Published as a conference paper at ICLR 2021
GRAPH CODE BERT: P RE-TRAINING CODE REPRESEN -
TATIONS WITH DATA FLOW
Daya Guo1∗, Shuo Ren2∗, Shuai Lu3∗, Zhangyin Feng4∗, Duyu Tang5, Shujie Liu5, Long Zhou5,
Nan Duan5, Alexey Svyatkovskiy6, Shengyu Fu6, Michele Tufano6, Shao Kun Deng6,
Colin Clement6, Dawn Drain6, Neel Sundaresan6, Jian Yin1, Daxin Jiang7, and Ming Zhou5
1School of Computer Science and Engineering, Sun Yat-sen University.
2Beihang University, 3Peking University, 4Harbin Institute of Technology,
5Microsoft Research Asia, 6Microsoft Devdiv, 7Microsoft STCA
ABSTRACT
Pre-trained models for programming language have achieved dramatic empirical
improvements on a variety of code-related tasks such as code search, code comple-
tion, code summarization, etc. However, existing pre-trained models regard a code
snippet as a sequence of tokens, while ignoring the inherent structure of code, which
provides crucial code semantics and would enhance the code understanding process.
We present GraphCodeBERT, a pre-trained model for programming language that
considers the inherent structure of code. Instead of taking syntactic-level structure
of code like abstract syntax tree (AST), we use data ﬂow in the pre-training stage,
which is a semantic-level structure of code that encodes the relation of “where-
the-value-comes-from” between variables. Such a semantic-level structure is less
complex and does not bring an unnecessarily deep hierarchy of AST, the property
of which makes the model more efﬁcient. We develop GraphCodeBERT based
on Transformer. In addition to using the task of masked language modeling, we
introduce two structure-aware pre-training tasks. One is to predict code structure
edges, and the other is to align representations between source code and code
structure. We implement the model in an efﬁcient way with a graph-guided masked
attention function to incorporate the code structure. We evaluate our model on four
tasks, including code search, clone detection, code translation, and code reﬁne-
ment. Results show that code structure and newly introduced pre-training tasks can
improve GraphCodeBERT and achieves state-of-the-art performance on the four
downstream tasks. We further show that the model prefers structure-level attentions
over token-level attentions in the task of code search.1
1 I NTRODUCTION
Pre-trained models such as ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin
et al., 2018) have led to strong improvement on numerous natural language processing (NLP) tasks.
These pre-trained models are ﬁrst pre-trained on a large unsupervised text corpus, and then ﬁne-tuned
on downstream tasks. The success of pre-trained models in NLP also promotes the development of
pre-trained models for programming language. Existing works (Kanade et al., 2019; Karampatsis &
Sutton, 2020; Feng et al., 2020; Svyatkovskiy et al., 2020; Buratti et al., 2020) regard a source code
as a sequence of tokens and pre-train models on source code to support code-related tasks such as
code search, code completion, code summarization, etc. However, previous works only utilize source
code for pre-training, while ignoring the inherent structure of code. Such code structure provides
useful semantic information of code, which would beneﬁt the code understanding process. Taking
the expression v= max value−min valueas an example, vis computed from max valueand
min value. Programmers do not always follow the naming conventions so that it’s hard to understand
the semantic of the variable vonly from its name. The semantic structure of code provides a way to
understand the semantic of the variable vby leveraging dependency relation between variables.
∗Work done while this author was an intern at Microsoft Research Asia. Contact: Daya Guo
(guody5@mail2.sysu.edu.cn)
1All the codes and data are available at https://github.com/microsoft/CodeBERT.
1
arXiv:2009.08366v4  [cs.SE]  13 Sep 2021
Published as a conference paper at ICLR 2021
In this work, we present GraphCodeBERT, a pre-trained model for programming language that
considers the inherent structure of code. Instead of taking syntactic-level structure of code like
abstract syntax tree (AST), we leverage semantic-level information of code, i.e. data ﬂow, for pre-
training. Data ﬂow is a graph, in which nodes represent variables and edges represent the relation of
“where-the-value-comes-from” between variables. Compared with AST, data ﬂow is less complex
and does not bring an unnecessarily deep hierarchy, the property of which makes the model more
efﬁcient. In order to learn code representation from source code and code structure, we introduce two
new structure-aware pre-training tasks. One is data ﬂow edges prediction for learning representation
from code structure, and the other is variable-alignment across source code and data ﬂow for aligning
representation between source code and code structure. GraphCodeBERT is based on Transformer
neural architecture (Vaswani et al., 2017) and we extend it by introducing a graph-guided masked
attention function to incorporate the code structure.
We pre-train GraphCodeBERT on the CodeSearchNet dataset (Husain et al., 2019), which includes
2.3M functions of six programming languages paired with natural language documents. We evaluate
the model on four downstream tasks: natural language code search, clone detection, code translation,
and code reﬁnement. Experiments show that our model achieves state-of-the-art performance on the
four tasks. Further analysis shows that code structure and newly introduced pre-training tasks can
improve GraphCodeBERT and the model has consistent preference for attending data ﬂow.
In summary, the contributions of this paper are: (1) GraphCodeBERT is the ﬁrst pre-trained model
that leverages semantic structure of code to learn code representation. (2) We introduce two new
structure-aware pre-training tasks for learning representation from source code and data ﬂow. (3)
GraphCodeBERT provides signiﬁcant improvement on four downstream tasks, i.e. code search, clone
detection, code translation, and code reﬁnement.
2 R ELATED WORKS
Pre-Trained Models for Programming LanguagesInspired by the big success of pre-training in
NLP (Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019; Raffel et al., 2019), pre-trained models
for programming languages also promotes the development of code intelligence (Kanade et al., 2019;
Feng et al., 2020; Karampatsis & Sutton, 2020; Svyatkovskiy et al., 2020; Buratti et al., 2020).
Kanade et al. (2019) pre-train a BERT model on a massive corpus of Python source codes by masked
language modeling and next sentence prediction objectives. Feng et al. (2020) propose CodeBERT, a
bimodal pre-trained model for programming and natural languages by masked language modeling and
replaced token detection to support text-code tasks such as code search. Karampatsis & Sutton (2020)
pre-train contextual embeddings on a JavaScript corpus using the ELMo framework for program
repair task. Svyatkovskiy et al. (2020) propose GPT-C, which is a variant of the GPT-2 trained from
scratch on source code data to support generative tasks like code completion. Buratti et al. (2020)
present C-BERT, a transformer-based language model pre-trained on a collection of repositories
written in C language, and achieve high accuracy in the abstract syntax tree (AST) tagging task.
Different with previous works, GraphCodeBERT is the ﬁrst pre-trained model that leverages code
structure to learn code representation to improve code understanding. We further introduce a graph-
guided masked attention function to incorporate the code structure into Transformer and two new
structure-aware pre-training tasks to learn representation from source code and code structure.
Neural Networks with Code StructureIn recent years, some neural networks leveraging code
structure such as AST have been proposed and achieved strong performance in code-related tasks like
code completion (Li et al., 2017; Alon et al., 2019; Kim et al., 2020), code generation (Rabinovich
et al., 2017; Yin & Neubig, 2017; Brockschmidt et al., 2018), code clone detection (Wei & Li, 2017;
Zhang et al., 2019; Wang et al., 2020), code summarization (Alon et al., 2018; Hu et al., 2018) and so
on (Nguyen & Nguyen, 2015; Allamanis et al., 2018; Hellendoorn et al., 2019). Nguyen & Nguyen
(2015) propose an AST-based language model to support the detection and suggestion of a syntactic
template at the current editing location. Allamanis et al. (2018) use graphs to represent programs
and graph neural network to reason over program structures. Hellendoorn et al. (2019) propose two
different architectures using a gated graph neural network and Transformers for combining local
and global information to leverage richly structured representations of source code. However, these
2
Published as a conference paper at ICLR 2021
works leverage code structure to learn models on speciﬁc tasks from scratch without using pre-trained
models. In this work, we study how to leverage code structure for pre-training code representation.
3 D ATA FLOW
In this section, we describe the basic concept and extraction of data ﬂow. In next section, we will
describe how to use data ﬂow for pre-training.
Data ﬂow is a graph that represents dependency relation between variables, in which nodes represent
variables and edges represent where the value of each variable comes from. Unlike AST, data ﬂow
is same under different abstract grammars for the same source code. Such code structure provides
crucial code semantic information for code understanding. Taking v= max value−min valueas
an example, programmers do not always follow the naming conventions so that it is hard to understand
the semantic of the variable. Data ﬂow provides a way to understand the semantic of the variable vto
some extent, i.e. the value of vcomes from max valueand min valuein data ﬂow. Besides, data
ﬂow supports the model to consider long-range dependencies induced by using the same variable
or function in distant locations. Taking Figure 1 as an example, there are four variables with same
name (i.e. x3, x7, x9 and x11) but with different semantic. The graph in the ﬁgure shows dependency
relation between these variables and supports x11 to pay more attention to x7 and x9 instead of x3.
Next, we describe how to extract data ﬂow from a source code.
defmax(a,b):x=0ifb>a:x=belse:x=areturnx
SourcecodeParseintoAST
CompilerTool
Identifyvariable sequence5
13 67 8
910
4 2
11
defmax(a,b):x=0ifb>a:x=belse:x=areturnx
Identify variable sequence in AST
Variable relation ab
x0babxaxx
1 2
3 4
6 5
1089 7
11
Extract variable relation from AST
Valuecomesfrom…
function definition
nameparameters
maxa b body
expressionstatement
assignment
leftright
x 0
If statementreturn statement
xcondition
left right
a b
operator
>
alternative
…
Figure 1: The procedure of extracting data ﬂow given a source code. The graph in the rightmost is
data ﬂow that represents the relation of ”where-the-value-comes-from” between variables.
Figure 1 shows the extraction of data ﬂow through a source code. Given a source code C =
{c1,c2,...,c n}, we ﬁrst parse the code into an abstract syntax tree (AST) by a standard compiler
tool2. The AST includes syntax information of the code and terminals (leaves) are used to identify
the variable sequence, denoted as V = {v1,v2,...,v k}. We take each variable as a node of the graph
and an direct edge ε= ⟨vi,vj⟩from vi to vj refers that the value of j-th variable comes from i-th
variable. Taking x= expras an example, edges from all variables in exprto xare added into the
graph. We denote the set of directed edges as E = {ε1,ε2,...,ε l}and the graph G(C) = (V,E) is
data ﬂow used to represent dependency relation between variables of the source code C.
4 G RAPH CODE BERT
In this section, we describe GraphCodeBERT, a graph-based pre-trained model based on Transformer
for programming language. We introduce model architecture, graph-guided masked attention and
pre-training tasks including standard masked language model and newly introduced ones. More
details about model pre-training setting are provided in the Appendix A.
2https://github.com/tree-sitter/tree-sitter
3
Published as a conference paper at ICLR 2021
Variable Sequence
Value comes from
… x=0 if b>  a  : x=b else …Return maximum value
Text Code
a
1
b
2
x
3
0
4
b
5
a
6
x
7
b
8
x
9
a
10
x
11
Source code
Comment
Return maximum value
Data Flow
[CLS] … x=0 if b> [MASK] : x=b else … [SEP]Return [MASK] value [SEP] a
1
b
2
x
3
0
4
b
5
a
6
x
7
b
8
x
9
a
10
x
11
…
L1
L2
L12
GraphCodeBERT
maximum a
1 2 4 5 6 8 10
data flow edge 
prediction among 
variables
7 9 113xx x
variable-alignment 
across source code 
and data flow
Masked Language Modeling
11
109
87
65
43
21
def max(a, b):
x=0
if b>a:
x=b
else:
x=a
return x
Figure 2: An illustration about GraphCodeBERT pre-training. The model takes source code paired
with comment and the corresponding data ﬂow as the input, and is pre-trained using standard masked
language modeling (Devlin et al., 2018) and two structure-aware tasks. One structure-aware task is to
predict where a variable is identiﬁed from (marked with orange lines) and the other is data ﬂow edges
prediction between variables (marked with blue lines).
4.1 M ODEL ARCHITECTURE
Figure 2 shows the model architecture of GraphCodeBERT. We follow BERT (Devlin et al., 2018)
and use the multi-layer bidirectional Transformer (Vaswani et al., 2017) as the model backbone.
Instead of only using source code, we also utilize paired comments to pre-train the model to support
more code-related tasks involving natural language such as natural language code search (Feng et al.,
2020). We further take data ﬂow, which is a graph, as a part of the input to the model.
Given a source code C = {c1,c2,...,c n}with its comment W = {w1,w2,...,w m}, we can obtain
the corresponding data ﬂow G(C) = (V,E) as discussed in the Section 3, whereV = {v1,v2,...,v k}
is a set of variables and E = {ε1,ε2,...,ε l}is a set of direct edges that represent where the value of
each variable comes from. We concatenate the comment, source code and the set of variables as the
sequence input X = {[CLS],W,[SEP],C, [SEP],V }, where [CLS] is a special token in front of
three segments and [SEP] is a special symbol to split two kinds of data types.
GraphCodeBERT takes the sequence Xas the input and then converts the sequence into input vectors
H0. For each token, its input vector is constructed by summing the corresponding token and position
embeddings. We use a special position embedding for all variables to indicate that they are nodes
of data ﬂow. The model applies N transformer layers over the input vectors to produce contextual
representations Hn = transformern(Hn−1),n ∈[1,N]. Each transformer layer contains an
architecturally identical transformer that applies a multi-headed self-attention operation (Vaswani
et al., 2017) followed by a feed forward layer over the input Hn−1 in the n-th layer.
Gn = LN(MultiAttn(Hn−1) +Hn−1) (1)
Hn = LN(FFN (Gn) +Gn) (2)
where MultiAttn is a multi-headed self-attention mechanism, FFN is a two layers feed forward
network, and LN represents a layer normalization operation. For the n-th transformer layer, the
output ˆGn of a multi-headed self-attention is computed via:
Qi = Hn−1WQ
i , Ki = Hn−1WK
i , Vi = Hn−1WV
i (3)
headi = softmax(QiKT
i√dk
+ M)Vi (4)
ˆGn = [head1; ...; headu]WO
n (5)
where the previous layer’s outputHn−1 ∈R|X|×dh is linearly projected to a triplet of queries, keys
and values using model parameters WQ
i ,WK
i ,WV
i ∈Rdh×dk , respectively. uis the number of heads,
dk is the dimension of a head, and WO
n ∈Rdh×dh is the model parameters. M ∈R|X|×|X|is a mask
matrix, where Mij is 0 if i-th token is allowed to attend j-th token otherwise −∞.
4
Published as a conference paper at ICLR 2021
4.2 G RAPH -GUIDED MASKED ATTENTION
To incorporate the graph structure into Transformer, we deﬁne a graph-guided masked attention
function to ﬁlter out irrelevant signals. The attention masking function could avoid the keyki attended
by the query qj by adding the attention score qT
j ki an inﬁnitely negative value so that the attention
weight becomes zero after using a softmax function. To represent dependency relation between
variables, a node-query qvi is allowed to attend to a node-key kvj if there is a direct edge from the
node vj to the node vi (i.e. ⟨vj,vi⟩∈ E) or they are the same node (i.e. i = j). Otherwise, the
attention is masked by adding an inﬁnitely negative value into the attention score. To represent
the relation between source code tokens and nodes of the data ﬂow, we ﬁrst deﬁne a set E
′
, where
⟨vi,cj⟩/⟨cj,vi⟩∈ E
′
if the variable vi is identiﬁed from the source code token cj. We then allow the
node qvi and code kcj attend each other if and only if ⟨vi,cj⟩/⟨cj,vi⟩∈ E
′
. More formally, we use
the following graph-guided masked attention matrix as the mask matrix M in the equation 4:
Mij =
{0 if qi ∈{[CLS],[SEP]}or qi,kj ∈W ∪Cor ⟨qi,kj⟩∈ E∪E
′
−∞ otherwise (6)
4.3 P RE-TRAINING TASKS
We describe three pre-training tasks used for pre-training GraphCodeBERT in this section. The ﬁrst
task is masked language modeling (Devlin et al., 2018) for learning representation from the source
code. The second task is data ﬂow edge prediction for learning representation from data ﬂow, where
we ﬁrst mask some variables’ data ﬂow edges and then let GraphCodeBERT predict those edges. The
last task is variable-alignment across source code and data ﬂow for aligning representation between
source code and data ﬂow, which predicts where a variable is identiﬁed from.
Masked Language Modeling We follow Devlin et al. (2018) to apply masked language modeling
(MLM) pre-training task. Specially, we sample randomly 15% of the tokens from the source code and
paired comment. We replace them with a [MASK] token 80% of the time, with a random token 10%
of the time, and leave them unchanged 10% of the time. The MLM objective is to predict original
tokens of these sampled tokens, which has proven effective in previous works (Devlin et al., 2018;
Liu et al., 2019; Feng et al., 2020). In particular, the model can leverage the comment context if the
source code context is not sufﬁcient to infer the masked code token, encouraging the model to align
the natural language and programming language representations.
Edge Prediction To learn representation from data ﬂow, we introduce a pre-training task of data
ﬂow edges prediction. The motivation is to encourage the model to learn structure-aware repre-
sentation that encodes the relation of “where-the-value-comes-from” for better code understanding.
Specially, we randomly sample 20% of nodes Vs in data ﬂow, mask direct edges connecting these
sampled nodes by add an inﬁnitely negative value in the mask matrix, and then predict these masked
edges Emask. Taking the variable x11 in Figure 2 for an example, we ﬁrst mask edges ⟨x7,x11⟩
and ⟨x9,x11⟩in the graph and then let the model to predict these edges. Formally, the pre-training
objective of the task is calculated as Equation 7, where Ec = Vs ×V ∪V ×Vs is a set of candidates
for edge prediction, δ(eij ∈E) is 1 if ⟨vi,vj⟩∈ Eotherwise 0, and the probability peij of existing
an edge from i-th to j-th node is calculated by dot product following a sigmoid function using
representations of two nodes from GraphCodeBERT. To balance positive-negative ratio of examples,
we sample negative and positive samples with the same number for Ec.
lossEdgePred = −
∑
eij∈Ec
[δ(eij ∈Emask)logpeij + (1−δ(eij ∈Emask))log(1 −peij )] (7)
Node Alignment To align representation between source code and data ﬂow, we introduce a pre-
training task of node alignment across source code and data ﬂow, which is similar to data ﬂow edge
prediction. Instead of predicting edges between nodes, we predict edges between code tokens and
nodes. The motivation is to encourage the model to align variables and source code according to
data ﬂow. Taking Figure 3 for an example, we ﬁrst mask edges between the variable x11 in data ﬂow
and code tokens, and then predict which code token the variable x11 in data ﬂow is identiﬁed from.
As we can see, the model could predict that the variable x11 is identiﬁed form the variable xin the
expression “return x” according to data ﬂow information (i.e. the value ofx11 comes from x7 or x9).
5
Published as a conference paper at ICLR 2021
GraphCodeBERT
Variable SequenceText Code
[CLS] Return [MASK] value [SEP] def max ( a , b ) : x=0 if b>a : x=b else: x=a return x [SEP] 𝑎𝑎1 𝑏𝑏2 𝑥𝑥3 04 𝑏𝑏5 𝑎𝑎6 𝑥𝑥7 𝑏𝑏8 𝑥𝑥9 𝑎𝑎10 𝑥𝑥11
Predict which code token the variable 𝑥𝑥11 in 
data flow is identified from 
Mask edges between the variable 𝑥𝑥11 in data flow and code tokens
Figure 3: An example of the Node Alignment task.
Specially, we randomly sample 20% nodes V
′
s in the graph, mask edges between code tokens and
sampled nodes, and then predict masked edges E
′
mask. The pre-training objective of this task is
similar to Equation 7, where E
′
c = V
′
s ×Cis a set of candidates for node alignment. Similarly, we
also sample negative and positive samples with the same number for E
′
c.
lossNodeAlign = −
∑
eij∈E′
c
[δ(eij ∈E
′
mask)logpeij + (1−δ(eij ∈E
′
mask))log(1 −peij )] (8)
5 E XPERIMENTS
We evaluate our model on four downstream tasks, including code search, clone detection, code
translation and code reﬁnement. Detailed experimental settings can be found in the Appendix.
5.1 N ATURAL LANGUAGE CODE SEARCH
Given a natural language as the input, the task aims to ﬁnd the most semantically related code from a
collection of candidate codes. We conduct experiments on the CodeSearchNet code corpus (Husain
et al., 2019), which includes six programming languages. Different from the dataset and the setting
used in the Husain et al. (2019), we ﬁlter low-quality queries by handcrafted rules and expand 1000
candidates to the whole code corpus, which is closer to the real-life scenario. We use Mean Reciprocal
Rank (MRR) as our evaluation metric and report results of existing methods in the Table 1. We
provide more details about the ﬁltered dataset and also give results using the same setting of Husain
et al. (2019) in the Appendix B.
model Ruby Javascript Go Python Java Php Overall
NBow 0.162 0.157 0.330 0.161 0.171 0.152 0.189
CNN 0.276 0.224 0.680 0.242 0.263 0.260 0.324
BiRNN 0.213 0.193 0.688 0.290 0.304 0.338 0.338
selfAtt 0.275 0.287 0.723 0.398 0.404 0.426 0.419
RoBERTa 0.587 0.517 0.850 0.587 0.599 0.560 0.617
RoBERTa (code) 0.628 0.562 0.859 0.610 0.620 0.579 0.643
CodeBERT 0.679 0.620 0.882 0.672 0.676 0.628 0.693
GraphCodeBERT 0.703 0.644 0.897 0.692 0.691 0.649 0.713
Table 1: Results on code search. GraphCodeBERT outperforms other models signiﬁcantly (p< 0.01).
All models calculate inner product of code and query encodings as relevance scores to rank candidate
codes. We follow Husain et al. (2019) to implement four methods as baselines in the ﬁrst group to
obtain the encodings, including bag-of-words, convolutional neural network, bidirectional recurrent
neural network, and multi-head attention. The second group is the results of pre-trained models.
Roberta (Liu et al., 2019) is a pre-trained model on text corpus with MLM learning objective,
while RoBERTa (code)is pre-trained only on code. CodeBERT (Feng et al., 2020) is pre-trained
6
Published as a conference paper at ICLR 2021
on code-text pairs with MLM and replaced token detection learning objectives. As we can see,
GraphCodeBERT that leverages code structure for pre-training brings a 2% gain of MRR, achieving
the state-of-art performance. We also conducted t-test between our GraphCodeBERT and other
baselines, and the results show the improvements are signiﬁcant with p< 0.01.
5.2 C ODE CLONE DETECTION
Code clones are multiple code fragments that output similar results when given the same input. The
task aims to measure the similarity between two code fragments, which can help reduce the cost of
software maintenance and prevent bugs. We conduct experiments on the BigCloneBench dataset
(Svajlenko et al., 2014) and report results in the Table 2.
Deckard (Jiang et al., 2007) is to compute vectors for structural information within ASTs and
then a Locality Sensitive Hashing (LSH) (Datar et al., 2004) is used to cluster similar vectors for
detection. RtvNN (White et al., 2016) trains a recursive autoencoder to learn representations for
AST. CDLH (Wei & Li, 2017) learn representations of code fragments via AST-based LSTM and
hamming distance is used to optimize the distance between the vector representation of AST pairs.
Model Precision Recall F1
Deckard 0.93 0.02 0.03
RtvNN 0.95 0.01 0.01
CDLH 0.92 0.74 0.82
ASTNN 0.92 0.94 0.93
FA-AST-GMN 0.96 0.94 0.95
RoBERTa (code) 0.949 0.922 0.935
CodeBERT 0.947 0.934 0.941
GraphCodeBERT 0.948 0.952 0.950
Table 2: Results on code clone detection. Graph-
CodeBERT outperforms other pre-trained methods
signiﬁcantly (p< 0.01).
ASTNN Zhang et al. (2019) uses RNNs to
encode AST subtrees for statements, then
feed the encodings of all statement trees into
an RNN to learn representation for a pro-
gram. FA-AST-GMN (Wang et al., 2020)
uses GNNs over a ﬂow-augmented AST to
leverages explicit control and data ﬂow in-
formation for code clone detection. Results
show that our GraphCodeBERT that lever-
ages code structure information signiﬁcantly
outperforms other methods with p < 0.01,
which demonstrates the effectiveness of our
pre-trained model for the task of code clone
detection.
5.3 C ODE TRANSLATION
Code translation aims to migrate legacy software from one programming language in a platform to
another. Following Nguyen et al. (2015) and Chen et al. (2018), we conduct experiments on a dataset
crawled from the same several open-source projects as them and report results in the Table 3.
The Naive method is directly copying the source code as the translation result. PBSMT is short
for phrase-based statistical machine translation (Koehn et al., 2003), and has been exploited in
previous works (Nguyen et al., 2013; Karaivanov et al., 2014). As for the Transformer, we use the
Method Java→C# C#→Java
BLEU Acc BLEU Acc
Naive 18.54 0.0 18.69 0.0
PBSMT 43.53 12.5 40.06 16.1
Transformer 55.84 33.0 50.47 37.9
RoBERTa (code) 77.46 56.1 71.99 57.9
CodeBERT 79.92 59.0 72.14 58.8
GraphCodeBERT 80.58 59.4 72.64 58.8
Table 3: Results on code translation.
same number of layers and hidden size
as pre-trained models. To leverage the
pre-trained models for translation, we ini-
tialize the encoder with pre-trained mod-
els and randomly initialize parameters of
the decoder and the source-to-target atten-
tion. Results show that the models initial-
ized with pre-trained models (i.e the second
group) outperform PBSMT and Transformer
models. Among them, GraphCodeBERT
achieves state-of-art performance, which
demonstrates the effectiveness of our model
for code translation.
5.4 C ODE REFINEMENT
Code reﬁnement aims to automatically ﬁx bugs in the code, which can contribute to reducing the cost
of bug-ﬁxes. We use the dataset released by Tufano et al. (2019) and report results in the Table 4.
7
Published as a conference paper at ICLR 2021
The Naive method directly copies the buggy code as the reﬁnement result. For the Transformer, we
use the same number of layers and hidden size as the pre-trained models. Same as the Section 5.3,
we initialize the encoder with pre-trained models and randomly initialize parameters of the decoder
Method small medium
BLEU Acc BLEU Acc
Naive 78.06 0.0 90.91 0.0
LSTM 76.76 10.0 72.08 2.5
Transformer 77.21 14.7 89.25 3.7
RoBERTa (code) 77.30 15.9 90.07 4.1
CodeBERT 77.42 16.4 91.07 5.2
GraphCodeBERT 80.02 17.3 91.31 9.1
Table 4: Results on code reﬁnement.
and the source-to-target attention. Then
we use the training data to ﬁne-tune
the whole model. In the table, we see
that the Transformer signiﬁcantly out-
performs LSTM. Results in the second
group shows that pre-trained models out-
perform Transformer models further, and
GraphCodeBERT achieves better per-
formance than other pre-trained models
on both datasets, which shows leveraging
code structure information are helpful to
the task of code reﬁnement.
5.5 M ODEL ANALYSIS
Ablation Study We conduct ablation study on the task of natural language code search to under-
stand various components in our approach impact overall performance. We remove two pre-training
tasks and data ﬂow, respectively, to analyze their contribution. Table 5 shows that the overall perfor-
mance drops from 71.3% to 70.3%∼70.7% when removing Node Alignment and Edge Prediction
pre-training tasks, respectively, which reveals the importance of two structure-aware pre-training
tasks. After ablating the data ﬂow totally, we can see that the performance drops from 71.3% to 69.3%,
which means leveraging data ﬂow to learn code representation could improve GraphCodeBERT.
Methods Ruby Javascript Go Python Java Php Overall
GraphCodeBERT 0.703 0.644 0.897 0.692 0.691 0.649 0.713
-w/o EdgePred 0.701 0.632 0.894 0.687 0.688 0.640 0.707
-w/o NodeAlign 0.685 0.635 0.887 0.682 0.690 0.640 0.703
-w/o Data Flow 0.679 0.620 0.882 0.672 0.676 0.628 0.693
Table 5: Ablation study on natural language code search
Node-vs. Token-level Attention Table 6 shows how frequently a special token [CLS] that is used
to calculate probability of correct candidate attends to code tokens (Codes) and variables (Nodes).
We see that although the number of nodes account for 5%∼20%, attentions over nodes overwhelm
node/code ratio (around 10% to 32%) across all programming languages. The results indicate that
data ﬂow plays an important role in code understanding process and the model pays more attention to
nodes in data ﬂow than code tokens.
Ruby Javascript Go Python Java Php
Codes/Nodes 90.1/9.9 94.6/5.4 95.0/5.03 80.6/19.4 93.2/6.8 87.5/12.5
[CLS] →Codes/Nodes 82.3/17.7 89.7/10.3 91.0/9.0 67.7/32.3 87.8/12.2 79.4/20.6
Table 6: Attention distribution (%) between code tokens (codes) and variables (nodes) across different
programming language on natural language code search test sets. The ﬁrst row is the ratio of the
number of code tokens to nodes, and the second row is attention distribution of [CLS] token.
Comparison between AST and Data FlowFigure 4 shows MRR score with respect to input
sequence length on the validation dataset of Ruby programming language for the task of code
search. AST Pre-order Traversalregards AST as a sequence by linearizing all AST nodes us-
ing pre-order traversal algorithm. AST Subtree Maskingregards AST as a tree and introduce
subtree masking (Nguyen et al., 2019) for self-attention of the Transformer. In subtree masking,
each node-query in AST attends only to its own subtree descendants, and each leaf-query only
attends to leaves of AST. Transformer has a self-attention component withO(n2) time and memory
complexity where nis the input sequence length, and thus is not efﬁcient to scale to long inputs.
8
Published as a conference paper at ICLR 2021
64 96 128 192 256 512
Sequence Length
0.60
0.62
0.64
0.66
0.68
0.70
0.72
0.74
0.76MRR Score
w/o code structure
AST Pre-order Traversal
AST Subtree Masking
GraphCodeBERT
Figure 4: MRR score on the validation dataset of Ruby for code
search with varying length of input sequence.
We observe that injecting AST
even hurts the performance when
the sequence length is short (e.g.
shorter than 128), while Graph-
CodeBERT consistently brings
performance boost on varying se-
quence length and obtains better
MRR score than AST-based meth-
ods. The main reason is that data
ﬂow is less complex and the num-
ber of nodes account for 5% ∼
20% (see Table 6), which does not
bring an unnecessarily deep hierar-
chy of AST and makes the model
more accurate and efﬁcient.
Case Study We also give a case study to demonstrate that data ﬂow would enhance the code
understanding process. Given a source code and a comment, we use GraphCodeBERT with and
without data ﬂow to predict whether the comment correctly describes the source code. Results are
given in Figure 5. We can see that both models make correct prediction in the original example, where
the threshold is 0.5 (left panel). To study the code understanding ability of models, we change the
source code (center panel) and the comment (right panel), respectively. Although we make a small
change on the source code (returna →returnb) and the comment (sumvalue →meanvalue),
the semantic of the source code and the comment are completely different and corresponding gold
labels change from 1 to 0. As we can see in the ﬁgure, GraphCodeBERT without using data ﬂow
fails these tests and still outputs high probability for negative examples. After leveraging data ﬂow,
GraphCodeBERT better understands the semantic of source code and makes correct predictions on all
tests, which demonstrates that data ﬂow could improve the code understanding ability of the model.
Unchanged Code: 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑎𝑎 → 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑏𝑏 NL: 𝑠𝑠𝑟𝑟𝑠𝑠𝑣𝑣𝑎𝑎𝑣𝑣𝑟𝑟𝑟𝑟 → 𝑠𝑠𝑟𝑟𝑎𝑎𝑟𝑟𝑣𝑣𝑎𝑎𝑣𝑣𝑟𝑟𝑟𝑟
Input
Label 1 0 0
Prediction
GraphCodeBERT: 0.6563 (1)
GraphCodeBERT: 0.8728 (1)
(w/o Data Flow)
GraphCodeBERT:  0.4615 (0)
GraphCodeBERT:  0.8608 (1)
(w/o Data Flow)
GraphCodeBERT: 0.2884 (0)
GraphCodeBERT: 0.9048 (1)
(w/o Data Flow)
NL:
Return sum value of an array
Code:
import numpy as np
def  f(array):
a=np.sum(array)
b=np.mean(array)
return a
NL:
Return sum value of an array
Code:
import numpy as np
def  f(array):
a=np.sum(array)
b=np.mean(array)
return b
NL:
Return mean value of an array
Code:
import numpy as np
def  f(array):
a=np.sum(array)
b=np.mean(array)
return a
Figure 5: We take a comment and a source code as the input (ﬁrst row), and use GraphCodeBERT
with and without data ﬂow to predict the probability of the source code matching the comment (third
row). The label is 1 if the comment correctly describes the source code otherwise 0 (second row).
6 C ONCLUSION
In this paper, we present GraphCodeBERT that leverages data ﬂow to learn code representation.
To the best of our knowledge, this is the ﬁrst pre-trained model that considers code structure for
pre-training code representations. We introduce two structure-aware pre-training tasks and show
that GraphCodeBERT achieves state-of-the-art performance on four code-related downstream tasks,
including code search, clone detection, code translation and code reﬁnement. Further analysis shows
that code structure and newly introduced pre-training tasks boost the performance. Additionally, case
study in the task of code search shows that applying data ﬂow in the pre-trained model improves code
understanding.
9
Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
Daya Guo and Jian Yin are supported by the Research Foundation of Science and Technology Plan
Project in Guangdong Province (2017B030308007).
REFERENCES
Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. Learning to represent programs
with graphs. In International Conference on Learning Representations, 2018.
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. code2seq: Generating sequences from
structured representations of code. arXiv preprint arXiv:1808.01400, 2018.
Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. arXiv, pp.
arXiv–1910, 2019.
Marc Brockschmidt, Miltiadis Allamanis, Alexander L Gaunt, and Oleksandr Polozov. Generative
code modeling with graphs. arXiv preprint arXiv:1805.08490, 2018.
Luca Buratti, Saurabh Pujar, Mihaela Bornea, Scott McCarley, Yunhui Zheng, Gaetano Rossiello,
Alessandro Morari, Jim Laredo, Veronika Thost, Yufan Zhuang, et al. Exploring software natural-
ness throughneural language models. arXiv preprint arXiv:2006.12641, 2020.
Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree neural networks for program translation. In
Advances in neural information processing systems, pp. 2547–2557, 2018.
Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab S Mirrokni. Locality-sensitive hashing
scheme based on p-stable distributions. In Proceedings of the twentieth annual symposium on
Computational geometry, pp. 253–262, 2004.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing
Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural
languages. arXiv preprint arXiv:2002.08155, 2020.
Daya Guo, Duyu Tang, Nan Duan, M. Zhou, and Jian Yin. Dialog-to-action: Conversational question
answering over a large-scale knowledge base. In NeurIPS, 2018.
Daya Guo, Duyu Tang, Nan Duan, M. Zhou, and Jian Yin. Coupling retrieval and meta-learning for
context-dependent semantic parsing. ArXiv, abs/1906.07108, 2019.
Vincent J Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. Global
relational models of source code. In International Conference on Learning Representations, 2019.
Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. Deep code comment generation. In2018 IEEE/ACM
26th International Conference on Program Comprehension (ICPC), pp. 200–20010. IEEE, 2018.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Code-
searchnet challenge: Evaluating the state of semantic code search.arXiv preprint arXiv:1909.09436,
2019.
Lingxiao Jiang, Ghassan Misherghi, Zhendong Su, and Stephane Glondu. Deckard: Scalable and
accurate tree-based detection of code clones. In 29th International Conference on Software
Engineering (ICSE’07), pp. 96–105. IEEE, 2007.
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Pre-trained contextual
embedding of source code. arXiv preprint arXiv:2001.00059, 2019.
Svetoslav Karaivanov, Veselin Raychev, and Martin Vechev. Phrase-based statistical translation of
programming languages. In Proceedings of the 2014 ACM International Symposium on New Ideas,
New Paradigms, and Reﬂections on Programming & Software, pp. 173–184, 2014.
10
Published as a conference paper at ICLR 2021
Rafael-Michael Karampatsis and Charles Sutton. Scelmo: Source code embeddings from language
models. arXiv preprint arXiv:2004.13214, 2020.
Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. Code prediction by feeding trees to
transformers. arXiv preprint arXiv:2003.13848, 2020.
Philipp Koehn, Franz J Och, and Daniel Marcu. Statistical phrase-based translation. Technical
report, UNIVERSITY OF SOUTHERN CALIFORNIA MARINA DEL REY INFORMATION
SCIENCES INST, 2003.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint
arXiv:1901.07291, 2019.
Jian Li, Yue Wang, Michael R Lyu, and Irwin King. Code completion with neural attention and
pointer networks. arXiv preprint arXiv:1711.09573, 2017.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike
Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining
approach. arXiv preprint arXiv:1907.11692, 2019.
Anh Tuan Nguyen and Tien N Nguyen. Graph-based statistical language model for code. In 2015
IEEE/ACM 37th IEEE International Conference on Software Engineering, volume 1, pp. 858–868.
IEEE, 2015.
Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. Lexical statistical machine translation
for language migration. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software
Engineering, pp. 651–654, 2013.
Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. Divide-and-conquer approach for
multi-phase statistical migration for source code (t). In 2015 30th IEEE/ACM International
Conference on Automated Software Engineering (ASE), pp. 585–596. IEEE, 2015.
Xuan-Phi Nguyen, Shaﬁq Joty, Steven Hoi, and Richard Socher. Tree-structured attention with
hierarchical accumulation. In International Conference on Learning Representations, 2019.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and
Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365,
2018.
Maxim Rabinovich, Mitchell Stern, and Dan Klein. Abstract syntax networks for code generation
and semantic parsing. arXiv preprint arXiv:1704.07535, 2017.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language
understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-
assets/researchcovers/languageunsupervised/language understanding paper. pdf, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi
Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a uniﬁed text-to-text
transformer. arXiv preprint arXiv:1910.10683, 2019.
Jeffrey Svajlenko, Judith F Islam, Iman Keivanloo, Chanchal K Roy, and Mohammad Mamun Mia.
Towards a big data curated benchmark of inter-project code clones. In 2014 IEEE International
Conference on Software Maintenance and Evolution, pp. 476–480. IEEE, 2014.
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. Intellicode compose:
Code generation using transformer. arXiv preprint arXiv:2005.08025, 2020.
Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys
Poshyvanyk. An empirical study on learning bug-ﬁxing patches in the wild via neural machine
translation. ACM Transactions on Software Engineering and Methodology (TOSEM), 28(4):1–29,
2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,Łukasz
Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information
processing systems, pp. 5998–6008, 2017.
11
Published as a conference paper at ICLR 2021
Wenhan Wang, Ge Li, Bo Ma, Xin Xia, and Zhi Jin. Detecting code clones with graph neural
networkand ﬂow-augmented abstract syntax tree. arXiv preprint arXiv:2002.08653, 2020.
Huihui Wei and Ming Li. Supervised deep features for software functional clone detection by
exploiting lexical and syntactical information in source code. In IJCAI, pp. 3034–3040, 2017.
Martin White, Michele Tufano, Christopher Vendome, and Denys Poshyvanyk. Deep learning
code fragments for code clone detection. In 2016 31st IEEE/ACM International Conference on
Automated Software Engineering (ASE), pp. 87–98. IEEE, 2016.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V
Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint
arXiv:1906.08237, 2019.
Pengcheng Yin and Graham Neubig. A syntactic neural model for general-purpose code generation.
In The 55th Annual Meeting of the Association for Computational Linguistics (ACL), Vancouver,
Canada, July 2017. URL https://arxiv.org/abs/1704.01696.
Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. A novel neural
source code representation based on abstract syntax tree. In 2019 IEEE/ACM 41st International
Conference on Software Engineering (ICSE), pp. 783–794. IEEE, 2019.
A P RE-TRAINING DETAILS
GraphCodeBERT includes 12 layers Transformer with 768 dimensional hidden states and 12 attention
heads. For fair comparison, we use the same dataset as CodeBERT (Feng et al., 2020) to pretrain
our model. The dataset is the CodeSearchNet dataset3 (Husain et al., 2019), which includes 2.3M
functions with document pairs for six programming languages. We train the model on two DGX-2
machines, each having 16 NVIDIA Tesla V100 with 32GB memory. We set the max length of
sequences and nodes as 512 and 128, respectively. We use the Adam optimizer to update model
parameters with 1,024 batch size and 2e-4 learning rate. To accelerate the training process, we adopt
the parameters of CodeBERT released by Feng et al. (2020) to initialize the model. The model is
trained with 200K batches and costs about 83 hours.
At each iteration, we alternate EdgePred and NodeAlign objectives in combination with MLM to
pre-train the model. And we follow Lample & Conneau (2019) to sample each batch from the same
programming language according to a multinomial distribution with probabilities {qi}i=1...N, where
ni is number of examples for i-th programming language and α=0.7. Sampling with this distribution
could alleviates the bias towards high-resource languages.
qi = pα
i∑j=1
N pα
j
with pi = ni
∑k=1
N nk
(9)
B N ATURAL LANGUAGE CODE SEARCH
Given a natural language as the input, code search aims to ﬁnd the most semantically related code
from a collection of candidate codes. We conduct experiments on the CodeSearchNet code corpus
(Husain et al., 2019) and follow Husain et al. (2019) to take the ﬁrst paragraph of the documentation
as the query for the corresponding function. However, we observe that some queries contain content
unrelated to the code, such as a link “http://...” that refers to external resources. Therefore, we ﬁlter
following examples to improve the quality of the dataset.
(1) Examples whose code could not be parsed into abstract syntax tree.
(2) Examples whose query tokens number is shorter than 3 or larger than 256.
(3) Examples whose query contains special tokens such as “http://”.
(4) Examples whose query is empty or not written in English.
3https://github.com/github/CodeSearchNet
12
Published as a conference paper at ICLR 2021
Different from the setting of Husain et al. (2019), the answer of each query is retrieved from the
whole development and testing code corpus instead of 1,000 candidate codes. We list data statistics
about the ﬁltered dataset in Table 7.
Code Search Training examples Dev queries Testing queries Candidate codes
Go 167,288 7,325 8,122 28,120
Java 164,923 5,183 10,955 40,347
JavaScript 58,025 3,885 3,291 13,981
PHP 241,241 12,982 14,014 52,660
Python 251,820 13,914 14,918 43,827
Ruby 24,927 1,400 1,261 4,360
Table 7: Data statistics about the ﬁltered dataset. For each query in the development and testing sets,
the answer is retrieved from the whole candidate codes (i.e. the last row).
We use GraphCodeBERT to separately encode query and source code with data ﬂow, and calculate
inner product of their representations of the special token[CLS] as relevance scores to rank candidate
codes. In the ﬁne-turning step, we set the learning rate as 2e-5, the batch size as 32, the max sequence
length of queries and codes as 128 and 256, and the max number of nodes as 64. We use the Adam
optimizer to update model parameters and perform early stopping on the development set.
We also report the results using the same setting of Husain et al. (2019) in Table 8. In this setting,
models are required to retrieve an answer for a query from 1000 candidates. The results show that
GraphCodeBERT also achieves the state-of-the-art performance.
model Ruby Javascript Go Python Java Php Overall
NBow 0.429 0.461 0.641 0.581 0.514 0.484 0.518
CNN 0.245 0.352 0.627 0.571 0.527 0.529 0.475
BiRNN 0.084 0.153 0.452 0.321 0.287 0.251 0.258
selfAtt 0.365 0.451 0.681 0.692 0.587 0.601 0.563
RoBERTa 0.625 0.606 0.820 0.809 0.666 0.658 0.697
RoBERTa (code) 0.661 0.640 0.819 0.844 0.721 0.671 0.726
CodeBERT 0.693 0.706 0.840 0.869 0.748 0.706 0.760
GraphCodeBERT 0.732 0.711 0.841 0.879 0.757 0.725 0.774
Table 8: Results on natural language code search using the setting of Husain et al. (2019).
C C ODE CLONE DETECTION
Code clone detection aims to measure the similarity between two code fragments. We use Big-
CloneBench dataset (Svajlenko et al., 2014), which contains over 6,000,000 true clone pairs and
260,000 false clone pairs from 10 different functionalities. We follow the settings in Wei & Li (2017),
discarding code fragments without any tagged true and false clone pairs and using 9,134 remaining
code fragments. Finally, the dataset provided by Wang et al. (2020) includes 901,724/416,328/416,328
examples for training/validation/testing. We treat the task as a binary classiﬁcation to ﬁne-tune Graph-
CodeBERT, where we use source code and data ﬂow as the input. The probability of true clone
is calculated by dot product from the representation of [CLS]. In the ﬁne-turning step, we set the
learning rate as 2e-5, the batch size as 16, the max sequence length as 512 the max number of nodes
as 128. We use the Adam optimizer to update model parameters and tune hyper-parameters and
perform early stopping on the development set.
We give a case of the GraphCodeBERT output for this task in Figure 6. In this example, two Java
source codes both download content from a given URL and convert the type of the content into string
type. Therefore, two codes are semantically similar since they output similar results when given the
same input. As we can see, our model gives a high score for this case and the pair is classiﬁed as true
clone pair.
13
Published as a conference paper at ICLR 2021
protected String downloadURLtoString(URL url) throws IOException
{
BufferedReader in = new BufferedReader(new
InputStreamReader(url.openStream()));
StringBuffer sb = new StringBuffer(100 * 1024);
String str;
while ((str = in.readLine()) != null) {
sb.append(str);
}
in.close();
return sb.toString();
}
public static String fetchUrl(String urlString) 
{
try {
URL url = new URL(urlString);
BufferedReader reader = new BufferedReader(new
InputStreamReader(url.openStream()));
String line = null;
StringBuilder builder = new StringBuilder();
while ((line = reader.readLine()) != null) {
builder.append(line);
}
reader.close();
return builder.toString();
} catch (MalformedURLException e) {
} catch (IOException e) {
}
return “";
}
Input: Two source codes Output: Semantically similar (score: 0.983)
Figure 6: A case of GraphCodeBERT output for the code clone detection task.
D C ODE TRANSLATION
Code translation aims to migrate legacy software from one programming language in a platform to
another. We conduct experiments on a dataset crawled from the same several open-source projects as
Nguyen et al. (2015) and Chen et al. (2018), i.e. Lucene 4, POI5, JGit6 and Antlr7. We do not use
Itext8 and JTS9 as they do because of the license problem. Those projects have both Java and C#
implementation. We pair the methods in the two languages based on their ﬁle names and method
names. After removing duplication and methods with null function body, the total number of method
pairs is 11,800, and we split 500 pairs from them as the development set and another 1,000 pairs for
test. To demonstrate the effectiveness of GraphCodeBERT on the task of code translation, we adopt
various pre-trained models as encoders and stay hyperparameters consistent. We set the learning rate
as 1e-4, the batch size as 32, the max sequence length as 256 and the max number of nodes as 64. We
use the Adam optimizer to update model parameters and tune hyper-parameters and perform early
stopping on the development set.
We give a case of the GraphCodeBERT output for this task in Figure 7. In this example, the model
successfully translates a piece of Java code into its C# version. The differences include the type
name (from “boolean” to “bool”) and the usage of getting a string value of a bool variable (from
“String.valueOf(b)” to “b.ToString()”).
Figure 7: A case of GraphCodeBERT output for the code translation task.
4http://lucene.apache.org/
5http://poi.apache.org/
6https://github.com/eclipse/jgit/
7https://github.com/antlr/
8http://sourceforge.net/projects/itext/
9http://sourceforge.net/projects/jts-topo-suite/
14
Published as a conference paper at ICLR 2021
E C ODE REFINEMENT
Code reﬁnement aims to automatically ﬁx bugs in the code. We use the dataset released by Tufano
et al. (2019). The source is buggy Java functions while the target is the according ﬁxed ones. Almost
all the names of variables and custom methods are normalized. The dataset contains two subsets
based on the code length. For the small dataset, the numbers of training, development and test
samples are 46,680, 5,835 and 5,835. For the medium dataset, the numbers are 52,364, 6,545 and
6,545. We also use the sequence-to-sequence Transformer model to conduct the experiments. In the
ﬁne-tuning step, we adopt various pre-trained models as encoders. We set the learning rate as 1e-4,
the batch size as 32, the max sequence length as 256 and the max number of nodes as 64. We use the
Adam optimizer to update model parameters and perform early stopping on the development set.
We give two cases of the GraphCodeBERT output for this task in Figure 8. In the ﬁrst example, the
model successfully ﬁxes the operation bug (from “*” to “+”) to match the function name “add”. In
the second case, the source function and type names are normalized. The return type of this function
is “void” but the buggy code gives a return value. Our model successfully removes the “return” word
so that the return type of the function matches its declaration.
Figure 8: Two cases of GraphCodeBERT output for the code reﬁnement task.
F C ASE STUDY
F.1 N ATURAL LANGUAGE CODE SEARCH
We give a case study to illustrate retrieved results by GraphCodeBERT on the natural language code
search task, with a comparison to CodeBERT and RoBERTa (code) models. Two examples are given
in Figure 9 and we can see that GraphCodeBERT successfully retrieves correct source codes for
given queries on both examples. As we can see in the ﬁrst case, incorporating data ﬂow will help
Graph-CodeBERT better understand the complicated expression “[(k, v) for k, v in self.items() if v
is not self.EMPTY]” by leveraging dependency relation among variables in data ﬂow graph. In the
second case, the terminology “%Y-%m-%d” in Python program language is a format of date time.
GraphCodeBERT and CodeBERT both successfully search the correct function. Compared with
RoBERTa (code), the second case shows that utilizing natural language descriptions for pre-training
helps models do better semantic matching between source codes and queries on the code search task.
F.2 C ODE CLONE DETECTION
We give a case study to compare GraphCodeBERT with CodeBERT and RoBERTa (code) models
on code clone detection task. An example is shown in Figure 10. The ﬁrst source code is to
return the HTML content from a given URL, while the second source code is to return the last
line from a ﬁxed URL “http://kmttg.googlecode.com/svn/trunk/version”. Their semantics are not
similar due to their different outputs. Data ﬂow could help GraphCodeBERT better understand
that the return value “pageHTML” in ﬁrst source code comes from “pageHTML.append(line);
pageHTML.append(“\r\n”);” instead of “bufferedWriter.write(pageHTML.toString());” and the
return value “version” in the second source code comes from “version = inputLine” or “version =
null;”. Although two source codes are highly overlapped (marked in yellow), GraphCodeBERT
successfully predict the gold label compared with other models without data ﬂow.
15
Published as a conference paper at ICLR 2021
Case 1
Query: Return copy of instance, omitting entries that are EMPTY
Gold Source Code:
def defined_items(self):
return self.__class__(
[(k, v) for k, v in self.items() if v is not self.EMPTY], is_empty=False
)
Search Results (Top1)
GraphCodeBERT:
def defined_items(self):
return self.__class__(
[(k, v) for k, v in self.items() if v is not self.EMPTY], is_empty=False
)
CodeBERT:
def copy(self):
context = CLIContext()
for item in dir(self):
if item[0] != '_' and item not in ('copy', 'write_headers’):
setattr(context, item, getattr(self, item))
return context
RoBERTa(code):
def copy(self):
x = self.to_dict()
x.pop(self._pkey)
return self.__class__(**x)
Case 2
Query: Fast %Y-%m-%d parsing
Gold Source Code:
def parse_date(s):
try:
return datetime.date(int(s[:4]), int(s[5:7]), int(s[8:10]))
except ValueError:
return datetime.datetime.strptime(s, '%d %B %Y').date()
Search Results (Top1)
GraphCodeBERT:
def parse_date(s):
try:
return datetime.date(int(s[:4]), int(s[5:7]), int(s[8:10]))
except ValueError:
return datetime.datetime.strptime(s, '%d %B %Y').date()
CodeBERT:
def parse_date(s):
try:
return datetime.date(int(s[:4]), int(s[5:7]), int(s[8:10]))
except ValueError:
return datetime.datetime.strptime(s, '%d %B %Y').date()
RoBERTa(code):
def parse(self, hcl, canonicalize=False):
return self.request("parse", json={"JobHCL": hcl, "Canonicalize": 
canonicalize}, method="post", allow_redirects=True).json()
Figure 9: Two examples on code search task and retrieved results from different models.
Source code 1:
private String getHTML(String pageURL, String encoding, String dirPath) 
throws IOException {
StringBuilder pageHTML = new StringBuilder();
HttpURLConnection connection = null;
try {
URL url = new URL(pageURL);
connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty("User-Agent", "MSIE 7.0");
connection.connect();
BufferedReader br = new BufferedReader(new 
InputStreamReader(connection.getInputStream(), encoding));
String line = null;
while ((line = br.readLine()) != null) {
pageHTML.append(line);
pageHTML.append("\r\n");
}
} catch (Exception e) {
e.printStackTrace();
} finally {
connection.disconnect();
}
if (dirPath != null) {
File file = new File(dirPath);
BufferedWriter bufferedWriter = new BufferedWriter(new 
FileWriter(file));
bufferedWriter.write(pageHTML.toString());
bufferedWriter.close();
}
return pageHTML.toString();
}
Source code 2:
private static String getVersion() {
debug.print("");
String version = null;
String version_url = 
"http://kmttg.googlecode.com/svn/trunk/version";
try {
URL url = new URL(version_url);
URLConnection con = url.openConnection();
BufferedReader in = new BufferedReader(new 
InputStreamReader(con.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) version = inputLine;
in.close();
} catch (Exception ex) {
version = null;
}
return version;
}
Gold Label: 
No semantically similar
Prediction:
GraphCodeBERT: 
No semantically similar 
CodeBERT: 
semantically similar 
RoBERTa(code):
semantically similar 
Figure 10: An examples on code clone detection task and model prediction from different models.
Overlapped code snippets between two source codes are marked in yellow.
F.3 C ODE TRANSLATION AND CODE REFINEMENT
We give a case study to compare GraphCodeBERT with Transformer without using data ﬂow on
code generation tasks, including code translation and code reﬁnement. We list three cases in Table
9 and Table 10, respectively. [src] represents the source input, [ref] represents the reference, [sys]
represents Transformer without data ﬂow and [ours] represents GraphCodeBERT. We can see that the
Transformer ([sys]) baseline makes several mistakes, including repeating tokens, logic errors and
syntax errors, while GraphCodeBERT ([ours]) as a encoder could improve the generation.
16
Published as a conference paper at ICLR 2021
Case1: Transformer outputs repeating tokens
[src] public static ﬁnal WeightedTerm[] getTerms(Query query){return getTerms(query,false);}
[ref] public static WeightedTerm[] GetTerms(Query query){return GetTerms(query, false);}
[sys] public static WeightedTerm[] GetTerms(Query query){return GetTerms(false, new static
static static static static static WeightTerms);}
[ours] public static WeightedTerm[] GetTerms(Query query){return GetTerms(query, false);}
Case2: Transformer outputs codes with severe logic and syntax errors
[src] public long skip(long n){int s = (int) Math.min(available(), Math.max(0, n));ptr += s;return s;}
[ref] public override long Skip(long n){int s = (int)Math.Min(Available(), Math.Max(0, n));
ptr += s;return s;}
[sys] public override long Skip(long n){int s = Math.Min(n) == 0 ? Math.Min(00.0 :
Math.Min(n, s.Length);return s;}
[ours] public override long Skip(long n){int s = (int)Math.Min(Available(), Math.Max(0, n));
ptr += s;return s;}
Case3: Transformer uses the wrong variable as a parameter.
[src] public UnbufferedCharStream(int bufferSize){n = 0;data = new int[bufferSize];}
[ref] public UnbufferedCharStream(int bufferSize){n = 0;data = new int[bufferSize];}
[sys] public UnbufferedCharStream(int bufferSize){data = new int[data];}
[ours] public UnbufferedCharStream(int bufferSize){n = 0;data = new int[bufferSize];}
Table 9: Three examples that translate from Java to C# programming language on code translation
task. [src] represents the source input, [ref] represents the reference, [sys] represents Transformer
without data ﬂow and [ours] represents GraphCodeBERT.
Case1: Transformer adds redundant parameters (android.view.View view)
[src] public void METHOD 1 ( ) { android.content.Intent V AR1 = new android.content.Intent
( V AR2 ) ; METHOD 2 ( V AR1 , 0 ) ; android.content.Intent i = new android.content.Intent
( this , V AR3 class ) ; METHOD 3 ( i ) ; }
[ref] public void METHOD 1 ( ) { android.content.Intent V AR1 = new android.content.Intent
( V AR2 ) ; METHOD 2 ( V AR1 , 0 ) ; }
[sys] public void METHOD 1 ( android.view.View view ){ android.content.Intent V AR1 =
new android.content.Intent ( V AR2 ) ; METHOD 2 ( V AR1 , 0 ) ; }
[ours] public void METHOD 1 ( ) { android.content.Intent V AR1 = new android.content.Intent
( V AR2 ) ; METHOD 2 ( V AR1 , 0 ) ; }
Case2: Transformer outputs codes with severe logic or irrelevant codes
[src] public java.util.Date METHOD 1 ( ) { return V AR1 . METHOD 1 ( ) . METHOD 2 ( ) ; }
[ref] public java.util.Date METHOD 1 ( ) { if ( ( V AR1 . METHOD 1 ( ) ) != null )
{ return V AR1 . METHOD 1 ( ) . METHOD 2 ( ) ; } else { return null ; } }
[sys] public java.util.Date METHOD 1 ( ) { if ( ( V AR1 ) == null )
{ return new java.util.Date ( ) ; } return V AR1 . METHOD 1 ( ) . METHOD 2 ( ) ; }
[ours] public java.util.Date METHOD 1 ( ) { if ( ( V AR1 . METHOD 1 ( ) ) != null )
{ return V AR1 . METHOD 1 ( ) . METHOD 2 ( ) ; } else { return null ; } }
Case3: Transformer makes no change
[src] public java.lang.String METHOD 1 ( TYPE 1 V AR1 ) { if ( V AR1 == null ) return null ;
return V AR1 . METHOD 2 ( ) . getText ( ) ; }
[ref] public java.lang.String METHOD 1 ( TYPE 1 V AR1 ) { return V AR1 . METHOD 2 ( ) .
getText ( ) ; }
[sys] public java.lang.String METHOD 1 ( TYPE 1 V AR1 ) { if ( V AR1 == null ) return null ;
return V AR1 . METHOD 2 ( ) . getText ( ) ; }
[ours] public java.lang.String METHOD 1 ( TYPE 1 V AR1 ) { return V AR1 . METHOD 2 ( ) .
getText ( ) ; }
Table 10: Three examples on code reﬁnement task. [src] represents the source input, [ref] represents
the reference, [sys] represents Transformer without data ﬂow and [ours] represents GraphCodeBERT.
17
Published as a conference paper at ICLR 2021
G E RROR ANALYSIS
We also conduct error analysis and summary two main classes of errors for both code understanding
and generation tasks.
Figure 11 gives three error cases of GraphCodeBERT on the natural language code search task. We
observe that GraphCodeBERR mainly fails to retrieve those source code that involves functions of
the library like “tf” (Tensorﬂow) in the ﬁrst case and “ GoogleCloudStorageHook” in the second
case. It’s difﬁcult for GraphCodeBERR to understand meanings of APIs like “tf.io.read ﬁle” and
“tf.image.decode image” without relevant information. A potential direction to mitigate the problem
is to incorporate deﬁnitions of the library. The other major problem is that there are some termi-
nologies like “unistr” in the query (corresponding to “decode(‘utf-8’)” in Python code) in third case.
Incorporating more text-code pairs for pre-training might alleviate this problem.
As for the code generation task, Table 11 shows two cases of GraphCodeBERT on the code translation
task. We ﬁnd that the major problems include semantic errors like identiﬁers from nowhere in the
ﬁrst case and syntax errors like missing a “ }” symbol before “return n” in the second case. This
problem might be mitigated by incorporating a dedicated decoder that takes into account grammar of
programming languages and different generation paradigm like generating a sequence of production
rules (Yin & Neubig, 2017; Guo et al., 2018; 2019) in a context-free grammar manner.
Case 1
Query: Return an image tensor.
Gold Source Code:
def read_image(filepath):
im_bytes = tf.io.read_file(filepath)
im = tf.image.decode_image(im_bytes, channels=CHANNELS)
im = tf.image.convert_image_dtype(im, tf.float32)
return im
GraphCodeBERT:
def get_image(self, float_key="floats", to_chw=True):
tensors = callBigDlFunc(self.bigdl_type,
"localImageFrameToImageTensor", self.value, float_key, to_chw)
return map(lambda tensor: tensor.to_ndarray(), tensors)
Case 2
Query: Uploads the file to Google cloud storage
Gold Source Code:
def execute(self, context):
hook = GoogleCloudStorageHook(
google_cloud_storage_conn_id=self.google_cloud_storage_conn_id,
delegate_to=self.delegate_to)
hook.upload(
bucket_name=self.bucket,
object_name=self.dst,
mime_type=self.mime_type,
filename=self.src,
gzip=self.gzip,
)
GraphCodeBERT:
def upload(remote_path, local_path):
storage = STORAGES['s3']()
conf = s3conf.S3Conf(storage=storage)
conf.upload(local_path, remote_path)
Case 3
Query: json.loads wants an unistr in Python3. Convert it.
Gold Source Code:
def _json_safe(data):
if not hasattr(data, 'encode'):
try:
data = data.decode('utf-8')
except UnicodeDecodeError:
raise ValueError(
'Expected valid UTF8 for JSON data, got %r' % (data,))
return data
GraphCodeBERT:
def parse_unstruct(unstruct):
my_json = json.loads(unstruct)
data = my_json['data']
schema = data['schema’]
if 'data' in data:
inner_data = data['data']
else:
raise SnowplowEventTransformationException(["Could not 
extract inner data field from unstructured event"])
fixed_schema = fix_schema("unstruct_event", schema)
return [(fixed_schema, inner_data)]
Figure 11: Error cases of GraphCodeBERT on the natural language code search.
Case1: semantic error – identiﬁers from nowhere.
[src] public String toString() {return getKey() + “: ” + getValue();}
[ref] public override string ToString(){return GetKey() + “: ” + GetValue();}
[ours] public override string ToString(){return Name + “: ” + GetValue();}
Case2: syntax errors – missing a “}” before “return n”)
[src] public static int numNonnull(Object[] data) {int n = 0;if ( data == null ) return n;
for (Object o : data) {if ( o!=null ) n++;}return n;}
[ref] public static int NumNonnull(object[] data){int n = 0;if (data == null){return n;}
foreach (object o in data){if (o != null){n++;}}return n;}
[ours] public static int NumNonNull(object[] data){int n = 0;if (data == null){return n;}
foreach (object o in data){if (o != null){n++;}return n;}
Table 11: Error cases of GraphCodeBERT on the code translation task. [src] represents the source
input, [ref] represents the reference and [ours] represents GraphCodeBERT.
18

Learning Compiler Pass Orders using Coreset and Normalized Value Prediction
Youwei Liang∗1 Kevin Stone∗2 Ali Shameli2 Chris Cummins2 Mostafa Elhoushi2 Jiadong Guo2
Benoit Steiner2 Xiaomeng Yang2 Pengtao Xie1 Hugh Leather2 Yuandong Tian2
Abstract
Finding the optimal pass sequence of compilation
can lead to a signiﬁcant reduction in program size
and/or improvement in program efﬁciency. Prior
works on compilation pass ordering have two ma-
jor drawbacks. They either require an excessive
budget (in terms of compilation steps) at compile
time or fail to generalize to unseen programs. In
this paper, for code-size reduction tasks, we pro-
pose a novel pipeline to ﬁnd program-dependent
pass sequences within 45 compilation calls. It
ﬁrst identiﬁes a coreset of 50 pass sequences
via greedy optimization of a submodular func-
tion, and then learns a policy with Graph Neural
Network (GNN) to pick the optimal sequence by
predicting the normalized values of the pass se-
quences in the coreset. Despite its simplicity, our
pipeline outperforms the default -Oz ﬂag by an
average of 4.7% over a large collection (4683) of
unseen code repositories from diverse domains
across 14 datasets. In comparison, previous ap-
proaches like reinforcement learning on the raw
pass sequence space may take days to train due
to sparse reward, and may not generalize well in
held-out ones from different domains. Our re-
sults demonstrate that existing human-designed
compiler ﬂags can be improved with a simple yet
effective technique that transforms the raw action
space into a small one with denser rewards.
1. Introduction
For more efﬁcient execution with fewer resources (e.g.,
memory, CPU, and storage), applying the right ordering for
compiler optimization passes to a given program, i.e., pass
ordering, is an important yet challenging problem. Manual
efforts require expert knowledge and are time-consuming,
error-prone, and often yield sub-par results, due to the huge
*Equal second author contribution, listed alphabetically.
1University of California, San Diego 2Meta AI. Correspondence
to: Kevin Stone <kevinleestone@meta.com>.
Coreset Optimization
Programs
Reward 
Matrix
Action sequences
Optimize
Coreset
...
...
...
Normalized Value Prediction
Inference 
Program
Value Softmax
...
Coreset
GNN-GEAN 
Program 
Observation
Figure 1.A depiction of our main contributions. (Top) Coreset
Optimization: A process for discovering a small set of pass se-
quences (coreset) that generalizes. (Bottom) Normalized Value
Prediction A process where our model learns to predict the nor-
malized value of pass sequences from the coreset. (Bottom inset)
Our model, Graph Edge Attention Network (GEAN), for encoding
program observations.
size of the search space. For example, the LLVM compiler
has 124 different compilation ﬂags. If the pass sequences
have a length of 45, then the possible number of sequences
(12445 ∼1094) is already more than the atoms in the uni-
verse (∼1080 (Planck Collaboration et al., 2016)).
In recent years, machine learning (ML)-guided pass order-
ing has emerged as an interesting ﬁeld to replace this labori-
ous process (Wang & O’Boyle, 2018). Along this line, many
works show promising results using various optimization
and/or ML techniques (e.g., reinforcement learning (Haj-Ali
et al., 2020a), language modelling (Cummins et al., 2017),
evolutionary algorithms (Kulkarni & Cavazos, 2012), etc).
However, there are several limitations. Some previous works
(e.g., MLGO (Troﬁn et al., 2021), MLGoPerf (Ashouri et al.,
2022)) run adaptive search algorithms to optimize a set of
programs for many hours. While this achieves strong per-
formance gain, the procedure can be slow and does not
distill knowledge from past experience and requires search-
ing from scratch for unseen programs. Recent works like
Autophase (Haj-Ali et al., 2020b) learn a pass policy via re-
arXiv:2301.05104v2  [cs.PL]  27 Jan 2023
Learning Compiler Pass Orders using Coreset and Normalized Value Prediction
inforcement learning, and applies it to unseen programs
without further search procedure, and GO (Zhou et al.,
2020) ﬁne-tunes the models for unseen programs. These ap-
proaches work for unseen programs from the same/similar
domain, but can still be quite slow in the training stage, and
do not show good generalization to programs from very
different domains.
In this work, we propose a novel pass ordering optimization
pipeline to reduce the code size of a program. As a ﬁrst
contribution, we formulate the search for a universal core
set of pass sequences (termed coreset) as an optimization
problem of a submodular reward function and use a greedy
algorithm to approximately solve it. The resulting coreset
consists of 50 pass sequences with a total number of passes
of 625. Importantly, it leads to very strong performance
across programs from diverse domains, ranging from the
Linux Kernel to BLAS. Speciﬁcally, for one unseen pro-
gram in the evaluation set, there exists one of the 50 pass
sequences that leads to an average code size reduction of
5.8% compared to the default -Oz setting, across 10 diverse
codebases (e.g. Cbench (Fursin, 2014), MiBench (Guthaus
et al., 2001), NPB (Bailey et al., 1995), CHStone (Hara
et al., 2008), and Anghabench (Da Silva et al., 2021)) of
over one million programs in total. Considering the huge
search space of compiler ﬂags, this is a very surprising
ﬁnding.
While it is time-consuming to ﬁnd an optimal pass sequence
with an exhaustive enumeration of the core subset of pass
sequences, as a second contribution, we ﬁnd that the (near)
optimal pass sequence can be directly predicted with high
accuracy via our Graph Edge Attention Network (GEAN), a
graph neural network (GNN) architecture adapted to encode
the augmented ProGraML (Cummins et al., 2021) graphs
of programs. Therefore, we can run a few pass sequences
selected by the model on an unseen program to obtain a
good code size reduction. This enables us to ﬁnd a good
ﬂag conﬁguration that leads to 4.7% improvement on aver-
age, with just 45 compilation passes, a reasonable trade-off
between the cost spent on trying compilation passes and the
resulting performance gain.
We compare our approach with extensive baselines, includ-
ing reinforcement learning (RL) -based methods such as
PPO, Q-learning, and behavior cloning. We ﬁnd that RL-
based approaches operating on the original compiler pass
space often suffer from unstable training (due to inaccurate
value estimation) and sparse reward space and fail to gener-
alize to unseen programs at inference. In comparison, our
approach transforms the vast action space into a smaller one
with much more densely distributed rewards. In this trans-
formed space, approaches as simple as behavior cloning can
be effective and generalizable to unseen programs.
2. Related Work
Graph structured data are present in numerous applications
and it has been shown that taking advantage of this data
can help us train very effective machine learning models.
(Brauckmann et al., 2020) use abstract syntax trees and con-
trol ﬂow graphs for learning compiler optimization goals.
They show that using such graphs allows them to outper-
form state-of-the-art in the task of heterogeneous OpenCL
mapping. (Guo et al., 2020) uses a transformer based model
with a graph guided masked attention that incorporates the
data ﬂow graph into the training. They achieve state of the
art performance in four tasks including code search, clone
detection, code translation, and code reﬁnement.
As a contender to graph neural networks, (Mialon et al.,
2021) uses transformers to process graphs. They show that
if we effectively encode positional and local sub-structures
of graphs and feed them to the transformer, then the trans-
former can outperform the classical GNN models. They
test their model on classiﬁcation and regression tasks and
achieve state of the art performance. In (Srinivas et al.,
2020), they used an unsupervised model to learn embed-
dings of high dimensional pixel data using contrastive learn-
ing. They then use this embedding for downstream rein-
forcement learning tasks.
3. Methodology
3.1. Action space
The CompilerGym framework (Cummins et al., 2022) pro-
vides a convenient interface for the compiler pass ordering
problem. The default environment allows choosing one of
124 discrete actions at each step corresponding to running
a speciﬁc compiler pass. In this work we will use the term
pass interchangeably with action. We ﬁx episode lengths to
45 steps to match the setup in (Haj-Ali et al., 2020b). Given
that our trajectories have a length of 45 steps, this means
we have 12445 ∼1.6 ×1094 possible pass sequences to
explore. To ﬁnd an optimal pass sequence for a program,
we can apply some existing reinforcement learning meth-
ods including Q learning like DQN (Mnih et al., 2015) and
policy gradient like PPO (Schulman et al., 2017).
Pass SequencesHowever for this problem it turns out that
certain pass sequences are good at optimizing many differ-
ent programs (where “good” is deﬁned as better than the
compiler default -Oz). We found that constraining the ac-
tion space to a learned set of pass sequences enables state
of the art performance and also signiﬁcantly reduces the
challenge of exploration. This allows us to cast the problem
as one of supervised learning over this set of pass sequences.
We use the following algorithm to ﬁnd a good set of pass
sequences.
Learning Compiler Pass Orders using Coreset and Normalized Value Prediction
Figure 2.An exemplar reward matrix for 67 programs and 50 pass
sequences. Most of the pass sequences do not lead to strong re-
wards, except for a few. On the other hand, certain pass sequences
(i.e., columns) can lead to high rewards for multiple programs
simultaneously and thus are good candidates for the coreset.
Suppose we have N programs and M promising pass se-
quences. Let R= [rij] ∈RN×M be the reward matrix, in
which rij >0 is the ratio of the codesize of i-th program if
applied with j-th pass sequence, compared to -O0. rij >1
means that the j-th pass sequence does better than -O0 in
codesize reduction for i-th program, and rij < 1 means
it performs worse. The reward matrix is normalized per
row, by the maximum reward for each program, so that the
optimal pass sequence has reward of 1 for each program.
Then we aim to pick a subset Sof Kpass sequences, called
the coreset, from all M pass sequences, so that the overall
saving J(S) is maximized:
max
|S|≤K
J(S) =
N∑
i=1
max
j∈S
rij (1)
Finding M candidate pass sequences. Note that there can
be exponential number of pass sequences, and we cannot
construct the entire reward matrix, instead we seed a list
of candidate action trajectories. For this, we run a random
policy on a subset of M (17500) selected training programs.
For each selected program, we run E (200) episodes and
pick the best pass sequence as one of the M candidates.
If part of the best pass sequence leads to the same state,
they are truncated so that the sequence becomes shorter.
If multiple pass sequences yield the same reward we only
retain the ﬁrst after ordering them length-lexicographically.
On average these last two steps reduce the length of the
candidate pass sequences by 80%.
We then construct the rewardrij by applying the j-th best
pass sequence to program i, and comparing it with -O0.
Finding the best coresetS with greedy algorithm. As
a function deﬁned in subsets, J(S) can be proven to be a
nonnegative and monotone submodular function (See Ap-
pendix). While maximizing a submodular function is NP-
hard (Ward & ˇZivn`y, 2016), the following greedy algorithm
is proven to be an efﬁcient approximate algorithm that leads
to fairly good solutions (Nemhauser et al., 1978). Start-
ing from S0 = ∅, at each iteration t, it picks a new pass
sequence jt as follows:
jt := arg max
j/∈St−1
J(St−1 ∪{j}) (2)
And St ←St−1 ∪{jt}until we pick Kpass sequences. We
set K = 50 in this paper.
Given the discovered coresetS, we deﬁne a generalized ac-
tion as a pass sequence in S. Applying a generalized action
to a program means that we roll out the corresponding pass
sequence on the program and return the best program state
(i.e., having the highest cumulative reward) (it is feasible
because we can cache the program state for each step).
3.2. Normalized Value Prediction
After discovering the “good” pass sequences (i.e., the core-
set), we can turn the problem of the sequential decision-
making on compiler passes into a problem of supervised
learning. The target is to train a model to predict the best
pass sequence conditioned on the program, where the label
of the program is the index of the pass sequence that results
in the largest code size reduction. However, one important
observation we have is that there are typically multiple pass
sequences in the coreset that all result in the largest code
size reduction (see Figure 3 for the examples). Therefore,
instead of using the single-class classiﬁcation method with
a cross entropy loss, we leverage the fact we have access to
the values for all pass sequences. We predict the softmax
normalized value of each pass sequence with a cross entropy
loss detailed below. This approach is similar to behavior
cloning (Pomerleau, 1988) but with soft targets over the
coreset.
For a program p, we roll out all pass sequences in the
coreset on it, obtaining a reward rp
j for the j-th sequence
(i.e., the highest cumulative reward observed during the
rollout of the pass sequence), which forms a value vector
rp = [rp
1,...,r p
K]. Then, the normalized values of the pass
sequences are deﬁned by
vp = Softmax(rp/T) (3)
where T is a temperature parameter.
For an initial observation op of a program, our model out-
puts a probability distribution, a = f(op), over the pass
sequences. The target of the training is to make a close to
the normalized values of the pass sequences. To this end, we
use the Kullback–Leibler (KL) divergence to supervise the
Learning Compiler Pass Orders using Coreset and Normalized Value Prediction
model, which can be reduced to the following cross entropy
loss up to a constant term.
L(ap,vp) = −
K∑
j=1
ap
j log vp
j (4)
3.3. Program Representations
Since we use the CompilerGym (Cummins et al., 2022)
environments for program optimization, we exploit the pro-
gram representations from CompilerGym, where program
source code is converted to LLVM Intermediate Represen-
tation (IR) (Lattner & Adve, 2004) and several representa-
tions are constructed from the IR, including the ProGraML
graph (Cummins et al., 2021), the Autophase feature (Haj-
Ali et al., 2020b), and the Inst2vec feature (Ben-Nun et al.,
2018). We choose to use the LLVM environment from
CompilerGym because the LLVM ecosystem is a popular
compiler infrastructure that powers Swift, Rust, Clang, and
more.
Autophase We use the Autophase features Haj-Ali et al.
(2020b) to build some baseline models for comparison,
which will be detailed in Section 4.2. The Autophase feature
is a 56-dimension integer feature vector summarizing the
LLVM IR representation, and it contains integer counts of
various program properties such as the number of instruc-
tions and maximum loop depth. we use an MLP to encode
it and output a program representation.
ProGraML As a part of our main method, we leverage
ProGraML (Cummins et al., 2021) graphs for training GNN
models. ProGraML is a graph-based representation that en-
codes semantic information of the program which includes
control ﬂow, data ﬂow, and function call ﬂow. This repre-
sentation has the advantage that it is not a ﬁxed size - it
does oversimplify large programs - and yet it is still a more
compact format than the original IR format. We list three
bullet points of the ProGraML graph below.
• Node featuresEach node in a ProGraML graph have
4 features described in Table 1. The “text” feature
is a textual representation and the main feature that
captures the semantics of a node. For example, it tells
us what an “instruction” node does (e.g., it can be
alloca, store, add, etc).
• Edges featuresEach edge in a ProGraML graph have
2 features described in Table 1.
• Type graph There is an issue with the ProGraML
graph. Speciﬁcally, a node of type variable/constant
node can end up with a long textual representation
(for the “text” feature) if it is a composite data struc-
ture. For example, a struct (as in C/C++) contain-
ing dozens of data members needs to include all the
members in its “text” feature. In other words, the
current ProGraML representation does not automati-
cally break down composite data types into their ba-
sic components. Since there is an unbounded num-
ber of possible structs, this prevents 100% vocabulary
coverage on any IR with structs (or other composite
types). To address this issue, we propose to expand
the node representing a composite data type into a
type graph. Speciﬁcally, a pointer node is expanded
into this type graph: [variable] <- [pointer] <-
[pointed-type], where [...]denotes a node and
<-denotes an edge connection. A struct node is ex-
panded into a type graph where all its members are
represented by individual nodes (which may be fur-
ther expanded into their components) and connected
to a structnode. An array is expanded into this
type graph:[variable] <- [array] <- [element-
type]. The newly added nodes are categorized as
type nodes and the edges connecting the type nodes
are type edges. The type nodes and type edges con-
stitute the type sub-graphs in the ProGraML graphs. In
this manner, we break down the composite data struc-
tures into the type graphs that consist of only primitive
data types such as float and i32.
3.4. Network Architecture
Since the Autophase feature can be encoded by a simple
MLP, we discuss only the network architectures for encod-
ing the ProGraML graphs in this section.
We use a graph neural network (GNN) as the backbone
to encode the ProGraML graphs and output a graph-level
representation. The GNN encodes the graph via multiple
layers of message passing and outputs a graph-level repre-
sentation by a global average pooling over the node features.
The goal of graph encoding is to use the structure and re-
lational dependencies of the graph to learn an embedding
that allows us to learn a better policy. To this end, we ex-
perimented with several different GNN architectures such
as Graph Convolutional Network (GCN) (Kipf & Welling,
2017), Gated Graph Convolutions Network (GGC) (Li et al.,
2015), Graph Attention Network (GAT) (Brody et al., 2022),
Graph Isomorphism Network (GIN) (Xu et al., 2019). To
better capture the rich semantics of node/edge features in
the ProGraML graphs, we propose Graph Edge Attention
Network ( GEAN), a variant of the graph attention net-
work (Veliˇckovi´c et al., 2017). These GNNs leverage both
the node and edge features, so we start by presenting how
to embed the node and edge features.
Node embeddingFor the “text” features of the nodes, we
build a vocabulary that maps from text to integer. The
vocabulary covers all the text ﬁelds of the nodes in the
graphs in the training set. The ﬁnal vocabulary consists of
Learning Compiler Pass Orders using Coreset and Normalized Value Prediction
Feature Description
Node
type One of {instruction, variable, constant,type}
text Semantics of the node
function Integer position in function
block Integer position in IR basic block
Edge
ﬂow Edge type. One of {call, control, data,type}
position Integer edge position in ﬂow branching
Table 1.Features in the ProGraML graph representation which we
augment with type information (changes highlighted). We ablate
the augmentations in Section 4.5.
117 unique textual representations, and we add an additional
item “unknown” to the vocabulary which denotes any text
features that may be encountered at inference time and we
have not seen before. The i-th textual representation is
embedded using a learnable vector vi ∈Rd, where d is
the embedding dimension. The “type” feature is not used
because it can be inferred from the “text” feature.
Edge embeddingThe edge embedding is the sum of three
types of embedding as the following.
• Type embeddingWe have 4 types of edge ﬂows, so
we use 4 learnable vectors to represent them.
• Position embeddingThe “position” feature of an edge
is a non-negative integer which does not have an upper
bound. We truncate any edge positions larger than 32
to 32 and use a set of 32 learnable vectors to represent
the edge positions.
• Block embedding We use the block indices of the
two nodes connected by the edge to construct a new
edge feature. The motivation is that whether the edge
goes beyond an IR basic block can inﬂuence program
optimization. Suppose the block indices of the source
node and the target node of an edge are respectively bi
and bj. We get the relative position of the two nodes
with respect to IR basic blocks in the following way:
pblock = sign(bi−bj). If the edge connects two nodes
in the same IR basic block, then pblock is 0. And
pblock = ±1 indicates the edge goes from a block
to the next/previous block. There are 3 possible values
for pblock, so it is embedded using 3 learnable vectors.
The ﬁnal embedding of an edge is the sum of its type, posi-
tion, and block embedding vectors.
Graph mixupWe note that the ProGraML graphs of two
programs can be composed into a single graph without af-
fecting the semantics of the two programs. And their value
vectors can be added up to correctly represent the value vec-
tor of the composite graph. In this manner, we can enrich
the input space to the GNNs and mitigate model overﬁtting
for the normalized value prediction method.
Graph Edge Attention NetworkWe introduce the GEAN
in this paragraph and defer its mathematical details to the
Appendix. There are two main differences between the GAT
and GEAN. 1) GEAN adopts a dynamic edge representa-
tion. Speciﬁcally, GAT uses the node-edge-node feature to
calculate the attention for neighborhood aggregation, while
GEAN uses the node-edge-node feature to calculate not
only the attention but also a new edge representation. Then,
the updated edge representation is sent to the next layer for
computation. Note that GAT uses the same edge embedding
in each layer. We conduct an ablation study showing that
the edge representation in GEAN improves the generaliza-
tion of the model. 2) GAT treats the graph as an undirected
graph while GEAN encodes the node-edge-node feature to
output an updated node-edge-node feature, where the two
updated node features represent the feature to be aggregated
in the source node and the target node, respectively. This
ensures that the directional information is preserved in the
neighborhood aggregation.
3.5. Dataset Preparation
Overﬁtting issues could happen if training is performed
on a small subset of programs, or the set of programs is
not diverse enough. To mitigate this we ﬁnd it helpful to
create an aggregate dataset that uses many different public
datasets as curated by CompilerGym. CompilerGym gives
us access to 14 different datasets constructed using two
different methods.
• Curated These are small collections of hand-picked
programs. They are curated to be distinct from one
another without overlap and are not useful for training.
Typically programs are larger as they may comprise
multiple source ﬁles combined into a single program.
These are commonly used for evaluating compiler op-
timization improvements.
• Uncurated These are comprised of individual com-
piler IRs from building open source repositories such
as Linux and Tensorﬂow. We also include synthetically
generated programs, targeted for compiler testing (not
optimization).
For our aggregate dataset we decided to holdout the entirety
of the four curated datasets for use as an out-of-domain test
set. This is important because they represent the types of
programs we expect to see in the wild. We also split the
uncurated datasets into train, validaton, and test programs.
3.6. Evaluation
For all our metrics and rewards we leverage the IR instruc-
tion count as value we are trying to minimize. We also
report metrics on each CompilerGym dataset as well as
Learning Compiler Pass Orders using Coreset and Normalized Value Prediction
Type Dataset Train Val Test
Uncurated
anghabench-v1 70,7000 1,000 2,000
blas-v0 133 28 29
github-v0 7,000 1,000 1,000
linux-v0 4,906 1,000 1,000
opencv-v0 149 32 32
poj104-v1 7,000 1,000 1,000
tensorﬂow-v0 415 89 90
clgen-v0 697 149 150
csmith-v0 222 48 48
llvm-stress-v0 697 149 150
Curated
cbench-v1 0 0 11
chstone-v0 0 0 12
mibench-v1 0 0 40
npb-v0 0 0 121
Total - 728,219 4,495 4,683
Table 2.CompilerGym dataset types and training splits. The hand
curated datasets are used solely to evaluate generalization to real
world program domains at test time.
the mean over datasets to get a single number to compare
overall results.
• The mean percent improved over-Oz (MeanOverOz)
deﬁned as following:
¯IOz = MeanOverOz := 1
|P|
∑
p
IOz
p −Iπθ
p
IOzp
, (5)
where pis a speciﬁc program from the set of programs
Pin the dataset. IOz
p is the number of IR instructions
in the program after running the default compiler pass
-Oz. Iπθ
p is the number of IR instructions in the pro-
gram after applying the policy under consideration.
We can think of this as a simple average of the percent
improvement over -Oz.
• We also look compare the geometric mean
(GMeanOverOz) of ﬁnal sizes across all pro-
grams relative to -Oz to give a weighted comparison
that is insensitive to outliers.
¯IOz
G = GMeanOverOz :=
(∏
p
IOz
p
Iπθ
p
) 1
|P|
(6)
4. Experiments
4.1. Experimental Setup
For the methods in Table 3, we search over a set of hyper-
parameters, including the temperature T in Eq. 3, number
of layers, the embedding dimension of the node/edge fea-
tures, and the output dimension in the hidden layers in the
MLPs. We select the best model of each method based on
the validation metric (validation MeanOverOz in 45 steps)
and report the best model’s metrics on the test set.
4.2. Baseline Methods
Oracle We consider a brute-force search over the coreset
in order to ﬁnd the best pass sequence for a given program.
This gives us an upper-bound of the downstream policy
network. In our case the coreset has a total of 50 sequences
(625 total passes).
Top-45We also consider how well we would do if the oracle
is only allowed to use the most popular pass sequences
but limited to 45 passes. We use 45 passes because this
is maximum allowed for our all other baselines and our
proposed method.
RL-PPO We reproduce the Autophase (Haj-Ali et al.,
2020b) pipeline by using the state-of-the-art RL algorithm
PPO (Schulman et al., 2017) to learn a policy model. We
have two program representations for training the RL mod-
els, including the Autophase feature and the ProGraML
graphs (note that Haj-Ali et al. (2020b) only used the
Autophase feature). The Autophase/ProGraML feature
is sent to a GNN/MLP for feature encoding, which out-
puts a program-level embedding. Following Haj-Ali et al.
(2020b), we add an additional action history feature to the
RL pipeline, which is a histogram of previously applied
passes. The vector of the histogram of action history is
divided by 45 (i.e., the number of the total passes in our
budget) for normalization. A 2-layer MLP is used to encode
the action history to obtain a feature vector, which is con-
catenated with the program embedding extracted from the
ProGraML graph or the Autophase feature. The concate-
nated feature is sent to a 2-layer MLP to output the action
probability for the policy network. The value network (i.e.,
the critic) in our PPO pipeline mimics the policy network
(i.e., the actor) in feature encoding and outputs a scalar to es-
timate the state values. The state values are the expectation
of the discounted cumulative rewards where the reward in
each step is the improvement over -O0: (I(t)
p −Iπθ
p )/I(t)
p ,
where I(t)
p denotes the current IR instruction count of the
program pat time step t. This reward is reasonable since
it makes the value approximation Markovian. At inference,
an action is sampled from the output of the learned policy
network at each time step until the total number of steps
reaches 45.
Q-value-rank We consider each pass sequence in the core-
set as a generalized action and train a Q network to predict
the value of each generalized action. Recall that the value
vector rp is the highest cumulative reward observed during
the rollout of each pass sequence in the coreset on program
p. The Q-value-rank model is trained to approximate the
value vector using a mean squared loss.
Learning Compiler Pass Orders using Coreset and Normalized Value Prediction
Figure 3.GEAN-Q-value-rank: ground truth of rewards and model predictions over the 50 generalized actions for three benchmarks.
BC We consider learning a standard behavior cloning model
to predict the best pass sequences from the coreset, where
the best pass sequence is deﬁned as the following. As in the
previous bullet point, the value vector is denoted byrp. If
there is only one isuch that rp
i = maxj∈nrp
j, then the clas-
siﬁcation label is i. If there are multiple such i’s (multiple
pass sequences) that achieve the largest reward maxj∈nrp
j,
then we order the corresponding pass sequences by length-
lexicographic ordering. The classiﬁcation label is selected
to be the ﬁrst one after the ordering. This ensures that our
deﬁnition for the best pass sequence (among the coreset) for
a program is unique and consistent. We use a standard cross
entropy loss to train the single-class classiﬁcation model.
NVP This is the normalized value prediction method de-
scribed in Section 3.2.
The last three methods (i.e., Q-value-rank, BC, and NVP)
share the same inference protocol. Note that they all output
a vector a of length 50 whose entries correspond to the pass
sequences in the coreset. At inference, we roll out the pass
sequences with the highest values in a one by one until our
budget of 45 passes is reached. Since the pass sequence has
an average length of 12.5, typically 3 or 4 pass sequences are
applied (anything beyond 45 passes will be truncated). For
BC and NVP, we also tried sampling pass sequences using
the model output a, but that resulted in worse performance.
Therefore, we stick to selection by maximum values.
4.3. Main Results
In Table 3 we present the main results of our experiments
comparing our proposed method -NVP to various base-
lines. The test programs were completely held-out during
both data-driven learning phases (pass sequence search and
model training).
The results show that our model achieves strong perfor-
mance over the prior method (Autophase-RL-PPO) pro-
posed in (Haj-Ali et al., 2020b). Additionally, we can see
that both the GEAN model and the normalized value pre-
diction over the discovered coreset are needed to achieve
the best performance within 45 passes. See Figure 6 in the
Method #passes ¯IOz ¯IOz
G
Compiler (-Oz) - 0% 1.000
Autophase-RL-PPO 45 -16.3% 0.960
GCN-RL-PPO 45 -12.2% 0.987
GGC-RL-PPO 45 -8.5% 1.000
GIN-RL-PPO 45 -11.3% 0.991
GAT-RL-PPO 45 -10.3% 0.999
GEAN-RL-PPO 45 -10.0% 0.997
GEAN-Q-value-rank 45 -0.3% 1.043
Autophase-BC 45 2.6% 1.043
GEAN-BC 45 2.1% 1.038
Autophase-NVP (Ours) 45 3.8% 1.054
GCN-NVP (Ours) 45 4.4% 1.058
GGC-NVP (Ours) 45 4.1% 1.056
GIN-NVP (Ours) 45 4.2% 1.056
GAT-NVP (Ours) 45 4.1% 1.055
GEAN-NVP (Ours) 45 4.7% 1.062
Top-45 45 -7.5% 0.992
Oracle 625 5.8% 1.075
Table 3.Evaluation results on held-out test setaveraged over all
datasets. ¯IOz denotes per-program MeanOverOz, and ¯IOz
G de-
notes GMeanOverOz over all programs. All methods except
Compiler and Oracle baselines use 45 compiler optimization
passes.
Appendix for a visualization of the improvement in program
size over the 45 passes on programs from the holdout set.
The Oracle shows strong performance but requires a large
number of interactions with the compiler. But, this shows
that the pass sequence search generalizes to new unseen
programs. This is somewhat unsurprising given that the
compiler’s built-in hand-tuned pass list (-Oz) works reason-
ably well for most programs.
The performance of Top-45 by itself is weak showing that
in order to achieve good results in a reasonable number
of passes (45) we need to leverage a general policy and
Learning Compiler Pass Orders using Coreset and Normalized Value Prediction
search to select the most likely candidate pass sequences to
evaluate.
4.4. Why Did the RL-PPO Baseline Fail?
We provide an empirical analysis of why the RL-PPO ap-
proaches obtain much lower performance compared to our
NVP approaches. We hypothesize two possible reasons for
the failures of RL-PPO. 1) Inaccurate state-value estima-
tion results in a high variance in training.In the PPO
algorithm, we have a policy network (the actor) to output
a probability distribution over the actions. And we have
a value network (the critic) for estimating the state val-
ues, where the approximation is based on regressing the
cumulative reward of trajectories sampled from the current
policy (Schulman et al., 2017). The update of the policy
network is based on the state value outputted by the value
network. Inaccurate state value estimation results in a high
variance in training the policy network. Due to the stochas-
tic nature of the value estimation that stems from the Monte
Carlo sampling in cumulative reward regression, it is difﬁ-
cult to analyze how accurately the value network approx-
imates the ground truth state values (which are unknown
even for the programs in the training set). We alleviate
this issue by analyzing the Q-value-rank approach (as
introduced in Section 4.2), which can be seen as a sim-
pliﬁed version of the value approximation in PPO. The
Q-value-rank approach is simpler because the values
to estimate are deterministic (i.e., the value vector rp is
ﬁxed for a program p). Moreover, since we consider the
50 pass sequences in our coreset as 50 generalized actions,
the Q-value-rank approach can be seen as the value
approximation in a PPO pipeline where a trajectory consists
of only a single step over the 50 generalized actions. In
this sense, the Q-value-rank approach is a simpliﬁed
version of the regular value estimation in PPO. Figure 3
shows that the value estimation is inaccurate for programs
in the held-out test set even for Q-value-rank approach.
Therefore, it is even more challenging to estimate the state
values in PPO. The inaccuracy leads to a high variance in
training the policy network. 2) The reward is very sparse.
As shown in Figure 2, the rewards are very sparse. There-
fore, the good states (i.e., the program states with a higher
chance to be optimized in code size reduction) are rarely
seen by the value/policy network during rollouts. Then, the
value network does not have a good value estimation for
those good states, and the policy network does not converge
to output a good policy for them. We conjecture these two
issues are the main reason for why the RL-PPO methods
obtain the worst performance as shown in Table 3.
4.5. Ablation Studies
Ablation for GEAN-NVPWe perform 3 ablation exper-
iments for GEAN-NVP, where we remove graph mixup,
Figure 4.The effect of temperature on GEAN-NVP.
Method Test MeanOverOz
GEAN-NVP 4.7% (0.0%)
- graph mixup 4.4% (-0.3%)
- edge embedding 4.4% (-0.3%)
- type graph -5.3% (-10.0 %)
Table 4.Ablation on GEAN-NVP components.
mask the edge embedding, and remove the type graph,
respectively. The results in Table 4 show that the test
MeanOverOz metric drops after removing any of the three
components. Speciﬁcally, the performance drops signiﬁ-
cantly after removing the type graph, which validates its
importance.
The effect of the temperatureThe temperature parameter
T in Eq. 3 controls how sharp the target distribution is. The
distribution tends to be sharper as the temperature decreases.
To analyze the inﬂuence of the temperature on the general-
ization of the model, We vary the temperature T in training
the GEAN-NVP model and report the results in Figure 4.
5. Conclusions
In this paper, we develop a pipeline for program size re-
duction under limited compilation passes. We ﬁnd that it is
a great challenge to approximate the state values (i.e., the
maximum code size reduction) for a diverse set of programs,
so existing state-of-the-art methods such as proximal policy
optimization (PPO) fail to obtain good performances. To
tackle this problem, we propose a search algorithm that dis-
covers a good set of pass sequences (i.e., the coreset), which
generalizes well to unseen programs. Moreover, we propose
to train a GNN to approximate the normalized state values
of programs over the coreset, for which we propose a variant
of the graph attention network, termed GEAN. Our pipeline
of coreset discovery and normalized value prediction via
GEAN perform signiﬁcantly better than the PPO baselines.
Learning Compiler Pass Orders using Coreset and Normalized Value Prediction
References
Ashouri, A. H., Elhoushi, M., Hua, Y ., Wang, X., Manzoor,
M. A., Chan, B., and Gao, Y . Mlgoperf: An ml guided
inliner to optimize performance, 2022. URL https:
//arxiv.org/abs/2207.08389.
Bailey, D., Harris, T., Saphir, W., Van Der Wijngaart, R.,
Woo, A., and Yarrow, M. The nas parallel benchmarks
2.0. Technical report, Technical Report NAS-95-020,
NASA Ames Research Center, 1995.
Ben-Nun, T., Jakobovits, A. S., and Hoeﬂer, T. Neural
code comprehension: A learnable representation of code
semantics. Advances in Neural Information Processing
Systems, 31, 2018.
Brauckmann, A., Goens, A., Ertel, S., and Castrillon, J.
Compiler-based graph representations for deep learning
models of code. In CC, pp. 201–211, 2020.
Brody, S., Alon, U., and Yahav, E. How attentive are
graph attention networks? In International Confer-
ence on Learning Representations, 2022. URL https:
//openreview.net/forum?id=F72ximsx7C1.
Cummins, C., Petoumenos, P., Wang, Z., and Leather, H.
End-to-end deep learning of optimization heuristics. In
26th International Conference on Parallel Architectures
and Compilation Techniques (PACT). IEEE, 2017.
Cummins, C., Fisches, Z. V ., Ben-Nun, T., Hoeﬂer, T.,
O’Boyle, M. F. P., and Leather, H. ProGraML: A Graph-
based Program Representation for Data Flow Analysis
and Compiler Optimizations. In ICML, 2021.
Cummins, C., Wasti, B., Guo, J., Cui, B., Ansel, J., Gomez,
S., Jain, S., Liu, J., Teytaud, O., Steiner, B., et al. Com-
pilergym: robust, performant compiler optimization en-
vironments for ai research. In 2022 IEEE/ACM Interna-
tional Symposium on Code Generation and Optimization
(CGO), pp. 92–105. IEEE, 2022.
Da Silva, A. F., Kind, B. C., de Souza Magalh ˜aes, J. W.,
Rocha, J. N., Guimaraes, B. C. F., and Pereira, F. M. Q.
Anghabench: A suite with one million compilable c
benchmarks for code-size reduction. In 2021 IEEE/ACM
International Symposium on Code Generation and Opti-
mization (CGO), pp. 378–390. IEEE, 2021.
Fursin, G. Collective tuning initiative. arXiv preprint
arXiv:1407.3487, 2014.
Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou,
L., Duan, N., Svyatkovskiy, A., Fu, S., et al. Graphcode-
bert: Pre-training code representations with data ﬂow.
arXiv preprint arXiv:2009.08366, 2020.
Guthaus, M. R., Ringenberg, J. S., Ernst, D., Austin, T. M.,
Mudge, T., and Brown, R. B. Mibench: A free, commer-
cially representative embedded benchmark suite. In Pro-
ceedings of the fourth annual IEEE international work-
shop on workload characterization. WWC-4 (Cat. No.
01EX538), pp. 3–14. IEEE, 2001.
Haj-Ali, A., Ahmed, N. K., Willke, T., Shao, Y . S., Asanovic,
K., and Stoica, I. Neurovectorizer: End-to-end vectoriza-
tion with deep reinforcement learning. In Proceedings of
the 18th ACM/IEEE International Symposium on Code
Generation and Optimization, pp. 242–255, 2020a.
Haj-Ali, A., Huang, Q. J., Xiang, J., Moses, W., Asanovic,
K., Wawrzynek, J., and Stoica, I. Autophase: Jug-
gling hls phase orderings in random forests with
deep reinforcement learning. In Dhillon, I., Pa-
pailiopoulos, D., and Sze, V . (eds.), Proceedings
of Machine Learning and Systems , volume 2, pp.
70–81, 2020b. URL https://proceedings.
mlsys.org/paper/2020/file/
4e732ced3463d06de0ca9a15b6153677-Paper.
pdf.
Hara, Y ., Tomiyama, H., Honda, S., Takada, H., and Ishii,
K. Chstone: A benchmark program suite for practical
c-based high-level synthesis. In 2008 IEEE International
Symposium on Circuits and Systems (ISCAS), pp. 1192–
1195. IEEE, 2008.
Kipf, T. N. and Welling, M. Semi-supervised classi-
ﬁcation with graph convolutional networks. In In-
ternational Conference on Learning Representations ,
2017. URL https://openreview.net/forum?
id=SJU4ayYgl.
Kulkarni, S. and Cavazos, J. Mitigating the compiler opti-
mization phase-ordering problem using machine learning.
In Proceedings of the ACM international conference on
Object oriented programming systems languages and ap-
plications, pp. 147–162, 2012.
Lattner, C. and Adve, V . Llvm: A compilation framework
for lifelong program analysis & transformation. In Inter-
national Symposium on Code Generation and Optimiza-
tion, 2004. CGO 2004., pp. 75–86. IEEE, 2004.
Li, Y ., Tarlow, D., Brockschmidt, M., and Zemel, R.
Gated graph sequence neural networks. arXiv preprint
arXiv:1511.05493, 2015.
Mialon, G., Chen, D., Selosse, M., and Mairal, J. Graphit:
Encoding graph structure in transformers, 2021.
Mnih, V ., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness,
J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidje-
land, A. K., Ostrovski, G., et al. Human-level control
Learning Compiler Pass Orders using Coreset and Normalized Value Prediction
through deep reinforcement learning. nature, 518(7540):
529–533, 2015.
Nemhauser, G. L., Wolsey, L. A., and Fisher, M. L. An
analysis of approximations for maximizing submodular
set functions—i. Mathematical programming, 14(1):265–
294, 1978.
Planck Collaboration et al. Planck 2015 results - xiii.
cosmological parameters. Astronomy & Astrophysics ,
594:A13, 2016. doi: 10.1051/0004-6361/201525830.
URL https://doi.org/10.1051/0004-6361/
201525830.
Pomerleau, D. A. Alvinn: An autonomous land vehicle in a
neural network. In Touretzky, D. (ed.), Advances in Neu-
ral Information Processing Systems, volume 1. Morgan-
Kaufmann, 1988. URL https://proceedings.
neurips.cc/paper/1988/file/
812b4ba287f5ee0bc9d43bbf5bbe87fb-Paper.
pdf.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and
Klimov, O. Proximal policy optimization algorithms.
arXiv preprint arXiv:1707.06347, 2017.
Srinivas, A., Laskin, M., and Abbeel, P. Curl: Contrastive
unsupervised representations for reinforcement learning,
2020.
Troﬁn, M., Qian, Y ., Brevdo, E., Lin, Z., Choromanski,
K., and Li, D. Mlgo: a machine learning guided com-
piler optimizations framework, 2021. URL https:
//arxiv.org/abs/2101.04808.
Veliˇckovi´c, P., Cucurull, G., Casanova, A., Romero, A.,
Lio, P., and Bengio, Y . Graph attention networks.arXiv
preprint arXiv:1710.10903, 2017.
Wang, Z. and O’Boyle, M. Machine learning in compiler
optimization. Proceedings of the IEEE, 106(11):1879–
1901, 2018.
Ward, J. and ˇZivn`y, S. Maximizing k-submodular functions
and beyond. ACM Transactions on Algorithms (TALG),
12(4):1–26, 2016.
Xu, K., Hu, W., Leskovec, J., and Jegelka, S. How powerful
are graph neural networks? In International Conference
on Learning Representations, 2019. URL https://
openreview.net/forum?id=ryGs6iA5Km.
Zhou, Y ., Roy, S., Abdolrashidi, A., Wong, D., Ma, P.,
Xu, Q., Liu, H., Phothilimtha, P., Wang, S., Goldie, A.,
et al. Transferable graph optimizers for ml compilers.
Advances in Neural Information Processing Systems, 33:
13844–13855, 2020.
Learning Compiler Pass Orders using Coreset and Normalized Value Prediction
A. Proofs
Lemma A.1. The objective J(S) deﬁned in Eqn. 1
max
|S|≤K
J(S) =
N∑
i=1
max
j∈S
rij (7)
(with the additional deﬁnition J(∅) = 0), is a nonnegative and monotone submodular function.
Proof. Since rij >0, it is clear that J(S) ≥0 is nonnegative.
To incorporate the special case J(∅) = 0, note that J(S) can be written as
max
|S|≤K
J(S) =
N∑
i=1
max
(
max
j∈S
rij,0
)
. (8)
Let ˆrij = rij and ˆri,0 = 0, then in order to prove J(S) is monotone and submodular, by additivity, we only need to prove
Ji(S) := maxj∈S∪{0}ˆrij is monotone and submodular.
For any A⊆B, it is clear that
Ji(A) = max
j∈A∪{0}
ˆrij ≤ max
j∈B∪{0}
ˆrij = Ji(B) (9)
So Ji(S) is monotone.
To prove submodularity, for any A ⊆B, we comare the quatity of Ji(A∪{j}) −Ji(A) and Ji(B∪{j}) −Ji(B) for
j /∈B.
Case 1: ˆrij is a maximum over the subsetB. In this case, then ˆrij is also a maximum over the subset A. Then
Ji(A∪{j}) = Ji(B∪{j}) = ˆrij, since Ji(A) ≤Ji(B), we have:
Ji(A∪{j}) −Ji(A) ≥Ji(B∪{j}) −Ji(B) (10)
Case 2: ˆrij is a maximum overAbut not inB. Then Ji(A∪{j}) −Ji(A) ≥0, but Ji(B∪{j}) −Ji(B) = 0 . So
Eqn. 10 still holds.
Case 3:ˆrij is neither a maximum inAor inB. Then both Ji(A∪{j}) −Ji(A) = 0 and Ji(B∪{j}) −Ji(B) = 0. So
Eqn. 10 still holds.
By deﬁnition of submodularity (Eqn. 10), we know Ji(S) is submodular and so does J(S).
B. GEAN Encoding
Our Graph Edge Attention Network (GEAN) has the following key features.
Attention with edge featuresWe modify the attention mechanism in GAT to output an edge embedding and two node
features for neighborhood aggregation. For clarity, we show a table containing the notations used in the GNN in Table 5. The
feature update process can be mathematically deﬁned by the following equations, where Mi,i = 1,..., 5 is an encoding
fully connected layer.
X′(t+1)
ij = M1(X(t)
i ,E(t)
i→j,X(t)
j ), (11)
a′(t+1)
ij = M2(X(t)
i ,E(t)
i→j,X(t)
j ), (12)
X(t+1)
ji = M3(X(t)
i ,E(t)
i→j,X(t)
j ), (13)
a(t+1)
ji = M4(X(t)
i ,E(t)
i→j,X(t)
j ), (14)
E(t+1)
i→j = M5(X(t)
i ,E(t)
i→j,X(t)
j ), (15)
Learning Compiler Pass Orders using Coreset and Normalized Value Prediction
1 2
5
3 𝑋!"#𝑋!$#
𝑋!%#
𝑋"&#𝑋"!
𝑋%!
𝑋$!
𝑋&"
4
Figure 5.Graph attention. Circles denote nodes and solid arrows denote edges. Squares are the calculated features, and dash arrows
represent feature aggregation. The orange/green squares denote the features to be aggregated in the target/source nodes of the edges. The
edge embedding and the attention are not shown.
Notation Meaning
E The set of edges in the graph
(i,j) Edge from node ipointing to node j
X(t)
i Representation of node iat layer t
E(t)
i→j Representation of the edge (i,j) at layer t
X(t)
ij Representation for node iassociated with edge (i,j)
X′(t)
ij Representation for node iassociated with edge (j,i)
a(t)
ij Raw attention associated with representation X(t)
ij
a′(t)
ij Raw attention associated with representation X′(t)
ij
α(t)
ij Normalized attention associated with a(t)
ij
α′(t)
ij Normalized attention associated with a′(t)
ij
Ti Target neighbors of node i: {j|(i,j) ∈E}
Si Source neighbors of node i: {j|(j,i) ∈E}
Table 5.The notations in GEAN.
In words, the node-edge-node triplet, (X(t)
i ,E(t)
i→j,X(t)
j ), associated with edge (i,j), is encoded by fully connected layers
to output 5 features, including X′(t+1)
ij and a′(t+1)
ij (a representation and attention to be aggregated in node i), and X(t+1)
ji
and a(t+1)
ji (a representation and attention to be aggregated in node j), and the updated edge representation E(t+1)
i→j . Note
that the features to be aggregated to a target node are marked with the ′, and those to a source node are without the ′(see
Figure 5). After the feature encoding, we perform an attention-weighted neighborhood aggregation for each node, which
can be mathematically described by the following equations.
[[
α(t+1)
ij
]
j∈Ti

[
α′(t+1)
ij
]
j∈Si
]
= Softmax
[[
a(t+1)
ij
]
j∈Ti

[
a′(t+1)
ij
]
j∈Si
]
(16)
X(t+1)
i =
∑
j∈Ti
α(t+1)
ij X(t+1)
ij +
∑
j∈Si
α′(t+1)
ij X′(t+1)
ij (17)
where ∥denotes concatenation.
Learning Compiler Pass Orders using Coreset and Normalized Value Prediction
In comparison, GAT only outputs an attention score by encoding the node-edge-node triplet: a(t+1)
ij = (X(t)
i ,E(t)
i→j,X(t)
j ),
and the feature for neighborhood aggregation is only conditioned on the neighbors: X(t+1)
ij = MLP(X(t)
j ). To summarize,
our encoding approach can ensure that the GNN model is aware of the direction of the edge and that the edge embedding is
updated in each layer, which helps improve the performance (as shown in Table 3).
C. Detailed Results
Dataset Oracle Top-45 Autophase-RL-PPO Autophase-NVP GEAN-RL-PPO GEAN-NVP
anghabench-v1 0.7%/1.011 -1.0%/0.996 -15.9%/0.974 -0.2%/1.002 -0.8%/0.996 -0.0%/1.003
blas-v0 2.6%/1.028 -0.4%/0.997 -1.7%/0.984 2.1%/1.023 -1.0%/0.990 2.4%/1.026
cbench-v1 3.5%/1.041 -2.4%/0.984 -10.1%/0.925 -0.1%/1.008 -1.6%/0.998 2.2%/1.028
chstone-v0 9.3%/1.106 1.2%/1.016 1.3%/1.018 8.3%/1.095 5.4%/1.060 8.8%/1.101
clgen-v0 5.4%/1.060 3.1%/1.034 -0.5%/0.998 4.6%/1.051 0.3%/1.005 5.0%/1.056
csmith-v0 21.2%/1.320 -96.3%/0.851 -116.0%/0.954 21.1%/1.320 -124.6%/0.965 21.1%/1.320
github-v0 1.0%/1.011 0.2%/1.002 0.1%/1.001 0.9%/1.010 -0.2%/0.999 0.9%/1.010
linux-v0 0.6%/1.007 -0.4%/0.998 -0.5%/0.997 0.6%/1.006 -2.3%/0.989 0.6%/1.007
llvm-stress-v0 6.3%/1.087 -18.9%/0.885 -67.0%/0.731 0.7%/1.035 -17.5%/0.888 2.1%/1.045
mibench-v1 1.7%/1.020 0.0%/1.003 -2.8%/0.976 -5.8%/0.963 -0.3%/1.000 -0.1%/1.003
npb-v0 9.8%/1.159 5.7%/1.085 0.9%/1.035 5.1%/1.079 3.7%/1.068 5.5%/1.085
opencv-v0 5.2%/1.061 1.0%/1.013 0.5%/1.007 4.2%/1.051 0.3%/1.004 4.8%/1.057
poj104-v1 7.8%/1.105 3.9%/1.055 -17.5%/0.876 6.1%/1.080 -0.7%/1.008 6.3%/1.082
tensorﬂow-v0 6.1%/1.077 -0.2%/0.998 0.2%/1.004 5.9%/1.075 -0.2%/0.998 5.9%/1.075
Average 5.8%/1.075 -7.5%/0.992 -16.3%/0.960 3.8%/1.054 -10.0%/0.997 4.7%/1.062
Table 6.Detailed evaluation results on held-out test sets.
Learning Compiler Pass Orders using Coreset and Normalized Value Prediction
D. Trajectory comparison
Figure 6.Program optimization example over many steps comparing the Autophase-RL-PPO (blue) approach with our GEAN-NVP
(orange) approach. The dashed line represents the compiler default -Oz performance and higher is better.
Learning Compiler Pass Orders using Coreset and Normalized Value Prediction
E. Compiler passes
Index Flag Index Flag Index Flag
0 -add-discriminators 42 -globalsplit 84 -lower-expect
1 -adce 43 -guard-widening 85 -lower-guard-intrinsic
2 -aggressive-instcombine 44 -hotcoldsplit 86 -lowerinvoke
3 -alignment-from-assumptions 45 -ipconstprop 87 -lower-matrix-intrinsics
4 -always-inline 46 -ipsccp 88 -lowerswitch
5 -argpromotion 47 -indvars 89 -lower-widenable-condition
6 -attributor 48 -irce 90 -memcpyopt
7 -barrier 49 -infer-address-spaces 91 -mergefunc
8 -bdce 50 -inferattrs 92 -mergeicmps
9 -break-crit-edges 51 -inject-tli-mappings 93 -mldst-motion
10 -simplifycfg 52 -instsimplify 94 -sancov
11 -callsite-splitting 53 -instcombine 95 -name-anon-globals
12 -called-value-propagation 54 -instnamer 96 -nary-reassociate
13 -canonicalize-aliases 55 -jump-threading 97 -newgvn
14 -consthoist 56 -lcssa 98 -pgo-memop-opt
15 -constmerge 57 -licm 99 -partial-inliner
16 -constprop 58 -libcalls-shrinkwrap 100 -partially-inline-libcalls
17 -coro-cleanup 59 -load-store-vectorizer 101 -post-inline-ee-instrument
18 -coro-early 60 -loop-data-prefetch 102 -functionattrs
19 -coro-elide 61 -loop-deletion 103 -mem2reg
20 -coro-split 62 -loop-distribute 104 -prune-eh
21 -correlated-propagation 63 -loop-fusion 105 -reassociate
22 -cross-dso-cﬁ 64 -loop-guard-widening 106 -redundant-dbg-inst-elim
23 -deadargelim 65 -loop-idiom 107 -rpo-functionattrs
24 -dce 66 -loop-instsimplify 108 -rewrite-statepoints-for-gc
25 -die 67 -loop-interchange 109 -sccp
26 -dse 68 -loop-load-elim 110 -slp-vectorizer
27 -reg2mem 69 -loop-predication 111 -sroa
28 -div-rem-pairs 70 -loop-reroll 112 -scalarizer
29 -early-cse-memssa 71 -loop-rotate 113 -separate-const-offset-from-gep
30 -early-cse 72 -loop-simplifycfg 114 -simple-loop-unswitch
31 -elim-avail-extern 73 -loop-simplify 115 -sink
32 -ee-instrument 74 -loop-sink 116 -speculative-execution
33 -ﬂattencfg 75 -loop-reduce 117 -slsr
34 -ﬂoat2int 76 -loop-unroll-and-jam 118 -strip-dead-prototypes
35 -forceattrs 77 -loop-unroll 119 -strip-debug-declare
36 -inline 78 -loop-unswitch 120 -strip-nondebug
37 -insert-gcov-proﬁling 79 -loop-vectorize 121 -strip
38 -gvn-hoist 80 -loop-versioning-licm 122 -tailcallelim
39 -gvn 81 -loop-versioning 123 -mergereturn
40 -globaldce 82 -loweratomic
41 -globalopt 83 -lower-constant-intrinsics
Table 7.A list of LLVM compiler pass indices and their corresponding command line ﬂag.

Consistent * Complete* W
ellDocumented*EasytoReuse* *
Evaluated
*
CGO
*
Artifact *
AEC
Synthesizing Benchmarks for Predictive Modeling
Chris Cummins
Pavlos Petoumenos
University of Edinburgh, UK
{c.cummins,ppetoume}@inf.ed.ac.uk
Zheng Wang
Lancaster University, UK
z.wang@lancaster.ac.uk
Hugh Leather
University of Edinburgh, UK
hleather@inf.ed.ac.uk
Abstract
Predictive modeling using machine learning is an effective
method for building compiler heuristics, but there is a short-
age of benchmarks. Typical machine learning experiments
outside of the compilation ﬁeld train over thousands or mil-
lions of examples. In machine learning for compilers, how-
ever, there are typically only a few dozen common bench-
marks available. This limits the quality of learned models,
as they have very sparse training data for what are often
high-dimensional feature spaces. What is needed is a way
to generate an unbounded number of training programs that
ﬁnely cover the feature space. At the same time the generated
programs must be similar to the types of programs that human
developers actually write, otherwise the learning will target
the wrong parts of the feature space.
We mine open source repositories for program fragments
and apply deep learning techniques to automatically con-
struct models for how humans write programs. We sample
these models to generate an unbounded number of runnable
training programs. The quality of the programs is such that
even human developers struggle to distinguish our generated
programs from hand-written code.
We use our generator for OpenCL programs, CLgen, to
automatically synthesize thousands of programs and show
that learning over these improves the performance of a
state of the art predictive model by 1.27×. In addition, the
ﬁne covering of the feature space automatically exposes
weaknesses in the feature design which are invisible with
the sparse training examples from existing benchmark suites.
Correcting these weaknesses further increases performance
by 4.30×.
Categories and Subject Descriptors D.3.4 [ Program-
ming Languages]: Processors—code generation, compilers,
optimization
Keywords Synthetic program generation, OpenCL, Bench-
marking, Deep Learning, GPUs
1. Introduction
Predictive modeling is a well researched method for building
optimization heuristics that often exceed human experts and
Ad-hoc 
Drivers
clsmithclsmithDatasets
clsmithclsmithTraining 
Programs
Feature 
Extractor
clsmithclsmithTraining 
Data
Predictive 
Model
Parameters
Features
Performance 
measurements
Figure 1. Training a predictive model.
reduces development time [1–11]. Figure 1 shows the process
by which these models are trained. A set of training programs
are identiﬁed which are expected to be representative of the
application domain. The programs are compiled and executed
with different parameter values for the target heuristic, to de-
termine which are the best values for each training program.
Each program is also summarized by a vector of features
which describe the information that is expected to be impor-
tant in predicting the best heuristic parameter values. These
training examples of program features and desired heuristic
values are used to create a machine learning model which,
when given the features from a new, unseen program, can
predict good heuristic values for it.
It is common for feature vectors to contain dozens of
elements. This means that a large volume of training data is
needed to have an adequate sampling over the feature space.
Without it, the machine learned models can only capture
the coarse characteristics of the heuristic, and new programs
which do not lie near to training points may be wrongly
predicted. The accuracy of the machine learned heuristic is
thus limited by the sparsity of the training points.
There have been efforts to solve this problem using tem-
plates. The essence of the approach is to construct a prob-
abilistic grammar with embedded semantic actions that de-
ﬁnes a language of possible programs. New programs may be
created by sampling the grammar and, through setting proba-
bilities on the grammar productions, the sampling is biased
towards producing programs from one part of the space or an-
other. This technique is potentially completely general, since
a grammar can theoretically be constructed to match any de-
sired program domain. However, despite being theoretically
possible, it is not easy to construct grammars which are both
suitably general and also produce programs that are in any
978-1-5090-4931-8/17 c⃝ 2017 IEEE CGO 2017, Austin, USA
Accepted for publication by IEEE.c⃝ 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/
republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
86
way similar to human written programs. It has been shown to
be successful over a highly restricted space of stencil bench-
marks with little control ﬂow or program variability [4, 8].
But, it is not clear how much effort it will take, or even if it
is possible for human experts to deﬁne grammars capable of
producing human like programs in more complex domains.
By contrast, our approach does not require an expert to
deﬁne what human programs look like. Instead, we automat-
ically infer the structure and likelihood of programs over a
huge corpus of open source projects. From this corpus, we
learn a probability distribution over sets of characters seen
in human written code. Later, we sample from this distribu-
tion to generate new random programs which, because the
distribution models human written code, are indistinguish-
able from human code. We can then populate our training
data with an unbounded number of human like programs,
covering the space far more ﬁnely than either existing bench-
mark suites or even the corpus of open source projects. Our
approach is enabled by two recent developments:
The ﬁrst is the breakthrough effectiveness of deep learning
for modeling complex structure in natural languages [12, 13].
As we show, deep learning is capable not just of learning the
macro syntactical and semantic structure of programs, but
also the nuances of how humans typically write code. It is
truly remarkable when one considers that it is given no prior
knowledge of the syntax or semantics of the language.
The second is the increasing popularity of public and open
platforms for hosting software projects and source code. This
popularity furnishes us with the thousands of programming
examples that are necessary to feed into the deep learning.
These open source examples are not, sadly, as useful for
directly learning the compiler heuristics since they are not
presented in a uniform, runnable manner, nor do they typically
have extractable test data. Preparing each of the thousands
of open source projects to be directly applicable for learning
compiler heuristics would be an insurmountable task. In
addition to our program generator, CLgen, we also provide
an accompanying host driver which generates datasets for,
then executes and proﬁles synthesized programs.
We make the following contributions:
• We are the ﬁrst to apply deep learning over source codes
to synthesize compilable, executable benchmarks.
• A novel tool CLgen 1 for general-purpose benchmark
synthesis using deep learning. CLgen automatically and
rapidly generates thousands of human like programs for
use in predictive modeling.
• We use CLgen to automatically improve the performance
of a state of the art predictive model by1.27×, and expose
limitations in the feature design of the model which, after
correcting, further increases performance by 4.30×.
1https://github.com/ChrisCummins/clgen
Rodinia
NVIDIA SDK
AMD SDK
Parboil
NAS
Polybench
SHOC
Ad-hoc
ISPASS
Ploybench
Lonestar
SPEC-Viewperf
MARS
GPGPUsim
0
1
2
3
4
5
6
7
#. benchmarks used
Figure 2. The average number of benchmarks used in
GPGPU research papers, organized by origin. In this work
we use the seven most popular benchmark suites.
2. Motivation
In this section we make the argument for synthetic bench-
marks. We identiﬁed frequently used benchmark suites in a
survey of 25 research papers in the ﬁeld of GPGPU perfor-
mance tuning from four top tier conferences between 2013–
2016: CGO, HiPC, PACT, and PPoPP. We found the average
number of benchmarks used in each paper to be 17, and that
a small pool of benchmarks suites account for the majority of
results, shown in Figure 2. We selected the 7 most frequently
used benchmark suites (accounting for 92% of results), and
evaluated the performance of the state of the art Grewe et
al. [14] predictive model across each. The model predicts
whether running a given OpenCL kernel on the GPU gives
better performance than on the CPU. We describe the full
experimental methodology in Section 7.
Table 1 summarizes our results. The performance of a
model trained on one benchmark suite and used to predict the
mapping for another suite is generally very poor. The bench-
mark suite which provides the best results, NVIDIA SDK,
achieves on average only 49% of the optimal performance.
The worst case is when training with Parboil to predict the
optimal mappings for Polybench, where the model achieves
only 11.5% of the optimal performance. From this it is clear
that heuristics learned on one benchmark suite fail to general-
ize across other suites.
This problem is caused both by the limited number of
benchmarks contained in each suite, and the distribution of
benchmarks within the feature space. Figure 3 shows the fea-
ture space of the Parboil benchmark suite, showing whether,
for each benchmark, the model was able to correctly predict
the appropriate optimization. We used Principle Component
Analysis to reduce the multi-dimensional feature space to aid
visualization.
As we see in Figure 3a, there is a dense cluster of neigh-
boring benchmarks, a smaller cluster of three benchmarks,
and two outliers. The lack of neighboring observations means
that the model is unable to learn a good heuristic for the two
outliers, which leads to them being incorrectly optimized. In
87
AMD NPB NVIDIA Parboil Polybench Rodinia SHOC
AMD - 38.0% 74.5% 76.7% 21.7% 45.8% 35.9%
NPB 22.7% - 45.3% 36.7% 13.4% 16.1% 23.7%
NVIDIA 29.9% 37.9% - 21.8% 78.3% 18.1% 63.2%
Parboil 89.2% 28.2% 28.2% - 41.3% 73.0% 33.8%
Polybench 58.6% 30.8% 45.3% 11.5% - 43.9% 12.1%
Rodinia 39.8% 36.4% 29.7% 36.5% 46.1% - 59.9%
SHOC 42.9% 71.5% 74.1% 41.4% 35.7% 81.0% -
Table 1. Performance relative to the optimal of the Grewe et al. predictive model across different benchmark suites on an AMD
GPU. The columns show the suite used for training; the rows show the suite used for testing.
Principle Component 1 →
Principle Component 2 →
Correct
Incorrect
(a)
Principle Component 1 →
Principle Component 2 →
Correct
Incorrect
Additional (b)
Figure 3. A two dimensional projection of the Grewe et al.
feature space, showing predictive model results over Parboil
benchmarks on an NVIDIA GPU. Two outliers in (a) are
incorrectly predicted due to the lack of nearby observations.
The addition of neighboring observations in (b) corrects this.
Figure 3b, we hand-selected benchmarks which are neigh-
bouring in the feature space and retrained the model. The
addition of these observations (and the information they pro-
vide about that part of the feature space) causes the two
outliers to be correctly optimized. We found such outliers in
all of the benchmark suites of Table 1.
These results highlight the signiﬁcant effect that the num-
ber and distribution of training programs has on the quality
of predictive models. Without good coverage of the feature
space, any machine learning methodology is unlikely to pro-
duce high quality heuristics, suitable for general use on ar-
bitrary real applications, or even applications from different
benchmark suites. Our novel approach, described in the next
section, solves this problem by generating an unbounded
number of programs to cover the feature space with ﬁne
granularity.
3. Overview of Our Approach
In this paper we present CLgen, a tool for synthesizing
OpenCL benchmarks, and an accompanying host driver for
executing synthetic benchmarks for gathering performance
data for predictive modeling. While we demonstrate our
approach using OpenCL, it is language agnostic. Our tool
CLgen learns the semantics and structure from over a million
lines of hand-written code from GitHub, and synthesizes
programs through a process of iterative model sampling.
CLgen
Host Driver
Language 
Corpus
GitHub
Software
Repositories
clsmithclsmithContent Files Rejection 
Filter
Search 
engine
Code 
Rewriter
Model 
parameters
Rejection 
Filter
LSTM network
Synthesizer
Synthesis 
parameters
Argument 
Extractor
Benchmark 
parameters
clsmithclsmithSynthesized 
Benchmarks
Benchmark 
Driver
clsmithclsmithSynthesized 
Payloads
clsmithclsmithPerformance 
Results
Dynamic 
Checker
Figure 4. Benchmark synthesis and execution pipeline.
We use a host driver to execute the synthesized programs
to gather performance data for use in predictive modeling.
Figure 4 provides an overview of the program synthesis and
execution pipeline. Our approach extends the state of the
art by providing a general-purpose solution for benchmark
synthesis, leading to better and more accurate predictive
models.
In the course of evaluating our technique against prior
work we discovered that it is also useful for evaluating the
quality of features. Since we are able to cover the space
so much more ﬁnely than the prior work, which only used
standard benchmark suites, we are able to ﬁnd multiple
programs with identical feature values but different best
heuristic values. This indicates that the features are not
sufﬁciently discriminative and should be extended with more
information to allow those programs to be separated. We
go on to show that doing this signiﬁcantly increases the
performance of the learned heuristics. We expect that our
technique will be valuable for feature designers.
88
4. CLgen: Benchmark Synthesis
CLgen is an undirected, general-purpose program synthesizer
for OpenCL. It adopts and augments recent advanced tech-
niques from deep learning to learn over massive codebases. In
contrast to existing grammar and template based approaches,
CLgen is entirely probabilistic. The systemlearns to program
using neural networks which model the semantics and usage
of a huge corpus of code fragments in the target programming
language. This section describes the assembly of an OpenCL
language corpus, the application of deep learning over this
corpus, and the process of synthesizing programs.
4.1 An OpenCL Language Corpus
Deep learning requires large datasets [15]. For the purpose of
modeling a programming language, this means assembling a
very large collection of real, hand-written source codes. We
assembled OpenCL codes by mining public repositories on
the popular code hosting site GitHub.
This is itself a challenging task since OpenCL is an
embedded language, meaning device code is often difﬁcult
to untangle since GitHub does not presently recognize it as
a searchable programming language. We developed a search
engine which attempts to identify and download standalone
OpenCL ﬁles through a process of ﬁle scraping and recursive
header inlining. The result is a 2.8 million line dataset of
8078 “content ﬁles” which potentially contain OpenCL code,
originating from 793 GitHub repositories.
We prune the raw dataset extracted from GitHub using
a custom toolchain we developed for rejection ﬁltering and
code rewriting, built on LLVM.
Rejection Filter The rejection ﬁlter accepts as input a
content ﬁle and returns whether or not it contains compilable,
executable OpenCL code. To do this we attempt to compile
the input to NVIDIA PTX bytecode and perform static
analysis to ensure a minimum static instruction count of three.
We discard any inputs which do not compile or contain fewer
than three instructions.
During initial development it became apparent that isolat-
ing the OpenCL device code leads to a higher-than-expected
discard rate (that is, seemingly valid OpenCL ﬁles being
rejected). Through analyzing 148k lines of compilation er-
rors, we discovered a large number of failures caused by
undeclared identiﬁers — a result of isolating device code
— 50% of undeclared identiﬁer errors in the GitHub dataset
were caused by only 60 unique identiﬁers. To address this,
we developed a shim header which contains inferred values
for common type deﬁnitions (e.g. FLOAT_T), and common
constants (e.g. WGSIZE), shown in Listing 1.
Injecting the shim decreases the discard rate from 40%
to 32%, responsible for an additional 88k lines of code in
the ﬁnal language corpus. The resulting dataset is 2.0 million
lines of compilable OpenCL source code.
1 / * Enable OpenCL f e a t u r e s * /
2 # d e f i n e c l _ c l a n g _ s t o r a g e _ c l a s s _ s p e c i f i e r s
3 # d e f i n e c l _ k h r _ f p 6 4
4 # i n c l u d e< c l c / c l c . h>
5
6 / * I n f e r r e d t y p e s * /
7 t y p e d e f f l o a t FLOAT_T ;
8 t y p e d e f unsigned i n t INDEX_TYPE ;
. . . (36 more )
9
10 / * I n f e r r e d c o n s t a n t s * /
11 # d e f i n e M_PI 3 . 1 40 2 5
12 # d e f i n e WG_SIZE 128
. . . (185 more )
Listing 1. The shim header ﬁle, providing inferred type
aliases and constants for OpenCL on GitHub.
Code Rewriter Programming languages have few of the
issues of semantic interpretation present in natural language,
though there remains many sources of variance at the syntac-
tic level. For example, the presence and content of comments
in code, and the choice of identifying names given to vari-
ables. We consider these ambiguities to be non-functional
variance, and developed a tool to normalize code of these
variances so as to make the code more amenable to machine
learning. This is a three step process:
1. The source is pre-processed to remove macros, conditional
compilation, and source comments.
2. Identiﬁers are rewritten to have a short but unique name
based on their order of appearance, using the sequential
series {a, b, c, . . . , aa, ab, ac, . . .}for variables and {A,
B, C, . . . , AA, AB, AC, . . .}for functions. This process
isolates the syntactic structure of the code, and unlike prior
work [16], our rewrite method preserves program behavior.
Language built-ins (e.g. get_global_id, asin) are not
rewritten.
3. A variant of the Google C++ code style is enforced to
ensure consistent use of braces, parentheses, and white
space.
An example of the code rewriting process is shown in Figure 5.
A side effect of this process is a reduction in code size,
largely due to the removal of comments and excess white
space. The ﬁnal language corpus contains 1.3 million lines
of transformed OpenCL, consisting of 9487 kernel functions.
Identiﬁer rewriting reduces the bag-of-words vocabulary size
by 84%.
4.2 Learning OpenCL
Generating valid, executable program code is an ambitious
and challenging goal for unsupervised machine learning. We
employ state of the art deep language modeling techniques to
achieve this task.
We use the Long Short-Term Memory (LSTM) architec-
ture of Recurrent Neural Network [17, 18] to learn a character-
level language model over the corpus of OpenCL compute
kernels. The LSTM network architecture comprises recurrent
89
1 # d e f i n eDTYPE f l o a t
2 # d e f i n eALPHA( a ) 3 . 5 f * a
3 i n l i n e DTYPE ax (DTYPE x ) { return ALPHA( x ) ; }
4
5 _ _ k e r n e l void saxpy ( / * SAXPY k e r n e l * /
6 _ _ g l o b a l DTYPE * i n p u t 1 ,
7 _ _ g l o b a l DTYPE * i n p u t 2 ,
8 c o n s t i n t nelem )
9 {
10 unsigned i n t i d x = g e t _ g l o b a l _ i d ( 0 ) ;
11 / / = ax + y
12 i f ( i d x < nelem ) {
13 i n p u t 2 [ i d x ] += ax ( i n p u t 1 [ i d x ] ) ; }}
(a) Example content ﬁle
1 i n l i n e f l o a t A( f l o a t a ) {
2 return 3 . 5 f * a ;
3 }
4
5 _ _ k e r n e l void B( _ _ g l o b a l f l o a t* b , _ _ g l o b a l f l o a t* c ,
↪→ c o n s t i n t d ) {
6 unsigned i n t e = g e t _ g l o b a l _ i d ( 0 ) ;
7
8 i f ( e < d ) {
9 c [ e ] += A( b [ e ] ) ;
10 }
11 }
(b) Content ﬁle after code rewriting
Figure 5. The code rewriting process, which transforms code
to make it more amenable to language modeling.
layers of memory cells, each consisting of an input, output,
and forget gate, and an output layer providing normalized
probability values from a 1-of-K coded vocabulary [19].
We use a 3-layer LSTM network with 2048 nodes per layer,
implemented in Torch. We train this 17-million parameter
model using Stochastic Gradient Descent for 50 epochs,
using an initial learning rate of 0.002, decaying by a factor
of one half every 5 epochs. Training took three weeks on a
single machine using an NVIDIA GTX Titan, with a ﬁnal
model size of 648MB. Training the network is a one-off cost,
and can be parallelized across devices. The trained network
can be deployed to lower-compute machines for use.
4.3 Synthesizing OpenCL
We synthesize OpenCL compute kernels by iteratively sam-
pling the learned language model. We implemented two
modes for model sampling: the ﬁrst involves providing an ar-
gument speciﬁcation, stating the data types and modiﬁers of
all kernel arguments. When an argument speciﬁcation is pro-
vided, the model synthesizes kernels matching this signature.
In the second sampling mode this argument speciﬁcation is
omitted, allowing the model to synthesize compute kernels of
arbitrary signatures, dictated by the distribution of argument
types within the language corpus.
In either mode we generate a seed text, and sample the
model, character by character, until the end of the compute
kernel is reached, or until a predetermined maximum number
of characters is reached. Algorithm 1 illustrates this process.
Algorithm 1 Sampling a candidate kernel from a seed text.
Require: LSTM model M, maximum kernel length n.
Ensure: Completed sample string S.
1: S ←“__kernel void A(const int a) {” Seed text
2: d ←1 Initial code block depth
3: for i ←|S|to n do
4: c ←predictcharacter(M, S) Generate new character
5: if c =“{” then
6: d ←d + 1 Entered code block, increase depth
7: else if c =“}” then
8: d ←d −1 Exited code block, decrease depth
9: end if
10: S ←S + c Append new character
11: if depth = 0then
12: break Exited function block, stop sampling
13: end if
14: end for
The same rejection ﬁlter described in Section 4.1 then either
accepts or rejects the sample as a candidate synthetic bench-
mark. Listing 6 shows three examples of unique compute
kernels generated in this manner from an argument speciﬁ-
cation of three single-precision ﬂoating-point arrays and a
read-only signed integer. We evaluate the quality of synthe-
sized code in Section 6.
5. Benchmark Execution
We developed a host driver to gather performance data from
synthesized CLgen code. The driver accepts as input an
OpenCL kernel, generates payloads of user-conﬁgurable
sizes, and executes the kernel using the generated payloads,
providing dynamic checking of kernel behavior.
5.1 Generating Payloads
A payload encapsulates all of the arguments of an OpenCL
compute kernel. After parsing the input kernel to derive
argument types, a rule-based approach is used to generate
synthetic payloads. For a given global size Sg: host buffers
of Sg elements are allocated and populated with random
values for global pointer arguments, device-only buffers
of Sg elements are allocated for local pointer arguments,
integral arguments are given the valueSg, and all other scalar
arguments are given random values. Host to device data
transfers are enqueued for all non-write-only global buffers,
and all non-read-only global buffers are transferred back to
the host after kernel execution.
5.2 Dynamic Checker
For the purpose of performance benchmarking we are not
interested in the correctness of computed values, but we
deﬁne a class of programs as performing useful work if they
predictably compute some result. We devised a low-overhead
runtime behavior check to validate that a synthesized program
does useful work based on the outcome of four executions of
a tested program:
90
1 _ _ k e r n e l void A( _ _ g l o b a l f l o a t* a ,
2 _ _ g l o b a l f l o a t* b ,
3 _ _ g l o b a l f l o a t* c ,
4 c o n s t i n t d ) {
5 i n t e = g e t _ g l o b a l _ i d ( 0 ) ;
6 f l o a t f = 0 . 0 ;
7 f o r ( i n t g = 0 ; g < d ; g ++) {
8 c [ g ] = 0 . 0 f ;
9 }
10 b a r r i e r ( 1 ) ;
11
12 a [ g e t _ g l o b a l _ i d ( 0 ) ] = 2 *b [ g e t _ g l o b a l _ i d ( 0 ) ] ;
13 }
(a) Vector operation with branching and synchronization.
1 _ _ k e r n e l void A( _ _ g l o b a l f l o a t* a ,
2 _ _ g l o b a l f l o a t* b ,
3 _ _ g l o b a l f l o a t* c ,
4 c o n s t i n t d ) {
5 i n t e = g e t _ g l o b a l _ i d ( 0 ) ;
6 i f ( e >= d ) {
7 return ;
8 }
9 c [ e ] = a [ e ] + b [ e ] + 2 * a [ e ] + b [ e ] + 4 ;
10 }
(b) Zip operation which computes ci = 3ai + 2bi + 4.
1 _ _ k e r n e l void A( _ _ g l o b a l f l o a t* a ,
2 _ _ g l o b a l f l o a t* b ,
3 _ _ g l o b a l f l o a t* c ,
4 c o n s t i n t d ) {
5 unsigned i n t e = g e t _ g l o b a l _ i d ( 0 ) ;
6 f l o a t 1 6 f = ( f l o a t 1 6) ( 0 . 0 ) ;
7 f o r ( unsigned i n t g = 0 ; g < d ; g ++) {
8 f l o a t 1 6 h = a [ g ] ;
9 f . s0 += h . s0 ;
10 f . s1 += h . s1 ;
11 f . s2 += h . s2 ;
12 f . s3 += h . s3 ;
13 f . s4 += h . s4 ;
14 f . s5 += h . s5 ;
15 f . s6 += h . s6 ;
16 f . s7 += h . s7 ;
17 f . s8 += h . s8 ;
18 f . s9 += h . s9 ;
19 f . sA += h . sA ;
20 f . sB += h . sB ;
21 f . sC += h . sC ;
22 f . sD += h . sD ;
23 f . sE += h . sE ;
24 f . sF += h . sF ;
25 }
26 b [ e ] = f . s0 + f . s1 + f . s2 + f . s3 + f . s4 + f . s5 +
↪→ f . s6 + f . s7 + f . s8 + f . s9 + f . sA + f . sB +
↪→ f . sC + f . sD + f . sE + f . sF ;
27 }
(c) Partial reduction over reinterpreted vector type.
Figure 6. Compute kernels synthesized with CLgen. All
three kernel were synthesized from the same argument spec-
iﬁcation: three single-precision ﬂoating-point arrays and a
read-only signed integer.
1. Create 4 equal size payloads A1in, B1in, A2in, B2in,
subject to restrictions: A1in = A2in, B1in = B2in,
A1in ̸= B1in.
2. Execute kernel k 4 times: k(A1in) →A1out, k(B1in) →
B1out, k(A2in) →A2out, k(B2in) →B2out.
3. Assert:
• A1out ̸= A1in and B1out ̸= B1in, else k has no output
(for these inputs).
• A1out ̸= B1out and A2out ̸= B2out, else k is input
insensitive t (for these inputs).
• A1out = A2out and B1out = B2out, else k is non-
deterministic.
Equality checks for ﬂoating point values are performed with
an appropriate epsilon to accommodate rounding errors, and
a timeout threshold is also used to catch kernels which are
non-terminating. Our method is based on random differential
testing [20], though we emphasize that this is not a general
purpose approach and is tailored speciﬁcally for our use
case. For example, we anticipate a false positive rate for
kernels with subtle sources of non-determinism which more
thorough methods may expose [21–23], however we deemed
such methods unnecessary for our purpose of performance
modeling.
6. Evaluation of Synthetic Programs
In this section we evaluate the quality of programs synthe-
sized by CLgen by their likeness to hand-written code, and
discuss limitations of the synthesis and execution pipeline.
6.1 Likeness to Hand-written Code
Judging whether a piece of code has been written by a
human is a challenging task for a machine, so we adopt a
methodology from machine learning research based on the
Turing Test [24–26]. We reason that if the output of CLgen
is human like code, then a human judge will be unable to
distinguish it from hand-written code.
We devised a double blind test in which 15 volunteer
OpenCL developers from industry and academia were shown
10 OpenCL kernels each. Participants were tasked with
judging whether, for each kernel, they believed it to have
been written by hand or by machine. Kernels were randomly
selected for each participant from two equal sized pools of
synthetically generated and hand-written code from GitHub.
We applied the code rewriting process to all kernels to
remove comments and ensure uniform identiﬁer naming. The
participants were divided into two groups, with 10 of them
receiving code generated by CLgen, and 5 of them acting as
a control group, receiving code generated by CLSmith [27],
a program generator for differential testing1.
We scored each participant’s answers, ﬁnding the average
score of the control group to be 96% (stdev. 9%), an unsurpris-
1An online version of this test is available at http://humanorrobot.uk/.
91
Raw Code Features
comp static #. compute operations
mem static #. accesses to global memory
localmem static #. accesses to local memory
coalesced static #. coalesced memory accesses
transfer dynamic size of data transfers
wgsize dynamic #. work-items per kernel
(a) Individual code features
Combined Code Features
F1: transfer/(comp+mem) commun.-computation ratio
F2: coalesced/mem % coalesced memory accesses
F3: (localmem/mem)×wgsize ratio local to global mem accesses
×#. work-items
F4: comp/mem computation-mem ratio
(b) Combinations of raw features
Table 2. Grewe et al. model features.
ing outcome as generated programs for testing have multiple
“tells”, for example, their only input is a singleulong pointer.
There were no false positives (synthetic code labeled human)
for CLSmith, only false negatives (human code labeled syn-
thetic). With CLgen synthesized programs, the average score
was 52% (stdev. 17%), and the ratio of errors was even. This
suggests that CLgen code is indistinguishable from hand-
written programs, with human judges scoring no better than
random chance.
6.2 Limitations
Our new approach enables the synthesis of more human-like
programs than current state of the art program generators,
and without the expert guidance required by template based
generators, but it has limitations. Our method of seeding the
language models with the start of a function means that we
cannot support user deﬁned types, or calls to user-deﬁned
functions. This means that we only consider scalars and ar-
rays as inputs; while 6 (2.3%) of the benchmark kernels from
Table 3 use irregular data types as inputs. We will address
this limitation through recursive program synthesis, whereby
a call to a user-deﬁned function or unrecognized type will
trigger candidate functions and type deﬁnitions to be synthe-
sized. Currently we only run single-kernel benchmarks. We
will extend the host driver to explore multi-kernel schedules
and interleaving of kernel executions. Our host driver gener-
ates datasets from uniform random distributions, as do many
of the benchmark suites. For cases where non-uniform in-
puts are required (e.g. proﬁle-directed feedback), an alternate
methodology for generating inputs must be adopted.
7. Experimental Methodology
7.1 Experimental Setup
Predictive Model We reproduce the predictive model from
Grewe, Wang, and O’Boyle [14]. The predictive model is
used to determine the optimal mapping of a given OpenCL
kernel to either a GPU or CPU. It uses supervised learning
to construct a decision tree with a combination of static and
Version #. benchmarks #. kernels
NPB (SNU [29]) 1.0.3 7 114
Rodinia [30] 3.1 14 31
NVIDIA SDK 4.2 6 12
AMD SDK 3.0 12 16
Parboil [31] 0.2 6 8
PolyBench [32] 1.0 14 27
SHOC [33] 1.1.5 12 48
Total - 71 256
Table 3. List of benchmarks.
Intel CPU AMD GPU NVIDIA GPU
Model Core i7-3820 Tahiti 7970 GTX 970
Frequency 3.6 GHz 1000 MHz 1050 MHz
#. Cores 4 2048 1664
Memory 8 GB 3 GB 4 GB
Throughput 105 GFLOPS 3.79 TFLOPS 3.90 TFLOPS
Driver AMD 1526.3 AMD 1526.3 NVIDIA 361.42
Compiler GCC 4.7.2 GCC 4.7.2 GCC 5.4.0
Table 4. Experimental platforms.
dynamic kernel features extracted from source code and the
OpenCL runtime, detailed in Table 2b.
Benchmarks As in [14], we test our model on the NAS
Parallel Benchmarks (NPB) [28]. We use the hand-optimized
OpenCL implementation of Seo, Jo, and Lee [29]. In [14]
the authors augment the training set of the predictive model
with 47 additional kernels taken from 4 GPGPU benchmark
suites. To more fully sample the program space, we use
a much larger collection of 142 programs, summarized in
Table 3. These additional programs are taken from all 7
of the most frequently used benchmark suites identiﬁed in
Section 2. None of these programs were used to train CLgen.
We synthesized 1,000 kernels with CLgen to use as additional
benchmarks.
Platforms We evaluate our approach on two 64-bit CPU-
GPU systems, detailed in Table 4. One system has an AMD
GPU and uses OpenSUSE 12.3; the other is equipped with an
NVIDIA GPU and uses Ubuntu 16.04. Both platforms were
unloaded.
Datasets The NPB and Parboil benchmark suites are pack-
aged with multiple datasets. We use all of the packaged
datasets (5 per program in NPB, 1-4 per program in Par-
boil). For all other benchmarks, the default datasets are used.
We conﬁgured the CLgen host driver to synthesize payloads
between 128B-130MB, approximating that of the dataset
sizes found in the benchmark programs.
7.2 Methodology
We replicated the methodology of [14]. Each experiment is
repeated ﬁve times and the average execution time is recorded.
The execution time includes both device compute time and
the data transfer overheads.
We use leave-one-out cross-validation to evaluate predic-
tive models. For each benchmark, a model is trained on data
from all other benchmarks and used to predict the mapping
92
1 _ _ k e r n e l void A( _ _ g l o b a l f l o a t* a ,
2 _ _ g l o b a l f l o a t* b ,
3 _ _ g l o b a l f l o a t* c ,
4 c o n s t i n t d ) {
5 i n t e = g e t _ g l o b a l _ i d ( 0 ) ;
6 i f ( e < 4 & & e < c ) {
7 c [ e ] = a [ e ] + b [ e ] ;
8 a [ e ] = b [ e ] + 1 ;
9 }
10 }
Listing 2. In the Grewe et al. feature space this
CLgen program is indistinguishable from AMD’s Fast
Walsh–Hadamard transform benchmark, but has very
different runtime behavior and optimal device mapping. The
addition of a branching feature ﬁxes this.
for each kernel and dataset in the excluded program. We re-
peat this process with and without the addition of synthetic
benchmarks in the training data. We do not test model predic-
tions on synthetic benchmarks.
8. Experimental Results
We evaluate the effectiveness of our approach on two hetero-
geneous systems. We ﬁrst compare the performance of a state
of the art predictive model [14] with and without the addi-
tion of synthetic benchmarks, then show how the synthetic
benchmarks expose weaknesses in the feature design and how
these can be addressed to develop a better model. Finally we
compare the ability of CLgen to explore the program feature
space against a state of the art program generator [27].
8.1 Performance Evaluation
Figure 7 shows speedups of the Grewe et al. predictive model
over the NAS Parallel Benchmark suite with and without the
addition of synthesized benchmarks for training. Speedups
are calculated relative to the best single-device mapping for
each experimental platform, which is CPU-only for AMD
and GPU-only for NVIDIA. The ﬁne grained coverage of the
feature space which synthetic benchmarks provide improves
performance dramatically for the NAS benchmarks. Across
both systems, we achieve an average speedup of 2.42×
with the addition of synthetic benchmarks, with prediction
improvements over the baseline for 62.5% of benchmarks on
AMD and 53.1% on NVIDIA.
The strongest performance improvements are on NVIDIA
with the FT benchmark which suffers greatly under a single-
device mapping. However, the performance on AMD for the
same benchmark slightly degrades after adding the synthetic
benchmarks, which we address in the next section.
8.2 Extending the Predictive Model
Feature designers are bound to select as features only prop-
erties which are signiﬁcant for the sparse benchmarks they
test on, which can limit a model’s ability to generalize over a
wider range of programs. We found this to be the case with
the Grewe et al. model. The addition of automatically gener-
ated programs exposed two distinct cases where the model
failed to generalize as a result of overspecializing to the NPB
suite.
The ﬁrst case is thatF3 is sparse on many programs. This is
a result of the NPB implementation’s heavy exploitation of lo-
cal memory buffers and the method by which they combined
features (we speculate this was a necessary dimensionality
reduction in the presence of sparse training programs). To
counter this we extended the model to use the raw feature
values in addition to the combined features.
The second case is that some of our generated programs
had identical feature values as in the benchmark set, but had
different behavior (i.e. optimal mappings). Listing 2 shows
one example of a CLgen benchmark which is indistinguish-
able in the feature space to one the of existing benchmarks
— the Fast Walsh-Hadamard transform — but with different
behavior. We found this to be caused by the lack of dis-
criminatory features for branching, since the NPB programs
are implemented in a manner which aggressively minimized
branching. To counter this we extended the predictive model
with an additional feature containing a static count of branch-
ing operations in a kernel.
Figure 8 shows speedups of our extended model across
all seven of the benchmark suites used in Section 2. Model
performance, even on this tenfold increase of benchmarks, is
good. There are three benchmarks on which the model per-
forms poorly: MatrixMul, cutcp, and pathfinder. Each
of those programs make heavy use of loops, which we be-
lieve the static code features of the model fail to capture. This
could be addressed by extracting dynamic instruction counts
using proﬁling, but we considered this beyond the scope of
our work. It is not our goal to perfect the predictive model,
but to show the performance improvements associated with
training on synthetic programs. To this extent, we are suc-
cessful, achieving average speedups of 3.56×on AMD and
5.04×on NVIDIA across a very large test set.
8.3 Comparison of Source Features
As demonstrated in Section 2, the predictive quality of a
model for a given point in the feature space is improved with
the addition of observations from neighboring points. By
producing thousands of artiﬁcial programs modeled on the
structure real OpenCL programs, CLgen is able to consis-
tently and automatically generate programs which are close
in the feature space to the benchmarks which we are testing
on.
To quantify this effect we use the static code features
of Table 2a, plus the branching feature discussed in the
previous subsection, to measure the number of CLgen kernels
generated with the same feature values as those of the
benchmarks we examined in the previous subsections. We
examine only static code features to allow comparison with
the GitHub kernels for which we have no automated method
to execute them and extract runtime features, and CLSmith
generated programs.
93
BT.A
BT.B
BT.S
BT.W
CG.A
CG.B
CG.C
CG.S
CG.W
EP.A
EP.B
EP.C
EP.W
FT.A
FT.B
FT.S
FT.W
LU.A
LU.B
LU.C
LU.S
LU.W
MG.A
MG.B
MG.C
MG.S
MG.W
SP.A
SP.B
SP.C
SP.S
SP.W
Average
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
Speedup over CPU
Grewe et al.
w. CLgen
(a) AMD Tahiti 7970
BT.A
BT.B
BT.S
BT.W
CG.A
CG.B
CG.C
CG.S
CG.W
EP.A
EP.B
EP.C
EP.W
FT.A
FT.B
FT.S
FT.W
LU.A
LU.B
LU.C
LU.S
LU.W
MG.A
MG.B
MG.C
MG.S
MG.W
SP.A
SP.B
SP.C
SP.S
SP.W
Average
1
3
5
7
9
11
13
15
17
Speedup over GPU
Grewe et al.
w. CLgen
(b) NVIDIA GTX 970
Figure 7. Speedup of programs using Grewe et al. predictive model with and without synthetic benchmarks. The predictive
model outperforms the best static device mapping by a factor of 1.26×on AMD and 2.50×on NVIDIA. The addition of
synthetic benchmarks improves the performance to 1.57×on AMD and 3.26×on NVIDIA.
Figure 9 plots the number of matches as a function of
the number of kernels. Out of 10,000 unique CLgen kernels,
more than a third have static feature values matching those
of the benchmarks, providing on average 14 CLgen kernels
for each benchmark. This conﬁrms our original intuition:
CLgen kernels, by emulating the way real humans write
OpenCL programs, are concentrated in the same area of the
feature space as real programs. Moreover, the number of
CLgen kernels we generate is unbounded, allowing us to
continually reﬁne the exploration of the feature space, while
the number of kernels available on GitHub is ﬁnite. CLSmith
rarely produces code similar to real-world OpenCL programs,
with only 0.53% of the generated kernels have matching
feature values with benchmark kernels. We conclude that the
unique contribution of CLgen is its ability to generate many
thousands of programs that are appropriate for predictive
modeling.
9. Related Work
Our work lies at the intersections of a number of areas: pro-
gram generation, benchmark characterization, and language
modeling and learning from source code. There is no existing
work which is similar to ours, in respect to learning from
large corpuses of source code for benchmark generation.
GENESIS [34] is a language for generating synthetic train-
ing programs. Users annotate template programs with sta-
tistical distributions over features, which are instantiated to
generate statistically controlled permutations of templates.
Template based approaches provide domain-speciﬁc solutions
for a constrained feature and program space, for example, gen-
erating permutations of Stencil codes [35, 36]. Our approach
provides general-purpose program generation over unknown
domains, in which the statistical distribution of generated
programs is automatically inferred from real world code.
Random program generation is an effective method for
software testing. Grammar-based fuzz testers have been de-
veloped for C [37] and OpenCL [27]. A mutation-based ap-
proach for the Java Virtual Machine is demonstrated in [38].
Goal-directed program generators have been used for a vari-
ety of domains, including generating linear transforms [39],
MapReduce programs [40], and data structure implementa-
tions [41]. Program synthesis from input/output examples
is used for simple algorithms in [42], string manipulation
in [43], and geometry constructions in [44].
Machine learning has been applied to source code to aid
software engineering. Naturalize employs techniques devel-
oped in the natural language processing domain to model cod-
ing conventions [45]. JSNice leverages probabilistic graphical
models to predict program properties such as identiﬁer names
for Javascript [46].
94
Figure 8. Speedups of predictions using our extended model over Grewe et al. on both experimental platforms. Synthetic
benchmarks and the additional program features outperform the original predictive model by a factor3.56×on AMD and 5.04×
on NVIDIA.
0 2000 4000 6000 8000 10000
#. kernels
0
500
1000
1500
2000
2500
3000
3500
#. matches
GitHub
CLSmith
CLgen
Figure 9. The number of kernels from GitHub, CLSmith,
and CLgen with static code features matching the bench-
marks. CLgen generates kernels that are closer in the feature
space than CLSmith, and can continue to do so long after we
have exhausted the extent of the GitHub dataset. Error bars
show standard deviation of 10 random samplings.
There is an increasing interest in mining source code repos-
itories at large scale [16, 47, 48]. Previous studies have in-
volved data mining of GitHub to analyze software engineer-
ing practices [49–52], for example code generation [53], code
summarization [54], comment generation [55], and code com-
pletion [56]. However, no work so far has exploited mined
source code for benchmark generation. This work is the ﬁrst
to do so.
10. Conclusion
The quality of predictive models is bound by the quantity and
quality of programs used for training, yet there is typically
only a few dozen common benchmarks available for experi-
ments. We present a novel tool which is the ﬁrst of it’s kind
— an entirely probabilistic program generator capable of gen-
erating an unbounded number of human like programs. Our
approach applies deep learning over a huge corpus of publicly
available code from GitHub to automatically infer the seman-
tics and practical usage of a programming language. Our tool
generates programs which to trained eyes are indistinguish-
able from hand-written code. We tested our approach using a
state of the art predictive model, improving its performance
by a factor of 1.27×. We found that synthetic benchmarks
exposed weaknesses in the feature set which, when corrected,
further improved the performance by 4.30×. Our hope for
this work is to demonstrate a proof of concept for an exciting
new avenue of program generation, and that the full release
of CLgen will expedite discovery in other domains. In future
work we will extend the approach to multiple programming
languages, and investigate methods for performing an auto-
matic directed search of feature spaces.
Acknowledgments
Our thanks to the volunteers at Codeplay Software Ltd and
the University of Edinburgh for participating in the qual-
itative evaluation. This work was supported by the UK
Engineering and Physical Sciences Research Council un-
der grants EP/L01503X/1 (CDT in Pervasive Parallelism),
EP/L000055/1 (ALEA), EP/M01567X/1 (SANDeRs), EP/M0
15823/1, and EP/M015793/1 (DIVIDEND). The code and
data for this paper are available at:
http://chriscummins.cc/cgo17.
95
References
[1] P. Micolet, A. Smith, and C. Dubach. “A Machine
Learning Approach to Mapping Streaming Workloads
to Dynamic Multicore Processors”. In: LCTES. 2016.
[2] Z. Wang, G. Tournavitis, B. Franke, and M. O’Boyle.
“Integrating Proﬁle-driven Parallelism Detection and
Machine-learning-based Mapping”. In: TACO (2014).
[3] A. Magni, C. Dubach, and M. O’Boyle. “Automatic
Optimization of Thread-Coarsening for Graphics Pro-
cessors”. In: PACT. ACM, 2014, pp. 455–466.
[4] C. Cummins, P. Petoumenos, M. Steuwer, and H.
Leather. “Towards Collaborative Performance Tuning
of Algorithmic Skeletons”. In: HLPGPU. 2016.
[5] Z. Wang and M. O’Boyle. “Mapping Parallelism to
Multi-cores: A Machine Learning Based Approach”.
In: PPoPP. 15. ACM, 2009, pp. 75–84.
[6] Y . Wen, Z. Wang, and M. O’Boyle. “Smart Multi-
Task Scheduling for OpenCL Programs on CPU/GPU
Heterogeneous Platforms”. In: HiPC. IEEE, 2014.
[7] Z. Wang and M. O’Boyle. “Partitioning Streaming
Parallelism for Multi-cores: A Machine Learning Based
Approach”. In: PACT. ACM, 2010, pp. 307–318.
[8] T. L. Falch and A. C. Elster. “Machine Learning Based
Auto-tuning for Enhanced OpenCL Performance Porta-
bility”. In: IPDPSW. IEEE, 2015.
[9] A. Collins, C. Fensch, and H. Leather. “Auto-Tuning
Parallel Skeletons”. In: Parallel Processing Letters
22.02 (June 2012), p. 1240005.
[10] H. Leather, E. Bonilla, and M. O’Boyle. “Automatic
Feature Generation for Machine Learning Based Opti-
mizing Compilation”. In: TACO 11 (2014).
[11] W. F. Ogilvie, P. Petoumenos, Z. Wang, and H. Leather.
“Fast Automatic Heuristic Construction Using Active
Learning”. In: LCPC. 2014.
[12] A. Graves. “Generating Sequences with Recurrent
Neural Networks”. In: arXiv:1308.0850 (2013).
[13] I. Sutskever, O. Vinyals, and Q. V . Le. “Sequence to
Sequence Learning with Neural Networks”. In: NIPS.
2014.
[14] D. Grewe, Z. Wang, and M. O’Boyle. “Portable Map-
ping of Data Parallel Programs to OpenCL for Hetero-
geneous Systems”. In: CGO. IEEE, 2013.
[15] Y . LeCun, Y . Bengio, and G. Hinton. “Deep learning”.
In: Nature 521.7553 (2015), pp. 436–444.
[16] M. Allamanis and C. Sutton. “Mining Source Code
Repositories at Massive Scale using Language Model-
ing”. In: MSR. 2013, pp. 207–216.
[17] M. Sundermeyer, R. Schl, and H. Ney. “LSTM Neural
Networks for Language Modeling”. In: Interspeech.
2012.
[18] T. Mikolov. “Recurrent Neural Network based Lan-
guage Model”. In: Interspeech. 2010.
[19] A. Graves and J. Schmidhuber. “Framewise Phoneme
Classiﬁcation with Bidirectional LSTM and Other
Neural Network Architectures”. In: Neural Networks
5.5 (18), pp. 602–610.
[20] W. M. McKeeman. “Differential Testing for Software”.
In: DTJ 10.1 (1998), pp. 100–107.
[21] A. Betts, N. Chong, and A. Donaldson. “GPUVerify: A
Veriﬁer for GPU Kernels”. In:OOPSLA. 2012, pp. 113–
131.
[22] J. Price and S. Mcintosh-Smith. “Oclgrind: An Exten-
sible OpenCL Device Simulator”. In: IWOCL. ACM,
2015.
[23] T. Sorensen and A. Donaldson. “Exposing Errors Re-
lated to Weak Memory in GPU Applications”. In:PLDI.
2016.
[24] H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and
W. Xu. “Are You Talking to a Machine? Dataset and
Methods for Multilingual Image Question Answering”.
In: arXiv:1505.05612 (2015).
[25] R. Zhang, P. Isola, and A. A. Efros. “Colorful Image
Colorization”. In: arXiv:1603.08511 (2016).
[26] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. “Show
and Tell: A Neural Image Caption Generator”. In:
CVPR (2015).
[27] C. Lidbury, A. Lascu, N. Chong, and A. Donald-
son. “Many-Core Compiler Fuzzing”. In:PLDI. 2015,
pp. 65–76.
[28] D. H. Bailey, E. Barszcz, J. Barton, D. Browning, R.
Carter, L. Dagum, R. Fatoohi, S. Fineberg, P. Frederick-
son, T. Lasinski, R. Schreiber, H. Simon, V . Venkatakr-
ishnan, and S. Weeratunga. “The NAS Parallel Bench-
marks”. In: IJHPCA 5.3 (1991), pp. 63–73.
[29] S. Seo, G. Jo, and J. Lee. “Performance Characteriza-
tion of the NAS Parallel Benchmarks in OpenCL”. In:
IISWC. IEEE, 2011.
[30] S. Che, M. Boyer, J. Meng, D. Tarjan, J. W. Sheaffer,
S. H. Lee, and K. Skadron. “Rodinia: A Benchmark
Suite for Heterogeneous Computing”. In:IISWC. IEEE,
Oct. 2009.
[31] J. A. Stratton, C. Rodrigues, I. Sung, N. Obeid, L.
Chang, N. Anssari, G. D. Liu, and W. W. Hwu. “Par-
boil: A Revised Benchmark Suite for Scientiﬁc and
Commercial Throughput Computing”. In: Center for
Reliable and High-Performance Computing (2012).
[32] S. Grauer-Gray, L. Xu, R. Searles, S. Ayalasomayajula,
and J. Cavazos. “Auto-tuning a High-Level Language
Targeted to GPU Codes”. In:InPar. 2012.
[33] A. Danalis, G. Marin, C. McCurdy, J. S. Meredith, P. C.
Roth, K. Spafford, V . Tipparaju, and J. S. Vetter. “The
Scalable HeterOgeneous Computing (SHOC) Bench-
mark Suite”. In: GPGPU. ACM, 2010.
96
[34] A. Chiu, J. Garvey, and T. S. Abdelrahman. “Genesis: A
Language for Generating Synthetic Training Programs
for Machine Learning”. In: CF. ACM, 2015, p. 8.
[35] J. D. Garvey and T. S. Abdelrahman. “Automatic Per-
formance Tuning of Stencil Computations on GPUs”.
In: ICPP (2015).
[36] C. Cummins, P. Petoumenos, M. Steuwer, and H.
Leather. “Autotuning OpenCL Workgroup Size for
Stencil Patterns”. In: ADAPT. 2016.
[37] X. Yang, Y . Chen, E. Eide, and J. Regehr. “Finding and
Understanding Bugs in C Compilers”. In: PLDI. 2011.
[38] Y . Chen, T. Su, C. Sun, Z. Su, and J. Zhao. “Coverage-
Directed Differential Testing of JVM Implementations”.
In: PLDI. 2016.
[39] Y . V oronenko, F. De Mesmay, and M. Püschel. “Com-
puter Generation of General Size Linear Transform
Libraries”. In: CGO. IEEE, 2009, pp. 102–113.
[40] C. Smith. “MapReduce Program Synthesis”. In:PLDI.
2016.
[41] C. Loncaric, T. Emina, and M. D. Ernst. “Fast Synthesis
of Fast Collections”. In: PLDI. 2016.
[42] W. Zaremba, T. Mikolov, A. Joulin, and R. Fergus.
“Learning Simple Algorithms from Examples”. In:
ICML. 2016.
[43] S. Gulwani. “Automating string processing in spread-
sheets using input-output examples”. In: POPL. 2011.
[44] S. Gulwani, V . A. Korthikanti, and A. Tiwari. “Synthe-
sizing geometry constructions”. In: PLDI. 2011.
[45] M. Allamanis, E. T. Barr, C. Bird, and C. Sutton.
“Learning Natural Coding Conventions”. In:FSE. 2014,
pp. 281–293.
[46] Veselin Raychev, Martin Vechev, and Andreas Krause.
“Predicting Program Properties from “Big Code””. In:
POPL. 2015.
[47] M. White, C. Vendome, M. Linares-Vasquez, and D.
Poshyvanyk. “Toward Deep Learning Software Reposi-
tories”. In: MSR. 2015.
[48] E. Kalliamvakou, L. Singer, G. Gousios, D. M. German,
K. Blincoe, and D. Damian. “The Promises and Perils
of Mining GitHub”. In: MSR. 2009.
[49] Y . Wu, J. Kropczynski, P. C. Shih, and J. M. Car-
roll. “Exploring the Ecosystem of Software Develop-
ers on GitHub and Other Platforms”. In: CSCW. 2014,
pp. 265–268.
[50] E. Guzman, D. Azócar, and Y . Li. “Sentiment Analysis
of Commit Comments in GitHub: an Empirical Study”.
In: MSR. 2014, pp. 352–355.
[51] R. Baishakhi, D. Posnett, V . Filkov, and P. Devanbu.
“A Large Scale Study of Programming Languages and
Code Quality in Github”. In: FSE. 2014.
[52] B. Vasilescu, V . Filkov, and A. Serebrenik. “Percep-
tions of Diversity on GitHub: A User Survey”. In:
Chase (2015).
[53] X. Gu, H. Zhang, D. Zhang, and S. Kim. “Deep API
Learning”. In: arXiv:1605.08535 (2016).
[54] M. Allamanis, H. Peng, and C. Sutton. “A Convolu-
tional Attention Network for Extreme Summarization
of Source Code”. In: arXiv:1602.03001 (2016).
[55] E. Wong, J. Yang, and L. Tan. “AutoComment: Mining
Question and Answer Sites for Automatic Comment
Generation”. In: ASE. IEEE, 2013, pp. 562–567.
[56] V . Raychev, M. Vechev, and E. Yahav. “Code Com-
pletion with Statistical Language Models”. In: PLDI.
2014.
97
A. Artifact description
A.1 Abstract
Our research artifact consists of interactive Jupyter notebooks.
For your convenience, we provide two methods of validating
our results: an ‘AE’ notebook which validates the main exper-
iments of the paper, and a comprehensive ‘Paper’ notebook
which replicates every experiment of the paper, including
additional analysis. The most convenient method to evaluate
our results is to access our pre-conﬁgured live server:
http://[redacted]:8888/notebooks/AE.ipynb
using the password [redacted], and to follow the instruc-
tions contained within.
A.2 Description
A.2.1 Check-list (Artifact Meta Information)
• Run-time environment: A web browser.
• Output: OpenCL code, runtimes, ﬁgures and tables from the
paper.
• Experiment workﬂow: Run (or install locally) Jupyter note-
books; interact with and observe results.
• Experiment customization: Edit code in Jupyter notebook; full
API and CLI for CLgen.
• Publicly available?: Yes, code and data. See:
http://chriscummins.cc/cgo17/
A.2.2 How Delivered
Jupyter notebooks which contain an annotated version of this
paper, interleaved with the code necessary to replicate results.
We provide three options to run the Jupyter notebooks:
1. Remote access to the notebook running on our pre-
conﬁgured experimental platform.
2. Download our pre-packaged VirtualBox image with
Jupyter notebook installed.
3. Install the project locally on your own machine.
A.3 Installation
Access the Jupyter notebooks using one of the three methods
we provide. Once accessed, proceed to Section A.4.
A.3.1 Remote Access
The Jupyter notebooks are available at:
http://[redacted]:8888, password [redacted].
A dashboard showing server load is available at:
http://[redacted]:19999
High system load may lead to inconsistent performance
results; this may occur if multiple reviewers are accessing the
server simultaneously.
A.3.2 Virtual Machine
Copy our pre-conﬁgured 5.21 GB VirtualBox image using:
$ scp cgo@[redacted]:vm.ova ~
Password: [redacted]
Install the virtual machine using VirtualBox’s “Import Appli-
ance” command:
The image was prepared using VirtualBox 5.1.8. It has the
following conﬁguration: Ubuntu 16.04, 4 GB RAM, 10 GB
hard drive, bridged network adapter with DHCP, US keyboard
layout, GMT timezone.
Start the machine and log in using username and password
cgo. Once at the shell, runlaunch. This will start the Jupyter
notebook server and print its address. You can access the
notebooks at this address using the browser of the host device.
Please note that the VirtualBox image does not have OpenCL,
so new runtimes cannot be generated.
A.3.3 Local Install
See http://chriscummins.cc/cgo17/ for instructions.
Note that we only support Ubuntu 16.04 or OS X, and sudo
privileges are required to install the necessary requirements.
Other Linux distributions may work but will require extra
steps to install the correct package versions.
98
A.4 Experiment Workﬂow
1. Access the Jupyter notebook server using one of the three
options described in Section A.3.
2. From the Jupyter server page, tick the checkbox next to
one of the two notebooks: AE.ipynb for minimal arti-
fact reproduction or Paper.ipynb for a comprehensive
interactive paper.
3. Click the button “Duplicate”.
4. Click on the name of the newly created copy, e.g.
Paper-Copy1.ipynb or AE-Copy3.ipynb.
5. Repeatedly press theplay button (tooltip is “run cell, select
below”) to step through each cell of the notebook.
OR select “Kernel” > “Restart & Run All” from the menu
to run all of the cells in order.
A.5 Evaluation and Expected Result
Each code cell within the Jupyter notebook generates an
output. Expected results are described in text cells.
We include both the code necessary to evaluate the data
used in the paper, and the code necessary to generate and
evaluate new data. For example, we include the large neural
network trained on all of the OpenCL on GitHub (which took
3 weeks to train), along with a small dataset to train a new
one.
A.6 Experiment Customization
The experiments are fully customizable. The Jupyter note-
book can be edited “on the ﬂy”. Simply type your changes
into the cells and re-run them. For example, in Table 1 of the
Paper.ipyn notebook we cross-validate the performance of
predictive models on an AMD GPU:
To replicate this experiment using the NVIDIA GPU, change
the ﬁrst line of the appropriate code cell to read data =
nvidia_benchmarks and re-run the cell:
Note that some of the cells depend on the values of prior cells
and must be executed in sequence.
CLgen has a documented API and command line interface.
You can create new corpuses, train new networks, sample
kernels, etc.
A.7 Notes
For more information about CLgen, visit:
http://chriscummins.cc/clgen
For more information about Artifact Evaluation, visit:
http://ctuning.org/ae/submission-20161020.html
99

A DEEP LEARNING BASED COST MODEL FOR AUTOMATIC CODE
OPTIMIZATION
Riyadh Baghdadi1 2 Massinissa Merouani3 Mohamed-Hicham Leghettas3 Kamel Abdous3 Taha Arbaoui4
Karima Benatchba3 Saman Amarasinghe1
ABSTRACT
Enabling compilers to automatically optimize code has been a longstanding goal for the compiler community.
Efﬁciently solving this problem requires using precise cost models. These models predict whether applying a
sequence of code transformations reduces the execution time of the program. Building an analytical cost model
to do so is hard in modern x86 architectures due to the complexity of the microarchitecture. In this paper, we
present a novel deep learning based cost model for automatic code optimization. This model was integrated in a
search method and implemented in the TIRAMISU compiler to select the best code transformations. The input
of the proposed model is a set of simple features representing the unoptimized code and a sequence of code
transformations. The model predicts the speedup expected when the code transformations are applied. Unlike
previous models, the proposed one works on full programs and does not rely on any heavy feature engineering.
The proposed model has only 16% of mean absolute percentage error in predicting speedups on full programs.
The proposed model enables TIRAMISU to automatically ﬁnd code transformations that match or are better
than state-of-the-art compilers without requiring the same level of heavy feature engineering required by those
compilers.
1 I NTRODUCTION
Writing high-performance software is essential in many ar-
eas from machine learning to science and engineering. In
nuclear physics, for example, researchers need to perform
large scale simulations to study the properties of matter.
A highly optimized implementation of these simulations
can be orders of magnitude faster compared to an unop-
timized implementation. In deep learning, an optimized
implementation of a state-of-the-art neural network such as
XLNet (Yang et al., 2019) is 1.8× faster than the equivalent
PyTorch implementation. Writing such a highly optimized
code requires ninja programmers and is time-consuming
while the results are error-prone, less understandable, and
non-portable. One of the longstanding goals in the compiler
community is to develop compilers that can automatically
optimize high-level code. These compilers automatically
apply code transformations to make the code run faster;
thus, avoiding the need for manual low-level program tun-
ing. They provide greater productivity, portability, and high
performance, and will be directly accessible by domain
scientists.
1Massachusetts Institute of Technology 2New York Univer-
sity Abu Dhabi 3Ecole Nationale Superieure d’Informatique
4University of Technology of Troyes. Correspondence to: Riyadh
Baghdadi <baghdadi@nyu.edu>.
Proceedings of the4 th MLSys Conference, San Jose, CA, USA,
2021. Copyright 2021 by the author(s).
Automatically generating efﬁcient code for high-
performance systems is a tedious task. In order for the
compiler to generate efﬁcient code, two problems have to
be solved. First, a large set of code transformations and a
mechanism to apply them to programs need to be provided.
Examples of such transformations include loop ﬁssion,
fusion, parallelization, and vectorization. Second, the right
sequence of code transformations from this large set has to
be chosen. The selected code transformations must preserve
the program semantics and provide the highest performance
for the input program. While state-of-the-art-compilers
have shown success in solving the ﬁrst problem (i.e., the
ability to provide a large set of transformations and correctly
apply a selected sequence of transformations (Wolf & Lam,
1991; Bondhugula et al., 2008; Trifunovic et al., 2010;
Grosser et al., 2014; Lefebvre & Feautrier, 1998; Quiller´e
& Rajopadhye, 2000)), they still do not successfully
solve the second problem (i.e., selecting the sequence of
transformations that will provide the best performance).
The problem of selecting the right sequence of code trans-
formations can be modeled as a search problem that can
be solved in three steps. In the ﬁrst step, the compiler uses
a search technique to explore the space of possible code
transformations. The result of this step is a set of candidates
where each one is a sequence of code transformations. In
the second step, the compiler checks the validity of each
candidate (i.e., checks that applying the transformations
A Deep Learning Based Cost Model for Automatic Code Optimization
does not change the program semantics). In the third step,
the compiler evaluates the valid candidates and chooses the
one that minimizes the execution time. This evaluation can
be done by running each candidate on the target hardware
to obtain the exact speedup. However, this is not a feasible
solution in practice as running a program takes a consider-
able amount of time. Moreover, the target hardware may
not be available at compile time. Another way to evaluate a
candidate is by using a cost model to predict the speedup.
Designing cost models manually is known to be a hard
task (Trifunovic et al., 2009; Bachir et al., 2013). This is
mainly due to the diversity of hardware architectures and
their complexity (out-of-order execution, complex memory
hierarchies, data prefetching, etc.). Complex interactions be-
tween code transformations make the problem more compli-
cated. Recently, cost models, such as Ithemal (Mendis et al.,
2018) and Halide (Adams et al., 2019), have demonstrated
how to overcome some of this complexity by using deep
learning. While these state-of-the-art cost models are more
accurate, they are limited in two ways: Ithemal (Mendis
et al., 2018) only predicts throughput for basic blocks of as-
sembly code (instead of full programs). It also assumes that
data is always in cache. The cost model in Halide (Adams
et al., 2019) requires heavy feature engineering (it uses 54
complex program features). Designing such features is te-
dious, error-prone, and time-consuming.
In this paper, we propose a novel DNN-based cost model
that avoids the problems of previous work. Our model op-
erates on full programs expressed in a high-level language
(not just basic blocks). It takes into consideration not only
memory accesses to the cache but also to the main memory.
Moreover, it does not require heavy feature engineering.
The proposed cost model takes the original unoptimized
code and a sequence of code transformations and predicts
the speedup that these transformations would yield when
applied. The model is designed for CPUs and is integrated
in the TIRAMISU compiler (Baghdadi et al., 2019), a com-
piler for the TIRAMISU domain-speciﬁc language (DSL).
Because this model is a regression model, it allows the com-
piler to select the best transformation candidates by ranking
the candidates selected by a search technique.
Contributions In summary, the contributions of this pa-
per are:
• A novel deep-learning-based cost model for code opti-
mization. This cost model is a regression cost model,
operates on full programs, and does not rely on extract-
ing complex features.
• An implementation of the proposed model and an inte-
gration into a search approach to enable the TIRAMISU
compiler to automatically search for the best code trans-
formations.
• We evaluate the proposed model and show that it has
a low error rate reaching 16% mean absolute percent-
age error. We show also that it enables TIRAMISU to
automatically ﬁnd code transformations that match or
outperform state-of-the-art compilers.
2 T IRAMISU EMBEDDED DSL
TIRAMISU (Baghdadi et al., 2019) is a domain-speciﬁc lan-
guage (DSL) embedded in C++. It provides a C++ API that
allows users to write a high level, architecture-independent
algorithm, and a set of API calls to select which code trans-
formations should be applied. The ﬁrst part of a TIRAMISU
program speciﬁes the algorithm without specifying how it
should be optimized. The second part speciﬁes which code
transformations to apply and how the results of computa-
tions should be stored. TIRAMISU uses a mathematical
model known as the polyhedral model internally (Feautrier,
1988; Baghdadi et al., 2015a; Bondhugula et al., 2008; Bagh-
dadi et al., 2015b; 2019) to represent code, code transfor-
mations, and to reason about the correctness of code trans-
formations. The following code shows an example of a
convolution algorithm written in TIRAMISU .
1 // Declare the iterators.
2 var n(0, batch), fout(0, out_features), fin
(0, in_features), y(0, H-2), x(0, W-2),
k0(0, 3), k1(0, 3);
3 // Algorithm.
4 conv(n, fout, y, x) += weights(fout, fin, y
, x) * input(n, fin, y + k0, x + k1);
The iterators in line 2 deﬁne the loop bounds around the
conv computation. The algorithm is semantically equiva-
lent to the following code.
1 for (n in 0..batch)
2 for (fout in 0..out_features)
3 for (y in 0..H-2)
4 for (x in 0..W-2)
5 for (fin in 0..in_features)
6 for (k0 in 0..3)
7 for (k1 in 0..3)
8 conv[n, fout, y, x] += weigths[fout,
fin, y, x] * input[n, fin, y+k0, x+k1];
The next code shows an example of code transformation
commands that can be applied to the previous convolution
kernel. These commands apply parallelization, loop inter-
change, tiling, vectorization, and unrolling.
1 // Provide the code transformation commands.
2 conv.parallelize(n);
3 conv.interchange(fout, fin);
4 conv.tile(y, x, 32, 32);
5 conv.vectorize(fout, 8);
6 conv.unroll(k0); conv.unroll(k1);
Currently, in TIRAMISU , a developer has to provide the
previous sequence of code transformations manually. Our
A Deep Learning Based Cost Model for Automatic Code Optimization
goal is to automate ﬁnding that sequence. We do this by
developing a cost model that predicts the speedup of using
a given transformation or any sequence of valid transfor-
mations. For example, the model can be used to predict
whether combining parallelization, loop interchange, and
loop tiling is useful. In addition, the model can be used
to choose the right arguments for each one of the previous
code transformations (e.g., choose the tile sizes).
3 D ATA GENERATION
As training DNNs requires a large data set and only a small
number of programs have ever been written in TIRAMISU ,
we decided to automatically generate a data set and use it
to train the model. We developed a code generator that
generates random programs and sequences of code transfor-
mations. Each one of these randomly generated programs
and code transformations is compiled, executed, and ﬁnally,
the actual speedup is measured. The speedup is the ratio
between the execution time of the original unoptimized pro-
gram and the optimized one. Each data point in the data
set is a triplet of the form (program, a sequence of code
transformations, measured speedup).
Random Code Generation A TIRAMISU program is a
sequence of computations where each computation is an as-
signment. There are three common patterns of assignments
that appear in TIRAMISU programs: (1) simple assignments
where the right-hand side is a function of input arrays or
array values computed previously; (2) stencils; (3) reduc-
tions. The random code generator generates sequences of
computations where each computation is a variant (or a
combination) of the previous patterns. Randomly generated
programs are correct by construction. A computation con-
sumes either constants, input arrays, or values computed
by previous computations. Code transformations are also
generated randomly but speciﬁc rules are used to guarantee
that code transformations are valid (for example, tiling is
not applied if the loop extent is smaller than the tile size).
The random code generator is designed to generate programs
that are representative of real programs. The three patterns
that the random code generator generates, when combined
together, cover all the patterns that we are interested in
supporting. The input data is also generated automatically
and the size of the input data is chosen randomly. Since
the generated programs are not data dependent (no data
dependent conditional or data dependent array access), the
actual data values are not important. Only the size of the
data is important for training the model.
For a given randomly generated program, the random code
generator chooses random data sizes, and then generates 32
random sequences of code transformations.
Dataset Construction The total generated dataset has ap-
proximately 1.8 million programs. To construct this dataset,
we generated 56250 random algorithms. For each algorithm,
we generated 32 random sequences of code transformations.
Therefore we obtained 56250 × 32 programs in total. We
followed the gold-standard in performance engineering and
executed each resulting program 30 times, and retained the
median value of the execution times in order to reduce the
impact of minor variance in execution times. Since data
generation is time consuming, we used a cluster of 16 nodes
of multicore CPUs to accelerate data generation. Generating
the whole data set took 3 weeks.
4 P ROGRAM CHARACTERIZATION AND
MODEL ARCHITECTURES
Our cost model is designed to support programs that can be
expressed in TIRAMISU . The latter is designed for express-
ing data parallel algorithms that operate over dense arrays
using loop nests and sequences of statements. These algo-
rithms are often found in image processing, deep learning,
dense linear algebra, tensor operations, and stencil com-
putations. A formal description of programs supported by
TIRAMISU can be found in (Baghdadi et al., 2019; 2020).
Code transformations supported by the proposed model in-
clude loop fusion, loop tiling, loop interchange, and loop
unrolling which are all challenging. For simpler transfor-
mations such as parallelization and vectorization, we use
simple heuristics similar to those used by the Halide au-
toscheduler (Ragan-Kelley et al., 2012). These heuristics
mainly parallelize the outermost loops and vectorize the
innermost loops when a set of conditions are met.
4.1 Program Characterization
Designing complex hand-engineered features is tedious,
error-prone, and time-consuming. Instead of using com-
plex hand-engineered features, we characterize programs by
extracting simple high-level information that is stored in a
compact variable-size representation.
Our program characterization is based on the AST (Abstract
Syntax Tree) representation of programs. A program is
characterized as an ordered tree of computation vectors
as shown in Figure 1b. A computation vectoris a vector
that includes three pieces of information: (1) loop nest
representation; (2) assignments representation; (3) loop
transformation representation. In the following paragraphs,
we describe each of these components, the key features we
aim to encode by each one, and how we combine them in a
compact way.
Loop Nest Representation The extent of each loop level
around the computation is stored in the computation vector
(the extent of a loop is calculated based on its lower and up-
per bounds). An example is shown in Figure 1c. After each
loop level extent, for each loop transformation, we insert a
boolean tag that represents whether that transformation is
A Deep Learning Based Cost Model for Automatic Code Optimization
for	i:	
		for	j:
				for	k:
						computation	A					
				for	l:
						computation	B				
				for	m:
						for	n:
								computation	C
								computation	D
						computation	E
Loop i
j
k l m
n
Computation A B
C D
E
Computation Vector
E
Loop Nest VectorAssignment Vector
Loop i
Loop j
Loop m
Transformations applied on loop i
Transformations applied on loop j
Transformations applied on loop m
Memory access 1
Memory access 2
Memory access n
Operations count
Memory access 3
(a) Program pseudocode. (b) Program tree representation. (c) Computation vector.
Figure 1: Our characterization of a typical program.
applied to that particular loop level. Each transformation is
followed by its parameters (when this applies).
Assignments Representation We represent both the left-
hand side and the right-hand side of the assignment. To
represent the left-hand side, we store the dimensions and the
size of each dimension of the buffer used on the left-hand
side. To represent the right-hand side of the assignment (the
assignment expression), we store the following information:
(1) the memory access pattern (access matrix described
later); (2) the ID of each accessed buffer (a number); (3) the
count of each arithmetic operation used on the right-hand
side (i.e., the number of times each arithmetic operation is
used).
We represent the array accesses using an access matrix that
stores the coefﬁcients of each array access. This matrix uses
exactly the same format used in the polyhedral model to
represent array accesses (Paul & Christian, 2011). It only
supports arrays that have afﬁne array accesses (i.e., array
accesses that are afﬁne in the loop iterators). Supporting
only afﬁne accesses is not a problem since TIRAMISU only
supports code that has afﬁne accesses. Supporting code that
has data-dependent array accesses or non-afﬁne accesses is
not within the scope of this paper.
The access matrix has k rows and n + 1columns where
k is the number of dimensions of the access buffer and n
is the loop depth. Each row in the matrix represents an
array dimension. Each array dimension is considered to be
a linear combination of the loop iterators. Each loop iterator
in the matrix is represented by a column. The coefﬁcient of
each loop iterator is stored in the column that corresponds to
that loop iterator. The last column in the matrix corresponds
to constants.
M =


1 0 0
1 1 0
0 1 −2


For example, the following memory access A[i0, i0 +
i1, i1 − 2] is represented using the matrix M. In this case,
the ﬁrst dimension of the buffer access is i0. It can also be
written as 1∗i0 +0 ∗i1 +0. It is represented by the ﬁrst row
in the matrix using 1 0 0where the ﬁrst column corresponds
to iterator i0, the second column corresponds to i1 and the
last column corresponds to constants. The second access
i0 +i1 is represented with the second row in the matrix1 1 0.
Each memory access matrixis succeeded by the identiﬁer
of the buffer.
Loop Transformation Representation Each loop trans-
formation can be characterized by two pieces of information:
transformation type and its parameters. Since transforma-
tions are applied on loop levels, we attach to the representa-
tion of each loop level the transformations that are applied
to that level (Figure 1c). The transformations that involve
changing the structure of the program (e.g. loop fusion) are
directly applied to the program structure representationthat
we will describe next.
Program Structure Representation The program is rep-
resented as a tree structure where leaves are thecomputation
vectors and internal nodes are the loop levels, as shown in
Figure 1b.
4.2 Detailed List of Features Composing the
Computation Vector
Table 1 details the full list of features that constitute the
Computation Vectorpresented in Figure 1c. All features
are integer values except Tag features which are boolean
values. In practice, we set n = 7and m = 21where n is
the maximum length of the loop nests in our dataset and
m is the maximum number of terms used in an assignment.
When needed, a zero-padding is added to the Loop Nest
Vector and Assignment Vector.
4.3 Hardware Characterization
The goal of the paper is not to develop a hardware indepen-
dent model. The proposed model is speciﬁc to a particular
CPU. The user can build a model for each CPU they want to
target. Building a new model does not require a large effort.
It can be done simply by running a script that generates new
data and retrains the model.
A Deep Learning Based Cost Model for Automatic Code Optimization
Loop Nest Vector
Loop1
Upper bound, Lower bound, Reduction tag
Fusion tag, Interchange tag, Tilling tag, Tilling factor
Loop2
Upper bound, Lower bound, Reduction tag
Fusion tag, Interchange tag, Tilling tag, Tilling factor
...
...
Loopn
Upper bound, Lower bound, Reduction tag
Fusion tag, Interchange tag, Tilling tag, Tilling factor,
Unroll tag, Unrolling factor.
Assignment Vector
Memory access 1 Access matrix, Buffer ID.
Memory access 2 Access matrix, Buffer ID.
...
...
Memory access m Access matrix, Buffer ID.
Operations count Number of Additions, Number of Multiplications,
Number of Subtractions, Number of Divisions.
Table 1: A detailed listing of the features that compose the Computation Vector.
In other words, we do not include any feature to repre-
sent the hardware because the model is speciﬁc to one and
only one hardware architecture. We also do not extract any
hardware-speciﬁc feature from the code for the same rea-
son. Designing a model that works for multiple hardware
architectures is beyond the scope of this paper.
4.4 Model Architecture
We model the problem of speedup estimation as a regression
problem: given an algorithm and a set of code transforma-
tions, our model predicts the speedup expected when ap-
plying the suggested code transformations compared to the
base program (i.e. without applying code transformations).
We design our cost model’s architecture to support the vari-
able size and recursive nature of our program characteri-
zation by combining Recurrent and Recursive Neural Net-
works. Our model’s architecture has three layers as shown
in Figure 2a.
Computation Embedding Layer All computation vec-
tors of the program are processed through a feedforward
network after log-transforming non-boolean features. This
log-transformation is necessary since these features have a
large dynamic range and most of them are expected to be
multiplied with other features to compute the ﬁnal speedup.
Recursive Loop Embedding Layer The computation
embeddings resulting from the previous layer are then pro-
cessed recursively following the tree structure of the pro-
gram using the loop embedding unit. At a given loop level,
the loop embedding unit summarizes the program up to that
loop into a loop embedding. The loop embedding unit, as
depicted in Figure 2b, is composed of two separate LSTM
cells (Hochreiter & Schmidhuber, 1997) and a feedforward
layer. At each loop level, the ﬁrst LSTM is fed with the
embedding vectors of computations that are nested directly
in that loop level while the second LSTM is fed with the
embedding vectors of the previous loop levels that resulted
from the previous loop embedding units. The two hidden
states of the LSTMs are merged using a feedforward layer
into a loop embedding vector.
The purpose of this recursive embedding is to selectively in-
corporate information from each computation respecting its
positional relations with the other computations. The output
of this layer, the program embedding vector, is assumed to
contain the needed set of automatically extracted features
covering the complete program.
Regression Layer The program embedding vectorpro-
duced by the previous layer is ﬁnally fed to a shallow feed-
forward neural network that performs a regression in order
to predict the speedup.
We choose MAPE (Mean Absolute Percentage Error) as
an objective function to train our model. The model is im-
plemented in PyTorch (Paszke et al., 2019) and optimized
using AdamW (Loshchilov & Hutter, 2017). All the imple-
mentation details including layer sizes and training policy
can be found in appendix A.1.
Other Neural Network Models Explored We also ex-
plored many other alternative architectures for the cost-
model, the architecture presented above has the lowest
MAPE error on both the test set and benchmarks set. For
instance, replacing the Recursive loop embedding layerwith
a simple Recurrent Neural Network that is directly fed with
the sequence of computation embeddingswithout taking in
consideration the loops hierarchy leads to a relative increase
of 1.15x in MAPE of the test set and 1.33x in the bench-
marks set compared to the presented architecture. Another
straightforward choice of using a simple Feedforwad Neural
Network, i.e. totally skipping the Recursive loop embedding
A Deep Learning Based Cost Model for Automatic Code Optimization
DC E
A B
C D
E
Computation embedding vector 
Feedforward NN 
Loop embedding unit
Loop embedding vector 
Feedforward NN 
Computation vector 
Recursive Loop
Embedding Layer Regression
Layer
Computation
Embedding Layer
A B
i
j
k l
m
n LSTM LSTM 
Child computation
embeddings 
Child loop
embeddings 
New loop
embedding 
ŷ Predicted Speedup
(a) Processing the program presented in Figure 1 through the three layers of the
cost-model.
(b) Loop embedding unit.
Figure 2: The cost model architecture
layer and feeding directly the concatenated computation em-
beddings to the regression layer, leads to a relative increase
of 1.39x in MAPE of the test set and 1.37x in the bench-
marks set compared to the presented model. In addition,
this alternative has the considerable limitation of not sup-
porting variable program sizes and supports only programs
that contain up to a certain number of computations (we
have set the maximum number of computations to 4 when
testing this alternative).
4.5 Level of Feature Extraction
One of the questions that we needed to answer is at which
level should the code representation be extracted: directly
from the source code or from the transformed code (interme-
diate representation obtained after applying code transfor-
mations)? Our choice was to extract the representation from
the TIRAMISU source code for a pragmatic reason. In order
for a model that takes transformed code to work, Tiramisu
will need to apply the transformations on the program before
using the model. Such a step is time-consuming especially
because it will be repeated a large number of times (given
that the search space is large). Furthermore, transformed
code is more complex and therefore is harder to learn from
compared to a program and a list of transformations.
5 S EARCH SPACE EXPLORATION
Finding the best code transformations is a hard combina-
torial optimization problem due to the fact that some of
the constraints (e.g., interaction between code transforma-
tions), and the objective (the speedup in this case), are hard
to represent mathematically using the program’s features.
Thus, the proposed model is used as an objective function
estimator to better navigate the search space. However, the
used search exploration approach should take into account
the estimator’s margin of error, thus requiring stochasticity
in the search space exploration.
Since the interaction between code transformations is hard
to characterize, one of the best ways to model the problem of
ﬁnding the best code transformations (and their parameters)
is to use a tree search. This allows us to use classical tree
search algorithms. In this paper, we use Beam Search and
MCTS (Monte Carlo Tree Search).
The Beam Search tree (as shown in Figure 3) explores
whether to apply a code transformation and which parame-
ters to use for that transformation. At each node of the tree,
an evaluation is conducted using the cost model to assess
whether the chosen transformations provide a good speedup.
In Figure 3, exploring the tree shows that applying tiling
with a tile size of (16, 8) and unrolling with a factor of 4
provides the best sequence of code transformations.
MCTS takes advantage of the search tree and takes into
account the stochasticity of the model. Particularly, the
proposed MCTS combines our model’s prediction and an
execution of the best evaluated code transformations as a
compromise between prediction and execution. MCTS ﬁrst
explores the different branches of the tree and selects a
number of promising code transformations. To this end,
the model is used in each node to obtain an estimate of the
speedup. MCTS keeps track of a set of the best evaluated
code transformations to execute them (the size of the set is a
parameter of the approach). Once the tree is explored, the set
of the best code transformations is executed. The advantage
of this two-step approach is to accelerate the exploration of
the search space using the model and to correct the model’s
error, if occurred, by executing a limited set of programs
and their code transformations. The proposed model is thus
A Deep Learning Based Cost Model for Automatic Code Optimization
C
yes no
C.tile()    C
tile?
Unroll?
C.tile(16,8).unroll() C.tile(16,8)
yes no
C.tile(16,8).unroll(2) C.tile(16,8).unroll(4)
2 4
C.tile(8,8) C.tile(16,8) C.tile(32,8)
8,8 16,8 32,8
Parameters?
Explore: tiling, unrolling
Figure 3: Example of the BS Tree for exploring the tiling and
unrolling code transformations
used to prune the search space and limit the execution to
selected code transformations.
6 E VALUATION
To evaluate our cost model: (1) we measure its accuracy on
a test set composed of random programs and compare the
predicted and the measured speedups on that data set; (2) we
measure the speedups obtained when the model is used to
search for code transformations in real-world benchmarks;
(3) we compare the accuracy of this model with the accuracy
of the model used in Halide (Ragan-Kelley et al., 2012), a
state-of-the-art model.
The model evaluation and the data collection are performed
on 16 identical multi-core CPU nodes. Each node has a
dual-socket, each socket is a 12-core Intel Xeon E5-2680v3
CPU, with 128 GB RAM. We used 60% of data for training,
20% for validation, and 20% for testing.
MAPE (y, ˆy) = 1
n
n∑
i=1
⏐⏐⏐yi − ˆyi
yi
⏐⏐⏐
Model Accuracy To measure the accuracy of the pro-
posed model, we use MAPE (Mean Absolute Percentage
Error), where y and ˆy are respectively the measured and the
predicted speedups. The MAPE of our cost model on the
test set is 16%.
The Pearson correlation coefﬁcient for the proposed model
is 0.90, showing that the linear correlation between pre-
dicted and measured speedups is strong. In addition, we
evaluate the ranking capabilities of the model with the Spear-
man’s rank correlation coefﬁcient, deﬁned as: rs(y, ˆy) =
r
(
rg(y), rg(ˆy)
)
where rg(y) converts the speedups to ranks
and r is the Pearson correlation coefﬁcient. The Spearman’s
rank coefﬁcient of our cost model is 0.95, which shows
that the predicted and measured ranks are highly linearly
correlated. This property is important when using the model
with a search method.
Comparing Predicted and Measured Speedups Fig-
ure 4 compares the predicted and measured speedups. To
simplify visualization, we use a subset of the test set. This
subset is composed of 100 random programs, each with 32
random sequences of code transformations (therefore, the
total is 3200 transformed programs). The horizontal axis
is the list of 3200 programs. These programs are sorted
based on their speedups in ascending order to simplify vi-
sualization. As the ﬁgure shows, the predicted speedups
are close to the measured ones. The error in prediction is
lower around the speedup 1 and is higher as the speedup
gets further from 1. We will comment more on this behavior
later in the section.
Figure 5 investigates the distribution of the model error rates
over the whole test set. On top, Absolute Percentage Error
(APE) is measured on the code transformations of each
program and the results are plotted through a histogram. On
bottom, APE is measured on all data points of the test set
and the measured speedups are plotted against their APE.
We can see that the error gets smaller as speedups approach
1 and gets higher as speedups get far from 1. Particularly,
the error is more signiﬁcant for speedups below 0.05. The
model is more accurate around speedup 1 because most
programs in the training data set have speedups close to 1.
Speedups below 0.05 are less frequent. The next experiment
will evaluate whether the accuracy of the model allows
ﬁnding the best code transformations when searching the
space.
Search Space Exploration Using the Cost Model In
this experiment, we evaluate the ability of search approach
combined with the cost model to ﬁnd good code transfor-
mation sequences for real-world benchmarks. We use BS
and MCTS to explore the search space. We use a set of
real-world benchmarks spanning different areas: image pro-
cessing, deep learning, linear algebra and stencils. The
benchmarks include box blur (an image processing ﬁlter
to blur images), conv + relu(two successive neural net-
work layers that beneﬁt from operator fusion), convolution
(a direct neural network convolution), cvtcolor (an image
processing ﬁlter for converting the colors of an input image
from RGB to gray), doitgen (a kernel from the multiresolu-
tion adaptive numerical scientiﬁc simulation (Louis-Noel,
2010)), heat2d (heat equation over 2D space), heat3d (heat
equation over 3D space), jacobi2d (a jacobi-style stencil
computation over 2D data with 5-point stencil pattern), mvt
(matrix vector multiplication composed with another matrix
vector multiplication but with transposed matrix), and sei-
del2d (Gauss-Seidel style stencil computation over 2D data
with 9-point stencil pattern). The sizes of the input data for
each benchmark is provided in the appendix.
Figure 6 shows the best speedups found for each benchmark.
The baseline is the original program where the outermost
A Deep Learning Based Cost Model for Automatic Code Optimization
Transformed programs ordered by their speedups
Speedup
0.005
0.01
0.05
0.1
0.5
1
5
10
50
100
Predicted speedup Measured speedup
Figure 4: Predicted speedups compared to measured speedups. The speedups are ordered in ascending order.
Absolute percentage error in predicted speedup
Number of programs0
10000
20000
30000
40000
0.00
0.06
0.12
0.18
0.24
0.30
0.36
0.42
0.48
0.54
0.60
0.66
0.72
0.78
0.84
0.90
0.96
Measured speedup
Absolute percentage error0%
10%
20%
30%
40%
0.05 0.1 0.5 1 5 10
Figure 5: The distribution of error rates for the whole test set.
On top, APE is measured for each transformed program, then
the histogram of measurements is plotted. On bottom, APE is
measured for each transformed program, then the speedups are
plotted with their APE.
loop is parallelized (no other code transformation is applied).
The ﬁrst column (blue), reports results obtained when beam
search is used to explore the search space. This column is
considered the reference in our comparison as execution
is used to obtain the speedups. In the second and third
columns, beam search and MCTS use the cost model to
predict speedups. The last column shows the speedups
Speedup
0
1
2
3
4
5
6
7
8
box blur
conv + reluconvolution
cvtcolordoitgenheat2dheat3djacobi2d
mvt
seidel2d
Beam search with execution Beam search with the cost model
MCTS with the cost model Halide Autoscheduler
Figure 6: Speedups for different benchmarks obtained by exploring
the search space.
obtained after applying the Halide autoscheduler (Halide
automatic optimizer) deﬁned in (Adams et al., 2019).
Beam search (BS) with the cost model is competitive in
most benchmarks, but does not ﬁnd the best code transfor-
mations in heat2d, jacobi2d and seidel2d. Beam search with
the cost model relies entirely on predictions to make deci-
sions. Bad predictions can thus mislead the search method
which is why beam search does not ﬁnd the best transfor-
mations in the previous benchmarks. MCTS has similar
performance, except in jacobi2d and seidel2d where it ﬁnds
better code transformations, and in cvtcolor where the code
transformations found are less good. MCTS can ﬁnd bet-
ter code transformations in these cases because it copes
with model imprecision taking into account its stochasticity.
However, since the tree space is explored differently, MCTS
might explore different nodes compared to BS and thus have
distinguishable results.
A Deep Learning Based Cost Model for Automatic Code Optimization
Comparison with Halide In this section, we compare
our cost model with the one of Halide (Adams et al., 2019),
a state-of-the-art cost model and the closest to ours. In
comparison with Halide, TIRAMISU ﬁnds transformation
sequences that are either competitive with those found by
Halide or better (except in box blur). This is mainly due
to miss predictions by the Halide model which lead Halide
to use transformations that degrade performance. These
wrong predictions happen in particular in benchmarks that
are from the area of scientiﬁc computing which Halide was
not trained to handle (heat2d, jacobi2d, mvt and seidel2d).
In benchmarks that fall in the categories of deep learning and
image processing, which Halide supports well, TIRAMISU
and Halide have comparable performance.
We also compare the performance of the Halide model
with that of TIRAMISU on randomly generated programs.
Halide’s paper usesR2 as an accuracy metric and uses MSE
(Mean Square Error) as a loss function, we thus use the same
metric and loss function for comparison. Halide has an R2
of 0.96, whereas TIRAMISU has 0.89. Both Halide and
TIRAMISU have comparable results but Halide uses heavy
feature engineering. The main advantage of TIRAMISU is
that it does not require feature engineering.
Tradeoff Between Search Time and Quality of Code
Transformations It is crucial to note that the execution
time of a search method is dependent on the time to evaluate
code transformations. If one compiles and executes every
transformed program to assess the speedup of code trans-
formations, the search time would increase considerably
and therefore becomes impractical. The proposed model
accelerates the search space exploration, reducing thus the
time needed to ﬁnd the best code transformations.
Table 2 illustrates the tradeoff that we make between the
time needed to explore the search space, and the perfor-
mance of the ﬁnal code transformations found. It compares
the gains from reducing the search time and performance
degradation due to the use of a model. To be speciﬁc, for
each benchmark: (1) we compare the time necessary to ex-
plore the search space using Beam Search with Execution
(BSE) and using either of Beam Search with the Cost Model
(BSM) on the left table, or MCTS on the right table. Com-
paring these two shows how faster BSM and MCTS are with
regard to BSE. (2) we compare the performance of the ﬁnal
code transformations returned by BSE and the ones returned
by BSM or MCTS.
The second column of the table represents the speedup in
search time (the time taken by BSE over the time taken
by BSM or MCTS). The third column is the degradation in
performance (execution time) of the code transformations
found by BSM or MCTS compared to the code transforma-
tions found by BSE. We can see that, on average, searching
with BSM is 106.5 times faster than searching with BSE,
while incurring an average decrease of 15% in the perfor-
mance of the code transformations found. Moreover, search-
ing with MCTS is 11.8 faster on average, with a loss of
12.5% in performance.
Note that for jacobi2d in beam search, and cvtcolor in
MCTS, the performance degradation is important. This
is mainly due to the imprecision of the model for these
benchmarks. The model predicts some bad code transfor-
mations as being good. This is mainly because a pattern
that appears in those benchmarks is not covered enough by
the training data. A solution to this problem would be to
generate more data that include the missing pattern.
Pearson correlation
0.00
0.25
0.50
0.75
1.00
Spearman's rank correlation
0.00
0.25
0.50
0.75
1.00
Figure 7: Correlation between predicted and measured speedups
for the data points in thesmall test set. Pearson correlation is on the
left and Spearman’s correlation is on the right. Each column repre-
sents the coefﬁcient measured on 32 random code transformations
for a single program.
More Detailed Evaluation Figure 7 gives more insights
on the behavior of our model. The Pearson and Spearman’s
coefﬁcients are measured on a subset of the test set between
the code transformations of each program. To simplify visu-
alization, we use the same subset of transformations as the
experiment of Figure 4. In most cases, the coefﬁcients are
close to 1, highlighting a strong linear correlation between
the predicted and measured speedups (Pearson correlation),
and between the predicted and measured ranks (Spearman’s
correlation). Particularly, the correlation between ranks is of
most importance for search methods as these methods rely
on the ranks of the explored points instead of their actual
evaluations.
7 R ELATED WORK
Several machine learning based cost models have been
developed for automatic code optimization. MILEPOST
GCC (Fursin et al., 2008) uses a 1-nearest-neighbor model
that takes manually engineered features and predicts the best
combinations of compiler ﬂags for GCC (GNU Compiler
Collection). Ithemal (Mendis et al., 2018) uses an LSTM
based model to predict the throughput of basic blocks of the
control-ﬂow graph (assembly-level code). Both of MILE-
POST GCC and Ithemal do not support high-level loop
transformations though. Modeling loop transformations is
in particular challenging as it requires the ability to model
the loop structure. This implies the ability to model cycles
in the control-ﬂow graph of the program, which is not trivial.
In addition, Ithemal does not model memory accesses to
A Deep Learning Based Cost Model for Automatic Code Optimization
Benchmark
Search time
improvement
(speedup)
Performance
degradation
for code
transforma-
tions
box blur 65x 0 %
conv + relu 12x 0 %
convolution 9x 0 %
cvtcolor 65x 7 %
doitgen 114x 7 %
heat2d 117x 18 %
heat3d 351x 13 %
jacobi2d 122x 65 %
mvt 83x 12 %
seidel2d 127x 28 %
Average 106.5x 15 %
Benchmark
Search time
improvement
(speedup)
Performance
degradation
for code
transforma-
tions
box blur 19x 0 %
conv + relu 14x 0 %
convolution 11x 0 %
cvtcolor 3x 41 %
doitgen 12x 5 %
heat2d 17x 17 %
heat3d 28x 9 %
jacobi2d 4x 26 %
mvt 5x 9 %
seidel2d 5x 18 %
Average 11.8x 12.5 %
Table 2: Search time improvement compared to performance degradation. On the left, the results for beam search with the cost model are
shown, and on the right, the results for MCTS.
different memory hierarchies (it only models accesses to
cache). Unlike both systems, our proposed model supports
loop transformations and takes into consideration different
memory hierarchy levels.
Other related work includes a model proposed by Rahman
et al. (Rahman et al., 2010). This model uses a feedforward
neural network to predict the execution time of programs af-
ter applying the tiling code transformation. Another model
proposed by Magni et al. (Magni et al., 2014) uses a feed-
forward neural network in a cascade fashion to predict the
best thread-coarsening factor. These two methods rely on
hand-engineered features though. They are also limited to a
single code transformation.
Halide (Adams et al., 2019) proposes a more comprehensive
method to ﬁnd efﬁcient code transformations. It combines
beam search with a feedforward neural network that pre-
dicts the execution time of programs from a set of manually-
engineered features. It uses 54 heavily engineered features
to perform its predictions. To the best of our knowledge,
both of Tiramisu and Halide consider the same code trans-
formations in their search spaces. In the same context, Au-
toTVM (Chen et al., 2018) uses a deep learning model to
search for code transformation parameters (TVM compares
two models, a TreeGRU and a Gradient Boosted Tree). The
TVM models are used with simulated annealing to search
the space. The authors only demonstrate the search for trans-
formation parameters though (tile size, unrolling factor, ...).
They do not demonstrate search for loop transformations
since the transformations themselves are provided by the
user. In a different fashion, DeepTune (Cummins et al.,
2017) proposes a neural network consisting of embedding
layers and LSTM cells to predict whether an OpenCL kernel
should be mapped to CPU or GPU. DeepTune also proposes
another model to predict whether thread coarsening (a code
transformation for GPUs) should be used. The DeepTune
models are classiﬁcation models though. This means that
they can classify whether a given transformation is beneﬁ-
cial, but they are not designed to rank different sequences
of code transformations. Therefore, they are not ideal for
use when searching a large space of code transformations
where many sequences need to be ranked and the best one
selected.
The goal of this paper is not to propose a cost model for
general purpose compilers. The goal is also not to propose
a cost model for general purpose transformations. In a
way similar to related work, this paper mainly focuses on a
domain speciﬁc compiler and on loop transformations.
Many polyhedral compilers including Pluto (Bondhugula
et al., 2008), PENCIL (Baghdadi et al., 2015a; 2013a),
LLVM Poly (Grosser et al., 2012), and Tensor Compre-
hensions (Vasilache et al., 2018) formalize the problem of
automatic code optimization as an integer linear program
(ILP). The objective of this ILP is to minimize the distance
between producer and consumer statements. The resulting
problem can be solved exactly, but the implicit cost model
does not capture all the complexity of the hardware architec-
ture and transformation interactions (Baghdadi et al., 2019).
This leads to suboptimal solutions (Baghdadi et al., 2019;
2015a; 2013b). Making the objective function more compre-
hensive makes the problem non-linear and thus it becomes
intractable.
8 C ONCLUSION
This paper presents a novel cost model for predicting
speedups. This cost model is a regression cost model that
operates on full programsand does not rely on extracting
complex features. It is not limited to transformation param-
eters but also includes code transformations. We develop
a random code generator to generate the training data and
release the generator publicly. We evaluated the proposed
model and show that it had a low error rate of 16% MAPE.
We integrate this model in a search space method and show
that the integrated approach enables TIRAMISU to auto-
matically ﬁnd sequences of code transformations that are
competitive with state of the art compilers.
A Deep Learning Based Cost Model for Automatic Code Optimization
REFERENCES
Adams, A., Ma, K., Anderson, L., Baghdadi, R., Li, T.-
M., Gharbi, M., Steiner, B., Johnson, S., Fatahalian, K.,
Durand, F., and Ragan-Kelley, J. Learning to optimize
halide with tree search and random programs. ACM
Trans. Graph., 38(4):121:1–121:12, July 2019. ISSN
0730-0301. doi: 10.1145/3306346.3322967. URL http:
//doi.acm.org/10.1145/3306346.3322967.
Bachir, M., Brault, F., Gregg, D., Cohen, A., et al. Minimal
unroll factor for code generation of software pipelining.
International Journal of Parallel Programming, 41(1):
1–58, 2013.
Baghdadi, R., Cohen, A., Guelton, S., Verdoolaege, S., In-
oue, J., Grosser, T., Kouveli, G., Kravets, A., Lokhmotov,
A., Nugteren, C., Waters, F., and Donaldson, A. F. PEN-
CIL: towards a platform-neutral compute intermediate
language for dsls. CoRR, abs/1302.5586, 2013a. URL
http://arxiv.org/abs/1302.5586.
Baghdadi, R., Cohen, A., Verdoolaege, S., and Trifunovic,
K. Improved loop tiling based on the removal of spurious
false dependences. TACO, 9(4):52, 2013b.
Baghdadi, R., Beaugnon, U., Cohen, A., Grosser, T., Kruse,
M., Reddy, C., Verdoolaege, S., Betts, A., Donaldson,
A. F., Ketema, J., Absar, J., Haastregt, S. v., Kravets,
A., Lokhmotov, A., David, R., and Hajiyev, E. Pencil:
A platform-neutral compute intermediate language for
accelerator programming. In Proceedings of the 2015 In-
ternational Conference on Parallel Architecture and Com-
pilation (PACT), PACT ’15, pp. 138–149, Washington,
DC, USA, 2015a. IEEE Computer Society. ISBN 978-1-
4673-9524-3. doi: 10.1109/PACT.2015.17. URL http:
//dx.doi.org/10.1109/PACT.2015.17.
Baghdadi, R., Cohen, A., Grosser, T., Verdoolaege, S.,
Lokhmotov, A., Absar, J., van Haastregt, S., Kravets,
A., and Donaldson, A. F. PENCIL language speciﬁca-
tion. Research Rep. RR-8706, INRIA, 2015b. URL
https://hal.inria.fr/hal-01154812.
Baghdadi, R., Ray, J., Romdhane, M. B., Del Sozzo, E.,
Akkas, A., Zhang, Y ., Suriana, P., Kamil, S., and Amaras-
inghe, S. Tiramisu: A polyhedral compiler for express-
ing fast and portable code. In Proceedings of the 2019
IEEE/ACM International Symposium on Code Genera-
tion and Optimization, CGO 2019, pp. 193–205, Piscat-
away, NJ, USA, 2019. IEEE Press. ISBN 978-1-7281-
1436-1. URL http://dl.acm.org/citation.
cfm?id=3314872.3314896.
Baghdadi, R., Debbagh, A. N., Abdous, K., Benhamida,
F. Z., Renda, A., Frankle, J. E., Carbin, M., and Amaras-
inghe, S. Tiramisu: A polyhedral compiler for dense and
sparse deep learning, 2020.
Bondhugula, U., Hartono, A., Ramanujam, J., and Sadayap-
pan, P. A practical automatic polyhedral parallelizer and
locality optimizer. In PLDI, pp. 101–113, 2008.
Chen, T., Zheng, L., Yan, E., Jiang, Z., Moreau, T., Ceze, L.,
Guestrin, C., and Krishnamurthy, A. Learning to optimize
tensor programs. In Advances in Neural Information
Processing Systems, pp. 3389–3400, 2018.
Cummins, C., Petoumenos, P., Wang, Z., and Leather, H.
End-to-end deep learning of optimization heuristics. In
2017 26th International Conference on Parallel Architec-
tures and Compilation Techniques (PACT), pp. 219–232.
IEEE, 2017.
Feautrier, P. Array expansion. In Proceedings of the 2nd
international conference on Supercomputing, pp. 429–
441, St. Malo, France, 1988. ACM. ISBN 0-89791-272-1.
doi: 10.1145/55364.55406. URL http://portal.
acm.org/citation.cfm?id=55406.
Fursin, G., Miranda, C., Temam, O., Namolaru, M., Yom-
Tov, E., Zaks, A., Mendelson, B., Bonilla, E., Thomson,
J., Leather, H., Williams, C., O’Boyle, M., Barnard, P.,
Ashton, E., Courtois, E., and Bodin, F. MILEPOST
GCC: machine learning based research compiler. In GCC
Summit, Ottawa, Canada, June 2008. URL https://
hal.inria.fr/inria-00294704.
Glorot, X. and Bengio, Y . Understanding the difﬁculty
of training deep feedforward neural networks. Journal
of Machine Learning Research - Proceedings Track, 9:
249–256, 01 2010.
Grosser, T., Groslinger, A., and Lengauer, C. Polly - per-
forming polyhedral optimizations on a low-level interme-
diate representation. Parallel Processing Letters, 22(4),
2012. URL http://dblp.uni-trier.de/db/
journals/ppl/ppl22.html#GrosserGL12.
Grosser, T., Cohen, A., Holewinski, J., Sadayappan, P., and
Verdoolaege, S. Hybrid hexagonal/classical tiling for
gpus. In Proceedings of Annual IEEE/ACM International
Symposium on Code Generation and Optimization, CGO
’14, pp. 66:66–66:75, New York, NY , USA, 2014. ACM.
Hochreiter, S. and Schmidhuber, J. Long short-term mem-
ory. Neural Comput., 9(8):1735–1780, November 1997.
ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735.
URL https://doi.org/10.1162/neco.1997.
9.8.1735.
Lefebvre, V . and Feautrier, P. Automatic storage man-
agement for parallel programs. Parallel Computing,
24:649–671, 1998. ISSN 01678191. doi: 10.1016/
S0167-8191(98)00029-5.
A Deep Learning Based Cost Model for Automatic Code Optimization
Loshchilov, I. and Hutter, F. Fixing weight decay regular-
ization in adam. CoRR, abs/1711.05101, 2017. URL
http://arxiv.org/abs/1711.05101.
Louis-Noel, P. PolyBench suite. http://www.cse.ohio-
state.edu/˜pouchet/software/polybench/, 2010.
URL http://www.cse.ohio-state.edu/
˜pouchet/software/polybench/.
Magni, A., Dubach, C., and O’Boyle, M. Automatic
optimization of thread-coarsening for graphics proces-
sors. In Proceedings of the 23rd International Confer-
ence on Parallel Architectures and Compilation, PACT
’14, pp. 455–466, New York, NY , USA, 2014. Associa-
tion for Computing Machinery. ISBN 9781450328098.
doi: 10.1145/2628071.2628087. URL https://doi.
org/10.1145/2628071.2628087.
Mendis, C., Amarasinghe, S. P., and Carbin, M. Ithe-
mal: Accurate, portable and fast basic block through-
put estimation using deep neural networks. CoRR,
abs/1808.07412, 2018. URL http://arxiv.org/
abs/1808.07412.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J.,
Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga,
L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Rai-
son, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang,
L., Bai, J., and Chintala, S. Pytorch: An imperative
style, high-performance deep learning library. In Wal-
lach, H., Larochelle, H., Beygelzimer, A., d’Alch´e Buc,
F., Fox, E., and Garnett, R. (eds.), Advances in Neural In-
formation Processing Systems 32, pp. 8024–8035. Curran
Associates, Inc., 2019.
Paul, F. and Christian, L. The polyhedron model. In Padua,
D. (ed.), Encyclopedia of Parallel Computing, pp. 1581,
1592. Springer, 2011.
Quiller´e, F. and Rajopadhye, S. Optimizing memory us-
age in the polyhedral model. ACM Trans. on Program-
ming Languages and Systems, 22(5):773–815, September
2000.
Ragan-Kelley, J., Adams, A., Paris, S., Levoy, M., Ama-
rasinghe, S., and Durand, F. Decoupling algorithms
from schedules for easy optimization of image processing
pipelines. ACM Trans. Graph., 31(4):32:1–32:12, July
2012. ISSN 0730-0301.
Rahman, M., Pouchet, L.-N., and Sadayappan, P. Neural
network assisted tile size selection. InInternational Work-
shop on Automatic Performance Tuning (IWAPT’2010).
Berkeley, CA: Springer Verlag, 2010.
Smith, L. N. and Topin, N. Super-convergence: Very fast
training of residual networks using large learning rates.
CoRR, abs/1708.07120, 2017. URL http://arxiv.
org/abs/1708.07120.
Trifunovic, K., Nuzman, D., Cohen, A., Zaks, A., and
Rosen, I. Polyhedral-model guided loop-nest auto-
vectorization. In 2009 18th International Conference
on Parallel Architectures and Compilation Techniques,
pp. 327–337, 2009.
Trifunovic, K., Cohen, A., Edelsohn, D., Li, F., Grosser,
T., Jagasia, H., Ladelsky, R., Pop, S., Sjodin, J., and
Upadrasta, R. GRAPHITE two years after: First lessons
learned from Real-World polyhedral compilation, January
2010.
Vasilache, N., Zinenko, O., Theodoridis, T., Goyal, P., De-
Vito, Z., Moses, W. S., Verdoolaege, S., Adams, A., and
Cohen, A. Tensor comprehensions: Framework-agnostic
high-performance machine learning abstractions. CoRR,
abs/1802.04730, 2018.
Wolf, M. E. and Lam, M. S. A loop transformation theory
and an algorithm to maximize parallelism. IEEE transac-
tions on parallel and distributed systems, 2(4):452–471,
1991.
Yang, Z., Dai, Z., Yang, Y ., Carbonell, J. G., Salakhut-
dinov, R., and Le, Q. V . Xlnet: Generalized autore-
gressive pretraining for language understanding. CoRR,
abs/1906.08237, 2019. URL http://arxiv.org/
abs/1906.08237.
A Deep Learning Based Cost Model for Automatic Code Optimization
A A PPENDIX
A.1 Model Architecture Details and Training
Methodology
The computation embedding layer is a fully connected mul-
tilayer perceptron (MLP), feedforward neural network that
takes 1235-dimensional computation vectors and generates
180-dimensional embeddings. We use 3 intermediate lay-
ers of 600, 350, 200 neurons respectively. The output of
each layer is transformed by the ELU function and fed to a
dropout layer with a dropout probability of 0.225, and then
passed to the next layer. This succession of the activation
function and the dropout layer is applied to all the neural
networks of this model. The two LSTM cells in the loop
embedding unit have identical input and hidden vector sizes
that correspond to the output of the computation embedding
layer (180). The feedforward neural network that maps
the concatenated hidden vectors to 180-dimensional loop
embedding have one intermediate layer of size 200. The
regression layer that maps the program embeddingvector to
a speedup value, has two intermediate layers with 200 and
180 neurons.
We implemented our model in PyTorch (Paszke et al., 2019)
(0.4.1.post2). All model parameters are learnable from the
computation embedding layer to the regression layer by way
of the recursive loop embedding layer. For the loss function,
we used MAPE, a normalized metric based on L1. This
loss function is suitable for speedup prediction because the
target value is positive by design. In addition, the function
motivates the model to be equitably accurate in the wide
range of speedups we have. The Glorot initialization (Glorot
& Bengio, 2010) is adopted for all weights of the model. We
train our model using AdamW (Loshchilov & Hutter, 2017)
with a weight decay coefﬁcient of 0.0075. The learning
rate is scheduled by the One Cycle Policy (Smith & Topin,
2017) with a maximum learning rate of 0.001. The other
optimizer parameters are left on their default values. The
best accuracy is achieved after about 700 epochs of training.
The training set is processed in batches of 32 data points.
Each batch is formed by code transformations belonging to
the same algorithm. The rationale for this grouping is that
it is faster to operate on data points having the same tree
structure.
A.2 Benchmark Sizes and Parameters
Table 3 summarizes the benchmarks’ input sizes and param-
eters used in Section 6.
A.3 More Detailed Evaluation
Figure 8 gives an overview of the correlation between the
predicted and measured speedups over 16 programs ran-
domly selected from the test set. Each chart represents the
Benchmark Input size and parameters
box blur 3 × 1024 × 1024
conv + relu
batch size: 8
input size: 1024 × 1024 × 3
kernel size: 3 × 3
output features: 2
convolution
batch size: 8
input size: 1024 × 1024 × 3
kernel size: 3 × 3
output features: 2
cvtcolor 3 × 1024 × 1024
doitgen 256 × 256 × 128, 256 × 256
heat2d 1024 × 1024
heat3d 770 × 898 × 1024
jacobi2d 130 × 1024
mvt 1024 × 1024
seidel2d 256 × 256
Table 3: Benchmarks input sizes and parameters
10 
50 
5 
1 0.5 
0.1 0.05 
0.001 
10 
50 
5 
1 0.5 
0.1 0.05 
0.001 
10 
50 
5 
1 0.5 
0.1 0.05 
0.001 
10 
50 
5 
1 0.5 
0.1 0.05 
0.001 
10 
50 
5 
1 
0.5 
0.1 
0.05 
0.001 
10 
50 
5 
1 
0.5 
0.1 
0.05 
0.001 
10 
50 
5 
1 
0.5 
0.1 
0.05 
0.001 
10 
50 
5 
1 
0.5 
0.1 
0.05 
0.001 
Measured speedups
Predicted speedups
Figure 8: Measured vs predicted speedup on 16 random programs
from the test set, each blue dot represents a code transformation
with respect to its measured speedup and its predicted speedup.
32 random transformations applied on each program with
blue dots, the closer a blue dot is to the red line the lower the
prediction error is. This ﬁgure shows that the cost model’s
predictions ﬁt well the distribution and the range of the
speedups and does not just predict an average value for each
program.

Evaluating Heuristic Optimization Phase Order Search Algorithms
Prasad A. Kulkarni, David B. Whalley, Gary S. Tyson
Florida State University
Computer Science Department
Tallahassee, FL 32306-4530
fkulkarni,whalley,tysong@cs.fsu.edu
Jack W. Davidson
University of Virginia
Department of Computer Science
Charlottesville, VA 22904-4740
jwd@virginia.edu
Abstract
Program-specic or function-specic optimization phase
sequences are universally accepted to achieve better over-
all performance than any xed optimization phase order-
ing. A number of heuristic phase order space search algo-
rithms have been devised to nd customized phase order-
ings achieving high performance for each function. How-
ever, to make this approach of iterative compilation more
widely accepted and deployed in mainstream compilers, it
is essential to modify existing algorithms, or develop new
ones that nd near-optimal solutions quickly. As a step in
this direction, in this paper we attempt to identify and un-
derstand the important properties of some commonly em-
ployed heuristic search methods by using information col-
lected during an exhaustive exploration of the phase order
search space. We compare the performance obtained by
each algorithm with all others, as well as with the optimal
phase ordering performance. Finally, we show how we can
use the features of the phase order space to improve exist-
ing algorithms as well as devise new, and better performing
search algorithms.
1. Introduction
Current compilers contain numerous different optimiza-
tion phases. Each optimization phase analyzes the program
and attempts to change the code in some way to produce
a semantically equivalent program that performs better for
one or some combination of constraints, such as speed,
code size and power. Most optimization phases share re-
sources (such as registers), and require certain conditions in
the code to be applicable before applying a transformation.
Each optimization phase may consume or release resources,
as well as create or destroy specic conditions. As a result,
phases interact with each other, by generating or remov-
ing opportunities for application of other phases. In many
performance-oriented application domains it is important to
nd the best order of applying optimization phases so that
very high-quality code can be produced. This problem of
nding the best sequence of optimization phases to apply is
known as the phase ordering problem in compilers.
The space of all possible orderings of optimization
phases is huge since most current compilers contain nu-
merous different optimization phases with few restrictions
imposed on the order of applying these phases. Prior re-
search has found that no single ordering of phases can
achieve optimal performance on all applications or func-
tions [1, 2, 3, 4, 5, 6]. The phase ordering problem is
difcult since even after decades of research the relation-
ships and interactions between optimization phases remain
ill-understood. The only consensus is that the phase re-
lationships change with the function being optimized, the
manner in which optimizations are implemented in the com-
piler, and the characteristics of the target architecture. Due
to such factors, iterative compilers that evaluate numerous
different orderings of optimization phases have found wide
appeal to search for the best phase order as well as phase
parameters on a per-function basis.
Exhaustive exploration of the optimization phase order
space, although possible in a reasonable amount of time for
a large majority of the functions [7, 8], takes prohibitively
long for most large functions to make it suitable for use in
typical iterative compilers. Instead, faster heuristic algo-
rithms that scan only a portion of the phase order space are
more commonly employed. However, such methods do not
evaluate the entire space to provide any guarantees about the
quality of the solutions obtained. Commonly used heuristic
algorithms to address the phase ordering problem include
genetic algorithms [3, 4], hill climbing algorithms [9, 10],
as well as random searches of the space [11].
In this paper we evaluate many different heuristic search
approaches to determine the important characteristics of
each algorithm as related to the phase order space. We also
evaluate their performance, comparing the performance
with other heuristic approaches as well as with optimal or-
derings. We are able to perform a very thorough study since
we have completely enumerated the phase order spaces
of our compiler for hundreds of functions and can sim-
ply lookup information instead of compiling and simulating
each ordering of phases. Thus, the main contributions of
this paper are:
1. This study is the most detailed evaluation of the perfor-
mance and cost of different heuristic search methods,
and the rst to compare their performance with the op-
timal phase ordering.
2. We isolate and evaluate several properties of each stud-
ied heuristic algorithm, and demonstrate the signi-
cance and difculty in selecting the correct optimiza-
tion phase sequence length, which is often ignored or
kept constant in most previous studies on optimization
phase ordering.
3. This study identies and illustrates the importance of
leaf function instances, and shows how we can exploit
the properties of leaf instances to enhance existing al-
gorithms as well as to construct new search algorithms.
The paper is organized as follows. In the next section
we will discuss research related to the current work. Our
experimental setup will be described in Section 3. In Sec-
tion 4 we will describe the results of our study to determine
the properties of the phase order search space and evaluate
various heuristic search algorithms. Finally, we will present
our conclusions in Section 5.
2. Related Work
Optimization phase ordering has been a long standing
and important issue in compiler optimizations, and as such
has received much recent attention from researchers. Sev-
eral researchers have attempted to formally specify com-
piler optimizations in order to systematically address the
phase ordering problem. Whiteld and Soffa developed a
framework based on axiomatic specications of optimiza-
tions [2, 12]. This framework was employed to list the
potential enabling and disabling interactions between op-
timizations, which were then used to derive an application
order for the optimizations. The main drawback was that
in cases of cyclic interactions between two optimizations, it
was not possible to determine a good ordering automatically
without detailed information about the compiler. Follow-up
work on the same topic has seen the use of additional an-
alytical models, including code context and resource (such
as cache) models, to determine and predict other proper-
ties of optimization phases such as the impact of optimiza-
tions [13], and the protability of optimizations [14]. Some
other work has proposed a Unied Transformation Frame-
work (UTF) to provide a uniform and systematic represen-
tation of iteration reordering transformations and their arbi-
trary combinations [15]. It is possible using UTF to trans-
form the optimization phase order space into a polyhedral
space, which is considered by some researchers to be more
convenient for a systematic exploration than the original
space [16]. However, this work is restricted to loop opti-
mizations, and needs to be extended to other optimizations
before it could be adopted in typical iterative compilers.
Earlier work in iterative compilation concentrated on
nding good parameter settings for a few optimizations,
such as loop unrolling and loop tiling [10, 17, 6]. In cases
where exhaustive exploration was expensive, researchers
used heuristic algorithms, such as grid-based searches, hill
climbers, and genetic algorithms, to scan only a portion of
the search space. A common deduction is that typical pro-
gram search spaces, on a variety of different architectures,
generally contain enough local minima that biased sampling
techniques should nd good solutions. Iterative compila-
tion usually comes at the cost of a large number of program
executions. In order to reduce the cost, some studies have
looked at incorporating static architectural models, particu-
larly cache models, into their approach [18].
Research on the phase ordering problem over all or most
of the optimization phases in a compiler has typically con-
centrated on nding effective (rather than optimal) phase
sequences by evaluating only a portion of the phase order
search space. A method, called Optimization-Space Ex-
ploration [5], uses static performance estimators to reduce
the search time. In order to prune the search they limit the
number of congurations of optimization-parameter value
pairs to those that are likely to contribute to performance
improvements. This area has also seen the application of
other search techniques to intelligently search the optimiza-
tion space. Hill climbers [9] and genetic algorithms [3, 4]
have been employed during iterative algorithms to nd opti-
mization phase sequences better than the default one used in
their compilers. Such techniques are often combined with
aggressive pruning of the search space [19, 20] to make
searches for effective optimization phase sequences faster
and more efcient. Successful attempts have also been
made to use predictive modeling and code context informa-
tion to focus search on the most fruitful areas of the phase
order space for the program being compiled [11]. Other ap-
proaches to reduce compilation time estimate the number of
instructions executed and use that as a means to prune the
number of versions of a program that need to be executed
or simulated for evaluation [21, 8].
Researchers have also attempted a near-optimal reso-
lution of the phase ordering problem considering all the
phases present in their compilers. Kulkarni et al. demon-
strated that for typical optimization phases applied in a
compiler backend it is possible to exhaustively evaluate the
phase order space for most of the functions in several em-
bedded benchmarks [7, 8]. This was made possible by using
techniques to drastically prune the space of distinct func-
tion instances, as well as reducing the number of program
executions to estimate the performances of all nodes in the
space. However, they were unable to exhaustively explore
the space for some of the largest functions, while some other
large functions each took several days to evaluate. This
shows that exhaustive space evaluation may still be infea-
sible for routine use in typical iterative compilers.
In spite of the wide-spread use of heuristic approaches,
there have been few attempts to evaluate and compare their
properties and performance in the context of the characteris-
tics of the phase order space. Kisuki et al. analyzed the per-
formance of ve different search algorithms, and reported
the observation that heuristic algorithms do not differ much
in their efciency [10]. However, this study was performed
on a space of only two optimizations (with different param-
eters), and did not take in to account properties of the phase
order space. A more detailed evaluation was performed by
Almagor et al. [9], which attempted to relate features of the
phase order space with the efciency and performance of
different heuristic algorithms. This study was, however, in-
complete since they had restricted their sequence length to
be of a xed size. This earlier work also did not have ac-
cess to the entire phase order space features, and lacked a
knowledge of the optimal phase order performance.
3. Experimental Framework
In this section we rst describe the compiler framework,
and then explain our experimental setup.
3.1. Compiler Framework
The research in this paper uses the Very Portable Opti-
mizer (VPO) [22], which is a compiler back end that per-
forms all its optimizations on a single low-level interme-
diate representation called RTLs (Register Transfer Lists).
Since VPO uses a single representation, it can apply most
analysis and optimization phases repeatedly and in an arbi-
trary order. VPO compiles and optimizes one function at a
time. This is important since different functions may have
very different best orderings, so any strategy that requires
all functions in a le to have the same phase order will al-
most certainly not be optimal. At the same time, restrict-
ing the phase ordering problem to a single function helps to
make the phase order space more manageable. The com-
piler has been targeted to generate code for the StrongARM
SA-100 processor using Linux as its operating system. We
used the SimpleScalar set of functional simulators [23] for
the ARM to get dynamic performance measures.
Table 1 describes each of the 15 candidate code-
improving phases that were used during the exhaustive ex-
ploration of the optimization phase order search space. In
addition, register assignment, which is a compulsory phase
that assigns pseudo registers to hardware registers, must be
performed. VPO implicitly performs register assignment
before the rst code-improving phase in a sequence that re-
quires it. After applying the last code-improving phase in
a sequence, VPO performs another compulsory phase that
inserts instructions at the entry and exit of the function to
manage the activation record on the run-time stack. Fi-
nally, the compiler also performs predication and instruc-
tion scheduling before the nal assembly code is produced.
These last two optimizations should only be performed late
in the compilation process in the VPO compiler, and so are
not included in the set of re-orderable optimization phases.
For the experiments described in this paper we used a
subset of the benchmarks from the MiBench benchmark
suite, which are C applications targeting specic areas of
the embedded market [24]. We selected two benchmarks
from each of the six categories of applications present in
MiBench. Table 2 contains descriptions of these programs.
VPO compiles and optimizes individual functions at a time.
The 12 benchmarks selected contained a total of 244 func-
tions, out of which 88 were executed (at least once) with the
input data provided with each benchmark.
3.2. Experimental Setup
For this study we exhaustively enumerate the phase or-
der space for a large number of functions. This enumer-
ation enables us to accurately investigate the properties of
the search space, to study heuristic search algorithms, to
tune the algorithms, to suggest new algorithms, as well as
to compare the efciency of different heuristic algorithms
in nding optimal phase ordering for each function. In this
section we will describe our setup for the current research.
We use the algorithm proposed by Kulkarni et al. [7, 8]
to exhaustively evaluate the phase order space for the func-
tions in our benchmark suite. The algorithm is illustrated in
Figure 1. The phase ordering problem is viewed as an eval-
uation of all possible distinct function instances that can be
generated by changing the order of optimization phases in
a compiler. This approach to the phase ordering problem
makes no assumptions about the phase sequence length for
each function, and allows phases to be repeated as many
times as they can possibly be active in the same sequence.
Thus, the phase ordering space is complete in each case.
However, it is important to note that a different compiler,
with a different or greater set of optimization phases can
possibly generate better code than the optimal instance pro-
duced by VPO. Thus, optimal in the context of this work
refers to the best code that can be produced by any opti-
mization phase ordering in VPO and is not meant to imply a
universally optimum solution. The algorithm can be briey
summarized as follows:
Optimization Phase Gene Description
branch chaining b Replaces a branch or jump target with the target of the last jump in the jump chain.
common subexpression
elimination
c Performs global analysis to eliminate fully redundant calculations, which also includes global
constant and copy propagation.
remove unreach. code d Removes basic blocks that cannot be reached from the function entry block.
loop unrolling g To potentially reduce the number of comparisons and branches at runtime and to aid scheduling
at the cost of code size increase.
dead assign. elim. h Uses global analysis to remove assignments when the assigned value is never used.
block reordering i Removes a jump by reordering blocks when the target of the jump has only a single predecessor.
minimize loop jumps j Removes a jump associated with a loop by duplicating a portion of the loop.
register allocation k Uses graph coloring to replace references to a variable within a live range with a register.
loop transformations l Performs loop-invariant code motion, recurrence elimination, loop strength reduction, and in-
duction variable elimination on each loop ordered by loop nesting level.
code abstraction n Performs cross-jumping and code-hoisting to move identical instructions from basic blocks to
their common predecessor or successor.
eval. order determ. o Reorders instructions within a single basic block in an attempt to use fewer registers.
strength reduction q Replaces an expensive instruction with one or more cheaper ones. For this version of the com-
piler, this means changing a multiply by a constant into a series of shift, adds, and subtracts.
reverse branches r Removes an unconditional jump by reversing a cond. branch when it branches over the jump.
instruction selection s Combines pairs or triples of instructions together where the instructions are linked by set/use
dependencies. Also performs constant folding and checks if the resulting effect is a legal in-
struction before committing to the transformation.
remove useless jumps u Removes jumps and branches whose target is the following positional block.
Table 1. Candidate Optimization Phases along with Their Designations
1. The algorithm starts with the un-optimized function
instance. A depth-rst search algorithm is used to
produce the next sequence to evaluate. Each new se-
quence appends one phase to a previous sequence re-
sulting in a new function instance, in depth rst order.
2. Phases not successful in changing the program repre-
sentation do not need further evaluation.
3. The next stage uses CRC hash values [25], calculated
on the entire function, to compare the current function
instance with all previous distinct function instances.
If the current function instance is identical to an in-
stance previously produced by some other phase se-
quence, then only one needs to be evaluated, and so
the current instance is discarded.
4. Even if two function instances are not identical, it is
possible that the only differences may lie in the reg-
ister numbers being used, or the labels assigned to the
various basic blocks. In such cases the two function in-
stances will still perform identically, and so the current
function instance no longer needs further evaluation.
5. The next step determines the performance of each dis-
tinct function instance. In order to reduce the num-
ber of program simulations, the algorithm only simu-
lates one function instance from the set of function in-
stances having the same basic block control ow. The
rst function instance with a new block control ow
is instrumented and simulated to obtain the number of
times each block is executed. The dynamic frequency
measure for the each function instance is determined
by multiplying the block execution counts by the esti-
mated number of static cycles for each block. These
dynamic frequency measures have been shown to bear
a strong correlation with simulated cycles.
Category Program Description
auto bitcount test proc. bit manipulation abilities
qsort sort strings using the quicksort algo.
network dijkstra Dijkstra's shortest path algorithm
patricia construct patricia trie for IP trafc
telecomm fft fast fourier transform
adpcm compress 16-bit linear PCM samples
to 4-bit samples
consumer jpeg image compression and decomp.
tiff2bw convert color tiff image to b&w image
security sha secure hash algorithm
blowsh symmetric block cipher with variable
length key
ofce search searches for given words in phrases
ispell fast spelling checker
Table 2. MiBench Benchmarks Used
The exhaustive enumeration of any function is stopped
if the time required exceeds an approximate limit of two
calculate
function
performance
generate
next
optimization
sequence
using
depth-first
approach
last phase
active?
Y Nidentical
function
instance?
equivalent N
instance?
function
control-flow
structure?
seenN
Y
N Y Y
simulate
application
Figure 1. Steps During an Exhaustive Evaluation of the Phase Order Space for Each Function
weeks. Using the above algorithm and cut-off criteria we
were able to enumerate completely the optimization phase
order space for 234 out the 244 functions that we studied.
Out of the 88 executed functions, we were able to com-
pletely enumerate 79. We represent the phase order space
for every function in the form of a DAG (Directed Acyclic
Graph), as shown in Figure 2. Nodes in the DAG represent
distinct function instances and edges represent transitions
from one node to the next on application of an optimization
phase. The DAG then enables much faster evaluation of any
search heuristic, since compilation as well as execution can
be replaced with a simple table lookup in the DAG to deter-
mine the performance of each phase ordering. As a result,
the study of the various algorithms is fast, and it is possible
to evaluate various parameters of the algorithms as well as
the search space.
b c
a d d
a
d
3 4
(b) Depth-first Traversal
6
7
8
1
52
a
c
Figure 2. Directed Acyclic Graph Represent-
ing the Phase Order Space of a Function
4 Study of Common Heuristic Search Tech-
niques
Over the past decades researchers have employed vari-
ous heuristic algorithms to cheaply nd effective solutions
to the phase ordering problem. However, several issues re-
garding the relative performance and cost of each algorithm,
as well as the effect of changing different algorithm param-
eters on that algorithm's performance are as yet uncertain
and not clearly understood. In this section we will perform
a thorough evaluation and comparison of commonly used
heuristic methods. Based on the phase order space charac-
teristics we will also develop new techniques and suggest
several improvements to existing algorithms.
4.1. Local Search Techniques
Local search techniques, such as hill climbing and sim-
ulated annealing, can only migrate to neighboring points
from one iteration to the next during their search for good
solutions. Central to these algorithms is the denition of
neighbors of any point in the space. For this study, we de-
ne the neighbors of a sequence to be all those sequences
that differ from the base sequence in only one position.
Thus, for a compiler with only three optimization phases a,
b and c, the sequence shown in the rst column of Table 3
will have the sequences listed in the following columns as
its neighbors. The position that differs in each neighbor
is indicated in bold. For a compiler with m optimization
phases, a sequence of length n will have (m   1)n neigh-
bors. Unless the search space is extremely smooth, these
local search algorithms have a tendency to get stuck in a
local minimum, which are points that are not globally mini-
mum, but have better tness values than any of their neigh-
bors. For a comprehensive study of these algorithms it is
important to rst understand relevant properties of the opti-
mization phase order space. The results from this study are
presented in the next section.
bseq neighbors
a b c a a a a a a
b b b a c b b b b
c c c c c a b c c
a a a a a a a b c
Table 3. Neighbors in Heuristic Searches
4.1.1 Distribution of Local Minima in the Phase Order
Space
Earlier studies have attempted to probe the properties of
the phase order space [17, 9]. Such studies, however, only
looked at a small portion of the space, and ignored im-
portant factors affecting the nature and distribution of lo-
cal minima in phase order spaces. One such factor, com-
monly ignored, is the optimization sequence length. It is
almost impossible to estimate the best sequence length to
use due to the ability and tendency of optimization phases
to enable other phases. During our experiments, maximum
sequence lengths of active phases varied from 3 to 44 over
different functions, with considerable variation within the
same function itself for different phase orderings. The goal
of analyzing the search space, in the context of local search
techniques, is to nd the properties and distribution of all
local minima in each phase order search space. However,
there are some difcult hurdles in achieving this goal:
Variable sequence length: Since the best sequence
length for each function is unknown, an ideal analysis
would require nding the properties of local minima for all
possible sequence lengths. This requirement is needed be-
cause any sequence of attempted phases of any length de-
nes a point in the search space DAG. Conversely, a sin-
gle point in the space can be dened by, potentially, innite
number of attempted sequences of different lengths. This
is important, since different sequences dening the same
point will have different neighbors. This implies that some
of those sequences may be locally optimum, while others
may be not, even though they dene the same point in the
phase order space. For example, the attempted sequences
fb ! ag, fc ! b ! ag, and fd ! b ! c ! ag all
dene the same node 4 in the DAG in Figure 2 (Note that,
the phases a and b, indicated in bold, are active, while c and
d are dormant). Thus, we can see that it is possible to have
sequences of different lengths pointing to the same node.
Thus, this ideal goal of nding the local minima for all pos-
sible sequence lengths is clearly impossible to achieve.
Fixed sequence length: A conceptually simpler ap-
proach would be to use some oracle to give us the best
sequence length to use for each function, and then only an-
alyze the space for this single sequence length. The mini-
mum reasonable length to use, so that all nodes in the DAG
can be reached, would be the maximum active sequence
length for each function. For an average maximum active
sequence length of 16, over all 234 enumerated functions,
we would need to evaluate 1516 different phase orderings
for each function. Evaluation of any phase ordering to de-
termine if that ordering is a local optimum would in turn re-
quire us to lookup the performance of that ordering as well
as that of its 15  16 neighbors. This, also, is clearly a huge
undertaking considering that the maximum active sequence
length we encountered during our exhaustive phase order
enumeration study was 44.
Due to such issues, in our present experiments, we de-
cided to use sampling to probe only a reasonable portion of
the phase order search space for some number of different
sequence lengths for each function. We use 16 different se-
quence lengths. The initial length is set to the length of the
sequence of active phases applied by the conventional VPO
compiler in batch mode. The remaining sequence lengths
are successive increments of one-fourth of the initial se-
quence length used for each function. The larger sequence
lengths may be needed to accommodate phases which may
be dormant at the point they are attempted. For each set of
experiments for each function, we rst randomly generate
a sequence of the specied length. We then compare the
performance of the node that this sequence denes with the
performance of all of its neighbors to nd if this sequence is
a local optimal. This base node is marked as done. All later
sequences are constrained to dene different nodes in the
space. As this sampling process progresses it will require
an increasing number of attempts to nd a sequence corre-
sponding to an unevaluated node in the search space. The
process terminates when the average number of attempts to
generate a sequence dening a new node exceeds 100.
Figures 3(a) and 3(b) illustrate the average phase order
space properties over all the executed functions that we
studied. The plot labeled % nodes touched in the DAG from
Figure 3(a) shows the percentage of nodes that were evalu-
ated for local minimum from amongst all nodes in the space
DAG. This number initially increases, reaches a peak, and
then drops off. This graph, in effect, shows the nature of
typical phase order spaces. Optimization phase order space
DAGs typically start out with a small width, reach a max-
imum around the center of the DAG, and again taper off
towards the leaf nodes as more and more function instances
generated are detected to be redundant. Smaller attempted
sequence lengths in Figure 3(a) dene points higher up in
the DAG, with the nodes dened dropping down in the DAG
as the length is increased. The next plot labeled avg local
minima % distance from optimal in Figure 3(a) measures
the average difference in performance from optimal over
all the samples at each length. As the sequence lengths in-
creased the average performance of the samples gets closer
and closer to optimal, until after a certain point the perfor-
mance remains more or less constant. This is expected,
and can be explained from the last plot in Figure 3(a), la-
beled %(avg.active seq. length / batch seq. length), which
shows the percentage increase in the average length of ac-
tive phases as the attempted sequence length is increased.
The ability to apply more active phases implies that the
function is better optimized, and thus we see a correspond-
ing increase in performance and a smaller percentage of ac-
tive phases.
The rst plot in Figure 3(b) shows the ratio of sequences
reaching local minima to the total sequences probed. This
ratio seems to remain more or less constant for different
lengths. The small percentage of local minima in the to-
tal samples indicates that there are not many local minima
(a) Local Minima Information (b) Global Minima Information
Figure 3. Search Space Properties
in the space. The next plot, %(num global minima / to-
tal minima), in this gure shows that the percentage of lo-
cally minimum nodes achieving global minima grows with
increase in sequence length. This increase is more pro-
nounced initially, but subsequently becomes steadier. In the
steady state around 45% of local minima display globally
optimum performance. This characteristic means that for
longer sequence lengths there is a good chance that the lo-
cal minimum found during local search algorithms will have
globally optimal performance. The nal plot in Figure 3(b),
%(functions for which at least one sample reached optimal),
presents the percentage of functions for which the probe is
able to nd optimal in at least one of its samples. This num-
ber shows a similar characteristic of continuously increas-
ing with sequence lengths, until it reaches a steady state at
close to 100% for larger sequence lengths. Hence, for small
multiples of the batch sequence length the local search al-
gorithms should be able to nd global minima with a high
probability for most of the functions.
Thus, this study illustrates that it is important to nd the
correct balance between increase in sequence length, per-
formance obtained, and the time required for the search. Al-
though larger sequence lengths tend to perform better they
are also more expensive to evaluate, since they have more
neighbors, and evaluation of each neighbor takes longer. It
is worthwhile to note that we do not need to increase the se-
quence lengths indenitely. After a modest increase in the
sequence lengths, as compared to the xed batch sequence
length, we are able to obtain most of the potential benets
of any further increases in sequence lengths.
4.1.2 Hill Climbing
In this section we evaluate the performance of the steepest
descent hill climbing heuristic algorithm for different se-
quence lengths [9]. The algorithm is initiated by randomly
populating a phase sequence of the specied length. The
performance of this sequence is evaluated, along with that
of all its neighbors. If the best performing neighbor has
equal or better performance than the base sequence, then
that neighbor is selected as the new base sequence. This
process is repeated until a local optimum is reached, i.e.,
the base sequence performs better than all of its neighbors.
For each sequence length, 100 iterations of this algorithm
are performed by selecting random starting points in the
search space. The sequence lengths were incremented 40
times starting from the length of the active batch sequence,
with each increment equal to one-fourth the batch length.
Figures 4(a) and 4(b) illustrate the results of the hill
climbing experiments. The plot marked % best perf. dis-
tance from optimal in Figure 4(a) compares the best solution
found by the hill climbing algorithm with optimal, averaged
over the 79 executed functions, and over all 100 iterations
for each sequence length. We can see that even for small
sequence lengths the algorithm is able to obtain a phase or-
dering whose best performance is very close to optimal. For
lengths greater than 1.5 times the batch sequence length,
the algorithm is able to reach optimal in most cases. The
plot avg. steps to local minimum in Figure 4(a) shows that
the simple hill climbing algorithm requires very few steps
to reach local optimal, and that the average distance to the
local optimal decreases with increase in sequence lengths.
This decrease in the number of steps is caused by better
performance delivered by each typical sequence when the
initial sequence length is increased, so that in effect the al-
gorithm starts out with a better initial sequence, and takes
fewer steps to the local minimum.
As mentioned earlier, the hill climbing algorithm is iter-
ated 100 times for each sequence length and each function
to eliminate the noise caused by the random component of
the algorithm. The rst plot in Figure 4(b), avg. % iter-
ations reaching optimal, illustrates that the average num-
ber of iterations reaching optimal increases with increase in
the sequence length up to a certain limit, after which it re-
mains more or less constant. A related measure % avg. perf.
(a) Local Minima Information (b) Global Minima Information
Figure 4. Properties of the Hill Climbing Algorithm
distance from optimal, shown in the second plot in Figure
4(b), is the average function performance over all the itera-
tions for each sequence length. This measure also shows a
marked improvement as the sequence length increases un-
til the average performance peaks at around 4% worse than
optimal. These results indicate the signicance of select-
ing a correct sequence length during the algorithm. Larger
sequence lengths lead to larger active sequences that result
in the initial performance improvement, but increasing the
length incessantly gives diminishing returns while making
the algorithm more expensive.
4.1.3 Simulated Annealing
Simulated annealing can be dened as a technique to nd a
good solution to an optimization problem by trying random
variations of the current solution. A worse variation is ac-
cepted as the new solution with a probability that decreases
as the computation proceeds. The slower the cooling sched-
ule, or rate of decrease, the more likely the algorithm is to
nd an optimal or near-optimal solution [26]. In our im-
plementation, the algorithm proceeds similarly to the hill
climbing algorithm by starting from a random initialization
point in the phase order space. The sequence length is xed
for each run of the algorithm. During each iteration the per-
formance of the base sequence is evaluated along with that
of all its neighbors. Similar to the hill climbing method, if
the performance of the best performing neighbor is better
than the performance of the base sequence, then that neigh-
bor is selected as the base sequence for the next iteration.
However, if the current iteration is not able to nd a neigh-
bor performing better than the base sequence, the algorithm
can still migrate to the best neighbor based on its current
temperature. The worse solution is generally accepted with
a probability based on the Boltzmann probability distribu-
tion:
prob = exp( f
T ) (1)
where, f is the difference in performance between the cur-
rent base sequence and the best neighbor, and T is the cur-
rent temperature. Thus, smaller the degradation and higher
the temperature the greater the probability of a worse solu-
tion being accepted.
An important component of the simulated annealing al-
gorithm is the annealing schedule, which determines the
initial temperature and how it is lowered from high to low
values. The assignment of a good schedule generally re-
quires physical insight and/or trial and error experiments. In
this paper, we attempt to study the effect of different anneal-
ing schedules on the performance of a simulated annealing
algorithm. For this study, the sequence length is xed at
1.5 times the batch compiler length of active phases. As
seen in the hill climbing experiments, this is the smallest
sequence length at which the average performance reaches
a steady state that is very close to optimal. We conducted
a total of 400 experimental runs by varying the initial tem-
perature and the annealing schedule. The temperature was
varied from 0 to 0.95 in steps of 0.5. For each temperature
we dened 20 different annealing schedules, which control
the temperature in steps from 0.5 to 0.95 per iteration. The
results for each conguration are averaged over 100 runs to
account for noise caused by random initializations.
Our results, shown in Figure 5, indicate that for the phase
ordering problem, as seen by our compiler, the initial tem-
perature as well as the annealing schedule do not have a
signicant impact on the performance delivered by the sim-
ulated annealing algorithm. The best performance obtained
over all the 400 experimental runs is, on average, 0.15% off
from optimal, with a standard deviation of 0.13%. Like-
wise, other measures obtained during our experiments are
also consistent across all 400 runs. The average number of
iterations achieving optimal performance during each run
is 41.06%, with a standard deviation of 0.81%. The av-
erage performance for each run is 15.95% worse than op-
timal, with a deviation of 0.55%. However, as expected,
the number of steps to a local minimum during each itera-
tion for each run increases with increase in the initial tem-
perature and the annealing schedule step. As the starting
temperature and annealing schedule step are increased, the
algorithm accepts more poorly performing solutions before
halting. However, this increase in the number of steps to
local optimal does not translate into any signicant perfor-
mance improvement for our experiments,
Figure 5. Increase in the Number of Steps to
Local Minimum with Increases in Initial Tem-
perature and Annealing Schedule Step
4.2. Greedy Algorithm
Greedy algorithms follow the policy of making the lo-
cally optimum choice at every step in the hope of nally
reaching the global optimum. Such algorithms are com-
monly used for addressing several optimization problems
with huge search spaces. For the phase ordering problem,
we start off with the empty sequence as the base sequence.
During each iteration the algorithm creates new sequences
by adding each available phase rst to the prex and then as
the postx of the base sequence. Each of these sequences
is evaluated to nd the best performing sequence in the cur-
rent iteration, which is consequently selected as the base se-
quence for the next iteration. If there are multiple sequences
obtaining the same best performance, then one of these is
selected at random. The algorithm is repeated 100 times in
order to reduce the noise that can potentially be caused by
this random component in the greedy method. Thus, in our
case the algorithm has a bounded cost, as it performs a xed
number of (15+15=30) evaluations in each step, where 15
is the number of available optimizations in our compiler.
Our current implementation of the greedy algorithm is
inspired by the approach used by Almagor et al. [9]. Simi-
lar to the hill climbing algorithm, the sequence lengths dur-
ing the greedy algorithm are varied from the active batch
sequence length for each function as the initial length to
11 times the batch length, in increments of one-fourth the
batch length. To minimize the effect of the random com-
ponent, the algorithm is repeated 100 times for each se-
quence length. The best and average performances dur-
ing these 100 iterations for each sequence length, averaged
over all executed functions, are illustrated in Figure 6. The
plots show a similar pattern to the hill climbing perfor-
mance graphs. However, it is interesting to note that the
best achievable performance during the greedy algorithm is
around 1.1% worse than optimal, whereas it is very close
to optimal (0.02%) for the hill climbing algorithm. Also,
the average performance during the greedy algorithm im-
proves more gradually and continues to improve for larger
sequence lengths as compared to hill climbing.
Figure 6. Greedy Algorithm Performance
4.3. Focusing on Leaf Sequences of Active
Phases
Leaf function instances are those that cannot be further
modied by the application of any additional optimizations
phases. These function instances represent leaves in the
DAG of the phase order space (e.g. nodes 3, 4, 6, and 8
in Figure 2). Sequences of active phases leading to leaf
function instances are called leaf sequences. Working with
only the leaf sequences has the advantage that the heuris-
tic algorithm no longer needs to guess the most appropriate
sequence length to minimize the algorithm running time,
while at the same time obtaining the best, or at least close
to the best possible performance. Since leaf function in-
stances are generated by different lengths of active phase
sequences, the length of the leaf sequences is variable. In
this section we describe our modications to existing algo-
rithms, as well as introduce new algorithms that deal with
only leaf sequences.
We rst motivate the reason for restricting the heuristic
searches to only leaf function instances. Figure 7 shows
the distribution of the dynamic frequency counts as com-
pared to the optimal for all distinct function instances ob-
tained during our exhaustive phase order space evaluation,
averaged over all 79 executed functions. From this gure
we can see that the performance of the leaf function in-
stances is typically very close to the optimal performance,
and that leaf instances comprise a signicant portion of op-
timal function instances with respect to the dynamic fre-
quency counts. This fact is quite intuitive since active op-
timizations generally improve performance, and very rarely
cause a performance degradation. The main drawback of
this approach is that the algorithm will not nd the optimal
phase ordering for any function that does not have an opti-
mal performing leaf instance. However, we have observed
that most functions do contain optimal performing leaf in-
stances. For more than 86% of the functions in our bench-
mark suite there is at least one leaf function instance that
achieved optimal dynamic frequency counts. The average
best performance for leaf function instances over all exe-
cuted functions is only 0.42% worse than optimal. More-
over, leaf function instances comprise only 4.38% of the to-
tal space of distinct function instances, which is in turn a mi-
nuscule portion of the total phase order search space. Thus,
restricting the heuristic search to only the leaf function in-
stances constrains the search to only look at a very small
portion of the search space that typically consists of good
function instances, and increases the probability of nding
a near-optimal solution quickly. In the next few sections
we will describe some modications to existing algorithms,
as well as describe new algorithms that take advantage of
the opportunity provided by leaf function instances to nd
better performance faster.
Figure 7. Average Distribution of Dynamic
Frequency Counts
4.3.1 Genetic Algorithm
Genetic algorithms are adaptive algorithms based on Dar-
win's theory of evolution [27]. These algorithms have been
successfully employed by several researchers to address the
phase ordering problem and other related issues in compil-
ers [4, 3, 28, 29]. Genes correspond to optimization phases
and chromosomes correspond to optimization sequences in
the genetic algorithm. The set of chromosomes currently
under consideration constitutes a population. The number
of generations is how many sets of populations are to be
evaluated. Our experiments with genetic algorithms sug-
gests that minor modications in the conguration of these
parameters do not signicantly affect the performance de-
livered by the genetic algorithms. For the current study we
have xed the number of chromosomes in each population
at 20. Chromosomes in the rst generation are randomly
initialized. After evaluating the performance of each chro-
mosome in the population, they are sorted in decreasing
order of performance. During crossover, 20% of chromo-
somes from the poorly performing half of the population
are replaced by repeatedly selecting two chromosomes from
the better half of the population and replacing the lower half
of the rst chromosome with the upper half of the second
and vice-versa to produce two new chromosomes each time.
During mutation we replace a phase with another random
phase with a small probability of 5% for chromosomes in
the upper half of the population and 10% for the chromo-
somes in the lower half. The chromosomes replaced during
crossover are not mutated.
The only parameter that seems to signicantly affect the
performance of the genetic algorithm is the length of each
chromosome. We conducted two different studies with ge-
netic algorithms. In the rst study we vary the length of the
chromosomes (attempted sequence) starting from the batch
sequence length to 11 times the batch sequence length, in
steps of one-fourth of the batch length. For the second
study we modied the genetic algorithm to only work with
leaf sequences of active phases. This approach requires
maintaining active leaf sequences of different lengths in the
same population. After crossover and mutation it is pos-
sible that the new sequences no longer correspond to leaf
function instances, and may also contain dormant phases.
The modied genetic algorithm handles such sequences by
rst squeezing out the dormant phases and then extending
the sequence, if needed, by additional randomly generated
phases to get a leaf sequence. Figures 8(a) and 8(b) shows
the performance results, as well as a comparison of the two
approaches. Since the modied genetic algorithm for leaf
instances does not depend on sequence lengths, the aver-
age performance and cost delivered by this new algorithm
are illustrated by single horizontal lines in Figures 8(a) and
(b).The number of generations is a measure of the cost of
the algorithm. Thus, by concentrating on only the leaf func-
tion instances, the genetic algorithm is able to obtain close
to the best performance at close to the least cost possible
for any sequence length. Interestingly, performance of the
genetic algorithm for leaf sequences (0.43%) is very close
to the best achievable average leaf performance (0.42%).
(a) Performance (b) Cost
Figure 8. Performance and Cost of Genetic Algorithms
(a) Performance (b) Cost
Figure 9. Performance and Cost of Random Search Algorithms
4.3.2 Random Search
Random sampling of the search space to nd good solutions
is an effective technique for search spaces that are typically
discrete and sparse, and when the relationships between the
various space parameters are not clearly understood. Exam-
ples are the search spaces that we are dealing with to address
the phase ordering problem. In this study we have attempted
to evaluate random sampling, again by performing two dif-
ferent sets of experiments similar to the genetic algorithm
experiments in the previous section. For the rst set, ran-
domly constructed phase sequences of different lengths are
evaluated until 100 consecutive sequences fail to show an
improvement over the current best. The second set of ex-
periments is similar, but only considers leaf sequences or
leaf function instances.
Figures 9(a) and 9(b) show the performance benets as
well as the cost for all our random search experiments. It
is interesting to note that random searches are also able to
achieve performance close to the optimal for each function
in a few number of attempts. Since our algorithm cong-
uration mandates the best performance to be held steady
for 100 consecutive sequences, we see that the cost of our
algorithm is always above 100 attempts. We again notice
that leaf sequences consistently obtain good performance
for the random search algorithm as well. In fact, for our
current conguration, random search algorithm concentrat-
ing on only the leaf sequences is able to cheaply outperform
the best achievable by any other random search algorithm
for any sequence length.
4.3.3 N-Lookahead Algorithm
This algorithm scans N levels down the search space DAG
from the current location to select the phase that leads to
the best subsequence of phases to apply. The critical pa-
rameter is the number of levels to scan. For a N lookahead
algorithm we have to evaluate 15N different optimization
phases to select each phase in the base sequence. This pro-
cess can be very expensive, especially for larger values of
the lookahead N. Thus, in order for this approach to be
feasible we need to study if small values of the lookahead
N can achieve near optimal performance for most of the
functions.
For the current set of experiments we have constrained
the values of N to be either 1, 2, or 3 levels of lookahead.
Due to the exponential nature of the phase order search
space, we believe that any further increase in the lookahead
value will make this method too expensive in comparison
with other heuristic approaches. Table 4 shows the average
performance difference from optimal for the three levels of
lookahead over all the executed functions in our set. As
expected, the performance improves as the levels of looka-
head are increased. However, even after using three lev-
els of lookahead the performance is far from optimal. This
illustrates the ragged nature of typical phase order search
spaces, where it is difcult to predict the nal performance
of a phase sequence by only looking at a few number of
phases further down from the current location.
Lookahead
1 2 3
% Performance 22.90 14.64 5.35
Table 4. Perf. of N-Lookahead Algorithm
5. Conclusions
In this paper we studied various properties of the opti-
mization phase order space, and evaluated various heuristic
search algorithms. Based on our observations regarding the
search space properties, we further suggested and evaluated
extensions to existing heuristic algorithms, and developed
new heuristic methods. This study is also the rst compar-
ison of the performance delivered by the different heuris-
tic algorithms with the optimal phase ordering. A study of
this magnitude would have normally required several man-
months to accomplish. However, the presence of the ex-
haustive phase order exploration data over a large number
of functions meant that this study required no further com-
pilation or simulation runs to determine the performance of
each unique phase sequence during the heuristic algorithms.
We have a number of interesting conclusions from our
detailed study: (1) Analysis of the phase order search space
indicates that the space is highly discrete and very sparse.
(2) The phase order space typically has a few local and
global minima. More importantly, the sequence length of
attempted phases denes the percentage of local and global
minima in the search space. Larger sequence lengths in-
crease the probability of nding a global minima, but can
also increase the search time to nd a good solution. Thus, it
is important to nd the correct sequence lengths to balance
algorithm cost, and its ability to reach better performing so-
lutions faster. (3) Due to the inherent difculty in determin-
ing the ideal sequence length to use during any heuristic
method, and the high probability of obtaining near-optimal
performance from leaf function instances, we modied ex-
isting algorithms to concentrate on only leaf sequences and
demonstrated that for many algorithms leaf sequences can
deliver performance close to the best, and often times even
better than that obtained by excessive increases in sequence
lengths for the same algorithms. Moreover, this can be
achieved at a fraction of the running cost of the original
algorithm since the space of leaf function instances is only
4.38% of the total space of all function instances. (4) On
comparing the performance and cost of different heuristic
algorithms we nd that simple techniques, such as local hill
climbing allowed to run over multiple iterations, can often
outperform more complex techniques such as genetic algo-
rithms and lookahead schemes. The added complexity of
simulated annealing, as compared to hill climbing, is found
to not signicantly affect the performance of the algorithm.
Random searches and greedy search algorithms achieve de-
cent performance, but not as good as the other heuristic ap-
proaches for the amount of effort expended. The unpre-
dictable nature of phase interactions is responsible for the
mediocre performance of the N-lookahead heuristic algo-
rithm. (5) Interestingly, most of the heuristic algorithms we
evaluated are able to achieve performance close to the best
phase ordering performance in acceptable running times for
all functions. Thus, in conclusion we nd that differences
in performance delivered by different heuristic approaches
are not that signicant when compared to the optimal phase
ordering performance. Selection of the correct sequence
length is important for algorithms that depend on it, but can
be safely bypassed without any signicant performance loss
wherever possible by concentrating on leaf sequences.
6. Acknowledgments
We thank the anonymous reviewers for their constructive
comments and suggestions. This research was supported
in part by NSF grants EIA-0072043, CCR-0208892, CCR-
0312493, CCF-0444207, CNS-0072043, CNS-0305144,
and CNS-0615085.
References
[1] Steven R. Vegdahl. Phase coupling and constant generation
in an optimizing microcode compiler. In Proceedings of the
15th annual workshop on Microprogramming, pages 125
133. IEEE Press, 1982.
[2] D. Whiteld and M. L. Soffa. An approach to ordering opti-
mizing transformations. In Proceedings of the second ACM
SIGPLAN symposium on Principles & Practice of Parallel
Programming, pages 137146. ACM Press, 1990.
[3] Keith D. Cooper, Philip J. Schielke, and Devika Subrama-
nian. Optimizing for reduced code space using genetic al-
gorithms. In Workshop on Languages, Compilers, and Tools
for Embedded Systems, pages 19, May 1999.
[4] Prasad Kulkarni, Wankang Zhao, Hwashin Moon, Kyungh-
wan Cho, David Whalley, Jack Davidson, Mark Bailey, Yun-
heung Paek, and Kyle Gallivan. Finding effective optimiza-
tion phase sequences. In Proceedings of the 2003 ACM SIG-
PLAN conference on Language, Compiler, and Tool for Em-
bedded Systems, pages 1223. ACM Press, 2003.
[5] Spyridon Triantafyllis, Manish Vachharajani, Neil Vachhara-
jani, and David I. August. Compiler optimization-space ex-
ploration. In Proceedings of the international symposium on
Code Generation and Optimization, pages 204215. IEEE
Computer Society, 2003.
[6] T. Kisuki, P.M.W. Knijnenburg, M.F.P. O'Boyle, F. Bodin,
and H.A.G. Wijshoff. A feasibility study in iterative compi-
lation. In Proc. ISHPC'99, volume 1615 of Lecture Notes in
Computer Science, pages 121132, 1999.
[7] P. Kulkarni, D. Whalley, G. Tyson, and J. Davidson. Exhaus-
tive optimization phase order space exploration. In Proceed-
ings of the Fourth Annual IEEE/ACM International Sympo-
sium on Code Generation and Optimization, March 26-29
2006.
[8] Prasad Kulkarni, David Whalley, Gary Tyson, and Jack
Davidson. In search of near-optimal optimization phase or-
derings. In LCTES '06: Proceedings of the 2006 ACM SIG-
PLAN/SIGBED conference on Language, compilers and tool
support for embedded systems, pages 8392, New York, NY,
USA, 2006. ACM Press.
[9] L. Almagor, Keith D. Cooper, Alexander Grosul, Timothy J.
Harvey, Steven W. Reeves, Devika Subramanian, Linda Tor-
czon, and Todd Waterman. Finding effective compilation se-
quences. In LCTES '04: Proceedings of the 2004 ACM SIG-
PLAN/SIGBED conference on Languages, Compilers, and
Tools for Embedded Systems, pages 231239, New York,
NY, USA, 2004. ACM Press.
[10] T. Kisuki, P. Knijnenburg, , and M.F.P. O'Boyle. Combined
selection of tile sizes and unroll factors using iterative com-
pilation. In Proc. PACT, pages 237246, 2000.
[11] F. Agakov, E. Bonilla, J. Cavazos, B. Franke, G. Fursin,
M. F. P. O'Boyle, J. Thomson, M. Toussaint, and C. K. I.
Williams. Using machine learning to focus iterative opti-
mization. In Proceedings of the International Symposium on
Code Generation and Optimization, pages 295305, Wash-
ington, DC, USA, 2006. IEEE Computer Society.
[12] Deborah L. Whiteld and Mary Lou Soffa. An approach
for exploring code improving transformations. ACM Trans.
Program. Lang. Syst., 19(6):10531084, 1997.
[13] Min Zhao, Bruce Childers, and Mary Lou Soffa. Predict-
ing the impact of optimizations for embedded systems. In
LCTES '03: Proceedings of the 2003 ACM SIGPLAN confer-
ence on Language, compiler, and tool for embedded systems,
pages 111, New York, NY, USA, 2003. ACM Press.
[14] Min Zhao, Bruce R. Childers, and Mary Lou Soffa. A model-
based framework: An approach for prot-driven optimiza-
tion. In Proceedings of the international symposium on Code
generation and optimization, pages 317327, Washington,
DC, USA, 2005.
[15] W. Kelly and W. Pugh. A framework for unifying reordering
transformations. Technical Report CS-TR-3193, 1993.
[16] Shun Long and Grigori Fursin. A heuristic search algorithm
based on unied transformation framework. In 7th workshop
on High Performance Scientic and Engineering Computing,
Norway, 2005. IEEE Computer Society.
[17] F. Bodin, T. Kisuki, P.M.W. Knijnenburg, M.F.P. O'Boyle,
and E. Rohou. Iterative compilation in a non-linear opti-
misation space. Proc. Workshop on Prole and Feedback
Directed Compilation, 1998.
[18] P.M.W. Knijnenburg, T. Kisuki, K. Gallivan, and M.F.P.
O'Boyle. The effect of cache models on iterative compi-
lation for combined tiling and unrolling. In Proc. FDDO-3,
pages 3140, 2000.
[19] Prasad Kulkarni, Steve Hines, Jason Hiser, David Whalley,
Jack Davidson, and Douglas Jones. Fast searches for effec-
tive optimization phase sequences. In Proceedings of the
ACM SIGPLAN '04 Conference on Programming Language
Design and Implementation, June 2004.
[20] Prasad Kulkarni, Steve Hines, David Whalley, Jason Hiser,
Jack Davidson, and Douglas Jones. Fast and efcient
searches for effective optimization-phase sequences. ACM
Trans. Archit. Code Optim., 2(2):165198, 2005.
[21] K. Cooper, A. Grosul, T. Harvey, S. Reeves, D. Subrama-
nian, L. Torczon, and T. Waterman. Acme: Adaptive com-
pilation made efcient. In Proceedings of the ACM SIG-
PLAN/SIGBED Conference on Languages, Compilers, and
Tools for Embedded Systems, pages 6978, June 15-17 2005.
[22] M. E. Benitez and J. W. Davidson. A portable global opti-
mizer and linker. In Proceedings of the SIGPLAN'88 con-
ference on Programming Language Design and Implemen-
tation, pages 329338. ACM Press, 1988.
[23] D. Burger and T. Austin. The SimpleScalar tool set, version
2.0. SIGARCH Comput. Archit. News, 25(3):1325, 1997.
[24] Matthew R. Guthaus, Jeffrey S. Ringenberg, Dan Ernst,
Todd M. Austin, Trevor Mudge, and Richard B. Brown.
MiBench: A free, commercially representative embedded
benchmark suite. IEEE 4th Annual Workshop on Workload
Characterization, December 2001.
[25] W. Peterson and D. Brown. Cyclic codes for error detec-
tion. In Proceedings of the IRE, volume 49, pages 228235,
January 1961.
[26] Paul E. Black. Simulated annealing. Dictionary of Algo-
rithms and Data Structures adopted by the U.S. National
Institute of Standards and Technology, December 2004.
http://www.nist.gov/dads/HTML/simulatedAnnealing.html.
[27] Melanie Mitchell. An Introduction to Genetic Algorithms.
Cambridge, Mass. MIT Press, 1996.
[28] A. Nisbet. Genetic algorithm optimized parallelization.
Workshop on Prole and Feedback Directed Compilation,
1998.
[29] Mark Stephenson, Saman Amarasinghe, Martin Martin, and
Una-May O'Reilly. Meta optimization: improving compiler
heuristics with machine learning. In Proceedings of the ACM
SIGPLAN 2003 conference on Programming Language De-
sign and Implementation, pages 7790. ACM Press, 2003.

Ansor : Generating High-Performance Tensor Programs for Deep LearningLianmin Zheng, Chenfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen, Joseph Gonzalez, Ion Stoica

Deep Learning System Stack

Introducing Compiler
𝑑𝑒𝑛𝑠𝑒!,#=&$𝑑𝑎𝑡𝑎!,$×𝑤𝑒𝑖𝑔ℎ𝑡#,$𝑟𝑒𝑙𝑢𝑏,𝑜=max(𝑑𝑒𝑛𝑠𝑒!,#,0)
A dense layer with ReLU activation•Math expression: •Declaration:dense(o, b) +=data(i, b) * weight(i, o);relu(o, b) =max(dense(o, b), 0.0)Halide
dense = compute(shape, lambdab, o: sum(data[b,i] *weight[o,i], i))relu  =compute(shape, lambdab, o: max(dense[b,o], 0.0))TVM
Billions of possible implementations for it!
Related Work on GeneratingHigh-Performance Tensor Programs
TVM's Approach
...
ManualTemplatefori.0 in range(  ):for j.0 inrange(  ):for k.0 inrange(  ):fori.1 in range(  ):for j.1 inrange(  ):C[...] +=A[...] * B[...]fori.2 inrange(  ):for j.2 inrange(  ):D[...] =max(C[...], 0.0)
???????
Parameter SearchAutoTVM: Template-guided searchUse templatesto define the search space for every operator
Drawbacks•Not fully-automated   -> Requires huge manual effort•Limited search space  -> Does not achieve optimal performance
TVM: An Automated End-to-End Optimizing Compiler for Deep Learning, OSDI 18
...
IncompleteProgramfori.0 in range(512):for j.0 inrange(512):D[...] =max(C[...], 0.0)How to build the next statement ?Candidate 1Candidate 2Candidate 3Candidate 4
Pruned
Pruned
KeptKept
Beam Search with Early Pruning
Learning to Optimize Halide with Tree Search and Random Programs, SIGGRAPH 19
Halide’s Auto-schedulerSequential Construction Based SearchUse beam search to generate the programs sequentiallyDrawbacks•Intermediate candidates are incomplete programs-> The cost model cannot do accurate prediction•Sequential order-> The error accumulates-> Limits the design of the search space
Challenges and our approachC1: How to build a large search space automatically?•Use a hierarchical search space
•Sample complete programs and fine-tune themC2: How to search efficiently?Fine-tuningBetter Programs
Low-level detail sampling
......fori.0 in range(64):for j.0 inrange(64):for k.0 inrange(512):fori.1 in range(8):for j.1 inrange(8):D[...] = ...
CompletePrograms
...for ...for ...for ...for ... ...for ...for ...for ...for ... ...for ...for ...for ...for ...
High-levelstructure generation? ?? ??
Challenges and our approach
•C3: How to allocate resource for many search tasks?•Utilize a task scheduler to prioritize important tasks 
Layer 1Layer 2
Layer 49Layer 50
Layer 3... Need to generate programs for all layers -> A lot of search tasks
System Overview
Deep Learning Models
Subgraph 1TaskSchedulerSubgraph 2Subgraph 3·· · 
Program SamplerSketch GenerationRandom Annotation
Performance TunerEvolutionary SearchLearned Cost Model
Intel CPUMeasurerARM CPUNVIDIA GPU·· · 
Partitioned subgraphs
One subgraph 
A batch of initial programs 
A batch of optimized programs 
Execution time of programs
Program Sampling
Deep Learning Models
Subgraph 1
TaskScheduler
Subgraph 2 Subgraph 3 ·· · 
Program Sampler
Sketch Generation Random Annotation
Performance Tuner
Evolutionary Search Learned Cost Model
Intel CPU
Measurer
ARM CPU NVIDIA GPU ·· · 
Partitioned subgraphs
One subgraph 
A batch of initial programs 
A batch of optimized programs 
Execution time of programs
•Goal: automatically construct a large search space and uniformly sample from the space•Approach•Two-level hierarchical search space: Sketch+ Annotation•Sketch: a few good high-level structures•Annotation: billions of low-level details
Program Sampling
ComputeDeclarationRule-basedSketch GenerationSketch 1Sketch 2... Random AnnotationCompletePrograms
•Sampling process:
Sketch Generation Examples 1/2
for i.0 in range(TILE_I0):
for j.0 in range(TILE_J0):
fori.1 in range(TILE_I1):
for j.1 in range(TILE_J1):
for k.0 in range(TILE_K0):
fori.2 in range(TILE_I2):
for j.2 in range(TILE_J2):
for k.1 in range(TILE_I1):
fori.3 in range(TILE_I3):
for j.3 in range(TILE_J3):
C[...] += A[...] * B[...]
fori.4 in range(TILE_I2 * TILE_I3):
for j.4 in range(TILE_J2 * TILE_J3):
D[...] = max(C[...], 0.0)
Generated	sketch	1
* The mathmeticalexpression:
!",$ =&'[",)]
 
,
	×	/[),$]
0",$ =max	(!",$ ,0.0)
where0≤",$,)<512
* The corresponding naïve program:
foriin range(512):
for j inrange(512):
for kinrange(512):
C[i, j] +=A[i, k] * B[k, j]
foriinrange(512):
for jinrange(512):
D[i, j] =max(C[i, j], 0.0)
* The corresponding DAG:
Example	Input	1:
A
B
C D
Derivation:
“SSRSRSS” multi-level tiling + fusion
Sketch Generation Examples 2/2

Random Annotation Examplesparalleli.0@j.0@i.1@j.1 in range(256):
for k.0 inrange(32):
fori.2 in range(16):
unroll k.1 inrange(16):
unrolli.3 in range(4):
vectorizej.3 inrange(16):
C[...] +=A[...] * B[...]
fori.4 inrange(64):
vectorizej.4 inrange(16):
D[...] =max(C[...], 0.0)
Sampled	 program	1		
paralleli.2 in range(16):
forj.2 in range(128):
for k.1 inrange(512):
for i.3 in range(32):
vectorizej.3 inrange(4):
C[...] +=A[...] * B[...]
paralleli.4 inrange(512):
for j.4 inrange(512):
D[...] = max(C[...], 0.0)
Sampled	 program	2
fori.0 in range(TILE_I0):
for j.0 inrange(TILE_J0):
fori.1 in range(TILE_I1):
for j.1 inrange(TILE_J1):
for k.0 inrange(TILE_K0):
fori.2 in range(TILE_I2):
for j.2 inrange(TILE_J2):
for k.1 inrange(TILE_I1):
fori.3 in range(TILE_I3):
for j.3 inrange(TILE_J3):
C[...] +=A[...] * B[...]
fori.4 inrange(TILE_I2 * TILE_I3):
for j.4 inrange(TILE_J2 * TILE_J3):
D[...] = max(C[...], 0.0)
Generated	sketch	1
paralleli.0@j.0@i.1@j.1 in range(256):
for k.0 inrange(32):
fori.2 in range(16):
unroll k.1 inrange(16):
unrolli.3 in range(4):
vectorizej.3 inrange(16):
C[...] +=A[...] * B[...]
fori.4 inrange(64):
vectorizej.4 inrange(16):
D[...] =max(C[...], 0.0)
Sampled	 program	1		
paralleli.2 in range(16):
forj.2 in range(128):
for k.1 inrange(512):
for i.3 in range(32):
vectorizej.3 inrange(4):
C[...] +=A[...] * B[...]
paralleli.4 inrange(512):
for j.4 inrange(512):
D[...] = max(C[...], 0.0)
Sampled	 program	2
fori.0 in range(TILE_I0):
for j.0 inrange(TILE_J0):
fori.1 in range(TILE_I1):
for j.1 inrange(TILE_J1):
for k.0 inrange(TILE_K0):
fori.2 in range(TILE_I2):
for j.2 inrange(TILE_J2):
for k.1 inrange(TILE_I1):
fori.3 in range(TILE_I3):
for j.3 inrange(TILE_J3):
C[...] +=A[...] * B[...]
fori.4 inrange(TILE_I2 * TILE_I3):
for j.4 inrange(TILE_J2 * TILE_J3):
D[...] = max(C[...], 0.0)
Generated	sketch	1
PerformanceFine-tuning
Deep Learning Models
Subgraph 1
TaskScheduler
Subgraph 2 Subgraph 3 ·· · 
Program Sampler
Sketch Generation Random Annotation
Performance Tuner
Evolutionary Search Learned Cost Model
Intel CPU
Measurer
ARM CPU NVIDIA GPU ·· · 
Partitioned subgraphs
One subgraph 
A batch of initial programs 
A batch of optimized programs 
Execution time of programs
Evolutionary Search•Random sampling does not guarantee the performance•Perform evolutionary search with learned cost model on sampled programs•mutation
•crossover+ =
•Randomly mutate tile size•Randomly mutate parallel/unroll/vectorize factor and granularity•Randomly mutate computation location
•Predict the score of each non-loop innermost statementfor i in range(10):for j in range(10):B[i][j] = A[i] * 2for i in range(10):C[i] = B[i][i] -3Statement B:Statement C: 
Example:
Cost = Cost of Statement B + Cost of Statement C
•Extract features for every non-loop innermost statement:•used cache lines, used memory, reuse distance, arithmetic intensity, ...•Train on-the-fly with measured programs (typically less than 30,000)
Learned Cost Model
TaskScheduler
Deep Learning Models
Subgraph 1
TaskScheduler
Subgraph 2 Subgraph 3 ·· · 
Program Sampler
Sketch Generation Random Annotation
Performance Tuner
Evolutionary Search Learned Cost Model
Intel CPU
Measurer
ARM CPU NVIDIA GPU ·· · 
Partitioned subgraphs
One subgraph 
A batch of initial programs 
A batch of optimized programs 
Execution time of programs
Task Scheduler•There are many subgraphs(search tasks) in a network•Example: ResNet-50 has 29 unique subgraphs after partition
•Predict each task’s impact on the end-to-end objective function•Using optimistic guess and similarity between tasks
Task 1 Task 2 Task 3
•Existing systems: sequential optimization with a fixed allocation 
•Our task scheduler: slice the time and prioritize important subgraphsTask 1Task 2Task 3Task 1Task 2Task 1Task 3Task 1
EvaluationResultsThree levels : single operator, subgraph and network
Single OperatorPlatform:Intel-Platinum 8124M (18 cores)Operators:conv1d (C1D), conv2d (C2D),conv3d (C3D), matmul (GMM)group conv2d (GRP),dilated conv2d (DIL)depthwise conv2d (DEP), conv2d transpose (T2D),capsule conv2d (CAP),matrix 2-norm (NRM)
Analysis: For most test cases, the best programs found by Ansor areoutside the search space of existing search-based frameworks.
Parallelize reduction loops
Unroll to simplify the multiplication of zeros in the strided caseExplore more tiling levels and computation locations
SubgraphPlatforms:"@C" for Intel CPU (8124M)"@G" for NVIDIA (V100)Subgraphs:ConvLayer = conv2d + bn + reluTBS = transpose + batch_matmul + softmax
Analysis:Comprehensive coverage of the search space gives 1.1–14.2×speedup against the best alternative.
Network
Analysis •Ansor performs best or equally the best in all test cases with up to 3.8xspeedup
Platforms:Intel CPU (8124M)NVIDIA GPU (V100)ARM CPU (A53)Networks: ResNet-50, Mobilenet V2, 3D-ResNet, DCGAN, BERT
Intel CPU
NetworkPlatforms:Intel CPU (8124M)NVIDIA GPU (V100)ARM CPU (A53)Networks: ResNet-50, Mobilenet V2, 3D-ResNet, DCGAN, BERT
Analysis •Ansor performs best or equally the best in all test cases with up to 3.8xspeedupNVIDIA GPU
NetworkPlatforms:Intel CPU (8124M)NVIDIA GPU (V100)ARM CPU (A53)Networks: ResNet-50, Mobilenet V2, 3D-ResNet, DCGAN, BERTARM CPU
Analysis •Ansor performs best or equally the best in all test cases with up to 3.8xspeedup•Ansor delivers portable performance
Ablation Study
Analysis•The most important factor is the search space•Fine-tuning improves the search results significantly•Task scheduler accelerates the search•Match the performance of AutoTVM with 10x less search time
AnsorAnsor w/o task schedulerAnsor w/o fine-tuning
Ansor w/o large search spaceBaseline (AutoTVM)
Summary•Search-based compilation enables to generate high-performance tensor programs for deep learning•Ansor introduces techniques to improve the search in three aspects:•Large search space•Efficient search algorithm•Smart search scheduling
•Thank you for listening•Email me to ask follow-up questions: lianminzheng@gmail.com

Rapidly Selecting Good Compiler Optimizations
using Performance Counters
John Cavazos1 Grigori Fursin2 Felix Agakov1 Edwin Bonilla1
Michael F.P. O’Boyle1 Olivier Temam2
Members of HiPEAC
1Institute for Computing Systems Architecture (ICSA)
School of Informatics, University of Edinburgh, UK
2ALCHEMY Group, INRIA Futurs and LRI, Paris-Sud University, France
Abstract
Applying the right compiler optimizations to a particu-
lar program can have a signiﬁcant impact on program per-
formance. Due to the non-linear interaction of compiler
optimizations, however, determining the best setting is non-
trivial. There have been several proposed techniques that
search the space of compiler options to ﬁnd good solutions;
however such approaches can be expensive. This paper pro-
poses a different approach using performance counters as a
means of determining good compiler optimization settings.
This is achieved by learning a model off-line which can then
be used to determine good settings for any new program.
We show that such an approach outperforms the state-of-
the-art and is two orders of magnitude faster on average.
Furthermore, we show that our performance counter-based
approach outperforms techniques based on static code fea-
tures. Using our technique we achieve a 17% improve-
ment over the highest optimization setting of the commer-
cial PathScale EKOPath 2.3.1 optimizing compiler on the
SPEC benchmark suite on a recent AMD Athlon 64 3700+
platform.
1 Introduction
Automatically selecting the best set of compiler op-
timizations for a particular program is a difﬁcult task
and there has been much previous work on automatically
searching for the best optimization settings [14, 22, 23, 29].
This previous work is based on iteratively enabling certain
optimizations, running the compiled program and, based on
its performance, deciding on a new optimization setting.
Pan et al. [22] introduce a new algorithm called combined
elimination (CE) that was shown to outperform all previ-
ous search-based techniques in ﬁnding good optimization
settings with considerably fewer evaluations.
However, these pure search or “orchestration” ap-
proaches do not use prior knowledge of the hardware, com-
piler, or program and instead attempt to obtain this knowl-
edge online. Thus, every time a new program is optimized,
the system starts with no prior knowledge. In our experi-
ments, this means on average over 600 evaluations (com-
pile + run) to tune an application. In contrast, the technique
presented in this paper uses knowledge about a program’s
behavior to automatically select the best optimizations with
as little as 3 program evaluations. Speciﬁcally, we use the
performance counter values collected from a few runs of
the program as input to an automatically constructed model
which outputs a probability distribution of good compiler
optimizations to use. Using dynamic knowledge of how a
particular program runs on a particular platform, on a set
of benchmark suites we are able to achieve the same per-
formance or better as combined elimination, two orders of
magnitude faster on average. Thus, obtaining knowledge a-
priori (involving a one-off-cost at the factory), we can sig-
niﬁcantly speedup the searching of good optimization se-
quences for any new program. In contrast, “pure search”
techniques always obtain knowledge online and must do so
for any new program they optimize.
Performance counters have been extensively used for
performance analysis in explaining program behavior [10,
3]. One of the ﬁrst papers to investigate how they could
be used systematically to select optimizations [24] showed
impressive performance gains. However, the heuristic used
was manually developed over a 12 month period using de-
tailed simulations. Furthermore, the optimizations selec ted
were also implemented by hand. A small change in the ar-
 0
 5
 10
 15
 20
 25
 30
 35
 40
L2_TCA
L1_TCH
L1_TCM
FP_OPS
FAD_INS
FML_INS
L1_TCA
L2_TCH
L1_ICR
L2_ICA
L1_ICA
L2_ICH
L1_ICH
L2_DCW
L2_DCR
L2_DCA
L1_DCA
L2_DCH
L1_DCH
TOT_CYC
RES_STL
VEC_INS
BR_INS
FP_INS
BR_MSP
BR_TKN
HW_INT
STL_ICY
L2_STM
L2_LDM
L1_STM
L1_LDM
TLB_TL
TLB_IM
TLB_DM
FPU_IDL
L2_TCM
L2_ICM
L2_DCM
L1_ICM
L1_DCM
Relative to Average Perf Cntrs 
       
181.mcf
Figure 1. Performance counter values for
181.mcfcompiled with -O0 relative to the aver-
age values for the entire set of benchmark suite
(SPECFP,SPECINT, MiBench, Polyhedron).
FAST
PC Model
 0
 0.2
 0.4
 0.6
 0.8
 1
 1.2
 1.4
 1.6
 1.8
 2
L2_TCAL1_TCHL1_TCMFP_OPSFAD_INS
FML_INSL1_TCAL2_TCH
L1_ICRL2_ICAL1_ICA
L2_ICHL1_ICHL2_DCW
L2_DCRL2_DCAL1_DCA
L2_DCHL1_DCHTOT_CYC
RES_STLVEC_INSBR_INSFP_INSTOT_INSBR_MSP
BR_TKNHW_INTSTL_ICY
L2_STML2_LDML1_STML1_LDMTLB_TLTLB_IM
TLB_DMFPU_IDLL2_TCM
L2_ICML2_DCML1_ICM
L1_DCM
Relative to −o0 Perf Cntrs 
       
181.mcf
Figure 2. Performance counter values of -Ofast
(FAST) and our scheme (PCModel) relative to -
O0 for each performance counter for 181.mcf.
chitecture would potentially require the entire process to be
repeated. In contrast, our scheme is entirely automatic. It
uses machine learning to automatically build a model which
maps performance counters to good optimization options
without human intervention and is thus portable.
The use of learned models to guide the selection of op-
timizations has also received recent attention [1, 5, 27].
Stephenson et al. [27], for instance, use a genetic program-
ming approach to automatically learn individual compiler
optimizations, such as the register allocation spill heuristic,
within the Trimaran compiler. In this paper, however, we
consider the problem of determining the best settings for a
large number of optimizations within a highly-tuned com-
mercial compiler, PathScale EKOPAth 2.3.1 [25] whose
performance is as good as or better than Intel 9.0. We show
signiﬁcant performance improvements over the highest op-
timization level for this compiler.
In our previous work [1], we used static code features
to obtain good optimizations for new programs being com-
piled. The static features were used to ﬁnd the most similar
program from a set of previously explored programs. This
was used for estimating a distribution of good sequences
for the matching program, from which optimizations to ap-
ply for the new program were drawn. The idea is that opti-
mizations which performed well on a “similar” (previously
explored) program will work well for the new program be-
ing compiled. This approach worked well on multimedia
kernels on embedded processors, but as we show in Sec-
tion 5.3 it performs poorly on larger general purpose appli-
cations. In fact, there is little or no performance improve-
ment over the highest optimization level provided by the
PathScale compiler. The main reason is that static code fea-
tures, which essentially characterize local code construc ts
such as loops, provide a poor global characterization once
aggregated over many such code sections. Furthermore,
code features are a poor mechanism to describe the dynamic
behavior of large control-ﬂow intensive programs. We show
that a performance counter-based scheme overcomes these
challenges. Using performance counters to select good op-
timizations is attractive as it exploits knowledge of the pro-
gram’s behavior without requiring knowledge of the pro-
gramming language syntax.
This paper is organized as follows. The next section
provides a motivating example showing that performance
counters can be used to select good optimizations. Sec-
tion 3 describes the performance counters used in this pa-
per and how they can characterize program behavior. This
section also includes a brief description of how we use a
simple modelling technique, logistic regression[4], to auto-
matically learn a global optimizing heuristic. Section 4 de -
scribes the experimental setup and is followed in Section 5
by the experimental results and their analysis. This includes
a comparison between our scheme and both combined elim-
ination and random selection. We also compare against a
static feature-based modelling approach. This is followed
by a summary of related work and concluding remarks.
2 Motivation
The information obtained from performance counters is
a compact summary of a program’s dynamic behavior. In
particular, they summarize important aspects of a program’s
performance, e.g., cache misses or ﬂoating point unit uti-
lization. Our approach uses this information to automati-
cally select compiler optimization settings likely to improve
program performance.
This section looks at just one of the programs evaluated
in this paper to illustrate how such performance counters
can be used to select good compiler optimizations. As the
performance counter values are related to actual program
performance, they can be used by a modelling technique
Compiler Evaluations Execution time Speedup
-O0 1 40.2 1
-Ofast 1 32.3 1.24
CE 240 18.0 2.23
PC Model 3 17.2 2.33
Figure 3. The execution time and speedup
over -O0 for the best compiler setting ob-
tained using different schemes on 181.mcf.
The column labelled Evaluations shows the
number of times the code must be run to
achieve this level of performance by each
scheme.
(described in the next section) to select good optimization
settings. Our model examines performance counter values
of a new program and, using prior knowledge from previ-
ously examined programs, determines the optimization set-
ting most likely to result in a speedup and improved perfor-
mance counter values.
2.1 Performance Counters
Figure 1 illustrates the use of performance counters.
This graph shows the performance counter values for the
181.mcfbenchmark relative to the average values for the
SPEC benchmark suite. These values were collected on an
AMD Athlon 64 3700+ processor using the commercially
available PathScale optimizing compiler. What is immedi-
ately apparent is that 181.mcf is an unusual program -
having a much greater number of memory accesses per in-
struction than average - up to 38 times more in the case of
L2 store misses (L2
STM). A learned model should iden-
tify this and enable transformations that reduce the impact
of cache accesses.
Figure 2 shows the performance counter values after
applying two optimization schemes, -Ofast (FAST), the
highest optimization setting available with PathScale and
the setting found by our performance counter model ( PC-
Model). PCModel is able to signiﬁcantly improve the use
of the L1 and L2 cache. This is shown in the third to the
last and the last bars of Figure 2 in the columns labelled
L1
TCM (L1 total cache miss) and L2 TCA (L2 total cache
accesses). These bars show that the model is able to reduce
the number of L1 cache misses by 20% which has the effect
of reducing the number of L2 accesses by 20%. The -Ofast
setting, on the other hand, has no effect on these values.
2.2 Performance
Figure 3 shows the speedups obtained by -Ofast, CE,
and PCModel over -O0. -Ofast is able to achieve a 1.24
speedup over -O0 while PCModel gives a speedup of 2.33,
i.e., a speedup of 1.88 over -Ofast. It achieves this perfor-
mance improvement with just 3 evaluations using a learned
model trained ofﬂine. If we compare this to the performance
of combined elimination (CE) [22], our scheme gives a
slightly greater improvement. Furthermore, combined elim-
ination requires 240 evaluations on this benchmark. Using
our trained model, we are able to achieve greater perfor-
mance improvement over the state-of-the-art, approaching
two orders of magnitude fewer evaluations.
2.3 T ransformations
Examining the transformations selected by PCModel, it
is apparent that locality enhancing loop optimizations hav e
been enabled. In effect, the automatically generated model
has learned that 181.mcfhas a problem with its memory
usage and has selected transformations to overcome this.
If, however, we examine the transformations selected by -
Ofast and PCModel we see that they both enable the loop
optimizer -LNO which is aimed at exploiting locality. On
closer inspection, the major difference is that our model de-
cides to turn on the -m32 ﬂag, i.e., generate 32-bit code
rather than the default 64-bit for the AMD. It does this be-
cause the number of data cache accesses and branch instruc-
tions are high. Figure 1 shows that the data cache accesses
are relatively high (L1
DCM and L2 DCM) for this bench-
mark. Also, looking at the number of branch instructions
(BR
INS) we see it is more than 2.5 times the average.
According to the AMD compiler manual [2] the -m32
option “can improve performance if your program has lots
of variables of the type long and/or pointers. As these data-
types are 32-bit in x86, this switch will reduce the memory
footprint of your program.” We note that -m32 is only use-
ful for a few programs, and our model decides based on
the dynamic characteristics (performance counters) of each
program when it should be applied. In fact, many manu-
facturers include the -m32 option in the SPEC “peak” ﬂags
for some codes when using PathScale. By examining the
code of 181.mcfwe see it accesses its main data structure
through pointers in a loop. Also, it has a large number of
branches executed proportional to the number of total in-
structions due to small tight loops. Reducing the pointer
size, by using -m32, reduces the number of I-cache data
misses dramatically for 181.mcf as can be seen in Fig-
ure 2.
Thus, our model has learned that data cache misses and
branch instructions (via the performance counter data) are
the critical characteristics of this program and suggests t he
compiler convert pointers from 64-bit to 32-bit, because 64-
bit pointers are reducing the effective cache capacity and
memory bandwidth. This demonstrates the strength of au-
tomatic model construction. It has no a priori human bias
 0
 0.5
 1
 1.5
 2
 2.5
L2_TCA
L1_TCH
L1_TCM
FP_OPS
FAD_INS
FML_INS
L1_TCA
L2_TCH
L1_ICR
L2_ICA
L1_ICA
L2_ICH
L1_ICH
L2_DCW
L2_DCR
L2_DCA
L1_DCA
L2_DCH
L1_DCH
TOT_CYC
RES_STL
VEC_INS
BR_INS
FP_INS
BR_MSP
BR_TKN
HW_INT
STL_ICY
L2_STM
L2_LDM
L1_STM
L1_LDM
TLB_TL
TLB_IM
TLB_DM
FPU_IDL
L2_TCM
L2_ICM
L2_DCM
L1_ICM
L1_DCM
Relative to Average Perf Cntrs 
       
SPEC FP
Figure 4. Performance counter values for SPEC
FP average values for the entire benchmark
suite compiled with -O0.
 0
 0.2
 0.4
 0.6
 0.8
 1
 1.2
 1.4
 1.6
 1.8
L2_TCA
L1_TCH
L1_TCM
FP_OPS
FAD_INS
FML_INS
L1_TCA
L2_TCH
L1_ICR
L2_ICA
L1_ICA
L2_ICH
L1_ICH
L2_DCW
L2_DCR
L2_DCA
L1_DCA
L2_DCH
L1_DCH
TOT_CYC
RES_STL
VEC_INS
BR_INS
FP_INS
BR_MSP
BR_TKN
HW_INT
STL_ICY
L2_STM
L2_LDM
L1_STM
L1_LDM
TLB_TL
TLB_IM
TLB_DM
FPU_IDL
L2_TCM
L2_ICM
L2_DCM
L1_ICM
L1_DCM
Relative to Average Perf Cntrs 
       
MiBench
Figure 5. Performance counter values for
MiBench average values for the entire bench-
mark suite compiled with -O0.
about what are important program characteristics or trans-
formations – it learns solely based on empirical evidence.
This example show that performance counters can be
used by a model to select optimizations that improve per-
formance by examining the values of the counters. Further-
more, it can ﬁnd good sequences rapidly. The next sec-
tions describe the performance counters used in this paper
in greater detail and explains the technique used to automat-
ically build an automatic optimizing heuristic.
3 Optimization selection based on
performance counters
This section ﬁrst looks at the performance counters used
in this paper and illustrates that they can be used to char-
acterize well-known properties of SPEC FP and MiBench.
This is followed by a description of the modelling technique
we use which is based on logistic regression. This is a stan-
dard machine learning technique which can learn whether
an optimization is good or bad for a certain set of perfor-
mance counter values and associates a probability with this
decision.
3.1 Dynamic characterization of program
behavior using performance counters
Modern processors are often equipped with a special set
of registers that allow for measuring performance counter
events with no disruption to the running program. These
events can describe several characteristics of the running
program, such as, cache hits and misses and branch pre-
diction statistics. On the AMD Athlon, there are 4 regis-
ters for measuring performance counter events, but up to
60 different events can be measured. It is possible to col-
lect anywhere between 4 and 60 types of events per run by
multiplexing the use of the special registers. Since we aim
at broadly characterizing the program behavior rather than
studying a particular performance phenomenon, we have
collected all 60 events. Using multiplexing, we collected
all 60 performance counter values of a benchmark in 3 runs.
The performance counters used in this study are shown in
Table 1. In order to use the collected statistics as inputs to
our model, we normalized the value of each performance
counter by TOT
INS, the total number of instructions exe-
cuted. Normalizing the performance counters is important
since it allows us to generalize across different benchmarks
regardless of how long each benchmark executes.
Table 1 also presents the average values for each counter
across our benchmark suites. As can be seen, some counter
events are relatively common, such as L1 data cache misses
(L1
DCA) while others are relatively rare, such as instruc-
tion TLB misses (TLB IM).
To see how the performance counters can character-
ize program behavior, we have examined two benchmark
suites, SPEC FP and MiBench, out of the four benchmark
suites we evaluated (SPEC INT, SPEC FP, MiBench, Poly-
hedron). We plotted the performance counter values for
each of these suites relative to the average values for the
entire collection of benchmarks. These values are shown in
Figures 4 and 5 where the values are collected when com-
piled with -O0. It is interesting to see that MiBench exer-
cises the cache hierarchy much less than SPEC FP which
can be seen by the much larger number of L2 cache misses
(L2
TCA) for SPEC FP. Also, MiBench has a relatively
large number of branches compare to SPEC FP, almost
twice as many, and many of the branches in MiBench are
mispredicted, while more branches for SPEC FP codes are
easier to predict (BR
MSP).
These results are not surprising and have been discussed
in prior work. However, these graphs show that there are
important aspects of program behavior that can be captured
using performance counters. Furthermore, this informatio n
can serve as an input to a model that selects compiler op-
Performance Counter Name Meaning Average Values
HW INT Hardware interrupts 0.000
RES STL Cycles stalled on any resource 0.660
STL ICY Cycles with no instruction issue 0.035
TOT CYC Total cycles 1.099
TOT INS Instructions completed 1.000
VEC INS Vector/SIMD instructions 0.017
Floating Point Instruction Statistics
FAD INS, FML INS, FP INS, FP OPS, FPU IDL Floating point: Adds, Multiplies, Total Insns, Total
Ops, Cycles Idle
0.030, 0.036, 0.066, 0.066, 0.473
Branch Instruction Statistics
BR INS, BR MSP, BR TKN Branch instructions, Cond. Branches Mispredicted,
Cond. Branches Taken
0.047, 0.002, 0.035
Level 1 Cache Statistics
DCA, DCH, DCM Data Cache: Accesses, Hits, Misses 0.475, 0.472, 0.004
ICA, ICH, ICM, ICR Instruction Cache: Accesses, Hits, Misses, Reads 0.316, 0.315, 0.0006, 0.315
LDM, STM Load Misses, Store Misses 0.0015, 0.0016
TCA, TCH, TCM Total Cache: Accesses, Hits, Misses 0.789, 0.790, 0.004
Level 2 Cache Statistics
DCA, DCH, DCM, DCR, DCW Data Cache: Accesses, Hits, Misses, Reads, Writes 0.003, 0.003, 0.0005, 0.0015, 0.0016
ICA, ICH, ICM Instruction Cache: Accesses, Hits, Misses 0.0006, 0.0006, 0.000002
LDM, STM Load Misses, Store Misses 0.0004, 0.00008
TCA, TCH, TCM Total Cache: Accesses, Hits, Misses 0.004, 0.003686, 0.0005
TLB Statistics
TLB DM Data translation lookaside buffer misses 0.0002
TLB IM Instruction translation lookaside buffer misses 0.000001
TLB TL Total translation lookaside buffer misses 0.0002
Table 1. Performance counters used. The ﬁrst column lists th e performance counter acronyms, the
second column gives a description, and the third column give s the average counter values normal-
ized by total instructions executed across the entire set of benchmark suites.
timizations. Using these statistics our model can learn to
apply optimizations that will reduce the impact of cache
misses for SPEC FP or branch misses for MiBench. The
next section shows how we can use this information to se-
lect good optimizations automatically using machine learn-
ing.
3.2 Automatically learning a good model
The goal of model construction is to learn a mapping
x → t between a set of performance counters featuresx, and
a set of good optimizations t. Here x is a vector of the 60
normalized performance counters values, and t is a vector
mask indicating which transformations are used in the se-
quence (i.e., each vector entry corresponds to a transforma-
tion, and t corresponds to a sequence of 0/1, 1 meaning the
transformation is used). The goal of the model is to pre-
dict the best possible t for a program described by features
x. We emphasize this is not the phase-ordering problem.
That is, our models do not ﬁnd the best order in which to
apply transformations. While this is an interesting problem,
most compilers do not allow changing arbitrarily the order
optimizations are applied.
In the following paragraphs, we ﬁrst describe how the
training set required for ofﬂine training is collected whic h
is then used to construct a model. We then describe how
this model is used.
Training set
At ﬁrst, a large number (500) of transformation sequences
are randomly sampled and applied to each program and the
speedup for each of these sequences is recorded. Also, we
require three additional runs of the program to collect the
performance counter values. The runs where the speedup,
relative to -Ofast, is smaller than 1 are ﬁltered out, and
the remaining data forms the training set. Then, we use
the standard Leave One Out Cross-Validationprocedure for
evaluating our models. That is, the models are trained on
N −1 benchmarks and tested on the Nth benchmark that
has been left out. In our experiments, N = 57, therefore our
models were trained with 56 ×500 = 28000 training data
points. The cost of obtaining 28000 training points is ex-
pensive, however it is a one-off-cost incurred ofﬂine at the
factory.
Model construction
The model is now built using the training set. The predic-
tive modelling process is summarized in Figure 6. We use
a probabilistic approach for predictive modelling, where we
determine for each optimization ti the probability pi that it
should be evaluated. The technique used is logistic regres-
sion [4]. Intuitively, it attempts to ﬁnd the set of perfor-
mance counter values for which enabling the transforma-
tion ti leads to improved performance in the training set and
also determines when disabling a transformation is prefer-
able. In effect, it tries to draw a hyperplane in the multi-
dimensional hardware counter space between those occa-
sions when the transformation is best enabled and those
occasions when it is best disabled. Borderline cases near
the hyperplane have a probability, p, around 0.5 associ-
ated with them. Those which should be deﬁnitely enabled
have p >> 0.5 and those which should be deﬁnitely dis-
abled have p << 0.5. This is a standard machine learning
technique and is computationally inexpensive - see [4] for a
more detailed description. Note that gathering training data
and construction of the model is an ofﬂine process, that is,
it would take place “at the factor” before the compiler is
shipped to the costumer.
Using the model
Given a new target benchmark, we ﬁrst extract the perfor-
mance counter features x by running the benchmark. This
requires 3 runs of the benchmark. This features vector is
then fed as input to our trained models which then outputs
a probability pi for each transformation ti showing whether
each transformation should be applied or not. We then sam-
ple from this probability distribution to generate a suitab le
compiler optimization setting. The program is then opti-
mized based on the transformations selected and the new
speedup is measured as shown in Figure 6. An advantage
of this technique is that we can sample as many times as
we wish to generate different settings. We later show in
Section 5.1 that very few samples are required to achieve
good performance. Furthermore, increasing the number of
samples which are evaluated increases the performance ob-
tained.
4 Experimental Setup
This section brieﬂy describes the experimental setup.
First, the hardware platform, OS, and optimizing compiler
are described. Second, we describe the benchmarks and the
optimizations available for selection.
4.1 Platform
We perform all experiments on a cluster of AMD Athlon
64 3700+ 2.4GHz processors with an L1 cache of 64KB,
an L2 cache of 1MB, 3GB of memory, and each running
Mandriva Linux 2006. We use the latest PAPI 3.2.1 hard-
ware counter library [21] and PAPIEx 0.99rc2 tool to collect
hardware performance counters for the benchmarks. Ta-
ble 1 has a brief description of all the counters we use. PA-
PIEx works in the multiplexing mode allowing us to col-
lect a large number of counters in one run. We collect
performance counters using level -O0 so that characteris-
tics of a benchmark are not masked by higher optimization
levels (e.g., -Ofast). We use the latest open-source com-
mercial PathScale EKOPath Compiler 2.3.1 [25] with the
-Ofast ﬂag, which we refer to as our baseline. This com-
piler is tuned to AMD processors and on average performs
similarly or better than the Intel 9.0 compilers on the same
platform.
In order to evaluate the stability of our measurements,
we have executed multiple runs of each benchmark using
-Ofast and have found there to be very little variance in
the execution time, on average less than 0.3%.
4.2 Benchmarks
We evaluate our approach on widely-used benchmark
suites written in C, C++, Fortran and Fortran 90. These are
SPEC 95 FP (ref dataset), SPEC 2000 FP and INT (train
dataset), Polyhedron 2005 [18], and MiBench [13]. We
used train inputs for the SPEC 2000 benchmarks due to
the large number of experiments we ran, on average over
1000 for each benchmark. SPEC and Polyhedron bench-
marks are relatively large programs used for performance
evaluation of servers and for comparison of performance of
various compilers on these servers. These benchmarks are
used by PathScale to tune their compiler suite. MiBench is
a free, commercially representative embedded benchmark
suite consisting of a large number of applications and ker-
nels. We believe that all these programs cover a large vari-
ety of different dynamic behaviors. In the experimental sec-
tion, we partition the benchmark results into SPEC, which
includes all SPEC benchmarks and Others, which includes
MiBench and Polyhedron.
4.3 T ransformations
The PathScale EKOPath compiler suite includes a
PathOpt tool that randomly selects from a variety of global
compiler settings. PathOpt iterates for a user-speciﬁed
amount of evaluations and is used to iteratively ﬁnd the best
optimization settings for a program on a targeted platform.
We select 121 ﬂags that are known to inﬂuence performance
and use PathOpt to apply 500 random settings of these ﬂags
on all benchmarks. We also allow our model to generate
sequences with optimizations that are mutually exclusive,
e.g, different unroll factors. When this happens, the com-
piler simply ignores all but the last option. We could have
easily adapted our models to handle this directly (e.g., us-
ing a soft-max approach where the optimization with the
 
 
New program  
predicted set of best 
transformations 
Architecture 
   (t A
1  t A
2  … t A
M ) 
   (t B
1  t B
2  … t B
M ) 
TB (baseline option) X 
 
performance  
counter features 
for the baseline 
PCModel
  
(b)  Inference using a predictive model.  Given a new benchmark, we first  extract per formance 
counter features. These features are then fed into our trained models which then output a set of 
transformation sequences to apply to the new benchmark. 
 
… 
best 
speedups 
 
… 
sA 
sB 
Programs (training set)  
  bes t  set of transformations 
(option sequences) 
Architecture     (t  
1  t 1
2  … t 1
M ) 
  (t 2
1  t 2
2  … t 2
M ) 
… 
   (t N
1  t N
2  … t N
M ) 
… 
TB (baseline option) 
s1 
 
s2 
sN 
Speedups 
 
… 
X 
 
 performance  
counter features 
for the baseline 
PCModel  
(a) Summary of the predictive modelling  procedure. We use the features x, the transfor mations 
t, and (implicitly) the speed -ups {s} for constructing the training data < x , t > . We then 
evaluate the mapping from the performance counters to the transformation sequences x 
 
 t  by 
fitting a probabilistic model to the training set. 
 
Figure 6. Training and using the models.
highest probability is chosen), however this would make it
harder to compare with combined elimination and random.
The speedups for each setting were used along with perfor-
mance hardware counters as training data for our logistic
regression model.
5 Experimental results
In this section we evaluate our proposed technique in a
number of ways. Initially, we report the performance im-
provement achieved by our technique compared to com-
bined elimination and the number of evaluations each
scheme needs to achieve such a level of performance. Since
we use a probabilistic model we have generated our re-
sults for multiple trials. For random selection, we have also
generated our results for multiple trials. This is then fol-
lowed by a comparison of the performance of a number
of different schemes with respect to the number of eval-
uations available. Here, we compare combined elimina-
tion (CE), our models (PCModel), and a random selec-
tion (RAND) approach which generates random transfor-
mation sequences. RAND is implemented using a pseudo-
random number generator to choose which optimizations to
apply in the sequence. Each optimization has a .5 proba-
bility of being used in each sequence. We then provide a
study examining the use of static code features as a means
of selecting compiler optimizations. Finally, we perform
some analysis to evaluate the most important performance
counter features. All results in this section, unless other -
wise stated, are relative to -Ofast, the highest optimizati on
level available in PathScale. Note, all of our techniques
(CE,PCModel,Random) start with the -Ofast as their initial
sample, so none of our techniques can perform worse than
-Ofast. We emphasize that our models are built using leave-
one-out cross validation, so the models are not trained using
any information of the programs it is optimizing.
CE
PC Model
 1
 1.1
 1.2
 1.3
 1.4
 1.5
 1.6
 1.7
 1.8
 1.9
average
301.apsi
300.twolf
256.bzip2
197.parser
191.fma3d
189.lucas
188.ammp
186.crafty
183.equake
181.mcf
179.art
178.galgel
177.mesa
175.vpr
173.applu
172.mgrid
171.swim
168.wupwise
164.gzip
146.wave5
141.apsi
125.turb3d
110.applu
107.mgrid
104.hydro2d
103.su2cor
102.swim
101.tomcatv
Relative to −ofast
       
Combined Elimination (CE) and PC Model
Figure 7. The speedup of Combined Elimina-
tion (CE) versus our model (PCModel) for the
SPEC benchmarks relative to the performance
of -Ofast. The number of evaluations used by
PCModel is limited to 25 while CE uses on aver-
age 609.
CE
PC Model
 1
 1.1
 1.2
 1.3
 1.4
 1.5
 1.6
 1.7
 1.8
 1.9
average
tfft
test_fpu
gsm
CRC32
adpcm_c
sha
rijndael
rnflow
protein
pix
stringsearch
patricia
dijkstra
mdbx
linpk
gas_dyn
fatigue
drag
doduc
lame
jpeg_d
jpeg_c
channel
susan_s
susan_e
susan_c
bitcount
air
ac
Relative to −ofast
       
Combined Elimination (CE) versus PC Model
Figure 8. The speedup of Combined Elimination
(CE) versus our model (PCModel) for the Other
(non-SPEC) benchmarks relative to the perfor-
mance of -Ofast. The number of evaluations
used by PCModel is limited to 25 while CE uses
on average 609.
5.1 Speedup and number of evaluations
for PCModel and CE
Figure 7 shows the speedups of our model and combined
elimination relative to -Ofast on the SPEC benchmarks. PC-
Model achieves a speedup of 1.17 on average compared to
1.09 by CE. In this example, we have set the number of
evaluations selected by our model to 25 evaluations. How-
ever, CE needs on average 609 evaluations of the program
to achieve this. We note that both PCModel and CE can
ﬁnd signiﬁcant improvements for several programs. For in-
stance, PCModel ﬁnds improvements of 10% or more over
-Ofast for half (14 of 28) of the SPEC benchmarks. Thus we
are able to achieve better performance than CE with consid-
erable fewer evaluations. In fact, our model achieves the
same performance as CE on the SPEC benchmarks in only
3 iterations. Figure 8 shows a similar comparison for the
other benchmarks. In this case both schemes give approx-
imately the same performance improvement of 1.17. Fig-
ure 9 however, shows that CE needs a large number of eval-
uations to achieve this performance level, ranging form 240
evaluations up to 1550 with an average of 609 needed.
It is not suprising that the CE algorithm requires a large
number of evaluations. The CE algorithm ﬁrst evaluates
the effect of each of the optimizations by turning them off
one at a time. This is 121 evaluations for the entire set of
optimizations we explored. Then it “combines” the knowl-
edge gathered by these initial evaluations to choose opti-
mizations that lead to better improvement when turned off.
After several iterations of turning off single optimizations it
converges to a setting where no additional ﬂags turned off
improve performance. We refer the reader to the combined
elimination paper [22] for further details of the algorithm.
5.2 Performance versus number of
evaluations for diﬀerent schemes
To give a different view of how our model performs we
considered the performance it achieves as a function of the
number of evaluations we select from it. Figure 10 shows
the performance achieved averaged across all the bench-
marks versus the number of evaluations allowed. We also
compare our approach relative to combined elimination and
random selection. For each technique, the more evaluations
allowed the better the performance achieved. Our model
achieves the same performance after 60 evaluations as ran-
dom does after 200 evaluations. Surprisingly, random se-
lection does quite well! Other papers [15, 1] have reported
on the excellent performance of pure random search.
CE’s initial step involves turning off each optimization
in turn which gives a small speedup of 1.04 across the ﬁrst
121 evaluations. Only after all 121 optimizations have been
evaluated does it improve its behavior by “combining” its
results. However, after 200 evaluations it only achieves the
same performance as that achieved by our model after 2
evaluations. Random selection achieves this in just 10 eval-
uations. RAND and CE both found an improvement of 17%
over -Ofast using 200 and 60 iterations, respectively. CE
ﬁnds less improvement at 12%. We note that for these ex-
periments the CE algorithm was run to completion. On the
other hand, given the nature of the RAND and PCModel
algorithms, we could continue to construct sequences with
these algorithms until we are satisﬁed with the optimizied
performance of our application. Both, RAND and PCModel
whould reach the same maximum available speedup in the
limit, however PCModel should reach that speedup sooner.
We believe RAND and PCModel perform well in our ex-
 0
 200
 400
 600
 800
 1000
 1200
 1400
 1600
PCModel-----------averagetffttest_fpugsmCRC32adpcm_csharijndaelrnflowproteinpixstringsearchpatriciadijkstramdbxlinpkgas_dynfatiguedragdoduclamejpeg_djpeg_cchannelsusan_ssusan_esusan_cbitcountairac301.apsi300.twolf256.bzip2197.parser191.fma3d189.lucas188.ammp186.crafty183.equake181.mcf179.art178.galgel177.mesa175.vpr173.applu172.mgrid171.swim168.wupwise164.gzip146.wave5141.apsi125.turb3d110.applu107.mgrid104.hydro2d103.su2cor102.swim101.tomcatv
Iteration Count
       
Combined Elimination and PC Model Iterations
Figure 9. The number of evaluations of the CE algorithm per be nchmark. The mininum evaluations
is 240 (181.mcf), maximum is 1562 (fatigue), and the average is 609. We compared CE to PCModel
with an evaluation count 25 (far right).
periments because the search space has many good points.
However, in certain scenarios it is possible that CE could
outperform RAND (and models trained with RAND data,
such as PCModel). CE’s main goal is to eliminate opti-
mizations that degrade performance. If we encouter an op-
timization space with many optimizations that degrade per-
formance of a benchmark, it would take a large number of
iterations for RAND to construct a sequence with none (or
few) of these degrading optimizations.
5.3 Static versus dynamic features
As mentioned in the introduction, we previously used a
model-based approach [1] to characterize programs using
static code features. In this section, we quantitatively co m-
pare the merit of static (code) versus dynamic (performance
counter) features using the same number of evaluations. In
previous work [1], we applied this approach to the UTDSP
benchmarks, which are small embedded kernels often con-
taining only a single loop nest with afﬁne array accesses.
Here, we extracted the same code features for several of the
SPEC INT benchmarks, which are large control-ﬂow inten-
sive programs. We also used a K-nearest neighbor approach
and built IID distributions similar to what we used in our
previous work to be as fair as possible. We choose neigh-
bors using either static code features or dynamic perfor-
mance counter features and drew samples from the neigh-
bor’s IID distribution to apply to a new program.
The results of this experiment are shown in Figure 11.
The static code feature-based approach ﬁnds some improve-
ment in 4 of the 7 programs achieving an average speedups
of just 1.01 when compared to -Ofast. In contrast, our
performance counter model signiﬁcantly improves over the
static code feature-based approach giving a speedup aver-
age of 1.08 over -Ofast. The graph shows performance
counters are signiﬁcantly better for characterizing large
programs with complex control ﬂow, e.g., 181.mcf and
186.crafty.
5.4 Analysis of the importance of the
performance counters
The goal of this analysis is to understand which perfor-
mance counters are most important for predicting good opti-
mization sequences. The fundamental objective in this con-
text is mutual information between a subset of the perfor-
mance counters and good optimization sequences (for deﬁ-
nition of mutual information, see e.g. [19]). Our goal was to
maximize the mutual information for subsets of the retained
features.
In general, it is intractable to compute the mutual infor-
mation exactly, therefore approximations need to be consid-
 1
 1.05
 1.1
 1.15
 1.2
 1.25
 1  10  100  1000
Speedup
Number of Evaluations
Performance versus Number of Evaluations (PC Model, CE, RAND)
PC Model
RAND
CE
Figure 10. The speedup of Combined Elimina-
tion (CE), PCModel, and random selection aver-
aged across all benchmarks versus the number
of program evaluations used.
STATIC
DYNAMIC
 1
 1.05
 1.1
 1.15
 1.2
 1.25
 1.3
 1.35
average
300.twolf
256.bzip2
197.parser
186.crafty
181.mcf
175.vpr
164.gzip
Relative to −ofast
       
Static vs Dynamic Features
Figure 11. Performance of SPEC INT 2000
Benchmarks using static code features and dy-
namic features.
ered. For our analysis we applied a novel subset-selection
approach which greedily maximizes the Gaussian approx-
imation of the mutual information. For training data con-
taining one good optimization sequence per benchmark, we
found that over 95% of the total information (if all features
are retained) was typically contained in just 15 performance
counters. In contrast, conventional approximations disre -
garding interactions between the inputs (e.g. [28]) would
typically require twice as many features to retain the same
amount of information, which in our case is not much bet-
ter than random selection (see Figure 12 left). Interestingly,
while the choice of the informative features generally de-
pends on the training data, we found that there was a lot of
overlap between the performance counters found for various
good transformation sequences (typically with one or two
unique features per training set). Figure 12 right shows in-
formative performance counters for training sequences con-
structed with the CE algorithm.
6 Related Work
Parello et al. [24] presents a systematic, but manual it-
erative approach for program optimization using dynamic
features. At each iteration, performance counters are used
to identify a performance anomaly of a program and a set
of program transformations is suggested to solve this prob-
lem. Then the transformed program is run again to detect
further performance anomalies. The process is manual and
can take several weeks per benchmark. Our technique is
fully automated and can generate heuristics to predict good
optimizations in seconds.
In the area of predictive modelling, Zhao et al. use man-
ually constructed cost/beneﬁt models to predict whether to
apply PRE or LICM[34]. They achieve 1% to 2% improve-
ment over always applying an optimization, but at a cost of
greatly increasing compilation time (by up to 68%). Their
models appear to be quite complicated and have to be manu-
ally constructed. Our models, on the other hand, are simple
and automatically constructed using machine learning.
During the past several years, the beneﬁts of iterative
compilation have been widely reported [15, 8, 9, 11, 14].
Iterative compilation is able to ﬁnd optimization sequences
that out-perform the highest optimization settings in com-
mercial compilers and when applied to library subroutines
they ﬁnd solutions that compare favorably with highly-
optimized hand-tuned vendor libraries [31, 12, 26, 30].
However, iterative compilation requires searching a com-
binatorially large space deﬁned by the optimizations of in-
terest. This search can take several days to weeks depend-
ing on the running time of the program, speed of the com-
piler and target architecture, and thoroughness of the search.
There have been a number of papers focusing on reducing
the cost of iterative optimization.
Kulkarni et al. [16] introduce the VISTA system, an
interactive compilation system which concentrates on re-
ducing compilation time. This system uses a variety of
techniques to reduce the number of different compilation
sequences evaluated. Their system stores a representation
of each program compiled then detects when identical or
equivalent code has been generated and only executes code
that has not been previously generated. They also prohibit
speciﬁc optimizations and optimization sequences from be-
ing performed if it is unlikely that these optimizations wil l
not change the code. These techniques are only effective
when programs are extremely small, such as those used in
embedded domains.
Kulkarni et al. [17] also introduce techniques to allow
exhaustive enumeration of all distinct function instances
that would be produced from the different phase-orderings
of 15 optimizations. This exhaustive enumeration allowed
them to construct probabilities of enabling/disabling int er-
2 4 6 8 10 12 140
2
4
6
8
10
12
14
Retained PCs
I({x},t)
Feature Selection by Information Maximization
Gaussian I({x},t)
Gaussian I(xi,t)
Quantized I(xi,t)
Uniform
Most Informative Performance Counters
1) L1 TCA 2) L1 DCH 3) TLB DM
4) BR INS 5) RES STL 6) TOT CYC
7) L2 ICH 8) VEC INS 9) L2 DCH
10) L2 TCA 11) L1 DCA 12) HW INT
13) L2 TCH 14) L1 TCH 15) BR MS
Figure 12. Analysis of the importance of the performance counters. The data contains one good opti-
mization sequence per benchmark. Left: Approximate amount of information retained in the selected
features. Top curve: our method; lower curves: simpler heur istics. Conventional approach [28]
which uses binning and ignores input interactions (dash-do tted curve) is comparable with the uni-
form random selection. Right: 15 most informative performance counters sorted in the orde r of the
descending importance.
actions between the different optimization passes. Using
these probabilities, they constructed a probabilistic batch
compiler that determined which optimization should be ap-
plied next depending on which one had the highest proba-
bility of being enabled. This method however does not con-
sider the beneﬁts each optimization can potentially provide.
In contrast, we train our model based on the impact of opti-
mizations applied, and therefore our technique learns which
optimizations are beneﬁcial to apply to “unseen” programs
with similar characteristics.
Another system to speedup iterative compilation was
recently introduced by Cooper et al. called ACME [7].
ACME utilizes a technique called estimated virtual execu-
tion (EVE) which estimates changes to the execution counts
of basic blocks when an optimization that changes the CFG
is performed. This is done by inserting a pass into the opti-
mization sequence after each invocation of a CFG-changing
optimization and ﬁxes the basic block counts based on the
changes. They can then model the advantages and disad-
vantages of applying optimizations by multiplying the num-
ber of instructions in a block by its dynamic frequency then
summing over all blocks. This technique can estimate the
performance of very simple models, however this method
is vastly inaccurate when estimating the performance of to-
day’s complex machines, especially out-of-order issue pro-
cessors. It also requires signiﬁcant changes to be made to
each optimization in a compiler.
Triantafyllis et al. [29] develop an alternative approach
to reduce the total number of evaluations of a new program.
Here the space of compiler options is examined off-line on
a per function basis and the best performing ones are clas-
siﬁed into a small tree of compiler options. When compil-
ing a new program, the tree is searched by compiling and
executing the best path in the tree. As long as the best se-
quences can be categorized into a small tree, this proves to
be a highly effective technique.
Pan et al. [22] develop an algorithm called combined
elimination which selectively turns off optimizations until
the best performance is found for a new application. This al-
gorithm was compared to other algorithms for tuning com-
piler settings [14, 29] and was shown to achieve the same or
better performance as these algorithms while dramatically
reducing the tuning time. In a recent paper [23], they parti-
tion a program into tuning sections and then use combined
elimination to ﬁnd the best combination of optimizations
for each of these tuning sections. By using rating methods,
the authors can evalute several different optimization set -
tings for a tuning section during one run of the program.
They are able to reduce the time to ﬁnd good optimization
settings from hours to minutes. However, the techniques of
partitioning a program and rating methods are orthogonal to
the particular search algorithm used, therefore we could as
well use our models to ﬁnd good sequences even faster.
Yotovet al. [32] describe a model-based approach to op-
timize BLAS libraries that can is effective as empirical eval-
uation. In a later paper [33], they reﬁne analytical models
based on the results of the empirical search for the ATLAS
library. A local neighborhood search around the best found
points is used to further improve the solutions and perform
comparable to the ATLAS global search strategy. However,
the analytical models require manual tuning and are com-
plicated to design.
Cooper et al. [8] use genetic algorithms to solve the
compilation phase-ordering problem. They were concerned
with ﬁnding “good” compiler optimization sequences that
reduced code size. Their technique was successful at reduc-
ing code size by as much as 40%. Unfortunately, their tech-
nique is application-speciﬁc. That is, a genetic algorithm
has to retrain for each program to decide the best optimiza-
tion sequence for that program.
Several researchers have also looked at using machine
learning to construct heuristics that control a single opti -
mization. Stephenson et al. [27] used genetic program-
ming (GP) to tune heuristic priority functions for three com-
piler optimizations: hyperblock selection, register allo ca-
tion, and data prefetching within the Trimaran’s IMPACT
compiler. For two optimizations, hyperblock selection and
data prefetching, they achieved signiﬁcant improvements.
However, a closer look at the results indicate that all the
improvement was obtained from the initial population indi-
cating that these two pre-existing heuristics were not well
tuned. For the third optimization, register allocation, th ey
were able to achieve on average only a 2% increase over the
manually tuned heuristic.
Cavazos et al. [5] describe an idea of using supervised
learning to control whether or not to apply instruction
scheduling. They induced heuristics that used features of
a basic block to predict whether scheduling would beneﬁt
that block or not. Using the induced heuristic, they were
able to reduce scheduling effort by as much as 75% while
still retaining the effectiveness of scheduling all blocks.
Monsifrot et al. [20] use a classiﬁer based on decision
tree learning to determine which loops to unroll. They
looked at the performance of compiling Fortran programs
from the SPEC benchmark suite using g77 for two different
architectures, an UltraSPARC and an IA64. They showed
an improvement over the hand-tuned heuristic of 3% and
2.7% over g77’s unrolling strategy on the IA64 and Ultra-
SPARC, respectively.
These results highlight the diminishing results obtained
when only controlling a single optimization and highlight
the need to control the application of multiple compiler op-
timizations.
Recently, Cavazos et al. [6] describe using static code
features and supervised learning to control several opti-
mizations to apply during method compilation in a JIT com-
piler. Since Java methods are typically small, static code
features were successfully used to characterizing them. In
future work, we will compare static versus dynamic fea-
tures to predict which optimizations to apply to local code
segments.
7 Conclusions
In this paper we address the problem of predicting good
compiler optimizations by using performance counters to
automatically generate compiler heuristics. We do this by
using machine learning techniques that predict good code
transformations to apply given a program’s performance
counter features. Our technique automates the tuning pro-
cess and eliminates the need for manual experimentation.
Additionally, the heuristics induced by these techniques can
generalize to programs that have not been seen before. Us-
ing performance counters allows us to apply transforma-
tions that will beneﬁt the program being compiled while
avoiding optimizations that will degrade performance. Us-
ing our models, we can achieve a 10% average speedup over
the highest optimization setting in the PathScale compiler
on SPEC benchmarks on a recent AMD Athlon much faster
than the current state-of-the-art pure search techniques.
References
[1] F. Agakov, E. Bonilla, J. Cavazos, B. Franke, G. Fursin,
M. OBoyle, J. Thomson, M. Toussaint, and C. Williams.
Using machine learning to focus iterative optimization. In
Proceedings of the International Symposium on Code
Generation and Optimization, pages 295–305, 2006.
[2] AMD. Compiler usage guidelines for 64-bit operating
systems on amd64 platforms. http://www.amd.com/us-
en/assets/content
type/white papers and tech docs/32035.pdf,
2006.
[3] R. Azimi, M. Stumm, and R. W. Wisniewski. Online
performance analysis by statistical sampling of
microprocessor performance counters. In ICS ’05:
Proceedings of the 19th annual international conference on
Supercomputing, pages 101–110, New York, NY , USA,
2005. ACM Press.
[4] C. M. Bishop. Neural Networks for Pattern Recognition.
Oxford University Press, Oxford, UK, 1996.
[5] J. Cavazos and J. E. B. Moss. Inducing heuristics to decid e
whether to schedule. In Proceedings of the ACM SIGPLAN
’04 Conference on Programming Language Design and
Implementation, pages 183–194, Washington, D.C., June
2004. ACM Press.
[6] J. Cavazos and M. O’Boyle. Method-speciﬁc dynamic
compilation using logistic regression. In Proceedings of the
ACM SIGPLAN ’06 Conference on Object Oriented
Programming, Systems, Languages, and Applications,
Portland, Or., October 2006. ACM Press.
[7] K. D. Cooper, A. Grosul, T. J. Harvey, S. Reeves,
D. Subramanian, L. Torczon, and T. Waterman. Acme:
adaptive compilation made efﬁcient. In Proceedings of the
2005 ACM SIGPLAN/SIGBED Conference on Languages,
Compilers, and Tools for Embedded Systems, pages 69–77,
New York, NY , USA, 2005. ACM Press.
[8] K. D. Cooper, P. J. Schielke, and D. Subramanian.
Optimizing for reduced code space using genetic
algorithms. In Workshop on Languages, Compilers, and
Tools for Embedded Systems, pages 1–9, Atlanta, Georgia,
July 1999. ACM Press.
[9] K. D. Cooper, D. Subramanian, and L. Torczon. Adaptive
optimizing compilers for the 21st century. Journal of
Supercomputing, 23(1):7–22, August 2002.
[10] J. Dean, J. E. Hicks, C. A. Waldspurger, W. E. Weihl, and
G. Z. Chrysos. Proﬁleme : Hardware support for
instruction-level proﬁling on out-of-order processors. In
International Symposium on Microarchitecture, pages
292–302, 1997.
[11] B. Franke, M. O’Boyle, J. Thomson, and G. Fursin.
Probabilistic source-level optimisation of embedded
programs. In Proceedings of the 2005 ACM
SIGPLAN/SIGBED Conference on Languages, Compilers,
and Tools for Embedded Systems, pages 78–86, New York,
NY , USA, 2005. ACM Press.
[12] M. Frigo and S. G. Johnson. The design and
implementation of FFTW3. Proceedings of the IEEE,
93(2):216–231, 2005. special issue on ”Program
Generation, Optimization, and Platform Adaptation”.
[13] M. R. Guthaus, J. S. Ringenberg, D. Ernst, T. M. Austin,
T. Mudge, and R. B. Brown. Mibench: A free,
commercially representative embedded benchmark suite. In
IEEE 4th Annual Workshop on Workload Characterization,
Austin, TX, December 2001.
[14] M. Haneda, P. M. W. Knijnenburg, and H. A. G. Wijshoff.
Automatic selection of compiler options using
non-parametric inferential statistics. In Proceedings of the
International Conference on Parallel Architectures and
Compilation Techniques, pages 123–132, Washington, DC,
USA, 2005. IEEE Computer Society.
[15] T. Kisuki, P. M. W. Knijnenburg, and M. F. P. O’Boyle.
Combined selection of tile sizes and unroll factors using
iterative compilation. In Proceedings of the International
Conference on Parallel Architectures and Compilation
Techniques, page 237, Washington, DC, USA, 2000. IEEE
Computer Society.
[16] P. Kulkarni, S. Hines, J. Hiser, D. Whalley, J. Davidson, and
D. Jones. Fast searches for effective optimization phase
sequences. In Proceedings of the ACM SIGPLAN ’04
Conference on Programming Language Design and
Implementation, pages 171–182, New York, NY , USA,
2004. ACM Press.
[17] P. A. Kulkarni, D. B. Whalley, G. S. Tyson, and J. W.
Davidson. Exhaustive optimization phase order space
exploration. In Proceedings of the International Symposium
on Code Generation and Optimization, pages 306–318,
Washington, DC, USA, 2006. IEEE Computer Society.
[18] P. S. Ltd. http://www.polyhedron.com, 2006.
[19] R. J. McEliece. The Theory of Information and Coding.
Addison-Wesley, 1977.
[20] A. Monsifrot, F. Bodin, and R. Quiniou. A machine
learning approach to automatic production of compiler
heuristics. In AIMSA ’02: Proceedings of the 10th
International Conference on Artiﬁcial Intelligence:
Methodology, Systems, and Applications, pages 41–50.
Springer-Verlag, 2002.
[21] P. Mucci. Papi – the performance application programmi ng
interface. http://icl.cs.utk.edu/papi/index.html, 2000.
[22] Z. Pan and R. Eigenmann. Fast and effective orchestrati on
of compiler optimizations for automatic performance
tuning. In Proceedings of the International Symposium on
Code Generation and Optimization, pages 319–332, 2006.
[23] Z. Pan and R. Eigenmann. Fast automatic procedure-leve l
performance tuning. In Proceedings of the International
Conference on Parallel Architectures and Compilation
Techniques, Seattle, WA, September 2006. IEEE Computer
Society.
[24] D. Parello, O. Temam, A. Cohen, and J.-M. Verdun.
Towards a systematic, pragmatic and architecture-aware
program optimization process for complex processors. In
SC ’04: Proceedings of the 2004 ACM/IEEE conference on
Supercomputing, page 15, Washington, DC, USA, 2004.
IEEE Computer Society.
[25] I. PathScale. http://www.pathscale.com, 2006.
[26] M. Puschel, J. Moura, J. Johnson, D. Padua, M. Veloso,
B. Singer, J. Xiong, F. Franchetti, A. Gacic, Y . V oronenko,
K. Chen, R. W. Johnson, and N. Rizzolo. Spiral: Code
generation for dsp transforms. Proceedings of the IEEE,
93(2):232–275, 2005. special issue on ”Program
Generation, Optimization, and Platform Adaptation”.
[27] M. Stephenson, S. Amarasinghe, M. Martin, and U.-M.
O’Reilly. Meta optimization: Improving compiler heuristics
with machine learning. In Proceedings of the ACM
SIGPLAN ’03 Conference on Programming Language
Design and Implementation, pages 77–90, San Diego, Ca,
June 2003. ACM Press.
[28] M. Stephenson and S. P. Amarasinghe. Predicting unroll
factors using supervised classiﬁcation. In Proceedings of
the International Symposium on Code Generation and
Optimization, pages 123–134, 2005.
[29] S. Triantafyllis, M. Vachharajani, N. Vachharajani, and D. I.
August. Compiler optimization-space exploration. In
Proceedings of the International Symposium on Code
Generation and Optimization, pages 204–215, Washington,
DC, USA, 2003. IEEE Computer Society.
[30] R. Vuduc, J. W. Demmel, and J. A. Bilmes. Statistical
models for empirical search-based performance tuning. Int.
J. High Perform. Comput. Appl., 18(1):65–94, 2004.
[31] R. C. Whaley and J. J. Dongarra. Automatically tuned
linear algebra software. In SC ’98: Proceedings of the 1998
ACM/IEEE conference on Supercomputing, pages 1–27,
Washington, DC, USA, 1998. IEEE Computer Society.
[32] K. Yotov, D. Padua, K. Pingali, P. Stodghill, and P. Wu. A
comparison of empirical and model-driven optimization. In
Proceedings of the ACM SIGPLAN ’03 Conference on
Programming Language Design and Implementation, pages
63–76, San Diego, Ca, June 2003. ACM Press.
[33] K. Yotov, K. Pingali, and P. Stodghill. Think globally,
search locally. In ICS ’05: Proceedings of the 19th annual
international conference on Supercomputing, pages
141–150, New York, NY , USA, 2005. ACM Press.
[34] M. Zhao, B. R. Childers, and M. L. Soffa. A model-based
framework: an approach for proﬁt-driven optimization. In
Proceedings of the International Symposium on Code
Generation and Optimization, pages 317–327, 2005.

A Flexible Approach to Autotuning
Multi-Pass Machine Learning Compilers
Phitchaya Mangpo Phothilimthana, Amit Sabne, Nikhil Sarda, Karthik Srinivasa Murthy, Y anqi Zhou,
Christof Angermueller, Mike Burrows, Sudip Roy, Ketan Mandke, Rezsa Farahani, Y u Emma Wang, Berkin Ilbeyi,
Blake Hechtman, Bjarke Roune, Shen Wang, Y uanzhong Xu, and Samuel J. Kaufman∗
Google, ∗University of Washington
Email: mangpo@google.com
Abstract—Search-based techniques have been demonstrated
effective in solving complex optimization problems that arise in
domain-speciﬁc compilers for machine learning (ML). Unfortu-
nately, deploying such techniques in production compilers is im-
peded by two limitations. First, prior works require factorization
of a computation graph into smaller subgraphs over which search
is applied. This decomposition is not only non-trivial but also
signiﬁcantly limits the scope of optimization. Second, prior works
require search to be applied in a single stage in the compilation
ﬂow, which does not ﬁt with the multi-stage layered architecture
of most production ML compilers.
This paper presents XTAT, an autotuner for production ML
compilers that can tune both graph-level and subgraph-level
optimizations across multiple compilation stages. XTAT applies
XTAT-M, a ﬂexible search methodology that deﬁnes a search
formulation for joint optimizations by accurately modeling the
interactions between different compiler passes. XTAT tunes tensor
layouts, operator fusion decisions, tile sizes, and code generation
parameters in XLA, a production ML compiler, using various
search strategies. In an evaluation across 150 ML training and
inference models on Tensor Processing Units (TPUs) at Google,
XTAT offers up to 2.4 × and an average 5% execution time
speedup over the heavily-optimized XLA compiler .
Keywords-compiler, autotuning, machine learning
I. I NTRODUCTION
Machine learning (ML) compilers solve multiple optimiza-
tion problems to translate an ML program, typically repre-
sented as a tensor computation graph, to an efﬁcient executable
for a hardware target. Recent works have demonstrated that
search-based techniques can be used to solve many of these
problems effectively [1]–[10]. However, production ML com-
pilers (e.g., XLA [11] and Glow [12]) still rely on heuristics
to solve these problems quickly, albeit often sub-optimally.
Current search-based techniques [1]–[10] have at least one
of the two key shortcomings that prevent them from being
deployed in production ML compilers.
First, they rely on the assumption that performance-critical
optimization decisions are localized within a subgraph and
hence can be made independently from the rest of the graph
[1]–[8]. This is often not the case. For example, decisions
to fuse1 tensor operations affect memory requirements, which
1When operators are fused, intermediate tensors can be used by the
consuming operator directly without saving them to the slow main memory.
in turn affect decisions to rematerialize2 tensors in different
portions of the graph. Furthermore, the subgraph-focused solu-
tions assume that they can easily partition a tensor computation
graph into suitable subgraphs. However, ﬁnding an optimal
partitioning for a particular optimization task is a non-trivial
combinatorial optimization problem by itself. A common
strategy is to partition a graph according to the neural network
layers [3], [5], ignoring cross-layer optimization opportunities.
We empirically observed a regression of up to 2.6× and 32%
on average across 150 ML models by limiting fusions in XLA
to be within layers. Furthermore, prior approaches [1]–[10]
have primarily studied inference graphs. Training graphs, on
the other hand, can be up to two orders of magnitude larger
than inference graphs in terms of number of nodes, rendering
subgraph decomposition less effective.
Second, the optimizations enabled in prior works [1]–
[10] are applied at the same stage in the compilation ﬂow,
speciﬁcally in the loop transformation stage. However, this is
impractical in production compilers because compiler transfor-
mations are organized as passes to reduce complexity, and have
strict ordering constraints. For example, in the XLA compiler,
tensor layout assignment occurs before operator fusion. This
is required because operator fusion’s goal is to reduce memory
trafﬁc, which is impossible to estimate without layouts. The
automatic cross-replica sharding [13] happens between the
layout and fusion passes. It requires layouts to calculate
parallelization overheads correctly, and must execute before
the fusion pass because otherwise the distribution granularity
becomes coarse, lowering performance gains. Therefore, joint
optimization of layout and fusion cannot occur at the same
compilation stage. To perform joint layout-fusion optimization
using the previous research approaches, all intermediate trans-
formations must be co-optimized too, which does not scale in
practice.
We next discuss our approach to overcome these short-
comings. To address the issue of the optimization scope, we
develop an autotuner for ML compilers that supports tuning
decisions made at both whole graph and subgraph levels.
Speciﬁcally, we implement XTAT (pronounced “stat”), an
autotuner for the XLA compiler.XTAT can tune tensor layouts
2Rematerialization reduces memory usage by recomputing a tensor when
it is needed instead of saving it in memory.
1
2021 30th International Conference on Parallel Architectures and Compilation Techniques (PACT)
978-1-6654-4278-7/21/$31.00 ©2021 IEEE
DOI 10.1109/PACT52795.2021.00008
6HDUFK&RQ
)RUPXODWLRQ
RI6LPXODWHG
$QQHDOLQJRU
(YROXWLRQDU\
6HDUFK
;7$7
(YDOXDWRU
/&0
+DUGZDUH
&RQILJ
&RVW
*UDSK([WUDFWRU
;/$
/RZOHYHO&RGH
*HQHUDWRU
7HQVRU)ORZ
3\7RUFK
3URJUDP
738
*UDSKOHYHO
2SWLPL]DWLRQV
/D\RXW$VVLJQPHQW
2SHUDWRU)XVLRQ
&URVV5HSOLFD
6KDUGLQJ
5HPDWHULDOL]DWLRQ
$OJHEUDLF
6LPSOLILFDWLRQHWF
.HUQHOOHYHO
2SWLPL]DWLRQV
7LOLQJ9HFWRUL]DWLRQ
)ODJVHWF
;7$7
RSWLPL]DWLRQ
JUDSK
FRQILJ
%HVW
&RQILJ
7XQH
&RPSLOHU
(YDOXDWRU
+DUGZDUH
&RVW0RGHO
VHDUFKVSDFH
GHIDXOWFRQILJ
RXWSXWJUDSK
JUDSK
FRQILJ FRVW
;7$70EDVHG
IRUPXODWLRQRIVHDUFK
VFKHGXOHDQGVHDUFK
VWUDWHJ\
;7$70EDVHG
IRUPXODWLRQRIWKH
FRQILJXUDWLRQVIRU
FRPSLOHURSWLPL]DWLRQV
Fig. 1: Overview ofXTAT, an autotuner for XLA.XTAT takes as inputs: (a) a program/graph to optimize, (b) a formulation of
the search strategy inXTAT-M, and (c) a formulation of the compiler optimization conﬁgurations inXTAT-M. XTAT currently
supports tuning optimization passes highlighted in red.XTAT uses the learned cost model and/or hardware to evaluate the
performance of different optimization conﬁgurations and outputs the best conﬁguration found.
and fusion decisions at the graph level, and tune tile sizes3 and
critical code generation parameters at the kernel (subgraph)
level for TPUs [14], [15]. Employing search at the graph level
is challenging due to search spaces that are exponentially large
in the number of nodes. Therefore, it is important to ensure
that the search space for an optimization pass is well designed,
as the design can greatly affect the quality of the ﬁnal solution
found by a search method. To this end, we present effective
search spaces for optimizations tuned byXTAT.
To tune multiple compiler optimizations at different stages,
we develop a ﬂexible methodology to apply search to multiple
compiler passes, called XTAT-M. The complexity of multi-
stage joint optimizations tuning comes from the fact that
decisions made in one optimization pass affect input graphs
to subsequent passes and, thereby, their conﬁgurations and
search spaces. To address this complexity,XTAT-M deﬁnes
conﬁguration-update speciﬁcations, which co-relate the state
of the intermediate graph left behind by one compiler op-
timization to a partial solution for a subsequent compiler
optimization. Through such speciﬁcations, XTAT-M allows
composing search-based strategies in certain passes (e.g., for
layout and fusion) with existing heuristic-based strategies in
intermediate passes (e.g., for cross-replica sharding). Addi-
tionally, XTAT-M enables ﬂexibility to trade off time spent
searching for a solution and solution optimality through a
conﬁgurable search schedule. This ﬂexibility is critical to
achieve the best performance given a time budget. Lastly,
XTAT-M allows us to apply a wide range of search techniques
including exhaustive search, simulated annealing, evolutionary
search, model-based optimization, and reinforcement learning.
To summarize this paper’s contributions:
• We developXTAT-M, a novel methodology to formulate
search for multiple optimizations at different stages of an
ML compiler with a ﬂexible search schedule.
• We build theXTAT autotuner based onXTAT-M to tune
tensor layouts, operator fusion decisions, tile sizes, and
3A tile of tensor is processed at a time to effectively utilize fast memory
(e.g., scratchpad and cache).
code generation parameters in XLA, as shown in Fig. 1.
• XTAT is the ﬁrst autotuner that can jointly tune multiple
optimizations at different stages of an ML compiler.
• XTAT handles much larger search spaces compared to
prior works since XTAT optimizes at both graph and
subgraph levels. For instance, the number of valid fusion
conﬁgurations alone for the EfﬁcientNet training model
with approximately 40,000 nodes is 240,000. In contrast,
prior works [1]–[8] tune one subgraph, which typically
has fewer than 10 nodes, at a time.
• We demonstrate how to incorporate advanced techniques
such as a learned cost model and various search strategies
into XTAT to reduce autotuning time.
• We evaluateXTAT on 150 ML models (comprising both
training and inference models from Google’s production
and research workloads) on TPUs and achieve signiﬁcant
improvement: up to 2.4× speedup and 5% on average
over the heavily-optimized XLA compiler.
II. XTAT-M:M ETHODOLOGY &F ORMULA TIONS
A. Overview
We deﬁne a conﬁguration on a graph for an optimization
pass as a collection of per-node conﬁgurations that control how
the pass transforms each node in the graph. For example, con-
sider the pass that determines a tensor layout (i.e., a physical
ordering of tensor dimensions in memory). A conﬁguration for
this pass is a collection of physical layouts assigned to tensors
at each node in the graph.
Searching for the best conﬁguration for a compiler opti-
mization involves (i) exploration of candidate conﬁgurations,
(ii) application of the optimization pass according to the
candidates, and (iii) evaluation of the output graphs. Imple-
menting these steps for a single optimization is relatively
straightforward, while the quality of results depends on the
expressiveness and compactness of the search space as well
as the capability of the search technique.
However, searching for optimal conﬁgurations for multi-
ple optimizations requires that the search satisﬁes a critical
2
condition: the search space and the conﬁguration chosen for
an optimization pass must be consistent with decisions made
in prior passes. For example, consider Fig. 2 with two opti-
mization passes,layout assignment(which determines tensor
layouts) and operator fusion (which fuses several operators
together). Since the performance of a fused node depends on
the layouts of the relevant tensors, the decisions in the fusion
pass must take into account the layouts decided in the prior
pass. Further, the search space for the fusion pass (i.e., the
set of node conﬁgurations to consider for search) depends on
the layout assignment since the layout pass may transform the
graph. As shown in Fig. 2, two layout conﬁgurations lead to
different input graphs for the fusion pass: one contains acopy
node, while the other does not; as a result, the fusion search
space for the left graph must contain conﬁgurations forcopy,
but the search space for the right must not.
How can we orchestrate searches for joint optimizations to
satisfy the above condition? To do so, we identify two key
properties that a search should implement. To describe these
properties, consider the scenario where an optimization pass,
say A, is applied before another optimization pass, sayB.
1) The ordering between the produced graphs is
g
A(conﬁgA)
−−−−−−→ g′ B(conﬁgB)
−−−−−−→ g′′. This ordering implies
that: (i) the graphg′ is a result of applying a candidate
conﬁguration conﬁgA selected for A and (ii) the search
space of candidate conﬁgurations and a selected
candidate conﬁgB for B is based ong′.
2) All conﬁgurations are well-formed; a conﬁguration
conﬁgB contains valid conﬁgurations for all conﬁgurable
nodes in g′. Further, when conﬁgA is changed, causing
graph g′ to change, conﬁgB must be updated to be
compatible with g′ using the best nodes’ conﬁgurations
observed so far forB. This ensures that the search forB
does not begin the exploration from scratch.
To realize this orchestration, we develop a generic method-
ology XTAT-M to formulate the interactions between multi-
stage compiler optimizations, and search strategies. Our pro-
posed search methodology is applicable to compilers that
apply multiple optimization passes in sequence.
B. Methodology
The search formulation ofXTAT-M is displayed in Fig. 3.
We model the search process as a sequence of search steps.
Each search step takes a set of current candidates C and
produces a new set of candidatesC′for the next round. When
tuning n optimizations, a candidatec in C captures n graphs
(c.graphs) andn conﬁgurations (c.conﬁgs), wherec.graphs[id]
and c.conﬁgs[id] are an input graph and a conﬁguration for an
optimization pass id respectively. As color coded in Fig. 3,
XTAT-M allows one to conﬁgure its routines to implement
a desired search schedule (red) and search strategy (green),
and tailor the conﬁguration-update procedures to speciﬁc op-
timizations (purple). Each search step (SearchSteplines 8–16)
performs the following actions:
Ôôô
͞ʾ̶ˀ̶ʽ˂͟
ŎøŒđÔŋø
͞ʽʾ˄͟
îĲĬű
͞ʾ̶˄͟
īÔŷ
͞ˀ̶ʽ˂̶˄͟
͟͞
͜ʼ͝
̶˄͟
͜ʼ̶ʽ͝
͜ʽ̶ʼ͝
͜ʼ̶ʽ̶ʾ͝
͜ʽ̶ʼ̶ʾ͝
͜ʼ̶ʾ̶ʽ͝
͜ʾ̶ʼ̶ʽ͝
͜ʾ̶ʽ̶ʼ͝
͜ʽ̶ʼ̶ʾ͝
͜ʼ̶ʾ̶ʽ͝
Ôôô
͞ʾ̶ˀ̶ʽ˂͟͜ʽ̶ʼ̶ʾ͝
ŎøŒđÔŋø
͞ʽʾ˄͟͜ʼ͝
îĲĬű
͞ʾ̶˄͟͜ʽ̶ʼ͝
īÔŷ
͞ˀ̶ʽ˂̶˄͟͜ʼ̶ʾ̶ʽ͝
îĲŋŸ
͞ʾ̶ˀ̶ʽ˂͟͜ʼ̶ʽ̶ʾ͝
ʽ̶ʼ̶ʾ͝
ʽ
ʼ ʼ
TÔŸĲŞř
ŒŒĔČĬīøĬř
aŋøŎÔřĲŎ
;ŞŒĔĲĬ
Ôôô
͞ʾ̶ˀ̶ʽ˂͟
ŎøŒđÔŋø
͞ʽʾ˄͟
îĲĬű
͞ʾ̶˄͟
īÔŷ
͞ˀ̶ʽ˂̶˄͟
͟͞
͜ʼ͝
̶͟
͜ʼ̶ʽ͝
͜ʽ̶ʼ͝
͜ʼ̶ʽ̶ʾ͝
͜ʽ̶ʼ̶ʾ͝
͜ʼ̶ʾ̶ʽ͝
͜ʾ̶ʼ̶ʽ͝
͜ʾ̶ʽ̶ʼ͝
͜ʽ̶ʼ̶ʾ͝
͜ʼ̶ʾ̶ʽ͝
Ôôô
͞ʾ̶ˀ̶ʽ˂͟͜ʽ̶ʼ̶ʾ͝
ŎøŒđÔŋø
͞ʽʾ˄͟͜ʼ͝
îĲĬű
͞ʾ̶˄͟͜ʽ̶ʼ͝
īÔŷ
͞ˀ̶ʽ˂̶˄͟͜ʼ̶ʾ̶ʽ͝
͟͜ʽ̶ʼ
ʽ
˄͟͜ʼ̶ʾ
ʼ
Ôôô
ŎøŒđÔŋø
îĲĬű
īÔŷ
ÔôôÔôô
ŎøŒđÔŋø
îĲĬű
īÔŷîĲŋŸ
Ôôô
TÔŸĲŞř
ŒŒĔČĬīøĬř
aŋøŎÔřĲŎ
;ŞŒĔĲĬ
Fig. 2: Layout conﬁgurations determine the input graphs to
the operator fusion pass. A node is annotated with its output
tensor shape[n0,n1,...], where ni is the size of dimensioni. Layout
{d0,d1,...} represents minor to major ordering in memory. Applied
conﬁgurations are highlighted inred, and other valid conﬁgurations
are highlighted inblue. A layout conﬁguration speciﬁes the layouts
of inputs and outputs of inﬂuential operators (i.e., convolution and
reshape). The compiler propagates layouts from the inﬂuential nodes
to others. A copy operator is inserted when there is a layout mismatch.
A fusion conﬁguration speciﬁes which nodes are fused, where a node
with decision value 1 must be fused with all of its consumers. If a
node being fused has multiple consumers (e.g.,add), it is duplicated
(recomputed).The fusion conﬁguration foraddis the same (fused)
between left and right, but the outcomes differ;conv is fused in
the right but not in the left scenario.This ﬁgure shows operator
fusion directly following layout assignment, but other intermediate
passes can alter the input to the fusion pass further.
• SelectOpt: select an optimization to tune
• GenerateCandidates: develop search candidates
• FixAndApplyCandidate: ﬁx and apply a candidate
• Evaluate: evaluate a candidate
• SelectCandidates: select candidates for the next step
T erminate(line 3) determines when the search stops, e.g.,
when all candidates are explored, or a certain number of steps
have been performed, or the time limit is reached. The search
process also maintains a global ConfStore that captures the
best conﬁgurations observed for all optimizations.
1) SelectOpt: The ﬂexibility to control thesearch schedule
is critical to achieve the best performance given a desired
time budget. Tuning optimizations jointly often enables more
performance improvement opportunities only if we are able
to explore the search space sufﬁciently. XTAT-M allows
users to conﬁgure the search schedule by controlling the
return values of SelectOpt. To tune one optimization pass,
say, B, after another, say A, SelectOpt can return a series
of A,A,..., B,B,...,C,C,... across calls. To enable joint op-
3
Global variables: Opts,ConfStore,best_candidate
1: function MAINSEARCH (ginput )
2: C ←Init(ginput )
3: while !Terminate() do
4: C = SearchStep(C)
5: end while
6: end function
7:
8: function SEARCH STEP(C)
9: op tid ←SelectOpt(Op ts)
10: C′←GenerateCandidates(op tid ,C)
11: for c : C′ do
12: FixAndApplyCandidate(op tid ,c)
13: Evaluate(c)
14: end for
15: return SelectCandidates(C,C′)
16: end function
17:
18: function FIXANDAPPL YCANDIDA TE(op tid ,c)
19: g ←ApplyOpt............ (op tid ,c.graphs[optid],c.conﬁgs[optid])
20: // Update conﬁgs of later passes to be compatible with g
21: for id : SubsequentOpts(Op ts,op tid ) do
22: c.graphs[id] ←g
23: c.conﬁgs[id] ←InferConﬁg(c.conﬁgs[id],g,ConfStore[id])
24: g ←ApplyOpt............ (id,g,c.conﬁgs[id])
25: end for
26: c.graphs[ﬁnal] ←g
27: end function
28:
29: function EVA L U AT E(c)
30: c.cost ←ExecuteGraph(c.graphs[ﬁnal])
31: if c.cost < best_candidate.cost then
32: best_candidate ←c
33: UpdateStore(ConfStore,c.conﬁgs)
34: end if
35: end function
36:
37: function INIT(g)
38: for id : Op tsdo
39: con f ig←GetInitConﬁg................ (id,g)
40: gcombo[id] ←g
41: con f igcombo[id] ←con f ig
42: g ←ApplyOpt............ (id,g,con f ig)
43: end for
44: gcombo[ f inal] ←g
45: cost ←ExecuteGraph(g)
46: C[0] ←{graphs : gcombo,conﬁgs: conﬁgcombo,cost : cost}
47: return C
48: end function
Fig. 3: Formulation of XTAT-M search methodology.
Routines in red are conﬁgured by users to control the search
schedule. Routines in greenare implemented by search strategies.
Routines in purpleare optimization-speciﬁc and essential for joint
optimizations....................Routines in blueare interfaces to retrieve information
from a compiler.
timizations, we can conﬁgure SelectOpt to alternate passes:
A,B,C,A,B,C,.... XTAT-M also supports a mixture of sequen-
tial and joint tuning, e.g., tuningA and B jointly followed by
tuning C and D jointly: A,B,A,B,...,C,D,C,D,.... Note that a
mixed search schedule alone does not enable a joint optimiza-
tion, it is the combination of a mixed search schedule and
candidates ﬁxing (described later in FixAndApplyCandidate)
that enables a joint optimization.
2) GenerateCandidates: XTAT-M lets one apply various
search strategies by implementingGenerateCandidates, which
generates a new set of candidates from the current set of
candidates. Note that a candidate contains conﬁgurations for
all optimizations being tuned.GenerateCandidatesprimarily
focuses on generating new conﬁgurations for a speciﬁc opti-
mization op tid, and optionally new conﬁgurations for the sub-
sequent interacting optimizations. Note that a new candidate
created from this routine may be ill-formed; i.e., c.conﬁgs
may not be compatible with c.graphs because this routine
changes conﬁgurations without updating or considering the
graphs. Consider Fig. 2 as an example, let the left layout and
fusion conﬁgurations compose a candidatec. If we mutate
only the layout ofc to be the layout shown on the right side
of the ﬁgure, the fusion conﬁguration is no longer valid for
the right graph because thecopynode has been removed. We
allow GenerateCandidatesto generate ill-formed candidates in
this step and then ﬁx them in the next step. This is how we
enable search strategy implementers to develop their search
algorithms without having to deal with complex compiler
transformation effects between interacting optimizations.
3) FixAndApplyCandidate: In this step, we update the
intermediate graphs and the conﬁgurations of all optimiza-
tions to be well-formed using ApplyOpt and InferConﬁg.
ApplyOpt(optid,g,conﬁg) transforms g by applying optimiza-
tion id and potentially more optimizations (using compiler’s
heuristics) betweenid and id +1. For example, whenop tid is
layout, to obtain the input graph for the fusion pass, we apply
layout assignment pass with respect toconﬁg and many more
transformations such as conditional code motion and cross-
replicas sharding. Note that not all of these transformations
are worthy of tuning;XTAT-M enables applying the heuristic-
based decisions for them naturally.
When a conﬁguration for an optimization, say A (e.g.,
layout), is changed byGenerateCandidates, the conﬁgurations
for subsequent interacting optimizations, sayB (e.g., fusion),
must be updated to be well-formed. InferConﬁg (line 23)
does so by ﬁxing c.conﬁgs[B] to be compatible with the
graph g, generated from applying c.conﬁgs[A] for pass A.
InferConﬁg ﬁrst identiﬁes the nodes that areunchanged be-
tween g and c.conﬁgs[B] and carries forward the conﬁgurations
in c.conﬁgs[B] for such nodes. For the other nodes ing, it pulls
the best conﬁgurations found so far from the globalConfStore
if present; otherwise, it generates optimization-speciﬁc de-
faults using GetInitConﬁg(B,g)[n]. By default, GetInitConﬁg
returns the default conﬁguration generated by the compiler’s
heuristic, but one can override it to return a random con-
ﬁguration or others for more exploration. The semantics of
InferConﬁg is shown in theinfer-configrule in Fig. 12
in the appendix.
ConfStore[optid][fp(n)] stores the best conﬁguration for op-
timization optid for node n with ﬁngerprintfp(n). Two nodes
are considered to be the same or unchanged if they have
the same ﬁngerprint. This approach of reusing conﬁgurations
of unchanged nodes assumes that the conﬁgurations of these
nodes still work well in a new context, provided we can
deﬁne theunchangedrelation appropriately. A simple strategy
is to compare only the nodes’ attributes (e.g., operator type,
4
T ABLE I: Implementations of search-strategy-speciﬁc functions inXTAT-M. Column |C| describes the number of candidates
returned by the functionSelectCandidates. ParametersM and K in EVO, MBO, and RL are conﬁgurable by users.
Strategy Function GenerateCandidates Function SelectCandidates |C|
Exhaustive Return the next candidate (not visited before). N/A N/A
SA Mutate conﬁgurations of some nodes in the current
candidate.
Return either the old or new candidate depend-
ing on their costs and the annealing temperature.
1
EVO Generate M new candidates by crossing over parent
candidates and mutate some nodes’ conﬁgurations.
Return the K (where K > M) most recent
(unique) candidates, using costs for tie breaking.
K
MBO Generate M new candidates from the model’s latent state. N/A N/A
RL Generate M new candidates from the learned policy at
the current state (i.e., the current candidate).
Return the best candidate so far. 1
input/output shapes, layouts etc). Fig. 2 shows how this
strategy might infer the fusion conﬁguration on the right from
the left one. Theadd is identiﬁed as unchanged, soadd will
be fused with bothreshapeand conv,b u tadd and conv
are not fused together in the left scenario.
To deﬁne unchanged more conservatively, we consider a
context around a node by computing a node’s ﬁngerprintfp(n)
from both its attributes and its neighborhood withink hops. If
k is set to one, only reshape remains unchanged between
the middle left and the middle right graphs in Fig. 2, so we
will not reuse the conﬁguration foradd. In practice,k is set
to ﬁve for the fusion pass (from hyperparameter tuning). For a
node-level optimization where an optimal conﬁguration for a
node is independent from the rest of the graph, we can simply
set k = 0 to ignore the neighborhood.
4) Evaluate: This step evaluates the performance of a
candidate. If the new best candidate is found, we update
the global ConfStore (line 33). The update overwrites a
previous conﬁguration for a node with respect to fp(n) if
its conﬁguration changes, while retaining conﬁgurations for
nodes that do not belong to the candidate, as formalized in
the update-storerule in Fig. 12 in the appendix. It is
important to retain conﬁgurations for nodes that do not belong
to the candidate because the current best candidate may not be
optimal and future search steps taken by the search strategy
may favor graphs that contain these nodes.
5) SelectCandidates: The last action is selecting a set of
generated candidates to pass to the next round.
To support tuning an additional pass, one can simply add
the optimization to theOp tslist, conﬁgure the search schedule
through SelectOpt, set the neighborhood size forfp for that
speciﬁc optimization (which can be set through hyperpa-
rameter tuning), and modify the compiler pass to apply the
optimization (ApplyOpt) according to a given conﬁguration.
Note that ﬁxing illegal combinations of conﬁgurations is
the key to our joint autotuning methodology. Simply dropping
illegal combinations is insufﬁcient for joint autotuning because
changes applied to one pass conﬁguration without the ﬁx are
often illegal. This means one has to fall back to using default
conﬁgurations for later passes, resulting in tuning one pass
after another, but not a joint optimization.
C. Search Strategies
Below, we describe a wide spectrum of search strategies
that we have evaluated. Table I summarizes how the following
search strategies implement routinesGenerateCandidateand
SelectCandidate.
1) Exhaustive search within a node:The exhaustive strat-
egy generates a new candidate by selecting the next option
for the current node’s conﬁgurations. When all conﬁgurations
have been explored for the current node, it moves on to the
next node. This strategy can be used for node (kernel) level
optimizations, such as tile size selection.
2) Simulated annealing (SA): SA mutates a candidate and
probabilistically accepts a new one based on annealing tem-
perature that decreases over time. We have found the following
types of mutation operators to be effective:
• Single-node: mutates the conﬁguration of one randomly
selected node.
• Multi-node: mutates conﬁgurations of all nodes indepen-
dently with probabilityp, wherep decreases as a function
of temperature in simulated annealing.
• Group: mutate conﬁgurations of a randomly-selected set
of nodes grouped by a certain criterion.
We have empirically observed that a multi-node mutation
works well for fusion autotuning and a combination of group
and single-node mutations works best for layout autotuning.
3) Regularized evolution (EVO):EVO performs evolution-
ary search using a population of K individuals. Each new
individual is generated by selecting two parents from the
population using binary tournament selection, recombining
them with some crossover rateγ, and mutating the recombined
individual with some probabilityμ. Following [16], to promote
exploration, the population is updated by replacing the oldest
individuals by newly evaluated individuals. In our experiment,
we use the parametersK = 100, γ = 0.2, andμ = 0.01. We also
considered population-based evolutionary optimization using
P3BO [17] but did not ﬁnd clear performance improvement
over a single evolution optimizer.
4) Model-based optimization (MBO): MBO performs
model-based optimization with automatic model selection
[18]. At each optimization round, a set of candidate regression
models are ﬁt to the data acquired thus far, and their hyper-
parameters are optimized by randomized search and ﬁve fold
cross-validation. Models with a cross-validation score≥0.4
are ensembled to deﬁne an acquisition function. This function
is then optimized by regularized evolution to generate a new
batch of samples. Candidate models include ridge regression,
random forests, gradient boosting, and neural networks.
5
5) Deep reinforcement learning (RL):A deep RL method
(GO) [19] is designed speciﬁcally for ML compiler’s graph
optimizations. GO uses a graph neural network (GNN) to
create node embeddings and segmented recurrent attention
layers to capture long-range dependencies that appear in a
computation graph. The policy network transforms the graph
representation into optimization decisions with soft attention.
The learning objective is optimized using Proximal Policy
Optimization (PPO) [20]. Instead of using conventional RL
algorithms, we leverage the existing compiler heuristics by
initializing the search with the default heuristic solution, and
modify the RL sampling stage to encourage more exploitation.
Concretely, we modify the state transition function such that
RL only traverses to a state with a higher reward than the
default conﬁguration’s:S(t)= S(t −1) if R(t) < R(0), where
state S(t) is the embedding of the fusion conﬁguration at step
t, and R(0) is the reward for the default conﬁguration.
D. Limitations
There are some important optimizations, such as graph
rewrites, that do not naturally lend to a node-conﬁguration-
based representation. In such cases, we believe thatXTAT-M
can still be used as a subroutine, where the outer loop applies
rewrites, and for each candidate change, we useXTAT-M to
optimize other optimization decisions. Notice thatXTAT-M
does not have to start the search from scratch every iteration
because the global ConfStore persists across all iterations.
Our formulation also does not support tuning an unbounded
number of passes, for example, passes that run until a ﬁxpoint.
III. XTAT:I MPLEMENT A TION INXLA
We developXTAT based onXTAT-M to tune tensor layouts,
operator fusion decisions, tile sizes, and code generation
parameters in XLA.XTAT’s target accelerator is TPUs [15],
energy-efﬁcient ML accelerators.
A. Background on XLA
XLA is a ML compiler capable of generating code for
various hardware targets [11]. Its workﬂow can be split into
three stages. In the graph-level passes, XLA uses a graph-
based intermediate representation named High-Level Opera-
tion (HLO) to describe a tensor computation graph. V arious
compiler passes transform HLO so that the output graph is
algebraically optimized, and is ready to be mapped onto the
target hardware. At the end of the graph-level phase, the
optimized HLO graph consists of nodes that represent fusions
of multiple operations. We refer to a fused node as akernel.
Next is the hardware lowering phase; the compiler converts
each individual kernel into instructions that can be executed on
the target hardware. In the third phase, low-level architecture-
speciﬁc optimizations are applied, such as VLIW instruction
scheduling, peephole optimizations, and register allocation.
B. XTAT’s Optimization-Speciﬁc Search F ormulations
We instantiate XTAT-M to tune the most performance-
critical optimization passes in XLA including layout as-
signment, operator fusion, and tile size selection, as well
T ABLE II: Instantiation ofXTAT-M for speciﬁc optimization
passes tuned byXTAT.
Optimization Applicable node Node’s con f ig
layout assignment conv. & reshape input and output layouts
operator fusion fusible node fusion control bit
tile size kernel w/ tiling input and output tile sizes
ﬂags non-comm kernel lowering-phase ﬂag values
as compiler’s ﬂags used during the lowering phase. Layout
assignment and operator fusion effect graph-level changes and
hence are optimized globally. Tile size and ﬂags selection
are kernel-level optimizations and hence are optimized for
each kernel independently. Apart from these optimizations,
XTAT employs the other heuristic-based passes used in XLA.
Our choice of passes to tune was inﬂuenced by expert XLA
developers who suggested these optimizations as ones that
signiﬁcantly affect performance of most ML models. Note
that layout and/or fusion decisions heavily inﬂuence other opti-
mizations such as cross-replica sharding and rematerialization.
While rematerialization and operator scheduling are important
for making a program ﬁt in available memory, they do not
typically reduce program’s execution time.
For each optimization pass we tune, we aim to deﬁne
a search space that is: (1) expressive, i.e., contain diverse
candidates that lead to optimal outcomes; (2)compact, i.e.,
include only valid candidates, few candidates that have the
same behavior, and not too many more bad candidates than
good ones. We specializeXTAT-M’s generic formulation to
speciﬁc optimizations as summarized in Table II. The rest of
this section details the search formulation for each optimiza-
tion. We also describe existing XLA heuristics and alternative
search formulations, which are the baselines in our evaluations.
Layout Assignment: The layout assignment pass chooses
the physical layouts of the input and output tensors of each
node to satisfy constraints from the user’s program, the com-
piler backend, and the underlying hardware, while minimizing
the program’s execution time. An example layout constraint
for convolution on TPUs is that the input and output must
have input feature, output feature, or batch dimensions as
their most minor dimensions. Figure 2 displays the valid
input layouts ofconv in blue. Layout{d0,d1,...} represents
minor to major dimensions, where elements in the most minor
dimension are physically consecutive. If an edge connects an
output to an input with a different layout, the compiler inserts
a copy (transpose) operator to convert the layout. In Fig. 2
(left), the compiler assigns layout{1,0,2} to the output of
add but {0,1,2} to the ﬁrst input ofconv, causing a layout
mismatch, and the insertion of a copy operator. The compiler
must trade-off between selecting the best layouts for each
speciﬁc operator and the overhead from copy operators.
Compiler’s heuristic:XLA performs layout assignment in
multiple rounds. Initially, the pass includes only constraints
from the program’s inputs and outputs. In each round, the
layout propagation algorithm propagates layouts from the
constrained operators to others through element-wise, pad, and
slice operators. This optimistic propagation may cause layout
6
constraint violations at some operators. After each round, the
pass uses various heuristics — ranging from a rough cost
model to hard-coded decisions depending on the operators
— to rectify the violations. This process continues until all
constraints are satisﬁed.
Naïve search formulation:We could deﬁne the search space
to cover all permutations of the dimensions of input and output
tensor of every node. However, this leads to an extremely large
search space with many invalid and inefﬁcient candidates.
Our search formulation:We deﬁne the search to conﬁgure
only the most layout-performance-critical nodes, which are
convolution4 and reshape operations because they are common
operations and have the most constrained implementations for
TPUs. The search queries the compiler for valid input-output
layout combinations for these nodes. Once layouts of reshape
and convolution nodes are assigned, we leverage the existing
layout propagation algorithm to propagate layouts from these
nodes to others. This search space contains only valid and
relatively efﬁcient candidates.
Operator Fusion: When operators are fused, intermediate
tensors can be used by the consuming operator directly without
saving them to the slow main memory. Fusion also reduces
kernel launch overheads. While most ML compilers [1], [3],
[4], [6], [10], [21] support fusions of a limited set of oper-
ations, XLA can fuse more complex operations (e.g., gather,
scatter, reshape, reduce, etc.) resulting in more optimization
opportunities as well as more decisions to make. Typically,
when an operator is fused into multiple consumers, it must be
duplicated (recomputed) in each consumer because consumers
can have different iteration spaces (loop structures). This is
illustrated in Fig. 2, whereadd is fused into bothreshape
and copy. The compiler must trade between recomputation
and reduced memory communication.
Compiler’s heuristic:The XLA fusion algorithm maintains
a priority queue of all nodes in the graph. A cost model
computes the priority of a node as the beneﬁt of fusing that
node into its consumers. The algorithm iteratively fuses a node
with the highest priority value until all nodes have negative
priority values. The fused node and its consumers are removed
from the queue, newly formed fused nodes are inserted, and
the priority values of affected nodes are updated.
Our search formulation:We assign a boolean value to each
fusible node to control whether it is fused with its consumers.
Alternative formulation (edge):We can assign a control bit
per edge (instead of per node) such that a parent node is
fused with only a consumer whose edge is marked. However,
when a parent node u is fused with a consumer nodev but
not a consumer nodew, the compiler saves the intermediate
output of u in the slow memory; as a result, we do not get
the full beneﬁt of fusion. While this formulation is more
expressive than the per-node formulation, we believe this
additional coverage is unnecessary.
Alternative formulation (node priority):Another approach
4All tensor contraction operations (e.g., multiplication and einsum) are
converted into convolutions before layout assignment in XLA.
is to assign a node a priority value instead of controlling
its fusion behavior explicitly, similar to GO [19]. We can
adapt the existing heuristic to use these priorities as its initial
node priorities, instead of using the cost model. However, the
heuristic dynamically adjusts the priorities when nodes are
fused; a conﬁguration cannot deﬁne these values in advance
because we do not know how many adjustments will occur.
A workaround is to set the priority of the newly fused node
to the sum of its constituents’ priorities. This search space is
larger than ours, but covers the same set of behaviors.
Alternative formulation (ﬂags):Another approach is to tune
the compiler’s fusion-related ﬂags. We try tuning ﬂags that (1)
limit fusions of inputs into convolutions, (2) limit fusions of
outputs into convolutions, and (3) parameterize the fusion cost
model. This search space is small, and does not grow as the
number of nodes increases. However, this approach provides
less control than the other formulations.
Tile-Size Selection:The goal is to pick an optimal tile size
for each kernel’s input and output tensors such that they ﬁt in
scratchpad memory.
Compiler’s heuristic: XLA enumerates all possible tile
sizes and chooses the optimal one according to hand-written
analytical cost models.
Search formulation:We query the compiler to get a set of
valid tile sizes for each kernel to form the search space. The
number of tile sizes we autotune ranges from 2 to 500,000.
Lowering Flags: Apart from tile size selection, the
lowering phase is also responsible for many performance-
critical decisions. Many of these decisions are determined by
heuristics that are controlled by ﬂags. Some examples include
instruction window size for hoisting load/store instructions,
enabling overlaps of input/output DMAs, and scratchpad limit
to allocate to an operator.
Compiler’s heuristic:The compiler developers set these ﬂag
values to “magic" numbers that work well on most benchmarks
in the regression suite.
Search formulation:For an integer ﬂag, we limit the search
space to four sensible values.XTAT tunes the total of eight
integer and boolean ﬂags.
C. XTAT’s Search Schedule
We conﬁgure SelectOpt in XTAT-M to explore different
search schedules for tuning our four target optimizations. For
scalability, we choose to decouple graph-level optimizations
(layout and fusion) from kernel-level optimizations (tile size
and ﬂags) for scalability.
1) Joint layout-fusion autotuning: Following the formu-
lation presented in Section II-B, we conﬁgure SelectOpt to
alternate between layout and fusion everyS steps, whereS is
set to ﬁve via hyperparameter tuning. Mutating both layout
and fusion conﬁgurations in one search step performs slightly
worse than the alternating strategy.
2) Joint tile-size-ﬂags autotuning: The tile size selection
pass does not change the graph, merely annotating nodes with
tile sizes, so the joint tile size and ﬂags optimization is less
complex. Further, the joint search space of tile sizes and ﬂags
7
is small enough for enumeration. Hence, we performed an
exhaustive search within each node on the cross-product of tile
size and ﬂags search spaces until the time limit is reached. We
also explore tuning tile size and then ﬂags sequentially. We
conﬁgure SelectOpt to tune tile size until exhaustion before
switching to ﬂags. The result in our evaluation (Section IV -B)
reveals that tuning them in sequence is superior to tuning them
jointly because we can only explore a small fraction of the
entire joint search space before timeout. Thus, we conﬁgure
XTAT to tune tile size and ﬂags separately.
D. Execution Time Measurement
1) Measurement on hardware:
a) Kernel execution time: Most kernels’ runtimes do
not depend on input data, so they can be measured reliably
using random inputs; this is an approximation in a few cases
such as gather and scatter kernels. We do not tune tile sizes
for kernels that communicate across multiple devices; their
optimal choices are trivial since there is no complex tradeoff
between communication and compute. We parallelize runtime
measurements across machines, and use multi-threading to
further parallelize compilation.
b) Graph execution time: We measure the runtime of
a graph by summing its kernels’ runtimes. We ignore kernels
that communicate across multiple devices, as layout and fusion
decisions rarely affect these kernels’ runtimes, enabling tuning
a multi-device model on a single device. We ignore loops
and conditionals, considering only the kernels in their bodies.
We found these approximations to be accurate enough for
ranking in autotuning. One can improve the ﬁdelity of the
estimates using program traces to weight kernels’ runtimes
by their execution counts. Finally, we ignore interactions
between kernels as they do not execute concurrently on a
TPU, leaving only negligible effects like code prefetching
and device temperature. We use caching to avoid measuring
identical kernels repeatedly since small conﬁguration changes
leave most kernels in a graph unchanged.
2) Learned cost model:To avoid expensive candidate eval-
uations (compiling and executing on real hardware), we train
a learned cost model to predict execution time, following
the TPU learned performance model [22]. The model uses
a GNN to encode operation features and the structure of a
kernel subgraph. Node embeddings outputted from the GNN
are summarized using a simple reduction function to generate
a kernel embedding, which is fed into a feed-forward layer to
produce a ﬁnal prediction. We train one model for all graph-
level optimization tasks, one for tile-size selection, and one
for ﬂags selection. The GNN is shared between tile-size and
ﬂags models. For graph-level tasks, the model predicts the
absolute runtime of a kernel based on the default tile size
and ﬂag values, selected by the compiler’s heuristics. The
entire program’s runtime is the summation of the kernels’
predictions. For kernel-level optimizations, the model predicts
relative runtimes of different conﬁgurations of a given kernel.
To ensure accuracy of the model for newer types of work-
loads and to keep up with compiler’s changes, we ﬁnetune
the model periodically; we freeze the GNN layers and retrain
only the feed-forward head. Since the model can never be
100% accurate, we execute the topk conﬁgurations on real
hardware and pick the best.
IV . EV ALUA TIONS
Our benchmarks comprise 150 machine learning models
from the XLA TPU regression suite representing both produc-
tion and research usage at Google. They include both inference
and training graphs, with sizes ranging from 100 to 56,000
nodes. We report execution time speedup of each benchmark
over itsdefault execution time when compiled using the XLA
compiler’s heuristics, which have been continuously improved
by a large team of experts since 2016 for production usage.
Execution time is measured on TPU v3 [15].
A. End-to-End Autotuning
1) Tuning on real hardware:According to the results from
Section IV -B, we set our end-to-end autotuning schedule as
follows: ﬁrst, jointly tune layout and fusion via SA, followed
by exhaustive search for tile-size autotuning, and ending with
exhaustive search for ﬂags tuning. The joint layout-fusion
optimization was tuned for two hours on 10 TPU machines
(each with a host and an accelerator) or at most 10,000
candidate evaluations on each machine for each model. The
exhaustive search for tile-size and ﬂags autotuning is sharded
to run on 10 machines with one hour timeout for each task.
The results in this section show whatXTAT can achieve given
a moderate amount of resources and time; techniques to reduce
tuning time and resources will be evaluated later.
Figure 4 reports the real execution time speedup using
representative program inputs and complete control ﬂow. The
ﬁgure breaks down the speedup contributions from autotuning
different passes. We display 43 models that achieve perfor-
mance improvement of 5% or more. Overall,XTAT offers 5%
speedup on average over the production compiler across 150
models, where tile size and fusion autotuning contributes the
most to the total speedup, followed by layout, and ﬂags auto-
tuning. We see a huge speedup on A VSpeech inference (2.4×)
and Translate Transformer inference (1.5×). Nine models also
exhibit more than 15% improvement. Note that the XLA
compiler has been heavily optimized for most of these models;
nevertheless, the autotuner still provides substantial speedup:
14% on MLPerf DLRM training (recommendation model),
13% on MLPerf Mask RCNN training, 11% on MLPerf SSD
training, and 7% on a few ResNet training models.
2) Tuning with learned cost model: Next, we evaluate
the learned cost model in terms of its efﬁcacy on reducing
tuning time. We generated training data for the learned cost
model from compiling and running the 150 ML models using
random layout, fusion, tile size, and ﬂag conﬁgurations. Eight
benchmarks were held out for testing.
On the hold-out benchmarks, we compared: (1) using only
real hardware for evaluations and (2) using the learned cost
model to select topk conﬁgurations to evaluate on real hard-
ware. Both settings employed the same autotuning schedule
8
6SHHGXS




DYVSHHFKBL
WUDQVODWHBWU
PDJHQWDBG
WWVBQDWBV\QW
WUDQVODWHBWU
JUDSKQHWVB
JUDSKQHWVB
XQHWG
RSHQDLBYB
POSHUIBGOUP
POSHUIBPDV
JUDSKQHWVB
POSHUIBVVG
XQHWBWUDLQLQ
HIILFLHQWQHW
POSHUIBPDV
PQDVQHWBD
PQDVQHWBE
PDVNBUFQQ
EDEHOILVKBWW
RSHQDLBYB
WUDQVODWHBK
UHVQHWBYB
WHQVRUWHQV
UHVQHWBYB
WWVBV\QWKHVL
KDORQHW
S\WRUFKBWSX
WXQDVBVLPS
XQHWGBVSD
GOUPBVKDP
LQFHSWLRQBY
GOUPB[BZL
GOUPBGHQVH
PDVNBUFQQ
DOH[QHWBWUDL
JDPHSOD\B
GHQVHBVXE
LPDJHBHPE
POSHUIBVVG
UHVQHWBPRG
GOUPBGHQVH
WUDQVODWHBK
)ODJV 7LOH6L]H )XVLRQ /D\RXW
Fig. 4: End-to-end model speedups from autotuning 150 ML models. The ﬁgure shows only models that achieve 5% or more
improvement. The ﬁrst bar (A VSpeech inference) has 2.36× speedup (2.34× from fusion and 2% from tile size). There is no
performance regression on the rest of the benchmarks; in the worst case, the model’s execution time remains the same.
(joint layout fusion, followed by tile size, and ended with
ﬂags tuning). For (1), we used the same autotuning setup as
in Section IV -A. For (2), we also used 10 TPU machines
to perform real evaluations on hardware but used 10 CPU
machines at the beginning to select top layout and fusion
conﬁgurations in order to cut down time using TPU machines,
which are in high demand. In particular, we ﬁrst ran the joint
fusion-layout autotuning on 10 CPU machines for two hours
or at most 10,000 steps using the learned cost model, selected
the top (k = 10) candidates from each machine, and ran them
on 10 TPU machines. Then, we used the learned cost model
to pick the best (k = 5) tile sizes and ﬂags from each shard to
execute on a TPU. Here, we ran both cost model evaluations
and real hardware evaluations on TPU machines as time spent
on cost model evaluations are negligible.
Figure 5a reports the execution time speedup using represen-
tative program inputs and complete control ﬂow, and Fig. 5b
reports the tuning time. Overall, using the learned cost model
achieves almost the same speedup as using real hardware
alone, while drastically reducing tuning time. On average, it
reduces the tuning time of tile sizes and ﬂags by 6 × and
8× respectively. Notice that although it does not help reduce
the total layout-fusion tuning time signiﬁcantly, it reduces the
tuning time on TPU machines by 240×.
This improved tuning time makes it possible to run the
autotuner at scale. We have deployed the tile size and ﬂags
autotuning to automatically optimize the top workloads in
Google’s ﬂeet daily. The learned cost model enabledXTAT
to tune 20× more kernels per day. In the past 10 days,XTAT
has sped up the most heavily-executed kernels by 3.5% and
1.6% on average from tile size and ﬂags tuning respectively.
3) P erformance analysis: The speedup from autotuning
generally comes from higher FLOPs utilization. Good tile sizes
balance computation and memory bandwidth utilization.XTAT
doubles the tile sizes of many copy operations in the Translate
Transformer benchmark, increasing the memory bandwidth
utilization of these operations from 45% to 70%, resulting
in an overall speedup of 1.3×.
(a) Execution time speedup on benchmarks in the hold-out set when
using the learned cost model vs. using hardware evaluations alone.
Higher is better. Using the learned cost model achieves the same
speedup of 2.4× on AvSpeech as using the real hardware.
(b) Tuning time in minutes (maximum across 10 machines). Lower is
better. Light blue is tuning time on CPU machines, which are cheaper
and more abundant resources.
Fig. 5: End-to-end autotuning: the learned model drastically
reduces tuning time while achieving almost the same speedup
as using real hardware alone.
Fusion autotuning generates better fused operators, enabling
more efﬁcient tile sizes for corresponding operations. E.g.,
in A VSpeech inference, fusion tuning moves one reshape
operation out of a convolution fusion operation, enabling it
to use a tile size that is 12.5× larger than before. As a result,
this convolution fusion is sped up by 12.5× with over 2×
9
FLOPs utilization and over 3×memory bandwidth utilization,
resulting in an overall speedup of 2.3× for that benchmark.
Good layout assignment generally reduces the overall data-
formatting overheads. However, autotuning Translate Trans-
former reveals an intricate relationship between different op-
timizations. Layout tuning adds a copy operation, removing
a bitcast from a long-running fused operation. As a result,
the FLOPs and memory bandwidth utilization of the fusion
operation increases by over 2× and over 1.5× respectively,
yielding an overall speedup of 1.16× for that benchmark.
4) Result discussion: Our performance improvements may
at ﬁrst seem modest. However, our benchmarks are repre-
sentative of large, real world deployments, so even small
improvements translate to signiﬁcant resource savings. Further,
the baseline compiler we compare against is actively being
tuned against the same benchmarks (and sometimes in the
light of our own tuning results!), making it a constant battle
to stay ahead. WhenXTAT was applied to top workloads from
the TPU ﬂeet (beyond the benchmark suite), we saw higher
speedups (approximately 15% on average).
Prior works [1]–[7], [10] evaluate only on inference models,
and show impressive performance improvements. However,
their baselines are library kernels (from CuDNN, MKL,
etc) that are generally optimized for training workloads. In
contrast, our baseline is the XLA TPU compiler, which is
optimized for both inference and training workloads.
B. Search F ormulations
To evaluate the search formulations presented in Sec-
tion III-B, we used SA as the search strategy, measured execu-
tion time using real hardware, and selected 10 benchmarks that
showed signiﬁcant speedup from autotuning for a particular
optimization problem. We ran 10 replicas of SA with random
seeds in parallel and reported the best conﬁguration found
among all the replicas. On each replica, we ran the search for
10,000 steps with a two-hour time limit. We experimented with
two modes: starting the search from a default conﬁguration
(from the compiler’s heuristic) and starting from random con-
ﬁgurations (different replicas starting from different random
conﬁgurations). This subsection reports speedup with respect
to execution time measured as described in Section III-D1
(ignoring control ﬂow and using randomly generated data).
1) How effective is our layout assignment formulation?:
As shown in Fig. 6, when starting the search from a default
conﬁguration, our proposed search formulation drastically
outperforms the naïve formulation, as we hypothesized. When
starting from random conﬁgurations, the search using the naïve
formulation is unable to ﬁnd any valid layout conﬁguration
for any of the benchmarks. In contrast, the search using the
proposed formulation is able to ﬁnd valid conﬁgurations for
all benchmarks with an average speedup of 8.6%, which is
almost the same as the average speedup of 9% when starting
from the default conﬁguration.
2) How effective is our operator fusion search formula-
tion?: Figure 7 shows the results when starting the search from
a default conﬁguration. The per-node control bit formulation
6SHHGXS





UPB[BZLGHBGHQVHDYVSHHFKBWUDLQLQJJUDSKQHWVBQPJUDSKQHWVBQWUDQVODWHBWUDQVIRUP
POSHUIBGOUPPOSHUIBVVGPOSHUIBVVGPOSHUIBVVG
RSHQDLBYBUQQBQDW
JHRPHDQ
3URSRVHG 1DLYH
Fig. 6: Layout autotuning: execution time speedup using
different search spaces (starting from a default conﬁguration).
Our proposed formulation outperforms the naïve formulation.
6SHHGXS




DXWRHQFRGHUHIILFLHQWQHWJUDSKQHWVBQJUDSKQHWVBQJUDSKQHWVBQPQDVQHWBEUHVQHWBSDUDO
UHVQHWBYXQHWBWUDLQLQZDYHQHWBWUDL
JHRPHDQ
3HU1RGH 3HU(GJH 1RGHSULRLUW\ )ODJV
Fig. 7: Fusion autotuning: execution time speedup with differ-
ent search spaces (starting from a default conﬁguration). The
per-node search space offers the most speedup.
is slightly (2 percentage points) better than the per-edge al-
ternative, as hypothesized. The difference is more pronounced
when starting from random conﬁgurations (not shown in the
ﬁgure), where the per-node formulation’s average speedup is 8
percentage points higher than that of the per-edge formulation.
Another search formulation uses node priority, and is sig-
niﬁcantly worse than the control bit formulations. Even when
starting from default, the best candidates found in this search
space are worse than the default; it is impossible to faithfully
reconstruct the default conﬁguration in this formulation as
priority values change dynamically during the fusion pass.
The last alternative search formulation is tuning ﬂags.
Because the entire search space has only 52 candidates, we
performed an exhaustive search here. As we hypothesized, this
formulation is worse than the control bit formulations due to its
limited expressivity, offering tiny speedup on all benchmarks.
3) How much beneﬁt does joint layout and fusion autotun-
ing offer?: To investigate the potential of joint autotuning of
multiple passes, we compared two strategies: tuning layout
and then fusion sequentially (each for two hours), and tuning
them jointly (for four hours). Figure 8 shows the results
for benchmarks where joint autotuning improves over tuning
layout and then fusion. For the remaining six benchmarks,
sequential and joint tuning offer the same speedup. According
10
6SHHGXS







JUDSKQHWVBQPJUDSKQHWVBQ
UHVQHWBY XQHWBWUDLQLQJ
/D\RXWWKHQ)XVLRQ -RLQW -RLQWQRUHXVH
-RLQWUDQGRPVWDUW -RLQWQRUHXVHUDQGRPVWDUW
Fig. 8: Autotuning layout then fusion vs. autotuning them
jointly (with/without conﬁguration reuse). The ﬁrst three in
each cluster start from a default conﬁguration; the last two start
from random conﬁgurations. ‘No reuse’ does not employ the
conﬁguration reuse mechanism (the globalConfStore). Joint
tuning with the reuse mechanism offers the most speedup.
to Fig. 8, joint tuning is better than sequential tuning. This
is because a better layout conﬁguration may disable an even
better fusion conﬁguration.
In addition to evaluating the achievable speedups, we also
compared tuning time between sequential tuning and joint
tuning. We measured the time taken to reach within 0.1
percentage point of the highest speedup. On average, joint
tuning took 2.7× less time than sequential tuning to reach the
highest speedup across 10 benchmarks. This is likely because,
in some benchmarks, the sequential search schedule wasted
time optimizing layouts (in the ﬁrst half of the search) when
there was not much room for improvement.
This experiment also suggests that our simple search tech-
nique is still effective in an extremely large search space,
thanks to our strategy of leveraging compiler heuristics.
4) How important is the conﬁguration reuse mechanism
for joint autotuning?: The conﬁguration reuse mechanism
(Section II-B3) enables the autotuner to reuse parts of the best
fusion conﬁgurations found so far when a layout conﬁguration
changes. Without this mechanism, the autotuner must reset to
a default or random fusion conﬁguration when a new graph
is generated by a new layout conﬁguration. According to
Fig. 8, when the search starts from a default conﬁguration,
there is no difference between using and not using the reuse
mechanism. However, if the search starts from a random
conﬁguration and GetInitConﬁg in Fig. 3 returns a random
conﬁguration, we observe signiﬁcant beneﬁt from employing
the reuse mechanism: up to 15% execution time improvement.
In terms of time to reach the highest speedup (within 0.1
percentage point) when starting from a default conﬁguration,
reuse or no reuse mechanism took similar time on all but two
benchmarks. On MLPerf DLRM and GraphNets n4k, the reuse
mechanism reached the highest speedup 4.5× and 1.8× faster
than no reuse mechanism did. Hence, the reuse mechanism is
useful for reducing tuning time for some models and essential
if one does not have good default conﬁgurations.
5) How much beneﬁt does joint tile size and ﬂags autotun-
ing offer?: Similar to joint layout-fusion, we compared tuning
tile size and then ﬂags, and tuning them jointly. Since tile size
and ﬂags search space is small, we enumerated all candidates
and evaluated them in a random order up to a time limit (10
minutes). For the sequential schedule, we tuned tile size and
then ﬂags on every kernel. For the joint strategy, the time limit
per kernel is 20 minutes. Tuning tile size then ﬂags offers 6.6%
average speedup, while tuning them jointly offers only 1.9%
average speedup. This is because while the individual tile size
and ﬂags search spaces are small, their joint search space is
too large; as a result, traversing the search space in a random
order fails to discover good candidates by the time limit.
C. Search Strategies
To show that XTAT-M allows various search strategies
without changing the search formulation, we ran the search
techniques from Section II-C (excluding exhaustive search) on
fusion autotuning. This also evaluates the search strategies at
ﬁnding good candidates in a large search space. We measured
execution time on real hardware, using 1,000 or 10,000
candidate evaluations, depending on the benchmark’s time for
one evaluation. We ran each search strategy on 10 replicas
starting from the default conﬁguration. We report speedup
using randomly generated data and ignoring control ﬂow.
Figure 9 shows the average speedup over the default con-
ﬁguration. Fig. 10 in the Appendix shows the optimization
trajectories for one example benchmark. We ﬁnd rather simple
evolution-based techniques (SA and EVO) reliably ﬁnd better
solutions than advanced model-based optimization techniques
(MBO and RL). We hypothesize that the high-dimensional
search spaces (2960 for AutoEncoder; up to 239568 for Efﬁcient-
Net) make it challenging to ﬁt an accurate model without prior
knowledge. Furthermore, the evolution-based techniques take
less time to propose new candidates, reducing compute costs
and overall tuning time. While SA and EVO are comparable
for most benchmarks, SA is worse than EVO and RL on
GraphNet models. With more samples, however, SA achieves
Fig. 9: Fusion autotuning: average execution time speedup
using different search strategies (starting from a default con-
ﬁguration) when allowing up to 1,000 or 10,000 samples
(depending on benchmarks). Error bars show 95% conﬁdence
intervals over ten replicas.
11
the highest speedup. We conclude that when resources are
limited (one or a few machines available for autotuning), EVO
ﬁnds good conﬁgurations faster than other algorithms.
V. RELA TEDWORK
A. Autotuning in ML Compilers
Autotuning has been effective at optimizing code in various
domains [23]–[28]. We apply this technique to a multi-pass
ML compiler. Recent search-based ML compilers [1]–[8] ap-
ply autotuning at the kernel or subgraph level. Template-based
approaches [1]–[7] are similar to XLA ’s lowering algorithm;
based on a kernel’s subgraph, a loop structure template is
used to generate code. Ansor [3] proposed a template-based
approach that allows more ﬂexible loop fusions compared to
prior work. However, their tiling structure is still ﬁxed, and
their fusion capability is more limited than XLA ’s. Halide [5],
[29]–[31] covers a larger space of possible loop implementa-
tions compared to template-based approaches, but performed
worse than FlexTensor [4] and Ansor [3]. Mind Mappings [32]
and AKG [33] focus on operator-level optimizations and code
generation for custom hardware accelerators. UnlikeXTAT,
these prior works do not tune layout decisions.
The value learning approach [9] ﬁrst applied search to loop
optimizations for an entire graph at once. However, its search
must be applied in a single compilation stage, making it
inapplicable to multi-pass production ML compilers, and it
does not tune tensor layouts. This approach has been applied to
inference graphs with up to 400 nodes, while our approach has
been evaluated on both inference and training graphs with up
to 500,000 nodes. DeepCuts [34] applies a greedy exploration
guided by an analytical cost model to tune graph-level fusion
decisions along with some GPU kernel parameters. However,
its fusion capability is limited, and it is not obvious how
to extend DeepCuts to support layout tuning at the graph
level. Graph substitution approaches [10], [35] optimize entire
tensor computation graphs, but work in a limited search space
reachable via graph rewrites. Without exploding the number of
rewrite rules, they cannot represent arbitrary fusions of tensor
operations or change layouts of arbitrary tensors. GO [19]
optimizes device placement, operator fusion, and operator
scheduling decisions for an entire TensorFlow graph. However,
it does not consider the effect of multiple compilation passes
on the intermediate computation graph, so we cannot adopt
their search formulation when dealing with a full compiler
stack. Rammer [36] can jointly optimize inter- and intra-
operator parallelism, but it does not address other kinds of
optimizations, such as layout and fusion, addressed in this
paper. Rammer’s approach is also inapplicable to TPUs, since
TPUs do not support concurrent execution of multiple kernels.
Many existing works use heuristics or analytical cost models to
tackle a speciﬁc graph-level optimization, including operator
fusion [21], [37], [38] and layout assignment [39].
B. Joint Autotuning Capability
In a multi-pass compiler, changing a conﬁguration for one
optimization pass changes the search spaces for subsequent op-
timizations, as explained in Section II-A. Generic autotuning
frameworks, such as OpenTuner [28], require the entire search
space to be speciﬁed upfront, thus disallowing search spaces
that change dynamically. Our joint autotuning methodology is
a generic method to formulate the interactions between multi-
pass compiler optimizations. It is applicable to any multi-
pass compiler and is not speciﬁc to the XLA compiler or
XTAT. Therefore, it can be implemented in existing autotuning
frameworks (if they can be extended to support dynamic search
spaces), enabling them to perform joint optimizations.
The approach used in most search-based ML compilers [1]–
[9], inspired by Halide [29], supports joint optimizations by
default. However, when a search space is large, it requires a
cost model that can accurately evaluate a (partial) schedule
or conﬁguration, which is extremely challenging to create for
an entire tensor graph. Therefore, most of these compilers
optimize only one small subgraph at a time. While the value
learning approach [9] can optimize an entire inference graph,
it requires carefully-crafted feature engineering, which is not
easily applied to new hardware. Furthermore, one cannot easily
apply a Halide-like approach to multi-pass production compil-
ers without completely reimplementing them. Our goal is to
provide autotuning capability to such multi-pass compilers.
C. Learned Cost Model
Similar to prior works [2], [5], [7], [9], [22], [40]–[42],
we use a learned cost model to accelerate autotuning. We
additionally propose a pretraining-ﬁnetuning method to reduce
training time while keeping the model up-to-date.
VI. C ONCLUSION
We proposed a search methodology that enables autotun-
ing various graph-level and subgraph-level optimizations at
different compilation stages for production ML compilers.
Our search formulation allows solving optimization problems
jointly or one-at-a-time, as well as using a spectrum of search
strategies to solve them. Based on this, we developed an
autotuner to tune the key optimization passes in the XLA TPU
compiler. The execution time speedups on 150 ML models
from Google’s workloads found by the autotuner averaged 5%,
with many cases over 15%, and one improving by a factor of
2.4. We substantially reduced tuning time via a learned cost
model.
XTAT has been used in different ways to optimize produc-
tion models. Compiler developers have usedXTAT to detect
opportunities to improve the baseline heuristics multiple times.
For instance, fusion autotuning led to a ﬁx in the heuristics,
yielding 18% latency reduction on a model served in produc-
tion. Such improvements are invisible in our experiments as
they have already been incorporated into the compiler. Besides
being used by compiler developers,XTAT has been deployed
to automatically tune tile sizes and ﬂags for the most heavily-
used production models in Google’s ﬂeet everyday.
ACKNOWLEDGMENT
We thank Albert Cohen, Charith Mendis, Jacques Pienaar,
Jason Ansel, and the reviewers for their insightful feedback.
12
REFERENCES
[1] T. Chen, T. Moreau, Z. Jiang, L. Zheng, E. Y an, M. Cowan, H. Shen,
L. Wang, Y . Hu, L. Ceze, C. Guestrin, and A. Krishnamurthy, “TVM:
An Automated End-to-End Optimizing Compiler for Deep Learning,” in
Proceedings of the 13th USENIX Conference on Operating Systems De-
sign and Implementation, ser. OSDI ’18. USA: USENIX Association,
2018, p. 579–594.
[2] T. Chen, L. Zheng, E. Y an, Z. Jiang, T. Moreau, L. Ceze, C. Guestrin,
and A. Krishnamurthy, “Learning to Optimize Tensor Programs,” in
Proceedings of the 32nd International Conference on Neural Information
Processing Systems, ser. NeurIPS’18. Red Hook, NY , USA: Curran
Associates Inc., 2018, p. 3393–3404.
[3] L. Zheng, C. Jia, M. Sun, Z. Wu, C. H. Y u, A. Haj-Ali, Y . Wang, J. Y ang,
D. Zhuo, K. Sen, J. E. Gonzalez, and I. Stoica, “Ansor: Generating High-
Performance Tensor Programs for Deep Learning,” in14th USENIX
Symposium on Operating Systems Design and Implementation , ser.
OSDI ’20. USENIX Association, Nov. 2020, pp. 863–879.
[4] S. Zheng, Y . Liang, S. Wang, R. Chen, and K. Sheng, “FlexTensor:
An Automatic Schedule Exploration and Optimization Framework for
Tensor Computation on Heterogeneous System,” in Proceedings of
the Twenty-Fifth International Conference on Architectural Support for
Programming Languages and Operating Systems, ser. ASPLOS ’20.
New Y ork, NY , USA: Association for Computing Machinery, 2020, p.
859–873.
[5] A. Adams, K. Ma, L. Anderson, R. Baghdadi, T.-M. Li, M. Gharbi,
B. Steiner, S. Johnson, K. Fatahalian, F. Durand, and J. Ragan-Kelley,
“Learning to Optimize Halide with Tree Search and Random Programs,”
ACM Trans. Graph., vol. 38, no. 4, Jul. 2019.
[6] B. H. Ahn, P . Pilligundla, A. Y azdanbakhsh, and H. Esmaeilzadeh,
“Chameleon: Adaptive Code Optimization for Expedited Deep Neural
Network Compilation,” inInternational Conference on Learning Repre-
sentations, 2020.
[7] M. Li, M. Zhang, C. Wang, and M. Li, “AdaTune: Adaptive Tensor
Program Compilation Made Efﬁcient,” in34th Conference on Neural
Information Processing Systems, ser. NeurIPS’20, 2020.
[8] N. V asilache, O. Zinenko, T. Theodoridis, P . Goyal, Z. DeVito, W . S.
Moses, S. V erdoolaege, A. Adams, and A. Cohen, “Tensor Compre-
hensions: Framework-Agnostic High-Performance Machine Learning
Abstractions,”arXiv preprint arXiv:1802.04730, 2018.
[9] B. Steiner, C. Cummins, H. He, and H. Leather, “V alue Learning for
Throughput Optimization of Deep Learning Workloads,” inProceedings
of MLSys Conference, 2021.
[10] Z. Jia, O. Padon, J. Thomas, T. Warszawski, M. Zaharia, and A. Aiken,
“T ASO: Optimizing Deep Learning Computation with Automatic Gen-
eration of Graph Substitutions,” in Proceedings of the 27th ACM
Symposium on Operating Systems Principles, ser. SOSP ’19. New Y ork,
NY , USA: Association for Computing Machinery, 2019, p. 47–62.
[11] TensorFlow, “XLA: Optimizing Compiler for TensorFlow,”
https://www.tensorﬂow.org/xla, [Online; accessed 19-September-2019].
[12] N. Rotem, J. Fix, S. Abdulrasool, G. Catron, S. Deng, R. Dzhabarov,
N. Gibson, J. Hegeman, M. Lele, R. Levenstein, J. Montgomery,
B. Maher, S. Nadathur, J. Olesen, J. Park, A. Rakhov, M. Smelyanskiy,
and M. Wang, “Glow: Graph Lowering Compiler Techniques for Neural
Networks,”arXiv preprint arXiv:1805.00907, 2019.
[13] Y . Xu, H. Lee, D. Chen, H. Choi, B. A. Hechtman, and S. Wang,
“Automatic cross-replica sharding of weight update in data-parallel
training,”CoRR, vol. abs/2004.13336, 2020.
[14] N. P . Jouppi, C. Y oung, N. Patil, D. Patterson, G. Agrawal, R. Bajwa,
S. Bates, S. Bhatia, N. Boden, A. Borchers, R. Boyle, P .-l. Cantin,
C. Chao, C. Clark, J. Coriell, M. Daley, M. Dau, J. Dean, B. Gelb,
T. V . Ghaemmaghami, R. Gottipati, W . Gulland, R. Hagmann, C. R.
Ho, D. Hogberg, J. Hu, R. Hundt, D. Hurt, J. Ibarz, A. Jaffey,
A. Jaworski, A. Kaplan, H. Khaitan, D. Killebrew, A. Koch, N. Kumar,
S. Lacy, J. Laudon, J. Law, D. Le, C. Leary, Z. Liu, K. Lucke,
A. Lundin, G. MacKean, A. Maggiore, M. Mahony, K. Miller, R. Na-
garajan, R. Narayanaswami, R. Ni, K. Nix, T. Norrie, M. Omernick,
N. Penukonda, A. Phelps, J. Ross, M. Ross, A. Salek, E. Samadiani,
C. Severn, G. Sizikov, M. Snelham, J. Souter, D. Steinberg, A. Swing,
M. Tan, G. Thorson, B. Tian, H. Toma, E. Tuttle, V . V asudevan,
R. Walter, W . Wang, E. Wilcox, and D. H. Y oon, “In-Datacenter
Performance Analysis of a Tensor Processing Unit,” inProceedings of
the 44th Annual International Symposium on Computer Architecture,
ser. ISCA ’17, 2017.
[15] N. P . Jouppi, D. H. Y oon, G. Kurian, S. Li, N. Patil, J. Laudon, C. Y oung,
and D. Patterson, “A Domain-Speciﬁc Supercomputer for Training Deep
Neural Networks,”Commun. ACM, vol. 63, no. 7, p. 67–78, Jun. 2020.
[16] E. Real, A. Aggarwal, Y . Huang, and Q. V . Le, “Regularized evolution
for image classiﬁer architecture search,” in Proceedings of the aaai
conference on artiﬁcial intelligence, vol. 33, 2019, pp. 4780–4789.
[17] C. Angermueller, D. Belanger, A. Gane, Z. Mariet, D. Dohan,
K. Murphy, L. Colwell, and D. Sculley, “Population-Based Black-
Box Optimization for Biological Sequence Design,” arXiv preprint
arXiv:2006.03227, 2020.
[18] C. Angermueller, D. Dohan, D. Belanger, R. Deshpande, K. Murphy,
and L. Colwell, “Model-based reinforcement learning for biological
sequence design,” inInternational Conference on Learning Represen-
tations, 2019.
[19] Y . Zhou, S. Roy, A. Abdolrashidi, D. Wong, P . Ma, Q. Xu, H. Liu,
P . Phothilimtha, S. Wang, A. Goldie, A. Mirhoseini, and J. Laudon,
“Transferable Graph Optimizers for ML Compilers,” inProceedings of
International Conference on Neural Information Processing Systems, ser.
NeurIPS’20, 2020.
[20] J. Schulman, F. Wolski, P . Dhariwal, A. Radford, and O. Klimov, “Proxi-
mal Policy Optimization Algorithms,”CoRR, vol. abs/1707.06347, 2017.
[21] J. Roesch, S. Lyubomirsky, M. Kirisame, L. Weber, J. Pollock, L. V ega,
Z. Jiang, T. Chen, T. Moreau, and Z. Tatlock, “Relay: A High-Level
Compiler for Deep Learning,”arXiv preprint arXiv:1904.08368, 2019.
[22] S. J. Kaufman, P . M. Phothilimthana, Y . Zhou, C. Mendis, S. Roy,
A. Sabne, and M. Burrows, “A Learned Performance Model for Tensor
Processing Units,” in Proceedings of Machine Learning for Systems,
2021.
[23] R. C. Whaley and J. J. Dongarra, “Automatically Tuned Linear Alge-
bra Software,” in Proceedings of the 1998 ACM/IEEE Conference on
Supercomputing, ser. SC ’98, 1998.
[24] M. Frigo and S. G. Johnson, “FFTW: an adaptive software architecture
for the FFT,” in Proceedings of the 1998 IEEE International Confer-
ence on Acoustics, Speech and Signal Processing, ICASSP ’98 (Cat.
No.98CH36181), 1998.
[25] G. Fursin, Y . Kashnikov, A. W . Memon, Z. Chamski, O. Temam,
M. Namolaru, B. Mendelson, A. Zaks, E. Courtois, F. Bodin, P . Barnard,
E. Ashton, E. Bonilla, J. Thomson, C. Williams, and M. O’Boyle,
“Milepost GCC: Machine Learning Enabled Self-tuning Compiler,”
International Journal of P arallel Programming, vol. 39, pp. 296–327,
2011.
[26] J. Ansel, C. Chan, Y . L. Wong, M. Olszewski, Q. Zhao, A. Edelman,
and S. Amarasinghe, “PetaBricks: A Language and Compiler for Algo-
rithmic Choice,” inProceedings of the 30th ACM SIGPLAN Conference
on Programming Language Design and Implementation, ser. PLDI ’09.
Association for Computing Machinery, 2009.
[27] P . M. Phothilimthana, J. Ansel, J. Ragan-Kelley, and S. Amarasinghe,
“Portable Performance on Heterogeneous Architectures,” inProceedings
of the Eighteenth International Conference on Architectural Support
for Programming Languages and Operating Systems, ser. ASPLOS ’13.
Association for Computing Machinery, 2013.
[28] J. Ansel, S. Kamil, K. V eeramachaneni, J. Ragan-Kelley, J. Bosboom,
U. O’Reilly, and S. Amarasinghe, “OpenTuner: An extensible frame-
work for program autotuning,” in2014 23rd International Conference
on P arallel Architecture and Compilation T echniques, ser. P ACT ’14,
2014, pp. 303–315.
[29] J. Ragan-Kelley, C. Barnes, A. Adams, S. Paris, F. Durand, and
S. Amarasinghe, “Halide: A language and compiler for optimizing
parallelism, locality, and recomputation in image processing pipelines,”
in Proceedings of the 34th ACM SIGPLAN Conference on Programming
Language Design and Implementation, ser. PLDI ’13. Association for
Computing Machinery, 2013.
[30] T.-M. Li, M. Gharbi, A. Adams, F. Durand, and J. Ragan-Kelley,
“Differentiable programming for image processing and deep learning
in halide,”ACM Trans. Graph., vol. 37, no. 4, Jul. 2018.
[31] R. T. Mullapudi, A. Adams, D. Sharlet, J. Ragan-Kelley, and K. Fata-
halian, “Automatically Scheduling Halide Image Processing Pipelines,”
ACM Trans. Graph., vol. 35, no. 4, Jul. 2016.
[32] K. Hegde, P .-A. Tsai, S. Huang, V . Chandra, A. Parashar, and C. W .
Fletcher, “Mind Mappings: Enabling Efﬁcient Algorithm-Accelerator
Mapping Space Search,” inProceedings of the 26th ACM International
Conference on Architectural Support for Programming Languages and
Operating Systems, ser. ASPLOS 2021. New Y ork, NY , USA: Asso-
ciation for Computing Machinery, 2021, p. 943–958.
13
[33] J. Zhao, B. Li, W . Nie, Z. Geng, R. Zhang, X. Gao, B. Cheng, C. Wu,
Y . Cheng, Z. Li, P . Di, K. Zhang, and X. Jin, “AKG: Automatic
Kernel Generation for Neural Processing Units Using Polyhedral Trans-
formations,” inProceedings of the 42nd ACM SIGPLAN International
Conference on Programming Language Design and Implementation, ser.
PLDI ’21. New Y ork, NY , USA: Association for Computing Machinery,
2021, p. 1233–1248.
[34] W . Jung, T. T. Dao, and J. Lee, “DeepCuts: A Deep Learning Op-
timization Framework for V ersatile GPU Workloads,” inProceedings
of the 42nd ACM SIGPLAN International Conference on Programming
Language Design and Implementation, ser. PLDI ’21. New Y ork, NY ,
USA: Association for Computing Machinery, 2021, p. 190–205.
[35] Z. Jia, J. Thomas, T. Warszawski, M. Gao, M. Zaharia, and A. Aiken,
“Optimizing DNN Computation with Relaxed Graph Substitutions,” in
Proceedings of MLSys Conference, 2019.
[36] L. Ma, Z. Xie, Z. Y ang, J. Xue, Y . Miao, W . Cui, W . Hu, F. Y ang,
L. Zhang, and L. Zhou, “Rammer: Enabling Holistic Deep Learning
Compiler Optimizations with rTasks,” in 14th USENIX Symposium
on Operating Systems Design and Implementation , ser. OSDI ’20.
USENIX Association, Nov. 2020, pp. 881–897.
[37] Nvidia, “Nvidia TensorRT: Programmable Inference Accelerator,” 2017.
[38] Z. Zheng, P . Zhao, G. Long, F. Zhu, K. Zhu, W . Zhao, L. Diao, J. Y ang,
and W . Lin, “FusionStitching: Boosting Memory Intensive Computations
for Deep Learning Workloads,”arXiv preprint arXiv:2009.10924, 2020.
[39] Y . Liu, Y . Wang, R. Y u, M. Li, V . Sharma, and Y . Wang, “Optimizing
CNN Model Inference on CPUs,” in2019 USENIX Annual T echnical
Conference), ser. A TC ’19. Renton, W A: USENIX Association, Jul.
2019, pp. 1025–1040.
[40] C. Dubach, J. Cavazos, B. Franke, G. Fursin, M. F. O’Boyle, and
O. Temam, “Fast Compiler Optimisation Evaluation Using Code-feature
Based Performance Prediction,” inProceedings of the 4th International
Conference on Computing Frontiers, ser. CF ’07, 2007.
[41] Z. Jia, S. Lin, M. Gao, M. Zaharia, and A. Aiken, “Improving the Accu-
racy, Scalability, and Performance of Graph Neural Networks with Roc,”
in Proceedings of MLSys Conference, I. Dhillon, D. Papailiopoulos, and
V . Sze, Eds., vol. 2, 2020, pp. 187–198.
[42] R. Baghdadi, M. Merouani, M.-H. Leghettas, K. Abdous, T. Arbaoui,
K. Benatchba, and S. Amarasinghe, “A Deep Learning Based Cost
Model for Automatic Code Optimization,” inProceedings of MLSys,
2021.
14
APPENDIX
A. Additional Evaluation Results on Search Strategies
Fig. 10: Fusion autotuning: optimization trajectories for one
example benchmark. Shaded areas show 95% conﬁdence in-
tervals over ten replicas.
In the experiment that compares different search strategies
in Section IV -C, we collect the speedup trajectories of various
search strategies over time on one example benchmark, shown
in Fig. 10. We notice that RL has very high variance on
most benchmarks compared to the rest, as also evidenced in
Fig. 10. This means in the setting where we can run 10 search
replicas in parallel and take the best, RL often achieves the
most speedup compared to the others at the same number of
samples, especially earlier in the search. Future work may
consider incorporating prior knowledge, e.g. pretraining the
models on a diverse set of graphs, to improve the performance
and the consistency of model-based optimizations.
B. Semantic Rules for XTAT-M
In this section, we describe (a) the implementation of
GenerateCandidatesfor simulated annealing, and (b) brieﬂy
present the semantics of operations in XTAT-M and derive
two search steps in a joint layout-fusion tuning for the graph
in Fig. 2.
a) Implementation of GenerateCandidates for simulated
annealing: As described in Section II-C, simulated anneal-
ing uses a mutation operator, e.g., single-node, to mutate
a given conﬁguration and accept a new one based on an-
nealing temperature. Fig. 11 presents an implementation of
GenerateCandidatesfor simulated annealing. This implemen-
tation utilizes a routine,Mu ta te, which implements the muta-
tion operator, generating a new conﬁguration from the current
conﬁguration.
b) Semantics of operations inXTAT-M: Fig. 12 presents
the semantics ofInferConﬁg, UpdateStore, Mutate (based on
Fig. 11), and ApplyOpt. Speciﬁcally, to keep the discussion
concise, we focus on a single-node mutation-based search
strategy applied to tuning layout assignment and operator
fusion optimizations.
• InferConﬁg: This routine updates a conﬁgurationconﬁg
for an optimizationop tid to be compatible with the graph
1: function GENERA TECANDIDA TES(op tid ,C)
2: c ←C[0] // C has one candidate.
3: c′←c // Init conﬁgs for all optimizations.
4: // Mutate only the conﬁg for optimizationop tid .
5: c′.conﬁgs[optid] ←Mutate(c.conﬁgs[optid])
6: return {c′}
7: end function
Fig. 11: Implementation ofGenerateCandidatesfor simulated
annealing.
g, an input to the optimization pass.InferConﬁgidentiﬁes
the nodes in g and conﬁg that have the same ﬁngerprint
fp(n) and carries forward the conﬁgurations inconﬁg for
such nodes. For the other nodes in g, it pulls the best
conﬁgurations found so far from the global ConfStore
if present; otherwise, it generates optimization-speciﬁc
defaults.
• UpdateStore: When a search candidate conﬁguration is
evaluated and it performs superior to prior observed
evaluations, then the candidate conﬁguration is captured
by updating the globalConfStore.
• Mutate: Mu ta teconstructs a new conﬁguration from the
current conﬁguration. We present a simpliﬁed rule for a
single-node mutation, soMu ta tereturns a conﬁguration
change for a single node in the graph.
• ApplyOpt: As described in Section II-B,ApplyOpt trans-
forms an input graph by applying the optimization.
A derivation of the joint layout-fusion tuning performed on
the graph in Fig. 2 is presented in Fig. 13. For conciseness, we
do not describe all the tensors involved in the graph. Specif-
ically, we focus on the mutation of the layout conﬁgurations
for tensors,t5 and t6, in the search step for layout, and focus
on the add node in the search step for fusion. In each search
step, the Mu ta teoperation for layout begins with the graph
g, applies a layout mutation, and infers theconﬁg for fusion.
Additionally, the ﬁrst step performs theMu ta teoperation for
fusion. Both steps apply the inferred fusion conﬁguration and
execute the output graph. Based on the evaluation, each search
step updates the global store, i.e.,ConfStore. In the derivation,
we note that applying a search candidate for layout in step
1 results in the addition of a copy node, while no copy
is introduced when a different search candidate for layout is
explored in step 2.
15
Fig. 12: Semantics of functionsInferConﬁg, UpdateStore, Mutate (for a single node), andApplyOpt in XTAT-M.
infer-conﬁg
fr o m_con f ig:= {tuple (fp (n),con f ig[ fp (n)])}, ∀(n).(n ∈g.Nodes ∧fp (n) ∈con f ig)
fr o m_store := {tuple (fp (n),Con f Store[op tid ][fp (n)])}, ∀(n).(n ∈g.Nodes ∧fp (n) ̸∈con f ig∧fp (n) ∈Con f Store[op tid ])
fr o m_init := {tuple (fp (n),GetInitCon f ig(op tid ,g)[n])}, ∀(n).(n ∈g.Nodes ∧fp (n) ̸∈con f ig∧fp (n) ̸∈Con f Store[op tid ])
E ⊢ con f ig′:= In f erCon f ig(con f ig,g,Con f Store[op tid ]) ⇓E{con f ig′↦→{ fr o m_con f ig∪fr o m_store∪fr o m_init}}
update-store
new_con f ig:= {tuple (fp (n),con f igs[op tid ][fp (n)])}, ∀(n).(fp (n) ∈con f igs[op tid ])
old_con f ig:= {tuple (fp (n),Con f Store[op tid ][fp (n)])}, ∀(n).(fp (n) ̸∈con f igs[op tid ]∧fp (n) ∈Con f Store[op tid ])
E ⊢U pdateStore(Con f Store,con f igs) ⇓E{Con f Store[op tid ] ↦→{old_con f ig∪new_con f ig},∀(op tid ∈Con f Store)}
Mutate layout
{n1}∈ Nodes n 1 := {op,t1} op tid := layoutID
con f ig[n1] := layoutt1 layout′t1 ∈rand(LayoutGen(t1)) ∧ (layoutt1 ̸= layout′t1 )
E ⊢ con f ig′:= Mu ta te(con f ig) ⇓E{con f ig′↦→{(n1,layout′t1 )}}
apply-layout1
g; {n1,n2}∈ Nodes; {(n1,n2)}∈ Edg es∧(n2.tin == n1.tout)
con f ig[n1] := l1 con f ig[n2] := l2 E ⊢ l1 == l2 ⇓FALSE
E ⊢ g := ApplyOp t(layoutID,g,con f ig) ⇓E{g ↦→{g ∪{n1.con f ig.layouttout = l1,n2.con f ig.layouttin = l2}∪{Insert(copy,n1,n2)}}
apply-layout2
g; {n1,n2}∈ Nodes; {(n1,n2)}∈ Edg es∧(n2.tin == n1.tout)
con f ig[n1] := l1 con f ig[n2] := l2 E ⊢ l1 == l2 ⇓TR UE
E ⊢ g := ApplyOp t(layoutID,g,con f ig) ⇓E{g ↦→{g ∪{n1.con f ig.layouttout = l1},n2.con f ig.layouttin = l2}}}
Mutate fusion {n1}∈ Nodes n 1 := {op,t1} op tid := f usionID con f ig[n1] := 0
E ⊢ con f ig:= Mu ta te(con f ig) ⇓E{con f ig↦→{(n1,1)}}
apply-fusion g; {n1,n2}∈ Nodes; {(n1,n2)}∈ Edges con f ig[n1] := 1
E ⊢ g := ApplyOp t(f usionID,g,con f ig)) ⇓E{g ↦→{g \{n1,n2}∪{n′
2 := n1 ◦n2}}
Fig. 13: Two search steps in a joint layout-fusion tuning for the graphs in Fig. 2: left for step 1 and right for step 2. The
neighborhood size for the node’s ﬁngerprint calculation used byConfStoreis set to 0 (ignoring neighborhood) in this example.
{n1,n2,n3}∈ Nodes {(n1,n3),(n1,n4),(n2,n4)}∈ Edg es n 1 := {add,t1} n2 := {max,t2}
n3 := {reshape,t3,t4} n4 := {conv,t5,t6,t7} g := c.graphs(layoutID)
LayoutGen(t1) := LayoutGen(t3) := LayoutGen(t5) := {{0,2,1},{2,1,0},{1,0,2}} LayoutGen(t2) := LayoutGen(t6) := {{{0,2,1},{2,0,1}}
Step 1: Tune layout & fusion
E ⊢ con f ig:= Mu ta te(c.con f igs(layoutID)) ⇓E{con f ig↦→{(n4,layoutt5 ↦→{0,1,2}),(n4,layoutt6 ↦→{0,2,1})}}
E ⊢ g′:= ApplyOp t(layoutID,g,con f ig) ⇓E{g′↦→{g ∪{n4.con f ig.layoutt5 = {0,1,2},n4.con f ig.layoutt6 = {0,2,1}}∪{Insert(copy,n1,n4)}}
E ⊢ con f ig′:= In f erCon f ig(c.con f igs(f usionID),g′,Con f Store(f usionID)) ⇓E{con f ig′↦→{(n1,0),(n2,0),(n3,0),(n4,0),(copy,0)}}
E ⊢ con f ig′:= Mu ta te(con f ig′) ⇓E{con f ig′↦→{(n1,1)}
E ⊢ g′′:= ApplyOp t(f usionID,g′,con f ig′) ⇓E{g′′↦→{g′\{n1,n3,copy}∪{n1 ◦n3,n1 ◦copy}}}
ExecuteGraph(g′′)
E ⊢U pdateStore(Con f Store{layoutID : con f ig, f usionID: con f ig′})a
⇓E{Con f Store↦→{layoutID : {(n4,layoutt5 ↦→{0,1,2}),(n4,layoutt6 ↦→{0,2,1})}, f usionID: {(n1,1),(n2,0),(n3,0),(n4,0),(copy,0)}}}
Step 2: Tune layout (fusion conﬁg is inferred)
E ⊢ con f ig:= Mu ta te(c.con f igs(layoutID)) ⇓E{con f ig↦→{(n4,layoutt5 ↦→{1,0,2}),(n4,layoutt6 ↦→{0,2,1})}}
E ⊢ g′:= ApplyOp t(layoutID,g,con f ig) ⇓E{g′↦→{g ∪{n4.con f ig.layoutt5 = {1,0,2},n4.con f ig.layoutt6 = {0,2,1}}}
E ⊢ con f ig′:= In f erCon f ig(c.con f igs(f usionID),g′,Con f Store(f usionID)) ⇓E{c.con f igs(f usionID) ↦→{(n1,1),(n2,0),(n3,0),(n4,0)}}
E ⊢ g′′′:= ApplyOp t(f usionID,g′,con f ig′) ⇓E{g′′′↦→{g′\{n1,n3,n4}∪{n1 ◦n3,n1 ◦n4}}}
ExecuteGraph(g′′′)
E ⊢U pdateStore(Con f Store,{layoutID : con f ig, f usionID: con f ig′})b
⇓E{Con f Store↦→{layoutID : {(n4,layoutt5 ↦→{1,0,2}),(n4,layoutt6 ↦→{0,2,1})}, f usionID: {(n1,1),(n2,0),(n3,0),(n4,0)}}}
(g,2 steps f usion+layout tune) ⇝Optimal(g′′,g′′′)
acontinues in the next line
bcontinues in the next line
16

BenchPress: A Deep Active Benchmark Generator
Foivos Tsimpourlas
Meta AI Research
University of Edinburgh
F.Tsimpourlas@sms.ed.ac.uk
Pavlos Petoumenos
University of Manchester
pavlos.petoumenos@manchester.ac.uk
Min Xu
Meta AI Research
m1n@fb.com
Chris Cummins
Meta AI Research
cummins@fb.com
Kim Hazelwood
Meta AI Research
kimhazelwood@fb.com
Ajitha Rajan
University of Edinburgh
arajan@inf.ed.ac.uk
Hugh Leather
Meta AI Research
hleather@fb.com
ABSTRACT
Finding the right heuristics to optimize code has always been a
difficult and mostly manual task for compiler engineers. Today this
task is near-impossible as hardware-software complexity has scaled
up exponentially. Predictive models for compilers have recently
emerged which require little human effort but are far better than
humans in finding near optimal heuristics. As any machine learning
technique, they are only as good as the data they are trained on but
there is a severe shortage of code for training compilers. Researchers
have tried to remedy this with code generation but their synthetic
benchmarks, although thousands, are small, repetitive and poor
in features, therefore ineffective. This indicates the shortage is of
feature quality more than corpus size. It is more important than
ever to develop a directed program generation approach that will
produce benchmarks with valuable features for training compiler
heuristics.
We develop BenchPress, the first ML benchmark generator for
compilers that is steerable within feature space representations
of source code. BenchPress synthesizes compiling functions by
adding new code in any part of an empty or existing sequence
by jointly observing its left and right context, achieving excel-
lent compilation rate. BenchPress steers benchmark generation
towards desired target features that has been impossible for state
of the art synthesizers (or indeed humans) to reach. It performs
better in targeting the features of Rodinia benchmarks in 3 differ-
ent feature spaces compared with (a) CLgen - a state of the art ML
synthesizer, (b) CLSmith fuzzer, (c) SRCIROR mutator or even (d)
human-written code from GitHub. BenchPress is the first gener-
ator to search the feature space with active learning in order to
generate benchmarks that will improve a downstream task. We
show how using BenchPress, Grewe’s et al. CPU vs GPU heuristic
model can obtain a higher speedup when trained on BenchPress’s
We release BenchPress’s source code on a publicly available repository at
https://github.com/fivosts/BenchPress.
This work was supported by the Engineering and Physical Sciences Research Council
(grant EP/L01503X/1), EPSRC Centre for Doctoral Training in Pervasive Parallelism at
the University of Edinburgh, School of Informatics.
PACT ’22, October 10–12, 2012, Chicago,IL
2018. ACM ISBN 978-1-4503-XXXX-X/18/06. . . $15.00
benchmarks compared to other techniques. BenchPress is a pow-
erful code generator: Its generated samples compile at a rate of 86%,
compared to CLgen’s 2.33%. Starting from an empty fixed input,
BenchPress produces 10×more unique, compiling OpenCL bench-
marks than CLgen, which are significantly larger and more feature
diverse.
ACM Reference Format:
Foivos Tsimpourlas, Pavlos Petoumenos, Min Xu, Chris Cummins, Kim
Hazelwood, Ajitha Rajan, and Hugh Leather. 2018. BenchPress: A Deep
Active Benchmark Generator. In PACT ’22: International Conference on Par-
allel Architectures and Compilation Techniques (PACT), October 10–12, 2018,
Chicago, IL. ACM, New York, NY, USA, 12 pages.
1 INTRODUCTION
Estimating compiler optimization heuristics through predictive
modeling has been shown to outperform human experts and reduce
development time in previous studies [6, 8].
However, designing effective predictive models requires exten-
sive and diverse training data to help learn accurate compiler opti-
mization heuristics. Training data typically take the form of static
code features in benchmarks and their label for optimization heuris-
tics [18]. Static code features are used to characterize the behaviour
of programs and they are typically derived at the (1) Syntax level -
by traversing the Abstract Syntax Tree (AST) (e.g. count number of
operations in a function) or the (2) Intermediate Representation (IR)
of a program - with the help of compiler passes (e.g. count of each
LLVM-IR [25] instruction type). As shown in Figure 1, predictive
models learn to infer optimal heuristics for a program based on
program features and inputs.
Predictive models would be an elegant solution for generat-
ing compiler heuristics, if only we did not face an acute shortage
of benchmarks, both in quantity and diversity [ 8, 36]. The aver-
age number of benchmarks used in performance tuning papers is
17 [2, 5, 6, 14, 22, 36]. Well established areas of machine learning
meanwhile rely on orders of magnitude more data. ImageNet [10],
for example, holds 1.3M images for training and 50k for valida-
tion. A shortage of benchmarks in training leads to poor feature
space coverage that degrades the performance of predictive mod-
els [8, 16]. Cummins et al. [6] show that enhancing datasets with
synthetic benchmarks, using their tool CLgen, can improve the
arXiv:2208.06555v2  [cs.AI]  16 Aug 2022
PACT ’22, October 10–12, 2012, Chicago,IL Foivos Tsimpourlas, Pavlos Petoumenos, Min Xu, Chris Cummins, Kim Hazelwood, Ajitha Rajan, and Hugh Leather
Figure 1: Training pipeline of a predictive model.
performance of the Grewe et al. [ 18] partitioning task by 1.27x.
However, a case study by Goens et al. [ 16] shows that CLgen’s
synthetic benchmarks lack feature diversity with most generated
benchmarks being small, repetitive and with little new features
compared to existing benchmarks. We aim to address this with
BenchPress, a targeted benchmark generator, that can generate
compiler benchmarks with a desired set of features. In this work,
we focus on generating OpenCL benchmarks, as predictive models
for heterogeneous systems is a rapidly advancing field and training
examples for them are very sparse.
We developBenchPress, a BERT-based OpenCL benchmark gen-
erator [11, 34] that targets and synthesizes benchmarks in desired
parts of the feature space. We use active learning to choose parts of
the feature space and beam search to steerBenchPress’s generated
samples towards the requested features. We trainBenchPress with
OpenCL code samples that we collect by mining BigQuery [17] and
GitHub directly using its API [15]. We support composite data types
and calls to user-defined functions in our dataset and benchmark
generation. BenchPress is a bidirectional generative model and
learns to generate code in any part of a sequence by jointly con-
sidering left and right context. We achieve this with a new learnt
token, the [HOLE], which hides a sequence from the input, whose
length is unknown to BenchPress during training. BenchPress
learns to fill [HOLE] by iteratively predicting an arbitrary number
tokens that are likely to lead to a compiling function.
BenchPress outperforms CLgen in the task of undirected pro-
gram generation from a fixed input feed, generating 10×more
unique OpenCL kernels that are 7.5×longer on average, with a
compilation rate of 86% compared to CLgen’s 2.33%. BenchPress
strongly outperforms benchmark synthesizers CLgen, CLSmith [1,
37], and human written code from GitHub in reaching close to the
features of Rodinia benchmarks, developed by compiler experts.
Finally, BenchPress uses active learning, specifically query by com-
mittee [32], to search the feature space and find missing features to
improve Grewe’s et al. [18] CPU vs GPU heuristic. Enhancing the
heuristic’s dataset with BenchPress’s benchmarks improves the
heuristic’s speedup relative to the optimal static decision by 50%,
increasing it from 4% to 6%, when the maximum possible speedup
for this task is 12%.
In this paper, we present the following contributions:
(1) We are the first to develop a feature-space agnostic, steerable
code generator towards desired program features.
(2) We develop an automated approach to rank the feature space
of downstream tasks with active learning.
(3) We enable bidirectional source code generation by inserting
[HOLE] tokens in any part of a sequence.
2 MOTIVATION
Figure 2 shows a two-dimensional slice of the Grewe’s et al. [18]
feature space: number of computational instructions vs number of
memory instructions. Figure 2 also shows how the OpenCL bench-
marks found in the Rodinia suite map into this plane, represented
as purple diamonds. We find much of this two dimensional space
is uncovered. 54 of the 58 Rodinia examples cluster in the lower
left corner, the rest of the space having only four examples. Any
optimization decision for programs in this area of the space would
not be accurate due to lack of representative examples.
Figure 2: # Memory operations and # computational instruc-
tions for (a) Rodinia benchmarks in purple diamonds and (b)
CLgen’s samples in red dots. Generating samples with miss-
ing features is vital for predictive modeling’s performance.
CLgen attempted to address this problem by automatically gen-
erating more training examples. However, the generated kernels
lacked feature diversity and provided even poorer coverage of the
feature space. Figure 2 represents their position in the 2D space as
red dots. Almost all of them are concentrated in a corner covering
a small percentage of the feature space. While CLgen can generate
hundreds of millions of unique kernels, almost all of them will fail to
compile. As the probability of having at least one illegal token in the
kernel body increases with the number of tokens, only tiny kernels
are valid. In our experiments in Section 5, the longest compiling
CLgen kernel had 8 lines and 102 tokens. Given the small number
of tokens in valid kernels, there is a high degree of repetitiveness in
the generated corpus, not only in terms of features but also in terms
of structure and functionality. As a result, this approach is not well
suited to augmenting the training set with diverse feature bench-
marks. There is a compelling need to generate training points for
uncovered regions of the feature space and we attempt to address
this need with BenchPress. In the following Sections, we discuss
our approach and evaluation of BenchPress, comparing it to the
existing state-of-the art for feature space coverage.
3 APPROACH
We present BenchPress, a deep learning model for directed com-
piler benchmark generation. BenchPress is the first steerable
Short Title PACT ’22, October 10–12, 2012, Chicago,IL
synthesizer that can synthesize compiling functions with target
features. BenchPress steers its generation with a feature space-
agnostic beam search algorithm to search the space and steer
BenchPress’s generation towards the target features. Given a down-
stream task, BenchPress learns what features to target in order to
improve its performance by searching the space with active learn-
ing. BenchPress’s language model is based on BERT [11], which
we transform into a generative model.
The key feature in BenchPress that enables bidirectional code
generation is a new token, namely, the [HOLE] token. We train
BenchPress to learn and understand how to iteratively fill holes of
unknown length that can be found in any part of an input sequence
by conditioning it on the left and right context of the[HOLE]. Later,
this approach enables us to use beam search and steer benchmark
generation into the feature space iteratively as it can regress to
previously generated benchmarks with new holes and produce
newer samples with better features.
Figure 3 illustrates an overview of our approach. BenchPress
consists of three main components:
(1) Learning corpus collection and processing.
(2) Source code language modeling.
(3) Feature space search and benchmark generation.
We discuss each step in the following subsections.
Figure 3: BenchPress’s high-level approach.
3.1 Learning Corpus
Modeling source code accurately requires large amounts of data [26]
similarly to other deep learning tasks. We develop a tool to collect
data from BigQuery’s GitHub dataset [17]. We also use GitHub’s
API [15] and mine directly extra repositories that are not included
in BigQuery.
There are a few innovations in how we pre-process the code
compared to previous works. First, we inline included header files
recursively into source files to resolve type dependencies. Addi-
tionally, we automatically extract custom data types (e.g. struct,
typedef) and utility functions found in the unprocessed corpus
and place them into header files that are accessible throughout
BenchPress’s pipeline. This way, we resolve most type dependen-
cies by retaining the functionality and semantics of the original,
human-written programs. These two steps enable us to increase
significantly the amount of compiling kernels we end up with in
our training dataset. Second, we isolate kernels into single instances
because BenchPress is trained on complete functions. From the
previous steps, the type dependencies of each kernel are known
and we automatically provide them to the compiler, retaining their
compilability. Finally, we compile all kernels with Clang and reject
those that do not compile.
Next, we re-write identifiers by randomly sampling the alphabet,
eliminating spurious naming patterns in the corpus. All kernels
are padded to BenchPress’s sequence length and kernels that are
longer than this are truncated to fit. This helpsBenchPress train its
later indices’ positional embeddings more effectively, for which we
have less training information compared to earlier indices. Finally,
we derive a tokenizer by parsing the AST of all source code. We
reserve tokens for all OpenCL keywords and all intrinsic OpenCL
function name identifiers found in the official OpenCL specifica-
tions [33]. We analyze the dataset and tokenize by word the most
common function names and custom data type identifiers that we
have collected. We encode all literals and infrequently used custom
types and functions character by character to avoid exploding the
size of the vocabulary. We define 5 meta tokens:[START], [END],
[PAD], [HOLE], [ENDHOLE]. The derived tokenizer holds in total
2,201 unique tokens.
3.2 Language Modeling
BenchPress is based on BERT [ 11], a Transformer-based model
originally designed for natural language modeling. BERT is trained
to predict words that have been randomly hidden by[MASK] tokens.
This way BERT learns fitting words with respect to their position in
a sequence and also the left and right context, i.e., the text sequence
before and after the masked token to be predicted. This type of
training helps BERT learn what words mean within a given context,
improving downstream tasks that rely on that knowledge.
While this is a useful property, it is not enough to turn BERT
into a generative model. We also want to be able to extend a kernel
by inserting an arbitrary number of tokens in arbitrary positions.
We could iteratively add a [MASK] token to get one extra token at a
time, until we have a full statement. This would be limiting. Each
time the new token would be selected based on its probability of
completing forming a plausible kernel. Every intermediate kernel in
the iterative process would have to be plausible or almost plausible,
which is not a general way for augmenting kernels.
Clusters of [MASK] tokens could allow us to insert multiple to-
kens in each iteration. This is still unsatisfactory. The number of
[MASK] tokens in the cluster biases the kind of code that will be
generated: if we ask such a generator to produce five tokens, it
will give us a five token statement that could be expected to close
this gap, not a five token sequence that could be the start of a
much longer statement. We could place the left and right context
to the edges of a sequence and fill intermediate positions with
[MASK] tokens. BenchPress could predict a vocabulary or a stop
token for a [MASK], allowing for arbitrary sequences. We test this
configuration and sample a trained model with a fixed input feed.
BenchPress is unable to learn the [MASK]s’ left and right context
conditionally, when many [MASK]s are in a sequence, which leads
to zero samples to compile or even resemble reasonable code.
PACT ’22, October 10–12, 2012, Chicago,IL Foivos Tsimpourlas, Pavlos Petoumenos, Min Xu, Chris Cummins, Kim Hazelwood, Ajitha Rajan, and Hugh Leather
What we do instead is to extend BERT’s functionality with a
new pair of learnt tokens, the [HOLE] and the [ENDHOLE]. [HOLE]
follows the same logic with [MASK], however the number of tokens
that have been hidden behind it is unknown to the model during
training. The model only learns to predict the first token of an
arbitrarily long missing sequence. At inference-time, we iteratively
predict the first token of the remaining sequence and re-insert it
just before the [HOLE]. This way BenchPress learns to generate
arbitrarily large code sequences within any part of a sequence.
Figure 4: When a [HOLE] is inserted to a kernel at a random
index, it hides a random number of tokens, unknown to
BenchPress. On this example, BenchPress learns to predict
the first hidden token, p.
Figure 4 shows how a[HOLE] is inserted into a function to create
a datapoint. A random starting index and a random length are
selected. The choice of index and length are only restricted by a
potential overlap of the prospective hidden sequence with any of
the other meta token or the maximum hole length that is defined
as a training parameter for the architecture as a percentage of
each function’s length. When the specifications of a hole have been
settled, the hidden sequence is discarded. Only the first token of it is
kept as the target prediction for that hole. A hole can also represent
an empty sequence, i.e. hiding 0 tokens. In this case, the target
prediction during training is [ENDHOLE]. The training instances
are randomly generated on demand, the entire space of possible
instances is too large to be pre-generated. In this paper, we only
insert 1 hole per training instance forBenchPress to learn. Multiple
holes could be used during training, but this is not needed during
BenchPress’s current benchmark generation task.
3.3 Benchmark Generation
BenchPress’s synthesizer operates as a generative model with the
help of [HOLE] / [ENDHOLE] tokens. It receives an input with 1 or
more [HOLE] tokens and returns a completed benchmark. For each
[HOLE], BenchPress predicts one token that fits in the sequence
at the [HOLE]’s index, with respect to its left and right context. If
the predicted token is not [ENDHOLE], it moves the [HOLE] and
all subsequent tokens one position to the right and inserts the
predicted token to the initial target index. This intermediate kernel
is iteratively provided as an input for the next token prediction and
the process is repeated until BenchPress predicts [ENDHOLE]. This
marks a [HOLE] is complete and the final sample is returned, as
shown in Figure 5.
On its own, this process only augments kernels. We also make
it the first to target desired parts of a feature space by repeatedly
Figure 5: During sampling, BenchPress receives an input and
predicts iteratively the fitting tokens. BenchPress predicts
[ENDHOLE] to indicate a [HOLE] is complete.
generating kernels, selecting the ones closer to the target features,
inserting new holes, and generating new augmented kernels. We
use beam search to steer generation.
Given a target feature vector, BenchPress samples a starting,
fixed input feed ‘kernel void [HOLE]’ and yields a collection of
starting benchmarks. We reject benchmarks that do not compile
and for the remaining we measure the Euclidean distance between
their feature vectors and the target features. We select thetop-K can-
didates that have the shortest distance from the target and we use
them as inputs for the next generation. To improve diversity among
promoted benchmarks we introduce randomness in the selection
of top-K candidates: Each top-K sample, has a fixed probability
𝑝 = 0.15 to be replaced by another random candidate of its genera-
tion. BenchPress lazily creates multiple different input instances
for each selected candidate by placing a random [HOLE] of random
length in order to synthesize a new sample. BenchPress generates
a successive collection of benchmarks, of which K compiling ones
with the shortest distance from the target again are selected with
p-randomness and used as inputs. This search continues until a
sample achieves a distance of 0 from the target, or until a threshold
of generations (i.e. beam search depth) is exhausted. BenchPress
returns the closest benchmark to the target’s features along with
all beam search’s intermediate benchmarks that cover the model’s
traversal of the feature space starting from the origin and ending
near the target features. For the benchmark synthesis process, we
use categorical sampling with temperature to sampleBenchPress’s
probabilities. The sampling temperature, beam search’s width K
and depth are defined as sampling parameters.
In the worst case, BenchPress’s directed program generation is
slow, ranging from a few seconds to one hour, as it typically requires
thousands of language model inferences. However, BenchPressis
the first program synthesizer that can target a set of desired program
features.
3.4 Feature Space Search
A steerable synthesizer allows the generation of benchmarks with
desired features. However, the automatic selection of those parts
of the feature space that are worth targeting is challenging and
depends on the downstream task.
Short Title PACT ’22, October 10–12, 2012, Chicago,IL
BenchPress attempts to solve this by searching the feature space
with query by committee [32], a well-known active learning tech-
nique. We implement a committee of (a) 7 NN, (b) 7 k-NN and (c)
7 K-means models. We set their initial state by passively training
on a small portion of the downstream task’s data. We sample the
committee with thousands of random points in the space, we collect
the predicted labels and measure the entropy for each sample. The
entropy shows the level of uncertainty among the committee about
the predicted label of a given point and is defined as:
𝐻 = −
𝑙𝜖𝐿∑︁
(𝑝(𝑙)∗log(𝑝(𝑙))) (1)
where 𝐿is the set of all predicted labels and 𝑝(𝑙)the probabil-
ity of label 𝑙 in the committee’s prediction set for a given input.
The highest entropy point is an important feature vector to target
and BenchPress steers benchmark generation towards it with the
approach explained in 3.3. We collect the labels of generated bench-
marks and we train incrementally the committee with them. Then,
we sample it to find the next highest entropy point. We continue
this process until we saturate the feature space.BenchPress’s com-
mittee is agnostic to the downstream task or the feature space and
its I/O dimensions are hyper-parameters selected with respect to
the task’s feature and prediction dimensions.
4 EXPERIMENTAL SETUP
We describe the configurations used in training BenchPress, and
the parameters used in evaluation, namely (1) Feature Spaces - we
use three different representations of program features, (2) Target
Benchmarks - We use Rodinia benchmarks [5] and their features as
the target for synthesis by BenchPress, (3) Comparison to SOTA -
we compareBenchPress with code synthesizers and human written
code in improving Grewe’s et al. heuristic model.
4.1 Platforms
We trainBenchPress and conduct all our experiments on two 64-bit
systems each having one Intel Xeon E5-2620 16-core CPU, 2x Nvidia
GeForce GTX 1080 GPU and 32 Gigabytes of RAM. We use Ubuntu
18.04, PyTorch 1.9.1 [ 29], CUDA version 11.4 and Nvidia driver
version 510.47.03. We use Clang-10 as BenchPress’s compiler and
LLVM-10 to compile and execute InstCount and Autophase [ 20]
extracting tools. For compatibility reasons, we are required to use
Clang LibTooling from LLVM-6 to execute Grewe’s et al. [18] feature
extractor.
4.2 Language Modeling for source code
We collect OpenCL code from GitHub and split it into single func-
tion instances. We ensure no kernels that come from benchmarks
suites used in the evaluation are included in our corpus. We pre-
process text, re-write variables and reject OpenCL kernels that do
not compile. In total we mine 63,918 OpenCL kernels across 12,860
GitHub repositories and we successfully compile 19,637 of them
(31% compilation rate).
We train BenchPress on our OpenCL Corpus for 10M steps
with a batch size of 32. For BenchPress’s BERT model parameters,
we select 2 hidden layers, 12 attention heads. We set intermediate
size, hidden size and max position embeddings to 768. We set the
maximum length of holes to be 90% of a kernel’s token length, i.e. a
hole can hide almost all tokens of a training instance. We optimize
the model using Adam optimizer with a learning rate that reaches a
maximum of 45𝑥10−6 after 20,000 warmup steps and decays linearly
over the remaining training steps. We trainBenchPress’s language
model to a final loss value of 0.28.
4.3 Feature Spaces
Compiler predictive models use static code features to represent
programs and learn optimisation heuristics. A vector of indepen-
dent characteristics represent a single program. Each of them are
typically an integer or float value. Features are extracted at the
Syntax level by traversing the AST or at the IR level using the com-
piler’s middle end (e.g. LLVM-IR). A feature space is the collection
of all possible program feature vectors.
BenchPress is a generative model that can be steered to gen-
erate samples for a desired part of the feature space. We evaluate
BenchPress on three source feature representations we find across
the literature, (a) Syntax-level Grewe’s et al. features [18], (b) IR-
level LLVM-InstCount [25] and (c) IR-level Autophase [20].
Grewe’s et al. features are extracted with Clang’s LibTooling
and used to train their predictive model on the CPU vs GPU task
for OpenCL kernels. This feature space holds 8 dimensions. 4 di-
mensions describe the number of 1) computational, 2) relational, 3)
atomic and 4) memory access instructions. The feature space also
counts the different type of memory instructions, local memory or
coalesced. Finally, the computational to memory and coalesced to
memory ratios are defined.
InstCount is a standard pass provided by LLVM-IR framework
and used in Compiler Gym by Cummins et al. [7]. InstCount holds
70 dimensions: 67 dimensions each counting all 67 LLVM-IR in-
struction types and total number of 1) instructions, 2) basic blocks
and 3) functions. Autophase by Huang et al. [20] holds 56 dimen-
sions. While many of the features used in Autophase are shared
with InstCount, they introduce new ones such as number of input
arguments to PHI Nodes or total number of memory instructions.
On the other hand, they do not include the count of some LLVM
instructions that are not considered to contribute to a program’s
representation, e.g. CatchPad instruction.
4.4 Analysis of BenchPress and CLgen
language models
CLgen [6] is the current state of the art in OpenCL benchmark gen-
eration. Its synthetic benchmarks improve the accuracy of Grewe’s
et al. predictive model [18] by 1.27×. However, Goens et al. [16]
perform a case study and show evidence that CLgen’s synthetic
benchmarks do not improve the quality of training data and, con-
sequently, performance of predictive models. They show that a
predictive model in fact performs worse with synthetic benchmarks
as opposed to human written benchmarks or code from GitHub.
This study motivates us to perform an analysis of BenchPress’s
language model, BERT, with CLgen in the task of undirected pro-
gram generation. In this first experiment, we reproduce CLgen
using the authors’ artifacts and we sample it with a fixed input
‘kernel void’ to collect a dataset of unique OpenCL kernels. We
use BenchPress on the same generative task and sample the model
PACT ’22, October 10–12, 2012, Chicago,IL Foivos Tsimpourlas, Pavlos Petoumenos, Min Xu, Chris Cummins, Kim Hazelwood, Ajitha Rajan, and Hugh Leather
with the same fixed input‘kernel void [HOLE]’to obtain another
dataset of unique benchmarks. In this experiment we focus on the
language model’s inference performance. We compare both gener-
ative models on their throughput, their ability to create compiling
code, feature distribution and code size. In this experiment, we do
not direct program generation. BenchPress generates compiling
kernels in a single inference step.
4.5 Targeted Benchmark Generation
Next, we evaluate BenchPress’s ability to steer towards desired
program features. We use well-established compiler benchmarks
as our reference and target their features within this space. These
benchmarks usually perform intensive operations, such as matrix
multiplications or FFT analysis, they contain hundreds of computa-
tional and memory instructions and are specifically fine-tuned by
experts to exercise compilers from different angles. As a result, we
believe features in these benchmarks provide a good target to assess
performance of BenchPress’s ability to target complex features.
We choose target benchmarks within the Rodinia suite [ 4, 5]
as it is widely used in the literature [ 6, 8]. Similar to the train-
ing corpus, we collect the suite’s source files, we inline header
files and dependent OpenCL libraries into them, we split kernels
into single source files and reject those that do not compile. In
total, we collect 61 target Rodinia benchmarks out of which 58
compile. For the remaining benchmarks, we collect their features
using the feature extractors for Grewe’s et al., InstCount and Au-
tophase feature spaces [18, 20, 25]. We target the feature vectors of
these benchmarks and request BenchPress to generate at least one
matching benchmark for each. We end up with three collective syn-
thetic benchmark datasets, one for each feature space, that contain
code with features matching Rodinia benchmarks. For each Ro-
dinia benchmark’s target feature vector, we measure the minimum
Euclidean distance to it achieved between BenchPress, code from
GitHub, CLgen and CLSmith [1, 37]. For GitHub’s and CLSmith’s
kernels, we use SRCIROR [21] to apply code mutations exhaustively
with beam search.
To make our experiment more intuitive we use two datasets for
GitHub: a) GitHub consisting of all OpenCL kernels we collected
and b) GitHub-768, a proper subset of GitHub which contains only
the kernels that do not exceed BenchPress’s sequence length of
768 tokens. Since BenchPress benchmarks’ size are restricted to
the architecture’s sequence length, we feel it is important to make
this distinction in order to present a view of BenchPress’s actual
performance on features that may be unreachable within the current
sequence length. For example, it may be impossible to generate
2,000 computational instructions within 768 tokens. For such cases,
we believe GitHub-768 with its equally restricted sequence length
would allow for a fairer comparison.
For all three feature spaces, we weed out the Rodinia benchmarks
that have an exact matching sample (i.e. a Euclidean distance of
0) in GitHub-768. Since we already have matching samples for
them, we do not need to target them withBenchPress or any other
generative model. However, we do not skip benchmarks whose
features exist only in GitHub’s full dataset as we wanted to explore
the feasibility of using BenchPress to generate a sample with the
same features but smaller sequence length. Applying this restriction
we end up with 22 Rodinia benchmarks for Grewe’s et al., 52 for
InstCount and 36 for Autophase feature spaces.
4.6 Active Learning for Feature Selection
BenchPress’s steerable generation is vital for searching the feature
space while also finding useful features to target with active learn-
ing. In this experiment, we evaluateBenchPress in the downstream
task of training the predictive model proposed by Grewe et al. [18],
a well-tested problem used by many baseline models.
Grewe et al. train a decision tree model to predict the optimal
device to execute a benchmark, choosing between a CPU and a GPU.
They measure their model’s performance as speedup achieved with
using the predicted device for execution versus statically executing
all benchmarks on the GPU. To train the predictive model, they use
OpenCL benchmarks from 7 well-known benchmarks suites [6, 18].
In this experiment, we reproduce Grewe’s et al. heuristic using their
artifact and we also retrain it with datasets enriched with executable
benchmarks from BenchPress using active learning and passive
learning (i.e. targeting random parts of the feature space instead
of searching it), CLgen and GitHub. We measure the speedup over
static mapping for each of them.
To collect our evaluated datasets, we execute OpenCL bench-
marks with CLDrive [6] by Cummins et al. CLDrive automatically
generates inputs and drives kernels to the hardware. It measures
the execution time per device across thousands of runs and it re-
jects kernels that produce runtime errors, do not modify any of
the inputs (no output) or modify them differently for each run (not
deterministic). For (a) the 7 human-written benchmarks suites, (b)
BenchPress, (c) CLgen and (d) GitHub, we execute their kernel on
CLDrive using a range of different local and global size configura-
tions. We label each instance with the fastest measured device (the
CPU or the GPU), in the same way Cummins et al. [6] and Grewe
et al. [18] performed their evaluation.
5 RESULTS AND ANALYSIS
In this section, we show our experiments’ results and compare
BenchPress with state of the art techniques in OpenCL benchmark
synthesis. We present case studies of (a) BenchPress’s throughput
as a generative model compared to CLgen, (b) its ability to steer
benchmark generation towards desired features and (c) its perfor-
mance in searching the feature space to enhance a downstream
task’s performance.
5.1 Analysis of BenchPress and CLgen language
models
We perform an analysis ofBenchPress and CLgen as language mod-
els and compare them in generating a collection of benchmarks
from a fixed input feed, ‘kernel void [HOLE]’ and ‘kernel
void’ respectively. We compare the two approaches measuring
(a) the generative models’ throughput and (b) the quality of their
generated benchmarks in terms of code size and features. In this
experiment, we do not use any directed search or iterative ap-
proach for BenchPress’s generation. We perform this evaluation
to measure how BERT, BenchPress’s underlying language model,
compares with CLgen as a generative model. Table 1 presents the
Short Title PACT ’22, October 10–12, 2012, Chicago,IL
# unique
benchmarks
# compiling
benchmarks
compilation
rate
max
tokens
max inst
(LLVM-IR)
time per
sample (ms)
BenchPress 190,460 142,607 86% 750 161 162
CLgen 1,564,011 13,035 2.33% 102 32 103
Table 1: Throughput comparison between BenchPress and CLgen on generated OpenCL benchmarks when BenchPress does not
use feature-directed program generation.
aggregate measurements for the generated benchmarks using both
approaches.
Figure 6: Probability distribution of (a) token length and (b)
LLVM-IR Instruction count among BenchPress’s and CLgen’s
generated benchmarks. BenchPress’s benchmarks presented
here are generated at a single inference step without itera-
tively directing program synthesis.
Compilation rate and code quality. BenchPress generates over
10×more unique compiling benchmarks than CLgen. This result is
observed despite BenchPress generating 8×fewer unique bench-
marks than CLgen. The compilation rate with BenchPress is 86%
while CLgen has an exceedingly small rate of 2.3%. BenchPress’s
largest sample is 750 tokens compiling to 161 LLVM-IR instruc-
tions. This is a 7.5×and 5×increase in number of tokens and
number of LLVM-IR instructions compared to CLgen’s largest ker-
nel. The only drawback of BenchPress compared to CLgen is that
it is considerably slower in generating candidates. This is because
the transformer-based architecture in BenchPress is significantly
larger in number of parameters than CLgen’s LSTM. Additionally,
BenchPress tends to generate longer kernels than CLgen, necessi-
tating more inference steps and longer generation time.
In Figures 6a and 6b, we show the frequency distribution of
the number of tokens and number of LLVM-IR instructions for
compiling kernels for both datasets. To visualize our results better,
we focus on synthesized kernels with token lengths ≤100 and
instructions lengths ≤25 where the vast majority of benchmarks
are found. Most of BenchPress’s benchmarks are found to have
20 to 80 tokens and 3 to 16 LLVM-IR instructions. The majority of
CLgen’s benchmarks are found to have 5 to 45 tokens and only up
to 4 LLVM-IR instructions. 94% of CLgen’s generated benchmarks
have only 1 instruction when compiled to LLVM-IR. We analyze
the dataset to explain this phenomenon and find CLgen generates
a lot of comments, repeated dead statements and awkward non-
human-like code such as multiple semi-colons. These results agree
with the case study by Goens et al. [16] that shows the AST depth
distribution of CLgen’s code is significantly narrower compared to
code from GitHub or standard benchmarks.
Feature space coverage. To further enhance our comparison, we
perform an analysis on the feature space coverage ofBenchPress’s
and CLgen’s synthesized programs in all three feature spaces. Fea-
ture coverage is the most critical metric when evaluating the effec-
tiveness of a benchmark synthesizer for predictive modeling. We
use Principal Component Analysis (PCA-2) to represent the feature
spaces in an easy to visualize 2-dimensional space. In Figures 7a, 7b
and 7c we show the extent of feature space covered by candidates
in the two approaches. CLgen’s samples are clustered around the
origin, while there is one outlier for Autophase and two for Grewe’s
et al. and InstCount features. Candidates generated by BenchPress
are more scattered achieving a much wider coverage of the feature
space.
5.2 Targeted Benchmark Generation
We use beam search to generate samples that target desired parts
of the feature space. We compareBenchPress with human-written
benchmarks from GitHub and synthetic benchmarks from CLgen
and CLSmith in targeting the features of Rodinia benchmarks on
three feature spaces. We use SRCIROR code mutator with beam
search to collect GitHub and CLSmith benchmarks with closer fea-
tures. For each target benchmark, we gather one OpenCL kernel
per evaluated dataset whose features have the minimum available
Euclidean distance from the target features. Figures 8a, 8b and 8c
show the relative proximity of each benchmark to the target. This
proximity is the complement of the relative distance of the two
kernels, i.e, 1 minus the distance between the two kernels in the
feature space relative to the distance of the Rodinia kernel from the
axes origin. This allows us to express the quality of the match with
an intuitive 0% to 100% scale: 100% means the two kernels have the
same features, 0% means the best kernel is as close to the target as
an empty kernel. We mark perfect matches with a white asterisk
(*).
Performance on syntactic features. On Grewe’s et al. feature space,
BenchPress generates kernels that are the closest ones in features
for all 22 Rodinia Benchmarks compared to CLgen and CLSmith,
PACT ’22, October 10–12, 2012, Chicago,IL Foivos Tsimpourlas, Pavlos Petoumenos, Min Xu, Chris Cummins, Kim Hazelwood, Ajitha Rajan, and Hugh Leather
Figure 7: PCA-2 representation of feature space coverage of BenchPress and CLgen for (a) Grewe’s et al., (b) InstCount and (c)
Autophase feature spaces. In this experiment, BenchPress’s generation is undirected and no iterative space search is performed.
and 20 out of 22 compared toGitHub and GitHub-768. BenchPress
synthesizes an exact match (100% relative proximity) for 14 target
benchmarks.
We pick out and discuss a few examples from our results. The
absolute distance achieved for ‘nw-1’ and ‘ellipse_opt’, is 1.0. For
both targets, almost all features match except for one missing in-
struction (coalesced mem access and atomic inst respectively).
For ‘hotspot’ GitHub and BenchPress produce a candidate kernel
with exact matching features. However, BenchPress generates the
matching candidate kernel in 421 tokens, unlike GitHub’s closest
benchmark that has 798 tokens. For the two target benchmarks that
BenchPress’s candidates were not closest to, we found onlyGitHub
contains better samples for ‘com_dwt-3’ and and ‘gpu-1’, while
BenchPress does not. We find both benchmarks to be fairly large
(901 and 5,200 tokens respectively) and BenchPress cannot reach
these features within 768 tokens. For the same reason,GitHub-768,
CLgen and CLSmith does worse than BenchPress on these targets.
Performance on LLVM IR features. Autophase and InstCount
features are extracted from the LLVM-IR of a program that has
been compiled with -O1 flag to apply basic optimisations such
as dead code elimination. BenchPress occasionally generates re-
peating operations that a compiler will remove or numerical op-
erations that may be reduced to simple assignments. Owing to
these optimisations, we find targeting benchmarks on these two
feature spaces is more challenging than Grewe’s et al. syntax-level
features. With InstCount features, BenchPress generates candi-
dates whose features completely match 2 out of the 52 Rodinia
benchmarks. Among the remaining 50, BenchPress outperforms
CLgen, CLSmith, GitHub and GitHub-768 for all target benchmarks,
achieving higher proximity.SRCIROR significantly improvesGitHub
leading to GitHub+SRCIROR to achieve better proximity for 18 out
of 52 Rodinia benchmarks compared toBenchPress. On Autophase
features, BenchPress generates candidates matching the same 2 tar-
get benchmarks, while outperforming CLgen, CLSmith and GitHub
on 30 out of 36 Rodinia benchmarks in total. GitHub+SRCIROR per-
forms better than BenchPress for 8 out of 36 target benchmarks
and produces an exact match for ‘hotspotKernel’.
We previously explain the importance of having diverse features
in compiler benchmarks and we show, in Figure 2, how sparse Ro-
dinia benchmarks are on Grewe’s et al. reduced feature space and
how CLgen fails to provide any additional features. Now we intro-
duce into this 2-dimensional space allBenchPress’s kernels that are
generated while performing directed space search to target Rodinia
benchmarks and we present them in Figure 9. BenchPress densely
populates the space around the target benchmarks that are clus-
tered around the lower left corner. We findBenchPress’s samples
progressively converge to the target benchmark features with suc-
cessive generations. For example, BenchPress targets ‘com_dwt-3’
at 385 computational and 137 memory instructions, starting from
the axes origin and attempting to reach its features from different
directions. One of the directions prevail but does not manage to
exactly reach the target. The same happens for the top right point,
‘gpu-1’. BenchPress’s samples get closer developing a straight line
from the origin to 1,000 computational and 100 memory instruc-
tions. At this point BenchPress is restricted by its sequence length
and cannot augment further its samples. This is depicted by its
attempt to reduce the distance by swapping the two instruction
types within the same token length, forming a perpendicular line
with a negative slope. We argue the area of Grewe’s et al. feature
space that BenchPress can cover within 768 tokens to be the area
of the triangle formed by the intersections of the axes with the
extension of the negative slope line developed by BenchPress’s
samples.
Summary - BenchPress vs GitHub vs CLgen vs CLSmith. 6 of the
targeted Rodinia benchmarks exceed BenchPress’s maximum se-
quence length of 768 tokens. In LLVM-IR feature spaces, care must
be taken to generate code that will not be removed by compiler
optimisations. This is a difficult challenge for source code gener-
ative models. However, our results demonstrate that BenchPress
can generate OpenCL kernels that approach target human-written
benchmarks compared to GitHub code and CLgen candidates. Our
experiments also show BenchPress is dramatically better in all
cases than CLgen, the current state of the art in OpenCL synthetic
benchmark generation. We further elaborate on BenchPress’s per-
formance in the next subsections.
5.3 Active Learning for Feature Selection
We combineBenchPress’s ability to generate benchmarks targeting
desired features with active learning in order to generate bench-
marks that improve the training of the Grewe et al. heuristic. We
evaluate this against passive training withCLgen, GitHub code, and
Short Title PACT ’22, October 10–12, 2012, Chicago,IL
Figure 8: Relative proximity to each Rodinia benchmark of the candidate kernel with the closest features. We report the best
match for seven datasets ( BenchPress’s, CLgen’s, GitHub’s and GitHub-768’s datasets also combined with exhaustive mutations
with SRCIROR) over three feature spaces ((a) Grewe’s et al., (b) InstCount and (c) Autophase). Relative proximity is 1 minus the
distance of the two kernels in the feature space relative to the distance of the Rodinia benchmark from the axes origin. 100%
means an exact match in features and is highlighted with a white asterisk (*). A score towards 0% indicates the closest match
is closer to the axes origin than the benchmark, i.e., a very small or empty kernel.
BenchPress with randomly selected target features. All approaches augment the same baseline training set that is taken from [6], con-
taining 7 benchmark suites1. Table 2 shows the effect of each ap-
proach on the predictive power of the heuristic. Training only on
1The benchmarks have been updated with a wider range of global and local sizes.
PACT ’22, October 10–12, 2012, Chicago,IL Foivos Tsimpourlas, Pavlos Petoumenos, Min Xu, Chris Cummins, Kim Hazelwood, Ajitha Rajan, and Hugh Leather
Figure 9: # Memory operations and # computational instruc-
tions for (a) Rodinia benchmarks in purple diamonds, (b)
CLgen’s samples in red dots and BenchPress’s benchmarks
in green crosses after performing directed search for all Ro-
dinia benchmarks.
Speedup % Precision Recall Specificity
Benchmarks +4% 0.81 0.86 0.61
BenchPress-AL +6% 0.84 0.86 0.64
BenchPress-P +1% 0.84 0.85 0.48
CLgen -1% 0.52 0.86 0.43
GitHub +1% 0.85 0.83 0.61
Table 2: Grewe et al. heuristic model’s performance, preci-
sion, recall, and specificity when trained on each technique.
Speedup is the geometrical mean of speedups over all bench-
marks relative to the optimal static decision, i.e. running on
the GPU. Precision, recall, and specificity treat GPU labels
as positive and CPU labels as negative.
human written benchmarks improves the heuristic’s performance
by 4%, as shown in Table 2’s first row. To understand the maxi-
mum achievable improvement in the heuristic, we compute the
best speedup ( = 12%) that is achieved if the model chooses the
optimal device as opposed to always picking the GPU. For 71% of
the benchmarks, GPU is the optimal device, so no speedup improve-
ment is possible. For the remaining 29% benchmarks, predicting
the ‘CPU’ label correctly with Grewe et al. will result in a speedup
improvement.
BenchPress using active learning (BenchPress-AL) clearly out-
performs all other approaches in terms of average speedup, improv-
ing it by 6%. When trained on BenchPress with passive/random
feature selection ( BenchPress-P), the speedup achieved is only
1%. To our surprise, the same speedup is achieved with GitHub,
which is worse compared with training only on the original bench-
mark suites. We further analyze the dataset collected from GitHub
code and we find it to be imbalanced with 90% of its training in-
stances are labelled as ‘GPU’. This leads the model having a higher
precision of 0.85, i.e. predicting correctly that a kernel should ex-
ecute on the GPU, but falling short when it comes to correctly
Figure 10: BenchPress’s performance enhancement of Grewe
et al. heuristic model when using active learning compared
to passively targeting random parts of the feature space over
the course of 10 sampling epochs.
predicting the ‘CPU’ label. Training the heuristic with CLgen ac-
tually leads to a slowdown: it is 1% slower to execute kernels on
the predicted devices compared to statically executing everything
on the GPU, the baseline device. We analyze CLgen’s dataset and
observe the opposite pattern found in GitHub’s dataset. 63% of its
training data execute faster on the CPU than on the GPU. This is
a direct consequence of CLgen generating small benchmarks that
are poor in features, as the CPU may be slower than the GPU but
the large overhead of transferring data to the GPU makes the CPU
a better choice for small workloads. CLgen containing too many
CPU-labeled kernel explains the heuristic’s low precision and speci-
ficity, as it becomes biased to select the CPU very often leading to
a slowdown.
Our main motivation behind using active learning is that it gives
BenchPress the ability to target directly those parts of the feature
space that will maximize a downstream task’s performance. To
assess the active learner’s performance, we compare the Grewe
et al. heuristic’s speedup when trained on BenchPress’s bench-
marks that target areas of the feature space selected by the active
learner versus benchmarks that target random features. In both
cases, we execute BenchPress for the same amount of time, 10
sampling epochs (i.e., performing steered generation for 10 target
feature vectors). In Figure 10, we show the speedup achieved by
the heuristic when trained on the data collected at that step. Using
active learning to target features, BenchPress’s dataset improves
the heuristic’s speedup by 50% after 5 sampling steps, from 4% to
6%. Targeting random features never leads to a speedup higher than
1%. BenchPress can still develop the same speedup by targeting
random features if infinite amount of time was available. Our ac-
tive learner ensures that missing features are going to be quickly
targeted, improving the state of the art within 5 sampling epochs.
6 CONCLUSION
Predictive models for compilers have been shown to outperform
compiler experts but they are restricted by the amount and quality
of training data they are exposed to. What is needed is an approach
that can synthesize benchmarks and enhance datasets with missing
features. In this paper we propose BenchPress, a powerful code
generator that uses active learning to search the feature space and
Short Title PACT ’22, October 10–12, 2012, Chicago,IL
steers generation towards desired features. BenchPress generates
10×more and 7.5×larger undirected benchmarks with 37×greater
compilation rate than CLgen - a state of the art compiler bench-
mark generator - from a fixed input feed. BenchPress outperforms
CLgen, CLSmith, code from GitHub and applied mutations with
SRCIROR in generating OpenCL kernels that target the features
of Rodinia benchmarks developed by human experts. BenchPress
applies active learning to enhance Grewe’s et al. dataset with bench-
marks with missing features and leads to improving the heuristic’s
speedup by 50%. We hope this work to demonstrate a sustainable
method to direct feature space search of program generation and
that BenchPress’s release to researchers will enable research in
related domains.
7 RELATED WORK
BenchPress is inspired by BERT, a representation model by Devlin
et al. [11]. Contrary to previous techniques [30, 31], BERT learns
on unlabeled text data by jointly conditioning on both left and right
context. BERT enables multiple applications of this architecture
to a wide variety of difficult machine learning tasks, including
programming languages. In CuBERT [24], Kanade et al. apply BERT
over Python programs and evaluate it on finding typical mutation
faults. In CodeBERT [13], Feng et al. fine-tune BERT to perform
NL-PL and PL-NL transformations. In this work, we extend BERT
to a bidirectional generative model, with the help of [HOLE] token.
Cummins et al. [ 6] develop CLgen, a deep learning generator
based on LSTM [ 23] for OpenCL programs. They try to tackle
the compiler benchmark shortage by providing synthetic bench-
marks as training data for compiler heuristics. The authors present
the Grewe et al. [ 18] heuristic model improved its performance
by 1.27×when trained on their synthetic benchmarks. However,
Goens et al. [16] show that training with CLgen’s synthetic sam-
ples lead to a slowdown compared to training on human-written
benchmarks only. To explain this, they measure the AST depth of
CLgen’s samples and show it is 3×smaller compared to human-
written benchmarks and code from GitHub and poor in features,
therefore unrealistic. This motivates us to develop BenchPress,
which produces 10×more unique kernels that are 7.5×larger on
average.
In 2019, Nye et al. develop SketchAdapt [ 28], which uses a
generator-synthesizer [3, 12] to generate program sketches given
I/O specifications. The synthesizer samples sketches and the gener-
ator fills <HOLE> tokens with statements. SketchAdapt performs
better than other architectures [3, 12], however it samples only a
pre-defined pool of operations, which restricts its diversity. Bruen
et al. [9], propose a Tree2Tree approach for code generation using
VAE. They encode AST nodes using Tree-LSTMs (Tai et al. [35])
and train their model on C++ functions. They test their approach
against a VAE with an LSTM Seq2Seq model. They use their model
as a synthesizer by sampling random AST representations which
they extend to new programs. Their Seq2Seq model achieves a com-
pilation rate of up to 67% with greedy search, however this happens
because the model greedily selects the most probable labels, lead-
ing to repetitive samples. When sampling with temperature, their
Tree2Tree architecture is able to generate a wider variety of sam-
ples, but only achieves a compilation rate of 22%, which translates
to a few functions.
Gupta et al. [19] develop SED, a two-stage generator. A synthe-
sizer receives I/O specifications and generates programs likely to
satisfy them and a neural debugger applies program repair to reform
them into functions that match specifications. Gupta et al. evaluate
three synthesizer architectures and measure (a) the correctness
of generated programs across tests and (b) the accuracy of their
debugger to repair code. While SED is an innovative work, Karel
is a small-scale language and SED’s generative performance on a
complex programming language is not evaluated. Faustino et al.
develop Anghabench [8] to tackle the benchmark shortage [6, 36].
Anghabench is a collection of C programs mined from GitHub. In
order to make it compilable, they use Psyche-C [ 27] type infer-
ence engine to apply type reconstruction and resolve dependencies.
Structs, unions and other composite data types are omitted or re-
declared with primitive types. Their benchmarks are compiling,
but cannot be executed. Compared to AnghaBench, BenchPress
resolves type dependencies of composite types and user-defined
functions without changing the functionality or semantics of pro-
grams.
PACT ’22, October 10–12, 2012, Chicago,IL Foivos Tsimpourlas, Pavlos Petoumenos, Min Xu, Chris Cummins, Kim Hazelwood, Ajitha Rajan, and Hugh Leather
REFERENCES
[1] [n.d.]. https://github.com/ChrisLidbury/CLSmith. [Online; accessed 25-Apr-
2022].
[2] R. Bagrodia, R. Meyer, M. Takai, Yu-An Chen, Xiang Zeng, J. Martin, and Ha Yoon
Song. 1998. Parsec: a parallel simulation environment for complex systems.
Computer 31, 10 (1998), 77–85. https://doi.org/10.1109/2.722293
[3] Matej Balog, Alexander Gaunt, Marc Brockschmidt, Sebastian Nowozin, and
Daniel Tarlow. 2016. DeepCoder: Learning to Write Programs. (11 2016).
[4] Rodinia Benchmarks. [n.d.]. http://lava.cs.virginia.edu/Rodinia/download.htm.
[Online; accessed 25-Apr-2022].
[5] Shuai Che, Michael Boyer, Jiayuan Meng, David Tarjan, Jeremy W. Sheaffer, Sang-
Ha Lee, and Kevin Skadron. 2009. Rodinia: A benchmark suite for heterogeneous
computing. In 2009 IEEE International Symposium on Workload Characterization
(IISWC). 44–54. https://doi.org/10.1109/IISWC.2009.5306797
[6] Chris Cummins, Pavlos Petoumenos, Zheng Wang, and Hugh Leather. 2017.
Synthesizing benchmarks for predictive modeling. In2017 IEEE/ACM International
Symposium on Code Generation and Optimization (CGO) . 86–99. https://doi.org/
10.1109/CGO.2017.7863731
[7] Chris Cummins, Bram Wasti, Jiadong Guo, Brandon Cui, Jason Ansel, Sahir
Gomez, Somya Jain, Jia Liu, Olivier Teytaud, Benoit Steiner, Yuandong Tian, and
Hugh Leather. 2021. CompilerGym: Robust, Performant Compiler Optimization
Environments for AI Research. arXiv:2109.08267 [cs.PL]
[8] Anderson Faustino da Silva, Bruno Conde Kind, José Wesley de Souza Magalhães,
Jerônimo Nunes Rocha, Breno Campos Ferreira Guimarães, and Fernando Magno
Quinão Pereira. 2021. ANGHABENCH: A Suite with One Million Compilable C
Benchmarks for Code-Size Reduction. In2021 IEEE/ACM International Symposium
on Code Generation and Optimization (CGO) . 378–390. https://doi.org/10.1109/
CGO51591.2021.9370322
[9] Sander de Bruin, Vadim Liventsev, and Milan Petković. 2021. Autoencoders as
Tools for Program Synthesis. arXiv:2108.07129 [cs.AI]
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Im-
ageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on
Computer Vision and Pattern Recognition . 248–255. https://doi.org/10.1109/CVPR.
2009.5206848
[11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT:
Pre-training of Deep Bidirectional Transformers for Language Understanding. In
NAACL.
[12] Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman
Mohamed, and Pushmeet Kohli. 2017. RobustFill: Neural Program Learning under
Noisy I/O. In Proceedings of the 34th International Conference on Machine Learning
- Volume 70 (Sydney, NSW, Australia)(ICML’17). JMLR.org, 990–998.
[13] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming
Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020.
CodeBERT: A Pre-Trained Model for Programming and Natural Languages.
arXiv:2002.08155 [cs.CL]
[14] Grigori Fursin and Olivier Temam. 2011. Collective Optimization: A Practical
Collaborative Approach. ACM Trans. Archit. Code Optim. 7, 4, Article 20 (dec
2011), 29 pages. https://doi.org/10.1145/1880043.1880047
[15] GitHub. [n.d.]. https://docs.github.com/en/rest. [Online; accessed 25-Apr-2022].
[16] Andrés Goens, Alexander Brauckmann, Sebastian Ertel, Chris Cummins, Hugh
Leather, and Jeronimo Castrillon. 2019. A Case Study on Machine Learning for
Synthesizing Benchmarks . Association for Computing Machinery, New York, NY,
USA, 38–46. https://doi.org/10.1145/3315508.3329976
[17] Google. [n.d.]. https://cloud.google.com/bigquery. [Online; accessed 25-Apr-
2022].
[18] Dominik Grewe, Zheng Wang, and Michael F. P. O’Boyle. 2013. Portable mapping
of data parallel programs to OpenCL for heterogeneous systems. InProceedings of
the 2013 IEEE/ACM International Symposium on Code Generation and Optimization
(CGO). 1–10. https://doi.org/10.1109/CGO.2013.6494993
[19] Kavi Gupta, Peter Christensen, Xinyun Chen, and Dawn Song. 2020. Synthesize,
Execute and Debug: Learning to Repair for Neural Program Synthesis.
[20] Ameer Haj-Ali, Qijing (Jenny) Huang, John Xiang, William Moses, Krste
Asanovic, John Wawrzynek, and Ion Stoica. 2020. AutoPhase: Juggling HLS
Phase Orderings in Random Forests with Deep Reinforcement Learning. In
Proceedings of Machine Learning and Systems , I. Dhillon, D. Papailiopoulos,
and V. Sze (Eds.), Vol. 2. 70–81. https://proceedings.mlsys.org/paper/2020/file/
4e732ced3463d06de0ca9a15b6153677-Paper.pdf
[21] Farah Hariri and August Shi. 2018. SRCIROR: A Toolset for Mutation Testing of
C Source Code and LLVM Intermediate Representation. In 2018 33rd IEEE/ACM
International Conference on Automated Software Engineering (ASE) . 860–863. https:
//doi.org/10.1145/3238147.3240482
[22] John L. Henning. 2006. SPEC CPU2006 Benchmark Descriptions. SIGARCH
Comput. Archit. News 34, 4 (sep 2006), 1–17. https://doi.org/10.1145/1186736.
1186737
[23] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory.
Neural Comput. 9, 8 (nov 1997), 1735–1780. https://doi.org/10.1162/neco.1997.9.
8.1735
[24] Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi.
2020. Learning and Evaluating Contextual Embedding of Source Code.
arXiv:2001.00059 [cs.SE]
[25] Chris Lattner and Vikram Adve. 2004. LLVM: A Compilation Framework for
Lifelong Program Analysis and Transformation. In CGO. San Jose, CA, USA,
75–88.
[26] Yann LeCun, Y. Bengio, and Geoffrey Hinton. 2015. Deep Learning. Nature 521
(05 2015), 436–44. https://doi.org/10.1038/nature14539
[27] Leandro T. C. Melo, Rodrigo G. Ribeiro, Breno C. F. Guimarães, and Fernando
Magno Quintão Pereira. 2020. Type Inference for C: Applications to the Static
Analysis of Incomplete Programs. ACM Trans. Program. Lang. Syst. 42, 3, Article
15 (nov 2020), 71 pages. https://doi.org/10.1145/3421472
[28] Maxwell Nye, Luke B. Hewitt, Joshua B. Tenenbaum, and Armando Solar-Lezama.
2019. Learning to Infer Program Sketches. In ICML.
[29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory
Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Des-
maison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan
Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith
Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning
Library. In Advances in Neural Information Processing Systems 32 , H. Wallach,
H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Cur-
ran Associates, Inc., 8024–8035. http://papers.neurips.cc/paper/9015-pytorch-
an-imperative-style-high-performance-deep-learning-library.pdf
[30] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark,
Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representa-
tions. (02 2018).
[31] Alec Radford and Karthik Narasimhan. 2018. Improving Language Understanding
by Generative Pre-Training.
[32] H. S. Seung, M. Opper, and H. Sompolinsky. 1992. Query by Committee. InProceed-
ings of the Fifth Annual Workshop on Computational Learning Theory (Pittsburgh,
Pennsylvania, USA) (COLT ’92) . Association for Computing Machinery, New
York, NY, USA, 287–294. https://doi.org/10.1145/130385.130417
[33] OpenCL specification. [n.d.]. https://www.khronos.org/registry/OpenCL/specs/
3.0-unified/html/OpenCL_C.html. [Online; accessed 25-Apr-2022].
[34] John E. Stone, David Gohara, and Guochun Shi. 2010. OpenCL: A Parallel Pro-
gramming Standard for Heterogeneous Computing Systems.Computing in Science
Engineering 12, 3 (2010), 66–73. https://doi.org/10.1109/MCSE.2010.69
[35] Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved
Semantic Representations From Tree-Structured Long Short-Term Memory Net-
works. In Proceedings of the 53rd Annual Meeting of the Association for Computa-
tional Linguistics and the 7th International Joint Conference on Natural Language
Processing (Volume 1: Long Papers) . Association for Computational Linguistics,
Beijing, China, 1556–1566. https://doi.org/10.3115/v1/P15-1150
[36] Zheng Wang and Michael O’Boyle. 2018. Machine Learning in Compiler Opti-
mization. Proc. IEEE 106, 11 (2018), 1879–1901. https://doi.org/10.1109/JPROC.
2018.2817118
[37] Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and Un-
derstanding Bugs in C Compilers. In Proceedings of the 32nd ACM SIGPLAN
Conference on Programming Language Design and Implementation (San Jose, Cali-
fornia, USA) (PLDI ’11) . Association for Computing Machinery, New York, NY,
USA, 283–294. https://doi.org/10.1145/1993498.1993532

Absinthe: Learning an Analytical Performance
Model to Fuse and Tile Stencil Codes in One Shot
Tobias Gysi
Department of Computer Science
ETH Zurich, Switzerland
tobias.gysi@inf.ethz.ch
Tobias Grosser
Department of Computer Science
ETH Zurich, Switzerland
tobias.grosser@inf.ethz.ch
Torsten Hoeﬂer
Department of Computer Science
ETH Zurich, Switzerland
htor@inf.ethz.ch
Abstract—Expensive data movement makes the optimal target-
speciﬁc selection of data-locality transformations essential. Loop
fusion and tiling are the most important data-locality trans-
formations. Their optimal selection is hard since good tile
size choices inherently depend on the fusion choices and vice
versa. Existing approaches avoid this difﬁculty by optimizing
independent analytical models or by reverting to heuristics.
Absinthe formulates the ﬁrst uniﬁed linear optimization problem
to derive single shot fusion and tile size decisions for stencil codes.
At the core of our optimization problem, we place a learned
analytic performance model that captures the characteristics of
the target system. The tuned application kernels demonstrate
excellent performance within 10% of exhaustively auto-tuned
versions and up to 74% faster than the results of independent
optimization with max fusion heuristic and Absinthe tile size
selection. While the full search space is non-linear , bounding
it to relevant solutions enables the efﬁcient exploration of the
exponential search space using linear solvers. As a result, the
tuning of our application kernels takes less than one minute. Our
approach thus establishes the foundations for next-generation
compilers, which exploit empirical information to guide target-
speciﬁc code transformations.
Index T erms—stencils, performance model, data-locality
I. I NTRODUCTION
The cost of data movement in terms of energy and time has
long exceeded the cost of computation. Thus, data locality
recently became the most important optimization target for
performance engineers [1]. Today, most programmers either
rely on the compilation toolchain or manually optimize data
locality by tiling and fusing loops. Manual loop optimizations
are tedious and require a high porting effort to exploit different
architectures efﬁciently because tiling and fusion parameters
need to be adjusted for each target system. V arious frameworks
such as Halide [2] and Polymage [3] focus their tuning on
this parameter selection, but they either apply heuristics or
optimize tiling and fusion separately to control the exponen-
tial search space. However, fusion and tiling are inherently
linked—for optimizing one, one needs to assume a speciﬁc
conﬁguration for the other. For example, the optimal tile size
depends on the memory footprint of the loop, which changes
with fusion. This missing modularity of the problem requires
us to consider tiling and fusion in tandem.
Stencil computations on regular grids are ubiquitous in
scientiﬁc computing applications such as climate modeling [4],
seismic imaging [5], and electromagnetic simulations [6].
absinthe
64x4x3
64x4x5
20 1
3 4
6
5
7 8
unfused 
tiled
64x64x1
20 1
3 4
6
5
7 8
auto-tuning
64x4x1
64x4x4
model prediction [ms]
measured execution time [ms]
1.080.730.67
0.94
0.62
0.58 -6.5%
20 1
3 4
6
5
7 8
Fig. 1: Absinthe optimization example
In this work, we use the COSMO atmospheric model [4],
which is used in operational weather forecasting in most of
Europe [7] as well as in large-scale climate modeling [8],
as a motivating example. The 300′000 lines of code contain
more than 16′000 loops, most of which implement single
stencils. These stencils logically form complex producer-
consumer relationships, calledstencil programs [9]. We select
three representative COSMO stencil programs to evaluate the
effectiveness of our approach. Due to the very low arithmetic
intensity of every single stencil, tiling and fusion are crucial
for achieving good performance for stencil programs.
We show an example in Fig. 1—COSMO’s fastwaves
stencil program which implements parts of the sound wave
forward integration. The directed graphs show the data-ﬂow
(edges) between the stencils (nodes) of the fastwaves program.
Our optimization framework,Absinthe1, uses an automatically
learned performance model to guide the program optimization.
The ﬁgure plots the model prediction versus the measured
execution time for the tile size (annotated) and fusion (shaded
shapes) choices of Absinthe compared to auto-tuning and an
1https://github.com/spcl/absinthe.git
369
2019 28th International Conference on Parallel Architectures and Compilation Techniques (PACT)
978-1-7281-3613-4/19/$31.00 ©2019 IEEE
DOI 10.1109/PACT.2019.00036

target 
system
model learner
benchmarkparameters
optimizer
code generator
C++
transfor-
    mations
P, B
1
2
3
ILP solver
stencil DSL
Fig. 2: Absinthe architecture overview
unfused tiled implementation.
Absinthe consists of three main pieces: (1) a model learner,
(2) an optimizer, and (3) a code generator. Themodel learner
generates a performance model speciﬁc to each target architec-
ture. The optimizer derives an integer linear program encoding
the structure of the stencil program and the performance model
to tune tiling and fusion together. The code generator then
emits an implementation with the optimal tiling and fusion
parameters returned by the integer linear programming solver.
In this way, Absinthe combines automated model learning with
integer linear programming to control the exponential search
space and to automatically ﬁnd the best conﬁguration for each
target architecture.
In summary, we make the following key contributions:
• A linear formulation of parametric tiling for bound tile
sizes (assumed to be non-linear in general).
• A linear performance model that learns the target sys-
tem characteristics and enables the use of integer linear
programming to explore the search space.
• A single holistic optimization problem which applies the
linear performance model to derive optimal fusion and
tile size selection choices for stencil codes.
II. B ACKGROUND
The execution of stencils in succession provides plenty of
opportunities for data locality improvements.
A. Architecture Overview
Absinthe lowers stencil programs written in a high-level
domain-speciﬁc language (DSL) to efﬁcient C++ code. An
automatically learned performance model drives the selection
of target system speciﬁc code transformations. Fig. 2 shows
the interplay of the Absinthe components.
The model learner (1) runs once for every target system
to learn the model parameters. The optimizer (2) combines
the model parameters with the memory access patterns of
the stencil program to instantiate a target-speciﬁc performance
model. An integer linear programming (ILP) solver searches
the optimal data-locality transformations with respect to the
performance model. The code generator (2) applies the optimal
data-locality transformations to the high-level stencil program
representation and generates tuned C++ code.
1 for(int x=xbeg; x<=xend; ++x)
2 for(int y=ybeg; y<=yend; ++y)
3 for(int z=zbeg; z<=zend; ++z)
4 s0(x,y,z) = 0.5 * (i0(x+1,y,z) + i0(x,y,z));
5 for(int x=xbeg; x<=xend; ++x)
6 for(int y=ybeg; y<=yend; ++y)
7 for(int z=zbeg; z<=zend; ++z)
8 s1(x,y,z) = i1(x,y,z) * (s0(x,y+1,z) - s0(x,y-1,z));
Fig. 3: Example stencil sequence with lengthN =2
Absinthe targets three-dimensional stencil programs and
optimizes them to utilize all processors of the target system,
assuming exclusive system access. Our implementation has the
following limitations: 1) we support only three-dimensional
arrays, 2) we do not optimize the boundary conditions, and 3)
we tile the codes only for one memory hierarchy level.
B. Stencil Sequences
A stencil is an element-wise computation with a position
independent access pattern. Every stencil evaluation accesses
the input arrays at ﬁxed offsets relative to the updated output
array element. We assume that every stencil writes a single
array. We apply stencils to all array elements except for a
constant width halo at the array boundary which prevents out-
of-bounds accesses.
A stencil sequence is a program formed of several subse-
quent stencil applications. Fig. 3 shows an example stencil
sequence with length N =2 . The short example sequence
allows us to illustrate our approach with less complexity
compared to the fastwaves kernel introduced in Fig. 1.
C. Data-Locality Transformations
Absinthe combines rectangular tiling with redundant com-
putation at the tile boundaries to satisfy the data dependencies
of fused stencils. This overlapped tiling [10] enables major
performance improvements. The tuned fastwaves kernel shown
by Fig. 1 executes 1.5× faster compared to the unfused tiled
implementation variant.
Loop tiling decomposes the domain into hyper-rectangular
tiles of equal size. To increase the data-locality, we evaluate the
stencil on the entire tile before proceeding with the next one.
We thus introduce an additional outermost loop that iterates
over all tiles. To support arbitrary domain sizes, we cut the
tiles at the domain boundary.
Loop fusion replaces the tile loops of consecutive stencils
with a single tile loop that evaluates one stencil after another
before proceeding with the next tile. After fusion, the data
dependencies of producer-consumer stencils cross the tile
boundaries. To enable the parallel tile execution, we extend
the loop bounds of the producer stencils to perform redundant
computation at the tile boundaries which satisﬁes all data
dependencies locally.
The combination of fusion and tiling effectively increases
the spatial and temporal locality for stencils with overlapping
working sets. The code generator introduces one tile loop for
every group of fused stencils and allocates temporary storage
370
TABLE I: Important constants and variables
constants
N number of stencils in the stencil sequence
Dx,D y,D z domain sizes
Hx,H y,H z halo widths
T number of processors
C cache capacity
P f ,B f fast memory peel & body parameters
P b,B b slow memory peel & body base parameters
P v,B v slow memory peel & body variable parameters
variables
gi group index
nx
i ,n y
i ,n z
i tile counts
pf
i ,b f
i fast memory peel & body cost
pb
i ,b b
i slow memory peel & body base cost
pv
i ,b v
i slow memory peel & body variable cost
ex+
i ,e y+
i ,e z+
i evaluation boundary widths (positive axis direction)
ex−
i ,e y−
i ,e z−
i evaluation boundary widths (negative axis direction)
target system
STENCILS = {
  S0 : 
    0.5 * (
    i0(x+1,y,z) + 
    i0(x,y,z))
  S1 : 
    i1(x,y,z) * (
    s0(x,y+1,z) +
    s0(x,y-1,z))
}
stencil sequence
performance model
fast memory slow memory
data-locality transformations
loop fusion loop tiling
parameters
Ai, Bi
x, Bi
y, Bi
z
N, Dx, Dy, Dz, T, C 
Pf, Bf, Pb, Bb, Pv, Bv learn
analyze
min t
ILP
Fig. 4: Absinthe ILP parameters and components
to buffer intra-tile data dependencies with minimal memory
footprint.
III. M ODELING
The optimizer automatically instantiates an integer linear
program (ILP) to ﬁnd good data-locality transformations.
Fig. 4 shows the main components of the ILP: theparameters
component captures the stencil sequence and target system
properties that provide the basis for the optimization, thedata-
locality transformations component deﬁnes the optimization
variables that span the space of possible transformations, and
the performance model component estimates the execution
time for the selected code transformations. At optimization
time, the ILP solver searches the code transformations with
minimal estimated execution time.
We present the ILP for three-dimensional stencils, but the
formulation generalizes to stencils with different dimension-
ality. If not mentioned otherwise, the variables are positive
and integer-valued, while lowercase and uppercase identiﬁers
distinguish optimization variables and constants, respectively.
Table I lists important constants and variables.
A. Stencil Sequences
The optimizer requires an analysis of the stencil access
patterns to instantiate the ILP shown by Fig. 4. The access
patterns provide the basis to compute the data-ﬂow and to
estimate the performance of the stencil sequence.
We use positive indexes to number the stencils in execution
order and negative indexes to identify the input arrays. For
example, the indexes[0, 1] refer to the stencils[s0,s 1] and the
indexes [−1, −2] to the input arrays [i0,i 1] of the example
stencil sequence shown by Fig. 3. The stencil indexes also
map one-to-one to the output arrays since every stencil writes
precisely one output. The resulting index space thus uniquely
identiﬁes the input and output arrays of the stencil sequence.
To specify the data access patterns, we deﬁne for every
stencil i the access set Ai holding (index, offset) tuples that
deﬁne the array and the three-dimensional relative offset of
every input element access. The access sets
A0 = {(−1, (1, 0, 0)), (−1, (0, 0, 0))},
A1 = {(−2, (0, 0, 0)), (0, (0, 1, 0)), (0, (0, −1, 0))}
include all accesses of the example stencils. We also compute
minimal bounding boxes that contain all access offsets. To
represent the bounding boxes, we deﬁne for every stencil i
and dimension d the bounds set Bd
i holding (index, range)
tuples that specify the array and the minimal and maximal
access offset along the dimension. The bounds sets
Bx
0 = {(−1, (0, 1))}, By
1 = {(−2, (0, 0)), (0, (−1, 1))}
contain all accesses of the example stencils along the selected
dimensions.
To execute the stencil sequence, we deﬁne for every di-
mension d the constant domain sizeDd and the constant halo
width Hd along the dimension. The domain sizes determine
the stencil loop bounds, while the halo sizes together with the
domain sizes specify the array allocation size. For example,
we may execute the example stencils on the domain
Dx =6 4,D y =6 4,D z =6 0
and select the halo widths
Hx =1 ,H y =1 ,H z =0
to accommodate the transitive stencil access offsets, which
results in the array allocation size66 ×66 ×60.
B. Data-Locality Transformations
The optimizer also deﬁnes the optimization variables that
span the space of possible data-locality transformations and
introduces constraints to exclude solutions that suffer from
load imbalance or exceed the cache capacity.
To model loop tiling, we select for every stencili ∈ [0,N )
and every dimension d the tile count nd
i along the dimension
from the range
[
1,D d]
. For example, the tile counts
nx
0 =2 ,n y
0 =2 ,n z
0 =2
split the domain of the ﬁrst stencil in the example stencil
sequence into two tiles along every dimension.
To modelloop fusion, we select for every stencili ∈ [0,N )
the group indexgi and fuse stencils with the same group index.
371
We set the group index of the ﬁrst stencil to zero and increment
the group index with every additional group along the stencil
sequence. For every stencil, we thus have the choice to retain
or increment the group index of the preceding stencil, which
spans an exponential search space in the number of stencils.
For example, the group index tuples
(g0,g 1) ∈{ (0, 0), (0, 1)}
enumerate all possible group assignments for the example
stencil sequence. The group indexes g0 =0 ,g 1 =0 assign
the stencils to the same group to model fusion while the
group indexes g0 =0 ,g 1 =1 assign the stencils to different
groups that execute consecutively. To quantify theredundant
computation, we also extend for every stencil i ∈ [0,N )
and for every dimension d the tile size with the evaluation
boundary widths ed+
i and ed−
i along both directions. For
example, the evaluation boundary widths
ey+
0 =1 ,e y−
0 =1
extend the tile size of the ﬁrst stencil to satisfy all data
dependencies of the example stencil sequence locally. We
deﬁne the tile sizes for every stencil, but the stencils of each
group share the same tile loop and tile size. We thus enforce
tile count equality for succeeding stencils with the same group
index.
To guaranteedata-locality, we exclude tile sizes that exceed
the cache capacityC (L2 cache). To estimate the cache utiliza-
tion, we multiply for every stencil group the tile size with the
number of accessed arrays. This approximation optimistically
models a fully associative cache with a least recently used
cache replacement policy and does not consider the accesses
at the tile boundaries. We thus enforce the cache utilization for
a single tile to be lower than one-third of the cache capacity.
This choice compensates for our optimistic cache modeling
and ensures that not only the current but also the next and
the previous tile executed by the same processor mostly ﬁt
the cache. As a result, the data-locality improves since the
overlapping boundaries of consecutive tiles stay in cache.
To guarantee parallel efﬁciency, we enforce a total number
of tiles within 5% of an integer multiple of the number of
processors T and for every dimension a tile count within2%
of an integer multiple of the domain size.
C. Performance Model
The optimizer ﬁnally instantiates the performance model
based on the stencil sequence and target system parameters
shown by Fig. 4.
The performance model distinguishes two cost components:
(1) the peel cost models the latency and (2) the body cost
models the throughput of the innermost loop executions. In
other words, thepeel cost accounts for loop startup overheads
– examples are the over fetch at the loop boundaries or the
execution of scalar peel loops – while thebody cost models the
steady-state of the loop execution. For both components, we
model the memory accesses for two memory hierarchy levels:
(1) the fast memory (L2 cache) and (2) theslow memory (L3
nx=ny=2 
Dx=Dy=4 
DxDy
body domain
nxny
peel domain
nxDy
DxnyDynx
y
x
y
x
y
x
Fig. 5: Illustration of the peel and body domain computation
for domain size8×8 split into 2×2 tiles with boundary width
one along the positive axis directions.
cache or DDR memory). For every data element, we assume
the slow memory handles the ﬁrst and the fast memory all
subsequent accesses during the tile execution. To estimate the
execution time, the performance model multiplies the number
of memory accesses with the learned model parameters.
The loop startup overheads make long tiles along the
innermost loop dimension more efﬁcient. To model this effect,
we distinguish the peel cost proportional to the number of
innermost loop executions (peel domain) and the body cost
proportional to the number of innermost loop iterations (body
domain). This separation allows us to assign a higher cost to
memory accesses executed during the loop startup. The two
cost components and the goal to employ efﬁcient integer linear
programming solvers result in linear cost functions of the form
Px +By that sum the peel costPx and the body costBy. The
variables x and y denote the number of memory accesses for
the peel and body domains, respectively. The learned model
parameters P and B convert the memory accesses to execution
times. Sec. III-D details how themodel learner determines the
model parameters.
The performance model combines multiple cost functions
to estimate the stencil sequence execution time. To deﬁne
the cost functions, we next introduce the peel and body
functions that compute the weighted size of the peel and body
domains, respectively. Fig. 5 shows the computation of the
peel domain (left) and the body domain (right) for a simpliﬁed
two-dimensional domain (middle) with all weights set to one.
The peel domain counts the blue points (squares) while the
body domain counts all points (squares and circles). The peel
and body functions extend this computation with additional
terms and factors to model our three-dimensional domain and
parametric weights.
a) peel function: The peel cost is proportional to the
number of innermost loop executions. Without loss of gen-
erality, we assume the innermost loops execute along the
x-dimension, which means the number of innermost loop
executions corresponds to the size of the tiles projected to
the yz-plane.
The product DyDz of the domain sizes is equal to the sum
of the tile domains and the productsDzny
i and Dynz
i of the
tile counts with the perpendicular domain size approximate
the tile boundaries. To sum the tiles along the innermost loop
372
dimension, we multiply the terms with the tile count along the
x-dimension. This approximation is exact except for the tile
corners. To evaluate the peel cost, we deﬁne for every stencil
i ∈ [0,N ) the peel function
fp
i (w, wy,w z)= nx
i (DyDzw + Dzny
i wy + Dynz
i wz)
which scales the inner domain and the boundary terms with
the inner weight w and the boundary weights wy and wz,
respectively. For example, we set the inner weight to one and
the boundary weights to the evaluation boundary widths to
count the innermost loop executions.
b) body function: The body cost is proportional to the
number of innermost loop iterations scaled with cost function-
speciﬁc weights. The number of innermost loop iterations
is equal to the sum of the tile volumes. To compute the
volume of the overlapping tiles, we add the productDxDyDz
of the domain sizes to the tile counts multiplied with the
perpendicular domain sizes. This approximation again includes
the tile domains and the tile boundaries without the tile
corners. To evaluate the body cost, we deﬁne for every stencil
i ∈ [0,N ) the body function
fb
i (w, wx,w y,w z)= DxDyDzw+DyDznx
i wx+
DxDzny
i wy+DxDynz
i wz
which scales the inner domain and the boundary terms with
the inner weight w and the boundary weights wx, wy, and
wz, respectively. For example, we set the inner weight to one
and the boundary weights to the evaluation boundary widths
to count the innermost loop iterations.
The peel and body functions next allow us to deﬁne the cost
functions for the two memory hierarchy levels.
c) fast memory: The fast memory model counts the
memory accesses to estimate the stencil execution time. We
assume that every evaluation of the stencili loads the entire
access set Ai and stores the result. The stencili thus performs
1+|Ai|memory accesses per evaluation. To count the memory
accesses, we set for every stencili ∈ [0,N ) the weight
ci =1+ |Ai|
to the number of memory accesses per stencil evaluation and
for every dimension d the boundary weight
cd
i =( 1+ |Ai|)(ed−
i + ed+
i )
to the number of memory accesses per stencil evaluation
scaled with the evaluation boundary widths. The multiplication
reﬂects that the stencils are evaluated at every evaluation
boundary line. We then set for every stencil i ∈ [0,N ) the
peel cost pf
i and the body cost bf
i of the fast memory model
to the products
pf
i = P f fp
i (ci,c y
i ,c z
i ),b f
i = Bf fb
i (ci,c x
i ,c y
i ,c z
i )
which evaluate the peel and body functions to obtain the
number of memory accesses for the peel and body domains,
respectively. The learned model parametersP f and Bf con-
vert the memory accesses to execution times.
d) slow memory: The slow memory model determines
the communication volume to estimate the execution time. We
observe that the memory throughput improves with the number
of parallel access streams. To model this behavior, we sum two
cost functions that estimate the base cost and the variable
cost with respect to the number of access streams. We assume
that every stencil group loads and stores an array only once.
Repeated accesses of the same array hit the fast memory and
are not relevant for the slow memory model.
We compute the slow memory loads and stores based on
the group indexes. A stencil only loads an array from slow
memory if the group index of the stencil that accessed the
array last differs. Otherwise, the array was already loaded to
the fast memory. A stencil only stores an array to slow memory
if the group index of the last stencil that accesses the array
differs. Otherwise, the array is not used outside of the stencil
group, and storing to slow memory is not necessary.
To estimate thebase cost, we set for every stencili ∈ [0,N )
the weight mi to one if the stencil loads or stores at least one
array and to zero otherwise. We also set for every dimension
d the boundary weight
md
i = mi(ed−
i + ed+
i )
to the weight times the evaluation boundary widths. We then
set for every stencil i ∈ [0,N ) the peel cost pb
i and the body
cost bb
i of the base cost to the products
pb
i = P bfp
i (mi,m y
i ,m z
i ),b b
i = Bbfb
i (mi,m x
i ,m y
i ,m z
i )
which evaluate the peel and body functions to obtain the
number of stencil evaluations that access at least one array
for the peel and body domains, respectively. The learned
model parameters P b and Bb convert the stencil evaluations
to execution times.
To estimate the variable cost , we set for every stencil
i ∈ [0,N ) the weight si to the number of accessed arrays and
for every dimension d the boundary weights sd
i to the sum
of the array access boundary widths along the dimension. We
consider only arrays and boundary lines that have not been
accessed by a preceding stencil of the same stencil group.
To compute access boundary widths, we extend for every
data dependency (j, (B−,B +)) ∈B d
i the evaluation boundary
widths with the access bounds B− and B+. We then set for
every stencil i ∈ [0,N ) the peel cost pv
i and the body costbv
i
of the variable cost to the products
pv
i = P vfp
i (si,s y
i ,s z
i ),b v
i = Bvfb
i (si,s x
i ,s y
i ,s z
i )
which evaluate the peel and body functions to obtain the
number of access streams for the peel and body domains,
respectively. The learned model parametersP v and Bv convert
the access streams to execution times.
The slow memory model ﬁnally sums thebase cost and the
variable cost to estimate the execution time.
To estimate the overall stencil execution time, we assume
that the fast memory and the slow memory accesses overlap.
We thus compute for every stencil the maximumpeel cost and
373
the maximum body cost of the two memory hierarchy levels.
The sum
∑ N−1
i=0 max(pf
i ,p b
i + pv
i )+m a x (bf
i ,b b
i + bv
i )
accumulates the individual stencil execution times to obtain
the execution time of the entire stencil sequence. The term
∑ N−1
i=0 2 ·3(Bb + Bv)nx
i ny
i nz
i
emulates the slow memory access cost to load the two pre-
computed tile loop bounds for all three dimensions to account
for the tile execution overheads. We include this term in the
estimated execution time to favor implementation variants with
fewer tiles. Together, the estimated execution time and the tile
execution overheads deﬁne the objective function of the integer
linear program.
D. Learning the Performance Model
The model learner adapts the performance model param-
eters to the performance characteristics of the target system.
To learn the parameters, we implemented training stencils that
either stress the slow or the fast memory and measure their
execution time for different tile sizes. We then compute the
model parameters using least absolute deviations (LAD) [11]
regression, which compared to least squares regression has
better outlier robustness.
As the performance depends on the tile shape, we bench-
mark the training stencils with tile sizes ranging from 10 to
80 elements along thex-dimension and from1 to 55 elements
along the other dimensions. We exclude tiles with a volume
below 500 or above 2000 elements to ensure that the tiles ﬁt
the fast memory (L2 cache). In total, we run 103 tile size
conﬁgurations.
When learning the fast memory model, the fast memory
accesses have to dominate the execution times of the training
stencils. We used three training stencils that access 12, 16, and
20 array positions. We always connect nine identical stencils
that access the same input array to one training sequence. The
repeated accesses of the same input array guarantee that the
fast memory accesses dominate the execution time.
We benchmark the three training sequences for all tile
size conﬁgurations. For every run r ∈ [0,R ), we collect the
measured execution time tr and compute the number of fast
memory accesses xr and yr for the peel and body domain,
respectively. The LAD regression
(P f ,B f ) = argmin
(P,B)∈R2
∑
r∈[0,R) |(Px r + Byr) −tr|
then selects the fast memory model parameters P f and Bf
that minimize the L1-norm of the prediction error.
When learning the slow memory model, the slow memory
accesses have to dominate the execution times. We used nine
training stencils that access 1, 2, or 3 input arrays with access
boundary width 0, 1, or 2. The stencils access the input arrays
at three diagonal offsets to avoid unnecessary fast memory
accesses. We always connect nine identical stencils that access
different input and output arrays to one training sequence. The
many loaded and stored arrays guarantee that the slow memory
accesses dominate the execution time.
We benchmark the nine training sequences for all tile size
conﬁgurations. For every run r ∈ [0,R ), we collect the mea-
sured execution time tr. To learn the base cost, we compute
the number of stencil evaluationsxr and yr that perform slow
memory accesses for the peel and body domain, respectively.
To learn the variable cost, we compute the number of slow
memory accesses ur and vr for the peel and body domain,
respectively. The LAD regression
(P b,B b,P v,B v) = argmin
(P ′,B′,P ′′,B′′)∈R4
∑
r∈[0,R) |(P ′xr+
B′yr + P ′′ur+B′′vr) −tr|
then selects the slow memory model parameters P b, Bb, P v,
and Bv that minimize the L1-norm of the prediction error.
All training sequences are synthetic and differ from the
application kernels tuned in Sec. V -D.
IV . OPTIMIZA TION
The number of possible data-locality transformations de-
ﬁned in Sec. III-B makes the manual tuning of stencil pro-
grams difﬁcult. To automate the process, we could exhaus-
tively search for the optimal data-locality transformations
according to the performance model introduced in Sec. III-C.
However, for stencil sequences of lengthN the search space
contains O(2N ND xDyDz) implementation variants which
decompose into 2N fusion choices multiplied with up to N
stencil groups and DxDyDz tile size choices. This large
search space motivates advanced optimization methods.
To explore the search space, we rely on the well established
mixed-integer linear programming (MILP) approach, which
ﬁnds or approximates the optimal solution within some pre-
deﬁned objective function gap. The optimizer translates the
performance model and the space of data-locality transforma-
tions to an MILP that deﬁnes the optimization problem. We
next detail the automatic translation of the performance model
to linear constraints.
A. Linearizing Multiplications
The performance model multiplies the tile count variables
with other variables. Linear programs cannot directly express
the product of two integer variables. An implementation
trick [12] nevertheless allows us to multiply two variablesx
and y with known upper bounds X and Y .
We ﬁrst observe that the product of the binary variable b
and the variable x with known upper bound X translates to
three constraints. The constraint0 ≤ p ≤ x limits the product
p to the range [0,x ], while the constraints
p −Xb ≤ 0 and p −x −Xb ≥− X
force the product to zero ifb is zero and to x otherwise.
To express the product of two variables x and y with the
known upper bounds X and Y , we next encode the variable
y with the sum
y = ∑ ⌊log2(Y )⌋
i=0 2iyi
374
where the binary variables yi represent the digits of y. Then
the product p = xy corresponds to the sum
p = ∑ ⌊log2(Y )⌋
i=0 2ixyi
of the binary products xyi scaled with the power of two
associated with the respective digit. All binary products are
translated to the constraints introduced before.
The optimizer implements the performance model by intro-
ducing binary representations for all tile count variables and
lowers the products as shown above. This solution works since
we know that for every dimensiond the range [1,D d] limits
the tile count variables.
B. Modeling Stencil Groups
The number of stencil groups is an optimization variable
not known during the generation of the optimization problem.
Allocating one variable per stencil group to store group prop-
erties such as the tile count is thus not possible. Instead, we
model stencil group properties with the help of stencil speciﬁc
variables. At optimization time, the group index variables of
Sec. III-B allow us to compute stencil group properties and to
assign them to all stencil speciﬁc variables of the group.
The group indexes increase monotonically along the stencil
sequence. The constraint g0 =0 sets the group index of the
ﬁrst stencil to zero. To limit the remaining group indexes, we
deﬁne for every stencili ∈ [0,N −1) the constraint
0 ≤ gi+1 −gi ≤ 1,
which sets the group index difference of succeeding stencils
to zero or one for fusion and no fusion, respectively.
With the help of the group indexes, we deﬁne constraints
that apply to the stencil groups. For example, the tile counts
have to be equal within the stencil group. To enforce equality,
we deﬁne for every stencil i ∈ [0,N −1) and for every
dimension d the constraints
nd
i+1 −nd
i + Dd(gi+1 −gi) ≥ 0,
nd
i+1 −nd
i −Dd(gi+1 −gi) ≤ 0
which limit the tile count difference to zero if the stencils have
the same group index. Otherwise, the group index difference
gi+1 −gi is positive since the indexes increase along the stencil
sequence. Then the group index difference multiplied with
the upper bound Dd for the tile count difference nd
i+1 − nd
i
disables the constraints for all possible tile count assignments.
The upper bound follows from the observation that the tile
counts range from one to the domain sizeDd.
The optimizer uses the group index variables to model the
tile counts, the cache utilization, and the number of slow
memory accesses.
V. EV ALUA TION
To validate our approach, we learn the performance model
for three target systems and compare application kernels tuned
with Absinthe to heuristically tuned, hand-tuned, and auto-
tuned implementation variants.
1 STENCILS = {
2 "s0":"auto res = 0.5*(i0(x+1,y,z)+i0(x,y,z));",
3 "s1":"auto res = i1(x,y,z)*(s0(x,y+1,z)+s0(x,y-1,z));"}
Fig. 6: Absinthe version of the example stencil sequence
1 #pragma omp parallel for schedule(static)
2 for(int i d x=0 ;i d x<1 * 3 * 12; ++idx) {
3 // views of the input and output arrays
4 loop_info l = _tiles_group0[idx];
5 array_view_3d i1(&__i1(l.xbeg, l.ybeg, l.zbeg));
6 array_view_3d i0(&__i0(l.xbeg, l.ybeg, l.zbeg));
7 array_view_3d s1(&__s1(l.xbeg, l.ybeg, l.zbeg));
8 // stack allocated temporary arrays
9 tarray0_3d ___s0;
10 tarray0_view_3d s0(&___s0(HX, HY, HZ));
11
12 { // apply s0 stencil
13 int xbeg = _loops_s0[idx].xbeg;
14 int xend = _loops_s0[idx].xend;
15 int ybeg = _loops_s0[idx] ybeg;
16 int yend = _loops_s0[idx].yend;
17 int zbeg = _loops_s0[idx].zbeg;
18 int zend = _loops_s0[idx].zend;
19 for(int z = zbeg; z < zend; ++z)
20 for(int y = ybeg; y < yend; ++y)
21 #pragma omp simd
22 for(int x = xbeg; x < xend; ++x) {
23 auto res = 0.5 (i0(x+1,y,z) + i0(x,y,z));
24 s0(x,y,z) = res;
25 }}
26
27 { // apply s1 stencil
28 int xbeg = _loops_s1[idx].xbeg;
29 int xend = _loops_s1[idx].xend;
30 int ybeg = _loops_s1[idx] ybeg;
31 int yend = _loops_s1[idx].yend;
32 int zbeg = _loops_s1[idx].zbeg;
33 int zend = _loops_s1[idx].zend;
34 for(int z = zbeg; z < zend; ++z)
35 for(int y = ybeg; y < yend; ++y)
36 #pragma omp simd
37 for(int x = xbeg; x < xend; ++x) {
38 auto res = i1(x,y,z) * (s0(x,y+1,z) + s0(x,y-1,z));
39 s1(x,y,z) = res;
40 }}
41 }
Fig. 7: Optimized code for the example stencil sequence
A. Setup & Methodology
The target systems feature Xeon E5-2695 v4, Xeon Phi
7210, and Power8NVL sockets. We conﬁgure the Xeon Phi
sockets with two NUMA domains, each of them with 32
processors and three DDR channels, and run the experiments
on one of the two NUMA domains. We optimize the linear
programs with CPLEX 12.6.3 and compile the generated C++
codes with GCC 5.3 on the Xeon and Xeon Phi systems and
with GCC 5.4 on the Power system.
To perform the experiments, we set the domain size to64×
64 × 60 elements with 3 × 3 × 3 halo elements similar to
the COSMO [4] production conﬁguration. All experiments are
performed using double-precision ﬂoating-point numbers.
We set the number of processors to the available coresT =
18, T =3 2, and T =1 0 for the Xeon, Xeon Phi, and Power
systems, respectively.
To measure the execution time, we repeat every experiment
64 times and discard the ﬁrst16 measurements to warmup the
375
p=12
p=16
p=20
Pf
= 2e-07  Bf
= 1.8e-080.00
0.05
0.10
20 40 60 80
x
execution time [ms]
fast memory (Xeon)
p=12
p=16
p=20
Pf
= 4.4e-07  Bf
= 1.9e-080.00
0.10
0.20
0.30
20 40 60 80
x
execution time [ms]
fast memory (Xeon Phi)
p=12
p=16
p=20
Pf
= 1.6e-07  Bf
= 1.7e-080.00
0.02
0.04
0.06
20 40 60 80
x
execution time [ms]
fast memory (Power)
i=1,b=1
i=2,b=1
i=3,b=1
Pb
= 1.7e-06  Bb
= 7.5e-08
Pv
= 8.9e-07  Bv
= 6.5e-080.00
0.05
0.10
0.15
0.20
20 40 60 80
x
execution time [ms]
slow memory (Xeon)
i=1,b=1
i=2,b=1
i=3,b=1
Pb
= 3.4e-06  Bb
= 2.6e-07
Pv
= 7.9e-07  Bv
= 1.8e-070.00
0.25
0.50
0.75
20 40 60 80
x
execution time [ms]
slow memory (Xeon Phi)
i=1,b=1
i=2,b=1
i=3,b=1
Pb
= 4.7e-06  Bb
= 2.4e-07
Pv
= 1.8e-07  Bv
= 3.7e-080.00
0.03
0.06
0.09
20 40 60 80
x
execution time [ms]
slow memory (Power)
Fig. 8: Measured (polygons) and estimated (lines) execution times for the fast memory (p=positions) and slow memory (i=input
arrays and b=boundary width) training stencils and variable tile sizes along thex-dimension.
memory hierarchy. Before every run, except when learning
the fast memory model, we ﬂush the L1 and L2 caches with
dummy data. As we assume exclusive system access, we run
one thread per processor. We time only the stencil executions,
which excludes the initialization logic and the boundary con-
ditions. All plots show median values and nonparametric95%
conﬁdence intervals [13] to visualize the distribution of the
measurements.
B. Implementation
Absinthe provides a high-level stencil DSL to implement
stencil programs. Fig. 6 and Fig. 7 show the DSL version and
the generated code for the example stencil sequence introduced
in Sec. II-B, respectively. Absinthe parses the DSL to extract
the accesses patterns. Based on this analysis, the optimizer
derives the integer linear program and determines the optimal
solution using the CPLEX solver [14]. After the optimization,
the code generator emits C++ code that implements the fusion
and tile size choices of the optimal solution.
The code generator performs overlapped tiling [10] with
one tiling hierarchy level and periodic boundary conditions.
In addition to the stencil sequence, we also generate the
boilerplate necessary to execute, benchmark, and verify the
stencil sequence. The veriﬁcation compares the results of the
parallel implementation to naive sequential code. The code
generator utilizes the Jinja2 template engine [15] to specialize
a generic stencil sequence template with the program-speciﬁc
logic.
C. Learning the Target Systems
Absinthe learns the performance model parameters once for
every target system and then tunes all stencil programs using
the same parameter set. Sec. III-D discusses the performance
model learning. We next evaluate the quality of the learned
model parameters.
To improve the noise robustness, we use all 48 measure-
ments per experiment when learning the model parameters
using LAD regression [11]. We use the median of the repeated
measurements when computing the R2 values.
Fig. 8 compares for the tile sizes 5 × 5 × x the measured
execution times of the training stencils to the learned fast
memory and slow memory models. We observe that for the
shown tile sizes, the execution times increase almost linearly
with the tile size along thex-dimension with model predictions
close to the measured execution times of the training stencils.
The annotations mark the different training stencils. For ex-
ample, the annotationp =1 2refers to the training stencil that
accesses twelve positions, and the annotation i =3 ,b =1
refers to the training stencil that accesses three input arrays
with boundary width one.
376
absinthe
hand
min
max
(74.0%)
auto-tuning
(-6.5%)0.5
0.7
0.9
1.1
1.3
0.5 0.7 0.9 1.1 1.3
estimated time [ms]
measured time [ms]
fastwaves (Xeon)
absinthe
hand
min
max
auto-tuning
(-0.8%)
0.4
0.8
1.2
1.6
0.4 0.8 1.2 1.6
estimated time [ms]
measured time [ms]
diusion (Xeon)
absinthehand
minmax
auto-tuning
(-3.4%)
0.3
0.4
0.5
0.6
0.7
0.8
0.3 0.4 0.5 0.6 0.7 0.8
estimated time [ms]
measured time [ms]
advection (Xeon)
absinthe
hand min
max
auto-tuning
(-0.7%)
1.5
2.0
2.5
1.5 2.0 2.5
estimated time [ms]
measured time [ms]
fastwaves (Xeon Phi)
absinthehand
min
max
auto-tuning
(-7.8%)1.0
2.0
3.0
4.0
1.0 2.0 3.0 4.0
estimated time [ms]
measured time [ms]
diusion (Xeon Phi)
absinthe
hand
min
max
auto-tuning
(-7.1%)
1.0
1.5
2.0
1.0 1.5 2.0
estimated time [ms]
measured time [ms]
advection (Xeon Phi)
absinthe
hand
min
max
auto-tuning
(-6.1%)
0.6
0.8
1.0
1.2
0.6 0.8 1.0 1.2
estimated time [ms]
measured time [ms]
fastwaves (Power)
absinthe
hand
min
max
auto-tuning
(-2.5%)0.4
0.8
1.2
1.6
0.4 0.8 1.2 1.6
estimated time [ms]
measured time [ms]
diusion (Power)
absinthe
hand
min
max
auto-tuning
(-1.7%)
0.8
1.2
1.6
0.8 1.2 1.6
estimated time [ms]
measured time [ms]
advection (Power)
Fig. 9: Measured and estimated execution times for the optimal (triangle), selected (squares), and random (dots) implementation
variants of the fastwaves (N =9 ), diffusion (N =1 6), and advection (N =8 ) kernels.
The R2 values of 0.87, 0.95, and 0.94 for the fast memory
model and of0.96, 0.96, and 0.90 for the slow memory model
conﬁrm the quality of the learned model parameters for the
Xeon, Xeon Phi, and Power systems, respectively.
D. Tuning the Application Kernels
Existing benchmark suites such as PolyBench [16] often
contain stencil programs that iterate only one stencil instead of
multiple different stencils. To evaluate the quality of our fusion
and tile size selection choices, we thus implement three stencil
sequences from the COSMO atmospheric model [4]. These
real-world benchmark kernels contain one-, two-, and three-
dimensional stencils from ﬁrst to ﬁfth order. The fastwaves
kernel consists of nine stencils that compute the pressure gradi-
ent, update the horizontal wind speeds, and compute the wind
divergence. The diffusion kernel consists of sixteen stencils
that update the pressure and the wind speeds. The advection
kernel consists of eight stencils that transport the horizontal
wind speeds. The two-dimensional advection and diffusion
stencils access only neighbor elements in the horizontal xy-
plane, while the fastwaves stencils perform three-dimensional
accesses.
To perform the experiments, we adapt the COSMO stencils
377
to match the current implementation of our code generator,
which supports only three-dimensional arrays and periodic
boundary conditions. We thus replace the original boundary
conditions and remove accesses to lower-dimensional arrays.
Fig. 9 shows the performance of Absinthe for all application
kernels and target systems. We compare the measured and
estimated execution times of the optimal solution found by
Absinthe to selected and random implementation variants with
group index and tile size constraints. Data points close to
the diagonal imply good model prediction. The dashed lines
delimit the region with 20% prediction error. The timings
include the stencil computation without boundary conditions.
Additionally, we add the following selected implementation
variants: the min and max heuristics combine minimal and
maximal fusion with the Absinthe tile size selection, the
hand approach reproduces the hand-tuned fusion and tile size
choices of the COSMO production code, and theauto-tuning
approach combines tile size auto-tuning with the Absinthe
fusion choices. As the hand and auto-tuning variants may
violate the cache size or load imbalance constraints, their
estimated execution times are possibly invalid.
The optimal solutions for the three application kernels con-
tain at most four stencil groups. To sample random implemen-
tation variants, we select20 random group index assignments
with at most four groups and repeat the optimization with
constraints that ﬁx the group indexes. To examine different
tile sizes, we also introduce tile size constraints that enforce
smaller or larger tiles along one dimension. In total, we sample
60 random implementation variants.
The auto-tuning exhaustively searches for every stencil
group the tile sizes1, 2, 4, 12, 30, and 60 in the z-dimension
and the powers of two in thexy-plane. The tuning of isolated
stencil groups does not consider the cache reuse of consecutive
stencil groups. However, the approach circumvents the joint
evaluation of all stencil group tile size combinations. We
use the Absinthe fusion strategy to avoid auto-tuning the
exponential fusion search space.
a) fastwaves: The optimal solution for all target systems
splits the fastwaves kernel into two groups (Fig. 1 shows the
optimal solution for the Xeon system). The tile shapes reﬂect
the three-dimensional access patterns detailed in Fig. 10.
b) diffusion: The optimal solutions split the diffusion
kernel into two, one, and four groups of equal sizes with tile
size 64×13×1, 64×16×1, and64×32×1 for the Xeon, Xeon
Phi, and Power systems, respectively. These choices reﬂect the
two-dimensional access pattern of the stencils and the different
L2 cache capacities. Most implementation variants perform
better than expected. We attribute this bias to the peel cost
of the slow memory model, which do not consider the cache
reuse of consecutive innermost loop executions that span the
full domain.
c) advection: The optimal solution of the advection
kernel fuses all stencils with tile size64 ×16 ×1, 64 ×32 ×1,
and 64 ×32 ×1 for the Xeon, Xeon Phi, and Power systems,
respectively. The fast memory model dominates the predicted
execution time of the compute-intensive seven-point stencils.
j+1
k+1 
j+1,k+1
i+1
k+1 
i+1,k+1
k-1
k+1
i+1 i+1j+1
j+1
j+1 i+1
j+1 i+1
k-1k-1
i-1j-1
i-1
k+1
i-1,k+1
j-1
k+1
j-1,k+1
j-1 i-1
input
temp
output
arrays: example:
y(i,j,k) =
  x(i,j,k) + 
  x(i+1,j,k);
i+1x y
Fig. 10: Data-ﬂow graph of the fastwaves kernel. All edges are
annotated with the non-center access offsets that the stencils
read in addition to the center position (i,j,k).
Especially for the Xeon Phi and Power systems, the fast
memory model tends to underestimate the measured execution
times. For example, since the cache accesses may not fully
overlap with the actual stencil computation.
The auto-tuned versions of the fastwaves, diffusion, and
advection kernels perform 6.5%, 0.8%, and 3.4% faster than
Absinthe for the Xeon system, 0.7%, 7.8%, and 7.1% faster
than Absinthe for the Xeon Phi system, and6.1%, 2.5%, and
1.7% faster than Absinthe for the Power system, respectively.
The small performance penalty compared to the much slower
auto-tuning and the relative agreement of estimated and mea-
sured execution times demonstrate the effectiveness of our
approach for different stencils and hardware architectures.
The hand-tuned kernels perform well, but the manual opti-
mization of large codes is tedious and time-consuming. The
combination of fusion heuristics with Absinthe demonstrates
the challenge of independent fusion and tile size selection.
The auto-tuning approach always works best since it does
not depend on the performance model assumptions. For ex-
ample, the auto-tuned tile sizes violate the cache capacity
constraints of the Power system, which means tiles ﬁtting the
L2 cache are not optimal for this architecture. Auto-tuning
generates 277 implementation variants for every stencil group,
which on the Xeon system results in40 minutes search time
for the diffusion kernel. Extending the auto-tuning to the215
fusion choices increases the search time beyond10′000 hours.
Absinthe explores the full search space in40 seconds.
378
1.66x
1.29x
1x
2.03x
3.7x
1x
1.4x
1.06x1x
0
10
20
fastwaves advection di usion
execution time [ms]
Absinthe
Halide
Polymage
Fig. 11: Execution times of the Absinthe, Halide, and Poly-
mage tuned application kernels for domain size256×256×60
on the Xeon system (slowdowns relative to Absinthe).
E. Comparison with Halide and Polymage
To compare Absinthe, we implement the application kernels
with Polymage [3] (git:a8a101b) and Halide [2] (git:3af2386).
We optimize the stencil sequences with the built-in auto-
schedulers [17], [18], compile the Absinthe and Polymage
kernels with GCC 5.3, and adapt the scheduling parameters
to match the processor count of the Xeon system.
Fig. 11 compares the execution times for the Absinthe,
Polymage, and Halide tuned application kernels. We set the
domain size to 256 × 256 × 60 elements since Halide and
Polymage do not perform well for small domains. Absinthe
and Polymage apply the same code transformations and use
the same compiler, which makes the results comparable.
Halide compiles with LL VM and performs loop reordering and
stencil inlining, which reduces the signiﬁcance of the results.
Absinthe performs best for all kernels, which emphasizes the
quality of the fusion and tile size selection choices.
VI. R ELA TED WORK
Tile size selection is a well-researched topic with two main
directions: purely analytic approaches [19], [20], [21], [22],
[23], [24], [25], [26], [27], [28] and empirical approaches [29],
[30], [31], [32] that search different conﬁgurations for optimal
performance. Y uki et al. [33] learn machine-speciﬁc static tile
size selection models. Artiﬁcial neuronal networks are also
effective for both instruction throughput prediction [34] and
tile size selection [35]. All of these works regard tile size
selection as a problem that must be solved after having already
committed to a certain loop structure.
D. Cociorva et al. [36] observe that program scheduling
and tile size selection are intertwined and propose a dynamic
programming based approach for combined scheduling and tile
size selection speciﬁc for tensor sequences. Their work does
not consider stencil computations. Quasem and Kennedy [37]
propose a model guided empirical approach for loop fusion
and tiling. Beaugnon et al. [38] also combine analytical mod-
eling and empirical search space exploration. They present an
analytical model to compute a lower bound for the execution
time of partially-speciﬁed program variants that allows them to
prune the search space early-on. None of these works provide
a linear programming formulation.
There exist several approaches for generating code for
iterative stencils. Patus [39] is a code generator for iterative
stencils on CPUs and GPUs. Henretty et al. [40] introduced
a code generator for iterative multi-statement stencils that
implements the DLT [41] data layout transformation. Both
code generators rely on tile size auto-tuning. Pochoir [42] is
an iterative stencil compiler that uses cache-oblivious tiling
techniques to avoid the tile size selection problem. Prajapati
et al. [43] manually derive tile size selection models for
single statement stencils executed on GPUs. They require non-
linear integer programming which takes hours to terminate and
commonly does not guarantee optimal solutions.
STELLA [44] is a domain-speciﬁc language for climate
modeling. Halide [2] and Polymage [3] are domain-speciﬁc
languages for image processing pipelines. All approaches
support the optimization of stencil programs with data-locality
transformations. MODESTO [9] is an analytic performance
model to derive optimal fusion patterns for stencil programs
based on memory bandwidth estimates. However, the model
does not consider loop overheads and other metrics impor-
tant for good tile size selection. For Polymage, Jangda and
Bondhugula [18] employ dynamic programming to explore the
space of fusion choices according to a cost function. During
the optimization, a heuristic selects suitable tile sizes. For
Halide, Liao et al. [45] and Mullapudi et al. [17] suggest
cost functions and custom optimization strategies to perform
automatic scheduling, which covers tile size selection. These
solutions do not integrate the fusion and tile size selection
choice in a single linear model. Absinthe thus provides the ﬁrst
holistic integer linear programming formulation that simulta-
neously schedules stencil programs and chooses matching tile
sizes.
VII. C ONCLUSION
Absinthe instantiates an optimization problem that evaluates
a learned performance model to select target system speciﬁc
data-locality transformations for stencil codes. Surprisingly six
performance model parameters are sufﬁcient to capture the
relevant target system characteristics. The evaluation of the
performance model is fast and requires no complex operations.
We also demonstrate how to linearize the performance model
for stencil codes with known domain sizes of limited range.
These properties facilitate the efﬁcient exploration of the
exponential search space with the help of powerful linear
solvers. Our empirical evaluation provides strong evidence that
learning a target-speciﬁc performance model is a competitive
alternative to auto-tuning.
ACKNOWLEDGMENTS
We want to thank the Swiss National Supercomputing
Center (CSCS) for granting access to the computing resources.
This project has received funding from the European Research
Council (ERC) under the European Union’s Horizon 2020
programme (grant agreement DAPP , No. 678880), the Swiss
National Science Foundation under the Ambizione programme
(grant PZ00P2168016), and ARM Holdings plc and Xilinx Inc
in the context of Polly Labs.
379
REFERENCES
[1] D. Unat, A. Dubey, T. Hoeﬂer, J. Shalf, M. Abraham, M. Bianco,
B. L. Chamberlain, R. Cledat, H. C. Edwards, H. Finkel, K. Fuerlinger,
F. Hannig, E. Jeannot, A. Kamil, J. Keasler, P . H. J. Kelly, V . Leung,
H. Ltaief, N. Maruyama, C. J. Newburn, , and M. Pericas, “Trends in
Data Locality Abstractions for HPC Systems,” IEEE Transactions on
Parallel and Distributed Systems (TPDS), vol. 28, no. 10, Oct. 2017.
[2] J. Ragan-Kelley, C. Barnes, A. Adams, S. Paris, F. Durand, and
S. Amarasinghe, “Halide: a language and compiler for optimizing
parallelism, locality, and recomputation in image processing pipelines,”
ACM SIGPLAN Notices, vol. 48, no. 6, pp. 519–530, 2013.
[3] R. T. Mullapudi, V . V asista, and U. Bondhugula, “Polymage: Automatic
optimization for image processing pipelines,”SIGARCH Comput. Archit.
News, vol. 43, no. 1, pp. 429–443, Mar. 2015.
[4] M. Baldauf, A. Seifert, J. F¨orstner, D. Majewski, M. Raschendorfer, and
T. Reinhardt, “Operational convective-scale numerical weather predic-
tion with the COSMO model: Description and sensitivities,” Monthly
Weather Review, vol. 139, no. 12, pp. 3887–3905, 2011.
[5] G. A. McMechan, “Migration by Extrapolation of Time-Dependent
Boundary V ALUES*,”Geophysical Prospecting, vol. 31, pp. 413–420,
Jun. 1983.
[6] A. Taﬂove, “Review of the formulation and applications of the ﬁnite-
difference time-domain method for numerical modeling of electromag-
netic wave interactions with arbitrary structures,”Wave Motion, vol. 10,
no. 6, pp. 547 – 582, 1988, special Issue on Numerical Methods for
Electromagnetic Wave Interactions.
[7] (1998) Consortium for small-scale modeling. [Online]. Available:
http://www.cosmo-model.org/
[8] O. Fuhrer, T. Chadha, T. Hoeﬂer, G. Kwasniewski, X. Lapillonne,
D. Leutwyler, D. L ¨uthi, C. Osuna, C. Sch ¨ar, T. C. Schulthess, and
H. V ogt, “Near-global climate simulation at 1 km resolution: establishing
a performance baseline on 4888 gpus with cosmo 5.0,” Geoscientiﬁc
Model Development Discussions, vol. 2017, pp. 1–27, 2017.
[9] T. Gysi, T. Grosser, and T. Hoeﬂer, “MODESTO: Data-centric analytic
optimization of complex stencil programs on heterogeneous architec-
tures,” in Proceedings of the 29th ACM on International Conference on
Supercomputing, ser. ICS ’15. New Y ork, NY , USA: ACM, 2015, pp.
177–186.
[10] X. Zhou, J.-P . Giacalone, M. J. Garzar ´an, R. H. Kuhn, Y . Ni, and
D. Padua, “Hierarchical overlapped tiling,” inProceedings of the Tenth
International Symposium on Code Generation and Optimization , ser.
CGO ’12. New Y ork, NY , USA: ACM, 2012, pp. 207–218.
[11] B. S. Cade and J. D. Richards, “Permutation tests for least absolute
deviation regression,” Biometrics, vol. 52, no. 3, pp. 886–902, 1996.
[12] “ILP Part 6 Faster multiplication,” https://blog.adamfurmanek.pl/2015/
09/26/ilp-part-6/, September 26th 2015.
[13] T. Hoeﬂer and R. Belli, “Scientiﬁc benchmarking of parallel computing
systems: Twelve ways to tell the masses when reporting performance
results,” in Proceedings of the International Conference for High Per-
formance Computing, Networking, Storage and Analysis, ser. SC ’15.
New Y ork, NY , USA: ACM, 2015, pp. 73:1–73:12.
[14] IBM, IBM ILOG CPLEX Optimization Studio CPLEX User’s Manual,
2011.
[15] “Welcome to Jinja2,” http://jinja.pocoo.org/docs/2.10/, 2008.
[16] L.-N. Pouchet, “Polybench: The polyhedral benchmark suite,” URL:
https://sourceforge.net/projects/polybench/, 2012.
[17] R. T. Mullapudi, A. Adams, D. Sharlet, J. Ragan-Kelley, and K. Fa-
tahalian, “Automatically scheduling halide image processing pipelines,”
ACM Trans. Graph., vol. 35, no. 4, pp. 83:1–83:11, Jul. 2016.
[18] A. Jangda and U. Bondhugula, “An effective fusion and tile size model
for optimizing image processing pipelines,” inProceedings of the 23rd
ACM SIGPLAN Symposium on Principles and Practice of Parallel
Programming, ser. PPoPP ’18. New Y ork, NY , USA: ACM, 2018,
pp. 261–275.
[19] M. D. Lam, E. E. Rothberg, and M. E. Wolf, “The cache performance
and optimizations of blocked algorithms,” inACM SIGARCH Computer
Architecture News, vol. 19, no. 2. ACM, 1991, pp. 63–74.
[20] R. Schreiber and J. J. Dongarra, “Automatic blocking of nested loops,”
1990.
[21] S. Coleman and K. S. McKinley, “Tile size selection using cache
organization and data layout,” inACM SIGPLAN Notices, vol. 30, no. 6.
ACM, 1995, pp. 279–290.
[22] C.-h. Hsu and U. Kremer, “A quantitative analysis of tile size selection
algorithms,” The Journal of Supercomputing, vol. 27, no. 3, pp. 279–294,
2004.
[23] V . Sarkar and N. Megiddo, “An analytical model for loop tiling and
its solution,” in Performance Analysis of Systems and Software, 2000.
ISPASS. 2000 IEEE International Symposium on . IEEE, 2000, pp.
146–153.
[24] N. Mitchell, K. H ¨ogstedt, L. Carter, and J. Ferrante, “Quantifying
the multi-level nature of tiling interactions,” International Journal of
Parallel Programming, vol. 26, no. 6, pp. 641–670, 1998.
[25] K. Y otov, X. Li, G. Ren, M. Garzaran, D. Padua, K. Pingali, and
P . Stodghill, “Is search really necessary to generate high-performance
blas?” Proceedings of the IEEE, vol. 93, no. 2, pp. 358–386, 2005.
[26] K. Esseghir, “Improving data locality for caches,” Ph.D. dissertation,
Rice University, 1993.
[27] G. Rivera and C.-W. Tseng, “A comparison of compiler tiling algo-
rithms,” in Compiler Construction. Springer, 1999, pp. 1–99.
[28] S. Mehta, G. Beeraka, and P .-C. Y ew, “Tile size selection revisited,”
ACM Trans. Archit. Code Optim., vol. 10, no. 4, pp. 35:1–35:27, Dec.
2013.
[29] R. C. Whaley and J. J. Dongarra, “Automatically tuned linear algebra
software,” in Proceedings of the 1998 ACM/IEEE conference on Super-
computing. IEEE Computer Society, 1998, pp. 1–27.
[30] P . M. Knijnenburg, T. Kisuki, K. Gallivan, and M. F. O’Boyle, “The
effect of cache models on iterative compilation for combined tiling and
unrolling,” Concurrency and Computation: Practice and Experience ,
vol. 16, no. 2-3, pp. 247–270, 2004.
[31] B. B. Fraguela, M. G. Carmueja, and D. Andrade, “Optimal tile size
selection guided by analytical models,”parameters, vol. 10, p. 14, 2005.
[32] J. Shirako, K. Sharma, N. Fauzia, L.-N. Pouchet, J. Ramanujam,
P . Sadayappan, and V . Sarkar, “Analytical bounds for optimal tile
size selection,” in Proceedings of the 21st International Conference
on Compiler Construction, ser. CC’12. Berlin, Heidelberg: Springer-
V erlag, 2012, pp. 101–121.
[33] T. Y uki, L. Renganarayanan, S. Rajopadhye, C. Anderson, A. E.
Eichenberger, and K. O’Brien, “Automatic creation of tile size selection
models,” in Proceedings of the 8th annual IEEE/ACM international
symposium on Code generation and optimization . ACM, 2010, pp.
190–199.
[34] C. Mendis, A. Renda, S. Amarasinghe, and M. Carbin, “Ithemal:
Accurate, portable and fast basic block throughput estimation using deep
neural networks,” 2018.
[35] M. Rahman, L.-N. Pouchet, and P . Sadayappan, “Neural network assisted
tile size selection,” 2010.
[36] D. Cociorva, J. W. Wilkins, C. Lam, G. Baumgartner, J. Ramanujam, and
P . Sadayappan, “Loop optimization for a class of memory-constrained
computations,” in Proceedings of the 15th International Conference on
Supercomputing, ser. ICS ’01. New Y ork, NY , USA: ACM, 2001, pp.
103–113.
[37] A. Qasem and K. Kennedy, “Proﬁtable loop fusion and tiling using
model-driven empirical search,” in Proceedings of the 20th Annual
International Conference on Supercomputing, ser. ICS ’06. New Y ork,
NY , USA: ACM, 2006, pp. 249–258.
[38] U. Beaugnon, A. Pouille, M. Pouzet, J. Pienaar, and A. Cohen, “Op-
timization space pruning without regrets,” in Proceedings of the 26th
International Conference on Compiler Construction, ser. CC 2017. New
Y ork, NY , USA: ACM, 2017, pp. 34–44.
[39] M. Christen, O. Schenk, and H. Burkhart, “Patus: A code generation
and autotuning framework for parallel iterative stencil computations
on modern microarchitectures,” in 2011 IEEE International Parallel
Distributed Processing Symposium, May 2011, pp. 676–687.
[40] T. Henretty, R. V eras, F. Franchetti, L.-N. Pouchet, J. Ramanujam, and
P . Sadayappan, “A stencil compiler for short-vector simd architectures,”
in Proceedings of the 27th International ACM Conference on Interna-
tional Conference on Supercomputing, ser. ICS ’13. New Y ork, NY ,
USA: ACM, 2013, pp. 13–24.
[41] T. Henretty, K. Stock, L.-N. Pouchet, F. Franchetti, J. Ramanujam, and
P . Sadayappan, “Data layout transformation for stencil computations on
short-vector simd architectures,” inProceedings of the 20th International
Conference on Compiler Construction: Part of the Joint European
Conferences on Theory and Practice of Software, ser. CC’11/ETAPS’11.
Berlin, Heidelberg: Springer-V erlag, 2011, pp. 225–245.
[42] Y . Tang, R. A. Chowdhury, B. C. Kuszmaul, C.-K. Luk, and C. E. Leis-
erson, “The pochoir stencil compiler,” inProceedings of the twenty-third
annual ACM symposium on Parallelism in algorithms and architectures.
ACM, 2011, pp. 117–128.
380
[43] N. Prajapati, W. Ranasinghe, S. Rajopadhye, R. Andonov, H. Djidjev,
and T. Grosser, “Simple, accurate, analytical time modeling and optimal
tile size selection for gpgpu stencils,” in Proceedings of the 22Nd
ACM SIGPLAN Symposium on Principles and Practice of Parallel
Programming, ser. PPoPP ’17. New Y ork, NY , USA: ACM, 2017,
pp. 163–177.
[44] T. Gysi, C. Osuna, O. Fuhrer, M. Bianco, and T. C. Schulthess, “Stella:
A domain-speciﬁc tool for structured grid methods in weather and
climate models,” in Proceedings of the International Conference for
High Performance Computing, Networking, Storage and Analysis, ser.
SC ’15. New Y ork, NY , USA: ACM, 2015, pp. 41:1–41:12.
[45] S. W. Liao, S. J. Tsai, C. H. Y ang, and C. K. Lo, “Locality-aware
scheduling for stencil code in halide,” in 2016 45th International
Conference on Parallel Processing Workshops (ICPPW), Aug 2016, pp.
72–77.
381

National Institutional Ranking Framework
Ministry of Education
Government of India
Welcome to Data Capturing System: ENGINEERING
Submitted Institute Data for NIRF'2024'
Institute Name: V. S. B. Engineering College [IR-E-C-36944]
Sanctioned (Approved) Intake
Academic Year 2022-23 2021-22 2020-21 2019-20 2018-19 2017-18
UG [4 Years Program(s)] 1080 1080 1050 1050 - -
PG [2 Year Program(s)] 45 45 - - - -
Total Actual Student Strength (Program(s) Offered by Your Institution)
(All programs
of all years)
No. of Male
Students
No. of Female
Students
Total Students Within State
(Including male
& female)
Outside State
(Including male
& female)
Outside
Country
(Including male
& female)
Economically
Backward
(Including male
& female)
Socially
Challenged
(SC+ST+OBC
Including male
& female)
No. of students
receiving full
tuition fee
reimbursement
from the State
and Central
Government
No. of students
receiving full
tuition fee
reimbursement
from Institution
Funds
No. of students
receiving full
tuition fee
reimbursement
from the Private
Bodies
No. of students
who are not
receiving full
tuition fee
reimbursement
UG [4 Years
Program(s)]
2267 1254 3521 3514 7 0 1578 1898 1146 771 0 1559
PG [2 Year
Program(s)]
5 16 21 21 0 0 9 12 12 0 0 9
Placement & Higher Studies
UG [4 Years Program(s)]: Placement & higher studies for previous 3 years
Academic Year No. of first year
students intake in the
year
No. of first year
students admitted in
the year
Academic Year No. of students
admitted through
Lateral entry
Academic Year No. of students
graduating in
minimum stipulated
time
No. of students
placed
Median salary of
placed
graduates(Amount in
Rs.)
No. of students
selected for Higher
Studies
2017-18 690 634 2018-19 12 2020-21 558 472 450000(Four Lakh Fifty
Thousand)
17
2018-19 870 684 2019-20 14 2021-22 560 487 476000(Four Lakh
seventy six Thousand)
16
2019-20 1050 843 2020-21 20 2022-23 766 680 485000(Four Lakhs
and eighty five
thousand)
5
PG [2 Years Program(s)]: Placement & higher studies for previous 3 years
Academic Year No. of first year
students intake in the
year
No. of first year
students admitted in
the year
Academic Year No. of students graduating in minimum
stipulated time
No. of students
placed
Median salary of
placed
graduates(Amount in
Rs.)
No. of students
selected for Higher
Studies
2019-20 54 10 2020-21 7 2 300000(Three Lakhs) 0
2020-21 45 5 2021-22 4 1 300000(Three Lakhs) 0
2021-22 45 9 2022-23 4 2 306000( three lakhs sis
thousand)
0
Ph.D Student Details
Ph.D (Student pursuing doctoral program till 2022-23 Students admitted in the academic year 2023-24 should not be entered here.)
Total Students
Full Time 0
Part Time 0
No. of Ph.D students graduated (including Integrated Ph.D)
2022-23 2021-22 2020-21
Full Time 0 0 0
Part Time 0 0 1
Financial Resources: Utilised Amount for the Capital expenditure for previous 3 years
Academic Year 2022-23 2021-22 2020-21
Utilised Amount Utilised Amount Utilised Amount
Annual Capital Expenditure on Academic Activities and Resources (excluding expenditure on buildings)
Library ( Books, Journals and e-Resources only) 3807023 (Thirty eight Lakhs Seven Thousand and twenty three
)
2902009 (Twenty Nine Lakhs Two Thousand and Nine) 237715 (Two Lakhs Thirty Seven Thousand Seven Hundred
and Fifteen)
New Equipment and software for Laboratories 7920927 (Seventy Nine Lakhs Twenty Thousand Nine Hundred
Twenty Seven )
5822398 (Fifty Eight Lakhs Twenty Two Thousand Three
Hundred and Ninety Eight)
5797130 (Fifty Seven Lakhs Ninety Seven Thousand One
Hundred and Thirty)
Engineering Workshops 2401745 (Twenty Four Lakhs One Thousand Seven Hundred
Forty Five )
1450542 (Fourteen Lakhs Fifty Thousand Five Hundred and
Forty Two)
1351945 (Thirteen Lakhs Fifty One Thousand Nine Hundred
and Forty Five)
Other expenditure on creation of Capital Assets (For setting up
classrooms, seminar hall, conference hall , library, Lab, Engg
workshops excluding expenditure on Land and Building)
22745562 (Two Crore Twenty Seven Lakhs Forty Five
Thousand Five Hundred Sixty Two)
2954641 (Twenty Nine Lakhs Fifty Four Thousand Six Hundred
and Forty One)
37378481 (Three Crore Seventy Three Lakhs Seventy Eight
Thousand Four Hundred and Eighty One)
Financial Resources: Utilised Amount for the Operational expenditure for previous 3 years
Academic Year 2022-23 2021-22 2020-21
Utilised Amount Utilised Amount Utilised Amount
Annual Operational Expenditure
Salaries (Teaching and Non Teaching staff) 107478268 (Ten Crore Seventy Four Lakhs Seventy Eight
Thousand Two Hundred Sixty Eight)
88842585 (Eight Crore Eighty Eight Lakhs Forty Two
Thousand Five Hundred and Eighty Five)
75443418 (Seven Crore Fifty Four Lakhs Forty Three
Thousand Four Hundred and Eighteen)
Maintenance of Academic Infrastructure or consumables and
other running expenditures(excluding maintenance of hostels
and allied services,rent of the building, depreciation cost, etc)
27973800 (Two Crore Seventy Nine Lakhs Seventy Three
Thousand Eight Hundred)
88922338 (Eight Crore Eighty Nine Lakhs Twenty Two
Thousand Three Hundred and Thirty Eight)
53584970 (Five Crore Thirty Five Lakhs Eighty Four Thousand
Nine Hundred and Seventy)
Seminars/Conferences/Workshops 2244064 (Twenty Two Lakhs Forty Four Thousand Sixty Four) 1009963 (Ten Lakhs Nine Thousand Nine Hundred and Sixty
Three)
1445441 (Fourteen Lakhs Forty Five Thousand Four Hundred
and Forty One)
IPR
Calendar year 2022 2021 2020
No. of Patents Published 24 24 9
No. of Patents Granted 10 1 0
Sponsored Research Details
Financial Year 2022-23 2021-22 2020-21
Total no. of Sponsored Projects 6 4 6
Total no. of Funding Agencies 2 3 1
Total Amount Received (Amount in Rupees) 198388 420763 3195777
Amount Received in Words One Lakh Ninety Eight Thousand Three Hundred and Eighty
Eight
Four Lakhs Twenty Thousand Seven Hundred and Sixty Three Thirty One Lakh Ninety Five thousand Seven Hundred and
Seventy Seven Only
Consultancy Project Details
Financial Year 2022-23 2021-22 2020-21
Total no. of Consultancy Projects 4 14 1
Total no. of Client Organizations 2 2 1
Total Amount Received (Amount in Rupees) 193265 436472 300000
Amount Received in Words One Lakh Ninety Three Thousand Two Hundred and Sixty Five Four Lakhs Thirty Six Thousand Four Hundred and Seventy
Two
Three lakhs only
PCS Facilities: Facilities of physically challenged students
1. Do your institution buildings have Lifts/Ramps? Yes, more than 80% of the buildings
2. Do your institution have provision for walking aids, including wheelchairs and transportation from one building to another for
handicapped students?
Yes
3. Do your institution buildings have specially designed toilets for handicapped students? Yes, more than 80% of the buildings
Faculty Details
Srno Name Age Designation Gender Qualification Experience (In
Months)
Currently working
with institution?
Joining Date Leaving Date Association type
1 DINESH M 33 Assistant Professor Male M.E. 132 Yes 04-06-2019 -- Regular
2 BALAJI V 31 Assistant Professor Male M.E. 123 Yes 14-05-2012 -- Regular
3 SENTHILKUMAR P
R
36 Assistant Professor Male M. Phil 157 No 01-08-2012 26-09-2023 Regular
4 MOHAMED
NASURUDEEN M
33 Assistant Professor Male M.Tech 132 Yes 16-05-2019 -- Regular
5 Dr
UMAMAHESWARI
K
41 Professor Female Ph.D 207 Yes 01-12-2017 -- Regular
6 KRISHNAN R 38 Assistant Professor Male M.E. 151 Yes 03-07-2013 -- Regular
7 GUNASEKARAN S 45 Associate Professor Male M.Tech 292 Yes 02-06-2010 -- Regular
8 DHANABAL M 32 Assistant Professor Male M.E. 137 Yes 02-07-2014 -- Regular
9 DR
KALYANASUNDAR
AM R
49 Assistant Professor Male Ph.D 234 Yes 02-07-2014 -- Regular
10 THANGAVEL M 32 Assistant Professor Male M.E. 131 Yes 02-01-2015 -- Regular
11 PRABAKARAN S 34 Assistant Professor Male M.E. 161 Yes 04-05-2012 -- Regular
12 MOHANAVEL G 32 Assistant Professor Male M.E. 136 Yes 27-04-2018 -- Regular
13 SRI
VENKATALAKSHMI
R
28 Assistant Professor Female M.E. 92 Yes 06-06-2018 -- Regular
14 KAPIL DEV N 35 Assistant Professor Male M.E. 152 Yes 18-07-2018 -- Regular
15 Dr GANESH BABU
P
53 Professor Male Ph.D 235 Yes 20-08-2018 -- Regular
16 RAJAGOPAL R 37 Assistant Professor Male M. Phil 147 Yes 16-07-2018 -- Regular
17 GOWRI SHANKAR
R
34 Assistant Professor Male M.E. 151 Yes 03-06-2015 -- Regular
18 ANBUMANI P 34 Assistant Professor Male M.E. 153 Yes 04-01-2016 -- Regular
19 VINOTH KUMAR S 36 Assistant Professor Male M.E. 131 Yes 09-06-2014 -- Regular
20 Dr SUMITHIRADEVI
C
42 Associate Professor Female Ph.D 252 Yes 02-01-2009 -- Regular
21 ASHOK KUMAR R 38 Assistant Professor Male M.E. 137 Yes 13-06-2016 -- Regular
22 DR RAJU P 41 Associate Professor Male Ph.D 236 Yes 19-03-2012 -- Regular
23 DHAYABARANI R 51 Associate Professor Female M.E. 305 Yes 09-06-2003 -- Regular
24 Dr PRABAKARAN T
R
59 Professor Male Ph.D 201 Yes 30-06-2017 -- Regular
25 DEVARAJ P 31 Assistant Professor Male M.E. 113 Yes 04-06-2016 -- Regular
26 VASUDEVAN M 34 Assistant Professor Male M.E. 127 Yes 07-05-2018 -- Regular
27 PRAGASH S 32 Assistant Professor Male M.E. 132 Yes 29-05-2018 -- Regular
28 Rajkumar K 31 Assistant Professor Male M.E. 92 Yes 01-07-2017 -- Regular
29 NARESHKUMAR P 37 Assistant Professor Male M.E. 190 Yes 01-07-2014 -- Regular
30 POORNIMA P 34 Assistant Professor Female M.E. 163 Yes 01-07-2014 -- Regular
31 MOHANASELVAN
C
31 Assistant Professor Male M.E. 122 Yes 01-07-2015 -- Regular
32 KIRUBASHANKAR
T S
42 Associate Professor Male M.E. 243 Yes 04-07-2013 -- Regular
33 SIVAKUMAR R 44 Associate Professor Male M.E. 273 Yes 10-04-2006 -- Regular
34 SELVARASU A 40 Assistant Professor Male M.E. 190 Yes 16-07-2012 -- Regular
35 KARTHEESWARAN
R
31 Assistant Professor Male M.E. 119 Yes 02-01-2015 -- Regular
36 ARUN B 31 Assistant Professor Male M.E. 114 Yes 08-06-2017 -- Regular
37 ARUL N 32 Assistant Professor Male M.E. 119 Yes 02-06-2015 -- Regular
38 Dr RAGHUPATHY R 35 Associate Professor Male Ph.D 154 Yes 21-02-2011 -- Regular
39 DINESH KUMAR D 31 Assistant Professor Male M.E. 130 Yes 13-08-2018 -- Regular
40 GEETHA S 28 Assistant Professor Female M.E. 92 Yes 09-08-2018 -- Regular
41 NELSON S 36 Assistant Professor Male M.E. 188 Yes 23-04-2018 -- Regular
42 KEERTHANA S 32 Assistant Professor Female M.E. 100 Yes 05-06-2018 -- Regular
43 SWARNALATHA A
P
33 Assistant Professor Female M.E. 80 Yes 06-06-2018 -- Regular
44 Dr MAHARAJAN C 31 Assistant Professor Male Ph.D 100 Yes 07-06-2018 -- Regular
45 Dr GOMATHI P S 46 Professor Female Ph.D 275 Yes 24-05-2010 -- Regular
46 MANGALARAJ K 38 Assistant Professor Male M. Phil 163 Yes 09-02-2015 -- Regular
47 BRINDHA G 35 Assistant Professor Female M. Phil 102 Yes 05-10-2017 -- Regular
48 Dr PARTHIBAN M 39 Associate Professor Male Ph.D 102 Yes 24-10-2017 -- Regular
49 KALAICHELVI K 55 Associate Professor Female M.E. 240 Yes 01-06-2009 -- Regular
50 MANIVANNAN K 38 Assistant Professor Male M.Tech 102 Yes 03-10-2017 -- Regular
51 SIVALINGAM T 34 Assistant Professor Male M.E. 125 Yes 01-07-2015 -- Regular
52 Dr NIRMALKANNAN
V
41 Dean / Principal /
Director / Vice
Chancellor
Male Ph.D 233 Yes 08-11-2017 -- Regular
53 THARANI B 35 Assistant Professor Female M.E. 104 Yes 16-08-2018 -- Regular
54 VALLI SUSEELA R 36 Assistant Professor Female M.E. 153 Yes 01-06-2018 -- Regular
55 DR RAJASEKARAN
E
57 Professor Male Ph.D 236 Yes 01-06-2018 -- Regular
56 JEGAN R R 36 Assistant Professor Male M.E. 161 Yes 15-12-2017 -- Regular
57 ARUNKUMAR R 37 Assistant Professor Male M.E. 178 Yes 11-12-2017 -- Regular
58 REVATHI M 34 Assistant Professor Female M. Phil 132 Yes 17-01-2018 -- Regular
59 ANANDHAKUMAR
P
31 Assistant Professor Male M.E. 91 Yes 04-06-2019 -- Regular
60 SUJITHA S 27 Assistant Professor Female M.E. 60 Yes 12-06-2019 -- Regular
61 JAINULAFDEEN A 35 Assistant Professor Male M.E. 175 Yes 10-06-2019 -- Regular
62 Dr SANGEETHA M 43 Professor Female Ph.D 240 Yes 12-06-2019 -- Regular
63 PREMALATHA N 34 Assistant Professor Female M.E. 133 Yes 01-06-2019 -- Regular
64 SATHYANARAYAN
AN M
33 Assistant Professor Male M.E. 128 Yes 07-06-2019 -- Regular
65 MOHAN
PERIYASAMY M
33 Assistant Professor Male M.E. 97 No 10-06-2019 14-11-2022 Regular
66 Dr SELVAKUMAR J 44 Professor Male Ph.D 235 Yes 07-08-2019 -- Regular
67 VELAVAN R 41 Assistant Professor Male M.E. 143 Yes 06-06-2019 -- Regular
68 SANTHASEELAN 36 Assistant Professor Male M.Tech 146 Yes 06-06-2019 -- Regular
69 JENISH S 49 Assistant Professor Male M.Tech 117 Yes 08-06-2019 -- Regular
70 Dr
GNANAPRAGASAM
G
41 Associate Professor Male Ph.D 154 Yes 06-12-2019 -- Regular
71 LATHA P 44 Associate Professor Female M.E. 187 Yes 15-07-2019 -- Regular
72 Maalini D 35 Assistant Professor Female M.E. 75 Yes 16-12-2019 -- Regular
73 Ramprakash A 36 Assistant Professor Male M.E. 104 Yes 18-06-2019 -- Regular
74 Karthi K 28 Assistant Professor Male M.E. 67 Yes 24-06-2019 -- Regular
75 Dr Anishkumar M 42 Professor Male Ph.D 225 Yes 22-07-2019 -- Regular
76 Subashini K 29 Assistant Professor Female M.Tech 76 Yes 13-06-2019 -- Regular
77 Suvitha S 28 Assistant Professor Female M. Phil 59 Yes 15-07-2019 -- Regular
78 SHOBANA M 35 Assistant Professor Female M.E. 78 Yes 13-12-2017 -- Regular
79 DR AROCKIA
MARRY P
44 Professor Female Ph.D 249 Yes 31-01-2020 -- Regular
80 Dr KARTHI S 42 Associate Professor Male Ph.D 157 Yes 02-11-2020 -- Regular
81 MOHANAPRAKASH
K
36 Assistant Professor Male M.E. 179 Yes 04-11-2020 -- Regular
82 PRASANTH B 31 Assistant Professor Male M.E. 96 Yes 17-08-2020 -- Regular
83 Dr SENTHIL
KUMAR B
47 Professor Male Ph.D 263 Yes 12-10-2020 -- Regular
84 CAPTAN
PRABAKARAN A
32 Assistant Professor Male M.E. 116 Yes 24-08-2020 -- Regular
85 NAVANEETHA
KRISHNAN G
36 Assistant Professor Male M.E. 99 Yes 05-11-2020 -- Regular
86 RAJASEKAR K 39 Assistant Professor Male M.E. 121 Yes 06-11-2020 -- Regular
87 MAHESH S 34 Assistant Professor Male Ph.D 144 Yes 03-11-2020 -- Regular
88 MAHALAKSHMI V 26 Assistant Professor Female M. Phil 41 Yes 04-01-2021 -- Regular
89 PRAKASH
AYYAPPAN B
44 Assistant Professor Male M.E. 131 No 30-12-2020 30-05-2022 Regular
90 Dr ARULKUMAR P 42 Professor Male Ph.D 235 Yes 29-12-2020 -- Regular
91 NANDHINIDEVI S 42 Assistant Professor Female M.E. 172 Yes 30-12-2020 -- Regular
92 Dr KARTHIGA D 32 Assistant Professor Female Ph.D 41 Yes 19-01-2021 -- Regular
93 Dr
POORNACHANDRA
N R
30 Assistant Professor Male Ph.D 41 Yes 28-12-2020 -- Regular
94 SAHEBZATHI S 39 Assistant Professor Female M.E. 88 Yes 23-01-2021 -- Regular
95 PRABAKARAN A 32 Assistant Professor Male M.E. 85 Yes 25-01-2021 -- Regular
96 PRAVEENKUMAR
G
38 Assistant Professor Male M.E. 152 Yes 25-01-2021 -- Regular
97 PURUSHOTHAMAN
S
34 Assistant Professor Male M.E. 132 Yes 29-12-2020 -- Regular
98 LAKSHMI R 33 Assistant Professor Female M.E. 70 Yes 27-01-2021 -- Regular
99 KAMALI R 26 Assistant Professor Female M.E. 41 Yes 27-01-2021 -- Regular
100 ARIVARASAN S 38 Associate Professor Male M.Tech 206 Yes 29-01-2021 -- Regular
101 INDUMATHY S 27 Assistant Professor Female M.Tech 41 Yes 29-01-2021 -- Regular
102 MERLIN FLAVIA E 26 Assistant Professor Female M. Phil 41 Yes 29-01-2021 -- Regular
103 VENKATRAMANA
G
30 Assistant Professor Male M.Tech 87 Yes 29-01-2021 -- Regular
104 VIJAYARAGHAVAN
T
32 Assistant Professor Male M.E. 99 Yes 30-01-2021 -- Regular
105 MOHAMMED
KEERAN V
33 Assistant Professor Male M.E. 116 Yes 30-01-2021 -- Regular
106 VENGAIMARBHAN
D
38 Assistant Professor Male M.E. 124 Yes 10-02-2021 -- Regular
107 JANANI S 27 Assistant Professor Female M.E. 40 Yes 10-02-2021 -- Regular
108 JAISHANKAR V 34 Assistant Professor Male M.E. 41 Yes 31-01-2021 -- Regular
109 SARANYADEVI M 31 Assistant Professor Female M.E. 69 Yes 09-02-2021 -- Regular
110 MANOJSAGAR K 33 Assistant Professor Male M.E. 125 Yes 12-03-2021 -- Regular
111 Dr AMIT KUMAR
SINGH
45 Associate Professor Male Ph.D 111 Yes 07-04-2021 -- Regular
112 KATHU SARA
RENJI
32 Assistant Professor Female M.E. 39 Yes 30-03-2021 -- Regular
113 Dr MURUGESA
THANU N
57 Professor Male Ph.D 303 Yes 24-02-2020 -- Regular
114 NANDHINI P 31 Assistant Professor Female M.E. 89 Yes 17-08-2020 -- Regular
115 Dr KATHIR I 54 Professor Male Ph.D 371 Yes 12-08-2021 -- Regular
116 ANANDAN D 38 Assistant Professor Male M.E. 161 Yes 13-08-2021 -- Regular
117 RAJESHKUMAR S 36 Assistant Professor Male M.E. 134 Yes 12-08-2021 -- Regular
118 VINAYAGAMOORT
HY A
41 Assistant Professor Male M. Phil 203 Yes 11-08-2021 -- Regular
119 SYED FAZLULLAH
S
50 Assistant Professor Male M. Phil 239 Yes 13-08-2021 -- Regular
120 BHAVADHARANI K 25 Assistant Professor Female M.E. 21 Yes 06-07-2022 -- Regular
121 PALRAJ R 53 Assistant Professor Male M.E. 192 Yes 11-05-2022 -- Regular
122 RANJITH G 29 Assistant Professor Male M.E. 45 Yes 14-03-2022 -- Regular
123 SENTHIL KUMAR K 36 Assistant Professor Male M.E. 150 Yes 14-03-2022 -- Regular
124 MOHAN S 52 Assistant Professor Male M.E. 144 Yes 07-03-2022 -- Regular
125 MANIKANDAN T 35 Assistant Professor Male M.E. 42 Yes 04-03-2022 -- Regular
126 ABIRAMI M 31 Assistant Professor Female M.E. 60 Yes 30-11-2021 -- Regular
127 DR ZULAIKHA
BEEVI S
52 Assistant Professor Female Ph.D 208 Yes 16-09-2021 -- Regular
128 SRINIVAS B 38 Assistant Professor Male M.Tech 148 Yes 15-07-2022 -- Regular
129 SARANYA R 33 Assistant Professor Female M.E. 84 Yes 15-06-2022 -- Regular
130 FRANKLIN JINO R
E
42 Assistant Professor Male M.Tech 168 Yes 15-06-2022 -- Regular
131 ARIVOLI R 38 Assistant Professor Male M.Tech 177 Yes 11-05-2022 -- Regular
132 DR KALAIARASI G 36 Associate Professor Female Ph.D 144 Yes 18-07-2022 -- Regular
133 DR ASHOK J 43 Associate Professor Male Ph.D 187 Yes 15-07-2022 -- Regular
134 DR KAVITHA K 50 Professor Female Ph.D 268 Yes 20-04-2022 -- Regular
135 MAHESHKUMAR K 36 Assistant Professor Male M.Tech 96 Yes 14-03-2022 -- Regular
136 PERIYATHAMBI P 43 Assistant Professor Male M.Tech 194 Yes 07-02-2022 -- Regular
137 DHINESH G 34 Assistant Professor Male M.E. 90 Yes 29-09-2021 -- Regular
138 SHARMELA K 31 Assistant Professor Female M.E. 36 Yes 07-03-2022 -- Regular
139 DR RAMESH BABU
R
43 Associate Professor Male Ph.D 216 Yes 15-09-2021 -- Regular
140 VIJAYA BALA N 54 Assistant Professor Female M.E. 195 Yes 26-08-2021 -- Regular
141 SUBRAMANI RAO
R
37 Assistant Professor Male M.Tech 120 Yes 18-07-2022 -- Regular
142 DR SARAVANAN G 44 Professor Male Ph.D 219 Yes 24-03-2022 -- Regular
143 RAJESH S 35 Assistant Professor Male M.E. 82 Yes 03-09-2021 -- Regular
144 LAVANYA M 33 Assistant Professor Female M.Tech 50 Yes 01-09-2021 -- Regular
145 KUMARAVEL J 35 Assistant Professor Male M.E. 120 Yes 28-07-2022 -- Regular
146 VAIRAMANI G 34 Assistant Professor Male M.E. 142 Yes 24-01-2022 -- Regular
147 DR THANGADURAI
K R
50 Professor Male Ph.D 233 Yes 24-01-2022 -- Regular
148 ASHOK JAIN X S 36 Assistant Professor Male M.E. 117 Yes 21-01-2022 -- Regular
149 DR KANAGARAJU
T
41 Assistant Professor Male Ph.D 193 Yes 19-01-2022 -- Regular
150 DR AMRISHRAJ D 33 Assistant Professor Male Ph.D 39 Yes 12-01-2022 -- Regular
151 SARAVANAN G 27 Assistant Professor Male M.Tech 13 Yes 05-01-2022 -- Regular
152 BASKAR BABU 34 Assistant Professor Male M.E. 72 Yes 08-09-2021 -- Regular
153 DR NATESH M 36 Assistant Professor Male Ph.D 129 Yes 30-08-2021 -- Regular
154 PRASATH S 30 Assistant Professor Male M.Tech 48 Yes 08-06-2022 -- Regular
155 SOWMIYA M 26 Assistant Professor Male M.E. 13 Yes 02-03-2022 -- Regular
156 MANGAIYAR
THILAGAM P
32 Assistant Professor Female M.E. 81 Yes 25-02-2022 -- Regular
157 JOSEPH VIANNY X 27 Assistant Professor Male M.E. 21 Yes 15-12-2021 -- Regular
158 DHANAYENDRAN
R
26 Assistant Professor Male M.E. 24 Yes 14-12-2021 -- Regular
159 PRASANNA
KUMAR M
27 Assistant Professor Male M.E. 13 Yes 27-08-2021 -- Regular
160 VINOTH T 33 Assistant Professor Male M. Phil 86 Yes 04-07-2022 -- Regular
161 SARAVANAN A 35 Assistant Professor Male M. Phil 72 Yes 15-07-2022 -- Regular
162 AARDRA C 25 Assistant Professor Female M.A 28 Yes 04-05-2022 -- Regular
163 RAVICHANDIRAN R 33 Assistant Professor Male M. Phil 84 Yes 04-05-2022 -- Regular
164 BALASUBRAMANIA
N K
42 Assistant Professor Male M. Phil 200 Yes 06-09-2021 -- Regular
165 SHEELA V 39 Associate Professor Female M. Phil 109 Yes 06-09-2021 -- Regular
166 DR KRISHNARAJ S 65 Professor Male Ph.D 399 Yes 27-10-2022 -- Regular
167 SURESH KUMAR V 38 Assistant Professor Male M. Phil 132 Yes 06-06-2022 -- Regular
168 DR SARAVANA
KUMAR R
47 Associate Professor Male Ph.D 226 Yes 08-12-2021 -- Regular
169 DR
THULASIDHASAN J
36 Assistant Professor Male Ph.D 72 Yes 04-10-2021 -- Regular
170 DR BHASKAR C 41 Assistant Professor Male Ph.D 180 Yes 06-09-2021 -- Regular
171 DR SYED IBRAHIM
P S
52 Assistant Professor Male Ph.D 190 Yes 06-09-2021 -- Regular
172 MEENAL ABIRAMI
R
36 Assistant Professor Female Ph.D 55 Yes 20-07-2022 -- Regular
173 MUTHU KUMAR S 26 Assistant Professor Male Ph.D 13 Yes 08-08-2022 -- Regular
174 SATHYARAJU S 35 Assistant Professor Male M. Phil 56 Yes 01-11-2022 -- Regular
175 RAMESH BABAU K
D
46 Assistant Professor Male M. Phil 114 Yes 04-05-2022 -- Regular
176 NAGAMANI S 34 Assistant Professor Male M. Phil 51 Yes 20-04-2022 -- Regular
177 VIJAYAKUMAR M 37 Assistant Professor Male M. Phil 143 Yes 20-04-2022 -- Regular
178 DR VINOTH S 30 Assistant Professor Male Ph.D 13 Yes 04-10-2021 -- Regular
179 PRABHAKARAN T 24 Assistant Professor Male M.Sc. 13 Yes 01-10-2021 -- Regular
180 SARAVANAKUMAR
R
27 Assistant Professor Male NET 13 Yes 12-10-2022 -- Regular
181 DR CHINNASAMI S 29 Assistant Professor Male Ph.D 52 No 06-06-2022 13-01-2023 Regular
182 SUGANYA M 30 Assistant Professor Male M. Phil 48 Yes 06-12-2021 -- Regular
183 KANNAN S 40 Assistant Professor Male Ph.D 70 Yes 01-12-2021 -- Regular
184 PRIYA K 33 Assistant Professor Female M. Phil 28 Yes 13-10-2021 -- Regular
185 BASKAR B 28 Assistant Professor Male M. Phil 39 Yes 04-10-2021 -- Regular
186 KAVITHA V 40 Assistant Professor Female M.E. 220 Yes 08-03-2022 -- Regular
187 JAYACANTH
SURESH G
55 Assistant Professor Male M.Tech 63 Yes 04-03-2022 -- Regular
188 KEERTHIGA K 32 Assistant Professor Female M.Tech 49 Yes 18-07-2022 -- Regular
189 DR SAM
EBENEZER R
36 Associate Professor Male M.Tech 99 Yes 04-05-2022 -- Regular
190 SRIDHIVYA M 28 Assistant Professor Male M.Tech 24 Yes 07-03-2022 -- Regular
191 SARAVANA
KUMAR A S
42 Assistant Professor Male M.Tech 132 Yes 02-03-2022 -- Regular
192 DR GOBAL SAMY 38 Associate Professor Male Ph.D 74 Yes 03-12-2021 -- Regular
193 DR SARAVANAN S 47 Associate Professor Male Ph.D 204 Yes 04-10-2021 -- Regular
194 BALASUBRAMANI 37 Assistant Professor Male M.Tech 150 Yes 26-12-2022 -- Regular
195 GOMATHI P 41 Assistant Professor Female Ph.D 186 Yes 01-12-2022 -- Regular
196 PRIYA M 35 Assistant Professor Female M.E. 96 Yes 15-09-2022 -- Regular
197 Dr K
BARANITHARAN
44 Associate Professor Male Ph.D 196 Yes 24-01-2022 -- Regular
198 Dr SHUKHASHI
ADAK
36 Assistant Professor Male Ph.D 77 No 21-02-2022 22-12-2022 Regular
199 DR RAHUL SINGH 34 Assistant Professor Male Ph.D 16 No 30-03-2022 22-12-2022 Regular
200 Sabari K 32 Assistant Professor Male M.E. 94 Yes 19-08-2021 -- Regular
201 P Anitha 35 Assistant Professor Female M. Phil 50 Yes 01-10-2021 -- Regular
202 DR
VENKATACHALAM
N
44 Associate Professor Male Ph.D 153 Yes 17-10-2022 -- Regular
203 MR KISHORE
KUMAR R
27 Assistant Professor Male NET 6 Yes 07-02-2023 -- Regular
204 DR VENKATESH
KUMAR N
47 Assistant Professor Male Ph.D 209 Yes 03-05-2023 -- Regular
205 MR GOBINATH P 30 Assistant Professor Male M.Sc. 5 Yes 09-02-2023 -- Regular
206 MR KARTHIKEYAN
M
48 Assistant Professor Male M.E. 6 Yes 14-09-2022 -- Regular
207 MR ANBU A 28 Assistant Professor Male M.Tech 34 Yes 27-01-2023 -- Regular
208 DR BITANG
KWRUNG TRIPURA
35 Assistant Professor Male Ph.D 11 Yes 03-01-2023 -- Regular
209 DR TRIPTI DE T 35 Assistant Professor Female Ph.D 42 Yes 30-12-2022 -- Regular
210 MRS NITHYA C 34 Assistant Professor Female M.E. 13 Yes 01-09-2021 -- Regular
211 MRS AAYISHA
BEGAM A
27 Assistant Professor Female M.E. 7 Yes 03-05-2023 -- Regular
212 DR SURENDAR P 35 Assistant Professor Male Ph.D 111 Yes 17-10-2022 -- Regular
213 MR ELUMALAI V 36 Assistant Professor Male M.E. 157 Yes 07-04-2023 -- Regular
214 DR
VIGNESHWARAN S
32 Assistant Professor Male Ph.D 74 Yes 28-12-2022 -- Regular
215 MRS PARVATHI M 47 Assistant Professor Female M.E. 151 Yes 12-10-2022 -- Regular
216 MR KUMARESAN V 29 Assistant Professor Male M.E. 11 Yes 30-12-2022 -- Regular
217 MRS NITHIYA K 34 Assistant Professor Female M.E. 8 Yes 03-04-2023 -- Regular
218 MR VIJAYAKUMAR
G
30 Assistant Professor Male M.E. 8 Yes 03-04-2023 -- Regular
219 MS RAAJYA
VARDHINI A
36 Assistant Professor Female M.Tech 11 Yes 30-12-2022 -- Regular
220 MS KAVITHA R P 39 Assistant Professor Female M.Sc. 151 Yes 10-02-2023 -- Regular
221 MR NAGARAJ D 42 Assistant Professor Male M. Phil 102 Yes 28-04-2023 -- Regular
222 DR KEERTHANA R 27 Assistant Professor Female M.Sc. 10 Yes 20-02-2023 -- Regular
223 MR BOOMI P 39 Assistant Professor Male M. Phil 187 Yes 01-02-2023 -- Regular
224 MS JEYASHREE G 26 Assistant Professor Female M.Tech 4 Yes 30-08-2022 -- Regular
225 DR VIJAYAPRITHA
S
31 Assistant Professor Female M.Sc. 3 Yes 03-04-2023 -- Regular
226 DR MANIKANDAN
R
55 Professor Male Ph.D 294 Yes 30-08-2022 -- Regular
227 MRS KAVIYA G 29 Assistant Professor Female M.E. 38 Yes 03-03-2023 -- Regular
228 MR BASKAR D 51 Assistant Professor Male M.Tech 263 Yes 03-03-2023 -- Regular
229 MRS DEEPIKA D 36 Assistant Professor Female M.E. 11 Yes 28-12-2022 -- Regular
230 MR KUMARAN R 44 Associate Professor Male M.E. 233 Yes 06-06-2019 -- Regular
231 MRS JAYAKUMARI
DS
42 Assistant Professor Female M.E. 208 Yes 01-04-2023 -- Regular
232 MS SRIDEVI R 24 Assistant Professor Female M.Sc. 10 Yes 08-02-2023 -- Regular
233 MR KARTHIKEYAN
J
31 Assistant Professor Male M.E. 13 Yes 30-12-2021 -- Regular
234 MS NITHYA A 34 Assistant Professor Female M.Tech 42 Yes 27-04-2022 -- Regular
235 MRS MENAGA G 33 Assistant Professor Female M.E. 37 Yes 05-03-2022 -- Regular
236 MRS KAVYA PRIYA
J
31 Assistant Professor Female M.E. 13 Yes 04-03-2022 -- Regular
237 MR MANI P 32 Assistant Professor Male M.E. 11 Yes 12-04-2022 -- Regular
238 MRS REVATHI R 40 Assistant Professor Female M.E. 113 Yes 01-04-2023 -- Regular
239 MRS AJITHA KALA
DS
47 Assistant Professor Female M.Tech 8 Yes 31-03-2023 -- Regular
240 MRS RAJESWARI S 32 Assistant Professor Female M.Tech 20 Yes 04-03-2022 -- Regular
241 MR KALIMUTHAN C 45 Assistant Professor Male M.Tech 108 Yes 01-03-2023 -- Regular
242 MR
BHARATHIRAJA M
32 Assistant Professor Male M.E. 26 Yes 01-03-2023 -- Regular
243 MR SUBRAMANI S 34 Assistant Professor Male M.E. 11 Yes 29-12-2022 -- Regular
244 MRS
NAGALAKSHMI N
41 Assistant Professor Female M.E. 20 Yes 03-10-2022 -- Regular
245 MR ASHWIN
KUMAR KG
30 Assistant Professor Male M.E. 68 Yes 06-07-2017 -- Regular
246 MR ILANGOVAN B 35 Assistant Professor Male M.E. 112 Yes 01-03-2023 -- Regular
247 MR ELAYARAJA
MP
36 Assistant Professor Male M. Phil 25 Yes 03-04-2023 -- Regular
248 MRS LAKSHMI B 34 Assistant Professor Female M. Phil 47 Yes 04-05-2023 -- Regular
249 MRS REBECCAL
ANGEL M
31 Assistant Professor Female M.E. 8 Yes 31-03-2023 -- Regular
250 MRS
GEETHANJALI G
29 Assistant Professor Female NET 30 Yes 22-02-2023 -- Regular
251 MS SATHYA K 32 Assistant Professor Female M. Phil 19 Yes 06-04-2023 -- Regular
252 MRS EYALARASI V 31 Assistant Professor Female M.Tech 10 Yes 01-02-2023 -- Regular
253 MS
RENGALAKSHMI P
25 Assistant Professor Female M.Tech 11 Yes 31-08-2022 -- Regular
254 MRS
MOHANAPRIYA M
37 Assistant Professor Female MCA 165 Yes 05-05-2014 -- Regular
255 MAHALAKSHMI V 27 Assistant Professor Female M.E. 28 Yes 19-07-2021 -- Regular
256 PRIYADHARSHINI
S
27 Assistant Professor Female M.E. 13 Yes 08-12-2021 -- Regular
257 GOWSALYA S 26 Assistant Professor Female M.Tech 19 Yes 30-12-2021 -- Regular
258 GOWRI T 25 Assistant Professor Female M.Tech 19 Yes 30-12-2021 -- Regular

This is the post peer-review accepted manuscript of:
Amir Hossein Ashouri, Andrea Bignoli, Gianluca Palermo, Cristina Silvano, Sameer
Kulkarni, John Cavazos
MiCOMP: Mitigating the Compiler Phase-Ordering Problem Using Optimization
Sub-Sequences and Machine Learning
ACM Transactions on Architecture and Code Optimization, TACO 2015
The published version is available online at: https://doi.org/10.1145/3124452
©2018 ACM. (Association for Computing Machinery, Inc.). Permission to make digital or hard
copies of part or all of this work for personal or classroom use is granted without fee provided that
copies are not made or distributed for pro/f_it or commercial advantage and that copies bear this
notice and the full citation on the /f_irst page. Copyrights for components of this work owned by
others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to
republish, to post on servers, or to redistribute to lists, requires prior speci/f_ic permission and/or a
fee. Request permissions from Publications Dept., ACM, Inc., fax +1 (212) 869-0481, or
permissions@acm.org. "
Personal use of this material is permitted. Permission from the editor must be obtained for all other
uses, in any current or future media, including reprinting/republishing this material for advertising
or promotional purposes, creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.
brought to you by COREView metadata, citation and similar papers at core.ac.uk
provided by Archivio istituzionale della ricerca - Politecnico di Milano
1
MiCOMP: Mitigating the Compiler Phase-ordering Problem using
Optimization Sub-sequences and Machine Learning
AMIR HOSSEIN ASHOURI, Politecnico di Milano
ANDREA BIGNOLI, Politecnico di Milano
GIANLUCA PALERMO, Politecnico di Milano
CRISTINA SILVANO, Politecnico di Milano
SAMEER KULKARNI, University of Delaware
JOHN CAVAZOS,University of Delaware
Recent compilers oﬀer a vast number of multilayered optimizations, capable of targeting diﬀerent code
segments of an application. Choosing among these optimizations can signi/f_icantly impact the performance
of the code being optimized. The selection of the right set of compiler optimizations for a particular code
segment is a very hard problem, but /f_inding the best ordering of these optimizations adds further complexity.
. The traditional approach of constructing compiler heuristics to solve this problem simply can not cope
with the enormous complexity of choosing the right ordering of optimizations for every code segment in an
application.
The predictive model uses (i) a platform-independent dynamic features, (ii) an encoded version of the
compiler sequence and (iii) an exploration heuristic to tackle the problem.
Experimental results using the LLVM compiler framework and the Cbench suite show the eﬀectiveness of
the clustering and encoding techniques to application-based reordering of passes while using a number of
predictive models. We perform statistical analysis on the prediction space and compare against (i) standard
optimization levels O2 and O3, (ii) random iterative compilation, and (iii) two recent non-iterative approaches.
We demonstrate that our proposed methodology outperforms the performance of -O1, -O2, and -O3 optimiza-
tion levels in just a few iterations, reaching an average performance speedup of 1.31 (up to 1.51) on the Cbench
benchmark suite.
Additional Key Words and Phrases: compilers, machine learning, autotuning, recommender systems
ACM Reference format:
Amir Hossein Ashouri, Andrea Bignoli, Gianluca Palermo, Cristina Silvano, Sameer Kulkarni, and John cavazos.
2017. MiCOMP: Mitigating the Compiler Phase-ordering Problem using Optimization Sub-sequences and
Machine Learning. ACM Transactions on Architecture and Code Optimization , , Article 1 (September 2017),
22 pages.
DOI: 10.1145/3124452
1 INTRODUCTION
Compiler developers typically design optimization passes in order to transform each code segment
of a program to produce an optimized version of an application. The optimizations can be applied
at diﬀerent stages of the compilation process. Optimizing source code by hand is a tedious task
and therefore compiler optimizations are provided to automatically transform code. However,
these code optimizations are programming language, application, and architecture dependent.
Additionally, the word optimization is a misnomer and there is no guarantee the transformed code
will perform better than the original version. Understanding the behavior of the optimizations
and the actual eﬀect on the source-code and the interaction of the optimizations with each other
This work is partially supported by the European Commission Call H2020-FET-HPC program under the grant ANTAREX-
671623.
© 2017 Copyright held by the owner/author(s). Publication rights licensed to ACM. This is the author’s version of the
work. It is posted here for your personal use. Not for redistribution. The de/f_initive Version of Record was published inACM
Transactions on Architecture and Code Optimization , http://dx.doi.org/10.1145/3124452.
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
1:2 A. H. Ashouri et al.
are complex modeling problems. The problem is particularly diﬃcult because compiler developers
have to deal with hundreds of diﬀerent optimizations that can be applied during the diﬀerent
compilation phases and this creates the phase-ordering problem. The phase-ordering problem has
been an open-problem in the /f_ield of compiler research for many decades [25, 42]. The inability of
researchers to solve the phase-ordering problem has led to advances in the more simple problem of
selecting the right set of optimizations, but even this problem has yet to be solved [9, 12].
This process of selecting the right optimizations for each code segment is typically done manually,
and the sequence of optimizations is constructed with little insight into the interaction between the
preceding compiler optimizations in the sequence. The task of manually constructing heuristics to
select the right sequence of compiler optimizations is infeasible given the ever growing number of
compiler optimizations being integrated into compiler frameworks. As an example, GCC has more
than 200 compiler passes, and LLVM-clang and LLVM-opt each have more than 100 transformations.
Additionally, these optimizations are applied at very diﬀerent phases of the compilation, including
analysis passes and loop-nest passes. Most optimization /f_lags are turned oﬀ by default, and compiler
developers rely on software developers to know which optimizations will bene/f_it their code.
Compiler developers provide standard optimization levels, e.g. -O1, -O2, -Os, etc. to introduce a
/f_ixed-sequence of compiler optimizations that, on average, bring good performance on a set of
benchmarks the compiler developers tested. Finding the best ordering of compiler optimizations
can have substantial bene/f_its for performance metrics such as execution time, power consumption,
and code-size. To this end, using prede/f_ined optimizations usually is not good enough to bring
the best achievable application-speci/f_ic performance. In this paper, we propose a framework in
order to mitigate the complexity of the phase-ordering problem. So far, there are two potential
techniques we could use to predict good optimization orders for code being optimized:
•(i) Intermediate Sequence Prediction : This technique uses a model to predict the current
best optimization (from a given set of optimizations) that should be applied based on the
characteristics of code in its present state. [5, 24].
•(ii) Complete Sequence Prediction : This technique uses a model to predict the complete se-
quence of optimizations that needs to be applied to the code just by looking at characteristics
of the code in its original state [10, 34–36].
The framework proposed in this paper, MiCOMP, falls under the second category. It uses pre-
dictive models on complete optimization sequences rather than individual optimizations. We
characterize applications as a vector of dynamic features that are independent from the target
architecture. Predicting the complete optimization sequence to apply to a piece of code, i.e. complete
sequence prediction, has the bene/f_it of only requiring a single-round of feature collection of the code
before any optimizations are applied to it. In order to use classic machine learning algorithms with
the phase-ordering problem, we adapt an encoding scheme to transform variable-length vectors
of optimizations into /f_ixed-length vectors. Our prediction models are trained oﬄine and program
features and diﬀerent compiler con/f_igurations are fed as inputs. As outputs, a prediction model
produces a speedup number without the need to actually run the code on the target architecture.
The dynamic characterization is independent from the architecture the code is running; thus, it can
bring portability among diﬀerent architectures. Additionally, we de/f_ine exploration heuristics to
/f_ind the best models in the shortest time. Our metric of time is de/f_ined as the minimum number of
predictions from the model to obtain the best version of the code being optimized. The heuristic is
based on Adjusted Cosine Similarity [38] to correlate diﬀerent con/f_igurations of optimizations with
their corresponding predicted speedups across all the training data. A recommendation algorithm
enables us to explore only a fraction of the con/f_iguration space to reach the best speedups rather
than a state-of-the-art sorting/ranking [10, 34–36]. In our experimental results, we show that our
technique can outperform LLVM’s highest optimization level of -O3 by just a few predictions
(up to 3 in the worst observed case). . We selected the full set of applications from the Ctuning
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
MiCOMP: Mitigating the Compiler Phase-ordering Problem using Opt. Sub-sequences 1:3
Cbench benchmark [14] to assess and evaluate the bene/f_its of the proposed approach and to prove
its feasibility. The main contributions of the proposed approach are as follows:
•An independent predictive-modeling framework, capable of capturing the correlation
between diﬀerent compiler optimizations and their predicted speedup without having to
run optimized code variants on the target platform. Our autotuning framework can be
paired with any desired predictive models.
•. We have clustered diﬀerent compiler optimizations, all taken from LLVM’s O3 into 5
diﬀerent groups. The order of optimizations within a group is internally /f_ixed but the
ordering of the groups can be altered. In this work, these groups are called sub-sequences
and we exploit the phase-ordering by using these sub-sequences rather than the individual
optimizations. By starting from no optimizations (as the baseline) and exploring diﬀer-
ent orderings of the sub-sequences using the same optimizations available to -O3, we
outperformed -O3.
•Adapting a simple mapping technique to encode an optimization sequence into a bit string.
The proposed technique transforms a variable-length representation to a /f_ixed-length
feature vector representation. It allows us to apply traditional machine learning algorithms
since they are mostly designed to cope with /f_ixed-length feature vectors.
•Adapting a Recommender System (RS) approach on the prediction space to use dynamic
information.
The rest of the paper organized as follows: Section 2 presents related work. Section 3 introduces
our proposed methodology including all its components. In Section 4, we present our experimental
results and evaluate the results by means of several comparisons in the Section 5. We conclude the
paper with future work and the conclusion.
2 RELATED WORK
Literature on the phase-ordering problem is closely related to the problem of selecting the best
set of compiler optimizations in a /f_ixed ordering. Recent literature can be classi/f_ied into two main
classes: (i) autotuning and iterative compilation approaches and (ii) applying machine learning to
the problem of optimization selection.
Autotuning addresses automatic code-generation and optimization by using diﬀerent scenarios
and architectures. It involves building techniques for automatic optimization of diﬀerent parame-
ters in order to maximize or minimize the satisfaction of an objective function. One strategy in
autotuning consists of coupling the approach with random generation of code-variants at each run.
This technique can generally improve application performance in reference to static-handcrafted
compiler optimization sequences [1]. Given the complexity of the iterative compilation problem
[9], it has been shown that applying compiler optimization sequences at random can be as good
as using other algorithms such as Genetic algorithms or Simulated Annealing to choose which
optimizations to apply [1, 10, 12]. Other authors [3, 8] explored compiler Design Space Exploration
(DSE) techniques jointly with architectural DSE for VLIW architectures.
Applying machine learning to the problem of selecting the best compiler optimizations has
been extensively investigated by many researchers in the past. Proposed methodologies [11, 13, 22,
28, 29, 41] were among the /f_irst notable works introducing the use of machine learning to solve
compilation problems. Recent related work [4, 6, 7, 34, 35] also tackled the problem of selecting
the best compiler optimizations to apply by utilizing Bayesian Networks with an application-
independent characterization technique, predictive modeling with dynamic characterization, and
predictive modeling with compiler representations (Intermediate Representation (IR)) . There
have been diﬀerent objective functions used with machine learning on the problem: i) A speedup
predictor takes as input both the characterization of the program being compiled and an optimization
sequence, and it predicts as output the speedup when applying that optimization sequence relative
to a default optimization setting. [10, 34, 35] ii) A sequence predictor characterizes a program being
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
1:4 A. H. Ashouri et al.
compiled and uses it as input to a model, and the model predicts a probability distribution of
optimizations to apply to that program. [1, 4, 6, 37]. iii) A tournament predictor [36]) takes as input
a triple corresponding to the characterization of the program and two optimization sequences. This
model predicts whether the speedup after applying the the /f_irst optimization sequence will be more
or less than speedup if applying the second optimization sequence.
. However, there are a few notable published studies that attempted to solve the problem.
Kulkarni and Cavazos [24] have applied Neuro-Evolution for Augmenting Topologies (NEAT) in
the Java JikesRVM compiler to phase-ordering by using intermediate sequence prediction. They
built prediction models that use as input features of the current state of the transformed source-
code and de/f_ine certain stop-condition rules to complete the /f_inal predicted sequence at each
iteration. They used source-code features and the Java JikesRVM JIT compiler to experimentally
evaluate their approach. In contrast, we tackle the problem using predictive modeling and dynamic
independent-characterization of the applications and our proposed methodology enables us to
predict the full-sequence in one-shot.
Matrins et al. [27] tackled the problem of phase-ordering by a DSE approach that uses a clustering-
based selection method for grouping functions with similarities and exploration of a reduced search
space resulting from the combination of optimizations previously suggested for the functions
in each group. Authors used DNA encoding where program elements (e.g., operators and loops
in function granularity) are encoded in a sequence of symbols, and followed by calculating the
distance matrix and a tree construction of the optimization set. Consequently, they applied the
compiler optimization passes already included in the DSE to measure the reduction in the total
exploration time of the search space such as Genetic algorithm. Our proposed approach on the
other hand is mainly diﬀerent, as we mitigate the phase-ordering problem by inducing a prediction
model rather than a design space exploration scheme. Once our model is trained, it can be further
used for any number of applications under analysis to induce a prediction inexpensively and we
believe it will bring scalability in autotuning compilers.
Other related work has approached the problem by exhaustively exploring the optimization
ordering space at the granularity of functions [ 23]. The exhaustive enumeration these authors
proposed, constructs probabilities of enabling/disabling interactions between diﬀerent optimization
sequences, but these probabilities are not speci/f_ic to any program. Jantz et al. [19] proposed two
pruning techniques to downsample the optimization space. As a result the authors could employ
faster exhaustive phase-ordering searches on the new space. Ashouri et al. introduced an approach
that uses predictive modeling to construct an intermediate sequence of optimizations for code being
compiled [5]. Other related work used iterative Design Space Exploration (DSE) and clustering-
based approaches to down-sample and cluster the available optimizations targeting performance
gain and power reduction [30–32].
Our approach, MiCOMP, is signi/f_icantly diﬀerent compared with those mentioned in the literature.
Our work mostly resembles the approach of Park et al. [35, 36]. However, our techniques tackle
the signi/f_icantly harder problem of the phase-ordering. We introduce a mapping function that
encodes an optimization sequence into a bit string. It preserves the ordering and the repetition of
the optimizations. At the same time, the proposed work is able to predict the complete optimization
sequence to apply to the unoptimized code, rather than predicting the best optimization to apply to
the current state of the optimized code [5, 24]. . We use dynamic architecture independent features
to feed into our model. Moreover, we used clustering over all passes in LLVM’s-O3, that tended to
perform well, to signi/f_icantly outperform the single optimization sequence performed by-O3 itself.
We do that by re-ordering these sub-sequences automatically based on the type of the application
under optimization. To summarize, the presented work is the /f_irst approach that uses machine-
learning based techniques on the phase-ordering problem to predict the complete sequence of
optimizations. In Section 4, we improve the machine-learning model through Recommender System
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
MiCOMP: Mitigating the Compiler Phase-ordering Problem using Opt. Sub-sequences 1:5
techniques and assess the experimental results we obtain against the state-of-the-art phase-ordering
approaches.
3 THE PROPOSED METHODOLOGY
Compilers typically ship with standard optimization levels (e.g. -O2, -O3 and -Ofast) each
tuned during compiler development to obtain a certain level of performance on a standard set
of benchmarks. These optimization levels do not always translate to good performance on other
applications. The main objective of the proposed methodology is to introduce a compiler autotuning
framework, which is able to dynamically reorder the compiler passes within LLVM’s optimization
level -O3 , to achieve the maximum speedup for the applications being optimized. We found that if
we could reorder sub-sequences of optimizations that tended to perform well, we could signi/f_icantly
outperform the single optimization sequence performed by-O3. This process should be customized
based on the features of the application under analysis. To mitigate the phase-ordering problem, a
model has to be constructed in such a way that it can correlate the eﬀect of using diﬀerent compiler
sequences and the corresponding achievable speedup. MiCOMP uses such a model, and it can (i)
recommend good sequences of optimizations that maximize an application’s performance (ii) with
very few predictions.
Phase-ordering is also complicated by allowing the possibility of variable-length compiler se-
quences. State-of-the-art approaches for selecting the right set of optimizations used /f_ixed length
feature-vectors [6, 10, 35, 36] to induce a prediction model. During the prediction phase, MiCOMP
proposes an iterative process in which diﬀerent solutions are explored by evaluating diﬀerent
optimization sequences with the potential of leading to higher speedups. We predict optimization
sequences that will perform well against using state-of-the-art ranking [35, 36] techniques.
Figure 1 illustrates the two main phases of MiCOMP: (i) oﬄine training and (ii) online prediction .
The oﬄine training phase is used to learn about the eﬀects of compiler optimizations when
compiling an application. In particular, this phase is used to induce a prediction model considering
application features and applied optimizations (including order and repetitions). This phase is
performed once for each compiler and the model is built on a set of representative applications.
In this phase, each application is passed through a single round of feature collection to extract
an application’s characteristics. A dynamic pro/f_iler is used to generate a representation of the
program in terms of its features. Since a very large set of features is extracted for each application,
we apply a dimension-reduction technique to reduce the number of features that is fed as input
to the prediction model (e.g. PCA âĂŞ Principal Component Analysis [ 21]). This speeds up the
learning during the model construction process. Application-pro/f_iling and dimension-reduction
techniques are extensively described in Section 3.1. Next, an application is compiled with diﬀerent
con/f_igurations of compiler optimizations, executed and pro/f_iled in terms of speedup with respect
to LLVM ’s-O3. The speedup values together with the reduced program features and an encoded
version of the used compiler optimizations (characterized by a /f_ixed-length binary output, see
Section 3.3) are fed to a machine learning algorithm to induce the speedup predictor (see Section
3.4). This model can then used during the online phase.
The online prediction phase , is used every time a new application is optimized. We use the same
feature extraction and dimension reduction techniques described in the oﬄine training phase. The
collected features are used to query the speedup prediction model to predict the best set of compiler
sequences to apply to an application. The goal of our method is to discover the fewest number of
predictions that will be needed to obtain the optimization sequence that gives the best speedup
possible. Thus, MiCOMP has been coupled with a heuristic derived from the /f_ield of Recommender
Systems (see Section 3.5). This technique is used to obtain a predicted set of optimization sequences
where each sequence is as diverse as possible to the other sequences in the set, thus guaranteeing
coverage of a large part of the optimization con/f_iguration space, consequently obtaining a set of
optimization sequences that are robust to model inaccuracies.
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
1:6 A. H. Ashouri et al.
Applica'on	
Characteriza
'on	
Applica'on	
Characteriza'on	
Feature	
Vector	
Compiler	
Op'miza'on	
sub-sequences	
Phase-ordering	
Design	Space	
Explora'on	
LLVM	
Backend	
Training	
applica'on	
set	
 Mapper	
Trained		
Speedup	
Model	
Predic've	
Modeling	
Speedup	
w.r.t	–O3	
Applica'on	
Characteriza'on	
Dimension	
Reduc'on	
Feature	
Vector	
Best	
Compiler	
Sequences	
Target	
Applica'on	
¤	
Trained		
Speedup	
Model	
II)	Predic'on	Phase	(online)	
I)	Training	Phase	(oﬄine)	
Dimension	
Reduc'on	
Recommender	
System	
Predicted	
Speedup	
Fig. 1. Proposed framework. (i) oﬀline-training phase which is done once and (ii) online-prediction phase for
optimizing new unseen applications
3.1 Application Characterization
In this work, we used a PIN-based [26] dynamic instrumentation framework to analyze and charac-
terize the behavior of applications at execution-time. In particular, our framework provides a high
level Micro-architectural Independent Characterization of Applications (MICA) [17] suitable for
characterizing applications in a target architecture agnostic manner. There is no static syntactic
analysis, but the framework is solely based on dynamic MICA pro/f_iling. The MICA framework
reports information about instruction types, memory and register access pattern, potential in-
struction level parallelism and a dynamic control /f_low analysis in terms of branch predictability.
Overall, the MICA framework characterizes an application in reference to 99 diﬀerent metrics
(or features). Many of these 99 features are strongly correlated (e.g. the number of memory reads
with stride smaller than 1K is bounded by the number of reads with stride smaller than 2K). To
signi/f_icantly improve the speed of model construction, we applied a dimension reduction by using
Principal Component Analysis (PCA) [21] to reduce the number of features used to characterize an
application. PCA is a technique to transform a set of correlated features into a set of orthogonal, i.e.,
uncorrelated principal components. The PCA transformation sorts the principal components by
descending order based on their variance [20]. For instance, the /f_irst principal component includes
the most input data variability, i.e., this component represents most of the information contained
in the input data. To reduce the number of input features, while keeping most of the information
contained in the input data, one simply needs to use the /f_irstk principal components as suggested
in previous work [17]. In particular, we set k = 5, which captures more than 98% of the overall
variance across all training data. .
3.2 Constructing Compiler Sub-sequences
In this section, we brie/f_ly explain our novel idea behind clustering certain compiler optimizations as
sub-sequences. A phase-ordering optimization sequence represented by the vectoro belongs to the n
dimensional factorial space|Ωphases |= n!, wheren represents the number of compiler optimizations
under study. However, the mentioned bound is for a simpli/f_ied phase-ordering problem having a
/f_ixed length optimization sequence length and no repetitive application of optimizations. Allowing
optimizations to be repeatedly applied and a variable length sequence of optimizations will expand
the problem space to:
|Ωphases _repetition |=
mÕ
i=0
ni (1)
Where n is the number of optimizations under study and m is the maximum desired length
for the optimization sequence. Even for reasonable values for n and m, the entire search space is
enormous. For example, assumingn and m are both equal to 10, this leads to an optimization search
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
MiCOMP: Mitigating the Compiler Phase-ordering Problem using Opt. Sub-sequences 1:7
  aa
  adce
  alignment-from-assumptions
  argpromotion
  assumption-cache-tracker
  barrier
  basicaa
  basiccg
  bdce
  block-freq
  branch-prob  constmerge  correlated-propagation  deadargelim  demanded-bits  domtree
dse  
early-cse  
elim-avail-extern  
float2int  forceattrs  functionattrs  
globaldce  
globalopt  
globals-aa  
gvn  indvars  
inferattrs  
inline  
instcombine  
ipsccp  
jump-threading  
lazy-value-info  
lcssa  
licm  
loop-accesses  
loop-deletion  
loop-idiom  
loop-rotate  
loop-simplify  loop-unroll  
loop-unswitch  
loop-vectorize  
loops  
mem2reg  
memcpyopt  
memdep  
mldst-motion  
  prune-eh  reassociate  rpo-functionattrs  scalar-evolution  sccp  scoped-noalias
  simplifycfg
  slp-vectorizer
  sroa
  strip-dead-prototypes
  tailcallelim
  targetlibinfo
  tbaa
  tti
  verify
Fig. 2. Generated directed graph for LLVM’s -O3. Each node in the graph represents an optimization pass.
The edge thickness depicts the strengths in the connection between two nodes.
space of more than 11 billion diﬀerent optimization sequences to select from for each piece of code
being optimized [5] 1.
3.2.1 The Optimization Dependence Graph.Mitigating the phase-ordering problem with previous
approaches is not practical due to the large number of diﬀerent possible optimization sequences
to select for each piece of code being optimized. MiCOMP proposes to group optimizations into
clusters of sub-sequences that are known to perform well, . There are 157 compiler passes in LLVM
optimization level -O3 (more than 60 unique compiler passes) and selecting the most promising sub-
sequences from these optimizations can positively aﬀect the autotuning process. Among all these
157 compiler passes, some are analysis passes (i.e. basicaa, memdep, etc) which do not transform the
code directly, but instead provide analysis information to other compiler passes that follow them.
The rest are transformation passes, i.e., Aggressive Dead Code Elimination (adce), Loop Invariant
Code Motion (licm), loop-rotate, etc., which perform optimizations on the code 2.
In this paper, we introduce the idea of clustering sub-sequences of all the passes available to the
optimization level -O3 and adapt prediction models to order these sub-sequences in ways that
improve the performance of a particular application. We show that this technique can improve
the performance of an application over using -O3 by evaluating a few predicted orderings of the
sub-sequences of optimizations.
This graph can be represented by a Weighted Adjacency Matrix that has the size of N×N. This
matrix can be used for clustering optimizations into diﬀerent sub-sequences. Figure 2 shows the
constructed graph on -O3.
3.2.2 Graph and Sub-sequence Clustering.For our clustering of optimizations into sub-sequences,
we could have used any number of the numerous clustering methods proposed in the literature
related to Pattern Recognition (e.g. iterative, hierarchical, divisive, etc,) [39]. We selected agglom-
erative clustering [43] which is an iterative clustering technique that merges smaller clusters and
improves the complexity of k-mean clustering on graphs [39]. A key insight of this method is that
1The problem of phase-ordering does not have deterministic upper-bound in the case of unbounded length.
2http://llvm.org/docs/Passes.html
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
1:8 A. H. Ashouri et al.
it treats clusters as a dynamical system and its samples as states. The algorithm works as follows:
Agglomerative clustering, receives as input the matrix of the graph G and the number of desired
clusters (nT ) and builds (i) the graph G with k-nearest-neighbors upon computing its Weighted
Adjacency Matrix (W ). (ii) The algorithm then calculates the transition probabilities and (iii) forms
sample clusters C = {c1, ...,cnc }. (iv) It enters a loop to iteratively try to add more sub-clusters to
the already available clusters in C as long as the conditional sum of the all-path integrals within
the new sub-clusters maximizes some objective function (argmax) [43]. A path integral is a metric
to measure the stability of a dynamical system and is computed by summing the paths within
the cluster on the directed graph weighted by transition probabilities. We used the algorithm and
tentatively increased the number of max desired clusters until no clusters could be added. The /f_inal
/f_ive clusters, namely, the best optimization sub-sequences the algorithm could /f_ind are reported in
Section 4 Table 4.
3.2.3 Benefits of Sub-sequences. Clustering optimizations into sub-sequences makes sense.
Certain analysis algorithms typically should be done before an optimization in order for the
optimization to have any signi/f_icant impact. For example, we may want to run analysis that performs
basic block counts and predicts branch instruction outcomes before applying an optimization
that reorders the code blocks in an application. Additionally, it is likely that -O3 will contain
optimizations that should follow other optimizations in order to obtain the best performance. Thus,
forming a cluster of optimizations that should be applied together makes a lot of sense.
3.3 The Proposed Encoder
Constructing prediction models for the problem of selecting the right compiler optimizations with
/f_ixed-length feature vectors has been extensively studied [10, 34–36]. However, prediction models
fall short when correlating program characterizations with the right compiler optimizations to
apply when it comes to a variable optimization sequence length [2]. Therefore, we adapt a simple
mapping technique to encode an optimization sequence into a bit string. The proposed technique
transforms a variable-length representation to a /f_ixed-length feature vector representation.
Let A= {α1, . . . ,αN }be the set of all variables, which can be thought of as an alphabet. Every
αi is a letter. A /f_inite string of not necessarily distinct letters is called aword. Thus, each word is
a concatenation of the form αi1 αi2 ··· αik , where i1, i2, . . . ,ik ∈{1, . . . ,M}. The integer k is the
length of the word. We will also allow the empty word which by de/f_inition has length zero.
There is a simple way of encoding the space Wof all words of length at most M using the
space described by {0, 1}N ×M consisting of all binary strings of the /f_ixed lengthN ×M. To see this,
consider the mapping function f : A→{ 0, 1}N which encodes each letter αi to the binary string
f (αi )= b1 ···bM , where
bj =
{
1 if j = i
0 if j , i.
Now we de/f_ine the mapping functionF : W→{ 0, 1}N ×M by encoding each wordαi1 αi2 ··· αik to the binary
string
F(αi1 αi2 ··· αik )= f (αi1 )f (αi2 )··· f (αik ) 0 ··· 0
N −k times
,
where 0 = 0 ··· 0 is the zero string of length N . Evidently the map F is one-to-one. The image F(W)is much
smaller than the target space {0, 1}N ×M , as these sets have ÍM
k=0 Nk and 2N ×M elements, respectively.
If we identify each element of{0, 1}N ×M with a concatenations1 ···sN of N elements of {0, 1}N , the image
F(W)can be simply characterized by the following two requirements:
1. Each si has at most one non-zero binary digit.
2. If si = 0 and sj , 0, then i > j.
Given the proposed encoding, there exists a one-to-one (1:1) mapping, F, for every instance of A =
{α1, . . . ,αN }with the binary size of N ×M that has the same characteristics of the original presentation
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
MiCOMP: Mitigating the Compiler Phase-ordering Problem using Opt. Sub-sequences 1:9
ADDEBC	
Α	
B	
10000	 00010	 00010	 00001	 01000	
 00100	
Fig. 3. An example of the proposed encoding scheme on the example where we have repetitions and {N =
5, M = 6}: Each le/t_ter represents a compiler sub-sequences containing diﬀerent compiler optimizations.
with the bene/f_it of having a /f_ixedN ×M length. An example of the proposed encoding scheme is shown
on the Figure 3. Our adapted mapping function uses a one-hot encoding approach [33] for N=5 and M=6
to assign a single high (1) for each subsequences while other bits are turned oﬀ (0). This technique can
inexpensively preserve the order and the repetitions of optimizations in a sequence, at the same time it assures
the transformed feature vector has /f_ixed-length size.
3.4 Predictive Modeling
To this end, the proposed methodology in Figure 1 illustrates the use of predictive modeling in both the oﬄine
(training) and online (testing) phases of the process. We used the predictive modeling in the oﬄine training
phase to (i) construct the model and in (ii) the online prediction phase we exploit the constructed model on
the target application to predict the speedup of a complete optimization sequence without the need to actually
apply the sequence of optimizations to the code.
3.4.1 Constructing the Prediction Model. Predictive modeling is the process of constructing, testing, and
validating a model to predict an unobserved outcome based on characterization of a state from which to
predict the outcome. In this paper, the state being characterized is the code being optimized, and the predicted
outcome corresponds to the speedup metric calculated by normalizing the execution time of the current
optimization sequence by the execution time of the baseline optimization sequence. The general formulation
of the optimization problem is to construct a function that takes as input the features of the unoptimized
program being compiled. In other words, this model takes as an input a tuple (F,T )where F is the feature
vector of the collected instrumentation of the program being optimized; and T is one of the several possible
compiler optimization sequences predicted to perform well on this program. Its output is a prediction of the
speedup T should achieve when applied to the original code.
3.4.2 Analysis of Selecting the Compilation Baseline.As explained in Section 3.2, we do not use any of the
default compilation optimization levels as a baseline to start from since we used all compiler optimizations
passes that are used in -O3 for our clustering purposes (see Section 3.2.2). Additionally, we found that using a
baseline compiler optimization level to start from ultimately reduces the speedup achievable from the sequence
we construct with predictive modeling.
Results suggests that using the MiCOMP optimization sequence without an optimization level as a baseline
can lead to substantial bene/f_its compared with using any of-OX optimization levels as a baseline. . Note
that using a baseline of -O1, -O2, or -O3 each converge to a sub-optimal speedup. Thus, applying certain
sequences causes a degradation in performance as can be seen by using these standard optimization levels as
a baseline. The better option is to not use a baseline sequence at all and to allow MICOMP to predict the best
sequence to apply on its own.
The insights of this experiment are threefold: (i) The clustering technique is bene/f_icial; /f_irst, to gain better
speedup values and second, . (ii) The sub-sequences can be coupled with machine-learning techniques so
they can be reordered based on the applications being optimized while outperforming the highest standard
optimizations levels. (iii) Phase-ordering does matter in the /f_ield of compilers; i.e., using the same set of
optimization /f_lags available to-O3, MiCOMP can signi/f_icantly outperform-O3 itself.
3.4.3 Application-specific Prediction. Our machine-learning constructed models can be used for unseen
target applications to predict the speedup when applying compiler sequences to them. The predicted speedup
values correspond to the optimization sequence applied to the program. For a given input program, /f_irst a
feature vector containing dynamic instrumentation is collected. Then our prediction model is fed the features
of the program being compiled to predict the expected speedup if an optimization sequence T was applied to
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
1:10 A. H. Ashouri et al.
0 500 1000 1500 2000 2500 3000 3500 4000
Applied optimization sequences sorted by increasing actual speedup
0.5
0.6
0.7
0.8
0.9
1
1.1
1.2
1.3Normalized speedup to -O3
MiCOMP Proposed Sub-seq Only
Baseline -O1 + MiCOMP Proposed Sub-seq
Baseline -O2 + MiCOMP Proposed Sub-seq
Baseline -O3 + MiCOMP Proposed Sub-seq
Region
of
interest
Fig. 4. Empirical analysis of having diﬀerent compilation baseline across all CBench applications (Harmonic
mean). Region of interest is depicted where MiCOMP sub-sequences outperformed other compiler sequences
having a fixed standard compilation baseline.
it. By predicting the performance of each possible optimization sequence that can be applied, it is possible to
rank the optimization sequences according to their expected speedup and only select the sequences to actually
apply that are predicted to give the highest speedups.
A state-of-the-art ranking approach [34, 36] was used to rank optimization sequences in descending order,
and we only select the top N optimization sequences to evaluate their actual optimization quality. In this
work, we propose an iterative process in which diﬀerent solutions are explored to /f_ind those leading to higher
speedups. In other words, our proposed exploration technique uses the output of our prediction model to
generate an initial exploration strategy, and the exploration strategy dynamically updates itself in order to
reach the highest speedup values in the least number of predictions.
3.5 Recommender System Heuristic
In the initial steps taken by [5, 24], the authors de/f_ined iterative exploration heuristics, based on the current
optimized state of the target application being compiled, to select the next best optimization to apply, which will
bring the eventual best speedup. As the current state of the optimized application depends on the optimizations
that were already applied, this previous approach required several rounds of feature collection. In this paper,
we propose a predictive approach that generates the complete optimization sequence for a program that has
not been optimized, thus it needs to collect features only once before any optimizations are applied.
3.5.1 Adjusted Cosine Similarity. Many of the aforementioned state-of-the-art approaches, tackling both
the selection and the phase-ordering problem, de/f_ine exploration strategies on the optimizations design space.
Yet, to the best of the authors’ knowledge, none of them make use of information in order to dynamically
improve the strategy itself. Dynamic information, in our particular case, is the predicted speedup on the
sequences already explored and evaluated. The knowledge can be eﬀectively used to improve the initial
exploration. The technique we propose, leverages the similarity between the unexplored and the explored
optimization sequences. In particular, our proposed technique prioritizes the evaluation of solutions less
similar to the ones already explored. This is especially important for the phase-ordering problem where there
are a plethora of optimization sequences that need to be explored. The similarity measure is based on how
close the achieved speedup is for predicted solutions across all the training data. As an example, let Sp, i and
Sp, j be the predicted speedups of the sequences i and j when applied to program p in the set of programs P.
We de/f_ine an iterative process to look for predicted similarities ini and j when they were applied to diﬀerent
programs in P.
In recommender system (RS), an algorithm called Basic Cosine Similarity is used to correlate users and
items. However, computing the similarity using this algorithm has one important drawback: the diﬀerence in
rating scale are not taken into account. The Adjusted Cosine Similarity oﬀsets this drawback by subtracting
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
MiCOMP: Mitigating the Compiler Phase-ordering Problem using Opt. Sub-sequences 1:11
the corresponding user-average from each co-rated pair and is shown to have the lowest error-rate amongst
the diﬀerent similarity measurement techniques [ 38]. . Using this method, we can compute the Adjusted
Cosine Similarity between optimization sequence i and j as:
sim (i, j)=
Í
p∈P (Sp, i − ¯Sp )(Sp, j − ¯Sp )
√Í
p∈P (Sp, i − ¯Sp )2
√Í
p∈P (Sp, j − ¯Sp )2
(2)
where Sp, i is the speedup achieved by sequence i when applied to program p of all set of programs P, and
¯Sp is the average speedup on program p. We use the computed measure to evaluate the correlation between a
pair of optimization sequences to bias our exploration strategy.
We de/f_ine the exploration strategy inspired by ACS as follows:
(1) Sort predicted speedup solutions in decreasing order in a list.
(2) Test solutions in order. If the solution to test is too similar to one already tested in the current list iteration, skip it.
(3) If the end of the list has been reached and there are still optimization sequences to test, go to 2. Start from the head of the list
and exclude already tested solutions.
High values of ACS for a pair of optimization sequences are the consequence of achieving pairwise similar
speedups across all training data. We employ this measure to hint exploration priority to the solutions that
are less similar to the ones already tested. .
4 EXPERIMENTAL RESULTS
In this section we evaluate our proposed methodology on an Intel Xeon architecture. We adapted our in-
strumentation and architecture-independent tool (Section 3.1) to extract characteristics from a large set of
benchmarks from the Ctuning CBench suite [14]. We have used LLVM compilation framework v3.8 (Clang for
the frontend/backend and Opt for the optimization passes). The training set consists of diﬀerent applications
ranging from automotive, security, oﬃce, and telecom. The list of applications we evaluated is reported
in Table 3. Table 4 illustrates the list of diﬀerent compiler optimizations that are clustered into 5 diﬀerent
sub-sequences (refer to Section 3.2.2) that are derived from LLVM’s-O3. We used the sub-sequences with no
baseline in MiCOMP and generate the design space enabling orderings and repetitions of these sub-sequences.
The optimizations are /f_ixed within a sub-sequence, but sub-sequences are allowed to appear in any order in
the full optimization sequence.
The application execution time has been estimated by using the Linux Perf tool. . The execution time is
done by averaging three loop-wraps of the speci/f_ic compiled binary with 1s of sleep in between three diﬀerent
executions of those loop-wraps. Therefore, in total, each individual transformed application binary has been
executed 9 times as three packages of three loop-wraps to ensure better accuracy of estimations and fairness
among the generation of executions. This technique is used both in the training and inference phases. In order
to implement a dimension reduction technique we applied PCA.
This analysis reveals that by using a 5-D vector of features (a single-dimensional vector of length 5) we
can capture 98% of the variance available in the training set. The Principal Components (PCs) have been
computed using the MICA features collected from application executions (it is required only once), normalized
by standard deviation across all data sets.
The proposed methodology is prediction model independent, and we report the results using three diﬀerent
models described in Table 5. To this end, we used (i) a Linear Regression (LR) classi/f_ier using the M5 attribute
selection method with default ridge parameter, (ii) a Multilayer Perceptron using the default con/f_iguration,
and the (iii) K* algorithm using default settings. In this work, the WEKA machine learning tool [15] has been
integrated in our framework. We trained diﬀerent speedup predictors, each one by excluding from the training
application set, one of the applications. This technique is called Leave-One-Out-Cross Validation (LOOCV)
and ensures a fair evaluation of our trained models. Validation data is used on the application excluded from
the training set for prediction purpose.
4.1 Analysis of the variability of the distributions
4.2 Analysis of the MiCOMP’s training data dependency
4.3 Analysis of the MiCOMP’s timing breakdown
An application characterization phase takes between 15 to 40 seconds depending on the type of the application.
We noticed a small factor of slowdown when we perform the feature collection phase versus measuring the
pure application’s execution time. The overhead is negligible /f_irst, as it is required once and second, the
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
1:12 A. H. Ashouri et al.
Table 1. Analysis of paucity of data when diﬀerent categories are used
Scenario Speedup w.r.t. MiCOMP’s defaultTraining Testing
Automotive Security 0.9443
Automotive Telcom 0.9743
Automotive Consumer 0.9432
Automotive Network 0.9896
Automotive Oﬃce 0.9896
Table 2. MiCOMP timing breakdown for oﬀline training and online inference
Phase Category Time
Oﬄine Training (A) Oﬄine Data-collection (32 App) 5 days
(B) Model Construction (MLP) 120 sec
Online Prediction
(C) Susan-c Feature Collection 17.4 sec
(D) Susan-c Compilation 4.5 sec
(E) Susan-c Execution 9.7 sec
(F) Prediction (MLP) 2 sec
(G) Recommendation (ACS) 12 sec
0 10 102 103 104 105
Applied optimization sequences sorted by increasing actual speedup
0.7
0.8
0.9
1
1.1
1.2
1.3
1.4
1.5
Normalized speedup to -O3
Max Seq Length 7 (Total: 97 K)
Max Seq Length 6 (Total: 19 K)
Max Seq Length 5 (Total= 4K)
Max Seq Length 4 (Total= 0.781 K)
Max Seq Length 3 (Total= 0.156 K)
1.34
1.38
1.44 1.45
1.23
Fig. 5. Empirical analysis of having diﬀerent compiler sequence lengths on 5 candidate applications: telecom_adpcm_d,
jpeg_d, bzipd, network_dijkstra, automotive_bitcount. Note that X axis is in logarithmic scale.
speedup gained by using MiCOMP is far higher. In MiCOMP, cross validation is done in a few minutes for
each application under analysis. Model construction is heavily correlated with the type of machine learning
algorithm we use. We observed LR to be the fastest and MLP to be slowest for our data.
4.4 Analysis of Longer Sequence Length
As described in Section 3, MiCOMP requires having upper bound on the sequence length for using the encoding
scheme. To this end, we evaluate MiCOMP by having diﬀerent maximum values for the sequence length. A
speedup prediction model requires a one time expensive training be done in order to construct an accurate
model. We believe that the longer the sequence length, the better the chance of /f_inding higher speedup values.
We have tested our proposed sub-sequences with diﬀerent maximum sequence lengths to empirically /f_ind the
most eﬀective length across all the training applications. This is done also with the goal of scalability and
ultimately speeding up the training phase.
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
MiCOMP: Mitigating the Compiler Phase-ordering Problem using Opt. Sub-sequences 1:13
Table 3. Full Cbench Applications under analysis (CTuning CBench suite v1.1 [14])
No. cBench list Description
1 automotive_bitcount Bit counter
2 automotive_qsort1 Quick sort
3 automotive_susan_c Smallest Univalue Segment Assimilating Nucleus Corners
4 automotive_susan_e Smallest Univalue Segment Assimilating Nucleus Edges
5 automotive_susan_s Smallest Univalue Segment Assimilating Nucleus Smoothing
6 security_blow/f_ish_d Symmetric-key block cipher Decoder
7 security_blow/f_ish_e Symmetric-key block cipher Encoder
8 security_rijndael_d AES algorithm Rijndael Decoder
9 security_rijndael_e AES algorithm Rijndael Encoder
10 security_sha NIST Secure Hash Algorithm
11 security_pgp_d public key cryptography for the masses
12 security_pgp_e public key cryptography for the masses
13 telecom_adpcm_c Intel/dvi adpcm coder/decoder Coder
14 telecom_adpcm_d Intel/dvi adpcm coder/decoder Decoder
15 telecom_gsm gsm encoder/decoder
16 telecom_CRC32 32 BIT ANSI X3.66 CRC checksum /f_iles
17 consumer_jpeg_c JPEG kernel
18 consumer_jpeg_d JPEG kernel
19 consumer_lame MP3 encoding engine
20 consumer_mad MPEG audio decoder
21 consumer_tiﬀ2bw convert a color TIFF image to grey scale
22 consumer_tiﬀ2rgba convert a TIFF image to RGBA color space
23 consumer_tiﬀdither convert a TIFF image to dither noisespace
24 consumer_tiﬀmedian convert a color TIFF image to create a TIFF palette /f_ile
25 network_dijkstra Dijkstra’s algorithm
26 network_patricia Patricia Trie data structure
27 oﬃce_stringsearch1 Boyer-Moore-Horspool pattern match
28 oﬃce_ghostscript Aladdin Ghostscript
29 oﬃce_ispell An interactive spelling corrector
30 oﬃce_rsynth Klatt synthesizer
31 bzip2d BurrowsâĂŞWheeler compression algorithm
32 bzip2e BurrowsâĂŞWheeler compression algorithm
Table 4. Candidate clusters of compiler optimizations into sub-sequences (all derived from LLVM -O3)
sub-
seq
Compiler Passes
A -alignment-from-assumptions -argpromotion -barrier -bdce -block-freq -branch-prob -constmerge -deadargelim -demanded-
bits -dse /f_loat2int -forceattrs -functionattrs -globaldce -globalopt -globals-aa -gvn -indvars -inferattrs -inline -ipsccp -jump-
threading -lcssa -loop-accesses -loop-deletion -loop-idiom -loop-unroll -loop-unswitch -loop-vectorize -mldst-motion -prune-
eh -reassociate -rpo-functionattrs -sccp -simplyfycfg -sroa -strip-dead-prototypes
B -licm -mem2reg
C -instcombine -loop-rotate -loop-simplyfy
D -memcpyopt
E -adce -loop-unswitch -slp-vectorize -tailcallelim
Figure 5 gives the Harmonic mean (as suggested by [16] 3) values of the actual speedups using /f_ive selected
applications each having diﬀerent upper bound sequence lengths. We randomly selected an application from
each of CBench categories (automotive, compression, telecom, consumer, oﬃce and network) since it was
impractical to do this analysis with all applications. Having the upper bounds set to 3, 4, 5, 6 and 7 respectively,
gives search spaces of 156, 781, 3909, 19k and 97k distinct permutations of sub-sequences with repetitions
enabled (refer to Equation 1 for the optimization space). The /f_ive speedup lines show the trend of reaching a
higher speedup value by iteratively exploring larger fraction of the optimization space. The maximum speedup
found against -O3 using sequence lengths of 3, 4, 5, 6, and 7, respectively, are 1.23, 1.34, 1.38, 1.44 and 1.45.
These results suggest to set the maximum length to 6 as this ensures achieving good speedups while avoiding
a potential exploration of 100K sequences per each application in the training set.
3We provide harmonic mean rather than geometric mean as we are dealing with averaging speedups. Note that harmonic-
mean is always less than or equal to geometric mean.
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
1:14 A. H. Ashouri et al.
Table 5. List of the predictive models used in our experiments. Note that the proposed methodology is
independent from any specific machine-learning algorithm (classifier) and it can be paired with any algorithm
desired.
Predictive Model Description
MultilayerPerceptron (MLP) A feedforward arti/f_icial neural network model that maps sets of input data onto a set of appropri-
ate outputs. A MLP consists of multiple layers of nodes in a directed graph, with each layer fully
connected to the next one
LinearRegression (LR) An approach for modeling the relationship between a scalar dependent variable y and one or more
explanatory variables (or independent variables) denoted X. In linear regression, the relationships
are modeled using linear predictor functions whose unknown model parameters are estimated from
the data.
KStar It is an instance-based classi/f_ier, that is the class of a test instance is based upon the class of those train-
ing instances similar to it, as determined by some similarity function. It diﬀers from other instance-
based learners in that it uses an entropy-based distance function.
Table 6. Average error rate for the proposed encoding function versus an arbitrary encoding
M.L MiCOMP Encoding Arbitrary Encoding Improvement Factor
MAE AE MAE AE MAE AE
MLP 0.06778 0.05439 0.10826 0.11838 1.59× 2.16×
LR 0.07515 0.07795 0.12879 0.13974 1.71× 1.79×
KStar 0.05129 0.05078 0.09188 0.10866 1.77× 2.13×
4.5 MiCOMP’s Prediction Accuracy
Unlike sequence prediction models [1, 6, 36] in speedup prediction approaches, prediction quality is measured
by means of prediction error. This metric demonstrates how close the prediction values were to the actual
speedups given the same sequence. We use the following diﬀerent error measurement techniques.
Mean Absolute Error. In statistics, the Mean Absolute Error (MAE) [18] is a quantity used to measure how
close predictions are to the eventual outcomes. The mean absolute error is given by:
MAE = 1
n
nÕ
i=1
|fi −/y.alti | (3)
where we de/f_ineei as |fi −/y.alti |given fi as the prediction values and /y.alti the actual values. Consequently, the
value ei is inverse proportional to the accuracy of the prediction.
Approximation Error. Complementary to MAE, Approximation Error (AE) [40] is a common error mea-
surement whereas in some data there is some discrepancy between an exact value and the approximation.
An approximation error can occur because (i) certain measurements of the data are not precise (which we
consider it can be the case for any computer scienti/f_ic measurement) and (ii) approximated values are used
instead of the real values (the iterative prediction way keeps using the predicted values). It is calculated as:
δ = |/uni03F5 |
|/v.alt|= |/v.alt −/v.altapprox |
|/v.alt| (4)
where the absolute error is the magnitude of the diﬀerence between the exact value and the approximation.
These de/f_initions can be extended to the case when/v.alt and /v.altapproximate are n-dimensional vectors, then by
replacing the absolute error with an n-norm error.
4.5.1 Prediction Accuracy. We provide the prediction’s error rate in Table 6. We observe that the arbitrary
encoding leads to higher error rates in the prediction values. . Table 6 shows that the KStar model does slightly
better in terms of accuracy compared with other models, it achieves around 5% error rate on average. In
general, having a smaller error rate does not always guarantee higher performance gain but rather showcases
the accuracy of the prediction model to capture the correlation between diﬀerent compiler sub-sequences and
the speedup values. 4
4We are aware of the many other encoding possibilities that are more eﬃcient (currently having N ×M length). However,
we believe that extending the current encoding scheme to a more sophisticated version is out of scope of the work. Moreover,
the proposed clustering technique can eﬀectively reduce the number of N , thus the encoding scheme is scalable for higher
orders.
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
MiCOMP: Mitigating the Compiler Phase-ordering Problem using Opt. Sub-sequences 1:15
Table 7. Best compiler optimization sub-sequences found using an exhaustive iterative compilation and their
related speedups
Application Best sub-sequence Speedup w.r.t. -O3 Kruskal-wallis p-value
telecom_adpcm_c ECDDCC 1.35 0.9999
security_sha ACCACE 1.06 0.9934
security_blow/f_ish_e.csv BCCEEA 1.13 0.9909
automotive_susan_e.csv AABACA 1.15 0.9981
consumer_tiﬀdither DCEDCD 1.20 0.9999
security_rijndael_e CAEEC 1.10 0.9999
consumer_tiﬀ2bw CCDCD 1.30 0.9999
bzip2e CBADCA 1.30 0.9944
automotive_susan_s ECCCDE 1.22 0.9999
oﬃce_stringsearch1 ABCBAC 1.07 0.9999
telecom_adpcm_d DCAACA 1.13 0.9939
consumer_jpeg_c DDC 1.41 0.9932
network_patricia CECBAA 1.18 0.9954
automotive_susan_c BDBCCB 1.32 0.9999
consumer_tiﬀ2rgba DEDDC 1.32 0.9999
automotive_qsort1 CBAAAC 1.04 0.9969
security_blow/f_ish_d DACECA 1.09 0.9949
network_dijkstra EECBBE 1.51 0.9991
security_rijndael_d ECEACD 1.09 0.9992
bzip2d CBDACA 1.29 0.9994
automotive_bitcount BEACCA 1.19 0.9999
consumer_jpeg_d CCED 1.18 0.9999
consumer_tiﬀmedian BCBACB 1.15 0.9929
telecom_CRC32 DCAACA 1.26 0.9999
telecom_pgp_d DCAACA 1.21 0.9999
telecom_pgp_e DCA 1.22 0.9949
oﬃce_ispell ABEBAE 1.09 0.9999
oﬃce_ghostscript ABCBAC 1.08 0.9999
oﬃce_rsynth ABCBA 1.10 0.9969
consumer_mad DDCA 1.17 0.9999
consumer_lame DDCAB 1.15 0.9999
Harmonic mean 1.31 0.9976
4.5.2 Iterative Compilation Max Speedups. Iterative compilation is known to be able to achieve good
performance results when compiling applications [ 9]. However, the approach is expensive and should be
combined with more intelligent search algorithms [1, 6]. Table 7 reports the maximum speedups found by
an iterative compilation approach using our proposed clustering while exploring the full optimization space
exhaustively. This experiment empirically con/f_irms that the proposed clustering is useful on the phase-ordering
space since we show that we can achieve on average a 31% speedup versus -O3. Figure 4 illustrates the trend
when using MiCOMP sub-sequences with no baseline compared with having a baseline (e.g.: -O1, -O2 or
-O3). The best optimization sequence for each of applications under-analysis and its speedup value are reported
in the second and the third columns of Table 7. Readers can refer to Table 4 to /f_ind the exact set of compiler
optimizations clustered in each sub-sequence.
4.6 Performance Gain of The MiCOMP Technique Against The Ranking Approach
Our approach can improve the exploration to /f_ind the best optimization sequences in an optimization search
space and to /f_ind the best speedups using a fewer number of predictions. Table 8 reports the comparison
between the best speedup found by our approach and a state-of-the-art N-shot approach [34, 36]. The results,
averaged using a Harmonic mean across all applications, show that using the same number of predictions from
both models, our exploration technique can outperform the ranking approach on every number of predicted
optimization sequences used (1, 5, 10, 15 and 20). This shows that our proposed methodology can eﬀectively
predicts the best compiler sequences to use and converges faster to better solutions in the space.
5 COMPARATIVE RESULTS
In this section we evaluate the results of our model against three diﬀerent techniques:(i) Standard optimization
levels, (ii) Random Iterative Compilation (RIC) and, (iii) state-of-the-art prediction Models. We use our
MiCOMP exploration policy and compare the performance of predictions to a previously published ranking
approach [34, 35]. For each application under analysis, we tested the speedup gained using 1, 5 and 10
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
1:16 A. H. Ashouri et al.
Table 8. Prediction improvement of MiCOMP based on Adjusted Cosine Similarities against the Ranking
(N-shot approach)
Exploration Techniques Top-1 Top-5 Top-10 Top-15 Top-20
MiCOMP 1.01 1.06 1.09 1.10 1.12
Ranking 0.93 1.02 1.06 1.07 1.08
Table 9. MLP’s speedup table against LLVM’s-O3. Reported numbers are A (B% ): (A) speedup and (B) percentage
speedup w.r.t. the optimal speedup value of exhaustive exploration. Values are reported for 1 prediction, 5 predictions and
10 predictions.
Application 1 prediction 5 predictions 10 predictions
automotive_bitcount 1.04 (95.38%) 1.07 (98.12%) 1.08 (98.92%)
automotive_qsort1 1.01 (95.32%) 1.03 (96.93%) 1.03 (97.55%)
automotive_susan_c 1.04 (96.61%) 1.06 (98.53%) 1.06 (99.07%)
automotive_susan_e 1.04 (96.47%) 1.03 (98.41%) 1.04 (99.00%)
automotive_susan_s 0.99 (96.26%) 1.01 (98.42%) 1.02 (98.98%)
bzip2d 0.93 (92.77%) 0.96 (94.02%) 1.00 (94.37%)
bzip2e 1.09 (83.77%) 1.10 (86.02%) 1.12 (90.37%)
consumer_jpeg_c 1.01 (85.18%) 1.07 (90.35%) 1.10 (94.51%)
consumer_jpeg_d 1.09 (84.70%) 1.14 (88.97%) 1.17 (97.85%)
consumer_tiﬀ2bw 0.96 (75.54%) 0.99 (80.59%) 1.02 (82.46%)
consumer_tiﬀ2rgba 0.91 (80.61%) 0.95 (86.19%) 1.07 (88.08%)
consumer_tiﬀdither 1.02 (80.14%) 1.09 (85.86%) 1.11 (87.68%)
consumer_tiﬀmedian 0.94 (79.21%) 1.02 (85.72%) 1.06 (89.31%)
consumer_mad 1.02 (82.14%) 1.09 (85.86%) 1.11 (87.68%)
consumer_lame 0.99 (89.21%) 1.02 (90.72%) 1.06 (92.31%)
network_dijkstra 1.13 (60.00%) 1.29 (68.46%) 1.38 (73.00%)
network_patricia 0.91 (74.99%) 0.93 (80.79%) 0.97 (93.91%)
oﬃce_ispell 0.98 (84.99%) 1.01 (90.79%) 1.03 (93.91%)
oﬃce_ghostscript 0.99 (79.99%) 1.03 (82.79%) 1.03 (90.91%)
oﬃce_rsynth 1.01 (84.99%) 1.02 (90.79%) 1.03 (93.91%)
oﬃce_stringsearch1 0.98 (64.99%) 1.02 (70.79%) 1.01 (73.91%)
security_sha 0.93 (64.99%) 1.01 (70.79%) 1.03 (73.91%)
security_blow/f_ish_e 0.97 (64.99%) 1.03 (70.79%) 1.03 (73.91%)
security_blow/f_ish_d 0.97 (64.99%) 0.99 (70.79%) 1.02 (73.91%)
security_rijndael_e 0.99 (64.99%) 1.02 (70.79%) 1.01 (73.91%)
security_rijndael_d 1.00 (64.99%) 1.01 (70.79%) 1.04 (73.91%)
telecom_adpcm_c 0.96 (64.99%) 1.01 (70.79%) 1.02 (73.91%)
telecom_adpcm_d 0.98 (64.99%) 1.02 (70.79%) 1.01 (73.91%)
telecom_gsm_d 0.93 (64.99%) 1.03 (70.79%) 1.04 (73.91%)
telecom_CRC32 1.01 (85.18%) 1.07 (90.35%) 1.10 (94.51%)
telecom_pgp_d 1.04 (96.61%) 1.06 (98.53%) 1.06 (99.07%)
telecom_pgp_e 1.02 (80.14%) 1.09 (85.86%) 1.11 (87.68%)
Harmonic mean 1.03 (84.74%) 1.05 (87.51%) 1.09 (91.52%)
predictions and provide the Harmonic mean values. For example, one can see that for the network_dijkstra
application we can gain a higher speedup values using MiCOMP and, on average even better than -O3
from just the /f_irst prediction. Moreover, we can achieve a 4% performance improvement over-O3 when we
use 5 predicted optimization sequences from our model. Over all our benchmarks, using our model we can
achieve 1%, 4%, and 9% speedups over -O3 using 1, 5, and 10 predicted optimization sequences, respectively.
Consequently, our technique allows MiCOMP to outperform -O3 by high margins.
5.1 Comparison with Standard Optimization Levels
Standard optimization levels have been introduced to achieve good performance on average. However, they
come short of the customized auto-tuning frameworks per architecture/application/dataset. As we showed
in Table 9, MiCOMP can surpass the performance of -O3 with a few predictions on application bases. Here
we provide Table 10 which reports more /f_ine-grained speedup over all standard optimization levels. This
demonstrates how fast (/f_irst number in the tuple) and in what percentage of the explored space (the second
number), the framework is reaching a sequence which can outperform the speci/f_ic standard optimization level.
Each column is reporting two values: (i) in how many predictions and (ii) in what percentage of the whole
con/f_iguration space the propped methodology can outperformOX levels.
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
MiCOMP: Mitigating the Compiler Phase-ordering Problem using Opt. Sub-sequences 1:17
Table 10. Average Speedup w.r.t LLVM -O3. Numbers are A (B%): (A) How fast (in terms of number of
predictions) in average the proposed methodology outperforms LLVM standard Optimizations. (B) The
percentage of the optimization space explored to satisfy the goal.
Predictive Modeling -O1 -O2 -O3
MultilayerPerceptron 1 (0.01%) 1 (0.01%) 2 (0.016%)
LinearRegression 1 (0.01%) 1 (0.01%) 3 (0.02%)
KStar 1 (0.01%) 1 (0.01%) 2 (0.016%)
5.2 Comparison with State-of-the-art Iterative Compilation Models
In this section, we compare MiCOMP with two state-of-the-art intermediate-sequence prediction approaches
proposed in [5, 24].
5.2.1 Intermediate Speedup Comparison Case. (A).Kulkarni et al., [24] used Neuro-Evolution for Augmenting
Topologies (NEAT) to predict the best compiler optimization to apply given the state of source-code being
optimized by the dynamic JIT Jikes RVM compiler. . Contrary to the technique we propose in this paper where
we obtain features of the code only once before it is optimized. Kulkarni et al. used NEAT, a machine-learning
framework based on genetic evolution, to generate many neural-networks where each network was evaluated
on the task of using static source code features to predict the next compiler optimization to apply. NEAT
can make optimization predictions to any given maximum-length to predict the most bene/f_icial sequence of
optimizations for the target application being compiled. In NEAT training time was reported around 10 days
while the current approach requires a few hours to construct the model. Another advantage of the current
work is the fact that it supports multiple predictions from the prediction-space while the NEAT approach can
produce one-shot results based on the stop condition for each application and neural network con/f_iguration.
We reproduced the work by Kulkarni et al. [24] by using 100 chromosomes and 500 generations on 12 core
Xeon(R) CPU E5-1650 v2 @ 3.50GHz with 12GB running on Ubuntu and we report the result in Table 11. We
ran NEAT in parallel with average running time of 1.75 hours per model (the longest took 4 hours). . The
training and prediction is done with leave-one-out cross-validation in order to produce uniform results. .
5.2.2 Intermediate Speedup Comparison Case. (B).Ashouri et al. [5] demonstrated a predictive methodology
to predict an intermediate speedup OF an optimization from the con/f_iguration space given the current state of
the application.
However, unlike [24], they employed dynamic features of the application under study. As mentioned in
Section 5.2.1, a major downside in an intermediate speedup approach is that application feature should be
collected on every state by means feature extraction and this makes the system impractical on large-scale data,
specially, when dynamic features are collected on every state. In addition to an eﬃcient feature collection
process and predicting the complete optimization sequence to apply to the unoptimized code at once, MiCOMP
brings two extensions to the aforementioned work. Second, comparison baseline in [5] was LLVM’s default
optimization, while in this work we provide a comparison against LLVM’s -O3 (we show MiCOMP can
outperform an aggressive optimization setting in LLVM, that is, -O3, in only a few predictions.). Figure 6
demonstrates the comparison. For this comparison, we used the same training data of up to the length of 4 (as
it was declared in [5]) for both models to be uniform on both comparisons. We observe that, except the /f_irst
two predictions, the proposed approach outperforms the intermediate speedup methodology reported in this
work and on average MiCOMP brings 11% speedup gain.
5 10 15 20 25 30 35 40 45 50
number of extractions
0.0
0.2
0.4
0.6
0.8
1.0normalized speedup w.r.t  O3
Proposed approach 
Immediate speedup predictor
Fig. 6. Performance of MiCOMP w.r.t the performance of intermediate speedup predictor approach [5]
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
1:18 A. H. Ashouri et al.
Table 11. Performance comparison of the single prediction by MiCOMP against the intermediate speedup
approach reported in previous work [24]. All values are normalized by -O3.
Application NEAT MiCOMP
Best NN Size 1 prediction 1 prediction
automotive_qsort1 1105 1.0336 1.0385
automotive_bitcount 1536 1.0923 1.0898
automotive_susan_c 607 1.0012 1.0491
automotive_susan_e 613 1.0211 1.0481
automotive_susan_s 1295 1.0135 1.0195
bzip2d 1159 1.0514 0.9898
bzip2e 1259 1.0012 0.9798
consumer_jpeg_c 1327 1.0205 1.1882
consumer_jpeg_e 596 1.0712 1.0981
consumer_tiﬀ2bw 1038 0.9522 0.9491
consumer_tiﬀ2rgba 1147 0.9905 0.9295
consumer_tiﬀdither 612 1.0222 1.0288
consumer_tiﬀmedian 1356 0.9097 0.9497
consumer_lame 1612 1.0221 1.0288
consumer_mad 1256 0.9097 0.9497
network_dijkstra 1343 1.0353 1.1382
network_patricia 622 0.7971 0.8585
oﬃce_ispell 1056 0.9197 0.9897
oﬃce_ghostscript 1356 0.9097 1.0997
oﬃce_rsynth 1306 0.9797 1.0497
oﬃce_stringsearch1 858 0.9897 1.0397
telecom_adpcm_c 958 0.9754 1.0192
telecom_adpcm_d 948 0.9897 1.0232
telecom_gsm_d 924 0.9997 1.0397
telecom_CRC32 886 0.9423 1.0012
telecom_pgp_d 843 0.9697 1.0234
telecom_pgp_e 1002 0.9891 1.0254
security_sha 1536 1.0193 1.0178
security_blow/f_ish_e 1221 1.0023 1.0298
security_blow/f_ish_d 1534 1.0923 1.0898
security_rijndael_e 1132 1.0113 1.0395
security_rijndael_d 1033 1.0123 1.0598
Harmonic Mean 0.9632 1.0295
5.3 Comparison with Random Iterative Optimization
As we illustrated in Section 4.5.2, iterative compilation can improve application performance over standard
compiler optimization sequences [ 1, 9]. Additionally, several published works have shown that drawing
compiler optimization sequences at random can often be as good as using other more complicated search
algorithms, such as genetic algorithms or simulated annealing [ 1, 10, 12]. In this section, we compare the
eﬀectiveness of MiCOMP to a Random Iterative Compilation (RIC) method that samples our clustered subse-
quences from a uniform distribution. We randomized the distribution of predictions 10000 times to make sure
the obtained model is totally uniform. The purpose of this comparison is to show how eﬀective the MiComp
predictive modeling works against random iterative compilation. Our results are presented in Figure 7. To
present our results, we de/f_ineNormalized Performance Improvement (NPI) as the ratio of the performance
improvement achieved over the potential performance improvement:
N P I=
Er ef −E
Er ef −Ebest
(5)
where E is the execution time achieved by the methodology under consideration, Eref is the execution
time achieved with a reference compilation methodology and Ebest is the best execution time that can be
obtained through an exhaustive exploration of all possible compiler optimization sequences in the optimization
space we are exploring. As the execution time E of the iterative compilation methodology under analysis
gets closer to the reference execution time Eref , the value of NPI gets closer to 0, where 0 indicates no
improvement was obtained. As E approaches the best execution time, Ebest , the value of NPI approaches 1.
An NPI value of 1 indicates that the optimal performance available was achieved. The goal of the evaluation
in this section is to show how eﬀective MiCOMP is at exploring the optimization sequence space compared to
RIC. . The X axis pertains to the number of predicted optimization sequences used and the Y axis shows their
corresponding speedup values. We used NPI (scaled within [−∞, 1]) and the speedups are all normalized by
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
MiCOMP: Mitigating the Compiler Phase-ordering Problem using Opt. Sub-sequences 1:19
5 1 0 1 5 2 0 2 5 3 0 3 5 4 0 4 5 5 0
number of predictions
0 .0
0 .2
0 .4
0 .6
0 .8
1 .0normalized speedup w .r.t O3
Proposed Approach
Random
(a) Let both MiCOMP and random explore the opt. space
1-Shot 5-Shot 10-Shot 15-Shot 25-Shot 35-Shot 45-Shot 55-Shot 65-Shot 75-Shot 85-Shot
0.9
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
normalized speedup w.r.t -O3
number of predictions (b) Let only random explore the opt. space
Fig. 7. MiCOMP performance comparison versus random iteration compilation
-O3 performance. Thus, Y = 0 is the speedup line corresponding to -O3. We observe that the performance of
MiCOMP outperforms Random Iterative Compilation with fairly a clear margin on each number of predicted
optimization sequences used. Table 7 gives the the absolute speedup values.
Figure 7b displays another result where we compare a /f_ixed number of predicted optimization sequences for
MiCOMP, that is 5 predicted sequences, versus diﬀerent number of predicted sequences from RIC. That is we
observe the prediction quality of MiCOMP compared to diﬀerent numbers of predicted optimization sequences
drawn from a random distribution. Figure 7b depicts this scenario using a violin plot where the Y axis pertains
to the speedup with respect to the RIC and the X axis corresponds to the diﬀerent predicted optimization
sequences obtained from RIC. Statistically, we observe that the quality of the 5-prediction of MiCOMP is as
good as using 35 prediction optimization sequences from RIC thus we observe that the predicted optimization
sequences derived by MiCOMP can give up to 7×exploration speedup versus the RIC method.
6 CONCLUSION
We proposed and presented a clustering technique for all the compiler optimizations in LLVM’s -O3 and
clustered them in /f_ive diﬀerent optimization sub-sequences to speedup the training and exploration phase.
This method helps us outperform LLVM’s -O3 optimization sequence. Moreover, MiCOMP has a simple
encoding function that encodes an optimization sequence into a bit string, allowing us to apply standard
machine learning techniques that require /f_ixed length feature vectors. We incorporated analogies between
the analyzed problem and the context of Recommender Systems, and integrate similarity measures to boost
exploration eﬃciency. We show that MiCOMP can outperform LLVM’s standard optimization levels with just
a few predicted of optimizations sequences and achieves top 80% of the available speedup by traversing less
than 5% of the optimization sequence space. .
REFERENCES
[1] Felix Agakov, Edwin Bonilla, John Cavazos, Björn Franke, Grigori Fursin, Michael FP O’Boyle, John Thomson, Marc
Toussaint, and Christopher KI Williams. 2006. Using machine learning to focus iterative optimization. In Proceedings
of the International Symposium on Code Generation and Optimization . IEEE Computer Society, 295–305.
[2] Alan Agresti, I Liu, and others. 1999. Modeling a categorical variable allowing arbitrarily many category choices.
Biometrics 55, 3 (1999), 936–943.
[3] Amir Hossein Ashouri. 2012. Design space exploration methodology for compiler parameters in VLIW processors . Master’s
thesis. Politecnico di Milano, Italy. http://hdl.handle.net/10589/72083.
[4] Amir Hossein Ashouri. 2016. Compiler Autotuning Using Machine Learning Techniques . Ph.D. Dissertation. Politecnico
di Milano, Italy. http://hdl.handle.net/10589/129561.
[5] Amir Hossein Ashouri, Andrea Bignoli, Gianluca Palermo, and Cristina Silvano. 2016. Predictive Modeling Methodology
for Compiler Phase-Ordering. In Proceedings of 7th Workshop on Parallel Programming and Run-Time Management
Techniques for Many-core Architectures and 5th Workshop on Design Tools and Architectures for Multicore Embedded
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
1:20 A. H. Ashouri et al.
Computing Platforms . ACM. DOI:http://dx.doi.org/10.1145/2872421.2872424
[6] Amir Hossein Ashouri, Giovanni Mariani, Gianluca Palermo, Eunjung Park, John Cavazos, and Cristina Silvano. 2016.
COBAYN: Compiler Autotuning Framework Using Bayesian Networks. ACM Trans. Archit. Code Optim. 13, 2, Article
21 (June 2016), 25 pages. DOI:http://dx.doi.org/10.1145/2928270
[7] Amir Hossein Ashouri, Giovanni Mariani, Gianluca Palermo, and Cristina Silvano. 2014. A Bayesian network approach
for compiler auto-tuning for embedded processors. In Embedded Systems for Real-time Multimedia (ESTIMedia), 2014
IEEE 12th Symposium on . IEEE, 90–97. DOI:http://dx.doi.org/10.1109/ESTIMedia.2014.6962349
[8] Amir Hossein Ashouri, Vittorio Zaccaria, Sotirios Xydis, Gianluca Palermo, and Cristina Silvano. 2013. A framework
for Compiler Level statistical analysis over customized VLIW architecture. In Very Large Scale Integration (VLSI-SoC),
2013 IFIP/IEEE 21st International Conference on . IEEE, 124–129. DOI:http://dx.doi.org/10.1109/VLSI-SoC.2013.6673262
[9] François Bodin, Toru Kisuki, Peter Knijnenburg, Mike O’Boyle, and Erven Rohou. 1998. Iterative compilation in a
non-linear optimisation space. In Workshop on Pro/f_ile and Feedback-Directed Compilation.
[10] John Cavazos, Grigori Fursin, Felix Agakov, Edwin Bonilla, Michael F. P. O’Boyle, and Olivier Temam. 2007. Rapidly
Selecting Good Compiler Optimizations Using Performance Counters. In Proceedings of the International Symposium
on Code Generation and Optimization (CGO ’07) . IEEE Computer Society, Washington, DC, USA, 185–197. DOI:
http://dx.doi.org/10.1109/CGO.2007.32
[11] John Cavazos and J Eliot B Moss. 2004. Inducing heuristics to decide whether to schedule. In ACM SIGPLAN Notices ,
Vol. 39. ACM, 183–194.
[12] Yang Chen, Shuangde Fang, Yuanjie Huang, Lieven Eeckhout, Grigori Fursin, Olivier Temam, and Chengyong Wu.
2012. Deconstructing iterative optimization. ACM Transactions on Architecture and Code Optimization (TACO) 9, 3
(2012), 21.
[13] Keith D Cooper, Philip J Schielke, and Devika Subramanian. 1999. Optimizing for reduced code space using genetic
algorithms. In ACM SIGPLAN Notices , Vol. 34. ACM, 1–9.
[14] Grigori Fursin. 2010. Collective benchmark (cbench), a collection of open-source programs with multiple datasets
assembled by the community to enable realistic benchmarking and research on program and architecture optimization.
(2010).
[15] Mark Hall, Eibe Frank, Geoﬀrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H Witten. 2009. The WEKA
data mining software: an update. ACM SIGKDD explorations newsletter 11, 1 (2009), 10–18.
[16] Torsten Hoe/f_ler and Roberto Belli. 2015. Scienti/f_ic benchmarking of parallel computing systems: twelve ways to tell
the masses when reporting performance results. In Proceedings of the International Conference for High Performance
Computing, Networking, Storage and Analysis . ACM, 73.
[17] Kenneth Hoste and Lieven Eeckhout. 2007. Microarchitecture-independent workload characterization. IEEE Micro 27,
3 (2007), 63–72.
[18] Rob J Hyndman and Anne B Koehler. 2006. Another look at measures of forecast accuracy. International journal of
forecasting 22, 4 (2006), 679–688.
[19] Michael R Jantz and Prasad A Kulkarni. 2013. Exploiting phase inter-dependencies for faster iterative compiler
optimization phase order searches. In Compilers, Architecture and Synthesis for Embedded Systems (CASES), 2013
International Conference on . IEEE, 1–10.
[20] Kevin J Johnson and Robert E Synovec. 2002. Pattern recognition of jet fuels: comprehensive GC×GC with ANOVA-
based feature selection and principal component analysis. Chemometrics and Intelligent Laboratory Systems 60, 1
(2002), 225–237.
[21] Richard Arnold Johnson, Dean W Wichern, and others. 1992. Applied multivariate statistical analysis . Vol. 4. Prentice
hall Englewood Cliﬀs, NJ.
[22] Toru Kisuki, Peter M. W. Knijnenburg, and Michael F. P. O’Boyle. 2000. Combined Selection of Tile Sizes and Unroll
Factors Using Iterative Compilation. In Proceedings of the 2000 International Conference on Parallel Architectures
and Compilation Techniques (PACT’00), Philadelphia, Pennsylvania, USA, October 15-19, 2000 . 237–248. DOI:http:
//dx.doi.org/10.1109/PACT.2000.888348
[23] Prasad A Kulkarni, David B Whalley, Gary S Tyson, and Jack W Davidson. 2009. Practical exhaustive optimization
phase order exploration and evaluation. ACM Transactions on Architecture and Code Optimization (TACO) 6, 1 (2009),
1.
[24] Sameer Kulkarni and John Cavazos. 2012. Mitigating the compiler optimization phase-ordering problem using machine
learning. ACM SIGPLAN Notices 47, 10 (2012), 147–162.
[25] David B Loveman. 1977. Program improvement by source-to-source transformation. Journal of the ACM (JACM) 24, 1
(1977), 121–145.
[26] Chi-Keung Luk, Robert Cohn, Robert Muth, Harish Patil, Artur Klauser, Geoﬀ Lowney, Steven Wallace, Vijay Janapa
Reddi, and Kim Hazelwood. 2005. Pin: building customized program analysis tools with dynamic instrumentation.
In Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation (PLDI ’05) .
ACM, New York, NY, USA, 190–200. DOI:http://dx.doi.org/10.1145/1065010.1065034
[27] Luiz GA Martins, Ricardo Nobre, Joao MP Cardoso, Alexandre CB Delbem, and Eduardo Marques. 2016. Clustering-
Based Selection for the Exploration of Compiler Optimization Sequences. ACM Transactions on Architecture and Code
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.
MiCOMP: Mitigating the Compiler Phase-ordering Problem using Opt. Sub-sequences 1:21
Optimization (TACO) 13, 1 (2016), 8.
[28] Antoine Monsifrot, François Bodin, and Rene Quiniou. 2002. A machine learning approach to automatic production of
compiler heuristics. In Arti/f_icial Intelligence: Methodology, Systems, and Applications. Springer, 41–50.
[29] J Eliot B Moss, Paul E Utgoﬀ, John Cavazos, Doina Precup, Darko Stefanovic, Carla Brodley, and David Scheeﬀ. 1997.
Learning to schedule straight-line code. In NIPS, Vol. 97. 929–935.
[30] Ricardo Nobre, Luiz GA Martins, and João MP Cardoso. 2015. Use of Previously Acquired Positioning of Optimizations
for Phase Ordering Exploration. In Proceedings of the 18th International Workshop on Software and Compilers for
Embedded Systems . ACM, 58–67.
[31] Ricardo Nobre, Luiz GA Martins, and João MP Cardoso. 2016a. A graph-based iterative compiler pass selection and
phase ordering approach. In Proceedings of the 17th ACM SIGPLAN/SIGBED Conference on Languages, Compilers, Tools,
and Theory for Embedded Systems . ACM, 21–30.
[32] Ricardo Nobre, Luıs Reis, and Joao MP Cardoso. 2016b. Compiler Phase Ordering as an Orthogonal Approach for
Reducing Energy Consumption. In Proceedings of the 19th Workshop on Compilers for Parallel Computing (CPC2016),
Valladolid, Spain, July 6–8, 2016 .
[33] Holger Orup. 1999. On-the-/f_ly one-hot encoding of leading zero count. (Oct. 26 1999). US Patent 5,974,432.
[34] EunJung Park, John Cavazos, and Marco A Alvarez. 2012. Using graph-based program characterization for predictive
modeling. In Proceedings of the Tenth International Symposium on Code Generation and Optimization . ACM, 196–206.
[35] Eunjung Park, John Cavazos, Louis-Noël Pouchet, Cédric Bastoul, Albert Cohen, and P Sadayappan. 2013. Predictive
modeling in a polyhedral optimization space. International Journal of Parallel Programming 41, 5 (2013), 704–750.
[36] Eunjung Park, Sameer Kulkarni, and John Cavazos. 2011. An evaluation of diﬀerent modeling techniques for iterative
compilation. In Proceedings of the 14th international conference on Compilers, architectures and synthesis for embedded
systems. ACM, 65–74.
[37] Suresh Purini and Lakshya Jain. 2013. Finding good optimization sequences covering program space.ACM Transactions
on Architecture and Code Optimization (TACO) 9, 4 (2013), 56.
[38] Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001. Item-based collaborative /f_iltering recommenda-
tion algorithms. In Proceedings of the 10th international conference on World Wide Web . ACM, 285–295.
[39] Satu Elisa Schaeﬀer. 2007. Graph clustering. Computer Science Review 1, 1 (2007), 27–64.
[40] Steve Smale and Ding-Xuan Zhou. 2003. Estimating the approximation error in learning theory. Analysis and
Applications 1, 01 (2003), 17–41.
[41] Mark Stephenson, Saman Amarasinghe, Martin Martin, and Una-May O’Reilly. 2003. Meta Optimization: Improving
Compiler Heuristics with Machine Learning. In Proceedings of the ACM SIGPLAN 2003 Conference on Programming
Language Design and Implementation (PLDI ’03) . ACM, New York, NY, USA, 77–90. DOI:http://dx.doi.org/10.1145/
781131.781141
[42] Steven R. Vegdahl. 1982. Phase coupling and constant generation in an optimizing microcode compiler.ACM SIGMICRO
Newsletter 13, 4 (1982), 125–133.
[43] Wei Zhang, Deli Zhao, and Xiaogang Wang. 2013. Agglomerative clustering via maximum incremental path integral.
Pattern Recognition 46, 11 (2013), 3056–3065.
ACM Transactions on Architecture and Code Optimization, Vol. , No. , Article 1. Publication date: September 2017.

Using Machine Learning to Focus Iterative Optimization
F. Agakov, E. Bonilla, J.Cavazos, B.Franke, G. Fursin,
M.F.P. O'Boyle, J. Thomson, M. Toussaint, C.K.I. Williams
School of Informatics
University of Edinburgh
UK
Abstract
Iterative compiler optimization has been shown to out-
perform static approaches. This, however, is at the cost of
large numbers of evaluations of the program. This paper de-
velops a new methodology to reduce this number and hence
speed up iterative optimization. It uses predictive modelling
from the domain of machine learning to automatically focus
search on those areas likely to give greatest performance.
This approach is independent of search algorithm, search
space or compiler infrastructure and scales gracefully with
the compiler optimization space size. Off-line, a training
set of programs is iteratively evaluated and the shape of the
spaces and program features are modelled. These models
are learnt and used to focus the iterative optimization of a
new program. We evaluate two learnt models, an indepen-
dent and Markov model, and evaluate their worth on two
embedded platforms, the Texas Instrument C6713 and the
AMD Au1500. We show that such learnt models can speed
up iterative search on large spaces by an order of magni-
tude. This translates into an average speedup of 1.22 on the
TI C6713 and 1.27 on the AMD Au1500 in just 2 evalua-
tions.
1 Introduction
Using iterative search as a basis for compiler optimiza-
tion has been widely demonstrated to give superior perfor-
mance over static schemes [1, 3, 11]. The main drawback
of these schemes is the amount of search time needed to
achieve performance improvements given that each point
of the search is a recompilation and execution of the pro-
gram. Although multiple recompilations/executions are ac-
ceptable for embedded code, libraries and persistent appli-
cations, these long compilation/execution cycles restrict the
space of options searched. On a larger scale, they are a sig-
nicant barrier to adoption in general purpose compilation.
There have been a number of papers focusing on reduc-
ing the cost of iterative optimization. As a single evaluation
consists of a compilation plus execution of the program, two
recent papers have investigated reducing the cost of an indi-
vidual compilation or execution [7]. In [1, 6] a more radical
approach is used to reduce the total number of evaluations.
Cooper et al [6] examine the structure of the search space,
in particular the distribution of local minima relative to the
global minima and devise new search based algorithms that
outperform generic search techniques. An alternative ap-
proach is developed in [20]. Here the space of compiler
options is examined off-line on a per function basis and the
best performing ones classied into a small tree of com-
piler options. When compiling a new program, the tree is
searched by compiling and executing the best path in the
tree. As long as the best sequences can be categorized into
a small tree, this proves to be a highly effective technique.
This paper develops a new methodology to speed up it-
erative optimization. It automatically focuses any search on
those areas likely to give the greatest performance. This
methodology is based on machine learning, is independent
of search algorithm, search space or compiler infrastruc-
ture and scales gracefully with the compiler optimization
space size. It uses program features to correlate the pro-
gram to be optimized with previous knowledge in order to
focus the search. Off-line, a training set of programs is iter-
atively evaluated and the shape of the spaces and program
features are recorded. From this data, our scheme automat-
ically learns a model which predicts those parts of the op-
timization space that are likely to give good performance
improvements for different classes of programs. When a
new program is then encountered, an appropriate predictive
model is selected, based on program features, which then
biases the search to a certain area of the space. Using this
technique we are able to speed up search by an up to an
order of magnitude on large spaces.
This paper is structured as follows. Section 2 provides
a motivating example demonstrating how learning where to
search can signicantly reduce the number of evaluations
needed to nd good performance improvements. This is
1
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
 1  10  100  1000
Percent of Max Improvement Available
Evaluations
86%
38%
RANDOM
FOCUSSED
(a) (b)
Figure 1. (a) Points corresponding to those transformation sequences whose performance is within
5 per cent of the optimum for adpcm on the TI C6713. The contour is the predicted area for good
optimizations. (b) How close to the best performance random and focused search achieve for each
program evaluation. The random algorithm achieves 38 % of the maximum improvement in 10 evalau-
tions; the focused search 86%.
followed in section 3 by a description of the experimental
setup, an analysis of the optimization spaces encountered
and an examination of how two standard search algorithms,
random and genetic, perform. This is followed in section 4
by a description of two predictive models and a demonstra-
tion of how they can be used to speed up search. Section
5 describes how the standard machine learning techniques
of principal components analysis and nearest neighbors are
used to learn the predictive models. In section 6, these learnt
models are then tested on (i) a medium sized exhaustively
enumerated space and (ii) a very large space, where they
are shown to improve search performance by up to an or-
der of magnitude. Section 7 describes related work and is
followed in section 8 with some brief conclusions.
2 Motivation & Example
This paper focuses on embedded applications where per-
formance is critical and consequently there has been a large
body of work aimed at improving the performance of opti-
mizing compilers, e.g. [14]. Most of this work focuses on
improving back-end, architecture specic compiler phases
such as code generation, register allocation and schedul-
ing. However, the investment in ever more sophisticated
back-end algorithms produces diminishing returns. Iterative
approaches based on back-end optimizations consequently
give relatively small improvements [5]. In this paper, we
consider source-level transformations [19, 8] for embedded
systems. Such an approach is, by denition, highly portable
from one processor to the next and provides additional ben-
et to the manufacturer's highly tuned compiler. However,
this portability comes at cost. For example, in [8], Franke
et al require 1000 evaluations to achieve reasonable perfor-
mance improvements.
Search space The reason for this excessive search time
is that determining the best high level sequence of transfor-
mations for a particular program is non-trivial. Consider the
diagram in gure 1 (a) showing the behavior of the adpcm
program on the Texas Instrument's C6713. This diagram
is an attempt at plotting all of the good performing points
(within 5% of the optimum) in the space of all transforma-
tions of length 5 selected from a set of 14 transformations.
It therefore covers a space of size
 
. It is difcult to rep-
resent a large 5 dimensional space graphically so each good
performing transformation sequence (
	


is plotted
at position 
	

on the x-axis, which denotes prexes of
length 2, and position 

on the y axis, which denotes
sufces of length 3. The most striking feature is that min-
ima are scattered throughout the space and nding the very
best is a difcult task. Prior knowledge about where good
points were likely to be, could focus our search allowing
the minimal point to be found faster. Alternatively, given a
xed number of evaluations, we can expect improved per-
formance if we know good areas to search within.
2
Focused search In this paper we develop a technique that
learns off-line, ahead of time, a predictive model from iter-
ative evaluations of other programs. This predictive model
then denes good regions of the space to search. In gure
1 (a) the contour lines enclose those areas where our tech-
nique predicts there will be good points. Using this predic-
tion we are able to reduce the number of searches to achieve
the same performance - rapidly reducing the cost of iterative
search. This can be seen in gure 1 (b), which compares
random search (averaged over 20 trials to be statistically
meaningful) with and without the predictive model focus.
The x-axis denotes (logarithmic scale) the number of evalu-
ations performed by the search. The y-axis denotes the best
performance achieved so far by the search ; 0% represents
the original code performance, 100% the maximum perfor-
mance achievable. It is immediately apparent that the pre-
dictive model rapidly speedups up the search. For instance,
after 10 evaluations, random searching achieves 38% of the
potential improvement available while the focused search
achieves 86%. As can be seen from gure 1 (b), such a
large improvement would require over 80 evaluations using
random search, justifying further investigation of predictive
models.
3 Optimization Space
This paper develops machine learning techniques to im-
prove the search performance of iterative optimization.
This section briey describes the benchmarks we use, the
program transformations which make up the optimization
space and the embedded platforms we evaluate. It then
characterises the space and presents two standard search al-
gorithms which are later used to show how learnt predictive
models can dramatically speed up search.
  
			 
Benchmarks The UTDSP [13, 17] benchmark suite was
designed to evaluate the quality of code generated by a
high-level language (such as C) compiler targeting a pro-
grammable digital signal processor (DSP) [13]. This set
of benchmarks contains small, but compute-intensive DSP
kernels as well as larger applications composed of more
complex algorithms. The size of programs ranges from 20-
500 lines of code where the runtime is usually below 1 sec-
ond. However, these programs represent compute-intensive
kernels widely regarded most important by DSP program-
mers and are used indenitely in stream-processing appli-
cations.
Transformations In this paper we consider source to
source transformations (Many of these transformations
also appear within the optimisation phases of a native
Label Transformation
1,2,3,4 Loop unrolling
f Loop attening
n FOR loop normalization
t Non-perfectly nested loop conversion
k Break load constant instructions
s Common subexpression elimination
d Dead code elimination
h Hoisting of loop invariants
i IF hoisting
m Move loop-invariant conditionals
c Copy propagation
Table 1. The labeled transformations used
for the exhaustive enumeration of the space.
1,2,3,4 corresponds to the loop unroll factor
compiler[1]), applicable to C programs and available within
the restructuring compiler SUIF 1 [10]. For the purpose
of this paper, we have selected eleven transformations de-
scribed and labeled in table 1. As we (arbitrarily) consider
four loop unroll factors, this increases the number of trans-
formations considered to 14. We then exhaustively evalu-
ated all transformations sequences of length 5 selected from
these 14 options. This allows us to evaluate the relative per-
formance of our proposed techniques. In the later evalua-
tion section (see section 7), we also consider searching, non
exhaustively, a much larger space.
Platforms Our experiments were performed on two dis-
tinct platforms to demonstrate that our technique is generic.
TI: The Texas Instrument C6713 is a high end oating point
DSP. The wide clustered VLIW processor has 256kB of in-
ternal memory. The programs were compiled using the TI's
Code Composer Studio Tools Version 2.21 compiler with
the highest -O3 optimization level and -ml3 ag (generates
large memory model code). AMD: The AMD Alchemy
Au1500 processor is an embedded SoC processor using a
MIPS32 core (Au1), running at 500MHz. It has 16KB in-
struction cache and 16KB non-blocking data cache. The
programs were compiled with GCC 3.2.1 with the -O3 com-
pile ag. According to the manufacturer, this version/option
gives the best performance - better than later versions of
GCC - and hence was used in our experiments.
 "! #%$&(')	*+ ,-*$&	./ ('	
In order to characterize the optimization space, we ex-
haustively enumerated all
  
transformation sequences on
both platforms. Table 2 summarizes the performance avail-
3
TI AMD
Prog. Improv. Seq. Improv. Seq.
fft 3.64%   3nm 4.49%   4hns
r 45.5%   4 26.7%   3
iir 16.3%   3h 29.5%   h4
latnrm 0.34%   nsch 27.1%   csh4
lmsr 0.39%   1s 30.3%   s3
mult 0.00%   30.5%   4
adpcm 24.0%   1ish 0.75%   ism
compress 39.1%   4s 24.0%   hs4
edge 5.06%   3 23.1%   ch4
histogram 0.00%   24.7%   4
lpc 10.7%   sn2 6.01%   h4cnm
spectral 7.46%   n4 8.53%   sh4
Average 15.2% - 19.64% -
Table 2. Summary of optimization space on
the TI and AMD using exhaustive search
able; columns 2 and 3 refer to the TI while columns 4 and 5
refer to the AMD respectively.
Improved execution time The columns labeled
Improv. (cols. 2 and 4) shows the maximum re-
duction in execution time obtained on the TI and AMD
within this exhaustively enumerated space. Eight (out
of twelve) benchmarks for Texas Instruments and eleven
(out of twelve) benchmarks for AMD achieved signicant
improvement. The best execution time reduction was
45.5% on the TI and 30.5% on the AMD. On average, a
12.7% reduction was achieved for the TI and 13.8% for the
AMD. This translates into an average speedup of 1.15 and
1.16 over the platform specic optimizing compiler.
Best performing sequences The columns labeled Seq.,
(columns 3 and 5) in table 2 contain the best performing
sequence for each benchmark on each machine. The indi-
vidual letters within each entry refer to the labeled trans-
formations in table 1, e.g. i = if hoisting. These en-
tries show that the complexity and type of good transforma-
tion sequences is program dependent. While benchmarks
such as ﬁr and edge detect for the TI and ﬁr, mult and his-
togram for the AMD reach their best performance with sin-
gle transformations, other benchmarks such as adpcm for
the TI and lpc for the AMD obtain their minimum execu-
tion time with four and ve-length sequences respectively.
Similarly, transformations that yield good performance on
some benchmarks do not appear in the best sequences of
other programs. For example, on the AMD the sequence
  ism makes adpcm run at its minimum execution time;
however, none of these three individual transformations is
present in the best performing sequence of compress.
Critically, the best performing sequence on one program
is never the best for another. Therefore, a technique which
tries to simply apply the best sequence found on other pro-
grams is unlikely to succeed.
    . 	(*'$ ,  $ -
In this section, we describe two common methods used
to search the transformation spaces: a blind random search
(RA) and a smarter genetic algorithm (GA). Random
search generates a random string of transformations where
each transformation is equally likely to be chosen and per-
forms surprisingly well in our experience. We congured
our GA in the same manner as best GA in [6] with an
initial randomly selected population of 50.
For the exhaustively enumerated space, both algorithms
have similar performance as can be seen in gure 2. Here
we plot the best performance achieved so far by each al-
gorithm against how many program evaluations have been
performed. This plot is averaged over all programs. Im-
provements by either algorithm are more easily achieved on
the TI due to the much greater number of sequences giving
a signicant speedup.
Both algorithms have similar overall performance with
the GA performing well on the AMD in the early part of
the search. However, random search performs better after
a large number of evaluations as the GA appears to more
likely to be stuck in local minima. In both cases, however,
large numbers of evaluations are needed to gain any signif-
icant performance improvements. In the next section we
investigate how predictive models can speed up this search.
4 Models to focus search
In order to speed up the search algorithm we wish to
focus our attention on protable areas of the optimization
space. We wish to build a model of those transformation
sequences for which a program obtained good performance
in the hope that this can be learnt and used on later pro-
grams. We could simply record the best sequence achieved
on other programs and hope that it improves our current
program. However, this has a number of aws. Firstly, as
the results in table 2 show, the best transformation on one
program is never the best on others. Secondly, knowing the
best sequence on another program only provides one single
option and cannot guide subsequent search within a larger
space.
Alternatively, we can build intricate models that char-
acterize the performance of all transformation sequences.
Here the problem is that we can easily overt the model to
the data so that it cannot be generalized to other programs.
4
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
 1  10  100  1000
Percent of Max Improvement Available
Evaluations
RANDOM
GA
0%
20%
40%
60%
80%
100%
 1  10  100  1000
Percent of Max Improvement Available
Evaluations
RAND
GA
(a) TI (b) AMD
Figure 2. Performance with respect to evaluations for the random (RAND) and genetic (GA) search
algorithms on the TI (a) and AMD (b). The x-axis denotes (logarithmic scale) the number of evalu-
ations performed by each search. The y-axis denotes the best performance achieved so far by the
search; 0 % represents the original code performance, 100% the maximum performance achievable.
Results averaged over all benchmarks
Furthermore, such a complex model will require extensive
training data, which may be costly to gather and is unre-
alistic in practise. In this section we consider two differ-
ent models which try to summarize the optimization space
without excessive overtting. We consider (i) a simple inde-
pendent distribution model and (ii) a more complex Markov
model. Both of these require relatively small amounts of
training data to construct and should be easy to learn (see
section 5).
 &  	
	 	  	')    * 	
	

   	
It makes sense to start with the simplest approach rst:
modelling program transformations as if they were inde-
pendent. We know that this assumption does not hold in
general, but it might be sufcient to better focus search
algorithms. Consider a set of  transformations 
   
	  . Let   	!" be a sequence of
transformations  of length # , where each element %$ is cho-
sen from the transformations in  . Under the independent
model we assume that the probability of a sequence of trans-
formations being good is simply the product of each of the
individual transformations in the sequence being good, i.e.:
&
 %!	'!"


"
(
$*) 
&
$

 (1)
Here
&
 +

is the probability that the transformation ,+ oc-
curs in good sequences. For our dataset we have chosen the
set of good sequences to be those sequences that have an
improvement in performance of at least -
.0/ of the maxi-
mum possible improvement. We calculate
&
 1+

by simply
counting the number of times 1+ occurs in good sequences
and normalize the distribution i.e. 2

$3) 
&

+


 
. We
then record within a vector the probability of each of the
4
 
transformations.
For each benchmark we can build this probability vector
or IID distribution. We refer to this as the IID-oracle. It is
an oracle in the sense that we can only know its value once
we have exhaustively enumerated the space, which in prac-
tise is unrealistic. Our goal is to be able to predict this oracle
by using machine learning techniques based on a training
set of programs in order to improve search. However, it
is necessary to prove rst that this oracle distribution does
indeed lead to better search algorithms.
 &"! 5 (
6 8795  	
As described above, the IID probability distribution
function assumes that all transformations are mutually in-
dependent neglecting the effect of interactions among trans-
formations. This can be very restrictive, particularly when
there are transformations that enable the applicability of
other transformations or when some of them only yield
good performance when others are applied. Therefore, in-
cluding these interactions in our technique makes possible
the construction of richer models that ideally will improve
biased search algorithms and will obtain good performance
in fewer evaluations.
In order to keep the number of samples needed for build-
ing our probability density function low while including in-
5
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
 1  10  100  1000
Percent of Max Improvement Available
Evaluations
32%
75%
87%
RANDOM
IID-ORC
MAR-ORC
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
 1  10  100  1000
Percent of Max Improvement Available
Evaluations
27%
79%
88%
GA
GA-IID-ORC
GA-MAR-ORC
(a) TI: Random (b) TI: GA
Figure 3. TI: Random (a) and GA (b) search versus IID-oracle and Markov oracle. Results averaged
over all benchmarks
teractions among transformations, we can take one step fur-
ther from the IID distribution by using a Markov chain. A
Markov chain for transformation sequences can be dened
as follows:
&



&
 

"
(
$*) 	
&
$  $ 


The equation above states that the probability of a trans-
formation applied in the sequence depends upon the trans-
formations that have been applied before. The main as-
sumption under this model is that these probabilities do not
change along the sequence, i.e. they are the same at any
position of the sequence, and therefore the model is often
referred as a stationary Markov chain. This oversimplica-
tion prevents the number of parameters of the model from
increasing with the length of the sequences considered.
Thus, the parameters of the model are the probability
at the rst position of the sequence
&
 

and the transi-
tion matrix
&
$  $ 

with  
 
' # , which as be-
fore can be learnt from data by counting. Once again
2

+
) 
&
   
+


 
and
2

+
) 
&
$  
+
  $ 


 
must be satised.
As in section 4.1 the parameters of the model have been
learnt from those sequences that have an improvement in
performance at least -
.
/ of the maximum possible im-
provement. Using this model gives a 14 x 14 matrix.
 &   ./
		 &,   	(*' $  7 &(&, $ 	
 	  $ 	   	
To test the potential of our scheme, we compared each
baseline search algorithm against this same algorithm us-
ing each predictive model. For the random algorithm, in-
stead of having a uniform probability of a transformation
being selected, each model biases certain transformations
over others. In the case of the GA, the initial population is
selected based on the model's probabilities and then the GA
is allowed to evolve as usual.
We construct each model using the results obtained from
searching a particular program's space and then test each
model-enabled search algorithm on the same benchmark;
we call these two learnt models: IID-oracle and Markov-
oracle. These oracles form an upper-bound on the perfor-
mance we can expect to achieve when later trying to learn
each model. This is to evaluate whether such models can
improve the search. Clearly, if the best a model oracle can
achieve is insignicant, it is not worth expending effort in
trying to learn it.
Figure 4 (a) depicts the average performance, over all
our benchmarks, of the baseline random algorithm against
random search biased with the two oracles on the TI. Sim-
ilarly, Figure 4 (b) depicts the performance of the baseline
GA algorithm versus using the two oracles to generate the
initial population. In both gures, we see that the oracles
can signicantly speed up nding a good solution. For ex-
ample, at evaluation 10, randomachieves less than 20% of
the maximum available performance. In contrast, random
+ IID-oracleachieves more than 50% of the available
performance and random + Markov-oracleachieves
around 80% of the performance. Figures 5 depicts a sim-
ilar picture on the AMD architecture. On the AMD ar-
chitecture, our two oracles signicantly improve the per-
formance of each baseline algorithm. The baseline ran-
dom search algorithm only achieves 4% of the available
performance after 10 evaluations. In contrast, random
+ IID-oracle achieves about 20% of the available
performance (5 times better than base) and random +
Markov-oracle achieves 40% of the available perfor-
6
0%
20%
40%
60%
80%
100%
 1  10  100  1000
Percent of Max Improvement Available
Evaluations
22%
41%
66%
RAND
IID-ORC
MAR-ORC
0%
20%
40%
60%
80%
100%
 1  10  100  1000
Percent of Max Improvement Available
Evaluations
22%
41%
72%
GA
GA-IID-ORC
GA-MAR-ORC
(a) AMD: Random (b) AMD: GA
Figure 4. AMD: Random (a) and GA (b) search versus IID-oracle and Markov-oracle. Results averaged
over all benchmarks
mance, an order of magnitude better than the base random
algorithm. On average, the baseline algorithm needs 100
evaluations to achieve the same performance as the base-
line + the Markov oracle achieves with just 10 evaluations.
We can see from these gures, that the IID and Markov
models have the potential to dramatically improve the per-
formance of both search algorithms. In the next section we
describe how we can learn these models from previous off-
line runs to build a predictive model.
5 Learning a model
The biggest difculty in applying knowledge learnt off-
line to a novel input is considering exactly which portions of
this knowledge are relevant to the new program. We show
that, as is the case in many other domains, programs can
be successfully represented by program features, which can
then be used to gauge their similarity and thus the applica-
bility of previously learnt off-line knowledge.
Obviously, the selection of these program features is crit-
ical to the success of this method, and so we employ a well
known statistical technique, principal component analysis
(PCA) [2], to assist the selection. Initially, we identied
thirty-six loop-level features we thought might describe the
characteristics of a program well, and use them as input for
the PCA process as shown in table 3. PCA tells us that, in
this instance, due to redundancy and covariance in the fea-
tures' values, these thirty-six features can be combined in
such a way that they can be reduced to only ve features,
whilst retaining 99% of the variance in the data. The output
of this process is a 5-D feature vector for each benchmark,
containing these ve condensed feature values.
Nearest Neighbors By using a nearest neighbors classi-
er [2], we can select which of our previously analyzed pro-
grams our new program is most similar to. Learning using
nearest neighbors is simply a matter of mapping each 5-D
feature vector of our training programs (all our benchmarks)
onto a 5-D feature space.
Classiﬁcation When a novel program is compiled, it is
rst put through a feature extractor, and those features pro-
cessed by PCA. The resulting 5-D feature vector is mapped
onto the 5-D feature space, and the Euclidean distance be-
tween it and every other point in the space calculated. The
closest point is considered to be the 'nearest neighbor' and
thus the program associated with that point is the most sim-
ilar to the new program.
We can apply this process to each of our twelve bench-
marks by using leave-one-out cross-validation, where we
disallow the use as training data of the feature vector asso-
ciated with the program that is currently being evaluated,
otherwise a program would always select itself as its near-
est neighbor.Having selected a neighbor, a previously learnt
probability distribution for that selected neighbor is then
used as the model for the new program to be iteratively op-
timized.
 /  7    , 	(  ,
It is useful to know how close our learnt distribution is
to the oracle distribution for both models, IID and Markov.
Averaged across all benchmarks, the learnt distribution
achieves approximately 0/ of the performance per eval-
uation of the IID-oracle and the Markov-oracle on the TI.
On the AMD, we achieve a similar result - approximately
75 % of both oracles' performance.
7
Features
for loop is simple?
for loop is nested?
for loop is perfectly nested?
for loop has constant lower bound?
for loop has constant upper bound?
for loop has constant stride?
for loop has unit stride? number of iterations in for loop
loop step within for loop
loop nest depth
no, of array references within loop
no. of instructions in loop
no. of load instructions in loop
no. of store instructions in loop
no. of compare instructions in loop
no. of branch instructions in loop
no. of divide instructions in loop no. of noop instructions in loop
no. of call instructions in loop
no. of generic instructions in loop
no. of array instructions in loop
no. of memory copy instructions in loop
no. of other instructions in loop
no. of oat variables in loop
no. of int variables in loop
both int and oats used in loop?
loop contains an if-construct?
loop contains an if statement in for-construct?
loop contains an if-construct?
loop iterator is an array index?
all loop indices are constants?
array is accessed in a non-linear manner?
loop strides on leading array dimensions only?
loop has calls?
loop has branches?
loop has regular control ow?
Table 3. Features used
As the oracles have been shown to improve performance
and we are able to achieve a signicant percentage of
their improvement, this suggests that both learnt models
should give signicant performance improvement over ex-
isting schemes. This is evaluated in the next section.
6 Evaluation
This section evaluates our focused search approach on
two optimization spaces. The rst space is the exhaus-
tively enumerated
  
space described throughout this pa-
per. The second is a much larger space of size  
	
i.e.
transformation sequences of length 20 with each transfor-
mation selected from one of 82 possible transformations
available in SUIF 1 [10]. This was achieved using the stan-
dard leave one out cross-validation scheme i.e. learn the
IID and Markov models based on the training data from all
other programs except for the one about to be optimized or
tested
  7       	&$ &7 	  	/ -	
 	   ('	
Initially, we ran both the baseline random and GA search
algorithms for 500 program evaluations and recorded their
speedup over time on both the TI and AMD. We then ran the
same algorithms again, this time using the two learnt mod-
els: IID and Markov. This was achieved using the standard
leave one out cross-validation scheme i.e. learn the IID and
Markov models based on the training data from all other
programs except for the one about to be optimized or tested.
The results for the TI and AMD are shown in gures 6
and 7 respectively. On the TI the learnt IID based mod-
els achieve approximately twice the potential performance
of either baseline algorithm after 10 evaluations (60%/62%
vs 32%/27%) . The learnt Markov model does even better,
achieving 79% of the perfomance available after the same
number of evaluations. The baseline algorithms would need
over 40 evaluations to achieve this same performance im-
provement. On the AMD, the performance improvements
are less dramatic, yet the learnt Markov based algorithms
achieves more than twice the performance of the baseline
algorithms after 10 evaluations.
"!  7         ,	  ('	
Experiments within an exhaustively enumerated space
are useful as the performance of a search algorithm can be
evaluated relative to the absolute minima. However, in prac-
tise when we wish to search across a large range of transfor-
mations, it is infeasible to run exhaustive experiments. In-
stead we ran a random search for 1000 evaluations on each
program space as off-line training data.
This time we wish to focus on the performance achieved
in the early parts of iterative optimization. So, we ran the
baseline random search algorithm and both learnt models
for just 50 evaluations. As the genetic algorithm and ran-
dom search have the same behaviour for the rst 50 evalua-
tions, the GA was not separately evaluated
The speedups for each benchmark after 2, 5, 10 and 50
evaluations on the TI is shown in gure 7. Due to time
constraints, only those benchmarks with non-negligible
speedup on the exhaustively enumerated space are evalu-
ated. The learnt models both deliver good performance and
the random + IID learnt model achieves an average speedup
of 1.26 after just 2 evaluations. Furthermore, the random +
IID learnt model achieves a greater average performance af-
ter 5 evaluations (1.34) than the baseline random algorithm
does after 50 evaluations (1.29).
Surprisingly, the IID learnt model achieves better perfor-
mance than the Markov learnt model after 50 evaluations
1.41 vs 1.30 speedup in contrast to the results of the ex-
haustively enumerated space (see gures 6 and 7). The rea-
8
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
 1  10  100  1000
Percent of Max Improvement Available
Evaluations
32%
60%
79%
RANDOM
IID-LRN
MAR-LRN
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
 1  10  100  1000
Percent of Max Improvement Available
Evaluations
27%
62%
79%
GA
GA-IID-LRN
GA-MAR-LRN
(a) TI: Random (b) TI: GA
Figure 5. TI: Random (a) and GA (b) search versus IID-learnt and Markov-learnt. Results averaged
over all benchmarks
0%
20%
40%
60%
80%
100%
 1  10  100  1000
Percent of Max Improvement Available
Evaluations
22%
35%
51%
RAND
IID-LRN
MAR-LRN
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
 1  10  100  1000
Percent of Max Improvement Available
Evaluations
17%
28%
44%
GA
GA-IID-LRN
GA-MAR-LRN
(a) AMD: Random (b) AMD: GA
Figure 6. AMD: Random (a) and GA (b) search versus IID-learnt and Markov-learnt. Results averaged
over all benchmarks
son is that the Markov model needs a greater number of
training evaluations than the IID model to model the space
accurately. Here we have only 1000 evaluations to build a
model.
Similarly, the speedups for the AMD are shown for each
benchmark after 2, 5, 10 and 50 evaluations on in gure 8.
Again both learnt models signicantly outperform the base-
line random algorithm. In fact the random + Markov learnt
model achieves a greater average performance (1.33) after
5 evaluations than random does after 50 evaluations (1.32).
It therefore achieves this level of performance an order of
magnitude faster - the same is also true for the TI. Once
again random + IID unexpectedly outperforms random +
Markov at 50 evaluations. Thus after just 2 evaluations a
speedup of 1.27 is found on average, almost three times the
performance of the baseline algorithm.
Finally, the single sequence that gives the best perfor-
mance on average on the AMD in the small space is humc3.
This gives an average speedup of 1.11, signicantly less
than that achieved by random + Markov after just 2 eval-
uations. On the TI, there does not exist a single sequence
which gives any performance improvement on average.
Discussion The Markov predictor performs less well on
the exhaustive space due to the reduced amount of train-
ing data. This suggests that the IID model should initially
be used on a new platform when there is relatively small
amounts of training data available. Over time, once suf-
cient new data is accrued by iterative optimization, this can
be used for a second stage of learning using the Markov
9
`
TI 2 Evaluations 5 Evaluations 10 Evaluations 50 Evaluations
Benchmark R M I R M I R M I R M I
fft 1.00 1.00 1.00 1.01 1.01 1.34 1.00 1.01 1.65 1.34 1.21 1.81
r 1.18 1.66 1.67 1.25 1.66 1.83 1.37 1.66 1.85 1.70 1.85 1.85
iir 1.14 1.20 1.19 1.18 1.23 1.19 1.19 1.23 1.21 1.19 1.23 1.23
adpm 1.08 1.33 1.17 1.18 1.33 1.18 1.25 1.35 1.24 1.28 1.43 1.28
edg 1.08 1.13 1.27 1.15 1.13 1.28 1.21 1.13 1.28 1.25 1.13 1.29
lpc 1.09 1.05 1.13 1.10 1.05 1.16 1.10 1.10 1.18 1.24 1.12 1.27
spe 1.01 1.10 1.15 1.03 1.17 1.16 1.05 1.17 1.16 1.07 1.17 1.18
AVG 1.08 1.21 1.22 1.12 1.22 1.34 1.16 1.23 1.36 1.29 1.30 1.41
Figure 7. Speedups up achieved by random search (R), random + Markov learnt model (M), random +
IID learnt model (I) after 2, 5, 10 and 50 evaluations on each benchmark on the TI processor. Random
+ IID learnt model achieves greater average performance (1.34) after 5 evaluations than random does
after 50 evaluations (1.29)
model.
7 Related Work
Iterative search-based optimization As well as the work
of Almagor et al [1] and Triantafyllis et al [20] described
in the introduction, there have been a number of related
projects. A partially user-assisted approach to select opti-
misation sequences for embedded applications is described
in [11]. This approach combines user guides and perfor-
mance information with a genetic algorithm to select lo-
cal and global optimisation sequences. Other authors [9, 5]
have explored ways to search program- or domain-specic
command line parameters to enable and disable specic op-
tions of various optimising compilers. In [8] iterative high
level optimizations are applied to several embedded proces-
sors using two probabilistic algorithms. Good speedups are
obtained at the expense of very large number of evaluations.
Finally, in [?], it is shown that carefully hand generated
models can approach the performance of iterative optimi-
sation.
Machine Learning Machine learning predictive mod-
elling has been recently used for non-search based optimisa-
tion. Here the compiler attempts to learn off-line a good op-
timization heuristic which is then used instead of the com-
piler writer's hand-tuned method.
Stephenson et al. [18] used genetic programming to
tune heuristic priority functions for three compiler opti-
mizations: within the Trimaran's IMPACT compiler. For
two optimizations they achieved signicant improvements.
However, these two pre-existing heuristics were not well
implemented. Turning off data prefetching completely is
preferable and reduces many of their signicant gains. For
the third optimization, register allocation, they were only
able to achieve on average a 2% increase over the manually
tuned heuristic.
Cavazos et al. [4] describe using supervised learning to
control whether or not to apply instruction scheduling. No
absolute performance improvements were reported how-
ever.
Finally, Monsifrot et al. [15] use a classier based on
decision tree learning to determine which loops to unroll.
They looked at the performance of compiling Fortran pro-
grams from the SPEC benchmark suite using g77 for two
different architectures, an UltraSPARC and an IA64 where
there learnt scheme showed modest improvement.
8 Conclusion and Future Work
This paper develops a new methodology to speed up it-
erative compilation. It automatically focuses any search
on those areas likely to give greatest performance. It use
predictive modelling and program features to learn prof-
itable areas of the optimization space to search. Experi-
ments demonstrate that this approach is highly effective in
speeding up iterative optimization.
Currently, we have a one-off training/learning phase to
build a model which is then applied to each new program.
An obvious next step is to continuously update the learnt
model after each new program is iteratively optimized, sim-
ilar in spirit to lifelong compilation [12]. Future work will
investigate different predictive models on new spaces to fur-
ther improve the performance of search based optimization.
10
`
AMD 2 Evaluations 5 Evaluations 10 Evaluations 50 Evaluations
Benchmark R M I R M I R M I R M I
fft 1.00 1.04 1.04 1.00 1.05 1.07 1.00 1.07 1.10 1.00 1.15 1.17
r 1.22 1.33 1.46 1.28 1.44 1.51 1.37 1.44 1.54 1.48 1.55 1.94
iir 1.13 1.29 1.10 1.20 1.32 1.13 1.27 1.37 1.18 1.32 1.39 1.32
lat 1.04 1.48 1.40 1.23 1.53 1.43 1.32 1.53 1.52 1.41 1.53 1.53
lms 1.13 1.15 1.19 1.20 1.22 1.22 1.31 1.33 1.29 1.42 1.44 1.40
mul 1.05 1.54 1.85 1.26 1.89 1.88 1.48 1.89 1.90 1.69 1.92 1.93
adpcm 1.08 1.24 1.27 1.17 1.33 1.31 1.24 1.36 1.35 1.32 1.41 1.44
com 1.11 1.34 1.50 1.22 1.59 1.63 1.27 1.62 1.69 1.60 1.70 1.74
edg 1.10 1.11 1.20 1.21 1.16 1.25 1.29 1.26 1.30 1.32 1.31 1.34
his 1.08 1.27 1.16 1.21 1.31 1.29 1.28 1.32 1.33 1.33 1.33 1.36
lpc 1.00 1.00 1.05 1.00 1.02 1.09 1.00 1.04 1.13 1.06 1.09 1.23
spe 1.00 1.04 1.01 1.00 1.09 1.01 1.00 1.10 1.01 1.00 1.12 1.04
AVG 1.08 1.24 1.27 1.17 1.33 1.31 1.24 1.36 1.35 1.32 1.41 1.44
Figure 8. Speedups up achieved by random search (R), random + Markov learnt model (M), random
+ IID learnt model (I) after 2, 5, 10 and 50 evaluations on each benchmark on the AMD processor.
Random + Markov learnt model achieves greater average performance (1.33) after 5 evaluations than
random does after 50 evaluations (1.32)
References
[1] L. Almagor, K.D. Cooper, A. Grosul, T. J. Harvey, S. W.
Reeves, D. Subramanian, L. Torczon and T. Waterman: Find-
ing effective compilation sequences In LCTES 2004: 231-
239
[2] C. Bishop, Neural Networks for Pattern Recognition, OUP,
2005
[3] F. Bodin, T. Kisuki, P.M.W. Knijnenburg, M.F.P. O'Boyle,
and E. Rohou. Iterative Compilation in a Non-Linear Op-
timisation Space. Workshop on Pro®le Directed Feedback-
Compilation, PACT'98, October 1998.
[4] J. Cavazos and J. E.B. Moss, Inducing Heuristics to Decide
Whether to Schedule, In ACM PLDI, May 2004.
[5] K. Chow and Y. Wu. Feedback-directed selection and charac-
terization of compiler optimizations. In (FDDO-4), Decem-
ber 2001.
[6] K. D. Cooper, A. Grosul, T.J. Harvey, S. Reeves, D. Subra-
manian, L. Torczon, and T. Waterman. Searching for compi-
lation sequences. Rice technical report, 2005.
[7] K. D. Cooper, A. Grosul, T.J. Harvey, S. Reeves, D. Sub-
ramanian, L. Torczon, and T. Waterman. ACME: adaptive
compilation made efcient. In ACM LCTES, Chicago, IL,
2005.
[8] B. Franke and M.F.P. O'Boyle, J. Thomson and G. Fursin.
Probabilistic Source-Level Optimisation of Embedded Pro-
grams In ACM LCTES 2005.
[9] E.F. Granston and A. Holler. Automatic recommendation of
compiler options. In (FDDO-4), December 2001.
[10] M. Hall, L. Anderson, S. Amarasinghe, B. Murphy,
S.W. Liao, E. Bugnion, M. and Lam. Maximizing multi-
processor performance with the SUIF compiler. IEEE Com-
puter, 29(12), 8489, 1999
[11] P. Kulkarni, W. Zhao, H. Moon, K. Cho, D. Whalley,
J. Davidson, M. Bailey, Y. Park, and K. Gallivan. Finding ef-
fective optimization phase sequences. In ACM LCTES, June
2003.
[12] C.Lattner and V.Adve, LLVM: a compilation framework for
lifelong program analysis & transformation, In CGO, 2004.
[13] C. Lee. UTDSP benchmark suite.
http://www.eecg.toronto.edu/˜corinna/
DSP/infrastructure/UTDSP.html, 1998.
[14] S.Liao, S. Devadas, K. Keutzer, A. Tjiang and A. Wang Op-
timization Techniques for Embedded DSP Micro-processors
In DAC, 1995.
[15] A.Monsifrot, F.Bodin and R.Quiniou, A machine learning
approach to automatic production of compiler heuristics, In
International Conference on Arti®cial Intelligence: Method-
ology, Systems, Applications, 2002.
[16] K. Yotov, X.Li, G.Ren, M.Cibulskis, G. DeJong, M.Garzarn,
D.Padua, K.Pingali, P.Stodghill, and P. Wu. A Comparison
of Empirical and Model-driven Optimization. In PLDI 2003
[17] M. Saghir, P. Chow, and C. Lee. A comparison of traditional
and VLIW DSP architecture for compiled DSP applications.
In CASES '98, Washington, DC, USA, 1998.
[18] M. Stephenson, S. Amarasinghe, M. Martin and U-M.
O'Reilly Meta Optimization: Improving Compiler Heuris-
tics with Machine Learning In PLDI 2003.
[19] B. Su, J. Wang, and A. Esguerra. Source-level loop opti-
mization for DSP code generation. In Proceedings of 1999
IEEE International Conference on Acoustic, Speech and Sig-
nal Processing (ICASSP '99), volume 4, pages 21552158,
Phoenix, AZ, 1999.
[20] S. Triantafyllis, M. Vachharajani, N. Vachharajani, and D. I.
August Compiler Optimization-Space Exploration In CGO
March 2003.
11

OpenTuner: An Extensible Framework for
Program Autotuning
Jason Ansel Shoaib Kamil Kalyan Veeramachaneni
Jonathan Ragan-Kelley Jeﬀrey Bosboom Una-May
O’Reilly Saman Amarasinghe
MIT - CSAIL
August 27, 2014
1 / 30
Raytracer Example
An example ray tracer program: raytracer.cpp
$ g++ −O3 −o r a y t r a c e ra r a y t r a c e r . cpp
$ time . / r a y t r a c e ra
. / r a y t r a c e ra 0 . 1 7 s u s e r 0 . 0 0 s system 99% cpu 0.175 t o t a l
1.47x speedup with:
$ g++ −O3 −o r a y t r a c e rb apps / r a y t r a c e r . cpp−f u n s a f e−math−o p t i m i z a t i o n s−fwrapv
↪→−fno −e x p e n s i v e−o p t i m i z a t i o n s−−param=max−p e e l−b r a n c h e s =115−fweb −fno −
↪→ cx−f o r t r a n−r u l e s−−param=max−i n l i n e−r e c u r s i v e−depth=25 −fno −btr −bb−
↪→ e x c l u s i v e−fno −t r e e−ch −−param=i v−max−c o n s i d e r e d−u s e s =69−f g c s e−l a s−
↪→ f t r e e−loop −d i s t r i b u t i o n−−param=max−goto −d u p l i c a t i o n−i n s n s =11−−param=
↪→ max−h o i s t−depth=44 −fs ch ed−s t a l l e d−i n s n s−dep −−param=max−once −p e e l e d−
↪→ i n s n s =165−−param=max−p i p e l i n e−r e g i o n−i n s n s =316−−param=i v−c o n s i d e r−a l l
↪→−c a n d i d a t e s−bound=75
$ time . / r a y t r a c e rb
. / r a y t r a c e rb 0 . 1 2 s u s e r 0 . 0 0 s system 99% cpu 0.119 t o t a l
2 / 30
Raytracer Example
An example ray tracer program: raytracer.cpp
$ g++ −O3 −o r a y t r a c e ra r a y t r a c e r . cpp
$ time . / r a y t r a c e ra
. / r a y t r a c e ra 0 . 1 7 s u s e r 0 . 0 0 s system 99% cpu 0.175 t o t a l
1.47x speedup with:
$ g++ −O3 −o r a y t r a c e rb apps / r a y t r a c e r . cpp−f u n s a f e−math−o p t i m i z a t i o n s−fwrapv
↪→−fno −e x p e n s i v e−o p t i m i z a t i o n s−−param=max−p e e l−b r a n c h e s =115−fweb −fno −
↪→ cx−f o r t r a n−r u l e s−−param=max−i n l i n e−r e c u r s i v e−depth=25 −fno −btr −bb−
↪→ e x c l u s i v e−fno −t r e e−ch −−param=i v−max−c o n s i d e r e d−u s e s =69−f g c s e−l a s−
↪→ f t r e e−loop −d i s t r i b u t i o n−−param=max−goto −d u p l i c a t i o n−i n s n s =11−−param=
↪→ max−h o i s t−depth=44 −fs ch ed−s t a l l e d−i n s n s−dep −−param=max−once −p e e l e d−
↪→ i n s n s =165−−param=max−p i p e l i n e−r e g i o n−i n s n s =316−−param=i v−c o n s i d e r−a l l
↪→−c a n d i d a t e s−bound=75
$ time . / r a y t r a c e rb
. / r a y t r a c e rb 0 . 1 2 s u s e r 0 . 0 0 s system 99% cpu 0.119 t o t a l
2 / 30
Raytracer Example
An example ray tracer program: raytracer.cpp
$ g++ −O3 −o r a y t r a c e ra r a y t r a c e r . cpp
$ time . / r a y t r a c e ra
. / r a y t r a c e ra 0 . 1 7 s u s e r 0 . 0 0 s system 99% cpu 0.175 t o t a l
1.47x speedup with:
$ g++ −O3 −o r a y t r a c e rb apps / r a y t r a c e r . cpp−f u n s a f e−math−o p t i m i z a t i o n s−fwrapv
↪→−fno −e x p e n s i v e−o p t i m i z a t i o n s−−param=max−p e e l−b r a n c h e s =115−fweb −fno −
↪→ cx−f o r t r a n−r u l e s−−param=max−i n l i n e−r e c u r s i v e−depth=25 −fno −btr −bb−
↪→ e x c l u s i v e−fno −t r e e−ch −−param=i v−max−c o n s i d e r e d−u s e s =69−f g c s e−l a s−
↪→ f t r e e−loop −d i s t r i b u t i o n−−param=max−goto −d u p l i c a t i o n−i n s n s =11−−param=
↪→ max−h o i s t−depth=44 −fs ch ed−s t a l l e d−i n s n s−dep −−param=max−once −p e e l e d−
↪→ i n s n s =165−−param=max−p i p e l i n e−r e g i o n−i n s n s =316−−param=i v−c o n s i d e r−a l l
↪→−c a n d i d a t e s−bound=75
$ time . / r a y t r a c e rb
. / r a y t r a c e rb 0 . 1 2 s u s e r 0 . 0 0 s system 99% cpu 0.119 t o t a l
2 / 30
iv-consider-all-candidates-bound what???
This command is brittle and confusing:
$ g++ −O3 −o r a y t r a c e rb apps / r a y t r a c e r . cpp−f u n s a f e−math−o p t i m i z a t i o n s−fwrapv
↪→−fno −e x p e n s i v e−o p t i m i z a t i o n s−−param=max−p e e l−b r a n c h e s =115−fweb −fno −
↪→ cx−f o r t r a n−r u l e s−−param=max−i n l i n e−r e c u r s i v e−depth=25 −fno −btr −bb−
↪→ e x c l u s i v e−fno −t r e e−ch −−param=i v−max−c o n s i d e r e d−u s e s =69−f g c s e−l a s−
↪→ f t r e e−loop −d i s t r i b u t i o n−−param=max−goto −d u p l i c a t i o n−i n s n s =11−−param=
↪→ max−h o i s t−depth=44 −fs ch ed−s t a l l e d−i n s n s−dep −−param=max−once −p e e l e d−
↪→ i n s n s =165−−param=max−p i p e l i n e−r e g i o n−i n s n s =316−−param=i v−c o n s i d e r−a l l
↪→−c a n d i d a t e s−bound=75
▶ Speciﬁc to:
▶ raytracer.cpp
▶ Same ﬂags are 1 .42x slower than -O1 for fft.c
▶ GCC 4.8.2-19ubuntu1
▶ Intel Core i7-4770S
▶ Autotuners can help!
3 / 30
iv-consider-all-candidates-bound what???
This command is brittle and confusing:
$ g++ −O3 −o r a y t r a c e rb apps / r a y t r a c e r . cpp−f u n s a f e−math−o p t i m i z a t i o n s−fwrapv
↪→−fno −e x p e n s i v e−o p t i m i z a t i o n s−−param=max−p e e l−b r a n c h e s =115−fweb −fno −
↪→ cx−f o r t r a n−r u l e s−−param=max−i n l i n e−r e c u r s i v e−depth=25 −fno −btr −bb−
↪→ e x c l u s i v e−fno −t r e e−ch −−param=i v−max−c o n s i d e r e d−u s e s =69−f g c s e−l a s−
↪→ f t r e e−loop −d i s t r i b u t i o n−−param=max−goto −d u p l i c a t i o n−i n s n s =11−−param=
↪→ max−h o i s t−depth=44 −fs ch ed−s t a l l e d−i n s n s−dep −−param=max−once −p e e l e d−
↪→ i n s n s =165−−param=max−p i p e l i n e−r e g i o n−i n s n s =316−−param=i v−c o n s i d e r−a l l
↪→−c a n d i d a t e s−bound=75
▶ Speciﬁc to:
▶ raytracer.cpp
▶ Same ﬂags are 1 .42x slower than -O1 for fft.c
▶ GCC 4.8.2-19ubuntu1
▶ Intel Core i7-4770S
▶ Autotuners can help!
3 / 30
iv-consider-all-candidates-bound what???
This command is brittle and confusing:
$ g++ −O3 −o r a y t r a c e rb apps / r a y t r a c e r . cpp−f u n s a f e−math−o p t i m i z a t i o n s−fwrapv
↪→−fno −e x p e n s i v e−o p t i m i z a t i o n s−−param=max−p e e l−b r a n c h e s =115−fweb −fno −
↪→ cx−f o r t r a n−r u l e s−−param=max−i n l i n e−r e c u r s i v e−depth=25 −fno −btr −bb−
↪→ e x c l u s i v e−fno −t r e e−ch −−param=i v−max−c o n s i d e r e d−u s e s =69−f g c s e−l a s−
↪→ f t r e e−loop −d i s t r i b u t i o n−−param=max−goto −d u p l i c a t i o n−i n s n s =11−−param=
↪→ max−h o i s t−depth=44 −fs ch ed−s t a l l e d−i n s n s−dep −−param=max−once −p e e l e d−
↪→ i n s n s =165−−param=max−p i p e l i n e−r e g i o n−i n s n s =316−−param=i v−c o n s i d e r−a l l
↪→−c a n d i d a t e s−bound=75
▶ Speciﬁc to:
▶ raytracer.cpp
▶ Same ﬂags are 1 .42x slower than -O1 for fft.c
▶ GCC 4.8.2-19ubuntu1
▶ Intel Core i7-4770S
▶ Autotuners can help!
3 / 30
How to Autotune a Program
Program
4 / 30
How to Autotune a Program
Program
Search Space
Definition
Run MethodExecutes
4 / 30
How to Autotune a Program
Configuration
Program
Search Space
Definition
Run Method MeasurementExecutes
Program 
Autotuner
Machine 
Learning
Search
Technique(s)
4 / 30
How to Autotune a Program
Configuration
Program
Search Space
Definition
Run Method MeasurementExecutes
Program 
Autotuner
Machine 
Learning
Search
Technique(s)
Optimized 
Configuration
4 / 30
OpenTuner
▶ OpenTuner is an general framework for program autotuning
▶ Extensible conﬁguration representation
▶ Uses ensembles of techniques to provide robustness to diﬀerent
search spaces
▶ As an example, lets implement a GCC ﬂags autotuner with
OpenTuner
Search Space
Definition Run Method
(1) (2)
5 / 30
OpenTuner
▶ OpenTuner is an general framework for program autotuning
▶ Extensible conﬁguration representation
▶ Uses ensembles of techniques to provide robustness to diﬀerent
search spaces
▶ As an example, lets implement a GCC ﬂags autotuner with
OpenTuner
Search Space
Definition Run Method
(1) (2)
5 / 30
Deﬁne the Search Space with OpenTuner
▶ Optimization level: O0, O1, O2, O3
m a n i p u l a t o r =C o n f i g u r a t i o n M a n i p u l a t o r( )
m a n i p u l a t o r . addparameter ( I n t e g e r P a r a m e t e r( ’ o p tl e v e l ’ , 0 , 3) )
▶ On/oﬀ ﬂags, eg: ’-falign-functions’ vs
’-fno-align-functions’
GCC FLAGS = [
’ a l i g n−f u n c t i o n s ’ , ’ a l i g n−jumps ’ , ’ a l i g n−l a b e l s ’ ,
’ branch −count −r e g ’ , ’ branch−p r o b a b i l i t i e s ’ ,
# . . . (176 t o t a l )
]
f o r f l a gi n GCC FLAGS :
m a n i p u l a t o r . addparameter ( EnumParameter ( f l a g , [ ’ on ’ , ’ o f f ’ , ’ d e f a u l t ’ ] ) )
▶ Parameters, eg: ’--param early-inlining-insns=512’
# ( name , min , max)
GCC PARAMS = [
( ’ e a r l y−i n l i n i n g−i n s n s ’ , 0 , 1000) ,
( ’ gcse−cost −d i s t a n c e−r a t i o ’ , 0 , 100) ,
# . . . (145 t o t a l )
]
f o r param , min val , max val i n GCC PARAMS :
m a n i p u l a t o r . addparameter ( I n t e g e r P a r a m e t e r( param , min val , max val ) )
6 / 30
Deﬁne the Search Space with OpenTuner
▶ Optimization level: O0, O1, O2, O3
m a n i p u l a t o r =C o n f i g u r a t i o n M a n i p u l a t o r( )
m a n i p u l a t o r . addparameter ( I n t e g e r P a r a m e t e r( ’ o p tl e v e l ’ , 0 , 3) )
▶ On/oﬀ ﬂags, eg: ’-falign-functions’ vs
’-fno-align-functions’
GCC FLAGS = [
’ a l i g n−f u n c t i o n s ’ , ’ a l i g n−jumps ’ , ’ a l i g n−l a b e l s ’ ,
’ branch −count −r e g ’ , ’ branch−p r o b a b i l i t i e s ’ ,
# . . . (176 t o t a l )
]
f o r f l a gi n GCC FLAGS :
m a n i p u l a t o r . addparameter ( EnumParameter ( f l a g , [ ’ on ’ , ’ o f f ’ , ’ d e f a u l t ’ ] ) )
▶ Parameters, eg: ’--param early-inlining-insns=512’
# ( name , min , max)
GCC PARAMS = [
( ’ e a r l y−i n l i n i n g−i n s n s ’ , 0 , 1000) ,
( ’ gcse−cost −d i s t a n c e−r a t i o ’ , 0 , 100) ,
# . . . (145 t o t a l )
]
f o r param , min val , max val i n GCC PARAMS :
m a n i p u l a t o r . addparameter ( I n t e g e r P a r a m e t e r( param , min val , max val ) )
6 / 30
Deﬁne the Search Space with OpenTuner
▶ Optimization level: O0, O1, O2, O3
m a n i p u l a t o r =C o n f i g u r a t i o n M a n i p u l a t o r( )
m a n i p u l a t o r . addparameter ( I n t e g e r P a r a m e t e r( ’ o p tl e v e l ’ , 0 , 3) )
▶ On/oﬀ ﬂags, eg: ’-falign-functions’ vs
’-fno-align-functions’
GCC FLAGS = [
’ a l i g n−f u n c t i o n s ’ , ’ a l i g n−jumps ’ , ’ a l i g n−l a b e l s ’ ,
’ branch −count −r e g ’ , ’ branch−p r o b a b i l i t i e s ’ ,
# . . . (176 t o t a l )
]
f o r f l a gi n GCC FLAGS :
m a n i p u l a t o r . addparameter ( EnumParameter ( f l a g , [ ’ on ’ , ’ o f f ’ , ’ d e f a u l t ’ ] ) )
▶ Parameters, eg: ’--param early-inlining-insns=512’
# ( name , min , max)
GCC PARAMS = [
( ’ e a r l y−i n l i n i n g−i n s n s ’ , 0 , 1000) ,
( ’ gcse−cost −d i s t a n c e−r a t i o ’ , 0 , 100) ,
# . . . (145 t o t a l )
]
f o r param , min val , max val i n GCC PARAMS :
m a n i p u l a t o r . addparameter ( I n t e g e r P a r a m e t e r( param , min val , max val ) )
6 / 30
Deﬁning the Run Function
▶ Optimization level: O0, O1, O2, O3
d e f run ( s e l f , d e s i r e dr e s u l t , programinput , l i m i t ) :
c f g = d e s i r e dr e s u l t . c o n f i g u r a t i o n . data
gcc cmd = ’ g++ r a y t r a c e r . cpp−o . / tmp . b i n ’
gcc cmd += ’ −O{0}’ .format ( c f g [ ’ o p tl e v e l ’ ] )
▶ On/oﬀ ﬂags:
f o r f l a gi n GCC FLAGS :
i f c f g [ f l a g ] == ’ on ’ :
gcc cmd += ’ −f {0}’ .format ( f l a g )
e l i f c f g [ f l a g ] == ’ o f f ’ :
gcc cmd += ’ −fno −{0}’ .format ( f l a g )
▶ Parameters:
f o r param , min value , max value i n GCC PARAMS :
gcc cmd += ’ −−param {0}={1}’ .format ( param , c f g [ param ] )
▶ Measure how well it performs:
c o m p i l er e s u l t = s e l f .c a l l p r o g r a m( gcc cmd )
r u nr e s u l t = s e l f .c a l l p r o g r a m( ’ . / tmp . b i n ’ )
r e t u r n R e s u l t ( time=r u nr e s u l t [ ’ time ’ ] )
7 / 30
Deﬁning the Run Function
▶ Optimization level: O0, O1, O2, O3
d e f run ( s e l f , d e s i r e dr e s u l t , programinput , l i m i t ) :
c f g = d e s i r e dr e s u l t . c o n f i g u r a t i o n . data
gcc cmd = ’ g++ r a y t r a c e r . cpp−o . / tmp . b i n ’
gcc cmd += ’ −O{0}’ .format ( c f g [ ’ o p tl e v e l ’ ] )
▶ On/oﬀ ﬂags:
f o r f l a gi n GCC FLAGS :
i f c f g [ f l a g ] == ’ on ’ :
gcc cmd += ’ −f {0}’ .format ( f l a g )
e l i f c f g [ f l a g ] == ’ o f f ’ :
gcc cmd += ’ −fno −{0}’ .format ( f l a g )
▶ Parameters:
f o r param , min value , max value i n GCC PARAMS :
gcc cmd += ’ −−param {0}={1}’ .format ( param , c f g [ param ] )
▶ Measure how well it performs:
c o m p i l er e s u l t = s e l f .c a l l p r o g r a m( gcc cmd )
r u nr e s u l t = s e l f .c a l l p r o g r a m( ’ . / tmp . b i n ’ )
r e t u r n R e s u l t ( time=r u nr e s u l t [ ’ time ’ ] )
7 / 30
Deﬁning the Run Function
▶ Optimization level: O0, O1, O2, O3
d e f run ( s e l f , d e s i r e dr e s u l t , programinput , l i m i t ) :
c f g = d e s i r e dr e s u l t . c o n f i g u r a t i o n . data
gcc cmd = ’ g++ r a y t r a c e r . cpp−o . / tmp . b i n ’
gcc cmd += ’ −O{0}’ .format ( c f g [ ’ o p tl e v e l ’ ] )
▶ On/oﬀ ﬂags:
f o r f l a gi n GCC FLAGS :
i f c f g [ f l a g ] == ’ on ’ :
gcc cmd += ’ −f {0}’ .format ( f l a g )
e l i f c f g [ f l a g ] == ’ o f f ’ :
gcc cmd += ’ −fno −{0}’ .format ( f l a g )
▶ Parameters:
f o r param , min value , max value i n GCC PARAMS :
gcc cmd += ’ −−param {0}={1}’ .format ( param , c f g [ param ] )
▶ Measure how well it performs:
c o m p i l er e s u l t = s e l f .c a l l p r o g r a m( gcc cmd )
r u nr e s u l t = s e l f .c a l l p r o g r a m( ’ . / tmp . b i n ’ )
r e t u r n R e s u l t ( time=r u nr e s u l t [ ’ time ’ ] )
7 / 30
OpenTuner Results for GCC Flags
 0.4
 0.45
 0.5
 0.55
 0.6
 0  300  600  900  1200  1500  1800
Execution Time (seconds)
Autotuning Time (seconds)
g++ -O1
g++ -O2
g++ -O3
OpenTuner
Autotune GCC ﬂags for tsp ga.cpp. Median of 30 runs, error bars
are 20th and 80th percentiles.
8 / 30
Related Projects
A small selection of many related projects:
Package Domain Search Method
ATLAS Dense Linear AlgebraExhaustive
Code Perforation Compiler Exhaustive + Simulated Annealing
FFTW Fast Fourier TransformExhaustive / Dynamic Prog.
OSKI Sparse Linear AlgebraExhaustive + Heuristic
Active Harmony Runtime System Nelder-Mead
PATUS Stencil ComputationsNelder-Mead or Evolutionary
Sepya Stencil ComputationsRandom-Restart Gradient Ascent
Dynamic Knobs Runtime System Control Theory
Milepost GCC / cTuningCompiler IID Model + Central DB
SEEC / Heartbeats Runtime System Control Theory
Insieme Compiler Diﬀerential Evolution
PetaBricks Programming LanguageBottom-up Evolutionary
SPIRAL DSP Algorithms Pareto Active Learning
▶ Simple techniques (exhaustive, hill climbers, etc) are popular
▶ No single technique is best for all problems
▶ Representations are often just integers/ﬂoats/booleans
9 / 30
Related Projects
A small selection of many related projects:
Package Domain Search Method
ATLAS Dense Linear AlgebraExhaustive
Code Perforation Compiler Exhaustive + Simulated Annealing
FFTW Fast Fourier TransformExhaustive / Dynamic Prog.
OSKI Sparse Linear AlgebraExhaustive + Heuristic
Active Harmony Runtime System Nelder-Mead
PATUS Stencil ComputationsNelder-Mead or Evolutionary
Sepya Stencil ComputationsRandom-Restart Gradient Ascent
Dynamic Knobs Runtime System Control Theory
Milepost GCC / cTuningCompiler IID Model + Central DB
SEEC / Heartbeats Runtime System Control Theory
Insieme Compiler Diﬀerential Evolution
PetaBricks Programming LanguageBottom-up Evolutionary
SPIRAL DSP Algorithms Pareto Active Learning
▶ Simple techniques (exhaustive, hill climbers, etc) are popular
▶ No single technique is best for all problems
▶ Representations are often just integers/ﬂoats/booleans
9 / 30
Limits of Current Approaches
▶ We believe simple techniques limit the scope and eﬃciency of
autotuning
▶ A hill climber works great for a block size, but fails for more
complex applications
▶ Many users of autotuning work hard to prune their search
spaces to ﬁt techniques such as exhaustive search
▶ Real problems have large search spaces
10 / 30
Limits of Current Approaches
▶ We believe simple techniques limit the scope and eﬃciency of
autotuning
▶ A hill climber works great for a block size, but fails for more
complex applications
▶ Many users of autotuning work hard to prune their search
spaces to ﬁt techniques such as exhaustive search
▶ Real problems have large search spaces
10 / 30
Over 10806 Combinations of GCC Optimizationsg++ apps/raytracer.cpp -o ./raytracerc -O3 -fno-align-functions -fno-align-loops -fasynchronous-unwind-tables -fbranch-count-reg -fbranch-probabilities-fno-branch-target-load-optimize -fbtr-bb-exclusive -fno-combine-stack-adjustments -fno-common -fcompare-elim -fcrossjumping -fcse-follow-jumps-fcx-fortran-rules -fcx-limited-range -fdata-sections -fno-dce -fdelete-null-pointer-checks -fno-devirtualize -fno-dse -fearly-inlining -fexceptions-ffinite-math-only -fforward-propagate -fgcse -fgcse-after-reload -fno-gcse-las -fno-graphite-identity -fno-if-conversion2 -fno-inline-functions-fno-inline-small-functions -fno-ipa-cp -fno-ipa-matrix-reorg -fno-ipa-profile -fno-ipa-pta -fipa-pure-const -fipa-reference -fno-ipa-sra -fno-ivopts-fno-loop-block -fno-loop-flatten -floop-interchange -fno-loop-parallelize-all -floop-strip-mine -fmath-errno -fno-merge-all-constants -fno-modulo-sched-fno-non-call-exceptions -fno-optimize-sibling-calls -fno-optimize-strlen -fpeel-loops -fpeephole -fno-peephole2 -fno-predictive-commoning-fno-prefetch-loop-arrays -fno-reg-struct-return -fno-regmove -frename-registers -fno-reorder-blocks -freorder-blocks-and-partition -freorder-functions-fno-rerun-cse-after-loop -fno-rounding-math -fno-rtti -fno-sched-critical-path-heuristic -fno-sched-dep-count-heuristic -fno-sched-group-heuristic-fno-sched-interblock -fno-sched-pressure -fsched-rank-heuristic -fsched-spec-insn-heuristic -fsched-spec-load -fno-sched-stalled-insns -fsched-stalled-insns-dep-fno-sched2-use-superblocks -fno-schedule-insns -fschedule-insns2 -fno-sel-sched-pipelining -fno-sel-sched-pipelining-outer-loops -fsel-sched-reschedule-pipelined-fno-short-wchar -fno-shrink-wrap -fsignaling-nans -fsingle-precision-constant -fno-split-ivs-in-unroller -fstrict-enums -fno-thread-jumps-ftrapping-math -fno-trapv -fno-tree-builtin-call-dce -fno-tree-ccp -fno-tree-copy-prop -ftree-copyrename -fno-tree-cselim -fno-tree-dce -ftree-dse-fno-tree-forwprop -ftree-fre -ftree-loop-distribute-patterns -fno-tree-loop-distribution -ftree-loop-if-convert -fno-tree-loop-if-convert-stores-fno-tree-loop-ivcanon -ftree-pta -fno-tree-reassoc -fno-tree-scev-cprop -fno-tree-slp-vectorize -ftree-sra -ftree-switch-conversion -fno-tree-ter-fno-tree-vectorize -ftree-vrp -fno-unit-at-a-time -fno-unroll-all-loops -fno-unroll-loops -funsafe-loop-optimizations -funwind-tables -fno-var-tracking-fvar-tracking-assignments-toggle -fno-var-tracking-uninit -fno-vect-cost-model -fno-vpt -fweb -fwhole-program -fwrapv --param=align-loop-iterations=16--param=align-threshold=28 --param=allow-load-data-races=1 --param=allow-packed-load-data-races=1 --param=allow-packed-store-data-races=0--param=allow-store-data-races=1 --param=case-values-threshold=3 --param=comdat-sharing-probability=14 --param=cxx-max-namespaces-for-diagnostic-help=1008--param=early-inlining-insns=19 --param=gcse-after-reload-critical-fraction=15 --param=gcse-after-reload-partial-fraction=10 --param=gcse-cost-distance-ratio=14--param=gcse-unrestricted-cost=5 --param=ggc-min-expand=66 --param=ggc-min-heapsize=15449 --param=graphite-max-bbs-per-function=248 --param=graphite-max-nb-scop-params=10--param=hot-bb-count-ws-permille=271 --param=hot-bb-frequency-fraction=2357 --param=inline-min-speedup=36 --param=inline-unit-growth=26 --param=integer-share-limit=511--param=ipa-cp-eval-threshold=222 --param=ipa-cp-loop-hint-bonus=18 --param=ipa-cp-value-list-size=18 --param=ipa-max-agg-items=13 --param=ipa-sra-ptr-growth-factor=6--param=ipcp-unit-growth=3 --param=ira-loop-reserved-regs=8 --param=ira-max-conflict-table-size=261 --param=ira-max-loops-num=25 --param=iv-always-prune-cand-set-bound=17--param=iv-consider-all-candidates-bound=26 --param=iv-max-considered-uses=85 --param=l1-cache-line-size=128 --param=l1-cache-size=24 --param=l2-cache-size=356--param=large-function-growth=237 --param=large-function-insns=4444 --param=large-stack-frame=431 --param=large-stack-frame-growth=250 --param=large-unit-insns=2520--param=lim-expensive=10 --param=loop-block-tile-size=40 --param=loop-invariant-max-bbs-in-loop=2500 --param=loop-max-datarefs-for-datadeps=816--param=lto-min-partition=261 --param=lto-partitions=96 --param=max-average-unrolled-insns=22 --param=max-completely-peel-loop-nest-depth=18--param=max-completely-peel-times=31 --param=max-completely-peeled-insns=325 --param=max-crossjump-edges=30 --param=max-cse-insns=251 --param=max-cse-path-length=8--param=max-cselib-memory-locations=1202 --param=max-delay-slot-insn-search=137 --param=max-delay-slot-live-search=84 --param=max-dse-active-local-stores=1250--param=max-early-inliner-iterations=2 --param=max-fields-for-field-sensitive=0 --param=max-gcse-insertion-ratio=50 --param=max-gcse-memory=13107200--param=max-goto-duplication-insns=15 --param=max-grow-copy-bb-insns=23 --param=max-hoist-depth=101 --param=max-inline-insns-auto=43 --param=max-inline-insns-recursive=126--param=max-inline-insns-recursive-auto=135 --param=max-inline-insns-single=421 --param=max-inline-recursive-depth=24 --param=max-inline-recursive-depth-auto=28--param=max-iterations-computation-cost=24 --param=max-iterations-to-track=253 --param=max-jump-thread-duplication-stmts=21 --param=max-last-value-rtl=2794--param=max-modulo-backtrack-attempts=14 --param=max-once-peeled-insns=105 --param=max-partial-antic-length=25 --param=max-peel-branches=84--param=max-peel-times=23 --param=max-peeled-insns=25 --param=max-pending-list-length=10 --param=max-pipeline-region-blocks=44 --param=max-pipeline-region-insns=578--param=max-predicted-iterations=28 --param=max-reload-search-insns=356 --param=max-sched-extend-regions-iters=1 --param=max-sched-insn-conflict-delay=1--param=max-sched-ready-insns=101 --param=max-sched-region-blocks=15 --param=max-sched-region-insns=36 --param=max-slsr-cand-scan=12 --param=max-stores-to-sink=2--param=max-tail-merge-comparisons=24 --param=max-tail-merge-iterations=1 --param=max-tracked-strlens=351 --param=max-unroll-times=26 --param=max-unrolled-insns=570--param=max-unswitch-insns=17 --param=max-unswitch-level=11 --param=max-variable-expansions-in-unroller=0 --param=max-vartrack-expr-depth=14--param=max-vartrack-reverse-op-size=15 --param=max-vartrack-size=12500164 --param=min-crossjump-insns=18 --param=min-inline-recursive-probability=9--param=min-insn-to-prefetch-ratio=23 --param=min-spec-prob=15 --param=min-vect-loop-bound=2 --param=omega-eliminate-redundant-constraints=0--param=omega-hash-table-size=138 --param=omega-max-eqs=43 --param=omega-max-geqs=68 --param=omega-max-keys=378 --param=omega-max-vars=32--param=omega-max-wild-cards=55 --param=partial-inlining-entry-probability=68 --param=predictable-branch-outcome=0 --param=prefetch-latency=115--param=prefetch-min-insn-to-mem-ratio=2 --param=sccvn-max-alias-queries-per-access=2543 --param=sccvn-max-scc-size=2504 --param=scev-max-expr-complexity=32--param=scev-max-expr-size=45 --param=sched-mem-true-dep-cost=0 --param=sched-pressure-algorithm=1 --param=sched-spec-prob-cutoff=79 --param=sched-state-edge-prob-cutoff=2--param=selsched-insns-to-rename=6 --param=selsched-max-lookahead=14 --param=selsched-max-sched-times=1 --param=simultaneous-prefetches=9--param=sink-frequency-threshold=53 --param=slp-max-insns-in-bb=279 --param=sms-dfa-history=3 --param=sms-loop-average-count-threshold=2--param=sms-max-ii-factor=35 --param=sms-min-sc=3 --param=ssp-buffer-size=13 --param=switch-conversion-max-branch-ratio=2 --param=tm-max-aggregate-size=32--param=tracer-dynamic-coverage=66 --param=tracer-dynamic-coverage-feedback=46 --param=tracer-max-code-growth=200 --param=tracer-min-branch-probability=82--param=tracer-min-branch-probability-feedback=70 --param=tracer-min-branch-ratio=21 --param=tree-reassoc-width=2 --param=uninit-control-dep-attempts=415--param=use-canonical-types=0 --param=vect-max-version-for-alias-checks=11 --param=vect-max-version-for-alignment-checks=23
11 / 30
Large Search Spaces are a Challenge
Project Benchmark Possible Conﬁgurations
GCC/G++ Flags all 10806
Halide Blur 1025
Halide Wavelet 1032
Halide Bilateral 10176
HPL n/a 109.9
PetaBricks Poisson 103657
PetaBricks Sort 1090
PetaBricks Strassen 10188
PetaBricks TriSolve 101559
Stencil all 106.5
Unitary n/a 1021
Mario n/a 106328
12 / 30
Ensembles of Techniques
▶ OpenTuner contains many techniques such as:
▶ Diﬀerential Evolution
▶ Genetic Algorithms
▶ Greedy Mutation
▶ Multi-armed Bandit
▶ Nelder Mead
▶ Partial Swarm Optimization
▶ Pattern Search
▶ Pseudo Annealing
▶ Torczon
▶ Uses ensembles of techniques to provide robustness to
diﬀerent search spaces
13 / 30
Ensembles of Techniques in OpenTuner
Differential 
Evolution
Particle 
Swarm 
Optimization
Torczon
Hill 
Climber
14 / 30
Ensembles of Techniques in OpenTuner
Differential 
Evolution
Particle 
Swarm 
Optimization
Torczon
Hill 
Climber
Information sharing through ResultsDB
14 / 30
Ensembles of Techniques in OpenTuner
Differential 
Evolution
Particle 
Swarm 
Optimization
Torczon
Hill 
Climber
Information sharing through ResultsDB
AUC Bandit
14 / 30
Ensembles of Techniques in OpenTuner
Differential 
Evolution
Particle 
Swarm 
Optimization
Torczon
Hill 
Climber
Information sharing through ResultsDB
AUC Bandit
Which configuration should we try next?
?
14 / 30
Ensembles of Techniques in OpenTuner
Differential 
Evolution
Particle 
Swarm 
Optimization
Torczon
Hill 
Climber
Information sharing through ResultsDB
AUC Bandit
Which configuration should we try next?
33%
Exploration
33% 33%
14 / 30
Ensembles of Techniques in OpenTuner
Differential 
Evolution
Particle 
Swarm 
Optimization
Torczon
Hill 
Climber
Information sharing through ResultsDB
AUC Bandit
Which configuration should we try next?
100%
Exploitation
0% 0%
14 / 30
OpenTuner’s General Representation
▶ Large search spaces do not mean haphazard ones
▶ Choosing the right representation is critical
▶ OpenTuner allows programmers to easily express structured
search spaces
▶ Supports complex parameter types such as permutations,
schedules, mappings
▶ User deﬁned parameter types
▶ Next, a demonstration of the versatility of OpenTuner
15 / 30
OpenTuner’s General Representation
▶ Large search spaces do not mean haphazard ones
▶ Choosing the right representation is critical
▶ OpenTuner allows programmers to easily express structured
search spaces
▶ Supports complex parameter types such as permutations,
schedules, mappings
▶ User deﬁned parameter types
▶ Next, a demonstration of the versatility of OpenTuner
15 / 30
OpenTuner Can Play Super Mario Bros!
(Video 1)
1http://youtu.be/pTi_tHpj6Ow
16 / 30
OpenTuner Can Play Super Mario Bros!
▶ Only feedback is number of pixels moved to the right
▶ e.g. approximately 1500 pixels for ﬁrst pit
▶ OpenTuner doesn’t see the screen
▶ Super Mario Bros is deterministic, single run suﬃces
17 / 30
Naive Representation
▶ Bad, because most conﬁgurations make no sense.
▶ Just mashing random buttons.
▶ Doesn’t work at all (Video 2).
2http://youtu.be/nyYdq1jJQrw
18 / 30
Naive Representation
▶ Bad, because most conﬁgurations make no sense.
▶ Just mashing random buttons.
▶ Doesn’t work at all (Video 2).
2http://youtu.be/nyYdq1jJQrw
18 / 30
Better Representation
▶ Movements (list):
▶ Direction (left, right, run left, or run right)
▶ Duration (frames)
▶ Jumps (list):
▶ Start frame
▶ Duration (frames)
Choosing the right representation is critical
▶ Search space size 10 6328
▶ Winning run found in 13641 ( ≈ 104) attempts
▶ Under 5 minutes of training time
19 / 30
Better Representation
▶ Movements (list):
▶ Direction (left, right, run left, or run right)
▶ Duration (frames)
▶ Jumps (list):
▶ Start frame
▶ Duration (frames)
Choosing the right representation is critical
▶ Search space size 10 6328
▶ Winning run found in 13641 ( ≈ 104) attempts
▶ Under 5 minutes of training time
19 / 30
Better Representation
▶ Movements (list):
▶ Direction (left, right, run left, or run right)
▶ Duration (frames)
▶ Jumps (list):
▶ Start frame
▶ Duration (frames)
Choosing the right representation is critical
▶ Search space size 10 6328
▶ Winning run found in 13641 ( ≈ 104) attempts
▶ Under 5 minutes of training time
19 / 30
Super Mario Bros Results
 1000
 1500
 2000
 2500
 3000
 3500
 0  60  120  180  240  300
Pixels Moved Right (Progress)
Autotuning Time (seconds)
Win Level
OpenTuner
20 / 30
OpenTuner Generating Halide Schedules
▶ A domain speciﬁc language for image
processing and photography
▶ Used for camera pipeline in Google Glass,
HDR+ in Android, some ﬁlters in
Photoshop
▶ Separate algorithm language and
scheduling language
▶ We use OpenTuner to generate the
scheduling language
21 / 30
Simple Halide Example
Algorithm:
ImageParam i n p u t (UInt ( 1 6 ) , 2) ;
Func a ( ”a” ) , a ( ”b” ) , a ( ” c ” ) ;
Var x ( ” x ” ) , y ( ” y ” ) ;
a ( x , y ) = i n p u t ( x , y ) ;
b ( x , y ) = a ( x , y ) ;
c ( x , y ) = b ( x , y ) ;
Complex schedules:
▶ Split
▶ Reorder / reorder storage
▶ Vectorize / Parallel
▶ Compute at / compute root
OpenTuner Generated Schedule:
Var x0 , y1 , x2 , x4 , y5 ;
a . s p l i t( x , x , x0 , 4)
. s p l i t( y , y , y1 , 16)
. r e o r d e r( y1 , x0 , y , x )
. v e c t o r i z e( y1 , 4)
. compute at ( b , y ) ;
b . s p l i t( x , x , x2 , 64)
. r e o r d e r( x2 , x , y )
. r e o r d e rs t o r a g e( y , x )
. v e c t o r i z e( x2 , 8)
. compute at ( c , x4 ) ;
c . s p l i t( x , x , x4 , 8)
. s p l i t( y , y , y5 , 2)
. r e o r d e r( x4 , y5 , y , x )
. p a r a l l e l( x )
. compute root ( ) ;
22 / 30
Simple Halide Example
Algorithm:
ImageParam i n p u t (UInt ( 1 6 ) , 2) ;
Func a ( ”a” ) , a ( ”b” ) , a ( ” c ” ) ;
Var x ( ” x ” ) , y ( ” y ” ) ;
a ( x , y ) = i n p u t ( x , y ) ;
b ( x , y ) = a ( x , y ) ;
c ( x , y ) = b ( x , y ) ;
Complex schedules:
▶ Split
▶ Reorder / reorder storage
▶ Vectorize / Parallel
▶ Compute at / compute root
OpenTuner Generated Schedule:
Var x0 , y1 , x2 , x4 , y5 ;
a . s p l i t( x , x , x0 , 4)
. s p l i t( y , y , y1 , 16)
. r e o r d e r( y1 , x0 , y , x )
. v e c t o r i z e( y1 , 4)
. compute at ( b , y ) ;
b . s p l i t( x , x , x2 , 64)
. r e o r d e r( x2 , x , y )
. r e o r d e rs t o r a g e( y , x )
. v e c t o r i z e( x2 , 8)
. compute at ( c , x4 ) ;
c . s p l i t( x , x , x4 , 8)
. s p l i t( y , y , y5 , 2)
. r e o r d e r( x4 , y5 , y , x )
. p a r a l l e l( x )
. compute root ( ) ;
22 / 30
Simple Halide Example
Algorithm:
ImageParam i n p u t (UInt ( 1 6 ) , 2) ;
Func a ( ”a” ) , a ( ”b” ) , a ( ” c ” ) ;
Var x ( ” x ” ) , y ( ” y ” ) ;
a ( x , y ) = i n p u t ( x , y ) ;
b ( x , y ) = a ( x , y ) ;
c ( x , y ) = b ( x , y ) ;
Complex schedules:
▶ Split
▶ Reorder / reorder storage
▶ Vectorize / Parallel
▶ Compute at / compute root
OpenTuner Generated Schedule:
Var x0 , y1 , x2 , x4 , y5 ;
a . s p l i t( x , x , x0 , 4)
. s p l i t( y , y , y1 , 16)
. r e o r d e r( y1 , x0 , y , x )
. v e c t o r i z e( y1 , 4)
. compute at ( b , y ) ;
b . s p l i t( x , x , x2 , 64)
. r e o r d e r( x2 , x , y )
. r e o r d e rs t o r a g e( y , x )
. v e c t o r i z e( x2 , 8)
. compute at ( c , x4 ) ;
c . s p l i t( x , x , x4 , 8)
. s p l i t( y , y , y5 , 2)
. r e o r d e r( x4 , y5 , y , x )
. p a r a l l e l( x )
. compute root ( ) ;
22 / 30
Simple Halide Example
Algorithm:
ImageParam i n p u t (UInt ( 1 6 ) , 2) ;
Func a ( ”a” ) , a ( ”b” ) , a ( ” c ” ) ;
Var x ( ” x ” ) , y ( ” y ” ) ;
a ( x , y ) = i n p u t ( x , y ) ;
b ( x , y ) = a ( x , y ) ;
c ( x , y ) = b ( x , y ) ;
Complex schedules:
▶ Split
▶ Reorder / reorder storage
▶ Vectorize / Parallel
▶ Compute at / compute root
OpenTuner Generated Schedule:
Var x0 , y1 , x2 , x4 , y5 ;
a . s p l i t( x , x , x0 , 4)
. s p l i t( y , y , y1 , 16)
. r e o r d e r( y1 , x0 , y , x )
. v e c t o r i z e( y1 , 4)
. compute at ( b , y ) ;
b . s p l i t( x , x , x2 , 64)
. r e o r d e r( x2 , x , y )
. r e o r d e rs t o r a g e( y , x )
. v e c t o r i z e( x2 , 8)
. compute at ( c , x4 ) ;
c . s p l i t( x , x , x4 , 8)
. s p l i t( y , y , y5 , 2)
. r e o r d e r( x4 , y5 , y , x )
. p a r a l l e l( x )
. compute root ( ) ;
22 / 30
Simpliﬁed Schedules (Placement Only)
Schedule:
a.compute at(b, y)
b.compute at(c, x)
c.compute root()
Logical Loop Structure:
for c x:
for b x:
for b y:
for a x:
for a y:
compute a()
compute b()
for c y:
compute c()
Resulting Code:
for x:
for y:
tmp a = input[x, y]
tmp b[y] = tmp a
for y:
output[x, y] = tmp b[y]
23 / 30
Simpliﬁed Schedules (Placement Only)
Schedule:
a.compute at(b, y)
b.compute at(c, x)
c.compute root()
Logical Loop Structure:
for c x:
for b x:
for b y:
for a x:
for a y:
compute a()
compute b()
for c y:
compute c()Resulting Code:
for x:
for y:
tmp a = input[x, y]
tmp b[y] = tmp a
for y:
output[x, y] = tmp b[y]
23 / 30
Simpliﬁed Schedules (Placement Only)
Schedule:
a.compute at(b, y)
b.compute at(c, x)
c.compute root()
Logical Loop Structure:
for c x:
for b x:
for b y:
for a x:
for a y:
compute a()
compute b()
for c y:
compute c()
Resulting Code:
for x:
for y:
tmp a = input[x, y]
tmp b[y] = tmp a
for y:
output[x, y] = tmp b[y]
23 / 30
Simpliﬁed Schedules (Placement Only)
Schedule:
a.compute at(b, y)
b.compute at(c, x)
c.compute root()
Logical Loop Structure:
for c x:
for b x:
for b y:
for a x:
for a y:
compute a()
compute b()
for c y:
compute c()
Resulting Code:
for x:
for y:
tmp a = input[x, y]
tmp b[y] = tmp a
for y:
output[x, y] = tmp b[y]
23 / 30
Simpliﬁed Schedules (Placement Only)
Schedule:
a.compute at(b, y)
b.compute at(c, x)
c.compute root()
Logical Loop Structure:
for c x:
for b x:
for b y:
for a x:
for a y:
compute a()
compute b()
for c y:
compute c()
Resulting Code:
for x:
for y:
tmp a = input[x, y]
tmp b[y] = tmp a
for y:
output[x, y] = tmp b[y]
23 / 30
Simpliﬁed Schedules (Placement Only)
Schedule:
a.compute at(b, y)
b.compute at(c, x)
c.compute root()
Logical Loop Structure:
for c x:
for b x:
for b y:
for a x:
for a y:
compute a()
compute b()
for c y:
compute c()Resulting Code:
for x:
for y:
tmp a = input[x, y]
tmp b[y] = tmp a
for y:
output[x, y] = tmp b[y]
23 / 30
Naive Halide Representation
Based on Halide
scheduling language:
a.compute at(b, y)
b.compute at(c, x)
c.compute root()
▶ 8 possible placements:
▶ compute at(a, x),
▶ compute at(a, y),
▶ compute at(b, x),
▶ compute at(b, y),
▶ compute at(c, x),
▶ compute at(c, y),
▶ compute root(),
▶ inline
▶ 3 computations that must be
placed (a, b, c):
▶ 512 possible schedules
24 / 30
Naive Halide Representation
Based on Halide
scheduling language:
a.compute at(b, y)
b.compute at(c, x)
c.compute root()
▶ 8 possible placements:
▶ compute at(a, x),
▶ compute at(a, y),
▶ compute at(b, x),
▶ compute at(b, y),
▶ compute at(c, x),
▶ compute at(c, y),
▶ compute root(),
▶ inline
▶ 3 computations that must be
placed (a, b, c):
▶ 512 possible schedules
24 / 30
Naive Halide Representation
Based on Halide
scheduling language:
a.compute at(b, y)
b.compute at(c, x)
c.compute root()
▶ 8 possible placements:
▶ compute at(a, x),
▶ compute at(a, y),
▶ compute at(b, x),
▶ compute at(b, y),
▶ compute at(c, x),
▶ compute at(c, y),
▶ compute root(),
▶ inline
▶ 3 computations that must be
placed (a, b, c):
▶ 512 possible schedules
24 / 30
Naive Representation Does Not Work
▶ Naive representation works for
simple halide programs
▶ Fails completely for more complex
programs
▶ 474 of 512 schedules are invalid
▶ Callgraph orderings not respected
▶ Exponentially worse with larger
programs
▶ Poor locality
▶ Small changes move large
subtrees around
25 / 30
Naive Representation Does Not Work
▶ Naive representation works for
simple halide programs
▶ Fails completely for more complex
programs
▶ 474 of 512 schedules are invalid
▶ Callgraph orderings not respected
▶ Exponentially worse with larger
programs
▶ Poor locality
▶ Small changes move large
subtrees around
25 / 30
Better Representation
for c x:
for b x:
for b y:
for a x:
for a y:
compute a()
compute b()
for c y:
compute c()
▶ Representation based logical loop structure
▶ Loop structure can be reconstructed from token order
▶ Representation is a permutation of tokens that:
▶ Respects callgraph orderings
▶ Respects loop orderings
▶ Handling of some subtle corner cases and reorder()
discussed in the paper
26 / 30
Better Representation
for c x:
for b x:
for b y:
for a x:
for a y:
compute a()
compute b()
for c y:
compute c()
▶ Representation based logical loop structure
▶ Loop structure can be reconstructed from token order
▶ Representation is a permutation of tokens that:
▶ Respects callgraph orderings
▶ Respects loop orderings
▶ Handling of some subtle corner cases and reorder()
discussed in the paper
26 / 30
Better Representation
for c x:
for b x:
for b y:
for a x:
for a y:
compute a()
compute b()
for c y:
compute c()
▶ Representation based logical loop structure
▶ Loop structure can be reconstructed from token order
▶ Representation is a permutation of tokens that:
▶ Respects callgraph orderings
▶ Respects loop orderings
▶ Handling of some subtle corner cases and reorder()
discussed in the paper
26 / 30
Better Representation
for c x:
for b x:
for b y:
for a x:
for a y:
compute a()
compute b()
for c y:
compute c()
▶ Representation based logical loop structure
▶ Loop structure can be reconstructed from token order
▶ Representation is a permutation of tokens that:
▶ Respects callgraph orderings
▶ Respects loop orderings
▶ Handling of some subtle corner cases and reorder()
discussed in the paper
26 / 30
Halide Blur
 0
 0.005
 0.01
 0.015
 0.02
 0.025
 0.03
 0  100  200  300  400  500
Execution Time (seconds)
Autotuning Time (seconds)
Hand-optimized
OpenTuner
27 / 30
Halide Bilateral Grid
 0
 0.2
 0.4
 0.6
 0.8
 1
 1.2
 1.4
 0  5000  10000  15000
Execution Time (seconds)
Autotuning Time (seconds)
Hand-optimized
OpenTuner
28 / 30
Conclusions
▶ A lot of performance is left on the
ﬂoor due to poorly optimized
programs
▶ OpenTuner makes state of the art
machine learning accessible to all
▶ Extensible conﬁguration
representation
▶ Ensembles of techniques
▶ Conventional wisdom
underestimates the size tractable
search spaces
▶ However, choosing the right
representation is critical to
successful autouners
http://opentuner.org/
pip install opentuner
Try OpenTuner today!
29 / 30
A Final Video 3
▶ OpenTuner learning to play Super
Mario Bros
▶ Every run that achieves a high
score
▶ Runs that don’t make
improvements are skipped
▶ Run # in top left caption
▶ Thanks!
http://opentuner.org/
pip install opentuner
Try OpenTuner today!
3http://youtu.be/O5IK9f2nBsE
30 / 30

Comparative Code Structure Analysis using Deep
Learning for Performance Prediction
Tarek Ramadan, Tanzima Z. Islam, Chase Phelps
{t r297,tanzima,chaseleif}@txstate.edu
Texas State University
Nathan Pinnow, Jayaraman J. Thiagarajan
{pinnow2,jayaramanthi1}@llnl.gov
Lawrence Livermore National Laboratory
Abstract—Performance analysis has always been an af-
terthought during the application development process, focusing
on application correctness ﬁrst. The learning curve of the existing
static and dynamic analysis tools are steep, which requires
understanding low-level details to interpret the ﬁndings for
actionable optimizations. Additionally, application performance
is a function of a number of unknowns stemming from the
application-, runtime-, and interactions between the OS and
underlying hardware, making it difﬁcult to model using any deep
learning technique, especially without a large labeled dataset. In
this paper, we address both of these problems by presenting
a large corpus of a labeled dataset for the community and
take a comparative analysis approach to mitigate all unknowns
except their source code differences between different correct
implementations of the same problem. We put the power of deep
learning to the test for automatically extracting information from
the hierarchical structure of abstract syntax trees to represent
source code. This paper aims to assess the feasibility of using
purely static information (e.g., abstract syntax tree or AST) of
applications to predict performance change based on the change
in code structure. This research will enable performance-aware
application development since every version of the application
will continue to contribute to the corpora, which will enhance
the performance of the model. We evaluate several deep learning-
based representation learning techniques for source code. Our
results show that tree-based Long Short-Term Memory (LSTM)
models can leverage source code’s hierarchical structure to dis-
cover latent representations. Speciﬁcally, LSTM-based predictive
models built using a single problem and a combination of multiple
problems can correctly predict if a source code will perform
better or worse up to 84% and 73% of the time, respectively.
Index Terms—Comparative performance modeling, machine
learning, Long Short-Term Memory Networks, Deep Graph
Learning
I. I NTRODUCTION
The application development life cycle comprises a perpet-
ual loop of design-development-testing. At least one-fourth of
the life cycle for both scientiﬁc and commercial applications
is spent in rigorous testing. However, performance modeling
and analysis have mostly been an afterthought, with pro-
gram correctness taking center. Most production environments
assess program correctness through nightly regression tests
to ensure building integrity in commercial and government
research laboratories but do not consider performance. To
combat the absence of performance prediction tools during
code development, developers often spend a great deal of
time in posterior dynamic (runtime) analysis by executing
the target program and measuring the metrics of interest,
e.g., time or hardware performance counters. Even though
dynamic analysis is more thorough, this process costs hours
for data collection and analysis time and requires a human in
the loop. The cost of dynamic analysis motivates the need
for static analysis based on information available before a
code runs, such as code structure represented as an abstract
syntax tree (AST). This work takes the ﬁrst step toward
developing such static analysis tools that can eventually predict
the execution time of applications before running them by
correlating code patterns to performance problems. While the
need for performance regression tests and dynamic analysis
will exist, this effort aims at reducing the time and effort spent
by developers in dynamic analysis.
The problem of predicting the absolute execution time of
applications based on code structure alone is challenging since
the execution time is a function of many factors, including
the underlying architecture, the input parameters, and the
application’s interactions with the OS. Hence, all work in
the literature aiming to predict the absolute execution time
based on static information (e.g., code structure) alone suf-
fers from poor accuracy [20; 24]. In contrast, this paper
circumvents that problem by taking a comparative approach
to attribute changes in source code structure to performance
f(δCode) →δPerformance and subsequently assess whether a
new application will run faster or slower on the same system
for similar input. In this context, we deﬁne model accuracy
as the percentage of times the model (trained on an arbitrary
dataset) correctly predicts the label (slower or otherwise) for
a test code compared to another one, where the train and the
test datasets are disjoint. E.g., DP-vs-DFS = 82% means
that a model trained on data from several solutions to a
Dynamic Programming (DP) problem can accurately predict
the performance difference between a random pair of solutions
to the Depth First Search (DFS) problem 82% of the time.
Since we take a comparative approach, factors that impact
applications outside of code structure get nulliﬁed. The three
use cases of this research are—selecting the best algorithm to
solve a problem out of several alternative solutions, predicting
performance as a code evolves, and automatically generating
semantically similar code suggestions with expected perfor-
mance change. While the ﬁrst two use cases can directly apply
the source code embeddings generated in this research, the last
one requires combining embeddings with code generation.
To learn the correlation between δ(Code) and
δ(Performance), we propose a novel static analysis
approach that leverages a deep learning technique on Abstract
Syntax Trees (AST). By directly operating on ASTs, we gain
1
arXiv:2102.07660v2  [cs.LG]  22 Apr 2021
Tag Contest Count Min (ms) Median (ms) Max (ms) StdDev Algorithms
A 4 C 6616 86 1269 4063 445 Hashing [5]
B 230 B 6099 31 658 1872 386 Binary search and number theory [6]
C 1027 C 832 72 437 1455 344 Greedy [3]
D 914 D 612 206 534 1965 464 Data structure and number theory [4]
E 1004 C 505 3 80 137 48 Constructive algorithm
F 1006 E 599 51 214 1647 471 DFS, Graphs, and Trees [2]
G 1037 D 207 5 90 450 63 DFS, Graphs, and Trees [2]
H 489 C 5192 2 9 29 15 Dynamic programming (DP)
I 919 D 475 2 285 800 202 DFS, DP, Graphs
TABLE I: Information about the selected problems. Run times are in milliseconds. The Contest column indicates a speciﬁc problem (e.g.
C in 4 C) in a contest (e.g., 4 in 4 C) from Codeforces. The tags ( A-I) are only assigned for ease of reference.
crucial structural information about code while dispensing
variations in coding styles. To analyze the AST using deep
learning, we build upon techniques initially designed for
natural language processing, given the relevant nature between
coding language and natural language. We propose to leverage
tree-structured Long Short-Term Memory (LSTM) [34], which
automatically learns to represent information inherent to a
hierarchical data structure such as an AST to a vector form.
In this effort, we also present a curated dataset comprising
over 4M programs, annotated with execution times and mem-
ory usage on the same system, from the online programming
contest platform Codeforces [1]. We identiﬁed thousands of
submissions that have signiﬁcant variations in execution times
to build a reliable model. Our work’s biggest strength is
the models’ generalizability since they produce high-quality
predictions ( 84% accurate on a single problem and 73% on
multiple problems) for different algorithms solving different
problems. While we do not anticipate entirely dispensing
the need for performance analysts during optimizations, our
methodology can reduce the amount of time, effort, and
resources consumed in a posterior analysis by informing
developers of cases where code changes introduce inefﬁciency.
In summary, the contributions of this paper include:
• An extensive labeled dataset of programs representing 1,278
different problems and 4,313,322 correct solutions to those
problems with varying ranges of execution times. Such a
dataset will be useful for further investigating generative
deep learning techniques for automatic well-performing
code generation.
• Demonstrate the applicability of deep-learning for static
code analysis to learn relationships between code structure
changes and performance, assuming code versions run on
the same machine.
• An empirical study to make suitable recommendations for
deep learning architecture design, training data sampling,
and data augmentation to build predictive models.
• A pipeline that can be integrated into the development
phase of applications to improve prediction accuracy during
production.
• A discussion on the frontiers of generating well-performing
code based on the work presented in this paper.
II. D ATASET DESCRIPTION
This section describes the data collection process, along
with a brief discussion of the data characteristics. This dataset
will be made publicly available via Github along with the
pipeline presented in this paper.
A. Data Collection
We collected the dataset from an online platform, Code-
forces [1], that organizes programming contests regularly.
Each contest consists of a set of problems to which users
submit their solutions. The online judge system automatically
evaluates each solution for correctness using several test cases
(typically 5 to 13, though the number varies among problems)
and reports their corresponding runtime and memory usage.
The dataset contains several unique solutions to each problem
with varying runtime and memory usage characteristics, from
which a deep learning model can “learn”.
We develop a Python tool to automatically retrieve a list
of contests from the Codeforces website using their provided
API and disregard any contest that has not yet ﬁnished.
Subsequently, our data collection tool carries out an API
request for a list of submission IDs for each contest in the
retrieved list. For each problem, our tool parses all submissions
ignoring those marked by Codeforces as incorrect solutions.
Finally, our tool enters each problem set along with source
code, source language, runtime, and memory usage properties
to a database for each test case.
This process results in a total of 4,313,322 correct solu-
tions, spanning across 1,278 problems. The distribution of so-
lution times ranges from 6 problems, each having over 40,000
submissions to 600 problems, each having less than 1,000
submissions. Each problem is unique in regards to its difﬁculty
and popularity. For training, it is crucial to select problem sets
that have a sufﬁcient number of solutions and are of sufﬁcient
difﬁculty such that runtime and memory usage across solutions
show non-trivial variability. In this paper, we only focus on
submissions written in C++, and the tests are averaged to
obtain a mean runtime for each problem. However, the tool
is generic in its ability to collect all programs written in all
languages. We present the performance of the built models in
Section VI from seven groups of algorithms. Table I presents
statistics and descriptions of these nine selected problems
2
ROSE Prediction
c1
c2 AST2
AST1
Neural 
Network
Node 
Rep.
Code 
Rep.
Input Processing Deep Representation Learning Predictive ModelSource 
code
Node 
Rep.
Code 
Rep.
Fig. 1: An overview of the proposed deep-learning methodology. Deep representation learning model builds code representation that the
predictive model uses to predict if the second code is expected to be more efﬁcient (in terms of execution time).
(Tags A-I). These problems are automatically selected based
on sufﬁcient variation in execution times and more than 100
correct solutions. While not all of the algorithms in our dataset
represent scientiﬁc applications, Dynamic Programming (DP)
and Graph Traversal are two of the 13 dwarfs of scientiﬁc
applications, and shortest path algorithms are core to several
commercial ones.
B. Generating Code Pairs
As described in the previous section, we formulate the
problem of comparative performance analysis as correlating
δ(Code) for a pair of source codes with δ(performance).
This formulation takes a differential approach instead of
predicting the absolute performance of a new or a variant
of the same application, which can be intractable given that
execution time is a factor of a large number of variables.
Our pipeline automatically generates pairs of codes from
the selected problems for training a robust model to facilitate
this formulation. Since the ordering of every set of codes
could be considered as a unique pair, for N submissions,
there exists a total of N2 possible pairs. Though data-driven
approaches such as deep neural networks are typically known
to require large amounts of data, we argue that all possible
pairs are not required to train a robust model. Not all pairs
add unique information for the model to learn, and repetitive
training creates a model that has been overﬁtted. Hence, in
Section VI-D, we evaluate the data requirements to build a
robust model. The state-of-practice method for reducing the
training dataset is to use random subsets of input. Hence,
we explore the use of random subsets (of code pairs) of
varying sizes for training the models and make appropriate
recommendations. For every pair of programs, we generate
the target variable (binary) as follows: if the ﬁrst element of
the pair has a higher execution time, we label it as positive,
otherwise negative. This formulation emulates a developer
looking to determine if a new version of the program will
have a lower (improved) runtime. Our experiments also study
the impact of including two-way ordering of every pair of
codes, i.e., (a,b) and (b,a) on the model accuracy.
III. P ROPOSED APPROACH
Figure 1 shows an overview of our approach that con-
verts each source code to an AST by using the ROSE [30]
compiler and subsequently utilizes a deep neural network
to learn embeddings from ASTs that incorporate structural
information automatically. Embedding learning is an active
area of research with several algorithms such as deep graph
convolutional network [31], deep graph attention network [36],
and LSTM [33]. An adequate representation improves the
accuracy of the downstream analysis tasks (e.g., performance
prediction in our case) by capturing discriminatory information
that contributes to determining a label. These embeddings
then become the input to a classiﬁer module that implements
a fully connected neural network to perform the prediction.
While this paper mainly focuses on ASTs to build a deep
learning pipeline for static program modeling, we expect that
information gathered during compile-time, such as control and
data ﬂow graphs, may improve a model’s accuracy.
A. Problem Formulation:
Formally, let us denote the ASTs for a pair of source code
by pi and pj respectively, where every pi ∈P is a submission
pertinent to the problem P. We deﬁne a deep feature extractor
F that processes an AST to produce a latent representation
z∈Z, where Z denotes the latent space. Mathematically, this
process is expressed as F : P ↦→ Z. Since the goal is to
predict if the AST of pj is expected to have a lower execution
time than pi, we ﬁrst concatenate their features to produce
¯zij = [zi,zj]. As a result, when the dimensionality of the latent
space Z is d, the size of the concatenated feature ¯zij becomes
2 ∗d. The classiﬁer function C maps the concatenated feature
¯zij ∈ ¯Z into the target variable yij ∈Y, where the output
space Y is discrete and assumes one of the two values 0 or
1. In other words, the classiﬁer mapping can be expressed as
C : ¯Z ↦→Y.
The feature extractor F consists of two components. First,
it learns to represent each code construct of an AST ( node of a
tree) using an embedding lookup function that assigns a feature
vector to each node depending on its type (e.g., for loops
or if statements). Second, it learns representation for the
entire AST or a sub-tree using the deep learning algorithm ( z).
In this paper, we propose tree-LSTM for automatic feature
representation. Section III-B discusses our rationale in details.
Our proposed approach jointly infers the representations for
each of the nodes and subsequently for the entire AST. This
automated learning approach alleviates the need for any man-
ual feature engineering technique, which presents a nontrivial
3
Fig. 2: Proposed training strategies based on tree-structured LSTMs for processing ASTs. Arrows going from root to leaves (and vice versa)
indicate information ﬂow in the LSTM, arrows between trees indicate information ﬂow between layers.
challenge for source code. The classiﬁer is implemented as
a feed-forward network and produces the likelihood of one
version of the code being superior to the other in terms of
expected performance. This likelihood is then encoded into a
decision label using Equation ??:
pred(pi,pj) →
{
0 ti <tj; piis faster
1 ti ≥tj; pjis faster or equivalent (1)
B. Tree-Structured LSTMs for AST Modeling
Since conventional solutions such as fully connected net-
works and convolutional neural networks typically utilize
unstructured high-dimensional and image data, they can not
directly handle ASTs. Consequently, there has been a recent
surge in deep learning techniques designed speciﬁcally for
these challenging data types. Tree LSTM [34] is a deep
learning architecture for processing tree-structured data. Re-
cent work has demonstrated that leveraging the hierarchical
structures of languages, both natural and programming, gives
models salient characteristics of the data and improves perfor-
mance on the downstream tasks [12; 34]. Consequently, we
propose using tree-structured LSTM, a speciﬁc construction
of recurrent neural networks, to produce concise vector repre-
sentations for each source code.
Intuitively, our model uses hierarchical accumulation to
encode each non-terminal node’s representation by aggregating
the hidden states of all of its descendants. The accumulation
process occurs in two stages. First, the model induces the value
states of non-terminals with hierarchical embeddings, which
helps the model become aware of the hierarchical and sibling
relationships between the nodes. Second, the model performs
an upward cumulative-sum operation on each target node (the
node whose representation is being learned), accumulating all
elements in the branches originating from the target node to its
descendant leaves. In Section VI, we thoroughly evaluate the
performance of tree-LSTM with Graph Convolution Network
(GCN) based deep representation learning models suitable for
structured graph data. We demonstrate that the accuracy of the
predictions are higher with embeddings built using tree-LSTM
compared to that of GCN.
Broadly, tree-LSTM is a recurrent neural network
(RNN) [37], designed to perform feature extraction from
arbitrary length sequence data via the recursive application
of a transition function on a hidden state vector. It operates
by taking in the hidden state of the previous element of the
sequence and the input for its current element. At each time
step t, the hidden state vector ht is a function of the input
vector xt (vector representation of the tth element in the
sequence) and its previous hidden state ht−1. Consequently,
the ht can be interpreted as a concise representation of the
sequence of elements observed up to time t. In a typical RNN,
the transition function is implemented as follows:
ht = tanh Wxt + Uht−1 + b, (2)
where W, U, and bare learnable parameters, and tanh denotes
the hyperbolic tangent nonlinearity. An inherent limitation of
RNNs is that as the sequence length grows the problem of
exploding or vanishing gradients makes training very difﬁ-
cult [26]. Consequently, the LSTM architecture addresses this
limitation in learning long-term dependencies by introducing
a memory cell that preserves the states over long periods of
time [13; 33]. For a time step t, an LSTM unit is typically
comprised of vectors from an input gate it, forget gate ft,
output gate ot, memory cell state ct and hidden state ht.
Intuitively, the forget gate controls the extent to which the
memory cell’s previous states are forgotten, the input gate
controls how much each unit is updated, and the output
gate controls the exposure of the internal memory state. In
a nutshell, the hidden state vector in an LSTM unit is a gated,
partial view of the internal memory cell state. Mathematically,
the transition equations can be derived as follows:
it = σ(Wixt + Uiht−1 + bi),
ft = σ(Wf xt + Uf ht−1 + bf ),
ot = σ(Woxt + Uoht−1 + bo),
ut = σ(Wuxt + Uuht−1 + bu),
ct = it ⊙ut + ft ⊙ct−1,
ht = ot ⊙tanh(ct). (3)
A limitation of this architecture is that it allows only strictly
sequential information propagation. However, in the case of
ASTs, the information ﬂow happens to multiple children from
a given parent node. Hence, we propose a new architecture
for the tree-LSTM technique to deal with information ﬂow
through an AST [34]. The crucial difference between an LSTM
unit and a tree-LSTM unit is that the gating vectors and
memory cell updates are dependent on the states of possibly
4
many child units. Additionally, instead of a single forget gate,
the tree-LSTM unit contains one forget gate for each child,
which allows it to selectively leverage information from each
child. Section IV-B describes the input vector at each node.
Figure 2(a) shows that the transition function is applied to the
leaf nodes ﬁrst and then progressively moves up the tree to the
root node. Mathematically, this can be described as follows:
Assuming that the C(j) denotes the set of children of a node
j,
˜hj =
∑
k∈C(j)
hk,
ij = σ(Wixj + Ui˜hj + bi),
fjk = σ(Wf xj + Uf hk + bf ),
oj = σ(Woxj + Uo˜hj + bo),
uj = σ(Wuxj + Uu˜hj + bu),
cj = ij ⊙uj +
∑
k∈C(j)
fjk ⊙ck,
hj = oj ⊙tanh(cj). (4)
Note that, after processing the entire AST (or a selected sub-
tree), the classiﬁer function uses the ﬁnal hidden representation
at the root node (of the sub-tree) for prediction.
IV. M ETHODS
As described in the previous section, our goal is to jointly
infer node representations (depending on the node type) and
AST representations, with the overall objective of predicting
performance. To achieve this goal, we need to address the
following tasks: (1) generate ASTs, (2) represent nodes of an
AST as a vector (encoding lookup), and (3) combine node
embeddings for representing source code (AST modeling). In
this section, we describe how we address each of the tasks
in detail, along with implementation speciﬁcs. We follow the
notations introduced in the previous section.
A. AST Generation
The ﬁrst step towards applying deep neural networks to
codes is to create an appropriate representation. In general,
the code can be treated as a text excerpt and processed with
standard language modeling tools such as word or document
embeddings. However, we advocate using abstract syntax trees
since they are better descriptors of a code structure. This
transformation leads to an additional challenge that the neural
network should leverage the inherent tree structure of ASTs.
To generate the ASTs, we use the ROSE [30] compiler in-
frastructure. ROSE is a ﬂexible, portable, and scalable source-
to-source compiler infrastructure widely used in the scientiﬁc
community comprising the national laboratories, universities,
and the industry. The AST from ROSE is modiﬁed only to
include internal nodes that are part of the source code’s func-
tion deﬁnitions. This process removes irrelevant information
from the tree and allows models to train faster. For simplicity,
the source code’s function deﬁnitions are all set as children of
a root node. While these simpliﬁcations make the embedding
learning process simpler, more ﬁne-grained representation of
the tree nodes could provide additional information for the
model to exploit. Finally, the AST generation process outputs
a list of the node IDs and a list of links between nodes to
represent the tree.
B. Constructing Node Embeddings
We assign a unique ID to each type of internal node (e.g.,
for, while), consistent across all trees in the database. A
node type gets the same ID even when they appear multiple
times in the same tree. Following standard practices in the nat-
ural language processing area, our machine learning pipeline
transforms the user-deﬁned tokens (IDs in ASTs) into vector
representations. A naive approach for constructing a vector
representation from an AST is using one-hot encoding, where
each ID can be assigned a 1−sparse binary vector with the
value 1 at the location of the ID and 0 elsewhere. Since such
a vector’s size is a function of the total number of unique
IDs in the dataset, it is high-dimensional (often referred to
as cursed representations). Such representations can lead to
severe overﬁtting when building predictive models.
Hence, we investigate a different approach in this work
that assigns a speciﬁc vector representation for each ID using
an embedding lookup structure. We ﬁx the dimensionality of
the embedding at λ, and initialize the embeddings randomly.
However, the neural network training process can subsequently
tune the embeddings. The total number of parameters to
be optimized in this step is λ ×D, where D is the total
number of unique IDs in our dataset. This embedding layer
will allow the model to infer similarities between nodes in
terms of their performance impact, much more effectively than
simple one-hot encoding. Once encoded, these embeddings are
passed along to train a model for generating representations
for the entire tree. In this paper, we initialize using random
embeddings. In the future, we will investigate using pre-
trained embeddings by adapting word embedding techniques
(e.g., Skip-gram [21], GloVe [27]) from the Natural Language
Processing (NLP) literature.
C. Training Models
Following the state-of-practice in neural networks, we ad-
vocate using multiple layers in our tree-LSTM architecture. In
this design, the hidden states at the end of one layer are used
as the next layer’s node representations. This process typically
leads to greater reﬁnement of each sub-tree’s representations
as each layer provides a better representation of the tree
structure. In addition to this native implementation, which
we refer to as uni-directional tree-LSTM, we also consider
two variants: bi-directional tree-LSTM, and alternating tree-
LSTM. In the ﬁrst variant, we allowed the tree transition to
be bi-directional, i.e., from root to leaf nodes and leaf to root
nodes. As illustrated in Figure 2(b), two different tree-LSTMs
run independently, with one having hidden states going from
child to parent and the other going from parent to child.
The parent node copies its representation to all its children
instead of just sending its representation to a single node.
This information propagation pattern enables a node’s hidden
state to include information from its children and its ancestors.
Finally, the two representations are concatenated together to
form a uniﬁed representation for a node. Since our approach
uses the ﬁnal root node representation to make the prediction,
5
the downward pass in the bi-directional training’s ﬁnal layer
is not required. In the second variant of the tree-LSTM,
bi-directional training is simpliﬁed by alternating forward
and backward passes. As shown in Figure 2(c), a 3−layer
tree-LSTM will be constructed with 2 forward layers and 1
backward layer between them. Compared to the bi-directional
training, this contains only half the number of parameters
to train, avoids overﬁtting in practice, and produces highly
effective latent representations for ASTs. In our experiments,
we ﬁnd that the alternating tree-LSTM consistently produces
the best performance in all cases.
D. Classiﬁer Design
Since our approach’s overall objective is to perform compar-
ative analysis, we ﬁrst concatenate the hidden representations
for the two ASTs and subsequently pass it to a fully connected
classiﬁer with the sigmoid activation. This classiﬁer’s number
of parameters is 2 ∗d, where d is the size of the latent
representation in our tree-LSTM. We then compute the binary
cross-entropy loss between the predicted probabilities and the
correct labels to optimize the network’s unknown parameters.
V. E XPERIMENTAL SETUP
This paper evaluates our proposed methodology using nine
individual problems (presented in Table I), and a combined
dataset comprising 100 submissions picked randomly from
100 different problems (referred to as MP in Figure 3). We
calculate the accuracy of a model based on the percentage
of times the model correctly classiﬁes a code as slower or
otherwise compared to another one.
A. System
We run our experiments on the Google Cloud Platform
(GCP). Speciﬁcally, we use a machine with eight virtual CPUs,
30 GB RAM, and one NVIDIA Tesla P100 GPU with 16GB
memory. Each CPU consists of 2.30GHz Intel (R) CPU with
4 cores and is based on the x86 64 Architecture. The GPUs
are equipped with CUDA 10.0 toolkit by NVIDIA.
B. Deep Representation Learning Techniques
In this paper, we evaluate the efﬁcacy of our proposed
tree-LSTM (described in Section III-B) based representation
learning technique compared to that of a more generic one—
Graph Convolution Network (GCN) [16; 31]. The GCN is a
generalization of Convolution Neural Networks (CNN) [15],
where the GCN stacks multiple graph convolution layers to ex-
tract a high-level node representation that takes N-dimensional
data to graph data. A GCN model takes the graph-structured
data as input and generates a vector representing a source-
code. The GCN applies semi-supervised node classiﬁcation,
which classiﬁes each node of the tree to help decide the type
for the whole AST. We extend the GCN model by creating
a wrapper layer that combines information from an internal
node’s directly connected nodes. The signiﬁcant difference
between GCN and tree-LSTM is in information ﬂow to each
internal node–GCN leverages all neighboring nodes compared
to tree-LSTM which leverages knowledge of parent-child
relationships. The source-code embeddings are then passed
from GCN to the classiﬁer, as described in Section IV-D.
A B C D E F G H I MP0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Tree-LSTM GCN
Training Dataset
Normalized Accuracy
Fig. 3: The overall model evaluation and generalizability of our pro-
posed tree-LSTM approach compared to a traditional GCN method.
The X-axis shows the training dataset, and the Y-axis shows the
accuracy. The lines show the accuracy of models in classifying
the performance differences (lower or otherwise) between random
pairs of disjoint submissions for the same problem (as the training
problem). The boxplots show models’ accuracy in classifying the
performance difference between random pairs of disjoint submissions
from all others problems except the training one.
C. Hyper-parameter Tuning
For automated hyper-parameter tuning, we leverage the
Optuna optimization framework [7]. For GCN, we observe that
the most critical parameters to tune for the given downstream
prediction task are the number of convolution layers and the
hidden layer’s size. We vary the number of convolutional
layers from 1 to 16 for GCN (more than that exhausts GPU
memory) and the hidden layer’s size from 8 to 256. Our
experiments show that (6, 117) as the number of convolutional
layers and the hidden layer’s size, respectively achieves the
best accuracy ( 68.5%). For tree-LSTM, we use 100 hidden
states and the feature embedding vectors of length 120. With
these parameters, we achieve the best accuracy ( 73%) for the
large dataset.
VI. R ESULTS
In this section, we present a detailed evaluation of the
proposed approach. The details about the dataset can be found
in Section II. We used a subset of the submissions (disjoint
from training dataset) to demonstrate results. In this study, we
evaluate the following: (a) the effectiveness of our proposed
tree-LSTM architecture compared to GCN in building source-
code representations and the generalizability of the predictive
models built on them; (b) the architectural choices for the
best model; (c) the design choices for data sampling and
augmentation; and (d) the prediction sensitivity to runtime
variation.
A. Model Evaluation and Generalization
This experiment’s objective is two-fold–(a) test the effec-
tiveness of our proposed tree-LSTM architecture compared to
GCN in building source-code representations and their impact
on model accuracy, and (b) determine how well the predictive
model can generalize to unseen problems. For this experi-
6
ment, we train models on submissions from a problem and
measure the accuracy of the model in predicting the change
in performance (positive or negative) for unseen submissions
from (i) the same problem (disjoint set), (ii) different problem
but the same algorithmic group, and (iii) other problems from
diverse algorithmic groups. Figure 3 shows training datasets
along the X-axis. The boxplots along the Y-axis show models’
accuracy in classifying the performance difference between
random pairs of disjoint submissions from diverse problems.
The line plots show the accuracy of pi vs pi, where the training
and testing datasets are disjoint submissions of the problem pi.
Generalization: From Figure 3, we can observe that a model
built for a speciﬁc problem predicts the label of a disjoint
set of submissions from (i) the same problem with up to
81% accuracy (line chart for training set E); (ii) different
problems from different algorithmic groups with up to 80%
accuracy (boxplot for the problem E). Further investigation
shows that the highest accuracy is incurred when the con-
structive algorithm to solve problem E classiﬁes random pairs
of submissions from a DFS problem G. Also, a model built
on the same problem can accurately classify the difference in
execution times between random pairs of disjoint submissions
from other problems from the same algorithmic group with up
to 82% accuracy. E.g., model built on DFS problemF classiﬁes
submissions from another DFS-based problem G with up to
82% accuracy. These observations show that the model is
learning problem characteristics and not just remembering
programming constructs.
To test our approach’s generalizability, we build a model
using a large dataset by randomly selecting 100 submissions
from 100 different problems with sufﬁcient variation in exe-
cution times and more than 1000 correct solutions. We denote
this dataset as MP (for multiple problems). We then evaluate
a model trained on MP to predict performance differences of
submissions from both A-I and these 100 problems (disjoint
test set). Figure 3 shows that the predictive model trained on
MP can accurately classify performance difference between
random pairs of solutions with up to 84% accuracy for
A-I (boxplot) and 73% accuracy for a disjoint set of MP
submissions (line chart).
Tree-LSTM vs GCN: From Figure 3 we can observe that
the prediction task with the tree-LSTM-based embeddings
consistently outperforms that built using the GCN model. The
tree-LSTM based representations capture crucial hierarchical
information about code structure that a generic graph-based
model fails to do, which explains GCN’s poor performance.
B. ROC vs Accuracy Metric
In addition to the accuracy metric, we evaluate all our
models based on the receiver operator curve (ROC) to study
how the prediction task’s performance varies as the conﬁdence
threshold changes. The conﬁdence threshold of the models’
output (probability) determines if the performance difference
between two code pairs should be classiﬁed as positive or
negative. Increasing the conﬁdence threshold lowers the false
positive rate and equivalently the true positive rate. Having a
lower false positive means that if a model classiﬁes the change
F G I
F .80 .72 .67
G .82 .76 .68
I .76 .67 .77
TABLE II: Models trained and evaluated on different problems
in similar algorithm groups (DFS and Graphs). Rows indicate the
training dataset, and columns display the test set. While problems F
and G share the same algorithmic classes (DFS, Graphs, and Trees),
the problem I has a partial overlap (DFS, DP, Graphs). This result
indicates that a more considerable overlap in problem characteristics
will result in higher prediction accuracy.
Layers Uni-Directional Bi-Directional
1 0.773 0.769
0.780 0.78
2 0.765 0.767
0.789 0.786
3 0.766 0.77
0.783 0.767
Alternating layers
0.77 (A) and 0.804 (C)
TABLE III: Prediction performance of the proposed approach
problem set A and C . We report the results obtained using different
architectural choices.
in execution time as increasing, the application developer
can conﬁdently invest the time and effort to resolve coding
inefﬁciencies (perhaps by using other tools). E.g., in Figure 4,
we can observe that the ROC for the problemA, obtained using
the 3−layer alternating tree-LSTM architecture, achieves a
high area under the ROC metric of 0.85, which implies 85%
true positive rate. Since this measure agrees with the accuracy
metric, we only report the experiments’ accuracy scores.
C. Impact of Architectural Choice
This experiment’s objective is to evaluate the impact of the
different architectural choices for the tree-LSTM (the best
representation learning model for source-code, as found in
Section VI-A) on the overall prediction accuracy. Table III
presents the impact of three architectural choices on the
accuracy of the prediction task. We increase the number of
layers from 1 to 3 for the uni- and bi-directional architectures
and observe an insigniﬁcant change in the accuracy. The bi-
directional architecture is signiﬁcantly more complicated and
takes much longer to train since information is combined from
both the forward and backward passes at every layer. The lack
of improvement in model accuracy indicates overﬁtting due to
the arbitrary increase in model complexity. The alternating
architecture produces an equivalent representation compared
to the other architectures, e.g., the alternating architecture
improves the downstream predictive task’s performance by 2%
for problem C. However, the alternating architecture combines
information once during the forward pass followed by the
backward pass, thereby gathering more information than a uni-
directional one. In comparison, its accuracy is similar to that
of the bi-directional architecture while being faster to train.
D. Impact of Data Sampling and Augmentation
A crucial aspect of modern machine learning methods is the
complex trade-off between the task complexity and the amount
7
Fig. 4: ROC curve on the validation set obtained using the multi-layer
alternating Tree-LSTM architecture on problem set A .
0 1500 2000 4000500 1000 2500 35003000 0 40 10020 60 80
Percentage of Pairs used for TrainingNumber of Submissions in Train Set
Test Accuracy
0.66
0.68
0.70
0.72
0.74
0.76
0.64
0.73
0.74
0.75
0.76
0.77
0.72
Test Accuracy
(a) (b)
Fig. 5: (a) Accuracy of the model based on a percentage of maximum
pairs for 2048 submissions. (b) Accuracy changes based on training
set characteristics.
of training data required to produce reliable models. Hence, we
study the impact of data sampling on our approach’s observed
performance. We vary the number of submissions, with a ﬁxed
ratio of pairs for every case and the number of pairs for a given
number of submissions.
Impact of number of submissions during training: First,
we increase the number of submissions in the training set from
32 to 4096 by powers of 2. For each case, we construct the
training pairs by selecting a random 75% of all possible pairs
in that case. For all tests, we use the same test set of submis-
sions. Figure 5(a) shows results for the problem set A with
the multi-layer alternating tree-LSTM architecture. Figure 5(a)
shows that the accuracy steadily improves as the number of
submissions grows. However, beyond 1000 submissions, there
is a diminishing return. Since data collection and annotation
are time-consuming, having a reasonable number of training
samples improves this methodology’s adaptability in practice.
Impact of the percentage of pairs during training: We also
investigate the question of how many pairs should be included
from those submissions for training. We perform this study by
increasing the percentage of pairs used for a ﬁxed number of
submissions. In particular, we select the number of submis-
sions at 2048 and vary the ratio of pairs used. Interestingly,
Figure 5(b) shows that the accuracy initially improves rapidly
as the number of pairs (randomly chosen) increases, achieving
an accuracy improvement of 10%, however, the accuracy score
then begins to dip. However, there is a dip in the accuracy
score. We continue to include more pairs since complex
models such as deep networks tend to overﬁt when the training
Test Accuracy
0 500 1000 1500 2000 0 400 1000 1200 1600200 800 1400 0 400 1000 1200 1600200 800 1400
0.800
0.825
0.850
0.875
0.900
0.925
0.950
0.975
1.000
0.820
0.840
0.860
0.880
0.900
0.920
0.940
0.960
0.980
0.750
0.800
0.850
0.900
0.950
1.000
Minimum Runtime Difference to Distinguish (ms)
(a) Problem A (b) Problem B (c) Problem C
Fig. 6: Studying the sensitivity of the proposed approach.
-20 0 30-10 10 20-100 0 100-50 50
-100
-50
0
50
100
150
-150
-10
-5
0
5
10
15
-15
(a) Node embeddings (b) Code embeddings
Fig. 7: Visualization of the learned representations of nodes and ASTs
obtained using t-SNE. (a) Two-Dimensional Representation of the
node embeddings. Green are operations, red are other expressions,
blue are statements, yellow are literal values, and black are support
nodes. (b) Two-Dimensional Representation of the AST latent repre-
sentations. Each color corresponds to one of the problem sets.
data’s complexity is high. This observation motivates the need
for further investigation of sampling strategies for optimal
performance.
Impact of the ordering of pairs during training: We
evaluate how important it is for the model to train on both
orderings of a single pair ( (a,b) and (b,a)). We compare
a training set containing only one ordering pair to a model
trained on the same number of overall teams, with half
being the others’ reverse. We ﬁnd that the accuracy improves
marginally, up to 2% from using symmetrical pairs as opposed
to non-symmetrical ones (the ﬁgure is not included due to
space limitation).
E. Prediction Sensitivity
When comparing the execution times, the small differences
are less signiﬁcant than the large differences, e.g., a 1-
millisecond difference is treated the same as that with 4000.
To evaluate how sensitive the model predictions are to the
variation in the submissions’ execution times, we sort the
evaluation sets and record accuracy for pairs with a difference
beyond a certain threshold.
Figure 6 shows the results for models trained on problems
A, B, and C. With these three problems, we can observe that
the accuracy of the prediction task consistently improves with
the increase in the minimum difference that the model needs
to resolve. Further investigation uncovers that a massive differ-
ence in execution time for source code typically comes from
having either loop constructs (e.g., for, while) or signiﬁcantly
longer code. Hence, it becomes easier for the model to spot
discriminatory structure in the source code when source code
variants differ signiﬁcantly in execution times. The execution
times of all the problems in this dataset are reasonably close.
These results indicate that as we move toward problems with a
more signiﬁcant difference in execution time between versions,
the model will perform better.
8
F . Visualizing Learned Representations
We initialize nodes with random embedding vectors, and the
model subsequently learns representations of each node from
the data. To evaluate the effectiveness of the learning process,
we map the embeddings down from the λ= 120 dimensional
space to a two-dimensional space and plot in Figure 7. For
the low-dimensional projection, we leverage the unsupervised,
non-linear t-Distributed Stochastic Neighbor Embedding (t-
SNE) technique [18] that is primarily used for data exploration
and visualizing high-dimensional data.
Node representations: Near nodes in t-SNE have similar
representations, nodes with equal value along one axis have
some similarities, and unequal nodes separated across both
axes are signiﬁcantly different, hence they should have dif-
fering representations. Figure 7(a) shows that the tree-LSTM
model discovers that the string and char literal representations
are closely related. The model learns that plus plus and
plus assign operators are close in nature, hence group
them closely. Since the for and while representations share
similar values along a single axis, it indicates that the model
can encode their similarities into the representations while still
capturing their differences.
Code representations: Similarly, we use our model to
generate code embeddings for three different problems with
100 submissions each and project those to two-dimensional
representations (red, blue, and green). In Figure 7(b), we
observe that the model can create distinctly different represen-
tations from other problems. We also observe that problems
represented with red and green often have grouped clusters
indicating more similarity than to the problem in blue.
VII. D ISCUSSIONS
Figure 3 shows that the predictive model based on tree-
LSTM representations can achieve up to 84% accuracy for a
model built on an individual problem E and 73% accuracy
for model built using a mixture of submissions from hundreds
of problems. While the former shows the applicability of the
proposed method in studying performance evolution of the
same problem over time, the latter demonstrates that in saving
time and effort for application developers a majority of the
time. Once deployed in a continuous learning environment,
this framework can improve the model’s accuracy by adding
new observations based on the nightly tests performed in most
production-based software development environments.
Section VI-E discusses that it is easier for the model to
resolve a higher threshold (large increase in execution time to
be considered poor performance) than a smaller one. A devel-
oper can select our model through a threshold to prefer true
positive cases over false negatives (or vice versa). E.g., with a
runtime difference in 1 second between two applications, the
model’s accuracy reaches close to 100%. Also, we observe that
even though the random initial embeddings are useful, using
other sophisticated methods such as Deepwalk [28] to generate
initial node representations may help the models to train faster
and perform better. Further, these node representations could
be made transferable between tasks in analyzing an AST.
Additionally, adding more information, such as the data ﬂow
graph, can help the model learn to distinguish problems at a
higher level.
VIII. R ELATED WORK
The area of software engineering has seen a surge in
applying LSTM and its variants in generating code, code sum-
mation using natural language [32], bug detection [11], bug
report classiﬁcation [29], author attribution [8], source code
retrieval [35], and source code defect prediction [11]. Machine
learning-based program analysis has been studied long in the
literature [9; 17]. Hindle et al. [14] compare programming
languages to natural languages and conclude that programs
have rich statistical properties. These properties are difﬁcult
for a human to capture, but they justify using learning-based
approaches to analyze programs. Others have applied deep
learning for performance analysis, [10] used sequential LSTMs
on normalized source code to predict optimization heuristics.
Related work such as [22; 23] applied a convolutional neural
network over the AST to classify the type of problem the
source code is trying to solve. [34] performed sentiment
analysis for natural language processing using a single layer
and directional tree LSTM to build up the representation of
the whole tree based upon each node’s children, with the
leaves being the words in the sentence. Others have looked
at using AST for performing static performance analysis [19].
While using similar methods, including upward and downward
passes, their approaches did not include an in-depth learning
analysis of the AST in the context of comparative performance
prediction. Neamtiu et al. [25] have applied deep learning
to the program structure information to classify programs.
In contrast, our approach, albeit similar, solves a different
problem of attributing the change in performance to code
structure changes.
IX. C ONCLUSION
This work aims at designing a generic and accurate method
for performance modeling that can provide insight into how a
change in the program impacts the performance (positive or
negative). We implement a tree-structured deep-neural network
architecture to represent the code as a vector to perform the
comparison. We then compare two such vector representations
from two applications using a feed-forward neural network.
This framework can inform a developer about how changes
made to their code will affect the performance without nec-
essarily needing to be trained on the speciﬁc problem. We
trained and evaluated our models on both individual problems
and a combined dataset of hundreds of them. Such information
available during coding can signiﬁcantly reduce the need for
running applications every time a code change is made and
assist in performance-aware application development.
REFERENCES
[1] Codeforces: Online Programming Competition. https://
Codeforces.com.
[2] Contest 1027C – Military Problem. http://codeforces.
com/problemset/problem/21006/E.
[3] Contest 1027C – Minimum Value Rectangle. http:
//codeforces.com/problemset/problem/21027/C.
9
[4] Contest 914D – Bash and a Tough Math Puzzle. http:
//codeforces.com/problemset/problem/2914/D.
[5] Contest 4C – Registration. http://codeforces.com/
problemset/problem/24/C.
[6] Contest 230B – TPrime. http://codeforces.com/
problemset/problem/230/B.
[7] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru
Ohta, and Masanori Koyama. Optuna: A next-generation
hyperparameter optimization framework. In Proceedings
of the 25th ACM SIGKDD International Conference on
Knowledge Discovery & Data Mining, pages 2623–2631,
2019.
[8] Bander Alsulami, Edwin Dauber, Richard Harang, Spiros
Mancoridis, and Rachel Greenstadt. Source code au-
thorship attribution using long short-term memory based
networks. In European Symposium on Research in
Computer Security, pages 65–82. Springer, 2017.
[9] Kyle R Canavera, Naeem Esfahani, and Sam Malek.
Mining the execution history of a software system to
infer the best time for its adaptation. In Proceedings
of the ACM SIGSOFT 20th International Symposium on
the Foundations of Software Engineering, page 18. ACM,
2012.
[10] Chris Cummins, Pavlos Petoumenos, Zheng Wang, and
Hugh Leather. End-to-end deep learning of optimization
heuristics. 2017 26th International Conference on Par-
allel Architectures and Compilation Techniques (PACT) ,
pages 219–232, 2017.
[11] Hoa Khanh Dam, Trang Pham, Shien Wee Ng, Truyen
Tran, John Grundy, Aditya Ghose, Taeksu Kim, and
Chul-Joo Kim. A deep tree-based model for software
defect prediction, 2018.
[12] Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa
Tsuruoka. Tree-to-sequence attentional neural machine
translation. arXiv preprint arXiv:1603.06075 , 2016.
[13] Alex Graves and J ¨urgen Schmidhuber. Framewise
phoneme classiﬁcation with bidirectional lstm and other
neural network architectures. Neural networks, 18(5-6):
602–610, 2005.
[14] Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel,
and Premkumar Devanbu. On the naturalness of soft-
ware. In 2012 34th International Conference on Software
Engineering (ICSE), pages 837–847. IEEE, 2012.
[15] Nal Kalchbrenner, Edward Grefenstette, and Phil Blun-
som. A convolutional neural network for modelling
sentences. arXiv preprint arXiv:1404.2188 , 2014.
[16] Thomas N Kipf and Max Welling. Semi-supervised
classiﬁcation with graph convolutional networks. arXiv
preprint arXiv:1609.02907, 2016.
[17] Huihua Lu, Bojan Cukic, and Mark Culp. Software
defect prediction using semi-supervised learning with
dimension reduction. In Proceedings of the 27th
IEEE/ACM International Conference on Automated Soft-
ware Engineering, pages 314–317. ACM, 2012.
[18] Laurens van der Maaten and Geoffrey Hinton. Visu-
alizing data using t-sne. Journal of machine learning
research, 9(Nov):2579–2605, 2008.
[19] Kewen Meng and Boyana Norris. Mira: A framework
for static performance analysis. CoRR, abs/1705.07575,
2017. URL http://arxiv.org/abs/1705.07575.
[20] Kewen Meng and Boyana Norris. Mira: A framework for
static performance analysis. In 2017 IEEE International
Conference on Cluster Computing (CLUSTER) , pages
103–113. IEEE, 2017.
[21] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey
Dean. Efﬁcient estimation of word representations in
vector space. arXiv preprint arXiv:1301.3781 , 2013.
[22] Lili Mou, Ge Li, Zhi Jin, Lu Zhang, and Tao
Wang. TBCNN: A tree-based convolutional neural
network for programming language processing. CoRR,
abs/1409.5718, 2014. URL http://arxiv.org/abs/1409.
5718.
[23] Lili Mou, Ge Li, Yuxuan Liu, Hao Peng, Zhi Jin, Yan Xu,
and Lu Zhang. Building program vector representations
for deep learning. CoRR, abs/1409.3358, 2014. URL
http://arxiv.org/abs/1409.3358.
[24] Sri Hari Krishna Narayanan, Boyana Norris, and Paul D
Hovland. Generating performance bounds from source
code. In 2010 39th International Conference on Parallel
Processing Workshops, pages 197–206. IEEE, 2010.
[25] Iulian Neamtiu, Jeffrey S Foster, and Michael Hicks. Un-
derstanding source code evolution using abstract syntax
tree matching. ACM SIGSOFT Software Engineering
Notes, 30(4):1–5, 2005.
[26] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio.
On the difﬁculty of training recurrent neural networks.
In International conference on machine learning , pages
1310–1318, 2013.
[27] Jeffrey Pennington, Richard Socher, and Christopher D
Manning. Glove: Global vectors for word representation.
In Proceedings of the 2014 conference on empirical
methods in natural language processing (EMNLP), pages
1532–1543, 2014.
[28] Bryan Perozzi, Rami Al-Rfou’, and Steven Skiena. Deep-
walk: online learning of social representations. In KDD,
2014.
[29] Hanmin Qin and Xin Sun. Classifying bug reports into
bugs and non-bugs using lstm. In Proceedings of the
Tenth Asia-Paciﬁc Symposium on Internetware , Internet-
ware ’18, New York, NY , USA, 2018. Association for
Computing Machinery. ISBN 9781450365901. doi:
10.1145/3275219.3275239. URL https://doi.org/10.1145/
3275219.3275239.
[30] Dan Quinlan and Chunhua Liao. The rose source-
to-source compiler infrastructure. In Cetus Users and
Compiler Infrastructure Workshop, in conjunction with
PACT 2011, October 2011.
[31] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem,
Rianne Van Den Berg, Ivan Titov, and Max Welling.
Modeling relational data with graph convolutional net-
works. In European Semantic Web Conference , pages
593–607. Springer, 2018.
10
[32] Yusuke Shido, Yasuaki Kobayashi, Akihiro Yamamoto,
Atsushi Miyamoto, and Tadayuki Matsumura. Auto-
matic source code summarization with extended tree-
lstm, 2019.
[33] Martin Sundermeyer, Ralf Schl ¨uter, and Hermann Ney.
Lstm neural networks for language modeling. In Thir-
teenth annual conference of the international speech
communication association, 2012.
[34] Kai Sheng Tai, Richard Socher, and Christopher D
Manning. Improved semantic representations from tree-
structured long short-term memory networks. In Pro-
ceedings of the 53rd Annual Meeting of the Association
for Computational Linguistics and the 7th International
Joint Conference on Natural Language Processing (Vol-
ume 1: Long Papers) , pages 1556–1566, 2015.
[35] Yao Wan, Jingdong Shu, Yulei Sui, Guandong Xu, Zhou
Zhao, Jian Wu, and Philip S. Yu. Multi-modal attention
network learning for semantic source code retrieval,
2019.
[36] Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang
Ye, Peng Cui, and Philip S Yu. Heterogeneous graph
attention network. In The World Wide Web Conference ,
pages 2022–2032, 2019.
[37] Ronald J Williams and David Zipser. A learning al-
gorithm for continually running fully recurrent neural
networks. Neural computation, 1(2):270–280, 1989.
11

HAL Id: lirmm-03366078
https://hal-lirmm.ccsd.cnrs.fr/lirmm-03366078
Submitted on 5 Oct 2021
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in F rance or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
Mapping Computations in Heterogeneous Multicore
Systems with Statistical Regression on Program Inputs
Junio Cezar Ribeiro da Silva, Lorena Leão, Vinicius Petrucci, Abdoulaye
Gamatié, F ernando Magno Quintão Pereira
T o cite this version:
Junio Cezar Ribeiro da Silva, Lorena Leão, Vinicius Petrucci, Abdoulaye Gamatié, F ernando Magno
Quintão Pereira. Mapping Computations in Heterogeneous Multicore Systems with Statistical Re-
gression on Program Inputs. ACM T ransactions on Embedded Computing Systems (TECS), ACM,
2021, 20 (6), pp.#112. ￿10.1145/3478288￿. ￿lirmm-03366078￿
1
Mapping Computations in Heterogeneous Multicore
Systems with Statistical Regression on Program Inputs
JUNIO CEZAR RIBEIRO DA SILVA, Universidade Federal de Minas Gerais
LORENA LE ˜AO, Universidade Federal de Minas Gerais
VINICIUS PETRUCCI, Universidade Federal da Bahia and University of Pi/t_tsburgh
ABDOULAYE GAMATI´E, LIRMM, Univ. Montpellier, CNRS
FERNANDO MAGNO QUINT ˜AO PEREIRA, Universidade Federal de Minas Gerais
A hardware con/f_iguration is a set of processors and their frequency levels in a multi-core heterogeneous system.
/T_his paper presents a compiler-based technique to match functions with hardware con/f_igurations. Such a
technique consists in using multivariate linear regression to associate function arguments with particular
hardware con/f_igurations. By showing that this classi/f_ication space tends to be convex in practice, this paper
demonstrates that linear regression is not only an eﬃcient tool to map computations to heterogeneous
hardware, but also an eﬀective one. To demonstrate the viability of multivariate linear regression as a way to
perform adaptive compilation for heterogeneous architectures, we have implemented our ideas onto the Soot
Java bytecode analyzer. Code that we produce can predict the best con/f_iguration for a large class of Java and
Scala benchmarks running on an Odroid XU4 big.LITTLE board; hence, outperforming prior techniques such
as ARM’s GTS and CHOAMP, a recently released static program scheduler.
CCS Concepts: •So/f_tware and its engineering→Compilers; •Computing methodologies→Parallel
programming languages; Machine learning;
ACM Reference format:
Junio Cezar Ribeiro da Silva, Lorena Le˜ao, Vinicius Petrucci, Abdoulaye Gamati´e, and Fernando Magno /Q_uint˜ao
Pereira. 2021. Mapping Computations in Heterogeneous Multicore Systems with Statistical Regression on
Program Inputs. ACM Trans. Embedd. Comput. Syst. 1, 1, Article 1 (January 2021), 35 pages.
DOI: 10.1145/3478288
1 INTRODUCTION
Several modern computer architectures combine, into a single device, fast and slow processing
cores able to execute the same set of instructions (Kumar et al. 2004; Orgerie et al. 2014; Singh et al.
2020). Fast cores perform computations eﬃciently, but are power-hungry; Slow cores show the
inverse behavior. /T_he ARM big.LITTLE design, ubiquitous in smartphones, is an example of such
technology (H¨ahnel and H¨artig 2014). As an illustration, the Apple A14, launched in September of
2020, has two high-performance cores calledFirestorm and four energy-eﬃcient cores called Icestorm,
all running the ARMv8.4-A instruction set (Gurman et al. 2020). /T_he combination of heterogeneous
processors featuring multiple frequency levels gives programmers many con/f_igurations to choose
from when running their applications. However, performing this choice is challenging (Azhar et al.
2019; Nejat et al. 2020; Nishtala et al. 2017; Paul et al. 2020; Petrucci et al. 2015).
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee
provided that copies are not made or distributed for pro/f_it or commercial advantage and that copies bear this notice and the
full citation on the /f_irst page. Copyrights for components of this work owned by others than the author(s) must be honored.
Abstracting with credit is permi/t_ted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires
prior speci/f_ic permission and/or a fee. Request permissions from permissions@acm.org.
© 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM. 1539-9087/2021/1-ART1 $15.00
DOI: 10.1145/3478288
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
1:2 Ribeiro, et al.
A recent solution to this problem is CHOAMP, a compilation technique invented by Sreelatha et
al. (Sreelatha et al. 2018). CHOAMP uses supervised machine learning to map program functions
to the con/f_iguration that best /f_its them. Sreelathaet al. capture characteristics of the target
architecture’s runtime behavior. From this knowledge, they predict the ideal con/f_iguration to a
program, given its syntactic characteristics. Sreelatha et al.’s approach is fully static: interventions
on the program remain con/f_ined into the compiler, and no extra runtime support is required from
the hardware. /T_his approach has been made popular by Shelepovet al. (Shelepov et al. 2009)’s
HASS system, a scheduler for same-ISA heterogeneous systems.
We observe that CHOAMP and HASS share a fundamental shortcoming: they do not consider
program inputs when performing mapping decisions. As we explain in Section 2, there exist
programs for which the best hardware con/f_iguration for a given function varies depending on the
function’s inputs. We make the case that inputs are key to determine good mappings between
programs and con/f_igurations supported by the evidence that such mappings do not necessarily
converge to a single, ideal con/f_iguration, as the size of inputs grows.
Our Solution. In this paper, we introduce a compilation approach to map program parts to
hardware con/f_igurations. Our technique explicitly takes function inputs into consideration when
deciding which hardware con/f_igurations to use. Input-based code optimizations are not a new idea,
as we explain in Section 5.3; however, to the best of our knowledge, this paper is the /f_irst to use
related techniques to /f_ind optimized hardware con/f_igurations for programs running on big.LITTLE
architectures. As we discuss in Section 3, our idea is based on statistical regression on the values of
program inputs. Given a function foo, a collection of its inputs {t1, t2, . . . ,tm }available for training,
plus a set of hardware con/f_igurations{h1, h2, . . . ,hn }, we run foo(ti ), 1 ≤i ≤m, onto a sample of
the con/f_iguration space{hj |1 ≤j ≤n}. Training gives us the ideal con/f_iguration for each input,
in terms of a measurable goal, such as runtime or energy consumption. When producing code for
foo, we augment its binary representation with this knowledge to predict the best con/f_iguration for
potentially unseen inputs.
Our Results.We have implemented our technique in S/o.sc/o.sc/t.sc(Vall´ee-Rai et al. 1999), a bytecode
optimizer, and have evaluated it onto an Odroid XU4 big.LITTLE architecture. S/o.sc/o.sc/t.sclets generate
code that, at runtime, changes the hardware con/f_iguration per program function, based on train-
ing knowledge. We call this code generator the J/i.sc/n.sc/n.sc/hyphen.scCcompiler, a tool that reads and outputs
Java bytecodes. As we explain in Section 4, we have evaluated J/i.sc/n.sc/n.sc/hyphen.scCon the Program Based
Benchmark Suite (Shun et al. 2012) used by Acar et al (Acar et al. 2018), and on programs from
Renaissance (Prokopec et al. 2019). We have evaluatedJ/i.sc/n.sc/n.sc/hyphen.scCwith two objective functions: runtime
and energy consumption. We measure energy for the entire board using physical probes, following
Bessa et al’s methodology (Bessa et al. 2017). Below we summarize the bene/f_its of our solution in
the context of the existing literature:
•Simple: we show that, for typical parallel code, the value of scalar inputs and the size of
aggregate inputs yield useful information to feed linear regression models, because the
function that maps these values to optimized hardware con/f_igurations form a convex space.
•Eﬀective : in most of our benchmarks, ten input sets are already suﬃcient to let us train a
predictor to a high level of accuracy. Variety is, of course, important: the more diﬀerent
the inputs we have, the more accurate the predictions we perform.
•Eﬃcient : our approach does not require active runtime monitoring. Inputs must be
evaluated upon function invocation, and only then. Evaluation amounts to one vector-
matrix multiplication, and is proportional to the number of function arguments (not to
their sizes) times the number of valid hardware con/f_igurations. Training is costly, but this
cost is paid oﬄine, before programs run in production mode.
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
Mapping in Parallel Heterogenous Architectures 1:3
•Semi-Automatic: our approach requires minimum user interference. Developers annotate
which functions must be adapted. No further interventions are necessary.
•Easily-deployable: our solution does not require runtime monitoring; thus, it can be
deployed in any modern hardware and operating system, independent on them providing
advanced runtime capability such as performance counters. We require only the capability
to change the hardware con/f_iguration at runtime.
Convex Space./T_his journal paper extends an earlier conference paper of ours (da Silva et al. 2020),
with more experiments and a more detailed explanation of our technique. /T_he most important
addition over that /f_irst publication is an empirical demonstration that statistical functions that
map program inputs to hardware con/f_igurations tend to be convex. As we show in Section 4.4,
convexity means that by varying only one function argument, while /f_ixing the others, the ideal
con/f_iguration is unlikely to oscillate, for instance, going fromhi to hj and then back to hi . We have
not found one single case, among 18 benchmarks, each with 14 diﬀerent input sets, where such
oscillations could be discerned. /T_he consequence of this observation, discussed in Section 4.4.2,
is that derivative-based search methods are expected to converge to an optimal result, and linear
regression is capable to accurately predict this optimum. We emphasize that this observation shows
an intuitive tendency that has been veri/f_ied in the benchmarks used in this paper, for the inputs
available to them. Although it is easy to construct benchmarks whose space of (inputs ×ideal
hardware con/f_igurations) is not convex, we believe that such benchmarks are unlikely to emerge
as part of well-known algorithms, as we explain in Section 4.4.
2 OVERVIEW
/T_he termhardware con/f_igurationis used with diﬀerent meanings by diﬀerent researchers, thus we
shall restrict ourselves to the following de/f_inition:
De/f_inition 2.1 (Hardware Con/f_iguration).Let Π = {π1, π2, . . . ,πn }be a set of n processors, and
let Freq be a function that maps each processor to a list of possible frequency levels. A hardware
con/f_iguration is a set of pairsh = {(π, f )|π ∈Π, f ∈Freq(π)}. If (πi , fj )∈ h, for some fj ∈Freq(πi ),
then processor πi is said to be active in h with frequency fj , otherwise it is said to be inactive.
Example 2.2 (Hardware Con/f_iguration)./T_he HardKernel Odroid XU4 has four big cores{b0, b1, b2, b3}
and four LITTLE cores {L0, L1, L2, L3}. Big cores have 19 frequency levels {200MHz, 300MHz, . . . ,
1.9GHz, 2.0GHz}. LITTLE cores have 14 {200MHz, . . . ,1.5GHz}. /T_his SoC supports any number of
active processors; however, big cores must always use the same frequency level. /T_he same is true
for LITTLE cores. An example of hardware con/f_iguration is{b0, b2}×2.0GHz, {L1, L2, L3}×1.3GHz.
/T_he notion of hardware con/f_iguration leads to an interesting problem in the /f_ield ofadaptive
compilation. In the words of Cooper et al (Cooper et al. 2005), “an adaptive compiler uses a compile-
execute-analyze feedback loop to /f_ind the combination of optimizations and parameters that minimizes
[optimizes] some performance goal, such as code size or execution time”. In this paper we are interested
in solving the adaptive compilation problem introduced by De/f_inition 2.3.
De/f_inition 2.3.I/n.sc/p.sc/u.sc/t.sc/hyphen.scA/w.sc/a.sc/r.sc/e.sc S/c.sc/h.sc/e.sc/d.sc/u.sc/l.sc/i.sc/n.sc/g.sc /i.sc/n.sc S/i.sc/n.sc/g.sc/l.sc/e.sc/hyphen.scISA H/e.sc/t.sc/e.sc/r.sc/o.sc/g.sc/e.sc/n.sc/e.sc/o.sc/u.sc/s.sc A/r.sc/c.sc/h.sc/i.sc/t.sc/e.sc/c.sc/t.sc/u.sc/r.sc/e.sc/s.sc
(I/s.sc/h.sc/a.sc) Input: a function F, its input i, a set of hardware con/f_igurationsH = {h1, . . .hn }, and a
cost function Oi
F : H ↦→R, which determines the cost of running F with input i on con/f_iguration
h ∈H. Examples of cost functions include runtime, energy, energy-delay product, throughput, etc.
Output: a con/f_igurationh ∈H that minimizes Oi
F .
As Section 3 explains, we insert code at the entry point of functions to switch the hardware
con/f_iguration; hence, providing a solution toI/s.sc/h.sc/a.sc. From that region on, the program will run in
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
1:4 Ribeiro, et al.
the chosen con/f_iguration, until its execution /f_low /f_inds the entry point of another function that has
also been instrumented. Any function might run at any hardware con/f_iguration, for although we
target a heterogeneous architecture, big and LITTLE cores run the same instruction set. Notice that
moving computations across cores bears a cost: cores do not share L1 caches, and big and LITTLE
clusters do not share the L2 cache, although all the cores share the main memory (Weber et al .
2017, Sec.2.2). /T_his cost is embedded in the numbers that we report in Section 4. In other words,
the speedups or regressions that we observe experimentally already include the cost of warming
up caches due to switching computation between cores.
2.1 Program Inputs and Hardware Configuration
Compilers, such as G/c.sc/c.scor C/l.sc/a.sc/n.sc/g.sc, do not try to capitalize on diﬀerences between cores when
producing binary programs: the same executable runs in both cores. Nevertheless, we know of
research artifacts that take these diﬀerences into consideration; for example, CHOAMP is a recent
technique in this direction (Sreelatha et al. 2018). CHOAMP matches program features, such as
branches, barriers, reductions and memory access operations with the ideal con/f_iguration for each
function. CHOAMP has been evaluated on the OpenMP version of the NAS benchmark suite (Bailey
et al. 1991) with great bene/f_it, producing code that was 65% more energy-eﬃcient than the default
Linux scheduler.
A/f_ter CHOAMP trains a regression model, the same core con/f_iguration decision applies for a
function, regardless of its actual inputs. /T_his shortcoming of purely static approaches has been
well-known. /Q_uoting Nie and Duan: “since the properties they have collected are based on the given
input set, those oﬄine pro/f_iling approaches are hard to adapt to various input sets and therefore will
drastically aﬀect the program performance” (Nie and Duan 2012). We corroborate this observation
and show that it is possible to /f_ind diﬀerent programs for which the ideal hardware con/f_iguration
varies according to their inputs. Example 2.4 illustrates this /f_inding with an actual experiment.
Example 2.4. Function /t.sc/a.sc/s.sc/k.scin Figure 1 inserts into a global map all the values stored in a
stream. Values are associated with a key, whose size varies according to the formal parameter
/k.sc/e.sc/y.scS/i.sc/z.sc/e.sc. T/a.sc/s.sc/k.schas a synchronized block; hence, it can be safely executed by multiple threads. /T_he
number of threads is an implicit input. /T_hese three values: size of input stream, size of keys, and
number of threads, form a three-dimensional space, which Figure 1 illustrates. /T_he ideal hardware
con/f_iguration for/t.sc/a.sc/s.sc/k.scvaries within this space. Figure 2 illustrates this variation for 3×25 diﬀerent
input sets. /T_he notationXbYL denotes X big cores, and Y LITTLE cores. In this experiment, we
have set Freq(b) = 1.8GHz, for any big core b, and Freq(L) = 1.5GHz, for any LITTLE core L.
/T_he construction of a key, at line 5 of Figure 1 is a CPU-heavy, synchronization-free task.
/T_he larger the key, the more incentive we have to use the big cores. However, the updating of
/g.sc/l.sc/o.sc/b.sc/a.sc/l.scM/a.sc/p.scat line 9 is a synchronization-heavy task: the more synchronization we have, the
heavier is the penalty on the big cores, relative to the LITTLE ones. Indeed, as already observed
by Kim et al. (Kim et al. 2014), context switches are more expensive in the big than in the LITTLE
cores. Reasons for this heavier penalty include the larger pipelines used in the big cores. Whereas
the ARM A15 (big) core features a pipeline with 15 integer and up to 25 /f_loating point stages,
the A7 (LITTLE) core features a pipeline with only 8 stages (Weber et al. 2017, Sec.2.2). Memory
accesses are also more expensive (relative to execution cycles) on the faster cores: L2 latency for
big cores is 21 cycles while for LITTLE cores it is 10 (Greenhalgh 2011). Furthermore, the larger
the input streams, the more o/f_ten we access the synchronized region between lines 7 and 10 of
Figure 1. We can observe results similar to those seen in Example 2.4 in algorithms like Integer
Sort, a benchmark used by Sreelatha et al. (Sreelatha et al. 2018), which we re-evaluate in Section 4.
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
Mapping in Parallel Heterogenous Architectures 1:5
// The number of threads is a hidden input
void task(Stream<Value> s, long keySize) {
  while (!s.empty()) {
    // Get a key of the proper size:
    BigInteger key = getNextKey(keySize);
    // Use key to update globalMap
    synchronized(globalMap) {
      Value value = s.next();
      globalMap.put(key, value);
    }
  }
}
T	=	4	
T	=	8	
T	=	16	
T	=	32	
0	
2000	
4000	
6000	
8000	
10000	
10^2	 10^3	 10^3	 10^4	
# Threads
s.size()
keySize
Fig. 1. (Le/f_t) A program whose behavior can be computation-heavy or synchronization-heavy, depending on
its inputs. (Right) The search space formed by the program’s inputs. (Taken from da Silva et al. (2020)).
oox 2b0Lx
4b0Lox ox
4b0Lx o ox
o o 4b0Lo o
4b0Loooo
keySize
s.size ( )
1 2 3 4 5
5
4
3
2
1
4T
xxx 2b0Lx
4b4L4b4Lx x-
4b4Lx 4b4L 4b4Lo
4b4L 4b4L 4b4L4b4L 4b4L
4b4L4b4L4b4L4b4L4b4L
8T
xxx 2b0Lx
oox -x
ox o ox
o o ox o
ooooo
16T
0b2L, 0b4L, 2b0L, 4b0L, 4b4L
0b4L, 2b0L, 4b0L, 4b4L
10
10
10
10
10
10 10 10 10 10
4b0L, 4b4Lo
-
x
Fig. 2. Ideal configuration for diﬀerent parameters of the /t.sc/a.sc/s.sc/k.scfunction (Fig. 1), for 4, 8 and 16 threads,
measured on an Odroid XU4 with the userspace governor. Default configuration is 4b4L. Names in boxes
indicate the best configuration(s) for that input. ’X’ indicates setups with three or more configurations tied
as best. To produce these charts, we followed a methodology to be described in Section 4.4. Even considering
4 threads, there is benefit to enable more than four processors, as the Java virtual machine creates threads for
garbage collection and JIT compilation, for instance (Taken from da Silva et al. (2020)).
Notice that allowing a program to use more threads does not necessarily mean that this program
will draw larger bene/f_it from con/f_igurations with more cores. /T_his counterintuitive behavior
happens because more cores might provoke more thread con/f_licts in synchronization-heavy se/t_tings.
Example 2.5 elaborates on this observation.
Example 2.5. Figure 2 shows that when the T/a.sc/s.sc/k.scfunction runs with eight threads, the con/f_igura-
tion with full resources (4b4L) is o/f_ten chosen. However, once 16 threads are employed to generate
keys, sometimes not all the cores are used in the fastest con/f_iguration. Figure 3 provides numbers
to explain this behavior. For small key sizes, collisions between threads happen frequently. In this
se/t_ting,1/b.sc0Lis the best con/f_iguration, regardless of the number of threads considered (even though
running times are short, the con/f_idence level of this result is above 99%). Once key size increases,
time spent on synchronization becomes less important, and extra cores start to be advantageous.
Notice that past certain point (a/f_ter keys with 10,000 bits), just-in-time compilation triggers, and the
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
1:6 Ribeiro, et al.
0.E+00	
1.E+05	
2.E+05	
3.E+05	
4.E+05	
5.E+05	
6.E+05	
7.E+05	
10^1	 10^2	 10^3	 10^4	 10^5	
0.E+00	
5.E+05	
1.E+06	
2.E+06	
2.E+06	
3.E+06	
10^1	 10^2	 10^3	 10^4	 10^5	
0.E+00	
1.E+06	
2.E+06	
3.E+06	
4.E+06	
5.E+06	
6.E+06	
7.E+06	
8.E+06	
10^1	 10^2	 10^3	 10^4	 10^5	
0b1L	 1b0L	 0b4L	 4b0L	 4b4L	
Average Waiting Time (Nanoseconds)
keySize (Number of bits in the key)
4 Threads
8 Threads
16 Threads
Total Running Time (seconds)
keySize (Number of bits in the key)
0.001	
0.01	
0.1	
1	
10	
10^1	 10^2	 10^3	 10^4	 10^5	
0.001	
0.01	
0.1	
1	
10	
10^1	 10^2	 10^3	 10^4	 10^5	
0.01	
0.1	
1	
10	
10^1	 10^2	 10^3	 10^4	 10^5	
4 Threads
8 Threads
16 Threads
Fig. 3. Average waiting time and total running time of function T/a.sc/s.sc/k.sc(Figure 1) considering diﬀerent key
sizes, and diﬀerent numbers of threads. To perform this experiment, we have modified T/a.sc/s.sc/k.scto use Java’s
ReentrantLock class. This new version of T/a.sc/s.sc/k.scis slower, due to the need to log events; hence, we present
total running times to give the reader some perspective on the program’s behavior.
average waiting time decreases by approximately 50%. Yet, the total application time still increases,
due to the excess of calculations required by the large keys.
Example 2.5 shows that two facts increase the average time that threads spent on critical sections:
the total number of working threads that the program uses, and the total number of cores where
these threads execute. /T_herefore, adding more resources to thread rich programs can cause them to
run more slowly due to extra con/f_licts. Such complex tradeoﬀs make the problem of choosing ideal
hardware con/f_igurations a non-trivial endeavor.
2.2 Accounting for Energy Eﬀiciency
Choosing good hardware con/f_igurations becomes more challenging, once we consider energy as a
dimension of eﬃciency. Low-frequency cores tend to be more power-eﬃcient than high-frequency
processors; hence, there is incentive to use them to save energy. However, low-frequency cores
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
Mapping in Parallel Heterogenous Architectures 1:7
tend to take longer to /f_inish tasks; possibly, using more energy to perform a job. /T_his observation
is critical in ba/t_tery-powered devices, such as smartphones. /T_he next example analyzes such
power-performance tradeoﬀs. In this experiment, we are measuring the actual power consumed in
the entire board, which includes not only its CPUs but also its peripherals, such as memory and
cooling. To this end, we use the apparatus described by da Silva et al. (2018), which samples power
at 20KHz.
Example 2.6. We have used the power measurement apparatus shown in Figure 4(a) to plot
runtime and energy consumption for the function /t.sc/a.sc/s.sc/k.scearlier seen in Fig. 1, considering two
diﬀerent input sets. Figure 4(b) shows the power pro/f_ile ofT/a.sc/s.sc/k.scfor a synchronization-free set
of inputs (top) and for a synchronization heavy set (bo/t_tom). Following Silvaet al. (da Silva et al.
2018), we call the chart relating runtime and energy a constellation. /T_he constellation in Figure 4(c)
shows the behavior of /t.sc/a.sc/s.sc/k.scfor the synchronization-free input. In this case, the size of keys is very
large, and the number of insertions in the /g.sc/l.sc/o.sc/b.sc/a.sc/l.scM/a.sc/p.scis very low, thus con/f_licts seldom happen.
On the other hand, if we make the size of keys very small, and the size of the stream very large,
then we obtain a rather diﬀerent constellation, which Figure 4(d) outlines. /T_his constellation shows
how T/a.sc/s.sc/k.scperforms in a synchronization-heavy environment.
30	
60	
90	
120	
4	 8	 12	 16	 20	 24	 28	 32	 36	
20	
40	
60	
80	
100	
4	 8	 12	 16	
Energy (J)
Energy (J)
Time (s) Time (s)
1B4L
0B4L
2B2L
2B1L
2B0L
3B3L
3B2L
3B1L
3B0L
4B4L
1B3L
0B3L
4B3L
0B2L
4B2L
0B1L
4B0L
4B1L
1B0L
1B1L
1B2L
4B3L
4B2L4B1L
4B0L
0B1L
0B4L
3B4L 
3B3L
3B2L
3B1L
3B0L
0B3L
1B0L
1B2L
1B3L
1B4L
1B1L
0B2L
2B3L
2B0L
2B1L
Best time and energy
Worst time 
and energy
Odroid 
XU4 Synchronization 
circuit
Power
meter
// This function is still multi-threaded
void syncFreeTask(Stream<Value> s, long keySize, 
                     Map<BigInteger, Value> privateMap) {
  while (!s.empty()) {
    // Get a key of the proper size:
    BigInteger key = getNextKey(keySize);
    // Use key to update the map (private per thread)
    Value value = s.next();
    privateMap.put(key, value);
  }
}
1
2
3
4
5
6
7
8
9
10
11
(a)
(c)
(b)
(d)
8 Threads
35
45
55
65
75
85
7 8 9 10 11 12 13 14
Energy (J)
10
30
50
70
90
2 7 12 17 22
Energy (J)
1b0L
0b1L1b1L2b0L
1b2L
0b2L
4b1L
3b2L
3b3L
4b2L
4b3L
0b3L
1b3L
2b1L
0b4L
2b2L
1b4L
3b1L
4b0L
3b0L
2b3L
2b0L
3b0L
1b0L
0b3L 0b2L
0b1L
0b4L
1b1L
3b2L
3b1L4b0L
4b4L
4b1L
4b2L
4b3L 3b4L
3b3L
2b1L
2b3L
2b3L1b4L
1b3L
1b1L
      (b)     
8 Threads
0 1 2 3 4 5
0 5 10 15 20
Time (s)
iPower (W)
0 2 4 6 8 10 12
0 5 10 15 20
Time (s)
iPower (W)
0
6
10
16
20
0
6
10
16
20
0 2 4 6 8 10 12
0 1 2 3 4 5
 Power (W)  Power (W)
Time (s)
Time (s)
Odroid 
XU4
Power
MeterSynchronization
Circuit
(b)
8 Threads
(c) (d)
Energy (J)
10
30
50
70
90
35
45
55
65
85
75
2 7 12 17 22 7 9 11 1312108 14
Time (s)
2b4L
3b4L
4b4L 1b2L
(a)     
Time (s)
Fig. 4. (a) The energy measurement apparatus adopted in this paper. (b) Power charts for configuration
4b4L with synchronization-free inputs (top) and synchronization-heavy inputs (bo/t_tom). (c) Time vs energy
constellation for the synchronization-free input set. (d) The constellation for the synchronization-heavy input
set. Big cores run at 2.0GHz and LITTLE cores run at 1.5GHz. (Taken from da Silva et al. (2020))
Example 2.6 shows how changes in inputs modify the disposition of hardware con/f_igurations in
the constellations. /T_he best energy and time con/f_iguration in the CPU-heavy se/t_ting,4b4L, is one
of the worst con/f_igurations in the synchronization-heavy se/t_ting. Such dramatic changes make
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
1:8 Ribeiro, et al.
it very diﬃcult for a completely static approach to /f_ind energy-eﬃcient hardware con/f_igurations
for program parts. /T_he size and type of program inputs are only known at runtime. To handle the
lack of information at compile time, existing prior work (David et al . 2014; Nishtala et al. 2017;
Petrucci et al. 2015) resorts to online monitoring; however, this may pose a potential overhead on
the system as the number of programs and hardware con/f_iguration increase.
3 MULTIVARIATE LINEAR REGRESSION OF PROGRAM INPUTS
We apply statistical regression on the arguments of a function to determine the ideal hardware
con/f_igurations for diﬀerent inputs of that function. /T_he pipeline in Figure 5 summarizes how code
is modi/f_ied in order to implement this idea. To ease our presentation, we shall be using source
code in our examples, as seen in Figure 5. However, our solution works at the Java bytecode level
and our interventions happen within the compiler –more precisely in the program’s intermediate
representation. Our techniques could have been applied directly onto Java sources or even onto
a diﬀerent programming language. Nevertheless, working at the bytecode level lets us optimize
programs wri/t_ten in diﬀerent languages that run on the Java Virtual Machine. In Section 4 we shall
validate our techniques using Java and Scala benchmarks.
3.1 Multiple Linear Regression
/T_he key ingredient of our work is the application of multivariate regression onto the arguments
of functions. Linear regression empowers a prediction model that matches function parameters
with resource-eﬃcient hardware con/f_igurations. We extend our regression model to a multivariate
system, as the output is a vector (of ideal con/f_igurations). In this model, we de/f_ine a number of
dependent variables, grouped into a matrix C, plus a number of independent variables, grouped
into a matrix A. /T_he goal of the regression model is to determine a matrixΘ that approximates
the product C = σ(AΘ). In this case, σ is the so/f_tmaxfunction, applied on the rows of the matrix
product AΘ. If Z is a 1 ×n vector, e.g., a line of AΘ, then σ(Z)is also an 1 ×n vector, whose jth
element is de/f_ined as:σ(Z)j = eZj /Ín
1 eZk . /T_he so/f_tmax function receives a vector of real numbers,
and produces a vector of the same size normalized over a probability distribution. Every σ(Z)j is a
number between 0.0 and 1.0, and the sum of all the elements within σ(Z)is 1.0.
Example 3.1. Figure 6 presents a formula for regression involving a function f that has three
formal parameters. We assume a universe of /f_ive valid con/f_igurations (0/b.sc1L, 1/b.sc0L, 1/b.sc1L, 2/b.sc0Land
2/b.sc1L). /T_he frequency level is immaterial for this example: big and LITTLE cores run at a certain
/f_ixed frequency, which is not necessarily the same for the two clusters. In this example we have a
training set containing four samples, each one representing a diﬀerent invocation of function f ,
ideally with diﬀerent actual arguments.
/T_he matrixA of independent variables.As Example 3.1 illustrates, the matrixA encodes known
values of function arguments. /T_hese values are called thetraining set of our regression. If we are
analyzing a function with n arguments, and our training set contains m function calls, then A is a
matrix withm rows andn +1 columns. /T_he extra column is the all-ones vector 1m, which represents
intercepts – constants that allow us to handle a scenario in which the training set contains only
null values. /T_his all-ones column is the /f_irst column of matrixA in Figure 6.
Example 3.2. Figure 7 shows how ten diﬀerent samples of functionT/a.sc/s.sc/k.sc, from Fig. 1, are organized
into a matrix A of independent variables.
/T_he matrixC of dependent variables.C represents the ideal hardware con/f_iguration for each
input in the training set. If we admit k valid con/f_igurations, and our training set hasm samples,
then C is an m ×k matrix. Each line of C is a unitary vector ei , which has all the components set to
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
Mapping in Parallel Heterogenous Architectures 1:9
void task(Stream<Value> s, long keySize) {
  Bundle b = new Bundle(0xA33F0251);
  b.addConﬁg(Env.getCurrentConﬁg());
  b.addVar((double)s.size());
  b.addVar((double)keySize);
  b.addVar((double)Thread.activeCount());
  b.startTimer();
  while (!s.empty()) {
    BigInteger key = getNextKey(keySize);
    synchronized(globalMap) {
      Value value = s.next();
      globalMap.put(key, value);
    }
  }
  b.endTimer();
}
Soot
add proﬁling
instrumentation
.jar/.class
instrumented
for training
 Bash+Java
Driver (java)
I/O hooks
List of
inputs
Training result:
(inputs, conﬁgs)
Python
Regression
Analysis
Soot
add prediction
instrumentation
.jar/.class
instrumented
for production
Regression
Coeﬃcients
(matrix /u1D767)
@AdaptiveMethod
@HiddenInput (expr=“Thread.activeCount()”)
void task(Stream<Value> s, long keySize) {
  while (!s.empty()) {
    BigInteger key = getNextKey(keySize);
    synchronized(globalMap) {
      Value value = s.next();
      globalMap.put(key, value);
    }
  }
}
void task(Stream<Value> s, long keySize) {
  … 
  conﬁg = // predicted conﬁguration for
         // (s.size(), keySize, Thread.activeCount());
  Regression.changeConﬁg(conﬁg);
  while (!s.empty()) {
    BigInteger key = getNextKey(keySize);
    synchronized(globalMap) {
      Value value = s.next();
      globalMap.put(key, value);
    }
  }
  // Restore original conﬁguration (See Fig.14)
}
Annotated
.jar/.class
Annotated
.java/.kt
Javac/Kotlinc
Pre-process
annotations
Fig. 5. The execution pipeline of J/i.sc/n.sc/n.sc/hyphen.scC, as shown in our previous work (da Silva et al. 2020).
zero, except its ith index, which is set to one. If Cji = 1, then i is the best con/f_iguration for inputj.
/T_he next example illustrates these notions with actual data.
Example 3.3. Figure 8 reuses the ten samples seen in Example 3.2 to build the matrix of dependent
variables. /T_his matrix has one line per sample, and one column per con/f_iguration of interest. /T_his
example considers only 10 out of the 4,654 possible con/f_igurations of the Odroid XU4 board.
/T_his need for bounding the search space might prevent us from discovering good optimization
opportunities; however, it ensures that our methodology is practical. Section 4 discusses the criteria
used to build the search space of allowed con/f_igurations.
Finding the parameter matrixΘ. /T_he problem of constructing a predictor based on multivariate
linear regression consists in /f_inding a matrixΘ that maximizes correct predictions on the training
set. /T_he underlying assumption is that ifΘ approximates the behavior of the training set, then it is
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
1:10 Ribeiro, et al.
/u1D72D00   /u1D72D01   /u1D72D02   /u1D72D03   /u1D72D04
/u1D72D10   /u1D72D11   /u1D72D12   /u1D72D13   /u1D72D14
/u1D72D20   /u1D72D21   /u1D72D22   /u1D72D23   /u1D72D24
/u1D72D30   /u1D72D31   /u1D72D32   /u1D72D33   /u1D72D34
0    1    0    0    0
1    0    0    0    0
0    0    0    0    1
0    0    0    0    10b1L
1b0L
1b1L
2b0L
2b1L
1   /u1D73601   /u1D73602   /u1D73603
1   /u1D73611   /u1D73612   /u1D73613
1   /u1D73621   /u1D73622   /u1D73623
1   /u1D73631   /u1D73632   /u1D73633
training inputs
function arguments
×/u1D6D4=
Best(f(/u1D73601, /u1D73602, /u1D73603)): 1b0L
Best(f(/u1D73611, /u1D73612, /u1D73613)): 0b1L
Best(f(/u1D73621, /u1D73622, /u1D73623)): 2b1L
Best(f(/u1D73631, /u1D73632, /u1D73633)): 2b1L
C A ϴ
Fig. 6. Formula to train a 3-ary function f (α0, α1, α2). The goal of multivariate linear regression is to find the
coeﬀicients Θ that approximate the product C = σ(AΘ). Training set contains four samples.
1 4 10100,0001 4 1001,0001 410,0001001 8 1001001 8 1,00010,0001 810,000101161 10,000116101,0001161001011610,000100,000
T"="4"
T"="8"
T"="16"
T"="32"
1.E+00"
1.E+01"
1.E+02"
1.E+03"
1.E+04"
1.E+05"
10^1" 10^2" 10^3" 10^3" 10^4"
# Threadss.size()
keySize
# Threadss.size()keySizeintercepts
1b0L
4b0L
4b2L
4b0L
4b4L
4b4L
1b0L
4b0L
4b2L
4b4L
matrix A with training inputs
Fig. 7. Training set for the T/a.sc/s.sc/k.scmethod (Fig. 1). The table on the right is matrix A of independent variables
(taken from da Silva et al. (2020)).
likely to yield good results on the test set. /T_here exist eﬃcient techniques to /f_indΘ –gradient descent
being the best well-known (Cauchy 1847). Our model is based on multivariate linear regression;
thus, searches happen over a linear space. By a linear search space, we mean that, for each element
(i, j)in C, we have that: Cij = Θ0j + αi1Θ1j + . . .+ αim Θmj . In other words, non-linear expressions
such as αip αiq do not contribute to the value ofCij . Because our model involves only searches over
a linear space, gradient descent converges to a global optimum (Shi 2004).
Example 3.4. Figure 9 shows a possible matrixΘ that gradient descent /f_inds for theT/a.sc/s.sc/k.scfunction,
when given the training set seen in Figures 7 and 8. Once we apply the so/f_tmax function onto
the product AΘ we obtain a predicted matrix C′, which approximates the target matrix C, e.g.,
C′= σ(AΘ). Each line of C′adds up to1 1.00. /T_he largest value in each linei of C′determines the
ideal con/f_iguration for the input setAi . /T_he matrixΘ seen in Figure 9 led us into aC′that correctly
matches the target C in all but two inputs. Some misses are expected. If we resort to more complex
regression models, for instance, with non-linear components, then we might /f_ind aΘ that correctly
1We are using only two decimal digits; hence, rounding errors prevent us from obtaining 1.00 in every line.
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
Mapping in Parallel Heterogenous Architectures 1:11
T"="4"
T"="8"
T"="16"
T"="32"
1.E+00"
1.E+01"
1.E+02"
1.E+03"
1.E+04"
1.E+05"
10^1" 10^2" 10^3" 10^3" 10^4"
# Threadss.size()
keySize
1b0L
4b0L
4b2L
4b0L
4b4L
4b4L
1b0L
4b0L
4b2L
4b4L
matrix C with ideal configurations
1b0L 4b0L 4b2L 4b4L
1 0 0 00 1 0 00 0 1 00 1 0 00 0 0 10 0 0 11 0 0 00 1 0 00 0 1 00 0 0 1
Fig. 8. Matrix of independent variables built for ten diﬀerent invocations of function T/a.sc/s.sc/k.scin Figure 1 (taken
from da Silva et al. (2020)).
predicts every row of C. However, this matrix, which /f_its too well the training set, might not yield
good predictions on unseen inputs.
1b4L
4b0L
4b2L
4b4L
/u1D6D4=
Goal matrix C Matrix A
Matrix ϴ
1b4L
4b0L
4b2L
4b4L
Predicted matrix C’
×
✔
✘
✔
✔
✔
✔
✔
✔
✘
✔
1.000.000.000.000.000.670.010.330.000.000.220.780.000.570.300.130.000.060.000.940.000.000.390.611.000.000.000.000.010.890.010.090.000.590.370.050.000.000.001.00
-0.01250.01140.0006-0.6481-0.19640.04720.0166-0.0759-0.1763-0.00080.00000.00020.0010-0.0003-0.00500.0000
1000010000100100000100011000010000100001
Fig. 9. The result of multivariate linear regression produced by the training set seen in Examples 3.2 and 3.3
(taken from da Silva et al. (2020)).
Using Θ to carry out predictions./T_he single output of regression is the matrixΘ. Once we /f_ind
a suitable Θ, we can use it to predict the ideal con/f_iguration for inputs that we have not observed
during training. To this eﬀect, as we shall be/t_ter explain in Section 3.3, the constants in Θ are
hardcoded into the binary text that we generate for the function f under analysis. If f is invoked
with a set of inputs Ai , then the expression σ(Ai Θ)is computed on-the-/f_ly. /T_he result of this
evaluation determines the active con/f_iguration.
Example 3.5. Figure 10 uses the matrix Θ found in Figure 9 to guess the best con/f_iguration
for four unseen input sets. /T_hese inputs appear as dark spheres in Figure 10. In this example,Θ
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
1:12 Ribeiro, et al.
correctly predicts the ideal con/f_iguration for three out of four samples. In one case, the last input in
Figure 10, we wrongly predict the best con/f_iguration as4b2L, whereas empirical evidence suggests
that it should be 4b4L.
/u1D6D4(                        × ϴ ) =
/u1D6D4(                        × ϴ ) =
/u1D6D4(                        × ϴ ) =
/u1D6D4(                        × ϴ ) =
T"="4"
T"="8"
T"="16"
T"="32"
1.E+00"
1.E+01"
1.E+02"
1.E+03"
1.E+04"
1.E+05"
10^1" 10^2" 10^3" 10^3" 10^4"
# Threadss.size()
keySize
4b0L
4b4L
4b4L
1b0L
4b0L
4b2L
4b4L
1b4L 4b0L
4b4L
1b4L
1b0L
4b0L
4b2L
1.000.000.000.00
1.000.000.000.00
0.010.680.260.05
0.000.000.570.43
✔
✘
✔
✔
1b4L
4b0L
4b2L
4b4L
Unseen inputs Predictions
14110000
18101E+05
1161100
1161000010
Fig. 10. The matrix Θ (see Figure 9) used to predict ideal configurations for unseen inputs. Light-grey points
form the training set. Inputs in the test set are dark-grey. (taken from da Silva et al. (2020))
3.2 Engineering the Training Phase
Users of J/i.sc/n.sc/n.sc/hyphen.scCspecify which methods must be optimized. For each one of these methods, J/i.sc/n.sc/n.sc/hyphen.scC
singles out its inputs, and instrument them to produce regression data. Are considered inputs: the
formal parameters of methods, the global variables used within these methods and the number of
active threads. Regression data consists of the size of these inputs. /T_he technique used to obtain
these sizes depends on the type of input. Currently, we use the following heuristics:
Primitive types: the size of a primitive type is its own value.
Wrappers: types such as Integer or Double, which work as wrappers of primitive types, give us a
size through their value() methods, e.g., intValue() for Integer.
Arrays and Strings: we derive the size of such types via the length property.
Collections: we derive the size of collections by invoking their size() method.
Other classes: we search within the declaration of the type, or in any of its super-types, for a
method called size(); otherwise, we search for a property called length. If such names are
not to be found, an error ensues.
Example 3.6. Figure 11 shows two instrumented programs. Pro/f_iling code is inserted in the
programs’ intermediate representation –source code is used only for readability. Instrumentation is
performed by a singleton objectInstrumenter, which stores “bundles” of data. Each bundle contains
an identi/f_ier, a hardware con/f_iguration, the independent variables of the adaptive method, and the
runtime for those variables. Identi/f_iers map methods to bundles. Multiple invocations of the same
method will produce one bundle per call.
3.2.1 Profiling, Logging and Training. Currently, we use a pro/f_iling infrastructure wri/t_ten as
a combination of Java code and bash scripts. /T_he part implemented in Java consists of a service
that runs the program that we want to optimize in a controlled environment. /T_his driver has two
responsibilities: warming up the target program and changing hardware con/f_igurations before
every pro/f_iling experiment.
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
Mapping in Parallel Heterogenous Architectures 1:13
void visit(ﬁnal int NT) throws ... {
  Bundle b = new Bundle(0xFF4AC08D);
  b.addConﬁg(getCurrentConﬁg());
  b.addInt(visited.length); // array
  b.addInt(graph.size()); // class has size()
  b.addInt(NT); // primitive type
  Instrumenter.save(b);
  b.startTime();
  Vector<Visitor> bots = new Vector<Visitor>(NT);
  for (int i = 0; i < NT; i++) {
    bots.add(new Visitor(graph, i));
  }
  for (Visitor v : bots) { v.start(); }
  for (Visitor v : bots) { v.join(); }
  b.stopTime();
}
void count(ﬁnal int START, ﬁnal int END) {
  Bundle b = new Bundle(0xFF4AC08E);
  b.addConﬁg(getCurrentConﬁg());
  b.addInt(START); // primitive type
  b.addInt(END); // primitive type
  b.addInt(forkJoinPool.getActiveThreadCount());
  Instrumenter.save(b);
  b.startTime();
  for (int j = START; j <= END; j++) {
    SingleCounter aux = counters[elements[j]];
    synchronized (aux) {
      aux.value += 1;
   }
  }
  b.stopTime();
}
Fig. 11. Instrumented version of two programs. Grey code is from the original method. (Le/f_t) Breadth-first
search. (Right) Sorting application. (taken from da Silva et al. (2020))
J/i.sc/n.sc/n.sc/hyphen.scCreceives an annotated program P, a set of diﬀerent inputs I = {ι1, ι2, . . . ,ιm }of P, and
a set of acceptable hardware con/f_igurationsH = {h1, h2, . . . ,hn }. It will test the program a pre-
determined number of times for each pair (h, ι), h ∈H, ι∈I. /T_he best con/f_iguration for each input
ιis chosen among the most frequent winner. /T_he objective function that determines the winner
is con/f_igurable. Currently, we consider time, energy consumption and energy-delay product. In
case of ties, we choose the con/f_iguration with the least resources. Resources are ordered according
to the number of big cores, the number of LITTLE cores, the frequency of the big cores and the
frequency of the LITTLE cores, in this sequence.
1.0
1.2
1.4
1.6
1.8
1.0
1.5
2.0
2.5
0.00
0.05
0.10
0.15
0.20
0.02
0.03
0.04
0.05
0.06
0.07
0.08
1b0L
4b0L
4b2L
4b4L
1b0L
4b0L
4b2L
4b4L
1b0L
4b0L
4b2L
4b4L
1b0L
4b0L
4b2L
4b4L
4 10100.0004 1001.000410.00010081.00010.000
1000 0100 0010 0001
Fig. 12. Training output produced by the driver on a few inputs seen in Figure 7. Y-axis is runtime.
Example 3.7. Figure 12 shows a typical output produced during J/i.sc/n.sc/n.sc/hyphen.scC’s training phase, con-
sidering runtime as the objective function. In this experiment, each pair formed by a hardware
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
1:14 Ribeiro, et al.
con/f_iguration and an input is sampled ten times. Vectors at the bo/t_tom of Figure 12 are the inputs
passed to function T/a.sc/s.sc/k.sc(Fig. 1). /T_hese vectors are the independent variables in the regression
model. Vectors at the top of Figure 12 are the best con/f_igurations. /T_hese vectors will give us the
dependent variables used in the regression.
Complexity of the Training Phase.Training involves running the target program in every
valid hardware con/f_iguration with every available input. /T_herefore, the asymptotic complexity of
this phase is determined by the number of valid hardware con/f_igurations, the number of input
sets available to run the program and the complexity of the program itself. In other words, the
instrumented program P will have to be pro/f_iled|H |×| I |times; H being the set of valid hardware
con/f_igurations andI the set of available inputs. /T_he third column of Table 1 (Page 17) reports
training times observed when evaluating the benchmarks used in this paper. /T_he longest time was
225 minutes; the shortest 21 minutes. Section 4.1 provides more details about these costs.
3.3 Generation of Adaptive Code
/T_he product of training is a matrixΘ of /f_loating-point numbers.Θ is hardcoded into the production
code that we want to optimize. Such step happens in the phase labeled “add prediction instrumen-
tation” in Figure 5. /T_he instrumentation that we add into a functionf of interest evaluates the
expression σ(Ai Θ), where Ai is a 1 ×n vector. /T_he size ofAi is one plus the number of inputs of
the target function. /T_he expressionσ(Ai Θ)yields a 1 ×k vector of probabilities, whose elements
add up to 1.0, where k is the number of hardware con/f_igurations considered as targets. /T_he largest
element within σ(Ai Θ)determines the next con/f_iguration that will be used during the current
invocation of f . /T_herefore, the complexity of choosing a hardware con/f_iguration is proportional to
the number of function arguments (n) times the number of valid hardware con/f_igurations (k). As
we shall discuss in Section 4, this time tends to be too short to be of practical consequence.
Example 3.8. Figure 13 shows the production version of our running example, the function
T/a.sc/s.sc/k.sc, originally seen in Figure 1. /T_he dashed box outlines the code that we add toT/a.sc/s.sc/k.scto change
the current hardware con/f_iguration. We show, on the right of the /f_igure, the key methods used
to change and restore the current hardware con/f_iguration. /T_he matrixΘ seen in the production
version of function T/a.sc/s.sc/k.scwas found a/f_ter training, as Example 3.4 explains.
4 EVALUATION
/T_his section demonstrates the eﬀectiveness of our technique when optimizing bytecodes that run
on top of the Java Virtual Machine. To this end, we shall provide answers to the following research
questions:
RQ1 – Training:what is the training time required by J/i.sc/n.sc/n.sc/hyphen.scC, and how does it compare to
CHOAMP’s?
RQ2 – Speed:what is the speed gain obtained byJ/i.sc/n.sc/n.sc/hyphen.scCwhen compared to competing techniques?
RQ3 – Energy:what is the energy improvement thatJ/i.sc/n.sc/n.sc/hyphen.scCdelivers on top of existing techniques?
RQ4 – Convexity:is convexity a common property of the space formed by the mapping of
function arguments to ideal hardware con/f_igurations?
We compare J/i.sc/n.sc/n.sc/hyphen.scCwith two approaches: Sreelatha et al. (Sreelatha et al. 2018)’s CHOAMP,
and ARM’s GTS (Jeﬀ 2013). GTS, short forGlobal Task Scheduling, is Linux’ heterogeneity-aware
scheduler in our big.LITTLE system. Yet, before delving into numbers, we introduce the runtime
environment we have used to evaluate J/i.sc/n.sc/n.sc/hyphen.scC.
/T_he Hardware.Experiments were performed in an Odroid XU4 development board. /T_his device
is powered by a Samsung Exynos 5422 SoC with four ARM Cortex A15 cores, running at up to
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
Mapping in Parallel Heterogenous Architectures 1:15
// Returns the product A×θ
double[] mul(double[] A, double[][] θ);
// Applies the σ function onto d
double[] softmax(double[] d);
// Returns the index that holds the largest
// value within vector pred
int indexLargestElement(double[] pred);
// Get the i-th hardware conﬁguration
Conﬁg getConﬁguration(int i);
// Get the conﬁguration currently in use
Conﬁg getCurrentConﬁguration();
// Change the conﬁguration currently in use
// to the new conﬁguration g
void changeConﬁguration(Conﬁg g);
void task(Stream<Value> s, long keySize) {
  double Theta[][] = {{-0.0125, 0.0114, 0.0006, -0.6481},
                                   {-0.1964, 0.0472, 0.0166, -0.0759},
                                   {-0.1763, -0.0008, 0.0000, 0.0002},
                                   {0.0010, -0.0003, -0.0050, 0.0000}};
  double A[] = {1.0, s.size(), keySize, Thread.activeCount()};
  double P[] = Regression.softmax(Regression.mul(A, Theta));
  int i = indexLargestElement(P);
  Conﬁg originalConﬁg = Regression.getCurrentConﬁguration();
  Conﬁg conﬁg = Regression.getConﬁguration(i);
  Regression.changeConﬁguration(conﬁg);
  while (!s.empty()) {
    BigInteger key = getNextKey(keySize);
    synchronized(globalMap) {
      Value value = s.next();
      globalMap.put(key, value);
    }
  }
  changeConﬁguration(originalConﬁg);
}
Fig. 13. The production version of function T/a.sc/s.sc/k.sc, first seen in Figure 1.
2.0GHz, and four Cortex A7 cores running at up to 1.5GHz. /T_he board features 2GB of LPDDR3
RAM. To measure the energy consumed exclusively by speci/f_ic functions, we send signals to the
synchronization circuit seen in Figure 4-a through one of the board’s GPIO pin. Code to emit the
signal is inserted into Java bytecodes via S/o.sc/o.sc/t.sc, immediately before the invocation of a function of
interest, and immediately a/f_ter that function returns. We use the energy measurement framework
proposed by Bessa et al. (Bessa et al. 2017). Power is measured by a National Instruments DAQ USB
6009 device, at a rate of 12,000 samples per second.
/T_he So/f_tware Stack.We use Oracle’s OpenJDK/JRE 11 LTS and Soot 3.2.0 to process bytecodes. No
modi/f_ications have been made in the Java Virtual Machine –transformations performed by either
J/i.sc/n.sc/n.sc/hyphen.scCor CHOAMP happen at the bytecode level, and are carried out via Soot. To mitigate the
eﬀect of JIT compilation in the execution time of benchmarks, each application has a warm-up stage
before actual execution (details in Table 1). We have used Python 3.4 and Scikit Learn (Pedregosa
et al. 2011) to implement regression. Python was also used, in addition to GNU Bash 4.4.19, to
generate the suite of micro-benchmarks used to train CHOAMP. /T_he Operating System in the
Odroid XU4 used in our experiments is the GNU/Linux Ubuntu 18.04 LTS with kernel 4.17.
/T_he Benchmark Suite./T_his paper uses the 18 benchmarks shown in Table 1. Eight of them were
taken from Acar et al. (Acar et al. 2018), who had selected nine programs from Problem Based
Benchmark Suite (PBBS) (Shun et al. 2012) to evaluate concurrency models. /T_he version of PBBS
used by Acar et al. (Acar et al. 2018) was implemented in C/C++, so we had to re-implement all the
benchmarks in Java. We removed D/e.sc/l.sc/a.sc/u.sc/n.sc/a.sc/y.scT/r.sc/i.sc/a.sc/n.sc/g.sc/u.sc/l.sc/a.sc/t.sc/i.sc/o.sc/n.scfrom our collection, because we could
not ensure that its parallel implementation always produces the same output. /T_he triangulation
varies depending on how threads are scheduled; hence, the output of diﬀerent versions of this
benchmark might not be equal.
We chose six programs from the Renaissance benchmark collection, which was recently released
by Prokopec et al. (Prokopec et al. 2019). Renaissance contains 21 benchmarks. All the programs in
that collection come with only one set of input values. We chose only six benchmarks because we
had to understand and augment each program with more inputs. We have also added veri/f_ication
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
1:16 Ribeiro, et al.
code to these benchmarks to check execution correctness. /T_he six benchmarks that we chose
are implemented in Scala. Our criterion when picking up programs was simplicity: we selected
benchmarks that were easy to modify. We have opted for Scala programs to demonstrate that
J/i.sc/n.sc/n.sc/hyphen.scCcan deal well with languages other than Java.
In addition to PBBS and Renaissance, J/i.sc/n.sc/n.sc/hyphen.scCis distributed with four typical parallel algorithms.
C/o.sc/l.sc/l.sc/i.sc/n.sc/e.sc/a.sc/r.scP/o.sc/i.sc/n.sc/t.sc/s.scand /r.sc/a.sc/n.sc/d.sc/o.sc/m.scN/u.sc/m.scC/o.sc/m.sc/p.scwere taken from public repositories; H/a.sc/s.sc/h.scS/y.sc/n.sc/c.sc, was
adapted from Butcher’s book; and/i.sc/n.sc/s.sc/e.sc/r.sc/t.scA/n.sc/d.scA/d.sc/d.scwas adapted from Zhang et al. (2020)’s Figure 3.
Table 1 presents an overview of the benchmarks, as well as basic characteristics of their code.
We recall an observation already made by Prokopec et al., when introducing the Renaissance
Benchmark Suite: the source code of the benchmarks is relatively small; however, they cause
the invocation of hundreds of diﬀerent methods, potentially millions of times. Many of these
methods belong to the Java Standard Library (e.g., java.util.*, java.lang.System, etc). /T_hree
of the Renaissance programs: textscAls, C/h.sc/i.scS/q.sc_u.sc/a.sc/r.sc/e.scand D/e.sc/c.scT/r.sc/e.sc/e.scalso rely heavily on Apache’s
spark engine. /T_he two last columns of Table 1 report the static and dynamic count of methods
invoked by each benchmark. To derive these numbers, we use Oracle’sVisualVM pro/f_iler on the
default inputs of each benchmark. Notice that numbers produced through this pro/f_iling technique
cannot be exactly reproduced: small variations are expected due to internal operations of the Java
Virtual Machine, concerning, for instance, the activation of the garbage collector or the just-in-time
compiler. Example 4.1 clari/f_ies this observation forS/p.sc/a.sc/n.sc/n.sc/i.sc/n.sc/g.scF/o.sc/r.sc/e.sc/s.sc/t.sc. /T_he choice of benchmark in
this case is arbitrary, because the other programs that we evaluate show similar behavior.
Example 4.1. /T_he benchmark calledS/p.sc/a.sc/n.sc/n.sc/i.sc/n.sc/g.scF/o.sc/r.sc/e.sc/s.sc/t.scis part of the Problem Based Benchmark
Suite. It computes a series of minimum spanning trees –one per connected component– of
undirected graphs. /T_his benchmark’s source code contains 410 lines of code, grouped into 31
methods organized in /f_ivejava /f_iles. We adapt the method called “benchmark”, which receives
a list of edges and an integer denoting the number of threads that will be created to process
these edges. By default, this benchmark receives a graph with 120 nodes and 1,000 edges. With
these default inputs, “benchmark” causes the invocation of 1,905 diﬀerent methods. Out of this
lot, 1,874 are either part of the Java Standard Library (in classes like java.util.ArrayList or
java.util.concurrent.Executors), or are part of the Java Virtual Machine. In a single run of
this benchmark with default inputs, these methods were invoked 106,462 times. /T_hese numbers,
1,874 and 106,462, might vary due to internal operations of the JVM.
Available Inputs.Each benchmark comes with a default input set. We have augmented every one
of them with 13 additional inputs. We tried, as best as our knowledge of the benchmark allowed it,
to maximize the diversity of inputs, having data of diﬀerent sizes. Ten of these inputs, randomly
chosen, are used for training. When evaluating the model trained for a benchmark, we use four
unseen and randomly chosen inputs. Sections 4.2 and 4.3 further discuss the impact of diﬀerent
inputs in the execution time and energy consumption of the applications. /T_he separation of inputs
into training and evaluation sets is random.
Choice of regression target.We optimize one method per benchmark: the routine invoked by
the benchmark’smain function. /T_his approach is equivalent to doing regression on inputs of the
whole program. In other words, in practice, we are tuning the entire program; not a single method
of it. As previously mentioned, each of these adapted functions will, during a normal execution of
the benchmark, cause the invocation of many methods external to the benchmark’s source code.
All these activations will run in the hardware con/f_iguration determined by the regression model.
To give the reader some perspective on this approach, Table 1 contains in the last two columns the
number of diﬀerent functions that are aﬀected by the change in hardware con/f_iguration (stCalls),
plus the number of times that these functions are dynamically invoked (dyCalls).
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
Mapping in Parallel Heterogenous Architectures 1:17
Source Benchmark TTime Lang. LoC W stCalls dyCalls
J/i.sc/n.sc/n.sc/hyphen.scCcollinearPoints 32m1 J 565 3 576 3,724
J/i.sc/n.sc/n.sc/hyphen.scChashSync 94m7s J 73 3 649 3,838
J/i.sc/n.sc/n.sc/hyphen.scCinsertAndAdd 47m30s J 130 4 645 47,978
J/i.sc/n.sc/n.sc/hyphen.scCrandomNumComp 26m7s J 89 6 631 3,399
(Shun et al. 2012) bfs 42m33s J 353 4 736 13,713,590
(Shun et al. 2012) radixSort 20m51s J 501 4 2,045 13,821
(Shun et al. 2012) sampleSort 26m17s J 414 3 625 3,173
(Shun et al. 2012) suﬃxArray 30m12s J 316 3 288 162,976,109
(Shun et al. 2012) removeDuplicates 30m31s J 174 4 1,204 12,341
(Shun et al. 2012) convexHull 56m30s J 499 5 246 28,488,450
(Shun et al. 2012) nearestNeighbors 30m29s J 715 3 1,335 3,664,153
(Shun et al. 2012) spanningForest 21m40s J 410 4 1,905 106,462
(Prokopec et al. 2019) als 80m12s S/J 97 1 165,108 2,123,294
(Prokopec et al. 2019) philosophers 21m15s S/J 146 1 10,446 96,129
(Prokopec et al. 2019) futureGenetic 26m8s S/J 115 1 909 8,426
(Prokopec et al. 2019) /f_inagleHTTP 225m10s S/J 119 1 785 1,554
(Prokopec et al. 2019) chiSquare 27m15s S/J 101 1 185,202 3,523,326
(Prokopec et al. 2019) decTree 64m22s S/J 129 1 836,283 5,717,648
Table 1. Benchmarks used for evaluating J/i.sc/n.sc/n.sc/hyphen.scC. Column TTime shows time to train each benchmark, which
will be further explained in Section 4.1. Lang. contains the source language of benchmarks: J stands for
Java and S for Scala. Column W shows the number of warm-up executions performed by each application.
StCalls approximates the number of methods invoked by a benchmark running with default inputs; dyCalls
approximates the number of times that those functions are invoked. These numbers were produced with
Oracle’sVisualVM profiler; hence, small variations are to be expected. /c.sc/o.sc/l.sc/l.sc/i.sc/n.sc/e.sc/a.sc/r.scP/o.sc/i.sc/n.sc/t.sc/s.scfinds three points
on the same line; /h.sc/a.sc/s.sc/h.scS/y.sc/n.sc/c.scinserts in a concurrent table; /r.sc/a.sc/n.sc/d.sc/o.sc/m.scN/u.sc/m.scC/o.sc/m.sc/p.schas several long sequences of
branches that are hard to predict; and /i.sc/n.sc/s.sc/e.sc/r.sc/t.scA/n.sc/d.scA/d.sc/d.scimplements parallel operations on a Database.
On the Choice of Hardware Con/f_igurations.When training J/i.sc/n.sc/n.sc/hyphen.scCand CHOAMP, we follow
the methodology proposed by Sreelatha et al (Sreelatha et al. 2018). /T_hus, we consider a universe of
six con/f_igurations: 4b4L (4 big and 4 LITTLE cores), 4b0L, 0b4L, 2b2L, 2b0L and 0b2L. LITTLE cores
run at maximum frequency: 1.5GHz. Big cores are statically set to run at either 1.6GHz or 1.8GHz
(instead of using the maximum 2.0GHz frequency level) due to known thermal issues (da Silva et al.
2019, Sec-4.2). GTS is allowed to choose among any possible hardware con/f_iguration involving
big and LITTLE cores, and the diﬀerent frequency levels available in the hardware. For the
sake of reproducibility and to be/t_ter understand the impacts of our technique, we have disabled
Dynamic Voltage and Frequency Scaling (DVFS) when using either J/i.sc/n.sc/n.sc/hyphen.scCor CHOAMP, but not
GTS. Regardless of the methodology used to choose hardware con/f_igurations, thread scheduling
uses Linux’ defaultHeterogeneous Multi-Processor (HMP) scheduler (Rezki and Wool 2015), which is
integrated into the kernel’sCompletely Fair Scheduler (CFS). /T_hus, threads might switch between
big and LITTLE cores. /T_he higher the CPU utilization of a thread, the higher the likelihood that it
will run on a big core.
On the Implementation of CHOAMP.CHOAMP is a system that, diﬀerent from our approach,
relies on the syntax of the program text –and on its implied semantics– to predict ideal hardware
con/f_igurations. CHOAMP represents this text of code as a set of characteristics that are useful for
training and prediction. Such characteristics, also called prime features by its authors, are split into
two diﬀerent groups: language dependent and independent. Language independent features, such
as number of branches or memory accesses, are easier to identify and port, as they tend to appear in
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
1:18 Ribeiro, et al.
Prime Feature Lang.Dep. OpenMP Java VM
Branch operations No - -
Memory operations No - -
Atomic operations Yes omp atomic atomic
Barriers Yes omp barrier CyclicBarrier, Phaser
Critical Sections Yes omp critical Synchronized
False Sharing No - -
Flush operations Yes omp /f_lush not used
Table 2. Prime features and their correspondent Java VM implementation.
most languages. On the other hand, features that depend on a speci/f_ic programming language need
to be adapted when porting the technique to new environments. CHOAMP was initially designed
to work with OpenMP applications implemented in C; therefore, some of the prime features used
by Sreelatha et al. depend on OpenMP constructs. Our re-implementation of CHOAMP targets
Java applications running on Hotspot; thus, some of its features had to be adapted to our needs.
Table 2 presents the list of program characteristics originally used by CHOAMP for OpenMP and
the new version of them, adapted to the JVM.
Most language dependent features /f_ind correspondents in the Java standard library, as is the case
of the omp atomic pragma, which we derived from classes in the package java.util.concurrent.
atomic. For instance, the occurrence of method incrementAndGet(), from the AtomicInteger
class, would add an “Atomic Operation” to the feature vector of the function whereincrementAnd
Get() is invoked. However, some features like /f_lush operations, proposed by Sreelatha et al. (Sree-
latha et al. 2018), were not reused in our implementation, due to a lack of correspondents.
Training and TuningFollowing Sreelatha et al., we have trained the probabilistic model of
CHOAMP by running it on a set of generic micro-benchmarks. As the original training set was
wri/t_ten in C and OpenMP, we had to create a new training set that suits Java. /T_he micro-benchmarks
we used were directly based on the scripts made public by Sreelatha et al. /T_hese scripts generate
hundreds of micro-benchmarks. /T_he user adjusts the intensity of each prime feature through
command line inputs. We used the originals generator scripts at h/t_tps://bitbucket.org/jkrishnavs/
openmp-eigenbench, adjusting the code to Java. We also used the same range and intensity of
features as used in the original work of CHOAMP. Sreelatha et al. have proposed three diﬀerent
regression models for CHOAMP. We have experimented with all of them, and end up choosing the
linear /f_it, because, in our setup, it yields be/t_ter results than the /Q_uadratic and Gaussian predictors.
/T_his result in on par with the /f_indings of Sreelathaet al.
4.1 RQ1: Training
J/i.sc/n.sc/n.sc/hyphen.scCand CHOAMP require training to adjust the parameters of the regression models. While this
cost is paid once by CHOAMP, when performing the training over a set of generic micro-benchmarks,
J/i.sc/n.sc/n.sc/hyphen.scCpays this cost for each application that it optimizes. CHOAMP uses micro-benchmarks for
training; J/i.sc/n.sc/n.sc/hyphen.scCuses the application itself. /T_he training time of CHOAMP is computed over a set of
285 micro-benchmarks over all the hardware con/f_igurations mentioned in our experimental setup.
In our hardware, we took about 780 minutes to train our implementation of CHOAMP.
To train J/i.sc/n.sc/n.sc/hyphen.scC, we follow the methodology described in Section 3.2.1. J/i.sc/n.sc/n.sc/hyphen.scC’s training time
depends on the target application’s run time, and on the number of available inputs. Table 1 shows
the training time of each benchmark. Using ten inputs and ten allowed hardware states (clock
speed ×hardware con/f_igurations) per benchmark, we took 903 minutes to train the 18 programs
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
Mapping in Parallel Heterogenous Architectures 1:19
used in this section. /T_he longest time, three hours and 45 minutes was spent in Renaissance’s
/f.sc/i.sc/n.sc/a.sc/g.sc/l.sc/e.scHTTP. PBBS’s/r.sc/a.sc/d.sc/i.sc/x.scS/o.sc/r.sc/t.scgave us the fastest training time: 20 minutes and 51 seconds.
Once the benchmark is trained, no further pre-processing is required. /T_he product of training, the
code earlier seen in Figure 13, is embedded directly into a program’s bytecode, and runs in constant
time. Training’s overhead bears no impact once the optimized code runs. /T_he only overhead that is
imposed onto optimized programs comes from the matrix multiplication that happens once a hot
function is invoked, as we have discussed in Section 3.3. As we will see in Section 4.2, this runtime
overhead is too low to be reliably measured.
4.2 RQ2: Speed
Figure 14 summarizes the comparison of the three diﬀerent schedulers, when the objective function
that J/i.sc/n.sc/n.sc/hyphen.scCand CHOAMP minimize is the execution time of target applications. We have tested
each benchmark with four input sets, adopting a signi/f_icance levelα = 0.05; i.e., a con/f_idence level
of 95%. So, if the results reported by, for instance, J/i.sc/n.sc/n.sc/hyphen.scCand CHOAMP cannot be distinguished
with a con/f_idence of more than 95%, then we consider them as originating from the same population.
/T_hus, we use Student’s Test to measure the p-value of two populations, and consider signi/f_icant
results with a p-value less than 0.05. White boxes with le/t_ters in Figure 14 identify the technique
which achieved the best result for a combination of benchmark and input. J stands for J/i.sc/n.sc/n.sc/hyphen.scC, C
for CHOAMP and G for GTS; X means that the winning systems have produced results statistically
similar (p-value greater than 0.05). Above each input set, we show the con/f_iguration thatJ/i.sc/n.sc/n.sc/hyphen.scC
chose for that input. /T_he grey box, at the right of the name of each benchmark, is the con/f_iguration
that CHOAMP chooses for that benchmark.
We notice that in 26 cases, out of 72 combinations of [benchmarks ×inputs], J/i.sc/n.sc/n.sc/hyphen.scCachieved
be/t_ter results when compared to the other techniques. In other 42 cases,J/i.sc/n.sc/n.sc/hyphen.scCwas at least as
fast as GTS or CHOAMP. CHOAMP, in turn, accounted for 3 best results, and GTS for only one, in
/h.sc/a.sc/s.sc/h.scS/y.sc/n.sc/c.sc’sI/n.sc4. /T_hese numbers show thatJ/i.sc/n.sc/n.sc/hyphen.scCusually outperforms the two other techniques
considered in this paper; however, J/i.sc/n.sc/n.sc/hyphen.scC’s performance still depends on a good choice of inputs
for training. Example 4.2 highlights this fact. Further discussion about the importance of /f_inding
good training sets shall appear in Example 4.6 (Page 27).
Example 4.2. J/i.sc/n.sc/n.sc/hyphen.scCperformed rather poorly in /c.sc/o.sc/l.sc/l.sc/i.sc/n.sc/e.sc/a.sc/r.scP/o.sc/i.sc/n.sc/t.sc/s.sc, due to an unfortunate choice
of training inputs. Indeed, the 10 training inputs chosen when optimizing /c.sc/o.sc/l.sc/l.sc/i.sc/n.sc/e.sc/a.sc/r.scP/o.sc/i.sc/n.sc/t.sc/s.sc/f_ind
in 4/b.sc4Ltheir best con/f_iguration. However, coincidentally, three of the test inputs ask for4/b.sc0L. It
suﬃces to switch one of the test and training inputs to put J/i.sc/n.sc/n.sc/hyphen.scCon pair with the other schedulers.
/T_his experiment shows that con/f_igurations impact in non-trivial ways the behavior of applications.
All the winning con/f_igurations, regardless of the technique, converged to the frequency of 1.8GHz
whenever at least one big core was present. /T_he most recurring con/f_igurations were 4b4L (16x for
CHOAMP and 37x for J/i.sc/n.sc/n.sc/hyphen.scC), 0b4L (2x/11x), 4b0L (17x for J/i.sc/n.sc/n.sc/hyphen.scConly), 2b0L (4x for J/i.sc/n.sc/n.sc/hyphen.scConly),
and 0b2L (2x for J/i.sc/n.sc/n.sc/hyphen.scConly). Example 4.3 further emphasizes the importance of the hardware
con/f_iguration in the behavior of a program.
Example 4.3. Considering /c.sc/h.sc/i.scS/q.sc_u.sc/a.sc/r.sc/e.sc’s last input (/w.sc/o.sc/r.sc/k.sc/e.sc/r.sc/s.sc= 2, SIZE = 1, 023, 464), J/i.sc/n.sc/n.sc/hyphen.scC
prediction of the con/f_iguration 4b0L led to a mean run time of 8.18 seconds, while CHOAMP
decision led to 8.47 and GTS to 9.00. For its second input ( /w.sc/o.sc/r.sc/k.sc/e.sc/r.sc/s.sc= 4, SIZE = 2, 250, 467), we
observed J/i.sc/n.sc/n.sc/hyphen.scCprediction (4b0L) leading to mean run time of 17.00 seconds, CHOAMP to 18.70
and GTS to 17.76. In every case, p-values comparing these populations were less than 0.008.
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
1:20 Ribeiro, et al.
● ●
●
●
●
●
●
2
3
4
5
In1 In2 In3 In4
collinearPoints 
●
●
●●● ●
●
●●
●
●
● ●
0
2
4
6
8
In1 In2 In3 In4
randomNumComp 
●
●
0
1
2
3
4
5
In1 In2 In3 In4
hashSync 
●
●● ● ●
0
10
20
30
40
In1 In2 In3 In4
bfs 
●
●
●●
●
●
●
0.050
0.075
0.100
0.125
0.150
In1 In2 In3 In4
spaningForest 
●● ●●0
10
20
30
In1 In2 In3 In4
insertAndAdd 
●
●
●
●
● ●
1
2
In1 In2 In3 In4
nearestNeighbors 
●
●
●
●●
●
●
●
●
0.4
0.8
1.2
1.6
In1 In2 In3 In4
convexHull 
●
●
1.5
2.0
2.5
3.0
In1 In2 In3 In4
removeDuplicates 
●
● ●●
●● ●
1
2
In1 In2 In3 In4
suffixArray 
●
●● ●
●
● ●●● ● ●
0
5
10
15
In1 In2 In3 In4
sampleSort 
● ●
●
●
●
0.2
0.4
0.6
In1 In2 In3 In4
radixSort 
●●
●
●
10
20
30
In1 In2 In3 In4
philosophers 
●
● ●
●
20
40
60
80
In1 In2 In3 In4
futureGenetic 
●
●
●
●
1.0
1.5
2.0
2.5
In1 In2 In3 In4
finagleHttp 
●
●
●
●
●
27.5
30.0
32.5
35.0
37.5
In1 In2 In3 In4
decTree 
●
●
●
●
10
15
In1 In2 In3 In4
chiSquare 
●
●
●
●20
40
60
80
In1 In2 In3 In4
als 
JJJ
J
X
X
4b0L 4b0L 4b0L
4b4L               
4b4L
X X
4b4L               4b4L               
4b4L               4b4L               4b4L               
0b4L               
0b2L 0b2L 0b4L 4b4L
4b4L               4b4L               
4b4L
4b4L               
4b0L 4b4L 4b0L 4b4L
4b4L               
0b4L 0b4L 0b4L 0b4L
4b4L               
4b4L
XX
X X
X X
J
X X X
J X
X
X X
X
G
J
J J
J
CX
4b4L 4b0L 4b4L 4b4L 4b4L 4b4L 4b4L 4b4L
0b4L 4b4L 4b4L 4b4L 4b0L 4b0L 4b0L 4b0L 4b4L 4b4L 4b4L 4b4L
4b4L 4b4L 4b4L 4b4L
4b4L 4b4L 4b4L
4b4L               
4b0L 4b4L 4b4L 4b4L
4b4L               
2b0L 4b0L 2b0L 4b0L
4b4L               
4b4L 4b4L 4b4L 4b4L
0b4L               
4b0L 4b0L 4b0L 4b0L
4b4L               
4b0L 0b4L 0b4L 0b4L
4b4L               
2b0L 2b0L 0b4L 0b4L
X
X
X X J
J
J J X X X C
X X X
X J
J J J J J
J
X
J
X X
XJ
J
C
X
J
XX
X
4b4L 4b4L 4b4L
J
XX
XX
Fig. 14. Execution time of benchmarks from Table 1. Y -axis shows time in seconds. X-axis shows diﬀerent
experiments; each experiment uses diﬀerent inputs. Boxplots are ordered by J/i.sc/n.sc/n.sc/hyphen.scC, CHOAMP and GTS.
4.3 RQ3: Energy
Figure 16 compares CHOAMP, GTS and J/i.sc/n.sc/n.sc/hyphen.scCregarding energy consumption. /T_he clock speed of
1.6GHz was the most common among all the schedulers, except for one input set of /r.sc/a.sc/d.sc/i.sc/x.scS/o.sc/r.sc/t.sc,
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
Mapping in Parallel Heterogenous Architectures 1:21
when CHOAMP chose to use 1.8GHz. GTS can choose any possible frequency levels from 200MHz
to 1.8GHz in the big cluster and from 200MHz to 1.5GHz in the li/t_tle one. /T_his /f_lexibility may lead
to performance degradation because GTS increases frequency gradually, even in computation-
intensive programs. Even with several warm-up rounds, GTS might take an excessively long time
to achieve maximum frequency levels for some applications. /T_hus,J/i.sc/n.sc/n.sc/hyphen.scCoutperforms GTS mostly
due to its ability to choose high-performance hardware con/f_igurations, such as 4b4L at 1.6GHz,
immediately. GTS, in turn, needs a warm-up period to arrive at them.
Figure 16 shows that J/i.sc/n.sc/n.sc/hyphen.scCachieved the best results in 20 experiments (out of 72); GTS won in
2, and CHOAMP in 6. In 44 experiments there was no clear winner. /T_his diﬃculty to pinpoint a best
technique is due to the fact that we measure energy for the entire board. /T_herefore, peripherals
like the fan and the memory bus increase the variance of results. /T_his decision was no accident: we
believe that measuring energy for the entire boards provides a more realistic assessment of the
behavior of our techniques if adopted in production. CHOAMP has chosen the 0b4L con/f_iguration
at 1.6GHz for almost all the samples in this evaluation. /T_his behavior is due to some features,
such as branching and memory operations, dominating the others in most of the functions that
constitute a benchmark. It is possible to improve this behavior by scaling the relative importance
of the features; however, this optimization is out of the scope of this work.
On the In/f_luence of Execution History.Execution history impacts the energy consumed by diﬀerent
programs. Take as an example the entry corresponding to/h.sc/a.sc/s.sc/h.scS/y.sc/n.sc/c.scin Figure 16. Although J/i.sc/n.sc/n.sc/hyphen.scC
and CHOAMP predicted the same con/f_iguration for the second input set (In2), the former consumed
marginally more energy. /T_his behavior is even more surprising once we consider thatJ/i.sc/n.sc/n.sc/hyphen.scC’s
and CHOAMP’s code run within about the same time, as Figure 14 reveals. /T_he culprit of this
counter-intuitive result is the board state at the time measurement started. /T_he warm-up phase, in
this case, is responsible for giving J/i.sc/n.sc/n.sc/hyphen.scC’s and CHOAMP’s code diﬀerent starting states. In the
discussion that follows, we shall separate the execution of a benchmark into two parts: warm-up,
when the target routine is called a number of times to stabilize the Java Virtual Machine; and
measurement, when the behavior of the benchmark is actually gauged.
Hysteresis. Figure 15 shows the power pro/f_ile of/h.sc/a.sc/s.sc/h.scS/y.sc/n.sc/c.sc, including warm-up and measurement
phases. Invocations in the warm-up stage use diﬀerent inputs than in the measurement stage. As
a result, our technique predicted the con/f_iguration 4b4L for the last warm-up invocation, which
is diﬀerent than 0b4L, the con/f_iguration predicted at measurement. /T_he use of big cores during
warm-up increased the amount of energy consumed by the board in the measurement step, due to
a well-known phenomenon: the hysteresis of power dissipation.
/T_he mean power dissipated byJ/i.sc/n.sc/n.sc/hyphen.scC’s version of/h.sc/a.sc/s.sc/h.scS/y.sc/n.sc/c.scin Figure 15(a) was 4.68W. CHOAMP’s
was 4.09W, as seen in Figure 15(b). J/i.sc/n.sc/n.sc/hyphen.scC’s program consumes more energy (9.47J vs 8.22J). How-
ever, if we /f_ix the hardware con/f_iguration in the warm-up phase ofJ/i.sc/n.sc/n.sc/hyphen.scC’s code, then the average
dissipation goes down to 3.95W. Figure 15(c) reports the power pro/f_ile of this setup. /T_he only
diﬀerence between the executions of J/i.sc/n.sc/n.sc/hyphen.scCin Figures 15 (a) and (c) is the con/f_iguration used in
the warm-up stage. /T_here is no statistically signi/f_icant diﬀerence between the amount of energy
consumed by CHOAMP and J/i.sc/n.sc/n.sc/hyphen.scCwhen both warm-up with the same hardware con/f_iguration.
/T_his behavior caused by diﬀerences between con/f_igurations chosen at warm-up and measurement
phases only aﬀects J/i.sc/n.sc/n.sc/hyphen.scC. CHOAMP always chooses the same hardware con/f_iguration per function,
and GTS increases frequency gradually. /T_he only further impact that this diﬀerence had inJ/i.sc/n.sc/n.sc/hyphen.scC’s
behavior was observed in /d.sc/e.sc/c.scT/r.sc/e.sc/e.scand /c.sc/o.sc/l.sc/l.sc/i.sc/n.sc/e.sc/a.sc/r.scP/o.sc/i.sc/n.sc/t.sc/s.sc. In both cases, only for the last input set
(In4), and only when measuring running time (Fig. 14). /T_he need to change con/f_iguration when
moving from warm-up to measurement has cost J/i.sc/n.sc/n.sc/hyphen.scC’s code some time. Nevertheless, when
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
1:22 Ribeiro, et al.
1 2 3 4 5 60 1 2 3 4 5 60 2 4 60 8 10
 Warm up
JINN-C 
Predicted
4b4L
Measured run
JINN-C 
Predicted
0b4L
Measured 
runWarm up
(c)
Warm up
Fixed 
Conﬁguration 
0b4L
Measured run
JINN-C 
Predicted
0b4L
Energy = 9.477 J (mean of 10x)  - P-value to (b) = 0.000 
Energy = 8.224 J (mean of 10x) 
Energy = 8.139 J (mean of 10x)  - P-value to (b) = 0.362 
iPower (W)
Time (s)
Time (s)
Time (s)
iPower (W) iPower (W)
(b)
(a)
43210
43210
43210
Fig. 15. Power consumption of hashSync with (a) J/i.sc/n.sc/n.sc/hyphen.scC, (b) CHOAMP, and (c)J/i.sc/n.sc/n.sc/hyphen.scCwith fixed configuration
at warm-up. P-values below 0.05 indicate that executions of J/i.sc/n.sc/n.sc/hyphen.scC’s and CHOAMP’s code are statistically
diﬀerent. For this benchmark, CHOAMP (b) predicted 0b4L as the best configuration for the parallel kernel.
This configuration is used in all warm-up stages and in the measurement phase.
reporting the results in this paper, we have opted to let the hardware con/f_iguration /f_luctuate during
warm-up, as this is the expected behavior of J/i.sc/n.sc/n.sc/hyphen.scC, once it is deployed in production.
4.4 RQ4: Convexity
A convex space is a region within a Euclidean Space whose intersection with any line results in a
continuous line segment. If the convex space can be described by a function, then said function is
also called convex. Convex functions are very important in classi/f_ication problems. Firstly, because
exploration methods based on derivatives, like gradient descent, converge to the optimal solution
when applied on them (Boyd and Vandenberghe 2004). Secondly, as we discuss in Section 4.4.2,
disjoint convex sets are linearly separable. /T_hus, linear techniques, like the one employed in this
paper, tend to yield good results when used to classify such sets.
In our se/t_ting, the search space is a function that maps program inputs to hardware con/f_igu-
rations. /T_his function is discrete, because its image is a /f_inite set of hardware con/f_igurations. As
a consequence of convexity, if we /f_ix all the program inputs and vary one of them, then every
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
Mapping in Parallel Heterogenous Architectures 1:23
●
●
●
●
●●
●●
●●
●
●
●
20
30
40
In1 In2 In3 In4
collinearPoints 
●●●
●●●
0
20
40
60
In1 In2 In3 In4
randomNumComp 
●
●
●
●
●
●● ●
●
●
●● ●
0
10
20
30
40
In1 In2 In3 In4
hashSync 
●
●
0
100
200
In1 In2 In3 In4
bfs 
●
●
●
●
●
●
●
●
0.6
0.9
1.2
1.5
In1 In2 In3 In4
spaningForest 
●
●
●
●●
●
●
●0
50
100
150
200
In1 In2 In3 In4
insertAndAdd 
●
●
●
●
●
●●
●
●
●
10
20
30
In1 In2 In3 In4
nearestNeighbors 
●
●
●
●
5
10
In1 In2 In3 In4
convexHull 
●
●
●
●
●
●
10
15
20
25
30
In1 In2 In3 In4
removeDuplicates 
●●
●
●
●
●●● ●
●
●
●
●
5
10
15
20
In1 In2 In3 In4
suffixArray 
● ●
●
●
●
●
●● ●●
●●
0
50
100
In1 In2 In3 In4
sampleSort 
● ● ●
●
●
●
●
●
●
2
4
6
8
In1 In2 In3 In4
radixSort 
●●
● ●
●0
100
200
300
In1 In2 In3 In4
philosophers 
●
●●
●
●
●
●
●●0
100
200
300
400
500
In1 In2 In3 In4
future−genetic 
●
● ●
●
8
12
16
20
In1 In2 In3 In4
finagle−http 
●
●
●
●
●
●
●●200
300
400
500
In1 In2 In3 In4
dec−tree 
●
●●
●
●
●
●
●
●
75
100
125
150
In1 In2 In3 In4
chi−square 
● ●
●
●●
●
● ●●
●
●
200
400
600
In1 In2 In3 In4
als 
JJJ
X
X
4b4L 4b0L 4b0L
0b4L-1.8  
4b4L
X
0b4L               0b4L               
0b4L               0b4L               0b4L               
0b4L               
0b2L 0b2L 0b4L 2b0L
0b4L               0b4L               
2b0L
0b4L               
4b4L 0b4L 0b4L 4b4L
0b4L               
0b4L 0b4L 0b4L 0b4L
0b4L               
0b4L
X
X X
J X
C X
4b4L 0b4L 4b4L 4b4L 4b4L 4b4L 4b4L 4b4L
0b4L 4b4L 4b4L 4b4L 4b0L 4b0L 2b0L 4b0L 2b2L 4b4L 2b2L 2b2L
4b4L 4b4L 4b4L 4b4L
0b4L 4b4L 0b4L
0b4L               
4b0L 4b4L 4b4L 4b4L
0b4L               
4b0L 2b0L 4b0L 2b0L
0b4L               
4b4L 4b4L 4b4L 4b4L
0b4L               
4b0L 4b0L 4b0L 4b0L
0b4L               
4b0L 0b4L 0b4L 0b4L
0b4L               
2b0L 2b0L 0b4L 0b4L
X X X XC
X
X X C CX
X X
X
X
J X
4b4L 4b4L 4b4L
C X
J
G X
J J
J J
X X X
X G
X J J
J J J J
C X X X XX
J J X
X
X X J
J X X
X
X
X
X
X
Fig. 16. Energy consumed by the benchmarks in Table 1.Y -axis shows energy in Joules.X-axis shows diﬀerent
experiments. Boxplots are sorted as in Figure 14.
region covered by the same optimal con/f_iguration is continuous. In other words, while varying
this single input monotonically, we will not leave a region r where a certain con/f_igurationh is
the best, /f_ind a new regionr ′governed by a diﬀerent con/f_igurationh′, only to /f_indh again later,
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
1:24 Ribeiro, et al.
once we cross the boundary between r ′and a third region r ′′. Notice that convexity is a tendency,
not a principle: it is possible to implement programs whose space of optimal con/f_igurations is not
convex, as Example 4.4 demonstrates.
Example 4.4. /T_he space formed by ideal hardware con/f_igurations associated withi in unlikely,
below, is non-convex:
void unlikely(int i) {
if (10 <= i && i <= 100) sync_intensive();
else comp_intensive();}
/T_heunlikely routine receives one input, namely, the integer i. If i is less than 10 or greater
than 100, it invokes a computationally intensive procedure; otherwise, it invokes a synchronization
intensive one. /T_he optimal con/f_igurations for these two pieces of code diﬀer. /T_he space occupied
by the optimal hardware con/f_iguration forcomp intensive, i.e., [−∞, 10[∪], 100, +∞], is non-
continuous; hence, concave.
4.4.1 Space Exploration. /T_he program discussed in Example 4.4 is unlikely to exist in the real-
world. To support this statement, Figure 17 provides a glimpse of the best hardware con/f_igurations
for diﬀerent inputs of four benchmarks in our collection. Each matrix in Figure 17 associates a pair
of inputs with the hardware con/f_igurations that yielded the shortest execution times for those inputs.
In this experiment, we chose the two benchmarks from our collection that contain two inputs. We
have augmented this set with /h.sc/a.sc/s.sc/h.scS/y.sc/n.sc/c.scand /f.sc/u.sc/t.sc/u.sc/r.sc/e.scG/e.sc/n.sc/e.sc/t.sc/i.sc/c.sc, to /f_it the /f_igure into a 2×2 matrix
(for aesthetic reasons). However, to avoid having to draw a 3D-/f_igure, we have /f_ixed one input for
each benchmark: number of Generations for /f.sc/u.sc/t.sc/u.sc/r.sc/e.scG/e.sc/n.sc/e.sc/t.sc/i.sc/c.sc, and number of workers, for
/h.sc/a.sc/s.sc/h.scS/y.sc/n.sc/c.sc. /T_he computational time involved in the production of each /f_igure follows:/r.sc/a.sc/d.sc/i.sc/x.scS/o.sc/r.sc/t.sc:
1 hour and 37 minutes; P/h.sc/i.sc/l.sc/o.sc/s.sc/o.sc/p.sc/h.sc/e.sc/r.sc/s.sc: 2 hours and 24 minutes; /f.sc/u.sc/t.sc/u.sc/r.sc/e.scG/e.sc/n.sc/e.sc/t.sc/i.sc/c.sc: 55 hours and
53 minutes; and /h.sc/a.sc/s.sc/h.scS/y.sc/n.sc/c.sc: 58 hours and 12 minutes. /T_he four tables in Figure 17 show convex
spaces: any sequence of rows or columns traverses a continuous region, as Example 4.5 illustrates:
Example 4.5. Consider /h.sc/a.sc/s.sc/h.scS/y.sc/n.sc/c.scin Figure 17(c). If we /f_ix the value for keySize in 106, and
vary s.size() in the set {10, 102, 103, 104, 105}, we observe that each one of the continuous intervals
[10, 10], [102, 102]and [103, 105]is governed by the same set of optimal hardware con/f_igurations.
Dealing with variance:To arrive at this result, we had to account for small time variations. To
generate the data in every table seen in Figure 17, we considered 5 ×5 combinations of inputs,
and the hardware con/f_igurations used in the previous sections. We run each pair of inputs with
every con/f_iguration of interest 20 times. To reduce variance, we removed the four fastest and the
four slowest samples; hence, considering 12 executions per input per con/f_iguration. Nevertheless,
this expedient only would not be enough to mitigate the problem of high variance, mostly when
considering input se/t_tings with small runtimes.
To mitigate variance for small inputs, we consider not the best, but the set of best hardware
con/f_igurations per input. For each input, we /f_i/t_ted our linear regression model using the least
squares method to estimate the model parameters. We analyze the diﬀerences among group means
with standard analysis of variance (ANOVA) (Fisher 1918); hence, generalizing the T-test beyond
two means. In the context of this work, we consider groups of hardware con/f_iguration; and the
null hypothesis states that samples from diﬀerent hardware con/f_igurations came from the same
probability distribution. /T_hus, the null hypothesis means that there is no statistical diﬀerence
between the execution time of diﬀerent hardware con/f_igurations. We checked if the data were
statistically signi/f_icant considering a con/f_idence of 95%, i.e., a P-value less than 0.05. ANOVA is
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
Mapping in Parallel Heterogenous Architectures 1:25
1684 126
75
125
25
100
50
Elements
workers
4b0L 4b4L
RadixSort
101010 1010
10
10
10
10
10
Iterations
insertions
HashSync
1
2
3
4
2 3 4 5 6
4b4L, 4b0L, 0b4L, 2b0L, 0b2L
4b4L, 4b0L, 0b4L, 2b0L
4b4L
2b0L
4b4L, 4b0L
1062 84
50
90
10
70
30
Chromosomes
workers
FutureGenetic
4b0L
0b4L
4b0L, 0b4L
1062 84
12
50
3
25
6
Meals
philosophers
Philosophers 
4b4L, 4b0L, 0b4L, 2b0L, 0b2L
0b4L
(b)(a)
(d)(c)
0b4L, 0b2L, 2b0L
5
x103x104
Fig. 17. Best configurations for 4 benchmarks used in our evaluation. The charts exemplify the convex space
over benchmarks inputs. HashSync and FutureGenetic receive 3 inputs each, but for this experiment we fixed
the number of workers in HashSync to 16 and the number of generations in FutureGenetic to 5000.
an omnibus test –it analyzes the data as a whole; hence, we performed a post-hoc test to /f_ind out
where the diﬀerences among the groups were.
/T_he post-hoc test consists of a series of T-tests between each pair of con/f_igurations. /T_he sig-
ni/f_icance level is adjusted to avoid spurious positives. To that end, we used the Bonferroni cor-
rection (Bonferroni 1936; Dunn 1958). Each individual hypothesis is tested with a threshold of
α/n, where α is the signi/f_icance level for the entire set of comparisons, e.g., 0.05, and n is the
number of statistical tests performed. /T_hus, analogously to the ANOVA test, if the resulting P-value
is lower than the signi/f_icance level given by the Bonferroni correction, then the null hypothesis
can be rejected. Rejection of the null hypothesis is equivalent to assume that the two groups of
con/f_igurations present a statistically signi/f_icant diﬀerence.
Monotonicity and Convexity. As Example 4.4 shows, Convexity is a tendency, not a rule.
Nevertheless, we believe that convexity is common because the asymptotic behavior of most
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
1:26 Ribeiro, et al.
2×102
2×103
2×104
2×105
2×106
2 6 10 14 18
Number of Threads
InsertAndAdd
Number of Insertions
0b4L, 4b0L, 4b4L0b4L
0b2L
0b2L, 0b4L
4b4L
0b2L, 0b4L, 2b0L
0b2L, 0b4L, 2b0L, 4b4L
(a)
5×101
5×102
5×103
5×104
5×105
2 6 10 14 18
Number of Threads
RemoveDuplicates
Number of Elements
(b)
0b4L
4b4L
0b4L, 2b0L, 4b0L, 4b4L
0b2L, 0b4L, 2b0L, 4b0L, 4b4L
Fig. 18. Best hardware configuration, considering running time as the objective function. I/n.sc/s.sc/e.sc/r.sc/t.scA/n.sc/d.scD/o.sc/u.sc/b.sc/l.sc/e.sc
receives four inputs. We fixed the “initial capacity” in 5,000 and the “change threshold” in 500.R/e.sc/m.sc/o.sc/v.sc/e.scD/u.sc/p.sc/l.sc/i.sc/hyphen.sc
/c.sc/a.sc/t.sc/e.sc/s.screceives three inputs. We fixed the “size of the keys” in 16 bytes. This choice of values is arbitrary, and
was necessary to give us 2D figures.
algorithms is described by monotonic –therefore convex– functions. /T_he running time of algorithms
tend to be governed by convex functions ranging on their inputs. /T_his observation can also be
projected onto parallel algorithms, once we consider the number of threads as an input. In other
words, variations in the running time of a parallel algorithm tend to be described by (typically
decreasing) monotonic functions ranging on the number of available threads (Keller et al. 2000).
As an example, Figure 18 shows the best hardware con/f_iguration for two of our benchmarks,
considering running time as the objective function. In this case, we are varying, in addition to
some input, the number of available threads. It is possible to observe the convex shape of the space
delimited by the best hardware con/f_igurations for these benchmarks.
4.4.2 Separability. In the context of classi/f_ication problems, the main bene/f_it of convexity is
separability. /T_heHyperplane Separation /T_heorem(Boyd and Vandenberghe 2004, Ch.4) states that
two disjoint convex sets are linearly separable. In other words, if C1 and C2 are closed convex sets
such that C1 ∩C2 = ∅, then there exists a linear function /afii10069.ital(x)= wT x + w0 such that /afii10069.ital(x)> 0 for
every x ∈C1 and /afii10069.ital(x)< 0 for every x in C2. Function /afii10069.italis called a linear discriminant, because it
can distinguish points from C1 and C2 (Chan 2020).
In this paper, we use multivariate linear regression to build linear discriminants. As already
mentioned in Section 2.1, these functions range on the space formed by program inputs. Completely
chartering this space would involve applying the methodology discussed in Section 3.2 to every
possible combination of inputs that a program might use. /T_his task is impractical—if at all possible;
thus, in Section 4.4.1 we have evaluated a handful of inputs for a few benchmarks. /T_his evaluation
indicates that, at least for these benchmarks, the regions in the search space covered by similar
hardware con/f_igurations tend to be convex.
Figure 17 shows that the regions in which speci/f_ic hardware con/f_iguration excel are not dis-
joint. In other words, the same set of function arguments could be mapped to diﬀerent hardware
con/f_igurations with similarly good eﬀects. Nevertheless, the linear discriminants that we build
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
Mapping in Parallel Heterogenous Architectures 1:27
4b0L
0b1L
4b4L
Input XInput Y
4b0L
0b1L
4b4L
Input XInput Y
(x1, y1)
(x0, y0)
(x2, y2)
́
́
́
Inferred separation
4b0L
0b1L 4b4L
Input XInput Y
Inferred separation
(a) (b) (c)
(x1, y1)
(x0, y0)
(x2, y2)
́
́
́
(x3, y3)
́
Fig. 19. (a) An ideal representation of the space of best hardware configurations, with three convex sets. (b)
Approximation of this space a/f_ter training with three inputs covering two convex sets. (c) Approximation of
this space a/f_ter training with inputs that cover all the convex sets. White regions are misclassified zones.
are disjoints, for, in case of ties, we choose the con/f_iguration with less resources as explained in
Section 3.2.1. /T_his said, the accurate classi/f_ication of regions into the right hardware con/f_igurations
still depends on a good assortment of inputs. Example 4.6 illustrates this fact.
Example 4.6. Figure 19 (a) shows an idealized representation of the search space for a hypothetical
program2. /T_his space is formed by three convex sets mapping inputs to con/f_igurations4/b.sc0L, 0/b.sc4L
and 4/b.sc4L. A perfect classi/f_ication model would /f_ind the three lines that separate these regions. In
practice, classi/f_ication approximates these lines by maximizing the distance between the known
inputs. In this example, we have three of them. /T_he white regions in each /f_igure show points for
which classi/f_ication would give wrong answer. /T_he more inputs are used in training, the smaller
these regions tend to be, as the comparison between Figures 19 (b) and (c) illustrates. For a concrete
discussion about the impact of training inputs on the quality of the classi/f_ication model, we refer
the reader to Example 4.2 (Page 19).
5 RELATED WORK
Our work explores a type of machine learning technique –multivariate linear regression– to solve
an instance of program scheduling in heterogeneous architectures. /T_here exists vast literature about
uses of machine learning in compilers (Wang and O’Boyle 2018) and Ashouriet al. (Ashouri et al.
2018). Equally abundant is the material about scheduling in heterogeneous multi-core systems. For
a comprehensive overview on this topic, we recommend the survey recently carried out by Singh
et al. (2020). In the rest of this section, we discuss some of this research, focusing on scheduling,
with the intention to explain how our work stands within the contemporary literature.
5.1 A General Overview on Program Scheduling in Heterogeneous Systems
/T_he general problem of scheduling computations in heterogeneous architectures has a/t_tracted
much a/t_tention, as Mi/t_tal and Ve/t_ter have discussed (Mi/t_tal and Ve/t_ter 2015). Table 3 provides a
taxonomy of previous solutions to this problem. We group them according to the level at which
they are implemented, and to the way they answer each of the following four questions:
•Architecture: do they apply to Single or Multi-ISA systems?
2We say “idealized” because, in practice, we would not be able to determine the best hardware con/f_igurations for every
input set, due to imprecisions in the measurement apparatus. Imprecisions are especially present when considering small
inputs, which tend to cause short running times.
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
1:28 Ribeiro, et al.
Work Level Arch. Source Input Auto Runtime Learn
(Poesia et al. 2017) C Multi Yes No Yes No Yes
(Barik et al. 2016) C Multi Yes No Yes Yes No
(Rossbach et al. 2013) C/L Multi Yes No No Yes No
(Luk et al. 2009) C/L Multi Yes No No Yes No
(Joao et al. 2012) A/L Multi Yes No No No No
(Lukefahr et al. 2016) A Multi No No Yes No No
(Van Craeynest et al. 2012a) A Multi No No Yes No No
(Imes et al. 2015) L Multi Yes No No Yes Yes
(Nishtala et al. 2017) O Single No No Yes Yes Yes
(Petrucci et al. 2015) O Single No No Yes Yes No
(Reddy et al. 2020) O Multi No No Yes Yes No
(Delimitrou and Kozyrakis 2014) O Multi No No Yes Yes Yes
(Augonnet et al. 2011) L Multi Yes No No No No
(Piccoli et al. 2014) O/C Single Yes No Yes Yes No
(Tang et al. 2013) O/C Multi Yes No Yes Yes No
(Cong and Yuan 2012) O/C Multi Yes No Yes Yes No
(Sreelatha et al. 2018) C Single Yes No Yes No Yes
J/i.sc/n.sc/n.sc/hyphen.scCC Single Yes Yes Yes No Yes
Table 3. Diﬀerent solutions to the problem of finding ideal hardware configurations. We consider the
following levels: Architecture (A), Operating System (O), Compiler (C) or Library/Programming model (L).
•Source: is the program’s code modi/f_ied?
•Input: is the approach input-aware?
•Auto: is user intervention required to choose con/f_iguration?
•Runtime: is runtime information exploited?
•Learn: is there any adaptation to runtime conditions?
Perhaps the most important diﬀerence among the several strategies proposed to /f_ind ideal hardware
con/f_igurations concerns the moment at which said strategy is used. In the rest of this section, we
consider the following three possible choices: at compilation time, at runtime, or both.
Static Solutions.Purely static approaches work at compilation time. /T_hey might be applied by
the compiler, either automatically, i.e., without user intervention (Cong and Yuan 2012; Jain et al.
2016; Luk et al . 2009; Poesia et al . 2017; Rossbach et al . 2013; Sreelatha et al . 2018; Tang et al .
2013), or not. In the la/t_ter case, users can use annotations (Mendonc ¸a et al. 2017), domain speci/f_ic
programming languages (Luk et al . 2009; Rossbach et al . 2013) or library calls (Augonnet et al .
2011) to indicate where each program part should run. /T_he main bene/f_it of static techniques is low
runtime overhead: because every scheduling decision is taken before the program runs, no dynamic
checks are necessary to schedule computations. However, these techniques tend to be in/f_lexible:
they are unable to take runtime information into consideration; hence, the same program phase is
always scheduled in the same way. In Table 3, techniques implemented at either the compiler or
library levels are purely static.
Dynamic Solutions.Purely dynamic approaches take into account runtime information. /T_hey can
be implemented at the architecture level (Joao et al. 2012; Lukefahr et al. 2016; Rangan et al. 2009;
Van Craeynest et al. 2012a; Yazdanbakhsh et al. 2015), or at the virtual machine (VM)/OS level (Barik
et al. 2016; Gaspar et al . 2015; Nishtala et al . 2017; Petrucci et al . 2015; Somu Muthukaruppan
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
Mapping in Parallel Heterogenous Architectures 1:29
et al. 2014; Zhang and Hoﬀmann 2016). By leveraging runtime information, the system can use
environment information, unknown at compilation time. Examples of such information include
varying input sizes and resource demands. However, there may be some overhead on accurately
collecting and processing runtime data. Besides, because scheduling decisions are taken on-the-
/f_ly, usually the scheduler cannot spend much time weighing choices. /T_hus, even though these
algorithms use runtime information, they might still take suboptimal decisions, due to their inability
to spend much time solving hard scheduling problems.
Hybrid Solutions.Approaches that mix static and dynamic techniques are calledhybrid. Examples
of hybrid solutions to scheduling include works from Piccoli et al. (2014), Cong and Yuan (2012),
and Tang et al. (2013). Piccoli et al (Piccoli et al. 2014) have used a compiler to instrument a program
with guards that determine, based on input sizes, where each loop should run. Cong and Yuan (Cong
and Yuan 2012), in turn, use the compiler to partition a program in regions of similar behavior, and
rely on runtime information to schedule computation so as to minimize the energy consumed by
each region. Finally, Tang et al. (Tang et al. 2013) use a compiler to populate a program code with
markers, so that low-priority applications can manage their own contentiousness to ensure the QoS
of high-priority co-runners. None of these previous work use any form of learning technique to
tune the behavior of the scheduler, as Table 3 indicates in the column Learn. Guards, once created,
behave always on the same way.
5.2 Scheduling in Single-ISA Heterogeneous Systems
Much a/t_tention has been dedicated to the problem of /f_inding good placements of computation
on heterogeneous multicore systems, as Mi/t_talet al. (Mi/t_tal 2016) has summarized. However,
we emphasize that a large part of this literature concerns the design of scheduling heuristics
implemented at the level of the hardware or the operating system (Cai et al. 2016; Garcia-Garcia
et al. 2018; Masci/t_ti et al. 2020; Mi/t_tal 2016; Neto et al. 2018; Park et al. 2018; Van Craeynest et al.
2012b). /T_his section describes works that, likeJ/i.sc/n.sc/n.sc/hyphen.scC, can be auto-tuned to characteristics of the
runtime environment, and that have been speci/f_ically designed for big.LITTLE architectures. By
characteristics of the runtime environment we mean the nature of the inputs or the behavior of
the hardware. We leave out of this comparison scheduling algorithms that rely on worst-case
execution estimates of tasks, such as Masci/t_ti et al. (2020)’s or Roeder et al. (2021). In other words,
no assumptions are made on the time or energy budget of a given task.
Table 4 categorizes these techniques along the following lines:
•Granularity: what is the data used for training? Most of the techniques use runtime
information (R) –available via performance counters. CHOAMP relies on features mined
from the program’s syntax (S). We use the program’s inputs to perform predictions (I).
•Training: when does learning occur? Oﬀ-line (oﬀ) systems calibrate prediction before the
target program runs; on-line (on) systems do it while the program executes.
•Data: what is the source of training data? OS-based oﬀ-line systems usually rely on micro-
benchmarks (µ-benchs) to perform calibration. CHOAMP uses features of the program,
which it extracts from its syntax. Techniques used in servers can rely on the target program
itself as the source of training data, for said program is bound to run for a long time.
•Target: in which scenario is the technique meant to be used? Most of the papers that
deal with I/s.sc/h.sc/a.sc(De/f_inition 2.3), ours included, present solutions for embedded devices and
smartphones (clients). S/l.sc/o.sc/o.sc/p.sc, S/a.scC, O/c.sc/t.sc/o.sc/p.sc/u.sc/s.sc/hyphen.scM/a.sc/n.scand H/i.sc/p.sc/s.sc/t.sc/e.sc/r.scguarantee QoS in servers.
•Level: the diﬀerent adaptive techniques that we list in Table 4 either run on the operating
system (OS), or are implemented at the compiler’s level (CP).
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
1:30 Ribeiro, et al.
Approach Gran. Tr. Data Target Level
O/c.sc/t.sc/hyphen.scM/a.sc/n.sc(Petrucci et al. 2015) R on self server OS
S/a.scC(Azhar et al. 2019) R on self server. OS
S/l.sc/o.sc/o.sc/p.sc(Azhar et al. 2017) R/S on self server OS/CP
S/p.sc/a.sc/r.sc/t.sc/a.sc(Donyanavard et al. 2016) R oﬀ µ-bench client OS
D/y.scPO(Gupta et al. 2017) R oﬀ µ-bench client OS
Tzilis et al (Tzilis et al. 2019) R oﬀ-line µ-bench client OS
H/i.sc/s.sc/p.sc/t.sc/e.sc/r.sc(Nishtala et al. 2017) R oﬀ/on µ-bench+self server OS
CHOAMP (Sreelatha et al. 2018) S oﬀ µ-bench client CP
SIAM (Krishna and Nasre 2018) S+I oﬀ self client CP
J/i.sc/n.sc/n.sc/hyphen.scCI oﬀ self client CP
Table 4. Recent solutions to I/s.sc/h.sc/a.sc. Granularity (Gran.) uses R for runtime, S for syntax and I for input.
Training (Tr.) uses on for on-line and oﬀ for oﬀ-line. Level uses OS for operating system and CP for compiler.
Two systems that solveI/s.sc/h.sc/a.sc(De/f_inition 2.3) in big.LITTLE architectures at the compiler level are
Sreelatha et al. (Sreelatha et al. 2018)’s CHOAMP, and Krishnaet al. (Krishna and Nasre 2018)’s SIAM.
We have compared J/i.sc/n.sc/n.sc/hyphen.scCwith CHOAMP extensively in this paper. SIAM, in turn, is a system that
targets speci/f_ically graph algorithms parallelized via OpenMP. It consists of a prediction model
that, given a particular shape of a graph, determines the best data-structure format and hardware
con/f_iguration for that shape. We could, in principle, adapt it to implement some of our benchmarks,
such as /s.sc/p.sc/a.sc/n.sc/n.sc/i.sc/n.sc/g.scF/o.sc/r.sc/e.sc/s.sc/t.scand BFS –graph-based algorithms. However, this implementation would
involve providing each algorithm with diﬀerent graph representations –a task to be paid at a
non-negligible programming cost.
5.3 Input-Aware Program Autotuning
Our work is centered around the idea that characteristics of the input can be used to determine the
behavior of a program. Autotuning techniques that take decisions based on inputs are well-known.
Even relatively old libraries such as FFTW (Frigo and Johnson 2005) provide code that is optimized
for diﬀerent input sizes (Guarrasi et al . 2013). And more recent work (Esper et al . 2021; Teich
et al. 2021) has demonstrated that input-awareness can be used to keep programs running in
multi-core systems within the limits of predetermined energy or time requirements. Along these
lines, a direction of research that has been much explored concerns the matching of inputs with
data-structures (Costa and Andrzejak 2018; Jung et al. 2011; Schiller et al. 2016). We emphasize that
none of these previous techniques embed regression code into the compiled program, in order to
bestow on said program the ability to choose hardware con/f_igurations based on input value.
In this regard, recent work by Oliveira et al. (2021) have shown that the implementation of data
structures bear much impact upon the energy consumption of mobile applications. From their
observations, Oliveira et al. provide developers with a number of recommendations to code energy
eﬃcient so/f_tware, following a methodology previously proposed by Couto et al. (2020). /T_his line
of work aims at building tools that work like code linters: tools that point out potential power
ineﬃciencies to programmers. /T_hismodus operandi is made clear by Melfe et al. (2018). /T_his paper,
in contrast, describes an automatic optimization: our code transformation works at the compiler
level, and does not require intervention from users.
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
Mapping in Parallel Heterogenous Architectures 1:31
6 CONCLUSION
/T_his paper presented an end-to-end code generation technique that matches programs to hardware
con/f_igurations in heterogeneous multicore systems. /T_his paper is centered around the thesis that
the values of a function’s arguments provide enough information to predict the best hardware
con/f_iguration for that function. Our technique is able to outperform, be it in energy consumption,
be it in execution time, the default Linux scheduler for ARM (the Global Task Scheduler), and
CHOAMP, a recently released tool that predicts the best hardware con/f_iguration to a parallel
program based on its syntax.
Limitations and Future Work. As any pro/f_iler-based technique,J/i.sc/n.sc/n.sc/hyphen.scChas limitations. Its eﬀec-
tiveness depends on having adequate training inputs for each program that it optimizes. However,
the more inputs are used, the longer the training time, as observed in Section 3.2. We speculate
that it might be possible to remove the need of training per application if we use static-pro/f_iling
techniques (Ball and Larus 1993; Wu and Larus 1994) to infer, at compilation time, how the inputs
of a program might in/f_luence that program’s behavior. In terms of engineering, we believe that the
techniques that we advocate in this paper could be ported to programming environments other
than the JDK’s ecosystem. For instance, to port our technique to C/OpenMP, J/i.sc/n.sc/n.sc/hyphen.scCcould be
reimplemented in LLVM (La/t_tner and Adve 2004). In this regard, LLVM would /f_ill in that se/t_ting the
role that Soot has /f_illed in this paper. Notice that our techniques do not depend on static program
features, so feature engineering would not be necessary for portability. We leave the investigation
of such possibilities as future work.
So/f_tware.J/i.sc/n.sc/n.sc/hyphen.scCis available at h/t_tps://github.com/lac-dcc/JINN-C under the GPL-3.0 License. De-
tails about this project can be found at h/t_tps://homepages.dcc.ufmg.br/∼juniocezar/intelligentDVFS
ACKNOWLEDGMENTS
/T_his work has been made possible by the support of the following agencies: ANR (the CONTINUUM
project: grant ANR-15-CE25-0007-01); CNPq (Grants PDE-202896/2017-0 and 406377/2018-9);
FAPEMIG (Grant PPM-00333-18) and CAPES (Edital CAPES P/r.scI/n.sc/t.sc). While working on this project,
Junio Cezar was the recipient of a scholarship generously donated by Google LLC (Research Awards
for Latin America—LARA ).
REFERENCES
Umut A. Acar, Arthur Chargu ´eraud, Adrien Gua/t_to, Mike Rainey, and Filip Sieczkowski. 2018. Heartbeat Scheduling:
Provable Eﬃciency for Nested Parallelism. In PLDI. ACM, New York, NY, USA, 769–782.
Amir H. Ashouri, William Killian, John Cavazos, Gianluca Palermo, and Cristina Silvano. 2018. A Survey on Compiler
Autotuning Using Machine Learning. Comput. Surv. 51, 5 (2018), 96:1–96:42. DOI:h/t_tp://dx.doi.org/10.1145/3197978
Cedric Augonnet, Samuel /T_hibault, Raymond Namyst, and Pierre-Andre Wacrenier. 2011. StarPU: A Uni/f_ied Platform for
Task Scheduling on Heterogeneous Multicore Architectures. Concurr. Comput. : Pract. Exper. 23, 2 (2011), 187–198.
Muhammad Waqar Azhar, Miquel Peric`as, and Per Stenstr¨om. 2019. SaC: Exploiting Execution-Time Slack to Save Energy
in Heterogeneous Multicore Systems. In ICPP. ACM, New York, NY, USA, 26:1–26:12. DOI:h/t_tp://dx.doi.org/10.1145/
3337821.3337865
M. Waqar Azhar, Per Stenstr¨om, and Vassilis Papaefstathiou. 2017. SLOOP: QoS-Supervised Loop Execution to Reduce
Energy on Heterogeneous Architectures. ACM Trans. Archit. Code Optim. 14, 4, Article 41 (2017), 25 pages. DOI:
h/t_tp://dx.doi.org/10.1145/3148053
D. H. Bailey, E. Barszcz, J. T. Barton, D. S. Browning, R. L. Carter, L. Dagum, R. A. Fatoohi, P. O. Frederickson, T. A. Lasinski, R. S.
Schreiber, H. D. Simon, V. Venkatakrishnan, and S. K. Weeratunga. 1991. /T_he NAS Parallel Benchmarks&Mdash;Summary
and Preliminary Results. In Supercomputing. ACM, New York, NY, USA, 158–165.
/T_homas Ball and James R. Larus. 1993. Branch Prediction for Free. SIGPLAN Not. 28, 6 (1993), 300–313. DOI:h/t_tp:
//dx.doi.org/10.1145/173262.155119
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
1:32 Ribeiro, et al.
Rajkishore Barik, Naila Farooqui, Brian T. Lewis, Chunling Hu, and Tatiana Shpeisman. 2016. A Black-box Approach to
Energy-aware Scheduling on Integrated CPU-GPU Systems. In CGO. ACM, New York, NY, USA, 70–81.
Tarsila Bessa, Ghristopher Gull, Pedro /Q_uint ao, Michael Frank, Jos ´e Nacif, and Fernando Magno /Q_uint ao Pereira.
2017. JetsonLEAP: A framework to measure power on a heterogeneous system-on-a-chip device. Science of Computer
Programming 33, 1 (2017), 1–37.
Carlo Emilio Bonferroni. 1936. Teoria statistica delle classi e calcolo delle probabilit`a. (1936).
Stephen Boyd and Lieven Vandenberghe. 2004. Convex Optimization. Cambridge University Press, New York, NY, USA.
Paul Butcher. 2014. Seven Concurrency Models in Seven Weeks (1st ed.). Pragmatic Bookshelf, Raleigh, NC, US.
Haoran Cai, Qiang Cao, Feng Sheng, Manyi Zhang, Chuanyi Qi, Jie Yao, and Changsheng Xie. 2016. Montgol/f_ier: Latency-
aware power management system for heterogeneous servers. In IPCCC. IEEE, Washington, DC, USA, 1–8.
M. Augustine Cauchy. 1847. M´ethode G´en´erale pour la r´esolution des syst`emes d’´Equations simultan´ees. Comptes Rendus
Hebd. S´eances Acad. Sci. 25, 10 (1847), 536–538.
Stanley Chan. 2020. Linear Separability. (2020). Lecture Notes on Machine Learning - STAT598.
Jason Cong and Bo Yuan. 2012. Energy-eﬃcient Scheduling on Heterogeneous Multi-core Architectures. In ISLPED. ACM,
New York, NY, USA, 345–350.
Keith D. Cooper, Alexander Grosul, Timothy J. Harvey, Steven Reeves, Devika Subramanian, Linda Torczon, and Todd
Waterman. 2005. ACME: Adaptive Compilation Made Eﬃcient. In LCTES. ACM, New York, NY, USA, 69–77.
Diego Costa and Artur Andrzejak. 2018. CollectionSwitch: A Framework for Eﬃcient and Dynamic Collection Selection. In
CGO. Association for Computing Machinery, New York, NY, USA, 16–26. DOI:h/t_tp://dx.doi.org/10.1145/3168825
Marco Couto, Jo˜ao Saraiva, and Jo˜ao Paulo Fernandes. 2020. Energy Refactorings for Android in the Large and in the Wild.
In SANER, Kostas Kontogiannis, Foutse Khomh, Alexander Chatzigeorgiou, Marios-Ele/f_therios Fokaefs, and Minghui
Zhou (Eds.). IEEE, New York, NY, USA, 217–228. DOI:h/t_tp://dx.doi.org/10.1109/SANER48275.2020.9054858
Junio Cezar Ribeiro da Silva, Lorena Le ao, Vin´ıcius Petrucci, Abdoulaye Gamati´e, and Fernando Magno /Q_uint ao Pereira.
2019. Scheduling in Heterogeneous Architectures via Multivariate Linear Regression on Function Inputs. Technical Report
LIRMM-02281112. CNRS.
Junio Cezar Ribeiro da Silva, Lorena Le ao, Vin´ıcius Petrucci, Abdoulaye Gamati´e, and Fernando Magno /Q_uint ao Pereira.
2020. Mapping Computations in Heterogeneous Multicore Systems with Statistical Regression on Inputs. In SBESC. IEEE,
USA, 42–49.
Junio Cezar Ribeiro da Silva, Fernando Magno /Q_uint˜ao Pereira, Michael Frank, and Abdoulaye Gamati´e. 2018. A Compiler-
Centric Infra-Structure for Whole-Board Energy Measurement on Heterogeneous Android Systems. In ReCoSoC. IEEE,
Washington, DC, USA, 1–8.
Florian David, Gael /T_homas, Julia Lawall, and Gilles Muller. 2014. Continuously Measuring Critical Section Pressure with
the Free-lunch Pro/f_iler.SIGPLAN Not. 49, 10 (2014), 291–307.
Christina Delimitrou and Christos Kozyrakis. 2014. /Q_uasar: Resource-eﬃcient and QoS-aware Cluster Management. In
ASPLOS. ACM, New York, NY, USA, 127–144.
Bryan Donyanavard, Tiago M ¨uck, Santanu Sarma, and Nikil Du/t_t. 2016. SPARTA: Runtime Task Allocation for Energy
Eﬃcient Heterogeneous Many-cores. In CODES. ACM, New York, NY, USA, 27:1–27:10.
Olive Jean Dunn. 1958. Estimation of the Means for Dependent Variables. Annals of Mathematical Statistics. 29 (1958),
1095–1111. Issue 4.
Khalil Esper, Stefan Wildermann, and J¨urgen Teich. 2021. A Comparative Evaluation of Latency-Aware Energy Optimization
Approaches in Many-Core Systems (Invited Paper). InNG-RES (OpenAccess Series in Informatics (OASIcs)), Marko Bertogna
and Federico Terraneo (Eds.), Vol. 87. Schloss Dagstuhl–Leibniz-Zentrum f¨ur Informatik, Dagstuhl, Germany, 1:1–1:12.
DOI:h/t_tp://dx.doi.org/10.4230/OASIcs.NG-RES.2021.1
Ronald A. Fisher. 1918. /T_he Correlation Between Relatives on the Supposition of Mendelian Inheritance. Philosophical
Transactions 52 (1918), 399–433.
M. Frigo and S.G. Johnson. 2005. The Design and Implementation of FFTW3. Proc. IEEE 93, 2 (2005), 216 –231. DOI:
h/t_tp://dx.doi.org/10.1109/JPROC.2004.840301
Adrian Garcia-Garcia, Juan Carlos Saez, and Manuel Prieto. 2018. Contention-Aware Fair Scheduling for Asymmetric Single-
ISA Multicore Systems. IEEE Trans. Computers 67, 12 (2018), 1703–1719. DOI:h/t_tp://dx.doi.org/10.1109/TC.2018.2836418
Francisco Gaspar, Luis Tanic ¸a, Pedro Tom´as, Aleksandar Ilic, and Leonel Sousa. 2015. A Framework for Application-Guided
Task Management on Heterogeneous Embedded Systems. ACM Trans. Archit. Code Optim. 12, 4 (Dec. 2015), 42:1–42:25.
Peter Greenhalgh. 2011. Big.LITTLE processing with ARM cortex-A15 & cortex-A7. (2011). h/t_tps://www.eetimes.com/
document.asp?doc id=1279167
Massimiliano Guarrasi, Giovanni Erbacci, and Andrew Emerson. 2013. Auto-tuning of the FFTW Library for Massively
Parallel Supercomputers. (2013).
Ujjwal Gupta, Chetan Arvind Patil, Ganapati Bhat, Prabhat Mishra, and Umit Y. Ogras. 2017. DyPO: Dynamic Pareto-Optimal
Con/f_iguration Selection for Heterogeneous MpSoCs.Trans. Embed. Comput. Syst. 16, 5s (2017), 123:1–123:20. DOI:
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
Mapping in Parallel Heterogenous Architectures 1:33
h/t_tp://dx.doi.org/10.1145/3126530
Mark Gurman, Debby Wu, and Ian King. 2020. Apple Aims to Sell Macs With Its Own Chips Starting in 2021. (2020).
Accessed on July 2021.
Marcus H¨ahnel and Hermann H¨artig. 2014. Heterogeneity by the Numbers: A Study of the ODROID XU+E Big. LITTLE
Platform. In HotPower. USENIX Association, Berkeley, CA, USA, 3–3.
Connor Imes, David H. K. Kim, Martina Maggio, and Henry Hoﬀmann. 2015. POET: a portable approach to minimizing
energy under so/f_t real-time constraints. InRTAS. IEEE, New York, NY, USA, 75–86. DOI:h/t_tp://dx.doi.org/10.1109/RTAS.
2015.7108419
A. Jain, M. A. Laurenzano, L. Tang, and J. Mars. 2016. Continuous shape shi/f_ting: Enabling loop co-optimization via near-free
dynamic code rewriting. In MICRO. IEEE, New York, NY, USA, 1–12.
Brian Jeﬀ. 2013. big.LITTLE Technology moves towards fully heterogeneous Global Task Scheduling. Technical Report. ARM.
White paper.
Jos´e A. Joao, M. Aater Suleman, Onur Mutlu, and Yale N. Pa/t_t. 2012. Bo/t_tleneck Identi/f_ication and Scheduling in Multithreaded
Applications. In ASPLOS. ACM, New York, NY, USA, 223–234.
Changhee Jung, Silvius Rus, Brian P. Railing, Nathan Clark, and Santosh Pande. 2011. Brainy: Eﬀective Selection of Data
Structures. In PLDI. ACM, New York, NY, USA, 86–97. DOI:h/t_tp://dx.doi.org/10.1145/1993498.1993509
J¨org Keller, Christoph Kessler, and Jesper Larsson Tr¨aﬀ. 2000. Practical Pram Programming. John Wiley & Sons, Inc., USA.
J. M. Kim, S. K. Seo, and S. W. Chung. 2014. Looking into heterogeneity: when simple is faster. (2014). h/t_tps://news.
ycombinator.com/item?id=8714613.
Jyothi Krishna and Rupesh Nasre. 2018. Optimizing Graph Algorithms in Asymmetric Multicore Processors. Trans. on CAD
of Integrated Circuits and Systems 37, 11 (2018), 2673–2684. DOI:h/t_tp://dx.doi.org/10.1109/TCAD.2018.2858366
Rakesh Kumar, Dean M. Tullsen, Parthasarathy Ranganathan, Norman P. Jouppi, and Keith I. Farkas. 2004. Single-ISA
Heterogeneous Multi-Core Architectures for Multithreaded Workload Performance. SIGARCH Comput. Archit. News 32,
2 (2004), 64. DOI:h/t_tp://dx.doi.org/10.1145/1028176.1006707
Chris La/t_tner and Sarita V. Adve. 2004. LLVM: a compilation framework for lifelong program analysis transformation. In
CGO. IEEE, New York, US, 75–86. DOI:h/t_tp://dx.doi.org/10.1109/CGO.2004.1281665
Chi-Keung Luk, Sunpyo Hong, and Hyesoon Kim. 2009. Qilin: Exploiting Parallelism on Heterogeneous Multiprocessors
with Adaptive Mapping. In MICRO. ACM, New York, NY, USA, 45–55.
A. Lukefahr, S. Padmanabha, R. Das, F. M. Sleiman, R. G. Dreslinski, T. F. Wenisch, and S. Mahlke. 2016. Exploring
Fine-Grained Heterogeneity with Composite Cores. Transactions on Computers 65, 2 (2016), 535–547.
Agostino Masci/t_ti, Tommaso Cucino/t_ta, and Mauro Marinoni. 2020. An Adaptive, Utilization-Based Approach to Schedule
Real-Time Tasks for ARM Big.LITTLE Architectures. SIGBED Rev. 17, 1 (2020), 18–23. DOI:h/t_tp://dx.doi.org/10.1145/
3412821.3412824
Gilberto Melfe, Alcides Fonseca, and Jo˜ao Paulo Fernandes. 2018. Helping developers write energy eﬃcient Haskell through
a data-structure evaluation. In GREENS@ICSE, Ivano Malavolta, Rick Kazman, and Jo˜ao Saraiva (Eds.). ACM, New York,
NY, USA, 9–15. DOI:h/t_tp://dx.doi.org/10.1145/3194078.3194080
Gleison Mendonc ¸a, Breno Guimar˜aes, P´ericles Alves, M´arcio Pereira, Guido Ara´ujo, and Fernando Magno /Q_uint˜ao Pereira.
2017. DawnCC: Automatic Annotation for Data Parallelism and Oﬄoading. Transactions on Architecture and Code
Optimization 14, 2 (2017), 13:1–13:25.
Sparsh Mi/t_tal. 2016. A Survey of Techniques for Architecting and Managing Asymmetric Multicore Processors.Comput.
Surv. 48, 3 (2016), 45:1–45:38. DOI:h/t_tp://dx.doi.org/10.1145/2856125
Sparsh Mi/t_tal and Jeﬀrey S. Ve/t_ter. 2015. A Survey of CPU-GPU Heterogeneous Computing Techniques.Comput. Surv. 47, 4
(2015), 69:1–69:35.
Mehrzad Nejat, Madhavan Manivannan, Miquel Pericas, and Per Stenstrom. 2020. Coordinated Management of Processor
Con/f_iguration and Cache Partitioning to Optimize Energy under QoS Constraints. InIPDPS. IEEE, USA, 303–313. DOI:
h/t_tp://dx.doi.org/10.1109/IPDPS.2019.00040
Jose Leal Domingues Neto, Se-Young Yu, Daniel F. Macedo, Jos´e Marcos S. Nogueira, Rami Langar, and Stefano Secci. 2018.
ULOOF: A User Level Online Oﬄoading Framework for Mobile Edge Computing. IEEE Trans. Mob. Comput. 17, 11 (2018),
2660–2674. DOI:h/t_tp://dx.doi.org/10.1109/TMC.2018.2815015
Pengcheng Nie and Zhenhua Duan. 2012. Eﬃcient and Scalable Scheduling for Performance Heterogeneous Multicore
Systems. J. Parallel Distrib. Comput. 72, 3 (2012), 353–361.
Rajiv Nishtala, Paul M. Carpenter, Vinicius Petrucci, and Xavier Martorell. 2017. Hipster: Hybrid Task Manager for
Latency-Critical Cloud Workloads. In HPCA. IEEE, New York, NY, USA, 409–420.
Wellington Oliveira, Renato Oliveira, Fernando Castor, Gustavo Pinto, and Jo ˜ao Paulo Fernandes. 2021. Improving
energy-eﬃciency by recommending Java collections. Empir. So/f_tw. Eng.26, 3 (2021), 55. DOI:h/t_tp://dx.doi.org/10.
1007/s10664-021-09950-y
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
1:34 Ribeiro, et al.
Anne-Cecile Orgerie, Marcos Dias de Assun c ¸˜ao, and Laurent Lefevre. 2014. A Survey on Techniques for Improving
the Energy Eﬃciency of Large-scale Distributed Systems. ACM Comput. Surv. 46, 4 (2014), 47:1–47:31. DOI:h/t_tp:
//dx.doi.org/10.1145/2532637
Jinsu Park, Seongbeom Park, and Woongki Baek. 2018. RPPC: A Holistic Runtime System for Maximizing Performance
Under Power Capping. In CCGRID. IEEE, Washington, DC, USA, 41–50.
Suraj Paul, Navonil Cha/t_terjee, Prasun Ghosal, and Jean-Philippe Diguet. 2020. Adaptive Task Allocation and Scheduling on
NoC-Based Multicore Platforms with Multitasking Processors. ACM Trans. Embed. Comput. Syst. 20, 1, Article 4 (2020),
26 pages. DOI:h/t_tp://dx.doi.org/10.1145/3408324
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. /T_hirion, O. Grisel, M. Blondel, P. Pre/t_tenhofer, R. Weiss, V. Dubourg,
J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning
in Python. Journal of Machine Learning Research 12 (2011), 2825–2830.
Vinicius Petrucci, Orlando Loques, Daniel Moss´e, Rami Melhem, Neven Abou Gazala, and Sameh Gobriel. 2015. Energy-
Eﬃcient /T_hread Assignment Optimization for Heterogeneous Multicore Systems.ACM Trans. Embed. Comput. Syst. 14, 1
(2015), 15:1–15:26.
Guilherme Piccoli, Henrique N. Santos, Raphael E. Rodrigues, Christiane Pousa, Edson Borin, and Fernando M. /Q_uint˜ao
Pereira. 2014. Compiler Support for Selective Page Migration in NUMA Architectures. In PACT. ACM, New York, NY,
USA, 369–380.
Gabriel Poesia, Breno Campos Ferreira Guimar˜aes, Fabricio Ferracioli, and Fernando Magno /Q_uint˜ao Pereira. 2017. Static
placement of computation on heterogeneous devices. PACMPL 1, OOPSLA (2017), 50:1–50:28.
Aleksandar Prokopec, Andrea Ros`a, David Leopoldseder, Gilles Duboscq, Petr T˚uma, Martin Studener, Lubom´ır Bulej, Yudi
Zheng, Alex Villaz´on, Doug Simon, /T_homas W¨urthinger, and Walter Binder. 2019. Renaissance: Benchmarking Suite for
Parallel Applications on the JVM. In PLDI. ACM, New York, NY, USA, 31–47.
Krishna K. Rangan, Gu-Yeon Wei, and David Brooks. 2009. /T_hread Motion: Fine-grained Power Management for Multi-core
Systems. In ISCA. ACM, New York, NY, USA, 302–313.
Basireddy Karunakar Reddy, Amit Kumar Singh, Bashir M. Al-Hashimi, and Geoﬀ V. Merre/t_t. 2020. AdaMD: Adaptive
Mapping and DVFS for Energy-Eﬃcient Heterogeneous Multicores. Trans. Comput. Aided Des. Integr. Circuits Syst. 39, 10
(2020), 2206–2217. DOI:h/t_tp://dx.doi.org/10.1109/TCAD.2019.2935065
Uladizislau Rezki and Vitaly Wool. 2015. Doing big.LITTLE right: li/t_tle and big obstacles. (2015).
Julius Roeder, Sebastian Altmeyer, Benjamin Rouxel, and Clemens Grelck. 2021. Energy-aware Scheduling of Multi-version
Tasks onHeterogeneous Real-time Systems. In SAC. ACM, New York, NY, USA, 1–10.
Christopher J. Rossbach, Yuan Yu, Jon Currey, Jean-Philippe Martin, and Dennis Fe/t_terly. 2013. Dandelion: A Compiler and
Runtime for Heterogeneous Systems. In SOSP. ACM, New York, NY, USA, 49–68.
Benjamin Schiller, Clemens Deusser, Jer´onimo Castrill´on, and /T_horsten Strufe. 2016. Compile- and run-time approaches for
the selection of eﬃcient data structures for dynamic graph analysis. Appl. Netw. Sci. 1 (2016), 9. DOI:h/t_tp://dx.doi.org/10.
1007/s41109-016-0011-2
Daniel Shelepov, Juan Carlos Saez Alcaide, Stacey Jeﬀery, Alexandra Fedorova, Nestor Perez, Zhi Feng Huang, Sergey
Blagodurov, and Viren Kumar. 2009. HASS: A Scheduler for Heterogeneous Multicore Systems. SIGOPS Oper. Syst. Rev.
43, 2 (2009), 66–75.
Zhen-Jun Shi. 2004. Convergence of Line Search Methods for Unconstrained Optimization. Appl. Math. Comput. 157, 2
(2004), 393–405. DOI:h/t_tp://dx.doi.org/10.1016/j.amc.2003.08.058
Julian Shun, Guy E. Blelloch, Jeremy T. Fineman, Phillip B. Gibbons, Aapo Kyrola, Harsha Vardhan Simhadri, and Kanat
Tangwongsan. 2012. Brief Announcement: /T_he Problem Based Benchmark Suite. InSPAA. ACM, New York, NY, USA,
68–70.
Amit Kumar Singh, Somdip Dey, Klaus D. McDonald-Maier, Basireddy Karunakar Reddy, Geoﬀ V. Merre/t_t, and Bashir M.
Al-Hashimi. 2020. Dynamic Energy and /T_hermal Management of Multi-core Mobile Platforms: A Survey.Des. Test 37, 5
(2020), 25–33. DOI:h/t_tp://dx.doi.org/10.1109/MDAT.2020.2982629
/T_hannirmalai Somu Muthukaruppan, Anuj Pathania, and Tulika Mitra. 2014. Price /T_heory Based Power Management for
Heterogeneous Multi-cores. In ASPLOS. ACM, New York, NY, USA, 161–176.
Jyothi Krishna Viswakaran Sreelatha, Shankar Balachandran, and Rupesh Nasre. 2018. CHOAMP: Cost Based Hardware
Optimization for Asymmetric Multicore Processors. Trans. Multi-Scale Computing Systems 4, 2 (2018), 163–176.
Lingjia Tang, Jason Mars, Wei Wang, Tanima Dey, and Mary Lou Soﬀa. 2013. ReQoS: Reactive Static/Dynamic Compilation
for QoS in Warehouse Scale Computers. In ASPLOS. ACM, New York, NY, USA, 89–100.
J¨urgen Teich, Pouya Mahmoody, Behnaz Pourmohseni, Sascha Roloﬀ, Wolfgang Schr¨oder-Preikschat, and Stefan Wildermann.
2021. Run-Time Enforcement of Non-functional Program Properties on MPSoCs. In A Journey of Embedded and
Cyber-Physical Systems - Essays Dedicated to Peter Marwedel on the Occasion of His 70th Birthday , Jian-Jia Chen (Ed.).
Springer-Verlag, Berlin, Heidelberg, 125–149. DOI:h/t_tp://dx.doi.org/10.1007/978-3-030-47487-4 9
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.
Mapping in Parallel Heterogenous Architectures 1:35
Stavros Tzilis, Pedro Trancoso, and Ioannis Sourdis. 2019. Energy-Eﬃcient Runtime Management of Heterogeneous
Multicores using Online Projection. TACO 15, 4 (2019), 63:1–63:26.
Raja Vall´ee-Rai, Phong Co, Etienne Gagnon, Laurie Hendren, Patrick Lam, and Vijay Sundaresan. 1999. Soot - a Java
Bytecode Optimization Framework. In CASCON. IBM Press, Indianapolis, US, 13–.
Kenzo Van Craeynest, Aamer Jaleel, Lieven Eeckhout, Paolo Narvaez, and Joel Emer. 2012a. Scheduling Heterogeneous
Multi-cores /T_hrough Performance Impact Estimation (PIE). InISCA. IEEE, New York, NY, USA, 213–224.
Kenzo Van Craeynest, Aamer Jaleel, Lieven Eeckhout, Paolo Narvaez, and Joel Emer. 2012b. Scheduling Heterogeneous
Multi-cores /T_hrough Performance Impact Estimation (PIE). InISCA. IEEE Computer Society, Washington, DC, USA,
213–224.
Zheng Wang and Michael F. P. O’Boyle. 2018. Machine Learning in Compiler Optimization. Proc. IEEE 106, 11 (2018),
1879–1901. DOI:h/t_tp://dx.doi.org/10.1109/JPROC.2018.2817118
Anton Weber, Kim-Anh Tran, Stefanos Kaxiras, and Alexandra Jimborean. 2017. Decoupled Access-Execute on ARM
big.LITTLE. CoRR abs/1701.05478 (2017), 1–25. arXiv:1701.05478 h/t_tp://arxiv.org/abs/1701.05478
Youfeng Wu and James R. Larus. 1994. Static Branch Frequency and Program Pro/f_ile Analysis. InMICRO. ACM, New York,
NY, USA, 1–11. DOI:h/t_tp://dx.doi.org/10.1145/192724.192725
A. Yazdanbakhsh, J. Park, H. Sharma, P. Lot/f_i-Kamran, and H. Esmaeilzadeh. 2015. Neural acceleration for GPU throughput
processors. In MICRO. IEEE, New York, NY, USA, 482–493.
Huazhe Zhang and Henry Hoﬀmann. 2016. Maximizing Performance Under a Power Cap: A Comparison of Hardware,
So/f_tware, and Hybrid Techniques. InASPLOS. ACM, New York, NY, USA, 545–559.
Yunming Zhang, Ajay Brahmakshatriya, Xinyi Chen, Laxman Dhulipala, Shoaib Kamil, Saman Amarasinghe, and Julian
Shun. 2020. Optimizing Ordered Graph Algorithms with GraphIt. In CGO. Association for Computing Machinery, New
York, NY, USA, 158–170. DOI:h/t_tp://dx.doi.org/10.1145/3368826.3377909
ACM Transactions on Embedded Computing Systems, Vol. 1, No. 1, Article 1. Publication date: January 2021.

VALUE LEARNING FOR THROUGHPUT OPTIMIZATION
OF DEEP NEURAL NETWORKS
Benoit Steiner1 Chris Cummins1 Horace He1 Hugh Leather1
ABSTRACT
As the usage of machine learning techniques is becoming ubiquitous, the efﬁcient execution of neural networks is
crucial to many applications. Frameworks, such as Halide and TVM, separate the algorithmic representation of
the deep learning model from the schedule that determines its implementation. Finding good schedules, however,
remains extremely challenging. Auto-tuning methods, which search the space of valid schedules and execute
each candidate on the hardware, identify some of the best performing schedules, but the search can take hours,
hampering the productivity of deep learning practitioners. What is needed is a method that achieves a similar
performance without extensive search, delivering the needed efﬁciency quickly.
We model the scheduling process as a sequence of optimization choices, and present a new technique to accurately
predict the expected performance of a partial schedule using a LSTM over carefully engineered features that
describe each DNN operator and their current scheduling choices. Leveraging these predictions we are able to
make these optimization decisions greedily and, without any executions on the target hardware, rapidly identify an
efﬁcient schedule.
Our evaluation shows that our performance predictions are one order of magnitude more accurate than the state
of the art. This enables us to ﬁnd schedules that improve the execution performance of deep neural networks by
2.6×over Halide and 1.5×over TVM. Moreover, our technique is two to three orders of magnitude faster than
that of these tools, and completes in seconds instead of hours.
1 I NTRODUCTION
The rise of machine learning techniques has been accompa-
nied by the development of deep learning frameworks such
as PyTorch (Paszke et al., 2019) and TensorFlow (Abadi
et al., 2015). The majority of these tools provide a collection
of primitive tensor operators to perform tasks such as matrix
multiplication, convolution and pooling, which are applied
in sequence by a runtime interpreter to derive the outputs of
the neural network from its input tensors.
Though pervasive, this approach has two main downsides.
First, it requires the development, optimization, and main-
tenance of a large library of operators, which necessitates
scarce human expertise. As a result, these frameworks tend
to focus on the most common operators that are only opti-
mized for a limited set of use cases, leaving a lot of perfor-
mance on the table. Second, operators can only exchange
data through global memory, which is a signiﬁcant bot-
1Facebook AI Research, Menlo Park, California. Correspon-
dence to: Benoit Steiner <benoitsteiner@fb.com>.
Proceedings of the4 th MLSys Conference, San Jose, CA, USA,
2021. Copyright 2021 by the author(s).
tleneck, especially in the case of low arithmetic intensity
operators such as the activation functions.
To avoid these problems, projects such as Halide (Ragan-
Kelley et al., 2012) and TVM (Chen et al., 2018a) proposed
to represent tensor computations, such as the ones under-
lying deep learning, using a declarative domain speciﬁc
language based on Einstein’s notation. This high-level rep-
resentation is then compiled into assembly code that can
be executed directly on hardware. This approach abstracts
away from the user key implementation choices such as
loop ordering, blocking, vectorization, unrolling, or par-
allelization, and leaves it up to the compiler to ﬁgure out
which solution, a.k.a. schedule, most efﬁciently leverages
the available hardware resources.
Various efforts, such as Chen et al. (2018b); Zheng et al.
(2020b); Adams et al. (2019); Zheng et al. (2020a), have
attempted to tackle the scheduling problem by framing it
as a search in the space of valid implementations. Given
the combinatorial nature of the problem, it is impossible to
exhaustively consider all the possible solutions. To avoid
this issue, these efforts use various heuristics to attempt
to prune the solution space and focus the search inside its
Value Learning for Throughput Optimization of Deep Neural Networks
most promising regions. However, in doing so they fail to
consider many efﬁcient schedules. Moreover, even after
pruning, they are still left with many candidates that need
to be ranked by their performance. Due to the inaccuracies
of their cost models, these projects ultimately measure the
execution time of the top contenders they identiﬁed on real
hardware, and this benchmarking process is time consuming.
Altogether, these projects suffer from very high search times,
yet may still generate suboptimal implementations.
In this work, we propose to overcome these limitations.
First, we detail how to build and train a model capable
of accurately predicting the performance of deep learning
workloads. Second, through an iterative process, we derive
from this initial model a value function capable of predict-
ing, given a partial set of scheduling decisions, the best
achievable performance over all remaining decisions. The
addition of this look ahead makes it possible to quickly
schedule an entire pipeline by making a sequence of local
decisions that are globally optimal.
We make the following contributions:
• A method to derive a set of valid implementation can-
didates.
• A performance model capable of predicting the speed
of a fully scheduled pipeline almost as accurately as
benchmarking on actual hardware while being many
times faster. Our cost model is more than one order of
magnitude more accurate than the ones proposed by
(Adams et al., 2019) and (Chen et al., 2018b)
• A technique to iteratively derive from our cost model
a value function that forecasts the best performance
achievable for a partially scheduled pipeline. We
use this value function to greedily navigate the so-
lution space towards the best candidate two to three
orders of magnitude faster than previously published
approaches.
• Combining these techniques, we are able to identify
schedules that outperform Halide and TVM by 2.6×
and 1.5×respectively.
2 R ELATED WORK
Polyhedral compilers (Bondhugula et al., 2008b) such as
Vasilache et al. (2018), Baghdadi et al. (2015), or Bond-
hugula et al. (2008a) offer automatic scheduling of afﬁne
loop nests. They restrict the program transformations they
consider to a set of afﬁne loop transformation that maxi-
mizes outer loop parallelism and minimizes statement-to-
statement reuse distance. This formulation can be solved
exactly using an ILP solver. However, important program
optimizations such as introducing redundant computation to
improve locality and parallelism must be excluded, and the
implicit cost model is only weakly correlated with actual
performance. This limits the robustness of the solution as
well as the performance that can be ultimately delivered.
Alternatively, high-performance computing solutions such
as ATLAS (Whaley & Dongarra, 1998) and FFTW (Frigo,
1999) apply black box optimization techniques to automati-
cally tune the implementation of key components of their
libraries. They work by providing a parametrized code gen-
erator capable of producing many functionally equivalent
implementations of an algorithm. They can then automati-
cally identify the most efﬁcient candidate implementation
by measuring the actual run time of each one on the target
hardware. Over the years, many different search techniques
have been applied to auto tuning of compiler optimizations,
including random sampling, genetic algorithms and simu-
lated annealing (Kisuki et al., 2000; Cooper et al., 2005).
Halide (Ragan-Kelley et al., 2012) introduced a domain
speciﬁc language capable of encoding almost every tensor
based computation. It relies on a combination of various cost
models to guide its search, either handcrafted (Mullapudi
et al., 2016; Sioutas et al., 2018; 2020) or learned (Adams
et al., 2019), with optional benchmarking to rank the top
solutions identiﬁed during the search. However, due to the
extensive nature of the search, this process can take several
hours for large problems such as the ones corresponding
to commonly used neural networks. Moreover, the limited
accuracy of these cost models limits the quality of the result.
More recently, TVM (Chen et al., 2018b) provided an imple-
mentation of many common deep learning (DL) operators in
a DSL derived from Halide. This allowed it to annotate each
operator with handwritten schedule templates that restrict
the search space for each DL operator to good subspaces
identiﬁed before hand. However, constructing good tem-
plates requires huge efforts and scarcely available expertise
in both tensor operators and hardware. As a result, these
templates fail to take into consideration good but far from
obvious scheduling solutions. Furthermore, a cost model
is used to quickly discard the solutions that are predicted
to perform poorly. However, the limited accuracy of this
cost model prevents TVM from leveraging it much. Instead,
TVM heavily depends on benchmarking to assess the ﬁtness
of a schedule. As a result, autotuning deep neural network
takes several hours.
To reduce the need for human expertise, FlexTensor (Zheng
et al., 2020b) demonstrates how to build generic sched-
ule templates, and navigates the larger search space using
a combination of heuristics, simulated annealing, and Q-
learning (Watkins & Dayan, 1992). However, they still
benchmark workloads on real hardware to generate their
reward signal on CPU and GPU, which results in search
times of several hours. Ansor (Zheng et al., 2020a) uses a
set of hardcoded rules to automatically generate schedule
templates without human intervention. Like TVM, it relies
Value Learning for Throughput Optimization of Deep Neural Networks
on a cost model to guide its search through the solution
space, and auto-tunes when its cost model suggests good
points. However, the process takes several hours.
Machine Learning based methods are also used for other
applications. For example, Neo (Marcus et al., 2019) relies
on deep neural networks to generate database query execu-
tions plans. It bootstraps its query optimization model from
existing optimizers and continues to learn from incoming
queries.
3 B ACKGROUND
Neural networks are commonly modeled using a graph of
deep learning tensor operators. Indeed, after being intro-
duced by TensorFlow (Abadi et al., 2016), this type of rep-
resentation has gained wide acceptance and has become a
de-facto standard available in PyTorch (Paszke et al., 2019)
(through TorchScript), ONNX (ONN, 2017 (accessed Octo-
ber 8, 2020), MXNet (Chen et al., 2015), and others. How-
ever, as the semantics of the operators is implicit and their
implementation is opaque, these graph representations are
not directly amenable to an end-to-end analysis. Moreover,
any optimization at this level is inherently limited to the
set of rewrites that can be represented using a combination
of the available set of tensor operators. To avoid these two
problems, compilers such as Halide and TVM encode each
operator using one or more tensor expressions based on Ein-
stein notation (Einstein, 1923). We use the same approach
in our work, and in fact, we implement our contributions
on top of Halide. In this section, we introduce the Halide
tensor expression representation and scheduling directives,
as well as the terminology we use throughout this paper.
Readers already familiar with this topic can skip directly to
section 4.
The equations in (1) illustrate how one can model the for-
ward pass of a simple neural network using tensor expres-
sions. The indices (aka variables) i, j, and k, identify the
location of the coefﬁcients in the tensors X, Y, Z, T, A, B,
and W. The variables iand j, used to index the result of
various expressions, are called pure variables. The variable
k, which doesn’t appear in any result, is called areduction
variable.
X(i,j) =
∑
k
A(i,k) ·W(k,j)
Y(j) = tanh(B(j))
Z(i,j) =X(i,j) +Y(j)
T(i,j) = max(0,Z(i,j))
(1)
This computation (aka pipeline) consists of 4 statements,
which we call stages. Zis a consumer of X, and a producer
for T. The number of variables used to index a tensor is
called its rank. The span of each variable is known as its
extent. It is automatically inferred by the compiler from the
dimensions of the pipeline inputs, by tracing the propagation
of variables from the inputs through the statements of the
program.
The tensor expressions are converted into a collection of
nested loops. The schedule, a set of high level directives,
instructs the compiler how to perform this task. Sched-
ules may leave some aspects of the implementation undeter-
mined, in which case the compiler uses default choices to
complete the schedule. Listing 1 depicts the loop nest gen-
erated by a simple schedule that forces the materialization
of every expression in memory.
X = buffer[I, J]
for i in range(0, I, 1):
for j in range(0, J, 1):
X[i, j] = 0
for k in range(0, K, 1):
X[i, j] += A[i, k] * W[k, j]
Y = buffer[J]
for j in range(0, J, 1):
Y[j] = tanh(B[j])
Z = buffer[I, J]
for i in range(0, I, 1):
for j in range(0, J, 1):
Z[i, j] = X[i, j] + Y[j]
T = buffer[I, J]
for i in range(0, I, 1):
for j in range(0, J, 1):
T[i, j] = max(0, Z[i, j])
Listing 1.Naive implementation. Each variable corresponds to a
loop. The result of each expression is materialized in memory
using a buffer.
Such a mapping would not perform well, however. We can
use a combination of the Halide scheduling primitives listed
in Table 1 to improve the situation.
For example, as depicted in Listing 2, we can leverage the
split and reorder directives to tile (Wikipedia, 2020)
the computation of X and improve the locality of the mem-
ory it accesses.
Furthermore, we can altogether remove the need to materi-
alize the values of Zby inlining its computation into that of
T using a compute at directive.
Listing 3 demonstrates how to leverage modern hardware by
parallelizing the computation and making use of SIMD in-
structions. Furthermore, it demonstrates how to take advan-
tage of the store at scheduling directive to only partially
materialize the matrix multiplication, thus further improving
the locality of memory accesses.
Listings 2 and 3 illustrate some of the possible ways to
organize the computation of our example pipeline using
scheduling directives.
Value Learning for Throughput Optimization of Deep Neural Networks
Optimization Description Parameter
split Transform a loop into two nested loops. Split factor.
reorder Exchange two loops. IDs of the loops to swap.
vectorize Use SIMD instructions to encode the loop. ID of the loop to vectorize.
parallel Parallelize the computation over multiple CPU cores. ID of the loop to parallelize.
compute at Inline the evaluation of a loop into another one. ID to the producer to inline and ID of the loop
into which to inline it.
store at Store the values generated by a stage into a temporary buffer. ID of the stage to materialize and location of the
loop in which to materialize it.
Table 1.Primitive scheduling directives used to organize the computation of a deep learning computation pipeline.
X = buffer[I, J]
for ii in range(0, I, B_I):
for jj in range(0, J, B_J):
for i in range(ii, ii+B_I, 1):
for j in range(jj, jj+B_J, 1):
X[i, j] = 0
for k in range(0, K, 1):
X[i, j] += A[i, k] * W[k, j]
T = buffer[I, J]
for i in range(0, I, 1):
for j in range(0, J, 1):
T[i, j] = max(0, X[i, j]+tanh(B[j]))
Listing 2.Partially tiled implementation of the matrix multiplica-
tion X. Additionally, the evaluation of Y is inlined into that of Z,
and the evaluation of Z is inlined into that of T. This reduces the
amount of memory accesses but forces the recomputation of Y for
each iteration over variable i.
T = buffer[I, J]
parallel for ii in range(0, I, B_I):
Xp = buffer[B_I, J]
for jj in range(0, J, B_J):
for i in range(ii, ii+B_I, 1):
for j in range(jj, jj+B_J, 1):
Xp[i-ii, j] = 0
vectorize for k in range(0, K, 1):
Xp[i-ii, j] += A[i, k] * W[k, j]
vectorize for j in range(0, J, 1):
for i in range(ii, ii+B_I, 1):
T[i, j] = max(0, Xp[i-ii, j]+tanh(B[j]))
Listing 3.Partially parallelized and vectorized implementation. In
addition, the evaluation of the matrix multiplication results is
cached inside the loop over variable i.
4 A UTOMATED SCHEDULING
To automate the process of ﬁnding good schedules, we
process the stages of a pipeline one by one. For each stage,
we build a set of candidate solutions as follows:
1. Each variable v in an expression implies a loop of extent
Nv in the naive implementation of the corresponding
stage. We build an initial set of scheduling candidates
C0 by splitting every loop into up to 3 subloops.
2. We extend C0 into C1 by adding to our initial set of
solutions all the possible reorderings of the subloops.
3. We build C2 on top of C1 by adding the option to
vectorize each of the loops of each of the candidates
contained in C1.
4. We then choose at what granularity the stage should
be computed with respect to its consumers, or in other
words, if and how the loops of the stage should be
inlined into the loops of subsequent stages. We also
decide whether the computation should be materialized
in a temporary buffer or not. This grows C2 into C3.
5. Finally, for each solution in C3, we choose which loop
to parallelize. This gives us our ﬁnal set of candidates
C4.
The action space grows combinatorially with the rank of the
tensors as well as the extent of each dimension. We attempt
to combat this by removing choices that do not work well
together. For example, we only consider split factors that are
a multiple of the vector length for a vectorized loop. Table 2
summarizes the landscape of our ﬁnal solution space for
a dozen popular neural networks. On average, there are
almost 300,000 choices for each stage, and for an entire
Avg. Branch
Factor
Scheduling
Decisions
Est. Search
Space Size
AlexNet 2.389e5 54 3.3e290
Inception 6.175e4 370 1.5e1773
MNASNet 1.801e4 206 2.3e876
MobileNetV2 3.600e3 207 1.4e736
ResNet18 2.669e5 170 3.9e922
ResNet3D 1.559e6 91 3.8e563
ResNet50 3.304e5 309 3.0e1705
ShufﬂeNet 1.362e3 301 1.1e944
SqueezeNet 1.421e3 166 3.1e524
Transformer 4.554e4 104 2.0e484
VGG19 9.140e6 178 1.1e1239
Wavenet 6.605e3 133 2.4e376
Average 2.852e5 214 2.4e1167
Table 2. Optimization landscape for 12 neural networks. Branch
factors and search space sizes are estimated using 1,000 randomly
chosen schedules for each model.
Value Learning for Throughput Optimization of Deep Neural Networks
pipeline there are more options than atoms in the universe.
Note that in order to generate the set of candidates C3 for
a stage σi, we need to have settled on the loop structure
of all its consumers. In other words, the set of schedul-
ing solutions available to σi is only determined once all its
consumers have been scheduled. This forces us to gener-
ate candidate schedules for a pipeline sequentially and in
reverse topological order.
We model the problem of choosing the best schedule
amongst all the candidates we identiﬁed as a determinis-
tic Markov Decision Process, or MDP (Bellman, 1957),
over a ﬁnite horizon with a dynamic but ﬁnite action space.
In our formulation, a state Si is a partial schedule, that is
the sequence of individual scheduling decisions sk, with
0 ≤k≤iapplied to the stages σ0 through σi of a pipeline.
Note that we index stages starting from the last one since we
process the pipeline in reverse topological order. The set of
actions ai available in state Si is the set of valid scheduling
options available for stage σi+1 given the loop structure
already imposed to stages σ0 to σi by the scheduling deci-
sions s0 through si. We use the notations Si, s0,...si and s
interchangeably depending on the context.
We propose to solve the MDP by learning an approximation
V(s) of the optimal value function V∗(s). In layman’s
terms, V∗(s) is a function capable of predicting the lowest
runtime achievable from state s assuming that we make
optimal scheduling decisions for all the subsequent stages
in the pipeline.
We explain in Section 5 how we implement our value func-
tion using a neural network, and in section Section 6 we
detail how we train it. Once we have our value function ap-
proximation V(s), we greedily schedule the pipeline stage
by stage as explained in algorithm 1.
Algorithm 1Pipeline scheduling
Input1: Pipeline p with n stages σ1 ... σn
Input2: Value function V(s)
s0 = InitialState
for i= 1to ndo
Ci = GenerateSchedulingCandidates(σi, si−1)
vi = ∞
for sin Ci do
if V(s) <vi then
vi = V(s)
si = s
end if
end for
end for
Return: s1 ... sn
For a N-stage pipeline, with an average of M choices avail-
able per stage, we only need to consider N ·M candidates
out of the MN available complete schedules. This enables
us to schedule deep learning workloads extremely quickly
as we will see in Section 7.3.
Note that if we could learn the true value function V∗(s) in-
stead of an approximation V(s) our approach would ensure
that we ﬁnd the optimal solution. We will see in section
7.2 how each iteration improves the efﬁciency of the sched-
ules identiﬁed by our approach and discuss how well our
approach performs in practice in Section 7.3.
5 P ERFORMANCE PREDICTION
We propose to use a deep neural network to predict the
performance of fully scheduled pipelines (our cost model)
as well as the best achievable performance for a partially
scheduled pipeline (our value function). To achieve the best
possible accuracy and guide our search towards the best
candidates, we paid great attention to the choice of input
features as well as the architecture of the neural network.
5.1 Input Features
The throughput of a neural network depends on two main
factors: the amount of computation and data access to be
performed, and the overall organization of the computation.
Consequently, we devised two groups of input features to
capture this information: a set of intrinsic features that are
invariant to the schedule, and a collection of schedule depen-
dant features that are acquired as the process of scheduling
a pipeline progresses.
When trying to predict the cost of a fully scheduled pipeline,
we can extract acquired features for all the stages. However,
when we try to predict the value of a state Si, we can only
compute acquired features for stages σ0 through σi. We use
zeros for all the remaining stages.
In this setting, the intrinsic features enable our model to
predict how fast each stage could be executed if scheduled
optimally, while the acquired features enable us to capture
how well the scheduled stages are expected to perform.
5.1.1 Intrinsic Features
In order to predict the performance of a computation, we
ﬁrst capture its intrinsic scale and complexity independently
from the way it is implemented on the hardware. To that
end, we extract directly from the tensor expressions a set of
schedule invariant features that describe each stage of the
computation.
The bulk of a neural network computation is done by prim-
itive ﬂoating point operations. We therefore analyze each
tensor expression to extract the total number of primitive
ﬂoating point operations to be performed (namely addition,
subtraction, multiplication, divisions, min and max). In or-
Value Learning for Throughput Optimization of Deep Neural Networks
der to tally accurate counts, we make sure to only use these
primitive operations when encoding the neural network into
tensor expression. For example, we replace all the intrinsic
math functions used in neural networks (exp, tanh, ...) with
their Pad´e approximants (Brezinski, 1996).
Moreover, tensor indexing operations can impact the perfor-
mance of computations signiﬁcantly since integer divisions
and modulo have very low throughput on CPUs. We there-
fore pay attention to capturing all the integer (additions,
subtractions, multiplications, divisions, modulos, min, and
max) and boolean (logical and, or, xor, not, as well as com-
parisons and select operation) operations required to carry
out each tensor expression.
Last but not least, bringing data from memory to an ALU is
up to two orders of magnitude slower than actually per-
forming a computation on this data. We therefore pay
close attention to modeling memory access patterns. In
our DSL, data transfers can happen when an expression Y
references the data produced by one or more expression Xk.
For each stage Y we compute the total number of references
to the coefﬁcients generated by the expressionsXk. We also
model strided, transposed, and broadcast memory accesses
by building a N by M matrix Afor each Xk, where N is the
rank of Y and M the rank of Xk. We populate the matrix A
as follow: A(i,j) = αj if and only if the ith pure variable
used to index Y is the jth variable used to index Xk. αj is
the corresponding stride.
We made room in the feature vector of each stage for refer-
ences to up to 3 other stages. To deal with this limitation,
we encode operators that require more than 3 inputs using
several stages. For example, the concatenation of 5 tensors
would be encoded a chain of 2 concatenations of 3 tensors.
We support tensors of up to 5 dimensions, which enables
us to handle most neural networks, including the ones mak-
ing use of 3d convolutions. We ﬁll the unused slots in the
feature vector with zeros.
5.1.2 Acquired Features
The reorganization of the computation caused by a schedule
impacts how efﬁciently the ALUs, the memory subsystem
with its cache hierarchy, and the many cores of modern
CPUs can be utilized. We designed a set of schedule de-
pendent features to capture these effects, as well as various
overheads inherent in the schedule. We call these features
acquired features as they are progressively collected as we
schedule the pipeline. These features include:
• The number of vectorized as well as scalar calls to
each ﬂoating point and integer primitive arithmetic
operation.
• The extent of each loop (after splitting).
• For each split loop, the total number of bytes read
and written, the total number of unique cache lines
accessed, the number of bytes read and used more
than once as well as the distance in bytes between
reuses. We designed these features to allow the model
to predict whether a schedule can leverage the CPU
caches, and if so at which level and how efﬁciently.
• The amount of recomputation that needs to be per-
formed due to inlining.
• The CPU core utilization deﬁned as the ratio of the
number of parallel tasks over the number of CPU cores.
• The number of ancillary overheads that impact perfor-
mance, such as allocations and deallocations of buffers
on the heap, an estimate of the number of context
switches required to wake up threads when paralleliz-
ing a computation, an estimate of the number of page
faults, the number of false sharings (Scott & Bolosky,
1993).
5.1.3 Derived Features
Products and ratios of features are hard to learn for neural
networks without signiﬁcantly increasing the number of lay-
ers beyond what would be acceptable from a search time
perspective. To avoid this we extended our intrinsic and
acquired feature sets with several metrics derived from the
original features, such as various measures of the arithmetic
intensity imposed by the schedule (e.g. the ratio of vec-
tor instructions by unique bytes read). We determined the
combinations of primitive features that are useful through
extensive empirical testing.
5.2 Model Architecture
The performance of each stage σ in the pipeline we are
trying to optimize depends on three factors:
• Its own schedule.
• The schedule of the stages σk consumed by σ since
this determines the layout of the data handed over to S
as well as the likelihood this data resides in one of the
CPU caches.
• The schedule of the stages σl (if any) that are executed
between all the σk and σ, since these intermediate
stages σl have the potential to ﬂush the data left by the
σk out of the caches, thus forcing its reload by σfrom
a higher level cache or even from main memory.
Moreover, the set of scheduling decisions made for the
initial stages of a pipeline impact the performance of the yet
to be scheduled stages.
To capture these effects, we architected our model around
a bidirectional LSTM (Hochreiter & Schmidhuber, 1997)
that enables latent features to ﬂow from scheduled stages
towards unscheduled stages, as well as from producer stages
to consumer stages, as depicted in Figure 1.
Value Learning for Throughput Optimization of Deep Neural Networks
Normalization
Embedding
Normalization
Embedding
Concat
Normalization
Embedding
Normalization
Embedding
Concat
Normalization
Embedding
Normalization
Embedding
Concat
...
Sum over stages
Mul MulMul
LinearLinear Linear
Linear
Stage 1 Stage 2 Stage N
LSTM Cell                  LSTM Cell                  LSTM Cell                  
Intrinsic 
Feature 
Extraction
Acquired 
Feature 
Extraction
Intrinsic 
Feature 
Extraction
Acquired 
Feature 
Extraction
Optimization ChoicesOptimization Choices Unoptimized
Intrinsic 
Feature 
Extraction
Zeros
Figure 1.Architecture of the neural network we use to predict the throughput of a N-stage pipeline.
We start by normalizing the features extracted by our com-
piler for each stage using the normalization technique intro-
duced by (Ioffe & Szegedy, 2015). However, we normalize
the features over the entire training set instead of doing so
per batch. This helps us better deal with some of the fea-
tures that are rare in our training set, such as the number of
logical or/and/not operations. To further reduce the runtime
of the model, we embed the normalized input features into
a low-dimensional vector space, which we can feed directly
into our LSTM.
The outputs of each LSTM timestep act as a weighting factor
that we multiply coefﬁcient-wise with the embedded feature
vector of each stage. We then reduce the size of the latent
space using a simple linear layer, sum the resulting feature
vectors across all the stages, and lastly pass that sum to a
linear layer to generate our ﬁnal prediction.
6 I TERATIVE VALUE TRAINING
To train our value function V(s), we use an iterative ap-
proach to progressively reﬁne each estimate Vj(s) from its
previous approximation Vj−1(s). We bootstrap the process
by training an initial cost model that we use as a rough
estimate of V0(s).
6.1 Pipeline Dataset
To train our value function and initial cost model, we created
over 10,000 random neural networks, ranging in size from
1 to 200 operators, with up to 4 inputs (excluding weights)
and 15 outputs. The inputs have up to 4 dimensions of
size ranging between 1 and 1024. All these parameters
are sampled uniformly. The sequence of operators, their
connectivity, and parameters are also chosen at random
among all the legal options (e.g. we cannot apply a 5 by 5
convolution on a 3 by 3 input without padding).
6.2 Initial Cost Model
To train our initial cost model, we generated approximately
1,000 random complete schedules for each of our pipelines.
We benchmarked each schedule 10 times and recorded the
corresponding runtimes along with the corresponding intrin-
sic and acquired features.
To ensure that our model is not biased toward over or under
predictions, we use the absolute value of the relative error,
or P/M where P is the prediction of our model and M is
the mean measured performance over ten runs. Furthermore,
we add two components to the loss:
• A criticality factor C, deﬁned as the ratio of the best
runtime measured for the pipeline over the runtime
of the schedule. As as long as our cost model ranks
poorly performing schedules below the best ones, it
serves its intended purpose. Cenables us to relax the
accuracy requirement for the least efﬁcient schedules
and instead focus on the most critical ones.
• A deweighting factor D, based on the conﬁdence in
the reliability of the ground truth for that schedule,
estimated as the inverse of the standard deviation of all
the measurements for the schedule.
Our ﬁnal loss is the product of these 3 components.
L= |log(P/M)|·D·C
6.3 Value Function Iteration
Our technique is inspired from the well known value it-
eration approach summarized in (Sutton & Barto, 2018).
Value Learning for Throughput Optimization of Deep Neural Networks
Algorithm 2Iteration of our value function approximation.
Input: set of pipelines P
Input: value function Vi−1(s)
Initialize Vi(s) to Vi−1(s)
for pin P do
for kin [0,100] do
s0,...sn = BestSchedule(p,Vi,ϵk )
for sj in s0,...sn do
tj+1,...tn = BeamSearch(sj,Vi−1))
r= Benchmark(s0,...sj,tj+1,...tn)
Vi(sj) = min(r,Vi(sj))
end for
end for
end for
Return: Vi(s)
Starting from a randomly initialized value function, the al-
gorithm iterates over all the states sand all the actions a
of the MDP. It reﬁnes the value function V for each state s
from the expected reward associated with taking action aby
using the following rule until the process converges:
V(s) = max
a
(E[r|s,a] +V(s′))
However, we cannot use this approach directly. Indeed, we
face two main hurdles. First, we cannot measure the unit
reward associated with taking an action. We can only access
the end to end runtime of a fully scheduled pipeline. Second,
it would be impractical to exhaustively visit all the states
associated with the scheduling of a single pipeline for all
but the shortest ones.
On the other hand, we can make a few simpliﬁcations. First,
we do not need to uniformly sample all the actions available
from a state s. Although we need a precise estimate of
the value function associated with the “best“ states, we
only need a rough estimate for the less interesting states
(using the intuition behind the criticality factor that we use
in our loss function). Consequently, we can undersample
the actions that lead to the less interesting states. Second,
we can derive an estimate for the value function for a state
sby searching for the best schedule starting from sinstead
of relying on rewards. Third, we do not need to start from a
random value function. We can instead use our cost model to
bootstrap the process of estimating the value function. This
helps the initial search ﬁnd much more accurate estimates
for V1(s).
Based on these observations, we built Algorithm 2: we
extract 100 schedules for each pipeline in our training set,
using the BestSchedule procedure previously described in
Algorithm 1. We inject a small amount of random noise ϵto
the predictions made by the value function to ensure that we
cover a signiﬁcant portion of the interesting states of each
pipeline.
For each state si in the greedy schedules, we identify how
to best schedule the remaining stages of the pipeline by
running a beam search starting from si. We use a beam size
of 320, and our previous estimate of the value function to
guide the search. We benchmark the resulting schedule and
use its measured runtime to reﬁne the estimate of the value
function V for the state si.
Starting from V0, which is our initial cost model, we then it-
eratively build progressively more accurate estimates Vi(s).
7 E VALUATION
We implemented our approach by coupling C++ code to
quickly generate candidate schedules, extract intrinsic and
acquired features, and run our search policy with Pytorch
code to implement, train, and evaluate our cost model and
value functions. We used Halide to compile our schedules
into assembly code.
We ran all our experiments on an Intel Xeon D-2191A CPU
running at 1.60GHz with 48GB of RAM and a NVidia Tesla
M40 GPU.
7.1 Cost Model
We trained our initial cost model on a total of 13,352
pipelines converted from a set of randomly created ONNX
models, with an average of 1,184 schedules per pipeline.
We evaluated its accuracy on 12,000 schedules extracted
from the models listed in Table 2.
We compared the accuracy of our cost model against that of
Halide (Adams et al., 2019) and TVM (Chen et al., 2018b).
We retrained the Halide cost model on our training set and
evaluated it on our test set. The TVM cost models are trained
online directly on the data generated by the benchmarking
process of individual deep learning operators. We used this
same data as the test set on which we evaluated the accuracy
of their XGBoost based model. Note that overﬁtting is not a
concern in this case since TVM retrains its cost model for
every pipeline it optimizes.
We summarized the average prediction error, the maximum
prediction error, and the coefﬁcient of determination R2 for
all three models in Figure 2.
Our predictions have an average error of 4.9%, representing
a signiﬁcant improvement over the Halide and TVM cost
models which have an average error of 64.7% and 103.8%
respectively. Furthermore, our prediction error is almost
down to the level of noise inherent in any performance mea-
surement, which is 2.4% on average and 282% in the worst
case on both our training and test sets. We deﬁne the mea-
surement noise for each schedule as the difference between
Value Learning for Throughput Optimization of Deep Neural Networks
20
40
60
80
100
TVM
Halide
Ours
(a) Average prediction error (in %) on a linear scale
500
1000
5000
10000
50000 TVM
Halide
Ours
(b) Maximum prediction error (in %) on a log scale
0.75
0.80
0.85
0.90
0.95
1.00 TVM
Halide
Ours
(c) Coefﬁcient of determination
Figure 2.Aggregate accuracy of TVM, Halide, and our approaches
for estimating the runtime of deep learning workloads.
the two extreme measurements for that schedule divided
by the average measurement. The average (resp. worst
case) noise for the test and training set is the average (resp.
maximum) of the measurement noise of each schedule.
7.2 Value Function
We don’t have a direct way to demonstrate that our succes-
sive value function approximations Vi(s) converge towards
the optimal value functionV∗(s). Instead we show that with
each iteration our value function estimates are better able to
guide a search. In Figure 3 we plot the relative performance
of the schedules selected by our greedy search as well as a
standard beam search under the guidance of three successive
estimates V0, V1, and V2 on the models listed in Table 2.
V0 is our initial cost model. Due to its lack of look-ahead
ability, it is a poor guide for a greedy search. Unsurprisingly,
beam search performs substantially better. As V1 and V2
reﬁne the estimates of our value function the gap between
the quality of the schedules identiﬁed by a greedy and a
beam search decreases, while the overall performance of the
schedules increases.
Figure 3.Relative speedups of the schedules generated by our
value function. The schedules are found using either a greedy
or a beam search guided by successive approximations of our
value function V0, V1, V2.
7.3 Benchmarks
We evaluate the ability of our search strategy to identify
good schedules on a diverse set of deep learning workloads
summarized in Table 2, comprehending a range of runtime
comprised between 10 milliseconds and 3.4 seconds.
In Figure 4a, we compare the quality of the implementations
found by the following systems: PyTorch 1.5, AutoTVM
version 0.6, and the Halide auto-scheduler version 8.0.0.
PyTorch does not auto-tune, or search any optimization
space. We conﬁgured AutoTVM to use 1 thread per core.
We ran the Halide auto-scheduler with its default settings
of 5 search passes with each pass identifying 32 candidate
schedules, and benchmarking to do the ﬁnal ranking of the
160 candidates.
Our approach ﬁnds schedules that outperform PyTorch, Au-
toTVM, and Halide, producing geomean speedups of 5.1×,
1.5×, and 2.6×respectively.
RL based methods can be sensitive to stochastic initializa-
tion. To measure the impact of this effect on the quality of
the schedule our approach generates, we trained 10 distinct
value functions making sure we initialize our random num-
ber generator with a different seed each time. We then ran
the schedule search under the guidance of these 10 value
functions, and measured the performance of the correspond-
ing implementations. On average the performance varies by
±8.3%, as represented by the error bars on Figure 4a.
We show the search time of the systems in Figure 4b. Py-
Torch does not search for good solutions. AutoTVM takes
considerable time, requiring 3 hours on average and up to 12
hours to complete the search. Halide takes an average of 20
minutes and up to 2.5 hours. Our system with cost and value
functions takes only an average of 13 seconds to optimize
Value Learning for Throughput Optimization of Deep Neural Networks
2x
4x
6x
8x
10x
12x
14x
16x
18x
Alexnet Inception MNASNet Mobilenet ResNet18 ResNet3D ResNet50 Shufflenet Squeezenet Transformer VGG19 WaveNet Geomean
Adams2019 AutoTVM Ours
(a) Speedup of schedules over PyTorch. Higher is better.
1s
10s
100s
1000s
10000s
Alexnet Inception MNASNet Mobilenet ResNet18 ResNet3D ResNet50 ShufflenetSqueezenet Transformer VGG19 WaveNet Geomean
Adams2019 AutoTVM Ours
(b) Schedule search time. Lower is better.
Figure 4. Performance of ﬁnal schedules relative to PyTorch (a) and time taken to schedule deep learning workloads (b). We plot the
search times on a log scale. We compare our results against the Halide autoscheduler (Adams2019) and AutoTVM using the published
conﬁguration. TVM failed to load the Transformer and Wavenet models. For fairness, we exclude those models from the AutoTVM result
aggregates.
a neural network and less than 1 minute in the worst case.
Our policy based search is by far the most efﬁcient.
8 C ONCLUSION
By combining machine learning techniques with a strong
set of features, it is possible to predict the latency of a deep
learning workload nearly as precisely as by benchmarking
the workload on real hardware. Moreover, by leveraging this
cost model we built a value function capable of accurately
predicting, given a partial set of scheduling decisions, the
best achievable performance over all remaining decisions.
We use this value function to greedily navigate the space of
solutions without having to benchmark candidate schedules
to evaluate their ﬁtness. This enables us to both ﬁnd better
candidates while speeding up the scheduling process by
several orders of magnitude compared to previous state of
the art.
REFERENCES
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z.,
Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M.,
Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard,
M., Jia, Y ., Jozefowicz, R., Kaiser, L., Kudlur, M., Lev-
enberg, J., Man´e, D., Monga, R., Moore, S., Murray, D.,
Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever,
I., Talwar, K., Tucker, P., Vanhoucke, V ., Vasudevan,
V ., Vi´egas, F., Vinyals, O., Warden, P., Wattenberg, M.,
Wicke, M., Yu, Y ., and Zheng, X. TensorFlow: Large-
scale machine learning on heterogeneous systems, 2015.
URL https://www.tensorflow.org/. Software
available from tensorﬂow.org.
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean,
J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur,
M., Levenberg, J., Monga, R., Moore, S., Murray, D.,
Steiner, B., Tucker, P., Vasudevan, V ., Warden, P., Wicke,
M., Yu, Y ., and Zhang, X. Tensorﬂow: A system for
large-scale machine learning. In OSDI, 2016.
Adams, A., Ma, K., Anderson, L., Baghdadi, R., Li, T.-
M., Gharbi, M., Steiner, B., Johnson, S., Fatahalian, K.,
Durand, F., and Ragan-Kelley, J. Learning to optimize
halide with tree search and random programs. ACM
Trans. Graph., 38(4):121:1–121:12, July 2019. ISSN
0730-0301. doi: 10.1145/3306346.3322967. URL http:
//doi.acm.org/10.1145/3306346.3322967.
Baghdadi, R., Beaugnon, U., Cohen, A., Grosser, T., Kruse,
M., Reddy, C., Verdoolaege, S., Betts, A., Donaldson,
A. F., Ketema, J., Absar, J., Haastregt, S. v., Kravets,
A., Lokhmotov, A., David, R., and Hajiyev, E. Pencil:
A platform-neutral compute intermediate language for
accelerator programming. In Proceedings of the 2015
International Conference on Parallel Architecture and
Value Learning for Throughput Optimization of Deep Neural Networks
Compilation (PACT), PACT ’15, pp. 138–149, USA, 2015.
IEEE Computer Society. ISBN 9781467395243. doi:
10.1109/PACT.2015.17. URL https://doi.org/
10.1109/PACT.2015.17.
Bellman, R. A markovian decision process. Indiana Univ.
Math. J., 6:679–684, 1957. ISSN 0022-2518.
Bondhugula, U., Hartono, A., Ramanujam, J., and Sa-
dayappan, P. A practical automatic polyhedral paral-
lelizer and locality optimizer. SIGPLAN Not., 43(6):
101–113, June 2008a. ISSN 0362-1340. doi: 10.
1145/1379022.1375595. URL https://doi.org/
10.1145/1379022.1375595.
Bondhugula, U., Hartono, A., Ramanujam, J., and Sadayap-
pan, P. A practical automatic polyhedral parallelizer
and locality optimizer. In Proceedings of the 29th ACM
SIGPLAN Conference on Programming Language De-
sign and Implementation, PLDI ’08, pp. 101–113, New
York, NY , USA, 2008b. ACM. ISBN 978-1-59593-
860-2. doi: 10.1145/1375581.1375595. URL http:
//doi.acm.org/10.1145/1375581.1375595.
Brezinski, C. Extrapolation algorithms and pad´e approxima-
tions: a historical survey. Applied numerical mathematics,
20(3):299–318, 1996.
Chen, T., Li, M., Li, Y ., Lin, M., Wang, N., Wang, M., Xiao,
T., Xu, B., Zhang, C., and Zhang, Z. MXNet: A Flexible
and Efﬁcient Machine Learning Library for Heteroge-
neous Distributed Systems. ArXiv e-prints, December
2015.
Chen, T., Moreau, T., Jiang, Z., Zheng, L., Yan, E., Shen,
H., Cowan, M., Wang, L., Hu, Y ., Ceze, L., Guestrin, C.,
and Krishnamurthy, A. TVM: An automated end-to-end
optimizing compiler for deep learning. In 13th USENIX
Symposium on Operating Systems Design and Implemen-
tation (OSDI 18), pp. 578–594, Carlsbad, CA, October
2018a. USENIX Association. ISBN 978-1-939133-08-3.
URL https://www.usenix.org/conference/
osdi18/presentation/chen.
Chen, T., Zheng, L., Yan, E., Jiang, Z., Moreau, T., Ceze,
L., Guestrin, C., and Krishnamurthy, A. Learning to op-
timize tensor programs. In Proceedings of the 32Nd
International Conference on Neural Information Pro-
cessing Systems, NIPS’18, pp. 3393–3404, USA, 2018b.
Curran Associates Inc. URL http://dl.acm.org/
citation.cfm?id=3327144.3327258.
Cooper, K., Grosul, A., Harvey, T., Reeves, S., Subramanian,
D., Torczon, L., and Waterman, T. Acme: Adaptive
compilation made efﬁcient. volume 40, 07 2005. doi:
10.1145/1070891.1065921.
Einstein, A. The foundation of the general theory of rel-
ativity. In Annalen der Physik, pp. 81–124. Springer,
1923.
Frigo, M. A fast fourier transform compiler. In PLDI, 1999.
Hochreiter, S. and Schmidhuber, J. Long short-term memory.
Neural computation, 9:1735–80, 12 1997. doi: 10.1162/
neco.1997.9.8.1735.
Ioffe, S. and Szegedy, C. Batch normalization: Accelerating
deep network training by reducing internal covariate shift.
ArXiv, abs/1502.03167, 2015.
Kisuki, T., Knijnenburg, P. M. W., and O’Boyle, M. F. P.
Combined selection of tile sizes and unroll factors using
iterative compilation. In Proceedings 2000 International
Conference on Parallel Architectures and Compilation
Techniques (Cat. No.PR00622), pp. 237–246, 2000.
Marcus, R., Negi, P., Mao, H., Zhang, C., Alizadeh, M.,
Kraska, T., Papaemmanouil, O., and Tatbul, N. Neo:
A learned query optimizer. Proc. VLDB Endow., 12
(11):1705–1718, July 2019. ISSN 2150-8097. doi: 10.
14778/3342263.3342644. URL https://doi.org/
10.14778/3342263.3342644.
Mullapudi, R. T., Adams, A., Sharlet, D., Ragan-Kelley, J., ,
and Fatahalia, K. Automatically scheduling halide image
processing pipelines. In ACM Trans. Graph, 2016.
Open Neural Network Exchange (ONNX). ONNX, 2017
(accessed October 8, 2020). URL https://github.
com/onnx/onnx.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J.,
Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga,
L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison,
M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L.,
Bai, J., and Chintala, S. Pytorch: An imperative style,
high-performance deep learning library. In Advances
in Neural Information Processing Systems 32, pp. 8026–
8037. Curran Associates, Inc., 2019.
Ragan-Kelley, J., Adams, A., Paris, S., Levoy, M., Ama-
rasinghe, S., and Durand, F. Decoupling algorithms
from schedules for easy optimization of image processing
pipelines. ACM Transactions on Graphics - TOG, 31, 07
2012. doi: 10.1145/2185520.2185528.
Scott, M. and Bolosky, W. False sharing and its effect
on shared memory performance. In Proceedings of the
USENIX Symposium on Experiences with Distributed and
Multiprocessor Systems (SEDMS), volume 57, pp. 41,
1993.
Value Learning for Throughput Optimization of Deep Neural Networks
Sioutas, S., Stuijk, S., Corporaal, H., Basten, T., and Somers,
L. Loop transformations leveraging hardware prefetch-
ing. In Proceedings of the 2018 International Sym-
posium on Code Generation and Optimization, CGO
2018, pp. 254–264, New York, NY , USA, 2018. Associa-
tion for Computing Machinery. ISBN 9781450356176.
doi: 10.1145/3168823. URL https://doi.org/10.
1145/3168823.
Sioutas, S., Stuijk, S., Basten, T., Corporaal, H., and Somers,
L. Schedule synthesis for halide pipelines on gpus. ACM
Trans. Archit. Code Optim., 17(3), August 2020. ISSN
1544-3566. doi: 10.1145/3406117. URL https://
doi.org/10.1145/3406117.
Sutton, R. S. and Barto, A. G. Reinforcement Learn-
ing: An Introduction. The MIT Press, second edition,
2018. URL http://incompleteideas.net/
book/the-book-2nd.html.
Vasilache, N., Zinenko, O., Theodoridis, T., Goyal, P., De-
Vito, Z., Moses, W. S., Verdoolaege, S., Adams, A., and
Cohen, A. Tensor comprehensions: Framework-agnostic
high-performance machine learning abstractions, 2018.
Watkins, C. J. C. H. and Dayan, P. Q-learning. Machine
Learning, 8(3):279–292, 1992. ISSN 1573-0565. doi:
10.1007/BF00992698. URL https://doi.org/10.
1007/BF00992698.
Whaley, R. C. and Dongarra, J. Automatically tuned lin-
ear algebra software. In SuperComputing 1998: High
Performance Networking and Computing, 1998.
Wikipedia. Loop tiling, 2020. URL https:
//en.wikipedia.org/w/index.php?title=
Loop_tiling. [Online; accessed 07-October-2020].
Zheng, L., Jia, C., Sun, M., Wu, Z., Yu, C. H., Haj-Ali, A.,
Wang, Y ., Yang, J., Zhuo, D., Sen, K., Gonzalez, J. E., and
Stoica, I. Ansor: Generating high-performance tensor
programs for deep learning. In 14th USENIX Sympo-
sium on Operating Systems Design and Implementation
(OSDI 20), Banff, Alberta, November 2020a. USENIX
Association. URL https://www.usenix.org/
conference/osdi20/presentation/zheng.
Zheng, S., Liang, Y ., Wang, S., Chen, R., and Sheng,
K. Flextensor: An automatic schedule exploration
and optimization framework for tensor computation on
heterogeneous system. ASPLOS ’20, pp. 859–873,
New York, NY , USA, 2020b. Association for Com-
puting Machinery. ISBN 9781450371025. doi: 10.
1145/3373376.3378508. URL https://doi.org/
10.1145/3373376.3378508.

Visualizing and Understanding Recurrent Networks
Andrej Karpathy∗ Justin Johnson∗ Li Fei-Fei
Department of Computer Science, Stanford University
{karpathy,jcjohns,feifeili}@cs.stanford.edu
Abstract
Recurrent Neural Networks (RNNs), and speciﬁcally a variant with Long Short-
Term Memory (LSTM), are enjoying renewed interest as a result of successful
applications in a wide range of machine learning problems that involve sequential
data. However, while LSTMs provide exceptional results in practice, the source
of their performance and their limitations remain rather poorly understood. Using
character-level language models as an interpretable testbed, we aim to bridge this
gap by providing a comprehensive analysis of their representations, predictions
and error types. In particular, our experiments reveal the existence of interpretable
cells that keep track of long-range dependencies such as line lengths, quotes and
brackets. Moreover, an extensive analysis with ﬁnite horizon n-gram models sug-
gest that these dependencies are actively discovered and utilized by the networks.
Finally, we provide detailed error analysis that suggests areas for further study.
1 Introduction
Recurrent Neural Networks, and speciﬁcally a variant with Long Short-Term Memory (LSTM) (14),
have recently emerged as an effective model in a wide variety of applications that involve sequential
data. These include language modeling (22), handwriting recognition and generation (9), machine
translation (28; 1), speech recognition (10), video analysis (7) and image captioning (30; 19).
However, the source of their impressive performance remains poorly understood. This raises con-
cerns of interpretability and limits our ability design better architectures. A few recent ablation
studies analyzed the effects on performance as they removed or modiﬁed various gates and connec-
tions (12; 5). While this analysis illuminates the performance-critical pieces of the architecture, it is
still limited to measuring the test set perplexity alone.
An often cited advantage of the LSTM architecture is that it can store and retrieve information over
long time scales using its gating mechanisms. However, it is not immediately clear that good so-
lutions can be discovered with stochastic gradient descent and truncated backpropagation through
time, and that these mechanisms are effectively utilized on real-world data. In this work we use
character-level language models as an interpretable testbed for illuminating the long-range depen-
dencies learned by the LSTM architecture. Our analysis reveals the existence of cells that robustly
identify interpretable, high-level patterns such as line lengths, brackets and quotes. We further quan-
tify the LSTM predictions with comprehensive comparison to n-gram models, where we ﬁnd that
LSTMs perform signiﬁcantly better on characters that require long-range reasoning. Finally, we
conduct a detailed error analysis in which we “peel the onion”of errors with a sequence of oracles.
These results allow us to quantify the extent of remaining errors in several categories and to suggest
speciﬁc areas for further study.
2 Related Work
Recurrent Networks. Recurrent Neural Networks (RNNs) have a long history of applications in
various sequence learning tasks (31; 26; 25). Despite their early successes, the difﬁculty of training
∗Both authors contributed equally to this work.
1
arXiv:1506.02078v1  [cs.LG]  5 Jun 2015
simple recurrent networks (2; 24) has encouraged various proposals for improvements to their basic
architecture. Among the most successful variants are the Long Short Term Memory networks (14),
which can in principle store and retrieve information over long time periods with explicit gating
mechanisms and a built-in constant error carousel. In the recent years there has been a renewed
interest in further improving on the basic architecture by modifying the functional form as seen with
Gated Recurrent Units (4), incorporating content-based soft attention mechanisms (1; 32), push-
pop stacks (18), or more generally external memory arrays with both content-based and relative
addressing mechanisms (11). In this work we focus the majority of our analysis on the LSTM due
to its widespread popularity and proven track record.
Understanding Recurrent Networks. While there is an abundance of work that modiﬁes or extends
the basic LSTM architecture, relatively little attention has been paid to understanding its computa-
tional properties or the source of its performance. Greff et al. (12) recently conducted a comprehen-
sive study of different LSTM components and concluded that the forget gates are its most critical
components. Chung et al. evaluated GRU compared to LSTMs (5). Pascanu et al. examined the
effects of depth (23). These approaches study recurrent network architectures based on variations in
the ﬁnal performance, while our approach dives deep into the statistical patterns in their activations
and predictions. We also conduct detailed error analysis that breaks down the the ﬁnal performance
into interpretable categories.
3 Experimental Setup
We ﬁrst describe three variants of a deep recurrent network (RNN, LSTM and the GRU models),
then explain how they are used in sequence learning and ﬁnally describe the optimization.
3.1 Recurrent Neural Network Models
The simplest instantiation of a deep recurrent network arranges hidden state vectors hl
t in a two-
dimensional grid, where t = 1 . . . Tis thought of as time and l = 1 . . . Lis the depth. The bottom
row of vectors h0
t = xt at depth zero holds the input vectors xt and each vector in the top row {hL
t }
is used to predict an output vector yt. All intermediate vectors hl
t are computed as a function of
hl
t−1 and hl−1
t . Through these hidden vectors, each output yt at some particular time step t becomes
a function of all input vectors up to that time, {x1, . . . , xt}. The precise mathematical form of the
recurrence (hl
t−1 , hl−1
t ) →hl
t varies from model to model and we describe these details next.
Vanilla Recurrent Neural Network (RNN) has a recurrence of the form
hl
t = tanh Wl
(
hl−1
t
hl
t−1
)
where we assume that all h ∈Rn. The parameter matrix Wl on each layer has dimensions [n ×2n]
and tanh is applied elementwise. Note that W varies between layers but is shared through time. We
omit the bias vectors for brevity. Interpreting the equation above, the inputs from the layer below
hl−1
t and before in time hl
t−1 are transformed and interact through additive interaction before the
non-linearity. This is known to be a weak form of coupling (27). Both the LSTM and the GRU
(discussed next) include more powerful multiplicative interactions.
Long Short-Term Memory (LSTM) (14) was designed to address the difﬁculties of training
RNNs (2). In particular, it was observed that the backgropagation dynamics led the gradients in
an RNN to either vanish or explode. It was later found that the exploding gradient concern can
be alleviated with a heuristic of clipping the gradients at some maximum value (24). On the other
hand, LSTMs were designed to mitigate the vanishing gradient problem. In addition to a hidden
state vector hl
t, LSTMs also maintain a memory vector cl
t. At each time step the LSTM can choose
to read from, write to, or reset the cell using explicit gating mechanisms. The precise form of the
update is as follows:


i
f
o
g

=


sigm
sigm
sigm
tanh

Wl
(
hl−1
t
hl
t−1
) cl
t = f ⊙cl
t−1 + i ⊙g
hl
t = o ⊙tanh(cl
t)
Here, the sigmoid function sigm and tanh are applied element-wise, and Wl is a [4n ×2n] matrix.
The three vectors i, f, o∈Rn are each thought of as binary gates that control whether each memory
cell is updated, whether it is reset to zero, and whether its local state is revealed in the hidden vector,
2
respectively. The activations of these gates are based on the sigmoid function and hence allowed to
range smoothly between zero and one to keep the model differentiable. The vector g ∈Rn ranges
between -1 and 1 and is used to additively modify the memory contents. This additive interaction
is a critical feature of the LSTM’s design, because during backpropagation a sum operation merely
distributes gradients. This allows gradients on the memory cells c to ﬂow backwards through time
uninterrupted for long time periods, or at least until the ﬂow is disrupted with the multiplicative
interaction of an active forget gate. Lastly, note that an implementation of the LSTM requires one
to maintain both hl
t and cl
t at every point in the network.
Gated Recurrent Unit (GRU) (4) was recently proposed as a simpler alternative to the LSTM:
(
r
z
)
=
(
sigm
sigm
)
Wl
r
(
hl−1
t
hl
t−1
) ˜hl
t = tanh(Wl
xhl−1
t + Wl
g(r ⊙hl
t−1))
hl
t = (1 −z) ⊙hl
t−1 + z ⊙˜hl
t
Here, Wl
r are [2n ×2n] and Wl
g and Wl
x are [n ×n]. The GRU has the interpretation of computing
a candidate hidden vector ˜hl
t and then smoothly interpolating towards it, as gated by z.
3.2 Character-level Language Modeling
We use character-level language modeling as an interpretable testbed for sequence learning. In this
setting, the input to the network is a sequence of characters and the network is trained to predict
the next character in the sequence with a Softmax classiﬁer at each time step. Concretely, assuming
a ﬁxed vocabulary of K characters we encode all characters with K-dimensional 1-of-K vectors
{xt}, t= 1, . . . , T, and feed these to the recurrent network to obtain a sequence of D-dimensional
hidden vectors at the last layer of the network{hL
t }, t= 1, . . . , T. To obtain predictions for the next
character in the sequence we project this top layer of activations to a sequence of vectors{yt}, where
yt = WyhL
t and Wy is a [K ×D] parameter matrix. These vectors are interpreted as holding the
(unnormalized) log probability of the next character in the sequence and the objective is to minimize
the average cross-entropy loss over all targets.
3.3 Optimization
Following previous work (28) we initialize all parameters uniformly in range [−0.08, 0.08]. We use
mini-batch stochastic gradient descent with batch size 100 and RMSProp (6) per-parameter adaptive
update with base learning rate2×10−3 and decay 0.95. These settings work robustly with all of our
models. The network is unrolled for 100 time steps. We train each model for 50 epochs and decay
the learning rate after 10 epochs by multiplying it with a factor of 0.95 after each additional epoch.
We use early stopping based on validation performance and cross-validate the amount of dropout for
each model individually. Our supplementary material contains additional details regarding efﬁcient
implementation and further experiments with other initialization ranges and updates.
4 Experiments
Datasets. Two datasets previously used in the context of character-level language models are the
Penn Treebank dataset (20) and the Hutter Prize 100MB of Wikipedia dataset (16) . However, both
datasets contain a mix of common language and special markup. Our goal is not to compete with
previous work but rather to study recurrent networks in a controlled setting and on both ends on
the spectrum of degree of structure. Therefore, we chose to use Leo Tolstoy’sWar and Peace(WP)
novel, which consists of 3,258,246 characters of almost entirely English text with minimal markup,
and at the other end of the spectrum the source code of the Linux Kernel (LK). We shufﬂed all
header and source ﬁles randomly and concatenated them into a single ﬁle to form the 6,206,996
character long dataset. We split the data into train/val/test splits as 80/10/10 for WP and 90/5/5 for
LK. Therefore, there are approximately 300,000 characters in the validation/test splits in each case.
The total number of characters in the vocabulary is 87 for WP and 101 for LK.
4.1 Comparing Recurrent Networks
We ﬁrst train several recurrent network models to support further analysis and to compare their
performance in a controlled setting. In particular, we train models in the cross product of type
(LSTM/RNN/GRU), number of layers (1/2/3), number of parameters (4 settings), and both datasets
(WP/KL). The 4 parameter sizes were chosen to be in units of 1-layer LSTMs with 64, 128, 256,
and 512 cells. With our character vocabulary sizes this gives approximately 50K, 130K, 400K, and
1.3M parameters respectively. The sizes of hidden layers in the other models were always chosen to
match the closest prototype size as close as possible.
3
LSTM RNN GRU
Layers 1 2 3 1 2 3 1 2 3
Size War and Peace Dataset
64 1.449 1.442 1.540 1.446 1.401 1.396 1.398 1.373 1.472
128 1.277 1.227 1.279 1.417 1.286 1.277 1.230 1.226 1.253
256 1.189 1.137 1.141 1.342 1.256 1.239 1.198 1.164 1.138
512 1.161 1.092 1.082 - - - 1.170 1.201 1.077
Linux Kernel Dataset
64 1.355 1.331 1.366 1.407 1.371 1.383 1.335 1.298 1.357
128 1.149 1.128 1.177 1.241 1.120 1.220 1.154 1.125 1.150
256 1.026 0.972 0.998 1.171 1.116 1.116 1.039 0.991 1.026
512 0.952 0.840 0.846 - - - 0.943 0.861 0.829
GRU­2 (512)
GRU­3 (256)
GRU­1 (128)
LSTM­2 (512)
LSTM­1 (64)
GRU­1 (64)
LSTM­2 (256)
GRU­3 (128)
RNN­2 (256)
RNN­3 (128)
GRU­1 (512)
RNN­3 (64)
RNN­1 (64)
GRU­2 (128)
LSTM­2 (64)
RNN­1 (128)
LSTM­3 (512)
RNN­2 (64)
GRU­2 (256)
GRU­3 (64)
LSTM­1 (256)
LSTM­1 (128) LSTM­3 (128)
GRU­2 (64)
LSTM­3 (256)
LSTM­1 (512)
LSTM­3 (64)
GRU­1 (256)
RNN­2 (128)
RNN­1 (256)
RNN­3 (256)
LSTM­2 (128)
GRU­3 (512)
Figure 1: Left: The test set cross-entropy loss for all models and datasets (low is good). Models
in each row have nearly equal number of parameters. The test set has 300,000 characters. The
standard deviation, estimated with 100 bootstrap samples, is less than 4 ×10−3 in all cases. Right:
A t-SNE embedding based on the probabilities assigned to test set characters by each model on War
and Peace. The color, size, and marker correspond to model type, model size, and number of layers.
The test set results are shown in Figure 1. Our consistent ﬁnding is that depth of at least two is
beneﬁcial. However, between two and three layers our results are mixed. Additionally, the results
are mixed between the LSTM and the GRU, but both signiﬁcantly outperform the RNN. We also
computed the fraction of times that each pair of models agree on the most likely character and use it
to render a t-SNE (29) embedding (we found this more stable and robust than the KL divergence).
The plot (Figure 1, right) further supports the claim that the LSTM and the GRU make similar
predictions while the RNNs form their own cluster.
4.2 Internal Mechanisms of an LSTM
Interpretable, long-range LSTM cells. LSTMs can in principle use their memory cells to remem-
ber long-range information and keep track of various attributes of text they are in. For instance, it is
a simple exercise to write down toy cell weights that would allow the cell to keep track of whether
it is inside a quoted string. However, to our knowledge, the existence of such cells has never been
experimentally demonstrated on real-world data. In particular, it could be argued that even if the
LSTM is in principle capable of using these operations, practical optimization challenges (i.e. SGD
dynamics, or approximate gradients due to truncated backpropagation through time) might prevent
it from discovering these solutions. In this experiment we verify that multiple interpretable cells do
in fact exist in these networks. We show several examples in Figure 2. Note that truncated backprop-
agation prevents the gradient signal from noticing dependencies longer than 100 characters, but we
still observe cells that reliably keep track of quotes or comment blocks for periods much longer than
100 characters. We hypothesize that these cells ﬁrst develop on patterns shorter than 100 characters
but then generalize to longer sequences.
Gate activation statistics. We also recorded and examined the statistics of gate activations in the
networks while evaluating the test set. We were particularly interested in studying the distributions of
saturation regimes in the networks, where we deﬁne a gate to be left or right-saturated if its activation
is less than 0.1 or more than 0.9, respectively. We then compute the fraction of times that each gate’s
activation falls into either category and show these statistics in Figure 3. The amount of right-
saturated forget gate activations in an LSTM is particularly interesting, since this corresponds to
cells that remember their value from the previous iteration. For instance, note that there are multiple
cells that are almost always right-saturated and hence function as perfect integrators. Conversely,
there are no cells that function in purely feed-forward fashion, as these would show as consistently
left-saturated forget gates. The output gate statistics also reveal that there are no cells that get
consistently revealed to the hidden state. Lastly, a surprising fact is that unlike the other two layers,
the activations in the ﬁrst layer are diffuse. This is a ﬁnding that we struggle to explain but it is
present across all of our models, including GRUs.
4.3 Understanding Long-Range Interactions
Good performance of LSTMs is frequently attributed to their ability to store long-range information.
In this section we test this hypothesis by comparing an LSTM with baseline models that can only
utilize information from a ﬁxed number of previous steps. In particular, we consider two baselines:
4
Figure 2: Several examples of cells with interpretable activations discovered in our best Linux Ker-
nel and War and Peace LSTMs. Text color corresponds to tanh(c), where -1 is red and +1 is blue.
Figure 3: Left three: Saturation plots for an LSTM. Each circle is a gate in the LSTM and its
position is determined by the fraction of time it is left and right-saturated. These fractions must add
to at most one (indicated by the diagonal line).Right two: Saturation plot for a 3-layer GRU model.
1. n-NN: A fully-connected neural network with one hidden layer and tanh nonlinearities. The
input to the network is a sparse binary vector of dimension nK that concatenates the one-of- K
encodings of n consecutive characters. We optimize the model as described in Section 3.3 and
cross-validate the size of the hidden layer.
2. n-gram: An unpruned (n + 1)-gram character-level language model using modiﬁed Kneser-
Ney smoothing (3). This is a standard smoothing method for language models (15). All models
were trained using the popular KenLM software package (13).
Performance comparisons. The performance of both n-gram models is shown in Table 2. The
n-gram and n-NN models perform nearly identically for small values of n, but for larger values
the n-NN models start to overﬁt and the n-gram model performs better. Moreover, we see that on
both datasets our best recurrent network outperforms the 20-gram model (1.077 vs. 1.195 on WP
and 0.84 vs.0.889). It is difﬁcult to make a direct model size comparison, but the 20-gram model
ﬁle has 3GB, while our largest checkpoints are 11MB. However, the assumptions encoded in the
Kneser-Ney smoothing model are intended for word-level modeling of natural language and may
not be optimal for character-level data. Despite this concern, these results provide some evidence
that the recurrent networks are effectively utilizing information beyond 20 characters.
Error Analysis. It is instructive to delve deeper into the errors made by both recurrent networks and
n-gram models. In particular, we deﬁne a character to be an error if the probability assigned to it
by a model is below 0.5. Figure 4 (left) shows the overlap between the test-set errors for the 3-layer
LSTM, and the best n-NN and n-gram models. We see that the majority of errors are shared by all
three models, but each model also has its own unique errors.
5
PPPPPPModel
n 1 2 3 4 5 6 7 8 9 20
War and Peace Dataset
n-gram 2.399 1.928 1.521 1.314 1.232 1.203 1.194 1.194 1.194 1.195
n-NN 2.399 1.931 1.553 1.451 1.339 1.321 - - - -
Linux Kernel Dataset
n-gram 2.702 1.954 1.440 1.213 1.097 1.027 0.982 0.953 0.933 0.889
n-NN 2.707 1.974 1.505 1.395 1.256 1.376 - - - -
Table 2: The test set cross-entropy loss on both datasets for n-gram models (low is good). The
standard deviation estimate using 100 bootstrap samples is below 4 ×10−3 in all cases.
@ = ; } \t ) { ] : <s>W M j S 9 7 3 J H V0.0
0.2
0.4
0.6
0.8
1.0Mean probability
LSTM 20­gram
\r <s> " 7 d 8 ? y k n ( G : V q J 2 9 5 00.0
0.2
0.4
0.6
0.8
1.0Mean probability
LSTM 20­gram
Figure 4: Left: Overlap between test-set errors between our best 3-layer LSTM and the n-gram
models (low area is good).Middle/Right: Mean probabilities assigned to a correct character (higher
is better), broken down by the character, and then sorted by the difference between two models.
“<s>” is the space character. LSTM (red) outperforms the 20-gram model (blue) on special char-
acters that require long-range reasoning. Middle: LK dataset, Right: WP dataset.
To gain deeper insight into the errors that are unique to the LSTM or the 20-gram model, we compute
the mean probability assigned to each character in the vocabulary across the test set. In Figure 4
(middle,right) we display the 10 characters where each model has the largest advantage over the
other. On the Linux Kernel dataset, the LSTM displays a large advantage on special characters that
are used to structure C programs, including whitespace and brackets. The War and Peace dataset
features an interesting long-term dependency with the carriage return, which occurs approximately
every 70 characters. Figure 4 (right) shows that the LSTM has a distinct advantage on this character.
To accurately predict the presence of the carriage return the model likely needs to keep track of
its distance since the last carriage return. The cell example we’ve highlighted in Figure 2 (top,
left) seems particularly well-tuned for this speciﬁc task. Similarly, to predict a closing bracket
or quotation mark, the model must be aware of the corresponding open bracket, which may have
appeared many time steps ago. The fact that the LSTM performs signiﬁcantly better than the 20-
gram model on these characters provides strong evidence that the model is capable of effectively
keeping track of long-range interactions.
Case study: closing brace . Of these structural characters, the one that requires the longest-term
reasoning is the closing brace (“ }”) on the Linux Kernel dataset. Braces are used to denote blocks
of code, and may be nested; as such, the distance between an opening brace and its corresponding
closing brace can range from tens to hundreds of characters. This feature makes the closing brace
an ideal test case for studying the ability of the LSTM to reason over various time scales. We
group closing brace characters on the test set by the distance to their corresponding open brace and
compute the mean probability assigned by the LSTM and the 20-gram model to closing braces within
each group. The results are shown in Figure 5 (left). We see not only that the LSTM consistently
outperforms the 20-gram model at predicting closing braces, but that the performance difference
increases up to a distance of 60. After this point the performance difference between the models
remains relatively constant, suggesting that no additional time horizon ranges are being captured
and that perhaps the LSTM has trouble keeping track of braces longer than this amount.
Training dynamics. We can gain further insight into the training dynamics of the LSTM by com-
paring it with trained n-NN models during training using the (symmetric) KL divergence between
the predictive distributions on the test set. We visualize the divergence and the difference in the mean
loss in Figure 5 (right). Notably, we see that in the ﬁrst few iterations the LSTM behaves like the 1-
NN model but then diverges from it soon after. The LSTM then behaves most like the 2-NN, 3-NN,
and 4-NN models in turn. This experiment suggests that the LSTM “grows” its competence over
increasingly longer dependencies. We believe that this insight is related to why Sutskever et al. (28)
6
0­1010­3030­4040­5050­6060­7070­8080­9090­100100­200200­300300­400
Distance between "{" and "}"
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Difference between P("}")
for LSTM and 20­gram
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5
LSTM epoch
1
2
3
4
5
6
7
8mean KL
1­NN
2­NN
3­NN
4­NN
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5
LSTM epoch
−2.5
−2.0
−1.5
−1.0
−0.5
0.0
0.5
1.0
1.5
2.0
∆loss
Figure 5: Left: Mean difference between the probabilities that the LSTM and 20-gram model
assign to the “ }” character, bucketed by the distance to the matching “ {”. Right: Comparison of
the similarity between 3-layer LSTM and the n-NN baselines over the ﬁrst 3 epochs of training, as
measured by the symmetric KL-divergence (middle) and the test set loss (right). Low KL indicates
similar predictions, and positive ∆loss indicates that the LSTM outperforms the baseline.
had to reverse the source sentences in their encoder-decoder architecture for machine translation.
Without the inversion it is as if the model was immediately presented with n-grams of very high n.
This prevents the RNN from slowly “growing” its competence from shorter to longer dependencies.
4.4 Error Analysis: Breaking Down the Failure Cases
In this section we break down LSTM’s errors into categories to understand its limitations, the relative
severity of each error type, and to suggest areas for further study. We focus on the War and Peace
dataset where it is easier to analyze the errors. Our approach is to “peel the onion”by iteratively
removing the errors with a series of constructed oracles. We discuss these ﬁrst:
n-gram oracle. First, we construct optimistic n-gram oracles that eliminate errors that might be
ﬁxed with better modeling of the previousn characters. In particular, we evaluate ourn-gram model
(n = 1, . . . ,9) and remove a character error if it is correctly classiﬁed by any of these models.
Dynamic n-long memory oracle. Consider the string “Jon yelled at Mary but Mary couldn’t hear
him. ”One interesting and consistent failure mode is that if the LSTM fails to recognize the ﬁrst
occurrence of “Mary” then it will almost always also fail to recognize the second, with an identical
pattern of errors. In principle the ﬁrst mention should make the second easier: the LSTM could
conceivably store a summary of previously seen characters in the data and fall back on this memory
when it is uncertain. However, this does not appear to take place in practice. This observation is
related to the improvements seen in “dynamic evaluation” (21; 17) of recurrent language models,
where an RNN is allowed to train on the test set characters during evaluation as long as it sees them
only once. This oracle optimistically removes errors in all words (starting with the second character)
that can also be found as a substring in the last n characters (we use n ∈{100, 500, 1000, 50000}).
Rare words oracle. Next, we construct an oracle that eliminates errors for rare words that occur
only up to n times in the training data ( n = 0, . . . ,5). Note that a character-level model could in
principle hypothesize the spelling of unseen words by recognizing common letter patterns. Errors
corrected by this oracle indicate its failure to do so.
Word model oracle. We noticed that a large portion of the errors occur on the ﬁrst character of each
word. Intuitively, the task of selecting the next word in the sequence is much harder than completing
the last few characters of a known word. We constructed an oracle that eliminated all errors after
a space, quote or a newline. Interestingly, a high portion of errors can be found after a newline
character, since the models have to learn that new line is identical to a space.
Punctuation oracle. This oracle removes errors on all punctuation.
Boost oracles. The remaining errors that do not show salient structures or patterns are removed by
an oracle that boosts the probability of the correct letter by a ﬁxed amount. These oracles allow us
to understand the distribution of the difﬁculty of the remaining errors.
We now subject two LSTM models to the error analysis: First, our best LSTM model and second,
the best LSTM model in the smallest model category (50K parameters). The smaller second model
will allow us to quantify the reduction in each error category as we scale up our models. For the
following analysis we deﬁne a character to be an error when its assigned probability is less than
0.5. We then iterate over oracles in the order presented above, remove each error type in turn and
visualize the fractions of errors attributed to each type in Figure 6.
7
1-gram to 5-gram
Punctuation
Up to 500 memory
Less than 3 training examples of word
After space or quote
After newline
0.4 to 0.5 boost
LSTM-3 (512)
LSTM-2 (64)
Figure 6: Left: LSTM errors removed one by one with oracles, starting from top of the pie chart
and going counter-clockwise. The area of each slice corresponds to fraction of errors contributed.
“n-memory” refers to dynamic memory oracle with context ofn previous characters. “Wordt-train”
refers to the rare words oracle with word count threshold oft. Right: Concrete examples of text from
the test set for each error type. Blue color highlights the relevant characters with the associated error.
For the memory category we also highlight the repeated substrings with red bounding rectangles.
The error breakdown. The best LSTM model makes a total of 140K errors out of 330K test set
characters (42%). Of these, the n-gram oracle eliminates 18%, suggesting that the model is not
taking full advantage of the last 9 characters. The dynamic memory oracle eliminates 6% of the
errors. In principle, a dynamic evaluation scheme could be used to mitigate this type of error, but
we believe that a more principled approach could involve a model similar to Memory Networks
(32), where the model is allowed to attend to a recent history of the sequence while making its next
prediction. Finally, the rare words oracle accounts for 9% of the errors. This error type could be
mitigated with transfer learning, or by increasing the size of the training set. The majority of the
remaining errors (37%) follow a space, a quote, or a newline. This suggests that substantially longer
time dependencies, and possibly hierarchical context models, could eliminate a large fraction of the
errors. See Figure 6 for examples of each error type.
Errors eliminated by scaling up. The smaller LSTM model makes a total of 184K errors (or 56%
of the test set), approximately 44K more than the large model. Surprisingly, 36K of these errors
(81%) are n-gram errors, 5K come from the boost category, and the remaining 3K are distributed
across the other categories relatively evenly. That is, scaling the model up by a factor 26 in the
number of parameters has almost entirely provided gains in the local, n-gram error rate and has left
the other error categories untouched in comparison. This analysis suggests that in order to fully
remove all errors it might be necessary to develop new architectural improvements instead of simply
scaling up the basic model. For full details of this analysis refer to our supplementary material.
5 Conclusion
We have presented a comprehensive analysis of Recurrent Neural Networks and their representa-
tions, predictions and error types. In particular, qualitative visualization experiments, cell activation
statistics and in-depth comparisons to ﬁnite horizonn-gram models demonstrate that these networks
learn powerful, long-range interactions. We have also conducted a detailed error analysis that illumi-
nates the limitations of recurrent networks and allows us to suggest speciﬁc areas for further study.
In particular, n-gram errors can be signiﬁcantly reduced by scaling up the models and rare words
could be addressed with bigger datasets. However, further architectural innovations may be needed
to eliminate the remaining errors.
8
References
[1] D. Bahdanau, K. Cho, and Y . Bengio. Neural machine translation by jointly learning to align and translate.
arXiv preprint arXiv:1409.0473, 2014.
[2] Y . Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difﬁcult.
Neural Networks, IEEE Transactions on, 5(2):157–166, 1994.
[3] S. F. Chen and J. Goodman. An empirical study of smoothing techniques for language modeling. Com-
puter Speech & Language, 13(4):359–393, 1999.
[4] K. Cho, B. van Merri ¨enboer, D. Bahdanau, and Y . Bengio. On the properties of neural machine translation:
Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014.
[5] J. Chung, C. Gulcehre, K. Cho, and Y . Bengio. Empirical evaluation of gated recurrent neural networks
on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
[6] Y . N. Dauphin, H. de Vries, J. Chung, and Y . Bengio. Rmsprop and equilibrated adaptive learning rates
for non-convex optimization. arXiv preprint arXiv:1502.04390, 2015.
[7] J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell.
Long-term recurrent convolutional networks for visual recognition and description. CVPR, 2015.
[8] J. Duchi, E. Hazan, and Y . Singer. Adaptive subgradient methods for online learning and stochastic
optimization. The Journal of Machine Learning Research, 12:2121–2159, 2011.
[9] A. Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013.
[10] A. Graves, A.-R. Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks.
In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages
6645–6649. IEEE, 2013.
[11] A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014.
[12] K. Greff, R. K. Srivastava, J. Koutn ´ık, B. R. Steunebrink, and J. Schmidhuber. LSTM: A search space
odyssey. CoRR, abs/1503.04069, 2015.
[13] K. Heaﬁeld, I. Pouzyrevsky, J. H. Clark, and P. Koehn. Scalable modiﬁed Kneser-Ney language model
estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics,
pages 690–696, Soﬁa, Bulgaria, August 2013.
[14] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
[15] X. Huang, A. Acero, H.-W. Hon, and R. Foreword By-Reddy. Spoken language processing: A guide to
theory, algorithm, and system development. Prentice Hall PTR, 2001.
[16] M. Hutter. The human knowledge compression contest. 2012.
[17] F. Jelinek, B. Merialdo, S. Roukos, and M. Strauss. A dynamic language model for speech recognition.
In HLT, volume 91, pages 293–295, 1991.
[18] A. Joulin and T. Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. CoRR,
abs/1503.01007, 2015.
[19] A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. CVPR,
2015.
[20] M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini. Building a large annotated corpus of english: The
penn treebank. Computational linguistics, 19(2):313–330, 1993.
[21] T. Mikolov. Statistical language models based on neural networks. Presentation at Google, Mountain
View, 2nd April, 2012.
[22] T. Mikolov, M. Karaﬁ ´at, L. Burget, J. Cernock `y, and S. Khudanpur. Recurrent neural network based
language model. In INTERSPEECH 2010, 11th Annual Conference of the International Speech Commu-
nication Association, Makuhari, Chiba, Japan, September 26-30, 2010, pages 1045–1048, 2010.
[23] R. Pascanu, C ¸ . G¨ulc ¸ehre, K. Cho, and Y . Bengio. How to construct deep recurrent neural networks.CoRR,
abs/1312.6026, 2013.
[24] R. Pascanu, T. Mikolov, and Y . Bengio. On the difﬁculty of training recurrent neural networks. arXiv
preprint arXiv:1211.5063, 2012.
[25] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error propagation.
Technical report, DTIC Document, 1985.
[26] J. Schmidhuber. Deep learning in neural networks: An overview. Neural Networks, 61:85–117, 2015.
Published online 2014; based on TR arXiv:1404.7828 [cs.NE].
[27] I. Sutskever, J. Martens, and G. E. Hinton. Generating text with recurrent neural networks. InProceedings
of the 28th International Conference on Machine Learning (ICML-11), pages 1017–1024, 2011.
[28] I. Sutskever, O. Vinyals, and Q. V . Le. Sequence to sequence learning with neural networks. InAdvances
in Neural Information Processing Systems, pages 3104–3112, 2014.
[29] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research,
9(2579-2605):85, 2008.
[30] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator.CVPR,
2015.
[31] P. J. Werbos. Generalization of backpropagation with application to a recurrent gas market model. Neural
Networks, 1(4):339–356, 1988.
[32] J. Weston, S. Chopra, and A. Bordes. Memory networks. CoRR, abs/1410.3916, 2014.
9
Supplementary Material
Exact Model Sizes
LSTM RNN GRU
Layers 1 2 3 1 2 3 1 2 3
Size War and Peace Dataset
64 64 45 37 141 96 78 77 53 44
128 128 89 68 273 174 139 151 98 79
256 256 159 126 531 325 257 300 185 146
512 512 308 241 1045 623 487 596 357 280
Linux Kernel Dataset
64 64 46 38 142 98 81 78 54 45
128 128 85 69 274 178 143 152 100 81
256 256 161 128 534 329 260 300 187 149
512 512 310 243 1048 628 492 596 359 282
Table 3: Exact sizes of the models used in all experiments. The models in each row have approxi-
mately the same number of parameters.
Training Details
In this section we provide additional details regarding training.
Efﬁcient implementation . It is impractical to compute the loss function over the entire training sequence,
which can in practice be be millions of characters long. Instead, a good strategy is to take the input sequence of
characters are reshape it into one data matrix Xof size [B×N], where Bis the batch size (we use B= 100).
We then march from left to right in chunks of length S(we use S = 100), and every non-overlapping block of
[B×S] is fed as input to the network. The ground truth labels matrix Y is the same as X, but shifted by 1
columnwise (i.e. next character in the sequence). It is important to keep track of the hidden state activationshl
T
at the last time step in the previous batch and copy them into the vectors hl
0 in the next batch. We evaluate the
loss and backpropagate only on the [B×S] chunk of data. In particular, note that the backpropagated signal is
truncated S steps backwards in time, limiting the extent of long term interactions that the networks can learn.
In practice the networks do learn much longer dependencies (as supported by our qualitative cell visualization
experiments), but this can liekly only happen if the same dependencies also occur on time scales less than S
and then happen to generalize to longer sequences.
Initialization. Following previous work (28) we initialize all parameters uniformly in the range [−0.08,0.08].
We experimented with multiple settings of this hyperparameter on multiple architectures and found its sensi-
tivity to be relatively low. In particular, It is safer to err on the side of making this range smaller, since some
models start to diverge when it is greater than 1, but the performance degrades gracefully, even down to 10−4
and less.
Optimization. We experimented with multiple per-parameter learning rate updates and consistently found
RMSProp (6) to provide the best results, and with the widest tolerance to the learning rate setting. Adagrad
(8) also provided a wide tolerance but its performance did not match that of RMSProp. SGD with momentum
was able to nearly match the performance of RMSProp but required a precise setting of the learning rate. For
example, for a 3-layer LSTM trained with SGD converged only in the range[10−0.4,10−0.8] and diverged oth-
erwise, while RMSProp diverged outside of[10−2.2,10−3.2] interval. We clipped the gradients elementwise in
range [−5,5] and found [−1,1] to be too aggressive, at least in our setting (these settings correspond to case
where we average the loss over both the batch size and the sequence length). We did not use L2 regularization
because cross-validation always preferred very small coefﬁcients. We used early stopping based on the valida-
tion performance. We decay the learning rate after 10 epochs by multiplying it with a factor of 0.95 after each
additional epoch. We train each model for 50 epochs.
Activation Distributions
During our analysis we noticed a distinct peak of the tanh c (Figure 7, bottom) at ±0.76, which is exactly
tanh(1). This is caused by the fact that the largest amount that the LSTM can add or subtract to a cell is 1,
since gis the output of tanh. This raises a concern because the hidden statehis computed as h= o⊙tanh(c).
Therefore if a cell happens to be zero because it was reset in the previous iteration, the LSTM does not have the
capacity to fully saturate the cell in a single time step. We investigated a simple ﬁx to this problem by changing
the update to the form cl
t = f ⊙cl
t−1 + αi⊙g where αis a constant. We observed signiﬁcant signiﬁcant
reduction in performance for α< 0.25. Using α= 2allows the LSTM to saturate the cell in one step. In this
case we observed a consistently faster convergence (e.g. 700 vs. 900 iterations to reach loss of 2.0) across all
architectures. However, the ﬁnal performance at epoch 50 remains unchanged.
10
Figure 7: Histograms of tanh(c) for 20 randomly chosen cells (each in different color) across three
layers. Distinct peaks show at tanh(1), which are caused when cell memory is reset with a forget
gate and then maimally added to with the input gate.
Figure 8: t-SNE of the weights connecting to the individual characters. Each circle corresponds to
an input/forget/output gate. For example, numbers cluster together, indicating that their inﬂuence on
the input/forget gates are very similar. Additionally, the brackets (except for curly brackets) are all
clustered in the forget gate indicating similar inﬂuence, but they are spread out in the other gates.
Weights
We examined the weights of a 1-layer LSTM to gain an understanding of the effect of the inputs on the
LSTM’s representation. First, we compute the total strength of inﬂuence that each character exhibits on the
representation, which we quantify as the sum of absolute values of its weights to all gates. For an LSTM
trained on the Linux Kernel dataset, the characters with the strongest inﬂuence turn out to be special char-
acters (\;}(’{v.w+gb+), while the characters that exhibit weakest inﬂuence include rare unicode symbols,
followed by `$789ˆ5. This suggests that the LSTMs learns to some extent ignore numerals.
We wanted to gain a more holistic understanding of the inﬂuence of each character on the activations of the
network. To this effect, we embed the rows of the LSTM’s weight matrix connecting the characters to the gates
with t-SNE (29) (Figure 8). This ﬁgure shows, for instance, that numerals, which are clustered, give rise to very
similar write/forget behavior in the LSTM.
11
Error Analysis
LSTM-3 (512) LSTM-2 (64)
Oracle # Errors Fraction of Errors # Errors Fraction of Errors
1-gram 1581 0.011 1697 0.009
2-gram 4001 0.029 4637 0.025
3-gram 2959 0.021 8200 0.045
4-gram 3425 0.025 14053 0.076
5-gram 3183 0.023 11602 0.063
6-gram 2974 0.021 8257 0.045
7-gram 2825 0.020 5859 0.032
8-gram 2406 0.017 4268 0.023
9-gram 1635 0.012 2634 0.014
100 memory 989 0.007 1074 0.006
500 memory 2714 0.019 2985 0.016
1000 memory 1407 0.010 1566 0.009
5000 memory 3409 0.024 3828 0.021
Word 0-train 3937 0.028 4026 0.022
Word 1-train 2220 0.016 2302 0.013
Word 2-train 1687 0.012 1775 0.010
Word 3-train 1557 0.011 1636 0.009
Word 4-train 1330 0.001 1377 0.008
Word 5-train 1197 0.009 1244 0.007
after space or quote 45114 0.323 45834 0.250
After newline 6843 0.049 7277 0.040
Punctuation 8544 0.061 8728 0.048
0.1 boost 3060 0.022 2445 0.013
0.2 boost 4384 0.031 4288 0.023
0.3 boost 6113 0.044 7069 0.038
0.4 boost 8337 0.060 10091 0.055
0.5 boost 11986 0.09 14993 0.082
Table 4: Full breakdown of errors corrected by all oracles for Figure 6 from main paper.
LSTM-3 (512) LSTM-2 (64)
Oracle Group # Errors Fraction of Errors # Errors Fraction of Errors
n-gram 24989 0.179 61207 0.333
memory 8519 0.061 9453 0.051
rare words 11928 0.085 12360 0.067
spacing 60501 0.433 61839 0.337
boost 33880 0.242 38886 0.212
Table 5: Breakdown of errors correct by oracles, grouped by oracle type.
12
Figure 9: A t-SNE visualization of LSTM cell state representations on the War and Peace dataset.
Each point is colored based on the character most recently read by the LSTM, and is captioned with
both the previous and next character. Spaces are shown as underscores.
Cell state representation
In Figure 9 we use t-SNE to visualize the (gated) cell statetanh(c) of an LSTM, allowing us to gain insight into
the type of information it encodes. In the lower left we see a cluster of cell states that occur immediately after
a space, and in the upper right we see a cluster of cell states where a space is about to occur. This demonstrates
that the LSTM uses its cell state both to record characters that have been recently seen and to hypothesize about
characters that may appear next.
13

A Reinforcement Learning Environment for
Polyhedral Optimizations
Alexander Brauckmann
∗
Chair for Compiler Construction
TU Dresden, Germany
alexander.brauckmann@tu-dresden.de
Andr´es Goens
∗
Chair for Compiler Construction
TU Dresden, Germany
andres.goens@tu-dresden.de
Jeronimo Castrillon
Chair for Compiler Construction
TU Dresden, Germany
jeronimo.castrillon@tu-dresden.de
Abstract—The polyhedral model allows a structured way of
deﬁning semantics-preserving transformations to improve the
performance of a large class of loops. Finding proﬁtable points
in this space is a hard problem which is usually approached by
heuristics that generalize from domain-expert knowledge. Exist-
ing problem formulations in state-of-the-art heuristics depend
on the shape of particular loops, making it hard to leverage
generic and more powerful optimization techniques from the
machine learning domain. In this paper, we propose PolyGym, a
shape-agnostic formulation for the space of legal transformations
in the polyhedral model as a Markov Decision Process (MDP).
Instead of using transformations, the formulation is based on an
abstract space of possible schedules. In this formulation, states
model partial schedules, which are constructed by actions that are
reusable across different loops. With a simple heuristic to traverse
the space, we demonstrate that our formulation is powerful
enough to match and outperform state-of-the-art heuristics. On
the Polybench benchmark suite, we found transformations that
led to a speedup of 3.39x over LLVM O3, which is 1.83x better
than the speedup achieved by ISL. Our generic MDP formula-
tion enables using reinforcement learning to learn optimization
policies over a wide range of loops. This also contributes to the
emerging ﬁeld of machine learning in compilers, as it exposes a
novel problem formulation that can push the limits of existing
methods.
Index Terms—polyhedral optimization, loop scheduling, ma-
chine learning, reinforcement learning, PolyGym
I. I NTRODUCTION
In most compute-intensive applications, a signiﬁcant amount
of the execution time is spent in loops. Loop optimization has
thus received plenty of attention in the compiler community,
where one ﬁnds as diverse strategies for optimization as are
the loops themselves. Among them, polyhedral compilation is
an important class of strategies for loop optimization which
focuses on a particular class of (nested) loops with regular
properties [1]–[3]. While this signiﬁcantly restricts the fam-
ily of loops that can be optimized, polyhedral compilation
methods are very relevant in practice. Compilers for important
domain-speciﬁc languages like TensorComprehensions [4] and
Tiramisu [5] build atop these methods. Polyhedral compilation
methods have also been used for efforts on recognizing motifs,
or well-known high-level computations [6]. The models have
been extended beyond explicit loops to support recursive func-
tions calls [7], or used to target emerging memory technologies
by exploiting the detailed information in the model [8], [9].
∗Equal contribution
With the polyhedral model we can ﬁnd semantics-preserving
transformations that signiﬁcantly improve the loops’ per-
formance. We do this by exploring constraints exposed by
geometric properties of the dependencies between statements
in different iterations of the loops. However, this exploration
requires considering a prohibitively large space of possible
code transformations. The two main methods to explore this
space are heuristics, which use domain-speciﬁc models to
directly ﬁnd a good candidate transformation, and iterative
meta-heuristics, which iteratively explore the space by evalu-
ating multiple points and adapting the solutions. Model-based
heuristics, like Pluto [10] and ISL [2], are more applicable in
practice, but the resulting code is consistently outperformed
by signiﬁcantly more time-costly iterative meta-heuristics,
like [11], [12].
A promising strategy to bridge this gap is reinforcement
learning (RL). RL has proven to be very successful for
navigating vast discrete spaces, while training a decision policy
that can later be leveraged to ﬁnd good solutions directly [13],
[14]. Precisely for this reason, RL is well-suited for learning
heuristics in the spaces of compiler optimizations, seeking a
sweet spot between applicable heuristics and iterative methods:
While iterating the search space, a heuristic model is learned
that can later be re-used in future iterations to lead to a
proﬁtable transformation.
No RL method has been applied to the search spaces of
polyhedral loop scheduling yet. This is partly because the
common way of thinking of loop optimizations in terms of
transformations, like loop fusion, ﬁssion or tiling, is not well-
suited for RL. The ﬁrst disadvantage concerns correctness:
not every transformation can be applied to every schedule.
This happens often and is computationally costly. The greater
problem is that the transformations themselves depend on the
concrete instance of the problem. For example, loop ﬁssion
needs to select the concrete statements and loop bounds,
which depend on the concrete loop. If we encode these
transformations as actions in a markov decision process (MDP)
for RL, any policy for ﬁnding proﬁtable schedules would be
speciﬁc to that loop or kernel by the very structure of the
formulation.
In this paper we propose a formulation that overcomes
the limitations of transformations for an MDP. We utilize a
known formulation [15], [16] that leverages Farkas’ lemma
arXiv:2104.13732v2  [cs.LG]  29 Apr 2021
to construct an abstract space of exclusively valid schedules.
We give present a clear overview of the background from
discrete geometry, emphasizing why the different steps in the
construction are important for this use-case (Section II). Our
MDP navigates this abstract space in an instance-independent
fashion by leveraging its algebraic properties (Section IV).
The resulting search space boasts the potential for achieving
great performance and learning, a fact we demonstrate by
traversing it with simple heuristics (Section V). Even without
sophisticated RL algorithms, simple heuristics that guide the
iterative process outperform the state-of-the-art heuristics in
ISL [2] in the Polybench benchmark suite [17] in a small
number of iterations. Importantly, as well, the ﬁeld of machine
learning for compilers is an emerging ﬁeld of study on its
own [18]. RL algorithms are a particularly relevant class of
methods for compiler optimization [19], and have successfully
been applied to certain subproblems already [20]–[22]. Even
in the industrial LLVM compiler, a RL-trained heuristic for in-
lining was recently integrated into the mainline version [23]. 1
However, deﬁning RL environments for problems in compiler
optimization is challenging in general, for many of the same
reasons it is challenging in the case of polyhedral optimiza-
tions. Deﬁning the problem in a way that action policies can
be learned in a uniform fashion, generalizing across multiple
samples, is an important step to advance the ﬁeld of machine
learning in compilers as well.
II. B ACKGROUND : POLYHEDRAL COMPILATION
This section introduces the basics of the polyhedral model,
based on the original work of Feautrier [15], [16], and the
algorithms for schedule space construction due to Pouchet
and others [11], [24]. Consider the following C kernel for the
matrix-vector multiplication y= Ax:
int i, j;
for (i = 0; i < N; i++)
S: y[i] = 0;
for (i = 0; i < N; i++)
for (j = 0; j < N; j++)
T: y[i] += A[i][j] *x[j];
The iterations of the loops in this kernel exhibit a regular
structure. The set of points i,j of the values the variables i,j
take during the execution of the loops is I := {i,j ∈Z|0 ≤
i,j ≤N −1}. This kernel is what we call a Static Control
Part (SCoP). A detailed description of SCoPs and how they
can be constructed can be found in [3]. For the purposes of
this paper however, it sufﬁces to know that these programs
have regular iteration spaces. Concretely, if we consider the
iteration variables in the loops of the SCoP, they have a
geometric interpretation as a lattice polytope. A polytope is
the n-dimensional generalization of a polygon in 2 dimensions
and polyhedral in 3 dimensions. The lattice contains a set
of discrete points (e.g. integer values) in the interior of the
polytope. For example, the set I of the example kernel can be
1https://github.com/llvm/llvm-project/blob/main/llvm/lib/Analysis/
InlineAdvisor.cpp
seen as the discrete points in the lattice Z2, restricted to an
N ×N square starting at the origin, as depicted in Figure 1.
i
j
Fig. 1. The lattice polytope interpretation of the iteration space of the matrix-
vector multiplication kernel for N=5.
In the kernel, there are two distinct statements that are
executed in the loop, which we labelled by S and T. The
ﬁrst statement S : y[i] = 0 does not actually depend on j,
and is executed N times, which we denote by S(i) for i =
0,...,N −1. The statement T : y[i] += A[i,j]*x[j]
depends on both variables i,j and is executed N2 times,
which we denote by T(i,j) for i,j = 0 ,...,N −1. In
particular, executing this kernel amounts to executing these
N2 + N statements. Analyzing the statements we can see
which statements have to be executed before which, and where
it does not matter. Indeed, we know that for every i, the
instance S(i) has to be executed before all T(i,j). Similarly,
we need to execute T(i,j) before we execute T(i,j + 1)
for all valid i,j. 2 We can use the geometric interpretation
as polytopes (in this example, polygons) to visualize these
dependencies, as in Figure 2.
i
j
i
j
i
j
i
Dependency T(i, j) → S(i)
Dependency T(i, j+ 1)→ T(i, j)
Fig. 2. The geometric interpretation of dependencies in the matrix-vector
multiplication kernel for N=5.
Any permutation of the N2 + N statements S(i),T(i,j)
that respects the dependencies visualized in Figure 2
will produce the same (correct) result. They are all
semantics-preserving permutations. For example, the
2If we know that addition is commutative, we can relax these conditions on
the T(i, j), but we ignore the commutativity of addition for this discussion.
original code executes the statements in the order
S(0),S(1),...,S (N − 1),T(0,0),T(0,1),...,T (N −
1,N −1). However, we could also execute them in the
order S(0),T(0,0),S(1),T(0,1),T(1,0),S(2),...,S (N −
1),T(0,N −1),...,T (N −1,0),T(1,N −1),...,T (N −
1,N −2),T(N−1,N −1), as this respects all dependencies.
This order would translate to the code:
int i, j;
for (i = 0; i < N; i++){
y[i] = 0;
for (j = 0; j <= i; j++)
y[j] += A[j][i-j]*x[i-j];}
for (i = 0; i < N; i++)
for (j = i+1; j < N; j++)
y[j] += A[j][i-j+N]*x[i-j+N];
The semantics-preserving permutations of the statements
can be represented as schedules. Using the lexicographic
ordering, we can encode schedules as vectors. For example,
the original statement order for N = 5 could be encoded as the
1-dimensional vectors for S(i) = (i), and T(i,j) = i+5·j+5
respectively, which results in the order (0),..., (4) for S
and (5),..., (29) for T, encoding the order of the statement
executions. The crucial observation is that these vectors are
afﬁne functions
ΘS : Z2 →Z1,(i,j) ↦→(1,0,0)T (i,j, 1),
ΘT : Z2 →Z1,(i,j) ↦→(1,5,5)T (i,j, 1).
This schedule is called the identity schedule, as it represents
the original untransformed code.
In general, when the schedule transformations are afﬁne
linear functions, we can use a powerful result of discrete geom-
etry called Farkas’ lemma in its afﬁne form [25] to characterize
all such transformations [15]. The transformed schedule we
showed above, for example, has the order 0,2,5,9,14 for the
instances S(i),i = 1,..., 4, which is not an afﬁne function
of i. We can express the schedule as an afﬁne function,
however, if we use two-dimensional vectors S(i) = (i,0) and
T(i,j) = (i+j,i+1). These vectors, sorted lexicographically,
give the order of the schedule above and are afﬁne functions as
well. When scheduling SCoPs, this characterization results in
a set of linear inequalities describing the valid schedules [24].
This set of inequalities comes about by encoding each depen-
dency between statements as a precedence constraint on the
iteration spaces. Farkas’ lemma then allows us to transform
afﬁne functions respecting these dependencies into this set of
inequalities. For example, for the matrix-vector multiplication
kernel, the valid 1-dimensional schedules are integer solutions
to the inequalities on the 11 variables i0,...,i 10
i2 >0 and i6 ≥i3 and i7 ≥i4
and i8 ≥i5 and i7 ≥i0 −i1 + i3 + i4 −i6
and i10 >i3 + i4 + i5 −i6 −i7 −i8 + i9.
(1)
Geometrically, this set of inequalities can be equivalently
considered as another lattice polytope, where each point in
this polytope represents a valid schedule. Importantly, this
polytope should not be confused with those for the iteration
space or the dependencies, where the points represent the
values of the iteration variables. Figure 3 depicts this set
for one-dimensional schedules. The space of valid schedules
is itself a polytope in 11 dimensions, corresponding to the
11 variables in Equation 1. As such, we depict it with a
parallel plot, where each horizontal line in the grid is a
dimension and each point has a different color. In fact, this
lattice polytope has an inﬁnite number of points; we show
only the meet of the polytope with the fundamental region
of the lattice (∼= Z11) in this plot. Each point represents a
schedule, like the original schedule which is depicted explicitly
in the ﬁgure. Note that one-dimensional refers to the schedule
vectors, i.e. the images of ΘS : Z→Z1 and ΘT : Z2 →Z1.
This dimension is different from the points depicted, which
encode the constraints as characterized by Farkas’s lemma.
The polytope corresponding to the constraints in Equation 1
includes both schedule functions, ΘS and ΘT .
i
j
i
−2
0
2
Fig. 3. The geometric interpretation of the space of valid one-dimensional
schedules in the matrix-vector multiplication kernel for N=5. The space is a
lattice polytope on 11 dimensions, visualized as a parallel plot.
Any valid schedule can be expressed as a one-dimensional
transformation function. The caveat is, however, that this
transformation is not necessarily an afﬁne function. As such,
Farkas’ lemma does not apply, and we cannot capture these
schedules in the lattice polytope of valid one-dimensional
transformations. We can use more dimensions to describe more
schedules as afﬁne functions [16]. To deﬁne a k-dimensional
schedule, we deﬁne k different polytopes P1,...,P k using
Farkas’ lemma, as described above for the one-dimensional
case (cf. Figure 3). Each polytope Pd encodes the functions for
the corresponding component, i.e. (ΘS)d,(ΘT )d. In the exam-
ple of the two-dimensional schedule for the matrix-vector mul-
tiplication example, we have (ΘS)1(i,j) = i,(ΘS)2(i,j) = 0
and similarly, (ΘT )1(i,j) = i+ 1,(ΘT )2(i,j) = i+ 1. Each
of these is an afﬁne function of the inputs.
The characerization of the space of all possible multi-
dimensional schedules is not as straightforward as it is in the
one-dimensional case. For this we distinguish between two
types of precedence constraints. For a dependency S →T we
say that the polytope Pi carries the dependency strongly if it
encodes that the value of ΘS <ΘT . If instead it only encodes
the weaker constraint ΘS ≤ΘT , we say that Pi carries this
dependency weakly. While this does not enforce the constraint,
it ensures that this constraint is not violated either. A proper
deﬁnition of these concepts can be found in [11]. For an
example, consider the dependency S →T in the matrix vector
multiplication. In the the two-dimensional schedule deﬁned
above, this dependency is carried weakly by the polytope P1
corresponing to the ﬁrst dimension, since i≤i+jbut i≮i+j
for (i,j) ∈I. The dependency T →T, on the other hand, is
carried strongly by P1, since i + j < i+ (j + 1) for all
(i,j) ∈I. The polytope P2 for second dimension then carries
the dependency S → T strongly, since 0 < i+ 1 for all
(i,j) ∈I.
If we consider a polytope carrying all dimensions strongly,
we leave out some possible schedules. In some cases, the
polytope carrying all dimensions strongly might even be
empty, an example of this is given in [16]. Instead, we can
iteratively construct a multi-dimensional schedule by carrying
only some dimensions strongly and ensuring all the other
dependencies are carried weakly. Since we interpret schedules
in lexicographical order, once a dependency is carried strongly
by a dimension, all subsequent dimensions need not carry it
weakly anymore. If we iteratively add dependencies to dimen-
sions in this way, we will always ﬁnd a multi-dimensional
schedule for every SCoP [16]. However, the order in which
we let dimensions carry the dependencies strongly affects the
space of possible schedules. In Section IV we deﬁne a general
algorithm with free parameters to explore all possible such
constructions.
To ﬁnd proﬁtable schedules for a SCoP, we need to choose
a schedule after having constructed the space of all possible
(multi-dimensional) schedules. For this, we can independently
consider the polytopes Pd corresponding to each dimension
d of the schedule space. Points in this schedule space are
integer solutions to a set of inequalities, but equivalently, they
are points with integer values in a special linear combination
of terms:
p=
s∑
i
λivi,d +
t∑
i
αiri,d,λi,αi ∈R≥0,
k1∑
i
αi = 1. (2)
Equation 2 describes a convex combination of terms vi,d called
vertices, and a positive linear combination of terms ri,d called
rays, where the index dcorresponds to the dimension d. Every
such combination with integer values, i.e., where p∈Zn, is a
point in the lattice polytope of valid schedules. In other words,
Pd = Zn ∩{
∑
i
λivd,i +
∑
i
αird,i |λi,αi ≥0,
∑
i
λi = 1}.
This representation can be calculated with Chernikova’s al-
gorithm [26] and is equivalent to the constraint set repre-
sentation (as e.g. in Equation 1). We call the set Gd =
{vd,1,vd,1,...,r d,1,... }the set of generators of Pd. Choos-
ing a point pd ∈Pd in the corresponding polytope for every
dimension thus deﬁnes a multi-dimensional schedule.
As we have seen, a confusing aspect of the polyhedral model
is how there are several lattice polytopes associated to a SCoP
region: the iteration space, geometric interpretations of the
dependencies, and the spaces of valid schedules. Figure 4 gives
an overview of the model and the different lattice polytopes,
as well as the relationships between them, on the basis of the
matrix-vector multiplication example.
int i, j;
for (i = 0; i ¡ N; i++);
S: y[i] = 0;
for (i = 0; i ¡ N; i++)
for (j = 0; j ¡ N; j++)
T: y[i] += A[i][j]*x[j];
Dependencies
ij
ij
ij
i
Iteration space
i
j
Schedule space
−2
0
2
i
j
i
Schedules
i
j
i
Fig. 4. An overview of schedules in the polytope model
Optimizing a loop amounts to ﬁnding a good schedule for
the loop and a target architecture. Since the construction of
a multi-dimensional space of valid schedules is not unique,
this can be considered as two distinct steps: Constructing a
schedule space (of valid schedules) and exploring it to ﬁnd
a proﬁtable schedule. An important contribution of this paper
are MDP models for ﬁnding such a schedule based on these
two steps, as we describe in Section IV. In Section III we
describe how state-of-the-art methods go about ﬁnding them.
III. R ELATED WORK
There are two main families of methods for schedule opti-
mization in the polyhedral model: iterative methods that are
often guided by meta-heuristics and model-driven heuristics.
Iterative models evaluate multiple schedules iteratively by
generating code and benchmarking it, eventually returning the
best schedule [24]. The iterative exploration of such large
spaces is typically guided by meta heuristics such as genetic
algorithms, where the results of evaluations inﬂuence future
sampling points in the space [11], [12], [27]. Since these
methods require multiple executions of the program, they
are time-consuming in practical compilation ﬂows and need
additional benchmark tooling. In contrast to this, model-driven
heuristics like Pluto [10] and ISL [2] propose a reasonably
good solution directly without benchmarking. Using prede-
ﬁned assumptions about communication and locality, these
approaches formulate the optimization problem as an Integer
Linear Program. The assumptions require domain knowledge
that may not generalize to other architectures, and are usually
outperformed by their iterative counterparts. Learned heuris-
tics for schedule optimization have been investigated in the
scope of domain-speciﬁc languages (DSLs) that do loop-
heavy computations, like TensorComprehensions for machine
learning [4] or Halide [28] for image processing. A recent
extension to Halide [29] uses machine learning on a set of
program features to build a cost model. The authors use
the cost model to steer a beam search to ﬁnd proﬁtable
schedules iteratively without benchmarking the programs. In
similar fashion, TVM [30] uses a learned cost model to
estimate the performance of a candidate schedule in its form
of a program representation. TensorComprehensions, Halide
and TVM work on a similar space of loop transformation
primitives. Compared to the search space presented in this
work, their space does not include afﬁne transformations that
lead to potentially fast loop variants. These methods are also
restricted to programs expressed in the corresponding DSLs.
In contrast, this work presents a search space that is applicable
to general C code.
Recent work using Tiramisu [31] overcomes some of these
limitations by considering polyhedral optimizations as well.
In addition to that, they use machine learning to estimate
the execution time of a loop after the transformation, but
use an iterative monte-carlo tree search method to explore
the possible transformations. Our contribution is orthogonal to
their approach. Unlike our formulation, however, the search-
space in [31] includes incorrect transformations and, more
importantly, is speciﬁc to the loop being optimized. This
makes it more challenging to design and learn a policy that
generalizes across different loops.
RL has successfully been applied to certain subproblems of
compiler optimizations. NeuroVectorizer learns a model that
aims to predict a solution in the search space of vectorization
factors for a given loop [20]. Other work has focused on
maximizing speedup [22] and code size [21] on the search
space of analysis and transformation passes in LLVM, which
is enormously large. A smaller search space for code size
optimization is used by [23] to predict whether or not to inline
a function. In PolyGym, we construct a medium-sized search
space that contains many proﬁtable solutions, as we will show
in Section V.
More generally, machine learning has recently been applied
successfully to multiple tasks in compilers [32]–[35]. While
these works have advanced the models of code, they have
focused on rather coarse-grained tasks, like mapping OpenCL
kernels to a CPU or GPU. An important reason for this is that
formulating problems in compilers in an instance-independent
fashion is challenging. Finding optimal loop schedules is a
more subtle problem than, for instance, selecting a particular
GPU or CPU for the execution of a kernel. This paper
contributes to the ﬁeld of machine learning in compilers by
providing more challenging, yet lucrative task that can better
challenge the models of code.
IV. P OLYGYM’S MARKOV DECISION PROCESS
In this section we present our formulation of the problem
of ﬁnding a proﬁtable schedule as a markov decision process
(MDP). This enables us to explore the space of possible
schedules for a SCoP in the context of RL. In particular, the
actions in our formulation are independent of the particular
instance, i.e. the SCoP being optimized. If, instead, the actions
depended on the particular instance, then a different model
needs to be used for every instance. This property is thus a
necessary requirement for a heuristic to learn to generalize
over different instances.
As explained in Section II, the problem of ﬁnding a prof-
itable schedule can be divided in two stages: the construction
of the schedule space and the selection of a schedule. We
deﬁne two different MDPs, one for each subproblem. From the
point of view of the formulation, these are two distinct MDPs,
even if the space of the state space of the second MDP depends
on the options chosen in the ﬁrst. Thus, a single combined RL
agent can operate on the combined space, or alternatively, two
distinct agents can be trained to explore them.
A. Schedule space construction
We look for a schedule space in its general form, and
thus deﬁne it as a multi-dimensional space [11], [16]. As
illustrated in Algorithm 1, we construct a k-dimensional
schedule space iteratively, by going through the dimensions
of the space. For each dimension, we decide which depen-
dencies to include as strong dependencies, until we have
selected all dimensions. Crucially, the function that decides
this, select_dependency, is left as an unspeciﬁed, free
function. It then calculates the polytope of possible schedules
strongly satisfying these dependencies and weakly satisfying
the remaining unselected dependencies. Using Chernikova’s
algorithm [26] it calculates generators as vertices and rays
for this polytope. This algorithm goes back to the princi-
ples outlined by Feautrier [16], which is the same basis
for the heuristic in Section 3.2 in [11] and Algorithm 1
in [12]. Algorithm 1 generalizes these principles by leaving
the decision function select_dependency unspeciﬁed,
instead of proposing a concrete heuristic. Note that we write
select_dependency(d) to specify that this function de-
pends on the representation of the dependency d. In general,
this decision could also depend on other parameters like
properties of the SCoP.
Algorithm 1 General construction of the schedule space
input: A set D of dependencies
output: A schedule space S = (G1,G2,...,G k), where each
Gi corresponds to the set of generators of the lattice
polytope Pifor the i-th dimension.
1: i←1
2: while D̸= ∅do
3: Depsi ←∅
4: for d∈D do
5: if select_dependency(d) then
6: Depsi ←Depsi ∪{d}
7: Di ← strong_deps(Depsi) ∩weak_deps(D \
Depsi)
8: Gi ←chernikova(Di)
9: i←i+ 1
return (G1,...,G i−1)
This simple change, making select_dependency a free
function, has profound consequences. The greedy heuristics
of [11], [16] construct one deterministic schedule space, which
fundamentally limits the space of possible schedules that can
be found with the method. On the other hand, the randomized
heuristic of [12] could in principle produce any schedule space.
The statistical bias of the randomized algorithm, however,
implies that this is never the case in practice, by the law of
big numbers. While this is good for ﬁnding good heuristics in
most cases, it fundamentally limits the ceiling of possible im-
provement. By leaving select_dependency unspeciﬁed,
a model could learn a good heuristic, which can also leverage
properties of the concrete instance of the problem.
Based on Algorithm 1 we can deﬁne an MDP that constructs
this space. It considers the free select_dependency func-
tion as an action, which is combined with two additional
actions for controlling the iteration in from Algorithm 1. This
allows a walk through the MDP to steer the iteration through
the construction. The state space of the MDP is the countably
inﬁnite space
Scons = {(idim,idep,d1,...,d |D|)
|idim,idep,d1,...,d |d|∈N,idim >0}.
In this space, the ﬁrst component idim represents
the dimension, the second idep represents the
current dependency being selected, while the other
components represent the strong dependencies included
in that dimension. We deﬁne the set of actions as
Actcons = {next_dim,next_dep,select_dep}. The
transition probabilities Pcons : Scons ×Actcons ×Scons →[0,1]
are deﬁned as follows:
Pcons((idim,idep,d1,...,d |D|),next_dim,
(i′
dim + 1,i′
dep,d′
1,...,d ′
|D|))
= δidim,i′
dim
·δidep,i′
dep
·δd1,d′
1 ···δd|D|,d′
|D|
,
Pcons((idim,idep,d1,...,d |D|),next_dep,
(i′
dim,i′
dep + n,d′
1,...,d ′
|D|))
= δidim,i′
dim
·δidep,i′
dep
·δd1,d′
1 ···δd|D|,d′
|D|
,
Pcons((idim,idep,d1,...,d |D|),select_dep,
(i′
dim,i′
dep,d′
1,...,d ′
idep + idim,...,d ′
|D|))
= δidim,i′
dim
·δidep,i′
dep
·δd1,d′
1 ···δd|D|,d′
|D|
·δdidep ,0,
where δi,j = 0 if i ̸= j and 1 if i = j is the Kronecker
delta and n is a value that depends on the concrete state 3.
Concretely, we distingish between two cases. If there is a
k with k > idep such that dk = 0 , then we choose the
ﬁrst (minimal) such k and set n = k −idep. If there is
none, i.e. di > 0 for all i > idep, then we start back
from 0 and choose the smallest k with dk = 0 (without
any additional requirements on k). In this case we also set
n = k −idep. This means that, as their names suggest,
next_dim increases the dimension and select_dep adds
the current dependency to the set of strong dependencies if it
was not previously added. The action next_dep increases
the current available dependency, skipping those that have
already been selected. Accepting states are all state of the form
3We omit the indices indicating this dependency for readability.
(1, 1, 0, 0)start (2, 1, 0, 0)
(2, 2, 0, 0) (3, 2, 0, 0)
(3, 1, 0, 0)
(3, 2, 0, 3) (4, 2, 0, 3)
(4, 1, 0, 3)
(4, 1, 4, 3)next_dim
next_dep
next_dim
next_depnext_dep
select_dep
next_dim
next_depnext_dep
select_dep
(1, 1, 0, 1)
select_dep
(1, 2, 0, 0)
next_dep
. . .
. . .
(2, 2, 0, 2)
select_dep
(2, 1, 0, 2)
next_dep
. . .
. . .
. . . . . .
. . . . . .
Fig. 5. An example of the schedule space construction MDP and a concrete
action sequence for the matrix-vector multiplication example.
(i,j,d 1,...,d |D|),i,j,d 1,...,d |D|>0, where all dependen-
cies have been chosen to be strongly carried. Finally, the initial
state is (1,1,0,..., 0) ∈Scons, starting with no dependencies
selected. We will discuss the rewards later, in Section IV-C.
Note that while the state space depends on the concrete
instance, i.e. the SCoP being considered, the set of actions
does not. This is important since it allows us to learn a policy
to navigate these spaces in a fashion that is independent of the
problem instance.
Consider the example in Figure 5. It shows the state space
for the schedule space construction of the example of the
matrix-vector multiplication from Section II. The ﬁgure also
shows an example run through the state space, corresponding
to the sequence of actions next_dim, next_dep,
next_dim, next_dep, next_dep, select_dep,
next_dim, next_dep, select_dep. The ﬁrst action,
next_dim, increases the dimension, which is the ﬁrst
component (1,... ) →(2,... ). Then, the next_dep action
changes the second index, indicating the current dependency:
(2,1,0,0) → (2,2,0,0). Since no dependency has been
selected so far, the value of j is 2, corresponding to the the
ﬁrst case outlined above, yielding n= 1. After increasing the
dimension with a next_dim action, the next next_dep
action again changes the dependency. This time there is no
dependency with index > 2, and thus j = 1, by the second
case, corresponding to n= −1. A second next_dep returns
the second index, marking the dependency, to 2, returning
to the state (3,2,0,0). At this point the select_dep adds
an index to the current dependency ( 2), marking the current
dimension to carry this dependency strongly: (3,2,0,3). The
dependencies S →T and T →T are sorted, in that order.
This process continues until it reaches the state (4,1,4,3).
This corresponds to a 4-dimensional space, as indicated by
the ﬁrst component. As indicated by the third component 4,
the third dimension strongly carries the ﬁrst dependency,
S →T, and the fourth dimension strongly carries the second
dependency T → T, as indicated by the last component.
In particular, the accepting state itself uniquely deﬁnes the
schedule space, as the order in which the dependencies are
selected within a dimension is irrelevant to the construction,
it only matters which dimensions speciﬁcally carry the
dependencies strongly.
B. Schedule space exploration
Once the schedule space is generated, we look into how to
ﬁnd proﬁtable schedules in this space. From Algorithm 1, the
schedule space is given by a vector S = ( G1,G2,...,G k),
where each Gd corresponds to the generators of a lattice
polytope for the d-th dimension. A valid schedule consists
precisely of a point in each of the lattice polytopes corre-
sponding to the dimensions. To construct each of these points
we need to consider the generators Gd. These are given as a
set of vertices v1,...,v s and rays r1,...,r t as explained in
Section II. An arbitrary point p is in the polytope iff it can
be written as a convex combination of vertices and a positive
linear combination of rays, i.e.
p=
s∑
i=1
λivi +
t∑
i=1
αiri,
where λi ≥0 for all i= 1,...,s and ∑s
i=1 λi = 1 and αi ≥0
for all i= 1,...,t . Note that some work uses a third generator
type, lines. This can always be converted to an equivalent
set of generators as we deﬁne here. We choose to have only
rays instead of rays and lines as it is more uniform this way,
making this representation more ameanable for RL. Schedules
correspond to the points in the lattice polytope, which is a
subset of this general polytope. This means that phas to have
integer coefﬁcients.
To formulate this as an MDP, we introduce Algorithm 2
which generates points in the polytopes following the
same principles as [11], [12], [15], [16], [24], [27]. Once
again, our algorithm leaves an unspeciﬁed, free function
select_coeff. The goal of this function is to ﬁnd the
values of the coefﬁcients λi,i = 1,...,s and αi,i = 1,...,t .
We do this by iterating over all vertices and rays and selecting
a coefﬁcient in this iteration. Like the authors in [12], we use a
correction step, multiplying by the least common denominator
(LCD) in the end to ensure the point is in the lattice polytope,
i.e. has integer coefﬁcients. While this can be avoided through
the design of the select_coeff function, this correction
step can simplify the design of the function.
4Since the ﬁrst two indicate the current dimension and dependency.
Algorithm 2 General exploration of the schedule space
input: A schedule space S = (G1,G2,...,G k) as a vector,
where each Gi corresponds to the generators of a lattice
polytope for the i-th dimension.
output: A point in the schedule polytope for each dimension
p= (p1,...,p k)
1: for i∈{1,...,k }do
2: v1,...,v s,r1,...,r t ←Gi
3: pi ←0
4: for x∈{v1,...,v s,r1,...,r t}do
5: pi ←pi + select_coeff(x) ·x
6: if pi ∈Qn \Zn then
7: pi ←LCD(pi) ·pi
return (p1,...,p k)
To deﬁne a corresponding MDP, we start by deﬁning the
state space. For an integer N >0, we deﬁne the space as the
ﬁnite set 5 of the coefﬁcients
Sexpl = {(λ1,1,...,λ 1,s1 ,α1,1,α1,t1 ,...,λ k,1,...,α k,tk)
|λi,j,αi,j ∈{0,...,N, ⊥}for all i,j}
Note that since we have multiple polytopes P1,...,P k, cor-
responding to the multiple dimensions, we use two indices
for the generators, where the ﬁrst index d corresponds to the
polytope Pd and the second index iterates over the generators
in Gd. This can also be conceptually understood as unrolling
the two loops in Algorithm 2 in the state space. We use the
symbol ⊥to mark coefﬁcients that have not been selected,
as this is distinct from selecting 0 as a coefﬁcient. Choosing
the coefﬁcients λi ∈{0,...,N }means that in most cases,∑s
i=1 λi >1. In that case we norm the coefﬁcients by building
the convex combination as ∑s
i=1 λivi/∑s
i=1 λi. Additionally,
when s= 1 it is clear that λ1 = 1, since for a single point there
is only one possible convex combination. We thus remove the
corresponding coefﬁcients entirely from the state space 6.
Since we need to choose a coefﬁcient for each term,
we do not include actions to steer the exploration as we
did for the schedule space construction. Thus, the action
space corresponds directly to the function select_coeff.
We deﬁne actions select_coeff0, . . . ,select_coeffN
accordingly. We set
Pexpl((a1,...,...,a i,⊥,..., ⊥),select_coeffX,
(a1,...,...,a i,X, ⊥,..., ⊥)) = 1,
and for all other states s,s′∈Sexpl and actions a∈Actexpl,
we set Pexpl(s,a,s ′) = 0 . The initial state corresponds
the starting conﬁguration with no coefﬁcients deﬁned, i.e.,
(⊥,..., ⊥).
Similar to the space construction, a crucial aspect of this
formulation is that the actions are independent of the SCoP.
5This is thus parameterized by N. Parameters like these are sometimes also
called hyper-parameters, especially in the context of machine learning.
6This was the case for almost all examples we evaluated in this paper.
(⊥, ⊥, . . . ,⊥)
start
(0, ⊥, ⊥, . . . ,⊥)(1, ⊥, ⊥, . . . ,⊥) (2, ⊥, ⊥, . . . ,⊥)
select_coeff0
select_coeff1
select_coeff2
. . .
select_coeff0
(. . . ,0, ⊥, ⊥, ⊥, ⊥, . . .)(. . . ,1, ⊥, ⊥, ⊥, . . .) (. . . ,2, ⊥, ⊥, ⊥, . . .)
select_coeff0
select_coeff1
select_coeff2
(. . . ,0, 0, ⊥, ⊥, . . . ,⊥)(. . . ,0, 1, ⊥, ⊥, . . . ,⊥) (. . . ,0, 2, ⊥, . . . ,⊥)
select_coeff0
select_coeff1
select_coeff2
(. . . ,0, 2, 0, ⊥, . . .)(. . . ,0, 2, 1, ⊥, . . .) (. . . ,0, 2, 2, ⊥, . . .)
select_coeff0
select_coeff1
select_coeff2
······ ···
Fig. 6. An example of the schedule space exploration MDP and a concrete
action sequence for the matrix-vector multiplication example.
This way, a heuristic model can learn to ﬁnd good schedules
by selecting the correct coefﬁcients.
Consider again the example of the matrix-vector multiplica-
tion kernel. With the schedule space constructed as depicted
in Figure 5, the schedule space has four dimensions, where
the third carries the dependency S → T strongly and the
fourth one T → T. This yields four polytopes, one for
each dimension, which after using Chernikova’s algorithm can
be generated each by a single vertex and 10,10,10,13 rays
respectively. These all live in a 7 dimensional vector space.
Note that neither the 4 dimensions of the schedule space nor
the 7 dimensions of the polytopes for each schedule space
dimension have a direct interpretation in terms of the loop
bounds. They represent loop schedules according to Farkas’
lemma (cf. Section II) and they are not intuitively easy to
understand in terms of the loops’ abstract syntax tree (AST).
As mentioned above, the coefﬁcient selection for the vertex is
not part of the coefﬁcient selection in the state space of the
MDP, since it is a single vertex in all four dimensions.
Figure 6 shows the beginning of a sample run through this
space, given by the following action sequence (omitting the
select_coeff part of the name): 0, 0, 0, 0, 0, 2, 2, 1, 0,
2, 0, 0, 0, 0, 2, 0, 0, 1, 0, 2, 0, 0, 0, 1, 0, 2, 0, 0, 0, 0, 0,
0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0. The coefﬁcients of the rays
are selected dimension by dimension: α1,1 = 0 ,...,α 1,6 =
2,α1,7 = 2,α1,8 = 1,...,α 1,10 = 2,α2,1 = 0,...,α 4,13 = 0.
After selecting all coefﬁciens with that sequence, the resulting
points are given by:
p1 = (0 ,0,0,0,0,0,0) + (0 ,0,0,0,0,−1,0)
+ 2 ·(0,0,0,0,0 −1,−1) + 2 ·(0,0,0,−1,−1,0,0)
+ (−1,−1,0,0,0,0,0) = ( −1,−1,0,−2,−2,−3,−1)
...
p4 = (0 ,0,1,0,0,0,0) + 2 ·(0,0,1,0,0,0,0)
= (0,0,3,0,0,0,0).
We see how this MDP can produce concrete schedules.
Finally, this points can be translated into a valid transformation
(i,j) →(k,l) as explained in Section II. This translates to the
following transformed C kernel:
if (N >= 1)
for (int k = -N; k <= 0; k += 1) {
if (N + k >= 1)
for (int l = 0; l < N; l += 1)
T: y[-k] += y[-k][l] *x[l];
if (k <= -1)
S: y[-k - 1] = 0;}
Note again that the correspondence between the representa-
tion as points in the schedule space and the transformed kernel
is not directly visible. The dimensions of a multidimensional
schedule deﬁne how the individual points in the schedule are
ordered (lexicographically), it does not directly translate e.g.
to loop nesting.
This concrete example was in fact found by a simple
heuristic, with a bias towards deﬁning coefﬁcients to be 0.
We describe the heuristic in Section V.
C. Rewards
The rewards of an MDP deﬁne a feedback loop – the quality
of an action taken in a given state. This is needed for a learning
algorithm so it can decide about what action is favorable to
take in a given state and adjust the agent’s model in case the
action was infavorable. Reinforcement Learning can then be
used to train an agent that navigates an MDP. Recent work has
shown that it is possible to use Deep Neural Networks as agent
models that are able to generalize over the large search space
and able to learn to make good decisions on in previously
unseen states [13], [14].
Figure 7 shows the ﬂow with two MDP models in PolyGym
in a RL context. Note that PolyGym does not deﬁne an agent,
it only deﬁnes the environment to learn with such an agent.
The two separate MDPs produce a single schedule, which is
then compiled with LLVM/Poly and executed, to determine the
reward. There is no immediate reward to the construction of
the schedule space, nor to the partial states in the exploration
step, since these do not deﬁne an actual schedule. As such, we
deﬁne the rewards for both individual MDPs uniformly, based
on the ﬁnal constructed schedule, as follows. If the action leads
to a complete schedule, which is a terminal state, then we give
the speedup over a traditional optimizing compiler, e.g., using
LLVM with the -O3 ﬂag, as a sample-independent metric as
reward. If it leads to an incomplete schedule, we give 0 as
PolyGym
Clang
LLVMPolly
Schedule Space
Construction
MDP
Schedule Space
Exploration
MDP
Polly
LLVM
Execu-
tion
SourceCode SCoP
Generators
Schedule
Execu-table
Agent Reward
Actions
Fig. 7. The PolyGym ﬂow
reward. Finally, if it leads to any invalid state, we give negative
reward. Invalid states can occur when constructing an empty
polytope in the MDP of the schedule space construction. They
can also occur in the exploration MDP, when there are multiple
vertices in a polytope and all vertex coefﬁcients are selected
to be 0 (leading to no convex combination).
In this way, (positive) rewards are only given for the ﬁnal
construction which obtains a valid schedule. This constitutes
what is called a sparse reward setting. For example, the
schedule for the matrix-vector multiplication kernel 7 found
by the examples described in Figures 5 and 6, the resulting
kernel runs an impressive 140×faster than the original kernel
compiled with -O38. This level of improvement is thus not
easy to obtain in more complex kernels and larger input
sizes. Thus, the ﬁnal action select_coeff0 (not shown
in Figure 6 for space reasons) has a reward of 140, while all
other actions have a reward of 0. State-of-the-art methods in
RL trickle down this reward to the other actions [13], [14].
Importantly, this reward should also be trickled down to the
schedule space construction in a learning setting, even if the
MDP is conceptually a different one.
D. Limitations
While our formulation of the schedule search problem
as an MDP with an instance-independent action set enables
reinforcement learning for this problem, it still has some
limitations.
An important limitation is that some accepting states do not
yield schedules, as discussed previously. There are two posso-
bilities of this happening. The ﬁrst corresponds to the schedule
space construction. Not every schedule space constructed with
Algorithm 1 actually has schedules. If too many dependencies
are carried strongly in a dimension, the intersection of the
corresponding polytopes can be empty, leading to a space
without any valid schedules. The randomized construction
7For the vector dimension N = 2000, with data generated procedurally
using the Polybench tools.
8The full setup is described in Section V.
of [12], similarly, does not provide such a (formal) guarantee.
In practice, however, biasing the construction towards many
dimensions, as they do, yields non-empty spaces. We are
conﬁdent this fact can easily be learned by pertinent methods.
The second possibility has to do with the vertices in a
generating set for a polytope. If all vertices have a coefﬁcient
of 0, Algorithm 2 does not produce a schedule, since the point
constructed is not in the lattice polytope. The authors in [12]
choose the vertex at random. We cannot do the same, as it
breaks the MDP abstraction, where the reward of a state is
non-deterministic.9 We did not run into this problem in any
example considered in this paper. Having more than one vertex
seems to be an extremely rare occurrence in practice. This is
both a beneﬁt and a problem: While we mostly do not have to
deal with this problem, if at some point we ever do, we will
probably not have enough samples to learn to deal with it.
A ﬁnal limitation corresponds to the exploration phase.
There are two ways in which we exclude some points. By
choosing a ﬁnite N as a hyper-parameter, we technically
limit the possible coefﬁcients of the schedule space. This is
necessary, however, since otherwise there is an inﬁnite amount
of possible schedule polytopes 10. Similarly, the fact that we
use integer coefﬁcients and calculate the LCD in Algorithm 2
might exclude some points that would otherwise be found
with rational coefﬁcients. With an unbounded N this problem
would not exist. Consequently, a larger N mitigates it.
While the limitations discussed here technically exclude
some solutions or allow algorithms 1 and 2 to fail, they do
so only in rare corner-cases. Moreover, our reward design and
the hyper-parameter N allow us to avoid or at least mitigate
these limitations in those rare corner-cases.
V. E VALUATION
In this section we evaluate the performance potential of the
search space of our MDP formulation by exploring it with
simple heuristics. We compare the best schedules we ﬁnd in
a ﬁxed number of explorations with the ones from the state-
of-the-art heuristic implemented in ISL and Polly in recent
versions. In the following, we describe the experimental setup
we used and discuss the results obtained.
A. Experimental setup
We implemented the MDPs of the schedule space as an
OpenAI Gym Environment, so it is conveniently usable for
future machine learning heuristics [36]. The algorithms are
implemented in Python, relying on the ISL binding islpy 11 for
operations on integer sets. To transform the polytopes into the
generator representation, we used Polyite’s implementation of
the Chernikova algorithm12. Finally, we use LLVM and Polly
9Since the execution times of the SCoPs are non-deterministic as well, this
is the case for all rewards. However, if the statistical variance of execution
times is not negligible, then the whole problem of selecting an optimal
schedule is ill-posed in the ﬁrst place. This is not the case in practice.
10Many of these are equivalent, since the number of orderings is ﬁnite
11https://github.com/inducer/islpy
12https://github.com/stganser/chernikova
seidel-2d syr2k syrk trisolv trmm aggregated
gesummv gramschmidt jacobi-1d jacobi-2d lu mvt nussinov
deriche doitgen durbin fdtd-2d ﬂoyd-warshall gemm gemver
2mm 3mm atax bicg cholesky correlation covariance
2-10
2-5
20
25
2-10
2-5
20
25
2-10
2-5
20
25
2-10
2-5
20
25
speedup over O3-clang12 (log)
ISL-clang3.9
ISL-clang12
O3-clang3.9
Fig. 8. Violin plot of the distribution of speedups by benchmark, with speedups as reference lines (geometric mean, 4 executions).
in version 3.9 to extract polyhedral representations of SCoPs
and to transform them according to the computed schedules.
All schedules were evaluated on the same system, consisting
of an AMD 3960X with 24 cores and 48 threads, organized in
8 units, each with 3 cores. The units have a dedicated 16 MB
L3 cache. Each core has a dedicated 32 KB L1 and a 512 KB
L2 cache. The benchmarks are executed in a single unit, that
is, using at most 3 cores (and 6 threads). We do so because
of the low interference between units, which allows us to run
up to four experiments in parallel without adding too much
noise in the observations. The conﬁguration of four units has
been identiﬁed to have necligible side-effects in preliminary
experiments.
For our evaluation we use the Polybench benchmark
suite [17], which consists of 30 numerical computations from
various domains, such as image processing, statistics, and
linear algebra. Each of the benchmarks contains a SCoP
kernel, i.e. a loop nest that satisﬁes the conditions of the
polyhedral model. We had to exclude the ludcmp, heat-3d,
and adi kernels because of their large number of dependence
polytopes. The sheer size of these polytopes signiﬁcantly slows
down the runtime of the Chernikova algorithm and thus the
construction of the schedule space. Since programs generated
by unfavorable schedules could potentially run for a long time,
we set a timeout that is proportional to the LLVM O3 runtime
by a factor of 10 when executing.
We compare the schedules found by PolyGym against
standard optimizing compilers and against ISL. As standard
optimizing compiler we use LLVM in version 12 with
the -O3 ﬂag (LLVM O3). The ISL experiments were
run with ﬂags -polly -polly-parallel=true
-polly-vectorizer=none -polly-tiling=true
-polly-default-tile-size=64. In the experiments,
we used 4 measurements and report the minimum to eliminate
measurement inaccuracies.
B. Result analysis
To analyze how many proﬁtable schedules the space
spawned by our MDP formulation includes, we generate 1000
schedules per benchmark kernel with the a simple bias the ex-
ploration of the schedule space towards the select_coeff0
2mm3mmataxbicg
choleskycorrelationcovariance
derichedoitgendurbinfdtd-2d
ﬂoyd-warshall
gemmgemvergesummv
gramschmidt
jacobi-1djacobi-2d
lu mvt
nussinovseidel-2d
syr2ksyrktrisolvtrmm
geo
mean
1
2
4
8
16
32
speedup over O3-clang12 (log)
ISL-clang3.9ISL-clang12PolyGym-biascoeff0
Fig. 9. Speedups over Clang O3 of individual kernels of Polybench. PolyGym was conﬁgured with the select coeff 0 heuristic and 1000 iterations.
action. This bias results in overall less complex schedules.
Figure 8 shows the distributions of speedups by individual
samples, along with the performance of ISL and LLVM
O3. We can observe that the search space contains many
proﬁtable schedules. Many benchmarks, like 2mm,doitgen,
gemm,jacobi-2d or seidel-2d, show a sigiﬁcant distri-
bution of points better than those found by ISL. This suggests
that an agent could learn to achieve this better performance
results without iteratively executing the kernel.
Figure 9 shows the maximum measured speedups for in-
dividual kernels of the polybench suite. We ﬁnd schedules
with an overall speedup of 3.39x over O3-clang12, which is
1.83x faster than the ones of ISL-clang12 and 1.67x faster than
ISL-clang3.9. For 20 of the 26 kernels, the heuristic iteratively
ﬁnds more proﬁtable schedules than ISL-clang12, for 22 of 26
for ISL-clang3.9. The results are not directly comparable to
Polyite [12], [27], because they we use a different hardware
system. However, the overall results seem to be comparable
in terms of the improvement, which is not surprising, since
we use a similar search space and a similar random sampling
process. Compared to Polyite, our MDP formulation is shape-
agnostic. This can enable an agent to learn to navigate this
space without requiring an iterative execution.
We further analyze the inﬂuence that a potential heuristic
has on the sampling process. We employ different trivial
heuristics for demonstration.
• In bias select dep, we bias the schedule space con-
struction phase towards the select_dep action, which
results in less schedule dimensions.
• In bias coeff 0, we bias towards the select_coeff0
action in the schedule space exploration phase.
• In uniform, we select actions uniformly at random.
Figure 10 shows the performance of the PolyGym search
space with different heuristics on the Polybench suite. The plot
shows the geometric mean across the different benchmarks as
the number of schedules sampled increases. We see that differ-
1 10 100 1000
0
1
2
3
ISL-clang3.9ISL-clang12
number of schedules (log)
speedup over O3-clang12 PolyGym-biascoeff0PolyGym-biasselectdepPolyGym-uniform
Fig. 10. Maximum speedup over LLVM O3 achieved by the different
heuristics for action selection with increasing number of explored schedules.
The plot depicts the geometric mean over all selected benchmarks of the
Polybench suite. As reference, the aggregated speedups of ISL over LLVM
O3 in different version (ISL-clang3.9, ISL-clang12) is also included.
ent heuristics impact the space exploration achieving different
performances overall. All evaluated heuristics outperform the
schedules found by ISL-clang3.9 and ISL-clang12 in less
than 40 sampling iterations. Most notably, when biasing the
selection towards the coefﬁcient 0 ( bias coeff 0), the cross-
over is at 11 iterations for ISL-clang12 and after 13 iterations
for ISL-clang3.9. Crucially, this shows that there is potential
to learn to explore this space efﬁciently. All of the evaluated
heuristics, however, are very simple and make not use of the
MDP’s states. In future work, this heuristic can be a Deep
Neural Network that is optimized to take the best actions using
Reinforcement Learning algorithms.
VI. C ONCLUSIONS
In this paper we presented PolyGym – an environment for
polyhedral schedule optimizations for reinforcement learning
(RL), based on a generic, SCoP-independent markov deci-
sion process (MDP). With this formalization, it is possible
to develop heuristic with RL algorithms that learn models
that produce proﬁtable schedules directly, without iterative
benchmarking. Using the PolyBench suite, we have shown
that the search space offers the potential to learn to produce
results that signiﬁcantly outperform state-of-the-art heuristics.
PolyGym is usable as an environment conforming to the Gym
interface, which is widely-used by RL algorithms, enabling its
integration with minimal effort. Therefore, PolyGym allows
to investigate learning models in this task without diving into
the theory of the polyhedral models, which is a high barrier.
We believe that this will expose polyhedral optimizations as a
learning problem, advancing the ﬁeld of machine learning for
compilers.
In future work, we plan to investigate models and RL
algorithms to learn proﬁtable schedules. We also plan to inte-
grate reward functions for further objectives, such as energy
consumption, into PolyGym.
ACKNOWLEDGMENTS
This work was funded in part by the German Federal
Ministry of Education and Research (BMBF) within the
project ScaDS AI (BMBF 01IS18026B), the German Research
Council (DFG) through the TraceSymm project (number
366764507) and the Studienstiftung des deutschen V olkes.
REFERENCES
[1] S. Pop, A. Cohen, C. Bastoul, S. Girbal, G.-A. Silber, and N. Vasilache,
“Graphite: Polyhedral analyses and optimizations for gcc,” in Proceed-
ings of the 2006 GCC Developers Summit , 2006, p. 2006.
[2] S. Verdoolaege, “isl: An integer set library for the polyhedral model,”
in International Congress on Mathematical Software . Springer, 2010,
pp. 299–302.
[3] T. Grosser, H. Zheng, R. Aloor, A. Simb ¨urger, A. Gr ¨oßlinger, and L.-
N. Pouchet, “Polly-polyhedral optimization in llvm,” in Proceedings of
the First International Workshop on Polyhedral Compilation Techniques
(IMPACT), vol. 2011, 2011, p. 1.
[4] N. Vasilache, O. Zinenko, T. Theodoridis, P. Goyal, Z. DeVito, W. S.
Moses, S. Verdoolaege, A. Adams, and A. Cohen, “Tensor comprehen-
sions: Framework-agnostic high-performance machine learning abstrac-
tions,” arXiv preprint arXiv:1802.04730 , 2018.
[5] R. Baghdadi, J. Ray, M. B. Romdhane, E. D. Sozzo, A. Akkas, Y . Zhang,
P. Suriana, S. Kamil, and S. Amarasinghe, “Tiramisu: A polyhedral
compiler for expressing fast and portable code,” in 2019 IEEE/ACM
International Symposium on Code Generation and Optimization (CGO) ,
2019, pp. 193–205.
[6] R. Gareev, T. Grosser, and M. Kruse, “High-performance generalized
tensor operations: A compiler-oriented approach,” ACM Transactions on
Architecture and Code Optimization (TACO) , vol. 15, no. 3, pp. 1–27,
2018.
[7] S. Kobeissi, A. Ketterlin, and P. Clauss, “Rec2poly: Converting re-
cursions to polyhedral optimized loops using an inspector-executor
strategy,” in International Conference on Embedded Computer Systems .
Springer, 2020, pp. 96–109.
[8] K. Vadivel, L. Chelini, A. BanaGozar, G. Singh, S. Corda, R. Jordans,
and H. Corporaal, “Tdo-cim: Transparent detection and ofﬂoading for
computation in-memory,” in 2020 Design, Automation Test in Europe
Conference Exhibition (DATE), 2020, pp. 1602–1605.
[9] A. A. Khan, H. Mewes, T. Grosser, T. Hoeﬂer, and J. Castrillon,
“Polyhedral compilation for racetrack memories,” IEEE Transactions
on Computer-Aided Design of Integrated Circuits and Systems (TCAD).
Special issue on Compilers, Architecture, and Synthesis of Embedded
Systems (CASES’20) , vol. 39, no. 11, pp. 3968–3980, Oct. 2020.
[Online]. Available: https://ieeexplore.ieee.org/document/9216560
[10] U. Bondhugula and J. Ramanujam, “Pluto: A practical and fully auto-
matic polyhedral parallelizer and locality optimizer,” 2007.
[11] L.-N. Pouchet, C. Bastoul, A. Cohen, and J. Cavazos, “Iterative opti-
mization in the polyhedral model: Part ii, multidimensional time,” ACM
SIGPLAN Notices, vol. 43, no. 6, pp. 90–100, 2008.
[12] S. Ganser, A. Gr ¨osslinger, N. Siegmund, S. Apel, and C. Lengauer,
“Iterative schedule optimization for parallelization in the polyhedron
model,” ACM Transactions on Architecture and Code Optimization
(TACO), vol. 14, no. 3, pp. 1–26, 2017.
[13] V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G.
Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski
et al. , “Human-level control through deep reinforcement learning,”
nature, vol. 518, no. 7540, pp. 529–533, 2015.
[14] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van
Den Driessche, J. Schrittwieser, I. Antonoglou, V . Panneershelvam,
M. Lanctot et al., “Mastering the game of go with deep neural networks
and tree search,” nature, vol. 529, no. 7587, pp. 484–489, 2016.
[15] P. Feautrier, “Some efﬁcient solutions to the afﬁne scheduling problem.
i. one-dimensional time,”International journal of parallel programming,
vol. 21, no. 5, pp. 313–347, 1992.
[16] ——, “Some efﬁcient solutions to the afﬁne scheduling problem. part ii.
multidimensional time,” International journal of parallel programming ,
vol. 21, no. 6, pp. 389–420, 1992.
[17] L.-N. Pouchet et al. (2012) Polybench: The polyhedral benchmark
suite. [Online]. Available: http://web.cs.ucla.edu/ ∼pouchet/software/
polybench/
[18] Z. Wang and M. O’Boyle, “Machine learning in compiler optimization,”
Proceedings of the IEEE , vol. 106, no. 11, pp. 1879–1901, 2018.
[19] H. Leather and C. Cummins, “Machine learning in compilers: Past,
present and future,” in 2020 Forum for Speciﬁcation and Design Lan-
guages (FDL). IEEE, 2020, pp. 1–8.
[20] A. Haj-Ali, N. K. Ahmed, T. Willke, Y . S. Shao, K. Asanovic, and
I. Stoica, “Neurovectorizer: End-to-end vectorization with deep rein-
forcement learning,” inProceedings of the 18th ACM/IEEE International
Symposium on Code Generation and Optimization , 2020, pp. 242–255.
[21] C. Cummins, H. Leather, B. Steiner, H. He, and S. Chintala, “Compil-
erGym: A reinforcement learning toolkit for compilers,” https://github.
com/facebookresearch/CompilerGym/, 2020.
[22] R. Mammadli, A. Jannesari, and F. Wolf, “Static neural compiler
optimization via deep reinforcement learning,” in 2020 IEEE/ACM 6th
Workshop on the LLVM Compiler Infrastructure in HPC (LLVM-HPC)
and Workshop on Hierarchical Parallelism for Exascale Computing
(HiPar). IEEE, 2020, pp. 1–11.
[23] M. Troﬁn, Y . Qian, E. Brevdo, Z. Lin, K. Choromanski, and D. Li,
“Mlgo: a machine learning guided compiler optimizations framework,”
arXiv preprint arXiv:2101.04808 , 2021.
[24] L.-N. Pouchet, C. Bastoul, A. Cohen, and N. Vasilache, “Iterative
optimization in the polyhedral model: Part i, one-dimensional time,”
in International Symposium on Code Generation and Optimization
(CGO’07). IEEE, 2007, pp. 144–156.
[25] A. Schrijver, Theory of linear and integer programming . John Wiley
& Sons, 1998.
[26] H. Le Verge, “A Note on Chernikova’s algorithm,”
INRIA, Research Report RR-1662, 1992. [Online]. Available:
https://hal.inria.fr/inria-00074895
[27] S. Ganser, A. Gr ¨oßlinger, N. Siegmund, S. Apel, and C. Lengauer,
“Speeding up iterative polyhedral schedule optimization with surrogate
performance models,” ACM Transactions on Architecture and Code
Optimization (TACO), vol. 15, no. 4, pp. 1–27, 2018.
[28] J. Ragan-Kelley, C. Barnes, A. Adams, S. Paris, F. Durand, and S. Ama-
rasinghe, “Halide: a language and compiler for optimizing parallelism,
locality, and recomputation in image processing pipelines,” Acm Sigplan
Notices, vol. 48, no. 6, pp. 519–530, 2013.
[29] A. Adams, K. Ma, L. Anderson, R. Baghdadi, T.-M. Li, M. Gharbi,
B. Steiner, S. Johnson, K. Fatahalian, F. Durand et al., “Learning to op-
timize halide with tree search and random programs,” ACM Transactions
on Graphics (TOG) , vol. 38, no. 4, pp. 1–12, 2019.
[30] T. Chen, T. Moreau, Z. Jiang, L. Zheng, E. Yan, H. Shen, M. Cowan,
L. Wang, Y . Hu, L. Ceze et al. , “{TVM}: An automated end-to-end
optimizing compiler for deep learning,” in 13th {USENIX}Symposium
on Operating Systems Design and Implementation ( {OSDI}18), 2018,
pp. 578–594.
[31] R. Baghdadi, M. Merouani, M.-H. Leghettas, K. Abdous, T. Arbaoui,
K. Benatchba et al. , “A deep learning based cost model for automatic
code optimization,” Proceedings of Machine Learning and Systems ,
vol. 3, 2021.
[32] C. Cummins, P. Petoumenos, Z. Wang, and H. Leather, “End-to-end deep
learning of optimization heuristics,” in 2017 26th International Con-
ference on Parallel Architectures and Compilation Techniques (PACT) .
IEEE, 2017, pp. 219–232.
[33] A. Brauckmann, A. Goens, S. Ertel, and J. Castrillon, “Compiler-
based graph representations for deep learning models of code,” in
Proceedings of the 29th ACM SIGPLAN International Conference on
Compiler Construction (CC 2020) , ser. CC 2020. New York, NY ,
USA: Association for Computing Machinery, Feb. 2020, p. 201–211.
[Online]. Available: https://doi.org/10.1145/3377555.3377894
[34] G. Ye, Z. Tang, H. Wang, D. Fang, J. Fang, S. Huang, and Z. Wang,
“Deep program structure modeling through multi-relational graph-based
learning,” in Proceedings of the ACM International Conference on
Parallel Architectures and Compilation Techniques, 2020, pp. 111–123.
[35] C. Cummins, Z. V . Fisches, T. Ben-Nun, T. Hoeﬂer, and H. Leather,
“Programl: Graph-based deep learning for program optimization and
analysis,” arXiv preprint arXiv:2003.10536 , 2020.
[36] G. Brockman, V . Cheung, L. Pettersson, J. Schneider, J. Schul-
man, J. Tang, and W. Zaremba, “Openai gym,” arXiv preprint
arXiv:1606.01540, 2016.

Reinforcement Learning Guided Software Debloating∗
Nham Le Van Ashish Gehani Arie Gurﬁnkel Susmit Jha Jorge A. Navas
Abstract
Modern software engineering practices encourage software bloat. Libraries provide
more APIs than any one client needs; programs have many command line and
conﬁguration options, with most never used by the user and remaining set to their
default values. The result is software systems that are bloated, slow, and, most
importantly, vulnerable to attacks. Software debloating is an emerging technique
to address this problem by automatically trimming (or specializing) a program to
a user-deﬁned execution environment (e.g., command line options, conﬁguration
ﬁles, common usage scenarios, etc.). Since deciding on what to specialize is
computationally expensive, current state-of-the-art software debloaters rely on
hand-crafted heuristics. This is problematic because these heuristics are not always
effective, and, more importantly, difﬁcult to adjust to different debloating metrics.
In this paper, we propose a reinforcement learning based approach to automatically
learn a good debloating policy. We have implemented the approach as a tool called
DEEP OCCAM , and evaluated it on debloating several C programs. Our results show
that the technique outperforms the base-line policy of a state-of-the-art debloater
OCCAM , while being easily adaptable to a variety of debloating metrics.
1 Introduction
The rapid increase of software productivity in the last decades was fueled by the extensive use of
abstraction and reuse of software components. While this enabled building larger and more complex
software systems, these gains came at the expense of less efﬁcient and less secure software systems,
and contribute to a troublesome trade-off between productivity and performance/security. When an
application built using general-purpose components is deployed, the majority of the general-purpose
functionality is never used. This creates two problems: ﬁrst, as the use of abstraction layers makes
it more difﬁcult to optimize the software, this has a detrimental impact on performance; second, it
increases the attack surface for security vulnerabilities.
An emerging solution to this problem is a set of tools, called Software Debloaters, that automatically
customize a program to a user-speciﬁed environment. One successful approach for software debloat-
ing is based on partial evaluation (PE) [12] in which a partial evaluator takes a program and some of
its input values and produces a residual (or specialized) program in which those inputs are replaced
with their values. While PE has been extensively applied to functional and logic programs, it was less
successful on imperative C/C++ programs (with a noteable exception of C-M IX [3] and TEMPO [9]).
With the advent of LLVM [15], several new partial evaluators of LLVM bitcode have arisen during
the last few years (e.g., LLPE [21], OCCAM [16], and TRIMMER [20]). These tools leverage LLVM
optimizations such as constant propagation, function inlining, and others to either reduce the size of
the residual program (e.g., OCCAM and TRIMMER ) or improve performance (e.g., LLPE ).
The key step of a PE-based debloater is to estimate whether specializing (or inlining) a function will
result in a smaller (or more secure) residual program. Specialization naturally increases the size of
∗This material is based upon work supported by the National Science Foundation under Grants ACI-1440800
and CNS-1740079. Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are
those of the authors and do not necessarily reﬂect the views of the National Science Foundation.
33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.
Callee Caller Module Call-Site
Basic blocks Basic blocks Functions Arguments
Instructions Instructions Instructions Known arguments
Store instructions Store instructions Basic blocks
Call instructions Call instructions Direct calls
Branch instructions Branch instructions
Loops Loops
Uses Uses
Inﬂuenced branches Untouched call-sites
Inﬂuenced instructions
Table 1: RL features. Inﬂuenced instructions (branches) are the those whose operands have statically
known arguments. Uses indicates how many times the caller and the callee are invoked.
the code since it adds extra functions. However, since some inputs in the new functions have been
replaced by constant values, optimizations such as constant propagation, might become enabled and
result in new optimization opportunities that reduce the code size. Therefore, the decision whether or
not a function should be specialized for a particular call-site is non-trivial. Current PE-based software
debloaters use naive heuristics. For example, OCCAM implements two heuristics of “never specialize”
or “always specialize”, respectively;TRIMMER specializes a function only if it is called once in the
whole program.
In this paper, we present a new approach based on reinforcement learning (RL) to automatically infer
effective heuristics for specializing functions for PE-based software debloating. RL is a good ﬁt
for the problem for two reasons. First, code specialization resembles a Markov Decision Process:
each decision moves the code from one state to another, and specialization depends only on the
current state rather than the history of the transformations. Second, the quality of debloating can
only be measured after all specialization actions and corresponding compiler optimizations have been
performed.
The main challenge in applying RL is in deciding on a good state representation. While it is
tempting to use the source code (or LLVM intermediate representation (IR)) as a state, this is not
computationally tractable. The IR is typically hundreds of MBs in size. Instead, an adequate set of
features that captures meaningful information about the code while avoiding state aliasing is required.
In particular, these features must capture the calling context in order to distinguish between call-sites.
The main contributions of this paper are threefold: (1) calling context features that enable RL to
ﬁnd a useful heuristic for PE-based debloating software; (2) implementation of our method in a tool
called DEEP OCCAM ; and (3) evaluation on the reduction of the program size and number of possible
code-reuse attacks, by comparing our handcrafted features with features learnt automatically via
embedding LLVM IR into a vector space using INST 2VEC [5]. Our initial evaluation suggests that
RL is a viable method for developing an effective specialization policy for debloating, and learning
with handcrafted features is easier than with INST 2VEC .
2 Methods
In this section, we formulate the debloating problem as a reinforcement learning problem, describe
the learning procedure, and its hyperparameters.
Action, state, and reward. To determine whether a call-site should be specialized or not, we train
a policy using reinforcement learning. This requires deﬁning action, state, and reward.
An action is whether to run a code transformation, called specialization, or not. Consider a call to
function calleeF with arguments A at a call-site ci. Let formals be a map from call-site arguments
to their corresponding formal parameters. Let B, B ⊆A, be the arguments whose values are are
statically known. Then, specialization of the call-site does:
1. Create a new function spec_calleeF (formals(A \B)), such that the body of spec_calleeF
is identical to the body of the original calleeF except that formals(B) are replaced with
their values.
2. Perform constant propagation to push forward the information to the rest of callee’s body,
potentially specializing more call-sites in the callee.
3. Replace calleeF(A) with spec_calleeF (A \B) at the call-site ci.
2
Note that after specialization, a software debloater can trigger other optimization passes such as
inlining and dead-code elimination.
A state is a vector capturing all the relevant information about the code. We experiment with two
state representations: (a) a vector of hand crafted features (HF), and (b) an embedding of LLVM IR
via INST 2VEC (IV).
In HF, the state is a concatenation of four feature vectors, summarized in Table 1. They capture
relevant information about the callee, caller, compilation unit (module), and call-site. Most features
are self-explanatory, and hence, we focus only on those which are novel or more relevant for
avoiding state aliasing. The state aliasing problem occurs when two different states have the same
representation in the RL model. In our context, our features must distinguish between two call-sites in
the same basic block when invoking the same callee with the same arguments if one is specialized and
the other not. In [14], the features InLoop (whether a call-site is in a loop), InlineDepth (the current
inlining depth), and currentGraphSize (how many instructions are there in the caller, including one
added by inlining), are used to decide inlining. However, in our case, these features are insufﬁcient to
prevent state aliasing. Thus, we also add Untouched call-sites that counts the number of call-sites yet
to be processed. Since the call-sites are processed in a ﬁxed order, Untouched call-sites is sufﬁcient
to distinguish any two call-sites in a function.
In IV, we use INST 2VEC embedding from [5]. We extract the LLVM IR of the caller, the callee,
and the calling context (a window of n instructions around the call-site) as a lists of instructions.
Each instruction is embedded into a vector space using INST 2VEC . Additionally, we encode the
arguments at the call-site as a bitvector, in which each statically known argument is encoded by1,
and an unknown argument by 0. This bitvector is then embedded using a different embedding matrix.
Finally, the IV state is a tuple of the above four 2-D matrices. Note that in this case, the calling
context of each call is explicitly represented by the IR.
For rewards, we focus on two different metrics. First, we measure the number of instructions in the
ﬁnal binary produced by the software debloater after specialization took place. Second, we measure
the reduction of the attack surface, focusing on code reuse attacks.
Code reuse attacks are exploits in which an attacker makes use of the available instructions in the
binary to chain together short sequences called gadgets, and use those gadgets to compromise the
control ﬂow of the program, causing a malicious effect. Those sequences are often categorized based
on their last instruction, into ROP (return-oriented programming), JOP (jump-oriented programming),
and COP (call-oriented programming) gadgets, respectively. Since the relationship between the
number of gadgets and exploitability is an open question [6], we focus on reducing the number of
gadgets without making any further claim about the security of the debloated code.
The rewards are the negations of the number of instructions, ROP, JOP, and COP gadgets. The
negation is necessary because we are interested in minimizing our metrics. We only have one
measurement at the end of an episode (when the ﬁnal binary is produced). Immediate rewards after
each action are set to zero.
Learning procedure, policy network, and hyper-parameters. Each state representation requires
a different neural net for the policy network. For HF, we use a 3-layer fully connected network.
Before training, we run the pipeline with a random policy 100 times to calculate the mean and
standard deviation of each feature, and use this metadata to normalize all features into mean of 0
deviation of 1.
For IV, we run the caller, the callee, the calling context and the arguments bitvector through four
separate 2-layer GRU [8] blocks, concatenate the last hidden outputs of these 4 blocks, and then feed
it to a 3-layer fully connected network. The architecture is depicted in Fig. 1a.
Both networks use ReLU [17] as the activation function. We use REINFORCE [22] with normalized
rewards to update the policy for both models. At each REINFORCE iteration, we roll out k runs of the
current policy, batch them together, and use the Adam optimizer [13] to update the network. For all
metrics, we use the same hyper-parameters: INST 2VEC calling context n = 10, number of runs in
each policy rollout k = 75, Adam’s learning rate =0.001, and train up to 340 iterations.
3
Caller
Embedding
Callee
Embedding
Context
Embedding
Args bitvector
Embedding
2-layers 
GRU
2-layers 
GRU
2-layers 
GRU
2-layers 
GRU
h_caller h_callee h_ctx h_args
Fully connected 1
Fully connected 2
output
(a) Policy network architecture using INST 2VEC .
App + 
Libraries
  Occam
Pytorch
Debloated   
   Binary
   Metrics 
measuring
State: how does the 
code look like? Action: specialize or not
Manifest
  gRPC
Rewards
(b) DEEP OCCAM architecture. Boxes in blue cor-
respond to this paper.
3 Implementation and Evaluation
We have implemented our prototype, called DEEP OCCAM , using OCCAM [16]. The architecture of
DEEP OCCAM is depicted in Fig. 1b. DEEP OCCAM takes as inputs a set of LLVM modules (main
application and libraries) and a manifest (i.e., a user-deﬁned execution environment) and produces a
specialized binary. We used Gadget Set Analyzer (GSA ) [6] to evaluate the specialized binary and
PYTORCH [18] for training the deep learning models.
At each call-site, OCCAM calculates the features described in Table 1 from theLLVM module, sending
them to the PYTORCH server, and transforming the code based on the returned decision. During
inference, the PYTORCH server receives the message and runs the learnt policy to decide whether to
specialize that call-site. During training, the PYTORCH server also keeps track of all the decisions
it has made and uses that information to update its policy. We run multiple copies of OCCAM on
multiple copies of the PYTORCH server to scale up the learning process, which is justiﬁed by the
Markov property of the state.
We compare DEEP OCCAM with OCCAM running in two modes. First, OCCAM AGG that runs OC-
CAM with a nonrecursive-aggressive policy. This always specializes a call-site if the callee is not a
recursive function. Second, OCCAM NONE that runs OCCAM without specializing any call-site. Fig. 2
shows the comparision between DEEP OCCAM using HF , DEEP OCCAM using IV, OCCAM AGG, and
OCCAM NONE for debloating GNU T REE . On average, DEEP OCCAM outperforms both OCCAM -
NONE and OCCAM AGG in reducing the number of ROP and JOP gadgets, matches OCCAM NONE
on COP gadgets, and underperforms in reducing the number of instructions. Interestingly, while
DEEP OCCAM matches OCCAM NONE on COP, it ﬁnds a different policy that specialize some call-sites.
For optimizing the number of instructions, we run 200 more training iterations but are not able to
outperform OCCAM NONE . We include the results for the number of instructions only to demonstrate
the ﬂexibility of our approach.
In all of our experiments, the learning procedure does not converge when we use INST 2VEC pre-
trained embedding. Upon closer inspection, we observe that the software in our test suite, once
compiled to LLVM IR, contained many instructions regarded as low-frequency by INST 2VEC .
Consequently, they are all mapped into the same UNK! token in the cutoff dictionary. Hence, the
calling context window that we use suffers from the state aliasing problem: different calling contexts
are mapped to the same vector of tokens.
This is surprising since INST 2VEC claims to train the embedding on a wide range of software written
in C, C++, FORTRAN, and OpenCL, speciﬁcally to avoid overﬁtting to a small family of code bases.
4 Related work and Conclusions
Recent advances in deep learning and RL have opened new frontiers to the design of compiler
heuristics [4]. Most current approaches deﬁne actions at the compilation unit level: whether to run
a particular optimization pass, or how to schedule optimization passes. For example, Cummins et
al. [10] model code as a natural language problem to predict whether to run the code on CPU or GPU,
as well as the optimal GPU thread coarsening factors. Kulkarni et al. [14] use NEAT — a genetic
algorithm — to learn an inlining heuristic for MaxineVM as a neural network. For interpretability,
4
0 50 100 150 200 250 300 350
8900
9000
9100
9200
9300
9400
9500
9600
9700
(a) Instructions (HF)
0 20 40 60 80 100 120
6
8
10
12
14
16
18
20
 (b) COP gadgets (HF)
0 20 40 60 80 100 120
400
420
440
460
480
500
(c) ROP gadgets (HF)
0 20 40 60 80 100 120
10
15
20
25
30
35
 (d) JOP gadgets (HF)
0 20 40 60 80 100 120
400
420
440
460
480
500
(e) ROP gadgets (IV)
0 20 40 60 80 100 120
10
15
20
25
30
35
 (f) JOP gadgets (IV)
Figure 2: Results for optimizing different metrics using HF and IV. x-axis is the number of RL
iterations. y-axis is the results. Dots and bars are mean and std. dev. of k runs in each iteration.
HF results are in blue while IV results are in green. Orange and red lines are OCCAM NONE and
OCCAM AGG results, respectively.
they then approximate the neural network by a decision tree. We, on the other hand, use actions at
much lower granularity, focusing on specialization at each call-site.
Program code is a rich structure that can be presented in a variety of ways, e.g., as raw text, control-
ﬂow and call- graphs, etc. Recent application of deep learning for code experiment with models
based on techniques from Natural Language Processing and Graph Neural Network (e.g., [2, 5, 1, 7]).
Among them, INST 2VEC [5] is the only one that works on LLVM IR.
The closest related work is Chisel [11] – a software debloater based on RL. Inspired by C-Reduce [19],
Chisel takes a program P and a property of interest ϕ (i.e., P must compile and pass tests deﬁning
ϕ) and produces the smallest program P′ that satisﬁes ϕ. RL accelerates the search for the reduced
program providing a scalability boost. Compared to DEEP OCCAM , Chisel has very different state
and action space, and, is generally only as sound as the test cases deﬁning ϕ.
In this paper, we present DEEP OCCAM — an end-to-end tool to learn specialization heuristics for
software debloating. Our preliminary results suggest that it is feasible to use RL to learn an effective
specialization heuristic to optimize a variety of related metrics. They also suggest that out of the
box, pretrained embedding such as INST 2VEC might not be applicable for the task. We hope that
5
DEEP OCCAM contributions in feature engineering and architecture might be applicable to other
compiler optimization tasks such as inlining.
References
[1] Uri Alon, Omer Levy, and Eran Yahav. code2seq: Generating sequences from structured
representations of code. CoRR, abs/1808.01400, 2018.
[2] Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: learning distributed
representations of code. pages 40:1–40:29, 2019.
[3] Lars Ole Andersen. Program analysis and specialization for the C programming language .
PhD thesis, University of Copenhagen, 1994.
[4] Amir H. Ashouri, William Killian, John Cavazos, Gianluca Palermo, and Cristina Silvano. A
survey on compiler autotuning using machine learning. ACM Comput. Surv., 51(5):96:1–96:42,
2019.
[5] Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoeﬂer. Neural code comprehension: A
learnable representation of code semantics. CoRR, abs/1806.07336, 2018.
[6] Michael D. Brown and Santosh Pande. Is less really more? towards better metrics for measuring
security improvements realized through software debloating. In 12th USENIX Workshop on
Cyber Security Experimentation and Test (CSET 19), Santa Clara, CA, August 2019. USENIX
Association.
[7] Zimin Chen and Martin Monperrus. A literature study of embeddings on source code. CoRR,
abs/1904.03061, 2019.
[8] Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk,
and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical
machine translation. CoRR, abs/1406.1078, 2014.
[9] Charles Consel, Luke Hornof, Renaud Marlet, Gilles Muller, Scott Thibault, E-N V olanschi,
Julia Lawall, and Jacques Noyé. Tempo: Specializing systems applications and beyond. ACM
Computing Surveys, 30(3es), 1998.
[10] C. Cummins, P. Petoumenos, Z. Wang, and H. Leather. End-to-end deep learning of optimization
heuristics. In 2017 26th International Conference on Parallel Architectures and Compilation
Techniques (PACT), pages 219–232, Sep. 2017.
[11] Kihong Heo, Woosuk Lee, Pardis Pashakhanloo, and Mayur Naik. Effective program debloating
via reinforcement learning. In CCS, pages 380–394, 2018.
[12] Neil D. Jones, Carsten K. Gomard, and Peter Sestoft. Partial evaluation and automatic program
generation. Prentice Hall international series in computer science. Prentice Hall, 1993.
[13] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd
International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May
7-9, 2015, Conference Track Proceedings, 2015.
[14] Sameer Kulkarni, John Cavazos, Christian Wimmer, and Doug Simon. Automatic construction
of inlining heuristics using machine learning. In CGO, pages 9:1–9:12, 2013.
[15] Chris Lattner and Vikram Adve. Llvm: A compilation framework for lifelong program analysis
& transformation. In CGO, pages 75–, 2004.
[16] Gregory Malecha, Ashish Gehani, and Natarajan Shankar. Automated software winnowing. In
SAC, pages 1504–1511, 2015.
[17] Vinod Nair and Geoffrey E. Hinton. Rectiﬁed linear units improve restricted boltzmann
machines. In Proceedings of the 27th International Conference on International Conference on
Machine Learning, ICML’10, pages 807–814, USA, 2010. Omnipress.
6
[18] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito,
Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in
PyTorch. In NIPS Autodiff Workshop, 2017.
[19] John Regehr, Yang Chen, Pascal Cuoq, Eric Eide, Chucky Ellison, and Xuejun Yang. Test-case
reduction for C compiler bugs. In PLDI, pages 335–346, 2012.
[20] Hashim Sharif, Muhammad Abubakar, Ashish Gehani, and Fareed Zaffar. Trimmer: Application
specialization for code debloating. In ASE, pages 329–339, 2018.
[21] Christopher S.F. Smowton. I/O Optimisation and elimination via partial evaluation. Technical
Report UCAM-CL-TR-865, University of Cambridge, Computer Laboratory, December 2014.
[22] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforce-
ment learning. Mach. Learn., 8(3-4):229–256, May 1992.
7

OpenTuner: An Extensible Framework for Program
Autotuning
Jason Ansel Shoaib Kamil Kalyan Veeramachaneni
Jonathan Ragan-Kelley Jeffrey Bosboom
Una-May O’Reilly Saman Amarasinghe
Massachusetts Institute of Technology
Cambridge, MA
{jansel, skamil, kalyan, jrk, jbosboom, unamay, saman}@csail.mit.edu
ABSTRACT
Program autotuning has been shown to achieve better
or more portable performance in a number of domains.
However, autotuners themselves are rarely portable between
projects, for a number of reasons: using a domain-informed
search space representation is critical to achieving good
results; search spaces can be intractably large and require
advanced machine learning techniques; and the landscape of
search spaces can vary greatly between diﬀerent problems,
sometimes requiring domain speciﬁc search techniques to
explore eﬃciently.
This paper introduces OpenTuner, a new open source
framework for building domain-speciﬁc multi-objective pro-
gram autotuners. OpenTuner supports fully-customizable
conﬁguration representations, an extensible technique rep-
resentation to allow for domain-speciﬁc techniques, and an
easy to use interface for communicating with the program to
be autotuned. A key capability inside OpenTuner is the use
of ensembles of disparate search techniques simultaneously;
techniques that perform well will dynamically be allocated
a larger proportion of tests. We demonstrate the eﬃcacy
and generality of OpenTuner by building autotuners for 7
distinct projects and 16 total benchmarks, showing speedups
over prior techniques of these projects of up to 2 .8×with
little programmer eﬀort.
1. INTRODUCTION
Program autotuning is increasingly being used in domains
such as high performance computing and graphics to opti-
mize programs. Program autotuning augments traditional
human-guided optimization by oﬄoading some or all of
the search for an optimal program implementation to an
automated search technique. Rather than optimizing a
program directly, the programmer expresses a search space
of possible implementations and optimizations. Autotuning
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proﬁt or commercial advantage and that copies bear this notice and the full citation
on the ﬁrst page. Copyrights for components of this work owned by others than
ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior speciﬁc permission
and/or a fee. Request permissions from Permissions@acm.org.
PACT’14,August 24–27, 2014, Edmonton, AB, Canada.
Copyright 2014 ACM 978-1-4503-2809-8/14/08 ...$15.00.
http://dx.doi.org/10.1145/2628071.2628092.
can often make the optimization process more eﬃcient
as autotuners are able to search larger spaces than is
possible by hand. Autotuning also provides performance
portability, as the autotuning process can easily be re-run on
new machines which require diﬀerent sets of optimizations.
Finally, multi-objective autotuning can be used to trade oﬀ
between performance and accuracy, or other criteria such
as energy consumption and memory usage, and provide
programs which meet given performance or quality of service
targets.
While the practice of autotuning has increased in
popularity, autotuners themselves often remain relatively
simple and project speciﬁc. There are three main challenges
which make the development of autotuning frameworks
diﬃcult.
The ﬁrst challenge is using the right conﬁguration
representation for the problem. Conﬁgurations can contain
parameters that vary from a single integer for a block
size to a much more complex type such as an expression
tree representing a set of instructions. The creator of
the autotuner must ﬁnd ways to represent their complex
domain-speciﬁc data structures and constraints. When
these data structures are naively mapped to simpler
representations, such as a point in high dimensional space,
locality information is lost which makes the search problem
much more diﬃcult. Picking the right representation for the
search space is critical to having an eﬀective autotuner. To
date, all autotuners that have used a representation other
than the simplest ones have had custom project-speciﬁc
representations.
The second challenge is the size of the valid conﬁguration
space. While some prior autotuners have worked hard to
prune the conﬁguration space, we have found that for many
problems excessive search space pruning will miss out on
non-intuitive good conﬁgurations. We believe providing all
the valid conﬁgurations of these search spaces is better than
artiﬁcially constraining search spaces and possibly missing
optimal solutions. Search spaces can be very large, up to
103600 possible conﬁgurations for one of our benchmarks.
Full exhaustive search of such a space will not complete
in human lifetimes! Thus, intelligent machine learning
techniques are required to seek out a good result with a
small number of experiments.
The third challenge is the landscape of the conﬁguration
space. If the conﬁguration space is a monotonic function, a
search technique biased towards this type of search space
(such as a hill climber) will be able to ﬁnd the optimal
conﬁguration. If the search space is discontinuous and
haphazard an evolution algorithm may perform better.
However, in practice search spaces are much more complex,
with discontinuities, high dimensionality, plateaus, hills
with some of the conﬁguration parameters strongly coupled
and some others independent from each other. A search
technique that is optimal in one type of conﬁguration space
may fail to locate an adequate conﬁguration in another. It
is diﬃcult to provide a robust system that performs well
in a variety of situations. Additionally, many application
domains will have domain-speciﬁc search techniques (such
as scheduling or blocking heuristics) which may be critical
to ﬁnding an optimal solution eﬃciently. This has caused
most prior autotuners to use customized search techniques
tailored to their speciﬁc problem. This requires machine
learning expertise in addition to the individual domain
expertise to build an autotuner for a system. We believe
that this is one of the main reasons that, while autotuners
are recognized as critical for performance optimization, they
have not seen commodity adoption.
In this paper we present OpenTuner, a new framework for
building domain-speciﬁc program autotuners. OpenTuner
features an extensible conﬁguration and technique repre-
sentation able to support complex and user-deﬁned data
types and custom search heuristics. It contains a library
of predeﬁned data types and search techniques to make
it easy to setup a new project. Thus, OpenTuner solves
the custom conﬁguration problem by providing not only a
library of data types that will be suﬃcient for most projects,
but also extensible data types that can be used to support
more complex domain speciﬁc representations when needed.
A core concept in OpenTuner is the use of ensembles of
search techniques. Many search techniques (both built in
and user-deﬁned) are run at the same time, each testing
candidate conﬁgurations. Techniques which perform well
by ﬁnding better conﬁgurations are allocated larger budgets
of tests to run, while techniques which perform poorly
are allocated fewer tests or disabled entirely. Techniques
are able to share results using a common results database
to constructively help each other in ﬁnding an optimal
solution. algorithms add results from other techniques as
new members of their population. To allocate tests between
techniques we use an optimal solution to the multi-armed
bandit problem using area under the curve credit assignment.
Ensembles of techniques solve the large and complex search
space problem by providing both a robust solutions to
many types of large search spaces and a way to seamlessly
incorporate domain speciﬁc search techniques.
1.1 Contributions
This paper makes the following contributions:
•To the best of our knowledge, OpenTuner is the ﬁrst
to introduce a general framework to describe complex
search spaces for program autotuning.
•OpenTuner introduces the concept of ensembles of
search techniques to program autotuning, which allow
many search techniques to work together to ﬁnd an
optimal solution.
•OpenTuner provides more sophisticated search tech-
niques than typical program autotuners. This enables
expanded uses of program autotuning to solve more
complex search problems and pushes the state of the
art forward in program autotuning in a way that can
easily be adopted by other projects.
•We demonstrate the versatility of our framework
by building autotuners for 7 distinct projects and
demonstrate the eﬀectiveness of the system with 16
total benchmarks, showing speedups over existing
techniques of up to 2 .8×.
•We show that OpenTuner is able to succeed both
in massively large search spaces, exceeding 10 3600
possible conﬁgurations in size, and in smaller search
spaces using less than 2% of the tests required for
exhaustive search.
2. RELATED WORK
Package Domain Search Method
Active Harmony [31]Runtime System Nelder-Mead
ATLAS [34] Dense Linear AlgebraExhaustive
FFTW [14] Fast Fourier TransformExhaustive/Dynamic Prog.
Insieme [19] Compiler Diﬀerential Evolution
OSKI [33] Sparse Linear AlgebraExhaustive+Heuristic
PATUS [9] Stencil ComputationsNelder-Mead or Evolutionary
PetaBricks [4] Programming LanguageBottom-up Evolutionary
Sepya [21] Stencil ComputationsRandom-Restart Gradient Ascent
SPIRAL [28] DSP Algorithms Pareto Active Learning
Figure 1: Summary of selected related projects using
autotuning
A number of oﬄine empirical autotuning frameworks have
been developed for building eﬃcient, portable libraries in
speciﬁc domains; selected projects and techniques used are
summarized in Figure 1. ATLAS [34] utilizes empirical
autotuning to produce an optimized matrix multiply
routine. FFTW [14] uses empirical autotuning to combine
solvers for FFTs. Other autotuning systems include
SPIRAL [28] for digital signal processing PATUS [9] and
Sepya [21] for stencil computations, and OSKI [33] for sparse
matrix kernels.
The area of iterative compilation contains many projects
that use diﬀerent machine learning techniques to optimize
lower level compiler optimizations [2, 15, 26, 1]. These
projects change both the order that compiler passes are
applied and the types of passes that are applied.
In the dynamic autotuning space, there have been a
number of systems developed [18, 17, 22, 6, 8, 5] that focus
on creating applications that can monitor and automatically
tune themselves to optimize a particular objective. Many of
these systems employ a control systems based autotuner that
operates on a linear model of the application being tuned.
For example, PowerDial [18] converts static conﬁguration
parameters that already exist in a program into dynamic
knobs that can be tuned at runtime, with the goal of
trading QoS guarantees for meeting performance and power
usage goals. The system uses an oﬄine learning stage to
construct a linear model of the choice conﬁguration space
which can be subsequently tuned using a linear control
system. The system employs the heartbeat framework [16]
to provide feedback to the control system. A similar
technique is employed in [17], where a simpler heuristic-
based controller dynamically adjusts the degree of loop
perforation performed on a target application to trade QoS
for performance.
Results Database
Search 
TechniquesSearch 
Driver
Search
Reads: Results
Writes: Desired Results
Measurement
User Defined 
Measurement 
Function
Measurement 
Driver
Configuration 
Manipulator
Reads: Desired Results
Writes: Results
Figure 2: Overview of the major components in the
OpenTuner framework.
Related to our Mario benchmark, Murphy [23] presents
algorithms for playing NES games based on example human
inputs. In contrast, this paper’s Mario example has no
knowledge of the game except for pixels moved to the right.
3. THE OPENTUNER FRAMEWORK
Our terminology reﬂects that the autotuning problem is
cast as a search problem. The search space is made up
of conﬁgurations, which are concrete assignments of a set
of parameters. Parameters can be primitive such as an
integer or complex such as a permutation of a list. When
the performance, output accuracy, or other metrics of a
conﬁguration are measured (typically by running it in a
domain-speciﬁc way), we call this measurement a result.
Search techniques are methods for exploring the search
space and make requests for measurement called desired
results. Search techniques can change conﬁgurations using a
user-deﬁned conﬁguration manipulator, which also includes
parameters corresponding directly the parameters in the
conﬁguration. Some parameters include manipulators,
which are opaque functions that make stochastic changes
to a speciﬁc parameter in a conﬁguration.
Figure 2 provides an overview of the major components
in OpenTuner. The search process includes techniques,
which use the user deﬁned conﬁguration manipulator in
order to read and write conﬁgurations. The measurement
processes evaluate candidate conﬁgurations using a user
deﬁned measurement function. These two components
communicate exclusively through a results database used to
record all results collected during the tuning process, as well
as providing ability to perform multiple measurements in
parallel.
3.1 OpenTuner Usage
To implement an autotuner with OpenTuner, ﬁrst, the
user must deﬁne the search space by creating aconﬁguration
manipulator. This conﬁguration manipulator includes a set
of parameter objects which OpenTuner will search over.
Second, the user must deﬁne a run function which evaluates
the ﬁtness of a given conﬁguration in the search space to
produce a result. These must be implemented in a small
Python program in order to interface with the OpenTuner
API.
Figure 3 shows an example of using OpenTuner to search
over the space of GCC compiler ﬂags in order to minimize
1import opentuner
2from opentuner import ConfigurationManipulator
3from opentuner import EnumParameter
4from opentuner import IntegerParameter
5from opentuner import MeasurementInterface
6from opentuner import Result
7
8GCC_FLAGS = [
9’align-functions’, ’align-jumps’, ’align-labels’,
10’branch-count-reg’, ’branch-probabilities’,
11# ... (176 total)
12]
13
14# (name, min, max)
15GCC_PARAMS = [
16(’early-inlining-insns’, 0, 1000),
17(’gcse-cost-distance-ratio’, 0, 100),
18# ... (145 total)
19]
20
21class GccFlagsTuner(MeasurementInterface):
22
23def manipulator(self):
24"""
25Define the search space by creating a
26ConfigurationManipulator
27"""
28manipulator = ConfigurationManipulator()
29manipulator.add_parameter(
30IntegerParameter(’opt_level’, 0, 3))
31for flag in GCC_FLAGS:
32manipulator.add_parameter(
33EnumParameter(flag,
34[’on’, ’off’, ’default’]))
35for param, min, max in GCC_PARAMS:
36manipulator.add_parameter(
37IntegerParameter(param, min, max))
38return manipulator
39
40def run(self, desired_result, input, limit):
41"""
42Compile and run a given configuration then
43return performance
44"""
45cfg = desired_result.configuration.data
46gcc_cmd = ’g++ raytracer.cpp -o ./tmp.bin’
47gcc_cmd += ’ -O{0}’.format(cfg[’opt_level’])
48for flag in GCC_FLAGS:
49if cfg[flag] == ’on’:
50gcc_cmd += ’ -f{0}’.format(flag)
51elif cfg[flag] == ’off’:
52gcc_cmd += ’ -fno-{0}’.format(flag)
53for param, min, max in GCC_PARAMS:
54gcc_cmd += ’ --param {0}={1}’.format(
55param, cfg[param])
56
57compile_result = self.call_program(gcc_cmd)
58assert compile_result[’returncode’] == 0
59run_result = self.call_program(’./tmp.bin’)
60assert run_result[’returncode’] == 0
61return Result(time=run_result[’time’])
62
63if __name__ == ’__main__’:
64argparser = opentuner.default_argparser()
65GccFlagsTuner.main(argparser.parse_args())
Figure 3: GCC/G++ ﬂags autotuner using OpenTuner.
execution time of the resulting program. In Section 4, we
present results on an expanded version of this example which
obtains up to 2.8x speedup over -O3.
This example tunes three types of ﬂags to GCC. First
it chooses between the four optimization levels -O0, -O1, -
O2, -O3. Second, for 176 ﬂags listed on line 8 it decides
between turning the ﬂag on (with -fFLAG), oﬀ (with -fno-
FLAG), or omitting the ﬂag to allow the default value to
take precedence. Including the default value as a choice is
not necessary for completeness, but speeds up convergence
and results in shorter command lines. Finally, it assigns a
bounded integer value to the 145 parameters on line 15 with
the --param NAME=VALUE command line option.
The method manipulator (line 23), is called once at
startup and creates a ConfigurationManipulator object
which deﬁnes the search space of GCC ﬂags. All accesses
to conﬁgurations by search techniques are done through
the conﬁguration manipulator. For optimization level, an
IntegerParameter between 0 and 3 is created. For each
ﬂag, a EnumParameter is created which can take the values
on, off, and default. Finally, for the remaining bounded
GCC parameters, an IntegerParameter is created with the
appropriate range.
The method run (line 40) implements the measurement
function for conﬁgurations. First, the conﬁguration is
realized as a speciﬁc command line to g++. Next, this g++
command line is run to produce an executable, tmp.bin,
which is then run using call_program. call_program is a
convenience function which runs and measures the execution
time of the given program. Finally, a Result is constructed
and returned, which is a database record type containing
many other optional ﬁelds such as time, accuracy, and
energy. By default OpenTuner minimizes the time ﬁeld,
but this can be customized.
3.2 Search Techniques
To provide robust search, OpenTuner includes techniques
that can handle many types of search spaces and runs
a collection of search techniques at the same time.
Techniques which perform well are allocated more tests,
while techniques which perform poorly are allocated fewer
tests. Techniques share results through the results database,
so that improvements made by one technique can beneﬁt
other techniques. This sharing occurs in technique-speciﬁc
ways; for example, evolutionary techniques add results
found by other techniques as members of their population.
OpenTuner techniques are meant to be extended. Users
can deﬁne custom techniques which implement domain-
speciﬁc heuristics and add them to ensembles of pre-deﬁned
techniques.
Ensembles of techniques are created by instantiating a
meta technique, which is a technique made up of a collection
of other techniques. The OpenTuner search driver interacts
with a single root technique, which is typically a meta
technique. When the meta technique gets allocated tests,
it incrementally decides how to divide theses tests among
its sub-techniques. OpenTuner contains an extensible class
hierarchy of techniques and meta techniques, which can be
combined together and used in autotuners.
3.2.1 AUC Bandit Meta Technique
In addition to a number of simple meta techniques, such
as round robin, OpenTuner’s core meta technique used
in results is the multi-armed bandit with sliding window,
area under the curve credit assignment (AUC Bandit)
meta technique. A similar technique was used in [25] in
the diﬀerent context of online operator selection. It is
based on an optimal solution to the multi-armed bandit
problem [12]. The multi-armed bandit problem is the
problem of picking levers to pull on a slot machine with
many arms each with an unknown payout probability. The
sliding window makes the technique only consider a subset of
history (the length of the window) when making decisions.
This meta-technique encapsulates a fundamental trade-oﬀ
between exploitation (using the best known technique) and
exploration (estimating the performance of all techniques).
The AUC Bandit meta technique assigns each test to
the technique, t, deﬁned by the formula arg max t(AUC t +
C
√
2 lg|H|
Ht
) where |H|is the length of the sliding history
window, Ht is the number of times the technique has been
used in that history window, C is a constant controlling
the exploration/exploitation trade-oﬀ, and AUC t is the
credit assignment term quantifying the performance of the
technique in the sliding window. The second term in the
equation is the exploration term which gets smaller the more
often a technique is used.
The area under the curve credit assignment mechanism,
based on [13], draws a curve by looking at the history for a
speciﬁc technique and looking only at if a technique yielded a
new global best or not. If the technique yielded a new global
best, a upward line is drawn, otherwise a ﬂat line is drawn.
The area under this curve (scaled to a maximum value of 1)
is the total credit attributed to the technique. This credit
assignment mechanism can be described more precisely by
the formula: AUC t = 2
|Vt|(|Vt|+1)
∑|Vt|
i=1 iVt,i where Vt is the
list of uses of t in the sliding window history. Vt,i is 1 if
using technique t the ith time in the history resulted in a
speedup, otherwise 0.
3.2.2 Other Techniques
OpenTuner includes implementations of the techniques:
diﬀerential evolution; many variants of Nelder-Mead search
and Torczon hillclimbers; a number of evolutionary mutation
techniques; pattern search; particle swarm optimization; and
random search. OpenTuner also includes a bandit mutation
technique which uses the same AUC Bandit method to
decide which manipulator function across all parameters to
call on the best known conﬁguration. These techniques
span a range of strategies and are each biased to perform
best in diﬀerent types of search spaces. They also each
contain many settings which can be conﬁgured to change
their behavior. Each technique has been modiﬁed so that
with some probability it will use information found by
other techniques if other techniques have discovered a better
conﬁguration.
The default meta technique, used in results in this paper
and meant to be robust, uses an AUC Bandit meta technique
to combine greedy mutation, diﬀerential evolution, and two
hill climber instances.
3.3 Conﬁguration Manipulator
The conﬁguration manipulator provides a layer of ab-
straction between the search techniques and the raw
conﬁguration structure. It is primarily responsible for
managing a list of parameter objects, each of which can be
Parameter
Primitive Complex
Integer ScaledNumericFloat
LogInteger LogFloat PowerOfTwo
Switch Enum Permutation
Schedule
SelectorBoolean
Figure 4: Hierarchy of built in parameter types. User
deﬁned types can be added at any point below Primitive
or Complex in the tree.
used by search techniques to read and write parts of the
underlying conﬁguration.
The default implementation of the conﬁguration ma-
nipulator uses a ﬁxed list of parameters and stores
the conﬁguration as a dictionary from parameter name
to parameter-dependent data type. The conﬁguration
manipulator can be extended by the user either to change
the underlying data structure used for conﬁgurations or to
support a dynamic list of parameters that is dependent on
the conﬁguration instance.
3.3.1 Parameter Types
OpenTuner has a hierarchy of built-in parameter types.
Each parameter type is responsible for interfacing between
the raw representation of a parameter in the conﬁguration
and the standardized view of that parameter presented to
the search techniques. Parameter types can be extended
both to change the underlying representation, and to change
the abstraction provided to search techniques to cause a
parameter to be search in diﬀerent ways.
From the viewpoint of search techniques there are two
main types of parameters, each of which provides a diﬀerent
abstraction to the search techniques:
Primitive parameters present a view to search techniques
of a numeric value with an upper and lower bound.
These upper and lower bounds can be dependent on the
conﬁguration instance.
The built in parameter types Float and LogFloat (and
similarly Integer and LogInteger) both have identical
representations in the conﬁguration, but present a diﬀerent
view of the underlying value to the search techniques.
Float is presented directly to to search techniques, while
LogFloat presents a log scaled view of the underlying
parameter to search techniques. To a search technique,
halving and doubling a log scaled parameter are changes
of equal magnitude. Log scaled variants of parameters are
often better for parameters such as block sizes where ﬁxed
changes in values have diminishing eﬀects the larger the
parameter becomes. PowerOfTwo is a commonly used special
case, similar to LogInteger, where the legal values of the
parameter are restricted to powers of two.
Complex parameters present a more opaque view to search
techniques. Complex parameters have a variable set of ma-
nipulation operators (manipulators) which make stochastic
changes to the underlying parameter. These manipulators
are arbitrary functions deﬁned on the parameter which
can make high level type dependent changes. Complex
parameters are meant to be easily extended to add domain
speciﬁc structures to the search space. New complex
parameters deﬁne custom manipulation operators for use by
the conﬁguration manipulators.
The built in parameter types Boolean, Switch, and
Enum could theoretically also be represented as primitive
parameters, since they each can be translated directly to
a small integer representation. However, in the context
of search techniques they make more sense as complex
parameters. This is because, for primitive parameters,
search techniques will attempt to follow gradients. These
parameter types are unordered collections of values for which
no gradients exist. Thus, the complex parameter abstraction
is a more eﬃcient representation to search over.
The Permutation parameter type assigns an order to a
given list of values and has manipulators which make various
types of random changes to the permutation. A Schedule
parameter is a Permutation with a set of dependencies
that limit the legal order. Schedules are implemented as a
permutation that gets topologically sorted after each change.
Finally, a Selector parameter is a special type of tree which
is used to deﬁne a mapping from an integer input to an
enumerated value type.
In addition to these primary primitive and complex
abstractions for parameter types, there are a number of
derived ways that search techniques will interact with
parameters in order to more precisely convey intent. These
are additional methods on parameter which contain default
implementations for both primitive and complex parameter
types. These methods can optionally be overridden for
speciﬁc parameters types to improve search techniques.
Parameter types will work without these methods being
overridden, but implementing them can improve results.
As an example, a common operation in many search
techniques is to add the diﬀerence between conﬁguration A
and B to conﬁguration C. This is used both in diﬀerential
evolution and many hill climbers. Complex parameters have
a default implementation of this indent which compares the
value of the parameter in the 3 conﬁgurations: if A = B,
then there is no diﬀerence and the result is C; similarly,
if B = C, then A is returned; otherwise a change should
be made so random manipulators are called. This works in
general, but for individual parameter types there are often
better interpretations. For example, with permutations one
could calculate the positional movement of each item in
the list and calculate a new permutation by applying these
movements again.
3.4 Objectives
OpenTuner supports multiple user-deﬁned objectives.
Result records have ﬁelds for time, accuracy, energy, size,
confidence, and user deﬁned data. The default objective is
to minimize time. Many other objectives are supported,
such as: maximize accuracy; threshold accuracy while
minimizing time; and maximize accuracy then minimize size.
The user can easily deﬁne their own objective by deﬁning
comparison operators and display methods on a subclass of
Objective.
3.5 Search Driver and Measurement
OpenTuner is divided into two submodules, search and
measurement. The search driver and measurement driver in
each of these modules orchestrate most of the framework of
the search process. These two modules communicate only
through the results database. The measurement module
is minimal by design and is primarily a wrapper around
the user deﬁned measurement function which creates results
from conﬁgurations.
This division between search and measurement is moti-
vated by a number of diﬀerent factors:
•To allow parallelism between multiple search mea-
surement processes, possibly across diﬀerent machines.
Parallelism is most important in the measurement
processes since in most autotuning applications mea-
surement costs dominate. To allow for parallelism the
search driver will make multiple requests for desired
results without waiting for each request to be fulﬁlled.
If a speciﬁc technique is blocked waiting for results,
other techniques in the ensemble will used to ﬁll out
requests to prevent idle time.
•The separation of the measurement modules is
desirable to support online learning and sideline
learning. In these setups, autotuning is not done before
deployment of an application, but is done online as an
application is running or during idle time. Since the
measurement module is minimal by design, it can be
replaced by an domain speciﬁc online learning module
which periodically examines the database to decide
which conﬁguration to use and records performance
back to the database.
•Finally, in many embedded or mobile settings which
require constrained measurement environments it is
desirable to have a minimal measurement module
which can easily be re-implemented in other languages
without needing to modify the majority of the
OpenTuner framework.
3.6 Results Database
The results database is a fully featured SQL database. All
major database types are supported, and SQLite is used if
the user has not conﬁgured a database type so that no setup
is required. It allows diﬀerent techniques to query and share
results in a variety of ways and is useful for introspection
about the performance of techniques across large numbers
of runs of the autotuner.
4. EXPERIMENTAL RESULTS
Project BenchmarkPossible Conﬁgurations
GCC/G++ Flags all 10806
Halide Blur 1025
Halide Wavelet 1032
Halide Bilateral 10176
HPL n/a 109.9
PetaBricks Poisson 103657
PetaBricks Sort 1090
PetaBricks Strassen 10188
PetaBricks TriSolve 101559
Stencil all 106.5
Unitary n/a 1021
Mario n/a 106328
Figure 5: Search space sizes in number of possible
conﬁgurations, as represented in OpenTuner.
We validated OpenTuner by using it to implemented
autotuners for seven distinct projects. This section describes
these seven projects, the autotuners we implemented, and
presents results comparing to prior practices in each project.
Figure 5 lists, for each benchmark, the number of distinct
conﬁgurations that can be generated by OpenTuner. This
measure is not perfect because some conﬁgurations may be
semantically equivalent and the search space depends on
the representation chosen in OpenTuner. It does, however,
provide a sense of the relative size of each search space,
which is useful as a ﬁrst approximation of tuning diﬃculty.
For many of these benchmarks, the size of the search space
makes exhaustive search intractable. Thus, we compare to
the best performance obtained by the benchmark authors
when optimality is impossible to quantify.
For each benchmark, we keep the environment as constant
as possible, but do not use the same environment for
all benchmarks, due to particular requirements for each
benchmark, and because one (the stencil benchmark) is
based on data from an environment outside our control.
4.1 GCC/G++ Flags
The GCC/G++ ﬂags autotuner is described in detail
in Section 3.1. There are a number features that were
omitted from the earlier example code for simplicity, which
are included in the full version of the autotuner.
First, we added error checking to gracefully handle the
compiler or the output program hanging, crashing, running
out of memory, or otherwise going wrong. Our tests
uncovered a number of bugs in GCC which triggered
internal compiler errors and we implemented code to detect,
diagnose, and avoid error-causing sets of ﬂags. We are
submitting bug reports for these crashes to the GCC
developers.
Second, instead of using a ﬁxed list of ﬂags and
parameters (which the example does for simplicity), our full
autotuner automatically extracts the supported ﬂags from
g++ --help=optimizers. Parameters and legal ranges are
extracted automatically fromparams.def in the GCC source
code.
Additionally, there were a number of smaller features
such as: time limits to abort slow tests which will not be
optimal; use of LogInteger parameter types for some values;
a save_final_config method to output the ﬁnal ﬂags; and
many command line options to control autotuner behavior.
We ran experiments using gcc 4.7.3-1ubuntu1, on an 8
total core, 2-socket Xeon E5320. We allowed ﬂags such as
-ffast-math which can change rounding or NaN behavior of
ﬂoating-point numbers and have small impacts on program
results. We still observe speedups with these ﬂags removed.
For target programs to optimize we used: a fast Fourier
transform in C, ﬀt.c, taken from the SPLASH2 [35]
benchmark suite; a C++ template matrix multiply,matrix-
multiply.cpp, written by Xiang Fan [11] (version 3); a C++
ray tracer, raytracer.cpp, taken from the scratchpixel
website [27]; and a genetic algorithm to solve the traveling
salesman program in C++, tsp_ga.cpp, by Kristoﬀer
Nordkvist [24], which we modiﬁed to run deterministically.
These programs were chosen to span a range from
highly optimized codes, like fft.c which contains cache
aware tiling and threading, to less optimized codes, like
matrixmultiply.cpp which contains only a transpose of one
of the inputs.
Figure 6 shows the performance for autotuning GCC ﬂags
on these four diﬀerent sample programs. Final speedups
ranged from 1.15×for FFT to 2 .82×for matrix multiply.
Figure 7 shows a separate experiment to try to understand
which ﬂags contributed the most to the speedups obtained.
Full GCC command lines found contained over 250 options
 0.8
 0.85
 0.9
 0.95
 1
 0  300  600  900  1200  1500  1800
Execution Time (seconds)
Autotuning Time (seconds)
gcc -O1
gcc -O2
gcc -O3
OpenTuner
(a) ﬀt.c
 0.1
 0.15
 0.2
 0.25
 0.3
 0  300  600  900  1200  1500  1800
Execution Time (seconds)
Autotuning Time (seconds)
g++ -O1
g++ -O2
g++ -O3
OpenTuner (b) matrixmultiply.cpp
 0.4
 0.45
 0.5
 0.55
 0.6
 0.65
 0.7
 0.75
 0  300  600  900  1200  1500  1800
Execution Time (seconds)
Autotuning Time (seconds)
g++ -O1
g++ -O2
g++ -O3
OpenTuner
(c) raytracer.cpp
 0.4
 0.45
 0.5
 0.55
 0.6
 0  300  600  900  1200  1500  1800
Execution Time (seconds)
Autotuning Time (seconds)
g++ -O1
g++ -O2
g++ -O3
OpenTuner (d) tsp ga.cpp
Figure 6: GCC/G++ Flags: Execution time (lower is better) as a function of autotuning time. Aggregated performance
of 30 runs of OpenTuner, error bars indicate median, 20th, and 80th percentiles. Note that in (b) the O1/O2/O3 and in (c)
the O2/O3 lines are on top of each other and may be diﬃcult to see.
and are diﬃcult understand by hand. To approximate the
importance of each ﬂag, we measured the slowdown resulting
from removing it from the autotuned GCC command. We
normalize all slowdowns to sum to 100%. This experiment
was conducted on a single run of the autotuner. This
measurement of importance is imperfect because ﬂags can
interact in complex ways and are not independent. We
note that the total slowdown from removing each ﬂag
independently is larger than the slowdown of removing all
ﬂags.
The benchmarks show interesting behavior and varying
sets of optimal ﬂags. The matrix multiply benchmark
reaches optimal performance through a few large steps, and
its speedup is dominated by 3 ﬂags: -fno-exceptions, -
fwrapv, and -funsafe-math-optimizations. On the other
hand, TSP takes many small, incremental steps and its
speedup is spread over a large number of ﬂags with smaller
eﬀects. The FFT benchmark is heavily hand optimized and
performs more poorly at -O3 than at -O2; the ﬂags found
for it are primarily disabling, rather than enabling, various
compiler optimizations which interact poorly with its hand-
written optimizations. While there are some patterns, each
benchmark requires a diﬀerent set of ﬂags to get the best
performance.
4.2 Halide
Halide [29, 30] is a domain-speciﬁc language and compiler
for image processing and computational photography,
speciﬁcally targeted towards image processing pipelines
(graphs) that contain many stages. Halide separates
the expression of the kernels and pipeline itself from the
pipeline’s schedule, which deﬁnes the order of execution
and placement of data by which it is mapped to machine
execution. The schedule dictates how the Halide compiler
synthesizes code for a given pipeline. This allows expert
programmers to easily deﬁne and explore the space of
complex schedules which result in high performance.
Until today, production uses of Halide have relied on
hand-tuned schedules written by experts. The autotuning
problem in Halide is to automatically synthesize schedules
and search for high-performance conﬁgurations of a given
pipeline on a given architecture. The Halide project
previously integrated a custom autotuner to automatically
search over the space of schedules, but it was removed
because it became too complex to maintain and was diﬃcult
to use in practice. 1
1Unfortunately, the original Halide autotuner cannot be
used as a baseline to compare against, because the Halide
language and compiler have changed too much since its
removal to make direct comparison meaningful.
Flag (Importance)
-fno-tree-vectorize (25.0%)-funroll-loops (6.7%)-fno-jump-tables (5.4%)-fno-inline (5.0%)-fno-ipa-pure-const (4.8%)-fno-tree-cselim (4.7%)-fno-rerun-cse-after-loop (4.5%)-fno-tree-forwprop (4.0%)-fno-tree-tail-merge (4.0%)-fno-cprop-registers (3.2%)264 other ﬂags (32.7%)
(a) ﬀt.c
Flag (Importance)
-fno-exceptions (28.5%)-fwrapv (27.8%)-funsafe-math-optimizations (27.0%)–param=large-stack-frame=65 (1.1%)-fschedule-insns2 (1.0%)-funroll-loops (0.9%)-fno-ivopts (0.8%)–param=sccvn-max-scc-size=2995 (0.7%)–param=max-sched-extend-regions-iters=2(0.6%)
–param=slp-max-insns-in-bb=1786(0.6%)
272 other ﬂags (10.9 %)
(b) matrixmultiply.cpp
Flag (Importance)
-funsafe-math-optimizations (21.1%)-ﬃnite-math-only (9.4%)-frename-registers (3.9%)-fwhole-program (2.8%)–param=selsched-insns-to-rename=2 (2.4%)-fno-tree-dominator-opts (2.3%)–param=min-crossjump-insns=17 (2.2%)–param=max-crossjump-edges=31 (2.1%)–param=sched-state-edge-prob-cutoﬀ=17(2.0%)
–param=sms-loop-average-count-threshold=4(1.8%)
265 other ﬂags (49.9 %)
(c) raytracer.cpp
Flag (Importance)
-freorder-blocks-and-partition (2.8%)-funroll-all-loops (1.9%)–param=omega-max-geqs=64 (1.8%)–param=predictable-branch-outcome=2 (1.5%)–param=min-insn-to-prefetch-ratio=36 (1.5%)-fno-rename-registers (1.4%)–param=max-unswitch-insns=200 (1.4%)–param=omega-max-keys=2000 (1.4%)–param=max-delay-slot-live-search=83 (1.3%)–param=prefetch-latency=50 (1.3%)279 other ﬂags (83.7%)
(d) tsp ga.cpp
Figure 7: Top 10 most important GCC ﬂags for each benchmark. Importance is deﬁned as the slowdown when the ﬂag is
removed from the ﬁnal conﬁguration. Importance is normalized to sum to 100%.
 0
 0.005
 0.01
 0.015
 0.02
 0.025
 0.03
 0  100  200  300  400  500
Execution Time (seconds)
Autotuning Time (seconds)
Hand-optimized
OpenTuner
(a) Blur
 0
 0.005
 0.01
 0.015
 0.02
 0  500  1000
Execution Time (seconds)
Autotuning Time (seconds)
Hand-optimized
OpenTuner (b) Wavelet
 0
 0.2
 0.4
 0.6
 0.8
 1
 1.2
 1.4
 0  5000  10000  15000
Execution Time (seconds)
Autotuning Time (seconds)
Hand-optimized
OpenTuner (c) Bilateral
Figure 8: Halide: Execution time (lower is better) as a function of autotuning time. Aggregated performance of 30 runs of
OpenTuner, error bars indicate median, 20th, and 80th percentiles.
Halide’s model of schedules is based on deﬁning three
things for each function in a pipeline:
1. In what order pixels within the stage should be
evaluated.
2. At what granularity pixels should be interleaved
between producer and consumer stages.
3. At what granularity pixels should be stored for reuse.
In the Halide language, these choices are speciﬁed by
applying schedule operators to each function. The
composition of potentially many schedule operators for each
function together deﬁne the organization of computations
and data for a whole pipeline.
For example, the best hand-tuned schedule (against which
we compare our autotuner) for a two-stage separable blur
pipeline is written as follows:
blur_y.split(y, y, yi, 8)
.parallel(y)
.vectorize(x, 8);
blur_x.store_at(blur_y, y)
.compute_at(blur_y, yi)
.vectorize(x, 8);
blur_y(x, y) and blur_y(x, y) are the Halide functions
which make up the pipeline. The scheduling operators
available to the autotuner are the following:
•split introduces a new variable and loop nest by
adding a layer of blocking. Recursive splits make the
search space theoretically inﬁnite; we limit the number
of splits to at most 4 per dimension per function, which
is suﬃcient in practice. We represent each of these
splits as a PowerOfTwoParameter, where setting the
size of the split to 1 corresponds to not using the split
operator.
•parallel, vectorize, and unroll cause the loop nest
associated with a given variable in the function to be
executed in parallel (over multiple threads), vectorized
(e.g., with SSE), or unrolled. OpenTuner represents
each of these operators as a parameter per variable
per function (including variables introduced by splits).
The parallel operator is a BooleanParameter, while
vectorize and unroll are PowerOfTwoParameters like
splits.
•reorder / reorder_storage take a list of variables
and reorganizes the loop nest order or storage
order for those variables. We encode a single
reorder and a single reorder_storage for each
function. We represent reorder_storage as a
PermutationParameter over the original (pre-split)
variables in the function. The reorder operator for
each function is a ScheduleParameter over all the
post-split variables, with the permutation constraint
that the split children of a single variable are ordered
in a ﬁxed order relative to each other, to remove
redundancy in the choice space.
Together, these operators deﬁne the order of execution
(as a perfectly nested loop) and storage layout within
the domain of each function, independently. Finally, the
granularity of interleaving and storage between stages are
speciﬁed by two more operators:
•compute_at speciﬁes the granularity at which a given
function should be computed within its consumers.
•store_at speciﬁes the granularity at which a given
function should be stored for reuse by its consumers.
 8.6
 8.65
 8.7
 8.75
 8.8
 8.85
 8.9
 8.95
 9
 9.05
 0  200  400  600  800  1000 1200 1400 1600 1800
Execution Time (seconds)
Autotuning Time (seconds)
Vendor-optimized
OpenTuner
Figure 9: High Performance Linpack : Execution
time (lower is better) as a function of autotuning time.
Aggregated performance of 30 runs of OpenTuner, error bars
indicate median, 20th, and 80th percentiles.
Both granularity choices are speciﬁed as a level in the loop
nest of a downstream function whose scope encloses all
uses of the producer. These producer-consumer granularity
choices are the most diﬃcult parameters to encode and
search, because the valid choices for these parameters are
inter-dependent with both granularity and order choices
for all other functions in the pipeline. Na ¨ıvely encoding
these parameters without respecting this interdependence
creates numerous invalid schedules—which are meaningless
in Halide’s schedule language and rejected by the compiler—
for each valid schedule.
The result of all these operators is a schedule space that
is an exceedingly complex tree; transformations can add or
delete subtrees, for example. Searching over such a space
is far more diﬃcult when it must be encoded in simple
integer parameters, so we take advantage of OpenTuner’s
ability to encode more complex parameter types. We
represented the set of all compute and storage interleaving
choices for a whole pipeline as a single instance of a custom,
domain-speciﬁc parameter type. The parameter extends a
ScheduleParameter, encoding open and close pairs for each
scope in the scheduled pipeline’s loop nest, with a correction
step which projects any invalid permutations back into the
space of meaningful schedules.
Figure 8 presents results for three benchmarks drawn
from Halide’s standard demo applications: a two-stage
blur, an inverse Daubechies wavelet transform, and a fast
bilateral ﬁlter using the bilateral grid. For all three of
these examples OpenTuner is able to create schedules that
match or beat the best hand optimized schedules shipping
with the Halide source code. Results were collected on
three IvyBridge-generation Intel processors2 using the latest
release of Halide.
4.3 High Performance Linpack
The High Performance Linpack benchmark [10] is used
to evaluate ﬂoating point performance of machines ranging
from small multiprocessors to large-scale clusters, and is
the evaluation criterion for the Top 500 [32] supercomputer
benchmark. The benchmark measures the speed of
2Blur was tuned on a Core i5-3550, wavelet on a Core i7-
3770, and bilateral grid on a 12 cores of a dual-socket Xeon
E5-2695 v2 server.
solving a large random dense linear system of equations
using distributed memory. Achieving optimal performance
requires tuning about ﬁfteen parameters, including matrix
block sizes and algorithmic parameters. To assist in tuning,
HPL includes a built in autotuner that uses exhaustive
search over user-provided discrete values of the parameters.
We run HPL on a 2.93 GHz Intel Sandy Bridge quad-core
machine running Linux kernel 3.2.0, compiled with GCC
4.5 and using the Intel Math Kernel Library (MKL) 11.0
for optimized math operations. For comparison purposes,
we evaluate performance relative to Intel’s optimized HPL
implementation3. We encode the input tuning parameters
for HPL as na¨ıvely as possible, without using any machine-
speciﬁc knowledge. For most parameters, we utilize
EnumParameter or SwitchParameter, as they generally
represent discrete choices in the algorithm used. The major
parameter that controls performance is the blocksize of the
matrix; this we represent as an IntegerParameter to give
as much freedom as possible for the autotuner for searching.
Another major parameter controls the distribution of the
matrix onto the processors; we represent this by enumerating
all 2D decompositions possible for the number of processors
on the machine.
Figure 9 shows the results of 30 tuning runs using Open-
Tuner, compared with the vendor-provided performance.
The median performance across runs, after 1200 seconds
of autotuning, exceeds the performance of Intel’s optimized
parameters. Overall, OpenTuner obtains a best performance
of 86.5% of theoretical peak performance on this machine,
while exploring a miniscule amount of the overall search
space. Furthermore, the blocksize chosen is not a power
of two, and is generally a value unlikely to be guessed for
use in hand-tuning.
4.4 PetaBricks
PetaBricks [3] is an implicitly parallel language and
compiler which incorporates the concept of algorithmic
choice into the language. The PetaBricks language provides
a framework for the programmer to describe multiple
ways of solving a problem while allowing the autotuner to
determine which combination of ways is best for the user’s
situation. The search space of PetaBricks programs is both
over low level optimizations and over diﬀerent algorithms.
The autotuner is used to synthesize poly-algorithms which
weave many individual algorithms together by switching
dynamically between them at recursive call sites.
The primary components in the search space for PetaBricks
programs are algorithmic selectors which are used to
synthesize instances of poly-algorithms from algorithmic
choices in the PetaBricks language. A selector is used to
map input sizes to algorithmic choices, and is represented
by a list of cutoﬀs and choices. As an example,
the selector [InsertionSort , 500, QuickSort, 1000, MergeSort]
would correspond to synthesizing the function:
void Sort(List& list) {
if(list.length < 500)
InsertionSort(list);
else if( list.length < 1000)
QuickSort(list);
else
MergeSort(list);
}
3Available at http://software.intel.com/en-us/
articles/intel-math-kernel-library-linpack-download .
 0
 0.02
 0.04
 0.06
 0.08
 0.1
 0.12
 0.14
 0.16
 0.18
 0.2
 0  600  1200  1800  2400  3000  3600
Execution Time (seconds)
Autotuning Time (seconds)
PetaBricks Autotuner
OpenTuner
(a) Poisson
 0
 0.02
 0.04
 0.06
 0.08
 0.1
 0  600  1200
Execution Time (seconds)
Autotuning Time (seconds)
PetaBricks Autotuner
OpenTuner (b) Sort
 0
 0.05
 0.1
 0.15
 0.2
 0  600  1200
Execution Time (seconds)
Autotuning Time (seconds)
PetaBricks Autotuner
OpenTuner
(c) Strassen
 0.0095
 0.01
 0.0105
 0.011
 0.0115
 0  600  1200
Execution Time (seconds)
Autotuning Time (seconds)
PetaBricks Autotuner
OpenTuner (d) Tridiagonal Solver
Figure 10: PetaBricks: Execution time (lower is better) as a function of autotuning time. Aggregated performance of 30
runs of OpenTuner, error bars indicate median, 20th, and 80th percentiles.
where QuickSort and MergeSort recursively call Sort so
the program dynamically switches between sorting methods
as recursive calls are made on smaller and smaller lists.
We used the general SelectorParameter to represent this
choice type, which internally keeps track of the order of the
algorithmic choices and the cutoﬀs. PetaBricks programs
contain many algorithmic selectors and a large number of
other parameter types, such as block sizes, thread counts,
iteration counts, and program speciﬁc parameters.
Results using OpenTuner compared to the built-in
PetaBricks autotuner are shown in Figure 10. The
PetaBricks autotuner uses a diﬀerent strategy that starts
with tests on very small problem inputs and incrementally
works up to full sized inputs [4]. In all cases, the autotuners
arrive at similar solutions, and for Strassen, the exact same
solution. For Sort and Tridiagonal Solver, OpenTuner
beats the native PetaBricks autotuner, while for Poisson
the PetaBricks autotuner arrives at a better solution, but
has much higher variance.
The Poisson equation solver (Figure 10a) presents the
most diﬃcult search space. The search space for Poisson
in PetaBricks is described in detail in [7]. It is a variable
accuracy benchmark where the goal of the autotuner is
to ﬁnd a solution that provides 8-digits of accuracy while
minimizing time. All points in Figure 10a satisfy the
accuracy target, so we do not display accuracy. Open-
Tuner uses the ThresholdAccuracyMinimizeTime objective
described in Section 3.4. The Poisson search space selects
between direct solvers, iterative solvers, and multigrid
solvers where the shape of the multigrid V-cycle/W-cycle
is deﬁned by the autotuner. The optimal solution is a
poly-algorithm composed of multigrid W-cycles. However,
it is diﬃcult to obtain 8 digits of accuracy with randomly
generated multigrid cycle shapes, though it is easy with
a direct solver (which solves the problem exactly). This
creates a large “plateau” which is diﬃcult for the autotuners
to improve upon, and is shown near 0.16. The native
PetaBricks autotuner is less aﬀected by this plateau because
it constructs algorithms incrementally bottom up; however
the use of these smaller input sizes causes larger variance as
mistakes early on get ampliﬁed.
4.5 Stencil
In [20], the authors describe a generalized system for
autotuning memory-bound stencil computations on modern
multicore machines and GPUs. By composing domain-
speciﬁc transformations, the authors explore a large space
of implementations for each kernel; the original autotuning
methodology involves exhaustive search over thousands of
implementations for each kernel.
We obtained the raw execution data, courtesy of the
authors, and use OpenTuner instead of exhaustive search
on the data from a Nehalem-class 2.66 GHz Intel Xeon
machine, running Linux 2.6. We compare against the
optimal performance obtained by the original autotuning
 2e+07
 4e+07
 6e+07
 8e+07
 1e+08
 1.2e+08
 1.4e+08
 1.6e+08
 1.8e+08
 2e+08
 0  10  20  30  40  50  60
Execution Time (cycles)
Tests
Optimal
OpenTuner
(a) Laplacian
 1e+08
 2e+08
 3e+08
 4e+08
 5e+08
 6e+08
 0  10  20  30  40  50  60
Execution Time (cycles)
Tests
Optimal
OpenTuner
(b) Divergence
Figure 11: Stencil: Execution time (lower is better) as
a function of tests. Aggregated performance of 30 runs
of OpenTuner, error bars indicate median, 20th, and 80th
percentiles.
system through exhaustive search. The search space for
this problem involves searching for parameters for the
parallel decomposition, cache and thread blocking, and loop
unrolling for each kernel; to limit the impact of control
ﬂow and cache misalignment, these parameters depend on
one another (for example, the loop unrolling will be a
small integer divisor of the thread blocking). We encode
these parameters as PowerOfTwoParameters but ensure that
invalid combinations are discarded.
Figure 11 shows the results of using OpenTuner for the
Laplacian and divergence kernel benchmarks, showing the
median performance obtained over 30 trials as a function
of the number of tests (results for the gradient kernel are
similar, so we omit them for brevity). OpenTuner is able
to obtain peak performance on Laplacian after less than 35
tests of candidate implementations and 25 implementations
for divergence; thus, using OpenTuner, less than 2% of
the search space needs to be explored to reach optimal
performance. These results show that even for problems
where exhaustive search is tractable (though it may take
days), OpenTuner can drastically improve convergence to
the optimal performance with little programmer eﬀort.
4.6 Unitary Matrices
As a somewhat diﬀerent example, we use OpenTuner in an
example from physics, namely the quantum control problem
of synthesizing unitary matrices in SU (2) in optimal time,
 0.99
 0.992
 0.994
 0.996
 0.998
 1
 0  10  20  30  40  50  60
Accuracy
Autotuning Time (seconds)
OpenTuner (easy problem instances)
OpenTuner (hard problem instances)
Figure 12: Unitary: Accuracy (higher is better) as a
function of autotuning time. Aggregated performance of 30
runs of OpenTuner, error bars indicate median, 20th, and
80th percentiles.
using a ﬁnite control set composed of rotations around two
non-parallel axes. (Such rotations generate the complete
space SU (2).)
Unlike other examples, which use OpenTuner as a
traditional autotuner to optimize a program, the Unitary
example uses OpenTuner to perform a search over the
problem space as a subroutine at runtime in a program.
The problem has a ﬁxed set of operators (or controls),
represented as matrices, and the goal is to ﬁnd a sequence of
operators that, when multiplied together, produce a given
target matrix. The objective function is an accuracy value
deﬁned as a function of the distance of the product of the
sequence to the goal (also called the trace ﬁdelity).
Figure 12 shows the performance of the Unitary example
on both easy and hard instances of the problem. For both
types of problem instance OpenTuner is able to meet the
accuracy target within the ﬁrst few seconds. This is example
shows that OpenTuner can be used for more types of search
problems than just program autotuning.
4.7 Mario
To demonstrate OpenTuner’s versatility, we used the
system to learn to play Super Mario Bros., a Nintendo
Entertainment System game in which the player, as Mario,
runs to the right, jumping over pits and onto enemies’
heads to reach the ﬂagpole at the end of each level. We
use OpenTuner to search for a sequence of button presses
that completes the game’s ﬁrst level. The button presses
are written to a ﬁle and played back in the NES emulator
FCEUX. The emulator is conﬁgured to evaluate candidate
input sequences faster than real time. OpenTuner evaluates
96 input sequences in parallel in diﬀerent instances of the
emulator. A Lua script running in the emulator detects
when Mario dies or reaches the ﬂagpole by monitoring a
location in the emulated memory. At that time it reports the
number of pixels Mario has moved to the right, which is the
objective function OpenTuner tries to maximize. The game
is deterministic, so only one trial is necessary to evaluate this
function. OpenTuner does not interpret the game’s graphics
or read the emulated memory; the objective function’s value
is the only information OpenTuner receives. Horizontal
movement is controlled by many EnumParameters choosing
 1000
 1500
 2000
 2500
 3000
 3500
 0  60  120  180  240  300
Pixels Moved Right (Progress)
Autotuning Time (seconds)
Win Level
OpenTuner
Figure 13: Mario: Pixels Mario moves to the right (higher
is better) as a function of autotuning time. Reaching
pixel 3161 (the ﬂagpole) completes the level. Aggregated
performance of 30 runs of OpenTuner, error bars indicate
median, 20th, and 80th percentiles.
to run or walk left or right or to not move, each paired with
an IntParameter specifying the duration of the movement
in frames. As the goal is to move right, the search space is
given a 3 to 1 right-moving bias. Jumping is controlled by
IntParameters specifying on which frames to jump and how
long to hold the jump button.
Figure 13 shows the performance of the Mario example.
OpenTuner rapidly reaches 1500 pixels because many initial
(mostly random) conﬁgurations will make it that far in the
level. The bottlenecks evident in the ﬁgure correspond
to pits which Mario must jump over, which requires
coordinating all four kinds of parameter: Mario must jump
on the correct frame for a long enough duration, having
built up speed from walking or running right for a long
enough duration, and then must not return left into the
pit. Performance is aided by a“ratcheting” eﬀect: due to the
NES’s memory limits, portions of the level that have scrolled
oﬀ the screen are quickly overwritten with new portions, so
the game only allows Mario to move left one screen (up
to 256 pixels) from his furthest-right position. The ﬁnal
sequences complete the level after between 3500 and 5000
input actions.
5. CONCLUSIONS
We have shown OpenTuner, a new framework for building
domain-speciﬁc multi-objective program autotuners. Open-
Tuner supports fully customizable conﬁguration represen-
tations and an extensible technique representation to allow
for domain-speciﬁc techniques. OpenTuner introduces the
concept of ensembles of search techniques in autotuning,
which allow many search techniques to work together to ﬁnd
an optimal solution and provides a more robust search than
a single technique alone.
While implementing these seven autotuners in Open-
Tuner, the biggest lesson we learned reinforced a core
message of this paper of the need for domain-speciﬁc
representations and domain-speciﬁc search techniques in
autotuning. As an example, the initial version of the
PetaBricks autotuner we implemented just used a point in
high dimensional space as the conﬁguration representation.
This generic mapping of the search space did not work at
all. It produced ﬁnal conﬁgurations an order of magnitude
slower than the results presented from our autotuner that
uses selector parameter types. Similarly, Halide’s search
space strongly motivates domain speciﬁc techniques that
make large coordinated jumps, for example, swapping
scheduling operators on x and y across all functions
in the program. We were able to add domain-speciﬁc
representations and techniques to OpenTuner at a fraction
of the time and eﬀort of building a fully custom system for
that project. OpenTuner was able to seamlessly integrate
the techniques with its ensemble approach.
OpenTuner is free and open source4 and as the community
adds more techniques and representations to this ﬂexible
framework, there will be less of a need to create a new
representation or techniques for each project and we hope
that the system will work out-of-the-box for most creators
of autotuners. OpenTuner pushes the state of the art
forward in program autotuning in a way that can easily
be adopted by other projects. We hope that OpenTuner
will be an enabling technology that will allow the expanded
use of program autotuning both to more domains and
by expanding the role of program autotuning in existing
domains.
6. ACKNOWLEDGEMENTS
We would like to thank Clarice Aiello for contributing the
Unitary benchmark program. We gratefully acknowledge
Connelly Barnes for helpful discussions and bug ﬁxes related
to autotuning the Halide project.
This work is partially supported by DOE award DE-
SC0005288 and DOD DARPA award HR0011-10-9-0009.
This research used resources of the National Energy
Research Scientiﬁc Computing Center, which is supported
by the Oﬃce of Science of the U.S. Department of Energy
under Contract No. DE-AC02-05CH11231.
7. REFERENCES
[1] F. Agakov, E. Bonilla, J. Cavazos, B. Franke,
G. Fursin, M. F. P. O’boyle, J. Thomson,
M. Toussaint, and C. K. I. Williams, “Using machine
learning to focus iterative optimization,” in CGO’06,
2006, pp. 295–305.
[2] L. Almagor, K. D. Cooper, A. Grosul, T. J. Harvey,
S. W. Reeves, D. Subramanian, L. Torczon, and
T. Waterman, “Finding eﬀective compilation
sequences.” in LCTES’04, 2004, pp. 231–239.
[3] J. Ansel, C. Chan, Y. L. Wong, M. Olszewski,
Q. Zhao, A. Edelman, and S. Amarasinghe,
“PetaBricks: A language and compiler for algorithmic
choice,” in PLDI, Dublin, Ireland, Jun 2009.
[4] J. Ansel, M. Pacula, S. Amarasinghe, and U.-M.
O’Reilly, “An eﬃcient evolutionary algorithm for
solving bottom up problems,” in Annual Conference
on Genetic and Evolutionary Computation , Dublin,
Ireland, July 2011.
[5] W. Baek and T. Chilimbi, “Green: A framework for
supporting energy-conscious programming using
controlled approximation,” in PLDI, June 2010.
[6] V. Bhat, M. Parashar, . Hua Liu, M. Khandekar,
N. Kandasamy, and S. Abdelwahed, “Enabling
self-managing applications using model-based online
4Available from http://opentuner.org/
control strategies,” in International Conference on
Autonomic Computing, Washington, DC, 2006.
[7] C. Chan, J. Ansel, Y. L. Wong, S. Amarasinghe, and
A. Edelman, “Autotuning multigrid with PetaBricks,”
in Supercomputing, Portland, OR, Nov 2009.
[8] F. Chang and V. Karamcheti, “A framework for
automatic adaptation of tunable distributed
applications,” Cluster Computing, vol. 4, March 2001.
[9] M. Christen, O. Schenk, and H. Burkhart, “Patus: A
code generation and autotuning framework for parallel
iterative stencil computations on modern
microarchitectures.” in IPDPS. IEEE, 2011.
[10] J. J. Dongarra, P. Luszczek, and A. Petitet, “The
LINPACK Benchmark: past, present and future,”
Concurrency and Computation: Practice and
Experience, vol. 15, no. 9, pp. 803–820, 2003.
[11] X. Fan, “Optimize your code: Matrix multiplication,”
https://tinyurl.com/kuvzbp9, 2009.
[12] A. Fialho, L. Da Costa, M. Schoenauer, and M. Sebag,
“Analyzing bandit-based adaptive operator selection
mechanisms,” Annals of Mathematics and Artiﬁcial
Intelligence – Special Issue on Learning and Intelligent
Optimization, 2010.
[13] A. Fialho, R. Ros, M. Schoenauer, and M. Sebag,
“Comparison-based adaptive strategy selection with
bandits in diﬀerential evolution,” in PPSN’10, ser.
LNCS, R. S. et al., Ed., vol. 6238. Springer,
September 2010.
[14] M. Frigo and S. G. Johnson, “The design and
implementation of FFTW3,” IEEE, vol. 93, no. 2,
February 2005.
[15] G. Fursin, C. Miranda, O. Temam, M. Namolaru,
E. Yom-Tov, A. Zaks, B. Mendelson, E. Bonilla,
J. Thomson, H. Leather, C. Williams, M. O’Boyle,
P. Barnard, E. Ashton, E. Courtois, and F. Bodin,
“MILEPOST GCC: machine learning based research
compiler,” in Proceedings of the GCC Developers’
Summit, Jul 2008.
[16] H. Hoﬀmann, J. Eastep, M. D. Santambrogio, J. E.
Miller, and A. Agarwal, “Application heartbeats: a
generic interface for specifying program performance
and goals in autonomous computing environments,” in
ICAC, New York, NY, 2010.
[17] H. Hoﬀmann, S. Misailovic, S. Sidiroglou, A. Agarwal,
and M. Rinard, “Using code perforation to improve
performance, reduce energy consumption, and respond
to failures,” Massachusetts Institute of Technology,
Tech. Rep. MIT-CSAIL-TR-2209-042, Sep 2009.
[18] H. Hoﬀmann, S. Sidiroglou, M. Carbin, S. Misailovic,
A. Agarwal, and M. Rinard, “Power-aware computing
with dynamic knobs,” in ASPLOS, 2011.
[19] H. Jordan, P. Thoman, J. J. Durillo, S. Pellegrini,
P. Gschwandtner, T. Fahringer, and H. Moritsch, “A
multi-objective auto-tuning framework for parallel
codes,” in Proceedings of the International Conference
on High Performance Computing, Networking, Storage
and Analysis, ser. SC ’12, 2012.
[20] S. Kamil, C. Chan, L. Oliker, J. Shalf, and
S. Williams, “An auto-tuning framework for parallel
multicore stencil computations,” in IPDPS’10, 2010,
pp. 1–12.
[21] S. A. Kamil, “Productive high performance parallel
programming with auto-tuned domain-speciﬁc
embedded languages,” Ph.D. dissertation, EECS
Department, University of California, Berkeley, Jan
2013.
[22] G. Karsai, A. Ledeczi, J. Sztipanovits, G. Peceli,
G. Simon, and T. Kovacshazy, “An approach to
self-adaptive software based on supervisory control,”
in International Workshop in Self-adaptive software ,
2001.
[23] T. Murphy VII, “The ﬁrst level of Super Mario Bros.
is easy with lexicographic orderings and time travel,”
April 2013.
[24] K. Nordkvist, “Solving TSP with a genetic algorithm
in C++,” https://tinyurl.com/lq3uqlh, 2012.
[25] M. Pacula, J. Ansel, S. Amarasinghe, and U.-M.
O’Reilly, “Hyperparameter tuning in bandit-based
adaptive operator selection,” in European Conference
on the Applications of Evolutionary Computation ,
Malaga, Spain, Apr 2012.
[26] E. Park, L.-N. Pouche, J. Cavazos, A. Cohen, and
P. Sadayappan, “Predictive modeling in a polyhedral
optimization space,” in CGO’11, April 2011, pp. 119
–129.
[27] S. Pixel, “3D Basic Lessons: Writing a simple
raytracer,” https://tinyurl.com/lp8ncnw, 2012.
[28] M. P ¨uschel, J. M. F. Moura, B. Singer, J. Xiong, J. R.
Johnson, D. A. Padua, M. M. Veloso, and R. W.
Johnson, “Spiral: A generator for platform-adapted
libraries of signal processing alogorithms,” IJHPCA,
vol. 18, no. 1, 2004.
[29] J. Ragan-Kelley, A. Adams, S. Paris, M. Levoy,
S. Amarasinghe, and F. Durand, “Decoupling
algorithms from schedules for easy optimization of
image processing pipelines,” ACM Trans. Graph.,
vol. 31, no. 4, pp. 32:1–32:12, Jul. 2012.
[30] J. Ragan-Kelley, C. Barnes, A. Adams, S. Paris,
F. Durand, and S. Amarasinghe, “Halide: a language
and compiler for optimizing parallelism, locality, and
recomputation in image processing pipelines,” in
Proceedings of the 34th ACM SIGPLAN conference on
Programming language design and implementation ,
ser. PLDI ’13. New York, NY, USA: ACM, 2013, pp.
519–530.
[31] C. Tapus, I.-H. Chung, and J. K. Hollingsworth,
“Active harmony: Towards automated performance
tuning,” in In Proceedings from the Conference on
High Performance Networking and Computing , 2003.
[32] Top500, “Top 500 supercomputer sites,”
http://www.top500.org/, 2010.
[33] R. Vuduc, J. W. Demmel, and K. A. Yelick, “OSKI: A
library of automatically tuned sparse matrix kernels,”
in Scientiﬁc Discovery through Advanced Computing
Conference, San Francisco, CA, June 2005.
[34] R. C. Whaley and J. J. Dongarra, “Automatically
tuned linear algebra software,” in Supercomputing,
Washington, DC, 1998.
[35] S. Woo, M. Ohara, E. Torrie, J. Singh, and A. Gupta,
“The SPLASH-2 programs: characterization and
methodological considerations,” in Symposium on
Computer Architecture News, June 1995.

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536–1547
November 16 - 20, 2020.c⃝2020 Association for Computational Linguistics
1536
CodeBERT:
A Pre-Trained Model for Programming and Natural Languages
Zhangyin Feng1∗, Daya Guo2∗, Duyu Tang3, Nan Duan3, Xiaocheng Feng1
Ming Gong4, Linjun Shou4, Bing Qin1, Ting Liu1, Daxin Jiang4, Ming Zhou3
1 Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology, China
2 The School of Data and Computer Science, Sun Yat-sen University, China
3 Microsoft Research Asia, Beijing, China
4 Microsoft Search Technology Center Asia, Beijing, China
{zyfeng,xcfeng,qinb,tliu}@ir.hit.edu.cn
guody5@mail2.sysu.edu.cn
{dutang,nanduan,migon,lisho,djiang,mingzhou}@microsoft.com
Abstract
We present CodeBERT, a bimodal pre-trained
model for programming language (PL) and
natural language (NL). CodeBERT learns
general-purpose representations that support
downstream NL-PL applications such as nat-
ural language code search, code documen-
tation generation, etc. We develop Code-
BERT with Transformer-based neural architec-
ture, and train it with a hybrid objective func-
tion that incorporates the pre-training task of
replaced token detection, which is to detect
plausible alternatives sampled from generators.
This enables us to utilize both “ bimodal” data
of NL-PL pairs and “ unimodal” data, where
the former provides input tokens for model
training while the latter helps to learn bet-
ter generators. We evaluate CodeBERT on
two NL-PL applications by ﬁne-tuning model
parameters. Results show that CodeBERT
achieves state-of-the-art performance on both
natural language code search and code docu-
mentation generation. Furthermore, to inves-
tigate what type of knowledge is learned in
CodeBERT, we construct a dataset for NL-PL
probing, and evaluate in a zero-shot setting
where parameters of pre-trained models are
ﬁxed. Results show that CodeBERT performs
better than previous pre-trained models on NL-
PL probing.1
1 Introduction
Large pre-trained models such as ELMo (Peters
et al., 2018), GPT (Radford et al., 2018), BERT
(Devlin et al., 2018), XLNet (Yang et al., 2019)
∗Work done while this author was an intern at Microsoft
Research Asia.
1All the codes and data are available at https://
github.com/microsoft/CodeBERT
and RoBERTa (Liu et al., 2019) have dramati-
cally improved the state-of-the-art on a variety of
natural language processing (NLP) tasks. These
pre-trained models learn effective contextual repre-
sentations from massive unlabeled text optimized
by self-supervised objectives, such as masked
language modeling, which predicts the original
masked word from an artiﬁcially masked input
sequence. The success of pre-trained models in
NLP also drives a surge of multi-modal pre-trained
models, such as ViLBERT (Lu et al., 2019) for
language-image and VideoBERT (Sun et al., 2019)
for language-video, which are learned from bi-
modal data such as language-image pairs with bi-
modal self-supervised objectives.
In this work, we present CodeBERT, abimodal
pre-trained model for natural language (NL) and
programming language (PL) like Python, Java,
JavaScript, etc. CodeBERT captures the seman-
tic connection between natural language and pro-
gramming language, and produces general-purpose
representations that can broadly support NL-PL
understanding tasks (e.g. natural language code
search) and generation tasks (e.g. code documen-
tation generation). It is developed with the multi-
layer Transformer (Vaswani et al., 2017), which is
adopted in a majority of large pre-trained models.
In order to make use of both bimodal instances
of NL-PL pairs and large amount of available uni-
modal codes, we train CodeBERT with a hybrid
objective function, including standard masked lan-
guage modeling (Devlin et al., 2018) and replaced
token detection (Clark et al., 2020), where uni-
modal codes help to learn better generators for
producing better alternative tokens for the latter
objective.
We train CodeBERT from Github code reposito-
1537
ries in 6 programming languages, where bimodal
datapoints are codes that pair with function-level
natural language documentations (Husain et al.,
2019). Training is conducted in a setting similar
to that of multilingual BERT (Pires et al., 2019),
in which case one pre-trained model is learned for
6 programming languages with no explicit mark-
ers used to denote the input programming lan-
guage. We evaluate CodeBERT on two down-
stream NL-PL tasks, including natural language
code search and code documentation generation.
Results show that ﬁne-tuning the parameters of
CodeBERT achieves state-of-the-art performance
on both tasks. To further investigate what type of
knowledge is learned in CodeBERT, we construct
a dataset for NL-PL probing, and test CodeBERT
in a zero-shot scenario, i.e. without ﬁne-tuning the
parameters of CodeBERT. We ﬁnd that CodeBERT
consistently outperforms RoBERTa, a purely natu-
ral language-based pre-trained model. The contri-
butions of this work are as follows:
• CodeBERT is the ﬁrst large NL-PL pre-
trained model for multiple programming lan-
guages.
• Empirical results show that CodeBERT is ef-
fective in both code search and code-to-text
generation tasks.
• We further created a dataset which is the ﬁrst
one to investigate the probing ability of the
code-based pre-trained models.
2 Background
2.1 Pre-Trained Models in NLP
Large pre-trained models (Peters et al., 2018; Rad-
ford et al., 2018; Devlin et al., 2018; Yang et al.,
2019; Liu et al., 2019; Raffel et al., 2019) have
brought dramatic empirical improvements on al-
most every NLP task in the past few years. Suc-
cessful approaches train deep neural networks on
large-scale plain texts with self-supervised learning
objectives. One of the most representative neural
architectures is the Transformer (Vaswani et al.,
2017), which is also the one used in this work. It
contains multiple self-attention layers, and can be
conventionally learned with gradient decent in an
end-to-end manner as every component is differen-
tiable. The terminology “self-supervised” means
that supervisions used for pre-training are auto-
matically collected from raw data without manual
annotation. Dominant learning objectives are lan-
guage modeling and its variations. For example,
in GPT (Radford et al., 2018), the learning objec-
tive is language modeling, namely predicting the
next word wk given the preceding context words
{w1,w2,...,w k−1}. As the ultimate goal of pre-
training is not to train a good language model, it is
desirable to consider both preceding and following
contexts to learn better general-purpose contextual
representations. This leads us to the masked lan-
guage modeling objective used in BERT (Devlin
et al., 2018), which learns to predict the masked
words of a randomly masked word sequence given
surrounding contexts. Masked language modeling
is also used as one of the two learning objectives
for training CodeBERT.
2.2 Multi-Modal Pre-Trained Models
The remarkable success of the pre-trained model
in NLP has driven the development of multi-modal
pre-trained model that learns implicit alignment
between inputs of different modalities. These mod-
els are typically learned from bimodal data, such
as pairs of language-image or pairs of language-
video. For example, ViLBERT (Lu et al., 2019)
learns from image caption data, where the model
learns by reconstructing categories of masked im-
age region or masked words given the observed
inputs, and meanwhile predicting whether the cap-
tion describes the image content or not. Simi-
larly, VideoBERT (Sun et al., 2019) learns from
language-video data and is trained by video and
text masked token prediction. Our work belongs
to this line of research as we regard NL and PL
as different modalities. Our method differs from
previous works in that the fuels for model train-
ing include not only bimodal data of NL-PL pairs,
but larger amounts of unimodal data such as codes
without paired documentations.
A concurrent work (Kanade et al., 2019) uses
masked language modeling and next sentence pre-
diction as the objective to train a BERT model on
Python source codes, where a sentence is a log-
ical code line as deﬁned by the Python standard.
In terms of the pre-training process, CodeBERT
differs from their work in that (1) CodeBERT is
trained in a cross-modal style and leverages both
bimodal NL-PL data and unimodal PL/NL data, (2)
CodeBERT is pre-trained over six programming
languages, and (3) CodeBERT is trained with a
new learning objective based on replaced token
1538
detection.
3 CodeBERT
We describe the details about CodeBERT in this
section, including the model architecture, the input
and output representations, the objectives and data
used for training CodeBERT, and how to ﬁne-tune
CodeBERT when it is applied to downstream tasks.
3.1 Model Architecture
We follow BERT (Devlin et al., 2018) and
RoBERTa (Liu et al., 2019), and use multi-layer
bidirectional Transformer (Vaswani et al., 2017) as
the model architecture of CodeBERT. We will not
review the ubiquitous Transformer architecture in
detail. We develop CodeBERT by using exactly the
same model architecture as RoBERTa-base. The
total number of model parameters is 125M.
3.2 Input/Output Representations
In the pre-training phase, we set the input as the
concatenation of two segments with a special sepa-
rator token, namely [CLS],w1,w2,..wn,[SEP],
c1,c2,...,c m,[EOS]. One segment is natural lan-
guage text, and another is code from a certain pro-
gramming language. [CLS] is a special token in
front of the two segments, whose ﬁnal hidden repre-
sentation is considered as the aggregated sequence
representation for classiﬁcation or ranking. Follow-
ing the standard way of processing text in Trans-
former, we regard a natural language text as a se-
quence of words, and split it as WordPiece (Wu
et al., 2016). We regard a piece of code as a se-
quence of tokens.
The output of CodeBERT includes (1) contextual
vector representation of each token, for both natural
language and code, and (2) the representation of
[CLS], which works as the aggregated sequence
representation.
3.3 Pre-Training Data
We train CodeBERT with bothbimodal data, which
refers to parallel data of natural language-code
pairs, and unimodal data, which stands for codes
without paired natural language texts and natural
language without paired codes.
We use datapoints from Github repositories,
where each bimodal datapoint is an individual
function with paired documentation, and each uni-
modal code is a function without paired documen-
tation. Speciﬁcally, we use a recent large dataset
TRAINING DATA bimodal DATA unimodal CODES
GO 319,256 726,768
JAVA 500,754 1,569,889
JAVASCRIPT 143,252 1,857,835
PHP 662,907 977,821
PYTHON 458,219 1,156,085
RUBY 52,905 164,048
ALL 2,137,293 6,452,446
Table 1: Statistics of the dataset used for training Code-
BERT.
provided by Husain et al. (2019), which includes
2.1M bimodal datapoints and 6.4Munimodal codes
across six programming languages (Python, Java,
JavaScript, PHP, Ruby, and Go). Data statistics is
shown in Table 1.2
The data comes from publicly available open-
source non-fork GitHub repositories and are ﬁl-
tered with a set of constraints and rules. For ex-
ample, (1) each project should be used by at least
one other project, (2) each documentation is trun-
cated to the ﬁrst paragraph, (3) documentations
shorter than three tokens are removed, (4) func-
tions shorter than three lines are removed, and (5)
function names with substring “test” are removed.
An example of the data is given in Figure 1 3.
Figure 1: An example of the NL-PL pair, where NL is
the ﬁrst paragraph (ﬁlled in red) from the documenta-
tion (dashed line in black) of a function.
3.4 Pre-Training CodeBERT
We describe the two objectives used for training
CodeBERT here. The ﬁrst objective is masked
language modeling (MLM), which has proven ef-
fective in literature (Devlin et al., 2018; Liu et al.,
2Since we will evaluate on the natural language code
search task, we only use the training data of Husain et al.
(2019) to train CodeBERT with no access to the dev and test-
ing data.
3The source of the illustrating example comes from
https://github.com/apache/spark/blob/
618d6bff71073c8c93501ab7392c3cc579730f0b/
python/pyspark/rdd.py#L125-L138
1539
6
NL Generator
NL-Code 
Discriminator
[𝑀𝐴𝑆𝐾]𝑤
CodeBERT V2:
A Pre-trained Model for NL-Code Understanding and Generation 
𝑤2
𝑤3
𝑤4
[𝑀𝐴𝑆𝐾]𝑤
𝑤1
𝑤2
𝑤3
𝑤4
𝑤5
[𝑀𝐴𝑆𝐾]𝑐
𝑐5
𝑐3
𝑐4
[𝑀𝐴𝑆𝐾]𝑐
𝑐1
𝑐2
𝑐3
𝑐4
𝑐5
𝑐6
𝑐1
𝑤51
𝑤2
𝑤3
𝑤4
𝑤5
𝑐29
𝑐5
𝑐3
𝑐4
𝑐162
𝑐1
original
replaced
Code Generator
original
original
original
replaced
original
original
original
original
replaced
sample
sample
sample
sample
Figure 2: An illustration about the replaced token detection objective. Both NL and code generators are language
models, which generate plausible tokens for masked positions based on surrounding contexts. NL-Code discrimi-
nator is the targeted pre-trained model, which is trained via detecting plausible alternatives tokens sampled from
NL and PL generators. NL-Code discriminator is used for producing general-purpose representations in the ﬁne-
tuning step. Both NL and code generators are thrown out in the ﬁne-tuning step.
2019; Sun et al., 2019). We apply masked language
modeling on bimodal data of NL-PL pairs. The sec-
ond objective is replaced token detection (RTD),
which further uses a large amount ofunimodal data,
such as codes without paired natural language texts.
Detailed hyper-parameters for model pre-training
are given in Appendix B.1.
Objective #1: Masked Language Modeling
(MLM) Given a datapoint of NL-PL pair (x =
{w, c}) as input, where w is a sequence of NL
words and c is a sequence of PL tokens, we ﬁrst
select a random set of positions for both NL and PL
to mask out (i.e. mw and mc, respectively), and
then replace the selected positions with a special
[MASK] token. Following Devlin et al. (2018),
15% of the tokens from x are masked out.
mw
i ∼unif{1,|w|}for i= 1to |w| (1)
mc
i ∼unif{1,|c|}for i= 1to |c| (2)
wmasked = REPLACE(w,mw,[MASK]) (3)
cmasked = REPLACE(c,mc,[MASK]) (4)
x = w + c (5)
The MLM objective is to predict the original to-
kens which are masked out, formulated as follows,
where pD1 is the discriminator which predicts a
token from a large vocabulary.
LMLM(θ)=
∑
i∈mw∪mc
−log pD1 (xi|wmasked,cmasked)
(6)
Objective #2: Replaced Token Detection (RTD)
In the MLM objective, only bimodal data (i.e. data-
points of NL-PL pairs) is used for training. Here we
present the objective of replaced token detection.
The RTD objective (Clark et al., 2020) is origi-
nally developed for efﬁciently learning pre-trained
model for natural language. We adapt it in our sce-
nario, with the advantage of using both bimodal
and unimodal data for training. Speciﬁcally, there
are two data generators here, an NL generator pGw
and a PL generator pGc , both for generating plau-
sible alternatives for the set of randomly masked
positions.
ˆwi ∼pGw (wi|wmasked) for i∈mw (7)
ˆci ∼pGc (ci|cmasked) for i∈mc (8)
wcorrupt = REPLACE(w,mw, ˆw) (9)
ccorrupt = REPLACE(c,mc,ˆc) (10)
xcorrupt = wcorrupt + ccorrupt (11)
The discriminator is trained to determine whether
a word is the original one or not, which is a binary
classiﬁcation problem. It is worth noting that the
RTD objective is applied to every position in the
input, and it differs from GAN (generative adver-
sarial network) in that if a generator happens to
produce the correct token, the label of that token
is “real” instead of “fake” (Clark et al., 2020). The
loss function of RTD with regard to the discrimina-
tor parameterized by θis given below, whereδ(i) is
1540
an indicator function and pD2 is the discriminator
that predicts the probability of the i-th word being
original.
LRTD(θ) =
|w|+|c|∑
i=1
(
δ(i)log pD2 (xcorrupt,i)+
(
1 −δ(i)
)(
1 −log pD2 (xcorrupt,i)
))
(12)
δ(i) =
{
1, if xcorrupt
i = xi.
0, otherwise. (13)
There are many different ways to implement the
generators. In this work, we implement two ef-
ﬁcient n-gram language models (Jurafsky, 2000)
with bidirectional contexts, one for NL and one
for PL, and learn them from corresponding uni-
model datapoints, respectively. The approach is
easily generalized to learn bimodal generators or
use more complicated generators like Transformer-
based neural architecture learned in a joint manner.
We leave these to future work. The PL training data
is the unimodal codes as shown in Table 1, and the
NL training data comes from the documentations
from bimodal data. One could easily extend these
two training datasets to larger amount. The ﬁnal
loss function are given below.
min
θ
LMLM(θ) +LRTD(θ) (14)
3.5 Fine-Tuning CodeBERT
We have different settings to use CodeBERT in
downstream NL-PL tasks. For example, in natural
language code search, we feed the input as the
same way as the pre-training phase and use the
representation of [CLS] to measure the semantic
relevance between code and natural language query,
while in code-to-text generation, we use an encoder-
decoder framework and initialize the encoder of
a generative model with CodeBERT. Details are
given in the experiment section.
4 Experiment
We present empirical results in this section to verify
the effectiveness of CodeBERT. We ﬁrst describe
the use of CodeBERT in natural language code
search (§4.1), in a way that model parameters of
CodeBERT are ﬁne-tuned. After that, we present
the NL-PL probing task (§4.2), and evaluate Code-
BERT in a zero-shot setting where the parameters
of CodeBERT are ﬁxed. Finally, we evaluate Code-
BERT on a generation problem, i.e. code documen-
tation generation (§4.3), and further evaluate on a
programming language which is never seen in the
training phase (§4.4).
4.1 Natural Language Code Search
Given a natural language as the input, the objec-
tive of code search is to ﬁnd the most semantically
related code from a collection of codes. We con-
duct experiments on the CodeSearchNet corpus
(Husain et al., 2019) 4. We follow the ofﬁcial evalu-
ation metric to calculate the Mean Reciprocal Rank
(MRR) for each pair of test data (c, w) over a ﬁxed
set of 999 distractor codes. We further calculate the
macro-average MRR for all languages as an overall
evaluation metric. It is helpful to note that this met-
ric differs from the AVG metric in the original pa-
per, where the answer is retrieved from candidates
from all six languages. We ﬁne-tune a language-
speciﬁc model for each programming language 5.
We train each model with a binary classiﬁcation
loss function, where a softmax layer is connected
to the representation of [CLS]. Both training and
validation datasets are created in a way that posi-
tive and negative samples are balanced. Negative
samples consist of balanced number of instances
with randomly replaced NL (i.e. ( c, ˆw)) and PL
(i.e. (ˆc, w)). Detailed hyper-parameters for model
ﬁne-tuning are given in Appendix B.2.
Model Comparisons Table 2 shows the results
of different approaches on the CodeSearchNet cor-
pus. The ﬁrst four rows are reported by Husain
et al. (2019), which are joint embeddings of NL and
PL (Gu et al., 2018; Mitra et al., 2018). NBOW
represents neural bag-of-words. CNN, BIRNN
and SELFATT stand for 1D convolultional neu-
ral network (Kim, 2014), bidirectional GRU-based
recurrent neural network (Cho et al., 2014), and
multi-head attention (Vaswani et al., 2017), respec-
tively.
We report the remaining numbers in Table 2.
We train all these pre-trained models by regarding
codes as a sequence of tokens. We also continu-
ously train RoBERTa only on codes from Code-
SearchNet with masked language modeling. Re-
sults show that CodeBERT consistently performs
4More details about the dataset are given in Appendix A.
5We have ﬁne-tuned a multi-lingual model for six program-
ming languages, but ﬁnd that it performs worse that ﬁne-tuning
a language-speciﬁc model for each programming language.
1541
MODEL RUBY JAVASCRIPT GO PYTHON JAVA PHP MA-AVG
NBOW 0.4285 0.4607 0.6409 0.5809 0.5140 0.4835 0.5181
CNN 0.2450 0.3523 0.6274 0.5708 0.5270 0.5294 0.4753
BIRNN 0.0835 0.1530 0.4524 0.3213 0.2865 0.2512 0.2580
SELF ATT 0.3651 0.4506 0.6809 0.6922 0.5866 0.6011 0.5628
ROBERTA 0.6245 0.6060 0.8204 0.8087 0.6659 0.6576 0.6972
PT W/ CODE ONLY (INIT =S) 0.5712 0.5557 0.7929 0.7855 0.6567 0.6172 0.6632
PT W/ CODE ONLY (INIT =R) 0.6612 0.6402 0.8191 0.8438 0.7213 0.6706 0.7260
CODE BERT (MLM, INIT =S) 0.5695 0.6029 0.8304 0.8261 0.7142 0.6556 0.6998
CODE BERT (MLM, INIT =R) 0.6898 0.6997 0.8383 0.8647 0.7476 0.6893 0.7549
CODE BERT (RTD, INIT =R) 0.6414 0.6512 0.8285 0.8263 0.7150 0.6774 0.7233
CODE BERT (MLM+RTD, INIT =R) 0.6926 0.7059 0.8400 0.8685 0.7484 0.7062 0.7603
Table 2: Results on natural language code retrieval. Baselines include four joint embeddings (ﬁrst group) of NL
and PL, RoBERTa, and RoBERTa which is continuously trained with masked language modeling on codes only
(second group). PT stands for pre-training. We train CodeBERT (third group) with different settings, including
using different initialization (from scratch (INIT =S) or initialized with the parameters of RoBERTa (INIT =R)) and
using different learning objectives (MLM, RTD, or the combination of both).
better than RoBERTa and the model pre-trained
with code only. CodeBERT (MLM) learned from
scratch performs better than RoBERTa. Unsur-
prisingly, initializing CodeBERT with RoBERTa
improves the performance 6.
4.2 NL-PL Probing
In the previous subsection, we show the empirical
effectiveness of CodeBERT in a setting that the
parameters of CodeBERT are ﬁne-tuned in down-
stream tasks. In this subsection, we further inves-
tigate what type of knowledge is learned in Code-
BERT without modifying the parameters.
Task Formulation and Data ConstructionFol-
lowing the probing experiments in NLP (Petroni
et al., 2019; Talmor et al., 2019), we study NL-
PL probing here. Since there is no existing work
towards this goal, we formulate the problem of
NL-PL probing and create the dataset by ourselves.
Given an NL-PL pair ( c, w), the goal of NL-PL
probing is to test model’s ability to correctly pre-
dict/recover the masked token of interest (either a
code token ci or word token wj) among distractors.
There are two major types of distractors: one is the
whole target vocabulary used for the masked lan-
guage modeling objective (Petroni et al., 2019), and
another one has fewer candidates which are ﬁlter or
curated based on experts’ understanding about the
ability to be tested (Talmor et al., 2019). We follow
the second direction and formulate NL-PL probing
as a multi-choice question answering task, where
the question is cloze-style in which a certain token
6We further give a learning curve of different pre-trained
models in the ﬁne-tuning process in Appendix C.
is replaced by [MASK] and distractor candidate
answers are curated based on our expertise.
Speciﬁcally, we evaluate on the NL side and PL
side, respectively. To ease the effort of data col-
lection, we collect data automatically from NL-PL
pairs in both validation and testing sets of Code-
SearchNet, both of which are unseen in the pre-
training phase. To evaluate on the NL side, we
select NL-PL pairs whose NL documentations in-
clude one of the six keywords (max, maximize, min,
minimize, less, greater), and group them to four
candidates by merging ﬁrst two keywords and the
middle two keywords. The task is to ask pre-trained
models to select the correct one instead of three
other distractors. That is to say, the input in this
setting includes the complete code and a masked
NL documentation. The goal is to select the correct
answer from four candidates. For the PL side, we
select codes containing keywordsmax and min, and
formulate the task as a two-choice answer selection
problem. Here, the input includes complete NL
documentation and a masked PL code, and the goal
is to select the correct answer from two candidates.
Since code completion is an important scenario,
we would like to test model’s ability in predicting
the correct token merely based on preceding PL
contexts. Therefore, we add an additional setting
for PL side, where the input includes the complete
NL documentation and preceding PL codes. Data
statistics is given in the top two rows in Table 3.
Model Comparisons Results are given in Table
3. We report accuracy, namely the number of cor-
rectly predicted instances over the number of all
instances, for each programming language. Since
1542
RUBY JAVASCRIPT GO PYTHON JAVA PHP ALL
NUMBER OF DATAPOINTS FOR PROBING
PL (2 CHOICES ) 38 272 152 1,264 482 407 2,615
NL (4 CHOICES ) 20 65 159 216 323 73 856
PL PROBING
ROBERTA 73.68 63.97 72.37 59.18 59.96 69.78 62.45
PRE-TRAIN W / CODE ONLY 71.05 77.94 89.47 70.41 70.12 82.31 74.11
CODE BERT (MLM) 86.84 86.40 90.79 82.20 90.46 88.21 85.66
PL PROBING WITH PRECEDING CONTEXT ONLY
ROBERTA 73.68 53.31 51.32 55.14 42.32 52.58 52.24
PRE-TRAIN W / CODE ONLY 63.16 48.53 61.84 56.25 58.51 58.97 56.71
CODE BERT (MLM) 65.79 50.74 59.21 62.03 54.98 59.95 59.12
NL PROBING
ROBERTA 50.00 72.31 54.72 61.57 61.61 65.75 61.21
PRE-TRAIN W / CODE ONLY 55.00 67.69 60.38 68.06 65.02 68.49 65.19
CODE BERT (MLM) 65.00 89.23 66.67 76.85 73.37 79.45 74.53
Table 3: Statistics of the data for NL-PL probing and the performance of different pre-trained models. Accuracies
(%) are reported. Best results in each group are in bold.
datasets in different programming languages are
extremely unbalanced, we report the accumulated
metric with the same way. We use CodeBERT
(MLM) here because its output layer naturally ﬁts
for probing. Results show that CodeBERT per-
forms better than baselines on almost all languages
on both NL and PL probing. The numbers with
only preceding contexts are lower than that with
bidirectional contexts, which suggests that code
completion is challenging. We leave it as a future
work.
We further give a case study on PL-NL probing.
We mask NL token and PL token separately, and
report the predicted probabilities of RoBERTa and
CodeBERT. Figure 3 illustrates the example of a
python code7. We can see that RoBERTa fails in
both cases, whereas CodeBERT makes the correct
prediction in both NL and PL settings.
4.3 Code Documentation Generation
Although the pre-training objective of Code-
BERT does not include generation-based objectives
(Lewis et al., 2019), we would like to investigate
to what extent does CodeBERT perform on gen-
eration tasks. Speciﬁcally, we study code-to-NL
generation, and report results for the documenta-
tion generation task on CodeSearchNet Corpus in
six programming languages. Since the generated
documentations are short and higher order n-grams
may not overlap, we remedy this problem by using
smoothed BLEU score (Lin and Och, 2004).
7The example comes from https://
github.com/peri-source/peri/blob/
61beed5deaaf978ab31ed716e8470d86ba639867/
peri/comp/psfcalc.py#L994-L1002
def vec_to_halfvec(vec):
d = vec[1:] - vec[:-1]
if ((d/d.mean()).std() > 1e-14) or (d.mean() < 0):
raise ValueError('vec must be np.arange() in increasing order')
dx = d.mean()
lowest = np.abs(vec). min ()
highest = np.abs(vec).max()
return np.arange(lowest, highest + 0.1*dx, dx).astype(vec.dtype)
"Transforms a vector np.arange(-N, M, dx) to np.arange( min (|vec|), 
max(N,M),dx)]"
masked NL token
masked PL token
max min less greater
NL
Roberta 96.24% 3.73% 0.02% 0.01%
CodeBERT (MLM) 39.38% 60.60% 0.02% 0.0003%
PL
Roberta 95.85% 4.15% - -
CodeBERT (MLM) 0.001% 99.999% - -
Figure 3: Case study on python language. Masked to-
kens in NL (in blue) and PL (in yellow) are separately
applied. Predicted probabilities of RoBERTa and Code-
BERT are given.
Model Comparisons We compare our model
with several baselines, including a RNN-based
model with attention mechanism (Sutskever et al.,
2014), the Transformer (Vaswani et al., 2017),
RoBERTa and the model pre-trained on code only.
To demonstrate the effectiveness of CodeBERT
on code-to-NL generation tasks, we adopt various
pre-trained models as encoders and keep the hyper-
parameters consistent. Detailed hyper-parameters
are given in Appendix B.3.
Table 4 shows the results with different mod-
els for the code-to-documentation generation task.
As we can see, models pre-trained on program-
ming language outperform RoBERTa, which illus-
trates that pre-trainning models on programming
1543
MODEL RUBY JAVASCRIPT GO PYTHON JAVA PHP OVERALL
SEQ 2SEQ 9.64 10.21 13.98 15.93 15.09 21.08 14.32
TRANSFORMER 11.18 11.59 16.38 15.81 16.26 22.12 15.56
ROBERTA 11.17 11.90 17.72 18.14 16.47 24.02 16.57
PRE -TRAIN W / CODE ONLY 11.91 13.99 17.78 18.58 17.50 24.34 17.35
CODE BERT ( RTD ) 11.42 13.27 17.53 18.29 17.35 24.10 17.00
CODE BERT ( MLM ) 11.57 14.41 17.78 18.77 17.38 24.85 17.46
CODE BERT ( RTD +MLM ) 12.16 14.90 18.07 19.06 17.65 25.16 17.83
Table 4: Results on Code-to-Documentation generation, evaluated with smoothed BLEU-4 score.
language could improve code-to-NL generation.
Besides, results in the Table 4 show that CodeBERT
pre-trained with RTD and MLM objectives brings
a gain of 1.3 BLEU score over RoBERTa overall
and achieve the state-of-the-art performance8.
4.4 Generalization to Programming
Languages NOT in Pre-training
We would like to evaluate CodeBERT on the pro-
gramming language which is never seen in the pre-
training step. To this end, we study the task of gen-
erating a natural language summary of a C# code
snippet. We conduct experiments on the dataset
of CodeNN (Iyer et al., 2016)9, which consists of
66,015 pairs of questions and answers automati-
cally collected from StackOverﬂow. This dataset
is challenging since the scale of dataset is orders
of magnitude smaller than CodeSearchNet Corpus.
We evaluate models using smoothed BLEU-4 score
and use the same evaluation scripts as Iyer et al.
(2016).
MODEL BLEU
MOSES (K OEHN ET AL ., 2007) 11.57
IR 13.66
SUM-NN (R USH ET AL ., 2015) 19.31
2-LAYER BILSTM 19.78
TRANSFORMER (VASWANI ET AL ., 2017) 19.68
TREE LSTM (T AI ET AL ., 2015) 20.11
CODE NN (I YER ET AL ., 2016) 20.53
CODE 2SEQ (ALON ET AL ., 2019) 23.04
ROBERTA 19.81
PRE -TRAIN W / CODE ONLY 20.65
CODE BERT (RTD) 22.14
CODE BERT (MLM) 22.32
CODE BERT (MLM+RTD) 22.36
Table 5: Code-to-NL generation on C# language.
Model Comparisons Table 5 shows that our
model with MLM and RTD pre-training objectives
achieves 22.36 BLEU score and improves by 2.55
points over RoBERTa, which illustrates CodeBERT
8We further give some output examples in Appendix E.
9https://github.com/sriniiyer/codenn
could generalize better to other programming lan-
guage which is never seen in the pre-training step.
However, our model achieve slightly lower results
than code2seq (Alon et al., 2019). The main reason
could be that code2seq makes use of compositional
paths in its abstract syntax tree (AST) while Code-
BERT only takes original code as the input. We
have trained a version of CodeBERT by traversing
the tree structure of AST following a certain order,
but applying that model does not bring improve-
ments on generation tasks. This shows a potential
direction to improve CodeBERT by incorporating
AST.
5 Conclusion
In this paper, we present CodeBERT, which to the
best of our knowledge is the ﬁrst large bimodal
pre-trained model for natural language and pro-
gramming language. We train CodeBERT on both
bimodal and unimodal data, and show that ﬁne-
tuning CodeBERT achieves state-of-the-art perfor-
mance on downstream tasks including natural lan-
guage code search and code-to-documentation gen-
eration. To further investigate the knowledge em-
bodied in pre-trained models, we formulate the task
of NL-PL probing and create a dataset for probing.
We regard the probing task as a cloze-style answer
selection problem, and curate distractors for both
NL and PL parts. Results show that, with model
parameters ﬁxed, CodeBERT performs better than
RoBERTa and a continuously trained model using
codes only.
There are many potential directions for further
research on this ﬁeld. First, one could learn better
generators with bimodal evidence or more compli-
cated neural architecture to improve the replaced to-
ken detection objective. Second, the loss functions
of CodeBERT mainly target on NL-PL understand-
ing tasks. Although CodeBERT achieves strong
BLEU scores on code-to-documentation genera-
tion, the CodeBERT itself could be further im-
proved by generation-related learning objectives.
1544
How to successfully incorporate AST into the pre-
training step is also an attractive direction. Third,
we plan to apply CodeBERT to more NL-PL re-
lated tasks, and extend it to more programming
languages. Flexible and powerful domain/language
adaptation methods are necessary to generalize
well.
Acknowledgments
Xiaocheng Feng is the corresponding author of this
work. We thank the anonymous reviewers for their
insightful comments. Zhangyin Feng, Xiaocheng
Feng, Bing Qin and Ting Liu are supported by the
National Key R&D Program of China via grant
2018YFB1005103 and National Natural Science
Foundation of China (NSFC) via grant 61632011
and 61772156.
References
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav.
2019. code2seq: Generating sequences from struc-
tured representations of code. International Confer-
enceon Learning Representations.
Kyunghyun Cho, Bart Van Merri ¨enboer, Caglar Gul-
cehre, Dzmitry Bahdanau, Fethi Bougares, Holger
Schwenk, and Yoshua Bengio. 2014. Learning
phrase representations using rnn encoder-decoder
for statistical machine translation. arXiv preprint
arXiv:1406.1078.
Kevin Clark, Minh-Thang Luong, Quoc V . Le, and
Christopher D. Manning. 2020. {ELECTRA}: Pre-
training text encoders as discriminators rather than
generators. In International Conference on Learn-
ing Representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2018. Bert: Pre-training of deep
bidirectional transformers for language understand-
ing. arXiv preprint arXiv:1810.04805.
Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018.
Deep code search. In 2018 IEEE/ACM 40th Interna-
tional Conference on Software Engineering (ICSE),
pages 933–944. IEEE.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis
Allamanis, and Marc Brockschmidt. 2019. Code-
searchnet challenge: Evaluating the state of seman-
tic code search. arXiv preprint arXiv:1909.09436.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and
Luke Zettlemoyer. 2016. Summarizing source code
using a neural attention model. In Proceedings
of the 54th Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers),
pages 2073–2083.
Dan Jurafsky. 2000. Speech & language processing.
Pearson Education India.
Aditya Kanade, Petros Maniatis, Gogul Balakrish-
nan, and Kensen Shi. 2019. Pre-trained contex-
tual embedding of source code. arXiv preprint
arXiv:2001.00059.
Yoon Kim. 2014. Convolutional neural net-
works for sentence classiﬁcation. arXiv preprint
arXiv:1408.5882.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris
Callison-Burch, Marcello Federico, Nicola Bertoldi,
Brooke Cowan, Wade Shen, Christine Moran,
Richard Zens, et al. 2007. Moses: Open source
toolkit for statistical machine translation. In Pro-
ceedings of the 45th annual meeting of the associ-
ation for computational linguistics companion vol-
ume proceedings of the demo and poster sessions,
pages 177–180.
Mike Lewis, Yinhan Liu, Naman Goyal, Mar-
jan Ghazvininejad, Abdelrahman Mohamed, Omer
Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.
Bart: Denoising sequence-to-sequence pre-training
for natural language generation, translation, and
comprehension. arXiv preprint arXiv:1910.13461.
Chin-Yew Lin and Franz Josef Och. 2004. Orange: a
method for evaluating automatic evaluation metrics
for machine translation. In Proceedings of the 20th
international conference on Computational Linguis-
tics, page 501. Association for Computational Lin-
guistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man-
dar Joshi, Danqi Chen, Omer Levy, Mike Lewis,
Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: A robustly optimized bert pretraining ap-
proach. arXiv preprint arXiv:1907.11692.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan
Lee. 2019. Vilbert: Pretraining task-agnostic visi-
olinguistic representations for vision-and-language
tasks. In Advances in Neural Information Process-
ing Systems, pages 13–23.
Bhaskar Mitra, Nick Craswell, et al. 2018. An intro-
duction to neural information retrieval. Foundations
and Trends® in Information Retrieval, 13(1):1–126.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt
Gardner, Christopher Clark, Kenton Lee, and Luke
Zettlemoyer. 2018. Deep contextualized word repre-
sentations. arXiv preprint arXiv:1802.05365.
Fabio Petroni, Tim Rockt ¨aschel, Patrick Lewis, Anton
Bakhtin, Yuxiang Wu, Alexander H Miller, and Se-
bastian Riedel. 2019. Language models as knowl-
edge bases? arXiv preprint arXiv:1909.01066.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.
How multilingual is multilingual bert? arXiv
preprint arXiv:1906.01502.
1545
Alec Radford, Karthik Narasimhan, Tim Salimans,
and Ilya Sutskever. 2018. Improving language
understanding by generative pre-training. URL
https://s3-us-west-2. amazonaws. com/openai-
assets/researchcovers/languageunsupervised/language
understanding paper. pdf.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine
Lee, Sharan Narang, Michael Matena, Yanqi Zhou,
Wei Li, and Peter J Liu. 2019. Exploring the limits
of transfer learning with a uniﬁed text-to-text trans-
former. arXiv preprint arXiv:1910.10683.
Alexander M Rush, Sumit Chopra, and Jason We-
ston. 2015. A neural attention model for ab-
stractive sentence summarization. arXiv preprint
arXiv:1509.00685.
Chen Sun, Austin Myers, Carl V ondrick, Kevin Mur-
phy, and Cordelia Schmid. 2019. Videobert: A joint
model for video and language representation learn-
ing. arXiv preprint arXiv:1904.01766.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.
Sequence to sequence learning with neural networks.
In Advances in neural information processing sys-
tems, pages 3104–3112.
Kai Sheng Tai, Richard Socher, and Christopher D
Manning. 2015. Improved semantic representations
from tree-structured long short-term memory net-
works. arXiv preprint arXiv:1503.00075.
Alon Talmor, Yanai Elazar, Yoav Goldberg, and
Jonathan Berant. 2019. olmpics–on what lan-
guage model pre-training captures. arXiv preprint
arXiv:1912.13283.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In Advances in neural information pro-
cessing systems, pages 5998–6008.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V
Le, Mohammad Norouzi, Wolfgang Macherey,
Maxim Krikun, Yuan Cao, Qin Gao, Klaus
Macherey, et al. 2016. Google’s neural machine
translation system: Bridging the gap between hu-
man and machine translation. arXiv preprint
arXiv:1609.08144.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car-
bonell, Ruslan Salakhutdinov, and Quoc V Le.
2019. Xlnet: Generalized autoregressive pretrain-
ing for language understanding. arXiv preprint
arXiv:1906.08237.
A Data Statistic
Data statistics of the training/validation/testing data
splits for six programming languages are given in
Table 6.
CODE SEARCH TRAINING DEV TESTING
GO 635,635 28,483 14,291
JAVA 908,886 30,655 26,909
JAVASCRIPT 247,773 16,505 6,483
PHP 1,047,406 52,029 28,391
PYTHON 824,342 46,213 22,176
RUBY 97,580 4,417 2,279
Table 6: Data statistics about the CodeSearchNet Cor-
pus for natural language code search.
B Train Details
B.1 Pre-training
We train CodeBERT on one NVIDIA DGX-2 ma-
chine using FP16. It combines 16 interconnected
NVIDIA Tesla V100 with 32GB memory. We use
the following set of hyper-parameters to train mod-
els: batchsize is 2,048 and learning rate is 5e-4. We
use Adam to update the parameters and set the num-
ber of warmup steps as 10K. We set the max length
as 512 and the max training step is 100K. Training
1,000 batches of data costs 600 minutes with MLM
objective, 120 minutes with RTD objective.
B.2 CodeSearch
In the ﬁne-turning step, we set the learning rate as
1e-5, the batch size as 64, the max sequence length
as 200 and the max ﬁne-tuning epoch as 8. As the
same with pre-training, We use Adam to update the
parameters. We choose the model performed best
on the development set, and use that to evaluate on
the test set.
B.3 Code Summarization on Six
Programming Languages
We use Transformer with 6 layers, 768 dimensional
hidden states and 12 attention heads as our decoder
in all settings. We set the max length of input
and inference as 256 and 64, respectively. We use
the Adam optimizer to update model parameters.
The learning rate and the batch size are 5e-5 and
64, respectively. We tune hyperparameters and
perform early stopping on the development set.
B.4 Code Summarization on C#
Since state-of-the-art methods use RNN as their de-
coder, we choose a 2-layer GRU with an attention
mechanism as our decoder for a comparison. We
ﬁne-tune models using a grid search with the fol-
lowing set of hyper-parameters: batchsize is in{32,
64}and learning rate is in {2e-5, 5e-5}. We report
1546
the number when models achieve best performance
on the development set.
C Learning Curve of CodeSearch
From Figure 4, we can see that CodeBERT per-
forms better at the early stage, which reﬂects that
CodeBERT provides good initialization for learn-
ing downstream tasks.
Figure 4: Learning curve of different pre-trained mod-
els in the ﬁne-tuning step. We show results on Python
and Java.
D Late Fusion
In section §4.1 , we show that CodeBERT per-
forms well in the setting where natural languages
and codes have early interactions. Here, we in-
vestigate whether CodeBERT is good at working
as a uniﬁed encoder. We apply CodeBERT for
natural language code search in a later fusion set-
ting, where CodeBERT ﬁrst encodes NL and PL
separately, and then calculates the similarity by dot-
product. In this way, code search is equivalent to
ﬁnd the nearest codes in the shared vector space.
This scenario also facilitates the use of CodeBERT
in an online system, where the representations of
codes are calculated in advance. In the runtime, a
system only needs to compute the representation
of NL and vector-based dot-product.
We ﬁne-tune CodeBERT with the following ob-
jective, which maximizes the dot-product of the
ground truth while minimizing the dot-product of
distractors.
−1
N
∑
i
log
( exp
(
Enc(ci)⊺Enc(wi)
)
∑
j exp
(
Enc(cj)⊺Enc(wi)
)
)
(15)
Results are given in Table 7. We just do this
setting on two languages with a relatively small
amount of data.
We can see that CodeBERT performs better than
RoBERTa and the model pre-trained with codes
MODEL RUBY GO
ROBERTA 0.0043 0.0030
PRE-TRAIN W / CODE ONLY 0.1648 0.4179
CODE BERT 0.6870 0.8372
Table 7: Results on natural language code search by
late fusion.
only. And late fusion performs comparable with
the standard way. What’s more, late fusion is more
efﬁcient and this setting could be used in an online
system.
E Case Study
To qualitatively analyze the effectiveness of Code-
BERT, we give some cases for code search and
code documentation generation tasks.
Considering the limited space, we only give the
top2 results of the query for python programming
language. As show in Figure 5, search results are
very relevant with query.
Figure 6 and Figure 7 show the outputs with
different models for the code documentation gen-
eration task. As we can see, CodeBERT performs
better than all baselines.
1547
Figure 5: Python CodeSearch example. The results are searched from 1,156,085 python code data. We only give
the top2 results because space is limited.
Figure 6: Java code documentation generation output example.
Figure 7: Python code documentation generation output example.

CompilerGym: Robust, Performant Compiler
Optimization Environments for AI Research
Chris Cummins, Bram Wasti, Jiadong Guo, Brandon Cui, Jason Ansel, Sahir Gomez,
Somya Jain, Jia Liu, Olivier Teytaud, Benoit Steiner, Yuandong Tian, Hugh Leather
Facebook
cummins@fb.com
Abstract—Interest in applying Artiﬁcial Intelligence (AI) tech-
niques to compiler optimizations is increasing rapidly, but com-
piler research has a high entry barrier. Unlike in other domains,
compiler and AI researchers do not have access to the datasets
and frameworks that enable fast iteration and development
of ideas, and getting started requires a signiﬁcant engineering
investment. What is needed is an easy, reusable experimental
infrastructure for real world compiler optimization tasks that
can serve as a common benchmark for comparing techniques,
and as a platform to accelerate progress in the ﬁeld.
We introduce CompilerGym 1, a set of environments for real
world compiler optimization tasks, and a toolkit for exposing
new optimization tasks to compiler researchers. CompilerGym
enables anyone to experiment on production compiler optimiza-
tion problems through an easy-to-use package, regardless of their
experience with compilers. We build upon the popular OpenAI
Gym interface enabling researchers to interact with compilers
using Python and a familiar API.
We describe the CompilerGym architecture and implemen-
tation, characterize the optimization spaces and computational
efﬁciencies of three included compiler environments, and provide
extensive empirical evaluations. Compared to prior works, Com-
pilerGym offers larger datasets and optimization spaces, is 27 ×
more computationally efﬁcient, is fault-tolerant, and capable of
detecting reproducibility bugs in the underlying compilers.
In making it easy for anyone to experiment with compilers –
irrespective of their background – we aim to accelerate progress
in the AI and compiler research domains.
I. I NTRODUCTION
There is a growing body of work that shows how the
performance and portability of compiler optimizations can
be improved through autotuning [ 1], machine learning [ 2],
and reinforcement learning [ 3], [ 4], [ 5]. The goal of these
approaches is to supplement or replace the optimization
decisions made by hand-crafted heuristics with decisions
derived from empirical data. Autotuning makes these decisions
by automatically searching over a space of conﬁgurations. This
is effective, but search may be prohibitively costly for large
search spaces, and must be repeated from scratch for each new
problem instance. The promise of supervised and reinforcement
learning techniques is to reduce or completely eliminate this
search cost by inferring optimization decisions from patterns
observed in past data.
Despite many strong experimental results showing that
these techniques outperform human experts [ 2], [1], [6], the
complexity of experimental infrastructure for compiler research
1Available at: https://compilergym.ai
CompilerGym 
Environment
Benchmark 
Datasets
Compiler APIs
Feature 
Extractors
Observation
View of Program
Reward
Performance Metric
Action Space
Optimization Decisions
Action
User
(Python API, 
web interface,
or commandline)
Fig. 1: The CompilerGym interaction loop. A CompilerGym
environment exposes an observation, reward, and action space.
The user’s goal is to select the action that will lead to
the greatest cumulative reward. This may be through hand-
crafted heuristics, search, supervised machine learning, or
reinforcement learning.
hampers progress in the ﬁeld. In many other ﬁelds there are
simple environments, each using standard APIs that machine
learning researchers can interact with. From Atari games to
physics simulations, a known interface abstracts the problems
to the point that AI researchers do not need deep knowledge
of the problem to apply their machine learning techniques.
CompilerGym provides just that for compilers. AI researchers
can solve compiler problems without being compiler experts,
and compiler experts can integrate state-of-the-art ML without
being AI experts.
To support this ease of use and performance CompilerGym
offers the following key features:
1) Easy to install. Precompiled binaries for Linux and
macOS can be installed with a single command.
2) Easy to use. Builds on the Gym [ 7] API that is easy to
learn and widely used by researchers.
3) Comprehensive. Includes a full suite of millions of
benchmarks. Provides multiple kinds of pre-computed
program representations and appropriate optimization
targets and reward functions out of the box.
4) Reproducible. Provides validation for correctness of
results and public leaderboards to aggregate results.
5) Accessible. Includes code-free ways to explore Compil-
arXiv:2109.08267v2  [cs.PL]  22 Dec 2021
Frontend
Gym APIs
Resource 
Management
Error Handlers
Compilation 
Session
RPC 
Interface
LLVM Service
GCC Service
Other compiler
…
Python C++ / Python / other
Fig. 2: The client-server architecture. The dashed line indicates
the boundary between the frontend Python process that the
user interacts with and the backend compiler services. The pro-
cesses communicate over RPC, providing simple distribution,
parallelization, and fault tolerance.
erGym environments, such as an interactive command
line shell and a browser-based graphical user interface.
6) Performant. Supports the high throughput required for
large-scale experiments on massive datasets.
7) Fault-tolerant. Detects and gracefully recovers from
ﬂaky compiler errors that can occur during autotuning.
8) Extensible. Removes the substantial engineering effort
required to expose new compiler problems for research
and integrate new machine learning techniques.
In this paper, we make the following contributions:
• We introduce CompilerGym, a Python library that for-
mulates compiler optimization problems as easy-to-use
Gym [7] environments with a simple API.
• We provide environments for three compiler optimization
problems: LLVM phase ordering, GCC ﬂag selection, and
CUDA loop nest generation. The environments are de-
signed from the ground up for large-scale experimentation:
they are 27×faster than prior works, expose larger search
spaces, include millions of programs for training, and
support optimizing for both code size and runtime.
• We demonstrate the utility of CompilerGym as a platform
for research by evaluating a multitude of autotuning and
reinforcement learning techniques. By using a standard
interface, CompilerGym seamlessly integrates with third
party libraries, offering a substantial reduction in the
engineering effort required to create compiler experiments.
• We release a suite of tools to lower the barrier-to-entry to
compiler optimization research: the core CompilerGym
library and environments, a toolkit for integrating new
compiler optimization problems, public leaderboards to
aggregate and verify research results, a web interface
and API, extensive command line tools, and large ofﬂine
datasets comprising millions of performance results.
II. S YSTEM ARCHITECTURE
CompilerGym’s architecture comprises two components: a
Python frontend that implements the Gym APIs and other
user-facing tools, and a backend that provides the integrations
with speciﬁc compilers.
Frontend: The CompilerGym frontend is a Python library
that exposes compiler optimization tasks using the OpenAI
import compiler_gym
# Create a new environment, selecting the compiler to
# use, the program to compile, the feature vector to
# represent program states, and the optimization target:
env = compiler_gym.make(
"llvm-v0",
benchmark="cbench-v1/qsort",
observation_space="Autophase",
reward_space="IrInstructionCount",
)
# Start a new compilation session:
observation = env.reset()
# Run a thousand random optimizations. Each step of the
# environment produces a new state observation and reward:
for _ in range(1000):
observation, reward, done, info = env.step(
env.action_space.sample() # User selects action.
)
if done:
env.reset()
# Save output program:
env.write_bitcode("/tmp/output.bc")
Listing 1: Example of the CompilerGym environment API. A
CompilerGym environment builds on gym.Env, formulating
a compiler optimization task as a Markov Decision Process,
and provides additional compiler-speciﬁc functionality such as
saving the compiled program to disk.
Gym [7] environment interface. Figure 1 shows the interaction
loop for the Gym environments. This allows researchers to
interact with important compiler optimization problems in
a familiar language and vocabulary with which many are
comfortable. The frontend is described in Section III.
Backend: CompilerGym uses a client-server architecture,
shown in Figure 2. This design provides separation of concerns
as systems developers can easily add support for new compiler
problems by implementing a simple Compilation Session
interface that comprises only four methods. The backend is
described in Section IV.
III. F RONTEND API AND TOOLS
This section describes CompilerGym’s user-facing tools. We
ﬁrst describe the core formulation of compiler optimization
problems as Gym environments, then the API extensions and
other features tailored for compiler optimization research.
A. OpenAI Gym Environments
We formulate compiler optimization tasks as Markov Deci-
sion Processes (MDPs) and expose them as environments using
the popular OpenAI Gym [ 7] interface. A Gym environment
comprises ﬁve ingredients:
1) An Action Space deﬁnes the set of possible actions that
can be taken from a given MDP state. In CompilerGym, action
spaces can be composed of discrete choices (e.g. selecting an
optimization pass from a ﬁnite set), continuous choices (e.g.
selecting a function inlining threshold), or any combination of
the two. The action space can change between states, such as
in the case where one optimization precludes another.
2) An Observation Space from which observations of the
MDP state are drawn. CompilerGym environments support
multiple observation types such as numeric feature vectors
generated by compiler analyses, control ﬂow graphs, and strings
2
of compiler IR. Each environment exposes multiple observation
spaces that can be selected from or composed.
3) A Reward Space deﬁnes the range of values generated by
the reward function, used to provide feedback on the quality of
a chosen action, either positive or negative. In CompilerGym,
reward spaces can be nondeterministic (e.g. change in program
runtime), platform speciﬁc (e.g. change in the size of a compiled
binary), or entirely deterministic.
4) A Step operator applies an action at the current state and
responds with a new observation, a reward, and a signal that
indicates whether the MDP has reached a terminal state. Not
all compiler optimization problems have terminal states.
5) A Reset operator resets the environment to an initial state
and returns an initial observation.
Listing 1 demonstrates how the core CompilerGym API
is used. A make() function instantiates a subclass of the
gym.Env environment that represents a particular compiler op-
timization task. The Gym interface is self describing: the action
space and observation spaces are described by action_space
and observation_space attributes, respectively. This enables
CompilerGym environments to be integrated directly with
techniques that are compatible with other Gym environments.
Listing 2 shows one such integration.
In interacting with an environment the user’s goal is to
select the sequence of actions that maximizes the cumulative
reward. Although Gym is designed primarily for reinforcement
learning research, it makes no assumptions about the structure
of user code and therefore can be used with a wide range of
approaches. For a single environment, the best sequence of
actions may be found through search. To generalize a solution
that works for unseen environments, a policy is learned to map
from observation to optimal actions, or a Q-function is learned
to give expected cumulative rewards for state-action pairs.
B. API Extensions for Compiler Optimization
The advantage of the Gym interface is that it is simple and
can be used across a range of domains. We supplement this
interface with additional APIs that are speciﬁc to compilers.
1) Benchmark Datasets: An instance of a compiler optimiza-
tion environment requires a program to optimize. We refer to
these programs as benchmarks, and collections of benchmarks
as datasets. We designed an API to manage datasets that
efﬁciently scales to millions of benchmarks, and a mechanism
for downloading datasets from public servers. This API supports
program generators (like CSmith [ 8]), compiling user-supplied
code to use as benchmarks, iterating and looping over sets
of benchmarks, and specifying an input dataset and execution
environment for running compiled binaries.
2) State Serialization: We provide a mechanism to save and
restore environment state that includes the benchmark, action
history, and cumulative reward.
3) Validating States: Serialized states can be replayed to
validate that results are reproducible. We use this to ensure
reproducibility of the underlying compiler infrastructure. For
import compiler_gym
from ray import tune
from ray.rllib.agents.ppo import PPOTrainer
def make_env(config):
# Create an LLVM environment using the Autophase
# observation space and instruction count rewards.
env = compiler_gym.make("llvm-autophase-ic-v0")
# Optionally create a time limit for the RL agent.
env = compiler_gym.wrappers.TimeLimit(env, 45)
# Loop over the NPB benchmark suite for training.
dataset = env.datasets["benchmark://npb-v0"]
env = compiler_gym.wrappers.CycleOverBenchmarks(
env, dataset.benchmarks()
)
return env
tune.register_env("CompilerGym", make_env)
tune.run(PPOTrainer, config={"env": "CompilerGym"})
Listing 2: Using RLlib [ 9] to train a reinforcement learning
agent on one of the CompilerGym environments.
example, we detected a nondeterminism bug in an LLVM
optimization pass1; we removed this pass from CompilerGym.
4) Validating Semantics: For runnable benchmarks, we pro-
vide an additional layer of results validation that automatically
applies a differential testing [ 10] regime to detect correctness
errors in the compiled binaries. For the LLVM environments we
also integrate LLVM’s address, thread, and undeﬁned behavior
sanitizers to detect program logic errors.
5) Lazy and batched operations: Typically, the observation
and reward spaces of a Gym environment are determined at
construction time, and each step() operation takes a single
action and produces a single observation and reward. We extend
this method in CompilerGym environments to optionally accept
multiple actions, and a list of observation and reward spaces
to compute and return. Passing multiple actions enables the
backend to more efﬁciently execute them in a single batch
and return a ﬁnal state and reward, evaluated in Section VII-A.
Specifying the observation and reward spaces as arguments to
step() enables efﬁcient lazy computation of observations or
rewards in cases where the values are not needed at every step,
or to ﬂexibly change observation and reward space during the
liftetime of an environment.
6) Lightweight deep copy operator: CompilerGym envi-
ronments provide a fork() operator that efﬁciently creates
independent deep copies of environment states. This can be
used to optimize backtracking or other techniques that require
frequently evaluating a common subsequence of actions. For
example, a greedy search can be implemented by creating n
forks of an environment with an n-dimensional action space,
running a single action in each fork, and selecting the one
which produced the greatest reward. Backtracking is especially
expensive in compilers because most actions have no “undo”.
C. Customizing Environment Behavior
The Gym [ 7] library deﬁnes environment wrapper classes
to mutate the MDP formulation of a wrapped environment.
1LLVM’s-gvn-sink pass contains an operation that sorts a vector of
basic block pointers by address, causing inconsistent output.
3
Fig. 3: CompilerGym Explorer, a website that enables users
to interact with the LLVM phase ordering environment. The
left side of the page renders the phase ordering search space
as an interactive tree; the right side of the page visualizes the
program features and their trends.
CompilerGym provides an additional suite of environment
wrappers for a broad range of compiler research uses. These
include specifying a subset of command line ﬂags to use in an
action space, iterating over a suite of benchmarks, and deﬁning
derived observation spaces such as using custom compiler
analyses on compiler IR. These wrappers can be composed.
Listing 2 shows integration with the popular RLlib [ 9] library
using two of these wrappers.
D. Command Line Tools
We include a complete set of command line tools for
CompilerGym, including scripts to run parallelized searches,
replay and validate results from past runs, and an interactive
shell that includes inline documentation and tab completion,
enabling users to interact with the compiler optimization
environments without writing any code.
E. Web Service and CompilerGym Explorer
We designed a REST API to enable CompilerGym environ-
ments to be used over a network, and CompilerGym Explorer 1,
a web frontend that makes it easy to navigate compiler
optimization spaces, implemented using React. CompilerGym
Explorer presents a visualization of the search tree, shown in
Figure 3, and asynchronously calls the REST API to update
the tree in real time as the user interacts with it.
A key feature of the tool is to visualize not only the current
state, but also historical trends of the rewards and observation
metrics. This allows users to easily pinpoint interesting actions
in a large search tree and trigger new explorations. We expect
this to be valuable for feature engineering, debugging the
behavior of agents, and as a general educational tool.
1Available at: https://compilergym.ai/explorer
benchmark_uri: str
actions: int[]
state_id: sha1
end_of_episode: bool
rewards: ﬂoat[]
Steps
state_id: sha1
compressed_ir: bytes
instcounts: int[]
…
Observations
state_id: sha1
action: int
next_state_id: sha1
rewards: ﬂoat[]
StateTransitions
n 
1
n  n 
Fig. 4: The relational database schema for state transitions
in LLVM environments. Fields that comprise unique primary
keys are emboldened. We are releasing an instance of this
database comprising over 1M unique states that can be used
for pre-training, off-policy learning, or general ofﬂine analysis.
F . State Transition Dataset
We designed a relational database schema to log the state
transitions of CompilerGym environments for later ofﬂine
analysis, shown in Figure 4. A Steps table records every
unique action sequence for a particular benchmark and a hash of
the environment state. An Observations table stores various
representations of each unique state, indexed by state hash.
A StateTransitions table encodes the unique transitions
between states and the rewards received for each.
We implemented a wrapper class for CompilerGym en-
vironments that asynchronously populates the Steps and
Observations tables of a state transition database upon every
step of an environment. A post-processing script de-duplicates
and populates the StateTransitions table.
We are releasing a large instance of this database ( 50+GB)
which contains over 1M unique LLVM environment states,
suitable for a range of ofﬂine supervised and unsupervised
learning tasks. We evaluate an example usage in Section VII-F.
IV. B ACKEND RUNTIME AND INTERFACE
The CompilerGym backend comprises a CompilationSession
interface for integrating compilers and a common client-server
runtime that map this interface to the Gym API.
A. The CompilationSession Interface
CompilerGym is designed for seamless compiler integration.
The integration centers around implementing a state machine to
interact with the compiler called a CompilationSession. A Com-
pilationSession exposes actions and observations using a simple
schema and must implement two methods, apply_action
and get_observation, as shown in Figure 5. We provide
CompilationSession interfaces for Python and C++. Listing 3
demonstrates an example implementation.
B. Compiler Service Runtime
A common runtime maps implementations of the Compila-
tionSession interface (Listing 3) to the Gym API (Listing 1).
This runtime is shared by all compiler integrations and is
architected to be performant and scalable. The design is
4
Fig. 5: A graphical representation of the CompilationSession
integration. To add a new compiler to CompilerGym, users
need only deﬁne the boxes highlighted in blue. Grey boxes
demonstrate the integration of the CompilerSession with a
typical reinforcement learning loop.
resilient to failures, crashes, inﬁnite loops, and nondeterministic
behavior in backend compiler services. All compiler service
operations have appropriate timeouts, graceful error handling,
or retry loops. Improvements to the runtime can be made
without changing compiler integration or user code.
A key design point of the CompilerGym runtime is that the
service that provides the compiler integration is isolated in a
separate process to the user’s Python interpreter. The Python
interpreter invokes operations on the compiler service through
Remote Procedure Calls (RPCs). The beneﬁts of this are fault
tolerance and recovery in cases where the compiler crashes or
terminates abruptly; support for compiling on a different system
architecture than the host by running the compiler service on
a remote machine; and scalability as the expensive compute
work is ofﬂoaded, enabling many user threads to interact with
separate compiler environments without contention on Python’s
global interpreter lock.
V. E NVIRONMENTS
This section describes three compiler integrations shipped
in CompilerGym.
A. LLVM Phase Ordering
LLVM [ 11] is a modular compiler infrastructure used
throughout academia and industry. After parsing an input source
program to a language-agnostic Intermediate Representation
(IR), the LLVM optimizer applies a conﬁgurable pipeline of
optimization passes to the IR. The selection and ordering of
compiler optimizations – known as phase ordering – greatly
impacts the quality of the ﬁnal binary and has been the focus
of much research [1], [12].
We include a phase ordering environment in CompilerGym
as an example of a challenging, high-dimensional optimization
problem in which signiﬁcant gains can be achieved.
#include "compiler_gym/service/CompilationSession.h"
#include "compiler_gym/service/runtime/Runtime.h"
using namespace compiler_gym;
struct MyCompilationSession: public CompilationSession{
vector<ActionSpace> getActionSpaces() {...}
vector<ObservationSpace> getObservationSpaces() {...}
Status init(
const ActionSpace& actionSpace,
const Benchmark& benchmark) {...}
Status applyAction(
const Action& action,
bool& endOfEpisode,
bool& actionSpaceChanged) {...}
Status setObservation(
const ObservationSpace& observationSpace,
Observation& observation) {...}
};
int main(int argc, char** argv) {
runtime::createAndRunService<MyCompilationSession>(
argc, argv, "My compiler service");
}
Listing 3: A C++ implementation of the CompilationSession
interface to add support for a new compiler. Method bodies
are omitted for brevity. There is an equivalent API for Python.
Actions: The action space consists of a discrete choice from
124 optimization passes extracted automatically from LLVM.
There is no maximal episode length as episodes can run forever
(except in the case of a compiler bug leading to an error), the
user must estimate when no further gains can be achieved and
no further actions should be taken. For any particular program
the optimal phase ordering may omit or repeat actions.
Rewards: We support optimizing for three metrics: code
size, which is the number of instructions in the IR; binary
size, which is the size of the .text section in the compiled
object ﬁle; and runtime, which is the wall time of the compiled
program when run using a speciﬁc conﬁguration of inputs
on the machine hosting the CompilerGym backend. When
used as a reward signal each metric returns the change in
value between the previous environment state and the new
environment state. Each reward signal can optionally be scaled
against the gains achieved by the compiler’s default phase
orderings, -Oz for size reduction and -O3 for runtime. Code
size is platform-independent and determinsitic, binary size
is platform-dependent and deterministic, and runtime is both
platform-speciﬁc and nondeterministic.
Observations: We provide ﬁve observation spaces for LLVM
ranging from counter-based numeric feature vectors [ 4] to
sequential language models [ 13] up to graph-based program
representations [14]. See Table III for a comparison.
Datasets: We provide millions of programs for evaluation,
summarized in Table I. We aggregate C, C++, OpenCL,
and Fortran programs from benchmark suites in a variety
of domains, open source programs, and synthetic program
generators. Accessing these datasets within CompilerGym is
as simple as specifying the name of the dataset and optionally
the name of a speciﬁc benchmark. Presently only cBench [ 15]
and Csmith [8] support optimizing for runtime.
5
Number of Benchmarks
Dataset Autophase [4] MLGO [3] CompilerGym
AnghaBench [16] 1,041,333
BLAS [17] 300
cBench [15] 23
CHStone [18] 9 12
CLgen [19] 996
GitHub [14] 49,738
Linux kernel 13,894
MiBench [20] 40
NPB [21] 122
OpenCV 442
POJ-104 [22] 49,816
TensorFlow [23] 1,985
Csmith [8] 100 232†
llvm-stress [11] 232†
Proprietary 28,000
TABLE I: LLVM benchmark datasets included in Compiler-
Gym, compared to the number of benchmarks used in two
recent machine learning works. †denotes random program
generators with 32-bit seeds. Excluding the program generators,
the total number of benchmarks is 1,145,499.
B. GCC Flag Tuning
We include an environment that exposes the optimization
space deﬁned by GCC’s command line ﬂags. The environment
works with any version of GCC from 5 up to and including the
current version at time of writing, 11.2. The environment uses
Docker images to enable hassle free install and consistency
across machines. Alternatively, any local installation of the
compiler can be used. This selection is made by simple string
speciﬁer of the path or docker image name. The only change
that an RL agent needs to make to work with GCC instead of
LLVM is to call env = gym.make("gcc-v0"), instead of
using "llvm-v0".
While the LLVM phase ordering action space is unbounded
as passes may be executed forever, the number of GCC
command line conﬁgurations is bounded. GCC’s action space
consists of all the available optimization ﬂags and parameters
that can be speciﬁed from the command line. These are automat-
ically extracted from the “help” documentation of whichever
GCC version is used. For GCC 11.2.0 1, the optimization space
includes 502 options:
• the six -O<n> ﬂags, e.g. -O0, -O3, -Ofast, -Os.
• 242 ﬂags such as -fpeel-loops, each of which may be
missing, present, or negated (e.g. -fno-peel-loops).
Some of these ﬂags may take integer or enumerated
arguments which are also included in the space.
• 260 parameterized command line ﬂags such as --param
inline-heuristics-hint-percent=<number>. The
number of options for each of these varies. Most take
numbers, a few take enumerated values.
This gives a ﬁnite optimization space with a modest size of
approximately 104461. Earlier versions of GCC report their
parameter spaces less clearly and so the tool ﬁnds smaller
111.2.0 is the latest stable version of GCC at time of writing.
for a in 1048576 : L0 [thread]
for a’ in 1 : L1
for a’’ in 1 : L2
%0[a] <- read()
for a’’ in 1 : L4
%1[a] <- read()
for a’’ in 1 : L6
%2[a] <- add(%0, %1)
for a’’ in 1 : L8
%3[a] <- write(%2)
Listing 4: An example loop tree in the loop tool environment.
spaces when pointed at those. For example, on GCC 5, the
optimization space is only 10430.
Actions: We provide two action spaces that can be used
interchangeably. The ﬁrst directly exposes the optimization
space via a list of integers, each encoding the choice for one
option with a known cardinality. A second action space is
intended to make it easy for RL tools that operate on a ﬂat
list of categorical actions. For every option with a cardinality
of fewer than ten, we provide actions that directly set the
choice for that action. For options with greater cardinalities
we provide actions that add and subtract 1, 10, 100, and 1000
to the choice integer corresponding to the option. For GCC
11.2.0, this creates a set of 2281 actions that can modify the
choices of the current state.
Rewards: We provide two deterministic reward signals: the
sizes in bytes of the assembly or the object code.
Observations: We provide four observation spaces: a nu-
meric instruction count observation, the Register Transfer
Language code at the end of compilation, the assembly code
as text, and the object code as a binary.
C. CUDA Loop Nest Code Generation
Manually tuning CUDA code requires sweeping over many
parameters. Due to the sheer size of the tunable space, the
problem of generating fast CUDA is well suited for automated
techniques [24], [25]. As a ﬂexible compilation environment,
CompilerGym is well equipped to handle compilers for tuning
GPU workloads. We integrated loop_tool, a simple dense
linear algebra compiler [ 26]. loop_tool takes a minimalist
approach to linear algebra representations by decomposing
standard BLAS-like routines into a DAG of n-dimensional
applications of arithmetic primitives. The DAG is then anno-
tated with three pieces of information about loop ordering: the
order in which loops are emitted, the nesting structure of each
loop, and the reuse of loops by subsequent operations. This is
lowered to a loop tree that can be annotated with which loop
should be run in parallel. These four annotations across a slew
of point-wise operations represent a large optimization space.
Actions: We map interacting with the loop structure for
point-wise additions to a cursor-based discrete action space.
At any point the cursor will refer to an individual loop in the
loop hierarchy and will have an associated “mode” to control
either moving the cursor or modifying the current loop. There
is an action “toggle mode” to swap between these two. When
moving the cursor, the actions “up” and “down” will shift
6
the cursor inward and outward respectively. When modifying
the current loop, the action “up” will increase its size by
one. This is done by changing the size of the parent loop
to accommodate the new inner size. Often this induces tail
logic, which is handled automatically. Finally, any loop can
be changed to be threaded. This will schedule loop execution
across CUDA threads which may span multiple warps or even
multiple streaming multiprocessors. A second, extended action
space allows loops to be split, creating a larger hierarchy.
Rewards: The environment reward signal is a measurement
of ﬂoating point operations per second (FLOPs) achieved by
benchmarking the loop nest in the given state. This is both
platform dependent and non-deterministic due to the noise
involved in benchmarking.
Observations: There are two observations spaces: action
state, which describes the cursor position and mode, and loop
tree structure, which is a textual dump of the current state of
the loop_tool environment, as shown in Listing 4.
VI. I MPLEMENTATION
CompilerGym is implemented in a mixture of Python and
C++. The core runtime comprises 12k lines of code. The
compiler integrations comprise 6k lines of code for LLVM,
3k for GCC and 0.5k for loop_tool. CompilerGym is open
source and available under a permissive license.
Binary Releases: Periodic versioned releases are made from
a stable branch. We ship pre-compiled release binaries for
macOS and Linux (Ubuntu 18.04, Fedora 28, Debian 10 or
newer equivalents) that can be installed as Python wheels.
Documentation: Our public facing documentation includes
full API references for Python and C++, a getting started
guide, FAQ, and code samples demonstrating integration with
RLlib [9], implementations of exhaustive, random, and greedy
searches, and Q-learning [27] and Actor Critic [28].
Testing: We have a comprehensive unittest suite with 85.8%
branch coverage that is run on every code change across a test
matrix of all supported operating systems and Python versions.
Additionally, a suite of fuzz and stress tests are ran daily by
continuous integration services to proactively identify issues.
VII. E VALUATION
We evaluate CompilerGym ﬁrst by comparing the computa-
tional efﬁciency of the environments to prior works. We then
show how the simplicity of the CompilerGym APIs enables
large-scale autotuning and reinforcement learning experiments
to be engineered with remarkably few lines of code.
Experimental Platforms: Results in this section are obtained
from shared compute servers equipped with Intel Xeon 8259CL
CPUs, NVIDIA GP100 GPUs, and ﬂash storage.
A. Computational Efﬁciency
A key design goal of CompilerGym is to provide the best
performance possible, enabling researchers to train larger mod-
els, try more conﬁgurations, and get better results in less time.
We evaluate the computational efﬁciency of CompilerGym’s
LLVM phase ordering environment and compare to two prior
works: Autophase [4] and OpenTuner [29].
100 101 102 103
Step time (ms, log axis)
0.00
0.25
0.50
0.75
1.00Probability
Fig. 6: Cumulative density plot of step times for each of the 23
programs in cBench [ 15]. Each line shows a different program.
The difference between the median step times of the fastest
program (crc32) and the slowest (ghostbench) is 560.3 ×.
We use code size rewards signals for all three platforms
and the observation space used in [ 4] for Autophase and
CompilerGym; OpenTuner is a black box search framework
and so does not provide observation spaces. We measure the
computational efﬁciencies of each environment by measuring
the wall times of operations during 1M random trajectories.
For CompilerGym, which uses a client-server architecture, we
also measure the initial server startup time.
Table II shows the results. CompilerGym achieves a much
higher throughput than Autophase while offering the same
interface, observation space, and reward signal. This is enabled
by CompilerGym’s client-server architecture. After initially
reading and parsing the bitcode ﬁle from disk, the Compiler-
Gym server incrementally applies an individual optimization
pass at each step. In contrast, Autophase and OpenTuner must,
at each step, read and parse the IR, apply the entire sequence
of passes, and then serialize the result. OpenTuner, which
was designed for uses where the search time is dominated by
compilation time, has the highest environment initialization
cost, as it requires several disk operations and the creation
of a database. The CompilerGym server maintains a cache of
parsed unoptimized bitcodes that enables an amortized O(1)
cost of environment initialization.
The distribution of operation wall times depends on the action
being performed and the program being optimized. Figure 6
shows the wide distribution of wall times within the benchmarks
of a single dataset.
B. Computational Efﬁciency of Observation Spaces
This experiment evaluates the computational efﬁciency of
the LLVM environment observation and reward spaces. We
recorded 1M wall times of each using random trajectories.
Table III summarizes the results. There is a 192×range
in observation space times, demonstrating a tradeoff between
observation space computational cost and ﬁdelity; and 4727×
range in reward space times, motivating the development of
fast approximate proxy rewards and cost models [30], [31].
C. Autotuning LLVM Phase Ordering
We evaluate various autotuning techniques on the LLVM
phase ordering task to demonstrate the ease and speed of
CompilerGym. We use the following autotuning techniques:
Greedy search, which at each step evaluates all possible actions
7
Service Startup Environment Initialization Environment Step
Cost p50 p99 µ Cost p50 p99 µ Cost p50 p99 µ
Autophase [4] — — — — O(n) 22.4ms 388.4ms 53.3ms O(nm) 71.0ms 2,489.8ms 205.9ms
OpenTuner [29] — — — — O(n) 269.6ms 8,515.3ms 777.5ms O(nm) 50.7ms 1,491.1ms 131.2ms
CompilerGym O(1) 119.7ms 131.8ms 120.8ms O(1)† 2.2ms 198.6ms 21.3ms O(n) 1.0ms 108.6ms 7.5ms
CompilerGym-batched ” ” ” ” ” ” ” ” ” 0.2ms 37.4ms 2.6ms
TABLE II: Computational costs of CompilerGym operations compared to prior works when computing the same actions,
observations, and rewards. After paying a one-off startup penalty, CompilerGym is 27×faster than an equivalent prior work.
This much higher throughput enables training larger models with larger datasets. Cost denotes average-case time complexities
wrt. n the size of the program being compiled and m the number of actions in the episode. †denotes amortized cost, achieved
by caching initial environment states. p50 and p99 denote the 50th and 99th percentile of wall times, respectively. µ denotes
the arithmetic mean wall time. We also measured the throughput of CompilerGym when batches of actions are processed in a
single environment step, denoted CompilerGym-batched above. This improves throughput by a further 2.9×by reducing RPC
round trips, but loses intermediate observations and rewards. Measurements taken from 1M randon trajectories, evenly divided
across all benchmark datasets.
Type p50 p99 µ
LLVM-IR String 0.9ms 72.1ms 5.9ms
InstCount 70-D int64 vector 0.5ms 6.9ms 0.9ms
Autophase [4] 56-D int64 vector 0.7ms 38.0ms 3.4ms
inst2vec [13] 200-D ﬂoat vector list 15.8ms 31,847ms 738.1ms
ProGraML [14] Directed multigraph 104.5ms 14,194ms 821.5ms
Code size Int64 count 0.4ms 3.6ms 0.4ms
Binary size Int64 byte count 56.2ms 703.7ms 98.1ms
Runtime Float wall time 75.9ms 8,406ms 614.4ms
TABLE III: Computational costs of the observation and
reward spaces of LLVM environments from 1M wall time
measurements, evenly divided across all benchmark datasets.
p50 and p99 denote the 50th and 99th percentile, respectively,
and µ denotes the arithmetic mean.
and selects the action which provides the greatest reward,
terminating once no positive reward can be achieved by any
action; LaMCTS [ 32], an extension of Monte Carlo Tree
Search [ 33] that partitions the search space on the ﬂy to
focus on important search regions [ 33]; Nevergrad [ 34] and
OpenTuner [29], two black box optimization frameworks that
contain ensembles of techniques; and Random Search, which
selects actions randomly until a conﬁgurable number of steps
have elapsed without a positive reward.
We run single threaded versions of each autotuning technique
on each benchmark in the cBench [ 15] suite for one hour.
Hyperparameters for all techniques were tuned on a validation
set of 50 Csmith [ 8] benchmarks. We evaluate each technique
when optimizing for three different targets: code size, binary
size, and runtime. For runtime we use the median of three
measurements to provide the reward signals during search, and
the median of 30 measurements for ﬁnal reported values. Each
experiment was repeated 10 times.
The standard interface exposed by CompilerGym makes
it simple to integrate with third party autotuning libraries or
to develop new autotuning approaches. Table IV shows the
number of lines of code required to integrate each search
technique, and the performance achieved.
Phase ordering is challenging because the optimization space
is unbounded, high-dimensional, and contains sparse rewards.
Nevertheless, autotuning – when furnished with a sufﬁciently
Lines
of code
Geomean
code size
reduction
Geomean
binary size
reduction
Geomean
runtime
speedup
Greedy Search 10 1.053× 1.267× 1.059×
LaMCTS [32] 35 1.051× 1.273× 1.053×
Nevergrad [34] 41 1.083× 1.318× 1.093×
OpenTuner [29] 165 1.060× 1.102× 0.822×
Random Search 24 1.048× 1.278× 1.078×
TABLE IV: Lines of code required to integrate or implement
autotuning techniques for the LLVM phase ordering task, and
performance achieved when optimizing for three targets on
the cBench suite [ 15] given a 1hr search budget. Results are
compared against -Oz for size reduction and -O3 for runtime.
generous search budget – outperforms the default compiler
heuristics by tailoring the conﬁguration to each benchmark.
We note that the optimal conﬁguration differs between all
benchmarks and optimization targets.
D. Autotuning GCC Command Line Flags
For GCC we show a different aspect of the CompilerGym.
For these experiments we explore the GCC environment’s
high-dimensional action space using a number of simple
search techniques. These experiments are performed using
GCC version 11.2.0 in Docker. That version of GCC has 502
optimization settings that can be selected. We evaluate three
search techniques on the the CHstone [18] suite:
1) Random search. A random list of 502 integers from the
allowable range is selected at each step.
2) Hill climbing search. At each step a small number of
random changes are made to the current choices. If this
improves the objective then the current state is accepted and
future steps modify from there.
3) Genetic algorithm (GA). A population of 100 ran-
dom choices is maintained. We use the Python library
geneticalgorithm [35] with its default parameters.
Table V shows the geometric mean of the object code size
objective across the benchmarks in CHstone [ 18], averaged
over 3 searches. Each search was allowed 1000 compilations.
8
Lines of code Geomean binary size
reduction
Genetic Algorithm [35] 27 1.27×
Hill Climbing 14 1.04×
Random Search 9 1.21×
TABLE V: Lines of code required to integrate or implement
autotuning techniques for the GCC ﬂag tuning task, and
performance achieved when optimizing the CHstone suite [ 18],
given a search budget of 1000 compilations per benchmark.
Results are compared against -Os.
Fig. 7: A sweep over a particular conﬁguration in the Com-
pilerGym provided search space for point-wise addition on a
GPU using loop_tool to generate CUDA.
E. Autotuning CUDA Loop Nests
The loop_tool environment provides an easily accessible
interface to being exploring the landscape of GPU optimizations.
Tuning a simple space by searching threading and then sizing
the inner loop reaches 73.5% of theoretical peak performance
on our GP100 test hardware ( ∼6e10 FLOPs or ∼750GB/s for
two 4-byte ﬂoating point reads and one write), and parity with
PyTorch performance on the same operation across a variety
of problem sizes. Figure 7 shows the results for different loop
conﬁgurations, demonstrating potentially useful hardware and
compiler characteristics, notably a drop in performance near
100k threads.
F . Learning a Cost Model using the State Transition Dataset
Auxiliary tasks are commonly used in reinforcement learn-
ing to produce better representation learning and help with
downstream tasks [ 36], [ 37]. This experiment demonstrates
using the State Transition Dataset (Section III-F) to learn a
cost model of instruction count from a graph representation of
program state.
We implemented a Gated Graph Neural Network [ 38] in
PyTorch [ 39] and used Mean Squared Error loss to train a
regressor to predict the instruction count of a program after
two rounds of message passing on the PROGRAML [14] graph
representation built into CompilerGym. We trained on 80% of
the State Transition Database by iterating over pairs of (graph,
instruction count) from the database. We used the remaining
20% of the database as a validation set. Figure 8 shows the
convergence of the neural network. The network achieves a
relative error of 0.025, while a naive mean prediction scores
1.393.
Fig. 8: Predicting program instruction using a Graph Neural
Networks trained on CompilerGym’s State Transition Dataset.
This shows performance on a holdout validation set as a
function of the number of training epochs.
Geomean code size reduction
Test Dataset A2C [40] APEX [41] IMPALA [42] PPO [43]
AnghaBench [16] 0.951× 0.659× 0.958× 0.776×
BLAS [17] 0.928× 0.934× 0.861× 0.906×
cBench [15] 0.804× 0.698× 0.814× 0.964×
CHStone [18] 0.823× 0.704× 0.707× 1.014×
CLgen [19] 0.950× 0.687× 0.916× 0.843×
Csmith [8] 1.023× 0.692× 1.144× 1.245×
GitHub [14] 0.975× 0.987× 0.976× 0.984×
Linux kernel 0.987× 0.998× 0.983× 0.995×
llvm-stress [11] 0.838× 0.493× 0.736× 0.097×
MiBench [20] 0.996× 0.996× 0.996× 1.000×
NPB [21] 0.961× 0.816× 0.958× 0.923×
OpenCV 0.976× 0.969× 0.986× 0.945×
POJ-104 [22] 0.778× 0.651× 0.805× 0.801×
TensorFlow [23] 0.976× 0.976× 0.966× 0.933×
TABLE VI: Comparison of four reinforcement learning al-
gorithms, trained for 100k episodes on Csmith [ 8] programs,
when evaluated on datasets from a range of program domains.
The programs used for testing on Csmith are different from
those used for training. Results are compared to -Oz.
G. Reinforcement Learning for LLVM Phase Ordering
CompilerGym offers seamless integration with third party
reinforcement learning frameworks. For example, by changing
a single parameter value in Listing 2 we can use any of the
26 reinforcement learning algorithms included in RLlib [9].
We use CompilerGym to replicate the LLVM phase ordering
environment used in [ 4]. Speciﬁcally: we ﬁx episode lengths
to 45 steps, use the same observation space comprising a
feature vector concatenated with a histogram of the agent’s
previous actions, and we use a subset of the full action space 1.
We note that each of these modiﬁcations to the base LLVM
environment can be achieved using the wrapper classes built
into CompilerGym (Section III-C). Our environment differs
from [4] in that we use a code size reward signal rather than
simulated cycle counts.
We train three different reinforcement learning algorithms
for 100k episodes and periodically evaluate performance on a
holdout validation set. We use Csmith to generate both training
and validation sets, as in [4].
Table VI shows the performance of the trained agents when
evaluated on a random 50 programs from each of the datasets
available out of the box in CompilerGym. 3 of the 4 algorithms
1We use 42 actions (out of 124 total) rather than the 45 actions used in [ 4]
as three of the actions have been removed in recent versions of LLVM.
9
Training Set
Csmith [8] Github [14] TensorFlow [23]
Test Set
Csmith [8] 1.245× 0.567× 0.723×
Github [14] 0.984× 0.981× 0.995×
TensorFlow [23] 0.932× 0.950× 0.998×
TABLE VII: CompilerGym includes millions of programs to
train on that can be selected by simply specifying the name of
the dataset(s) to use. Here we cross-validate the generalization
performance of a PPO [ 43] agent by varying the training and
test sets. The row indicates the dataset used for training, the
column indicates the dataset used for testing. The values are
geomean code size reduction relative to -Oz.
achieve positive results when generalizing to programs within
the same domain (Csmith), but only PPO [43] is able to achieve
a positive score on two of the 13 other datasets. This highlights
the challenge of generalization across program domains.
H. Effect of Training Set on RL
The generalization of reinforcement learning agents across
domains is the subject of active research [ 44], [45], [46]. As
demonstrated in the previous experiment, the performance of
agents trained on one dataset can differ wildly on datasets
from other domains. We evaluate the effect of training set on
generalization by training a PPO [43] agent on different training
sets and then evaluating their generalization performance on test
sets from different domains. All other experimental parameters
are as per the previous experiment.
Table VII shows the results. As can be seen, each algorithm
performs best when generalizing to benchmarks from within
the same dataset, suggesting the importance of training on
benchmarks across a wide range of program domains.
I. Effect of Program Representation on Learning
Representation learning and feature engineering is an area
of much research [ 47], [48], [2]. CompilerGym environments
provide multiple state representations for each environment.
We evaluate the performance of two different program repre-
sentations, and their performance when concatenated with a
histogram of the agent’s previous actions, as used in [ 4]. We
use the same experimental setup as in the prior sections.
The results are shown in Figure 9. In both cases stronger
performance is achieved when coupling the program repre-
sentation with a histogram of the agent’s previous actions.
The Autophase representation encodes more attributes of the
structure of programs than InstCount and achieves greater
performance. We believe that representation learning is one of
the most exciting areas for future research, and CompilerGym
provides the supporting infrastructure for this research.
VIII. R ELATED WORK
We present a suite of tools for compiler optimization research.
Other compiler research tools include OpenTuner [ 29] and
YaCoS [49], autotuning frameworks that include an ensemble
of techniques for compiler optimizations; cTuning [ 50], a
framework for distributing autotuning results; TenSet [ 51] and
1e+03 1e+04 1e+05
#. training episodes (log)
0.25
0.50
0.75
1.00Mean codesize reduction
Autophase w. hist
Autophase
InstCount w. hist
InstCount
Fig. 9: The convergence rate and ﬁnal performance of learned
agents depends on the observation space. CompilerGym
includes several observation spaces out of the box. By changing
one line of code we trained four PPO [ 43] agents on different
observation spaces. This plot shows the performance on a
holdout validation set as a function of the number of training
episodes. We applied a Gaussian ﬁlter ( σ = 5) to aid in
visualizing the trends.
LS-CAT [52], large-scale performance datasets suitable for
ofﬂine learning; and ComPy-Learn [ 53], a library of program
representations for LLVM. CompilerGym has a broader set
of features than these prior works, providing several compiler
problems, program representations, optimization targets, and
ofﬂine datasets all in a single package.
There is a growing body of research that applies AI
techniques to compilers optimizations [ 2]. Many approaches
have been proposed to phase ordering, including collaborative
ﬁltering [ 54], design space exploration [ 55], and Bayesian
Networks [ 56]. Even removing passes from standard opti-
mization pipelines has been shown to sometimes improve
performance [57]. Autophase [4] and CORL [ 58] use reinforce-
ment learning to tackle the LLVM phase ordering problem.
Both works identify generalization across programs as a key
challenge. Our work aims to accelerate progress on this problem
by combining several observation spaces with millions of
training programs to serve as a platform for research.
Other reinforcement learning compiler works include
MLGO [3] which learns a policy for a function inling heuristic,
NeuroVectorizer [5] which formulates instruction vectorization
as single-step environments, and PolyGym [ 59] which targets
Polyhedral loop transformations. Compared to these works, the
search spaces in CompilerGym environments are far larger.
CompilerGym is not limited to reinforcement learning.
Prior work has cast compiler optimization tasks as supervised
learning problems using classiﬁcation to select optimization
decisions [60], [2] or regression to learn cost models [ 61], [30],
[31]. CompilerGym is as an ideal platform for gathering the
data to train and evaluate these approaches, including both
ofﬂine datasets and the infrastructure to generate new ones.
IX. C ONCLUSIONS
We aim to lower the barrier-to-entry to compiler optimization
research. We present CompilerGym, a suite of tools that
removes the signiﬁcant engineering investment required try out
new ideas on production compiler problems.
10
REFERENCES
[1] Amir H Ashouri, William Killian, John Cavazos, Gianluca Palermo, and
Cristina Silvano. A Survey on Compiler Autotuning using Machine
Learning. CSUR, 51(5), 2018.
[2] Hugh Leather and Chris Cummins. Machine Learning in Compilers:
Past, Present and Future. In FDL, 2020.
[3] Mircea Troﬁn, Yundi Qian, Eugene Brevdo, Zinan Lin, Krzysztof
Choromanski, and David Li. MLGO: a Machine Learning Guided
Compiler Optimizations Framework. arXiv:2101.04808, 2021.
[4] Ameer Haj-Ali, Qijing Huang, William Moses, John Xiang, John
Wawrzynek, Krste Asanovic, and Ion Stoica. Autophase: Juggling hls
phase orderings in random forests with deep reinforcement learning. In
MLSys, 2020.
[5] Ameer Haj-Ali, Nesreen K Ahmed, Ted Willke, Yakun Sophia Shao, Krste
Asanovic, and Ion Stoica. Neurovectorizer: End-to-end vectorization
with deep reinforcement learning. In CGO, 2020.
[6] Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles
Sutton. A Survey of Machine Learning for Big Code and Naturalness.
CSUR, 51(4), 2018.
[7] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider,
John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym.
arXiv:1606.01540, 2016.
[8] Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. Finding and
Understanding Bugs in C Compilers. In PLDI, 2011.
[9] Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox,
Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. RLlib:
Abstractions for Distributed Reinforcement Learning. In ICML, 2018.
[10] William M McKeeman. Differential Testing for Software. Digital
Technical Journal, 10(1), 1998.
[11] Chris Lattner and Vikram Adve. LLVM: A Compilation Framework for
Lifelong Program Analysis & Transformation. In CGO, 2004.
[12] Yang Chen, Yuanjie Huang, Lieven Eeckhout, Grigori Fursin, Liang Peng,
Olivier Temam, and Chengyong Wu. Evaluating Iterative Optimization
Across 1000 Datasets. In PLDI, 2010.
[13] Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoeﬂer. Neural
Code Comprehension: A Learnable Representation of Code Semantics.
In NeurIPS, 2018.
[14] Chris Cummins, Zacharias Fisches, Tal Ben-Nun, Torsten Hoeﬂer,
Michael O’Boyle, and Hugh Leather. ProGraML: A Graph-based Program
Representation for Data Flow Analysis and Compiler Optimizations. In
ICML, 2021.
[15] Grigori Fursin, John Cavazos, Michael O’Boyle, and Olivier Temam.
MiDataSets: Creating the conditions for a more realistic evaluation of
iterative optimization. In HiPEAC, 2007.
[16] Anderson Faustino da Silva, Bruno Conde Kind, Jos ´e Wesley
de Souza Magalh ˜aes, Jer ˆonimo Nunes Rocha, Breno Campos Ferreira
Guimaraes, and Fernando Magno Quin ˜ao Pereira. AnghaBench: A Suite
with One Million Compilable C Benchmarks for Code-Size Reduction.
In CGO, 2021.
[17] Chuck L Lawson, Richard J. Hanson, David R Kincaid, and Fred T.
Krogh. Basic Linear Algebra Subprograms for Fortran Usage. TOMS,
5(3), 1979.
[18] Yuko Hara, Hiroyuki Tomiyama, Shinya Honda, Hiroaki Takada, and
Katsuya Ishii. CHStone: A Benchmark Program Suite for Practical
C-based High-Level Synthesis. In ISCAS, 2008.
[19] Chris Cummins, Pavlos Petoumenos, Zheng Wang, and Hugh Leather.
Synthesizing Benchmarks for Predictive Modeling. In CGO, 2017.
[20] Matthew R Guthaus, Jeffrey S Ringenberg, Dan Ernst, Todd M Austin,
Trevor Mudge, and Richard B Brown. MiBench: A Free, Commercially
Representative Embedded Benchmark Suite. In WWC, 2001.
[21] David Bailey, Tim Harris, William Saphir, Rob Van Der Wijngaart, Alex
Woo, and Maurice Yarrow. The NAS Parallel Benchmarks 2.0. Technical
report, Technical Report NAS-95-020, NASA Ames Research Center,
1995.
[22] Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional Neural
Networks Over Tree Structures for Programming Language Processing.
In AAAI, 2016.
[23] Mart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis,
Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving,
Michael Isard, et al. TensorFlow: A System for Large-Scale Machine
Learning. In OSDI, 2016.
[24] Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu,
Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen,
et al. Ansor: Generating High-Performance Tensor Programs for Deep
Learning. In OSDI, 2020.
[25] Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris,
Fr´edo Durand, and Saman Amarasinghe. Halide: A Language and
Compiler for Optimizing Parallelism, Locality, and Recomputation in
Image Processing Pipelines. In PLDI, 2013.
[26] Bram Wasti. loop tool. https://github.com/facebookresearch/loop tool,
2021.
[27] Christopher JCH Watkins and Peter Dayan. Q-Learning. Machine
learning, 8(3-4), 1992.
[28] Vijay R Konda and John N Tsitsiklis. Actor-Critic Algorithms. In
NeurIPS, 2000.
[29] Jason Ansel, Shoaib Kamil, Kalyan Veeramachaneni, Jonathan Ragan-
Kelley, Jeffrey Bosboom, Una-May O’Reilly, and Saman Amarasinghe.
OpenTuner: An Extensible Framework for Program Autotuning. In PACT,
2014.
[30] Charith Mendis, Alex Renda, Saman Amarasinghe, and Michael Carbin.
Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation
using Deep Neural Networks. In ICML, 2019.
[31] Benoit Steiner, Chris Cummins, Horace He, and Hugh Leather. Value
Learning for Throughput Optimization of Deep Learning Workloads. In
MLSys, 2021.
[32] Linnan Wang, Rodrigo Fonseca, and Yuandong Tian. Learning Search
Space Partition for Black-Box Optimization using Monte Carlo Tree
Search. In NeurIPS, 2020.
[33] Guillaume Chaslot, Sander Bakkes, Istvan Szita, and Pieter Spronck.
Monte-Carlo Tree Search: A New Framework for Game AI. AIIDE, 8,
2008.
[34] Jeremy Rapin and Olivier Teytaud. Nevergrad - A Gradient-Free
Optimization Platform. https://github.com/facebookresearch/nevergrad,
2018.
[35] Ryan Solgi. geneticalgorithm. https://pypi.org/project/geneticalgorithm/,
2020.
[36] Marc G. Bellemare, Will Dabney, Robert Dadashi, Adrien Ali Ta ¨ıga,
Pablo Samuel Castro, Nicolas Le Roux, Dale Schuurmans, Tor Lattimore,
and Clare Lyle. A Geometric Perspective on Optimal Representations
for Reinforcement Learning. CoRR, abs/1901.11530, 2019.
[37] Max Jaderberg, V olodymyr Mnih, Wojciech Marian Czarnecki, Tom
Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Re-
inforcement Learning with Unsupervised Auxiliary Tasks. CoRR,
abs/1611.05397, 2016.
[38] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated
Graph Sequence Neural Networks. arXiv:1511.05493, 2015.
[39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury,
Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca
Antiga, et al. PyTorch: An Imperative Style, High-performance Deep
Learning Library. arXiv:1912.01703, 2019.
[40] V olodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves,
Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.
Asynchronous Methods for Deep Reinforcement Learning. In ICML,
2016.
[41] Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo
Hessel, Hado Van Hasselt, and David Silver. Distributed Prioritized
Experience Replay. In ICML, 2018.
[42] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad
Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning,
et al. IMPALA: Scalable Distributed Deep-RL with Importance Weighted
Actor-Learner Architectures. In ICML, 2018.
[43] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg
Klimov. Proximal Policy Optimization Algorithms. arXiv:1707.06347,
2017.
[44] Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman.
Quantifying Generalization in Reinforcement Learning. In ICML, 2019.
[45] Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving
Generalization in Reinforcement Learning with Mixture Regularization.
In NeurIPS, 2020.
[46] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley,
Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and
Jasper Snoek. Can You Trust Your Model’s Uncertainty? Evaluating
Predictive Uncertainty under Dataset Shift. In NeurIPS, 2019.
11
[47] Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi.
Learning and Evaluating Contextual Embedding of Source Code. In
ICML, 2020.
[48] Hong Jin Kang, Tegawend ´e F Bissyand ´e, and David Lo. Assessing the
Generalizability of code2vec Token Embeddings. In ASE, 2019.
[49] Andr´e Felipe Zanella, Anderson Faustino da Silva, and Fernando Magno
Quint˜ao. YaCoS: a Complete Infrastructure to the Design and Exploration
of Code Optimization Sequences. In SBLP, 2019.
[50] Grigori Fursin. Collective Tuning Initiative: Automating and Accelerating
Development and Optimization of Computing Systems. In GCC
Developers’ Summit, 2009.
[51] Lianmin Zheng, Ruochen Liu, Ameer Haj Ali, Junru Shao, Tianqi Chen,
Joseph E Gonzalez, and Ion Stoica. TenSet: A Large-scale Program
Performance Dataset for Learned Tensor Compilers. In NeurIPS, 2021.
[52] Lars Bjertnes, Jacob O Tørring, and Anne C Elster. LS-CAT: A Large-
Scale CUDA AutoTuning Dataset. arXiv:2103.14409, 2021.
[53] Alexander Brauckmann, Andr´es Goens, and Jeronimo Castrillon. ComPy-
Learn: A toolbox for exploring machine learning representations for
compilers. In FDL, 2020.
[54] Stefano Cereda, Gianluca Palermo, Paolo Cremonesi, and Stefano Doni.
A Collaborative Filtering Approach for the Automatic Tuning of Compiler
Optimisations. In LCTES, 2020.
[55] Ricardo Nobre, Luiz GA Martins, and Jo ˜ao MP Cardoso. A graph-based
iterative compiler pass selection and phase ordering approach. In LCTES,
2016.
[56] Amir Hossein Ashouri, Giovanni Mariani, Gianluca Palermo, Eunjung
Park, John Cavazos, and Cristina Silvano. COBAYN: Compiler
Autotuning Framework Using Bayesian Networks. TACO, 13(2), 2016.
[57] Kyriakos Georgiou, Craig Blackmore, Samuel Xavier-de Souza, and
Kerstin Eder. Less is more: Exploiting the standard compiler optimization
levels for better performance and energy consumption. In SCOPES, 2018.
[58] Rahim Mammadli, Ali Jannesari, and Felix Wolf. Static Neural Compiler
Optimization via Deep Reinforcement Learning. In LLVM-HPC, 2020.
[59] Alexander Brauckmann, Andr ´es Goens, and Jeronimo Castrillon. A
Reinforcement Learning Environment for Polyhedral Optimizations.
arXiv:2104.13732, 2021.
[60] Chris Cummins, Pavlos Petoumenos, Zheng Wang, and Hugh Leather.
End-to-end Deep Learning of Optimization Heuristics. In PACT, 2017.
[61] Rahim Mammadli, Marija Selakovic, Felix Wolf, and Michael Pradel.
Learning to Make Compiler Optimizations More Effective. In MAPS,
2021.
12

Compiler Auto-Vectorization with Imitation Learning
Charith Mendis
MIT CSAIL
charithm@mit.edu
Cambridge Yang
MIT CSAIL
camyang@csail.mit.edu
Yewen Pu
MIT CSAIL
yewenpu@mit.edu
Saman Amarasinghe
MIT CSAIL
saman@csail.mit.edu
Michael Carbin
MIT CSAIL
mcarbin@csail.mit.edu
Abstract
Modern microprocessors are equipped with single instruction multiple data (SIMD)
or vector instruction sets which allow compilers to exploit ﬁne-grained data level
parallelism. To exploit this parallelism, compilers employ auto-vectorization
techniques to automatically convert scalar code into vector code. Larsen & Amaras-
inghe (2000) ﬁrst introduced superword level parallelism (SLP) based vectorization,
which is a form of vectorization popularly used by compilers. Current compilers
employ hand-crafted heuristics and typically only follow one SLP vectorization
strategy which can be suboptimal. Recently, Mendis & Amarasinghe (2018) for-
mulated the instruction packing problem of SLP vectorization by leveraging an
integer linear programming (ILP) solver, achieving superior runtime performance.
In this work, we explore whether it is feasible to imitate optimal decisions made by
their ILP solution by ﬁtting a graph neural network policy. We show that the learnt
policy, Vemal, produces a vectorization scheme that is better than the well-tuned
heuristics used by the LLVM compiler. More speciﬁcally, the learnt agent produces
a vectorization strategy that has a 22.6% higher average reduction in cost compared
to the LLVM compiler when measured using its own cost model, and matches the
runtime performance of the ILP based solution in 5 out of 7 applications in the
NAS benchmark suite.
1 Introduction
Modern microprocessors have introduced single instruction multiple data (SIMD) or vector units (e.g.
Intel x86 A VX extensions1) to accelerate execution of performance critical applications by performing
computations on multiple data items in parallel. In order to use these vector units, programmers
must either code using platform speciﬁc vector assembly instructions (Figure 1(c)), which is tedious,
error-prone and results in non-portable code or use existing compiler auto-vectorization techniques
to automatically discover data parallel portions of programs and to transform scalar instructions
(Figure 1(a)) in such regions into vector instructions (Figure 1(b),(c)).
Larsen & Amarasinghe (2000) ﬁrst showed how to perform compiler auto-vectorization using ﬁne
grained parallelism available in programs at the instruction level (superword level parallelism)
targeting ﬁxed-width vector instructions available in modern microprocessors. Superword level
parallelism (SLP) based vectorization provided a more general alternative to loop vectorization
techniques proposed earlier (Allen & Kennedy, 1987; Sreraman & Govindarajan, 2000).
In order to exploit SLP for vectorization, compilers need to select a set of scalar instructions which
can be executed in parallel and merge them into a single vector instruction. It is shown that this
process reduces to the optimal subset selection problem, which is known to be NP-hard. Therefore,
many existing SLP vectorization schemes are driven by hand-crafted heuristics (Liu et al., 2012;
1https://software.intel.com/en-us/articles/introduction-to-intel-advanced-vector-extensions
33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.
 a[0] = b[0] + c[0]
 a[1] = b[1] + c[1]  {a[0],a[1]} = {b[0],b[1]} + {c[0],c[1]}
movdqa xmm0, c
paddq  xmm0, b
movdqa a, xmm0
(a) (b) (c) 
Figure 1: (a) scalar code storing result of adding elements loaded from array b,c into array a (b)
compiler auto-vectorized code; expressions inside {.} are executed in parallel. (c) manually written
vector assembly instructions / executable code generated by the compiler for auto-vectorized code (b)
Shin et al., 2003, 2005, 2002), which can be suboptimal. Recently, goSLP (Mendis & Amarasinghe,
2018) introduced a SLP vectorization strategy with certain optimality guarantees guided by an Integer
Linear Programming (ILP) solver. It has shown superior vectorization schemes compared to heuristics
guided solutions, achieving end-to-end speedups on well-known compiler benchmark suites.
In this work, we propose a technique to learn how to vectorize by imitating the solution presented
in goSLP (Mendis & Amarasinghe, 2018). Speciﬁcally, we formulate the decision procedure as a
Markov Decision Process (MDP) and then use the DAGGER algorithm (Ross et al., 2011) to collect
traces on how the ILP solver solves the SLP vectorization problem. We use these trace aggregates
as supervision to train a parametrized policy modeled by a Gated Graph Neural Network (Li et al.,
2015; Allamanis et al., 2017).
We show that the learnt policy,Vemal, outperforms one of the well-tuned heuristics used in the LLVM
compiler (Lattner & Adve, 2004) both in terms of static metrics and dynamic runtime performance,
while matching the performance of goSLP in 5 out of 7 programs in the NAS benchmark suite held
out for testing.
Speciﬁcally, we make the following contributions:
• Formulation of the SLP vectorization problem as a Markov Decision Process (MDP), where
the vectorization strategy is constructed sequentially.
• Modeling of the agent policy that solves the MDP as a Gated Graph Neural Network
(GGNN) (Li et al., 2015; Allamanis et al., 2017) and using imitation learning to train the
policy network to mimic the optimal decisions made by the ILP solver.
• Evaluation of the learnt policy on representative compiler benchmark suites (SPEC2006fp
C/C++ (Henning, 2006), SPEC2017fp C/C++ (Bucek et al., 2018) and NAS benchmark
suites (Bailey et al., 1991)). Speciﬁcally, we show that the learnt policy has an average static
cost reduction which is 22.6% higher than LLVM as measured by LLVM’s own static cost
model. We also show that the learnt policy achieves a geometric mean runtime speedup
of 1.015×on the NAS benchmark suite (held out during training) compared to LLVM,
matching runtime speedups of goSLP in 5 out of 7 applications.
In summary, we have shown that it is possible to learn end-to-end compiler optimization policies
which surpass the performance of hand-crafted compiler heuristics by imitating an optimal solution.
2 The SLP Vectorization Problem
Superword Level Parallelism (SLP) is a type of ﬁne-grained parallelism ﬁrst introduced by Larsen &
Amarasinghe (2000) that is suitable for vector code generation. SLP is available in cases where two
or more scalar instructions are independent (the second instruction does not require the output of the
ﬁrst instruction, and vice-versa) and are isomorphic (of the same instruction type, such as addition
or subtraction). For example, consider the code snippet from Figure 2(a). It computes intermediate
values A1 up to A3 by dividing values loaded from array L. Next, these intermediate values are used
in a series of subtraction operations and the results are stored in array S, which is disjoint from L.
Both the division and subtraction instruction groups, adjacent loads and stores are independent and
isomorphic. Hence, they are amenable to SLP vectorization.
Such scalar instructions can be executed in parallel and hence can be packed to form a single vector
instruction. We name a set of scalar instructions which are packed as a vector pack. For example
considering the code snippet shown in Figure 2(a), we can form a vector pack out of A1 and A2 and
perform the division in vector form as: {A1,A2}= {L[5],L[6]}/{L[2],L[3]}.
Forming vector instructions by packing multiple scalar instructions can lead to better runtime perfor-
mance. However, not all packing opportunities are proﬁtable. For instance, if a set of instructions is
packed while their operands remain scalar, there will be an additional packing overhead to bring the
2
operands also into packed form. On the other hand, if the output of packed instructions is used by
other scalar instructions, there will be an unpacking overhead.
The task of the compiler is to ﬁnd the most proﬁtable set of vector packs to form among all available
packing opportunities. This combinatorial optimization task is not easy even in the case of pairwise
instruction packing (Mendis & Amarasinghe, 2018) and is NP-hard.
2.1 Motivating Example
(a) Code example (b) LLVM’s SLP algorithm (c) goSLP’s optimal algorithm
  A1   = L[5] / L[2]
  A2   = L[6] / L[3]
  A3   = L[7] / L[4]
  S[1] = L[1] - A2
  S[2] = L[2] - A3
  S[3] = L[3] - A1
{L[6],L[7]} {L[3],L[4]}
{A2,A3}
{S[1],S[2]}
{L[1],L[2]}
L[2]
L[3]
Vector instructions : 3
Overhead instructions: 3
Vector instructions : 5
Overhead instructions: 2
{S[2],S[3]}
{L[2],L[3]} {A3,A1}
{L[5],L[7]} {L[2],L[4]}
L[2]
Figure 2: Comparison of SLP vectorization strategies (a) code example; S and Lare two disjoint
arrays, (b)-(c) show dependency graphs of packed instructions under each instruction packing strategy
(b) under LLVM’s SLP vectorization algorithm (c) optimal packing. Solid arrows show dependencies.
Groupings with solid circles show vector packs. Groupings with dotted circles show packs created
using overhead packing instructions and dotted lines show unpacking of values from vector packs.
We compare LLVM’s greedy packing strategy with goSLP’s optimal packing strategy for the code
snippet shown in Figure 2(a) to illustrate how instruction packing affects the quality of vectorization.
Figure 2(b) shows the packing solution of LLVM’s SLP vectorization algorithm. LLVM’s greedy
algorithm chooses to vectorize adjacent stores S[2] and S[3] ﬁrst. Next, it tries to pack the operands
of already packed values recursively. Using this procedure, operands of {S[2],S[3]}are packed next,
forming vector packs {L[2],L[3]}and {A1,A3}. However, operands of {A1,A3}cannot be packed,
since they access non-adjacent memory locations, forcing LLVM to emit two overhead packing
instructions. Further, L[2] needs to be unpacked before it can be repacked with L[4]. Altogether,
LLVM’s strategy yields 3 vector instructions while incurring a cost of 3 overhead instructions.
Figure 2(c) shows the optimal packing solution found by goSLP using an ILP solver. The optimal
strategy creates vector pack {S[1],S[2]}instead of {S[2],S[3]}, which proves to be globally optimal.
Altogether, this strategy yields 5 vector instructions and requires only 2 overhead instructions.
In this paper, we evaluate whether it is possible to learn a policy that imitates goSLP’s ILP solution
for the pairwise instruction packing problem. Note that, we are not aiming to learn a general purpose
ILP solver, but to imitate the ILP solution for this problem domain. We will now formalize the
pairwise instruction packing problem.
2.2 The Pairwise Instruction Packing Problem
Let I = {I1,I2,...,I n−1,In,Iϵ}be the set of n+ 1 instructions, where Iϵ is an artiﬁcial empty
Instruction. A valid instruction packing ¯P = {P1 ...P m}is a collection of pairs:
Pi ∈I×I ,
⋃
i
Pi = I
that satisﬁes two kinds of packing constraints: CI (within pairs) and CII (between pairs).
Within PairA legal pack P = (Ii,Ij) must satisfy the within pair constraint CI(P)
• Ii and Ij must be isomorphic: perform the same operation on same data types which results
in values of the same type.
• Ii and Ij must be independent: Ii and Ij cannot be directly or transitively dependent, where
they cannot be reachable by one another in the same data-dependency graph.
3
• If Ii and Ij require reordering, it should be possible under the hardware memory model.
• If Ii and Ij access memory they must access adjacent memory locations.
• Ii ≺Ij under some ordering of statements I1 ≺···≺ In ≺Iϵ
Between Pairs packs Pi and Pj can both be created iff they satisfy the between pairs constraint
CII(Pi,Pj):
• Pi and Pj are schedulable: there shouldn’t be any circular dependencies between the two
packs. For example, if Ii,1,Ii,2 ∈Pi and Ij,1,Ij,2 ∈Pj, it shouldn’t be the case that
Ii,1 δIj,1 and Ij,2 δIi,2, where δdenotes dependency.
• Pi and Pj are not overlapping: ∀Ii ∈Pi =⇒ Ii /∈Pj. That is, a single statement can only
belong to one pack.
We write CII(Pi,Pj), if the packs satisfy the between pairs constraint.
Static Cost.We evaluate the efﬁcacy of the packing strategies using LLVM’s static cost model. The
static cost model assigns costs for each vector and scalar instruction. The total cost of a block of code
is calculated as the addition of these per instruction costs.
Objective.Let F be the performance (measured using static cost) of executing the block of code with
vectorized instructions formed by a packing. The goal of any SLP vectorization scheme is to ﬁnd a
packing strategy P that minimizes the cost (argmin ¯P F( ¯P)) subject to constraints CI and CII.
3 Learnt Solution - Vemal
Our learnt vectorization policy, Vemal, imitates goSLP’s (Mendis & Amarasinghe, 2018) ILP solution
to perform SLP vectorization. We cast the pairwise instruction packing problem as a Markov Decision
Process (MDP), which we solve by a neural-network policy that imitates the ILP solution via imitation
learning using the DAGGER algorithm (Ross et al., 2011).
3.1 MDP formulation
We formulate the pairwise instruction packing problem as a MDP by iteratively forming one vector
pack at a time following a particular instruction traversal policy:
• Bottom-up Traversal.The learnt policy starts making packing decisions traversing the
function in reverse, starting from the ﬁnal instruction with valid packing candidates. At
the end of each iteration i, we choose a predecessor of Ii with valid packing candidates to
consider for packing next.
• Top-down Traversal.The learnt policy starts making packing decisions traversing down-
wards from the ﬁrst instruction with valid packing candidates. At the end of each iteration i,
we choose a successor of Ii with valid packing candidates to consider for packing next.
The instruction traversal policy selects a speciﬁc instruction Ii in each iteration to be considered for
packing next. The learnt vectorization policy decides which valid packing candidate instruction Ij
should Ii be packed with. We deﬁne the components of our MDP as follows:
State (Si), A state S in our MDP is a tuple, S = ( Ii,PB ). Here, Ii represents the instruction
selected by the ﬁxed traversal order; PB = {P1 ...P k}represents a set of formed packs.
Start State (S1), S1 = (I1,{}). The current instruction to consider for packing is I1 and there are
no vector packs formed yet. The traversal policy decides I1.
Action (A), On a given state S = (Ii,PB ), the set of legal 2 actions A[S] is given as follows:
A[(Ii,PB )] = {Ij such that Pi = {Ii,Ij},CI(Pi) and ∀Pj ∈PB, CII(Pi,Pj)}
Transition (T), On a given state (Ii,PB ) and action A= I, the transition function T simply adds
the newly formed pack to PB: T[(Ii,PB ),I] = (Ii+1,PB ∪{Ii,I}). The traversal policy decides
on Ii+1. If there are no instructions with valid packing opportunities Ii+1 = Iϵ.
Reward (R), We deﬁne the reward function as follows:
R((Ii,PB )) =
{0 if Ii ̸= Iϵ
F(PB) if Ii = Iϵ
2satisﬁes CI and CII
4
  I1 : A1   = C1  / L[1]
  I2 : A2   = C2  / L[2]
  I3 : A3   = L[4] / L[2]
  
L[1] L[2] PC
A1 A2 A3F UP
L[1] L[2]C
A1 A2F
P
A3 UP
(a) (b) (c)
Focus Edge Depedency Edge (position 1) Depedency Edge (position 2)
Pack Edge Possible Pack Edge
Figure 3: Graph formulation of MDP states for the code snippet shown in (a) under the forward
traversal policy. Nodes: C-Constant, F-Focus, P-Pack, UP-Unpack and all other nodes are instruction
nodes. Edges: different types are shown in the ﬁgure. Edges with arrows are directed. Figure (a)
shows the initial state and Figure (b) shows the state after packing instruction nodes {L[1],L[2]}.
3.2 Graph Formulation of the MDP State
We use a Gated Graph Neural Network (GGNN) (Li et al., 2015) as part of the policy network
modeling to make packing decisions for each state. To leverage the GGNN, we formulate the state of
our MDP as a graph as follows:
Nodes. We consider 5 types of nodes to encode the graph features of a state Si:
• Instruction Node:correspond to each instruction with at least one valid packing opportunity
or instructions which are already packed.
• Pack Node:common node representing overhead packing instructions.
• Unpack Node:common node representing overhead unpacking instructions.
• Constant Node:common node representing any constant value used by instructions.
• Focus Node:special node that is connected to the instruction that is considered for packing
in this iteration (Ii).
Edges. Following are the 4 types of edges connecting the above nodes:
• Dependency Edge:encodes if an instruction must be executed after another one in sequen-
tial order. More over, depending on the position of the arguments the instruction depends
on, a different dependency edge type is created for a maximal of 5, with further additional
arguments collapsed to the same dependency edge type 6. If a vector pack is formed and it
requires overhead packing instructions a suitable dependency edge is added between the
pack node and the two instruction nodes which form the vector pack. Similarly, if a vector
pack needs to be used by a scalar a suitable dependency edge is added between the relevant
instruction node and the unpack node. Note that, all dependency edges are directed.
• Possible Pack Edge:encodes whether two instructions can be packed together.
• Packed Edge:encodes instructions that are already packed together.
• Focus Edge:the focus edge connects the focus node to the instruction node that is consid-
ered for packing. This marks the node that we are making decisions on.
We illustrate our graph formulation for the code snippet shown in Figure 3(a) assuming a forward
traversal policy. Figure 3(b) shows the initial state with focus edge directed at L[1]. Further, it shows
edges for dependencies for each operand position, possible packs ({A1,A2}, {A2,A3}, {A1,A3}
and {L[1],L[2]}) as well as mandatory packing of non-adjacent load L[4] which is used by A3.
Figure 3(c) shows the next state assuming pack {L[1],L[2]}is formed. Notice the packed edge
between L[1] and L[2] and the update of the focus edge to A1, which is considered for packing next.
3.3 Neural Network Architecture
We use a GGNN to encode the aforementioned graph representation of the MDP state. GGNNs
have shown promise in the ﬁeld of code modeling (Brockschmidt et al., 2018), being able to capture
intricate graph structures present in code sequences. GGNN maintains a hidden state vector for each
node. During forward simulation, it passes some ﬁxed round of messages between nodes through
their connecting edges. The hidden state of a node is updated on each message passing iteration by
ﬁrst aggregating the messages passed from adjacent nodes and next by running the aggregate through
a Gated Recurrent Unit (Cho et al., 2014) cell. Once message passing completes, we pass the hidden
states of neighbors connected through possible pack edges of the node selected for packing through
a multi-layer perceptron. Finally, we feed its output through a softmax layer to produce the action
probabilities indicating how likely the selected node will be packed with its neighbors.
5
3.4 Imitation Learning
Vemal uses both supervised pre-training and imitation learning using the DAGGER algorithm (Ross
et al., 2011) to imitate the packing decisions made by the ILP solver.
Supervised Pre-training.We ﬁrst precompute optimal packing decisions using the ILP solver for
the functions in our training set. We then pick a particular instruction traversal policy and create
optimal trajectories of state-action pairs using the packing decisions made by the ILP solver, whilst
updating the focus edge according to the traversal policy for each function. Here, the action is the
selection of a node for packing among all valid packing candidates. The MDP state serves as the
input to the GGNN based policy. We use cross entropy loss to train the policy by encoding the target
action as a 1-hot vector. We perform supervised pre-training for a designated amount of batches.
Imitation learning with DAGGER . In the imitation learning phase, we sample a speciﬁc roll-out
of the GGNN policy using the chosen instruction traversal policy for a randomly sampled batch of
functions. For every MDP state visited by the GGNN policy, we encode the packing decisions already
made into the ILP formulation, and use the ILP solver to provide optimal packings for the remaining
instructions. We use this to ﬁnd the optimal action for every state visited by the GGNN policy. The
DAGGER algorithm augments the current dataset to include these state-action pairs. We then train on
this augmented dataset similar to supervised pre-training. At each epoch, we continue to augment the
dataset using the aforementioned strategy. This allows the GGNN policy to learn how to rectify its
policy in cases where it falls out of the optimal trajectory.
3.5 Function Partitioning
Graph neural networks suffer from over-smoothing and scaling problems specially when the graph’s
diameter is large (Zhou et al., 2018). Real world functions can be arbitrarily sized, in some cases
reaching more than 190000 instructions in our dataset. To alleviate scaling problems in our formula-
tion, we partition functions based on their instruction counts and learn vectorization for each partition
separately considering them as full functions using the formulation mentioned in Sections 3.1 to 3.4.
Note that, we solve ILP problems considering each partition individually and use their solutions to
train our agent. During inference time, we take separate rollouts using our agent for each partition of
the function and ﬁnally merge all the decisions to form the ﬁnal vectorization scheme.
4 Dataset
We use the same set of benchmark programs from goSLP (Mendis & Amarasinghe, 2018) to train
and evaluate our imitation learning approach. Our dataset is composed of all individual functions
collected out of the benchmark programs listed in Table 1. The benchmark programs represent
ﬂoating-point C/C++ programs from SPEC2006 (Henning, 2006), SPEC2017 (Bucek et al., 2018)
and NAS (Bailey et al., 1991) benchmark suites, which are well known benchmark suites used for
evaluating compiler optimizations.
Benchmark Suite Benchmark Programs
SPEC2006 433.milc, 444.namd, 447.dealII, 450.soplex, 453.povray, 470.lbm, 482.sphinx3
SPEC2017 508.namd_r, 510.parest_r, 511.povray_r, 519.lbm_r, 538.imagick_r, 544.nab_r
NAS BT, SP, LU, MG, FT, CG, EP
Table 1: Benchmark programs used for training and testing our learnt agent
4.1 Collection
We ﬁrst compiled each source ﬁle to LLVM’s intermediate representation (IR) just before LLVM’s
existing SLP vectorizer runs. By this way, we obtain the same IR that would have been seen by the
vectorizer during an end-to-end compilation. Each source ﬁle has a number of functions and goSLP
builds ILP problems considering a single function as the vectorization unit. Hence, we collected both
the compiled LLVM IR (just before SLP vectorization) as well as the corresponding pairwise packing
opportunities for each function for all programs in our benchmark suite.
4.2 Preparation
Using the methodology outlined in Section 4.1, we collected 35635 functions in total. However, only
3981 (11.17%) functions are vectorized by goSLP. If we use all collected functions during training, it
induces a natural bias towards not vectorizing due to the imbalance in our dataset, even for functions
6
with abundant vectorizable opportunities. The goal of our learnt agent is to mimic goSLP as closely
as possible when there are vectorizable opportunities. In cases where our learnt agent suggests an
unproﬁtable scheme, it can be eliminated by a cost model similar to the current LLVM SLP vectorizer.
This asymmetric learning objective and the imbalance in our collected functions motivated us to
create the ﬁnal dataset that is biased towards functions which are vectorized by goSLP. We select all
functions which are vectorized by goSLP as well as a random subset of non-vectorized functions
such that, 80% (3981) of our dataset has functions with proﬁtable vectorization schemes and 20%
(995) do not. Finally, we split the dataset into a training set (80%) and a test set (20%) such that the
proportionality of the vectorized and non-vectorized functions remains the same for both. There are
3169 and 812 vectorized functions in our training and test respectively. Our training set does not
include any functions from the NAS benchmark suite which we use for evaluating the end-to-end
runtimes of our learnt policy.
We evaluate 2 partitioning sizes, namely partitioning each function at 100 and 200 instruction counts.
For each such partition, we create a new function with only the instructions from that partition and
solve an ILP problem (goSLP’s formulation) to retrieve the set of optimal actions for the partition.
We report the ﬁnal training and test set compositions for each partitioning scheme in Table 2.
Scheme Name Partition Size Train set (partitioned functions) Test set (partitioned functions)
vectorized non-vectorized vectorized non-vectorized
p100 100 7203 1802 1776 446
p200 200 5378 1346 1357 341
Table 2: Partitioned dataset statistics
5 Training and Testing
We now explain how we train Vemal’s GGNN policy (Section 5.1) and use it in inference (Section 5.2)
for making ﬁnal vectorization decisions.
5.1 Training Setup
We learn the GGNN policy using the training set for each partition size for both backward and forward
instruction traversal policies. Initially, there are 144944 and 163618 optimal state-action pairs for
partition sizes 100 and 200 respectively under forward traversal policy. Under backward traversal the
respective numbers are 144026 and 163448.
We pre-train each network using 3000 randomly sampled batches. At the beginning of each epoch,
we randomly sample 400 of partitioned functions and augment our dataset using rollouts obtained for
those functions. We use a mixed student-teacher policy similar to that used by the original DAGGER
algorithm (Ross et al., 2011) to take rollouts, with the probability of choosing the teacher agent (ILP
solver) exponentially decayed by β = 0.9 at the beginning of each epoch. Finally, we use goSLP’s
ILP solution to compute the optimal actions for each state we encounter during rollouts.
We use 20 message passing iterations in our GGNN. We train the neural network using stochastic
gradient descent with momentum of 0.9, initial learning rate of 0.002 and with an exponentially
decaying learning rate schedule (decay of 0.95). We randomly sample 50 state-action pairs for each
batch and sample replay buffer
batch size number of batches for each epoch.
5.2 Evaluation Criteria
In order to evaluate whether our trained policy is better than LLVM’s SLP vectorization algorithm,
we use three different metrics in our experiments.
• Average cost reduction across all vectorized functions in the test set compared to scalar.
• Geometric mean speedup across all vectorized functions in the test set compared to scalar.
• Geometric mean speedup of actual runtimes for NAS benchmark suite over LLVM SLP.
For the ﬁrst two metrics we use values reported by LLVM’s cost model. For the ﬁnal metric we use
actual wall clock times. For each partition size and instruction traversal policy, we evaluate each
policy both when it uses the action with the highest probability (argmax policy) for each state as well
as when it uses the best trace among n-rollouts (multi-rollout policy).
7
6 Experimental Results
We trained all four agents – partition sizes 100 and 200 and instruction traversal policies forward and
backward – for 40 epochs. We use LLVM SLP (clang-6.0), goSLP and a random packing agent as our
baselines for comparison. Note that, we restrict goSLP and our learnt agents to only perform pairwise
packing (vectorization factor = 2), where as for LLVM SLP we do not restrict its vectorization factor
and use its implementation without any change. Random packing agent is an agent which chooses
uniformly from alternative actions for a given MDP state.
6.1 Static Results
Table 3 shows the average cost reduction and geometric mean speedup for the functions in the test set
which are vectorized by goSLP (812). We include two additional comparison points, goSLP-p100
and goSLP-p200, which are policies that solve multiple ILP problems for partitioned functions using
goSLP’s formulation and merges the decisions to perform ﬁnal vectorization.
Vectorization
Policy
Traversal
Policy # of rollouts Average Cost Reduction
(LLVM cost model)
Geo-mean Speedup
(LLVM cost model)
goSLP 23.6182 1.1226
goSLP-p100 19.4815 1.1148
goSLP-p200 21.2857 1.1193
LLVM SLP 12.1872 1.0873
random forward 1.0320 1.0150
random backward 1.0567 1.0126
p100 forward 1 13.3633 1.0833
p100 forward 10 14.9421 1.1018
p100 backward 1 9.2180 1.0685
p100 backward 10 11.3227 1.0911
p200 forward 1 13.5259 1.0829
p200 forward 10 14.7340 1.1020
p200 backward 1 9.5801 1.0693
p200 backward 10 11.5631 1.0912
Table 3: Average cost reduction and geometric mean speedups for vectorized functions in our test set
based on LLVM’s cost model under different vectorization policies
We notice that the agents using forward traversal learn a better vectorization policy than when using
backward traversal. Also, all learnt agents except agents using backward traversal surpass LLVM
SLP’s average cost reduction. This fact is magniﬁed with more rollouts, showing the efﬁcacy of our
learnt policy compared to LLVM SLP’s greedy algorithm. The best performing agent (p100 with 10
rollouts) has an average cost reduction compared to scalar which is 22.6% higher than that of LLVM.
This is in spite of the fact, LLVM is not restricted to pairwise packing.
Also, notice that partitioned versions of goSLP achieve a lower average cost reduction compared to
that of goSLP. This is because of the sub-optimality introduced by solving subproblems as opposed
to solving vectorization for the entire function. The maximum average cost reduction of learnt
agents p100 and p200 are capped at those of goSLP-p100 and goSLP-p200 respectively. We should
therefore, expect p200 to learn a better policy than p100. However, the learnt policies only have small
overall average cost reduction differences. This is because the GGNN is not as good at approximating
goSLP’s packing policy when it comes to larger graphs. This gives rise to a trade-off space between
sub-optimality of the solution and the learnability of a packing strategy at various partition sizes.
6.2 Runtime Results
We use our learnt GGNN policy for both partition sizes 100 and 200 to perform end-to-end vectoriza-
tion for the NAS benchmark suite. We use the agents learnt under forward traversal and use both
argmax and multi-rollout policy to evaluate the efﬁcacy of the learnt policy on end-to-end runtimes.
All benchmark programs are run on a Haswell Intel(R) Xeon(R) CPU E5-2680 v3 machine running
at 2.50GHz with 32kB L1 and 256kB L2 cache sizes. We use Class-A workloads in our evaluation.
We run each benchmark program 3 times and report the median as is the common reporting method
for compiler benchmarks. Figure 4 shows the runtime speedups for each program under goSLP and
our learnt policies compared to LLVM SLP. The table shows the ﬁnal geometric mean speedup for all
policies compared to LLVM SLP.
8
0.7
0.8
0.9
1
1.1
1.2
1.3
BT SP LU MG FT CG EP
goSLP p100-1 p100-10 p200-1 p200-10
Speedup compared to LLVM SLP
Policy Rollouts Speedup over
LLVM SLP
goSLP 1.041
p100 1 0.979
p100 10 1.015
p200 1 0.987
p200 10 1.003
Figure 4: Speedup of goSLP, p100 with 1 and 10 rollouts, p200 with 1 and 10 rollouts compared to
LLVM SLP for individual benchmarks in the NAS benchmark suite. The table shows the geometric
mean speedups for the entire benchmark suite.
The best performing learnt agent achieves 1.015×geometric mean speedup over all NAS benchmarks.
In fact, both p100 and p200 learnt agents were able to beat LLVM SLP’s performance with only 10
rollouts. This shows the efﬁcacy of our learnt agents on end-to-end runtime performance.
More notably, all agents except p100 with 1 rollout beat or replicate the substantial runtime speedup
of goSLP on BT benchmark over LLVM SLP. This signiﬁes that the agents have learned a non-trivial
vectorization policy not covered by LLVM SLP. Also, note that for SP and MG benchmarks all agents
consistently beat goSLP in terms of performance. Even though goSLP performs optimal packing
according to LLVM’s cost model, for these benchmarks there exist other vectorization strategies, as
uncovered by our learnt agents, which are better in terms of runtimes, but are suboptimal according
to LLVM’s cost model. This shows inaccuracies of the LLVM cost model. This provides evidence to
our hypothesis, that in future a reinforcement learning based policy with a better cost model has the
potential to learn an even better end-to-end vectorization policy than goSLP.
7 Related Work
Since the inception of vector machines, loop vectorization (Allen & Kennedy, 1987; Nuzman &
Zaks, 2008) has been introduced to speedup scientiﬁc computing kernels. Larsen & Amarasinghe
(2000) introduced superword level parallelism based vectorization targeting shorter SIMD width
machines. Subsequently many techniques have emerged suggesting better heuristics to perform SLP
vectorization (Porpodas & Jones, 2015; Porpodas et al., 2015; Liu et al., 2012; Shin et al., 2003, 2005,
2002). Recently, Mendis & Amarasinghe (2018) introduced a pairwise optimal packing algorithm
using an ILP solver for statement packing which outperforms previous greedy approaches.
There has been work to ﬁnd better compiler heuristics using machine learning (Stephenson et al.,
2003; Cummins et al., 2017). More speciﬁcally, there has been previous attempts at identifying better
heurisitics or program orders for vectorization using machine learning (Stock et al., 2012). However,
it does not provide an end-to-end learnt solution for vectorization. Reinforcement Learning has been
used to perform compiler instruction scheduling (McGovern et al., 2002) prior to the era of deep
neural networks. In this work, we have shown how to imitate an ILP based solution using a graph
neural network based policy to come up with the ﬁrst end-to-end learnt auto-vectorizer.
In our formulation, we solve a NP-hard packing problem. Previously, reinforcement learning has
been used to solve combinatorial optimization problems (Dai et al., 2017; Bello et al., 2016; Li et al.,
2018). Comparatively, we have stronger supervision through an oracle with optimal actions.
8 Conclusion
Compiler auto-vectorization allows compilers to harness ﬁne-grained parallelism within programs.
Many greedy heuristic based solutions were proposed, and recently Mendis & Amarasinghe (2018)
introduced a tractable solution with optimality guarantees using an ILP solver. Our work shows the
feasibility of learning an end-to-end vectorization policy by imitating this optimal solution and we
show that it outperforms well-tuned compiler heuristics used by the LLVM compiler. This holds out
the promise that learnt compiler optimizations can be a better alternative to hand-written counterparts
in the near future.
9
Acknowledgements
We would like to thank Darsh Shah who was initially involved with this project and all reviewers for
insightful comments and suggestions. This research was supported by DARPA D3M Award #FA8750-
17-2-0126, DARPA HACCS Award #HR0011-18-C-0059 and DARPA SDH Award #HR0011-18-3-
0007. Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are
those of the authors and do not necessarily reﬂect the views of the funding agencies.
References
Allamanis, M., Brockschmidt, M., and Khademi, M. Learning to represent programs with graphs.
arXiv preprint arXiv:1711.00740, 2017.
Allen, R. and Kennedy, K. Automatic translation of fortran programs to vector form. ACM Trans.
Program. Lang. Syst., 9(4):491–542, October 1987. ISSN 0164-0925. doi: 10.1145/29873.29875.
URL http://doi.acm.org/10.1145/29873.29875.
Bailey, D., Barszcz, E., Barton, J., Browning, D., Carter, R., Dagum, L., Fatoohi, R., Frederick-
son, P., Lasinski, T., Schreiber, R., Simon, H., Venkatakrishnan, V ., and Weeratunga, S. The
nas parallel benchmarks. Int. J. High Perform. Comput. Appl. , 5(3):63–73, September 1991.
ISSN 1094-3420. doi: 10.1177/109434209100500306. URL http://dx.doi.org/10.1177/
109434209100500306.
Bello, I., Pham, H., Le, Q. V ., Norouzi, M., and Bengio, S. Neural combinatorial optimization with
reinforcement learning. ArXiv, abs/1611.09940, 2016.
Brockschmidt, M., Allamanis, M., Gaunt, A. L., and Polozov, O. Generative code modeling with
graphs. arXiv preprint arXiv:1805.08490, 2018.
Bucek, J., Lange, K.-D., and v. Kistowski, J. Spec cpu2017: Next-generation compute benchmark.
In Companion of the 2018 ACM/SPEC International Conference on Performance Engineering,
ICPE ’18, pp. 41–42, New York, NY , USA, 2018. ACM. ISBN 978-1-4503-5629-9. doi: 10.1145/
3185768.3185771. URL http://doi.acm.org/10.1145/3185768.3185771.
Cho, K., Van Merriënboer, B., Bahdanau, D., and Bengio, Y . On the properties of neural machine
translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014.
Cummins, C., Petoumenos, P., Wang, Z., and Leather, H. End-to-end deep learning of optimization
heuristics. In 2017 26th International Conference on Parallel Architectures and Compilation
Techniques (PACT), pp. 219–232. IEEE, 2017.
Dai, H., Khalil, E. B., Zhang, Y ., Dilkina, B., and Song, L. Learning combinatorial optimization
algorithms over graphs. In Proceedings of the 31st International Conference on Neural Information
Processing Systems, NIPS’17, pp. 6351–6361, USA, 2017. Curran Associates Inc. ISBN 978-1-
5108-6096-4. URL http://dl.acm.org/citation.cfm?id=3295222.3295382.
Henning, J. L. Spec cpu2006 benchmark descriptions. SIGARCH Comput. Archit. News , 34
(4):1–17, September 2006. ISSN 0163-5964. doi: 10.1145/1186736.1186737. URL http:
//doi.acm.org/10.1145/1186736.1186737.
Larsen, S. and Amarasinghe, S. Exploiting superword level parallelism with multimedia instruction
sets. In Proceedings of the ACM SIGPLAN 2000 Conference on Programming Language Design
and Implementation, PLDI ’00, pp. 145–156, New York, NY , USA, 2000. ACM. ISBN 1-58113-
199-2. doi: 10.1145/349299.349320. URL http://doi.acm.org/10.1145/349299.349320.
Lattner, C. and Adve, V . LLVM: A Compilation Framework for Lifelong Program Analysis &
Transformation. In Proceedings of the 2004 International Symposium on Code Generation and
Optimization (CGO’04), Palo Alto, California, Mar 2004.
Li, Y ., Tarlow, D., Brockschmidt, M., and Zemel, R. Gated graph sequence neural networks.arXiv
preprint arXiv:1511.05493, 2015.
Li, Z., Chen, Q., and Koltun, V . Combinatorial optimization with graph convolutional networks and
guided tree search. In NeurIPS, 2018.
10
Liu, J., Zhang, Y ., Jang, O., Ding, W., and Kandemir, M. A compiler framework for extract-
ing superword level parallelism. In Proceedings of the 33rd ACM SIGPLAN Conference
on Programming Language Design and Implementation , PLDI ’12, pp. 347–358, New York,
NY , USA, 2012. ACM. ISBN 978-1-4503-1205-9. doi: 10.1145/2254064.2254106. URL
http://doi.acm.org/10.1145/2254064.2254106.
McGovern, A., Moss, E., and Barto, A. G. Building a basic block instruction scheduler with
reinforcement learning and rollouts. Machine learning, 49(2-3):141–160, 2002.
Mendis, C. and Amarasinghe, S. goslp: Globally optimized superword level parallelism framework.
Proc. ACM Program. Lang., 2(OOPSLA):110:1–110:28, October 2018. ISSN 2475-1421. doi:
10.1145/3276480. URL http://doi.acm.org/10.1145/3276480.
Nuzman, D. and Zaks, A. Outer-loop vectorization: Revisited for short simd architectures. In
Proceedings of the 17th International Conference on Parallel Architectures and Compilation
Techniques, PACT ’08, pp. 2–11, New York, NY , USA, 2008. ACM. ISBN 978-1-60558-282-5.
doi: 10.1145/1454115.1454119. URL http://doi.acm.org/10.1145/1454115.1454119.
Porpodas, V . and Jones, T. M. Throttling automatic vectorization: When less is more. InProceedings
of the 2015 International Conference on Parallel Architecture and Compilation (PACT), PACT ’15,
pp. 432–444, Washington, DC, USA, 2015. IEEE Computer Society. ISBN 978-1-4673-9524-3.
doi: 10.1109/PACT.2015.32. URL https://doi.org/10.1109/PACT.2015.32.
Porpodas, V ., Magni, A., and Jones, T. M. Pslp: Padded slp automatic vectorization. InProceedings of
the 13th Annual IEEE/ACM International Symposium on Code Generation and Optimization, CGO
’15, pp. 190–201, Washington, DC, USA, 2015. IEEE Computer Society. ISBN 978-1-4799-8161-8.
URL http://dl.acm.org/citation.cfm?id=2738600.2738625.
Ross, S., Gordon, G., and Bagnell, D. A reduction of imitation learning and structured prediction to
no-regret online learning. In Proceedings of the fourteenth international conference on artiﬁcial
intelligence and statistics, pp. 627–635, 2011.
Shin, J., Chame, J., and Hall, M. W. Compiler-controlled caching in superword register ﬁles for
multimedia extension architectures. In Proceedings of the 2002 International Conference on
Parallel Architectures and Compilation Techniques, PACT ’02, pp. 45–55, Washington, DC, USA,
2002. IEEE Computer Society. ISBN 0-7695-1620-3. URL http://dl.acm.org/citation.
cfm?id=645989.674318.
Shin, J., Chame, J., and Hall, M. Exploiting superword-level locality in multimedia extension
architectures, volume 5. 4 2003.
Shin, J., Hall, M., and Chame, J. Superword-level parallelism in the presence of control ﬂow. In
Proceedings of the International Symposium on Code Generation and Optimization , CGO ’05,
pp. 165–175, Washington, DC, USA, 2005. IEEE Computer Society. ISBN 0-7695-2298-X. doi:
10.1109/CGO.2005.33. URL http://dx.doi.org/10.1109/CGO.2005.33.
Sreraman, N. and Govindarajan, R. A vectorizing compiler for multimedia extensions. Int. J. Parallel
Program., 28(4):363–400, August 2000. ISSN 0885-7458. doi: 10.1023/A:1007559022013. URL
http://dx.doi.org/10.1023/A:1007559022013.
Stephenson, M., Amarasinghe, S., Martin, M., and O’Reilly, U.-M. Meta optimization: improving
compiler heuristics with machine learning. In ACM SIGPLAN Notices, volume 38, pp. 77–90.
ACM, 2003.
Stock, K., Pouchet, L.-N., and Sadayappan, P. Using machine learning to improve automatic
vectorization. ACM Transactions on Architecture and Code Optimization (TACO), 8(4):50, 2012.
Zhou, J., Cui, G., Zhang, Z., Yang, C., Liu, Z., and Sun, M. Graph neural networks: A review of
methods and applications. arXiv preprint arXiv:1812.08434, 2018.
11

A Pattern Based Algorithmic Autotuner for Graph
Processing on GPUs
K
e Meng 1,2, Jiajia Li 3, Guangming T an 1,2, Ninghui Sun 1,2
1State Key Laboratory of Computer Architecture, Institute of Computing T echnology , Chinese Academy of Sciences
2University of Chinese Academy of Sciences
3Paciﬁc Northwest National Laboratory
mengke@ncic.ac.cn Jiajia.Li@pnnl.gov {tgm, snh}@ict.ac.cn
Abstract
This paper proposes G S W I T C H, a pattern-based algorithmic
auto-tuning system that dynamically switches between opti-
mization variants with negligible overhead. Its novelty lies in
a small set of algorithmic patterns that allow for the conﬁg-
urable assembly of variants of the algorithm. The fast tran-
sition of G S W I T C H is based on a machine learning model
trained using 644 real graphs. Moreover, G S W I T C H provides
a simple programming interface that conceals low-level tun-
ing details from the user. W e evaluate G S W I T C H on typical
graph algorithms (BFS, CC, PR, SSSP , and BC) using Nvidia
Kepler and Pascal GPUs. The results show that G S W I T C H
runs up to 10×faster than the best conﬁguration of the state-of-
the-art programmable GPU-based graph processing libraries
on 10 representative graphs. G S W I T C H outperforms Gunrock
on 92.4% cases of 644 graphs which is the largest dataset
evaluation reported to date.
CCS Concepts • Software and its engineering → Soft-
ware libraries and repositories;
Keywords GPU, Graph processing, Auto-tuning
1 Introduction
Graph algorithms support a broad spectrum of applications.
For example, social network analytics, such as PageRank [ 11]
and HITS [ 54], rely heavily on graph models to represent rela-
tionships. Analysis tasks such as bug ﬁnding and network rout-
ing have used graph-based algorithms to improve their perfor-
mance and accuracy [ 15, 57]. However, graph processing, es-
pecially for large-scale graphs, is computationally expensive.
As GPUs provide higher parallelism and memory bandwidth
than traditional CPUs, they are a promising hardware for
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not
made or distributed for proﬁt or commercial advantage and that copies bear
this notice and the full citation on the ﬁrst page. Copyrights for components
of this work owned by others than ACM must be honored. Abstracting with
credit is permitted. T o copy otherwise, or republish, to post on servers or to
redistribute to lists, requires prior speciﬁc permission and /or a fee. Request
permissions from permissions@acm.org.
PP oPP ’19, F ebruary 16–20, 2019, W ashington, DC, USA
© 2019 Association for Computing Machinery .
ACM ISBN 978-1-4503-6225-2/19/02. . . $15.00
https://doi.org/10.1145/3293883.3295716
accelerating graph applications. Some programmable graph
processing frameworks have been developed on GPUs, such
as CuSha [ 27], Frog [ 50], WS-VR [ 26], and Gunrock [ 58].
They integrate graph primitives and tend to provide a general-
purpose library .
However, from the perspective of optimization, these frame-
works often fail owing to the inherent properties of input sen-
sitivity and algorithmic diversity, which are two fundamental
problems in the performance tuning of many applications. (i)
In the scenario featuring input sensitivity , dynamic graphs and
the inﬂuence of diverse graph topologies are rarely considered
in current graph algorithm libraries. A single optimization
variant is typically used during the entire course of execution,
which leads to signiﬁcant bias in the best performance at-
tained. For an instance of breadth-ﬁrst search (BFS), the Gun-
rock library [ 58], the best hand-tuned implementation to date,
achieves up to 47GTEPS on a scale-free graph and retards to
44MTEPs on a road-net graph. (ii) In the scenario featuring
algorithmic diversity , an algorithm-speciﬁc optimization does
not always work well for all cases. For example, WS-VR [
26]
and CuSha [ 27] perform well on PageRank-like algorithms
that require dense workloads but show poor performance on
traversal-based algorithms like SSSP with sparse workloads.
On the contrary , direction optimization [ 7], which is applica-
ble to a greater variety of graph algorithms, is used and tuned
for only BFS in Gunrock. This isolated performance tuning
leads to difﬁculties in exploring all potential features and fails
to reuse tuning strategies among different graph algorithms
and datasets.
Several studies have recently developed new auto-tuning
techniques [ 5, 14, 32, 38] targeting the input sensitivity for
performance critical applications. In contrast to traditional
auto-tuning systems [ 16, 60] that generate libraries in an off-
line way , an input sensitivity auto-tuning system requires an
on-line manner of determining the best conﬁguration or as-
sembly of algorithms, referred to as algorithmic auto-tuning.
The key is to specify an algorithm-choosing strategy that
extracts cost-effective input features and maps them to a per-
formance space, as in Ding et al. ’s work [ 14]. Our contribu-
tion here is a set of algorithmic patterns that helps build an
efﬁcient algorithmic auto-tuner. This work focuses on GPU
architectures because of their powerful capability in terms of
both parallel computation and memory bandwidth.
201

PPoPP ’19, February 16–20, 2019, Washington, DC, USA Ke Meng, Jiajia Li, Guangming T an, Ninghui Sun
W e present G S W I T C H 1 . It abstracts from a graph algorithm
as a small set of algorithmic patterns that are characterized
by quantitative parameters (features). Given different graph
inputs, a graph algorithm is dynamically composed of con-
crete implementations of these algorithmic patterns during
runtime. G S W I T C H leverages a machine learning model to
select good algorithmic conﬁgurations. Speciﬁcally , our main
contributions are as follows:
•W e deﬁne two properties, generality and signiﬁcance,
to identify good algorithmic patterns. Through a com-
prehensive study of graph algorithms using more than
1,200 real graphs, we extract ﬁve composable patterns
satisfying these properties.
•W e propose a machine learning based algorithmic auto-
tuner called G S W I T C H to dynamically generate high-
performance graph algorithms. G S W I T C H dynamically
chooses the best choice of an algorithm with minimal
overhead and no user intervention.
•GS W I T C H is highly productive in terms of user pro-
gramming.
It consists of a front-end exposed to users
and a transparent back-end. The front-end provides
a succinct yet ﬂexible abstraction for programming a
graph processing algorithm. The back-end integrates in-
dividual highly optimized kernels into a parameterized
kernel library .
•GS W I T C H was implemented and evaluated on GPUs.
Experiments on typical graph applications show that it
outperforms the state-of-the-art libraries/frameworks
by a factor of
2 ∼10. It achieves better performance
than the highly hand-tuned counterpart Gunrock li-
brary [58] for 92.4% instances of 644 graphs.
T o the best of our knowledge, this work is the ﬁrst study to
systematically investigate tuning variants of graph algorithms
and develop an auto-tuning system for them on GPUs.
W e have organized the remainder of this paper as follows:
Section 2 provides related deﬁnitions and our motivations.
Section 3 explores ﬁve algorithmic patterns, and Section 4
outlines the framework of G S W I T C H and shows details of its
implementation. In Section 5, we evaluate our system and
analyze the effectiveness of our auto-tuning method. Section
6 surveys graph processing work on GPUs, and Section 7
concludes this work.
2 Background and Motivation
In this section, we introduce basic deﬁnitions for the graph
processing framework. W e also provide our motivations to
give an intuition of why a pattern-based auto-tuner is both
easy to implement algorithms and provides higher perfor-
mance than state-of-the-art hand-tuned systems.
1 GS W I T C H is currently available in an open-source repository at
https://github.com/P AA-NCIC/gswitch.
Filter Expand (75%)
Runtime breakdo
wn of BFS
Filter (86.1%)
Expand
Runtime breakdown of BFS
Frontier Size
M
M
Distanc
e from source 
100× K
K
Distanc
e from source 
Frontier Size
(a) (b)
(c)
Performance Loss (%)
Datasize (nnz)
4.8K 530M
Figure 1. A motivating example of the BFS: The frontier
expansion on (a) a scale-free graph com-youtube and (b)
a road-net graph roadNet-CA; (c) Performance loss if we
only use the Push [9] optimization variant on 1288 graphs.
2.1 Graph Algorithms
This work considers BSP (Bulk Synchronous Parallel) graph
algorithms using the widely adopted neighbor-based prop-
erty [ 62], where all vertices communicate only with their
neighbors. A graph algorithm in the BSP model is treated
as a sequence of super-steps that iteratively converge, with
each step requiring global synchronization. Only a subset of
the vertices or edges, known as the active set, participates in
the computation in each iteration. An algorithm terminates
with no more active elements. G S W I T C H standardizes a graph
algorithm as a quadratic-work process: The ﬁltering step (re-
ferred as Filter in G S W I T C H) generates the active set and
updates
their private data; then the expansion step (referred
as Expand in G S W I T C H) processes all the neighbors of the
generated active set. Our Filter-Expand abstraction is inspired
by the Gather-Apply-Scatter approach in PowerGraph [ 19]
and the Advance-Filter-Update mechanism in Gunrock [ 58].
The main difference is that we simplify the abstraction by
integrating the “ Apply/Update” step into our Filter. In this
paper, we use ﬁve typical graph applications to benchmark
GS W I T C H: Breadth-First Search (BFS) [ 7], Connected Com-
ponents (CC) [ 53], PageRank (PR) [ 11], Single Source Short-
est Path (SSSP) [ 42], and Betweenness Centrality (BC) [ 47]
algorithms.
2.2 Motivation
Such challenges of graph processing on GPUs as irregular
memory access patterns and unbalanced workloads have led
to numerous optimization studies [ 7, 9, 44, 52, 58, 61, 66].
The idea of the algorithmic pattern is partially inspired by
these efforts.
For example, directional optimization [ 7] has been widely
used in some frameworks/libraries to handle irregular work-
loads. Figure 1 shows a breakdown of the runtime of the BFS,
and the behavior at the edge and vertex frontiers of two types
202
Gswitch PP oPP ’19, February 16–20, 2019, Washington, DC, USA
Push Pull
1 83
2
6
5
4 7
1 83
2
6
5
4 7
ActiveVisited Unvisited Work-item
edge skipping
reduce
at
omic
Figure 2. Direction: Push vs. Pull for BFS.
of graphs: a scale-free graph com-youtube and a road-net
graph roadNet-CA. The graph com-youtube (Figure 1
(a)) has a small diameter (13 from the x-axis) and workload
explosion, its edges to be processed in middle iterations can
be 100×more than the vertices in number. The Expand is its
performance bottleneck (75% Expand vs. 25% Filter). By
contrast, the graph roadNet-CA (Figure 1 (b)) has a larger
diameter yet edges to process in each iteration. Thus, it spends
more time on the Filter than the Expand (86.1% vs. 13.9%). If
a single optimization variant (e.g., push [ 9]) is used on the two
graphs, the loss of performance of the BFS can be as high as
80% (Figure 1 (c)). Some past work [ 7, 9, 21, 44, 52, 61, 66]
has reported this performance variance and proposed speciﬁc
tuning algorithms. W e further abstract these characteristics as
algorithmic patterns which heavily affect performance.
As we know , many inﬂuential performance-tuning strate-
gies are available for implementing high-performance graph
algorithms. Examples are strategies used to handle the load-
imbalance within and between each warp/block, atomic-free
operations, kernel fusion, coalesced memory access, among
others [
7, 9, 21, 22, 34, 35, 41, 44, 52, 61, 63, 66]. W e advo-
cate that a group of low-level optimizations should be con-
sidered as a single algorithmic pattern that can reduce the
tuning space signiﬁcantly . This motivates us to explore more
algorithmic patterns and use them to build our algorithmic
auto-tuner.
3 Exploring Algorithmic Patterns
A pattern here means a group of methods, strategies, or tech-
niques to optimize the similar algorithmic paradigms in differ-
ent graph applications, such as load-balance strategies, data
structure. W e call the optimization variants of a pattern as
candidates. Speciﬁcally , we claim that an algorithmic pattern
should have two properties:
•Generality: The pattern is effective for more than one
application.
•Signiﬁcance: The behavior of candidates is signiﬁcantly
different, depending on inclusion of the pattern.
The pattern set of G S W I T C H includes dir ection, the active
set format, load balancing, stepping, and kernel fusion . W e
describe each pattern from the perspective of their two prop-
erties below .
Runtime (ms)
Iteration No.  of BC 
Iteration No.  of Delta-PR
It
eration No.  of BFS 
Iteration No.  of BF-SSSP 
Runtime (ms)
PullPush
Figure 3. Runtime per-iteration in push and pull modes for
graph hollywood-09.
Pattern 1: Direction (Candidates: Push and Pull).
Generality: Direction is the push-pull dichotomy general-
ized form [ 7]. Figure 2 plots an example of the BFS. The push
mode touches the neighboring edges of active vertices and
updates a vertex property via an atomic operation. The pull
mode touches the neighboring edges of inactive vertices and
updates a vertex property via a reduction operation. The push-
pull dichotomy can be applied to all the ﬁve graph algorithms
as demonstrated in previous work [ 9, 52, 66]. In contrast to
Gunrock, which only enables this optimization for BFS, we
generalize this dichotomy as a common optimization for all
the graph algorithms.
Signiﬁcance : Figure 3 shows that the difference in perfor-
mance between the push and pull modes is signiﬁcant. For
BFS and BC, the pull mode skips a large number of edges
in the middle iterations. For Delta-PageRank [ 19], the push
mode outperforms the pull mode when the active vertices
decrease in length. For BF 2 , the pull mode is faster than the
push mode in some iterations with heavy workloads. Based
on our observations of thousands of cases, we conclude that
the pull mode is preferable in the middle iterations when the
number of the active edges is greater than that of inactive
edges. Therefore, using the numbers (and ratios) of the active
and inactive elements in an iteration as tuning parameters is
beneﬁcial for auto-tuning.
Pattern 2: Active-set data-structure(Candidates: Bitmap,
Unsorted queue, and Sorted queue).
Generality: As introduced in Section 2, the active set is
a common data representation for most graph algorithms.
GS W I T C H chooses the data structure between the bitmap and
the queue to maintain the active set [ 44]. Figure 4 shows these
data structures for an example active vertex set ({2,3} for the
second-level iteration) and their generations. The bitmap uses
one bit for each vertex, where the value one represents an
active vertex and zero represents an inactive vertex. Active
vertices can also be described by a compact sorted/unsorted
queue, which is generated by performing a scan-based method
2 In this paper, SSSP refers to our implementation of dynamic stepping , BF
is short for the unordered Bellman-Ford algorithm, and ∆ -stepping represents
the traditional implementation of stepping with the static priority threshold.
203
PPoPP ’19, February 16–20, 2019, Washington, DC, USA Ke Meng, Jiajia Li, Guangming T an, Ninghui Sun
Unsorted Queue Sorted Queue
1 2 3 4 5 6 1 2 3 4 5 6
23 2 3
scan
coalesced non-coalesced
out-of-order sequential
Bitmap
10 01 00
cause threads idle 
low overhead 23 2 4 11 5 6 6 3 5 3
21 X2 XXLevel[]
Filt
er(i): Level[i]==2
scan
Active Set
less spac
e
Level[]
Figure 4. The second iteration of BFS with an active vertex
set {2,3}. T o select the active set in this iteration, the bitmap
marks the two vertices as 1s and the rest as 0s, whereas the
queue copies them to a compact in- or out-of-order array .
Runtime (ms)
Iteration No.  of PR Iteration No.  of SSSP 
Bitmap Sorted queue Unsorted queue
(a) (
b)
Figure 5. Runtime for each number of iterations in bitmap,
unsorted queue, and sorted queue formats for (a) Pager-
ank in graph kron_g500-log21 and (b) SSSP for graph
msdoor.
that computes the offset of each thread. Comparing these two
formats, the bitmap has no scan overhead but threads may be
idle when assigned an inactive element, especially for small
active sets. On the contrary , the queue does not cause idle
threads but its generation step may be costly , especially for
large active sets. The queue format can be either unsorted or
sorted as shown in Figure 4. The generation of the unsorted
queue is fast because it ensures coalesced memory access. The
generation of the sorted queue is unfriendly to memory access,
but the following Expand step beneﬁts from the potentially
contiguous memory access of the queue [33].
Signiﬁcance : The performance of the tested candidates is
shown in Figure 5. For an algorithm with a dense workload,
like PageRank, all the vertices and edges are active in each
iteration. Thus, the bitmap can avoid overhead due to the
enqueue. For an algorithm with a sparse workload, like SSSP ,
compressing the sporadic active vertices into the queue is use-
ful. In conclusion, bitmap is more friendly to dense workloads,
the unsorted queue is more friendly to sparse workloads, and
the sorted queue is in the middle. Tuning parameters such as
the active set size and the number of corresponding edges are
critical to this pattern.
Pattern 3: Load balance (Candidates: TWC, WM, CM,
and STRICT ).
Generality: Due to the mismatch between the irregular
computations in graph algorithms and the Single Instruction
Multiple Threads (SIMT) architecture of GPUs, this pattern
is important for GPU optimizations. Figure 6 illustrates four
load-balancing strategies:
•The Thread, W arp, and CT A (TWC ) method was ﬁrst
introduced by the B40C [ 41]. For Nvidia GPUs, a warp
is a set of threads working in a lockstep fashion, while
CT A (Cooperative Thread Array) is a set of warps. The
active vertex set is divided into low-, medium-, and
high-degreed vertices according to their out-degrees,
which are mapped to a thread, a thread warp, and a CT A
separately .
•The W arp Mapping (WM ) method redresses the imbal-
ance in a warp. A warp processes continuous vertices
from the active set as a batch. It loads the warpsize of
their neighbors into GPU shared memory each time,
until all their neighbors have been processed. A binary
search on the loaded edges is performed to ﬁnd the
sources of the neighbors. Finally , the results are written
back to global memory .
•The CT A Mapping (CM ) method is similar to WM, and
deals with the balance in a CT A. A CT A loads contin-
uous blocksize vertices as a batch and their blocksize
neighbors into the shared memory until all neighbors
have been processed. A larger binary search is used
to compute the source of the loaded neighbors, and
synchronization is needed for threads in a CT A.
•The STRICT method handles the balance both in CT As
and across them. It is modiﬁed from the load-balance
partitioning (LB) in [ 13]. This method forces each CT A
to maintain an equal number of vertices and edges, by
using a sorted search to ﬁnd the optimal partitioning of
the workload. Then, the edge and vertex lists are then
both partitioned.
Signiﬁcance : STRICT obtains the best load balance but its
overhead is the highest, WM/CM is in between, and TWC
has the lowest overhead but it is not balanced well. The main
overhead of WM is the binary search, which can be completed
in no more than log2 (warpsize ) steps, while that of CM is
log2 (blocksize ) steps. As shown in Figure 7, TWC runs faster
owing to its lower overhead, and CM and WM perform better
with irregular workloads. When the workload contains a hub
vertex which is connected to a large number of neighboring
vertices, STRICT is the best method.
Pattern 4: Stepping (Candidates: Increase, Decrease, and
Remain).
Generality: This pattern is also related to the active set as it
speciﬁes its execution behavior for different graph processing
algorithms. A monotonic algorithm, such as SSSP , can be
implemented in either ordered or unordered mode [
21]. An
ordered implementation updates only high-priority elements
of the active set (e.g., the ∆ -stepping algorithm [ 42]) to avoid
204
Gswitch PP oPP ’19, February 16–20, 2019, Washington, DC, USA
WM/CM STRICT
warp size/CTA size=8
A
ctive set (queue or bitmap)
1 2
6 1 3 4 52
 2 1
2
 2 1
6 2 4
__global__
__global__
Adj_List
__shared__ local_v_value
load value

seg1 seg2
local_v_value
seg_reduce
3
 4
 5
 6
2
1__global__
write back
vdata
filter
Ne
xt Active set__global__
__shared__
grid siz
e=2
Active set (queue or bitmap)
1 2__global__
0
 4
 1012
15
17
20__global__ Row_offset + vid
sorted_search
CTA

CTA

__global__
 6 1 3 4 52
 2 1
2
 2 1
6 2 41
0
4 8
__global__
 3
 4
 5
 6
2
1 vdata
filter
__global__ Ne
xt Active set
atomic
Adj_List
1 2
TWC
BalancedL ow Overhead
warp size=4
Active set
__global__
Thread Warp CTA

(2, 4) (4, 32)
6 1 3 4 52
 2 1
2
 2 1
6 2 4
smallq
1
middle
q
2
large
q
Adj_List
__global__
barrier
3
 4
 5
 6
2
1__global__ vdata
filter
__global__
<2
Ne
xt Active setsmallq middleq largeq
Figure 6. Load-balancing strategies: TWC, WM, CM, and STRICT .
Iteration No.  of PR Iteration No.  of BFS Iteration No.  of BFS
Runtime (ms)
131 170
Push Mode
235240 232151
Pull Mode
(a) ( b) (c)
STRICTCMTWC
 WM
Figure 7. Runtime per iteration on TWC, WM, CM, and
STRICT methods for Load-Balancing. (a) The PageRank for
soc-orkut. (b) The push mode of BFS for soc-orkut.
(c) The pull mode of BFS for soc-orkut.
Iteration No.  of SSSP
T ouched Edges  
Runtime (ms)
Bellman-Ford Dynamic stepping∆-stepping
Iteration No.  of SSSP
Figure 8. A comparsion among the unordered Bellman-Ford,
the ordered ∆ -stepping , and dynamic stepping for SSSP on
graph soc-orkut.
Runtime (ms)
Iteration No.  of BFS Iteration No.  of BFS
36 55
Runtime (ms)
(a) ( b)
Figure 9. Runtime per iteration runtime of variants with fused
or standalone candidate for graphs (a) roadNet-CA and (b)
soc-orkut.
unnecessary computation. An unordered implementation en-
gages as many active elements as possible to pursue high
parallelism (e.g., the Bellman-Ford algorithm). With online al-
gorithmic auto-tuning in mind, we propose a dynamic priority
threshold and amend it by comparing the (estimated) number
of edges between the previous iterations and the ones follow-
ing. If the number of edges increases or decreases by some
margin, we increase or decrease the priority threshold accord-
ingly by a pre-set step size; otherwise, the priority threshold
stays constant. From our experiments, the increase/decrease
ratio was set to 35%.
Signiﬁcance : Figure 8 shows the difference among the un-
ordered version, the static ordered version ( ∆ -stepping), and
the dynamic ordered version. W e set the priority threshold for
the static unordered version to the cw/d, used in [ 13]. The
dynamic ordered version was more ﬂexible when facing work-
load explosion. T o determine whether to increase or decrease
the stepping threshold, G S W I T C H collects the characteristics
of the current workload (the number of edges to be explored
and the edge distribution) as tuning parameters.
Pattern 5: Kernel fusion (Candidates: Standalone and
Fused).
Generality: Kernel fusion fuses multiple operations into a
single kernel to improve the operational intensity . T o avoid
processing duplicated vertices, active set generation and pro-
cessing are usually implemented as two separate kernels such
as in Gunrock and Enterprise. A duplicate removal process
is required for them. However, for some sparse graphs with
stable workloads, if the attributes of the dataset and historical
information imply a shortcut, we use the kernel fusion to
speed up the execution. Duplication-tolerant techniques such
as bitmap marking is applied to guarantee the correctness in a
kernel-fusion variant. If the runtime of the last iteration is far
205
PPoPP ’19, February 16–20, 2019, Washington, DC, USA Ke Meng, Jiajia Li, Guangming T an, Ninghui Sun
Online
Ofﬂine
 Record runtime characteristic Statistics 
Traning
Kernel Libs 
L
earning Model 
Expand 
Active-set 
Filter
Use filter function to
generat
e active set
filter
update vertex data status 
A
ctive Inactive Fixed
Inspector 
Feature Extraction Features 
Che
ck whether the algorithm
has converged or not is converged?
use emit function to send 
vmsg to another vertex
  
emit
comp/compAtomic
comp/compAtomic function 
describes ho
w the vmsg are 
proc
essed
Runtime Characteristics
 number of active vertices
 number of inactive vertices
 number of active edges
 number of inactive edges
   
...
Selector 
Estimate the performance
 and make decisions Conﬁgurations 
Executor
Kernel Libs 
Hist
orical Information
 last filter time
 last expand time   
...
Feature Database
Feedback
1
2
3
[Y] Bypass the decision making [N] T o reconﬁgure 
Is stable?
Host
Host
Devic e
Dataset Attributes
 number of vertices
 number of edges
 the average degree
 Gini coefficient
   
...
Figure 10. The overview of G S W I T C H. On the host side, the Inspector extracts the feature vectors that are fed to the learning
model in the Selector. On the device side, the Executor processes the selected kernels and generates the feedback for the next
iteration.
T able 1.Feature vector in G S W I T C H.
Parameter Meaning Correlation
Dataset attributes
N , M the number of vertices and edges P3,5
d the average of degrees P3,5
σ d the standard deviation of degrees P3,5
rd the relative range of degrees P3,5
G
I Gini coefﬁcient [29] P3,5
He
r Relative edge distribution entropy [29] P3,5
Runtime Characteristics
Va , Ea the number of active vertices and edges P1,2,3,4
Vi
a , Eia the number of inactive vertices and edegs P1,2,3,4
Va
p , Eap the ratio of active vertices and edges P1,2,3,4
Vi
ap , Eiap the ratio of inactive vertices and edges P1,2,3,4
c
d the average of degrees in current workload P1,2,3,4
rc
d the relative range of degrees in current workload P1,2,3,4
Historical Information
tf the time of the last Filter step P5
te the time of the last Expand step P5
Tf the average time of previous Filter steps P5
Te the average time of previous Expand steps P5
longer than the average runtime in the fused mode, G S W I T C H
switches back to the standalone mode.
Signiﬁcance : As illustrated in Figure 9 (a), the fused ver-
sion runs 12×faster than the standalone one, which improves
performance signiﬁcantly on road-net graphs. However, for
dense inputs like those from social graphs, where the work-
load is irregular and unpredictable, too many duplicates are
generated from fused version to achieve reasonable perfor-
mance (Figure 9 (b)). Features of the input graph and the
runtime of previous iterations are relevant tuning parameters
for this pattern.
The parameters of these patterns are summarized in T able 1.
W e refer to Pattern i as P i for short. By determining a proper
conﬁguration of these parameterized patterns, G S W I T C H ob-
tains an optimized execution of a given graph processing
algorithm. The machine learning technique used to train these
candidate values will be discussed in the next section.
4 Implementation of G S W I T C H
4.1 G S W I T C HOverview
As the algorithmic optimizations are abstracted as assembled
parameterized patterns, it is natural to adopt a machine learn-
ing based method, as they have been successfully used in
recent auto-tuners [ 12, 32, 48, 64]. As shown in Figure 10, in
every iteration (or super-step) G S W I T C H selects optimized
variants on-the-ﬂy through the Inspector, Selector, and Ex-
ecutor stages. T wo knowledge databases (feature database
and parameterized kernel library) are constructed to train the
model ofﬂine.
•Inspector. The inspector (labeled as 1 ) ﬁrst checks
for the convergence of the previous iteration. If the
previous iteration has not converged, it starts a new
iteration, collects the values of the pre-deﬁned features:
dataset attributes, runtime characteristics, and historical
information, and saves them as a vector. These feature
values are the inputs to the selector.
•Selector. The selector (labeled as 2 ) is a black-box
learning model which is trained ofﬂine from the feature
database. It takes the feature vector from the Inspector
to predict the optimal kernel conﬁguration for the next
iteration. The conﬁguration is saved as an entry in the
kernel library .
•Executor. The executor (labeled as 3 ) chooses the de-
termined kernels from the kernel library and executes
them appropriately on the GPU. Meanwhile, some run-
time characteristics are proﬁled for use as feedback for
the next inspector.
As shown in Figure 10, the inspector and selector stages are
executed on the host (CPUs), while the executor is processed
206
Gswitch PP oPP ’19, February 16–20, 2019, Washington, DC, USA
on the device (GPUs). The kernel library stores the imple-
mentation variants of the Filter and Expand primitives. At
the end of each iteration, a feedback stage copies the runtime
characteristics and historical information of the last iteration
from the device to the host.
4.2 Programming APIs
T o implement the customized graph application, users need to
provide four functions for G S W I T C H to implement their graph
applications: filter, emit, comp and compAtomic. Figure 11
gives an example of these functions that together implement
BFS. The filter function provides a predicate to reﬁne the
active set for the current iteration. The emit function deﬁnes
the message that the source vertex should send. The comp
and compAtomic functions both describe how the message
is processed in the receiving vertex, with the former being
atomic-free while the latter is not. No tuning parameters are
required and all the tuning details are opaque to users.
__device__ void compAtomic(int* u, int lvl, G g){
  atomicExch(u, lvl); 
}
__device__ void comp(int* u, int lvl, G g){
  *u = lvl;
}
__device__ int emit(int vid, Empty* weight, G g){
  return data_of(vid)+1;
}
__device__ Status filter(int vid, G g){
  int lvl = data_of(vid);
  if(lvl == g.level()){return Active;}
  else if(lvl < 0) return Inactive;
  else return Fixed;
}
Figure 11. An example of BFS algorithm in G S W I T C H.
4.3 Feature Extraction
The feature vector for training and inference on the model
is listed in T able 1. These features are easily collected even
during runtime because most of the required runtime char-
acteristics are side products of the Filter and Expand steps.
These features fall in three dimensions:
•The dataset attributes provide the basic description of
the graph. First, two parameters are used to represent
the basic structure of the graph: N (the number of ver-
tices) and M (the number of edges). Second, d, σd ,
and rd are added to describe the distribution of the
edges. T o classify whether a graph is regular or irreg-
ular, we introduce the Gini coefﬁcient and the relative
edge distribution entropy to measure the equality of
degree distributions [ 29]. The attributes of the dataset
are computed only once while loading the data into the
memory .
•The runtime characteristics describe the workload in
each iteration. The numbers and ratios of active and
inactive elements in the graph help determine the direc-
tion (P 1). Once this is done, G S W I T C H chooses the ac-
tive or inactive elements as the current workload. cd and
rcd describe the degree distribution of the given work-
load. The active set format (P 2) and load-balancing
strategies (P 3) are highly relevant to the features of the
given workload.
•Historical information reveals the workload imbalance
of previous iterations. By comparing the time taken for
the last iteration with the average time for all previous
iterations, we can estimate the workload of the last
iteration without copying the runtime characteristics
from the device.
The features are selected based on a statistical analysis.
Figure 12 shows six most prominent features for the ﬁve pat-
terns on 644 graphs [ 1], where the distribution of each pattern
reﬂects the inﬂuence of the features. Figure 12 (a) shows that
the pull mode improvements when the number of inactive
edges is small; By contrast, the push mode is preferred when
the number of inactive edges increases. Figure 12 (b) shows
that the queue mode performs well when the number of active
vertices is small. Figure 12 (c) and (d) show that the STRICT
mode is beneﬁcial when the data are irregular and the number
of active vertices is large. Figure 12 (e) indicates that the
stepping threshold should be increased when the number of
active edges is small. Figure 12 (f) shows the Gini coefﬁcient
that describes the inequality of the degree distribution, from
which we can conclude that fused kernels are preferred when
the vertices in a graph have similar numbers of neighbors.
4.4 Model Generation
W e treat patterns as independent decision goals, and the sta-
tistics of each iteration constitute a record in the feature data-
base. For example, one record of the graph web-it-2004
is R ={509338, 7178410, 28.2, 58.7, 16.6, 0.55, 0.94, 289,
508987, 2694, 14353300, 0.0006, 0.9994, 0.0002, 0.9998, 9.32,
137.7, 1.2, 1.3, 0.4, 0.15}. W e ran all the implementations of
the kernel library on 644 graphs [ 1] for all the benchmarks
and gathered a total of 386,780 records (one record for each
iteration). The true optimal conﬁgurations were attained via
brute-force experimentation.
Decision making can be regarded as a classiﬁcation prob-
lem. G S W I T C H builds a classiﬁer for each pattern. The clas-
siﬁcation process can be formulated as f (R) = OPT . For
example, the output OPT here in the directional classiﬁer can
be either push or pull. The CAR T (Classiﬁcation and Regres-
sion Tree) algorithm is used to generate the decision tree. W e
choose it as our learning model for two main reasons. First,
this model is easy to convert the resulting rules to if-else sen-
tences, which is convenient for transplanting them to diverse
platforms and integrating them with other runtime systems.
Second, the model is easy to understand and interpret while
its prediction overhead is low . However, the decision-tree
207
PPoPP ’19, February 16–20, 2019, Washington, DC, USA Ke Meng, Jiajia Li, Guangming T an, Ninghui Sun
Percentage(%)
(a) Eiap
 (b) Vap
 (d) Eap
 (f) GI
(e) E
(1e5)
a
(c) Her
Figure 12. The distribution of the optimal strategy with different parameter values, where the y-axis shows the percentage of
each parameter falling into its value intervals.
model is prone to overﬁtting. T o improve the generality of the
model, we tailor the generated decision tree and keep the its
height as low as possible.
expand<push, LB>(. ..){
...
for u in neighbor(v)
  c
ompAtomic(u, wa_of(v), g);
...
}
e
xpand<push, CM>(...){
...
load blocksiz
e edges of v 
...
}
e
xpand<push, CM>(...){
...
for v in AS<Queue>
 
vmsg = emit(v, NULL, g);
...
}
e
xpand<push, CM>(...){
...
if(Fused==True)
  writ
e u to AS directly
...
}
Push
Pull
CM
WM
TWC
STRICT
Bitmap
Unsorted Queue
Sorted Queue
Remain
Increase
Decrease
Fused
Standalone
Pattern 1 Pattern 2 Pattern 3 Pattern 4 Pattern 5
Direction Load Balance A.S. Format Stepping Kernel Fusion
ﬁlter<push>(...){
for v in active V:
  if priority(v) > th
    write v to AS
...
}
Non-selected strategy Selected strategy
Figure 13. The kernel searching process of the selector along
with some interpretable rules in the decision path of one
iteration.
0.22
0.59
1.61
38.7
58.5
61.8
0.97
0.20
0.46
0.82
4.74
15.5
27.2
0.68
0.21
0.40
0.76
2.03
17.7
30.4
0.68
0.21
0.40
0.54
1.50
17.9
29.7
0.82
97.0
98.6
96.5
58.8
1.80
0.38
0.39
49.5
50.4
50.4
49.2
33.5
0.86
0.40
44.4
46.3
44.9
44.8
31.3
2.01
0.39
124
123
126
122
71.3
2.74
0.40
1
2
3
4
5
6
7
TWC WM CM STRICT TWC WM CM STRICT
Push Pull
Gswitch Best
0.20
0.40
0.54
1.50
1.80
0.38
0.39
0.20
0.40
0.54
1.50
1.80
0.38
0.39
Gswitch Best
∑ 162 49.7 52.2 51.1 354 234 214 569 5.21 5.21
lvl
Figure 14. The runtime of different strategies of BFS for
graph orkut. "Gswitch" indicates the predicted strategies of
GS W I T C H, "Best" indicates the true optimal strategies.
4.5 Kernel Searching
In the low-level implementation, the parameterized kernels
are described as C++ templates. Some patterns (e.g., P 1: di-
rection) are implemented as standalone kernels, while others
(e.g., P 2: active set formats) are implemented as a combina-
tion of branches and conditions. In total, we had 12 standalone
kernels supporting 156 implementation variants. In each itera-
tion, we needed to choose a ﬁlter primitive from 12 candidates
and one expand primitive from 144 candidates.
W e made decisions following a given order and each pat-
tern had one classiﬁer. Figure 13 illustrates the kernel search-
ing process for one iteration of BFS. W e ﬁrst determine the
direction, because the workloads in push and pull are com-
pletely different. Load-balancing strategies are then consid-
ered, followed by the active set format. W e ﬁnally decide
the stepping conﬁguration as well as whether to enable the
kernel fusion. Figure 14 shows the searching path of the
graph orkut for the BFS algorithm along the dimensions of
the direction and the load-balancing. In this case, G S W I T C H
chooses the optimal strategy in each iteration.
T able 2.Representative graphs for benchmarking.
Graphs V ertices Edges Max Degree Domain
soc-orkut 3M 212.7M 27,466 SN
soc-pok
ec 1.6M 61M 20,518 SN
web-uk-2005 129K 23M 850 WG
web-wikipedia-2009
1.8M 9M 2,623 WG
kron_g500-log21 2.1M 182.1M 213,904 GG
r
gg_n_2_24 16.8M 265.1M 40 GG
roadNet-CA 1.9M 5.5M 121 RN
roadNet-TX
1.4M 3.8M 12 RN
sc-msdoor 415K 19.8M 76 SC
sc-ldoor
952K 42M 76 SC
Graphs domains are: SN: Social Network, WG: W eb Graph, GG: Generated
Graph, RN: Road Network, SC: Scientiﬁc Computing.
5 Evaluation
5.1 Experimental Setup
Platform
. W e ran all experiments on two systems, both on
Linux servers with 2.10GHz E5–2620 v2 Intel Xeon CPUs
and 48 GB of main memory . One had an Nvidia K40m GPU
with 12 GB of global memory , and the other had an Nvidia
P100 GPU with 16 GB of global memory . W e compiled all
the GPU programs using NVIDIA ’s nvcc compiler (version
7.5.17) and the -O3 ﬂag.
Dataset. W e randomly chose 1,288 graphs from the net-
work repository [ 1] 3 . Half of them (644) were used as the
3 All graphs were transformed into undirected ones.
208
Gswitch PP oPP ’19, February 16–20, 2019, Washington, DC, USA
T able 3.The evaluation time of G S W I T C H vs. other GPU graph processing libraries and hardcoded implementations.
Runtime (ms) [lower is better]
Social Network W eb Graph Generated Graph Road Network Scientiﬁc Computing
Alg. Lib. orkut pokec web-uk-05 web-wp-09 kron-21 rgg_n24 roadnNet-CA roadNet-TX msdoor ldoor
BFS
Enterprise 30.7 11.2 3.1 87 10 - - 47 44 68
Gunrock 6.1 12 1.6 25 4.2 606 104 82 42 64
Gswitch 5.5 3.8 1.2 18 2.9 344 39 47 23 33
CC
GPUCC 84 21 5.7 14 128 - 9.1 6.7 6.3 13
Gunrock 161 38 50 36 236 305 17 13 17 40
Gswitch 111 35 3.9 11 89 143 6.1 4.7 3.9 7.5
PR
WS-VR 16567 (79) 2718 (51) 116 (27) 408 (49) 6600 (35) 7769 (67) 61 (52) 43 (51) 253 (60) 421 (58)
Gunrock 4069 (22) 733 (21) 70 (22) 1420 (24) 3009 (19) 1533 (19) 190 (21) 134 (21) 54 (18) 107 (18)
Gswitch 2201 (22) 433 (21) 59 (22) 117 (24) 1759 (19) 1011 (19) 32 (20) 20 (20) 41 (18) 84 (18)
SSSP
Frog 2424 534 35 436 847 - 44 28 199 363
Gunrock 1096 385 29 112 248 99663 179 196 175 582
Gswitch 407 144 16 83 247 2772 84 90 46 83
BC
GPUBC 286 46 7.6 100 57 1298 194 182 87 250
Gunrock 427 92 11 88 371 1669 116 227 62 98
Gswitch 85 28 2.9 55 91 916 78 101 56 79
The numbers in brackets are the numbers of iterations. As Gunrock updates its code continuously , some performance results we reproduced for it are inconsistent
with
the results of the original paper [58] (e.g., on rgg_n24) but consistent with the results of the authors’ more recent paper [59].
training set for model generation, while the rest were used as
the evaluation set. The evaluation set has no overlap with the
training set. T en-fold cross-validation was used to evaluate
the accuracy of the model. As shown in T able 2, we chose 10
representative graphs from the evaluation set to analyze.
Baseline. G S W I T C H was compared with several state-of-
the-art
programmable GPU-based graph processing frame-
works/libraries. According to the latest results in [ 2, 59],
Gunrock [ 58] delivered relatively better performance than
the others owing to its continuous evolution. W e compared
our method with Gunrock on all test cases. For each graph
algorithm, we selected a specialized, optimized implementa-
tion from other frameworks/libraries that beats Gunrock in
some cases, which were Enterprise [ 33], GPUCC [ 53], WS-
VR [ 26], Frog [ 50] and GPUBC [ 47] for BFS, CC, PR, SSSP
and BC, respectively .
5.2 Overall Performance
T able 3 shows the performance comparison of the graph li-
braries as described in Section 5.1. For the BFS benchmark,
Gunrock, G S W I T C H, and the hand-tuned version of Enter-
prise enabled the direction-switching and idempotence opti-
mization in BFS. However, Gunrock needed user-provided
tuning parameters, such as do_a and do_b to decide when
to switch. These parameters can vary signiﬁcantly for differ-
ent graphs. For example, Gunrock’s BFS achieved its best
performance on graph soc-orkut when do_a = 0.12 and
do_b = 0.1, while on graph roadNet-CA, the best set-
ting was do_a = 1 and do_b = 10. The Enterprise adopted
static rule-based switching that led to suboptimal results
in some cases, for instance, on graphs soc-orkut and
web-wikipedia-2009. For the CC benchmark, our method
beat all the programmable GPU graph libraries largely due to
the transition of the active set format. G S W I T C H was slower
than GPUCC in some cases because the latter used speciﬁc op-
timizations, which can not be generalized. For the PageRank
benchmark, each library had a different implementation but
used the same terminal condition. WS-VR used the pull mode
and the WM load-balancing strategy for all cases, whereas
Gunrock used the push mode and the LB [ 13] load-balancing
strategy for all cases. By contrast, G S W I T C H used differ-
ent
strategies for different inputs. For the SSSP benchmark,
our dynamic stepping optimization reduced the number of
touched edges signiﬁcantly compared to the static ∆ -stepping
version. Frog performed well on some graphs because it used
an asynchronous algorithm that convergences more quickly .
For the BC benchmark, both the GPUBC and Gunrock used
a push-based implementation, while G S W I T C H performed
f
aster than Gunrock due to the generalized directional opti-
mization.
Figure 15 shows the performance of G S W I T C H compared
with that of Gunrock on K40m (a) and P100 (b) on the eval-
uation set. On K40m, the results show that the average per-
formance of G S W I T C H was approximately 2.5
∼4.6×faster
than Gunrock on average, and it achieved positive speedups
in about 84% ∼96% of the cases. On P100, we retrained the
model to test the portability of G S W I T C H. The results show
that G S W I T C H was about 2 ∼3.3×faster than Gunrock on
average and 94% ∼99% of the cases were positive. G S W I T C H
can thus choose the suitable strategies automatically , while
Gunrock relies on user-provided tuning parameters that are
frequently unavailable in real-world situations.
209
PPoPP ’19, February 16–20, 2019, Washington, DC, USA Ke Meng, Jiajia Li, Guangming T an, Ninghui Sun
Nvidia K40Nvidia P100
 BFS
(a)
(f
)
Datasize(nnz)
Average
Runtime 
Gunrock:
Gswitch: 
41.2ms
15.4ms
88.1% positiv
e
Datasize(nnz)
Average
Runtime 
Gunrock:
Gswitch: 
38.7ms
13.2ms
97
.5% positive
(b)
CC
(g)
Datasiz
e(nnz)
Average
Runtime 
Gunrock:
Gswitch: 
9.6ms
3.6ms
95.3% positiv
e
Datasize(nnz)
Average
Runtime 
Gunrock:
Gswitch: 
3.9ms
1.2ms
99.6% positiv
e
(c)
PR
(h)
Datasiz
e(nnz)
Average
Runtime 
Gunrock:
Gswitch: 
114.4ms
24.7
ms
96.1% positive
Datasize(nnz)
Average
Runtime 
Gunrock:
Gswitch: 
30.9ms
9.6ms
96.8% positiv
e
(d)
 SSSP
(i)
A
verage
Runtime 
Gunrock:
Gswitch: 
399.8ms
162.6ms
84.5% positiv
e
Datasize(nnz)
Datasiz
e(nnz)
Average
Runtime 
Gunrock:
Gswitch: 
162.6ms
82.1ms
94.7% positiv
e
(e)
(j)
BC
Datasize(nnz)
A
verage
Runtime 
Gunrock:
Gswitch: 
86.4ms
32.4ms
88.8% positiv
e
Datasize(nnz)
A
verage
Runtime 
Gunrock:
Gswitch: 
65.2ms
22.3ms
97
.3% positive
4.8K 530M 4.8K 530M 4.8K 530M
4.8K 530M 4.8K 530M 4.8K 530M 4.8K 530M
4.8K 530M 4.8K 530M
4.8K 530M
Figure 15. Performance improvement normalized to Gunrock on both K40m and P100 GPUs for all the ﬁve graph algorithms.
5.3 Effect of Individual Patterns
P1
P1+P2
P1+P2+P3
P1+P2+P3+P4
P1+P2+P3+P4+P5
GSWITCH baseline
Gunrock
BFS-orkut BFS-msdoor
CC-orkut CC-msdoor PR-orkut PR-msdoor
SSSP-orkut SSSP-msdoor BC-orkut BC-msdoor
Normalized PerformanceNormalized PerformanceNormalized Performance
Normalized PerformanceNormalized Performance
Figure 16. The incremental perfomance of G S W I T C H com-
pared with Gunrock.
T o further understand the improvement in performance ef-
fected by each pattern, we illustrate the incremental speedups
of a scale-free graph ( soc-orkut) and a mesh-like graph
(msdoor) in Figure 16. W e used Gunrock as the static op-
timal version of each algorithm. The results show that the
GS W I T C H baseline (the non-switching version) delivers a
similar performance to that of the Gunrock implementation.
Therefore, the superiority of G S W I T C H in performance over
Gunrock is mainly owing to dynamic switching. Moreover,
the importance of the different algorithmic patterns on perfor-
mance improvement varied from algorithm to algorithm. For
traversal-based algorithms such as BFS and BC, the direc-
tional optimization (P1) yielded an approximately 2 ×bump
Overhead Filter Expand
BFS PR CC SSSP BC
Runtime Breakdown
Overhead per Iteration (µs)

0
1100
200
0
100%
6%
BFS PR CC SSSP BC
200
0
100
81
96
156
120 121
Figure 17. The normalized time breakdown of ﬁve bench-
marks and the overhead of dynamical switching on
soc-orkut.
in performance. The active set format (P2) brought about a
10% performance improvement on dense workloads. Load
balancing (P3) is important for scale-free graphs, where the
workload of each iteration varies signiﬁcantly . Stepping op-
timization (P4) proﬁled in only monotonic algorithms, and
the kernel fusion (P5) reduced the ﬁlter time, which is the
bottleneck in mesh-like graphs.
5.4 Accuracy and Overhead
Each record contained 21 features and one label showing the
optimal strategy as described in Section 3. G S W I T C H had
a
corresponding classiﬁer for each pattern to decide which
strategy to choose. W e statistically analyzed the accuracy of
each pattern during the online decision making. The accura-
cies were 98%, 97%, 85%, 82% and 94% for the ﬁve patterns,
respectively . It is interesting to note that G S W I T C H achieved
good performance even though the learning model could not
correctly predict in some cases. More features or a more com-
plex model can improve accuracy , but will also increase the
runtime of feature extraction and training.
Figure 17 shows the runtime breakdown for the ﬁve bench-
marks. For edge-centric algorithms such as CC, the active
210
Gswitch PP oPP ’19, February 16–20, 2019, Washington, DC, USA
set is large. Thus the Filter and Expand steps cost similar
time. For applications such as PR and BC, which have more
edges to process, the Expand step dominates. The overhead in
GS W I T C H includes collecting the runtime characteristics and
making the decisions. As described in Section 4.3, the feature
vector can be collected efﬁciently without extra computation,
which cost 58 ∼120 µs in each iteration. The overhead of dy-
namic switching in G S W I T C H was affordable, as it cost at
most 6% of total runtime.
6 Related W ork
GS W I T C H was developed as an auto-tuner to implement
adapti
ve graph algorithms. In contrast to traditional auto-
tuners [ 16, 60], it derives from the emerging algorithmic auto-
tuning approaches [ 5, 14, 32, 36]. For the optimization of
graph processing algorithms, its design and implementation
share with and are inspired from past work. W e summarize re-
lated work to clarify the common features with other work and
highlight G S W I T C H’s superiority in implementing a highly
efﬁcient graph algorithm library .
First, a large number of hardwired GPU implementations
provide enough references for the kernel library of the G S W I T C H
auto-tuner. W e refer them to variants of training and search-
ing optimization on GPUs. For example, Merrill et al. ’s BFS
algorithm [ 41] proposed an adaptive load-balanced strategy to
process a frontier with a thread, a warp, or a CT A, according to
the number of neighbors of the given vertex. Enterprise [ 33]
implements the direction-optimized BFS with streamlined
GPU thread scheduling. V erstraaten et al. [ 56] implemented
the directional optimization using a model-driven approach.
Soman et al. [ 53] implemented the CC algorithm with edge-
centric abstraction and accelerated the root ﬁnding procedure
with the pointer jumping technique. Davidson et al. ’s [ 13]
SSSP algorithm uses a near-far approach to reduce the num-
ber of touched edges during the traversal, trading parallelism
for work-efﬁciency . Mclaughlin and Bader [ 40] implemented
an efﬁcient BC algorithm to solve the load imbalance issue
in threads. A straightforward combination of these optimiza-
tion methods fails to achieve better performance. G S W I T C H
leverages a machine learning model to determine a proper
conﬁguration.
Second, several studies have discussed the trade-off among
performance tuning strategies. Unfortunately , they are con-
ﬁned to only a speciﬁc graph algorithm. For example, Beamer
et al. ’s work [ 7] ﬁrst introduced the directional optimization
in an implementation of BFS on CPUs. Nasre et al. [ 44]
analyzed the trade-offs between data-driven and topology-
driven approaches. Hassaan et al. ’s work [
21] discussed the
pros and cons of the ordered and unordered implementations
of irregular algorithms and attempted to balance parallelism
with work-efﬁciency . Inspired by these individual efforts,
GS W I T C H goes further to examine the correlation among
multiple algorithms in terms of tuning. The key point is that
our extracted algorithmic patterns have not been revealed in
previous work.
Last, Gunrock [ 58] integrated the above-mentioned tuning
strategies to implement a number of graph primitives. T o some
extent, this applies common tuning strategies to different
primitives. However, the optimization depends on the users’
priori knowledge. It is a static conﬁguration that may become
suboptimal. G S W I T C H breaks the barrier by automatically
building a better algorithm by assembling strategies at runtime
owing to the algorithmic patterns.
7 Conclusion
In this work, we proposed G S W I T C H, an adaptive perfor-
mance
tuning system for graph processing algorithms. Its
novelty is in a set of algorithmic patterns that naturally en-
able the adoption of a machine learning model. The results
of evaluations show that G S W I T C H yields better performance
than the best conﬁgurations of four other state-of-the-art GPU
graph processing libraries. This work showcases algorithmic
auto-tuning techniques. W e advocate that more research ef-
fort should be put into advancing algorithmic auto-tuning,
especially for input-sensitive workloads such as graph and
sparse tensor algorithms. However, our patterns are not fully
complete yet. W e believe that many opportunities and open
research questions persist for extracting additional patterns.
8 Acknowledgments
W e thank all reviewers for constructive comments that helped
us further clarify this paper. This research was funded by The
National Key Research and Development Program of China
(2016YFB0201305, 2016YFB0200803, 2016YFB0200300),
the National Natural Science Foundation of China under grant
nos. (61521092, 91430218, 31327901, 61472395, 61432018).
This research was also funded by the US Department of En-
ergy , Ofﬁce for Advanced Scientiﬁc Computing (ASCR) un-
der A ward No. 66150: "CENA TE: The Center for Advanced
T echnology Evaluation". Paciﬁc Northwest National Labora-
tory (PNNL) is a multiprogram national laboratory operated
for DOE by Battelle Memorial Institute under Contract DE-
AC05-76RL01830.
References
[1] 2017. Network Repository. http://networkrepository .com/networks.
php
[2] 2018. Comparison with Other Engines. https://gunrock.github.io/
docs/engines_topc.html
[3] September 2016. nvgraph. https://developer.nvidia.com/nvgraph
[4] September 2017. graph500. http://www.graph500.org/
[5] Jason Ansel, Cy P . Chan, Y ee Lok W ong, Marek Olszewski, Qin Zhao,
Alan Edelman, and Saman P . Amarasinghe. 2009. PetaBricks: a lan-
guage and compiler for algorithmic choice. In Proceedings of the 2009
ACM SIGPLAN Conference on Programming Language Design and
Implementation, PLDI 2009, Dublin, Ireland, June 15-21, 2009. 38–49.
https://doi.org/10.1145/1542476.1542481
211
PPoPP ’19, February 16–20, 2019, Washington, DC, USA Ke Meng, Jiajia Li, Guangming T an, Ninghui Sun
[6] Arash Ashari, Naser Sedaghati, John Eisenlohr, Srinivasan
Parthasarathy , and P . Sadayappan. 2015. Fast sparse matrix-
vector multiplication on GPUs for graph applications. In High
P erformance Computing, Networking, Storage and Analysis, SC14:
International Conference for. 781–792.
[7] Scott Beamer, Krste Asanovi ´c, and David Patterson. 2013. Direction-
optimizing breadth-ﬁrst search. Scientiﬁc Programming 21, 3-4 (2013),
137–148.
[8] T al Ben-Nun, Michael Sutton, Sreepathi Pai, and Keshav Pingali. 2017.
Groute: An Asynchronous Multi-GPU Programming Model for Irreg-
ular Computations. In ACM Sigplan Symposium on Principles and
Practice of P arallel Programming. 235–248.
[9] Maciej Besta, MichaÅ ´C Podstawski, Linus Groner, Edgar Solomonik,
and T orsten Hoeﬂer. 2017. T o Push or T o Pull: On Reducing Commu-
nication and Synchronization in Graph Computations. In The Interna-
tional Symposium. 93–104.
[10] Ulrik Brandes. 2001. A faster algorithm for betweenness centrality .
Journal of mathematical sociology 25, 2 (2001), 163–177.
[11] Sergey Brin and Lawrence Page. 2012. Reprint of: The anatomy of a
large-scale hypertextual web search engine. Computer networks 56, 18
(2012), 3825–3833.
[12] Jee W Choi, Amik Singh, and Richard W V uduc. 2010. Model-driven
autotuning of sparse matrix-vector multiply on GPUs. In ACM sigplan
notices, V ol. 45. ACM, 115–126.
[13] Andrew Davidson, Sean Baxter, Michael Garland, and John D. Owens.
2014. W ork-Efﬁcient Parallel GPU Methods for Single-Source Short-
est Paths. In IEEE International P arallel and Distributed Processing
Symposium. 349–359.
[14] Y ufei Ding, Jason Ansel, Kalyan V eeramachaneni, Xipeng Shen, Una-
May Oâ ˘A ´ZReilly , and Saman Amarasinghe. 2015. Autotuning algo-
rithmic choice for input sensitivity . In ACM SIGPLAN Notices, V ol. 50.
ACM, 379–390.
[15] Jens Domke, T orsten Hoeﬂer, and W olfgang E Nagel. 2011. Deadlock-
free oblivious routing for arbitrary topologies. In P arallel & Distributed
Processing Symposium (IPDPS), 2011 IEEE International. IEEE, 616–
627.
[16] Matteo Frigo and Steven G Johnson. 2005. The design and implemen-
tation of FFTW3. Proc. IEEE 93, 2 (2005), 216–231.
[17] Evangelos Georganas, A ydin Buluç, Jarrod Chapman, Leonid Oliker,
Daniel Rokhsar, and Katherine Y elick. 2014. Parallel de bruijn graph
construction and traversal for de novo genome assembly . In Proceedings
of the International Conference for High P erformance Computing,
Networking, Storage and Analysis. IEEE Press, 437–448.
[18] Abdullah Gharaibeh, Lauro BeltrÃ ˇco Costa, Elizeu Santos-Neto, and
Matei Ripeanu. 2017. A yoke of oxen and a thousand chickens for
heavy lifting graph processing. In International Conference on P arallel
Architectures and Compilation T echniques. 345–354.
[19] Joseph E Gonzalez, Y ucheng Low , Haijie Gu, Danny Bickson, and
Carlos Guestrin. 2012. PowerGraph: Distributed Graph-Parallel Com-
putation on Natural Graphs.. In OSDI, V ol. 12. 2.
[20] T ae Jun Ham, Lisa Wu, Narayanan Sundaram, Nadathur Satish, and
Margaret Martonosi. 2016. Graphicionado: A high-performance and
energy-efﬁcient accelerator for graph analytics. In Microarchitecture
(MICRO), 2016 49th Annual IEEE/ACM International Symposium on.
IEEE, 1–13.
[21] Muhammad Amber Hassaan, Martin Burtscher, and Keshav Pingali.
2011. Ordered vs. unordered: a comparison of parallelism and work-
efﬁciency in irregular algorithms. Acm Sigplan Notices 46, 8 (2011),
3–12.
[22] Changwan Hong, Aravind Sukumaran-Rajam, Jinsung Kim, and P
Sadayappan. 2017. MultiGraph: Efﬁcient Graph Processing on GPUs.
In P arallel Architectures and Compilation T echniques (P ACT), 2017
26th International Conference on. IEEE, 27–40.
[23] Sungpack Hong, Kyun Kim Sang, T ayo Oguntebi, and Kunle Olukotun.
2011. Accelerating CUDA graph algorithms at maximum warp. Acm
Sigplan Notices 46, 8 (2011), 267–276.
[24] Rashid Kaleem, Anand V enkat, Sreepathi Pai, Mary Hall, and Keshav
Pingali. 2016. Synchronization Trade-Offs in GPU Implementations of
Graph Algorithms. In P arallel and Distributed Processing Symposium,
2016 IEEE International. 514–523.
[25] Jeremy Kepner, Peter Aaltonen, David Bader, A ydin Buluç, Franz
Franchetti, John Gilbert, Dylan Hutchison, Manoj Kumar, Andrew
Lumsdaine, Henning Meyerhenke, et al . 2016. Mathematical founda-
tions of the GraphBLAS. In High P erformance Extreme Computing
Conference (HPEC), 2016 IEEE. IEEE, 1–9.
[26] Farzad Khorasani, Rajiv Gupta, and Laxmi N. Bhuyan. 2016. Scalable
SIMD-Efﬁcient Graph Processing on GPUs. In International Confer-
ence on P arallel Architecture and Compilation. 39–50.
[27] Farzad Khorasani, Keval V ora, Rajiv Gupta, and Laxmi N. Bhuyan.
2014. CuSha:vertex-centric graph processing on GPUs. In Proceedings
of the 23rd international symposium on High-performance parallel and
distributed computing. 239–252.
[28] John Kloosterman, Jonathan Beaumont, Mick W ollman, Ankit Sethia,
Ron Dreslinski, Trevor Mudge, and Scott Mahlke. 2015. W arpPool:
sharing requests with inter-warp coalescing for throughput processors.
In Proceedings of the 48th International Symposium on Microarchitec-
ture. ACM, 433–444.
[29] Jérôme Kunegis and Julia Preusse. 2012. Fairness on the web: Alter-
natives to the power law . In Proceedings of the 4th Annual ACM W eb
Science Conference. ACM, 175–184.
[30] Aapo Kyrola, Guy E Blelloch, and Carlos Guestrin. 2012. Graphchi:
Large-scale graph computation on just a pc. USENIX.
[31] Da Li and Michela Becchi. 2013. Deploying Graph Algorithms on
GPUs: An Adaptive Solution. In IEEE International Symposium on
P arallel & Distributed Processing. 1013–1024.
[32] Jiajia Li, Guangming T an, Mingyu Chen, and Ninghui Sun. 2013.
SMA T: an input adaptive auto-tuner for sparse matrix-vector multi-
plication. In ACM SIGPLAN Notices, V ol. 48. ACM, 117–126.
[33] Hang Liu and H. Howie Huang. 2015. Enterprise: breadth-ﬁrst graph
traversal on GPUs. In International Conference for High PERFOR-
MANCE Computing, Networking, Storage and Analysis. 68.
[34] Junhong Liu, Xin He, W eifeng Liu, and Guangming T an. [n. d.].
Register-A ware Optimizations for Parallel Sparse Matrix-Matrix Multi-
plication. International Journal of P arallel Programming ([n. d.]).
[35] Junhong Liu, Xin He, W eifeng Liu, and Guangming T an. 2018.
Register-based Implementation of the Sparse General Matrix-matrix
Multiplication on GPUs. SIGPLAN Not. 53, 1 (Feb. 2018), 407–408.
https://doi.org/10.1145/3200691.3178529
[36] Yixun Liu, Eddy Z Zhang, and Xipeng Shen. 2009. A cross-input
adaptive framework for GPU program optimizations. In P arallel & Dis-
tributed Processing, 2009. IPDPS 2009. IEEE International Symposium
on. IEEE, 1–10.
[37] Y ucheng Low , Joseph E. Gonzalez, Aapo Kyrola, Danny Bickson,
Carlos E. Guestrin, and Joseph Hellerstein. 2014. GraphLab: A New
Framework For Parallel Machine Learning. Computer Science (2014).
[38] Y ulong Luo, Guangming T an, Zeyao Mo, and Ninghui Sun. 2015.
F AST: A fast stencil autotuning framework based on an optimal-
solution space model. In Proceedings of the 29th ACM on International
Conference on Supercomputing. ACM, 187–196.
[39] Grzegorz Malewicz, Matthew H. Austern, Aart J. C Bik, James C. Dehn-
ert, Ilan Horn, Naty Leiser, and Grzegorz Czajkowski. 2010. Pregel:
a system for large-scale graph processing. In ACM SIGMOD Interna-
tional Conference on Management of Data. 135–146.
[40] Adam Mclaughlin and David A. Bader. 2015. Scalable and High
Performance Betweenness Centrality on the GPU. In High P erformance
Computing, Networking, Storage and Analysis, SC14: International
Conference for. 572–583.
[41] Duane Merrill, Michael Garland, and Andrew Grimshaw . 2012. Scal-
able GPU graph traversal. Acm Sigplan Notices 47, 8 (2012), 117–128.
212
Gswitch PP oPP ’19, February 16–20, 2019, Washington, DC, USA
[42] Ulrich Meyer and Peter Sanders. 2003. ∆ -stepping: a parallelizable
shortest path algorithm. Journal of Algorithms 49, 1 (2003), 114–152.
[43] Rupesh Nasre, Martin Burtscher, and Keshav Pingali. 2013. Atomic-
free irregular computations on GPUs. In The W orkshop on General
Purpose Processor Using Graphics Processing Units. 96–107.
[44] Rupesh Nasre, Martin Burtscher, and Keshav Pingali. 2013. Data-driven
versus topology-driven irregular computations on GPUs. In P arallel &
Distributed Processing (IPDPS), 2013 IEEE 27th International Sympo-
sium on. IEEE, 463–474.
[45] Y uechao Pan, Y angzihao W ang, Y uduo Wu, Carl Y ang, and John D
Owens. 2017. Multi-GPU graph analytics. In P arallel and Distributed
Processing Symposium (IPDPS), 2017 IEEE International. IEEE, 479–
490.
[46] Amitabha Roy , Ivo Mihailovic, and Willy Zwaenepoel. 2013. X-stream:
Edge-centric graph processing using streaming partitions. In Proceed-
ings of the T wenty-F ourth ACM Symposium on Operating Systems
Principles. ACM, 472–488.
[47] Ahmet Erdem SariyÃijce, Kamer Kaya, Erik Saule, and ÃIJmit V
Ã ˘GatalyÃijrek. 2013. Betweenness centrality on GPUs and heteroge-
neous architectures. Advances in Journalism & Communication 01, 4
(2013), 50–53.
[48] Naser Sedaghati, T e Mu, Louis-Noel Pouchet, Srinivasan Parthasarathy ,
and P Sadayappan. 2015. Automatic selection of sparse matrix repre-
sentation on GPUs. In Proceedings of the 29th ACM on International
Conference on Supercomputing. ACM, 99–108.
[49] Dipanjan Sengupta, Shuaiwen Leon Song, Kapil Agarwal, and Karsten
Schwan. 2015. GraphReduce: processing large-scale graphs on
accelerator-based systems. (2015), 1–12.
[50] Xuanhua Shi, Junling Liang, Sheng Di, Bingsheng He, Hai Jin, Lu
Lu, Zhixiang W ang, Xuan Luo, and Jianlong Zhong. 2015. Optimiza-
tion of asynchronous graph processing on GPU with hybrid coloring
model. In Acm Sigplan Symposium on Principles & Practice of P arallel
Programming. 271–272.
[51] Xuanhua Shi, Zhigao Zheng, Y ongluan Zhou, Hai Jin, Ligang He, Bo
Liu, and Qiang-Sheng Hua. 2018. Graph processing on GPUs: a survey .
ACM Computing Surveys (CSUR) 50, 6 (2018), 81.
[52] Julian Shun and Guy E. Blelloch. 2013. Ligra: A Lightweight Graph
Processing Framework for Shared Memory . Acm Sigplan Notices 48, 8
(2013), 135–146.
[53] Jyothish Soman, Kothapalli Kishore, and P J Narayanan. 2010. A fast
GPU algorithm for graph connectivity . (2010), 1–8.
[54] Nikita Spirin and Jiawei Han. 2012. Survey on web spam detection:
principles and algorithms. ACM SIGKDD Explorations Newsletter 13,
2 (2012), 50–64.
[55] Bryan Thompson, Bryan Thompson, and Bryan Thompson. 2014. Map-
Graph: A High Level API for Fast Development of High Performance
Graph Analytics on GPUs. In The W orkshop on Graph Data Manage-
ment Experiences and Systems. 1–6.
[56] Merijn V erstraaten, Ana Lucia V arbanescu, and Cees de Laat. 2017.
Using Graph Properties to Speed-up GPU-based Graph Traversal: A
Model-driven Approach. arXiv preprint arXiv:1708.01159 (2017).
[57] Kai W ang, Aftab Hussain, Zhiqiang Zuo, Guoqing Xu, and Ardalan
Amiri Sani. 2017. Graspan: A single-machine disk-based graph sys-
tem for interprocedural static analyses of large-scale systems code. In
Proceedings of the T wenty-Second International Conference on Archi-
tectural Support for Programming Languages and Operating Systems.
ACM, 389–404.
[58] Y angzihao W ang, Andrew Davidson, Y uechao Pan, Y uduo Wu, Andy
Riffel, and John D. Owens. 2015. Gunrock: a high-performance graph
processing library on the GPU. In Acm Sigplan Symposium on Princi-
ples & Practice of P arallel Programming. 265–266.
[59] Y angzihao W ang, Y uechao Pan, Andrew Davidson, Y uduo Wu, Carl
Y ang, Leyuan W ang, Muhammad Osama, Chenshan Y uan, W eitang
Liu, Andy T Riffel, et al . 2017. Gunrock: GPU graph analytics. ACM
Transactions on P arallel Computing (TOPC) 4, 1 (2017), 3.
[60] R Clint Whaley and Jack J Dongarra. 1998. Automatically tuned linear
algebra software. In Proceedings of the 1998 ACM/IEEE conference on
Supercomputing. IEEE Computer Society , 1–27.
[61] Chenning Xie, Rong Chen, Haibing Guan, Binyu Zang, and Haibo
Chen. 2015. SYNC or ASYNC: time to fuse for distributed graph-
parallel computation. In ACM Sigplan Symposium on Principles and
Practice of P arallel Programming. 194–204.
[62] Da Y an, Yingyi Bu, Y uanyuan Tian, Amol Deshpande, et al . 2017. Big
graph analytics platforms. F oundations and Trends ® in Databases 7,
1-2 (2017), 1–195.
[63] Xiuxia Zhang, Guangming T an, Shuangbai Xue, Jiajia Li, Keren Zhou,
and Mingyu Chen. 2017. Understanding the GPU Microarchitecture to
Achieve Bare-Metal Performance Tuning. In Proceedings of the 22Nd
ACM SIGPLAN Symposium on Principles and Practice of P arallel
Programming (PP oPP ’17). ACM, New Y ork, NY , USA, 31–43. https:
//doi.org/10.1145/3018743.3018755
[64] Y ue Zhao, Jiajia Li, Chunhua Liao, and Xipeng Shen. [n. d.]. Bridging
the gap between deep learning and sparse matrix format selection.
[65] Jianlong Zhong and Bingsheng He. 2014. Medusa: Simpliﬁed Graph
Processing on GPUs. IEEE Transactions on P arallel & Distributed
Systems 25, 6 (2014), 1543–1552.
[66] Xiaowei Zhu, W enguang Chen, W eimin Zheng, and Xiaosong Ma.
2016. Gemini: A Computation-Centric Distributed Graph Processing
System.. In OSDI. 301–316.
213

Published as a conference paper at ICLR 2019
GENERATIVE CODE MODELING WITH GRAPHS
Marc Brockschmidt, Miltiadis Allamanis, Alexander Gaunt
Microsoft Research
Cambridge, UK
{mabrocks,miallama,algaunt}@microsoft.com
Oleksandr Polozov
Microsoft Research
Redmond, W A, USA
polozov@microsoft.com
ABSTRACT
Generative models for source code are an interesting structured prediction prob-
lem, requiring to reason about both hard syntactic and semantic constraints as
well as about natural, likely programs. We present a novel model for this prob-
lem that uses a graph to represent the intermediate state of the generated output.
Our model generates code by interleaving grammar-driven expansion steps with
graph augmentation and neural message passing steps. An experimental evalua-
tion shows that our new model can generate semantically meaningful expressions,
outperforming a range of strong baselines.
1 I NTRODUCTION
Learning to understand and generate programs is an important building block for procedural artiﬁ-
cial intelligence and more intelligent software engineering tools. It is also an interesting task in the
research of structured prediction methods: while imbued with formal semantics and strict syntactic
rules, natural source code carries aspects of natural languages, since it acts as a means of commu-
nicating intent among developers. Early works in the area have shown that approaches from natural
language processing can be applied successfully to source code (Hindle et al., 2012), whereas the
programming languages community has had successes in focusing exclusively on formal semantics.
More recently, methods handling both modalities (i.e., the formal and natural language aspects) have
shown successes on important software engineering tasks (Raychev et al., 2015; Bichsel et al., 2016;
Allamanis et al., 2018b) and semantic parsing (Yin & Neubig, 2017; Rabinovich et al., 2017).
However, current generative models of source code mostly focus on only one of these modalities at
a time. For example, program synthesis tools based on enumeration and deduction (Solar-Lezama,
2008; Polozov & Gulwani, 2015; Feser et al., 2015; Feng et al., 2018) are successful at generating
programs that satisfy some (usually incomplete) formal speciﬁcation but are often obviously wrong
on manual inspection, as they cannot distinguish unlikely from likely, “natural” programs. On the
other hand, learned code models have succeeded in generating realistic-looking programs (Maddison
& Tarlow, 2014; Bielik et al., 2016; Parisotto et al., 2017; Rabinovich et al., 2017; Yin & Neubig,
2017). However, these programs often fail to be semantically relevant, for example because variables
are not used consistently.
In this work, we try to overcome these challenges for generative code models and present a general
method for generative models that can incorporate structured information that is deterministically
available at generation time. We focus our attention on generating source code and follow the ideas
of program graphs(Allamanis et al., 2018b) that have been shown to learn semantically meaning-
ful representations of (pre-existing) programs. To achieve this, we lift grammar-based tree decoder
models into the graph setting, where the diverse relationships between various elements of the gen-
erated code can be modeled. For this, the syntax tree under generation is augmented with additional
edges denoting known relationships (e.g., last use of variables). We then interleave the steps of the
generative procedure with neural message passing (Gilmer et al., 2017) to compute more precise
representations of the intermediate states of the program generation. This is fundamentally differ-
ent from sequential generative models of graphs (Li et al., 2018; Samanta et al., 2018), which aim
to generate all edges and nodes, whereas our graphs are deterministic augmentations of generated
trees.
To summarize, we present a) a general graph-based generative procedure for highly structured ob-
jects, incorporating rich structural information; b) ExprGen, a new code generation task focused
1
arXiv:1805.08490v2  [cs.LG]  16 Apr 2019
Published as a conference paper at ICLR 2019
Algorithm 1Pseudocode for Expand
Input: Context c, partial AST a, node v to ex-
pand
1: hv ←getRepresentation(c,a,v )
2: rhs ←pickProduction(v,hv)
3: for child node type ℓ∈rhs do
4: (a,u) ←insertChild(a,ℓ)
5: if ℓis nonterminal type then
6: a←Expand(c,a,u )
7: return a
int ilOffsetIdx =Array.IndexOf(sortedILOffsets, map.ILOffset);int nextILOffsetIdx = ilOffsetIdx + 1;int nextMapILOffset =
nextILOffsetIdx < sortedILOffsets.Length
? sortedILOffsets[nextILOffsetIdx]: int.MaxValue;
Figure 1: Example for ExprGen, target ex-
pression to be generated is marked . Taken
from BenchmarkDotNet, lightly edited for
formatting.
on generating small, but semantically complex expressions conditioned on source code context; and
c) a comprehensive experimental evaluation of our generative procedure and a range of baseline
methods from the literature.
2 B ACKGROUND & TASK
The most general form of the code generation task is to produce a (partial) program in a program-
ming language given some context information c. This context information can be natural language
(as in, e.g., semantic parsing), input-output examples (e.g., inductive program synthesis), partial pro-
gram sketches, etc. Early methods generate source code as a sequence of tokens (Hindle et al., 2012;
Hellendoorn & Devanbu, 2017) and sometimes fail to produce syntactically correct code. More re-
cent models are sidestepping this issue by using the target language’s grammar to generate abstract
syntax trees (ASTs) (Maddison & Tarlow, 2014; Bielik et al., 2016; Parisotto et al., 2017; Yin &
Neubig, 2017; Rabinovich et al., 2017), which are syntactically correct by construction.
In this work, we follow the AST generation approach. The key idea is to construct the AST a
sequentially, by expanding one node at a time using production rules from the underlying program-
ming language grammar. This simpliﬁes the code generation task to a sequence of classiﬁcation
problems, in which an appropriate production rule has to be chosen based on the context informa-
tion and the partial AST generated so far. In this work, we simplify the problem further — similar
to Maddison & Tarlow (2014); Bielik et al. (2016) — by ﬁxing the order of the sequence to always
expand the left-most, bottom-most nonterminal node. Alg. 1 illustrates the common structure of
AST-generating models. Then, the probability of generating a given AST agiven some context cis
p(a|c) =
∏
t
p(at |c,a<t), (1)
where at is the production choice at step tand a<t the partial syntax tree generated before step t.
Code Generation as Hole CompletionWe introduce theExprGen task of ﬁlling in code within a
hole of an otherwise existing program. This is similar, but not identical to the auto-completion func-
tion in a code editor, as we assume information about the following code as well and aim to generate
whole expressions rather than single tokens. The ExprGen task also resembles program sketch-
ing (Solar-Lezama, 2008) but we give no other (formal) speciﬁcation other than the surrounding
code. Concretely, we restrict ourselves to expressions that have Boolean, arithmetic or string type,
or arrays of such types, excluding expressions of other types or expressions that use project-speciﬁc
APIs. An example is shown in Fig. 1. We picked this subset because it already has rich semantics
that can require reasoning about the interplay of different variables, while it still only relies on few
operators and does not require to solve the problem of open vocabularies of full programs, where an
unbounded number of methods would need to be considered.
In our setting, the context cis the pre-existing code around a hole for which we want to generate an
expression. This also includes the set of variables v1,...,v ℓ that are in scope at this point, which
can be used to guide the decoding procedure (Maddison & Tarlow, 2014). Note, however, that our
method is not restricted to code generation and can be easily extended to all other tasks and domains
that can be captured by variations of Alg. 1 (e.g. in NLP).
2
Published as a conference paper at ICLR 2019
3 G RAPH DECODING FOR SOURCE CODE
To tackle the code generation task presented in the previous section, we have to make two de-
sign choices: (a) we need to ﬁnd a way to encode the code context c,v1,...,v ℓ and (b) we need
to construct a model that can learn p(at | c,a<t) well. We do not investigate the question of
encoding the context in this paper, and use two existing methods in our experiments in Sect. 5.
Both these encoders yield a distributed vector representation for the overall context, representations
ht1 ,..., htT for all tokens in the context, and separate representations for each of the in-scope vari-
ables v1,...,v ℓ, summarizing how each variable is used in the context. This information can then
be used in the generation process, which is the main contribution of our work and is described in
this section.
Overview Our decoder model follows the grammar-driven AST generation strategy of prior work
as shown in Alg. 1. The core difference is in how we compute the representation of the node to
expand. Maddison & Tarlow (2014) construct it entirely from the representation of its parent in
the AST using a log-bilinear model. Rabinovich et al. (2017) construct the representation of a node
using the parents of the AST node but also found it helpful to take the relationship to the parent node
(e.g. “condition of a while”) into account. Yin & Neubig (2017) on the other hand propose to take
the last expansion step into account, which may have ﬁnished a subtree “to the left”. In practice,
these additional relationships are usually encoded by using gated recurrent units with varying input
sizes.
We propose to generalize and unify these ideas using a graph to structure the ﬂow of information
in the model. Concretely, we use a variation of attribute grammars (Knuth, 1967) from compiler
theory to derive the structure of this graph. We associate each node in the AST with two fresh nodes
representing inherited resp. synthesized information (or attributes). Inherited information is derived
from the context and parts of the AST that are already generated, whereas synthesized information
can be viewed as a “summary” of a subtree. In classical compiler theory, inherited attributes usu-
ally contain information such as declared variables and their types (to allow the compiler to check
that only declared variables are used), whereas synthesized attributes carry information about a sub-
tree “to the right” (e.g., which variables have been declared). Traditionally, to implement this, the
language grammar has to be extended with explicit rules for deriving and synthesizing attributes.
To transfer this idea to the deep learning domain, we represent attributes by distributed vector
representations and train neural networks to learn how to compute attributes. Our method for
getRepresentation from Alg. 1 thus factors into two parts: a deterministic procedure that turns a
partial AST a<t into a graph by adding additional edges that encode attribute relationships, and a
graph neural network that learns from this graph.
Notation Formally, we represent programs as graphs where nodes u,v,... are either the AST
nodes or their associated attribute nodes, and typed directed edges ⟨u,τ,v ⟩∈E connect the nodes
according to the ﬂow of information in the model. The edge types τ represent different syntactic
or semantic relations in the information ﬂow, discussed in detail below. We write Ev for the set of
incoming edges into v. We also use functions like parent(a,v) and lastSibling(a,v) that look up
and return nodes from the AST a(e.g. resp. the parent node of vor the preceding AST sibling of v).
Algorithm 2Pseudocode for ComputeEdge
Input: Partial AST a, node v
1: Edge set E← ∅
2: if v is inherited then
3: E←E∪{⟨ parent(a, v), Child, v⟩}
4: if v is terminal node then
5: E←E∪{⟨ lastToken(a, v), NextToken, v⟩}
6: if v is variable then
7: E←E∪{⟨ lastUse(a, v), NextUse, v⟩}
8: if v is not ﬁrst child then
9: E←E∪{⟨ lastSibling(a, v), NextSib, v⟩}
10: else
11: E←E∪{⟨ u, Parent, v⟩| u ∈children(a, v)}
12: E←E∪{⟨ inheritedAttr(v), InhToSyn, v⟩}
13: return E
Example Consider the AST of the expres-
sion i - j shown in Fig. 2 (annotated with
attribute relationships) constructed step by step
by our model. The AST derivation using the
programming language grammar is indicated
by shaded backgrounds, nonterminal nodes are
shown as rounded rectangles, and terminal
nodes are shown as rectangles. We addition-
ally show the variables given within the context
as dashed rectangles at the bottom. First, the
root node, Expr, was expanded using the pro-
duction rule (1) :Expr =⇒Expr - Expr.
Then, its two nonterminal children were in turn
3
Published as a conference paper at ICLR 2019
Expr
0
i
1
j
2
(1)  Expr ⟹ Expr − Expr
Expr
0
Expr
3
Expr-
i
1
j
2
Initial State Step 1
Child
(2)  Expr ⟹ 𝒱
Expr
0
Expr
3
Expr
i
4
-
i
1
j
2
Step 2
Child
NextUse
Expr
0
Expr
3 5
Expr
i
4
-
i
1
j
2
Step 3
Child
NextUse
Expr
0
Expr
3 5
Expr
i
4
-
6
i
1
j
2
Step 4
Child
NextUse
NextSib
Expr
0
Expr
3 5
Expr
7
i
4
-
6
i
1
j
2
Step 5
Child
NextUse
NextSib NextSib
(2)  Expr ⟹ 𝒱Expr
0
Expr
3 5
Expr
7
i
4
j
8
-
6
i
1
j
2
Step 6
Child
NextUse
NextSib NextSib
NextUse
Expr
0
Expr
3 5
Expr
7 9
i
4
j
8
-
6
i
1
j
2
Step 7
Child
NextUse
NextSib NextSib
NextUse
Expr
0 10
Expr
3 5
Expr
7 9
i
4
j
8
-
6
i
1
j
2
Step 8
Child
NextUse
NextSib NextSib
NextUse
Parent
Figure 2: Example AST with attribute dependencies, shown constructed step by step in the order of
generation. Each AST node (labeled by a terminal or non-terminal) has either one or two associated
attribute nodes, shown as its left/right parts. The node IDs are highlighted at the corresponding
generation step. Edge color and label indicate edge type. Edges are computed using Alg. 2, but are
only depicted after use in message passing. Best viewed in color.
expanded to the set of known variables using
the production rule (2) : Expr =⇒ V, choosing i for the ﬁrst variable and j for the second
variable (cf. below for details on picking variables).
Attribute nodes are shown overlaying their corresponding AST nodes. For example, the root node
is associated with its inherited attributes node 0 and with node 10 for its synthesized attributes. For
simplicity, we use the same representation for inherited and synthesized attributes of terminal nodes.
Edges ina<t We discuss the edges used in ourneural attribute grammars(NAG) on our example
below, and show them in Fig. 2 using different edge drawing styles for different edge types. Once a
node is generated, the edges connecting this node can be deterministically added to a<t (precisely
deﬁned in Alg. 2). The list of different edge types used in our model is as follows:
• Child (red) edges connect an inherited attribute node to the inherited attributes nodes of its
children, as seen in the edges from node 0. These are the connections in standard syntax-
driven decoders (Maddison & Tarlow, 2014; Parisotto et al., 2017; Yin & Neubig, 2017;
Rabinovich et al., 2017).
• Parent (green) edges connect a synthesized attribute node to the synthesized attribute node
of its AST parent, as seen in the edges leading to node 10. These are the additional con-
nections used by the R3NN decoder introduced by Parisotto et al. (2017).
• NextSib (black) edges connect the synthesized attribute node to the inherited attribute node
of its next sibling (e.g. from node 5 to node 6). These allow information about the synthe-
sized attribute nodes from a fully generated subtree to ﬂow to the next subtree.
• NextUse (orange) edges connect the attribute nodes of a variable (since variables are al-
ways terminal nodes, we do not distinguish inherited from synthesized attributes) to their
4
Published as a conference paper at ICLR 2019
next use. Unlike Allamanis et al. (2018b), we do not perform a dataﬂow analysis, but in-
stead just follow the lexical order. This can create edges from nodes of variables in the
context c (for example, from node 1 to 4 in Fig. 2), or can connect AST leaf nodes that
represent multiple uses of the same variable within the generated expressions.
• NextToken (blue) edges connect a terminal node (a token) to the next token in the program
text, for example between nodes 4 and 6.
• InhToSyn edges (not shown in Fig. 2) connect the inherited attributes nodes to its synthe-
sized attribute nodes. This is not strictly adding any information, but we found it to help
with training.
The panels of Fig. 2 show the timesteps at which the representations of particular attribute nodes are
computed and added to the graph. For example, in the second step, the attributes for the terminal
token i (node 4) in Fig. 2 are computed from the inherited attributes of its AST parent Expr (node
3), the attributes of the last use of the variablei (node 1), and the node labeli. In the third step, this
computed attribute is used to compute the synthesized attributes of its AST parent Expr (node 5).
Attribute Node Representations To compute the neural attribute representationhv of an attribute
node vwhose corresponding AST node is labeled with ℓv, we ﬁrst obtain its incoming edges using
Alg. 2 and then use the state update function from Gated Graph Neural Networks (GGNN) (Li et al.,
2016). Thus, we take the attribute representations hui at edge sources ui, transform them according
to the corresponding edge type ti using a learned function fti , aggregate them (by elementwise
summation) and combine them with the learned embedding emb(ℓv) of the node label ℓv using a
function g:
hv = g(emb(ℓv),
∑
⟨ui,ti,v⟩∈Ev
fti (hui )) (2)
In practice, we use a single linear layer for fti and implement g as a gated recurrent unit (Cho
et al., 2014). We compute node representations in such an order that all hui appearing on the right
of (2) are already computed. This is possible as the graphs obtained by repeated application of
Alg. 2 are directed acyclic graphs rooted in the inherited attribute node of the root node of the AST.
We initialize the representation of the root inherited attribute to the representation returned by the
encoder for the context information.
Choosing Productions, Variables & LiteralsWe can treat picking production rules as a simple
classiﬁcation problem over all valid production rules, masking out those choices that do not corre-
spond to the currently considered nonterminal. For a nonterminal node vwith label ℓv and inherited
attributes hv, we thus deﬁne
pickProduction(ℓv,hv) = arg maxP(rule |ℓv,hv) = arg max [e(hv) +mℓv ] . (3)
Here, mℓv is a mask vector whose value is 0 for valid productions ℓv ⇒... and −∞for all other
productions. In practice, we implement eusing a linear layer.
Similarly, we pick variables from the set of variables Vin scope using their representations hvvar
(initially the representation obtained from the context, and later the attribute representation of the
last node in the graph in which they have been used) by using a pointer network (Vinyals et al.,
2015). Concretely, to pick a variable at node v, we use learnable linear function kand deﬁne
pickVariable(V,hv) = arg max
var∈V
P(var |hv) = arg max
var∈V
k(hv,hvvar ). (4)
Note that since the model always picks a variable from the set of in-scope variablesV, this generation
model can never predict an unknown or out-of-scope variable.
Finally, to generate literals, we combine a small vocabulary Lof common literals observed in the
training data and special UNK tokens for each type of literal with another pointer network that can
copy one of the tokens t1 ...t T from the context. Thus, to pick a literal at node v, we deﬁne
pickLiteral(V,hv) = arg max
lit∈L∪{t1...tT }
P(lit |hv). (5)
5
Published as a conference paper at ICLR 2019
Note that this is the only operation that may produce an unknown token ( i.e. an UNK literal). In
practice, we implement this by learning two functions sL and sc, such that sL(hv) produces a score
for each token from the vocabulary and sc(hv,hti ) computes a score for copying token ti from the
context. By computing a softmax over all resulting values and normalizing it by summing up entries
corresponding to the same constant, we can learn to approximate the desired P(lit |hv).
Training & Training ObjectiveThe different shapes and sizes of generated expressions compli-
cate an efﬁcient training regime. However, note that given a ground truth target tree, we can easily
augment it with all additional edges according to Alg. 2. Given that full graph, we can compute a
propagation schedule (intuitively, a topological ordering of the nodes in the graph, starting in the root
node) that allows to repeatedly apply(2) to obtain representations for all nodes in the graph. By rep-
resenting a batch of graphs as one large (sparse) graph with many disconnected components, similar
to Allamanis et al. (2018b), we can train our graph neural network efﬁciently. We have released the
code for this onhttps://github.com/Microsoft/graph-based-code-modelling .
Our training procedure thus combines an encoder (cf. Sect. 5), whose output is used to initialize the
representation of the root and context variable nodes in our augmented syntax graph, the sequential
graph propagation procedure described above, and the decoder choice functions (3) and (4). We
train the system end-to-end using a maximum likelihood objective without pre-trained components.
Additional Improvements We extend (3) with an attention mechanism (Bahdanau et al., 2014;
Luong et al., 2015) that uses the state hv of the currently expanded node vas a key and the context
token representations ht1 ,...,h tT as memories. Experimentally, we found that extending Eqs. 4, 5
similarly did not improve results, probably due to the fact that they already are highly dependent on
the context information.
Following Rabinovich et al. (2017), we provide additional information for Child edges. To allow
this, we change our setup so that some edge types also require an additional label, which is used
when computing the messages sent between different nodes in the graph. Concretely, we extend(2)
by considering sets of unlabeled edges Ev and labeled edges Eℓ
v:
hv = g(emb(ℓv),
∑
(ui,ti,v)∈Ev
fti (hui ) +
∑
(ui,ti,ℓi,v)∈Eℓv
fti (hui ,embe(ℓi))) (6)
Thus for labeled edge types, fti takes two inputs and we additionally introduce a learnable embed-
ding for the edge labels. In our experiments, we found it useful to labelChild with tuples consisting
of the chosen production and the index of the child, i.e., in Fig. 2, we would label the edge from 0
to 3 with (2,0), the edge from 0 to 6 with (2,1), etc.
Furthermore, we have extended pickProduction to also take the information about available
variables into account. Intuitively, this is useful in cases of productions such as Expr =⇒
Expr.Length, which can only be used in a well-typed derivation if an array-typed variable is
available. Thus, we extend e(hv) from (3) to additionally take the representation of all variables
in scope into account, i.e., e(hv,r({hvvar |var ∈V})), where we have implemented r as a max
pooling operation.
4 R ELATED WORK
Source code generation has been studied in a wide range of different settings (Allamanis et al.,
2018a). We focus on the most closely related works in language modeling here. Early works ap-
proach the task by generating code as sequences of tokens (Hindle et al., 2012; Hellendoorn &
Devanbu, 2017), whereas newer methods have focused on leveraging the known target grammar and
generate code as trees (Maddison & Tarlow, 2014; Bielik et al., 2016; Parisotto et al., 2017; Yin &
Neubig, 2017; Rabinovich et al., 2017) ( cf. Sect. 2 for an overview). While modern models suc-
ceed at generating “natural-looking” programs, they often fail to respect simple semantic rules. For
example, variables are often used without initialization or written several times without being read
inbetween.
Existing tree-based generative models primarily differ in what information they use to decide which
expansion rule to use next. Maddison & Tarlow (2014) consider the representation of the immediate
6
Published as a conference paper at ICLR 2019
parent node, and suggest to consider more information ( e.g., nearby tokens). Parisotto et al. (2017)
compute a fresh representation of the partial tree at each expansion step using R3NNs (which in-
tuitively perform a leaf-to-root traversal followed by root-to-leaf traversal of the AST). The PHOG
model (Bielik et al., 2016) conditions generation steps on the result of learned (decision tree-style)
programs, which can do bounded AST traversals to consider nearby tokens and non-terminal nodes.
The language also supports a jump to the last node with the same identiﬁer, which can serve as
syntactic approximation of data-ﬂow analysis. Rabinovich et al. (2017) only use information about
the parent node, but use neural networks specialized to different non-terminals to gain more ﬁne-
grained control about the ﬂow of information to different successor nodes. Finally, Amodio et al.
(2017) and Yin & Neubig (2017) follow a left-to-right, depth-ﬁrst expansion strategy, but thread up-
dates to single state (via a gated recurrent unit) through the overall generation procedure, thus giving
the pickProduction procedure access to the full generation history as well as the representation of
the parent node. Amodio et al. (2017) also suggest the use of attribute grammars, but use them to
deﬁne a deterministic procedure that collects information throughout the generation process, which
is provided as additional feature.
As far as we are aware, previous work has not considered a task in which a generative model ﬁlls a
hole in a program with an expression. Lanuage model-like methods take into account only the lexi-
cographically previous context of code. The task of Raychev et al. (2014) is near to our ExprGen,
but instead focuses on ﬁlling holes in sequences of API calls. There, the core problem is identifying
the correct function to call from a potentially large set of functions, given a sequence context. In
contrast, ExprGen requires to handle arbitrary code in the context, and then to build possibly com-
plex expressions from a small set of operators. Allamanis et al. (2018b) consider similar context, but
are only picking a single variable from a set of candidates, and thus require no generative modeling.
5 E VALUATION
Dataset We have collected a dataset for our ExprGen task from 593 highly-starred open-source
C# projects on GitHub, removing any near-duplicate ﬁles, following the work of Lopes et al. (2017).
We parsed all C# ﬁles and identiﬁed all expressions of the fragment that we are considering ( i.e.,
restricted to numeric, Boolean and string types, or arrays of such values; and not using any user-
deﬁned functions). We then remove the expression, perform a static analysis to determine the nec-
essary context information and extract a sample. For each sample, we create an abstract syntax
tree by coarsening the syntax tree generated by the C # compiler Roslyn. This resulted in 343 974
samples overall with 4.3 (±3.8) tokens per expression to generate, or alternatively 3.7 ( ±3.1) pro-
duction steps. We split the data into four separate sets. A “test-only” dataset is made up from∼100k
samples generated from 114 projects. The remaining data we split into training-validation-test sets
(3 : 1 : 1), keeping all expressions collected from a single source ﬁle within a single fold. Samples
from our dataset can be found in the supplementary material. Our decoder uses the grammar made
up by 222 production rules observed in the ASTs of the training set, which includes rules such as
Expr =⇒Expr + Expr for binary operations, Expr =⇒Expr.Equals(Expr) for built-in
methods, etc.
Encoders We consider two models to encode context information. Seq is a two-layer bi-
directional recurrent neural network (using a GRU (Cho et al., 2014)) to encode the tokens before
and after the “hole” in which we want to generate an expression. Additionally, it computes a rep-
resentation for each variable var in scope in the context in a similar manner: For each variable var
it identiﬁes usages before/after the hole and encodes each of them independently using a second
bi-directional two-layer GRU, which processes a window of tokens around each variable usage. It
then computes a representation for var by average pooling of the ﬁnal states of these GRU runs.
The second encoder Gis an implementation of the program graph approach introduced by Allamanis
et al. (2018b). We follow the transformation used for the Varmisuse task presented in that paper,
i.e., the program is transformed into a graph, and the target expression is replaced by a fresh dummy
node. We then run a graph neural network for 8 steps to obtain representations for all nodes in the
graph, allowing us to read out a representation for the “hole” (from the introduced dummy node)
and for all variables in context. The used context information captured by the GNN is a superset of
what existing methods (e.g. language models) consider.
7
Published as a conference paper at ICLR 2019
Baseline Decoders We compare our model to re-implementations of baselines from the litera-
ture. As our ExprGen task is new, re-using existing implementations is hard and problematic in
comparison. Most recent baseline methods can be approximated by ablations of our model. We
experimented with a simple sequence decoder with attention and copying over the input, but found
it to be substantially weaker than other models in all regards. Next, we consider Tree, our model
restricted to using only Child edges without edge labels. This can be viewed as an evolution of
Maddison & Tarlow (2014), with the difference that instead of a log-bilinear network that does
not maintain state during the generation, we use a GRU. ASN is similar to abstract syntax net-
works (Rabinovich et al., 2017) and arises as an extension of theTreemodel by adding edge labels
on Child that encode the chosen production and the index of the child (corresponding to the “ﬁeld
name” Rabinovich et al. (2017)). Finally, Synfollows the work of Yin & Neubig (2017), but uses
a GRU instead of an LSTM. For this, we extend Treeby a new NextExp edge that connects nodes
to each other in the expansion sequence of the tree, thus corresponding to the action ﬂow (Yin &
Neubig, 2017).
In all cases, our re-implementations improve on prior work in our variable selection mechanism,
which ensures that generated programs only use variables that are deﬁned and in scope. Both Rabi-
novich et al. (2017) and Yin & Neubig (2017) instead use a copying mechanism from the context.
On the other hand, they use RNN modules to generate function names and choose arguments from
the context (Yin & Neubig, 2017) and to generate string literals (Rabinovich et al., 2017). Our
ExprGen task limits the set of allowed functions and string literals substantially and thus no RNN
decoder generating such things is required in our experiments.
The authors of the PHOG (Bielik et al., 2016) language model kindly ran experiments on our data
for the ExprGen task, to provide baseline results of a non-neural language model. Note, however,
that PHOG does not consider the code context to the right of the expression to generate, and does no
additional analyses to determine which variable choices are valid. Extending the model to take more
context into account and do some analyses to restrict choices would certainly improve its results.
5.1 Q UANTITATIVE EVALUATION
Metrics We are interested in the ability of a model to generate valid expressions based on the
current code context. To evaluate this, we consider four metrics. As our ExprGen task requires
a conditional language model of code, we ﬁrst consider the per-token perplexity of the model; the
lower the perplexity, the better the model ﬁts the real data distribution. We then evaluate how often
the generated expression is well-typed ( i.e., can be typed in the original code context). We report
these metrics for the most likely expression returned by beam search decoding with beam width 5.
Finally, we compute how often the ground truth expression was generated (reported for the most
likely expression, as well as for the top ﬁve expressions). This measure is stricter than semantic
equivalence, as an expression j > i will not match the equivalent i < j.
Results We show the results of our evaluation in Tab. 1. Overall, the graph encoder architecture
seems to be best-suited for this task. All models learn to generate syntactically valid code (which
is relatively simple in our domain). However, the different encoder models perform very differently
on semantic measures such as well-typedness and the retrieval of the ground truth expression. Most
of the type errors are due to usage of an “UNK” literal (for example, the G→NAG model only
has 4% type error when ﬁltering out such unknown literals). The results show a clear trend that
correlates better semantic results with the amount of information about the partially generated pro-
grams employed by the generative models. Transferring a trained model to unseen projects with
a new project-speciﬁc vocabulary substantially worsens results, as expected. Overall, our NAG
model, combining and adding additional signal sources, seems to perform best on most measures,
and seems to be least-impacted by the transfer.
5.2 Q UALITATIVE EVALUATION
As the results in the previous section suggest, the proposed ExprGen task is hard even for the
strongest models we evaluated, achieving no more than 50% accuracy on the top prediction. It
is also unsolvable for classical logico-deductive program synthesis systems, as the provided code
context does not form a precise speciﬁcation. However, we do know that most instances of the task
8
Published as a conference paper at ICLR 2019
Table 1: Evaluation of encoder and decoder combinations on predicting an expression from code
context. †: PHOG (Bielik et al., 2016) is only conditioned on the tokens on the left of the expression.
Model Test (from seen projects) Test-only (from unseen projects)
Perplexity Well-Typed Acc@1 Acc@5 Perplexity Well-Typed Acc@1 Acc@5
PHOG† – – 34.8% 42.9% – – 28.0% 37.3%
Seq→Seq 87.48 32.4% 21.8% 28.1% 130.46 23.4% 10.8% 16.8%
Seq→NAG 6.81 53.2% 17.7% 33.7% 8.38 40.4% 8.4% 15.8%
G→Seq 93.31 40.9% 27.1% 34.8% 28.48 36.3% 17.2% 25.6%
G→T ree 4.37 49.3% 26.8% 48.9% 5.37 41.2% 19.9% 36.8%
G→ASN 2.62 78.7% 45.7% 62.0% 3.03 74.7% 32.4% 48.1%
G→Syn 2.71 84.9% 50.5% 66.8% 3.48 84.5% 36.0% 52.7%
G→NAG 2.56 86.4% 52.3% 69.2% 3.07 84.5% 38.8% 57.0%
int methParamCount = 0;if (paramCount > 0) {IParameterTypeInformation[] moduleParamArr =GetParamTypeInformations(Dummy.Signature, paramCount);methParamCount = moduleParamArr.Length;}
if (paramCount > methParamCount) {
IParameterTypeInformation[] moduleParamArr =GetParamTypeInformations(Dummy.Signature,paramCount - methParamCount);}
G→NAG :
paramCount > methParamCount (34.4%)
paramCount == methParamCount (11.4%)
paramCount < methParamCount (10.0%)
G→ASN :
paramCount == 0 (12.7%)
paramCount < 0 (11.5%)
paramCount > 0 (8.0%)
public static String URItoPath(String uri) {if (System.Text.RegularExpressions.Regex.IsMatch(uri, "^file:\\\\[a-z,A-Z]:")) {return uri.Substring(6);}
if (uri.StartsWith(@"file:")) {
return uri.Substring(5);}return uri;}
G→NAG :
uri.Contains(UNK_STRING_LITERAL) (32.4%)
uri.StartsWith(UNK_STRING_LITERAL) (29.2%)
uri.HasValue() (7.7%)
G→S yn:
uri == UNK_STRING_LITERAL (26.4%)
uri == "" (8.5%)
uri.StartsWith(UNK_STRING_LITERAL) (6.7%)
Figure 3: Two lightly edited examples from our test set and expressions predicted by different
models. More examples can be found in the supplementary material.
are (easily) solvable for professional software developers, and thus believe that machine learning
systems can have considerable success on the task.
Fig. 3 shows two (abbreviated) samples from our test set, together with the predictions made by the
two strongest models we evaluated. In the ﬁrst example, we can see that the G→NAG model cor-
rectly identiﬁes that the relationship between paramCount and methParamCount is important
(as they appear together in the blocked guarded by the expression to generate), and thus generates
comparison expressions between the two variables. The G → ASNmodel lacks the ability to
recognize that paramCount (or any variable) was already used and thus fails to insert both rel-
evant variables. We found this to be a common failure, often leading to suggestions using only
one variable (possibly repeatedly). In the second example, both G →NAGand G →Syn have
learned the common if (var.StartsWith(...)) { ... var.Substring(num)
... } pattern, but of course fail to produce the correct string literal in the condition. We show
results for all of our models for these examples, as well as for as additional examples, in the supple-
mentary material B.
6 D ISCUSSION & CONCLUSIONS
We presented a generative code model that leverages known semantics of partially generated pro-
grams to direct the generative procedure. The key idea is to augment partial programs to obtain a
graph, and then use graph neural networks to compute a precise representation for the partial pro-
gram. This representation then helps to better guide the remainder of the generative procedure. We
9
Published as a conference paper at ICLR 2019
have shown that this approach can be used to generate small but semantically interesting expressions
from very imprecise context information. The presented model could be useful in program repair
scenarios (where repair proposals need to be scored, based on their context) or in the code review
setting (where it could highlight very unlikely expressions). We also believe that similar models
could have applications in related domains, such as semantic parsing, neural program synthesis and
text generation.
REFERENCES
Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. A survey of machine learning for
big code and naturalness. ACM Computing Surveys, 2018a.
Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. Learning to represent programs with
graphs. In International Conference on Learning Representations (ICLR), 2018b.
Matthew Amodio, Swarat Chaudhuri, and Thomas W. Reps. Neural attribute machines for program generation.
arXiv preprint arXiv:1705.09231, 2017.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to
align and translate. In International Conference on Learning Representations (ICLR), 2014.
Benjamin Bichsel, Veselin Raychev, Petar Tsankov, and Martin Vechev. Statistical deobfuscation of android
applications. In Conference on Computer and Communications Security (CCS), 2016.
Pavol Bielik, Veselin Raychev, and Martin Vechev. PHOG: probabilistic model for code. In International
Conference on Machine Learning (ICML), 2016.
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural
machine translation: Encoder–decoder approaches. Syntax, Semantics and Structure in Statistical Transla-
tion, 2014.
Yu Feng, Ruben Martins, Osbert Bastani, and Isil Dillig. Program synthesis using conﬂict-driven learning. In
Programming Languages Design and Implementation (PLDI), 2018.
John K. Feser, Swarat Chaudhuri, and Isil Dillig. Synthesizing data structure transformations from input-output
examples. In Programming Languages Design and Implementation (PLDI), 2015.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message
passing for quantum chemistry. In International Conference on Machine Learning (ICML), 2017.
Vincent J. Hellendoorn and Premkumar Devanbu. Are deep neural networks the best choice for modeling
source code? In Foundations of Software Engineering (FSE), 2017.
Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. On the naturalness of
software. In International Conference on Software Engineering (ICSE), 2012.
Donald E. Knuth. Semantics of context-free languages. Mathemtical Systems Theory, 2(2):127–145, 1967.
Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. In
International Conference on Learning Representations (ICLR), 2016.
Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning deep generative models of
graphs. CoRR, abs/1803.03324, 2018.
Cristina V Lopes, Petr Maj, Pedro Martins, Vaibhav Saini, Di Yang, Jakub Zitny, Hitesh Sajnani, and Jan Vitek.
DéjàVu: a map of code duplicates on GitHub. In Object-Oriented Programming, Systems, Languages, and
Applications (OOPSLA), 2017.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural
machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2015.
Chris J Maddison and Daniel Tarlow. Structured generative models of natural source code. In International
Conference on Machine Learning (ICML), 2014.
Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli.
Neuro-symbolic program synthesis. In International Conference on Learning Representations (ICLR), 2017.
10
Published as a conference paper at ICLR 2019
Oleksandr Polozov and Sumit Gulwani. FlashMeta: a framework for inductive program synthesis. In Object-
Oriented Programming, Systems, Languages, and Applications (OOPSLA), 2015.
Maxim Rabinovich, Mitchell Stern, and Dan Klein. Abstract syntax networks for code generation and semantic
parsing. In Annual Meeting of the Association for Computational Linguistics (ACL), 2017.
Veselin Raychev, Martin Vechev, and Eran Yahav. Code completion with statistical language models. In
Programming Languages Design and Implementation (PLDI), 2014.
Veselin Raychev, Martin Vechev, and Andreas Krause. Predicting program properties from Big Code. In
Principles of Programming Languages (POPL), 2015.
Bidisha Samanta, Abir De, Niloy Ganguly, and Manuel Gomez-Rodriguez. Designing random graph models
using variational autoencoders with applications to chemical design. CoRR, abs/1802.05283, 2018.
Armando Solar-Lezama. Program synthesis by sketching. University of California, Berkeley, 2008.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Advances in Neural Information
Processing Systems, 2015.
Pengcheng Yin and Graham Neubig. A syntactic neural model for general-purpose code generation. In Annual
Meeting of the Association for Computational Linguistics (ACL), 2017.
11
Published as a conference paper at ICLR 2019
A D ATASET SAMPLES
Below we list some sample snippets from the training set for our ExprGen task. The highlighted
expressions are to be generated.
for (int i=0; i < 3*timeSpanUnits + 1 ; ++i) {
consolidator.Update(new TradeBar { Time = refDateTime });
if (i < timeSpanUnits) { // before initial consolidation happens
Assert.IsNull(consolidated);
}
else {
Assert.IsNotNull(consolidated);
if ( i % timeSpanUnits == 0 ) { // i = 3, 6, 9
Assert.AreEqual(refDateTime.AddMinutes(-timeSpanUnits), consolidated.Time);
}
}
refDateTime = refDateTime.AddMinutes(1);
}
Figure 4: Sample snippet from the Lean project. Formatting has been modiﬁed.
var words = (from word in phrase.Split(’ ’)
where word.Length > 0 select word.ToLower()).ToArray();
Figure 5: Sample snippet from the BotBuilder project. Formatting has been modiﬁed.
_hasHandle = _mutex.WaitOne( timeOut < 0 ? Timeout.Infinite
: timeOut,
exitContext: false);
Figure 6: Sample snippet from the Chocolatey project. Formatting has been modiﬁed.
public static T retry<T>(int numberOfTries, Func<T> function,
int waitDurationMilliseconds = 100,
int increaseRetryByMilliseconds = 0) {
if (function == null) return default(T);
if (numberOfTries == 0)
throw new ApplicationException("You must specify a number"
+ " of retries greater than zero.");
var returnValue = default(T);
var debugging = log_is_in_debug_mode();
var logLocation = ChocolateyLoggers.Normal;
for (int i = 1; i <= numberOfTries ; i++)
{
Figure 7: Sample snippet from the Chocolatey project. Formatting has been modiﬁed and the snippet
has been abbreviated.
12
Published as a conference paper at ICLR 2019
while ( count >= startIndex )
{
c = s[count];
if ( c != ’ ’ && c != ’n’ ) break;
count--;
}
Figure 8: Samples snippet in the CommonMark.NET project. Formatting has been modiﬁed.
private string GetResourceForTimeSpan(TimeUnit unit, int count)
{
var resourceKey = ResourceKeys.TimeSpanHumanize.GetResourceKey(unit, count);
return count == 1 ? Format(resourceKey) : Format(resourceKey, count);
}
Figure 9: Sample snippet from the Humanizer project. Formatting has been modiﬁed.
var indexOfEquals = segment.IndexOf(’=’) ;
if ( indexOfEquals == -1 ) {
var decoded = UrlDecode(segment, encoding);
return new KeyValuePair<string, string>(decoded, decoded);
}
Figure 10: Samples snippet from the Nancy project. Formatting has been modiﬁed.
private bool ResolveWritableOverride(bool writable)
{
if (!Writable && writable)
throw new StorageInvalidOperationException("Cannot open writable storage"
+ " in readonly storage.");
bool openWritable = Writable;
if ( openWritable && !writable )
openWritable = writable;
return openWritable;
}
Figure 11: Sample snippet from the OpenLiveWriter project. Formatting has been modiﬁed.
char c = html[j];
if ( c == ’;’ || (!(c >= ’a’ && c <= ’z’) && !(c >= ’A’ && c <= ’Z’) && !(c >= ’0’ && c <= ’9’)) )
{
break;
}
Figure 12: Sample snippet from the OpenLiveWriter project. Formatting has been modiﬁed.
13
Published as a conference paper at ICLR 2019
string entityRef = html.Substring(i + 1, j - (i + 1)) ;
Figure 13: Sample snippet from the OpenLiveWriter project. Formatting has been modiﬁed.
B S AMPLE GENERATIONS
On the following pages, we list some sample snippets from the test set for our ExprGen task,
together with suggestions produced by different models. The highlighted expressions are the ground
truth expression that should be generated.
14
Published as a conference paper at ICLR 2019
Sample 1
if (context.Context == _MARKUP_CONTEXT_TYPE.CONTEXT_TYPE_Text &&
!String.IsNullOrEmpty(text)) {
idx = originalText.IndexOf(text) ;
if (idx == 0) {
// Drop this portion from the expected string
originalText = originalText.Substring(text.Length);
// Update the current pointer
beginDamagePointer.MoveToPointer(currentRange.End);
}
else if (idx > 0 &&
originalText.Substring(0, idx)
.Replace("\r\n", string.Empty).Length == 0)
{
// Drop this portion from the expected string
originalText = originalText.Substring(text.Length + idx);
// Update the current pointer
beginDamagePointer.MoveToPointer(currentRange.End);
}
else
{
return false;
}
}
Sample snippet from OpenLiveWriter. The following suggestions were made:
Seq→Seq:
UNK_TOKEN[i] (0.6%)
input[inputOffset + 1] (0.3%)
UNK_TOKEN & UNK_NUM_LITERAL (0.3%)
Seq→NAG:
MarshalUrlSupported.IndexOf(UNK_CHAR_LITERAL) (0.9%)
IsEditFieldSelected.IndexOf(UNK_CHAR_LITERAL) (0.8%)
marshalUrlSupported.IndexOf(UNK_CHAR_LITERAL) (0.7%)
G→S eq:
UNK_TOKEN.IndexOf(UNK_CHAR_LITERAL) (21.6%)
UNK_TOKEN.LastIndexOf(UNK_CHAR_LITERAL) (14.9%)
UNK_TOKEN.GetHashCode() (8.1%)
G→T ree:
UNK_CHAR_LITERAL.IndexOf(UNK_CHAR_LITERAL) (8.1%)
UNK_CHAR_LITERAL.IndexOf(originalText) (8.1%)
originalText.IndexOf(UNK_CHAR_LITERAL) (8.1%)
G→ASN :
originalText.GetHashCode() (37.8%)
originalText.IndexOf(UNK_CHAR_LITERAL) (14.8%)
originalText.LastIndexOf(UNK_CHAR_LITERAL) (6.2%)
G→S yn:
text.IndexOf(UNK_CHAR_LITERAL) (20.9%)
text.LastIndexOf(UNK_CHAR_LITERAL) (12.4%)
originalText.IndexOf(UNK_CHAR_LITERAL) (11.6%)
G→NAG :
originalText.IndexOf(UNK_CHAR_LITERAL) (32.8%)
originalText.LastIndexOf(UNK_CHAR_LITERAL) (12.4%)
originalText.IndexOf(text) (8.7%)
15
Published as a conference paper at ICLR 2019
Sample 2
caretPos--;
if (caretPos < 0) {
caretPos = 0;
}
int len = inputString.Length;
if (caretPos >= len) {
caretPos = len - 1 ;
}
Sample snippet from acat. The following suggestions were made:
Seq→Seq:
UNK_TOKEN+1 (2.1%)
UNK_TOKEN+UNK_TOKEN] (1.8%)
UNK_TOKEN.IndexOf(UNK_CHAR_LITERAL) (1.3%)
Seq→NAG:
wordToReplace - 1 (3.2%)
insertOrReplaceOffset - 1 (2.9%)
inputString - 1 (1.9%)
G→S eq:
len + 1 (35.6%)
len - 1 (11.3%)
len >> UNK_NUM_LITERAL (3.5%)
G→T ree:
len + len (24.9%)
len - len (10.7%)
1 + len (3.7%)
G→ASN :
len + 1 (22.8%)
len - 1 (10.8%)
len + len (10.3%)
G→S yn:
len + 1 (13.7%)
len - 1 (11.5%)
len - len (11.0%)
G→NAG :
len++ (33.6%)
len-1 (21.9%)
len+1 (14.6%)
16
Published as a conference paper at ICLR 2019
Sample 3
public static String URItoPath(String uri) {
if (System.Text.RegularExpressions
.Regex.IsMatch(uri, "^file:\\\\[a-z,A-Z]:")) {
return uri.Substring(6);
}
if ( uri.StartsWith(@"file:") ) {
return uri.Substring(5);
}
return uri;
}
Sample snippet from acat. The following suggestions were made:
Seq→Seq:
!UNK_TOKEN (11.1%)
UNK_TOKEN == 0 (3.6%)
UNK_TOKEN != 0 (3.4%)
Seq→NAG:
!uri (7.6%)
!MyVideos (4.7%)
!MyDocuments (4.7%)
G→S eq:
action == UNK_STRING_LITERAL (22.6%)
label == UNK_STRING_LITERAL (14.8%)
file.Contains(UNK_STRING_LITERAL) (4.6%)
G→T ree:
uri == uri (7.4%)
uri.StartsWith(uri) (5.5%)
uri.Contains(uri) (4.3%)
G→ASN :
uri == UNK_STRING_LITERAL (11.7%)
uri.Contains(UNK_STRING_LITERAL) (11.7%)
uri.StartsWith(UNK_STRING_LITERAL) (8.3%)
G→S yn:
uri == UNK_STRING_LITERAL (26.4%)
uri == "" (8.5%)
uri.StartsWith(UNK_STRING_LITERAL) (6.7%)
G→NAG :
uri.Contains(UNK_STRING_LITERAL) (32.4%)
uri.StartsWith(UNK_STRING_LITERAL) (29.2%)
uri.HasValue() (7.7%)
17
Published as a conference paper at ICLR 2019
Sample 4
startPos = index + 1;
int count = endPos - startPos + 1;
word = (count > 0) ? input.Substring(startPos, count) : String.Empty;
Sample snippet from acat. The following suggestions were made:
Seq→Seq:
UNK_TOKEN.Trim() (3.4%)
UNK_TOKEN.Replace(UNK_STRING_LITERAL, UNK_STRING_LITERAL) (2.1%)
UNK_TOKEN.Replace(‘UNK_CHAR‘, ‘UNK_CHAR‘) (3.4%)
Seq→NAG:
input[index] (1.4%)
startPos[input] (0.9%)
input[count] (0.8%)
G→S eq:
val.Trim() (6.6%)
input.Trim() (6.5%)
input.Substring(UNK_NUM_LITERAL) (4.0%)
G→T ree:
UNK_STRING_LITERAL + UNK_STRING_LITERAL (8.4%)
UNK_STRING_LITERAL + startPos (7.8%)
startPos + UNK_STRING_LITERAL (7.8%)
G→ASN :
input.Trim() (15.6%)
input.Substring(0) (6.4%)
input.Replace(UNK_STRING_LITERAL, UNK_STRING_LITERAL) (2.8%)
G→S yn:
input.Trim() (7.8%)
input.ToLower() (6.4%)
input + UNK_STRING_LITERAL (5.6%)
G→NAG :
input+StartPos (11.8%)
input+count (9.5%)
input.Substring(startPos, endPos - count) (6.3%)
18
Published as a conference paper at ICLR 2019
Sample 5
protected virtual void CrawlSite() {
while ( !_crawlComplete )
{
RunPreWorkChecks();
if (_scheduler.Count > 0) {
_threadManager.DoWork(
() => ProcessPage(_scheduler.GetNext()));
}
else if (!_threadManager.HasRunningThreads()) {
_crawlComplete = true;
} else {
_logger.DebugFormat("Waiting for links to be scheduled...");
Thread.Sleep(2500);
}
}
}
Sample snippet from Abot. The following suggestions were made:
Seq→Seq:
!UNK_TOKEN (9.4%)
UNK_TOKEN > 0 (2.6%)
UNK_TOKEN != value (1.3%)
Seq→NAG:
!_maxPagesToCrawlLimitReachedOrScheduled (26.2%)
!_crawlCancellationReported (26.0%)
!_crawlStopReported (21.8%)
G→S eq:
!UNK_TOKEN (54.9%)
!done (18.8%)
!throwOnError (3.3%)
G→T ree:
!_crawlCancellationReported (23.6%)
!_crawlStopReported (23.3%)
!_maxPagesToCrawlLimitReachedOrScheduled (18.9%)
G→ASN :
!_crawlStopReported (26.6%)
!_crawlCancellationReported (26.5%)
!_maxPagesToCrawlLimitReachedOrScheduled (25.8%)
G→S yn:
!_crawlStopReported (19.6%)
!_maxPagesToCrawlLimitReachedOrScheduled (19.0%)
!_crawlCancellationReported (15.7%)
G→NAG :
!_crawlStopReported (38.4%)
!_crawlCancellationReported (31.8%)
!_maxPagesToCrawlLimitReachedOrScheduled (27.0%)
19
Published as a conference paper at ICLR 2019
Sample 6
char character = originalName[i];
if ( character == ’<’ ) {
++startTagCount;
builder.Append(’‘’);
} else if (startTagCount > 0) {
if (character == ’>’) {
--startTagCount;
}
Sample snippet from StyleCop. The following suggestions were made:
Seq→Seq:
x == UNK_CHAR_LITERAL (5.9%)
UNK_TOKEN == 0 (3.3%)
UNK_TOKEN > 0 (2.7%)
Seq→NAG:
!i == 0 (5.1%)
character < 0 (2.7%)
character (2.2%)
G→S eq:
character == UNK_CHAR_LITERAL (70.8%)
character == UNK_CHAR_LITERAL || character == UNK_CHAR_LITERAL (5.8%)
character != UNK_CHAR_LITERAL (3.1%)
G→T ree:
character == character (9.9%)
UNK_CHAR_LITERAL == character (8.2%)
character == UNK_CHAR_LITERAL (8.2%)
G→ASN :
character == UNK_CHAR_LITERAL (43.4%)
character || character (3.3%)
character == UNK_CHAR_LITERAL == UNK_CHAR_LITERAL (3.0%)
G→S yn:
character == UNK_CHAR_LITERAL (39.6%)
character || character == UNK_STRING_LITERAL (5.2%)
character == UNK_STRING_LITERAL (2.8%)
G→NAG :
character == UNK_CHAR_LITERAL (75.5%)
character == ’’ (2.6%)
character != ’UNK_CHAR (2.5%)
20
Published as a conference paper at ICLR 2019
Sample 7
public void AllowAccess(string path)
{
if (path == null) throw new ArgumentNullException("path");
if ( !path.StartsWith(" /") )
throw new ArgumentException(
string.Format(
"The path \"{0}\" is not application relative."
+ " It must start with \"~/\".",
path),
"path");
paths.Add(path);
}
Sample snippet from cassette. The following suggestions were made:
Seq→Seq:
UNK_TOKEN < 0 (14.6%)
!UNK_TOKEN (7.5%)
UNK_TOKEN == 0 (3.3%)
Seq→NAG:
path == UNK_STRING_LITERAL (18.1%)
path <= 0 (5.6%)
path == "" (4.8%)
G→S eq:
!UNK_TOKEN (48.0%)
!discardNulls (6.3%)
!first (2.7%)
G→T ree:
!path (67.4%)
path && path (8.4%)
!!path (5.5%)
G→ASN :
!path (91.5%)
!path && !path (0.9%)
!path.Contains(UNK_STRING_LITERAL) (0.7%)
G→S yn:
!path (89.6%)
!path && !path (1.5%)
!path.Contains(UNK_STRING_LITERAL) (0.5%)
G→NAG :
!path (42.9%)
!path.StartsWith(UNK_STRING_LITERAL) (23.8%)
!path.Contains(UNK_STRING_LITERAL) (5.9%)
21
Published as a conference paper at ICLR 2019
Sample 8
int methodParamCount = 0;
IEnumerable<IParameterTypeInformation> moduleParameters =
Enumerable<IParameterTypeInformation>.Empty;
if (paramCount > 0) {
IParameterTypeInformation[] moduleParameterArr =
this.GetModuleParameterTypeInformations(Dummy.Signature, paramCount);
methodParamCount = moduleParameterArr.Length;
if (methodParamCount > 0)
moduleParameters = IteratorHelper.GetReadonly(moduleParameterArr);
}
IEnumerable<IParameterTypeInformation> moduleVarargsParameters =
Enumerable<IParameterTypeInformation>.Empty;
if ( paramCount > methodParamCount ) {
IParameterTypeInformation[] moduleParameterArr =
this.GetModuleParameterTypeInformations(
Dummy.Signature, paramCount - methodParamCount);
if (moduleParameterArr.Length > 0)
moduleVarargsParameters = IteratorHelper.GetReadonly(moduleParameterArr);
}
Sample snippet from Afterthought. The following suggestions were made:
Seq→Seq:
!UNK_TOKEN (10.9%)
UNK_TOKEN == UNK_TOKEN (4.6%)
UNK_TOKEN == UNK_STRING_LITERAL (3.3%)
Seq→NAG:
dummyPinned != 0 (2.2%)
paramCount != 0 (2.1%)
dummyPinned == 0 (1.5%)
G→S eq:
newValue > 0 (9.7%)
zeroes > 0 (9.0%)
paramCount > 0 (6.0%)
G→T ree:
methodParamCount == methodParamCount (3.4%)
0 == methodParamCount (2.8%)
methodParamCount == paramCount (2.8%)
G→ASN :
paramCount == 0 (12.7%)
paramCount < 0 (11.5%)
paramCount > 0 (8.0%)
G→S yn:
methodParamCount > 0 (10.9%)
paramCount > 0 (7.9%)
methodParamCount != 0 (5.6%)
G→NAG :
paramCount > methodParamCount (34.4%)
paramCount == methodParamCount (11.4%)
paramCount < methodParamCount (10.0%)
22
Published as a conference paper at ICLR 2019
Sample 9
public CodeLocation(int index, int endIndex, int indexOnLine,
int endIndexOnLine, int lineNumber, int endLineNumber)
{
Param.RequireGreaterThanOrEqualToZero(index, "index");
Param.RequireGreaterThanOrEqualTo(endIndex, index, "endIndex");
Param.RequireGreaterThanOrEqualToZero(indexOnLine, "indexOnLine");
Param.RequireGreaterThanOrEqualToZero(endIndexOnLine, "endIndexOnLine");
Param.RequireGreaterThanZero(lineNumber, "lineNumber");
Param.RequireGreaterThanOrEqualTo(endLineNumber, lineNumber, "endLineNumber");
// If the entire segment is on the same line,
// make sure the end index is greater or equal to the start index.
if ( lineNumber == endLineNumber ) {
Debug.Assert(endIndexOnLine >= indexOnLine,
"The end index must be greater than the start index,"
+ " since they are both on the same line.");
}
this.startPoint = new CodePoint(index, indexOnLine, lineNumber);
this.endPoint = new CodePoint(endIndex, endIndexOnLine, endLineNumber);
}
Sample snippet from StyleCop. The following suggestions were made:
Seq→Seq:
!UNK_TOKEN (14.0%)
UNK_TOKEN == 0 (4.4%)
UNK_TOKEN > 0 (3.5%)
Seq→NAG:
endIndex < 0 (3.8%)
endIndex > 0 (3.4%)
endIndex == 0 (2.2%)
G→S eq:
lineNumber < 0 (9.4%)
lineNumber == 0 (7.4%)
lineNumber <= 0 (5.1%)
G→T ree:
lineNumber == lineNumber (3.4%)
0 == lineNumber (2.5%)
lineNumber > lineNumber (2.5%)
G→ASN :
endLineNumber == 0 (9.6%)
endLineNumber < 0 (7.9%)
endLineNumber > 0 (6.1%)
G→S yn:
lineNumber > 0 (11.3%)
lineNumber == 0 (7.3%)
lineNumber != 0 (6.7%)
G→NAG :
lineNumber > endLineNumber (20.7%)
lineNumber < endLineNumber (16.5%)
lineNumber == endLineNumber (16.2%)
23
Published as a conference paper at ICLR 2019
Sample 10
public static Bitmap RotateImage(Image img, float angleDegrees,
bool upsize, bool clip) {
// Test for zero rotation and return a clone of the input image
if (angleDegrees == 0f) return (Bitmap)img.Clone();
// Set up old and new image dimensions, assuming upsizing not wanted
// and clipping OK
int oldWidth = img.Width; int oldHeight = img.Height;
int newWidth = oldWidth; int newHeight = oldHeight;
float scaleFactor = 1f;
// If upsizing wanted or clipping not OK calculate the size of the
// resulting bitmap
if ( upsize || !clip ) {
double angleRadians = angleDegrees * Math.PI / 180d;
double cos = Math.Abs(Math.Cos(angleRadians));
double sin = Math.Abs(Math.Sin(angleRadians));
newWidth = (int)Math.Round((oldWidth * cos) + (oldHeight * sin));
newHeight = (int)Math.Round((oldWidth * sin) + (oldHeight * cos));
}
// If upsizing not wanted and clipping not OK need a scaling factor
if (!upsize && !clip) {
scaleFactor = Math.Min((float)oldWidth / newWidth,
(float)oldHeight / newHeight);
newWidth = oldWidth; newHeight = oldHeight;
}
Sample snippet from ShareX. The following suggestions were made:
Seq→Seq:
UNK_TOKEN > 0 (8.3%)
!UNK_TOKEN (4.4%)
UNK_TOKEN == 0 (2.6%)
Seq→NAG:
newHeight > 0 (5.1%)
clip > 0 (3.2%)
oldWidth > 0 (2.9%)
G→S eq:
UNK_TOKEN && UNK_TOKEN (15.0%)
UNK_TOKEN || UNK_TOKEN (13.6%)
trustedForDelegation && !appOnly (12.1%)
G→T ree:
upsize && upsize (21.5%)
upsize && clip (10.9%)
clip && upsize (10.9%)
G→ASN :
upsize && clip (13.9%)
upsize && !clip (9.8%)
clip && clip (9.3%)
G→S yn:
upsize && !upsize (6.9%)
clip && !upsize (6.3%)
upsize || upsize (5.7%)
G→NAG :
upsize || clip (19.1%)
upsize && clip (18.8%)
upsize && ! clip (12.2%)
24

A Critical Review of Recurrent Neural Networks
for Sequence Learning
Zachary C. Lipton
University of California, San Diego
zlipton@cs.ucsd.edu
June 5th, 2015
Abstract
Countless learning tasks require awareness of time. Image captioning,
speech synthesis, and video game playing all require that a model generate
sequences of outputs. In other domains, such as time series prediction,
video analysis, and music information retrieval, a model must learn from
sequences of inputs. Signiﬁcantly more interactive tasks, such as natu-
ral language translation, engaging in dialogue, and robotic control, often
demand both.
Recurrent neural networks (RNNs) are a powerful family of connec-
tionist models that capture time dynamics via cycles in the graph. Unlike
feedforward neural networks, recurrent networks can process examples one
at a time, retaining a state, or memory, that reﬂects an arbitrarily long
context window. While these networks have long been diﬃcult to train
and often contain millions of parameters, recent advances in network archi-
tectures, optimization techniques, and parallel computation have enabled
large-scale learning with recurrent nets.
Over the past few years, systems based on state of the art long short-
term memory (LSTM) and bidirectional recurrent neural network (BRNN)
architectures have demonstrated record-setting performance on tasks as
varied as image captioning, language translation, and handwriting recog-
nition. In this review of the literature we synthesize the body of research
that over the past three decades has yielded and reduced to practice these
powerful models. When appropriate, we reconcile conﬂicting notation and
nomenclature. Our goal is to provide a mostly self-contained explication
of state of the art systems, together with a historical perspective and
ample references to the primary research.
1 Introduction
Recurrent neural networks (RNNs) are a superset of feedforward neural net-
works, augmented with the ability to pass information across time steps. They
1
arXiv:1506.00019v1  [cs.LG]  29 May 2015
are a rich family of models capable of nearly arbitrary computation. A well-
known result by Siegelman and Sontag from 1991 demonstrated that a ﬁnite
sized recurrent neural network with sigmoidal activation functions can simulate
a universal Turing machine [54]. In practice, the ability to model temporal
dependencies makes recurrent neural networks especially suited to tasks where
input and/or output consist of sequences of points that are not independent.
1.1 Comparison to Prior Work
The literature on recurrent neural networks can seem impenetrable to the unini-
tiated. Shorter papers assume familiarity with a large body of background lit-
erature. Diagrams are frequently underspeciﬁed, failing to indicate which edges
span time steps and which don’t. Worse, jargon abounds while notation is fre-
quently inconsistent across papers or overloaded within papers. Readers are
frequently in the unenviable position of having to synthesize conﬂicting infor-
mation across many papers in order to understand but one. For example, in
many papers subscripts index both nodes and time steps. In others, h simul-
taneously stands for link functions and a layer of hidden nodes. The variable t
simultaneously stands for both time indices and targets, sometimes in the same
equation. Many terriﬁc breakthrough papers have appeared recently, but clear
reviews of recurrent neural network literature are rare.
Among the most useful resources are Alex Graves’ 2012 book on supervised
sequence labelling with recurrent neural networks [24] and Felix Gers’ doctoral
thesis [19]. More recently, [15] covers recurrent neural nets for language mod-
eling. Other resources focus on a speciﬁc technical aspect such as [50], which
surveys gradient calculations in recurrent neural networks. In this review pa-
per, we aim to provide a readable, intuitive, and consistently notated review on
recurrent neural networks for sequence learning. We emphasize models, algo-
rithms, and results, but focus equally on distilling the intuitions that have guided
this largely heuristic and empirical ﬁeld. In addition to concrete modeling de-
tails, we oﬀer qualitative arguments, a historical perspective, and comparisons
to alternative methodology where appropriate.
1.2 Why Recurrent Nets?
In this section, we address the fundamental reasons why recurrent neural net-
works warrant serious study for modeling sequential input and output. To be
clear, we are motivated by a desire to achieve empirical results. This warrants
clariﬁcation because recurrent nets have roots in both cognitive modeling and
supervised machine learning, and owing to this diﬀerence of perspectives, many
of these papers have diﬀerent aims and priorities. In the foundational papers,
generally published in cognitive science and computational neuroscience jour-
nals ([33], [34], [18]), biologically plausible mechanisms are emphasized. In other
papers ([53], [57], [35]), biological inspiration is downplayed in favor of achiev-
ing empirical results on important tasks and datasets. Given the empirical aim,
2
we now address three signiﬁcant questions one might reasonably want answered
before reading further.
Why Explicitly Model Time?
In light of the practical success and economic value of time-agnostic models,
this is a fair question. Support vector machines, logistic regression, and feedfor-
ward networks have proved immensely useful without explicitly modeling time.
Arguably, it is precisely the assumption of independence that has led to much
recent progress in machine learning. Further, many models implicitly capture
time by concatenating each input with some number of its immediate prede-
cessors and successors, presenting the machine learning model with a sliding
window of context about each point of interest. This approach has been used
with deep belief nets for speech modeling in [43].
Unfortunately, despite the usefulness of the independence assumption, it
precludes modeling long-range time-dependencies. For example, a model trained
using a ﬁnite-length context window of length 5 could never be trained to answer
the simple question, “what was the data point seen 10 time steps ago?” For
a practical application such as call center automation, such a limited system
might learn to route calls, but could never participate in an extended dialogue.
Since the earliest conception of artiﬁcial intelligence, we have sought to build
systems that interact with humans in time. In Alan Turing’s groundbreaking
paper Computing Machinery and Intelligence, he proposes an “Imitation Game”
which judges a machine’s intelligence by its ability to convincingly engage in
dialogue [65]. Besides dialogue systems, modern interactive systems of economic
importance include self-driving cars and robotic surgery, among others. Ignoring
an explicit model of time, it seems unlikely that any combination of classiﬁers
or regressors can be cobbled together to provide this functionality.
Why Neural Networks and Not Markov Models?
Recurrent neural networks are not the ﬁrst models to capture time dependen-
cies. Markov chains, which model transitions between observed sequences of
states ( s(1), s(2), ... , s(T)), were ﬁrst described by mathematician Andrey
Markov in 1906. Hidden Markov models (HMMs), which model observed data
(o(1),o(2),...,o (T)) as probabilistically dependent upon unobserved states, were
described in the 1950s and have been widely studied since the 1960s. However,
traditional Markov model approaches are limited because their states must be
drawn from a modestly sized discrete state space sj ∈S. The Viterbi algorithm,
which is used to perform eﬃcient inference on hidden Markov models, scales in
time O(|S|2). Further, the transition table capturing the probability of moving
between any two adjacent states is of size |S|2. Thus, standard operations are
infeasible with an HMM when the set of possible hidden states is larger than
roughly 106 states. Further, each hidden state s(t) can depend only on the pre-
vious state s(t−1). While it is possible to extend any Markov model to account
for a larger context window by creating a new state-space equal to the cross
3
product of the possible states at each time in the window, this procedure grows
the state space exponentially with the size of the window, rendering Markov
models computationally impractical for modeling long-range dependencies.
Given the limitations of Markov models, we ought to explain why it is sen-
sible that connectionist models, i.e., artiﬁcial neural networks, should fare bet-
ter. First, recurrent neural networks can capture long-range time dependencies,
overcoming the chief limitation of Markov models. This point requires a careful
explanation. As in Markov models, any state in a traditional RNN depends
only on the current input as well as the state of the network at the previous
time step 1. However, the hidden state at any time step can contain information
from an arbitrarily long context window. This is possible because the number
of distinct states that can be represented in a hidden layer of nodes grows ex-
ponentially with the number of nodes in the layer. Even if each node took only
binary values, the network could represent 2N states where N is the number of
nodes in the hidden layer. Given real-valued outputs, even assuming the lim-
ited precision of 64 bit numbers, a single hidden layer of nodes can represent
264N
distinct states. While the potential expressive power grows exponentially
with the number of nodes in the hidden representation, the complexity of both
inference and training grows only quadratically.
Second, it is generally desirable to extend neural networks to tackle any
supervised learning problem because they are powerful learning models that
achieve state of the art results in a wide range of supervised learning tasks.
Over the past several years, storage has become more aﬀordable, datasets have
grown far larger, and the ﬁeld of parallel computing has advanced considerably.
In the setting of such large high-dimensional datasets, simple linear models
under-ﬁt and often underutilize computing resources. Deep learning methods, in
particular those based on deep belief networks (DNNs), which are greedily built
by stacking restricted Boltzmann machines, and convolutional neural networks,
which exploit the local dependency of visual information, have demonstrated
record-setting results on many important applications. Neural networks are
especially well-suited for machine perception tasks, where the raw underlying
features are not individually informative. This success is attributed to their
ability to learn hierarchical representations, unlike traditional algorithms, which
rely upon hand-engineered features. However, despite their power, feedforward
neural nets have limitations. Most notably, they rely on the assumption of
independence among the data points. Additionally, these networks generally
rely on input consisting of ﬁxed length vectors. Thus it is sensible to extend
these powerful learning tools to model data with temporal structure, especially
in the many domains where neural nets are already the state of the art.
1 Bidirectional recurrent neural networks (BRNNs) [53] extend RNNs to model dependence
on both past states and future states. Traditional RNNs only model the dependence of each
event on the past. This extension is especially useful for sequence to sequence learning with
ﬁxed length examples.
4
Are RNNs Too Expressive?
As described earlier, ﬁnite-sized RNNs with sigmoidal activations are Turing
complete. The capability of RNNs to run arbitrary computation clearly demon-
strates their expressive power, but one could argue that the C programming
language is equally capable of expressing arbitrary programs. And yet there are
no papers claiming that the invention of C represents a panacea for machine
learning. Out of the box, C oﬀers no simple way of eﬃciently exploring the
space of programs. There is no straightforward way to calculate the gradient
of an arbitrary C program to minimize a chosen loss function. Further, the
biggest problem with treating the set of programs expressible in C as a family
of machine learning models is that this set is far too large. Given any ﬁnite sized
dataset, there exist countless programs which can overﬁt the data, generating
desired output but failing to generalize to test data.
Why then should RNN’s not suﬀer from similar problems? First, given any
ﬁxed architecture (set of nodes, edges, and activation functions), the recurrent
neural networks described in this paper are fully diﬀerentiable end to end. The
derivative of the loss function can always be calculated with respect to each of
the parameters (weights) in the model. Second, while the Turing-completeness
of ﬁnite-length RNNs is an impressive property, given any ﬁxed-size RNN and
a speciﬁc architecture, it is not actually possible to generate any arbitrary pro-
gram. Further, unlike an arbitrary program composed in C, a recurrent neu-
ral network can be regularized via standard techniques such as weight decay,
dropout, and limiting the degrees of freedom.
2 Background
Here we introduce formal notation and provide a brief background on neural
networks.
2.1 Time
To be clear, RNNs are not limited to sequences which index time. They have
been used successfully on non-temporal sequence data, including genetic data
[3]. However, computation proceeds in time, and many important applications
have an explicit or implicit temporal aspect. While we refer to time throughout
this paper, the methods described here are applicable to wider family of tasks.
In this paper, by time, we refer to data points x(t) that arrive and desired
outputs y(t) that are generated in a discrete sequence of time steps indexed by
t. We use superscripts with parentheses and not subscripts to obviate confu-
sion between time steps and neurons. Our sequences may be of ﬁnite length
or countably inﬁnite. When they are ﬁnite, we call the maximum time in-
dex of the sequence T. Thus a sequence of consecutive inputs can be notated
(x(1),x(2),..., x(T)) and outputs can be notated ( y(1),y(2),..., y(T)) These time
steps may be equally spaced samples from a continuous real-world process. Ex-
amples would include the still images that comprise the frames of videos or the
5
discrete amplitudes sampled at ﬁxed intervals to comprise audio recordings. The
time steps may also be ordinal, with no exact correspondence to durations. In
fact, these techniques can be extended to domains including genetic sequences,
where the sequence has a deﬁned order but no real correspondence to time. This
is the case with natural language. In the word sequence “John Coltrane plays
the saxophone”, x(1) = John, x(2) = Coltrane, etc.
2.2 Neural Networks
Neural networks are biologically inspired models of computation. Generally,
a neural network consists of a set of artiﬁcial neurons, commonly referred to
as nodes or units, and a set of directed edges between them, which intuitively
represent the synapses in a biological neural network. Associated with each
neuron j is an activation function lj, which is sometimes called a link function.
We use the notation “lj” and not “hj” (unlike some other papers) to distinguish
the activation functionlj from the values of the hidden nodes in a network, which
is commonly notated h in the RNN literature.
Associated with each edge from node j′ to j is a weight wjj′ . Following
the convention adopted in several foundational recurrent net papers ([32], [19],
[21], [63]), we index neurons with j and j′, and by wjj′ , we denote the weight
corresponding to the directed edge from node j′ to node j. It is important to
note that in many papers, textbooks, and lecture notes, the indices are ﬂipped
and wj′j ̸= wjj′ denotes the weight on the directed edge from the node j′to the
node j as in [17] and on Wikipedia [69].
The value vj of each neuronjis calculated by applying its activation function
to a weighted sum of its inputs (Figure 1):
vj = lj

∑
j′
wjj′ ·vj′

.
For convenience, we term the weighted sum inside the parenthesis the incom-
ing activation and notate it as aj. We represent this entire process in ﬁgures
by depicting neurons as circles and edges as arrows connecting them. When
appropriate, we mark the exact activation function with a symbol, e.g., σ for
sigmoid.
Common choices for the activation function include the sigmoid σ(z) =
1/(1 + e−z) and the tanh function φ(z) = ( ez −e−z)/(ez + e−z) which has
become common in feedforward neural nets and was applied to recurrent nets
in [63]. Another activation which has become the state of the art in deep learning
research is the rectiﬁed linear unit (ReLU) lj(z) = max(0,z). These units have
been demonstrated to improve the performance of many deep neural networks
([43], [47], [73]) on tasks as varied as speech processing and object recognition,
and have been used in recurrent neural networks by [7].
The activation function at the output nodes depends upon the task. For
multi-class classiﬁcation, we apply a softmax nonlinearity to the layer. The
6
Figure 1: An artiﬁcial neuron computes a nonlinear function of a weighted sum
of its inputs.
softmax function calculates
ˆyk = eak
∑K
k′=1 eak′
where K is the total number of outputs. The denominator is normalization
consisting of a sum of exponentials over all output nodes, ensuring that the
output sums to 1. For multilabel classiﬁcation the activation function is simply
a point-wise sigmoid. For regression we may have linear output. Due to the
overwhelming number of current applications involving multi-class classiﬁcation,
especially for recurrent nets, in this paper, unless otherwise speciﬁed, we assume
that softmax is applied at the output.
2.3 Feedforward Neural Networks
With a neural model of computation, one must determine the order in which
computation should proceed. Should nodes be sampled one at a time and up-
dated, or should the value of all nodes be calculated at once and then all updates
applied simultaneously? Feedforward neural networks (Figure 2) are a restricted
class of neural networks which deal with this problem by forbidding cycles in
the graph. Thus all nodes can be arranged into layers. The outputs in each
layer can be calculated given the outputs from the lower layers.
The input x to a feedforward network is presented by setting the values of
the lowest layer. Each higher layer is then successively computed until out-
put is generated at the topmost layer ˆy. These networks are frequently used
for supervised learning tasks such as classiﬁcation and regression. Learning is
accomplished by iteratively updating each of the weights to minimize a loss
function, L(ˆy,y), which penalizes the distance between the output ˆy and the
target y. Backpropagation, an algorithm introduced to neural networks in [52],
7
Figure 2: A feedforward neural network. An example is presented to the network
by setting the values of the blue nodes. Each layer is calculated successively as
a function of the prior layers until output is produced at the topmost layer.
computes the gradient of the loss with respect to each parameter using the chain
rule. While the optimization surfaces for neural networks are highly non-convex,
and exact optimization is known to be an NP-Hard problem, a large body of
work on heuristic pre-training and optimization techniques has led to impres-
sive empirical success on many supervised learning tasks. Convolutional neural
networks, developed by Yann LeCun, [38] are a variant of feedforward neural
network that, since 2012, hold records in many computer vision tasks, such as
object detection [36].
Feedforward networks, however, are limited. After each example is pro-
cessed, the entire state of the network is lost. If each data point is independently
sampled, this presents no problem. But if data points are related in time, this
is unacceptable. Frames from video, snippets of audio, and words pulled from
sentences, represent settings where the independence assumption fails.
2.4 Training Neural Networks via Backpropagation
The most successful algorithm for training neural networks is backpropagation,
introduced to neural networks by Rumelhart et al. in 1985 [52]. Backpropa-
gation uses the chain rule to calculate the derivative of a loss function Lwith
respect to each parameter in the network. The weights are then adjusted by gra-
dient descent. Because the loss surface is non-convex, there is no assurance that
backpropagation will reach a global minima. However, in practice, networks
trained with backpropagation and gradient following techniques have been re-
markably successful. In practice, most networks are trained with stochastic
gradient descent (SGD) using mini-batches. Here, w.l.o.g. we discuss only the
8
case with batch size equal to 1. The stochastic gradient update equation is given
by
w ←w −η∇wFi
where η is the learning rate and ∇wFi is the gradient of the objective function
with respect to the parameters w as calculated on a single example ( xi,yi).
In practice, many variants of SGD are used to accelerate learning. Some
popular heuristics, such as AdaGrad [16], AdaDelta [72], and RMSprop [1],
adaptively tune the learning rate for each feature. AdaGrad, arguably the most
popular, adapts the learning rate by caching the sum of squared gradients with
respect to each parameter at each time step. The step size for each feature is
scaled to the inverse of this cache. This leads to fast convergence on convex error
surfaces, but because the cached sum is monotonically increasing, AdaGrad has
a monotonically decreasing learning rate. This may be undesirable on highly
non-convex loss surfaces. RMSprop modiﬁes AdaGrad by introducing a decay
factor on the cache, transforming the monotonic growing cache into a moving
average. Momentum methods are another common SGD variant used to train
neural networks. These methods add to each update a decaying sum of the pre-
vious updates. When the momentum parameter is well-tuned and the network
is initialized well, momentum methods can train deep nets and recurrent nets
competitively with more computationally expensive methods like the Hessian
Free optimizer [62].
To calculate the gradient in a feedforward neural network, backpropagation
proceeds as follows. First, an example is forward propagated through the net-
work to produce a value vj at each node and outputs ˆy at the topmost layer.
Then, a loss function L(ˆyk,yk) is assessed at each output node k. Subsequently,
for each output node k, we can calculate
δk = ∂L(ˆyk,yk)
∂ˆyk
·l′
k(ak).
Given these values δk, for each node in the level prior we can calculate
δj = l′(aj)
∑
k
δk ·wkj.
This calculation is performed successively for each lower layer to calculate δj for
every node j given the δ values for each node connected by an outgoing edge.
Each value δj represents the derivative ∂L/aj of the total loss function w.r.t.
that node’s incoming activation. Given the values vj calculated on the forward
pass, and the values δj calculated on the backward pass, the derivative of the
loss Lwith respect a given parameter wjj′ is given by
∂L
∂wjj′
= δjvj′ .
Large scale feedforward neural networks trained via backpropagation have set
many large-scale machine learning records, most notably on the computer vision
task of object detection ([39], [36]).
9
Several other methods have been explored for learning the weights in a neu-
ral network. Early work, including Hopﬁeld nets [33], learned via a Hebbian
principle but did not produce networks useful for discriminative tasks. A num-
ber of papers from the 1990s ([6], [27]) championed the idea of learning neural
networks with genetic algorithms with some even claiming that achieving suc-
cess on real-world problems by applying many small changes to a network’s
weights was impossible. Despite the subsequent success of backpropagation,
interest in genetic algorithms persists. Several recent papers explore genetic
algorithms for neural networks, especially as means of learning the architecture
of neural networks, a problem not addressed by backpropagation ([5], [28]). By
the architecture we mean the number of layers, the number of nodes in each, the
connectivity pattern among the layers, the choice of activation functions, etc.
One open question in neural network research is how to exploit sparsity in
training. In a neural network with sigmoidal or tanh activation functions, the
nodes in each layer never take value exactly 0. Thus, even if the inputs are
sparse, the nodes at each hidden layer are not. However, a rectiﬁed linear units
(ReLUs) introduce sparsity to hidden layers [22]. In this setting, a promising
path may be to store the sparsity pattern when computing each layer’s values
and use it to speed up computation of the next layer in the network. A growing
body of recent work ([12], [37], [55], [40]), thus read shows that given sparse
inputs to a linear model with any standard regularizer, sparsity can be fully ex-
ploited even if the gradient is not sparse (owing to regularization). Given sparse
hidden layers, these approaches can be extended to the layer-wise computations
in neural networks.
3 Recurrent Neural Networks
Recurrent neural networks are a strict superset of feedforward neural networks,
augmented by the inclusion of recurrent edges that span adjacent time steps,
introducing a notion of time to the model. While RNNs may not contain cycles
among the conventional edges, recurrent edges may form cycles, including self-
connections. At time t, nodes receiving input along recurrent edges receiveinput
activation from the current example x(t) and also from hidden nodes h(t−1) in
the network’s previous state. The output ˆy(t) is calculated given the hidden
state h(t) at that time step. Thus, input x(t−1) at time t−1 can inﬂuence the
output ˆy(t) at time t by way of these recurrent connections.
We can show in two equations, all calculations necessary for computation at
each time step on the forward pass in a simple recurrent neural network:
h(t) = σ(Whxx + Whhh(t−1) + bh)
ˆy(t) = softmax(Wyhh(t) + by)
Here Whx is the matrix of weights between the input and hidden layers and
Whh is the matrix of recurrent weights between the hidden layers at adjacent
10
time steps. The vectors bh and by are biases which allow each node to learn an
oﬀset.
Most of the models discussed in this paper consist of networks with recurrent
hidden layers. However, some proposed models, such as Jordan Networks, allow
for connections between the outputs in one state and the hidden layer in the
next. Others, such as Sutskever et al.’s model for sequence to sequence learning
[64], compute the output at each time step and pass a representation of this
information as the input at the following time step.
Figure 3: A simple recurrent network. At each time step t, activation is passed
along solid edges as in a feedforward network. Dashed edges connect the source
node j′at time t, i.e., j′(t) to the target node at the following time step j(t+1).
A simple recurrent network is depicted in Figure 3. The dynamics of this
network across time steps can be visualized by unfolding the network (Figure
4). Given this picture, the model can be interpreted not as cyclic, but rather
as a deep network with one layer per time step and shared weights across time
steps. It’s then clear that the unfolded network can be trained across many time
steps using backpropagation. This algorithm is called backpropagation through
time (BPTT), and was introduced in 1990 [68].
3.1 Past Approaches
Early foundational work on recurrent networks took in the 1980s. In 1982,
Hopﬁeld introduced a family of recurrent neural networks [33]. We’ll avoid
a prolonged discussion of these networks as they are not currently used for
sequence learning. Hopﬁeld’s nets have some pattern recognition capabilities
but oﬀer no clear method for supervised training. These networks are deﬁned
by the values of the weights between nodes and the link functions are simple
thresholding at 0. In these nets, a pattern is placed in the network by setting
11
Figure 4: Visualizing the network unfolded across time steps.
the values of the nodes. The network then runs for some time according to its
update rules, and eventually a pattern is read out. The networks are useful for
recovering a stored pattern from a corrupted version and are the forerunners of
Boltzmann machines and auto-encoders.
Figure 5: A recurrent neural network as proposed by Jordan (1986).
Jordan networks (Figure 5), introduced by Michael Jordan in 1986, present
an early architecture for supervised learning on sequences ([34]). A Jordan
network resembles a feedforward network with a single hidden layer, but is ex-
12
Figure 6: An Elman network as described in Finding Structure in Time (1990)
[18]. Hidden units are connected 1-to-1 to context units across time steps, which
in turn feed back into the corresponding hidden units.
tended with context units. 2 Outputs are fed into the context units, which then
feed in to the hidden nodes at the following time step. Additionally, context
units have self-connected edges. Intuitively, these self-connected edges gives Jor-
dan networks a means to send information across multiple time steps without
perturbing the output at each intermediary time step. In Jordan’s setup, the
network was used to plan actions and thus these context units allow the network
to remember actions taken (outputs) at the previous time steps. Several modern
architectures utilize some form of direct feedback from output nodes. [64] trans-
lates sentences between natural languages. When generating text sequences, the
word chosen at each time step is fed into the network at the following time step.
Elman networks, introduced in [18], simplify the structure in the Jordan
network. Associated with each unit in the hidden layer is a single context unit.
Each context unit j′takes as input the state of the corresponding hidden node
j at the previous time step, along an edge of unit weight wj′j = 1. Eﬀectively,
this entire setup is equivalent to a simple RNN in which each hidden node has
a single self-connected recurrent edge. Some of Elman’s ideas, including ﬁxed-
weight edges and the importance of self-connected recurrent edges in hidden
nodes survive in Hochreiter and Schmidhuber’s subsequent work on Long Short-
Term Memory [32].
In the paper, Elman trains the network using backpropagation and demon-
strates that the network can learn long-range time dependencies. The paper
features two sets of experiments. The ﬁrst extends the logic operation exclusive
OR (XOR) to the time domain by concatenating sequences of three tokens. For
2 In Jordan’s paper, he calls these “state units”. Elman calls a corresponding structure
“context units”. In this paper we simplify terminology by using only “context units”.
13
each three-token segment, e.g. “011”, the ﬁrst two tokens (“01”) are chosen
randomly and the third (“1”) is chosen by performing XOR on them. A com-
pletely random guess should achieve accuracy of 50%. A perfect system should
perform the same as random for the ﬁrst two tokens, but guess the third token
perfectly, achieving accuracy of 66 .6...%. Elman’s simple network does in fact
do get close to this maximum achievable score.
3.2 Training Recurrent Networks
Learning with recurrent neural networks has long been considered to be diﬃ-
cult. As with all neural networks, the optimization is NP-Hard. But learning
on recurrent networks can be especially hard due to the diﬃculty of learning
long-range dependencies as described by Bengio et al in 1994 [9] and expanded
upon in [31]. The well known problems of vanishing and exploding gradients
occur when propagating errors across many time steps. As a trivial example,
consider a network with a single input node, a single output node, and a single
recurrent hidden node (Figure 7). Now consider an input passed to the network
at time τ and an error calculated at time t, assuming input of 0 in the inter-
vening time steps. Owing to the weight tying across time steps (the recurrent
edge at hidden node j always has the same weight), the impact of the input
at time τ on the output at time t will either explode exponentially or rapidly
approach zero as t−τ grows large, depending on whether the weight |wjj|>1
or |wjj|< 1 and also upon the activation function in the hidden node (Figure
8). Given activation function lj = σ, the vanishing gradient problem is more
pressing, but with a rectiﬁed linear unit max(0,x), it’s easier to imagine the
exploding gradient, even with this trivial example. In [49], Pascanu et al. give
a thorough mathematical treatment of the vanishing and exploding gradient
problems, characterizing exact conditions under which these problems may oc-
cur. Given these conditions under which the gradient may vanish or explode,
they suggest an approach to training via a regularization term, which forces the
weights to values where the gradient neither vanishes nor explodes.
Truncated backpropagation through time (TBPTT) is one solution to this
problem for continuously running networks [70]. With TBPTT, some maximum
number of time steps is set along which error can be propagated. While TBPTT
with a small cutoﬀ can be used to alleviate the exploding gradient problem, it
requires that one sacriﬁce the ability to learn long-range dependencies.
The optimization problem represents a more fundamental obstacle that can-
not as easily be dealt with by modifying network architecture. It has been
known since at least 1993 that optimizing even a 3-layer neural network is NP-
Complete [11]. However, recent empirical and theoretical studies suggest that
the problem may not be as hard in practice as once thought. [14] shows that
while many critical points exist on the error surfaces of large neural networks,
the ratio of saddle points to true local minima increases exponentially with the
size of the network
Fast implementations and improved gradient following heuristics have ren-
dered RNN training feasible. For example, implementations of forward and
14
Figure 7: A simple recurrent net with one input unit, one output unit, and one
recurrent hidden unit.
Figure 8: A visualization of the vanishing gradient problem, using the architec-
ture depicted in Figure 7. If the weight along the purple edge is less than one,
the eﬀect of the input at the ﬁrst time step on the output at the ﬁnal time step
will rapidly diminish as a function of the size of the interval in between. An
illustration like this appears in [24]
15
backward propagation using GPUs, such as Theano ( [10]) and Torch ([13]),
have made it easy to implement fast training algorithms. In 1996, prior to
the introduction of the LSTM, attempts to train recurrent nets to bridge long
time gaps were shown to perform no better than random guessing [30]. How-
ever, successfully trained RNNs are now relatively common. Sutskever and
Martens reported success training recurrent neural networks with a Hessian-
Free, i.e., truncated Newton approach [45] and applied it to a network which
learns to generate text one character at a time in [63]. In the paper that de-
scribed the abundance of saddle points on the error surfaces of neural networks
([14]), the authors present a saddle-free version of Newton’s method. Unlike
Newton’s method, which is attracted to critical points, including saddle points,
this variant is specially designed to escape from them. Experimental results
include a demonstration of improved performance on recurrent networks. New-
ton’s method requires computing the Hessian, which is prohibitively expensive
for large networks, scaling quadratically with the number of parameters. While
their algorithm only approximates the Hessian, it is still computationally expen-
sive compared to SGD. Thus the authors describe a hybrid approach whereby
the saddle-free Newton method is applied in places where SGD appears to be
stuck.
3.3 Modern RNNs
The most successful RNN architectures for sequence learning date to two papers
from 1997. The ﬁrst, Long Short-Term Memory, by Hochreiter and Schmidhu-
ber, introduces the memory cell, a unit of computation that replaces traditional
artiﬁcial neurons in the hidden layer of a network. With these memory cells,
networks are able to overcome some diﬃculties with training encountered in
earlier recurrent nets. The second, Bidirectional Recurrent Neural Networks, by
Schuster and Paliwal, introduces the BRNN architecture in which information
from both the future and the past are used to determine the output at any
time t. This is in contrast to previous systems, in which only past input can
aﬀect the output, and has been used successfully for sequence labeling tasks in
natural language processing, among others. Fortunately, the two innovations
are not mutually exclusive, and have been successfully combined by Graves et
al. for phoneme classiﬁcation [26] and handwriting recognition [25].
3.3.1 Long Short-Term Memory (LSTM)
In 1997, to overcome the problem of vanishing gradients, Hochreiter and Schmid-
huber introduced the LSTM model. This model resembles a standard neural
network with a recurrent hidden layer, only each ordinary node (Figure 1) in
the hidden layer is replaced with a memory cell (Figure 9). The memory cell
contains a node with a self-connected recurrent edge of weight 1, ensuring that
the gradient can pass across many time steps without vanishing or exploding.
To distinguish that we are referencing a memory cell and not an ordinary
node, we use the index c.
16
Figure 9: LSTM memory cell as initially described in Hochreiter [32]. The
self-connected node is the internal state s. The diagonal line indicates that it
is linear, i.e. no link function is applied there. Nodes marked “Π” output the
product of their inputs. Dashed lines indicate recurrent edges and pink edges
have ﬁxed weight of 1.
17
The term “Long Short-Term Memory” comes from the following intuition.
Simpler recurrent neural networks havelong term memory in the form of weights.
The weights change very slowly over time encoding general knowledge about the
data. They also have short term memory in the form of ephemeral activations,
which pass from each node’s output to successive nodes. The LSTM model
introduces an intermediary sort of memory via the memory cell. A memory cell
is a composite of simpler units with the novel addition of multiplicative nodes,
represented in diagrams with Π. All elements of the LSTM cell are enumerated
and described below.
• Internal State: At the heart of each memory cell is a node s with linear
activation, which is referred to in the original paper as the ”internal state”
of the cell. We index cells with c and thus the internal state of a cell c is
sc.
• Constant error carousel: The internal state sc has a self-connected (re-
current) edge with weight 1. This edge, called the constant error carousel,
spans adjacent time steps with constant weight, assuring that error can
ﬂow across time steps without vanishing.
• Input Node: This node behaves as an ordinary neuron, taking input
from the rest of the network (at the previous time step) as well as from
the input. In the original paper and most subsequent work the input
node is labeled g. We adhere to this convention but note that it may be
confusing as g does not stand for gate. In the original paper, the gates
are called yin and yout but this is confusing because y generally stands for
output in the machine learning literature. Seeking comprehensibility, we
break with this convention and use i, f, and o to refer to input, forget
and output gates respectively as in [64]. When we use vector notation
we are referring to the values of the nodes in an entire layer of cells. For
example, g is a vector containing the value of g at each memory cell in a
layer. When the subscript cis used, it is to refer to an individual memory
cell.
• Multiplicative Gating: Multiplicative gates are distinctive features of
the LSTM model. Here a sigmoidal unit called a gate is learned given
the input and the incoming recurrent connections from the previous time
step. Some value of interest is then multiplied by this output. If the
gate outputs 0, ﬂow through the gate is cut oﬀ. If the gate outputs 1, all
activation is passed through the gate.
– Input gate: The original LSTM contains two gates. The ﬁrst is an
input gate ic, which is multiplied by the input node gc.
– Output gate: The second gate is termed the output gate, which we
notate as oc. This gate is multiplied by the value of the internal state
sc to produce the value of vc output by the memory cell . This then
feeds into the LSTM hidden layer at the next time step h(t+1) as well
as the output nodes ˆy(t) at the current time step.
18
Figure 10: LSTM memory cell with forget gate as described by Felix Gers et al.
in [21].
Since the original LSTM was introduced, several variations have been proposed.
Forget gates, proposed in 2000 by Gers and Schmidhuber [21], add a gate similar
to input and output gates to allow the network to ﬂush information from the
constant error carousel. Also in 2000, Gers and Schmidhuber proposed peephole
connections [20], which pass from the carousel directly to the input and output
gates of that same node without ﬁrst having to pass through the output gate.
They report that these connections improve performance on timing tasks where
the network must learn to measure precise intervals between events. While peep-
hole connections are not common in modern papers, forget gates have become
a mainstay of LSTM work.
Put formally, computation in the LSTM model proceeds according to the
following calculations which must be evaluated at each time step. This gives
the full algorithm for a modern LSTM with forget gates.
g(t) = φ(Wgxx(t) + Wihh(t−1) + bg)
i(t) = σ(Wixx(t) + Wihh(t−1) + bi)
f(t) = σ(Wfxx(t) + Wfhh(t−1) + bf)
o(t) = σ(Woxx(t) + Wohh(t−1) + bo)
s(t) = g(t) ⊙i(i) + s(t−1) ⊙f(t)
h(t) = s(t) ⊙o(t)
19
where ⊙stands for element-wise multiplication. The calculations for the simpler
LSTM without forget gates is given by setting f(t) = 1 for all t. We use the
tanh function φfor the input node gfollowing the latest state of the art setup of
Zaremba and Sutskever in [71]. However, in the original LSTM paper [32], the
activation function for g is the sigmoid σ. Again, h(t−1) is a vector containing
the values vc output by each memory cell c in the hidden layer at the previous
time step.
Intuitively, in terms of the forward pass, the LSTM can learn when to let
activation into the internal state. So long as the input gate takes value 0, no
activation can get in. Similarly, the output gate learns when to let the value
out. When both gates are closed, the activation is trapped in the LSTM, neither
growing nor shrinking, nor aﬀecting the output at the intermediary time steps.
In terms of the backwards pass, the constant error carousel enables the gradient
to propagate back across many time steps, neither exploding nor vanishing.
In this sense, the gates are learning when to let error in, and when to let it
out. In practice, the LSTM has shown a superior ability to learn long-range
dependencies as compared to simple RNNs. Consequently, the majority of state
of the art application papers covered in this review use the LSTM model.
3.4 Bidirectional Recurrent Neural Networks (BRNNs)
Along with the LSTM, one of the most used RNN setups is the bidirectional
recurrent neural network (BRNN) (Figure 11) ﬁrst described in [53]. In this
architecture, there are two layers of hidden nodes. Both hidden layers are con-
nected to input and output. The two hidden layers are diﬀerentiated in that the
ﬁrst has recurrent connections from the past time steps while in the second the
direction of recurrent of connections is ﬂipped, passing activation backwards in
time. Given a ﬁxed length sequence, the BRNN can be learned with ordinary
backpropagation. The following three equations describe a BRNN:
h(t)
f = σ(Whf xx + Whf hf h(t−1)
f + bhf )
h(t)
b = σ(Whbxx + Whbhb hb
(t+1) + bhb )
ˆy(t) = softmax(Wyhf hf
(t) + Wyhb hb
(t) + by)
Where h(t)
f and h(t)
b correspond to the hidden layers in the forwards and back-
wards directions respectively.
One limitation of the BRNN is that cannot run continuously, as it requires
a ﬁxed endpoint in both the future and in the past. Further, it is not an
appropriate machine learning algorithm for the online setting, as it is implausible
to receive information from the future, i.e., sequence elements that have not been
observed. But for sequence prediction over a sequence of ﬁxed length, it is often
sensible to account for both past and future data. Consider the natural language
task of part of speech tagging . Given a word in a sentence, information about
both the words which precede and those which succeed it is useful for predicting
20
Figure 11: Structure of a bidirectional recurrent neural network as described by
Schuster and Paliwal in [53].
that word’s part of speech. Karpathy et al. use such a network for generating
captions for images [35].
The LSTM and BRNN are in fact compatible ideas. The former introduces a
new basic unit from which to compose a hidden layer, while the latter concerns
the wiring of the hidden layers, regardless of what nodes they contain. Such an
approach, termed a BLSTM was used by Graves et al. to achieve state of the
art results on handwriting recognition and phoneme classiﬁcation [25] [26].
4 Applications of LSTM and BRNN
In the previous sections, we introduced the basic building blocks from which
nearly all state of the art recurrent neural networks are composed. Now, we’ll
look at several application areas where recurrent networks have been employed
successfully. We’ll quickly introduce the representations used for input and
output and the commonly used performance metrics. Then we will survey state
of the art results in machine translation, image captioning, video captioning,
and handwriting recognition.
4.1 Representing Natural Language for Input and Output
Many applications of RNNs involve text processing. Some applications, e.g.
image captioning, involve generating strings of text. Others, such as machine
translation and dialogue systems, require both inputting and outputting text.
21
In this section, we provide the background necessary to understand how text is
represented in most recurrent net models.
When words are output at each time step, generally the output consists of
a softmax vector y(t) ∈R K where K is the size of the vocabulary. A softmax
layer is like an element-wise logistic function that is normalized so that all of
its components sum to 1. Intuitively these outputs correspond to probabilities
that each word is the correct output at that time step.
For application where inputs consist of sequences of words, typically the
words are fed to the network one at a time as inputs in consecutive time steps.
In these cases, the simplest way to represent words is a one-hot encoding, using
binary vectors with a length equal to the size of the vocabulary, e.g. “1000” and
“0100” would represent the ﬁrst and second words in the vocabulary respectively.
Such an encoding is discussed in [18] among others. However, this encoding is
ineﬃcient, requiring as many bits as the vocabulary is large. Further, it oﬀers
no direct way to capture diﬀerent aspects of similarity between words in the
encoding itself. Thus it is more common to model words with a distributed
representation using a meaning vector. In some cases, these meanings for words
are learned given a large corpus of supervised data, but it is common to initialize
the meaning vectors using an embedding based on word co-occurence statistics.
Freely available code to produce word vectors from co-occurrence stats include
Glove from Pennington Socher and Manning [51], and word2vec [23], which
implements a word embedding algorithm from Mikolov et al. [46].
Such distributed representations for symbolic data were described by Hinton
in 1986 [29], used extensively for natural language by Bengio et al. in 2003 [8],
and more recently brought to wider attention in the deep learning community
by Socher, Manning and Ng in a number of papers describing Recursive Auto-
encoder (RAE) networks ([59], [56], [60], [58]). For clarity we point out that
these recursive networks, which are sometimes referred to as “RNNs” are not
recurrent neural nets. In contrast to RNNs which model arbitrary dependencies
between inputs at diﬀerent points in the sequence, recursive networks assume a
tree structure where each word in a sentence is a leaf and each sentence can be
represented as a binary tree. In this model, each internal node has a meaning
vector which can be calculated by concatenating the meanings of its children
and multiplying the composite vector by an encoding matrix. Each meaning
vector mj ∈R d and the encoding matrix Ae ∈R 2d×d. The composite meaning
mp ∈R d given to the parent p of two child nodes l and r in the syntax tree is
mp = σ(Ae[ml; mr]T)
This method seems restricted to sentences, and incompatible with the online
setting where the full sequence may not be ﬁxed in advance. Additionally,
while it has been successfully used for classiﬁcation tasks, it oﬀers no generative
model for composing text.
In many experiments with recurrent neural networks ([18], [63], [71]), input
is fed in one character at a time (and output generated one character at a time).
While the output is nearly always a softmax layer, many papers omit details
22
of how they represent single-character inputs. It seems reasonable to infer that
characters are encoded with a one-hot encoding. We know of no cases of paper
using a distributed representation at the single-character level.
4.1.1 Evaluation Methodology
One serious obstacle to training systems to output variable length sequences of
words are the ﬂaws of the available performance metrics. Developed in 2002,
BLEU score, a common evaluation metric for machine translation, is related to
modiﬁed unigram precision [48]. It is extended by taking the geometric mean
of the n-gram precisions for all values of n between 1 and some upper limit
N. In practice, 4 is a typical value for N, shown to maximize agreement with
human raters. Because precision can be made high by oﬀering absurdly short
translations, the authors of BLEU introduce a brevity penalty BP. Where q
is the length of the candidate translation and r is the length of the reference
translations the brevity penalty is expressed as
BP =
{
1 if q >r
e(1−r/q) if q <= r.
Then the BLEU score can be calculated as
BLEU = BP ·exp
( N∑
n=1
wnlog pn
)
where wn is a uniform weight 1 /N and pn is the modiﬁed n-gram precision.
BLEU score is commonly used in recent papers to evaluate both translation
and captioning systems. While BLEU score does appear highly correlated
with human judgments, there is no guarantee that any given translation with a
higher BLEU score is superior to another which receives a lower BLEU score.
In fact, while BLEU scores agree with human judgement across large sets of
translations, they are signiﬁcantly less accurate at the sentence level.
METEOR , introduced in 2005 by Banerjee and Lavie is an alternative met-
ric intended to overcome these weaknesses of the BLEU score [4]. METEOR
is scored based on explicit word to word matches between candidates and ref-
erences. When multiple references exist, the best score is used. Unlike BLEU,
METEOR exploits known synonyms and stemming. First they compute an
F-score
Fα = P ·R
α·P + (1 −α) ·R
based on single word matches where P is the precision and Ris the recall. Then
they calculate a fragmenting penalty Pen ∝ch/m where ch is the smallest
number of chunks of consecutive words such that the words are adjacent in
both the candidate and the reference, and m is the total number of matched
unigrams yielding the score:
METEOR = (1 −Pen) ·Fα.
23
Empirically, this metric has been found to agree more with human raters than
BLEU score.
However, METEOR is less straight-forward to calculate than BLEU. To
replicate the METEOR score reported by another party, one must exactly
replicate their stemming and synonym matching, as well as the calculations.
Both metrics rely upon having the exact same set of candidate translations.
Even in the straightforward case of binary classiﬁcation, without time de-
pendencies, commonly used performance metrics like F1 give rise to complicated
thresholding strategies which may not accord with any reasonable intuition of
what should constitute good performance [41]. Along the same lines, given these
new performance metrics which are weak proxies for our objectives and which
cannot be optimized directly, it may be diﬃcult to diﬀerentiate systems which
are truly stronger and those which most meticulously game the performance
metrics of interest.
4.2 Text Translation
Natural language translation is a fundamental problem in machine learning that
resists solutions with shallow methods. Some tasks, like document classiﬁcation,
can be performed successfully with representations like bag-of-words, which ig-
nore word order. But word order is essential in translation. The sentences
“Scientist killed by raging virus” and “Virus killed by raging scientist” have
identical bag-of-words representations.
In Sequence to Sequence Learning, published in 2014, Sutskever et al. present
a translation model using two multilayered LSTMs and demonstrate impressive
performance translating from English to French. The ﬁrst LSTM is used for
encoding an input phrase from the source language and one for decoding the
output phrase in the target language. Their model works according to the
following procedure (Figure 12):
• The source phrase is fed to the encode LSTM, one word at a time. The in-
puts are word vectors. This LSTM does not output anything. In practice,
the authors found that signiﬁcantly better results were achieved when the
input sentence was fed into the network in reverse order.
• When the end of the phrase is reached, a special input which indicates the
beginning of the sentence is fed to the decoder LSTM. This second LSTM
gets as input the start token, and also the ﬁnal state of the encoder This
LSTM outputs softmax probabilities over the vocabulary at each time
step.
• A word is chosen and fed to the network at the next step. At this time
step, the network receives activation along all recurrent edges as well as
the word vector of the chosen word.
For training, the true inputs are fed to the encoder, the true translation is
fed to the decoder, and loss is propagated back from the outputs of the decoder.
24
Figure 12: Sequence to sequence LSTM model as described by Sutskever et al.
[64]. The input blocks (blue and purple) all correspond to word vectors, which
are fully connected to the corresponding hidden state. Red nodes are softmax
outputs. The network consists of an encoding model (LSTM1) and a decoding
model (LSTM2). Weights are tied among all encoding steps and among all
decoding time steps.
The network is trained to maximize the likelihood of the correct translation.
At inference time a left to right beam search is used to determine which words
to output. A few among the most likely next words are chosen for expansion
after each time step. The beam search ends when the network outputs an end-
of-sentence (EOS) token. They train the model using only stochastic gradient
descent, without momentum, halving the learning rate every half epoch, after
the ﬁrst ﬁve. Using this approach, they achieve a BLEU score of 34.81, which
outperforms the best previous neural network NLP systems, and is very close to
the best published results, including systems that have explicitly programmed
domain expertise. When their system is used to rerank candidate translations
from another system, they achieved a BLEU score of 36.5
Another RNN approach to language translation is presented by Auli et al.
in Joint Language and Translation Modeling with Recurrent Neural Networks .
Their RNN model uses the word embeddings of Mikolov [2] and a lattice rep-
resentation of the decoder output to facilitate search over the space of possible
translations. In their lattice, each node corresponds to a sequence of words.
They report a BLEU score of 28.5 on French-English translation tasks. Both
papers report results on data from WMT datasets but Sutskever et al. only re-
port results on English to French translation while Auli et al. report French to
English translation. Thus it is impossible to directly compare the performance
of the two models.
4.3 Image Captioning
Recently, recurrent neural networks have been used successfully for image cap-
tioning, [35, 44, 67]. In this task, a training set consists of input images x and
target captions y. Given a large set of image-caption pairs, a model is trained
25
to predict the appropriate caption for an image.
Vinyals et al. follow up on the success in language to language translation
by considering captioning as a case of image to language translation. Instead of
both encoding and decoding with LSTMs, they introduce the idea of encoding
an image with a convolutional neural network, and then decoding it with an
LSTM. This idea is the core of several papers which address this topic. This
work clearly precedes [35] as it is cited throughout, but [44] claim it as an
independent innovation. With this architecture, they achieve then state of the
art results on the Pascal, Flickr30k, and COCO datasets.
Karpathy et al. similarly use a convolutional neural network to encode im-
ages together with a bidirectional neural network to decode translations, using
word2vec embeddings as word representations [35]. They consider both full-
frame evaluations and a model that captures correspondences between image
regions and text snippets. At inference, their procedure resembles the one de-
scribed for Sutskever et al., where sentences are decoded one word at a time.
The most probable word is chosen and fed to the network at the next time step.
This process is repeated until an EOS token is produced.
4.4 Other Interesting Applications
Handwriting recognition is an application area where bidirectional LSTMs have
been used to achieve state of the art results. In work by Liwicki, Graves et al.
( [42], [25]), data is collected from a whiteboard using an eBeam interface. The
interface collects the ( x,y) coordinates of the pen at regularly sampled time
steps. In the more recent paper, they use a bidirectional LSTM model, outper-
forming an HMM model by achieving 81 .5% word-level accuracy, compared to
70.1% for the HMM.
In the last year, a number of papers have emerged, that extend the success
of recurrent networks for translation and image captioning to new domains.
Among the most interesting of these applications are unsupervised video encod-
ing [61], video captioning [66] and program execution [71]. In [66], Venugopalan
et al. demonstrate a sequence to sequence architecture similar to that which
Sutskever uses for natural language translation. However, instead of both en-
coding and decoding words, they encode frames from a video and decode words.
At each time step the input to the encoding LSTM is the topmost hidden layer
of convolutional neural network. At decoding time, the network outputs prob-
abilities over the vocabulary at each time step.
In Learning to Execute ([71]), Zaremba and Sutskever experiment with net-
works which read computer programs one character at a time and predict their
output. They focus on programs which output integers and ﬁnd that for simple
programs, including adding two nine-digit numbers, their network, which uses
LSTM units, several stacked hidden layers, and makes a single left to right pass
through the program, can predict the output with 99% accuracy.
26
5 Conclusion
Over the past thirty-ﬁve years, recurrent networks have gone from impractical
models, primarily of interest for cognitive modeling and computational neuro-
science, to powerful and practical tools for large-scale supervised learning. This
progress owes to advances in model architectures, training algorithms, and par-
allel computing. Recurrent nets are especially interesting because they seem
able to overcome many of the extreme restrictions generally placed on data by
traditional machine learning approaches. With recurrent nets, the assumption
of independence between consecutive examples is broken, the assumption of
ﬁxed-dimension input is broken, and yet RNN models perform competitively
with or outperform the state of the art on many tasks.
While LSTMs and BRNNs set records, we ﬁnd it noteworthy that many
advances come from novel architectures rather than fundamentally novel algo-
rithms. It seems that research exploring the space of possible models, either
via genetic algorithms or a Markov Chain Monte Carlo based approach, should
be promising. Along these lines, we note that neural networks seem to of-
fer a wide range of easily transferable techniques. New activation functions,
training procedures, initializations procedures, etc. are generally transferable
across networks and tasks, often conferring similar beneﬁts. As the number of
such techniques grows, the practicality of testing all combinations diminishes.
However it seems reasonable to infer that as a community, neural network re-
searchers are exploring the space of a model architectures and conﬁgurations
much as a genetic algorithm might, mixing and matching techniques, with a
ﬁtness function in the form of evaluation metrics on major datasets of interest.
This suggests two things. First, as stated before, this body of research would
beneﬁt from automated procedures to explore the space of models. Second, as
we build systems designed to perform more complex tasks, we would beneﬁt from
improved ﬁtness functions. BLEU inspires less conﬁdence than the accuracy
reported on a binary classiﬁcation task. To this end, when possible, it seems
prudent to individually test techniques ﬁrst with classic feedforward networks
on datasets with established benchmarks before applying them to recurrent
networks in settings with less concrete evaluation criteria.
Lastly, the rapid success of recurrent neural networks on natural language
tasks leads us to believe that extensions of this work to longer form text would
be fruitful. Additionally, we imagine that dialogue systems could be built along
similar principles to the architectures used for translation, encoding prompts and
generating responses, retaining the entirety of conversation history as contextual
information.
Acknowledgements
My research is funded by generous support from the Division of Biomedical In-
formatics at UCSD, via a training grant from the National Library of Medicine.
This review has beneﬁted from insightful comments from Charles Elkan, John
27
Berkowitz, Balakrishnan Narayanaswamy, Stefanos Poulis, Sharad Vikram.
References
[1] Lecture 6.5- rmsprop: Divide the gradient by a running average of its recent
magnitude.
[2] Michael Auli, Michel Galley, Chris Quirk, and Geoﬀrey Zweig. Joint
language and translation modeling with recurrent neural networks. In
EMNLP, pages 1044–1054, 2013.
[3] Pierre Baldi and Gianluca Pollastri. The principled design of large-scale re-
cursive neural network architectures–DAG-RNNs and the protein structure
prediction problem. The Journal of Machine Learning Research, 4:575–602,
2003.
[4] Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for
MT evaluation with improved correlation with human judgments. In Pro-
ceedings of the acl workshop on intrinsic and extrinsic evaluation measures
for machine translation and/or summarization , pages 65–72, 2005.
[5] Justin Bayer, Daan Wierstra, Julian Togelius, and J¨ urgen Schmidhuber.
Evolving memory cell structures for sequence learning. In Artiﬁcial Neural
Networks–ICANN 2009, pages 755–764. Springer, 2009.
[6] Richard K Belew, John McInerney, and Nicol N Schraudolph. Evolving
networks: Using the genetic algorithm with connectionist learning. In In.
Citeseer, 1990.
[7] Yoshua Bengio, Nicolas Boulanger-Lewandowski, and Razvan Pascanu. Ad-
vances in optimizing recurrent networks. In Acoustics, Speech and Sig-
nal Processing (ICASSP), 2013 IEEE International Conference on , pages
8624–8628. IEEE, 2013.
[8] Yoshua Bengio, R´ ejean Ducharme, Pascal Vincent, and Christian Janvin.
A neural probabilistic language model. The Journal of Machine Learning
Research, 3:1137–1155, 2003.
[9] Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term
dependencies with gradient descent is diﬃcult. Neural Networks, IEEE
Transactions on, 5(2):157–166, 1994.
[10] James Bergstra, Olivier Breuleux, Fr´ ed´ eric Bastien, Pascal Lamblin, Raz-
van Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley,
and Yoshua Bengio. Theano: a CPU and GPU math expression compiler.
In Proceedings of the Python for scientiﬁc computing conference (SciPy) ,
volume 4, page 3. Austin, TX, 2010.
28
[11] Avrim L Blum and Ronald L Rivest. Training a 3-node neural network
is NP-complete. In Machine learning: From theory to applications , pages
9–28. Springer, 1993.
[12] Bob Carpenter. Lazy sparse stochastic gradient descent for regularized
multinomial logistic regression. Alias-i, Inc., Tech. Rep, pages 1–20, 2008.
[13] Ronan Collobert, Koray Kavukcuoglu, and Cl´ ement Farabet. Torch7: A
matlab-like environment for machine learning. In BigLearn, NIPS Work-
shop, number EPFL-CONF-192376, 2011.
[14] Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho,
Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle
point problem in high-dimensional non-convex optimization. In Advances
in Neural Information Processing Systems , pages 2933–2941, 2014.
[15] Wim De Mulder, Steven Bethard, and Marie-Francine Moens. A survey on
the application of recurrent neural networks to statistical language model-
ing. Computer Speech & Language, 30(1):61–98, 2015.
[16] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods
for online learning and stochastic optimization. The Journal of Machine
Learning Research, 12:2121–2159, 2011.
[17] Charles Elkan. Learning meanings for sentences.
[18] Jeﬀrey L Elman. Finding structure in time. Cognitive science, 14(2):179–
211, 1990.
[19] Felix Gers. Long short-term memory in recurrent neural networks. Unpub-
lished PhD dissertation, ´Ecole Polytechnique F´ ed´ erale de Lausanne, Lau-
sanne, Switzerland, 2001.
[20] Felix A Gers and J¨ urgen Schmidhuber. Recurrent nets that time and count.
In Neural Networks, 2000. IJCNN 2000, Proceedings of the IEEE-INNS-
ENNS International Joint Conference on , volume 3, pages 189–194. IEEE,
2000.
[21] Felix A Gers, J¨ urgen Schmidhuber, and Fred Cummins. Learning to forget:
Continual prediction with LSTM. Neural computation, 12(10):2451–2471,
2000.
[22] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectiﬁer
networks. In Proceedings of the 14th International Conference on Artiﬁcial
Intelligence and Statistics. JMLR W&CP Volume , volume 15, pages 315–
323, 2011.
[23] Yoav Goldberg and Omer Levy. word2vec explained: deriving mikolov
et al.’s negative-sampling word-embedding method. arXiv preprint
arXiv:1402.3722, 2014.
29
[24] Alex Graves. Supervised sequence labelling with recurrent neural networks ,
volume 385. Springer, 2012.
[25] Alex Graves, Marcus Liwicki, Santiago Fern´ andez, Roman Bertolami, Horst
Bunke, and J¨ urgen Schmidhuber. A novel connectionist system for uncon-
strained handwriting recognition. Pattern Analysis and Machine Intelli-
gence, IEEE Transactions on, 31(5):855–868, 2009.
[26] Alex Graves and J¨ urgen Schmidhuber. Framewise phoneme classiﬁcation
with bidirectional LSTM and other neural network architectures. Neural
Networks, 18(5):602–610, 2005.
[27] Frederic Gruau et al. Neural network synthesis using cellular encoding and
the genetic algorithm. 1994.
[28] Steven A Harp and Tariq Samad. Optimizing neural networks with ge-
netic algorithms. In Proceedings of the 54th American Power Conference,
Chicago, volume 2, 2013.
[29] Geoﬀrey E Hinton. Learning distributed representations of concepts.
[30] Sepp Hochreiter. Bridging long time lags by weight guessing and long
short-term memory.
[31] Sepp Hochreiter, Yoshua Bengio, and Paolo Frasconi. Gradient ﬂow in
recurrent nets: the diﬃculty of learning long-term dependencies.
[32] Sepp Hochreiter and J¨ urgen Schmidhuber. Long short-term memory.Neu-
ral computation, 9(8):1735–1780, 1997.
[33] John J Hopﬁeld. Neural networks and physical systems with emergent
collective computational abilities. Proceedings of the national academy of
sciences, 79(8):2554–2558, 1982.
[34] Michael I Jordan. Serial order: A parallel distributed processing approach.
Advances in psychology, 121:471–495, 1997.
[35] Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for gen-
erating image descriptions. arXiv preprint arXiv:1412.2306 , 2014.
[36] Alex Krizhevsky, Ilya Sutskever, and Geoﬀrey E Hinton. Imagenet clas-
siﬁcation with deep convolutional neural networks. In Advances in neural
information processing systems, pages 1097–1105, 2012.
[37] John Langford, Lihong Li, and Tong Zhang. Sparse online learning via
truncated gradient. In Advances in neural information processing systems,
pages 905–912, 2009.
[38] B Boser Le Cun, John S Denker, D Henderson, Richard E Howard, W Hub-
bard, and Lawrence D Jackel. Handwritten digit recognition with a back-
propagation network. In Advances in neural information processing sys-
tems. Citeseer, 1990.
30
[39] Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and
Zhuowen Tu. Deeply-supervised nets. arXiv preprint arXiv:1409.5185 ,
2014.
[40] Zachary C Lipton and Charles Elkan. Eﬃcient elastic net regularization
for sparse linear models. 2015.
[41] Zachary C Lipton, Charles Elkan, and Balakrishnan Naryanaswamy. Opti-
mal thresholding of classiﬁers to maximize F1 measure. In Machine Learn-
ing and Knowledge Discovery in Databases , pages 225–239. Springer, 2014.
[42] Marcus Liwicki, Alex Graves, Horst Bunke, and J¨ urgen Schmidhuber. A
novel approach to on-line handwriting recognition based on bidirectional
long short-term memory networks. In Proc. 9th Int. Conf. on Document
Analysis and Recognition, volume 1, pages 367–371, 2007.
[43] Andrew L Maas, Quoc V Le, Tyler M O’Neil, Oriol Vinyals, Patrick
Nguyen, and Andrew Y Ng. Recurrent neural networks for noise reduc-
tion in robust ASR. In INTERSPEECH. Citeseer, 2012.
[44] Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, and Alan Yuille. Deep caption-
ing with multimodal recurrent neural networks (m-RNN). arXiv preprint
arXiv:1412.6632, 2014.
[45] James Martens and Ilya Sutskever. Learning recurrent neural networks
with hessian-free optimization. In Proceedings of the 28th International
Conference on Machine Learning (ICML-11) , pages 1033–1040, 2011.
[46] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeﬀrey Dean. Eﬃ-
cient estimation of word representations in vector space. arXiv preprint
arXiv:1301.3781, 2013.
[47] Vinod Nair and Geoﬀrey E Hinton. Rectiﬁed linear units improve restricted
boltzmann machines. In Proceedings of the 27th International Conference
on Machine Learning (ICML-10) , pages 807–814, 2010.
[48] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a
method for automatic evaluation of machine translation. In Proceedings of
the 40th annual meeting on association for computational linguistics , pages
311–318. Association for Computational Linguistics, 2002.
[49] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the diﬃculty of
training recurrent neural networks. arXiv preprint arXiv:1211.5063 , 2012.
[50] Barak A Pearlmutter. Gradient calculations for dynamic recurrent neural
networks: A survey. Neural Networks, IEEE Transactions on , 6(5):1212–
1228, 1995.
[51] Jeﬀrey Pennington, Richard Socher, and Christopher D Manning. Glove:
Global vectors for word representation.Proceedings of the Empiricial Meth-
ods in Natural Language Processing (EMNLP 2014) , 12, 2014.
31
[52] David E Rumelhart, Geoﬀrey E Hinton, and Ronald J Williams. Learn-
ing internal representations by error propagation. Technical report, DTIC
Document, 1985.
[53] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural net-
works. Signal Processing, IEEE Transactions on, 45(11):2673–2681, 1997.
[54] Hava T Siegelmann and Eduardo D Sontag. Turing computability with
neural nets. Applied Mathematics Letters, 4(6):77–80, 1991.
[55] Yoram Singer and John C Duchi. Eﬃcient learning using forward-backward
splitting. In Advances in Neural Information Processing Systems , pages
495–503, 2009.
[56] Richard Socher, Eric H Huang, Jeﬀrey Pennin, Christopher D Manning,
and Andrew Y Ng. Dynamic pooling and unfolding recursive autoencoders
for paraphrase detection. In Advances in Neural Information Processing
Systems, pages 801–809, 2011.
[57] Richard Socher, Andrej Karpathy, Quoc V Le, Christopher D Manning,
and Andrew Y Ng. Grounded compositional semantics for ﬁnding and de-
scribing images with sentences. Transactions of the Association for Com-
putational Linguistics, 2:207–218, 2014.
[58] Richard Socher, Cliﬀ C Lin, Chris Manning, and Andrew Y Ng. Pars-
ing natural scenes and natural language with recursive neural networks.
In Proceedings of the 28th international conference on machine learning
(ICML-11), pages 129–136, 2011.
[59] Richard Socher, Christopher D Manning, and Andrew Y Ng. Learning con-
tinuous phrase representations and syntactic parsing with recursive neural
networks. In Proceedings of the NIPS-2010 Deep Learning and Unsuper-
vised Feature Learning Workshop, pages 1–9, 2010.
[60] Richard Socher, Jeﬀrey Pennington, Eric H Huang, Andrew Y Ng, and
Christopher D Manning. Semi-supervised recursive autoencoders for pre-
dicting sentiment distributions. In Proceedings of the Conference on Empir-
ical Methods in Natural Language Processing , pages 151–161. Association
for Computational Linguistics, 2011.
[61] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsu-
pervised learning of video representations using LSTMs. arXiv preprint
arXiv:1502.04681, 2015.
[62] Ilya Sutskever, James Martens, George Dahl, and Geoﬀrey Hinton. On the
importance of initialization and momentum in deep learning. In Proceed-
ings of the 30th International Conference on Machine Learning (ICML-13),
pages 1139–1147, 2013.
32
[63] Ilya Sutskever, James Martens, and Geoﬀrey E Hinton. Generating text
with recurrent neural networks. In Proceedings of the 28th International
Conference on Machine Learning (ICML-11) , pages 1017–1024, 2011.
[64] Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to sequence learn-
ing with neural networks. In Advances in Neural Information Processing
Systems, pages 3104–3112, 2014.
[65] Alan M Turing. Computing machinery and intelligence. Mind, pages 433–
460, 1950.
[66] Subhashini Venugopalan, Marcus Rohrbach, Jeﬀ Donahue, Raymond
Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence–video
to text. arXiv preprint arXiv:1505.00487 , 2015.
[67] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.
Show and tell: A neural image caption generator. arXiv preprint
arXiv:1411.4555, 2014.
[68] Paul J Werbos. Backpropagation through time: what it does and how to
do it. Proceedings of the IEEE, 78(10):1550–1560, 1990.
[69] Wikipedia. Backpropagation — Wikipedia, the free encyclopedia, 2015.
[Online; accessed 18-May-2015].
[70] Ronald J Williams and David Zipser. A learning algorithm for continually
running fully recurrent neural networks. Neural computation, 1(2):270–280,
1989.
[71] Wojciech Zaremba and Ilya Sutskever. Learning to execute. arXiv preprint
arXiv:1410.4615, 2014.
[72] Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv
preprint arXiv:1212.5701, 2012.
[73] Matthew D Zeiler, M Ranzato, Rajat Monga, M Mao, K Yang, Quoc Viet
Le, Patrick Nguyen, A Senior, Vincent Vanhoucke, Jeﬀrey Dean, et al. On
rectiﬁed linear units for speech processing. In Acoustics, Speech and Sig-
nal Processing (ICASSP), 2013 IEEE International Conference on , pages
3517–3521. IEEE, 2013.
33

Performance-Detective: Automatic Deduction of
Cheap and Accurate Performance Models
Larissa Schmid
Karlsruhe Institute of Technology,
KASTEL
Germany
Marcin Copik
ETH Zurich, D-INFK
Switzerland
Alexandru Calotoiu
ETH Zurich, D-INFK
Switzerland
Dominik Werle
Karlsruhe Institute of Technology,
KASTEL
Germany
Andreas Reiter
University of Applied Sciences
Karlsruhe, IDM
Germany
Michael Selzer
Karlsruhe Institute of Technology,
IAM-MMS
Germany
Anne Koziolek
Karlsruhe Institute of Technology,
KASTEL
Germany
Torsten Hoefler
ETH Zurich, D-INFK
Switzerland
ABSTRACT
The many configuration options of modern applications
make it difficult for users to select a performance-optimal
configuration. Performance models help users in understand-
ing system performance and choosing a fast configuration.
Existing performance modeling approaches for applications
and configurable systems either require a full-factorial ex-
periment design or a sampling design based on heuristics.
This results in high costs for achieving accurate models. Fur-
thermore, they require repeated execution of experiments
to account for measurement noise. We propose Performance-
Detective, a novel code analysis tool that deduces insights
on the interactions of program parameters. We use the in-
sights to derive the smallest necessary experiment design
and avoiding repetitions of measurements when possible,
significantly lowering the cost of performance modeling. We
evaluate Performance-Detective using two case studies where
we reduce the number of measurements from up to 3125
to only 25, decreasing cost to only 2.9% of the previously
needed core hours, while maintaining accuracy of the re-
sulting model with 91.5% compared to 93.8% using all 3125
measurements.
ICS ’22, June 28–30, 2022, Virtual Event, USA
© 2022 Copyright held by the owner/author(s). Publication rights licensed
to ACM.
This is the author’s version of the work. It is posted here for your personal
use. Not for redistribution. The definitive Version of Record was published
in 2022 International Conference on Supercomputing (ICS ’22), June 28–30,
2022, Virtual Event, USA , https://doi.org/10.1145/3524059.3532391.
CCS CONCEPTS
• Software and its engineering →Software performance.
KEYWORDS
automatic performance modeling, empirical performance
modeling, experiment design, configurable systems
ACM Reference Format:
Larissa Schmid, Marcin Copik, Alexandru Calotoiu, Dominik Werle,
Andreas Reiter, Michael Selzer, Anne Koziolek, and Torsten Hoefler.
2022. Performance-Detective: Automatic Deduction of Cheap and
Accurate Performance Models. In 2022 International Conference
on Supercomputing (ICS ’22), June 28–30, 2022, Virtual Event, USA.
ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/3524059.
3532391
1 INTRODUCTION
The costs of operating high-performance systems are mil-
lions of euros per year [8, 10]. While computing centers do
their best to design and run clusters economically, developers
must ensure that their applications scale efficiently on com-
puting clusters. Modern software systems are configurable
and allow users to set many parameters according to their
needs. For example, in the Pace3D material simulation [21],
users select algorithm settings and properties to simulate,
impacting performance metrics such as response time and
throughput [40, 50]. Choosing a set of parameters that yields
the best performance is challenging. Developers and users of-
ten do not know how configuration options interact or how a
single configuration option influences performance [19, 39].
An example of such a challenge is understanding application
ICS ’22, June 28–30, 2022, Virtual Event, USA Schmid et al.
Performance-
Detective
[ours]
Full-factorial
[Extra-P]
Sparse Modeling
[Extra-P]
Plackett-
Burman [PIM]
0
20
40
60
80
100
Core hours
Samples
Accuracy 2.9%
100%
1.5%
44.7%
4%
100%
12.5%
39.2%
91.5%93.8%
74.9%
62.6%
Figure 1: Performance-Detective vs Extra-P [12] and
Performance-Influence Models (PIM) [39]. Cost-
accuracy trade-offs on Pace3D with three parameters.
scalability, i.e., the interaction between problem size and the
number of processes [4].
Performance modeling helps users understand application
behavior by expressing application performance as func-
tions of input parameters [20, 25]. Since the advent of High-
Performance Computing (HPC), performance experts man-
ually identify and model the parts of the application they
consider critical to performance. Due to the high cost of per-
formance experts, HPC developers often intuitively select a
subset of the configuration space for evaluation. Automatic
performance modeling generates models from empirical mea-
surements that cover all configuration options. Even though
configuration spaces are constrained and smaller than combi-
natorial explosion would suggest [34], the number of options
still makes exhaustive measurements infeasible.
Modeling frameworks for applications with many con-
figuration parameters can be classified into black-box and
white-box approaches. The former suffers from high sam-
pling costs and relies on heuristics to reduce the experiment
design [39], introducing the risk of excluding interactions
between options from the sample set. The likelihood of this
problem can be decreased by using more samples – leading to
a hard to quantify trade-off between model accuracy and the
number of samples [17, 27, 36]. White-box approaches can
support modeling of numerical options but do not use known
interactions between options, resulting in an expensive full-
factorial experiment design [13] or using heuristics to reduce
the number of samples [48]. Other white-box procedures con-
centrate on binary and binary-encoded options [45, 46] and
do not consider numerical parameters such as the problem
size or the number of processes.
In this paper, we introduce Performance-Detective, a novel
white-box modeling methodology that significantly lowers
experimentation costs while maintaining the accuracy of
the resulting models (Figure 1). In contrast to previous sam-
pling optimizations that used imprecise heuristics, we use
program information to deduce an optimized, minimal ex-
periment design, removing measurement points that do not
affect known interactions between non-functional parame-
ters. We use parametric performance models obtained from
the taint-based analysis [ 13] to understand the impact of
parameters on program functions. Through a step-by-step
analysis of parameter interactions, we derive conclusions on
parameter interactions in the program’s control flow. Ap-
plying the deduced conclusions to the experiment design
removes measuring points unnecessary to model parameter
interactions. Thus, Performance-Detective can reduce the di-
mensionality of experiments from exponential to polynomial
while avoiding the risk of excluding parameter dependencies
from the experiment. The experiment design ofPerformance-
Detective is orthogonal to the modeling approach and can be
used with black-box and white-box performance modeling.
The deduced experiment design makes performance mod-
eling more affordable and is easily applicable alongside mod-
ern performance modeling systems. To quantify the increased
efficiency and validate model correctness, we empirically
evaluate Performance-Detective using a multi-physics solver
(see Figure 1) and a particle transport application.Performance-
Detective maintains accuracy of 91.4% while reducing the
costs of measurements by a factor of up to 34 times, compared
against both the Extra-P empirical performance modeling
tool [12, 36] and Performance-Influence Models [39].
Contributions:
•Performance-Detective, a white-box measurement
methodology using a noveldeductive analysis that uses
the results of performance tainting to derive an opti-
mized minimal subset of required measurements out
of a multi-dimensional configuration space1.
•An extensive evaluation against state-of-the-art mod-
eling workflows, proving high accuracy and reduced
experiment size.
•We identify main loops within applications using our
deductive analysis, and leverage this to reduce the
cost of experiments even further, in essence applying
classical analytical performance modeling techniques
to modern automatic modeling approaches.
•Two case studies of modeling and evaluating applica-
tions extrapolated and interpolated test points.
2 FOUNDATIONS
The deduction process of Performance-Detective is based on
a program’s parametric profile obtained with taint-based per-
formance modeling provided by Perf-Taint (Sec. 2.1).
Performance-Detective overcomes the limitations of exist-
ing performance modeling tools (Sec. 2.2), enabling efficient
and reliable white-box modeling.
2.1 Perf-Taint
Perf-Taint [13, 14] is a hybrid modeling tool that enhances
the black-box and analytical modeling tools with program
1Workflow prototype and replication package: https://doi.org/10.5445/IR/
1000146001
Performance-Detective: Automatic Deduction of Performance Models ICS ’22, June 28–30, 2022, Virtual Event, USA
void important(int x, int* arr) {
for(int i = 0; i < x; ++i)
arr[i] += i;
}
void unimportant(int x, int* arr) {
for(int i = 0; i < 16; ++i)
arr[i] += x;
}
params: [ x ]
functions: {
important: {
loops: [ x ],
MPI: None
}
}
LLVM Static
Analysis
Dynamic Taint
Analysis
Figure 2: Perf-Taint: applying the taint-based analysis
produces a JSON-like parametric performance profile.
information. Perf-Taint applies a sequence of static and dy-
namic analyses to the program to understand how its com-
putational effort is affected by a change in input parameters.
The result of the operation is a parametric profile of a pro-
gram, consisting of a list of performance-relevant functions
and parameters whose values can change the performance.
Perf-Taint is built on top of LLVM [ 29] and supports dis-
tributed parallel applications implemented in C/C++ with
MPI. Internally, the tool uses static LLVM loop analysis [16]
and DFSan [15], a dataflow taint analysis library extended
with support for control-flow tainting [49].
We present Perf-Taint on an example program with two
functions in Figure 2. The functionimportant is performance-
relevant because its complexity changes with parameter x.
Thus, both the parameter and the function are included in
the performance profile on the right side. On the other hand,
the function unimportant involves only a constant amount
of computation. This information obtained from program
analysis improves the overall modeling workflow by restrict-
ing the modeling to performance-relevant program elements
In practice, many short-running functions are particularly
affected by noise and could otherwise generate false models
that would make the modeling process more difficult and
error-prone. These can be automatically excluded from the
modeling process if Perf-Taint identified them as constant.
2.2 Limitations of modeling frameworks
The state-of-the-art modeling approaches do not allow for
efficient modeling of computing applications with many pa-
rameters and numerical options.
Performance-Influence Models (PIMs) [48] derive perfor-
mance models on the function level, but they do not use pa-
rameter dependencies and require expensive tracing instead
of profiling. White-Box PIMs [ 45, 46] do not support nu-
merical options. Though numerical options can be encoded
and discretized [46], the approach does not use a learning
method and, therefore, cannot predict inter- or extrapolating
training configurations.
Extra-P [12] relies on the full-factorial experiment design
where all parameter combinations must be considered, lead-
ing to a combinatorial explosion of the number of measure-
ments. Heuristics have been proposed to optimize the ex-
periment design [36], but they introduce a risk of missing
parameter interactions and negatively affecting model qual-
ity. Furthermore, the impact of the noise present in measure-
ments makes modeling with more than three parameters
particularly challenging.
Perf-Taint [13] does not propose a deterministic and effec-
tive methodology for user analysis of the experiment design.
While Perf-Taint detects no function depending on both x1
and x2 in the example in the listing below, it does not use
parameter knowledge to reduce the sampling set.
void f( int x) { void g( int x) {
for (int i : irange (0 , x)) for ( int i : irange (0 , x))
calculate (); calculate ();
} }
f(x1); g(x2);
In the next code example, all functions are called in the main
loop. If the main loop is executed often enough, repetitions
of experiments are not necessary as the repetition of the
calculations already accounts for measurement noise.
for ( int i = 0; i < iters ; i ++) {
f(x1); g(x2);
}
However, Perf-Taint does not make use of this to suggest
fewer experiments. Even though 25 and 5 measurements are
sufficient to model the performance in these examples, Extra-
P and Perf-Taint still require and suggest 125 measurements
in both cases. Even though heuristics for sampling can lower
experiment size, they do so at the cost of reducing model
accuracy. While more samples can increase the accuracy,
there is no strategy for selecting them, leading to a full-
factorial experiment design.
3 APPROACH
Figure 3 depicts the overall workflow of performance model-
ing that we will detail in the following subsections. We start
by analyzing the system to trace the influence of configura-
tion options on single functions (Section 3.1). Performance-
Detective then uses the insights about which parameter in-
fluences the performance of which function to deduce a
minimal experiment design that exploits insights about the
interplay of options (Section 3.2). After executing the ex-
periments (Section 3.3), the performance of single functions
can be modeled using any learning methodology. Finally, we
derive a whole system model by summing up single models.
We assume that the system’s performance varies in func-
tions that contain non-constant loops, i.e., a configuration
option determines how often they are executed and how
many iterations they include. First, we distinguish between
functional and non-functional configuration options: while
the functional options define the problem to be solved, non-
functional options are free to vary.
Functional 
Non- 
functional 
Performance- 
relevant 
Problem sizePhysical
constant
Number of
processes
Not 
Performance- 
relevant 
Then, we distinguish between con-
figuration options that affect per-
formance and those that do not. In
ICS ’22, June 28–30, 2022, Virtual Event, USA Schmid et al.
3.3 Instrumented
Experiments
3.2 Experiment
design 
3.1 System 
analysis 
Deterministically
reduced
experiment
design
Instrument
relevant functions
Instrument all
functions
Parametric
profile
Heuristic
samplingparam2
param1 param2
param1
Modeling
Performance-
Detective:  
White-Box
Performance
Modeling
Extra-P:
Black-Box
Performance
Modeling 
Model relevant
functions
Model all
functions
1.3 *
param1
Model: -   -
1.3 * param1  
+ 10-4 * param2
...
Noise
Sample all, trace
relevant functions 
Heuristic
samplingparam2
param1
Weber et al.:
Performance-
Influence
Modeling 
Model all
functions
...
Noise
Expert selects
parameters
Expert selects
parameters
?
Extra-P      Weber  
                       et al.
func2: param2
func3: param1
func4: constant
func1: param1
func2: param2
func3: param1
func4: constant
func1: param1
func4: constant
func1: param1
func2
func3
func4
func1
func2
func3
func4
func1
func2
func3
func4
func1
func2
func3
func4
func1
func1: param1
func4
10-6 * param1 
func1: param1
func4?
?
Figure 3: Modeling workflow of Performance-Detective
compared to Extra-P [9, 36] and Weber et al. [48].
Based on the system analysis, Performance-Detective
deterministically reduces the experiment design.
modeling, we ignore options that af-
fect neither the result nor the perfor-
mance of a program. However, the
exact distinction between functional
and non-functional options is sub-
jective and domain-specific [46].
We will use the example program
in Figure 4 as a running example. It takes the three param-
eters x1, x2, and iters as configuration options and does
calculations based on them.
3.1 System analysis
In the first step, we analyze the system to find out how config-
uration options influence its performance. We do this using
the Perf-Taint approach introduced in Section 2.1. The anal-
ysis of Perf-Taint outputs parameter dependencies for each
performance-relevant function: a function always depends
on variables that determine the loop iteration count in the
function.
Example. Figure 4 shows functions’ dependencies in the
colored boxes. Function preCalculate is not influenced by
the configuration options and is identified as constant. foo
is dependent on x1. bar and baz iterate over the parameterz
and depend onx2. We observe thatiters has a multiplicative
influence on the runtime of all functions and deduce that it
linearly affects all computations. We do not consider foo to
void baz(int z) {
  int flag = preCalculate(z);
  for (int i = flag; i < z; i++) {
    //calculate something 
  } 
}
int preCalculate(int z) { 
  if (x2 < 10) return 0;
  return 1; 
}
void foo(int y, int z, int iters) {
  for (int i = 0; i < iters; i++) {
    for (int j = 0; j < y; j++) {
      //calculate something
    }
    bar(z);
    baz(z);
  }
}
void bar(int z) { 
  for (int i = 0; i < z; i++) {
    //calculate something 
  } 
time(foo) = f(x1, iters)
time(baz) = f(x2, iters)
time(preCalculate) =
constant
time(bar) = f(x2, iters)
int main(int argc, char ** argv) {
  int x1 = atoi(argv[1]);
  int x2 = atoi(argv[2]);
  int iters = atoi(argv[3]);
  register_variables(x1, x2, iters) 
  y = x1 * 5;
  z = x2 / 7;
  foo(y, z, iters); 
}
System: Parameters x1, x2, and iters Full Experiment
Design Space
Insight 1: iters
influences
computations
linearly 
Deduced Experiment
Design
y depends on x1
z depends on x2
Insight 2: no
function
depending on 
both x1 and x2
Figure 4: Running example. The system takes the con-
figuration options x1, x2, and iters.
depend on x2 because we exclusively measure time spent in
functions (cf. Section 3.3).
3.2 Experiment design
Based on the derived dependencies from the system analy-
sis, Performance-Detective deduces the minimal experiment
design. From a black-box view of the system in Figure 4,
Performance-Detective starts with a full-factorial experiment
design of the three parameters x1, x2, and iters. Since
Performance-Detective detects that iters influences the run-
time of foo, bar, and baz linearly, measuring variations of
iters will not provide us additional insights into the sys-
tem’s performance. Therefore, Performance-Detective can
exclude this parameter from the experiment design. While
the linear influence is easy to see in the example, this may
not be the case for more complex real-world programs. To
verify that the influence of the iters parameter is linear in
such a case, we can measure execution times of loop itera-
tions. If the runtime does not change significantly between
them, i.e., the coefficient of variation is sufficiently small
between iterations, we verify that the influence of theiters
parameter is indeed linear.
Furthermore, we observe that x1 and x2 influence the per-
formance of distinct functions and do not interact with each
Performance-Detective: Automatic Deduction of Performance Models ICS ’22, June 28–30, 2022, Virtual Event, USA
other. Therefore, additional samples measuring their inter-
actions will not provide further insights, and Performance-
Detective can exclude them from the experiment design.
In general, we assume that if options influence different
sets of functions, they do not interact with each other, and
Performance-Detective can vary them simultaneously. Also,
we do not consider options linearly influencing system run-
time, such as the number of main loop iterations.
Extra-P requires a full-factorial measurement setup [ 9]
(Section 2.2). Even the sparse modeling approach with heuris-
tics [36] requires at least five measurements per parame-
ter to capture interactions between parameters. In contrast,
Performance-Detective can detect the lack of interaction and
strike out measurement configurations where only one pa-
rameter is changed, and the others are kept constant. This
improvement reduces the dimensionality of the experiment
compared to a full-factorial setup and provides further sav-
ings when compared to sparse modeling.
In the example above, Performance-Detective determin-
istically reduces the number of measurements from 125 in
a full-factorial experiment design to only five measurement
points without sacrificing accuracy.
3.3 Profiling
We use an instrumentation-based approach to capture the
total time spent in a function during one execution of the
program, summarizing time across all invocations. We mea-
sure time spent in a function exclusively, i.e., not including
time spent in calls to other instrumented functions. We in-
strument functions identified as performance-relevant by
Perf-Taint, i.e., containing loops dependent on configuration
options and the main function. Previous works suggested
repeating each sample five times to account for measurement
noise. However, when Performance-Detective identifies a lin-
ear dependency of parameter on the main calculation, we
can skip the repeated measurements and even halt execution
after gathering at least five iterations of the main calculation
loop. While this approach is common in analytical modeling,
it has been outside the reach of automated modeling due to
the difficulty of identifying configuration parameters that
control the number of executions of the main loop.
Example. In Figure 4, Performance-Detective disregards the
constant function preCalculate for instrumentation. We
instrument foo, bar, and baz, as they depend on annotated
input parameters, and main to capture the total runtime of
the application. In exclusive measurement, the time obtained
for foo does not include the time spent in bar. However, the
time of baz includes the time spent in preCalculate.
3.4 Limitations
If all parameters are intertwined, no pruning of parameter
combinations to measure is possible. Also, we do not re-
gard binary options as switching between them results in
performance jumps. We assume that performance-relevant
behavior is located in computational loops and MPI commu-
nication routines. Modeling of recursion is not supported,
but recursive computations are rare in HPC [13].
4 MODELING WORKFLOWS
Performance-Detective is orthogonal to the instrumentation
and learning methodology, and it can reduce the costs of ex-
periments in different performance modeling toolchains. We
consider two state-of-the-art modeling workflows: the black-
box, empirical Extra-P performance modeling tool (Sec. 4.1)
and the white-box Performance-Influence Models (Sec. 4.2).
4.1 Extra-P
Extra-P [12] is a performance modeling tool that expresses
the effect of configuration parameters 𝑥𝑖, 𝑖 ∈ {1, . . . , 𝑚}
on a performance metric 𝑓 (𝑥1, . . . , 𝑥𝑚). The result, such as
𝑡 (𝑛, 𝑝)= 10·𝑛 ·log(𝑝), is a familiar, human-readable function.
In practice, the configuration parameters most often an-
alyzed are problem size and process count, and the metric
of interest is usually the runtime. This allows developers
to identify performance bottlenecks in their applications,
especially when using it in conjunction with tools such as
Score-P [26] that allows automatic instrumentation and mea-
surement at the granularity of individual function calls.
The core assumption of the methods is that the complexity
of most algorithms can be expressed using a small number of
building blocks, summarized in Equation 1. The Performance
Model Normal Form (PMNF) models a metric as a combination
of polynomial and logarithmic expressions of configuration
parameters. This limits the possible search space sufficiently
to allow for a fast traversal while still being sufficiently flex-
ible to cover the overwhelming majority of applications.
𝑓 (𝑥1, . . . , 𝑥𝑚)=
𝑛Õ
𝑘=1
𝑐𝑘 ·
𝑚Ö
𝑙=1
𝑥𝑖𝑘𝑙
𝑙 ·log𝑗𝑘𝑙
2 (𝑥𝑙 ) (1)
An issue not addressed in the state-of-the-art approaches
is modeling more than three parameters. In practice, noise
makes detecting parameters with a smaller impact on metrics
of interest effectively impossible for even four parameters.
4.2 Performance-Influence Models
Performance-Influence Models (PIMs) [39] describe how con-
figuration options and their interactions influence the perfor-
mance of a system. They support a theoretically unlimited
number of binary and numerical parameters. PIMs can be
created using different sampling and learning techniques.
ICS ’22, June 28–30, 2022, Virtual Event, USA Schmid et al.
The models are learned in a black-box manner by providing
the configurations used and measured total system runtime
as inputs. Due to a high number of supported techniques, it is
unclear which combination of methods will provide the best
accuracy. However, there are works [17, 24] investigating
the interplay of sampling and learning techniques and the
influence of the sample size on model accuracy.
To further overcome these trade-offs between measure-
ment effort and accuracy, Velez et al. [45, 46] proposed White-
Box Performance-Influence Modeling for binary and binary-
encoded configuration options. They analyze the options’
influence on the system’s execution and help find an optimal
sample set. However, numerical options such as the problem
size are not supported since continuous parameters would
be tedious to encode. Weber et al. [ 48] present a two-step
white-box process for creating PIMs on the function level.
First, they use samples of the application execution to learn
a PIM for every function. Then, they use tracing to derive
more accurate models for functions that could not be learned
with a specified accuracy. They use the extended Plackett-
Burmann design [47] for sampling the numerical options
and decision trees for learning models.
5 CASE STUDIES
We illustrate the deduction process of Performance-Detective
presented in Section 3 with two case studies: Kripke, a 3D Sn
particle-transport proxy application, and a real-world case
study from Pace3D (Parallel Algorithms for Crystal Evolu-
tion) [21], a multi-physics framework that simulates how a
material reacts to outside influences. For both case studies,
we only consider the execution of the main calculation loops
as this is where most of the work happens.
Pace3D: Pressure calculation with projected conjugate gradi-
ent method. The calculation consists of two steps: First, an
approximate solution is calculated on a coarse grid. This ap-
proximate solution is then used to calculate the real solution
on the fine grid. The grids represent the material, with each
cell corresponding to one cube of the material. The solver
iterates until it reaches a convergence criterion or a given
maximum number of iterations. We consider the number
of processes and the number of coarse grid cubes as non-
functional parameters. We also use the size of the material
to predict how much material each process should get to
achieve the best performance. Additionally, we consider the
maximum number of iterations.
Kripke. Angular fluxes are calculated using different num-
bers of directions and groups. Kripke was built to research
how different data layouts, programming paradigms, and
architectures influence performance [ 1]. We consider the
number of direction sets as well as the number of processes
1 for ( int i = 0; i < cubes ; i ++) {
2 // spacing = vol / cubes
3 for ( int j = 0; j < spacing ; j ++) {
4 // complexity = cubes · spacing
5 // complexity = cubes · ( vol / cubes )
6 // complexity = vol
7 }
8 }
Listing 1: Program iterating over the spacing
Constant:
12
vol: 20
cubes: 28
procs: 26
cubes and
procs: 4
vol and
procs: 18
(a) Pace3D
Constant:
19 dirsets: 33
procs: 2
(b) Kripke
Figure 5: Number of functions depending on the anno-
tated parameters per case study as Venn diagrams.
as parameters. Additionally, we are interested in the number
of iterations.
5.1 System analysis
We analyze the scenarios with Perf-Taint [13]. The only nec-
essary modifications to the source code are annotating and
registering the variables corresponding to the parameters of
interest. We run the analysis on a small problem size using
only a few iterations, assuming that the computations and
analysis results are representative, and validate the results
with larger measurements in Section 6.2.
5.1.1 Pace3D. We annotate variables in the code corre-
sponding to the number of processes (procs), material size
(vol), coarse grid size (cubes), and spacing of the coarse grid
(spacing). Additionally, we annotate the number of iterations
for the fine as well as for the coarse grid. This results in a
total of 21 lines of code added.
The coarse grid spacing is a dependent parameter (not
linearly independent) and is calculated by dividing the mate-
rial size by the coarse grid size (spacing = vol/cubes). Hence,
we can replace the spacing by vol/cubes, substituting it by
the independent parameters it is defined by. Listing 1 shows
an example where we can conclude that the material size
determines the performance of the loops.
From the results, Performance-Detective identifies that the
maximum number of iterations on the fine grid determines
the runtime of the main calculation. To verify that the in-
fluence is linear, we trace the iterations of the loop for one
execution of the program and check whether the runtime of a
single loop iteration changes. To do so,Performance-Detective
calculates the coefficient of variation between them. In our
Performance-Detective: Automatic Deduction of Performance Models ICS ’22, June 28–30, 2022, Virtual Event, USA
Insight 1: 
iters influences
computations
linearly
Parameters:  
vol - total volume 
cubes - coarse grid size 
procs - number of processes 
iters - number of iterations 
Full Experiment Design
Space: 
3125 experiment executions 
Insight 2: 
vol and cubes
do not interact
Choose iters  >= 5  
instead of measurement
repetitions
Full-factorial
Experiment Design
with 3 parameters:
125 experiment
executions
Deterministically
reduced Experiment
Design: 
25 experiment
executions 
Figure 6: Deducing the minimal experiment design for
Pace3D.
scenario, the coefficient of variation is 0.05 for 100 loop it-
erations. We conclude that the influence of iters is indeed
linear.
Furthermore, the analysis results show that 12 functions
are constant, and 54 are dependent on the annotated param-
eters. For the 2 functions dependent on the spacing, we can
automatically assess that they in fact depend on the total vol-
ume, using replacement as shown in the example in Listing
1. The results, as shown in Figure 5a, show that the material
size and the coarse grid size influence different functions.
5.1.2 Kripke. To annotate the number of processes (procs)
as well as the number of direction sets (dirsets) and iterations
(iters), we add a total of five lines of code. Using the analysis
results, Performance-Detective analyzes that the number of
iterations determines the runtime of the main calculation. As
before, to verify that the influence is linear, we trace the loop
iterations of one program execution. As the coefficience of
variation between the measured 10 iterations is 0.0067, we
conclude that the influence of iters is linear. Moreover, the
analysis shows (cf. Figure 5b) that 19 functions are constant
and 35 are dependent on the annotated parameters, with 33
depending on dirsets and 2 onprocs. The number of processes
influences different functions than the number of direction
sets.
5.2 Minimal Experiment Design
Figure 6 shows the process of Performance-Detective deduc-
ing the experiment design for Pace3D based on the insights
gained from analyzing the system. From a black-box view
at the system, Performance-Detective starts with four param-
eters and a full-factorial experiment design, leading to 625
measurement points. To account for measurement noise,
each point has to be executed five times, leading to a total of
3125 experiment executions.
As we know that iters linearly influences the runtime of
the main computation, Performance-Detective can exclude
iters from the parameter space (Insight 1 ). Also, we can skip
repetitions of the execution of measurement points and
choose a sufficiently high value (≥5) for iters instead. This
reduces the experiment design to 125 points that need to
be executed only once. We also know that the functions af-
fected by the coarse grid size are distinct from those affected
by the total size. Thus, Performance-Detective can strike out
configurations aiming to find interactions between them
from the experiment design ( Insight 2 ). This means that
Performance-Detective varies 𝑣𝑜𝑙 and 𝑐𝑢𝑏𝑒𝑠 simultaneously
and only the number of processes independently of them. As
procs interacts with vol and cubes according to the analysis,
Performance-Detective includes procs as interacting parame-
ter into the experiment design. Figure 7 (Sec. 6.2) shows the
resulting training data points for Pace3D as crossed circles.
For Kripke, Performance-Detective deduces the experiment
design similarly: As iters has a linear influence on the run-
time of the main computation, it can exclude it from the
parameter space (Insight 1) and we can skip repetitions of ex-
periments. Furthermore, Performance-Detective can remove
configurations aimed at finding interactions between 𝑝𝑟𝑜𝑐𝑠
and𝑑𝑖𝑟𝑠𝑒𝑡𝑠 as they influence distinct sets of functions (Insight
2). Thus, Performance-Detective varies 𝑝𝑟𝑜𝑐𝑠 independently
of 𝑑𝑖𝑟𝑠𝑒𝑡𝑠 , resulting in 5 measuring points with (p, dirsets)=
(8, 8), (16, 16), (32, 24), (64, 32), (128, 64).
Performance-Detective reduced the full experiment design
space of 3125 experiment executions to only 25 executions
needed for Pace3D, and from 625 to 5 for Kripke.
6 EVALUATION
To evaluate Performance-Detective, we assess the accuracy of
the performance models generated for the case studies pre-
sented in Section 5. We evaluate the reduction of repetitions
by inclusion of iterations and variation of independent pa-
rameters individually. Therefore, we formulate the following
research questions:
RQ1 What is the model accuracy when generating it from a
single measurement with a high number of iterations?
RQ2 What is the model accuracy when generating it from
a minimal experiment design by varying independent
parameters simultaneously?
To answer the RQs, we compare our models with mod-
els generated following conventional experiment and sam-
pling designs. The expected outcome of the evaluation is
maintained accuracy while reducing the dimensionality of
experiment design and not repeating experiment executions,
resulting in significantly decreased cost for measurements.
6.1 Instrumented Experiments
Table 1 shows the hardware and software systems used for
measuring. We instrument the application using the list of
important functions generated by Perf-Taint and repeat the
measurement of each configuration five times to assess RQ1
and RQ2 separately. The coefficient of variation between
ICS ’22, June 28–30, 2022, Virtual Event, USA Schmid et al.
CPU Intel Xeon Gold 6230 2.1GHz
Cores 40 on 2 sockets
Memory 96 GB
GCC 10.2 (Pace3D), 11.2 (Kripke)
MPI OpenMPI 4.0.5 (Pace3D), OpenMPI 4.1.2 (Kripke)
Software Score-P 7.0 [26] (Pace3D), Score-P 7.1 (Kripke),
Perf-Taint, Extra-P
Table 1: Measurement environment
Modeling
approach
#executions Mean error Pace3D Mean error Kripke
interpolate extrapolate interpolate extrapolate
Extra-P 5 8.31 % 9.31 % 4.56 % 18.32 %
Extra-P 1 8.05 % 8.74 % 4.33 % 15.17 %
Decision Trees 5 21.61 % 60.15 % 21.79 % 25.68 %
Decision Trees 1 22.62 % 62.80 % 22.38 % 25.31 %
Table 2: Mean error of models generated from
Performance-Detective experiment design using a sin-
gle application execution vs five repetitions
105.33 105.94105.64 106.24 106.41
103.9
103.24
103
103.53
104.43
105.51 105.81 106.11 106.33 106.54 106.71
103.41
104.08
103.61
104.73
105.03
103.18
vol
cubes
Figure 7: Overview of training and test points on log-
arithmic scales. Training data Performance-Detective:
, training data full-factorial:
 and
 . Test data
interpolated:
 , test data extrapolated:
 .
the repetitions of the same configuration is 0.1 or less for
all configurations. We always use the filter file containing
important functions gained from system analysis to instru-
ment only relevant functions to compare the predictions of
all models. Otherwise, the evaluation would be less mean-
ingful because the models generated when instrumenting all
functions cannot predict the actual execution time as they
have more profiling overhead that we cannot remove from
the measurements.
6.2 RQ1: Modeling using a single
measurement
To evaluate whether the inclusion of iterations in the model
can simplify the experiment design, saving us repetitions
of the measurements, we generate a model from a single
execution of each measurement and compare the predictions
of this model with test points. For evaluating the accuracy,
we use the mean time of the five repeated executions of each
test measurement point. To obtain a prediction of the total
execution time, we sum up the predictions for the single
functions and evaluate them against the execution time of
the evaluation configurations.
We use two testing sets, containing either inter- or ex-
trapolated measurement points. Figure 7 shows an overview
of the test data sets used for Pace3D. For the interpolated
test data of both case studies, we measure configurations in
between the training data points. For Pace3D, we do not in-
terpolate the number of processes as this would not provide
insight into the quality of the minimal experiment design:
We still treat the number of processes as a separate parameter
and vary it separately from vol and cubes, as it interacts with
both according to the analysis. For the respective indepen-
dent parameters, we use points between each value used for
training, as shown in Figure 7. This results in 80 measuring
points for Pace3D and 16 for Kripke with the interpolated
values procs = (12, 24, 48, 96)and dirsets = (12, 20, 28, 48).
For testing how well the model predicts extrapolated con-
figuration points, we extrapolate parameters identified as
independent and measure them together with the respec-
tive other parameter using a value used for the training set.
This means that we measure the extrapolated values 54,000
and 108,000 for cubes together with the extrapolated values
3,456,000 and 5,184,000 for vol as well as with all the val-
ues for vol used for the training set, shown in Figure 7, for
Pace3D. Following this approach results in 119 measuring
points for the extrapolated test data set of Pace3D and 24 for
Kripke, for which we extrapolate procs to 256 and 512 and
dirsets to 96 and 128.
The results in Table 2 show that the accuracy remains
about the same when using only one execution as for the
models generated from the mean times of five repetitions.
This suggests that it is sufficient to execute the measurement
of each configuration point only once if the main calculation
is executed sufficiently often. For Pace3D, we set the number
of iterations of the main calculation to 100, and to 10 for
Kripke. However, even five iterations per application run
should be enough compared to one iteration per repetition
of the experiment. While the total time measured in the main
calculation stays the same for both variants, we can save the
initialization overhead by executing it more often in only
one execution.
Performance-Detective: Automatic Deduction of Performance Models ICS ’22, June 28–30, 2022, Virtual Event, USA
Experiment Design Modeling approach Perf-Taint? Cost Pace3D Mean error Pace3D Cost Kripke Mean error Kripke
core hours #experiments test set test set core hours #experiments test set test set
interpolated extrapolated interpolated extrapolated
Performance-Detective Extra-P ✓ 10.9 25 8.05 % 8.74 % 5.3 5 4.3 % 15.2 %
Full-factorial Extra-P ✓ 367.55 625 9.5 % 10.7 % 85.9 125 7.3 % 17.0 %
Full-factorial Extra-P – 367.55 625 6.2 % 6.3 % 85.9 125 6.7 % 11.2 %
Sparse Extra-P ✓ 5.54 80 8.9 % 17.7 % 8.8 50 24.6 % 34.2 %
Sparse Extra-P – 5.54 80 15.0 % 31.8 % 8.8 50 7.7 % 16.7 %
Performance-Detective Decision Trees ✓ 10.9 25 22.6 % 47.7 % 5.31 5 22.4 % 25.3 %
Plackett-Burman Decision Trees ✓ 164.37 245 20.1 % 45.4 % 35.66 50 15.8 % 35.1 %
Plackett-Burman Decision Trees – 164.37 245 27.0 % 44.2 % 35.66 50 21.8 % 31.2 %
Table 3: Mean error of different models for interpolated and extrapolated test points.
0
20
40
60
80
100
vol 324000 648000 12960002160000
cubes 12000 12000 12000 12000
(a) 4.41 6.32 9.80 10.92
(b)
(c)
4.78 7.09 10.37 11.04
7.38 6.75 7.00 5.68
(d)
(e)
26.00 19.65 10.71 0.84
6.48 1.00 14.37 31.39
(f) 24.75 28.56 33.25 26.69
(g)
(h)
23.13 25.04 31.21 25.28
34.16 31.52 33.96 24.10
interpolated
Model error [%]
216000 345600051840005184000
cost
108000 108000 1000 108000
12.11 11.92 7.85 12.11 2.9 %
18.66 16.09 6.47 15.38 100.0 %
4.92 3.58 1.36 3.11 100.0 %
36.61 16.40 30.37 7.10 1.5 %
9.92 18.11 89.37 34.98 1.5 %
74.52 60.64 51.57 67.12 2.9 %
74.65 60.56 51.08 67.05 44.7 %
73.33 57.52 48.40 64.51 44.7 %
extrapolated
0
20
40
60
80
100
procs 12 24 48 96
dirsets 48 28 20 12
(a) 17.93 0.25 0.88 3.04
(b)
(c)
5.27 7.66 5.65 0.32
4.67 8.67 7.88 7.81
(d)
(e)
19.49 28.50 26.87 22.17
9.57 9.78 1.64 1.21
(f) 30.14 23.38 10.83 19.06
(g)
(h)
15.81 10.25 7.78 27.93
25.18 12.91 9.85 45.68
interpolated
Model error [%]
16 16 256 512
cost
96 128 96 128
6.89 9.90 12.19 15.02 6.2 %
4.04 0.25 24.57 27.71 100.0 %
1.10 3.96 11.95 12.09 100.0 %
28.38 25.81 44.40 47.13 10.2 %
9.28 4.81 27.84 39.44 10.2 %
26.02 41.11 40.12 56.11 6.2 %
37.12 49.95 52.00 64.82 41.5 %
31.77 45.69 45.07 59.74 41.5 %
extrapolated
(a) Performance-Detective (Extra-P)
(b) [Cal16], Perf-Taint
(c) [Cal16]
(d) [Rit20], Perf-Taint
(e) [Rit20]
(f) Performance-Detective (Decision Trees)
(g) [Web21], Perf-Taint
(h) [Web21]
(a) Accuracy of models for the Pace3D case study
for excerpts of the interpolated and extrapolated
data set with 64 processes.
(b) Accuracy of models for the Kripke case study
for excerpts of the interpolated and extrapolated
data set.
Figure 8: Accuracy of different models for evaluation testing points.
6.3 RQ2: Varying independent parameters
simultaneously
The central question in our evaluation is whether we can
maintain accuracy of the model when generating it from a
minimal experiment design. To answer this, we compare the
accuracy of our Performance-Detective model to models cre-
ated with conventional experiment or sampling designs. For
the Extra-P multi-parameter modeler [9], this means using
a full-factorial setup with 125 measuring points for Pace3D
and 25 for Kripke. For the Extra-P sparse modeler [36], we
use 5 values for each parameter while keeping the others
constant and one interaction point for each parameter com-
bination, resulting in 16 points for Pace3D and 10 for Kripke.
To learn PIMs, we use the extended Plackett-Burman De-
sign [47] with 49 samples (Pace3D) or 10 samples (Kripke)
and five levels, effectively being a subset of the full-factorial
measuring points. We use the script of Weber et al. [48] to
learn PIMs on a function level based on known parameter
dependencies. However, we do not use their two-step process
but only one profiling step using compiler instrumentation.
In contrast to them, we use mean values derived from the
repetitions of the experiment for modeling. They build a
separate model for each repetition of the experiment.
As we showed in Section 6.2 that the accuracy of the mod-
els generated from a single application execution per mea-
surement point is comparable to the accuracy of the model
generated from repeating measurements five times for each
point, we will continue with these models for Performance-
Detective.
Table 3 shows the required number of measurements of
each approach and the cost in total core hours. The cost of
individual samples are not uniform across the configuration
space, as the impact the parameters have on performance can
mean some samples are orders of magnitude more expensive
than others. We therefore consider the total core hours to
be the more important metric. To evaluate the predictions’
ICS ’22, June 28–30, 2022, Virtual Event, USA Schmid et al.
accuracy, we use the same test configurations as in Section
6.2, inter- and extrapolating the training data points. Ta-
ble 3 depicts the mean error of the models from the different
approaches.
6.3.1 Interpolated test data. Figure 8 shows the comparison
of prediction error among models learned using Extra-P
(depicted as [Cal16] and [Rit20]) and Decision Trees (depicted
as [Web21]) both with and without the information about
dependencies from Perf-Taint. We compare our approach to
the full-factorial setup and the sparse modeling approach.
One would expect all approaches to be reasonably effec-
tive at predicting configuration options within the range of
measurements already available. In the case of Extra-P, using
the full set of measurements leads to accurate models, with
model errors seldom passing 10%, and the use of Perf-Taint
has a relatively small impact on results. The results of the
sparse modeler are worse overall, but for Pace3D, they are
significantly improved by the use of Perf-Taint.Performance-
Detective achieves results comparable to the full factorial
experiment design of Extra-P, while requiring only a frac-
tion (2.9% and 6.18%, respectively) of the cost.
The performance-influence models, however, show a con-
sistently higher model error rate of over 20%. When using
Performance-Detective, the quality of the models remains ap-
proximately the same, while the cost is reduced by a factor
of 15 for Pace3D and 7 for Kripke.
6.3.2 Extrapolated test data. Figure 8 shows an overview of
the accuracy of the different Extra-P (depicted as [Cal16] and
[Rit20]) and PIM models ([Web21]), both using information
about dependencies from Perf-Taint and without it.
When extrapolating, predicting the runtime is more chal-
lenging as any errors are quickly magnified. Overall, we ob-
serve the same trends as for interpolated evaluation points,
with a couple of differences. For Pace3D, the sampling ap-
proach of Extra-P generates significantly higher errors when
not used in conjunction with Perf-Taint. However, for Kripke,
the errors of the sampling approach are increased when used
together with Pef-Taint. The performance-influence models
show a larger model error rate, this time of over 40% for
Pace3D and over 30% for Kripke when using the Plackett-
Burman sampling design. However, the approach for learn-
ing performance-influence models does not extrapolate, as
we can see in our data: It only predicts the time measured
for the highest training value of the respective extrapolated
parameter.
While providing information about the dependencies de-
rived by Perf-Taint can be beneficial to the accuracy of the
sparse Extra-P modeler, it decreases accuracy for the full-
factorial experiment design. This is because with a full-facto-
rial design, there is a lot of measurement data and Perf-Taint
only removes wrong dependencies that model noise. The
accuracy of the sparse modeler increases because it has less
measurement data and very few information about parame-
ter interplay.
6.3.3 Discussion. For both case studies, we can maintain the
accuracy of the model generated with the experiment design
deduced by Performance-Detective (25 and 5 experiment exe-
cutions) as compared to a full-factorial design (625 and 125
experiment executions) and Plackett-Burman sampling (245
and 50 experiment executions). While for Pace3D, the sparse
modeler is even less expensive than Performance-Detective,
it has a lower accuracy than Performance-Detective (mean
accuracy of 85.9% and 74.9% with and without Perf-Taint as
compared to 91.5%) with especially worse predictions for
extrapolated test points (mean accuracy of 17.7% and 31.8%
with and without Perf-Taint compared to 9.3%).
For the PIMs, we observe a generally significant predic-
tion error in both evaluation sets, which does not change
much depending on using our minimal experiment design
or sampling, and usage of Perf-Taint. This indicates that the
learning method decision trees is not well-suited to model
the performance of these applications.
Across evaluation scenarios, we observe that Performance-
Detective always dramatically decreases the cost of experi-
ments, while not meaningfully degrading model quality.
6.4 Threats to validity
A threat to external validity is that we ran the taint analysis
on a small problem size. This was possible for the scenarios
shown because the calculations are the same and still repre-
sentative. However, this is not guaranteed in the general case,
as sometimes different branches can be active depending on
problem size. We can detect this by using control-flow taint-
ing, provided by Perf-Taint. The only impact this scenario
would have is that the taint analysis will have a higher cost.
Regarding internal validity, not inlining functions defined
as inline is a potential issue. Functions are not inlined if we
detect them as being performance-relevant. This could dis-
tort the measurements, as not inlining might incur a perfor-
mance penalty on the applications as a whole. The overhead
introduced by the profiling itself is a further source of dis-
tortion. We mitigate this by only instrumenting functions
detected as performance-relevant, and keep the overhead as
low as possible.
7 RELATED WORK
Parametric Performance Models. The difficulty of modeling
modern software systems with many parameters has been
addressed by parametric software performance models [ 5,
28, 32, 35] and Performance-Influence Models (PIMs) [39].
Performance-Detective: Automatic Deduction of Performance Models ICS ’22, June 28–30, 2022, Virtual Event, USA
The parametric software performance modelling approach
Palladio [5, 35] allows the modelling of parametric dependen-
cies. However, automated extraction of these models [28, 32]
assumes that the system consists of software components
with well-defined interfaces, which is usually not the case in
HPC applications.
In white-box PIMs, machine learning and heuristic meth-
ods are used to iteratively learn models representing the
influences and interactions of various application parame-
ters. However, white-box PIMs are either limited to binary-
encoded arguments or require expensive trace measurements
(cf. Sections 2.2, 4.2). Other methods of learning efficient per-
formance models for highly configurable systems use Fourier
transformations to reduce the number of samples and pa-
rameter combinations [18, 51].
HPC Performance Modeling. Analytical performance mod-
els can be created manually through source code inspection
and guidance by performance engineers [20, 25]. Unfortu-
nately, such models require significant effort, and excluding
empirical data makes the models prone to underestimate the
effects of hardware congestion and network performance.
Extra-P [12] is the state-of-the-art workflow for empirical
and parametric performance modeling, extended with multi-
parameter modeling [9], validation of high-performance li-
braries [38], prototyping hardware requirements for HPC
applications [11], and with white-box modeling [13, 14]. We
present a wider discussion in Sections 2.1 and 4.1.
PALM [43] constructs performance models from anno-
tated application source code and enhances them with user-
provided insights, program analysis, and measurements. As-
pen [41] is a domain-specific language for the manual speci-
fication of program operations. COMPASS [31] uses Aspen
annotations to generate application models statically, but
it requires user intervention when the kernels cannot be
analyzed automatically. In contrast, Performance-Detective
does not require user input and manual analysis steps.
Online learning improves the accuracy of static and com-
piler-based performance models [ 6, 7]. Machine learning
methods have been applied successfully to model perfor-
mance [22, 30, 42] and decrease the negative effects of mea-
surement noise [37]. Performance-Detective provides a com-
plete and white-box workflow that requires neither heuristics
nor approximations to construct performance models, and
we provide validated parametric dependencies of models.
Auto-tuning. Auto-tuning applies an optimization method
to achieve one or more goals, such as minimizing runtime [2]
or floating-point operations per second [44] of an applica-
tion. To find the best configuration among the search space,
global or local search methods are employed [3, 23, 33]. In
contrast to auto-tuning approaches that find the best config-
uration for a given optimization goal, Performance-Detective
enables efficient modeling of the whole configuration space
and interactions among configuration options.
8 CONCLUSION
We have shown that we can significantly lower the cost
of automatic performance modeling of applications with
multiple configuration parameters. We deduce a minimal
experiment design with Performance-Detective by exploiting
automatically derived insights about parameter interplay
and main loops and thus reduce the number and cost of re-
quired measurements while achieving comparable accuracy
to methods costing more than an order of magnitude more
compute hours. With Performance-Detective, we model the
Pace3D real-world multi-physics solver using 25 rather than
3125 measurements, require 34 times fewer core hours and
achieve and still maintain a model accuracy of 91.5% com-
pared to 93.8% and 90.6% when all measurements are used.
Furthermore, we model Kripke, reducing needed measure-
ments from 125 to 5, leading to 16 times fewer core hours
needed, while maintaining an accuracy of 89.2% compared
to 90.6% using all measurements.
ACKNOWLEDGMENTS
Larissa Schmid was supported by the Ministry of
Science, Research and the Arts Baden-Württemberg
(Az: 7712.14-0821-2). This work has received fund-
ing from the European Research Council (ERC) under the Eu-
ropean Union’s Horizon 2020 programme (grant agreement
PSAP, No. 101002047), and from the Schweizerische National-
fonds zur Förderung der wissenschaftlichen Forschung (SNF,
Swiss National Science Foundation) through Project 170415.
The authors acknowledge the financial support by the Fed-
eral Ministry for Economic Affairs and Energy of Germany
in the project ProStroM (project number 03ETB026C). This
work was supported by KASTEL Security Research Labs.
The authors acknowledge support by the state of Baden-
Württemberg through bwHPC.
REFERENCES
[1] A. J. Kunen, T. S. Bailey, and P. N. Brown. 2015. Kripke - a mas-
sively parallel transport mini-app. In Joint International Conference
on Mathematics and Computation (M&C), Supercomputing in Nuclear
Applications (SNA) and the Monte Carlo (MC) Method (ANS MC ’15)
(Nashville, Tennessee).
[2] Jason Ansel, Shoaib Kamil, Kalyan Veeramachaneni, Jonathan Ragan-
Kelley, Jeffrey Bosboom, Una-May O’Reilly, and Saman Amarasinghe.
2014. OpenTuner: An extensible framework for program autotuning.
In 2014 23rd International Conference on Parallel Architecture and Com-
pilation Techniques (PACT). 303–315. https://doi.org/10.1145/2628071.
2628092
[3] Prasanna Balaprakash, Stefan M. Wild, and Paul D. Hovland. 2013.
An Experimental Study of Global and Local Search Algorithms in
Empirical Performance Tuning. In High Performance Computing for
Computational Science - VECPAR 2012 , Michel Daydé, Osni Marques,
ICS ’22, June 28–30, 2022, Virtual Event, USA Schmid et al.
and Kengo Nakajima (Eds.). Springer Berlin Heidelberg, Berlin, Hei-
delberg, 261–269.
[4] Bradley J. Barnes, Barry Rountree, David K. Lowenthal, Jaxk Reeves,
Bronis de Supinski, and Martin Schulz. 2008. A Regression-Based
Approach to Scalability Prediction. In Proceedings of the 22nd Annual
International Conference on Supercomputing (Island of Kos, Greece)
(ICS ’08) . Association for Computing Machinery, New York, NY, USA,
368–377. https://doi.org/10.1145/1375527.1375580
[5] Steffen Becker, Heiko Koziolek, and Ralf Reussner. 2009. The Palladio
component model for model-driven performance prediction. Journal
of Systems and Software 82, 1 (2009), 3–22. https://doi.org/10.1016/j.
jss.2008.03.066 Special Issue: Software Performance - Modeling and
Analysis.
[6] Arnamoy Bhattacharyya and Torsten Hoefler. 2014. PEMOGEN: Auto-
matic Adaptive Performance Modeling During Program Runtime. In
Proc. of the 23rd International Conference on Parallel Architectures and
Compilation (Edmonton, AB, Canada) (PACT ’14). ACM, New York, NY,
USA, 393–404. https://doi.org/10.1145/2628071.2628100
[7] Arnamoy Bhattacharyya, Grzegorz Kwasniewski, and Torsten Hoefler.
2015. Using Compiler Techniques to Improve Automatic Performance
Modeling. In Proc. of the 24th International Conference on Parallel Ar-
chitectures and Compilation Techniques (PACT’15) (San Francisco, CA,
USA). San Francisco, CA, USA, 1–12.
[8] Christian Bischof, Dieter an Mey, and Christian Iwainsky. 2012. Brain-
ware for green HPC. 27, 4 (2012), 227–233. https://doi.org/10.1007/
s00450-011-0198-5
[9] Alexandru Calotoiu, David Beckinsale, Christopher W. Earl, Torsten
Hoefler, Ian Karlin, Martin Schulz, and Felix Wolf. 2016. Fast Multi-
parameter Performance Modeling. In 2016 IEEE International Confer-
ence on Cluster Computing (CLUSTER) (Taipei, Taiwan, 2016-09). IEEE,
172–181. https://doi.org/10.1109/CLUSTER.2016.57
[10] Alexandru Calotoiu, Marcin Copik, Torsten Hoefler, Marcus Ritter,
Sergei Shudler, and Felix Wolf. 2020. ExtraPeak: Advanced Au-
tomatic Performance Modeling for HPC Applications. In Software
for Exascale Computing - SPPEXA 2016-2019 (Cham) (Lecture Notes
in Computational Science and Engineering) , Hans-Joachim Bungartz,
Severin Reiz, Benjamin Uekermann, Philipp Neumann, and Wolf-
gang E. Nagel (Eds.). Springer International Publishing, 453–482.
https://doi.org/10.1007/978-3-030-47956-5_15
[11] Alexandru Calotoiu, Alexander Graf, Torsten Hoefler, Daniel Lorenz,
Sebastian Rinke, and Felix Wolf. 2018. Lightweight Requirements Engi-
neering for Exascale Co-design. In Proc. of the 2018 IEEE International
Conference on Cluster Computing (CLUSTER), Belfast, UK . IEEE, 1–11.
[12] Alexandru Calotoiu, Torsten Hoefler, Marius Poke, and Felix Wolf.
2013. Using automated performance modeling to find scalability bugs
in complex codes. In SC ’13: Proceedings of the International Conference
on High Performance Computing, Networking, Storage and Analysis
(2013-11). 1–12. https://doi.org/10.1145/2503210.2503277 ISSN: 2167-
4337.
[13] Marcin Copik, Alexandru Calotoiu, Tobias Grosser, Nicolas Wicki, Felix
Wolf, and Torsten Hoefler. 2021. Extracting Clean Performance Models
from Tainted Programs. In Proceedings of the 26th ACM SIGPLAN
Symposium on Principles and Practice of Parallel Programming (Virtual
Event, Republic of Korea) (PPoPP ’21) . Association for Computing
Machinery, New York, NY, USA, 403–417. https://doi.org/10.1145/
3437801.3441613
[14] Marcin Copik and Torsten Hoefler. 2019. perf-taint: Taint
Analysis for Automatic Many-Parameter Performance Modeling.
ACM Student Research Competition at ACM/IEEE Supercomputing
(2019). https://sc19.supercomputing.org/proceedings/src_poster/src_
poster_pages/spostg110.html
[15] dfsan 2019. Clang 9 Documentation - DataFlowSanitizer. https://clang.
llvm.org/docs/DataFlowSanitizer.html.
[16] Dan Gohman. 2009. ScalarEvolution and Loop Optimization. Talk at
LLVM Developer’s Meeting.
[17] Alexander Grebhahn, Norbert Siegmund, and Sven Apel. 2019. Pre-
dicting Performance of Software Configurations: There is no Silver
Bullet. (2019). arXiv:1911.12643 http://arxiv.org/abs/1911.12643
[18] Huong Ha and Hongyu Zhang. 2019. Performance-Influence Model for
Highly Configurable Software with Fourier Learning and Lasso Regres-
sion. In2019 IEEE International Conference on Software Maintenance and
Evolution (ICSME). 470–480. https://doi.org/10.1109/ICSME.2019.00080
[19] Xue Han and Tingting Yu. 2016. An Empirical Study on Performance
Bugs for Highly Configurable Software Systems. In Proceedings of the
10th ACM/IEEE International Symposium on Empirical Software Engi-
neering and Measurement (Ciudad Real, Spain) (ESEM ’16). Association
for Computing Machinery, New York, NY, USA, Article 23, 10 pages.
https://doi.org/10.1145/2961111.2962602
[20] Torsten Hoefler, William Gropp, William Kramer, and Marc Snir. 2011.
Performance modeling for systematic performance tuning. In SC’11:
Proceedings of 2011 International Conference for High Performance Com-
puting, Networking, Storage and Analysis . IEEE, 1–12.
[21] J. Hötzer, A. Reiter, H. Hierl, P. Steinmetz, M. Selzer, and Britta
Nestler. 2018. The parallel multi-physics phase-field framework
Pace3D. Journal of Computational Science 26 (2018), 1–12. https:
//doi.org/10.1016/j.jocs.2018.02.011
[22] Engin Ipek, Bronis R. de Supinski, Martin Schulz, and Sally A. McKee.
2005. An Approach to Performance Prediction for Parallel Applications.
In Proceedings of the 11th International Euro-Par Conference on Parallel
Processing (Lisbon, Portugal) (Euro-Par’05). Springer-Verlag, Berlin,
Heidelberg, 196–205.
[23] Herbert Jordan, Peter Thoman, Juan J. Durillo, Simone Pellegrini,
Philipp Gschwandtner, Thomas Fahringer, and Hans Moritsch. 2012.
A multi-objective auto-tuning framework for parallel codes. In SC
’12: Proceedings of the International Conference on High Performance
Computing, Networking, Storage and Analysis . 1–12. https://doi.org/
10.1109/SC.2012.7
[24] Christian Kaltenecker, Alexander Grebhahn, Norbert Siegmund, and
Sven Apel. 2020. The Interplay of Sampling and Machine Learning for
Software Performance Prediction. IEEE Software 37, 4 (2020), 58–66.
https://doi.org/10.1109/MS.2020.2987024
[25] Darren J Kerbyson, Henry J Alme, Adolfy Hoisie, Fabrizio Petrini,
Harvey J Wasserman, and Mike Gittings. 2001. Predictive performance
and scalability modeling of a large-scale application. In Proceedings of
the 2001 ACM/IEEE conference on Supercomputing . 37–37.
[26] Andreas Knüpfer, Christian Rössel, Dieter an Mey, Scott Biersdorff,
Kai Diethelm, Dominic Eschweiler, Markus Geimer, Michael Gerndt,
Daniel Lorenz, Allen Malony, Wolfgang E. Nagel, Yury Oleynik, Pe-
ter Philippen, Pavel Saviankou, Dirk Schmidl, Sameer Shende, Ronny
Tschüter, Michael Wagner, Bert Wesarg, and Felix Wolf. 2012. Score-
P: A Joint Performance Measurement Run-Time Infrastructure for
Periscope,Scalasca, TAU, and Vampir. InTools for High Performance
Computing 2011, Holger Brunst, Matthias S. Müller, Wolfgang E. Nagel,
and Michael M. Resch (Eds.). Springer Berlin Heidelberg, Berlin, Hei-
delberg, 79–91.
[27] Sergiy Kolesnikov, Norbert Siegmund, Christian Kästner, Alexander
Grebhahn, and Sven Apel. 2019. Tradeoffs in modeling performance
of highly configurable software systems. 18, 3 (2019), 2265–2283.
https://doi.org/10.1007/s10270-018-0662-9
[28] Klaus Krogmann, Michael Kuperberg, and Ralf Reussner. 2010. Using
Genetic Search for Reverse Engineering of Parametric Behavior Models
for Performance Prediction. IEEE Transactions on Software Engineering
36, 6 (2010), 865–877. https://doi.org/10.1109/TSE.2010.69
Performance-Detective: Automatic Deduction of Performance Models ICS ’22, June 28–30, 2022, Virtual Event, USA
[29] C. Lattner and V. Adve. 2004. LLVM: A Compilation Framework for
Lifelong Program Analysis & Transformation. In Proc. of the Inter-
national Symposium on Code Generation and Optimization: Feedback-
directed and Runtime Optimization (Palo Alto, California) (CGO ’04) .
IEEE Computer Society, Washington, DC, USA.
[30] Benjamin C. Lee, David M. Brooks, Bronis R. de Supinski, Martin
Schulz, Karan Singh, and Sally A. McKee. 2007. Methods of inference
and learning for performance modeling of parallel applications. In
Proc. of the 12th ACM SIGPLAN Symposium on Principles and Practice of
Parallel Programming (San Jose, California, USA) ((PPoPP ’07)) . ACM,
249–258.
[31] Seyong Lee, Jeremy S Meredith, and Jeffrey S Vetter. 2015. Compass:
A framework for automated performance modeling and prediction. In
Proceedings of the 29th ACM on International Conference on Supercom-
puting. 405–414.
[32] Manar Mazkatli, David Monschein, Johannes Grohmann, and Anne
Koziolek. 2020. Incremental Calibration of Architectural Performance
Models with Parametric Dependencies. In IEEE International Con-
ference on Software Architecture (ICSA 2020) . Salvador, Brazil, 23–34.
https://doi.org/10.1109/ICSA47634.2020.00011
[33] Philip Pfaffe, Martin Tillmann, Sigmar Walter, and Walter F. Tichy. 2017.
Online-Autotuning in the Presence of Algorithmic Choice. In2017 IEEE
International Parallel and Distributed Processing Symposium Workshops
(IPDPSW). 1379–1388. https://doi.org/10.1109/IPDPSW.2017.28
[34] Elnatan Reisner, Charles Song, Kin-Keung Ma, Jeffrey S. Foster, and
Adam Porter. 2010. Using symbolic evaluation to understand behavior
in configurable software systems. In Proceedings of the 32nd ACM/IEEE
International Conference on Software Engineering - ICSE ’10 (Cape Town,
South Africa), Vol. 1. ACM Press, 445. https://doi.org/10.1145/1806799.
1806864
[35] Ralf H. Reussner, Steffen Becker, Jens Happe, Robert Heinrich, Anne
Koziolek, Heiko Koziolek, Max Kramer, and Klaus Krogmann. 2016.
Modeling and Simulating Software Architectures – The Palladio Ap-
proach. MIT Press, Cambridge, MA. http://mitpress.mit.edu/books/
modeling-and-simulating-software-architectures
[36] Marcus Ritter, Alexandru Calotoiu, Sebastian Rinke, Thorsten Reimann,
Torsten Hoefler, and Felix Wolf. 2020. Learning Cost-Effective Sam-
pling Strategies for Empirical Performance Modeling. In 2020 IEEE
International Parallel and Distributed Processing Symposium (IPDPS)
(New Orleans, LA, USA, 2020-05). IEEE, 884–895. https://doi.org/10.
1109/IPDPS47924.2020.00095
[37] Marcus Ritter, Alexander Geiß, Johannes Wehrstein, Alexandru Calo-
toiu, Thorsten Reimann, Torsten Hoefler, and Felix Wolf. 2021. Noise-
Resilient Empirical Performance Modeling with Deep Neural Networks.
In 2021 IEEE International Parallel and Distributed Processing Sympo-
sium (IPDPS) . 23–34. https://doi.org/10.1109/IPDPS49936.2021.00012
[38] Sergei Shudler, Alexandru Calotoiu, Torsten Hoefler, Alexandre Strube,
and Felix Wolf. 2015. Exascaling Your Library: Will Your Implemen-
tation Meet Your Expectations?. In Proceedings of the 29th ACM on
International Conference on Supercomputing (Newport Beach, Califor-
nia, USA) (ICS ’15) . Association for Computing Machinery, New York,
NY, USA, 165–175. https://doi.org/10.1145/2751205.2751216
[39] Norbert Siegmund, Alexander Grebhahn, Sven Apel, and Christian
Kästner. 2015. Performance-influence models for highly configurable
systems. In Proceedings of the 2015 10th Joint Meeting on Foundations
of Software Engineering - ESEC/FSE 2015 (Bergamo, Italy). ACM Press,
284–294. https://doi.org/10.1145/2786805.2786845
[40] Connie U. Smith. 1993. Software performance engineering. In Perfor-
mance Evaluation of Computer and Communication Systems , Lorenzo
Donatiello and Randolph Nelson (Eds.). Springer Berlin Heidelberg,
Berlin, Heidelberg, 509–536.
[41] K. L. Spafford and J. S. Vetter. 2012. Aspen: A Domain Specific Language
for Performance Modeling. In Proc. of the International Conference on
High Performance Computing, Networking, Storage and Analysis (Salt
Lake City, Utah) (SC ’12) . IEEE Computer Society Press, Los Alamitos,
CA, USA, Article 84, 11 pages.
[42] Jingwei Sun, Guangzhong Sun, Shiyan Zhan, Jiepeng Zhang, and Yong
Chen. 2020. Automated Performance Modeling of HPC Applications
Using Machine Learning. IEEE Trans. Comput. 69, 5 (2020), 749–763.
https://doi.org/10.1109/TC.2020.2964767
[43] N. R. Tallent and A. Hoisie. 2014. Palm: Easing the Burden of Analytical
Performance Modeling. In Proc. of the 28th ACM International Confer-
ence on Supercomputing (Munich, Germany) (ICS ’14). ACM, New York,
NY, USA, 221–230. https://doi.org/10.1145/2597652.2597683
[44] Sébastien Varrette, Frédéric Pinel, Emmanuel Kieffer, Grégoire Danoy,
and Pascal Bouvry. 2020. Automatic Software Tuning of Parallel Pro-
grams for Energy-Aware Executions. InParallel Processing and Applied
Mathematics, Roman Wyrzykowski, Ewa Deelman, Jack Dongarra, and
Konrad Karczewski (Eds.). Springer International Publishing, Cham,
144–155.
[45] Miguel Velez, Pooyan Jamshidi, Florian Sattler, Norbert Siegmund,
Sven Apel, and Christian Kästner. 2020. ConfigCrusher: Towards
White-Box Performance Analysis for Configurable Systems. Auto-
mated Software Engineering 27, 3 (2020), 265–300. https://doi.org/10.
1007/s10515-020-00273-8
[46] Miguel Velez, Pooyan Jamshidi, Norbert Siegmund, Sven Apel, and
Christian Kästner. 2021. White-box analysis over machine learning:
Modeling performance of configurable systems. In 2021 IEEE/ACM
43rd International Conference on Software Engineering (ICSE) . IEEE,
1072–1084.
[47] JC Wang and CF Jeff Wu. 1995. A hidden projection property of
Plackett-Burman and related designs. Statistica Sinica (1995), 235–250.
[48] Max Weber, Sven Apel, and Norbert Siegmund. 2021. White-Box
Performance-Influence Models: A Profiling and Learning Approach. In
2021 IEEE/ACM 43rd International Conference on Software Engineering
(ICSE) (Madrid, Spain, 2021-05). IEEE, 1059–1071. https://doi.org/10.
1109/ICSE43902.2021.00099
[49] Nicolas Wicki. 2020. Control Flow Taint Analysis for Performance
Modeling in LLVM. Bachelor’s Thesis.
[50] Murray Woodside, Greg Franks, and Dorina C. Petriu. 2007. The
Future of Software Performance Engineering. In Future of Software
Engineering (FOSE ’07) (Minneapolis, MN, USA, 2007-05). IEEE, 171–
187. https://doi.org/10.1109/FOSE.2007.32
[51] Yi Zhang, Jianmei Guo, Eric Blais, and Krzysztof Czarnecki. 2015. Per-
formance Prediction of Configurable Software Systems by Fourier
Learning (T). In 2015 30th IEEE/ACM International Conference on
Automated Software Engineering (ASE) (2015-11). 365–373. https:
//doi.org/10.1109/ASE.2015.15

Halide: A Language and Compiler for Optimizing Parallelism,
Locality, and Recomputation in Image Processing Pipelines
Jonathan Ragan-Kelley
MIT CSAIL
Connelly Barnes
Adobe
Andrew Adams
MIT CSAIL
Sylvain Paris
Adobe
Frédo Durand
MIT CSAIL
Saman Amarasinghe
MIT CSAIL
Abstract
Image processing pipelines combine the challenges of stencil com-
putations and stream programs. They are composed of large graphs
of different stencil stages, as well as complex reductions, and stages
with global or data-dependent access patterns. Because of their
complex structure, the performance difference between a naive im-
plementation of a pipeline and an optimized one is often an order of
magnitude. Efﬁcient implementations require optimization of both
parallelism and locality, but due to the nature of stencils, there is a
fundamental tension between parallelism, locality, and introducing
redundant recomputation of shared values.
We present a systematic model of the tradeoff space fundamen-
tal to stencil pipelines, a schedule representation which describes
concrete points in this space for each stage in an image processing
pipeline, and an optimizing compiler for the Halide image process-
ing language that synthesizes high performance implementations
from a Halide algorithm and a schedule. Combining this compiler
with stochastic search over the space of schedules enables terse,
composable programs to achieve state-of-the-art performance on a
wide range of real image processing pipelines, and across different
hardware architectures, including multicores with SIMD, and hetero-
geneous CPU+GPU execution. From simple Halide programs writ-
ten in a few hours, we demonstrate performance up to5×faster than
hand-tuned C, intrinsics, and CUDA implementations optimized by
experts over weeks or months, for image processing applications
beyond the reach of past automatic compilers.
Categories and Subject Descriptors D.3.4 [ Programming Lan-
guages]: Processors – compilers, optimization, code generation
Keywords domain speciﬁc language; compiler; image processing;
locality; parallelism; redundant computation; optimization; GPU;
vectorization
1. Introduction
Image processing pipelines are everywhere, and are essential to
capturing, analyzing, mining, and rendering the rivers of visual
information gathered by our countless cameras and imaging-based
sensors. Applications from raw processing, to object detection and
recognition, to Microsoft’s Kinect, to Instagram and Photoshop, to
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proﬁt or commercial advantage and that copies bear this notice and the full citation
on the ﬁrst page. To copy otherwise, to republish, to post on servers or to redistribute
to lists, requires prior speciﬁc permission and/or a fee.
PLDI ’13 June 16-21, Seattle.
Copyright c⃝2013 ACM 978-1-4503-2014-6/13/06. . . $15.00
medical imaging and neural scanning all demand extremely high
performance to cope with the rapidly rising resolution and frame rate
of image sensors and the increasing complexity of algorithms. At the
same time, the shrinking cameras and mobile devices on which they
run require extremely high efﬁciency to last more than a few minutes
on battery power. While power-hungry radios and video CODECs
can implement slow-changing standards in custom hardware, image
processing pipelines are rapidly evolving and diverse, requiring high
performance software implementations.
Image processing pipelines combine the challenges of stencil
computation and stream programs. They are composed of large
graphs of many different operations, most of which are stencil
computations. These pipelines are simultaneously wide and deep:
each stage exhibits data parallelism across the many pixels which
it must process, and whole pipelines consist of long sequences
of different operations, which individually have low arithmetic
intensity (the ratio of computation performed to data read from prior
stages and written to later stages). For example, an implementation
of one recent algorithm, local Laplacian ﬁlters [3, 22], is a graph of
99 different stages (Fig. 1), including many different stencils and a
large data-dependent resampling.
As a result of this structure, the performance difference between
a naive implementation of a given pipeline and a highly optimized
one is frequently an order of magnitude or more. With current tools,
often the only way to approach peak performance is by hand-writing
parallel, vectorized, tiled, and globally fused code in low-level C,
CUDA, intrinsics, and assembly. Simple pipelines become hun-
dreds of lines of intricately interleaved code; complex pipelines, like
Adobe’s Camera Raw engine, become hundreds of thousands. Tun-
ing them requires immense effort by expert programmers, and the
end result is not portable to different architectures, nor composable
with other algorithms, without sacriﬁcing much of this painstakingly
earned performance. Libraries of optimized subroutines do not solve
the problem, either, since many critical optimizations involve fusion
for producer-consumer locality across stages.
We address this challenge by raising the level of abstraction, and
decoupling the algorithm deﬁnition from its execution strategy, to
improve portability and composability, while automating the search
for optimized mappings of the resulting pipelines to parallel ma-
chines and complex memory hierarchies. Effective abstraction and
automatic optimization enable radically simpler programs to achieve
higher performance than hand-tuned expert implementations, while
running across a wide range of architectures.
1.1 Image Processing Pipelines
Stencils have been well studied in scientiﬁc applications in the form
of iterated stencil computations, where one or a few small stencils
are applied to the same grid over many iterations [ 10, 16, 19]. In
contrast, we are interested in other applications, in image processing
DOWN
DOWN
DOWN
LUT
LUT: look-up table
   O(x,y,k) ← lut(I(x,y) − kσ)
DDA: data-dependent access
   k ← floor(I
1(x,y) / σ)
   α ← (I1(x,y) / σ) − k
   O(x,y) ← (1−α) I2(x,y,k) + α I2(x,y,k+1)
DDA
DDA
ADD: addition
   O(x,y) ← I1(x,y) + I2(x,y) 
DOWN
DOWN: downsample
   T1 ← I ⊗x [1 3 3 1]
   T2 ← T1 ⊗y [1 3 3 1]
   O(x,y) ← T2(2x,2y)
UP: upsample
   T
1(2x,2y) ← I(x,y)
   T2 ← T1 ⊗x [1 3 3 1]
   O ← T2 ⊗y [1 3 3 1]
UP
UP
SUB
SUB: subtraction
   O(x,y) ← I
1(x,y) − I2(x,y)
COPY COPY
SUB
UP
DDA
COPYCOPY
UP
ADD
ADD
...
...
...
...
...
The algorithm uses 8 pyramid levels
level size
w × h
w × h
2 2
w  ×   h
128 128
Figure 1. Imaging pipelines employ large numbers of interconnected, heterogeneous stages. Here we show the structure of the local Laplacian
ﬁlter [ 3, 22], which is used for a variety of tasks in photographic post-production. Each box represents intermediate data, and each arrow
represents one or more functions that deﬁne that data. The pipeline includes horizontal and vertical stencils, resampling, data-dependent
gathers, and simple pointwise functions.
and computer graphics, where stencils are common, but often in a
very different form: stencil pipelines. Stencil pipelines are graphs of
different stencil computations. Iteration of the same stencil occurs,
but it is the exception, not the rule; most stages apply their stencil
only once before passing data to the next stage, which performs
different data parallel computation over a different stencil.
Graph-structured programs have been studied in the context
of streaming languages [ 4, 11, 29]. Static communication analy-
sis allows stream compilers to simultaneously optimize for data
parallelism and producer-consumer locality by interleaving compu-
tation and communication between kernels. However, most stream
compilation research has focussed on 1D streams, where sliding win-
dow communication allows 1D stencil patterns. Image processing
pipelines can be thought of as programs on 2D and 3D streams and
stencils. The model of computation required by image processing is
also more general than stencils, alone. While most stages are point
or stencil operations over the results of prior stages, some stages
gather from arbitrary data-dependent addresses, while others scatter
to arbitrary addresses to compute operations like histograms.
Pipelines of simple map operations can be optimized by tradi-
tional loop fusion: merging multiple successive operations on each
point into a single compound operation improves arithmetic intensity
by maximizing producer-consumer locality, keeping intermediate
data values in fast local memory (caches or registers) as it ﬂows
through the pipeline. But traditional loop fusion does not apply to
stencil operations, where neighboring points in a consumer stage
depend on overlapping regions of a producer stage. Instead, sten-
cils require a complex tradeoff between producer-consumer locality,
synchronization, and redundant computation. Because this tradeoff
is made by interleaving the order of allocation, execution, and com-
munication of each stage, we call it the pipeline’sschedule. These
tradeoffs exist in scheduling individual iterated stencil computations
in scientiﬁc applications, and the complexity of the choice space
is reﬂected by the many different tiling and scheduling strategies
introduced in past work [10, 16, 19]. In image processing pipelines,
this tradeoff must be made for each producer-consumer relationship
between stages in the graph—often dozens or hundreds—and the
ideal schedule depends on the global interaction among every stage,
often requiring the composition of many different strategies.
1.2 Contributions
Halide is an open-source domain-speciﬁc language for the complex
image processing pipelines found in modern computational pho-
tography and vision applications [26]. In this paper, we present the
optimizing compiler for this language. We introduce:
• a systematic model of the tradeoffs between locality, parallelism,
and redundant recomputation in stencil pipelines;
• a scheduling representation that spans this space of choices;
• a DSL compiler based on this representation that combines
Halide programs and schedule descriptions to synthesize points
anywhere in this space, using a design where the choices forhow
to execute a program are separated not just from the deﬁnition
of what to compute, but are pulled all the way outside the black
box of the compiler;
• a loop synthesizer for data parallel pipelines based on simple
interval analysis, which is simpler and less expressive than
polyhedral model, but more general in the class of expressions
it can analyze;
• a code generator that produces high quality vector code for
image processing pipelines, using machinery much simpler than
the polyhedral model;
• and an autotuner that can infer high performance schedules—up
to 5×faster than hand-optimized programs written by experts—
for complex image processing pipelines using stochastic search.
Our scheduling representation composably models a range of
tradeoffs between locality, parallelism, and avoiding redundant
work. It can naturally express most prior stencil optimizations,
as well as hierarchical combinations of them. Unlike prior stencil
code generation systems, it does not describe just a single stencil
scheduling strategy, but separately treats every producer-consumer
edge in a graph of stencil and other image processing computations.
Our split representation, which separates schedules from the
underlying algorithm, combined with the inside-out design of
our compiler, allows our compiler to automatically search for the
best schedule. The space of possible schedules is enormous, with
hundreds of inter-dependent dimensions. It is too high dimensional
for the polyhedral optimization or exhaustive parameter search
employed by existing stencil compilers and autotuners. However,
we show that it is possible to discover high quality schedules using
stochastic search.
Given a schedule, our compiler automatically synthesizes high
quality parallel vector code for x86 and ARM CPUs with SSE/A VX
and NEON, and graphs of CUDA kernels interwoven with host
management code for hybrid GPU execution. It automatically infers
all internal allocations and a complete loop nest using simple
but general interval analysis [ 18]. Directly mapping data parallel
dimensions to SIMD execution, including careful treatment of
strided access patterns, enables high quality vector code generation,
without requiring any general-purpose loop auto-vectorization.
The end result is a system which enables terse, composable
programs to achieve state-of-the-art performance on a wide range
of real image processing pipelines, and across different hardware
architectures, including multicores with SIMD, and heterogeneous
CPU+GPU execution. From simple Halide programs written in a
few hours, we demonstrate performance up to 5×faster than hand-
tuned C, intrinsics, and CUDA implementations written by experts
over weeks or months, for image processing applications beyond
the reach of past automatic compilers.
2. The Halide DSL
We use the Halide DSL to describe image processing pipelines
in a simple functional style [ 26]. A simple C++ implementation
of local Laplacian ﬁlters (Fig. 1) is described by dozens of loop
nests and hundreds of lines of code. This is not practical to globally
optimize with traditional loop optimization systems. The Halide
version distills this into 62 lines describing just the essential dataﬂow
and computation in the 99 stage pipeline, and all choices for how
the program should be synthesized are described separately (Sec. 3).
In Halide, values that would be mutable arrays in an impera-
tive language are instead functions from coordinates to values. It
represents images as pure functions deﬁned over an inﬁnite inte-
ger domain, where the value of a function at a point represents the
color of the corresponding pixel. Pipelines are speciﬁed as chains
of functions. Functions may either be simple expressions in their
arguments, or reductions over a bounded domain. The expressions
that deﬁne functions are side-effect free, and are much like those in
any simple functional language, including:
• Arithmetic and logical operations;
• Loads from external images;
• If-then-else expressions;
• References to named values (which may be function arguments,
or expressions deﬁned by a functional let construct);
• Calls to other functions, including external C ABI functions.
For example, a separable 3 ×3 unnormalized box ﬁlter is expressed
as a chain of two functions in x, y:
UniformImage in(UInt(8), 2)
Var x, y
Func blurx(x,y) = in(x-1,y) + in(x,y) + in(x+1,y)
Func out(x,y) = blurx(x,y-1) + blurx(x,y) + blurx(x,y+1)
This representation is simpler than most functional languages.
It does not include higher-order functions, dynamic recursion, or
additional data structures like lists. Functions simply map from
integer coordinates to a scalar result. Constrained versions of more
advanced features such as higher-order functions are added as
syntactic sugar, but do not change the underlying representation.
This representation is sufﬁcient to describe a wide range of image
processing algorithms, and these constraints enable ﬂexible analysis
and transformation of algorithms during compilation. Critically,
this representation is naturally data parallel within the domain of
each function. Also, since functions are deﬁned over an inﬁnite
domain, boundary conditions can be handled safely and efﬁciently
by computing arbitraryguard bandsof extra values as needed. Guard
bands are a common pattern in image processing code, both for
performance concerns like alignment, and for safety. Wherever
speciﬁc boundary conditions matter to the meaning of an algorithm,
the function may deﬁne its own.
Reduction functions. In order to express operations like his-
tograms and general convolutions, Halide also needs a way to ex-
press iterative or recursive computations, like summation, histogram,
and scan. Reductions are deﬁned in two parts:
• An initial value function, which speciﬁes a value at each point
in the output domain.
• A recursive reduction function, which redeﬁnes the value at
points given by anoutput coordinate expressionin terms of prior
values of the function.
Unlike a pure function, the meaning of a reduction depends on the
order in which the reduction function is applied. The programmer
speciﬁes the order by deﬁning a reduction domain, bounded by
minimum and maximum expressions for each dimension. The value
at each point in the output domain is deﬁned by the ﬁnal value of
the reduction function at that point, after recursing in lexicographic
order across the reduction domain.
This pattern can describe a range of algorithms outside the scope
of traditional stencil computation, but essential to image processig
pipelines, in a way that bounds side effects. For example, histogram
equalization combines multiple reductions and a data-dependent
gather. A scattering reduction computes a histogram, a recursive
scan integrates it into a CDF, and a point-wise operation remaps the
input using the CDF:
UniformImage in(UInt(8), 2)
RDom r(0..in.width(), 0..in.height()), ri(0..255)
Var x, y, i
Func histogram(i) = 0; histogram(in(r.x, r.y))++
Func cdf(i) = 0; cdf(ri) = cdf(ri-1) + histogram(ri)
Func out(x, y) = cdf(in(x, y))
The iteration bounds for the reduction and scan are expressed by the
programmer using explicit reduction domains (RDoms).
3. Scheduling Image Processing Pipelines
Halide’s representation of image processing algorithms avoids
imposing constraints on the order of execution and placement of
data. Values need to be computed before they can be used, to respect
the fundamental dependencies in the algorithm, but many choices
remain unspeciﬁed:
• When and where should the value at each coordinate in each
function be computed?
• Where should they be stored?
• How long are values cached and communicated across multiple
consumers, and when are they independently recomputed by
each?
These choices can not change the meaning or results of the algo-
rithm, but they are essential to the performance of the resulting
implementation. We call a speciﬁc set of choices for when and
where values are computed the pipeline’sschedule.
In the presence of stencil access patterns, these choices are bound
by a fundamental tension between producer-consumer locality,
parallelism, and redundant recomputation of shared values. To
understand this tradeoff space, it is useful to look at an example.
3.1 Motivation: Scheduling a Two-Stage Pipeline
Consider the simple two-stage blur algorithm, which computes a
3×3 box ﬁlter as two 3×1 passes. The ﬁrst stage, blurx, computes
a horizontal blur of the input by averaging over a 3 ×1 window:
blurx(x,y) = in(x-1,y) + in(x,y) + in(x+1,y)
The second stage,out, computes the ﬁnal isotropic blur by averaging
a 1 ×3 window of the output from the ﬁrst stage:
out(x,y) = blurx(x,y-1) + blurx(x,y) + blurx(x,y+1)
A natural way to think about scheduling the pipeline is from the
perspective of the output stage: how should it compute its input?
There are three obvious choices for this pipeline.
valid
schedules
storage
granularity
compute
granularity
coarse storage granularity
no redundant computation
fine storage granularity
redundant computation
coarse interleaving
low locality
fine interleaving
high locality
high parallelism
low locality
more parallelism
less synchronization
breadth first
sliding
window
overlapping tiles
loop fusion
sliding windows
within tiles
Figure 2.A natural way to visualize the space of scheduling choices
is by granularity of storage (x-axis), and granularity of computation
(y-axis). Breadth-ﬁrst execution does coarse-grain computation into
coarse-grain storage. Total fusion performs ﬁne-grain computation
into ﬁne-grain storage (small temporary buffers). Sliding window
strategies allocate enough space for the entire intermediate stage,
but compute it in in ﬁne-grain chunks as late as possible. These
extremes each have their pitfalls. Breadth-ﬁrst execution has poor
locality, total fusion often does redundant work, and using sliding
windows to avoid redundant recomputation constrains parallelism by
introducing dependencies across loop iterations. The best strategies
tend to be mixed, and lie somewhere in the middle of the space.
First, it could compute and store every required point in blurx
before evaluating any points in out. Applied to a 6 megapixel
(3k ×2k) image, this is equivalent to the loop nest:
alloc blurx[2048][3072]
for each y in 0..2048:
for each x in 0..3072:
blurx[y][x] = in[y][x-1] + in[y][x] + in[y][x+1]
alloc out[2046][3072]
for each y in 1..2047:
for each x in 0..3072:
out[y][x]=blurx[y-1][x] + blurx[y][x] + blurx[y+1][x]
This is the most common strategy in hand-written pipelines, and
what results from composing library routines together: each stage
executes breadth-ﬁrst across its input before passing its entire output
to the next stage. There is abundant parallelism, since all the required
points in each stage can be computed and stored independently, but
there is little producer-consumer locality, since all the values of
blurx must be computed and stored before the ﬁrst one is used by
out.
At the other extreme, the out stage could compute each point in
blurx immediately before the point which uses it. This opens up a
further choice: should points in blurx which are used by multiple
points in out be stored and reused, or recomputed independently by
each consumer?
Interleaving the two stages, without storing the intermediate
results across uses, is equivalent to the loop nest:
alloc out[2046][3072]
for each y in 1..2047:
for each x in 0..3072:
alloc blurx[-1..1]
for each i in -1..1:
blurx[i]= in[y-1+i][x-1]+in[y-1+i][x]+in[y-1+i][x+1]
out[y][x] = blurx[0] + blurx[1] + blurx[2]
Each pixel can be computed independently, providing the same
abundant data parallelism from the breadth-ﬁrst strategy. The dis-
tance from producer to consumer is small, maximizing locality. But
because shared values in blurx are not reused across iterations, this
strategy performs redundant work. This can be seen as the result of
Strategy Span (iterations)Max. reuse dist. (ops)Work ampl.
Breadth-ﬁrst≥3072×2046 3072×2048×3 1.0×
Full fusion ≥3072×2046 3×3 2.0×
Sliding window3072 3072×(3 + 3) 1.0×
Tiled ≥3072×2046 34×32×3 1.0625×
Sliding in tiles≥3072×2048/8 3072×(3 + 3) 1.25×
Figure 3. Different points in the choice space in Figure 2 each
make different trade-offs between locality, redundant recomputation,
and parallelism. Here we quantify these effects for our two-stage
blur pipeline. The span measures the degree of parallelism available,
by counting how many threads or simd lanes could be kept busy
doing useful work. The Max. reuse distance measures locality, by
counting the maximum number of operations that can occur between
computing a value and reading it back. Work ampliﬁcation measures
redundant work, by comparing the number of arithmetic operations
done to the breadth-ﬁrst case. Each of the ﬁrst three strategies
represent an extreme point of the choice space, and is weak in
one regard. The fastest schedules are mixed strategies, such as the
tiled ones in the last two rows.
applying classical loop fusion through a stencil dependence pattern:
the body of the ﬁrst loop is moved into the second loop, but its work
is ampliﬁed by the size of the stencil.
The two stages can also be interleaved while storing the values
of blurx across uses:
alloc out[2046][3072]
alloc blurx[3][3072]
for each y in -1..2047:
for each x in 0..3072:
blurx[(y+1)%3][x]=in[y+1][x-1]+in[y+1][x]+in[y+1][x+1]
if y < 1: continue
out[y][x] = blurx[(y-1)%3][x]
+ blurx[ y % 3 ][x]
+ blurx[(y+1)%3][x]
This interleaves the computation over a sliding window, with out
trailing blurx by the stencil radius (one scanline). It wastes no work,
computing each point in blurx exactly once, and the maximum
distance between a value being produced in blurx and consumed
in out is proportional to the stencil height (three scanlines), not the
entire image. But to achieve this, it has introduced a dependence
between the loop iterations: a given iteration of out depends on
the last three outer loop iterations of blurx. This only works if
these loops are evaluated sequentially. Interleaving the stages while
producing each value only once requires tightly synchronizing the
order of computation, sacriﬁcing parallelism.
Each of these strategies has a major pitfall: lost locality, redun-
dant work, or limited parallelism (Fig. 3). In practice, the right
choice for a given pipeline is almost always somewhere in between
these extremes. For our two-stage example, a better balance can be
struck by interleaving the computation of blurx and out at the level
of tiles:
alloc out[2046][3072]
for each ty in 0..2048/32:
for each tx in 0..3072/32:
alloc blurx[-1..33][32]
for y in -1..33:
for x in 0..32:
blurx[y][x] = in[ty*32+y][tx*32+x-1]
+ in[ty*32+y][tx*32+x]
+ in[ty*32+y][tx*32+x+1]
for y in 0..32:
for x in 0..32:
out[ty*32+y][tx*32+x] = blurx[y-1][x]
+ blurx[y ][x]
+ blurx[y+1][x]
This trades off a small amount of redundant computation on tile
boundaries for much greater producer-consumer locality, while still
breadth first: each function is
entirely evaluated before the next
one.
total fusion: values are computed
on the fly each time that they are
needed.
sliding window: values are 
computed when needed then 
stored until not useful anymore.
tiles: overlapping regions are
processed in parallel, functions
are evaluated one after another.
sliding windows within tiles:
tiles are evaluated in parallel
using sliding windows.
in blurx out in blurx
blurx
out
in out
in blurx out in blurx out
serial y, serial x serial x, serial y serial y
vectorized x
parallel y
vectorized x
split x into 2xo+xi,
split y into 2yo+yi,
serial yo, xo, yi, xi
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
1 2 3 4 5 6
7 8 9 10 11 12
13 14 15 16 17 18
19 20 21 22 23 24
25 26 27 28 29 30
31 32 33 34 35 36
1 2
3 4
5 6
7 8
9 10
11 12
13 14
15 16
17 18
19 20
21 22
23 24
25 26
27 28
29 30
31 32
33 34
35 36
1 2
3 4
5 6
7 8
9 10
11 12
1
1
2
2
1
1
2
2
1
1
2
2
domain order
Figure 4. Even simple pipelines exhibit a rich space of scheduling choices, each expressing its own trade-off between parallelism, locality,
and redundant recompute. The choice made for each stage is two-fold. The simpler choice is the domain order, which can express thread
parallelism, vectorization, and traversal order (row-major vs column-major). Dimensions can be split into inner and outer components, which
recursively expands the choice space, and can express tiling strategies. The more complex question each stage must answer is when its inputs
should be computed. Choices include computing all dependencies ahead of time (breadth-ﬁrst), computing values as late as possible and
then discarding them (total fusion), and computing values as late as possible, but reusing previous computations (sliding window). The two
categories of choice interact. For example, the fastest schedules often split the domain into tiles processed in parallel, and then compute either
breadth-ﬁrst or with sliding windows within each tile.
leaving parallelism unconstrained both within and across tiles. (In
the iterated stencil computation literature, the redundant regions are
often called “ghost zones,” and this strategy is sometimes called
“overlapped tiling” [17, 31].) On a modern x86, this strategy is
10×faster than the breadth-ﬁrst strategy using the same amount of
multithreaded and vector parallelism. This is because the lack of
producer-consumer locality leaves the breadth-ﬁrst version limited
by bandwidth. This difference grows as the pipeline gets longer,
increasing the ratio of intermediate data to inputs and outputs,
and it will only grow further as the computational resources scale
exponentially faster than external memory bandwidth under Moore’s
Law.
The very fastest strategy we found on this architecture interleaves
the computation of the two stages using a sliding window over
scanlines, while splitting the image into strips of independent
scanlines which are treated separately:
alloc out[2046][3072]
for each ty in 0..2048/8:
alloc blurx[-1..1][3072]
for y in -2..8:
for x in 0..3072:
blurx[(y+1)%3][x] = in[ty*8+y+1][tx*32+x-1]
+ in[ty*8+y+1][tx*32+x]
+ in[ty*8+y+1][tx*32+x+1]
if y < 0: continue
for x in 0..3072:
out[ty*8+y][x] = blurx[(y-1)%3][x]
+ blurx[ y % 3][x]
+ blurx[(y+1)%3][x]
Relative to the original sliding window strategy, this sacriﬁces
two scanlines of redundant work on the overlapping tops and
bottoms of independently-processed strips of blurx to instead
reclaim ﬁne-grained parallelism within each scanline and coarse-
grained parallelism across scanline strips. The end result is 10%
faster still than the tiled strategy on one benchmark machine, but
10% slower on another. The best choice between these and many
other strategies varies across different target architectures. There
are also global consequences to the decision made for each stage in
a larger pipeline, so the ideal choice depends on the composition
of stages, not just each individual stages in isolation. Real image
processing pipelines often have tens to hundreds of stages, making
the choice space enormous.
3.2 A Model for the Scheduling Choice Space
We introduce a model for the space of important choices in schedul-
ing stencil pipelines, based on each stage choosing at what granular-
ity to compute each of its inputs, at what granularity to store each
for reuse, and within those grains, in what order its domain should
be traversed (Fig. 4).
The Domain Order Our model ﬁrst deﬁnes the order in which
the required region of each function’s domain should be traversed,
which we call the domain order, using a traditional set of loop
transformation concepts:
• Each dimension can be traversed sequentially or in parallel.
• Constant-size dimensions can be unrolled or vectorized.
• Dimensions can be reordered (e.g. from column- to row-major).
• Finally, dimensions can be split by a factor, creating two new
dimensions: an outer dimension, over the old range divided by
the factor, and an inner dimension, which iterates within the
factor. After splitting, references to the original index become
outer × factor + inner.
Splitting recursively opens up further choices, and enables many
common patterns like tiling when combined with other transforma-
tions. Vectorization and unrolling are modeled by ﬁrst splitting a
dimension by the vector width or unrolling factor, and then schedul-
ing the new inner dimension as vectorized or unrolled. Because
Halide’s model of functions is data parallel by construction, dimen-
sions can be interleaved in any order, and any dimension may be
scheduled serial, parallel, or vectorized.
For reduction functions, the dimensions of the reduction domain
may only be reordered or parallelized if the reduction update is
associative. Free variable dimensions may be scheduled in any order,
just as with pure functions.
Our model considers only axis-aligned bounding regions, not
general polytopes—a practical simpliﬁcation for image processing
and many other applications. But this also allows the regions to
be deﬁned and analyzed using simple interval analysis. Since our
model of scheduling relies on later compiler inference to determine
the bounds of evaluation and storage for each function and loop,
it is essential that bounds analysis be capable of analyzing every
expression and construct in the Halide language. Interval analysis
is simpler than modern tools like polyhedral analysis, but it can
effectively analyze through a wider range of expressions, which is
essential for this design.
The Call Schedule In addition to the order of evaluationwithin the
domain of each function, the schedule also speciﬁes the granularity
of interleaving the computation of a function with the storage and
computation of each function on which it depends. We call these
choices the call schedule. We specify a unique call schedule for each
function in the entire call graph of a Halide pipeline. Each function’s
call schedule is deﬁned as the points in the loop nest of its callers
where it is stored and computed (Fig. 4, top). For example, the three
extremes from the previous section can be viewed along these axes:
• The breadth-ﬁrst schedule both stores and computes blurx at
the coarsest granularity (which we call the root level—outside
any other loops).
• The fused schedule both stores and computes blurx at the
ﬁnest granularity, inside the innermost (x) loop of out. At this
granularity, values are produced and consumed in the same
iteration, but must be reallocated and recomputed on each
iteration, independently.
• The sliding window schedule stores at the root granularity, while
computing at the ﬁnest granularity. With this interleaving, values
of blurx are computed in the same iteration as their ﬁrst use,
but persist across iterations. To exploit this by reusing shared
values in subsequent iterations, the loops between the storage
and computation levels must be strictly ordered, so that a single
unique ﬁrst iteration exists for each point, which can compute it
for later consumers.
Together, the call schedule and domain order deﬁne an algebra
for scheduling stencil pipelines on rectangular grids. Composing
these choices can deﬁne an inﬁnite range of schedules, including
the vast majority of common patterns exploited by practitioners in
hand-optimized image processing pipelines.
The loop transformations deﬁned by the domain order interact
with the inter-stage interleaving granularity chosen by the call
schedule because the call schedule is deﬁned by specifying the
loop level at which to store or compute. A function call site may be
stored or computed at any loop from the innermost dimensions of
the directly calling function, to the surrounding dimensions at which
it is itself scheduled to be computed, and so on through its chain
of consumers. Splitting dimensions allows the call schedule to be
speciﬁed at ﬁner granularity than the intrinsic dimensionality of the
calling functions, for example interleaving by blocks of scanlines
instead of individual scanlines, or tiles of pixels instead of individual
pixels. Since every value computed needs a logical location into
which its result can be stored, the storage granularity must be equal
to, or coarser than, the computation granularity.
Schedule Examples Revisiting the optimized examples from
Sec. 3.1, the tiled schedule can be modeled as follows:
• The domain order ofout is split(y, 32) →ty, y; split(x, 32) →
tx, x; order(ty, tx, y, x). (This is similar to the tiled domain or-
der shown rightmost in Fig. 4.) All dimensions may be parallel.
x is vectorized for performance.
• The call schedule of blurx is set to both store and compute for
each iteration of tx in out.
• The domain of blurx under tx is scheduled order(y, x), and x
is vectorized for performance.
The parallel tiled sliding window schedule is modeled by making
the call schedule of blurx store at ty in out, but compute at ﬁner
granularity, at out’sy dimension. Then:
• The domain order ofout is split(y, 8) →ty, y; order(ty, y, x).
ty may be parallel, x is vectorized for performance. y must be
sequential, to enable reuse.
• The domain of blurx under y only has dimension x, which is
vectorized for performance.
4. Compiling Scheduled Pipelines
Our compiler combines the functions describing a Halide pipeline,
with a fully-speciﬁed schedule for each function, to synthesize the
machine code for a single procedure which implements the entire
pipeline. The generated pipeline is exposed as a C ABI callable
function which takes buffer pointers for input and output data, as
well as scalar parameters. The implementation is multithreaded and
vectorized according to the schedule, internally manages the alloca-
tion of all intermediate storage, and optionally includes synthesized
GPU kernels which it also manages automatically.
The compiler makes no heuristic decisions about which loop
transformations to apply or what will generate fast code. For all such
questions we defer to the schedule. At the same time, the generated
code is safe by construction. The bounds of all loops and allocations
are inferred. Bounds inference generates loop bounds that ultimately
depend only on the size of the output image. Bounded loops are
our only means of control ﬂow, so we can guarantee termination.
All allocations are large enough to cover the regions used by the
program.
Given the functions deﬁning a Halide pipeline and a fully
speciﬁed schedule as input (Fig. 5, left), our compiler proceeds
through the major steps below.
4.1 Lowering and Loop Synthesis
The ﬁrst step of our compiler is a lowering process that synthesizes
a single, complete set of loop nests and allocations, given a Halide
pipeline and a fully-speciﬁed schedule (Fig. 5, middle).
Lowering begins from the function deﬁning the output (in this
case, out). Given the function’s domain order from the schedule,
it generates a loop nest covering the required region of the output,
whose body evaluates the function at a single point in that domain
(Fig. 5, middle-top). The order of loops is given by the schedule, and
includes additional loops for split dimensions. Loops are deﬁned
by their minimum value and their extent, and all loops implicitly
stride by 1. This process rounds up the total traversed domain of
dimensions which have been split to the nearest multiple of the split
factor, since all loops have a single base and extent expression.
At this stage, loop bounds are left as simple symbolic expressions
of the required regionof the output function, which is resolved later.
The bounds cannot have inter-dependent dimensions between the
loops for a single function, so they represent a dense iteration over
an axis-aligned bounding box. Each loop is labeled as being serial,
parallel, unrolled, or vectorized, according to the schedule.
Lowering then proceeds recursively up the pipeline, from callers
to callees (here, from out to blurx). Callees (apart from those
scheduled inline) are scheduled to be computed at the granularity
of some dimension of some caller function. This corresponds to an
existing loop in the code generated so far. This site is located, and
code evaluating the callee is injected at the beginning of that loop
body. This code takes the form of a loop nest constructed using the
domain order of the callee. The allocation for the callee is similarly
injected at some containing loop level speciﬁed by the schedule.
In Fig. 5, middle, blurx is allocated at the level of tiles ( out.xo),
while it is computed as required for each scanline within the tile
algorithm flatteninglowering bound
inference
vectorization
& unrollingautotuner schedule
running time
code
generation
COMPILER
sliding window
optimization
& storage folding
Input: Algorithm
blurx(x,y)  = in(x-1,y)
 + in(x,y)
 + in(x+1,y)
out(x,y)  = blurx(x,y-1)
 + blurx(x,y)
 + blurx(x,y+1)
Sec 4.1: Lowering out
par for out.yo in 0..out.y.extent/4
 for out.xo in 0..out.x.extent/4
  for out.yi in 0..4
   vec for out.xi in 0..4
    out(4*x o+xi,4*yo+yi) =
     blurx(x i,yi-1)
     + blurx(x i,yi)
     + blurx(x i,yi+1)
Sec 4.2: Bounds inference
let blurx.y.min = 
 4*out.y o.min + out.yi.min - 1
Sec 4.5: Vectorization
vec for blurx.xi in 0..4
 blurx[blurx.y.stride*blurx.y+4*blurx.x o+xi) = ...
 ↓  
blurx[blurx.y.stride*blurx.y+4*blurx.xo+ramp(4)]
= in[in.y.stride*(blurx.y.min+blurx.y)+4*blurx.xo+ramp(4)]
+ ...
Sec 4.4: Flattening
out[out.y.stride*(4*(out.yo-out.yo.min)+out.yi)
    +4*(out.xo-out.xo.min)+out.xi]
= blurx[blurx.y.stride*(out.yi-1-blurx.y.min)+out.xi-blurx.x.min]
+ blurx[blurx.y.stride*(out.yi  -blurx.y.min)+out.xi-blurx.x.min]
+ blurx[blurx.y.stride*(out.yi+1-blurx.y.min)+out.xi-blurx.x.min]Sec 4.1: Lowering blurx
alloc blurx[blurx.y.extent][blurx.x.extent]
for blurx.y in blurx.y.min..blurx.y.max
 for blurx.x
o in blurx.x.min/4..blurx.x.max/4
  vec for blurx.xi in 0..4
   blurx(4*x o+xi,y) =
    in(4*x o+xi-1,y)
    + in(4*x o+xi,y)
    + in(4*x o+xi+1,y)
Input: Schedule
blurx: split x by 4 → xo, xi
 vectorize: xi
 store at out.x0
 compute at out.yi
out: split x by 4 → xo, xi
   split y by 4 → yo, yi
 reorder: yo, xo, yi, xi
 parallelize: yo
 vectorize: xi
Figure 5. Our compiler is driven by an autotuner, which stochastically searches the space of valid schedules to ﬁnd a high performance
implementation of the given Halide program. The core of the compiler lowers a functional representation of an imaging pipeline to imperative
code using a schedule. It does this by ﬁrst constructing a loop nest producing the ﬁnal stage of the pipeline (in this case out), and then
recursively injecting the storage and computation of earlier stages of the pipeline at the loop levels speciﬁed by the schedule. The locations
and sizes of regions computed are symbolic at this point. They are resolved by the subsequent bound inference pass, which injects interval
arithmetic computations in a preamble at each loop level that set the region produced of each stage to be at least as large as the region consumed
by subsequent stages. Next, sliding window optimization and storage folding remove redundant computation and excess storage where the
storage granularity is above the compute granularity. A simple ﬂattening transform converts multidimensional coordinates in the inﬁnite
domain of each function into simple one-dimensional indices relative to the base of the corresponding buffer. Vectorization and unrolling
passes replace loops of constant with k scheduled as vectorized or unrolled with the corresponding k-wide vector code or k copies of the loop
body. Finally, backend code generation emits machine code for the scheduled pipeline via LLVM.
(out.yi). The allocation and computation for blurx are inserted at
the corresponding points in the loop nest.
Reductions are lowered to a pair of loop nests: the ﬁrst initializes
the domain, and the second applies the reduction rule. Both alloca-
tion and loop extents are tracked as symbols of the required region
of the function used by its callers. Once lowering has recursed to
the end of the pipeline, all functions have been synthesized into a
single set of loops.
4.2 Bounds Inference
At this stage, for allocation sizes and loop bounds the pipeline relies
on symbolic bounds variables for each dimension of each function.
The next stage of lowering generates and injects appropriate deﬁni-
tions for these variables. Like function lowering, bounds inference
proceeds recursively back from the output. For each function, it
symbolically evaluates the bounds of each dimension based on the
bounds required of its caller and the symbolic indices at which the
caller invokes it. At each step, the required bounds of each dimen-
sion are computed by interval analysis of the expressions in the
caller which index that dimension, given the previously computed
bounds of all downstream functions.
After bounds inference has recursed to the top of the pipeline, it
walks back down to the output, injecting deﬁnitions for the bounds
variables used as stand-ins during lowering. They are deﬁned by
expressions which compute concrete bounds as a preamble at each
loop level (e.g., in Fig. 5, right, the minimum bound of blurx.y is
computed from interval analysis of the index expressions at which it
is accessed combined with the bounds of the calling function, out).
In practice, hoisting dynamic bounds evaluation expressions to the
outermost loop level possible makes the runtime overhead of more
complex bounds expressions negligible.
Interval analysis is an unusual choice in a modern loop synthesis
and code generation system. The resulting min/max bounds for each
dimension are less expressive than the polyhedral model. They can
only describe iteration over axis-aligned boxes, rather than arbitrary
polytopes. However, it is trivial to synthesize efﬁcient loops for
any set of intervals, in contrast to the problem of scanning general
polyhedra. For many domains, including image processing, this is an
acceptable simpliﬁcation: most functions are applied over rectilinear
regions.
Most critically, interval analysis can analyze a more general class
of expressions: it is straightforward to compute intervals through
nearly any computation, from basic arithmetic, to conditional ex-
pressions, to transcendentals, and even loads from memory. As a
result, this analysis can be used pervasively to infer the complete
bounds of every loop and allocation in any pipeline represented
in Halide. It also generalizes through constructs like symbolic tile
sizes, which are beyond the scope of polyhedral analysis. For cases
where interval analysis is over-conservative (e.g., when computing
the bounds of a ﬂoating point number loaded from memory which
the programmer knows will be between 0 and 1), Halide includes a
simple clamp operator, which simultaneously declares and enforces
a bound on an expression.
4.3 Sliding Window Optimization and Storage Folding
After bounds inference, the compiler traverses the loop nests seeking
opportunities for sliding window optimizations. If a realization of
a function is stored at higher loop level than its computation, with
an intervening serial loop, then iterations of that loop can reuse
values generated by previous iterations. Using the same interval
analysis machinery as in bounds inference, we shrink the interval
to be computed at each iteration by excluding the region computed
by all previous iterations. It is this transformation that lets us trade
off parallelism (because the intervening loop must be serial) for
reuse (because we avoid recomputing values already computed by
previous iterations.)
For example, in Fig. 5, blurx is stored for reuse within each tile
of out, but computed as needed, for each scanline within the tile.
Because scanlines (out.yi) are traversed sequentially, intermediate
values of blurx are computed immediately before the ﬁrst scanline
of out which needs them, but may be reused my later scanlines
within the tile. For each iteration of out.yi, the range of blurx.y
is computed to exclude the interval covered by all prior iterations
computed within the tile.
Storage folding is a second similar optimization employed at this
stage of lowering. If a region is allocated outside of a serial loop but
only used within it, and the subregion used by each loop iteration
marches monotonically across the region allocated, we can “fold”
the storage, by rewriting indices used when accessing the region by
reducing them modulo the maximum extent of the region used in any
given iteration. For example, in Fig. 5, each iteration of out.yi only
needs access to the last 3 scanlines ofblurx, so the storage of blurx
can be reduced to just 3 scanlines, and the value blurx(x,y+3) will
reuse the same memory address as blurx(x,y), blurx(x,y-3), and
so on. This reduces peak memory use and working set size.
4.4 Flattening
Next, the compiler ﬂattens multi-dimensional loads, stores, and allo-
cations into their single-dimensional equivalent. This happens in the
conventional way: a stride and a minimum offset are computed for
each dimension, and the buffer index corresponding to a multidimen-
sional site is the dot product of the site coordinates and the strides,
minus the minimum. (Cf. Fig. 5, right.) By convention, we always
set the stride of the innermost dimension to 1, to ensure we can
perform dense vector loads and stores in that dimension. For images,
this lays them out in memory in scanline order. While our model of
scheduling allows extreme ﬂexibility in the order of execution, we
do not support more unusual layouts memory, such as tiled or sparse
storage. (We have found that modern caching memory hierarchies
largely obviate the need for tiled storage layouts, in practice.)
4.5 Vectorization and Unrolling
After ﬂattening, vectorization and unrolling passes replace loops of
constant size scheduled as vectorized or unrolled with transformed
versions of their loop bodies. Unrolling replaces a loop of size n
with n sequential statements performing each loop iteration in turn.
That is, it completely unrolls the loop. Unrolling by lesser amounts is
expressed by ﬁrst splitting a dimension into two, and then unrolling
the inner dimension.
Vectorization completely replaces a loop of size n with a single
statement. For example, in Fig. 5 (lower right), the vector loop
over blurx.xi is replaced by a single 4-wide vector expression.
Any occurrences of the loop index ( blurx.xi) are replaced with
a special value ramp(n) representing the vector [0 1...n −1]. A
type coercion pass is then run over this to promote any scalars
combined with this special value to n-wide broadcasts of the scalar
expression. All of our IR nodes are meaningful for vector types:
loads become gathers, stores become scatters, arithmetic becomes
vector arithmetic, ternary expressions become vector selects, and
so on. Later, during code generation, loads and stores of a linear
expression of k∗ramp(n)+o will become dense vector loads and
stores if the coefﬁcient k = 1, or strided loads and stores with
stride k otherwise. In contrast to many languages, Halide has no
divergent control ﬂow, so this transformation is always well-deﬁned
and straight-forward to apply. In our representation, we never split
a vector into a bundle of scalars. It is always a single expression
containing ramps and broadcast nodes. We have found that this
yields extremely efﬁcient code without any sort of generalized loop
auto-vectorization.
4.6 Back-end Code Generation
Finally, we perform low-level optimizations and emit machine code
for the resulting pipeline. Our primary backends use LLVM for low-
level code generation. We ﬁrst run a standard constant-folding and
dead-code elimination pass on our IR, which also performs symbolic
simpliﬁcation of common patterns produced by bounds inference.
At this point, the representation is ready to be lowered to LLVM IR.
There is mostly a one-to-one mapping between our representation
and LLVM’s, but two speciﬁc patterns warrant mention.
First, parallel for loops are lowered to LLVM code that ﬁrst
builds a closure containing state referred to in the body of a for loop.
The loop body is lowered to a separate function that accepts the
closure as an argument and performs one iteration of the loop. We
ﬁnally generate code that enqueues the iterations of the loop onto a
task queue, which a thread pool consumes at runtime.
Second, many vector patterns are difﬁcult to express or generate
poor code if passed directly to LLVM. We use peephole optimization
to reroute these to architecture-speciﬁc intrinsics. For example, we
perform our own analysis pass to determine alignment of vector
loads and stores, and we catch common patterns such as interleaving
stores, strided loads, vector averages, clamped arithmetic, ﬁxed-
point arithmetic, widening or narrowing arithmetic, etc. By mapping
speciﬁc expression IR patterns to speciﬁc SIMD opcodes on each
architecture, we provide a means for the programmer to make use
of all relevant SIMD operations on ARM (using NEON) and x86
(using SSE and A VX).
GPU Code Generation The data parallel grids deﬁning a Halide
pipeline are a natural ﬁt for GPU programming models. Our com-
piler uses the same scheduling primitives, along with a few simple
conventions, to model GPU execution choices. GPU kernel launches
are modeled as dimensions (loops) scheduled to be parallel and
annotated with the GPU block and thread dimensions to which they
correspond.
The limitations of GPU execution place a few constraints on
how these dimensions can be scheduled. In particular, a sequence of
block and thread loops must be contiguous, with no other intervening
loops between the block and thread levels, since a kernel launch
corresponds to a single multidimensional, tiled, parallel loop nest.
Sets of kernel loops may not be nested within each other on current
GPUs which do not directly implement nested data parallelism.
Additionally, the extent of the thread loops must ﬁt within the
corresponding limits of the target device. Other than that, all the
standard looping constructs may still be scheduled outside or within
the block and grid dimensions. This corresponds to loops which
internally launch GPU kernels, and loops within each thread of a
GPU kernel, respectively.
Given a schedule annotated with GPU block and thread dimen-
sions, our compiler proceeds exactly as before, synthesizing a single
set of loop nests for the entire pipeline. No stage before the backend
is aware of GPU execution; block and thread dimensions are treated
like any other loops. The GPU backend extends the x86 backend,
including its full feature set. Outside the loops over block and thread
dimensions, the compiler generates the same optimized SSE code as
it would in the pure CPU target. At the start of each GPU block loop
nest, we carve off the sub-nest much like a parallel for loop in the
CPU backend, only it is spawned on the GPU. We ﬁrst build a clo-
sure over all state which ﬂows into the GPU loops. We then generate
a GPU kernel from the body of those loops. And ﬁnally, we generate
the host API calls to launch that kernel at the corresponding point in
the host code, passing the closure as an argument. We also generate
dynamic code before and after launches to track which buffers need
to be copied to or from the device. Every allocated buffer which is
used on the GPU has a corresponding device memory allocation,
and their contents are lazily copied only when needed.
The end result is not individual GPU kernels, but large graphs
of hybrid CPU/GPU execution, described by the same scheduling
model which drives the CPU backends. A small change in the
schedule can transform a graph of dozens of GPU kernels and
vectorized CPU loop nests, tied together by complex memory
management and synchronization, into an entirely different graph
of kernels and loops which produce the same result, expressively
modeling an enormous space of possible fusion and other choices
in mapping a given pipeline to a heterogeneous machine.
5. Autotuning Pipeline Schedules
We apply stochastic search to automatically ﬁnd good schedules for
Halide pipelines. The automatic optimizer takes a ﬁxed algorithm
and attempts to optimize the running time by searching for the most
efﬁcient schedule. The schedule search space is enormous—far too
large to search exhaustively. For example in the local Laplacian
ﬁlters pipeline, we estimate a lower bound of 10720 schedules. This
is derived by labeling functions with three tilings per function and all
possible store and compute granularities. The actual dimensionality
of the space is likely much higher. The optimal schedule dependends
on machine architecture, image dimensions, and code generation in
complex ways, and exhibits global dependencies between choices
due to loop fusion and caching behavior.
In this section, we describe our autotuner. Because the search
problem has many local minima, we use a genetic algorithm to
seek out a plausible approximate solution, inspired by the search
procedure in PetaBricks [2]. We ﬁrst describe the schedule search
space. We show how domain-speciﬁc knowledge can be used to
select reasonable starting points. Then we explain the general
operations of the genetic algorithm. Finally, we show how further
knowledge can be incorporated as search priors for more effective
mutation rules.
Schedule Search Space Our full model of scheduling is per call
but to simplify autotuning we schedule each function identically
across all call sites. The domain transformations include splitting
and reordering dimensions, and marking them parallel, vectorized,
or unrolled, or mapping a pair of dimensions to a GPU grid launch.
Variable and block size arguments are randomized and chosen from
small powers of two.
Because schedules have complex global dependencies, not all
schedules are valid: for example, a schedule could be computed or
stored inside a dimension that does not exist in the caller’s loop order.
Genetic operations such as mutate and crossover may invalidate
correct parent schedules. In general therefore we reject any partially
completed schedules that are invalid, and continue sampling until
we obtain valid schedules. We also verify the program output against
a correct reference schedule, over several input images. This is just
a sanity check: all valid schedules should generate correct code.
Finally to prevent explosion of generated code due to complex
pipelines, we limit the number of domain scheduling operations for
each function.
Search Starting Point One valid starting schedule is to label all
functions as computed and stored breadth ﬁrst (at the outermost,
root granularity). The tuner converges from this starting point, albeit
slowly. We can often do better by seeding the initial population
with reasonable schedules. For each function we ﬁnd its rectangular
footprint relative to the caller (via bounds inference) and inline
functions with footprint one. Remaining functions are stochastically
scheduled as either (1) fully parallelized and tiled or (2) simply
parallelized over y. We deﬁne fully parallelized and tiled as tiled
over x and y, vectorized within the tile’s inner x coordinate, and
parallelized over the y outer tile dimension. These choices are
selected by a weighted coin that has ﬁxed weight from zero to
one depending on the individual. This allows us to often discover
good starting points for functions that vectorize well, or fall back to
naive parallelism when that is not the case. The dimensions x and y
are chosen from adjacent dimensions at random, except when there
are optional bounds annotations provided by the Halide programmer
(such as the number of color channels): dimensions with small
bound are not tiled.
Genetic Algorithm Search We use a ﬁxed population size (128
individuals per generation, for all examples in this paper) and
construct each new generation with population frequencies of
elitism, crossover, mutated individuals, and random individuals.
Elitism copies the top individuals from the previous generation.
Crossover selects parents by tournament selection, followed by
two-point crossover, with crossover points selected at random
between functions. Random individuals are generated either by the
reasonable schedules described previously, or with equal probability,
by scheduling each function independently with random schedule
choices. This is directly derived from the PetaBricks autotuner [2].
Schedule Mutation Rules We incorporate further prior knowledge
about schedules into our mutation rules. Mutation selects a function
at random and modiﬁes its schedule with one of eight operations
selected at random. Six are fairly generic: randomize constants,
replace a function’s schedule with one randomly generated, copy
from a randomly selected function’s schedule, and for the function’s
list of domain transformations, either add, remove, or replace with a
randomly selected transformation.
Our remaining two mutations incorporate speciﬁc knowledge
about imaging. These are chosen with higher probability. First, a
pointwise mutation of compute or storage granularity is generally
ineffective at fusing loops. Thus we have a loop fusion rule which
schedules the chosen function as fully parallelized and tiled , fol-
lowed by scheduling callees as vectorized by x, and computed and
stored under the tile’s innerx dimension. The callee scheduling is
repeated recursively until a coin ﬂip fails. Finally, we incorporate
prior knowledge by applying a template: we replace a function’s
schedule with one of three common schedule patterns sampled from
a text ﬁle. These are: (1) compute and store under x, and vectorize
by x, (2) fully parallelized and tiled, and (3) parallelized over y and
vectorized over x. If generating a CUDA schedule we inject a fourth
pattern which simply tiles on the GPU. The x and y dimensions are
determined as in the starting point.
6. Results
To evaluate our representation and compiler, we applied them to a
range of image processing applications, and compared the best
autotuned result found by our compiler to the best previously
published expert implementation we could ﬁnd. We selected this set
of examples to cover a diversity of algorithms and communication
patterns. It includes pipelines ranging from two to 99 stages, and
including many different stencils, data-dependent access patterns,
and histogram reductions. We describe each application below.
Fig. 6 summarizes their properties, and Fig. 7 summarizes their
performance. All evaluation was performed on a quad core Xeon
W3520 x86 CPU, and an NVIDIA Tesla C2070 GPU.
Blur is the simple example used in Sec. 3.1, which convolves the
image with two 3 ×1 box kernels in two steps, a horizontal 3 ×1
kernel then a vertical 1 ×3 kernel. This is a simple example of two
consecutive stencils. Our reference comparison is a hand-optimized,
manually fused and multithreaded loop nest deﬁned entirely in SSE
intrinsics [26]. This version is 12×faster than a simple pair of loops
in C compiled by GCC 4.7. The version found by our autotuner
is 10% faster, still, while being generated from a two line Halide
algorithm rather than 35 lines of intrinsics.
# functions # stencils graph structure
Blur 2 2 simple
Bilateral grid 7 3 moderate
Camera pipeline 32 22 complex
Local Laplacian ﬁlters 99 85 very complex
Multi-scale interpolation 49 47 complex
Figure 6. Properties of the example applications. In some cases, the
number of functions exceeds the number of program lines in Fig. 7,
because Halide functions are meta-programmed using higher-order
functions.
Camera pipeline transforms the raw data recorded by the camera
sensor into a usable image. Its demosaicking, alone, is a complex
combination of 21 interleaved and inter-dependent stencils.
The reference comparison is a single carefully tiled and fused
loop nest from the Frankencamera, expressed in 306 lines of C++ [1].
All producer-consumer communication is staged through scratch
buffers, tiles are distributed over parallel threads using OpenMP, and
the tight loops are autovectorized by GCC. The Halide algorithm is
145 lines describing 32 functions and 22 different stencils, literally
translated from the pseudocode in the comments explaining the
original source. It compiles to an implementation 3.4×faster than
the hand-tuned original. The autotuned schedule fuses long chains
of stages through complex, interleaved stencils on overlapping tiles,
fully fuses other stages, vectorizes every stage, and distributes blocks
of scanlines across threads.
Multi-scale interpolation uses an image pyramid to interpolate
pixel data for seamless compositing. This requires dealing with data
at many different resolutions. The resulting pyramids are chains of
stages which locally resample over small stencils, but through which
dependence propagates globally across the entire image.
The reference implementation is a carefully-structured set of
loop nests which were hand-tuned by an Adobe engineer to generate
a fully vectorized implementation in GCC. The Halide algorithm is
substantially simpler, but compiles to an implementation 1.7×faster
than the original parallel vector code. The same Halide algorithm
also automatically compiles to a graph of CUDA kernels and x86
code which is 5.9×faster.
Bilateral grid is an efﬁcient implementation of the bilateral ﬁlter,
which smoothes an image while preserving its main edges [5, 21].
It ﬁrst scatters the image data into a 3D grid, effectively building
a windowed histogram in each column of the grid, then blurs the
grid along each of is axes with three 5-point stencils. Finally, the
output image is constructed by trilinear interpolation within the grid
at locations determined by the input image.
The CPU reference code is a tuned but clean implementation
from the original authors in 122 lines of C++. It is partially autovec-
torized by GCC, but is nontrivial to multithread (a naive OpenMP
parallelization of major stages results in a slowdown on our bench-
mark CPU), so the reference is single-threaded. The Halide algo-
rithm is 34 lines, and compiles to an implementation 4.2 ×faster
than the original. The speedup comes from a combination of par-
allelism, tile-level fusion of some stages, and careful reordering of
dimensions to control parallel grain size in the grid.
We also compared our implementation to a hand-tuned GPU
implementation from the original authors, written in 370 lines of
CUDA code. The same Halide algorithm—less than 1/10th the
code—found a different schedule which was 2.3 ×faster than
the hand-written CUDA. The Halide compiler generates similar
CUDA code to the reference, but the autotuner found an unintuitive
point in the schedule space which sacriﬁced some parallelism in
the grid construction step to reduce synchronization overhead in
the scattering reduction. It also uses a tiled fusion strategy which
x86
Halide Expert Speedup Lines Lines Factor
tuned tuned Halide expert shorter
(ms) (ms)
Blur 11 13 1.2× 2 35 18×
Bilateral grid 36 158 4.4× 34 122 4×
Camera pipe 14 49 3.4× 123 306 2×
Interpolate 32 54 1.7× 21 152 7×
Local Laplacian 113 189 1.7× 52 262 5×
CUDA
Halide Expert Speedup Lines Lines Factor
tuned tuned Halide expert shorter
(ms) (ms)
Bilateral grid 8.1 18 2.3× 34 370 11×
Interpolate 9.1 54* 5.9× 21 152* 7×
Local Laplacian 21 189* 9× 52 262* 5×
Figure 7. Comparison of autotuned Halide program running times
to hand-optimized programs created by domain experts in C, in-
trinsics, and CUDA. Halide programs are both faster and require
fewer code lines. (*No GPU reference available, compared to CPU
reference.)
passes intermediate results through GPU scratchpad memory to
improve locality through the blur steps at the expense of redundant
computation. These tradeoffs were counter-intuitive to the original
author, and also much harder to express in CUDA, but are easily
described by our schedule representation and found by our autotuner.
Local Laplacian ﬁlters uses a multi-scale approach to tone map
images and enhance local contrast in an edge-respecting fashion [3,
22]. It is used in the clarity, tone mapping, and other ﬁlters in Adobe
Photoshop and Lightroom. The algorithm builds and manipulates
several image pyramids with complex dependencies between them.
The ﬁlter output is produced by a data-dependent resampling from
several pyramids. With the parameters we used, the pipeline contains
99 different stages, operating at many scales, and with different
computational patterns.
The reference implementation is 262 lines of C++, developed
at Adobe, and carefully parallelized with OpenMP, and ofﬂoading
most intensive kernels to tuned assembly routines from Intel Per-
formance Primitives [14, 20]. It has very similar performance to a
version deployed in their products, which took several months to
develop, including 2-3 weeks dedicated to optimization. It is 10×
faster than an algorithmically identical reference version written
by the authors in pure C++, without IPP or OpenMP. The Halide
version was written in one day, in 52 lines of code. It compiles to an
implementation which is 1.7×faster than the highly optimized ex-
pert implementation (roughly 20×faster than the clean C++ without
IPP and OpenMP). The resulting schedule is enormously complex,
mixing different fusion, tiling, vectorization, and multithreading
strategies throughout the expansive 99 stage graph. In C, it would
correspond to hundreds of loops over more than 10,000 lines of
code.
The same program compiles with a different automatically
generated schedule to a hybrid CPU/GPU program with 58 unique
GPU kernels, each representing a differently tiled and partially fused
subset of the overall graph, and with the lookup table and several
smaller levels of the pyramids scheduled as vector code on the CPU.
The generated program has more distinct CUDA kernels than the
Halide algorithm describing it has lines of code. It runs 7.5×faster
than the hand-tuned Adobe implementation, and is more than 4×
faster than the best parallel vector implementation on the CPU. This
Source Target Cross- Autotuned Slowdown
size size tested on target (vs target
(MP) (MP) time (ms) (ms) autotuned)
Blur 0.3 30 13 11 1.2×
Bilateral grid 0.3 2 35 36 0.97×
Interpolate 0.3 2 31 32 0.97×
Blur 30 0.3 1.1 0.07 16×
Bilateral grid 2 0.3 9.6 6.7 1.4×
Interpolate 2 0.3 9.7 5.2 1.9×
Figure 8. Cross-testing of autotuned schedules across resolutions.
Each program is autotuned on a source image size. The resulting
schedule is tested on a target image size giving a “cross-tested time."
This is compared to the result of running the autotuner directly on
the target resolution. We report the ratio of the cross-tested time to
the autotuned-on-target time as the “slowdown." Note that schedules
generalize better from low resolutions to high resolutions. In theory
the slow-down should always be at least one, but due to the stochastic
nature of the search some schedules were slower when autotuned
on the target.
is by far the fastest implementation of local Laplacian ﬁlters we
know of.
6.1 Autotuning Performance
These examples took between 2 hours and 2 days to tune (from
10s to 100s of generations). In all cases, the tuner converged to
within 15% of the ﬁnal performance after less than one day tuning
on a single machine. Improvements to the compiling and tuning
infrastructure (for example, distributing tests across a cluster) could
reduce these times signiﬁcantly.
We generally found the tuned schedules to be insensitive to
moderate changes in resolution or architecture, but extreme changes
can cause the best schedule to change dramatically. Table 8 shows
experiments in cross-testing schedules tuned at different resolutions.
We observe that schedules generalize better from low resolutions to
high resolutions. We also mapped the best GPU schedule for local
Laplacian ﬁlter to the CPU, and found that this is 7×slower than
the best CPU schedule.
6.2 Discussion
Across a range of image processing applications and target architec-
tures, our scheduling representation is able to model, our compiler
is able to generate, and our autotuner is able to discover implemen-
tation strategies which deliver state-of-the-art performance. This
performance comes from careful navigation of the extremely high
dimensional space of tradeoffs between locality, parallelism, and re-
dundant recomputation in image processing pipelines. Making these
tradeoffs by hand is challenging enough, as shown by the much
greater complexity of hand-written implementations, but ﬁnding
the ideal points is daunting when each change a programmer might
want to test can require completely rewriting a complex loop nest
hundreds of lines long. The performance advantage of the Halide
implementations is a direct result of simplytesting many more points
in the space than a human programmer ever could manually describe
at the level of explicit loops.
Nonetheless, based on our experience, brute force autotuning
still poses signiﬁcant challenges of robustness and usability in
a real system. The tuning process is an entire extra stage added
to a programmer’s normal development process. Straightforward
implementations are sensitive to noise in testing environment and
many other factors. In Halide, while simple pipelines (like blur) can
tune effectively with only trivial mutation rules, we found heuristic
mutation rules are essential to converge in a reasonable amount
of time when tuning more complex imaging pipelines. However
these rules may be speciﬁc to the algorithm structure. For example,
different template mutation rules may be necessary for 3D voxel data
or unconventional color layouts. We also found that it is necessary
to maintain diversity to avoid becoming trapped in local minima,
which we did by using a large population (128 individuals per
generation). Even so, the tuner can occasionally become trapped,
requiring restarting with a new random initialization, and taking
the best of several runs. In general, we feel that high dimensional
stochastic autotuning has not yet developed the robust methodology
or infrastructure for real-world use found elsewhere in the compiler
community.
7. Prior Work
Image processing pipelines include similar structure to several
problems well studied in compilers.
Split Compilers Sequoia’s “mappings” and SPIRAL’s loop syn-
thesis algebra echo our separation of the model of scheduling from
the description of the algorithm, and its lifting outside our com-
piler [9, 25].
Stream Programs Compiler optimization of stream programs was
studied extensively in the StreamIt and Brook projects [4, 11, 29]. In
this framework, sliding window communication implements stencils
on 1D streams [ 12, 24, 30]. Stream compilers generally did not
consider the introduction of redundant work as a major optimization
choice, and nearly all work in stream compilation has focussed
on these 1D streams. Image processing pipelines, in contrast, are
effectively stream programs over 2D-4D streams.
Stencil Optimization Iterated stencil computations are important
to many scientiﬁc applications, and have been studied for decades.
Frigo and Strumpen proposed a cache oblivious traversal for efﬁ-
cient stencil computation [10]. This view of locality optimization
by interleaving stencil applications in space and time inspired our
model of scheduling. The Pochoir compiler automatically trans-
forms stencil codes from serial form into a parallel cache oblivious
form using similar algorithms [28].
Overlapping tiling is a strategy which divides a stencil compu-
tation into tiles, and trades off redundant computation along tile
boundaries to improve locality and parallelism [16], modeled in our
schedule representation as interleaving both storage and computa-
tion inside the tile loops. Other tiling strategies represent different
points in the tradeoff space modeled by our representation [ 19].
Past compilers have automatically synthesized parallel loop nests
with overlapped tiling on CPUs and GPUs using the polyhedral
model [13, 16]. These compilers focussed on synthesizing high
quality code given a single, user-deﬁned set of overlapped tiling
parameters. Autotuning has also been applied to iterated stencil com-
putations, but past tuning work has focussed on exhaustive search
of small parameter spaces for one or a few strategies [15].
Image Processing Languages Critically, many optimizations for
iterated stencil computations are based on the assumption that the
time dimension of iteration is large relative to the spatial dimension
of the grid. In image processing pipelines, most individual stencils
are applied only once, while images are millions of pixels in size.
Image processing pipelines also include more types of computa-
tion than stencils alone, and scheduling them requires choices not
only of different parameters, but of entirely different strategies, for
each of many heterogeneous stages, which is infeasible with either
exhaustive search or polyhedral optimization. Most prior image pro-
cessing languages and systems have focused on efﬁcient expression
of individual kernels, as well as simple fusion in the absence of
stencils [6, 8, 23, 27]. Recently, Cornwall et al. demonstrated fast
GPU code generation for image processing code using polyhedral
optimization [7].
Earlier work on the Halide language included a weaker model of
schedules, and required programmers to explicitly specify schedules
by hand [26]. This is the ﬁrst automatic optimizer, and therefore
the ﬁrst fully automatic compiler, for Halide programs. We show
how they can be automatically inferred, starting from just the
algorithm deﬁning the pipeline stages, and using relatively little
domain-speciﬁc knowledge beyond the ability to enumerate points
in the space of schedules. Our state-of-the-art performance shows
the effectiveness of our scheduling model for representing the
underlying choice space. The scheduling model presented here is
also richer than in past work. In particular, it separates computation
frequency from storage frequency in the call schedule, enabling
sliding window schedules and similar strategies which trade off
parallelism for redundant work while maintaining locality.
In all, this gives a dramatic new result: automatic optimization of
stencil computations, including full consideration of the parallelism,
locality, and redundancy tradeoffs. Past automatic stencil optimiza-
tions have targeted individual points in the space, but have not auto-
matically chosen among different strategies spanning these multi-
dimensional tradeoffs, and none have automatically optimized large
heterogeneous pipelines, only individual or small multi-stencils.
Acknowledgments
Eric Chan provided feedback and inspiration throughout the design
of Halide, and helped compare our local Laplacian ﬁlters imple-
mentation to his in Camera Raw. Jason Ansel helped inform the
design of our autotuner. This work was supported by DOE Award
DE-SC0005288, NSF grant 0964004, grants from Intel and Quanta,
and gifts from Cognex and Adobe.
References
[1] A. Adams, E. Talvala, S. H. Park, D. E. Jacobs, B. Ajdin, N. Gelfand,
J. Dolson, D. Vaquero, J. Baek, M. Tico, H. P. A. Lensch, W. Matusik,
K. Pulli, M. Horowitz, and M. Levoy. The Frankencamera: An
experimental platform for computational photography. ACM Trans.
Graph., 29(4), 2010.
[2] J. Ansel, C. Chan, Y . L. Wong, M. Olszewski, Q. Zhao, A. Edelman, and
S. Amarasinghe. PetaBricks: A language and compiler for algorithmic
choice. In ACM Programming Language Design and Implementation,
2009.
[3] M. Aubry, S. Paris, S. W. Hasinoff, J. Kautz, and F. Durand. Fast and
robust pyramid-based image processing. Technical Report MIT-CSAIL-
TR-2011-049, Massachusetts Institute of Technology, 2011.
[4] I. Buck, T. Foley, D. Horn, J. Sugerman, K. Fatahalian, M. Houston,
and P. Hanrahan. Brook for GPUs: Stream computing on graphics
hardware. In SIGGRAPH, 2004.
[5] J. Chen, S. Paris, and F. Durand. Real-time edge-aware image
processing with the bilateral grid. ACM Trans. Graph., 26(3), 2007.
[6] CoreImage. Apple CoreImage programming guide, 2006.
[7] J. L. T. Cornwall, L. Howes, P. H. J. Kelly, P. Parsonage, and B. Nico-
letti. High-performance SIMT code generation in an active visual
effects library. In Conf. on Computing Frontiers, 2009.
[8] C. Elliott. Functional image synthesis. In Proceedings of Bridges,
2001.
[9] K. Fatahalian, D. R. Horn, T. J. Knight, L. Leem, M. Houston, J. Y . Park,
M. Erez, M. Ren, A. Aiken, W. J. Dally, and P. Hanrahan. Sequoia:
programming the memory hierarchy. In ACM/IEEE conference on
Supercomputing, 2006.
[10] M. Frigo and V . Strumpen. Cache oblivious stencil computations. In
ICS, 2005.
[11] M. I. Gordon, W. Thies, M. Karczmarek, J. Lin, A. S. Meli, C. Leger,
A. A. Lamb, J. Wong, H. Hoffman, D. Z. Maze, and S. Amarasinghe.
A stream compiler for communication-exposed architectures. In Inter-
national Conf. on Architectural Support for Programming Languages
and Operating Systems, 2002.
[12] P. L. Guernic, A. Benveniste, P. Bournai, and T. Gautier. Signal – A
data ﬂow-oriented language for signal processing. IEEE Transactions
on Acoustics, Speech and Signal Processing, 34(2):362–374, 1986.
[13] J. Holewinski, L. Pouchet, and P. Sadayappan. High-performance code
generation for stencil computations on gpu architectures. In ICS, 2012.
[14] IPP. Intel Integrated Performance Primitives.
http://software.intel.com/en-us/articles/intel-ipp/.
[15] S. Kamil, C. Chan, L. Oliker, J. Shalf, , and S. Williams. An auto-tuning
framework for parallel multicore stencil computations. In IPDPS, 2010.
[16] S. Krishnamoorthy, M. Baskaran, U. Bondhugula, J. Ramanujam,
A. Rountev, and P. Sadayappan. Effective automatic parallelization of
stencil computations. In PLDI, 2007.
[17] J. Meng and K. Skadron. A performance study for iterative stencil
loops on gpus with ghost zone optimizations. In IJPP, 2011.
[18] R. Moore. Interval Analysis. 1966.
[19] A. Nguyen, N. Satish, J. Chhugani, C. Kim, and P. Dubey. 3.5-d
blocking optimization for stencil computations on modern cpus and
gpus. In Supercomputing, 2010.
[20] OpenMP. OpenMP. http://openmp.org/.
[21] S. Paris, P. Kornprobst, J. Tumblin, and F. Durand. Bilateral ﬁlter-
ing: Theory and applications. Foundations and Trends in Computer
Graphics and Vision, 2009.
[22] S. Paris, S. W. Hasinoff, and J. Kautz. Local Laplacian ﬁlters: Edge-
aware image processing with a Laplacian pyramid.ACM Trans. Graph.,
30(4), 2011.
[23] PixelBender. Adobe PixelBender reference, 2010.
[24] H. Printz. Automatic Mapping of Large Signal Processing Systems to a
Parallel Machine. Ph.D. Thesis, Carnegie Mellon University, 1991.
[25] M. Puschel, J. M. F. Moura, J. R. Johnson, D. Padua, M. M. Veloso,
B. W. Singer, J. Xiong, F. Franchetti, A. Gacic, Y . V oronenko, K. Chen,
R. W. Johnson, and N. Rizzolo. SPIRAL: Code generation for DSP
transforms. In Proceedings of the IEEE, volume 93, 2005.
[26] J. Ragan-Kelley, A. Adams, S. Paris, M. Levoy, S. Amarasinghe, and
F. Durand. Decoupling algorithms from schedules for easy optimization
of image processing pipelines. ACM Trans. Graph., 31(4), 2012.
[27] M. A. Shantzis. A model for efﬁcient and ﬂexible image computing.
In ACM SIGGRAPH, 1994.
[28] Y . Tang, R. Chowdhury, B. Kuszmaul, C.-K. Luk, and C. Leiserson.
The Pochoir stencil compiler. In SPAA, 2011.
[29] W. Thies, M. Karczmarek, and S. Amarasinghe. StreamIt: A language
for streaming applications. In International Conference on Compiler
Construction, 2002.
[30] P.-S. Tseng. A Parallelizing Compiler for Disributed Memory Parallel
Computers. PhD thesis, Carnegie Mellon University, 1989.
[31] X. Zhou, J.-P. Giacalone, M. J. Garzarán, R. H. Kuhn, Y . Ni, and
D. Padua. Hierarchical overlapped tiling.

A Game-Based Framework to Compare Program
Classifiers and Evaders
Thaís Damásio
UFMG
Minas Gerais, Brazil
thais.damasio@dcc.ufmg.br
Michael Canesche
UFMG
Minas Gerais, Brazil
michaelcanesche@dcc.ufmg.br
Vinícius Pacheco
UFMG
Minas Gerais, Brazil
vinicius.pacheco@dcc.ufmg.br
Marcus Botacin
Texas A&M University
Texas, USA
botacin@tamu.edu
Anderson Faustino da Silva
UEM
Paraná, Brazil
anderson@din.uem.br
Fernando M. Quintão Pereira
UFMG
Minas Gerais, Brazil
fernando@dcc.ufmg.br
Abstract
Algorithm classification consists in determining which algo-
rithm a program implements, given a finite set of candidates.
Classifiers are used in applications such malware identifi-
cation and plagiarism detection. There exist many ways to
implement classifiers. There are also many ways to imple-
ment evaders to deceive the classifiers. This paper analyzes
the state-of-the-art classification and evasion techniques. To
organize this analysis, this paper brings forward a system
of four games that matches classifiers and evaders. Games
vary according to the amount of information that is given to
each player. This setup lets us analyze a space formed by the
combination of nine program encodings; seven obfuscation
passes; and six stochastic classification models. Observations
from this study include: (i) we could not measure substantial
advantages of recent vector-based program representations
over simple histograms of opcodes; (ii) deep neural networks
recently proposed for program classification are no better
than random forests; (iii) program optimizations are almost
as effective as classic obfuscation techniques to evade clas-
sifiers; (iv) off-the-shelf code optimizations can completely
remove the evasion power of naïve obfuscators; (v) control-
flow flattening and bogus-control flow tend to resist the
normalizing power of code optimizations.
CCS Concepts: • Software and its engineering →Com-
pilers; Software libraries and repositories .
Keywords: algorithm classification, obfuscation
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for components of this work owned by others than the author(s) must
be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee. Request permissions from permissions@acm.org.
CGO ’23, February 25 – March 1, 2023, Montréal, QC, Canada
© 2023 Copyright held by the owner/author(s). Publication rights licensed
to ACM.
ACM ISBN 979-8-4007-0101-6/23/02. . . $15.00
https://doi.org/10.1145/3579990.3580012
ACM Reference Format:
Thaís Damásio, Michael Canesche, Vinícius Pacheco, Marcus Botacin,
Anderson Faustino da Silva, and Fernando M. Quintão Pereira. 2023.
A Game-Based Framework to Compare Program Classifiers and
Evaders. In Proceedings of the 21st ACM/IEEE International Sympo-
sium on Code Generation and Optimization (CGO ’23), February 25 –
March 1, 2023, Montréal, QC, Canada. ACM, New York, NY, USA,
14 pages. https://doi.org/10.1145/3579990.3580012
1 Introduction
The problem of algorithm classification can be informally
defined as follows: given a finite set of different specifica-
tions of algorithms, plus a program that implements one of
them, find which algorithm the problem implements. This
challenge has been called task classification by Allamanis
et al. [1] and program classification by Mou et al. [27]. The
name algorithm classification seems to have its origin in Ben-
Nun et al. [4]’s work. The problem of algorithm classification
has been much studied in the machine learning literature—
for an overview, we recommend Section 2 of Peng et al .
[32]. Algorithm classification is important because it has
many applications: redundancy elimination [3] and name re-
placement [2, 21] in source code, malware identification [22],
plagiarism detection [6, 29], task identification [42], etc.
Classifiers: the Good Side. As consequence of Rice[35]’s
Theorem, constructing a perfect algorithm classifier is impos-
sible. Thus, typical solutions to this problem are of stochastic
nature. Given the importance of the problem, and its open
essence, in recent years, many approaches have been pro-
posed for the construction of classifiers. These approaches
vary in terms of the way to represent programs, or in the
classification model adopted, or in the dataset used to train
the classifier. Program representations include, for example,
Inst2Vec [4], Code2Vec [2], Ir2Vec [40], Asm2Vec [16],
Milepost [28] and ProGraML [8]. Classification models in-
clude neural networks proposed by Mou et al. [27], Cummins
et al. [9] and Brauckmann et al. [5], for instance. And large
datasets of reference algorithms have been released by Mou
et al. [27], and Puri et al. [33], for instance.
108
CGO ’23, February 25 – March 1, 2023, Montréal, QC, Canada T. Damásio, M. Canesche, V. Pacheco, M. Botacin, A. Silva and F. Pereira
Evaders: the Bad Side. Just like there exists much effort
to improve the precision of algorithm classifiers, there ex-
ists also effort to evade them. Evasion enhances the ability
of malware to escape anti-virus systems [ 44] or improves
the chances that plagiarism might rest undiscovered [ 15].
The common approach to defeat algorithm classifiers is code
transformation [36]. Typically, evaders transform programs
via code obfuscation; however, recent studies have demon-
strated that standard compiler optimizations are also effec-
tive to hide programs [ 34]. Given that there are so many
evasion techniques, and yet so many ways to implement
classifiers, an immediate question that we pose, as a research
community, is where do we stand in this arms race. The goal
of this paper is to provide some answers to this question.
The Contributions of This Work. This paper proposes a
system of four games to compare algorithm classifiers (“clas-
sifiers" for short) and evaders. Every game uses a variation
of Mou et al. [27]’s balanced data set of solutions to program-
ming problems. The classifier is trained with part of this
dataset, and the evader challenges it with the rest. As we ex-
plain in Section 2, games differ on the resources given to each
player. Two games are symmetric: the classifier knows any
obfuscation strategy that is allowed to the evader. The two
other games are asymmetric: the evader can transform pro-
grams with an unknown obfuscator. In one of these games,
the classifier can try to revert the effects of obfuscation using
code optimizations as a normalization strategy.
The Arena. This paper does not propose classification ap-
proaches nor obfuscation strategies. Instead, as we explain
in Section 3, we create a playing arena by extracting arti-
facts from previous work. In common these artifacts have
the fact that they manipulate programs in the LLVM [ 23]
intermediate representation (IR). We have built classifiers
by combining nine different program embeddings (i.e., a
vector-based representation of a program) with six different
stochastic classification models. We have built evaders out
of eight code obfuscation techniques. Section 4 compares
classifiers and evaders using a curated version of Mou et al.
[27]’s Poj-104 dataset, plus 48 implementations of theMirai
malware [19]. From this comparison, we draw a number of
conclusions, some of which we list below.
Histograms: When evaluating symmetric games, we ob-
served no advantage of program embeddings designed
for code classification over histograms of opcodes. Al-
though simple, histograms resist well against transfor-
mations such as control-flow flattening, and can work
in tandem with different classification models.
Models: we could observe no improvement of neural
networks previously used for program classification [45]
over an off-the-shelf implementation of random forests.
Random forests seem to surpass Zhang et al.’s neural
networks by a wider margin once evaders are allowed
to transform programs.
Optimizers: optimizers are as effective as code obfus-
cators as an evasion strategy, while producing code
about 60x faster. However, code optimization tends to
resist poorly to a classifier that is aware of the opti-
mization approach used by the evader, whereas there
exist obfuscation techniques that can resist disclosure.
Normalization: code optimizations revert the effects
of many obfuscation strategies. A classifier that is al-
lowed to optimize programs with clang-O3 is imper-
vious to transformations like those proposed by Zhang
et al. [46] (built after Devore-McDonald and Berger
[15]’s). However, obfuscation approaches like control-
flow flattening and bogus control flow tend to resist
optimization-based normalization.
2 The Game Framework
Definition 2.1 formalizes the notion of programming prob-
lem. Definition 2.1 leaves the concept of reference function
vague on purpose: it could be a black-box implementation
of a program; a table with inputs and expected outputs; a
specification written in natural language, etc. Example 2.2
shows examples of typical programming problems.
Definition 2.1 (Programming Problem). Let 𝑓ref be a refer-
ence oracle: a function whose implementation is unknown,
but that can be consulted (i.e., invoked). Function 𝑓ref de-
fines a programming problem. A solution to this problem is a
function 𝑠of known implementation that produces the same
output as 𝑓ref , when given the same inputs.
Example 2.2. Poj-104 [27] is a set of 104 problems taken
from an online programming judge. Each problem consists of
500 sample solutions, each written by a potentially different
human programmer. Problems are defined by a set of inputs
and the expected outputs. CodeNet [33] is a set of 4,053
programming problems with 13,916,868 submissions written
in 55 different programming languages, out of which 53.6%
represent correct solutions to the reference oracles. These
oracles are tables with inputs and expected outputs.
Definition 2.3 formalizes algorithm classification as the
challenge of guessing which problem a program solves. No-
tice that algorithm classification deals with a closed universe
of algorithms: it involves a finite set of possible problems
from which the classifier must choose one. This restriction is
in contrast to the problem ofcode summarization [38], which
assumes an open universe of possible algorithms.
Definition 2.3 (Algorithm Classification). Given a set {𝑓1,
𝑓2,...𝑓 𝑚 }of 𝑚different programming problems, plus a solu-
tion 𝑠 for one of them, the algorithm classification problem
asks to find the problem 𝑓𝑖,1 ≤𝑖 ≤𝑚that 𝑠 solves.
A classifier categorizes programs into algorithms. A pro-
gram that transforms other programs to evade classification
is an evader. In this paper, transformations are either opti-
mizations, which try to make a program more efficient; or
109
A Game-Based Framework to Compare Program Classifiers and Evaders CGO ’23, February 25 – March 1, 2023, Montréal, QC, Canada
obfuscations, which try to hide the purpose of the program,
often making it less efficient. From these concepts, we define
an adversarial game as follows:
Definition 2.4 (Adversarial Game). A game consists of a
classifier 𝐶, an evader 𝐸, a set 𝐹 of 𝑚programming problems
{𝑓1,𝑓2,...,𝑓 𝑚 }, a set 𝑆 of 𝑛challenges {𝑠1,𝑠2,...,𝑠 𝑛 }and an
accuracy threshold 𝐾,0 ≤𝐾 ≤1. Each challenge 𝑠𝑖 ∈𝑆,1 ≤
𝑖 ≤𝑛must solve exactly one problem 𝑓𝑗 ∈𝐹,1 ≤𝑗 ≤𝑚. For
each 𝑠𝑖 , the game proceeds as follows:
1. The evader produces a solution 𝑠′
𝑖 = 𝐸(𝑠𝑖 );
2. The classifier chooses problem 𝑓𝑗 = 𝐶(𝑠′
𝑖 ),1 ≤𝑗 ≤𝑚.
The classifier wins the game if it can find the right problem
𝑓𝑗 that the solution 𝑠′
𝑖 solves with probability greater than 𝐾.
It loses the game otherwise.
The evader, according to Definition 2.4, is allowed to mod-
ify a program before giving it to the classifier. Modifications
must preserve semantics. Thus, if𝑠is a solution to a program-
ming problem 𝑓 (as per Definition 2.1), and 𝐸is an evader,
then 𝑠′= 𝐸(𝑠)is also a solution to 𝑓. From this observation,
Figure 1 lists the four games that we consider in this paper.
Game 0 (symmetric) Game 1 (asymmetric)
Game 2 (symmetric) Game 3 (asymmetric)
Classifier
Evader
0.8 Dataset
0.2 Dataset
Classifier
Evader
0.8 Dataset
0.2 Dataset +
Transformer
Classifier 0.8 Dataset +
Transformer
Evader 0.2 Dataset +
Transformer
Classifier 0.8 Dataset +
Optimizer
Evader 0.2 Dataset +
Transformer
Figure 1. The four games to evaluate classifiers and evaders.
Symmetric vs Asymmetric Games. In Figure 1, a “sym-
metric" game means that classifiers and evaders have access
to the same resources. An “asymmetric" game means that
classifiers and evaders have access to different resources. In
Game0, programs are used without transformation, e.g., the
evader 𝐸is the identity function. In Game1, the evader is al-
lowed to transform the challenge that it gives to the classifier,
but the classifier is tuned with the original training set. In
Game2, classifier and evader have access to the same code
transformation, which can be either an obfuscation or an op-
timization. The transformation is one-way: if 𝑠′= 𝐸(𝑠), the
classifier cannot recover 𝑠 by inspecting 𝑠′. However, it can
apply the transformation onto the training set. In Game3,
the two players have access to some code transformation,
albeit not the same. Our hypothesis is that optimizing trans-
formations let the classifier normalize programs, approxi-
mating challenges to the training set. Example 2.5 backs up
this hypothesis with some intuition.
Example 2.5. Figure 2 illustrates Game3. The evader modi-
fies a program before challenging the classifier. In this exam-
ple, the evader uses O-LLVM’s “instruction substitution" [20].
This pass replaces arithmetic instructions with other instruc-
tions that implement the same semantics. The classifier, in
turn, can optimize programs in its training set, using, for in-
stance, the -O1 level of clang. When given a challenge, like
function foo, the classifier applies the same transformation,
e.g., clang -O1, onto foo, before classifying it. In this exam-
ple, clang -O1 partially undoes the transformations carried
out by the evader. Notice that the example uses source code
for clarity, but the transformations that we evaluate in this
paper happen in the LLVM intermediate representation.
int foo(int b, int c) {
  int a = b + c;
  return a;
}
01
02
03
04
int foo(int b, int c) {
  int r = rand();
  int a = b + r;
  a = a + c;
  a = a - r;
  return a;
}
01
02
03
04
05
06
07
%add = add i32 %c, %b
ret i32 %add
01
02
%call = call i32 @rand()
%add = add i32 %c, %b
ret i32 %add
01
02
(a)
(b)
(c)
(d)
clang -O1 ollvm inst-sub
Figure 2. (a) A solution to the programming problem “sum
up two integers". (b) The optimized LLVM representation of
function foo. (c) A version of foo obfuscated with O-LLVM’s
instruction substitution. (d) The version of the obfuscated
function optimized with clang -O1.
3 The Classification Arena
This paper reuses artifacts from previous works to build
classifiers and evaders. These artifacts work on programs in
the LLVM intermediate representation; hence, support some
level of cross-language analysis [30]. However, our exper-
iments use only codes derived from C and C++ programs.
Figure 3 shows the two embeddings, nine models and six
code normalizers used to build classifiers. In total, this paper
evaluates 2×9×6 classifiers. Figure 4 shows the nine evaders
tried in this paper. Evaders differ on the transformation that
they apply on the challenge, before giving it to the classi-
fier. Game0 uses a passive evader, which does not use any
transformation. Figure 4 places this player in the last row.
The rest of this section explains the embeddings, models and
transformations that we have used in this paper.
3.1 Program Embeddings
A program is a string of characters. This string needs to be
transformed into a numerical format to be classified. Defini-
tion 3.1 calls this format a program embedding .
110
CGO ’23, February 25 – March 1, 2023, Montréal, QC, Canada T. Damásio, M. Canesche, V. Pacheco, M. Botacin, A. Silva and F. Pereira
challenge.ll + {f1, f2, …, fm}
dgcnn [45]
− or −
cnn
rf
svm
knn
lr
cfg, Brauckmann et al. [5]
cfg_compact, Faustino [17]
cdfg, Brauckmann et al. [5]
cdfg_compact, Faustino [17]
cdfg_plus, Brauckmann et al. [5]
PrograML, Cummins et al. [8]
ir2vec, VenkataKeerthy et al. [40]
milepost, Namolaru et al. [28]
histogram, Silva et al. [11]
-O0
-O3
Programming Problem fi,1 ≤ i ≤ m
Norm. Embedding functions Model
Standard scikit 
models [31]
mlp
Figure 3. The different classifiers used in this work. “Norm. "
denotes code normalizer , i.e., a transformation that the clas-
sifier is allowed to apply on code, to ease the task of doing
algorithm classification in Game3. We have experimented
with clang -O3 and with clang -O0 (no normalization).
challenge.c
ollvm
ollvm -bcf
ollvm -fla
ollvm -sub
challenge.ll
clang -S -emit-llvm
Junod et al. [20]
ch_mcmc.ll
ch_drlsc.ll
ch_rs.ll
ch_ollvm.ll
ch_bcf.ll
ch_fla.ll
ch_sub.ll
ch_id.ll
ch_o3.llclang -O3
mcmc
drlsg
rs
Zhang et al. [46]
Figure 4. The nine evaders evaluated in this paper.
Definition 3.1 (Program Embedding). A program embed-
ding is an array of numerical values, i.e., a tensor, that rep-
resents a program. The function that maps programs onto
embeddings is called an embedding function .
This paper evaluates the embedding functions in Figure 3.
Our criteria to choose embeddings was (i) the existence of a
publicly available artifact and (ii) the input of said artifact
being the LLVM intermediate representation (IR)1. Ir2Vec,
Milepost and Histogram map programs in the LLVM IR
into arrays. The other embeddings transform a graph-based
representation of the program into a tensor formed by three
arrays representing vertex attributes, edge attributes and
adjacencies. Figure 7 of Brauckmann et al . [5] shows how
a graph can be converted into an array. Graph-based for-
mats are extracted from programs in the LLVM IR. These
graphs differ on the amount of information that they ex-
pose: instructions, control flow, data dependencies, function
calls, etc. Two graph representations are compact ( cfg_c
and cdfg_c) [17], meaning that instructions are grouped
into basic blocks, instead of represented as individual nodes.
1We tried to use Ben-Nun et al. [4]’s Inst2Vec, but failed to reproduce their
experiments: the artifact runs out of memory even for small training sets.
3.2 Classification Models
Classifiers use stochastic models to solve algorithm classifi-
cation. We define a model as follows:
Definition 3.2 (Model). A model is a function that receives
(i) a program embedding; (ii) a set 𝐹 of 𝑚 programming
problems; and (iii) a challenge 𝑠, and outputs the problem
𝑓𝑖 ∈𝐹 that 𝑠 is believed to solve.
This paper evaluates the six models in Figure 3. Five
of these models are standard implementations available in
SciKit-Learn [31]: rf: random forest; svm: support vector
machine; knn: k-nearest neighbors; lr: logistic regression;
and mlp: multi-layer perceptron. The sixth model (dgcnn) is
the Deep Graph Convolutional Neural Network proposed by
Zhang et al. [45]. This model—a learning architecture that
receives attributes characterizing graphs—is built as follows:
1. four graph convolutional layers, with 32, 32, 32, and
one unit with hyperbolic tangent activation;
2. a one-dimensional convolutional layer;
3. a max pooling layer;
4. a one-dimensional convolutional layer;
5. a dense layer followed by a dropout layer;
6. a final dense layer that does the classification.
Zhang et al.’s model will only be used as is in Section 4.1
of this paper, because it requires graph-based program em-
beddings. Thus, dgcnn is used with the six graph-based em-
beddings from Figure 3. When applied onto the three other
embeddings: Ir2Vec, Milepost and Histogram, we use a
simpler version of it, cnn, which lacks the four first layers.
These layers only exist to merge the attributes of vertices
and edges into a single array, which is then used to start
the classification process. Thus, when receiving arrays as
the program embedding, these layers find no service. For
comparison purposes, we shall evaluate a standard imple-
mentation of a neural network fromSciKit (the mlp model),
with only one hidden layer with 100 perceptrons and ReLu
activation—which uses between 10 and 100x less memory.
3.3 Code Transformations
Evaders transform programs to confuse classifiers. This pa-
per recognizes nine different transformations, which Figure 4
shows. One of these transformations is the identity function;
the other is the sequence of compiler optimizations applied
by clang-O3. Three other transformations,bcf, fla and sub,
are available in O-LLVM [20], a code obfuscation framework
built on top of LLVM. These transformations can be used in
combination or separately. In this paper we either use them
all together (in the pass that we call simply ollvm), or use
them individually. The three transformations are as follows:
bcf: Bogus Control Flow inserts extra jumps into the pro-
gram’s control flow graph. These jumps are controlled
by conditions that are always false, but that are not
resolved by LLVM’s standard optimizations.
111
A Game-Based Framework to Compare Program Classifiers and Evaders CGO ’23, February 25 – March 1, 2023, Montréal, QC, Canada
fla: Control-Flow Flattening replaces every jump in the
program’s CFG with a case from a switch statement
within a loop. A counter at the end of every basic block
determines the next block to be executed.
sub: Instruction Substitution replaces logic and arithmetic
instructions with sequences of commands that are se-
mantically equivalent (See Example 2.5).
Three other obfuscations used in this paper were proposed
by Zhang et al. [46]to evade stochastic clone detectors. These
obfuscation techniques operate directly in the source code of
programs. Each of them is a different strategy to combine 15
simpler transformations. These simpler transformations con-
sist, for instance, in replacing a for loop with a while loop;
replacing a switch statement with a chain of conditional
branches; replacing constants with arithmetic operations;
etc. The three strategies that we reuse from Zhang et al. are:
rs: combines the 15 transformations randomly, but with-
out repetition, via a Random Search.
mcmc: uses a Markov-Chain Monte Carlo algorithm to
combine transformations, favoring sequences that lead
to programs harder to understand.
drlsg: uses a Deep Reinforcement Learning to combine
transformations, maximizing the distance between the
new program and its original version.
We could not make Zhang et al .’s scripts work directly.
Thus, we use its dataset: Zhang et al.’s repository contains
transformed versions of all the programs in Poj-104. We
had to fix some of these programs, inserting header files, to
ensure that they could be all compiled via clang.
4 Evaluation
This section provides answers to eight research questions,
discussed in Sections 4.1-4.8. The hardware adopted in these
experiments bears no influence on their results, except in
Section 4.6. Experiments were carried out on Linux Ubuntu
20.4. The intermediate representation of programs was ex-
tracted with LLVM 4.0, except in Section 4.1. In that case, we
use LLVM 15.0. We adopt an old version of LLVM, because
that is the only version which can be used with O-LLVM. Sec-
tion 4.1 uses LLVM 15.0 because that is the only version that
provides passes to extract all the program representations.
Box plots, whenever used, summarize ten samples.
Datasets: Sections 4.1, 4.2, 4.3, 4.4 and 4.7 use a perfectly
balanced dataset formed by the 104 classes of programming
problems released by Mou et al . [27]. Each class contains
500 solutions to a programming problem; hence, in total,
we use 104 ×500 programs (Section 4.1 uses a subset of
this dataset). Solutions were presumably written by different
people. Classifiers are trained with a random selection of 375
samples from each class (the training set ), and challenged
with 125 samples (the test set ). Training and classification
happens anew in Sections 4.1, 4.2, 4.3, 4.4 and 4.7; thus, results
might vary slightly, even if they discuss the same experiment.
Evaluation Metric: our evaluations of classification games
use perfectly balanced datasets, meaning that the number of
programming problems used to train the classifiers, and the
number of challenges (𝑠𝑗 in Def. 2.4) are equally distributed
among the different classes of problems. In this case, the
F1-score and the simple Accuracy (number of hits divided
by number of tries) yield the same information (as an illus-
tration, we report both metrics in Figure 12). Therefore, we
shall report only accuracy in most of our analyses.
4.1 RQ1: Comparing Program Embeddings
Question 1 (RQ1). How effective are typical program embed-
dings to support algorithm classification?
This section compares the embeddings in Figure 3 with
32 classes of programming problems taken randomly from
Poj-104. We do not use all the 104 classes of problems in
Poj-104 to reduce evaluation times. In total, running one
round of evaluation (nine representations) takes 11-12 hours
in a commodity machine (2.0GHz); and each experiment
requires ten rounds. The comparison uses Zhang et al. [45]
neural network—the only model that fits all the embeddings.
The other models can only receive histograms.
0.550.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.60
0.65
0.70
0.75
0.80
0.85
0.90
cfg cfg_c cdfg cdfg_c cdfg+
programl
ir2vecmileposthistogram
Accuracy: hits / (hits + misses)
dgcnn (Zhang et al. [44]’s model) cnn (adapt. of dgcnn)
73.49%
85.35%
59.90%
79.74%
70.95%
56.30%
81.35%
80.29%
81.82%
Figure 5. Comparison of program embeddings in Game0
using 32 problems. Numbers next to boxes are means.
Discussion: Figure 5 compares nine embeddings in Game0.
The most accurate results were observed when cfg_c was
the embedding used to classify programs. Mean accuracy
was 85.36%. We could not observe statistically significant
differences (pairwise p-values above 0.2) between cdfg_c,
Ir2Vec, Mile- Post and Histogram: they all had mean ac-
curacy between 81 and 82%. We found this result surprising,
because cfg_c also uses histograms: it produces a histogram
of opcodes for every basic block in the program, plus an
adjacency matrix to represent control-flow information. And
yet, it is only marginally better than Histogram, which is a
vector of 63 positions counting instruction opcodes. In other
words, cfg_c is asymptotically more costly thanHistogram,
but its observed benefit is small.
112
CGO ’23, February 25 – March 1, 2023, Montréal, QC, Canada T. Damásio, M. Canesche, V. Pacheco, M. Botacin, A. Silva and F. Pereira
cfg cfg_c cdfg cdfg_c cdfg+
programl
ir2vecmileposthistogramdgcnn (Zhang et al.’s model) cnn (adapt. of dgcnn)
0.0Accuracy: hits / (hits + misses)
0.2
0.4
0.6
0.8
Game1 (transformer: O-LLVM)
Game2 (transformer: O-LLVM)
Game3 (transf: O-LLVM; opt: clang -O3)
Figure 6. Comparison of program embeddings in Games 1,
2 and 3 using 32 problems. Dots are averages of 10 samples.
Trends observed in Figure 5 emerged in the other games,
as Figure 6 shows. In this case, Histogram and cfg_c yield
the best accuracies for Game2: about 76%. Nevertheless, the
different embeddings produce similar results. Accuracy drops
on the asymmetric games. This result is the consequence
of the obfuscator used: O-LLVM with its four transformation
passes. The next sections will discuss this result in more
detail. The cdfg+ embedding seems to better resist code
transformations, but its accuracy was never above 20%.
4.2 RQ2: Comparing Classification Models
Question 2 (RQ2). How effective are typical classification
models to support algorithm classification?
This section compares the accuracy and the memory con-
sumption of different models in regards to Game0. Results
are shown as boxplots aggregating ten executions of each
model. Variations, albeit small, are possible, because the six
different models that we compare, except knn, have parame-
ters that are initialized with random weights.
Discussion: Figure 7 shows the relative performance of the
six models on the test set formed by 104 problems ( 𝑓𝑖,1 ≤
𝑖 ≤104); each with 125 challenges (𝑠𝑗,1 ≤𝑗 ≤125). Random
forest showed the best accuracy (80.0%). This number means
that, given a program and 104 problems that it might solve,
rf is correct in four out of five guesses. However, the dif-
ference to Zhang et al. [45]’s neural network or to SciKit’s
is very small: less than 1.0%. Nevertheless, the difference
is still statistically significant within a confidence level of
0.99. Although the difference between Zhang et al.’s cnn and
SciKit’s mlp is small, the latter uses much less memory:mlp
(also knn, svm and lr) runs with less than 0.5GB of RAM.
Cnn uses 2.0GB, and rf uses 2.2GB.
4.3 RQ3: Measuring Evasion
This section evaluates Game1 and Game2 (see Figure 1)
using the six models analyzed in Section 4.2. We consider
0.4
0.6
C
NN
K NN MLP
S VM LR RF
1
2
3
Memory (GB)                    Accuracy
Figure 7. Comparison of different models used in Game0,
using 104 problems and the Histogram embedding.
only the Histogram embedding, as it is the only one that
can be passed as input to the different models.
Question 3 (RQ3). How effective are typical code obfuscation
techniques as a means to evade algorithm classification?
Figure 8 evaluates Game1. Each dot is the average of ten
measurements. The first column of Figure 8 shows the base-
line; i.e., the average of the accuracy results earlier seen
in Figure 7. Almost every obfuscation strategy reduces the
accuracy of the classifier. Drlsg, which Zhang et al . [46]
have introduced as the most effective obfuscation approach
among their choices (including mcmc and rs) has no ef-
fect upon the classifier. Once we inspect the bytecodes that
drlsg produces, the SSA conversion that LLVM uses reverts
all the effects of it. Control-flow flattening and instruction
substitution have almost no effect on the classifier that uses
random forests. Flattening barely changes the histogram of
instructions in a program—an observation that advocates for
the usage of histograms of opcodes as a classification embed-
ding. Substitution changes the instructions, but preserves
proportions; hence, random forests are still able to perform
effective classification.
Figure 9 evaluates Game2. In this case, the classifier is
aware of the obfuscation strategy adopted by the evader.
Thus, each sample in the training dataset is obfuscated be-
fore being given to the classifier. Notice that the classifier is
trained only with obfuscated codes, not with original pro-
grams. The adversary, similarly, only uses obfuscated chal-
lenges. In this case, classifiers and evaders behave almost like
in Game0. Thus, knowledge of the obfuscation approach is
enough to give the classifier power to resist evasion.
To explain the results seen in Figures 8 and 9, we compare
the average distance between histograms of original and ob-
fuscated programs. We use Euclidean Distance computed in
113
A Game-Based Framework to Compare Program Classifiers and Evaders CGO ’23, February 25 – March 1, 2023, Montréal, QC, Canada
Accuracy
BaselineOLLVM
FLA O3
MCMC
SUB BCF
DRLSG
RS
1.0
0.8
0.6
0.4
0.2
0.0
cnn knn mlp svm lr rf
Figure 8. Evaluation of Game1 on 104 problems, using the
Histogram embedding plus six different models.
cnn knn mlp svm lr rf
Accuracy
1.0
0.8
0.6
0.4
0.2
0.0
BaselineOLLVM
O3
MCMC
SUB BCF
DRLSG
RSFLA
Figure 9. Evaluation of Game2 on 104 problems, using the
Histogram embedding plus six different models.
a space of 63 dimensions (the number of opcodes). Figure 10
shows this comparison. The greater the distance between old
and new histograms, the greater the capacity of an evader to
deceive the classifier. In this sense, the most effective evasion
techniques are the optimizations used byclang -O3 and the
combination of transformations used by O-LLVM.
4.4 RQ4: The Normalization Hypothesis
Question 4 (RQ4). Can compiler optimizations improve al-
gorithm classification by reverting the effects of obfuscation?
This section evaluates Game3 to verify if optimizations
can revert the effects of obfuscation techniques. To this end,
we train the classifiers with normalized programs (the clas-
sifier is not aware of any obfuscator the evader can use).
When receiving a challenge—which consists of an obfus-
cated program—the classifier optimizes that code.
Discussion: Figure 11 summarizes the evaluation ofGame3.
O-LLVM is resistant against code optimizations. However,
this ability is due to bogus control flow (bcf)—which cannot
be easily optimized—plus an interesting accident: the appli-
cation of optimizations onto code subject to control-flow
Figure 10. Analysis of the distance between program em-
beddings extracted from different obfuscated codes.
cnn knn mlp svm lr rf
Accuracy
1.0
0.8
0.6
0.4
0.2
0.0
FLA
Baseline OLLVM MCMC
SUB BCF
DRLSG
RS
Figure 11. Evaluation of Game3 with the Histogram em-
bedding plus six models. The normalizer is clang -O3.
flattening changes substantially the mix of instructions in
that code (important: the classifier has no access to flattened
samples). In this case, optimizations caused the opposite of
what we expected: they increase the ability of fla to hide
the purpose of a program. And yet, when used alone, fla
fails to deceive a histogram-based classifier (see Fig. 8). The
obfuscation techniques designed by Zhang et al . [46] are
ineffective in face of optimizations. These transformations
might confuse source-code analyses; however, their effects
disappear from optimized codes.
4.5 RQ5: On the Number of Available Classes
Question 5 (RQ5). How does the effectiveness of algorithm
classifiers vary with the number of classes of algorithms?
Classification becomes harder as the number of classes
grow. For instance, a random classifier has an expected hit
114
CGO ’23, February 25 – March 1, 2023, Montréal, QC, Canada T. Damásio, M. Canesche, V. Pacheco, M. Botacin, A. Silva and F. Pereira
rate of 50% when dealing with two classes, and of 0.96% when
dealing with the 104 classes from Poj-104. Our classifiers
enjoy higher accuracy. This section investigates how this
accuracy varies with the number of classes of algorithms.
CNN KNN MLP SVM LR RF
CNN KNN MLP SVM LR RF
F1-Score                                 Accuracy
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
4           8           16          32          64
Figure 12. Evaluation of Game0 using the Histogram em-
bedding, considering a varying number 𝑚of classes of pro-
gramming problems.
Discussion: Figure 12 shows the accuracy (and the F1-score)
of the histogram-based classifiers in Game0, when dealing
with 2𝑖,2 ≤𝑖 ≤6 classes of programming problems. Figure 7
already shows accuracy for the maximum number of classes:
104. As expected, accuracy drops almost linearly with the
number of classes; however, the constant of the linear factor
is small. The classifier built with random forests still shows
a hit rate of about 80% when dealing with 64 classes—more
than 50x the expected accuracy of a random classifier.
4.6 RQ6: Performance Impact
Question 6 (RQ6). How does the performance of code pro-
duced by typical obfuscators and typical optimizers vary?
We evaluate Question 6 on the C programs from “The
Benchmark Game" ( https://benchmarksgame-team.pages.
debian.net/benchmarksgame/). We show numbers forclang
-O3 and O- LLVM. Numbers for Zhang et al . [46]’s transfor-
mations cannot be shown, for they have released only obfus-
cated versions of Poj-104, but not the transformers.
Discussion: Figure 13 compares the running times of codes
produced with clang-O3 and with O-LLVM. Times are rela-
tive to programs compiled with clang -O0. O-LLVM slows
down every program; clang-O3 speeds up all of them. The
16 programs, compiled with clang -O0, run in 203.45 sec-
onds (std = 3.08 sec). On average (geometric mean), the 16
obfuscated programs are 8.33x slower than the programs
compiled with clang -O0. The optimized programs are 2.32x
faster than what clang -O0 produces. Yet, variations are
important: obfuscated ary3 experiences a slowdown of more
0.01 0.1 1 10
ackermann
ary3
bin-trees
fannkuch
fasta
fib2
hash
heapsort
lists
matrix
methcall
random
regex
sieve
sptl-norm
strcat
O0/ollvm, geomean = 0.12x O0/O3, geomean = 2.32x
Figure 13. Running times of optimized and obfuscated pro-
grams. Times are relative to code produced by clang -O0.
than 30x, and optimizedmatrix [multiplication] experiences
a speedup of almost 7x. Optimized programs were 1.23x-
136.18x faster than their obfuscated counterparts.
4.7 RQ7: Detecting the Obfuscator
Question 7 (RQ7). Can the algorithm classifiers evaluated in
this paper detect the obfuscator used in games 1 and 2?
Game2 relies on the premise that the classifier “knows"
the obfuscation strategy used by the evader. However, dis-
covering the obfuscation strategy applied onto programs is
not trivial, as this section shows.
Experimental Setup. To answer Question 7, we have eval-
uated a classifier on a game formed by a set of 10 code trans-
formers, plus a dataset of 500 programs. We train the classi-
fier onto 10 ×400 transformed programs, and challenge it
with the 10 ×100 remaining programs. We consider the fol-
lowing transformers: (i): clang -O0; (ii): clang -mem2reg;
(iii): clang -O3 ; (iv): ollvm -bcf ; (v): ollvm -fla ; (vi):
ollvm -sub; (vii): drlsc; (viii): mcmc; (ix): rs and (x): ga. We
thus have ten classes to classify programs. Class-ii is formed
by programs in the LLVM IR optimized with just themem2reg
pass, which maps variables onto symbolic registers, instead
of leaving them in memory. Class-x is the genetic algorithm
used by Zhang et al. [46]. We could not use it in the other ex-
periments, because we could not apply it onto every program
in Poj-104. However, ga was available to answer Question 7,
given the restricted datasets used in this section. We eval-
uate Question 7 onto four datasets, each containing 5,000
samples: dataset1: a transformed version of each one of the
500 programs from a single—random—problem taken from
Poj-104; dataset2: a transformed version of each one of the
5 random solutions of a different problem from Poj-104 (we
eliminated four classes randomly, to remain with 100 classes);
dataset3: same as dataset1, except that each transformer
115
A Game-Based Framework to Compare Program Classifiers and Evaders CGO ’23, February 25 – March 1, 2023, Montréal, QC, Canada
is given 500 solutions from the same programming problem.
dataset4: same as dataset2, except that each transformer
is given 5 solutions from the same programming problem.
Notice dataset1 and dataset2 give the same programs to
each transformer, whereas the other two datasets give differ-
ent programs to each transformer.
CNN KNN MLP SVM LR RF
dataset1 
0.00
0.25
0.50
0.75
1.00
CNN KNN MLP SVM LR RF
dataset2
0.00
0.25
0.50
0.75
1.00
CNN KNN MLP svm LR RF
dataset3
0.00
0.25
0.50
0.75
1.00
CNN KNN MLP SVM LR RF
dataset4
0.00
0.25
0.50
0.75
1.00
Accuracy
Figure 14. Evaluation of Question 7 on four different
datasets. Programs in each dataset are originally produced
with clang -O0, and then transformed via either optimiza-
tions or obfuscations.
Discussion: Figure 14 compares the performance of the
classifier on the four different setups that we consider. No-
tice that in three setups the performance of the classifier
is approximately the same: it obtains a hit rate of 25%. Al-
though higher than a random hit rate (expected 10%), this
result shows that “obfuscation classification" is harder than
algorithm classification—at least given the stochastic tools
evaluated in this paper. The only situation where we ob-
served a high hit rate was in dataset3. However, this is
a spurious correlation: every obfuscator is applied onto a
different programming problem. In this case, the classifier
is rather discovering that programming problem, not the
obfuscator used to transform the program.
4.8 RQ8: Actual Malware
Question 8 (RQ8). Can the algorithm classifiers evaluated in
this paper be used to identify real-world malware?
This section evaluates the classifiers discussed in Section 3
on 48 versions of Mirai2, a malware that turns networked
devices running Linux into remotely controlled bots.
Building Mirai Identifiers. A classifier cannot be effec-
tively trained with 48 samples. Thus, to increase the number
of samples, we proceed as follows. First, we separate 36 posi-
tive samples (i.e., actual malware) for training; and add to
this suite 36 negative samples (i.e., benign software) taken
from SPEC CPU2017. Let this initial collection be the seed
2Retrieved on August 1st, 2022, from https://github.com/vxunderground/
MalwareSourceCode/tree/main/Linux/Mirai-Family
suite. The benign samples were chosen by size: for each neg-
ative sample, we chose the C file from SPEC that had the
closest size, in the number of LLVM instructions. From this
seed collection, we can build seven classifiers by choosing a
classification model (see Figure 3) and varying the training
set. Each training set 𝑡𝑖,1 ≤𝑖 ≤6, is formed by adding to
𝑡𝑖+1 all the programs in 𝑡𝑖 , plus a transformed version of the
seed suite. We obtain these six new versions by applying
the following transformers: clang -O1, clang -O2, clang
-O3, ollvm -fla, ollvm -bcf and , ollvm -sub. Thus, the
classifier 𝑡𝑖 is trained with 36 ×𝑖 positive samples and this
same number of negative samples.
Challenging Mirai Identifiers. Figure 15 shows how the
accuracy of two different families of classifiers vary with the
growth of the training set. We compare a classifier based
on the cnn model adapted from Zhang et al . [45] and the
random forest model (rf). Each challenge consists of 12 posi-
tive samples (malware) and 12 negative samples (from SPEC
CPU2017). We tried seven different challenges. Each chal-
lenge takes the 24 samples, and transforms them using—
exclusively—one of the seven transformers used in training.
The maximum accuracy is 7.0, in which case the classifier cor-
rectly identifies the 7 ×(12 +12)= 168 challenges. Figure 15
shows that the accuracy of the classifiers increases with the
growth of the training set. Both the classifiers considered in
Figure 15 achieve an almost perfect hit rate when trained
with all the 7 ×2 ×36 = 504 samples. The rf approach, in
particular, misses only one out of 168 challenges (a benign
file obfuscated with clang -O1).
Comparison with Anti-Virus Systems. We have com-
pared the accuracy of the classifiers evaluated in Figure 15
with the anti-viruses available at VirusTotal[39]. VirusTo-
tal aggregates several industry-quality anti-virus systems
into a single scanner, which can be tried online. We do not
know how many different anti-viruses are used by Virus-
Total—its website claims that it scans binaries with over 70
implementations [39]. Figure 16 summarizes results for the
168 challenges used to construct Figure 15. We have tried
VirusTotalin two ways: first, asking if a software is mal-
ware; second, asking if it is Mirai. We show results for the
best antivirus, and compare them with our best classifier:
the rf model trained with 504 samples. The accuracy of our
classifier is higher. However, whereasVirusTotalhas been
engineered to deal with any binary program, our classifier
can only tell if a program is a version of Mirai or not.
5 Related Work
Algorithm classification is a form ofcode diffing: the problem
of detecting coding patterns within programs. The literature
on code diffing is extensive. For an overview, we recommend
Section 2 of Ren et al. [34]’s recent work. There are, essen-
tially, two approaches to perform code diffing: semantic anal-
ysis and syntactic analysis. The former relies on approaches
116
CGO ’23, February 25 – March 1, 2023, Montréal, QC, Canada T. Damásio, M. Canesche, V. Pacheco, M. Botacin, A. Silva and F. Pereira
0.00
1.75
3.50
5.25
7.00
0.00
1.75
3.50
5.25
7.00
sub
bcf
fla
O3
O2
O1
O0
sub
bcf
fla
O3
O2
O1
O0
O0 +O1 +O2 +O3 +fla +bcf +sub
|train| 72 216 288 360 432144 504
cnn (Zhang et al.’s model)
rf
Stacked accuracy on the test set of 12+12 challenges (perfect hit-rate == 7.0)
Challenges (12 SPEC + 12 Mirai)Challenges (12 SPEC + 12 Mirai)
Figure 15. Evaluation of different algorithm classifiers on
168 challenges consisting of different versions of malicious
(Mirai) and benign (SPEC CPU2017) codes.
subbcfflaO3O2O1O0
is malw.
is mirai
96.77% 87.09% 83.87% 87.09% 96.77% 90.32% 96.77%
77.41% 74.19% 70.96% 74.19% 80.64% 80.64% 80.64%
rf(504) 100.0% 95.83% 100.0% 100.0% 100.0% 100.0% 100.0%
Figure 16. Comparison between the best classifier in Fig-
ure 15 and the best antivirus in VirusTotal.
such as symbolic execution/theorem proving [25, 26] or post-
mortem analysis of execution traces [ 7, 41] to determine
similarities between programs. The latter—syntactic-based
approaches—do not assume any semantic knowledge about
the target programming language. Thus, they tend to be
computationally cheaper. There are also hybrid code diffing
techniques, which combine semantic and syntactic method-
ologies. Algorithm classification is typically solved via ex-
clusively syntactic-based approaches [16, 18, 24, 43].
The Four Games in Perspective. As pointed out by Ren
et al. [34], most of the code diffing literature focus on the
construction of classifiers; hence, they would be described as
instances of Game0. New program embeddings evaluated as
Game0 include Inst2Vec [4], Code2Vec [2], Ir2Vec [40],
Asm2Vec [16], and ProGraML [10]. Brauckmann et al. [5]
and Siow et al. [37] use Game0 to compare the effectiveness
of abstract syntax trees, sequences of tokens, and a graph-
based embedding. In previous work [11], we have also used
Game0 to compare 13 variations of program embeddings.
However, to make our experiments practical, we have used
only five classes of algorithmic problems.
Work that aims at building evaders tends to focus on
Game1, where the classifier has no previous knowledge
of the transformations allowed to the evader. Such is the
case of Ren et al . [34]’s BinTuner, for instance. David et
al. [13, 14] perform experiments that we classify as varia-
tions of Game3. They try to perform code diffing in binaries
produced by different compilers. To ease this task, David
et al. use optimizations to put binaries into a canonical form.
In this sense, the evader would be a different compiler, not
some code-obfuscation approach.
6 Conclusion
This paper has compared state-of-the-art program classi-
fiers and evaders on a framework formed by four adversarial
games. The current implementation of this framework lets
us evaluate 9 ×6 ×2 classifiers, whose design varies accord-
ing to the representation used to encode programs (nine
embeddings); the stochastic model used to classify programs
(six implementations) and the code normalization approach
pre-applied onto programs (two optimizers). This framework
has allowed us to evaluate nine evaders, which differ in how
they obfuscate programs to hide their purposes. In hindsight,
some of our observations were expected, such as the fact that
optimizers are equally as effective as obfuscators to evade
classification [34]; or the fact that compiler optimizations
can revert naïve code obfuscation [13]. Yet, we found some
of our empirical findings surprising. For instance, in sym-
metric games, we have observed that histograms of opcodes
are as effective as program embeddings recently designed
to do algorithm classification. We also could not observe ad-
vantages of recent models proposed for stochastic algorithm
classifications over off-the-shelf machine learning libraries.
Finally, knowledge of the obfuscator improves substantially
the ability of a classifier to categorize programs ( Game2),
but discovering the obfuscator applied onto a program seems
to be a hard task (Sec. 4.7). All this said, our study has limita-
tions. In particular, most of our conclusions have been drawn
from experiments performed on a single dataset, formed by
104 programming problems, each with 500 samples. The only
different dataset that we have evaluated consists of 48 actual
malware plus 48 benign programs of similar sizes. Perhaps,
observations derived from different datasets could lead to
conclusions other than those enumerated in this paper.
Data Availability Statement. Software artifact [12] is
available at https://doi.org/10.5281/zenodo.7374649.
Acknowledgement
The authors would like to thank CNPq, FAPEMIG and CAPES
for financial support. The authors are also indebted to the
CGO reviewers and Michael O’Boyle, this paper’s shepherd,
for many useful suggestions and comments.
117
A Game-Based Framework to Compare Program Classifiers and Evaders CGO ’23, February 25 – March 1, 2023, Montréal, QC, Canada
A Artifact Appendix
A.1 Abstract
This artifact compares different program classification tech-
niques, and pit them against different evasion techniques. In
total, this artifact let us evaluate nine program encoding tech-
niques; seven code obfuscation passes; and seven stochastic
classification models. The artifact consists of a docker con-
tainer with accompanying scripts to replicate Figures 5-15 au-
tomatically, plus the dataset and accompanying instructions
to replicate Figure 16 manually. For a more up-to-date version
of this source code, check https://github.com/lac-dcc/yali.
A.2 Artifact Check-List (Meta-Information)
•Program: Seven code classification models: two from Zhang
et al. [45]: “dgcnn" (Deep Graph Convolution Neural Net-
work), “cnn" (Convolutional Neural Network); and five from
SciKit [31]: “rf" (Random Forest); “svm" (Support Vector
Machine); “knn" (K-Nearest Neighbors); “lr" (Logistic Regres-
sion); and “mlp" (Multilayer Perceptron). One code obfusca-
tor: O-LLVM [20]. Optimization passes in clang.
•Compilation: clang, gcc, cmake.
•Dataset: Training data samples are included.
•Run-time environment: Any operating system that sup-
ports Docker, Docker-compose, Python3, Wget, Tar, and Sed.
•Hardware: Any x86-64 machine with at least 64 GB of RAM
memory.
•Metrics: Accuracy, F1-Score, Memory (GB), and time.
•Output: Jupyter notebooks replicating Figures 5-15.
•How much disk space required (approx.)?: 80 GB.
•How much time is needed to prepare workflow (ap-
proximately)?: 1 hour.
•How much time is needed to complete experiments
(approximately)?: All the experiments take approximately
19 days:
– Speedup analysis (Figure 13) takes ∼23∗Rounds
60 hours. We
ran 10 rounds, so it takes ∼4 hours
– Memory analysis (Figure 7) takes ∼14∗𝑅𝑜𝑢𝑛𝑑𝑠
60 hours. We
ran 10 rounds, so it takes ∼2 hours and 20 minutes
– Game 0 (Figure 7) takes ∼ 14∗𝑅𝑜𝑢𝑛𝑑𝑠
60 hours. We ran 10
rounds, so it takes ∼2 hours and 20 minutes
– Game 1 and Game 2 (Figure 8 and Figure 9) take∼112∗𝑅𝑛𝑑𝑠
60
hours. We ran 10 rounds, so each one of them takes ∼18
hours and 30 minutes
– Game 3 (Figure 11) takes ∼98∗𝑅𝑜𝑢𝑛𝑑𝑠
60 hours. We ran 10
rounds, so each one of them takes ∼16 hours and 20
minutes
– The experiment to reproduce Figure 12 takes ∼10∗𝑅𝑜𝑢𝑛𝑑𝑠
60
hours. We ran 10 rounds, so it takes ∼1 hour and 40
minutes
– The experiment to reproduce Figure 15 takes ∼10∗𝑅𝑜𝑢𝑛𝑑𝑠
60
hours. We ran 10 rounds, so it takes ∼1 hour and 40
minutes
– The experiment to reproduce Figure 14 takes∼𝑅𝑜𝑢𝑛𝑑𝑠∗4
minutes. We ran 10 rounds, so it takes ∼40 minutes
– Comparison between program representations (Figure 5)
takes ∼𝑅𝑜𝑢𝑛𝑑𝑠∗1.7 days. We ran 10 rounds, so it takes∼
17 days
•Publicly available?: Yes
•Code licenses (if publicly available)?: GPL-3.0.
•Archived (provide DOI) ?: https://doi.org/10.5281/zenodo.
7374649
A.3 Description
A.3.1 Delivered. https://doi.org/10.5281/zenodo.7374649
A.3.2 Hardware Dependencies. Any x86-64 machine
with at least 64 GB of RAM memory.
A.3.3 Software Dependencies. Docker, Docker-compose,
Python3, Jupyter Notebook, Wget, Tar, and Sed. The Docker
image contains all the other dependencies to reproduce it.
A.3.4 Data Sets.
•“OJClone": POJ-104 dataset used by Mou et al. [27].
•“BCF": The OJClone dataset obfuscated by OLLVM’s
Bogus Control Flow.
•“FLA": The OJClone dataset obfuscated by OLLVM’s
Control Flow Flattening.
•“SUB": The OJClone dataset obfuscated by OLLVM’s
Instructions Substitution.
•“OLLVM": The OJClone dataset obfuscated by all the
OLLVM’s passes.
•“MCMC": The OJClone dataset obfuscated by the Monte
Carlo approach of Zhang et al. [46].
•“DRLSG": The OJClone dataset obfuscated by the Deep
Reinforcement Learning Sequence Generation of Zhang
et al. [46].
•“RS": The OJClone dataset obfuscated by the Random-
Search strategy of Zhang et al. [46].
•“dataset1", “dataset2", “dataset3", “dataset4": They are
a set of programs from OJClone originally produced
with clang -O0 and then transformed via either op-
timizations or obfuscations. There are ten classes of
code transformers in each of them.
•“Mirai": 48 different versions of theMirai malware [19],
plus 48 benign programs of similar sizes.
A.4 Installation
1. Download and unpack the zip file fromhttps://doi.org/
10.5281/zenodo.7374649. You will get a folder called
yali-main.
2. Copy the .env.example file3 into a new file called
.env.
3. Run the script setup.sh, e.g., ( $> ./setup.sh ) to
prepare the environment.
A.5 Experiment Workflow
To execute the experiments, run the scriptrun.sh as follows:
$ ./run.sh all
3This is a hidden file located in the project’s root directory.
118
CGO ’23, February 25 – March 1, 2023, Montréal, QC, Canada T. Damásio, M. Canesche, V. Pacheco, M. Botacin, A. Silva and F. Pereira
It is possible to execute the experiments separately. Below
we show how to generate data for specific figures:
•Figure 5: $> ./run.sh embeddings
•Figure 7 (second chart) and 12:$> ./run.sh resource
•Figure 7 (first chart): $> ./run.sh game0
•Figure 8: $> ./run.sh game1
•Figure 9: $> ./run.sh game2
•Figure 10: This figure is generated by a Jupyter Note-
book
•Figure 11: $> ./run.sh game3
•Figure 13: $> ./run.sh speedup
•Figure 14: $> ./run.sh discover
•Figure 15: $> ./run.sh malware
•Figure 16: The data in this figure cannot be reproduced
automatically, more information in section A.6.1.
A.6 Evaluation and Expected Results
The Statistics folder contains Jupyter Notebooks that plot the
data generated by the experiments. Each notebook describes
the chart(s) it contains and provides the steps to generate
this data. The artifact contains the following notebooks:
•EmbeddingResults: Presents comparison of program
embeddings in Game0 using 32 problems. Notebook
replicates Figure 5.
•GameResults: Presents information about the 4 games
proposed in our work and the Figure 15. Notebook
replicates Figures 7 (first chart), 8, 9, 11, Figure 15 and
12.
•ResourceResults: Presents information about resource
consumption (memory and time) of each model. Note-
book replicate Figure 7 (second chart).
•StrategiesResults: Notebook replicates Figures 10, 14
and 13, with analyses of each strategy.
A.6.1 Reproducing Figure 16. The only figure in the pa-
per that cannot be reproduced automatically is Figure 16, be-
cause it requires us submitting files to a website that classifies
them as either malicious or benign software. To reproduce
Figure 16, we recommend the following steps:
1. Go to www.virustotal.com
2. Click “Choose File" and select file to upload fromhttps:
//github.com/lac-dcc/yali/tree/main/MalwareDataset/
mirai (malicious code) or https://github.com/lac-dcc/
yali/tree/main/MalwareDataset/spec_cpu_2006_range
(benign code).
3. Retrieve detection score from the upper left bullet. This
is the malware detection rate.
4. Retrieve detection labels in front of the names of each
anti-virus in the central window.
5. Count the ratio of anti-virus (AVs) assigning the Mi-
rai label to the malicious samples. This is the family
classification rate.
6. Repeat for every file.
References
[1] Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, and Charles
Sutton. 2018. A Survey of Machine Learning for Big Code and Nat-
uralness. ACM Comput. Surv. 51, 4, Article 81 (jul 2018), 37 pages.
https://doi.org/10.1145/3212695
[2] Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019.
Code2Vec: Learning Distributed Representations of Code. Proc. ACM
Program. Lang. 3, POPL (2019), 40:1–40:29. https://doi.org/10.1145/
3290353
[3] Ira D. Baxter, Andrew Yahin, Leonardo Moura, Marcelo Sant’Anna,
and Lorraine Bier. 1998. Clone Detection Using Abstract Syntax Trees.
In ICSM (’98) . IEEE Computer Society, USA, 368.
[4] Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. 2018.
Neural Code Comprehension: A Learnable Representation of Code
Semantics. In NIPS (Montréal, Canada). Curran Associates Inc., Red
Hook, NY, USA, 3589–3601.
[5] Alexander Brauckmann, Andrés Goens, Sebastian Ertel, and Jeronimo
Castrillon. 2020. Compiler-Based Graph Representations for Deep
Learning Models of Code. In CC. ACM, New York, NY, USA, 201–211.
https://doi.org/10.1145/3377555.3377894
[6] Romain Brixtel, Mathieu Fontaine, Boris Lesner, Cyril Bazin, and Ro-
main Robbes. 2010. Language-Independent Clone Detection Applied
to Plagiarism Detection. InSCAM. IEEE Computer Society, USA, 77–86.
https://doi.org/10.1109/SCAM.2010.19
[7] Joan Calvet, José M. Fernandez, and Jean-Yves Marion. 2012. Aligot:
Cryptographic Function Identification in Obfuscated Binary Programs.
In CCS. Association for Computing Machinery, New York, NY, USA,
169–182. https://doi.org/10.1145/2382196.2382217
[8] Chris Cummins, Zacharias V. Fisches, Tal Ben-Nun, Torsten Hoefler,
Michael F P O’Boyle, and Hugh Leather. 2021. ProGraML: A Graph-
based Program Representation for Data Flow Analysis and Compiler
Optimizations. In ICML, Vol. 139. PMLR, Baltimore, Maryland, USA,
2244–2253.
[9] Chris Cummins, Pavlos Petoumenos, Alastair Murray, and Hugh
Leather. 2018. Compiler Fuzzing Through Deep Learning. In ISSTA.
ACM, New York, NY, USA, 95–105. https://doi.org/10.1145/3213846.
3213848
[10] Chris Cummins, Bram Wasti, Jiadong Guo, Brandon Cui, Jason Ansel,
Sahir Gomez, Somya Jain, Jia Liu, Olivier Teytaud, Benoit Steiner,
Yuandong Tian, and Hugh Leather. 2021. CompilerGym: Robust, Per-
formant Compiler Optimization Environments for AI Research. CoRR
abs/2109.08267 (2021), 12 pages. arXiv:2109.08267
[11] Anderson Faustino da Silva, Edson Borin, Fernando Magno Quintao
Pereira, Nilton Luiz Queiroz Junior, and Otavio Oliveira Napoli. 2022.
Program representations for predictive compilation: State of affairs
in the early 20’s. Journal of Computer Languages 73, null (dec 2022),
101171–101185. https://doi.org/10.1016/j.cola.2022.101171
[12] Thais Damasio, Michael Canesche, Vinicius Pacheco, Anderson
Faustino, Marcus Botacin, and Fernando Pereira. 2022. A Game-Based
Framework to Compare Program Classifiers and Evaders - Artifact . Uni-
versidade Federal de Minas Gerais. https://doi.org/10.5281/zenodo.
7374649
[13] Yaniv David, Nimrod Partush, and Eran Yahav. 2017. Similarity of
Binaries through Re-Optimization. In PLDI. Association for Comput-
ing Machinery, New York, NY, USA, 79–94. https://doi.org/10.1145/
3062341.3062387
[14] Yaniv David, Nimrod Partush, and Eran Yahav. 2018. FirmUp: Precise
Static Detection of Common Vulnerabilities in Firmware. In ASPLOS.
Association for Computing Machinery, New York, NY, USA, 392–404.
https://doi.org/10.1145/3173162.3177157
[15] Breanna Devore-McDonald and Emery D. Berger. 2020. Mossad: De-
feating Software Plagiarism Detection. Proc. ACM Program. Lang. 4,
OOPSLA, Article 138 (nov 2020), 28 pages. https://doi.org/10.1145/
3428206
119
A Game-Based Framework to Compare Program Classifiers and Evaders CGO ’23, February 25 – March 1, 2023, Montréal, QC, Canada
[16] Steven H. H. Ding, Benjamin C. M. Fung, and Philippe Charland. 2019.
Asm2Vec: Boosting Static Representation Robustness for Binary Clone
Search against Code Obfuscation and Compiler Optimization. In SP.
IEEE, Washington, DC, USA, 472–489. https://doi.org/10.1109/SP.2019.
00003
[17] Anderson Faustino. 2021. Graphs Based on IR as Representation
of Code: Types and Insights. In SBLP. ACM, New York, USA, 75–82.
https://doi.org/10.1145/3475061.3475063
[18] Jian Gao, Xin Yang, Ying Fu, Yu Jiang, and Jiaguang Sun. 2018.
VulSeeker: A Semantic Learning Based Vulnerability Seeker for Cross-
Platform Binary. In ASE. Association for Computing Machinery, New
York, NY, USA, 896–899. https://doi.org/10.1145/3238147.3240480
[19] Harm Griffioen and Christian Doerr. 2020. Examining Mirai’s Battle
over the Internet of Things . Association for Computing Machinery, New
York, NY, USA, 743–756. https://doi.org/10.1145/3372297.3417277
[20] Pascal Junod, Julien Rinaldini, Johan Wehrli, and Julie Michielin. 2015.
Obfuscator-LLVM–software protection for the masses. InInternational
Workshop on Software Protection . IEEE, Washington, DC, US, 3–9.
[21] Toshihiro Kamiya, Shinji Kusumoto, and Katsuro Inoue. 2002.
CCFinder: A Multilinguistic Token-Based Code Clone Detection Sys-
tem for Large Scale Source Code. IEEE Trans. Softw. Eng. 28, 7 (jul
2002), 654–670. https://doi.org/10.1109/TSE.2002.1019480
[22] Saruhan Karademir, Thomas Dean, and Sylvain Leblanc. 2013. Us-
ing Clone Detection to Find Malware in Acrobat Files. In CASCON
(Ontario, Canada). IBM Corp., USA, 70–80.
[23] C. Lattner and V. Adve. 2004. LLVM: a compilation framework for
lifelong program analysis amp; transformation. In International Sym-
posium on Code Generation and Optimization, 2004. CGO 2004. IEEE,
Washington, DC, US, 75–86.
[24] Bingchang Liu, Wei Huo, Chao Zhang, Wenchao Li, Feng Li, Aihua
Piao, and Wei Zou. 2018. ADiff: Cross-Version Binary Code Similarity
Detection with DNN. In ASE. Association for Computing Machinery,
New York, NY, USA, 667–678. https://doi.org/10.1145/3238147.3238199
[25] Lannan Luo, Jiang Ming, Dinghao Wu, Peng Liu, and Sencun Zhu.
2014. Semantics-Based Obfuscation-Resilient Binary Code Similar-
ity Comparison with Applications to Software Plagiarism Detection.
In FSE. Association for Computing Machinery, New York, NY, USA,
389–400. https://doi.org/10.1145/2635868.2635900
[26] Jiang Ming, Dongpeng Xu, Yufei Jiang, and Dinghao Wu. 2017. BinSim:
Trace-Based Semantic Binary Diffing via System Call Sliced Segment
Equivalence Checking. In SEC. USENIX Association, USA, 253–270.
[27] Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016. Convolutional
Neural Networks over Tree Structures for Programming Language
Processing. In AAAI. AAAI Press, Palo Alto, CA, US, 1287–1293.
[28] Mircea Namolaru, Albert Cohen, Grigori Fursin, Ayal Zaks, and Ari
Freund. 2010. Practical Aggregation of Semantical Program Properties
for Machine Learning Based Optimization. In CASES. ACM, New York,
NY, USA, 197–206. https://doi.org/10.1145/1878921.1878951
[29] Lawton Nichols, Kyle Dewey, Mehmet Emre, Sitao Chen, and Ben
Hardekopf. 2019. Syntax-Based Improvements to Plagiarism Detectors
and Their Evaluations. In ITiCSE (Aberdeen, Scotland Uk) (ITiCSE ’19) .
Association for Computing Machinery, New York, NY, USA, 555–561.
https://doi.org/10.1145/3304221.3319789
[30] Lawton Nichols, Mehmet Emre, and Ben Hardekopf. 2019. Structural
and Nominal Cross-Language Clone Detection. In FASE (Lecture Notes
in Computer Science, Vol. 11424) , Reiner Hähnle and Wil M. P. van der
Aalst (Eds.). Springer, Germany, 247–263. https://doi.org/10.1007/978-
3-030-16722-6_14
[31] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent
Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Pret-
tenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre
Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and
Édouard Duchesnay. 2011. Scikit-Learn: Machine Learning in Python.
J. Mach. Learn. Res. 12, null (nov 2011), 2825–2830.
[32] Dinglan Peng, Shuxin Zheng, Yatao Li, Guolin Ke, Di He, and Tie-Yan
Liu. 2021. How could Neural Networks understand Programs?. InICML
(Proceedings of Machine Learning Research, Vol. 139) , Marina Meila and
Tong Zhang (Eds.). PMLR, Online, 8476–8486. http://proceedings.mlr.
press/v139/peng21b.html
[33] Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo
Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir R. Choud-
hury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar,
Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. 2021.
CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity
of Coding Tasks. In NeurIPS Datasets and Benchmarks , Joaquin Van-
schoren and Sai-Kit Yeung (Eds.). Curran Associates Inc., Red Hook,
NY, USA, 21 pages.
[34] Xiaolei Ren, Michael Ho, Jiang Ming, Yu Lei, and Li Li. 2021. Un-
leashing the Hidden Power of Compiler Optimization on Binary Code
Difference: An Empirical Study. In PLDI. Association for Computing
Machinery, New York, NY, USA, 142–157. https://doi.org/10.1145/
3453483.3454035
[35] Henry Gordon Rice. 1953. Classes of recursively enumerable sets and
their decision problems. Trans. Amer. Math. Soc. 74, 2 (1953), 358–366.
https://doi.org/10.2307/1990888
[36] Sebastian Schrittwieser, Stefan Katzenbeisser, Johannes Kinder, Georg
Merzdovnik, and Edgar Weippl. 2016. Protecting Software through
Obfuscation: Can It Keep Pace with Progress in Code Analysis? ACM
Comput. Surv. 49, 1, Article 4 (apr 2016), 37 pages. https://doi.org/10.
1145/2886012
[37] Jing Kai Siow, Shangqing Liu, Xiaofei Xie, Guozhu Meng, and Yang
Liu. 2022. Learning Program Semantics with Code Representations:
An Empirical Study. https://doi.org/10.48550/ARXIV.2203.11790
[38] Sean Stapleton, Yashmeet Gambhir, Alexander LeClair, Zachary Eber-
hart, Westley Weimer, Kevin Leach, and Yu Huang. 2020. A Hu-
man Study of Comprehension and Code Summarization . Associa-
tion for Computing Machinery, New York, NY, USA, 2–13. https:
//doi.org/10.1145/3387904.3389258
[39] https://support.virustotal.com/hc/en-us/articles/115002146809-
Contributors. 2022. VirusTotal. VT Intelligence: Combine
Google and Facebook and apply it to the field of Malware.
https://www.virustotal.com/gui/intelligence-overview
[40] S. VenkataKeerthy, Rohit Aggarwal, Shalini Jain, Maunendra Sankar
Desarkar, Ramakrishna Upadrasta, and Y. N. Srikant. 2020. IR2VEC:
LLVM IR Based Scalable Program Embeddings. ACM Trans. Archit.
Code Optim. 17, 4, Article 32 (2020), 27 pages. https://doi.org/10.1145/
3418463
[41] Shuai Wang and Dinghao Wu. 2017. In-Memory Fuzzing for Binary
Code Similarity Analysis. In ASE. IEEE Press, USA, 319–330.
[42] Jackson Woodruff, Jordi Armengol-Estapé, Sam Ainsworth, and
Michael F. P. O’Boyle. 2022. Bind the Gap: Compiling Real Software
to Hardware FFT Accelerators. In PLDI. Association for Computing
Machinery, New York, NY, USA, 687–702. https://doi.org/10.1145/
3519939.3523439
[43] Xiaojun Xu, Chang Liu, Qian Feng, Heng Yin, Le Song, and Dawn Song.
2017. Neural Network-Based Graph Embedding for Cross-Platform
Binary Code Similarity Detection. In CCS. Association for Computing
Machinery, New York, NY, USA, 363–376. https://doi.org/10.1145/
3133956.3134018
[44] Ilsun You and Kangbin Yim. 2010. Malware Obfuscation Techniques:
A Brief Survey. In BWCCA. IEEE Computer Society, USA, 297–300.
https://doi.org/10.1109/BWCCA.2010.85
[45] Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. 2018.
An End-to-End Deep Learning Architecture for Graph Classification.
In Proceedings of the AAAI Conference on Artificial Intelligence (New
Orleans, Riverside, USA) (AAAI ’32). Association for the Advancement
of Artificial Intelligence, New York, NY, USA, 4423–4445.
120
CGO ’23, February 25 – March 1, 2023, Montréal, QC, Canada T. Damásio, M. Canesche, V. Pacheco, M. Botacin, A. Silva and F. Pereira
[46] Weiwei Zhang, Shengjian Guo, Hongyu Zhang, Yulei Sui, Yinxing Xue,
and Yun Xu. 2021. Challenging Machine Learning-based Clone Detec-
tors via Semantic-preserving Code Transformations. arXiv:2111.10793
https://arxiv.org/abs/2111.10793
Received 2022-09-02; accepted 2022-11-07
121

National Institutional Ranking Framework
Ministry of Education
Government of India
Welcome to Data Capturing System: ENGINEERING
Submitted Institute Data for NIRF'2025'
Institute Name: V. S. B. Engineering College [IR-E-C-36944]
Sanctioned (Approved) Intake
Academic Year 2023-24 2022-23 2021-22 2020-21 2019-20 2018-19
UG [4 Years Program(s)] 1230 1080 1080 1050 - -
PG [2 Year Program(s)] 45 45 - - - -
Total Actual Student Strength (Program(s) Offered by Your Institution)
(All programs
of all years)
No. of Male
Students
No. of Female
Students
Total Students Within State
(Including male
& female)
Outside State
(Including male
& female)
Outside
Country
(Including male
& female)
Economically
Backward
(Including male
& female)
Socially
Challenged
(SC+ST+OBC
Including male
& female)
No. of students
receiving full
tuition fee
reimbursement
from the State
and Central
Government
No. of students
receiving full
tuition fee
reimbursement
from Institution
Funds
No. of students
receiving full
tuition fee
reimbursement
from the Private
Bodies
No. of students
who are not
receiving full
tuition fee
reimbursement
UG [4 Years
Program(s)]
2557 1692 4249 3180 1068 1 1042 1497 614 559 492 874
PG [2 Year
Program(s)]
3 16 19 19 0 0 12 7 11 0 0 8
Placement & Higher Studies
UG [4 Years Program(s)]: Placement & higher studies for previous 3 years
Academic Year No. of first year
students intake in the
year
No. of first year
students admitted in
the year
Academic Year No. of students
admitted through
Lateral entry
Academic Year No. of students
graduating in
minimum stipulated
time
No. of students
placed
Median salary of
placed
graduates(Amount in
Rs.)
No. of students
selected for Higher
Studies
2018-19 870 684 2019-20 14 2021-22 560 487 476000(Four Lakh
seventy six Thousand)
16
2019-20 1050 843 2020-21 20 2022-23 766 680 485000(Four Lakhs
and eighty five
thousand)
5
2020-21 1050 778 2021-22 50 2023-24 795 766 700000(Seven Lakhs) 29
PG [2 Years Program(s)]: Placement & higher studies for previous 3 years
Academic Year No. of first year
students intake in the
year
No. of first year
students admitted in
the year
Academic Year No. of students graduating in minimum
stipulated time
No. of students
placed
Median salary of
placed
graduates(Amount in
Rs.)
No. of students
selected for Higher
Studies
2020-21 45 5 2021-22 4 1 300000(Three Lakhs) 0
2021-22 45 9 2022-23 4 2 306000( three lakhs sis
thousand)
0
2022-23 45 13 2023-24 12 10 600000(Six Lakhs) 2
Ph.D Student Details (including Integrated Ph.D)
Ph.D (Student pursuing doctoral program till 2023-24 Students admitted in the academic year 2024-25 should not be entered here.)
Total Students
Full Time 1
Part Time 37
No. of Ph.D students graduated (including Integrated Ph.D)
2023-24 2022-23 2021-22
Full Time 7 0 0
Part Time 0 0 0
Financial Resources: Utilised Amount for the Capital expenditure for previous 3 years
Academic Year 2023-24 2022-23 2021-22
Utilised Amount Utilised Amount Utilised Amount
Annual Capital Expenditure on Academic Activities and Resources (excluding expenditure on buildings)
Library ( Books, Journals and e-Resources only) 7762317 (Seventy Seven Lakh Sixty Two Thousand Three
Hundred and Seventeen)
3807023 (Thirty eight Lakhs Seven Thousand and twenty three
)
2902009 (Twenty Nine Lakhs Two Thousand and Nine)
New Equipment and software for Laboratories 6728542 (Sixty Seven Lakhs Twenty Eight Thousand Five
Hundred and Forty Two)
7920927 (Seventy Nine Lakhs Twenty Thousand Nine Hundred
Twenty Seven )
5822398 (Fifty Eight Lakhs Twenty Two Thousand Three
Hundred and Ninety Eight)
Engineering Workshops 6634739 (Sixty Six Lakhs Thirty Four Thousand  Seven
Hundred and Thirty Nine )
2401745 (Twenty Four Lakhs One Thousand Seven Hundred
Forty Five )
1450542 (Fourteen Lakhs Fifty Thousand Five Hundred and
Forty Two)
Other expenditure on creation of Capital Assets (For setting up
classrooms, seminar hall, conference hall , library, Lab, Engg
workshops excluding expenditure on Land and Building)
96607727 (Nine Crore Sixty Six Lakhs Seven Thousand and
Seven Hundred and Twenty Seven)
22745562 (Two Crore Twenty Seven Lakhs Forty Five
Thousand Five Hundred Sixty Two)
2954641 (Twenty Nine Lakhs Fifty Four Thousand Six Hundred
and Forty One)
Financial Resources: Utilised Amount for the Operational expenditure for previous 3 years
Academic Year 2023-24 2022-23 2021-22
Utilised Amount Utilised Amount Utilised Amount
Annual Operational Expenditure
Salaries (Teaching and Non Teaching staff) 127782703 (Twelve Crore Seventy Seven Lakhs Eighty Two
Thousand Seven Hundred and Three)
107478268 (Ten Crore Seventy Four Lakhs Seventy Eight
Thousand Two Hundred Sixty Eight)
88842585 (Eight Crore Eighty Eight Lakhs Forty Two
Thousand Five Hundred and Eighty Five)
Maintenance of Academic Infrastructure or consumables and
other running expenditures(excluding maintenance of hostels
and allied services,rent of the building, depreciation cost, etc)
148647731 (Fourteen Crore Eighty Six Lakhs Forty Seven
Thousand Seven Hundred and Thirty One)
27973800 (Two Crore Seventy Nine Lakhs Seventy Three
Thousand Eight Hundred)
88922338 (Eight Crore Eighty Nine Lakhs Twenty Two
Thousand Three Hundred and Thirty Eight)
Seminars/Conferences/Workshops 1336674 (Thirteen Lakh Thirty Six Thousand Six Hundred and
Seventy Four)
2244064 (Twenty Two Lakhs Forty Four Thousand Sixty Four) 1009963 (Ten Lakhs Nine Thousand Nine Hundred and Sixty
Three)
IPR
Calendar year 2023 2022 2021
No. of Patents Published 164 24 24
No. of Patents Granted 20 10 1
Sponsored Research Details
Financial Year 2023-24 2022-23 2021-22
Total no. of Sponsored Projects 20 6 4
Total no. of Funding Agencies 15 2 3
Total Amount Received (Amount in Rupees) 5478397 198388 420763
Amount Received in Words Fifty Four Lakhs Seventy Eight Thousand Three Hundred
Ninety Seven
One Lakh Ninety Eight Thousand Three Hundred and Eighty
Eight
Four Lakhs Twenty Thousand Seven Hundred and Sixty Three
Consultancy Project Details
Financial Year 2023-24 2022-23 2021-22
Total no. of Consultancy Projects 10 4 14
Total no. of Client Organizations 7 2 2
Total Amount Received (Amount in Rupees) 9000000 193265 436472
Amount Received in Words Ninety Lakhs One Lakh Ninety Three Thousand Two Hundred and Sixty Five Four Lakhs Thirty Six Thousand Four Hundred and Seventy
Two
PCS Facilities: Facilities of physically challenged students
1. Do your institution buildings have Lifts/Ramps? Yes, more than 80% of the buildings
2. Do your institution have provision for walking aids, including wheelchairs and transportation from one building to another for
handicapped students?
Yes
3. Do your institution buildings have specially designed toilets for handicapped students? Yes, more than 80% of the buildings
Faculty Details
Srno Name Age Designation Gender Qualification Experience (In
Months)
Currently working
with institution?
Joining Date Leaving Date Association type
1 DINESH M 35 Assistant Professor Male M.E. 144 Yes 04-06-2019 -- Regular
2 MOHAMED
NASURUDEEN M
34 Assistant Professor Male M.Tech 144 Yes 16-05-2019 -- Regular
3 Dr
UMAMAHESWARI
K
43 Professor Female Ph.D 219 Yes 01-12-2017 -- Regular
4 KRISHNAN R 39 Assistant Professor Male M.E. 163 Yes 03-07-2013 -- Regular
5 GUNASEKARAN S 46 Associate Professor Male M.Tech 304 Yes 02-06-2010 -- Regular
6 DHANABAL M 33 Assistant Professor Male M.E. 149 Yes 02-07-2014 -- Regular
7 DR
KALYANASUNDAR
AM R
51 Assistant Professor Male Ph.D 246 Yes 02-07-2014 -- Regular
8 THANGAVEL M 33 Assistant Professor Male M.E. 143 Yes 02-01-2015 -- Regular
9 PRABAKARAN S 35 Assistant Professor Male Ph.D 173 Yes 04-05-2012 -- Regular
10 MOHANAVEL G 33 Assistant Professor Male M.E. 148 Yes 27-04-2018 -- Regular
11 SRI
VENKATALAKSHMI
R
30 Assistant Professor Female M.E. 104 Yes 06-06-2018 -- Regular
12 KAPIL DEV N 37 Assistant Professor Male M.E. 164 Yes 18-07-2018 -- Regular
13 Dr GANESH BABU
P
54 Professor Male Ph.D 247 Yes 20-08-2018 -- Regular
14 GOWRI SHANKAR
R
35 Assistant Professor Male M.E. 163 Yes 03-06-2015 -- Regular
15 ANBUMANI P 36 Assistant Professor Male M.E. 165 Yes 04-01-2016 -- Regular
16 VINOTH KUMAR S 37 Assistant Professor Male M.E. 143 Yes 09-06-2014 -- Regular
17 Dr SUMITHIRADEVI
C
43 Associate Professor Female Ph.D 264 Yes 02-01-2009 -- Regular
18 ASHOK KUMAR R 39 Assistant Professor Male M.E. 149 Yes 13-06-2016 -- Regular
19 DR RAJU P 42 Associate Professor Male Ph.D 248 Yes 19-03-2012 -- Regular
20 DHAYABARANI R 52 Associate Professor Female M.E. 317 Yes 09-06-2003 -- Regular
21 Dr PRABAKARAN T
R
60 Professor Male Ph.D 213 Yes 30-06-2017 -- Regular
22 DEVARAJ P 32 Assistant Professor Male M.E. 125 Yes 04-06-2016 -- Regular
23 PRAGASH S 33 Assistant Professor Male M.E. 144 Yes 29-05-2018 -- Regular
24 Rajkumar K 32 Assistant Professor Male M.E. 104 Yes 01-07-2017 -- Regular
25 NARESHKUMAR P 39 Assistant Professor Male M.E. 202 Yes 01-07-2014 -- Regular
26 POORNIMA P 35 Assistant Professor Female M.E. 175 Yes 01-07-2014 -- Regular
27 MOHANASELVAN
C
32 Assistant Professor Male M.E. 134 Yes 01-07-2015 -- Regular
28 KIRUBASHANKAR
T S
43 Associate Professor Male Ph.D 255 Yes 04-07-2013 -- Regular
29 SIVAKUMAR R 45 Associate Professor Male M.E. 285 Yes 10-04-2006 -- Regular
30 SELVARASU A 41 Assistant Professor Male M.E. 202 Yes 16-07-2012 -- Regular
31 ARUN B 33 Assistant Professor Male M.E. 126 Yes 08-06-2017 -- Regular
32 Dr RAGHUPATHY R 37 Associate Professor Male Ph.D 166 Yes 21-02-2011 -- Regular
33 DINESH KUMAR D 32 Assistant Professor Male M.E. 142 Yes 13-08-2018 -- Regular
34 GEETHA S 29 Assistant Professor Female M.E. 104 Yes 09-08-2018 -- Regular
35 KEERTHANA S 33 Assistant Professor Female M.E. 112 Yes 05-06-2018 -- Regular
36 SWARNALATHA A
P
34 Assistant Professor Female M.E. 92 Yes 06-06-2018 -- Regular
37 Dr GOMATHI P S 47 Professor Female Ph.D 287 Yes 24-05-2010 -- Regular
38 MANGALARAJ K 39 Assistant Professor Male M. Phil 175 Yes 09-02-2015 -- Regular
39 BRINDHA G 36 Assistant Professor Female M. Phil 114 Yes 05-10-2017 -- Regular
40 KALAICHELVI K 56 Associate Professor Female M.E. 252 Yes 01-06-2009 -- Regular
41 MANIVANNAN K 40 Assistant Professor Male M.Tech 114 Yes 03-10-2017 -- Regular
42 SIVALINGAM T 35 Assistant Professor Male M.E. 137 No 01-07-2015 04-07-2024 Regular
43 Dr NIRMALKANNAN
V
42 Dean / Principal /
Director / Vice
Chancellor
Male Ph.D 245 No 08-11-2017 21-10-2023 Regular
44 THARANI B 36 Assistant Professor Female M.E. 116 Yes 16-08-2018 -- Regular
45 JEGAN R R 37 Assistant Professor Male M.E. 173 Yes 15-12-2017 -- Regular
46 ARUNKUMAR R 38 Assistant Professor Male M.E. 190 Yes 11-12-2017 -- Regular
47 REVATHI M 35 Assistant Professor Female M. Phil 144 Yes 17-01-2018 -- Regular
48 ANANDHAKUMAR
P
32 Assistant Professor Male M.E. 103 Yes 04-06-2019 -- Regular
49 Dr SANGEETHA M 44 Professor Female Ph.D 252 Yes 12-06-2019 -- Regular
50 PREMALATHA N 35 Assistant Professor Female M.E. 145 Yes 01-06-2019 -- Regular
51 SATHYANARAYAN
AN M
35 Assistant Professor Male M.E. 140 Yes 07-06-2019 -- Regular
52 Dr SELVAKUMAR J 45 Professor Male Ph.D 247 Yes 07-08-2019 -- Regular
53 VELAVAN R 43 Assistant Professor Male M.E. 155 Yes 06-06-2019 -- Regular
54 SANTHASEELAN 37 Assistant Professor Male M.Tech 158 Yes 06-06-2019 -- Regular
55 Dr
GNANAPRAGASAM
G
43 Associate Professor Male Ph.D 166 Yes 06-12-2019 -- Regular
56 LATHA P 45 Associate Professor Female M.E. 199 Yes 15-07-2019 -- Regular
57 Maalini D 37 Assistant Professor Female M.E. 87 Yes 16-12-2019 -- Regular
58 Ramprakash A 37 Assistant Professor Male M.E. 116 Yes 18-06-2019 -- Regular
59 Karthi K 30 Assistant Professor Male M.E. 79 Yes 24-06-2019 -- Regular
60 Dr Anishkumar M 43 Professor Male Ph.D 237 Yes 22-07-2019 -- Regular
61 Subashini K 30 Assistant Professor Female M.Tech 88 Yes 13-06-2019 -- Regular
62 DR AROCKIA
MARRY P
45 Professor Female Ph.D 261 Yes 31-01-2020 -- Regular
63 Dr KARTHI S 43 Associate Professor Male Ph.D 169 Yes 02-11-2020 -- Regular
64 MOHANAPRAKASH
K
37 Assistant Professor Male M.E. 191 No 04-11-2020 11-04-2024 Regular
65 PRASANTH B 32 Assistant Professor Male M.E. 108 Yes 17-08-2020 -- Regular
66 Dr SENTHIL
KUMAR B
48 Professor Male Ph.D 275 Yes 12-10-2020 -- Regular
67 CAPTAN
PRABAKARAN A
34 Assistant Professor Male M.E. 128 Yes 24-08-2020 -- Regular
68 NAVANEETHA
KRISHNAN G
37 Assistant Professor Male M.E. 111 Yes 05-11-2020 -- Regular
69 RAJASEKAR K 41 Assistant Professor Male M.E. 133 Yes 06-11-2020 -- Regular
70 MAHESH S 35 Assistant Professor Male Ph.D 156 Yes 03-11-2020 -- Regular
71 MAHALAKSHMI V 28 Assistant Professor Female M. Phil 53 Yes 04-01-2021 -- Regular
72 Dr KARTHIGA D 33 Assistant Professor Female Ph.D 53 Yes 19-01-2021 -- Regular
73 PRABAKARAN A 34 Assistant Professor Male M.E. 97 Yes 25-01-2021 -- Regular
74 PRAVEENKUMAR
G
39 Assistant Professor Male M.E. 164 Yes 25-01-2021 -- Regular
75 LAKSHMI R 34 Assistant Professor Female M.E. 82 Yes 27-01-2021 -- Regular
76 KAMALI R 27 Assistant Professor Female M.E. 53 No 27-01-2021 08-06-2023 Regular
77 ARIVARASAN S 39 Associate Professor Male M.Tech 218 Yes 29-01-2021 -- Regular
78 INDUMATHY S 29 Assistant Professor Female M.Tech 53 Yes 29-01-2021 -- Regular
79 MERLIN FLAVIA E 27 Assistant Professor Female M. Phil 53 Yes 29-01-2021 -- Regular
80 VENGAIMARBHAN
D
39 Assistant Professor Male M.E. 136 Yes 10-02-2021 -- Regular
81 JANANI S 28 Assistant Professor Female M.E. 52 Yes 10-02-2021 -- Regular
82 NANDHINI P 32 Assistant Professor Female M.E. 101 Yes 17-08-2020 -- Regular
83 Dr KATHIR I 55 Professor Male Ph.D 383 Yes 12-08-2021 -- Regular
84 ANANDAN D 39 Assistant Professor Male Ph.D 173 Yes 13-08-2021 -- Regular
85 RAJESHKUMAR S 37 Assistant Professor Male M.E. 146 No 12-08-2021 30-04-2024 Regular
86 VINAYAGAMOORT
HY A
43 Assistant Professor Male M. Phil 215 Yes 11-08-2021 -- Regular
87 SYED FAZLULLAH
S
51 Assistant Professor Male M. Phil 251 Yes 13-08-2021 -- Regular
88 BHAVADHARANI K 26 Assistant Professor Female M.E. 33 Yes 06-07-2022 -- Regular
89 PALRAJ R 54 Assistant Professor Male M.E. 204 Yes 11-05-2022 -- Regular
90 RANJITH G 30 Assistant Professor Male M.E. 57 Yes 14-03-2022 -- Regular
91 SENTHIL KUMAR K 38 Assistant Professor Male M.E. 162 Yes 14-03-2022 -- Regular
92 MOHAN S 53 Assistant Professor Male M.E. 156 No 07-03-2022 31-05-2024 Regular
93 ABIRAMI M 33 Assistant Professor Female M.E. 72 Yes 30-11-2021 -- Regular
94 SRINIVAS B 39 Assistant Professor Male M.Tech 160 Yes 15-07-2022 -- Regular
95 SARANYA R 34 Assistant Professor Female M.E. 96 Yes 15-06-2022 -- Regular
96 FRANKLIN JINO R
E
43 Assistant Professor Male M.Tech 180 Yes 15-06-2022 -- Regular
97 ARIVOLI R 39 Assistant Professor Male M.Tech 189 Yes 11-05-2022 -- Regular
98 DR KAVITHA K 51 Professor Female Ph.D 280 Yes 20-04-2022 -- Regular
99 MAHESHKUMAR K 37 Assistant Professor Male M.Tech 108 Yes 14-03-2022 -- Regular
100 PERIYATHAMBI P 44 Assistant Professor Male M.Tech 206 No 07-02-2022 29-06-2024 Regular
101 DR RAMESH BABU
R
44 Associate Professor Male Ph.D 228 Yes 15-09-2021 -- Regular
102 VIJAYA BALA N 55 Assistant Professor Female M.E. 207 Yes 26-08-2021 -- Regular
103 SUBRAMANI RAO
R
38 Assistant Professor Male M.Tech 132 Yes 18-07-2022 -- Regular
104 DR SARAVANAN G 45 Professor Male Ph.D 231 Yes 24-03-2022 -- Regular
105 RAJESH S 36 Assistant Professor Male M.E. 94 Yes 03-09-2021 -- Regular
106 LAVANYA M 35 Assistant Professor Female M.Tech 62 Yes 01-09-2021 -- Regular
107 VAIRAMANI G 35 Assistant Professor Male M.E. 154 Yes 24-01-2022 -- Regular
108 DR THANGADURAI
K R
51 Professor Male Ph.D 245 Yes 24-01-2022 -- Regular
109 ASHOK JAIN X S 37 Assistant Professor Male M.E. 129 No 21-01-2022 10-09-2023 Regular
110 DR KANAGARAJU
T
42 Assistant Professor Male Ph.D 205 Yes 19-01-2022 -- Regular
111 DR AMRISHRAJ D 35 Assistant Professor Male Ph.D 51 Yes 12-01-2022 -- Regular
112 DR NATESH M 37 Assistant Professor Male Ph.D 141 Yes 30-08-2021 -- Regular
113 PRASATH S 31 Assistant Professor Male M.Tech 60 Yes 08-06-2022 -- Regular
114 MANGAIYAR
THILAGAM P
33 Assistant Professor Female M.E. 93 No 25-02-2022 08-07-2024 Regular
115 JOSEPH VIANNY X 28 Assistant Professor Male M.E. 33 Yes 15-12-2021 -- Regular
116 PRASANNA
KUMAR M
28 Assistant Professor Male M.E. 25 Yes 27-08-2021 -- Regular
117 VINOTH T 34 Assistant Professor Male M. Phil 98 Yes 04-07-2022 -- Regular
118 SARAVANAN A 36 Assistant Professor Male M. Phil 84 Yes 15-07-2022 -- Regular
119 AARDRA C 26 Assistant Professor Female M.A 40 No 04-05-2022 10-11-2023 Regular
120 BALASUBRAMANIA
N K
43 Assistant Professor Male M. Phil 212 Yes 06-09-2021 -- Regular
121 SHEELA V 40 Associate Professor Female M. Phil 121 Yes 06-09-2021 -- Regular
122 SURESH KUMAR V 40 Assistant Professor Male M. Phil 144 Yes 06-06-2022 -- Regular
123 DR SARAVANA
KUMAR R
49 Associate Professor Male Ph.D 238 Yes 08-12-2021 -- Regular
124 DR
THULASIDHASAN J
37 Assistant Professor Male Ph.D 84 Yes 04-10-2021 -- Regular
125 DR BHASKAR C 42 Assistant Professor Male Ph.D 192 No 06-09-2021 26-06-2024 Regular
126 DR SYED IBRAHIM
P S
53 Assistant Professor Male Ph.D 202 Yes 06-09-2021 -- Regular
127 MEENAL ABIRAMI
R
38 Assistant Professor Female Ph.D 67 Yes 20-07-2022 -- Regular
128 MUTHU KUMAR S 27 Assistant Professor Male Ph.D 25 Yes 08-08-2022 -- Regular
129 SATHYARAJU S 36 Assistant Professor Male M. Phil 68 Yes 01-11-2022 -- Regular
130 NAGAMANI S 35 Assistant Professor Male M. Phil 63 No 20-04-2022 27-06-2024 Regular
131 VIJAYAKUMAR M 38 Assistant Professor Male M. Phil 155 Yes 20-04-2022 -- Regular
132 DR VINOTH S 31 Assistant Professor Male Ph.D 25 Yes 04-10-2021 -- Regular
133 PRABHAKARAN T 25 Assistant Professor Male M.Sc. 25 Yes 01-10-2021 -- Regular
134 SARAVANAKUMAR
R
28 Assistant Professor Male NET 25 Yes 12-10-2022 -- Regular
135 KANNAN S 41 Assistant Professor Male Ph.D 82 Yes 01-12-2021 -- Regular
136 PRIYA K 34 Assistant Professor Female M. Phil 40 Yes 13-10-2021 -- Regular
137 BASKAR B 29 Assistant Professor Male M. Phil 51 Yes 04-10-2021 -- Regular
138 KAVITHA V 41 Assistant Professor Female M.E. 232 Yes 08-03-2022 -- Regular
139 KEERTHIGA K 33 Assistant Professor Female M.Tech 61 Yes 18-07-2022 -- Regular
140 DR SAM
EBENEZER R
37 Associate Professor Male M.Tech 111 No 04-05-2022 01-11-2023 Regular
141 SRIDHIVYA M 29 Assistant Professor Male M.Tech 36 No 07-03-2022 24-11-2024 Regular
142 DR GOBAL SAMY 39 Associate Professor Male Ph.D 86 No 03-12-2021 30-04-2024 Regular
143 DR SARAVANAN S 49 Associate Professor Male Ph.D 216 Yes 04-10-2021 -- Regular
144 BALASUBRAMANI 39 Assistant Professor Male M.Tech 162 Yes 26-12-2022 -- Regular
145 GOMATHI P 43 Assistant Professor Female Ph.D 198 Yes 01-12-2022 -- Regular
146 Sabari K 33 Assistant Professor Male M.E. 106 Yes 19-08-2021 -- Regular
147 DR
VENKATACHALAM
N
45 Associate Professor Male Ph.D 165 Yes 17-10-2022 -- Regular
148 MR KISHORE
KUMAR R
29 Assistant Professor Male NET 18 Yes 07-02-2023 -- Regular
149 MR GOBINATH P 31 Assistant Professor Male M.Sc. 17 Yes 09-02-2023 -- Regular
150 MR KARTHIKEYAN
M
49 Assistant Professor Male M.E. 18 Yes 14-09-2022 -- Regular
151 MR ANBU A 29 Assistant Professor Male M.Tech 46 Yes 27-01-2023 -- Regular
152 DR BITANG
KWRUNG TRIPURA
36 Assistant Professor Male Ph.D 23 Yes 03-01-2023 -- Regular
153 DR TRIPTI DE T 36 Assistant Professor Female Ph.D 54 Yes 30-12-2022 -- Regular
154 MRS AAYISHA
BEGAM A
29 Assistant Professor Female M.E. 19 Yes 03-05-2023 -- Regular
155 DR SURENDAR P 37 Assistant Professor Male Ph.D 123 Yes 17-10-2022 -- Regular
156 MRS PARVATHI M 49 Assistant Professor Female M.E. 163 Yes 12-10-2022 -- Regular
157 MR KUMARESAN V 30 Assistant Professor Male M.E. 23 Yes 30-12-2022 -- Regular
158 MS KAVITHA R P 40 Assistant Professor Female M.Sc. 163 Yes 10-02-2023 -- Regular
159 MR NAGARAJ D 43 Assistant Professor Male M. Phil 114 Yes 28-04-2023 -- Regular
160 DR KEERTHANA R 28 Assistant Professor Female M.Sc. 22 Yes 20-02-2023 -- Regular
161 MR BOOMI P 40 Assistant Professor Male M. Phil 199 Yes 01-02-2023 -- Regular
162 MS JEYASHREE G 28 Assistant Professor Female M.Tech 16 Yes 30-08-2022 -- Regular
163 DR MANIKANDAN
R
56 Professor Male Ph.D 306 Yes 30-08-2022 -- Regular
164 MR BASKAR D 52 Assistant Professor Male M.E. 275 Yes 03-03-2023 -- Regular
165 MRS DEEPIKA D 37 Assistant Professor Female M.E. 23 Yes 28-12-2022 -- Regular
166 MRS JAYAKUMARI
DS
43 Assistant Professor Female M.E. 220 No 01-04-2023 29-05-2024 Regular
167 MRS RAJESWARI S 33 Assistant Professor Female M.Tech 32 Yes 04-03-2022 -- Regular
168 MR KALIMUTHAN C 46 Assistant Professor Male M.Tech 120 No 01-03-2023 30-06-2024 Regular
169 MR SUBRAMANI S 35 Assistant Professor Male M.E. 23 Yes 29-12-2022 -- Regular
170 MR ASHWIN
KUMAR KG
31 Assistant Professor Male M.E. 80 Yes 06-07-2017 -- Regular
171 MR ILANGOVAN B 36 Assistant Professor Male M.E. 124 No 01-03-2023 30-05-2024 Regular
172 MR ELAYARAJA
MP
38 Assistant Professor Male M. Phil 37 Yes 03-04-2023 -- Regular
173 MRS LAKSHMI B 35 Assistant Professor Female M. Phil 59 Yes 04-05-2023 -- Regular
174 MRS
GEETHANJALI G
31 Assistant Professor Female NET 42 Yes 22-02-2023 -- Regular
175 MS SATHYA K 33 Assistant Professor Female M. Phil 31 Yes 06-04-2023 -- Regular
176 MS
RENGALAKSHMI P
26 Assistant Professor Female M.Tech 23 No 31-08-2022 30-05-2024 Regular
177 MRS
MOHANAPRIYA M
38 Assistant Professor Female MCA 177 Yes 05-05-2014 -- Regular
178 MAHALAKSHMI V 28 Assistant Professor Female M.E. 40 Yes 19-07-2021 -- Regular
179 PRIYADHARSHINI
S
28 Assistant Professor Female M.E. 25 Yes 08-12-2021 -- Regular
180 GOWSALYA S 27 Assistant Professor Female M.Tech 31 No 30-12-2021 04-07-2024 Regular
181 GOWRI T 26 Assistant Professor Female M.Tech 31 Yes 30-12-2021 -- Regular
182 L KARUPPASAMY 40 Assistant Professor Male M.E. 98 Yes 05-02-2024 -- Regular
183 G MAHESWARI 41 Assistant Professor Female M. Phil 55 Yes 15-06-2023 -- Regular
184 K SANGEETHA 37 Assistant Professor Female M. Phil 150 Yes 05-02-2024 -- Regular
185 M ANANTHI 35 Assistant Professor Female M.E. 89 Yes 30-01-2024 -- Regular
186 S VIDHYA 36 Assistant Professor Female M.E. 150 Yes 05-02-2024 -- Regular
187 A
BHARATHIDASAN
37 Assistant Professor Male M.E. 20 Yes 01-07-2024 -- Regular
188 ANUMOL MATHAI 36 Assistant Professor Female Ph.D 25 Yes 12-04-2024 -- Regular
189 V JAMUNA 40 Assistant Professor Female M.E. 198 Yes 02-05-2024 -- Regular
190 I NANDHINI 30 Assistant Professor Female M.E. 35 Yes 20-09-2021 -- Regular
191 S RAMAMOORTHY 33 Assistant Professor Male M.E. 63 Yes 24-04-2023 -- Regular
192 P MURUGAIYAN 46 Assistant Professor Male Ph.D 102 No 13-03-2024 28-06-2024 Regular
193 A MATHESWARAN 38 Assistant Professor Male M. Phil 152 Yes 13-03-2024 -- Regular
194 S KANJANADEVI 37 Assistant Professor Female Ph.D 48 Yes 18-03-2024 -- Regular
195 L SIVARAJ 43 Assistant Professor Male M.Tech 96 Yes 25-07-2023 -- Regular
196 V VELUMANI 47 Associate Professor Male Ph.D 182 Yes 10-04-2024 -- Regular
197 T PAVITHRA 33 Assistant Professor Female M. Phil 52 Yes 08-02-2023 -- Regular
198 G
MANGAIYARKARA
SI
39 Assistant Professor Female M. Phil 155 Yes 05-09-2023 -- Regular
199 PRIYA 48 Assistant Professor Female Ph.D 222 Yes 04-10-2023 -- Regular
200 P INDIRA 40 Professor Female Ph.D 195 Yes 30-01-2024 -- Regular
201 S SRI
JAYABHARATHI
26 Assistant Professor Female M.E. 11 Yes 30-06-2023 -- Regular
202 B VEDHIKA 24 Assistant Professor Female M.Tech 11 No 19-01-2024 16-05-2024 Regular
203 K UTHRADEVI 33 Assistant Professor Female M.E. 42 Yes 29-01-2024 -- Regular
204 LAKSHMI PRABHA 33 Assistant Professor Female M.E. 3 Yes 05-02-2024 -- Regular
205 ANANDHA
KRISHNAN
47 Associate Professor Male Ph.D 96 Yes 19-01-2024 -- Regular
206 R SULAIMAN 39 Assistant Professor Male M.Tech 114 Yes 03-01-2024 -- Regular
207 G YASIKA 24 Assistant Professor Female M.E. 7 Yes 01-02-2024 -- Regular
208 T KOSALAIRAMAN 40 Assistant Professor Male M.Tech 168 Yes 04-01-2023 -- Regular
209 S LAKSHMIPRIYA 24 Assistant Professor Female M.E. 12 Yes 31-08-2023 -- Regular
210 J ARUNA 32 Assistant Professor Female Ph.D 96 Yes 06-06-2024 -- Regular
211 C MANONMANI 37 Assistant Professor Female M.Tech 169 Yes 02-05-2024 -- Regular
212 S
SADEESHVKUMAR
40 Assistant Professor Male M.E. 158 Yes 29-01-2024 -- Regular
213 A VIJAYAKUMAR 41 Professor Male Ph.D 198 Yes 31-08-2023 -- Regular
214 S REENA 39 Assistant Professor Female M.E. 109 No 29-01-2024 05-02-2024 Regular
215 R PAVITHRA 33 Assistant Professor Female M.E. 120 Yes 14-02-2024 -- Regular
216 P MIRNALINI 33 Assistant Professor Female M.E. 121 Yes 11-03-2024 -- Regular
217 T SARANYA 29 Assistant Professor Female M.Tech 67 Yes 20-03-2024 -- Regular
218 T GAYATHRI 36 Assistant Professor Female M.E. 133 Yes 01-04-2024 -- Regular
219 K SRINATH YADAV 40 Assistant Professor Male M.E. 132 Yes 19-05-2022 -- Regular
220 R
MUTHUCHELVAN
31 Assistant Professor Male M.E. 84 Yes 30-12-2022 -- Regular
221 R SUGANYA 32 Assistant Professor Female M.E. 97 Yes 03-04-2024 -- Regular
222 G NIRANJANA
DEVI
30 Assistant Professor Female M. Phil 82 Yes 10-07-2023 -- Regular
223 K
UMAMAHESWARI
37 Assistant Professor Female M.E. 132 Yes 15-06-2023 -- Regular
224 G PRAVEENA 33 Assistant Professor Female M.Tech 112 Yes 21-06-2023 -- Regular
225 B ARTHI 28 Assistant Professor Female M.E. 12 Yes 30-08-2023 -- Regular
226 K MOHAMED
YOUNUS
46 Assistant Professor Male M.E. 144 Yes 17-07-2023 -- Regular
227 M BHUVANESWARI 31 Assistant Professor Female M.E. 74 Yes 31-08-2023 -- Regular
228 A JERMAN
SHIYAMALA
27 Assistant Professor Female M.E. 51 No 14-11-2023 13-06-2024 Regular
229 M ARUN 42 Assistant Professor Male M.E. 216 No 02-08-2023 08-10-2024 Regular
230 V VETRISELVAN 46 Assistant Professor Male M.Tech 144 Yes 17-07-2023 -- Regular
231 D SURESH 44 Assistant Professor Male M.E. 242 Yes 26-07-2023 -- Regular
232 THIRUMALAIKANN
AN
36 Assistant Professor Male M.E. 99 No 26-07-2023 11-07-2024 Regular
233 S HARSHA
NANDHINI
24 Assistant Professor Female M.Tech 10 Yes 21-08-2023 -- Regular
234 R GOWMEKHA 27 Assistant Professor Female M.E. 61 No 08-12-2021 30-05-2024 Regular
235 R DEVAKI 44 Assistant Professor Female M.Tech 97 No 23-02-2022 30-05-2024 Regular
236 VIJAIKUMAR
PATHAK
34 Assistant Professor Male Ph.D 53 No 21-08-2023 30-04-2024 Regular
237 B BHARATHI 36 Assistant Professor Female M.Tech 124 Yes 10-01-2024 -- Regular
238 V YUVARAJ 42 Associate Professor Male Ph.D 180 Yes 10-01-2024 -- Regular
239 T RAMESHKUMAR 49 Associate Professor Male Ph.D 293 Yes 12-01-2024 -- Regular
240 M A
SUNDARAMAHLIN
GHAM
31 Assistant Professor Male Ph.D 84 Yes 19-01-2024 -- Regular
241 NIHARIKA SINGH 40 Assistant Professor Female Ph.D 86 Yes 31-01-2024 -- Regular
242 S YUVARANI 34 Assistant Professor Female M.E. 121 Yes 19-01-2024 -- Regular
243 C BHUVANESWARI 32 Assistant Professor Female M.E. 109 Yes 22-01-2024 -- Regular
244 G GOWTHAM
SIDDHARTH
30 Assistant Professor Male M.Tech 26 Yes 31-08-2023 -- Regular
245 A MURUGAVEL 37 Assistant Professor Male M.E. 121 Yes 28-06-2023 -- Regular
246 A MEENAKSHI
SUNDAR
37 Assistant Professor Male Ph.D 165 Yes 09-06-2023 -- Regular
247 Dr T SRINATH 34 Assistant Professor Male Ph.D 62 No 28-08-2023 30-05-2024 Regular
248 V RADHIKA 48 Associate Professor Female Ph.D 214 No 20-10-2023 06-02-2024 Regular
249 P SANGAIYA 40 Assistant Professor Male Ph.D 162 Yes 15-07-2023 -- Regular
250 S KAMALI 27 Assistant Professor Female M.E. 44 Yes 17-06-2023 -- Regular
251 YAMINIPRIYA 31 Assistant Professor Female M.E. 45 Yes 16-03-2023 -- Regular
252 PUNITHA 35 Assistant Professor Female M.E. 120 Yes 16-03-2023 -- Regular
253 DR VENKADESH 44 Professor Male Ph.D 204 No 01-06-2023 30-04-2024 Regular
254 DR RINESH 38 Associate Professor Male Ph.D 156 No 31-08-2023 30-04-2024 Regular
255  GNANA
SUBRAMANIAN A
37 Assistant Professor Male M. Phil 145 Yes 01-08-2022 -- Regular
256 GANESAN T 45 Associate Professor Male M.E. 245 Yes 06-05-2023 -- Regular
257 CHELLAMUTHU P
S
52 Assistant Professor Male M.E. 54 Yes 01-06-2023 -- Regular
258 MALATHI B 40 Assistant Professor Female M. Phil 41 Yes 06-06-2023 -- Regular
259 UMAMAHESWARI
R
30 Assistant Professor Female M.E. 75 Yes 02-06-2023 -- Regular
260 DR RAJASEKARAN
E
58 Professor Male Ph.D 347 Yes 01-06-2018 -- Regular
261 RAMESH BABU K D 47 Assistant Professor Male M. Phil 255 Yes 04-05-2022 -- Regular
262 NITHYA C 35 Assistant Professor Female M.E. 111 Yes 01-09-2021 -- Regular
263 MRS REBECCAL
ANGEL M
32 Assistant Professor Female M.E. 133 Yes 31-03-2023 -- Regular
264 KARTHEESWARAN
R
32 Assistant Professor Male M.E. 131 Yes 22-06-2015 -- Regular
265 Dr ARULKUMAR P 43 Professor Male Ph.D 247 Yes 18-01-2021 -- Regular
266 NANDHINIDEVI S 43 Assistant Professor Female M.E. 184 Yes 18-01-2021 -- Regular
267 MRS
NAGALAKSHMI N
42 Assistant Professor Female M.E. 32 No 03-08-2022 14-05-2024 Regular
268 MR
BHARATHIRAJA M
33 Assistant Professor Male M.E. 38 No 05-04-2023 30-05-2024 Regular
269 MRS AJITHA KALA
DS
48 Assistant Professor Female M.Tech 32 No 04-05-2022 30-06-2024 Regular
270 MR KARTHIKEYAN
J
32 Assistant Professor Male M.E. 25 Yes 24-02-2023 -- Regular
271 MS RAAJYA
VARDHINI A
37 Assistant Professor Female M.Tech 85 Yes 24-02-2023 -- Regular
272 MR VIJAYAKUMAR
G
31 Assistant Professor Male M.E. 32 Yes 26-05-2023 -- Regular
273 MRS NITHIYA K 35 Assistant Professor Female M.E. 32 Yes 01-06-2023 -- Regular
274 Dr
POORNACHANDRA
N R
31 Assistant Professor Male Ph.D 53 Yes 21-01-2021 -- Regular
275 PURUSHOTHAMAN
S
35 Assistant Professor Male M.E. 144 No 25-01-2021 22-06-2024 Regular
276 SARANYADEVI M 32 Assistant Professor Female M.E. 98 Yes 01-03-2021 -- Regular
277 KATHU SARA
RENJI
33 Assistant Professor Male M.E. 51 Yes 11-04-2021 -- Regular
278 RAVICHANDIRAN R 34 Assistant Professor Male M. Phil 96 Yes 04-05-2022 -- Regular
279 PRIYA M 36 Assistant Professor Female M.E. 108 No 30-04-2022 08-06-2024 Regular
280 MR ELUMALAI V 37 Assistant Professor Male M.E. 169 Yes 10-04-2023 -- Regular
281 DR
VIGNESHWARAN S
33 Assistant Professor Male Ph.D 86 Yes 15-02-2023 -- Regular
282 DR VENKATESH
KUMAR N
48 Assistant Professor Male Ph.D 221 Yes 03-05-2023 -- Regular
283 NELSON S 37 Assistant Professor Male M.E. 152 Yes 24-04-2018 -- Regular
284 REVATHI R 41 Assistant Professor Female M.E. 202 Yes 02-06-2022 -- Regular
285 MANI P 33 Assistant Professor Male M.E. 135 Yes 12-05-2022 -- Regular
286 GOBINATH S 45 Assistant Professor Male M.E. 90 Yes 29-08-2022 -- Regular

ANGHA BENCH : a Suite with One Million
Compilable C Benchmarks for Code-Size Reduction
Anderson Faustino da Silva
Department of Informatics
UEM, Brazil
anderson@din.uem.br
Bruno Conde Kind
Department of Computer Science
UFMG, Brazil
condekind@dcc.ufmg.br
Jos´e Wesley de Souza Magalh ˜aes
Department of Computer Science
UFMG, Brazil
josewesleysouza@dcc.ufmg.br
Jerˆonimo Nunes Rocha
Department of Computer Science
UFMG, Brazil
jeronimonunes@dcc.ufmg.br
Breno Campos Ferreira Guimar ˜aes
Department of Computer Science
UFMG, Brazil
brenosfg@dcc.ufmg.br
Fernando Magno Quint ˜ao Pereira
Department of Computer Science
UFMG, Brazil
fernando@dcc.ufmg.br
Abstract—A predictive compiler uses properties of a program
to decide how to optimize it. The compiler is trained on a
collection of programs to derive a model which determines its
actions in face of unknown codes. One of the challenges of
predictive compilation is how to ﬁnd good training sets. Regard-
less of the programming language, the availability of human-
made benchmarks is limited. Moreover, current synthesizers
produce code that is very different from actual programs, and
mining compilable code from open repositories is difﬁcult, due
to program dependencies. In this paper, we use a combination
of web crawling and type inference to overcome these problems
for the C programming language. We use a type reconstructor
based on Hindley-Milner’s algorithm to produce ANGHA BENCH ,
a virtually unlimited collection of real-world compilable C pro-
grams. Although A NGHA BENCH programs are not executable,
they can be transformed into object ﬁles by any C compliant
compiler. Therefore, they can be used to train compilers for
code size reduction. We have used thousands of A NGHA BENCH
programs to train YACOS, a predictive compiler based on LLVM.
The version of Y ACOS autotuned with A NGHA BENCH generates
binaries for the LLVM test suite over 10% smaller than clang
-Oz. It compresses code impervious even to the state-of-the-art
Function Sequence Alignment technique published in 2019, as it
does not require large binaries to work well.
Index Terms—Benchmark, Repository, Synthesis, Training
I. I NTRODUCTION
The growing popularity of stochastic classiﬁcation tech-
niques is contributing to making compiler autotuning an
effective approach to the generation of efﬁcient programs [1],
[2], [3]. Autotuning is implemented as follows. A compiler
is trained on a collection of programs, and uses knowledge
acquired during training to optimize unseen codes. Samples
from the known collection of programs are compiled in dif-
ferent ways, and the best results, given some objective function
such as running time or size, are recorded. The compiler
uses this model to decide which optimizations to apply on
an unknown program Pu. Autotuning has been shown to be
effective along different dimensions of code efﬁciency, such
as running time [4], [5], [6], [7], [8], energy consumption [9],
[10], code size [11], [12], [13], hardware usage [14], [15], and
the size-speed relation [16], [17], for instance.
a) The Problem Posed by the Lack of Benchmarks:
A common shortcoming in this ﬁeld, extensively discussed
by Cummins et al. [14], is the small size of typical training
sets. Cummins et al. have analyzed 25 research papers pub-
lished between 2013 and 2016, from four conferences: CGO,
HiPC, PACT, and PPoPP. They observe that “ the average
number of benchmarks used in each paper [is] 17 ”. This
result, although at ﬁrst surprising, should not be unexpected.
Typical benchmarks contain a small number of programs:
SPEC CINT2006 [18] contains 12, SPEC CFP2006 [18]
contains 17, Parsec [19] contains 13, Rodinia [20] contains 23,
Polybench [21] contains 30, cBench [22] contains 30, and NPB
v.1 [23] contains 8. The problem with these small numbers is,
in the words of Cummins et al., that “heuristics learned on one
benchmark suite fail to generalize across other suites ”. Wang
and O’Boyle subsume well the essence of the problem: ” The
most immediate problem continues to be gathering enough suf-
ﬁcient high quality training data. Although there are numerous
benchmark sites publicly available, the number of programs
available is relatively sparse compared to the number that a
typical compiler will encounter in its lifetime. ” [3]
To circumvent the obstacle posed by a perceived lack of
benchmarks, compiler researchers resort to program genera-
tion. With such purpose, automatically constructed programs
have been used to tune compiler heuristics in speciﬁc sce-
narios [24], [10], [25], [26], [27]. However, these programs
cannot be easily employed in general purpose compilers: they
consist of micro-kernels that exercise particular aspects of the
target hardware or of the target programming language. As an
example, Sreelatha et al. [10] generate code snippets to ﬁnd
optimum constants for their code generation approach. Each
program performs one action several times, be it to access
memory, to synchronize threads, to force branch mispredic-
tions, etc. Such behavior, although beﬁtting Sreelatha et al.’s
needs, is unlikely to occur in real-world programs.
b) Our Contributions: We bring forward a new technique
to generate compilable benchmarks for C. As we explain in
Section III, we mine C code from open-source repositories. Al-
though an obvious solution to the synthesis of benchmarks, this
approach is not common due to one fundamental shortcoming:
it is difﬁcult to compile code downloaded from repositories
automatically, due to program dependencies. In the words of
Cummins et al. [14]: “preparing each of the thousands of open
source projects to be directly applicable for learning compiler
heuristics would be an insurmountable task.” In this paper, we
show how to compile these codes without human intervention.
Key to the success of this endeavor is type reconstruction. We
use PsycheC, a type inference engine for C [28], to ﬁll up
all the missing dependencies of code mined from the internet.
This combination of code crawler and type reconstruction
yields a collection of compilable programs that is practically
unbounded.
Summary of Results. We call A NGHA BENCH the collec-
tion of C benchmarks that we synthesize. Every benchmark
is compilable, albeit non-executable. A NGHA BENCH can be
parsed by any C analyzer, can be converted into intermediate
representations such as LLVM IR and gcc Gimple, and can be
translated into object ﬁles. A NGHA BENCH supports compiler
tuning for code size reduction, and lets researchers study
properties of real-world programs via static code analyses.
Such possibilities are summarized in the following list of
contributions:
• Reconstructor: Section III-A describes the infrastructure
that we use to obtain compilable C programs from public
repositories. This combination of web crawler and type
inference engine is able to produce one million compil-
able C functions in about one week, including the time
to download ﬁles and reconstruct types.
• Distribution: We provide a public collection of over
one million compilable C ﬁles, organized as single-
function and multi-function benchmarks. As we explain
in Section III-B, this collection is browsable—search
being guided by a vector of features that we extract from
the LLVM representation [29] of each program.
• Applications: Section IV-B shows that A NGHA BENCH
predicts well the behavior of compiler optimizations. In
Section IV-D, we use A NGHA BENCH to train Y ACOS,
a framework implemented by Filho et al. [6], [30] to
ﬁnd good optimization sequences for LLVM. Consider-
ing code size as the objective functions, we show that
ANGHA BENCH yields a training set 45.33% and 36.77%
more effective than programs generated by CS MITH [31]
and LDRG EN [32].
• Optimization: We have used A NGHA BENCH to produce
a code reduction tool, A NGHA Z, that improves clang-Oz
by 11.1% on average. To compile an unseen program Pu,
ANGHA Z searches a database of optimized functions for
the program Pk the closest to Pu, given a well-known
distance metric [5]. As discussed in Section IV-E, by
applying onto Pu optimizations known to be effective on
Pk, ANGHA Z can reduce codes impervious to even Rocha
et al.’s state-of-the-art approach published in CGO19’s
distinguished paper [33].
c) Why compilation matters: We call “compilable” a ﬁle
known in the C speciﬁcation as a translation unit [34, Sec-
5.1.1.1]. From a translation unit a C compiler can produce
an object ﬁle. There are analyses that can be performed on
the source code of programs, without the need to produce
compilable code [35], [36], [37]. However, all the analyses and
applications that we present in Section IV require compilable
code, because these techniques either run on LLVM bytecodes
or on machine code. In Section IV-F we show that type
inference is essential to give us a large quantity of compilable
benchmarks. Without type inference, we cannot go from the
program’s abstract syntax tree to object code, due to missing
dependencies.
d) The beneﬁts of a large code base: The reader could
think that it is simple to analyze any partial C function, even
if not compilable, as long as it is syntactically valid. This
statement is not true. The C grammar is not context free;
hence, most parsers, including clang’s and gcc’s, require all
the dependencies in place, otherwise, statements like T*c
become ambiguous: is T a type, or the ﬁrst operand of a
multiplication? Other ambiguities exist [28]. Having this large
and unambiguous code base opens up many opportunities to
understand real-world C programs. We have performed some
analysis to this end, which, for the sake of space, we discuss in
an extended technical report that accompanies this paper [38].
II. O VERVIEW
A. Predictive Compilation
As mentioned in Section I, a predictive compiler relies on
properties of known programs to approximate properties of
unknown programs. The collection formed by all the known
programs is called the Training Set. Predictions, in this context,
consist in matching static program properties , also called
features, with compilation actions 1. The concept of static
program property has been deﬁned in previous work; however,
because this is a central notion to this paper, we recall its
deﬁnition, using the notation proposed by Pereira et al. [39]:
Deﬁnition 1 (Static Program Feature) . Given a program P, a
static program feature f(P) is any characteristic of P, with
the following attributes:
• Static: f(P) depends only on the syntax of P;
• Consistent: if f(P) =x and f(P) =x′, then x = x′;
• Available: f(P) can be computed in polynomial time.
An ordered sequence of program features determines a
feature vector. The set of every possible feature vector gives us
a feature space. Such space can be explored in many different
ways. For instance, because it abides by Euclidean Laws, it is
sound to deﬁne distance between vectors. Example 1 illustrates
these concepts.
1Previous work on predictive compilation, such as Zanella et al’s [30], also
talk about dynamic program features. These are properties observed during
the execution of programs. In this paper, we focus entirely on static program
properties, as the benchmarks that we produce are not meant to run.
Example 1. Figure 1 shows a three-dimentional feature space.
The three features that form it are Number of Instructions,
Number of Storesand Loop Depth, i.e., the depth level of
the innermost natural loop in the program.
  0  20  40  60  80 100 120
 50 100 150 200 250
0
1
2
3
4
(93, 4, 234)
Number of stores
Number of instructions
Most nested loop
0  1  0  1  1
(88, 2, 191)
0  1  1  0  1
tailcallelimloop-simplifyscalarizerlicmloop-rotate
Feature vector
Optimization vector
Fig. 1. Training a predictive compiler.
The training phase of a predictive compiler consists in
a search, not necessarily exhaustive, for the most adequate
compilation action for each program in the training set. The
notion of “most adequate action” depends on two factors: (i)
the objective function that guides the search; and (ii) the rep-
resentation of the action. Typical objective functions include
runtime, size and energy consumption. Common representa-
tions include tuples and lists of optimizations. In the former
case, the order of application of an optimization is ﬁxed—
what varies is the occurrence or not of the optimization [6],
[40]. In the latter, any permutation of a known universe of
optimizations is acceptable [41], [42], [43].
Example 2. The property space seen in Figure 1 contains
twelve programs, each one represented as a dot. The ﬁgure
shows the feature vectors of two programs. The best tuple of
optimizations, from a universe of ﬁve candidates, for each one
of these two programs is also shown. A zero means that the
optimization is inactive; a one means that it should be applied
onto that program. Such tuples can be found using different
heuristics, including exhaustive search. In this example, we
assume that the objective function is size; thus, a tuple t1 is
better than a tuple t2 when the optimizations in t1 reduce code
size more than the optimizations in t2.
Once a compiler is trained, it can be used to optimize
unknown programs. Optimizations, in this case, are based on
approximations: the behavior observed in the training set is
used to approximate the behavior of the unseen code. There
are many ways to implement these approximations: neural
networks, supporting vector machines, decision trees, etc.
Example 3 uses one of such techniques: classiﬁcation based
on K-nearest neighbors [44], to perform predictions.
  0  20  40  60  80 100 120
 50 100 150 200 250
0
1
2
3
4
Number of stores
Number of instructions Most nested loop
Unknown program
(25, 1, 75)
(27, 0, 51) 11001(18, 0, 55) 11001
(3, 1, 63) 10001
(48, 1, 88) 11011 11001
Prediction
Fig. 2. Performing predictions.
Example 3. Figure 2 shows how the K-Nearest Neigh-
bors algorithm can predict the best optimization tuple for
a program. Given an unknown program with feature vector
(stores = 25, innermost = 1, instructions = 75), we ﬁnd
the four closest programs to this vector. The predictor activates
the ith optimization if said optimization is active among the
closest neighbors, and turns it off otherwise.
B. The Need for Benchmarks
If the training collection is small, then large chunks of the
feature space will remain uncovered by the known codes.
Compilers can still perform predictions; however, the infor-
mation available during training might not approximate the
behavior of unseen programs. To circumvent this problem,
researchers often use synthetic benchmark suites. In this
section, we analyze some of these approaches.
a) DeepSmith: One of the most successful benchmark
generators in use today is D EEP SMITH [45], an evolution
of Cummins et al. ’s CLG EN [14]. D EEP SMITH has been
able to produce realistic OpenCL programs. It is meant to
be programming language agnostic; however, our attempts to
use it towards generating C, instead of OpenCL, programs met
with no success. Below, we narrate three of our experiences.
In every case, we use compilable C programs drawn from a
collection of half-a-million samples mined from open-source
repositories as the initial training corpus:
• Training set : 30,000 randomly chosen C ﬁles. 107,264
candidate strings generated in 15 hours using a seed
function signature with one argument. Results: Nine
programs could be compiled. The largest LLVM bytecode
had ﬁve instructions (Clang -O0).
• Training set: 30,000 randomly chosen C ﬁles. Genera-
tion: 131,760 candidate strings in 30 hours using a seed
signature with four arguments. Results: 1,178 programs
could be successfully compiled. The largest program had
six lines of code, and 36 LLVM instructions.
• Training set: the 10,000 largest C ﬁles in the available
collection. Generation: 54,912 candidate strings gener-
ated in 10 hours using a seed function signature with
four arguments. Results: Seventeen programs could be
successfully compiled. The largest program had ﬁve lines
of code, and 16 LLVM instructions.
b) Compiler Fuzzers: A compiler fuzzer produces ran-
dom programs to uncover bugs in compilers. The most suc-
cessful tool of this sort is CS MITH [31]. Programs generated
by CS MITH have revealed hundreds of bugs in the LLVM
infrastructure, and dozens in gcc’s. LDRG EN, another tool of
similar purposes, has also been effectively employed to ﬁnd
bugs in different compilers. Although tremendously successful
as bug-ﬁnding resources, fuzzers are not meant to be used
to generate training data for predictive compilers. Programs
generated by fuzzers like CSMITH and LDRG EN tend to differ
from real-world codes. Thus, properties inferred from them
may not generalize to programs written by people. The next
example supports this statement with empirical data.
Example 4. Figure 3 shows the relation between stores and
loads in the 275 programs in the LLVM test suite. For
each store, we ﬁnd 3.35 load instructions. This analysis was
performed in LLVM bytecodes compiled with -O0, but opti-
mized with mem2reg. This optimization, mem2reg, has been
used in this experiment to remove memory access operations
related to stack-allocated variables. Without this optimization,
binaries would contain too many loads and stores without
a counterpart in the source code. Analyzing 10K programs
generated by CS MITH , we ﬁnd the inverse behavior: 0.47
loads per store. 10K programs produced by LDRG EN fare no
better: they contain only one store instruction. The programs
synthesized by D EEP SMITH (1,204 samples) approximate the
ratio found in the actual benchmarks: for each store, we ﬁnd
2.97 load instructions. Nevertheless, they are too small: the
largest program contains only two store instructions.
III. T HE ANGHA BENCH COLLECTION
A. The Program Reconstruction Framework
We developed a completely automated process for gener-
ating compilable programs from open source projects. This
infrastructure has three major components: (i) Repository
Crawler; (ii) Function Extractor; and (iii) Type Inference
Engine. Each of these steps is described in the rest of this
section.
a) The Repository Crawler: The ﬁrst stage in our frame-
work consists in gathering the source-code from which we
shall build benchmarks. Programs are mined from GitHub,
via a web crawler. We ﬁlter out projects tagged as using the C
programming language, then sort them by popularity (we use
GitHub stars as a metric of popularity). The crawler traverses
a preﬁx of this sorted list, whose length is determined by the
user, cloning each of the repositories in order. The codebase
of each repository is cleaned to remove ﬁles which are not
C source or header ﬁles. This corpus of C programs is then
provided as input to the Function Extractor.
b) The Function Extractor: The function extractor sep-
arates mined ﬁles into a collection of would-be programs,
consisting of one C function per ﬁle. The extractor is a clang
plugin, which runs after Clang’s Abstract Syntax Tree building
step. It traverses the program’s AST, looking for function
declarations. If a declaration is found and has a matching
deﬁnition, we outline its implementation to a separate ﬁle. The
plugin can run in two modes: it can either create a ﬁle for each
function found, or one single ﬁle that aggregates all function
deﬁnitions found within that input source ﬁle. Clang builds an
AST for a program even if errors occur during compilation.
However, unless dependencies can be solved, it cannot move
from this point towards a ﬁnal object ﬁle. Example 5 illustrates
some issues that prevent compilation.
Example 5. Figure 4 shows a function extracted from the
source code of the Tox peer-to-peer messaging application.
This function (without the declarations in the grey box) is not
compilable, namely because it calls another function whose
declaration is unavailable in line 7, and contains references to
an unknown type BS LIST in lines 6, 7 and 12.
c) The Type Inference Engine: Once we have a large
number of candidate programs, the next challenge is to make
them compilable. To solve issues that prevent compilation,
such as those seen in Example 5, we run each program through
the PsycheC type inference engine [28]. PsycheC will ﬁll
the missing pieces within the candidate program, generating a
version of its code that compiles.
Example 6. The grey box in Figure 4 shows the result of
running the function bs_list_find through PsycheC. The
resulting program has a function declaration for the missing
function ﬁnd, as well as a valid deﬁnition for the missing
type BS LIST. This is all the absent information that prevented
compilation of the original code. Therefore, any C compiler
can successfully compile this new version without errors.
On the Preservation of Information. PsycheC does not
change the control ﬂow graph of programs. However, changes
may happen in the types of variables: ﬁelds present in structs
and unions in the original program might be omitted in the
reconstructed code, or might be declared with different types.
Omissions happen if these ﬁelds are not used. Thus, the
intermediate representations of the original and reconstructed
programs are identical, except for the types of the variables.
B. The Code Distribution Framework
To distribute the programs assembled using the techniques
seen in Section III-A, we have created a public website.
Different benchmark suites can be downloaded from it. All
these collections include only compilable codes. Compilation
has been certiﬁed using gcc 7.5, LLVM v.6, v.8 and v10.
Currently, we distribute the following suites:
0 14K
15K
0
0 12K
12K
0
400
0
0 400 0 20
5
0
Number of Loads
Number of Stores
Number of Stores
Number of Stores
Number of Stores
Number of Loads Number of Loads Number of Loads
LLVM Test Suite CSmith LDRGen
DeepSmithslope = 0.298 slope = -0.470 slope = 0.000
slope = 0.337
Fig. 3. A comparison between the number of stores and loads found in different benchmark collections. To ease visual comparison, each plot shows the line
(in pink) produced for the programs in the LLVM test suite and the main diagonal (in grey), i.e., X = Y .
int bs_list_ﬁnd
  (const BS_LIST *list, const uint8_t *data) {
  int r = ﬁnd(list, data);
  //return only -1 and positive values
  if (r < 0) {
    return -1;
  }
  return list->ids[r];
}
6
7
8
9
10
11
12
13
1
2
3
4
typedef int uint8_t;
struct TYPE_4__ { int* ids; };
typedef struct TYPE_4__ BS_LIST ;
int ﬁnd (BS_LIST const*, int const*);
Fig. 4. The code outside the grey area is an example of non-compilable
candidate program extracted from the toxcore repository. The code in the
grey area was introduced by PsycheC, to ensure compilation.
• The A NGHA BENCH collection:
– a set with 1M ﬁles containing single functions;
– a set with 530K ﬁles containing single functions;
– the 10K largest ﬁles from the above set;
– 15K ﬁles containing multiple functions.
• The LLVM test suite: 275 programs.
• The 10K largest programs among 530K programs gener-
ated with LDRG EN.
• The 10K largest programs among 530K programs gener-
ated with CS MITH .
• All the 1,204 programs that we have produced with
DEEP SMITH (see Section II).
Figure 5 reports data about the size of the programs in the
different collections. Program size is measured as the number
of instructions of these programs in the LLVM intermediate
representation. When converting programs to LLVM, we use
the mem2reg pass, to move to virtual registers all the program
variables that, otherwise, would be allocated on the stack.
a) A Protocol to Build Benchmarks: The largest
ANGHA BENCH collection contains 1,033,890 programs. For
faster experiments, we also provide smaller suites. To build
collections of N benchmarks, we follow the procedure below:
1) Let R be a list with the most popular git repositories,
in number of stars, with a majority of ﬁles in C,
in descending order, and let C be the collection of
benchmarks.
2) While C has less than N ﬁles, we:
a) Remove r, the current most popular repository
from R;
b) Add to C every function from r (see Section III-A).
b) The Code Search Engine: The public distribution
contains a code search engine, which lets users retrieve the
K closest program to a given code. Proximity is measured
CollectionQuantityGranularityMeanSDMedian1MFunctions61.6081.4136530KFunctions63.2497.323610KFunctions534.07336.3843315KWhole files266.64419.79119530KFunctions5,844.675,876.673,93310KFunctions20,190.903,649.0419,161530KFunctions1,950.541,216.822,00710KFunctions4,753.50322.654,668DeepSmith1KFunctions13.002.9812LLVM+SPEC06288Whole files6,737.3541,262.08584
AnghaBench
CSmith
LDRGen
Fig. 5. Instructions per benchmarks in the collections that we distribute. SD
is Standard Deviation.
as the Euclidean Distance computed on the feature vectors
introduced in Section II-A. We use LLVM to mine features
from the intermediate representation of programs. Today, users
can assemble vectors using features taken from a collection of
239 candidate program characteristics. We also provide three
predeﬁned feature vectors:
• LLVMSTS: 59 features produced by the LLVM’s
--stats ﬂag applied on clang -O0.
• NUMERICAL FEATURES : 43 features taken from Filho et
al. [6]. They project Namolaru et al. [5] features onto the
LLVM IR.
• DEFAULT: A seven-features vector for fast searches,
formed by (number of) instructions, stores, loads, basic
blocks, Control Flow edges , variables and variable uses
Our similarity search does not relate programs based on
semantic equivalence, `a la Alon et al. [46]. Rather, close
programs, in our context, are codes that tend to behave simi-
larly when exposed to the same set of compiler optimizations.
Therefore, it is assumed that the chosen feature vectors will
relate programs by the effect that optimizations provoke on
them. Notice, nevertheless, that while this is the objective, this
result is not guaranteed to hold, given the statistical nature of
every experiment described in this paper.
IV. E VALUATION
This section investigates the six research questions enu-
merated below. Further studies are available in the extended
version of this work [38]:
RQA What is the mining rate of the infrastructure de-
scribed in Section III-A?
RQB Can A NGHA BENCH better predict the impact of
compiler optimizations on programs?
RQC Can A NGHA BENCH better approximate the proper-
ties of human-written benchmarks than code pro-
duced by other program synthesizers?
RQD How does A NGHA BENCH compare to other syn-
thetic benchmarks as training data for a predictive
compiler?
RQE How does A NGHA Z compare to the state-of-the-art
binary reducer of Rocha et al [33]?
RQF Can we build a collection similar to A NGHA BENCH
without the support of type inference?
Ground Truth. RQB presupposes the existence of a ground
truth, that is, a collection of “typical” real-world bench-
marks. Different benchmarks have been used at different times
and places throughout the still short history of compilers.
Therefore, ﬁnding a universally acceptable ground truth is an
endeavor of improbable success. In this paper, we settle for a
collection of 288 programs, which includes every benchmark
available in the LLVM test collection (275 programs), plus the
programs in the SPEC CINT CPU2006 suite (13 programs).
In all, this collection gives us 1,450,035 lines of code, spread
across 31,366 functions from 2,315 ﬁles.
A. RQA: Mining Throughput
This section investigates the rate in which the infrastructure
from Section III-A produces valid benchmarks. A benchmark
is considered valid when we can use both clang and gcc to
convert it to an object ﬁle.
Methodology: We set up our framework to collect and re-
construct code, until a threshold of 1,000,000 compilable
programs had been reached. The metric used for ranking
repositories by popularity was GitHub’s star feature. We
executed the extraction-reconstruction process in parallel on
an 8-core Intel i7-3770, with 16 GBs of RAM, running
Ubuntu 16.04. We set a maximum execution timeout of 5
seconds for the type inference’s constraint-solving step, as its
uniﬁcation algorithm has a potentially exponential worst-case
performance [28]. We were concerned with answering two
questions about our framework’s performance:
• How long does it take, on average, to generate a compi-
lable program?
• What is the success rate of the program reconstructor?
Discussion: To reach the threshold of 1,000,000 programs, our
framework collected code from 148 repositories. It produced
1,044,023 compilable programs in 145 hours. This gives
us an average rate of one benchmark per 0.5 seconds. In
total, 1,882,687 candidate functions were extracted. Thus, the
success rate for the reconstruction process was approximately
55.5%. Only 3,666 reconstructions failed due to the timeout.
The most common reasons for failures were unpreprocessed
macros that were not syntactically valid in C.
B. RQB: Predicting Compiler Behavior
The goal of this section is to support the thesis that
ANGHA BENCH predicts more accurately the behavior of com-
piler optimizations than other synthetic benchmarks suites.
Methodology: We shall investigate the code size reduction
obtained by two different optimization levels of clang: -O1
and -O3, when applied on different benchmark collections.
The choice for these two optimization levels is arbitrary; the
ANGHA BENCH distribution contains the same study for other
optimization levels, and results are similar. We use the Mean
Square Prediction Error (MSPE) as a measure of accuracy,
deﬁned as (predicted value −observed value)2. To carry out
predictions, we ﬁt a linear model M in a synthetic benchmark,
relating the size of programs when compiled with clang -O0
and clang -O1 (or -O3). M is then used to predict program
size in the ground-truth collection.
Discussion: Figure 6 summarizes our ﬁndings. Each ﬁgure
shows a main diagonal, and the regression line. Because
optimizations tend to remove instructions, the regression line
is always under the main diagonal. We give the slope of the
regression line (Slp), and a measure of accuracy ( Err). Err
denotes the ratio between the MSPE of a given collection
(ANGHA BENCH , CS MITH , LDRG EN, D EEP SMITH ) and the
MSPE of the ground-truth. We compute Err as follows:
1) Let ℓ be the regression line that we ﬁt into a given
synthetic collection.
2) Let ℓg be the regression line that we ﬁt into the ground-
truth collection.
3) Let mg be the MSPE that we obtain using ℓg on the
ground-truth collection (this is the standard deﬁnition of
MSPE).
4) Let m be the MSPE that we obtain when using ℓ also
on the ground-truth collection.
5) We let Err = m/mg.
The lower the value of Err, the better the predictor used
to compute it. We report the error for the second largest
collection in A NGHA BENCH with 530K single functions (the
collection with 1M samples makes this experiment too slow).
Figure 6 shows that A NGHA BENCH ’s error is one order of
magnitude smaller than errors produced by the other collec-
tions. Programs from the CS MITH and from the LDRG EN
collections are easy to optimize. They are made to execute
without undeﬁned behavior. To avoid undeﬁnedness, they
contain hardcoded inputs. The excess of constants leads to
code that is easy to simplify. The programs generated by
DEEP SMITH are also easy to optimize, although they do not
contain hard-coded inputs (they are not meant to run). In this
case, optimization opportunities come from an excess of dead-
code.
C. RQC: Code Similarity
This paper defends the thesis that A NGHA BENCH approx-
imates more closely the properties of real-world code than
other synthetic program sets. This section provides evidence
that such is the case. To this end, we shall rely on the measure
of “distance” between programs, which we discuss below.
500 500
4000 4000
0
50
0
400
0
50
0
400
AnghaBench -O1
(530K single funcs.)
CSmith -O1 CSmith -O3
LDRGen -O1 LDRGen -O3
600 600
60
0
60
0
Slp: 0.63 
Err: 1.60
Slp: 0.20 
Err: 33.39
Slp: 0.40 
Err: 8.57
Slp: 0.58 
Err: 1.12
Slp: 0.25 
Err: 11.33
Slp: 0.40 
Err: 4.72
DeepSmith -O1 DeepSmith -O3
Slp: 0.29 
Err: 20.56
Slp: 0.29 
Err: 9.35
40
0
40
0
400 400
1000 1000
0
100
0
100LLVM Test Suite -O1 LLVM Test Suite -O3
#insts (x100) at -O1
#insts at -O3
#insts (x100) at -O0 #insts at -O0
Slp: 0.58 
Err: 1.00
Slp: 0.61 
Err: 1.00
AnghaBench -O3
(530K single funcs.)
Fig. 6. The effect of compiler optimizations on the size of programs.
Instructions are measured in hundreds.
Methodology: There exists a rich assortment of functions to
measure the distance between programs [47]. We have adopted
two of them: the Euclidean distance on numerical feature
vectors, and the MCoeff relation between two programs pro-
posed by Filho et al. [6]. The latter is a metric that measures
how similarly two programs respond to the same sequence
of compiler optimizations. Our choice is purely pragmatic:
the infrastructure described in Section III-B already simpliﬁes
the computation of these two metrics. Furthermore, these two
functions meet the following properties, assuming that p1 and
p2 are programs: (i) d(p1, p2) ≥0, (ii) d(p1, p2) = d(p2, p1),
and (iii) d(p1, p2) ≤ d(p1, p3) + d(p3, p2), for any p3 /∈
{p1, p2}. Finally, although there exist similarity metrics that
are likely to be more expressive than those that we chose, e.g.,
`a la D EEP SIM [48], they tend to be more costly to compute.
Discussion: Figure 7 shows the distance of each one of the
288 programs in the ground-truth to the different synthetic
collections. The distance of a program pg in the ground-truth
collection to a collection C of synthetic benchmarks is given
by d(pg, pc), where pc is the program in C that is the closest to
pg, and d is either the Euclidean distance on numerical feature
vectors, or MCoeff.
The average Euclidean distance from the ground-truth
collection to A NGHA BENCH is 6.7x shorter than to the
CSMITH (available in our public distribution) collection, and
33.3x shorter than to the LDRG EN (available in our public
distribution—See Sec. III-B) collection. This difference is
smaller once we consider MCoeff, but it is still noticeable.
ANGHA BENCH is approximately 21% and 29% closer to the
ground-truth than the CS MITH collection and the LDRG EN
collection, respectively.
Diversity. We use the notions of distance seen in this section
to argue that A NGHA BENCH is more diverse than the other
synthetic collections that we use. The data in Figure 8 supports
this statement. For each point (Nb, K)(c,d) in Figure 8, we as-
sume that c is either A NGHA BENCH , CS MITH , or LDRG EN,
and d is a distance function, e.g., numerical features, or
MCoeff. In this case, Nb is the number of benchmarks from
collection c that wins as one of the K closest programs to
some benchmark in the ground-truth collection. Thus, a very
homogeneous collection c would have all the programs with
the same features; leading to (Nb, K)(c,d) = K always. A
very diverse collection, in turn, would give us (Nb, K)(c,d) =
Ng×K, where Ng is the number of benchmarks in the ground-
truth set. Therefore, according to these deﬁnitions, the larger
(Nb, K)(c,d), the more heterogenous is collection c, and the
better it covers the feature space.
D. RQD: Predictive Compilation
This section provides evidence that A NGHA BENCH yields
better training sets than other benchmark generators. To this
end, we have used different synthetic collections of bench-
marks to train Y ACOS, the predictive compiler implemented
by Filho et al. [6], [30]. Y ACOS uses a heuristic based on
Kennedy et al. [49]’s particle swarm optimization (PSO) to
ﬁnd good optimization sequences.
Numerical 
Feature 
vectors
MCoeﬀ
(the larger, the more similar)
AnghaBench CSmith
LDRGen
AnghaBench CSmith
LDRGen
Fig. 7. Distance between the ground-truth and different synthetic benchmarks.
Each collection is formed by the 10K largest programs out of a pool of 530K
candidates. Red dots are programs from the SPEC CPU2006 suite.
Methodology - Benchmarks : We have trained Y ACOS using
the three collections mentioned in Section III-B, each with
10K ﬁles: A NGHA BENCH , CS MITH and LDRG EN. For the
last two collections, we produced 530K ﬁles, and took the
10K largest. We adopted this expedient for fairness, as the 10K
programs from ANGHA BENCH were selected from a collection
of 530K benchmarks. Programs in the CS MITH collection are
larger; hence, training based on them takes much longer: on
average, three hours per ﬁle. The other two collections yield
faster training time: on average 20 minutes per ﬁle. In total,
training took 87 days, using 16 cores running at 3.40GHz.
Methodology - Training : Training consists in associating
each benchmark in the training set with 100 sequences of
optimizations—each sequence with 60 optimizations. These
optimizations come from a set of 83 passes available in LLVM.
Searching the feature space, in this setting, is the problem of
associating with the feature vector of a program P the best
list of optimizations for P. Y ACOS’s PSO is parameterized
0	
200	
400	
600	
800	
1000	
1200	
1400	
1600	
1	 2	 3	 4	 5	 6	 7	 8	 9	 10	
Angha-NM	 CSmith-NM	 LdrGen-NM	
Angha-MC	 CSmith-MC	 LdrGen-MC	
211 benchmarks in our synthetic 
collection win as the closest code to 
some ground-truth program. 1,493 
programs win among the 10 closest 
to some ground-truth program.
Fig. 8. Number of programs that win for K-nearest of some program in the
ground-truth. The X-axis is K.
with an initial population of 100 particles, which evolves for
10 generations. Nevertheless, the exact implementation of this
heuristic is immaterial to the understanding of this paper—it
sufﬁces to know that its quality varies with the training set.
Methodology - Prediction : Once training is complete,
YACOS uses similarity search (i.e., KNN with K=1) to ﬁnd
the known program Pk in the training set that is the closest to
a given unknown program Pu. We have used the two measures
of distance seen in Section IV-C: Euclidean distance applied
on numerical feature vectors, and MCoeff. We let Sk be the
list of optimizations associated with Pk. From Sk, Y ACOS
can build different list of optimizations to be applied onto Pu
following four strategies:
• Elite: let Se ⊆Sk be a subset of Sk (|Sk|= 100) formed
by the lists that improve on clang -Oz. We apply on Pu
every sequence in Se, and keep the best result. If Se is
empty, this strategy has no effect on Pu.
• JX, X ∈{1, 10, 100}:] we apply on Pu Just the X best
sequences in Sk, and keep the best result. The sequences
in Sk are ordered by their effect on Pk. The most effective
optimization comes ﬁrst.
Discussion: For validation, we use the ground-truth collection
mentioned in Section IV-C. Figures 9 and 10 summarize the
results of this experiment. A winning strategy is the pair in
{Elite, J1, J10, J100}×{ANGHA BENCH , CSMITH , LDRG EN}
that yields the smallest bytecodes when applied to the val-
idation set. We omit results involving benchmarks produced
by D EEP SMITH . Due to their simplicity, their feature vectors
contain mostly zeros. Boldface fonts mark winners consid-
ering minimum, maximum, mean and median code reduction.
ANGHA BENCH wins in most cases. When using the Euclidean
Distance with the Elite choice, A NGHA BENCH reduces code
by 10.6% on average. If we use MCoeff, gains are higher:
11.1%. These results were not obtained in small programs:
the ground-truth used as validation contains the 13 integer
programs from SPEC CPU2006.
The gray cells in Figures 9 and 10 contain results for
AnghaBench CSmith LDRGen
min
avg
med
max
min
avg
med
max
#P
Elite J1 J10 J100
-46-86-83-46-15-61-61-33-121-145-103-3310.62.18.610.70.13.98.310.64.4-7.36.510.78.43.57.48.40.03.97.68.53.5-4.94.88.341.233.236.441.212.034.235.339.632.616.632.644.40.00.10.00.02.90.00.00.30.30.80.00.611.18.49.711.07.87.69.511.08.24.68.411.18.56.57.98.58.06.37.78.66.53.36.69.041.233.236.441.212.034.235.339.632.616.632.644.42801952692821221126728219939250280Elite J1 J10 J100 Elite J1 J10 J100
Fig. 9. Percentage of code reduction, measured over number of LLVM
bytecodes, produced from the ground truth using Y ACOS parameterized with
the Euclidean distance. #P reports the number of programs in which we have
observed positive results over clang -Oz. Averages are geometric mean.
programs in which we could ﬁnd a list of optimization better
than clang -Oz. If we consider average values for these
programs, then the difference between A NGHA BENCH and
the other collections is less apparent, as we are counting
only positive results. However, A NGHA BENCH is able to
ﬁnd non-trivial lists of optimization for more programs than
the other synthetic collections. For instance, using the Elite
strategy with Euclidean distance (Fig. 9), Y ACOS trained with
ANGHA BENCH was able to reduce the size of 280 programs
(compared with clang -Oz), vs only 12 if we train Y ACOS
with CS MITH , and 199, if we train Y ACOS with LDRG EN.
CSmith LDRGen
min
avg
med
max
min
avg
med
max
#P
Elite J1 J10 J100 Elite J1 J10 J100 Elite J1 J10 J100
-38-98-61-38-21-148-59-47-55-176-70-4511.14.89.611.10.7-2.56.510.09.70.48.510.79.34.88.19.30.0-1.24.58.27.83.26.88.641.241.241.241.231.421.538.038.537.437.437.437.40.80.00.30.80.20.10.10.20.10.20.10.111.58.310.611.410.55.67.910.610.87.69.511.09.56.48.79.57.24.25.38.38.35.57.18.841.241.241.241.231.421.538.038.537.437.437.437.428223127328324112251277269192271282
AnghaBench
Fig. 10. Percentage of reduction (same as Fig. 9) achieved on the ground
truth using the MCoeff distance.
Using the ground-truth as the training set. We have also
used the ground-truth (288 programs, including SPEC CINT
CPUT2006) as the training set to predict good optimization
sequences to itself. In this experiment, we have used a leave-
one-out methodology with the MCoeff distance. Leave-one-
out is applied per benchmark. The ground-truth is formed
by the combination of 36 different benchmark collections.
Thus, we use 35 suites as the training set, and one suite
as the test set. On the J1 strategy, A NGHA BENCH ﬁnds a
better list of optimization than clang -Oz for 231 out of
288 programs. The ground-truth beats clang -Oz in only 30
programs. The average size reduction across all programs is
4.8% using A NGHA BENCH (Figure 10) vs -14.2% using the
ground-truth. On J10, A NGHA BENCH outperforms clang -Oz
in 273 programs, and the ground-truth in 109. On J100, they
obtain statistically similar results: 283 vs 284 improvements.
We emphasize that the ground-truth is much smaller than
ANGHA BENCH : we are comparing a training set of 10,000
samples with a training set with strictly less than 288.
E. RQE: Code Size Reduction
This section provides some perspective on the code size
reduction achieved by a compiler trained with ANGHA BENCH .
To this end, we analyze the effects of this compiler on
MIBENCH [50]. MIBENCH has been used by Rocha et al. [33]
as a challenging case study. Rocha et al. have designed and
implemented a technique to reduce code size by merging
common sequences of instructions. Their Function Merging by
Sequence Alignment (FMSA) approach excels when applied
onto large code bases, as there are more opportunities for
merging redundant code. However, their technique yields
poor results when applied onto small programs—a natural
consequence of a statistical lack of redundancies. Rocha et
al. have used M IBENCH to demonstrate this last point.
Methodology: We compile M IBENCH with Y ACOS trained
with the 10K largest programs from A NGHA BENCH . Program
similarity is measured with the Euclidean distance. Euclidean
distance is used in this experiment because it is the default pro-
gram metric used in Y ACOS. We use the subset of M IBENCH
available in the LLVM test suite.
Discussion: Figure 11 reports the results that we have obtained
after compiling MIBENCH with YACOS. The baseline is clang
-Os. This is the same baseline adopted by Rocha et al. For
reference, Figure 11 also reports, on top, the percentages of
code size reduction observed by Rocha et al. Numbers refer
to the size of the object ﬁle produced after compilation.
-10	
-5	
0	
5	
10	
15	
20	
Percentage of code reduction reported by Rocha et al., relative to clang -Os
Percentage of code reduction relative to clang -Ospatriciadijkstraqsortsha
basicmath
CRC32
FFT
rijndael
jpegsusan
strsearchbitcount
4.2
0.7
20.6
2.0
1.2
0.0
0.0
0.0
0.0
0.0
0.0
0.0
blowﬁsh
0.0
0.84
1.75
2.94
4.34 5.62
5.85
8.53
10.54
10.96
11.0211.42
12.71
18.39
Elite
J1
J10
J100
Fig. 11. Percentage of code size reduction achieved on M IBENCH , measured
over size of object ﬁles, in bytes. Numbers on top are percentage of reduction
reported by Rocha et al. [33]. Numbers at the bottom are percentage of
reduction achieved by Y ACOS, using the Elite approach.
The amount of code size reduction obtained by Y ACOS
tends to be higher than the reduction achieved by FMSA; how-
ever, the bad performance of FMSA on M IBENCH had been
noticed by Rocha et al. [33]. As we have mentioned, FMSA
does better the larger the target code base, for in this case,
there will be more redundancies to explore 2. We emphasize
that these numbers cannot be directly compared: they were
not produced in the same empirical setup. Nevertheless, in at
least eight benchmarks from M IBENCH , where Rocha et al.
have reported no gains, we could observe reductions of 6.0%
on average (geo-mean), compared to clang -Os. Another fact
that this experiment highlights is that clang -Os and clang -
Oz still leave much room for improvement. In benchmarks like
bitcount and strsearch it is possible to ﬁnd sequences
of optimizations that are almost twice as efﬁcient as clang -
Os, and approximately 8% better than clang -Oz. Finally, it
must be understood that FMSA is a compiler optimization,
whereas our approach is rather a replacement to the LLVM
pass manager. In other words, it would have been possible to
include FMSA as one of the optimizations of Y ACOS’s search
strategy. We have not performed this new experiment simply
because FMSA is not available by default in LLVM.
F . RQF: The Role of Type Inference
Provided that we can mine open-source repositories, and
there are so many of them, one could expect that some of
these programs naturally compile without type reconstruction.
In this section, we show that such code base is worse than
ANGHA BENCH when used to train a predictive compiler.
1) The Need for Type Inference: One of the benchmark col-
lections distributed in our website consists of 529,498 C func-
tions and their respective LLVM bytecodes. This collection of
over half-a-million compilable benchmarks has been produced
out of 54,431 ﬁles taken from 79 open source repositories. Out
of these ﬁles, we extracted 698,449 functions, sizes varying
from one line to 45,263 lines of code ( Radare2’s assembler).
Thus, we produced an initial code base of 698,449 C ﬁles,
each ﬁle containing a single function. We run PsycheC with
a timeout of 30 seconds on this code base. PsycheC has
been able to reconstruct dependencies of 529,498 functions;
thus, ensuring their compilation. Compilation consists in the
generation of an object ﬁle out of the function— a task
performed with clang 6.0.1.
Out of the 698,449 functions, 31,935 were directly compi-
lable as-is, that is, without PsycheC’s inference. To perform
automatic compilation, we invoke clang on a preprocessed C
ﬁle containing an individual function extracted as-is. Hence,
without type inference, we could ensure compilation of 4.6%
of the programs. With type inference, we could ensure com-
pilation of 75,8% of all the programs. Failures to reconstruct
types were mostly due to macros that were not syntactically
valid in C without preprocessing.
2Rocha et al. have published a new code size reduction technique that
extends sequence alignment to SSA-form programs [51]. We believe that their
new algorithm, SalSSA, is the most effective size reduction technique in use
today. When applied onto M IBENCH , SalSSA achieves a geo-mean reduction
of 1.4% to 1.6%; twice as much as FMSA. However, a direct comparison
with our work is not possible, for they reported results only for ARM.
2) On the Consequence of using Small Functions: As we
have mentioned, we can compile automatically less than 5%
of the functions that we download, even considering all the
dependencies in the C ﬁles where these functions exist. Nev-
ertheless, given that we can download millions of functions,
5% is already enough to give us a non-negligible number of
benchmarks. However, these compilable functions tend to be
very small. The median number of LLVM bytecodes is seven
(in contrast to 36, using type inference). Said functions are
unlikely to contain features such as arrays of structs, type casts,
recursive types, double pointer dereferences, etc. It is typical
that developers separate deﬁnitions from implementations into
different ﬁles in the C ecosystem. It sufﬁces to have one
missing ﬁle or the wrong version of one library, and the entire
program will fail to compile.
basicmathbitcountblowﬁshCRC32
FFT
dijkstra
jpeq
patriciaqsortrijndael
sha
stringsearch
susan
-15	
-10	
-5	
0	
5	
10	
15	
20	
25	
30	
Myst.	 Comp.	
Percentage of variation on number of LLVM 
bytecodes (instructions) in the ﬁnal linked ﬁle
Angha
Fig. 12. Performance of Y ACOS (with the Elite choice) on the M IBENCH
programs. We use two training sets: A NGHA BENCH (10K largest functions)
and the functions that we can compile without type inference. Percentage of
reduction is measured with regards to clang -Oz on the number of LLVM
bytecodes in optimized programs.
Due to their small size, the naturally compilable functions
are not effective to tune a compiler. Figure 12 demonstrates
this fact with data. The ﬁgure compares Y ACOS, when trained
with our collection of half-a-million functions (from which
we selected the 10K largest samples), or with the collection
of 31.9K directly compilable functions (from which, again,
we selected the 10K largest functions). The small size of
the latter group prevents it from being an effective training
set. For instance, using these functions, Y ACOS reduces
the size of MiBench’s bitcount by 10%, whereas using
ANGHA BENCH , it achieves 16.9%. In susan, the naturally
compilable functions lead to an increase of code size (5.4%),
whereas A NGHA BENCH reduces it by 1.7%. Although there
are benchmarks in M IBENCH where the naturally compilable
functions lead to more compression, these gains are close to
those obtained by A NGHA BENCH , and seldom occur.
V. R ELATED WORK
This paper deals with synthesis of benchmarks and code size
reduction. Because the former has been discussed in Section II,
we now focus on the latter. Yet, we start our discussion
mentioning some work on the synthesis of benchmarks.
Synthesis of Benchmarks. The creation of synthetic bench-
marks has become a frequent focus of research. Random
program generators, such as CS MITH [31], LDRG EN [32] and
Orange3 [52], [53] have been successfully used to produce C
programs for stress-testing compilers, often ﬁnding correctness
bugs in industrial implementations [54]. Although conceived
to ﬁnd bugs, these tools have also been used to improve
the quality of the optimized code emitted by mainstream C
compilers [55], [56]. Nevertheless, such tools, given their
goals, are not designed to produce realistic code.
Recent effort to create human-like C programs has leveraged
Deep Learning techniques to generate code similar to real-
world examples. CLG EN [14] uses this approach to generate
OpenCL kernels, while D EEP SMITH [45] generalizes this
technique to other languages. Nonetheless, we have found that
DEEP SMITH has trouble synthesizing non-trivial C programs
when trained with corpora of open source projects, as we
have explained in Section II-B. We could not use it to train
YACOS, in Section IV-D, because the feature vectors of
its programs contained mostly zeros. Following a different
approach, Richards et al. have produced realistic JavaScript
benchmarks, out of monitored browser sections [57]. Type re-
construction is not an issue in this scenario, because JavaScript
is dynamically typed. One shortcoming of this modus operandi
is scalability, because it is not fully automatic. Richards’
technique still requires users to create a browsing section,
which will then be instrumented.
Code Size Reduction. Several compiler transformations have
code reduction as either the main goal or a desirable con-
sequence [33], [51], [58], [59], [60], [61], [62]. Often, such
techniques involve Code Factoring , the identiﬁcation of re-
dundant code within the program. Code motion techniques
search for identical instructions and merge them to avoid
redundancy [62], [63], [64], [63]. Function Merging is the
other main category in the ﬁeld. This optimization ﬁnds
functions that are semantically similar or equivalent, and
generates new functions to replace them [65], [66], [67]. It
is possible to merge even functions that are slightly different.
When functions that meet a similarity threshold are found, a
new procedure is created. The new function contains additional
control-ﬂow to choose between which original implementation
should be executed [33], [51], [68].
In spite of program compression being an old problem,
effective and elegant code size reduction techniques have been
discovered as recently as the current year [51]. Nevertheless,
the contributions of this paper are orthogonal to these tech-
niques, because we do not propose new optimizations. Rather,
our synthetic benchmarks can help a compiler identify when
each of these optimizations might be proﬁtable. To support
this observation, in Section IV-E we showed how to augment
a C compiler with knowledge extracted from our benchmarks.
Autotuning for Binary Size Reduction. Code size has been
a common objective function of predictive compilers. The
earliest works in this direction used genetic algorithms to
continually improve the size of the code generated for target
applications [42], [69], [70]. These early approaches evolve
a sequence of optimizations for each individual program;
thus, search runs until convergence for each program being
optimized. The technique described in section IV-E, on the
other hand, can simply ﬁnd the program in the training set that
better approximates the target application. Therefore, once the
predictive model has been trained, the impact on compilation
time is minimal.
VI. C ONCLUSION
This paper has presented a framework to produce compil-
able C programs out of open-source repositories, which we
have used to generate more than one million benchmarks.
Compilation is ensured via type inference. Different appli-
cations of these benchmarks have been shown, with empha-
sis on predictive size reduction. In this regard, we showed
that in benchmarks like bitcount and blowfish, from
MIBENCH , it is possible to ﬁnd sequences of optimizations
that are about 15% better than clang -Os, and approximately
8% better than clang -Oz.
a) License: Although the process of building the
compilable benchmarks is automatic, adding the licenses
to the ﬁles is not. To fulﬁll this task, we need to
ﬁnd the license(s) used in the repository, which is not
necessarily present in the program’s source code. We have
preserved the original licenses of a subset of 128,411 ﬁles
from the following repositories: FFmpeg, DeepMind,
openssl, SoftEtherVPN, libgit2, php-src,
radare2, darwin-xnu, mongoose, reactos, git,
nodemcu-firmware, redis, h2o, and obs-studio.
These repositories give us a total of 38.9K C ﬁles. These ﬁles
have been organized as an external LLVM test suite.
b) Further uses of ANGHA BENCH : In addition to the ap-
plications seen in this paper, we are aware of a few other uses
of A NGHA BENCH . For instance, A NGHA BENCH has been
used to: (i) stress-test two C-to-Verilog compilers: LEGUP and
VIVADO ; (ii) test routing algorithms that convert C programs
to FPGA circuits; (iii) ﬁne-tune register assignment heuristics;
(iv) compare the speed of C parsers; (v) check the effectiveness
of a termination checker, the U LTIMATE Automizer; and (vi)
scale up program synthesis, following an idea proposed by
Bornholt and Torlak [71].
c) Software: ANGHA BENCH is publicly available at http:
//cuda.dcc.ufmg.br/angha. In addition to the benchmarks, this
webpage contains links to the infrastructure used to build
them, including crawler, function extractor and type inference
engine [28].
ACKNOWLEDGEMENT
This work has been made possible by grants from different
research agencies, namely CNPq, CAPES and FAPEMIG. We
thank Luigi Soares and Augusto Noronha for reading a draft
of this paper. We also thank the CGO reviewers, for all the
time and expertise that they have put into our manuscript.
REFERENCES
[1] A. H. Ashouri, W. Killian, J. Cavazos, G. Palermo, and C. Silvano, “A
survey on compiler autotuning using machine learning,” Comput. Surv.,
vol. 51, no. 5, pp. 96:1–96:42, 2018.
[2] H. Leather and C. Cummins, “Machine learning in compilers: Past,
present and future,” in FDL. Washington, DC, USA: IEEE, 2020.
[3] Z. Wang and M. F. P. O’Boyle, “Machine learning in compiler opti-
mization,” Proceedings of the IEEE , vol. 106, no. 11, pp. 1879–1901,
2018.
[4] A. H. Ashouri, G. Mariani, G. Palermo, E. Park, J. Cavazos, and
C. Silvano, “COBAYN: Compiler autotuning framework using bayesian
networks,” TACO, vol. 13, no. 2, pp. 21:1–21:25, 2016. [Online].
Available: http://doi.acm.org/10.1145/2928270
[5] M. Namolaru, A. Cohen, G. Fursin, A. Zaks, and A. Freund, “Practical
aggregation of semantical program properties for machine learning based
optimization,” in CASES. New York, NY , USA: ACM, 2010, pp. 197–
206.
[6] J. F. Filho, L. G. A. Rodriguez, and A. F. da Silva, “Yet another
intelligent code-generating system: A ﬂexible and low-cost solution,”
J. Comput. Sci. Technol. , vol. 33, no. 5, pp. 940–965, 2018.
[7] P. M. Phothilimthana, A. Thakur, R. Bodik, and D. Dhurjati, “Scaling
up superoptimization,” in ASPLOS. New York, NY , USA: ACM, 2016,
pp. 297–310.
[8] T. C. de Souza Xavier and A. F. da Silva, “Exploration of compiler
optimization sequences using a hybrid approach,” Computing and Infor-
matics, vol. 37, no. 1, pp. 165–185, 2018.
[9] M. Novaes, V . Petrucci, A. Gamati ´e, and F. M. Q. a. Pereira, “Compiler-
assisted adaptive program scheduling in big.little systems: Poster,” in
PPoPP. New York, NY , USA: ACM, 2019, pp. 429–430.
[10] J. K. V . Sreelatha, S. Balachandran, and R. Nasre, “CHOAMP: cost
based hardware optimization for asymmetric multicore processors,”
Trans. Multi-Scale Computing Systems, vol. 4, no. 2, pp. 163–176, 2018.
[11] S. Bansal and A. Aiken, “Binary translation using peephole superopti-
mizers,” in OSDI. Berkeley, CA, USA: USENIX Association, 2008,
pp. 177–192.
[12] R. Bunel, A. Desmaison, M. P. Kumar, P. H. S. Torr, and P. Kohli,
“Learning to superoptimize programs,” in ICLR. Toulon, France:
OpenReview, 2017.
[13] E. Schkufza, R. Sharma, and A. Aiken, “Stochastic program optimiza-
tion,” Commun. ACM, vol. 59, no. 2, pp. 114–122, 2016.
[14] C. Cummins, P. Petoumenos, Z. Wang, and H. Leather, “Synthesizing
benchmarks for predictive modeling,” in CGO. Piscataway, NJ, USA:
IEEE, 2017, pp. 86–99.
[15] G. Poesia, B. C. F. Guimar ˜aes, F. Ferracioli, and F. M. Q. Pereira, “Static
placement of computation on heterogeneous devices,” PACMPL, vol. 1,
no. OOPSLA, pp. 50:1–50:28, 2017.
[16] D. Simon, J. Cavazos, C. Wimmer, and S. Kulkarni, “Automatic
construction of inlining heuristics using machine learning,” in CGO.
Washington, DC, USA: IEEE Computer Society, 2013, pp. 1–12.
[17] P. Zhao and J. N. Amaral, “To inline or not to inline? enhanced inlining
decisions,” in LCPC. Heidelberg, Germany: Springer, 2003, pp. 405–
419.
[18] J. L. Henning, “SPEC CPU2006 benchmark descriptions,” SIGARCH
Comput. Archit. News , vol. 34, no. 4, pp. 1–17, Sep. 2006. [Online].
Available: http://doi.acm.org/10.1145/1186736.1186737
[19] R. Bagrodia, R. Meyer, M. Takai, Y .-a. Chen, X. Zeng, J. Martin, and
H. Y . Song, “Parsec: A parallel simulation environment for complex
systems,” Computer, vol. 31, no. 10, pp. 77–85, 1998. [Online].
Available: https://doi.org/10.1109/2.722293
[20] S. Che, M. Boyer, J. Meng, D. Tarjan, J. W. Sheaffer, S.-H. Lee, and
K. Skadron, “Rodinia: A benchmark suite for heterogeneous computing,”
in IISWC. Washington, DC, USA: IEEE, 2009, pp. 44–54.
[21] L.-N. Pouchet and T. Yuki, “PolyBench/C 4.2.1: The polyhedral C
benchmark suite,” 2018. [Online]. Available: http://polybench.sf.net
[22] G. Fursin and O. Temam, “Collective optimization: A practical collab-
orative approach,” Trans. Archit. Code Optim. , vol. 7, no. 4, pp. 20:1–
20:29, 2010.
[23] D. Bailey, E. Barszcz, J. Barton, D. Browning, R. Carter,
L. Dagum, R. Fatoohi, P. Frederickson, T. Lasinski, R. Schreiber,
H. Simon, V . Venkatakrishnan, and S. Weeratunga, “The
NAS parallel benchmarks,” Int. J. High Perform. Comput.
Appl., vol. 5, no. 3, pp. 63–73, 1991. [Online]. Available:
http://dx.doi.org/10.1177/109434209100500306
[24] R. Barik, N. Farooqui, B. T. Lewis, C. Hu, and T. Shpeisman, “A
black-box approach to energy-aware scheduling on integrated cpu-gpu
systems,” in CGO. New York, NY , USA: ACM, 2016, pp. 70–81.
[25] Z. Wang and M. F. O’Boyle, “Mapping parallelism to multi-cores: A
machine learning based approach,” in PPoPP. New York, NY , USA:
ACM, 2009, pp. 75–84.
[26] Z. Wang and M. F. O Boyle, “Partitioning streaming parallelism for
multi-cores: A machine learning based approach,” in PACT. New York,
NY , USA: ACM, 2010, pp. 307–318.
[27] Y . Wen, Z. Wang, and M. F. P. O’Boyle, “Smart multi-task scheduling
for opencl programs on CPU/GPU heterogeneous platforms,” in HiPC.
Los Alamitos, CA, USA: IEEE, 2014, pp. 1–10.
[28] L. T. C. Melo, R. G. Ribeiro, M. R. de Ara ´ujo, and F. M. Q. a.
Pereira, “Inference of static semantics for incomplete c programs,”
Proc. ACM Program. Lang. , vol. 2, no. POPL, pp. 29:1–29:28, Dec.
2018. [Online]. Available: http://doi.acm.org/10.1145/3158117
[29] C. Lattner and V . Adve, “LLVM: A compilation framework for lifelong
program analysis & transformation,” in CGO. Washington, DC, USA:
IEEE, 2004, pp. 75–. [Online]. Available: http://dl.acm.org/citation.
cfm?id=977395.977673
[30] A. F. Zanella, A. F. da Silva, and F. M. Q. ao Pereira, “Y ACOS: a com-
plete infrastructure to the design and exploration of code optimization
sequences,” in SBLP. New York, NY , USA: ACM, 2020, pp. 56–63.
[31] X. Yang, Y . Chen, E. Eide, and J. Regehr, “Finding and understanding
bugs in c compilers,” in PLDI. New York, NY , USA: ACM, 2011, pp.
283–294.
[32] G. Barany, “Liveness-driven random program generation,” in LOPSTR.
Heidelberg, Germany: Springer, 2017, pp. 112–127.
[33] R. C. O. Rocha, P. Petoumenos, Z. Wang, M. Cole, and H. Leather,
“Function merging by sequence alignment,” in CGO. Piscataway, NJ,
USA: IEEE Press, 2019, pp. 149–163.
[34] ISO-Standard, “ISO/IEC 9899:tc3 - committee draft of the C99 stan-
dard,” 2011.
[35] R. Dyer, H. Rajan, and Y . Cai, Language Features for Software Evo-
lution and Aspect-Oriented Interfaces: An Exploratory Study . Berlin,
Heidelberg: Springer-Verlag, 2013, p. 148–183.
[36] R. Dyer, H. Rajan, H. A. Nguyen, and T. N. Nguyen, “Mining billions of
ast nodes to study actual and potential usage of java language features,”
in ICSE. New York, NY , USA: ACM, 2014, p. 779–790.
[37] J. Eyolfson and P. Lam, “How C++ developers use immutability decla-
rations: An empirical study,” in ICSE. Washington, DC, USA: IEEE,
2019, p. 362–372.
[38] A. F. da Silva, B. Kind, J. W. M. aes, J. Rocha, B. G. aes, and F. M. Q.
ao Pereira, “Anghabench: a synthetic collection of benchmarks mined
from open-source repositories,” Universidade Federal de Minas Gerais,
Tech. Rep. 01-2020, 2020.
[39] F. M. Q. a. Pereira, G. V . Leobas, and A. Gamati ´e, “Static prediction
of silent stores,” ACM Trans. Archit. Code Optim. , vol. 15, no. 4, pp.
44:1–44:26, 2018.
[40] G. Fursin, Y . Kashnikov, A. W. Memon, Z. Chamski, O. Temam,
M. Namolaru, E. Yom-Tov, B. Mendelson, A. Zaks, E. Courtois et al.,
“Milepost GCC: Machine learning enabled self-tuning compiler,” Inter-
national journal of parallel programming , vol. 39, no. 3, pp. 296–327,
2011.
[41] M. Stephenson, S. Amarasinghe, M. Martin, and U.-M. O’Reilly,
“Meta optimization: Improving compiler heuristics with machine
learning,” in Proceedings of the ACM SIGPLAN 2003 Conference on
Programming Language Design and Implementation , ser. PLDI ’03.
New York, NY , USA: ACM, 2003, pp. 77–90. [Online]. Available:
http://doi.acm.org/10.1145/781131.781141
[42] K. D. Cooper, P. J. Schielke, and D. Subramanian, “Optimizing for
reduced code space using genetic algorithms,” in Proceedings of the
ACM SIGPLAN 1999 Workshop on Languages, Compilers, and Tools for
Embedded Systems, ser. LCTES ’99. New York, NY , USA: ACM, 1999,
pp. 1–9. [Online]. Available: http://doi.acm.org/10.1145/314403.314414
[43] F. Agakov, E. Bonilla, J. Cavazos, B. Franke, G. Fursin, M. F. P.
O’Boyle, J. Thomson, M. Toussaint, and C. K. I. Williams, “Using
machine learning to focus iterative optimization,” in Proceedings of the
International Symposium on Code Generation and Optimization , ser.
CGO ’06. Washington, DC, USA: IEEE Computer Society, 2006, pp.
295–305. [Online]. Available: http://dx.doi.org/10.1109/CGO.2006.37
[44] T. Cover and P. Hart, “Nearest neighbor pattern classiﬁcation,” Trans.
Inf. Theor., vol. 13, no. 1, pp. 21–27, 2006.
[45] C. Cummins, P. Petoumenos, A. Murray, and H. Leather, “Compiler
fuzzing through deep learning,” in ISSTA. New York, NY , USA: ACM,
2018, pp. 95–105.
[46] U. Alon, M. Zilberstein, O. Levy, and E. Yahav, “Code2vec: Learning
distributed representations of code,” Proc. ACM Program. Lang., vol. 3,
no. POPL, pp. 40:1–40:29, 2019.
[47] R. Naseem, O. Maqbool, and S. Muhammad, “Improved similarity
measures for software clustering,” in CSMR. Washington, DC, USA:
IEEE Computer Society, 2011, pp. 45–54.
[48] G. Zhao and J. Huang, “DeepSim: Deep learning code functional
similarity,” in ESEC/FSE. New York, NY , USA: ACM, 2018, p.
141–151.
[49] J. Kennedy and R. Eberhart, “Particle swarm optimization,” in Proceed-
ings of the International Conference on Neural Networks , vol. 4, Nov
1995, pp. 1942–1948 vol.4.
[50] M. R. Guthaus, J. S. Ringenberg, D. Ernst, T. M. Austin, T. Mudge, and
R. B. Brown, “MiBench: A free, commercially representative embedded
benchmark suite,” in WWC. Washington, DC, USA: IEEE, 2001, pp.
3–14.
[51] R. C. O. Rocha, P. Petoumenos, Z. Wang, M. Cole, and H. Leather,
“Effective function merging in the ssa form,” in PLDI. New York, NY ,
USA: ACM, 2020, p. 854–868.
[52] E. Nagai, A. Hashimoto, and N. Ishiura, “Reinforcing random testing of
arithmetic optimization of c compilers by scaling up size and number of
expressions,” IPSJ Trans. System LSI Design Methodology , vol. 7, pp.
91–100, 2014.
[53] K. Nakamura and N. Ishiura, “Introducing loop statements in random
testing of c compilers based on expected value calculation,” in Proc. the
Workshop on Synthesis And System Integration of Mixed Information
Technologies (SASIMI 2015), 2015, pp. 226–227.
[54] J. Chen, J. Patra, M. Pradel, Y . Xiong, H. Zhang, D. Hao, and L. Zhang,
“A survey of compiler testing,”ACM Comput. Surv., vol. 53, no. 1, 2020.
[55] A. Hashimoto and N. Ishiura, “Detecting arithmetic optimization oppor-
tunities for c compilers by randomly generated equivalent programs,”
IPSJ Transactions on System LSI Design Methodology , vol. 9, pp. 21–
29, 2016.
[56] G. Barany, “Finding missed compiler optimizations by differential
testing,” in Proceedings of the 27th International Conference on
Compiler Construction , ser. CC 2018. New York, NY , USA:
ACM, 2018, pp. 82–92. [Online]. Available: http://doi.acm.org/10.1145/
3178372.3179521
[57] G. Richards, A. Gal, B. Eich, and J. Vitek, “Automated construction
of javascript benchmarks,” SIGPLAN Not., vol. 46, no. 10, p. 677–694,
2011.
[58] J. Knoop, O. R ¨uthing, and B. Steffen, “Partial dead code elimination,”
SIGPLAN Not. , vol. 29, no. 6, pp. 147–158, Jun. 1994. [Online].
Available: http://doi.acm.org/10.1145/773473.178256
[59] J. Cocke, “Global common subexpression elimination,” SIGPLAN
Not., vol. 5, no. 7, pp. 20–24, Jul. 1970. [Online]. Available:
http://doi.acm.org/10.1145/390013.808480
[60] J. Ernst, W. Evans, C. W. Fraser, T. A. Proebsting, and S. Lucco, “Code
compression,” in Proceedings of the ACM SIGPLAN 1997 Conference
on Programming Language Design and Implementation , ser. PLDI ’97.
New York, NY , USA: ACM, 1997, pp. 358–365. [Online]. Available:
http://doi.acm.org/10.1145/258915.258947
[61] A. S. Tanenbaum, H. van Staveren, and J. W. Stevenson, “Using
peephole optimization on intermediate code,” ACM Trans. Program.
Lang. Syst. , vol. 4, no. 1, pp. 21–36, Jan. 1982. [Online]. Available:
http://doi.acm.org/10.1145/357153.357155
[62] P. Briggs and K. D. Cooper, “Effective partial redundancy elimination,”
in Proceedings of the ACM SIGPLAN 1994 Conference on Programming
Language Design and Implementation , ser. PLDI ’94. New York,
NY , USA: ACM, 1994, pp. 159–170. [Online]. Available: http:
//doi.acm.org/10.1145/178243.178257
[63] G. L ´oki, ´A. Kiss, J. J ´asz, and ´A. Besz ´edes, “Code factoring in gcc,” in
Proceedings of the 2004 GCC Developers’ Summit , 2004, pp. 79–84.
[64] A. Dreweke, M. Worlein, I. Fischer, D. Schell, T. Meinl, and
M. Philippsen, “Graph-based procedural abstraction,” in Proceedings
of the International Symposium on Code Generation and Optimization ,
ser. CGO ’07. Washington, DC, USA: IEEE Computer Society, 2007,
pp. 259–270. [Online]. Available: https://doi.org/10.1109/CGO.2007.14
[65] S. Tallam, C. Coutant, I. L. Taylor, X. D. Li, and C. Demetriou, “Safe
icf: Pointer safe and unwinding aware identical code folding in gold,”
in GCC Developers Summit , 2010. [Online]. Available: http://gcc.gnu.
org/wiki/summit2010?action=AttachFile&do=view&target=tallam.pdf
[66] M. Liska, “Optimizing large applications,” arXiv, 2014.
[67] L. C. Infrastructure. (2019, nov) Mergefunctions pass, how it
works. [Online]. Available: http://llvm.org/docs/MergeFunctions.html#
mergefunctions-pass-how-it-works
[68] T. J. Edler von Koch, B. Franke, P. Bhandarkar, and A. Dasgupta,
“Exploiting function similarity for code size reduction,” in Proceedings
of the 2014 SIGPLAN/SIGBED Conference on Languages, Compilers
and Tools for Embedded Systems , ser. LCTES ’14. New York,
NY , USA: ACM, 2014, pp. 85–94. [Online]. Available: http:
//doi.acm.org/10.1145/2597809.2597811
[69] P. Kulkarni, W. Zhao, H. Moon, K. Cho, D. Whalley, J. Davidson,
M. Bailey, Y . Paek, and K. Gallivan, “Finding effective optimization
phase sequences,” in Proceedings of the 2003 ACM SIGPLAN
Conference on Language, Compiler, and Tool for Embedded Systems ,
ser. LCTES ’03. New York, NY , USA: ACM, 2003, pp. 12–23.
[Online]. Available: http://doi.acm.org/10.1145/780732.780735
[70] D. Fatiregun, M. Harman, and R. M. Hierons, “Evolving transformation
sequences using genetic algorithms,” in Source Code Analysis and
Manipulation, Fourth IEEE International Workshop on , Sep. 2004, pp.
65–74.
[71] J. Bornholt and E. Torlak, “Scaling program synthesis by exploiting
existing code,” 2015.

Bayesian Optimization is Superior to Random Search for
Machine Learning Hyperparameter Tuning:
Analysis of the Black-Box Optimization Challenge 2020
Ryan Turner
rturner@twitter.com
Twitter
David Eriksson
deriksson@fb.com
Facebook
Michael McCourt
mccourt@sigopt.com
SigOpt, an Intel company
Juha Kiili
juha@valohai.com
Valohai
Eero Laaksonen
eero@valohai.com
Valohai
Zhen Xu
xuzhen@4paradigm.com
4Paradigm
Isabelle Guyon
guyon@chalearn.org
ChaLearn
Abstract
This paper presents the results and insights from the black-box optimization
(BBO) challenge at NeurIPS 2020 which ran from July–October, 2020. The chal-
lenge emphasized the importance of evaluating derivative-free optimizers for tun-
ing the hyperparameters of machine learning models. This was the ﬁrst black-box
optimization challenge with a machine learning emphasis. It was based on tuning
(validation set) performance of standard machine learning models on real datasets.
This competition has widespread impact as black-box optimization (e.g., Bayesian
optimization) is relevant for hyperparameter tuning in almost every machine learn-
ing project as well as many applications outside of machine learning. The ﬁnal
leaderboard was determined using the optimization performance on held-out (hid-
den) objective functions, where the optimizers ran without human intervention.
Baselines were set using the default settings of several open-source black-box op-
timization packages as well as random search.
1 Introduction
In black-box optimization we aim to solve the problem minx∈Ω f(x), where f is a computation-
ally expensive black-box function and the domain Ω is commonly a hyper-rectangle. The fact that
evaluations are computationally expensive typically limits the number of evaluations of f to a few
hundred. In the black-box setting, no additional information is known about f and we observe no
ﬁrst- or second-order information when evaluatingf. This is commonly referred to as derivative-free
optimization.
Black-box optimization problems of this form appear everywhere. Most machine learning (ML)
models have hyperparameters that require tuning via black-box (i.e., derivative-free) optimiza-
tion [60]. Likewise, black-box optimization has wide applications in closely related areas such
as signal processing [63]. These black-box optimization problems are often solved using Bayesian
optimization (BO) methods [19]. BO methods rely on a (probabilistic) surrogate model for the ob-
jective function that provides a measure of uncertainty. This model is often a Gaussian process
(GP) [55], but other models such as Bayesian neural networks are also commonly used as long as
they provide a measure of uncertainty. Using this surrogate model, an acquisition function is used
to determine the most promising point to evaluate next, where popular options include expected
improvement (EI) [35], knowledge gradient (KG) [18], and entropy search (ES) [31]. There are
also other surrogate optimization methods that rely on deterministic surrogate models such as radial
basis functions [16, 66], see Forrester et al. [17] for an overview. The choice of surrogate model
and acquisition function are both problem-dependent and the goal of this challenge is to compare
arXiv:2104.10201v2  [cs.LG]  31 Aug 2021
different approaches over a large number of different problems. This was the ﬁrst challenge aiming
to ﬁnd the best black box optimizers specially for ML-related problems.
Moreover, many non-ML problems also beneﬁt from the use of BO. For example, chemical en-
gineering, materials discovery, manufacturing design, control systems and drug discovery are also
effective uses of BO [8, 9, 20, 23, 26, 33, 36, 49, 64]. Even real-world experiments, such as opti-
mal web interfaces evaluated using A/B tests, have been optimized using BO [42]. Hyperparameter
tuning is such a demanded tool that all the major cloud platforms offer parameter tuning tools [57].
Additionally, there are small businesses (e.g., SigOpt) offering hyperparameter optimization as a
service (OPTaaS).
Evolutionary algorithms (EAs) [69] are another popular approach to black-box is optimization; they
include methods such as differential evolution (DE), genetic algorithms (GAs), and the covariance
matrix adaptation evolution strategy (CMA-ES) [27]. However, EAs generally require thousands of
evaluations to be competitive with more sample-efﬁcient methods such as BO, which may not be
feasible when f is computationally expensive. Other possible options for solving black-box opti-
mization problems include the Nelder-Mead algorithm [51] and the popular quasi-Newton method
BFGS [44] with gradients obtained via ﬁnite differences.
While BO has gained a lot of traction over the last few years and open-source packages have become
very mature and robust, the study by Bouthillier and Varoquaux [7] shows that there is still work
to do to convince ML practitioners to use it to tune their algorithms. Surveying authors of papers
published at NeurIPS 2019 and ICLR 2020 they found that while 80% of the NeurIPS papers and
88% of the ICLR papers tuned their hyperparameters, the vast majority used manual tuning, random
search, or grid search (GS). In fact, only 7% of the NeurIPS papers and 6% of the ICLR papers
used a different method such as BO. Motivated by the original NeurIPS 2019 survey we decided to
submit a proposal for a competition with the goal of decisively showing that BO and similar methods
are superior choices over random search and grid search for tuning hyperparameters of ML models.
This competition showed this advantage decisively and also provided guidance on how to select the
best performing black-box optimization method.
2 Background
There have been benchmarks and competitions on compare black-box optimization such as the an-
nual COCO competition [28]. However, the COCO competition focused on synthetic problems like
the Ackely and Rastrigin functions. Although being well suited towards unit test-like development
cycles, these problems are systematically different from many real-world tasks: They often have
highly correlated dimensions and high dynamic range. Many real-world tasks have irrelevant di-
mensions as well as complex noise patterns. Therefore, the goal of our competition was based on
tuning for real-world ML tasks.
In this sense, the AutoML competition series [46] is perhaps the closest competition to our com-
petition. Other similar AutoML competitions were the NIPS 2006 model selection game 1 [24,
Sec. 1.5.1] and The “Beat AutoSKLearn” challenge 2. However, there are some key differences
between black-box optimization and AutoML. In AutoML, it is up to the algorithm to determine the
search space and which ML method to try (e.g., MLP or random forest). To make matters more con-
crete, the following gives an example of a search space conﬁguration in a scikit-learn SVM tuning
problem that was provided to the optimization algorithms in our competition:
svm_cfg = {
"C": {" type ": " real ", " space ": " log ", " range ": (1.0, 1e3 )},
" gamma ": {" type ": " real ", " space ": " log ", " range ": (1e-4, 1e-3)},
" tol ": {" type ": " real ", " space ": " log ", " range ": (1e-5, 1e-1)},
}
Furthermore, an AutoML algorithm is given a training dataset, and must perform as well as possible
on the test set without any human ever seeing the training set. By contrast, in black-box optimization,
the search space is assigned to the algorithm. The algorithm is given access only to a black-box
1http://clopinet.com/isabelle/Projects/NIPS2006/
2https://worksheets.codalab.org/worksheets/0x18a13ee4b0db4e098679f390bbd97fb2
2
objective function, which in the general case may not even be ML-related. There is no training (or
validation) data given to a black-box optimization algorithm.
There has been a related series of competitions to the COCO benchmark at the GECCO conference:
BBComp [47, 48]. Although the BBComp objective functions are black-box, they have typically
been synthetic. The exception is the EMO 2017 edition of BBComp, which used real-world prob-
lems. However, this challenge differed in several ways: EMO used only 10 test problems, of which 8
were multi-objective physical simulations and therefore not representative of typical ML use cases.
There were also no practice problems given to the participants before evaluating them on the 10 test
problems. In the spirit of ML, we think it is important to have a “training set” to iterate on. Finally,
the BBComp explicitly allowed closed-source submissions and human-in-the-loop optimizers. In
this challenge, all submissions had to be open-sourced to be eligible for a prize, which is essential
as advancing the ﬁeld is a goal of this challenge.
3 Competition Setup
This is a new competition as there have been no ML-oriented black-box optimization competitions
in the past.3 The most similar competition, the previously mentioned AutoML series, maintains key
differences (endemic between any black-box optimization competition and any AutoML competi-
tion). In AutoML, the algorithm gets to pick the search space while in black-box optimization, it
is assigned to the algorithm by the user. The user often has intuition for reasonable hyperparameter
ranges in ML problems, which makes it possible to frame it as a black-box optimization problem.
While the survey by Bouthillier and Varoquaux [7] showed that most ML researchers tune their
hyperparameters, they do so using simple methods such as random search, grid search, and manual
tuning. In addition, there are a large number of possible algorithms from BO and EAs, so concluding
which method is preferable on real-world problems is a clear technical advance.
There were three different sets of problems where the participants’ algorithms were evaluated:
1) The local practice problems; these problems were included in Bayesmark [62] and run locally on
the participants’ machines with full visibility into the models and data. 2) Thefeedback leaderboard
problems; these problems were used to calculate the leaderboard score on the website during the
three month feedback phase of the challenge. These problems were run in a cloud environment and
completely hidden from the participant. 3) The ﬁnal leaderboard problems; these problems were
used to calculate the ﬁnal leaderboard score, which was posted on the website after the closing of
submissions and used to determine prizes. These problems were also hidden like the feedback prob-
lems. To prevent overﬁtting, the participants’ algorithms were only evaluated a single time on the
ﬁnal leaderboard problems. Problems were randomly split between feedback and ﬁnal. The set of
problems is discussed in more detail in Section 3.2.
However, the local practice problems were made from tuning ML models on public scikit-learn
datasets, and therefore, not a random split of the other problems.
We provided astarter kit4 to allow the participants to: 1) Locally score their submissions on the local
practice problems with a single shell command; and 2) package their submissions into a compliant
zip ﬁle for submission with a single shell command. All baseline and evaluation code runs on CPU
without the need for a GPU.
3.1 Baselines
The starter kit provided examples using the default conﬁgurations of several different optimization
packages: Hyperopt [5], Nevergrad [54], OpenTuner [1], pySOT [14], Scikit-Optimize [30], and
TuRBO [15], which serve as baselines. However, the most natural single reference point is the
performance of (included) random search.
These baselines were meant to give participants a good starting point, but there are many other
possible packages such as Ax/BoTorch [3, 4], Cornell-MOE [67], Dragonﬂy [37], Emukit [52],
GPFlowOpt [41], GPyOpt [61], pycma [29], RBFOpt [10], RoBO [39], ProBO [50], and
Spearmint [60].
3https://bbochallenge.com/
4https://github.com/rdturnermtl/bbo challenge starter kit/
3
3.2 The Optimization Problems
The “dataset” for this competition was a collection of optimization problems. Therefore, we fol-
lowed the same protocols that were followed in the AutoML competition for a “dataset of datasets”.
A collection of public scikit-learn datasets5 were provided to the participants in local practice prob-
lems.
We obtain novel optimization problems via the Cartesian product of datasets, ML models, and eval-
uation metrics. For example, the following are all examples of optimization problems,
• Tune a GBDT on MNIST evaluated on accuracy on the validation set.
• Tune logistic regression on MNIST evaluated on log loss on the validation set.
• Tune an MLP on Boston housing evaluated on RMSE on the validation set.
Thus, informally,
{set of opt. problems}= {set of models}×{set of datasets}×{set of loss functions}.
The Cartesian product is violated slightly as different loss functions are used for classiﬁcation and
regression problems. Keeping many of these datasets completely hidden allows us to have test
(optimization) problems unknown to the participants for both the feedback and ﬁnal leaderboards.
The search space varied by ML model and was provided to the algorithm by the benchmark. We
summarize this space of problems across phases in Table 1.
Table 1: A summary of the different model, loss, and data set combinations that made up the dif-
ferent phases. Note that only the (local) practice problems were visible to the participants; both the
feedback and ﬁnal problems were hidden.
Practice Feedback Final
Models 9 6 6
Loss functions 2 2 2
Datasets 6 5 5
This structure was chosen because in industrial settings we are often more concerned withwall clock
time than raw CPU time. To keep this wall clock time reasonable, each submission was allowed a
budget of 30 minutes per optimization run. Therefore, algorithms that perform well when making
parallel suggestions are highly desirable. Much of the BO literature is focused on limiting the num-
ber function evaluations rather than iterations. The limitation of 16 rounds of guesses (iterations) is
very small in the broader world of optimization.
3.3 Evaluation Metrics
In this challenge we use the open-source package Bayesmark [62] to execute all the experiments
inside the docker and for scoring. The Bayesmark package has routines designed to deal with the
subtleties of scoring black-box optimization algorithms. Scoring an optimization algorithm on a
single problem is easy; simply take the minimum found by the optimizer aftern function evaluations.
Averaging over repeated trials can be done in noisy settings. Each repeated trial of a particular
problem is known as a study.
However, averaging over many different problems becomes more subtle. We cannot simply average
scores because they are all on different scales (units); such an approach builds in an arbitrary implicit
weighting across problems. The Bayesmark package has a scoring system designed to deal with this
problem. First, we normalize the performance on each problem so that a single RS suggestion has an
average score of 1, and the global optimum has a score of 0. Then, we can average the performance
across multiple problems because the units are all the same.
Appendix B contains the equations for scoring taken directly from the Bayesmark documentation.
The (feedback and ﬁnal) leaderboard score is from Equation 7: 100 ×(1 −norm-mean-perf). Ac-
cordingly, the scoring is normalized such that scores vary from 0 (The optimizer on average is about
5The example scikit-learn datasets were: digits, iris, wine, breast, boston, and diabetes.
4
the same as a single random search guess) to 100 (The optimizer ﬁnds the best known optimum
every single time). This places the scoring on a normalized, unitless, and intuitive scale.
3.4 Evaluation Environment
The scoring and execution of runs in this challenge was handled using the open source Bayesmark
package. Bayesmark is designed around optimizers that use a suggest-observe framework; and it
provides a Python abstract class with the API. This suggest-observe framework is known as an
open-loop optimizer. The participant’s algorithm suggestsk = 8 guesses to be evaluated in parallel
via the suggest function. The benchmark then evaluates the k different guesses and returns them
to the algorithm via the observe function. The user just needs to provide a Python ﬁle with the
Bayesmark AbstractOptimizer class implemented. This open-loop setup is desirable as it gives
the user more ﬂexibility on how (and if) to evaluate a suggestion. Furthermore, if the black box
being optimized is a real experiment (not a function in code) an open-loop setup is required.
In the context of a black-box optimization competition, each “data point” in the “training” or “test”
set is an independent black-box optimization problem. This is similar to the AutoML competition
where each “data point” is a dataset. For each optimization problem, the algorithm had access only
to a search space speciﬁcation and a black-box that evaluates the objective function.
For the local practice optimization problems, the evaluation of the objective functions was done on
the participants’ hardware. However, for the test problems (the feedback and ﬁnal leaderboards),
the objective function had to be hidden, and therefore the participants’ submissions were run inside
a Docker container in a cloud environment. The optimizers had a total of 640 seconds compute
time for making suggestions on each problem (16 iterations with batch size of 8); or 40 seconds per
iteration. Optimizers exceeding the time limits were cut off from making further suggestions and the
best optima found before being killed was used. The participants were limited to ﬁve submissions
per day. However, few teams made more than one submission per day.
In this challenge, we used the average score (see Equation 7) overM = 60 problems on the feedback
leaderboard. A separate set of M = 60 problems were used for the ﬁnal leaderboard. The two sets
of problems were split randomly. The feedback leaderboard was run with N = 10 repeated trials.
The ﬁnal problems were run with N = 30 repeated trials. To ensure the ﬁnal score was not due to
chance, we re-ran the top-20 with N = 100 for the ﬁnal leaderboard ranking.
The submissions were executed in a docker hosted on the Valohai platform. 6 Valohai provided
a robust backend where resources could automatically scale alongside the master scoring queue
load. More than 300 worker machines were executing submissions during peak hours, and at quieter
moments, not a single one. To prevent data exﬁltration about the feedback leaderboard problems, the
docker images had no network/internet access and the participants were not allowed to see the logs.
However, the Valohai platform provided intuitive access for the organizers to inspect the scoring
tasks. The organizers could still easily comment on issues through manual channels as they had
access to the logs.
Post-challenge the evaluation environment has been adapted to CodaLab for use as a “ever-lasting
benchmark” in hyper-parameter optimization courses.7
4 Learnings and Key Results
By the end of the challenge, there were 65 teams after ﬁltering accounts which could not be veriﬁed
through GitHub or LinkedIn. When testing on the ﬁnal leaderboard problems, which were not previ-
ously available to competitors, most teams saw their gains persist. This implies that the submissions
did not simply overﬁt to the local practice problems or the feedback leaderboard problems: There
were actual insights which worked on previously unseen ML problems. Out of the 65 total teams
whose results appeared in the ﬁnal leaderboard in Table 2, 61 beat the baseline random search and
23 beat TuRBO which was the strongest baseline provided in the starter kit. The ﬁnal competition
results are shown in Table 2 and in Figure 1.
6https://valohai.com/
7https://competitions.codalab.org/competitions/28609
5
4.1 Error Analysis
Just getting a sensible scale for scoring is not enough to gain scientiﬁc rigor from this challenge; we
need to do an error analysis. Also, for fairness, we wanted to be conﬁdent that we did not give prizes
to a team due to chance. Our error analysis gave us conﬁdence we did enough repeated trials such
that the ranking of the ﬁnal leaderboard was not due to a “lucky” random seed.
We ran the top-20 participants for N = 100 repeated trials to provide statistical conﬁdence in
the ﬁnal results followed by a bootstrap-based analysis to get a “conﬁdence set”. Based on this
analysis, we are 90% certain in the ﬁnal top-5 as is shown in Table 3 and our resulting conclusions
and learnings. Note that this bootstrap procedure is entirely a post-hoc error analysis and did not
determine the ﬁnal ranking on the leaderboard.
More precisely, we wanted to get error-bars on what the scores would be with an inﬁnite number of
repeated trials on these same problems. Note that the ﬁnal normalized score is a grand-mean across
all problems, which means we cannot use a simple t-test. Furthermore, we also wanted to translate
that to a “conﬁdence set” on the rankings. So, we opted to use a bootstrap-based analysis of the
scores from each study.
To get the bootstrap re-sampled score, we re-sampled with replacement N scores for each problem
separately and then took the average of those to get a bootstrap score. We repeated this process
B = 104 times (more than enough for scalar estimation) to get the bootstrap distribution on scores.
More formally,
S(i)
pn ∼Cat({Spn}N
n=1) , i ∈{1, . . . , B}, (1)
where Cat is a categorical distribution that is uniform over the elements provided as its argument. In
other words, the score on problem p in the nth study of the ith bootstrap replication is re-sampled
from the actual scores on problem p. We then ranked the teams within each bootstrap replication.
We got separate parallel rankings for each bootstrap replication. This gives a bootstrap distribution
of ranks.
4.2 Performance of the Baselines
In this section we report the score for the baselines provided in the starter kit with the addition of
Ax, GPyOpt, and pycma. The score for the different baselines with the default options can be found
in Table 4.
We see that TuRBO8 performs the best with a score of 88.921 followed by pySOT which uses the
stochastic RBF (SRBF) method [56]. Both TuRBO and pySOT use trust-region inspired meth-
ods, showing that a more local approach is advantageous for ML hyperparameter tuning. Scikit-
Optimize, Ax, and GpyOpt, all use Bayesian optimization with a GP model. Scikit-Optimize uses a
hedging strategy that uses multiple acquisition functions. Ax uses batch noisy EI (qNEI) [42] while
GPyOpt uses EI with local penalization [22], which is the only option in GPyOpt that supports batch
evaluations. The hyperopt package uses a tree-structured Parzen estimator (TPE) with EI, so these
results indicate that using a GP model leads to better performance than using a TPE. Oopentuner
and pycma use EAs and are clearly not competitive with the BO packages.
Note that this comparison does not necessarily show that one package is better than another; it
rather compares the performance of their default methods. Table 4 shows that the trust-region in-
spired method TuRBO and SRBF (pySOT) perform best out-of-the box, followed by packages with
traditional BO methods as defaults. Both groups of methods perform better than the three packages
that rely on EAs.
4.3 Bayesian Optimization was Consistently Effective
The submissions immediately bring one signiﬁcant realization to the forefront: surrogate-assisted
optimization is very effective. We discussed the performance of the baselines relative to random
search in Section 4.2, but the performance of the participants was even more compelling (see Ta-
ble 2). All of the top-20 participants used some form of surrogate-assisted optimization. This is
strongly indicative of the value of using a “surrogate model” and that intelligent modeling/decision
8https://github.com/uber-research/TuRBO
6
Table 2: The ﬁnal rankings on the ﬁnal leaderboard for the top-20. We show the ﬁnal rank, team
name, and (mean) score. For completeness, we also show the score using the median instead of
the mean to aggregate scores across runs. We provide an analysis comparing this algorithm to the
equivalent number of random search iterations (“RS Iters.”) needed to obtain the same (mean) score.
This is in comparison to the actual number of function evaluations in the challenge: 16 ×8 = 128;
the ratio yields the “RS Efﬁciency” factor. The AutoML.org submission would have gotten 92.551
(3rd place) after correcting a minor bug in their submission that prevented their code from executing;
so, we present them with rank “*” in this table.
Rank Team Score Median RS Iters. RS Efﬁciency
1 Huawei Noah’s Ark Lab 93.519 99.166 15,512 121.188
2 NVIDIA RAPIDS.AI 92.928 98.616 12,089 94.445
* AutoML.org 92.551 98.693 10,353 80.883
3 JetBrains Research 92.509 99.131 10,179 79.523
4 Duxiaoman DI 92.212 99.027 9,032 70.562
5 Optuna Developers 91.806 99.156 7,698 60.141
6 Ambitious Audemer 91.107 96.668 5,899 46.086
7 jumpshot 91.089 97.056 5,861 45.789
8 KAIST OSI 90.872 98.659 5,409 42.258
9 Able Anteater 90.302 95.954 4,405 34.414
10 Oxford BXL 90.143 98.792 4,165 32.539
11 Innovatrics 90.081 97.062 4,076 31.844
12 IBM AI RBFOpt 90.050 96.117 4,032 31.500
13 Jim Liu 89.996 97.279 3,957 30.914
14 Jzkay 89.969 99.037 3,920 30.625
15 Better call Bayes 89.846 97.395 3,757 29.352
16 dannynguyen 89.706 98.800 3,581 27.977
17 AlexLekov 89.403 99.099 3,232 25.250
18 ABO 89.354 97.893 3,180 24.844
19 a2i2team 89.237 98.781 3,058 23.891
20 Tiny, Shiny & Don 89.229 96.513 3,050 23.828
Random Search (RS) 75.815 88.746 128 1.000
Table 3: Variation in ﬁnal rankings (of all teams) across bootstrap replications. Each ranking’s
frequency in the bootstrap replications is shown in the bottom row. Teams outside these top-5 only
appeared in the top-5 in< 0.1% of bootstrap replications. This analysis gives us near certainty these
top-5 (and the prize amounts) are not due to chance. Duxiaoman DI was the only source of variation.
Their solution appears to have a higher variance than the others in the top-5.
Most likely ranking 2nd most likely ranking 3rd most likely ranking
Huawei Noah’s Ark Lab Huawei Noah’s Ark Lab Huawei Noah’s Ark Lab
NVIDIA RAPIDS.AI NVIDIA RAPIDS.AI NVIDIA RAPIDS.AI
JetBrains Research JetBrains Research Duxiaoman DI
Duxiaoman DI Optuna Developers JetBrains Research
Optuna Developers Duxiaoman DI Optuna Developers
90% 5.2% 4.5%
making can signiﬁcantly improve the optimization performance. Most top teams used a GP model
and at least one of the commonly used acquisition functions, but as we will describe in the next
section discovered the need for ensembling.
Note also that the baseline random search samples uniformly in the warped space from the search
conﬁguration. Thus, this baseline already outperforms a more naive random search that does not use
the warping information, e.g., log scale vs linear scale. Nonetheless, the participants gained orders
of magnitude greater search efﬁciency than random search as seen in Table 2.
7
(1+1)
Figure 1: Top methods on the ﬁnal leaderboard and examples submissions vs random search (RS):
On the left, we show what RS would have done given more function evaluations than allowed in the
challenge (128). This performance curve is based on an unbiased estimate from pooling the data of
N = 256 RS runs, which gives 256 ×128 = 32 ,768 function evaluations for each problem. As a
reference we also show the function evaluation limit in the challenge (128) and the performance of
RS at 128 function evaluations. On the right, we zoom into the relevant part of the plot and show
the performance of the top-5 submissions and the example submissions from the starter kit. The
top submissions clearly pull ahead of all the starter kit examples. The NVIDIA submission is a
simple ensemble of the TuRBO and Scikit-Optimize examples. We note how much improvement is
obtained over each of those solutions individually. Based on the intersection of the curves with the
RS curve we can see the “RS Iters.” from Table 4. Note that the x-axis is in logarithmic scale. This
demonstrates the orders of magnitude more function evaluations that would be required to obtain the
same performance as the top submissions using random search. Here, we show the performance on
the ﬁnal (not feedback) leaderboard. Thus the submissions were only evaluated one time on this test
suite; this gives us conﬁdence the performance we see here is not simply overﬁtting to the feedback
leaderboard.
Table 4: Performance of the example submissions provided to the participants. Like Table 2, we
also show the random search equivalents.
Example Submission Score Median RS Iters. RS Efﬁciency
TuRBO 88.921 98.927 2,756 21.531
pySOT 88.419 97.324 2,346 18.328
Scikit-Optimize 88.085 96.054 2,114 16.516
Ax 86.977 97.042 1,516 11.844
GpyOpt 85.384 94.443 978 7.641
hyperopt 82.389 93.506 477 3.727
Nevergrad (1+1) 80.012 92.681 288 2.250
pycma 78.658 95.285 220 1.719
OpenTuner 76.854 90.073 156 1.219
Random Search (RS) 75.815 88.746 128 1.000
4.4 Ensemble Methods
Many published papers on BO propose using only one surrogate model and one acquisition function,
despite some prior research having discussed the beneﬁts of ensembling BO methods [34]. Still,
teams discovered that the large set of somewhat disparate problems were best treated through a
mixture of methods. In fact, all of the methods in the top-10 had some sort of ensembling strategy.
8
Decision
Tree
Multi-Layer
Perceptron
Random
Forest
Adaptive
Boosting
Lasso
model for tuning
60
80
100
120score
Scikit-Optimize TuRBO NVIDIA RAPIDS.AI
Figure 2: Comparison of the NVIDIA RAPIDS.AI solution and its components (Scikit-Optimize and
TuRBO): Distribution of scores split across the different ML models for Scikit-Optimize, TuRBO,
and NVIDIA RAPIDS.AI. The thick horizontal line shows the median performance and the vertical
black line shows the interquartile range, the difference between 25th and 75th percentiles. While
TuRBO often has a similar median to the NVIDIA RAPIDS.AI solution, the 25th percentile is
much smaller on Lasso as well as on the decision tree and random forest problems, which affects
the mean performance. Ensembling with another high-performance optimizer like Scikit-Optimize
helps avoid this failure mode.
For the purposes of this article we use the term ensemble in a broad sense. Ensembles could be built
from multiple surrogates, acquisition functions, or potentially entire optimization algorithms, each
of which could independently be used to fully power the optimization. The level at which these
ensembling decisions were made varied across each of the teams. The ﬁrst-placed team Huawei
Noah’s Ark Lab [11] used an elaborate compilation of acquisition functions and incorporated them
into a multi-objective optimization strategy to select the next suggestions from the Pareto frontier.
Other popular approaches were to alternate the kernel in the GP model or to use multiple surrogate
models such as a GP and an XGBoost model.
Evolutionary methods also found their way into some ensembles — 2 of the the top-10 submissions
incorporated differential evolution into their optimization process. Of particular note, the AutoML
team [2] allocated the ﬁnal 5 of their 16 batches to differential evolution in order to improve con-
vergence close to the best point found in the ﬁrst 11 batches. Similarly, Better call Bayes [6] used
pySOT and switched to DE for the ﬁnal batches. This is a great example of how BO methods, while
powerful, can be supplemented by other tools to help balance out their weaknesses.
Second-placed NVIDIA RAPIDS.AI [45] and fourth-placed Duxiaoman DI [68] lived on the other
extreme of simple ensemble methods; they employed straightforward ensembles of TuRBO +
Scikit-Optimize and TuRBO + pySOT, respectively. NVIDIA’s ensemble, selecting 50% of sugges-
tions from each method, got a score of 92.928, beating both TuRBO (88.921) and Scikit-Optimize
(88.085) when used individually by a large margin. Figure 2 shows an analysis comparing the
NVIDIA RAPIDS.AI ensemble with its components. This analysis hints that ensembling may be
useful in avoiding failed models where an individual BO algorithm makes little progress. The suc-
cess of ensembles further justiﬁes the use of open-loop optimizers. The implementation of the
NVIDIA RAPIDS.AI solution was trivial due to the open-loop nature of the optimizers they were
ensembling.
The results of this challenge show that a strong ensemble can be created by combining open-source
tools without prior knowledge of the underlying components. However, further analysis and ablation
studies would be useful for fully understanding the mechanism of why ensembling works so well
for BO.
9
4.5 Building with (and from) Open Source Tools
On the topic of Scikit-Optimize, TuRBO, and open source packages, all of the top-20 submissions
had some open source elements present (including NumPy/SciPy). Realistically, nobody worked
on a solution entirely independent of existing code: NVIDIA RAPIDS.AI stitched together a solu-
tion built entirely from unaltered open-source projects; Huawei built on tools like GPy (which are
common in the BO and GP community) and built their own strategy using them; AutoML started
from open-source tools that their research group previously built. This is a testament to the maturity
of the open-source computational Python community, and in particular, the ML/GP niche of that
community.
Six out of the top-10 teams incorporated TuRBO into their solution, showing that trust region-
based optimization works well even for the lower dimensional problems represented by these ML
hyperparameter tuning tasks. In particular, JetBrains Research [58] combined TuRBO withk-means
to learn a partitioning of the search space similar to Wang et al. [65]. This prevalence of TuRBO
may indicate that the function landscape is often non-smooth and that it can be beneﬁcial to ﬁt a
local model rather than a global model. Or, it may indicate that TuRBO was the highest performing
baseline provided to participants, and it was a logical starting place.
4.6 Discrete and Categorical Parameters
While surrogate-assisted optimization is very powerful most literature on the topic deals with only
continuous parameters. The Bayesmark tool allowed users to ignore the presence of integer and
categorical parameters and computationally treat all parameters as continuous by encoding discrete
and categorical parameters in a continuous space. Furthermore, tree based methods such as TPEs are
generally considered to more naturally manage categorical parameters, but none of the participants
used it in their solutions.
Still, a few participants chose to more actively recognize and manage integer and categorical param-
eters. The Optuna Developers [59] built on top of TuRBO, but changed the size of the dimensions
in the trust region to make sure that at least one value for each discrete parameters was always vi-
able (that the trust region never moved/shrunk so much that none of the points in the trust region
represent actionable parameters). KAIST OSI [38] took that approach and added in a multi-armed
bandit strategy for recovering categorical parameters from their continuous embeddings. The other
participants may have identiﬁed other avenues for improving performance that they felt were more
beneﬁcial investments of their energy.
4.7 Meta-Learning and Warm Starting
The competition was divided into a feedback session (which the participants could monitor thor-
ough a practice leaderboard) and a ﬁnal testing session (the results of which produced the ﬁnal
leaderboard, as seen in Table 2). The goal of the feedback period was to allow participants to
measure their performance on problems which they could not observe and improve through that
feedback. Many of the successful participants used this as an opportunity to set tunable elements of
their submissions to high performing values; this, in effect, was meta black box optimization.
Some participants used this as an opportunity for meta-learning. While this was not the goal of the
black-box optimization setting, the participants realized that this meta-learning can further improve
the performance by transferring information from hyperparameter conﬁgurations applied to similar
ML problems. To preserve the black-box nature of the challenge, the ﬁnal testing was conducted
with all anonymized parameter names (e.g., P1, P2). This negated the beneﬁt of most meta-learning
strategies.
But we were so excited by the effort put in to meta-learning by these teams that we reran all sub-
missions with full visibility into parameter names. This allowed teams to employ strategies such as
making initial guesses using the found optima from problems with the same variable names under
the premise that the objective functions are likely similar. Such warm starting of the optimization
process led to major improvements for AutoML.org, DeepWisdom, dangnguyen, and Tiny, Shiny &
Don; participants who ignored this data saw no signiﬁcant change in performance from this extra
information. These results were compiled into an alternate “warm start friendly” leaderboard in
Table 5 where AutoML [2] emerged victorious. More details can be found in Appendix A.
10
5 Discussion
This competition is only one of many addressing the automation of machine learning model develop-
ment [25]. We hope that future organizers will take the progress made here and continue to develop
new competitions which address aspects of automated machine learning which were ignored in this
competition.
One point of focus in this competition was the black-box nature of each optimization problem. In
other competitions, more knowledge about the machine learning circumstances were made avail-
able to the participants, but here we wanted to see how well optimization could be conducted on
such problems without any knowledge of the problems. In future competitions, it might be inter-
esting to ﬁnd a middle ground – perhaps one where the type of model were known (e.g., XGBoost,
which would give a beneﬁt similar to what was seen on the warm start leaderboard) or the modeling
circumstances were known (e.g., maximizing the F1 score for a classiﬁcation problem on imbal-
anced data). Even without access to the training data, there may still be signiﬁcant opportunities for
improved performance.
Multi-ﬁdelity (or multi-information source) computations were not available in this competition, but
they may be common in practical circumstances [53]. Research has observed potential beneﬁts from
studying cheaply available (but lower ﬁdelity) information such as through evaluating only a fraction
of the training data or a small number of epochs. Of particular interest is early stopping setups in
ML models [43]. Often algorithms can guess a hyperparameter setting will perform poorly based on
the start of the learning curve without completing the training algorithm. Such a mechanism could
be made available in a black-box setting, which would give competitors the opportunity to more
intelligently use their computational budget.
This competition required batches of 8 suggestions be created, to emphasize the need for parallelism
which is required for practical circumstances. Some of those circumstances would prefer asyn-
chronous parallelism, where one suggestion is created given the other outstanding 7 suggestions cur-
rently being evaluated. Additionally, while we only focused on unconstrained single-objective opti-
mization performance, many relevant problems have additional black-box constraints that need to be
satisﬁed or are more naturally phrased as multi-objective optimization problems [12, 13, 21, 32, 40].
6 Conclusions
The novelty and importance of the black-box optimization challenge gathered large interest with 65
teams and hundreds of participants; it is also hosted an ongoing benchmark on CodaLab. It was
the ﬁrst optimization challenge evaluating derivative-free optimizers on ML-related problems. As
such, it demonstrated decisively the beneﬁts of Bayesian optimization over random search. The
top submissions showed over 100 ×sample efﬁciency gains compared to random search. First,
all of the top teams used some form of BO ensemble; sometimes with very simple and easy to
productionize strategies such as alternating the surrogate, acquisition function, or potentially entire
optimization algorithms.. Second, the warm start leaderboard demonstrated how warm starting from
even loosely related problems often yields large performance gains. Finally, this challenge offers
many opportunities for extensions to test and push the boundaries on other aspects of black-box
optimization.
Acknowledgments and Disclosure of Funding
We would like to thank the NeurIPS 2020 competition track organizers, Hugo Jair Escalante and
Katja Hofmann, for their guidance during the challenge. We would also like to thank Newton Le
and Serim Park for their feedback on the initial proposal.
11
References
[1] J. Ansel, S. Kamil, K. Veeramachaneni, J. Ragan-Kelley, J. Bosboom, U.-M. O’Reilly, and
S. Amarasinghe. OpenTuner: An extensible framework for program autotuning. In Proceed-
ings of the 23rd international conference on Parallel architectures and compilation , pages
303–316, 2014.
[2] N. Awad, G. Shala, D. Deng, N. Mallik, M. Feurer, K. Eggensperger, A. Biedenkapp, D. Ver-
metten, H. Wang, C. Doerr, M. Lindauer, and F. Hutter. Squirrel: A switching hyperparameter
optimizer, 2020.
[3] E. Bakshy, L. Dworkin, B. Karrer, K. Kashin, B. Letham, A. Murthy, and S. Singh. AE:
A domain-agnostic platform for adaptive experimentation. In Workshop on Systems for ML ,
2018.
[4] M. Balandat, B. Karrer, D. R. Jiang, S. Daulton, B. Letham, A. G. Wilson, and E. Bakshy.
Botorch: A framework for efﬁcient Monte-Carlo Bayesian optimization. InAdvances in Neural
Information Processing Systems 33, 2020.
[5] J. Bergstra, B. Komer, C. Eliasmith, D. Yamins, and D. D. Cox. Hyperopt: A Python library
for model selection and hyperparameter optimization. Computational Science & Discovery, 8
(1):014008, 2015.
[6] S. Biswas, A. D. Cobb, A. Sistrunk, N. Ramakrishnan, and B. Jalaian. Better call surrogates:
A hybrid evolutionary algorithm for hyperparameter optimization, 2020.
[7] X. Bouthillier and G. Varoquaux. Survey of machine-learning experimental methods at
NeurIPS2019 and ICLR2020. PhD thesis, Inria Saclay Ile de France, 2020.
[8] R. Calandra, A. Seyfarth, J. Peters, and M. P. Deisenroth. Bayesian optimization for learning
gaits under uncertainty. Annals of Mathematics and Artiﬁcial Intelligence, 76(1):5–23, 2016.
[9] A. Candelieri, R. Perego, and F. Archetti. Bayesian optimization of pump operations in water
distribution systems. Journal of Global Optimization, 71(1):213–235, 2018.
[10] A. Costa and G. Nannicini. RBFOpt: An open-source library for black-box optimization with
costly function evaluations. Mathematical Programming Computation, 10(4):597–629, 2018.
[11] A. I. Cowen-Rivers, W. Lyu, Z. Wang, R. Tutunov, H. Jianye, J. Wang, and H. B. Ammar.
HEBO: Heteroscedastic evolutionary Bayesian optimisation, 2020.
[12] S. Daulton, M. Balandat, and E. Bakshy. Differentiable expected hypervolume improvement
for parallel multi-objective Bayesian optimization. In Advances in Neural Information Pro-
cessing Systems 33, 2020.
[13] D. Eriksson and M. Poloczek. Scalable constrained Bayesian optimization. Proceedings of The
24th International Conference on Artiﬁcial Intelligence and Statistics, pages 730–738, 2021.
[14] D. Eriksson, D. Bindel, and C. A. Shoemaker. pySOT and POAP: An event-driven asyn-
chronous framework for surrogate optimization, 2019. arXiv preprint arXiv:1908.00420.
[15] D. Eriksson, M. Pearce, J. R. Gardner, R. Turner, and M. Poloczek. Scalable global optimiza-
tion via local Bayesian optimization. In Advances in Neural Information Processing Systems
32, pages 5497–5508, 2019.
[16] G. E. Fasshauer and M. J. McCourt. Kernel-based approximation methods using MATLAB ,
volume 19. World Scientiﬁc Publishing Company, 2015.
[17] A. Forrester, A. Sobester, and A. Keane. Engineering Design via Surrogate Modelling: A
Practical Guide. John Wiley & Sons, 2008.
[18] P. Frazier, W. Powell, and S. Dayanik. The knowledge-gradient policy for correlated normal
beliefs. INFORMS Journal on Computing, 21(4):599–613, 2009.
[19] P. I. Frazier. A tutorial on Bayesian optimization. arXiv preprint arXiv:1807.02811, 2018.
12
[20] P. I. Frazier and J. Wang. Bayesian optimization for materials design. In Information Science
for Materials Discovery and Design, pages 45–75. Springer, 2016.
[21] J. R. Gardner, M. J. Kusner, Z. E. Xu, K. Q. Weinberger, and J. P. Cunningham. Bayesian
optimization with inequality constraints. In Proceedings of the 31th International Conference
on Machine Learning , volume 32 of JMLR Workshop and Conference Proceedings , pages
937–945. JMLR.org, 2014.
[22] J. Gonz ´alez, Z. Dai, P. Hennig, and N. D. Lawrence. Batch Bayesian optimization via local
penalization. In Proceedings of the 19th International Conference on Artiﬁcial Intelligence
and Statistics, volume 51 of JMLR Workshop and Conference Proceedings , pages 648–657.
JMLR.org, 2016.
[23] R. B. Gramacy, G. A. Gray, S. Le Digabel, H. K. Lee, P. Ranjan, G. Wells, and S. M. Wild.
Modeling an augmented Lagrangian for blackbox constrained optimization.Technometrics, 58
(1):1–11, 2016.
[24] I. Guyon, G. Cawley, G. Dror, and A. Saffari. Hands-On Pattern Recognition Challenges in
Machine Learning, volume 1. Microtome Publishing, 2011.
[25] I. Guyon, L. Sun-Hosoya, M. Boull ´e, H. J. Escalante, S. Escalera, Z. Liu, D. Jajetic, B. Ray,
M. Saeed, M. Sebag, A. Statnikov, W. Tu, and E. Viegas. Analysis of the AutoML challenge
series 2015–2018. In AutoML, Springer Series on Challenges in Machine Learning, 2019.
[26] S. Haghanifar, M. McCourt, B. Cheng, J. Wuenschell, P. Ohodnicki, and P. W. Leu. Creating
glasswing butterﬂy-inspired durable antifogging superomniphobic supertransmissive, super-
clear nanostructured glass through Bayesian learning and optimization. Materials Horizons, 6
(8):1632–1642, 2019.
[27] N. Hansen, S. D. M ¨uller, and P. Koumoutsakos. Reducing the time complexity of the de-
randomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary
Computation, 11(1):1–18, 2003.
[28] N. Hansen, A. Auger, O. Mersmann, T. Tusar, and D. Brockhoff. COCO: A platform for
comparing continuous optimizers in a black-box setting. arXiv preprint arXiv:1603.08785 ,
2016.
[29] N. Hansen, Y . Akimoto, and P. Baudis. CMA-ES/pycma on Github. Zenodo, doi, 10, 2019.
[30] T. Head, G. L. MechCoder, I. Shcherbatyi, et al. scikit-optimize/scikit-optimize: v0. 5.2, 2018.
[31] P. Hennig and C. J. Schuler. Entropy search for information-efﬁcient global optimization.
Journal of Machine Learning Research, 13(6), 2012.
[32] J. M. Hern ´andez-Lobato, M. A. Gelbart, M. W. Hoffman, R. P. Adams, and Z. Ghahramani.
Predictive entropy search for Bayesian optimization with unknown constraints. InProceedings
of the 32nd International Conference on Machine Learning, volume 37 ofJMLR Workshop and
Conference Proceedings, pages 1699–1707. JMLR.org, 2015.
[33] J. M. Hern ´andez-Lobato, J. Requeima, E. O. Pyzer-Knapp, and A. Aspuru-Guzik. Parallel and
distributed Thompson sampling for large-scale accelerated exploration of chemical space. In
Proceedings of the 34th International Conference on Machine Learning , volume 70 of Pro-
ceedings of Machine Learning Research, pages 1470–1479. PMLR, 2017.
[34] M. Hoffman, E. Brochu, and N. de Freitas. Portfolio allocation for Bayesian optimization. In
UAI 2011, Proceedings of the Twenty-Seventh Conference on Uncertainty in Artiﬁcial Intelli-
gence, pages 327–336. AUAI Press, 2011.
[35] D. R. Jones, M. Schonlau, and W. J. Welch. Efﬁcient global optimization of expensive black-
box functions. Journal of Global Optimization, 13(4):455–492, 1998.
[36] S. Ju, T. Shiga, L. Feng, Z. Hou, K. Tsuda, and J. Shiomi. Designing nanostructures for phonon
transport via Bayesian optimization. Physical Review X, 7(2):021024, 2017.
13
[37] K. Kandasamy, K. R. Vysyaraju, W. Neiswanger, B. Paria, C. R. Collins, J. Schneider, B. Poc-
zos, and E. P. Xing. Tuning hyperparameters without grad students: Scalable and robust
Bayesian optimisation with Dragonﬂy. Journal of Machine Learning Research, 21(81):1–27,
2020.
[38] T. Kim, J. Ahn, N. Kim, and S. Yun. Adaptive local Bayesian optimization over multiple
discrete variables, 2020.
[39] A. Klein, S. Falkner, N. Mansur, and F. Hutter. RoBO: A ﬂexible and robust Bayesian opti-
mization framework in Python. In NIPS 2017 Bayesian Optimization Workshop, 2017.
[40] J. Knowles. ParEGO: A hybrid algorithm with on-line landscape approximation for expensive
multiobjective optimization problems. IEEE Transactions on Evolutionary Computation , 10
(1):50–66, 2006.
[41] N. Knudde, J. van der Herten, T. Dhaene, and I. Couckuyt. GPﬂowOpt: A Bayesian opti-
mization library using TensorFlow. arXiv preprint arXiv:1711.03845 , 2017. URL https:
//github.com/GPflow/GPflowOpt/.
[42] B. Letham, B. Karrer, G. Ottoni, E. Bakshy, et al. Constrained Bayesian optimization with
noisy experiments. Bayesian Analysis, 14(2):495–519, 2019.
[43] L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: A novel
bandit-based approach to hyperparameter optimization. The Journal of Machine Learning
Research, 18(1):6765–6816, 2017.
[44] D. C. Liu and J. Nocedal. On the limited memory BFGS method for large scale optimization.
Mathematical Programming, 45(1-3):503–528, 1989.
[45] J. Liu, B. Tunguz, and G. Titericz. GPU accelerated exhaustive search for optimal ensemble
of black-box optimization algorithms, 2020.
[46] Z. Liu, Z. Xu, S. Rajaa, M. Madadi, J. C. J. Junior, S. Escalera, A. Pavao, S. Treguer, W.-W.
Tu, and I. Guyon. Towards automated deep learning: Analysis of the AutoDL challenge series
2019. In NeurIPS 2019 Competition and Demonstration Track, pages 242–252. PMLR, 2020.
[47] I. Loshchilov and T. Glasmachers. Black box optimization competition BBComp, 2019. URL
https://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/index.html.
[48] D. Molina, A. LaTorre, and F. Herrera. An insight into bio-inspired and evolutionary algo-
rithms for global optimization: Review, analysis, and lessons learnt over a decade of competi-
tions. Cognitive Computation, 10(4):517–544, 2018.
[49] D. M. Negoescu, P. I. Frazier, and W. B. Powell. The knowledge-gradient algorithm for se-
quencing experiments in drug discovery. INFORMS Journal on Computing , 23(3):346–363,
2011.
[50] W. Neiswanger, K. Kandasamy, B. Poczos, J. Schneider, and E. Xing. ProBO: A framework for
using probabilistic programming in Bayesian optimization. arXiv preprint arXiv:1901.11515,
9, 2019.
[51] J. A. Nelder and R. Mead. A simplex method for function minimization. The Computer
Journal, 7(4):308–313, 1965.
[52] A. Paleyes, M. Pullin, M. Mahsereci, N. Lawrence, and J. Gonz ´alez. Emulation of physical
processes with Emukit. In Second Workshop on Machine Learning and the Physical Sciences,
NeurIPS, 2019.
[53] M. Poloczek, J. Wang, and P. I. Frazier. Multi-information source optimization. In Advances
in Neural Information Processing Systems 30, pages 4288–4298, 2017.
[54] J. Rapin and O. Teytaud. Nevergrad - A gradient-free optimization platform. version 0.2. 0,
https://GitHub.com/FacebookResearch/Nevergrad, 2018.
14
[55] C. E. Rasmussen. Gaussian processes in machine learning. In Summer School on Machine
Learning, pages 63–71. Springer, 2003.
[56] R. G. Regis and C. A. Shoemaker. A stochastic radial basis function method for the global
optimization of expensive functions. INFORMS Journal on Computing, 19(4):497–509, 2007.
[57] J. Rodriguez. Hyperparameter tuning platforms are becoming a new market in the deep
learning space. Hacker Noon, 2018. URL https://hackernoon.com/hyperparameter-
tuning-platforms-are-becoming-a-new-market-in-the-deep-learning-space-
7106f0ac1689.
[58] M. Sazanovich, A. Nikolskaya, Y . Belousov, and A. Shpilman. Solving black-box optimization
challenge via learning search space partition for local Bayesian optimization, 2020.
[59] M. Shibata, T. Yanase, H. Imamura, M. Nomura, T. Ohta, S. Sano, and H. V . Yamazaki. Team
Optuna developers’ method for black-box optimization challenge 2020, 2020. URL https:
//valohaichirpprod.blob.core.windows.net/papers/optuna.pdf.
[60] J. Snoek, H. Larochelle, and R. P. Adams. Practical Bayesian optimization of machine learning
algorithms. In Advances in Neural Information Processing Systems 25 , pages 2960–2968,
2012.
[61] The GPyOpt authors. GPyOpt: A Bayesian optimization framework in Python, 2016. URL
http://github.com/SheffieldML/GPyOpt/.
[62] R. Turner. The Bayes opt benchmark documentation, 2019. URL https://
bayesmark.readthedocs.io/.
[63] R. Turner and C. E. Rasmussen. Model based learning of sigma points in unscented Kalman
ﬁltering. Neurocomputing, 80:47–53, 2012.
[64] T. Ueno, T. D. Rhone, Z. Hou, T. Mizoguchi, and K. Tsuda. COMBO: An efﬁcient Bayesian
optimization library for materials science. Materials Discovery, 4:18–21, 2016.
[65] L. Wang, R. Fonseca, and Y . Tian. Learning search space partition for black-box optimization
using Monte Carlo tree search. arXiv preprint arXiv:2007.00708, 2020.
[66] H. Wendland. Scattered Data Approximation, volume 17. Cambridge University Press, 2004.
[67] J. Wu and P. I. Frazier. The parallel knowledge gradient method for batch Bayesian optimiza-
tion. In Advances in Neural Information Processing Systems 29, pages 3126–3134, 2016.
[68] J. Wu, M. Cao, L. Shan, and Q. Ying. Higher performance for AutoML: The
beneﬁt of various ensemble Bayesian optimization strategy, 2020. URL https://
valohaichirpprod.blob.core.windows.net/papers/duxiaoman.pdf.
[69] X. Yu and M. Gen. Introduction to Evolutionary Algorithms . Springer Science & Business
Media, 2010.
A Warm-starting
Figure 3 shows the score of the top submissions on the warm-start leaderboard compared to random
search. Table 5 shows the top-20 leaderboard on the warm-start leaderboard with the submission by
AutoML.org coming out on top.
15
(1+1)
Figure 3: Top methods on the warm start leaderboard and examples submissions vs random search
(RS). This plot follows the same analysis as Figure 1. The most notable message from this plot
is how the intensive use of warm starting by AutoML.org really allows them to “pull ahead of the
pack”.
Table 5: The ﬁnal rankings on the warm start leaderboard for the top-20 in the same format as
Table 2. We show the ﬁnal rank, team name, and score. Here, AutoML.org, DeepWisdom, dan-
gnguyen, and Tiny, Shiny & Don make big gains by warm starting, from the parameter names,
leveraging solutions from different problems with the same search space.
Rank Team Score RS Iters. RS Efﬁciency
1 AutoML.org 94.845 27,977 218.570
2 DeepWisdom 93.380 14,615 114.180
3 Huawei Noah’s Ark Lab 93.241 13,777 107.633
4 dangnguyen 93.082 12,891 100.711
5 NVIDIA RAPIDS.AI 92.537 10,294 80.422
6 JetBrains Research 92.509 10,179 79.523
7 Duxiaoman DI 92.242 9,140 71.406
8 Optuna Developers 92.142 8,785 68.633
9 Tiny, Shiny & Don 92.108 8,667 67.711
10 KAIST OSI 91.272 6,277 49.039
11 jumpshot 91.115 5,917 46.227
12 Ambitious Audemer 90.999 5,669 44.289
13 Innovatrics 90.757 5,186 40.516
14 Jzkay 90.525 4,769 37.258
15 Oxford BXL 90.403 4,566 35.672
16 IBM AI RBFOpt 90.370 4,513 35.258
17 Better call Bayes 90.104 4,108 32.094
18 Able Anteater 90.036 4,012 31.344
19 Jim Liu 89.972 3,924 30.656
20 IBM Research-China 89.834 3,741 29.227
Random Search (RS) 75.815 128 1.000
B How scoring works
The scoring system is about aggregating the function evaluations of the optimizers. We represent
Fptn as the function evaluation of objective functionp (TEST CASE) at batcht (ITER) under repeated
trial n (TRIAL). In the case of batch sizes greater than 1, Fptn is the minimum function evaluation
16
across the suggestions in batch t. The ﬁrst transformation is that we consider the cumulative mini-
mum over batches t as the performance of the optimizer on a particular trial:
Sptn = cumm-mintFptn . (2)
From a decision theoretical perspective it is more sensible to consider the mean (possibly warped)
score. Robust measures like the median score can hide a high percentage of runs that completely
fail. However, when we look at the mean score we ﬁrst take the clipped score with a baseline value:
S′
ptn = min(Sptn, clipp) . (3)
This is largely because there may be a non-zero probability of F = ∞(as in when the objective
function crashes), which means that mean random search performance is inﬁnite loss. We set clip p
to the median score after a single function evaluation of random search. The mean performance on
a single problem then becomes:
mean-perfpt = meann S′
ptn . (4)
Which then becomes a normalized performance (via linear rescaling between optimal and RS per-
formance) of:
norm-mean-perfpt = mean-perfpt −optp
clipp −optp
. (5)
Again, we can aggregate this into all objective function performance with:
mean-perft = meanp norm-mean-perfpt , (6)
which is a mean-of-means (or grand mean).
For the leaderboard, we use 16 batches (t = 16) of batch size 8; and we transform the scores on a 0
to 100 scale. In equations,
leaderboard-score = 100 ×(1 −mean-perf16) . (7)
This means that an algorithm whose ﬁnal solution is only as good as a single guess of random
search has score 0. Meanwhile, an algorithm whose ﬁnal solution always ﬁnds the best known
global optimum has score 100. This scoring system is invariant to independent linear rescaling of
any of the objective functions.
17

Nature | Vol 610 | 6 October 2022 | 47
Article
Discovering faster matrix multiplication 
algorithms with reinforcement learning
Alhussein Fawzi1,2 ✉, Matej Balog1,2, Aja Huang1,2, Thomas Hubert1,2, 
Bernardino Romera-Paredes1,2, Mohammadamin Barekatain1, Alexander Novikov1, 
Francisco J. R. Ruiz1, Julian Schrittwieser1, Grzegorz Swirszcz1, David Silver1, Demis Hassabis1 
& Pushmeet Kohli1
Improving the efficiency of algorithms for fundamental computations can have a 
widespread impact, as it can affect the overall speed of a large amount of computations. 
Matrix multiplication is one such primitive task, occurring in many systems—from 
neural networks to scientific computing routines. The automatic discovery of 
algorithms using machine learning offers the prospect of reaching beyond human 
intuition and outperforming the current best human-designed algorithms. However, 
automating the algorithm discovery procedure is intricate, as the space of possible 
algorithms is enormous. Here we report a deep reinforcement learning approach 
based on AlphaZero
1 for discovering efficient and provably correct algorithms for the 
multiplication of arbitrary matrices. Our agent, AlphaT ensor, is trained to play a 
single-player game where the objective is finding tensor decompositions within a 
finite factor space. AlphaT ensor discovered algorithms that outperform the state- 
of-the-art complexity for many matrix sizes. Particularly relevant is the case of 4 × 4 
matrices in a finite field, where AlphaT ensor’s algorithm improves on Strassen’s two- 
level algorithm for the first time, to our knowledge, since its discovery 50 years ago
2.  
We further showcase the flexibility of AlphaT ensor through different use-cases: 
algorithms with state-of-the-art complexity for structured matrix multiplication and 
improved practical efficiency by optimizing matrix multiplication for runtime on 
specific hardware. Our results highlight AlphaT ensor’s ability to accelerate the 
process of algorithmic discovery on a range of problems, and to optimize for different 
criteria.
We focus on the fundamental task of matrix multiplication, and use 
deep reinforcement learning (DRL) to search for provably correct and 
efficient matrix multiplication algorithms. This algorithm discovery 
process is particularly amenable to automation because a rich space of 
matrix multiplication algorithms can be formalized as low-rank decom-
positions of a specific three-dimensional (3D) tensor2, called the matrix 
multiplication tensor3–7. This space of algorithms contains the stand-
ard matrix multiplication algorithm and recursive algorithms such as 
Strassen’s2, as well as the (unknown) asymptotically optimal algorithm. 
Although an important body of work aims at characterizing the com-
plexity of the asymptotically optimal algorithm8–12, this does not yield 
practical algorithms5. We focus here on practical matrix multiplication 
algorithms, which correspond to explicit low-rank decompositions of 
the matrix multiplication tensor. In contrast to two-dimensional matri-
ces, for which efficient polynomial-time algorithms computing the rank 
have existed for over two centuries13, finding low-rank decompositions 
of 3D tensors (and beyond) is NP-hard14 and is also hard in practice. 
In fact, the search space is so large that even the optimal algorithm 
for multiplying two 3 × 3 matrices is still unknown. Nevertheless, in a 
longstanding research effort, matrix multiplication algorithms have 
been discovered by attacking this tensor decomposition problem using 
human search2,15,16, continuous optimization17–19 and combinatorial 
search20. These approaches often rely on human-designed heuristics, 
which are probably suboptimal. We instead use DRL to learn to recog-
nize and generalize over patterns in tensors, and use the learned agent 
to predict efficient decompositions.
We formulate the matrix multiplication algorithm discovery pro-
cedure (that is, the tensor decomposition problem) as a single-player 
game, called T ensorGame. At each step of T ensorGame, the player 
selects how to combine different entries of the matrices to multiply.  
A score is assigned based on the number of selected operations required 
to reach the correct multiplication result. This is a challenging game 
with an enormous action space (more than 1012 actions for most inter-
esting cases) that is much larger than that of traditional board games 
such as chess and Go (hundreds of actions). T o solve T ensorGame and 
find efficient matrix multiplication algorithms, we develop a DRL agent, 
AlphaT ensor. AlphaT ensor is built on AlphaZero1,21, where a neural net-
work is trained to guide a planning procedure searching for efficient 
matrix multiplication algorithms. Our framework uses a single agent 
to decompose matrix multiplication tensors of various sizes, yielding 
https://doi.org/10.1038/s41586-022-05172-4
Received: 2 October 2021
Accepted: 2 August 2022
Published online: 5 October 2022
Open access
 Check for updates
1DeepMind, London, UK. 2These authors contributed equally: Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert and Bernardino Romera-Paredes. ✉e-mail: afawzi@deepmind.com
48 | Nature | Vol 610 | 6 October 2022
Article
transfer of learned decomposition techniques across various tensors. 
T o address the challenging nature of the game, AlphaT ensor uses a 
specialized neural network architecture, exploits symmetries of the 
problem and makes use of synthetic training games.
AlphaT ensor scales to a substantially larger algorithm space than 
what is within reach for either human or combinatorial search. In fact, 
AlphaT ensor discovers from scratch many provably correct matrix 
multiplication algorithms that improve over existing algorithms in 
terms of number of scalar multiplications. We also adapt the algo -
rithm discovery procedure to finite fields, and improve over Strassen’s 
two-level algorithm for multiplying 4 × 4 matrices for the first time, to 
our knowledge, since its inception in 1969. AlphaT ensor also discovers a 
diverse set of algorithms—up to thousands for each size—showing that 
the space of matrix multiplication algorithms is richer than previously 
thought. We also exploit the diversity of discovered factorizations to 
improve state-of-the-art results for large matrix multiplication sizes. 
Through different use-cases, we highlight AlphaT ensor’s flexibility 
and wide applicability: AlphaT ensor discovers efficient algorithms 
for structured matrix multiplication improving over known results, 
and finds efficient matrix multiplication algorithms tailored to spe -
cific hardware, by optimizing for actual runtime. These algorithms 
multiply large matrices faster than human-designed algorithms on 
the same hardware.
Algorithms as tensor decomposition
As matrix multiplication (A, B) ↦ AB is bilinear (that is, linear in both 
arguments), it can be fully represented by a 3D tensor: see Fig. 1a for 
how to represent the 2 × 2 matrix multiplication operation as a 3D ten-
sor of size 4 × 4 × 4, and refs. 3,5,7 for more details. We write nT  for the 
tensor describing n × n matrix multiplication. The tensor Tn is fixed 
(that is, it is independent of the matrices to be multiplied), has entries 
in {0, 1}, and is of size n2 × n2 × n2. More generally, we use nm p,,T  to 
describe the rectangular matrix multiplication operation of size n × m 
with m × p (note that =n nnn,,T T ). By a decomposition of Tn into R 
rank-one terms, we mean
T ∑=⊗ ⊗, (1)n
r
R
rr r
=1
() () ()uv w
where ⊗ denotes the outer (tensor) product, and u(r), v(r) and w(r) are all 
vectors. If a tensor T  can be decomposed into R rank-one terms, we say 
the rank of T  is at most R, or RRank () ≤T . This is a natural extension 
from the matrix rank, where a matrix is decomposed into ∑⊗r
R rr
=1
() ()uv .
A decomposition of Tn into R rank-one terms provides an algorithm 
for multiplying arbitrary n × n matrices using R scalar multiplications 
(see Algorithm 1). We refer to Fig. 1b,c for an example algorithm mul-
tiplying 2 × 2 matrices with R = 7 (Strassen’s algorithm).
Crucially, Algorithm 1 can be used to multiply block matrices. By using 
this algorithm recursively, one can multiply matrices of arbitrary size, with 
the rank R controlling the asymptotic complexity of the algorithm. In par-
ticular, N × N matrices can be multiplied with asymptotic complexity 
O N() Rlog( )n ; see ref. 5 for more details.
DRL for algorithm discovery
We cast the problem of finding efficient matrix multiplication algo-
rithms as a reinforcement learning problem, modelling the environ-
ment as a single-player game, T ensorGame. The game state after step 
t is described by a tensor St, which is initially set to the target tensor 
we wish to decompose: S T= n0 . In each step t of the game, the player 
selects a triplet (u(t), v(t), w(t)), and the tensor St is updated by subtract-
ing the resulting rank-one tensor: ←− ⊗⊗tt
tt t
−1
() () ()S S uv w . The goal 
of the player is to reach the zero tensor 0=tS  by applying the smallest 
number of moves. When the player reaches the zero tensor, the 
sequence of selected factors satisfies uv wT =∑ ⊗⊗nt
R tt t
=1
() () ()  (where 
R denotes the number of moves), which guarantees the correctness of 
the resulting matrix multiplication algorithm. T o avoid playing unnec-
essarily long games, we limit the number of steps to a maximum value, 
Rlimit.
For every step taken, we provide a reward of −1 to encourage finding 
the shortest path to the zero tensor. If the game terminates with a 
non-zero tensor (after Rlimit steps), the agent receives an additional 
terminal reward equal to γ−( )R limitS , where γ() R limitS  is an upper bound 
on the rank of the terminal tensor. Although this reward optimizes for 
rank (and hence for the complexity of the resulting algorithm), other 
reward schemes can be used to optimize other properties, such as 
practical runtime (see ‘ Algorithm discovery results’). Besides, as our 
aim is to find exact matrix multiplication algorithms, we constrain 
{u(t), v(t), w(t)} to have entries in a user-specified discrete set of coeffi -
cients F (for example, F = {−2, −1, 0, 1, 2}). Such discretization is com -
mon practice to avoid issues with the finite precision of floating 
points15,18,20.
T o play T ensorGame, we propose AlphaT ensor (Fig. 2), an agent based 
on AlphaZero1, which achieved tabula rasa superhuman performance 
in the classical board games of Go, chess and shogi, and on its extension 
to handle large action spaces Sampled AlphaZero 21. Similarly to 
AlphaZero, AlphaT ensor uses a deep neural network to guide a Monte 
c1 c2
c3 c4
=
a1 a2
a3 a4
·
b1 b2
b3 b4
a
U =
1 0 1 0 1 0
00 00 11
0 1 00 00
11 0 1 0 –1
V =
10
00 0
01 1
10 1
W =
10 01 1
00 10 10
01 01 00
10 0a1 a2 a3 a4
b1
b2
b3
b4
c1
c3
c2
c4
bc
–1
0
1
0
1 0
01
0 0
0 –1
–1 0 1
01
0 0
1 0
–1 1 0
0
0
0
1
–1
m1 = ( a1 + a4 )( b1 + b4 )
m2 = ( a3 + a4 ) b1
m3 = a1 ( b2 – b4 )
m4 = a4 ( b3 – b1 )
m5 = ( a1 + a2 ) b4
m6 = ( a3 – a1 )( b1 + b2 )
m7 = ( a2 – a4 )( b3 + b4 )
c1 = m1 + m4 – m5 + m7
c2 = m3 + m5
c3 = m2 + m4
c4 = m1 – m2 + m3 + m6
Fig. 1 | Matrix multiplication tensor and algorithms. a, Tensor T2 representing 
the multiplication of two 2 × 2 matrices. Tensor entries equal to 1 are depicted 
in purple, and 0 entries are semi-transparent. The tensor specifies which entries 
from the input matrices to read, and where to write the result. For example,  
as c1 = a1b1 + a2b3, tensor entries located at ( a1, b1, c1) and (a2, b3, c1) are set to 1.  
b, Strassen's algorithm2 for multiplying 2 × 2 matrices using 7 multiplications.  
c, Strassen's algorithm in tensor factor representation. The stacked factors  
U, V and W (green, purple and yellow, respectively) provide a rank-7 
decomposition of 2T  (equation (1)). The correspondence between arithmetic 
operations ( b) and factors (c) is shown by using the aforementioned colours.
Nature | Vol 610 | 6 October 2022 | 49
Carlo tree search (MCTS) planning procedure. The network takes as 
input a state (that is, a tensor tS  to decompose), and outputs a policy 
and a value. The policy provides a distribution over potential actions. 
As the set of potential actions (u(t), v(t), w(t)) in each step is enormous, 
we rely on sampling actions rather than enumerating them21,22. The 
value provides an estimate of the distribution z of returns (cumulative 
reward) starting from the current state tS . With the above reward 
scheme, the distribution z models the agent’s belief about the rank of 
the tensor St. T o play a game, AlphaT ensor starts from the target tensor 
( nT ) and uses the MCTS planner at each step to choose the next action. 
Finished games are used as feedback to the network to improve the 
network parameters.
Overcoming the challenges posed by T ensorGame—namely, an enor-
mous action space, and game states described by large 3D tensors 
representing an abstract mathematical operation—requires multiple 
advances. All these components, described briefly below,  substantially 
improve the overall performance over a plain AlphaZero agent (see 
Methods and Supplementary Information for details).
Neural network architecture
We propose a transformer-based 23 architecture that incorporates 
inductive biases for tensor inputs. We first project the S × S × S input 
tensor into three S × S grids of feature vectors by using linear layers 
applied to the three cyclic transpositions of the tensor. The main part of 
the model comprises a sequence of attention operations, each applied 
to a set of features belonging to a pair of grids (Extended Data Figs. 3 
and 4). This generalizes axial attention24 to multiple grids, and is both 
more efficient and yields better results than naive self-attention. The 
proposed architecture, which disregards the order of rows and columns 
in the grids, is inspired by the invariance of the tensor rank to slice 
reordering. The final feature representation of the three matrices is 
passed both to the policy head (an autoregressive model) and the value 
head (a multilayer perceptron).
Synthetic demonstrations
Although tensor decomposition is NP-hard, the inverse task of con-
structing the tensor from its rank-one factors is elementary. Hence, 
we generate a large dataset of tensor-factorization pairs (synthetic 
demonstrations) by first sampling factors uv w{(, ,) }rr r
r
R() () ()
=1 at random, 
and then constructing the tensor =∑ ⊗⊗r
R rr r
=1
() () ()D uv w . We train the 
network on a mixture of supervised loss (that is, to imitate synthetic 
demonstrations) and standard reinforcement learning loss (that is, 
learning to decompose a target tensor nT ) (Fig. 2). This mixed training 
strategy—training on the target tensor and random tensors— substan-
tially outperforms each training strategy separately. This is despite 
randomly generated tensors having different properties from the tar-
get tensors.
Change of basis
 nT  (Fig. 1a) is the tensor representing the matrix multiplication bilinear 
operation in the canonical basis. The same bilinear operation can be 
expressed in other bases, resulting in other tensors. These different 
Algorithm 1
A meta-algorithm parameterized by =uv w{, ,}rr r() () ()
r
R
1 for computing 
the matrix product C = AB. It is noted that R controls the number of 
multiplications between input matrix entries.
Parameters: =uv w{, ,}rr r() () ()
r
R
1: length-n2 vectors such that 
Tn
rr r() () ()
r
R
1=∑ ⊗⊗= uv w
Input: A, B: matrices of size n × n
Output: C = AB
(1) for r = 1, …, R do
(2)     ←+ ++ +/midhorizellipsis/midhorizellipsismu au av bv b() ()r n n11
r
n
rr
n
r
1
() ()
1
() ()2 222
(3) for i = 1, …, n2 do
(4)     /midhorizellipsiscw mw miR 1ii
R(1)( )←+ +
return C
Change of basis
Pre-generated 
synthetic 
demonstrations
Played games
buffer
Played
game
Sample
random state
Neural network
Policy head
Value head
Acting
...
LearningUpdated 
model
Network inputTraining labels
(u, v, w)
(u(1), v(1), w(1))
 (u(2), v(2), w(2))( u(3), v(3), w(3))
Fig. 2 | Overview of AlphaTensor. The neural network (bottom box) takes  
as input a tensor St, and outputs samples ( u, v, w) from a distribution  
over potential next actions to play, and an estimate of the future returns  
(for example, of S−Rank( )t ). The network is trained on two data sources: 
previously played games and synthetic demonstrations. The updated network 
is sent to the actors (top box), where it is used by the MCTS planner to generate 
new games.
50 | Nature | Vol 610 | 6 October 2022
Article
tensors are equivalent: they have the same rank, and decompositions 
obtained in a custom basis can be mapped to the canonical basis, hence 
obtaining a practical algorithm of the form in Algorithm 1. We leverage 
this observation by sampling a random change of basis at the beginning 
of every game, applying it to Tn, and letting AlphaT ensor play the game 
in that basis (Fig. 2). This crucial step injects diversity into the games 
played by the agent.
Data augmentation
From every played game, we can extract additional tensor-factorization 
pairs for training the network. Specifically, as factorizations are 
order invariant (owing to summation), we build an additional 
tensor-factorization training pair by swapping a random action with 
the last action from each finished game.
Algorithm discovery results
Discovery of matrix multiplication algorithms
We train a single AlphaT ensor agent to find matrix multiplication algo-
rithms for matrix sizes n × m with m × p, where n, m, p ≤ 5. At the begin-
ning of each game, we sample uniformly a triplet ( n, m, p) and train 
AlphaT ensor to decompose the  tensor Tnm p,, . Although we consider 
tensors of fixed size (Tnm p,,  has size nm × mp × pn), the discovered algo-
rithms can be applied recursively to multiply matrices of arbitrary size. 
We use AlphaT ensor to find matrix multiplication algorithms over 
different arithmetics—namely, modular arithmetic (that is, multiplying 
matrices in the quotient ring 2Z ), and standard arithmetic (that is, mul-
tiplying matrices in R).
Figure  3 (left) shows the complexity (that is, rank) of the algo -
rithms discovered by AlphaT ensor. AlphaT ensor re-discovers the 
best algorithms known for multiplying matrices (for example, 
Strassen’s 2 and Laderman’s 15 algorithms). More importantly, AlphaT -
ensor improves over the best algorithms known for several matrix 
sizes. In particular, AlphaT ensor finds an algorithm for multiplying 
4 × 4 matrices using 47 multiplications in 2Z , thereby outperforming 
Strassen’s two-level algorithm 2, which involves 7 2 = 49 multiplica -
tions. By applying this algorithm recursively, one obtains a practical 
matrix multiplication algorithm in Z2 with complexity O N() 2.778 . 
Moreover, AlphaT ensor discovers efficient algorithms for multiply -
ing matrices in standard arithmetic; for example, AlphaT ensor finds 
a rank-76 decomposition of 4,5,5T , improving over the previous 
state-of-the-art complexity of 80 multiplications. See Extended 
Data Figs. 1 and 2 for examples.
AlphaT ensor generates a large database of matrix multiplication 
algorithms—up to thousands of algorithms for each size. We exploit 
this rich space of algorithms by combining them recursively, with the 
aim of decomposing larger matrix multiplication tensors. We refer 
to refs. 25,26 and Appendix H in Supplementary Information for more 
details. Using this approach, we improve over the state-of-the-art 
results for more than 70 matrix multiplication tensors (with 
n, m, p ≤ 12). See Fig. 3 (right) and Extended Data Table 1 for the results.
A crucial aspect of AlphaT ensor is its ability to learn to transfer knowl-
edge between targets (despite providing no prior knowledge on their 
relationship). By training one agent to decompose various tensors, 
AlphaT ensor shares learned strategies among these, thereby improv-
ing the overall performance (see Supplementary Information for 
analysis). Finally, it is noted that AlphaT ensor scales beyond current 
computational approaches for decomposing tensors. For example, to 
our knowledge, no previous approach was able to handle T4, which has 
an action space 1010 times larger than T3. Our agent goes beyond this 
limit, discovering decompositions matching or surpassing 
state-of-the-art for large tensors such as 5T .
AlphaTensor rank
Modular Standard
Size
(n, m, p)
Best method
known
Best rank
known
(2, 2, 2) (Strassen, 1969)2 7 7 7
(3, 3, 3) (Laderman, 1976)15 23 23 23
49 47 49
98 96 98
(2,2,3 ) 11 11 11
(2,2,4 ) 14 14 14
(2,2,5 ) 18 18 18
(2,3,3 ) 15 15 15
(2,3,4 ) 20 20 20
(2,3,5 ) 25 25 25
(2,4,4 ) 26 26 26
(2,4,5 ) 33 33 33
(2,5,5 ) 40 40 40
(3,3,4 ) (Smirnov, 2013)18 29 29 29
(3,3,5 ) 36 36 36
(3,4,4 ) 38 38 38
(3,4,5 ) 48 47 47
(3,5,5 ) 58 58 58
(4,4,5 ) 64 63 63
(4,5,5 ) 80 76 76 200 4006 00 8001 ,000
Best rank known
0
5
10
15
20
25
30Improvement in rank
(9, 9, 9)
(9, 9, 11)
(9, 10, 10)
(9, 11, 11)
(10, 10, 10)
(10, 11, 12)
(10, 12, 12)
(11, 11, 11)
(11, 12, 12)
(Strassen, 1969)2
(Hopcroft and Kerr, 1971)16
(Hopcroft and Kerr, 1971)16
(Hopcroft and Kerr, 1971)16
(Hopcroft and Kerr, 1971)16
(Hopcroft and Kerr, 1971)16
(Smirnov, 2013)18
(Smirnov, 2013)18
(Smirnov, 2013)18
(Sedoglavic and Smirnov, 2021)19
(3, 5, 5) + (2, 5, 5)
(2, 2, 2) ^ (2, 2, 2)
(2, 2, 2) + (2, 2, 1)
(2, 2, 2) + (2, 2, 2)
(2, 2, 2) + (2, 2, 3)
(4, 4, 2) + (4, 4, 3)
(2, 5, 5) ^ (2, 1, 1)
(4, 4, 4)
(5, 5, 5)
(Hopcroft and Kerr, 1971)16
Fig. 3 | Comparison between the complexity of previously known matrix 
multiplication algorithms and the ones discovered by AlphaTensor. Left: 
column (n, m, p) refers to the problem of multiplying n × m with m × p matrices. 
The complexity is measured by the number of scalar multiplications (or 
equivalently, the number of terms in the decomposition of the tensor). ‘Best 
rank known’ refers to the best known upper bound on the tensor rank (before 
this paper), whereas ‘AlphaTensor rank’ reports the rank upper bounds 
obtained with our method, in modular arithmetic (
Z2) and standard arithmetic. 
In all cases, AlphaTensor discovers algorithms that match or improve over 
known state of the art (improvements are shown in red). See Extended Data 
Figs. 1 and 2 for examples of algorithms found with AlphaTensor. Right: results 
(for arithmetic in 
R) of applying AlphaTensor-discovered algorithms on larger 
tensors. Each red dot represents a tensor size, with a subset of them labelled. 
See Extended Data Table 1 for the results in table form. State-of-the-art results 
are obtained from the list in ref. 
64.
Nature | Vol 610 | 6 October 2022 | 51
 
Analysing the symmetries of matrix multiplication algorithms
From a mathematical standpoint, the diverse algorithms discovered 
by AlphaT ensor show that the space is richer than previously known. 
For example, while the only known rank-49 factorization decomposing 
T TT=⊗42 2 before this paper conforms to the product structure (that 
is, it uses the factorization of T2 twice, which we refer to as Strassen- 
square2), AlphaT ensor finds more than 14,000 non-equivalent fac-
torizations (with standard arithmetic) that depart from this scheme, 
and have different properties (such as matrix ranks and sparsity—see 
Supplementary Information). By non-equivalent, we mean that it is 
not possible to obtain one from another by applying a symmetry trans-
formation (such as permuting the factors). Such properties of matrix 
multiplication tensors are of great interest, as these tensors represent 
fundamental objects in algebraic complexity theory3,5,7. The study of 
matrix multiplication symmetries can also provide insight into the 
asymptotic complexity of matrix multiplication5. By exploring this rich 
space of algorithms, we believe that AlphaT ensor will be useful for 
generating results and guiding mathematical research. See Supple-
mentary Information for proofs and details on the symmetries of 
factorizations.
Beyond standard matrix multiplication
T ensors can represent any bilinear operation, such as structured matrix 
multiplication, polynomial multiplication or more custom bilinear 
operations used in machine learning 27,28. We demonstrate here a 
use-case where AlphaT ensor finds a state-of-the-art algorithm for 
multiplying an n x n skew-symmetric matrix  with a vector of length n. 
Figure 4a shows the obtained decompositions for small instance sizes n.  
We observe a pattern that we generalize to arbitrary n, and prove that 
this yields a general algorithm for the skew-symmetric matrix-vector 
product (Fig.  4b). This algorithm, which uses nn n(− 1)(+ 2)/2 ~
1
2
2  
multiplications (where ∼ indicates asymptotic similarity), outperforms 
the previously known algorithms using  asymptotically n2 multiplica-
tions29, and is asymptotically optimal. See Supplementary Information 
for a proof, and for another use-case showing AlphaT ensor’s ability 
to re-discover the Fourier basis (see also Extended Data Table 2). This 
shows that AlphaT ensor can be applied to custom bilinear operations, 
and yield efficient algorithms leveraging the problem structure.
Rapid tailored algorithm discovery
We show a use-case where AlphaT ensor finds practically efficient matrix 
multiplication algorithms, tailored to specific hardware, with zero 
prior hardware knowledge. T o do so, we modify the reward of AlphaT-
ensor: we provide an additional reward at the terminal state (after the 
agent found a correct algorithm) equal to the negative of the runtime 
of the algorithm when benchmarked on the target hardware. That is, 
we set r rλ b′= +tt t, where rt is the reward scheme described in ‘DRL for 
algorithm discovery’ , bt is the benchmarking reward (non-zero only at 
the terminal state) and λ is a user-specified coefficient. Aside from the 
different reward, the exact same formulation of T ensorGame is used.
We train AlphaT ensor to search for efficient algorithms to multiply 
4 × 4 block matrices, and focus on square matrix multiplication of size 
8,192 (each block is hence of size 2,048) to define the benchmarking 
reward. AlphaT ensor searches for the optimal way of combining the 
16 square blocks of the input matrices on the considered hardware. We 
do not apply the 4 × 4 algorithm recursively, to leverage the efficient 
implementation of matrix multiplication on moderate-size matrices 
(2,048 × 2,048 in this case). We study two hardware devices commonly 
used in machine learning and scientific computing: an Nvidia V100 
graphics processing unit (GPU) and a Google tensor processing unit 
(TPU) v2. The factorization obtained by AlphaT ensor is transformed 
into JAX30 code, which is compiled ( just in time) before benchmarking.
Figure 5a,b shows the efficiency of the AlphaT ensor-discovered 
algorithms on the GPU and the TPU, respectively. AlphaT ensor dis-
covers algorithms that outperform the Strassen-square algorithm, 
which is a fast algorithm for large square matrices 31,32. Although 
the discovered algorithm has the same theoretical complexity as 
Strassen-square, it outperforms it in practice, as it is optimized for 
the considered hardware. Interestingly, AlphaT ensor finds algorithms 
n = 3
Extrapolationa
( n – 1 )( n + 2 )
2
multiplications.
(2)
(3) wij = aij ( bj – bi ) Computing the /f_irst ( n – 2 )( n + 1 )/2 intermediate products 
(4)
(5) qi = bi
n
j=1 aji
(7)
(8)
(9)
Input: n × n skew-symmetric matrix A, vector b.
Output: The resulting vector c = Ab computed in
(1) for i = 1, . . . , n − 2 do
for j = i + 1, . . . , n do
(4) for i = 1, . . . , n do
/uni03A3
(6) for i = 1, . . . , n − 2 do
Computing the /f_inal n intermediate products 
ci = i–1
j=1 wji +/uni03A3n
j=i+1wij – qi/uni03A3
cn–1 = – n–2
i=1/uni03A3wij –n–2
j=i+1/uni03A3 wjn +n–2
j=1/uni03A3 qi
n
i=1,i≠n–1/uni03A3
cn = – n–1
i=1/uni03A3wij +n
j=i+1/uni03A3 qi 
n–1
i=1/uni03A3
b WV U
n = 4 n = 5 n = 6 n = 10
Fig. 4 | Algorithm discovery beyond standard matrix multiplication.  
a, Decompositions found by AlphaTensor for the tensors of size nn××
nn(− 1)
2  
(with n = 3, 4, 5, 6) representing the skew-symmetric matrix-vector multiplication. 
The red pixels denote 1, the blue pixels denote −1 and the white pixels denote 0. 
Extrapolation to n = 10 is shown in the rightmost figure. b, Skew-symmetric 
matrix-by-vector multiplication algorithm, obtained from the examples solved 
by AlphaTensor. The wij and qi terms in steps 3 and 5 correspond to the m r terms 
in Algorithm 1. It is noted that steps 6–9 do not involve any multiplications.
52 | Nature | Vol 610 | 6 October 2022
Article
with a larger number of additions compared with Strassen-square (or 
equivalently, denser decompositions), but the discovered algorithms 
generate individual operations that can be efficiently fused by the 
specific XLA33 grouping procedure and thus are more tailored towards 
the compiler stack we use. The algorithms found by AlphaT ensor also 
provide gains on matrix sizes larger than what they were optimized 
for. Finally, Fig. 5c shows the importance of tailoring to particular 
hardware, as algorithms optimized for one hardware do not perform 
as well on other hardware.
Discussion
Trained from scratch, AlphaT ensor discovers matrix multiplication 
algorithms that are more efficient than existing human and 
computer-designed algorithms. Despite improving over known 
algorithms, we note that a limitation of AlphaT ensor is the need 
to pre-define a set of potential factor entries F , which discretizes 
the search space but can possibly lead to missing out on efficient 
algorithms. An interesting direction for future research is to adapt 
AlphaT ensor to search for F. One important strength of AlphaT ensor 
is its flexibility to support complex stochastic and non-differentiable 
rewards (from the tensor rank to practical efficiency on specific hard-
ware), in addition to finding algorithms for custom operations in a 
wide variety of spaces (such as finite fields). We believe this will spur 
applications of AlphaT ensor towards designing algorithms that opti-
mize metrics that we did not consider here, such as numerical stability 
or energy usage.
The discovery of matrix multiplication algorithms has far-reaching 
implications, as matrix multiplication sits at the core of many com -
putational tasks, such as matrix inversion, computing the determi -
nant and solving linear systems, to name a few7. We also note that our 
methodology can be extended to tackle related primitive mathemati-
cal problems, such as computing other notions of rank (for example, 
border rank—see Supplementary Information), and NP-hard matrix 
factorization problems (for example, non-negative factorization). By 
tackling a core NP-hard computational problem in mathematics using 
DRL—the computation of tensor ranks—AlphaT ensor demonstrates the 
viability of DRL in addressing difficult mathematical problems, and 
potentially assisting mathematicians in discoveries.
Online content
Any methods, additional references, Nature Research reporting summa-
ries, source data, extended data, supplementary information, acknowl-
edgements, peer review information; details of author contributions 
and competing interests; and statements of data and code availability 
are available at https://doi.org/10.1038/s41586-022-05172-4.
1. Silver, D. et al. A general reinforcement learning algorithm that masters chess, shogi, and 
Go through self-play. Science 362, 1140–1144 (2018).
2. Strassen, V. Gaussian elimination is not optimal. Numer. Math. 13, 354–356 (1969).
3. Bürgisser, P., Clausen, M. & Shokrollahi, A. Algebraic Complexity Theory Vol. 315 (Springer 
Science & Business Media, 2013).
4. Bläser, M. Fast matrix multiplication. Theory Comput. 5, 1–60 (2013).
5. Landsberg, J. M. Geometry and Complexity Theory 169 (Cambridge Univ. Press, 2017).
6. Pan, V. Y. Fast feasible and unfeasible matrix multiplication. Preprint at https://arxiv.org/
abs/1804.04102 (2018).
7. Lim, L.-H. Tensors in computations. Acta Numer. 30, 555–764 (2021).
8. Schönhage, A. Partial and total matrix multiplication. SIAM J. Comput. 10, 434–455 (1981).
9. Coppersmith, D. & Winograd, S. Matrix multiplication via arithmetic progressions. In ACM 
Symposium on Theory of Computing 1–6 (ACM, 1987).
10. Strassen, V. The asymptotic spectrum of tensors and the exponent of matrix 
multiplication. In 27th Annual Symposium on Foundations of Computer Science 49–54 
(IEEE, 1986).
11. Le Gall, F. Powers of tensors and fast matrix multiplication. In International Symposium on 
Symbolic and Algebraic Computation 296–303 (ACM, 2014).
12. Alman, J. & Williams, V. V. A refined laser method and faster matrix multiplication. In 
ACM-SIAM Symposium on Discrete Algorithms 522–539 (SIAM, 2021).
13. Gauss, C. F. Theoria Motus Corporum Coelestium in Sectionibus Conicis Solum 
Ambientium (Perthes and Besser, 1809).
14. Hillar, C. J. & Lim, L.-H. Most tensor problems are NP-hard. J. ACM 60, 1–39 (2013).
15. Laderman, J. D. A noncommutative algorithm for multiplying 3 × 3 matrices using 23 
multiplications. Bull. Am. Math. Soc. 82, 126–128 (1976).
16. Hopcroft, J. E. & Kerr, L. R. On minimizing the number of multiplications necessary for 
matrix multiplication. SIAM J. Appl. Math. 20, 30–36 (1971).
17. Vervliet, N., Debals, O., Sorber, L., Van Barel, M. & De Lathauwer, L. Tensorlab 3.0 (2016); 
https://www.tensorlab.net/
18. Smirnov, A. V. The bilinear complexity and practical algorithms for matrix multiplication. 
Comput. Math. Math. Phys. 53, 1781–1795 (2013).
19. Sedoglavic, A. & Smirnov, A. V. The tensor rank of 5x5 matrices multiplication is bounded 
by 98 and its border rank by 89. In Proc. 2021 on International Symposium on Symbolic 
and Algebraic Computation 345–351 (ACM, 2021).
20. Heule, M. J., Kauers, M. & Seidl, M. New ways to multiply 3 × 3-matrices. J. Symb. Comput. 
104, 899–916 (2021).
21. Hubert, T. et al. Learning and planning in complex action spaces. In International 
Conference on Machine Learning 4476–4486 (PMLR, 2021).
22. Zhang, W. & Dietterich, T. G. A reinforcement learning approach to job-shop scheduling. 
In International Joint Conferences on Artificial Intelligence Vol. 95, 1114–1120  
(Morgan Kaufmann Publishers, 1995).
8,192
10,240
12,288
14,336
16,384
18,432
20,480
Matrix size
4.3%
8.5%
6.8%
10.7%
10.1%
13.3%
16.1%
19.6%
13.8%
16.6%
15.3%
17.9%
21.3%
23.9%
AlphaTensor
8,192
10,240
12,288
14,336
16,384
18,432
20,480
Matrix size
6.6%
10.3%
9.0%
12.4%
8.9%
13.9%
9.2%
13.4%
7.2%
11.2%
6.9%
12.3%
8.4%
13.9%
TPU
GPUBenchmark device
10.3%
2.8%
4.4%
8.5%
Optimized for TPU
Optimized for GPU
Speed-up on Nvidia V100 GPU Speed-up on TPU v2 Speed-up of tailored agorithms
on both devices
a b c
Strassen-square
AlphaTensor
Strassen-square
Fig. 5 | Speed-ups of the AlphaTensor-discovered algorithm. a,b, Speed-ups 
(%) of the AlphaTensor-discovered algorithms tailored for a GPU (a) and a TPU 
(b), optimized for a matrix multiplication of size 8,192 × 8,192. Speed-ups are 
measured relative to standard (for example, cuBLAS for the GPU) matrix 
multiplication on the same hardware. Speed-ups are reported for various 
matrix sizes (despite optimizing the algorithm only on one matrix size). We also 
report the speed-up of the Strassen-square  algorithm. The median speed-up is 
reported over 200 runs. The standard deviation over runs is <0.4 percentage 
points (see Supplementary Information for more details). c, Speed-up of both 
algorithms (tailored to a GPU and a TPU) benchmarked on both devices.
Nature | Vol 610 | 6 October 2022 | 53
23. Vaswani, A. Attention is all you need. In International Conference on Neural Information 
Processing Systems Vol 30, 5998–6008 (Curran Associates, 2017).
24. Ho, J., Kalchbrenner, N., Weissenborn, D. & Salimans, T. Axial attention in 
multidimensional transformers. Preprint at https://arxiv.org/abs/1912.12180 (2019).
25. Drevet, C.-É., Islam, M. N. & Schost, É. Optimization techniques for small matrix 
multiplication. Theor. Comput. Sci. 412, 2219–2236 (2011).
26. Sedoglavic, A. A non-commutative algorithm for multiplying (7 × 7) matrices using 250 
multiplications. Preprint at https://arxiv.org/abs/1712.07935 (2017).
27. Battaglia, P. W. et al. Relational inductive biases, deep learning, and graph networks. 
Preprint at https://arxiv.org/abs/1806.01261 (2018).
28. Balog, M., van Merriënboer, B., Moitra, S., Li, Y. & Tarlow, D. Fast training of sparse graph 
neural networks on dense hardware. Preprint at https://arxiv.org/abs/1906.11786 (2019).
29. Ye, K. & Lim, L.-H. Fast structured matrix computations: tensor rank and Cohn–Umans 
method. Found. Comput. Math. 18, 45–95 (2018).
30. Bradbury, J. et al. JAX: composable transformations of Python+NumPy programs. GitHub 
http://github.com/google/jax (2018).
31. Benson, A. R. & Ballard, G. A framework for practical parallel fast matrix multiplication. 
ACM SIGPLAN Not. 50, 42–53 (2015).
32. Huang, J., Smith, T. M., Henry, G. M. & Van De Geijn, R. A. Strassen’s algorithm reloaded.  
In International Conference for High Performance Computing, Networking, Storage and 
Analysis 690–701 (IEEE, 2016).
33. Abadi, M. et al. Tensorflow: a system for large-scale machine learning. In USENIX 
Symposium On Operating Systems Design And Implementation 265–283 (USENIX,  
2016).
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in 
published maps and institutional affiliations.
Open Access This article is licensed under a Creative Commons Attribution 
4.0 International License, which permits use, sharing, adaptation, distribution 
and reproduction in any medium or format, as long as you give appropriate 
credit to the original author(s) and the source, provide a link to the Creative Commons license, 
and indicate if changes were made. The images or other third party material in this article are 
included in the article’s Creative Commons license, unless indicated otherwise in a credit line 
to the material. If material is not included in the article’s Creative Commons license and your 
intended use is not permitted by statutory regulation or exceeds the permitted use, you will 
need to obtain permission directly from the copyright holder. To view a copy of this license, 
visit http://creativecommons.org/licenses/by/4.0/.
© The Author(s) 2022
Article
Methods
TensorGame
T ensorGame is played as follows. The start position S0 of the game 
corresponds to the tensor T  representing the bilinear operation of 
interest, expressed in some basis. In each step t of the game, the player 
writes down three vectors (u(t), v(t), w(t)), which specify the rank-1 tensor 
u(t) ⊗ v(t) ⊗ w(t), and the state of the game is updated by subtracting the 
newly written down factor:
SS uv w←− ⊗⊗ . (2)tt
tt t
−1
() () ()
The game ends when the state reaches the zero tensor, 0=RS . This 
means that the factors written down throughout the game form a 
factorization of the start tensor S0, that is, =∑ ⊗⊗t
R tt t
0= 1
() () ()S uv w . 
This factorization is then scored. For example, when optimizing for 
asymptotic time complexity the score is −R , and when optimizing  
for practical runtime the algorithm corresponding to the factorization 
{(, ,) }tt t
t
R() () ()
=1uv w  is constructed (see Algorithm 1) and then bench-
marked on the fly (see Supplementary Information).
In practice, we also impose a limit Rlimit on the maximum number of 
moves in the game, so that a weak player is not stuck in unnecessarily 
(or even infinitely) long games. When a game ends  because it has run 
out of moves, a penalty score is given so that it is never advantageous 
to deliberately exhaust the move limit. For example, when optimizing 
for asymptotic time complexity, this penalty is derived from an upper 
bound on the tensor rank of the final residual tensor R limitS . This upper 
bound on the tensor rank is obtained by summing the matrix ranks of 
the slices of the tensor.
TensorGame over rings. We say that the decomposition of nT  in equa-
tion (1) is in a ring E (defining the arithmetic operations) if each of the 
factors u(t), v(t) and w(t) has entries belonging to the set E, and additions 
and multiplications are interpreted according to E. The tensor rank 
depends, in general, on the ring. At each step of T ensorGame, the ad-
ditions and multiplications in equation ( 2) are interpreted in E. For 
example, when working in 2Z , (in this case, the factors u(t), v(t) and w(t) 
live in F = {0, 1}), a modulo 2 operation is applied after each state update 
(equation (2)).
We note that integer-valued decompositions u(t), v(t) and w(t) lead to 
decompositions in arbitrary rings E. Hence, provided F only contains 
integers, algorithms we find in standard arithmetic apply more gener-
ally to any ring.
AlphaTensor
AlphaT ensor builds on AlphaZero 1 and its extension Sampled 
AlphaZero21, combining a deep neural network with a sample-based 
MCTS search algorithm.
The deep neural network, fθ(s) = (π, z) parameterized by θ, takes as 
input the current state s of the game and outputs a probability distribu-
tion π(⋅∣s) over actions and z(⋅∣s) over returns (sum of future rewards) G. 
The parameters θ of the deep neural network are trained by reinforce-
ment learning from self-play games and synthetic demonstrations. 
Self-play games are played by actors, running a sample-based MCTS 
search at every state st encountered in the game. The MCTS search 
returns an improved probability distribution over moves from which 
an action at is selected and applied to the environment. The sub-tree 
under at is reused for the subsequent search at st+1. At the end of the 
game, a return G is obtained and the trajectory is sent to the learner to 
update the neural network parameters θ. The distribution over returns 
z(⋅∣st) is learned through distributional reinforcement learning using 
the quantile regression distributional loss34, and the network policy 
π(⋅∣st) is updated using a Kullback–Leibler divergence loss, to maximize 
its similarity to the search policy for self-play games or to the next 
action for synthetic demonstrations. We use the Adam optimizer 35 
with decoupled weight decay36 to optimize the parameters θ of the 
neural network.
Sample-based MCTS search. The sample-based MCTS search is very 
similar to the one described in Sampled AlphaZero. Specifically, the 
search consists of a series of simulated trajectories of T ensorGame that 
are aggregated in a tree. The search tree therefore consists of nodes 
representing states and edges representing actions. Each state-action 
pair (s, a) stores a set of statistics Ns aQ sa πs a(, ), (, ), ˆ( ,) , where N(s, a) 
is the visit count, Q(s, a) is the action value and π saˆ( ,)  is the empirical 
policy probability. Each simulation traverses the tree from the root 
state s0 until a leaf state sL is reached by recursively selecting in each 
state s an action a that has not been frequently explored, has high em-
pirical policy probability and high value. Concretely, actions within 
the tree are selected by maximizing over the probabilistic upper con-
fidence tree bound
21,37
∑
Qs ac sπ sa
Ns b
Ns aargmax (, )+ () ⋅ˆ (, )
(, )
1+ (, ) ,
a
b
where c(s) is an exploration factor controlling the influence of the 
empirical policy π saˆ( ,)  relative to the values Q(s, a) as nodes are visited 
more often. In addition, a transposition table is used to recombine 
different action sequences if they reach the exact same tensor. This 
can happen particularly often in T ensorGame as actions are commuta-
tive. Finally, when a leaf state sL is reached, it is evaluated by the neural 
network, which returns K actions {ai} sampled from π(a∣sL), alongside 
the empirical distribution ∑π as δˆ( )= Ki aaL
1
, i  and a value v (sL) con-
structed from z(⋅∣sL). Differently from AlphaZero and Sampled Alp-
haZero, we chose v not to be the mean of the distribution of returns 
z(⋅∣sL) as is usual in most reinforcement learning agents, but instead to 
be a risk-seeking value, leveraging the facts that T ensorGame is a deter-
ministic environment and that we are primarily interested in finding 
the best trajectory possible. The visit counts and values on the simu-
lated trajectory are then updated in a backward pass as in Sampled 
AlphaZero.
Policy improvement. After simulating N(s) trajectories from state s 
using MCTS, the normalized visit counts of the actions at the root of 
the search tree N(s, a)/N(s) form a sample-based improved policy. Dif-
ferently from AlphaZero and Sampled AlphaZero, we use an adaptive 
temperature scheme to smooth the normalized visit counts distribution 
as some states can accumulate an order of magnitude more visits than 
others because of sub-tree reuse and transposition table. Concretely, 
we define the improved policy as 
πs aN sa Ns bˆ( ,) =( ,) /∑ (, )τs
b
τs1/ () 1/ ()I  
where τs Ns NN N() =l og () /log if >  and 1 otherwise, with N  being a 
hyperparameter. For training, we use Iπˆ directly as a target for the 
network policy π. For acting, we additionally discard all actions that 
have a value lower than the value of the most visited action, and sample 
proportionally to πˆI  among those remaining high-value actions.
Learning one agent for multiple target tensors. We train a single 
agent to decompose the different tensors nm p,,T  in a given arithmetic 
(standard or modular). As the network works with fixed-size inputs, we 
pad all tensors (with zeros) to the size of the largest tensor we consider 
( 5T , of size 25 × 25 × 25). At the beginning of each game, we sample uni-
formly at random a target nm p,,T , and play T ensorGame. Training a sin-
gle agent on different targets leads to better results thanks to the trans-
fer between targets. All our results reported in Fig. 3 are obtained using 
multiple runs of this multi-target setting. We also train a single agent 
to decompose tensors in both arithmetics. Owing to learned transfer 
between the two arithmetics, this agent discovers a different distribu-
tion of algorithms (of the same ranks) in standard arithmetic than the 
agent trained on standard arithmetic only, thereby increasing the over-
all diversity of discovered algorithms.
Synthetic demonstrations. The synthetic demonstrations buffer  
contains tensor-factorization pairs, where the factorizations 
uv w{(, ,) }rr r
r
R() () ()
=1 are first generated at random, after which the tensor 
uv w=∑ ⊗⊗r
R rr r
=1
() () ()D  is formed. We create a dataset containing  
5 million such tensor-factorization pairs. Each element in the factors 
is sampled independently and identically distributed (i.i.d.) from a 
given categorical distribution over F (all possible values that can be 
taken). We discarded instances whose decompositions were clearly 
suboptimal (contained a factor with u = 0, v = 0, or w = 0).
In addition to these synthetic demonstrations, we further add to 
the demonstration buffer previous games that have achieved large 
scores to reinforce the good moves made by the agent in these games.
Change of basis. The rank of a bilinear operation does not depend on 
the basis in which the tensor representing it is expressed, and for any 
invertible matrices A, B and C we have Rank( )= Rank () (, ,)TT AB C , where 
(, ,)AB CT  is the tensor after change of basis given by
∑∑ ∑=. (3)ijk
a
S
b
S
c
S
ia jb kc abc
(, ,)
=1 =1 =1
TT AB CAB C
Hence, exhibiting a rank-R decomposition of the matrix multiplica-
tion tensor nT  expressed in any basis proves that the product of two 
n × n matrices can be computed using R scalar multiplications. Moreo-
ver, it is straightforward to convert such a rank-R decomposition into 
a rank-R decomposition in the canonical basis, thus yielding a practical 
algorithm of the form shown in Algorithm 1. We leverage this observa-
tion by expressing the matrix multiplication tensor nT  in a large number 
of randomly generated bases (typically 100,000) in addition to the 
canonical basis, and letting AlphaT ensor play games in all bases in 
parallel.
This approach has three appealing properties: (1) it provides a natural 
exploration mechanism as playing games in different bases automati-
cally injects diversity into the games played by the agent; (2) it exploits 
properties of the problem as the agent need not succeed in all bases—it 
is sufficient to find a low-rank decomposition in any of the bases; (3) 
it enlarges coverage of the algorithm space because a decomposition 
with entries in a finite set F = {−2, −1, 0, 1, 2} found in a different basis 
need not have entries in the same set when converted back into the 
canonical basis.
In full generality, a basis change for a 3D tensor of size S × S × S is 
specified by three invertible S × S matrices A, B and C. However, in our 
procedure, we sample bases at random and impose two restrictions: 
(1) A = B = C, as this performed better in early experiments, and (2) 
unimodularity ( Adet∈ {−1, +1 }), which ensures that after converting 
an integral factorization into the canonical basis it still contains integer 
entries only (this is for representational convenience and numerical 
stability of the resulting algorithm). See Supplementary Information 
for the exact algorithm.
Signed permutations. In addition to playing (and training on) games in 
different bases, we also utilize a data augmentation mechanism when-
ever the neural network is queried in a new MCTS node. At acting time, 
when the network is queried, we transform the input tensor by applying 
a change of basis—where the change of basis matrix is set to a random 
signed permutation. We then query the network on this transformed 
input tensor, and finally invert the transformation in the network’s 
policy predictions. Although this data augmentation procedure can 
be applied with any generic change of basis matrix (that is, it is not 
restricted to signed permutation matrices), we use signed permuta-
tions mainly for computational efficiency. At training time, whenever 
the neural network is trained on an (input, policy targets, value target) 
triplet (Fig. 2), we apply a randomly chosen signed permutation to 
both the input and the policy targets, and train the network on this 
transformed triplet. In practice, we sample 100 signed permutations 
at the beginning of an experiment, and use them thereafter.
Action canonicalization. For any λ1, λ2, λ3 ∈ {−1, +1} such that λ1λ2λ3 = 1, 
the actions (λ1u, λ2v, λ3w) and (u, v, w) are equivalent because they lead 
to the same rank-one tensor (λ1u) ⊗ (λ2v) ⊗ (λ3w) = u ⊗ v ⊗ w. T o pre-
vent the network from wasting capacity on predicting multiple equiva-
lent actions, during training we always present targets (u, v, w) for the 
policy head in a canonical form, defined as having the first non-zero 
element of u and the first non-zero element of v strictly positive. This is 
well defined because u or v cannot be all zeros (if they are to be part of 
a minimal rank decomposition), and for any (u, v, w) there are unique 
λ1, λ2, λ3 ∈ {−1, +1} (with λ1λ2λ3 = 1) that transform it into canonical form. 
In case the network predicts multiple equivalent actions anyway, we 
merge them together (summing their empirical policy probabilities) 
before inserting them into the MCTS tree.
Training regime. We train AlphaT ensor on a TPU v3, with a total batch 
size of 2,048. We use 64 TPU cores, and train for 600,000 iterations. 
On the actor side, the games are played on standalone TPU v4, and we 
use 1,600 actors. In practice, the procedure  takes a week to converge.
Neural network
The architecture is composed of a torso, followed by a policy head that 
predicts a distribution over actions, and a value head that predicts a dis-
tribution of the returns from the current state (see Extended Data Fig. 3).
Input. The input to the network contains all the relevant information 
of the current state and is composed of a list of tensors and a list of 
scalars. The most important piece of information is the current 3D 
tensor St  of size S × S × S. (For simplicity, in the description here we 
assume that all the three dimensions of the tensor are equal in size. The 
generalization to different sizes is straightforward.) In addition, the 
model is given access to the last h actions (h being a hyperparameter 
usually set to 7), represented as h rank-1 tensors that are concatenated 
to the input. The list of scalars includes the time index t of the current 
action (where 0 ≤ t < R
limit).
Torso. The torso of the network is in charge of mapping both scalars and 
tensors from the input to a representation that is useful to both policy 
and value heads. Its architecture is based on a modification of transform-
ers23, and its main signature is that it operates over three S × S grids pro-
jected from the S × S × S input tensors. Each grid represents two out of 
the three modes of the tensor. Defining the modes of the tensor as 
,,U VW , the rows and columns of the first grid are associated to U and V, 
respectively, the rows and columns of the second grid are associated to 
W and U, and the rows and columns of the third grid are associated to V 
and W. Each element of each grid is a feature vector, and its initial value 
is given by the elements of the input tensors along the grid’s missing 
mode. These feature vectors are enriched by concatenating an S × S × 1 
linear projection from the scalars. This is followed by a linear layer pro-
jecting these feature vectors into a 512-dimensional space.
The rest of the torso is a sequence of attention-based blocks with the 
objective of propagating information between the three grids. Each of 
those blocks has three stages, one for every pair of grids. In each stage, 
the grids involved are concatenated, and axial attention24 is performed 
over the columns. It is noted that in each stage we perform in parallel S  
self-attention operations of 2S elements in each. The representation 
sent to the policy head corresponds to the 3S2 512-dimensional feature 
vectors produced by the last layer of the torso. A detailed description 
of the structure of the torso is specified in Extended Data Fig. 4 (top) 
and Appendix A.1.1 in Supplementary Information.
Policy head. The policy head uses the transformer architecture 23 
to model an autoregressive policy. Factors are decomposed into k 
Article
tokens of dimensionality d  such that k × d = 3S. The transformer con -
ditions on the tokens already generated and cross-attends to the fea -
tures produced by the torso. At training time, we use teacher-forcing, 
that is, the ground truth actions are decomposed into tokens and 
taken as inputs into the causal transformer in such a way that the 
prediction of a token depends only on the previous tokens. At infer -
ence time, K  actions are sampled from the head. The feature repre-
sentation before the last linear layer of the initial step (that is, the 
only step that is not conditioned on the ground truth) is used as an 
input to the value head, described below. Details of the architecture 
are presented in Extended Data Fig. 4 (centre) and Appendix A.1.2 in 
Supplementary Information.
Value head. The value head is composed of a four-layer multilayer 
perceptron whose last layer produces q outputs corresponding to the 
,, …qq
q
q
1
2
3
2
2− 1
2  quantiles. In this way, the value head predicts the distri-
bution of returns from this state in the form of values predicted for the 
aforementioned quantiles34. At inference time, we encourage the agent 
to be risk-seeking by using the average of the predicted values for quan-
tiles over 75%. A detailed description of the value head is presented in 
Extended Data Fig. 4 (bottom) and Appendix A.1.3 in Supplementary 
Information.
Related work
The quest for efficient matrix multiplication algorithms started with 
Strassen’s breakthrough in ref. 2, which showed that one can multiply 
2 × 2 matrices using 7 scalar multiplications, leading to an algorithm 
of complexity O n() 2.81 . This led to the development of a very active field 
of mathematics attracting worldwide interest, which studies the asymp-
totic complexity of matrix multiplication (see refs. 3–6). So far, the best 
known complexity for matrix multiplication is n() 2.37286O  (ref. 12), which 
improves over ref. 11, and builds on top of fundamental results in the 
field8–10. However, this does not yield practical algorithms, as such 
approaches become advantageous only for astronomical matrix sizes. 
Hence, a significant body of work aims at exhibiting explicit factoriza-
tions of matrix multiplication tensors, as these factorizations provide 
practical algorithms. After Strassen’s breakthrough showing that 
Trank( )≤ 72 , efficient algorithms for larger matrix sizes were 
found15,16,18,26,38. Most notably, Laderman showed in ref. 15 that 3 × 3 
matrix multiplications can be performed with 23 scalar multiplications. 
In addition to providing individual low-rank factorizations, an impor-
tant research direction aims at understanding the space of matrix 
multiplication algorithms—as opposed to exhibiting individual 
low-rank factorizations—by studying the symmetry groups and diver-
sity of factorizations (see ref. 5 and references therein). For example, 
the symmetries of 2 × 2 matrix multiplication were studied in refs. 39–42, 
where Strassen’s algorithm was shown to be essentially unique. The 
case of 3 × 3 was studied in ref. 43, whereas a symmetric factorization 
for all n is provided in ref. 44.
On the computational front, continuous optimization has been the 
main workhorse for decomposing tensors17,45,46, and in particular matrix 
multiplication tensors. Such continuous optimization procedures (for 
example, alternating least squares), however, yield approximate solu-
tions, which correspond to inexact matrix multiplication algorithms 
with floating point operations. T o circumvent this issue, regularization 
procedures have been proposed, such as ref. 18, to extract exact decom-
positions. Unfortunately, such approaches often require  substantial 
human intervention and expertise to decompose large tensors. A dif-
ferent line of attack was explored in refs. 47,48, based on learning the 
continuous weights of a two-layer network that mimics the structure 
of the matrix multiplication operation. This method, which is trained 
through supervised learning of matrix multiplication examples, finds 
approximate solutions to 2 × 2 and 3 × 3 matrix multiplications.  
In ref. 48, a quantization procedure is further used to obtain an exact 
decomposition for 2 × 2. Unlike continuous optimization-based 
approaches, AlphaT ensor directly produces algorithms from the 
desired set of valid algorithms, and is flexible in that it allows us to 
optimize a wide range of (even non-differentiable) objectives. This 
unlocks tackling broader settings (for example, optimization in finite 
fields, optimization of runtime), as well as larger problems (for exam-
ple, T4 and T5) than those previously considered. Different from con-
tinuous optimization, a boolean satisfiability (SAT) based  formulation 
of the problem of decomposing 3 × 3 matrix multiplication was 
recently proposed in ref. 20, which adds thousands of new decompo -
sitions of rank 23 to the list of known 3 × 3 factorizations. The approach 
relies on a state-of-the-art SAT solving procedure, where several 
assumptions and simplifications are made on the factorizations to 
reduce the search space. As is, this approach is, however, unlikely to 
scale to larger tensors, as the search space grows very quickly with 
the size.
On the practical implementation front, ref. 31 proposed several ideas 
to speed up implementation of fast matrix multiplication algorithms 
on central processing units (CPUs). Different fast algorithms are then 
compared and benchmarked, and the potential speed-up of such algo-
rithms is shown against standard multiplication. Other works focused 
on getting the maximal performance out of a particular fast matrix 
multiplication algorithm (Strassen’s algorithm with one or two levels 
of recursion) on a CPU32 or a GPU49. These works show that, despite 
popular belief, such algorithms are of practical value. We see writing 
a custom low-level implementation of a given algorithm to be distinct 
from the focus of this paper—developing new efficient algorithms—and 
we believe that the algorithms we discovered can further benefit from 
a more efficient implementation by experts.
Beyond matrix multiplication and bilinear operations, a growing 
amount of research studies the use of optimization and machine learn-
ing to improve the efficiency of computational operations. There 
are three levels of abstractions at which this can be done: (1) in the 
hardware design, for example, chip floor planning50, (2) at the hard-
ware–software interface, for example, program super-optimization 
of a reference implementation for specific hardware51, and (3) on the 
algorithmic level, for example, program induction52, algorithm selec-
tion53 or meta-learning54. Our work focuses on the algorithmic level of 
abstraction, although AlphaT ensor is also flexible to discover efficient 
algorithms for specific hardware. Different from previous works, we 
focus on discovering matrix multiplication algorithms that are prov-
ably correct, without requiring initial reference implementations. We 
conclude by relating our work broadly to existing reinforcement learn-
ing methods for scientific discovery. Within mathematics, reinforce-
ment learning was applied, for example, to theorem proving
55–58, and 
to finding counterexamples refuting conjectures in combinatorics and 
graph theory59. Reinforcement learning was further shown to be useful 
in many areas in science, such as molecular design60,61 and synthesis62 
and optimizing quantum dynamics63.
Data availability
The data used to train the system were generated synthetically accord-
ing to the procedures explained in the paper. The algorithms discov-
ered by AlphaT ensor are available for download at https://github.com/
deepmind/alphatensor.
Code availability
An interactive notebook with code to check the non-equivalence of algo-
rithms is provided. Moreover, the fast algorithms from the ‘ Algorithm 
discovery results’ section on a GPU and a TPU are provided. These are 
available for download at https://github.com/deepmind/alphatensor.  
A full description of the AlphaZero algorithm that this work is based 
on is available in ref. 1, and the specific neural network architecture we 
use is described using pseudocode in the Supplementary Information. 
34. Dabney, W., Rowland, M., Bellemare, M. & Munos, R. Distributional reinforcement learning 
with quantile regression. In AAAI Conference on Artificial Intelligence Vol. 32, 2892–2901 
(AAAI Press, 2018).
35. Kingma, D. P., & Ba, J. Adam: a method for stochastic optimization. In International 
Conference on Learning Representations (ICLR) (2015).
36. Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. In International 
Conference on Learning Representations (ICLR) (2019).
37. Silver, D. et al. Mastering the game of Go with deep neural networks and tree search. 
Nature 529, 484–489 (2016).
38. Sedoglavic, A. A non-commutative algorithm for multiplying 5x5 matrices using 99 
multiplications. Preprint at https://arxiv.org/abs/1707.06860 (2017).
39. de Groote, H. F. On varieties of optimal algorithms for the computation of bilinear 
mappings II. optimal algorithms for 2 × 2-matrix multiplication. Theor. Comput. Sci. 7,  
127–148 (1978).
40. Burichenko, V. P. On symmetries of the Strassen algorithm. Preprint at https://arxiv.org/
abs/1408.6273 (2014).
41. Chiantini, L., Ikenmeyer, C., Landsberg, J. M. & Ottaviani, G. The geometry of rank 
decompositions of matrix multiplication I: 2 × 2 matrices. Exp. Math. 28, 322–327 (2019).
42. Grochow, J. A. & Moore, C. Designing Strassen’s algorithm. Preprint at https://arxiv.org/
abs/1708.09398 (2017).
43. Ballard, G., Ikenmeyer, C., Landsberg, J. M. & Ryder, N. The geometry of rank 
decompositions of matrix multiplication II: 3 × 3 matrices. J. Pure Appl. Algebra 223, 
3205–3224 (2019).
44. Grochow, J. A. & Moore, C. Matrix multiplication algorithms from group orbits. Preprint at 
https://arxiv.org/abs/1612.01527 (2016).
45. Kolda, T. G. & Bader, B. W. Tensor decompositions and applications. SIAM Rev. 51,  
455–500 (2009).
46. Bernardi, A., Brachat, J., Comon, P. & Mourrain, B. General tensor decomposition, 
moment matrices and applications. J. Symb. Comput. 52, 51–71 (2013).
47. Elser, V. A network that learns Strassen multiplication. J. Mach. Learn. Res. 17, 3964–3976 
(2016).
48. Tschannen, M., Khanna, A. & Anandkumar, A, StrassenNets: deep learning with a 
multiplication budget. In International Conference on Machine Learning 4985–4994 
(PMLR, 2018).
49. Huang, J., Yu, C. D. & Geijn, R. A. V. D. Strassen’s algorithm reloaded on GPUs. ACM Trans. 
Math. Softw. 46, 1–22 (2020).
50. Mirhoseini, A. et al. A graph placement methodology for fast chip design. Nature 594, 
207–212 (2021).
51. Bunel, R., Desmaison, A., Kohli, P., Torr, P. H. & Kumar, M. P. Learning to superoptimize 
programs. In International Conference on Learning Representations (ICLR) (2017).
52. Li, Y., Gimeno, F., Kohli, P. & Vinyals, O. Strong generalization and efficiency in neural 
programs. Preprint at https://arxiv.org/abs/2007.03629 (2020).
53. Lagoudakis, M. G. et al. Algorithm selection using reinforcement learning. In International 
Conference on Machine Learning 511–518 (Morgan Kaufmann Publishers, 2000).
54. Schmidhuber, J. Evolutionary Principles in Self-Referential Learning. On Learning now to 
Learn: The Meta-Meta-Meta...-Hook. Diploma thesis, Technische Univ. Munchen (1987).
55. Kaliszyk, C., Urban, J., Michalewski, H. & Olšák, M. Reinforcement learning of theorem 
proving. In International Conference on Neural Information Processing Systems  
8836–8847 (Curran Associates, 2018).
56. Piotrowski, B. & Urban, J. ATPboost: learning premise selection in binary setting with ATP 
feedback. In International Joint Conference on Automated Reasoning 566–574 (Springer, 
2018).
57. Bansal, K., Loos, S., Rabe, M., Szegedy, C. & Wilcox, S. HOList: an environment for 
machine learning of higher order logic theorem proving. In International Conference on 
Machine Learning 454–463 (PMLR, 2019).
58. Zombori, Z., Urban, J. & Brown, C. E. Prolog technology reinforcement learning prover.  
In International Joint Conference on Automated Reasoning 489–507 (Springer, 2020).
59. Wagner, A. Z. Constructions in combinatorics via neural networks. Preprint at https://
arxiv.org/abs/2104.14516 (2021).
60. Popova, M., Isayev, O. & Tropsha, A. Deep reinforcement learning for de novo drug 
design. Sci. Adv. 4, eaap7885 (2018).
61. Zhou, Z., Kearnes, S., Li, L., Zare, R. N. & Riley, P. Optimization of molecules via deep 
reinforcement learning. Sci. Rep. 9, 10752 (2019).
62. Segler, M. H., Preuss, M. & Waller, M. P. Planning chemical syntheses with deep neural 
networks and symbolic AI. Nature 555, 604–610 (2018).
63. Dalgaard, M., Motzoi, F., Sørensen, J. J. & Sherson, J. Global optimization of quantum 
dynamics with AlphaZero deep exploration. npj Quantum Inf. 6, 6 (2020).
64. Fast matrix multiplication algorithms catalogue. Université de Lille https://fmm.univ- 
lille.fr/ (2021).
Acknowledgements We thank O. Fawzi, H. Fawzi, C. Ikenmeyer, J. Ellenberg, C. Umans and  
A. Wigderson for the inspiring discussions on the use of machine learning for maths; A. Davies, 
A. Gaunt, P. Mudigonda, R. Bunel and O. Ronneberger for their advice on early drafts of the 
paper; A. Ruderman, M. Bauer, R. Leblond, R. Kabra and B. Winckler for participating in a 
hackathon at the early stages of the project; D. Visentin, R. Tanburn and S. Noury for sharing 
their expertise on TPUs; P. Wang and R. Zhao for their help on benchmarking algorithms;  
G. Holland, A. Pierce, N. Lambert and C. Meyer for assistance coordinating the research; and 
our colleagues at DeepMind for encouragement and support.
Author contributions A.F. conceived the project, with support from B.R.-P. and P.K.; T.H., A.H. 
and J.S. developed the initial AlphaZero codebase, and B.R.-P., M. Balog, A.F., A.N., F.J.R.R. and 
G.S. developed an early supervised network prototype. A.H., T.H., B.R.-P., M. Barekatain and  
J.S. designed the network architecture used in the paper. T.H., J.S., A.H., M. Barekatain, A.F.,  
M. Balog and F.J.R.R. developed the tensor decomposition environment and data generation 
pipeline, and A.H., T.H., M. Barekatain, M. Balog, B.R.-P., F.J.R.R. and A.N. analysed the 
experimental results and algorithms discovered by AlphaTensor. A.N., A.F. and T.H. developed 
the benchmarking pipeline and experiments, and B.R.-P., F.J.R.R. and A.N. extended the 
approach to structured tensors. A.F., B.R.-P., G.S. and A.N. proved the results in the paper.  
D.S., D.H. and P.K. contributed technical advice and ideas. A.F., M. Balog, B.R.-P., F.J.R.R., A.N. 
and T.H. wrote the paper. These authors contributed equally, and are listed alphabetically by last 
name after the corresponding author: A.F., M. Balog, A.H., T.H., B.R.-P. These authors contributed 
equally, and are listed alphabetically by last name: M. Barekatain, A.N., F.J.R.R., J.S. and G.S.
Competing interests The authors of the paper are planning to file a patent application relating 
to subject matter contained in this paper in the name of DeepMind Technologies Limited.
Additional information
Supplementary information The online version contains supplementary material available at 
https://doi.org/10.1038/s41586-022-05172-4.
Correspondence and requests for materials should be addressed to Alhussein Fawzi.
Peer review information Nature thanks Grey Ballard, Jordan Ellenberg, Lek-Heng Lim, Michael 
Littman and the other, anonymous, reviewer(s) for their contribution to the peer review of this 
work.
Reprints and permissions information is available at http://www.nature.com/reprints.
Article
Extended Data Fig. 1 | Algorithm for multiplying 4 × 4 matrices in modular arithmetic ( 2Z ) with 47 multiplications. This outperforms the two-level Strassen’s 
algorithm, which involves 7 2 = 49 multiplications.
Extended Data Fig. 2 | Algorithm for multiplying 4 × 5 by 5 × 5 matrices in standard arithmetic with 76 multiplications. This outperforms the previously best 
known algorithm, which involves 80 multiplications.
Article
Extended Data Fig. 3 | AlphaTensor’s network architecture. The network 
takes as input the list of tensors containing the current state and previous 
history of actions, and a list of scalars, such as the time index of the current 
action. It produces two kinds of outputs: one representing the value, and the 
other inducing a distribution over the action space from which we can sample 
from. The architecture of the network is accordingly designed to have a 
common torso, and two heads, the value and the policy heads. c is set to 512 in 
all experiments.
Extended Data Fig. 4 | Detailed view of AlphaTensor’s architecture, included torso, policy and value head. We refer to Algorithms A.1-A.11 in Supplementary 
Information for the details of each component.
Article
Extended Data Table 1 | Rank results obtained by combining decompositions (in standard arithmetic)
The table shows the cases where we were able to obtain an improvement over state-of-the-art, for tensors Tnm p,,  (with n, m, p≤12). The recipe column indicates the low-level matrix multiplica-
tion algorithms used to build the corresponding factorization. 〈n, m, p〉 denotes the best known bound on the rank of Tnm p,, ; see Appendix H in Supplementary Information for more details. For 
tensors that were directly decomposed by AlphaTensor, the recipe shows a star mark, e.g. 〈3, 4, 5〉*. All the factorizations are made available.
Extended Data Table 2 | Result of applying AlphaTensor to the tensor representing the cyclic convolution operation
AlphaTensor finds the discrete Fourier matrix (DFT) and the inverse DFT matrix in finite fields. The figure shows the decompositions found by AlphaTensor of the n × n × n tensor representing the 
cyclic convolution of two vectors, for three different values of n in the finite field of order 17. The action space, characterized by the number of possible factor triplets {u(r), v(r), w(r)}, is thus 173n, 
which is of the order of 1029 for n = 8. Despite the huge action space, AlphaTensor finds the optimal rank-n decompositions for the three values of n. The factors in the figure are stacked vertically, 
i.e., U = [u(1), …, u(n)]. For ease of visualization, the factor entries have been expressed in terms of powers of an n-th primitive root of unity in the finite field. Within each column, each colour 
uniquely represents one element of the field (e.g., for the column n = 4, we have depicted in grey 40 = 44 = 4−4 = 1). By inspecting the patterns in the decompositions, one could extrapolate the 
results for other values of n and other fields. Indeed, the factors u(r) and v(r) correspond to the DFT coefficients, since ==u vz kr
k
r
k
r() () , whereas the factors w(r) correspond to the inverse DFT, since 
wz n/kr
k
r() = −  for 0≤k, r < n, where z is an n-th primitive root of unity (i.e., zn = 1 and zj ≠ 1 for any 1≤j < n).

 
Edinburgh Research Explorer 
 
 
Synthesizing Benchmarks for Predictive Modeling
Citation for published version:
Cummins, C, Petoumenos, P, Wang, Z & Leather, H 2017, Synthesizing Benchmarks for Predictive
Modeling. in 2017 IEEE/ACM International Symposium on Code Generation and Optimization (CGO).
International Symposium on Code Generation and Optimization, Institute of Electrical and Electronics
Engineers, Austin, TX, USA, pp. 86-99, International Symposium on Code Generation and Optimization
(CGO) 2017, Austin, Texas, United States, 4/02/17. https://doi.org/10.1109/CGO.2017.7863731
Digital Object Identifier (DOI):
10.1109/CGO.2017.7863731
Link:
Link to publication record in Edinburgh Research Explorer
Document Version:
Peer reviewed version
Published In:
2017 IEEE/ACM International Symposium on Code Generation and Optimization (CGO)
General rights
Copyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)
and / or other copyright owners and it is a condition of accessing these publications that users recognise and
abide by the legal requirements associated with these rights.
Take down policy
The University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorer
content complies with UK legislation. If you believe that the public display of this file breaches copyright please
contact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately and
investigate your claim.
Download date: 28. Dec. 2025
Synthesizing Benchmarks for Predictive Modeling
Chris Cummins
Pavlos Petoumenos
University of Edinburgh
{c.cummins,ppetoume}@inf.ed.ac.uk
Zheng Wang
Lancaster University
z.wang@lancaster.ac.uk
Hugh Leather
University of Edinburgh
hleather@inf.ed.ac.uk
Abstract
Predictive modeling using machine learning is an effective
method for building compiler heuristics, but there is a short-
age of benchmarks. Typical machine learning experiments
outside of the compilation ﬁeld train over thousands or mil-
lions of examples. In machine learning for compilers, how-
ever, there are typically only a few dozen common bench-
marks available. This limits the quality of learned models,
as they have very sparse training data for what are often
high-dimensional feature spaces. What is needed is a way
to generate an unbounded number of training programs that
ﬁnely cover the feature space. At the same time the generated
programs must be similar to the types of programs that human
developers actually write, otherwise the learning will target
the wrong parts of the feature space.
We mine open source repositories for program fragments
and apply deep learning techniques to automatically con-
struct models for how humans write programs. We sample
these models to generate an unbounded number of runnable
training programs. The quality of the programs is such that
even human developers struggle to distinguish our generated
programs from hand-written code.
We use our generator for OpenCL programs, CLgen, to
automatically synthesize thousands of programs and show
that learning over these improves the performance of a
state of the art predictive model by
1.27×. In addition, the
ﬁne covering of the feature space automatically exposes
weaknesses in the feature design which are invisible with
the sparse training examples from existing benchmark suites.
Correcting these weaknesses further increases performance
by 4.30×.
Categories and Subject Descriptors D.3.4 [ Program-
ming Languages]: Processors—code generation, compilers,
optimization
Keywords Synthetic program generation, OpenCL, Bench-
marking, Deep Learning, GPUs
1. Introduction
Predictive modeling is a well researched method for building
optimization heuristics that often exceed human experts and
Ad-hoc 
Drivers
clsmithclsmithDatasets
clsmithclsmithTraining 
Programs
Feature 
Extractor
clsmithclsmithTraining 
Data
Predictive 
Model
Parameters
Features
Performance 
measurements
Figure 1. Training a predictive model.
reduces development time [1–11]. Figure 1 shows the process
by which these models are trained. A set of training programs
are identiﬁed which are expected to be representative of the
application domain. The programs are compiled and executed
with different parameter values for the target heuristic, to de-
termine which are the best values for each training program.
Each program is also summarized by a vector of features
which describe the information that is expected to be impor-
tant in predicting the best heuristic parameter values. These
training examples of program features and desired heuristic
values are used to create a machine learning model which,
when given the features from a new, unseen program, can
predict good heuristic values for it.
It is common for feature vectors to contain dozens of
elements. This means that a large volume of training data is
needed to have an adequate sampling over the feature space.
Without it, the machine learned models can only capture
the coarse characteristics of the heuristic, and new programs
which do not lie near to training points may be wrongly
predicted. The accuracy of the machine learned heuristic is
thus limited by the sparsity of the training points.
There have been efforts to solve this problem using tem-
plates. The essence of the approach is to construct a prob-
abilistic grammar with embedded semantic actions that de-
ﬁnes a language of possible programs. New programs may be
created by sampling the grammar and, through setting proba-
bilities on the grammar productions, the sampling is biased
towards producing programs from one part of the space or an-
other. This technique is potentially completely general, since
a grammar can theoretically be constructed to match any de-
sired program domain. However, despite being theoretically
possible, it is not easy to construct grammars which are both
suitably general and also produce programs that are in any
way similar to human written programs. It has been shown to
be successful over a highly restricted space of stencil bench-
marks with little control ﬂow or program variability [4, 8].
But, it is not clear how much effort it will take, or even if it
is possible for human experts to deﬁne grammars capable of
producing human like programs in more complex domains.
By contrast, our approach does not require an expert to
deﬁne what human programs look like. Instead, we automat-
ically infer the structure and likelihood of programs over a
huge corpus of open source projects. From this corpus, we
learn a probability distribution over sets of characters seen
in human written code. Later, we sample from this distribu-
tion to generate new random programs which, because the
distribution models human written code, are indistinguish-
able from human code. We can then populate our training
data with an unbounded number of human like programs,
covering the space far more ﬁnely than either existing bench-
mark suites or even the corpus of open source projects. Our
approach is enabled by two recent developments:
The ﬁrst is the breakthrough effectiveness of deep learning
for modeling complex structure in natural languages [12, 13].
As we show, deep learning is capable not just of learning the
macro syntactical and semantic structure of programs, but
also the nuances of how humans typically write code. It is
truly remarkable when one considers that it is given no prior
knowledge of the syntax or semantics of the language.
The second is the increasing popularity of public and open
platforms for hosting software projects and source code. This
popularity furnishes us with the thousands of programming
examples that are necessary to feed into the deep learning.
These open source examples are not, sadly, as useful for
directly learning the compiler heuristics since they are not
presented in a uniform, runnable manner, nor do they typically
have extractable test data. Preparing each of the thousands
of open source projects to be directly applicable for learning
compiler heuristics would be an insurmountable task. In
addition to our program generator, CLgen, we also provide
an accompanying host driver which generates datasets for,
then executes and proﬁles synthesized programs.
We make the following contributions:
• We are the ﬁrst to apply deep learning over source codes
to synthesize compilable, executable benchmarks.
• A novel tool CLgen 1 for general-purpose benchmark
synthesis using deep learning. CLgen automatically and
rapidly generates thousands of human like programs for
use in predictive modeling.
• We use CLgen to automatically improve the performance
of a state of the art predictive model by1.27×, and expose
limitations in the feature design of the model which, after
correcting, further increases performance by4.30×.
1https://github.com/ChrisCummins/clgen
Rodinia
NVIDIA SDK
AMD SDK
Parboil
NAS
Polybench
SHOC
Ad-hoc
ISPASS
Ploybench
Lonestar
SPEC-Viewperf
MARS
GPGPUsim
0
1
2
3
4
5
6
7
#. benchmarks used
Figure 2. The average number of benchmarks used in
GPGPU research papers, organized by origin. In this work
we use the seven most popular benchmark suites.
2. Motivation
In this section we make the argument for synthetic bench-
marks. We identiﬁed frequently used benchmark suites in a
survey of 25 research papers in the ﬁeld of GPGPU perfor-
mance tuning from four top tier conferences between 2013–
2016: CGO, HiPC, P ACT, and PPoPP . We found the average
number of benchmarks used in each paper to be 17, and that
a small pool of benchmarks suites account for the majority of
results, shown in Figure 2. We selected the 7 most frequently
used benchmark suites (accounting for 92% of results), and
evaluated the performance of the state of the artGrewe et
al. [14] predictive model across each. The model predicts
whether running a given OpenCL kernel on the GPU gives
better performance than on the CPU. We describe the full
experimental methodology in Section 7.
Table 1 summarizes our results. The performance of a
model trained on one benchmark suite and used to predict the
mapping for another suite is generally very poor. The bench-
mark suite which provides the best results, NVIDIA SDK,
achieves on average only 49% of the optimal performance.
The worst case is when training with Parboil to predict the
optimal mappings for Polybench, where the model achieves
only 11.5% of the optimal performance. From this it is clear
that heuristics learned on one benchmark suite fail to general-
ize across other suites.
This problem is caused both by the limited number of
benchmarks contained in each suite, and the distribution of
benchmarks within the feature space. Figure 3 shows the fea-
ture space of the Parboil benchmark suite, showing whether,
for each benchmark, the model was able to correctly predict
the appropriate optimization. We used Principle Component
Analysis to reduce the multi-dimensional feature space to aid
visualization.
As we see in Figure 3a, there is a dense cluster of neigh-
boring benchmarks, a smaller cluster of three benchmarks,
and two outliers. The lack of neighboring observations means
that the model is unable to learn a good heuristic for the two
outliers, which leads to them being incorrectly optimized. In
AMD NPB NVIDIA Parboil Polybench Rodinia SHOC
AMD - 38.0% 74.5% 76.7% 21.7% 45.8% 35.9%
NPB 22.7% - 45.3% 36.7% 13.4% 16.1% 23.7%
NVIDIA 29.9% 37.9% - 21.8% 78.3% 18.1% 63.2%
Parboil 89.2% 28.2% 28.2% - 41.3% 73.0% 33.8%
Polybench 58.6% 30.8% 45.3% 11.5% - 43.9% 12.1%
Rodinia 39.8% 36.4% 29.7% 36.5% 46.1% - 59.9%
SHOC 42.9% 71.5% 74.1% 41.4% 35.7% 81.0% -
Table 1. Performance relative to the optimal of theGrewe et al.predictive model across different benchmark suites on an AMD
GPU. The columns show the suite used for training; the rows show the suite used for testing.
Principle Component 1 
Principle Component 2  Correct
Incorrect
(a)
Principle Component 1 
Principle Component 2  Correct
Incorrect
Additional
(b)
Figure 3. A two dimensional projection of theGrewe et al.
feature space, showing predictive model results over Parboil
benchmarks on an NVIDIA GPU. Two outliers in (a) are
incorrectly predicted due to the lack of nearby observations.
The addition of neighboring observations in (b) corrects this.
Figure 3b, we hand-selected benchmarks which are neigh-
bouring in the feature space and retrained the model. The
addition of these observations (and the information they pro-
vide about that part of the feature space) causes the two
outliers to be correctly optimized. We found such outliers in
all of the benchmark suites of Table 1.
These results highlight the signiﬁcant effect that the num-
ber and distribution of training programs has on the quality
of predictive models. Without good coverage of the feature
space, any machine learning methodology is unlikely to pro-
duce high quality heuristics, suitable for general use on ar-
bitrary real applications, or even applications from different
benchmark suites. Our novel approach, described in the next
section, solves this problem by generating an unbounded
number of programs to cover the feature space with ﬁne
granularity.
3. Overview of Our Approach
In this paper we present CLgen, a tool for synthesizing
OpenCL benchmarks, and an accompanying host driver for
executing synthetic benchmarks for gathering performance
data for predictive modeling. While we demonstrate our
approach using OpenCL, it is language agnostic. Our tool
CLgen learns the semantics and structure from over a million
lines of hand-written code from GitHub, and synthesizes
programs through a process of iterative model sampling.
CLgen
Host Driver
Language 
Corpus
GitHub
Software
Repositories
clsmithclsmithContent Files Rejection 
Filter
Search 
engine
Code 
Rewriter
Model 
parameters
Rejection 
Filter
LSTM network
Synthesizer
Synthesis 
parameters
Argument 
Extractor
Benchmark 
parameters
clsmithclsmithSynthesized 
Benchmarks
Benchmark 
Driver
clsmithclsmithSynthesized 
Payloads
clsmithclsmithPerformance 
Results
Dynamic 
Checker
Figure 4. Benchmark synthesis and execution pipeline.
We use a host driver to execute the synthesized programs
to gather performance data for use in predictive modeling.
Figure 4 provides an overview of the program synthesis and
execution pipeline. Our approach extends the state of the
art by providing a general-purpose solution for benchmark
synthesis, leading to better and more accurate predictive
models.
In the course of evaluating our technique against prior
work we discovered that it is also useful for evaluating the
quality of features. Since we are able to cover the space
so much more ﬁnely than the prior work, which only used
standard benchmark suites, we are able to ﬁnd multiple
programs with identical feature values but different best
heuristic values. This indicates that the features are not
sufﬁciently discriminative and should be extended with more
information to allow those programs to be separated. We
go on to show that doing this signiﬁcantly increases the
performance of the learned heuristics. We expect that our
technique will be valuable for feature designers.
4. CLgen: Benchmark Synthesis
CLgen is an undirected, general-purpose program synthesizer
for OpenCL. It adopts and augments recent advanced tech-
niques from deep learning to learn over massive codebases. In
contrast to existing grammar and template based approaches,
CLgen is entirely probabilistic. The systemlearns to program
using neural networks which model the semantics and usage
of a huge corpus of code fragments in the target programming
language. This section describes the assembly of an OpenCL
language corpus, the application of deep learning over this
corpus, and the process of synthesizing programs.
4.1 An OpenCL Language Corpus
Deep learning requires large datasets [15]. For the purpose of
modeling a programming language, this means assembling a
very large collection of real, hand-written source codes. We
assembled OpenCL codes by mining public repositories on
the popular code hosting site GitHub.
This is itself a challenging task since OpenCL is an
embedded language, meaning device code is often difﬁcult
to untangle since GitHub does not presently recognize it as
a searchable programming language. We developed a search
engine which attempts to identify and download standalone
OpenCL ﬁles through a process of ﬁle scraping and recursive
header inlining. The result is a 2.8 million line dataset of
8078 “content ﬁles” which potentially contain OpenCL code,
originating from 793 GitHub repositories.
We prune the raw dataset extracted from GitHub using
a custom toolchain we developed for rejection ﬁltering and
code rewriting, built on LL VM.
Rejection Filter The rejection ﬁlter accepts as input a
content ﬁle and returns whether or not it contains compilable,
executable OpenCL code. To do this we attempt to compile
the input to NVIDIA PTX bytecode and perform static
analysis to ensure a minimum static instruction count of three.
We discard any inputs which do not compile or contain fewer
than three instructions.
During initial development it became apparent that isolat-
ing the OpenCL device code leads to a higher-than-expected
discard rate (that is, seemingly valid OpenCL ﬁles being
rejected). Through analyzing 148k lines of compilation er-
rors, we discovered a large number of failures caused by
undeclared identiﬁers — a result of isolating device code
— 50% of undeclared identiﬁer errors in the GitHub dataset
were caused by only 60 unique identiﬁers. To address this,
we developed ashim header which contains inferred values
for common type deﬁnitions (e.g.FLOAT_T), and common
constants (e.g. WGSIZE), shown in Listing 1.
Injecting the shim decreases the discard rate from 40%
to 32%, responsible for an additional 88k lines of code in
the ﬁnal language corpus. The resulting dataset is 2.0 million
lines of compilable OpenCL source code.
1 / * Enable OpenCL f e a t u r e s * /
2 #define cl_clang_storage_class_specifiers
3 #define cl_khr_fp64
4 #include <clc / clc .h>
5
6 / * Inferred types * /
7 typedef float FLOAT_T ;
8 typedef un signed int INDEX_TYPE ;
. . . (36 more )
9
10 / * Inferred constants * /
11 #define M _ P I 3.14025
12 #define WG_SIZE 128
. . . (185 more )
Listing 1. The shim header ﬁle, providing inferred type
aliases and constants for OpenCL on GitHub.
Code Rewriter Programming languages have few of the
issues of semantic interpretation present in natural language,
though there remains many sources of variance at the syntac-
tic level. For example, the presence and content of comments
in code, and the choice of identifying names given to vari-
ables. We consider these ambiguities to benon-functional
variance, and developed a tool to normalize code of these
variances so as to make the code more amenable to machine
learning. This is a three step process:
1. The source is pre-processed to remove macros, conditional
compilation, and source comments.
2. Identiﬁers are rewritten to have a short but unique name
based on their order of appearance, using the sequential
series {a ,b ,c ,...,a a ,a b ,a c ,...} for variables and {A,
B,C ,...,A A ,A B,A C ,...} for functions. This process
isolates the syntactic structure of the code, and unlike prior
work [16], our rewrite method preserves program behavior.
Language built-ins (e.g.get_global_id, asin) are not
rewritten.
3. A variant of the Google C++ code style is enforced to
ensure consistent use of braces, parentheses, and white
space.
An example of the code rewriting process is shown in Figure 5.
A side effect of this process is a reduction in code size,
largely due to the removal of comments and excess white
space. The ﬁnal language corpus contains 1.3 million lines
of transformed OpenCL, consisting of 9487 kernel functions.
Identiﬁer rewriting reduces the bag-of-words vocabulary size
by 84%.
4.2 Learning OpenCL
Generating valid, executable program code is an ambitious
and challenging goal for unsupervised machine learning. We
employ state of the art deep language modeling techniques to
achieve this task.
We use the Long Short-Term Memory (LSTM) architec-
ture of Recurrent Neural Network [17, 18] to learn a character-
level language model over the corpus of OpenCL compute
kernels. The LSTM network architecture comprises recurrent
1 #define DTYPE float
2 #define ALPHA ( a ) 3 . 5 f* a
3 inline DTYPE a x ( DTYPE x ) { return ALPHA ( x ) ; }
4
5 __kernel void saxpy ( / * S A X P Y kernel * /
6 _ _ g l o b a l DTYPE * input1 ,
7 _ _ g l o b a l DTYPE * input2 ,
8 const int nelem )
9{
10 unsigned int idx = get_global_id (0) ;
11 / / =a x+y
12 if ( idx < nelem ) {
1 3 input2 [ idx ] + = ax( input1 [ idx ]) ; }}
(a) Example content ﬁle
1 inline float A( float a) {
2 return 3.5 f * a;
3}
4
5 __kernel void B( __global float * b , __global float * c,
↪→ const int d) {
6 unsigned int e = get_global_id (0) ;
7
8 if (e < d) {
9 c[e] + = A (b[e]) ;
10 }
11 }
(b) Content ﬁle after code rewriting
Figure 5. The code rewriting process, which transforms code
to make it more amenable to language modeling.
layers of memory cells, each consisting of an input, output,
and forget gate, and an output layer providing normalized
probability values from a 1-of-K coded vocabulary [19].
We use a 3-layer LSTM network with 2048 nodes per layer,
implemented in Torch. We train this 17-million parameter
model using Stochastic Gradient Descent for 50 epochs,
using an initial learning rate of 0.002, decaying by a factor
of one half every 5 epochs. Training took three weeks on a
single machine using an NVIDIA GTX Titan, with a ﬁnal
model size of 648MB. Training the network is a one-off cost,
and can be parallelized across devices. The trained network
can be deployed to lower-compute machines for use.
4.3 Synthesizing OpenCL
We synthesize OpenCL compute kernels by iteratively sam-
pling the learned language model. We implemented two
modes for model sampling: the ﬁrst involves providing anar-
gument speciﬁcation, stating the data types and modiﬁers of
all kernel arguments. When an argument speciﬁcation is pro-
vided, the model synthesizes kernels matching this signature.
In the second sampling mode this argument speciﬁcation is
omitted, allowing the model to synthesize compute kernels of
arbitrary signatures, dictated by the distribution of argument
types within the language corpus.
In either mode we generate aseed text, and sample the
model, character by character, until the end of the compute
kernel is reached, or until a predetermined maximum number
of characters is reached. Algorithm 1 illustrates this process.
Algorithm 1Sampling a candidate kernel from a seed text.
Require: LSTM model M, maximum kernel lengthn.
Ensure: Completed sample stringS.
1: S ←“__kernel void A(const int a) {” Seed text
2: d ← 1 Initial code block depth
3: for i ←| S| to n do
4: c ← predictcharacter(M, S) Generate new character
5: if c =“{” then
6: d ← d +1 Entered code block, increase depth
7: else ifc =“}” then
8: d ← d −1 Exited code block, decrease depth
9: end if
10: S ← S + c Append new character
11: if depth =0 then
12: break Exited function block, stop sampling
13: end if
14: end for
The same rejection ﬁlter described in Section 4.1 then either
accepts or rejects the sample as a candidate synthetic bench-
mark. Listing 6 shows three examples of unique compute
kernels generated in this manner from an argument speciﬁ-
cation of three single-precision ﬂoating-point arrays and a
read-only signed integer. We evaluate the quality of synthe-
sized code in Section 6.
5. Benchmark Execution
We developed a host driver to gather performance data from
synthesized CLgen code. The driver accepts as input an
OpenCL kernel, generates payloads of user-conﬁgurable
sizes, and executes the kernel using the generated payloads,
providing dynamic checking of kernel behavior.
5.1 Generating Payloads
A payload encapsulates all of the arguments of an OpenCL
compute kernel. After parsing the input kernel to derive
argument types, a rule-based approach is used to generate
synthetic payloads. For a given global sizeSg: host buffers
of Sg elements are allocated and populated with random
values for global pointer arguments, device-only buffers
of Sg elements are allocated for local pointer arguments,
integral arguments are given the valueSg, and all other scalar
arguments are given random values. Host to device data
transfers are enqueued for all non-write-only global buffers,
and all non-read-only global buffers are transferred back to
the host after kernel execution.
5.2 Dynamic Checker
For the purpose of performance benchmarking we are not
interested in the correctness of computed values, but we
deﬁne a class of programs as performinguseful work if they
predictably compute some result. We devised a low-overhead
runtime behavior check to validate that a synthesized program
does useful work based on the outcome of four executions of
a tested program:
1 __kernel void A( __global float * a,
2 __global float * b,
3 __global float * c,
4 const int d) {
5 int e = get_global_id (0) ;
6 float f = 0.0;
7 for ( int g= 0 ; g<d ; g + + ){
8c [ g ] = 0 . 0 f ;
9}
1 0 barrier (1) ;
11
1 2 a[ get_global_id (0) ] = 2 * b[ get_global_id (0) ];
13 }
(a) V ector operation with branching and synchronization.
1 __kernel void A( __global float * a,
2 __global float * b,
3 __global float * c,
4 const int d) {
5 int e = get_global_id (0) ;
6 if (e > = d) {
7 return ;
8}
9 c[e] = a[e] + b[e] + 2 * a[e] + b[e] + 4;
10 }
(b) Zip operation which computesci =3 ai +2 bi +4 .
1 __kernel void A( __global float * a,
2 __global float * b,
3 __global float * c,
4 const int d) {
5 unsigned int e = get_global_id (0) ;
6 float16 f=( float16 ) (0.0) ;
7 for ( unsigned int g= 0 ; g<d ; g + + ){
8 float16 h=a [ g ] ;
9 f.s 0 + = h.s 0;
1 0 f . s1 += h . s1 ;
1 1 f . s2 += h . s2 ;
1 2 f . s3 += h . s3 ;
1 3 f . s4 += h . s4 ;
1 4 f . s5 += h . s5 ;
1 5 f . s6 += h . s6 ;
1 6 f . s7 += h . s7 ;
1 7 f . s8 += h . s8 ;
1 8 f . s9 += h . s9 ;
19 f . sA += h . sA ;
20 f . sB += h . sB ;
21 f . sC += h . sC ;
22 f . sD += h . sD ;
23 f . sE += h . sE ;
24 f . sF += h . sF ;
25 }
2 6 b [ e ]=f . s 0+f . s 1+f . s 2+f . s 3+f . s 4+ f . s 5+
↪→ f . s 6+f . s 7+f . s 8+f . s 9+ f . s A +f . s B+
↪→ f.s C + f. s D + f.s E + f.s F;
27 }
(c) Partial reduction over reinterpreted vector type.
Figure 6. Compute kernels synthesized with CLgen. All
three kernel were synthesized from the same argument spec-
iﬁcation: three single-precision ﬂoating-point arrays and a
read-only signed integer.
1. Create 4 equal size payloads A1in, B1in, A2in, B2in,
subject to restrictions: A1in = A2in, B1in = B2in,
A1in ̸= B1in.
2. Execute kernel k 4 times: k(A1in) → A1out, k(B1in) →
B1out, k(A2in) → A2out, k(B2in) → B2out.
3. Assert:
• A1out ̸= A1in and B1out ̸= B1in, elsek has no output
(for these inputs).
• A1out ̸= B1out and A2out ̸= B2out, else k is input
insensitive t (for these inputs).
• A1out = A2out and B1out = B2out, else k is non-
deterministic.
Equality checks for ﬂoating point values are performed with
an appropriate epsilon to accommodate rounding errors, and
a timeout threshold is also used to catch kernels which are
non-terminating. Our method is based on random differential
testing [20], though we emphasize that this is not a general
purpose approach and is tailored speciﬁcally for our use
case. For example, we anticipate a false positive rate for
kernels with subtle sources of non-determinism which more
thorough methods may expose [21–23], however we deemed
such methods unnecessary for our purpose of performance
modeling.
6. Evaluation of Synthetic Programs
In this section we evaluate the quality of programs synthe-
sized by CLgen by their likeness to hand-written code, and
discuss limitations of the synthesis and execution pipeline.
6.1 Likeness to Hand-written Code
Judging whether a piece of code has been written by a
human is a challenging task for a machine, so we adopt a
methodology from machine learning research based on the
Turing Test[24–26]. We reason that if the output of CLgen
is human like code, then a human judge will be unable to
distinguish it from hand-written code.
We devised a double blind test in which 15 volunteer
OpenCL developers from industry and academia were shown
10 OpenCL kernels each. Participants were tasked with
judging whether, for each kernel, they believed it to have
been written by hand or by machine. Kernels were randomly
selected for each participant from two equal sized pools of
synthetically generated and hand-written code from GitHub.
We applied the code rewriting process to all kernels to
remove comments and ensure uniform identiﬁer naming. The
participants were divided into two groups, with 10 of them
receiving code generated by CLgen, and 5 of them acting as
a control group, receiving code generated by CLSmith [27],
a program generator for differential testing1.
We scored each participant’s answers, ﬁnding the average
score of the control group to be 96% (stdev. 9%), an unsurpris-
1An online version of this test is available athttp://humanorrobot.uk/.
Raw Code Features
comp static #. compute operations
mem static #. accesses to global memory
localmem static #. accesses to local memory
coalesced static #. coalesced memory accesses
transfer dynamic size of data transfers
wgsize dynamic #. work-items per kernel
(a) Individual code features
Combined Code Features
F1: transfer/(comp+mem) commun.-computation ratio
F2: coalesced/mem % coalesced memory accesses
F3: (localmem/mem)×wgsize ratio local to global mem accesses
× #. work-items
F4: comp/mem computation-mem ratio
(b) Combinations of raw features
Table 2. Grewe et al.model features.
ing outcome as generated programs for testing have multiple
“tells”, for example, their only input is a singleulong pointer.
There were no false positives (synthetic code labeled human)
for CLSmith, only false negatives (human code labeled syn-
thetic). With CLgen synthesized programs, the average score
was 52% (stdev. 17%), and the ratio of errors was even. This
suggests that CLgen code is indistinguishable from hand-
written programs, with human judges scoring no better than
random chance.
6.2 Limitations
Our new approach enables the synthesis of more human-like
programs than current state of the art program generators,
and without the expert guidance required by template based
generators, but it has limitations. Our method of seeding the
language models with the start of a function means that we
cannot support user deﬁned types, or calls to user-deﬁned
functions. This means that we only consider scalars and ar-
rays as inputs; while 6 (2.3%) of the benchmark kernels from
Table 3 use irregular data types as inputs. We will address
this limitation through recursive program synthesis, whereby
a call to a user-deﬁned function or unrecognized type will
trigger candidate functions and type deﬁnitions to be synthe-
sized. Currently we only run single-kernel benchmarks. We
will extend the host driver to explore multi-kernel schedules
and interleaving of kernel executions. Our host driver gener-
ates datasets from uniform random distributions, as do many
of the benchmark suites. For cases where non-uniform in-
puts are required (e.g. proﬁle-directed feedback), an alternate
methodology for generating inputs must be adopted.
7. Experimental Methodology
7.1 Experimental Setup
Predictive Model We reproduce the predictive model from
Grewe, Wang, and O’Boyle [14]. The predictive model is
used to determine the optimal mapping of a given OpenCL
kernel to either a GPU or CPU. It uses supervised learning
to construct a decision tree with a combination of static and
Version #. benchmarks #. kernels
NPB (SNU [29]) 1.0.3 7 114
Rodinia [30] 3.1 14 31
NVIDIA SDK 4.2 6 12
AMD SDK 3.0 12 16
Parboil [31] 0.2 6 8
PolyBench [32] 1.0 14 27
SHOC [33] 1.1.5 12 48
Total - 71 256
Table 3. List of benchmarks.
Intel CPU AMD GPU NVIDIA GPU
Model Core i7-3820 Tahiti 7970 GTX 970
Frequency 3.6 GHz 1000 MHz 1050 MHz
#. Cores 4 2048 1664
Memory 8G B 3G B 4G B
Throughput 105 GFLOPS 3.79 TFLOPS 3.90 TFLOPS
Driver AMD 1526.3 AMD 1526.3 NVIDIA 361.42
Compiler GCC 4.7.2 GCC 4.7.2 GCC 5.4.0
Table 4. Experimental platforms.
dynamic kernel features extracted from source code and the
OpenCL runtime, detailed in Table 2b.
Benchmarks As in [14], we test our model on the NAS
Parallel Benchmarks (NPB) [28]. We use the hand-optimized
OpenCL implementation of Seo, Jo, and Lee [29]. In [14]
the authors augment the training set of the predictive model
with 47 additional kernels taken from 4 GPGPU benchmark
suites. To more fully sample the program space, we use
a much larger collection of 142 programs, summarized in
Table 3. These additional programs are taken from all 7
of the most frequently used benchmark suites identiﬁed in
Section 2. None of these programs were used to train CLgen.
We synthesized 1,000 kernels with CLgen to use as additional
benchmarks.
Platforms We evaluate our approach on two 64-bit CPU-
GPU systems, detailed in Table 4. One system has an AMD
GPU and uses OpenSUSE 12.3; the other is equipped with an
NVIDIA GPU and uses Ubuntu 16.04. Both platforms were
unloaded.
Datasets
The NPB and Parboil benchmark suites are pack-
aged with multiple datasets. We use all of the packaged
datasets (5 per program in NPB, 1-4 per program in Par-
boil). For all other benchmarks, the default datasets are used.
We conﬁgured the CLgen host driver to synthesize payloads
between 128B-130MB, approximating that of the dataset
sizes found in the benchmark programs.
7.2 Methodology
We replicated the methodology of [14]. Each experiment is
repeated ﬁve times and the average execution time is recorded.
The execution time includes both device compute time and
the data transfer overheads.
We useleave-one-out cross-validationto evaluate predic-
tive models. For each benchmark, a model is trained on data
from all other benchmarks and used to predict the mapping
1 __kernel void A( __global float * a,
2 __global float * b,
3 __global float * c,
4 const int d) {
5 int e = get_global_id (0) ;
6 if (e < 4 & & e < c) {
7 c[e] = a[e] + b[e];
8a [ e ] = b [ e ] + 1 ;
9}
10 }
Listing 2. In the Grewe et al. feature space this
CLgen program is indistinguishable from AMD’s Fast
Walsh–Hadamard transform benchmark, but has very
different runtime behavior and optimal device mapping. The
addition of a branching feature ﬁxes this.
for each kernel and dataset in the excluded program. We re-
peat this process with and without the addition of synthetic
benchmarks in the training data. We do not test model predic-
tions on synthetic benchmarks.
8. Experimental Results
We evaluate the effectiveness of our approach on two hetero-
geneous systems. We ﬁrst compare the performance of a state
of the art predictive model [14] with and without the addi-
tion of synthetic benchmarks, then show how the synthetic
benchmarks expose weaknesses in the feature design and how
these can be addressed to develop a better model. Finally we
compare the ability of CLgen to explore the program feature
space against a state of the art program generator [27].
8.1 Performance Evaluation
Figure 7 shows speedups of theGrewe et al.predictive model
over the NAS Parallel Benchmark suite with and without the
addition of synthesized benchmarks for training. Speedups
are calculated relative to the best single-device mapping for
each experimental platform, which is CPU-only for AMD
and GPU-only for NVIDIA. The ﬁne grained coverage of the
feature space which synthetic benchmarks provide improves
performance dramatically for the NAS benchmarks. Across
both systems, we achieve an average speedup of 2.42×
with the addition of synthetic benchmarks, with prediction
improvements over the baseline for 62.5% of benchmarks on
AMD and 53.1% on NVIDIA.
The strongest performance improvements are on NVIDIA
with the FT benchmark which suffers greatly under a single-
device mapping. However, the performance on AMD for the
same benchmark slightly degrades after adding the synthetic
benchmarks, which we address in the next section.
8.2 Extending the Predictive Model
Feature designers are bound to select as features only prop-
erties which are signiﬁcant for the sparse benchmarks they
test on, which can limit a model’s ability to generalize over a
wider range of programs. We found this to be the case with
the Grewe et al.model. The addition of automatically gener-
ated programs exposed two distinct cases where the model
failed to generalize as a result of overspecializing to the NPB
suite.
The ﬁrst case is thatF3 is sparse on many programs. This is
a result of the NPB implementation’s heavy exploitation of lo-
cal memory buffers and the method by which they combined
features (we speculate this was a necessary dimensionality
reduction in the presence of sparse training programs). To
counter this we extended the model to use the raw feature
values in addition to the combined features.
The second case is that some of our generated programs
had identical feature values as in the benchmark set, but had
different behavior (i.e. optimal mappings). Listing 2 shows
one example of a CLgen benchmark which is indistinguish-
able in the feature space to one the of existing benchmarks
— the Fast Walsh-Hadamard transform — but with different
behavior. We found this to be caused by the lack of dis-
criminatory features for branching, since the NPB programs
are implemented in a manner which aggressively minimized
branching. To counter this we extended the predictive model
with an additional feature containing a static count of branch-
ing operations in a kernel.
Figure 8 shows speedups of our extended model across
all seven of the benchmark suites used in Section 2. Model
performance, even on this tenfold increase of benchmarks, is
good. There are three benchmarks on which the model per-
forms poorly:
MatrixMul, cutcp, and pathfinder. Each
of those programs make heavy use of loops, which we be-
lieve the static code features of the model fail to capture. This
could be addressed by extracting dynamic instruction counts
using proﬁling, but we considered this beyond the scope of
our work. It is not our goal to perfect the predictive model,
but to show the performance improvements associated with
training on synthetic programs. To this extent, we are suc-
cessful, achieving average speedups of3.56×on AMD and
5.04× on NVIDIA across a very large test set.
8.3 Comparison of Source Features
As demonstrated in Section 2, the predictive quality of a
model for a given point in the feature space is improved with
the addition of observations from neighboring points. By
producing thousands of artiﬁcial programs modeled on the
structure real OpenCL programs, CLgen is able to consis-
tently and automatically generate programs which are close
in the feature space to the benchmarks which we are testing
on.
To quantify this effect we use the static code features
of Table 2a, plus the branching feature discussed in the
previous subsection, to measure the number of CLgen kernels
generated with the same feature values as those of the
benchmarks we examined in the previous subsections. We
examine only static code features to allow comparison with
the GitHub kernels for which we have no automated method
to execute them and extract runtime features, and CLSmith
generated programs.
BT.A
BT.B
BT.S
BT.W
CG.A
CG.B
CG.C
CG.S
CG.W
EP.A
EP.B
EP.C
EP.W
FT.A
FT.B
FT.S
FT.W
LU.A
LU.B
LU.C
LU.S
LU.W
MG.A
MG.B
MG.C
MG.S
MG.W
SP.A
SP.B
SP.C
SP.S
SP.W
Average
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
Speedup over CPU
Grewe et al.
w. CLgen
(a) AMD Tahiti 7970
BT.A
BT.B
BT.S
BT.W
CG.A
CG.B
CG.C
CG.S
CG.W
EP.A
EP.B
EP.C
EP.W
FT.A
FT.B
FT.S
FT.W
LU.A
LU.B
LU.C
LU.S
LU.W
MG.A
MG.B
MG.C
MG.S
MG.W
SP.A
SP.B
SP.C
SP.S
SP.W
Average
1
3
5
7
9
11
13
15
17
Speedup over GPU
Grewe et al.
w. CLgen
(b) NVIDIA GTX 970
Figure 7. Speedup of programs usingGrewe et al.predictive model with and without synthetic benchmarks. The predictive
model outperforms the best static device mapping by a factor of1.26× on AMD and 2.50× on NVIDIA. The addition of
synthetic benchmarks improves the performance to1.57× on AMD and3.26× on NVIDIA.
Figure 9 plots the number of matches as a function of
the number of kernels. Out of 10,000 unique CLgen kernels,
more than a third have static feature values matching those
of the benchmarks, providing on average 14 CLgen kernels
for each benchmark. This conﬁrms our original intuition:
CLgen kernels, by emulating the way real humans write
OpenCL programs, are concentrated in the same area of the
feature space as real programs. Moreover, the number of
CLgen kernels we generate is unbounded, allowing us to
continually reﬁne the exploration of the feature space, while
the number of kernels available on GitHub is ﬁnite. CLSmith
rarely produces code similar to real-world OpenCL programs,
with only 0.53% of the generated kernels have matching
feature values with benchmark kernels. We conclude that the
unique contribution of CLgen is its ability to generate many
thousands of programs that are appropriate for predictive
modeling.
9. Related Work
Our work lies at the intersections of a number of areas: pro-
gram generation, benchmark characterization, and language
modeling and learning from source code. There is no existing
work which is similar to ours, in respect to learning from
large corpuses of source code for benchmark generation.
GENESIS [34] is a language for generating synthetic train-
ing programs. Users annotate template programs with sta-
tistical distributions over features, which are instantiated to
generate statistically controlled permutations of templates.
Template based approaches provide domain-speciﬁc solutions
for a constrained feature and program space, for example, gen-
erating permutations of Stencil codes [35, 36]. Our approach
provides general-purpose program generation over unknown
domains, in which the statistical distribution of generated
programs is automatically inferred from real world code.
Random program generation is an effective method for
software testing. Grammar-basedfuzz testershave been de-
veloped for C [37] and OpenCL [27]. A mutation-based ap-
proach for the Java Virtual Machine is demonstrated in [38].
Goal-directed program generators have been used for a vari-
ety of domains, including generating linear transforms [39],
MapReduce programs [40], and data structure implementa-
tions [41]. Program synthesis from input/output examples
is used for simple algorithms in [42], string manipulation
in [43], and geometry constructions in [44].
Machine learning has been applied to source code to aid
software engineering. Naturalize employs techniques devel-
oped in the natural language processing domain to model cod-
ing conventions [45]. JSNice leverages probabilistic graphical
models to predict program properties such as identiﬁer names
for Javascript [46].
Figure 8. Speedups of predictions using our extended model overGrewe et al.on both experimental platforms. Synthetic
benchmarks and the additional program features outperform the original predictive model by a factor3.56×on AMD and5.04×
on NVIDIA.
0 2000 4000 6000 8000 10000
#. kernels
0
500
1000
1500
2000
2500
3000
3500
#. matches
GitHub
CLSmith
CLgen
Figure 9. The number of kernels from GitHub, CLSmith,
and CLgen with static code features matching the bench-
marks. CLgen generates kernels that are closer in the feature
space than CLSmith, and can continue to do so long after we
have exhausted the extent of the GitHub dataset. Error bars
show standard deviation of 10 random samplings.
There is an increasing interest in mining source code repos-
itories at large scale [16, 47, 48]. Previous studies have in-
volved data mining of GitHub to analyze software engineer-
ing practices [49–52], for example code generation [53], code
summarization [54], comment generation [55], and code com-
pletion [56]. However, no work so far has exploited mined
source code for benchmark generation. This work is the ﬁrst
to do so.
10. Conclusion
The quality of predictive models is bound by the quantity and
quality of programs used for training, yet there is typically
only a few dozen common benchmarks available for experi-
ments. We present a novel tool which is the ﬁrst of it’s kind
— an entirely probabilistic program generator capable of gen-
erating an unbounded number of human like programs. Our
approach applies deep learning over a huge corpus of publicly
available code from GitHub to automatically infer the seman-
tics and practical usage of a programming language. Our tool
generates programs which to trained eyes are indistinguish-
able from hand-written code. We tested our approach using a
state of the art predictive model, improving its performance
by a factor of1.27×. We found that synthetic benchmarks
exposed weaknesses in the feature set which, when corrected,
further improved the performance by4.30×. Our hope for
this work is to demonstrate a proof of concept for an exciting
new avenue of program generation, and that the full release
of CLgen will expedite discovery in other domains. In future
work we will extend the approach to multiple programming
languages, and investigate methods for performing an auto-
matic directed search of feature spaces.
Acknowledgments
Our thanks to the volunteers at Codeplay Software Ltd and
the University of Edinburgh for participating in the qual-
itative evaluation. This work was supported by the UK
Engineering and Physical Sciences Research Council un-
der grants EP/L01503X/1 (CDT in Pervasive Parallelism),
EP/L000055/1 (ALEA), EP/M01567X/1 (SANDeRs), EP/M0
15823/1, and EP/M015793/1 (DIVIDEND). The code and
data for this paper are available athttp://chriscummins.
cc/cgo17.
References
[1] P . Micolet, A. Smith, and C. Dubach. “A Machine
Learning Approach to Mapping Streaming Workloads
to Dynamic Multicore Processors”. In:LCTES. 2016.
[2] Z. Wang, G. Tournavitis, B. Franke, and M. O’Boyle.
“Integrating Proﬁle-driven Parallelism Detection and
Machine-learning-based Mapping”. In:TACO (2014).
[3] A. Magni, C. Dubach, and M. O’Boyle. “Automatic
Optimization of Thread-Coarsening for Graphics Pro-
cessors”. In: PACT. ACM, 2014, pp. 455–466.
[4] C. Cummins, P . Petoumenos, M. Steuwer, and H.
Leather. “Towards Collaborative Performance Tuning
of Algorithmic Skeletons”. In:HLPGPU. 2016.
[5] Z. Wang and M. O’Boyle. “Mapping Parallelism to
Multi-cores: A Machine Learning Based Approach”.
In: PPoPP. 15. ACM, 2009, pp. 75–84.
[6] Y . Wen, Z. Wang, and M. O’Boyle. “Smart Multi-
Task Scheduling for OpenCL Programs on CPU/GPU
Heterogeneous Platforms”. In:HiPC. IEEE, 2014.
[7] Z. Wang and M. O’Boyle. “Partitioning Streaming
Parallelism for Multi-cores: A Machine Learning Based
Approach”. In: PACT. ACM, 2010, pp. 307–318.
[8] T. L. Falch and A. C. Elster. “Machine Learning Based
Auto-tuning for Enhanced OpenCL Performance Porta-
bility”. In: IPDPSW. IEEE, 2015.
[9] A. Collins, C. Fensch, and H. Leather. “Auto-Tuning
Parallel Skeletons”. In: Parallel Processing Letters
22.02 (June 2012), p. 1240005.
[10] H. Leather, E. Bonilla, and M. O’Boyle. “Automatic
Feature Generation for Machine Learning Based Opti-
mizing Compilation”. In:TACO 11 (2014).
[11] W . F. Ogilvie, P . Petoumenos, Z. Wang, and H. Leather.
“Fast Automatic Heuristic Construction Using Active
Learning”. In: LCPC. 2014.
[12] A. Graves. “Generating Sequences with Recurrent
Neural Networks”. In:arXiv:1308.0850 (2013).
[13] I. Sutskever, O. Vinyals, and Q. V . Le. “Sequence to
Sequence Learning with Neural Networks”. In:NIPS.
2014.
[14] D. Grewe, Z. Wang, and M. O’Boyle. “Portable Map-
ping of Data Parallel Programs to OpenCL for Hetero-
geneous Systems”. In:CGO. IEEE, 2013.
[15] Y . LeCun, Y . Bengio, and G. Hinton. “Deep learning”.
In: Nature 521.7553 (2015), pp. 436–444.
[16] M. Allamanis and C. Sutton. “Mining Source Code
Repositories at Massive Scale using Language Model-
ing”. In: MSR. 2013, pp. 207–216.
[17] M. Sundermeyer, R. Schl, and H. Ney. “LSTM Neural
Networks for Language Modeling”. In: Interspeech.
2012.
[18] T. Mikolov. “Recurrent Neural Network based Lan-
guage Model”. In:Interspeech. 2010.
[19] A. Graves and J. Schmidhuber. “Framewise Phoneme
Classiﬁcation with Bidirectional LSTM and Other
Neural Network Architectures”. In:Neural Networks
5.5 (18), pp. 602–610.
[20] W . M. McKeeman. “Differential Testing for Software”.
In: DTJ 10.1 (1998), pp. 100–107.
[21] A. Betts, N. Chong, and A. Donaldson. “GPUV erify: A
V eriﬁer for GPU Kernels”. In:OOPSLA. 2012, pp. 113–
131.
[22] J. Price and S. Mcintosh-Smith. “Oclgrind: An Exten-
sible OpenCL Device Simulator”. In:IWOCL. ACM,
2015.
[23] T. Sorensen and A. Donaldson. “Exposing Errors Re-
lated to Weak Memory in GPU Applications”. In:PLDI.
2016.
[24] H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and
W . Xu. “Are Y ou Talking to a Machine? Dataset and
Methods for Multilingual Image Question Answering”.
In: arXiv:1505.05612 (2015).
[25] R. Zhang, P . Isola, and A. A. Efros. “Colorful Image
Colorization”. In: arXiv:1603.08511 (2016).
[26] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. “Show
and Tell: A Neural Image Caption Generator”. In:
CVPR (2015).
[27] C. Lidbury, A. Lascu, N. Chong, and A. Donald-
son. “Many-Core Compiler Fuzzing”. In:PLDI. 2015,
pp. 65–76.
[28] D. H. Bailey, E. Barszcz, J. Barton, D. Browning, R.
Carter, L. Dagum, R. Fatoohi, S. Fineberg, P . Frederick-
son, T. Lasinski, R. Schreiber, H. Simon, V . V enkatakr-
ishnan, and S. Weeratunga. “The NAS Parallel Bench-
marks”. In: IJHPCA 5.3 (1991), pp. 63–73.
[29] S. Seo, G. Jo, and J. Lee. “Performance Characteriza-
tion of the NAS Parallel Benchmarks in OpenCL”. In:
IISWC. IEEE, 2011.
[30] S. Che, M. Boyer, J. Meng, D. Tarjan, J. W . Sheaffer,
S. H. Lee, and K. Skadron. “Rodinia: A Benchmark
Suite for Heterogeneous Computing”. In:IISWC. IEEE,
Oct. 2009.
[31] J. A. Stratton, C. Rodrigues, I. Sung, N. Obeid, L.
Chang, N. Anssari, G. D. Liu, and W . W . Hwu. “Par-
boil: A Revised Benchmark Suite for Scientiﬁc and
Commercial Throughput Computing”. In:Center for
Reliable and High-Performance Computing(2012).
[32] S. Grauer-Gray, L. Xu, R. Searles, S. A yalasomayajula,
and J. Cavazos. “Auto-tuning a High-Level Language
Targeted to GPU Codes”. In:InPar. 2012.
[33] A. Danalis, G. Marin, C. McCurdy, J. S. Meredith, P . C.
Roth, K. Spafford, V . Tipparaju, and J. S. V etter. “The
Scalable HeterOgeneous Computing (SHOC) Bench-
mark Suite”. In:GPGPU. ACM, 2010.
[34] A. Chiu, J. Garvey, and T. S. Abdelrahman. “Genesis: A
Language for Generating Synthetic Training Programs
for Machine Learning”. In:CF. ACM, 2015, p. 8.
[35] J. D. Garvey and T. S. Abdelrahman. “Automatic Per-
formance Tuning of Stencil Computations on GPUs”.
In: ICPP (2015).
[36] C. Cummins, P . Petoumenos, M. Steuwer, and H.
Leather. “Autotuning OpenCL Workgroup Size for
Stencil Patterns”. In:ADAPT. 2016.
[37] X. Y ang, Y . Chen, E. Eide, and J. Regehr. “Finding and
Understanding Bugs in C Compilers”. In:PLDI. 2011.
[38] Y . Chen, T. Su, C. Sun, Z. Su, and J. Zhao. “Coverage-
Directed Differential Testing of JVM Implementations”.
In: PLDI. 2016.
[39] Y . V oronenko, F. De Mesmay, and M. Püschel. “Com-
puter Generation of General Size Linear Transform
Libraries”. In: CGO. IEEE, 2009, pp. 102–113.
[40] C. Smith. “MapReduce Program Synthesis”. In:PLDI.
2016.
[41] C. Loncaric, T. Emina, and M. D. Ernst. “Fast Synthesis
of Fast Collections”. In:PLDI. 2016.
[42] W . Zaremba, T. Mikolov, A. Joulin, and R. Fergus.
“Learning Simple Algorithms from Examples”. In:
ICML. 2016.
[43] S. Gulwani. “Automating string processing in spread-
sheets using input-output examples”. In:POPL. 2011.
[44] S. Gulwani, V . A. Korthikanti, and A. Tiwari. “Synthe-
sizing geometry constructions”. In:PLDI. 2011.
[45] M. Allamanis, E. T. Barr, C. Bird, and C. Sutton.
“Learning Natural Coding Conventions”. In:FSE. 2014,
pp. 281–293.
[46] V eselin Raychev, Martin V echev, and Andreas Krause.
“Predicting Program Properties from “Big Code””. In:
POPL. 2015.
[47] M. White, C. V endome, M. Linares-V asquez, and D.
Poshyvanyk. “Toward Deep Learning Software Reposi-
tories”. In: MSR. 2015.
[48] E. Kalliamvakou, L. Singer, G. Gousios, D. M. German,
K. Blincoe, and D. Damian. “The Promises and Perils
of Mining GitHub”. In:MSR. 2009.
[49] Y . Wu, J. Kropczynski, P . C. Shih, and J. M. Car-
roll. “Exploring the Ecosystem of Software Develop-
ers on GitHub and Other Platforms”. In:CSCW. 2014,
pp. 265–268.
[50] E. Guzman, D. Azócar, and Y . Li. “Sentiment Analysis
of Commit Comments in GitHub: an Empirical Study”.
In: MSR. 2014, pp. 352–355.
[51] R. Baishakhi, D. Posnett, V . Filkov, and P . Devanbu.
“A Large Scale Study of Programming Languages and
Code Quality in Github”. In:FSE. 2014.
[52] B. V asilescu, V . Filkov, and A. Serebrenik. “Percep-
tions of Diversity on GitHub: A User Survey”. In:
Chase (2015).
[53] X. Gu, H. Zhang, D. Zhang, and S. Kim. “Deep API
Learning”. In: arXiv:1605.08535 (2016).
[54] M. Allamanis, H. Peng, and C. Sutton. “A Convolu-
tional Attention Network for Extreme Summarization
of Source Code”. In:arXiv:1602.03001 (2016).
[55] E. Wong, J. Y ang, and L. Tan. “AutoComment: Mining
Question and Answer Sites for Automatic Comment
Generation”. In: ASE. IEEE, 2013, pp. 562–567.
[56] V . Raychev, M. V echev, and E. Y ahav. “Code Com-
pletion with Statistical Language Models”. In:PLDI.
2014.

This paper is included in the Proceedings of the 
13th USENIX Symposium on Operating Systems Design 
and Implementation (OSDI ’18).
October 8–10, 2018 • Carlsbad, CA, USA
ISBN 978-1-939133-08-3
Open access to the Proceedings of the 
13th USENIX Symposium on Operating Systems 
Design and Implementation 
is sponsored by USENIX.
TVM: An Automated End-to-End Optimizing 
Compiler for Deep Learning
Tianqi Chen and Thierry Moreau, University of Washington; Ziheng Jiang, University of 
Washington, AWS; Lianmin Zheng, Shanghai Jiao Tong University; Eddie Yan, Haichen Shen, 
and Meghan Cowan, University of Washington; Leyuan Wang, UC Davis, AWS; Yuwei Hu, 
Cornell; Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy, University of Washington
https://www.usenix.org/conference/osdi18/presentation/chen
TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
Tianqi Chen1, Thierry Moreau 1, Ziheng Jiang 1,2, Lianmin Zheng 3, Eddie Yan 1
Meghan Cowan1, Haichen Shen 1, Leyuan Wang 4,2, Yuwei Hu 5, Luis Ceze1, Carlos Guestrin 1, Arvind Krishnamurthy 1
1Paul G. Allen School of Computer Science & Engineering, University of Washington
2 AWS, 3Shanghai Jiao Tong University,4UC Davis, 5Cornell
Abstract
There is an increasing need to bring machine learn-
ing to a wide diversity of hardware devices. Current
frameworks rely on vendor-speciﬁc operator libraries
and optimize for a narrow range of server-class GPUs.
Deploying workloads to new platforms – such as mo-
bile phones, embedded devices, and accelerators (e.g.,
FPGAs, ASICs) – requires signiﬁcant manual effort.
We propose TVM, a compiler that exposes graph-level
and operator-level optimizations to provide performance
portability to deep learning workloads across diverse
hardware back-ends. TVM solves optimization chal-
lenges speciﬁc to deep learning, such as high-level op-
erator fusion, mapping to arbitrary hardware primitives,
and memory latency hiding. It also automates optimiza-
tion of low-level programs to hardware characteristics by
employing a novel, learning-based cost modeling method
for rapid exploration of code optimizations. Experimen-
tal results show that TVM delivers performance across
hardware back-ends that are competitive with state-of-
the-art, hand-tuned libraries for low-power CPU, mo-
bile GPU, and server-class GPUs. We also demonstrate
TVM’s ability to target new accelerator back-ends, such
as the FPGA-based generic deep learning accelerator.
The system is open sourced and in production use inside
several major companies.
1 Introduction
Deep learning (DL) models can now recognize images,
process natural language, and defeat humans in challeng-
ing strategy games. There is a growing demand to deploy
smart applications to a wide spectrum of devices, rang-
ing from cloud servers to self-driving cars and embed-
ded devices. Mapping DL workloads to these devices is
complicated by the diversity of hardware characteristics,
including embedded CPUs, GPUs, FPGAs, and ASICs
(e.g., the TPU [21]). These hardware targets diverge in
L1D L1I L1D L1I
L2 L2
L3
RF RF RF RF
L1/TX L1/TX
L2
SM SM Activation
Buﬀer
Accum.
Register 
File
Wgt. FIFO
CPU GPU ‘TPU’
implicitly managed mixed explicitly managed
Memory Subsystem Architecture
· · ·
Compute  Primitive
scalar vector tensor
Figure 1: CPU, GPU and TPU-like accelerators re-
quire different on-chip memory architectures and com-
pute primitives. This divergence must be addressed when
generating optimized code.
terms of memory organization, compute functional units,
etc., as shown in Figure 1.
Current DL frameworks, such as TensorFlow, MXNet,
Caffe, and PyTorch, rely on a computational graph in-
termediate representation to implement optimizations,
e.g., auto differentiation and dynamic memory man-
agement [3, 4, 9]. Graph-level optimizations, however,
are often too high-level to handle hardware back-end-
speciﬁc operator-level transformations. Most of these
frameworks focus on a narrow class of server-class
GPU devices and delegate target-speciﬁc optimizations
to highly engineered and vendor-speciﬁc operator li-
braries. These operator-level libraries require signiﬁcant
manual tuning and hence are too specialized and opaque
to be easily ported across hardware devices. Providing
support in various DL frameworks for diverse hardware
back-ends presently requires signiﬁcant engineering ef-
fort. Even for supported back-ends, frameworks must
make the difﬁcult choice between: (1) avoiding graph
optimizations that yield new operators not in the prede-
ﬁned operator library, and (2) using unoptimized imple-
mentations of these new operators.
To enable both graph- and operator-level optimiza-
USENIX Association 13th USENIX Symposium on Operating Systems Design and Implementation    579
tions for diverse hardware back-ends, we take a fun-
damentally different, end-to-end approach. We built
TVM, a compiler that takes a high-level speciﬁcation of
a deep learning program from existing frameworks and
generates low-level optimized code for a diverse set of
hardware back-ends. To be attractive to users, TVM
needs to offer performance competitive with the multi-
tude of manually optimized operator libraries across di-
verse hardware back-ends. This goal requires addressing
the key challenges described below.
Leveraging Speciﬁc Hardware Features and Abstrac-
tions. DL accelerators introduce optimized tensor com-
pute primitives [1, 12, 21], while GPUs and CPUs con-
tinuously improve their processing elements. This poses
a signiﬁcant challenge in generating optimized code for
a given operator description. The inputs to hardware in-
structions are multi-dimensional, with ﬁxed or variable
lengths; they dictate different data layouts; and they have
special requirements for memory hierarchy. The system
must effectively exploit these complex primitives to ben-
eﬁt from acceleration. Further, accelerator designs also
commonly favor leaner control [21] and ofﬂoad most
scheduling complexity to the compiler stack. For spe-
cialized accelerators, the system now needs to gener-
ate code that explicitly controls pipeline dependencies to
hide memory access latency – a job that hardware per-
forms for CPUs and GPUs.
Large Search Space for Optimization Another chal-
lenge is producing efﬁcient code without manually tun-
ing operators. The combinatorial choices of memory ac-
cess, threading pattern, and novel hardware primitives
creates a huge conﬁguration space for generated code
(e.g., loop tiles and ordering, caching, unrolling) that
would incur a large search cost if we implement black
box auto-tuning. One could adopt a predeﬁned cost
model to guide the search, but building an accurate cost
model is difﬁcult due to the increasing complexity of
modern hardware. Furthermore, such an approach would
require us to build separate cost models for each hard-
ware type.
TVM addresses these challenges with three key mod-
ules. ( 1) We introduce a tensor expression language
to build operators and provide program transformation
primitives that generate different versions of the pro-
gram with various optimizations. This layer extends
Halide [32]’s compute/schedule separation concept by
also separating target hardware intrinsics from transfor-
mation primitives, which enables support for novel ac-
celerators and their corresponding new intrinsics. More-
over, we introduce new transformation primitives to ad-
dress GPU-related challenges and enable deployment to
specialized accelerators. We can then apply different se-
quences of program transformations to form a rich space
of valid programs for a given operator declaration. ( 2)
We introduce anautomated program optimization frame-
work to ﬁnd optimized tensor operators. The optimizer is
guided by an ML-based cost model that adapts and im-
proves as we collect more data from a hardware back-
end. ( 3) On top of the automatic code generator, we
introduce a graph rewriter that takes full advantage of
high- and operator-level optimizations.
By combining these three modules, TVM can take
model descriptions from existing deep learning frame-
works, perform joint high- and low-level optimizations,
and generate hardware-speciﬁc optimized code for back-
ends, e.g., CPUs, GPUs, and FPGA-based specialized
accelerators.
This paper makes the following contributions:
• We identify the major optimization challenges in pro-
viding performance portability to deep learning work-
loads across diverse hardware back-ends.
• We introduce novel schedule primitives that take ad-
vantage of cross-thread memory reuse, novel hardware
intrinsics, and latency hiding.
• We propose and implement a machine learning based
optimization system to automatically explore and
search for optimized tensor operators.
• We build an end-to-end compilation and optimiza-
tion stack that allows the deployment of deep learning
workloads speciﬁed in high-level frameworks (includ-
ing TensorFlow, MXNet, PyTorch, Keras, CNTK) to
diverse hardware back-ends (including CPUs, server
GPUs, mobile GPUs, and FPGA-based accelerators).
The open-sourced TVM is in production use inside
several major companies.
We evaluated TVM using real world workloads on a
server-class GPU, an embedded GPU, an embedded
CPU, and a custom generic FPGA-based accelerator.
Experimental results show that TVM offers portable
performance across back-ends and achieves speedups
ranging from 1.2× to 3.8× over existing frameworks
backed by hand-optimized libraries.
2 Overview
This section describes TVM by using an example to walk
through its components. Figure 2 summarizes execu-
tion steps in TVM and their corresponding sections in
the paper. The system ﬁrst takes as input a model from
an existing framework and transforms it into a computa-
tional graph representation. It then performs high-level
dataﬂow rewriting to generate an optimized graph. The
operator-level optimization module must generate efﬁ-
cient code for each fused operator in this graph. Oper-
ators are speciﬁed in a declarative tensor expression lan-
580    13th USENIX Symposium on Operating Systems Design and Implementation USENIX Association
Frameworks
High Level Graph Rewriting
Machine Learning Based 
Automated Optimizer
Optimized Computational Graph
Computational Graph
Hardware-Aware 
Optimization Primitives
Declarative
Tensor Expressions
Optimized Low Level Loop Program
LLVM IR CUDA/Metal/OpenCLAccelerator Backend
Deployable Module
Operator-level Optimization and Code Generation
Section 3
Section 4
Section 5
Figure 2: System overview of TVM. The current stack
supports descriptions from many deep learning frame-
works and exchange formats, such as CoreML and
ONNX, to target major CPU, GPU and specialized ac-
celerators.
guage; execution details are unspeciﬁed. TVM identiﬁes
a collection of possible code optimizations for a given
hardware target’s operators. Possible optimizations form
a large space, so we use an ML-based cost model to ﬁnd
optimized operators. Finally, the system packs the gen-
erated code into a deployable module.
End-User Example. In a few lines of code, a user can
take a model from existing deep learning frameworks and
call the TVM API to get a deployable module:
import tvm as t
# Use keras framework as example, import model
graph, params = t.frontend.from_keras(keras_model)
target = t.target.cuda()
graph, lib, params = t.compiler.build(graph, target, params)
This compiled runtime module contains three compo-
nents: the ﬁnal optimized computational graph (graph),
generated operators ( lib), and module parame-
ters (params). These components can then be used to
deploy the model to the target back-end:
import tvm.runtime as t
module = runtime.create(graph, lib, t.cuda(0))
module.set_input(**params)
module.run(data=data_array)
output = tvm.nd.empty(out_shape, ctx=t.cuda(0))
module.get_output(0, output)
TVM supports multiple deployment back-ends in lan-
guages such as C++, Java and Python. The rest of this
paper describes TVM’s architecture and how a system
programmer can extend it to support new back-ends.
3 Optimizing Computational Graphs
Computational graphs are a common way to represent
programs in DL frameworks [3, 4, 7, 9]. Figure 3 shows
conv2d relu conv2d relu ﬂatten
dense
softmax
operation
inputs
dataﬂow
dependency
w1 w2
w3
data
channels=32,
kernel_size=(3,3), 
padding=(1,1),
use_bias=0
example attributes
shape=(1,10)
Figure 3: Example computational graph of a two-layer
convolutional neural network. Each node in the graph
represents an operation that consumes one or more ten-
sors and produces one or more tensors. Tensor operations
can be parameterized by attributes to conﬁgure their be-
havior (e.g., padding or strides).
an example computational graph representation of a two-
layer convolutional neural network. The main differ-
ence between this high-level representation and a low-
level compiler intermediate representation (IR), such as
LLVM, is that the intermediate data items are large,
multi-dimensional tensors. Computational graphs pro-
vide a global view of operators, but they avoid specifying
how each operator must be implemented. Like LLVM
IRs, a computational graph can be transformed into func-
tionally equivalent graphs to apply optimizations. We
also take advantage of shape speciﬁcity in common DL
workloads to optimize for a ﬁxed set of input shapes.
TVM exploits a computational graph representation to
apply high-level optimizations: a node represents an op-
eration on tensors or program inputs, and edges represent
data dependencies between operations. It implements
many graph-level optimizations, including: operator fu-
sion, which fuses multiple small operations together;
constant-folding, which pre-computes graph parts that
can be determined statically, saving execution costs; a
static memory planning pass, which pre-allocates mem-
ory to hold each intermediate tensor; and data layout
transformations, which transform internal data layouts
into back-end-friendly forms. We now discuss operator
fusion and the data layout transformation.
Operator Fusion. Operator fusion combines multiple
operators into a single kernel without saving the interme-
diate results in memory. This optimization can greatly
reduce execution time, particularly in GPUs and spe-
cialized accelerators. Speciﬁcally, we recognize four
categories of graph operators: (1) injective (one-to-one
map, e.g., add), (2) reduction (e.g., sum), (3) complex-
out-fusable (can fuse element-wise map to output, e.g.,
conv2d), and (4) opaque (cannot be fused, e.g., sort). We
provide generic rules to fuse these operators, as follows.
Multiple injective operators can be fused into another in-
jective operator. A reduction operator can be fused with
input injective operators (e.g., fuse scale and sum). Op-
erators such as conv2d are complex-out-fusable, and we
USENIX Association 13th USENIX Symposium on Operating Systems Design and Implementation    581
conv+bn+relu
128x28x28
1x1x128x256
depthwise-
conv+bn+relu
512x14x14
3x3x512
rnn cell
hidden:128
lstm cell
hidden:128
0.00
0.50
1.00
1.50
2.00Relative Speedup
w/o fusion
w/ fusion
Figure 4: Performance comparison between fused and
non-fused operations. TVM generates both operations.
Tested on NVIDIA Titan X.
can fuse element-wise operators to its output. We can
apply these rules to transform the computational graph
into a fused version. Figure 4 demonstrates the impact
of this optimization on different workloads. We ﬁnd that
fused operators generate up to a 1.2 × to 2× speedup by
reducing memory accesses.
Data Layout Transformation. There are multiple
ways to store a given tensor in the computational graph.
The most common data layout choices are column major
and row major. In practice, we may prefer to use even
more complicated data layouts. For instance, a DL ac-
celerator might exploit 4×4 matrix operations, requiring
data to be tiled into 4 ×4 chunks to optimize for access
locality.
Data layout optimization converts a computational
graph into one that can use better internal data layouts
for execution on the target hardware. It starts by spec-
ifying the preferred data layout for each operator given
the constraints dictated by memory hierarchies. We then
perform the proper layout transformation between a pro-
ducer and a consumer if their preferred data layouts do
not match.
While high-level graph optimizations can greatly im-
prove the efﬁciency of DL workloads, they are only as
effective as what the operator library provides. Cur-
rently, the few DL frameworks that support operator fu-
sion require the operator library to provide an implemen-
tation of the fused patterns. With more network opera-
tors introduced on a regular basis, the number of possible
fused kernels can grow dramatically. This approach is
no longer sustainable when targeting an increasing num-
ber of hardware back-ends since the required number
of fused pattern implementations grows combinatorially
with the number of data layouts, data types, and accel-
erator intrinsics that must be supported. It is not feasi-
ble to handcraft operator kernels for the various opera-
tions desired by a program and for each back-end. To
for y in range(1024):
  for x in range(1024):
    C[y][x] = 0
    for k in range(1024):
      C[y][x] += A[k][y] * B[k][x]
for yo in range(128):
  for xo in range(128):
    C[yo*8:yo*8+8][xo*8:xo*8+8] = 0
    for ko in range(128):
      for yi in range(8):
        for xi in range(8):
          for ki in range(8):
            C[yo*8+yi][xo*8+xi] += 
               A[ko*8+ki][yo*8+yi] * B[ko*8+ki][xo*8+xi]
inp_buffer AL[8][8], BL[8][8]
acc_buffer CL[8][8]
for yo in range(128):
  for xo in range(128):
    vdla.fill_zero(CL)
    for ko in range(128):
      vdla.dma_copy2d(AL, A[ko*8:ko*8+8][yo*8:yo*8+8])
      vdla.dma_copy2d(BL, B[ko*8:ko*8+8][xo*8:xo*8+8])        
      vdla.fused_gemm8x8_add(CL, AL, BL)
    vdla.dma_copy2d(C[yo*8:yo*8+8,xo*8:xo*8+8], CL)
+ Cache Data on Accelerator Special Buﬀer 
A = t.placeholder((1024, 1024))
B = t.placeholder((1024, 1024))
k = t.reduce_axis((0, 1024))
C = t.compute((1024, 1024), lambda y, x: 
                t.sum(A[k, y] * B[k, x], axis=k))
s = t.create_schedule(C.op)
schedule schedule 
transformation
corresponding 
low-level code
+ Map to Accelerator Tensor Instructions
CL = s.cache_write(C, vdla.acc_buffer)
AL = s.cache_read(A, vdla.inp_buffer)
# additional schedule steps omitted …
s[CL].tensorize(yi, vdla.gemm8x8)
+ Loop Tiling
yo, xo, ko, yi, xi, ki = s[C].tile(y, x, k, 8, 8, 8)
Figure 5: Example schedule transformations that opti-
mize a matrix multiplication on a specialized accelerator.
this end, we next propose a code generation approach
that can generate various possible implementations for a
given model’s operators.
4 Generating Tensor Operations
TVM produces efﬁcient code for each operator by gen-
erating many valid implementations on each hardware
back-end and choosing an optimized implementation.
This process builds on Halide’s idea of decoupling de-
scriptions from computation rules (or schedule optimiza-
tions) [32] and extends it to support new optimizations
(nested parallelism, tensorization, and latency hiding)
and a wide array of hardware back-ends. We now high-
light TVM-speciﬁc features.
4.1 Tensor Expression and Schedule Space
We introduce a tensor expression language to support au-
tomatic code generation. Unlike high-level computation
graph representations, where the implementation of ten-
sor operations is opaque, each operation is described in
582    13th USENIX Symposium on Operating Systems Design and Implementation USENIX Association
Schedule primitives  used in various hardware backendsCPU ScheduleGPU Schedule ee
Accel. Schedule ule[Halide] Loop Transformations
✔
 ✔
 ✔[Halide] Thread Binding
✔
 ✔
 ✔[Halide] Compute Locality
✔
 ✔
 ✔[TVM] Special Memory Scope
✔
 ✔[TVM] Tensorization
✔
 ✔
 ✔[TVM] Latency Hiding
 ✔
Tensor Expression
Code Lowering
Select Schedule 
Primitives
Final Schedule
Low level code
Figure 6: TVM schedule lowering and code generation
process. The table lists existing Halide and novel TVM
scheduling primitives being used to optimize schedules
for CPUs, GPUs and accelerator back-ends. Tensoriza-
tion is essential for accelerators, but it can also be used
for CPUs and GPUs. Special memory-scope enables
memory reuse in GPUs and explicit management of on-
chip memory in accelerators. Latency hiding is speciﬁc
to TPU-like accelerators.
an index formula expression language. The following
code shows an example tensor expression to compute
transposed matrix multiplication:
m, n, h = t.var('m'), t.var('n'), t.var('h')
A = t.placeholder((m, h), name='A')
B = t.placeholder((n, h), name='B')
k = t.reduce_axis((0, h), name='k')
C = t.compute((m, n), lambda y, x: 
                   t.sum(A[k, y] * B[k, x], axis=k))
result shape
computing rule
Each compute operation speciﬁes both the shape of
the output tensor and an expression describing how to
compute each element of it. Our tensor expression
language supports common arithmetic and math oper-
ations and covers common DL operator patterns. The
language does not specify the loop structure and many
other execution details, and it provides ﬂexibility for
adding hardware-aware optimizations for various back-
ends. Adopting the decoupled compute/schedule princi-
ple from Halide [32], we use a schedule to denote a spe-
ciﬁc mapping from a tensor expression to low-level code.
Many possible schedules can perform this function.
We build a schedule by incrementally applying basic
transformations (schedule primitives) that preserve the
program’s logical equivalence. Figure 5 shows an ex-
ample of scheduling matrix multiplication on a special-
ized accelerator. Internally, TVM uses a data structure
to keep track of the loop structure and other information
as we apply schedule transformations. This information
can then help generate low-level code for a given ﬁnal
schedule.
Our tensor expression takes cues from Halide [32],
Darkroom [17], and TACO [23]. Its primary enhance-
ments include support for the new schedule optimiza-
tions discussed below. To achieve high performance
on many back-ends, we must support enough schedule
primitives to cover a diverse set of optimizations on dif-
ferent hardware back-ends. Figure 6 summarizes the
operation code generation process and schedule primi-
1024 2048
Matrix Size
0
1
2
3
4
5
6
7
8Time (ms)
cuBLAS
TVM w/o coop.
TVM
Figure 7: Performance comparison between TVM with
and without cooperative shared memory fetching on ma-
trix multiplication workloads. Tested on an NVIDIA Ti-
tan X.
tives that TVM supports. We reuse helpful primitives
and the low-level loop program AST from Halide, and
we introduce new primitives to optimize GPU and ac-
celerator performance. The new primitives are neces-
sary to achieve optimal GPU performance and essen-
tial for accelerators. CPU, GPU, TPU-like accelerators
are three important types of hardware for deep learning.
This section describes new optimization primitives for
CPUs, GPUs and TPU-like accelerators, while section 5
explains how to automatically derive efﬁcient schedules.
4.2 Nested Parallelism with Cooperation
Parallelism is key to improving the efﬁciency of
compute-intensive kernels in DL workloads. Modern
GPUs offer massive parallelism, requiring us to bake par-
allel patterns into schedule transformations. Most exist-
ing solutions adopt a model called nested parallelism, a
form of fork–join. This model requires a parallel sched-
ule primitive to parallelize a data parallel task; each task
can be further recursively subdivided into subtasks to ex-
ploit the target architecture’s multi-level thread hierarchy
(e.g., thread groups in GPU). We call this model shared-
nothing nested parallelism because one working thread
cannot look at the data of its sibling within the same par-
allel computation stage.
An alternative to the shared-nothing approach is to
fetch data cooperatively. Speciﬁcally, groups of threads
can cooperatively fetch the data they all need and place
it into a shared memory space. 1 This optimization can
take advantage of the GPU memory hierarchy and en-
1 Halide recently added shared memory support but without general
memory scope for accelerators.
USENIX Association 13th USENIX Symposium on Operating Systems Design and Implementation    583
able data reuse across threads through shared memory
regions. TVM supports this well-known GPU optimiza-
tion using a schedule primitive to achieve optimal per-
formance. The following GPU code example optimizes
matrix multiplication.
Barrier inserted 
automatically
by compiler
All threads cooperatively
load AS and BS in diﬀerent
parallel patterns
for thread_group (by, bx) in cross(64, 64):
  for thread_item (ty, tx) in cross(2, 2):
    local CL[8][8] = 0
    shared AS[2][8], BS[2][8] 
    for k in range(1024):
      for i in range(4):
        AS[ty][i*4+tx] = A[k][by*64+ty*8+i*4+tx]
      for each i in 0..4:
        BS[ty][i*4+tx] = B[k][bx*64+ty*8+i*4+tx]
      memory_barrier_among_threads()
      for yi in range(8):
        for xi in range(8):
          CL[yi][xi] += AS[yi] * BS[xi]
      for yi in range(8):
        for xi in range(8):
          C[yo*8+yi][xo*8+xi] = CL[yi][xi]
Figure 7 demonstrates the impact of this optimiza-
tion. We introduce the concept of memory scopes to the
schedule space so that a compute stage (AS and BS in the
code) can be marked as shared. Without explicit memory
scopes, automatic scope inference will mark compute
stages as thread-local. The shared task must compute
the dependencies of all working threads in the group.
Additionally, memory synchronization barriers must be
properly inserted to guarantee that shared loaded data is
visible to consumers. Finally, in addition to being use-
ful to GPUs, memory scopes let us tag special memory
buffers and create special lowering rules when targeting
specialized DL accelerators.
4.3 Tensorization
DL workloads have high arithmetic intensity, which
can typically be decomposed into tensor operators like
matrix-matrix multiplication or 1D convolution. These
natural decompositions have led to the recent trend of
adding tensor compute primitives [1, 12, 21]. These
new primitives create both opportunities and challenges
for schedule-based compilation; while using them can
improve performance, the compilation framework must
seamlessly integrate them. We dub this tensorization: it
is analogous to vectorization for SIMD architectures but
has signiﬁcant differences. Instruction inputs are multi-
dimensional, with ﬁxed or variable lengths, and each has
different data layouts. More importantly, we cannot sup-
port a ﬁxed set of primitives since new accelerators are
emerging with their own variations of tensor instructions.
We therefore need an extensible solution.
We make tensorization extensible by separating the
target hardware intrinsic from the schedule with a mech-
anism for tensor-intrinsic declaration. We use the same
tensor expression language to declare both the behavior
of each new hardware intrinsic and the lowering rule as-
sociated with it. The following code shows how to de-
clare an 8 ×8 tensor hardware intrinsic.
w, x = t.placeholder((8, 8)), t.placeholder((8, 8))
k = t.reduce_axis((0, 8))
y = t.compute((8, 8), lambda i, j: 
               t.sum(w[i, k] * x[j, k], axis=k))
def gemm_intrin_lower(inputs, outputs):
   ww_ptr = inputs[0].access_ptr(“r")
   xx_ptr = inputs[1].access_ptr("r")
   zz_ptr = outputs[0].access_ptr("w")
   compute = t.hardware_intrin("gemm8x8", ww_ptr, xx_ptr, zz_ptr)
   reset = t.hardware_intrin("fill_zero", zz_ptr)
   update = t.hardware_intrin("fuse_gemm8x8_add", ww_ptr, xx_ptr, zz_ptr)
   return compute, reset, update
gemm8x8 = t.decl_tensor_intrin(y.op, gemm_intrin_lower)
declare behavior
lowering rule to generate
hardware intrinsics to carry 
out the computation
Additionally, we introduce atensorize schedule primi-
tive to replace a unit of computation with the correspond-
ing intrinsics. The compiler matches the computation
pattern with a hardware declaration and lowers it to the
corresponding hardware intrinsic.
Tensorization decouples the schedule from speciﬁc
hardware primitives, making it easy to extend TVM
to support new hardware architectures. The generated
code of tensorized schedules aligns with practices in
high-performance computing: break complex operations
into a sequence of micro-kernel calls. We can also use
the tensorize primitive to take advantage of handcrafted
micro-kernels, which can be beneﬁcial in some plat-
forms. For example, we implement ultra low precision
operators for mobile CPUs that operate on data types
that are one- or two-bits wide by leveraging a bit-serial
matrix vector multiplication micro-kernel. This micro-
kernel accumulates results into progressively larger data
types to minimize the memory footprint. Presenting the
micro-kernel as a tensor intrinsic to TVM yields up to a
1.5× speedup over the non-tensorized version.
4.4 Explicit Memory Latency Hiding
Latency hiding refers to the process of overlapping mem-
ory operations with computation to maximize utilization
of memory and compute resources. It requires different
strategies depending on the target hardware back-end.
On CPUs, memory latency hiding is achieved implic-
itly with simultaneous multithreading [14] or hardware
prefetching [10, 20]. GPUs rely on rapid context switch-
ing of many warps of threads [44]. In contrast, special-
ized DL accelerators such as the TPU [21] usually favor
leaner control with a decoupled access-execute (DAE)
architecture [35] and ofﬂoad the problem of ﬁne-grained
synchronization to software.
Figure 9 shows a DAE hardware pipeline that reduces
runtime latency. Compared to a monolithic hardware de-
sign, the pipeline can hide most memory access over-
heads and almost fully utilize compute resources. To
achieve higher utilization, the instruction stream must be
augmented with ﬁne-grained synchronization operations.
Without them, dependencies cannot be enforced, leading
to erroneous execution. Consequently, DAE hardware
pipelines require ﬁne-grained dependence enqueuing/d-
equeuing operations between the pipeline stages to guar-
584    13th USENIX Symposium on Operating Systems Design and Implementation USENIX Association
l
d
l
d
e
x
e
x
l
d
l
d … e
x
e
x
vthread 0
vthread 1
barrier
ld ex …ld ld ex
ld ex …ld ld ex
barrier
vthread 0
vthread 1
ld ex
ld ex
…
…
ld
ld
ld ex
ld ex
Input: High-level Threaded Program Final Single Instruction StreamInject Synchronization Instructions
for vthread tx in range(2):
  acc_buffer CL[8]
  inp_buffer AL[8]
  for k in range(128):
    ld.dma_copy2d(AL, AL[k][tx*8:tx*8+8])
    ex.accumulate(AL, CL)
  acc_buffer CL[2][8]
  inp_buffer AL[2][8]
  ex.push_dep_to(ld)
  ex.push_dep_to(ld)
  for k in range(128):
    ld.pop_dep_from(ex)
    ld.dma_copy2d(AL[0], AL[k][0:8])
    ld.push_dep_to(ex)
    ld.pop_dep_from(ex)
    ld.dma_copy2d(AL[1], AL[k][8:16])
    ld.push_dep_to(ex)
    ex.pop_dep_from(ld)
    ex.accumulate(AL[0], CL[0])
    ex.push_dep_to(ld)
    ex.pop_dep_from(ld)
    ex.accumulate(AL[1], CL[1])
    ex.push_dep_to(ld)
  ld.pop_dep_from(ex)
  ld.pop_dep_from(ex)
for vthread tx in range(2):
  acc_buffer CL[8]
  inp_buffer AL[8]
  ex.push_dep_to(ld)
  for k in range(128):
    ld.pop_dep_from(ex)
    ld.dma_copy2d(AL, AL[k][tx*8:tx*8+8])
    ld.push_dep_to(ex)
    ex.pop_dep_from(ld)
    ex.accumulate(AL, CL)
    ex.push_dep_to(ld)
  ld.pop_dep_from(ex)
read after write  (RAW) dependence
read after write  (RAW) dependence
push RAW dependence
push WAR dependence
pop RAW dependence
pop WAR dependence
Figure 8: TVM virtual thread lowering transforms a virtual thread-parallel program to a single instruction stream; the
stream contains explicit low-level synchronizations that the hardware can interpret to recover the pipeline parallelism
required to hide memory access latency.
ld
0
ex 
0
ld
1
ex 
1
ld
2
ld
3
ex 
2
ex 
3
ld 
0
ex 
0
ld 
1
ex 
1
ld 
2
ex 
2
ld
3
ex 
3
execution savings
Decoupled Access-Execute Pipeline
t
Monolithic Pipeline
write after read  (WAR) dependence
read after write  (RAW) dependence
ld.perform_action(ld0)
ld.push_dep_to(ex)
ld.perform_action(ld1)
ld.push_dep_to(ex)
ex.pop_dep_from(ld)
ex.perform_action(ex0)
ex.push_dep_to(ld)
ex.pop_dep_from(ld)
ex.perform_action(ex1)
ex.push_dep_to(ld)
ld.pop_dep_from(ex)
ld.perform_action(ld2)
...
ld.perform_action(ld0)
ex.perform_action(ex0)
ld.perform_action(ld1)
ex.perform_action(ex1)
...
Instruction Stream
Figure 9: Decoupled Access-Execute in hardware hides
most memory access latency by allowing memory and
computation to overlap. Execution correctness is en-
forced by low-level synchronization in the form of de-
pendence token enqueueing/dequeuing actions, which
the compiler stack must insert in the instruction stream.
antee correct execution, as shown in Figure 9’s instruc-
tion stream.
Programming DAE accelerators that require explicit
low-level synchronization is difﬁcult. To reduce the
programming burden, we introduce a virtual threading
scheduling primitive that lets programmers specify a
high-level data parallel program as they would a hard-
ware back-end with support for multithreading. TVM
then automatically lowers the program to a single in-
struction stream with low-level explicit synchronization,
as shown in Figure 8. The algorithm starts with a high-
level multi-threaded program schedule and then inserts
the necessary low-level synchronization operations to
guarantee correct execution within each thread. Next,
it interleaves operations of all virtual threads into a sin-
gle instruction stream. Finally, the hardware recovers the
compute boundmemory bound
Figure 10: Rooﬂine [47] of an FPGA-based DL ac-
celerator running ResNet inference. With latency hid-
ing enabled by TVM, performance of the benchmarks is
brought closer to the rooﬂine, demonstrating higher com-
pute and memory bandwidth efﬁciency.
available pipeline parallelism dictated by the low-level
synchronizations in the instruction stream.
Hardware Evaluation of Latency Hiding. We now
demonstrate the effectiveness of latency hiding on a cus-
tom FPGA-based accelerator design, which we describe
in depth in subsection 6.4. We ran each layer of ResNet
on the accelerator and used TVM to generate two sched-
ules: one with latency hiding, and one without. The
schedule with latency hiding parallelized the program
with virtuals threads to expose pipeline parallelism and
therefore hide memory access latency. Results are shown
in Figure 10 as a rooﬂine diagram [47]; rooﬂine perfor-
mance diagrams provide insight into how well a given
system uses computation and memory resources for dif-
ferent benchmarks. Overall, latency hiding improved
performance on all ResNet layers. Peak compute utiliza-
tion increased from 70% with no latency hiding to 88%
with latency hiding.
USENIX Association 13th USENIX Symposium on Operating Systems Design and Implementation    585
5 Automating Optimization
Given the rich set of schedule primitives, our remaining
problem is to ﬁnd optimal operator implementations for
each layer of a DL model. Here, TVM creates a special-
ized operator for the speciﬁc input shape and layout as-
sociated with each layer. Such specialization offers sig-
niﬁcant performance beneﬁts (in contrast to handcrafted
code that would target a smaller diversity of shapes and
layouts), but it also raises automation challenges. The
system needs to choose the schedule optimizations –
such as modifying the loop order or optimizing for the
memory hierarchy – as well as schedule-speciﬁc param-
eters, such as the tiling size and the loop unrolling factor.
Such combinatorial choices create a large search space of
operator implementations for each hardware back-end.
To address this challenge, we built an automated sched-
ule optimizer with two main components: a schedule ex-
plorer that proposes promising new conﬁgurations, and
a machine learning cost model that predicts the perfor-
mance of a given conﬁguration. This section describes
these components and TVM’s automated optimization
ﬂow (Figure 11).
5.1 Schedule Space Speciﬁcation
We built a schedule template speciﬁcation API to let a
developer declare knobs in the schedule space. The tem-
plate speciﬁcation allows incorporation of a developer’s
domain-speciﬁc knowledge, as necessary, when specify-
ing possible schedules. We also created a generic mas-
ter template for each hardware back-end that automati-
cally extracts possible knobs based on the computation
description expressed using the tensor expression lan-
guage. At a high level, we would like to consider as many
conﬁgurations as possible and let the optimizer manage
the selection burden. Consequently, the optimizer must
search overbillions of possible conﬁgurations for the real
world DL workloads used in our experiments.
5.2 ML-Based Cost Model
One way to ﬁnd the best schedule from a large conﬁgu-
ration space is through blackbox optimization, i.e., auto-
tuning. This method is used to tune high performance
computing libraries [15, 46]. However, auto-tuning re-
quires many experiments to identify a good conﬁgura-
tion.
An alternate approach is to build a predeﬁned cost
model to guide the search for a particular hardware back-
end instead of running all possibilities and measuring
their performance. Ideally, a perfect cost model con-
siders all factors affecting performance: memory access
patterns, data reuse, pipeline dependencies, and thread-
Raspberry Pi
Tracker
Mali GPU
Nvidia GPU
TensorOp 
Speciﬁcation
Schedule Space 
Template
Database
Device Cluster
Schedule Explorer
ML Cost Model
log
querytraining 
data FPGA Board
rpc
get_perf
…
update
Figure 11: Overview of automated optimization frame-
work. A schedule explorer examines the schedule space
using an ML-based cost model and chooses experiments
to run on a distributed device cluster via RPC. To im-
prove its predictive power, the ML model is updated pe-
riodically using collected data recorded in a database.
Method Category Data
Cost
Model
Bias
Need
Hardware
Info
Learn
from
His-
tory
Blackbox auto-tuning high none no no
Predeﬁned cost model none high yes no
ML based cost model low low no yes
Table 1: Comparison of automation methods. Model bias
refers to inaccuracy due to modeling.
ing patterns, among others. This approach, unfortu-
nately, is burdensome due to the increasing complexity
of modern hardware. Furthermore, every new hardware
target requires a new (predeﬁned) cost model.
We instead take a statistical approach to solve the cost
modeling problem. In this approach, a schedule explorer
proposes conﬁgurations that may improve an operator’s
performance. For each schedule conﬁguration, we use
an ML model that takes the lowered loop program as in-
put and predicts its running time on a given hardware
back-end. The model, trained using runtime measure-
ment data collected during exploration, does not require
the user to input detailed hardware information. We up-
date the model periodically as we explore more conﬁg-
urations during optimization, which improves accuracy
for other related workloads, as well. In this way, the qual-
ity of the ML model improves with more experimental
trials. Table 1 summarizes the key differences between
automation methods. ML-based cost models strike a bal-
ance between auto-tuning and predeﬁned cost modeling
and can beneﬁt from the historical performance data of
related workloads.
Machine Learning Model Design Choices. We must
consider two key factors when choosing which ML
model the schedule explorer will use: quality and speed.
The schedule explorer queries the cost model frequently,
which incurs overheads due to model prediction time
and model reﬁtting time. To be useful, these overheads
must be smaller than the time it takes to measure per-
586    13th USENIX Symposium on Operating Systems Design and Implementation USENIX Association
0 100 200 300 400 500 600 700 800
Number of Trials
0.00
0.25
0.50
0.75
1.00
1.25
1.50
Relative Speedup
TVM: ML-based Model
TVM: Blackbox Genetic Algorithm
TVM: Random Search
Baseline: cuDNN
Figure 12: Comparison of different automation methods
for a conv2d operator in ResNet-18 on TITAN X. The
ML-based model starts with no training data and uses
the collected data to improve itself. The Y-axis is the
speedup relative to cuDNN. We observe a similar trend
for other workloads.
for yo in range(4):
  for xo in range(4):
    C[yo*2:yo*2+2][xo*2:xo*2+2] = 0
    for ko in range(8):
     for yi in range(2):
        for xi in range(2):
          C[yo*2+yi][xo*2+xi] +=  
            A[k][yo*2+yi] * B[k][xo*2+xi]
xi yi k xo yo
C 2 4 4 16 64
A 1 2 16 16 64
B 2 2 16 64 64
Feature ExtractionQuery: Loop AST
cost prediction
e.g. touched memory size
Schedule Explorer
XGBoost
alternatively, we can feed AST to TreeRNN
Figure 13: Example workﬂow of ML cost models. XG-
Boost predicts costs based on loop program features.
TreeRNN directly summarizes the AST.
formance on real hardware, which can be on the order
of seconds depending on the speciﬁc workload/hardware
target. This speed requirement differentiates our problem
from traditional hyperparameter tuning problems, where
the cost of performing measurements is very high rela-
tive to model overheads, and more expensive models can
be used. In addition to the choice of model, we need
to choose an objective function to train the model, such
as the error in a conﬁguration’s predicted running time.
However, since the explorer selects the top candidates
based only on the relative order of the prediction (A runs
faster than B), we need not predict the absolute execution
times directly. Instead, we use a rank objective to predict
the relative order of runtime costs.
We implement several types of models in our ML opti-
mizer. We employ a gradient tree boosting model (based
on XGBoost [8]), which makes predictions based on fea-
tures extracted from the loop program; these features in-
clude the memory access count and reuse ratio of each
memory buffer at each loop level, as well as a one-hot
encoding of loop annotations such as “vectorize”, “un-
roll”, and “parallel.” We also evaluate a neural network
model that uses TreeRNN [38] to summarize the loop
program’s AST without feature engineering. Figure 13
summarizes the workﬂow of the cost models. We found
that tree boosting and TreeRNN have similar predictive
quality. However, the former performs prediction twice
as fast and costs much less time to train. As a result, we
chose gradient tree boosting as the default cost model in
our experiments. Nevertheless, we believe that both ap-
proaches are valuable and expect more future research on
this problem.
On average, the tree boosting model does prediction
in 0.67 ms, thousands of times faster than running a real
measurement. Figure 12 compares an ML-based opti-
mizer to blackbox auto-tuning methods; the former ﬁnds
better conﬁgurations much faster than the latter.
5.3 Schedule Exploration
Once we choose a cost model, we can use it to select
promising conﬁgurations on which to iteratively run real
measurements. In each iteration, the explorer uses the
ML model’s predictions to select a batch of candidates
on which to run the measurements. The collected data is
then used as training data to update the model. If no ini-
tial training data exists, the explorer picks random candi-
dates to measure.
The simplest exploration algorithm enumerates and
runs every conﬁguration through the cost model, se-
lecting the top- k predicted performers. However, this
strategy becomes intractable with large search spaces.
Instead, we run a parallel simulated annealing algo-
rithm [22]. The explorer starts with random conﬁgura-
tions, and, at each step, randomly walks to a nearby con-
ﬁguration. This transition is successful if cost decreases
as predicted by the cost model. It is likely to fail (reject)
if the target conﬁguration has a higher cost. The random
walk tends to converge on conﬁgurations that have lower
costs as predicted by the cost model. Exploration states
persist across cost model updates; we continue from the
last conﬁguration after these updates.
5.4 Distributed Device Pool and RPC
A distributed device pool scales up the running of on-
hardware trials and enables ﬁne-grained resource sharing
among multiple optimization jobs. TVM implements a
customized, RPC-based distributed device pool that en-
ables clients to run programs on a speciﬁc type of de-
vice. We can use this interface to compile a program
on the host compiler, request a remote device, run the
USENIX Association 13th USENIX Symposium on Operating Systems Design and Implementation    587
Name Operator H,W IC , OC K , S
C1 conv2d 224, 224 3,64 7, 2
C2 conv2d 56, 56 64,64 3, 1
C3 conv2d 56, 56 64,64 1, 1
C4 conv2d 56, 56 64,128 3, 2
C5 conv2d 56, 56 64,128 1, 2
C6 conv2d 28, 28 128,128 3, 1
C7 conv2d 28, 28 128,256 3, 2
C8 conv2d 28, 28 128,256 1, 2
C9 conv2d 14, 14 256,256 3, 1
C10 conv2d 14, 14 256,512 3, 2
C11 conv2d 14, 14 256,512 1, 2
C12 conv2d 7, 7 512,512 3, 1
Name Operator H,W IC K , S
D1 depthwise conv2d 112, 112 32 3, 1
D2 depthwise conv2d 112, 112 64 3, 2
D3 depthwise conv2d 56, 56 128 3, 1
D4 depthwise conv2d 56, 56 128 3, 2
D5 depthwise conv2d 28, 28 256 3, 1
D6 depthwise conv2d 28, 28 256 3, 2
D7 depthwise conv2d 14, 14 512 3, 1
D8 depthwise conv2d 14, 14 512 3, 2
D9 depthwise conv2d 7, 7 1024 3, 1
Table 2: Conﬁgurations of all conv2d operators in
ResNet-18 and all depthwise conv2d operators in Mo-
bileNet used in the single kernel experiments. H/W
denotes height and width, IC input channels, OC out-
put channels, K kernel size, and S stride size. All ops
use “SAME” padding. All depthwise conv2d operations
have channel multipliers of 1.
function remotely, and access results in the same script
on the host. TVM’s RPC supports dynamic upload and
runs cross-compiled modules and functions that use its
runtime convention. As a result, the same infrastruc-
ture can perform a single workload optimization and
end-to-end graph inference. Our approach automates the
compile, run, and proﬁle steps across multiple devices.
This infrastructure is especially critical for embedded de-
vices, which traditionally require tedious manual effort
for cross-compilation, code deployment, and measure-
ment.
6 Evaluation
TVM’s core is implemented in C++ ( ∼50k LoC). We
provide language bindings to Python and Java. Earlier
sections of this paper evaluated the impact of several in-
dividual optimizations and components of TVM, namely,
operator fusion in Figure 4, latency hiding in Figure 10,
and the ML-based cost model in Figure 12. We now fo-
cus on an end-to-end evaluation that aims to answer the
following questions:
• Can TVM optimize DL workloads over multiple
platforms?
• How does TVM compare to existing DL frame-
ResNet-18 MobileNet0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0Time(ms)
LSTM LM DQN DCGAN0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Tensorﬂow XLA
Tensorﬂow
MXNet
TVM w/o graph opt
TVM
Figure 14: GPU end-to-end evaluation for TVM,
MXNet, Tensorﬂow, and Tensorﬂow XLA. Tested on the
NVIDIA Titan X.
works (which rely on heavily optimized libraries)
on each back-end?
• Can TVM support new, emerging DL workloads
(e.g., depthwise convolution, low precision opera-
tions)?
• Can TVM support and optimize for new specialized
accelerators?
To answer these questions, we evaluated TVM on four
types of platforms: (1) a server-class GPU, (2) an embed-
ded GPU, (3) an embedded CPU, and (4) a DL accelera-
tor implemented on a low-power FPGA SoC. The bench-
marks are based on real world DL inference workloads,
including ResNet [16], MobileNet [19], the LSTM Lan-
guage Model [48], the Deep Q Network (DQN) [28] and
Deep Convolutional Generative Adversarial Networks
(DCGAN) [31]. We compare our approach to exist-
ing DL frameworks, including MxNet [9] and Tensor-
Flow [2], that rely on highly engineered, vendor-speciﬁc
libraries. TVM performs end-to-end automatic optimiza-
tion and code generationwithout the need for an external
operator library.
6.1 Server-Class GPU Evaluation
We ﬁrst compared the end-to-end performance of
deep neural networks TVM, MXNet (v1.1), Tensor-
ﬂow (v1.7), and Tensorﬂow XLA on an Nvidia Titan
X. MXNet and Tensorﬂow both use cuDNN v7 for con-
volution operators; they implement their own versions
of depthwise convolution since it is relatively new and
not yet supported by the latest libraries. They also use
cuBLAS v8 for matrix multiplications. On the other
hand, Tensorﬂow XLA uses JIT compilation.
Figure 14 shows that TVM outperforms the base-
lines, with speedups ranging from 1.6 × to 3.8× due to
both joint graph optimization and the automatic opti-
mizer, which generates high-performance fused opera-
588    13th USENIX Symposium on Operating Systems Design and Implementation USENIX Association
C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C120.0
0.5
1.0
1.5
2.0
2.5
3.0Relative Speedup
cuDNN
TensorComprehensions
MX Kernel
TVM
TVM PT
D1 D2 D3 D4 D5 D6 D7 D8 D90.0
1.0
2.0
3.0
4.0
5.0Relative Speedup
Figure 15: Relative speedup of all conv2d operators in
ResNet-18 and all depthwise conv2d operators in Mo-
bileNet. Tested on a TITAN X. See Table 2 for op-
erator conﬁgurations. We also include a weight pre-
transformed Winograd [25] for 3x3 conv2d (TVM PT).
tors. DQN’s 3.8 x speedup results from its use of un-
conventional operators (4×4 conv2d, strides=2) that are
not well optimized by cuDNN; the ResNet workloads are
more conventional. TVM automatically ﬁnds optimized
operators in both cases.
To evaluate the effectiveness of operator level opti-
mization, we also perform a breakdown comparison for
each tensor operator in ResNet and MobileNet, shown in
Figure 15. We include TensorComprehension (TC, com-
mit: ef644ba) [42], a recently introduced auto-tuning
framework, as an additional baseline. 2 TC results in-
clude the best kernels it found in 10 generations × 100
population × 2 random seeds for each operator (i.e.,
2000 trials per operator). 2D convolution, one of the
most important DL operators, is heavily optimized by
cuDNN. However, TVM can still generate better GPU
kernels for most layers. Depthwise convolution is a
newly introduced operator with a simpler structure [19].
In this case, both TVM and TC can ﬁnd fast kernels com-
pared to MXNet’s handcrafted kernels. TVM’s improve-
ments are mainly due to its exploration of a large sched-
ule space and an effective ML-based search algorithm.
6.2 Embedded CPU Evaluation
We evaluated the performance of TVM on an ARM Cor-
tex A53 (Quad Core 1.2GHz). We used Tensorﬂow Lite
(TFLite, commit: 7558b085) as our baseline system.
Figure 17 compares TVM operators to hand-optimized
2According to personal communication [41], TC is not yet meant
to be used for compute-bound problems. However, it is still a good
reference baseline to include in the comparison.
ResNet-18 MobileNet0.0
100.0
200.0
300.0
400.0
500.0
600.0
700.0
800.0Time(ms)
DQN0.0
2.0
4.0
6.0
8.0
10.0
12.0
Tensorﬂow Lite TVM w/o graph opt TVM
Figure 16: ARM A53 end-to-end evaluation of TVM and
TFLite.
Figure 17: Relative speedup of all conv2d operators in
ResNet-18 and all depthwise conv2d operators in mo-
bilenet. Tested on ARM A53. See Table 2 for the con-
ﬁgurations of these operators.
ones for ResNet and MobileNet. We observe that TVM
generates operators that outperform the hand-optimized
TFLite versions for both neural network workloads. This
result also demonstrates TVM’s ability to quickly opti-
mize emerging tensor operators, such as depthwise con-
volution operators. Finally, Figure 16 shows an end-to-
end comparison of three workloads, where TVM outper-
forms the TFLite baseline.3
Ultra Low-Precision Operators We demonstrate
TVM’s ability to support ultra low-precision infer-
ence [13, 33] by generating highly optimized operators
for ﬁxed-point data types of less than 8-bits. Low-
precision networks replace expensive multiplication with
vectorized bit-serial multiplication that is composed of
bitwise and popcount reductions [39]. Achieving efﬁ-
cient low-precision inference requires packing quantized
data types into wider standard data types, such as int8
or int32. Our system generates code that outperforms
hand-optimized libraries from Caffe2 (commit: 39e07f7)
3DCGAN and LSTM results are not presented because they are not
yet supported by the baseline.
USENIX Association 13th USENIX Symposium on Operating Systems Design and Implementation    589
C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C120.0
2.0
4.0
6.0
8.0
10.0Relative Speedup
Hand optimized
TVM single-threaded
TVM multi-threaded
Figure 18: Relative speedup of single- and multi-
threaded low-precision conv2d operators in ResNet.
Baseline was a single-threaded, hand-optimized imple-
mentation from Caffe2 (commit: 39e07f7). C5, C3 are
1x1 convolutions that have less compute intensity, result-
ing in less speedup by multi-threading.
[39]. We implemented an ARM-speciﬁc tensorization
intrinsic that leverages ARM instructions to build an ef-
ﬁcient, low-precision matrix-vector microkernel.We then
used TVM’s automated optimizer to explore the schedul-
ing space.
Figure 18 compares TVM to the Caffe2 ultra low-
precision library on ResNet for 2-bit activations, 1-bit
weights inference. Since the baseline is single threaded,
we also compare it to a single-threaded TVM version.
Single-threaded TVM outperforms the baseline, particu-
larly for C5, C8, and C11 layers; these are convolution
layers of kernel size 1×1 and stride of 2 for which the ul-
tra low-precision baseline library is not optimized. Fur-
thermore, we take advantage of additional TVM capa-
bilities to produce a parallel library implementation that
shows improvement over the baseline. In addition to the
2-bit+1-bit conﬁguration, TVM can generate and opti-
mize for other precision conﬁgurations that are unsup-
ported by the baseline library, offering improved ﬂexi-
bility.
6.3 Embedded GPU Evaluation
For our mobile GPU experiments, we ran our end-to-end
pipeline on a Fireﬂy-RK3399 board equipped with an
ARM Mali-T860MP4 GPU. The baseline was a vendor-
provided library, the ARM Compute Library (v18.03).
As shown in Figure 19, we outperformed the baseline on
three available models for bothfloat16 and float32
(DCGAN and LSTM are not yet supported by the base-
line). The speedup ranged from 1.2× to 1.6×.
6.4 FPGA Accelerator Evaluation
Vanilla Deep Learning Accelerator We now relate
how TVM tackled accelerator-speciﬁc code generation
on a generic inference accelerator design we prototyped
on an FPGA. We used in this evaluation the Vanilla Deep
ﬂoat32 ﬂoat16
ResNet-18
ﬂoat32 ﬂoat16
MobileNet
0.0
50.0
100.0
150.0
200.0
250.0Time (ms)
ﬂoat32 ﬂoat16
DQN
0.0
1.0
2.0
3.0
4.0
5.0
ARMComputeLib TVM w/o graph opt TVM
Figure 19: End-to-end experiment results on Mali-
T860MP4. Two data types, ﬂoat32 and ﬂoat16, were
evaluated.
Learning Accelerator (VDLA) – which distills charac-
teristics from previous accelerator proposals [12, 21, 27]
into a minimalist hardware architecture – to demonstrate
TVM’s ability to generate highly efﬁcient schedules that
can target specialized accelerators. Figure 20 shows the
high-level hardware organization of the VDLA architec-
ture. VDLA is programmed as a tensor processor to
efﬁciently execute operations with high compute inten-
sity (e.g, matrix multiplication, high dimensional con-
volution). It can perform load/store operations to bring
blocked 3-dimensional tensors from DRAM into a con-
tiguous region of SRAM. It also provides specialized on-
chip memories for network parameters, layer inputs (nar-
row data type), and layer outputs (wide data type). Fi-
nally, VDLA provides explicit synchronization control
over successive loads, computes, and stores to maximize
the overlap between memory and compute operations.
Methodology. We implemented the VDLA design on a
low-power PYNQ board that incorporates an ARM Cor-
tex A9 dual core CPU clocked at 667MHz and an Artix-7
based FPGA fabric. On these modest FPGA resources,
we implemented a 16 ×16 matrix-vector unit clocked at
200MHz that performs products of 8-bit values and accu-
mulates them into a 32-bit register every cycle. The the-
oretical peak throughput of this VDLA design is about
102.4GOPS/s. We allocated 32kB of resources for ac-
tivation storage, 32kB for parameter storage, 32kB for
microcode buffers, and 128kB for the register ﬁle. These
on-chip buffers are by no means large enough to provide
sufﬁcient on-chip storage for a single layer of ResNet and
therefore enable a case study on effective memory reuse
and latency hiding.
We built a driver library for VDLA with a C runtime
API that constructs instructions and pushes them to the
target accelerator for execution. Our code generation al-
gorithm then translates the accelerator program to a se-
ries of calls into the runtime API. Adding the specialized
accelerator back-end took ∼2k LoC in Python.
590    13th USENIX Symposium on Operating Systems Design and Implementation USENIX Association
MEMORY LOAD
UNIT
MEMORY STORE
UNIT
DRAM
MICRO-OP SRAM
INPUT MEM
WEIGHT MEM
LOAD BUFFER STORE BUFFER
REGISTER FILE
GEMM
V_ALU
LOAD→EXE Q
EXE→LOAD Q
EXE→STORE Q
STORE→EXE Q
INSTRUCTION FETCH UNIT
LOAD 
CMD Q
COMPUTE 
CMD Q
COMPUTE
STORE 
CMD Q
controller
Figure 20: VDLA Hardware design overview.
End-to-End ResNet Evaluation. We used TVM to
generate ResNet inference kernels on the PYNQ plat-
form and ofﬂoaded as many layers as possible to VDLA.
We also used it to generate both schedules for the CPU
only and CPU+FPGA implementation. Due to its shal-
low convolution depth, the ﬁrst ResNet convolution layer
could not be efﬁciently ofﬂoaded on the FPGA and was
instead computed on the CPU. All other convolution lay-
ers in ResNet, however, were amenable to efﬁcient of-
ﬂoading. Operations like residual layers and activations
were also performed on the CPU since VDLA does not
support these operations.
Figure 21 breaks down ResNet inference time into
CPU-only execution and CPU+FPGA execution. Most
computation was spent on the convolution layers that
could be ofﬂoaded to VDLA. For those convolution lay-
ers, the achieved speedup was 40 ×. Unfortunately, due
to Amdahl’s law, the overall performance of the FPGA
accelerated system was bottlenecked by the sections of
the workload that had to be executed on the CPU. We
envision that extending the VDLA design to support
these other operators will help reduce cost even further.
This FPGA-based experiment showcases TVM’s ability
to adapt to new architectures and the hardware intrinsics
they expose.
7 Related Work
Deep learning frameworks [3, 4, 7, 9] provide convenient
interfaces for users to express DL workloads and deploy
them easily on different hardware back-ends. While ex-
isting frameworks currently depend on vendor-speciﬁc
tensor operator libraries to execute their workloads, they
can leverage TVM’s stack to generate optimized code for
a larger number of hardware devices.
High-level computation graph DSLs are a typical
way to represent and perform high-level optimiza-
tions. Tensorﬂow’s XLA [3] and the recently introduced
DLVM [45] fall into this category. The representations
Figure 21: We ofﬂoaded convolutions in the ResNet
workload to an FPGA-based accelerator. The grayed-out
bars correspond to layers that could not be accelerated
by the FPGA and therefore had to run on the CPU. The
FPGA provided a 40x acceleration on ofﬂoaded convo-
lution layers over the Cortex A9.
of computation graphs in these works are similar, and a
high-level computation graph DSL is also used in this
paper. While graph-level representations are a good ﬁt
for high-level optimizations, they are too high level to
optimize tensor operators under a diverse set of hard-
ware back-ends. Prior work relies on speciﬁc lowering
rules to directly generate low-level LLVM or resorts to
vendor-crafted libraries. These approaches require sig-
niﬁcant engineering effort for each hardware back-end
and operator-variant combination.
Halide [32] introduced the idea of separating comput-
ing and scheduling. We adopt Halide’s insights and reuse
its existing useful scheduling primitives in our compiler.
Our tensor operator scheduling is also related to other
work on DSL for GPUs [18, 24, 36, 37] and polyhedral-
based loop transformation [6,43]. TACO [23] introduces
a generic way to generate sparse tensor operators on
CPU. Weld [30] is a DSL for data processing tasks. We
speciﬁcally focus on solving the new scheduling chal-
lenges of DL workloads for GPUs and specialized accel-
erators. Our new primitives can potentially be adopted
by the optimization pipelines in these works.
High-performance libraries such as ATLAS [46] and
FFTW [15] use auto-tuning to get the best perfor-
mance. Tensor comprehension [42] applied black-box
auto-tuning together with polyhedral optimizations to
optimize CUDA kernels. OpenTuner [5] and existing
hyper parameter-tuning algorithms [26] apply domain-
agnostic search. A predeﬁned cost model is used to
automatically schedule image processing pipelines in
Halide [29]. TVM’s ML model uses effective domain-
aware cost modeling that considers program structure.
The based distributed schedule optimizer scales to a
larger search space and can ﬁnd state-of-the-art kernels
on a large range of supported back-ends. More impor-
tantly, we provide an end-to-end stack that can take de-
scriptions directly from DL frameworks and jointly opti-
mize together with the graph-level stack.
USENIX Association 13th USENIX Symposium on Operating Systems Design and Implementation    591
Despite the emerging popularity of accelerators for
deep learning [11, 21], it remains unclear how a com-
pilation stack can be built to effectively target these de-
vices. The VDLA design used in our evaluation provides
a generic way to summarize the properties of TPU-like
accelerators and enables a concrete case study on how
to compile code for accelerators. Our approach could
potentially beneﬁt existing systems that compile deep
learning to FPGA [34,40], as well. This paper provides a
generic solution to effectively target accelerators via ten-
sorization and compiler-driven latency hiding.
8 Conclusion
We proposed an end-to-end compilation stack to solve
fundamental optimization challenges for deep learning
across a diverse set of hardware back-ends. Our system
includes automated end-to-end optimization, which is
historically a labor-intensive and highly specialized task.
We hope this work will encourage additional studies of
end-to-end compilation approaches and open new op-
portunities for DL system software-hardware co-design
techniques.
Acknowledgement
We would like to thank Ras Bodik, James Bornholt, Xi
Wang, Tom Anderson and Qiao Zhang for their thorough
feedback on earlier versions of this paper. We would also
like to thank members of Sampa, SAMPL and Systems
groups at the Allen School for their feedback on the work
and manuscript. We would like to thank the anonymous
OSDI reviewers, and our shepherd, Ranjita Bhagwan, for
helpful feedbacks. This work was supported in part by a
Google PhD Fellowship for Tianqi Chen, ONR award
#N00014-16-1-2795, NSF under grants CCF-1518703,
CNS-1614717, and CCF-1723352, and gifts from Intel
(under the CAPA program), Oracle, Huawei and anony-
mous sources.
References
[1] NVIDIA Tesla V100 GPU Architecture: The World’s Most Ad-
vanced Data Center GPU, 2017.
[2] A BADI , M., A GARWAL , A., B ARHAM , P., BREVDO , E., C HEN ,
Z., C ITRO , C., C ORRADO , G. S., D AVIS, A., D EAN , J.,
DEVIN , M., G HEMAWAT, S., G OODFELLOW , I., H ARP, A.,
IRVING , G., I SARD , M., J IA, Y., J OZEFOWICZ , R., K AISER ,
L., K UDLUR , M., L EVENBERG , J., M AN ´E, D., M ONGA ,
R., M OORE , S., M URRAY, D., O LAH , C., S CHUSTER , M.,
SHLENS , J., S TEINER , B., S UTSKEVER , I., T ALWAR, K.,
TUCKER , P., V ANHOUCKE , V., VASUDEVAN , V., V I ´EGAS , F.,
VINYALS , O., W ARDEN , P., W ATTENBERG , M., W ICKE , M.,
YU, Y., AND ZHENG , X. TensorFlow: Large-scale machine
learning on heterogeneous systems, 2015. Software available
from tensorﬂow.org.
[3] A BADI , M., B ARHAM , P., C HEN , J., C HEN , Z., D AVIS, A.,
DEAN , J., D EVIN , M., G HEMAWAT, S., I RVING , G., I SARD ,
M., K UDLUR , M., L EVENBERG , J., M ONGA , R., M OORE , S.,
MURRAY, D. G., S TEINER , B., T UCKER , P., VASUDEVAN , V.,
WARDEN , P., W ICKE , M., Y U, Y., AND ZHENG , X. Tensor-
ﬂow: A system for large-scale machine learning. In12th USENIX
Symposium on Operating Systems Design and Implementation
(OSDI 16) (2016), pp. 265–283.
[4] A GARWAL , A., A KCHURIN , E., B ASOGLU , C., C HEN , G.,
CYPHERS , S., D ROPPO , J., E VERSOLE , A., G UENTER , B.,
HILLEBRAND , M., H OENS , R., H UANG , X., H UANG , Z.,
IVANOV, V., K AMENEV , A., K RANEN , P., K UCHAIEV , O.,
MANOUSEK , W., M AY, A., M ITRA , B., N ANO , O., N AVARRO,
G., O RLOV, A., P ADMILAC , M., P ARTHASARATHI , H., P ENG ,
B., R EZNICHENKO , A., S EIDE , F., S ELTZER , M. L., S LANEY ,
M., S TOLCKE , A., W ANG , Y., W ANG , H., Y AO, K., Y U, D.,
ZHANG , Y., AND ZWEIG , G. An introduction to computational
networks and the computational network toolkit. Tech. Rep.
MSR-TR-2014-112, August 2014.
[5] A NSEL , J., K AMIL , S., V EERAMACHANENI , K., R AGAN -
KELLEY , J., B OSBOOM , J., O’R EILLY, U.-M., AND AMARAS -
INGHE , S. Opentuner: An extensible framework for program au-
totuning. In International Conference on Parallel Architectures
and Compilation Techniques (Edmonton, Canada, August 2014).
[6] B AGHDADI , R., B EAUGNON , U., C OHEN , A., G ROSSER , T.,
KRUSE , M., R EDDY, C., V ERDOOLAEGE , S., B ETTS , A.,
DONALDSON , A. F., K ETEMA , J., A BSAR , J., H AASTREGT ,
S. V., K RAVETS , A., L OKHMOTOV , A., D AVID, R., AND HA-
JIYEV , E. Pencil: A platform-neutral compute intermediate lan-
guage for accelerator programming. In Proceedings of the 2015
International Conference on Parallel Architecture and Compila-
tion (PACT) (Washington, DC, USA, 2015), PACT ’15, IEEE
Computer Society, pp. 138–149.
[7] B ASTIEN , F., L AMBLIN , P., P ASCANU , R., B ERGSTRA , J.,
GOODFELLOW , I. J., B ERGERON , A., B OUCHARD , N., AND
BENGIO , Y. Theano: new features and speed improvements.
Deep Learning and Unsupervised Feature Learning NIPS 2012
Workshop, 2012.
[8] C HEN , T., AND GUESTRIN , C. Xgboost: A scalable tree boost-
ing system. In Proceedings of the 22Nd ACM SIGKDD Inter-
national Conference on Knowledge Discovery and Data Mining
(New York, NY , USA, 2016), KDD ’16, ACM, pp. 785–794.
[9] C HEN , T., L I, M., L I, Y., L IN, M., W ANG , N., W ANG , M.,
XIAO , T., X U, B., Z HANG , C., , AND ZHANG , Z. MXNet:
A ﬂexible and efﬁcient machine learning library for heteroge-
neous distributed systems. InNeural Information Processing Sys-
tems, Workshop on Machine Learning Systems (LearningSys’15)
(2015).
[10] C HEN , T.-F., AND BAER , J.-L. Effective hardware-based data
prefetching for high-performance processors. IEEE Transactions
on Computers 44, 5 (May 1995), 609–623.
[11] C HEN , Y., LUO, T., LIU, S., Z HANG , S., H E, L., WANG , J., L I,
L., C HEN , T., X U, Z., S UN, N., AND TEMAM , O. Dadiannao:
A machine-learning supercomputer. In Proceedings of the 47th
Annual IEEE/ACM International Symposium on Microarchitec-
ture (Washington, DC, USA, 2014), MICRO-47, IEEE Computer
Society, pp. 609–622.
[12] C HEN , Y.-H., E MER , J., AND SZE, V. Eyeriss: A spatial ar-
chitecture for energy-efﬁcient dataﬂow for convolutional neural
networks. In Proceedings of the 43rd International Symposium
on Computer Architecture (Piscataway, NJ, USA, 2016), ISCA
’16, IEEE Press, pp. 367–379.
[13] C OURBARIAUX , M., B ENGIO , Y., AND DAVID, J. Binarycon-
nect: Training deep neural networks with binary weights during
propagations. CoRR abs/1511.00363 (2015).
592    13th USENIX Symposium on Operating Systems Design and Implementation USENIX Association
[14] E GGERS , S. J., E MER , J. S., L EVY, H. M., L O, J. L., S TAMM ,
R. L., AND TULLSEN , D. M. Simultaneous multithreading: a
platform for next-generation processors. IEEE Micro 17, 5 (Sept
1997), 12–19.
[15] F RIGO , M., AND JOHNSON , S. G. Fftw: an adaptive software ar-
chitecture for the fft. InAcoustics, Speech and Signal Processing,
1998. Proceedings of the 1998 IEEE International Conference on
(May 1998), vol. 3, pp. 1381–1384 vol.3.
[16] H E, K., Z HANG , X., R EN, S., AND SUN, J. Identity mappings
in deep residual networks. arXiv preprint arXiv:1603.05027
(2016).
[17] H EGARTY , J., B RUNHAVER , J., D EVITO , Z., R AGAN -KELLEY ,
J., C OHEN , N., B ELL , S., VASILYEV , A., H OROWITZ , M., AND
HANRAHAN , P. Darkroom: Compiling high-level image pro-
cessing code into hardware pipelines. ACM Trans. Graph. 33, 4
(July 2014), 144:1–144:11.
[18] H ENRIKSEN , T., S ERUP, N. G. W., E LSMAN , M., H ENGLEIN ,
F., AND OANCEA , C. E. Futhark: Purely functional gpu-
programming with nested parallelism and in-place array updates.
In Proceedings of the 38th ACM SIGPLAN Conference on Pro-
gramming Language Design and Implementation (New York,
NY , USA, 2017), PLDI 2017, ACM, pp. 556–571.
[19] H OWARD , A. G., Z HU, M., C HEN , B., K ALENICHENKO , D.,
WANG , W., W EYAND , T., A NDREETTO , M., AND ADAM , H.
Mobilenets: Efﬁcient convolutional neural networks for mobile
vision applications. CoRR abs/1704.04861 (2017).
[20] J OUPPI , N. P. Improving direct-mapped cache performance
by the addition of a small fully-associative cache and prefetch
buffers. In [1990] Proceedings. The 17th Annual International
Symposium on Computer Architecture (May 1990), pp. 364–373.
[21] J OUPPI , N. P., Y OUNG , C., P ATIL , N., P ATTERSON , D.,
AGRAWAL , G., B AJWA , R., B ATES , S., B HATIA , S., B ODEN ,
N., B ORCHERS , A., B OYLE , R., C ANTIN , P.- L., C HAO , C.,
CLARK , C., C ORIELL , J., D ALEY, M., D AU, M., D EAN , J.,
GELB , B., G HAEMMAGHAMI , T. V., G OTTIPATI , R., G UL-
LAND , W., H AGMANN , R., H O, C. R., H OGBERG , D., H U,
J., H UNDT , R., H URT, D., I BARZ , J., J AFFEY, A., J AWORSKI ,
A., K APLAN , A., K HAITAN , H., K ILLEBREW , D., K OCH , A.,
KUMAR , N., L ACY, S., L AUDON , J., L AW, J., L E, D., L EARY,
C., L IU, Z., L UCKE , K., L UNDIN , A., M ACKEAN , G., M AG-
GIORE , A., M AHONY , M., M ILLER , K., N AGARAJAN , R.,
NARAYANASWAMI , R., N I, R., N IX, K., N ORRIE , T., O MER -
NICK , M., P ENUKONDA , N., P HELPS , A., R OSS , J., R OSS , M.,
SALEK , A., S AMADIANI , E., S EVERN , C., S IZIKOV , G., S NEL -
HAM , M., S OUTER , J., S TEINBERG , D., S WING , A., T AN, M.,
THORSON , G., T IAN , B., T OMA , H., T UTTLE , E., V ASUDE -
VAN, V., W ALTER , R., W ANG , W., W ILCOX , E., AND YOON ,
D. H. In-datacenter performance analysis of a tensor processing
unit. In Proceedings of the 44th Annual International Symposium
on Computer Architecture (New York, NY , USA, 2017), ISCA
’17, ACM, pp. 1–12.
[22] K IRKPATRICK , S., G ELATT, C. D., AND VECCHI , M. P. Op-
timization by simulated annealing. Science 220 , 4598 (1983),
671–680.
[23] K JOLSTAD , F., K AMIL , S., C HOU , S., L UGATO , D., AND
AMARASINGHE , S. The tensor algebra compiler. Proc. ACM
Program. Lang. 1, OOPSLA (Oct. 2017), 77:1–77:29.
[24] K L ¨OCKNER , A. Loo.py: transformation-based code generation
for GPUs and CPUs. In Proceedings of ARRAY ‘14: ACM SIG-
PLAN Workshop on Libraries, Languages, and Compilers for Ar-
ray Programming (Edinburgh, Scotland., 2014), Association for
Computing Machinery.
[25] L AVIN, A., AND GRAY, S. Fast algorithms for convolutional
neural networks. In 2016 IEEE Conference on Computer Vision
and Pattern Recognition, CVPR 2016, Las Vegas, NV , USA, June
27-30, 2016 (2016), pp. 4013–4021.
[26] L I, L., J AMIESON , K. G., D ESALVO, G., R OSTAMIZADEH ,
A., AND TALWALKAR , A. Efﬁcient hyperparameter optimiza-
tion and inﬁnitely many armed bandits. CoRR abs/1603.06560
(2016).
[27] L IU, D., C HEN , T., L IU, S., Z HOU , J., Z HOU , S., T EMAN , O.,
FENG , X., Z HOU , X., AND CHEN , Y. Pudiannao: A polyvalent
machine learning accelerator. In Proceedings of the Twentieth
International Conference on Architectural Support for Program-
ming Languages and Operating Systems (New York, NY , USA,
2015), ASPLOS ’15, ACM, pp. 369–381.
[28] M NIH , V., K AVUKCUOGLU , K., S ILVER , D., R USU , A. A.,
VENESS , J., B ELLEMARE , M. G., G RAVES , A., R IEDMILLER ,
M., F IDJELAND , A. K., O STROVSKI , G., ET AL . Human-level
control through deep reinforcement learning. Nature 518, 7540
(2015), 529.
[29] M ULLAPUDI , R. T., A DAMS , A., S HARLET , D., R AGAN -
KELLEY , J., AND FATAHALIAN , K. Automatically scheduling
halide image processing pipelines. ACM Trans. Graph. 35 , 4
(July 2016), 83:1–83:11.
[30] P ALKAR , S., T HOMAS , J. J., N ARAYANAN , D., S HANBHAG ,
A., P ALAMUTTAM , R., P IRK , H., S CHWARZKOPF , M., A MA-
RASINGHE , S. P., M ADDEN , S., AND ZAHARIA , M. Weld: Re-
thinking the interface between data-intensive applications. CoRR
abs/1709.06416 (2017).
[31] R ADFORD , A., M ETZ , L., AND CHINTALA , S. Unsupervised
representation learning with deep convolutional generative adver-
sarial networks. arXiv preprint arXiv:1511.06434 (2015).
[32] R AGAN -KELLEY , J., B ARNES , C., A DAMS , A., PARIS , S., D U-
RAND , F., AND AMARASINGHE , S. Halide: A language and
compiler for optimizing parallelism, locality, and recomputation
in image processing pipelines. In Proceedings of the 34th ACM
SIGPLAN Conference on Programming Language Design and
Implementation (New York, NY , USA, 2013), PLDI ’13, ACM,
pp. 519–530.
[33] R ASTEGARI , M., O RDONEZ , V., R EDMON , J., AND FARHADI ,
A. Xnor-net: Imagenet classiﬁcation using binary convolutional
neural networks. In European Conference on Computer Vision
(2016), Springer, pp. 525–542.
[34] S HARMA , H., P ARK , J., M AHAJAN , D., A MARO , E., K IM,
J. K., S HAO , C., M ISHRA , A., AND ESMAEILZADEH , H. From
high-level deep neural models to fpgas. InMicroarchitecture (MI-
CRO), 2016 49th Annual IEEE/ACM International Symposium on
(2016), IEEE, pp. 1–12.
[35] S MITH , J. E. Decoupled access/execute computer architectures.
In Proceedings of the 9th Annual Symposium on Computer Archi-
tecture (Los Alamitos, CA, USA, 1982), ISCA ’82, IEEE Com-
puter Society Press, pp. 112–119.
[36] S TEUWER , M., R EMMELG , T., AND DUBACH , C. Lift: A func-
tional data-parallel ir for high-performance gpu code generation.
In Proceedings of the 2017 International Symposium on Code
Generation and Optimization(Piscataway, NJ, USA, 2017), CGO
’17, IEEE Press, pp. 74–85.
[37] S UJEETH , A. K., L EE, H., B ROWN , K. J., C HAFI , H., W U, M.,
ATREYA , A. R., O LUKOTUN , K., R OMPF , T., AND ODERSKY ,
M. Optiml: An implicitly parallel domain-speciﬁc language for
machine learning. In Proceedings of the 28th International Con-
ference on International Conference on Machine Learning(USA,
2011), ICML’11, pp. 609–616.
[38] T AI, K. S., S OCHER , R., AND MANNING , C. D. Improved
semantic representations from tree-structured long short-term
memory networks. arXiv preprint arXiv:1503.00075 (2015).
USENIX Association 13th USENIX Symposium on Operating Systems Design and Implementation    593
[39] T ULLOCH , A., AND JIA, Y. High performance ultra-low-
precision convolutions on mobile devices. arXiv preprint
arXiv:1712.02427 (2017).
[40] U MUROGLU , Y., F RASER , N. J., G AMBARDELLA , G., B LOTT,
M., L EONG , P. H. W., JAHRE , M., AND VISSERS , K. A. FINN:
A framework for fast, scalable binarized neural network infer-
ence. CoRR abs/1612.07119 (2016).
[41] V ASILACHE , N. personal communication.
[42] V ASILACHE , N., Z INENKO , O., T HEODORIDIS , T., G OYAL, P.,
DEVITO , Z., M OSES , W. S., V ERDOOLAEGE , S., A DAMS ,
A., AND COHEN , A. Tensor comprehensions: Framework-
agnostic high-performance machine learning abstractions. CoRR
abs/1802.04730 (2018).
[43] V ERDOOLAEGE , S., C ARLOS JUEGA , J., C OHEN , A., I GNA -
CIO G ´OMEZ , J., T ENLLADO , C., AND CATTHOOR , F. Polyhe-
dral parallel code generation for cuda. ACM Trans. Archit. Code
Optim. 9, 4 (Jan. 2013), 54:1–54:23.
[44] V OLKOV, V. Understanding Latency Hiding on GPUs . PhD
thesis, University of California at Berkeley, 2016.
[45] W EI, R., A DVE , V., AND SCHWARTZ , L. Dlvm: A mod-
ern compiler infrastructure for deep learning systems. CoRR
abs/1711.03016 (2017).
[46] W HALEY , R. C., AND DONGARRA , J. J. Automatically tuned
linear algebra software. In Proceedings of the 1998 ACM/IEEE
Conference on Supercomputing (Washington, DC, USA, 1998),
SC ’98, IEEE Computer Society, pp. 1–27.
[47] W ILLIAMS , S., W ATERMAN , A., AND PATTERSON , D.
Rooﬂine: An insightful visual performance model for multicore
architectures. Commun. ACM 52, 4 (Apr. 2009), 65–76.
[48] Z AREMBA , W., S UTSKEVER , I., AND VINYALS , O. Recurrent
neural network regularization. arXiv preprint arXiv:1409.2329
(2014).
594    13th USENIX Symposium on Operating Systems Design and Implementation USENIX Association

Iterative Optimization in the Polyhedral Model:
Part II, Multidimensional Time
Louis-No¨el Pouchet1,3, C ´edric Bastoul1,3, Albert Cohen 1,3 and John Cavazos 2,3
1 ALCHEMY Group, INRIA Futurs and Paris-Sud University
{ﬁrstname.lastname}@inria.fr
2 Dept. of Computer & Information Science, University of Delaware
cavazos@cis.udel.edu
3 High Performance for Embedded Architectures and Compilers Network of Excellence members
Abstract
High-level loop optimizations are necessary to achieve good perfor-
mance over a wide variety of processors. Their performance impact
can be signiﬁcant because they involve in-depth program transfor-
mations that aiming to sustain a balanced workload over the com-
putational, storage, and communication resources of the target ar-
chitecture. Therefore, it is mandatory that the compiler accurately
models the target architecture and the effects of complex code re-
structuring.
However, because optimizing compilers (1) use simplistic per-
formance models that abstract away many of the complexities of
modern architectures, (2) rely on inaccurate dependence analysis,
and (3) lack frameworks to express complex interactions of trans-
formation sequences, they typically uncover only a fraction of the
peak performance available on many applications. We propose a
complete iterative framework to address these issues. We rely on
the polyhedral model to construct and traverse a large and expres-
sive search space. This space encompasses only legal, distinct ver-
sions resulting from the restructuring of any static control loop nest.
We ﬁrst propose a feedback-driven iterative heuristic tailored
to the search space properties of the polyhedral model. Though, it
quickly converges to good solutions for small kernels, larger bench-
marks containing higher dimensional spaces are more challenging
and our heuristic misses opportunities for signiﬁcant performance
improvement. Thus, we introduce the use of a genetic algorithm
with specialized operators that leverage the polyhedral representa-
tion of program dependences. We provide experimental evidence
that the genetic algorithm effectively traverses huge optimization
spaces, achieving good performance improvements on large loop
nests.
1. Introduction
In recent years, feedback-directed iterative optimization has be-
come a promising direction to harness the full potential of future
and emerging processors with modern compilers. Building on op-
eration research, statistical analysis and artiﬁcial intelligence, it-
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proﬁt or commercial advantage and that copies bear this notice and the full citation
on the ﬁrst page. To copy otherwise, to republish, to post on servers or to redistribute
to lists, requires prior speciﬁc permission and/or a fee.
Copyright c⃝ ACM [to be supplied]. . . $5.00
erative optimization generalizes proﬁle-directed approach to inte-
grate precise feedback from the runtime behavior of the program
into optimization algorithms. Through the many encouraging re-
sults that have been published in this area, it has become apparent
that achieving better performance with iterative techniques depends
on two major challenges.
Search space expressiveness. To achieve strong and portable per-
formance with iterative techniques, it is essential that transfor-
mation search space be expressive enough to let the optimiza-
tions target all important architecture components and address
all dominant performance anomalies,
Search space traversal. It is also important to construct search
algorithms (analytical, statistical, empirical) and acceleration
heuristics (performance models, machine learning) that effec-
tively traverse the search space by exploiting its static and dy-
namic characteristics.
This paper targets the optimization problem of selecting an
afﬁne transformation to optimize imperfectly nested loops. Gir-
bal et al. show that complex sequences of loop transformations are
needed to generate efﬁcient code for full-size loop nests on mod-
ern architectures [19]. They also show that such transformation se-
quences are out of reach of classical loop optimization frameworks,
although multidimensional afﬁne scheduling [18, 15] can success-
fully model them as one single optimization step [47], and scales to
large loop nests with hundreds of array references [19]. Within this
space of complex sequences of loop transformations, our work is
the ﬁrst to simultaneously address the two above-mentioned chal-
lenges.
We make the following contributions.
1. By considering multidimensional schedules, we tackle any pos-
sible static-control loop nest in a program, a major leap for-
ward from the state-of-the-art iterative optimization on one-
dimensional schedules [35]. We simultaneously extend the ap-
plication domain — applicability and scalability to larger codes
— and search space expressiveness — construction of more
complex sequences of transformations.
2. To harness the combinatorial explosion of the optimization
problem, we present a scalable traversal heuristic and origi-
nal genetic operators, tailored to efﬁciently traverse a space
of legal, distinct multidimensional schedules. These operators
leverage the algebraic structure and statistical properties of the
space.
3. We simultaneously demonstrate good performance gains and
excellent convergence speed on huge search spaces, even on
larger loop nests (up to 20 loops) where iterative afﬁne schedul-
ing has never been attempted before.
4. We demonstrate signiﬁcant performance improvements on
three different architectures.
The paper is structured as follows. Section 2 introduces multi-
dimensional scheduling in the polyhedral model. Section 3 con-
structs the search space of legal, distinct versions (multidimen-
sional schedules) for a program, and the key properties of this
space. Section 4 proposes a ﬁrst heuristic to efﬁciently traverse this
space in the case of small kernels. Section 5 deﬁnes a genetic algo-
rithm with specialized mutation and reproduction operators that can
effectively traverse huge search spaces associated with larger loop
nests. We show that our custom genetic algorithm achieves good
performance improvements despite the poor statistical distribution
of performance-enhancing schedules. Section 6 discusses related
work, before we conclude in Section 7.
2. Thinking in Polyhedra
Most compilers’ internal representations match the inductive se-
mantics of imperative programs (syntax tree, call tree, control-ﬂow
graph, SSA). In such reduced representations of the dynamic exe-
cution trace, a statement of a high-level program occurs only once,
even if it is executed many times (e.g., when enclosed within a
loop). Representing a program this way is not convenient for ag-
gressive optimizations which often need to consider a representa-
tion granularity at the level of dynamic statement instances . For
example, complex transformations like loop interchange, fusion or
tiling operate on the execution order of statement instances [48].
Due to compilation-time constraints and to the lack of an adequate
algebraic representation of the semantics of loop nests, classical
(non-iterative) compilers are unable to adapt the schedule of state-
ment instances of a program to best exploit the architecture re-
sources. For example, compilers can typically not apply any trans-
formation if data dependences are non-uniform (unimodular trans-
formations, tiling), if the loop trip counts differ (fusion) or simply
because proﬁtability is too unpredictable. As a simple illustration,
consider the Ring-Roberts edge detection ﬁlter shown in Figure 1.
While it is straightforward to detect a high level of data reuse be-
tween the two loop nests, none of the compilers we considered —
Open64 4.0, ICC 10.0, PathScale 3.0, GCC 4.2.0 — were able to
apply loop fusion for a potentially 50% cache miss reduction when
arrays do not ﬁt in the data cache (plus additional scalar promotion
and instruction-level-parallelism improvements). Indeed, this ap-
parently simple transformation actually requires a non-trivial com-
position of (two-dimensional) loop shifting, fusion and peeling.
/* Ring blur filter */
for (i=1;i<length-1;i++)
for (j=1;j<width-1;j++)
R Ring[i][j]=(Img[i-1][j-1]+Img[i-1][j]+Img[i-1][j+1]+
Img[i][j+1] + Img[i][j-1] +
Img[i+1][j-1]+Img[i+1][j]+Img[i+1][j+1])/8;
/* Roberts edge detection filter */
for (i=1;i<length-2;i++)
for (j=2;j<width-1;j++)
P Img[i][j]=abs(Ring[i][j]-Ring[i+1][j-1])+
abs(Ring[i+1][j]-Ring[i][j-1]);
Figure 1. Ring-Roberts edge detection for noisy images
To build complex loop transformations, a well known alterna-
tive is to represent programs in the polyhedral model . It is a ﬂexi-
ble and expressive representation for loop nests with statically pre-
dictable control ﬂow. The polyhedral model captures control-ﬂow
and data-ﬂow with three linear algebraic structures, described in
the following subsections. Such loop nests amenable to algebraic
representation are called static control parts (SCoP) [16, 19].
2.1 Iteration Domains
Iteration domains capture the dynamic instances of all statements
— all possible values of surrounding loop iterators — through a
set of afﬁne inequalities. For example, statement R in Figure 1 is
executed for every value of the pair of surrounding loop counters,
called the iteration vector : the iteration vector of statement R is
⃗xR = (i, j). Hence, the iteration domain of R is deﬁned by its
enclosing loop bounds:
DR = {i, j |1 ≤i ≤length −1 ∧1 ≤j ≤width −1},
which forms a parametric polyhedron (a space bounded by inequal-
ities, a.k.a. hyperplanes or faces). Each integral point inside this
polyhedron corresponds to exactly one execution of statement R.
This model allows the compiler to manipulate statement execution
and iteration ordering at the most precise level.
2.2 Subscript Functions
Subscript functions capture the data locations on which statement
operate. In static control parts, memory accesses are performed
through array references (a variable being a particular case of an
array). We restrict ourselves to subscripts of the form of afﬁne
expressions which may depend on surrounding loop counters (e.g.,
i and j for statement R) and global parameters (e.g., length and
width). Each subscript function is linked to an array that represents
a read or a write access. For instance, the subscript function for
the read reference Img[i-1][j] of statement R is simply f (i, j) =
(i −1, j).
2.3 Multidimensional Schedules
Iteration domains deﬁne exactly the set of dynamic instances for
each statement. However this algebraic structure does not describe
the order in which each instance has to be executed with respect
to each other. Of course, we do not want to rely on the inductive
semantics of the sequence and loop iteration for this purpose, as it
would break the algebraic reasoning about loop nests.
A convenient way to express the execution order is to give each
instance an execution date. It is obviously impractical to deﬁne
them one by one since instance number may be either very large
or unknown at compile time. An appropriate solution is to do it at
the statement level by deﬁning, for each statement, a scheduling
function that speciﬁes the execution date for each instance of a
corresponding statement. For tractability reasons, we restrict these
functions to be afﬁne (relaxation of this constraint may exists [6],
but challenges the code generation step [7]).
This work deals with multidimensional schedules: given a state-
ment S, it is an afﬁne form on the outer loop iterators ⃗xS and the
global parameters ⃗n. It is written
θS(⃗xS) =TS


⃗xS
⃗n
1


where TS is a matrix of constants (possibly not integers). Multi-
dimensional dates can be seen as clocks: the ﬁrst dimension cor-
responds to days (most signiﬁcant), next one is hours (less sig-
niﬁcant), the third to minutes, and so on. Unlike one-dimensional
afﬁne schedules, every static control program has a multidimen-
sional afﬁne schedule [18]. Hence the application domain of the
present work extends to all static control parts in general programs.
For a concrete intuition of scheduling functions, lets go back
to Ring-Roberts example of Figure 1. A possible multidimensional
schedule is: θR(i, j) = (i, j) and θP(i, j) = (i + length −2, j). This
means the schedule of statement R orders its instances according
to i ﬁrst and then j. This matches the structure of the ﬁrst loop
nest. This is similar for statement P, except for the offset on the
ﬁrst time-dimension which states that the ﬁrst nest runs before the
second one: while the largest value of the ﬁrst time-dimension for
R is length −2, the smallest value of the ﬁrst dimension of P is
length −1. Hence the loop surrounding P “starts” after the loop
surrounding R. Efﬁcient algorithms and tools exist to generate a
target code from a polyhedral representation with multidimensional
afﬁne schedules [38, 7]. Recent work by Girbal et al. [19] and
Vasilache et al. [43] improved these algorithms to scale up to
thousands of statements.
2.4 Beneﬁts of a Polyhedral Representation
Reasoning about programs in a polyhedral representation has many
advantages, for both program analysis and transformation.
1. exact dependence analysis is possible [16, 36, 30];
2. there exist efﬁcient algorithms and tools to regenerate impera-
tive code [38, 7];
3. loop transformation sequences of arbitrary complexity can be
constructed and transparently applied in one single step.
A more complete description of static control parts was given by
Xue [49] and their applicability to compute intensive, scientiﬁc or
embedded applications have been extensively discussed by Girbal
et al. [19, 32]. Frameworks to highlight SCoPs in general programs
and to extract both iteration domains and subscript functions al-
ready exist or are in active development in compiler platforms like
WRAP-IT/URUK for Open64 [19], and Graphite for GCC [33].
Multidimensional afﬁne schedules support arbitrary complex
compositions of a wide range of program transformations. Several
frameworks have been designed to facilitate the expression of such
transformations [22], or to enable their composition and semi-
automatic construction [19, 44]. As illustration, a trivial loop fusion
is not possible to improve data locality on the Ring-Roberts kernel
in Figure 1. Because of both data dependences and non-matching
loop bounds, only a partial loop fusion is possible, which translates
into a sequence of, e.g., fusion, shifting and index-set splitting [48].
Using multidimensional schedules, a correct transformation (found
using chunking [8]) is simply: θR(i, j) = (i, j) and θP(i, j) = (i +
2, j). One may care to check, using any polyhedral code generator,
that the corresponding target code corresponds to a quite complex
composition of syntactic transformations.
3. Generating Program Versions
The space of multidimensional afﬁne schedules is very expressive.
Each point in the space corresponds to potentially very different
program versions, exposing a wide spectrum of interactions be-
tween architectural components and back-end compiler optimiza-
tions. This section presents the formal construction of the space of
legal, distinct schedules only . We also give a practical heuristic to
reduce the combinatorics of any algorithm to traverse this space
while preserving the legality property.
3.1 Generating All Legal Schedules
Nisbet [31], then Long and Fursin [28] noticed that choosing a
schedule at random likely to lead to an illegal program version,
and that the probability of ﬁnding a legal one (which do not alter
semantics) decreases exponentially with program size.
This challenge can only be tackled when integrating data de-
pendence information into the construction of the search space.
Two statement instances are in dependence if they access the same
memory location and at least one of these accesses is a write. Main-
taining the relative order of such instances is a sufﬁcient condition
to preserve the original program semantics [9].
Dependences in static control parts can be expressed by depen-
dence polyhedra whose formal description has been proposed by
Feautrier [16]. A dependence polyhedron DR,S is a subset of the
Cartesian product of the iteration domains of statements R and S.
Each integral point of a dependence polyhedron corresponds to a
pair of instances of the statements in dependence. Thus, a schedule
does not change the semantics of the original program if it satisﬁes
the precedence constraint : for all pairs of iteration vectors ⃗xR and
⃗xS in all dependence polyhedra,
θR(⃗xR) ≺θS(⃗xS),
where ≺denotes the lexicographic ordering.1
The afﬁne form of Farkas lemma allows to translate such con-
straints into an afﬁne equivalent [40]. Feautrier used this result to
express every constraint that a one-dimensional schedule must re-
spect to preserve the semantics of the original program [17]. Those
constraints bound a space where each integral point corresponds to
a legal schedule. Pouchet et al. showed it is possible to traverse this
space efﬁciently for small programs that accept one-dimensional
schedules [35]. But dealing with multidimensional schedules leads
to a combinatorial explosion.
Using one-dimensional schedules, all dependences have to be
satisﬁed within a single time dimension: the precedence constraint
is simply θR(⃗xR) < θS(⃗xS) and θ is a row vector. In multidimen-
sional schedules, the legality constraints can also be built time di-
mension per time dimension, with the difference that a dependence
needs to be weakly solved — θS(⃗xS) −θR(⃗xR) ⪰0 — for the ﬁrst
time dimensions until it is strongly solved — θS(⃗xS) −θR(⃗xR) > 0
— at a given time dimension d. Once a dependence has been
strongly solved, no additional constraint is required for legality at
dimensions d′> d. Reciprocally, a dependence must be weakly
solved for all d′′< d. There is freedom to decide at which time
dimension a dependence will be strongly solved. Each possible de-
cision leads to a potentially different search space. Furthermore, it
is possible to arbitrarily increase the number of time dimensions of
the schedule, resulting in an inﬁnite set of scenarios in general.
The ouptut is in the form of a list of polyhedra of legal sched-
ules, one for each time dimension.
A naive solution to build (a representative subset of) all multidi-
mensional schedules would be to restrict the possible scenarios by
setting an upper bound on the number of time dimensions (e.g., the
loop nest depth + 1), which is already unrealistic for programs of
more than a few dependences.
Note that loop tiling (a.k.a. blocking) is not directly expressible
on multidimensional schedules. It requires modiﬁcations of the it-
eration domain (insertion of new dimensions) [2, 19] or speciﬁc
handling in the code generator [39]. Because of this, our search
space does not currently encompass loop tiling. Recent results by
Renganarayanan et al. and Bondhugula et al. are promising direc-
tions towards fully integrating loop tiling with afﬁne scheduling
algorithms [39, 11].
3.2 Building a Practical Search Space
We have to face a double combinatorial problem. First, there are too
many polytopes to be considered. For instance, the Ring-Roberts
ﬁlter shown in Figure 1 has 12 dependence polyhedra, from which
follows a huge number of possible strongly/weakly solved depen-
dence scenarios. Second, one needs to limit the search to bounded
1 (a1,..., an) ≺(b1,..., bm) iff there exists an integer 1 ≤i ≤min(n,m) s.t.
(a1,..., ai−1) = (b1,..., bi−1) and ai < bi.
polytopes. Yet, even the smallest bound leads to polytopes that are
too large to be explored exhaustively for complex loop nests.
Feautrier found a systematic solution to the explosion of the
number of polyhedra: he considers a space of legal schedules lead-
ing to maximum ﬁne-grain parallelism [18, 45]. To achieve this, a
greedy algorithm maximizes the number of dependences solved for
a given dimension. While this solution is interesting because it re-
duces the number of dimensions and exhibits inner parallelism, it
is not practical enough for several reasons. First, it needs to solve
a system of linear inequalities involving every schedule coefﬁcient
plus a decision variable per dependence [18]. This makes the prob-
lem untractable for all but small kernels. Moreover, minimizing
the number of dimensions often translates into big schedule coefﬁ-
cients; these generally lead to algorithmic complexity and control
overhead after generation of the target imperative code [22].
We suggest a simple variation to overcome those issues. The
following algorithm sketches our search space construction for a
given static control part:
1. Compute the exact set G of dependences for the SCoP through
instancewise analysis [16]
2. d ←1
3. while G ̸= /0 do
(a) Initialize Ld (the space of legal schedules for time dimen-
sion d) to the full-space polyhedron
(b) for each dependence DR,S ∈G
•Compute WDR,S — the space of legal schedules weakly
satisfying only DR,S — by enforcing, for all pairs of
points in DR,S:
θS(⃗xs) −θR(⃗xR) ≥0
•Ld ←Ld ∩WDR,S
(c) for each dependence DR,S ∈G
•Compute SDR,S — the space of legal schedules strongly
satisfying only DR,S — by enforcing, for all pairs of
points in DR,S:
θS(⃗xs) −θR(⃗xR) > 0
•if Ld ∩SDR,S ̸= /0 then
Ld ←Ld ∩SDR,S
G ←G −DR,S
(d) d ←d + 1
This heuristic outputs for each schedule dimension d a space Ld
of legal solutions.
The algorithm terminates; the proof uses the same argument as
Feautrier’s muldidimensional scheduling algorithm [45]: at least
one dependence can be strongly solved per time dimension d. Nev-
ertheless, it differs from Feautrier’s algorithm as it does not guar-
antee a maximal number of dependences solved per dimension.
Therefore it may not minimize the number of dimensions of the
schedule: this is not an issue as we only consider sequential codes. 2
However, this algorithm is efﬁcient and only needs one polyhe-
dron emptiness test per dependence, 3 and the elimination of Farkas
multipliers used to enforce the precedence constraint on sched-
ule coefﬁcients is performed dependence per dependence (i.e., on
very small systems) [35]. Since we consider sequential codes only,
we can bound the coefﬁcient values within {−1,0,1}to minimize
control-ﬂow overhead. This would have been very restrictive if we
2 Afﬁne partitioning may be better suited to characterize parallelism in the
polyhedral model [27].
3 Over Ld which contains exactly one variable per schedule coefﬁcient.
were constrained to one-dimensional schedules. In the multidimen-
sional case, although it eliminates some schedules from the space
(e.g., non-unit skewing), these bounds are compatible with the ex-
pression of arbitrary compositions of loop fusion, distribution, in-
terchange, code motion; in the worst case, it translates into addi-
tional time dimensions. Overall, this solution gives an interesting
tradeoff between scalability and expressiveness (performance of
the generated code).
So far, we did not deﬁne the order in which dependences are
considered when checking against strong satisfaction. This order
can have a dramatic impact on the constructed space. A long term
approach would be to consider this order as part of the search space,
but this is not currently practical (combinatorial explosion). In-
stead, we use two analytical criteria to order the dependences. First
of all, each dependence is assigned a priority, depending on the
memory trafﬁc generated by the pair of statements in dependence.
We use a simpliﬁed version of the model by Bastoul and Feautrier
[8]: for each array A and dimension d, we approximate the traf-
ﬁc as mrA
d , where md is the size of the dth dimension of the array,
and rA is the rank of the concatenation of the subscript matrices of
all references to dimension d of array A in the statement. Thus the
generated trafﬁc evaluation for a given statement is a multivariate
polynomial in the parametric sizes of all arrays. We use proﬁling to
instantiate these size parameters. Intuitively, maximizing the depth
where a dependence is strongly solved maximizes reuse in inner
loops and minimizes the memory trafﬁc in outer loops. Therefore,
we start with dependences involved in the statements with the less
trafﬁc. Our second criterion is based on dependence interference ;
it is used in case of non-discriminating priorities resulting from the
ﬁrst criterion. Two dependences interfere if it is impossible to build
a one-dimensional schedule strongly satisfying these two depen-
dences. We ﬁrst try to solve dependences interfering with the lower
number of other dependences, maximizing our chance to strongly
solve more dependences within the current time dimension.
3.3 Scanning the Search Space Polytopes
The algorithm presented in Section 3.2 constructs one polytope
per dimension of the schedule. Picking one point in every poly-
tope fully describes one multidimensional schedule, hence one pro-
gram version: the generated imperative codes will be distinct if the
scheduling matrices are distinct. This is reminiscent of the clas-
sical polyhedron scanning problem [4, 38, 7]; however, none of
the existing algorithms scale to the hundreds of dimensions we are
considering. Fortunately, our problem happens to be simpler than
“static” loop nest generation: we only need to “dynamically” enu-
merate every integral point which respects the set of constraints.
Each program version is represented by a unique scheduling
matrix Θ. The ﬁrst columns are schedule coefﬁcients associated
with each loop iterator surrounding a statement in the original pro-
gram, for all statements ( ⃗i). The next set of columns are sched-
ule coefﬁcients associated with global parameters ( ⃗p), for all state-
ments. The last column are the schedule coefﬁcients associated
with the constant ( c), for all statements.
Since we represent legal schedules as multidimensional afﬁne
functions, each row Θd of the scheduling function corresponds to
an integer point in the polytope of legal coefﬁcients Ld , built ex-
plicitly for this dimension. A program version in the optimization
space can thus be represented as follows, for a SCoP of l state-
ments, a schedule of dimension s, and the iteration vector ⃗x:
Θ. ⃗x =


⃗ı1
1 ··· ⃗ı1
p ⃗p1
1 ··· ⃗p1
p c1
1 ··· c1
p
...
...
⃗ıs
1 ··· ⃗ıs
p ⃗ps
1 ··· ⃗ps
p cs
1 ··· cs
p

.


⃗x1
...
⃗xp
⃗n1
...
⃗np
1...
1


To build each row Θd , we scan the legal polytope Ld , by suc-
cessively instantiating values for each coefﬁcient in a predeﬁned
order.4 Fourier-Motzkin elimination — a.k.a. projection — [5] pro-
vides a representation of the afﬁne constraints of a polytope ﬁt-
ted for its dynamic traversal. Computing the projection of all vari-
ables of a polytope Ld results in an equivalent polytope where
it is guaranteed that the value of vk is a function of v1,..., vk−1,
with afﬁne inequalities involving only v1,..., vk. Thus, the sequen-
tial order to build coefﬁcients is simply the reverse order of the
elimination steps. This scheme guarantees that provided a value in
the projection of v1,..., vk−1, a value exists for vk, for all k.5 In
order to achieve scalability, we use a modiﬁed and redundancy-
aware version of the Fourier-Motzkin algorithm. In its basic form,
the algorithm is known to generate many redundant constraints;
these redundancies reduce its scalability on large polyhedra. We
improved it, while maintaining the following properties for each
variable elimination:
1. any constraint deﬁning a hyperplane parallel to an existing con-
straint is removed (this is trivially computed since the con-
straints are kept normalized);
2. any variable which is linearly dependent to any other one(s) is
removed (thanks to implicit equalities detection and Gaussian
elimination);
3. constraints are removed if, once opposed, no point exists in
the solution polytope (we apply the Le Fur descending method
[25]).
In practice, this modiﬁed algorithm scales to hundreds of vari-
ables (schedule coefﬁcients) in the original system.
3.4 Schedule Completion Algorithm
For SCoPs with more than 4 or 5 statements, the previous construc-
tion leads to very large search spaces, challenging any traversal
algorithm. It is possible to focus the search on some coefﬁcients
of the schedule with maximal impact on performance, postpon-
ing the instantiation of a full schedule in a second heuristic step.
We show that such a two-step procedure can be designed with-
out breaking the fundamental legality property of the search space.
This approach will be used extensively to simplify the optimization
problem.
The previous projection pass guarantees it is possible to com-
plete — or even correct — any vector, slightly modifying its coef-
ﬁcients to make it lie within a given polytope. We use this property
to design the following completion algorithm . Given a vector ⃗v of
size n, for k ∈[1,n]:
1. compute the lower bound and upper bound of vk, provided the
coefﬁcient values for v1 ... vk−1, and
4 The order has no impact on the completeness of the traversal.
5 The case of holes in Z-polyhedra is handled through a schedule comple-
tion algorithm described in the next section.
2. if vk /∈[lb,ub], then vk = lb if vk < lb or vk = ub if vk > ub.6
Therefore it is possible to partially build a schedule preﬁx, e.g.,
values for the ⃗ıcoefﬁcients, while setting all other coefﬁcients to
0. Then, applying this correction principle will result in ﬁnding
the minimal amount of complementary transformations to make
the transformation lie in the computed legal space. The comple-
tion algorithm motivates the order of coefﬁcients in the Θ matrix.
We showed that the most performance impacting transformations
(interchange, skewing, reversal) are embedded in the ﬁrst coefﬁ-
cients of Θ — the ⃗ı coefﬁcients; followed by coefﬁcients usually
involved in fusion and distribution — the ⃗p coefﬁcients; and ﬁ-
nally the less impacting c coefﬁcients, representing loop shifting
and peeling [34]. The completion algorithm ﬁnds complementary
transformations in order of least to most impacting, as it will not
alter any vector preﬁx if a legal vector sufﬁx exists in the space.
Three fundamental properties are embedded in this completion
algorithm:
1. if v1,..., vk is a preﬁx of a legal point v, a completion is always
found;
2. this completion will only update vk+1,..., vdmax , if needed;
3. when v1,..., vk are the ⃗ı coefﬁcients, the heuristic looks for
the smallest absolute value for the ⃗p and constant coefﬁcients,
which corresponds to maximal (nested) loop fusion — relative
to the ⃗ıcoefﬁcients.
Picking coefﬁcients as close as possible to 0 has several advan-
tages in general: smaller coefﬁcients tend to simplify code genera-
tion, improve locality, reduce latency, and increase the size of basic
blocks in inner loops.
4. Traversing the search space
While it is possible to exhaustively traverse the constructed space
of legal versions for little SCoPs, in the case of one-dimensional
schedules, it becomes unpractical in the multidimensional case.
Pouchet et al. give a preliminary answer by means of a heuristic
to narrow this space and accelerate the traversal [35]. We build on
this result to design a powerful heuristic suitable for the multidi-
mensional case.
4.1 A Multidimensional Decoupling Heuristic
Our approach is called the decoupling heuristic as it leverages the
completion algorithm of Section 3.4 to stage the exploration of
large search spaces. It derives from the observation of the per-
formance distribution, where density patterns hinted that not all
schedule coefﬁcients have a signiﬁcant impact on performance [35,
34]. The principle of the decoupling heuristic for one-dimensional
schedules is (1) to enumerate different values for the ⃗ıcoefﬁcients,
(2) to instantiate full schedules with the completion algorithm, and
(3) to select the best completed schedules and further enumerate
the different coefﬁcients for the ⃗p part.
A direct extension to the multidimensional case exhibits two
major drawbacks. First, the relative performance impact of the
different schedule dimensions must be quantiﬁed. Second, an ex-
haustive enumeration of ⃗ıcoefﬁcients for all dimensions is out of
reach, as the number of points exponentially increases with the
number of dimensions. Figure 2 illustrates this assertion by sum-
marizing the size of the legal polytopes for different benchmarks,
for all schedule dimensions. We consider 10 SCoPs extracted from
classical benchmarks. The ﬁrst eight are UTDSP benchmarks [26]
directly amenable to polyhedral representation: compress-dct
is an image compression kernel (8x8 discrete cosine transform),
6 Z-holes are detected by checking if lb > ub.
edge-convolve2d is an edge detection kernel (different from
Ring-Roberts), fir is a Finite Impulse Response ﬁlter, lmsfir is
a Least Mean Square adaptive FIR ﬁlter, iir is an Inﬁnite Impulse
Response ﬁlter, matmult is a matrix multiplication kernel, latnrm
is a normalized lattice ﬁlter, and lpc (LPC analysis) is the hot
function of a linear predictive coding encoder. We considered two
additional benchmarks: ludcmp solves simultaneous linear equa-
tions by LU decomposition, and radar is a real code for the anal-
ysis of radar pulses. For each benchmark, we report the number
of (complex) instructions carrying array accesses ( #Inst), the num-
ber of loops ( #Loops), dependences ( #Dep), schedule dimension
(#Dim), and the total number of points for those dimensions (still
only legal schedules).
Relations between schedule dimensions To extend the decou-
pling approach to multidimensional schedules, we need to inte-
grate interactions between dimensions. For instance, to distribute
the outer loop of a nest (which can improve locality and vectoriza-
tion [3]), one can operate on the ⃗p and c parts of the schedule for
the ﬁrst dimension (a parametric shift). On the other hand, altering
the⃗ıparts will lead to the most signiﬁcant changes in the loop con-
trols. Indeed, the largest performance variation is usually captured
through the⃗ıparts [34], and a careful selection of those coefﬁcients
is mandatory to attain the best performance; conversely, it is likely
that the best performing transformations will share similar ⃗ıcoefﬁ-
cients in their schedules.
Furthermore, the ﬁrst dimension is highly constrained in gen-
eral, since all dependences need to be — weakly or strongly —
considered. Conversely, the last dimension is the less constrained
and often carries only very few dependences. 7
The decoupling heuristic in a nutshell We conducted an exten-
sive experiment showing that Θ1 (the ﬁrst time dimension of the
schedule) is a major discriminant of the overall performance dis-
tribution [34]. Therefore, the heuristic starts with an exploration of
values for coefﬁcients of Θ1, completing the schedule with a sin-
gle value for the remaining time dimensions (that is, the rest of
the schedule is set to 0 and the completion algorithm is called on
it). Like the decoupling heuristic, this exploration is limited to the
subspace associated with the ⃗ıcoefﬁcients of Θ1, except if this sub-
space is smaller than a given constant L1 — L1 = 50 in our experi-
ments. L1 drives the exhaustiveness of the procedure: the larger the
degree of freedom, the slower the convergence. By limiting to the ⃗ı
class we target only the most performance impacting subspaces.
To enumerate points in the polytopes, we incrementally pick a
dimension then pick an integer in the polyhedron’s projection onto
this dimension. Note that the full projection is computed once and
for all by the Fourier-Motzkin algorithm presented in Section 3.3,
before traversal. Technically, to enumerate integer points of the
subspace composed of the ﬁrst m columns of Ld , we deﬁne the
following recursive procedure to build a points ⃗v:
EXPLORE (⃗v,k) :
1. compute the lower bound and upper bound of vk, provided the
coefﬁcient values for v1 ... vk−1;
2. for each x ∈[lb,ub], set vk = x;
if k < m call E XPLORE (⃗v,k + 1) else output ⃗v.
The enumeration is initialized with a call to E XPLORE (⃗v,1).
The completion algorithm is then called on each vector ⃗v generated,
to compute a legal sufﬁx for ⃗v (corresponding to the columns [m +
1,n] of Ld ), ﬁnally instanciating a legal point of full dimensionality.
Then, the heuristic selects the x% best values for Θ1 (x = 5% in
our experiments), it proceeds with the exploration of values for co-
7 This is typically the case when the ﬁnal dimension is required to order the
statements within an innermost loop.
efﬁcients of Θ2, and recursively until the last but one dimension of
the schedule. The last dimension (corresponding to the innermost
nesting depth in the generated code) is not traversed, but completed
with a single value: exploring it would yield a huge number of it-
erations, with limited impact on the generated code, and negligible
impact on performance. Eventually, the exploration is bound with
a static limit — 1000 evaluations in our experiments.
4.2 Experiments
We consider three target architectures. The AMD Alchemy Au1500
is an embedded SoC with a MIPS32 core (Au1) running at
500MHz. We used GCC 3.2.1 with the -O3 ﬂag (version of
GCC and option with peak performance numbers, according to
the manufacturer). The STMicroelectronics ST231 is an embed-
ded SoC with a 4-issue VLIW core running at 400MHz and a
blocking cache. We used st200cc 1.9.0B (Open64) with the ﬂags
-O3 -mauto-prefetch -OPT:restrict. The AMD Athlon X64
3700+ has a 1MB L2 cache and runs at 2.4GHz. It runs Mandriva
Linux and the native compiler is GCC 4.1.1. We used the following
optimization settings for this platform which are known to bring ex-
cellent performance: -O3 -msse2 -ftree-vectorize . For this
particular machine, hardware counters were used to collect ﬁne-
grained cycle counts, and we used a real-time priority scheduler to
minimize OS interference. We picked the average of 10 runs for all
performance evaluations.
We implemented an instancewise dependence analysis, the con-
struction of the space of legal transformations, and the efﬁcient
scanning algorithms introduced in this paper. 8 We used free soft-
ware such as PipLib [16, 46] (a polyhedral library and parametric
integer linear programming solver) and CLooG [7] (an efﬁcient code
generator for the polyhedral model). For each point in the search
space, (1) we generated the kernel C code with CLooG,9 (2) then we
integrated this kernel in the original benchmark along with instru-
mentation to measure running time (we use performance counters
when available), (3) we compiled this code with the native compiler
and appropriate options, (4) and ﬁnally run the program on the tar-
get architecture and gather performance results. The original code
is included in this procedure starting at the second step, for appro-
priate performance comparison. Finally, the full iterative compila-
tion and execution process takes a few seconds on the heuristics,
and up to a few minutes on the GA described Section 5 for the
largest benchmark (up to 1000 tested versions). The time to com-
pute the legal space and to generate points is negligible with respect
to the total running time of the tested versions.
Results Figure 3 shows the results for the three architectures we
considered. We report the total numbers of tested versions ( Tested),
the run index of the best performing version ( Id Best ; the lower,
the earlier), and the performance improvement in percentage ( Perf.
Imp.). We also imposed a static limit of evaluating 1000 data points
in the search space. 10
All UTDSP experiments use the reference parameters. Increas-
ing data size would emphasize locality effects, yielding higher
speedups. E.g., matmult on Athlon with n = 250 yields 3 .61
speedup, n = 64 yields 3 .18 speedup, whereas the reference value
n = 10 yields 1 .43% speedup.
Discussion Our results show signiﬁcant improvement on all ker-
nels of the UTDSP suite. In addition, about 50 runs were sufﬁcient
for kernels with less than 10 statements (all but lpc and radar).
8 LETSEE, the LEgal Transformation SpacE Explorator, beta version avail-
able at http://www-rocq.inria.fr/ ∼pouchet/software/letsee
9 We use CLooG version 0.14.0 with default options.
10 It matches the maximum number of versions considered by the genetic
algorithm in Section 5.
Benchmark #Inst. #Loops #Dep. #Dim. dim 1 dim 2 dim 3 dim 4 Total
compress-dct 6 6 56 3 20 136 10857025 n/a 2.9 ×1010
edge 3 4 30 4 27 54 90534 43046721 5.6 ×1015
iir 8 2 66 3 18 6984 > 1015 n/a > 1019
fir 4 2 36 2 18 52953 n/a n/a 9.5 ×107
lmsfir 9 3 112 2 27 10534223 n/a n/a 2.8 ×108
matmult 2 3 7 1 912 n/a n/a n/a 912
latnrm 11 3 75 3 9 1896502 > 1015 n/a > 1022
lpc 12 7 85 2 63594 > 1020 n/a n/a > 1025
ludcmp 14 10 187 3 36 > 1020 > 1025 n/a > 1046
radar 17 20 153 3 400 > 1020 > 1025 n/a > 1048
Figure 2. Search space statistics
compress-dct edge iir ﬁr lmsﬁr matmult latnrm lpc ludcmp radar Average
#Runs 480 243 1000 77 1000 81 1000 1000 1000 1000
AMD Athlon Id. Best 19 11 34 33 51 16 6 489 37 405
Perf. Imp. 37.11% 5.58% 37.50% 40.24% 30.98% 42.87% 15.11% 31.15% 4.50% 6.42% 25.14%
#Runs 480 243 1000 77 1000 81 1000 1000 1000 1000
ST231 Id. Best 39 12 6 2 9 16 13 158 391 709
Perf. Imp. 15.11% 3.10% 24.91% 17.96% 10.17% 17.91% 2.61% 1.99% 6.33% 4.12% 10.42%
#Runs 480 243 1000 77 1000 81 1000 1000 1000 1000
Au1500 Id Best 30 17 38 27 11 17 43 82 175 454
Perf. Imp. 22.37% 2.51% 3.12% 14.00% 15.80% 20.18% 15.19% 14.08% 3.66% 3.39% 11.43%
Figure 3. Results of the decoupling heuristic for AMD Athlon, ST231 and Au1500
For all benchmarks, the best program version is syntactically
very far from the original one.
A good illustration of this is given for the Ring-Roberts run-
ning example, which achieve a 1 .47 speedup on a full HD image
on AMD Athlon; hardware counter details show a 54% reduction
of the L1 hit/miss ratio and a 51% of the data TLB misses. This
complex transformation is the result of multidimensional shifting
and peeling of the iterations preventing from fusion, and the com-
plete fusion of the remaining iterations.
The limited performance improvement for edge-convolve2d
is directly correlated to the code structure: this benchmark performs
a convolution of a 3x3 kernel, and is an excellent candidate for
optimization with loop unrolling — a transformation not embedded
in our search space. Our technique is fully compatible with other
iterative search techniques such as parameters tuning [1], and it is
expected that this combination would bring excellent performance
in this case.
We also noticed that performance improvements are often the
result of indirect enabling of back-end compiler optimizations (e.g.,
vectorization or scalar promotion), in addition to the direct im-
pact on hardware components (e.g, locality). Modern compiler op-
timization heuristics are still very fragile, and the interactions be-
tween optimization phases are not captured in their design. Pre-
dicting this interaction on non-trivial codes is still out of reach,
and slight syntactic differences can trigger different optimization
results. Testing different source code having the same semantics is
one way to circumvent the compiler’s optimization unpredictabil-
ity.
In addition, the best iteratively found transformation for a given
benchmark is different when considering a different target architec-
ture. This is due to different interactions with the compiler, as well
as different architectural features to optimize for. Note that it is not
a consequence of working with more expressive schedules: we al-
ready highlighted a similar pattern for the case of one-dimensional
schedules [35]. It conﬁrms the complexity of the optimization prob-
lem and the relevance of a feedback-directed approach.
The heuristic heavily relies on the observation that the ﬁrst
dimension of the schedule contains very few points — it traverses
this dimension exhaustively. However, exhaustive enumeration is
only possible for small kernels, such as most UTDSP benchmarks.
Unfortunately, for larger programs like lpc, ludcmp, radar, and to
some extent on latnrm, this approach does not scale.
To address this scalability issue, we substitute the exhaustive
search with a genetic algorithm.
5. Evolutionary Traversal of the Polytope
This section introduces novel genetic operators tailored to the
traversal of polytopes of legal afﬁne schedules.
Genetic algorithms (GA) [20] are known for their genericity:
we chose an evolutionary approach because of the natural encoding
of the geometric properties of the search space into crossover and
mutation operators. The two main properties are the following:
1. to enforce legality and uniqueness of the program versions, the
search space polytope must be closed for the genetic operators;
we construct dedicated mutation and crossover operators satis-
fying this property;
2. unlike random search, the traversal is characterized by its non-
uniformity (from the initial population and the crossovers); this
is utterly important as the largest part of the search space is
generally plagued with poor or similar performing versions
[35, 34].
Genetic algorithms have often be used in program optimization.
Our contribution is to reconcile ﬁne-grain control of a transforma-
tion heuristics — as opposed to optimization ﬂag or pass selection
[42, 1] — with the guaranteed legality of the transformed program
— as opposed to ﬁltering approaches [31, 29, 28] or always-correct
transformations [41, 24].
5.1 Genetic Algorithm
Using classical GA operators would not be an efﬁcient way to gen-
erate data points in our search space. This is because legal sched-
ules lie in afﬁne bounds that are strongly constrained and changing
them at random has a very low probability of preserving legality.
Moreover, in general, this probability decreases exponentially with
the space dimension [31]. We thus need to understand the proper-
ties of the space of legal schedules, and to embed them into dedi-
cated GA operators.
Some properties of afﬁne schedules The construction algorithm
outputs one polytope per schedule dimension. We can deduce nu-
merous properties on these polytopes, either deriving from the con-
struction algorithm or from afﬁne scheduling itself. In the follow-
ing, the term afﬁne constraint refers to any dependence, iteration
domain, or search bound constraint on coefﬁcients of the schedule
— the columns of Θ.
1. No afﬁne constraint involves coefﬁcients from different rows
of Θ, since those coefﬁcients are computed from distinct poly-
topes. Of course, multiple coefﬁcients inside a row can be in-
volved in a constraint.
2. Multiple coefﬁcients involved in a constraint are called depen-
dent. Each row can be partitioned into classes of dependent co-
efﬁcients, where no constraint involves coefﬁcients from dif-
ferent classes. E.g., in the polyhedron deﬁned by {x1 + x2 ≥
0 ∧x3 ≥0}we say that the set {x1,x2}is independent from the
set {x3}. Legality preservation is local to each class of depen-
dent coefﬁcients.
We design novel genetic operators exploiting and preserving
these properties.
Initialization We ﬁrst introduce an individual with a statically
computed schedule, computed by applying the completion algo-
rithm on all schedule coefﬁcents (which were previously all set to
0). This choice shares its motivation with the decoupling heuristic
in Section 4.1.
The rest of the population is initialized by performing aggres-
sive mutations on this static schedule; we generate 30 to 100 indi-
viduals, depending on the space dimension. The initial population
is heavily biased towards a particular subspace (typically the sub-
space of the ⃗ıcoefﬁcients), emphasizing the non-uniformity of the
traversal.
Mutation The mutation operator starts with the computation of
the distribution of probabilities to alter every coefﬁcient. This prob-
ability is driven by three factors; the ﬁrst one derives directly from
the heuristic of the one-dimensional case [35]:
•coefﬁcients of the iteration vectors have a dramatic impact on
the structure of the generated code; minor modiﬁcations trigger
wild jumps in the search space;
•conversely, coefﬁcients with little linear dependences with oth-
ers may require ample mutations to trigger signiﬁcant changes.
•the schedule dimension considered: lower dimensions and es-
pecially the scalar ones usually have a lower impact on perfor-
mance.
In addition, we weigh the probabilities with a uniform annealing
factor, to tune the aggressiveness of the mutation operator along
with the maturation of the population.
We randomly pick a value within the legal bounds for this
coefﬁcient, and according to the distribution of probabilities. As
this mutation may cause other coefﬁcients to become incorrect, we
then update the schedule with the completion algorithm depicted in
Section 3.4; it is a simple update because the schedule preﬁx can
be kept in the legal space, computing mutated coefﬁcients in the
reverse order of Fourier-Motzkin elimination.
We also experimented a simpler mutation operator, where the
bounds to pick mutated values where not adjusted to the corre-
sponding polytope of legal versions, applying our correction mech-
anism a posteriori. This approach did not prove very effective as
coefﬁcients are often correlated or severely constrained: randomly
picking values for multiple correlated coefﬁcients often leads to
identical schedules after correction. Only an incremental applica-
tion of the correction mechanism avoids the generation of many
duplicates (which strongly degrade the effectiveness of the muta-
tion operator).
Crossover We provide two crossover operators. The row crossover
is dedicated to compensating the row-wise aspect of the mutation
operator. Given two individuals represented by Θ and Θ′, the row
crossover operator randomly picks rows of either Θ or Θ′to build
a new individual Θ′′. This operator obviously preserves legality
since there are no dependences between rows. Since the mutation
operates within a schedule dimension, it may succeed in ﬁnding
good candidates for a given row of Θ or Θ′, but may mix these
with ineffective rows. Combining these rows may lead to a good
schedule, with a much higher probability than with mutation alone.
The column crossover, is dedicated to crossing independent
classes of schedule coefﬁcients (represented by sets of columns not
connected by any afﬁne constraint); this operator is quite original
and speciﬁc to the geometrical properties of the search space. It can
be seen as a ﬁner-grained crossover operator. From two individuals
Θ and Θ′, it randomly selects an independent class from either
parent — at every dimension — to build Θ′′. When there is only
one independent class in a given schedule dimension this operator
behaves like the row crossover. We rely on the geometric properties
of the polytope to compute linearly independent sets of variables
for a given schedule dimension. These sets are computed once and
for all, immediately after the search space polytope is built. This
operator preserves legality as it only modiﬁes independent sets of
schedule coefﬁcients.
Dependences constrain schedule coefﬁcients in pairs of state-
ments. Several transitive steps are needed to characterize all cor-
relations between coefﬁcients in a dependent class. This operator
carefully reﬁnes the grain of schedule transformations, while pre-
serving legality as it only modiﬁes independent sets of coefﬁcients.
Selection The selection process is a classical numerus clausus ﬁl-
tering, keeping half of the current population for the next genera-
tion. A better option would be to combine multiple metrics, includ-
ing performance predictors (to avoid running the code) or hardware
counters. We are currently investigating such techniques.
5.2 Experimental Results
Figure 4 summarize the results of the genetic algorithm applied
to all benchmarks for all architectures. Row Heuristic/GA shows
the fraction of the speedup achieved by the decoupling heuristic
w.r.t. the genetic algorithm, and fractions are averaged for the
benchmarks of less than 10 statements versus more than 10. We
initialized the population with 30 to 100 individuals, and performed
at most 10 generations; therefore, the maximum number of runs for
each program was 1000.
Comparing these results with the table in Figure 3 shows the
efﬁciency and scalability of our method. The genetic algorithm
achieves strong speedups for the larger kernels; these speedups are
much stronger than those of the decoupling heuristic for the larger
benchmarks. On the other hand, the decoupling heuristic exposes
78–100% of the speedup obtained with genetic algorithms within
the ﬁrst 50 runs, for all kernels of less than 10 statements
Architecture compress-dct edge iir ﬁr lmsﬁr matmult latnrm lpc ludcmp radar Average
AMD Athlon 44.17% 7.86% 32.18% 40.70% 24.23% 42.87% 28.23% 45.84% 69.63% 40.18% 37.58%
Heuristic/GA 84.31% 82.26% 93.58% 98.86% 80.71% 100% 53.57% 67.68% 6.52% 16.05% 89.95% / 35.95%
ST231 18.42% 3.29% 27.40% 18.81% 8.63% 17.91% 0.86% 3.44% 5.96% 28.32% 13.30%
Heuristic/GA 83.33% 94.52% 90.91% 95.48% 85.14% 100% 92.30% 32.20% 22.26% 30.82% 91.56% / 44.39%
AMD Au1500 25.11% 3.03% 4.07% 14.10% 19.18% 22.67% 27.01% 17.43% 15.71% 30.87% 17.91%
Heuristic/GA 89.21% 82.83% 78.29% 99.50% 81.64% 88.93% 55.55% 82.35% 16.56% 10.91% 86.73% / 41.35%
Figure 4. Results of the genetic algorithm. The Decoupling heuristics succeeds in discovering 78-100% of the speedup achieved by GA for
all benchmarks of less than 10 statements. For larger benchmarks, the GA performs 2 .46×better in average, and up to 16 ×better.
Results are better on AMD Athlon than on embedded proces-
sors, probably because the architecture is more complex: a good
interaction between architectural components is harder to achieve
and brings higher improvements. Conversely, the ST231 and AMD
Au1500 have a predictable behaviour, more effectively harnessed
by the back-end compiler, and showing less room for improvement;
yet our results are still signiﬁcant for such targets.
We report a detailed study of the representative compress-dct
benchmark, on AMD Athlon. Figure 5 summarizes the results, and
conﬁrms the huge advantage of the GA given the statistically sparse
and chaotic occurrence of performance-enhancing schedules. The
ﬁrst graph shows the convergence of our GA approach versus a Ran-
dom traversal in the space of legal schedules (only legal points are
drawn). The GA algorithm ran for 10 generations from an initial
population of 50 individuals. Both plots are an average of 100 com-
plete runs. On the second graph, we report the performance distri-
bution of the legal space. We exhaustively enumerated and eval-
uated all points with a distinct value for the ⃗ı+ ⃗p coefﬁcients of
the ﬁrst schedule dimension, combined with all points with a dis-
tinct⃗ıvalue for the second one; a total of 1 .29 ×106 schedules are
evaluated. For each distinct value of the ﬁrst schedule dimension
(plotted in the horizontal axis), we report the performance of the
Best schedule, the Worst one, and the Average for all tested values
of the second schedule dimension. The third graph shows the per-
formance distribution for all tested points of the second schedule
dimension, provided a single value for the ﬁrst one, sorted from
the best performing one to the worse (the best performing sched-
ule belongs to this chart). The difﬁculty to reach the best points in
the search space is emphasized by their extremely low proportion:
only 0.14% of points achieve at least 80% of the maximal speedup,
while only 0.02% achieve 95% and more. Conversely, 61.11% de-
grade performance of the original code, while in total 10.88% de-
grade the performance by a factor 2.
Figure 5(a) shows that our GA converges much faster than ran-
dom search: in 500 runs, the random search is only able to dis-
cover a 18% performance improvement; the GA takes only 120
runs to match this ﬁgure, converging towards the space maximal
44.1% improvement after 350 runs, at the 7 th generation (i.e.,
before the imposed limit of 10 generations). This the maximum
speedup available, as shown by the exhaustive search experiments
in Figure 5(b). The effectiveness of the genetic operators is illus-
trated by the decorrelation of the performance improvements and
the actual performance distribution. Conversely, random traversal
follows the shape of the performance distribution, and in average is
not able to reach the best performing schedules — as their density
in the space is very low.
Finally, we studied the behavior of multiple schedules for
the compress-dct benchmark, analyzing hardware counters on
Athlon. This study highlights complex interactions between the
memory hierarchy (both L1 and L2 accesses must be minimized
to achieve good performance), vectorization, and the activity of
functional units. The best performing transformation reduces the
numbers of stall cycles by a factor of 3, while improving the L2
hit/miss ratio by 10%. Transformation sequences achieving the op-
timal performance are opaque at ﬁrst glance: they involve complex
combinations of skewing, reversal, distribution and index-set split-
ting. These transformations address speciﬁc performance anoma-
lies of the loop nest, but they are often associated with the interplay
of multiple architecture components. Moreover, we observe that the
best optimizations are usually associated with more complex con-
trol ﬂow than the original code. The number of dynamic branches
is increased in most cases [34], although stall cycles are heavily
reduced due to locality and ILP improvements.
Overall, our results conﬁrms the potential of iterative optimiza-
tion to accurately capture the complex behavior of the processor
and back-end compiler, and extends its applicability to optimiza-
tion problems far more complex than those commonly solved in
adaptive compilation.
6. Related Work
In recent years, the beneﬁts of iterative compilation have been
widely reported [23, 13, 14, 21]. Iterative compilation is often
able to ﬁnd optimization sequences that out-perform the highest
optimization settings in commercial compilers.
Kulkarni et al. [24] introduce the VISTA system, an interactive
compilation system which concentrates on reducing the time to ﬁnd
good solutions. Another system that attempted to speedup iterative
compilation was introduced by Cooper et al. called ACME [12].
Triantafyllis et al. [42] develop an alternative approach to reduce
the total number of evaluations of a new program. Here the space
of compiler options is examined off-line on a per function basis and
the best performing ones are classiﬁed into a small tree of compiler
options.
Iterative optimization has been used effectively on a variety
of compilation and parallelization problems and its applicability
and practicality has been demonstrated beyond the academic world
[37]. Although multidimensional afﬁne scheduling is an obvious
target for iterative optimization, its proﬁtability is one of the most
difﬁcult to assess, due to (1) the model’s intrinsic expressiveness
(the downside of its effectiveness) and (2) its lack of analytical
models for the impact of transformations on the target architec-
ture. Hence, related work has been very limited up to this point. To
the best of our knowledge, Nisbet pioneered research in the area
with one of the very ﬁrst papers in iterative optimization. He de-
veloped the GAPS framework [31] which used a genetic algorithm
to traverse a search space of afﬁne schedules for automatic paral-
lelization. In addition, Long and O’Boyle [29] considered a larger
search space of transformation sequences represented as multidi-
mensional afﬁne schedules. Both of these approaches suffer from
under-constraining the search space by considering all possible
schedules, including illegal ones. Downstream ﬁltering approaches
do not scale, due to the exponentially diminishing proportion of
legal schedules with respect to the program size. For instance, Nis-
bet obtains only 3 −5% of legal schedules for the ADI benchmark
(6 statements). Moreover, under-constraining the search space lim-
 0
 0.2
 0.4
 0.6
 0.8
 1
 1.2
 1.4
 1.6
 50  100  150  200  250  300  350  400  450  500
Speedup
Tested versions
GA versus random - compress-dct
GA
Random
(a)
 0
 0.2
 0.4
 0.6
 0.8
 1
 1.2
 1.4
 1.6
 10  20  30  40  50  60
Speedup
Point index of the first schedule dimension
Performance distribution - compress-dct
Best
Average
Worst (b)
 0
 0.2
 0.4
 0.6
 0.8
 1
 1.2
 1.4
 1.6
 0  2000  4000  6000  8000  10000 12000 14000 16000 18000
Speedup
Point of the second schedule dimension, first dimension fixed
Performance distribution (sorted) - compress-dct (c)
Figure 5. Performance Distribution of compress-dct, AMD Athlon. GA discovers the maximum speedup available in the search space.
its the possibility to narrow the search to the most promising sub-
spaces.
Pouchet et al. demonstrated a more efﬁcient approach, by em-
bedding program dependences and afﬁne scheduling properties into
the search space itself. However, this approach was only applied to
small kernels with single dimensional schedules. We remove the
expressiveness limitations of this prior result, extending the search
space construction, preconditioning and traversal algorithms to ar-
bitrary multidimensional afﬁne schedules.
7. Conclusion
Present day compilers fail to model the complex interplay between
different optimizations and their effect on code on all the different
processor architecture components. Empirical search has become
essential to achieve portable high performance in spite of the ana-
lytically intractable hardware complexity. Most iterative compila-
tion techniques target compiler optimization ﬂags, parameters, de-
cision heuristics, or phase ordering [10, 14, 41, 24, 42, 1]. We take
a more aggressive stand, aiming for the construction and tuning of
complex sequences of transformations.
Afﬁne schedules build a very expressive search space, since a
single schedule can represent an arbitrarily complex sequence of
loop transformations. The ﬁrst attempts to traverse such a space
faced legality problems and showed poor results because only few
legal afﬁne schedules were found [31, 29]. Pouchet et al. recently
proposed a solution for a restricted class of loop nests and trans-
formations [35]. This paper targets all static control programs and,
by construction, enables iterative optimization in a closed space of
semantics-preserving transformations . To overcome the combina-
torial nature of the optimization search space, we designed heuris-
tics and a genetic algorithm with specialized operators that leverage
the algebraic properties of this space, embedding the legality con-
straints into the operators themselves. We simultaneously demon-
strate good performance gains and excellent convergence speed on
huge search spaces, even on larger loop nests where fully iterative
afﬁne scheduling has never been attempted before.
For future work, we intend on looking at automatic paralleliza-
tion, and at incorporating loop tiling for both locality and paral-
lelism. While the back-end compiler may support some form of
tiling, early experiments show that applying it as a post-pass does
not signiﬁcantly improve performance. Complexity of the gener-
ated code is partly responsible for this.
Acknowledgments Uday+Ram+Saday for fruitful talks and Cor-
nero/Rohou for the traviata board
References
[1] F. Agakov, E. Bonilla, J. Cavazos, B. Franke, G. Fursin, M. F. P.
O’Boyle, J. Thomson, M. Toussaint, and C. K. I. Williams. Using
machine learning to focus iterative optimization. In Proc. of the Intl.
Symposium on Code Generation and Optimization (CGO’06) , pages
295–305, Washington, DC, USA, 2006. IEEE Computer Society.
[2] N. Ahmed, N. Mateev, and K. Pingali. Tiling imperfectly-nested loop
nests. In SC’2000 High Performance Networking and Computing ,
Dallas, november 2000.
[3] J. Allen and K. Kennedy. Optimizing Compilers for Modern
Architectures. Morgan Kaufmann Publishers, 2002.
[4] C. Ancourt and F. Irigoin. Scanning polyhedra with DO loops. In 3rd
ACM SIGPLAN Symposium on Principles and Practice of Parallel
Programming, pages 39–50, june 1991.
[5] U. Banerjee. Loop Transformations for Restructuring Compilers .
Kluwer Academic, 1993.
[6] D. Barthou, J.-F. Collard, and P. Feautrier. Fuzzy array dataﬂow
analysis. J. of Parallel and Distributed Computing , 40:210–226,
1997.
[7] C. Bastoul. Code generation in the polyhedral model is easier than you
think. In IEEE Intl. Conf. on Parallel Architectures and Compilation
Techniques (PACT’04), pages 7–16, Juan-les-Pins, september 2004.
[8] C. Bastoul and P. Feautrier. Improving data locality by chunking.
In CC’12 Int. Conf. on Compiler Construction, LNCS 2622 , pages
320–335, Warsaw, april 2003.
[9] A. Bernstein. Analysis of programs for parallel processing. IEEE
Transactions on Electronic Computers , 15(5):757–763, october 1966.
[10] F. Bodin, T. Kisuki, P. M. W. Knijnenburg, M. F. P. O’Boyle, and
E. Rohou. Iterative compilation in a non-linear optimisation space.
In W. on Proﬁle and Feedback Directed Compilation , Paris, October
1998.
[11] U. Bondhugula, M. Baskaran, S. Krishnamoorthy, J. Ramanujam,
A. Rountev, and P. Sadayappan. Afﬁne transformations for
communication minimal parallelization and locality optimization
of arbitrarily nested loop sequences. Technical Report OSU-CISRC-
5/07-TR43, Ohio State Univ., 2007.
[12] K. D. Cooper, A. Grosul, T. J. Harvey, S. Reeves, D. Subramanian,
L. Torczon, and T. Waterman. Acme: adaptive compilation made
efﬁcient. In Proceedings of the 2005 ACM SIGPLAN/SIGBED
Conference on Languages, Compilers, and Tools for Embedded
Systems, pages 69–77, New York, NY , USA, 2005. ACM Press.
[13] K. D. Cooper, P. J. Schielke, and D. Subramanian. Optimizing
for reduced code space using genetic algorithms. In Workshop on
Languages, Compilers, and Tools for Embedded Systems , pages 1–9,
Atlanta, Georgia, July 1999. ACM Press.
[14] K. D. Cooper, D. Subramanian, and L. Torczon. Adaptive optimizing
compilers for the 21st century. J. Supercomputing, 23(1):7–22, 2002.
[15] A. Darte, Y . Robert, and F. Vivien. Scheduling and Automatic
Parallelization. Birkhauser, 2000.
[16] P. Feautrier. Parametric integer programming. RAIRO Recherche
Op´erationnelle, 22(3):243–268, 1988.
[17] P. Feautrier. Some efﬁcient solutions to the afﬁne scheduling problem,
part I: one dimensional time. Intl. Journal of Parallel Programming ,
21(5):313–348, october 1992.
[18] P. Feautrier. Some efﬁcient solutions to the afﬁne scheduling problem,
part II: multidimensional time. Intl. Journal of Parallel Programming ,
21(6):389–420, december 1992.
[19] S. Girbal, N. Vasilache, C. Bastoul, A. Cohen, D. Parello, M. Sigler,
and O. Temam. Semi-automatic composition of loop transformations
for deep parallelism and memory hierarchies. Intl. J. of Parallel
Programming, 34(3), 2006.
[20] D. E. Goldberg. Genetic Algorithms in Search, Optimization and
Machine Learning . Addison-Wesley Longman Publishing Co., Inc.,
Boston, MA, USA, 1989.
[21] M. Haneda, P. M. W. Knijnenburg, and H. A. G. Wijshoff. Automatic
selection of compiler options using non-parametric inferential
statistics. In Proceedings of the International Conference on
Parallel Architectures and Compilation Techniques , pages 123–132,
Washington, DC, USA, 2005. IEEE Computer Society.
[22] W. Kelly. Optimization within a Uniﬁed Transformation Framework .
PhD thesis, Univ. of Maryland, 1996.
[23] T. Kisuki, P. M. W. Knijnenburg, and M. F. P. O’Boyle. Combined
selection of tile sizes and unroll factors using iterative compilation. In
Proceedings of the International Conference on Parallel Architectures
and Compilation Techniques, page 237, Washington, DC, USA, 2000.
IEEE Computer Society.
[24] P. A. Kulkarni, S. R. Hines, D. B. Whalley, J. D. Hiser, J. W. Davidson,
and D. L. Jones. Fast and efﬁcient searches for effective optimization-
phase sequences. ACM Trans. Archit. Code Optim. , 2(2):165–198,
2005.
[25] M. Le Fur. Scanning parameterized polyhedron using fourier-motzkin
elimination. Concurrency - Practice and Experience , 8(6):445–460,
1996.
[26] C. Lee. UTDSP benchmark suite, 1998.
http://www.eecg.toronto.edu/˜corinna/DSP.
[27] A. W. Lim and M. S. Lam. Maximizing parallelism and minimizing
synchronization with afﬁne transforms. In ACM Symp. on Principles
of Programming Languages (PoPL’97) , pages 201–214, New York,
NY , USA, 1997. ACM Press.
[28] S. Long and G. Fursin. Systematic search within an optimisation
space based on uniﬁed transformation framework. IJCSE Intl. Journal
of Computational Science and Engineering , 2006.
[29] S. Long and M. O’Boyle. Adaptive Java optimisation using instance-
based learning. In Proc. ICS, pages 237–246, 2004.
[30] D. E. Maydan, S. P. Amarasinghe, and M. S. Lam. Array dataﬂow
analysis and its use in array privatization. In ACM Symp. on Principles
of Programming Languages (PoPL’93) , pages 2–15, Charleston,
South Carolina, Jan. 1993.
[31] A. Nisbet. GAPS: A compiler framework for genetic algorithm (GA)
optimised parallelisation. In HPCN Europe 1998: Proceedings of
the Intl. Conf. and Exhibition on High-Performance Computing and
Networking, pages 987–989, London, UK, 1998. Springer-Verlag.
[32] M. Palkovi ˇc. Enhanced Applicability of Loop Transformations . PhD
thesis, T. U. Eindhoven, The Netherlands, Sept. 2007.
[33] S. Pop, A. Cohen, C. Bastoul, S. Girbal, P. Jouvelot, G.-A. Silber,
and N. Vasilache. GRAPHITE: Loop optimizations based on the
polyhedral model for GCC. In Proc. of the 4th GCC Developper’s
Summit, Ottawa, Canada, June 2006.
[34] L.-N. Pouchet, C. Bastoul, J. Cavazos, and A. Cohen. A note on
the performance distribution of afﬁne schedules. 2nd Workshop on
Statistical and Machine learning approaches to ARchitectures and
compilaTion (SMART’08), G¨oteborg, Sweden, January 2008.
[35] L.-N. Pouchet, C. Bastoul, A. Cohen, and N. Vasilache. Iterative
optimization in the polyhedral model: Part I, one-dimensional time.
In IEEE/ACM Intl. Conf. on Code Generation and Optimization
(CGO’07), pages 144–156, San Jose, California, Mar. 2007.
[36] W. Pugh. The Omega test: a fast and practical integer program-
ming algorithm for dependence analysis. In Proceedings of the
third ACM/IEEE conference on Supercomputing , pages 4–13, Albu-
querque, Aug. 1991.
[37] M. P ¨uschel, B. Singer, J. Xiong, J. Moura, J. Johnson, D. Padua,
M. Veloso, and R. W. Johnson. SPIRAL: A generator for platform-
adapted libraries of signal processing algorithms. J. of High
Performance Computing and Applications, special issue on Automatic
Performance Tuning, 18(1):21–45, 2004.
[38] F. Quiller ´e, S. Rajopadhye, and D. Wilde. Generation of efﬁcient
nested loops from polyhedra. Intl. Journal of Parallel Programming ,
28(5):469–498, october 2000.
[39] L. Renganarayanan, D. Kim, S. Rajopadhye, and M. M. Strout.
Parameterized tiled loops for free. SIGPLAN Notices, Proceedings of
the 2007 PLDI conference , 42(6):405–414, 2007.
[40] A. Schrijver. Theory of Linear and Integer Programming . John Wiley
& Sons, 1986.
[41] M. Stephenson, S. Amarasinghe, M. Martin, and U.-M. O’Reilly.
Meta optimization: improving compiler heuristics with machine
learning. SIGPLAN Not. , 38(5):77–90, 2003.
[42] S. Triantafyllis, M. Vachharajani, and D. I. August. Compiler
optimization-space exploration. In Journal of Instruction-level
Parallelism, 2005.
[43] N. Vasilache, C. Bastoul, and A. Cohen. Polyhedral code generation
in the real world. In Proceedings of the Intl. Conf. on Compiler
Construction (ETAPS CC’06) , LNCS, pages 185–201, Vienna,
Austria, Mar. 2006. Springer-Verlag.
[44] N. Vasilache, A. Cohen, and L.-N. Pouchet. Automatic correction of
loop transformations. In IEEE Intl. Conf. on Parallel Architectures
and Compilation Techniques (PACT’07), Brasov, september 2007. To
appear.
[45] F. Vivien. On the optimality of Feautrier’s scheduling algorithm. In
Euro-Par ’02: Proceedings of the 8th Intl. Euro-Par Conf. on Parallel
Processing, pages 299–308, London, UK, 2002. Springer-Verlag.
[46] D. Wilde. A library for doing polyhedral operations. Technical report,
IRISA, 1993.
[47] M. Wolf. Improving Locality and Parallelism in Nested Loops . PhD
thesis, Dept. of computer science, Stanford University, California,
1992.
[48] M. Wolfe. High performance compilers for parallel computing .
Addison-Wesley Publishing Company, 1995.
[49] J. Xue. Transformations of nested loops with non-convex iteration
spaces. Parallel Computing, 22(3):339–368, 1996.

1
Learning Semantic Program Embeddings with Graph Interval
Neural Network
YU WANG, Nanjing University
FENGJUAN GAO, Nanjing University
LINZHANG WANG, Nanjing University
KE WANG,Visa Research.
Learning distributed representations of source code has been a challenging task for machine learning models.
Earlier works treated programs as text so that natural language methods can be readily applied. Unfortunately,
such approaches do not capitalize on the rich structural information possessed by source code. Of late, Graph
Neural Network (GNN) was proposed to learn embeddings of programs from their graph representations.
Due to the homogeneous ( i.e. do not take advantage of the program-speci/f_ic graph characteristics) and
expensive (i.e. require heavy information exchange among nodes in the graph) message-passing procedure,
GNN can suﬀer from precision issues, especially when dealing with programs rendered into large graphs.
In this paper, we present a new graph neural architecture, called Graph Interval Neural Network (GINN),
to tackle the weaknesses of the existing GNN. Unlike the standard GNN, GINN generalizes from a curated
graph representation obtained through an abstraction method designed to aid models to learn. In particular,
GINN focuses exclusively on intervals (generally manifested in looping construct) for mining the feature
representation of a program, furthermore, GINN operates on a hierarchy of intervals for scaling the learning
to large graphs.
We evaluate GINN for two popular downstream applications: variable misuse prediction and method name
prediction. Results show in both cases GINN outperforms the state-of-the-art models by a comfortable margin.
We have also created a neural bug detector based on GINN to catch null pointer deference bugs in Java code.
While learning from the same 9,000 methods extracted from 64 projects, GINN-based bug detector signi/f_icantly
outperforms GNN-based bug detector on 13 unseen test projects. Next, we deploy our trained GINN-based
bug detector and Facebook Infer, arguably the state-of-the-art static analysis tool, to scan the codebase of 20
highly starred projects on GitHub. /T_hrough our manual inspection, we con/f_irm 38 bugs out of 102 warnings
raised by GINN-based bug detector compared to 34 bugs out of 129 warnings for Facebook Infer. We have
reported 38 bugs GINN caught to developers, among which 11 have been /f_ixed and 12 have been con/f_irmed
(/f_ix pending). GINN has shown to be a general, powerful deep neural network for learning precise, semantic
program embeddings.
1 INTRODUCTION
Learning distributed representations of source code has a/t_tracted growing interest and a/t_ten-
tion in program language research over the past several years. Inspired by the seminal work
word2vec (Mikolov et al. 2013), many methods have been proposed to produce vectorial representa-
tion of programs. Such vectors, commonly known as program embeddings, capture the semantics
of a program through their numerical components such that programs denoting similar semantics
will be located in close proximity to one another in the vector space. A bene/f_it of learning program
2020. XXXX-XXXX/2020/1-ART1 $15.00
DOI:
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
arXiv:2005.09997v2  [cs.SE]  27 May 2020
1:2 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
embedding is to enable the application of neural technology to a board range of programming
language (PL) tasks, which were exclusively approached by logic-based, symbolic methods before.
Existing Models of Source Code.Earlier works predominately considered source code as text,
and mechanically transferred natural language methods to discover their shallow, textual pat-
terns (Gupta et al . 2017; Hindle et al . 2012; Pu et al . 2016). Following approaches aim to learn
semantic program embeddings from Abstract Syntax Tree (AST) (Alon et al. 2019a,b; Maddison and
Tarlow 2014). Despite the signi/f_icant progress, tree-based models are still con/f_ined to learning syn-
tactic program features due to the limited oﬀering from AST. Recently, another neural architecture,
called Graph Neural Network (GNN), has been introduced to the programming domain (Allamanis
et al. 2018; Li et al. 2016). /T_he idea is to reduce the problem of learning embeddings from source
code to learning from their graph representations. Powered by a general, eﬀective message-passing
procedure (Gilmer et al. 2017), GNN has achieved state-of-the-art results in a variety of problem
domains: variable misuse prediction (Allamanis et al. 2018), program summarization (Fernandes
et al. 2019), and bug detection and /f_ixing (Dinella et al. 2020). Even though GNN has substantially
improved the prior works, their precision and eﬃciency issues remain to be dealt with. First, GNN
is indiscriminate in which types of graph data they deal with. In other words, once programs are
converted into graphs, they will be processed in the same manner as any other graph data ( e.g.
social networks, molecular structures). As a result, it misses out on the opportunity to capitalize on
the unique graph characteristics possessed by programs. Second, perhaps more severely, GNN has
diﬃculties in learning from large graphs due to the high cost of its underlying message-passing
procedure. Ideally, every node should pass messages directly or indirectly to every other node in
the graph to allow suﬃcient information exchange. However, such an expensive propagation is
hard to scale to large graphs without incurring a signi/f_icant precision loss.
Graph Interval Neural Network.In this paper, we present a novel, general neural architecture
called Graph Interval Neural Network (GINN) for learning semantic embeddings of source code.
/T_he design of GINN is based on a key insight that by learning from abstractions of programs, models
can focus on code constructs of greater importance, and ultimately capture the essence of program
semantics in a precise manner. At the technical level, we adopt GNN’s learning framework for
their cu/t_ting-edge performance. An important challenge we need to overcome is how to abstract
programs into more eﬃcient graph representations for GINN to learn. To this end, we develop
a principled abstraction method directly applied on control /f_low graphs. In particular, we derive
from the control /f_low graph a hierarchy of intervals — subgraphs that generally represent looping
constructs — as a new form of program graphs. Under this abstracted graph representation, GINN
not only focuses exclusively on intervals for extracting deep, semantic program features but also
scales its generalization to large graphs eﬃciently. Note that GINN’s scalability-enhancing approach
is fundamentally diﬀerent from the existing techniques (Allamanis et al. 2018), where extra edges
are required to link nodes that are far apart in a graph. Moreover, Allamanis et al. (2018) have not
quanti/f_ied the improvement of model scalability due to the added edges, let alone identify which
edges to add w.r.t. the diﬀerent graphs and downstream tasks.
We have realized GINN as a new model of source code and extensively evaluated it. To demon-
strate its generality, we evaluate GINN in multiple downstream tasks. First, we pick two prediction
problems de/f_ined by the prior works (Allamanis et al. 2018; Alon et al. 2019b) in which GINN signif-
icantly outperforms RNN Sandwich and sequence GNN, the state-of-the-art models in predicting
variable misuses and method names respectively. Second, we evaluate GINN in detecting null
pointer dereference bugs in Java code, a task that is barely a/t_tempted by learning-based approaches.
Results show GINN-based bug detector is signi/f_icantly more eﬀective than the baseline built upon
GNN. We also deploy our trained bug detector and Facebook Infer (Berdine et al. 2006; Calcagno
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:3
et al. 2015), arguably the state-of-the-art static analysis tool, to search bugs in 20 Java projects that
are among the most starred on GitHub. A/f_ter manually inspecting the alarms raised by both tools,
we /f_ind that our bug detector yields a higher precision (38 bugs out of 102 alarms) than Infer (34
bugs out of 129 alarms). We have reported the 38 bugs detected by GINN to developers, out of
which 11 are /f_ixed and 12 are con/f_irmed (/f_ix pending). Our evaluation suggests GINN is a general,
powerful deep neural network for learning precise, semantic program embeddings.
Contributions. Our main contributions are:
•We propose a novel, general deep neural architecture, Graph Interval Neural Network,
which generalizes from a curated graph representation, obtained through a graph abstrac-
tion method on the control-/f_low graph.
•We realize GINN as a new deep model of code, which readily serves multiple downstream
PL tasks. More importantly, GINN outperforms the state-of-the-art model by a comfortable
margin in each downstream task.
•We present the details of our extensive evaluation of GINN in variable misuse prediction,
method name prediction, and null pointer dereference detection.
•We publish our code and data at https:///f_igshare.com/articles/datasetstar gz/8796677 to
aid future research activity.
2 PRELIMINARY
In this section, we brie/f_ly revisit connected, directed graphs, interval (Allen 1970) and GNN (Gori
et al. 2005), which lay the foundation for our work.
2.1 Graph
A graph G = (V , E)consists of a set of nodes V = {/v.alt1, ...,/v.altn }, and a list of directed edge sets
E = (E1, ...,EK )where K is the total number of edge types and Ek (1 ≤k ≤K) is a set of edges
of type k. We denote by (/v.alts ,/v.altd , k)∈ Ek an edge of type k directed from node /v.alts to node /v.altd . For
graphs with only one edge type, an edge is represented as (/v.alts ,/v.altd ).
/T_he immediate successors of a node/v.alti , denoted by post (/v.alti ), are all of the nodes /v.altj for which
(/v.alti ,/v.altj )is an edge in one edge set in E. /T_he immediate predecessors of node/v.altj , denoted by pre (/v.altj ),
are all of the nodes /v.alti for which (/v.alti ,/v.altj )is an edge in one edge set in E.
A path is an ordered sequence of nodes (/v.altp, ...,/v.altq)and their connecting edges, in which each
node /v.altt , t ∈(p, . . . ,q −1), is an immediate predecessor of /v.altt+1. A closed path is a path in which
the /f_irst and last nodes are the same. /T_he successors of a node/v.altt , denoted by post ∗(/v.altt ), are all of
the nodes /v.altx for which there exists a path from/v.altt to /v.altx . /T_he predecessors of a node/v.altt , denoted by
pre ∗(/v.altt ), are all of the nodes /v.alt/y.altfor which there exists a path from /v.alt/y.altto /v.altt .
2.2 Interval
Introduced by Allen (1970), an interval I(h) is the maximal, single entry subgraph in which h
is the only entry node and all closed paths contain h. /T_he unique interval nodeh is called the
interval head or simply the header node. An interval can be expressed in terms of the nodes in it:
I(h)= {/v.altl ,/v.alt2, ...,/v.altm }.
By selecting the proper set of header nodes, a graph can be partitioned into a set of disjoint
intervals. We show the partition procedure proposed by Allen (1970) in Algorithm 1. /T_he key is to
add to an interval a node only if all of its immediate predecessors are already in the interval (Line 8
to 9). /T_he intuition is such nodes when added to an interval keep the original header node as the
single entry of an interval. To /f_ind a header node to form another interval, a node is picked that is
not a member of any existing intervals although it must have a (but not all) immediate predecessor
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:4 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
being a member of the interval that is just computed (Line 11 to 12). We repeat the process until
reaching the /f_ixed-point where all nodes are members of an interval.
/T_he intervals on the original graph are called the /f_irst order intervals, denoted byI1(h), and
the graph from which they were derived is called /f_irst order graphG1. Partitioning the /f_irst order
graph results in a set of /f_irst order intervals, denoted byS1 s.t. I1(h)∈S 1. By making each /f_irst
order interval into a node and each interval exit edge into an edge, the second order graph can be
derived, from which the second order intervals can also be de/f_ined. /T_he procedure can be repeated
to derive successively higher order graphs until the n-th order graph consists of a single interval.
Figure 1 illustrates such a sequence of derived graphs.
Algorithm 1:Finding intervals for a given graph
Input: a set of nodes V on a graph
Output: a set of intervals S
1 // Assume /v.alt0 is the unique entry node of the graph
2 H = {/v.alt0};
3 while H , ∅do
4 // remove next h from H
5 h = H.pop();
6 I(h) = {h};
7 // only nodes that are neither in the current interval nor any other interval will be considered
8 while {/v.alt ∈V |/v.alt < I(h)∧∄s(s ∈S∧ /v.alt ∈s)∧pre (/v.alt)⊆ I(h)}, ∅do
9 I(h) = I(h) ∪{ v };
10 // /f_ind next headers
11 while {/v.alt ∈V |∄s1(s1 ∈S∧ /v.alt ∈s1)∧∃m1,m2(m1,m2 ∈pre (/v.alt)∧m1 ∈I(h)∧m2 < I(h))},∅do
12 H = H ∪{ v };
13 S= S∪ I(h);
Fig. 1. n-th order intervals and graphs. The set of intervals onS1 are I1(1)={1}, I1(2)={2}, I1(3)={3,4,5,6},
I1(7)={7}. The set of intervals onS2 are I2(1)={1} and I2(2)={2,7,8}. I3(1)={1,9} and I4(10)={10} are the
only intervals onS3 and S4 respectively.
2.3 Graph Neural Network
Graph Neural Network (GNN) (Gori et al. 2005) is a specialized machine learning model designed to
learn from graph data. /T_he intuitive idea underlying GNN is that nodes in a graph represent objects
and edges represent their relationships. /T_hus, each node/v.alt can be a/t_tached to a vector, called state,
which collects a representation of the object denoted by /v.alt. Naturally, the state of /v.alt can be speci/f_ied
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:5
using the information provided by nodes in the neighborhood of /v.alt. Technically, there are many
realizations of this idea. Here, we describe the message-passing technique (Gilmer et al. 2017), a
widely applied method for GNN to compute the states of nodes.
We reuse the terminology of graph introduced in Section 2.1. Given a graph G =(V , E)where V
and E are the inputs, a neural message-passing GNN computes the state vector for each node in a
sequence of steps. In each step, every node /f_irst sends messages to all of its neighbors, and then
update its state vector by aggregating the information received from its neighbors.
µ(l+1)
/v.alt= ϕ({µ(l)
u }u ∈N(/v.alt)) (1)
µ(l+1)
/v.altdenotes the state of /v.alt in l + 1 steps, which is determined by the state of its neighbors in the
previous step. N(/v.alt)denotes the neighbors that are connected to /v.alt. Formally, N(/v.alt)= {u|(u,/v.alt, k)∈
Ek , ∀k ∈{1, 2, ...,K}}. ϕ(·)is a non-linear function that performs the aggregation. A/f_ter the state
of each node is computed with many rounds of message passing, a graph representation can also
be obtained through various pooling operations (e.g. average or sum).
Some GNN (Si et al . 2018) computes a separate node state w.r.t. an edge type ( i.e. µ(l+1), k
/v.alt in
Equation 2) before aggregating them into a /f_inal state (i.e. µ(l+1)
/v.altin Equation 3).
µ(l+1), k
/v.alt= ϕ1(
Õ
u ∈Nk (/v.alt)
W1µ(l)
u ), ∀k ∈{1, 2, ...,K} (2) µ(l+1)
/v.alt= ϕ2(W2[µ(l), 1
/v.alt, µ(l), 2
/v.alt, ...,µ(l), K
/v.alt]) (3)
W1 and W2 are variables to be learned, and ϕ1 and ϕ2 are some nonlinear activation functions.
Li et al. (2016) proposed Gated Graph Neural Network (GGNN) as a new variant of GNN. /T_heir
major contribution is a new instantiation of ϕ(·)using Gated Recurrent Units (Cho et al. 2014). /T_he
following equations describe how GGNN works:
ml
/v.alt=
Õ
u ∈N(/v.alt)
f (µ(l)
u ) ( 4) µ(l+1)
/v.alt= GRU (ml
/v.alt, µ(l)
/v.alt) ( 5)
To update the state of node/v.alt, Equation 4 computes a messageml
/v.altusing f (·)(e.g. a linear function)
from the states of its neighboring nodes N(/v.alt). Next, a GRU takes ml
/v.altand µ(l)
/v.alt— the current state
of node /v.alt — to compute the new state µ(l+1)
/v.alt(Equation 5).
GNN’s Weakness. Although the message-passing technique has highly empowered GNN, it
severely hinders GNN’s capability of learning pa/t_terns formed by nodes that are far apart in a
graph. From a communication standpoint, a message sent out by the start node needs to go through
many intermediate nodes en route to the end node. Due to the aggregation operation (regardless of
which implementation in Equation 1, 3, or 5 is adopted), the message gets diluted whenever it is
absorbed by an intermediate node for updating its own state. By the time the message reaches the
end node, it is too imprecise to bear any reasonable resemblance to the start node. /T_his problem
is particularly challenging in the programming domain where long-distance dependencies are
common, important properties models have to capture. Allamanis et al. (2018) designed additional
edges to connect nodes that exhibit a relationship of interest ( e.g. data or control dependency)
but are otherwise far away from each other in an AST. /T_he drawbacks are: /f_irst, their idea, albeit
conceptually appealing, has not been rigorously tested in practice, therefore, the extent to which
it addresses the scalability issue is unknown; second, they have not oﬀered a principled solution
regarding which edges to add in order to achieve the best scalability. Instead, they universally
added a prede/f_ined set of edges to any graph, an approach that is far from satisfactory.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:6 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
Fig. 2
 Fig. 3
 Fig. 4
 Fig. 5
 Fig. 6
3 METHODOLOGY AND APPROACH
Our intuition is that generalization can be made easier when models are given abstractions (of
programs) to learn. Considering its generality and capacity, we choose GNN’s learning framework
to incorporate our abstraction methodology. A key challenge arises: how to abstract programs into
favorable graph representations for GNN to learn. A natural idea would be abstracting programs
at the level of source code and then reducing the abstracted programs into graphs. On the other
hand, a simpler and perhaps more elegant approach is to abstract programs directly on their graph
representation, which this paper explores. Speci/f_ically, we propose a new graph model, called Graph
Interval Neural Network (GINN), which uses control /f_low graph to represent an input program,1
and abstracts it with three primitive operators: partitioning, heightening, and lowering. Each
primitive operator is associated with an extra operator for computing the state of each node on the
resultant graph a/f_ter the primitive operator is applied. Using the control /f_low graph depicted in
Figure 2 as our running example, we explain in details of GINN’s learning approach.
To start with, GINN uses the partitioning operator to split the graph into a set of intervals.
Figure 3 adopts diﬀerent styling of the nodes to indicate the four intervals on the graph. /T_hen,
it proceeds to compute the node states using the extra operator. Unlike the standard message-
passing mechanism underpinning the existing GNN, GINN restricts messages to be passed within
an interval. If an interval is a node itself, its state will not get updated by the extra operator. For
example, node 3, 4, 5, and 6 in Figure 3 will freely pass messages to the neighbors while evolving
their own states. On the other hand, node 1 and 2, which are alone in their respective intervals, keep
their states as they were. /T_he propagation that occurs within an interval is carried out in the same
manner as it is in the existing GNN (Equation 1, 3, or 5 depending on the actual implementation).
We formalize the partitioning operator as follows:
De/f_inition 3.1.(Partitioning) Partitioning operator, denoted by ρ, is a sequence (ρabs, ρcom) where
ρabs : Gn →Sn is a function that maps an n-th order graph into a set of n-th order intervals, and
ρcom : ∀In(h)∈S n →µ/v.alt1 , µ/v.alt2 , . . .µ/v.altT ∈Re (/v.alt1,/v.alt2, . . . ,/v.altT ∈In(h))is a function that computes a
e-dimensional vector for each node in each n-th order interval. Equation 6 de/f_inesρcom.
Depends on how many nodes In(h)is composed of, two cases to be considered:
{
µ(l+1)
/v.alt= ϕ({µ(l)
u }u ∈N(/v.alt)) /v.alt ∈In(h)∧u ∈In(h)
µ(l+1)
/v.alt= µ(l)
/v.alt /v.alt ∈In(h)∧|In(h)|= 1 (6)
Next, GINN applies the heightening operator to convert the /f_irst order graph into the second
order graph (Figure 4). In particular, the heightening operation replaces each active interval (of
1We discuss a few variations in Section 4.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:7
multiple nodes) on the lower order graph with a single node on the higher order graph (e.g. node
3, 4, 5, and 6 are replaced with node 8). In terms of the states of the freshly created nodes, GINN
initializes them to be the weighted sum of the states of replaced nodes. To motivate this design
decision, we also consider simply averaging the states of replaced nodes in Section 4.3.5 and 4.4.2.
/T_he heightening operator is de/f_ined below.
De/f_inition 3.2.(Heightening) Heightening operator, denoted byη, is a sequence (ηabs, ηcom) where
ηabs : Gn →Gn+1 is a function that maps an n-th order graph into an (n+1)-th order graph, and
ηcom : µ/v.altn
1 , µ/v.altn
2 , . . .µ/v.altn
N ∈Re →µ/v.altn+1
1
, µ/v.altn+1
2
, . . .µ/v.altn+1
N ′ ∈Re ({/v.altn
1 ,/v.altn
2 , . . ./v.altn
N }denotes the set of
nodes on Gn; {/v.altn+1
1 ,/v.altn+1
2 , . . ./v.altn+1
N ′ }denotes the set of nodes on Gn+1)is a function that maps a set
of e-dimensional vectors for each node on Gn to another set of e-dimensional vectors for each node
on Gn+1. Equation 7 de/f_inesηcom.
Like Equation 6, we consider two cases:
 

µ(0)
/v.altn+1
i
= µ(l)
/v.altn
i
|In(h)|= 1 ∧/v.altn
i ∈In(h)
µ(0)
/v.altn+1
h
= Í
/v.alt∈In (h)α/v.altµ(l)
/v.alt|In(h)|> 1 (7) α/v.alt= exp (||µ(l)
/v.alt||)
Í
/v.alt′∈In (h)
exp (||µ(l)
/v.alt′||)
(8)
given that /v.altn+1
i on the (n+1)-th order graph is the same node as /v.altn
i on the n-th order graph, and
/v.altn+1
h is the node that replaces the interval In(h). /T_he weight of each state vector,α/v.alt, is determined
based on its length (Equation 8).
Now GINN uses the partitioning operator again to produce a set of intervals from the second
order graph (Figure 5). Due to the restriction outlined earlier, messages are only passed among
node 2, 7, and 8. However, since node 8 can be deemed as a proxy of node 3˜6, this exchange, in
essence, covers all but node 1 on the /f_irst order graph. In other words, even though messages
are always exchanged within an interval, heightening abstraction expands the region an interval
covers, thus enabling the communication of nodes that were otherwise far apart.
/T_hen, GINN applies the heightening operator the second time, as a result, the third order graph
emerges (Figure 6). Since the third order graph is an interval itself, GINN computes the state of
node 1 and 9 in the same way as existing GNN using partitioning operator. Now, GINN reaches
an important point, which we call suﬃcient propagation. /T_hat is every node has passed messages
either directly or indirectly to every other reachable node in the original control /f_low graph. In
other words, the heightening operator is no longer needed.
Upon completion of the message-exchange between node 1 and 9, GINN applies for the /f_irst time
the lowering operator to convert the third order graph back to the second order graph. In particular,
nodes that were created on the higher order graphs for replacing an interval on the lower order
graph will be split back into the nodes of which the interval previously consists (e.g. node 9 to node
2, 7, and 8). /T_he idea is, a/f_ter reaching suﬃcient propagation, GINN begins to recover the state of
each node on the original control /f_low graph. We de/f_ine the lowering operator below.
De/f_inition 3.3.(Lowering) Lowering operator, denoted by λ, is a sequence ( λabs, λcom) where
λabs : Gn+1 →Gn is the inverse function of ηabs, which maps an (n+1)-th order graph into an n-th
order graph, and λcom : µ/v.altn+1
1
, µ/v.altn+1
2
, . . .µ/v.altn+1
N ′ ∈Re →µ/v.altn
1 , µ/v.altn
2 , . . .µ/v.altn
N ∈Re ({/v.altn+1
1 ,/v.altn+1
2 , . . ./v.altn+1
N ′ }
denotes the set of nodes on Gn+1; {/v.altn
1 ,/v.altn
2 , . . ./v.altn
N }denotes the set of nodes on Gn)is a function
that maps a set of e-dimensional vectors for each node on Gn+1 to another set of e-dimensional
vectors for each node on Gn. Equation 9 de/f_inesλcom.
In general, λcom can be considered as a reversing operation of ηcom:
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:8 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
 

µ(0)
/v.altn
i
= µ(l)
/v.altn+1
i
|In+1(h)|= 1 ∧/v.altn+1
i ∈In+1(h)
µ(0)
/v.alt= α/v.altµ(l)
/v.altn+1
h
∗|In+1(h)| |In+1(h)|> 1 (9)
given that /v.altn
i on the n-th order graph is the same node as /v.altn+1
i on the (n+1)-th order graph, and
/v.altn+1
h is the node that is split into a multitude of nodes, each of which is assigned α/v.alt, the weight
de/f_ined in Equation 8.|In+1(h)|is the regulation term that aims to bring node /v.alt’s initial state
and its /f_inal state (prior to being replaced by the heightening operator) within the same order of
magnitude.
As GINN revisits the second order graph, it re-applies the partitioning operator to produce
the only interval to be processed (i.e. node 2, 7, and 8). However, there lies a crucial distinction
in computing the states of node 2, 7, and 8 this time around. /T_hat is the communication among
the three nodes is no longer local to their own states, but also inclusive of node 1’s thanks to
the exchange between node 1 and 9 on the third order graph. In general, such recurring interval
propagation improves the precision of node representations by incorporating knowledge gained
about the entire graph.
Next, GINN applies the lowering operator again to recover the original control /f_low graph. A/f_ter
the partitioning operation, GINN computes the state vector for every node on the graph, signaling
the completion of a whole learning cycle. /T_he process will be repeated for a /f_ixed number of steps.
/T_hen, we use the state vectors from the last step as the node representations. We generalize the
above explanation with the running example to formalize GINN’s abstraction cycle.
De/f_inition 3.4.(Abstraction Cycle) GINN’s abstraction cycle is a sequence composed of three
primitive operators: partitioning, denoted by ρ; heightening, denoted by η; and lowering, denoted
by λ. /T_he sequence can be wri/t_ten in the form of ((ρ →η)∗ → ρ′ → (λ → ρ)∗)∗ where
|(ρ →η)∗|= |(λ →ρ)∗|. ρ′denotes the partitioning operator on the highest order graph in which
case the graph is a single interval itself.
To sum up, GINN adopts control /f_low graphs as the program representation. At /f_irst, it uses
heightening operator to increase the order of graphs, as such, involving more nodes to communicate
within an interval. Later, by applying the lowering operator, GINN restores local propagation on the
reduced order of graphs, and eventually recovers the node states on the original control /f_low graph.
Worth mentioning heightening and lowering operators are complementary and their cooperation
bene/f_its the overall GINN’s learning objective. Speci/f_ically, heightening operator enables GINN to
capture the global properties of a graph, with which lowering operator helps GINN to compute
precise node representations. As the interleaving between heightening and lowering operator goes
on, node representations will continue to be re/f_ined to re/f_lect the properties of a graph.
Abstracting the Source Code.We aim to demystify what the above graph abstraction method
translates to at the source code level, and why it helps GINN to learn. Since intervals play a pivotal
role in all three operators of the graph abstraction method, it is imperative that we understand
what an interval represents in a program. As a simple yet eﬀective measure, we use a program,
which has the same control /f_low graph as the one in Figure 2, to disclose what code construct an
interval correlates to, and more importantly, how the abstraction method works at the source code
level. /T_he program is depicted in Figure 7, which computes the starting indices of the substring
s1 in string s2. Recall the only interval GINN processed on the /f_irst order graph, the one that
consists of node 3˜6, which represents the innerdo-while loop in the function. In other words,
GINN focuses exclusively on the inner loop when learning from the /f_irst order graph. Similarly,
node 2, node 7, and node 8 consist of the only interval from which GINN learns at the second order
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:9
void substringindices Hstring sQ L
string sR L int attempts L int {} arr I
{
int m ] sQNlengthHI[
int n ] sRNlengthHI[
int i ] PLj ] PLres ] PLpos ] P[
do {
if Hpos < attemptsI {
do {
log HB begin at index : B K iI[
if HsRN charat HiKjI A]
sQN charat HjII {
res ] MQ[
log HB differs at: B K j I[}
else
log HB equal at: B K jI[
j KK[
} while Hj < mI[
}
arr{posKK} ] res[
iKK[ j ] P[
res ] pos < attempts _ i : P[
} while Hi <] n M mI[
}
void substringindices Hstring sQ L
string sR L int attempts L int {} arr I
{
int m ] sQNlengthHI[
int n ] sRNlengthHI[
int i ] PLj ] PLres ] PLpos ] P[
do {
if H pos < attempts I {
do {
log HB begin at index : B K iI[
if HsRN charat HiKjI A]
sQN charat HjII {
res ] MQ[
log HB differs at: B K j I[}
else
log HB equal at: B K jI[
j KK[
} while Hj < mI[
}
arr { pos KK} ] res [
i KK[ j ] P[
res ] pos < attempts _ i : P[
} while Hi <] n M mI[
}
void substringindices Hstring sQ L
string sR L int attempts L int {} arr I
{
int m ] sQN length HI[
int n ] sRN length HI[
int i ] PLj ] PL res ] PL pos ] P[
do {
if H pos < attempts I {
do {
log HB begin at index : B K iI[
if HsRN charat HiKjI A]
sQN charat HjII {
res ] MQ[
log HB differs at: B K j I[}
else
log HB equal at: B K jI[
j KK[
} while Hj < mI[
}
arr { pos KK} ] res [
i KK[ j ] P[
res ] pos < attempts _ i : P[
} while Hi <] n M mI[
}
Fig. 7. GINN’s focus in source code when transitioning back and forth between diﬀerent order graphs.
graph. /T_his interval happens to be the nesteddo-while loop. Finally, GINN spans across the whole
program as node 1 and node 9 cover every node in the /f_irst order graph. Figure 7 illustrates GINN’s
transition among the /f_irst, second, and third order graph, in particular, the distinct focus on each
order graph. Inspired from the exposition above, we present /T_heorem 3.5 and Corollary 3.6.
T/h.sc/e.sc/o.sc/r.sc/e.sc/m.sc 3.5 (I/n.sc/t.sc/e.sc/r.sc/v.sc/a.sc/l.sc C/o.sc/n.sc/s.sc/t.sc/i.sc/t.sc/u.sc/e.sc/n.sc/t.sc/s.sc).Given a loop structurel on a control /f_low graph,l’s loop
header, denoted byh, is the entry node of an interval that does not contain any predecessor ofh outside
of l. However, the interval contains every node inside ofl assuming l does not contain an inner loop
construct.
P/r.sc/o.sc/o.sc/f.sc.First, we prove the non-coexistence ofh and its predecessors outside ofl. Assume otherwise,
there is an intervalI containing at least one predecessor ofh outside ofl and h itself. By construction,
I already had an entry node h′. Due to the loop-forming back edge connecting node m back to h
within l, I now has two entry nodes,h′and h, ifm is not already inI, which violates the single-entry
rule of an interval. For the other possibility that if m is in I, then by construction all immediate
predecessors of m are also in I given that l does not contain an inner loop, recursively all nodes in
l are in I, meaning there will be closed paths within I that do not go through h′(i.e. those going
through h), which violates the de/f_inition of interval.
Second, we prove h is the entry node of the interval it is in. Assume otherwise, there exists an
interval I, which h is a part of, having a diﬀerent entry node h′. Since none of h’s predecessors
outside of l are part of I, at least one of them will connect to h, thus making h the other entry node
of I in addition to h′. /T_herefore, the single-entry rule of an interval is violated.
Finally, we prove that all nodes of l are in the same interval as h if l does not contain an inner
loop construct. Assume otherwise, there exists at least one node in l that is not in the interval, I,
for which h is the entry node. By construction, at least one immediate predecessor of the node
not in I is also not in I. Since l does not contain an inner loop construct, none of the nodes
in l has a predecessor outside of l. By recursion, ultimately, the loop header h will be the only
immediate predecessor of a node that is not in I. /T_herefore,h is also not in I, which contradicts the
assumption. □
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:10 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
var clazz ] classtypes {B root B}N single HI as NNN
assert N notnull H clazz I[
var first ] classtypes {B recclass B}N single HI as NNN
assert N notnull Hclazz I[
assert N equal HB string BL first N properties {B name B}N name I[
assert N false H clazz N properties {B name B}N isarray I[
Fig. 8. An example (extracted from (Allamanis et al.
2018)) of variable misuse.clazz highlighted in red is
misused. Instead,first should have been used.
public string {} fH string {} array I
{
string {} newarray ] new string { array N length }[
for H int index ] P[ index < array N length [ index KKI
newarray { array N length M index MQ} ] array { index }[
return newarray [
}
Fig. 9. An example function extracted from (Alon et al.
2019b) whose name is le/f_t out for models to predict.
The answer in this case isreverseArray.
C/o.sc/r.sc/o.sc/l.sc/l.sc/a.sc/r.sc/y.sc 3.6 (H/i.sc/e.sc/r.sc/a.sc/r.sc/c.sc/h.sc/i.sc/c.sc/a.sc/l.sc I/n.sc/t.sc/e.sc/r.sc/v.sc/a.sc/l.sc/s.sc /f.sc/o.sc/r.sc N/e.sc/s.sc/t.sc/e.sc/d.sc L/o.sc/o.sc/p.sc/s.sc).Given a nested loop structure on
a control /f_low graph, nodes of the n-th most inner loop lie in an interval on the n-th order graph.
P/r.sc/o.sc/o.sc/f.sc.Let P(n) be the statement that nodes of the n-th most inner loop lie in an interval on the
n-th order graph. We give a proof by induction on n.
Base case: Show that the statement holds for the smallest number n = 1.
According to /T_heorem 3.5, P(0) is clearly true: nodes of the most inner loop lie in an interval on the
/f_irst order graph.
Inductive step: Show that for any k ≥1, if P(k) holds, then P(k+1) also holds.
Assume the induction hypothesis that for a particular k, the single case n = k holds, meaning P(k)
is true: nodes of the k-th most inner loop lie in an interval on the k-th order graph. By de/f_inition,
nodes of the k-th most inner loop will be rendered into a single node within the (k+1)-th most inner
loop on the (k+1)-th order graph. According to /T_heorem 3.5, we deduce that nodes of the (k+1)-th
most inner loop will also lie in an interval on the (k+1)-th order graph. /T_hat is, the statement P(k+1)
also holds, establishing the inductive step.
Conclusion: Since both the base case and the inductive step have been shown, by mathematical
induction the statement P(n) holds for every number n. □
GINN’s Strengths. To conclude, the graph abstraction method translates to a loop-based ab-
straction scheme at the source code level. In particular, it sorts programs into hierarchies, where
each level presents unique looping constructs for models to learn. Since looping constructs are
integral parts of a program, this loop-based abstraction scheme helps GINN to focus on the core
of a program for learning its feature representation. In scalability regard, the graph abstraction
method also oﬀers a crucial advantage. Given the abstracted program graphs, even without the
manually designed edges, GINN mostly deals with small, concise intervals. As a result, information
rarely needs to be propagated over long distances, thus signi/f_icantly enhancing GINN’s capability
in generalizing from the large graphs, an aspect that the existing GNN struggles with.
4 EVALUATION
/T_his section presents an extensive evaluation of GINN on three PL tasks. For each task, we compare
GINN against the state-of-the-art.
4.1 Selection Criteria
To select the right downstream applications for evaluating GINN, we devise a list of criteria below
in the order of importance.
(1) We prefer tasks for which datasets are publicly accessible and baseline models are open
sourced. Furthermore, we only consider models that are built upon Tensor/f_low, a widely
used deep learning library, for reducing the engineering load.
(2) We prefer tasks solved by an in/f_luential work that is highly cited in the literature.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:11
(3) We prefer tasks where implementations of GNN-based models are publicly accessible so that
we can save our eﬀort in building the baseline with GNN. /T_he reason is it’s not suﬃcient to
only compare GINN with the state-of-the-art, which GNN may also outperform.
4.2 Evaluation Tasks
In the end, we select variable misuse prediction task (Allamanis et al. 2018) (150+ citations) in which
models aim to infer which variable should be used in a program location. As shown in Figure 8,
the variable clazz highlighted in red is misused. Instead, variable first should have been passed
as the argument of the function Assert.NotNull. At /f_irst, Allamanis et al. (2018) use GGNN to
predict the misuse variable from AST-based program graphs (more precisely AST with additional
edges). Later, the joint model proposed by (Hellendoorn et al. 2020; Vasic et al. 2019) has achieved
be/t_ter results and thus becomes the new state-of-the-art.
In addition, we select method name prediction in which models aim to infer the name of a
method when given its body. Figure 9 shows the problem se/t_ting where the function name has been
stripped for models to predict. /T_he correct answer in this case isreverseArray. Alon et al. (2019b)
(120+ citations) create the /f_irst large-scale, cross-project classi/f_ication task in method names. Later
works adopted a generative approach that generates method names as sequences of words (Alon
et al. 2019a; Fernandes et al. 2019; Wang and Su 2020).
Last, we design a new task to evaluate GINN. Granted, machine learning has inspired a distinctive
approach to a board range of problems in program analysis. However, most problems are catered to
the strength of machine learning models in mining statistical pa/t_terns from data. It’s unclear how
well they can solve the long-standing problems that are traditionally tackled by symbolic, logic-
based methods. In this task, we examine if models can accurately detect null pointer dereference, a
type of deep, semantic, and prevalent bugs in so/f_tware. In particular, we compare GINN against
not only the standard GNN but also arguably the state-of-the-art static analysis tool Facebook
Infer (Berdine et al. 2006; Calcagno et al. 2015).
4.3 Variable Misuse Prediction
/T_he state-of-the-art model for predicting variable misuse bugs is proposed by Hellendoorn et al.
(2020). /T_heir model is an instantiation of the conceptual framework invented by Vasic et al. (2019).
In this section, we /f_irst describe this framework, then we explain how to instantiate it using GINN.
4.3.1 Joint Model for Localization and Repair.Vasic et al. (2019) propose a joint model structure
that learns to localize and repair variable misuse bugs simultaneously. Intuitively, their design
exploits an important property of variable misuse bugs. /T_hat is both the misuse (i.e. incorrectly
used) and the repair (i.e. should have been used) variable have already been de/f_ined somewhere in
the program, therefore, the key is to identify their locations. Vasic et al. (2019) learned probability
distributions over the sequence of tokens of a buggy program from which they pick the token of
the highest probability to be the misuse or repair variable. At a high-level, the joint model consists
of three major components: initial embedding layer, core model, and two separate classi/f_iers. Below
we describe each component in detail.
Initial Embedding Layer.Initial embedding layer is responsible for turning each symbol from
the input vocabulary (e.g. a token or a type of AST node) into a numerical vector, the format that is
amenable to deep neural networks. A simple, crude method is to use the one-hot vectors, a N ×N
matrix representing every word in a vocabulary (N denotes the size of the vocabulary). Each row
corresponds to a word, which consists of 0s in all cells with the exception of a single 1 in a cell
used uniquely to identify the word. For example, given a vocabulary, {a, +, b, binary-exp}, the
matrix M in Equation 10 depicts the one-hot vectors of each token. A drawback of one-hot vectors
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:12 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
is they don’t capture the semantic relationship of words in the vocabulary as vectors are uniformly
sca/t_tered in a high dimensional space (i.e. the Euclidean distance between any vector with every
other vector is a constant). A common remedy is to have the one-hot vectors multiply another
matrix W1 ∈RN ×d to produce E∈ RN ×d . /T_he intuition is to expand the network’s capacity with
more learnable parameters in W1 for searching a precise embedding matrix Efor each word in
the vocabulary. d denotes the number of columns in W1, which is also the size of the embedding
vectors. Equation 10 gives an example where d equals to 3. /T_he values ofW1 is initialized randomly
and will be learned simultaneously with the network during training.
Core Model. Given the embedding vector of each token, the core model computes the numerical
representation of the input program, expressed as another matrix Cwhere each row represents
a token. Many neural architectures that can play the role of core model. Vasic et al. (2019) used
Long Short Term Memory networks (LSTM) (Hochreiter and Schmidhuber 1997), a specialization of
Recurrent Neural Networks (RNN). Hellendoorn et al. (2020) explored other alternatives including
GGNN, Transformer (Vaswani et al. 2017), the state-of-the-art sequence model, and RNN Sandwich,
a mixture of sequence and graph models, which achieves the state-of-the-art results. For now, a
core model can be considered as a black-box with an abstract interface ϕ : E→C . We defer the
discussion of two instantiations of the core model to later sections.
(10)
Classi/f_iers for Misuse and Repair Variables./T_his layer is responsible for predicting the location
of misuse and repair variable2. It distributes two probabilities to each token in the program: one
for identi/f_ication of the misuse variable and the other for the repair variable. /T_he token with the
highest probability of being the misuse or repair variable will be picked as the output. At a technical
level, Vasic et al. (2019) performed a linear transformation on matrix C∈ Rk×h using another
matrix W2 ∈Rh×2 to produce P/v.altal ∈Rk×2. As explained before, k is the number of tokens in a
program; h is the size of the embedding vector produced by a core model.
P/v.altal = CW2
Since P/v.altal is a non-normalized output, we applyso/f_tmaxfunction, a common practice in machine
learning, to map P/v.altal into probability distributions over the token sequence of an input program.
Note that axis = 1 in Equation 11 means the normalization happens along each column in which
each value denotes the probability of one token being the misuse or repair variable.
Ppro = so/f_tmax(P/v.altal , axis = 1) (11)
Finally, the model is trained to nudge Ppro as close to the true distribution of misuse and repair
variables as possible. In particular, we minimize the cross-entropy loss expressed by the diﬀerence
between Ppro and the true distributions, denoted by Loc and Rep. Similar to the one-hot vectors, Loc
(resp. Rep) assigns a value of 1 to the actual index of misuse (resp. repair) variable among all tokens
of the input program and 0 otherwise. We give the details of the loss function in Appendix A.
2We followed Hellendoorn et al. (2020)’s approach to consider a single variable misuse bug per program.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:13
4.3.2 Instantiations of the Core Model.In this section, we review two instantiations of the core
model presented in (Hellendoorn et al. 2020): GGNN and RNN Sandwich. In particular, we discuss
how each of them computes the matrix Cintroduced earlier.
As explained in Section 4.2, GGNN works with AST as the backbone of graphs. To collect the
numerical representations of each token in the program, Hellendoorn et al. (2020) extracted states
of terminals nodes a/f_ter GGNN had completed the message-passing routine, and then stacked them
into matrix C(Equation 12).
C= stack([µ/v.alt1 , . . . ,µ/v.altn ], axis = 0), ∀/v.alti ∈Σ (12)
where Σ denotes the set of terminal nodes in AST; axis = 0 indicates each µ/v.altforms a row in C.
Next, we discuss RNN Sandwich (Hellendoorn et al . 2020), the state-of-the-art model in pre-
dicting variable misuse bugs. In a nutshell, RNN Sandwich adopts a hybrid neural architecture
combining a sequence model RNN and a graph model GGNN intending to get the best of both
worlds. /T_hat is, not only exploiting the semantic code structure using GGNN but also streamlining
information /f_low through token sequences using RNN. Technically, the way it works is the follow-
ing: (1) /f_irst, the matrixEproduced in the initial embedding layer will be fed into an RNN. /T_he
results are a list of vectors, h1, . . . ,hn, each of which represents a token. Compared to the initial
embedding matrix E, h1, . . . ,hn increases the precision of the token representation by capturing
the temporal properties they display in a sequence. Equation 13 computes h1, . . . ,hn. In simplest
terms, RNN takes two vectors at each time step — the embedding vector of the t-th token in the
sequence, denoted by E[t](assuming the t-th row in Eis the embedding of the t-th token), and
RNN’s current hidden state a/f_ter consuming the embedding of the previous token. /T_he output is
the new hidden state ht . For interested readers, Appendix B provides a more detailed description
ht = RNN(E[t], ht−1) (13)
of RNN’s computation model. (2) Next, GGNN takes over and computes the state vector for each
node via the message-passing protocol. For node initialization, the token representations computed
in Step (1) are assigned to the terminal nodes in an AST (Equation 14) while the non-terminal nodes
µ/v.alti = hi , ∀/v.alti ∈Σ, ∀hi ∈[h1, . . . ,hn] (14)
keep their representations computed from the initial embedding layer. (3) Finally, a/f_ter GGNN
has computed the state of each node in an AST, RNN takes back those of the terminal nodes and
computes a new representation for each token, h′
1, . . . ,h′
n, according to Equation 15 where µ/v.altt
denotes the state vector of the node corresponding to the t-th token in the program. Finally, the
h′
t = RNN(µ/v.altt , h′
t−1) (15)
matrix Ccan be computed by Equation 16.
C= stack([h′
1, . . . ,h′
n], axis = 0) (16)
4.3.3 New Instantiations of Core Models Using GINN.We propose two new instantiations built
upon GINN to pair with GGNN and RNN Sandwich. In order to adapt GINN as an instantiation of
the core model, an important issue needs to be addressed. Recall the interface de/f_ined for the core
model: ϕ : E→C , since GINN can not compute representations of tokens from a standard control
/f_low graph, matrixCcan’t be produced. /T_herefore, we modify control /f_low graphs as follows: (1)
/f_irst we split a graph node representing a basic block into multiple nodes, each of which represents
a single statement. Subsequently, we add additional edges to connect every statement with its
immediate successor within the same basic block. For edges on the original control /f_low graphs,
we change their start (resp. end) nodes from a basic block to its last (resp. /f_irst) statement a/f_ter the
split; (2) next, we replace each statement node with a sequence of nodes, each of which represents
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:14 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
a token of the statement. Speci/f_ically, every token node is connected to its immediate successor,
and the /f_irst token node will become the new start or end node of edges that were connecting the
statement nodes before.
Figure 10b depicts an example of the new graph representation given the function in Figure 10a.
Note that our modi/f_ication will not alter the partitioning of a graph into intervals. Instead, we
only replace nodes denoting basic blocks with those denoting token as the new constituents of
an interval. We have included a detailed explanation in Appendix C for readers/f_i perusal. Under
the new graph representation, GINN can compute the state vector for every token in the program,
and in turn the matrix C. /T_herefore, we can now instantiate the core model with GINN. Similarly,
we can easily construct a variant of RNN Sandwich by swapping out GGNN for GINN as the new
graph component.
static bool findindexofmaxelement H int {} arr I
{
int max ] int N minvalue L idx ] P[
for H int i ] P[ i < arr N length [ i KKI
{
if H arr {i} > max I
{
max ] arr {i}[
idx ] i[
}
}
return idx [
}
(a)
 (b)
Fig. 10. The program graph in (b) that represents the function in (a). Edges with hallow arrow heads represent
the control flows between program statements. The other edges connect tokens in a sequence.
4.3.4 Experimentation. Given the conceptual framework of the joint model and various instanti-
ations discussed above, we now describe the experiment setup.
Dataset, Metric, and Baseline.Hellendoorn et al. (2020) worked with ETH Py150 (Raychev et al.
2016), a publicly accessible dataset of python programs, on top of which they introduced buggy
programs to suit the task of variable misuse prediction. Since the actual dataset used in their
experiment is not publicly available, we follow their pre-processing steps in an a/t_tempt to replicate
their dataset. Major tasks include the deduplication of ETH Py150, generation of buggy programs,
and preservation of original programs as correct examples. Like their dataset, we maintain a balance
between buggy and correct programs. In the end, we have 3M programs in total, among which
We use 2M for training, 245K for validation, and 755K for test. A detailed description of the data
generation is provided in Appendix D. Similarly, we adopt the metrics proposed in (Vasic et al .
2019) to measure the model performance, such as (1) classi/f_ication accuracy, the percentage of
programs in the test set that are correctly classi/f_ied as either buggy or correct; (2) localization
accuracy, the percentage of buggy programs for which the bug location is correctly predicted; and
(3) localization+repair accuracy (or joint accuracy), the percentage of buggy programs for which
both the location and repair are correctly predicted. Regarding baselines, we use GGNN and RNN
Sandwich as the two instantiations of the core model.
Implementation. Since ETH Py150 provides AST for programs in the dataset, we convert each AST
into the graph representation that GINN consumes (Appendix E). We also implement Algorithm 1
to partition the graphs into intervals. Since prior works have not open-sourced the joint model
framework, we implemented our own in Tensor/f_low. Regarding the core models, we use Allamanis
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:15
et al. (2018)’s implementation of GGNN, on top of which we realize GINN’s graph abstraction
method while keeping all GGNN’s default parameters (e.g. size of node embeddings, number of
layers, etc.) intact. We only implement one abstraction cycle of GINN’s (original graph→highest
order graphs→original graphs) as more cycles do not make a signi/f_icant diﬀerence. For RNN
Sandwich, we use the large sandwich model (Hellendoorn et al . 2020), which is shown to work
be/t_ter than the vanilla model presented in Section 4.3.2. /T_he improvement is due to the increased
frequency of the alternation between sequence and graph models. /T_hat is, instead of inserting
RNN only before and a/f_ter GNN’s computation, they wrap every few rounds of message-passing
inside of GNN with an RNN to facilitate a thorough exchange between two neural architectures.
We couple GINN (resp. GGNN) with an RNN to implement the GINN- (resp. GGNN-) powered RNN
Sandwich. Note that Hellendoorn et al. (2020) also propose a lightweight graph representation by
keeping only the terminal nodes from an AST as the input for both GGNN and RNN Sandwich.
Even though these new program graphs have made models faster to train, their accuracy o/f_ten
decreases due to the lesser information contained in the input graphs. Hence, we don’t consider
this lightweight graphs in our evaluation.
Table 1. Results for the diﬀerent instantiations of core models. For GINN and GINN-powered RNN Sandwich,
we also include their results of an ablation study and an evaluation on an alternative design.
Models Con/f_iguration Classi/f_ication
Accuracy
Localization
Accuracy
Localization+Repair
Accuracy
GGNN Original 74.0% 58.1% 56.0%
GINN
Original 74.9% 60.5% 60.0%
Ablated 74.7% 58.8% 57.6%
Alternative 74.0% 59.5% 58.7%
Sandwich (GGNN) Original 74.7% 62.9% 61.1%
Sandwich (GINN)
Original 75.8% 69.3% 68.4%
Ablated 75.1% 62.4% 62.3%
Alternative 75.0% 63.1% 62.0%
All experiments including those to be presented later are performed on a desktop that runs
Ubuntu 16.04 having 3.7GHz i7-8700K CPU, 32GB RAM, and NVIDIA GTX 1080 GPU. As a pre-test,
we repeat the experiment presented in (Hellendoorn et al. 2020), and observed comparable model
performance for both GGNN and GGNN-based RNN Sandwich, signaling the validity of both our
model implementation and data curation. Interested readers may refer to Appendix F for details.
4.3.5 Results. First, we compare the performance of GINN (resp. GINN-powered RNN Sandwich)
against GGNN (resp. GGNN-powered RNN Sandwich). /T_hen we investigate the scalability of these
four neural architectures. Later, we conduct ablation studies to gain a deeper understanding of
GINN’s inner workings. Finally, we evaluate an alternative design of GINN.
Accuracy. Table 1 depicts the results of all four models using the aforementioned metrics (Appen-
dix G shows their training time). We focus on rows for the original con/f_iguration of each model,
and discuss the rest later. Regarding the classi/f_ication accuracy, all models perform reasonably
well, and GINN-powered RNN Sandwich and GINN are the top two core models despite by a small
margin. For more challenging tasks like localization or repair, GINN-powered RNN Sandwich now
displays a signi/f_icant advantage over all other models, in particular, it outperforms GGNN-powered
RNN Sandwich, the state-of-the-art model in variable misuse prediction, by 6.4% in localization and
7.3% in joint accuracy. To dig deeper, We manually inspect the predictions made by each model and
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:16 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
/f_ind that GINN-powered RNN Sandwich is considerably more precise at reasoning the semantics of
a program. Below we gave two examples to illustrate our /f_indings.
For Figure 11a, GINN-powered RNN Sandwich is the only model that not only locates but also
/f_ixes the misused variable (i.e. orig sys path highlighted within the shadow box). In contrast,
all baseline models consider orig sys path as the correctly used variable, which is not totally
unreasonable. In fact, appending an item (item) when it is not yet in the list (orig sys path) is a
very common pa/t_tern models need to learn. However, in this case, iforig sys path was the correct
variable, new sys path would have been an empty list when being assigned to sys.path[:0] in
the last line. GINN-powered RNN Sandwich is capable of capturing the nuance of the semantics
this program denotes, and not ge/t_ting trapped by the common programming paradigms. As for
Figure 11b, GINN and GGNN-powered RNN Sandwich also correctly localizes the misused variable
request, but none of them predicts the right repair variable sq. /T_he signal there is the fact that
fts is a list, which does not have keys() method. GINN-powered RNN Sandwich is again the
only neural architecture that produces the correct prediction end-to-end, demonstrating its higher
precision in reasoning the semantics of a program.
def add_vendor_lib HI:
orig_sys_path ] set H sys N path I
new_sys_path ] {}
for item in list H sys N path I:
if item not in orig_sys_path :
orig_sys_path N append H item I
sys N path N remove H item I
sys N path {:P} ] new_sys_path
(a)
def list_stored_queries Hself L request I:
sq ] super H geodjangowfsadapter L
self IN list_stored_queries H request I
fts ] list H self N models N keys HII
for k in request N keys HI:
sq{k} ] storedquerydescription H name ]kL feature_types ]fts L
title ]kL parameters ]{}I
return sq
(b)
Fig. 11. Two programs only GINN-powered RNN Sandwich makes the correct prediction end-to-end.
Scalability. We further analyze the results we obtained from the previous experiment to investigate
the scalability of the four neural architectures. In particular, we divide the entire test set into ten
subsets, each of which consists of graphs of similar size. We then record the performance of
all models on each subset starting from the smallest to the largest graphs. Note that GGNN,
at the core of both compared baselines, has already had additional edges incorporated into the
program graphs to alleviate the scalability concern (Allamanis et al. 2018). However, as explained
in Section 1, Allamanis et al. (2018) do not provide a principled guideline as to where exactly to
add the edges given a program graph speci/f_ically. /T_herefore, we can only follow their approach by
universally adding all types of edges they proposed (9 in total which we list in Appendix H) to all
program graphs. In contrast, we do not incorporate any additional edge apart from those that are
present in the graph representation for GINN.
Figure 12 shows how the performance of each model varies with the size of graphs. We skip the
metric of classi/f_ication accuracy under which models are largely indistinguishable. For localization
and joint accuracy, we make several observations of the performance trend for each model. First,
GGNN suﬀers the largest performance drop among all models — more than 10% ( resp. 20%) under
location (resp. joint) accuracy. /T_his phenomenon shows there is indeed a scalability issue with
GGNN. Second, the combination of sequence and graph model helps to make the neural architecture
more scalable than GGNN alone. Furthermore, by replacing GGNN with GINN in RNN Sandwich,
we obtain a model that is clearly the most scalable. Barring few sets of graphs, GINN-powered RNN
Sandwich is considerably more accurate than any other model, especially under the joint accuracy.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:17
1 2 3 4 5 6 7 8 9 100.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
N-th Set (Ranked by Size of Graphs)
Classification Accuracy
GGNN GINN
Sand I-Sand
1 2 3 4 5 6 7 8 9 100.4
0.5
0.6
0.7
0.8
0.9
1
1.1
N-th Set (Ranked by Size of Graphs)
Location Accuracy
GGNN GINN
Sand I-Sand
1 2 3 4 5 6 7 8 9 100.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
N-th Set (Ranked by Size of Graphs)
Location+Repair Accuracy
GGNN GINN
Sand I-Sand
Fig. 12. Investigating the scalability of all four neural architectures. I-Sand (resp. Sand) denotes the GINN-
powered RNN Sandwich (resp. GGNN-powered RNN Sandwich).
Finally, GINN alone also manages to improve the scalability of GGNN. As described earlier, the
improvement is magni/f_ied under the assistance of the RNN in the sandwich model.
Ablation Study.Unlike the prior work (Allamanis et al. 2018), all edges in our graph representation
are indispensable, thus can not be ablated. /T_herefore, the goal of this ablation study is to quantify
the in/f_luence of the graph abstraction method, the crux of GINN’s learning approach, on GINN’s
performance. Because GINN is identical to GGNN without the graph abstraction operators, we
evaluate GGNN when GINN’s program graphs — a variant of control /f_low graph — are provided
as input data. Rows for ablated con/f_iguration in Table 1 show GINN (resp. GINN-powered RNN
Sandwich) becomes only slightly more accurate than GGNN (resp. GGNN-powered RNN Sandwich)
a/f_ter the ablation, in other words, our graph abstraction method indeed helps models to learn.
Alternative Design. As mentioned in Section 3, we evaluate an alternative design of GINN in
which freshly created nodes — due to the heightening operator — are initialized to be the average
of the replaced nodes; conversely, for nodes that are created by the lowering operator, they receive
an equal share of the split node that they emerge from. In essence, we assign the weight, α/v.alt, used
in Equation 7 and 9 to be 1/|In
h |. Rows of alternative con/f_iguration in Table 1 show that the new
design results in a notable decrease in model accuracy, which motivates the original design of
GINN.
4.4 Method Name Prediction
/T_he state-of-the-art model in method name prediction, sequence GNN (Fernandes et al. 2019),
adopts a typical encoder-decoder architecture (Cho et al . 2014; Devlin et al. 2014).3 In a similar
vein to RNN Sandwich, sequence GNN also employs a combination of graph and sequence model
to encode a program, however, the synergy between the two neural architectures in this case is
considerably simpler. /T_hey only perform the /f_irst two steps of the vanilla joint model presented
in Section 4.3.2. /T_hat is, RNN /f_irst learns the sequence representation of each token in a program
before GGNN computes the state for every node in the AST. For decoder, they use another RNN,
which generates the method name as a sequence of words. While decoding, it also a/t_tends to the
states of nodes in the AST, a common technique for improving the accuracy of models adopting
encoder-decoder architecture (Bahdanau et al. 2015).
Instantiating the framework of sequence GNN’s with GINN is a straightforward task. We use
GINN as the new graph model for the encoder while keeping the remaining parts of the framework
3Wang and Su (2020) proposed a model to solve the same problem. Since (1) they did not perform a head-to-head comparison
against sequence GNN; and (2) their methodology requires program executions, we do not consider their model for this task.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:18 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
private intextractmodifiersH groovysourceast ast I {
groovysourceast modifiers ] ast N childoftype H modifiers I[
if H modifiers ]] null I return P[
int modifierflags ] P[
for H groovysourceast child ]
H groovysourceast I modifiers N getfirstchild HI[
child A] null [
child ] H groovysourceast I child N getnextsibling HII {
switch H child N gettype HII {
case literal_private :
modifierflags |] modifier N private [ break [
case literal_protected :
modifierflags |] modifier N protected [ break [
case literal_public :
modifierflags |] modifier N public [ break [
case final :
modifierflags |] modifier N final [ break [
case literal_static :
modifierflags |] modifier N static [ break [
}}
return modifierflags [
}
(a) Predicted to beSelectModifierby the baseline.
public void applyrulesH@nullable string pluginid L class <_ > clazz I {
modelregistry modelregistry ] target N getmodelregistry HI[
iterable < class <_ extends rulesource >> declaredsources ]
ruledetector N getdeclaredsources H clazz I[
for H class <_ extends rulesource > rulesource : declaredsources I {
iterable < extractedmodelrule > rules ]
ruleinspector N extract H rulesource IN getrules HI[
for H extractedmodelrule rule : rules I {
for H class <_ > dependency : rule N getruledependencies HII {
target N getpluginmanager HIN apply H dependency I[
}
rule N apply H modelregistry L modelpath N root I[
}
}
}
(b) Predicted to beCheckSourceby the baseline.
Fig. 13. Two programs sequence GINN predicts correctly but not the baseline.
intact. Like prior experiments, GINN works with the same graph format derived from control /f_low
graphs and customized to include nodes of tokens.
4.4.1 Experimentation. Given the sequence GNN’s framework and the instantiation with GINN,
we describe the setup of our experiment.
Dataset, Metric, and Baseline.We consider Java-small proposed by Alon et al. (2019a), a publicly
available dataset used in (Fernandes et al. 2019), adopting the same train-validation-test splits they
have de/f_ined. We also follow Fernandes et al. (2019)’s approach to measure performance using F1
and ROGUE score over the generated words (Appendix I). For sequence GNN baseline, we use the
model that achieves state-of-the-art results on Java-small, which employs bidirectional-LSTM as
the sequence and GGNN as the graph model for encoder and another LSTM for decoder. Apart
from the aforementioned a/t_tention mechanism, decoder also integrates a pointer network (Vinyals
et al. 2015) to directly copy tokens from the input program.
Implementation. We use JavaParser to extract AST out of programs in Java-small. We adopt
the same procedure as before to convert AST into the graph representation that GINN consumes.
Regarding the model implementation, we use the model code of sequence GNN open-sourced
on GitHub, within which we implemented GINN’s graph abstraction methods. Like before, all
default model parameters in their implementation are kept as they are and we only implement one
abstraction cycle within GINN. We name our model built out of GINN sequence GINN.
4.4.2 Results. In this section, we /f_irst compare the performance of sequence GINN against
sequence GNN. /T_hen we investigate the scalability of both neural architectures. Finally, we conduct
a similar ablation study and alternative design evaluation with those presented in the previous task.
Accuracy. Table 2 depicts the results of the two models. Sequence GINN signi/f_icantly outperforms
the sequence GNN across all metrics (e.g. close to 10% in F1). /T_hrough our manual inspection, we
/f_ind that sequence GINN’s strength stands out when dealing with programs of greater complexity.
Take programs in Figure 13 as examples, even though both of them denote relatively simple
semantics, sequence GNN struggles with their complexity (e.g. large program size for Figure 13a4,
and nested looping construct for Figure 13b). /T_herefore, its predictions do not precisely re/f_lect
the semantics of either program. sequence GNN predicts the name of the program in Figure 13a
4/T_he average size of the AST in Java-small is around 100 while this program has 150+ nodes in its AST.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:19
(resp. 13b) to be SelectModifier (resp. CheckSoucre) whereas sequence GINN produces correct
predictions for both programs, which are highlighted within the shadow boxes.
Table 2. The first two rows show the results for sequence GINN and sequence GNN; next two rows are the
results of the ablation study and the evaluation on alternative design.
Models Con/f_iguration F1 ROGUE-2 ROGUE-L
Sequence GNN Original 51.3% 24.9% 49.8%
Sequence GINN
Original 60.2% 29.5% 54.7%
Ablated 56.3% 26.3% 51.6%
Alternative 52.2% 24.8% 50.4%
Scalability. In the same setup as the scalability experiment of variable misuse prediction task,
we investigate the scalability of both models. To strengthen sequence GNN, we incorporate all
additional edges to its program graphs (Allamanis et al. 2018). As depicted in Figure 14, under all
three metrics, sequence GINN outperforms sequence GNN throughout the entire test set, especially
in F1 where sequence GINN displays a wider margin over sequence GNN on larger graphs (i.e. last
5 sets of the graphs except the seventh set) than it does on smaller graphs (i.e. /f_irst 5 sets of graphs).
1 2 3 4 5 6 7 8 9 100
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
N-th Set (Ranked by Size of Graphs)
F1
Baseline GINN
1 2 3 4 5 6 7 8 9 100
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
N-th Set (Ranked by Size of Graphs)
ROGUE-2
Baseline GINN
1 2 3 4 5 6 7 8 9 100
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
N-th Set (Ranked by Size of Graphs)
ROGUE-L
Baseline GINN
Fig. 14. Investigating the scalability of all four neural architectures.
Ablation Study. We conduct the same ablation study to quantify the in/f_luence of the graph
abstraction method on the performance of sequence GINN. As shown in Table 2, without the graph
abstraction operators, sequence GINN still performs somewhat be/t_ter than the baseline but notably
worse than the original con/f_iguration, indicating the considerable in/f_luence of the abstraction
method on sequence GINN’s performance.
Alternative Design. Following the evaluation of the alternative design in variable misuse predic-
tion, we adopt the same approach to initializing the state of freshly created nodes. /T_he last row in
Table 2 shows that this design causes GINN to be less accurate by a fairly wide margin.
4.5 Detection of Null Pointer Dereference
First, we present a framework for creating neural bug detectors in catching null pointer dereference.
/T_hen, we describe our experiment setup. Finally, we report our evaluation results.
Neural Bug Detection Framework.We devise a simpler program representation for this task.
Given a control /f_low graph, we only split a node of basic block into multiple nodes of statement
to aid the localization of bugs at the level of lines. When checking a method, we also include its
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:20 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
context by stitching the graphs of all its callers and callees to that of itself. Regarding the initial
state of each statement node, we use RNN to learn a representation of its token sequence. As
explained in Section 4.3.1, a/f_ter each token is embedded into a numerical vector (by the initial
embedding layer), we feed the embeddings of all tokens in a sequence to the network whose /f_inal
hidden state (i.e. the last ht in Equation 13) will be extracted as the initial node state.
A/f_ter GINN computes the state vector for each node, and in turn the entire graph (by aggregating
the states of all nodes), we train a feed-forward neural network (Svozil et al. 1997) to predict (using
the representation of the entire graph) whether or not a method is buggy. If it is, we will use the
same network to predict the bug locations. Our rationale is a graph as a whole provides stronger
signals for models to determine the correctness of a method. Correct methods will be refrained
from further predictions, leading to fewer false warnings. On the other hand, if a method is indeed
predicted to be buggy, our model will then pick the top N statements ranked by their probabilities
as potential bug locations. /T_his prediction mode intends to provide /f_lexibility for developers to
tune their analysis toward either producing less false warnings or identifying more bugs.
Dataset. We target Java code due to the popularity and availability of Java datasets. In addition to
bugs provided by existing datasets (Just et al. 2014; Saha et al. 2018; Ye et al. 2014), we extract addi-
tional bugs from Bugswarm projects (Tomassi et al. 2019) to enrich our dataset for this experiment.
For example, given a bug description ”Issue #2600…” contained in a commit message, we search
in the bug tracking system (e.g. Bugzilla) using id #2600 to retrieve the details of the bug, against
which we match pre-de/f_ined keywords to determine if it’s a null pointer dereference bug. Next,
we refer back to the commit to /f_ind out bug locations, speci/f_ically, we consider the lines that are
modi/f_ied by this commit to be buggy and the rest to be clean. We acknowledge the location of a
bug and its /f_ix may not be precisely the same thing. If there are ever cases where two locations
signi/f_icantly diﬀer, GINN and the compared baseline will be equally aﬀected.
Any detection is in nature a classi/f_ication problem having subjects of interest one class of the
data, and the rest the other. /T_herefore, in the context of bug detection, it’s also necessary to supply
the correct methods apart from the buggy methods for training. A natural approach would be
directly taking the /f_ixed version of each buggy method. However, this approach is unlikely to work
well in practice. Because in a real problem se/t_ting, an analyzer will never set out to diﬀerentiate
the two versions (i.e. correct and buggy) of the same program, instead, it has to separate buggy
programs from correct programs that are almost guaranteed to be functionally diﬀerent. For this
reason, we pair each buggy method with a correct method that is syntactically the closest (de/f_ined
by the tree edit distance between their AST) from the same project. However, due to the shortage
of bug instances in existing datasets even a/f_ter taking into account the additional bugs we extracted
ourselves, we include more correct methodsw.r.t.each buggy method to further enlarge our dataset.
Table 7 in Appendix J shows the details including the projects from which we extract the code
snippet, a brief description of their functionality, size of their codebase, and the number of buggy
methods we extract from each project. We maintain an approximately 3:1 ratio between the correct
and buggy methods for each project. In total, we use 64 projects for training and 13 for test.
Objective, Metric, and Baseline.We evaluate how accurately the neural bug detector can localize
a null pointer dereference within a method. Because warning developers about buggy functions
hardly makes a valuable tool. As we deal with an unbalanced dataset (i.e. cleaning lines are more
than buggy lines), we forgo the accuracy metric since a bug detector predicting all lines to be correct
would yield a decent accuracy but make a totally useless tool. Instead, we opt for precision, recall,
and F1 score (Equation 18–20 in Appendix K), metrics are commonly used in defect prediction
literature (Pradel and Sen 2018; Wang et al. 2016). To show our interval-based graph abstraction
method can improve a variety of graph models, we use the standard GNN to build a baseline bug
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:21
Table 3. Precision.
Methods Top-1 Top-3 Top-5
GNN (0) 0.209 0.147 0.117
GNN (1) 0.285 0.166 0.131
GNN (2) 0.218 0.138 0.101
GINN (0) 0.326 0.174 0.138
GINN (1) 0.351 0.167 0.130
GINN (2) 0.305 0.195 0.136
Gain 0.351- 0.285= 0.066
Table 4. Recall.
Methods Top-1 Top-3 Top-5
GNN (0) 0.243 0.373 0.450
GNN (1) 0.171 0.252 0.302
GNN (2) 0.210 0.337 0.375
GINN (0) 0.282 0.374 0.447
GINN (1) 0.329 0.431 0.507
GINN (2) 0.304 0.428 0.500
Gain 0.507- 0.450= 0.057
Table 5. F1 score.
Methods Top-1 Top-3 Top-5
GNN (0) 0.224 0.211 0.186
GNN (1) 0.214 0.201 0.183
GNN (2) 0.214 0.196 0.159
GINN (0) 0.303 0.238 0.211
GINN (1) 0.339 0.241 0.207
GINN (2) 0.304 0.268 0.214
Gain 0.339- 0.224= 0.115
detector for this experiment. We do not include classical static analysis tools as additional baselines
since comparing them against learning-based bug detectors is likely to be unfair due to the noise
issue raised earlier. /T_hat is static analyzers could very well discover hidden bugs or report diﬀerent
locations of the same bug. Properly handling those situations require manual inspection, which is
hard to scale. On the other hand, machine learning models are less susceptible to this problem as
the training set yield in principle the same distribution of the test set.
Implementation. We construct the control /f_low and interval graphs using Spoon (Pawlak et al.
2015), an open-source library for analyzing Java source code. To eﬃciently choose correct methods
to pair with a buggy method, we run DECKARD (Jiang et al. 2007), a clone detection tool adopting
an approximation of the shortest tree edit distance algorithm to identify similar code snippets.
To realize the neural bug detector powered by GNN, we make two major changes to Allamanis
et al. (2018)’s implementation of GGNN. First, we adopt the method de/f_ined in Equation 2 and 3
for updating the states of nodes in the message-passing procedure. Second, we design two loss
functions both in the form of cross-entropy for the bug detection task. /T_he /f_irst one is designed
for the classi/f_ication of buggy and correct methods and the other for the classi/f_ication of buggy
and clean lines inside of a buggy method. Next, we realize the graph abstraction method within
the implementation of GNN-powered bug detector to build GINN-powered bug detector. Similar
to prior tasks, we implement only one abstraction cycle for GINN. For the remaining GGNN’s
parameters a/f_ter our re-implementation, we keep them in both bug detectors.
Performance. Table 3–5 depict the precision, recall, and F1 for both bug detectors. We also
examine the impact of the program context — denoted by the number in parenthesis — on the
performance of each bug detector. More concretely, (0) means each method will be handled by itself.
(1) stitches control /f_low graphs of all callers and callees (denoted byF) to that of the target method,
and (2) further integrates the graphs of all callers and callees of every method in F. Exceeding 2
has costly consequences. First, most data points will have to be /f_iltered out for over/f_lowing the
GPU memory. Moreover, a signi/f_icant portion of the remaining ones would also have to be placed
in a batch on its own, resulted in a dramatic increase in training time. We provide the same amount
of context for each method in the test set as we do in the training set to maintain a consistent
distribution across the entire dataset. Columns in each table correspond to three prediction modes
in which models pick 1, 3, or 5 statements to be buggy a/f_ter a method is predicted buggy.
Overall, the neural bug detector built out of GINN consistently outperforms the baseline using
all proposed metrics, especially in F1 score where GINN beats GNN by more than 10%. Regarding
the impact of the context, we /f_ind that more context mostly but not always leads to improved
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:22 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
performance of either bug detector. /T_he reason is, on one hand, more information will always be
bene/f_icial. On the other hand, exceedingly large graphs hinders the generalization of graph models,
resulted in the degraded performance of the neural bug detectors.
Scalability. Similar to the prior studies, we investigate the scalability of both neural bug detectors.
Again, we have strengthened the baseline by adding extra edges to its program graphs (Allamanis
et al. 2018). We /f_ix the context for each method to be 2, which provides the largest variation in
graph size. Figure 15 depicts the precision, recall, and F1 score for GINN and the baseline under
top-1 prediction mode (top-3 and top-5 yield similar results). Even though GINN was beaten by
GNN on few subsets of small graphs, it consistently outperforms the baseline as the size of graphs
increases. Overall, we conclude bug detectors built out of GINN are more scalable.
1 2 3 4 5 6 7 8 9 100
0.1
0.2
0.3
0.4
0.5
0.6
0.7
N-th Set (Ranked by Size of Graphs)
Top-1 Precision
GNN GINN
1 2 3 4 5 6 7 8 9 100
0.1
0.2
0.3
0.4
0.5
0.6
0.7
N-th Set (Ranked by Size of Graphs)
Top-1 Recall
GNN GINN
1 2 3 4 5 6 7 8 9 100
0.1
0.2
0.3
0.4
0.5
0.6
0.7
N-th Set (Ranked by Size of Graphs)
Top-1 F1 Score
GNN GINN
Fig. 15. Investigating the scalability of baseline and GINN.
Catching Bugs in the Wild.We conduct another study to test if the GINN-based bug detector
can discover new bugs in the real world. We set up both GINN-based bug detector (previously
trained on 64 projects and con/f_igured with top-1 prediction mode) and Facebook Infer (Berdine
et al. 2006; Calcagno et al. 2015), arguably the state-of-the-art static analysis tool for Java code, to
scan the codebase of 20 projects on GitHub that are highly starred and actively maintained. Note
that although Infer is sound by design (based on abstract interpretation (Cousot and Cousot 1977)),
it makes engineering compromises for the sake of usability, thus triggering both false positives
and false negatives in practice. On the other hand, GINN-powered bug detector is by nature an
unsound tool, therefore we only study the utility of both tools in engineering terms.
A/f_ter manually inspecting all alarms produced by both tools, We con/f_irmed 38 bugs out of 102
alarms GINN-based bug detector raises, which rounds up to a precision of 37.3%. In comparison,
Facebook Infer emi/t_ted 129 alarms out of which 34 are con/f_irmed (i.e. precision = 26.3% ). To
understand why our bug detector is more precise, we make several important observations about
the behavior of both tools through our manual inspection. All examples are included in Appendix L.
First, Infer in general raised more alarms that are provably false (e.g. Figure 19, 21, and 23) and
missed more bugs that are demonstrably real (e.g. Figure 25 and 26, both of which have been /f_ixed
a/f_ter our reporting) than GINN-based bug detector. Second, Infer raised many alarms due to its
confusion of some common APIs (e.g. Figure 27 and 29). /T_hird, regarding path-sensitivity, there’s
still much room for infer to improve. For example, the path along which the bug is reported by
Infer (Figure 31) is clearly unsatis/f_iable. Speci/f_ically, the branch (Line 112) that triggers the null
pointer exception is guarded against the same condition under which the supposed null pointer is
in fact instantiated properly (Line 100). We have manually veri/f_ied that no code in between changes
the evaluation of the path condition at Line 111. Unlike Facebook Infer, GINN-based bug detector
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:23
searches for bug pa/t_terns it learned from the training data. Because a large number of the false
alarms Infer reports clearly deviate from the norm exhibited in the training data, GINN-based bug
detector is be/t_ter at suppressing them, resulted in a higher precision.
We also look into wrong predictions our bug detector made, and /f_ind that it tends to struggle
when a bug pa/t_tern is not local — the dereference is quite distant from where the null pointer is
assigned. We conjecture that the cause is the signal networks receive gets weaker when a sca/t_tered
bug pa/t_tern is overwhelmed by irrelevant statements. To address this issue, we apply the slicing
technique (Weiser 1981) and obtain encouraging results in a preliminary study. By simply slicing
the graph of each tested method, the same model (without retraining) can now detect 39 bugs out
of 93 alarms, a moderate improvement over the previous result. We plan to test this idea out more
systematically and at a larger scale.
We reported the con/f_irmed bugs (i.e. 38 in total mentioned above) that GINN-based bug detector
found, out of which 11 are /f_ixed and another 12 are con/f_irmed (/f_ixes pending). Reasons for
uncon/f_irmed bugs are: (1) no response from developers; (2) requiring actual inputs that trigger the
reported bugs; or (3) third party code developers did not write themselves. Our /f_indings show that
GINN-based bug detector is capable of catching new bugs in real-world Java projects.
4.6 Discussion
We summarize the lessons learned from our extensive evaluation. First and foremost, the interval-
based graph abstraction method has shown to improve the generalization of various graph-based
models (e.g. standard GNN, GGNN, RNN Sandwich) across all three downstream PL tasks. In each
task, the GINN-based model outperforms a GNN-based model — on a dataset that the GNN-based
model achieves state-of-the-art results — by around 10% in a metric adopted by the GNN-based
model. Based on the collective experience of the community (Appendix M), it is evident GINN has
signi/f_icantly advanced the state-of-the-art GNN as a general, powerful model in learning semantic
program embeddings. Second, our evaluation reveals the scalability issues existing graph models
suﬀer from — manifested in the signi/f_icant performance drop against the increasing size of graphs
— and suggests an eﬀective solution built out of the graph abstraction method. In fact, the overall
accuracy improvement mentioned earlier is in large part a/t_tributed to GINN’s eﬃcient handling
of the large graphs. Finally, our evaluation also shows model-based static analyzers can be a
promising alternative to the classical static analysis tools for catching null pointer dereference bugs.
Even though our results are still preliminary for drawing a general conclusion on the comparison
between statistical-based and logic-based tools, it shows our neural bug detector is not only notably
more precise than Infer but also capable of catching new bugs on many popular codebases on
GitHub.
5 RELATED WORK
In this section, we survey prior works on learning models of source code. Hindle et al . (2012)
pioneer the /f_ield of machine learning for source code. In particular, Hindleet al. /f_ind that programs
that people write are mostly simple and rather repetitive, and thus they are amenable to statistical
language models. /T_heir /f_inding is based on the evidence that the n-gram language model captures
regularities in so/f_tware as well as it does in English.
Later, many works propose to elevate the learning from the level of token sequences (Gupta et al.
2017; Hindle et al. 2012; Nguyen et al. 2013; Pu et al. 2016) to that of abstract syntax trees (Alon et al.
2019a,b; Maddison and Tarlow 2014) in an a/t_tempt to capture the structure properties exhibited in
source code. Notably, Alon et al. (2019b) present a method for function name prediction. Speci/f_ically,
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:24 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
it decomposes a program to a collection of paths in its abstract syntax tree, and learns the atomic
representation of each path simultaneously with learning how to aggregate a set of them.
Nowadays, graph neural networks have become undoubtedly the most popular deep model of
source code. Since the introduction of graph neural networks to the programming domain (Alla-
manis et al. 2018; Li et al. 2016), they have been applied to a variety of PL tasks, such as program
summarization (Fernandes et al. 2019), bug localization and /f_ixing (Dinella et al. 2020), and type
inference (Wei et al. 2020). Apart from being thoroughly studied and constantly improved in the
machine learning /f_ield, GNN’s capability of encoding semantic structure of code through a graph
representation is a primary contributor to their success. While our work also builds upon GNN, we
oﬀer a fundamentally diﬀerent perspective which no prior works have explored. /T_hat is program
abstraction helps machine learning models to capture the essence of the semantics programs denote,
and in turn facilitating the execution of downstream tasks in a precise and eﬃcient manner.
In parallel to all the aforementioned works, a separate line of works has emerged recently
that use program executions (i.e. dynamic models) (Wang 2019; Wang et al. 2018; Wang and Su
2020) rather than source code ( i.e. static models) for learning program representations. /T_heir
argument is that source code alone may not be suﬃcient for models to capture the semantic
program properties. Wang and Christodorescu (2019) show simple, natural transformations, albeit
semantically-preserving, can heavily in/f_luence the predictions of models learned from source
code. In contrast, executions that oﬀer direct, precise, and canonicalized representations of the
program behavior help models to generalize beyond syntactic features. On the /f_lip side, dynamic
models are likely to suﬀer from the low quality of training data since high-coverage executions are
hard to obtain. Wang and Su (2020) address this issue by blending both source code and program
executions. Our work is set to improve static models via program abstraction, therefore we don’t
consider runtime information as a feature dimension, which can be an interesting future direction
to explore.
6 CONCLUSION
In this paper, we present a new methodology of learning models of source code. In particular, we
argue by learning from abstractions of source code, models have an easier time to distill the key
features for program representation, thus be/t_ter serving the downstream tasks. At a technical
level, we develop a principled interval-based abstraction method that directly applies to control
/f_low graph. /T_his graph abstraction method translates to a loop-based program abstraction at
the source code level, which in essence makes models focus exclusively on looping construct for
learning feature representations of source code. /T_hrough a comprehensive evaluation, we show
our approach signi/f_icantly outperforms the state-of-the-art models in two highly popular PL tasks:
variable misuse and method name prediction. We also evaluate our approach in catching null
pointer dereference bugs in Java programs. Results again show GINN-based bug detector not only
beats the GNN-based bug detector but also yields a lower false positive ratio than Facebook Infer
when deployed to catch null pointer deference bugs in the wild. We reported 38 bugs found by
GINN to developers, among which 11 have been /f_ixed and 12 have been con/f_irmed (/f_ixing pending).
GINN is a general, powerful deep neural network that we believe is readily available to tackle a
wide range of problems in program analysis, program comprehension, and developer productivity.
REFERENCES
Miltiadis Allamanis. 2019. /T_he Adverse Eﬀects of Code Duplication in Machine Learning Models of Code. InProceedings of
the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Re/f_lections on Programming and
So/f_tware (Onward! 2019).
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:25
Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2018. Learning to represent programs with graphs.
International Conference on Learning Representations(2018).
Frances E. Allen. 1970. Control Flow Analysis. In Proceedings of a Symposium on Compiler Optimization.
Uri Alon, Omer Levy, and Eran Yahav. 2019a. code2seq: Generating sequences from structured representations of code.
International Conference on Learning Representations(2019).
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019b. Code2Vec: Learning Distributed Representations of Code.
Proc. ACM Program. Lang.POPL (2019).
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and
translate. International Conference on Learning Representations(2015).
Josh Berdine, Cristiano Calcagno, and Peter W. O’Hearn. 2006. Smallfoot: Modular Automatic Assertion Checking with
Separation Logic. In Proceedings of the 4th International Conference on Formal Methods for Components and Objects
(FMCO’05).
Cristiano Calcagno, Dino Distefano, Jeremy Dubreil, Dominik Gabi, Pieter Hooimeijer, Martino Luca, Peter O’Hearn, Irene
Papakonstantinou, Jim Purbrick, and Dulma Rodriguez. 2015. Moving Fast with So/f_tware Veri/f_ication. InNASA Formal
Methods, Klaus Havelund, Gerard Holzmann, and Rajeev Joshi (Eds.).
Kyunghyun Cho, Bart van Merri¨enboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua
Bengio. 2014. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In
Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).
P. Cousot and R. Cousot. 1977. Abstract interpretation: a uni/f_ied la/t_tice model for static analysis of programs by construction
or approximation of /f_ixpoints. InConference Record of the Fourth Annual ACM SIGPLAN-SIGACT Symposium on Principles
of Programming Languages.
Jacob Devlin, Rabih Zbib, Zhongqiang Huang, /T_homas Lamar, Richard Schwartz, and John Makhoul. 2014. Fast and
robust neural network joint models for statistical machine translation. In Proceedings of the 52nd Annual Meeting of the
Association for Computational Linguistics (Volume 1: Long Papers).
Elizabeth Dinella, Hanjun Dai, Ziyang Li, Mayur Naik, Le Song, and Ke Wang. 2020. Hoppity: Learning Graph Transforma-
tions to Detect and Fix Bugs in Programs. International Conference on Learning Representations(2020).
Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Structured Neural Summarization. International
Conference on Learning Representations(2019).
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. 2017. Neural Message Passing for
/Q_uantum Chemistry. InProceedings of the 34th International Conference on Machine Learning - Volume 70 (ICML/f_i17).
Marco Gori, Gabriele Monfardini, and Franco Scarselli. 2005. A new model for learning in graph domains. In Proceedings.
2005 IEEE International Joint Conference on Neural Networks, 2005.
Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017. DeepFix: Fixing Common C Language Errors by Deep
Learning. In /T_hirty-First AAAI Conference on Arti/f_icial Intelligence.
Vincent J. Hellendoorn, Charles Su/t_ton, Rishabh Singh, Petros Maniatis, and David Bieber. 2020. Global Relational Models
of Source Code. International Conference on Learning Representations(2020).
Abram Hindle, Earl T. Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. 2012. On the Naturalness of So/f_tware. In
Proceedings of the 34th International Conference on So/f_tware Engineering (ICSE /f_i12).
Sepp Hochreiter and Jrgen Schmidhuber. 1997. Long Short-term Memory. Neural computation(1997).
L. C. Jain and L. R. Medsker. 1999. Recurrent Neural Networks: Design and Applications(1st ed.). CRC Press, Inc., USA.
L. Jiang, G. Misherghi, Z. Su, and S. Glondu. 2007. DECKARD: Scalable and Accurate Tree-Based Detection of Code Clones.
In 29th International Conference on So/f_tware Engineering (ICSE’07).
Ren´e Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing
studies for Java programs. In Proceedings of the 2014 International Symposium on So/f_tware Testing and Analysis. ACM,
437–440.
Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2016. Gated graph sequence neural networks. International
Conference on Learning Representations(2016).
Chris Maddison and Daniel Tarlow. 2014. Structured generative models of natural source code. In International Conference
on Machine Learning.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeﬀrey Dean. 2013. Distributed Representations of Words and
Phrases and /T_heir Compositionality. InNeural Information Processing Systems (NIPS).
Tung /T_hanh Nguyen, Anh Tuan Nguyen, Hoan Anh Nguyen, and Tien N. Nguyen. 2013. A Statistical Semantic Language
Model for Source Code. In Proceedings of the 2013 9th Joint Meeting on Foundations of So/f_tware Engineering (ESEC/FSE
2013).
Renaud Pawlak, Martin Monperrus, Nicolas Petitprez, Carlos Noguera, and Lionel Seinturier. 2015. Spoon: A Library for
Implementing Analyses and Transformations of Java Source Code. So/f_tware: Practice and Experience(2015).
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:26 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
Michael Pradel and Koushik Sen. 2018. Deepbugs: a learning approach to name-based bug detection. Proceedings of the
ACM on Programming LanguagesOOPSLA (2018).
Yewen Pu, Karthik Narasimhan, Armando Solar-Lezama, and Regina Barzilay. 2016. Sk P: A Neural Program Corrector
for MOOCs. In Companion Proceedings of the 2016 ACM SIGPLAN International Conference on Systems, Programming,
Languages and Applications: So/f_tware for Humanity (SPLASH).
Veselin Raychev, Pavol Bielik, and Martin Vechev. 2016. Probabilistic Model for Code with Decision Trees. In Proceedings of
the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications
(OOPSLA 2016).
Ripon Saha, Yingjun Lyu, Wing Lam, Hiroaki Yoshida, and Mukul Prasad. 2018. Bugs. jar: a large-scale, diverse dataset of
real-world java bugs. In 2018 IEEE/ACM 15th International Conference on Mining So/f_tware Repositories (MSR). IEEE, 10–13.
Xujie Si, Hanjun Dai, Mukund Raghothaman, Mayur Naik, and Le Song. 2018. Learning Loop Invariants for Program
Veri/f_ication. InProceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS/f_i18).
Daniel Svozil, Vladimir Kvasnicka, and Jiri Pospichal. 1997. Introduction to multi-layer feed-forward neural networks.
Chemometrics and intelligent laboratory systems(1997).
David A Tomassi, Naji Dmeiri, Yichen Wang, Antara Bhowmick, Yen-Chuan Liu, Premkumar T Devanbu, Bogdan Vasilescu,
and Cindy Rubio-Gonz´alez. 2019. Bugswarm: mining and continuously growing a dataset of reproducible failures and
/f_ixes. In2019 IEEE/ACM 41st International Conference on So/f_tware Engineering (ICSE). IEEE, 339–349.
Marko Vasic, Aditya Kanade, Petros Maniatis, David Bieber, and Rishabh singh. 2019. Neural Program Repair by Jointly
Learning to Localize and Repair. International Conference on Learning Representations(2019).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,  Lukasz Kaiser, and Illia
Polosukhin. 2017. A/t_tention is all you need. InAdvances in neural information processing systems.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer Networks. In Advances in Neural Information Processing
Systems 28.
Ke Wang. 2019. Learning Scalable and Precise Representation of Program Semantics. arXiv preprint arXiv:1905.05251(2019).
Ke Wang and Mihai Christodorescu. 2019. COSET: A Benchmark for Evaluating Neural Program Embeddings.arXiv preprint
arXiv:1905.11445 (2019).
Ke Wang, Rishabh Singh, and Zhendong Su. 2018. Dynamic Neural Program Embedding for Program Repair. International
Conference on Learning Representations(2018).
Ke Wang and Zhendong Su. 2020. Blended, Precise Semantic Program Embeddings. In Proceedings of the 41st ACM SIGPLAN
International Conference on Programming Language Design and Implementation (PLDI ’20).
Song Wang, Taiyue Liu, and Lin Tan. 2016. Automatically learning semantic features for defect prediction. In2016 IEEE/ACM
38th International Conference on So/f_tware Engineering (ICSE). IEEE.
Jiayi Wei, Maruth Goyal, Greg Durre/t_t, and Isil Dillig. 2020. LambdaNet: Probabilistic Type Inference using Graph Neural
Networks. In International Conference on Learning Representations.
Mark Weiser. 1981. Program Slicing. In Proceedings of the 5th International Conference on So/f_tware Engineering (ICSE /f_i81).
Xin Ye, Razvan Bunescu, and Chang Liu. 2014. Learning to rank relevant /f_iles for bug reports using domain knowledge. In
Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of So/f_tware Engineering. ACM, 689–699.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:27
Appendix
A LOSS FUNCTION
We present the loss function of the joint model below. First, we use the cross-entropy loss to specify
the error between the predicted location and the true distribution, Loc, of the misuse variable.
H(Loc, Ppro)= −
Õ
i ∈[0, k)
Loc[i]logPpro[i][0]= −logPpro[itrue ][0]
where itrue is the actual index of the misuse variable in an input program. /T_hat is, the loss is the
negative logarithm ofPpro[itrue ][0], the probability that the model assigns to the token at indexitrue .
As Ppro[itrue ][0]tends to 1, the loss approaches zero. /T_he furtherPpro[itrue ][0]goes below 1, the
greater the loss becomes. /T_hus, minimizing this loss is equivalent to maximizing the log-likelihood
that the model assigns to the true labelsitrue . Similarly, the cross-entropy loss for repair variable is:
H(Rep, Ppro)= −
Õ
i ∈[0, k)
Rep[i]logPpro[i][1]= −logPpro[itrue ][1]
/T_he network will be trained to minimize bothH(Loc, Ppro)and H(Rep, Ppro). For inference, the
network predicts the token at iLoc (resp. iRep) to be the misuse (resp. repair) variable.
iLoc = argmax
i ∈[0, k)
Ppro[i][0]
iRep = argmax
i ∈[0, k)
Ppro[i][1]
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:28 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
B RECURRENT NEURAL NETWORK
A recurrent neural network (RNN) (Jain and Medsker 1999) is a class of arti/f_icial neural networks
that are distinguished from feedforward networks by their feedback loops. /T_his allows RNNs to
ingest their own outputs as inputs. It is o/f_ten said that RNNs have memory, enabling them to
process sequences of inputs.
Here we brie/f_ly describe the computation model of a vanilla RNN. Given an input sequence,
embedded into a sequence of vectors x = (x1, ···, xTx ), an RNN with N inputs, a single hidden layer
with M hidden units, and Q output units. We de/f_ine the RNN’s computation as follows:
ht = f (W ∗xt + V ∗ht−1) (17)
ot = so/f_tmax(Z ∗ht )
where xt ∈RN , ht ∈RM , ot ∈RQ is the RNN’s input, hidden state and output at time t, f is a
non-linear function (e.g. tanh or sigmoid), W ∈RM∗N denotes the weight matrix for connections
from input layer to hidden layer,V ∈RM∗M is the weight matrix for the recursive connections (i.e.
from hidden state to itself) and Z ∈RQ∗M is the weight matrix from hidden to the output layer.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:29
C PARTITIONING THE NEW GRAPHS INTO INTERVALS
We split our exposition into two steps: convert standard control /f_low graph to statement-based
control /f_low graph and from statement-based control /f_low graph to token-based control /f_low graphs.
Recall the de/f_inition of intervals: one node, called header, is the only entry node of a subgraph
in which all closed paths contain the header node. In both steps, the intervals on the original
control /f_low graph will be preserved because (1) there is no external node that connects to any
statement or token node (2) the closed path is fundamentally the same as before except there will be
more statement/token nodes on the path. /T_herefore, we say intervals consist of the same program
statements regardless of the conversion from the standard to token-based control /f_low graph.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:30 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
D DATASET GENERATION
We used the ETH Py150 dataset (Raychev et al . 2016), which is based on GitHub Python code,
and already partitioned into train and test splits (100K and 50K /f_iles, respectively). We further
split the 100K train /f_iles into 90K train and 10K validation examples and applied a deduplication
step on that dataset (Allamanis 2019). We extracted all top-level function de/f_initions from these
/f_iles; any function that uses multiple variables can be turned into a training example by randomly
replacing one variable usage with another. As there may be many candidates for such bugs in a
function, Hellendoorn et al. (2020) limited their extraction to up to three samples per function to
avoid biasing the dataset too strongly towards longer functions. Since an important goal of our
work is to improve the model scalability, we included more longer functions by creating 4 samples
per function. To keep our dataset the same size as Hellendoorn et al. (2020)’s (only for training and
testing as they did not report the size of their validation set), we trimmed small functions from the
other end. For every synthetically generated buggy example, an unperturbed, bug-free example of
the function is included as well to keep our dataset balanced. Our dataset contains 2M programs
for training, 245K for validation, and 755K for testing.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:31
E CONVERTING AST TO GINN’S GRAPH REPRESENTATION
First, we walk each AST to distill all the statement nodes; then, for each statement node, we
identify its immediate successor, such as the next statement within the same basic block, or the /f_irst
statement of another basic block (due to control constructs like if statement or for loop). Next,
we add an edge from each statement node to its immediate successor. Finally, we replace statement
nodes with sequences of token nodes, and move the start and end node for each control /f_low edge
from statement nodes to their /f_irst token nodes.
Note that all programs used in (Hellendoorn et al. 2020) consist of single functions only, therefore
we don’t need to consider method calls when converting AST to CFG.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:32 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
F REPRODUCING THE RESULTS REPORTED IN PRIOR WORKS
We compare the performance of our own model implementations again those reported in (Hel-
lendoorn et al. 2020). In particular, we measure the independent localization and repair accuracy,
the metrics adopted by (Hellendoorn et al. 2020), of the core models instantiated by GGNN and
RNN Sandwich. Table 6 shows the model performance is close across both neural architectures,
signaling the validity of both our model implementations and data collection.
Note that the accuracy below is what models achieved on the training set. Table 1 in the paper
shows the results of both GGNN and GGNN-powered RNN Sandwich on the test set, which is around
5%–10% lower than what’s reported in (Hellendoorn et al. 2020).5 We conjecture the accuracy drop
is likely caused by the higher diﬃculty level of our testing set. As explained in Appendix D, our
dataset contains more longer functions. Speci/f_ically, programs exceeding 250 tokens only make up
6.5% of their test set (Hellendoorn et al. 2020) whereas programs of the same size take around 14% in
our test set. In addition, our experiment mainly focuses on the improvement of GINN over GGNN
(or GNN). Since both models are implemented and executed with the exact same con/f_igurations (i.e.
server con/f_iguration, number of GPU cores, Tensor/f_low version,etc.), we believe our experimental
results for variable misuse prediction task is legitimate.
Table 6
Models Localization
Accuracy
Repair
Accuracy
GGNN (reported) 79% 74%
GGNN (self-implemented) 77% 71%
RNN Sandwich
(reported) 81% 86%
RNN Sandwich
(self-implemented) 80% 84%
5Hellendoorn et al. (2020) used classi/f_ication accuracy and location+repair accuracy only for the test set.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:33
G TRAINING TIME
In this section, we report the training time for every evaluated model under each program analysis
task. We ignore the inference time as all models make predictions instantaneously.
Variable Misuse PredictionFigure 16 presents the training time of all evaluated models in variable
misuse prediction task. We use the localization accuracy as other metrics show the same trend.
0 10 20 30 40 50 600
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Training Time (Hours)
Test localization accuracy
GGNN GINN
Sand I-Sand
Fig. 16. Models’ training time for variable misuse prediction.
Method Name PredictionFigure 17 presents the training time of all evaluated models in the
method name prediction task.
0 20 40 60 80 100 1200
0.2
0.4
0.6
0.8
Training Time (Minutes)
F1 achieved on test set
GGNN GINN
Fig. 17. Models’ training time for method name prediction.
Null Pointer DereferenceFigure 18 presents the training time of all evaluated models in null
pointer dereference detection task. Y-axis denotes precision score models achieved on the test set
(with context (1)). Recall and F1 shows a similar trend.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:34 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
0 10 20 30 40 50 600
0.1
0.2
0.3
0.4
0.5
Training Time (Minutes)
Test Precision in Top-1
GNN GINN
Fig. 18. Models’ training time for null pointer dereference detection.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:35
H ADDITION EDGES PROPOSED BY ALLAMANIS ET AL.
Below we list the edges Allamanis et al. (2018) designed on top of AST.
•NextToken connects each terminal node (syntax token) to its successor.
•LastRead connects a terminal node of a variable to all elements of the set of terminal
nodes at which the variable could have been read last.
•LastWrite: connects a terminal node of a variable to all elements of the set of syntax
tokens at which the variable was could have been last wri/t_ten to.
•ComputedFrom connects a terminal node of a variable /v.alt to all variable tokens occurring
in expr when expr is assigned to /v.alt.
•LastLexicalUse chains all uses of the same variable.
•ReturnsTo connects return tokens to the method declaration.
•FormalArgName connects arguments in method calls to the formal parameters that they
are matched to.
•GuardedBy connects every token corresponding to a variable (in the true branch of a if
statement) to the enclosing guard expression that uses the variable.
•GuardedByNegation connects every token corresponding to a variable (in the false branch
of a if statement) to the enclosing guard expression that uses the variable.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:36 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
I METRICS USED IN METHOD NAME PREDICTION
We adopted the measure used by previous works (Alon et al. 2019a,b; Fernandes et al. 2019; Wang and
Su 2020) in method name prediction, which measured F1 score (i.e. the harmonic mean of precision
and recall) over subtokens, case-insensitive. /T_he intuition is the quality of a method name prediction
largely depends on the constituent sub-words. For example, for a method called computeDiff,
a prediction of diffCompute is considered as an exact match, a prediction of compute has a full
precision but low recall, and a prediction of computeFileDiff has a full recall but low precision.
Below is extracted from https://en.wikipedia.org/wiki/ROUGE(metric ).
ROUGE, which stands for Recall-Oriented Understudy for Gisting Evaluation, is a set of metrics
for evaluating automatic summarization and machine translation in natural language processing.
/T_he metrics compare an automatically produced summary or translation against a reference or a
set of references (human-produced) summary or translation.
•ROUGE-1 refers to the overlap of unigram (each word) between the system and reference
summaries.
•ROUGE-2 refers to the overlap of bigrams between the system and reference summaries.
•ROUGE-L: Longest Common Subsequence based statistics. Longest common subsequence
problem takes into account sentence level structure similarity naturally and identi/f_ies
longest co-occurring in sequence n-grams automatically.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:37
J DATASET FOR NPE DETECTION TASK
We give the details about the dataset we assembled for the evaluation of detecting null pointer
dereference.
Table 7
Dataset Projects Description Size (KLoC) Number of Buggy Methods
Test
Lang Java lang library 50 7
Closure A JavaScript checker and optimizer 260 18
Chart Java chart library 149 17
Mockito Mocking framework for unit tests 45 14
Math Mathematics and statistics components 165 16
Accumulo Key/value store 194 6
Camel Enterprise integration framework 560 17
Flink System for data analytics in clusters 258 13
Jackrabbit-oak hierarchical content repository 337 23
Log4j2 Logging library for Java 70 29
Maven Project management and comprehension tool62 6
Wicket Web application framework 206 7
Birt Data visualizations platform 1,093 678
Number of Buggy Methods in Total for Test 793
Number of Methods in Total for Test 3,000
Validation SWT Eclipse Platform project repository 460 276
Tomcat Web server and servlet container 222 111
Number of Buggy Methods in Total for Validation 387
Number of Methods in Total for Validation 1,000
Training
JDT UI User interface for the Java IDE 508 897
Platform UI User interface and help components of Eclipse595 920
AspectJ An aspect-oriented programming extension289 151
from BugSwarm 61 projects on GitHub 5,191 156
Number of Buggy Methods in Total for Training 2,124
Number of Methods in Total for Training 9,000
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:38 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
K PRECISION, RECALL AND F1 SCORE
Below we explain precision, recall and F1 score for bug detection.
Precision = TP
TP + FP (18)
Recall = TP
TP + FN (19)
F1 Score= 2 ∗Precision ∗Recall
Precision + Recall (20)
TP, FP, and FN denote true positive, false positive, and false negative respectively. True positive
is the number of the predicted buggy lines that are truly buggy, while false positive is the number
of predicted buggy line that are not buggy. False negative records the number of buggy lines that
are predicted as non-buggy. A higher precision suggests a relatively low number of false alarms
while high recall indicates a relatively low number of missed bugs. F1 takes both precision and
recall into consideration.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:39
L MANUAL INVESTIGATION OF THE BUG REPORTS
Below is the program extracted from https://github.com/crossoverJie/JCSprout/blob/master/src/m
ain/java/com/crossoverjie/algorithm/TwoSum.java.
48 public int {} gettwoR H int {} nums L int target I{
49 int {} result ] new int {R} [
50 map < integer L integer > map ] new hashmap < >HRI [
51 for H int i ]P[i< nums N length [i KKI{
52
53 if H map N containskey H nums {i }II{
54 result ] new int {}{ map N get H nums {i}I Li} [
55 }
56 map N put H target M nums {i}LiI [
57 }
58 return result [
59 }
Fig. 19
Below is the report produced by Facebook Infer for the program in Figure 19. /T_he alarm is proven
to be false.
Fig. 20
Below is the program extracted from https://github.com/alibaba/Sentinel/blob/master/sentinel-cor
e/src/main/java/com/alibaba/csp/sentinel/node/metric/MetricTimerListener.java.
59 private void aggregate Hmap <long L list < metricnode >> maps L map <long L metricnode > metrics L clusternode node I {
60 for H entry <long L metricnode > entry : metrics N entryset HII {
61 long time ] entry N getkey HI[
62 metricnode metricnode ] entry N getvalue HI[
63 metricnode N setresource H node N getname HII[
64 metricnode N setclassification H node N getresourcetype HII[
65 if H maps N get H time I ]] null I {
66 maps N put Htime L new arraylist < metricnode >HII[
67 }
68 list < metricnode > nodes ] maps N get H time I[
69 nodes N add H entry N getvalue HII[
70 }
71 }
Fig. 21
Figure 22 shows the report produced by Facebook Infer for the program in Figure 21. /T_his alarm is
also proven to be false.
Figure 23 shows the program extracted from https://github.com/stagemonitor/stagemonitor/blob/0.
89.0/stagemonitor-tracing/src/main/java/org/stagemonitor/tracing/pro/f_iler/CallStackElement.ja
va.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:40 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
Fig. 22
75 public void recycle HI {
76 if HA useobjectpooling I {
77 return [
78 }
79 parent ] null [
80 signature ] null [
81 executiontime ] P[
82 for H callstackelement child : children I {
83 child N recycle HI[
84 }
85 children N clear HI[
86 objectpool N offer H this I[
87 }
88
89 public void removecallsfasterthan H long thresholdns I {
90 for H iterator < callstackelement > iterator ] children N iterator HI[ iterator N hasnext HI[ I {
91 callstackelement child ] iterator N next HI[
92 if H child N executiontime < thresholdns FF A child N isioquery HI I {
93 iterator N remove HI[
94 child N recycle HI[
95 } else {
96 child N removecallsfasterthan H thresholdns I[
97 }
98 }
99 }
100
101 public boolean isioquery HI {
102 OO that might be a bit ugly L but it saves reference to a boolean and thus memory
103 return signature N charat H signature N length HI M QI ]] G G [
104 }
Fig. 23
Figure 24 is the report produced by Facebook Infer for the program in Figure 23. /T_he reason Infer
think a npe is thrown by child.isIOQuery on line 92 is because inside of child.recycle(),
called at line 94, signature will be assigned as null (line 80). /T_hen in the next iteration when
child.isIOQuery is invoked charAt function will be called on a null pointer at line 103. However,
this report is provably false because the statement at line 91 guarantees child object refers to
diﬀerent memory cells in an array during each iteration, meaning recyle function will never be
called on the same object as isQuery.
Fig. 24
Figure 25 shows the program extracted from https://github.com/winder/Universal-G-Code-Sender/
blob/54a1db413980d4c90b5b704c2723f0e740de20be/ugs-core/src/com/willwinder/universalgcode
sender/i18n/Localization.java. When region is null at line 7 inside of function getString, a null
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:41
pointer exception will be thrown in function initialize at line 2. /T_his bug, which has been /f_ixed
a/f_ter our reporting, is missed by Facebook Infer but caught by GINN-based bug detector.
1 synchronized public static boolean initialize H string language I {
2 string {} lang ] language N split HB_BI[
3 return initialize H lang {P} L lang {Q}I[
4 }
5
6 public static string getstring H string id L string region I {
7 if H region ]] null || A region N equals H localization N region II {
8 initialize H region I[
9 }
10 return getstring Hid I[
11 }
Fig. 25
Figure 26 is the program extracted from https://github.com/brianfrankcooper/YCSB/blob/cd1589ce
6f5abf96e17aa8ab80c78a4348fdf29a/jdbc/src/main/java/site/ycsb/db/JdbcDBClient.java. If driver
is checked against null at line 9, it is reasonable to add a check before calling driver.contains at
line 3. Like the previous bug, this one is also missed by Facebook Infer but caught by GINN-based
bug detector, in addition, it is /f_ixed a/f_ter we report the bug.
1 string driver ] props N getproperty H driver_class I[
2
3 if H driver N contains HB sqlserver BII {
4 sqlserver ] true [
5 }
6
7 NNN
8 try {
9 if H driver A] null I {
10 class N forname H driver I[
11 }
12
13 NNN
14 }
Fig. 26
Figure 27 is the program extracted from https://github.com/ctripcorp/apollo/blob/master/apollo-c
ore/src/main/java/com/ctrip/framework/foundation/internals/Utils.java. /T_his alarm is provably
false, and the reason seems that Facebook Infer is confused by the API Strings.nullToEmpty in
com.google.common.base.Strings.
Figure 28 shows the report produced by Facebook Infer for the program in Figure 27. /T_his is a false
alarm.
Figure 29 is the program extracted from https://github.com/brianfrankcooper/YCSB/blob/master/s
3/src/main/java/site/ycsb/db/S3Client.java. /T_his alarm is provably false, and the reason seems that
Facebook Infer is confused by the API propsCL.getProperty(/quotedbl.Vars3.protocol/quotedbl.Var, /quotedbl.VarHTTPS/quotedbl.Var);(a
method in java.util.Properties) at line 210, which assigns a default value HTTPS to protocol.
Figure 30 shows the report produced by Facebook Infer for the program in Figure 29. /T_his is another
false alarm.
Figure 31 is the program extracted from https://github.com/stagemonitor/stagemonitor/b
lob/0.89.0/stagemonitor-web-servlet/src/main/java/org/stagemonitor/web/servlet/f
ilter/HttpRequestMonitorFilter.java. /T_he path on which Infer reports a NPE is infeasible.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:42 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
5 public class utils {
6 public static boolean isblank H string str I {
7 return strings N nulltoempty H str IN trim HIN isempty HI[
8 }
9
10 public static boolean isoswindows HI {
11 string osname ] system N getproperty HBosN name BI[
12 if H utils N isblank H osname II {
13 return false [
14 }
15 return osname N startswith HB windows BI[
16 }
17 }
Fig. 27
Fig. 28
208 protocol ] props N getproperty HBsSN protocol BI[
209 if H protocol ]] null I{
210 protocol ] propscl N getproperty HBsSN protocol BL B https BI[
211 }
212
213 NNN
214 clientconfig ] new clientconfiguration HI[
215 clientconfig N setmaxerrorretry H integer N parseint H maxerrorretry II[
216 ifH protocol N equals HB http BII {
217 clientconfig N setprotocol H protocol N http I[
218 } else {
219 clientconfig N setprotocol H protocol N https I[
220 }
Fig. 29
Fig. 30
isInjectContentToHtml(request) if evaluated to betrue, httpServletResponseBufferWrapper
was instantiated properly under the same condition. We have manually veri/f_ied that no code
in between the two isInjectContentToHtml(request) (line 99 and 111) aﬀects the output of
isInjectContentToHtml(request). We have a/t_tached a detailed call chain formonitorRequest
in the supplemental material. /T_he remaining part of the code is straight-forward, and can be easily
checked.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:43
95 private void domonitor H httpservletrequest request L httpservletresponse response L filterchain filterchain I {
96
97 final statusexposingbytecountingservletresponse responsewrapper [
98 httpservletresponsebufferwrapper httpservletresponsebufferwrapper ] null [
99 if H isinjectcontenttohtml H request II {
100 httpservletresponsebufferwrapper ] new httpservletresponsebufferwrapper H response I[
101 responsewrapper ] new statusexposingbytecountingservletresponse H httpservletresponsebufferwrapper I[
102 } else {
103 responsewrapper ] new statusexposingbytecountingservletresponse H response I[
104 }
105
106 try {
107 monitorrequest H filterchain L request L responsewrapper I[
108 } catch H exception eI {
109 handleexception HeI[
110 } finally {
111 if H isinjectcontenttohtml H request II {
112 injecthtml H response L request L httpservletresponsebufferwrapper I[
113 }
114 }
115 }
Fig. 31
Figure 32 shows the report produced by Facebook Infer for the program in Figure 31. /T_his is a false
alarm.
Fig. 32
List of /f_ixed and con/f_irmed bugs:
https://github.com/ctripcorp/apollo/issues/3012
https://github.com/spring-projects/spring-boot/issues/20890
https://github.com/SynBioDex/libSBOLj/issues/601
https://github.com/SynBioDex/libSBOLj/issues/603
https://github.com/konsoletyper/teavm/issues/438
https://github.com/brianfrankcooper/YCSB/issues/1371
https://github.com/ome/bioformats/issues/3464
https://github.com/nutzam/nutz/issues/1532
https://github.com/winder/Universal-G-Code-Sender/issues/1304
https://github.com/SeleniumHQ/selenium/issues/8183
https://github.com/SeleniumHQ/selenium/issues/8184
https://github.com/SeleniumHQ/selenium/issues/8185
https://github.com/kdn251/interviews/pull/159
https://github.com/spring-projects/spring-boot/issues/20901
https://github.com/crossoverJie/JCSprout/issues/195
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
1:44 Yu Wang, Fengjuan Gao, Linzhang Wang, and Ke Wang
M PUBLICATION HISTORY
Variable Misuse PredictionGGNN (Allamanis et al . 2018) is the /f_irst notable model, which
achieves 74.1% accuracy on a subset of MSR-VarMisuse. As depicted in Table 8, on the exact same
dataset using Allamanis et al. (2018)’s metric, the joint model proposed by Vasic et al. (2019) is in
fact notably worse than GGNN. /T_he joint model performed be/t_ter than GGNN on a new dataset
with a set of new metrics.
Table 8
Model MSR-VarMisuse ETH-Py150
end-to-end
Accuracy
Classi/f_ication
Accuracy
Localization
Accuracy
Localization+Repair
Accuracy
GGNN 71.4% 71.1% 64.6% 55.8%
Joint Model 62.3% 82.4% 71% 65.7%
Improvement -9.1% 11.3% 6.4% 9.9%
Given the conceptual framework laid out by the joint model, Hellendoorn et al. (2020) proposed
an engineering upgrade by simply replacing the RNN with a Transformer (Vaswani et al . 2017)
as the new instantiation of the core model. Table 9 shows how RNN Sandwich (the state-of-the-
art) compared to the engineering upgrade of the model proposed by Vasic et al . (2019). Note
that Hellendoorn et al. (2020) curated a diﬀerent dataset than Vasic et al. (2019)’s from ETH-Py150.
≤250 (resp. 1000) represents the set of programs that consist of less than 250 (resp. 1000) tokens.
Table 9
Model Classi/f_ication Accuracy Location+Repair Accuracy
≤250 ≤1000 ≤250 ≤1000
Transformer 75.9% 73.2% 67.7% 63.0%
RNN Sandwich 82.5% 81.9% 75.8% 73.8%
Improvement 6.6% 8.7% 8.1% 10.8%
Method Name Predictioncode2seq (Alon et al. 2019a) set the bar for large-scale, cross-project
method name prediction. Table 10 shows how it compares against a simple 2-layer bidirectional
LSTM on code2seq’s datasets: Java-small, Java-med, and Java-large.
Table 10
Model Java-small Java-med Java-large
Precision Recall F1 Precision Recall F1 Precision Recall F1
2-layer bi-LSTM 42.63% 29.97% 35.20% 55.15% 41.75% 47.52% 63.53% 48.77% 55.18%
code2seq 50.64% 37.40% 43.02% 61.24% 47.07% 53.23% 64.03% 55.02% 59.19%
Improvement 8.01% 7.43% 7.82% 6.09% 5.32% 5.71% 0.50% 6.25% 4.01%
Table 11 shows how Sequence GNN (Fernandes et al . 2019) compares against code2seq on
Java-small.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network 1:45
Table 11
Model Java-small
F1 ROGUE-2 ROGUE-L
code2seq 43.0% - -
Sequence GNN 51.4% 25.0 50.0
Improvement 8.4% - -
Table 12
Model Java-med Java-large
Precision Recall F1 Precision Recall F1
code2seq 32.95% 20.23% 25.07% 36.49% 22.51% 27.84%
Liger 39.88% 27.14% 32.30% 43.28% 31.43% 36.42%
Improvement 6.93% 6.91% 7.23% 6.79% 8.92% 8.58%
Wang and Su (2020) proposed a blended model, called Liger, that combines both static and
dynamic program features for predicting method names. Table 12 shows how Liger compares with
code2seq on a subset of Java-med and Java-large.
In conclusion, the improvement the GINN-based model made over the GNN-based model in
each PL task presented in this paper is comparable to that each aforementioned model made over
the prior state-of-the-art. More importantly, GINN-based models only work with the datasets on
which GNN-based models achieved state-of-the-art results and they are measured using the metrics
proposed by the GNN-based models.
Proc. ACM Program. Lang., Vol. 1, No. OOPSLA, Article 1. Publication date: January 2020.

MISIM: A Neural Code Semantics Similarity System
Using the Context-Aware Semantics Structure
Fangke Ye∗
Intel Labs and Georgia Institute of Technology
yefangke@gatech.edu
Shengtian Zhou ∗
Intel Labs
shengtian.zhou@intel.com
Anand Venkat
Intel Labs
anand.venkat@intel.com
Ryan Marcus
Intel Labs and MIT
ryanmarcus@csail.mit.edu
Nesime Tatbul
Intel Labs and MIT
tatbul@csail.mit.edu
Jesmin Jahan Tithi
Intel Labs
jesmin.jahan.tithi@intel.com
Niranjan Hasabnis
Intel Labs
niranjan.hasabnis@intel.com
Paul Petersen
Intel
paul.petersen@intel.com
Timothy Mattson
Intel Labs
timothy.g.mattson@intel.com
Tim Kraska
MIT
kraska@mit.edu
Pradeep Dubey
Intel Labs
pradeep.dubey@intel.com
Vivek Sarkar
Georgia Institute of Technology
vsarkar@gatech.edu
Justin Gottschlich
Intel Labs and University of Pennsylvania
justin.gottschlich@intel.com
Abstract
Code semantics similarity can be used for many tasks such as code recommendation,
automated software defect correction, and clone detection. Yet, the accuracy of
such systems has not yet reached a level of general purpose reliability. To help
address this, we present Machine Inferred Code Similarity (MISIM), a neural code
semantics similarity system consisting of two core components: (i) MISIM uses a
novel context-aware semantics structure, which was purpose-built to lift semantics
from code syntax; (ii) MISIM uses an extensible neural code similarity scoring
algorithm, which can be used for various neural network architectures with learned
parameters. We compare MISIM to four state-of-the-art systems, including two
additional hand-customized models, over 328K programs consisting of over 18
million lines of code. Our experiments show that MISIM has8.08% better accuracy
(using MAP@R) compared to the next best performing system.
∗Lead authors.
Preprint. Under review.
arXiv:2006.05265v6  [cs.LG]  2 Jun 2021
1 Introduction
The ﬁeld of machine programming (MP) is concerned with the automation of software develop-
ment [Gottschlich et al., 2018]. In recent years, there has been an emergence of many MP systems
due, in part, to advances in machine learning, formal methods, data availability, and computing
efﬁciency [Allamanis et al., 2018a, Alon et al., 2018, 2019b,a, Ben-Nun et al., 2018, Cosentino et al.,
2017, Li et al., 2017, Luan et al., 2019, Odena and Sutton, 2020, Finkel and Laguna, 2020, Tufano
et al., 2018, Wei and Li, 2017, Zhang et al., 2019, Zhao and Huang, 2018]. An open challenge in MP
is in the construction of accurate code similarity systems. Code similarity, which determines if two
or more code fragments are similar, can be reasoned about in many ways. Two principle ways are
through syntactic similarity and semantic analysis. While attention has historically centered around
code syntax similarity (i.e., whether two or more code fragments are syntactically similar) recent
work has revealed many advantages of code semantics similarity (i.e., whether two or more code
fragments are similar in meaning, even in the presence of syntactic differences) [Iyer et al., 2020, Lee
et al., 2021]. Precisely, we deﬁne two code fragments Ci and Cj to be semantically equivalent if for
a given set of inputs, I, both Ci and Cj, produce an identical respective set of outputs, O.
(b) Abstract Syntax Tree (AST)(a) Context-Aware Semantics Structure (CASS1-2-0-0-1) (c) Simplified Parse Tree (SPT)
compound statement boolean conditioncall expression argument listglobal attributes table parenthetical expression
nodes that are invariant across treesexpression statement condition/argument listidentifier
compound statement that encodes the cardinality of child statements
two non-keyword tokens or SPT
link to direct descendent link to indirect descendent hidden subtree…
#compound_stmt#{#}
#while_stmt#
#condition# #compound_stmt#{#}
#if_stmt#
#condition# #condition#
… …
…
#call_expr#
isdigit
#arg_list#
cin peek
…
compound_stmt
while_stmt … …
paren_expr
paren_expr paren_exprexpr_stmt expr_stmt
call_expr
compound_stmt
… while_stmt if_stmt
identifier arg_list
isdigit identifier
cin peek
{$$$}
while$$ … …
($) {$$}
while$$ if$$…
identifier
($) ($)
$$
isdigit
cin peek
($)
… … … …
Global attributes table (GAT)
0input_cardinality
1output_cardinality
Syntactically dense Ambiguity introduction
Legend
#while_stmt#
Figure 1: Three structural representations of Program A: CASS, AST, and SPT.
Code semantics similarity systems aim to determine if code fragments are solving a similar (or
equivalent) problem, even if implemented differently (e.g., various algorithms ofsort() [Cormen
et al., 2009]). They can be used in many ways. One way is to improve programmer productivity on
tasks such as code recommendation, clone detection, bug detection and mitigation, and language-
to-language translation (i.e., transpilation), to name a few [Allamanis et al., 2018b, Ahmad et al.,
2019, Bader et al., 2019, Barman et al., 2016, Bhatia et al., 2018, Dinella et al., 2020, Kamil et al.,
2016, Luan et al., 2019, Pradel and Sen, 2018]. Moreover, with improved accuracy, code semantics
similarity systems can likely be leveraged to automate many other parts of software development,
which may become necessary to keep pace with the heterogeneous growth of programming languages
and hardware systems [Ahmad et al., 2019, Batra et al., 2018, Bogdan et al., 2019, Chen et al., 2020,
Deng et al., 2020, Hannigan et al., 2019]. Yet, despite advances in the space of code semantics
similarity, fundamental questions remain open. One open question, which is the principle focus of
this paper, is regarding how the structural representations impact learning semantics similarity in
syntactically diverse code sets [Alam et al., 2019, Allamanis et al., 2018b, Becker and Gottschlich,
2017, Ben-Nun et al., 2018, Dinella et al., 2020, Iyer et al., 2020, Luan et al., 2019].
While prior work has explored some structural representations of code in the space of code similarity
and understanding, these explorations are still in their early stages. The classical abstract syntax tree
(AST) is used in the code2vec and code2seq systems [Alon et al., 2019b,a], while two novel structures
called the conteXtual ﬂow graph (XFG) and the simpliﬁed parse tree (SPT) are used in Neural Code
Comprehension (NCC) [Ben-Nun et al., 2018] and Aroma [Luan et al., 2019], respectively. While
each of these representations has beneﬁts in certain contexts, they possess one or more limitations
when considered more broadly. For example, the AST – while having a notable historical importance
2
for optimizing compilers – can be syntactically dense (see Figure 1(b)). Such syntax density can
mislead code similarity systems into memorizing syntax, rather than learning semantics (i.e., the
meaning behind the syntax). Alternatively, the XFG can capture important data dependencies, but is
obtained from an intermediate representation (IR) that requires code compilation. This restriction
can limit its application in interactive developer environments such as live code auto-completion or
recommendation. The SPT is structurally driven, which enables it to lift certain semantic meaning
from code, yet, it does not always resolve syntactic ambiguities. Instead, it can sometimes introduce
them, due to its intentional coarse-grain approach (see Figure 1(c)).
Learning from these observations, we present our code semantics similarity system called Machine
Inferred Code Similarity (MISIM). We principally focus on MISIM’s two main novelties:(i) its novel
code structural representation, the context-aware semantics structure(CASS), and (ii) its neural-based
learned code similarity scoring algorithm. This paper makes the following technical contributions:
• We present MISIM’scontext-aware semantics structure (CASS), a conﬁgurable representation of
code designed to (i) lift semantic meaning from code syntax, which (ii) supports language-speciﬁc
and language-agnostic extensibility.
• We present MISIM’s learned deep neural network (DNN) semantics similarity scoring framework
and show its efﬁcacy across three DNN topologies: (i) bag-of-features, (ii) a recurrent neural
network (RNN), and (iii) a graph neural network (GNN).
• We compare MISIM to four existing code similarity systems: (i) code2vec, (ii) code2seq, (iii)
Neural Code Comprehension, and (iv) Aroma. To deepen the experimental analysis, we also
include two customized source code token sequence models and include reported results from IBM
and MIT’s CodeNet repository, who conducted their own comparison of MISIM to Aroma. Across
approximately 18 million lines of code (not including CodeNet), our results show that MISIM has,
at its worst, 8.08% better accuracy than the next best performing system.
2 Code Representations for Code Semantics Similarity Analysis
Code representation is a core component of code semantics similarity analysis. Existing approaches
fall under two categories: syntax-based representations and semantics-based representations. We
provide a brief anecdotal analysis of these representations, discuss their strengths and weaknesses for
code semantics similarity, and motivate a new approach that offers more conﬁgurability.
A Real-World Example. We analyze two simple code examples (Program A and B) taken from the
POJ-104 dataset [Mou et al., 2016]. These programs correctly solve the same problem (#88), where
the goal is to emit all digits in a given input string. Hence, while the implementations of Program A
and B are syntactically dissimilar, they are semantically equivalent. Code for Program A and B is
shown below; Figure 1 illustrates three different ways to represent Program A.
Program A
int a;
// algorithm
while (!cin.eof()) {
while (!cin.eof() && !isdigit(cin.peek()))
cin.get(); // ignore
// print out result
if (cin >> a)
cout << a << endl;
}
Program B
char *p, *head, c;
p = (char *) malloc(sizeof(char) * 30);
head = p; scanf("%c", p);
while (*p != ’\n’) { p++; *p = getchar();}
*p = ’\0’; p = head;
for (; *p != ’\0’; p++) {
if(*p <= ’9’ && *p >= ’0’){printf("%c",*p);}
else if(*(p+1) < 58 && *(p+1) > 47){putchar(’\n’);}
}
Syntax-based Representations. Compilers have successfully used syntax-based representations
of programs, such as parse trees (a.k.a., concrete syntax trees) and abstract syntax trees (ASTs),
for several decades [Baxter et al., 1998]. More recently, the code understanding systems code2vec
and code2seq have utilized ASTs as their basic representation of code for program semantics
analysis [Alon et al., 2019b,a]. Parse trees are typically built by a language-speciﬁc parser during
the compilation process and faithfully capture every syntactic detail of the source program. ASTs,
on the other hand, abstract away certain syntactic details (e.g., parentheses), which can simplify
program analysis. However, ASTs can still be syntactically dense, which can mislead code similarity
systems into memorizing syntax rather than learning semantics. For instance, the AST in Figure 1(b)
represents the parentheses of while statements in Program A as paren_expr nodes, while missing
3
the semantic binding they have to the condition expression of the while statements. Syntax-based
representations are also conceptually ill-ﬁt for cross-language code semantics similarity in general.
Semantics-based Representations. Semantics-based representations capture the semantics of vari-
ous program constructs instead of syntax. ConteXtual ﬂow graph (XFG) used by NCC and simpliﬁed
parse tree (SPT) used by Aroma are two such representations. NCC hypothesizes that program
statements that operate in similar contexts are semantically similar, where the context of a statement
is deﬁned as the other surrounding statements with direct data- and control-ﬂow dependencies. These
dependencies are captured at the IR level by deﬁning programs’ XFG representations. inst2vec
embeddings are then trained for IR instructions a la word2vec [Mikolov et al., 2013a] and skip-gram
models [Mikolov et al., 2013b]. Overall, NCC views instructions as semantically similar, if their
embedding vectors are closer. Unfortunately, NCC’s dependency on a compiler IR restricts XFG
only to compilable code. Aroma uses SPT as a program representation to enable ML-based code
search and recommendation. Aroma builds SPT of a given query program, featurizes it using a set of
manually-selected features, and performs dot product of the query’s feature vector and that of the
candidates to ﬁnd similar code snippets. SPT is different than AST in that, it only consists of program
tokens and does not use any special language-speciﬁc rule names. Thus, SPT is language-agnostic,
which enables uniform handling of code written in different programming languages.
Limitations. While semantics-based code representations, and SPT in particular, are more suitable
for general-purpose code semantics similarity analysis, our example reveals some critical limitations
with SPT. First of all, although SPT is structurally-driven (as opposed to the syntax-driven AST [Luan
et al., 2019]), it may unintentionally carry semantic ambiguity. For instance, the SPT in Figure 1(c)
does not distinguish between the argument list of a function call (e.g., " (cin.peek())") and the
condition node of an if statement (e.g., " (cin >> a)"), and represents both of them by " ($)".
Moreover, it also captures program details that may be irrelevant for code semantics similarity. For
instance, it captures the number of program statements in main in the root node (e.g., as 3 $’s for
Program A, and 8 $’s for Program B), yet, while both of these programs have different number of
statements, they are semantically equivalent. Therefore, featurizing on this metric could mislead and
mistrain an ML system to infer such information as semantically meaningful.
A New Approach. We propose a new way to represent code, called the context-aware semantics
structure (CASS), to improve upon the limitations of the existing representations. CASS is a
semantics-based representation that builds on and extends SPT into a conﬁgurable structure, which
enables it to ﬂexibly capture a wide variety of structural representations (details in Section 3).
Figure 1(a) shows the CASS for Program A, obtained from one of its 216 different conﬁgurations.
In this example, CASS resolves the aforementioned ambiguity introduced by SPT by representing
argument list and condition node of an if and while statement differently, while simultaneously
eliding away unnecessary syntactic density produced by the AST (e.g., identifier nodes). These
modiﬁcations help in building a more accurate neural backend system (e.g., for this particular case,
CASS-based MISIM outperforms both its AST- and SPT-based counterparts by more than a 4.3%
margin).
3 The MISIM System
In Figure 2, we provide an overview of the MISIM system. A core component of MISIM is
the novel context-aware semantics structure (CASS), which aims to capture semantically salient
properties of the input code. Moreover, CASS is designed to be context-aware. That is, it can
capture information that describes the context of the code (e.g., parenthetical operator disambiguation
between a function call, mathematical operator precedence, Boolean logic ordering, etc.) that may
otherwise be ambiguous without such context-sensitivity. Once these CASSes are constructed,
they are vectorized and used as input to a neural network, which produces a feature vector for a
corresponding CASS. Once a feature vector is generated, a code similarity measurement (e.g., cosine
similarity [Baeza-Yates and Ribeiro-Neto, 1999]) calculates the similarity score.
3.1 Context-Aware Semantics Structure (CASS)
We have designed CASS with the following guiding principles: (i) it should not require compilation,
(ii) it should be a ﬂexible representation that captures code semantics, and (iii) it should be capable of
resolving code ambiguities in both its context sensitivity to the code and its environment. The ﬁrst
4
Phase 0 : source code Phase 1 : CASS featurization
Phase 3 : code similarity scoring
……CASS i1CASS i2
Learned embeddingFeatured CASS i1Featured CASS i2
Sub-network (e.g., RNN, GNN)Score
Output
Learned embedding
Sub-network (e.g., RNN, GNN)
OutputInferenceTrainingCASS representation•Manual features•Serialized CASS•CASS tree 
Snippet 1 CASS representation
DNN 1
DNN 2DNN•Neural bag-of-features•Recurrent neural net•Graph neural net
Similarity metric (e.g., cosine, dot product, etc.)
Metric learning loss (e.g., circle loss, triplet loss, etc.)
Error minimizationfunction 1…function Nfunction 1…function MGlobal Attributes TableGlobal Attributes Table
Similarity metric
Snippet 2 CASS representationPhase 2 : twin code semantic DNN trainingTraining Code Snippet i1Training Code Snippet i2……CASS j1CASS j2Featured CASS j1Featured CASS j2
Snippet 1 CASS representationfunction 1…function Xfunction 1…function YGlobal Attributes Table
Snippet 2 CASS representationTesting Code Snippet j1
 Twin code semantic DNNTwin code semantic DNNTesting Code Snippet j2j = 0..Ttest , where Ttestis the maximum number of testing tuples.i = 0..Ttrain , where Ttrainis the maximum number of training tuples. N, M, X, Y are the maximum number of functions for code snippets i1, i2, j1, j2  respectively.
Figure 2: Overview of the MISIM system.
principle (i) originates from the observation that unlike programs written in higher-level scripting
languages (e.g., Python [Van Rossum and Drake, 2009], JavaScript [Flanagan, 2006]), C/C++
programs found “in the wild” may not be well-formed (e.g., due to specialized compiler dependencies)
or exhaustively include all of their dependencies (e.g., due to assumptions about library availability)
and therefore may not compile. Moreover, for code recommendation systems that are expected to
function in a live setting, requiring compilation may severely constrain their practical application.
We address this by introducing a structure such that it does not require compilation (Section 3.1.1).
The second (ii) and third (iii) principles originate from the observation that different scenarios may
require attention to different semantics (e.g., embedded memory-bound systems may prefer to use
algorithms that do not use recursion due to a potential call stack overﬂow) and that programming
languages (PLs) evolve and new PLs continue to emerge. We attempt to address these issues with
CASS’s conﬁguration categories (Section 3.1.2).
3.1.1 CASS tree and global attributes table
Table 1: CASS Conﬁguration Options.
Language-speciﬁc
Type Option
Node Preﬁx
Label
0. No change (original SPT)
1. Add a preﬁx to each internal node label
2. Add a preﬁx to parenthesis node label
(C/C++ Speciﬁc)
Language-agnostic
Type Option
Compound
Statements
0. No change (original SPT)
1. Drop all features relevant to compound
statements
2. Replace with ‘{#}’
Global
Variables
0. No change (original SPT)
1. Drop all features relevant to global vars
2. Replace with ‘#GV AR’
3. Replace with ‘#V AR’ (label for local vars)
Global
Functions
0. No change (original SPT)
1. Drop all features relevant to global functions
2. Drop function identiﬁer and replace with
‘#EXFUNC’
Function I/O
Cardinality
0. No change
1. Include the input and output cardinality
per function in GAT
Here we provide an informal deﬁnition of CASS
(a formal deﬁnition is in Appendix A). The CASS
consists of one or more CASS trees and an optional
global attributes table (GAT). A CASS tree is a tree,
in which the root node represents the entire span
of the code snippet. During the construction of a
CASS tree, the program tokens are mapped to their
corresponding node labels using the grammar of the
high-level programming language. A CASS’s GAT
contains exactly one entry per unique function def-
inition in the code snippet. A GAT entry currently
includes only the input and output cardinality values
for each corresponding function, but can be extended
as new global attributes are needed.
3.1.2 CASS conﬁguration categories
In general, CASS conﬁgurations can be broadly clas-
siﬁed into two categories: language-speciﬁc and
language-agnostic. Exact values of the options for
5
each of the conﬁguration categories are described in
Table 1. Below we provide an intuitive description of the categories and their values.
Language-speciﬁc conﬁgurations (LSCs). Language-speciﬁc conﬁgurations are meant to capture
semantic meaning by resolving syntactic ambiguities that could be present in the concrete syntax
trees. It also introduces speciﬁcity related to the high-level programming language. For example,
the parentheses operator is overloaded in many programming languages to enforce an order of
evaluation of operands as well as to enclose a list of function arguments, amongst other things. CASS
disambiguates these by explicitly embedding the semantic contextual information in the CASS tree
nodes using the node preﬁx label (deﬁned in Appendix A).
(A) Node Preﬁx Label. The conﬁguration options for node preﬁx labels1 correspond to various levels
of semantic to syntactic information. In Table 1, option 0 corresponds to the extreme case of a
concrete syntax embedding, option 1 corresponds to eliminating irrelevant syntax, and option 2 is
principally equivalent to option 1, except it applies only to parentheticals, which we have identiﬁed –
through empirical evaluation – to often have notably divergent semantic meaning based on context.
Language-agnostic conﬁgurations (LACs). LACs can improve code similarity analysis by unbind-
ing overly-speciﬁc semantics that may be present in the original concrete syntax tree structure.
(B) Compound Statements. The compound statements conﬁguration option enables the user to control
how much non-terminal node information is incorporated into the CASS. Option 0 is equivalent to
Aroma’s SPT, option 1 omits separate features for compound statements altogether, and option 2 does
not discriminate between compound statements of different lengths and speciﬁes a special label to
denote the presence of a compound statement.
(C) Global Variables. The global variables conﬁguration speciﬁes the degree of global variable-
speciﬁc information contained in a CASS. In other words, it provides the user with the ability to
control the level of abstraction – essentially binding or unbinding global variable names as needed. If
all code similarity analysis will be performed against the same software program, retaining global
variable names may help elicit deeper semantic meaning. If not, unbinding global variable names
may improve semantic meaning.
(D) Global Functions. The global functions conﬁguration serves the dual purpose of (i) controlling
the amount of function-speciﬁc information to featurize and (ii) to explicitly disambiguate between
the usage of global functions and global variables (a feature that is absent in Aroma’s SPT design).
(E) Function I/O Cardinality. The function I/O cardinality conﬁguration aims to abstract the
semantics of certain groups of functions through input and output cardinality (i.e., embedded semantic
information that can be implicitly derived by analyzing the number of input and output parameters of
a function). I/O cardinality values of every function are recorded in GAT, if this option is enabled.
We have found that the speciﬁc context in which code similarity is performed provides an indication
of the optimal CASS conﬁguration. We discuss this in greater detail in Appendix A.1.
3.2 Neural Scoring Algorithm
MISIM’s neural scoring algorithm aims to compute the similarity score of two input programs. The
algorithm consists of two phases. The ﬁrst phase involves a neural network model that maps a
featurized CASS to a real-valued code vector. The second phase generates a similarity score between
a pair of code vectors using a similarity metric.2 We describe the details of the scoring model, its
training strategy, and other neural network model choices in this section.
3.2.1 Model
We investigated three neural network approaches for MISIM’s scoring algorithm:(i) a graph neural
network (GNN), (ii) a recurrent neural network (RNN), and (iii) a bag of manual features (BoF)
neural network. We name these models MISIM-GNN, MISIM-RNN, and MISIM-BoF respectively.
The graphical nature of CASS, as well as the recent success in applying GNNs in the program
domain Allamanis et al. [2018b], Brockschmidt et al. [2019], Dinella et al. [2020], Wei et al. [2020],
1Analytically deriving the optimal selection of node preﬁx labels across all C/C++ code may be untenable.
To accommodate this, we currently provide two levels of granularity for C/C++ node preﬁx labels in CASS.
2For this work, we have chosen cosine similarity as the similarity metric used within MISIM.
6
leads us to design the MISIM-GNN model that directly encodes the graphical structure of CASS.
MISIM-RNN is based on Hu et al. [2018], which serializes a CASS into a sequence and uses an RNN
to encode the structure.3 Unlike the two aforementioned models, MISIM-BoF takes in not the CASS
but a bag of manual features extracted from it, and uses a feed-forward network to encode them into a
vector. We compared the three models in our experiments and observed that MISIM-GNN performed
the best overall. Therefore, we describe it in detail in this section. Appendix B has details of the
MISIM-RNN and MISIM-BoF models.
MISIM-GNN. In the MISIM-GNN model, an input program’s CASS representation is transformed
into a graph. Then, each node in the graph is embedded into a trainable vector, serving as the node’s
initial state. Next, a GNN is used to update each node’s state iteratively. Finally, a global readout
function is applied to extract a vector representation of the entire graph from the ﬁnal states of the
nodes. We describe each of these steps in more detail below.
Input Graph Construction. We represent each program as a single CASS instance. Each instance
can contain one or more CASS trees, where each tree corresponds to a unique function of the program.
The CASS instance is converted into a single graph representation to serve as the input to the model.
The graph is constructed by ﬁrst transforming each CASS tree and its GAT entry into an individual
graph. These graphs are then merged into a single (disjoint) graph. For a CASS consisting of a CASS
tree T = (V,E) and a GAT entry a, we transform it into a directed graph G= (V′,E′,R), where
V′is the set of graph nodes, Ris the set of edge types, and E′= {(v,u,r ) |v,u ∈V′,r ∈R}is
the set of graph edges. The graph is constructed as follows:
V′= V ∪{a}, R = {p,c}, E ′= {(v,u,p ) |(v,u) ∈E}∪{(v,u,c ) |(u,v) ∈E}.
The two edge types, pand c, represent edges from CASS tree nodes to their parent and children
nodes, respectively.
Graph Neural Network. MISIM embeds each node v ∈V′in the input graph Ginto a vector by
assigning a trainable vector to each unique node label (with the optional preﬁx) and GAT attribute.
The node embeddings are then used as node initial states (h(0)
v ) by a relational graph convolutional
network (R-GCN [Schlichtkrull et al., 2018]) speciﬁed as the following:
h(l)
v = ReLU
(
1∑
r∈R |Nrv |
∑
r∈R
∑
u∈Nrv
W(l)
r h(l−1)
u + W(l)
0 h(l−1)
v
)
v ∈ V ′, l∈ [1, L],
where Lis the number of GNN layers, Nr
v = {u|(u,v,r ) ∈E′}is the set of neighbors of vthat
connect to vthrough an edge of type r∈R, and W(l)
r , W(l)
0 are weight matrices to be learned.
Code Vector Generation. To obtain a code vector c that represents the entire input graph, we apply
a graph-level readout function as speciﬁed below:
c = FC
([
AvgPool
({
h(L)
v
⏐⏐⏐v ∈ V ′
})
; MaxPool
({
h(L)
v
⏐⏐⏐v ∈ V ′
})])
The output vectors of average pooling and max pooling on the nodes’ ﬁnal states are concatenated
and fed into a fully-connected layer, yielding the code vector for the entire input program.
3.2.2 Training
We train the neural network model following the setting of metric learning [Schroff et al., 2015,
Hermans et al., 2017, Musgrave et al., 2020, Sun et al., 2020], which tries to map input data to a
vector space where, under a distance (or similarity) metric, similar data points are close together (or
have large similarity scores) and dissimilar data points are far apart. The metric we use is the cosine
similarity in the code vector space. As shown in the lower half of Figure 2, we use pair-wise labels to
train the model. Each pair of input programs are mapped to two code vectors by the model, from
which a similarity score is computed and optimized using a metric learning loss function.
3We did not include a Transformer-based model using the serialized CASS because in our experiments, we
had observed consistently worse performance from Transformers than that from RNNs when applying them to
source code token sequences.
7
code2veccode2seq
NCC
NCC-w/o-inst2vec
Aroma-DotAroma-CosSeq-RNN
Seq-Transformer
MISIM-GNNMISIM-RNNMISIM-BoF
0
20
40
60
80
100MAP@R (%)
(a) MAP@R on GCJ.
code2veccode2seq
NCC
NCC-w/o-inst2vec
Aroma-DotAroma-CosSeq-RNN
Seq-Transformer
MISIM-GNNMISIM-RNNMISIM-BoF
0
20
40
60
80
100AP (%)
 (b) AP on GCJ.
code2veccode2seq
NCC
NCC-w/o-inst2vec
Aroma-DotAroma-CosSeq-RNN
Seq-Transformer
MISIM-GNNMISIM-RNNMISIM-BoF
0
20
40
60
80
100MAP@R (%)
 (c) MAP@R on POJ-104.
code2veccode2seq
NCC
NCC-w/o-inst2vec
Aroma-DotAroma-CosSeq-RNN
Seq-Transformer
MISIM-GNNMISIM-RNNMISIM-BoF
0
20
40
60
80
100AP (%)
 (d) AP on POJ-104.
Figure 3: Summarized accuracy results on the test sets for code2vec, code2seq, NCC, Aroma, token
sequence models, and MISIM (avg over 3 runs; min/max values for error bars.)
AST-DotSPT-DotCASS-DotAST-CosSPT-CosCASS-Cos
0
50
100MAP@R (%)
45.1252.0755.5947.3955.0860.78
AST-DotSPT-DotCASS-DotAST-CosSPT-CosCASS-Cos
0
50
100AP (%)
35.98
45.9448.3145.31
55.3360.42
(a) AST vs SPT vs. CASS (2-1-3-1-1)
SPT-DotSPT-CosSPT-GNNSPT-RNNSPT-BoF
0
50
100MAP@R (%)
52.07 55.08
82.45 74.01 74.38
SPT-DotSPT-CosSPT-GNNSPT-RNNSPT-BoF
0
50
100AP (%)
45.94
55.33
82.00 81.64 82.95 (b) SPT vs. SPT + neural backends
Figure 4: Figures showing effects of CASS conﬁgurability and neural backend separately on POJ-104.
4 Experimental Evaluation
In this section, we analyze the performance of MISIM compared to code2vec, code2seq, NCC, and
Aroma on two datasets containing a total of more than 328,000 programs.4 We also compare it against
our own hand-tuned and best performing recurrent neural network (Seq-RNN) and Transformer
(Seq-Transformer), which take tokenized source code as input (see Appendix C.3 for details). Overall,
we ﬁnd that MISIM has greater accuracy than these systems across two metrics. We also perform an
abbreviated analysis of two MISIM variants, each trained with a different CASS conﬁguration, to
provide insight into when different conﬁgurations may be better ﬁt for different code corpora.
Datasets. Our experiments are conducted on two datasets: the Google Code Jam (GCJ) dataset [Ul-
lah et al., 2019] and the POJ-104 dataset [Mou et al., 2016]. The GCJ dataset consists of so-
lutions to programming problems in Google’s Code Jam coding competitions. We use a subset
of it that consisting of C/C++ programs that solve 297 problems. The POJ-104 dataset consists
of student-written C/C++ programs solving 104 problems. For both datasets, we label two pro-
grams as similar if they are solutions to the same problem. After a ﬁltering step, which removes
unparsable/non-compilable programs, we split each dataset by problem into three subsets for train-
ing, validation, and testing. Detailed statistics of the dataset partitioning are shown in Table 2.
Table 2: Dataset statistics.
Split GCJ POJ-104
#Problems #Programs #Problems #Programs
Training 237 223,171 64 28,137
Validation 29 36,409 16 7,193
Test 31 22,795 24 10,450
Total 297 282,375 104 45,780
Training. Unless otherwise speciﬁed, we use the same
training procedure in all experiments. The models are
built and trained using PyTorch [Paszke et al., 2019].
To train the models, we use the Circle loss [Sun et al.,
2020], a state-of-the-art metric learning loss function
that has been tested effective in various similarity learn-
ing tasks. Following the P-K sampling strategy [Her-
mans et al., 2017], we construct a batch of programs
by ﬁrst randomly sampling 16 different problems, and then randomly sampling at most 5 different
solutions for each problem. The loss function takes the similarity scores of all intra-batch pairs and
their pair-wise labels as input. Further details about the training procedure are in Appendix C.1.
Evaluation Metrics. The accuracy metrics we use for evaluation are Mean Average Precision at
R (MAP@R) [Musgrave et al., 2020] and Average Precision (AP) [Baeza-Yates and Ribeiro-Neto,
1999]. Since these metrics are already deﬁned, we do not detail them here (Details in Appendix C.4.)
Conﬁguration Identiﬁer. In the following sections, we refer to a conﬁguration of CASS by its unique
identiﬁer (ID). A conﬁguration ID is formatted as A-B-C-D-E. Each of the ﬁve letters corresponds
4Although other code similarity systems exist, we were not able to compare to them due to the differences in
target languages, problem settings, and lack of open-source availability.
8
to a conﬁguration type in the second column of Table 1, and will be replaced by an option number
speciﬁed in the third column of the table. Conﬁguration 0-0-0-0-0 corresponds to Aroma’s SPT.
4.1 End-to-End Results
Figure 3 shows the accuracy of code similarity systems compared in our experiments. 5 The blue
bars show the results of the MISIM system variants trained using CASS conﬁguration 2-1-3-1-1.
The orange bars show the results of code2vec, code2seq, NCC, Aroma, and the two token sequence
models. We observe that MISIM-GNN results in the best performance for MAP@R, yielding 1.08×
to 43.89×improvements over the other systems. In summary, MISIM system performs better than
other systems on both the metrics.
Table 3: Similarity MAP@R score from
CodeNet (credit: [Puri et al., 2021]).
C++1000 C++1400
Aroma 0.17 0.15
MISIM 0.75 0.75
Additional Results. Table 3 shows additional experimental
results of Aroma and MISIM gathered by the authors of Co-
deNet [Puri et al., 2021]. In their experiment, they evaluated
Aroma and MISIM on code semantics similarity problem
using their proposed C++1000 and C++1400 datasets. These
two datasets are similar to the POJ-104 dataset. The C++1000 dataset consists of 1000 classes and
500,000 programs, and the C++1400 dataset consists of 1400 classes and 420,000 programs.6
4.2 Specialized Experiment: Ablation Study
To understand the effectiveness of CASS’s conﬁgurability and neural backends separately, we
performed two additional experiments. In the ﬁrst experiment, we compared AST and Aroma’s
SPT representation (CASS conﬁg 0-0-0-0-0) with CASS’s 2-1-3-1-1 conﬁguration and excluded
neural backends. We replaced neural backends with dot product and cosine similarity, and obtained
MAP@R and AP scores on POJ-104 dataset. Figure 4a shows the results. To summarize, CASS’s
conﬁguration delivers 3-5% better accuracy over SPT and 10-13% better accuracy over AST. In
the second experiment, we added MISIM’s three neural backends to Aroma (Figure 4b) and found
that Aroma’s accuracy improves by 22-27%. In essence, both CASS’s conﬁgurability and the neural
backends independently help in improving MISIM’s accuracy over existing systems.
4.3 Specialized Experiment: CASS Conﬁgurations
We provide early an abbreviated investigation indicating that no CASS conﬁguration is invariably best
for all code. We ran experiments that train MISIM-GNN models with two CASS conﬁgurations (C1:
0-0-0-0-0 and C2: 2-2-3-2-1) on randomly sampled sub-training sets and compared their accuracy.
Table 4 shows the results from four selected sub-training sets, named TA, TB, TC, and TD from
POJ-104. When trained on TA or TB, the system using conﬁguration C2 performs better than that
using C1 in both the accuracy metrics. However, for TC or TD, the results are inverted.
Table 4: Test accuracy (avg, min, max) of MISIM-
GNN trained on different training subsets
Subset Conﬁg MAP@R (%) AP (%)
TA
C1 69.78 (-0.42/+0.21) 76.39 (-1.68/+1.51)
C2 71.99 (-0.26/+0.45) 79.89 (-1.20/+0.71)
TB
C1 63.45 (-1.58/+1.92) 68.58 (-2.51/+2.85)
C2 67.40 (-1.85/+1.23) 69.86 (-3.34/+1.79)
TC
C1 63.53 (-1.08/+1.53) 72.47 (-0.95/+1.24)
C2 61.23 (-2.04/+1.57) 69.83 (-1.03/+1.60)
TD
C1 61.78 (-0.46/+0.47) 66.86 (-2.31/+2.81)
C2 60.86 (-1.59/+0.90) 63.86 (-3.06/+3.43)
We compared the semantics features ofTA∩TB
to TC∩TD. We observed CASS-deﬁned seman-
tically salient features (e.g., global variables)
that C2 had been customized to extract, oc-
curred less frequently in TA ∩TB. For the
POJ-104 dataset, when global variables are used
more frequently, they are more likely to have
consistent meaning across different programs.
Abstracting them away as C2 does for TC,TD,
leads to a loss in semantic information salient
to code similarity. Conversely, when global
variables are not frequently used, there is an
increased likelihood that the semantics they ex-
tract are program-speciﬁc. Retaining their names in a CASS, may increase syntactic noise, reducing
model performance. C2 eliminates them for TA,TB, and has improved accuracy.
5A table for these results can be found in Appendix C.5.
6IBM/MIT’s CodeNet was publicly available a week prior to the NeurIPS submissions. We only include their
results on MISIM and Aroma. If accepted, we will perform a comprehensive study for camera-ready submission.
9
5 Related Work
There is a body of work on code comprehension not directly intended for code semantics similarity,
but may provide value to it. Some researchers have studied applying machine learning to learn from
the AST for completing various tasks on code [Alon et al., 2019a, Chen et al., 2018, Hu et al., 2018, Li
et al., 2018, Mou et al., 2016]. Odena and Sutton [2020] and Odena et al. [2021] represent a program
as property signatures inferred from input-output pairs, which may be used to improve program
synthesizers, amongst other things. There has also been work exploring graph representations. For
bug detection and code generation, Allamanis et al. [2018b] and Brockschmidt et al. [2019] represent
a program as a graph with a backbone AST and additional edges representing lexical ordering
and semantic relations between the nodes. Dinella et al. [2020] also use an AST-backboned graph
representation of programs to learn bug ﬁxing through graph transformation. Hellendoorn et al.
[2020] introduce a simpliﬁed graph containing only AST leaf nodes for program repair. Wei et al.
[2020] extract type dependency graphs from JavaScript programs for probabilistic type inference.
6 Conclusion
This paper presented MISIM, a code semantics similarity system. MISIM has two core novelties.
First, it uses the context-aware semantics structure (CASS) designed to lift semantic meaning from
code syntax. Second, it provides a neural-based code semantics similarity scoring algorithm for
learning semantics similarity scoring using CASS. Puri et al. [2021] and our experimental evaluation
showed that MISIM outperforms four state-of-the-art code semantics similarity systems and two
hand-optimized models. We also provided an anecdotal analysis illustrating that there may not be
one universally optimal CASS conﬁguration. An open research question for MISIM is in how to
automatically derive the proper conﬁguration of its various components for a given code corpus,
speciﬁcally the CASS and neural scoring algorithms, which we plan to explore in future work.
References
Maaz Bin Safeer Ahmad, Jonathan Ragan-Kelley, Alvin Cheung, and Shoaib Kamil. Automatically
Translating Image Processing Libraries to Halide. ACM Trans. Graph., 38(6), November 2019.
ISSN 0730-0301. doi: 10.1145/3355089.3356549.
Mejbah Alam, Justin Gottschlich, Nesime Tatbul, Javier S Turek, Tim Mattson, and Abdullah
Muzahid. A Zero-Positive Learning Approach for Diagnosing Software Performance Regressions.
In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlchBuc, E. Fox, and R. Garnett, editors,
Advances in Neural Information Processing Systems 32 , NeurIPS 2019, pages 11623–11635.
Curran Associates, Inc., 2019.
Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, and Charles Sutton. A Survey of Machine
Learning for Big Code and Naturalness. ACM Computing Surveys, 51(4), September 2018a.
Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. Learning to Represent Programs
with Graphs. In International Conference on Learning Representations, 2018b.
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. A General Path-Based Representation
for Predicting Program Properties. In Proceedings of the 39th ACM SIGPLAN Conference on
Programming Language Design and Implementation, PLDI 2018, page 404–419, New York, NY ,
USA, 2018. Association for Computing Machinery. ISBN 9781450356985. doi: 10.1145/3192366.
3192412.
Uri Alon, Omer Levy, and Eran Yahav. code2seq: Generating Sequences from Structured Represen-
tations of Code. In International Conference on Learning Representations, 2019a.
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: Learning Distributed Rep-
resentations of Code. Proc. ACM Program. Lang., 3(POPL):40:1–40:29, January 2019b. ISSN
2475-1421. doi: 10.1145/3290353.
Johannes Bader, Andrew Scott, Michael Pradel, and Satish Chandra. Getaﬁx: Learning to Fix Bugs
Automatically. Proc. ACM Program. Lang., 3(OOPSLA), October 2019. doi: 10.1145/3360585.
10
Ricardo A. Baeza-Yates and Berthier Ribeiro-Neto. Modern Information Retrieval. Addison-Wesley
Longman Publishing Co., Inc., USA, 1999. ISBN 020139829X.
Shaon Barman, Sarah Chasins, Rastislav Bodik, and Sumit Gulwani. Ringer: Web Automation
by Demonstration. SIGPLAN Not., 51(10):748–764, October 2016. ISSN 0362-1340. doi:
10.1145/3022671.2984020.
Gaurav Batra, Zach Jacobson, Siddarth Madhav, Andrea Queirolo, and Nick Santhanam. Artiﬁcial-
Intelligence Hardware: New Opportunities for Semiconductor Companies, 2018.
I. D. Baxter, A. Yahin, L. Moura, M. Sant’Anna, and L. Bier. Clone detection using abstract syntax
trees. In Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272),
pages 368–377, 1998.
Kory Becker and Justin Gottschlich. AI Programmer: Autonomously Creating Software Programs
Using Genetic Algorithms. CoRR, abs/1709.05703, 2017.
Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoeﬂer. Neural Code Comprehension: A
Learnable Representation of Code Semantics. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman,
N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31,
pages 3585–3597. Curran Associates, Inc., 2018.
Sahil Bhatia, Pushmeet Kohli, and Rishabh Singh. Neuro-Symbolic Program Corrector for Introduc-
tory Programming Assignments. In Proceedings of the 40th International Conference on Software
Engineering, ICSE ’18, page 60–70, New York, NY , USA, 2018. Association for Computing
Machinery. ISBN 9781450356381. doi: 10.1145/3180155.3180219.
Paul Bogdan, Fan Chen, Aryan Deshwal, Janardhan Rao Doppa, Biresh Kumar Joardar, Hai (Helen)
Li, Shahin Nazarian, Linghao Song, and Yao Xiao. Taming Extreme Heterogeneity via Machine
Learning Based Design of Autonomous Manycore Systems. In Proceedings of the International
Conference on Hardware/Software Codesign and System Synthesis Companion, CODES/ISSS ’19,
New York, NY , USA, 2019. Association for Computing Machinery. ISBN 9781450369237. doi:
10.1145/3349567.3357376.
Marc Brockschmidt, Miltiadis Allamanis, Alexander L. Gaunt, and Oleksandr Polozov. Generative
Code Modeling with Graphs. In International Conference on Learning Representations, 2019.
Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree Neural Networks for Program Translation.
In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors,
Advances in Neural Information Processing Systems 31, pages 2547–2557. Curran Associates, Inc.,
2018.
Yiran Chen, Yuan Xie, Linghao Song, Fan Chen, and Tianqi Tang. A Survey of Accelerator
Architectures for Deep Neural Networks. Engineering, 6(3):264 – 274, 2020. ISSN 2095-8099.
doi: https://doi.org/10.1016/j.eng.2020.01.007.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares,
Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder–
Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical
Methods in Natural Language Processing (EMNLP) , pages 1724–1734, Doha, Qatar, October
2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-1179.
Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. Introduction to
Algorithms, Third Edition. The MIT Press, 3rd edition, 2009. ISBN 0262033844.
V . Cosentino, J. L. Cánovas Izquierdo, and J. Cabot. A Systematic Mapping Study of Software
Development With GitHub. IEEE Access, 5:7173–7192, 2017. ISSN 2169-3536. doi: 10.1109/
ACCESS.2017.2682323.
S. Deng, H. Zhao, W. Fang, J. Yin, S. Dustdar, and A. Y . Zomaya. Edge Intelligence: The Conﬂuence
of Edge Computing and Artiﬁcial Intelligence. IEEE Internet of Things Journal, 7(8):7457–7469,
2020.
11
Elizabeth Dinella, Hanjun Dai, Ziyang Li, Mayur Naik, Le Song, and Ke Wang. Hoppity: Learning
Graph Transformations to Detect and Fix Bugs in Programs. In International Conference on
Learning Representations, 2020.
D. Feitelson, A. Mizrahi, N. Noy, A. Ben Shabat, O. Eliyahu, and R. Sheffer. How Developers
Choose Names. IEEE Transactions on Software Engineering, pages 1–1, 2020. ISSN 2326-3881.
doi: 10.1109/TSE.2020.2976920.
Hal Finkel and Ignacio Laguna. Program Synthesis for Scientiﬁc Computing, August 2020.
David Flanagan. JavaScript: The Deﬁnitive Guide. "O’Reilly Media, Inc.", 2006.
Luciano Floridi. Artiﬁcial Intelligence, Deepfakes and a Future of Ectypes. Philosophy & Technology,
31, 08 2018. doi: 10.1007/s13347-018-0325-3.
Edward M Gellenbeck and Curtis R Cook. An Investigation of Procedure and Variable Names as
Beacons During Program Comprehension. In Empirical studies of programmers: Fourth workshop,
pages 65–81. Ablex Publishing, Norwood, NJ, 1991.
Justin Gottschlich, Armando Solar-Lezama, Nesime Tatbul, Michael Carbin, Martin Rinard, Regina
Barzilay, Saman Amarasinghe, Joshua B. Tenenbaum, and Tim Mattson. The Three Pillars of
Machine Programming. In Proceedings of the 2nd ACM SIGPLAN International Workshop on
Machine Learning and Programming Languages, MAPL 2018, pages 69–80, New York, NY , USA,
2018. ACM. ISBN 978-1-4503-5834-7. doi: 10.1145/3211346.3211355.
Eileen Hannigan, Ondrej Burkacky, Peter Kenevan, Abhijit Mahindroo, Richard Johnson, Jonathon
Rivait, Heather Byer, Venetia Simcock, Elizabeth Brown, Roger Draper, Gwyn Herbein, Pamela
Norton, Katya Petriwsky, Charmaine Rice, John C. Sanchez, Dana Sand, Sneha Vats, Pooja
Yadav, Belinda Yu, Lucia Rahilly, Michael T. Borruso, Bill Javetski, Mark Staples, and Leff
Communications. McKinsey on Semiconductors, 2019.
Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. Global
Relational Models of Source Code. In International Conference on Learning Representations,
2020.
Alexander Hermans, Lucas Beyer, and Bastian Leibe. In Defense of the Triplet Loss for Person
Re-Identiﬁcation. CoRR, abs/1703.07737, 2017.
Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. Deep Code Comment Generation. In Proceedings
of the 26th Conference on Program Comprehension, ICPC ’18, page 200–210, New York, NY ,
USA, 2018. Association for Computing Machinery. ISBN 9781450357142. doi: 10.1145/3196321.
3196334.
Roshni Iyer, Yizhou Sun, Wei Wang, and Justin Gottschlich. Software language comprehension
using a program-derived semantics graph. In NeurIPS 2020 Workshop on Computer-Assisted
Programming, 2020.
Shoaib Kamil, Alvin Cheung, Shachar Itzhaky, and Armando Solar-Lezama. Veriﬁed Lifting of
Stencil Computations. In Proceedings of the 37th ACM SIGPLAN Conference on Programming
Language Design and Implementation, PLDI ’16, pages 711–726, New York, NY , USA, 2016.
ACM. ISBN 978-1-4503-4261-2. doi: 10.1145/2908080.2908117.
Celine Lee, Justin Gottschlich, and Dan Roth. Toward Code Generation: A Survey and Lessons from
Semantic Parsing, 2021.
Jian Li, Yue Wang, Michael R. Lyu, and Irwin King. Code Completion with Neural Attention
and Pointer Networks. In Proceedings of the 27th International Joint Conference on Artiﬁcial
Intelligence, IJCAI’18, page 4159–25. AAAI Press, 2018. ISBN 9780999241127.
Liuqing Li, He Feng, Wenjie Zhuang, N. Meng, and B. Ryder. Cclearner: A deep learning-based
clone detection approach. 2017 IEEE International Conference on Software Maintenance and
Evolution (ICSME), pages 249–260, 2017.
12
Tie-Yan Liu. Learning to Rank for Information Retrieval. Found. Trends Inf. Retr., 3(3):225–331,
March 2009. ISSN 1554-0669. doi: 10.1561/1500000016.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International
Conference on Learning Representations, 2019.
Sifei Luan, Di Yang, Celeste Barnaby, Koushik Sen, and Satish Chandra. Aroma: Code Recom-
mendation via Structural Code Search. Proc. ACM Program. Lang., 3(OOPSLA):152:1–152:28,
October 2019. ISSN 2475-1421. doi: 10.1145/3360578.
Tomas Mikolov, Kai Chen, Greg S. Corrado, and Jeffrey Dean. Efﬁcient estimation of word
representations in vector space, 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representa-
tions of words and phrases and their compositionality. In Proceedings of the 26th International
Conference on Neural Information Processing Systems - Volume 2, NIPS’13, page 3111–3119,
Red Hook, NY , USA, 2013b. Curran Associates Inc.
Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional Neural Networks over Tree
Structures for Programming Language Processing. InProceedings of the Thirtieth AAAI Conference
on Artiﬁcial Intelligence, AAAI ’16, page 1287–1293. AAAI Press, 2016.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A Metric Learning Reality Check, 2020.
Augustus Odena and Charles Sutton. Learning to Represent Programs with Property Signatures. In
International Conference on Learning Representations, 2020.
Augustus Odena, Kensen Shi, David Bieber, Rishabh Singh, Charles Sutton, and Hanjun Dai.
{BUSTLE}: Bottom-up program synthesis through learning-guided exploration. In International
Conference on Learning Representations, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor
Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward
Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner,
Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance
Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox,
and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035.
Curran Associates, Inc., 2019.
Michael Pradel and Koushik Sen. DeepBugs: A Learning Approach to Name-Based Bug Detection.
Proc. ACM Program. Lang., 2(OOPSLA), October 2018. doi: 10.1145/3276517.
Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladmir Zolotov,
Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh
Pujar, and Ulrich Finkler. Project codenet: A large-scale ai for code dataset for learning a diversity
of coding tasks. 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language
Models are Unsupervised Multitask Learners, 2019.
Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max
Welling. Modeling relational data with graph convolutional networks. In Aldo Gangemi, Roberto
Navigli, Maria-Esther Vidal, Pascal Hitzler, Raphaël Troncy, Laura Hollink, Anna Tordai, and
Mehwish Alam, editors, The Semantic Web - 15th International Conference, ESWC 2018, Herak-
lion, Crete, Greece, June 3-7, 2018, Proceedings, volume 10843 of Lecture Notes in Computer
Science, pages 593–607. Springer, 2018. doi: 10.1007/978-3-319-93417-4\_38.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: A Uniﬁed Embedding for Face
Recognition and Clustering. In The IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), Jun 2015.
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen
Wei. Circle Loss: A Uniﬁed Perspective of Pair Similarity Optimization. In The IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), 2020.
13
Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys
Poshyvanyk. Deep Learning Similarities from Different Representations of Source Code. In
Proceedings of the 15th International Conference on Mining Software Repositories , MSR ’18,
page 542–553, New York, NY , USA, 2018. Association for Computing Machinery. ISBN
9781450357166. doi: 10.1145/3196398.3196431.
F. Ullah, H. Naeem, S. Jabbar, S. Khalid, M. A. Latif, F. Al-turjman, and L. Mostarda. Cyber
Security Threats Detection in Internet of Things Using Deep Learning Approach. IEEE Access, 7:
124379–124389, 2019.
Guido Van Rossum and Fred L. Drake. Python 3 Reference Manual. CreateSpace, Scotts Valley, CA,
2009. ISBN 1441412697.
Huihui Wei and Ming Li. Supervised deep features for software functional clone detection by
exploiting lexical and syntactical information in source code. In Proceedings of the Twenty-Sixth
International Joint Conference on Artiﬁcial Intelligence, IJCAI-17, pages 3034–3040, 2017. doi:
10.24963/ijcai.2017/423.
Jiayi Wei, Maruth Goyal, Greg Durrett, and Isil Dillig. LambdaNet: Probabilistic Type Inference
using Graph Neural Networks. In International Conference on Learning Representations, 2020.
W. Wulf and Mary Shaw. Global Variable Considered Harmful.SIGPLAN Not., 8(2):28–34, February
1973. ISSN 0362-1340. doi: 10.1145/953353.953355.
Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. A Novel
Neural Source Code Representation Based on Abstract Syntax Tree. In Proceedings of the 41st
International Conference on Software Engineering, ICSE ’19, page 783–794. IEEE Press, 2019.
doi: 10.1109/ICSE.2019.00086.
Gang Zhao and Jeff Huang. DeepSim: Deep Learning Code Functional Similarity. In Proceedings of
the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium
on the Foundations of Software Engineering, ESEC/FSE 2018, page 141–151, New York, NY ,
USA, 2018. Association for Computing Machinery. ISBN 9781450355735. doi: 10.1145/3236024.
3236068.
14
A Context-Aware Semantics Structure Details
The following is the formal deﬁnition of CASS.
Deﬁnition 1 (Context-aware semantics structure (CASS)). A CASS consists of one or more CASS
trees and an optional global attributes table (GAT). A CASS tree, T, is a collection of nodes,
V = {v1,v2,...,v |V |}, and edges, E = {e1,e2,...,e |E|}, denoted as T = (V,E). Each edge is
directed from a parent node,vp to a child node, vc, or ek = (vp,vc) where ek ∈Eand vp,vc ∈V.
The root node, vr, of the tree signiﬁes the beginning of the code snippet and has no parent node, i.e.,
∄vp,(vp,vr) ∈E. A child node is either an internal node or a leaf node. An internal node has at
least one child node while a leaf node has no child nodes. A CASS tree can be empty, in which it
has no nodes. The CASS GAT contains exactly one entry per unique function deﬁnition in the code
snippet. A GAT entry includes the input and output cardinality values for the corresponding function.
Deﬁnition 2 (Node labels). Every CASS node has an associated label, lv. During the construction of
a CASS tree, the program tokens at each node,tv are mapped to its corresponding label orlv = f(tv).
This is depicted with an expression grammar for node labels and the function mapping tokens to
labels below.7
⟨bin-op⟩ ::= ‘+’ | ‘-’ | ‘*’ | ‘/’ | ...
⟨unary-op⟩ ::= ‘++’ | ‘--’ | ...
⟨leaf-node-label⟩ ::= LITERAL | IDENT | ‘#VAR’ | ‘#GVAR’ | ‘#EXFUNC’ | ‘#LIT’ | ...
⟨exp⟩ ::= ‘$’ | ‘$’⟨bin-op⟩‘$’ |⟨unary-op⟩‘$’ | ...
⟨internal-node-label⟩ ::= ‘for’ ‘(’⟨exp⟩‘;’⟨exp⟩‘;’⟨exp⟩‘)’⟨exp⟩‘;’
| ‘ int’⟨exp⟩‘;’
| ‘ return’⟨exp⟩‘;’
| ⟨exp⟩
| ...
lv = f(tv) =
{⟨leaf-node-label⟩ if vis a leaf node
⟨internal-node-label⟩ otherwise
Deﬁnition 3 (Node preﬁx label). A node preﬁx label is a string preﬁxed to a node label. A node
preﬁx label may or may not be present.
A.1 Discussion
We believe there is no silver bullet solution for code similarity for all programs and programming
languages. Based on this belief, a key intuition of CASS’s design is to provide a structure that
is semantically rich based on structure, with inspiration from Aroma’s SPT, while simultaneously
providing a range of customizable parameters to accommodate a wide variety of scenarios. CASS’s
language-agnostic and language-speciﬁc conﬁgurations and their associated options serve for explo-
ration of a series of tree variants, each differing in their granularity of detail of abstractions.
For instance, the compound statements conﬁguration provides three levels of abstraction. Option 0
is Aroma’s baseline conﬁguration and is the ﬁnest level of abstraction, as it featurizes the number
of constituents in a compound statement node. Option 2 reduces compound statements to a single
token and represents a slightly higher level of abstraction. Option 1 eliminates all features related to
compound statements and is the coarsest level of abstraction. The same trend applies to the global
variables and global functions conﬁgurations. It is our belief, based on early evidence, that the
appropriate level of abstraction in CASS is likely based on many factors such as (i) code similarity
purpose, (ii) programming language expressiveness, and (iii) application domain.
Aroma’s original SPT seems to work well for a common code base where global variables have
consistent semantics and global functions are standard API calls also with consistent semantics (e.g.,
a single code-base). However, for cases outside of such spaces, some question about applicability
arise. For example, assumptions about consistent semantics for global variables and functions may
not hold in cases of non-common code-bases or non-standardized global function names [Wulf and
7Note: the expression grammar we provide is non-exhaustive due to space limitations. The complete set of
standard C/C++ tokens or binary and unary operators is collectively denoted in shorthand as ‘...’.
15
Shaw, 1973, Gellenbeck and Cook, 1991, Feitelson et al., 2020]. Having the capability to differentiate
between these cases, and others, is a key motivation for CASS.
We do not believe that CASS’s current structure is exhaustive. With this in mind, we have designed
CASS to be extensible, enabling a seamless mechanism to add new conﬁgurations and options.
Our intention with this paper is to present initial ﬁndings in exploring CASS’s structure. Based on
our early experimental analysis, presented in Section C.7, CASS seems to be a promising research
direction for code similarity.
An Important Weakness. While CASS provides added ﬂexibility over SPT, such ﬂexibility may
be misused. With CASS, system developers are free to add or remove as much syntactic differ-
entiation detail they choose for a given language or given code body. Such overspeciﬁcation (or
underspeciﬁcation), may result in syntactic overload (or underload) which may cause reduced code
similarity accuracy over the original SPT design, as we illustrate in Section C.7.
B MISIM Models
In this section, we describe the models evaluated in our experiments other than MISIM-GNN, and
discuss the details of the experimental procedure.
B.1 MISIM-BoF
The MISIM-BoF model takes a set of manual features extracted from a CASS as its input. The features
include the ones extracted from CASS trees, using the same procedure described in Aroma [Luan
et al., 2019], as well as the entries in CASS GATs. The MISIM-BoF model is speciﬁed as below:
c = FC (AvgPool ({ex |x∈S})) ,
where S is the feature set of the input program and ex is the embedding vector of feature x. The
output code vector is computed by performing average pooling on the feature embeddings and
projecting its result into the code vector space with a fully connected layer.
B.2 MISIM-RNN
The input to the MISIM-RNN model is a serialized representation of a CASS. Each CASS tree,
representing a function in the program, is converted to a sequence using the technique proposed
in [Hu et al., 2018]. The GAT entry associated with a CASS tree is both prepended and appended to
the tree’s sequence, forming the sequence of the corresponding function. The model architecture can
be expressed as:
hf = biGRU (¯ef ) ,
c = FC ([AvgPool ({hf |f ∈F}) ; MaxPool ({hf |f ∈F})]) ,
where F is the set of functions in the input program and ¯ef is the sequence of embedding vectors
for the serialized CASS of function f. Each function’s sequence ﬁrst has its tokens embedded, and
then gets summarized to a function-level vector by a bidirectional GRU layer [Cho et al., 2014].
The code vector for the entire program is subsequently computed by performing average and max
pooling on the function-level vectors, concatenating the resulting vectors, and passing it through a
fully connected layer.
C Experimental Details
C.1 Training Procedure and Hyperparameters
We use the AdamW optimizer [Loshchilov and Hutter, 2019] with a learning rate of 10−3 for all
the models except Seq-Transformer, for which a learning rate of 10−4 is used to stabilize training.
The training runs for 100 epochs, each containing 1,000 iterations, and the model that gives the best
validation accuracy is used for testing.8 The hyperparameters used for the Circle loss are γ = 80
8We have observed that the validation accuracy stops to increase before the 100th epoch in all experiments.
16
and m= 0.4. For all of our MISIM models, we use 128-dimensional embedding vectors, hidden
states, and code vectors. We also apply dropout with a probability of 0.5 to the embedding vectors.
To handle rare or unknown tokens, a token that appears less than 5 times in the training set is replaced
with a special UNKNOWN token.
C.2 Modiﬁcations to code2vec, code2seq, NCC, and Aroma
To compare with code2vec, code2seq, NCC, and Aroma, we adapt them to our experimental setting in
the following ways. The original code2vec takes a function as an input, extracts its AST paths to form
the input to its neural network, and trains the network using the function name prediction task. In our
experiments, we feed the AST paths from all function(s) in a program into the neural network and train
it using the metric learning task described in Section 3.2.2. We make similar adaptions to code2seq
by combining AST paths from the whole program as one input sample. Additionally, we replace
the sequence decoder of code2vec with an attention-based path aggregator used in code2vec. NCC
contains a pre-training phase, named inst2vec, on a large code corpus for generating instruction
embeddings, and a subsequent phase that trains an RNN for a downstream task using the pre-trained
embeddings. We train the downstream RNN model on our metric learning task in two ways. The ﬁrst
uses the pre-trained embeddings (labeled as NCC in our results). The second trains the embeddings
from scratch on our task in an end-to-end fashion (labeled as NCC-w/o-inst2vec). For both code2vec
and NCC, we use the same model architectures and embedding/hidden sizes suggested in their papers
and open-sourced implementations. The dimension of their output vectors (i.e., code vectors) is set
to the same as our MISIM models. Aroma extracts manual features from the code and computes
the similarity score of two programs by taking the dot product of their binary feature vectors. We
experiment with both its original scoring mechanism (labeled as Aroma-Dot) and a variant that uses
the cosine similarity (labeled as Aroma-Cos).
C.3 Token Sequence Models
Here we provide details of the two token sequence model (Seq-RNN and Seq-Transformer) used in
our experiments.
The input to both models is tokenized source code. Identiﬁers and non-identiﬁer tokens are embedded
differently. For identiﬁers, we ﬁrst split their text values into subtokens according to camel case
and snake case patterns. Then we embed each subtoken into a trainable vector and generate an
identiﬁer’s embedding by taking the sum of its subtokens’ embedding vectors. For non-identiﬁer
tokens, we assign a trainable embedding vector to each unique token. All embedding vectors are
128-dimensional.
Seq-RNN employs a stacked two-layer bidirectional GRU that operates on token embedding se-
quences. Each direction in a layer has a 128-dimensional hidden state, and a dropout of rate 0.1
is applied between the two layers. The last hidden states in the second layer in both directions are
concatenated and projected into a 128-dimensional code vector using a linear layer. This code vector
is then used to compute cosine similarities between programs.
Seq-Transformer employs a six-layer Transformer encoder with a learned positional embedding.
The dimensionality of the Transformer is 128, and the feed-forward layers’ hidden states are 512
dimensional. A dropout of rate 0.1 is applied inside each Transformer encoder layer. The length of
the input token sequence is capped to 512. A special token is prepended to each sequence, and the
output at the location of this token after the last encoder layer is passed into a linear layer to generate
a 128-dimensional code vector for computing cosine similarities.
C.4 Evaluation Metrics
MAP@R measures how accurately a model can retrieve similar (or relevant) items from a database
given a query. MAP@R rewards a ranking system (e.g., a search engine, a code recommendation
engine, etc.) for correctly ranking relevant items with an order where more relevant items are ranked
higher than less relevant items. It is deﬁned as the mean of average precision scores, each of which is
evaluated for retrieving R most similar samples given a query. In our case, the set of queries is the
set of all test programs. For a program, R is the number of other programs in the same class (i.e., a
POJ-104 problem). MAP@R is applied to both validation and testing. We use AP to measure the
17
performance in a binary classiﬁcation setting, in which the models are viewed as binary classiﬁers
that determine whether a pair of programs are similar by comparing their similarity score with a
threshold. AP is only used for testing. They are computed from the similarity scores of all program
pairs in the test set, as well as their pair-wise labels. For the systems that require training (i.e., systems
with ML learned similarity scoring), we train and evaluate them three times with different random
seeds.
C.5 MISIM Accuracy Results (in tabular form)
Table 5 shows the results of MISIM in comparison to other systems. Same results are presented in
the graphical form in Figure 3.
Table 5: Code similarity system accuracy. Results are shown as the average and min/max values,
relative to the average, over 3 runs. We had to make a few modiﬁcations to adapt code2vec, code2seq,
NCC and Aroma to our experimental settings. Please refer to Appendix C.2 for details.
Method GCJ POJ-104
MAP@R (%) AP (%) MAP@R (%) AP (%)
code2vec 7.76 (-0.79/+0.88) 17.95 (-1.24/+1.76) 1.90 (-0.43/+0.38) 5.30 (-0.80/+0.60)
code2seq 11.67 (-1.98/+1.73) 23.09 (-3.24/+2.49) 3.12 (-0.45/+0.67) 6.43 (-0.37/+0.48)
NCC 17.26 (-1.11/+0.57) 31.56 (-1.11/+1.46) 39.95 (-2.29/+1.64) 50.42 (-2.98/+1.61)
NCC-w/o-inst2vec 34.88 (-5.72/+7.63) 56.12 (-7.63/+9.96) 54.19 (-3.18/+3.52) 62.75 (-5.49/+4.42)
Aroma-Dot 29.08 42.47 52.07 45.94
Aroma-Cos 29.67 36.21 55.08 55.33
Seq-RNN 69.27 (-0.52/+0.54) 82.66 (-0.23/+0.27) 72.28 (-0.76/+1.10) 79.19 (-1.11/+1.27)
Seq-Transformer 47.81 (-2.28/+2.90) 71.66 (-2.30/+4.22) 48.81 (-1.10/+1.66) 54.65 (-3.41/+3.55)
MISIM-GNN 74.87 (-0.10/+0.15) 91.32 (-0.18/+0.20) 83.39 (-0.30/+0.59) 84.69 (-1.82/+1.24)
MISIM-RNN 72.50 (-3.62/+2.09) 86.65 (-1.76/+2.32) 75.61 (-2.97/+2.54) 82.37 (-2.20/+1.36)
MISIM-BoF 71.25 (-0.64/+0.42) 89.28 (-0.41/+0.45) 74.85 (-0.27/+0.31) 82.97 (-0.36/+0.41
C.6 CASS vs AST
Some recent research on code representation uses the AST-based representation [Dinella et al., 2020]
or AST paths [Alon et al., 2019b,a]. In this subsection, we explore how AST and CASS perform on
the task of code semantic representation described here.
We compared the code similarity performance of ASTs and CASSes on the test set of POJ-104,
as shown in table 2, by transforming both kinds of representations into feature vectors. using the
same method described in [Luan et al., 2019] and compute the similarity scores using dot or cosine
similarity. For each program in the dataset, we extracted its CASS under three different conﬁgurations:
0-0-0-0-0 9, the base conﬁguration, and 2-1-3-1-1/1-2-1-0-0, the best/worst performing conﬁguration
according to our preliminary evaluation of CASS (see Appendix C.7 for details). We also extracted
the ASTs of function bodies in a program. Each syntax node in the AST is labeled by its node type,
and an identiﬁer (or literal) node also gets a single child labeled by the corresponding identiﬁer name
(or literal text).
Table 6: Test Accuracy for AST and CASS conﬁgurations on POJ-104.
Method MAP@R (%) AP (%)
AST-Dot 45.12 35.98
AST-Cos 47.39 45.31
SPT-Dot 52.07 45.94
SPT-Cos 55.08 55.33
CASS (2-1-3-1-1)-Dot 55.59 48.31
CASS (2-1-3-1-1)-Cos 60.78 60.42
CASS (1-2-1-0-0)-Dot 52.74 40.73
CASS (1-2-1-0-0)-Cos 57.99 54.75
9Conﬁguration 0-0-0-0-0 is the duplicate of SPT. As shown in Table 6, conﬁguration 2-1-3-1-1 shows better
accuracy than conﬁguration 0-0-0-0-0.
18
As shown in Table 6, CASS conﬁgurations show an improvement in accuracy over the AST up to
1.33×in both evaluation metrics described in Appendix C.4. To better understand the performance
difference, we investigated a few solutions for the same problems from the POJ-104 dataset. One of
the interesting observations we found is that for the same problem, a solution may have a different
naming convention for local variables than that of another solution (e.g., English vs Mandarin
description of variables), but the resulting different variable names may carry the same semantic
meaning. AST uses variable names in its structure, but CASS has the option to not use variable
names. Thus the erasure of local variable names in CASS might help in discovering the semantic
similarity between code with different variable names. This might explain some of the performance
differences between AST and CASS in this experiment.
C.7 Experimental Results of Various CASS conﬁgurations
In this section, we discuss our experimental setup and analyze the performance of CASS compared
to Aroma’s simpliﬁed parse tree (SPT). In Section C.7.1, we explain the dataset grouping and
enumeration for our experiments. We also discuss the metrics used to quantitatively rank the
different CASS conﬁgurations and those chosen for the evaluation of code similarity. Section C.7.2
demonstrates that, a code similarity system built using CASS (i) has a greater frequency of improved
accuracy for the total number of problems and (ii) is, on average, more accurate than SPT. For
completeness, we also include cases where CASS conﬁgurations perform poorly.
C.7.1 Experimental Setup
In this section, we describe our experimental setup. At the highest level, we compare the performance
of various conﬁgurations of CASS to Aroma’s SPT. The list of possible CASS conﬁgurations is
shown in Table 1.
Dataset. The experiments use the same POJ-104 dataset introduced in Section 4.
Problem Group Selection. Given that POJ-104 consists of 104 unique problems and nearly 50,000
programs, depending on how we analyze the data, we might face intractability problems in both
computational and combinatorial complexity. With this in mind, our initial approach is to construct
1000 sets of ﬁve unique, pseudo-randomly selected problems for code similarity analysis. Using
this approach, we evaluate every conﬁguration of CASS and Aroma’s original SPT on each pair of
solutions for each problem set. We then aggregate the results across all the groups to estimate their
overall performance. While this approach is not exhaustive of possible combinations (in set size or set
combinations), we aim for it to be a reasonable starting point. As our research with CASS matures,
we plan to explore a broader variety of set sizes and a more exhaustive number of combinations.
Code Similarity Performance Evaluation. For each problem group, we exhaustively calculate
code similarity scores for all unique solution pairs, including pairs constructed from the same program
solution (i.e., program Acompared to program A). We use Gto refer to the set of groups and gto
indicate a particular group in G. We denote |G|as the number of groups in G(i.e. cardinality) and
|g| as the number of solutions in group g. For g= Gi, where i= {1,2,..., 1000}, the total unique
program pairs (denoted by gP ) in Gi is |gP|= 1
2 |g|(|g|+ 1).
To compute the similarity score of a solution pair, we use Aroma’s approach. This includes calculating
the dot product of two feature vectors (i.e., a program pair), each of which is generated from a CASS
or SPT structure. The larger the magnitude of the dot product, the greater the similarity.
We evaluate the quality of the recommendation based on average precision. Precision is the ratio of
true positives to the sum of true positives and false positives. Here, true positives denote solution
pairs correctly classiﬁed as similar and false positives refer to solution pairs incorrectly classiﬁed as
similar. Recall is the ratio of true positives to the sum of true positives and false negatives, where
false negatives are solution pairs incorrectly classiﬁed as different. As we monotonically increase the
threshold from the minimum value to the maximum value, precision generally increases while recall
generally decreases. The average precision (AP) summarizes the performance of a binary classiﬁer
under different thresholds for categorizing whether the solutions are from the same equivalence class
(i.e., the same POJ-104 problem) [Liu, 2009]. AP is calculated using the following formula over all
thresholds.
19
1. All unique values from the M similarity scores, corresponding to the solution pairs, are
gathered and sorted in descending order. Let N be the number of unique scores and
s1,s2,...,s N be the sorted list of such scores.
2. For iin {1,2,...,N }, the precision pi and recall ri for the classiﬁer with the threshold
being si is computed.
3. Let r0 = 0. The average precision is computed as:
AP =
N∑
i=1
(ri −ri−1)pi
C.7.2 Results
2-0-0-0-1 0-0-0-0-1 2-2-3-0-0 1-0-1-0-0 1-2-1-0-0
Configuration
1000
500
0
500
1000Number of Problem Groups
859 828
146
5 1
141 172
854
995 999
Greater AP than SPT 
Less AP than SPT 
(a) Breakdown of the Number of
Groups with AP Greater or Less
than SPT.
SPT 0-1-3-0-1 2-1-3-0-1 1-0-2-1-0 1-1-1-0-0 1-2-1-0-0
Configuration
0.0
0.2
0.4
0.6
0.8
1.0Average Precision
0.681
0.893 0.892
0.702
0.650 0.644
(b) Average Precision for the
Group Containing the Best Case.
SPT 2-1-3-1-1 2-1-2-1-1 1-2-2-2-0 1-2-1-0-1 1-2-1-0-0
Configuration
0.5
0.6
0.7
0.8Mean of Average Precision
0.711 0.720 0.720
0.695
0.674 0.672
(c) Mean of Average Precision
Over All Program Groups.
Figure 5: Comparison of CASS and SPT. The blue bars in (a) and (b), and all the bars in (c), from
left to right, correspond to the best two, the median, and the worst two CASS conﬁgurations, ranked
by the metric displayed in each subﬁgure.
Figure 5a depicts the number of problem groups where a particular CASS variant performed better
(blue) or worse (orange) than SPT. For example, the CASS conﬁguration 2-0-0-0-1 outperformed SPT
in 859 of 1000 problem groups, and underperformed in 141 problem groups. This equates to a 71.8%
accuracy improvement of CASS over SPT. Figure 5a shows the two best (2-0-0-0-1 and 0-0-0-0-1),
the median (2-2-3-0-0), and the two worst (1-0-1-0-0 and 1-2-1-0-0) conﬁgurations with respect
to SPT. Although we have seen certain conﬁgurations that perform better than SPT, there are also
conﬁgurations that perform worse. We observed that the conﬁgurations with better performance have
function I/O cardinality option as 1. We also observed that the conﬁgurations with worse performance
have function I/O cardinality option as 0. These observations indicate that function I/O cardinality
seems to improve code similarity accuracy, at least, for the data we are considering. We speculate
that these conﬁguration results may vary based on programming language, problem domain, and
other constraints.
Figure 5b shows the group containing the problems for which CASS achieved the best performance
relative to SPT, among all 1000 problem groups. In other words, Figure 5b shows the performance
of SPT and CASS for the single problem group with the greatest difference between a CASS
conﬁguration and SPT. In this single group, CASS achieves the maximum improvement of more
than 30% over SPT for this problem group on two of its conﬁgurations. We note that, since we
tested 216 CASS conﬁgurations across 1000 different problem groups, there is a reasonable chance
of observing such a large difference even if CASS performed identically to SPT in expectation. We do
not intend for this result to demonstrate statistical signiﬁcance, but simply to illustrate the outcome of
our experiments.
Figure 5c compares the mean of AP over all 1000 problem groups. In it, the blue bars, moving left to
right, depict the CASS conﬁgurations that are (i) the two best, (ii) the median, and (iii) the two worst
in terms of average precision. Aroma’s baseline SPT conﬁguration is highlighted in orange. The best
two CASS conﬁgurations show an average improvement of more than 1% over SPT, while the others
degraded performance relative to the baseline SPT conﬁguration.
These results illustrate that certain CASS conﬁgurations can outperform the SPT on average by a
small margin, and can outperform the SPT on speciﬁc problem groups by a large margin. However,
we also note that choosing a good CASS conﬁguration for a domain is essential. We leave automating
this conﬁguration selection to future work.
20
C.7.3 Analysis of Conﬁgurations
0-*-*-*-* 1-*-*-*-* 2-*-*-*-*
Configuration
0.68
0.70
0.72Mean of Average Precision
(a) Node Preﬁx Labels.
*-0-*-*-* *-1-*-*-* *-2-*-*-*
Configuration
0.68
0.70
0.72Mean of Average Precision
 (b) Compound Statements.
*-*-0-*-* *-*-1-*-* *-*-2-*-* *-*-3-*-*
Configuration
0.68
0.70
0.72Mean of Average Precision (c) Global Variables.
*-*-*-0-* *-*-*-1-* *-*-*-2-*
Configuration
0.68
0.70
0.72Mean of Average Precision
(d) Global Functions.
*-*-*-*-0 *-*-*-*-1
Configuration
0.68
0.70
0.72Mean of Average Precision (e) function I/O cardinality.
Figure 6: The Distributions of Performance for Conﬁgurations with a Fixed Option Type.
Figures 6a-6e serve to illustrate the performance variation for individual conﬁgurations. Figure 6a
shows the effect of varying the options for the node preﬁx label conﬁguration. Applying the node
preﬁx label for the parentheses operator (option 2) results in the best overall performance while
annotating every internal node (option 1) results in a concrete syntax tree and the worst overall
performance. This underscores the trade-offs in incorporating syntax-binding transformations in
CASS. In Figure 6b we observe that removing all features relevant to compound statements (option
1) leads to the best overall performance when compared with other options. This indicates that
adding separate features for compound statements obscures the code’s intended semantics when the
constituent statements are also individually featurized.
Figure 6c shows that removing all features relevant to global variables (option 1) degrades perfor-
mance. We also observe that eliminating the global variable identiﬁers and assigning a label to signal
their presence (option 2) performs best overall, possibly because global variables appearing in similar
contexts may not use the same variable identiﬁers. Further, option 2 performs better than the case
where global variables are indistinguishable from local variables (option 3). Figure 6d indicates
that removing features relevant to identiﬁers of global functions, but ﬂagging their presence with a
special label as done in option 2, generally gives the best performance. This result is consistent with
the intuitions for eliminating features of function identiﬁers in CASS as discussed in Section A.1.
Figure 6e shows that capturing the input and output cardinality improves the average performance.
This aligns with our assumption that function I/O cardinality may abstract the semantics of certain
groups of functions.
A Subtle Observation. A more nuanced and subtle observation is that our results seem to indicate
that for each CASS conﬁguration the optimal granularity of abstraction detail is different. For
compound statements, the best option seems to correspond to the coarsest level of abstraction detail,
while for node preﬁx label, global variables, and global functions the best option seems to corresponds
to one of the intermediate levels of abstraction detail. Additionally, for function I/O cardinality, the
best option has a ﬁner level of detail. For our future work, we aim to perform a deeper analysis on
this and hopefully learn such conﬁgurations, to reduce (or eliminate) the overhead necessary of trying
to manually discover such conﬁgurations.
D Broader Impact
To discuss the broader impact of our project, we will categorize impacts by their degree of inﬂuence.
For example, by the phrase “ﬁrst-degree negative impact” we will refer to a scenario where a given
21
research idea can be directly used for harm (e.g., DeepFake [Floridi, 2018], DeepNude 10, so on).
Similarly, by "second-degree negative impact" we will refer to a scenario where a research idea may
have a direct negative or positive impact based on how it is used (e.g., facial recognition for security
vs. oppressing minorities, GPT [Radford et al., 2019] to create an empathetic chatbot vs. malicious
fake news, etc). We call a research idea to have a "third-degree negative impact" if the idea by itself
represents an abstract concept (e.g., a similarly metric) and cannot harm by its own, but can be used
to build a second application which can then have a negative impact based on its use.
We envision the following positive broader impacts of the research idea presented in this paper. As
brieﬂy mentioned in the introduction, an end-to-end code similarity system can be incorporated in
programming tools (e.g., Visual Studio, Eclipse, etc.) to improve the productivity of a programmer
by offering him/her a similar but "known to be more efﬁcient" code snippet. It can be used in coding
education by displaying better (e.g., concise, faster, space-efﬁcient, etc.) code for a given code snippet,
in assisting program debugging by identifying potential missing parts, for plagiarism detection, for
automated bug-detection and ﬁxing, in automatic code transformations (e.g., replacing a Python
function with an equivalent C function) and so on. If used wisely with proper control and governance,
we believe it can create many positive impacts.
We can envision the following third-degree negative impacts. If a tool that uses code similarity
becomes mature enough to automatically generate correct compilable codes, it can be potentially
used to automatically replace code from one language to another or to replace a slow code with a fast
one. A malicious person can leverage the code similarity tool to crawl the web and steal codes on the
web, ﬁnd common patterns and security ﬂaws in the code available on the web, and then ﬁnd ways
to hack at a massive scale. Codes generated from the same code generators are likely to be more
vulnerable to such attacks. If systems allow automatic code patching/ﬁxing based on code-similarity
without proper testing, it might create security ﬂaws if hacked. If programmers get used to getting
help from a programming tool, that might negatively reduce the learning ability of programmers
unless the tool also offers explainability. Explainability would be required to understand what the
tool is learning about the code similarity and to educate the programmers about it.
To summarize, code similarity is an abstract concept that is likely to have numerous positive applica-
tions. However, if used in other tools, it might also play a role in creating a third-order negative impact.
It may be used to develop tools and applications which, if mature enough, may cause unacceptable
or dangerous situations. To mitigate the negative impacts, we would need to ensure proper policy
and security measures are in place to prevent negative usage. In particular, such secure systems may
require a human-in-the-loop so that any such tool is used to enhance the capability and productivity
of programmers.
10We have intentionally not included a citation to this work. We do not want to be seen, in any way, as
endorsing or promoting it. We believe such an act would be ethically irresponsible.
22

arXiv:submit/0500581  [cs.LG]  24 Jun 2012
Practical Recommendations for Gradient-Based Training ofDeep
Architectures
Yoshua Bengio
June 24, 2012
Abstract
Learning algorithms related to artiﬁcial neural net-
works and in particular for Deep Learning may seem
to involve many bells and whistles, called hyper-
parameters. This chapter is meant as a practical
guide with recommendations for some of the most
commonly used hyper-parameters, in particular in
the context of learning algorithms based on back-
propagated gradient and gradient-based optimiza-
tion. It also discusses how to deal with the fact that
more interesting results can be obtained when allow-
ing one to adjust many hyper-parameters. Overall, it
describes elements of the practice used to successfully
and eﬃciently train and debug large-scale and often
deep multi-layer neural networks. It closes with open
questions about the training diﬃculties observed with
deeper architectures.
1 Introduction
Following a decade of lower activity, research in arti-
ﬁcial neural networks was revived after a 2006 break-
through (Hinton et al. , 2006; Bengio et al. , 2007;
Ranzato et al. , 2007) in the area of Deep Learning .
See (Bengio, 2009) for a review. Many of the practical
recommendations that justiﬁed the previous edition
of this book are still valid, and new elements were
added, while some survived longer by virtue of the
practical advantages they provided. The panorama
presented in this chapter regards some of these sur-
viving or novel elements of practice, focusing on
learning algorithms aiming at training deep neural
networks, but leaving most of the material speciﬁc
to the Boltzmann machine family to another chap-
ter (Hinton, 2013).
Although such recommendations come out of a liv-
ing practice that emerged from years of experimenta-
tion and to some extent mathematical justiﬁcation,
they should be challenged. They constitute a good
starting point for the experimenter and user of learn-
ing algorithms but very often have not been formally
validated, leaving open many questions that can be
answered either by theoretical analysis or by solid
comparative experimental work (ideally by both). A
good indication of the need for such validation is that
diﬀerent researchers and research groups do not al-
ways agree on the practice of training neural net-
works.
Several of the recommendations presented here can
be found implemented in the Deep Learning Tutori-
als 1 and in the related Pylearn2 library 2, all based
on the Theano library (discussed below) written in
the Python programming language.
The 2006 Deep Learning breakthrough centered
on the use of unsupervised representation learning
to help learning internal representations 3 by pro-
viding a local training signal at each level of a hi-
erarchy of features 4. Unsupervised representation
1 http://deeplearning.net/tutorial/
2 http://deeplearning.net/software/pylearn2
3 A neural network computes a sequence of data transfor-
mations, each step encoding the raw input into an intermedia te
or internal representation, in principle to make the predic tion
or modeling task of interest easier.
4 In standard multi-layer neural networks trained using
back-propagated gradients, the only signal that drives par am-
eter updates is provided at the output of the network (and
1
learning algorithms can be applied several times to
learn diﬀerent layers of a deep model. Several un-
supervised representation learning algorithms have
been proposed since then. Those covered in this
chapter (such as auto-encoder variants) retain many
of the properties of artiﬁcial multi-layer neural net-
works, relying on the back-propagation algorithm to
estimate stochastic gradients. Deep Learning algo-
rithms such as those based on the Boltzmann ma-
chine and those based on auto-encoder or sparse cod-
ing variants often include a supervised ﬁne-tuning
stage. This supervised ﬁne-tuning as well as the
gradient descent performed with auto-encoder vari-
ants also involves the back-propagation algorithm,
just as like when training deterministic feedforward
or recurrent artiﬁcial neural networks. Hence this
chapter also includes recommendations for training
ordinary supervised deterministic neural networks or
more generally, most machine learning algorithms re-
lying on iterative gradient-based optimization of a
parametrized learner with respect to an explicit train-
ing criterion.
This chapter assumes that the reader already un-
derstands the standard algorithms for training super-
vised multi-layer neural networks, with the loss gra-
dient computed thanks to the back-propagation algo-
rithm (Rumelhart et al. , 1986). It starts by explain-
ing basic concepts behind Deep Learning and the
greedy layer-wise pretraining strategy (Section 1.1),
and modern unsupervised pre-training algorithms
(denoising and contractive auto-encoders) that are
closely related in the way they are trained to stan-
dard multi-layer neural networks (Section 1.2). It
then reviews in Section 2 basic concepts in itera-
tive gradient-based optimization and in particular
the stochastic gradient method, gradient computa-
tion with a ﬂow graph, automatic diﬀerentation, and
online learning. The main section of this chapter is
Section 3, which explains hyper-parameters in gen-
eral, their optimization, and speciﬁcally covers the
main hyper-parameters of neural networks. Section 4
brieﬂy describes simple ideas and methods to debug
then propagated backwards). Some unsupervised learning al -
gorithms provide a local source of guidance for the paramete r
update in each layer, based only on the inputs and outputs of
that layer.
and visualize neural networks, while Section 5 covers
parallelism, sparse high-dimensional inputs, symbolic
inputs and embeddings, and multi-relational learn-
ing. The chapter closes (Section 6) with open ques-
tions on the diﬃculty of training deep architectures
and improving the optimization methods for neural
networks.
1.1 Deep Learning and Greedy Layer-
Wise Pretraining
The notion of reuse, which explains the power of
distributed representations (Bengio, 2009), is also at
the heart of the theoretical advantages behind Deep
Learning. The depth of a circuit is the length of the
longest path from an input node of the circuit to an
output node of the circuit. Formally, one can change
the depth of a given circuit by changing the deﬁni-
tion of what each node can compute, but only by a
constant factor (Bengio, 2009). The typical compu-
tations we allow in each node include: weighted sum,
product, artiﬁcial neuron model (such as a mono-
tone non-linearity on top of an aﬃne transforma-
tion), computation of a kernel, or logic gates. Theo-
retical results (H ˚ astad, 1986; H ˚ astad and Goldmann,
1991; Bengio et al. , 2006b; Bengio and LeCun, 2007;
Bengio and Delalleau, 2011) clearly identify families
of functions where a deep representation can be ex-
ponentially more eﬃcient than one that is insuﬃ-
ciently deep. If the same family of functions can
be represented with fewer parameters (or more pre-
cisely with a smaller VC-dimension 5), learning the-
ory would suggest that it can be learned with fewer
examples, yielding improvements in both computa-
tional eﬃciency and statistical eﬃciency.
Another important motivation for feature learning
and Deep Learning is that they can be done with un-
labeled examples , so long as the factors (unobserved
random variables explaining the data) relevant to the
questions we will ask later (e.g. classes to be pre-
dicted) are somehow salient in the input distribution
itself. This is true under the manifold hypothesis ,
which states that natural classes and other high-level
5 Note that in our experiments, deep architectures tend to
generalize very well even when they have quite large numbers
of parameters.
2
concepts in which humans are interested are asso-
ciated with low-dimensional regions in input space
(manifolds) near which the distribution concentrates,
and that diﬀerent class manifolds are well-separated
by regions of very low density. It means that a small
semantic change around a particular example can
be captured by changing only a few numbers in a
high-level abstract representation space. As a conse-
quence, feature learning and Deep Learning are in-
timately related to principles of unsupervised learn-
ing, and they can be exploited in the semi-supervised
setting (where only a few examples are labeled), as
well as in the transfer learning and multi-task settings
(where we aim to generalize to new classes or tasks).
The underlying hypothesis is that many of the under-
lying factors are shared across classes or tasks. Since
representation learning aims to extract and isolate
these factors, representations can be shared across
classes and tasks.
One of the most commonly used approaches for
training deep neural networks is based on greedy
layer-wise pre-training (Bengio et al. , 2007). The
idea, ﬁrst introduced in Hinton et al. (2006), is to
train one layer of a deep architecture at a time us-
ing unsupervised representation learning. Each level
takes as input the representation learned at the pre-
vious level and learns a new representation. The
learned representation(s) can then be used as in-
put to predict variables of interest, for example to
classify objects. After unsupervised pre-training,
one can also perform supervised ﬁne-tuning of the
whole system 6, i.e., optimize not just the classi-
ﬁer but also the lower levels of the feature hierar-
chy with respect to some objective of interest. Com-
bining unsupervised pre-training and supervised ﬁne-
tuning usually gives better generalization than pure
supervised learning from a purely random initializa-
tion. The unsupervised representation algorithms for
pre-training proposed in 2006 were the Restricted
Boltzmann Machine or RBM (Hinton et al. , 2006),
the auto-encoder (Bengio et al. , 2007) and a spar-
sifying form of auto-encoder similar to sparse cod-
ing (Ranzato et al. , 2007).
6 the whole system composes the computation of the rep-
resentation with computation of the predictor’s output,
1.2 Denoising and Contractive Auto-
Encoders
An auto-encoder has two parts: an encoder function
f that maps the input x to a representation h = f(x),
and a decoder function g that maps h back in the
space of x in order to reconstruct x. In the regu-
lar auto-encoder the reconstruction r(x) = g(f(x))
function is trained to minimize the average value of
a reconstruction loss on the training examples (but
reconstruction loss should be high for most other
input conﬁgurations 7). Examples of reconstruction
loss functions include ||x −r(x)||2 (for real-valued in-
puts) and −∑
i xi log ri(x) + (1 − xi) log(1 − ri(x))
(when interpreting xi as a bit or a probability of
a binary event). Auto-encoders capture the input
distribution by learning to better reconstruct more
likely input conﬁgurations . The diﬀerence between
the reconstruction vector and the input vector can
be shown to be related to the log-density gradient as
estimated by the learner (Vincent, 2011) and the Ja-
cobian matrix of the reconstruction with respect to
the input gives information about the second deriva-
tive of the density, i.e., in which direction the den-
sity remains high when you are on a high-density
manifold (Rifai et al. , 2011). In the Denoising Auto-
Encoder (DAE) and the Contractive Auto-Encoder
(CAE), the training procedure also introduces robust-
ness (insensitivity to small variations), respectively
in the reconstruction r(x) or in the representation
f(x). In the DAE (Vincent et al. , 2008, 2010), this
is achieved by training with stochastically corrupted
inputs, but trying to reconstruct the uncorrupted in-
puts. In the CAE (Rifai et al. , 2011), this is achieved
by adding an explicit regularizing term in the train-
ing criterion, proportional to the norm of the Jaco-
bian of the encoder, ||∂f (x)
∂x ||2. But the CAE and the
DAE are very related: when the noise is Gaussian and
small, the denoising error minimized by the DAE is
equivalent to minimizing the norm of the Jacobian of
the reconstruction function r(·) = g(f(·)) (whereas
7Diﬀerent mechanisms have been proposed to push recon-
struction error up in low density areas: denoising criterio n,
contractive criterion, and code sparsity. It has been argue d
that such constraints play a role similar to the partition fu nc-
tion for Boltzmann machines (Ranzato et al., 2008a)
3
the CAE minimizes the Jacobian of the encoder f(·).
Besides Gaussian noise, another interesting form of
corruption has been very successful with DAEs: it
is called the masking corruption and consists in ran-
domly zeroing out a large fraction (like 20% or even
50%) of the inputs, where the zeroed out subset is
randomly selected for each example. In addition to
the contraction eﬀect, it forces the learned encoder
to be able to rely only on an arbitrary subset of the
input features.
Another way to prevent the auto-encoder from per-
fectly reconstructing everywhere is to introduce a
sparsity penalty on h, discussed below (Section 3.1).
2 Gradients
2.1 Gradient Descent and Learning
Rate
Many learning algorithms rely on gradient-based nu-
merical optimization of a training criterion. Let
L(z, θ) be the loss incurred on example z when the
parameter vector takes value θ. The gradient vector
for the loss associated with a single example is ∂L (z,θ )
∂θ
and it is used as the core part the computation of pa-
rameter updates for gradient-based numerical opti-
mization algorithms. For example, simple online (or
stochastic) gradient descent (Robbins and Monro,
1951; Bottou and LeCun, 2004) updates the param-
eters after each example is seen according to
θ(t) ← θ(t−1) −ǫt
∂L(zt, θ)
∂θ
where zt is an example sampled at iteration t and
where ǫt is a hyper-parameter that is called the learn-
ing rate and whose choice is crucial. If the learn-
ing rate is too large 8, the average loss will increase.
The optimal learning rate is usually close to the
largest learning rate that does not cause divergence of
the training criterion , an observation that can guide
heuristics for setting the learning rate (Bengio, 2011),
e.g., start with a large learning rate and if the train-
ing criterion diverges, try again with 3 times smaller
8 above a value which is approximately 2 times the largest
eigenvalue of the average loss Hessian matrix.
learning rate, etc., until no divergence is observed.
The learning rate can be kept constant (with no guar-
antee of convergence), or it can be decreased appro-
priately (e.g., typically in O(1/t)) to ensure conver-
gence. A common heuristic we use is
ǫt = ǫ0τ
τ + t (1)
where we now have two hyper-parameters (instead
of one in the constant learning rate case): the ini-
tial learning rate ǫ0 and the time-constant of decay,
τ. In practice, we use mini-batch updates based on
the average of the gradients 9 inside each block of B
examples:
θ(t) ← θ(t−1) −ǫt
1
B
B(t+1)∑
t′=Bt+1
∂L(zt′ , θ)
∂θ . (2)
With B = 1 we are back to ordinary online gradient
descent, while with B equal to the training set size,
this is standard (also called ’batch’) gradient descent.
With intermediate values of B there is generally a
sweet spot. When B increases we can get faster com-
putation by taking advantage of parallelism or eﬃ-
cient matrix-matrix multiplications (instead of sep-
arate matrix-vector multiplications), often gaining a
factor of 2 in practice in overall training time. On the
other hand, as B increases, the number of updates
per computation done decreases, which slows down
clock-time convergence because less updates can be
done in the same computing time. Keep in mind that
the even the true gradient direction (averaging over
the whole training set) is only the steepest descent
direction locally but may not point in the right di-
rection when considering larger steps. In addition,
because the training criterion is not quadratic in the
parameters, as one moves in parameter space the op-
timal descent direction keeps changing. Because the
gradient direction is not quite the right direction of
descent, there is no point in spending a lot of com-
putation to estimate it precisely for gradient descent.
Instead, doing more updates more frequently helps to
9 Compared to a sum, an average makes a small change in
B have only a small eﬀect on the optimal learning rate, with an
increase in B generally allowing a small increase in the learning
rate because of the reduced variance of the gradient.
4
explore more and faster, especially with large learning
rates. In addition, smaller values of B may beneﬁt
from more exploration in parameter space and a form
of regularization both due to the ’noise’ injected in
the gradient estimator, which may explain the better
test results sometimes observed with smaller B.
When the training set is ﬁnite, training proceeds
by sweeps through the training set called an epoch,
and full training usually requires many epochs (itera-
tions through the training set). Note that stochastic
gradient (either one example at a time or with mini-
batches) is diﬀerent from ordinary gradient descent ,
sometimes called ’batch gradient descent’, which cor-
responds to the case where B equals the training set
size, i.e., there is one parameter update per epoch).
The great advantage of stochastic gradient descent
and other online or minibatch update methods is that
their convergence does not depend on the size of the
training set, only on the number of updates and the
richness of the training distribution. In the limit of a
large or inﬁnite training set, a batch method (which
updates only after seeing all the examples) is hope-
less. In fact, even for ordinary datasets of tens or hun-
dreds of thousands of examples (or more!), stochastic
gradient descent converges much faster than ordinary
(batch) gradient descent, and beyond some dataset
sizes the speed-up is almost linear (i.e., doubling the
size almost doubles the gain). It is really important
to use the stochastic version in order to get reason-
able clock-time convergence speeds.
As for any stochastic gradient descent method (in-
cluding the mini-batch case), it is important for ef-
ﬁciency of the estimator that each example or mini-
batch be sampled approximately independently. Be-
cause random access to memory (or even worse, to
disk) is expensive, a good approximation, called in-
cremental gradient (Bertsekas, 2010), is to visit the
examples (or mini-batches) in a ﬁxed order corre-
sponding to their order in memory or disk (repeating
the examples in the same order on a second epoch,
if not in the pure online case where each example is
visited only once), provided the examples or mini-
batches are ﬁrst put in a random order (to make
sure this is the case, it could be useful to ﬁrst shuﬄe
the examples). Faster convergence has been observed
if the order in which the mini-batches are visited is
shuﬄed before each epoch, which is eﬃcient if the
training set holds in computer memory.
2.2 Gradient Computation and Auto-
matic Diﬀerentiation
The gradient can be either computed manually or
through automatic diﬀerentiation. Either way, it
helps to structure this computation as a ﬂow graph ,
in order to prevent mathematical mistakes and make
sure an implementation is computationally eﬃcient.
The computation of the loss L(z, θ) as a function of
θ can be laid out in a graph whose nodes correspond
to elementary operations such as addition, multipli-
cation, and non-linear operations such as the neural
networks activation function (e.g., sigmoid or hyper-
bolic tangent), possibly at the level of vectors, matri-
ces or tensors. The ﬂow graph is directed and acyclic
and has three types of nodes: input nodes, internal
nodes, and output nodes. Each of its nodes is as-
sociated with a numerical output which is the result
of the application of that computation (none in the
case of input nodes), taking as input the output of
previous nodes in a directed acyclic graph. z and θ
(or their elements) are the input nodes of the graph
(they do not have inputs themselves) and L(z, θ) is
a scalar output of the graph. Note that here, in the
supervised case, z can include an input part x (e.g.
an image) and a target part y (e.g. a target class
associated with an object in the image). In the unsu-
pervised case z = x. In a semi-supervised case, there
is a mix of labeled and unlabeled examples, and z
includes y on the labeled examples but not on the
unlabeled ones.
In addition to associating a numerical output oa to
each node a of the ﬂow graph, we can associate a gra-
dient ga = ∂L (z,θ )
∂o a
. The gradient will be deﬁned and
computed recursively in the graph, in the opposite
direction of the computation of the nodes’ outputs,
i.e., whereas oa is computed using outputs op of pre-
decessor nodes p of a, ga will be computed using the
gradients gs of successor nodes s of a. More precisely,
the chain rule dictates
ga =
∑
s
gs
∂os
∂oa
5
where the sum is over immediate successors of a.
Only output nodes have no successor, and in par-
ticular for the output node that computes L, the
gradient is set to 1 since ∂L
∂L = 1, thus initializing
the recursion. Manual or automatic diﬀerentiation
then only requires to deﬁne the partial derivative as-
sociated with each type of operation performed by
any node of the graph. When implementing gradi-
ent descent algorithms with manual diﬀerentiation
the result tends to be verbose, brittle code that lacks
modularity – all bad things in terms of software en-
gineering. A better approach is to express the ﬂow
graph in terms of objects that modularize how to
compute outputs from inputs as well as how to com-
pute the partial derivatives necessary for gradient de-
scent. One can pre-deﬁne the operations of these ob-
jects (in a “forward propagation” or fprop method)
and their partial derivatives (in a “backward prop-
agation” or bprop method) and encapsulate these
computations in an object that knows how to com-
pute its output given its inputs, and how to com-
pute the gradient with respect to its inputs given
the gradient with respect to its output. This is the
strategy adopted in the Theano library 10 with its
Op objects (Bergstra et al. , 2010), as well as in li-
braries such as Torch 11 (Collobert et al. , 2011b) and
Lush 12.
Compared to Torch and Lush, Theano adds an in-
teresting ingredient which makes it a full-ﬂedged au-
tomatic diﬀerentiation tool: symbolic computation.
The ﬂow graph itself (without the numerical values
attached) can be viewed as a symbolic representation
(in a data structure) of a numerical computation. In
Theano, the gradient computation is ﬁrst performed
symbolically, i.e., each Op object knows how to create
other Ops corresponding to the computation of the
partial derivatives associated with that Op. Hence the
symbolic diﬀerentiation of the output of a ﬂow graph
with respect to any or all of its input nodes can be
performed easily in most cases, yielding another ﬂow
graph which speciﬁes how to compute these gradi-
ents, given the input of the original graph. Since the
gradient graph typically contains the original graph
10 http://deeplearning.net/software/theano/
11 http://www.torch.ch
12 http://lush.sourceforge.net
(mapping parameters to loss) as a sub-graph, in or-
der to make computations eﬃcient it is important to
automate (as done in Theano) a number of simpliﬁca-
tions which are graph transformations preserving the
semantic of the output (given the input) but yield-
ing smaller (or more numerically stable or more eﬃ-
ciently computed) graphs (e.g., removing redundant
computations). To take advantage of the fact that
computing the loss gradient includes as a ﬁrst step
computing the loss itself, it is advantageous to struc-
ture the code so that both the loss and its gradient are
computed at once, with a single graph having multi-
ple outputs. The advantages of performing gradient
computations symbolically are numerous. First of all,
one can readily compute gradients over gradients, i.e.,
second derivatives, which are useful for some learn-
ing algorithms. Second, one can deﬁne algorithms or
training criteria involving gradients themselves, as re-
quired for example in the Contractive Auto-Encoder
(which uses the norm of a Jacobian matrix in its
training criterion, i.e., really requires second deriva-
tives, which here are cheap to compute). Third, it
makes it easy to implement other useful graph trans-
formations such as graph simpliﬁcations or numerical
optimizations and transformations that help making
the numerical results more robust and more eﬃcient
(such as working in the domain of logarithms of prob-
abilities rather than in the domain of probabilities
directly). Other potential beneﬁcial applications of
such symbolic manipulations include parallelization
and additional diﬀerential operators (such as the R-
operator, recently implemented in Theano, which is
very useful to compute the product of a Jacobian
matrix partialf (x)
∂x or Hessian matrix ∂ 2L(x,θ )
∂θ 2 with a
vector without ever having to actually compute and
store the Hessian matrix itself (Pearlmutter, 1994)).
2.3 Online Learning and Optimization
of Generalization Error
The objective of learning is not to minimize train-
ing error or even the training criterion. The latter
is a surrogate for generalization error, i.e., perfor-
mance on new (out-of-sample) examples, and there
are no hard guarantees that minimizing the training
6
criterion will yield to good generalization error: it de-
pends on the appropriateness of the parametrization
and training criterion (with the corresponding prior
they imply) for the task at hand.
Many learning tasks of interest will require huge
quantities of data (most of which will be unlabeled)
and as the number of examples increases, so long as
capacity is limited (the number of parameters is small
compared to the number of examples), training er-
ror and generalization approach each other. In the
regime of such large datasets, we can consider that
the learner sees an unending stream of examples (e.g.,
think about a process that harvests text and images
from the web and feeds it to a machine learning algo-
rithm). In that context, it is most eﬃcient to simply
update the parameters of the model after each exam-
ple or few examples, as they arrive. This is the ideal
online learning scenario, and in a simpliﬁed setting,
we can even consider each new example z as being
sampled i.i.d. from an unknown generating distribu-
tion with probability density p(z). More realistically,
examples in online learning do not arrive i.i.d. but
instead from an unknown stochastic process which
exhibits serial correlation and other temporal depen-
dencies. However, if we consider the simpliﬁed case
of i.i.d. data, there is an interesting observation to
be made: the online learner is performing stochastic
gradient descent on its generalization error. Indeed,
the generalization error of a learner with parameters
θ and loss function L is
C = E[L(z, θ)] =
∫
p(z)L(z, θ)dz
while the stochastic gradient from sample z is
ˆg = ∂L(z, θ)
∂θ
with z a random variable sampled from p. The gra-
dient of generalization error is
∂C
∂θ = ∂
∂θ
∫
p(z)L(z, θ)dz =
∫
p(z)∂L(z, θ)
∂θ dz = E[ˆg]
showing that the online gradient ˆ g is an unbiased es-
timator of the generalization error gradient ∂C
∂θ . It
means that online learners, when given a stream of
non-repetitive training data, really optimize (maybe
not in the optimal way, i.e., using a ﬁrst-order gra-
dient technique) what we really care about: general-
ization error.
3 Hyper-Parameters
A pure learning algorithm can be seen as a func-
tion taking data as input and producing as output
a function (e.g. a predictor) or model (i.e. a bunch
of functions). However, in practice, many learning
algorithms involve hyper-parameters, i.e., annoying
buttons to be adjusted. In many algorithms such
as Deep Learning algorithms the number of hyper-
parameters (ten or more!) can make the idea of hav-
ing to adjust all of them unappealing. In addition,
it has been shown that the use of computer clusters
for hyper-parameter selection can have an important
eﬀect on results (Pinto et al. , 2009). Choosing hyper-
parameter values is formally equivalent to the ques-
tion of model selection , i.e., given a family or set of
learning algorithms, how to pick the most appropri-
ate one? We deﬁne a hyper-parameter for a learning
algorithm A as a value to be selected prior to the ac-
tual application of A to the data, a value that is not
directly selected by the learning algorithm itself . It
is basically an outside control knob. It can be dis-
crete (as in model selection) or continuous (such as
the learning rate discussed above). Of course, one
can hide these hyper-parameters by wrapping an-
other learning algorithm, say B, around A, to selects
A’s hyper-parameters (e.g. to minimize validation set
error). We can then call B a hyper-learner, and if B
has no hyper-parameters itself then the composition
of B over A could be a “pure” learning algorithm,
with no hyper-parameter. In the end, to apply a
learner to data, one has to have a pure learning algo-
rithm. The hyper-parameters can be ﬁxed by hand or
tuned by an algorithm, but they have to be selected.
Some hyper-parameters can be selected based on the
performance of A on its training data, but most can-
not. For any hyper-parameter that has an impact
on the eﬀective capacity of a learner, it makes more
sense to select its value based on out-of-sample data
(outside the training set), e.g., a validation set perfor-
7
mance, online error, or cross-validation error. Note
that some learning algorithms (in particular unsu-
pervised learning algorithms such as algorithms for
training RBMs by approximate maximum likelihood)
are problematic in this respect because we cannot di-
rectly measure the quantity that is to be optimized
(e.g. the likelihood) because it is intractable. On
the other hand, the expected denoising reconstruc-
tion error is easy to estimate (by just averaging the
denoising error over a validation set).
Once some out-of-sample data has been used for se-
lecting hyper-parameters, it cannot be used anymore
to obtain an unbiased estimator of generalization per-
formance, so one typically uses a test set (or double
cross-validation 13, in the case of small datasets) to
estimate generalization error of the pure learning al-
gorithm (with hyper-parameter selection hidden in-
side).
3.1 Neural Network Hyper-
Parameters
Diﬀerent learning algorithms involve diﬀerent sets of
hyper-parameters, and it is useful to get a sense of the
kinds of choices that the practitioners have to make,
and we focus here mostly on those relevant to neural
networks and Deep Learning algorithms.
3.1.1 Hyper-Parameters of the Approximate
Optimization
First of all, several learning algorithms can be viewed
as the combination of a training criterion and a
model on the one hand and of a particular proce-
dure for approximately optimizing this criterion on
the other hand. Correspondingly, one should dis-
tinguish hyper-parameters associated with the op-
timizer from hyper-parameters associated with the
model itself, i.e., typically the function class, regular-
izer and loss function. We have already mentioned
13 Double cross-validation applies recursively the idea of
cross-validation, using an outer loop cross-validation to evalu-
ate generalization error and then applying an inner loop cro ss-
validation inside each outer loop split’s training subset ( i.e.,
splitting it again into training and validation folds) in or der to
select hyper-parameters for that split.
above some of the hyper-parameters typically asso-
ciated with stochastic gradient descent with mini-
batches. Here is a more extensive descriptive list,
focusing on those used in stochastic (mini-batch) gra-
dient descent (although number of training iterations
is used for all iterative optimization algorithms).
•The initial learning rate (ǫ0 above, Eq.(1)).
This is often the single most important hyper-
parameter and one should always make sure that
it has been tuned (up to approximately a fac-
tor of 2). Typical values for a neural network
with standardized inputs (or inputs mapped to
the (0,1) interval) are less than 1 and greater
than 10 −6 but these should not be taken as strict
ranges and greatly depend on the parametriza-
tion of the model. A default value of 0.01 typi-
cally works for standard multi-layer neural net-
works but it would be foolish to rely exclu-
sively on this default value. If there is only
time to optimize one hyper-parameter and one
uses stochastic gradient descent, then this is the
hyper-parameter that is worth tuning.
•The hyper-parameters of the learning rate
schedule (such as the time constant τ in
Eq. (1)). The default value of τ →∞ means
that the learning rate is constant over training
iterations. In many cases the beneﬁt of choosing
other than this default value is small. An alter-
native to Eq. (1) used in Bergstra and Bengio
(2012) is
ǫt = ǫ0τ
max(t, τ )
which keeps the learning rate constant for the
ﬁrst τ steps and then decreases it in O(1/tα ),
with traditional recommendations (based on
asymptotic analysis of the convex case) suggest-
ing α = 1. See Bach and Moulines (2011) for a
recent analysis of the rate of convergence for the
general case of α ≤ 1, suggesting that smaller
values of α should be used in the non-convex
case, especially when using a gradient averaging
or momentum technique (see below). An adap-
tive and heuristic way of automatically setting
τ above is to keep ǫt constant until the training
8
criterion stops decreasing signiﬁcantly (by more
than some relative improvement threshold) from
epoch to epoch. That threshold is a less sensitive
hyper-parameter than τ itself.
•The mini-batch size (B in Eq. (2)) is typically
chosen between 1 and a few hundreds, e.g. B =
32 a good default value, with values above 10 to
take advantage of the speed-up of matrix-matrix
products over matrix-vector products. The im-
pact of B is mostly computational, i.e., larger B
yield faster computation (with appropriate im-
plementations) but requires visiting more exam-
ples in order to reach the same error, since there
are less updates per epoch. In theory, this hyper-
parameter should impact training time and not
really test performance , so it can be optimized
separately of the other hyper-parameters, by
comparing training curves (training and valida-
tion error vs amount of training time), after the
other hyper-parameters (except learning rate)
have been selected. B and ǫ0 may slightly inter-
act so both should be re-optimized at the end.
Once B is selected, it can generally be ﬁxed while
the other hyper-parameters can be further opti-
mized.
•Number of training iterations T (measured
in mini-batch updates). This hyper-parameter
is particular in that it can be optimized almost
for free using the principle of early stopping : by
keeping track of the out-of-sample error (as for
example estimated on a validation set) as train-
ing progresses (every N updates), one can decide
how long to train for any given setting of all the
other hyper-parameters. Early stopping is an
inexpensive way to avoid strong overﬁtting, i.e.,
even if the other hyper-parameters would yield
to overﬁtting, early stopping will considerably
reduce the overﬁtting damage that would other-
wise ensue. It also means that it hides the over-
ﬁtting eﬀect of other hyper-parameters, possibly
obscuring the analysis that one may want to do
when trying to ﬁgure out the eﬀect of individual
hyper-parameters, i.e., it tends to even out the
performance obtained by many otherwise overﬁt-
ting conﬁgurations of hyper-parameters by com-
pensating a too large capacity with a smaller
training time. For this reason, it might be use-
ful to turn early-stopping oﬀ when analyzing the
eﬀect of individual hyper-parameters. Now let
us turn to implementation details. Practically,
one needs to continue training beyond the se-
lected number of training iterations ˆT (which
should be the point of lowest validation error
in the training run) in order to ascertain that
validation error is unlikely to go lower than at
the selected point. A heuristic introduced in the
Deep Learning Tutorials 14 is based on the idea
of patience (set initially to 10000 examples in the
MLP tutorial), which is a minimum number of
training examples to see after the candidate se-
lected point ˆT before deciding to stop training
(i.e. accept this candidate as the ﬁnal answer).
As new candidate selected points ˆT (new minima
of the validation error) are observed, patience
is increased, either multiplicatively or additively
on top of the last ˆT found. Hence, if we ﬁnd a
new minimum 15 at t, we save the current best
model, update ˆT ← t and we increase our pa-
tience up to t + constant or t× constant. Note
that validation error should not be estimated af-
ter each training update (that would be really
wasteful) but after every N examples, where N
is at least as large as the validation set (ideally
several times larger so that the early stopping
overhead remains small).
•Momentum β. It has long been advo-
cated (Hinton, 1978, 2010) to temporally smooth
out the stochastic gradient samples obtained
during the stochastic gradient descent. For ex-
ample, a moving average of the past gradient
can be computed with ¯ g ← (1 − β)¯g + βg,
where g = ∂L (zt,θ )
∂θ is the instantaneous gradi-
ent and β is a small positive coeﬃcient that
controls how fast the old examples get down-
weighted in the moving average. The simplest
14 http://deeplearning.net/tutorial/
15Ideally, we should use a statistical test of signiﬁcance and
accept a new minimum (over a longer training period) only if
the improvement is statistically signiﬁcant, based on the s ize
and variance estimates one can compute for the validation se t.
9
momentum recipe is to make the updates pro-
portional to this smoothed gradient estimator
¯g instead of the instantaneous gradient g. The
idea is that it removes some of the noise and
oscillations that gradient descent has, in partic-
ular in the directions of high curvature of the
loss function 16. A default value of β = 1
works well in many cases but in some cases
momentum seems to make a positive diﬀer-
ence. Polyak averaging (Polyak and Juditsky,
1992) is a related form of parameter aver-
aging 17 that has theoretical advantages and
been advocated and shown to bring improve-
ments on some unsupervised learning procedures
such as RBMs (Swersky et al. , 2010). More re-
cently, several mathematically motivated algo-
rithms (Nesterov, 2009; Le Roux et al. , 2012)
have been proposed that incorporate some form
of momentum and that also ensure much faster
convergence (linear rather than sublinear) com-
pared to stochastic gradient descent, at least
for convex optimization problems. Note how-
ever that in the pure online case (stream of ex-
amples) and under some assumptions, the sub-
linear rate of convergence of stochastic gradi-
ent descent with O(1/t) decrease of learning
rate is an optimal rate, at least for convex
problems (Nemirovski and Yudin, 1983). That
would suggest that for really large training
sets it may not be possible to obtain better
rates than ordinary stochastic gradient descent,
albeit the constants in front (which depend
on the condition number of the Hessian) may
still be greatly reduced by using second-order
information online (Bottou and LeCun, 2004;
Bottou and Bousquet, 2008).
•Layer-wise optimization hyper-
16 Think about a ball coming down a valley. Since it has not
started from the bottom of the valley it will oscillate betwe en
its sides as it settles deeper, forcing the learning rate to b e
small to avoid large oscillations that would kick it out of th e
valley. Averaging out the local gradients along the way will
cancel the opposing forces from each side of the valley.
17 Polyak averaging uses for predictions a moving average of
the parameters found in the trajectory of stochastic gradie nt
descent.
parameters: although rarely done, it is
possible to use diﬀerent values of optimization
hyper-parameters (such as the learning rate) on
diﬀerent layers of a multi-layer network. This is
especially appropriate (and easier to do) in the
context of layer-wise unsupervised pre-training,
since each layer is trained separately (while the
layers below are kept ﬁxed). This would be
particularly useful when the number of units
per layer varies a lot from layer to layer. See
the paragraph below entitled Layer-wise opti-
mization of hyper-parameters (Sec. 3.3.4).
Some researchers also advocate the use of
diﬀerent learning rates for the diﬀerent types
of parameters one ﬁnds in the model (such as
biases and weights in the standard multi-layer
network, but the issue becomes more important
when parameters such as precision or variance
are included in the lot (Courville et al. , 2011).
With other optimization algorithms, the hyper-
parameters are typically diﬀerent. For example,
Conjugate Gradients (CG) algorithms typically have
a number of line search steps (which is a hyper-
parameter) and a tolerance for stopping each line
search (another hyper-parameter). An optimization
algorithm like L-BFGS (limited-memory BFGS) also
has a hyper-parameter controlling the memory usage
of the algorithm, the rank of the Hessian approxima-
tion kept in memory, which also has an inﬂuence on
the eﬃciency of each step. Both CG and L-BFGS are
iterative (e.g. the number of line searchers) and the
number of iterations can be optimized as described
above for stochastic gradient descent, with early stop-
ping.
3.2 Hyper-Parameters of the Model
and Training Criterion
Let us now turn to “model” and “criterion” hyper-
parameters typically found in neural networks, espe-
cially deep neural networks.
•Number of hidden units nh. Each layer in a
multi-layer neural network typically has a size
that we are free to set and that controls ca-
pacity. Because of early stopping and possibly
10
other regularizers (e.g., weight decay, discussed
below), it is mostly important to choose nh large
enough. Larger than optimal values typically do
not hurt generalization performance much, but
of course they require proportionally more com-
putation (in O(n2
h) if scaling all the layers at
the same time in a fully connected architecture).
Like for many other hyper-parameters, there is
the option of allowing a diﬀerent value of nh for
each hidden layer 18 of a deep architecture. See
the paragraph below entitled Layer-wise opti-
mization of hyper-parameters (Sec. 3.3.4).
In a large comparative study (Larochelle et al. ,
2009), we found that using the same size for all
layers worked generally better or the same as us-
ing a decreasing size (pyramid-like) or increasing
size (upside down pyramid), but of course this
may be data-dependent. For most tasks that we
worked on, we ﬁnd that an overcomplete 19 ﬁrst
hidden works better than an undercomplete one.
Another even more often validated empirical ob-
servation is that the optimal nh is much larger
when using unsupervised pre-training in a super-
vised neural network, e.g., going from hundreds
of units to thousands of units.
•Weight decay regularization coeﬃcient λ. A
way to reduce overﬁtting is to add a regulariza-
tion term to the training criterion, which limits
the capacity of the learner. The parameters of
machine learning models can be regularized by
pushing them towards a prior value, which is typ-
ically 0. L2 regularization adds a term λ ∑
i θ2
i
to the training criterion, while L1 regularization
adds a term λ ∑
i |θi|. Both types of terms can
be included. There is a clean Bayesian justiﬁ-
cation for such a regularization term: it is the
negative log-prior −log P (θ) on the parameters
θ. The training criterion then corresponds to the
negative joint likelihood of data and parameters,
−log P (data|θ) − log P (θ), with the loss func-
tion L(z, θ) being interpreted as −log P (z|θ) and
−log P (data|θ) = −∑ T
t=1 L(zt, θ) if the data
18 A hidden layer is a group of units that is neither an input
layer nor an output layer.
19 larger than the input vector
consists of T i.i.d. examples zt. This detail
is important to note because when one is do-
ing stochastic gradient-based learning, there is a
need to use the gradient of an unbiased estima-
tor of the gradient of the total training criterion
(including both the total loss and the regular-
izer), but one only considers a single mini-batch
or example at a time. How should the regularizer
be weighted in this sum, which is diﬀerent from
the sum of the regularizer and the total loss on
all examples? On each mini-batch update, the
gradient of the regularization penalty should be
multiplied not just by λ but also by B
T , i.e., one
over the number of updates needed to go once
through the training set. When the training set
size is not a multiple of B, the last mini-batch
will have size B′ < B and the contribution of
the regularizer to the mini-batch gradient should
therefore be modiﬁed accordingly (i.e. scaled by
B′
B compared to other mini-batches). In the pure
online setting (there is no ﬁxed ahead of time
training set size nor iterating again on the ex-
amples), it would then make sense to use B
t at
example t, or one over the number of updates
to date. L2 regularization penalizes large val-
ues more strongly and corresponds to a Gaus-
sian prior ∝ exp(−1
2
||θ||2
σ 2 ) with prior variance
σ2 = 1 /(2λ). Note that there is a connection
between early stopping (see above, choosing the
number of training iterations) and L2 regular-
ization (Collobert and Bengio, 2004), with one
basically playing the same role as the other (but
early stopping allowing a much more eﬃcient se-
lection of the hyper-parameter value, which sug-
gests dropping L2 regularization altogether when
early-stopping is used). L1 regularization makes
sure that parameters that are not really very use-
ful are driven to zero (i.e. encouraging sparsity
of the parameter values), and corresponds to a
Laplace density prior ∝ e− |θ |
s with scale parame-
ter s = 1
λ . L1 regularization often helps to make
the input ﬁlters 20 cleaner (more spatially local-
20 The input weights of a 1st layer neuron are often called
“ﬁlters” because of analogies with signal processing techn iques
such as convolutions.
11
ized) and easier to interpret. Stochastic gradi-
ent descent will not yield actual zeros but values
hovering around zero. If both L1 and L2 regu-
larization are used, a diﬀerent coeﬃcient (i.e. a
diﬀerent hyper-parameter) should be considered
for each, and one may also use a diﬀerent coeﬃ-
cient for diﬀerent layers. In particular, the input
weights and output weights may be treated dif-
ferently.
One reason for treating output weights diﬀer-
ently is that we know that it is suﬃcient to reg-
ularize only the output weights in order to con-
strain capacity: in the limit case of the num-
ber of hidden units going to inﬁnity, L2 regular-
ization corresponds to Support Vector Machines
(SVM) while L1 regularization corresponds to
boosting (Bengio et al. , 2006a). Another reason
for treating inputs and outputs diﬀerently from
hidden units is because they may be sparse. For
example, some input features may be 0 most of
the time while others are non-zero frequently. In
that case, there are fewer examples that inform
the model about that rarely active input feature,
and the corresponding parameters (weights out-
going from the corresponding input units) should
be more regularized than the parameters associ-
ated with frequently observed inputs. A similar
situation may occur with target variables that
are sparse (e.g., trying to predict rarely observed
events). In both cases, the eﬀective number of
meaningful updates seen by these parameters is
less than the actual number of updates. This
suggests to scale the regularization coeﬃcient of
these parameters by one over the eﬀective num-
ber of updates seen by the parameter. A related
formula turns up in Bayesian probit regression
applied to sparse inputs (Graepel et al. , 2010).
Some practitioners also choose to penalize only
the weights w and not the biases b associated
with the hidden unit activations w′z+b for a unit
taking the vector of values z as input. This guar-
antees that even with strong regularization, the
predictor would converge to the optimal constant
predictor, rather than the one corresponding to
0 activation. For example, with the mean-square
loss and the cross-entropy loss, the optimal con-
stant predictor is the output average.
•Sparsity of activation regularization coeﬃ-
cient α. A common practice in the Deep
Learning literature (Ranzato et al. , 2007, 2008b;
Lee et al. , 2008, 2009; Bagnell and Bradley,
2009; Glorot et al. , 2011a; Coates and Ng, 2011;
Goodfellow et al. , 2011) consists in adding a
penalty term to the training criterion that en-
courages the hidden units to be sparse, i.e.,
with values at or near 0. Although the L1
penalty (discussed above in the case of weights)
can also be applied to hidden units activations,
this is mathematically very diﬀerent from the
L1 regularization term on parameters. Whereas
the latter corresponds to a prior on the pa-
rameters, the former does not because it in-
volves the training distribution (since we are
looking at data-dependent hidden units out-
puts). Although we will not discuss this much
here, the inspiration for a sparse representa-
tion in Deep Learning comes from the ear-
lier work on sparse coding (Olshausen and Field,
1997). As discussed in Goodfellow et al. (2009)
sparse representations may be advantageous be-
cause they encourage representations that dis-
entangle the underlying factors of representa-
tion. A sparsity-inducing penalty is also a
way to regularize (in the sense of reducing the
number of examples that the learner can learn
by heart) (Ranzato et al. , 2008b), which means
that the sparsity coeﬃcient is likely to interact
with the many other hyper-parameters which in-
ﬂuence capacity. In general, increased sparsity
can be compensated by a larger number of hid-
den units.
Several approaches have been proposed to in-
duce a sparse representation (or with more hid-
den units whose activation is closer to 0). One
approach (Ranzato et al. , 2008b; Le et al. , 2011;
Zou et al. , 2011) is simply to penalize the L1
norm of the representation or another function of
the hidden units activation (such as the student-
t log-prior). This typically makes sense for non-
linearities such as the sigmoid which have a sat-
12
urating output around 0, but not for hyper-
bolic tangent non-linearity (whose saturation is
near the -1 and 1 extremes rather than near the
origin). Another option is to penalize the bi-
ases of the hidden units, to make them more
negative (Ranzato et al. , 2007; Lee et al. , 2008;
Goodfellow et al. , 2009; Larochelle and Bengio,
2008). Note that penalizing the bias runs the
danger that the weights could compensate for
the bias 21, which could hurt the numerical op-
timization of parameters. When directly pe-
nalizing the hidden unit outputs, several vari-
ants can be found in the literature, but no
clear comparative analysis has been published
to evaluate which one works better. Although
the L1 penalty (i.e., simply α times the sum
of output elements hj in the case of sigmoid
non-linearity) would seem the most natural (be-
cause of its use in sparse coding), it is used in
few papers involving sparse auto-encoders. A
close cousin of the L1 penalty is the Student-
t penalty (log(1 + h2
j )), originally proposed for
sparse coding (Olshausen and Field, 1997). Sev-
eral researchers penalize the average output ¯
hj
(e.g. over a mini-batch), and instead of push-
ing it to 0, encourage it to approach a ﬁxed tar-
get. This can be done through a mean-square
error penalty such as ∑
j (ρ − ¯hj )2, or maybe
more sensibly (because ˆ
h behaves like a prob-
ability), a Kullback-Liebler divergence with re-
spect to the binomial distribution with probabil-
ity ρ, −ρ log ¯hj − (1 − ρ) log(1 − ¯hj )+constant,
e.g., with ρ = 0 .05, as in (Hinton, 2010). In ad-
dition to regularization penalty itself, the choice
of activation function can have a strong impact
on the sparsity obtained. In particular, rectify-
ing non-linearities (such as max(0 , x), instead of
a sigmoid) have been very successful in several
instances (Jarrett et al. , 2009; Nair and Hinton,
2010; Glorot et al. , 2011a; Mesnil et al. , 2011;
Glorot et al. , 2011b). In sparse coding and
sparse predictive coding (Kavukcuoglu et al. ,
2009) the activations are directly optimized and
21 because the input to the layer generally has a non-zero
average, that when multiplied by the weights acts like a bias
actual zeros are the expected result of the op-
timization. In that case, ordinary stochastic
gradient is not guaranteed to ﬁnd these zeros
(it will oscillate around) and other methods
such as proximal gradient are more appropri-
ate (Bertsekas, 2010).
•Neuron non-linearity . The typical neuron
output is s(a) = s(w′x + b), where x is the
vector of inputs into the neuron, w the vec-
tor of weights and b the oﬀset or bias pa-
rameter, while s is a scalar non-linear func-
tion. Several non-linearities have been proposed
and some choices of non-linearities have been
shown to be more successful (Jarrett et al. , 2009;
Glorot and Bengio, 2010; Glorot et al. , 2011a).
The most commonly used by the author, for hid-
den units, are the sigmoid 1 /(1+e−a), the hyper-
bolic tangent ea−e−a
ea +e−a and the rectiﬁer max(0 , a).
Note that the sigmoid was shown to yield se-
rious optimization diﬃculties when used as the
top hidden layer of a deep supervised network
(Glorot and Bengio, 2010) without unsupervised
pre-training, but works well for auto-encoder
variants 22 For output (or reconstruction) units,
hard neuron non-linearities like the rectiﬁer do
not make sense because when the unit is satu-
rated (e.g. a < 0 for the rectiﬁer) and associ-
ated with a loss, no gradient is propagated in-
side the network, i.e., there is no chance to cor-
rect the error 23 In the case of hidden layers the
gradient manages to go through a subset of the
hidden units, even if the others are saturated.
For output units a good recipe is to obtain the
output non-linearity and the loss by considering
the associated negative log-likelihood and choos-
ing appropriate (conditional) output probability
22 The author hypothesizes that this discrepency is due
to the fact that the weight matrix W of an auto-encoder of
the form r(x) = W T sigmoid(W x) is pulled towards being or-
thonormal since this would make the auto-encoder closer to t he
identity function, because W T W x≈ x when W is orthonormal
and x is in the span of the rows of W .
23 A hard non-linearity for the output units non-linearity is
very diﬀerent from a hard non-linearity in the loss function ,
such as the hinge loss. In the latter case the derivative is 0
only when there is no error.
13
model, usually in the exponential family. For ex-
ample, one can typically take squared error and
linear outputs to correspond to a Gaussian out-
put model, cross-entropy and sigmoids to cor-
respond to a binomial output model, and loss
−log output[target class] with softmax outputs
to correspond to multinomial output variables.
For reasons yet to be elucidated, having a sig-
moidal non-linearity on the output (reconstruc-
tion) units (along with target inputs normalized
in the (0,1) interval) seems to be helpful when
training the contractive auto-encoder.
•Weights initialization scaling coeﬃcient .
Biases can generally be initialized to zero but
weights need to be initialized carefully to break
the symmetry between hidden units of the same
layer 24 Because diﬀerent output units receive
diﬀerent gradient signals, this symmetry break-
ing issue does not concern the output weights
(into the output units), which can therefore
also be set to zero. Although several ﬁxed
recipes (LeCun et al. , 1998a; Glorot and Bengio,
2010) for initializing the weights into hid-
den layers have been proposed (i.e. a
hyper-parameter is the discrete choice between
them), Bergstra and Bengio (2012) also inserted
as extra hyper-parameter a scaling coeﬃcient
for the initialization range. These recipes
are based on the idea that units with more
inputs (the fan-in of the unit) should have
smaller weights. Both LeCun et al. (1998a) and
Glorot and Bengio (2010) recommend scaling by
the inverse of the square root of the fan-in, al-
though Glorot and Bengio (2010) and the Deep
Learning Tutorials use a combination of the fan-
in and fan-out, e.g., sample a Uniform( −r, r)
with r =
√
6/(fan-in + fan-out) for hyperbolic
tangent units and r = 4
√
6/(fan-in + fan-out)
for sigmoid units. We have found that we could
avoid any hyper-parameter related to initializa-
tion using these formulas (and the derivation in
24 By symmetry, if hidden units of the same layer share the
same input and output weights, they will compute the same
output and receive the same gradient, hence performing the
same update and remaining identical, thus wasting capacity .
Glorot and Bengio (2010) can be used to derive
the formula for other settings). Note however
that in the case of RBMs, a zero-mean Gaussian
with a small standard deviation around 0.1 or
0.01 works well (Hinton, 2010) to initialize the
weights, while visible biases are typically set to
their optimal value if the weights were 0, i.e.,
log(¯x/(1 − ¯x) in the case of a binomial visible
unit whose corresponding binary input feature
has empirical mean ¯ x in the training set.
An important choice is whether one should use
unsupervised pre-training (and which unsuper-
vised feature learning algorithm to use) in or-
der to initialize parameters. In most settings
we have found unsupervised pre-training to help
and very rarely to hurt, but of course that
implies additional training time and additional
hyper-parameters.
•Random seeds. There are often several sources
of randomness in the training of neural net-
works and deep learners (such as for random
initialization, sampling examples, sampling hid-
den units in stochastic models such as RBMs,
or sampling corruption noise in denoising auto-
encoders). Some random seeds could therefore
yield better results than others. Because of the
presence of local minima in the training criterion
of neural networks (except in the linear case or
with ﬁxed lower layers), parameter initialization
matters. See Erhan et al. (2010b) for an exam-
ple of measuring the histogram of test errors for
hundreds of diﬀerent random seeds. Typically,
the choice of random seed only has a slight eﬀect
on the result and can mostly be ignored general
or for most of the hyper-parameter search pro-
cess. If computing power is available, then a
ﬁnal set of jobs with diﬀerent random seeds (5
to 10) for a small set of best choices of hyper-
parameter values can squeeze a bit more perfor-
mance. Another way to exploit computing power
to push performance a bit is model averaging, as
in Bagging (Breiman, 1994) and Bayesian meth-
ods. After training them, the outputs of dif-
ferent networks (or in general diﬀerent learning
algorithms) can be averaged. For example, the
14
diﬀerence between the neural networks being av-
eraged into a committee may come from the dif-
ferent seeds used for parameter initialization, or
the use of diﬀerent subsets of input variables, or
diﬀerent subsets of training examples (the latter
being called Bagging).
•Preprocessing. Many preprocessing steps have
been proposed to massage raw data into ap-
propriate inputs for neural networks and model
selection must also choose among them. In
addition to element-wise standardization (sub-
tract mean and divide by standard devia-
tion), Principal Components Analysis (PCA)
has often been advocated (LeCun et al. , 1998a;
Bergstra and Bengio, 2012) and also allows di-
mensionality reduction, at the price of an extra
hyper-parameter (the number of principal com-
ponents retained, or the proportion of variance
explained). A convenient non-linear preprocess-
ing is the uniformization (Mesnil et al. , 2011) of
each feature (which estimates its cumulative dis-
tribution Fi and then transforms each feature
xi by its quantile F −1
i (xi), i.e., returns an ap-
proximate normalized rank or quantile for the
value xi). A simpler to compute transform that
may help reduce the tails of input features is a
non-linearity such as the logarithm or the square
root, in an attempt to make to make them more
Gaussian-like.
In addition to the above somewhat generic choices,
more choices arise with diﬀerent architectures and
learning algorithms. For example, the denois-
ing auto-encoder has a hyper-parameter scaling the
amount of input corruption and the contractive auto-
encoder has as hyper-parameter a coeﬃcient scaling
the norm of the Jacobian of the encoder, i.e., control-
ling the importance of the contraction penalty. The
latter seems to be a rather sensitive hyper-parameter
that must be tuned carefully. The contractive auto-
encoder’s success also seems sensitive to the weight
tying constraint used in many auto-encoder archi-
tectures: the decoder’s weight matrix is equal to the
transpose of the encoder’s weight matrix. The spe-
ciﬁc architecture used in the contractive auto-encoder
(with tied weights, sigmoid non-linearies on hidden
and reconstruction units, along with squared loss or
cross-entropy loss) works quite well but other related
variants do not always train well, for reasons that
remain to be understood.
There are also many architectural choices that
are relevant in the case of convolutional architec-
tures (e.g. for modeling images, time-series or
sound) (LeCun et al. , 1989, 1998b; Le et al. , 2010) in
which hidden units have local receptive ﬁelds. Their
discussion is postponed to another chapter (LeCun,
2013).
3.3 Manual Search and Grid Search
Many of the hyper-parameters or model choices de-
scribed above can be ignored by picking a standard
recipe suggested here or in some other paper. Still,
one remains with a substantial number of choices to
be made, which may give the impression of neural
network training as an art. With modern comput-
ing facilities based on large computer clusters, it is
however possible to make the optimization of hyper-
parameters a more reproducible and automated pro-
cess, using techniques such as grid search or better,
random search, or even hyper-parameter optimiza-
tion, all discussed below.
3.3.1 General guidance for the exploration of
hyper-parameters
First of all, let us consider recommendations for ex-
ploring hyper-parameter settings, whether with man-
ual search, an automated procedure, or a combina-
tion of both. We call a numerical hyper-parameter
one that involves choosing a real number or an in-
teger, as opposed to making a discrete choice from
an unordered set. Examples of numerical hyper-
parameters are regularization coeﬃcients, number of
hidden units, number of training iterations, etc. One
has to think of hyper-parameter selection as a diﬃ-
cult form of learning. The training criterion for this
learning is the validation error, which is a proxy for
generalization error. Unfortunately, the relation be-
tween hyper-parameters and validation error can be
complicated. Although to ﬁrst approximation we ex-
pect a kind of U-shaped curve (when considering only
15
a single hyper-parameter, the others being ﬁxed), this
curve can also have noisy variations, in part due to
the use of ﬁnite data sets.
•Best value on the border. When considering
the validation error obtained for diﬀerent values
of a numerical hyper-parameter one should pay
attention as to whether or not the best value
found is near the border of the investigated in-
terval. If it is near the border, then it is recom-
mended to explore further beyond that border,
for the obvious reason that such a result sug-
gests strongly that better values could be found
beyond. Because the relation between a hyper-
parameter and validation error can be noisy, it
is generally not enough to try very few values.
For instance, trying only 3 values for a numer-
ical hyper-parameter is insuﬃcient, even if the
best value found is the middle one.
•Scale of values considered. Exploring values
of a numerical hyper-parameter entails choosing
a starting interval, which is therefore a kind of
hyper-hyper-parameter. By choosing the inter-
val large enough to start with, but based on pre-
vious experience with this hyper-parameter, we
ensure that we do not get completely wrong re-
sults. Now instead of choosing the intermedi-
ate values linearly in the chosen interval, it of-
ten makes much more sense to consider a linear
or uniform sampling in the log-domain (in the
space of the logarithm of the hyper-parameter).
For example, the results obtained with a learn-
ing rate of 0.01 are likely to be very similar to
the results with 0.011 while results with 0.001
could be quite diﬀerent from results with 0.002
even though the diﬀerence is the same in both
cases. The ratio between diﬀerent values is a bet-
ter guide of the expected impact of the change.
That is why exploring uniformly or regularly-
spaced values in the space of the logarithm of
the numerical hyper-parameter is typically pre-
ferred.
•Computational considerations. Validation
error is actually not the only measure to consider
in selecting hyper-parameters. Often, one has to
consider computational cost, either of training
or prediction. Computing resources for training
and prediction are limited and generally condi-
tion the choice of intervals of values to consider
(for example increasing the number of hidden
units or number of training iterations also scales
up computation). An interesting idea is to use
computationally cheap estimators of validation
error to select some hyper-parameters. For ex-
ample, Saxe et al. (2011) showed that the archi-
tecture hyper-parameters of convolutional net-
works could be selected using random weights in
the lower layers of the network. While this yields
a noisy and biased (pessimistic) estimator of the
validation error which could be obtained with
full training, it appears to be correlated with it,
which is enough for hyper-parameter selection
(or for keeping under consideration for further
and more expensive evaluation only the few best
choices found). Even without cheap estimators
of generalization errors, high-throughput com-
puting (e.g., on clusters, GPUs, or clusters of
GPUs) can be exploited to run not just hun-
dreds but thousands of training jobs, something
not conceivable only a few years ago, with each
job taking on the order of hours or days for
larger datasets. With cheap surrogates, some
researchers have run on the order of ten thou-
sands trials, and we can expect future advances
in parallelized computing power to boost these
numbers.
3.3.2 Coordinate Descent and Multi-
Resolution Search
When performing a manual search and with access to
only a single computer, a reasonable strategy is coor-
dinate descent : change only one hyper-parameter at
time, always making a change from the best conﬁgu-
ration of hyper-parameters found up to now. Instead
of a standard coordinate descent (which systemati-
cally cycles through all the variables to be optimized)
one can make sure to regularly ﬁne-tune the most
sensitive variables, such as the learning rate.
Another important idea is that there is no point
in exploring the eﬀect of ﬁne changes before one or
16
more reasonably good settings have been found. The
idea of multi-resolution search is to start the search
by considering large changes of the numerical hyper-
parameters. One can then start from the one or few
best conﬁgurations found and explore more locally
around them with smaller changes around these val-
ues.
3.3.3 Automated and Semi-automated Grid
Search
Once some interval or set of values has been selected
for each hyper-parameter (thus deﬁning a search
space), a simple strategy that exploits parallel com-
puting is the grid search. One ﬁrst needs to con-
vert the numerical intervals into lists of values (e.g.,
K regularly-spaced values in the log-domain of the
hyper-parameter). The grid search is simply an ex-
haustive search through all the combinations of these
values. The cross-product of these lists contains a
number of elements that is unfortunately exponen-
tial in the number of hyper-parameters (e.g., with
5 hyper-parameters, each allowed to take 6 diﬀerent
values, one gets 6 5 = 7776 conﬁgurations). In sec-
tion 3.4 below we consider an approach that works
more eﬃciently than the grid search when the num-
ber of hyper-parameters increases beyond 2 or 3.
The advantage of the grid search, compared to
many other optimization strategies (such as coordi-
nate descent), is that it is fully parallelizable. If a
large computer cluster is available, it is tempting to
choose a model selection strategy that can take ad-
vantage of parallelization. One practical disadvan-
tage of grid search, with a parallelized set of jobs on
a cluster, is that if only one of the jobs fails (for all
kinds of hardware and software reasons this is very
common) then one has to launch another volley of
jobs to complete the grid (and yet a third one if any
of these fails, etc.), thus multiplying the overall com-
puting time.
Typically, a single grid search is not enough and
practitioners tend to proceed with a sequence of grid
searches, each time adjusting the ranges of values
considered based on the previous results obtained.
Although this can be done manually, this procedure
can also be automated by considering the idea of
multi-resolution search to guide this outer loop. Dif-
ferent more local grid searches can be launched in
the neighborhood of the best solutions found previ-
ously. In addition, the idea of coordinate descent can
also be thrown in, by making each grid search focus
on only a few of the hyper-parameters. For exam-
ple, it is common practice to start by exploring the
initial learning rate while keeping ﬁxed (and initially
constant) the learning rate descent schedule. Once
the shape of the schedule has been chosen, it may be
possible to further reﬁne the learning rate, but in a
smaller interval around the best value found.
Humans can get very good at performing hyper-
parameter search, and having a human in the loop
also has the advantage that it can help detect bugs
or unwanted or unexpected behavior of a learning
algorithm. However, for the sake of reproducibil-
ity, machine learning researchers should strive to use
procedures that do not involve human decisions in
the middle, only at the outset (e.g., setting hyper-
parameter ranges, which can be speciﬁed in a paper
describing the experiments).
3.3.4 Layer-wise optimization of hyper-
parameters
In the case of Deep Learning with unsupervised
pre-training there is an opportunity for combin-
ing coordinate descent and cheap relative valida-
tion set performance evaluation associated with
some hyper-parameter choices. The idea, described
by Mesnil et al. (2011); Bengio (2011), is to perform
greedy choices for the hyper-parameters associated
with lower layers (near the input) before training the
higher layers. One ﬁrst trains (unsupervised) the
ﬁrst layer with diﬀerent hyper-parameter values and
somehow estimates the relative validation error that
would be obtained from these diﬀerent conﬁgurations
if the ﬁnal network only had this single layer as in-
ternal representation. In the common case where the
ultimate task is supervised, it means training a sim-
ple supervised predictor (e.g. a linear classiﬁer) on
top of the learned representation. In the case of linear
predictor (e.g. regression or logistic regression) this
can even be done on the ﬂy while unsupervised train-
ing of the representation progresses (i.e. can be used
17
for early stopping as well), as in (Larochelle et al. ,
2009). Once a set of apparently good (according
to this greedy evaluation) hyper-parameters values
has been found (or possibly using only the best one
found), these good values can be used as starting
point to train (and hyper-optimize) a second layer in
the same way, etc. The completely greedy approach is
to keep only the best conﬁguration up to now (for the
lower layers), but keeping the K best conﬁgurations
overall only multiplies computational costs of hyper-
parameter selection by K for layers beyond the ﬁrst
one. Since greedy layer-wise pre-training does not
modify the lower layers when pre-training the upper
layers, this is also very eﬃcient computationally. This
procedure allows one to set the hyper-parameters as-
sociated with the unsupervised pre-training stage,
and then there remains hyper-parameters to be se-
lected for the supervised ﬁne-tuning stage, if one
is desired. A ﬁnal supervised ﬁne-tuning stage is
strongly suggested, especially when there are many
labeled examples (Lamblin and Bengio, 2010).
3.4 Random Sampling of Hyper-
Parameters
A serious problem with the grid search approach
to ﬁnd good hyper-parameter conﬁgurations is that
it scales exponentially with the number of hyper-
parameters considered. In the above sections we have
discussed numerous hyper-parameters and if all of
them were to be explored at the same time it would
be impossible to use only a grid search to do so.
One may think that there are no other options sim-
ply because this is an instance of the curse of di-
mensionality. But like we have found in our work
on Deep Learning (Bengio, 2009), if there is some
structure in a target function we are trying to dis-
cover, then there is a chance to ﬁnd good solutions
without paying an exponential price. It turns out
that in many practical cases we have encountered,
there is a kind of structure that random sampling
can exploit (Bergstra and Bengio, 2012). The idea
of random sampling is to replace the regular grid
by a random (typically uniform) sampling. Each
tested hyper-parameter conﬁguration is selected by
independently sampling each hyper-parameter from
a prior distribution (typically uniform in the log-
domain, inside the interval of interest). For a dis-
crete hyper-parameter, a multinomial distribution
can be deﬁned according to our prior beliefs on
the likely good values (at worse a uniform distri-
bution across the allowed values). In fact, we can
use our prior knowledge to make this prior distri-
bution quite sophisticated. For example, we can
readily include knowledge that some values of some
hyper-parameters only make sense in the context of
other particular values of hyper-parameters. This is
a practical consideration for example when consider-
ing layer-speciﬁc hyper-parameters when the number
of layers itself is a hyper-parameter.
The experiments performed (Bergstra and Bengio,
2012) show that random sampling can be many times
more eﬃcient than grid search as soon as the number
of hyper-parameters goes beyond the 2 or 3 typically
seen with SVMs and vanilla neural networks. The
main reason why faster convergence is observed is
because it allows one to explore more values for each
hyper-parameter, whereas in grid search, the same
value of a hyper-parameter is repeated in exponen-
tially many conﬁgurations (of all the other hyper-
parameters). In particular, if only a small subset of
the hyper-parameters really matters, then this proce-
dure can be shown to be exponentially more eﬃcient.
What we found is that for diﬀerent datasets and ar-
chitectures, the subset of hyper-parameters that mat-
tered most was diﬀerent, but if was often the case
that only few hyper-parameters made a big diﬀer-
ence (and the learning rate is always one of them!).
When marginalizing (by averaging or minimizing) the
validation performance to visualize the eﬀect of one
or two hyper-parameters, we get a more noisy picture
(because of the random variations of the other hyper-
parameters) but one with much more resolution (be-
cause so many more diﬀerent values are considered).
Practically, one can plot the curves of best validation
error as the number of random trials 25 is increased
(with mean and standard deviation, obtained by con-
sidering, for each choice of number of trials, all possi-
ble same-size subsets of trials), and these curves tell
25 each random trial corresponding to a training job with a
particular choice of hyper-parameter values
18
us that we are approaching a plateau, i.e., that tells
us whether it is worth it or not to continue launching
jobs, i.e., we can perform a kind of early stopping in
the outer optimization over hyper-parameters. Note
that one should distinguish the curve of the “best
trial in ﬁrst N trials” with the curve of the mean (and
standard deviation) of the “best in a subset of size
N”. The latter is a better statistical representative of
the improvements we should expect if we increase the
number of trials. Even if the former has a plateau,
the latter may still be on the increase, pointing for the
need to more hyper-parameter conﬁguration samples
(more trials) (Bergstra and Bengio, 2012). Compar-
ing these curves with the equivalent obtained from
grid search we see faster convergence with random
search. On the other hand, note that one advan-
tage of grid search compared to random sampling is
that the qualitative analysis of results is easier be-
cause one can consider variations of a single hyper-
parameter with all the other hyper-parameters being
ﬁxed. It may remain a valid option to do a small
grid search around the best solutions found by ran-
dom search, considering only the hyper-parameters
that were found to matter or which concern a scien-
tiﬁc question of interest 26.
Random search maintains the advantage of easy
parallelization provided by grid search and improves
on it. Indeed, a practical advantage of random search
compared to grid search is that if one of the jobs
fails (due to hardware or software problems, a com-
mon issue on large clusters) then there is no need
to re-launch that job. It also means that if one has
launched 100 random search jobs, and ﬁnds that the
convergence curves still has an interesting slope, one
can launch another 50 or 100 without wasting the
ﬁrst 100. It is not that simple to combine the results
of two grid searches because they are not always com-
patible (i.e., one is not a subset of the other).
Finally, although random search is a useful ad-
dition to the toolbox of the practitioner, semi-
automatic exploration is still helpful and one will
often iterate between launching a new volley of
26 This is often the case in machine learning research, e.g.,
does depth of architecture matter? then we need to control ac -
curately for the eﬀect of depth, with all other hyper-parame ters
optimized for each value of depth.
jobs and analysis of the results obtained with
the previous volley in order to guide model de-
sign and research. What we need is more, and
more eﬃcient, automation of hyper-parameter op-
timization. There are some interesting steps in
this direction (Hutter, 2009; Bergstra et al. , 2011;
Hutter et al. , 2011; Srinivasan and Ramakrishnan,
2011) but much more needs to done.
4 Debugging and Analysis
4.1 Gradient Checking and Con-
trolled Overﬁtting
A very useful debugging step consists in verifying
that the implementation of the gradient ∂L
∂θ is com-
patible with the computation of L as a function of
θ. If the analytically computed gradient does not
match the one obtained by a ﬁnite diﬀerence approx-
imation, this signals that a bug is probably present
somewhere. First of all, looking at for which i one
gets important relative change between ∂L
∂θ i
and its
ﬁnite diﬀerence approximation, we can get hints as
to where the problem may be. An error in sign is
particularly troubling, of course. A good next step is
then to verify in the same way intermediate gradients
∂L
∂a with a some quantities that depend on the faulty
θ, such as intervening neuron activations.
As many researchers know, the gradient can be
approximated by a ﬁnite diﬀerence approximation
obtained from the ﬁrst-order Taylor expansion of a
scalar function f with respect to a scalar argument
x:
∂f (x)
∂x ≈ f(x + ε) −f(x)
ε
But a less known fact is that a second order approx-
imation can be achieved by considering the following
alternative formula:
∂f (x)
∂x ≈ f(x + ε) −f(x − ε)
2ε .
The second order terms of the Taylor expansion of
f(x + ε) and f(x −ε) cancel each other because they
are even, i.e., this formula is twice more expensive
but provides quadratically more precision.
19
Note that because of ﬁnite precision computation,
there will be a diﬀerence between the analytic (even
correct) and ﬁnite diﬀerence gradient. Contrary to
naive expectations, the relative diﬀerence may grow
if we choose an ε that is too small, i.e., the error
should ﬁrst decrease as ε is decreased, and then may
worsen when numerical precision kicks in, due to non-
linearities. We have often used a value of ε = 10 −4
in neural networks, a value that is suﬃciently small
to detect most bugs.
Once the gradient is known to be well computed,
another sanity check is that gradient descent (or any
other gradient-based optimization) should be able
to overﬁt on a small training set 27. In particu-
lar, to factor out eﬀects of SGD hyper-parameters, a
good sanity check for the code (and the other hyper-
parameters) is to verify that one can overﬁt on a small
training set using a powerful second order method
such as L-BFGS. For any optimizer, though, as the
number of examples is increased, the degradation of
training error should be gradual while validation er-
ror should improve. And one typically sees the advan-
tages of SGD over batch second-order methods like
L-BFGS increase as the training set size increases.
The break-even point may depend on the task, paral-
lelization (multi-core or GPU, see Sec.5 below), and
architecture (number of computations compared to
number of parameters, per example).
Of course, the real goal of learning is to achieve
good generalization error, and the latter can be es-
timated by measuring performance on an indepen-
dent test set. When test error is considered too high,
the ﬁrst question to ask is whether it is because of a
diﬃculty in optimizing the training criterion or be-
cause of overﬁtting. Comparing training error and
test error (and how they change as we change hyper-
parameters that inﬂuence capacity, such as the num-
ber of training iterations) helps to answer that ques-
tion. Depending on the answer, of course, the ap-
27 In principle, bad local minima could prevent that, but in
the overﬁtting regime, e.g., with more hidden units than exa m-
ples, the global minimum of the training error can generally be
reached almost surely from random initialization, presuma bly
because the training criterion becomes convex in the parame -
ters that suﬃce to get the training error to zero (Bengio et al.,
2006a), i.e., the output weights of the neural network.
propriate ways to improve test error are diﬀerent.
Optimization diﬃculties can be ﬁxed by looking for
bugs in the training code, inappropriate values of op-
timization hyper-parameters, or simply in insuﬃcient
capacity (e.g. not enough degrees of freedom, hidden
units, embedding sizes, etc.). Overﬁtting diﬃculties
can be addressed by collecting more training data,
introducing more or better regularization terms, or
considering diﬀerent function families (or neural net-
work architectures). In a multi-layer neural network,
both problems can be simultaneously present. For
example, as discussed in Bengio et al. (2007); Bengio
(2009), it is possible to have zero training error with
a large top-level hidden layer that allows the output
layer to overﬁt, while the lower layer are not doing
a good job of extracting useful features because they
were not properly optimized.
Unless using a framework such as Theano which
automatically handles the eﬃcient allocation of
buﬀers for intermediate results, it is important to pay
attention such buﬀers in the design of the code. The
ﬁrst objective is to avoid memory allocation in the
middle of the training loop, i.e., all memory buﬀers
should be allocated once and for all. Careless reuse
of the same memory buﬀers for diﬀerent uses can
however lead to bugs, which can be checked, in the
debugging phase, by initializing buﬀers to the NaN
(Not-A-Number) value, which propagates into down-
stream computation (making it easy to detect that
uninitialized values were used) 28.
4.2 Visualizations and Statistics
The most basic statistics that should be measured
during training are error statistics. The average loss
on the training set and the validation set and their
evolution during training are very useful to monitor
progress and diﬀerentiate overﬁtting from poor op-
timization. To make comparisons easier, it may be
useful to compare neural networks during training in
terms of their “age” (number of updates made times
mini-batch size B, i.e., number of examples visited)
rather than in terms of number of epochs (which is
very sensitive to the training set size).
28 Personal communication from David Warde-Farley, who
learned this trick from Sam Roweis.
20
When using unsupervised training to learn the ﬁrst
few layers of a deep architecture, a very common de-
bugging and analysis tool is the visualization of ﬁl-
ters, i.e., of the weight vectors associated with in-
dividual hidden units. This is simplest in the case
of the ﬁrst layer and where the inputs are images
(or image patches), time-series, or spectrograms (all
of which are visually interpretable). Several recipes
have been proposed to extend this idea to visualize
the preferred input of hidden units in layers that
follow the ﬁrst one (Lee et al. , 2008; Erhan et al. ,
2010a). In the case of the ﬁrst layer, since one of-
ten obtains Gabor ﬁlters, a parametric ﬁt of these
ﬁlters to the weight vector can be done so as to vi-
sualize the distribution of orientations, position and
scale of the learned ﬁlters. An interesting special
case of visualizing ﬁrst-layer weights is the visual-
ization of word embeddings (see Section 5.3 below)
using a dimensionality reduction technique such as
t-SNE (van der Maaten and Hinton, 2008).
In the case where the inputs are not images or eas-
ily visualizable, or to get a sense of the weight values
in diﬀerent hidden units, Hinton diagrams (Hinton,
1989) are also very useful, using small squares whose
color (black or white) indicates a weight’s sign and
whose area represents its magnitude.
Another way to visualize what has been learned
by an unsupervised (or joint label-input) model is
to look at samples from the model. Sampling pro-
cedures have been deﬁned at the outset for RBMs,
Deep Belief Nets, and Deep Boltzmann Machines,
for example based on Gibbs sampling. When weights
become larger, mixing between modes can become
very slow with Gibbs sampling. An interesting alter-
native is rates-FPCD (Tieleman and Hinton, 2009;
Breuleux et al. , 2011) which appears to be more ro-
bust to this problem and generally mixes faster, but
at the cost of losing theoretical guarantees.
In the case of auto-encoder variants, it was not
clear until recently whether they were really captur-
ing the underlying density (since they are not opti-
mized with respect to the maximum likelihood prin-
ciple or an approximation of it). It was therefore
even less clear if there existed appropriate sampling
algorithms for auto-encoders, but a recent proposal
for sampling from contractive auto-encoders appears
to be working very well (Rifai et al. , 2012), based on
arguments about the geometric interpretation of the
ﬁrst derivative of the encoder.
To get a sense of what individual hidden units rep-
resent, it has also been proposed to vary only one
unit while keeping the others ﬁxed, e.g., to the value
obtained by ﬁnding the hidden units representation
associated with a particular input example.
Another interesting technique is the visual-
ization of the learning trajectory in function
space (Erhan et al. , 2010b). The idea is to asso-
ciate the function (as opposed to simply the pa-
rameters) computed by a neural network with a
low-dimensional (2-D or 3-D) representation, e.g.,
with the t-SNE (van der Maaten and Hinton, 2008)
or Isomap (Tenenbaum et al. , 2000) algorithms, and
then plot the evolution of this function during train-
ing, or the population of such trajectories for diﬀerent
initializations. This provides visualization of eﬀec-
tive local minima 29 and shows that no two diﬀerent
random initializations ended up in the same eﬀective
local minimum.
Finally, another useful type of visualization is to
display statistics (e.g., histogram, mean and stan-
dard deviation) of activations (inputs and outputs
of the non-linearities at each layer), activation gradi-
ents, parameters and parameter gradients, by groups
(e.g. diﬀerent layers, biases vs weights) and across
training iterations. See Glorot and Bengio (2010) for
a practical example.
5 Other Recommendations
5.1 Multi-core machines, BLAS and
GPUs
Matrix operations are the most time-consuming in
eﬃcient implementations of many machine learning
algorithms and this is particularly true of neural net-
works and deep architectures. The basic operations
are matrix-vector products (forward propagation and
back-propagation) and vector times vector outer
29 It is diﬃcult to know for sure if it is a true local minima
or if it appears like one because the optimization algorithm is
stuck.
21
products (resulting in a matrix of weights for the
weight gradient computation). Matrix-matrix mul-
tiplications can be done substantially faster than the
equivalent sequence of matrix-vector products for two
reasons: by smart caching mechanisms such as imple-
mented in the BLAS library (which is called from
many higher-level environments such as python’s
numpy and Theano, Matlab, Torch or Lush), and
thanks to parallelism. Appropriate versions of BLAS
can take advantage of multi-core machines to dis-
tribute these computations on multi-core machines.
The speed-up is however generally a fraction of the
total speedup one can hope (e.g. 3 × to 4 × on a 4-
core machine), because of communication overheads
and because not all computation is parallelized. Par-
allelism becomes more eﬃcient when the sizes of
these matrices is increased, which is why mini-batch
updates can be computationally advantageous, and
more so when more cores are present.
The extreme multi-core machines are the GPUs
(Graphics Processing Units), with hundreds of cores.
Unfortunately, they also come with constraints and
specialized compilers which make it more diﬃcult to
fully take advantage of their potential. On 512-core
machines, we are routinely able to get speed-ups of
10× to 40 × for large neural networks. To make the
use of GPUs practical, it really helps to use existing
libraries that eﬃciently implement computations on
GPUs. See Bergstra et al. (2010) for a comparative
study of the Theano library (which compiles numpy-
like code for GPUs). One practical issue is that only
the GPU-compiled operations will typically be done
on the GPU, and that transfers between the GPU
and CPU considerably slow things down. It is im-
portant to use a proﬁler to ﬁnd out what is done
on the GPU and how eﬃcient these operations are
in order to quickly invest one’s time where needed
to make an implementation GPU-eﬃcient and keep
most operations on the GPU card.
5.2 Sparse High-Dimensional Inputs
Sparse high-dimensional inputs can be eﬃciently han-
dled by traditional supervised neural networks by us-
ing a sparse matrix multiplication. Typically, the in-
put is a sparse vector while the weights are in a dense
matrix, and one should use an eﬃcient implementa-
tion made for just this case in order to optimally take
advantage of sparsity. There is still going to be an
overhead on the order of 2 × or more (on the multiply-
add operations, not the others) compared to a dense
implementation of the matrix-vector product.
For many unsupervised learning algorithms there is
unfortunately a diﬃculty. The computation for these
learning algorithms usually involves some kind of re-
construction of the input (like for all auto-encoder
variants, but also for RBMs and sparse coding vari-
ants), as if the inputs were in the output space of
the learner. Two exceptions to this problem are
semi-supervised embedding (Weston et al. , 2008) and
Slow Feature Analysis (Wiskott and Sejnowski, 2002;
Berkes and Wiskott, 2002). The former pulls the rep-
resentation of nearby examples near each other while
also tuning the representation for a supervised learn-
ing task, and the latter maximizes the learned fea-
tures’ variance while minimizing their covariance and
maximizing their temporal auto-correlation.
For algorithms that do need a form of input re-
construction, an eﬃcient approach based on sam-
pled reconstruction (Dauphin et al. , 2011) has been
proposed, successfully implemented and evaluated
for the case of auto-encoders and denoising auto-
encoders. The ﬁrst idea is that on each example (or
mini-batch), one samples a subset of the elements of
the reconstruction vector, along with the associated
reconstruction loss. Only the loss for these sampled
elements (and the associated back-propagation oper-
ations into hidden units and reconstruction weights)
needs to be computed. That alone would reduce the
computational cost but make the gradient much more
noisy and possibly biased as well, if the sampling dis-
tribution was chosen not uniform. To reduce the
variance of that estimator, the idea is to guess for
which features the reconstruction loss will be larger
and to sample with higher probability these features
(and their loss). In particular, the authors always
sample the features associated with a non-zero in the
input (or the corrupted input, in the denoising case),
and uniformly sample an equal number of zeros. To
make the estimator unbiased now requires introduc-
ing a weight on the reconstruction loss associated
with each sampled feature, inversely proportional to
22
the probability of sampling it, i.e., this is an impor-
tance sampling scheme. The experiments show that
the speed-up increases linearly with the amount of
sparsity while the average loss is optimized as well as
in the deterministic full-computation case.
5.3 Symbolic Variables, Embeddings,
Multi-Task Learning and Multi-
Relational Learning
Parameter sharing (Lang and Hinton, 1988; LeCun,
1989; Lang and Hinton, 1988; Caruana, 1993; Baxter,
1995, 1997) is an old neural network technique for in-
creasing statistical power: if a parameter is used in N
times more contexts (diﬀerent tasks, diﬀerent parts of
the input, etc.) then it may be as if we had N times
more training examples for tuning its value. More
examples to estimate a parameter reduces its vari-
ance (with respect to sampling of training examples),
which is directly inﬂuencing generalization error: for
example the generalization mean squared error can
be decomposed as the sum of a bias term and a vari-
ance term (Geman et al. , 1992). The reuse idea was
ﬁrst exploited by applying the same parameter to dif-
ferent parts of the input, as in convolutional neu-
ral networks (Lang and Hinton, 1988; LeCun, 1989).
Reuse was also exploited by sharing the lower lay-
ers of a network (and the representation of the input
that they capture) across multiple tasks associated
with diﬀerent outputs of the network (Caruana, 1993;
Baxter, 1995, 1997). This idea is also one of the key
motivations behind Deep Learning (Bengio, 2009) be-
cause one can think of the intermediate features com-
puted in higher (deeper) layers as diﬀerent tasks that
can share the sub-features computed in lower layers
(nearer the input). This very basic notion of reuse
is key to improving generalization in many settings,
guiding the design of neural network architectures in
practical applications as well.
An interesting special case of these ideas is in the
context of learning with symbolic data. If some in-
put variables are symbolic, taking value in a ﬁnite
alphabet, they can be represented as neural network
inputs by a one-hot subvector (with a 0 everywhere
except at the position associated with the particular
symbol) of the input vector. Now, sometimes diﬀer-
ent input variables refer to diﬀerent instances of the
same type of symbol. A patent example is with neural
language models (Bengio et al. , 2003; Bengio, 2008),
where the input is a sequence of words. In these
models, the same input layer weights are reused for
words at diﬀerent positions in the input sequence (as
in convolutional networks). The product of a one-hot
sub-vector with this shared weight matrix is a gener-
ally dense vector, and this associates each symbol in
the alphabet with a point in a vector space (the result
of the matrix multiplication, which equals one of the
columns of the matrix), which we call its embedding.
The idea of vector space representations for words
and symbols is older (Deerwester et al. , 1990) and is
a particular case of the notion of distributed repre-
sentation (Hinton, 1986, 1989) central to the connec-
tionist approaches. Learned embeddings of symbols
(or other objects) can be conveniently visualized us-
ing a dimensionality reduction algorithm such as t-
SNE (van der Maaten and Hinton, 2008).
In addition to sharing the embedding parame-
ters across positions of words in an input sentence,
Collobert et al. (2011a) share them across natural
language processing tasks such as Part-Of-Speech
tagging, chunking and semantic role labeling. Param-
eter sharing is a key idea behind convolutional nets,
recurrent neural networks and dynamic Bayes nets, in
which the same parameters are used for diﬀerent tem-
poral or spatial slices of the data. This idea has been
generalized from sequences and 2-D images to arbi-
trary graphs with recursive neural networks or recur-
sive graphical models (Pollack, 1990; Frasconi et al. ,
1998; Bottou, 2011; Socher et al. , 2011), Markov
Logic Networks (Richardson and Domingos, 2006)
and relational learning (Getoor and Taskar, 2006). A
relational database can be seen as a set of objects (or
values) and relations between them, of the form (ob-
ject1, relation-type, object2). The same global set of
parameters can be shared to characterize such rela-
tions, across relations (which can be seen as tasks)
and objects. Object-speciﬁc parameters are the pa-
rameters specifying the embedding of a discrete ob-
ject. One can think of the elements of each embed-
ding vector as implicit learned attributes . Diﬀerent
tasks may demand diﬀerent attributes, so that ob-
23
jects which share some underlying characteristics and
behavior should end up having similar values of some
of their attributes. For example, words appearing in
semantically and syntactically similar contexts end
up getting a very close embedding (Collobert et al. ,
2011a). If the same attributes can be useful for sev-
eral tasks, then statistical power is gained through
parameter sharing, and transfer of information be-
tween tasks can happen, making the data of some
task informative for generalizing properly on another
task.
The idea proposed in Bordes et al. (2011, 2012) is
to learn an energy function that is lower for posi-
tive (valid) relations present in the training set, and
parametrized in two parts: on the one hand the sym-
bol embeddings and on the other hand the rest of
the neural network that maps them to a scalar en-
ergy. In addition, by considering relation types them-
selves as particular symbolic objects, the model can
reason about relations themselves and have relations
between relation types. For example, “to be” can act
as a relation type (in subject-attribute relations) but
in the statement “’To be’ is a verb” it appears both
as a relation type and as an object of the relation.
Such multi-relational learning opens the door to
the application of neural networks outside of their
traditional applications, which was based on a single
homogeneous source of data, often seen as a matrix
with one row per example and one column (or group
of columns) per random variable. Instead, one of-
ten has multiple heterogeneous sources of data (typ-
ically providing examples seen as a tuple of values),
each involving diﬀerent random variables. So long
as these diﬀerent sources share some variables , then
these multi-relational multi-task learning approaches
can be applied. Each variable can be associated with
its embedding function (that maps the value of a vari-
able to a generic representation space that is valid
across tasks and data sources ). This framework can
be applied not only to symbolic data but to mixed
symbolic/numeric data if the mapping from object
to embedding is generalized from a table look-up to
a parametrized function (the simplest being a linear
mapping) from its raw attributes (e.g., image fea-
tures) to its embedding. This has been exploited
successfully to design image search systems in which
images and queries are mapped to the same semantic
space (Weston et al. , 2011).
6 Open Questions
6.1 On the Added Diﬃculty of Train-
ing Deeper Architectures
There are experimental results which provide some
evidence that, at least in some circumstances, deeper
neural networks are more diﬃcult to train than
shallow ones, in the sense that there is a greater
chance of missing out on better minima when start-
ing from random initialization. This is borne out
by all the experiments where we ﬁnd that initializ-
ing global training of a deep architecture can dras-
tically improve performance. In the Deep Learn-
ing literature this has been shown with the use of
unsupervised pre-training, both applied to super-
vised tasks — training a neural network for clas-
siﬁcation (Hinton et al. , 2006; Bengio et al. , 2007;
Ranzato et al. , 2007) — and unsupervised tasks —
training a Deep Boltzmann Machine to model the
data distribution (Salakhutdinov and Hinton, 2009).
The learning trajectories visualizations
of Erhan et al. (2010b) have shown that even
when starting from nearby conﬁgurations in function
space, diﬀerent initializations seem to always fall in
a diﬀerent eﬀective local minimum. Furthermore,
the same study showed that the minima found when
using unsupervised pre-training were far in function
space from those found from random initialization,
in addition to giving better generalization error.
Both of these ﬁndings highlight the importance of
initialization, hence of local minima eﬀects, in deep
networks. Finally, it has been shown that these
eﬀects were both increased when considering deeper
architectures (Erhan et al. , 2010b).
There are also results showing that speciﬁc ways
of setting the initial distribution and ordering of
examples (“curriculum learning”) can yield bet-
ter solutions (Elman, 1993; Bengio et al. , 2009;
Krueger and Dayan, 2009). This also suggest that
very particular ways of initializing parameters, very
diﬀerent from uniformly sampled, can have a strong
24
impact on the solutions found by gradient descent.
The hypothesis proposed in (Bengio et al. , 2009) is
that curriculum learning can act similarly to a con-
tinuation method , i.e., starting from an easier opti-
mization task (e.g. convex) and tracking the local
minimum as the learning task is gradually made more
diﬃcult and closer to the real task of interest.
Why would training deeper networks be more dif-
ﬁcult? This is clearly still an open question. A
plausible partial answer is that deeper networks are
also more non-linear (since each layer composes more
non-linearity on top of the previous ones), making
gradient-based methods less eﬃcient. It may also be
that the number and structure of local minima both
change qualitatively as we increase depth. Theoreti-
cal arguments support a potentially exponential gain
in expressive power of deeper architectures (Bengio,
2009; Bengio and Delalleau, 2011) and it would be
plausible that with this added expressive power com-
ing from the combinatorics of composed reuse of sub-
functions could come a corresponding increase in the
number (and possibly quality) of local minima. But
the best ones could then also be more diﬃcult to ﬁnd.
On the practical side, several experimental results
point to factors that may help training deep architec-
tures:
•A local training signal. What many success-
ful procedures for training deep networks have
in common is that they involve a local training
signal that helps each layer decide what to do
without requiring the back-propagation of gradi-
ents through many non-linearities. This includes
of course the many variants of greedy layer-wise
pre-training but also the less well-known semi-
supervised embedding algorithm (Weston et al. ,
2008).
•Initialization in the right range. Based
on the idea that both activations and gradients
should be able to ﬂow well through a deep ar-
chitecture without signiﬁcant reduction in vari-
ance, Glorot and Bengio (2010) proposed set-
ting up the initial weights to make the Jaco-
bian of each layer have singular values near 1 (or
preserve variance in both directions). In their
experiments this clearly helped greatly reduc-
ing the gap between purely supervised and pre-
trained (with layer-wise unsupervised learning)
deep networks.
•Choice of non-linearities . In the same
study (Glorot and Bengio, 2010) and a follow-
up (Glorot et al. , 2011a) it was shown that the
choice of hidden layer non-linearities interacted
with depth. In particular, without unsupervised
pre-training, a deep neural network with sig-
moids in the top hidden layer would get stuck
for a long time on a plateau and generally pro-
duce inferior results, due to the special role of
0 and of the initial gradients from the output
units. Symmetric non-linearities like the hy-
perbolic tangent did not suﬀer from that prob-
lem, while softer non-linearities (without ex-
ponential tails) such as the softsign function
s(a) = a
1+|a| worked even better. In Glorot et al.
(2011a) it was shown that an asymmetric but
hard-limiting non-linearity such as the rectiﬁer
(s(a) = max(0 , a), see also (Nair and Hinton,
2010)) actually worked very well (but should not
be used for output units), in spite of the prior be-
lief that the fact that when hidden units are sat-
urated gradients would not ﬂow well into lower
layers. In fact gradients ﬂow very well, but on
selected paths. A recent heuristic that is related
to the diﬃculty of gradient propagation through
neural net non-linearities is the idea of “cen-
tering” the non-linear operation such that each
hidden unit has zero average output and zero
average slope (Schraudolph, 1998; Raiko et al. ,
2012).
6.2 Adaptive Learning Rates and
Second-Order Methods
To improve convergence and remove learning rates
from the list of hyper-parameters, many authors have
advocated exploring adaptive learning rate methods,
either for a global learning rate (Cho et al. , 2011),
a layer-wise learning rate, a neuron-wise learning
rate, or a parameter-wise learning rate (Bordes et al. ,
2009) (which then starts to look like a diagonal New-
25
ton method). LeCun (1987); LeCun et al. (1998a)
advocate the use of a second-order diagonal New-
ton (always positive) approximation, with one learn-
ing rate (associated with the approximated inverse
second derivative of the loss with respect to the
parameter). Hinton (2010) proposes scaling learn-
ing rates so that the average weight update is on
the order of 1/1000th of the weight magnitude.
LeCun et al. (1998a) also propose a simple power
method in order to estimate the largest eigenvalue
of the Hessian (which would be the optimal learn-
ing rate). An interesting alternative to variants of
Newton’s method are variants of the natural gradi-
ent method (Amari, 1998), but like the basic Newton
method it is computationally too expensive, requir-
ing operations on a too large square matrix (num-
ber of parameters by number of parameters). Diag-
onal and low-rank online approximations of natural
gradient (Le Roux et al. , 2008; Le Roux et al. , 2011)
have been proposed and shown to speed-up training
in some contexts.
Whereas stochastic gradient descent converges
very quickly initially it is generally slower than
second-order methods for the ﬁnal convergence, and
this may be important in some applications. As a
consequence, batch training algorithms (performing
only one update after seeing the whole training set)
such as the Conjugate Gradients method (a second
order method) have dominated stochastic gradient
descent for not too large datasets (e.g. less than
thousands or tens of thousands of examples). Fur-
thermore, it has recently been proposed and success-
fully applied to use second-order methods over large
mini-batches (Le et al. , 2011; Martens, 2010). The
idea is to do just a few iterations of the second-order
methods on each mini-batch and then move on to
the next mini-batch, starting from the best previous
point found.
At this point in time however, although the second-
order and natural gradient methods are appealing
conceptually, have demonstrably helped in the stud-
ied cases and may in the end prove to be very impor-
tant, they have not yet become a standard for neural
networks optimization and need to be validated and
maybe improved by other researchers, before displac-
ing simple (mini-batch) stochastic gradient descent
variants.
6.3 Conclusion
In spite of decades of experimental and theoretical
work on artiﬁcial neural networks, and with all the
impressive progress made since the ﬁrst edition of
this book, in particular in the area of Deep Learning,
there is still much to be done to better train neural
networks and better understand the underlying issues
that can make the training task diﬃcult. As stated in
the introduction, the wisdom distilled here should be
taken as a guideline, to be tried and challenged, not
as a practice set in stone. The practice summarized
here, coupled with the increase in available comput-
ing power, now allows researchers to train neural net-
works on a scale that is far beyond what was possible
at the time of the ﬁrst edition of this book, helping
to move us closer to artiﬁcial intelligence.
Acknowledgements
The author is grateful for the comments and feed-
back provided by Nicolas Le Roux, Ian Goodfel-
low, James Bergstra, Guillaume Desjardins, Razvan
Pascanu, David Warde-Farley, Eric Larsen, Frederic
Bastien, and Sina Honari, as well as for the ﬁnan-
cial support of NSERC, FQRNT, CIFAR, and the
Canada Research Chairs.
References
Amari, S. (1998). Natural gradient works eﬃciently
in learning. Neural Computation , 10(2), 251–276.
Bach, F. and Moulines, E. (2011). Non-asymptotic
analysis of stochastic approximation algorithms. In
NIPS’2011.
Bagnell, J. A. and Bradley, D. M. (2009). Diﬀeren-
tiable sparse coding. In NIPS’2009, pages 113–120.
Baxter, J. (1995). Learning internal representations.
In COLT’95, pages 311–320.
Baxter, J. (1997). A Bayesian/information theoretic
model of learning via multiple task sampling. Ma-
chine Learning , 28, 7–40.
26
Bengio, Y. (2008). Neural net language models.
Scholarpedia, 3(1), 3881.
Bengio, Y. (2009). Learning deep architectures for
AI . Now Publishers.
Bengio, Y. (2011). Deep learning of representations
for unsupervised and transfer learning. In JMLR
W&CP: Proc. Unsupervised and Transfer Learn-
ing.
Bengio, Y. and Delalleau, O. (2011). On the expres-
sive power of deep architectures. In ALT’2011.
Bengio, Y. and LeCun, Y. (2007). Scaling learning
algorithms towards AI. In Large Scale Kernel Ma-
chines.
Bengio, Y., Ducharme, R., Vincent, P., and Jauvin,
C. (2003). A neural probabilistic language model.
JMLR, 3, 1137–1155.
Bengio, Y., Le Roux, N., Vincent, P., Delalleau, O.,
and Marcotte, P. (2006a). Convex neural networks.
In NIPS’2005, pages 123–130.
Bengio, Y., Delalleau, O., and Le Roux, N. (2006b).
The curse of highly variable functions for local ker-
nel machines. In NIPS’2005, pages 107–114.
Bengio, Y., Lamblin, P., Popovici, D., and
Larochelle, H. (2007). Greedy layer-wise training
of deep networks. In NIPS’2006.
Bengio, Y., Louradour, J., Collobert, R., and We-
ston, J. (2009). Curriculum learning. In ICML’09.
Bergstra, J. and Bengio, Y. (2012). Random search
for hyper-parameter optimization. J. Machine
Learning Res., 13, 281–305.
Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P.,
Pascanu, R., Desjardins, G., Turian, J., Warde-
Farley, D., and Bengio, Y. (2010). Theano: a
CPU and GPU math expression compiler. In Proc.
Python for Scientiﬁc Comp. Conf. (SciPy) .
Bergstra, J., Bardenet, R., Bengio, Y., and K´ egl, B.
(2011). Algorithms for hyper-parameter optimiza-
tion. In NIPS’2011.
Berkes, P. and Wiskott, L. (2002). Applying slow fea-
ture analysis to image sequences yields a rich reper-
toire of complex cell properties. In ICANN’02,
pages 81–86.
Bertsekas, D. P. (2010). Incremental gradient, sub-
gradient, and proximal methods for convex opti-
mization: a survey. Technical Report 2848, LIDS.
Bordes, A., Bottou, L., and Gallinari, P. (2009). Sgd-
qn: Careful quasi-newton stochastic gradient de-
scent. Journal of Machine Learning Research , 10,
1737–1754.
Bordes, A., Weston, J., Collobert, R., and Bengio, Y.
(2011). Learning structured embeddings of knowl-
edge bases. In AAAI 2011 .
Bordes, A., Glorot, X., Weston, J., and Bengio, Y.
(2012). Joint learning of words and meaning rep-
resentations for open-text semantic parsing. AIS-
TATS’2012.
Bottou, L. (2011). From machine learning to machine
reasoning. Technical report, arXiv.1102.1808.
Bottou, L. and Bousquet, O. (2008). The tradeoﬀs of
large scale learning. In NIPS’2008.
Bottou, L. and LeCun, Y. (2004). Large-scale on-line
learning. In NIPS’2003.
Breiman, L. (1994). Bagging predictors. Machine
Learning, 24(2), 123–140.
Breuleux, O., Bengio, Y., and Vincent, P. (2011).
Quickly generating representative samples from an
rbm-derived process. Neural Computation , 23(8),
2053–2073.
Caruana, R. (1993). Multitask connectionist learn-
ing. In Proceedings of the 1993 Connectionist Mod-
els Summer School , pages 372–379.
Cho, K., Raiko, T., and Ilin, A. (2011). Enhanced
gradient and adaptive learning rate for training
restricted boltzmann machines. In ICML’2011,
pages 105–112.
27
Coates, A. and Ng, A. Y. (2011). The importance
of encoding versus training with sparse coding and
vector quantization. In ICML’2011.
Collobert, R. and Bengio, S. (2004). Links between
perceptrons, MLPs and SVMs. In ICML’2004.
Collobert, R., Weston, J., Bottou, L., Karlen, M.,
Kavukcuoglu, K., and Kuksa, P. (2011a). Natural
language processing (almost) from scratch. Journal
of Machine Learning Research , 12, 2493–2537.
Collobert, R., Kavukcuoglu, K., and Farabet, C.
(2011b). Torch7: A matlab-like environment for
machine learning. In BigLearn, NIPS Workshop .
Courville, A., Bergstra, J., and Bengio, Y. (2011).
Unsupervised models of images by spike-and-slab
RBMs. In ICML’2011.
Dauphin, Y., Glorot, X., and Bengio, Y. (2011). Sam-
pled reconstruction for large-scale learning of em-
beddings. In Proc. ICML’2011.
Deerwester, S., Dumais, S. T., Furnas, G. W., Lan-
dauer, T. K., and Harshman, R. (1990). Indexing
by latent semantic analysis. J. Am. Soc. Informa-
tion Science , 41(6), 391–407.
Elman, J. L. (1993). Learning and development in
neural networks: The importance of starting small.
Cognition, 48, 781–799.
Erhan, D., Courville, A., and Bengio, Y. (2010a).
Understanding representations learned in deep ar-
chitectures. Technical Report 1355, Universit´ e de
Montr´ eal/DIRO.
Erhan, D., Bengio, Y., Courville, A., Manzagol, P.-
A., Vincent, P., and Bengio, S. (2010b). Why does
unsupervised pre-training help deep learning? J.
Machine Learning Res. , 11, 625–660.
Frasconi, P., Gori, M., and Sperduti, A. (1998). A
general framework for adaptive processing of data
structures. IEEE Transactions on Neural Net-
works, 9(5), 768–786.
Geman, S., Bienenstock, E., and Doursat, R. (1992).
Neural networks and the bias/variance dilemma.
Neural Computation , 4(1), 1–58.
Getoor, L. and Taskar, B. (2006). Introduction to
Statistical Relational Learning . MIT Press.
Glorot, X. and Bengio, Y. (2010). Understanding
the diﬃculty of training deep feedforward neural
networks. In AISTATS’2010, pages 249–256.
Glorot, X., Bordes, A., and Bengio, Y. (2011a).
Deep sparse rectiﬁer neural networks. In AIS-
TATS’2011.
Glorot, X., Bordes, A., and Bengio, Y. (2011b). Do-
main adaptation for large-scale sentiment classiﬁ-
cation: A deep learning approach. In ICML’2011.
Goodfellow, I., Le, Q., Saxe, A., and Ng, A.
(2009). Measuring invariances in deep networks.
In NIPS’2009, pages 646–654.
Goodfellow, I., Courville, A., and Bengio, Y. (2011).
Spike-and-slab sparse coding for unsupervised fea-
ture discovery. In NIPS Workshop on Challenges
in Learning Hierarchical Models .
Graepel, T., Candela, J. Q., Borchert, T., and Her-
brich, R. (2010). Web-scale Bayesian click-through
rate prediction for sponsored search advertising in
microsoft’s bing search engine. In ICML’2010.
H ˚ astad, J. (1986). Almost optimal lower bounds for
small depth circuits. In STOC’86, pages 6–20.
H ˚ astad, J. and Goldmann, M. (1991). On the power
of small-depth threshold circuits. Computational
Complexity, 1, 113–129.
Hinton, G. E. (1978). Relaxation and its role in vi-
sion. Ph.D. thesis, University of Edinburgh.
Hinton, G. E. (1986). Learning distributed represen-
tations of concepts. In Proc. 8th Annual Conf. Cog.
Sc. Society , pages 1–12.
Hinton, G. E. (1989). Connectionist learning proce-
dures. Artiﬁcial Intelligence , 40, 185–234.
28
Hinton, G. E. (2010). A practical guide to train-
ing restricted Boltzmann machines. Technical Re-
port UTML TR 2010-003, Department of Com-
puter Science, University of Toronto.
Hinton, G. E. (2013). A practical guide to training
restricted boltzmann machines. In K.-R. M¨ uller,
G. Montavon, and G. B. Orr, editors, Neural Net-
works: Tricks of the Trade, Reloaded . Springer.
Hinton, G. E., Osindero, S., and Teh, Y.-W. (2006).
A fast learning algorithm for deep belief nets. Neu-
ral Computation , 18, 1527–1554.
Hutter, F. (2009). Automated Conﬁguration of Algo-
rithms for Solving Hard Computational Problems .
Ph.D. thesis, University of British Columbia.
Hutter, F., Hoos, H., and Leyton-Brown, K. (2011).
Sequential model-based optimization for general
algorithm conﬁguration. In LION-5 .
Jarrett, K., Kavukcuoglu, K., Ranzato, M., and Le-
Cun, Y. (2009). What is the best multi-stage ar-
chitecture for object recognition? In ICCV’09.
Kavukcuoglu, K., Ranzato, M.-A., Fergus, R., and
LeCun, Y. (2009). Learning invariant features
through topographic ﬁlter maps. In CVPR’2009.
Krueger, K. A. and Dayan, P. (2009). Flexible shap-
ing: how learning in small steps helps. Cognition,
110, 380–394.
Lamblin, P. and Bengio, Y. (2010). Important gains
from supervised ﬁne-tuning of deep architectures
on large labeled sets. NIPS*2010 Deep Learning
and Unsupervised Feature Learning Workshop.
Lang, K. J. and Hinton, G. E. (1988). The develop-
ment of the time-delay neural network architecture
for speech recognition. Technical Report CMU-CS-
88-152, Carnegie-Mellon University.
Larochelle, H. and Bengio, Y. (2008). Classiﬁca-
tion using discriminative restricted Boltzmann ma-
chines. In ICML’2008.
Larochelle, H., Bengio, Y., Louradour, J., and Lam-
blin, P. (2009). Exploring strategies for training
deep neural networks. J. Machine Learning Res. ,
10, 1–40.
Le, Q., Ngiam, J., Chen, Z., hao Chia, D. J., Koh,
P. W., and Ng, A. (2010). Tiled convolutional neu-
ral networks. In NIPS’2010.
Le, Q., Ngiam, J., Coates, A., Lahiri, A., Prochnow,
B., and Ng, A. (2011). On optimization methods
for deep learning. In ICML’2011.
Le Roux, N., Manzagol, P.-A., and Bengio, Y. (2008).
Topmoumoute online natural gradient algorithm.
In NIPS’07.
Le Roux, N., Bengio, Y., and Fitzgibbon, A. (2011).
Improving ﬁrst and second-order methods by mod-
eling uncertainty. In Optimization for Machine
Learning. MIT Press.
Le Roux, N., Schmidt, M., and Bach, F. (2012).
A stochastic gradient method with an exponen-
tial convergence rate for strongly-convex optimiza-
tion with ﬁnite training sets. Technical report,
arXiv:1202.6258.
LeCun, Y. (1987). Mod` eles connexionistes de
l’apprentissage. Ph.D. thesis, Universit´ e de Paris
VI.
LeCun, Y. (1989). Generalization and network de-
sign strategies. Technical Report CRG-TR-89-4,
University of Toronto.
LeCun, Y. (2013). to appear. In K.-R. M¨ uller,
G. Montavon, and G. B. Orr, editors, Neural Net-
works: Tricks of the Trade, Reloaded . Springer.
LeCun, Y., Boser, B., Denker, J. S., Henderson, D.,
Howard, R. E., Hubbard, W., and Jackel, L. D.
(1989). Backpropagation applied to handwritten
zip code recognition. Neural Computation , 1(4),
541–551.
LeCun, Y., Bottou, L., Orr, G. B., and M¨ uller, K.
(1998a). Eﬃcient backprop. In Neural Networks,
Tricks of the Trade .
29
LeCun, Y., Bottou, L., Bengio, Y., and Haﬀner, P.
(1998b). Gradient based learning applied to docu-
ment recognition. IEEE, 86(11), 2278–2324.
Lee, H., Ekanadham, C., and Ng, A. (2008). Sparse
deep belief net model for visual area V2. In
NIPS’07.
Lee, H., Grosse, R., Ranganath, R., and Ng, A. Y.
(2009). Convolutional deep belief networks for scal-
able unsupervised learning of hierarchical represen-
tations. In ICML’2009.
Martens, J. (2010). Deep learning via Hessian-free
optimization. In ICML’2010, pages 735–742.
Mesnil, G., Dauphin, Y., Glorot, X., Rifai, S., Ben-
gio, Y., Goodfellow, I., Lavoie, E., Muller, X.,
Desjardins, G., Warde-Farley, D., Vincent, P.,
Courville, A., and Bergstra, J. (2011). Unsuper-
vised and transfer learning challenge: a deep learn-
ing approach. In JMLR W&CP: Proc. Unsuper-
vised and Transfer Learning , volume 7.
Nair, V. and Hinton, G. E. (2010). Rectiﬁed linear
units improve restricted Boltzmann machines. In
ICML’2010.
Nemirovski, A. and Yudin, D. (1983). Problem com-
plexity and method eﬃciency in optimization . Wi-
ley.
Nesterov, Y. (2009). Primal-dual subgradient meth-
ods for convex problems. Mathematical program-
ming, 120(1), 221–259.
Olshausen, B. A. and Field, D. J. (1997). Sparse
coding with an overcomplete basis set: a strategy
employed by V1? Vision Research, 37, 3311–3325.
Pearlmutter, B. (1994). Fast exact multiplication by
the Hessian. Neural Computation , 6(1), 147–160.
Pinto, N., Doukhan, D., DiCarlo, J. J., and Cox,
D. D. (2009). A high-throughput screening ap-
proach to discovering good forms of biologically
inspired visual representation. PLoS Comput Biol ,
5(11), e1000579.
Pollack, J. B. (1990). Recursive distributed represen-
tations. Artiﬁcial Intelligence , 46(1), 77–105.
Polyak, B. and Juditsky, A. (1992). Acceleration of
stochastic approximation by averaging. SIAM J.
Control and Optimization , 30(4), 838–855.
Raiko, T., Valpola, H., and LeCun, Y. (2012). Deep
learning made easier by linear transformations in
perceptrons. In AISTATS’2012.
Ranzato, M., Poultney, C., Chopra, S., and LeCun,
Y. (2007). Eﬃcient learning of sparse representa-
tions with an energy-based model. In NIPS’06.
Ranzato, M., Boureau, Y.-L., and LeCun, Y. (2008a).
Sparse feature learning for deep belief networks.
In J. Platt, D. Koller, Y. Singer, and S. Roweis,
editors, Advances in Neural Information Process-
ing Systems 20 (NIPS’07) , pages 1185–1192, Cam-
bridge, MA. MIT Press.
Ranzato, M., Boureau, Y., and LeCun, Y. (2008b).
Sparse feature learning for deep belief networks. In
NIPS’2007.
Richardson, M. and Domingos, P. (2006). Markov
logic networks. Machine Learning , 62, 107–136.
Rifai, S., Vincent, P., Muller, X., Glorot, X., and
Bengio, Y. (2011). Contracting auto-encoders:
Explicit invariance during feature extraction. In
ICML’2011.
Rifai, S., Bengio, Y., Dauphin, Y., and Vincent, P.
(2012). A generative process for sampling contrac-
tive auto-encoders. In ICML’2012.
Robbins, H. and Monro, S. (1951). A stochastic
approximation method. Annals of Mathematical
Statistics, 22, 400–407.
Rumelhart, D. E., Hinton, G. E., and Williams,
R. J. (1986). Learning representations by back-
propagating errors. Nature, 323, 533–536.
Salakhutdinov, R. and Hinton, G. (2009). Deep
Boltzmann machines. In AISTATS’2009.
30
Saxe, A. M., Koh, P. W., Chen, Z., Bhand, M.,
Suresh, B., and Ng, A. (2011). On random weights
and unsupervised feature learning. In ICML’2011.
Schraudolph, N. N. (1998). Centering neural network
gradient factors. In G. B. Orr and K.-R. Muller, ed-
itors, Neural Networks: Tricks of he Trade , pages
548–548. Springer.
Socher, R., Lin, C., Ng, A. Y., and Manning, C.
(2011). Learning continuous phrase representa-
tions and syntactic parsing with recursive neural
networks. In ICML’2011.
Srinivasan, A. and Ramakrishnan, G. (2011). Pa-
rameter screening and optimisation for ILP using
designed experiments. Journal of Machine Learn-
ing Research , 12, 627–662.
Swersky, K., Chen, B., Marlin, B., and de Freitas,
N. (2010). A tutorial on stochastic approximation
algorithms for training restricted boltzmann ma-
chines and deep belief nets. In Information Theory
and Applications Workshop .
Tenenbaum, J., de Silva, V., and Langford, J. C.
(2000). A global geometric framework for nonlin-
ear dimensionality reduction. Science, 290(5500),
2319–2323.
Tieleman, T. and Hinton, G. (2009). Using fast
weights to improve persistent contrastive diver-
gence. In ICML’2009.
van der Maaten, L. and Hinton, G. E. (2008). Visual-
izing data using t-sne. J. Machine Learning Res. ,
9.
Vincent, P. (2011). A connection between score
matching and denoising autoencoders. Neural
Computation, 23(7).
Vincent, P., Larochelle, H., Bengio, Y., and Man-
zagol, P.-A. (2008). Extracting and composing
robust features with denoising autoencoders. In
ICML 2008 .
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y.,
and Manzagol, P.-A. (2010). Stacked denoising au-
toencoders: Learning useful representations in a
deep network with a local denoising criterion. J.
Machine Learning Res. , 11.
Weston, J., Ratle, F., and Collobert, R. (2008). Deep
learning via semi-supervised embedding. In ICML
2008.
Weston, J., Bengio, S., and Usunier, N. (2011). Ws-
abie: Scaling up to large vocabulary image anno-
tation. In Proceedings of the International Joint
Conference on Artiﬁcial Intelligence, IJCAI .
Wiskott, L. and Sejnowski, T. J. (2002). Slow fea-
ture analysis: Unsupervised learning of invari-
ances. Neural Computation , 14(4), 715–770.
Zou, W. Y., Ng, A. Y., and Yu, K. (2011). Unsu-
pervised learning of visual invariance with tempo-
ral coherence. In NIPS 2011 Workshop on Deep
Learning and Unsupervised Feature Learning .
31

FBNet: Hardware-Aware Efﬁcient ConvNet Design
via Differentiable Neural Architecture Search
Bichen Wu1∗, Xiaoliang Dai2∗, Peizhao Zhang3, Yanghan Wang3, Fei Sun3,
Yiming Wu3, Yuandong Tian3, Peter Vajda3, Yangqing Jia3, and Kurt Keutzer1
1UC Berkeley, 2Princeton University, 3Facebook Inc.
{bichen, keutzer}@berkeley.edu, xdai@princeton.edu,
{stzpz, yanghan, feisun, wyiming, yuandong, vajdap, jiayq}@fb.com
Abstract
Designing accurate and efﬁcient ConvNets for mobile
devices is challenging because the design space is com-
binatorially large. Due to this, previous neural archi-
tecture search (NAS) methods are computationally expen-
sive. ConvNet architecture optimality depends on fac-
tors such as input resolution and target devices. How-
ever, existing approaches are too resource demanding for
case-by-case redesigns. Also, previous work focuses pri-
marily on reducing FLOPs, but FLOP count does not al-
ways reﬂect actual latency. To address these, we propose
a differentiable neural architecture search (DNAS) frame-
work that uses gradient-based methods to optimize Con-
vNet architectures, avoiding enumerating and training in-
dividual architectures separately as in previous methods.
FBNets (Facebook-Berkeley-Nets), a family of models dis-
covered by DNAS surpass state-of-the-art models both de-
signed manually and generated automatically. FBNet-B
achieves 74.1% top-1 accuracy on ImageNet with 295M
FLOPs and 23.1 ms latency on a Samsung S8 phone, 2.4x
smaller and 1.5x faster than MobileNetV2-1.3[17] with
similar accuracy. Despite higher accuracy and lower la-
tency than MnasNet[20], we estimate FBNet-B’s search
cost is 420x smaller than MnasNet’s, at only 216 GPU-
hours. Searched for different resolutions and channel sizes,
FBNets achieve 1.5% to 6.4% higher accuracy than Mo-
bileNetV2. The smallest FBNet achieves 50.2% accuracy
and 2.9 ms latency (345 frames per second) on a Sam-
sung S8. Over a Samsung-optimized FBNet, the iPhone-X-
optimized model achieves a 1.4x speedup on an iPhone X.
FBNet models are open-sourced at https://github.
com/facebookresearch/mobile-vision.
∗Work done while interning at Facebook.
… …Stochastic	super	netDistribution
Operators
Probability
Training	super	net	Proxydataset
Sampling
OperatorLatencyLUTDeploy
Target	deviceBenchmark
…
Search	space
… …Neural	Architectures
Figure 1. Differentiable neural architecture search (DNAS) for
ConvNet design. DNAS explores a layer-wise space that each
layer of a ConvNet can choose a different block. The search space
is represented by a stochastic super net. The search process trains
the stochastic super net using SGD to optimize the architecture
distribution. Optimal architectures are sampled from the trained
distribution. The latency of each operator is measured on target
devices and used to compute the loss for the super net.
1. Introduction
ConvNets are the de facto method for computer vision.
In many computer vision tasks, a better ConvNet design
usually leads to signiﬁcant accuracy improvement. In pre-
vious works, accuracy improvement comes at the cost of
higher computational complexity, making it more challeng-
ing to deploy ConvNets to mobile devices, on which com-
puting capacity is limited. Instead of solely focusing on
accuracy, recent work also aims to optimize for efﬁciency,
especially latency. However, designing efﬁcient and accu-
rate ConvNets is difﬁcult due to the challenges below.
Intractable design space : The design space of a Con-
vNet is combinatorial. Using VGG16 [18] as a motivat-
ing example: VGG16 contains 16 layers. Assume for
each layer of the network, we can choose a different ker-
nel size from {1,3,5}and a different ﬁlter number from
{32,64,128,256,512}. Even with such simpliﬁed de-
sign choices and shallow layers, the design space contains
(3 ×5)16 ≈ 6 ×1018 possible architectures. However,
arXiv:1812.03443v3  [cs.CV]  24 May 2019
…
Search	space… …Neural	Architectures
Training	&	Evaluation	
TargetdatasetPerformance
Accuracy
Latency
Manual	Design
(a) A typical ﬂow of manual ConvNet design.
… …Neural	Architectures
Training	&	Evaluation	ProxydatasetPerformance
Accuracy
Latency
Controller	sampling
Training	controller
…
Search	space
(b) A typical ﬂow of reinforcement learning based neural architec-
ture search.
Figure 2. Illustration of manual ConvNet design and reinforcement
learning based neural architecture search.
training a ConvNet is very time-consuming, typically taking
days or even weeks. As a result, previous ConvNet design
rarely explores the design space. A typical ﬂow of man-
ual ConvNet design is illustrated in Figure 2(a). Design-
ers propose initial architectures and train them on the target
dataset. Based on the performance, designers evolve the ar-
chitectures accordingly. Limited by the time cost of training
ConvNets, the design ﬂow has to stop after a few iterations,
which is far too few to sufﬁciently explore the design space.
Starting from [30], recent works adopt neural architec-
ture search (NAS) to explore the design space automati-
cally. Many previous works [30, 31, 20] use reinforce-
ment learning (RL) to guide the search and a typical ﬂow
is illustrated in Figure 2(b). A controller samples architec-
tures from the search space to be trained. To reduce the
training cost, sampled architectures are trained on a smaller
proxy dataset such as CIFAR-10 or trained for fewer epochs
on ImageNet. The performance of the trained networks
is then used to train and improve the controller. Previ-
ous works [30, 31, 20] has demonstrated the effectiveness
of such methods in ﬁnding accurate and efﬁcient ConvNet
models. However, training each architecture is still time-
consuming, and it usually takes thousands of architectures
to train the controller. As a result, the computational cost of
such methods is prohibitively high.
Nontransferable optimality: the optimality of ConvNet
architectures is conditioned on many factors such as input
resolutions and target devices. Once these factors change,
the optimal architecture is likely to be different. A common
practice to reduce the FLOP count of a network is to shrink
the input resolution. A smaller input resolution may require
a smaller receptive ﬁeld of the network and therefore shal-
lower layers. On a different device, the same operator can
have different latency, so we need to adjust the ConvNet ar-
chitecture to achieve the best accuracy-efﬁciency trade-off.
Ideally, we should design different ConvNet architectures
case-by-case. In practice, however, limited by the computa-
tional cost of previous manual and automated approaches,
we can only realistically design one ConvNet and use it for
all conditions.
Inconsistent efﬁciency metrics : Most of the efﬁciency
metrics we care about are dependent on not only the Con-
vNet architecture but also the hardware and software con-
ﬁgurations on the target device. Such metrics include la-
tency, power, energy, and in this paper, we mainly focus
on latency. To simplify the problem, most of the previ-
ous works adopt hardware-agnostic metrics such as FLOPs
(more strictly, number of multiply-add operations) to evalu-
ate a ConvNet’s efﬁciency. However, a ConvNet with lower
FLOP count is not necessarily faster. For example, NasNet-
A [31] has a similar FLOP count as MobileNetV1 [6], but
its complicated and fragmented cell-level structure is not
hardware friendly, so the actual latency is slower [17]. The
inconsistency between hardware agnostic metrics and ac-
tual efﬁciency makes the ConvNet design more difﬁcult.
To address the above problems, we propose to use dif-
ferentiable neural architecture search (DNAS) to discover
hardware-aware efﬁcient ConvNets. The ﬂow of our algo-
rithm is illustrated in Figure 1. DNAS allows us to explore
a layer-wise search space where we can choose a different
block for each layer of the network. Following [21], DNAS
represents the search space by a super net whose operators
execute stochastically. We relax the problem of ﬁnding the
optimal architecture to ﬁnd a distribution that yields the op-
timal architecture. By using the Gumbel Softmax technique
[9], we can directly train the architecture distribution us-
ing gradient-based optimization such as SGD. The search
process is extremely fast compared with previous reinforce-
ment learning (RL) based method. The loss used to train the
stochastic super net consists of both the cross-entropy loss
that leads to better accuracy and the latency loss that penal-
izes the network’s latency on a target device. To estimate the
latency of an architecture, we measure the latency of each
operator in the search space and use a lookup table model
to compute the overall latency by adding up the latency of
each operator. Using this model allows us to quickly es-
timate the latency of architectures in this enormous search
space. More importantly, it makes the latency differentiable
with respect to layer-wise block choices.
We name the models discovered by DNAS as FBNets
(Facebook-Berkeley-Nets). FBNets surpass the state-of-
the-art efﬁcient ConvNets designed manually and automat-
ically. FBNet-B achieves 74.1% top-1 accuracy with 295M
FLOPs and 23.1 ms latency on an Samsung S8 phone, 2.4x
smaller and 1.5x faster than MobileNetV2-1.3. Being better
than MnasNet, FBNet-B’s search cost is 216 GPU-hours,
421x lower than the cost for MnasNet estimated based on
[20]. Such low search cost enables us to re-design ConvNets
case-by-case. For different resolution and channel scaling,
FBNets achieve 1.5% to 6.4% absolute gain in top-1 ac-
curacy compared with MobileNetV2 models. The smallest
FBNet achieves 50.2% accuracy and 2.9 ms latency (345
frames per second) with a batch size of 1 on Samsung
S8. Using DNAS to search for device-speciﬁc ConvNet,
an iPhone-x-optimized model achieves 1.4x speedup on an
iPhone X compared with a Samsung-optimized model.
2. Related work
Efﬁcient ConvNet models : Designing efﬁcient Con-
vNet has attracted many research attention in recent years.
SqueezeNet [8] is one of the early works focusing on reduc-
ing the parameter size of ConvNet models. It is originally
designed for classiﬁcation, but later extended to object de-
tection [22] and LiDAR point-cloud segmentation [24, 26].
Following SqueezeNet, SqueezeNext [3] and ShiftNet [23]
achieve further parameter size reduction. Recent works
change the focus from parameter size to FLOPs. Mo-
bileNetV1 and MobileNetV2 [6, 17] use depthwise con-
volutions to replace the more expensive spatial convolu-
tions. ShufﬂeNet [29] uses group convolution and shuf-
ﬂe operations to reduce the FLOP count further. More re-
cent works realize that FLOP count does not always reﬂect
the actual hardware efﬁciency. To improve actual latency,
ShufﬂeNetV2 [13] proposes a series of practical guide-
lines for efﬁcient ConvNet design. Synetgy [28] combines
ideas from ShufﬂeNetV2 and ShiftNet to co-design hard-
ware friendly ConvNets and FPGA accelerators.
Neural Architecture Search: [30, 31] ﬁrst proposes to
use reinforcement learning (RL) to search for neural archi-
tectures to achieve competitive accuracy with low FLOPs.
Early NAS methods are computationally expensive. Recent
works try to reduce the computational cost by weight shar-
ing [16] or using gradient-based optimization [12]. [25, 1]
further develop the idea of differentiable neural architec-
ture search combining Gumbel Softmax [9]. Early works of
NAS [31, 16, 12] focus on the cell level architecture search,
and the same cell structure is repeated in all layers of a
network. However, such fragmented and complicated cell-
level structures are not hardware friendly, and the actual ef-
ﬁciency is low. Most recently, [20] explores a stage-level
hierarchical search space, allowing different blocks for dif-
ferent stages of a network, while blocks inside a stage are
still the same. Instead of focusing on FLOPs, [20] aims to
optimize the latency on target devices. Besides searching
for new architectures, works such as [27, 5] focus on adapt-
ing existing models to improve efﬁciency.
3. Method
In this paper, we use differentiable neural architecture
search (DNAS) to solve the problem of ConvNet design.
We formulate the neural architecture search problem as
min
a∈A
min
wa
L(a,wa). (1)
Given an architecture space A, we seek to ﬁnd an optimal
architecture a ∈A such that after training its weights wa,
it can achieve the minimal loss L(a,wa). In our work, we
focus on three factors of the problem: a) the search spaceA.
b) The loss function L(a,wa) that considers actual latency.
c) An efﬁcient search algorithm.
3.1. The Search Space
Previous works [30, 31, 16, 11, 12] focus on cell level
architecture search. Once a cell structure is searched, it
is used in all the layers across the network. However,
many searched cell structures are very complicated and
fragmented and are therefore slow when deployed to mo-
bile CPUs [17, 13]. Besides, at different layers, the same
cell structure can have a different impact on the accuracy
and latency of the overall network. As shown in [20] and in
our experiments, allowing different layers to choose differ-
ent blocks leads to better accuracy and efﬁciency.
In this work, we construct a layer-wise search space with
a ﬁxed macro-architecture, and each layer can choose a dif-
ferent block. The macro-architecture is described in Table
1. The macro architecture deﬁnes the number of layers and
the input/output dimensions of each layer. The ﬁrst and the
last three layers of the network have ﬁxed operators. For
the rest of the layers, their block type needs to be searched.
The ﬁlter numbers for each layer are hand-picked empiri-
cally. We use relatively small channel sizes for early layers,
since the input resolution at early layers is large, and the
computational cost (FLOP count) is quadratic to input size.
Each searchable layer in the network can choose a differ-
ent block from the layer-wise search space. The block struc-
ture is inspired by MobileNetV2 [17] and ShiftNet [23], and
is illustrated in Figure 3. It contains a point-wise (1x1) con-
volution, a K-by-K depthwise convolution where K denotes
the kernel size, and another 1x1 convolution. “ReLU” ac-
tivation functions follow the ﬁrst 1x1 convolution and the
depthwise convolution, but there are no activation functions
following the last 1x1 convolution. If the output dimension
stays the same as the input dimension, we use a skip con-
nection to add the input to the output. Following [17, 23],
we use a hyperparameter, the expansion ratio e, to control
Input shape Block f n s
2242 ×3 3x3 conv 16 1 2
1122 ×16 TBS 16 1 1
1122 ×16 TBS 24 4 2
562 ×24 TBS 32 4 2
282 ×32 TBS 64 4 2
142 ×64 TBS 112 4 1
142 ×112 TBS 184 4 2
72 ×184 TBS 352 1 1
72 ×352 1x1 conv 1504 (1984) 1 1
72 ×1504 (1984) 7x7 avgpool - 1 1
1504 fc 1000 1 -
Table 1. Macro-architecture of the search space. Column-“Block”
denotes the block type. “TBS” means layer type needs to be
searched. Column-f denotes the ﬁlter number of a block. Column-
n denotes the number of blocks. Column- s denotes the stride of
the ﬁrst block in a stage. The ﬁlter size of the last 1x1 conv is 1504
for FBNet-A and 1984 for FBNet-{B, C}.
1x1	(group)	Conv,	ReLU
K	x	K	DWConv,	ReLU
H	x	W	x	Cin
H	x	W	x	(e	x	Cin)
(H/s)	x	(W/s)	x	(e	x	Cin)
(H/s)	x	(W/s)	x	Cout
+
1x1	(group)	Conv
Figure 3. The block structure of the micro-architecture search
space. Each candidate block in the search space can choose a dif-
ferent expansion rate, kernel size, and number of groups for group
convolution.
the block. It determines how much do we expand the output
channel size of the ﬁrst 1x1 convolution compared with its
input channel size. Following [20], we also allow choos-
ing a kernel size of 3 or 5 for the depthwise convolution.
In addition, we can choose to use group convolution for the
ﬁrst and the last 1x1 convolution to reduce the computation
complexity. When we use group convolution, we follow
[29] to add a channel shufﬂe operation to mix the informa-
tion between channel groups.
In our experiments, our layer-wise search space contains
9 candidate blocks, with their conﬁgurations listed in Table
2. Note we also have a block called “skip”, which directly
feed the input feature map to the output without actual com-
putations. This candidate block essentially allows us to re-
duce the depth of the network.
In summary, our overall search space contains 22 layers
and each layer can choose from 9 candidate blocks from
Block type expansion Kernel Group
k3 e1 1 3 1
k3 e1 g2 1 3 2
k3 e3 3 3 1
k3 e6 6 3 1
k5 e1 1 5 1
k5 e1 g2 1 5 2
k5 e3 3 5 1
k5 e6 6 5 1
skip - - -
Table 2. Conﬁgurations of candidate blocks in the search space.
Table 2, so it contains 922 ≈1021 possible architectures.
Finding the optimal layer-wise block assignment from such
enormous search space is a non-trivial task.
3.2. Latency-Aware Loss Function
The loss function used in (1) has to reﬂect not only the
accuracy of a given architecture but also the latency on the
target hardware. To achieve this goal, we deﬁne the follow-
ing loss function:
L(a,wa) = CE(a,wa) ·αlog(LAT(a))β. (2)
The ﬁrst term CE (a,wa) denotes the cross-entropy loss of
architecture awith parameter wa. The second term LAT(a)
denotes the latency of the architecture on the target hard-
ware measured in micro-second. The coefﬁcient αcontrols
the overall magnitude of the loss function. The exponent
coefﬁcient βmodulates the magnitude of the latency term.
The cross-entropy term can be easily computed. How-
ever, the latency term is more difﬁcult, since we need to
measure the actual runtime of an architecture on a target de-
vice. To cover the entire search space, we need to measure
about 1021 architectures, which is an impossible task.
To solve this problem, we use a latency lookup table
model to estimate the overall latency of a network based
on the runtime of each operator. More formally, we assume
LAT(a) =
∑
l
LAT(b(a)
l ), (3)
where b(a)
l denotes the block at layer-lfrom architecture a.
This assumes that on the target processor, the runtime of
each operator is independent of other operators. The as-
sumption is valid for many mobile CPUs and DSPs, where
operators are computed sequentially one by one. This way,
by benchmarking the latency of a few hundred operators
used in the search space, we can easily estimate the actual
runtime of the 1021 architectures in the entire search space.
More importantly, as will be explained in section 3.3, using
the lookup table model makes the latency term in the loss
function (2) differentiable with respect to layer-wise block
choices, and this allows us to use gradient-based optimiza-
tion to solve problem (1).
3.3. The Search Algorithm
Solving the problem (1) through brute-force enumera-
tion of the search space is very infeasible. The inner prob-
lem of optimizing wa involves training a neural network.
For ImageNet classiﬁcation, training a ConvNet typically
takes several days or even weeks. The outer problem of op-
timizing a∈A has a combinatorially large search space.
Most of the early works on NAS [30, 31, 20] follow the
paradigm above. To reduce the computational cost, the in-
ner problem is replaced by training candidate architectures
on an easier proxy dataset. For example, [30, 31] trains the
architecture on the CIFAR10 dataset, and [20] trains on Im-
ageNet but only for 5 epochs. The learned architectures are
then transferred to the target dataset. To avoid exhaustively
iterating through the search space, [30, 31, 20] use rein-
forcement learning to guide the exploration. Despite these
improvements, solving problem (1) is still prohibitively ex-
pensive – training a network on the proxy dataset is still
time-consuming, and thousands of architectures need to be
trained before reaching the optimal solution.
We adopt a different paradigm of solving problem (1).
We ﬁrst represent the search space by a stochastic super net.
The super net has the same macro-architecture as described
in Table 1, and each layer contains 9 parallel blocks as de-
scribed in Table 2. During the inference of the super net,
only one candidate block is sampled and executed with the
sampling probability of
Pθl (bl = bl,i) =softmax(θl,i; θl) = exp(θl,i)∑
iexp(θl,i). (4)
θl contains parameters that determine the sampling proba-
bility of each block at layer- l. Equivalently, the output of
layer-lcan be expressed as
xl+1 =
∑
i
ml,i ·bl,i(xl), (5)
where ml,i is a random variable in {0,1}and is evaluated
to 1 if block bl,i is sampled. The sampling probability is
determined by equation (4). bl,i(xl) denotes the output of
block-i at layer l given the input feature map xl. We let
each layer sample independently, therefore, the probability
of sampling an architecture acan be described as
Pθ(a) =
∏
l
Pθl (bl = b(a)
l,i ), (6)
where θ denotes the a vector consists of all the θl,i for each
block-iat layer-l. b(a)
l,i denotes that in the sampled architec-
ture a, block-iis chosen at layer-l.
Instead of solving for the optimal architecture a ∈A,
which has a discrete search space, we relax the problem to
optimize the probability Pθ of the stochastic super net to
achieve the minimum expected loss. Formally, we re-write
the discrete optimization problem (1) as
min
θ
min
wa
Ea∼Pθ {L(a,wa)}. (7)
It is obvious the loss function in (7) is differentiable with
respect to the architecture weights wa and therefore can be
optimized by stochastic gradient descent (SGD). However,
the loss is not directly differentiable to the sampling param-
eter θ, since we cannot pass the gradient through the dis-
crete random variable ml,i to θl,i. To sidestep this, we relax
the discrete mask variable ml,i to be a continuous random
variable computed by the Gumbel Softmax function [9, 14]
ml,i = GumbelSoftmax(θl,i|θl)
= exp[(θl,i + gl,i)/τ]∑
iexp[(θl,i + gl,i)/τ], (8)
where gl,i ∼Gumbel(0, 1) is a random noise following the
Gumbel distribution. The Gumbel Softmax function is con-
trolled by a temperature parameter τ. As τ approaches 0,
it approximates the discrete categorical sampling following
the distribution in (6). As τ becomes larger, ml,i becomes
a continuous random variable. Regardless of the value ofτ,
the mask ml,i is directly differentiable with respect to the
parameter θl,i. The technique of using Gumbel Softmax for
neural architecture search is also proposed in [25, 1].
As a result, it is clear that the cross-entropy term from
the loss function (2) is differentiable with respect to the
mask ml,i and therefore θl,i. For the latency term, since we
use the lookup table based model for efﬁciency estimation,
equation (3) can be written as
LAT(a) =
∑
l
∑
i
ml,i ·LAT(bl,i). (9)
The latency of each operator LAT (bl,i) is a constant coef-
ﬁcient, so the overall latency of architecture- a is differen-
tiable with respect to the mask ml,i, therefore θl,i.
As a result, the loss function (2) is fully differentiable
with respect to both weights wa and the architecture distri-
bution parameter θ. This allows us to use SGD to efﬁciently
solve problem (1).
Our search process is now equivalent to training the
stochastic super net. During the training, we compute
∂L/∂wa to train each operator’s weight in the super net.
This is no different from training an ordinary ConvNet. Af-
ter operators get trained, different operators can have a dif-
ferent contribution to the accuracy and the efﬁciency of the
overall network. Therefore, we compute ∂L/∂θ to update
the sampling probability Pθ for each operator. This step se-
lects operators with better accuracy and lower latency and
suppresses the opposite ones. After the super net training
ﬁnishes, we can then obtain the optimal architectures by
sampling from the architecture distribution Pθ.
As will be shown in the experiment section, the proposed
DNAS algorithm is orders of magnitude faster than previous
RL based NAS while generating better architectures.
4. Experiments
4.1. ImageNet Classiﬁcation
To demonstrate the efﬁcacy of our proposed method, we
use DNAS to search for ConvNet models on ImageNet 2012
classiﬁcation dataset [2], and we name the discovered mod-
els FBNets. We aim to discover models with high accuracy
and low latency on target devices. In our ﬁrst experiment,
we target Samsung Galaxy S8 with a Qualcomm Snap-
dragon 835 platform. The model is deployed with Caffe2
with int8 inference engine for mobile devices.
Before the search starts, we ﬁrst build a latency lookup
table described in section 3.2 on the target device. Next,
we train a stochastic super net with search space described
in section 3.3. We set the input resolution of the network
to 224-by-224. To reduce the training time, we randomly
choose 100 classes from the original 1000 classes to train
the stochastic super net. We train the stochastic super net for
90 epochs. In each epoch, we ﬁrst train the operator weights
wa and then the architecture probability parameter θ. wa is
trained on 80% of ImageNet training set using SGD with
momentum. The architecture distribution parameter θ is
trained on the rest 20% of ImageNet training set with Adam
optimizer [10]. To control the temperature of the Gumbel
Softmax from equation (8), we use an exponentially de-
caying temperature. After the search ﬁnishes, we sample
several architectures from the trained distribution Pθ, and
train them from scratch. Our architecture search framework
is implemented in pytorch [15] and searched models are
trained in Caffe2. More training details will be provided
in the supplementary materials.
Our experiment results are summarized in Table 3. We
compare our searched models with state-of-the-art efﬁcient
models both designed automatically and manually. The pri-
mary metrics we care about are top-1 accuracy on the Im-
ageNet validation set and the latency. If the latency is not
available, we use FLOP as the secondary efﬁciency metric.
For baseline models, we directly cite the parameter size,
FLOP count, and top-1 accuracy from the original paper.
Since our network is deployed with caffe2 with highly efﬁ-
cient in8 implementation, we have an unfair latency advan-
tage against other baselines. Therefore, we implement the
baseline models ourselves and measure their latency under
the same environment for a fair comparison. For automati-
cally designed models, we also compare the search method,
search space, and search cost.
Table 3 divides the models into three categories accord-
ing to their accuracy level. In the ﬁrst group, FBNet-
A achieves 73.0% accuracy, better than 1.0-MobileNetV2
(+1.0%), 1.5-ShufﬂeNet V2 (+0.4%), and CondenseNet
(+2%), and are on par with DARTS and MnasNet-65. Re-
garding latency, FBNet-A is 1.9 ms (relative 9.6%), 2.2
ms (relative 11%), and 8.6 ms (relative 43%) better than
the MobileNetV2, ShufﬂeNetV2, and CondenseNet coun-
terparts. Although we did not optimize for FLOP count di-
rectly, FBNet-A’s FLOP count is only 249M, 50M smaller
(relative 20%) than MobileNetV2 and ShufﬂeNetV2, 20M
(relative 8%) smaller than MnasNet, and 2.4X smaller than
DARTS. In the second group, FBNet-B achieves com-
parable accuracy with 1.3-MobileNetV2, but the latency
is 1.46x lower, and the FLOP count is 1.73x smaller,
even smaller than 1.0-MobileNetV2 and 1.5-ShufﬂeNet V2.
Compared with MnasNet, FBNet-B’s accuracy is 0.1%
higher, latency is 0.6ms lower, and FLOP count is 22M
(relative 7%) smaller. We do not have the latency of
NASNet-A and PNASNet, but the accuracy is compara-
ble, and the FLOP count is 1.9x and 2.0x smaller. In the
third group, FBNet-C achieves 74.9% accuracy, same as
2.0-ShufﬂeNetV2 and better than all others. The latency is
28.1 ms, 1.33x and 1.19x faster than MobileNet and Shuf-
ﬂeNet. The FLOP count is 1.56x, 1.58x, and 1.03x smaller
than MobileNet, ShufﬂeNet, and MnasNet-92.
Among all the automatically searched models, FBNet’s
performance is much stronger than DARTS, PNAS, and
NAS, and better than MnasNet. However, the search cost
is orders of magnitude lower. MnasNet [20] does not dis-
close the exact search cost (in terms of GPU-hours). How-
ever, it mentions that the controller samples 8,000 mod-
els during the search and each model is trained for ﬁve
epochs. According to our experiments, training of MNas-
Net for one epoch takes 17 minutes using 8 GPUs. So
the estimated cost for training 8,000 models for 5 epochs
is about 17/60 ×5 ×8 ×8,000 ≈91 ×103 GPU hours.
In comparison, the FBNet search takes 8 GPUs for only 27
hours, so the computational cost is only 216 GPU hours,
or 421x faster than MnasNet, 222x faster than NAS, 27.8x
faster than PNAS, and 1.33x faster than DARTS.
We visualize some of our searched FBNets, Mo-
bileNetV2, and MnasNet in Figure 4.
4.2. Different Resolution and Channel Size Scaling
A common technique to reduce the computational cost of
a ConvNet is to reduce the input resolution or channel size
without changing the ConvNet structure. This approach is
likely to be sub-optimal. We hypothesize that with a dif-
ferent input resolution and channel size scaling, the optimal
ConvNet structure will be different. To test this, we use
DNAS to search for several different combinations of input
resolution and channel size scaling. Thanks to the superior
efﬁciency of DNAS, we can ﬁnish the search very quickly.
The result is summarized in Table 4. Compared with Mo-
bileNetV2 under the same input size and channel size scal-
Model Search
method
Search
space
Search cost
(GPU hours / relative) #Params #FLOPs CPU
Latency
Top-1
acc (%)
1.0-MobileNetV2 [17] manual - - 3.4M 300M 21.7 ms 72.0
1.5-ShufﬂeNetV2 [13] manual - - 3.5M 299M 22.0 ms 72.6
CondenseNet (G=C=8) [7] manual - - 2.9M 274M 28.4 ‡ms 71.0
MnasNet-65 [13] RL stage-wise 91K ∗/ 421x 3.6M 270M - 73.0
DARTS [12] gradient cell 288 / 1.33x 4.9M 595M - 73.1
FBNet-A (ours) gradient layer-wise 216 / 1.0x 4.3M 249M 19.8 ms 73.0
1.3-MobileNetV2 [17] manual - - 5.3M 509M 33.8 ms 74.4
CondenseNet (G=C=4) [7] manual - - 4.8M 529M 28.7 ‡ms 73.8
MnasNet [20] RL stage-wise 91K ∗/ 421x 4.2M 317M 23.7 ms 74.0
NASNet-A [31] RL cell 48K / 222x 5.3M 564M - 74.0
PNASNet [11] SMBO cell 6K †/ 27.8x 5.1M 588M - 74.2
FBNet-B (ours) gradient layer-wise 216 / 1.0x 4.5M 295M 23.1 ms 74.1
1.4-MobileNetV2 [17] manual - - 6.9M 585M 37.4 ms 74.7
2.0-ShufﬂeNetV2 [13] manual - - 7.4M 591M 33.3 ms 74.9
MnasNet-92 [20] RL stage-wise 91K ∗/ 421x 4.4M 388M - 74.8
FBNet-C (ours) gradient layer-wise 216 / 1.0x 5.5M 375M 28.1 ms 74.9
Table 3. ImageNet classiﬁcation performance compared with baselines. For baseline models, we directly cite the parameter size, FLOP
count and top-1 accuracy on the ImageNet validation set from their original papers. For CPU latency, we deploy and benchmark the models
on the same Samsung Galaxy S8 phone with Caffe2 int8 implementation. The details of MnasNet- {64, 92} are not disclosed from [20]
so we cannot measure the latency. *The search cost for MnasNet is estimated according to the description in [20]. † The search cost is
estimated based on the claim from [11] that PNAS [11] is 8x lower than NAS[31]. ‡ The inference engine is faster than other models.
FBNet-A
FBNet-96-0.35-1
FBNet-s8
FBNet-iPhoneX
MobileNetV2
MnasNet
SkipK=3E=6K=3E=3K=3E=1K=3E=1G=2K=5E=6K=5E=3K=5E=1K=5E=1G=2
FBNet-B
FBNet-C
Figure 4. Visualization of some of the searched architectures. We
use rectangle boxes to denote blocks for each layer. We use differ-
ent colors to denote the kernel size of the depthwise convolution,
blue for kernel size of 3, green for kernel size of 5, and empty for
skipping. We use height to denote the expansion rate of the block:
6, 3, 1, and 1 with group-2 convolution.
ing, our searched models achieve 1.5% to 6.4% better accu-
racy with similar latency. Especially the FBNet-96-0.35-1
model achieves 50.2% (+4.7%) accuracy and 2.9 ms latency
(345 frames per second) on a Samsung Galaxy S8.
We visualize the architecture of FBNet-96-0.35-1 in Fig-
ure 4, we can see that many layers are skipped, and the net-
work is much shallower than FBNet-{A, B, C}, whose input
size is 224. We conjecture that this is because with smaller
input size, the receptive ﬁeld needed to parse the image also
becomes smaller, so having more layers will not effectively
increase the accuracy.
4.3. Different Target Devices
In previous ConvNet design practices, the same ConvNet
model is deployed to many different devices. However, this
is sub-optimal since different computing platforms and soft-
ware implementation can have different characteristics. To
validate this, we conduct search targeting two mobile de-
vices: Samsung Galaxy S8 with Qualcomm Snapdragon
835 platforms, and iPhone X with A11 Bionic processors.
We use the same architecture search space, but different la-
tency lookup tables collected from two target devices. All
the architecture search and training protocols are the same.
After we searched and trained two models, we deploy them
to both Samsung Galaxy S8 and iPhone X to benchmark the
overall latency. The result is summarized in Table. 5.
As we can see, the two models reach similar accuracy
(73.20% vs. 73.27%). FBNet-iphoneX model’s latency
is 19.84 ms on its target device, but when deployed to a
Samsung S8, its latency increases to 23.33 ms. On the
other hand, FBNet-S8 reaches a latency of 22.12 ms on a
Samsung S8, but when deployed to an iPhone X, the la-
tency hikes to 27.53 ms, 7.69 ms (relatively 39%) higher
than FBNet-iPhone X. This demonstrates the necessity of
re-designing ConvNets for different target devices.
Two models are visualized in Figure 4. Note that FBNet-
Input size &
Channel Scaling Model #Parameters #FLOPs CPU Latency Top-1 acc (%)
(224, 0.35)
MobileNetV2-224-0.35 1.7M 59M 9.3 ms 60.3
MNasNet-scale-224-0.35 1.9M 76M 10.7 ms 62.4 (+2.1)
FBNet-224-0.35 2.0M 72M 10.7 ms 65.3 (+5.0)
(192, 0.50)
MobileNetV2 2.0M 71M 8.4 ms 63.9
MnasNet-search-192-0.5 - - - 65.6 (+1.7)
FBNet-192-0.5 (ours) 2.6M 73M 9.9 ms 65.9 (+2.0)
(128, 1.0)
MobileNetV2 3.5M 99M 8.4 ms 65.3
MnasNet-scale-128-1.0 4.2M 103M 9.2 ms 67.3 (+2.0)
FBNet-128-1.0 (ours) 4.2M 92M 9.0 ms 67.0 (+1.7)
(128, 0.50) MobileNetV2 2.0M 32M 4.8 ms 57.7
FBNet-128-0.5 (ours) 2.4M 32M 5.1 ms 60.0 (+2.3)
(96, 0.35)
MobileNetV2 1.7M 11M 3.8 ms 45.5
FBNet-96-0.35-1 (ours) 1.8M 12.9M 2.9 ms 50.2 (+4.7)
FBNet-96-0.35-2 (ours) 1.9M 13.7M 3.6 ms 51.9 (+6.4)
Table 4. FBNets searched for different input resolution and channel scaling. MnasNet-scale is the MnasNet model with input and channel
size scaling. MnasNet-search-192-0.5 is a model searched with an input size of 192 and channel scaling of 0.5. Details of it are not
disclosed in [20], so we only cite the accuracy.
Model #Parameters #FLOPs Latency on
iPhone X
Latency on
Samsung S8 Top-1 acc (%)
FBNet-iPhoneX 4.47M 322M 19.84 ms (target) 23.33 ms 73.20
FBNet-S8 4.43M 293M 27.53 ms 22.12 ms (target) 73.27
Table 5. FBNets searched for different devices.
0 200 400 600 800 10001200
hw28_c192_k5_s2
hw14_c192_k5_s1
hw14_c336_k5_s1
hw28_c192_k3_s1
hw14_c336_k3_s1
hw14_c384_k3_s1
Comparison	of	Operator	Runtime	(us)
iPhone	XSamsung	S8
Ops	adopted	by	FBNet-iPhoneX
Ops	adopted	in	FBNet-S8
Figure 5. Comparison of operator runtime on two devices. Run-
time is in micro-second (us). Orange bar denotes the runtime on
iPhone X and blue bar denotes the runtime on Samsung S8. The
upper three operators are faster on iPhone X, therefore they are au-
tomatically adopted in FBNet-iPhoneX. The lower three operators
are faster on Samsung S8, and they are also automatically adopted
in FBNet-S8.
S8 uses many blocks with 5x5 depthwise convolution while
FBNet-iPhoneX only uses them in the last two stages. We
examine the depthwise convolution operators used in the
two models and compare their runtime on both devices.
As shown in Figure 5, the upper three operators are faster
on iPhone X, therefore they are automatically adopted in
FBNet-iPhoneX. The lower three operators are signiﬁcantly
faster on Samsung S8, and they are also automatically
adopted in FBNet-S8. Notice the drastic runtime differ-
ences of the lower three operators on two target devices.
It explains why the Samsung-S8-optimized model performs
poorly on an iPhone X. This shows DNAS can automati-
cally optimize the operator adoptions and generate different
ConvNets optimized for different devices.
5. Conclusion
We present DNAS, a differentiable neural architecture
search framework. It optimizes over a layer-wise search
space and represents the search space by a stochastic su-
per net. The actual target device latency of blocks is used to
compute the loss for super net training. FBNets, a family of
models discovered by DNAS surpass state-of-the-art mod-
els, both manually and automatically designed: FBNet-B
achieves 74.1% top-1 accuracy with 295M FLOPs and 23.1
ms latency, 2.4x smaller and 1.5x faster than MobileNetV2-
1.3 with the same accuracy. It also achieves better accuracy
and lower latency than MnasNet, the state-of-the-art efﬁ-
cient model designed automatically; we estimate the search
cost of DNAS is 420x smaller. Such efﬁciency allows us to
conduct searches for different input resolutions and channel
scaling. Discovered models achieve 1.5% to 6.4% accu-
racy gains. The smallest FBNet achieves 50.2% accuracy
with a latency of 2.9 ms (345 frames/sec) with batch size 1.
Over the Samsung-optimized FBNet, the improved FBNet
achieves 1.4x speed up on an iPhone X, showing DNAS is
able to adapt to different target devices automatically.
References
[1] Anonymous. Snas: stochastic neural architecture search. In
Submitted to International Conference on Learning Repre-
sentations, 2019. under review.
[2] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-
Fei. Imagenet: A large-scale hierarchical image database.
In Computer Vision and Pattern Recognition, 2009. CVPR
2009. IEEE Conference on, pages 248–255. Ieee, 2009.
[3] A. Gholami, K. Kwon, B. Wu, Z. Tai, X. Yue, P. Jin, S. Zhao,
and K. Keutzer. Squeezenext: Hardware-aware neural net-
work design. arXiv preprint arXiv:1803.10615, 2018.
[4] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learn-
ing for image recognition. In Proceedings of the IEEE con-
ference on computer vision and pattern recognition , pages
770–778, 2016.
[5] Y . He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han. Amc:
Automl for model compression and acceleration on mobile
devices. In Proceedings of the European Conference on
Computer Vision (ECCV), pages 784–800, 2018.
[6] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang,
T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efﬁ-
cient convolutional neural networks for mobile vision appli-
cations. arXiv preprint arXiv:1704.04861, 2017.
[7] G. Huang, S. Liu, L. van der Maaten, and K. Q. Weinberger.
Condensenet: An efﬁcient densenet using learned group con-
volutions. group, 3(12):11, 2017.
[8] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J.
Dally, and K. Keutzer. Squeezenet: Alexnet-level accuracy
with 50x fewer parameters and¡ 0.5 mb model size. arXiv
preprint arXiv:1602.07360, 2016.
[9] E. Jang, S. Gu, and B. Poole. Categorical reparameterization
with gumbel-softmax. arXiv preprint arXiv:1611.01144 ,
2016.
[10] D. P. Kingma and J. Ba. Adam: A method for stochastic
optimization. arXiv preprint arXiv:1412.6980, 2014.
[11] C. Liu, B. Zoph, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei,
A. Yuille, J. Huang, and K. Murphy. Progressive neural ar-
chitecture search. arXiv preprint arXiv:1712.00559, 2017.
[12] H. Liu, K. Simonyan, and Y . Yang. Darts: Differentiable
architecture search. arXiv preprint arXiv:1806.09055, 2018.
[13] N. Ma, X. Zhang, H.-T. Zheng, and J. Sun. Shufﬂenet
v2: Practical guidelines for efﬁcient cnn architecture design.
arXiv preprint arXiv:1807.11164, 2018.
[14] C. J. Maddison, A. Mnih, and Y . W. Teh. The concrete dis-
tribution: A continuous relaxation of discrete random vari-
ables. arXiv preprint arXiv:1611.00712, 2016.
[15] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. De-
Vito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Auto-
matic differentiation in pytorch. 2017.
[16] H. Pham, M. Y . Guan, B. Zoph, Q. V . Le, and J. Dean. Efﬁ-
cient neural architecture search via parameter sharing. arXiv
preprint arXiv:1802.03268, 2018.
[17] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C.
Chen. Mobilenetv2: Inverted residuals and linear bottle-
necks. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, pages 4510–4520, 2018.
[18] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. arXiv preprint
arXiv:1409.1556, 2014.
[19] C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed,
D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich.
Going deeper with convolutions. In Proceedings of the
IEEE conference on computer vision and pattern recogni-
tion, pages 1–9, 2015.
[20] M. Tan, B. Chen, R. Pang, V . Vasudevan, and Q. V . Le.
Mnasnet: Platform-aware neural architecture search for mo-
bile. arXiv preprint arXiv:1807.11626, 2018.
[21] T. Veniat and L. Denoyer. Learning time/memory-efﬁcient
deep architectures with budgeted super networks. arXiv
preprint arXiv:1706.00046, 2017.
[22] B. Wu, F. N. Iandola, P. H. Jin, and K. Keutzer. Squeezedet:
Uniﬁed, small, low power fully convolutional neural net-
works for real-time object detection for autonomous driving.
In CVPR Workshops, pages 446–454, 2017.
[23] B. Wu, A. Wan, X. Yue, P. Jin, S. Zhao, N. Golmant, A. Gho-
laminejad, J. Gonzalez, and K. Keutzer. Shift: A zero ﬂop,
zero parameter alternative to spatial convolutions. arXiv
preprint arXiv:1711.08141, 2017.
[24] B. Wu, A. Wan, X. Yue, and K. Keutzer. Squeezeseg: Con-
volutional neural nets with recurrent crf for real-time road-
object segmentation from 3d lidar point cloud. In 2018
IEEE International Conference on Robotics and Automation
(ICRA), pages 1887–1893. IEEE, 2018.
[25] B. Wu, Y . Wang, P. Zhang, Y . Tian, P. Vajda, and K. Keutzer.
Mixed precision quantization of convnets via differentiable
neural architecture search. arXiv preprint arXiv:1812.00090,
2018.
[26] B. Wu, X. Zhou, S. Zhao, X. Yue, and K. Keutzer. Squeeze-
segv2: Improved model structure and unsupervised domain
adaptation for road-object segmentation from a lidar point
cloud. arXiv preprint arXiv:1809.08495, 2018.
[27] T.-J. Yang, A. Howard, B. Chen, X. Zhang, A. Go, M. San-
dler, V . Sze, and H. Adam. Netadapt: Platform-aware neural
network adaptation for mobile applications. Energy, 41:46,
2018.
[28] Y . Yang, Q. Huang, B. Wu, T. Zhang, L. Ma, G. Gam-
bardella, M. Blott, L. Lavagno, K. Vissers, J. Wawrzynek,
et al. Synetgy: Algorithm-hardware co-design for con-
vnet accelerators on embedded fpgas. arXiv preprint
arXiv:1811.08634, 2018.
[29] X. Zhang, X. Zhou, M. Lin, and J. Sun. Shufﬂenet: An
extremely efﬁcient convolutional neural network for mobile
devices. arxiv 2017. arXiv preprint arXiv:1707.01083.
[30] B. Zoph and Q. V . Le. Neural architecture search with rein-
forcement learning. arXiv preprint arXiv:1611.01578, 2016.
[31] B. Zoph, V . Vasudevan, J. Shlens, and Q. V . Le. Learn-
ing transferable architectures for scalable image recognition.
arXiv preprint arXiv:1707.07012, 2(6), 2017.
A. Experiment details
We describe more experiment details in this appendix
to facilitate other researchers to reproduce our work. Our
architecture search is divided into two stages. In the ﬁrst
stage, we train the stochastic super net to ﬁnd an opti-
mal architecture distribution. In the second stage, we sam-
ple architectures from the distribution and train them from
scratch.
To train the stochastic super net, we randomly sample
100 classes from the original 1,000 classes of ImageNet.
Training the super net on this smaller proxy dataset is much
faster. We train the stochastic super net for 90 epochs with a
batch size of 192. In each epoch, we ﬁrst train the operator
parameters wa on 80% of the training set using stochastic
gradient descent with momentum. The initial learning rate
is 0.1, and decay following a cosine decaying schedule. The
momentum is 0.9, and weight decay is10−4. Next, we train
the architecture distribution parameter θ on the rest 20% of
the training set with Adam optimizer [10] with a learning
rate of 10−2 and weight decay of 5 ×10−4. The split of
weight and architecture parameter training ensure the archi-
tecture generalize to the validation dataset. To control the
Gumbel Softmax in (8), we use an initial temperature of 5.0
and exponentially anneal it by exp(−0.045) ≈0.956 every
epoch. For the loss function in (2), we set αto 0.2 and β
to 0.6. We use the standard ResNet data augmentation [4]
to process the input images. We found that at the begin-
ning of the training, operators are usually not sufﬁciently
trained, so their contributions to the accuracy are not clear.
However, their computational costs are always signiﬁcantly
different from each other. As a consequence, the super net
may always pick low computational cost operators at the
beginning of the training. To prevent this, we postpone the
training of the architecture parameter θ by 10 epochs to al-
low operator weights to be sufﬁciently trained ﬁrst. At the
end of the super net training, we sample 6 architectures from
the ﬁnal distribution to be trained from scratch.
To train the sampled architectures, the training proto-
cols are different for different models. Here we describe
the training protocol for FBNet- {A, B, C }. These models
have an input resolution of 224, channel size scaling of 1.0.
We train the models with a batch size of 256 on 8 GPUs
for 360 epochs. We set the initial learning rate to be 0.1,
and decay 10x at 90, 180, and 270 epochs. The momentum
is 0.9, weight decay is 4 ×10−5. We use dropout at the
last convolution layer of the network, and the dropout ratio
is 0.2. We use the standard GoogleNet data augmentation
[19] to randomly resize the image during training.

This paper is included in the Proceedings of the 
14th USENIX Symposium on Operating Systems 
Design and Implementation
November 4–6, 2020
978-1-939133-19-9
Open access to the Proceedings of the 
14th USENIX Symposium on Operating 
Systems Design and Implementation 
is sponsored by USENIX
Ansor: Generating High-Performance 
Tensor Programs for Deep Learning
Lianmin Zheng, UC Berkeley; Chengfan Jia, Minmin Sun, and Zhao Wu, Alibaba Group; 
Cody Hao Yu, Amazon Web Services, Inc; Ameer Haj-Ali, UC Berkeley; Yida Wang, 
Amazon Web Services; Jun Yang, Alibaba Group; Danyang Zhuo, UC Berkeley and 
Duke University; Koushik Sen, Joseph E. Gonzalez, and Ion Stoica, UC Berkeley
https://www.usenix.org/conference/osdi20/presentation/zheng
Ansor: Generating High-Performance Tensor Programs for Deep Learning
Lianmin Zheng1, Chengfan Jia2, Minmin Sun2, Zhao Wu2, Cody Hao Yu3 ,
Ameer Haj-Ali1, Yida Wang3, Jun Yang2, Danyang Zhuo1,4 ,
Koushik Sen1, Joseph E. Gonzalez1, Ion Stoica1
1 UC Berkeley,2Alibaba Group,3Amazon Web Services,4 Duke University
Abstract
High-performance tensor programs are crucial to guarantee
efﬁcient execution of deep neural networks. However, obtain-
ing performant tensor programs for different operators on
various hardware platforms is notoriously challenging. Cur-
rently, deep learning systems rely on vendor-provided kernel
libraries or various search strategies to get performant tensor
programs. These approaches either require signiﬁcant engi-
neering effort to develop platform-speciﬁc optimization code
or fall short of ﬁnding high-performance programs due to
restricted search space and ineffective exploration strategy.
We present Ansor, a tensor program generation framework
for deep learning applications. Compared with existing search
strategies, Ansor explores many more optimization combina-
tions by sampling programs from a hierarchical representation
of the search space. Ansor then ﬁne-tunes the sampled pro-
grams with evolutionary search and a learned cost model to
identify the best programs. Ansor can ﬁnd high-performance
programs that are outside the search space of existing state-of-
the-art approaches. In addition, Ansor utilizes a task scheduler
to simultaneously optimize multiple subgraphs in deep neural
networks. We show that Ansor improves the execution perfor-
mance of deep neural networks relative to the state-of-the-art
on the Intel CPU, ARM CPU, and NVIDIA GPU by up to
3.8⇥,2 .6⇥, and 1.7⇥, respectively.
1 Introduction
Low-latency execution of deep neural networks (DNN) plays
a critical role in autonomous driving [14], augmented real-
ity [3], language translation [15], and other applications of
AI. DNNs can be expressed as a directed acyclic compu-
tational graph (DAG), in which nodes represent the opera-
tors (e.g., convolution, matrix multiplication) and directed
edges represent the dependencies between operators. Existing
deep learning frameworks (e.g., Tensorﬂow [1], PyTorch [39],
MXNet [10]) map the operators in DNNs to vendor-provided
kernel libraries (e.g., cuDNN [13], MKL-DNN [27]) to
achieve high performance. However, these kernel libraries
require signiﬁcant engineering effort to manually tune for
each hardware platform and operator. The signiﬁcant manual
effort required to produce efﬁcient operator implementations
for each target accelerator limits the development and innova-
tion of new operators [7] and specialized accelerators [35].
Given the importance of DNNs’ performance, researchers
and industry practitioners have turned to search-based com-
pilation [2, 11, 32, 49, 59] for automated generation oftensor
programs, i.e., low-level implementations of tensor operators.
For an operator or a (sub-)graph of multiple operators, users
deﬁne the computation in a high-level declarative language
(§2 ), and the compiler then searches for programs tailored
towards different hardware platforms.
To ﬁnd performant tensor programs, it is necessary for a
search-based approach to explore a large enough search space
to cover all the useful tensor program optimizations. However,
existing approaches fail to capture many effective optimiza-
tion combinations, because they rely on either predeﬁned
manually-written templates (e.g., TVM [12], FlexTensor [59])
or aggressive pruning by evaluating incomplete programs
(e.g., Halide auto-scheduler [2]), which prevents them from
covering a comprehensive search space (§2 ). The rules they
use to construct the search space are also limited.
In this paper, we explore a novel search strategy for gener-
ating high-performance tensor programs. It can automatically
generate a large search space with comprehensive coverage of
optimizations and gives every tensor program in the space a
chance to be chosen. It thus enables to ﬁnd high-performance
programs that existing approaches miss.
Realizing this goal faces multiple challenges. First, it re-
quires automatically constructing a large search space to cover
as many tensor programs as possible for a given computation
deﬁnition. Second, we need to search efﬁciently without com-
paring incomplete programs in the large search space that can
be orders of magnitude larger than what existing templates
can cover. Finally, when optimizing an entire DNN with many
subgraphs, we should recognize and prioritize the subgraphs
that are critical to the end-to-end performance.
USENIX Association 14th USENIX Symposium on Operating Systems Design and Implementation    863
To this end, we design and implementAnsor, a framework
for automated tensor program generation. Ansor utilizes a
hierarchical representation to cover a large search space. This
representation decouples high-level structures and low-level
details, enabling ﬂexible enumeration of high-level structures
and efﬁcient sampling of low-level details. The space is con-
structed automatically for a given computation deﬁnition.
Ansor then samples complete programs from the search space
and ﬁne-tunes these programs with evolutionary search and a
learned cost model. To optimize the performance of DNNs
with multiple subgraphs, Ansor dynamically prioritizes sub-
graphs of the DNNs that are more likely to improve the end-
to-end performance.
We evaluate Ansor on both standard deep learning bench-
marks and emerging new workloads against manual libraries
and state-of-the-art search-based frameworks. Experiment re-
sults show that Ansor improves the execution performance
of DNNs on the Intel CPU, ARM CPU, and NVIDIA GPU
by up to3.8⇥, 2.6⇥, and1.7⇥, respectively. For most com-
putation deﬁnitions, the best program found by Ansor is out-
side the search space of existing search-based approaches.
The results also show that, compared with existing search-
based approaches, Ansor searches more efﬁciently, generating
higher-performance programs in a shorter time, despite its
larger search space. Ansor can match the performance of a
state-of-the-art framework with an order of magnitude less
search time. Besides, Ansor enables automatic extension to
new operators by only requiring their mathematical deﬁnitions
without manual templates.
In summary, this paper makes the following contributions:
• A mechanism to generate a large hierarchical search
space of tensor programs for a computational graph.
• An evolutionary strategy with a learned cost model to
ﬁne-tune the performance of tensor programs.
• A scheduling algorithm based on gradient descent to
prioritize important subgraphs when optimizing the end-
to-end performance of DNNs.
• An implementation and comprehensive evaluation of the
Ansor system demonstrating that the above techniques
outperform state-of-the-art systems on a variety of DNNs
and hardware platforms.
2 Background
The deep learning ecosystem is embracing a rapidly growing
diversity of hardware platforms including CPUs, GPUs, FP-
GAs, and ASICs. In order to deploy DNNs on these platforms,
high-performance tensor programs are needed for the opera-
tors used in DNNs. The required operator set typically con-
tains a mixture of standard operators (e.g., matmul, conv2d)
and novel operators invented by machine learning researchers
(e.g., capsule conv2d [23], dilated conv2d [57]).
C =compute((N, M), lambdai, j: sum(A[i, k]*B[k, j], [k]))Matrix	Multiplication	 				!",	%=∑(",	)*),	% )
Figure 1: The computation deﬁnition of matrix multiplication.
To deliver portable performance of these operators on a
wide range of hardware platforms in a productive way, multi-
ple compiler techniques have been introduced (e.g., TVM [11],
Halide [41], Tensor Comprehensions [49]). Users deﬁne the
computation in a form similar to mathematical expressions
using a high-level declarative language, and the compiler gen-
erates optimized tensor programs according to the deﬁnition.
Figure 1shows the computation deﬁnition of matrix multipli-
cation in the TVM tensor expression language. Users mainly
need to deﬁne the shapes of the tensors and how each element
in the output tensor is computed.
However, automatically generating high-performance ten-
sor programs from a high-level deﬁnition is extremely dif-
ﬁcult. Depending on the architecture of the target platform,
the compiler needs to search in an extremely large and com-
plicated space containing combinatorial choices of optimiza-
tions (e.g., tile structure, tile size, vectorization, paralleliza-
tion). Finding high-performance programs requires the search
strategy to cover a comprehensive space and explore it efﬁ-
ciently. We describe two recent and effective approaches in
this section and other related work in§8 .
Template-guided search.In template-guided search, the
search space is deﬁned by manual templates. As shown inFig-
ure 2a, the compiler (e.g., TVM) requires the user to manually
write a template for a computation deﬁnition. The template
deﬁnes the structure of the tensor programs with some tunable
parameters (e.g., tile size and unrolling factor). The compiler
then searches for the best values of these parameters for a spe-
ciﬁc input shape conﬁguration and a speciﬁc hardware target.
This approach has achieved good performance on common
deep learning operators. However, developing templates re-
quires substantial effort. For example, the code repository of
TVM already contains more than 15K lines of code for these
templates. This number continues to grow as new operators
and new hardware platforms emerge. Besides, constructing a
quality template requires expertise in both tensor operators
and hardware. It takes non-trivial research effort [32, 55, 59]
to develop quality templates. Despite the complexity of tem-
plate design, manual templates only cover limited program
structures because manually enumerating all optimization
choices for all operators is prohibitive. This approach typi-
cally requires deﬁning one template for each operator. Flex-
Tensor [59] proposes a general template to cover multiple
operators, but its template is still designed for single operator
granularity, which fails to include optimizations involving
multiple operators (e.g., operator fusion). The search space
of optimizing a computational graph with multiple operators
should contain different ways to compose the operators. A
template-based approach fails to achieve this because it can-
not break down their ﬁxed templates and re-compose them
864    14th USENIX Symposium on Operating Systems Design and Implementation USENIX Association
...
(a) T emplate-guided Search
Fixed Manual T emplate
fori.0 in range(  ):for j.0 inrange(  ):for k.0 inrange(  ):fori.1 in range(  ):for j.1 inrange(  ):C[...] +=A[...] * B[...]fori.2 inrange(  ):for j.2 inrange(  ):D[...] =max(C[...], 0.0)
??????
??
(b) Sequential Construction Based Search
...
IncompleteProgram
fori.0 in range(512):for j.0 inrange(512):D[...] =max(C[...], 0.0)
How	to	build	the	next	statement	 ?
Candidate	1
Candidate	2
Candidate	3
Candidate	4
Pruned
Pruned
Kept
Kept Evolutionary fine-tuning
Better	Programs
Low-level detail sampling
...for ...for ...for ...for ... ...for ...for ...for ...for ... ...for ...for ...for ...for ...
(c) Ansor’sHierarchical Approach
High-levelstructure generation
......fori.0 in range(64):for j.0 inrange(64):for k.0 inrange(512):fori.1 in range(8):for j.1 inrange(8):D[...] = ...
CompletePrograms
? ?
?
?
?
Beam Search with Early PruningParameter Serach
Figure 2: Search strategy comparison. The pseudo-code shows tensor programs with loop nests. The question marks in orange
background denote low-level parameters.
during the search.
Sequential construction based search.This approach de-
ﬁnes the search space by decomposing the program construc-
tion into a ﬁxed sequence of decisions. The compiler then
uses an algorithm such as beam search [34] to search for good
decisions (e.g., Halide auto-scheduler [2]). In this approach,
the compiler constructs a tensor program by sequentially un-
folding all nodes in the computational graph. For each node,
the compiler makes a few decisions on how to transform it
into low-level tensor programs (i.e., deciding computation
location, storage location, tile size,etc.). When all nodes are
unfolded, a complete tensor program is constructed. This ap-
proach uses a set of general unfolding rules for every node,
so it can search automatically without requiring manual tem-
plates. Because the number of possible choices of each de-
cision is large, to make the sequential process feasible, this
approach keeps only top-k candidate programs after every de-
cision. The compiler estimates and compares the performance
of candidate programs with a learned cost model to select the
top-k candidates; while other candidates are pruned. During
the search, the candidate programs are incomplete because
only part of the computational graph is unfolded or only some
of the decisions are made.Figure 2bshows this process.
However, estimating the ﬁnal performance of incomplete
programs is difﬁcult in several respects: (1) the cost model
trained on complete programs cannot accurately predict the
ﬁnal performance of incomplete programs. The cost model
can only be trained on complete programs because we need
to compile programs and measure their execution time to
get the labels for training. Directly using this model to com-
pare the ﬁnal performance of incomplete programs will result
in poor accuracy. As a case study, we train our cost model
(§5.2 ) on 20,000 random complete programs from our search
space and use the model to predict the ﬁnal performance of
incomplete programs. The incomplete programs are obtained
by only applying a fraction of loop transformations of the
complete programs. We use two ranking metrics for evalua-
tion: the accuracy of pairwise comparison and the recall@k
Figure 3: Pairwise comparison accuracy and top-k recall curve
on random partial programs. In both subﬁgures, higher values
are better.
score of top-k programs 1 (k = 10). As shown inFigure 3,
the two curves start from50% and 0% respectively, meaning
that random guess with zero information gives50% pairwise
comparison accuracy and0% top-k recall. The two curves
increase quickly as the programs become complete, which
means the cost model performs very well for complete pro-
grams but fails to accurately predict the ﬁnal performance of
incomplete programs. (2) The ﬁxed order of sequential deci-
sions limits the design of the search space. For example, some
optimization needs to add new nodes to the computational
graph (e.g., adding cache nodes, usingrfactor [46]). The
number of decisions for different programs becomes different.
It is hard to align the incomplete programs for a fair compari-
son. (3) Sequential construction based search is not scalable.
Enlarging the search space needs to add more sequential con-
struction steps, which, however, leads to a worse accumulated
error.
Ansor’s hierarchical approachAs shown inFigure 2c,
Ansor is backed by a hierarchical search space that decouples
high-level structures and low-level details. Ansor constructs
the search space for a computational graph automatically,
eliminating the need to manually develop templates. Ansor
then samples complete programs from the space and performs
ﬁne-tuning on complete programs, avoiding the inaccurate es-
timation of incomplete programs.Figure 2shows the key dif-
1recall@k of top-k = |G\P|
k , whereG is the set of top-k programs according
to the ground truth andP is the set of top-k programs predicted by the model.
USENIX Association 14th USENIX Symposium on Operating Systems Design and Implementation    865
ference between Ansor’s approach and existing approaches.
3 Design Overview
Ansor is an automated tensor program generation framework.
Figure 4shows the overall architecture of Ansor. The input
of Ansor is a set of to be optimized DNNs. Ansor uses the
operator fusion algorithm from Relay [42] to convert DNNs
from popular model formats (e.g., ONNX [6], TensorFlow
PB) to partitioned small subgraphs. Ansor then generates
tensor programs for these subgraphs. Ansor has three major
components: (1) a program sampler that constructs a large
search space and samples diverse programs from it; (2) a
performance tuner that ﬁne-tunes the performance of sampled
programs; (3) a task scheduler that allocates time resources
for optimizing multiple subgraphs in the DNNs.
Program sampler.One key challenge Ansor has to ad-
dress is generating a large search space for a given computa-
tional graph. To cover diverse tensor programs with various
high-level structures and low-level details, Ansor utilizes a
hierarchical representation of the search space with two lev-
els: sketch and annotation (§4 ). Ansor deﬁnes the high-level
structures of programs as sketches and leaves billions of low-
level choices (e.g., tile size, parallel, unroll annotations) as
annotations. This representation allows Ansor to enumerate
high-level structures ﬂexibly and sample low-level details ef-
ﬁciently. Ansor includes a program sampler that randomly
samples programs from the space to provide comprehensive
coverage of the search space.
Performance tuner.The performance of randomly sam-
pled programs is not necessarily good. The next challenge
is to ﬁne-tune them. Ansor employs evolutionary search and
a learned cost model to perform ﬁne-tuning iteratively (§5 ).
At each iteration, Ansor uses re-sampled new programs as
well as good programs from previous iterations as the ini-
tial population to start the evolutionary search. Evolutionary
search ﬁne-tunes programs by mutation and crossover which
perform out-of-order rewrite and address the limitation of
sequential construction. Querying the learned cost model is
orders of magnitude faster than actual measurement, so we
can evaluate thousands of programs in seconds.
Task scheduler.Using program sampling and performance
ﬁne-tuning allows Ansor to ﬁnd high-performance tensor pro-
grams for a computational graph. Intuitively, treating a whole
DNN as a single computational graph and generating a full
tensor program for it could potentially achieve the optimal
performance. This, however, is inefﬁcient because it has to
deal with the unnecessary exponential explosion of the search
space. Typically, the compiler partitions the large computa-
tional graph of a DNN into several small subgraphs [11,42].
This partition has a negligible effect on the performance
thanks to the layer-by-layer construction nature of DNNs.
This brings the ﬁnal challenge of Ansor: how to allocate time
resources when generating programs for multiple subgraphs.
Deep	Learning	Models
Subgraph	 1TaskSchedulerSubgraph	 2Subgraph	 3··	·	
Program SamplerSketch	GenerationRandom	Annotation
Performance TunerEvolutionary	SearchLearned	Cost	Model
Intel	CPUMeasurerARM	CPUNVIDIA	GPU··	·	
Section 6
Section 5
Section 4
Partitioned subgraphs
One subgraph 
A batch of initial programs 
A batch of opimizedprograms 
Execution time of programs(training data for future iterations)
Figure 4: System Overview. The gray arrows show the ﬂow
of extracting subgraphs from deep learning models and gen-
erating optimized programs for them. The green arrows mean
the measurer returns proﬁling data to update the status of all
components in the system.
The task scheduler (§6 ) in Ansor uses a scheduling algorithm
based on gradient descent to allocate resources to the sub-
graphs that are more likely to improve the end-to-end DNN
performance.
4 Program Sampling
The search space an algorithm explores determines the best
programs it can ﬁnd. The considered search spaces in existing
approaches are limited by the following factors: (1) Manual
enumeration (e.g., TVM [12]). It is impractical to manually
enumerate all possible choices by templates, so existing man-
ual templates only cover a limited search space heuristically.
(2) Aggressive early pruning (e.g., Halide auto-scheduler [2]).
Aggressive early pruning based on evaluating incomplete pro-
grams prevents the search algorithm from exploring certain
regions in the space.
In this section, we introduce techniques to push the bound-
ary of the considered search space by addressing the above
limitations. To solve (1), we automatically expand the search
space by recursively applying a set of ﬂexible derivation rules.
To avoid (2), we randomly sample complete programs in the
search space. Since random sampling gives an equal chance
to every point to be sampled, our search algorithm can po-
tentially explore every program in the considered space. We
do not rely on random sampling to ﬁnd the optimal program,
because every sampled program is later ﬁned-tuned (§5 ).
To sample programs that can cover a large search space, we
deﬁne a hierarchical search space with two levels:sketch and
annotation. We deﬁne the high-level structures of programs
as sketches and leave billions of low-level choices (e.g., tile
866    14th USENIX Symposium on Operating Systems Design and Implementation USENIX Association
No Rule Name Condition Application
1 Skip ¬ IsStrictInlinable (S, i) S0 = S;i0 = i  1
2 Always Inline IsStrictInlinable (S, i) S0 = Inline(S, i); i0 = i  1
3 Multi-level Tiling HasDataReuse(S, i) S0 = MultiLevelTiling (S, i);i0 = i  1
4 Multi-level Tiling with Fusion HasDataReuse(S, i) ^HasFusibleConsumer (S, i) S0 = FuseConsumer(MultiLevelTiling (S, i), i);i0 = i  1
5 Add Cache Stage HasDataReuse(S, i) ^¬ HasFusibleConsumer (S, i) S0 = AddCacheW rite(S, i);i = i0
6 Reduction Factorization HasMoreReductionParallel (S, i) S0 = AddR f actor(S, i);i0 = i  1
... User Deﬁned Rule ... ...
Table 1: Derivation rules used to generate sketches. The condition runs on the current states =( S, i). The application derives the
next states0 =( S0, i0) from the current states. Note that some function (e.g., AddR f actor, FuseConsumer) can return multiple
possible values ofS0. In this case we collect all possibleS0, and return multiple next statess0 for a single input states.
size, parallel, unroll annotations) as annotations. At the top
level, we generate sketches by recursively applying a few
derivation rules. At the bottom level, we randomly annotate
these sketches to get complete programs. This representation
summarizes a few basic structures from billions of low-level
choices, enabling the ﬂexible enumeration of high-level struc-
tures and efﬁcient sampling of low-level details.
While Ansor supports both CPU and GPU, we explain the
sampling process for CPUs in§4.1 and §4.2 as an example.
We then discuss how the process is different for GPU in§4.3 .
4.1 Sketch Generation
As shown inFigure 4, the program sampler accepts partitioned
subgraphs as input. The ﬁrst column inFigure 5shows two
examples of the input. The input has three equivalent forms:
the mathematical expression, the corresponding naive pro-
gram obtained by directly expanding the loop indices, and the
corresponding computational graph (directed acyclic graph,
or DAG).
To generate sketches for a DAG with multiple nodes, we
visit all the nodes in a topological order and build the structure
iteratively. For computation nodes that are compute-intensive
and have a lot of data reuse opportunities (e.g., conv2d, mat-
mul), we build basic tile and fusion structures for them as the
sketch. For simple element-wise nodes (e.g., ReLU, element-
wise add), we can safely inline them. Note that new nodes
(e.g., caching nodes, layout transform nodes) may also be
introduced to the DAG during the sketch generation.
We propose a derivation-based enumeration approach to
generate all possible sketches by recursively applying several
basic rules. This process takes a DAG as an input and returns
a list of sketches. We deﬁne the States =( S, i), whereS is
the current partially generated sketch for the DAG, andi is the
index of the current working node. The nodes in a DAG are
sorted in a topological order from output to input. The deriva-
tion begins from the initial naive program and the last node, or
the initial states =( naive program, index o f the last node).
Then we try to apply all derivation rules to the states re-
cursively. For each rule, if the current state satisﬁes the ap-
plication condition, we apply the rule tos =( S, i) and get
s0 =( S0, i0) where i0  i. This way the indexi (working node)
decreases monotonically. A state becomes a terminal state
when i = 0. During enumeration, multiple rules can be ap-
plied to one state to generate multiple succeeding states. One
rule can also generate multiple possible succeeding states.
So we maintain a queue to store all intermediate states. The
process ends when the queue is empty. Alls.S in terminal
states form a sketch list at the end of the sketch generation.
The number of sketches is less than 10 for a typical subgraph.
Derivation rules.Table 1lists derivation rules we used
for the CPU. We ﬁrst provide the deﬁnition of the used
predicates and then describe the functionality of each rule.
IsStrictInliable (S, i) indicates if the nodei in S is a sim-
ple element-wise operator that can always be inlined (e.g.,
element-wise add, ReLU).HasDataReuse(S, i) indicates if
the node i in S is a compute-intensive operator and has
plentiful intra-operator data reuse opportunity (e.g., mat-
mul, conv2d). HasFusibleConsumer (S, i) indicates if the
node i in S has only one consumer j and node j can be
fused into nodei (e.g., matmul + bias_add, conv2d + relu).
HasMoreReductionParallel (S, i) indicates if the nodei in S
has little parallelism in space dimensions but has ample paral-
lelism opportunity in reduction dimensions. (e.g., computing
2-norm of a matrix, matmulC2⇥2 = A2⇥512 · B512⇥2). We per-
form static analysis on the computation deﬁnitions to get the
values for these predicates. The analysis is done automatically
by parsing the read/write pattern in the mathematical expres-
sions. Next, we introduce the functionality of each derivation
rule.
Rule 1 just simply skips a node if it is not strictly inlinable.
Rule 2 always inlines strictly inlinable nodes. Since the condi-
tions of rule 1 and rule 2 are mutually exclusive, a state with
i > 1 can always satisfy one of them and continue to derive.
Rules 3, 4, and 5 deal with the multi-level tiling and fusion
for nodes that have data reuse. Rule 3 performs multi-level
tiling for data reusable nodes. For CPU, we use a “SSRSRS”
tile structure, where “S” stands for one tile level of space
loops and “R” stands for one tile level of reduction loops.
For example, in the matmulC(i, j)= Âk A[i, k] ⇥B[k, j], i and
j are space loops andk is a reduction loop. The “SSRSRS”
tile structure for matmul expands the original 3-level loop
(i, j, k) into a 10-level loop(i0, j0, i1, j1, k0, i2, j2, k1, i3, j3).
Although we do not permute the loop order, this multi-level
USENIX Association 14th USENIX Symposium on Operating Systems Design and Implementation    867
tiling can also cover some cases of reordering. For example,
the above 10-level loop can be specialized to just a simple
reorder (k0, j2, i3) by setting the length of other loops to one.
The "SSRSRS" tile structure is general for compute-intensive
dense operators (e.g., matmul, conv2d, conv3d) in deep learn-
ing, because they all consist of space loops and reduction
loops.
Rule 4 performs multi-level tiling and also fuses the fusible
consumers. For example, we fuse the element-wise nodes
(e.g., ReLU, bias add) into the tiled nodes (e.g., conv2d, mat-
mul). Rule 5 adds a caching node if the current data-reusable
node does not have a fusible consumer. For example, the ﬁ-
nal output node in a DAG does not have any consumer, so it
directly writes results into main memory by default and this
is inefﬁcient due to the high latency of memory accesses. By
adding a cache node, we introduce a new fusible consumer
into the DAG, then rule 4 can be applied to fuse this newly
added cache node into the ﬁnal output node. With the cache
node fused, now the ﬁnal output node writes its results into a
cache block, and the cache block will be written to the main
memory at once when all data in the block is computed.
Rule 6 can userfactor [46] to factorize a reduction loop
into a space loop to bring more parallelism.
Examples. Figure 5shows three examples of the gener-
ated sketches. The sketches are different from the manual
templates in TVM, because the manual templates specify
both high-level structures and low-level details while sketches
only deﬁne high-level structures. For the example input 1, the
sorted order of the four nodes in the DAG is(A, B,C, D).T o
derive the sketches for the DAG, we start from output node
D(i = 4) and apply rules to the nodes one by one. Speciﬁcally,
the derivation for generated sketch 1 is:
Input 1 !s(S0, i = 4) Rule 1   !s(S1, i = 3) Rule 4   !
s(S2, i = 2) Rule 1   !s(S3, i = 1) Rule 1   !Sketch 1
For the example input 2, the sorted order of the ﬁve nodes
is (A, B,C, D, E). Similarly, we start from the output node
E(i = 5) and apply rules recursively. The generated sketch 2
is derived by:
Input 2 !s(S0, i = 5) Rule 5   !s(S1, i = 5) Rule 4   !
s(S2, i = 4) Rule 1   !s(S3, i = 3) Rule 1   !
s(S4, i = 2) Rule 2   !s(S5, i = 1) Rule 1   !Sketch 2
Similarly, the generated sketch 3 is derived by:
Input 2 !s(S0, i = 5) Rule 6   !s(S1, i = 4) Rule 1   !
s(S2, i = 3) Rule 1   !s(S3, i = 2) Rule 2   !
s(S4, i = 1) Rule 1   !Sketch 3
Customization. While the presented rules are practical
enough to cover the structures for most operators, there are al-
ways exceptions. For example, some special algorithms (e.g.,
Winograd convolution [30]) and accelerator intrinsics (e.g.,
TensorCore [37]) require special tile structures to be effec-
tive. Although the template-guided search approach (in TVM)
can craft a new template for every new case, it needs a great
amount of design effort. On the other hand, the derivation-
based sketch generation in Ansor is ﬂexible enough to gen-
erate the required structures for emerging algorithms and
hardware, as we allow users to register new derivation rules
and integrate them seamlessly with existing rules.
4.2 Random Annotation
The sketches generated by the previous subsection are incom-
plete programs because they only have tile structures without
speciﬁc tile sizes and loop annotations, such as parallel, unroll,
and vectorization. In this subsection, we annotate sketches to
make them complete programs for ﬁne-tuning and evaluation.
Given a list of generated sketches, we randomly pick one
sketch, randomly ﬁll out tile sizes, parallelize some outer
loops, vectorize some inner loops, and unroll a few inner
loops. We also randomly change the computation location
of some nodes in the program to make a slight tweak to
the tile structure. All “random” in this subsection means a
uniform distribution over all valid values. If some special
algorithms require custom annotations to be effective (e.g.,
special unrolling), we allow users to give simple hints in the
computation deﬁnition to adjust the annotation policy. Finally,
since changing the layout of constant tensors can be done in
compilation time and brings no runtime overhead, we rewrite
the layouts of the constant tensors according to the multi-level
tile structure to make them as cache-friendly as possible. This
optimization is effective because the weight tensors of convo-
lution or dense layers are constants for inference applications.
Examples of random sampling are shown inFigure 5. The
sampled program might have fewer loops than the sketch
because the loops with length one are simpliﬁed.
4.3 GPU Support
For GPU, we change the multi-level tiling structure from
"SSRSRS" to "SSSRRSRS" to match the architecture of GPU.
The loops in the ﬁrst three space tiles are bound to BlockIdx,
virtual thread (for reducing bank conﬂicts), and ThreadIdx,
respectively. We add two sketch derivation rules, one for uti-
lizing shared memory by inserting a caching node (similar to
Rule 5) and the other for cross-thread reduction (similar to
Rule 6).
5 Performance Fine-tuning
The programs sampled by the program sampler have good cov-
erage of the search space, but their qualities are not guaranteed.
This is because the optimization choices, such as tile struc-
ture and loop annotations, are all randomly sampled. In this
868    14th USENIX Symposium on Operating Systems Design and Implementation USENIX Association
* The mathmeticalexpression:!",$=&'[",)] 
, 	×	/[),$]0",$=max	(!",$,0.0)where0≤",$,)<512* The corresponding naive program:foriin range(512):for j inrange(512):for kinrange(512):C[i, j] +=A[i, k] * B[k, j]foriinrange(512):for jinrange(512):D[i, j] =max(C[i, j], 0.0)* The corresponding DAG:
Example Input 1: paralleli.0@j.0@i.1@j.1 in range(256):for k.0 inrange(32):fori.2 in range(16):unroll k.1 inrange(16):unrolli.3 in range(4):vectorizej.3 inrange(16):C[...] +=A[...] * B[...]fori.4 inrange(64):vectorizej.4 inrange(16):D[...] =max(C[...], 0.0)
Sampled program 1  
paralleli.2 in range(16):forj.2 in range(128):for k.1 inrange(512):for i.3 in range(32):vectorizej.3 inrange(4):C[...] +=A[...] * B[...]paralleli.4 inrange(512):for j.4 inrange(512):D[...] =max(C[...], 0.0)
Sampled program 2
fori.0 in range(TILE_I0):for j.0 inrange(TILE_J0):fori.1 in range(TILE_I1):for j.1 inrange(TILE_J1):for k.0 inrange(TILE_K0):fori.2 in range(TILE_I2):for j.2 inrange(TILE_J2):for k.1 inrange(TILE_I1):fori.3 in range(TILE_I3):for j.3 inrange(TILE_J3):C[...] +=A[...] * B[...]fori.4 inrange(TILE_I2 * TILE_I3):for j.4 inrange(TILE_J2 * TILE_J3):D[...] =max(C[...], 0.0)
Generated sketch 1
foriinrange(8):for k inrange(512):C[i, k] =max(A[i, k],0.0) ifk < 400 else0foriin range(8):forj in range(4):fork_oin range(TILE_K0):fork_iin range(TILE_KI):E.rf[...] +=C[...] * D[...]foriin range(8):forj in range(4):fork_iin range(TILE_KI):E[...] +=E.rf[...]
Generated sketch 3
paralleliin range(8):for k inrange(512):C[i, k] =...forj in range(4):unrollk_oin range(32):vectorizedk_iin range(16):E.rf[...] +=C[...] * D[...]paralleliin range(8):forj in range(4):unrollk_iin range(16):E[...] +=E.rf[...]
Sampled  program 4  
* The mathmeticalexpression:/",==max	('",=,0.0)![",)]=>	/[",)], )<400							0			, )≥400A",$=&![",)] 
, 	×	0[),$]
where0≤"<8,0≤$<4,		0≤)<512,	0≤=<400* The corresponding naive program:foriinrange(8):for linrange(400):B[i, l] =max(A[i, l], 0.0)foriinrange(8):for k inrange(512):C[i, k] =B[i, k] ifk < 400 else0foriin range(8):for j inrange(4):for k inrange(512):E[i, j] +=C[i, k] * D[k, j]
* The corresponding DAG:
Example Input 2:
paralleli.0 in range(8):for k inrange(512):C[i, j] =max(A[i,k],0.0) ifk < 400 else0for k.0 inrange(512):vectorizej.3 inrange(4):E.cache[...] +=C[...] * D[...]vectorizej.4 inrange(4):E[...] =E.cache[...]
Sampled program 3
foriinrange(8):for k inrange(512):C[i, j] =max(A[i,k],0.0) ifk<400 else0fori.0 in range(TILE_I0):for j.0 inrange(TILE_J0):fori.1 in range(TILE_I1):for j.1 inrange(TILE_J1):for k.0 inrange(TILE_K0):fori.2 in range(TILE_I2):for j.2 inrange(TILE_J2):for k.1 inrange(TILE_I1):fori.3 in range(TILE_I3):for j.3 inrange(TILE_J3):E.cache[...] +=C[...] * D[...]fori.4 inrange(TILE_I2 * TILE_I3):for j.4 inrange(TILE_J2 * TILE_J3):E[...] =E.cache[...]
Generated sketch 2AB C D
A DB C E
Figure 5: Examples of generated sketches and sampled programs. This ﬁgure shows two example inputs, three generated sketches
and four sampled programs. The code example is pseudo code in a python-like syntax.
section, we introduce the performance tuner that ﬁne-tunes
the performance of the sampled programs via evolutionary
search and a learned cost model.
The ﬁne-tuning is performed iteratively. At each iteration,
we ﬁrst use evolutionary search to ﬁnd a small batch of promis-
ing programs according to a learned cost model. We then mea-
sure these programs on hardware to get the actual execution
time cost. Finally, the proﬁling data got from measurement is
used to re-train the cost model to make it more accurate.
The evolutionary search uses randomly sampled programs
as well as high-quality programs from the previous mea-
surement as the initial population and applies mutation and
crossover to generate the next generation. The learned cost
model is used to predict theﬁtness of each program, which is
the throughput of one program in our case. We run evolution
for a ﬁxed number of generations and pick the best programs
found during the search. We leverage a learned cost model be-
cause the cost model can give relatively accurate estimations
of the ﬁtness of programs while being orders of magnitudes
faster than the actual measurement. It allows us to compare
tens of thousands of programs in the search space in seconds,
and pick the promising ones to do actual measurements.
5.1 Evolutionary Search
Evolutionary search [54] is a generic meta-heuristic algorithm
inspired by biological evolution. By iteratively mutating high-
quality programs, we can generate new programs with poten-
tially higher quality. The evolution starts from the sampled
initial generation. To generate the next generation, we ﬁrst se-
lect some programs from the current generation according to
certain probabilities. The probability of selecting a program is
proportional to its ﬁtness predicted by the learned cost model
(§5.2 ), meaning that the program with a higher performance
score has a higher probability to be selected. For the selected
programs, we randomly apply one of the evolution operations
to generate a new program. Basically, for decisions we made
during sampling (§4.2 ), we design corresponding evolution
USENIX Association 14th USENIX Symposium on Operating Systems Design and Implementation    869
operations to rewrite and ﬁne-tune them.
Tile size mutation.This operation scans the program and
randomly selects a tiled loop. For this tiled loop, it divides a
tile size of one tile level by a random factor and multiplies this
factor to another level. Since this operation keeps the product
of tile sizes equal to the original loop length, the mutated
program is always valid.
Parallel mutation.This operation scans the program and
randomly selects a loop that has been annotated with parallel.
For this loop, this operation changes the parallel granularity
by either fusing its adjacent loop levels or splitting it by a
factor.
Pragma mutation.Some optimizations in a program are
speciﬁed by compiler-speciﬁc pragma. This operation scans
the program and randomly selects a pragma. For this pragma,
this operation randomly mutates it into another valid value.
For example, our underlying code generator supports auto
unrolling with a maximum number of steps by providing an
auto_unroll_max_step=N pragma. We randomly tweak the
number N.
Computation location mutation.This operation scans the
program and randomly selects a ﬂexible node that is not multi-
level tiled (e.g., a padding node in the convolution layer). For
this node, the operation randomly changes its computation
location to another valid attach point.
Node-based crossover.Crossover is an operation to gener-
ate new offspring by combining the genes from two or more
parents. The genes of a program in Ansor are its rewriting
steps. Every program generated by Ansor is rewritten from
its initial naive implementation. Ansor preserves a complete
rewriting history for each program during sketch generation
and random annotation. We can treat rewriting steps as the
genes of a program because they describe how this program
is formed from the initial naive one. Based on this, we can
generate a new program by combining the rewriting steps
of two existing programs. However, arbitrarily combining
rewriting steps from two programs might break the depen-
dencies in steps and create an invalid program. As a result,
the granularity of crossover operation in Ansor is based on
nodes in the DAG, because the rewriting steps across different
nodes usually have less dependency. Ansor randomly selects
one parent for each node and merges the rewriting steps of
selected nodes. When there are dependencies between nodes,
Ansor tries to analyze and adjust the steps with simple heuris-
tics. Ansor further veriﬁes the merged programs to guarantee
the functional correctness. The veriﬁcation is simple because
Ansor only uses a small set of loop transformation rewrit-
ing steps, and the underlying code generator can check the
correctness by dependency analysis.
The evolutionary search leverages mutation and crossover
to generate a new set of candidates repeatedly for several
rounds and outputs a small set of programs with the highest
scores. These programs will be compiled and measured on the
target hardware to obtain the real running time cost. The col-
lected measurement data is then used to update the cost model.
In this way, the accuracy of the learned cost model is grad-
ually improved to match the target hardware. Consequently,
the evolutionary search gradually generates higher-quality
programs for the target hardware platform.
Unlike the search algorithms in TVM and FlexTensor that
can only work in a ﬁxed grid-like parameter space, the evolu-
tionary operations in Ansor are speciﬁcally designed for ten-
sor programs. They can be applied to general tensor programs
and can handle a search space with complicated dependency.
Unlike the unfolding rules in Halide auto-scheduler, these op-
erations can perform out-of-order modiﬁcations to programs,
addressing the sequential limitations.
5.2 Learned Cost Model
A cost model is necessary for estimating the performance of
programs quickly during the search. We adopt a learned cost
model similar to related works [2, 12] with newly designed
program features. A system based on learned cost models
has great portability because a single model design can be
reused for different hardware backends by feeding in different
training data.
Since our target programs are mainly data parallel tensor
programs, which are made by multiple interleaved loop nests
with several assignment statements as the innermost state-
ments, we train the cost model to predict the score of one in-
nermost non-loop statement in a loop nest. For a full program,
we make predictions for each innermost non-loop statement
and add the predictions up as the score. We build the feature
vector for an innermost non-loop statement by extracting fea-
tures in the context of a full program. The extracted features
include arithmetic features and memory access features. A
detailed list of extracted features is in an appendix of the
extended version of this paper [58].
We use weighted squared error as the loss function. Be-
cause we mostly care about identifying the well-performing
programs from the search space, we put more weight on
the programs that run faster. Speciﬁcally, the loss func-
tion of the modelf on a programP with throughput y is
loss( f , P, y)= wp(Âs2S(P) f (s)  y)2 = y(Âs2S(P) f (s)  y)2
where S(P) is the set of innermost non-loop statements in
P. We directly use the throughputy as weight. We train a
gradient boosting decision tree [9] as the underlying model
f . A single model is trained for all tensor programs coming
from all DAGs, and we normalize the throughput of all pro-
grams coming from the same DAG to be in the range of[0, 1].
When optimizing a DNN, the number of measured programs
are typically less than 30,000. Training a gradient boosting
decision tree is very fast on such a small data sets, so we train
a new model every time instead of doing incremental updates.
870    14th USENIX Symposium on Operating Systems Design and Implementation USENIX Association
6 Task Scheduler
A DNN can be partitioned into many independent subgraphs
(e.g., conv2d + relu). For some subgraphs, spending time in
tuning them does not improve the end-to-end DNN perfor-
mance signiﬁcantly. This is due to two reasons: either (1) the
subgraph is not a performance bottleneck, or (2) tuning brings
only minimal improvement in the subgraph’s performance.
To avoid wasting time on tuning unimportant subgraphs,
Ansor dynamically allocates different amounts of time re-
sources to different subgraphs. Take ResNet-50 for example, it
has 29 unique subgraphs after the graph partitioning. Most of
these subgraphs are convolution layers with different shapes
conﬁgurations (input size, kernel size, stride, etc). We need
to generate different programs for different convolution lay-
ers because the best tensor program depends on these shape
conﬁgurations. In reality, users may have multiple DNNs for
all their applications. This leads to more subgraphs as well as
more opportunities to reduce the total tuning time, because
we can share and reuse knowledge between subgraphs. A
subgraph can also appear multiple times in a DNN or across
different DNNs.
We deﬁne a task as a process performed to generate high-
performance programs for a subgraph. It means that optimiz-
ing a single DNN requires ﬁnishing dozens of tasks (e.g., 29
tasks for ResNet-50). Ansor’s task scheduler allocates time
resources to tasks in an iterative manner. At each iteration,
Ansor selects a task, generates a batch of promising programs
for the subgraph, and measures the program on hardware. We
deﬁne such an iteration as one unit of time resources. When
we allocate one unit of time resources to a task, the task ob-
tains an opportunity to generate and measure new programs,
which also means the chance to ﬁnd better programs. We next
present the formulation of the scheduling problem and our
solution.
6.1 Problem Formulation
When tuning a DNN or a set of DNNs, a user can have various
types of goals, for example, reducing a DNN’s latency, meet-
ing latency requirements for a set of DNNs, or minimizing
tuning time when tuning no longer improves DNN perfor-
mance signiﬁcantly. We thus provide users a set of objective
functions to express their goals. Users can also provide their
own objective functions.
Suppose there aren tasks in total. Lett 2 Zn be the allo-
cation vector, whereti is the number of time units spent on
task i. Let the minimum subgraph latency taski achieves be
a function of the allocation vectorgi(t). Let the end-to-end
cost of the DNNs be a function of the latency of the sub-
graphs f (g1(t), g2(t),. . . ,g3(t)). Our objective is to minimize
the end-to-end cost:
minimize f (g1(t), g2(t),. . . ,g3(t))
f1 = Âm
j=1 Âi2S( j) wi ⇥gi(t)
f2 = Âm
j=1 max(Âi2S( j) wi ⇥gi(t), Lj)
f3 =  (’m
j=1
Bj
Âi2S( j) wi⇥gi(t) )
1
m
f4 = Âm
j=1 Âi2S( j) wi ⇥max(gi(t), ES(gi,t))
Table 2: Examples of objective functions for multiple neural
networks
To minimize the end-to-end latency of a single DNN, we
can deﬁne f (g1, g2,. . . ,gn)= Ân
i=1 wi ⇥gi, where wi is the
number of appearances of taski in the DNN. This formu-
lation is straightforward becausef is an approximation of the
end-to-end DNN latency.
When tuning a set of DNNs, there are several options.Ta-
ble 2 shows a number of example objective functions for
tuning multiple DNNs. Letm be the number of DNNs,S( j) is
the set of tasks that belong to DNNj. f1 adds up the latency
of every DNN, which means to optimize the cost of a pipeline
that sequentially runs all DNNs once. Inf2, we deﬁneLj as
the latency requirement of DNNj, meaning that we do not
want to spend time on a DNN if its latency has already met
the requirement. Inf3, we deﬁneBj as the reference latency
of a DNNj. As a result, our goal is to maximize the geo-
metric mean of speedup against the given reference latency.
Finally in f4, we deﬁne a functionES(gi,t) that returns an
early stopping value by looking at the history of latency of
task i. It can achieve the effect of per-task early stopping.
6.2 Optimizing with Gradient Descent
We propose a scheduling algorithm based on gradient descent
to efﬁciently optimize the objective function. Given the cur-
rent allocationt, the idea is to approximate the gradient of the
objective function∂f
∂ti
in order to choose the taski such that
i = argmaxi | ∂f
∂ti
|. We approximate the gradient by making an
optimistic guess and considering the similarity between tasks.
The derivation is in an appendix of the extended version of
this paper [58]. We approximate the gradient by
∂f
∂ti
⇡ ∂f
∂gi
(agi(ti)  gi(ti  Dt)
Dt +
(1  a)(min( gi(ti)
ti
, b Ci
maxk2N(i) Vk
 gi(ti))))
where Dt is a small backward window size,gi(ti) and gi(ti  
Dt) are known from the history of allocations.N(i) is the
set of similar tasks ofi, Ci is the number of ﬂoating point
operation in taski and Vk is the number of ﬂoating point
operation per second we can achieve in taskk. The parameter
a and b control the weight to trust some predictions.
To run the algorithm, Ansor starts fromt = 0 and warms
up with a round of round-robin to get an initial allocation
vector t =( 1, 1,. . . ,1). After the warm-up, at each iteration, we
USENIX Association 14th USENIX Symposium on Operating Systems Design and Implementation    871
compute the gradient of each task and pickargmaxi | ∂f
∂ti
|. Then
we allocate the resource unit to taski and update the allocation
vector ti = ti + 1. The optimization process continues until
we run out of the time budget. To encourage exploration, we
adopt ae-greedy strategy [47], which preserves a probability
of e to randomly select a task.
Taking the case of optimizing for a single DNN’s end-to-
end latency for example, Ansor prioritizes a subgraph that has
a high initial latency because our optimistic guess says we
can reduce its latency quickly. Later, if Ansor spends many
iterations on it without observing a decrease in its latency,
Ansor leaves the subgraph because its| ∂f
∂ti
| decreases.
7 Evaluation
The core of Ansor is implemented in C++ with about 12K
lines of code (3K for the search policy and 9K for other infras-
tructure). Ansor generates programs in its own intermediate
representation (IR). These programs are then lowered to TVM
IR for code generation targeting various hardware platforms.
Ansor only utilizes TVM as a deterministic code generator.
We evaluate the performance of generated programs on
three levels: single operator, subgraph, and entire neural net-
work. For each level of evaluation, we compare Ansor against
the state-of-the-art search frameworks and hardware-speciﬁc
manual libraries. We also evaluate the search efﬁciency and
the effectiveness of each component in Ansor.
The generated tensor programs are benchmarked on
three hardware platforms: an Intel CPU (18-core Platinum
8124M@3.0 GHz), an NVIDIA GPU (V100), and an ARM
CPU (4-core Cortex-A53@1.4GHz on the Raspberry Pi 3b+).
We use ﬂoat32 as the data type for all evaluations.
7.1 Single Operator Benchmark
Workloads. We ﬁrst evaluate Ansor on a set of common
deep learning operators, including 1D, 2D, and 3D convolu-
tion (C1D, C2D, and C3D respectively), matrix multiplica-
tion (GMM), group convolution (GRP), dilated convolution
(DIL) [57], depth-wise convolution (DEP) [24], transposed 2D
convolution (T2D) [40], capsule 2D convolution (CAP) [23],
and matrix 2-norm (NRM). For each operator, we select 4
common shape conﬁgurations and evaluate them with two
batch sizes (1 and 16). In total, there are10 operators ⇥4
shape conﬁgurations⇥2 batch size= 80 test cases. The shape
conﬁgurations used can be found in an appendix of the ex-
tended version of this paper [58]. We run these test cases on
the Intel CPU.
Baselines. We include PyTorch (v1.5) [39], Halide auto-
scheduler (commit: 1f875b0) [ 2], FlexTensor (commit:
7ac302c) [59], and AutoTVM (commit: 69313a7) [12] as
baselines. PyTorch is backed by the vendor-provided kernel
library MKL-DNN [27]. Halide auto-scheduler is a sequential
construction based search framework for Halide. AutoTVM
and FlexTensor are template-guided search frameworks based
on TVM. Since Halide auto-scheduler does not have a pre-
trained cost model for A VX-512, we disabled A VX-512 for
the evaluation in§7.1 and §7.2 . For every operator, we use
the best layout available in each framework, but the input and
output tensors must not be packed.
Search settings.We let search frameworks (i.e., Halide
auto-scheduler, FlexTensor, AutoTVM, and Ansor) to run
search or auto-tuning with up to1, 000 measurement trials
per test case. This means each framework can measure at
most 80⇥1, 000 programs for auto-tuning in this evaluation.
Using the same number of measurement trials makes it a fair
comparison without involving implementation details. In addi-
tion, using1, 000 measurement trials per test case is typically
enough for the search to converge in these frameworks.
Normalization. Figure 6 shows the normalized perfor-
mance. For each test case, we normalize the throughputs to
the best performing framework. We then plot the geometric
mean of the four shapes of each operator. The geometric mean
is also normalized to the best performing framework, so the
best framework has a normalized performance of 1 in the
ﬁgure. The error bar denotes the standard deviation of the
normalized throughput of four shapes of each operator.
Results. As shown in theFigure 6, Ansor performs the
best or equally the best in all operator and batch size set-
tings. Ansor outperforms existing search frameworks by
1.1  22.5⇥. The performance improvements of Ansor come
from both its large search space and effective exploration strat-
egy. For most operators, we found the best program generated
by Ansor is outside the search space of existing search frame-
works because Ansor is able to explore more optimization
combinations. For example, the signiﬁcant speedup on NRM
is because Ansor can parallelize reduction loops, while other
frameworks do not. The large speedup on T2D is because
Ansor can use correct tile structures and unrolling strategies to
let the code generator simplify the multiplication of zeros in
strided transposed convolution. In contrast, other frameworks
fail to capture many effective optimizations in their search
space, making them unable to ﬁnd the programs that Ansor
does. For example, the unfolding rules in Halide do not split
the reduction loop in GMM and do not split reduction loops
in C2D when padding is computed outside of reduction loops.
The templates in AutoTVM have limited tile structures, as
they cannot cover the structure of “Generated Sketch 1” in
Figure 5. The template in FlexTensor does not change the
computation location of padding. The template in FlexTensor
fails to run for reduction operators like NRM.
Ablation study.We run four variants of Ansor on a convo-
lution operator and report the performance curve. We pick the
last convolution operator in ResNet-50 with batch size=16 as
the test case, because its search space is sufﬁciently large to
evaluate the search algorithms. Other operators share a sim-
ilar pattern. InFigure 7, each curve is the median of 5 runs.
“Ansor (ours)” uses all our introduced techniques. “Beam
872    14th USENIX Symposium on Operating Systems Design and Implementation USENIX Association
Figure 6: Single operator performance benchmark on a 20-
core Intel-Platinum-8269CY . The y-axis is the throughput
normalized to the best throughput for each operator.
Figure 7: Ablation study of four variants of Ansor on a con-
volution operator. The y-axis is the throughput relative to the
throughput of the best program.
Search” means we prune incomplete programs with the cost
model during the sampling process and do not use ﬁne-tuning.
“No ﬁne-tuning” is based on “Ansor (ours)” but disables ﬁne-
tuning and only relies on random sampling. “Limited space”
is also based on “Ansor (ours)” but limits the search space
to make it similar to the space in existing manual templates
(e.g., limit tiling level, innermost tile sizes, and computation
location). As demonstrated byFigure 7, dropping either the
large search space or efﬁcient ﬁne-tuning decreases the ﬁnal
performance signiﬁcantly. The aggressive early pruning in
“Beam search” throws away incomplete programs with good
ﬁnal performance due to inaccurate estimation.
7.2 Subgraph Benchmark
We perform the subgraph benchmark on two common sub-
graphs in DNNs. The “ConvLayer” is a subgraph consisting
of 2D convolution, batch normalization [28], and ReLU ac-
tivation, which is a common pattern in convolutional neural
networks. The “TBS” is a subgraph consisting of two matrix
transposes, one batch matrix multiplication, and a softmax,
which is a pattern in the multi-head attention [51] in language
models. Similar to the single operator benchmark (§7.1 ), we
select four different shape conﬁgurations and two batch sizes,
run auto-tuning with up to1, 000 measurement trails per test
case, and report the normalized performance. We use the
Figure 8: Subgraph performance benchmark on a 20-core
Intel-Platinum-8269CY and an NVIDIA V100. "@C" denotes
CPU results and "@G" denotes GPU results. The y-axis is
the throughput normalized to the best throughput for each
subgraph.
same set of baseline frameworks and run the benchmark on
the Intel CPU and the NVIDIA GPU. We do not report the
performance of Halide auto-scheduler on GPU because as of
writing the paper its GPU support is still in an experimental
stage. FlexTensor fails to run on complicated subgraphs like
“TBS”.
Figure 8shows that Ansor outperforms manual libraries
and other search frameworks by1.1  14.2⇥. Ansor can gen-
erate high-performance programs consistently for these sub-
graphs on both platforms. FlexTensor performs well for single
operators but shows less advantage for subgraphs because it
lacks the support of operator fusion.
7.3 End-to-End Network Benchmark
Workloads. We benchmark the end-to-end inference execu-
tion time of several DNNs, which include ResNet-50 [22]
and MobileNet-V2 [43] for image classiﬁcation, 3D-ResNet-
18 [21] for action recognition, DCGAN [40] generator for
image generation, and BERT [15] for language understanding.
We benchmark these DNNs on three hardware platforms. For
the server-class Intel CPU and NVIDIA GPU, we report the
results for batch size 1 and batch size 16. For the ARM CPU
in the edge device, real-time feedback is typically desired, so
we only report the results for batch size 1.
Baselines and Settings.We include PyTorch (v1.5 with
torch script), TensorFlow (v2.0 with graph mode), TensorRT
(v6.0 with TensorFlow integration) [38], TensorFlow Lite
(V2.0), and AutoTVM as baseline frameworks. We do not in-
clude Halide auto-scheduler or FlexTensor because they lack
the support of widely-used deep learning model formats (e.g.,
ONNX, TensorFlow PB) and high-level graph optimizations.
As a result, we expect that the end-to-end execution time they
can achieve will be the sum of the latency of all subgraphs in
a DNN. In contract, AutoTVM can optimize a whole DNN
with its manual templates and various graph-level optimiza-
tions (e.g., graph-level layout search [32], graph-level constant
USENIX Association 14th USENIX Symposium on Operating Systems Design and Implementation    873
(a) Intel CPU
(b) NVIDIA GPU
(c) ARM CPU
Figure 9: Network inference performance benchmark on three
hardware platforms. The y-axis is the throughput relative to
the best throughput for each network.
folding [42]) which improve the performance signiﬁcantly.
Ansor also performs layout rewrite as described in§4.2 .W e
let both AutoTVM and Ansor run auto-tuning until they use
to 1000 ⇥n measurement trials on each DNN, wheren is the
number of subgraphs in the DNN. This is typically enough for
them to converge. We set the objective of the task scheduler
as minimizing the total latency of one DNN and generate
programs for these networks one by one. On the other hand,
PyTorch, TensorFlow, TensorRT, and TensorFlow Lite are all
backed by static kernel libraries (MKL-DNN on Intel CPU,
CuDNN on NVIDIA GPU, and Eigen on ARM CPU) and do
not need auto-tuning. We enable A VX-512 for all frameworks
on the Intel CPU in this network benchmark.
Results. Figure 9 shows the results on the Intel CPU,
Figure 10: Network performance auto-tuning curve. The y-
axis is the speedup relative to AutoTVM.
NVIDIA GPU and ARM CPU2. Overall, Ansor performs the
best or equally the best in all cases. Compared with search-
based AutoTVM, Ansor matches or outperforms it in all cases
with 1.0  21.8⇥ speedup. Compared with the best alterna-
tive, Ansor improves the execution performance of DNNs on
the Intel CPU, ARM CPU, and NVIDIA GPU by up to3.8⇥,
2.6⇥, and1.7⇥, respectively. The reason for the signiﬁcant
speedup on DCGAN is that DCGAN mainly consists of trans-
posed 2D convolution (T2D), which can be well optimized by
Ansor, as shown and explained in the single operator bench-
mark (§7.1 ). AutoTVM performs very well for ResNet-50 on
the Intel CPU thanks to its highly-optimized templates for
2D convolution and global layout search [32]. Ansor does
not run a global layout search but does rewrite the layout of
weight tensors as described in§4.2 . Ansor uses more levels
of tiling so it packs weight tensors into more levels. The lay-
out rewrite brings about 40% improvement to ResNet-50 in
Ansor. Compared with vendor-speciﬁc static libraries, Ansor
has more advantages on uncommon shapes and small batch
sizes, because it is not easy to manually optimize for these
cases.
Ablation study.We run variants of Ansor on two test cases
in Figure 10. In the left ﬁgure, we run four variants of Ansor to
generate programs for a single mobilenet-V2. In the right ﬁg-
ure, we run these variants for both mobilenet-V2 and ResNet-
50. We set the objective function of the task scheduler to be the
geometric mean of speedup against AutoTVM. As shown in
Figure 10, “No task scheduler” means we use a round-robin
strategy to allocate equal time resources to all subgraphs.
“Limited space” is based on “Ansor (ours)” but limits the
search space. “No ﬁne-tuning” is also based on “Ansor (ours)”
but disables ﬁne-tuning and relies on random sampling only.
As can be seen inFigure 10, “Limited space” performs the
worst in terms of the ﬁnal achieved performance, proving that
the best programs are not included in the limited space. The
ﬁnal achieved performance can be improved by enlarging the
search space, as depicted in “No ﬁne-tuning”. However, in
the right ﬁgure, randomly assigning tile sizes and annotations
23D-ResNet and DCGAN are not yet supported by TensorFlow Lite on
the ARM CPU.
874    14th USENIX Symposium on Operating Systems Design and Implementation USENIX Association
still cannot beat AutoTVM in the given time budget. After
enabling ﬁne-tuning, “No task scheduler” outperforms Au-
toTVM in both cases. Finally, “Ansor (ours)” employs the
task scheduler to prioritize performance bottlenecks (e.g., sub-
graphs contain 3x3 convolution), so it performs the best in
both search efﬁciency and the ﬁnal achieved performance.
7.4 Search Time
Ansor searches efﬁciently and can outperform or match Au-
toTVM with less search time. Ansor slices the time and uti-
lizes the task scheduler to simultaneously optimize all sub-
graphs together. In contrast, AutoTVM and other systems do
not have a task scheduler, so they generate programs for all
subgraphs one by one with a predeﬁned budget of measure-
ment trials for each subgraph. Ansor saves the search time
by prioritizing important subgraphs, while AutoTVM spends
predeﬁned time budget on every subgraph, which may be a
waste on the unimportant subgraphs.
Table 3shows the search time required for Ansor to match
the performance of AutoTVM on the Intel CPU network
benchmark (§7.3 ). We list the search time in two metrics:
number of measurements and wall-clock time. “Number of
measurements” is a metric agnostic to the implementation
of measurement and the overhead of search algorithm, while
“Wall-clock time” takes these factors into account. As shown
in the table, Ansor can match the performance of AutoTVM
with an order of magnitude less search time. InTable 3athe
saving in search time comes from the task scheduler, efﬁ-
cient ﬁne-tuning, and comprehensive coverage of effective
optimizations. InTable 3b, Ansor shows more time-saving
in wall-clock time. This is because Ansor does not introduce
much search overhead and has a better implementation of the
measurement (on the Intel CPU, Ansor can get accurate mea-
surement results with fewer repetitions by explicitly ﬂushing
the cache for some tensors). On other backends, Ansor can
match the performance of AutoTVM with a similar saving in
search time.
Typically, it takes several hours for Ansor to generate fully-
optimized programs for a DNN on a single machine. This is
acceptable for inference applications because it is a one-shot
effort before deployment. In addition, the whole architecture
of Ansor can be parallelized very easily.
7.5 Cost Model Evaluation
In this subsection, we evaluate the prediction quality of the
learned cost model. We use 25,000 programs measured dur-
ing tuning ResNet-50 on the Intel CPU as the data set. We
randomly pick 20,000 programs as the training set and use
the remaining 5,000 programs as the test set. We train the cost
model and let it make predictions for the test set.
Figure 11 plots the predicted throughputs vs. measured
throughputs. The measured throughputs are normalized to
AutoTVM Ansor Time-saving
ResNet-50 21,220 6,403 3.3 ⇥
Mobilenet-V2 31,272 1,892 16.5 ⇥
3D-ResNet 5,158 1,927 2.7 ⇥
DCGAN 3,003 298 10.1 ⇥
BERT 6,220 496 12.5 ⇥
(a) The number of measurements.
AutoTVM Ansor Time-saving
ResNet-50 39,250 4,540 8.6 ⇥
Mobilenet-V2 58,468 660 88.6 ⇥
3D-ResNet 7,594 2,296 3.3 ⇥
DCGAN 4,914 420 11.7 ⇥
BERT 12,007 266 45.1 ⇥
(b) Wall-clock time (seconds)
Table 3: The number of measurements and wall-clock time
used for Ansor to match the performance of AutoTVM on the
Intel CPU (batch size=1).
Figure 11: Measured throughputs vs. predicted throughputs.
the best performing programs in the test set. The predicted
throughputs are the output of the model, so they can be neg-
ative. InFigure 11a, the points scatter around the diagonal
line, meaning that the model makes accurate predictions. The
distribution is not uniform because the data set is collected
during the search. Good programs have a higher probability
to be chosen for measurements, so most of the programs are
in the top right corner. The points with measured through-
put 0.0 are programs that are invalid or killed due to timeout
during measurements. InFigure 11b, we sort the 5000 points
according to the predictions from the slowest to the fastest,
and use the relative ranking as x-axis. So the points are dis-
tributed uniformly over x-axis. It shows the distribution of
performance of the explored programs better.
The model archives 0.079 RMSE, 0.958R2 correlation,
0.851 pairwise comparison accuracy, and 0.624 recall@30 of
top-30 programs (see the deﬁnition atfootnote 1) on the test
set.
8 Related Work
Automatic tensor program generation based on schedul-
ing languages.Halide [41] introduces a scheduling language
USENIX Association 14th USENIX Symposium on Operating Systems Design and Implementation    875
that can describe loop optimization primitives. This language
is suitable for both manual optimization and automatic search.
Halide has three versions of auto-scheduler based on differ-
ent techniques [2, 31, 36]. The latest one with beam search
and learned cost model performs the best among them, which
is also used in our evaluation. TVM [11] utilizes a similar
scheduling language and includes a template-guided search
framework AutoTVM [12]. FlexTensor [59] proposes general
templates that can target a set of operators, but its templates
are designed for single operators. It is hard to use these tem-
plates for optimizations involving multiple operators (e.g., op-
erator fusion). A concurrent work ProTuner [19] uses Monte
Carlo tree search to solve the inaccurate estimation prob-
lem in Halide auto-scheduler. ProTuner mainly targets im-
age processing workloads, while Ansor targets deep learning
workloads and introduces new search space and other opti-
mizations.
Polyhedral compilation models.The polyhedral compila-
tion model [8,52,53] formulates the optimization of programs
as an integer linear programming (ILP) problem. It optimizes
a program with afﬁne loop transformation that minimizes the
data reuse distance between dependent statements. Tiramisu
[5] and TensorComprehensions [49] are two polyhedral com-
pilers that also target the deep learning domain. Tiramisu pro-
vides a scheduling language similar to the Halide language,
and it needs manual scheduling. TensorComprehensions can
search for GPU code automatically, but it is not yet meant to
be used for compute-bounded problems [11]. It cannot outper-
form TVM on operators like conv2d and matmul [11,48]. This
is because of the lack of certain optimizations [50] and the
inaccurate implicit cost model in the polyhedral formulation.
Graph-level optimization for deep learning.Graph-level
optimizations treat an operator in the computational graph
as a basic unit and perform optimization at graph level with-
out changing the internal implementations of operators. The
common optimizations at graph level include layout optimiza-
tions [32], operator fusion [11, 38, 60], constant folding [42],
auto-batching [33], automatic generation of graph substitu-
tion [29] and so forth. The graph-level optimizations are typ-
ically complementary to operator-level optimizations. Graph-
level optimizations can also beneﬁt from high-performance
implementations of operators. For example, general opera-
tor fusion relies on the code generation ability of Ansor. We
leave the joint optimization of Ansor and more graph-level
optimization as future work.
Search-based compilation and auto-tuning. Search
based compilation and auto-tuning have already shown their
effectiveness in domains other than deep learning. Stock
[44] is a super-optimizer based on random search. Stock
searches for loop-free hardware instruction sequences, while
Ansor generates tensor programs with nests of loops. Open-
Tuner [4] is a general framework for program auto-tuning
based on multi-armed bandit approaches. OpenTuner relies
on user-speciﬁed search space, while Ansor constructs the
search space automatically. Traditional high-performance li-
braries such as ATLAS [56] and FFTW [16] also utilize
auto-tuning. More recent works NeuroVectorizer [18] and
AutoPhase [20, 26] use deep reinforcement learning to au-
tomatically vectorize programs and optimize the compiler
phase ordering.
9 Limitations and Future work
One of Ansor’s limitations is that Ansor cannot optimize
graphs with dynamic shapes [45]. Ansor requires the shapes
in the computational graph to be static and known in ad-
vance to do analysis, construct the search space, and perform
measurements. How to generate programs for symbolic or
dynamic shape is an interesting future direction. Another
limitation is that Ansor only supports dense operators. To
support sparse operators (e.g., SpMM) that are commonly
used in sparse neural networks [17] and graph neural net-
works [25], we expect that a large portion of Ansor can still
be reused, but we need to redesign the search space. Lastly,
Ansor only performs program optimizations at a high level but
relies on other code generators (e.g., LLVM and NVCC) to
do platform-dependent optimizations (e.g., instruction selec-
tion). Ansor comes short of utilizing the special instructions,
such as Intel VNNI, NVIDIA Tensor Core, and ARM Dot for
mixed-precision and low-precision operators, which are not
handled well by the off-the-shelf code generators currently.
10 Conclusion
We propose Ansor, an automated search framework that
generates high-performance tensor programs for deep neu-
ral networks. By efﬁciently exploring a large search space
and prioritizing performance bottlenecks, Ansor ﬁnds high-
performance programs that are outside the search space of
existing approaches. Ansor outperforms existing manual li-
braries and search-based frameworks on a diverse set of neural
networks and hardware platforms by up to3.8⇥. By automat-
ically searching for better programs, we hope that Ansor will
help bridge the gap between the increasing demand in com-
puting power and limited hardware performance. Ansor is
integrated into the Apache TVM open-source project3.
11 Acknowledgement
We would like to thank Weizhao Xian, Tianqi Chen, Frank
Luan, anonymous reviewers, and our shepherd, Derek Mur-
ray, for their insightful feedback. In addition to NSF CISE
Expeditions Award CCF-1730628, this research is supported
by gifts from Alibaba Group, Amazon Web Services, Ant
Group, CapitalOne, Ericsson, Facebook, Futurewei, Google,
Intel, Microsoft, Nvidia, Scotiabank, Splunk, and VMware.
3https://tvm.apache.org/
876    14th USENIX Symposium on Operating Systems Design and Implementation USENIX Association
References
[1] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng
Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, San-
jay Ghemawat, Geoffrey Irving, Michael Isard, et al.
Tensorﬂow: a system for large-scale machine learning.
In 12th USENIX Symposium on Operating Systems De-
sign and Implementation (OSDI 16), pages 265–283,
2016.
[2] Andrew Adams, Karima Ma, Luke Anderson, Riyadh
Baghdadi, Tzu-Mao Li, Michaël Gharbi, Benoit Steiner,
Steven Johnson, Kayvon Fatahalian, Frédo Durand, et al.
Learning to optimize halide with tree search and ran-
dom programs.ACM Transactions on Graphics (TOG),
38(4):1–12, 2019.
[3] Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars
Mescheder, Andreas Geiger, and Carsten Rother. Aug-
mented reality meets deep learning for car instance seg-
mentation in urban scenes. InBritish machine vision
conference, volume 1, page 2, 2017.
[4] Jason Ansel, Shoaib Kamil, Kalyan Veeramachaneni,
Jonathan Ragan-Kelley, Jeffrey Bosboom, Una-May
O’Reilly, and Saman Amarasinghe. Opentuner: an ex-
tensible framework for program autotuning. InProceed-
ings of the 23rd international conference on Parallel
architectures and compilation, pages 303–316, 2014.
[5] Riyadh Baghdadi, Jessica Ray, Malek Ben Romdhane,
Emanuele Del Sozzo, Abdurrahman Akkas, Yunming
Zhang, Patricia Suriana, Shoaib Kamil, and Saman Ama-
rasinghe. Tiramisu: a polyhedral compiler for express-
ing fast and portable code. In2019 IEEE/ACM Interna-
tional Symposium on Code Generation and Optimiza-
tion (CGO), pages 193–205. IEEE, 2019.
[6] Junjie Bai, Fang Lu, Ke Zhang, et al. Onnx: open neural
network exchange, 2019.
[7] Paul Barham and Michael Isard. Machine learning sys-
tems are stuck in a rut. InProceedings of the Workshop
on Hot Topics in Operating Systems, pages 177–183,
2019.
[8] Uday Bondhugula, Albert Hartono, Jagannathan Ra-
manujam, and Ponnuswamy Sadayappan. A practical
automatic polyhedral parallelizer and locality optimizer.
In Proceedings of the 29th ACM SIGPLAN Conference
on Programming Language Design and Implementation,
pages 101–113, 2008.
[9] Tianqi Chen and Carlos Guestrin. Xgboost: a scalable
tree boosting system. InProceedings of the 22nd acm
sigkdd international conference on knowledge discovery
and data mining, pages 785–794, 2016.
[10] Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang,
Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang,
and Zheng Zhang. Mxnet: a ﬂexible and efﬁcient ma-
chine learning library for heterogeneous distributed sys-
tems. arXiv preprint arXiv:1512.01274, 2015.
[11] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin
Zheng, Eddie Yan, Haichen Shen, Meghan Cowan,
Leyuan Wang, Yuwei Hu, Luis Ceze, et al. Tvm: an auto-
mated end-to-end optimizing compiler for deep learning.
In 13th USENIX Symposium on Operating Systems De-
sign and Implementation (OSDI 18), pages 578–594,
2018.
[12] Tianqi Chen, Lianmin Zheng, Eddie Yan, Ziheng Jiang,
Thierry Moreau, Luis Ceze, Carlos Guestrin, and Arvind
Krishnamurthy. Learning to optimize tensor programs.
In Advances in Neural Information Processing Systems,
pages 3389–3400, 2018.
[13] Sharan Chetlur, Cliff Woolley, Philippe Vandermersch,
Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan
Shelhamer. cudnn: efﬁcient primitives for deep learning.
arXiv preprint arXiv:1410.0759, 2014.
[14] Marius Cordts, Mohamed Omran, Sebastian Ramos,
Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson,
Uwe Franke, Stefan Roth, and Bernt Schiele. The
cityscapes dataset for semantic urban scene understand-
ing. InProceedings of the IEEE conference on computer
vision and pattern recognition, pages 3213–3223, 2016.
[15] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. Bert: pre-training of deep bidirec-
tional transformers for language understanding.arXiv
preprint arXiv:1810.04805, 2018.
[16] Matteo Frigo and Steven G Johnson. Fftw: an adap-
tive software architecture for the fft. InProceedings
of the 1998 IEEE International Conference on Acous-
tics, Speech and Signal Processing, ICASSP’98 (Cat. No.
98CH36181), volume 3, pages 1381–1384. IEEE, 1998.
[17] Trevor Gale, Erich Elsen, and Sara Hooker. The state
of sparsity in deep neural networks.arXiv preprint
arXiv:1902.09574, 2019.
[18] Ameer Haj-Ali, Nesreen K Ahmed, Ted Willke,
Yakun Sophia Shao, Krste Asanovic, and Ion Stoica.
Neurovectorizer: end-to-end vectorization with deep
reinforcement learning. In Proceedings of the 18th
ACM/IEEE International Symposium on Code Genera-
tion and Optimization, pages 242–255, 2020.
[19] Ameer Haj-Ali, Hasan Genc, Qijing Huang, William
Moses, John Wawrzynek, Krste Asanovi´c, and Ion Sto-
ica. Protuner: tuning programs with monte carlo tree
search. arXiv preprint arXiv:2005.13685, 2020.
USENIX Association 14th USENIX Symposium on Operating Systems Design and Implementation    877
[20] Ameer Haj-Ali, Qijing Huang, William Moses, John
Xiang, John Wawrzynek, Krste Asanovic, and Ion Sto-
ica. Autophase: juggling hls phase orderings in random
forests with deep reinforcement learning. InThird Con-
ference on Machine Learning and Systems (ML-Sys),
2020.
[21] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh.
Can spatiotemporal 3d cnns retrace the history of 2d
cnns and imagenet? InProceedings of the IEEE Con-
ference on Computer Vision and Pattern Recognition
(CVPR), pages 6546–6555, 2018.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian
Sun. Deep residual learning for image recognition. In
Proceedings of the IEEE conference on computer vision
and pattern recognition, pages 770–778, 2016.
[23] Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst.
Matrix capsules with em routing. 2018.
[24] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry
Kalenichenko, Weijun Wang, Tobias Weyand, Marco
Andreetto, and Hartwig Adam. Mobilenets: efﬁcient
convolutional neural networks for mobile vision appli-
cations. arXiv preprint arXiv:1704.04861, 2017.
[25] Yuwei Hu, Zihao Ye, Minjie Wang, Jiali Yu, Da Zheng,
Mu Li, Zheng Zhang, Zhiru Zhang, and Yida Wang.
Featgraph: A ﬂexible and efﬁcient backend for
graph neural network systems. arXiv preprint
arXiv:2008.11359, 2020.
[26] Qijing Huang, Ameer Haj-Ali, William Moses, John Xi-
ang, Ion Stoica, Krste Asanovic, and John Wawrzynek.
Autophase: compiler phase-ordering for hls with deep
reinforcement learning. In2019 IEEE 27th Annual In-
ternational Symposium on Field-Programmable Custom
Computing Machines (FCCM), pages 308–308. IEEE,
2019.
[27] Intel. IntelR  math kernel library for deep learning net-
works, 2017.
[28] Sergey Ioffe and Christian Szegedy. Batch normaliza-
tion: accelerating deep network training by reducing in-
ternal covariate shift.arXiv preprint arXiv:1502.03167,
2015.
[29] Zhihao Jia, Oded Padon, James Thomas, Todd Warsza-
wski, Matei Zaharia, and Alex Aiken. Taso: optimizing
deep learning computation with automatic generation
of graph substitutions. InProceedings of the 27th ACM
Symposium on Operating Systems Principles, pages 47–
62, 2019.
[30] Andrew Lavin and Scott Gray. Fast algorithms for con-
volutional neural networks. InProceedings of the IEEE
Conference on Computer Vision and Pattern Recogni-
tion, pages 4013–4021, 2016.
[31] Tzu-Mao Li, Michaël Gharbi, Andrew Adams, Frédo
Durand, and Jonathan Ragan-Kelley. Differentiable
programming for image processing and deep learning
in halide. ACM Transactions on Graphics (TOG),
37(4):139, 2018.
[32] Yizhi Liu, Yao Wang, Ruofei Yu, Mu Li, Vin Sharma,
and Yida Wang. Optimizing cnn model inference on
cpus. In 2019 USENIX Annual Technical Conference
(USENIX ATC 19), pages 1025–1040, 2019.
[33] Moshe Looks, Marcello Herreshoff, DeLesley Hutchins,
and Peter Norvig. Deep learning with dynamic compu-
tation graphs.arXiv preprint arXiv:1702.02181, 2017.
[34] Mark F. Medress, Franklin S Cooper, Jim W. Forgie,
CC Green, Dennis H. Klatt, Michael H. O’Malley, Ed-
ward P Neuburg, Allen Newell, DR Reddy, B Ritea, et al.
Speech understanding systems: report of a steering com-
mittee. Artiﬁcial Intelligence, 9(3):307–316, 1977.
[35] Thierry Moreau, Tianqi Chen, Luis Vega, Jared Roesch,
Eddie Yan, Lianmin Zheng, Josh Fromm, Ziheng Jiang,
Luis Ceze, Carlos Guestrin, et al. A hardware–software
blueprint for ﬂexible deep learning specialization.IEEE
Micro, 39(5):8–16, 2019.
[36] Ravi Teja Mullapudi, Andrew Adams, Dillon Sharlet,
Jonathan Ragan-Kelley, and Kayvon Fatahalian. Auto-
matically scheduling halide image processing pipelines.
ACM Transactions on Graphics (TOG), 35(4):83, 2016.
[37] Nvidia. Nvidia tensor cores, 2017.
[38] Nvidia. Nvidia tensorrt: programmable inference accel-
erator, 2017.
[39] Adam Paszke, Sam Gross, Francisco Massa, Adam
Lerer, James Bradbury, Gregory Chanan, Trevor Killeen,
Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.
Pytorch: an imperative style, high-performance deep
learning library. In Advances in Neural Information
Processing Systems, pages 8024–8035, 2019.
[40] Alec Radford, Luke Metz, and Soumith Chintala. Un-
supervised representation learning with deep convolu-
tional generative adversarial networks.arXiv preprint
arXiv:1511.06434, 2015.
[41] Jonathan Ragan-Kelley, Connelly Barnes, Andrew
Adams, Sylvain Paris, Frédo Durand, and Saman Ama-
rasinghe. Halide: a language and compiler for optimiz-
ing parallelism, locality, and recomputation in image
878    14th USENIX Symposium on Operating Systems Design and Implementation USENIX Association
processing pipelines.Acm Sigplan Notices, 48(6):519–
530, 2013.
[42] Jared Roesch, Steven Lyubomirsky, Marisa Kirisame,
Josh Pollock, Logan Weber, Ziheng Jiang, Tianqi Chen,
Thierry Moreau, and Zachary Tatlock. Relay: a high-
level compiler for deep learning. arXiv preprint
arXiv:1904.08368, 2019.
[43] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey
Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: in-
verted residuals and linear bottlenecks. InProceedings
of the IEEE conference on computer vision and pattern
recognition, pages 4510–4520, 2018.
[44] Eric Schkufza, Rahul Sharma, and Alex Aiken. Stochas-
tic superoptimization.ACM SIGARCH Computer Archi-
tecture News, 41(1):305–316, 2013.
[45] Haichen Shen, Jared Roesch, Zhi Chen, Wei Chen,
Yong Wu, Mu Li, Vin Sharma, Zachary Tatlock, and
Yida Wang. Nimble: Efﬁciently compiling dynamic
neural networks for model inference.arXiv preprint
arXiv:2006.03031, 2020.
[46] Patricia Suriana, Andrew Adams, and Shoaib Kamil.
Parallel associative reductions in halide. In 2017
IEEE/ACM International Symposium on Code Gener-
ation and Optimization (CGO), pages 281–291. IEEE,
2017.
[47] Richard S Sutton and Andrew G Barto.Reinforcement
learning: an introduction. MIT press, 2018.
[48] Philippe Tillet, HT Kung, and David Cox. Triton: an
intermediate language and compiler for tiled neural net-
work computations. In Proceedings of the 3rd ACM
SIGPLAN International Workshop on Machine Learn-
ing and Programming Languages, pages 10–19, 2019.
[49] Nicolas Vasilache, Oleksandr Zinenko, Theodoros
Theodoridis, Priya Goyal, Zachary DeVito, William S
Moses, Sven Verdoolaege, Andrew Adams, and Albert
Cohen. Tensor comprehensions: framework-agnostic
high-performance machine learning abstractions.arXiv
preprint arXiv:1802.04730, 2018.
[50] Nicolas Vasilache, Oleksandr Zinenko, Theodoros
Theodoridis, Priya Goyal, Zachary Devito, William S
Moses, Sven Verdoolaege, Andrew Adams, and Albert
Cohen. The next 700 accelerated layers: from mathe-
matical expressions of network computation graphs to
accelerated gpu kernels, automatically.ACM Transac-
tions on Architecture and Code Optimization (TACO),
16(4):1–26, 2019.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser,
and Illia Polosukhin. Attention is all you need. InAd-
vances in neural information processing systems, pages
5998–6008, 2017.
[52] Sven Verdoolaege. Presburger formulas and polyhedral
compilation. 2016.
[53] Sven Verdoolaege, Juan Carlos Juega, Albert Cohen,
Jose Ignacio Gomez, Christian Tenllado, and Francky
Catthoor. Polyhedral parallel code generation for cuda.
ACM Transactions on Architecture and Code Optimiza-
tion (TACO), 9(4):1–23, 2013.
[54] Pradnya A Vikhar. Evolutionary algorithms: a critical
review and its future prospects. In2016 International
conference on global trends in signal processing, infor-
mation computing and communication (ICGTSPICC),
pages 261–265. IEEE, 2016.
[55] Leyuan Wang, Zhi Chen, Yizhi Liu, Yao Wang, Lianmin
Zheng, Mu Li, and Yida Wang. A uniﬁed optimization
approach for cnn model inference on integrated gpus.
In Proceedings of the 48th International Conference on
Parallel Processing, pages 1–10, 2019.
[56] R Clinton Whaley and Jack J Dongarra. Automatically
tuned linear algebra software. InSC’98: Proceedings
of the 1998 ACM/IEEE conference on Supercomputing,
pages 38–38. IEEE, 1998.
[57] Fisher Yu and Vladlen Koltun. Multi-scale context
aggregation by dilated convolutions. arXiv preprint
arXiv:1511.07122, 2015.
[58] Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu,
Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang,
Danyang Zhuo, Koushik Sen, et al. Ansor: generating
high-performance tensor programs for deep learning.
https://arxiv.org/abs/2006.06762, 2020.
[59] Size Zheng, Yun Liang, Shuo Wang, Renze Chen, and
Kaiwen Sheng. Flextensor: an automatic schedule ex-
ploration and optimization framework for tensor compu-
tation on heterogeneous system. InProceedings of the
Twenty-Fifth International Conference on Architectural
Support for Programming Languages and Operating
Systems, pages 859–873, 2020.
[60] Zhen Zheng, Pengzhan Zhao, Guoping Long, Feiwen
Zhu, Kai Zhu, Wenyi Zhao, Lansong Diao, Jun Yang,
and Wei Lin. Fusionstitching: boosting memory inten-
sive computations for deep learning workloads.arXiv
preprint arXiv:2009.10924, 2020.
USENIX Association 14th USENIX Symposium on Operating Systems Design and Implementation    879

1
Predicting New Workload or CPU Performance
by Analyzing Public Datasets
YU WANG,Harvard University
VICTOR LEE,Intel Corporation
GU-YEON WEI and DAVID BROOKS,Harvard University
/T_he marketplace for general-purpose microprocessors oﬀers hundreds of functionally similar models,
diﬀering by traits like frequency, core count, cache size, memory bandwidth, and power consumption. /T_heir
performance depends not only on microarchitecture, but also on the nature of the workloads being executed.
Given a set of intended workloads, the consumer needs both performance and price information to make
rational buying decisions. Many benchmark suites have been developed to measure processor performance,
and their results for large collections of CPUs are o/f_ten publicly available. However, repositories of benchmark
results are not always helpful when consumers need performance data fornew processorsor new workloads.
Moreover, the aggregate scores for benchmark suites designed to cover a broad spectrum of workload types
can be misleading. To address these problems, we have developed a deep neural network (DNN) model, and
we have used it to learn the relationship between the speci/f_ications of Intel CPUs and their performance on
the SPEC CPU2006 and Geekbench 3 benchmark suites. We show that we can generate useful predictions
for new processorsand new workloads. We also cross-predict the two benchmark suites and compare their
performance scores. /T_he results quantify the self-similarity of these suites for the /f_irst time in the literature.
/T_his work should discourage consumers from basing purchasing decisions exclusively on Geekbench 3, and it
should encourage academics to evaluate research using more diverse workloads than the SPEC CPU suites
alone.
ACM Reference format:
Yu Wang, Victor Lee, Gu-Yeon Wei, and David Brooks. 2018. Predicting New Workload or CPU Performance
by Analyzing Public Datasets. ACM Transactions on Architecture and Code Optimization1, 1, Article 1
(January 2018), 22 pages.
DOI: 10.1145/3284127
1 INTRODUCTION
Computer hardware evolves rapidly. In 2015 alone, Intel released over 200 CPU SKUs.1 /T_he SKUs
have diﬀerent characteristics like frequency, cache size, memory bandwidth, and core count. Be/t_ter
con/f_igurations usually cost more. /T_he performance of a microprocessor is not solely a function of
its microarchitecture; it depends critically on the nature of the workloads running on it. /T_hus both
CPU microarchitecture and workloads need to be taken into account when quantifying actual CPU
performance. Consumers can then estimate whether investing in a costly con/f_iguration helps to
meet their particular goals.
1Stock Keeping Unit. In this paper, a SKU is a CPU with a distinct processor number.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee
provided that copies are not made or distributed for pro/f_it or commercial advantage and that copies bear this notice and
the full citation on the /f_irst page. Copyrights for components of this work owned by others than ACM must be honored.
Abstracting with credit is permi/t_ted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires
prior speci/f_ic permission and/or a fee. Request permissions from permissions@acm.org.
© 2018 ACM. XXXX-XX/2018/1-ART1$15.00
DOI: 10.1145/3284127
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
1:2 Yu Wang, Victor Lee, Gu-Yeon Wei, and David Brooks
To help study processor performance, a variety of benchmark suites have been developed. For
the more popular suites, such as Geekbench, SPEC CPU, and Passmark, sizable repositories of
performance data have been collected, allowing the public to compare the behaviors of known
con/f_igurations on standard workloads. However, there are some issues in using these repositories.
/T_he /f_irst is that consumers need to wait for comprehensive benchmark results ofnew processorsto
be contributed to the public repositories. /T_he second issue is that fornew workloads, consumers
have to test a large number of possible con/f_igurations to /f_ind the most eﬀective hardware. Daunted
by this challenge, some consumers rely blindly on aggregate CPU scores in benchmark repositories.
To resolve these issues, we use statistical methods and machine learning techniques to analyze
data from the SPEC CPU2006 and Geekbench 3 repositories.2 SPEC CPU2006 is widely reported
in academic studies. Geekbench 3 is used by many consumers to make purchasing decisions. We
use deep neural networks (DNN) to predict the performance of Intel CPUs, and we compare the
DNN prediction with that by linear regression (LR). /T_he DNN predictive model learns interactions
between processor speci/f_ications for diﬀerent workloads, sharpening the usefulness of benchmark
data repositories.
/T_his paper makes the following contributions:
•We use DNN to predict the performance of SPEC and Geekbench workloads on new CPU
SKUs. (/T_he mean error is 5% for SPEC and 11% for Geekbench.) We show that DNN is more
accurate than traditional LR.
•With the same accuracy, we show that the performance of new workloads on just 10 SKUs
can predict their performance on other SKUs.
•For the /f_irst time in the literature, we quantify the self-similarity of these widely used
suites, by using DNN to cross-predict the results between the suites (with 25.9% and 14.9%
mean error when predicting SPEC and Geekbench, respectively), and by comparing their
CPU rankings (which are inconsistent, with footrule distance as high as 0.59).
/T_he rest of the paper is organized as follows. Section 2 gives our analysis of Intel processors in
terms of SKU speci/f_ications and SPEC performance. Section 3 describes our methodology. Section 4
summarizes our three case studies. Section 5 compares our prediction accuracy on those case
studies. Section 6 compares the SKU rankings in the three cases. Section 7 summarizes related
work and compares it with our own. Section 8 presents our conclusions.
2 INTEL PROCESSOR STATISTICS
In this section, we present some statistical facts about Intel processors, SPEC performance for the
Intel SKUs, and the SKUs’ microarchitectures and types in our SPEC and Geekbench datasets. We
show the large performance variations and a rich variety of SKUs in our datasets.
We construct our Intel processor speci/f_ication dataset from h/t_tp://ark.intel.com, where Intel pub-
lishes the speci/f_ications of all its processors. We collect speci/f_ications including microarchitecture,
launch year, number of cores, base frequency, cache size, power, and memory type. Figure 1 shows
the release years of Intel microarchitectures. /T_he size of a bubble corresponds to the number of
SKUs released in the corresponding year. /T_he /f_igure contains 17 microarchitectures that Intel has
released since 1993.
Intel has a “Tick-Tock” model for microarchitecture code names. A tick represents a shrinking
of the technology and a tock represents a new microarchitecture. In Figure 1, for example, West-
mere and Sandy Bridge have 32nm feature size, and Sandy Bridge and Ivy Bridge have the same
microarchitecture. Every year, there is expected to be a tick or tock. Starting in 2014, however,
Intel decided to add a “refresh” cycle, to update the current microarchitecture. /T_hat is why for 2014
2/T_he datasets in this paper are available at h/t_tps://github.com/Emma926/cpuselection.git.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
Predicting New Workload or CPU Performance
by Analyzing Public Datasets 1:3
Fig. 1. The number of SKUs released by Intel for 17 microarchitectures between 1993 and 2016.
Bubble size indicates the number of SKUs. Architectures include x86, Itanium, and Atom. (Data
from h/t_tp://ark.intel.com.)
in Figure 1 there are only a few Broadwell SKUs but more Haswell SKUs. A/f_ter 2016, Intel switched
their model to a three element cycle known as “Process-Architecture-Optimization”.
In this paper, we use relative run time as a measure of performance. We take a Sandy Bridge
processor (E3-1230) as the reference machine. We choose E3-1230 because it is common across
datasets. /T_he relative runtime of a workload is de/f_ined as
relati /v.alte runtime = runtime −runtime r ef
runtime r ef
(1)
where runtime r ef is the runtime of the corresponding workload on the reference machine. For
brevity, this paper will consistently use “performance” to indicate relative run time. /T_hus the scaled
runtime of E3-1230 is 0. In Figure 2, the E3-1230 performance is slightly below the average of all
Sandy Bridge SKUs, so the line is centered on a value greater than 0. Since all results are presented
relative to the same reference processor, the results are unaﬀected by the particular choice of the
reference.
To show the diverse performance of diﬀerent SKUs, we take the average SPEC performance on
SKUs, and we gather the SKUs based on their microarchitecture code names. In Figure 2, the dots
show the mean relative run time of the SKUs with the same microarchitecture code name, and
the lengths of the bars show the standard deviations. Longer bars indicate the performance of the
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
1:4 Yu Wang, Victor Lee, Gu-Yeon Wei, and David Brooks
0.5
 0.0 0.5 1.0
Relative Runtime
Nehalem
Westmere
Sandy Bridge
Ivy Bridge
Haswell
Broadwell
Skylake
Fig. 2. Means and standard deviations of SPEC relative run times. Each run time is the average
over all SPEC workloads of performance for a given SKU. The lengths of the bars show that SKUs
with the same microarchitecture have very diverse performance.
SKUs is more diverse. /T_he SKUs have the same microarchitecture but diﬀerent frequency, cache
size, memory size, and so on. /T_hese lead to variations in each bar, andthey contribute as much
to performance variation as the microarchitectures themselves./T_he bars overlap horizontally. /T_hat
means when selecting SKUs to optimize performance, there can be many choices with diﬀerent
microarchitectures but similar performance.
/T_he SKUs in the public repositories are random, depending on users’ submissions. SPEC and
Geekbench have their own setup and compilation instructions. /T_he setup of each data point is
included in the repositories. We summarize the SKUs in the SPEC and Geekbench datasets in
Figure 3. Our work focuses on performance prediction on the SKUs currently on the market, thus
Fig. 3. SKU breakdown in the SPEC (top) and Geekbench (bo/t_tom) datasets. Total number of SKUs
is 352 for SPEC and 119 for Geekbench.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
Predicting New Workload or CPU Performance
by Analyzing Public Datasets 1:5
we discard SKUs older than Sandy Bridge. We /f_irst do preprocessing to aggregate duplicate data.
Before explaining preprocessing, we /f_irst de/f_ine a CPU con/f_iguration as a SKU running at a certain
frequency with a certain memory size. In this paper, we say that a con/f_iguration (SKU, frequency,
memory size) determines a workload’s performance. Other factors such as compiler and OS aﬀect
performance as well, but we do not consider them in this work. In the repositories, multiple result
submissions may have the same con/f_iguration. We average the data points that have the same
workload and con/f_iguration.
Figure 3 shows the microarchitectures and types of the SKUs in our SPEC and Geekbench datasets.
/T_here are 352 SKUs in total for SPEC, and 119 SKUs for Geekbench. In both datasets, we have more
SKUs of Sandy Bridge, Ivy Bridge, and Haswell than Broadwell and Skylake. Figure 3 shows that
we have a fairly rich collection of SKUs.
Users can customize the memory size with a chosen SKU. We refer to a SKU with a certain
memory size as a con/f_iguration. In total, we have 639 con/f_igurations of 352 SKUs for the SPEC
workloads. To help visualize the performance of SPEC workloads on 639 con/f_igurations, we /f_irst
show, in Figure 4, the means and standard deviations of the relative run times of the workloads.
/T_he blue bars are SPECfp; the red bars, SPECint. SPECint workloads are very similar to each other,
except for 462.libquantum. SPECfp workloads have more diverse means and standard deviations.
/T_hat indicates FP workloads are more diverse in terms of performance on the 639 con/f_igurations.
/T_he performance of 462.libquantum is more similar to SPECfp. If clustering the workloads, we
would like to see that they form two clusters, with SPECint and some SPECfp workloads in one
cluster and the other FP workloads, which have quite diﬀerent means and standard deviations in
Figure 4, in the other cluster.
To visualize them in a two-dimensional space, we construct a matrix with 28 rows (for the 28
SPEC workloads) and 639 columns. Every matrix entry is the relative run time of the workload
on the corresponding SKU con/f_iguration. We then do a principal component analysis (PCA) to
410.bwaves
416.gamess
433.milc
434.zeusmp
435.gromacs
436.cactusADM
437.leslie3d
444.namd
450.soplex
453.povray
454.calculix
459.GemsFDTD
465.tonto
470.lbm
481.wrf
482.sphinx3
400.perlbench
401.bzip2
403.gcc
429.mcf
445.gobmk
456.hmmer
458.sjeng
462.libquantum
464.h264ref
471.omnetpp
473.astar
483.xalancbmk
1.0
0.5
0.0
0.5
1.0
Relative Runtime
SPEC FP
 SPEC Int
Fig. 4. Means and standard deviations of SPEC workloads’ relative run times on the SKUs in our
dataset. SPECfp is more diverse than SPECint. 462.libquantum is more similar to SPECfp.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
1:6 Yu Wang, Victor Lee, Gu-Yeon Wei, and David Brooks
visualize the matrix. Figure 5 shows the PCA space. We only show two principal components (PCs)
because the /f_irst and second PCs account for 82% of the variance.
/T_he closer two workloads are, the more similar their performance scaling across the 639 con-
/f_igurations. Blue dots are SPECfp and red are SPECint. SPECfp is more spread out than SPECint.
/T_hat means SPECfp workloads are more sensitive to SKU con/f_iguration diﬀerences, such as mi-
croarchitecture, frequency, and memory size. It is also an indication that it is harder for current
microarchitectures to extract performance from SPECint than FP. SPECint workloads have less
instruction level parallelism and more data dependencies. /T_his is also observed by Campanoni et
al. [2].
/T_he points at the bo/t_tom of Figure 5 form two clusters, and the point at the top (481.wrf) is
clearly an outlier. We want to /f_ind the centroids of the two clusters while leaving 481.wrf alone.
Running a k-means algorithm with three clusters leaves 481.wrf in a cluster by itself. We plot the
two centroids of the other two clusters using stars. 462.libquantum (the red dot at the bo/t_tom le/f_t)
is clustered with SPECfp. /T_he centroids will be used to explain our results in Section 5.2.
Although Figures 4 and 5 lead to consistent conclusions, they characterize the performance space
diﬀerently. Figure 4 focuses on the means and standard deviations of the performance data, while
PCA in Figure 5 emphasizes diﬀerent performance on diﬀerent SKU con/f_igurations. /T_his can be
explained with an example using two workloads (A, B) and three SKU con/f_igurations. Workload
A’s relative run times are 0.5, 0.7, -0.4, respectively. On the same SKU con/f_igurations, workload B’s
relative run times are 0.7, -0.4 and 0.5. /T_he two workloads could look exactly the same in Figure 4,
but PCA would show their diﬀerence.
Fig. 5. Principal component analysis of SPEC workloads’ performance scaling on 639 configura-
tions of 352 SKUs. The green stars are centroids of two clusters identified by k-means. SPECfp
workloads are more spread out than SPECint. 462.libquantum (red dot at the bo/t_tom le/f_t) is clus-
tered with SPECfp benchmarks. The outlier near the top is 481.wrf.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
Predicting New Workload or CPU Performance
by Analyzing Public Datasets 1:7
3 METHODOLOGY
/T_he previous section shows a large variation in performance. /T_hus a predictive model can be
helpful. /T_his section shows the method we use to build a predictive model. We present our features,
data preprocessing, model selection, and metrics (baselines) for evaluating the results.
A naive way to make a prediction is to use a previous generation to predict its successor. To do this,
one needs the exact con/f_iguration of interest from previous generation, because sometimes other
features are more important than microarchitecture generations, shown in the bars of Figure 2. For
the same reason, using other SKUs in one generation to predict a new SKU in the same generation
is diﬃcult. We have tried that, and our current predictive models achieve be/t_ter accuracy. We
eventually decide to use machine learning to learn the relations between CPU speci/f_ications and
performance, to avoid the missing data problem of the naive methods.
3.1 Features
Table 1 summarizes the data in this paper. /T_he data include features fed to the predictive model,
and the value we predict, the run time. /T_he features include CPU speci/f_ications from the Intel CPU
dataset and dynamic workload features from the SPEC and Geekbench datasets.
CPU Speci/f_ications:/T_he microarchitecture code names include microarchitecture changes
and technology scaling. Within one microarchitecture generation, the SKUs are diﬀerent in their
frequencies, last level cache sizes, TDP, and number of cores. We also add the release year as a
feature. /T_he type of the SKU is also an important feature. Intel categorizes the SKUs as mobile,
Table 1.Data in this paper. The data include CPU specifications (from the Intel processor dataset)
and dynamic workload features (from SPEC and Geekbench datasets), and what we predict (run
time).
Data Description Data Type
Workload /T_he names of the workloads One hot encode
Microarchitecture Code Name Intel code names Integer encode
Type Server, desktop, mobile, or embedded One hot encode
L3 Cache Size Last level cache size Numerical
Instruction Set Extensions SSE or AVX Integer encode
Memory Type DDR,DDR2,DDR3,DDR4 Integer encode
Memory Channels Number of memory channels Numerical
Max Memory Bandwidth Max memory bandwidth Numerical
ECC Memory Support Whether ECC memory is supported Binary
Base Frequency Nominal frequency Numerical
Turbo Frequency Turbo frequency Numerical
Turbo Boost Technology Whether Turbo Boost is supported Binary
Cores Number of cores Numerical
/T_hreads Number of threads Numerical
Hyper-/T_hreading Whether hyper-threading is supported Binary
TDP /T_hermal design power Numerical
Year Year of release Numerical
Frequency /T_he dynamic frequency Numerical
Memory Size /T_he size of oﬀ-chip memory Numerical
Run Time /T_he run times of the workloads Numerical
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
1:8 Yu Wang, Victor Lee, Gu-Yeon Wei, and David Brooks
desktop, server, and embedded. L2 and L1 caches per core do not change much across SKUs. /T_hus
we only include last level cache sizes.
For discrete data, we do integer encoding and one hot encoding. Integer encoding means mapping
discrete values to integers. Each feature needs only one entry. Discrete names and types usually
obey the order of the assigned integers. One hot encoding uses one feature entry for one discrete
value. It implies nothing more than that the discrete values are diﬀerent. /T_he encoding methods
are speci/f_ied in Table 1. We choose to map the microarchitecture code names, memory types, and
instruction set extensions into consecutive integers, with their natural order. For the SKU types, it
is easier to do one hot encoding. We also distinguish the workloads using one hot encoding. Before
training and testing, numerical and integer data need to be standardized, but one hot encoding
data do not.
Dynamic Workload Features:Every entry in the SPEC and Geekbench datasets contains a
con/f_iguration (SKU, current frequency, memory size), the name of the workload, and performance
(relative run time, which is the value to predict). We then get the CPU speci/f_ications from the Intel
processor dataset by the SKU processor number. We do not use the scores provided in SPEC and
Geekbench as performance metrics. For SPEC, we use the run time in seconds. For Geekbench, we
assume that the total amount of work is constant for a given workload and we use the inverse of
throughput to estimate run time. Details on the data preprocessing are in Section 3.2.
3.2 Data Preprocessing
/T_he SPEC2006 repository provides run time in seconds. Geekbench 3 provides throughput, e.g.,
as GB/second. We /f_irst average the performance of the workloads with the same (CPU SKU,
Frequency, Memory Size) con/f_iguration. Variations of system so/f_tware such as OS or compiler
are not considered in this work. Geekbench 3 run time is taken to be the inverse of throughput,
assuming that one workload has a constant total amount of work. /T_he average run time is then
converted into relative run time for the chosen reference machine based on Equation 1. /T_he
speci/f_ic choice of the reference machine does not aﬀect the prediction. /T_he relative performance
is normalized to have mean 0 and standard deviation 1. /T_he preprocessing steps are typical of
machine learning experiments [22].
3.3 Model Selection
In this section, we explain our choice of deep neural networks (DNN) and linear regression (LR) for
this work. We then illustrate our approach for tuning DNN topology and the hyperparameters of
DNN and LR.
Deep Neural Networks:Deep neural networks (DNN) have proven successful in many domains,
from regression and classi/f_ication to game playing. Compared with traditional methods, DNN
excels when data size is large, scales well with more data, and it does not require expertise in
feature engineering. /T_he advantages of DNN make it a very good /f_it for the performance prediction
scenario. /T_hus in this paper, we use DNN as our predictive model.
/T_he topology and hyperparameters are tuned through model selection. To do model selection,
we randomly split the dataset into a training set, a validation set, and a testing set. /T_he testing
set is the last hold out set, to avoid using all data to do model selection. We start by sweeping
the hyperparameters in all experiments, including the number of layers, the nodes per layer, the
learning rate, the number of training epochs, the activation function, and the optimizer. /T_he models
are trained with the training set. /T_he model with the lowest loss on the validation set is selected.
We /f_ind that the best set of hyperparameters is always the same. /T_herefore in all experiments
we use a DNN with 3 hidden layers and 100 nodes per layer. /T_he learning rate is 0.001 over 300
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
Predicting New Workload or CPU Performance
by Analyzing Public Datasets 1:9
training epochs. /T_he activation function is tanh and the optimizer is RMSprop. /T_hus, for example,
the number of weights for predicting SPEC is 50 million. Fortunately, training is oﬄine and has a
one-time cost.
Linear Regression: LR has been shown to work well in the CPU performance prediction
literature [20], while DNN is known to be able to explore non-linear interactions between features,
and it has been shown to be more accurate than LR at performance prediction [16, 17]. Unlike
DNNs, LR has interpretable weight parameters. In realistic scenarios, people can choose LR or
DNN based on the parameters, accuracy, and the value of interpretable weights. In order to make
it a fair comparison, we reimplement LR with MAE as the loss function. /T_hat is, the model is
optimized to yield the lowest MAE. Minimizing the MAE entails assuming the noise distribution to
be Laplacian [4]. In our experiments, both DNN and LR have L1 regularization with a weight of
0.01. /T_hey take exactly the same training sets, testing sets, and features.
Other models: Popular predictive models other than DNN include linear regression (LR),
support vector machines (SVM),k nearest neighbors (kNN), principal component analysis (PCA),
and genetic algorithms (GA). In Section 7, we list studies using some of those models. However
SVM is not suitable for large scale training because of its computational complexity [35]. kNN,
PCA and GA are be/t_ter when there are fewer data points (< 100) and simpler feature relations.
Besides, traditional methods need carefully selected features, and accuracy does not bene/f_it from
more training data. /T_herefore we choose to use DNN and compare it with one traditional method,
LR.
3.4 Metrics
Mean Absolute Error:We use mean absolute error (MAE) as the measure of accuracy, rather
than mean squared error (MSE). /T_he reasons are as follows.
(1) /T_here may be outliers and errors in the submi/t_ted data. With MSE, the optimizer penalizes
the outliers much more than normal data. However, we do not want to over/f_it the model to
the outliers [22].
(2) If a prediction isx and the MAE of the model is 0.01, the real value ofx is very likely to be
x ±0.01. MSE is not as easy to interpret.
Because the MAE is in a standardized space, we use two baselines to help understand it. We
assume two independent and identically distributed random variables,a and b, from Gaussian
distribution N(0, 1). /T_he expected distance betweena and b is 2√π , which is approximately 1.13 [32].
/T_herefore a prediction with MAE of 0.25 is a very good prediction. Another way to interpret the
MAE is to compare it with the standard deviation of the testing set. A naive way to do prediction is
to always predict the mean of the dataset. /T_his way, by de/f_inition the MAE is the standard deviation
MAE =
Ín
i=0 |xi −m|
n = Standard Deviation (2)
where n is the number of data points,m is the mean of the testing set, andxi is the real value of
predicted performance, a/f_ter normalization. Note that while the whole dataset is normalized to
have mean 0 and standard deviation 1, the testing sets may be in diﬀerent scales. In the captions of
Figures 6 and 7, we compare the MAEs with the standard deviations of the testing sets, and show
that the MAEs are much smaller. However what we just discussed are naive baselines. /T_he next
metric can show that our prediction is good enough in realistic scenarios.
Top-k Accuracy: To bring the MAE metric into realistic context, we use the predictive model
to help select SKUs. We predict performance and use that prediction to rank the SKUs.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
1:10 Yu Wang, Victor Lee, Gu-Yeon Wei, and David Brooks
We choose top-k accuracy, which measures whether the known best SKU is among the topk
predicted SKUs. It is a popular metric for image classi/f_ication tasks [13, 19]. Other metrics for
comparing rankings include Pearson correlation, Spearman footrule distance, and Kendall rank
correlation. /T_hey measure the diﬀerence between the tested ranking and the expected ranking, by
considering every item in each ranking. Top-k accuracy is a be/t_ter metric for this work, because
users care more about whether the best SKU is in the topk choices, than about how similar one
SKU ranking is to another. In this context, top-k accuracy is more interpretable than the ranking
correlations.
We construct three baselines to compare with our predictive model. Without the predictive
model, it is customary to compare microarchitecture code names (newer is be/t_ter), frequencies
(higher is be/t_ter), and cache sizes (larger is be/t_ter). /T_hus we construct the baselines taking those
factors into consideration. /T_hough the baselines are simple, this is how most customers make
purchasing decisions.
A Frequency: Rank the SKUs based on frequencies. If frequencies are the same, rank them
with cache sizes.
B Cache: Rank the SKUs based on cache sizes. If cache sizes are the same, rank them with
frequencies.
C Frequency + Cache:Rank the SKUs based on a summation of frequency and cache size
with weights. /T_he weights are discussed in Section 5.1. If the values are the same, rank
them with microarchitecture code names.
To measure the top-k accuracy, we shuﬄe our training or testing set, /f_ind the best SKU in the
testing set using real performance from the datasets, and measure how o/f_ten the best SKU is one of
the topk SKUs as ranked by our predictive model and the baselines.
4 CASE STUDIES OVERVIEW
In this section, we sketch the case studies, including prediction for new SKUs and for new workloads,
and cross-prediction between suites. We implement our models with Keras (h/t_tps://keras.io).
4.1 Case 1: Prediction for New SKUs
In this case, we show that we can predict the performance of SPEC and Geekbench workloads
on new SKUs. Using our open-source model, trained with SPEC and Geekbench data for existing
SKUs from online repositories, users can predict Geekbench and SPEC performance on new SKUs
of interest and choose the ones likely to perform the best.
In order to test the model on new SKUs, we adopt repeated random sub-sampling validation,
a widely-used cross-validation method [8]. We hold out several SKUs from the dataset, train the
model with the remaining data, and test the model with the hold out set.
We randomly choose 10 percent of the SKUs from one microarchitecture at a time as our testing
set. We use the rest of the data as our training set. We repeat this process 20 times for every
microarchitecture. We choose 20 repetitions a/f_ter trying 50 and /f_inding that the results were
statistically consistent. Comparing the MAEs of 20 DNN repetitions versus 50, the maximum
absolute diﬀerence is 0.005 and the standard deviation diﬀerence is 0.009. Both are minimal.
/T_he premise of this case study is that the predictive models are able to learn the performance
function of a workload using the CPU speci/f_ications in Table 1. Speci/f_ically, the training set may
contain SKUs with the same microarchitecture as the new SKU, and SKUs with similar features
such as L3 cache, frequency, and memory size, but diﬀerent microarchitecture. /T_he predictive
model then predicts the new SKU’s performance as though interpolating the training set.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
Predicting New Workload or CPU Performance
by Analyzing Public Datasets 1:11
Note that the prediction is nontrivial, even if the same microarchitecture or the full feature set
of a diﬀerent microarchitecture is in the training set. As shown in Figure 2, both microarchitecture
and other SKU features (such as frequency) introduce large variations in performance. Knowing
one of them does not make the prediction easy.
4.2 Case 2: Prediction for New Workloads
In this study, we show that a/f_ter measuring performance of a new workload on just a handful of
SKUs, we can train a model to predict the workload’s performance on other SKUs. Armed with such
predictions, consumers can choose the best processor for the workload based on both performance
and price.
To demonstrate the approach, we split our dataset, taking out one workload at a time and treating
it as the new workload. /T_hen we train the model with all the other workloads’ data plus a certain
amount of data from the chosen workload. We perform the experiment separately for SPEC and
Geekbench. We refer to this case as self-prediction of benchmark suites.
/T_he premise of this and the next case study is that, given the data of a new workload on
several SKUs, the model can /f_ind workloads with similar performance on those SKUs, and use
the performance of those workloads to predict the new workload’s performance on other SKUs.
/T_herefore the prediction accuracy should be related to the similarity of the new workload to the
workloads in the training set.
For every workload A, we remove the data for workload A from the overall dataset. We call
the remaining datadata rest . We then sample a /f_ixed set of SKUs for workload A as a testing set,
and we refer to the rest of the data for workload A astrain pool. We randomly pickn SKUs from
train pool (where n is 1, 5, 10, or 50) and combine their data withdata rest to form a training set.
We repeat the process for 20 iterations.
4.3 Case 3: Cross-Prediction Between Suites
/T_he experiments in case 2 are conducted within each benchmark suite. Case 3 is an extension of
case 2 that we call cross-prediction. We predict one workload in Geekbench using all data for SPEC
plus that collected using several SKUs for the chosen workload. /T_hen we repeat with Geekbench
and SPEC switched.
For every workload A in Geekbench, we split the whole dataset into the data of workload A
and the rest. We then sample a /f_ixed set of SKUs from workload A as a testing set, and we refer to
the remaining data of workload A astrain pool. We randomly pickn SKUs fromtrain pool and
combine their data with the SPEC data as a training set. We sweepn over 1, 5, 10, and 50, and we
repeat the process 20 times.
5 PREDICTION ACCURACY
In this section, we show prediction accuracy measured by mean absolute error (MAE) in the case
studies. We show that DNN is more accurate than LR, and case 2 (self-prediction) is more accurate
than case 3 (cross-prediction).
5.1 Case 1: Prediction for New SKUs
Figure 6 compares the MAEs of LR and DNN for SPEC and Geekbench, respectively. /T_he heights of
the bars represent the average MAEs of the 20 random test sets. /T_he error bars show one standard
deviation of the MAEs. A smaller error bar indicates that diﬀerent selections of testing set give
large diﬀerences in MAEs, therefore the model is more robust to predict diﬀerent SKUs.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
1:12 Yu Wang, Victor Lee, Gu-Yeon Wei, and David Brooks
Fig. 6. Results of predicting the performance of SPEC (top) and Geekbench (bo/t_tom) on new SKUs.
We compare linear regression (LR) and deep neural network (DNN) models. DNN always has lower
mean absolute errors (MAE) than LR. With DNN, the average MAE for SPEC is 5.5% and for Geek-
bench it is 11.2%. The standard deviations of the test sets are all over 20%.
For both SPEC and Geekbench, DNN always has lower MAEs and is less sensitive to the selection
of the test set than LR. /T_hat means the relationships between features and performance are not
simply linear. In Figure 6 (bo/t_tom), LR has a very large MAE and standard deviation when predicting
Geekbench on Broadwell SKUs. /T_hat is because in Figure 3 (bo/t_tom), there are only /f_ive Broadwell
SKUs, and one is desktop while the others are servers. Using servers to predict desktop performance
leads to large MAEs for LR. Apparently DNN addresses this be/t_ter. To compare the average MAEs
(the last set of bars in Figure 6), LR is 19.9% and DNN is 5.5% for SPEC, and LR is 20% and DNN is
11.2% for Geekbench. DNN is a be/t_ter choice when the data set is large and interactions between
features are complicated.
/T_he average MAE is 5.5% for SPEC and 11.2% for Geekbench using DNN. /T_he MAE for Geekbench
is higher than SPEC because Geekbench users are not like SPEC users, who are mostly computer
architects and system engineers, benchmarking on dedicated machines. One can easily install
Geekbench and produce scores on personal devices. Results may be noisy if other applications
are running at the same time, causing contention in computing and memory resources. We
observe larger variance in Geekbench data. We compute the average standard deviation of the
performance for the same workload with the same (SKU, frequency, memory) con/f_iguration. It is
4.4% for Geekbench, and 1.1% for SPEC. /T_he larger noise from the data makes it harder to predict
Geekbench.
/T_he accuracy of simulation-based prediction is about 4% [20], which is slightly lower than our
5.5% MAE of SPEC. Compared to simulation-based approach, the extra noise in our approach comes
from the performance variation introduced by compilers and operating systems in the datasets,
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
Predicting New Workload or CPU Performance
by Analyzing Public Datasets 1:13
and no architectural features. We use no architectural features which makes the prediction harder,
but it also makes the method easier to apply to any other public datasets, without the need for
tedious simulation.
5.2 Case 2: Prediction for New Workloads
In this section, we show the prediction results of case 2. We show that the DNN prediction accuracy
is largely determined by the similarity of the predicted workload to the rest of the benchmark
suite. We quantify the outlierness of a workload and show that it is positively correlated with the
prediction MAE, with a Pearson correlation of 0.69.
Prediction Results:/T_he results of SPEC are shown in Figure 7 (le/f_t). /T_he workloads are sorted
based on the MAEs a/f_ter adding 50 SKUs (brown line). (/T_he line for 5 added SKUs is omi/t_ted but
the average bar is shown.) /T_he standard deviations of all the testing sets are larger than 20%, and
our MAEs in all cases are lower than the standard deviation. For 5, 10, and 50 added SKUs, the
MAEs are 5.7%, 5.3%, and 4.7%, respectively. MAE improvements when moving from 10 SKUs to 50
are similar to those when moving from 5 SKUs to 10. Since adding 40 SKUs is more diﬃcult than
adding 5, we conclude that using 10 SKUs for the new workload is a reasonable choice.
Similarly, the Geekbench prediction results are shown in Figure 7 (right). /T_he workloads are
sorted and the rightmost workloads are outliers with very high MAEs. /T_he standard deviations of
Geekbench testing sets are all above 11%. /T_he outliers’ standard deviations are at least twice their
MAEs. /T_hat means our model shrinks the con/f_idence interval by at least 50%.
Case 2 is harder than case 1. In case 1, the average MAE is 11%. In case 2, the average MAE is
14.2%, with two obvious outlier workloads (Blackscholes and AES). We conclude that case 2 works
reasonably well, apart from the two outliers. Also, for the same reason as with SPEC, we choose 10
as the proper number of SKUs to use.
Insights on Benchmark Similarity:By studying MAEs across SPEC workloads, we /f_ind that
the prediction MAE is correlated with the similarity of the predicted workload to the rest of the
benchmark suite. /T_his observation supports the statement in Section 4.2 that the model /f_inds similar
workloads based on the data points of the new workload, and it uses the similar workloads to
predict the performance of the new workload on other SKUs.
Fig. 7. The results of case 2. The workloads are sorted by the MAE a/f_ter adding 50 SKUs. The
workloads on the right are outliers. The average MAEs are shown in the bars on the right. The line
for adding 5 SKUs is omi/t_ted, but the average bar for that case is shown. The standard deviations
of the testing sets are all over 11%.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
1:14 Yu Wang, Victor Lee, Gu-Yeon Wei, and David Brooks
In Figure 7 (le/f_t), a/f_ter adding 50 SKUs it is still relatively hard to predict the workloads on
the right. We call these workloads outliers. /T_he workload with highest MAE is 481.wrf and it is
also the outlier lying above all other workloads in Figure 5, which shows the workloads’ relative
performance scaling in the varied SKU con/f_igurations.
Some outliers are well-studied in prior works. Phansalkar et al. show that 436.cactusADM is an
outlier in terms of memory access characteristics in Figure 9 of [24]. However, workload features
collected in prior works are unable to explain why 481.wrf is such an outlier in SKU scaling space,
as shown in Figures 7 and 5. Future work will try to answer that question.
To study the outlierness of the workloads in Figure 7 (le/f_t), we ran a k-means algorithm to split
the workloads into two clusters,cluster1 and cluster2. /T_heir centroidsC1 and C2 are plo/t_ted as stars
in Figure 5. We compute the distance of a workload to its cluster’s centroid as
distance = I(x ∈cluster1)||x −C1||2
+I(x ∈cluster2)||x −C2||2
(3)
where x stands for the location of the workload in Figure 5 andI yields 1 if its argument is true
and 0 otherwise. /T_he distance quanti/f_ies the outlierness of a workload.
Figure 8 plots a workload’s MAE a/f_ter 50 added SKUs (/y.alt-axis) against its distance (x-axis).
Distance and MAE have a Pearson correlation of 0.6898. We can use distance to identify workloads
with high errors. Workloads with distance higher than 5 have MAEs higher than 10%. /T_his
correlation explains why the outliers have higher MAEs than others even a/f_ter adding 50 SKUs.
Conclusion: /T_his case shows that we are able to predict a new workload’s performance by
running it on 10 SKUs. /T_he accuracy of the prediction depends on the similarity of the workload to
those in the training set. Even if it is very diﬀerent, our model is still able to shrink the con/f_idence
interval.
Fig. 8. Correlation of SPEC workloads’ outlierness (distance) with MAE a/f_ter adding 50 SKUs. The
distance of a workload is its distance from the the centroid in its cluster. Centroids are in Figure 5.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
Predicting New Workload or CPU Performance
by Analyzing Public Datasets 1:15
5.3 Case 3: Cross-Prediction Between Suites
Figure 9 shows the results of cross-prediction: using Geekbench to predict SPEC (top) andvice versa
(bo/t_tom). Because the MAEs of the workloads and the outliers are similar to case 2 (self-prediction),
we only show the average MAE bars in this case.
To compare self-prediction and cross-prediction, we /f_irst compare Figure 9 (top) with the bars in
Figure 7 (le/f_t), both of which predict SPEC workloads. Cross-prediction gives higher MAEs, for
any number of added SKUs. A/f_ter adding 50 SKUs, the average MAE of cross-prediction is 16.9%;
the average MAE of self-prediction is 4.6%. Comparing the predictions of Geekbench in Figure 9
(bo/t_tom) and the bars in Figure 7 (right), a/f_ter adding 50 SKUs the average MAE for cross-prediction
is 12.6% and that for self-prediction is 10%.
Cross-prediction gives consistently higher errors because the workloads in SPEC and Geekbench
are very diﬀerent, more diﬀerent than the workloads in either benchmark suite itself. In another
words, the workloads in a benchmark suite are more similar to each other than to workloads outside
of the benchmark suite.
Benchmark suites like SPEC CPU2006 are expected to be diverse and representative of real-world
workloads. However, a benchmark suite is developed for a purpose, and diﬀerent suites usually
have diﬀerent purposes. /T_he purpose itself is the reason why the workloads in the suite are similar.
/T_he SPEC CPU2006 benchmark suite is intended for computer designers and architects to use for
pre-silicon design analysis. SPEC workloads are drawn from and representative of real workloads.
In contrast, Geekbench is intended to stress a certain part of the hardware. For example, memory
microbenchmarks do a lot of memory streaming operations to study the bandwidth limit of the
hardware system. From this point of view, it is reasonable that SPEC and Geekbench are very
diﬀerent and the workloads in each suite are more similar.
Conclusion: Cross-prediction is harder than self-prediction. To predict performance for new
workloads, one needs a training set with a very diverse collection of workloads. Benchmark suites
tend to be self-similar, because each serves a speci/f_ic purpose. Even so, cross-prediction is useful,
as shown in the next section.
Geekbench -> SPEC
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35 Case 3: Mean Absolute Error
DNN + 1 SKUDNN + 5 SKUsDNN + 10 SKUsDNN + 50 SKUs
SPEC -> Geekbench
0.00
0.05
0.10
0.15
0.20
0.25
0.30
Fig. 9. Case 3 is about cross-prediction. SPEC workloads are predicted with the Geekbench
dataset (top). Geekbench workloads are predicted with the SPEC dataset (bo/t_tom). Cross-
prediction has higher errors than self-prediction (case 2).
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
1:16 Yu Wang, Victor Lee, Gu-Yeon Wei, and David Brooks
DNN Freq Cache Freq+Cache
0.0
0.2
0.4
0.6
0.8
1.0Avg Top-3 Accuracy
Case 1: SPEC
DNN Freq Cache Freq+Cache
0.0
0.2
0.4
0.6
0.8
1.0
Geekbench
Fig. 10. SKU ranking results of case 1. Our DNN model has the highest top-3 accuracy.
6 SKU RANKING COMPARISON
In this section, we use the performance prediction results of the DNN model to rank SKUs. We
then compare the SKU rankings with the baseline metrics described in Section 3.4. /T_he baselines
represent the customary ways of making purchasing decisions. Our methodology makes SKU
selection more accurate without requiring expertise on workload characterization or computer
architecture speci/f_ications.
6.1 Case 1: Prediction for New SKUs
In this section, we show that our DNN model outperforms the three baselines in Section 3.4 when
using SPEC and Geekbench to select new SKUs.
/T_he results of SPEC and Geekbench are shown in Figure 10. For baseline C, a combination of
frequency and cache, we swept 100 combinations of weights. We combinedfrequency(GHz)×w
with cache(MB)×(1 −w), and sweptw from 0.01 to 1, with a step size of 0.01. In Figure 10, we use
0.9 ×f requenc/y.alt+ 0.1 ×cache so that the frequency and cache values are in about the same range.
/T_he top-3 accuracy of baseline C combining frequency and cache is no be/t_ter than A (frequency) or
B (cache).
Frequency is the best metric among the three baselines. /T_he following subsections show similar
results, therefore we show baseline A and drop the others in the rest of the paper.
Our predictive model outperforms all of the workload-indiﬀerent baselines. For Geekbench, the
frequency metric achieves accuracy comparable to that of our DNN model. /T_he Geekbench suite
as a whole is quite frequency-sensitive. /T_he SPEC suite, on the other hand, is sensitive to neither
frequency nor cache, which suggests that choosing SKUs for complicated workloads requires more
than simply comparing frequency or cache.
Conclusion: When selecting a SKU that optimizes SPEC or Geekbench performance, our
predictive model works be/t_ter than simply comparing frequencies, cache sizes, or a combination
of the two. Selecting a SKU by frequency is be/t_ter than by cache size, and is more successful for
microbenchmarks like Geekbench than for complicated workloads like SPEC.
6.2 Cases 2 and 3: Self- and Cross-Prediction
In this section, we compare the top-k accuracies of cases 2 and 3 and baseline A (frequency). Results
using baselines B and C are consistent with those presented, so we omit them. We show case 2 and
case 3 in the same plots to compare their results. Both cases use 10 additional SKUs for the new
workload. /T_he results appear in Figure 11.
It is very diﬃcult to /f_ind the best SKU for SPEC workloads for cases other than SPEC self-
prediction (case 2). /T_he top-3 accuracy of case 3 is very small, and that of frequency is 0. We start
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
Predicting New Workload or CPU Performance
by Analyzing Public Datasets 1:17
400.perlbench
401.bzip2
403.gcc
410.bwaves
416.gamess
429.mcf
433.milc
434.zeusmp
435.gromacs
436.cactusADM
437.leslie3d
444.namd
445.gobmk
450.soplex
453.povray
454.calculix
456.hmmer
458.sjeng
459.GemsFDTD
462.libquantum
464.h264ref
465.tonto
470.lbm
471.omnetpp
473.astar
481.wrf
482.sphinx3
483.xalancbmk
Avg
0.0
0.5
1.0Top-3 Accuracy
Case 2 and 3: SPEC
AES
BZip2 Compress
BZip2 Decompress
BlackScholes
Blur Filter
DFFT
DGEMM
Dijkstra
JPEG Compress
JPEG Decompress
Lua
Mandelbrot
N-Body
PNG Compress
PNG Decompress
Ray Trace
SFFT
SGEMM
SHA1
SHA2
Sharpen Filter
Sobel
Stream Add
Stream Copy
Stream Scale
Stream Triad
Twofish
Avg
0.0
0.5
1.0Top-3 Accuracy
Geekbench Self Cross Freq
Fig. 11. Based on SPEC (top) and Geekbench (bo/t_tom), compare the SKU rankings by self-
prediction (case 2), cross-prediction (case 3) and the best baseline (Frequency). Self-prediction
performs the best. Frequency is not a reasonable approach to select SKUs for new workloads.
to observe more accuracy from them when we relax the choices to top-7 SKUs. It indicates building
the predictive model is the only reasonable way to /f_ind good SKUs for SPEC. /T_he situation can get
worse for the workloads in people’s daily lives such as web browser, Microso/f_t Oﬃce suite, Adobe
suite, and other customized workloads from individual companies.
Self-prediction has higher accuracies than cross-prediction for most of the workloads, suggesting
self-prediction is easier. /T_he ranking accuracies further support what has been observed from
the prediction accuracy studies in Figures 7 and 9, and the self-similarity statement in Section 5.3.
Exceptions are AES, Dijkstra, and Stream Scale from Geekbench, which have higher accuracies
when trained with SPEC. Among them AES is the outlier with the highest MAE from the self-
prediction accuracy results in Figure 7 (right). /T_he results indicate that those three workloads are
more similar to SPEC in terms of selecting the best SKUs.
Self- and cross-prediction are more accurate than frequency. Note that in cases 2 and 3, the test
set is /f_ixed, and the only variation in the 50 iterations is the random additional 10 SKUs from the
new workload, as described in Section 4. /T_herefore for frequency bars in Figure 11, there is only
one ranking of SKUs to test, and the top-k accuracy is either 0 or 1, depending on whether the
best SKU is ranked top-k or not. By averaging the accuracies for the whole suite, we /f_ind that
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
1:18 Yu Wang, Victor Lee, Gu-Yeon Wei, and David Brooks
0 20 40 60 80 100 120
SPEC Rank
0
20
40
60
80
100
120GeekBench Rank
Rank Comparison, Footrule Distance = 0.594873243415
Fig. 12. Comparison of SKU rankings based on SPEC and Geekbench averages. Points are clus-
tered above the diagonal because Geekbench ranks SKUs lower than SPEC (giving them higher
ranking indexes). The horizontal cluster below shows that Geekbench poorly distinguishes SKUs
with quite diﬀerent SPEC behavior.
frequency performs the worst. /T_hough the previous subsection shows the accuracy of frequency to
be comparable to DNN prediction for Geekbench (Figure 10), individual workloads may or may not
be sensitive to frequency. Frequency is not a good way to select SKUs.
Conclusion: SPEC self-prediction is the only reasonable way to /f_ind the best SKU for SPEC
workloads. Self-prediction is easier than cross-prediction in most cases. Frequency has the lowest
accuracy, so frequency alone should not be the deciding factor when selecting SKUs for new
workloads.
6.3 Benchmark Suite Comparison
In this section, we compare the benchmark suites in terms of SKU ranking. While there is no
prediction involved, it is natural to compare the two benchmark suites a/f_ter comparing the individual
workloads in them.
Geekbench is very popular on computer enthusiast websites, with many articles claiming that
“Geekbench suggests A outperforms B”. Such articles aﬀect many consumers’ choices when pur-
chasing computing products. Most people use laptops to run workloads such as web browsers,
Microso/f_t’s Oﬃce suite, Adobe so/f_tware, and so on. We describe these as realistic workloads.
Consumers choose CPUs based on Geekbench scores because they assume that Geekbench predicts
the performance of realistic workloads. Here we view SPEC workloads as falling between realistic
workloads and Geekbench in terms of complexity and real-world relevance.
To show how the two suites produce diﬀerent rankings, we rank the SKUs that SPEC and
Geekbench datasets have in common by averaging each benchmark suite’s performance on each
such SKU. Figure 12 shows the ranking comparison. /T_hex-axis is the rank based on SPEC and the
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
Predicting New Workload or CPU Performance
by Analyzing Public Datasets 1:19
/y.alt-axis is the same SKU’s rank based on Geekbench. /T_he footrule distance between the two rankings
is 0.59. (/T_he range is from 0 to 1.) If the ranks were consistent on both suites, the points would lie
along the gray line (x = /y.alt). We observe two pa/t_terns in Figure 12. One is the cluster of points above
the gray line; Geekbench ranks those SKUs lower than SPEC (gives them higher ranking indexes).
Another pa/t_tern is the cluster of points below the gray line and close to thex-axis. SPEC ranks
those SKUs from 0 to 120, while Geekbench ranks them from 0 to around 50. In another words, the
SKUs have very diﬀerent SPEC performance, but are poorly diﬀerentiated by Geekbench.
/T_he Geekbench SKU ranking is not consistent with SPEC rankings. /T_hat means that if an enthu-
siast article says “Geekbench shows that A outperforms B”, SPEC may suggest that B outperforms
A. It is very unlikely that a Geekbench score can give accurate indications for realistic workloads
either. If one buys a CPU con/f_iguration based on its highest Geekbench score, the hardware may
not be the best for running the workloads that he or she runs daily.
/T_his suggests that people use Geekbench incorrectly. /T_he aggregate score of a benchmark suite
is misleading. Geekbench is designed to stress /f_loating point, integer, and memory operations. It
makes more sense to compare the performance of individual microbenchmarks that are relevant to
realistic workloads. If one runs a lot of matrix operations and image processing workloads, such as
MATLAB and Adobe Photoshop, the /f_loating point benchmarks in Geekbench are very helpful.
Conclusion: Aggregate Geekbench scores are misleading. Benchmarking relevant individual
Geekbench workloads makes more sense. SKU rankings by Geekbench are inconsistent with
rankings by SPEC, and probably also with rankings for realistic workloads. For a non-expert
consumer, the best way is to take all datasets available, plus the new workload’s performance on
10 SKUs, and train a predictive model as in cases 2 and 3.
7 RELATED WORK
7.1 Performance Prediction
For performance prediction, our work uses deep neural networks (DNNs), regarded as the state-of-
the-art machine learning algorithm. It has been proven to be able to simulate any function [5].
Unlike simulation-based performance prediction [16, 17, 20, 28], our approach does not use
microarchitecture features such as instruction issue width, read write buﬀer size, or number of
pipeline stages. We use only the CPU speci/f_ications that manufacturers provide. Microarchitecture
features would make performance prediction easier and more accurate, but our approach is more
helpful for consumers choosing CPUs. Even when manufacturers describe microarchitecture
features, consumers need expert knowledge to make use of such information.
Performance prediction is a well-studied /f_ield. /T_here are mechanistic models [3, 10–12, 18, 33, 34],
empirical models [9, 21, 31], and machine learning models [1, 36]. Zheng et al. used performance
counters and a ”stochastic dynamic coupling” heuristic for aligning host and target execution
samples to estimate performance using only binary code [37]. In contrast, we focus on CPU SKU
selection for consumers, rather than performance prediction for hardware/architecture designers.
Our method does not use performance counters or architecture features other than those accessible
to all consumers.
Piccart et al. use data transposition to rank commercial machines [27]. /T_heir work exploits
machine similarity rather than workload similarity. Our work has diﬀerent objectives and uses
diﬀerent metrics. It exploits workload similarity and benchmark suite self-similarity to help users
pick the best CPU SKU for their needs.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
1:20 Yu Wang, Victor Lee, Gu-Yeon Wei, and David Brooks
7.2 Workload Characterization
Phansalkar et al. develop a series of SPEC workload characterizations [23–26]. However they
do not focus on workload performance scaling as we do, so their classi/f_ication results are not
consistent with ours. Our approach provides an orthogonal way to characterize workloads.
Some researchers use static workload features to predict performance. Shelepov et al. use spatial
locality, cache size, and frequency to predict speedup on a variety of processors [29, 30]. Hoste
et al. collect microarchitecture-independent features to classify workload performance [14, 15].
Delimitrou et al. study workload characteristics for cloud scheduling [6, 7]. In our work, we predict
a new workload’s performance on hundreds of CPU SKUs. /T_he number of workloads is much
smaller than the number of hardware platforms. Because the workload features are static, we argue
that it is hard to use static features in this work.
8 CONCLUSION
To help customers select proper CPU SKUs and overcome the limitations of public benchmark
datasets, we have presented statistical and predictive analysis of workload performance, with data
collected from the SPEC CPU and Geekbench 3 suites and Intel processor speci/f_ications.
We demonstrated performance prediction fornew SKUsand new workloadswith deep neural
networks (DNNs). For new SKUs, the accuracy is 5% (SPEC) and 11% (Geekbench) mean absolute
error (MAE). With the same accuracy, we /f_ind that we can predict a new workload’s performance by
running it on 10 SKUS. We compared our predictive methods against workload-indiﬀerent metrics
for selecting processors. Our predictive model is the only approach that achieves reasonable
accuracy in all three case studies. Notably, workload-indiﬀerent methods of SKU selection do not
work for new workloads.
For the /f_irst time in the literature, we showed benchmark suite self-similarity quantitatively by
cross-prediction and comparison of SPEC and Geekbench SKU rankings. /T_he accuracy of cross-
prediction is lower than that for prediction within the suites because they are so diﬀerent: 25.9%
mean error to predict SPEC and 14.2% to predict Geekbench.
Rankings based on average Geekbench are inconsistent with those based on average SPEC.
Geekbench rankings do not imply SPEC rankings. It is also hard to draw any conclusions about
realistic workloads from a benchmark suite. We suggest studying the Geekbench or SPEC workloads
of interest, rather than relying on aggregate scores.
9 ACKNOWLEDGEMENT
/T_he authors would like to thank the anonymous reviewers for their thoughtful comments and
constructive suggestions which signi/f_icantly improved the quality of this paper. Part of this work
was supported as an internship project at the Intel Parallel Computing Lab under the supervision of
Victor Lee. /T_he authors would like to thank Bob Adolf, Udit Gupta, Glenn Holloway, Svilen Kanev,
Lillian Pentecost, Saketh Rama, Brandon Reagen, Margo Seltzer, Sophia Shao, and Sam Xi for their
feedback. /T_his work was supported in part by the National Science Foundation (NSF) under award
number 1533737.
REFERENCES
[1] N. Ardalani, C. Lestourgeon, K. Sankaralingam, and X. Zhu. Cross-architecture performance prediction (XAPP) using
CPU code to predict GPU performance. InProceedings of the 48th International Symposium on Microarchitecture, pages
725–737. ACM, 2015.
[2] S. Campanoni, K. Brownell, S. Kanev, T. M. Jones, G.-Y. Wei, and D. Brooks. HELIX-RC: an architecture-compiler
co-design for automatic parallelization of irregular programs. pages 217–228, 2014.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
Predicting New Workload or CPU Performance
by Analyzing Public Datasets 1:21
[3] X. E. Chen and T. M. Aamodt. Hybrid analytical modeling of pending cache hits, data prefetching, and MSHRs.ACM
Transactions on Architecture and Code Optimization (TACO), 8(3):10, 2011.
[4] V. Cherkassky and Y. Ma. Comparison of loss functions for linear regression. InNeural Networks, 2004. Proceedings.
2004 IEEE International Joint Conference on, volume 1, pages 395–400. IEEE, 2004.
[5] G. Cybenko. Approximation by superpositions of a sigmoidal function.Mathematics of control, signals and systems,
2(4):303–314, 1989.
[6] C. Delimitrou and C. Kozyrakis. Paragon: QoS-aware scheduling for heterogeneous datacenters. InProceedings of
the 18th international conference on Architectural support for programming languages and operating systems (ASPLOS),
volume 48, pages 77–88. ACM, 2013.
[7] C. Delimitrou and C. Kozyrakis. /Q_uasar: resource-eﬃcient and QoS-aware cluster management. InProceedings of
the 19th international conference on Architectural support for programming languages and operating systems (ASPLOS),
pages 127–144. ACM, 2014.
[8] W. Dubitzky, M. Granzow, and D. P. Berrar.Fundamentals of data mining in genomics and proteomics. Springer Science
& Business Media, 2007.
[9] P. Ein-Dor and J. Feldmesser. A/t_tributes of the performance of central processing units: a relative performance
prediction model.Communications of the ACM, 30(4):308–317, 1987.
[10] S. Eyerman, L. Eeckhout, T. Karkhanis, and J. E. Smith. A mechanistic performance model for superscalar out-of-order
processors. ACM Transactions on Computer Systems (TOCS), 27(2):3, 2009.
[11] S. Eyerman, K. Hoste, and L. Eeckhout. Mechanistic-empirical processor performance modeling for constructing CPI
stacks on real hardware. InPerformance Analysis of Systems and So/f_tware (ISPASS), 2011 IEEE International Symposium
on, pages 216–226. IEEE, 2011.
[12] A. Hartstein and T. R. Puzak. /T_he optimum pipeline depth for a microprocessor. InProceedings of the 29th annual
international symposium on Computer architecture (ISCA), pages 7–13. IEEE Computer Society, 2002.
[13] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference
on computer vision and pa/t_tern recognition, pages 770–778, 2016.
[14] K. Hoste, L. Eeckhout, and H. Blockeel. Analyzing commercial processor performance numbers for predicting
performance of applications of interest. InProceedings of the 2007 ACM SIGMETRICS international conference on
Measurement and modeling of computer systems, pages 375–376. ACM, 2007.
[15] K. Hoste, A. Phansalkar, L. Eeckhout, A. Georges, L. K. John, and K. De Bosschere. Performance prediction based on
inherent program similarity. InParallel Architectures and Compilation Techniques (PACT), 2006 International Conference
on, pages 114–122. IEEE, 2006.
[16] E. Ipek, B. R. De Supinski, M. Schulz, and S. A. McKee. An approach to performance prediction for parallel applications.
In European Conference on Parallel Processing, pages 196–205. Springer, 2005.
[17] E. ¨Ipek, S. A. McKee, R. Caruana, B. R. de Supinski, and M. Schulz. Eﬃciently exploring architectural design spaces
via predictive modeling. InProceedings of the 12th international conference on Architectural support for programming
languages and operating systems (ASPLOS), pages 195–206. ACM, 2006.
[18] T. S. Karkhanis and J. E. Smith. A /f_irst-order superscalar processor model. InComputer Architecture, 2004. Proceedings.
31st Annual International Symposium on, pages 338–349. IEEE, 2004.
[19] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classi/f_ication with deep convolutional neural networks. In
Advances in neural information processing systems (NIPS), pages 1097–1105, 2012.
[20] B. C. Lee et al. Accurate and eﬃcient regression modeling for microarchitectural performance and power prediction.
In Proceedings of the 12th international conference on Architectural support for programming languages and operating
systems, pages 185–194. ACM, 2006.
[21] A. Li, X. Zong, S. Kandula, X. Yang, and M. Zhang. CloudProphet: towards application performance prediction in
cloud. pages 426–427, 2011.
[22] K. P. Murphy.Machine learning: a probabilistic perspective. MIT press, 2012.
[23] A. Phansalkar, A. Joshi, L. Eeckhout, and L. K. John. Measuring program similarity: Experiments with SPEC CPU
benchmark suites. InPerformance Analysis of Systems and So/f_tware, 2005. ISPASS 2005. IEEE International Symposium
on, pages 10–20. IEEE, 2005.
[24] A. Phansalkar, A. Joshi, and L. K. John. Analysis of redundancy and application balance in the SPEC CPU2006
benchmark suite. InProceedings of the 34th annual international symposium on Computer architecture, pages 412–423.
ACM, 2007.
[25] A. Phansalkar, A. Joshi, and L. K. John. Subse/t_ting the SPEC CPU2006 benchmark suite.ACM SIGARCH Computer
Architecture News, 35(1):69–76, 2007.
[26] A. S. Phansalkar.Measuring program similarity for eﬃcient benchmarking and performance analysis of computer systems.
/T_he University of Texas at Austin, 2007.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.
1:22 Yu Wang, Victor Lee, Gu-Yeon Wei, and David Brooks
[27] B. Piccart, A. Georges, H. Blockeel, and L. Eeckhout. Ranking commercial machines through data transposition. In
Workload Characterization (IISWC), 2011 IEEE International Symposium on, pages 3–14. IEEE, 2011.
[28] S. Sharkawi, D. Desota, R. Panda, R. Indukuru, S. Stevens, V. Taylor, and X. Wu. Performance projection of HPC
applications using SPEC CFP2006 benchmarks. InParallel & Distributed Processing, 2009. IPDPS 2009. IEEE International
Symposium on, pages 1–12. IEEE, 2009.
[29] D. Shelepov, J. C. Saez Alcaide, S. Jeﬀery, A. Fedorova, N. Perez, Z. F. Huang, S. Blagodurov, and V. Kumar. HASS: a
scheduler for heterogeneous multicore systems.ACM SIGOPS Operating Systems Review, 43:66–75, 2009.
[30] D. Shepelow and A. Fedorova. Scheduling on heterogeneous multicore processors using architectural signatures. In
WIOSCA workshop of the 35th annual international symposium on Computer architecture (ISCA), 2008.
[31] K. Singh, E.˙Ipek, S. A. McKee, B. R. de Supinski, M. Schulz, and R. Caruana. Predicting parallel application performance
via machine learning approaches.Concurrency and Computation: Practice and Experience, 19(17):2219–2235, 2007.
[32] B. /T_hirey and R. Hickman. Distribution of euclidean distances between randomly distributed gaussian points in
n-space. arXiv preprint arXiv:1508.02238, 2015.
[33] S. Van den Steen, S. De Pestel, M. Mechri, S. Eyerman, T. Carlson, D. Black-Schaﬀer, E. Hagersten, and L. Eeckhout.
Micro-architecture independent analytical processor performance and power modeling. InPerformance Analysis of
Systems and So/f_tware (ISPASS), 2015 IEEE International Symposium on, pages 32–41. IEEE, 2015.
[34] S. Van den Steen, S. Eyerman, S. De Pestel, M. Mechri, T. E. Carlson, D. Black-Schaﬀer, E. Hagersten, and L. Eeckhout.
Analytical processor performance and power modeling using micro-architecture independent characteristics. InIEEE
Transaction on Computers, volume 65, 2016.
[35] Y. Wang and D. Wang. Towards scaling up classi/f_ication-based speech separation.IEEE Transactions on Audio, Speech,
and Language Processing, 21(7):1381–1390, 2013.
[36] X. Zheng, L. K. John, and A. Gerstlauer. Accurate phase-level cross-platform power and performance estimation. In
Design Automation Conference (DAC), 2016 53nd ACM/EDAC/IEEE, pages 1–6. IEEE, 2016.
[37] X. Zheng, H. Vikalo, S. Song, L. K. John, and A. Gerstlauer. Sampling-based binary-level cross-platform performance
estimation. InProceedings of the Conference on Design, Automation & Test in Europe, pages 1713–1718. European Design
and Automation Association, 2017.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: January 2018.

A Convolutional Attention Network
for Extreme Summarization of Source Code
Miltiadis Allamanis M.ALLAMANIS @ED.AC.UK
School of Informatics, University of Edinburgh, Edinburgh, EH8 9AB, United Kingdom
Hao Peng† PENGHAO .PKU @GMAIL .COM
School of Electronics Engineering and Computer Science, Peking University, Beijing 100871, China
Charles Sutton CSUTTON @INF.ED.AC.UK
School of Informatics, University of Edinburgh, Edinburgh, EH8 9AB, United Kingdom
Abstract
Attention mechanisms in neural networks have
proved useful for problems in which the input and
output do not have ﬁxed dimension. Often there
exist features that are locally translation invariant
and would be valuable for directing the model’s
attention, but previous attentional architectures
are not constructed to learn such features speciﬁ-
cally. We introduce an attentional neural network
that employs convolution on the input tokens to
detect local time-invariant and long-range topical
attention features in a context-dependent way. We
apply this architecture to the problem of extreme
summarization of source code snippets into short,
descriptive function name-like summaries. Using
those features, the model sequentially generates
a summary by marginalizing over two attention
mechanisms: one that predicts the next summary
token based on the attention weights of the input
tokens and another that is able to copy a code
token as-is directly into the summary. We demon-
strate our convolutional attention neural network’s
performance on 10 popular Java projects showing
that it achieves better performance compared to
previous attentional mechanisms.
1. Introduction
Deep learning for structured prediction problems, in which
a sequence (or more complex structure) of predictions need
to be made given an input sequence, presents special dif-
ﬁculties, because not only are the input and output high-
†Work partially done while author was as an intern at the
University of Edinburgh.
Proceedings of the 33 rd International Conference on Machine
Learning, New York, NY , USA, 2016. JMLR: W&CP volume 48.
Copyright 2016 by the author(s).
dimensional, but the dimensionality is not ﬁxed in advance.
Recent research has tackled these problems using neural
models of attention (Mnih et al., 2014), which have had
great recent successes in machine translation (Bahdanau
et al., 2015) and image captioning (Xu et al., 2015). Atten-
tional models have been successful because they separate
two different concerns: predicting which input locations are
most relevant to each location of the output; and actually
predicting an output location given the most relevant inputs.
In this paper, we suggest that many domains contain
translation-invariant features that can help to determine
the most useful locations for attention. For example, in
a research paper, the sequence of words “in this paper, we
suggest” often indicates that the next few words will be
important to the topic of the paper. As another example,
suppose a neural network is trying to predict the name of
a method in the Java programming language from its body.
If we know that this method name begins with get and
the method body contains a statement return ____ ; ,
then whatever token ﬁlls in the blank is likely to be useful
for predicting the rest of the method name. Previous ar-
chitectures for neural attention are not constructed to learn
translation-invariant features speciﬁcally.
We introduce a neural convolutional attentional model, that
includes a convolutional network within the attention mech-
anism itself. Convolutional models are a natural choice for
learning translation-invariant features while using only a
small number of parameters and for this reason have been
highly successful in non-attentional models for images (Le-
Cun et al., 1998; Krizhevsky et al., 2012) and text clas-
siﬁcation (Blunsom et al., 2014). But to our knowledge
they have not been applied within an attentional mechanism.
Our model uses a set of convolutional layers — without
any pooling — to detect patterns in the input and identify
“interesting” locations where attention should be focused.
We apply this network to an “extreme” summarization prob-
A Convolutional Attention Network for Extreme Summarization of Source Code
lem: We ask the network to predict a short and descriptive
name of a source code snippet (e.g. a method body) given
solely its tokens. Source code has two distinct roles: it not
only is a means of instructing a CPU to perform a computa-
tion but also acts as an essential means of communication
among developers who need to understand, maintain and
evolve software systems. For these reasons, software engi-
neering research has found that good names are important to
developers (Liblit et al., 2006; Takang et al., 1996; Binkley
et al., 2013). Additionally, learning to summarize source
code has important applications in software engineering,
such as in code understanding and in code search. The
highly structured form of source code makes convolution
naturally suited for the purpose of extreme summarization.
Our choice of problem is inspired by previous work (Al-
lamanis et al., 2015a) that tries to name existing methods
(functions) using a large set of hard-coded features, such
as features from the containing class and the method signa-
ture. But these hard-coded features may not be available
for arbitrary code snippets and in dynamically typed lan-
guages. In contrast, in this paper we consider a more general
problem: given an arbitrary snippet of code — without any
hard-coded features — provide a summary, in the form of a
descriptive method name.
This problem resembles a summarization task, where the
method name is viewed as the summary of the code. How-
ever, extreme source code summarization is drastically dif-
ferent from natural language summarization, because unlike
natural language, source code is unambiguous and highly
structured. Furthermore, a good summary needs to explain
how the code instructions compose into a higher-level mean-
ing and not naïvely explain what the code does. This ne-
cessitates learning higher-level patterns in source code that
uses both the structure of the code and the identiﬁers to
detect and explain complex code constructs. Our extreme
summarization problem may also be viewed as a translation
task, in the same way that any summarization problem can
be viewed as translation. But a signiﬁcant difference from
translation is that the input source code sequence tends to be
very large (72 on average in our data) and the output sum-
mary very small (3 on average in our data). The length of
the input sequence necessitates the extraction of both tempo-
rally invariant attention features and topical sentence-wide
features and — as we show in this paper — existing neural
machine translation techniques yield sub-optimal results.
Furthermore, source code presents the challenge of out-of-
vocabulary words. Each new software project and each new
source ﬁle introduces new vocabulary about aspects of the
software’s domain, data structures, and so on. This vocabu-
lary often does not appear in the training set. To address this
problem, we introduce a copy mechanism, which uses the
convolutional attentional mechanism to identify important
tokens in the input even if they are out-of-vocabulary tokens
that do not appear in the training set. The decoder, using
Emt
Kl1
Dw1
RELU
Kl2
k1
w2
ht−1
k2
k2
Kα
Kκ
SOFT MAX
Lfeat attention features
α
κ
attention weight vectorsattention weight vectors
Figure 1.The architecture of the convolutional attentional net-
work. attention_features learns location-speciﬁc attention fea-
tures given an input sequence {mi} and a context vector ht−1.
Given these features attention_weights —using a convolutional
layer and a SOFT MAX— computes the ﬁnal attention weight vec-
tors such as α and κ in this ﬁgure.
a meta-attention mechanism, may choose to copy tokens
directly from the input to the output sequence, resembling
the functionality of Vinyals et al. (2015).
The key contributions of our paper are: (a) a novel con-
volutional attentional network that successfully performs
extreme summarization of source code; (b) a comprehen-
sive approach to the extreme code summarization problem,
with interest both in the machine learning and software engi-
neering community; and (c) a comprehensive evaluation of
four competing algorithms on real-world data that demon-
strates the advantage of our method compared to standard
attentional mechanisms.
2. Convolutional Attention Model
Our convolutional attentional model receives as input a
sequence of code subtokens 1 c = [c<S>,c1,...c N,c</S>]
and outputs an extreme summary in the form of a con-
cise method name. The summary is a sequence of subto-
kens m = [m<s>,m1 ...m M,m</s>], where <s> and </s>
are the special start and end symbols of every subtoken
sequence. For example, in the shouldRender method
(top left of Table 3) the input code subtokens are c =
[<S>,try,{,return,render,requested,... ] while the tar-
1Subtokens refer to the parts of a source code token e.g.
getInputStream has the get, Input and Stream subtokens.
A Convolutional Attention Network for Extreme Summarization of Source Code
get output is m = [<s>,should,render,</s>]. The neural
network predicts each summary subtoken sequentially and
models P(mt|m<s>,...,m t−1,c). Information about the
previously produced subtokens m<s>,...,m t−1 is passed
into a recurrent neural network that represents the input state
with a vector ht−1. Our convolutional attentional neural
network (Figure 1) uses the input state ht−1 and a series of
convolutions over the embeddings of the tokens c to com-
pute a matrix of attention features Lfeat, (Figure 1) that
contains one vector of attention features for each sequence
position. The resulting features are used to compute one
or more normalized attention vectors ( e.g. α in Figure 1)
which are distributions over input token locations containing
a weight (in R(0,1)) for each subtoken in c. Finally, given
the weights, a context representation is computed and is
used to predict the probability distribution over the targets
mi. This model is a generative bimodal model of summary
text given a code snippet.
2.1. Learning Attention Features
We describe our model from the bottom-up (Figure 1). First
we discuss how to compute the attention featuresLfeat from
the input c and the previous hidden state ht−1. The basic
building block of our model is a convolutional network (Le-
Cun et al., 1990; Collobert & Weston, 2008) for extracting
position and context-dependent features. The input to atten-
tion_features is a sequence of code subtokens c of length
LEN(c) and each location is mapped to a matrix of attention
features Lfeat, with size (LEN(c) +const) ×k2 where the
constis a ﬁxed amount of padding. The intuition behind
attention_features is that given the inputc, it uses convolu-
tion to compute k2 features for each location. By then using
ht−1 as a multiplicative gating-like mechanism, only the
currently relevant features are kept in L2. In the ﬁnal stage,
we normalize L2. attention_features is described with the
following pseudocode:
attention_features (code tokens c, context ht−1)
C ←LOOKUP ANDPAD(c, E)
L1 ←RELU(C ONV 1D(C, Kl1))
L2 ←CONV 1D(L1,Kl2) ⊙ht−1
Lfeat ←L2/∥L2∥2
return Lfeat
Here E ∈ R|V|×D contains the D dimensional embed-
ding of each subtoken in names and code ( i.e. all possi-
ble cis and mis). The two convolution kernels are Kl1 ∈
RD×w1×k1 and Kl2 ∈Rk1×w2×k2 , where w1, w2 are the
window sizes of the convolutions and RELU refers to a
rectiﬁed linear unit (Nair & Hinton, 2010). The vector
ht−1 ∈Rk2 represents information from the previous subto-
kens m0 ...m t−1. CONV 1D performs a one-dimensional
(throughout the length of sentence c) narrow convolution.
Note that the input sequence c is padded by LOOKUP AND-
PAD. The size of the padding is such that with the nar-
row convolutions, the attention vector (returned by atten-
tion_weights) has exactly LEN(c) components. The ⊙op-
erator is the elementwise multiplication of a vector and a
matrix, i.e. B = A⊙v for v ∈RM and Aa M×N matrix,
Bij = Aijvi. We found the normalization of L2 into Lfeat
to be useful during training. We believe it helps because of
the widely varying lengths of inputs c. Note that no pooling
happens in this model; the input sequence c is of the same
length as the output sequence (modulo the padding).
To compute the ﬁnal attention weight vector — a vector
with non-negative elements and unit norm — we deﬁne
attention_weights as a function that accepts Lfeat from
attention_features and a convolution kernelK of size k2 ×
w3 ×1. attention_weights returns the normalized attention
weights vector with length LEN(c) and is described by the
following pseudocode:
attention_weights (attention features Lfeat, kernel K)
return SOFT MAX(CONV 1D(Lfeat,K))
Computing the Stateht. Predicting the full summary m
is a sequential prediction problem, where each subtoken mt
is sequentially predicted given the previous state containing
information about the previous subtokensm0 ...m t−1. The
state is passed through ht ∈Rk2 computed by a Gated
Recurrent Unit (Cho et al., 2014)i.e.
GRU(current input xt, previous state ht−1)
rt ←σ(xtWxr + ht−1Whr + br)
ut ←σ(xtWxu + ht−1Whu + bu)
ct ←tanh(xtWxc + rt ⊙(ht−1Whc) +bc)
ht ←(1 −ut) ⊙ht−1 + ut ⊙ct
return ht
During testing the next state is computed by ht =
GRU(Emt ,ht−1) i.e. using the embedding of the current
output subtoken mt. For regularization during training, we
use a trick similar to Bengio et al. (2015) and with proba-
bility equal to the dropout rate we compute the next state as
ht = GRU(ˆ n,ht−1), where ˆ nis the predicted embedding.
2.2. Simple Convolutional Attentional Model
We now use the components described above as building
blocks for our extreme summarization model. We ﬁrst build
conv_attention, a convolutional attentional model that uses
an attention vector α computed from attention_weights to
weight the embeddings of the tokens in c and compute the
predicted target embeddingˆ n∈RD. It returns a distribution
over all subtokens in V.
conv_attention (code c, previous state ht−1)
Lfeat ←attention_features(c,ht−1)
α ←attention_weights (Lfeat,Katt)
ˆ n←∑
iαiEci
n ←SOFT MAX(Eˆ n⊤+ b)
return TOMAP(n,V )
where b ∈ R|V| is a bias vector and TOMAP returns a
map of each subtoken in vi ∈V associated with its proba-
A Convolutional Attention Network for Extreme Summarization of Source Code
bility ni. We train this model using maximum likelihood.
Generating from the model works as follows: starting with
the special m0 = <s> subtoken and h0, at each timestep
t the next subtoken mt is generated using the probabil-
ity distribution n returned by conv_attention (c,ht−1).
Given the new subtoken mt, we compute the next state
ht = GRU(Emt ,ht−1). The process stops when the spe-
cial </s> subtoken is generated.
2.3. Copy Convolutional Attentional Model
We extend conv_attention by using an additional attention
vector κ as a copying mechanism that can suggest out-of-
vocabulary subtokens. In our data a signiﬁcant proportion
of the output subtokens (about 35%) appear in c. Motivated
by this, we extend conv_attention and allow a direct copy
from the input sequence c into the summary. Now the
network when predicting mt, with probability λcopies a
token from c into mt and with probability 1−λpredicts the
target subtoken as in conv_attention. Essentially, λacts as
a meta-attention. When copying, a token ci is copied into
mt with probability equal to the attention weight κi. The
process is the following:
copy_attention (code c, previous state ht−1)
Lfeat ←attention_features (c,ht−1)
α ←attention_weights (Lfeat,Katt)
κ ←attention_weights (Lfeat,Kcopy)
λ←max(σ(CONV 1D(Lfeat,Kλ)))
ˆ n←∑
iαiEci
n ←SOFT MAX(Eˆ n⊤+ b)
return λPOS2VOC(κ,c) + (1−λ)TOMAP(n,V )
where σ is the sigmoid function, Katt, Kcopy and Kλ are
different convolutional kernels,n ∈R|V|, α,κ ∈RLEN(c),
POS2VOC returns a map of each subtoken in c (which may
include out-of-vocabulary tokens) to the probabilities κi
assigned by the copy mechanism. Finally, the predictions
of the two attention mechanisms are merged, returning a
map that contains all potential target subtokens inV ∪c and
interpolating over the two attention mechanisms, using the
meta-attention weight λ. Note that α and κ are analogous
attention weights but are computed from different kernels,
and that n is computed exactly as in conv_attention.
Objective. To obtain signal for the copying mechanism
and λ, we input to copy_attention a binary vector Ic=mt
of size LEN(c) where each component is one if the code
subtoken is identical to the current target subtoken mt. We
can then compute the probability of a correct copy over the
marginalization of the two mechanisms, i.e.
P(mt|ht−1,c) =λ
∑
i
κiIci=mt + (1−λ)µrmt
where the ﬁrst term is the probability of a correct copy
(weighted by λ) and the second term is the probability of
the target token mt (weighted by 1 −λ). We use µ∈(0,1]
to penalize the model when the simple attention predicts
an UNK but the subtoken can be predicted exactly by the
copy mechanism, otherwise µ = 1. We arbitrarily used
µ= e−10, although variations did not affect performance.
2.4. Predicting Names
To predict a full method name, we use a hybrid breath-ﬁrst
search and beam search. We start from the specialm0 = <s>
subtoken and maintain a (max-)heap of the highest proba-
bility partial predictions so far. At each step, we pick the
highest probability prediction and predict its next subtokens,
pushing them back to the heap. When the </s> subtoken is
generated the suggestion is moved onto the list of sugges-
tions. Since we are interested in the top ksuggestions, at
each point, we prune partial suggestions that have a proba-
bility less than the current best kth full suggestion. To make
the process tractable, we limit the partial suggestion heap
size and stop iterating after 100 steps.
3. Evaluation
Dataset Collection. We are interested in the extreme sum-
marization problem where we summarize a source code
snippet into a short and concise method-like name. Al-
though such a dataset does not exist for arbitrary snippets of
source code, it is natural to consider existing method (func-
tion) bodies as our snippets and the method names picked
by the developers as our target extreme summaries.
To collect a good dataset of good quality, we cloned 11
open source Java projects from GitHub. We obtained the
most popular projects by taking the sum of the z-scores of
the number of watchers and forks of each project, using
GHTorrent (Gousios & Spinellis, 2012). We selected the
top 11 projects that contained more than 10MB of source
code ﬁles each and use libgdx as a development set. These
projects have thousands of forks and stars, being widely
known among software developers. The projects along
with short descriptions are shown in Table 1. We used this
procedure to select a mature, large, and diverse corpus of
real source code. For each ﬁle, we extract the Java methods,
removing methods that are overridden, are abstract or are
the constructors of a class. We ﬁnd the overridden methods
by an approximate static analysis that checks for inheritance
relationships and the @Override annotation. Overridden
methods are removed, since they are highly repetitive and
their names are easy to predict. Any full tokens that are
identical to the method name (e.g. in recursion) are replaced
with a special SELF token. We split and lowercase each
method name and code token into subtokens {mi}and {ci}
on camelCase and snake_case. The dataset and code
can be found at groups.inf.ed.ac.uk/cup/codeattention.
Experimental Setup. To measure the quality of our sug-
gestions we compute two scores. Exact match is the per-
centage of the method names predicted exactly, while the
A Convolutional Attention Network for Extreme Summarization of Source Code
Project Name Git SHA Description
F1
tf-idf Standard Attention conv_attention copy_attention
Rank 1 Rank 5 Rank 1 Rank 5 Rank 1 Rank 5 Rank 1 Rank 5
cassandra 53e370f Distributed Database 40.9 52.0 35.1 45.0 46.5 60.0 48.1 63.1
elasticsearch 485915b REST Search Engine 27.8 39.5 20.3 29.0 30.8 45.0 31.7 47.2
gradle 8263603 Build System 30.7 45.4 23.1 37.0 35.3 52.5 36.3 54.0
hadoop-common 42a61a4 Map-Reduce Framework 34.7 48.4 27.0 45.7 38.0 54.0 38.4 55.8
hibernate-orm e65a883 Object/Relational Mapping 53.9 63.6 49.3 55.8 57.5 67.3 58.7 69.3
intellij-community d36c0c1 IDE 28.5 42.1 23.8 41.1 33.1 49.6 33.8 51.5
liferay-portal 39037ca Portal Framework 59.6 70.8 55.4 70.6 63.4 75.5 65.9 78.0
presto 4311896 Distributed SQL query engine 41.8 53.2 33.4 41.4 46.3 59.0 46.7 60.2
spring-framework 826a00a Application Framework 35.7 47.6 29.7 41.3 35.9 49.7 36.8 51.9
wildﬂy c324eaa Application Server 45.2 57.7 32.6 44.4 45.5 61.0 44.7 61.7
Table 1.Open source Java projects used and F1 scores achieved. Standard attention refers to the model of Bahdanau et al. (2015).
F1 score is computed in a per-subtoken basis. When sug-
gesting summaries, each model returns a ranked list. We
compute exact match and F1 at rank 1 and 5, as the best
score achieved by any one of the top suggestions (i.e. if the
ﬁfth suggestion achieves the best F1 score, we use this one
for computing F1 at rank 5). Using BLEU (Papineni et al.,
2002) would have been possible, but it would not be differ-
ent from F1 given the short lengths of our output sequences
(3 on average). We use each project separately, training one
network for each project and testing on the respective test
set. This is because each project’s domain varies widely
and little information can be transferred among them, due
to the principle of code reusability of software engineering.
We note that we attempted to train a single model using
all project training sets but this yielded signiﬁcantly worse
results for all algorithms. For each project, we split the ﬁles
(top-level Java classes) uniformly at random into training
(65%), validation (5%) and test (30%) sets. We optimize hy-
perparameters using Bayesian optimization with Spearmint
(Snoek et al., 2012) maximizing F1 at rank 5.
For comparison, we use two algorithms: a tf-idf algorithm
that computes a tf-idf vector from the code snippet subto-
kens and suggests the names of the nearest neighbors using
cosine similarity. We also use the standard attention model
of Bahdanau et al. (2015) that uses a biRNN and fully con-
nected components, that has been successfully used in ma-
chine translation. We perform hyperparameter optimizations
following the same protocol on libgdx.
Training. To trainconv_attention and copy_attention we
optimize the objective using stochastic gradient descent with
RMSProp and Nesterov momentum (Sutskever et al., 2013;
Hinton et al., 2012). We use dropout (Srivastava et al., 2014)
on all parameters, parametric leaky RELUs (Maas et al.,
2013; He et al., 2015) and gradient clipping. Each of the
parameters of the model is initialized with normal random
noise around zero, except forb that is initialized to the log of
the empirical frequency of each target token in the training
set. For conv_attention the optimized hyperparameters are
k1 = k2 = 8, w1 = 24, w2 = 29, w3 = 10, dropout
rate 50% and D= 128. For copy_attention the optimized
hyperparameters are k1 = 32, k2 = 16, w1 = 18, w2 = 19,
w3 = 2, dropout rate 40% and D= 128.
3.1. Quantitative Evaluation
Table 1 shows the F1 scores achieved by the different meth-
ods for each project while Table 2 shows a quantitative
evaluation, averaged across all projects. “Standard Atten-
tion” refers to the machine translation model of Bahdanau
et al. (2015). The tf-idf algorithm seems to be performing
very well, showing that the bag-of-words representation of
the input code is a strong indicator of its name. Interestingly,
the standard attention model performs worse than tf-idf
in this domain, while conv_attention and copy_attention
perform the best. The copy mechanism gives a good F1
improvement at rank 1 and a larger improvement at rank
5. Although our convolutional attentional models have an
exact match similar to tf-idf, they achieve a much higher
precision compared to all other algorithms.
These differences in the data characteristics could be the
cause of the low performance achieved by the model of
Bahdanau et al. (2015). Although source code snippets re-
semble natural language sentences, they are more structured,
much longer and vary greatly in length. In our training sets,
each method has on average 72 tokens (median 25 tokens,
standard deviation 156) and the output method names are
made up from 3 subtokens on average (σ= 1.7).
OOV Accuracy. We measure the out-of-vocabulary (OOV)
word accuracy as the percentage of the out-of-vocabulary
subtokens that are correctly predicted by copy_attention.
On average, across our dataset, 4.4% of the test method
name subtokens are OOV. Naturally, the standard attention
model and tf-idf have an OOV accuracy of zero, since they
are unable to predict those tokens. On average we get a
10.5% OOV accuracy at rank 1 and 19.4% at rank 5. This
shows that the copying mechanism is useful in this domain
and especially in smaller projects that tend to have more
OOV tokens. We also note that OOV accuracy varies across
projects, presumably due to different coding styles.
Topical vs. Time-Invariant Feature Detection. The dif-
ference of the performance between the copy_attention
and the standard attention model of Bahdanau et al. (2015)
A Convolutional Attention Network for Extreme Summarization of Source Code
F1 (%) Exact Match (%) Precision (%) Recall (%)
Rank 1 Rank 5 Rank 1 Rank 5 Rank 1 Rank 5 Rank 1 Rank 5
tf-idf 40.0 52.1 24.3 29.3 41.6 55.2 41.8 51.9
Standard Attention 33.6 45.2 17.4 24.9 35.2 47.1 35.1 42.1
conv_attention 43.6 57.7 20.6 29.8 57.4 73.7 39.4 51.9
copy_attention 44.7 59.6 23.5 33.7 58.9 74.9 40.1 54.2
Table 2.Evaluation metrics averaged across projects. Standard Attention refers to the work of Bahdanau et al. (2015).
raises an interesting question. What does copy_attention
learn that cannot be learned by the standard attention model?
One hypothesis is that the biRNN of the standard attention
model fails to capture long-range features, especially in very
long inputs. To test our hypothesis, we shufﬂe the subtokens
in libgdx, essentially removing all features that depend on
the sequential information. Without any local features all
models should reduce to achieving performance similar to
tf-idf. Indeed, copy_attention now has an F1 at rank 1 that
is +1% compared to tf-idf (presumably thanks to the lan-
guage model-like structure of the output), while the standard
attention model worsens its performance getting an F1 score
(rank 1) of 26.2%, compared to the original 41.8%. This
suggests that the biRNN fails to capture long-range topical
attention features.
A simpler ht−1. Since the target summaries are quite
short, we tested a simpler alternative to the GRU, assigning
ht−1 = W ×[Gmt−1 ,Gmt−2 ], where G ∈RD×|V|is a
new embedding matrix (different from the embeddings in
E) and W is a k2 ×D×2 tensor. This model is simpler
and slightly faster to train and achieves similar performance
to copy_attention, reducing F1 by less than 1%.
3.2. Qualitative Evaluation
Figure 2 shows a visualization of a small method that il-
lustrates how copy_attention typically works. At the ﬁrst
step, it focuses its attention at the whole method and decides
upon the ﬁrst subtoken. In a large number of cases this
includes subtokens such as get, set, is, create etc. In
the next steps the meta-attention mechanism is highly conﬁ-
dent about the copying mechanism (λ= 0.97 in Figure 2)
and sequentially copies the correct subtokens from the code
snippet into the name. We note that across many examples
the copying mechanism tends to have a signiﬁcantly more
focused attention vector κ, compared to the attention vec-
tor α. Presumably, this happens because of the different
training signals of the attention mechanisms.
A second example of copy_attention is seen in Figure 3.
Although due to space limitations this is a relatively short
method, it illustrates how the model has learned both time-
invariant features and topical features. It correctly detects
the == operator and predicts that the method has a high
probability of starting withis. Furthermore, in the next step
(prediction of the m2 bullets subtoken) it successfully
learns to ignore the e preﬁx (prepended on all enumeration
variables in that project) and the flag subtoken that does
not provide useful information for the summary.
Table 3 presents a set of hand-picked examples from
libgdx that show interesting challenges of the domain
and how our copy_attention handles them. Understand-
ably, the model does not distinguish between should and
is — both implying a boolean return value — and in-
stead of shouldRender, isRender is suggested. The
getAspectRatio, surfaceArea and minRunLength
examples show the challenges of describing a previously
unseen abstraction. Interestingly, the model correctly recog-
nizes that a novel (UNK) token should be predicted afterget
in getAspectRatio. Most surprisingly, reverseRange
is predicted correctly, because of the structure of the code,
even though no code tokens contain the summary subto-
kens.
4. Related Work
Convolutional neural networks have been used for image
classiﬁcation with great success (Krizhevsky et al., 2012;
Szegedy et al., 2015; LeCun et al., 1990; 1998). More
related to this work is the use of convolutional neural net-
works for text classiﬁcation (Blunsom et al., 2014). Closely
related is the work of Denil et al. (2014) that learns rep-
resentations of documents using convolution but uses the
network activations to summarize a document rather than an
attentional model. Rush et al. (2015) use an attention-based
encoder to summarize sentences, but do not use convolution
for their attention mechanism. Our work is also related
to other work in attention mechanisms for text (Hermann
et al., 2015) and images (Xu et al., 2015; Mnih et al., 2014)
that does not use convolution to provide the attention val-
ues. Pointer networks (Vinyals et al., 2015) are similar to
our copy mechanism but use an RNN for providing atten-
tion. Finally, distantly related to this work is research on
neural architectures that learn code-like behaviors (Graves
et al., 2014; Zaremba & Sutskever, 2014; Joulin & Mikolov,
2015; Grefenstette et al., 2015; Dyer et al., 2015; Reed &
de Freitas, 2015; Neelakantan et al., 2015).
In recent years, thanks to the insight of Hindle et al. (2012)
the use of probabilistic models for software engineering ap-
plications has grown. Research has mostly focused on token-
level (Nguyen et al., 2013; Tu et al., 2014) and syntax-level
(Maddison & Tarlow, 2014) language models of code and
translation between programming languages (Karaivanov
A Convolutional Attention Network for Extreme Summarization of Source Code
boolean shouldRender() void reverseRange(Object[] a, int lo, int hi)
try {
return renderRequested||isContinuous;
} finally {
renderRequested = false;
}
hi--;
while (lo < hi) {
Object t = a[lo];
a[lo++] = a[hi];
a[hi--] = t;
}
Suggestions: ▶is,render (27.3%)
▶is,continuous (10.6%) ▶is,requested (8.2%)
▶render,continuous (6.9%) ▶get,render (5.7%)
Suggestions: ▶reverse,range (22.2%) ▶reverse (13.0%)
▶reverse,lo (4.1%) ▶reverse,hi (3.2%)
▶merge,range (2.0%)
int createProgram() VerticalGroup right()
GL20 gl = Gdx.gl20;
int program = gl.glCreateProgram();
return program != 0 ? program : -1;
align |= Align.right;
align &= ~Align.left;
return this;
Suggestions: ▶create (18.36%) ▶init (7.9%)
▶render (5.0%) ▶initiate (5.0%) ▶load (3.4%)
Suggestions: ▶left (21.8%) ▶top (21.1%) ▶right (19.5%)
▶bottom (18.5%) ▶align (3.7%)
boolean isBullet() float getAspectRatio()
return (m_flags & e_bulletFlag)
== e_bulletFlag;
return (height == 0) ?
Float.NaN : width / height;
Suggestions: ▶is (13.5%) ▶is,bullet (5.5%)
▶is,enable (5.1%) ▶enable (2.8%) ▶mouse (2.7%)
Suggestions: ▶get,UNK (9.0%) ▶get,height (8.7%)
▶get,width (6.5%) ▶get (5.7%) ▶get,size (4.2%)
int minRunLength(int n) JsonWriter pop()
if (DEBUG) assert n >= 0;
int r = 0;
while (n >= MIN_MERGE) {
r |= (n & 1);
n >>= 1;
}
return n + r;
if (named) throw
new IllegalStateException(UNKSTRING );
stack.pop().close();
current = stack.size == 0 ?
null : stack.peek();
return this;
Suggestions: ▶min (43.7%) ▶merge (13.0%)
▶pref (1.9%) ▶space (1.0%) ▶min,all (0.8%)
Suggestions: ▶close (21.4%) ▶pop (10.2%) ▶first (6.5%)
▶state (3.8%) ▶remove (2.2%)
Rectangle setPosition(float x, float y) float surfaceArea()
this.x = x;
this.y = y;
return this;
return 4 * MathUtils.PI *
this.radius * this.radius;
Suggestions: ▶set (54.0%) ▶set,y (12.8%)
▶set,x (9.0%) ▶set,position (8.6%)
▶set,bounds (1.68%)
Suggestions: ▶dot,radius (26.5%) ▶dot (13.1%)
▶crs,radius (9.0%) ▶dot,circle (6.5%) ▶crs (4.1%)
Table 3.A sample of handpicked snippets (the sample is necessarily limited to short methods because of space limitations) and the
respective suggestions that illustrate some interesting challenges of the domain and how the copy_attention model handles them or
fails. Note that the algorithms do not have access to the signature of the method but only to the body. Examples taken from the libgdx
Android/Java graphics library test set.
A Convolutional Attention Network for Extreme Summarization of Source Code
Target Attention Vectors λ
m1 set α = <s> { this . use Browser Cache = use Browser Cache ; } </s> 0.012κ = <s> { this . use Browser Cache = use Browser Cache ; } </s>
m2 use α = <s> { this . use Browser Cache = use Browser Cache ; } </s> 0.974κ = <s> { this . use Browser Cache = use Browser Cache ; } </s>
m3 browser α = <s> { this . use Browser Cache = use Browser Cache ; } </s> 0.969κ = <s> { this . use Browser Cache = use Browser Cache ; } </s>
m4 cache α = <s> { this . use Browser Cache = use Browser Cache ; } </s> 0.583κ = <s> { this . use Browser Cache = use Browser Cache ; } </s>
m5 END α = <s> { this . use Browser Cache = use Browser Cache ; } </s> 0.066κ = <s> { this . use Browser Cache = use Browser Cache ; } </s>
Figure 2.Visualization of copy_attention used to compute P(mt|m0 ...m t−1,c) for setUseBrowserCache in libgdx. The darker the
color of a subtoken, they higher its attention weight. This relationship is linear. Yellow indicates the convolutional attention weight of the
conv_attention component, while purple the attention of the copy mechanism. Since the values of α are usually spread across the tokens
the colors show a normalized α, i.e. α/∥α∥∞. In contrast, the copy attention weights κ are usually very peaky and we plot them as-is.
Underlined subtokens are out-of-vocabulary. λshows the meta-attention probability of using the copy attention κ vs. the convolutional
attention α. More visualizations of libgdx methods can be found at http://groups.inf.ed.ac.uk/cup/codeattention/.
Target Attention Vectors λ
m1 is α = <s> { return ( m Flags & e Bullet Flag ) == e Bullet Flag ; } </s> 0.012κ = <s> { return ( m Flags & e Bullet Flag ) == e Bullet Flag ; } </s>
m2 bullet α = <s> { return ( m Flags & e Bullet Flag ) == e Bullet Flag ; } </s> 0.436κ = <s> { return ( m Flags & e Bullet Flag ) == e Bullet Flag ; } </s>
m3 END α = <s> { return ( m Flags & e Bullet Flag ) == e Bullet Flag ; } </s> 0.174κ = <s> { return ( m Flags & e Bullet Flag ) == e Bullet Flag ; } </s>
Figure 3.Visualization of copy_attention modeling P(mt|m0 ...m t−1,c) for isBullet in libgdx. The copy_attention captures
location-invariant features and the topicality of the input code sequence. For information about the visualization see Figure 2.
et al., 2014; Nguyen et al., 2014). Movshovitz-Attias et al.
(2013) learns to predict code comments using a source code
topic model. Allamanis et al. (2015b) create a generative
model of source code given a natural language query and
Oda et al. (2015) use machine translation to convert source
code into pseudocode. Closer to our work, Raychev et al.
(2015) aim to predict names and types of variables, whereas
Allamanis et al. (2014) and Allamanis et al. (2015a) suggest
names for variables, methods and classes. Similar to Alla-
manis et al. (2015a), we predict method names but using
only the tokens within a method and no other features (e.g.
method signature). Mou et al. (2016) use syntax-level con-
volutional neural networks to learn vector representations
for code and classify student submissions into tasks without
considering naming. Piech et al. (2015) also learn program
embeddings from student submissions using the program
state, to assist MOOC students debug their submissions but
do not consider naming. Additionally, compared to Piech
et al. (2015) and Mou et al. (2016) our work looks into
highly diverse, non-student submission code that performs
a wide range of real-world tasks.
5. Discussion & Conclusions
Modeling and understanding source code artifacts through
machine learning can have a direct impact in software engi-
neering research. The problem of extreme code summariza-
tion is a ﬁrst step towards the more general goal of devel-
oping machine learning representations of source code that
will allow machine learning methods to reason probabilis-
tically about code resulting in useful software engineering
tools that will help code construction and maintenance.
Additionally, source code — and its derivative artifacts —
represent a new modality for machine learning with very
different characteristics compared to images and natural
language. Therefore, models of source code necessitate
research into new methods that could have interesting par-
allels to images and natural language. This work is a step
towards this direction: our neural convolutional attentional
model attempts to “understand” the highly-structured source
code text by learning both long-range features and local-
ized patterns, achieving the best performance among other
competing methods on real-world source code.
A Convolutional Attention Network for Extreme Summarization of Source Code
Acknowledgements
This work was supported by Microsoft Research through
its PhD Scholarship Programme and the Engineering
and Physical Sciences Research Council [grant number
EP/K024043/1]. We would like to thank Daniel Tarlow
and Krzysztof Geras for their insightful comments and sug-
gestions.
References
Allamanis, Miltiadis, Barr, Earl T, Bird, Christian, and Sut-
ton, Charles. Learning natural coding conventions. In
Symposium on the Foundations of Software Engineering
(FSE), 2014.
Allamanis, Miltiadis, Barr, Earl T, Bird, Christian, and
Sutton, Charles. Suggesting accurate method and class
names. In Proceedings of the 2015 10th Joint Meeting on
Foundations of Software Engineering. ACM, 2015a.
Allamanis, Miltiadis, Tarlow, Daniel, Gordon, Andrew, and
Wei, Yi. Bimodal modelling of source code and natural
language. In ICML, 2015b.
Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua.
Neural machine translation by jointly learning to align
and translate. In ICLR, 2015.
Bengio, Samy, Vinyals, Oriol, Jaitly, Navdeep, and
Shazeer, Noam. Scheduled sampling for sequence pre-
diction with recurrent neural networks. arXiv preprint
arXiv:1506.03099, 2015.
Binkley, Dave, Davis, Marcia, Lawrie, Dawn, Maletic,
Jonathan I, Morrell, Christopher, and Sharif, Bonita. The
impact of identiﬁer style on effort and comprehension.
Empirical Software Engineering, 2013.
Blunsom, Phil, Grefenstette, Edward, Kalchbrenner, Nal,
et al. A convolutional neural network for modelling sen-
tences. In Proceedings of the 52nd Annual Meeting of the
Association for Computational Linguistics, 2014.
Cho, Kyunghyun, van Merriënboer, Bart, Bahdanau,
Dzmitry, and Bengio, Yoshua. On the properties of neural
machine translation: Encoder–decoder approaches. Syn-
tax, Semantics and Structure in Statistical Translation ,
pp. 103, 2014.
Collobert, Ronan and Weston, Jason. A uniﬁed architecture
for natural language processing: deep neural networks
with multitask learning. In ICML, 2008.
Denil, Misha, Demiraj, Alban, Kalchbrenner, Nal, Blun-
som, Phil, and de Freitas, Nando. Modelling, visualising
and summarising documents with a single convolutional
neural network. arXiv preprint arXiv:1406.3830, 2014.
Dyer, Chris, Ballesteros, Miguel, Ling, Wang, Matthews,
Austin, and Smith, Noah A. Transition-based dependency
parsing with stack long short-term memory. In ACL,
2015.
Gousios, Georgios and Spinellis, Diomidis. GHTorrent:
Github’s data from a ﬁrehose. In Mining Software Repos-
itories (MSR), 2012 9th IEEE Working Conference on ,
2012.
Graves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural
turing machines. arXiv preprint arXiv:1410.5401, 2014.
Grefenstette, Edward, Hermann, Karl Moritz, Suleyman,
Mustafa, and Blunsom, Phil. Learning to transduce with
unbounded memory. In NIPS, 2015.
He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun,
Jian. Delving deep into rectiﬁers: surpassing human-level
performance on ImageNet classiﬁcation. arXiv preprint
arXiv:1502.01852, 2015.
Hermann, Karl Moritz, Kocisky, Tomas, Grefenstette, Ed-
ward, Espeholt, Lasse, Kay, Will, Suleyman, Mustafa,
and Blunsom, Phil. Teaching machines to read and com-
prehend. In NIPS, 2015.
Hindle, Abram, Barr, Earl T, Su, Zhendong, Gabel, Mark,
and Devanbu, Premkumar. On the naturalness of software.
In International Conference on Software Engineering
(ICSE), 2012.
Hinton, Geoffrey, Srivastava, Nitsh, and Swersky, Kevin.
Neural networks for machine learning. Online Course at
coursera. org, Lecture, 6, 2012.
Joulin, Armand and Mikolov, Tomas. Inferring algorith-
mic patterns with stack-augmented recurrent nets. arXiv
preprint arXiv:1503.01007, 2015.
Karaivanov, Svetoslav, Raychev, Veselin, and Vechev, Mar-
tin. Phrase-based statistical translation of programming
languages. In Proceedings of the 2014 ACM Interna-
tional Symposium on New Ideas, New Paradigms, and
Reﬂections on Programming & Software. ACM, 2014.
Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E.
ImageNet classiﬁcation with deep convolutional neural
networks. In NIPS, 2012.
LeCun, Y ., Boser, B., Denker, J. S., Howard, R. E., Habbard,
W., Jackel, L. D., and Henderson, D. Nips. chapter
Handwritten Digit Recognition with a Back-propagation
Network. 1990.
LeCun, Yann, Bottou, Leon, Bengio, Yoshua, and Haffner,
Patrick. Gradient-based learning applied to document
recognition. Proceedings of the IEEE, 1998.
A Convolutional Attention Network for Extreme Summarization of Source Code
Liblit, Ben, Begel, Andrew, and Sweetser, Eve. Cognitive
perspectives on the role of naming in computer programs.
In Proceedings of the 18th Annual Psychology of Pro-
gramming Workshop, 2006.
Maas, Andrew L., Hannun, Awni Y ., and Ng, Andrew Y .
Rectiﬁed linear units improve restricted Boltzmann ma-
chines. In ICML Workshop on Deep Learning for Audio,
Speech, and Language Processing (WDLASL), 2013.
Maddison, Chris and Tarlow, Daniel. Structured generative
models of natural source code. In ICML, 2014.
Mnih, V olodymyr, Heess, Nicolas, Graves, Alex, et al. Re-
current models of visual attention. In NIPS, 2014.
Mou, Lili, Li, Ge, Zhang, Lu, Wang, Tao, and Jin, Zhi.
Convolutional neural networks over tree structures for
programming language processing. In Proceedings of the
30th AAAI Conference on Artiﬁcial Intelligence, 2016.
Movshovitz-Attias, Dana, Cohen, WW, and W. Cohen,
William. Natural language models for predicting pro-
gramming comments. In ACL, 2013.
Nair, Vinod and Hinton, Geoffrey E. Rectiﬁed linear units
improve restricted Boltzmann machines. In ICML, 2010.
Neelakantan, Arvind, Le, Quoc V , and Sutskever, Ilya. Neu-
ral programmer: Inducing latent programs with gradient
descent. arXiv preprint arXiv:1511.04834, 2015.
Nguyen, Anh Tuan, Nguyen, Tung Thanh, and Nguyen,
Tien N. Migrating code with statistical machine transla-
tion. In Companion Proceedings of the 36th International
Conference on Software Engineering. ACM, 2014.
Nguyen, Tung Thanh, Nguyen, Anh Tuan, Nguyen,
Hoan Anh, and Nguyen, Tien N. A statistical seman-
tic language model for source code. In Joint Meeting
of the European Software Engineering Conference and
the ACM SIGSOFT Symposium on the Foundations of
Software Engineering (ESEC/FSE), 2013.
Oda, Yusuke, Fudaba, Hiroyuki, Neubig, Graham, Hata,
Hideaki, Sakti, Sakriani, Toda, Tomoki, and Nakamura,
Satoshi. Learning to generate pseudo-code from source
code using statistical machine translation. In Automated
Software Engineering (ASE), 2015 30th IEEE/ACM Inter-
national Conference on. IEEE, 2015.
Papineni, Kishore, Roukos, Salim, Ward, Todd, and Zhu,
Wei-Jing. BLEU: a method for automatic evaluation of
machine translation. In Proceedings of the 40th Annual
Meeting of the Association for Computational Linguistics,
2002.
Piech, Chris, Huang, Jonathan, Nguyen, Andy, Phulsuk-
sombati, Mike, Sahami, Mehran, and Guibas, Leonidas J.
Learning program embeddings to propagate feedback on
student code. In ICML, 2015.
Raychev, Veselin, Vechev, Martin, and Krause, Andreas.
Predicting program properties from “big code”. In Pro-
ceedings of the 42nd Annual ACM SIGPLAN-SIGACT
Symposium on Principles of Programming Languages .
ACM, 2015.
Reed, Scott and de Freitas, Nando. Neural programmer-
interpreters. arXiv preprint arXiv:1511.06279, 2015.
Rush, Alexander M, Chopra, Sumit, and Weston, Jason. A
neural attention model for abstractive sentence summa-
rization. In EMNLP, 2015.
Snoek, Jasper, Larochelle, Hugo, and Adams, Ryan P. Practi-
cal bayesian optimization of machine learning algorithms.
In NIPS, 2012.
Srivastava, Nitish, Hinton, Geoffrey E., Krizhevsky, Alex,
Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout: a
simple way to prevent neural networks from overﬁtting.
Journal of Machine Learning Research, 2014.
Sutskever, Ilya, Martens, James, Dahl, George, and Hin-
ton, Geoffrey. On the importance of initialization and
momentum in deep learning. In ICML, 2013.
Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet,
Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru,
Vanhoucke, Vincent, and Rabinovich, Andrew. Going
deeper with convolutions. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition,
2015.
Takang, Armstrong A, Grubb, Penny A, and Macredie,
Robert D. The effects of comments and identiﬁer names
on program comprehensibility: an experimental investi-
gation. J. Prog. Lang., 1996.
Tu, Zhaopeng, Su, Zhendong, and Devanbu, Premkumar.
On the localness of software. In Proceedings of the 22nd
ACM SIGSOFT International Symposium on Foundations
of Software Engineering. ACM, 2014.
Vinyals, Oriol, Fortunato, Meire, and Jaitly, Navdeep.
Pointer networks. In NIPS, 2015.
Xu, Kelvin, Ba, Jimmy, Kiros, Ryan, Cho, Kyunghyun,
Courville, Aaron C, Salakhutdinov, Ruslan, Zemel,
Richard S, and Bengio, Yoshua. Show, attend and tell:
Neural image caption generation with visual attention. In
ICML, 2015.
Zaremba, Wojciech and Sutskever, Ilya. Learning to execute.
arXiv preprint arXiv:1410.4615, 2014.

Verification Submitted Institute Data for Atal Ranking of Institutions on Innovation Achievements (ARIIA) - 2020-21
Institute Name: V. S. B. Engineering College [ARI-C-36944]
List all the departments / centers / units are part of innovation and startup ecosystem of the institite
Srno Department / Center / Unit Name Type of Department /
Center / Unit
Mention the Stream /
Discipline / Sector of
focus
Department / Center /
Unit Level Program Type Student / Benificiary
Strength (2019-20)
Academic Staff
Strength (2019-20)
Non-academic Staff
Strength (2019-20)
1 Tessolve Lab Industry Institute
Interface Unit
Mutidisciplinary UG / PG / PhD Technical 10 5 1
2 Design Lab Academic
Department with lab
facility
Multidisciplinary UG / PG / PhD Technical 101 3 1
3 Oracle Academy Centre of Excellence
with Advance Tools &
Equipment
Multidisciplinary UG / PG / PhD Technical 100 9 1
4 Amazon Web Services (AWS) Centre of Excellence
with Advance Tools &
Equipment
Multidisciplinary UG / PG / PhD Technical 100 2 1
5  Entrepreneurship Development Cell (EDC) Pre-incubation unit Multidisciplinary UG / PG / PhD Technical 5 12 1
6 Research and Development Centre Research Unit / Park Multidisciplinary UG / PG / PhD Technical 45 50 1
7 Electric Circuits Lab Academic
Department with lab
facility
Multidisciplinary UG / PG / PhD Technical 218 5 1
8 Electrical Machines Lab Academic
Department with lab
facility
Mutidisciplinary UG / PG / PhD Technical 210 5 1
9 Power Systems Simulation Lab Academic
Department with lab
facility
Mutidisciplinary UG / PG / PhD Technical 222 5 1
10 Power Electronics Lab Academic
Department with lab
facility
Electrical and
Electronics
Engineering
UG / PG / PhD Technical 57 1 1
11 Control System Lab Academic
Department with lab
facility
Electrical and
Electronics
Engineering
UG / PG / PhD Technical 57 1 1
12 Measurement and Instrumentation Lab Academic
Department with lab
facility
Electrical and
Electronics
Engineering
UG / PG / PhD Technical 56 2 1
13 Programming Lab1 Academic
Department with lab
facility
Mutidisciplinary UG / PG / PhD Technical 442 9 2
List all the departments / centers / units are part of innovation and startup ecosystem of the institite
14 Programming Lab 2 Academic
Department with lab
facility
Mutidisciplinary UG / PG / PhD Technical 291 5 2
15 Programming Lab 3 Academic
Department with lab
facility
Mutidisciplinary UG / PG / PhD Technical 291 6 2
16 Programming Lab 4 Academic
Department with lab
facility
Computer Science
and Engineering
UG / PG / PhD Technical 12 1 1
17 Programming Lab 5 Academic
Department with lab
facility
Computer Science
and Engineering
UG / PG / PhD Technical 336 3 2
18 Programming Lab 6 Academic
Department with lab
facility
Computer Science
and Engineering
UG / PG / PhD Technical 336 3 2
19 Concrete and Highway Engineering Lab Academic
Department with lab
facility
Civil Engineering UG / PG / PhD Technical 83 3 1
20 Survey Lab Academic
Department with lab
facility
Civil Engineering UG / PG / PhD Technical 27 1 1
21 Environmental Engineering Lab Academic
Department with lab
facility
Civil Engineering UG / PG / PhD Technical 56 2 1
22 Soil Mechanics Lab Academic
Department with lab
facility
Civil Engineering UG / PG / PhD Technical 60 2 1
23 Hydraulics Engineering Lab Academic
Department with lab
facility
Mutidisciplinary UG / PG / PhD Technical 238 5 1
24 Chemical Analysis Lab Academic
Department with lab
facility
Chemical
Engineering
UG / PG / PhD Technical 84 2 1
25 Biochemistry Lab Academic
Department with lab
facility
Biotechnology UG / PG / PhD Technical 94 2 1
26 Microbiology Lab Academic
Department with lab
facility
Biomedical
Engineering
UG / PG / PhD Technical 108 2 1
27 Strength of Materials Academic
Department with lab
facility
Mutidisciplinary UG / PG / PhD Technical 135 3 1
28 Engineering Practice Lab Academic
Department with lab
facility
Mutidisciplinary UG / PG / PhD Technical 830 15 1
29 CAD-CAM Lab Academic
Department with lab
facility
Mechanical
Engineering
UG / PG / PhD Technical 431 8 1
30 Manufacturing Technology Lab Academic
Department with lab
facility
Mechanical
Engineering
UG / PG / PhD Technical 268 5 1
List all the departments / centers / units are part of innovation and startup ecosystem of the institite
31 Dynamics Lab Academic
Department with lab
facility
Mechanical
Engineering
UG / PG / PhD Technical 160 3 1
32 Thermal Engineering Lab Academic
Department with lab
facility
Multidisciplinary UG / PG / PhD Technical 204 4 1
33 Metrology and Measurements Lab Academic
Department with lab
facility
Mechanical
Engineering
UG / PG / PhD Technical 160 3 1
34 Mechatronics Lab Academic
Department with lab
facility
Mechanical Lab UG / PG / PhD Technical 163 3 1
35 DSP Lab Academic
Department with lab
facility
Electronics and
Communication
Engineering
UG / PG / PhD Technical 483 9 1
36 Microwave and Optical Lab Academic
Department with lab
facility
Electronics and
Communication
Engineering
UG / PG / PhD Technical 165 3 1
37 Embedded and Microprocessor and
Microcontroller Lab
Academic
Department with lab
facility
Multidisciplinary UG / PG / PhD Technical 391 7 1
38 Electronic Devices and Circuits Lab 1 Academic
Department with lab
facility
Electronics and
Communication
Engineering
UG / PG / PhD Technical 318 6 1
39 Digital Lab Academic
Department with lab
facility
Multidisciplinary UG / PG / PhD Technical 341 7 1
40 Electronic Devices and Circuits Lab 2 Academic
Department with lab
facility
Multidisciplinary UG / PG / PhD Technical 370 7 1
41 M.E Lab Academic
Department with lab
facility
Electronics and
Communication
Engineering
UG / PG / PhD Technical 8 2 2
42 IPR Cell IPR Cell / Patent
Facilitation Unit /
Technology Transfer
Centre
Mutidisciplinary UG / PG / PhD Technical 48 50 1
43 Microsoft Campus Agreement Centre of Excellence
with Advance Tools &
Equipment
Mutidisciplinary UG / PG / PhD Technical 75 24 2
44 Wipro Talent Next Industry Institute
Interface Unit
Mutidisciplinary UG / PG / PhD Technical 200 3 1
45 TCS ion online assessment centre Industry Institute
Interface Unit
Mutidisciplinary UG / PG / PhD Technical 200 5 1
46 Physics Lab Academic
Department with lab
facility
Mutidisciplinary UG / PG / PhD Technical 830 7 1
47 Chemistry Lab Academic
Department with lab
facility
Multidisciplinary UG / PG / PhD Technical 830 6 1
List all the departments / centers / units are part of innovation and startup ecosystem of the institite
48 IT Lab 1 Academic
Department with lab
facility
Information
Technology
UG / PG / PhD Technical 272 9 1
49 IT Lab2 Academic
Department with lab
facility
Biotechnology UG / PG / PhD Technical 52 1 1
50 CISCO Centre of Excellence
with Advance Tools &
Equipment
Multidisciplinary UG / PG / PhD Technical 3 1 1
51 ICT Academy Industry Institute
Interface Unit
Multidisciplinary UG / PG / PhD Technical 50 2 0
52 TCS Accreditation Industry Institute
Interface Unit
Multidisciplinary UG / PG / PhD Technical 440 5 1
53 Breathe Electronics Industry Institute
Interface Unit
Electronics and
Communication
Engineering
UG / PG / PhD Technical 24 2 1
54 NIIT Industry Institute
Interface Unit
Multidisciplinary UG / PG / PhD Technical 30 1 1
55 Cognizant Partnered Institution Industry Institute
Interface Unit
Multidisciplinary UG / PG / PhD Technical 440 4 1
56 Litz Tech Pvt Ltd Industry Institute
Interface Unit
Multidisciplinary UG / PG / PhD Technical 12 4 1
57 LIVEWIRE Industry Institute
Interface Unit
Multidisciplinary UG / PG / PhD Technical 6 2 1
58 KKM Soft Pvt Ltd Industry Institute
Interface Unit
Civil Engineering UG / PG / PhD Technical 46 1 1
59 Veesha Technologies Industry Institute
Interface Unit
Mechanical
Engineering
UG / PG / PhD Technical 9 1 1
60 Prolific Systems and Technologies Industry Institute
Interface Unit
Electrical and
Electronics
Engineering
UG / PG / PhD Technical 16 1 1
61 Wipro Nodel Center Industry Institute
Interface Unit
Multidisciplinary UG / PG / PhD Technical 103 4 1
Participation of HEI in Innovation & Entrepreneurship Initiatives of Ministry of Education (Till Today)
Programs/Initiatives Status Registration ID
Adopted National Innovation and Start-up Policy at the HEI Yes Not Required
Establishment of Institution's Innovation Council (IIC) at HEI YES IIC Registration ID : IC202015325
Trained Innovation Ambassadors at HEI NO Not Required
Participation in Smart India Hackathon (SIH) Yes Not Required
Facilitated listing start-ups/ technologies in Young India combating COVID with Knowledge, Technology and Innovation (YUKTI 2.0) portal of MIC No Not Required
Participation in National Education Alliance of Technology (NEAT) No Not Required
Section 1 > i : Details of Co-curricular events/programs related to Innovation, IPR and Entrepreneurship/Start-up Conducted by the HEI
Srno
Name of
Program
(Full
Name)
Program
Theme
Program
Type
Program
Category
Organizin
g
Departm
ent /
Centre /
Student
Body
Name
Program
Start
Date
Program
End Date
Program
Duration(i
n Days)
Program
Location
Internal
Participa
nts
(Student)
Internal
Participa
nts
(Faculty)
External
Participa
nts
(Student)
External
Participa
nts
(Faculty)
Total
Number
of
Participa
nts
Had institute spent expenses
either from own
resource/received fund to
organise the
program/activity/event?
Did any of the participants
received any awards/recognition
for startup / innovation ideas /
research?
1 Motivatio
nal
Speak on
Special
Session
on
“Corporat
e
Expectati
ons”
 
Innovatio
n
Motivatio
nal speak
Institute
Lead
Activity
Placeme
nt
Training
30-06-20
19
00:00:00
30-06-20
19
00:00:00
1 Within
Campus
450 2 0 0 452 No No
2 FDP on
Phython
program
ming
 
Innovatio
n
Training Student
Lead
Activity
IT 01-07-20
19
00:00:00
07-10-20
19
00:00:00
99 Within
Campus
265 9 0 0 274 No No
3 Steps to
Start A
Small
Scale
Industry
 
Entrepre
neurship/
Startup
Awarene
ss
Institute
Lead
Activity
V.S.B.En
gineering
College
27-12-20
19
00:00:00
27-12-20
19
00:00:00
1 Within
Campus
66 2 0 0 68 Total Expense
Amount
Incurred by
Institute to
Organise the
Program (In
Rupees)
12000 No
4 Seminar
on recent
trends in
Civil
Engineeri
ng
 
Innovatio
n
Seminar Student
Lead
Activity
Civil
Engineeri
ng
14-03-20
20
00:00:00
14-03-20
20
00:00:00
1 Within
Campus
80 9 120 2 211 Total Expense
Amount
Incurred by
Institute to
Organise the
Program (In
Rupees)
10000 No
5 Mentorin
g
Session
 
Innovatio
n
Mentorin
g
Session
Student
Lead
Activity
Electroni
cs and
Communi
cation
Engineeri
ng
01-07-20
19
00:00:00
07-10-20
19
00:00:00
99 Within
Campus
482 25 0 0 507 No No
6 Worksho
p on
Research
Opportun
ities in
Power
Electroni
cs for
Power
Engineer
s
 
Innovatio
n
Worksho
p
Student
Lead
Activity
Electrical
and
Electroni
cs
Engineeri
ng
28-02-20
20
00:00:00
28-02-20
20
00:00:00
1 Within
Campus
2 0 0 0 2
Total Expense
Amount
Incurred by
Institute to
Organise the
Program (In
Rupees)
10000 No
7 Guest
Lecture
on Fluids
&
Hemodyn
amics
Derange
ment
 
Innovatio
n
Training Institute
Lead
Activity
Biomedic
al
Engineeri
ng
30-09-20
19
00:00:00
30-09-20
19
00:00:00
1 Within
Campus
54 3 0 0 57 Total Expense
Amount
Incurred by
Institute to
Organise the
Program (In
Rupees)
10000 No
Section 1 > i : Details of Co-curricular events/programs related to Innovation, IPR and Entrepreneurship/Start-up Conducted by the HEI
8 Technical
Seminar
on
Oppurtun
ities  for
Mechanic
al
Engineer
s in
Aerospac
e
 
Innovatio
n
Awarene
ss
Institute
Lead
Activity
Mechanic
al
Engineeri
ng
25-07-20
19
00:00:00
25-07-20
19
00:00:00
1 Within
Campus
163 5 0 0 168
Total Expense
Amount
Incurred by
Institute to
Organise the
Program (In
Rupees)
10000 No
9 KATTAD
AM’20
 
Innovatio
n
Skill
developm
ent
Student
Lead
Activity
CIVIL
ENGINE
ERING
06-03-20
20
00:00:00
06-03-20
20
00:00:00
1 Within
Campus
146 0 192 0 338 Total Expense
Amount
Incurred by
Institute to
Organise the
Program (In
Rupees)
30000 No
10 SKARA’2
0
 
Innovatio
n
Skill
developm
ent
Student
Lead
Activity
Compute
r science
and
Engineeri
ng
06-03-20
20
00:00:00
06-03-20
20
00:00:00
1 Within
Campus
62 0 65 0 127 Total Expense
Amount
Incurred by
Institute to
Organise the
Program (In
Rupees)
30000 No
11 ZREYAS’
20
 
Innovatio
n
Skill
developm
ent
Student
Lead
Activity
INFORM
ATION
TECHNO
LOGY
06-03-20
20
00:00:00
06-03-20
20
00:00:00
1 Within
Campus
62 0 80 0 142 Total Expense
Amount
Incurred by
Institute to
Organise the
Program (In
Rupees)
30000 No
12 TRYONI
X
 
Innovatio
n
Skill
developm
ent
Student
Lead
Activity
ELECTR
ONICS
AND
COMMU
NICATIO
N
ENGINE
ERING
06-03-20
20
00:00:00
06-03-20
20
00:00:00
1 Within
Campus
146 0 226 0 372 Total Expense
Amount
Incurred by
Institute to
Organise the
Program (In
Rupees)
30000 No
13 ARCHNI
X’20
 
Innovatio
n
Skill
developm
ent
Student
Lead
Activity
ELECTRI
CAL AND
ELECTR
ONICS
ENGINE
ERING
06-03-20
20
00:00:00
06-03-20
20
00:00:00
1 Within
Campus
32 0 204 0 236 Total Expense
Amount
Incurred by
Institute to
Organise the
Program (In
Rupees)
30000 No
14 PISTON
UTS
 
Innovatio
n
Skill
developm
ent
Student
Lead
Activity
MECHAN
ICAL
ENGINE
ERING
06-03-20
20
00:00:00
06-03-20
20
00:00:00
1 Within
Campus
0 0 156 0 156 Total Expense
Amount
Incurred by
Institute to
Organise the
Program (In
Rupees)
30000 No
Section 1 > ii : Details of Students, Faculties/Staff Participation/Representation in Co-curricular events/programs related to Innovation, IPR and Entrepreneurship/Start-up
at the State, National and (or) International Level (Organized by External Organizations) sent by the HEI
Srno
Name of
Program
(Full
Name)
Program
Theme
Program
Type
Organizin
g Agency
/ Institute
Name
Organizin
g Agency
Type
Program
Start
Date
Program
End Date
Program
Duration(i
n Days)
Participa
nts
(Students
)
Participa
nts
(Faculty)
Total
Number
of
Participa
nts
Awards/R
ecognitio
ns/Achiev
ements
Award /
Position /
Recogniti
on
Secured
Title of
Innovatio
n / Start-
up
Secured
the
Award /
Recogniti
on
Had institute spent expenses
either from own resource/
received fund towards sending
delegation to participate /
represent the activity /event?
Did any of the participants
received any awards/recognition
for startup / innovation ideas /
research
1 Worksho
p on
Python in
Data
Analytics
 
Innovatio
n
Worksho
p
PSG
College
of
Technolo
gy
Educatio
nal
Institute
21-02-20
20
00:00:00
23-02-20
20
00:00:00
3 2 0 2 No --- --- No No
2 Worksho
p on
Machine
Learning
 
Innovatio
n
Worksho
p
Kongu
Engineeri
ng
College
Educatio
nal
Institute
15-02-20
20
00:00:00
15-02-20
20
00:00:00
1 7 0 7 No --- --- No No
3 Worksho
p on
Machine
Leraning
 
Innovatio
n
Worksho
p
PSG
Institute
of
Technolo
gy and
Applied
Research
Educatio
nal
Institute
05-02-20
20
00:00:00
06-02-20
20
00:00:00
2 4 0 4 No --- --- No No
4 Worksho
p on Web
Develop
ment
 
Innovatio
n
Worksho
p
Appin
Technolo
gies
Corporat
e /
Industry
Associati
on
14-12-20
19
00:00:00
14-12-20
19
00:00:00
1 3 0 3 No --- --- No No
5 Worksho
p on Big
data
 
Innovatio
n
Worksho
p
Code
Bind
Technolo
ges
Corporat
e /
Industry
Associati
on
07-06-20
19
00:00:00
07-06-20
19
00:00:00
1 3 0 3 No --- --- No No
6 Worksho
p on Web
Develop
ment and
Artifical
Intelligen
ce
 
Innovatio
n
Worksho
p
Appin
Technolo
gy Lab
Corporat
e /
Industry
Associati
on
06-12-20
19
00:00:00
06-12-20
19
00:00:00
1 13 0 13 No --- --- No No
7 Worksho
p on
Web
Develop
ment
 
Innovatio
n
Worksho
p
Appin
Technolo
gy Lab
Corporat
e /
Industry
Associati
on
16-12-20
19
00:00:00
16-12-20
19
00:00:00
1 10 0 10 No --- --- No No
8 Worksho
p on Web
Develop
ment
Java
 
Innovatio
n
Worksho
p
Appin
Technolo
gy Lab
Corporat
e /
Industry
Associati
on
15-12-20
19
00:00:00
15-12-20
19
00:00:00
1 6 0 6 No --- --- No No
9 Worksho
p on Web
Develop
ment
 
Innovatio
n
Worksho
p
Appin
Technolo
gy Lab
Corporat
e /
Industry
Associati
on
30-12-20
19
00:00:00
30-12-20
19
00:00:00
1 2 0 2 No --- --- No No
Section 1 > ii : Details of Students, Faculties/Staff Participation/Representation in Co-curricular events/programs related to Innovation, IPR and Entrepreneurship/Start-up
at the State, National and (or) International Level (Organized by External Organizations) sent by the HEI
10 Worksho
p on Web
Develop
ment
Java
 
Innovatio
n
Worksho
p
Appin
Technolo
gy Lab
Corporat
e /
Industry
Associati
on
09-12-20
19
00:00:00
09-12-20
19
00:00:00
1 5 0 5 No --- --- No No
11 Worksho
p on Web
Develop
ment
 
Innovatio
n
Worksho
p
Appin
Technolo
gy Lab
Corporat
e /
Industry
Associati
on
13-12-20
19
00:00:00
13-12-20
19
00:00:00
1 1 0 1 No --- --- No No
12 Worksho
p on Web
Develop
ment
 
Innovatio
n
Worksho
p
Appin
Technolo
gy Lab
Corporat
e /
Industry
Associati
on
17-12-20
19
00:00:00
17-12-20
19
00:00:00
1 1 0 1 No --- --- No No
13 Worksho
p on Web
Develop
ment
 
Innovatio
n
Worksho
p
Appin
Technolo
gy Lab
Corporat
e /
Industry
Associati
on
12-12-20
19
00:00:00
12-12-20
19
00:00:00
1 3 0 3 No --- --- No No
14 Industrial
visit
 
Innovatio
n
Exposure
visit
Chip
Software
systems
Corporat
e /
Industry
Associati
on
22-09-20
19
00:00:00
23-09-20
19
00:00:00
2 103 0 103 No --- --- No No
15 Symposi
um on
Creoscad
z 20
 
Innovatio
n
Awarene
ss
K.Ramak
rishnan
college of
Technolo
gy
Educatio
nal
Institute
06-03-20
20
00:00:00
06-03-20
20
00:00:00
1 3 0 3 No --- --- No No
16 Symposi
um on
Shekraa
 
Innovatio
n
Awarene
ss
Governm
ent
College
of
Engineeri
ng
Educatio
nal
Institute
05-03-20
20
00:00:00
05-03-20
20
00:00:00
1 4 0 4 No --- --- No No
17 Symposi
um on
Power
Gale 20
 
Innovatio
n
Awarene
ss
PSNA
College
of
Engineeri
ng and
Technolo
gy
Educatio
nal
Institute
21-02-20
20
00:00:00
21-02-20
20
00:00:00
1 2 0 2 No --- --- No No
18 Symposi
um on
GiFT 20
 
Innovatio
n
Awarene
ss
Arulmuru
gan
College
of
Engineeri
ng
Educatio
nal
Institute
21-02-20
20
00:00:00
21-02-20
20
00:00:00
1 2 0 2 No --- --- No No
19 Symposi
um on
KRIYA
2020
 
Innovatio
n
Awarene
ss
PSG
College
of
Technolo
gy
Educatio
nal
Institute
21-02-20
20
00:00:00
23-02-20
20
00:00:00
3 4 0 4 No --- --- No No
20 Symposi
um on
Yuddhav
arna
2K20
 
Innovatio
n
Awarene
ss
M.Kumar
asamy
College
of
Engineeri
ng
Educatio
nal
Institute
29-02-20
20
00:00:00
29-02-20
20
00:00:00
1 4 0 4 No --- --- No No
Section 1 > ii : Details of Students, Faculties/Staff Participation/Representation in Co-curricular events/programs related to Innovation, IPR and Entrepreneurship/Start-up
at the State, National and (or) International Level (Organized by External Organizations) sent by the HEI
21 Symposi
um on
Hashtag
2.0
 
Innovatio
n
Awarene
ss
Coimbato
re
Institute
of
Technolo
gy
Corporat
e /
Industry
Associati
on
29-08-20
19
00:00:00
29-08-20
19
00:00:00
1 8 0 8 No --- --- No No
22 FDP on
Data
Science
with
Python
 
Innovatio
n
Awarene
ss
ICTACA
DEMY
Corporat
e /
Industry
Associati
on
16-12-20
19
00:00:00
18-12-20
19
00:00:00
3 0 9 9 No --- --- No No
23 Worksho
p on E-
Vehicle
and IC
Engine
assembly
 
Innovatio
n
Worksho
p
Coimbato
re
Institute
of
Engineeri
ng and
Technolo
gy
Educatio
nal
Institute
12-03-20
20
00:00:00
12-03-20
20
00:00:00
1 1 0 1 No --- --- No No
24 Internatio
nal
Conferen
ce-
Informati
on
Science
and
Renewab
le Energy
Sources
 
Innovatio
n
Conferen
ce
Selvam
College
Of
Technolo
gy,
Salem
Educatio
nal
Institute
30-08-20
19
00:00:00
30-08-20
19
00:00:00
1 1 0 1 No --- --- No No
25 Internatio
nal
Conferen
ce on
Adavnce
d
Nanomat
erials for
Energy
Engineeri
ng
Biological
and
Medical
Applicatio
ns
 
Innovatio
n
Conferen
ce
Chettinad
College
of
Engineeri
ng and
Technolo
gy, Karur
Educatio
nal
Institute
11-12-20
19
00:00:00
13-12-20
19
00:00:00
3 1 0 1 No --- --- No No
26 Data
Structure
s and
Algorithm
s using C
 
Innovatio
n
Skill
developm
ent
E-Box Corporat
e /
Industry
Associati
on
09-12-20
19
00:00:00
14-12-20
19
00:00:00
6 239 0 239 No --- --- TotalTotal
Expense
Amount
Incurred by
Institute? (In
Rupees)
12000 No
Section 2 > i : List the Academic Programs on Innovation, IPR and Entrepreneurship Development Offered During the Academic Period (2019-20)
Srno Title of Course/Subject
Offered Program Theme Type of Course
Course Offering
Department/Centr
e/Unit Name
Level of
Course
Program
Start Date
Program End
Date
Program
Duration(in Days) Mode of Delivery Total Numbers of
Participants
Total Numbers of
Contact Hours of Course
( Minimum 30 Contact
hours)
1 Value added Course on
Internet of Things
 Innovation Short term
Course
Bachelor Computer
Science and
Engineering
16-12-2019 21-12-2019 6 Offline 106 36
2 Human Computer
Interaction
 Innovation Elective/Core
Credit Course
Bachelor Computer
Science and
Engineering
09-12-2019 16-03-2020 99 Offline 111 45
3 Big Data Analytics  Innovation Elective/Core
Credit Course
Bachelor IT 08-07-2019 02-11-2019 118 Offline 51 45
4 Software Testing  Innovation Elective/Core
Credit Course
Bachelor IT 01-07-2019 02-11-2019 125 Offline 51 45
5 Software Testing  Innovation Elective/Core
Credit Course
Bachelor IT 01-07-2019 02-11-2019 125 Offline 51 45
6 Environment and
Agriculture
 Innovation Elective/Core
Credit Course
Bachelor CIVIL 01-07-2019 02-11-2019 125 Offline 46 45
7 Robotics  Innovation Elective/Core
Credit Course
Bachelor MECH 13-08-2020 06-11-2020 86 Online 163 45
Section 2 > ii : List the Employment Generation Skill Development Programs Organized by the HEI on Innovation, IPR and Entrepreneurship Development During the
Academic Period (2019-20)
Srno Title of the Training Program Program Theme Type of Course
Course Offering
Department/Centre/
Unit Name
Program Start
Date
Program End
Date
Program
Duration(in
Days)
Mode of Delivery Total Numbers of
Participants
Total Numbers of Contact
Hours of Course ( Minimum
30 Contact hours)
1 Deep Learning - Phase I  Innovation Employment
Generation Training
Programs
Computer Science
and Engineering
14-09-2020 19-09-2020 6 Online 45 30
2 Medical Image Analysis -
Phase I
 Innovation Employment
Generation Training
Programs
Electronics and
Communication
Engineering
14-09-2020 19-09-2020 6 Online 112 30
3 Deep Learning - Phase II  Innovation Employment
Generation Training
Programs
Computer Science
and Engineering
05-10-2020 10-10-2020 6 Online 43 30
4 Deep Learning - Phase III  Innovation Employment
Generation Training
Programs
Computer Science
and Engineering
02-11-2020 07-11-2020 6 Online 42 30
5 Medical Image Analysis -
Phase II
 Innovation Employment
Generation Training
Programs
Electronics and
Communication
Engineering
05-10-2020 10-10-2020 6 Online 112 30
6 Medical Image Analysis -
Phase III
 Innovation Employment
Generation Training
Programs
Electronics and
Communication
Engineering
02-11-2020 07-11-2020 6 Online 109 30
7 Technical Training (Service
and Product)
 Innovation Employment
Generation Training
Programs
SIXPHRASE 11-01-2020 22-03-2020 72 Offline 449 432
8 Technical Skills Training  Innovation Employment
Generation Training
Programs
Amphisoft 02-04-2019 02-06-2019 62 Offline 445 372
9 Data Structures and
Algorithms using C
 Innovation Employment
Generation Training
Programs
E-Box 09-12-2019 14-12-2019 6 Offline 239 42
10 Fast Track Training for TCS  Innovation Employment
Generation Training
Programs
Fast Track 11-07-2019 24-07-2019 14 Offline 429 98
11 Fast Track Training for CTS  Innovation Employment
Generation Training
Programs
Fast Track 20-08-2019 02-09-2019 14 Offline 367 98
12 Employability Training Camp  Innovation Employment
Generation Training
Programs
VSB
ENGINEERING
COLLEGE
27-05-2019 07-07-2019 42 Offline 480 294
Section 2 > iii : List the Full Time Faculties had with a Degree (UG/PG/PhD) in Innovation/IPR/Entrepreneurship and (or) Received Specialized Training program of I&E, IPR
(MDP, EDP, FDP, Short-term Certificate Course, of minimum 30 hours Duration) conducted by various State, Central Govt. agencies, and Knowledge Agencies in the
Academic Period (2019-20)
Srno Faculty Name Department/Centre/Unit Name Designation Do you have a degree in I&E & IPR? Had the faculty received training on I&E &
IPR during AY 2019-20
1 Mr.R.Vijayaganth Computer Science and Engineering Associate Professor No Yes
Training Name Training Theme Type of Course Training Agency
Name
Training Agency
Type Training Start Date Training End Date Training Duration
(in Days) Mode of Delivery
Digital Skill Readiness Program  Innovation Employment
Generation Training
Programs
WIPRO Corporate / Industry
Association
24-02-2020
00:00:00
06-03-2020
00:00:00
12 Online
2 Mr. M. Sathyanarayanan Information Technology Assistant Professor No Yes
Training Name Training Theme Type of Course Training Agency
Name
Training Agency
Type Training Start Date Training End Date Training Duration
(in Days) Mode of Delivery
Digital Skill Readiness Program  Innovation Employment
Generation Training
Programs
WIPRO Corporate / Industry
Association
24-02-2020
00:00:00
06-03-2020
00:00:00
12 Online
3 Dr.S.Anand Kumar Varma Civil Engineering Professor No Yes
Training Name Training Theme Type of Course Training Agency
Name
Training Agency
Type Training Start Date Training End Date Training Duration
(in Days) Mode of Delivery
Waste Water Treatment and
Recycling
 Innovation FDP in I&E, IPR NPTEL Corporate / Industry
Association
01-07-2019
00:00:00
30-10-2019
00:00:00
122 Online
4 Shriram.L Mechanical Engineering Assistant Professor No Yes
Training Name Training Theme Type of Course Training Agency
Name
Training Agency
Type Training Start Date Training End Date Training Duration
(in Days) Mode of Delivery
Design Practice  Innovation FDP in I&E, IPR NPTEL Corporate / Industry
Association
01-07-2019
00:00:00
30-09-2019
00:00:00
92 Online
5 Kannan N Mechanical Engineering Assistant Professor No Yes
Training Name Training Theme Type of Course Training Agency
Name
Training Agency
Type Training Start Date Training End Date Training Duration
(in Days) Mode of Delivery
Design Practice  Innovation FDP in I&E, IPR NPTEL Corporate / Industry
Association
01-07-2019
00:00:00
30-09-2019
00:00:00
92 Online
6 Balamurugan S Mechanical Engineering Assistant Professor No Yes
Training Name Training Theme Type of Course Training Agency
Name
Training Agency
Type Training Start Date Training End Date Training Duration
(in Days) Mode of Delivery
Design Practice  Innovation FDP in I&E, IPR NPTEL Corporate / Industry
Association
01-07-2019
00:00:00
30-09-2019
00:00:00
92 Online
7 Ram Prasad S Mechanical Engineering Assistant Professor No Yes
Training Name Training Theme Type of Course Training Agency
Name
Training Agency
Type Training Start Date Training End Date Training Duration
(in Days) Mode of Delivery
Design Practice  Innovation FDP in I&E, IPR NPTEL Corporate / Industry
Association
01-07-2019
00:00:00
30-09-2019
00:00:00
92 Online
8 Dr.G.Sathish Pandian Mechanical Engineering Associate Professor No Yes
Training Name Training Theme Type of Course Training Agency
Name
Training Agency
Type Training Start Date Training End Date Training Duration
(in Days) Mode of Delivery
Design and Instruction of
Engineering Course
 Innovation FDP in I&E, IPR NPTEL Corporate / Industry
Association
01-06-2019
00:00:00
30-09-2019
00:00:00
122 Online
9 Sangeetha M Mechanical Engineering Assistant Professor No Yes
Training Name Training Theme Type of Course Training Agency
Name
Training Agency
Type Training Start Date Training End Date Training Duration
(in Days) Mode of Delivery
Stress Management  Innovation FDP in I&E, IPR NPTEL Corporate / Industry
Association
01-07-2019
00:00:00
30-08-2019
00:00:00
61 Online
10 Ragunath N Mechanical Engineering Assistant Professor No Yes
Training Name Training Theme Type of Course Training Agency
Name
Training Agency
Type Training Start Date Training End Date Training Duration
(in Days) Mode of Delivery
Manufacturing of Composites  Innovation FDP in I&E, IPR NPTEL Corporate / Industry
Association
01-07-2019
00:00:00
30-10-2019
00:00:00
122 Online
11 Velavan R Mechanical Engineering Assistant Professor No Yes
Training Name Training Theme Type of Course Training Agency
Name
Training Agency
Type Training Start Date Training End Date Training Duration
(in Days) Mode of Delivery
Fundamentals of Manufacturing
Processes
 Innovation FDP in I&E, IPR NPTEL Corporate / Industry
Association
01-07-2019
00:00:00
30-10-2019
00:00:00
122 Online
12 Gowtham M P Mechanical Engineering Assistant Professor No Yes
Training Name Training Theme Type of Course Training Agency
Name
Training Agency
Type Training Start Date Training End Date Training Duration
(in Days) Mode of Delivery
Engineering Metrology  Innovation FDP in I&E, IPR NPTEL Corporate / Industry
Association
01-07-2019
00:00:00
30-12-2019
00:00:00
183 Online
13 A.Jainulafdeen EEE Assistant Professor No Yes
Training Name Training Theme Type of Course Training Agency
Name
Training Agency
Type Training Start Date Training End Date Training Duration
(in Days) Mode of Delivery
Fundamentals of Electric Drives  Innovation FDP in I&E, IPR NPTEL Corporate / Industry
Association
01-07-2019
00:00:00
30-09-2019
00:00:00
92 Online
14 Dr.G.Prabhakar EEE Associate Professor No Yes
Training Name Training Theme Type of Course Training Agency
Name
Training Agency
Type Training Start Date Training End Date Training Duration
(in Days) Mode of Delivery
Verification and Testing of VLSI
Circuits using EDA tools
 Innovation FDP in I&E, IPR AICTE Govt. Agency
(Central)
01-11-2019
00:00:00
14-11-2019
00:00:00
14 Online
15 Dr.K.Umamaheswari EEE Associate Professor No Yes
Training Name Training Theme Type of Course Training Agency
Name
Training Agency
Type Training Start Date Training End Date Training Duration
(in Days) Mode of Delivery
Modeling,Technologiesand control
on electric vechicles
 Innovation FDP in I&E, IPR AMET University Educational Institute 25-05-2020
00:00:00
29-05-2020
00:00:00
5 Online
16 Dr.Jayato Nayak Chemical Engineering Professor No Yes
Training Name Training Theme Type of Course Training Agency
Name
Training Agency
Type Training Start Date Training End Date Training Duration
(in Days) Mode of Delivery
Technologies for clean and
Renewable Energy Production
 Innovation FDP in I&E, IPR NPTEL Corporate / Industry
Association
01-07-2019
00:00:00
30-09-2019
00:00:00
92 Online
17 Dr.Nirmal Kannan VSB Engineering College Principal No Yes
Training Name Training Theme Type of Course Training Agency
Name
Training Agency
Type Training Start Date Training End Date Training Duration
(in Days) Mode of Delivery
18 Dr.Sundararajan V VSB Engineering College Head Technology Excellence
Group
No Yes
Training Name Training Theme Type of Course Training Agency
Name
Training Agency
Type Training Start Date Training End Date Training Duration
(in Days) Mode of Delivery
Section 3 > i : List Idea/Innovation Centric Student Clubs Are Functional at the HEI with Access to Co-working Space/Work-stations for Students with Facilities &
Equipment Available to promote and Support Innovation, Entrepreneurship and IPR Activities in Campus
Srno Name of Student Club Type of Club Year of Establishment Type of Facilities
Available at the Club
Student Club
with Access to
Other I&E
Facilities at the
HEI
No of
Programs/Activiti
es Organized by
the Student Club
During the AY
2019-20
Student Club
Team Lead
Name
Student Club Team
Lead Email
Student Club Team
Lead Phone
1 TALENTIA Innovation Club 2019 Co-working Space Innovation Labs 1 DEVAMATHANI
KA. G
mathanikasekar@gmai
l.com
9842962518
Section 3 > ii > a : List the Pre-Incubation Facility (Tinkering Lab/Makers’ Space/Design Centre/New Gen IEDC/IEDC/EDC/Innovation Centre/Startup Cell) (>= 600 Sq. Ft.
Floor Area)
Srno Name of Pre-
Incubation Centre
Type of
Facility
Year of
Establishme
nt
Type of
Facilities
Available at
the Centre
Pre-Incubation
with Access to
Other I&E
Facilities at the
HEI
Name of the
Funding
Schemes
Available
with Pre-
Incubation
Unit, if any
(Optional)
Facility Details
Total Nos of Dedicated
Executive/Management
team Strength Engaged at
the Facility
Had the Pre-Incubation
Unit Provided any
Funding Support for
Idea/Prototype/Innovati
on Development and
implementation during
the Financial Year
2019-20
Had institute
Generated Income
either from Services
Offered by Pre-
Incubation
Centre/Facilities?
Had institute spent expenses either
from own resource/received fund
towards Establishment/Operation
of Pre-Incubation Centre/Facilities?
Section 3 > ii > a : List the Pre-Incubation Facility (Tinkering Lab/Makers’ Space/Design Centre/New Gen IEDC/IEDC/EDC/Innovation Centre/Startup Cell) (>= 600 Sq. Ft.
Floor Area)
1 Entrepreneurship
Development Cell
EDC 2013 Work
Stations
Innovation Labs Innovation
Labs Facility
Incharge
Name
Dr.V.Ni
rmal
Kanna
n
Facility
Incharge
Email
v.nirma
lkanna
n@vsb
ec.com
Facility
Incharge
Phone
988488
3958
1 No No No
Section 3 > ii > b : List the Centre of Excellence (CoE) Facilities with Advance Tools and Equipment at the HEI (>= 1000 Sq. Ft. Floor Area)
No Data Flled!
Section 3 > ii > c : List the Incubation Facilities at the HEI (>= 1500 Sq. Ft. Floor Area)
No Data Flled!
Section 3 > ii > d : List the Innovation Park/Research Park Facilities at the HEI (>= 5000 Sq. Ft. Floor Area)
No Data Flled!
Section 3 > ii > e : List the IPR Cell/Technology Transfer Centre Facilities at the HEI
No Data Flled!
Section 3 > iii > a : Provide detail list of Faculties as Mentor to help Ideas/Innovations/Startups
Srno Name of the Faculty Designation of the Faculty Mentor Expert Area Years of Experience in Mentorship (In
Years)
1 Dr.Nirmal Kannan Principal Idea/Innovation Development 3
Name of Idea/Innovation/ Startup Mentored Mentoring Setup [Within (or) Outside] HEI CIN (in case it is Startup)
IPR Within
2 Dr.Sundararajan Head Technology Excellence Group Idea/Innovation Development 2
Name of Idea/Innovation/ Startup Mentored Mentoring Setup [Within (or) Outside] HEI CIN (in case it is Startup)
Aerospace Outside
Section 3 > iii > b : Provide detail list of External Experts/Agencies engaged as Mentor to help Ideas/Innovations/Startups
Srno Name of the External Expert/Agency as Mentor Mentor Expert Area Type of Mentorship
1 Dr.Nirmal Kannan IPR Protection & Commercialization Structured Mentorship
Section 3 > iii > b : Provide detail list of External Experts/Agencies engaged as Mentor to help Ideas/Innovations/Startups
Structured Mentorship Program Name Start Date End Date Total No of Days of
Mentorship Program No of Mentors Involved Mentorship Cohort Size
(No. of Mentee) Type of Mentee
IPR 01-07-2019 30-09-2019 92 1 1 Idea
Section 4 : List all the Innovations Generated Developed, Completed and Implemented in Ground with Support of Pre-incubation & Incubation facilities at the HEI and
Recognitions Received
Srn
o
Name of the
Innovation Generated
with the Support of
HEI
Type of
Innovation
Nature
of
Innovati
on
Name of
the HEI
Support
ed
Innovati
on
Develop
ment
Innov
ation
Devel
oped
by
Stage
of
Innov
ation/
Stage
of
TRL
it
starte
d
recei
ving
the
Supp
ort
durin
g the
FY
2019-
20
Stage
of
Innov
ation/
Stage
of
TRL
reach
ed as
on
31st
Marc
h
2020
Status
Pre-
Incubati
on/Incu
bation
Support
to
Innovati
on as
on 31st
March
2020
Had the
Innovation
Received any
Recognition at
State, National,
International
Level Programs
during the
financial period
2019-20
Had the
Innovati
on
Convert
ed into
Venture/
Startup/
SME
Unit
during
the
financial
period
2019-20
Had Centre/institute provided any
Financial support (either own
resource or fund received from
agencies) in terms of Seed
Grant/Fund Support to Idea /
Prototype / Innovations
Development during the financial
period 2019-20
Had the Innovations raised any
Seed fund/Angel Investment/VC
during the period of receiving pre-
incubation/incubation support at
your Pre-incubation/incubation or
HEI during the financial period
2019-20
Had the Pre-incubation/Incubation
Centre/HEI received any
income/revenue from
Idea/Prototype/Innovations for the
pre-incubation/incubation support
and the services offered by Pre-
incubation/incubation or institute
during the financial period 2019
1 A Machine  Learning
approach  for Fault
Detection  Prevention
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
NIL
No No No No
Name Type Gender Email Contact
Mr.P.Anbumani Faculty and Student Male anbuanc@gmail.com 9790262364
2 Expose and
Enumeration of  face
using image
processing
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
NIL
No No No No
Name Type Gender Email Contact
Mr.D.Vimalkumar Faculty and Student Male dvimalkumarr@gmail.com 9994040854
3 ATMIoT : An event
driven crime data
reporting service using
AI
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
NIL
No No No No
Name Type Gender Email Contact
Mr.S.Prabakaran Faculty and Student Male sugirpraba@gmail.com 9965029552
4 Secure wearable
Health  care sensing
devices
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
NIL
No No No No
Name Type Gender Email Contact
Mr.P.Anbumani Faculty and Student Male anbuanc@gmail.com 9790262364
5 Monitoring and
predicting  agriculture
sequences and
farmer assistance
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
NIL
No No No No
Name Type Gender Email Contact
Mr.V.Kumararaja Faculty and Student Male mail2kumarcs@gmail.com 9965029552
6 Client centric proxy
re-encryption for
outsourced data in
cloud
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
NIL
No No No No
Name Type Gender Email Contact
Mr.P.Anbumani Faculty and Student Male anbuanc@gmail.com 9790262364
7 An emergency alert
using eye  blinking
signals for real-time
driver-drowsiness
detection system
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
NIL
No No No No
Name Type Gender Email Contact
Mr.S.Prabakaran Faculty and Student Male sugirpraba@gmail.com 9965029552
8 Crop guidance for
young  farmers by
using data mining  and
big data analytics
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Mr.M.Vasudevan Faculty and Student Male vasumecse@gmail.com 8883344482
9 Block Cyber bullying
words  using Machine
Learning  Algorithm in
Social Network
Framework
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Mr.K.Suresh Kumar Faculty and Student Male suresh8788@gmail.com 9095844564
10 Device and sensors
are  applicable to 5G
implementation
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Dr.J.Suresh Faculty and Student Male jsuresh1983@gmail.com 9751062395
11 Water Monitoring
system i n Agriculture
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Mrs.S.Geetha Faculty and Student Female geethamazhilan@gmail.com 8838353121
12 Smart water utilization
Reduce and Reuse
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Mr.P.Anbumani Faculty and Student Male anbuanc@gmail.com 9790262364
13 Vehicle Theft
Identification  using
Raspberry Pi
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Mr.P.Anbumani Faculty and Student Male anbuanc@gmail.com 9790262364
14 An integrated voice
recognition  system
using Neuro –
heuristic method
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Mr.R.Vijayaganth Faculty and Student Male vijayaganth.vsbec@gmail.com 8925824433
15 VM consolidation and
migration for IAAS in
cloud storage
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Mr.P.Anbumani Faculty and Student Male anbuanc@gmail.com 9790262364
16 Smart electrolyte
bottle  using IOT
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Mr.S.Gunasekaran Faculty and Student Male gkarthikkumaran@gmail.com 9790051297
17 Automated toll
collection  system
using machine
learning techniques
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Mrs. B.Bhavani Devi Faculty and Student Female bavab.tech@gmail.com 9942576884
18 Medical data analysis
of  brain tumor type
classification  using 3D
deep convolution
neural networks
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Mrs.S.Abirami Faculty Female abiramis84@gmail.com 9943317689
19 Image encryption in
social  network
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Ms.M.Maharasi Faculty and Student Male mjmaharasi@gmail.com 9952826105
20 Camera vision based
trash  classification
and detection  system
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Dr.S.K.Senthilkumar Faculty and Student Male sksenthilkumar1974@gmail.com 9875642358
21 RPA based automatic
mail  receiving and
data entry  system
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Mr.S.Geerthik Faculty and Student Male geerthikcs@gmail.com 9994018380
22 Automatic Assessment
of  pavement condition
based  on road video
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Mr.P.Anbumani Faculty and Student Male anbuanc@gmail.com 9790262364
23 Data and user
confidentiality  privacy
preservation in
distributed server
Service
Innovation
Technol
ogical
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Mr.R Vijayaganth Faculty and Student Male vijayaganth.vsbec@gmail.com 8925824433
24 Block chain based
confidential  schema
for organized data in
distributed network
Service
Innovation
Technol
ogical
Block
Chainin
g
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Dr.J.Suresh Faculty and Student Male jsuresh1983@gmail.com 9952826105
25 Crop yield prediction
and  efficient fertilizer
usage
Service
Innovation
Technol
ogical
Data
Mining
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Mrs.P.Latha Faculty and Student Female platha19vsb@gmail.com 9344133141
26 "Cluster based VANET
oriented evolving
graph  model using
greedy detection  for
VANETs"
Service
Innovation
Technol
ogical
ADHOC
Network
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Mr.P.Anbumani Faculty and Student Male anbuanc@gmail.com 9790262364
27 "Lockers surveillance
using  Adriano kit"
Service
Innovation
Technol
ogical
Network
Security
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
Agen
cy
Nam
e
Pari
sho
dh
Jour
nal
Agen
cy
Type
Not-
Gov
t.
Age
ncie
s
Awar
d /
Positi
on /
Reco
gnitio
n
Secu
red
nil
No No No No
Name Type Gender Email Contact
Mr.P.Anbumani Faculty and Student Male anbuanc@gmail.com 9790262364
28 PID Control of Two
wheeling Inverted
Pendulum
Service
Innovation
Technol
ogical
Embedd
ed
System
s
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.A.Jainulafdeen Faculty and Student Male ajainulafdeen@gmail.com 8925824433
29 Traffic Light Pre-
Emption Control
System for Emergency
Vehicles
Service
Innovation
Technol
ogical
Renewa
ble
Energy
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Dr.P.Elamurugan Faculty and Student Male elamurugan.p@gmail.com 9962875859
30 Design and
Development of
Fractional order PID
Controller on Cruise
Control System
Service
Innovation
Technol
ogical
Embedd
ed
System
s
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Dr.G.Prabhakar Faculty and Student Male gprabhakar2488@gmail.com 7010732363
31 Fast Charging circuit
for EV using Arduino
Service
Innovation
Technol
ogical
Embedd
ed
System
s
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.R.ArunKumar Faculty and Student Male arunkumar1133@gmail.com 9790556393
32 Power Loss Reduction
based on DG using
Renewable Sources
Service
Innovation
Technol
ogical
Power
Electron
ics
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mrs.S.Sathya Faculty and Student Female sathiyashanmugavel@gmail.com 9965661395
33 Solar Powered BLDC
Motor for wide Range
Applications
Service
Innovation
Technol
ogical
Power
Electron
ics
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Dr.K.Umamaheswari Faculty and Student Female umakrishnan21@yahoo.co.in 8012124309
34 Line Follower Robot Service
Innovation
Technol
ogical
Embedd
ed
System
s
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.R.Sivakumar Faculty and Student Male odaisiva@gmail.com 9585419087
35 Smart LED Light
based on Solar Power
Supply
Service
Innovation
Technol
ogical
Power
Electron
ics
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr. K.Karthi Faculty and Student Male k.karthi94mr@gmail.com 9585966461
36 AUTOMATIC BROKE
FEEDING SYSTEM IN
PAPER MACHINE
USING PLC
Service
Innovation
Technol
ogical
Embedd
ed
System
s
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr. C.Maheshkumar Faculty and Student Male maheshkhumarc@gmail.com 9788060931
37 EMISSION
REDUCTION BY
USING SWITCHED
INTERGRATED
RECTIFIER IN
RECOVERY BOILER
Service
Innovation
Technol
ogical
Embedd
ed
System
s
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr. P.Parthasarathy Faculty and Student Male parthasarathy5326@gmail.com 9246711369
38 Hybrid Renewable
Energy System
Service
Innovation
Technol
ogical
Renewa
ble
Energy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr. S.Ragul Faculty and Student Male ragulsambath22@gmail.com 9952124414
39 METAL TRAP USING
RELAY CIRCUIT
Service
Innovation
Technol
ogical
Electric
Circuits
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mrs.S.Preetha Faculty and Student Female preethu.karur@gmail.com 9790262364
40 A Novel Approach of
Finger vein recognition
for personal
authentication
Service
Innovation
Technol
ogical
Image
Process
ing
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.R.Jegan Faculty and Student Male jeganvsb17@gmail.com 9566402260
41 Collision avoidance by
using advanced speed
control system
Service
Innovation
Technol
ogical
Embedd
ed
systems
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.Srivenkatalakshmi Faculty and Student Female srivenkatalakshmi94@gmail.com 9489486651
42 Novel Measurement of
Monitoring liquid level
in sewage water
Service
Innovation
Technol
ogical
Embedd
ed
systems
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Dr.V.Kavitha Faculty and Student Female drvkavitha@vsbec.com 9790556393
43 Eminent and safe child
rescue system
Service
Innovation
Technol
ogical
Embedd
ed
systems
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
D.S.Priyanka Faculty and Student Female rathi.priyanka5@gmail.com 7904864837
44 Pulsed Electric filed for
food preservation
Service
Innovation
Technol
ogical
Embedd
ed
systems
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
S.Meena Faculty and Student Female jeyammeena@gmail.com 9962875859
45 RF-ID based system
for school
transportation safety
enhancement
Service
Innovation
Technol
ogical
Wireles
s
network
s
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
S.Sujitha Faculty and Student Female sujithasubramani96@gmail.com 9455132585
46 Analysis and Design of
SRAM cell with
MTCMOS using micro
wind Tool
Service
Innovation
Technol
ogical
VLSI
design
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
S.Meena Faculty and Student Female jeyammeena@gmail.com 9962875859
47 Image Steganography
using Vignere cipher
and Knight’s tour
algorithm using dual
encryption technqiue
Service
Innovation
Technol
ogical
Cryptog
raphy
and
Network
Security
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
M.Thangavel Faculty and Student Male karurdinesh.vel@gmail.com 9500639349
48 Turmeric plant leaf
Disease Identification
using support vector
machine classifier
technique
Service
Innovation
Technol
ogical
Image
Process
ing
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
M.Dinesh Faculty and Student Male apdineshvsb@gmail.com 9945529924
49 Identification of Lung
caner using 3D deep
neural Network
Service
Innovation
Technol
ogical
Image
Process
ing
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
C.Moorthy Faculty and Student Male moorthy.ind@gmail.com 8870298982
50 Testing of Integrated
circuits using
automatic test
equipment
Service
Innovation
Technol
ogical
Embedd
ed
syetems
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
M.Thangavel Faculty and Student Male karurdinesh.vel@gmail.com 9500639349
51 Deformation IRIS
recognition using
DDCNN
Service
Innovation
Technol
ogical
Image
Process
ing
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
M.Vijay Faculty and Student Male vijayvsbece@gmail.com 8012124309
52 Life safety and
security helmet for
bike rider
Service
Innovation
Technol
ogical
Embedd
ed
syetems
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
P.Anandakumar Faculty and Student Male apanandhvsb@gmail.com 8870188907
53 Women safety using
live camera in IOT
Service
Innovation
Technol
ogical
IOT Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
S.Sathish Faculty and Student Male satishmtechece@gmail.com 8682947245
54 Data Security with
chaos signal
Service
Innovation
Technol
ogical
Wireles
s
Network
s
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
T.Sivalingam Faculty and Student Male t.sivalingam@gmail.com 8903936850
55 Minimum energy
constraints with cluster
head and spamming
tree in WSN with
routing algorithm
Service
Innovation
Technol
ogical
Wireles
s
Network
s
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
S.Sathish Faculty and Student Male satishmtechece@gmail.com 8682947245
56 Charge Recycling
based Redundant
write voltage
suppression technique
for low power SOT-
MRAM
Service
Innovation
Technol
ogical
VLSI
Design
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
K.Kalaichelvi Faculty and Student Female kalaish123@gmail.com 9698312857
57 A low power digital
background calibration
using new ultra
dynamic latch
comparator
Service
Innovation
Technol
ogical
VLSI
Design
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
K.Kalaichelvi Faculty and Student Female kalaish123@gmail.com 9698312857
58 Telegram Messenger
Based Digital Notice
Board With audio
system
Service
Innovation
Technol
ogical
Embedd
ed
systems
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.Vallisuseela Faculty and Student Female r.vallisuseela@gmail.com 9443283449
59 Field ploughing by
driverless tractor using
Ardunio
Service
Innovation
Technol
ogical
Embedd
ed
systems
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
M.Dinesh Faculty and Student Male apdineshvsb@gmail.com 9945529924
60 Border surveillance
using WSN
Service
Innovation
Technol
ogical
Wireles
s
Network
s
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
C.Moorthy Faculty and Student Male moorthy.ind@gmail.com 8870298982
61 An Ultra Low power
Bulk FINFET SRAM
Service
Innovation
Technol
ogical
VLSI
Design
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
P.Anandakumar Faculty and Student Male apanandhvsb@gmail.com 8870188907
62 Multimodal sensor
medical image fusion
based on laplacian
filter
Service
Innovation
Technol
ogical
Image
Process
ing
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
P.Anandakumar Faculty and Student Male apanandhvsb@gmail.com 8870188907
63 Removal of specular
reflection and early
detection of glaucoma
Service
Innovation
Technol
ogical
Image
processi
ng
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
S.Keerthana Faculty and Student Female keerthanaece20@gmail.com 7806801526
64 D2D communication
using dual hop
network for best path
scheduling in WSN
Service
Innovation
Technol
ogical
Wireles
s
Network
s
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.Ashokkumar Faculty and Student Male ashokvsbec@gmail.com 9994281810
65 ATM IOT-An Event
Driven crime data
reporting service using
AI
Service
Innovation
Technol
ogical
IOT Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
S.Keerthana Faculty and Student Female keerthanaece20@gmail.com 7806801526
66 Multi nodality medical
image fusion method
for cancer applications
Service
Innovation
Technol
ogical
Image
Process
ing
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.Ashokkumar Faculty and Student Male ashokvsbec@gmail.com 9994281810
67 Smart class room
attendance system
using group face
recognition
Service
Innovation
Technol
ogical
Embedd
ed
systems
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
M.Vijay Faculty and Student Male vijayvsbece@gmail.com 8682947245
68 Smart Electricity bill
payment system
management
Service
Innovation
Technol
ogical
Embedd
ed
systems
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Dr.V.Kavitha Faculty and Student Female drvkavitha@vsbec.com 8903936850
69 Vechicle Control Unit Service
Innovation
Technol
ogical
Embedd
ed
systems
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.Vallisuseela Faculty and Student Female r.vallisuseela@gmail.com 9443283449
70 Design of circularly
polarized dual band V
shaped patch antenna
Service
Innovation
Technol
ogical
Antenna
s
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.Dhayabarani Faculty and Student Female ece@vsbec.com 9994030854
71 Smart Irrigation for
better Crop
Management
Service
Innovation
Technol
ogical
Embedd
ed
systems
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
G.Mohanavel Faculty and Student Male mohan.vel48@gmail.com 9791924560
72 Smart Polling System Service
Innovation
Technol
ogical
Embedd
ed
systems
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.Ashokkumar Faculty and Student Male ashokvsbec@gmail.com 9994281810
73 Pulmonary Disease
Prediction using Deep
learning using lung
sound data base
Service
Innovation
Technol
ogical
Image
processi
ng
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
S.Keerthana Faculty and Student Female keerthanaece20@gmail.com 7806801526
74 Low cost powered
prosthetic leg with
auto tunning
Service
Innovation
Technol
ogical
Embedd
ed
systems
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.Dhayabarani Faculty and Student Female ece@vsbec.com 9994030854
75 Soldier Multipurpose
Suit
Service
Innovation
Technol
ogical
Embedd
ed
systems
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.R.Jegan Faculty and Student Male jeganvsb17@gmail.com 9566402260
76 A novel approach on
web application for
agriculture sector
Service
Innovation
Technol
ogical
Wireles
s
network
s
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.Dhayabarani Faculty and Student Female ece@vsbec.com 9994030854
77 A low cost system for
real time monitoring of
drinking water quality
and controlling using
IOT
Service
Innovation
Technol
ogical
Embedd
ed
systems
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.Krishnan Faculty and Student Male dgl.krishna@gmail.com 9944816768
78 A low duty cycle bust
mode telemeter signal
generation technique
for VHF inject tracking
and its CMOS
implementation
Service
Innovation
Technol
ogical
VLSI
Design
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
P.Anandakumar Faculty and Student Male apanandhvsb@gmail.com 8870188907
79 A self Powered air
quality monitoring
system within a
campus using WSN
(LORA)
Service
Innovation
Technol
ogical
Wireles
s
network
s
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
N.Arunkumar Faculty and Student Male arunshiva03@gmail.com 9688070556
80 Optimizing Energy
variable a cost control
measure
Service
Innovation
Technol
ogical
Wireles
s
network
s
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
N.M.Karthikeyan Faculty and Student Male santhoshkarthik87@gmail.com 9994718728
81 A Wearable Dual-
Band Low Profile High
Gain Low SAR
Antenna Amc-Backed
For WBAN
Applications
Service
Innovation
Technol
ogical
Embedd
ed
systems
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
M.Kaviarasu Faculty and Student Male kaviyarasum7@gmail.com 8489736186
82 Self Co-operative
Trust Scheme against
routing disruptions
attack minimization for
MANETS
Service
Innovation
Technol
ogical
Wireles
s
Network
s
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.Srivenkatalakshmi Faculty and Student Female srivenkatalakshmi94@gmail.com 9489486651
83 Experimental
Investigation on
Flexural Behaviour of
Engineered
Cementitious
Composite
Service
Innovation
Technol
ogical
Concret
e
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
B.YugeshKumar Faculty and Student Male yugeshkumar01@gmail.com 7806801526
84 Experimental Study on
Fly Ash Bricks by
Using Polystyrene
Plastic Powder
Service
Innovation
Technol
ogical
Concret
e
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
P.Suresh Kannan Faculty Male psureshkannan18@gmail.com 9894375878
85 Experimental
Investigations on High
Strength Fiber
Reinforced Permeable
Concrete
Service
Innovation
Technol
ogical
Concret
e
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
C.Mohanaselvan Faculty Male mohan.selvan79@gmail.com 9790201179
86 Experimental
Investigations on High
Strength Brick By
Using Natural
Admixtures
Service
Innovation
Technol
ogical
Concret
e
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.Kartheeswaran Faculty Male kartheeswarankarthik@gmail.com 9751135888
87 Experimental
Investigations on
Utilization of Municipal
Soild Waste in Bricks
Service
Innovation
Technol
ogical
Concret
e
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
N.Vigneshwaran Faculty Male ervigneshpuhazh2@gmail.com 9677093619
88 Experimental
Investigation on Partial
Replacement of
Ceramic Waste In
Coarse Aggregate
With Addition of
Bagasse Fiber
Service
Innovation
Technol
ogical
Concret
e
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
C.Mohanaselvan Faculty Male mohan.selvan79@gmail.com 9790201179
89 Behaviour of Bricks By
Using Paper as Major
Component
Service
Innovation
Technol
ogical
Concret
e
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
N.Vigneshwaran Faculty Male ervigneshpuhazh2@gmail.com 9677093619
90 Experimental
Investigation of
Thermal Insulation
Roof
Service
Innovation
Technol
ogical
Concret
e
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
N.Vigneshwaran Faculty Male ervigneshpuhazh2@gmail.com 9677093619
91 Experimental
Investigations & Study
of Rigid Pavement
with Fully
Replacement of Fine
Aggregate Municipal
Solid Waste
Incineration Ash &
Foundry Sand
Service
Innovation
Technol
ogical
Concret
e
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Dr.S.Anand Kumar Varma Faculty Male manju_sakv@yahoo.co.in 9994030854
92 Experimental Study on
Stabilization of Black
Cotton Soil Using
Baggasse Ash
Service
Innovation
Technol
ogical
Concret
e
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
B.YugeshKumar Faculty Male yugeshkumar01@gmail.com 9566402260
93 Experimental
Investigations on the
Implementation of
Transmitter and
Receiver in
Transportation System
to Prevent Accidents
Service
Innovation
Technol
ogical
Highwa
y
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.Kartheeswaran Faculty Male kartheeswarankarthik@gmail.com 9751135888
94 Experimental
Investigations On
Engineering Properties
of Recycled Fine
Aggregate concerete
Adding metakaolin
Service
Innovation
Technol
ogical
Concret
e
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
K.Rajkumar Faculty Male k.rajkumaraccet@gmail.com 9791597295
95 An Experimental Study
on Buoyant
Foundation
Service
Innovation
Technol
ogical
Foundat
ion
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
C.Mohanaselvan Faculty Male mohan.selvan79@gmail.com 9790201179
96 Experimental
Investigations on Self
Curing Concrete Using
Natural Super
Plasticizer
Service
Innovation
Technol
ogical
Concret
e
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
P.Suresh Kannan Faculty Male psureshkannan18@gmail.com 9894375878
97 An Analytical Study on
Design Of Hydraulic
Ram Pump
Service
Innovation
Technol
ogical
Hydrauli
c
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.Gowrishankar Faculty Male gowricivil@gmail.com 9750236832
98 An Experimental
Investigation of
Floating Concrete and
Boat With
Incorporated Rice
Husk Ash and
Foaming Agent
Service
Innovation
Technol
ogical
Concret
e
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
K.Rajkumar Faculty Male k.rajkumaraccet@gmail.com 9791597295
99 Study on Behavior of
Paper Industry
Treated Effluent Water
With Chemical
Admixture As Mixing
Water In Cocnrete
Service
Innovation
Technol
ogical
Environ
mental
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.Gowrishankar Faculty Male gowricivil@gmail.com 9750236832
100 Experimental
Investigation on Fly
Ash Bricks By Partial
Replacement of Fly
Ash With Lime Powder
Service
Innovation
Technol
ogical
Concret
e
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.Gowrishankar Faculty Male gowricivil@gmail.com 9750236832
101 Experimental
Investigation on
Cement Flooring Tiles
Using Coconut Shell
As Coarse Aggregate
Service
Innovation
Technol
ogical
Concret
e
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
K.Rajkumar Faculty Male k.rajkumaraccet@gmail.com 9791597295
102 Narural Air Purifier At
Traffic Signals
Powered By Vehicle
Movement
Service
Innovation
Technol
ogical
Environ
mental
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
D.Dineshkumar Faculty Male kongutamildinesh@gmail.com 7200302115
103 Municipal Solid Waste
Treatment By Using
Combination Of
Effective
Microorganisms And
Cou Dung
Service
Innovation
Technol
ogical
Environ
mental
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
M.Aathavan Faculty Male aathuguru3002@gmail.com 8773746977
104 Experimental
Investigation on
Behaviour of
Translucent Concrete
With Replacement of
Cement By Using
Lime Silica Powder
Service
Innovation
Technol
ogical
Concret
e
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
D.Dineshkumar Faculty Male kongutamildinesh@gmail.com 7200302115
105 Experimental
Investigation On
Design Of Readymade
Road Pavement
Service
Innovation
Technol
ogical
Highwa
y
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
R.Gowrishankar Faculty Male gowricivil@gmail.com 9750236832
106 Feasibility study and
implementation of dual
fuel system in two
wheelers
Service
Innovation
Technol
ogical
Automo
bile
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.R. Balamurugan Faculty Male balaapmech@gmail.com 9790201179
107 Design and fabrication
of 90 degree car
parking
Service
Innovation
Technol
ogical
Automo
bile
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.M.S.Pragash Faculty Male aeropragash@gmail.com 9965880994
108 "Design and
fabrication of dual side
water pumping system
using scotch yock
mechanism     "
Service
Innovation
Technol
ogical
Kinemat
ics of
Machine
ry
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Dr. P.Balu Faculty Male balumitauto@gmail.com 9750256355
109 "Coconut Multi
Processing machine"
Service
Innovation
Technol
ogical
Manufa
cturing
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Dr. V. Nirmal Kannan Faculty Male v.nirmalkannan@gmail.com 9944478641
110 "Mechanical properties
of natural fibre(White
madar) reinforced
polymer composites"
Service
Innovation
Technol
ogical
Material
Science
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.M. Dhanabal Faculty Male dhanabal59@gmail.com 9715010161
111 "Synthesis of fuel from
water     "
Service
Innovation
Technol
ogical
Automo
bile
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.S.Balamurugan Faculty Male balamuruganng892@gmail.com 7200302115
112 "Optimizing the turning
process by the effects
of machining
parameters     "
Service
Innovation
Technol
ogical
Manufa
cturing
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.M. P. Gowtham Faculty Male gowthammech21@gmail.com 8903461084
113 "Fatigue curve
parameters -
Description of 7075
aluminium alloy using
hyperbolic function     "
Service
Innovation
Technol
ogical
Material
Science
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Dr.J.Selvakumar Faculty Male selvakumar1979@gmail.com 6382023022
114 "Design and
fabrication of remote
controlled pick and
place robot     "
Service
Innovation
Technol
ogical
Robotic
s and
Automat
ion
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.S. Ramprasad Faculty Male ramprasad2601@gmail.com 8056905274
115 "Wheel chair with
standing
arrangements"
Service
Innovation
Technol
ogical
Automat
ion
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.N.Kapil Dev Faculty Male kapildevnr@gmail.com 9788981205
116 "Production of
extending bumper of
bike for accident
impact safety"
Service
Innovation
Technol
ogical
Automo
bile
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.P.Devaraj Faculty Male devarajp1992@gmail.com 9865294628
117 "Oscillating water
column power
generation"
Service
Innovation
Technol
ogical
Power
plant
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.G.Vijay Anand Faculty Male vijayapmech@gmail.com 9715010161
118 "Footstep power
generator using rack
and pinion
mechanism"
Service
Innovation
Technol
ogical
Kinemat
ics of
Machine
ry
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.S.Vinoth Kumar Faculty Male vkthermal@gmail.com 7598624455
119 "Acetylene gas used
as alternative fuel in
petrol engine"
Service
Innovation
Technol
ogical
Automo
bile
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.T.Raja Faculty and Student Male rajathambikannu@gmail.com 9944478641
120 "Multi nozzle pesticide
sprayer"
Service
Innovation
Technol
ogical
Manufa
cturing
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.R.Velavan Faculty Male velavangalvanisingwm@gmail.com 9003591698
121 "Drainage pipe block
removal     "
Service
Innovation
Technol
ogical
Manufa
cturing
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.C.Selvakumar Faculty Male cskmech141@gmail.com 9715010161
122 "Experimental
investigation and
macro property
analysis on Al 7075
metal matrix reinforced
with TiO2 & ES"
Service
Innovation
Technol
ogical
Material
Science
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.L.Shriram Faculty Male lshriram103@gmail.com 6380375265
123 "Thermal analysis of
Aluminium composite
piston"
Service
Innovation
Technol
ogical
Automo
bile
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.N.Yogaraj Faculty Male yogaraj0506@gmail.com 9500412199
124 "Comparitive analysis
and geomentrical
properties of drilling on
hardened tool steel
using cryogenic and
ordinary drill bits"
Service
Innovation
Technol
ogical
Material
Science
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Dr.M.Mohamed Abdul Hafeez Faculty Male mohamedabdulhafeez72@gmail.com 9894848268
125 "Driverless Renewable
energy hybrid vehicle
with bluetooth control"
Service
Innovation
Technol
ogical
Automo
bile
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Dr. V. Nirmal Kannan Faculty Male v.nirmalkannan@gmail.com 9944478641
126 "Design and
fabrication of
pneumatic sheet
bending machine     "
Service
Innovation
Technol
ogical
Manufa
cturing
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Dr. P.Balu Faculty Male balumitauto@gmail.com 7598624455
127 "Design of ion
propulsion system"
Service
Innovation
Technol
ogical
Material
Science
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.R. Balamurugan Faculty Male balaapmech@gmail.com 9715010161
128 "Synthesis of fuel from
from plastic waste"
Service
Innovation
Technol
ogical
Automo
bile
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.S.Balamurugan Faculty Male balamuruganng892@gmail.com 9715015648
129 "Tyre pressure
monitoring system"
Service
Innovation
Technol
ogical
Automo
bile
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Dr.J.Selvakumar Faculty Male selvakumar1979@gmail.com 6382023022
130 "Three axis pneumatic
modern trailor"
Service
Innovation
Technol
ogical
Manufa
cturing
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.S. Ramprasad Faculty Male ramprasad2601@gmail.com 8056905274
131 "Keyless bike" Service
Innovation
Technol
ogical
Automo
bile
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.S.Vinoth Kumar Faculty Male vkthermal@gmail.com 7598624455
132 "Automatic side stand
retrival system"
Service
Innovation
Technol
ogical
Automo
bile
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.M. Dhanabal Faculty Male dhanabal59@gmail.com 9003591698
133 "Automatic speed
braker deduction and
speed  controlling
system"
Service
Innovation
Technol
ogical
Autombi
le
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.N.Kapil Dev Faculty Male kapildevnr@gmail.com 9788981205
134 "Water splashing boat" Service
Innovation
Technol
ogical
Manufa
cturing
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.N.Kapil Dev Faculty Male kapildevnr@gmail.com 9788981205
135 " Fabrication of safety
bumper"
Service
Innovation
Technol
ogical
Automo
bile
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.M. P. Gowtham Faculty Male gowthammech21@gmail.com 8903461084
136 "Design of manual
seed sower"
Service
Innovation
Technol
ogical
Manufa
cturing
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.P.Devaraj Faculty Male devarajp1992@gmail.com 9865294628
137 "Mechanical
Properties of Natural
Fibre [Vaagai
Tree]Reinforced
Polymer Composite"
Service
Innovation
Technol
ogical
Material
Science
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.S.Balamurugan Faculty Male balamuruganng892@gmail.com 7598624455
138 "Optimization of
exhaust emissions in
aqua silencer by using
lithium hydroxide "
Service
Innovation
Technol
ogical
Automo
bile
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Dr. V. Nirmal Kannan Faculty Male v.nirmalkannan@gmail.com 9944478641
139 "Experimental
investigation and
optimization of
machining parameters
for EDM using copper
electrode on D2 steel"
Service
Innovation
Technol
ogical
Material
Science
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.R.Velavan Faculty Male velavangalvanisingwm@gmail.com 9003591698
140 "Fabrication of 360
degree pneumatic
power rotary modern
trailer"
Service
Innovation
Technol
ogical
Manufa
cturing
Technol
ogy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.L.Shriram Faculty Male lshriram103@gmail.com 6380375265
141 "Study of mechanical
properties of
aluminium metal
matrix composite"
Service
Innovation
Technol
ogical
Material
Science
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.S. Ramprasad Faculty Male ramprasad2601@gmail.com 8056905274
142 "Experimental
investigation and
mechanical behaviour
on EN24 steel with
and without PWHT
process"
Service
Innovation
Technol
ogical
Material
Science
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.T.Raja Faculty Male rajathambikannu@gmail.com 9894848268
143 "Analysis of
alternatives for brake
disc material using
various aluminium
MMC"
Service
Innovation
Technol
ogical
Automo
bile
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.N.Yogaraj Faculty Male yogaraj0506@gmail.com 9500412199
144 "Mechanical
Properties of Natural
Fibre[Indian Beech
Tree]Reinforced
Polymer Composites"
Service
Innovation
Technol
ogical
Automo
bile
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.N.Yogaraj Faculty Male yogaraj0506@gmail.com 9500412199
145 "Evaluation of
tribological properties
of AA7075 composite
reinforced with
graphite & titanium
carbide by powder
metallurgy process"
Service
Innovation
Technol
ogical
Material
Science
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.C.Selvakumar Faculty Male cskmech141@gmail.com 9003591698
146 Biodiesel performance
and emission in diesel
engine vagai seed oil
Service
Innovation
Technol
ogical
Automo
bile
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr.G.Vijay Anand Faculty Male vijayapmech@gmail.com 9788245897
147 "Experimental
investigation
perrormance and
emission of diesel and
biodiesel blends"
Service
Innovation
Technol
ogical
Automo
bile
Enginee
ring
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Dr.M.Mohamed Abdul Hafeez Faculty Male mohamedabdulhafeez72@gmail.com 9894848268
148 Warplane security and
communication system
using lifi
Service
Innovation
Technol
ogical
Wireles
Network
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr M.Sathya Narayanan Faculty Male sathyakcgit@gmail.com 9445360851
149 Driver drowsiness
monitoring system
using visual behaviour
and machine learning
Service
Innovation
Technol
ogical
Machine
Learnin
g
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr R.Kumaran Faculty Male kumaran.r.r@gmail.com 9944453651
150 Smart traffi
management and time
management using
bigdata analysis
Service
Innovation
Technol
ogical
Big data
Analytic
s
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mrs N.Premalatha Faculty Female prema.krnp@gmail.com 8056864396
151 Client centric proxy re-
encryption for
outsourced in the
cloud
Service
Innovation
Technol
ogical
Cloud
computi
ng
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr R.Rajiniganth Faculty Male rajiniganthvsbec@gmail.com 9787382047
152 Blockchain based
digital forensic
investigation
framework in the
internet of things and
social systems
Service
Innovation
Technol
ogical
Block
chain
technolo
gy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr S.Nelson Faculty Male selvinnelson@gmail.com 9994315031
153 Blockchin based
information
architecture for
medical product
supply chain
Service
Innovation
Technol
ogical
Block
chain
technolo
gy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Dr M.Sangeetha Faculty Female godsan2003@gmail.com 9688851998
154 Fingerprint based
antitheft for two
wheelers.
Service
Innovation
Technol
ogical
IOT Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mrs D.Maalini Faculty Female maalini.cse@gmail.com 8903686196
155 Anti theft technology of
museum cultural relics
using internet of things
Service
Innovation
Technol
ogical
IOT Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr M.MohanPeriyasamy Faculty Male mmohanperiyasamy@gmail.com 9003125953
156 Rumor detection on
social media
Service
Innovation
Technol
ogical
Data
Mining
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr R.Rajiniganth Faculty Male rajiniganthvsbec@gmail.com 9787382047
157 Blackchain Based
Implementation of
Electronic Medical
Health Records
Service
Innovation
Technol
ogical
Block
chain
technolo
gy
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr M.Sathya Narayanan Faculty Male sathyakcgit@gmail.com 9445360851
158 Spammer detection
and fake user
identification on social
networks
Service
Innovation
Technol
ogical
Machine
Learnin
g
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mrs D.Maalini Faculty Female maalini.cse@gmail.com 8903686196
159 Classification of lung
cancer using deep
learning algoritham
Service
Innovation
Technol
ogical
Data
Mining
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Dr M.Sangeetha Faculty Female godsan2003@gmail.com 9688851998
160 ARN - Authenticated
routing for adhoc
network for privacy
preserving
authentication scheme
for VANNET
Service
Innovation
Technol
ogical
Wireles
Network
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr K.Manivannan Faculty Male manivannan.vsbec@gmail.com 9894104234
161 Sentiment analysis in
E-commerce using
recommendation
system
Service
Innovation
Technol
ogical
Sentime
nt
analysis
-AI
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr R.Kumaran Faculty Male kumaran.r.r@gmail.com 9500412199
162 Vanet Based
Communication on
Vehicles for Accident
Prevention
Service
Innovation
Technol
ogical
Wireles
Network
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mrs N.Premalatha Faculty Female prema.krnp@gmail.com 8056864396
163 Combating online
abuse and online
reporting functions on
social networking sites
Service
Innovation
Technol
ogical
Data
Mining
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr M.MohanPeriyasamy Faculty Male mmohanperiyasamy@gmail.com 9003125953
164 Convolution neural
networks based fire
detection in
surveillance video
Service
Innovation
Technol
ogical
Wireles
Network
Stude
nt-
Facul
ty
TRL
0 :
Idea.
Unpr
oven
conc
ept,
no
testin
g has
been
perfor
med.
TRL
1 :
Basic
resea
rch.
Princi
ples
postu
lated
and
obser
ved
but
no
exper
iment
al
proof
availa
ble.
Innovati
on
Develop
ment
Complet
ed
No No No No No
Name Type Gender Email Contact
Mr S.Nelson Faculty Male selvinnelson@gmail.com 9994315031
Section 5 : List all the Ventures/Startups/SME Units Established with Support of Pre-incubation & Incubation facilities at the HEI
Srn
o
Name of the
Venture/Sta
rtup/SME
Unit
Established
with the
Support of
HEI
Type of
Innovation
of
Venture/Sta
rtup/SME
Unit
Nature
of
Innovat
ion
Name
of the
Pre-
Incubat
ion/Inc
ubation
Unit of
HEI
Suppor
ted
Innovat
ion
Develo
pment
Vent
ure
Foun
ded
by
Venture Registerd as
Year
of
Regi
strati
on
What is the
current status
of Pre-
Incubation/Inc
ubation
Support to
Venture/
Startup/ SME
Unit as on
31st March
2020?
Had the Venture Received any
Recognition at State, National,
International Level Programs
during the financial period 2019-
20?
Had Centre/institute provided
any Financial support (either
own resource or fund received
from agencies) in terms of Seed
Grant/Fund Support to Venture
establishment during the
financial period 2019-20
Had the venture/startup raised
any Seed fund/Angel
Investment/VC during the period
of receiving pre-
incubation/incubation support at
your Pre-incubation/incubation or
HEI during the financial period
2019-20
Had
the
Pre-
incubat
ion/Inc
ubation
Centre/
HEI
receive
d any
income
/revenu
e from
venture
/startup
for the
pre-
incubat
ion/inc
ubation
support
and the
service
s
offered
by Pre-
incubat
ion/inc
ubation
or
institut
e
during
the
financi
al
period
2019-
20
Has
the
Start-
up
Grown
to a
stage
of
Employ
ment
Genera
tion to
10
Numbe
rs or an
annual
Turnov
er to
Rs. 50
Lakhs
during
the
financi
al year
2019-
20
1 INNOTRIX
SOFT
SOLUTION
Service
Innovation
Techno
logical
507 Alum
ni Venture with
Corporate
Identification
Number (CIN)
Registration
INNOTRIX
SOFT
SOLUTION
2014
-15
Received
Incubation
Support from
Incubation
Unit
No No No No No
Name DIN number Type Gender Email Contact
B.KUMARAVADIVEL a125496896 Alumni Entrepreneur Male innotrixsoftsolutions@gmail.c
om
9842689745
2 ZERO
FLAG
TECHNOVA
TION
Service
Innovation
Techno
logical
839 Alum
ni Venture with
Corporate
Identification
Number (CIN)
Registration
ZERO
FLAG
TECHNOV
ATION
2014
-15
Graduated
from the
Incubation unit
No No No No No
Name DIN number Type Gender Email Contact
A.DHINESHKUMAR asf414984984 Alumni Entrepreneur Male innotrixsoftsolutions@gmail.c
om
9994568754
Section 7 > i : List all the Collaborations with External Agencies and Co-Creation of I&E Activities by the HEI
Srno Collaborated Agency Name Collaborated Agency Type Purpose of Collaboration Type of Support Offered
No. of Student
Projects/ Ideas /
Innovation / Startups
from the HEI get
benefitted through
this partnership
Date of MOU /
Agreement Signed
on
1 Accent Techno Soft Non-Govt. Agencies Collaboration with Startups/Industry Associations/Knowledge
Agencies to Promote I&E Activities and/or Internship Opportunities
Networking 110 09-07-2019
2 NHAI Govt. Agency (Central) Collaboration with Startups/Industry Associations/Knowledge
Agencies to Promote I&E Activities and/or Internship Opportunities
Incubation 50 13-08-2019
3 LITZ Tech Non-Govt. Agencies Collaboration with Startups/Industry Associations/Knowledge
Agencies to Promote I&E Activities and/or Internship Opportunities
Networking 5 22-10-2019
4 NIIT-LTD's Non-Govt. Agencies Collaboration with Startups/Industry Associations/Knowledge
Agencies to Promote I&E Activities and/or Internship Opportunities
Networking 52 24-09-2019
5 Breathe Electronics Corporate / Industry Association Collaboration with Startups/Industry Associations/Knowledge
Agencies to Promote I&E Activities and/or Internship Opportunities
Incubation 44 20-08-2019
6 Livewire - A division of CADD centre
training services Pvt Ltd
Corporate / Industry Association Collaboration with Startups/Industry Associations/Knowledge
Agencies to Promote I&E Activities and/or Internship Opportunities
Incubation 3 21-11-2019
Section 8 > ii : List the IPs Filed/Published/Granted (Copyright & Designs Only) by the HEI as an Applicant / Co-applicant / Co-owner
No Data Flled!
Section 8 > iii : List the IP (Patents only) Filed by the HEI as Applicant / Co-applicant / Co-owner
Srno Title/Name of Patent IP Registration Number Date of Filed/Published (Year, Month) Application Level Copy of receipt / application
screenshot [PDF or Image]
1 INTELLIGENT IOT BASED SOIL
FERTILIZATION TEST AND THEIR
SOLUTION
201941031950 National 2019 - 08 Uploaded
2 G glass for Electrical Power Generation and
Electrical Energy Storage
201941046543 National 2019 - 12 NA
3 G glass Smart Manufacturing Unit 201941046540 National 2019 - 12 Uploaded
4 Performance Improvement of Induction
generator in grid integrated wind energy
conversion system
201941046537 National 2019 - 12 Uploaded
5 High Performance Air Purifier Using Human
Hair and Vetiver (Chrysopogan  Zizanioides)
as Filter Media
201941051988 National 2020 - 01 Uploaded
6 A SYSTEM AND METHOD FOR
MONITORING AND PROCESSING
WASTEWATER AND  RAINWATER
HARVESTING IN FORMAL HABITATION
201941036236 National 2019 - 09 Uploaded
7 Method for Managing the uplink radiation and
Energy consumption of 5G Mobiles
202041009329 National 2020 - 03 Uploaded
8 Coconut Multi Processing Machine 202041009037 National 2020 - 03 Uploaded
9 Optimization of Carbon Content in Aqua
Silencer by using lithium hydroxide
202041009035 National 2020 - 03 Uploaded
10 Driverless Renewable Energy Hybrid Vehicle
with Bluetooth Control
202041009040 National 2020 - 03 Uploaded
11 SYSTEMATIC MODEL FOR CONSTANT
WATER QUALITY  MONITORING USING
IOT
202041006184A National 2020 - 02 NA

Program Representations for Predictive Compilation:
State of Aﬀairs in the Early 20’s
Anderson Faustino da Silva
UEM, Brazil
Edson Borin
UNICAMP, Brazil
Fernando Magno Quint˜ ao Pereira
UFMG, Brazil
Nilton Luiz Queiroz Junior
UEM, Brazil
Ot´ avio Oliveira Napoli
UNICAMP, Brazil
Abstract
In the last ﬁve years, predictive compilation has advanced with long strides.
Contributions in the ﬁeld include new program embeddings, new learning ar-
chitectures, and datasets with millions of programs. This paper evaluates 25
state-of-the-art program embeddings, three of them new, plus two learning
models from previous work. We have trained this apparatus with three large
datasets, and have applied it onto three classiﬁcation problems. When classi-
fying programs according to the problem that they solve, we reproduced the
high-accuracy results seen in previous work. However, we have not been able
to repeat these results in the two new classiﬁcation challenges that we study:
namely, determining the depth of the most nested loop in a program and de-
Email addresses: anderson@din.uem.br (Anderson Faustino da Silva),
edson@ic.unicamp.br (Edson Borin), fernando@dcc.ufmg.br (Fernando Magno Quint˜ ao
Pereira), niltonlqjr@gmail.com (Nilton Luiz Queiroz Junior),
onapoli@lmcad.ic.unicamp.br (Ot´ avio Oliveira Napoli)
Preprint submitted to Elsevier November 9, 2022
termining the best sequence of optimizations to reduce code size of programs.
Negative results emerged, even in spite of the large number of classiﬁers, 25,
that we have evaluated. Surprisingly, using the histogram of instruction op-
codes, a very simple program embedding, led to about the same classiﬁcation
accuracy than embeddings like Ir2Vec or Inst2Vec, which were designed to
solve stochastic compilation tasks.
1. Introduction
A predictive compilerinfers the characteristics of an unknown program based
on statistics collected from a set of known codes. This inferred knowledge can
be used in diﬀerent ways. A non-exhaustive list of examples include: proposing
meaningful names to variables and functions [1]; discovering algorithms hidden
in code [2]; predicting the performance of code [3, 4, 5]; determining the best
sequence of optimizations for each program [6, 7]; and approximating execution
proﬁles without running the code [8].
State of Aﬀairs. To be eﬀectively used, a predictive compiler requires a syner-
gistic combination of three elements: an embedding function, a learning model,
and training data. The embedding function maps the program into a set of nu-
meric arrays which, in turn, will be given as input to the learning model. These
arrays are known as the programembedding. The training data—programs (and
sometimes their inputs)—are used to ﬁt the model to the desired task, so that it
can accurately infer characteristics of unseen programs based on features present
in the known ones.
Much of the interest in the study of predictive compilation had its origin
in the late nineties [9, 10, 11, 12]. However, recent years have seen a surge in
the popularity of the subject. New embedding functions include Inst2Vec [2],
Code2Vec [1], Ir2Vec [13], Asm2Vec [14], and ProGraML [15], for in-
stance. Recent classiﬁcation and regression models include neural networks
proposed by Mou et al. [16], Cummins et al. [4] and Brauckmann et al. [17],
2
for instance. And datasets with hundreds of thousands of programs have been
released by Faustino et al. [6], Mou et al. [16], and Puri et al. [18].
Contributions of This Work. It is our understanding that the programming
language community would beneﬁt from an evaluation of these recently proposed
embeddings functions, datasets, and models. Motivated by this understanding,
we published in 2021 a suite of ﬁve sequences of compiler optimizations, with
the goal of using them as targets for stochastic prediction models [19]. At that
time, we left open the following question: “how diﬀerent regression models would
fare when classifying programs according to the sequence that best reduces code
size?” The present work provides answers to that question; hence, extending
our earlier study with a collection of new experiments that compare the relative
power of diﬀerent prediction models when used to solve compilation tasks.
Nevertheless, this paper goes beyond the questions left open in our previ-
ous publication. We compare the eﬀectiveness of seven embedding functions
to solve three code classiﬁcation problems, using two models designed for pro-
gram analysis. Section 2 describes the seven embedding functions. Three of
them—Subject of Section 2.2—are original material from this paper. The other
four embedding functions were taken from artifacts discussed in previous work.
We have combined these previous embedding functions with diﬀerent program
representations in various ways, so that in total we could evaluate 25 program
embeddings. Section 3 describes the classiﬁcation models that we use in this
paper—all of them techniques introduced in previous work. Finally, Section 4
describes the three classiﬁcation problems that we evaluate, namely:
Prob: given k programming problems, plus a programp, determine the problem
that p solves.
Loop: given a program p and k nesting levels, determine the level of p’s most
nested loop.
3
OptZ: given a program p, plus k optimization sequences1, determine the se-
quence that yields the smallest code for p.
The ﬁrst problem, Prob, is a classic challenge in the program analysis litera-
ture [2, 16, 20, 21]. Loop is new in the context of stochastic classiﬁcation. This
paper introduces it as a sanity check to verify the ability of diﬀerent classiﬁers
to detect structural information about a program that can be easily inferred by
any static analysis tool. Finally, OptZ is the classiﬁcation problem introduced
in the preliminary study that the current paper extends [19]. To keep this pa-
per self contained, Section 4.2 was taken almost without modiﬁcation from that
early work of ours. That ﬁrst publication has also inspired the methodology
that we adopt in Section 5. With regard to this methodology, notice that all
these problems encode exact objective functions, i.e., which are not subject of
statistical ﬂuctuations. That is the reason why OptZ focuses on code size as
the target metric. Contrary to execution speed or energy consumption, code
size does not vary across repetitions of the same experiment.
Baselines for future explorations. This paper evaluates eight diﬀerent instances
of the problems previously introduced: two for Prob, two for Loop, and four
for OptZ. These instances diﬀer with respect to the dataset. Two of the
datasets used in this paper, Poj (short for Programming Online Judge) [16]
and CodeNet [18], have been mined from programming online judges. The
other dataset, AnghaBench [6], was mined from general open-source repos-
itories. As we shall explain in Section 4.1, producing well-balanced datasets
requires caution, lest spurious correlations compromise the accuracy of the clas-
siﬁer. The evaluation of these diﬀerent instances of classiﬁcation problems de-
manded substantial computing time. However, adding new embedding functions
or classiﬁcation models to our collection of results does not require redoing these
1This paper considers only optimization sequences constructed from the optimization
passes available in the clang compiler. As an example, clang -sccp -instcombine -sccp
-simplifycfg is a sequence formed by the invocation of four optimizations. Notice that a
sequence of optimizations can contain repetitions. In this example, the sparse conditional
constant propagation, which is invoked by the -sccp ﬂag, appears twice.
4
experiments, as long as the datasets remain ﬁxed.
Summary of Observations. The evaluation in Section 5 shows that almost all
the combinations of embedding functions and learning models could solve Prob
on programs from Poj and CodeNet with high accuracy—often above 90%. In
contrast, none of these combinations—25 in total—when applied onto the other
classiﬁcation problems, fared much better than random guessing. For instance,
some classiﬁers could detect the existence of loops in programs with accuracy
between 60% and 70%; however, a random guesser is already expected to be
right 50% of the time on a balanced dataset. From these observations, we draw
the following conclusions:
Similarity: The majority of the 25 combinations of program representations
and embedding functions evaluated in this paper—22 from previous work—
tend to perform similarly. On Prob, they yield equally high accuracy. On
OptZ and Loop, they yield low accuracy.
Triviality: Section 5.1 shows that the high-accuracy results reported in the pro-
gramming languages literature on datasets extracted from programming
contests are to be expected, because they tend to be easy to separate and
classify.
Limitation: Section 5.2 shows that the graph-based neural network, fed with
six program representations from previous work, could not be trained to
recognize the nesting level of the deepest loop in a program—a simple
structural property of code.
Discretion: Section 5.3 indicates that caution must be exercised when using
POJ-based datasets to train models for classiﬁcation; otherwise, the clas-
siﬁer can be biased by the easily separable characteristics of the classes of
problems in the dataset.
Simplicity: Section 5.4 shows that none of the representations studied in this
paper consistently outperforms a histogram built with the opcodes of in-
5
structions for any of the three studied problems. In terms of implementa-
tion, the histogram is the simplest representation evaluated in this paper.
Consistency: Section 5.5 reports that classiﬁers like Support-Vector Machines
or Decision Trees, using histograms as embeddings, perform no worse than
graph-based neural networks used in previous work to do predictive com-
pilation, and depending on the problem, can even do better.
Image Based Program Representations. The embedding functions that we reuse
from previous work are one-way; hence, from a program embedding it is not pos-
sible to reconstruct the original program. This paper also evaluates an informa-
tion preserving embedding, which Section 2.2 introduces—the binary encoding
of the program. We call this embedding Prog2Image. Prog2Image gives
us the opportunity to reuse techniques already well-established in computer
vision to classify programs. Yet, even though Prog2Image encodes more in-
formation than the embeddings taken from previous work, it does improve the
ability of the considered models to solve OptZ or Loop. To demonstrate that
Prog2Image encodes useful information, we compare it with the Local Bi-
nary Pattern (LBP) [22], an embedding produced by an image summarization
technique. Prog2Image is considerably more eﬀective than the summarized
representations to solve the three problems considered in this paper, but it is
unable to outperform simple histograms of instructions in these very problems.
2. Representations and Embeddings
The machine learning models considered in this paper receive an embedding
extracted from a program and outputs aclass to which that program is expected
to belong. The class depends on the problem of interest: each one of the three
problems enumerated in Section 1 (Prob, Loop and OptZ) deals with diﬀerent
classes. Embeddings, in turn, relate programs and classiﬁcation models. This
notion, being a central concept to this paper, is given by Deﬁnition 1.
Deﬁnition 1 (Embedding). In the context of this paper, an embedding of
a program is an array of numerical values that represents that program. An
6
embedding function maps a program representation to an embedding with a ﬁxed
number of dimensions.
This paper considers seven embeddings that can be extracted directly from
the LLVM intermediate representation of a program. Table 1 shows them 2.
Four of these embeddings— Ir2Vec [13], Milepost [23], Inst2Vec [2], and
Histogram [24]—have been discussed in previous work. We refer the reader
to Section 6.2 for an explanation about the similarities and diﬀerences between
these embeddings. We have chosen these four embeddings because we could
ﬁnd their implementations in public artifacts. The three other embeddings—
Prog2Image, LBPif , and LBPeq—are introduced in Section 2.2. We adopt them
just to experiment with embeddings that preserve every information in the pro-
gram. In this sense, notice that Ir2Vec, Milepost, Inst2Vec, Histogram,
LBPif , and LBPeq are one-way: the original program cannot be recovered from
them. Prog2Image, in turn, is a two-way embedding, which can be mapped
back onto the original program.
Embedding Reference Dimensions Dim. Size (bits)
Ir2Vec VenkataKeerthy et al. [13] 300 64
Inst2Vec Ben-Nun et al. [2] Niv ×200 64
Milepost Namolaru et al. [23] 56 64
Histogram Cummins et al. [15] 63 64
Prog2Image Section 2.2.1 Npg ×256 1
LBPif Section 2.2.2 256 64
LBPeq Section 2.2.2 256 64
Table 1: Seven embeddings evaluated in this paper that can be derived directly from the
LLVM intermediate representation of programs. Dimensions is the number of cells in the
arrays that constitutes the embedding. The dimensionality of Inst2Vec and Prog2Image
depend on the dataset. In this case, users must deﬁne the parametersNiv and Npg. Dim.Size
is the size, in bits, of each cell of the array that constitutes the embedding.
Example 1. Figure 1 (a) shows a function written in C. Figure 1 (b) depicts
2Each name in the ﬁrst column of Table 1 denotes an embedding, i.e.. an array that can
be used as the input of a classiﬁcation model. In the rest of this paper, we also use these
names to denote embedding functions, i.e., functions that convert the LLVM intermediate
representation of a program to each embedding. Context will make clear if we are referring
to embedding functions or to embeddings—the product of these functions.
7
its control ﬂow graph. In this lower-level representation, the program is formed
by ﬁve diﬀerent instructions. Figure 1 (c) shows the histogram of opcodes—a
numerical array that represents the program, and that can be fed as input to
a learning model. Notice that arrays from a particular embedding have always
the same number of dimensions. In this example, any program will generate a
histogram as an array with a ﬁxed number of cells—each cell denoting a diﬀerent
opcode. The number of dimensions of the other embeddings is given in the
column Dimensions in Table 1.
l0: p = n > 1
l1: brn p l2 l3
l2: jmp l3
l3: r = phi(n, 0)
l4: ret r
int foo(int n) {
  int r = 0;
  if (n > 1)
    r = n;
  return r;
}
(a) (b)
1 1 10 11h
(c)
brngrtjmpphiretadd
Figure 1: Embedding based on the histogram of instructions. We assume an assembly language
with six opcodes. (a) Target program written in C. (b) Control ﬂow graph of the program. (c)
Histogram of program instructions, assuming an assembly language with six diﬀerent opcodes.
2.1. Graph-Based Program Representations
The domain, i.e., the input type, of the embeddings functions used in this
paper are either programs written in the LLVM intermediate representation (IR)
or in a graph-based version of the LLVM IR taken from previous work. Table 2
shows the six graph-based representations that we consider in this paper and
Deﬁnition 2 formalizes this concept. None of the six graph-based representa-
tions that we use are interpretable. In other words, in contrast to the LLVM IR,
the graph-based representations do not encode enough information to execute
the program that they represent. Rather, they aim at highlighting code char-
acteristics that tend to be good for predicting properties of programs. These
characteristics include the dependencies between instructions and between data,
the possible execution paths within the program, the occurrence of particular
instructions in the code, etc.
8
Deﬁnition 2 (Graph-Based Representation). Let p be a program with pv
elements of interest (the vertices), such as instructions, basic blocks, data loca-
tions, etc.; plus pe relations between elements (the edges), such as control and
data dependencies. A graph-based representation of p is a tuple (Ma, Mv, Me),
where:
• Ma is a |pv|×| pv|array that represents adjacent relations between the
graph nodes.
• Mv is a |pv|×nva array that encodes the nva attributes of each node.
• Me is a |pe|×nea array that encodes the nea attributes of each edge.
I: # instructions bb: # block nodes (==I) emb: Dimensions of embedding
B: # basic blocks jt: # jump targets ct: # instrs. that read constants
M: # mem. locations sq: # instrs. in sequence mm: # instrs. that read mem. locs.
C: # unique constants dt: # data dependencies
Rep. Reference Ma Mv Me
cfg Brauckmann et al. [17] I × I I × emb jt + sq
cfg c Faustino et al. [20] B × B B × emb jt
cdfg Brauckmann et al. [17] I × I I × emb jt + sq + dt
cdfg c Faustino et al. [20] B × B B × emb jt + dt
cdfg+ Brauckmann et al. [17] (I + B) ×
(I + B) (I + B) × emb jt +sq+dt+bb
ProGraML Cummins et al. [21] (I +M +C)×
(I + M + C) (I+M+C)×emb jt +sq+ct+mm
Table 2: Graph-based program representations considered in this paper.
From Graphs to Embeddings. Graph-based program representations cannot be
used directly as inputs of learning models: they need to be mapped onto arrays
for that purpose. These arrays will encode the vertex ( Mv) and edge ( Me)
attributes of the graph. Together with the adjacent matrix Ma, these arrays
will be the arguments eﬀectively passed to the model. The same graph attribute
can be translated into an array using diﬀerent embedding functions. In this
paper, we experiment with three diﬀerent embedding functions to map graph
attributes to arrays. We use functions that map these entities to either the
Histogram, the Inst2Vec, or the Ir2Vec embedding. Because we consider
six graph-based representations, and each one can be mapped via any one of the
9
three embedding functions, this combination gives us 18 more embeddings to
evaluate, in addition to the seven direct embeddings listed in Table 1. Figure 2
clariﬁes this modus operandi.
Fa
Fv
Fe
Mv
Me
Ma
|pe|×
{sq, tr}
|pv|×
HISTOGRAM
|pv|×|pv|×
{0, 1}
DGCNN
LLVM Intermediate 
Representation
(Low-level Instructions)
cdfg Embedding 
Function Embedding Model
P
Fhist
|1|×
HISTOGRAM DCNNP
cdfg_histogramhistogram
Figure 2: How diﬀerent program representations are mapped onto a format that can be fed
to a classiﬁcation model. The names cdfg inst2vec and inst2vec will be used in Section 5
to denote two diﬀerent classiﬁcation techniques.
An embedding function can translate the LLVM IR directly to an array,
like Fhist in Figure 2, or it can translate a graph-based representation of that
program, like Fa, Fv, and Fe in the same ﬁgure. Notice that, in the latter
case, an embedding function must be applied onto each vertex and each edge
of the graph; hence, producing three matrices. Example 2 explains how his-
tograms can be used as embeddings for graph-based representations. Regarding
the other embeddings, Ben-Nun et al. [2] deﬁne functions to map program in-
structions (vertices) and dependence relations (edges) to Inst2Vec vectors;
VenkataKeerty et al. [13] describe similar functions for Ir2Vec.
Example 2. Figures 3 (a) show the arrays Ma, Mv, and Me derived from the
CFG in Figure 1 (b). The embedding of Mv (Fig. 3-b) is a two-dimensional
array, with as many lines as there are vertices in the program, and as many
columns as there are opcodes in the assembly language. Thus, each line is the
Histogram-based embedding of a single vertex. The function that produces this
10
matrix is Fv in Figure 2. The embedding function of Me (Fig. 3-c) maps edges
to one of two constants: either sq, in case they represent instructions aligned
sequentially in memory; or to tr, in case they represent branches (conditional or
otherwise). As Figure 3 (c) shows, the embedding of edges is a one-dimensional
array with as many cells as there are edges in Me. The embedding function that
produces this matrix is Fe in Figure 2.
1
1
1
1
1
brn grt jmp phi retadd
l0
l1
l2
l3
l4
1
1
11
1l0
l1
l2
l3
l4
l0 l1 l2 l3 l4
l 0 → l 1
l 1 → l 2
l 2 → l 3
l 1 → l 3
l 3 → l 4
tr tr sqtrsq
(b)(a) (c)
Figure 3: Graph-based program representation of code seen in Figure 1 (a) mapped onto
a histogram-based embedding. (a) Adjacent matrix Ma derived from the CFG. (b) Vertex
attributes Mv. (c) Edge attributes Me.
Some of the graph representations are compact ( cfg c and cdfg c); which
means that instructions are grouped into basic blocks. In this case, the em-
bedding of a basic block is the cumulative function of the embeddings of the
individual nodes (see Section 6 for deﬁnitions). In other words, instead of gen-
erating a histogram for each instruction (which contains only one active bit),
a compact representation uses the histogram of all the instructions in a basic
block. This is the approach adopted by VenkataKeerthy et al. [13] and Ben-Nun
et al. [2].
2.2. Image-Based Program Representations
In this paper, we experiment with a few new ways to represent programs.
These representations are reinterpretations of approaches used in the ﬁeld of
computer vision. This section explains these representations.
2.2.1. Prog2Image
Prog2Image is a bit-map representation of the binary encoding of a pro-
gram. This encoding can be extracted from the program’s Executable and Link-
11
able Format (ELF) or from its LLVM intermediate representation. Our imple-
mentation of Prog2Image is parameterized by a natural number N, which
denotes the number of lines of the program’s image. Our implementation ﬁxes
the number of columns in the program’s image in 256. Thus, given some ma-
chine learning task that uses Prog2Image⟨N⟩, every program involved in this
task will be represented as an N ×256 array. Fixing these dimensions per
learning task is necessary because such a task uses a machine learning model
that receives, as inputs, always arrays with the same number of dimensions.
Deﬁnition 3 clariﬁes the usage of N.
Deﬁnition 3 (Prog2Image). Let Pbe the universe of programs, each encoded
as an array of bits. If p ∈P , then p[i] is the ith bit of p, where indexation
starts at zero; and |p|is the number of bits in p. For any p ∈P , such that
Prog2Image⟨N⟩(p) = m, we let:
m[i, j], i < N=



p[i ×256 + j] if i ×256 + j <|p|;
−1 otherwise.
According to Deﬁnition 3, Prog2Image⟨N⟩produces bit-maps of size N ×
256. If |p|≥ N ×256, then Prog2Image⟨N⟩(p) stores the ﬁrst N ×256 bits of
p. If p < N×256, then the unused bits are set to minus one. In the experiments
discussed in Section 5, N, is chosen according to the dataset of interest, so that
N ×256 exceeds the size of the largest program in that dataset. This approach
ensures that Prog2Image preserves all the information in the programs under
study. Thus, given a dataset of programs P, Prog2Image⟨N⟩(p), p∈P is
injective, as long as N ×256 is larger than the size, in bits, of the largest
pmax ∈P. Notice that Prog2Image⟨N⟩is not surjective, as there are images
that do not correspond to any valid program. Example 3 shows images of
diﬀerent programs.
Example 3. Figure 4 shows the binary images extracted from the LLVM in-
termediate representation of three diﬀerent programs. We let N = 300; thus,
Prog2Image can represent the ﬁrst 300 ×256 bits of any program, encoded in
12
the LLVM IR. The ﬁrst program in Figure 4 was larger than this limit; thus, its
image has been cropped at the end. The other two programs are smaller; thus,
the remaining bits have been ﬁlled with -1’s.
256
300
Figure 4: The Prog2Image view of three diﬀerent programs. The dashed area represents
unused bits in the image matrix.
2.2.2. Local Binary Patterns
Local Binary Patterns (LBP) are visual descriptors used in image classi-
ﬁcation [22]. In this paper, we experiment with two embedding functions
based on local binary patterns: LBPif and LBPeq. Both produce 64-bit arrays
h of size 1 ×256. Given a program p, h is produced in three steps, where
L = (⌈|p|/256⌉×256) and C = 256:
1. Build an L ×C matrix mimg as Prog2Image⟨L⟩(p). Each cell of mimg
has one bit.
2. Build an L ×C matrix mlbp from mimg. Each cell of mlbp has eight bits.
3. Compute the histogram h of mlbp (with 256 entries) and normalize its
entries by dividing each element by the sum of all the elements. Each
entry is a 64-bit double between 0.0 and 1.0.
The embeddings LBPif and LBPeq diﬀer only with respect to step (2), above. To
construct mlbp, we apply a function Neigh onto each cell of mimg. Neigh
produces an eight-bit value b for each element of mimg. This value, b, is
13
a function of eight neighbors of each bit, considering a neighborhood of ra-
dius two using the Manhattan distance 3. Figure 5(a-b) shows how the neigh-
borhood is used to index b. The neighborhood function used in LBPif corre-
sponds to the following boolean implication: if n is a neighbor of m[i, j], then
Neighif (m[i, j], n) ≡(m[i, j] →n). The neighborhood function used in LBPeq
is the identity applied onto the neighbor, e.g., Neigheq(m[i, j], n) ≡n.
xn0
n2
n4
n6
n1 n3
n5n7
n0 n1 n2 n3 n4 n5 n6 n7
NEIGHif(x, n)x n
0 0 1
0 1 1
1 0 0
1 1 1
(b)
NEIGHeq(x, n)n
0
1
0
1
(c) (d)
(a)
Figure 5: (a-b) The correspondence between neighborhood and positions in the LBP byte.
(c-d) The neighborhood functions used to build LBPif and LBPeq.
3. Classiﬁcation Models
In the context of this paper, amachine learning model is a function that takes
a program embedding as an input and outputs a class of interest. Example 4
clariﬁes this deﬁnition. There exist a very large number of diﬀerent ways to
implement such functions: random forests, support vector machines, logistic
regression, neural networks, etc. The bulk of our experiments use the Deep
Graph Convolutional Neural Network ( dgcnn) proposed by Zhang et al. [25],
although Section 5.5 evaluates additional models.
Example 4. A classiﬁcation model can receive the histogram of opcodes of a
program and output a number in the set {0, 1}. Each number denotes a class of
3In principle it should be possible to consider less neighbors ( e.g., the four cardinal neigh-
bors) or more neighbors ( e.g., the 24 points of distance two); however, in this paper we opted
to follow the description of the original LBPif algorithm [22].
14
programs: zero represents programs without loops; one represents programs with
one or more loops.
Deep Graph Convolutional Neural Networks. dgcnn is a learning architecture
that receives attributes that characterize graphs. In this paper, we have adapted
them to receive the three embeddings that emerge from a Graph-Based program
representation (see Deﬁnition 2). We refer the reader to Figure 2 and Exam-
ple 2 to recall these attributes. We have reused the original architecture of a
dgcnn proposed by Zhang et al. [25], which we describe below for the sake of
completeness.
1. four graph convolutional layers, with 32, 32, 32, and one unit with the
hyperbolic tangent activation function;
2. a one-dimensional convolutional layer;
3. a max pooling layer;
4. a one-dimensional convolutional layer;
5. two dense layers, the second being used for classiﬁcation; and
6. a dropout layer after the ﬁrst dense layer.
In the above architecture, the convolutional and dense layers, except the last,
use rectiﬁed linear unit ( ReLU) as the activation function. The last layer uses
softmax as the activation function. This is the setup originally proposed by
Zhang et al. [25].
Deep Convolutional Neural Networks. In this paper, in addition to using the
graph-based representation of programs, we also adopt array-based embeddings
such as Ir2Vec and Histogram, which can be fed directly to ordinary neural
networks. To learn program properties directly from these embeddings, we
opted to use a redesign of Zhang et al.’s [25] dgcnn architecture. We reuse the
same network conﬁguration, without the four ﬁrst layers. Notice that these four
layers only exist to merge the diﬀerent graph attributes (Ma, Mv, and Me) into
a single array, which will be then used to start the classiﬁcation process. Thus,
when receiving a simple array as the program embedding, these layers ﬁnd no
service.
15
Training. A model must be trained over an ensemble of known programs be-
fore being used to classify unknown programs. In our experiments, training of
either the dgcnn or the dcnn happens over 200 epochs, with a patience of 20
epochs. Patience is the number of epochs to wait before stopping calibration if
no progress on the validation set can be observed.
4. Classiﬁcation Problems
We let a classiﬁer be deﬁned by an embedding function (Table 1) plus a
learning model (Section 3). In Section 5, we shall probe the accuracy of diﬀerent
classiﬁers, a challenge that we summarize in Question 1. This section describes
the experimental setup to be used in that evaluation.
Question 1. What is the accuracy of the classiﬁer when applied onto the prob-
lems listed in Section 4.1.
Accuracy. We deﬁne accuracy as hits/guesses. Guesses is the number of
instances of the classiﬁcation problem, and hits is the number of instances cor-
rectly classiﬁed. Problems in Section 4.1 are balanced, meaning that, given a
code sample p, plus k classes C1, . . . Ck, the probability that p belongs into any
of the classes is exactly 1 /k. This simpliﬁcation of the dataset lets us equalize
accuracy with eﬀectiveness without having to resort to a weighted F1-score.
4.1. Problems of Interest
We evaluate Question 1 on the three classiﬁcation problems stated in Sec-
tion 1. Each problem contains multiple instances, which vary with regard to
the number k of classiﬁcation buckets (classes) and with regard to the dataset.
Table 3 shows the parameters used to build the diﬀerent instances of each clas-
siﬁcation problem. To distinguish them, we suﬃx the problem name with the
ﬁrst letter of the source dataset (A ngha, POJ-104, and CodeNet).
Instances of Loop diﬀer with relation to the number of classes: either two
or four—we distinguish them by suﬃxing them with the digits 2 and 4. In the
case of Loop-2, we want to ﬁnd programs either without (class 0) or with (class
16
Problem Dataset Classes S/Cls Train. Valid. Test.
Fig. 6 Prob-P POJ-104 5 500 1 500 500 500Fig. 7 Prob-C CodeNet
Fig. 8 Loop-2
Angha-10K
2 1 200 1 440 480 480
Fig. 9 Loop-4 4 300 720 240 240
Fig. 10 OptZ-A
5
500 1 500 500 500Fig. 11 OptZ-C CodeNet
Fig. 12 OptZ-P POJ-104 365 1095 365 365
Fig. 14 OptZ-P-sp 365 1095 365 365
Table 3: Parameters of the dataset used in each classiﬁcation problem. Classes indicates the
number of classes and S/Cls the number of samples per class. Loop-2 is a dataset formed by
programs with and without loops (classes 0 and 1). Loop-4 is a dataset formed by programs
with 0, 1, 2, or 3 (or more) nesting levels. Each class of OptZ-C and OptZ-P is formed by
programs taken from several classes of problems from CodeNet and POJ, respectively. Next
to each dataset, we show the ﬁgure that analyzes the performance of diﬀerent models when
applied onto that dataset.
1) loops. Loop-4 seeks to classify programs in four classes. The ﬁrst is formed
by programs without loops. The second, by programs with at least one loop,
but no nested loops. The third, by programs with at least a doubly nested loop,
but no triply nested loops. The fourth class is formed by programs with loops
containing three or more nesting levels.
4.1.1. On the Choice of Problems
The number of program properties that can be inferred through stochastic
models is virtually inﬁnite (We refer the reader to Section 6.3 for an overview).
Thus, given the vast number of diﬀerent problems that we could have studied,
an immediate question is why we chose Prob, Loop, and OptZ. This section
provides some rational on these choices.
Prob: Clone Detection. The ﬁrst problem of interest, Prob, has been cho-
sen as a mechanism to ensure that our experimental methodology is able to
reproduce results from previous work. This problem, often dubbed clone detec-
tion, is ubiquitous in the literature [16, 17, 25]. Indeed, the two learning models
described in Section 3 were borrowed from a work that seeks to solve Prob [25].
Loop: Nesting Depth. The choice for Loop was motivated by more subtle
reasons. Our goal with this experiment was to check whether the embeddings
17
and ML models evaluated in this work are capable of learning a simple code
property. Loop can be exactly solved via a graph-traversal algorithm applied
onto the program’s control ﬂow graph—something that static-analysis tools do
routinely. Interestingly, all the combinations of representations, models and
datasets that we used in an attempt to solve Loop failed to count the number
of nested loops in programs with more than 50% of accuracy. In other words,
many machine-learning models designed in previous work to solve compilation
tasks have limited capacity to identify structural program features that are
simple to extract via static analyses.
Given that Loop is parameterized on the maximum loop nesting level of
a program, the number of instances of this problem is unbounded. However,
the deepest nesting level in most of our benchmarks is two; hence, it is diﬃcult
to build instances of Loop using more than three classes. In particular, if we
consider more than three nested loops, then we would not ﬁnd 300 programs
among the largest ten thousand benchmarks in AnghaBench. Therefore, in
this paper, we consider only two instances of Loop, as seen in Table 3. The
choice for Loop-2 is natural: we want to separate programs into code with and
without loops. The second instance, Loop-4, is the highest nesting level for
which we could ﬁnd at least 300 programs in AnghaBench-10K.
OptZ: Optimization Sequences. Finally, we chose to study OptZ because it
has one of the largest search spaces out of the problems that are common in
the domain of predictive compilation. Thus, it provides an ideal benchmark
to the kind of study that this paper discusses. A simpler incarnation of this
problem, where the order—by not the presence—of optimizations is ﬁxed, has
been studied before in the early days of predictive compilation [26, 27]. Yet, after
those forays of Cavazos et al.in this area, mentions toOptZ have become rare in
the scientiﬁc literature. It is our understanding that this problem is suﬃciently
important to the craft of predictive compilers to merit further analysis.
18
4.1.2. On the Construction of the Datasets
The problems in Table 3 use only datasets with disjoint classes. That is,
when building datasets for the diﬀerent instances of each problem we ensured
that no ties are possible. Consequently, each program used in training or vali-
dation belongs to a unique class, regardless of the problem. Moreover, all the
datasets are balanced, i.e., each class in a dataset contains the same number of
samples (Smps/Cls).
Classes for Prob can only be built from Poj and CodeNet, because the
AnghaBench programs are not grouped per problem that they solve. We
use the ﬁrst ﬁve classes of programming problems in Poj and in CodeNet—
ordering being given by the identiﬁers in each dataset. Classes forLoop can only
be built from AnghaBench, since the other datasets did not contain loop-free
programs. We restricted the search to Angha-10K, a subset of AnghaBench
containing its 10,000 largest programs.
4.2. Finding The Optimization Sequences for OptZ
OptZ asks for the best sequence of optimizations for a given program, among
a number of candidate sequences. In this paper, we say that a sequence of op-
timizations S0 is better than a sequence S1 for a program p if S0 generates less
LLVM instructions for p than S1. The process of constructing the candidate
sequences is described in our early work [19]. This section revisits this method-
ology, showing how we have constructed ﬁve sequences of optimization such that
none of them is strictly better than the others.
A Dataset to Explore the Optimization Space. Exploring the optimization space
of a compiler like LLVM demands a large number of benchmarks. As already
observed in previous work, more than 1,000 diﬀerent programs are required, on
average, for a random sequence of 100 optimizations to outperform opt -Os
with more than 10% of chance [7]. We drew our samples from the 15,000 largest
functions available in AnghaBench. Functions are provided as independent
compilation units: C ﬁles with a single function plus all the declarations to
19
render it compilable. This choice is pragmatic: we constrain our search to
15,000 programs to make it practical.
Search Methodology. To explore the dataset of 15,000 functions, we employed a
sequence of three steps, that work per program p in the dataset of programs:
1. Generate optimization sequences for p, using the genetic algorithm pro-
vided by YaCoS [24]. In this algorithm, individuals are numeric vectors,
where each position contains the index of an analysis or optimization pass.
We use a population of 100 individuals, with 100 genes each, and have
evolved it over 50 generations. The crossover probability is 0.9 and the
mutation probability is 0.1.
2. Reduce the best sequence for p, via Purini and Jain’s reduction algo-
rithm [28]. Such algorithm removes optimizations from a sequence. An
optimization is removed if it does not increase the number of instructions
that the resulting sequence produces for p.
3. Filter out duplicates from the reduced sequence set. Two sequences are
duplicates if they have the same optimizations, in the same order.
Notice that this search approach is similar to the kind of property-based
testing methodology that is fostered in a tool like Haskell’s QuickCheck [29].
In this case, testing inputs are generated in step (1), using to this end a genetic
algorithm, and then are reduced in step (2), using Purini et al.’s approach [30].
The property of interest is that the reduced sequence of optimizations is as
eﬀective as the original sequence for the target program.
After applying the above steps onto our dataset of 15,000 functions, we
got 10,044 sequences of various sizes. From this set of 15,000 programs, and
10,044 sequences, we build an optimization matrix , a notion that Deﬁnition 4
formalizes. In Section 4.2.1 we show how the optimization matrix lets us choose
ﬁve optimization sequences via ﬁve diﬀerent criteria.
Deﬁnition 4 (Optimization Matrix). The optimization matrix M is a table
of programs ×sequences. An entry ( Pi, Sj) in M represents the number of
20
instructions that sequence Sj produced when applied onto program Pi.
In total, it took about six weeks to build the optimization matrix on a
single core of a commodity Intel Xeon E5-2650 processor, running at 2.30 GHz
with 32G RAM. Notice that the construction of the matrix can be parallelized;
however, we opted to do it on a single core to avoid using for too long a machine
that was shared with other researchers. Notice also that the construction of
the optimization matrix can be done gradually: the addition of new programs
or new optimization sequences to the matrix does not require rerunning old
sequences on old programs. In other words, adding new information to the
matrix does not require any recomputation of information already stored there.
4.2.1. Properties of Interest
The main goal of our optimization sequences is to minimize the size of the
generated code. Nonetheless, there are several ways of comparing these se-
quences when analyzing their eﬀect on a set of 15,000 programs. Moreover,
depending on the property of interest, one optimization sequence may be better
than the other. The following list introduces the main properties of interest that
we have used to compare the optimization sequences. Any of these properties
can be directly mined from the optimization matrix.
• Best frequency (Bst): this property indicates how often the optimiza-
tion sequence produced the best code, i.e., a code that is the smallest
among the code produced by the 10,044 sequences. To derive this met-
ric, we count how often a sequence Si was the minimum per line of the
optimization matrix, and returns the most frequent winner.
• Good in bounds (Lim) : this property indicates how often the opti-
mization sequence produced a code that is considered good, i.e., it is at
most X times (e.g., 1.03×) larger than the best code. We explored three
diﬀerent thresholds to deﬁne what would be considered good (1.03, 1.05,
and 1.07). This metric is particularly useful when the diﬀerence in the
21
code size generated by a sequence, in relation to the code size generated
by best sequence, is small, since it discards this small variation.
• Best total size (Sum): to compute this metric, we sum up the columns
of the optimization matrix, and return the smallest sum.
• Relative-size geometric mean (Geo): the relative code size indicates
how much larger is the code produced by the optimization sequence when
compared to the smallest code produced among all sequences. For exam-
ple, a relative code size of 1.35 ×indicates that the code produced by the
optimization sequence for a given program is 1.35 times larger than the
best code produced among all sequences. We employ the geometric mean
to summarize the relative code size of all programs for a given sequence.
• Best maximum relative size (Cap): The maximum relative size is the
worse relative size achieved by an optimization sequence for all program.
The best maximum relative size is the smallest maximum relative size. In
other words, the optimization sequence that produces the best maximum
relative size, is the one that yields the least-worse code increase.
4.2.2. Five Sequences
Table 4 shows the best sequences found according to each property of in-
terest. The second column reports the optimizations, in order, that constitute
the sequence and the third reports the total number of passes invoked by those
optimizations. Notice that the number of optimizations and the number of
passes invoked diﬀer. Diﬀerences are due to dependencies between passes. For
instance, any sequence of optimizations must start with -targetlibinfo and
-tti. Similarly, any sequence must end with -verify. Therefore, if the deﬁned
sequence does not have these passes, they are added automatically.
Table 5 summarizes the results achieved by the selected sequences accord-
ing to each property of interest. The ﬁrst row shows that the Bst sequence
produced the smallest code in 4,161 benchmarks, a number that corresponds to
27.74% of all programs. The Lim sequence produced code sizes that are up to
22
Seq. Passes Total
Bst
mem2reg, jmp-threading, instcombine, early-cse-memssa,
jump-threading, licm, early-cse-memssa, sroa, simplifycfg,
reassociate, instcombine, slp-vectorizer, early-cse-memssa
57
Lim
sroa, early-cse-memssa, reassociate, instcombine,
simplifycfg, licm, speculative-execution, jump-threading,
early-cse-memssa, simplifycfg, instcombine, simplifycfg
56
Sum
mem2reg, early-cse-memssa, correlated-propagation,
instcombine, reassociate, simplifycfg, early-cse-memssa,
instcombine, licm, jump-threading, simplifycfg, dse,
reassociate, early-cse-memssa, instcombine
40
Geo
loop-vectorize, sroa, gvn, instcombine, simplifycfg,
instcombine, licm, gvn, correlated-propagation,
jump-threading, mldst-motion, early-cse-memssa,
instcombine, simplifycfg, instsimplify
79
Cap
loop-rotate, sroa, correlated-propagation, indvars, gvn,
tailcallelim, instcombine, jump-threading, reassociate,
simplifycfg, instcombine, early-cse-memssa
58
Table 4: The best sequences. The second column reports the passes found by our methodology,
and the third column reports the total number of passes invoked by the sequence.
1.03, 1.05, and 1.07 times larger than the smallest code for 11,012, 12,712 and
13,546 programs. These numbers correspond to 73.4%, 84.7% and 93.8% of all
benchmarks. The Sum sequence produced, in total, 3,295,639 instructions for
all 15,000 programs. Instructions, in this case, correspond to LLVM instruc-
tions. The Geo sequence achieved the best relative-size geometric mean. When
compared to the best sequence for each available program, on average, Geo is
only 1.0324 times worse. Finally, the Cap sequence produced the least-worse
relative-size. When compared to the best sequence for a particular benchmark,
it produced code that was 16.31 times larger. For all the other sequences it was
possible to ﬁnd worse relative results.
Table 5 also shows, for each sequence, its rank according to each property
of interest. For example, the Bst sequence achieved the best result (rank #1)
according to the “Best Frequency” property of Section 4.2.1. It got the second
best result (rank #2) on the “Good in bounds” property, in this case with a
threshold of 1.03. The Lim sequence achieved very good results (ranked 1 st
or 2nd) in most properties of interest. In fact, the only property of interest for
23
Best Good in Bounds Total Rel. size Best max.
Freq. ≤1.03 ≤1.05 ≤1.07 size (geomean) relative size
Bst #1
4 161
#2
10 993
#2
12 655
#7
13 453
#45
3 306 270
#70
1.037
#7610
455.0
Lim #2
4 097
#1
11 012
#1
12 712
#1
13 546
#2
3 295 888
#2
1.032
#539
69.0
Sum #28
3 873
#6
10 783
#3
12 583
#3
13 511
#1
3 295 639
#9
1.034
#1558
132.0
Geo #282
3 446
#14
10 671
#5
12 547
#2
13 544
#4
3 298 863
#1
1.032
#522
68.5
Cap #2 774
2 431
#3 787
7 412
#3 775
9 948
#3 687
11 627
#2 896
3 355 929
#2 089
1.050
#1
16.3
Table 5: Overall performance of selected sequences. Each cell contains two numbers, the ﬁrst
one indicates the rank of the property of interest among all the 10,044 sequences considered
in our optimization matrix. The second one corresponds to the performance of that sequence
in regard to the property of interest.
which this sequence did not perform well is the Best Maximum Relative Size (last
column). The Bst, Sum, and Geo sequences also achieved good or moderate
results on most of the properties of interest, the exception being, again, the
Best Maximum Relative Size. The Cap sequence, on the other hand, oﬀers the
best Maximum Relative Size but achieved poor results on the other properties
of interest. We emphasize that these sequences can—in speciﬁc benchmarks—
greatly outperform the default optimization levels used in LLVM. Experiments
in Section 5 corroborate this claim.
5. Experimental Results
This section discusses a number of experiments performed to answer Ques-
tion 1 (Page 16). Results are organized in ﬁve observations:
1. Section 5.1: The apparatus discussed in this paper reproduces the high-
accuracy results reported in previous work on Prob.
2. Section 5.2: Said apparatus fails to project these high-accuracy results
onto solutions to OptZ and Loop.
3. Section 5.3: Caution must be exercised when using POJ-like datasets to
solve problems unrelated to programming problem classiﬁcation.
4. Section 5.4: The simple histogram of opcodes is no worse than any of the
other embeddings in the apparatus for the considered problems.
24
5. Section 5.5: Results, at least if restricted to histograms (the most eﬀective
embedding overall), tend to be consistent across diﬀerent machine learning
models.
Hardware. These experiments have been performed in a machine with an
AMD EPYC 7282 processor, 64GB of RAM, plus a GPU Nvidia GeForce RTX
3090 with 24GB of RAM.
Software. Except for the embedding functions introduced in Section 2.2, the
implementation of the program representations used in this paper have been
taken from previous work. The implementation of Ir2Vec comes from the
work of VenkataKeerthy et al. [13] (we use the “Symbolic” implementation); the
implementation of Inst2Vec comes from the work of Ben-Nun et al. [2]; the im-
plementation of ProGraML comes from the work of Cummins et al. [15]; The
other implementations seen in Table 1, including Milepost, have been taken
from YaCoS [24]. These representations were extracted from the intermediate
representation of LLVM, version 10.0. The image-based representations from
Section 2.2 were implemented via SciKit-Image [31].
Measurement Methodology. As mentioned in Section 4.1, our three prob-
lems use discrete objective functions. Hence, one single experiment, consisting
of one round of training and validation followed by one round of testing, is free
from variations. However, variations in multiple experiments are possible, be-
cause the learning models use random weights for initialization. Thus, results
that we report are averages of ten experiments. The ﬁgures that show these
results (Fig. 6-14) use boxplots, showing medians, quartiles and outliers. This
paper applies 25 diﬀerent classiﬁers onto eight diﬀerent problems, for a total of
200 experiments, each involving ten iterations. To make it practical, we set a
time limit of six hours to ﬁnish all the ten experiments for each combination of
classiﬁer and problem. There were ﬁve instances of experiments that could not
ﬁnish within this allotted time.
25
5.1. Reproducing Previous Work
Out of the three problems presented in Section 4.1, only Prob has been
explored in previous work. Mou et al. [16], Zhang et al. [32], and Brauckmann
et al. [17], for instance, have been able to report hit rates about 90% when
matching programs with the problems that they solve on the Poj dataset. We
have observed similar results.
Discussion. Figures 6 and 7 show the average accuracy achieved by each one of
the 25 combinations of models and embeddings when classifying the programs
extracted from the Poj and CodeNet datasets. The expected accuracy of
a random classiﬁer would be 20%. On Prob-P, the version of Prob applied
onto Poj, the three best classiﬁers— cdfg c histogram, cdfg c ir2vec and
histogram—achieved accuracies of 94%, 93%, and 93%, respectively.
cdfg_c : histogramcdfg_c : inst2veccdfg_c : ir2veccdfg : histogramcdfg : inst2veccdfg : ir2vec
cdfg+ : histogramcdfg+ : inst2veccdfg+ : ir2vec
cfg_c : histogramcfg_c : inst2veccfg_c : ir2veccfg : histogramcfg : inst2veccfg : ir2vechistograminst2vecir2veclbpeqlbpif
milepostprog2img
programl : histogramprograml : inst2vecprograml : ir2vec
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%Accuracy
Figure 6: Accuracy results for Prob-P. Whenever presenting these results, we follow the order
in which each embedding was introduced in Table 1. Each graph-based representation has
three variations, determined by the embedding function, e.g.: Histogram, Inst2Vec, and
Ir2Vec.
High accuracy is also observed when training the learning apparatus on
CodeNet. However, in this case, the three best classiﬁers use embedding func-
tions applied directly onto the LLVM IR of programs: histogram, programl
26
ir2vec, and inst2vec. The accuracies of these classiﬁers were, respectively,
96%, 96%, and 94%. Embedding functions that read graph-based representa-
tions of programs performed slightly worse. Nevertheless, the diﬀerence between
the graph-based classiﬁer and the other techniques is too small to be considered
relevant in this experiment.
cdfg_c : histogramcdfg_c : inst2veccdfg_c : ir2veccdfg : histogramcdfg : inst2veccdfg : ir2vec
cdfg+ : histogramcdfg+ : inst2veccdfg+ : ir2vec
cfg_c : histogramcfg_c : inst2veccfg_c : ir2veccfg : histogramcfg : inst2veccfg : ir2vechistograminst2vecir2veclbpeqlbpif
milepostprog2img
programl : histogramprograml : inst2vecprograml : ir2vec
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%Accuracy
Figure 7: Accuracy results for Prob-C.
5.2. Limitations of Previous Work
Standard machine learning techniques have been very successful at solving
Prob, as seen in Section 5.1. One could expect that said techniques would be
eﬀective to solve also OptZ and Loop. However, such is not the case, as we
explain in the rest of this section.
Discussion–Loops. Figures 8 and 9 show the average accuracy achieved by each
one of the 25 classiﬁers when given programs from AnghaBench according to
their deepest loop nesting level. We omit results for inst2vec, as Ben-Nun et
al.’s [2] tool did not allow us to extract such embeddings for loop datasets—
eventually the software that converts programs to Inst2Vec arrays runs out
27
of memory. The diﬀerent classiﬁers achieved similar and low accuracy, i.e.,
between 20 and 40% on average.
cdfg_c : histogramcdfg_c : inst2veccdfg_c : ir2veccdfg : histogramcdfg : inst2veccdfg : ir2vec
cdfg+ : histogramcdfg+ : inst2veccdfg+ : ir2vec
cfg_c : histogramcfg_c : inst2veccfg_c : ir2veccfg : histogramcfg : inst2veccfg : ir2vechistogram
ir2veclbpeqlbpif
milepostprog2img
programl : histogramprograml : inst2vecprograml : ir2vec
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%Accuracy
Figure 8: Accuracy results for Loop-2.
The graph-based neural networks did not outperform the traditional neural
networks discussed in Section 3. In principle, we found this result surprising,
because we failed to teach a graph-based neural network how to discover cycles.
However, such limitation is to be expected. Graph neural networks recognize
some substructures in particular graphs (star-shaped, for instance). Neverthe-
less, given any speciﬁc GNN, there are graphs whose substructures that GNN
will fail to recognize, as demonstrated by Zhengdao et al. [33]. For instance,
GNNs studied by Zhengdao et al. [33] fail to recognize cycles with eight or more
nodes.
Discussion–Optimizations. Figures 10, 11, and 12 report the accuracy of the dif-
ferent classiﬁers when solving OptZ for programs taken from Poj, CodeNet,
and AnghaBench, respectively. The accuracy of the diﬀerent classiﬁers is
low, typically remaining between 20 and 40% when applied onto Poj and
AnghaBench programs. It improves slightly on CodeNet, but never goes
above 50%. The instance of OptZ built out of Poj contains less samples than
28
cdfg_c : histogramcdfg_c : inst2veccdfg_c : ir2veccdfg : histogramcdfg : inst2veccdfg : ir2vec
cdfg+ : histogramcdfg+ : inst2veccdfg+ : ir2vec
cfg_c : histogramcfg_c : inst2veccfg_c : ir2veccfg : histogramcfg : inst2veccfg : ir2vechistogram
ir2veclbpeqlbpif
milepostprog2img
programl : histogramprograml : inst2vecprograml : ir2vec
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%Accuracy
Figure 9: Accuracy results for Loop-4.
cdfg_c : histogramcdfg_c : inst2veccdfg_c : ir2veccdfg : histogramcdfg : inst2veccdfg : ir2vec
cdfg+ : histogramcdfg+ : inst2veccdfg+ : ir2vec
cfg_c : histogramcfg_c : inst2veccfg_c : ir2veccfg : histogramcfg : inst2veccfg : ir2vechistograminst2vecir2veclbpeqlbpif
milepostprog2img
programl : histogramprograml : inst2vecprograml : ir2vec
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%Accuracy
Figure 10: Accuracy results for OptZ-A.
the equivalent instance built from CodeNet: 365 versus 500 samples per best
sequence of optimizations. The reason for this diﬀerence will be made clear in
Section 5.3. Nonetheless, we do not expect this diﬀerence to bear impact on the
accuracies observed in Figures 11 and 12.
29
The accuracy observed in Figures 10, 11, and 12 is low; however, we expected
such an outcome. The automatic prediction of the best optimization sequence
for a program is the hardest problem—for a human classiﬁer—among the three
problems studied in this paper. A software developer can solve Loop easily,
cdfg_c : histogramcdfg_c : inst2veccdfg_c : ir2veccdfg : histogramcdfg : inst2veccdfg : ir2vec
cdfg+ : histogramcdfg+ : inst2veccdfg+ : ir2vec
cfg_c : histogramcfg_c : inst2veccfg_c : ir2veccfg : histogramcfg : inst2veccfg : ir2vechistogram
ir2veclbpeqlbpif
milepost
programl : histogramprograml : inst2vecprograml : ir2vec
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%Accuracy
Figure 11: Accuracy results for OptZ-C.
cdfg_c : histogramcdfg_c : inst2veccdfg_c : ir2veccdfg : histogramcdfg : inst2veccdfg : ir2vec
cdfg+ : histogramcdfg+ : inst2veccdfg+ : ir2vec
cfg_c : histogramcfg_c : inst2veccfg_c : ir2veccfg : histogramcfg : inst2veccfg : ir2vechistograminst2vecir2veclbpeqlbpif
milepostprog2img
programl : histogramprograml : inst2vecprograml : ir2vec
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%Accuracy
Figure 12: Accuracy results for OptZ-P.
30
as long as the goto command is absent from the target code. Although Prob
is harder than Loop for a human classiﬁer, a seasoned programmer can often
infer the problem that a program solves, considering a limited set of possibilities.
However, the best optimization sequence for a program is not obvious; at least,
not for the authors of the present study, who have good experience with the
implementation of optimizing compilers.
OptZ is diﬃcult because the diﬀerence between the results produced by the
ﬁve sequences that we analyze in this paper is small. The ﬁfth column of Table 5
(Total Size) shows that the diﬀerence, in the number of LLVM instructions, of
codes produced by the best and the worst optimization sequences is of 1.8%
only, considering 15,000 benchmarks. These small diﬀerences persist in our
datasets, as Example 5 illustrates. Therefore, even though the sequences seen
in Table 4 are very diﬀerent, considering the passes that form them, the margin
for code-size reduction in typical programs is small.
Example 5. This paper analyzes four instances of OptZ. Table 3 shows the
sizes of the datasets curated for each instance. The ﬁrst instance, OptZ-A, uses
2,500 programs: ﬁve groups of 500 programs for which each optimization se-
quence from Table 4 outperforms the other four. If we compile the 500 programs
for which Cap is the best, we obtain respectively 131,442; 131,219; 131,751;
131,127 and 128,483 LLVM instructions using, respectively, Bst, Lim, Sum,
Geo, and Cap. On average, Cap outperforms its counterparts by less than
3%. In many samples, the diﬀerence is of only one LLVM instruction. Fig-
ure 13 shows how this diﬀerence spreads across the 500 benchmarks that com-
prises this dataset, both in absolute and relative terms. Similar behavior can
also be observed in each one of the four other datasets used in OptZ-A.
5.3. Spurious Correlations
The results reported in Figures 11 and 12 have been obtained over abalanced
dataset. The Poj and CodeNet datasets are organized into programming prob-
lems. This organization might provoke spurious correlations, because programs
31
Absolute difference in number of instructions Relative difference in number of instructions
0
20
40
60
80
0
0.04
0.08
0.12
0.16
500 programs, sorted by absolute difference 500 programs, sorted by relative difference
c_ZSTD_CCtxParams_setParameter
209 instructions with CAP vs
241 with the second bestextr_mkhash.c_SHA256_Transform
1,146 instructions with CAP vs
1,223 with the second best
In 150 programs, Cap is 
the best by one 
instruction
The different between Cap and 
the second best sequence is 
less than 2% for 372 programs
0.10
0.02
0.06
0.14
Figure 13: Comparison between results produced with optimization sequence Cap and the
second best sequence for the dataset where Cap is the best sequence.
that solve the same problem tend to be similar. To avoid spurious correlations,
in Section 5.2 we ensured that each optimization sequence is equally represented
across programming problems. Thus, each problem used to build the dataset
contributed 5 ×N samples, where N is the number of samples that were better
optimized by each of the ﬁve sequences. This precaution constrained the num-
ber of samples from Poj to 365 programs per optimization sequence; hence,
yielding 1,825 programs. Figure 14 shows what happens once such discretion is
not observed.
Discussion. Figure 14 reports the accuracy of the diﬀerent classiﬁers when ap-
plied onto a variation of OptZ-P, henceforth called OptZ-P-sp. The dataset
is formed by programs taken from 15 diﬀerent classes of algorithms from Poj.
Every three of these classes contribute programs that are best optimized by the
same optimization sequence. Compared to Fig. 12, Fig. 14 reports high accu-
racy. However, in this case, the classiﬁers are learning how to match programs
with the problems they solve, in opposition to the sequence of optimizations
that best ﬁt them.
To give the reader an idea of the potential for spurious correlations in ex-
periments using POJ-like samples, Figure 15 shows the t-distributed stochastic
neighbor embedding (t-SNE) [34] produced for diﬀerent datasets used in this
32
cdfg_c : histogramcdfg_c : inst2veccdfg_c : ir2veccdfg : histogramcdfg : inst2veccdfg : ir2vec
cdfg+ : histogramcdfg+ : inst2veccdfg+ : ir2vec
cfg_c : histogramcfg_c : inst2veccfg_c : ir2veccfg : histogramcfg : inst2veccfg : ir2vechistograminst2vecir2veclbpeqlbpif
milepostprog2img
programl : histogramprograml : inst2vecprograml : ir2vec
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%Accuracy
Figure 14: Accuracy results for OptZ-P-sp
paper. We use the histogram of instructions to produce the visual embedding.
Each point in the ﬁgure represents a program. Programs that are best optimized
by the same optimization sequence are represented with the same marker. Fig-
ures 15 (b) and (c) show unbalanced data, e.g., where the same programming
problems in Poj or CodeNet contribute programs best optimized by the same
sequence. In this case, clusters are easy to distinguish visually. Figure 15 (d)
shows the balanced dataset that we have used in Section 5.2. In this case, there
is much overlap among clusters.
5.4. The Eﬀectiveness of Instruction Histograms
Table 6 reports the average accuracy (out of ten evaluations) of each clas-
siﬁer analyzed in this Section. We have highlighted the best results obtained
in this evaluation. Overall, the histogram of instruction opcodes led to the
best observed results. In our opinion, this result is surprising. Histograms
discard much program information: they do not account for execution ﬂow or
dependencies between program elements. The graph-based representations that
use histograms as embeddings store both ﬂow and dependence relations. Nev-
ertheless, they could outperform the histogram-based embedding only in one
33
OPTZ-C with spurious correlationsOPTZ-A
OPTZ-P with spurious correlations OPTZ-P with ballanced dataset
(a) (b)
(c) (d)
Best sequences:   BST: ̗,     LIM: ̙,     SUM: ◁,     GEO: 
➕
,     CAP: ̋ ,
The optimizations sequences were taken from Faustino et al, 2021
Figure 15: Two-dimensional t-SNE plots for histogram-based embeddings.
problem: Prob-P.
In spite of the dominance of histogram-based embeddings, we emphasize
that the diﬀerence between the various classiﬁers is small. None of the classiﬁers
built with local-binary patterns ( LBPif and LBPeq) presented good results. The
other classiﬁers were highly accurate in Prob-like problems, and presented low
accuracy in the remaining problems. We speculate that the poor performance of
LBPif and LBPeq is due to excessive summarization, becauseProg2Image yields
much more accurate classiﬁcations. Prog2Image preserves all the information
that is summarized in these two local binary patterns.
34
Learning Model Prob
-P
Prob
-C
Loop
-2
Loop
-4
OptZ
-A
OptZ
-C
OptZ
-P
OptZ
-P-sp
cfg:Histogram 95.7 90.4 81.4 32.9 31.5 52.2 19.7 83.0
cfg:Inst2Vec 96.1 88.1 77.0 31.2 30.3 55.8 21.1 77.3
cfg:Ir2Vec 92.9 85.7 60.2 29.8 26.6 20.0 20.0 87.3
cfg c:Histogram 93.0 95.4 72.7 38.3 35.1 40.1 22.9 91.2
cfg c:Inst2Vec 95.7 90.2 65.2 35.1 30.8 41.9 19.3 93.7
cfg c:Ir2Vec 93.5 93.0 61.9 36.8 31.3 55.1 18.8 93.8
cdfg:Histogram 89.0 87.7 66.9 30.7 32.1 51.4 19.7 91.7
cdfg:Inst2Vec 96.7 92.7 55.0 32.9 25.3 37.7 22.3 92.2
cdfg:Ir2Vec 95.5 91.6 55.9 29.5 25.9 45.8 20.0 91.5
cdfg c:Histogram 93.6 95.8 64.7 43.4 36.1 52.4 19.6 87.0
cdfg c:Inst2Vec 91.4 93.4 71.7 38.8 31.8 52.2 21.1 91.2
cdfg c:Ir2Vec 90.5 92.4 62.5 35.6 32.7 54.8 17.5 93.3
cdfg+:Histogram 51.4 88.6 63.1 33.9 27.7 54.4 19.7 72.3
cdfg+:Inst2Vec 69.4 93.3 64.3 28.3 26.7 41.7 22.6 92.2
cdfg+:Ir2Vec 92.0 83.7 59.8 28.3 20.0 56.4 20.0 91.2
ProGraML:Histogram 89.2 89.9 71.6 31.7 25.0 57.7 20.5 75.8
ProGraML:Inst2Vec 84.0 91.2 79.9 28.8 29.2 42.3 20.3 80.0
ProGraML:Ir2Vec 86.0 83.0 60.2 31.7 20.0 57.3 21.0 75.7
Ir2Vec 91.7 95.8 63.4 34.6 31.4 45.9 23.8 92.0
Inst2Vec 94.8 94.8 - - 31.4 - 20.3 91.0
Milepost 90.6 93.5 57.3 37.6 33.2 44.5 23.3 89.7
Histogram 91.4 96.0 65.3 37.6 37.8 48.2 26.3 92.0
LBPif 42.3 78.4 49.0 29.0 21.6 31.3 20.3 30.5
LBPeq 52.3 89.9 50.3 30.2 23.3 29.9 19.5 50.5
Prog2Image 85.1 93.4 52.4 30.7 35.2 - 22.3 65.0
Figure 6 7 8 9 10 11 12 14
Table 6: Summary of results analyzed in this section. Dark-gray boxes show the best results;
light-gray, the second best. Dashes indicate experiments whose ten samples could not ﬁnish
within the allotted time limit of six hours.
5.5. Consistency Across Diﬀerent Learning Models
Since the histogram embedding presented good results for most of the experi-
ments, we have used it to evaluate the eﬃcacy of other machine learning models.
To this end, we have used Scikit-learn [35] to train classiﬁers based on the
following models: Logistic Regression (LR), Decision Trees (DT), K-Nearest
Neighbor (KNN), and Support Vector Machines (SVM). For the LR, KNN, and
SVM models we performed a grid-based hyperparameter search using the train-
ing and the validation datasets. For the LR model, we explored the following
35
hyperparameters: penalty (l1, elasticnet, l2, and none), solver (lbfgs), c (5/64,
10/64, 20/64, 50/64, 70/64, and 100/64), and tol (0.1, 0.5, and 1.0). For KNN:
n neighbours (5, 10, 15, 20, 50) and weights (uniform and distance). Finally, for
SVM: kernel (poly and rbf), C (0.1, 10.0, 100.0, 1000.0, and 10000.0), degree
(3), and gamma (scale).
Acc. with Histogram embedding Best
LR DT KNN SVM GNN
Prob-P 0.83 0.96 0.96 0.97 0.96 Fig. 6
Prob-C 0.41 0.90 0.91 0.93 0.95 Fig. 7
Loop-2 0.53 0.67 0.58 0.64 0.81 Fig. 8
Loop-4 0.38 0.45 0.32 0.37 0.43 Fig. 9
OptZ-A 0.29 0.32 0.39 0.43 0.36 Fig. 10
OptZ-C 0.34 0.54 0.53 0.55 0.57 Fig. 11
OptZ-P 0.27 0.21 0.24 0.24 0.22 Fig. 12
OptZ-P-sp 0.62 0.89 0.91 0.95 0.93 Fig. 14
Search space 72 1 10 10
Table 7: Accuracy of typical ML models using the histogram embedding. Column Best GNN
shows the best result seen in Table 6. Gray boxes highlight best results overall. The last row
shows the number of models considered when searching for good hyperparameters for each
ML model.
Discussion. Table 7 reports the accuracy obtained with the test dataset via
diﬀerent classiﬁcation models. This ﬁgure shows that the results reported in
this section remain consistent, regardless of the learning architecture used. In
other words, even more traditional ML models, such as SVM, achieve high
accuracy in Prob-P, Prob-C, and OptZ-P-sp. Yet, all these models perform
poorly on the other problems, except in the case where the dataset has not
been treated against spurious correlations ( OptZ-P-sp). In several cases, the
traditional ML models could outperform the graph neural networks used in the
rest of this paper. However, these results must be seen with caution: every
model in Table 7 results from a search in a small universe of hyperparameters.
Therefore, each one of these models is the best architecture out of a relatively
large number of possible variations, which Table 7 indicates in “Number of
models”.
36
6. Related Work
Research in predictive compilation started in earnest in the late nineties and
early two-thousands [9, 11, 26, 27]. As examples of early encouraging results,
Calder et al. [9] estimated with high accuracy the outcome of a branch without
running the program and Cavazos et al. [27] were able to predict the best register
allocator for a given program. Nevertheless, these pioneers faced challenges that,
at the time, would be diﬃcult to surmount; namely, lack of benchmarks and the
impossibility to use GPUs to improve training speed. Both these diﬃculties have
been solved in the last decade—a fact that fostered much progress in predictive
compilation.
6.1. On the Availability of Benchmarks
Much research has been done to equip compiler researchers with benchmarks.
An important step along this direction was the release ofPoj, by Mou et al. [16].
At about the same time, Cummins et al. [3] showed how to generate realistic
programs out of Markov models. More recently, AnghaBench [6] showed how
to use a type inference engine for C [36, 37] to synthesize compilable programs
from open-source repositories. To improve even more the availability of bench-
marks, Puri et al. [18] released CodeNet. Like Poj, CodeNet is built from
solutions to programming problems; however, CodeNet is orders of magnitude
larger.
6.2. The Embedding Zoo
The use of hardware accelerators in machine learning [38] was of vital im-
portance to the advent of new predictive compilation techniques. Faster models
shortened the development cycle of predictive techniques, and made it possible
to train classiﬁers with increasingly complex embeddings. The consequence was
a ﬂood of new program representations in the last four years [1, 2, 13, 14, 15].
One of the ﬁrst attempts to study the impact of the embedding function into the
quality of a predictive compiler was due to Namolaru et al. [23]. The Milepost
embedding, which we have evaluated in this paper, came from that work. The
37
other representations evaluated in this paper [2, 13, 17, 20, 21] are more recent.
They diﬀer from Namolaru et al. [23]’s Milepost embedding in the sense that
they are built via an accumulation function, whereas Namolaru et al. [23] build
vectors formed by independent features. Namolaru et al.’s vectors bind each
dimension to a program feature that can be extracted from the code indepen-
dently from the others. Examples of features include number of call instruc-
tions, number of edges in the CFG, number of basic blocks, etc. In addition to
Milepost, the histograms and the Prog2Image representation of Table 1 are
based on independent vectors of features. Independent feature vectors can also
be associated with program parts, like branches [9] or stores [39].
The other representations in Table 1 are built via an accumulation function.
Ben-Nun et al. [2]’s Inst2Vec maps ﬁxed-size groups of program statements
onto vectors of 200 dimensions, using well-known techniques to embedded words
into a vector space (Skip-Gram [40]). VenkataKeerthy et al. [13], in turn, deﬁne
a function that maps opcodes, arguments, and types of LLVM instructions onto
a 300-dimensional vector. They use simple addition to accumulate all these
vectors into a single, ﬁnal, embedding of the program.
6.3. Classiﬁcation Problems
Although predictive compilation is a popular topic in the programming lan-
guage community today, most evaluations still concern a small number of clas-
siﬁcation problems, three of which stand out:
• Programming-problem classiﬁcation [16]: determine which problem a pro-
gram solves [2, 20, 21].
• Device Mapping [41]: given an OpenCL program, predict if it runs faster
on a CPU or on a GPU [2, 13, 17, 21].
• Thread Coarsening [42]: given an OpenCL program, determine if the pro-
gram will run faster when mapped onto 2, 4, 8, 16, or 32 threads [2, 13, 17].
These three problems have been reused without modiﬁcation since their debut.
Their popularity is due to the availability of datasets and classiﬁers—a testimony
38
to the importance of public research artifacts. Recently, Brauckmann et al. [17]
have shown that typical learning architectures tend to solve device mapping
very well, although they stumble on thread coarsening. It has already been
observed that algorithmic mapping, at least if restricted to datasets extracted
from programming competitions, is easy to solve [18]. In this paper, we have
introduced two new problems, plus their datasets, and we have shown that
state-of-the-art classiﬁcation models cannot handle them satisfactorily yet.
This Paper in Perspective. As already mentioned in Section 1, this paper reuses
the ﬁve optimization sequences for code-size reduction that we have discovered
in 2021 [19]. Those sequences reﬁne a previous set of optimization sequences
that we have proposed one year earlier [6]. That ﬁrst collection of optimization
sequences formed what Purini et al.called a Default Covering Set [28]. Given a
certain optimization levelL of a compiler, e.g., clang -Oz, a default covering set
for L is an ensemble of optimization sequences S = {L1, L2, . . . , Ln}, such that,
for any program P, there exists, with high probability, at least oneLi, 1 ≤i ≤n,
that outperforms L on P. The ﬁve sequences of Section 4.1 do not form a
default covering set; however, they tend to outperform the sequences proposed
by Faustino et al. [6] for most programs in the LLVM test suite. This interest of
ours for optimization sequences that reduce the size of programs came forth with
the release of AnghaBench, a collection of more than one million programs
aimed at tuning compilers for code-size reduction [43].
7. Conclusion
This paper has evaluated 25 program classiﬁers onto three problems; two
of them new: matching programs with sequences of optimizations, and detect-
ing loops in programs via stochastic inference. The classiﬁers have been built
after 13 diﬀerent program representations: three of them new. Following re-
sults already observed in previous work, the classiﬁers were very eﬀective to
map programs to the problem they solve. However, this accuracy could not be
projected onto the problems that this paper introduced. Surprisingly, among
39
all the 25 classiﬁers, the one using histograms of instructions as the embedding
function was marginally better. Another observation that might be of general
interest is that caution must be exercised when using POJ-like datasets to train
predictive optimizers: programs that implement solutions to the same problem
tend to be very similar, even if written by diﬀerent coders. Thus, when using
these datasets, it is important to ensure that the classes of interest are equally
distributed across problems.
Extending this Work. The datasets and the infrastructure used in the experi-
ments carried out in this paper are publicly available, and we hope that they
can foster more research in the ﬁeld of predictive compilation. Adding new rep-
resentations to the pool of experiments already performed in this paper does not
require redoing said experiments. If a new program representation is proposed,
then this new representation will have to be evaluated on the diﬀerent datasets
in Table 3. Said evaluation will add a new column to Table 6; however, the
columns already there remain unchanged.
Software. All the material necessary to reproduce the experiments discussed in
this paper is available at https://github.com/otavioon/COLA-2022-Tools.
Acknowledgments
The authors would like to thank CNPq (grants 314645/2020-9 and 406377/2018-
9), FAPEMIG (grant PPM-00333-18), FAPESP (grant 2013/08293-7) and CAPES
(Edital PrInt and grant 88887.668980/2022-00). The authors would also like
to thank the Discovery Lab (https://discovery.ic.unicamp.br/) for its in-
frastructure and technical support.
References
[1] U. Alon, M. Zilberstein, O. Levy, E. Yahav, Code2Vec: Learning dis-
tributed representations of code, Proc. ACM Program. Lang. 3 (POPL)
(2019) 40:1–40:29. doi:10.1145/3290353.
40
[2] T. Ben-Nun, A. S. Jakobovits, T. Hoeﬂer, Neural code comprehension: A
learnable representation of code semantics, in: NIPS, Curran Associates
Inc., Red Hook, NY, USA, 2018, p. 3589–3601. doi:10.5555/3327144.
3327276.
[3] C. Cummins, P. Petoumenos, Z. Wang, H. Leather, Synthesizing bench-
marks for predictive modeling, in: CGO, IEEE, Piscataway, NJ, USA,
2017, pp. 86–99. doi:10.1109/CGO.2017.7863731.
[4] C. Cummins, P. Petoumenos, A. Murray, H. Leather, Compiler fuzzing
through deep learning, in: ISSTA, ACM, New York, NY, USA, 2018, pp.
95–105. doi:10.1145/3213846.3213848.
[5] V. M. do Rosario, A. F. da Silva, T. A. S. Camacho, O. O. Napoli, M. Bre-
ternitz, E. Borin, Smart selection of optimizations in dynamic compilers,
Concurr. Comput. Pract. Exp. 33 (18) (2021) 12. doi:10.1002/cpe.6089.
[6] A. F. da Silva, B. C. Kind, J. W. de Souza Magalh˜ aes, J. N. Rocha, B. C. F.
Guimar˜ aes, F. M. Q. Pereira, AnghaBench: A suite with one million compi-
lable C benchmarks for code-size reduction, in: CGO, IEEE, Los Alamitos,
CA, USA, 2021, pp. 378–390. doi:10.1109/CGO51591.2021.9370322.
[7] A. F. d. Silva, B. N. B. de Lima, F. M. Q. Pereira, Exploring the space
of optimization sequences for code-size reduction: Insights and tools, in:
Compiler Construction, Association for Computing Machinery, New York,
NY, USA, 2021, p. 47–58. doi:10.1145/3446804.3446849.
[8] A. Moreira, G. Ottoni, F. M. Q. Pereira, Vespa: Static proﬁling for binary
optimization, PACMPL 1 (OOPSLA) (2021) 144:1–144:28. doi:10.1145/
3485521.
[9] B. Calder, D. Grunwald, M. Jones, D. Lindsay, J. Martin, M. Mozer,
B. Zorn, Evidence-based static branch prediction using machine learn-
ing, ACM Trans. Program. Lang. Syst. 19 (1) (1997) 188–222. doi:
10.1145/239912.239923.
41
[10] A. Monsifrot, F. Bodin, R. Quiniou, A machine learning approach to au-
tomatic production of compiler heuristics, in: AIMSA, Springer-Verlag,
Berlin, Heidelberg, 2002, p. 41–50. doi:10.5555/646053.677574.
[11] E. Moss, P. Utgoﬀ, J. Cavazos, D. Precup, D. Stefanovic, C. Brodley,
D. Scheeﬀ, Learning to schedule straight-line code, in: NIPS, MIT Press,
Cambridge, MA, USA, 1997, p. 929–935. doi:10.5555/3008904.3009034.
[12] A. McGovern, E. Moss, Scheduling straight-line code using reinforcement
learning and rollouts, in: NIPS, MIT Press, Cambridge, MA, USA, 1999,
p. 903–909. doi:10.5555/340534.340836.
[13] S. VenkataKeerthy, R. Aggarwal, S. Jain, M. S. Desarkar, R. Upadrasta,
Y. N. Srikant, IR2VEC: LLVM IR based scalable program embeddings,
ACM Trans. Archit. Code Optim. 17 (4). doi:10.1145/3418463.
[14] S. H. H. Ding, B. C. M. Fung, P. Charland, Asm2Vec: Boosting static
representation robustness for binary clone search against code obfuscation
and compiler optimization, in: SP, IEEE, Washington, DC, USA, 2019, pp.
472–489. doi:10.1109/SP.2019.00003.
[15] C. Cummins, B. Wasti, J. Guo, B. Cui, J. Ansel, S. Gomez, S. Jain, J. Liu,
O. Teytaud, B. Steiner, Y. Tian, H. Leather, CompilerGym: Robust,
performant compiler optimization environments for AI research, CoRR
abs/2109.08267. arXiv:2109.08267.
[16] L. Mou, G. Li, L. Zhang, T. Wang, Z. Jin, Convolutional neural networks
over tree structures for programming language processing, in: AAAI, AAAI
Press, Phoenix, Arizona, 2016, p. 1287–1293. doi:10.5555/3015812.
3016002.
[17] A. Brauckmann, A. Goens, S. Ertel, J. Castrillon, Compiler-based graph
representations for deep learning models of code, in: CC, Association for
Computing Machinery, New York, NY, USA, 2020, p. 201–211. doi:10.
1145/3377555.3377894.
42
[18] R. Puri, D. S. Kung, G. Janssen, W. Zhang, G. Domeniconi, V. Zolotov,
J. Dolby, J. Chen, M. Choudhury, L. Decker, V. Thost, L. Buratti, S. Pujar,
S. Ramji, U. Finkler, S. Malaika, F. Reiss, Project CodeNet (2021). doi:
10.5281/zenodo.4814769.
[19] A. Faustino, E. Borin, F. M. Quint˜ ao Pereira, O. N´ apoli, V. Ros´ ario, New
optimization sequences for code-size reduction for the LLVM compilation
infrastructure, in: SBLP, ACM, New York, NY, USA, 2021, p. 33–40.
doi:10.1145/3475061.3475085.
[20] A. Faustino, Graphs based on IR as representation of code: Types and
insights, in: SBLP, Association for Computing Machinery, New York, NY,
USA, 2021, p. 1–8. doi:10.1145/3427081.3427089.
[21] C. Cummins, Z. V. Fisches, T. Ben-Nun, T. Hoeﬂer, M. F. P. O’Boyle,
H. Leather, ProGraML: A graph-based program representation for data
ﬂow analysis and compiler optimizations, in: ICML, Vol. 139, PMLR, Bal-
timore, Maryland, USA, 2021, pp. 2244–2253.
[22] T. Ojala, M. Pietik¨ ainen, D. Harwood, Performance evaluation of tex-
ture measures with classiﬁcation based on kullback discrimination of dis-
tributions, in: ICPR, IEEE, Washington, DC, USA, 1994, pp. 582–585.
doi:10.1109/ICPR.1994.576366.
[23] M. Namolaru, A. Cohen, G. Fursin, A. Zaks, A. Freund, Practical aggre-
gation of semantical program properties for machine learning based op-
timization, in: CASES, ACM, New York, NY, USA, 2010, pp. 197–206.
doi:10.1145/1878921.1878951.
[24] A. F. Zanella, A. F. da Silva, F. M. Quint˜ ao, YaCoS: A complete infras-
tructure to the design and exploration of code optimization sequences, in:
SBLP, Association for Computing Machinery, New York, NY, USA, 2020,
p. 56–63. doi:10.1145/3427081.3427089.
43
[25] M. Zhang, Z. Cui, M. Neumann, Y. Chen, An end-to-end deep learning
architecture for graph classiﬁcation, in: Proceedings of the AAAI Con-
ference on Artiﬁcial Intelligence, AAAI ’32, Association for the Advance-
ment of Artiﬁcial Intelligence, New York, NY, USA, 2018, p. 4423–4445.
doi:10.5555/3504035.3504579.
[26] F. Agakov, E. Bonilla, J. Cavazos, B. Franke, G. Fursin, M. F. P. O’Boyle,
J. Thomson, M. Toussaint, C. K. I. Williams, Using machine learning to fo-
cus iterative optimization, in: CGO, IEEE Computer Society, Washington,
DC, USA, 2006, pp. 295–305. doi:10.1109/CGO.2006.37.
[27] J. Cavazos, M. F. P. O’Boyle, Method-speciﬁc dynamic compilation using
logistic regression, in: OOPSLA, Association for Computing Machinery,
New York, NY, USA, 2006, p. 229–240. doi:10.1145/1167473.1167492.
[28] S. Purini, L. Jain, Finding good optimization sequences covering program
space, ACM Trans. Archit. Code Optim. 9 (4). doi:10.1145/2400682.
2400715.
[29] K. Claessen, J. Hughes, Quickcheck: A lightweight tool for random testing
of haskell programs, in: ICFP, Association for Computing Machinery, New
York, NY, USA, 2000, p. 268–279. doi:10.1145/351240.351266.
URL https://doi.org/10.1145/351240.351266
[30] A. Zeller, R. Hildebrandt, Simplifying and isolating failure-inducing input,
IEEE Trans. Softw. Eng. 28 (2) (2002) 183–200. doi:10.1109/32.988498.
URL https://doi.org/10.1109/32.988498
[31] S. V. der Walt, J. L. Schonberger, J. Nunez-Iglesias, F. Boulogne, J. D.
Warner, N. Yager, E. Gouillart, T. Yu, SciKit-image: Image processing in
python, PeerJ 453. doi:10.7717/peerj.453.
[32] J. Zhang, X. Wang, H. Zhang, H. Sun, K. Wang, X. Liu, A novel neural
source code representation based on abstract syntax tree, in: ICSE, IEEE
44
Press, New York, NY, US, 2019, p. 783–794. doi:10.1109/ICSE.2019.
00086.
[33] Z. Chen, L. Chen, S. Villar, J. Bruna, Can graph neural networks count
substructures?, CoRR abs/2002.04025. arXiv:2002.04025.
[34] T. Khot, Visualizing high-dimensional data, XRDS 23 (2) (2016) 66–67.
doi:10.1145/3021604.
[35] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel,
M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Pas-
sos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn:
Machine learning in Python, Journal of Machine Learning Research 12
(2011) 2825–2830. doi:10.5555/1953048.2078195.
[36] L. T. C. Melo, R. G. Ribeiro, M. R. de Ara´ ujo, F. M. Q. Pereira, Inference
of static semantics for incomplete C programs, Proc. ACM Program. Lang.
2 (POPL) (2018) 29:1–29:28. doi:10.1145/3158117.
[37] L. T. C. Melo, R. G. Ribeiro, B. C. F. Guimar˜ aes, F. M. Q. Pereira, Type
inference for C: Applications to the static analysis of incomplete programs,
ACM Trans. Program. Lang. Syst. 42 (3). doi:10.1145/3421472.
[38] S. R. Upadhyaya, Parallel approaches to machine learning-a comprehensive
survey, J. Parallel Distrib. Comput. 73 (3) (2013) 284–292. doi:10.1016/
j.jpdc.2012.11.001.
[39] F. M. Q. Pereira, G. V. Leobas, A. Gamati´ e, Static prediction of silent
stores, ACM Trans. Archit. Code Optim. 15 (4) (2018) 44:1–44:26. doi:
10.1145/3280848.
[40] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, J. Dean, Distributed rep-
resentations of words and phrases and their compositionality, in: NIPS,
NIPS’13, Curran Associates Inc., Red Hook, NY, USA, 2013, p. 3111–3119.
doi:10.5555/2999792.2999959.
45
[41] M. F. P. O’Boyle, Z. Wang, D. Grewe, Portable mapping of data parallel
programs to opencl for heterogeneous systems, in: CGO, CGO ’13, IEEE
Computer Society, USA, 2013, p. 1–10. doi:10.1109/CGO.2013.6494993.
[42] A. Magni, C. Dubach, M. O’Boyle, Automatic optimization of thread-
coarsening for graphics processors, in: PACT, Association for Comput-
ing Machinery, New York, NY, USA, 2014, p. 455–466. doi:10.1145/
2628071.2628087.
[43] A. F. da Silva, B. Kind, J. W. M. aes, J. Rocha, B. G. aes, F. M. Q. Pereira,
AnghaBench: a synthetic collection of benchmarks mined from open-source
repositories, Tech. Rep. 01-2020, Universidade Federal de Minas Gerais
(2020).
46

BaCO: A Fast and Portable Bayesian Compiler Optimization
Framework
Erik Hellsten1 Artur Souza2 Johannes Lenfers3
Lund University Federal University of Minas Gerais University of Münster
Rubens Lacouture4 Olivia Hsu5 Adel Ejjeh6
Stanford University Stanford University University of Illinois UC
Fredrik Kjolstad7 Michel Steuwer8 Kunle Olukotun9
Stanford University University of Edinburgh Stanford University
Luigi Nardi10
Lund University, Stanford University
Abstract
We introduce the Bayesian Compiler Optimization frame-
work (BaCO), a general purpose autotuner for modern
compilers targeting CPUs, GPUs, and FPGAs. BaCO
provides the ﬂexibility needed to handle the requirements
of modern autotuning tasks. Particularly, it deals with per-
mutation, ordered, and continuous parameter types along
with both known and unknown parameter constraints.
To reason about these parameter types and efﬁciently
deliver high-quality code, BaCO uses Bayesian optimiza-
tion algorithms specialized towards the autotuning do-
main. We demonstrate BaCO’s effectiveness on three
modern compiler systems: TACO, RISE & ELEVATE, and
HPVM2FPGA for CPUs, GPUs, and FPGAs respectively.
For these domains, BaCO outperforms current state-of-
the-art autotuners by delivering on average 1.36×–1.56×
faster code with a tiny search budget, and BaCO is able
to reach expert-level performance 2.9×–3.9×faster.
1. Introduction
Modern compilers are rapidly evolving to keep pace with
the growing range of increasingly specialized hardware
targets, as well as the ever-changing domains of inter-
est. A recent trend is to separate policy (what to compute)
from mechanism (transformations and code generation de-
scribing how to compute) by using scheduling languages.
Prominent examples of this paradigm include Halide [44],
TVM [7], TACO [49], and RISE & ELEV ATE [19, 55].
This design pushes the optimization task of ﬁnding good
1,10{erik.hellsten, luigi.nardi}@cs.lth.se, 2arturluis94@gmail.com,
3j.le@wwu.de, 4,5,7,9{rubensl, owhsu, kjolstad, kunle}@stanford.edu,
6aejjeh@illinois.edu, 8michel.steuwer@ed.ac.uk
schedules outside of the compiler core, where it can
be done manually or automatically by an autoscheduler.
Scheduling languages may express more complex opti-
mization spaces, and, thus, require more advanced au-
totuning features to effectively and efﬁciently tackle the
autoscheduling task. Modern hardware backends—like
GPUs, as in RISE & ELEV ATE [55], and FPGAs, as in
HPVM2FPGA [33, 13]—further increase the complexity
of relevant optimization spaces.
The separation of concerns between policy and mecha-
nism in compilers exposes a great opportunity. If we can
design a portable autoscheduler that is effective across
many compilers, like the design shown in Fig. 1, then we
can reduce the complexity of the overall ecosystem. New
compilers get an autoscheduler with minimal effort, and
improvements in the autoscheduler automatically beneﬁt
all compilers and their subsequent domains and backends.
However, a portable autoscheduler must be designed
with a rich input language to allow users to accurately
describe the search space exposed by their particular com-
piler. This autoscheduling search space is determined by
the product of the hardware target, the compiler’s schedul-
ing language features and conﬁguration tuning parameters.
In modern compilers, this search space is often complex,
including both continuous parameters (e.g., real-valued
tuning parameters), and discontinuous parameters bro-
ken down into integers (e.g., tiling factors), permutation
categories (e.g., loop reordering), ordinals/ordered cate-
gories (e.g., unroll factors), and categoricals/unordered
categories (parallelization schemes). These parameter
types are often abbreviated to RIPOC [41, 2, 22].
However, even the large class of search spaces that
can be generated as the Cartesian combination of these
parameters is often inadequate. Scheduling parameters
frequently depend on the settings of other parameters,
leading to constraints on the scheduling space. One such
arXiv:2212.11142v2  [cs.PL]  11 Apr 2023
TACO RISE/ELEVATE HPVM2FPGA
FPGA
Programs
Image 
Processing
Sparse
Tensor Algebra
schedules
code generation
transformations …
Autoscheduler
Figure 1: An autoscheduler that is portable across the
scheduling languages of diverse compilers.
example is a loop bound that must be an exact multiple
of a given tiling factor. We refer to these as known con-
straints, which are provided to the autoscheduler ahead
of time. Other constraints are initially unknown and must
be learned throughout the autoscheduling. An example
of this is learning sets of parameters that would gener-
ate programs that adhere to hardware constraints, such
as avoiding out-of-memory errors on a GPU. Such con-
straints are often referred to as hidden constraints. For a
general autotuning framework to be efﬁcient and portable
across a multitude of compilers, it needs to support as
many of these features as possible.
Once able to express a complex search space, the au-
toscheduler must be effective and efﬁcient in ﬁnding a
good schedule within this space. For optimizing compil-
ers, the performance of the generated code is of primary
concern. Therefore, the autoscheduler will invariably use
some sort of search combined with a cost model to evalu-
ate points of the search space. The cost model could be
analytic or data-driven, but the most accurate cost model
is to generate and run the code on its target platform. For
a general autoscheduler used across diverse compilers,
a cost model based on running the code makes it easy
to use the autoscheduler with a new compiler. We refer
to empirical autoschedulers, whose cost model is to run
the actual code generated by the compiler, as autotuners.
To achieve composability, and to work effectively and
easily across a diverse set of compilers, it is vital for an
autoscheduler to treat each compiler as a black-box sys-
tem. The autoscheduler’s job will then be to optimize the
black-box system using the smallest possible budget of
trials and errors, i.e., evaluations of the black-box system.
Many successful autotuning frameworks have been
proposed, some of which are listed in Table 1. These
frameworks have helped deliver high-performance soft-
ware in the past. However, they do not support all fea-
tures required to effectively search over the complex
search spaces described by diverse scheduling languages
across modern domain-speciﬁc compilers targeting var-
ious hardware backends. For example, Table 2 shows
the features required by three modern compilers (TACO,
RISE & ELEV ATE, and HPVM2FPGA), and Table 1
shows that none of 14 recently proposed autotuners sup-
port all required features in the manner we deﬁne below.
Parameters Constraints
RIOC Perm. Hidden Known
ATF [46]  ××× ××× 
OpenTuner [2]   ××× ×××
Ytopt [63]  ××× ××× 
Kernel Tuner [61]  ××× ××× 
KTT [43] ××× ××× ××× 
GPTune [38]  ××× ××× 
HyperMapper [41]  ×××  ×××
Bliss [48] ××× ××× ××× ×××
DeepHyper [11]  ××× ××× *
SMAC3 [36]  ××× ××× *
GpyOpt [3] ××× ××× ××× 
Spearmint [52]  ×××  ×××
GPﬂowOpt [28] ××× ×××  ×××
cBO [17] ××× ×××  ×××
BaCO (ours)    
Table 1: Framework capabilities; RIOC abbreviates Re-
al/Integer/Ordinal/Categorical parameters. Limited sup-
port for constraints is marked with *.
Parameters Constraints
RIOC Perm. Hidden Known
TACO [26]    
RISE & ELEV ATE [19]   
HPVM2FPGA [13]  
Table 2: Features needed by different compilers; RIOC
abbreviates Real/Integer/Ordinal/Categorical parameters.
Some of the frameworks in Table 1 do support some types
of user-deﬁned constraints. Several frameworks are using
ConﬁgSpace [35], which supports conjunctions of linear
constraints. While this is certainly useful, it is inadequate
for deﬁning non-linear dependencies (see frameworks
marked with *). The hidden constraints column in Ta-
ble 1 means that the framework uses specialized tools
to handle the constraint validity instead of assigning a
high objective value to infeasible conﬁgurations, such as
OpenTuner. Handling hidden constraints with a special-
ized tool distinguishes slow but valid conﬁgurations from
invalid conﬁgurations, providing more information to the
search mechanism. Lastly, while permutations can be cast
as categoricals, BaCO is the ﬁrst autotuning framework
to make use of their underlying structure.
Hence, we propose BaCO, a novel general autotuning
framework optimized towards the autotuning of modern
compilers, which efﬁciently handles all features men-
tioned above. BaCO does not require any user-provided
cost model but instead learns from observations from
running the generated code throughout the optimization
procedure. The support for sophisticated search spaces
and online learning means that BaCO ﬁnds good sched-
ules in fewer iterations than existing autotuners, while
being easy to use. Notably, we do not adapt BaCO to indi-
vidual compilers nor applications but show that it yields
2
expert-level performance out of the box for a wide range
of applications. Our contributions are:
• The ﬁrst Bayesian autotuning framework that sup-
ports all RIPOC features, including permutation types,
through the deﬁnition of separate distance metrics, thus
improving search performance (Sec. 3 and Sec. 4).
• The integration of a feasibility model for hidden con-
straints, simplifying the portability to new compiler
backends (Sec. 4).
• The ﬁrst system to use the chain-of-tree technique in a
Bayesian optimization setting for portable autotuning
on sparse search spaces (Sec. 4).
• Applying autotuning to three distinct compiler frame-
works for different domains and hardware targets, along
with a survey of the autotuning challenges of those three
recent domain-speciﬁc compilers (Sec. 2).
The end result is thatBaCO generates expert-level code
signiﬁcantly faster than the state of the art. We demon-
strate the effectiveness and robustness of BaCO across
three real-world compilers and code generation systems
targeting CPUs, GPUs, and FPGAs (Sec. 5). BaCO re-
liably produces good schedules across different compil-
ers without any customization for each compiler, such
as hyperparameter tuning or custom constraint ﬁltering.
It achieves expert performance on TACO 3.15 ×–5.0×
faster than the state-of-the-art, while RISE & ELEV ATE
achieves 1.35×–1.58×better performance with a tiny
autotuning budget, and HPVM2FPGA achieves peak per-
formance 2.43×–2.77×faster.
2. Complexity of Modern Autotuning
To develop the next generation general-purpose autotun-
ing framework, we need to better understand the real-life
challenges faced by various modern compiler frameworks.
Therefore, we investigate the autotuning features needed
by the TACO, RISE & ELEV ATE, and HPVM2FPGA
compiler frameworks. This will allow us to identify an
autotuning framework that is able to generalize across
a wide spectrum of compilers and backend targets. As
we shall see, this ideal general-purpose autotuning frame-
work needs to support the features described in Table 2,
which includes support for a wide range of parameter
types and both hidden and known constraints.
The Tensor Algebra Compiler (TACO). TACO [26]
is the state-of-the-art compiler for sparse tensor algebra. It
generates high-performance code for tensor operations ex-
pressed in a high-level Einstein notation, such as the sam-
pled dense-dense matrix multiplication (SDDMM) com-
putation represented as A(i, j) =B(i, j)∗C(i,k)∗D(k, j).
A particular strength of TACO is its capability to generate
code for a large variety of sparse tensor formats [8].
TACO’s scheduling language deﬁnes an iteration space
transformation framework that dictates how to traverse a
tensor stored in any particular format [49]. This provides
a way to introduce optimization transformations, such as
tiling, parallelization, vectorization, loop reordering, and
more. An autotuning framework selecting the optimiza-
tions exposed by the scheduling language needs to provide
not just traditional real, integer, ordinals, and categorical
parameters as provided by most frameworks in Table 1,
but also permutation parameters for selecting loop re-
ordering. Typically, optimization is performed inside the
compiler and controlled by a heuristic, but in TACO, as
well as other similar compilers with scheduling languages,
it is exposed as a tunable parameter. These optimization
parameters also need to follow known constraints that
TACO provides. An example is loop reordering variables
that TACO enforces for concordant traversal.
RISE & ELEV ATE. RISE [55] and ELEV ATE [19]
are a powerful combination of compiler and scheduling
languages. Computations are described in the RISE [55]
language using well-known data-parallel patterns like
map and reduce in the spirit of LIFT [20, 56]. Optimiza-
tions are applied and described in the ELEV ATE [ 19]
scheduling language as compositions of semantic pre-
serving rewrite-rules. The optimized RISE program is
compiled to high-performance CPU or GPU code.
Transformations, such as loop tiling, may introduce nu-
merical tuning parameters, such as a tile size, which are
often constrained by other numerical values, such as loop
bounds. When automatically optimizing RISE programs,
an explorative rewrite process speculatively applying pro-
gram transformations is performed. To evaluate the per-
formance of a transformed program, the system relies on
an autotuning framework to pick all numerical parame-
ters while respecting all known parameter constraints that
the system can collect automatically and provide to the
autotuning framework. Compiling for GPUs also intro-
duces hidden constraints for the autotuning framework,
such as choosing only parameter values that will result
in a program ﬁtting in the tight register and memory re-
quirements. When these constraints are not satisﬁed, the
compiler generates code that will fail to execute. There-
fore, the autotuning framework must be able to learn these
hidden constraints automatically.
HPVM2FPGA. HPVM2FPGA [ 13] is a compiler
that enables hardware-agnostic programming of ﬁeld-
programmable gate arrays (FPGAs). The compiler uses
sophisticated optimizations, coupled with design space
exploration (DSE), to automatically tune and gener-
ate well-performing FPGA designs from programs that
have not been written by hardware and FPGA experts.
HPVM2FPGA is part of the Heterogeneous Parallel Vir-
tual Machine (HPVM) compiler infrastructure [33, 12],
which provides a retargetable virtual ISA and compiler
IR for programming heterogeneous systems.
3
Feasibility
Model
Initial Configuration Generator
Acquisition
Function
Optimizer
Initial Phase
Learning Phase
BaCO
Evaluator
Learning
Recommending
Configurations
Runtime
Statistics
Prediction
Model
Compiler
Input Output
Compiler Toolchain
Config.
Search 
Space
Optimized
Config.
Program
Figure 2: Overview of the BaCO framework.
During HPVM2FPGA’s DSE, compiler transforma-
tions, such as loop unrolling, greedy loop fusion, ar-
gument privatization, and kernel fusion are explored.
HPVM2FPGA generates its parameter space automat-
ically through a static analysis of the IR, and the design
space varies depending on the size of the application be-
ing compiled. The majority of the parameters are boolean
parameters, with hidden constraints among them, making
it challenging to explore the space efﬁciently.
3. The BaCO Framework
We introduce Bayesian Compiler Optimization (BaCO)1,
a Bayesian optimization (BO) framework that learns high-
performing auto-scheduling strategies. BaCO thrives in
a small data world where conﬁguration evaluations are
costly, either due to high runtimes of the kernel or ex-
pensive simulations of code generation passes. BaCO
is backend-agnostic, and it can be equally applied to
CPU, GPU, and FPGA compilers. Building on the
BO paradigm, BaCO is centred around a conﬁguration
recommendation-evaluation loop: it recommends promis-
ing new conﬁgurations that are subsequently scheduled
and evaluated by the corresponding compiler toolchain.
The evaluation results are used to ﬁt two predictive mod-
els: one modelling the predicted value and one modelling
the predicted probability of feasibility of new conﬁgura-
tions. To initialize the two models, the procedure starts
with an initial phase, where the ﬁrst few conﬁgurations
are sampled uniformly at random from the search space.
However, for BO to reach its full potential, it needs to
be customized for autotuning tasks. This section explains
the various modules of BaCO’s architecture, shown in
Fig. 2, whereas further specialization towards autotuning
search spaces is emphasized in Sec. 4.
1Baco is Italian for bug.
3.1. Bayesian optimization
Bayesian optimization is a steadily growing methodology
for solving black-box optimization problems. Those are
problems where the objective function f (xxx) can only be
accessed point-wise through expensive evaluations. At
the core of BO is the use of a surrogate model, which esti-
mates the objective function. This helps with performing
well-informed decisions about which conﬁgurations to
evaluate next. The goal is to ﬁnd a good conﬁguration in
as few evaluations as possible. The most common choices
of surrogate model are Gaussian Process (GP) or Ran-
dom Forest (RF) predictors, which both provide useful
uncertainty estimates. For any given point xxx, the model
provides mean and variance, which is used to balance
exploration and exploitation. This trade-off is quantiﬁed
by an acquisition function. Common examples are Ex-
pected Improvement and Lower Conﬁdence Bound. The
surrogate model is dynamically updated to learn from
observations, creating a feedback loop where the BO
framework proposes new points, that are then evaluated.
The information from the evaluation is subsequently used
to train the model. BO was historically developed for
continuous compact domains, and the extension towards
more exotic search spaces is currently being used in this
work and studied by the BO community.
3.2. Surrogate models over compiler domains
Choice of probabilistic model One core element
of an efﬁcient BO algorithm is an accurate surrogate
model [ 15]. While complex parameter domains have
little impact on less intricate methods such as random
sampling, the success of BO depends greatly on clever
handling of such parameters. While traditionally, Ran-
dom Forests have been considered the natural choice as
surrogate models over discrete domains [ 23, 41, 4], re-
cent studies show that a careful implementation of Gaus-
sian Processes yields superior accuracy [18, 9]. However,
to achieve the true potential of GPs in autotuning and
DSE applications, signiﬁcant customization of the GP is
needed. This customization is explained in detail in the
following sections. Sec. 5.3 shows the impact of some
of these major design choices and a comparison between
GPs and RFs.
GP kernel similarity function A key feature in auto-
tuning and DSE is the mixed-variable search space. Thus,
the kernel needs to combine distance measures over differ-
ent parameter types. We propose the weighted Euclidean
norm ||ddd||2
2 = ∑D
i=1 (di/li)2 over the vector of individual
distance measures ddd, as a uniﬁed distance measure. D de-
notes the dimension of the search space, i.e., the number
of parameters being optimized, and li are the horizontal
lengthscales, learned using maximum likelihood estima-
4
tion (MLE) [40], weighting the different parameters. We
use the 5/2-Matérn kernel [47], given by
k(xxx,xxx′) =σ
(
1 +
√
5d +5d2
)
e−
√
5d, (1)
d =
√
D
∑
i=1
d(xi,x′
i)2
l2
i
(2)
where d(xi,x′
i) denotes the distance between xi and x′
i
(described in Sec. 4.1), as this has shown to be efﬁcient in
many real life applications [27, 58]. To increase stability,
we assume that the value observed in each evaluation,
y(xxx), is perturbed by some normally distributed noise [15],
such that y(xxx) =f (xxx)+ ε and ε ∼N(0,σε ).
GP hyperparameter optimization A crucial element
in effective optimization using GPs is to ﬁnd good hyper-
parameters for the model. Especially important are the
length-scales li presented in Eq. (2), which balance the
importance ratio between different parameters. The re-
maining hyperparameters are the outputscale σ in Eq. (1),
and the magnitude of the Gaussian noise σε . BaCO op-
timizes the hyperparameters using a multistart gradient
descent approach, which ﬁrst uniformly samples a num-
ber of possible hyperparameter settings, then chooses a
fraction of those with highest likelihood, and optimizes
them individually using L-BFGS [37].
Discrete parameter spaces offer a number of practi-
cal challenges when ﬁtting the GP model. One such
challenge is that the model hyperparameter optimization
method described above frequently prefers to give close
to zero lengthscale values to some parameters. In practice,
this means that conﬁgurations which take different values
for those parameters have close to zero similarity, making
the GP behave as a sparse model. This is undesirable
as it reduces the model’s expressive power. To address
this artifact of GP modeling, as well as to stabilize the
hyperparameter selection, BaCO uses gamma priors [40]
for the lengthscales. These priors are chosen to be ﬂexi-
ble while cutting out extreme hyperparameter settings. In
practice, stabilizing the lengthscales means that different
parameters are given more equal importance, preventing
certain parameters from becoming too dominant or too
insigniﬁcant due to model over-ﬁtting. Gamma prior dis-
tributions are chosen as they have positive support, can be
made reasonably concentrated and have long tails towards
both zero and inﬁnity. Other good alternatives with simi-
lar properties would be the log-normal or inverse-gamma
distributions. By normalizing the input data, BaCO can
use a single set of priors that works well for the majority
of parameters. Note that, this is an artifact from that many
parameters take identical values in discrete spaces, which
rarely occurs when working with continuous parameters.
3.3. Acquisition function
The acquisition function quantiﬁes the anticipated util-
ity of evaluating a new point. We use the Expected Im-
provement (EI) acquisition function [25], which balances
exploration and exploitation. Autotuning and DSE are
characterized by both discrete search spaces and noisy
function evaluations, in which case we observe that stan-
dard EI has a tendency to overly prioritize re-sampling
points with good values. To avoid this unintended be-
havior, we propose a modiﬁed EI acquisition function
which predicts the expected improvement of observing a
noise free evaluation of the blackbox function. Comput-
ing the EI without considering the noise in the GP makes
sampling repeated points less likely.
BaCO optimizes the acquisition function by multi-start
local search. Initially, a large number of conﬁgurations are
sampled at uniformly random, of which the best conﬁgu-
rations are chosen as starting points for the local search.
Neighbours are deﬁned as all conﬁgurations that can be
reached by modifying a single parameter.
4. Adapting to Exotic Search Spaces
When implementing an efﬁcient autotuning framework,
effectively handling all of the search space features is key.
As BaCO is built around a GP predictive model, careful
design of the distance metrics used for different variable
types is of additional importance. In this section we study
the intricacies of the different parameter types as well as
how to handle known and hidden constraints.
4.1. Parameter types
Continuous, integer, and ordinal parameters These
types of parameters have the property that the values
are comparable, i.e., you can use the greater or equal
sign to order them. This can naturally and explicitly be
translated into a distance metric, and in particular we
use the absolute difference, d(xi,x′
i) =|xi −x′
i|. However,
certain such parameters are innately exponential in nature,
such as tile size parameters. In that case, we use the
Euclidean distance over a log-transformed space instead,
d(xi,x′
i) =|logxi −logxi′|. The log transformation often
more accurately describes the relationship between values.
Consider tile sizes as an example. We expect the tile sizes
2 and 4 to be roughly as similar to each other as the tile
sizes 512 and 1024. However, the tile sizes 512 and 514
would be much more similar than the pairs above.
Categorical parameters Categorical parameters dif-
fer from ordinals in that they have no inherent order. Here,
we use the Hamming distance , deﬁned as dh(xi,x′
i) =
1 xi̸=x′
i
, where 1 is the indicator function, which returns 1
if xi ̸= x′
i and 0 otherwise. In other words, the Hamming
5
1  2  3  4 4  1  3  2
3 1 0 2
Figure 3: Illustration of similarity metrics between two
permutations. The number of discordant pairs (top
of right-hand box, green) is the Kendall distance, the
squared element movement (bottom of right-hand box,
blue) is the Spearman’s rank correlation, and the number
of changing elements (triangles, orange) is the Hamming
distance.
distance only considers whether the parameter values are
identical or not. The scale here is not relevant as the
distance is weighted by the lengthscale li in Eq. (2).
Permutation parameters Permutation parameters are
used to describe the reordering of a sequence of elements.
In compiler applications this most commonly appears as
the reordering of a set of loops [22]. Consider for example
a kernel with four nested loops (l1,l2,l3,l4) which can be
performed in any order. This ordering can be represented
by a single permutation variable πππ, which is a vector
whose element i, given by πi = j, describes the index j of
loop li in the new order. For example, the permutationπππ =
[2,4,3,1] corresponds to the following loop reordering:
for (l1 ... )
for (l2 ... )
for (l3 ... )
for (l4 ... )
...
→
for (l4 ... )
for (l1 ... )
for (l3 ... )
for (l2 ... )
...
Prior black-box optimization literature, autotuning, and
DSE frameworks lack the capability to effectively handle
this variable type, with the notable exception of Open-
Tuner [2]. In BO frameworks employing GPs, it is im-
portant to accurately estimate how different permutations
relate to each other. In other words, for the nested loop
reordering example above, the framework needs to deter-
mine if the two different loop orderings are likely to yield
a similar performance. One naive way of handling permu-
tation variables is to treat them as categorical variables,
e.g., to consider one nested loop ordering to be equally
similar to every other loop ordering (with the exception
of itself). This, however, ignores the underlying struc-
ture that can be used to deﬁne a more reﬁned similarity
measure. We instead present three different semimet-
rics for permutations: the Kendall distance, Spearman’s
rank correlation, and the Hamming distance. While the
semimetrics are not strictly distance metrics, Lomelí et
al. [39] show that they can be used to form a valid GP
kernel. These three semimetrics are illustrated in Fig. 3
on a set of four elements, where the two boxes represent
two permutations π = [1,2,3,4] and π′= [2,4,3,1].
The ﬁrst semimetric is the Kendall distance,
dk(π,π′) =∑m
i=1 ∑m
j=1 |1 π′
i >πj −1 πi>π′
j
|. The Kendall dis-
tance represents the number of discordant pairs, i.e., the
elements that have swapped order between the two permu-
tations. Each discordant pair is represented by a green, in-
terconnected double arrow. The second semimetric is the
Spearman’s rank correlation,ds(π,π′) =∑m
i=1(πi −π′
i )2,
which is the sum of squared movements of the elements
between two permutations. It is illustrated with blue
arrows in Fig. 3, where the dots represent the starting
points and the arrows the ﬁnal position. For example,
the number two starts in the second position in π (left)
and moves to the fourth position in π′(right), meaning
that it has travelled a distance of two. The Spearman’s
rank correlation then sums the squared displacement of
all elements. Note that the square substantially empha-
sizes large rank changes. Lastly, the Hamming distance,
dh(π,π′) =∑m
i=1 1 πi̸=π′
i
, is the number of elements in π
that are no longer at their original position in π′– repre-
sented with orange triangles in the ﬁgure.
For a given permutation set, the choice of semi-
metric depends on how those permutation parame-
ters are expected to impact the performance met-
ric. Intuitively, Kendall distance focuses more on pa-
rameter order, whereas Spearman’s rank correlation
emphasizes large movements of individual elements.
The Hamming distance only considers the number
of elements changed and ignores where they moved
to. As an example, consider the two loop orders
for (l2 ... )
for (l3 ... )
for (l1 ... )
for (l4 ... )
...
and
for (l4 ... )
for (l3 ... )
for (l1 ... )
for (l2 ... ).
...
They have a high Spearman’s rank correlation due to the
large movement of the ﬁrst and last element and rela-
tively smaller Kendall and Hamming distances, which is
intuitive given the compiler transformation that this repre-
sents. This is backed by our ablation analysis in Sec. 5.3,
where we observe that using Spearman’s rank correlation
outperforms the other alternatives. By consequence, we
use Spearman’s rank correlation as a default setting for
permutation variables in BaCO.
4.2. Parameter constraints
For an autotuning framework to be truly competitive in the
complex world of modern autoscheduling, it is essential
to effectively handle constraints in the parameter search
space. Constraints can be divided into known constraints,
which are known prior to optimization, and hidden con-
straints, which are only discovered during optimization.
BaCO is designed to support both these constraint types.
6
2
2
4
2 4
1
1
2 4 8
2
4 8
4
8
4
4
8
p1
p2
p3
p4
p5
Figure 4: Chain-of-Trees with 5 parameters, p1 to p5.
Known constraints In autotuning applications, users
often possess expert knowledge regarding parameter con-
ﬁgurations that lead to inefﬁcient or even infeasible sched-
ules. Incorporating this knowledge into the autotuning
framework leads to signiﬁcant performance improve-
ments. The improvement becomes even greater when
the feasible set makes up a small fraction of all possi-
ble conﬁgurations, i.e., when the search space is sparse.
BaCO handles known constraints during the acquisition
function optimization, and proposes only feasible conﬁg-
urations. As such, the surrogate model trains exclusively
on feasible points.
BaCO uses a Chain of Trees (CoT) data structure to
deal with sparse search spaces, which was ﬁrst presented
by Rasch et al. [46]. The CoT computes all of the feasible
conﬁgurations a-priori and stores them as a collection
(or “chain”) of trees. Each tree corresponds to a group
of co-dependent parameters, and parameters in different
trees are independent of one another. For each tree, each
level of the tree corresponds to a single parameter and
each node in that level corresponds to a possible value
for that parameter. Each path from the root to a leaf then
represents a partial conﬁguration, and the tree is built so
that only feasible conﬁgurations are included. Consider
for example the following search space:
p1 ∈{2,4}, p2 ∈{2,4}, p3 ∈{1,4}, p4 ∈{1,2,4}, p5 ∈{2,4,8}
p1 ≥p2, p4 ≥p3, p5 ≥2p4
In this example, there are ﬁve input parameters and
three constraints. Parameters p1 and p2 as well as pa-
rameters p3, p4, and p5 are co-dependent. We represent
them with the CoT shown in Fig. 4. As the parameters in
different trees are independent, any combination of partial
feasible conﬁgurations from the different trees yields a
feasible conﬁguration. For example, the leftmost path in
the left tree combined with the rightmost path in the right
tree yields the feasible conﬁguration
(p1, p2, p3, p4, p5) = (2,2,4,4,8).
The use of the Chain-of-trees in BaCO is threefold.
First random sampling can be made directly from the CoT.
Secondly, instead of evaluating the constraints explicitly,
it is signiﬁcantly faster to instead check whether the con-
ﬁguration belongs to the CoT. Thirdly, it allows working
with highly sparse search spaces which are common in
the autotuning domain. In such sparse spaces, operating
directly on the original domain becomes infeasible.2
Working with constrained spaces is inherently biased
since the optimization method presented in Rasch et
al. [45] prioritizes different conﬁgurations depending on
the structure of the CoT. Their approach is equivalent to
random sampling a conﬁguration from the CoT by start-
ing at each root node and then iteratively choosing a child
with uniform probability, which is biased towards con-
ﬁgurations in less dense parts of the tree. This bias is
furthermore dependent on the order in which the param-
eters appear in the tree, which is an undesirable feature.
We propose an alternative approach where we instead
sample uniformly from the leaves of the trees, which is
bias-free, meaning that the random sampling will be per-
formed uniformly on the all the conﬁgurations. We study
the impact of the bias in Sec. 5.
Another source of sparsity comes from how parameters
are deﬁned prior to optimization. In autotuning and DSE
applications, due to the nature of how computers store
information, parameters are often constrained to take ex-
ponential values. Treating such parameters as integers
leads to sparsity in the search spaces. BaCO instead ap-
plies the logarithmic transformation to such parameters.
This transformation makes the search space signiﬁcantly
denser and yields more natural distances for the GP. These
qualities improve performance, as we shall see in Sec. 5.
Hidden constraints Requiring the complete feasible
domain deﬁnition from the user would severely limit the
autotuner’s usability. Some constraints are either too
complicated to describe analytically or unknown a-priori.
Instead, BaCO supports the concept of hidden constraints,
learned automatically during optimization. BaCO uses
a Random Forest model to predict the probability of fea-
sibility for each conﬁguration. It then extends the EI
acquisition function presented in Sec. 3.3, by multiply-
ing the EI with the probability of feasibility [ 41]. This
should be compared with the naive approach of assigning
high objective values to infeasible conﬁgurations, which
suffers from difﬁculties with setting an accurate penalty.
Such penalty terms are further often detrimental for the
statistical model.
However, the practical interaction between the acqui-
sition function based on the GP model and the RF feasi-
bility predictor is complex. There is a constant trade-off
between the feasibility model wanting to select feasible
points and the value predictor that seeks to explore the
unexplored infeasible regions. If the surrogate model be-
comes excessively conﬁdent within the feasible region,
this balance tends to be skewed. In which case the selec-
2In the results section, we have used domain expert knowledge to man-
ually transform the search spaces to limit the sparsity. This allows a
more interesting comparison with previous methods.
7
tion fails to reliably ﬁnd feasible points. As a practical
solution, we consider a minimum feasible limit εf and
only consider conﬁgurations with probability of feasibil-
ity greater than εf for evaluation. By randomly sampling
a new εf each iteration, with p(εf = 0) > 0, we asymp-
totically guarantee to not cut away any solutions by doing
this.
5. Evaluation
We validate the efﬁciency, effectiveness, and generaliz-
ability of BaCO. We ﬁrst introduce the reference auto-
tuning methods that we use as a baseline to evaluate the
performance of BaCO, followed by the benchmarks from
the three real-world frameworks presented in Sec. 2. We
then show the performance results. For lack of space
we show the extensive empirical results on all the frame-
works and benchmarks in Appendix A. All experiments
are run for 30 repetitions. We also show a wall-clock time
analysis of all the autotuners used in Appendix B.
We answer the following research questions (RQ):
RQ1) Does BaCO achieve high performance with a
limited autotuning budget? The evaluation in Fig. 5
shows that, with a tiny budget of 20–40 evaluations,
depending on the complexity of the benchmark, BaCO
achieves 1.35×–1.55×better performance than the state-
of-the-art baselines. Furthermore, BaCO consistently
achieves expert-level performance with a small budget of
40–80 evaluations, where the baselines struggle to achieve
expert-level performance even with a much larger budget.
This demonstrates the advantage of BaCO to deliver high
performance for a small budget, even for complex search
spaces.
BaCO delivers expert-level performance on average
2.87×–3.87×faster than the state-of-the-art autotuning
frameworks. Fig. 6 highlights the quicker performance
evolution of BaCO for representative benchmarks, and
a more detailed breakdown is presented in Table 9 in
the appendix. These results show that BaCO delivers
performance much quicker than the baselines.
RQ2) Does BaCO generalize across compiler frame-
works and benchmarks? Our evaluation across three di-
verse real-world compiler frameworks shows consistently
that BaCO signiﬁcantly outperforms the baselines. In
fact, BaCO is the only framework that outperforms the
expert conﬁguration on all benchmarks across compiler
frameworks, as shown in Fig. 6, with a more detailed
breakdown in Table 5 in the appendix. This observation
suggests that the techniques discussed in the paper gener-
alize well across compiler frameworks and benchmarks.
RQ3) What is the performance beneﬁt of customizing
Bayesian optimization for compiler autotuning? Our sys-
tem demonstrates advantages over prior work BO-based
autotuning frameworks not speciﬁcally customized for
compiler domains (Ytopt in Fig. 8). As explained in
Sec. 3 and Sec. 4, these improvements validate our de-
sign choices and suggests that there are signiﬁcant per-
formance beneﬁts to be gained by customizing the BO
framework for this particular domain.
RQ4) What are the ﬁndings of autotuning our dis-
tinct real-world compilers using BaCO? This question
attempts to provide insight on why BaCO outperforms
baselines for our real-world compiler benchmarks. We
identify three main areas: exploration of new conﬁgura-
tions, testing schedules that did not previously exist in
prior work, and better handling of both known and un-
known constraints for complex real-world applications.
Evaluations for this question comes from Fig. 5, Fig. 7,
and Fig. 11 in the Appendix.
5.1. Baseline Methods
To contextualize the performance of BaCO, we evaluate
it alongside two state-of-the-art autotuning frameworks
and two random sampling approaches.
ATF with OpenTuner The Auto-Tuning Framework
(ATF) [46] extends the popular OpenTuner [2] to handle
known constraints. We chose OpenTuner as a baseline
since it is one of the leading frameworks for autotuning.
Ytopt Ytopt [63] is an autotuning framework using
BO and is part of the PROTEAS-TUNE project [ 1]. It
supports both Random Forests and Gaussian processes.
We run it here with Random Forests as the GP imple-
mentation does not support constraints. When infeasible
solutions are found due to hidden constraints, they are
added to the data set with a high objective value. We com-
pare against Ytopt since it is one of the only frameworks
that supports either constraints or GP. When addressing
RQ3, we run Ytopt with GPs.
Uniform and CoT random sampling These are uni-
form random sampling methods. The CoT random sam-
pling is a method that randomly samples at uniform di-
rectly from the CoT. This baseline allows us to study the
impact of the bias introduced by the known constraints,
as explained in Sec. 4.2.
Default and expert conﬁgurations For reference, we
show the performance of two baseline conﬁgurations: The
default conﬁguration, and an expert conﬁguration, care-
fully handcrafted by domain experts in the respective pro-
gramming languages. It is unlikely that a developer would
exceed the expert performance baseline, which makes
it a suitable data point for our empirical analysis. The
HPVM2FPGA benchmarks are automatically generated
by the autoscheduler and do not provide any expert con-
ﬁgurations, in which case we only report the default. The
expert conﬁgurations are taken from prior publications:
TACO [49] and RISE & ELEV ATE [19, 29, 54, 57]. The
original authors used manual or semi-automated methods
8
Benchmark Dim Constr. Feasible Full
Params Space size Budget
TACO
SpMV 7 O/C/P 1 .5 ×107 3.0 ×106 70
SpMM 6 O/C/P K 5 .2 ×1011 4.7 ×104 60
SDDMM 6 O/C/P K 5 .2 ×1011 7.8 ×104 60
TTV 7 O/C/P K/H 1 .5 ×107 6.0 ×106 70
MTTKRP 6 O/C/P K 1 .5 ×106 6.8 ×105 60
RISE & ELEV ATE
MM_CPU 5 O/P K/H 1 .0 ×107 2.9 ×104 100
MM_GPU 10 O K/H 1 .1 ×1011 1.5 ×108 120
Asum_GPU 5 O K 1 .2 ×106 6.3 ×104 60
Scal_GPU 7 O K/H 3 .9 ×107 4.2 ×106 60
K-means_GPU 4 O K/H 1 .4 ×104 3.6 ×103 60
Harris_GPU 7 O K 7 .7 ×109 1.0 ×107 100
Stencil_GPU 4 O K 1 .4 ×104 3.6 ×103 60
HPVM2FPGA
BFS 4 I/C H 256 256 20
Audio 15 I/C H 8 .4 ×105 8.4 ×105 60
PreEuler 7 I/C H 1 .5 ×104 1.5 ×104 60
Table 3: We evaluate BaCO on 15 important kernels from
domains like machine learning, statistics, and signal pro-
cessing. The benchmarks expose search spaces with
varying number of parameters ( Dim). They cover all pa-
rameter types considered ( Params): real (R), integer (I),
ordinal (O), categorical (C), and permutation (P). Con-
str. describes the type of constraints used by the bench-
mark: known (K) and hidden (H) constraints. Space size
describes the number of possible conﬁgurations in the
dense search space with Feasible denoting all valid con-
ﬁgurations with respect to the known constraints. Full
Budget is the total number of evaluations we allow for au-
totuning the kernel.
to determine well-performing conﬁgurations based on
their experience, hardware characteristics, or data prop-
erties. The authors had the incentive to produce the best
conﬁgurations but, presumably due to time constraints,
they may have occasionally missed better-performing con-
ﬁgurations.
5.2. Benchmarks
BaCO is evaluated over 15 kernels from linear algebra,
machine learning, image processing, statistics, and signal
processing. We integrate BaCO in the three real-world
frameworks presented in Sec. 2. The benchmarks have
been chosen based on prior work by the authors of the
three frameworks. Furthermore, most of these bench-
marks have an expert optimized code which allows for a
fair comparison. The search space size ranges from tens
of thousands to billions of conﬁgurations, as described in
Table 3, which is beyond the scope of exhaustive search.
To deﬁne the evaluation budget for each benchmark,
we ﬁrst establish a full budget for each benchmark, as
shown in the last column of Table 3. The full budget is
deﬁned using a rule of thumb of around 5 to 6 minutes.
Matrix Dimension Nonzeroes Dataset
ACTIVSg10K 20,000 ×20,000 135 ,888 SS
email-Enron 36,692 ×36,692 367 ,662 SS
Goodwin_040 17,922 ×17,922 561 ,677 SS
scircuit 170,998 ×170,998 958 ,936 SS
ﬁlter3D 106,437 ×106,437 2 ,707,179 SS
laminar_duct3D 67,173 ×67,173 3 ,788,857 SS
cage12 130,228 ×130,228 2 ,032,536 SS
smt 25,710 ×25,710 3 ,749,582 SS
random2 10,000 ×10,000 5 ,000,000 Rand
random1 1000 ×500 ×100 5 ,000,000 Rand
facebook 1,504 ×42,390 ×39,986 737 ,934 FB
uber 183 ×24 ×1,140 ×1,717 3 ,309,490 FT
nips 2,482 ×2,482 ×14,036 ×17 3 ,101,609 FT
chicago 6,186 ×24 ×77 ×32 5 ,330,673 FT
uber3 183 ×1,140 ×1,717 1 ,117,629 FT*
Table 4: Tensors used in our TACO benchmarks from the
SuiteSparse matrix collection (SS) [10, 31], Facebook Ac-
tivities graph (FB) [59], FROSTT (FT) collection [51] or
synthetically generated (Rand).
*We modify a FROSTT 4-tensor to a 3-tensor by dropping
one dimension since the next largest FROSTT tensor has
more than 75×the number of nonzeros.
This max compilation time is commonly adopted in large
companies, such as Google. We then deﬁnetiny and small
budgets as 1/3 and 2/3 of the full budget.
TACO benchmarks We benchmark 5 tensor algebra
expressions, commonly used in machine learning and ten-
sor factorization [16]. Namely, they comprise of sparse
matrix-vector multiply (SpMV) ai = ∑k Bi jck, sparse ma-
trix multiply (SpMM) Ai j = ∑k BikCk j, sampled dense-
dense matrix multiply (SDDMM) Ai j = ∑k Bi jCikDjk, ten-
sor times vector (TTV) Ai j = Bi jkck, and fourth-order
matricized tensor times Khatri-Rao product (MTTKRP)
Ai j = Biklm ∗Ck j ∗Dl j ∗Em j. Each tensor expression is
given a scheduling template that exposes tiling parame-
ters (split and unrolling factors) and permutation param-
eters (loop reorderings). BaCO searches for the set of
parameters, and therefore the schedule, that yields the
best performance. The characteristics of these parameters
and the search space is described in Table 3. We use
tensors from a wide variety of real-world applications
ranging from power networks and circuits to ﬂuid dynam-
ics and social networks. We run matrix expressions on
a subset of SuiteSparse matrices [ 10, 31] and synthetic
uniform random tensors, and we run higher-order expres-
sions on the Facebook Activities tensor [59], a subset of
the FROSTT tensor collection [51], and synthetic tensors
as well (see Table 4). The selected tensors vary widely
across tensor properties including number of nonzeroes,
dense dimension size, and irregular nonzero pattern.
The TACO benchmarks were run on nodes with two In-
tel Xeon Gold 6130 processors locked at 2100Ghz, using
all 32 cores and 96GB of RAM.
RISE & ELEV ATE benchmarksWe use six bench-
marks covering multiple domains, optimizations, and
hardware devices. This results in benchmarks requiring
various autotuning features, as described in Table 3.
9
The CPU Matrix Multiplication (MM_CPU) bench-
mark from [19] is run on a CPU and applies tiling, vector-
ization, and loop-permutation optimizations. The remain-
ing benchmarks are run on a GPU and apply GPU-speciﬁc
optimizations, including the OpenCL-speciﬁc work-group
conﬁguration, memory hierarchies, and coalescing. The
MM_GPU and K-means_GPU dense linear algebra bench-
marks are inspired by implementations used in [56]. The
linear algebra algorithms Asum_GPU and Scal_GPU are
from [54], the stencil from [ 57]. The remaining image
processing algorithm Harris_GPU is a corner detector
described in [29].
The RISE & ELEV ATE evaluation was executed on 8
cores of an Intel Xeon E5-2650 v3 @2.30Ghz processor
accompanied by 32 GB of RAM. For the GPU bench-
marks, we used a NVIDIA K80 GPU.
HPVM2FGA benchmarks We use the benchmarks
presented in [13]: (1) Breadth First Search (BFS), and
(2) the computational ﬂuid dynamics algorithm of euler
with pre-computed ﬂuxes (PreEuler), are taken from the
Rodinia Benchmark suite [6], and (3) 3D Spacial Audio
Encoder (Audio) from the ILLIXR testbed [ 24]. The
benchmarks represent diverse workloads from different
domains with varying parameter space sizes, ranging from
4 parameters for BFS, to 15 for Audio.
We ran these benchmarks through HPVM2FPGA’s op-
timizer, reporting the estimated execution time targeting
an Intel Arria 10 GX FPGA in our evaluation results.
5.3. Results
RQ1) Does BaCO achieve high performance with a
limited autotuning budget? Fig. 5 shows the average
performance of BaCO and the baselines for three differ-
ent levels of autotuning budget for all our benchmarks.
The tiny budget is only 20-40 evaluations.3 BaCO clearly
outperforms the baselines, delivering on average better
performance than all baselines for 19 out the 24 bench-
marks. For TACO, the tiny budget is even sufﬁcient to
deliver expert-level performance. With the small budget,
BaCO delivers expert performance for all three compiler
frameworks. The baselines struggle to deliver good per-
formance even with the full budget, particularly for the
challenging spaces in the RISE benchmarks. Tables 6,
7, and 8 in the appendix show the detailed performance
results for each individual benchmark and autotuning
framework.
BaCO also achieves performance faster, i.e., it reaches
the ﬁnal performance of the other baselines using fewer
conﬁguration evaluations. Fig. 6 shows the performance
evolution for three selected benchmarks, and that BaCO
delivers performance with 2.9 ×–5×fewer evaluations
than ATF and Ytopt. On average, BaCO ﬁnds the best
3Besides the BFS benchmark, for which it is only 6 evaluations
tiny small full
0.0x
0.2x
0.5x
0.8x
1.0x
1.2x
tiny small full
0.0x
0.2x
0.5x
0.8x
1.0x
1.2x
tiny small full
0.0x
0.5x
1.0x
1.5x
BaCO
ATF with OpenTuner
Ytopt
Uniform Sampling
CoT Sampling
Default
Autotuning BudgetAutotuning BudgetAutotuning Budget
Average Performance relative to Expert
HPVM2FPGA
RISE & ELEVATE
TACO
Figure 5: Average performance relative to expert with a
tiny budget (1/3 of the full budget), a small budget (2/3
of the full budget) and the full budget. BaCO delivers
the highest performance for each budget and achieves
expert-level performance consistently with the small bud-
get.
ATF and Ytopt conﬁgurations 2.87 ×and 3.82×faster,
respectively. Our experiments conﬁrm that these results
generalize well across our benchmarks, however, due to
space constraints these additional results are presented in
Table 6 in the Appendix.
RQ2) Does BaCO generalize across compiler
frameworks? To see how the performance generalizes,
we show in Fig. 7 (and Fig. 11 in the Appendix) how
the mean of the best-found solution by each framework
improves over time for each individual benchmark. In
the ﬁgures, the average performance is plotted for each
method and each benchmark. The goal is to achieve a
lower value, which identiﬁes a better-performing conﬁgu-
ration, and to ﬁnd low values as far to the left as possible,
which means using a low tuning budget. The performance
of the default conﬁguration and expert conﬁguration is
presented for reference when available. We further denote
10
5x
3.3x
3.7x
4.6x 2.9x
Figure 6: Evolution of average best runtime for one kernel from each framework. The ﬁgure is split vertically into
two different scales. BaCO reaches the ﬁnal performance of the state-of-the-art methods using as little as 30% of the
function evaluations of the other methods.
when each method reaches expert-level performance with
a star, such that a shorter distance between the y-axis and
the star is better. As ﬁnding improvement over the default
conﬁguration initially is easy, we split the plots into two
regions with different scales. This helps focus on the inter-
esting part closer to the expert-level performance. BaCO
reliably yields high-level performance and overall pro-
vides the best schedule in 22 out of the 24 benchmarks. It
is further frequently the only method to reach expert-level
performance within the given budget.
RQ3) What is the performance beneﬁt of customiz-
ing Bayesian optimization for compiler autotuning?
We study the BaCO design by running three matrices for
SpMM with default settings and with a number of major
features turned off. We use the SpMM benchmark as it
is reasonably well-behaved and only has few constraints.
The average speedup over expert is shown in Fig. 8. We
denote the restricted version by BaCO--. In more detail,
BaCO-- is BaCO without variable transformations, model
priors, and local search for the acquisition function. It
further uses the naive distance for permutation variables
that ignores their underlying structure and it does not use
BaCO’s more advanced ﬁtting of the GP hyperparameters.
We see that by doing those changes, BaCO takes about a
20% performance loss.
Next, we compare it with the GP implementation of
Ytopt. Ytopt uses none of the above mentioned features,
but additionally has a less advanced GP and BO toolkit.
Ytopt only supports constraints for RF so this Ytopt con-
ﬁguration does not support constraints. However, for this
benchmark we have manually pruned the search space for
Ytopt, so that the only remaining constraint is a single
less-than relationship between two variables. BO with
GPs requires a lot of care to be efﬁcient, which we see
from the difference between Ytopt (GP) and BaCO--.
Lastly, we show the difference between a well imple-
mented GP and RF as surrogate model. Especially for
smaller budgets, the GP model shows much stronger per-
formance. This is relevant as there is currently a paradigm
shift towards using GPs in discrete settings.
Ablation analysis To further understand the impact
of the different design choices in BaCO, we perform an
ablation analysis in Fig. 9. The impact of the permutation
kernel, variable transformations and model priors are stud-
ied in an ablation analysis. First, BaCO in default settings
is presented, which is using Spearman’s rank correlation
for permutation variables. Then we study the impact
of changing the permutation metric to Kendall distance,
Hamming distance, as well as the naive approach of treat-
ing permutations as categorical variables (Sec. 4.1). The
Spearman metric yields the best performance, especially
in early iterations.
Secondly, we study the impact of removing the loga-
rithmic transformations of variables and output (Sec. 4.2),
and the model priors (Sec. 3.2). Removing the log trans-
forms signiﬁcantly deteriorates the performance at all
evaluation counts. The lengthscale priors, however, have
a larger impact early on, where they work to stabilize the
procedure, and become less important when more data
has been observed with which to ﬁt the model.
Overall, we see that the changes have a much larger im-
pact early on. Except for transformations, ignoring any of
the individual features fails to prevent good performance
after more iterations. It is noteworthy that no individual
design choice has a major impact, but together they make
a large difference.
Hidden constraints Next, we study the impact of pre-
dicting feasibility with respect to hidden constraints on
two benchmarks from the RISE/ELEV ATE suite. In Fig-
ure 10, we show the average improvement over expert
after 20, 40, 60 iterations with and without the feasibil-
ity predictor. Additionally, we show the impact of the
minimum feasibility limit presented in Sec. 4.2. It shows
that the hidden constraints predictor has a signiﬁcant posi-
tive impact, particularly after more iterations where it has
had more samples to train on. But it also indicates that
the introduced minimum feasibility limit (Sec. 4.2) is im-
portant to stabilize the interaction between the feasibility
predictor and the surrogate model.
Chain-of-Trees Even after manual sparsity-reducing
11
2
4
HPVM2FPGA BFS
5 10 15 20
0.25
0.50
0.75
1000
2000
RISE/ELEVATEMM CPU
0 20 40 60 80 100
200
300
200
300
TACO SpMM scircuit
0 20 40 60
50
100
5000
6000
7000
HPVM2FPGA Audio
0 20 40 60
3000
4000
1000
2000
RISE/ELEVATEMM GPU
0 25 50 75 100 125
0
20
40
25000
50000
TACO TTV facebook
0 20 40 60
0
10
20
12
14
HPVM2FPGA PreEuler
0 20 40 60
5
10
100200300
RISE/ELEVATEAsum GPU
0 20 40 60
0
5
1.5
2.0
2.5
TACO SpMV cage12
0 20 40 60
0.5
1.0
BaCO
ATF with OpenTuner
Ytopt
Uniform Sampling
CoT Sampling
Expert Conﬁguration
Default Conﬁguration
DoE
Number of Conﬁguration Evaluations
Runtime [ms]
Evaluation of Average Best Runtime (Lower is better)
Figure 7: Evolution of average best runtime among evaluated conﬁgurations for selected benchmarks from Table 3.
Fig. 11 in the Appendix contains from Table 3 not shown here. The ﬁgure is split vertically into two different scales, and
we mark the iteration where each method beats the expert conﬁguration with a star.
transformations, some of the search spaces remain highly
sparse. When this is the case, CoTs greatly increases the
efﬁciency of sampling from, and searching, the parameter
domain. On the MM_GPU search space for example,
over ten runs, using the CoTs reduced the time spent on
evaluating constraints in the local search by a factor 6×
and the random sampling by a factor of80×. Overall, this
resulted in that the time spent by the internal working of
BaCO was reduced by 70%. For even more sparse search
spaces, operating directly on the search space quickly
becomes untenable.
RQ4) What are the ﬁndings of autotuning our dis-
tinct real-world compilers using BaCO?
Conﬁguration insight BaCO underperforms baselines
in only 1 of our 24 (4%) benchmarks across the 15 ker-
nels. Opentuner is able to beat BaCO (as shown in Fig. 7)
when running SpMV on cage12 (see Table 4). The SpMV
benchmark is interesting as it has a good default setting,
but ill-designed schedules can increase the run time by
several orders of magnitude. After inspection of the con-
20 40 60
Number of Conﬁguration Evaluations
0.0x
0.2x
0.5x
0.8x
1.0x
1.2x
1.5x
BaCO BaCO-- Ytopt (GP) RFs
Comparison between BO Implementations
Performance rel. to Expert
Figure 8: Geometric mean of the performance relative to
the expert conﬁguration for the TACO SpMM kernel ap-
plied to the ﬁlter3D, email-Enron and amazon0312 matri-
ces after 20, 40 and 60 evaluations.
ﬁgurations, it shows that ATF picks conﬁgurations simi-
lar to its previous conﬁgurations. This behavior in ATF
exploits sampling around prior good conﬁgurations each
evaluation, whereas BaCO’s algorithm is more explorative
12
Figure 9: Geometric mean of the performance relative to
the expert conﬁguration for the SpMM kernel applied to
the ﬁlter3D, email-Enron and amazon0312 matrices after
20, 40 and 60 evaluations. (Note the cut axis).
20 40 60
Number of Conﬁguration Evaluations
0.0x
1.0x
2.0x
3.0x
4.0x
5.0x
BaCO
No Hidden constraints
No Feasibility limit
Impact of Hidden Constraints
Performance rel. to Expert
Figure 10: Geometric mean of the performance relative to
the expert conﬁguration for the MM_GPU and Scal_GPU
kernels after 20, 40 and 60 evaluations.
in ﬁnding completely new conﬁgurations. Exploiting con-
ﬁgurations works for simple kernels, like SpMV , but fails
for real-world kernels with increased complexity and run-
time variance. Exploitation sampling is likely to get stuck
in a local minima, which is more likely to be globally bad
for complex problems (as is the case with TTV on the
random1 in Fig. 11 in the Appendix). We do not augment
BaCO in any speciﬁc manner to explore conﬁgurations,
but the global nature of the BO paradigm emphasizes
exploration over methods with local-exploration elements
such as OpenTuner.
Performance over expertBaCO is able to achieve bet-
ter than expert performance in some cases (see Fig. 5).
Even experts in the domain, with insight of the underly-
ing hardware architecture, may miss out on the optimal
conﬁguration and the best performance simply due to the
amount of user-time needed to explore the vast search
space of possible conﬁgurations. BaCO allows users to
automate that search while potentially offering better per-
formance than the expert could ﬁnd. For example, BaCO
is able to ﬁnd over a 1.1×speedup on average for TACO
(see Fig. 5) since experts in [49] only considered the de-
fault loop ordering (permutation) for all expressions. In
addition, many of the conﬁgurations the autotuner dis-
covers are hard to ﬁnd by hand due to the concordant
traversal of a compressed tensor data structure. Therefore,
it is difﬁcult for an expert to search the space of loop
orderings and know which of them are infeasible.
Constraints in real-world applications Over half
(8/15) of our benchmarks, and notably all of the
HPVM2FPGA benchmarks, use hidden constraints. Ad-
ditionally, all but one benchmark use known constraints,
signiﬁcantly reducing the feasible search space as shown
by the Feasible column in Table 3. Predictor modeling
of hidden constraints has a signiﬁcant impact on perfor-
mance (as discussed in RQ3), and this impact is apparent
in our real-world compiler benchmarks.
6. Related Work
Bayesian optimization for autotuningSeveral Bayesian
optimization frameworks have been presented for auto-
tuning [41, 63, 61, 38]. One of the earlier frameworks
was introduced by Nelson et al. [42], who present SURF
that uses Random Forests models to optimize tensor con-
straction operations on GPUs. This work is later extended
to become Ytopt by Wu et al. [63]. The authors use
the skopt Bayesian optimization framework to optimize
LLVM Clang/Polly pragma conﬁgurations on the Poly-
Bench benchmark suite. Ytopt further allows the usage of
additional surrogate models such as Gaussian Processes
and Boosted Trees. Ytopt implements a method based on
Bayesian optimization, which BaCO builds on. However,
we show that the Bayesian optimization pipeline needs
to be further customized to work well on autotuning do-
mains, which is the scope of our work.
The work by Sid-Lakhdar et al. [50] focuses on the
meta- and multi-task learning aspect. It was extended by
Liu et al. [38] into the GPTune framework. The authors
use linear coregionalization models (LCMs), to model
multiple similar problems simultaneously to increase ef-
ﬁciency. This was further extended by Zhu et al. [65] to
also handle multiﬁdelity applications. While this is out of
the scope of the current work, the use of meta-learning
can be used in combination to BaCO to achieve greater
efﬁciency. Willems et al. [61] use Bayesian optimization
to autotune GPU kernels using Gaussian Processes and
known constraints on the search space. Another recent
approach is Bliss by Roy et al. [48], that probabilistically
chooses a combination of models and acquisition func-
tions each new optimization iteration based on previous
performance observations. Bliss’ approach is orthogo-
nal to BaCO and it is possible that combining the meth-
ods further efﬁciency can be achieved. Recently, Dorier
et al. [11] present DeepHyper, a Bayesian optimization
framework for HPC storage system autotuning, that fo-
13
cuses on transfer learning through the use of variational
autoencoders.
Bayesian optimization for design space exploration
Nardi et al. [41, 30] use Bayesian optimization with a RFs
surrogate model to optimize FPGAs. They consider both
multiobjective and hidden constraints. Ejjeh et al. [13]
use the same DSE framework to tune hardware-agnostic
programs targeting FPGA backends. [53] design a human-
centric DSE approach, where expert priors accelerate the
convergence of the autotuner. While this is not the focus
of our work, a simple adaptation of the BaCO acquisition
function can beneﬁt the same user priors when available.
There is substantial work in the literature about DSE
techniques in HLS [5, 14, 62, 60, 64]. However, most ex-
isting work focuses on using DSE for tuning HLS, rather
than using it to select compiler optimizations [5, 14, 64].
These works are not based on Bayesian optimization and
we view them as complementary to our work.
The phase-ordering problem Autoscheduling tack-
les the task of applying a number of transformations to
optimize a kernel automatically. Typically, the schedul-
ing language parametrizes the application of those trans-
formations by a bounded set of options which we refer
to as parameters that are easier to optimize over. This
parametrization approach is the one used by TACO and
ELEV ATE. However, a different approach is to operate
directly on the space of transformations. Optimization
over this unbounded tree-like space is commonly known
as the phase-ordering problem [34, 32, 22, 21].
7. Conclusions and Future Work
We introduce the Bayesian Compiler Optimization frame-
work (BaCO), a plug-and-play solution to autoscheduling
tasks for modern scheduling languages targeting various
hardware backends. BaCO is able to reach expert-level
performance 2.7×-10×faster than the state of the art au-
totuners. The separation of concerns between policy and
mechanism allows compiler users to delegate the complex
and time-consuming task of scheduling to BaCO so that
they can focus on their applications instead.
While we show that BaCO can provide high-
performing solutions in less than 100 seconds, this time is
still too long for use in software development. The holy
grail of autoscheduling is to be able to use an autotuner
during the development, and ideally enable the user to
run autotuning every time they compile their code. That
way users can check both functional and non-functional
properties on a regular basis during the various program
lifecycle phases. Indeed, increasing the efﬁciency of
the autotuner would enable a new level of autotuning-
in-development-loop paradigm which is not accessible
with the current state of autotuning technology.
Acknowledgements
We would like to thank Jaeyeon Won for his help with
evaluation and Michael O’Boyle and Tobias Grosser for
their valuable feedback the a draft of this paper. Luigi
Nardi was supported in part by afﬁliate members and
other supporters of the Stanford DAWN project — Ant
Financial, Facebook, Google, Intel, Microsoft, NEC, SAP,
Teradata, and VMware and by the Wallenberg Launch
Pad (WALP) grant Dnr 2021.0348. Erik Hellsten and
Luigi Nardi were supported by the Wallenberg AI, Au-
tonomous Systems and Software Program (W ASP) funded
by the Knut and Alice Wallenberg Foundation. Adel
Ejjeh was supported in part by funding from an Intel
Research Award, from IBM under the DARPA DSSoC
program, and from the University of Illinois. Olivia Hsu
was supported by an NSF GRFP Fellowship. This re-
search was also supported in part by the Google Research
Scholar program. The computations were enabled by re-
sources provided by the Swedish National Infrastructure
for Computing (SNIC) at LUNARC, partially funded by
the Swedish Research Council through grant agreement
no. 2018-05973.
Any opinions, ﬁndings, and conclusions or recommen-
dations expressed in this material are those of the authors
and do not necessarily reﬂect the views of the aforemen-
tioned funding agencies.
14
References
[1] Proteas-tune. https://www.ornl.gov/project/
proteas-tune. Accessed: 2022-10-18.
[2] Jason Ansel, Shoaib Kamil, Kalyan Veeramachaneni, Jonathan
Ragan-Kelley, Jeffrey Bosboom, Una-May O’Reilly, and Saman
Amarasinghe. OpenTuner: An extensible framework for program
autotuning. In International Conference on Parallel Architectures
and Compilation Techniques (PACT), 2014.
[3] The GPyOpt authors. GPyOpt: A bayesian optimization frame-
work in python. http://github.com/SheffieldML/GPyOpt,
2016.
[4] Prasanna Balaprakash, Jack Dongarra, Todd Gamblin, Mary Hall,
Jeffrey K Hollingsworth, Boyana Norris, and Richard Vuduc.
Autotuning in high-performance computing applications. Pro-
ceedings of the IEEE, 106(11):2068–2083, 2018.
[5] Pedro Bruel, Alfredo Goldman, Sai Rahul Chalamalasetti, and
Dejan Milojicic. Autotuning high-level synthesis for fpgas using
opentuner and legup. In International Conference on ReConFig-
urable Computing and FPGAs (ReConFig). IEEE, 2017.
[6] Shuai Che, Michael Boyer, Jiayuan Meng, David Tarjan,
Jeremy W. Sheaffer, Sang-Ha Lee, and Kevin Skadron. Rodinia:
A benchmark suite for heterogeneous computing. In Interna-
tional Symposium on Workload Characterization (IISWC). IEEE,
2009.
[7] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng,
Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei
Hu, Luis Ceze, et al. TVM: An automated End-to-End optimizing
compiler for deep learning. In USENIX Symposium on Operating
Systems Design and Implementation (OSDI), 2018.
[8] Stephen Chou, Fredrik Kjolstad, and Saman Amarasinghe. For-
mat abstraction for sparse tensor algebra compilers. Proc. ACM
Program. Lang., 2(OOPSLA):123:1–123:30, October 2018.
[9] Jhouben Cuesta Ramirez, Rodolphe Le Riche, Olivier Roustant,
Guillaume Perrin, Cedric Durantin, and Alain Gliere. A com-
parison of mixed-variables bayesian optimization approaches.
Advanced Modeling and Simulation in Engineering Sciences ,
9(1):1–29, 2022.
[10] Timothy A Davis and Yifan Hu. The university of ﬂorida sparse
matrix collection. ACM Transactions on Mathematical Software
(TOMS), 38(1):1–25, 2011.
[11] Matthieu Dorier, Romain Egele, Prasanna Balaprakash, Jaehoon
Koo, Sandeep Madireddy, Srinivasan Ramesh, Allen D Malony,
and Rob Ross. Hpc storage service autotuning using variational-
autoencoder-guided asynchronous bayesian optimization. arXiv
preprint arXiv:2210.00798, 2022.
[12] Adel Ejjeh, Aaron Councilman, Akash Kothari, Maria Kotsi-
fakou, Leon Medvinsky, Abdul Rafae Noor, Hashim Sharif, Yifan
Zhao, Sarita Adve, Sasa Misailovic, et al. HPVM: Hardware-
agnostic programming for heterogeneous parallel systems. IEEE
Micro, 42(5):108–117, 2022.
[13] Adel Ejjeh, Leon Medvinsky, Aaron Councilman, Hemang Nehra,
Suraj Sharma, Vikram Adve, Luigi Nardi, Eriko Nurvitadhi,
and Rob A Rutenbar. HPVM2FPGA: Enabling true hardware-
agnostic fpga programming. In IEEE International Conference
on Application-speciﬁc Systems, Architectures, and Processors
(ASAP), 2022.
[14] Lorenzo Ferretti, Andrea Cini, Georgios Zacharopoulos, Cesare
Alippi, and Laura Pozzi. A graph deep learning framework for
high-level synthesis design space exploration, 2021.
[15] Peter I Frazier. A tutorial on bayesian optimization. arXiv
preprint arXiv:1807.02811, 2018.
[16] Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse
gpu kernels for deep learning. In Proceedings of the Interna-
tional Conference for High Performance Computing, Networking,
Storage and Analysis, SC ’20. IEEE Press, 2020.
[17] Jacob R Gardner, Matt J Kusner, Zhixiang Eddie Xu, Kilian Q
Weinberger, and John P Cunningham. Bayesian optimization with
inequality constraints. In International Conference on Machine
Learning (ICML), 2014.
[18] Eduardo C Garrido-Merchán and Daniel Hernández-Lobato.
Dealing with categorical and integer-valued variables in bayesian
optimization with gaussian processes. Neurocomputing, 380:20–
35, 2020.
[19] Bastian Hagedorn, Johannes Lenfers, Thomas Koehler, Xueying
Qin, Sergei Gorlatch, and Michel Steuwer. Achieving high-
performance the functional way: a functional pearl on expressing
high-performance optimizations as rewrite strategies. Proceed-
ings of the ACM on Programming Languages , 4(ICFP):1–29,
2020.
[20] Bastian Hagedorn, Larisa Stoltzfus, Michel Steuwer, Sergei Gor-
latch, and Christophe Dubach. High performance stencil code
generation with Lift. In International Symposium on Code Gen-
eration and Optimization (CGO), 2018.
[21] Ameer Haj-Ali, Hasan Genc, Qijing Huang, William Moses,
John Wawrzynek, Krste Asanovi ´c, and Ion Stoica. Protuner:
tuning programs with monte carlo tree search. arXiv preprint
arXiv:2005.13685, 2020.
[22] Ameer Haj-Ali, Qijing Jenny Huang, John Xiang, William Moses,
Krste Asanovic, John Wawrzynek, and Ion Stoica. Autophase:
Juggling hls phase orderings in random forests with deep re-
inforcement learning. Proceedings of Machine Learning and
Systems, 2:70–81, 2020.
[23] Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Se-
quential model-based optimization for general algorithm conﬁgu-
ration. In International conference on learning and intelligent
optimization (LION), 2011.
[24] Muhammad Huzaifa, Rishi Desai, Samuel Grayson, Xutao Jiang,
Ying Jing, Jae Lee, Fang Lu, Yihan Pang, Joseph Ravichandran,
Finn Sinclair, et al. ILLIXR: Enabling end-to-end extended
reality research. In 2021 IEEE International Symposium on
Workload Characterization (IISWC), pages 24–38. IEEE, 2021.
[25] Donald R Jones, Matthias Schonlau, and William J Welch. Efﬁ-
cient global optimization of expensive black-box functions. Jour-
nal of Global optimization, 13(4):455–492, 1998.
[26] Fredrik Kjolstad, Shoaib Kamil, Stephen Chou, David Lugato,
and Saman Amarasinghe. The tensor algebra compiler. Pro-
ceedings of the ACM on Programming Languages, 1(OOPSLA),
2017.
[27] Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and
Frank Hutter. Fast bayesian optimization of machine learning
hyperparameters on large datasets. In Artiﬁcial intelligence and
statistics, pages 528–536. PMLR, 2017.
[28] Nicolas Knudde, Joachim van der Herten, Tom Dhaene, and Ivo
Couckuyt. GPﬂowOpt: A Bayesian Optimization Library using
TensorFlow. arXiv preprint – arXiv:1711.03845, 2017.
[29] Thomas Koehler and Michel Steuwer. Towards a domain-
extensible compiler: optimizing an image processing pipeline on
mobile cpus. In International Symposium on Code Generation
and Optimization (CGO), 2021.
[30] David Koeplinger, Matthew Feldman, Raghu Prabhakar, Yaqi
Zhang, Stefan Hadjis, Ruben Fiszel, Tian Zhao, Luigi Nardi,
Ardavan Pedram, Christos Kozyrakis, et al. Spatial: A language
and compiler for application accelerators. In Proceedings of
the 39th ACM SIGPLAN Conference on Programming Language
Design and Implementation, pages 296–311, 2018.
[31] Scott P Kolodziej, Mohsen Aznaveh, Matthew Bullock, Jarrett
David, Timothy A Davis, Matthew Henderson, Yifan Hu, and
Read Sandstrom. The suitesparse matrix collection website inter-
face. Journal of Open Source Software, 4(35):1244, 2019.
[32] Jaehoon Koo, Prasanna Balaprakash, Michael Kruse, Xingfu Wu,
Paul Hovland, and Mary Hall. Customized monte carlo tree
search for llvm/polly’s composable loop optimization transfor-
mations. In International Workshop on Performance Modeling,
Benchmarking and Simulation of High Performance Computer
Systems (PMBS), 2021.
[33] Maria Kotsifakou, Prakalp Srivastava, Matthew D Sinclair,
Rakesh Komuravelli, Vikram Adve, and Sarita Adve. Hpvm:
Heterogeneous parallel virtual machine. In Proceedings of the
23rd ACM SIGPLAN Symposium on Principles and Practice of
Parallel Programming, pages 68–80, 2018.
[34] Michael Kruse, Hal Finkel, and Xingfu Wu. Autotuning search
space for loop transformations. In IEEE/ACM Workshop on
the LLVM Compiler Infrastructure in HPC (LLVM-HPC) and
Workshop on Hierarchical Parallelism for Exascale Computing
(HiPar), 2020.
[35] M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp, J. Mar-
ben, P. Müller, and F. Hutter. Boah: A tool suite for multi-
ﬁdelity bayesian optimization & analysis of hyperparameters.
arXiv:1908.06756 [cs.LG].
15
[36] Marius Lindauer, Katharina Eggensperger, Matthias Feurer, An-
dré Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhkopf,
René Sass, and Frank Hutter. SMAC3: A versatile bayesian
optimization package for hyperparameter optimization. Journal
of Machine Learning Research, 23(54):1–9, 2022.
[37] Dong C Liu and Jorge Nocedal. On the limited memory BFGS
method for large scale optimization. Mathematical Programming,
45(1):503–528, 1989.
[38] Yang Liu, Wissam M Sid-Lakhdar, Osni Marques, Xinran Zhu,
Chang Meng, James W Demmel, and Xiaoye S Li. GPTune: mul-
titask learning for autotuning exascale applications. In Principles
and Practice of Parallel Programming (PPoPP), 2021.
[39] Maria Lomeli, Mark Rowland, Arthur Gretton, and Zoubin
Ghahramani. Antithetic and monte carlo kernel estimators for
partial rankings. Statistics and Computing , 29(5):1127–1147,
2019.
[40] Kevin P Murphy. Machine learning: a probabilistic perspective.
MIT press, 2012.
[41] Luigi Nardi, David Koeplinger, and Kunle Olukotun. Practical
design space exploration. In Modeling, Analysis, and Simulation
of Computer and Telecommunication Systems (MASCOTS), 2019.
[42] Thomas Nelson, Axel Rivera, Prasanna Balaprakash, Mary Hall,
Paul D Hovland, Elizabeth Jessup, and Boyana Norris. Gen-
erating efﬁcient tensor contractions for gpus. In International
Conference on Parallel Processing (ICPP), 2015.
[43] Filip Petrovi ˇc, David St ˇrelák, Jana Hozzová, Jaroslav Ol’ha,
Richard Trembeck`y, Siegfried Benkner, and Ji ˇrí Filipoviˇc. A
benchmark set of highly-efﬁcient cuda and opencl kernels and its
dynamic autotuning with kernel tuning toolkit.Future Generation
Computer Systems, 108:161–177, 2020.
[44] Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Syl-
vain Paris, Frédo Durand, and Saman Amarasinghe. Halide: a
language and compiler for optimizing parallelism, locality, and
recomputation in image processing pipelines. ACM Sigplan No-
tices, 48(6):519–530, 2013.
[45] Ari Rasch, Michael Haidl, and Sergei Gorlatch. Atf: A generic
auto-tuning framework. In 2017 IEEE 19th International Con-
ference on High Performance Computing and Communications;
IEEE 15th International Conference on Smart City; IEEE 3rd
International Conference on Data Science and Systems (HPC-
C/SmartCity/DSS), pages 64–71. IEEE, 2017.
[46] Ari Rasch, Richard Schulze, Michel Steuwer, and Sergei Gor-
latch. Efﬁcient auto-tuning of parallel programs with interdepen-
dent tuning parameters via auto-tuning framework (ATF). ACM
Transactions on Architecture and Code Optimization, 18(1):1:1–
1:26, 2021.
[47] Carl Edward Rasmussen and Christopher K. I. Williams. Gaus-
sian processes for machine learning. MIT press, 2006.
[48] Rohan Basu Roy, Tirthak Patel, Vijay Gadepally, and Devesh
Tiwari. Bliss: auto-tuning complex applications using a pool of
diverse lightweight learning models. In Programming Language
Design and Implementation (PLDI), 2021.
[49] Ryan Senanayake, Changwan Hong, Ziheng Wang, Amalee Wil-
son, Stephen Chou, Shoaib Kamil, Saman Amarasinghe, and
Fredrik Kjolstad. A sparse iteration space transformation frame-
work for sparse tensor algebra. Proceedings of the ACM on
Programming Languages, 4(OOPSLA), 2020.
[50] Wissam M Sid-Lakhdar, Mohsen Mahmoudi Aznaveh, Xiaoye S
Li, and James W Demmel. Multitask and transfer learning for au-
totuning exascale applications. arXiv preprint arXiv:1908.05792,
2019.
[51] Shaden Smith, Jee W. Choi, Jiajia Li, Richard Vuduc, Jongsoo
Park, Xing Liu, and George Karypis. FROSTT: The formidable
repository of open sparse tensors and tools, 2017.
[52] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical
bayesian optimization of machine learning algorithms. 2012.
[53] Artur Souza, Luigi Nardi, Leonardo B Oliveira, Kunle Oluko-
tun, Marius Lindauer, and Frank Hutter. Bayesian optimization
with a prior for the optimum. In Joint European Conference
on Machine Learning and Knowledge Discovery in Databases.
Springer, 2021.
[54] Michel Steuwer, Christian Fensch, Sam Lindley, and Christophe
Dubach. Generating performance portable code using
rewrite rules: from high-level functional expressions to high-
performance opencl code. ACM SIGPLAN Notices, 50(9):205–
217, 2015.
[55] Michel Steuwer, Thomas Koehler, Bastian Köpcke, and Federico
Pizzuti. RISE & shine: Language-oriented compiler design.
CoRR, abs/2201.03611, 2022.
[56] Michel Steuwer, Toomas Remmelg, and Christophe Dubach.
Lift: a functional data-parallel ir for high-performance gpu code
generation. In Code Generation and Optimization (CGO), 2017.
[57] Larisa Stoltzfus, Bastian Hagedorn, Michel Steuwer, Sergei Gor-
latch, and Christophe Dubach. Tiling optimizations for stencil
computations using rewrite rules in lift. ACM Trans. Archit. Code
Optim., 16(4), dec 2019.
[58] Hakki Mert Torun, Madhavan Swaminathan, Anto Kavungal
Davis, and Mohamed Lamine Faycal Bellaredj. A global bayesian
optimization algorithm and its application to integrated system
design. IEEE Transactions on Very Large Scale Integration
(VLSI) Systems, 26(4):792–802, 2018.
[59] Bimal Viswanath, Alan Mislove, Meeyoung Cha, and Krishna P
Gummadi. On the evolution of user interaction in facebook. In
Proceedings of the 2nd ACM workshop on Online social networks,
pages 37–42, 2009.
[60] Jie Wang, Licheng Guo, and Jason Cong. AutoSA: a polyhedral
compiler for high-performance systolic arrays on FPGA. In
SIGDA International Symposium on Field-Programmable Gate
Arrays. ACM, 2021.
[61] Floris-Jan Willemsen, Rob van Nieuwpoort, and Ben van
Werkhoven. Bayesian optimization for auto-tuning gpu kernels.
In International Workshop on Performance Modeling, Bench-
marking and Simulation of High Performance Computer Systems
(PMBS), 2021.
[62] Nan Wu, Yuan Xie, and Cong Hao. Ironman: Gnn-assisted
design space exploration in high-level synthesis via reinforcement
learning. In Great Lakes Symposium on VLSI, 2021.
[63] Xingfu Wu, Michael Kruse, Prasanna Balaprakash, Hal Finkel,
Paul Hovland, Valerie Taylor, and Mary Hall. Autotuning Poly-
Bench benchmarks with LLVM Clang/Polly loop optimization
pragmas using bayesian optimization (extended version). arXiv
preprint arXiv:2104.13242, 2021.
[64] Guanwen Zhong, Alok Prakash, Yun Liang, Tulika Mitra, and
Smail Niar. Lin-Analyzer: A high-level performance analysis
tool for fpga-based accelerators. In ACM/EDAC/IEEE Design
Automation Conference (DAC), 2016.
[65] Xinran Zhu, Yang Liu, Pieter Ghysels, David Bindel, and Xi-
aoye S Li. GPTuneBand: Multi-task and multi-ﬁdelity autotuning
for large-scale high performance computing applications. In Par-
allel Processing for Scientiﬁc Computing, 2022.
16
A. Additional Results
In this Appendix, we present the results shown in Sec. 5.3 in greater detail.
Fig. 11 shows the remaining benchmarks not in Fig. 7. The takeaway is similar, that BaCO ﬁnds the expert-level-
performing conﬁgurations much faster than the other benchmarks, and that it consistently ﬁnds better conﬁgurations
than the other benchmarks after only a few iterations after the learning phase starts. However, the additional benchmarks
indicates that it generalizes well.
Table 5 shows for each autotuning framework and benchmark how many individual autotuning runs with a full budget
managed to reach the expert performance. The results show that BaCO reaches the expert-level performance in 575 out
of 750 runs (76%).
Tables 6, 7, and 8 show the relative performance achieved compared to an expert with the tiny, small, and full budgets.
Table 9 presents the factors showing how much faster BaCO reaches the best performance of the other methods. On
average, BaCO reaches ATF’s performance 2.87×faster and Ytopt’s performance 3.87×faster.
17
1000
2000
RISE/ELEVATEK-means GPU
0 20 40 60
25
50
75
2.5
5.0
TACO SpMV laminar duct3D
0 20 40 60
0.5
1.0
200
400
TACO SpMM cage12
0 20 40 60
50
100
150
2500
5000
TACO MTTKRP uber
0 20 40 60
500
1000
1500
2000
4000
TACO SDDMM email-Enron
0 20 40 60
300
400
25
50
RISE/ELEVATEHarris GPU
0 20 40 60 80 100
0.5
1.0
1.5
50
75
TACO SpMV random
0 20 40 60
10
20
30
200
400
TACO SpMM laminar duct3D
0 20 40 60
50
100
2500
5000
TACO MTTKRP nips
0 20 40 60
500
1000
1500
500
1000
TACO SDDMM ACTIVSg10K
0 20 40 60
60
80
100
100
200
RISE/ELEVATEStencil GPU
0 20 40 60
7.5
10.0
12.5
500
1000
RISE/ELEVATEScal GPU
0 20 40 60
0
50
1.0
1.5
TACO TTV uber3
0 20 40 60
0.2
0.4
0.6
5000
10000
TACO MTTKRP chicago
0 20 40 60
500
1000
1500
500
1000
TACO SDDMM Goodwin 040
0 20 40 60
60
80
BaCO
ATF with OpenTuner
Ytopt
Uniform Sampling
CoT Sampling
Expert Conﬁguration
Default Conﬁguration
DoE
Number of Conﬁguration Evaluations
Runtime [ms]
Evaluation of Average Best Runtime (Lower is better)
Figure 11: Evolution of average best runtime among evaluated conﬁgurations for all benchmarks in Table 3 that are not
shown in Fig. 7. The ﬁgure is split vertically into two different scales and the iteration where each method beats the
expert conﬁguration is marked with a star.
18
Framework Benchmark BO ATF Ytopt Uniform CoT
TACO (SpMM) scircuit 30 24 16 14 12
TACO (SpMM) cage12 30 22 8 12 13
TACO (SpMM) laminarduct3D 30 18 6 6 9
TACO (SDDMM) email-Enron 30 25 17 10 18
TACO (SDDMM) ACTIVSg10K 30 28 19 11 14
TACO (SDDMM) Goodwin040 30 24 18 10 12
TACO (MTTKRP) uber 24 21 4 3 6
TACO (MTTKRP) nips 24 20 9 7 15
TACO (MTTKRP) chicago 21 22 14 0 0
TACO (TTV) facebook 30 27 16 22 20
TACO (TTV) uber3 16 14 3 3 3
TACO (TTV) random 18 17 16 17 17
TACO (SpMV) laminarduct3D 13 14 1 1 3
TACO (SpMV) cage12 13 27 8 9 8
TACO (SpMV) ﬁlter3D 9 27 3 4 2
TACO 348 330 158 129 152
RISE & ELEV ATE MM-CPU 22 15 6 2 1
RISE & ELEV ATE MM-GPU 19 10 1 1 0
RISE & ELEV ATE Asum-GPU 29 9 5 3 3
RISE & ELEV ATE Scal-GPU 21 1 0 2 1
RISE & ELEV ATE K-means-GPU 25 15 3 5 6
RISE & ELEV ATE Stencil-GPU 18 4 8 0 0
RISE & ELEV ATE 149 68 23 23 13
HPVM2FPGA BFS 29 0 4 2 7
HPVM2FPGA Audio 23 4 1 0 0
HPVM2FPGA PreEuler 26 6 4 6 4
HPVM2FPGA 78 10 9 8 11
Table 5: Out of 30 autotuning runs with the full budget, how many reached expert-level performance.
19
Framework Benchmark BaCO ATF Ytopt Uniform CoT
TACO (SpMM) scircuit 1.16 0.67 0.58 0.59 0.61
TACO (SpMM) cage12 1.14 0.74 0.53 0.70 0.72
TACO (SpMM) laminarduct3D 0.96 0.34 0.35 0.38 0.35
TACO (SDDMM) email-Enron 1.05 0.91 0.78 0.86 0.91
TACO (SDDMM) ACTIVSg10K 1.02 0.90 0.83 0.88 0.89
TACO (SDDMM) Goodwin040 1.04 0.88 0.81 0.79 0.86
TACO (MTTKRP) uber 0.30 0.20 0.16 0.17 0.33
TACO (MTTKRP) nips 0.67 0.64 0.50 0.49 0.75
TACO (MTTKRP) chicago 0.76 0.58 0.61 0.39 0.60
TACO (TTV) facebook 0.70 0.25 0.33 0.52 0.47
TACO (TTV) uber3 0.73 0.38 0.46 0.51 0.38
TACO (TTV) random 1.30 0.86 0.35 0.92 1.26
TACO (SpMV) laminarduct3D 0.60 0.35 0.42 0.43 0.36
TACO (SpMV) cage12 0.53 0.85 0.33 0.30 0.39
TACO 0.83 0.63 0.49 0.54 0.61
RISE & ELEV ATE MM-CPU 0.89 0.81 0.79 0.78 0.78
RISE & ELEV ATE MM-GPU 0.28 0.17 0.28 0.25 0.14
RISE & ELEV ATE Asum-GPU 0.59 0.35 0.34 0.29 0.35
RISE & ELEV ATE Scal-GPU 0.34 0.06 0.12 0.11 0.06
RISE & ELEV ATE K-means-GPU 0.82 0.61 0.22 0.35 0.53
RISE & ELEV ATE Stencil-GPU 0.88 0.78 0.72 0.73 0.73
RISE & ELEV ATE 0.65 0.48 0.41 0.46 0.44
HPVM2FPGA BFS 0.48 0.18 0.60 0.28 0.29
HPVM2FPGA Audio 0.93 0.71 0.83 0.64 0.70
HPVM2FPGA PreEuler 0.46 0.36 0.32 0.37 0.38
HPVM2FPGA 0.62 0.42 0.58 0.43 0.46
All 0.76 0.56 0.49 0.50 0.55
Table 6: Relative performance compared to expert with the tiny budget (1/3 of the full budget). Values larger than 1
indicate a performance advantage over the expert. Values below 1 indicate a performance disadvantage compared with
the expert.
20
Framework Benchmark BaCO ATF Ytopt Uniform CoT
TACO (SpMM) scircuit 1.25 0.96 0.77 0.78 0.77
TACO (SpMM) cage12 1.35 0.99 0.80 0.84 0.82
TACO (SpMM) laminarduct3D 1.16 0.60 0.45 0.50 0.51
TACO (SDDMM) email-Enron 1.08 1.01 0.91 0.92 0.98
TACO (SDDMM) ACTIVSg10K 1.04 1.01 0.91 0.94 0.93
TACO (SDDMM) Goodwin040 1.08 1.00 0.90 0.91 0.95
TACO (MTTKRP) uber 0.42 0.38 0.24 0.27 0.43
TACO (MTTKRP) nips 0.94 0.92 0.68 0.68 0.85
TACO (MTTKRP) chicago 0.90 1.00 0.74 0.54 0.72
TACO (TTV) facebook 2.13 0.75 0.53 0.93 0.87
TACO (TTV) uber3 1.00 0.69 0.52 0.67 0.55
TACO (TTV) random 6.29 1.42 0.84 1.83 2.40
TACO (SpMV) laminarduct3D 0.88 0.74 0.48 0.56 0.52
TACO (SpMV) cage12 0.65 1.17 0.44 0.48 0.54
TACO (SpMV) ﬁlter3D 0.66 1.28 0.42 0.34 0.43
TACO 1.39 0.93 0.64 0.75 0.82
RISE & ELEV ATE MM-CPU 0.98 0.90 0.85 0.81 0.82
RISE & ELEV ATE MM-GPU 0.84 0.38 0.40 0.36 0.19
RISE & ELEV ATE Asum-GPU 1.20 0.58 0.38 0.41 0.49
RISE & ELEV ATE Scal-GPU 0.88 0.09 0.19 0.15 0.09
RISE & ELEV ATE K-means-GPU 1.00 0.74 0.34 0.44 0.65
RISE & ELEV ATE Stencil-GPU 0.94 0.86 0.76 0.81 0.80
RISE & ELEV ATE Harris-GPU 0.95 0.79 0.80 0.61
RISE & ELEV ATE 0.97 0.62 0.49 0.54 0.52
HPVM2FPGA BFS 1.00 0.34 0.70 0.56 0.50
HPVM2FPGA Audio 1.00 0.90 0.91 0.86 0.83
HPVM2FPGA PreEuler 0.78 0.45 0.36 0.50 0.48
HPVM2FPGA 0.92 0.57 0.66 0.64 0.61
All 1.22 0.80 0.61 0.67 0.71
Table 7: Relative performance compared to expert with the small budget (2/3 of the full budget). Values larger than 1
indicate a performance advantage over the expert. Values below 1 indicate a performance disadvantage compared with
the expert.
21
Framework Benchmark BaCO ATF Ytopt Uniform CoT
TACO (SpMM) scircuit 1.31 1.12 0.85 0.90 0.91
TACO (SpMM) cage12 1.37 1.08 0.82 0.90 0.92
TACO (SpMM) laminarduct3D 1.24 0.77 0.48 0.52 0.63
TACO (SDDMM) email-Enron 1.08 1.04 0.96 0.95 1.00
TACO (SDDMM) ACTIVSg10K 1.04 1.03 0.98 0.96 0.99
TACO (SDDMM) Goodwin040 1.09 1.05 0.96 0.95 0.98
TACO (MTTKRP) uber 1.00 0.64 0.24 0.38 0.54
TACO (MTTKRP) nips 1.00 1.00 0.81 0.78 0.91
TACO (MTTKRP) chicago 1.00 1.05 0.88 0.61 0.80
TACO (TTV) facebook 2.52 1.90 0.65 1.25 1.16
TACO (TTV) uber3 1.00 0.75 0.61 0.69 0.62
TACO (TTV) random 8.82 3.32 1.81 2.32 2.90
TACO (SpMV) laminarduct3D 0.98 0.91 0.54 0.60 0.68
TACO (SpMV) cage12 0.77 1.48 0.47 0.63 0.62
TACO 1.66 1.24 0.77 0.86 0.95
RISE & ELEV ATE MM-CPU 1.02 0.93 0.88 0.83 0.83
RISE & ELEV ATE MM-GPU 1.05 0.49 0.49 0.44 0.29
RISE & ELEV ATE Asum-GPU 1.25 0.67 0.43 0.50 0.58
RISE & ELEV ATE Scal-GPU 1.08 0.14 0.23 0.23 0.14
RISE & ELEV ATE K-means-GPU 1.02 0.84 0.40 0.54 0.79
RISE & ELEV ATE Stencil-GPU 0.98 0.91 0.85 0.84 0.82
RISE & ELEV ATE 1.06 0.69 0.55 0.61 0.59
HPVM2FPGA BFS 1.00 0.45 0.74 0.69 0.67
HPVM2FPGA Audio 1.00 0.94 0.94 0.90 0.91
HPVM2FPGA PreEuler 1.00 0.50 0.51 0.57 0.56
HPVM2FPGA 1.00 0.63 0.73 0.72 0.71
All 1.41 1.01 0.71 0.76 0.83
Table 8: Relative performance compared to expert with the full budget. Values larger than 1 indicate a performance
advantage over the expert. Values below 1 indicate a performance disadvantage compared with the expert.
22
Framework Benchmark ATF Ytopt Uniform CoT
TACO (SpMM) scircuit 3.33× 5.00× 4.62× 4.62×
TACO (SpMM) cage12 3.53× 6.00× 5.45× 5.00×
TACO (SpMM) laminarduct3D 4.00× 5.45× 5.00× 4.29×
TACO (SDDMM) email-Enron 3.75× 4.62× 4.62× 4.29×
TACO (SDDMM) ACTIVSg10K 1.88× 4.62× 5.45× 3.75×
TACO (SDDMM) Goodwin040 2.40× 4.29× 4.62× 4.00×
TACO (MTTKRP) uber 1.40× 4.62× 1.88× 1.43×
TACO (MTTKRP) nips 1.03× 2.31× 2.40× 1.58×
TACO (MTTKRP) chicago - 2.14 × 3.53× 2.86×
TACO (TTV) facebook 1.76× 3.00× 2.40× 2.50×
TACO (TTV) uber3 2.86× 3.53× 3.16× 3.33×
TACO (TTV) random 1.67× 2.07× 2.07× 1.67×
TACO (SpMV) laminarduct3D 1.09× 3.33× 2.86× 2.73×
TACO (SpMV) cage12 - 4.00 × 1.76× 1.76×
TACO 3.15× 5.00× 4.96× 4.32×
RISE & ELEV ATE MM-CPU 2.22× 3.03× 3.85× 3.85×
RISE & ELEV ATE MM-GPU 2.03× 1.97× 2.18× 2.93×
RISE & ELEV ATE Asum-GPU 2.73× 3.53× 3.33× 3.00×
RISE & ELEV ATE Scal-GPU 4.62× 3.75× 3.75× 4.62×
RISE & ELEV ATE K-means-GPU 2.73× 6.00× 5.00× 3.16×
RISE & ELEV ATE Stencil-GPU 2.50× 3.53× 3.53× 3.75×
RISE & ELEV ATE 2.68× 3.38× 3.51× 3.55×
HPVM2FPGA BFS 2.86× 2.00× 2.22× 2.22×
HPVM2FPGA Audio 2.86× 2.86× 3.16× 3.00×
HPMV2FPGA PreEuler 2.61× 2.61× 2.00× 2.07×
HPVM2FPGA 2.77× 2.49× 2.46× 2.43×
all 2.87× 3.82× 3.86× 3.63×
Table 9: Factors showing how much faster BaCO reach the best performance of the other methods. E.g., a factor of
3.33×indicates that BaCO required 3.33 ×less evaluations to achieve the same performance. ”-” indicates that the
average ﬁnal performance of BaCO was lower than ATF.
23
B. Wall Clock Time Analysis
In Table 10, we show the average wall clock time of BaCO and the baselines for the TACO SpMM and SDDMM
benchmarks. The breakdown of the wall clock time is given by two main factors: 1) the evaluations of the black-box
function, and 2) the computing time for the autotuner to generate its recommendations. The most expensive part of the
autotuning process is the ﬁrst factor, where the program kernel is evaluated a number of times corresponding to the
budget allocated. Thus, methods that recommend slow-to-evaluate conﬁgurations tend to use more wall clock time per
evaluation. The second factor relates to the internal workings of an autotuner search method. In this regard, the more
intricate model-based methods tend to be slower, but this cost becomes less prominent when optimizing larger program
kernels. This factor is highly dependent on the quality of the software implementation of the autotuner. It is beyond
the scope of this work to provide the fastest implementation of both BaCO and the baselines — In Table 10 we give a
rough analysis of the current average wall clock time for all autotuners, which will likely be improved in future releases
of these tools. We observe that while BaCO uses a more complex method than the baselines it is the second fastest
method behind ATF. ATF uses OpenTuner which employs search algorithms based on heuristics — These are usually
faster than model-based approaches such as BaCO. We can see this insight reﬂected also in the Ytopt wall clock time.
BaCO ATF w. Ytopt Uniform CoT
OpenTuner sampling samp.
SpMM 262 144 309 402 336
SDDMM 263 197 274 433 381
Table 10: Average wall clock time in seconds for BaCO and the baselines on the TACO SpMM and SDDMM benchmarks.
24

Minimizing the Cost of Iterative
Compilation with Active Learning
William F. Ogilvie
University of Edinburgh, UK
s0198982@sms.ed.ac.uk
Pavlos Petoumenos
University of Edinburgh, UK
ppetoume@inf.ed.ac.uk
Zheng Wang
Lancaster University, UK
z.wang@lancaster.ac.uk
Hugh Leather
University of Edinburgh, UK
hleather@inf.ed.ac.uk
Abstract
Since performance is not portable between platforms, en-
gineers must ﬁne-tune heuristics for each processor in turn.
This is such a laborious task that high-proﬁle compilers, sup-
porting many architectures, cannot keep up with hardware
innovation and are actually out-of-date. Iterative compilation
driven by machine learning has been shown to be efﬁcient
at generating portable optimization models automatically.
However, good quality models require costly, repetitive, and
extensive training which greatly hinders the wide adoption of
this powerful technique.
In this work, we show that much of this cost is spent
collecting training data, runtime measurements for differ-
ent optimization decisions, which contribute little to the ﬁ-
nal heuristic. Current implementations evaluate randomly
chosen, often redundant, training examples a pre-conﬁgured,
almost always excessive, number of times – a large source
of wasted effort. Our approach optimizes not only the selec-
tion of training examples but also the number of samples
per example, independently. To evaluate, we construct 11
high-quality models which use a combination of optimization
settings to predict the runtime of benchmarks from the SPAPT
suite. Our novel, broadly applicable, methodology is able to
reduce the training overhead by up to 26x compared to an
approach with a ﬁxed number of sample runs, transforming
what is potentially months of work into days.
Categories and Subject Descriptors D.3.4 [Programming
Languages]: Processors—compilers, optimization
Keywords Active Learning; Compilers; Iterative Compila-
tion; Machine Learning; Sequential Analysis;
1. Introduction
With Dennard scaling long dead and technology scaling stop-
ping within the next ﬁve years [1], performance improve-
ments rely increasingly on software improvements, especially
compiler and runtime optimizations. Accurate heuristics for
deciding the best way to optimize a program are hard to con-
struct. The space of possible decisions is vast, while their
effect on performance is complex and depends on both the
application and the targeted hardware. Manually develop-
ing such heuristics takes months or years, even for a single
target, meaning that tuning the compiler and runtime heur-
istics for each new released processor is unrealistic. Even
high-quality production compilers are out-of-date [31] and
cannot extract the full performance potential of the hardware.
The industry-wide trend towards heterogeneity only serves
to make the optimization decision space even more complex,
making effective heuristics near impossible to construct [44].
To overcome this problem, iterative compilation [30] was
proposed as a means by which heuristics can be automatically
produced, without the need for expert involvement. Different
optimization strategies are applied and the effect on speed,
size, or energy is measured. Augmented with machine learn-
ing [2], these data are used to train a predictor which selects
the best optimizations for a particular code and platform. This
approach not only produces heuristics much faster, but it also
outperforms heuristics crafted by human experts [31, 57].
The same methodology can be applied to radically differ-
ent problem domains: compilation [33, 55, 56], parallelism
mapping [22], runtime tuning [15], and hardware-software
co-design [60].
The time required to create these heuristics, while auto-
mated, is still substantial. Researchers have improved upon
this work by removing its reliance on random search and
used active learninginstead [4, 5, 39, 60]. Random search is
problematic because it selects optimization decisions and pro-
ﬁles the application multiple times under those optimizations
before it even knows whether this will actually improve our
knowledge of the decision space. In contrast, active learning
is a methodology which predicts the parts of the decision
978-1-5090-4931-8/17 c⃝ 2017 IEEE CGO 2017, Austin, USA
Accepted for publication by IEEE.c⃝ 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/
republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
245
space where as much information as possible can be gained
and directs the search towards them.
These works represent a substantial leap forward towards
making iterative compilation quick and easy. However, size-
able inefﬁciencies still exist. Previous work on iterative
compilation used a ﬁxed sampling plan: each unique train-
ing instance is repeatedly proﬁled a set number of times,
chosen a priori. Repeated measurements are necessary be-
cause runtime measurements are inherently noisy.
There are many sources of noise encountered for runtime
measurements. The most egregious of which is caused by
other user or system processes. Such processes compete for
resources with our application, especially cores, caches [41],
and memory [59], and they do so in non-deterministic ways.
In recent systems, the power and thermal walls lead to
more complex interference patterns. Intel’s Turbo Boost,
for example, might lower the frequency and the power
consumption of a process running on a core, when other
cores wake up [9].
Even ignoring interference from other applications, there
are still more sources of noise. Memory management mech-
anisms, such as dynamic memory allocators [26] and garbage
collectors [48], can introduce additional unpredictable over-
heads. On top of this, address space layout randomization and
the physical page allocation mechanism change the logical
and physical memory layout of the application every time it is
executed, potentially affecting the number of conﬂict misses
in the CPU caches and branch mispredictions [17, 38]. Multi-
threaded applications can even force non-deterministic beha-
vior on themselves, if the scheduler is not set to be perfectly
repeatable, or if small timing changes alter the communica-
tion patterns [45]. Any I/O can have non-repeatable timings,
and even changes to the environment variables between runs
can shift memory and alter runtimes [38].
Past research has investigated ways to reduce experimental
noise. Typical approaches include overriding the default
scheduling policy [43, 45], using more deterministic memory
management mechanisms [26, 43, 45], avoiding I/O, or
just minimizing the number of active processes, including
services and daemons. Doing so is not always enough or
desirable because of the following reasons. First, while they
do reduce noise, they do not eliminate it. Multiple proﬁling
runs are still needed to determine whether noise affects the
measurement signiﬁcantly. Even then the amount of variation
might be too high for optimization heuristics dependent
on accurate measurements [32]. Secondly, modiﬁcations
to reduce noise may do so at the expense of altering the
runtime behaviour that is meant to be measured or of risking
the wrong heuristic being learned. Heuristics targeting very
speciﬁc, low-noise runtime environments may not match well
when used in practice. For example, [16] showed that the
runtime variation caused by memory layout changes, such
as address space randomization, can dwarf the differences
between optimizations. If address space randomization is
disabled during training or only a single run is taken, then
an optimization could be selected which is not optimal on
average in deployment. Instead, multiple runs must be used
to smooth out the effects of random layout changes. Finally,
even when a low-noise environment would not actually
alter the heuristic, we have found it difﬁcult to convince
companies that tuning heuristics to an environment different
than their production one is acceptable. In short, what is
needed are techniques which perform well in realistic, noisy
environments.
Our work aims exactly at handling noise without having
to reduce it and without wasting time on large numbers of re-
peated performance measurements. Our insight was that each
additional observation, that is, each additional performance
measurement for the same optimization strategy, provides
diminishing amounts of information. Indeed, that extra in-
formation quickly reaches zero if there is little experimental
noise or if the observation ﬁts well with what we already
know about the decision space. In other words, extra proﬁling
runs for a decision are useful only if they are likely to con-
tradict what we predict about that decision. Our experiments
conﬁrm that iterative compilation can be slowed down by
using a ﬁxed sampling plan, spending most of its time getting
observations which provide no additional information.
In this paper, we introduce a novel active learning tech-
nique for iterative compilation which combines sequential
analysis, an approach where the number of samples are not
ﬁxed. By proﬁling the application under the same optimiz-
ation decision only as long as this improves our knowledge
of the decision space, we produce models quickly without
sacriﬁcing the heuristic’s quality. Speciﬁcally, our technique
begins by taking a single sample runtime for optimizations
that are deemed to be most proﬁtable to learn from, as deﬁned
by an active learner. As knowledge is built up, the algorithm
is able to revisit these examples instead of getting new ones.
This happens if it determines that they are of continued in-
terest, that is if it appears that measurement noise has affected
the data we previously collected on that conﬁguration.
To evaluate our approach we create predictors for 11
programs from the SPAPT suite [3]. These models can predict,
with low error, the runtime of a particular code given a
number of optimization options that we may want to apply,
and in this way can ﬁnd an optimal combination. Our results
show that we can create a high-quality heuristic on average
4x, and up to 26x, faster than a baseline approach which uses
35 samples per run.
2. Motivation
As previously stated, the research in this paper is based on
the realization that current procedures for creating machine
learning based heuristics do not consider sample size a para-
meter for optimization, but rather assume it to be a constant
value ﬁxed a priori. Moreover, little or no justiﬁcation is
ever provided for one chosen sample size over another. With
246
0
10
20
30
0 10 20 30
Loop i1 Unroll Factor
Loop i2 Unroll Factor
 1234MAE (ms)
mm compiled with −O21 sample per point
(a) mean absolute error for sample size of one
0
10
20
30
0 10 20 30
Loop i1 Unroll Factor
Loop i2 Unroll Factor
 1234MAE (ms)
mm compiled with −O2optimum samples per point
(b) mean absoulute error for optimal sample size
0
10
20
30
0 10 20 30
Loop i1 Unroll Factor
Loop i2 Unroll Factor
 0102030# Samples
mm compiled with −O2optimum samples per point
(c) optimal number of samples across the space
Figure 1. Error and sample size for each point in the decision space when using a sample size of one or the optimal sample size.
The decision is the unroll factor for two loops of themm kernel of SPAPT. For most but not all points, a single sample is enough.
active, iterative learning this need not be the case and we can
leverage the knowledge built up by the algorithm over time to
adaptively select a more appropriate sample size per example,
signiﬁcantly speeding-up training overall.
To motivate our work, we examined the way iterative
compilation works on the problem of tuning the unroll
factor for two loops,i1 and i2, of the matrix multiplication
kernel in the SPAPT suite on the system we describe in
Section 4. Using the-O2 optimization level as a baseline, we
compiled the kernel multiple times, each one with a different
combination of unroll factors for the two loops. Each binary
was then executed 35 times and its runtime measurements
recorded.
In Figure 1a, we present the Mean Absolute Error (MAE)
we would have incurred had we only taken a single observa-
tion. This gives us an estimated baseline for the worst error
we could pay in this space, as high as 4ms (5% of the mean)
for some binaries but practically zero for many. For the latter
getting even a second sample is a waste of effort. To estim-
ate the potential speed-up we could obtain if we knew how
many samples we should actually take for each optimization
setting we iterate through the space again, but at each point
we remove samples randomly from the group of 35 samples
we collected initially. We continue reducing the number of
samples as long as our calculated MAE is below 0.1ms.
Figures 1b and 1c show the error of this adaptive ap-
proach across the space and the number of samples needed
per conﬁguration to maintain such a small error, respectively.
These ﬁgures demonstrate that there is quite considerable
stochastic noise in measurements across the space and, there-
fore, that the number of samples needed for a low MAE
varies. If we take the naïve, ﬁxed sampling plan of 35, we
need 35 ⇥ 30 ⇥ 30 = 31, 500 individual executions, whereas
with ‘perfect knowledge’ we can incur an error of only 0.1ms
at the cost of15, 131 program runs –nearly half.
●
●
●
●
●
●
● ●● ● ●● ●
0 5 10 15 20 25 301.5 2.0 2.5 3.0
adi compiled with −O2
Loop i1 Unroll Factor
Runtime (s)
Figure 2. Runtime versus unroll factor for a loop of adi,
when using a sample size of one. A relationship between
unroll factor and runtime is relatively clear despite the noise –
i.e. stable around 2.1s until 10 where it climbs steadily and
plateaus at 3.1s for high levels of loop unrolling.
This example starts with ‘perfect’ information about each
point in the decision space and it removes samples until the
average runtime starts to deviate from that initially calculated.
A real sequential analysis approach on the other hand needs
to work the opposite way around: start from zero information
about each point and add samples until the distance between
their average runtime and the true mean is minimal. We
cannot know this distance without actually taking a large
247
number of samples, but we can approximate it by looking at
what the rest of the space tells us.
Consider Figure 2, where we unroll thei1 loop in the
adi benchmark a random number of times and take a single
sample each time. Despite the noise, we see that there is a
pattern easily identiﬁable to the human eye: a plateau starting
at 2.1s then climbs and levels off at 3.1s, around a loop
unroll factor of 10. We postulate, and it can be shown that,
points in areas where the pattern is clear and which ﬁt well in
that pattern are likely to be already close to their respective
population means. The points where we need more samples
are the rest.
Our active learning approach for iterative compilation
already uses Machine Learning to discover patterns in the
decision space during the training process. We can use that
same knowledge to determine whether a set of runtime
measurements for a point in the space ﬁts the local pattern or
not, that is whether it is likely to be affected by noise or not
given what is known about its neighbours.
3. Active Learning with Sequential Analysis
Previous research [18, 31] has shown that machine learning
can be used to generate compiler heuristics more accurately
than human experts. However, current implementations use
randomly collected examples for training and this is problem-
atic since randomness often leads to redundancy. In contexts
where obtaining data is cheap this is perfectly acceptable; but
for us, since each example requires compilation and multiple
runs to record average performance, a lot of effort can be
wasted.
Active learning [39] is speciﬁcally aimed at reducing
the occurrence of these unproﬁtable evaluations. Instead of
blindly selecting training examples, in our case binaries com-
piled with certain optimizations, it generates an intermedi-
ate model based on the examples already evaluated. The
algorithm considers a number of potential candidates (optim-
ization settings) that it could learn from next and assigns each
a score. This score represents the predicted extra information
that the example will provide. It is usually a function of the
uncertainty the model has with regards to the predicted value
– runtime. The example with the highest score is ‘labelled’
– compiled and proﬁled – and the information used to up-
date the intermediate model. This loop continues until some
completion criterion has been reached.
Our work in this paper introduces a novel approach to
active learning which is broadly applicable. The only as-
sumptions we make are that neighbouring examples in the
optimization design space do not signiﬁcantly differ in res-
ult from one another the majority of the time, and that the
signal-to-noise ratio of measurements are such that an ap-
proximately accurate model can be ﬁt to the data, where this
is in essence no different than techniques which attempt to
avoid over-ﬁtting. While traditional active learning is used
to reduce only the number of training examples,we wish
to reduce the number of samples needed per example.T o
the best of our knowledge, we are the ﬁrst to propose com-
bining active learning and sequential analysis in this way.
Previous research in this area has ignored sequential analysis.
The reason for this is that many implementations of active
learning are greedy [6, 47], so learning noisy data on pur-
pose will lead to incorrect conclusions being drawn from the
intermediate models. In particular, this will steer the search
towards the wrong areas of the decision space, signiﬁcantly
reducing further the quality of the ﬁnal model. In Section 3.3
we explain how we overcome this problem.
3.1 Sequential Analysis
Traditionally in active learning the training set (the set of
examples already seen) and the candidate set, a random
subset of all examples that could be learnt from next, are kept
disjoint. This makes sense because the information contained
in the training set is assumed to be of good quality; each
example will have been evaluated some ﬁxed number of
times to ameliorate the effect of noise, hence, there is little
to be gained from revisiting those examples. However, as
we have demonstrated, a ﬁxed sample count is often overly
conservative and wasteful, slowing down the training process
substantially.
In order to modify active learning to incorporate sequential
analysis we change the algorithm such that the initial sample
size is set to one. In case of noisy data, we need to be able to
revisit previously compiled programs so we keep them in the
candidate set – see Figure 3. That is to say, at each iteration of
the learning loop the algorithm will consider not only getting
a new example but also whether it is more proﬁtable to try an
old one again, similar to the multi-armed bandit problem [29].
We are able to do this because the particular model we use
provides a scoring function which quantiﬁes the uncertainty
the model has about a particular point in the space, given
what it knows. As knowledge is gained, given the shape
of the intermediate model, noisy examples or examples in
complex areas of the decision space will begin to ‘stick out’,
and will be more likely to be visited. In both cases, with each
iteration of the training loop, we select the example where
the highest amount of information can be extracted.
We outline our algorithm in more detail in Alg. 1. The
algorithm begins by constructing a modelM with ninit
training examples which have been randomly chosen from
all potential examplesF as a seed. To generate this initial
model we obtain some ﬁxed number of observationsnobs
for each training example to give the active learner a quick
and accurate look at the search space. The learning loop
then proceeds whilst the completion criterion has not been
satisﬁed. In Alg. 1 this criterion is set to a ﬁxed number of
training instances but could have been based on, for example,
wall-clock time or some estimate of error in the ﬁnal model
established through cross-validation [25]. At each iteration
of the loop the candidate setC combines nc random points
which have never been observed before and those examples
248
LearningAlgorithms NewObservation
InitialTrainingPoints
Active Learner
IntermediateModel FinalModelEvaluateCandidates
RandomUnseenPreviouslySeen
Figure 3. An overview of our active learning approach. To seed an initial model, we give the learner some good quality data.
We then choose a single training example from a set of candidate examples. We collect data for the chosen example and feed
that back into the algorithm. The process repeats until we reach some completion criterion. Contrary to existing active learning
approaches, we collect our (potentially noisy) training data one observation at a time. Visited training examples remain in the
candidate set and can be revisited if getting more observations for them is more proﬁtable than trying a new training example.
which have been seen previously but less thannobs times. We
choose the next training examplex based upon its predicted
usefulness (see Section 3.3) and we measure its runtimey one
more time. We then update the model as well as the required
data structures. It should be noted that this algorithm is easily
parallelized by selecting multiple training examples per loop
iteration instead of just one [4].
Tt Tt+1
StayPrune Grow
Xt+1
Figure 4. This diagram shows the three potential updates
that are stochastically applied to the Dynamic Tree upon
receiving a new training example xt+1. The tree either
remains unchanged, a leaf node is pruned back so that the
parent of the leaf becomes a leaf itself, or grown such that
two new children divide the relevant subspace.
3.2 Dynamic Trees
In regression problems where we wish to estimate the uncer-
tainty of a prediction the collective wisdom would be to use a
Gaussian Process (GP) [46]. However, GP inference is slow
with O(n3) efﬁciency forn examples. This is problematic,
particularly in active learning, since each time something
new is learned a model needs to be constructed and evaluated.
A more efﬁcient model which we leverage in this work is
the relatively new dynamic tree, which is based on the clas-
sical decision tree model [8] with modiﬁcations to include
Bayesian inference [10, 11]. The advantages of the dynamic
tree for our purposes are
• its ability to evolve over time as new data come in, without
reconstructing the model from the ground up with each
iteration;
• its estimation of uncertainty at any given point in the
space, like a GP but without the overhead;
• its avoidance of over-ﬁtting to the training data, which is
vital since we are learning potentially noisy information.
Full details on the model can be obtained from the article
by Taddyet al.[51]. The brief overview of how it works is
as follows. The static model used within the dynamic tree
framework is a traditional decision tree for regression applic-
ations. A set of rules recursively partitions the search space
into a set of hyper-rectangles such that training examples
with the same or similar output value are contained within
the same leaf node. The dynamic tree changes over time,
when new information is introduced, by a stochastic process
thereby avoiding the need to prune at the end. At timet, a tree
Tt is derived from the training data(x, y)t. When new data
(xt+1,y t+1) arrives, an updated treeTt+1 is created, identical
to Tt except that some mechanism has been randomly chosen
from three possibilities – see Figure 4. The leaf node⌘(xt+1)
containing xt+1 either (1) remains completely unchanged; (2)
is pruned, so that the parent of⌘(xt+1) becomes a leaf node;
(3) is grown, such that⌘(xt+1) becomes an internal node to
two new children. The choice of transformation is inﬂuenced
by yt+1 in a posterior distribution. This posterior distribution
depends upon the probability ofyt+1 given xt+1,T t, and
[x, y]t; hence, the dynamic tree is more resilient to noisy data
than other techniques.
3.3 Quantifying Usefulness
The most crucial part of the active learning loop is estimating
which training example from within the pool of potential
candidates C would be most proﬁtable to learn from next. The
dynaTree package for R [21] that we use offers two heuristics
out-of-the-box, both well cited in the literature for regression
problems. The ﬁrst was presented by Mackay [34] and selects
the candidate where the estimated variance of the output
is maximized relative to the other candidates. The second
heuristic by Cohn [13] selects the candidate it calculates will
most reduce the predicted average variance across the space.
To put this in a more accessible way, it selects the example
it believes will enable the model to best ﬁt what it is already
seeing, in an attempt to reveal key information that it may be
missing. Both are competitive with each other, and both solve
249
Algorithm 1 An active learning algorithm modiﬁed to re-
duce the number of samples, whereF contains all training
examples that could be chosen,ninit and nmax specify the
initial and total number of training examples to record,nc the
number of candidates per iteration, andnobs the number of
samples thought to be needed to reduce the affects of noise
in the output/performance values.
1: procedure ACTIVE LEARN (F,n init,n max,n c,n obs)
2: X  sample(F,n init)
3: Y  getObservations(X, nobs)
4: M  dynaTree(X, Y)
5: D  ;
6: for i = ninit,n max do
7: C  sample(F   X, nc)
8: for allk 2 keys(D) do
9: if D[k] <n obs then C  C [ k
10: end if
11: end for
12: x  ;
13: vmin  MAX_DOUBLE
14: for allc 2 C do
15: v  predictAvgModelVariance(M,c )
16: if v<v min then
17: vmin  v
18: x  c
19: end if
20: end for
21: y  getObservations(x, 1)
22: M  updateModel(M,x,y )
23: X  X [ x
24: if k 2 keys(D) then
25: D[k]  D[k]+1
26: else
27: D[k]  1
28: end if
29: end for
30: return M
31: end procedure
the greedy search problem discussed previously, although
the latter is more computationally intensive than the former –
O(|C|2) versus O(|C|). Despite this, we use the latter as our
scoring function, since it handles heteroskedasticity, non-
uniform variance across the space which we assume for
increased robustness, more effectively.
4. Experimental Setup
4.1 Optimization Problem
We evaluate our approach by examining how efﬁciently
we can construct models to solve a classical but complex
compilation problem. In particular, the problem we consider
in this work involves ﬁnding the optimal set of compilation
parameters for a program. The set of parameters includes loop
unrolling, cache tiling, and register tiling factors, where each
parameter has a range of possible values unique to each loop.
The combination of these parameters results in a massive
search space where we will need to ﬁnd a conﬁguration that
leads to a short program runtime. Our goal is to build a
program-speciﬁc model that can predict the runtime from the
given set of optimizations. This allows us to quickly search
over a large number of conﬁgurations to ﬁnd out the best
performing one without compiling and proﬁling the program
with every single option.
4.2 Platform and Benchmarks
Platform We evaluated our method on a server running
OpenSuse v12.3 with an Intel Core i7-4770K 4-coreCPU
at 3.4GHz. The machine contains 16GB of RAM and the
compiler used was gcc v4.7.2.
Environment We measured time using the C library func-
tion clock_gettime(). As in previous iterative compilation
literature, our machine was restricted to a single user and did
not have any processes running other than those enabled by
default under a standard OS installation. We took no further
steps to reduce experimental noise, such as pinning threads or
using a non-standard memory allocator; we decided against
this to avoid creating an artiﬁcial environment which could
alter our ﬁndings, as discussed previously in Section 1.
Benchmarks We used 11 applications taken from the
SPAPT suite [3], a collection of search problems (programs)
for automatic performance tuning. These benchmarks are
based on high-performance computing problems such as
stencil codes and dense linear algebra. These particular 11
were chosen based on an initial prototyping of our algorithm
using data kindly provided by the authors of [4], where only
these 11 were contained within that initial dataset. These
programs are sequential implementations, where dynamic
memory is allocated it is through the standardmalloc()
library function; again, we decided against using a low noise
allocator for reasons previously discussed.
Each problem of the SPAPT suite is deﬁned by three
primary variables – kernel, input size, and tunable conﬁg-
uration. The tunable parameters are further broken down into
a number of integer and binary values, with the values giv-
ing optimizing code transformations to apply, as speciﬁed
above. In our experiments binary ﬂags and input size were
not considered so that a fair comparison could be made with
the related work [4]. The precise size of each search space is
given in Table 1.
4.3 Evaluation Methodology
Baseline Approach Most machine learning in compilers
works use simple constant sampling plans [36, 37, 49], where
the number of observations in each sample is ﬁxed ahead of
time. Different sizes are chosen in the literature, for example,
[19, 23] use 10, [24] uses 20, [42] uses 80, and the work
against which we compare [4] uses 35. There is no statistical
criterion that can determine how many observations will be
sufﬁcient for this purpose. However, post hoc validation can
250
be performed, for example by calculating the ratio of the
Conﬁdence Interval (CI) to the mean and rejecting if that
breaches some threshold. Typically, this validation is not
presented in papers, if it is done at all. When it is done,
standard values are to use the 95% conﬁdence and a 1%
CI/mean threshold. In this paper, we compare against a
constant sampling plan of 35 observations, as that is what
is used in our comparison work [4]. We note that even 35
observations is not always enough. Across our benchmarks
we found that even though on the majority of examples there
was often very little noise, many did not fall into this pattern.
Fully 5% of examples broke the threshold. Even at a more
generous 5% threshold, we found that with 35 observations,
0.5% failed. With fewer observations the problem is worse.
At 5 observations, 3.3% fail that more generous threshold,
and at 2 observations (the minimum to have any statistical
certainty), 5% fail. This ﬁnding is corroborated by [32] which
samples until the threshold is met, and discovers that for
timing small code sequences it is sometimes necessary to take
hundreds of observations. Since active learning is susceptible
to bad data, these erroneous training examples can have a
detrimental affect on the quality of the learned heuristic and
its convergence.
Our technique, by avoiding a constant sampling plan, is
able to achieve far better results. Indeed, it is even able to
perform adequately with only one observation per example
in low noise parts of the space, and will spend effort with
multiple observations only on those parts of the space that
require it.
Based on classical methodologies, we consider two tech-
niques to be in competition with our own. For both, we get a
ﬁxed number of observations for each training example and
the candidate set is kept disjoint from past training examples.
The ﬁrst technique uses the average of the runtimes recorded
over 35 observations per single training example, as in [4].
The second technique records a single execution per example.
In this way we can compare how our approach fairs in rela-
tion to both very low and relatively high accuracy per training
point, in terms of estimating mean runtime.
In order to provide the best evaluation possible we com-
pare our methodology to the active learning approach by
Balaprakash et al.[4]; in particular, we use the same bench-
mark suite, model, parameters, and accuracy metric they do.
Evaluation Metrics Our evaluation examines the efﬁciency
of model construction and, more speciﬁcally, the evolution
of themodel errorover training timefor each one of the 11
benchmarks and the three different active learning approaches.
We quantify the accuracy of the models produced by each
approach using the Root Mean Squared Error of the predicted
runtimes (1). For each data point in a test set ofn instances
the runtime predicted by the current modelˆyt is compared to
the observed mean runtimeyt as follows:
RMSE =
sPn
i=1 (ˆyi   yi)2
n (1)
We measure the training time in each experiment as the
cumulative compilation and runtimes of any executables used
in training. The overhead of updating the Dynamic Trees
is not measured as it is a small part of the overall training
overhead and is near constant for all evaluated approaches.
4.4 Algorithm and Model Parameters
For each kernel the goal is to produce a model capable of
estimating mean serial code runtime for any set of optimiza-
tion settings. To this end, we used the following parameters
in constructing our model and overall learning algorithm.
With respect to Alg. 1, we start by seeding the algorithm
with just 5 random examplesninit. For each of these we
record 35 observationsnobs to calculate a mean runtime.
For the Dynamic Tree model we employ the RdynaTree
package. We use an entirely default conﬁguration except the
number of particlesN is set to 5,000. In each iteration of
the loop we consider500 random and new candidate training
instances nc.
The completion criterion for the experiments was set such
that the maximum size of the training setnmax does not
exceed 2,500. All experiments were repeated ten times with
new random seeds. The results reported in Section 5 are all
averaged over ten experimental runs.
4.5 Description of the Datasets
To collect the data for our experiments we proﬁle each pro-
gram with 10,000 distinct, randomly selected conﬁgurations.
For each one, we record its mean runtime, determined by
averaging 35 separate executions per example, and its com-
pilation time. Per experiment, we randomly mark 7,500 of
them as available for use in training, while we test the model
on the remaining 2,500 examples.
The feature values of each data point, which is to say
the values which make each example distinct from one
another, were all normalized through scaling and centring
to transform them into something similar to the Standard
Normal Distribution: a common practice in machine learning
work, where features are not all on comparative scales.
5. Experimental Results
In this Section we ﬁrst show that our approach can greatly
speed-up the learning process by reducing the cost of proﬁling
by up to 26x, as compared to a baseline approach that uses 35
observations for each data point. We then provide a detailed
analysis of our results for each program in turn.
5.1 Overall Analysis
To evaluate the overall efﬁciency of our proposed method-
ology versus the baseline active learning approach [4], we
251
Table 1.Lowest common RMS error achieved by both approaches, proﬁling time needed to reach this error level, and speed-up
for all 11 benchmarks
benchmark search space lowest common RMSE cost of the baseline cost of our approach speed-up
(sec) (sec)
adi 3.78 ⇥ 1014 0.087 2.62 ⇥ 104 9.08 ⇥ 104 0.29
atax 2.57 ⇥ 1012 0.097 3.33 ⇥ 103 2.39 ⇥ 102 13.93
bicgkernel 5.83 ⇥ 108 0.065 1.35 ⇥ 104 3.76 ⇥ 103 3.59
correlation 3.78 ⇥ 1014 0.589 57.46 8.13 7.07
dgemv3 1.33 ⇥ 1027 0.067 1.75 ⇥ 102 7.44 23.52
gemver 1.14 ⇥ 1016 0.342 2.99 ⇥ 103 1.15 ⇥ 102 26.00
hessian 1.95 ⇥ 107 0.006 5.76 ⇥ 103 1.56 ⇥ 103 3.69
jacobi 1.95 ⇥ 107 0.076 3.04 ⇥ 103 8.57 ⇥ 102 3.55
lu 5.83 ⇥ 108 0.013 2.57 ⇥ 103 7.09 ⇥ 102 3.62
mm 3.18 ⇥ 109 0.042 9.87 ⇥ 104 8.89 ⇥ 104 1.11
mvt 1.95 ⇥ 107 0.002 2.59 ⇥ 103 2.20 ⇥ 103 1.18
geometric mean 3.97
Table 2.This table gives an indication of the spread of the variance and 95% conﬁdence interval relative to the mean for all
benchmarks tested; the latter is given for two sample sizes, 5 and 35 observations. The values shown illustrate that although
noise can be low for many benchmarks, it is high for others.
benchmark variance 35-sample 95% C.I. / mean 5-sample 95% C.I. / mean
min mean max min mean max min mean max
adi 8.44 ⇥ 10 10 2.34 ⇥ 10 3 0.14 4.10 ⇥ 10 6 2.25 ⇥ 10 3 0.05 2.77 ⇥ 10 6 0.01 0.16
atax 7.54 ⇥ 10 10 9.72 ⇥ 10 5 0.03 2.22 ⇥ 10 5 2.31 ⇥ 10 3 0.06 1.79 ⇥ 10 5 0.01 0.25
bicgkernel 2.06 ⇥ 10 10 1.06 ⇥ 10 4 0.05 1.17 ⇥ 10 5 1.52 ⇥ 10 3 0.07 1.02 ⇥ 10 5 4.64 ⇥ 10 3 0.29
correlation 2.27 ⇥ 10 10 0.42 8.02 2.13 ⇥ 10 5 0.03 0.34 4.42 ⇥ 10 6 0.13 2.41
dgemv3 1.15 ⇥ 10 9 5.60 ⇥ 10 5 0.03 3.31 ⇥ 10 5 2.25 ⇥ 10 3 0.08 2.24 ⇥ 10 5 0.01 0.28
gemver 1.19 ⇥ 10 9 5.91 ⇥ 10 3 0.47 1.18 ⇥ 10 5 4.81 ⇥ 10 3 0.10 9.34 ⇥ 10 6 0.02 0.42
hessian 2.35 ⇥ 10 11 1.03 ⇥ 10 6 1.99 ⇥ 10 4 3.89 ⇥ 10 5 1.33 ⇥ 10 3 0.06 1.63 ⇥ 10 5 4.15 ⇥ 10 3 0.24
jacobi 2.54 ⇥ 10 10 1.20 ⇥ 10 4 0.09 1.32 ⇥ 10 5 1.29 ⇥ 10 3 0.09 4.12 ⇥ 10 6 3.83 ⇥ 10 3 0.39
lu 1.84 ⇥ 10 11 8.45 ⇥ 10 7 1.09 ⇥ 10 4 2.03 ⇥ 10 5 6.89 ⇥ 10 4 0.02 5.76 ⇥ 10 6 2.10 ⇥ 10 3 0.11
mm 2.76 ⇥ 10 10 4.87 ⇥ 10 6 1.31 ⇥ 10 3 2.26 ⇥ 10 5 7.44 ⇥ 10 4 0.02 1.36 ⇥ 10 5 2.37 ⇥ 10 3 0.09
mvt 9.97 ⇥ 10 12 1.07 ⇥ 10 8 7.87 ⇥ 10 6 6.29 ⇥ 10 5 8.28 ⇥ 10 4 0.03 3.98 ⇥ 10 5 2.44 ⇥ 10 3 0.11
adimmmvtjacobibigckernel
luhessiancorrelationataxdgemv3gemverGeo-mean
0123456789101112131415 26x
Reduction of proling cost
23.5x
Figure 5. Reduction of proﬁling overhead compared to a
baseline approach
measured the time needed for both techniques to ﬁrst reach a
common lowest average error.
In particular, Table 1 shows for each benchmark what this
lowest common average error is and how many seconds it
took to collect the proﬁling data needed to reach this level
for the competing methods. Figure 5 presents graphically
the acceleration achieved by our approach. In all but one
benchmark our algorithm is faster at reaching the lowest
average error. Speciﬁcally, our methodology is able to reduce
the overhead for 10 benchmarks by up to 26x. The only
benchmark in which our approach fails to reduce the overhead
is adi. However, the difference in errors between the two
techniques is comparable, within a few thousandths of a
second on average. Summarizing, our approach outperforms
the baseline by achieving on average a 4x reduction of
the proﬁling overhead, which translates to saving weeks of
compute time in practice for many compilation problems.
5.2 Detailed Analysis
In this Section we present our ﬁndings in more detail. Fig-
ures 6a–6f show the Root Mean Squared Error (1) against
evaluation time (cumulative proﬁling and compilation cost
in seconds) averaged over 10 runs for several representative
results. To make a fair comparison each graph shows the
range of time over which all three sampling plans are simul-
taneously active in processing up to 2,500 training instances.
What follows is a qualitative summary of those results.
adi: Figure 6a gives error against time for the three different
sampling techniques we evaluated for benchmarkadi. It
seems self-evident that there is some considerable noise in
the underlying data since a single observation per training
example plateaus in error fairly quickly and cannot achieve
the same results as the other two methods. Although our
252
variable observation approach is also unable to keep up with
a high ﬁxed number of observations per example it does
achieve comparably low error throughout.
atax, bicgkernel: The data of benchmarkatax in Figure
6b is quite different to that in Figure 6a and appears to
represent a case where the underlying noise in performance
measurements is comparatively low. This is exempliﬁed by
the fact that one sample per unique instance is enough to
do well, and indeed our technique appears to detect this;
compare these plot-lines to the 35 observations approach and
we see a good example of how much time can potentially be
saved by our technique over this baseline. Thebicgkernel
experiments follow the same sort of pattern.
correlation: Figure 6c, showing the results of the benchmark
correlation, is interesting since the error remains high
regardless of sampling technique. Data from Table 2 points
at a potential reason for this, that we outline below. As in
Figure 6a, we see that there must be noise present since one
observation does even worse. Our approach is not quite as
good as using a large number of observations per data point
but is competitive and within a few hundredths of a second
in terms of average error by the end of the displayed time.
dgemv3, gemver, hessian: In Figure 6d our variable ap-
proach is much faster than the classical method and the
simple but potentially noisy variant, similarly for the results
of dgemv3 and hessian.
jacobi, lu, mm, mvt: The data for thejacobi benchmark
(Figure 6e), which is also generally representative oflu, are
interesting since they show our algorithm to be slightly too
cautious but still much more efﬁcient than a ﬁxed sampling
plan. Themm benchmark gives a graph akin to that ofmvt,
showing our approach as giving slight speed-ups over the
classical methodology.
Table 2 details the distributions of the runtimes measured
during our experiments, and in particular the spread of the
variance and conﬁdence intervals relative to the means. The
level of noise across this set of benchmarks varies across
applications. Moreover, the variance is not constant across all
parts of the space for even a single benchmark in isolation;
some parts of the space suffer from extreme noise. An
adaptive algorithm, such as ours, is necessary to make the
best of these conditions.
Correlation shows very high noise and achieves a 7x
learning speed-up whereasgemver has lower noise but gave
us the highest learning speed-up – 26x. We think this is
because our experiments capped the number of executions
at 35, but many points incorrelation’s space need more
data. This limits the maximum speed-up that can be attained.
Gemver, in contrast tocorrelation has fewer points for
which 35 observations are inadequate. Foradi, where the
speed-up runs counter to our expectations, we ran longer
experiments but the outcome did not change. We believe that
this is due to the shape of the noisy regions in the space. We
will investigate these in future work.
6. Related Work
Our work lies at the intersection of optimization modeling
and active learning. No existing work has used sequential
analysis and active learning to reduce the overhead of iterative
compilation.
Analytic Modeling Analytic models have been widely used
to tackle complex optimization problems, such as auto-
parallelization [7, 40], runtime estimation [12, 27, 58], and
task mappings [28]. A particular problem with them, however,
is that the model has to be re-tuned for each new targeted
hardware [50].
Predictive Modeling Predictive modeling has been shown
to be useful in the optimization of both sequential and par-
allel programs [14, 23, 52, 53]. Its great advantage is that
it can adapt to changing platforms as it has noa priorias-
sumptions about their behavior, but it is expensive to train.
There are many studies showing it outperforms human based
approaches [19, 24, 31, 54, 61]. Prior work for machine learn-
ing in iterative compilation [20] often uses random sampling
or exhaustive search to collect training examples. The process
of collecting these examples can be expensive, taking several
months in practice. With active learning, our approach can
signiﬁcantly reduce the overhead of collecting these training
data, accelerating the process of tuning optimization heurist-
ics using machine learning.
Active Learning for Systems Optimization Active learn-
ing has recently emerged as a viable means for constructing
heuristics for systems optimization. Zuluagaet al.[60] pro-
posed an active learning algorithm to select parameters in a
multi-objective problem; Balaprakashet al.[4, 5] used act-
ive learning to ﬁnd optimizations for bothCPU and GPU
scientiﬁc codes; and Ogilvieet al.[39] proposed its use to
construct models to map programs in aCPU–GPU mixed
platform. In all these works, however, each training example
was proﬁled a ﬁxed number of times in order to compute an
average performance. Our work advances this prior work by
dynamically adjusting the number of proﬁling runs as needed,
which signiﬁcantly reduces the training overhead.
Program Runtime Variation The work by Mazouzet al.
[35] shows that parallel program execution time could vary
to various extents on different platforms. Thus, the number of
proﬁling runs needed for statistical soundness varies from one
platform to the other. Leatheret al.[32] proposed a statistical
method to determine the number of times to proﬁle a program
but for the much simpler problem of determining the best
performing binary version of a program during a random
search of the optimization space. In their work, binaries
whose conﬁdence interval of the runtime does not overlap
with that of the best performing binary are not revisited.
253
0 20000 40000 60000 80000
0.085 0.095 0.105
Results for adi benchmark
Evaluation Time (s)
Root Mean−Square Error (s)
all observationsone observationvariable observations
(a) adi
0 1000 2000 3000
0.06 0.08 0.10 0.12 0.14
Results for atax benchmark
Evaluation Time (s)
Root Mean−Square Error (s)
all observationsone observationvariable observations
(b) atax
0 200 400 600 800 1000
0.6 0.7 0.8 0.9 1.0
Results for correlation benchmark
Evaluation Time (s)
Root Mean−Square Error (s)
all observationsone observationvariable observations
(c) correlation
0 500 1000 1500 2000 2500 3000
0.25 0.30 0.35 0.40
Results for gemver benchmark
Evaluation Time (s)
Root Mean−Square Error (s)
all observationsone observationvariable observations
(d) gemver
0 500 1000 1500 2000 2500 3000
0.04 0.06 0.08 0.10 0.12 0.14
Results for jacobi benchmark
Evaluation Time (s)
Root Mean−Square Error (s)
all observationsone observationvariable observations
(e) jacobi
0 500 1000 1500 2000 2500
0.002 0.003 0.004 0.005 0.006
Results for mvt benchmark
Evaluation Time (s)
Root Mean−Square Error (s)
all observationsone observationvariable observations
(f) mvt
Figure 6. RMS error over time for six of our benchmarks for three different approaches: one observation, 35 observations, and
variable observations per training point
7. Conclusions and Future Work
While we need good software optimization heuristics to fully
exploit the performance potential of our hardware, it is be-
coming increasingly unrealistic to hand-tune them with expert
knowledge for every hardware architecture they have to target.
The result is that current compilers use out-of-date optimiza-
tion strategies, ultimately leading to sub-optimal binaries. To
alleviate this problem, previous research has proposed ma-
chine learning to automate the heuristic generation process.
Existing implementations are unnecessarily slow. They select
their training data randomly, much of which carry little useful
information despite being time consuming to acquire. Active
learning approaches have tackled this inefﬁciency, but their
inﬂexible sampling plan still causes them to collect training
data with little useful information.
In this paper, we present a unique approach, broadly applic-
able to heuristic generation. It combines sequential analysis,
which reduces the observations per training example, together
with active learning, which reduces training examples overall,
to greatly accelerate learning. We demonstrate our approach
by comparing it with a baseline 35-sample technique which
creates software optimization models derived through active
learning alone. Our approach achieves an average speed-up
of 4x, and up to 26x, without signiﬁcant penalties to the ﬁnal
heuristic quality.
We intend to test the bounds of our technique by artiﬁcially
introducing noise into the system to see how robustly it
performs in extreme cases. Success would allow our strategies
to be used in heavily loaded multi-user environments. This
would have been interesting in this paper, and is something
of an omission that was pointed out to us in a review. As such
we leave it to future work.
Acknowledgments
This work was partly supported by the UK Engineering
and Physical Sciences Research Council (EPSRC) under
grants EP/L000055/1 (ALEA), EP/M01567X/1 (SANDeRs),
EP/M015823/1, and EP/M015793/1 (DIVIDEND). We would
like to thank Dr. Balaprakash, of Argonne National Laborat-
ory, for his kind help in providing us the initial data for our
research.
254
References
[1] 2015 international technology roadmap for semicon-
ductors. http://www.semiconductors.org/main/
2015_international_technology_roadmap_for_
semiconductors_itrs/. Retrieved 08/09/16.
[2] F. Agakov, E. Bonilla, J. Cavazos, B. Franke, G. Fursin, M. F. P.
O’Boyle, J. Thomson, M. Toussaint, and C. K. I. Williams.
Using Machine Learning to Focus Iterative Optimization. In
CGO, 2006.
[3] P. Balaprakash, S. M. Wild, and B. Norris. SPAPT: Search
Problems in Automatic Performance Tuning. InICCS, 2012.
[4] P. Balaprakash, R. B. Gramacy, and S. M. Wild. Active-
Learning-Based Surrogate Models for Empirical Performance
Tuning. InCLUSTER, 2013.
[5] P. Balaprakash, K. Rupp, A. Mametjanov, R. B. Gramacy, P. D.
Hovland, and S. M. Wild. Empirical Performance Modeling of
GPU Kernels Using Active Learning. InParCo, 2013.
[6] M.-F. Balcan, A. Beygelzimer, and J. Langford. Agnostic
Active Learning. InICML, 2006.
[7] U. Bondhugula, A. Hartono, J. Ramanujam, and P. Sadayappan.
A Practical Automatic Polyhedral Parallelizer and Locality
Optimizer. InPLDI, 2008.
[8] L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone.
Classiﬁcation and regression trees. Wadsworth and Brooks,
1984.
[9] J. Charles, P. Jassi, N. S. Ananth, A. Sadat, and A. Fedorova.
Evaluation of the Intel Core i7 Turbo Boost feature. InIISWC,
2009.
[10] H. A. Chipman, E. I. George, and R. E. McCulloch. Bayesian
CART Model Search. Journal of the American Statistical
Association, 93, 1998.
[11] H. A. Chipman, E. I. George, and R. E. McCulloch. Bayesian
Treed Models.Machine Learning, 48, 2002.
[12] M. Clement and M. Quinn. Analytical Performance Prediction
on Multicomputers. InSC, 1993.
[13] D. A. Cohn. Neural Network Exploration Using Optimal
Experiment Design.Neural Networks, 9(6), 1996.
[14] K. D. Cooper, P. J. Schielke, and D. Subramanian. Optimizing
for Reduced Code Space using Genetic Algorithms. InLCTES,
1999.
[15] C. Cummins, P. Petoumenos, M. Steuwer, and H. Leather.
Autotuning OpenCL Workgroup Size for Stencil Patterns.
arXiv preprint arXiv:1511.02490, 2015.
[16] C. Curtsinger and E. D. Berger. STABILIZER: Statistically
Sound Performance Evaluation. InASPLOS, 2013.
[17] A. B. de Oliveira, J.-C. Petkovich, and S. Fischmeister. How
much does memory layout impact performance? A wide study.
In REPRODUCE 2014, 2014.
[18] C. Dubach, T. Jones, E. Bonilla, G. Fursin, and M. F. P.
O’Boyle. Portable Compiler Optimisation Across Embedded
Programs and Microarchitectures using Machine Learning. In
MICRO, 2009.
[19] M. K. Emani et al. Smart, Adaptive Mapping of Parallelism in
the Presence of External Workload. InCGO, 2013.
[20] G. Fursin et al. Milepost GCC: Machine Learning Enabled
Self-tuning Compiler.International Journal of Parallel Pro-
gramming, 39(3), 2011.
[21] R. B. Gramacy and M. A. Taddy. dynaTree: Dynamic Trees
for Learning and Design.http://faculty.chicagobooth.
edu/robert.gramacy/dynaTree.html, 2011. R package.
Retrieved 02/29/16.
[22] D. Grewe, Z. Wang, and M. F. O’Boyle. Portable Mapping of
Data Parallel Programs to OpenCL for Heterogeneous Systems.
In CGO, 2013.
[23] D. Grewe, Z. Wang, and M. F. P. O’Boyle. OpenCL Task
Partitioning in the Presence of GPU Contention. InLCPC,
2013.
[24] D. Grewe et al. A Workload-Aware Mapping Approach For
Data-Parallel Programs. InHiPEAC, 2011.
[25] T. Hastie, R. Tibshirani, and J. Friendman.The Elements of
Statistical Learning: Data Mining, Inference, and Prediction,
chapter 7. Springer, 2 edition, 2009.
[26] J. Herter, P. Backes, F. Haupenthal, and J. Reineke. CAMA:
A predictable cache-aware memory allocator. InEuromicro,
2011.
[27] S. Hong and H. Kim. An Analytical Model for a GPU
Architecture with Memory-level and Thread–level Parallelism
Awareness. InISCA, 2009.
[28] A. H. Hormati, Y . Choi, M. Kudlur, R. Rabbah, T. Mudge, and
S. Mahlke. Flextream: Adaptive Compilation of Streaming
Applications for Heterogeneous Architectures. InPACT, 2009.
[29] M. N. Katehakis and J. Arthur F. Veinott. The multi-armed
bandit problem: Decomposition and computation.Mathematics
of Operations Research, 12(2), 1987.
[30] P. M. W. Knijnenburg, T. Kisuki, and M. F. P. O’Boyle.
Iterative Compilation. 2002.
[31] S. Kulkarni and J. Cavazos. Mitigating the Compiler Optim-
ization Phase-Ordering Problem using Machine Learning. In
OOPSLA, 2012.
[32] H. Leather, M. F. P. O’Boyle, and B. Worton. Raced proﬁles:
efﬁcient selection of competing compiler optimizations. In
LCTES, 2009.
[33] H. Leather, E. Bonilla, and M. O’Boyle. Automatic Feature
Generation for Machine Learning–based Optimising Compila-
tion. ACM TACO, 2014.
[34] D. J. C. MacKay. Information-Based Objective Functions for
Active Data Selection.Neural Computation, 4, 1992.
[35] A. Mazouz, S. A. A. Touati, and D. Barthou. Study of
Variations of Native Program Execution Times on Multi-Core
Architectures. InCISIS, 2010.
[36] A. Monsifrot, F. Bodin, and R. Quiniou. A Machine Learning
Approach to Automatic Production of Compiler Heuristics. In
AIMSA, 2002.
[37] E. Moss, P. Utgoff, J. Cavazos, D. Precup, D. Stefanovic,
C. Brodley, and D. Scheeff. Learning to schedule straight-line
code. Advances in Neural Information Processing Systems,
1997.
255
[38] T. Mytkowicz, A. Diwan, M. Hauswirth, and P. F. Sweeney.
Producing Wrong Data Without Doing Anything Obviously
Wrong! InASPLOS XIV, 2009.
[39] W. F. Ogilvie et al. Fast Automatic Heuristic Construction
Using Active Learning. InLCPC. 2014.
[40] E. Park, J. Cavazos, L.-N. Pouchet, C. Bastoul, A. Cohen, and
P. Sadayappan. Predictive Modeling in a Polyhedral Optimiza-
tion Space.International Journal of Parallel Programming, 41
(5), 2013.
[41] P. Petoumenos, G. Keramidas, H. Zeffer, S. Kaxiras, and
E. Hagersten. STATSHARE: A Statistical Model for Managing
Cache Sharing via Decay. InMoBS, 2006.
[42] P. Petoumenos, L. Mukhanov, Z. Wang, H. Leather, and D. S.
Nikolopoulos. Power Capping: What Works, What Does Not.
In ICPADS, 2015.
[43] L.-N. Pouchet. Polybench: The polyhedral benchmark suite.
URL: http://polybench. sourceforge. net, 2012.
[44] J. Power, A. Basu, J. Gu, S. Puthoor, B. M. Beckmann, M. D.
Hill, S. K. Reinhardt, and D. A. Wood. Heterogeneous System
Coherence for Integrated CPU–GPU Systems. InMICRO,
2013.
[45] K. K. Pusukuri, R. Gupta, and L. N. Bhuyan. Thread Tran-
quilizer: Dynamically reducing performance variation.ACM
TACO, 2012.
[46] C. E. Rasmussen and C. K. I. Williams.Gaussian Processes
for Machine Learning. The MIT Press, 2006.
[47] B. Settles.Active Learning. Morgan and Claypool, 2012.
[48] F. Siebert. Constant-Time Root Scanning for Deterministic
Garbage Collection. InCC, 2001.
[49] M. Stephenson, S. Amarasinghe, M. Martin, and U.-M.
O’Reilly. Meta Optimization: Improving Compiler Heuristics
with Machine Learning. InPLDI, 2003.
[50] M. Stephenson, S. Amarasinghe, M. Martin, and U.-M.
O’Reilly. Meta Optimization: Improving Compiler Heuristics
with Machine Learning. InPLDI, 2003.
[51] M. A. Taddy, R. B. Gramacy, and N. G. Polson. Dynamic Trees
for Learning and Design.Journal of the American Statistics
Association, 106(493), 2009.
[52] Z. Wang and M. F. O’Boyle. Mapping Parallelism to Multi-
cores: A Machine Learning Based Approach. InPPoPP, 2009.
[53] Z. Wang and M. F. O’Boyle. Partitioning Streaming Parallelism
for Multi-cores: A Machine Learning Based Approach. In
PACT, 2010.
[54] Z. Wang and M. F. P. O’Boyle. Using Machine Learning to
Partition Streaming Programs.ACM TACO, 2013.
[55] Z. Wang, D. Powel, B. Franke, and M. F. O’Boyle. Exploitation
of GPUs for the Parallelisation of Probably Parallel Legacy
Code. InCC, 2014.
[56] Z. Wang, G. Tournavitis, B. Franke, and M. F. P. O’boyle.
Integrating Proﬁle-driven Parallelism Detection and Machine-
learning-based Mapping.ACM TACO, 2014.
[57] Y . Wen, Z. Wang, and M. O’Boyle. Smart multi-task schedul-
ing for OpenCL programs on CPU/GPU heterogeneous plat-
forms. InIEEE HiPC, 2014.
[58] R. Wilhelm et al. The Worst-Case Execution-Time Problem –
Overview of Methods and Survey of Tools.ACM TECS, 2008.
[59] S. Zhuravlev, S. Blagodurov, and A. Fedorova. Address-
ing Shared Resource Contention in Multicore Processors via
Scheduling. InASPLOS XV, 2010.
[60] M. Zuluaga, A. Krause, G. Sergent, and M. Püschel. Active
Learning for Multi–Objective Optimization. InICML, 2013.
[61] M. Zuluaga et al. “Smart” Design Space Sampling to Predict
Pareto–Optimal Solutions. InLCTES, 2012.
256

LSTM: A Search Space Odyssey
Klaus Greff KLAUS @IDSIA .CH
Rupesh Kumar Srivastava RUPESH @IDSIA .CH
Jan Koutn´ık HKOU @IDSIA .CH
Bas R. Steunebrink BAS @IDSIA .CH
J¨urgen Schmidhuber JUERGEN @IDSIA .CH
The Swiss AI Lab IDSIA
Istituto Dalle Molle di Studi sull’Intelligenza Artiﬁciale
Universit`a della Svizzera italiana (USI)
Scuola universitaria professionale della Svizzera italiana (SUPSI)
Galleria 2, 6928 Manno-Lugano, Switzerland
Abstract
Several variants of the Long Short-Term Mem-
ory (LSTM) architecture for recurrent neural net-
works have been proposed since its inception in
1995. In recent years, these networks have be-
come the state-of-the-art models for a variety of
machine learning problems. This has led to a re-
newed interest in understanding the role and util-
ity of various computational components of typ-
ical LSTM variants. In this paper, we present
the ﬁrst large-scale analysis of eight LSTM vari-
ants on three representative tasks: speech recog-
nition, handwriting recognition, and polyphonic
music modeling. The hyperparameters of all
LSTM variants for each task were optimized sep-
arately using random search and their impor-
tance was assessed using the powerful fANOV A
framework. In total, we summarize the results
of 5400 experimental runs ( ≈15 years of CPU
time), which makes our study the largest of its
kind on LSTM networks. Our results show
that none of the variants can improve upon the
standard LSTM architecture signiﬁcantly, and
demonstrate the forget gate and the output activa-
tion function to be its most critical components.
We further observe that the studied hyperparam-
eters are virtually independent and derive guide-
lines for their efﬁcient adjustment.
1. Introduction
Recurrent neural networks with Long Short-Term Memory
(which we will concisely refer to as LSTMs) have emerged
as an effective and scalable model for several learning
problems related to sequential data. Earlier methods
for attacking these problems were usually hand-designed
workarounds to deal with the sequential nature of data such
as language and audio signals. Since LSTMs are effec-
tive at capturing long-term temporal dependencies without
suffering from the optimization hurdles that plague sim-
ple recurrent networks (SRNs) (Hochreiter, 1991; Bengio
et al., 1994), they have been used to advance the state of the
art for many difﬁcult problems. This includes handwriting
recognition (Graves et al., 2009; Pham et al., 2013; Doetsch
et al., 2014) and generation (Graves et al., 2013), language
modeling (Zaremba et al., 2014) and translation (Luong
et al., 2014), acoustic modeling of speech (Sak et al., 2014),
speech synthesis (Fan et al., 2014), protein secondary struc-
ture prediction (Sønderby & Winther, 2014), analysis of
audio (Marchi et al., 2014), and video data (Donahue et al.,
2014) among others.
The central idea behind the LSTM architecture is a memory
cell which can maintain its state over time, and non-linear
gating units which regulate the information ﬂow into and
out of the cell. Most modern studies incorporate many im-
provements that have been made to the LSTM architecture
since its original formulation (Hochreiter & Schmidhuber,
1995; 1997). However, LSTMs are now applied to many
learning problems which differ signiﬁcantly in scale and
nature from the problems that these improvements were ini-
tially tested on. A systematic study of the utility of various
computational components which comprise LSTMs (see
Figure 1) was missing. This paper ﬁlls that gap and sys-
tematically addresses the open question of improving the
LSTM architecture.
We evaluate the most popular LSTM architecture ( vanilla
LSTM; Section 2) and eight different variants thereof on
three benchmark problems: acoustic modeling, handwrit-
arXiv:1503.04069v1  [cs.NE]  13 Mar 2015
LSTM: A Search Space Odyssey
unweighted connection
Legend
weighted connection
connection with time-lag
mutliplication
+ sum over all inputs
branching point
gate activation function
(always sigmoid)
+
+
+
+
forget gate
input gate
block input
cell
+
output gate
peepholes
LSTM block
g
h
...
input
...
...
...
...
...
...
...
...
recurrent
...
input
recurrent
input
recurrent
input
recurrent
output
recurrent
+
g
SRN
unit
...
input
...
recurrent
...
...output
recurrent
g input activation function
(usually tanh)
h output activation function
(usually tanh)
block output
Figure 1.Detailed schematic of the Simple Recurrent Network (SRN) unit (left) and a Long Short-Term Memory block (right) as used
in the hidden layers of a recurrent neural network.
ing recognition and polyphonic music modeling. Each one
differs from the vanilla LSTM by a single change. This al-
lows us to isolate the effect of each of these changes on the
performance of the architecture. Random search (Ander-
son, 1953; Solis & Wets, 1981; Bergstra & Bengio, 2012) is
used to ﬁnd the best performing hyperparameters for each
variant on each problem, enabling a reliable comparison of
the performance of different variants. We also provide in-
sights gained about hyperparameters and their interaction
using fANOV A (Hutter et al., 2014).
2. Vanilla LSTM
The LSTM architecture most commonly used in litera-
ture was originally described by Graves & Schmidhuber
(2005).1 We refer to it as vanilla LSTM and use it as a
reference for comparison of all the variants. The vanilla
LSTM incorporates changes by Gers et al. (1999) and Gers
& Schmidhuber (2000) into the original LSTM (Hochreiter
& Schmidhuber, 1997) and uses full gradient training. Sec-
tion 3 provides descriptions of these major LSTM changes.
A schematic of the vanilla LSTM block can be seen in Fig-
ure 1. It features three gates (input, forget and output),
block input, a single cell (the Constant Error Carousel),
an output activation function, and peephole connections.
The output of the block is recurrently connected back to
the block input and all of the gates.
The vector formulas for a vanilla LSTM layer forward
pass are given below. The corresponding Back-Propagation
Through Time (BPTT) formulas can be found in supple-
1But note that some studies omit peephole connections.
mentary material. Here xt is the input vector at time t, the
W are rectangular input weight matrices, the R are square
recurrent weight matrices, the p are peephole weight vec-
tors and b are bias vectors. Functions σ, g and h are
point-wise non-linear activation functions: logistic sigmoid
( 1
1+e−x ) is used for as activation function of the gates and
hyperbolic tangent is usually used as the block input and
output activation function. The point-wise multiplication
of two vectors is denoted with ⊙:
zt = g(Wzxt + Rzyt−1 + bz) block input
it = σ(Wixt + Riyt−1 + pi ⊙ct−1 + bi) input gate
ft = σ(Wfxt + Rfyt−1 + pf ⊙ct−1 + bf) forget gate
ct = it ⊙zt + ft ⊙ct−1 cell state
ot = σ(Woxt + Royt−1 + po ⊙ct + bo) output gate
yt = ot ⊙h(ct) block output
3. History of LSTM
3.1. Original Formulation
This initial version of the LSTM block (Hochreiter &
Schmidhuber, 1995; 1997) included (possibly multiple)
cells, input and output gates, but no forget gate and no
peephole connections. The output gate, unit biases, or input
activation function were omitted for certain experiments.
Training was done using a mixture of Real Time Recur-
rent Learning (RTRL) and Backpropogation Through Time
(BPTT). Only the gradient of the cell was propagated back
through time, and the gradient for the other recurrent con-
LSTM: A Search Space Odyssey
nections was truncated. Thus, that study did not use the
exact gradient for training. Another feature of that version
was the use of full gate recurrence, which means that all
the gates received recurrent inputs from all gates at the pre-
vious time-step in addition to the recurrent inputs from the
block outputs. This feature did not appear in any of the
later papers.
3.2. Forget Gate
The ﬁrst paper to suggest a modiﬁcation of the LSTM ar-
chitecture introduced the forget gate (Gers et al., 1999), en-
abling the LSTM to reset its own state. This allowed learn-
ing of continual tasks such as embedded Reber grammar.
3.3. Peephole Connections
Gers & Schmidhuber (2000) argued that in order to learn
precise timings, the cell needs to control the gates. So far,
this was only possible through an open output gate. Peep-
hole connections (connections from the cell to the gates,
blue in Figure 1) were added to the architecture in order
to make precise timings easier to learn. Additionally, the
output activation function was omitted, as there was no ev-
idence that it was essential for solving the problems that
LSTM had been tested on so far.
3.4. Full Gradient
The ﬁnal modiﬁcation towards the vanilla LSTM was done
by Graves & Schmidhuber (2005). This study presented
the full backpropagation through time (BPTT) training for
LSTM networks with the architecture described in Sec-
tion 2, and presented results on the TIMIT benchmark. Us-
ing full BPTT had the added advantage that LSTM gra-
dients could be checked using ﬁnite differences, making
practical implementations more reliable.
3.5. Other Variants
Since its introduction the vanilla LSTM has been the most
commonly used architecture, but other variants have been
suggested too. Before the introduction of full BPTT train-
ing, Gers et al. (2002) utilized a training method based on
Extended Kalman Filtering which enabled the LSTM to be
trained on some pathological cases at the cost of high com-
putational complexity. Schmidhuber et al. (2007) proposed
using a hybrid evolution-based method instead of BPTT for
training but retained the vanilla LSTM architecture.
Bayer et al. (2009) evolved different LSTM block architec-
tures that maximize ﬁtness on context-sensitive grammars.
Sak et al. (2014) introduced a linear projection layer that
projects the output of the LSTM layer down before recur-
rent and forward connections in order to reduce the amount
of parameters for LSTM networks with many blocks. By
introducing a trainable scaling parameter for the slope of
the gate activation functions, Doetsch et al. (2014) were
able to improve the performance of LSTM on an ofﬂine
handwriting recognition dataset. In what they callDynamic
Cortex Memory, Otte et al. (2014) improved convergence
speed of LSTM by adding recurrent connections between
the gates of a single block (but not between the blocks).
Cho et al. (2014) proposed a simpliﬁcation of the LSTM ar-
chitecture called Gated Recurrent Unit (GRU). They used
neither peephole connections nor output activation func-
tions, and coupled the input and the forget gate into an up-
date gate. Finally, their output gate (called reset gate) only
gates the recurrent connections to the block input ( Wz).
Chung et al. (2014) performed an initial comparison be-
tween GRU and LSTM and reported mixed results.
4. Evaluation Setup
The focus of our study is to compare different LSTM vari-
ants, and not to achieve state-of-the-art results. Therefore,
our experiments are designed to keep the setup simple and
the comparisons fair. The vanilla LSTM is used as a base-
line and evaluated together with eight of its variants. Each
variant adds, removes or modiﬁes the baseline in exactly
one aspect, which allows to isolate their effect. Three dif-
ferent datasets from different domains are used to account
for cross-domain variations.
Since hyperparameter space is large and impossible to tra-
verse completely, random search was used in order to ob-
tain the best-performing hyperparameters (Bergstra & Ben-
gio, 2012) for every combination of variant and dataset.
Thereafter, all analyses focused on the 10% best perform-
ing trials for each variant and dataset (Section 5.1), making
the results representative for the case of reasonable hyper-
parameter tuning efforts. Random search was also chosen
for the added beneﬁt of providing enough data for analyz-
ing the general effect of various hyperparameters on the
performance of each LSTM variant (Section 5.2).
4.1. Datasets
Each dataset is split into three parts: a training set, a valida-
tion set, which is used for early stopping and for optimizing
the hyperparameters, and a test set for the ﬁnal evaluation.
Details of preprocessing for each dataset are provided in
the supplementary material.
4.1.1. TIMIT
The TIMIT Speech corpus (Garofolo et al., 1993) is large
enough to be a reasonable acoustic modeling benchmark
for speech recognition, yet it is small enough to keep a
large study such as ours manageable. Our experiments fo-
cus on the frame-wise classiﬁcation task for this dataset,
LSTM: A Search Space Odyssey
where the objective is to classify each audio-frame as one
of 61 phones. The performance is measured as classiﬁca-
tion error percentage. The training, testing and validation
sets are split in line with Halberstadt (1998) into 3696, 400
and 192 sequences, having 304 frames on average.
4.1.2. IAM O NLINE
The IAM Online Handwriting Database (Liwicki & Bunke,
2005) consists of English sentences as time series of pen
movements that have to be mapped to characters. The net-
work uses four input features: the change inxand ypen po-
sitions, the time since the current stroke started and a binary
value indicating whether the pen is lifted. The performance
is measured in terms of the Character Error Rate (CER) af-
ter decoding. The size of the dataset was halved by sub-
sampling, which makes the experiments to run 2×faster
without harming the performance. The training, testing and
validation sets contained 5355, 2956, 3859 sequences with
an average length of 334 frames.
4.1.3. JSB C HORALES
JSB Chorales (Allan & Williams, 2005) is a polyphonic
music modeling dataset. The preprocessed data consists of
sequences of binary vectors and the task is next-step pre-
diction. The performance metric used is the negative log-
likelihood on the validation/test set. The complete dataset
consists of 229, 76, and 77 sequences respectively with an
average length of 61.
4.2. Network Architectures & Training
Bidirectional LSTM (Graves & Schmidhuber, 2005),
which consists of two hidden layers, one processing the
input forwards and the other one backwards in time, both
connected to a single softmax output layer, was used for
TIMIT and IAM Online tasks. A normal LSTM with one
hidden layer and a sigmoid output layer was used for the
JSB Chorales task. As loss function we employed Cross-
Entropy Error for TIMIT and JSB Chorales, while for the
IAM Online task the Connectionist Temporal Classiﬁcation
(CTC) Error by Graves et al. (2006) was used. The initial
weights for all networks were drawn from a normal dis-
tribution with standard deviation of0.1. Training was done
using Stochastic Gradient Descent with Nesterov-style mo-
mentum (Sutskever et al., 2013) with updates after each
sequence. The learning rate was rescaled by a factor of
(1 −momentum). Gradients were computed using full
BPTT for LSTMs (Graves & Schmidhuber, 2005). Train-
ing stopped after 150 epochs or once there was no improve-
ment on the validation set for more than ﬁfteen epochs.
4.3. LSTM Variants
The vanilla LSTM from Section 2 is referred as Vanilla (V).
The derived eight variants of the V architecture are the fol-
lowing:
1. No Input Gate (NIG)
2. No Forget Gate (NFG)
3. No Output Gate (NOG)
4. No Input Activation Function (NIAF)
5. No Output Activation Function (NOAF)
6. No Peepholes (NP)
7. Coupled Input and Forget Gate (CIFG)
8. Full Gate Recurrence (FGR)
The ﬁrst six variants are self-explanatory. The CIFG vari-
ant uses only one gate for gating both the input and the cell
recurrent self-connection (an LSTM modiﬁcation proposed
in GRU (Cho et al., 2014)). This is equivalent to setting
ft = 1 −it instead of learning the forget gate weights in-
dependently. The FGR variant adds recurrent connections
between all the gates as in the original formulation of the
LSTM (Hochreiter & Schmidhuber, 1997). It adds nine
additional recurrent weight matrices, thus signiﬁcantly in-
creasing the number of parameters.
4.4. Hyperparameter Search
While there are other methods to efﬁciently search for good
hyperparameters (cf. Snoek et al. 2012; Hutter et al. 2011),
random search has a couple of advantages for our setting:
it is easy to implement, trivial to parallelize and covers
the search space more uniformly, thereby improving the
follow-up analysis of hyperparameter importance.
Each hyperparameter search consists of 200 trials (for a
total of 5400 trials) of randomly sampling the following
hyperparameters:
• number of LSTM blocks per hidden layer: log-uniform
samples from [20,200];
• learning rate: log-uniform samples from [10−6,10−2];
• momentum: 1 −log-uniform samples from [0.01,1.0];
• standard deviation of Gaussian input noise: uniform
samples from [0,1].
In the case of the TIMIT dataset, two additional (boolean)
hyperparameters were considered (not tuned for the other
two datasets):
The ﬁrst one was the choice between traditional momen-
tum and Nesterov-style momentum (Sutskever et al., 2013).
Our analysis showed that this had no measurable effect
on performance so the latter was arbitrarily chosen for all
LSTM: A Search Space Odyssey
further experiments. The second one was whether to clip
the gradients to the range [−1,1]. This turned out to hurt
overall performance,2 therefore the gradients were never
clipped in the case of the other two datasets.
Note that, unlike an earlier small scale study (Chung et al.,
2014), the number of parameters was not kept ﬁxed for all
variants. Since different variants can utilize their parame-
ters differently, ﬁxing this number can bias comparisons.
5. Results & Discussion
Each of the 5400 experiments was run on one of 128 AMD
Opteron CPUs at 2.5 GHz and took 24.3 h on average to
complete. This sums up to a total single-CPU computa-
tion time of just below 15 years. For TIMIT and JSB
Chorales the test set performance of the best setup were
29.6% classiﬁcation error (CIFG) and a log-likelihood of
-8.38 (NIG) respectively. For the IAM Online dataset our
best result was a Character Error Rate of 9.26% (NP) on
the test set. The best previously published result is 11.5%
CER by Graves et al. (2008) using a different and much
more extensive preprocessing. 3
5.1. Comparison of the Variants
The twenty best runs (according to validation set perfor-
mance) for each variant were compared with those for the
baseline architecture (V). Welch’s t-test at a signiﬁcance
level of p= 0.05 was used4 to determine whether the mean
test set performance of each variant was signiﬁcantly dif-
ferent from that of the baseline. A summary of the results is
shown in Figure 2. The box for each variant for which the
mean differs signiﬁcantly from the baseline is highlighted
in blue. The mean number of parameters used by those
twenty best performing networks are also shown as grey
bar plots in the background.
5.1.1. G ENERAL OBSERVATIONS
The ﬁrst important observation based on Figure 2 is that
removing the output activation function (NOAF) or the for-
get gate (NFG) signiﬁcantly hurt performance on all three
datasets. Apart from the CEC, the ability to forget old in-
formation and the squashing of the cell state appear to be
critical for the LSTM architecture. Indeed, without the out-
2Although this may very well be the result of the range having
been chosen too tightly.
3Note that these numbers differ from the best test-set perfor-
mances that can be found in Figure 2. This is the case because
here we only report the single best performing setup as deter-
mined on the validation set. In Figure 2 on the other hand we
show the test-set performance of the 20 best setups for each vari-
ant.
4We applied theBonferroni adjustment to correct for perform-
ing eight different tests (one for each variant).
V CIFG FGR NP NOG NIAF NIG NFG NOAF
27
28
29
30
31
32
33
34
35classiﬁcation error in %
TIMIT
10
15
20
25
30character error rate
IAM Online
V CIFG FGR NP NOG NIAF NIG NFG NOAF
8.4
8.5
8.6
8.7
8.8
negative log-likelihood
JSB Chorales
1
2
3
4
5
number of parameters ∗105
0.5
1.0
1.5
2.0
2.5
3.0
number of parameters ∗105
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
number of parameters ∗105
Figure 2.Test set performance for top 10% (according to the val-
idation set) hyperparameter settings for each dataset and variant.
Boxes show the range between the 25th and the 75th percentile of
the data, while the whiskers indicate the whole range. The red dot
represents the mean and the red line the median of the data. The
boxes of variants that differ signiﬁcantly from the vanilla LSTM
are shown in blue with thick lines. The grey histogram in the
background presents the average number of parameters for the
top 10% performers of every variant. Figure best viewed in color.
LSTM: A Search Space Odyssey
Figure 3.Pie charts showing which fraction of variance of the test
set performance can be attributed to each of the hyperparameters.
The percentage of variance that is due to interactions between
multiple parameters is indicated as “higher order.”
put activation function, the block output can in principle
grow unbounded. Coupling the input and the forget gate
avoids this problem and might render the use of an out-
put non-linearity less important, which could explain why
GRU performs well without it.
Input and forget gate coupling (CIFG) did not signiﬁcantly
change mean performance on any of the datasets, although
the best performance improved slightly on music mod-
elling. Similarly, removing peephole connections (NP) also
did not lead to signiﬁcant changes, but the best perfor-
mance improved slightly for handwriting recognition. Both
of these variants simplify LSTMs and reduce the computa-
tional complexity, so it might be worthwhile to incorporate
these changes into the architecture.
Adding full gate recurrence (FGR) did not signiﬁcantly
change performance on TIMIT or IAM Online, but led to
worse results on the JSB Chorales dataset. Given that this
variant greatly increases the number of parameters, we gen-
erally advice against using it. Note that this feature was
present in the original proposal of the LSTM (Hochreiter
& Schmidhuber, 1995; 1997), but has been absent in all
following studies.
5.1.2. T ASK -SPECIFIC OBSERVATIONS
Removing the input gate (NIG), the output gate (NOG) and
the input activation function (NIAF) led to a signiﬁcant re-
duction in performance on speech and handwriting recog-
nition. However, there was no signiﬁcant effect on music
modelling performance. A small (but statistically insignif-
icant) average performance improvement was observed for
the NIG and NIAF architectures on music modeling. We
hypothesize that these behaviors will generalize to simi-
lar problems such as language modeling. For supervised
learning on continuous real-valued data (such as speech
and handwriting recognition), the input gate, output gate
and input activation function are all crucial for obtaining
good performance.
5.2. Impact of Hyperparameters
The fANOV A framework for assesing hyperparameter im-
portance by Hutter et al. (2014) is based on the observation
that marginalizing over dimensions can be done efﬁciently
in regression trees. This allows predicting the marginal er-
ror for one hyperparameter while averaging over all the oth-
ers. Traditionally this would require a full hyperparameter
grid search, whereas here the hyperparameter space can be
sampled at random.
Average performance for any slice of the hyperparameter
space is obtained by ﬁrst training a regression tree and then
summing over its predictions along the corresponding sub-
set of dimensions. To be precise, a random regression-
forest of 100 trees is trained and their prediction perfor-
mance is averaged. This improves the generalization and
allows for an estimation of uncertainty of those predictions.
The obtained marginals can then be used to decompose
the variance into additive components using the functional
ANalysis Of V Ariance (fANOV A) method (Hooker, 2007)
which provides an insight into the overall importance of
hyperparameters and their interactions.
5.2.1. A NALYSIS OF VARIANCE
Figure 3 shows what fraction of the test set performance
variance can be attributed to different hyperparameters. It
is obvious that the learning rate is by far the most impor-
tant hyperparameter, always accounting for more than two
thirds of the variance. The next most important hyper-
parameter is the hidden layer size, followed by the input
noise, leaving the momentum with less than one percent of
the variance. Higher order interactions play an important
role in the case of TIMIT, but are much less important for
the other two data sets. The hyperparameter interplay is
further discussed in Section 5.2.6.
5.2.2. L EARNING RATE
Learning rate is the most important hyperparameter, there-
fore it is very important to understand how to set it correctly
in order to achieve good performance. Figure 4 shows (in
blue) how setting the learning rate value affects the pre-
dicted average performance on the test set. It is important
to note that this is an average over all other hyperparam-
eters and over all the trees in the regression-forest. The
LSTM: A Search Space Odyssey
shaded area around the curve indicates the standard devia-
tion over tree predictions (not over other hyperparameters),
thus quantifying the reliability of the average. The same is
shown in green with the predicted average training time.
The plots in Figure 3 show that the optimal value for the
learning rate is dependent on the dataset. For each dataset,
there is a large basin (up to two orders of magnitude) of
good learning rates inside of which the performance does
not vary much. A related but unsurprising observation is
that there is a sweet-spot for the learning rate at the high
end of the basin. 5 In this region, the performance is good
and the training time is small. So while searching for a
good learning rate for the LSTM, it is sufﬁcient to do a
coarse search by starting with a high value (e.g. 1.0) and
dividing it by ten until performance stops increasing.
Figure 3 also shows that the fraction of variance caused
by the learning rate is much bigger than the fraction due
to interaction between learning rate and hidden layer size
(some part of the “higher order” piece, for more see Sec-
tion 5.2.6). This suggests that the learning rate can be
quickly tuned on a small network and then used to train
a large one.
5.2.3. H IDDEN LAYER SIZE
Not surprisingly the hidden layer size is an important hy-
perparameter affecting the LSTM network performance.
As expected, larger networks perform better, and the re-
quired training time increases with the network size.
5.2.4. I NPUT NOISE
Additive Gaussian noise on the inputs, a traditional regular-
izer for neural networks, has been used for LSTM as well.
However, we ﬁnd that not only does it almost always hurt
performance, it also slightly increases training times. The
only exception is TIMIT, where a small dip in error for the
range of [0.2,0.5] is observed.
5.2.5. M OMENTUM
One unexpected result of this study is that momentum af-
fects neither performance nor training time in any signiﬁ-
cant way. This follows from the observation that for none
of the datasets, momentum accounted for more than 1% of
the variance of test set performance. It should be noted that
for TIMIT the interaction between learning rate and mo-
mentum accounts for 2.5% of the total variance, but as with
learning rate ×hidden size (cf. Section 5.2.6) it does not
reveal any interpretable structure. This may be the result of
our choice to scale learning rates dependent on momentum
(Section 4.2). These observations suggest that momentum
5Note that it is outside the plotted range for IAM Online and
JSB Chorales.
Figure 5.Left: The predicted marginal error for combinations of
learning rate and hidden size. Right: The component that is
solely due to the interaction of the two and cannot be attributet
to changes in one of them alone. In other words the difference to
the case of them being perfectly independent. (Blue is better than
red.)
does not offer substantial beneﬁts when training LSTMs
with online stochastic gradient descent. It may, however,
be more important in the case of batch training, where the
gradients are less noisy.
5.2.6. I NTERACTION OF HYPERPARAMETERS
Here we focus on the higher order interactions for the
TIMIT dataset, for which they were strongest, but our anal-
ysis revealed very similar behavior for the other datasets:
learning rate ×hidden size = 6.7%
learning rate ×input noise = 4.4%
hidden size ×input noise = 2.0%
learning rate ×momentum = 1.5%
momentum ×hidden size = 0.6%
momentum ×input noise = 0.4%
The interaction between learning rate and the hidden size
is the strongest one, but Figure 5 does not reveal any sys-
tematic dependence between the two. In fact it may be the
case that more samples would be needed in order to prop-
erly analyse the ﬁne interplay between them, but given our
observations so far this might not be worth the effort. In
any case, it is clear that varying the hidden size does not
change the region of optimal learning rate.
6. Conclusion
This paper reports the results of a large scale study on vari-
ations of the LSTM architecture. We conclude that:
• The most commonly used LSTM architecture (vanilla
LSTM) performs reasonably well on various datasets
and using any of eight possible modiﬁcations does not
signiﬁcantly improve the LSTM performance.
LSTM: A Search Space Odyssey
30
35
40
45
50
55
60
65Classification Error
34
36
38
40
42
44
46 TIMIT
37
38
39
40
41
42
43
20
40
60
80
100Character Error Rate
40
50
60
70
80
90
100 IAM Online
44
46
48
50
52
54
56
58
10-6 10-5 10-4 10-3 10-2
learning rate
8.5
9.0
9.5
10.0
10.5
11.0
11.5
12.0
12.5Negative Log Likelihood
error
time
20 40 60 80 100 120 140 160 180 200
hidden size
9.70
9.75
9.80
9.85
9.90
9.95
10.00
10.05
10.10
10.15 JSB Chorales
0.0 0.2 0.4 0.6 0.8 1.0
input noise standard deviation
9.5
9.6
9.7
9.8
9.9
10.0
10.1
10.2
0
5
10
15
20
25
30
35
40
45
0
20
40
60
80
100
0.20
0.25
0.30
0.35
0.40
0.45
0.50
0.55
0.60
5
10
15
20
25
30
35
40
0
20
40
60
80
100
0.0
0.5
1.0
1.5
2.0
2.5
3.0
17.0
17.5
18.0
18.5
19.0
19.5
20.0
20.5
21.0
21.5
total time in h
26
27
28
29
30
31
32
33
34
total time in h
0.40
0.45
0.50
total time in h
Figure 4.Predicted marginal error (blue) and marginal time for different values of the learning rate, hidden size, and the input noise
(columns) for all three datasets (rows). The shaded area indicates the standard deviation between the tree-predicted marginals and thus
indicating the reliability of the predicted mean performance. Note that each plot is for the vanilla LSTM but curves for all variants that
are not signiﬁcantly worse look very similar.
• Certain modiﬁcations such as coupling the input
and forget gates or removing peephole connections
simplify LSTM without signiﬁcantly hurting perfor-
mance.
• The forget gate and the output activation function are
the critical components of the LSTM block. While
the ﬁrst is crucial for LSTM performance, the second
is necessary whenever the cell state is unbounded.
• Learning rate and network size are the most crucial
tunable LSTM hyperparameters. Surprisingly, the use
of momentum was found to be unimportant (in our set-
ting of online gradient descent). Gaussian noise on the
inputs was found to be moderately helpful for TIMIT,
but harmful for other datasets.
• The analysis of hyperparameter interactions revealed
that even the highest measured interaction (between
learning rate and network size) is quite small. This
implies that the hyperparameters can be tuned inde-
pendently. In particular, the learning rate can be cali-
brated ﬁrst using a fairly small network, thus saving a
lot of experimentation time.
Neural networks can be tricky to use for many practition-
ers compared to other methods whose properties are al-
ready well understood. This has remained a hurdle for
newcomers to the ﬁeld since a lot of practical choices are
based on the intuitions of experts, and experiences gained
over time. With this study, we have attempted to back
some of these intuitions with experimental results. We have
also presented new insights, both on architecture selection
and hyperparameter tuning for LSTM networks which have
emerged as the method of choice for solving complex se-
quence learning problems. In future work, we will explore
more complex modiﬁcations of the LSTM architecture.
Acknowledgments
This research was supported by the Swiss National Sci-
ence Foundation grants “Theory and Practice of Rein-
forcement Learning 2” (#138219) and “Advanced Re-
inforcement Learning” (#156682), and by EU projects
“NASCENCE” (FP7-ICT-317662) and “NeuralDynamics”
(FP7-ICT-270247).
References
Allan, Moray and Williams, Christopher KI. Harmonising
chorales by probabilistic inference. Advances in neural
information processing systems, 17:25–32, 2005.
Anderson, R. L. Recent Advances in Finding Best Op-
erating Conditions. Journal of the American Statistical
LSTM: A Search Space Odyssey
Association, 48(264):789–798, December 1953. ISSN
0162-1459. doi: 10.2307/2281072. URL http://
www.jstor.org/stable/2281072.
Bayer, Justin, Wierstra, Daan, Togelius, Julian, and
Schmidhuber, J ¨urgen. Evolving memory cell struc-
tures for sequence learning. In Artiﬁcial Neural Net-
works–ICANN 2009 , pp. 755–764. Springer, 2009.
URL http://link.springer.com/chapter/
10.1007/978-3-642-04277-5_76 .
Bengio, Y ., Simard, P., and Frasconi, P. Learning long-term
dependencies with gradient descent is difﬁcult. IEEE
Transactions on Neural Networks, 5(2):157–166, March
1994. ISSN 1045-9227. doi: 10.1109/72.279181.
Bergstra, James and Bengio, Yoshua. Random search for
hyper-parameter optimization. The Journal of Machine
Learning Research, 13(1):281–305, 2012. URL http:
//dl.acm.org/citation.cfm?id=2188395.
Cho, Kyunghyun, van Merrienboer, Bart, Gulcehre, Caglar,
Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua.
Learning Phrase Representations using RNN Encoder-
Decoder for Statistical Machine Translation. arXiv
preprint arXiv:1406.1078 , 2014. URL http://
arxiv.org/abs/1406.1078.
Chung, Junyoung, Gulcehre, Caglar, Cho, KyungHyun,
and Bengio, Yoshua. Empirical Evaluation of Gated
Recurrent Neural Networks on Sequence Modeling.
arXiv:1412.3555 [cs] , December 2014. URL http:
//arxiv.org/abs/1412.3555.
Doetsch, Patrick, Kozielski, Michal, and Ney, Hermann.
Fast and robust training of recurrent neural networks for
ofﬂine handwriting recognition. In 14th International
Conference on Frontiers in Handwriting Recognition ,
2014. URL http://people.sabanciuniv.
edu/berrin/cs581/Papers/icfhr2014/
data/4334a279.pdf.
Donahue, Jeff, Hendricks, Lisa Anne, Guadarrama, Sergio,
Rohrbach, Marcus, Venugopalan, Subhashini, Saenko,
Kate, and Darrell, Trevor. Long-term Recurrent Convo-
lutional Networks for Visual Recognition and Descrip-
tion. arXiv:1411.4389 [cs] , November 2014. URL
http://arxiv.org/abs/1411.4389. arXiv:
1411.4389.
Fan, Yuchen, Qian, Yao, Xie, Fenglong, and Soong,
Frank K. TTS synthesis with bidirectional LSTM based
recurrent neural networks. In Proc. Interspeech, 2014.
Garofolo, JS, Lamel, LF, Fisher, WM, Fiscus, JG, Pal-
lett, DS, and Dahlgren, NL. DARPA TIMIT Acoustic-
Phonetic Continuous Speech Corpus CD-ROM. Na-
tional Institute of Standards and Technology , NTIS Or-
der No PB91-505065, 1993.
Gers, Felix A. and Schmidhuber, J ¨urgen. Recurrent nets
that time and count. In Neural Networks, 2000. IJCNN
2000, Proceedings of the IEEE-INNS-ENNS Interna-
tional Joint Conference on , volume 3, pp. 189–194.
IEEE, 2000. ISBN 0769506194.
Gers, Felix A., Schmidhuber, J ¨urgen, and Cummins, Fred.
Learning to forget: Continual prediction with LSTM. In
Artiﬁcial Neural Networks, 1999. ICANN 99. Ninth In-
ternational Conference on (Conf. Publ. No. 470) , vol-
ume 2, pp. 850–855, 1999.
Gers, Felix A., P ´erez-Ortiz, Juan Antonio, Eck, Douglas,
and Schmidhuber, J ¨urgen. DEFK-LSTM. In ESANN
2002, Proceedings of the 10th Eurorean Symposium on
Artiﬁcial Neural Networks, 2002.
Graves, A, Liwicki, M, Fernandez, S, Bertolami, R, Bunke,
H, and Schmidhuber, J. A Novel Connectionist System
for Improved Unconstrained Handwriting Recognition.
IEEE Transactions on Pattern Analysis and Machine In-
telligence, 31(5), 2009.
Graves, Alex and Schmidhuber, J ¨urgen. Framewise
phoneme classiﬁcation with bidirectional LSTM
and other neural network architectures. Neural
Networks, 18(5–6):602–610, July 2005. ISSN 0893-
6080. doi: 10.1016/j.neunet.2005.06.042. URL
http://www.sciencedirect.com/science/
article/pii/S0893608005001206.
Graves, Alex, Fern´andez, Santiago, Gomez, Faustino, and
Schmidhuber, J ¨urgen. Connectionist temporal classiﬁ-
cation: labelling unsegmented sequence data with recur-
rent neural networks. In Proceedings of the 23rd inter-
national conference on Machine learning, pp. 369–376,
2006. URL http://dl.acm.org/citation.
cfm?id=1143891.
Graves, Alex, Liwicki, Marcus, Bunke, Horst, Schmidhu-
ber, J¨urgen, and Fern´andez, Santiago. Unconstrained on-
line handwriting recognition with recurrent neural net-
works. In Advances in Neural Information Processing
Systems, pp. 577–584, 2008.
Graves, Alex, Mohamed, Abdel-rahman, and Hinton, Ge-
offrey. Speech recognition with deep recurrent neural
networks. arXiv preprint arXiv:1303.5778, 2013. URL
http://arxiv.org/abs/1303.5778.
Halberstadt, Andrew K. Heterogeneous acoustic mea-
surements and multiple classiﬁers for speech recogni-
tion. PhD thesis, Massachusetts Institute of Technology,
1998.
LSTM: A Search Space Odyssey
Hochreiter, Sepp. Untersuchungen zu dynamischen neu-
ronalen Netzen. Masters Thesis, Technische Universit ¨at
M¨unchen, M¨unchen, 1991.
Hochreiter, Sepp and Schmidhuber, J ¨urgen. Long
Short Term Memory. Technical Report FKI-207-95,
Technische Universit ¨at M ¨unchen, M ¨unchen, August
1995. URL http://citeseerx.ist.psu.edu/
viewdoc/summary?doi=10.1.1.51.3117.
Hochreiter, Sepp and Schmidhuber, J ¨urgen. Long Short-
Term Memory. Neural Computation, 9(8):1735–1780,
November 1997. ISSN 0899-7667. doi: 10.1162/neco.
1997.9.8.1735. URL http://www.bioinf.jku.
at/publications/older/2604.pdf.
Hooker, Giles. Generalized Functional ANOV A Diag-
nostics for High-Dimensional Functions of Dependent
Variables. Journal of Computational and Graphical
Statistics, 16(3):709–732, September 2007. ISSN 1061-
8600, 1537-2715. doi: 10.1198/106186007X237892.
URL http://www.tandfonline.com/doi/
abs/10.1198/106186007X237892.
Hutter, F., Hoos, H. H., and Leyton-Brown, K. Sequential
Model-Based Optimization for General Algorithm Con-
ﬁguration. In Proc. of LION-5, pp. 507523, 2011.
Hutter, Frank, Hoos, Holger, and Leyton-Brown, Kevin.
An Efﬁcient Approach for Assessing Hyperpa-
rameter Importance. pp. 754–762, 2014. URL
http://jmlr.org/proceedings/papers/
v32/hutter14.html.
Liwicki, Marcus and Bunke, Horst. IAM-OnDB-an on-line
English sentence database acquired from handwritten
text on a whiteboard. In Document Analysis and Recog-
nition, 2005. Proceedings. Eighth International Confer-
ence on, pp. 956–961. IEEE, 2005.
Luong, Thang, Sutskever, Ilya, Le, Quoc V ., Vinyals, Oriol,
and Zaremba, Wojciech. Addressing the Rare Word
Problem in Neural Machine Translation. arXiv preprint
arXiv:1410.8206, 2014. URL http://arxiv.org/
abs/1410.8206.
Marchi, E., Ferroni, G., Eyben, F., Gabrielli, L., Squar-
tini, S., and Schuller, B. Multi-resolution linear predic-
tion based features for audio onset detection with bidi-
rectional LSTM neural networks. In 2014 IEEE Inter-
national Conference on Acoustics, Speech and Signal
Processing (ICASSP), pp. 2164–2168, May 2014. doi:
10.1109/ICASSP.2014.6853982.
Otte, Sebastian, Liwicki, Marcus, and Zell, An-
dreas. Dynamic Cortex Memory: Enhancing Re-
current Neural Networks for Gradient-Based Se-
quence Learning. In Artiﬁcial Neural Networks
and Machine Learning – ICANN 2014 , number
8681 in Lecture Notes in Computer Science, pp.
1–8. Springer International Publishing, September
2014. ISBN 978-3-319-11178-0, 978-3-319-11179-7.
URL http://link.springer.com/chapter/
10.1007/978-3-319-11179-7_1 .
Pham, Vu, Bluche, Th ´eodore, Kermorvant, Christopher,
and Louradour, J ´erˆome. Dropout improves Recur-
rent Neural Networks for Handwriting Recognition.
arXiv:1312.4569 [cs] , November 2013. URL http:
//arxiv.org/abs/1312.4569.
Sak, Hasim, Senior, Andrew, and Beaufays, Franc ¸oise.
Long short-term memory recurrent neural network archi-
tectures for large scale acoustic modeling. In Proceed-
ings of the Annual Conference of International Speech
Communication Association (INTERSPEECH) , 2014.
URL http://193.6.4.39/˜czap/letoltes/
IS14/IS2014/PDF/AUTHOR/IS141304.PDF.
Schmidhuber, J, Wierstra, D, Gagliolo, M, and Gomez,
F J. Training Recurrent Networks by EVOLINO.Neural
Computation, 19(3):757–779, 2007.
Snoek, Jasper, Larochelle, Hugo, and Adams, Ryan P.
Practical Bayesian Optimization of Machine Learning
Algorithms. In Pereira, F., Burges, C. J. C., Bottou, L.,
and Weinberger, K. Q. (eds.), Advances in Neural Infor-
mation Processing Systems 25 , pp. 2951–2959. Curran
Associates, Inc., 2012.
Solis, Francisco J. and Wets, Roger J.-B. Mini-
mization by Random Search Techniques. Mathe-
matics of Operations Research , 6(1):19–30, February
1981. ISSN 0364-765X. doi: 10.1287/moor.6.1.
19. URL http://pubsonline.informs.org/
doi/abs/10.1287/moor.6.1.19.
Sønderby, Søren Kaae and Winther, Ole. Protein Sec-
ondary Structure Prediction with Long Short Term Mem-
ory Networks. arXiv:1412.7828 [cs, q-bio] , Decem-
ber 2014. URL http://arxiv.org/abs/1412.
7828. arXiv: 1412.7828.
Sutskever, Ilya, Martens, James, Dahl, George, and Hin-
ton, Geoffrey. On the importance of initialization and
momentum in deep learning. In JMLR, pp. 1139–1147,
2013. URL http://jmlr.org/proceedings/
papers/v28/sutskever13.html.
Zaremba, Wojciech, Sutskever, Ilya, and Vinyals,
Oriol. Recurrent Neural Network Regularization.
arXiv:1409.2329 [cs] , September 2014. URL http:
//arxiv.org/abs/1409.2329.
Supplementary Material for LSTM: A Search Space Odyssey
A. LSTM formulas
Here we repeat the vectorized formulas for a vanilla LSTM
layer forward pass from the paper, and then present the for-
mulas for the backward pass. We also provide formulas for
all the studied variants.
A.1. Forward Pass
Here we reproduce the formulas of the forward pass from
the paper, but we split all gates and the block input into
activity before (¯⋆) and after non-linearity (⋆).
Let N be the number of LSTM blocks and M the number
of inputs. Then we get the following weights:
• Input weights: Wz, Ws, Wf , Wo ∈RN×M
• Recurrent weights: Rz, Rs, Rf , Ro ∈RN×N
• Peephole weights: ps, pf , po ∈RN
• Bias weights: bz, bs, bf , bo ∈RN
As in the paper we have xt as the input vector at time t,
σ, gand hare pointwise non-linear functions with σ(x) =
1
1+e−x being the logistic sigmoid. The pointwise multipli-
cation of two vectors is denoted with ⊙.
¯zt = Wzxt + Rzyt−1 + bz
zt = g(¯zt) block input
¯it = Wixt + Riyt−1 + pi ⊙ct−1 + bi
it = σ(¯it) input gate
¯ft = Wf xt + Rf yt−1 + pf ⊙ct−1 + bf
ft = σ(¯ft) forget gate
ct = zt ⊙it + ct−1 ⊙ft cell
¯ot = Woxt + Royt−1 + po ⊙ct + bo
ot = σ(¯ot) output gate
yt = h(ct) ⊙ot block output
A.2. Backpropagation Through Time
Here ∆t is the vector of deltas passed down from the layer
above. If E is the loss function it formally corresponds
to ∂E
∂yt , but not including the recurrent dependencies. The
deltas inside the LSTM block are then calculated as fol-
lows:
δyt = ∆t + RT
z δzt+1 + RT
i δit+1 + RT
f δft+1 + RT
o δot+1
δot = δyt ⊙h(ct) ⊙σ′(¯ot)
δct = δyt ⊙ot ⊙h′(ct) + po ⊙δot + pi ⊙δit+1
+ pf ⊙δft+1 + δct+1 ⊙ft+1
δft = δct ⊙ct−1 ⊙σ′(¯ft)
δit = δct ⊙zt ⊙σ′(¯it)
δzt = δct ⊙it ⊙g′(¯zt)
The deltas for the inputs are only needed if there is a layer
below that needs training:
δxt = WT
z δzt + WT
i δit + WT
f δft + WT
o δot
Finally the gradients for the weights are calculated like this,
where again ⋆can be any of {z,i,f,o}:
δW⋆ =
T∑
t=0
⟨δ⋆t,xt⟩ δpi =
T−1∑
t=0
ct ⊙δit+1
δR⋆ =
T−1∑
t=0
⟨δ⋆t+1,yt⟩ δpf =
T−1∑
t=0
ct ⊙δft+1
δb⋆ =
T∑
t=0
δ⋆t δpo =
T∑
t=0
ct ⊙δot
Here ⟨⋆1,⋆2⟩denotes the outer product of two vectors.
Supplementary Material for LSTM: A Search Space Odyssey
A.3. Variants
We only report differences to the formulas from Sec-
tion A.1:
1. No Input Gate (NOG): it = 1
2. No Forget Gate (NFG): ft = 1
3. No Output Gate (NIG): ot = 1
4. No Input Activation Function (NIAF): g(x) = x
5. No Output Activation Function (NOAF): h(x) = x
6. Coupled Input and Forget Gate (CIFG):
ft = 1 −it
7. No Peepholes (NP):
¯it = Wixt + Riyt−1 + bi
¯ft = Wf xt + Rf yt−1 + bf
¯ot = Woxt + Royt−1 + bo
8. Full Gate Recurrence (FGR):
¯it =Wixt + Riyt−1 + pi ⊙ct−1 + bi
+ Riiit−1 + Rfi ft−1 + Roiot−1
¯ft =Wf xt + Rf yt−1 + pf ⊙ct−1 + bf
+ Rif it−1 + Rff ft−1 + Rof ot−1
¯ot =Woxt + Royt−1 + po ⊙ct−1 + bo
+ Rioit−1 + Rfo ft−1 + Rooot−1
B. Datasets
This section provides details on the datasets and their pre-
processing that were used in the LSTM comparison tasks.
B.1. TIMIT
We use the TIMIT Speech corpus (Garofolo et al., 1993) for
framewise phone classiﬁcation. The full set of 61 phones
were used as targets. From the raw audio we extract 12 Mel
Frequency Cepstrum Coefﬁcients (MFCCs) (Mermelstein,
1976) + energy over 25ms hamming-windows with stride
of 10ms and a pre-emphasis coefﬁcient of 0.97. These 13
inputs along with their ﬁrst and second derivatives com-
prise the 39 inputs to the network and are normalized to
have mean 0 and variance of 1.
We restrict our study to an established subset of the full
TIMIT corpus as detailed by Halberstadt (1998). In short
that means we only use the core tests set and drop the SA
samples from the training set. For validation we use some
of the discarded samples from the full test set.
(a)
Ben Zoma said: "The days of 1thy
life means in the day-time; all the days
of 1thy life means even at night-time ."
(Berochoth .) And the Rabbis thought
it important that when we read the
(b)
Figure 6.(a) Example board ( a08-551z, training set) from the
IAM-OnDB dataset and (b) its transcription into character label
sequences.
B.2. IAM Online
The IAM On-Line Handwriting Database (IAM-OnDB; Li-
wicki & Bunke 2005)6 was used for the handwriting exper-
iments in the IAM Online task. The IAM-OnDB dataset
splits into one training, two validation sets and one test set,
having 775, 192, 216 and 544 boards each. Each board, see
6(a), contains multiple hand-written lines. Each line splits
into strokes represented by sequences of 3-dimensional
vectors of x, y (a pen position) and t (time) coordinates.
Begins and ends of the characters within each stroke are
not explicitely marked. The stroke data were joint together
and a fourth dimension that contains value of 1 at the time
of the pen lifting (a transition to the next stroke) and zeroes
at all other time steps. Each handwriting line is accompa-
nied with a target character sequence, see 6(b) assembled
from the following 81 ASCII characters:
abcdefghijklmnopqrstuvwxyz
ABCDEFGHIJKLMNOPQRSTUVWXYZ
0123456789 !"#&\’()*+,-./[]:;?
The board labeled as a08-551z (in the training set) con-
tains a sequence of 11 percent (%) characters that does not
have an image in the strokes and the percent character does
not occur in any other board. The board was removed from
the experiments.
6The IAM-OnDB was obtained from http:
//www.iam.unibe.ch/fki/databases/
iam-on-line-handwriting-database
Supplementary Material for LSTM: A Search Space Odyssey
V CIFG FGR NP NOG NIAF NIG NFG NOAF
30
40
50
60
70
80
90
100classiﬁcation error in %
TIMIT
V CIFG FGR NP NOG NIAF NIG NFG NOAF
20
40
60
80
100character error rate
IAM Online
V CIFG FGR NP NOG NIAF NIG NFG NOAF
8.5
9.0
9.5
10.0
10.5
11.0
11.5
12.0
12.5
negative log-likelihood
JSB Chorales
0.5
1.0
1.5
2.0
2.5
number of parameters ∗105
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
1.8
number of parameters ∗105
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
number of parameters ∗105
Figure 7.Test set performance of for all 200 runs for each dataset and variant. Boxes show the range between the 25 th and the 75 th
percentile of the data, while the whiskers indicate the whole range. The red dot represents the mean and the red line the median of the
data. The boxes of variants that differ signiﬁcantly from the vanilla LSTM are shown in blue with thick lines. The grey histogram in the
background presents the average number of parameters for every variant.
The two validation sets were joint together. The training,
validation and testing sets contain 5 355, 2 956 and 3 859
lines. The sequences were subsampled to half the length
(they still contain enough information but it speeds up the
training). Instead of absolute pen positions their differences
were used. The data was standardized. No additional pre-
processing (like base-line straightening, cursive correction
etc.) was used. The CTC-error function by Graves et al.
(2006) was used for labeling the 81 characters and best-
path decoding was used for determining the Character Er-
ror Rate.
B.3. JSB Chorales
JSB Chorales is a collection of 382 four-part harmonized
chorales by J. S. Bach (Allan & Williams, 2005), con-
sisting of 202 chorales in major keys and 180 chorals in
minor keys. We used the preprocessed piano-rolls pro-
vided by Boulanger-Lewandowski et al. (2012) currently
available at http://www-etud.iro.umontreal.
ca/˜boulanni/icml2012. These piano-rolls were
generated by transposing each MIDI sequence in C major
or C minor and sampling frames every quarter note.
C. Additional plots
Here we present some additional plots that didn’t make it
into the paper.
C.1. Full Boxplot for all Variants
In Figure 7 a box-whiskers-plot of the performance overall
200 runs (in contrast to only the top 20 as in the paper) is
shown for every variant.
C.2. Performance and Time Scatterplots
A scatterplot of training time vs performance for all runs
can be seen in Figure 9, 10, and 11. The individual variants
are shown with different markers. We were hoping to iden-
tify some clusters, along the pareto front of that tradeoff.
But no such structure could be found.
C.3. Hyperparameter Interactions
In Figure 8 we visualize the interaction between all pairs
of hyperparamters. It is divided vertically into three sub-
plots, one for every dataset (TIMIT, IAM Online, and JSB
Chorales). The subplots itself are divided horizontally into
two parts, each containing a lower triangular matrix of
heatmaps. The rows and columns of these matrices rep-
resent the different hyperparameters (learning rate, mo-
mentum, hidden size, and input noise) and there is one
heatmap for every combination. The color encodes the
performance as measured by the Classiﬁcation Error for
TIMIT, Character Error Rate for IAM Online and Nega-
tive Log-Likelihood for the JSB Chorales Dataset. For all
datasets low (blue) is better than high (red).
Each heatmap in the left part shows marginal performance
for different values of the respective two hyperparamers.
This is the average performance predicted by the decision
forest when marginalizing over all other hyperparameters.
So each one is the 2D version of the performance plots from
Figure 4 in the paper.
The right side employs the idea of ANOV A to better illus-
trate the interaction between the hyperparameters. So we
removed the variance of performance that can be explained
by varying a single hyperparameter and only plot what is
left. For the case that two hyperparameters do not interact
Supplementary Material for LSTM: A Search Space Odyssey
at all (are perfectly independent) that residual would be all
zero (grey).
So look for example at the pair hidden size and learning
rate on the left side for the TIMIT dataset. We can see
that in general performance varies strongly along the x-axis
(learning rate), ﬁrst decreasing and then increasing again.
This is what we would expect knowing the valley-shape of
the learning rate from Figure 4 in the paper. Along the y-
axis (hidden size) performance seems to decrease slightly
from top to bottom. Again this is roughly what we would
expect.
Now let’s look at the same pair on the right side. This plot
shows how the heatmap on the left differs from the case of
the two hyperparameters being independent. So here a blue
pixel means, that the marginal error for this combination
of learning rate and hidden size is lower (better) than you
would expect. You will notice the scale is much smaller for
the right side (-3 to 3 as opposed to 32 to 60) and still many
of the heatmaps are close to grey.
References
Allan, Moray and Williams, Christopher KI. Harmonising
chorales by probabilistic inference. Advances in neural
information processing systems, 17:25–32, 2005.
Boulanger-Lewandowski, Nicolas, Bengio, Yoshua, and
Vincent, Pascal. Modeling Temporal Dependencies
in High-Dimensional Sequences: Application to Poly-
phonic Music Generation and Transcription. pp. 1159–
1166, 2012. URL http://icml.cc/discuss/
2012/590.html.
Garofolo, JS, Lamel, LF, Fisher, WM, Fiscus, JG, Pal-
lett, DS, and Dahlgren, NL. DARPA TIMIT Acoustic-
Phonetic Continuous Speech Corpus CD-ROM. Na-
tional Institute of Standards and Technology , NTIS Or-
der No PB91-505065, 1993.
Graves, Alex, Fern´andez, Santiago, Gomez, Faustino, and
Schmidhuber, J ¨urgen. Connectionist temporal classiﬁ-
cation: labelling unsegmented sequence data with recur-
rent neural networks. In Proceedings of the 23rd inter-
national conference on Machine learning, pp. 369–376,
2006. URL http://dl.acm.org/citation.
cfm?id=1143891.
Halberstadt, Andrew K. Heterogeneous acoustic mea-
surements and multiple classiﬁers for speech recogni-
tion. PhD thesis, Massachusetts Institute of Technology,
1998.
Liwicki, Marcus and Bunke, Horst. IAM-OnDB-an on-line
English sentence database acquired from handwritten
text on a whiteboard. In Document Analysis and Recog-
nition, 2005. Proceedings. Eighth International Confer-
ence on, pp. 956–961. IEEE, 2005.
Mermelstein, P. Distance measures for speech recognition:
Psychological and instrumental. In Chen, C. H. (ed.),
Pattern Recognition and Artiﬁcial Intelligence, pp. 374–
388. Academic Press, New York, 1976.
Supplementary Material for LSTM: A Search Space Odyssey
-2.0
-1.5
-1.0
-0.5
-0.0 momentum
TIMIT
1.0
1.2
1.5
1.8
2.0 hidden size
-5.9 -5.0 -4.0 -3.0 -2.0
learning rate
0.0
0.3
0.5
0.7
1.0 input noise std
-2.0 -1.5 -1.0 -0.5 -0.0
momentum
1.0 1.2 1.5 1.8 2.0
hidden size
33
36
39
42
45
48
51
54
57
-2.0
-1.5
-1.0
-0.5
-0.0 momentum
TIMIT
1.0
1.2
1.5
1.8
2.0 hidden size
-5.9 -5.0 -4.0 -3.0 -2.0
learning rate
0.0
0.3
0.5
0.7
1.0 input noise std
-2.0 -1.5 -1.0 -0.5 -0.0
momentum
1.0 1.2 1.5 1.8 2.0
hidden size
2.4
1.6
0.8
0.0
0.8
1.6
2.4
(a)
-2.0
-1.5
-1.0
-0.5
-0.0 momentum
IAMOnline
1.0
1.2
1.5
1.7
2.0 hidden size
-6.0 -5.0 -4.0 -3.0 -2.0
learning rate
0.0
0.3
0.5
0.7
1.0 input noise std
-2.0 -1.5 -1.0 -0.5 -0.0
momentum
1.0 1.2 1.5 1.7 2.0
hidden size
20
30
40
50
60
70
80
90
-2.0
-1.5
-1.0
-0.5
-0.0 momentum
IAMOnline
1.0
1.2
1.5
1.7
2.0 hidden size
-6.0 -5.0 -4.0 -3.0 -2.0
learning rate
0.0
0.3
0.5
0.7
1.0 input noise std
-2.0 -1.5 -1.0 -0.5 -0.0
momentum
1.0 1.2 1.5 1.7 2.0
hidden size
24
16
8
0
8
16
24
(b)
-2.0
-1.5
-1.0
-0.5
-0.0 momentum
JSB Chorales
1.0
1.2
1.5
1.8
2.0 hidden size
-6.0 -5.0 -4.0 -3.1 -2.1
learning rate
0.0
0.2
0.5
0.7
1.0 input noise std
-2.0 -1.5 -1.0 -0.5 -0.0
momentum
1.0 1.2 1.5 1.8 2.0
hidden size
8.8
9.2
9.6
10.0
10.4
10.8
11.2
11.6
12.0
-2.0
-1.5
-1.0
-0.5
-0.0 momentum
JSB Chorales
1.0
1.2
1.5
1.8
2.0 hidden size
-6.0 -5.0 -4.0 -3.1 -2.1
learning rate
0.0
0.2
0.5
0.7
1.0 input noise std
-2.0 -1.5 -1.0 -0.5 -0.0
momentum
1.0 1.2 1.5 1.8 2.0
hidden size
0.32
0.24
0.16
0.08
0.00
0.08
0.16
0.24
0.32
(c)
Figure 8.Total marginal predicted performance for all pairs of hyperparameters (left) and the variation only due to their interaction
(right).
Supplementary Material for LSTM: A Search Space Odyssey
30 40 50 60 70 80 90 100
Classiﬁcation Error
104
105
106
time
TIMIT
 V
CIFG
FGR
NFG
NIAF
NIG
NOAF
NOG
NP
Figure 9.Scatterplots for all 1800 experiments of the TIMIT Dataset. We show performance on the x-axis vs training time on the y-axis
(logarithmically). The variants are displayed with different colors or markers
Supplementary Material for LSTM: A Search Space Odyssey
20 40 60 80 100
Character Error Rate
104
105
106
time
IAM Online
 V
CIFG
FGR
NFG
NIAF
NIG
NOAF
NOG
NP
Figure 10.Scatterplots for all 1800 experiments of the IAM Online Dataset. We show performance on the x-axis vs training time on the
y-axis (logarithmically). The variants are displayed with different colors or markers
Supplementary Material for LSTM: A Search Space Odyssey
8.0 8.5 9.0 9.5 10.0 10.5 11.0 11.5 12.0 12.5
Negative Log-Likelihood
102
103
104
time
JSB Chorales
 V
CIFG
FGR
NFG
NIAF
NIG
NOAF
NOG
NP
Figure 11.Scatterplots for all 1800 experiments of the JSB Chorales Dataset. We show performance on the x-axis vs training time on
the y-axis (logarithmically). The variants are displayed with different colors or markers

COSET: A Benchmark for Evaluating
Neural Program Embeddings
Ke Wang
Visa Research
Palo Alto, CA 94306
kewang@visa.com
Mihai Christodorescu
Visa Research
Palo Alto, CA 94306
mihai.christodorescu@visa.com
Abstract
Neural program embedding can be helpful in analyzing large software, a task that is
challenging for traditional logic-based program analyses due to their limited scala-
bility. A key focus of recent machine-learning advances in this area is on modeling
program semantics instead of just syntax. Unfortunately evaluating such advances
is not obvious, as program semantics does not lend itself to straightforward metrics.
In this paper, we introduce a benchmarking framework called COSET for standard-
izing the evaluation of neural program embeddings. COSET consists of a diverse
dataset of programs in source-code format, labeled by human experts according to
a number of program properties of interest. A point of novelty is a suite of program
transformations included in COSET. These transformations when applied to the
base dataset can simulate natural changes to program code due to optimization and
refactoring and can serve as a “debugging” tool for classiﬁcation mistakes. We con-
ducted a pilot study on four prominent models—TreeLSTM [ 1], gated graph neural
network (GGNN) [ 2], AST–Path neural network (APNN) [ 3], and DYPRO [4].
We found that COSET is useful in identifying the strengths and limitations of
each model and in pinpointing speciﬁc syntactic and semantic characteristics of
programs that pose challenges.
1 Introduction
Analyzing large software artifacts (so-called “Big Code”) has proven extremely challenging for
logic-base approaches, which quickly force a trade-off between accuracy and scalability. Applying
deep learning architectures to this task is a promising direction that ﬂourished in the last few years,
with a number of techniques being proposed around a variety of features extracted from software
programs, ranging from syntactic to statically derived from source code to dynamically inferred from
execution traces. Since these models operate in separate application domains, it is hard to draw a
comparison. Our goal in this paper is to measure how precisely models can capture the program
semantics and to this end we propose a framework, COSET, to standardize the evaluation of neural
program embeddings.
The idea is to reuse the knowledge distilled from existing code repositories in an attempt to simplify
the future development of software, and in turn improve the product quality, is quite powerful. Code
completion for example is a signiﬁcant boost to programmer productivity and as such is a worthwhile
task to pursue. Early methods in the ﬁeld applied NLP techniques to discover the textual patterns
existed in the source code [5; 6; 7]; following approaches proposed to learn the syntactic program
embedding from the Abstract Syntax Trees (AST) [ 8; 9; 10]. Such approaches addressed code
completion, but fell short of more complex tasks such as program synthesis or repair, where thorough
understanding and precise representations of program semantics are required. A number of new
deep-learning architectures developed to speciﬁcally address this issue [2; 3; 4; 11; 12] tried a variety
Preprint. Under review.
arXiv:1905.11445v1  [cs.LG]  27 May 2019
of improvements, in terms of features (static features based on program source, dynamic featured
from the execution of the program [4], abstract features from symbolic program traces [11], features
from the graph representation of a program [2]). This diversity also makes it hard to evaluate these
approaches as they do not uniformly or universally improve on existing work, but rather exhibit
trade-offs across the natural characteristics of programs.
We propose a new benchmarking framework, called COSET, that aims to provide a consistent
baseline for evaluating the performance of any neural program embedding. COSET captures both the
human-induced variations in software (due to, e.g., coding style, algorithmic choices, code layout
and structure) and algorithmic transformations to the software (due to, e.g., software refactoring,
code optimization). COSET consists of a dataset of almost eighty-ﬁve thousand programs, developed
by a large number of users, solving one of ten coding problems, and labeled with a number of
characteristics of interest (e.g. running time, pre- and post-condition, loop invariant). This dataset is
paired with a number of program transformations, which when applied to any program in the dataset
can produce variations of that program. These two elements (the dataset and the transformations) gives
COSET the power to distinguish between various neural program embeddings with high precision.
One use of COSET is to provide a classiﬁcation task for the neural program embeddings under
evaluation. The benchmark will measure the accuracy and the stability of the embedding, giving the
user a precise answer on how various embeddings compare. A second use ofCOSET is as a debugging
tool: once a user determines that a neural program embedding fails short of some goal, the COSET
program transformations can be used (in delta debugging fashion) to identify the characteristic of the
programming language, the program code, or the program execution that causes the accuracy drop.
We have conducted a pilot study with this benchmarking framework on four of the latest deep learning
models. We selected DYPRO [4], TreeLSTM [1], the gated graph neural network (GGNN) [2], and the
AST-Path neural network (APNN) [3] as evaluation subjects for COSET. Through our comprehensive
evaluation, we ﬁnd that DYPRO achieves the best result in the semantics prediction task and is
signiﬁcantly more stable with its prediction results than static models, which have difﬁculties in
capturing the program semantics. As a result, static models are much less stable against the syntax
variations even when the semantics are preserved. On the other hand, the generalization from static
program features as done in GGNN and APNN, while insufﬁciently accurate, is scalable to large or
long-running programs, which overwhelm the dynamic model of DYPRO. Through careful use of
COSET’s debugging features, we then identify a number of speciﬁc shortcomings in the tested models,
from lack of support for variable types, to confusion about logging and other ancillary program
aspects, and to limitations in representing APIs.
We make the following contributions:
• We design COSET, a novel benchmark framework that expresses both human and algorithmic
variations in software artifacts.
• We propose COSET for evaluating how precise the models can learn to interpret the program se-
mantics and for identifying speciﬁc program characteristics that are the source of misclassiﬁcation.
• We present our evaluation results including the strength and weakness of each model, followed by
an in-depth discussion of our ﬁndings.
2 Motivation and Running Example
For an intuition on why programs are difﬁcult to learn, consider the program of Figure 1. The
method Program.Difference() takes as input an array, sorts it, and returns the difference between
the smallest value and the largest value.
To learn the semantics of this program sufﬁciently well as to classify it correctly as a sorting routine
using the Bubblesort strategy requires bridging the gap between the syntactic representation and its
runtime execution. This gap is typically expressed in abstract interpretation by considering a program
to be the sum-total of all its possible execution traces and considering a program’s source code to be
an abstraction of the program-as-traces model. Intuitively this distinction implies that learning to
classify programs may be limited in accuracy when using only the source code, and correspondingly
benchmarks should focus on samples whose source code and execution traces are not trivially related.
2
The gap between syntax and (runtime) semantics is characterized by several properties, which become
the requirements for our COSET benchmark. In other words, we wish for COSET to have enough
training and test samples to cover the whole range of distinctions between syntax and semantics.
1 public static int Difference ( int [] a)
2 {
3 int swappbit =1;
4 while ( swappbit != 0)
5 {
6 swappbit =0;
7 for ( int i =0; i < a. Length - 1;i ++)
8 {
9 int temp1 , temp2 ;
10 temp1 =a[i];
11 temp2 =a[i + 1];
12 if( temp1 < temp2 )
13 {
14 a[i]= temp2 ;
15 a[i + 1]= temp1 ;
16 swappbit =1;
17 }
18 }
19 }
20 int b=a [0] - a[a. Length - 1];
21 return b;
22 }
Figure 1: Sample code from COSET dataset,
sorting an array using a Bubblesort-style strat-
egy and then computing the difference between
the smallest and the largest elements.
First, the gap between program syntax and seman-
tics is manifested in that similar programs of similar
syntax may have to vastly different semantics. For
example, consider the two sorting implementations,
both sorting an array via two nested loops, both
comparing the current element to its successor, and
both swapping them if the order is incorrect. Yet
the two functions could implement different sort-
ing strategies, namely Bubblesort and Insertionsort.
Therefore minor syntactic discrepancies can lead to
signiﬁcant semantic differences. Our ﬁrst require-
ment for the benchmark is to include sufﬁcient sam-
ples that vary both syntactically and semantically,
though solving the same software task.
Second, program statements are almost never inter-
preted in the order in which the corresponding token
sequence appears in the source code (the only excep-
tion being straight-line programs, i.e., ones without
any control-ﬂow statements). For example, a condi-
tional statement only executes one branch each time,
but its token sequence is expressed sequentially as
multiple branch bodies. Similarly, when iterating
over a looping structure at runtime, it is unclear in
which order any two tokens are executed when considering different loop iterations. In Figure 1
the statement on line 23 may or may not execute after the statement on line 25 from a previous
loop iteration. A related characteristic is that the order of independent statements (e.g., lines 19
and 20) is arbitrary and the learning model needs to be evaluated for syntactic-ordering biases. The
second requirement for our benchmark is to include samples that vary only in the syntactic order of
statements, without changing semantics.
Third, many tokens in the source code are not relevant to program semantics. For example, variable
names do not affect the results computed by a program, but rather the dependencies between variables
(both data and control dependencies) play an essential role in deﬁning program semantics. In Figure 1
the variables swappbit, i, temp1, temp2, b, and a could have any other names, distinct from each
other, and the program would behave the same way. The third requirement for our benchmark is to
include samples with arbitrarily values for syntactic tokens that do not affect semantics.
3 Benchmark Design
The requirements from the previous section determine the need for a wide range and wide diversity
of programs, both in terms of source code and in terms of execution behavior. Before describing
COSET’s design, we add an additional requirement, that of non-adversarial, natural style for our
dataset samples. The goal of COSET is to provide the capability of testing learning models on their
accuracy in capturing program semantics, and we wish to explicitly exclude adversarial examples from
this benchmark. The rationale is that adversarial examples, which underwent program obfuscation
in order to make their analysis and understanding exceedingly hard, are an orthogonal problem
that requires separate solutions and accordingly a distinct benchmark. For example, a common
obfuscation used by malicious programs is that of embedding an interpreter. The original (malicious)
program is replaced by a new program that consists of an interpreter and an reimplementation of the
original program in a randomly generated language, to be handled by the interpreter. Understanding
any adversarial examples obfuscated this way requires addressing three tasks: (1) identifying the
presence of an interpreter, (2) learning the semantics of that interpreter, and (3) learning the semantics
of the source code embedded alongside of the interpreter. COSET as a benchmark focuses only on
task (3) and thus explicitly excludes adversarially generated samples.
3
With this additional requirement in mind, we design the benchmark to include samples that capture
normal variation in terms of syntax, style, algorithms, etc. arising in day-to-day programming work.
This means using program samples from a diverse set of programmers, all solving the same coding
problem, as well as including tools to transform these samples according to two common steps in the
software-development lifecycle, refactoring and optimization.
3.1 Program Dataset
The dataset consists of 84,165 programs in total. They are obtained from a popular online coding
platform. Programs were written in several different languages: Java, C# and Python. All programs
solve a particular coding problem. We hand picked the problems to ensure the diversity of the
programs in the dataset. Speciﬁcally it contains introductory programming exercises for beginners,
coding puzzles that exhibit considerable algorithmic complexity and challenging problems frequently
appearing on coding interviews. We split the whole dataset into a training set containing 64,165
programs, a validation set of 10,000 programs, and a test set of the remaining 10,000 programs.
3.2 Program Labels
The dataset have been manually analyzed and labeled on the basis of operational semantics ( e.g.
Bubblesort, Insertionsort, Mergesort, etc., for a sorting routine). We allow certain kind of variations
to keep the total number of labels manageable. For example, we ignore local variables allocated for
temporary storage, the iterative style of looping or recursion, sort order: descending or ascending, etc.
The work is done by fourteen PhD students and exchange scholars at University of California, Davis.
They come from different research backgrounds such as programming language, database, security,
graphics, machine learning, etc. All of them have been interviewed and tested for their knowledge on
program semantics. To reduce the labeling error, we distributed programs solving the same coding
problem mostly to a single person and have them cross check the results for validation. The whole
process took more than three months to complete. In the end, we deﬁned 38 labels in total, with more
than 2,000 programs on average for each label.1
3.3 Program Transformations
In this section, we introduce the transformations we apply to COSET for generating program variants.
They serve two purposes in our experiments: debugging the model and gauging the model stability.
The former refers to identifying the reason of misclassiﬁcation; the latter measures how stable are
the models against software variations, both of which will be explained in details in Section 4.
COSET considers three types of transformations, semantics-preserving, semantics-approximating,
and semantics-changing.
Semantics-Preserving Transformations: This category can be further split into compiler opti-
mizations (i.e. for improving the performance of a program in code compilation) and software
refactoring (i.e. for improving the code readability and maintainability in software development).
Even though they are designed for different settings and purposes, all of them preserve the semantics
of the original programs. In this paper we choose Constant and Variable Propagation (CVP), Dead
Code Elimination (DCE), Loop Unrolling (LU) and Hoisting for compiler optimization and Variable
Renaming (VR), Nested Condition Simpliﬁcation (NCS), Control Flag Removal (CFR) and Control
Statement Uniﬁcation (CSU) for software refactoring. Interested readers are invited to consult Aho et
al.’s compiler textbook [13] and the Eclipse IDE documentation [14; 15] for the examples of each
transformation.
Semantics-Approximating Transformations: Semantics-approximating transformations change
a program in minor ways producing a new program with semanticsclose to that of the original. In par-
ticular we adopt two kinds of transformations. First we change type names to others within the same
group. For example, we replaceint with long in the group of integer types, ordouble with float in
the group of ﬂoating-point types. Second we also change an API in the program to another of the sim-
ilar semantics such as changing Array.Sort(a:array) to Array.Sort(a:array,c:comparer)
from Microsoft’s .NET Framework for C#.
1Readers are encouraged to consult the supplemental material.
4
(a) Prediction (accuracy).
 (b) Prediction (F1 score).
(c) Scalability (accuracy).
 (d) Scalability (F1 score).
Figure 2: Evaluation Results.
Semantics-Changing Transformations: We also include transformations that change the seman-
tics of a program. Those transformations are necessary solely for the purpose of debugging a model.
Speciﬁcally COSET can remove error handling code from a program and can ﬁx a buggy program with
SARFGEN [16], a productized repair tool often used in Massive Open Online Courses (MOOCs).
4 Evaluation
In this section, we describe in details how we use COSET to evaluate the strengths and weaknesses of
each deep neural architecture by measuring two characteristics—accuracy and stability.
4.1 Evaluating Classiﬁcation Accuracy Using C OSET
We train GGNN, APNN, TreeLSTM and DYPRO on COSET to predict the semantic label of a test
program. In this experiment we use prediction accuracy and F1 score to measure classiﬁcation results.
Results. Figure 2a depicts the prediction accuracy of all models on COSET. Overall, DYPRO
comes on top as the most accurate model at 84.7%, with GGNN achieves 75.2% prediction accuracy.
In terms of F1 score (Figure 2b), DYPRO and GGNN keep their top positions while APNN and
TreeLSTM swap their places at the bottom of the ranking. The other metric of interest is scalability in
terms of program size. In order to compare GGNN, APNN, and TreeLSTM (which operate on source
code, typically measured as number of lines) on one hand, and DYPRO (which operates on execution
trace, measured as number of state changes) on the other, we unify the program-size dimension as
number of bytes, with a line of code being on average 15 bytes in length and a trace state changes
20 bytes. As shown in Figures 2c and 2d, accuracy quickly drops as program size increases, with
static models (GGNN, APNN, and TreeLSTM) losing all accuracy once a program exceeds 100 lines.
Discussion of Misclassiﬁcation Types. COSET is able to trigger many types of misclassiﬁcations.
To gain a better understanding of the errors each model made, we have conducted a large scale manual
inspection and summarize our ﬁndings below.
Types and Variable Names:Type name can cause issues. For example, Figure 3a shows a program
in COSET that causes APNN to misclassify. Replacing float to double ﬁxes this error. As a more
subtle example, program in Figure 3b is misclassiﬁed by all of the static models. The error is due to
variable col and row. Since vast majority of programs in COSET use row (resp. col) as the induction
variable in the outer (resp. inner) loop , static models do not recognize the same semantics a program
5
static int Average ( int [] a)
{
var b = 0;
foreach ( int num in a)
b += num ;
float c = (float)
(b / a. Length );
return c;
}
(a)
static void PrintChessBoard ()
{
for ( int col=0;col <8;col ++) {
for ( int row=0;row <8;row ++)
if ((col +row )%2 == 0)
Console . Write ("X");
else
Console . Write ("O");
Console . WriteLine (); }
}
(b)
static int Ksmallest (
int [] a, int k)
{
Array. Sort (a, (x, y)
=> y.CompareTo(x));
return a[k - 1];
}
(c)
static string Reversal ( string s)
{
string temp = s.ElementAt(0) ;
for ( int i=s. Length -2;i >=1;i - -)
temp = temp +s.ElementAt(i);
return temp +
s.ElementAt(s.Length - 1) ;
}
(d)
bool IsUglyNumber ( int num )
{
if (num <= 0)
{11
return false;
}11
for ( int i =2;i <6; i ++)
{
while ( num %i ==0)
num /= i;
}
bool isUgly = num ==1;
return isUgly ;
}
(e)
char FindExtraChar ( string s,
string t)
{
int [] arr = new int [26];
int ground =
char.GetNumericValue(’a’);
for ( int i =0;i<s. Length ;i ++) {
int val =
char.GetNumericValue(s[i]);
arr [ val - ground ]++;}
for ( int j =0;j<t. Length ;j ++) {
int val =
char.GetNumericValue(t[i]);
if(-- arr [ val - ground ] < 0)
return s[j ];}
return '0';
}
(f)
static int CountParentheses ( string s)
{
int open = 0, close = 0, depth = 0;
foreach ( char c in s)
{
switch (c) {
case '(':
open ++;
if (open - depth > close )
depth ++;
break ;
case ')':
close ++;
break ;
default:
}}
if ( open != close )
depth = 0;
return depth ;
}
(g)
static int Difference ( int [] a)
{
int swappbit = 1;
while (swappbit != 0) {
swappbit = 0;
for ( int i =0;i<a. Length -1; i ++){
int temp1 , temp2 ;
temp1 = a[i];
temp2 = a[i + 1];
if ( temp1 > temp2 ) {
a[i] = temp2 ;
a[i + 1] = temp1 ;
swappbit = 1;
}}}111
return a[a. Length - 1] - a [0];
}
(h)
Figure 3: Misclassiﬁcations triggered by COSET. Underlined code is the root cause of the misclassiﬁ-
cation, as identiﬁed by COSET’s debugging capability. See Section 4 for a detailed analysis.
denotes after some of its variables (i.e. row and col) are interchanged. It sufﬁces to conclude that
static models weigh variable names in their prediction, where in fact they should not be considered
for any semantics-based classiﬁcation task.
Syntactic Structure: To capture human programming habits in terms of code style and code structure,
COSET exposes a uneven distribution of code samples in terms of program syntax ( i.e. some are
more frequently used than others), which helps to trigger misclassiﬁcations. For example, the switch
statement in program 3g is the reason of the error. In other words, models would have correctly
classiﬁed the program if the switch is converted to a if-else, a far more popular selection statement.
Scalability: COSET also reveals scalability being a major issue among all models including both
static and dynamic. As the size of programs/execution traces increases, all models suffer notable
drops in accuracy (resp. F1 score) depicted in Figure 2c ( resp. Figure 2d). In particular APNN is
shown to be the most vulnerable.
API Usage: COSET discloses issues among static models in generalizing API calls. Figures 3c and 3d
show two examples. Figure 3c (resp. Figure 3d) can be ﬁxed by replacing the underlined API with
Array.Sort(a) (resp. indexing operator []). Granted, COSET bears some blame as it contains very
few programs of the same label as the two examples that has the exact identical API signatures, but
to expect a sufﬁcient coverage of all API prototypes (e.g. 17 in total for Array.Sort according to
Microsoft .NET framework 4.7) is a challenging task for any dataset. Therefore generalizing to
largely similar despite partially unseen API is necessary and immediate. In addition, in both cases
such API variations account for a small portion of the whole program. In particular we ﬁnd 86.2%
6
Models Compiler Optimization Software Refactoring
CVP DCE LU Hoisting VR NCS CFR CSU
GGNN 4.2% 5.6% 19.1% 7.8% 8.4% 13.5% 11.0% 17.2%
APNN 7.4% 12.7% 27.7% 11.7% 7.1% 16.3% 14.9% 22.8%
TreeLSTM 7.2% 5.5% 14.6% 6.4% 5.4% 12.8% 8.5% 19.7%
DYPRO 2.6% 4.5% 5.3% 1.7% 0.0% 0.0% 0.0% 0.0%
Table 1: Results of measuring the stability using COSET, as percentage of changed predictions when
applying semantics-preserving transformations.
(resp. 68.9%) of the programs in the training set have a very similar body (i.e. differ by less than eight
tokens) to Figure 3c (resp. Figure 3d) according to DECKARD [17], arguably the state-of-the-art
clone detection algorithm. As a result, it is reasonable to expect static models to be immune from the
minor API changes.
Error Handling: COSET shows error handling code can be another cause of misclassiﬁcation.
Figure 3e depicts an example that is misclassiﬁed by APNN and TreeLSTM. By removing the if
clause at the beginning of the function, both models produce the correct result. This is another
example COSET demonstrates that static models are unstable against syntax change that barely affects
the program semantics.
Buggy Code: COSET reveals the DYPRO model as the most susceptible to buggy programs. The
reason being the dynamic nature of program representation DYPRO adopts. That is if there is a bug
in the program, its runtime dynamic traces will likely lead to greater discrepancies than the syntactic
representations (e.g. tokens, ASTs, etc.). Take the program in Figure 3f for example, the bug lies
in the API call in int val = char.GetNumericValue(‘a’) which converts a character of digit to an
integer. In contrast what the programmer intended is to directly get the ascii code/unicode of the
character using int val = ‘a’. Unfortunately this bug leads to a chaotic dynamic representation (i.e.
char.GetNumericValue() will be evaluated to -1 for any non-digit character), DYPRO is not able to
overcome. In comparison, all static models display a stronger fault tolerance by correctly classifying
the buggy program.
Semantic Property: COSET gives insights into the limitations of static models in learning semantic
program properties. The program in Figure 3h triggers a misclassiﬁcation of this kind on all static
models. Worth mentioning the program does not implement a standard Bubblesort strategy, in which
the inner loop systematically shifts its bound towards the beginning of an array after bigger numbers
are being bubbled up to the end of the array. Instead the inner loop in the program keeps sorting the
entire array until reaching an iteration where no items are swapped, at which point the array is sorted.
Our methodology for analyzing this misclassiﬁcation starts with formally deﬁning the semantic label
L of Bubblesort to apply to programs that have a function satisfying the following properties:
1. time complexity of O(n2);
2. input type signature ofa : array and post-condition that the arraya is sorted, ∀i ∈[0, a.Length−
1). a[i] ≤a[i + 1];
3. having a nested loop structure and an invariant of the outer loop assuming i is the number of its
iterations, ∀j ∈[a.Length −i, a.Length−1]. a[j −1] ≤a[j].
Similarly we give deﬁnitions of all other labels in COSET. To identify the source of the misclassiﬁ-
cation (i.e. a program not being classiﬁed as L = Bubblesort), we repeat the classiﬁcation task for
a broader set of classes, using one property at a time as a new label deﬁnition. For example, using
property (1) alone we collect all programs with time complexity O(n2) in COSET as the training
data for a label L′. Similarly for property (2) we use all programs that implement a sorting routine
and label them L′′; and for (3) we collect all programs that display the bubbling behavior under
label L′′′. Results show static models incorrectly predict the program in Figure 3h only when using
property (3) as label deﬁnition. In other words, the static models GGNN, APNN, and TreeLSTM fail
to learn a sufﬁciently expressive discriminator to understand the non-standard bubbling sort strategy.
This methodology demonstrates COSET’s utility in discovering the limitations of models in learning
semantic properties.
7
4.2 Evaluating Model Stability Using C OSET
In this experiment we evaluate how stable the models are with their predictions. The reason stability
is a relevant and important metric is because changes in source code are inevitable, and a model
that is resilient against changes, especially the semantics-preserving kind, will yield many beneﬁts.
As mentioned in the benchmark design (Section 3), COSET applies normal, standard, and widely
adopted program transformations to simulate how software evolves during its normal lifecycle (e.g.
code optimization or software refactoring).
Given the models that are trained for the experiment in Section 4.1 and COSET’s testing program, we
apply code transformations to generate a new test set, preserving semantics for all samples. Programs
with no applicable transformations are excluded from the new test set. We then examine if models
hold on to their prior predictions. Table 1 depicts the results. The number in each cell denotes the
percentage of programs in the new test set on which a model changed its prediction. Overall, all
models display decent stability against software transformations. No transformation was able to
sway 30% or more predictions for any model. For the static models, TreeLSTM is shown to be the
most stable deep neural architecture while APNN is the most sensitive model to the transformations.
Unsurprisingly, DYPRO handles almost all semantics-preserving transformations as its features are
extracted from execution traces and thus largely unaffected by these source-code transformations.
5 Related Work
MNIST [18] (Modiﬁed National Institute of Standards and Technology database) is a database of
handwritten digits that is widely used for training and testing machine learning algorithms. The
MNIST database contains 60,000 training images and 10,000 testing images. There have been
many attempts to achieve the lowest error rate, among which some indeed have accomplished "near-
human performance". ImageNet [ 19] is a visual database designed for visual object recognition
research. More than 14 million images have been hand-annotated to indicate what objects are pictured.
ImageNet contains more than 20,000 categories, each of which consists of several hundred images.
Since 2010, ImageNet has been adopted in an annual contest, the ImageNet Large Scale Visual
Recognition Challenge (ILSVRC), where machine learning models compete to correctly classify and
detect objects and scenes. CIFAR-10 [20] (Canadian Institute For Advanced Research) is another
database of images that are commonly used to train machine learning and computer vision algorithms.
It is one of the most widely used datasets for machine learning research. The CIFAR-10 dataset
contains 60,000 32x32 color images in 10 different classes e.g. cars, birds, cats, etc. Labeled Faces in
the Wild [21] is a database of face photographs designed for studying the problem of unconstrained
face recognition. The dataset contains more than 13,000 images of faces collected from the web.
Each face has been labeled with the name of the person pictured. 1680 of the people pictured have
two or more distinct photos in the data set. The only constraint on these faces is that they were
detected by the Viola-Jones face detector.
Unlike the existing databases designed almost exclusively for training and testing machine learning
models in computer vision and image processing domain, we present COSET to evaluate how precise
deep neural architecture can learn the semantics of a program. In addition we also introduce a
suite of program transformations for assessing the stability of model prediction and debugging the
classiﬁcation mistake.
6 Conclusion
We introduce a benchmark framework calledCOSET for evaluating the accuracy and stability of neural
program embeddings and related neural network architectures proposed for software classiﬁcation
task. COSET consists of a set of natural, non-adversarial source-code samples from a variety of
programmers tasked with well-deﬁned coding challenges, and supplements this set with source-code
transformations representative of common software optimization and refactoring. In our evaluation
we observed that COSET was effective in measuring differences among various models proposed in
literature and provided debugging capabilities to identify the root causes of misclassiﬁcations.
8
References
[1] Kai Sheng Tai, Richard Socher, and Christopher D. Manning. Improved semantic representations
from tree-structured long short-term memory networks. CoRR, abs/1503.00075, 2015. URL
http://arxiv.org/abs/1503.00075.
[2] Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. Learning to represent
programs with graphs. arXiv preprint arXiv:1711.00740, 2017.
[3] Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: Learning distributed
representations of code. arXiv preprint arXiv:1803.09473, 2018.
[4] Ke Wang. Learning scalable and precise representation of program semantics. arXiv preprint
arXiv:1905.05251, 2019.
[5] Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. On the
naturalness of software. In Software Engineering (ICSE), 2012 34th International Conference
on, pages 837–847. IEEE, 2012.
[6] Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. Deepﬁx: Fixing common c
language errors by deep learning. 2017.
[7] Yewen Pu, Karthik Narasimhan, Armando Solar-Lezama, and Regina Barzilay. sk_p: a
neural program corrector for moocs. In Companion Proceedings of the 2016 ACM SIGPLAN
International Conference on Systems, Programming, Languages and Applications: Software for
Humanity, pages 39–40. ACM, 2016.
[8] Chris Maddison and Daniel Tarlow. Structured generative models of natural source code. In
International Conference on Machine Learning, pages 649–657, 2014.
[9] Pavol Bielik, Veselin Raychev, and Martin Vechev. Phog: probabilistic model for code. In
International Conference on Machine Learning, pages 2933–2942, 2016.
[10] Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional neural networks over tree
structures for programming language processing. 2016.
[11] Jordan Henkel, Shuvendu Lahiri, Ben Liblit, and Thomas Reps. Code vectors: Understanding
programs through embedded abstracted symbolic traces. arXiv preprint arXiv:1803.06686,
2018.
[12] Daniel DeFreez, Aditya V . Thakur, and Cindy Rubio-González. Path-based function embeddings.
In Proceedings of the 40th International Conference on Software Engineering: Companion
Proceeedings, ICSE ’18, pages 430–431, New York, NY , USA, 2018. ACM. ISBN 978-1-4503-
5663-3. doi: 10.1145/3183440.3195042. URL http://doi.acm.org/10.1145/3183440.
3195042.
[13] Alfred V . Aho, Monica S. Lam, Ravi Sethi, and Jeffrey D. Ullman. Compilers: Principles,
Techniques, and Tools (2Nd Edition). Addison-Wesley Longman Publishing Co., Inc., Boston,
MA, USA, 2006. ISBN 0321486811.
[14] A brief history of eclipse. https://www.ibm.com/developerworks/rational/library/
nov05/cernosek/index.html. Accessed: 2019-05-22.
[15] Greg Goth. Beware the march of this ide: Eclipse is overshadowing other tool technologies.
IEEE software, 22(4):108–111, 2005.
[16] Ke Wang, Rishabh Singh, and Zhendong Su. Search, align, and repair: Data-driven feedback
generation for introductory programming exercises. In Proceedings of the 39th ACM SIGPLAN
Conference on Programming Language Design and Implementation, PLDI 2018, pages 481–495,
New York, NY , USA, 2018. ACM. ISBN 978-1-4503-5698-5. doi: 10.1145/3192366.3192384.
URL http://doi.acm.org/10.1145/3192366.3192384.
9
[17] Lingxiao Jiang, Ghassan Misherghi, Zhendong Su, and Stephane Glondu. Deckard: Scalable
and accurate tree-based detection of code clones. In Proceedings of the 29th International
Conference on Software Engineering, ICSE ’07, pages 96–105, Washington, DC, USA, 2007.
IEEE Computer Society. ISBN 0-7695-2828-7. doi: 10.1109/ICSE.2007.30. URL https:
//doi.org/10.1109/ICSE.2007.30.
[18] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning
applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
[19] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale
Hierarchical Image Database. In CVPR09, 2009.
[20] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report,
Citeseer, 2009.
[21] Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. Labeled faces in the
wild: A database forstudying face recognition in unconstrained environments. In Workshop on
faces in’Real-Life’Images: detection, alignment, and recognition, 2008.
10

CodeBERT:
A Pre-Trained Model for Programming and Natural Languages
Zhangyin Feng1∗, Daya Guo2∗, Duyu Tang3, Nan Duan3, Xiaocheng Feng1
Ming Gong4, Linjun Shou4, Bing Qin1, Ting Liu1, Daxin Jiang4, Ming Zhou3
1 Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology, China
2 The School of Data and Computer Science, Sun Yat-sen University, China
3 Microsoft Research Asia, Beijing, China
4 Microsoft Search Technology Center Asia, Beijing, China
{zyfeng,xcfeng,qinb,tliu}@ir.hit.edu.cn
guody5@mail2.sysu.edu.cn
{dutang,nanduan,migon,lisho,djiang,mingzhou}@microsoft.com
Abstract
We present CodeBERT, a bimodal pre-trained
model for programming language (PL) and
natural language (NL). CodeBERT learns
general-purpose representations that support
downstream NL-PL applications such as nat-
ural language code search, code documen-
tation generation, etc. We develop Code-
BERT with Transformer-based neural architec-
ture, and train it with a hybrid objective func-
tion that incorporates the pre-training task of
replaced token detection, which is to detect
plausible alternatives sampled from generators.
This enables us to utilize both “ bimodal” data
of NL-PL pairs and “ unimodal” data, where
the former provides input tokens for model
training while the latter helps to learn bet-
ter generators. We evaluate CodeBERT on
two NL-PL applications by ﬁne-tuning model
parameters. Results show that CodeBERT
achieves state-of-the-art performance on both
natural language code search and code docu-
mentation generation. Furthermore, to inves-
tigate what type of knowledge is learned in
CodeBERT, we construct a dataset for NL-PL
probing, and evaluate in a zero-shot setting
where parameters of pre-trained models are
ﬁxed. Results show that CodeBERT performs
better than previous pre-trained models on NL-
PL probing.1
1 Introduction
Large pre-trained models such as ELMo (Peters
et al., 2018), GPT (Radford et al., 2018), BERT
(Devlin et al., 2018), XLNet (Yang et al., 2019)
∗Work done while this author was an intern at Microsoft
Research Asia.
1 All the codes and data are available at https://
github.com/microsoft/CodeBERT
and RoBERTa (Liu et al., 2019) have dramati-
cally improved the state-of-the-art on a variety of
natural language processing (NLP) tasks. These
pre-trained models learn effective contextual repre-
sentations from massive unlabeled text optimized
by self-supervised objectives, such as masked
language modeling, which predicts the original
masked word from an artiﬁcially masked input
sequence. The success of pre-trained models in
NLP also drives a surge of multi-modal pre-trained
models, such as ViLBERT (Lu et al., 2019) for
language-image and VideoBERT (Sun et al., 2019)
for language-video, which are learned from bi-
modal data such as language-image pairs with bi-
modal self-supervised objectives.
In this work, we present CodeBERT, abimodal
pre-trained model for natural language (NL) and
programming language (PL) like Python, Java,
JavaScript, etc. CodeBERT captures the seman-
tic connection between natural language and pro-
gramming language, and produces general-purpose
representations that can broadly support NL-PL
understanding tasks (e.g. natural language code
search) and generation tasks (e.g. code documen-
tation generation). It is developed with the multi-
layer Transformer (Vaswani et al., 2017), which is
adopted in a majority of large pre-trained models.
In order to make use of both bimodal instances
of NL-PL pairs and large amount of available uni-
modal codes, we train CodeBERT with a hybrid
objective function, including standard masked lan-
guage modeling (Devlin et al., 2018) and replaced
token detection (Clark et al., 2020), where uni-
modal codes help to learn better generators for
producing better alternative tokens for the latter
objective.
We train CodeBERT from Github code reposito-
arXiv:2002.08155v4  [cs.CL]  18 Sep 2020
ries in 6 programming languages, where bimodal
datapoints are codes that pair with function-level
natural language documentations (Husain et al.,
2019). Training is conducted in a setting similar
to that of multilingual BERT (Pires et al., 2019),
in which case one pre-trained model is learned for
6 programming languages with no explicit mark-
ers used to denote the input programming lan-
guage. We evaluate CodeBERT on two down-
stream NL-PL tasks, including natural language
code search and code documentation generation.
Results show that ﬁne-tuning the parameters of
CodeBERT achieves state-of-the-art performance
on both tasks. To further investigate what type of
knowledge is learned in CodeBERT, we construct
a dataset for NL-PL probing, and test CodeBERT
in a zero-shot scenario, i.e. without ﬁne-tuning the
parameters of CodeBERT. We ﬁnd that CodeBERT
consistently outperforms RoBERTa, a purely natu-
ral language-based pre-trained model. The contri-
butions of this work are as follows:
•CodeBERT is the ﬁrst large NL-PL pre-
trained model for multiple programming lan-
guages.
•Empirical results show that CodeBERT is ef-
fective in both code search and code-to-text
generation tasks.
•We further created a dataset which is the ﬁrst
one to investigate the probing ability of the
code-based pre-trained models.
2 Background
2.1 Pre-Trained Models in NLP
Large pre-trained models (Peters et al., 2018; Rad-
ford et al., 2018; Devlin et al., 2018; Yang et al.,
2019; Liu et al., 2019; Raffel et al., 2019) have
brought dramatic empirical improvements on al-
most every NLP task in the past few years. Suc-
cessful approaches train deep neural networks on
large-scale plain texts with self-supervised learning
objectives. One of the most representative neural
architectures is the Transformer (Vaswani et al.,
2017), which is also the one used in this work. It
contains multiple self-attention layers, and can be
conventionally learned with gradient decent in an
end-to-end manner as every component is differen-
tiable. The terminology “self-supervised” means
that supervisions used for pre-training are auto-
matically collected from raw data without manual
annotation. Dominant learning objectives are lan-
guage modeling and its variations. For example,
in GPT (Radford et al., 2018), the learning objec-
tive is language modeling, namely predicting the
next word wk given the preceding context words
{w1,w2,...,w k−1}. As the ultimate goal of pre-
training is not to train a good language model, it is
desirable to consider both preceding and following
contexts to learn better general-purpose contextual
representations. This leads us to the masked lan-
guage modeling objective used in BERT (Devlin
et al., 2018), which learns to predict the masked
words of a randomly masked word sequence given
surrounding contexts. Masked language modeling
is also used as one of the two learning objectives
for training CodeBERT.
2.2 Multi-Modal Pre-Trained Models
The remarkable success of the pre-trained model
in NLP has driven the development of multi-modal
pre-trained model that learns implicit alignment
between inputs of different modalities. These mod-
els are typically learned from bimodal data, such
as pairs of language-image or pairs of language-
video. For example, ViLBERT (Lu et al., 2019)
learns from image caption data, where the model
learns by reconstructing categories of masked im-
age region or masked words given the observed
inputs, and meanwhile predicting whether the cap-
tion describes the image content or not. Simi-
larly, VideoBERT (Sun et al., 2019) learns from
language-video data and is trained by video and
text masked token prediction. Our work belongs
to this line of research as we regard NL and PL
as different modalities. Our method differs from
previous works in that the fuels for model train-
ing include not only bimodal data of NL-PL pairs,
but larger amounts of unimodal data such as codes
without paired documentations.
A concurrent work (Kanade et al., 2019) uses
masked language modeling and next sentence pre-
diction as the objective to train a BERT model on
Python source codes, where a sentence is a log-
ical code line as deﬁned by the Python standard.
In terms of the pre-training process, CodeBERT
differs from their work in that (1) CodeBERT is
trained in a cross-modal style and leverages both
bimodal NL-PL data and unimodal PL/NL data, (2)
CodeBERT is pre-trained over six programming
languages, and (3) CodeBERT is trained with a
new learning objective based on replaced token
detection.
3 CodeBERT
We describe the details about CodeBERT in this
section, including the model architecture, the input
and output representations, the objectives and data
used for training CodeBERT, and how to ﬁne-tune
CodeBERT when it is applied to downstream tasks.
3.1 Model Architecture
We follow BERT (Devlin et al., 2018) and
RoBERTa (Liu et al., 2019), and use multi-layer
bidirectional Transformer (Vaswani et al., 2017) as
the model architecture of CodeBERT. We will not
review the ubiquitous Transformer architecture in
detail. We develop CodeBERT by using exactly the
same model architecture as RoBERTa-base. The
total number of model parameters is 125M.
3.2 Input/Output Representations
In the pre-training phase, we set the input as the
concatenation of two segments with a special sepa-
rator token, namely [CLS],w1,w2,..wn,[SEP],
c1,c2,...,c m,[EOS]. One segment is natural lan-
guage text, and another is code from a certain pro-
gramming language. [CLS] is a special token in
front of the two segments, whose ﬁnal hidden repre-
sentation is considered as the aggregated sequence
representation for classiﬁcation or ranking. Follow-
ing the standard way of processing text in Trans-
former, we regard a natural language text as a se-
quence of words, and split it as WordPiece (Wu
et al., 2016). We regard a piece of code as a se-
quence of tokens.
The output of CodeBERT includes (1) contextual
vector representation of each token, for both natural
language and code, and (2) the representation of
[CLS], which works as the aggregated sequence
representation.
3.3 Pre-Training Data
We train CodeBERT with bothbimodal data, which
refers to parallel data of natural language-code
pairs, and unimodal data, which stands for codes
without paired natural language texts and natural
language without paired codes.
We use datapoints from Github repositories,
where each bimodal datapoint is an individual
function with paired documentation, and each uni-
modal code is a function without paired documen-
tation. Speciﬁcally, we use a recent large dataset
TRAINING DATA bimodal DATA unimodal CODES
GO 319,256 726,768
JAVA 500,754 1,569,889
JAVASCRIPT 143,252 1,857,835
PHP 662,907 977,821
PYTHON 458,219 1,156,085
RUBY 52,905 164,048
ALL 2,137,293 6,452,446
Table 1: Statistics of the dataset used for training Code-
BERT.
provided by Husain et al. (2019), which includes
2.1M bimodal datapoints and 6.4Munimodal codes
across six programming languages (Python, Java,
JavaScript, PHP, Ruby, and Go). Data statistics is
shown in Table 1.2
The data comes from publicly available open-
source non-fork GitHub repositories and are ﬁl-
tered with a set of constraints and rules. For ex-
ample, (1) each project should be used by at least
one other project, (2) each documentation is trun-
cated to the ﬁrst paragraph, (3) documentations
shorter than three tokens are removed, (4) func-
tions shorter than three lines are removed, and (5)
function names with substring “test” are removed.
An example of the data is given in Figure 1 3.
Figure 1: An example of the NL-PL pair, where NL is
the ﬁrst paragraph (ﬁlled in red) from the documenta-
tion (dashed line in black) of a function.
3.4 Pre-Training CodeBERT
We describe the two objectives used for training
CodeBERT here. The ﬁrst objective is masked
language modeling (MLM), which has proven ef-
fective in literature (Devlin et al., 2018; Liu et al.,
2Since we will evaluate on the natural language code
search task, we only use the training data of Husain et al.
(2019) to train CodeBERT with no access to the dev and test-
ing data.
3The source of the illustrating example comes from
https://github.com/apache/spark/blob/
618d6bff71073c8c93501ab7392c3cc579730f0b/
python/pyspark/rdd.py#L125-L138
6
NL Generator
NL-Code 
Discriminator
[𝑀𝐴𝑆𝐾]𝑤
CodeBERT V2:
A Pre-trained Model for NL-Code Understanding and Generation 
𝑤2
𝑤3
𝑤4
[𝑀𝐴𝑆𝐾]𝑤
𝑤1
𝑤2
𝑤3
𝑤4
𝑤5
[𝑀𝐴𝑆𝐾]𝑐
𝑐5
𝑐3
𝑐4
[𝑀𝐴𝑆𝐾]𝑐
𝑐1
𝑐2
𝑐3
𝑐4
𝑐5
𝑐6
𝑐1
𝑤51
𝑤2
𝑤3
𝑤4
𝑤5
𝑐29
𝑐5
𝑐3
𝑐4
𝑐162
𝑐1
original
replaced
Code Generator
original
original
original
replaced
original
original
original
original
replaced
sample
sample
sample
sample
Figure 2: An illustration about the replaced token detection objective. Both NL and code generators are language
models, which generate plausible tokens for masked positions based on surrounding contexts. NL-Code discrimi-
nator is the targeted pre-trained model, which is trained via detecting plausible alternatives tokens sampled from
NL and PL generators. NL-Code discriminator is used for producing general-purpose representations in the ﬁne-
tuning step. Both NL and code generators are thrown out in the ﬁne-tuning step.
2019; Sun et al., 2019). We apply masked language
modeling on bimodal data of NL-PL pairs. The sec-
ond objective is replaced token detection (RTD),
which further uses a large amount ofunimodal data,
such as codes without paired natural language texts.
Detailed hyper-parameters for model pre-training
are given in Appendix B.1.
Objective #1: Masked Language Modeling
(MLM) Given a datapoint of NL-PL pair (x =
{w, c}) as input, where w is a sequence of NL
words and c is a sequence of PL tokens, we ﬁrst
select a random set of positions for both NL and PL
to mask out (i.e. mw and mc, respectively), and
then replace the selected positions with a special
[MASK] token. Following Devlin et al. (2018),
15% of the tokens from x are masked out.
mw
i ∼unif{1,|w|}for i= 1to |w| (1)
mc
i ∼unif{1,|c|}for i= 1to |c| (2)
wmasked = REPLACE(w,mw,[MASK]) (3)
cmasked = REPLACE(c,mc,[MASK]) (4)
x = w + c (5)
The MLM objective is to predict the original to-
kens which are masked out, formulated as follows,
where pD1 is the discriminator which predicts a
token from a large vocabulary.
LMLM(θ)=
∑
i∈mw∪mc
−log pD1 (xi|wmasked,cmasked)
(6)
Objective #2: Replaced Token Detection (RTD)
In the MLM objective, only bimodal data (i.e. data-
points of NL-PL pairs) is used for training. Here we
present the objective of replaced token detection.
The RTD objective (Clark et al., 2020) is origi-
nally developed for efﬁciently learning pre-trained
model for natural language. We adapt it in our sce-
nario, with the advantage of using both bimodal
and unimodal data for training. Speciﬁcally, there
are two data generators here, an NL generator pGw
and a PL generator pGc , both for generating plau-
sible alternatives for the set of randomly masked
positions.
ˆwi ∼pGw (wi|wmasked) for i∈mw (7)
ˆci ∼pGc (ci|cmasked) for i∈mc (8)
wcorrupt = REPLACE(w,mw, ˆw) (9)
ccorrupt = REPLACE(c,mc,ˆc) (10)
xcorrupt = wcorrupt + ccorrupt (11)
The discriminator is trained to determine whether
a word is the original one or not, which is a binary
classiﬁcation problem. It is worth noting that the
RTD objective is applied to every position in the
input, and it differs from GAN (generative adver-
sarial network) in that if a generator happens to
produce the correct token, the label of that token
is “real” instead of “fake” (Clark et al., 2020). The
loss function of RTD with regard to the discrimina-
tor parameterized by θis given below, whereδ(i) is
an indicator function and pD2 is the discriminator
that predicts the probability of the i-th word being
original.
LRTD(θ) =
|w|+|c|∑
i=1
(
δ(i)log pD2 (xcorrupt,i)+
(
1 −δ(i)
)(
1 −log pD2 (xcorrupt,i)
))
(12)
δ(i) =
{
1, if xcorrupt
i = xi.
0, otherwise. (13)
There are many different ways to implement the
generators. In this work, we implement two ef-
ﬁcient n-gram language models (Jurafsky, 2000)
with bidirectional contexts, one for NL and one
for PL, and learn them from corresponding uni-
model datapoints, respectively. The approach is
easily generalized to learn bimodal generators or
use more complicated generators like Transformer-
based neural architecture learned in a joint manner.
We leave these to future work. The PL training data
is the unimodal codes as shown in Table 1, and the
NL training data comes from the documentations
from bimodal data. One could easily extend these
two training datasets to larger amount. The ﬁnal
loss function are given below.
min
θ
LMLM(θ) +LRTD(θ) (14)
3.5 Fine-Tuning CodeBERT
We have different settings to use CodeBERT in
downstream NL-PL tasks. For example, in natural
language code search, we feed the input as the
same way as the pre-training phase and use the
representation of [CLS] to measure the semantic
relevance between code and natural language query,
while in code-to-text generation, we use an encoder-
decoder framework and initialize the encoder of
a generative model with CodeBERT. Details are
given in the experiment section.
4 Experiment
We present empirical results in this section to verify
the effectiveness of CodeBERT. We ﬁrst describe
the use of CodeBERT in natural language code
search (§4.1), in a way that model parameters of
CodeBERT are ﬁne-tuned. After that, we present
the NL-PL probing task (§4.2), and evaluate Code-
BERT in a zero-shot setting where the parameters
of CodeBERT are ﬁxed. Finally, we evaluate Code-
BERT on a generation problem, i.e. code documen-
tation generation (§4.3), and further evaluate on a
programming language which is never seen in the
training phase (§4.4).
4.1 Natural Language Code Search
Given a natural language as the input, the objec-
tive of code search is to ﬁnd the most semantically
related code from a collection of codes. We con-
duct experiments on the CodeSearchNet corpus
(Husain et al., 2019) 4. We follow the ofﬁcial evalu-
ation metric to calculate the Mean Reciprocal Rank
(MRR) for each pair of test data (c, w) over a ﬁxed
set of 999 distractor codes. We further calculate the
macro-average MRR for all languages as an overall
evaluation metric. It is helpful to note that this met-
ric differs from the AVG metric in the original pa-
per, where the answer is retrieved from candidates
from all six languages. We ﬁne-tune a language-
speciﬁc model for each programming language 5.
We train each model with a binary classiﬁcation
loss function, where a softmax layer is connected
to the representation of [CLS]. Both training and
validation datasets are created in a way that posi-
tive and negative samples are balanced. Negative
samples consist of balanced number of instances
with randomly replaced NL (i.e. ( c, ˆw)) and PL
(i.e. (ˆc, w)). Detailed hyper-parameters for model
ﬁne-tuning are given in Appendix B.2.
Model Comparisons Table 2 shows the results
of different approaches on the CodeSearchNet cor-
pus. The ﬁrst four rows are reported by Husain
et al. (2019), which are joint embeddings of NL and
PL (Gu et al., 2018; Mitra et al., 2018). NBOW
represents neural bag-of-words. CNN, BIRNN
and SELFATT stand for 1D convolultional neu-
ral network (Kim, 2014), bidirectional GRU-based
recurrent neural network (Cho et al., 2014), and
multi-head attention (Vaswani et al., 2017), respec-
tively.
We report the remaining numbers in Table 2.
We train all these pre-trained models by regarding
codes as a sequence of tokens. We also continu-
ously train RoBERTa only on codes from Code-
SearchNet with masked language modeling. Re-
sults show that CodeBERT consistently performs
4More details about the dataset are given in Appendix A.
5We have ﬁne-tuned a multi-lingual model for six program-
ming languages, but ﬁnd that it performs worse that ﬁne-tuning
a language-speciﬁc model for each programming language.
MODEL RUBY JAVASCRIPT GO PYTHON JAVA PHP MA-AVG
NBOW 0.4285 0.4607 0.6409 0.5809 0.5140 0.4835 0.5181
CNN 0.2450 0.3523 0.6274 0.5708 0.5270 0.5294 0.4753
BIRNN 0.0835 0.1530 0.4524 0.3213 0.2865 0.2512 0.2580
SELF ATT 0.3651 0.4506 0.6809 0.6922 0.5866 0.6011 0.5628
ROBERTA 0.6245 0.6060 0.8204 0.8087 0.6659 0.6576 0.6972
PT W/ CODE ONLY (INIT =S) 0.5712 0.5557 0.7929 0.7855 0.6567 0.6172 0.6632
PT W/ CODE ONLY (INIT =R) 0.6612 0.6402 0.8191 0.8438 0.7213 0.6706 0.7260
CODE BERT (MLM, INIT =S) 0.5695 0.6029 0.8304 0.8261 0.7142 0.6556 0.6998
CODE BERT (MLM, INIT =R) 0.6898 0.6997 0.8383 0.8647 0.7476 0.6893 0.7549
CODE BERT (RTD, INIT =R) 0.6414 0.6512 0.8285 0.8263 0.7150 0.6774 0.7233
CODE BERT (MLM+RTD, INIT =R) 0.6926 0.7059 0.8400 0.8685 0.7484 0.7062 0.7603
Table 2: Results on natural language code retrieval. Baselines include four joint embeddings (ﬁrst group) of NL
and PL, RoBERTa, and RoBERTa which is continuously trained with masked language modeling on codes only
(second group). PT stands for pre-training. We train CodeBERT (third group) with different settings, including
using different initialization (from scratch (INIT =S) or initialized with the parameters of RoBERTa (INIT =R)) and
using different learning objectives (MLM, RTD, or the combination of both).
better than RoBERTa and the model pre-trained
with code only. CodeBERT (MLM) learned from
scratch performs better than RoBERTa. Unsur-
prisingly, initializing CodeBERT with RoBERTa
improves the performance 6.
4.2 NL-PL Probing
In the previous subsection, we show the empirical
effectiveness of CodeBERT in a setting that the
parameters of CodeBERT are ﬁne-tuned in down-
stream tasks. In this subsection, we further inves-
tigate what type of knowledge is learned in Code-
BERT without modifying the parameters.
Task Formulation and Data ConstructionFol-
lowing the probing experiments in NLP (Petroni
et al., 2019; Talmor et al., 2019), we study NL-
PL probing here. Since there is no existing work
towards this goal, we formulate the problem of
NL-PL probing and create the dataset by ourselves.
Given an NL-PL pair ( c, w), the goal of NL-PL
probing is to test model’s ability to correctly pre-
dict/recover the masked token of interest (either a
code token ci or word token wj) among distractors.
There are two major types of distractors: one is the
whole target vocabulary used for the masked lan-
guage modeling objective (Petroni et al., 2019), and
another one has fewer candidates which are ﬁlter or
curated based on experts’ understanding about the
ability to be tested (Talmor et al., 2019). We follow
the second direction and formulate NL-PL probing
as a multi-choice question answering task, where
the question is cloze-style in which a certain token
6We further give a learning curve of different pre-trained
models in the ﬁne-tuning process in Appendix C.
is replaced by [MASK] and distractor candidate
answers are curated based on our expertise.
Speciﬁcally, we evaluate on the NL side and PL
side, respectively. To ease the effort of data col-
lection, we collect data automatically from NL-PL
pairs in both validation and testing sets of Code-
SearchNet, both of which are unseen in the pre-
training phase. To evaluate on the NL side, we
select NL-PL pairs whose NL documentations in-
clude one of the six keywords (max, maximize, min,
minimize, less, greater), and group them to four
candidates by merging ﬁrst two keywords and the
middle two keywords. The task is to ask pre-trained
models to select the correct one instead of three
other distractors. That is to say, the input in this
setting includes the complete code and a masked
NL documentation. The goal is to select the correct
answer from four candidates. For the PL side, we
select codes containing keywordsmax and min, and
formulate the task as a two-choice answer selection
problem. Here, the input includes complete NL
documentation and a masked PL code, and the goal
is to select the correct answer from two candidates.
Since code completion is an important scenario,
we would like to test model’s ability in predicting
the correct token merely based on preceding PL
contexts. Therefore, we add an additional setting
for PL side, where the input includes the complete
NL documentation and preceding PL codes. Data
statistics is given in the top two rows in Table 3.
Model Comparisons Results are given in Table
3. We report accuracy, namely the number of cor-
rectly predicted instances over the number of all
instances, for each programming language. Since
RUBY JAVASCRIPT GO PYTHON JAVA PHP ALL
NUMBER OF DATAPOINTS FOR PROBING
PL (2 CHOICES ) 38 272 152 1,264 482 407 2,615
NL (4 CHOICES ) 20 65 159 216 323 73 856
PL PROBING
ROBERTA 73.68 63.97 72.37 59.18 59.96 69.78 62.45
PRE-TRAIN W / CODE ONLY 71.05 77.94 89.47 70.41 70.12 82.31 74.11
CODE BERT (MLM) 86.84 86.40 90.79 82.20 90.46 88.21 85.66
PL PROBING WITH PRECEDING CONTEXT ONLY
ROBERTA 73.68 53.31 51.32 55.14 42.32 52.58 52.24
PRE-TRAIN W / CODE ONLY 63.16 48.53 61.84 56.25 58.51 58.97 56.71
CODE BERT (MLM) 65.79 50.74 59.21 62.03 54.98 59.95 59.12
NL PROBING
ROBERTA 50.00 72.31 54.72 61.57 61.61 65.75 61.21
PRE-TRAIN W / CODE ONLY 55.00 67.69 60.38 68.06 65.02 68.49 65.19
CODE BERT (MLM) 65.00 89.23 66.67 76.85 73.37 79.45 74.53
Table 3: Statistics of the data for NL-PL probing and the performance of different pre-trained models. Accuracies
(%) are reported. Best results in each group are in bold.
datasets in different programming languages are
extremely unbalanced, we report the accumulated
metric with the same way. We use CodeBERT
(MLM) here because its output layer naturally ﬁts
for probing. Results show that CodeBERT per-
forms better than baselines on almost all languages
on both NL and PL probing. The numbers with
only preceding contexts are lower than that with
bidirectional contexts, which suggests that code
completion is challenging. We leave it as a future
work.
We further give a case study on PL-NL probing.
We mask NL token and PL token separately, and
report the predicted probabilities of RoBERTa and
CodeBERT. Figure 3 illustrates the example of a
python code7. We can see that RoBERTa fails in
both cases, whereas CodeBERT makes the correct
prediction in both NL and PL settings.
4.3 Code Documentation Generation
Although the pre-training objective of Code-
BERT does not include generation-based objectives
(Lewis et al., 2019), we would like to investigate
to what extent does CodeBERT perform on gen-
eration tasks. Speciﬁcally, we study code-to-NL
generation, and report results for the documenta-
tion generation task on CodeSearchNet Corpus in
six programming languages. Since the generated
documentations are short and higher order n-grams
may not overlap, we remedy this problem by using
smoothed BLEU score (Lin and Och, 2004).
7The example comes from https://
github.com/peri-source/peri/blob/
61beed5deaaf978ab31ed716e8470d86ba639867/
peri/comp/psfcalc.py#L994-L1002
def vec_to_halfvec(vec):
d = vec[1:] - vec[:-1]
if ((d/d.mean()).std() > 1e-14) or (d.mean() < 0):
raise ValueError('vec must be np.arange() in increasing order')
dx = d.mean()
lowest = np.abs(vec). min ()
highest = np.abs(vec).max()
return np.arange(lowest, highest + 0.1*dx, dx).astype(vec.dtype)
"Transforms a vector np.arange(-N, M, dx) to np.arange( min (|vec|), 
max(N,M),dx)]"
masked NL token
masked PL token
max min less greater
NL
Roberta 96.24% 3.73% 0.02% 0.01%
CodeBERT (MLM) 39.38% 60.60% 0.02% 0.0003%
PL
Roberta 95.85% 4.15% - -
CodeBERT (MLM) 0.001% 99.999% - -
Figure 3: Case study on python language. Masked to-
kens in NL (in blue) and PL (in yellow) are separately
applied. Predicted probabilities of RoBERTa and Code-
BERT are given.
Model Comparisons We compare our model
with several baselines, including a RNN-based
model with attention mechanism (Sutskever et al.,
2014), the Transformer (Vaswani et al., 2017),
RoBERTa and the model pre-trained on code only.
To demonstrate the effectiveness of CodeBERT
on code-to-NL generation tasks, we adopt various
pre-trained models as encoders and keep the hyper-
parameters consistent. Detailed hyper-parameters
are given in Appendix B.3.
Table 4 shows the results with different mod-
els for the code-to-documentation generation task.
As we can see, models pre-trained on program-
ming language outperform RoBERTa, which illus-
trates that pre-trainning models on programming
MODEL RUBY JAVASCRIPT GO PYTHON JAVA PHP OVERALL
SEQ 2SEQ 9.64 10.21 13.98 15.93 15.09 21.08 14.32
TRANSFORMER 11.18 11.59 16.38 15.81 16.26 22.12 15.56
ROBERTA 11.17 11.90 17.72 18.14 16.47 24.02 16.57
PRE -TRAIN W / CODE ONLY 11.91 13.99 17.78 18.58 17.50 24.34 17.35
CODE BERT ( RTD ) 11.42 13.27 17.53 18.29 17.35 24.10 17.00
CODE BERT ( MLM ) 11.57 14.41 17.78 18.77 17.38 24.85 17.46
CODE BERT ( RTD +MLM ) 12.16 14.90 18.07 19.06 17.65 25.16 17.83
Table 4: Results on Code-to-Documentation generation, evaluated with smoothed BLEU-4 score.
language could improve code-to-NL generation.
Besides, results in the Table 4 show that CodeBERT
pre-trained with RTD and MLM objectives brings
a gain of 1.3 BLEU score over RoBERTa overall
and achieve the state-of-the-art performance8.
4.4 Generalization to Programming
Languages NOT in Pre-training
We would like to evaluate CodeBERT on the pro-
gramming language which is never seen in the pre-
training step. To this end, we study the task of gen-
erating a natural language summary of a C# code
snippet. We conduct experiments on the dataset
of CodeNN (Iyer et al., 2016)9, which consists of
66,015 pairs of questions and answers automati-
cally collected from StackOverﬂow. This dataset
is challenging since the scale of dataset is orders
of magnitude smaller than CodeSearchNet Corpus.
We evaluate models using smoothed BLEU-4 score
and use the same evaluation scripts as Iyer et al.
(2016).
MODEL BLEU
MOSES (K OEHN ET AL ., 2007) 11.57
IR 13.66
SUM-NN (R USH ET AL ., 2015) 19.31
2-LAYER BILSTM 19.78
TRANSFORMER (VASWANI ET AL ., 2017) 19.68
TREE LSTM (T AI ET AL ., 2015) 20.11
CODE NN (I YER ET AL ., 2016) 20.53
CODE 2SEQ (ALON ET AL ., 2019) 23.04
ROBERTA 19.81
PRE -TRAIN W / CODE ONLY 20.65
CODE BERT (RTD) 22.14
CODE BERT (MLM) 22.32
CODE BERT (MLM+RTD) 22.36
Table 5: Code-to-NL generation on C# language.
Model Comparisons Table 5 shows that our
model with MLM and RTD pre-training objectives
achieves 22.36 BLEU score and improves by 2.55
points over RoBERTa, which illustrates CodeBERT
8We further give some output examples in Appendix E.
9https://github.com/sriniiyer/codenn
could generalize better to other programming lan-
guage which is never seen in the pre-training step.
However, our model achieve slightly lower results
than code2seq (Alon et al., 2019). The main reason
could be that code2seq makes use of compositional
paths in its abstract syntax tree (AST) while Code-
BERT only takes original code as the input. We
have trained a version of CodeBERT by traversing
the tree structure of AST following a certain order,
but applying that model does not bring improve-
ments on generation tasks. This shows a potential
direction to improve CodeBERT by incorporating
AST.
5 Conclusion
In this paper, we present CodeBERT, which to the
best of our knowledge is the ﬁrst large bimodal
pre-trained model for natural language and pro-
gramming language. We train CodeBERT on both
bimodal and unimodal data, and show that ﬁne-
tuning CodeBERT achieves state-of-the-art perfor-
mance on downstream tasks including natural lan-
guage code search and code-to-documentation gen-
eration. To further investigate the knowledge em-
bodied in pre-trained models, we formulate the task
of NL-PL probing and create a dataset for probing.
We regard the probing task as a cloze-style answer
selection problem, and curate distractors for both
NL and PL parts. Results show that, with model
parameters ﬁxed, CodeBERT performs better than
RoBERTa and a continuously trained model using
codes only.
There are many potential directions for further
research on this ﬁeld. First, one could learn better
generators with bimodal evidence or more compli-
cated neural architecture to improve the replaced to-
ken detection objective. Second, the loss functions
of CodeBERT mainly target on NL-PL understand-
ing tasks. Although CodeBERT achieves strong
BLEU scores on code-to-documentation genera-
tion, the CodeBERT itself could be further im-
proved by generation-related learning objectives.
How to successfully incorporate AST into the pre-
training step is also an attractive direction. Third,
we plan to apply CodeBERT to more NL-PL re-
lated tasks, and extend it to more programming
languages. Flexible and powerful domain/language
adaptation methods are necessary to generalize
well.
Acknowledgments
Xiaocheng Feng is the corresponding author of this
work. We thank the anonymous reviewers for their
insightful comments. Zhangyin Feng, Xiaocheng
Feng, Bing Qin and Ting Liu are supported by the
National Key R&D Program of China via grant
2018YFB1005103 and National Natural Science
Foundation of China (NSFC) via grant 61632011
and 61772156.
References
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav.
2019. code2seq: Generating sequences from struc-
tured representations of code. International Confer-
enceon Learning Representations.
Kyunghyun Cho, Bart Van Merri ¨enboer, Caglar Gul-
cehre, Dzmitry Bahdanau, Fethi Bougares, Holger
Schwenk, and Yoshua Bengio. 2014. Learning
phrase representations using rnn encoder-decoder
for statistical machine translation. arXiv preprint
arXiv:1406.1078.
Kevin Clark, Minh-Thang Luong, Quoc V . Le, and
Christopher D. Manning. 2020. {ELECTRA}: Pre-
training text encoders as discriminators rather than
generators. In International Conference on Learn-
ing Representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2018. Bert: Pre-training of deep
bidirectional transformers for language understand-
ing. arXiv preprint arXiv:1810.04805.
Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018.
Deep code search. In 2018 IEEE/ACM 40th Interna-
tional Conference on Software Engineering (ICSE),
pages 933–944. IEEE.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis
Allamanis, and Marc Brockschmidt. 2019. Code-
searchnet challenge: Evaluating the state of seman-
tic code search. arXiv preprint arXiv:1909.09436.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and
Luke Zettlemoyer. 2016. Summarizing source code
using a neural attention model. In Proceedings
of the 54th Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers),
pages 2073–2083.
Dan Jurafsky. 2000. Speech & language processing.
Pearson Education India.
Aditya Kanade, Petros Maniatis, Gogul Balakrish-
nan, and Kensen Shi. 2019. Pre-trained contex-
tual embedding of source code. arXiv preprint
arXiv:2001.00059.
Yoon Kim. 2014. Convolutional neural net-
works for sentence classiﬁcation. arXiv preprint
arXiv:1408.5882.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris
Callison-Burch, Marcello Federico, Nicola Bertoldi,
Brooke Cowan, Wade Shen, Christine Moran,
Richard Zens, et al. 2007. Moses: Open source
toolkit for statistical machine translation. In Pro-
ceedings of the 45th annual meeting of the associ-
ation for computational linguistics companion vol-
ume proceedings of the demo and poster sessions,
pages 177–180.
Mike Lewis, Yinhan Liu, Naman Goyal, Mar-
jan Ghazvininejad, Abdelrahman Mohamed, Omer
Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.
Bart: Denoising sequence-to-sequence pre-training
for natural language generation, translation, and
comprehension. arXiv preprint arXiv:1910.13461.
Chin-Yew Lin and Franz Josef Och. 2004. Orange: a
method for evaluating automatic evaluation metrics
for machine translation. In Proceedings of the 20th
international conference on Computational Linguis-
tics, page 501. Association for Computational Lin-
guistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man-
dar Joshi, Danqi Chen, Omer Levy, Mike Lewis,
Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: A robustly optimized bert pretraining ap-
proach. arXiv preprint arXiv:1907.11692.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan
Lee. 2019. Vilbert: Pretraining task-agnostic visi-
olinguistic representations for vision-and-language
tasks. In Advances in Neural Information Process-
ing Systems, pages 13–23.
Bhaskar Mitra, Nick Craswell, et al. 2018. An intro-
duction to neural information retrieval. Foundations
and TrendsR⃝in Information Retrieval, 13(1):1–126.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt
Gardner, Christopher Clark, Kenton Lee, and Luke
Zettlemoyer. 2018. Deep contextualized word repre-
sentations. arXiv preprint arXiv:1802.05365.
Fabio Petroni, Tim Rockt ¨aschel, Patrick Lewis, Anton
Bakhtin, Yuxiang Wu, Alexander H Miller, and Se-
bastian Riedel. 2019. Language models as knowl-
edge bases? arXiv preprint arXiv:1909.01066.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.
How multilingual is multilingual bert? arXiv
preprint arXiv:1906.01502.
Alec Radford, Karthik Narasimhan, Tim Salimans,
and Ilya Sutskever. 2018. Improving language
understanding by generative pre-training. URL
https://s3-us-west-2. amazonaws. com/openai-
assets/researchcovers/languageunsupervised/language
understanding paper. pdf.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine
Lee, Sharan Narang, Michael Matena, Yanqi Zhou,
Wei Li, and Peter J Liu. 2019. Exploring the limits
of transfer learning with a uniﬁed text-to-text trans-
former. arXiv preprint arXiv:1910.10683.
Alexander M Rush, Sumit Chopra, and Jason We-
ston. 2015. A neural attention model for ab-
stractive sentence summarization. arXiv preprint
arXiv:1509.00685.
Chen Sun, Austin Myers, Carl V ondrick, Kevin Mur-
phy, and Cordelia Schmid. 2019. Videobert: A joint
model for video and language representation learn-
ing. arXiv preprint arXiv:1904.01766.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.
Sequence to sequence learning with neural networks.
In Advances in neural information processing sys-
tems, pages 3104–3112.
Kai Sheng Tai, Richard Socher, and Christopher D
Manning. 2015. Improved semantic representations
from tree-structured long short-term memory net-
works. arXiv preprint arXiv:1503.00075.
Alon Talmor, Yanai Elazar, Yoav Goldberg, and
Jonathan Berant. 2019. olmpics–on what lan-
guage model pre-training captures. arXiv preprint
arXiv:1912.13283.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In Advances in neural information pro-
cessing systems, pages 5998–6008.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V
Le, Mohammad Norouzi, Wolfgang Macherey,
Maxim Krikun, Yuan Cao, Qin Gao, Klaus
Macherey, et al. 2016. Google’s neural machine
translation system: Bridging the gap between hu-
man and machine translation. arXiv preprint
arXiv:1609.08144.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car-
bonell, Ruslan Salakhutdinov, and Quoc V Le.
2019. Xlnet: Generalized autoregressive pretrain-
ing for language understanding. arXiv preprint
arXiv:1906.08237.
A Data Statistic
Data statistics of the training/validation/testing data
splits for six programming languages are given in
Table 6.
CODE SEARCH TRAINING DEV TESTING
GO 635,635 28,483 14,291
JAVA 908,886 30,655 26,909
JAVASCRIPT 247,773 16,505 6,483
PHP 1,047,406 52,029 28,391
PYTHON 824,342 46,213 22,176
RUBY 97,580 4,417 2,279
Table 6: Data statistics about the CodeSearchNet Cor-
pus for natural language code search.
B Train Details
B.1 Pre-training
We train CodeBERT on one NVIDIA DGX-2 ma-
chine using FP16. It combines 16 interconnected
NVIDIA Tesla V100 with 32GB memory. We use
the following set of hyper-parameters to train mod-
els: batchsize is 2,048 and learning rate is 5e-4. We
use Adam to update the parameters and set the num-
ber of warmup steps as 10K. We set the max length
as 512 and the max training step is 100K. Training
1,000 batches of data costs 600 minutes with MLM
objective, 120 minutes with RTD objective.
B.2 CodeSearch
In the ﬁne-turning step, we set the learning rate as
1e-5, the batch size as 64, the max sequence length
as 200 and the max ﬁne-tuning epoch as 8. As the
same with pre-training, We use Adam to update the
parameters. We choose the model performed best
on the development set, and use that to evaluate on
the test set.
B.3 Code Summarization on Six
Programming Languages
We use Transformer with 6 layers, 768 dimensional
hidden states and 12 attention heads as our decoder
in all settings. We set the max length of input
and inference as 256 and 64, respectively. We use
the Adam optimizer to update model parameters.
The learning rate and the batch size are 5e-5 and
64, respectively. We tune hyperparameters and
perform early stopping on the development set.
B.4 Code Summarization on C#
Since state-of-the-art methods use RNN as their de-
coder, we choose a 2-layer GRU with an attention
mechanism as our decoder for a comparison. We
ﬁne-tune models using a grid search with the fol-
lowing set of hyper-parameters: batchsize is in{32,
64}and learning rate is in {2e-5, 5e-5}. We report
the number when models achieve best performance
on the development set.
C Learning Curve of CodeSearch
From Figure 4, we can see that CodeBERT per-
forms better at the early stage, which reﬂects that
CodeBERT provides good initialization for learn-
ing downstream tasks.
1 2 3 4 5 6 7 8
The Number of Epoch
84.5
85.0
85.5
86.0
86.5
87.0
87.5
88.0
88.5Dev Accuracy of Python
Roberta
CodeBERT
Pre-train w/ code only
1 2 3 4 5 6 7 8
The Number of Epoch
79.5
80.0
80.5
81.0
81.5
82.0
82.5
83.0Dev Accuracy of Java
Roberta
CodeBERT
Pre-train w/ code only
Figure 4: Learning curve of different pre-trained mod-
els in the ﬁne-tuning step. We show results on Python
and Java.
D Late Fusion
In section §4.1 , we show that CodeBERT per-
forms well in the setting where natural languages
and codes have early interactions. Here, we in-
vestigate whether CodeBERT is good at working
as a uniﬁed encoder. We apply CodeBERT for
natural language code search in a later fusion set-
ting, where CodeBERT ﬁrst encodes NL and PL
separately, and then calculates the similarity by dot-
product. In this way, code search is equivalent to
ﬁnd the nearest codes in the shared vector space.
This scenario also facilitates the use of CodeBERT
in an online system, where the representations of
codes are calculated in advance. In the runtime, a
system only needs to compute the representation
of NL and vector-based dot-product.
We ﬁne-tune CodeBERT with the following ob-
jective, which maximizes the dot-product of the
ground truth while minimizing the dot-product of
distractors.
−1
N
∑
i
log
( exp
(
Enc(ci)⊺Enc(wi)
)
∑
j exp
(
Enc(cj)⊺Enc(wi)
)
)
(15)
Results are given in Table 7. We just do this
setting on two languages with a relatively small
amount of data.
We can see that CodeBERT performs better than
RoBERTa and the model pre-trained with codes
MODEL RUBY GO
ROBERTA 0.0043 0.0030
PRE-TRAIN W / CODE ONLY 0.1648 0.4179
CODE BERT 0.6870 0.8372
Table 7: Results on natural language code search by
late fusion.
only. And late fusion performs comparable with
the standard way. What’s more, late fusion is more
efﬁcient and this setting could be used in an online
system.
E Case Study
To qualitatively analyze the effectiveness of Code-
BERT, we give some cases for code search and
code documentation generation tasks.
Considering the limited space, we only give the
top2 results of the query for python programming
language. As show in Figure 5, search results are
very relevant with query.
Figure 6 and Figure 7 show the outputs with
different models for the code documentation gen-
eration task. As we can see, CodeBERT performs
better than all baselines.
Figure 5: Python CodeSearch example. The results are searched from 1,156,085 python code data. We only give
the top2 results because space is limited.
Figure 6: Java code documentation generation output example.
Figure 7: Python code documentation generation output example.

Neural Code Comprehension: A Learnable
Representation of Code Semantics
Tal Ben-Nun
ETH Zurich
Zurich 8092, Switzerland
talbn@inf.ethz.ch
Alice Shoshana Jakobovits
ETH Zurich
Zurich 8092, Switzerland
alicej@student.ethz.ch
Torsten Hoeﬂer
ETH Zurich
Zurich 8092, Switzerland
htor@inf.ethz.ch
Abstract
With the recent success of embeddings in natural language processing, research
has been conducted into applying similar methods to code analysis. Most works
attempt to process the code directly or use a syntactic tree representation, treating it
like sentences written in a natural language. However, none of the existing methods
are sufﬁcient to comprehend program semantics robustly, due to structural features
such as function calls, branching, and interchangeable order of statements. In this
paper, we propose a novel processing technique to learn code semantics, and apply
it to a variety of program analysis tasks. In particular, we stipulate that a robust
distributional hypothesis of code applies to both human- and machine-generated
programs. Following this hypothesis, we deﬁne an embedding space, inst2vec,
based on an Intermediate Representation (IR) of the code that is independent of the
source programming language. We provide a novel deﬁnition of contextual ﬂow
for this IR, leveraging both the underlying data- and control-ﬂow of the program.
We then analyze the embeddings qualitatively using analogies and clustering, and
evaluate the learned representation on three different high-level tasks. We show that
even without ﬁne-tuning, a single RNN architecture and ﬁxed inst2vec embeddings
outperform specialized approaches for performance prediction (compute device
mapping, optimal thread coarsening); and algorithm classiﬁcation from raw code
(104 classes), where we set a new state-of-the-art.
1 Introduction
The emergence of the “Big Data era” manifests in the form of a dramatic increase in accessible code.
In the year 2017 alone, GitHub reports [25] approximately 1 billion git commits (code modiﬁcation
uploads) written in 337 different programming languages. Sifting through, categorizing, and under-
standing code thus becomes an essential task for a variety of ﬁelds. Applications include identifying
code duplication, performance prediction, algorithm detection for alternative code suggestion (guided
programming), vulnerability analysis, and malicious code detection. These tasks are challenging,
as code can be modiﬁed such that it syntactically differs (for instance, via different or reordered
operations, or written in a different language altogether), but remains semantically equivalent (i.e.,
produces the same result). However, these tasks are also ideal for machine learning, since they can be
represented as classic regression and classiﬁcation problems.
In order to mechanize code comprehension, the research community typically employs reinforcement
learning and stochastic compilation forsuper-optimization [13, 56]; or borrows concepts from Natural
Language Processing (NLP) for human-authored code, relying on the following hypothesis:
The naturalness hypothesis [3]. Software is a form of human communication;
software corpora have similar statistical properties to natural language corpora;
and these properties can be exploited to build better software engineering tools.
32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada.
C/C++FORTRANPythonJavaCUDAOpenCL. . .SSA Representation(LLVM IR)Neural Code Comprehension
RNN
RNN
RNN
RNN
inst2vec
Contextual Flow Graph (XFG)Malicious Code DetectionGuided ProgrammingCode OptimizationHardware Mapping
Anti-Virus
IDE
CompilerHigh-Level Tasks
FrontendSourceCode
Figure 1: Component overview of the Neural Code Comprehension pipeline.
For NLP-based approaches, input code is usually processed into tokens (e.g., keywords, braces) [18]
or other representations [ 4, 7, 53], and optionally undergoes embedding in a continuous lower-
dimensional space. In the spirit of the successful word2vec model [47, 48], the mapping to the
embedding space is learned by pairing a token with its surrounding tokens. Following this process,
RNNs [23] are trained on sequences of such tokens. This model has been successfully used for NLP-
like tasks, such as summarization [6], function name prediction [7], and algorithm classiﬁcation [49].
Although the results for stochastic code optimization and NLP embeddings are promising, two issues
arise. Firstly, in prior works, the source programming language (or machine code for optimization) is
ﬁxed, which does not reﬂect the plethora of languages, nor generalizes to future languages. Secondly,
existing methods process tokens (or instructions) sequentially, targeting function- and loop-free code.
Such codes, however, do not represent the majority of the applications.
This paper presents Neural Code Comprehension1: a general-purpose processing pipeline geared
towards representing code semantics in a robust and learnable manner. The pipeline, depicted in
Fig. 1, accepts code in various source languages and converts it to statements in an Intermediate
Representation (IR), using the LLVM Compiler Infrastructure [39]. The LLVM IR, which is explained
in detail in Section 4, is then processed to a robust representation that we callconteXtual Flow Graphs
(XFGs). XFGs are constructed from both the data- and control-ﬂow of the code, thus inherently
supporting loops and function calls. In turn, the XFG structure is used to train an embedding space
for individual statements, called inst2vec (from the word “instruction”), which is fed to RNNs for
a variety of high-level tasks.
Neural Code Comprehension is evaluated on multiple levels, using clustering and analogies for
inst2vec , as well as three different code comprehension tasks for XFGs: algorithm classiﬁcation;
heterogeneous compute device (e.g., CPU, GPU) mapping; and optimal thread coarsening factor
prediction, which model the runtime of an application without running it. Our datasets contain CPU
and GPU code written in C, C++, OpenCL, and FORTRAN, though LLVM supports additional
languages such as Python, Rust, Swift, Go, and CUDA. Our work makes the following contributions:
• We formulate a robust distributional hypothesis for code, from which we draw a novel
distributed representation of code statements based on contextual ﬂow and LLVM IR.
• We detail the construction of the XFG, the ﬁrst representation designed speciﬁcally for
statement embeddings that combines data and control ﬂow.
• We evaluate the representation using clustering, analogies, semantic tests, and three funda-
mentally different high-level code learning tasks.
• Using one simple LSTM architecture and ﬁxed pre-trained embeddings, we match or surpass
the best-performing approaches in each task, including specialized DNN architectures.
2 Related Work
Distributed representations of code were ﬁrst suggested by Allamanis et al. [2], followed by several
works leveraging embeddings to apply NLP techniques to programming languages [3, 61].
Code Representation Previous research focuses on embedding high-level programming languages
such as Java [20, 30], C [41], or OpenCL [18] in the form of tokens or statements, as well as lower
1Code, datasets, trained embeddings, and results available at https://www.github.com/spcl/ncc
2
level representations such as object code [41]. To the best of our knowledge, however, no attempt
has been made to train embeddings for compiler IRs prior to this work. As for representing the
context of a token, which is necessary for training embeddings, some works rely on lexicographical
locality [2, 18, 20], whereas others exploit the structural nature of code, using Data Flow Graphs [4],
Control Flow Graphs [ 51, 53, 64], Abstract Syntax Trees (ASTs) [ 12, 30], paths in the AST [ 8],
or an augmented AST, for instance with additional edges connecting different uses and updates of
syntax tokens corresponding to variables [5]. We differ from all previous approaches by introducing
contextual ﬂow, a graph representation that captures both data and control dependencies. In compiler
research, similar graphs exist but have not been successfully exploited for machine learning. Examples
include the Program Dependence Graph (PDG) [ 24] and the IR known as Sea of Nodes [ 15, 16].
Unlike these representations, our graphs are not designed to be optimized by a compiler nor translated
to machine code, which allows us to introduce ambiguity (e.g., ignoring parameter order) in favor
of preserving context. Other works applying Machine Learning techniques to PDGs exist: Hsiao et
al. [34] use PDGs to compute n-gram models for program analysis, and Wang et al. [62] use them for
detecting copy direction among programs using Extreme Learning Machines. However, our work is
the ﬁrst to leverage a hybrid of control and data ﬂow for the training of embeddings.
Automated Tasks on Code Learned representations of code are commonly used for two types of
tasks: uncovering program semantics or optimizing programs. For the former task, code embeddings
have been used to perform function or variable naming [ 2, 7], clone detection [63], code comple-
tion [54, 65], summarization [6], and algorithm classiﬁcation [ 49]. As for program optimization,
research has been conducted on automatic feature generation for code [ 40, 50]; and Cummins et
al. [18] notably leverage embeddings of OpenCL code to predict optimal device mapping and thread
coarsening factors. Their work differs from ours in that the method is restricted to the OpenCL
language, and that they process programs in a sequential order, which does not capture complex code
structures. Furthermore, the state-of-the-art in automatic tuning for program optimization [10] uses
surrogate performance models and active learning, and does not take code semantics into account.
Embedding Evaluation Previous works that use code embeddings do not evaluate the quality of
the trained space on its own merit, but rather through the performance of subsequent (downstream)
tasks. One exception is Allamanis et al. [2], who present empirical evidence of vector similarities
for similar method names. To the best of our knowledge, we are the ﬁrst to quantify the quality of a
code embedding space itself in the form of clustering, syntactic analogies, semantic analogies, and
categorical distance tests.
3 A Robust Distributional Hypothesis of Code
The linguistic Distributional Hypothesis [32, 52] is given by: Words that occur in the same contexts
tend to have similar meanings. We stipulate that code, which describes a sequence of operations to a
processor, behaves similarly, and paraphrase this hypothesis to:
Statements that occur in the same contexts tend to have similar semantics.
However, the above wording is vague, due to the possible meanings of the highlighted elements. Below
we attempt to provide adequate deﬁnitions, upon which we build a learnable code representation.
Statements To choose the right abstraction for statements, we take two concerns into account:
universality and uniformity. As stated above, source code comes in many languages and thus
ﬁxating on a single one would hinder universality. At the other extreme, machine code (assembly)
is target-speciﬁc, containing specialized instructions and relying on hardware characteristics, such
as registers and memory architectures. As for uniformity, in a high-level language one statement
may represent simple arithmetics, multiple operations, or even class deﬁnitions (for example, the
Java statementbutton.setOnClickListener(new View.OnClickListener(){...}) ). On the
other hand, assembly is too limited, since instructions are reused for different purposes. We thus wish
to choose statements that are independent of the source language, as well as the hardware architecture.
Context The deﬁnition of a context for code statements should also be carefully considered. We
deﬁne context as statements whose execution directly depends on each other. Learning from consec-
utive statements in code does not necessarily fulﬁll this deﬁnition, as, for example, a programmer
3
double thres = 5.0;
if (x < thres)
x = y * y;
else
x = 2.0 * y;
x += 1.0;
(a) Source code
%cmp = fcmp olt double %x, 5.0
br i1 %cmp, label %LT, label %GE
LT:
%2 = fmul double %y, %y
GE:
%3 = fmul double 2.0, %y
AFTER:
%4 = phi double [%2,%LT], [%3,%GE]
%5 = fadd double %4, 1.0
(b) LLVM IR
%0
5.0 %cmp
%LT %GE
%2 %3
%AFTER
%y 2.0 %y
1.0
%5%3
%2
%4
%x (c) Dataﬂow basic blocks
%x
%y
%LT
%cmp
%AFTER
%5
%4
fadd
phi
%3%2
%GE
fmul
brbr
fcmp
phi (d) Contextual Flow Graph
Figure 2: Contextual ﬂow processing scheme.
may use a variable in the ﬁrst line of a function, but only use it again in the last line. Moreover,
such long-term relationships may vanish when using RNNs and attention learning. It is possible to
determine the data dependencies of each statement by analyzing dataﬂow, however, branches and
function calls do not necessarily generate such dependencies. Another way of representing execution
dependence is through the notion of causality (i.e., the “happens-before” relation [38]), which can
be used to complement dataﬂow. In our representation, context is the union of data dependence and
execution dependence, thereby capturing both relations.
Similarity To deﬁne similarity, one ﬁrst needs to deﬁne the semantics of a statement. We draw the
deﬁnition of semantics from Operational Semantics in programming language theory, which refers
to the effects (e.g., preconditions, postconditions) of each computational step in a given program.
In this paper, we speciﬁcally assume that each statement modiﬁes the system state in a certain way
(e.g., adds two numbers) and consumes resources (e.g., uses registers and ﬂoating-point units). It
follows that semantic similarity can be deﬁned by two statements consuming the same resources or
modifying the system state in a similar way. Using this deﬁnition, two versions of the same algorithm
with different variable types would be synonymous.
4 Contextual Flow Processing
The aforementioned statements and contexts cannot be directly extracted from source code, but rather
require processing akin to partial compilation (e.g., dataﬂow extraction). In this section, we brieﬂy
describe a popular compilation pipeline and proposed modiﬁcations to create a learnable vocabulary
of statements and their context.
4.1 Compilation, Static Single Assignment, and LLVM IR
Major contemporary compilers, such as GCC and LLVM, support multiple programming languages
and hardware targets. To avoid duplication in code optimization techniques, they enforce a strict
separation between the source language (frontend), an Intermediate Representation (IR) that can be
optimized, and the target machine code (backend) that should be mapped to a speciﬁc hardware. In
particular, the LLVM IR [45] supports various architectures (e.g., GPUs), and can represent optimized
code (e.g., using vector registers) inherently. Figures 2a and 2b depict an example code and its LLVM
IR equivalent, and the structure of an LLVM IR statement is shown in Fig. 3.
%5 = load float, float* %a1, align 4, !tbaa !1 ; comment
Output 
Identifier
Instruction Types Input
Identifier
Other
Parameters
Metadata
Figure 3: Anatomy of an LLVM IR statement.
x0 = a * b;
x1 = c * c;
x2 = x0 + x1;
x3 = x + x2;
Figure 4: SSA of x += (a*b)+(c*c) .
In the LLVM infrastructure, the IR is given in Static Single Assignment (SSA) form [19]. Brieﬂy,
an SSA IR ensures that every variable is assigned only once, which makes it easy to track dataﬂow
between IR statements, as shown in Fig. 4. To overcome analysis issues resulting from control-ﬂow,
such as loops, SSA deﬁnes φ-expressions. These expressions enumerate all possible outcomes that
4
can lead to a variable (depending on the runtime control-ﬂow), and can be used to optimize code
across branches. In Fig. 2b, the identiﬁer %4 is constructed from a φ-expression that can take either
the value of %2 or %3 , depending on the value of x .
4.2 Contextual Flow Graphs
To analyze dataﬂow for optimization, LLVM divides the IR statements into “basic blocks”, which
contain no control-ﬂow divergence, illustrated in Fig. 2c. Within a basic block, statements naturally
create traceable dataﬂow as SSA lists data dependencies in the form of input identiﬁers (even if
conditional), and assigns the results to a single identiﬁer. However, as shown in Section 3, dataﬂow
alone does not sufﬁce to provide context for a given statement, e.g., when in the vicinity of a branch.
Therefore, we deﬁne a representation that incorporates both the relative data- and control-ﬂow of a
statement, which we call the conteXtual Flow Graph (XFG).
XFGs (e.g., Fig. 2d) are directed multigraphs, where two nodes can be connected by more than
one edge. XFG nodes can either be variables or label identiﬁers (e.g., basic block, function name),
appearing in the ﬁgure as ovals or rectangles respectively. Correspondingly, an edge either represents
data-dependence (in black), carrying an LLVM IR statement; or execution dependence (light blue).
XFG Construction We generate XFGs incrementally from LLVM IR, as follows:
1. Read LLVM IR statements once, storing function names and return statements.
2. Second pass over the statements, adding nodes and edges according to the following rule-set:
(a) Data dependencies within a basic block are connected.
(b) Inter-block dependencies (e.g., φ-expressions) are both connected directly and through
the label identiﬁer (statement-less edges).
(c) Identiﬁers without a dataﬂow parent are connected to their root (label or program root).
It follows that XFGs create paths through dataﬂow as well as branches, loops, and functions (including
recursion). Owing to the two passes, as well as the linear-time construction of LLVM IR [58], XFGs
are constructed in O(n) for a program with nSSA statements. This is especially valuable when
learning over large code corpora, such as Tensorﬂow.
External Code Calls to external code (e.g., libraries, frameworks) can be divided into two cate-
gories: statically- and dynamically-linked. If the code is accessible during compilation (header-only
frameworks and static libraries), LLVM IR is available and the statements are traversed as part of the
XFG. In the dynamic case, the library code is not included and is represented as a call statement.
5 inst2vec : Embedding Statements in Continuous Space
With XFGs providing a notion of context, we can now train an embedding space for individual
statements. To support learnability, desiderata for such a space include: (a) statements that are in
close proximity should have similar artifacts on a system (i.e., use the same resources); and (b)
changing the same attributes (e.g., data type) for different instructions should result in a similar offset
in the space. We train LLVM IR statement embeddings using the skip-gram model [48], following
preprocessing to limit the vocabulary size.
5.1 Statement Preprocessing and Training
Preprocessing First, we ﬁlter out comments and metadata from statements. Then, identiﬁers and
immediate values (numeric constants, strings) are replaced with %ID and <INT/FLOAT/STRING>
respectively, where immediate values are fed separately to downstream RNNs. Lastly, data structures
are “inlined”, that is, their contents are encoded within the statement. Fig. 5 lists statements before
and after preprocessing.
store float %250, float* %82, align 4, !tbaa !1
%10 = fadd fast float %9, 1.3
%8 = load %"struct.aaa"*, %"struct.aaa"** %2
(a) LLVM IR
store float %ID, float* %ID, align 4
%ID = fadd fast float %ID, <FLOAT>
%ID = load { float, float }*, { float, float }** %ID
(b) inst2vec statements
Figure 5: Before and after preprocessing LLVM IR to inst2vec statements.
5
Table 1: inst2vec training dataset statistics
Discipline Dataset Files LLVM IR V ocabulary XFG Stmt.
Lines Size Pairs
Machine Learning Tensorﬂow [1] 2,492 16,943,893 220,554 260,250,973
High-Performance Computing AMD APP SDK [9] 123 1,304,669 4,146 45,081,359
BLAS [22] 300 280,782 566 283,856
Benchmarks NAS [57] 268 572,521 1,793 1,701,968
Parboil [59] 151 118,575 2,175 151,916
PolybenchGPU [27] 40 33,601 577 40,975
Rodinia [14] 92 103,296 3,861 266,354
SHOC [21] 112 399,287 3,381 12,096,508
Scientiﬁc Computing COSMO [11] 161 152,127 2,344 2,338,153
Operating Systems Linux kernel [42] 1,988 2,544,245 136,545 5,271,179
Computer Vision OpenCV [36] 442 1,908,683 39,920 10,313,451
NVIDIA samples [17] 60 43,563 2,467 74,915
Synthetic Synthetic 17,801 26,045,547 113,763 303,054,685
Total (Combined) — 24,030 50,450,789 8,565 640,926,292
Dataset Table 1 summarizes the code corpora and vocabulary statistics of the inst2vec dataset.
We choose corpora from different disciplines, including high-performance computing, benchmarks,
operating systems, climate sciences, computer vision, machine learning (using Tensorﬂow’s own
source code), and synthetically-generated programs. The code in the dataset is written in C, C++,
FORTRAN, and OpenCL, and is compiled for Intel CPUs as well as NVIDIA and AMD GPUs. The
ﬁles in the dataset were compiled to LLVM IR with Clang [44] and Flang [43], using compilation ﬂags
from the original code (if available) and randomly chosen compiler optimization (e.g.,-ffast-math )
and target architecture ﬂags.
For the synthetic corpus, we use both C code and the Eigen [ 31] C++ library. In particular, ran-
dom linear algebra operations are procedurally generated from high-level templates using different
parameters, such as data types, operations, and dimensions.
Setup and Training Given a set of XFGs created from the LLVM IR ﬁles, we generate neighboring
statement pairs up to a certain context size, following the skip-gram model [48]. A context of size
N includes all statement pairs that are connected by a path shorter or equal to N. To obtain the
pairs, we construct a dual graph in which statements are nodes, omitting duplicate edges. Following
this process, we discard statements that occur less than 300 times in the dataset, pairs of identical
statements, and perform subsampling of frequent pairs, similarly to Mikolov et al. [ 48]. We train
inst2vec with an embedding dimension of 200 for 5 epochs using Tensorﬂow [ 1]. The Adam
optimizer [37] is used with the default published hyperparameters and softmax cross-entropy loss.
5.2 Evaluation
Clustering Fig. 6 depicts the t-SNE [ 60] plots for trained inst2vec spaces with different XFG
context sizes, colored by statement and data type (legend in Appendix A). In the plots, we see that
both a context size of 1 statement in each direction (Fig. 6a) or 3 statements (Fig. 6c) generate large,
multi-type clusters, as well as outliers. This phenomenon eventually contributes to a lower ﬁnal
analogy score, due to inappropriate representation of inter-statement relations, as can be seen below.
Owing to these results, we choose a context size of 2 statements (Fig. 6b), which mostly consists of
separate, monochromatic clusters, indicating strong clustering w.r.t. instruction and data types. While
data type syntactic clusters are unsurprising, their existence is not trivial, since the dataset contains
diverse codebases rather than copies of the same functions with different types.
An example of a semantically-similar statement cluster can be found in data structures. In particular,
the top-5 nearest neighbors of operations on the complex data type “std::complex<float> ” include
“2 x float ” (i.e., a vector type). In fact, LLVM IR represents the complex data type as {float,
float} , so this property is generalized to any user-deﬁned data structure (struct ) with two ﬂoats.
Analogies and Tests We also evaluateinst2vec by automatically generating a list of statement
analogies (“a” is to “b” as “c” is to “?”, or “a:b; c:? ”) that appear in our vocabulary using the LLVM
6
50
-50
0
-50 0 50
(a) Context size = 1
50
0
-50
-100
100
50 0-50 (b) Context size = 2
80
100
60
40
20
0
-20
-60
-40
-80 -60 -40 0 -20 20 40 60 80 (c) Context size = 3
Figure 6: Two-dimensional t-SNE plots for learned embeddings (best viewed in color).
IR syntax. We then use the embeddings to ﬁnd the result by computing a-b+c and asking whether
the result is in the top-5 neighbors (cosine distance). Additionally, we automatically create relative
distance expressions using the LLVM IR reference categories [45] of the form d(a,b) <d(a,c) to
test whether statements that use different resources are further away than those who use the same.
Table 2 shows the analogy and test results for inst2vec trained on XFG as well as on CFG (control
ﬂow-only) and DFG (data ﬂow-only) for different context sizes. The analogies are divided into
different categories, including data types (i.e., transitions between types), options (e.g., fast math),
conversions (e.g., bit casting, extension, truncation), and data structures (e.g., vector-type equivalents
of structures). Below are examples of a type analogy:
%ID = add i64 %ID, %ID : %ID = fadd float %ID, %ID;
%ID = sub i64 %ID, %ID :? %ID = fsub float %ID, %ID
and a data structure analogy:
%ID = extractvalue { double, double } %ID, 0 : %ID = extractelement <2 x double> %ID, <TYP> 0;
%ID = extractvalue { double, double } %ID, 1 :? %ID = extractelement <2 x double> %ID, <TYP> 1
The results conﬁrm that over all scores, a context size of 2 is the best-performing conﬁguration, and
show that the XFG representation is more complete and leads to better embeddings than taking into
account control or data ﬂow alone.
Table 2: Analogy and test scores for inst2vec
Context
type
Context
Size
Syntactic Analogies Semantic Analogies Semantic Distance Test
Types Options Conversions Data Structures
CFG 1 0 (0 %) 1 (1.89 %) 1 (0.07 %) 0 (0 %) 51.59 %
2 1 (0.18 %) 1 (1.89 %) 0 (0 %) 0 (0 %) 50.47 %
3 0 (0 %) 1 (1.89 %) 4 (0.27 %) 0 (0 %) 53.79 %
DFG 1 53 (9.46 %) 12 (22.64 %) 2 (0.13 %) 4 (50.00 %) 56.79 %
2 71 (12.68 %) 12 (22.64 %) 12 (0.80 %) 3 (37.50 %) 57.44 %
3 67 (22.32 %) 18 (33.96 %) 40 (2.65 %) 4 (50.00 %) 60.38 %
XFG 1 101 (18.04 %) 13 (24.53 %) 100 (6.63 %) 3 (37.50 %) 60.98 %
2 226 (40.36 %) 45 (84.91 %) 134 (8.89 %) 7 (87.50 %) 79.12 %
3 125 (22.32 %) 24 (45.28 %) 48 (3.18 %) 7 (87.50 %) 62.56 %
6 Code Comprehension Experiments
In this section, we evaluate inst2vec on three different tasks, comparing with manually-extracted
features and state-of-the-art specialized deep learning approaches. Throughout all tasks, we use
the same neural network architecture and our pre-trained embedding matrix from Section 5, which
remains ﬁxed during training.
Training Our recurrent network (see schematic description in the Appendix B) consists of an
inst2vec input with an XFG context size of 2, followed by two stacked LSTM [33] layers with 200
units in each layer, batch normalization [35], a dense 32-neuron layer with ReLU activations, and
output units matching the number of classes. The loss function is a categorical cross-entropy trained
7
using Adam [37] with the default hyperparameters. Additionally, for the compute device mapping
and optimal thread coarsening factor prediction tasks, we train the LLVM IR statements with the
immediate values that were stripped from them during preprocessing (see Section 5). Further details
are given in Appendix C.
Datasets The algorithm classiﬁcation task uses the POJ-104 [49] dataset2, collected from a Peda-
gogical Open Judge system. The dataset contains 104 program classes written by 500 different people
(randomly selected subset per class). For the compute device mapping and optimal thread coarsening
factor prediction tasks, we use an OpenCL code dataset3 provided by Cummins et al. [18].
6.1 Algorithm Classiﬁcation
Using inst2vec , we construct an RNN that reads embedded source code and outputs a predicted
program class. We compare our approach with Tree-Based CNNs (TBCNN) [49], the best-performing
algorithm classiﬁer in the POJ-104 dataset. TBCNN constructs embeddings from Astract Syntax Tree
nodes of source code, and employs two specialized layers: tree convolutions and dynamic pooling.
Their network comprises 5 layers, where convolution and fully connected layers are 600-dimensional.
Our data preparation follows the experiment conducted by Mou et al. [49], splitting the dataset 3:1:1
for training, validation, and testing. To compile the programs successfully, we prepend#include
statements to each ﬁle. Data augmentation is then applied on the training set by compiling each ﬁle 8
times with different ﬂags (-O{0-3} , -ffast-math ).
Table 3: Algorithm classiﬁcation test accuracy
Metric Surface Features [49] RNN [49] TBCNN [49] inst2vec
(RBF SVM + Bag-of-Trees)
Test Accuracy [%] 88.2 84.8 94.0 94.83
Table 3 compares inst2vec (trained for 100 epochs) with the reported results of Mou et al. [ 49],
which contain TBCNN as well as a 600-cell RNN and a manual feature extraction approach (Surface
Features). The results show that inst2vec sets a new state-of-the-art with a 13.8 % decrease in error,
even though the dataset used to generate the embeddings does not include POJ-104 (see Table 1).
6.2 Heterogeneous Compute Device Mapping
Next, we use Neural Code Comprehension to predict whether a given OpenCL program will run
faster on a CPU (Intel Core i7-3820) or a GPU (AMD Tahiti 7970 and NVIDIA GTX 970) given
its code, input data size, and work-group size (i.e., number of threads that work in a group with
shared memory). To achieve that, we use the same experimental methodology presented by Cummins
et al. [ 18], removing their specialized OpenCL source rewriter and replacing their code token
embeddings with our XFGs and inst2vec . We concatenate the data and work-group sizes to the
network inputs, and train with stratiﬁed 10-fold cross-validation. We repeat the training 10 times with
random initialization of the network’s weights and report the best result.
In Table 4, inst2vec and inst2vec-imm (i.e., with immediate value handling) are compared with a
manual code feature extraction approach by Grewe et al. [29] and DeepTune [18], in terms of runtime
prediction accuracies and resulting speedup. The baseline for the speedup is a static mapping, which
selects the device that yields the best average case performance over all programs in the data set:
in the case of AMD Tahiti versus Intel i7-3820, that is the CPU and in the case of NVIDIA GTX
versus Intel i7-3820, it is the GPU. The results indicate that inst2vec outperforms Grewe et al. and
is on-par with DeepTune. We believe that the better predictions in DeepTune are the result of training
the embedding matrix in tandem with the high-level task, thereby specializing it to the dataset. This
specialized training is, however, surpassed by taking immediate values into account during training.
We present the result of the best immediate value handling method in Table 4 (inst2vec-imm ), and
the exhaustive results can be found in Appendix D.
2https://sites.google.com/site/treebasedcnn/
3https://www.github.com/ChrisCummins/paper- end2end- dl
8
Table 4: Heterogeneous device mapping results
Architecture Prediction Accuracy [%]
GPU Grewe et al. [29] DeepTune [18] inst2vec inst2vec-imm
AMD Tahiti 7970 41.18 73.38 83.68 82.79 88.09
NVIDIA GTX 970 56.91 72.94 80.29 82.06 86.62
Speedup
GPU Grewe et al. DeepTune inst2vec inst2vec-imm
AMD Tahiti 7970 3.26 2.91 3.34 3.42 3.47
NVIDIA GTX 970 1.00 1.26 1.41 1.42 1.44
6.3 Optimal Thread Coarsening Factor Prediction
Our third example predicts the best-performing thread coarsening factor, a measure of the amount of
work done per GPU thread, for a given OpenCL code. We again compare the achieved speedups of
inst2vec with manual features [46], DeepTune, and DeepTune with transfer learning applied from
the task in Section 6.2 (denoted by DeepTune-TL). Possible values for the coarsening factor are 1
(baseline for speedups), 2, 4, 8, 16, and 32. The results in Table 5 show that while inst2vec yields
better speedups than DeepTune-TL in only half of the cases (possibly due to the embedding special-
ization in DeepTune), the manually-extracted features are consistently outperformed by inst2vec .
Moreover, inst2vec-imm is consistently on-par with DeepTune, but improves inconsistently on
inst2vec (on the AMD Tahiti and the NVIDIA GTX only), and fails to outperform DeepTune-TL.
This can be explained by the small size of the training data for this task (17 programs with 6 different
thread coarsening factors for each hardware platform). The optimal device mapping task (Section
6.2), on the other hand, features 680 programs for each platform.
Table 5: Speedups achieved by coarsening threads
Computing Platform Magni et al. [46] DeepTune [18] DeepTune-TL [18] inst2vec inst2vec-imm
AMD Radeon HD 5900 1.21 1.10 1.17 1.37 1.28
AMD Tahiti 7970 1.01 1.05 1.23 1.10 1.18
NVIDIA GTX 480 0.86 1.10 1.14 1.07 1.11
NVIDIA Tesla K20c 0.94 0.99 0.93 1.06 1.00
7 Conclusion
In this paper, we have empirically shown that semantics of statements can be successfully recovered
from their context alone. This recovery relies both on proper granularity, where we propose to use
ﬁltered LLVM IR instructions; and on the grouping of statements, for which we use a mixture of data-
and control-ﬂow. We use our proposed representation to perform three high-level classiﬁcation and
prediction tasks, outperforming all manually-extracted features and achieving results that are on-par
with (and better than) two inherently different state-of-the-art specialized DNN solutions.
With this work, we attempt to pave the way towards mechanized code comprehension via machine
learning, whether the code was authored by a human or automatically-generated. Further research
could be conducted in various directions. Rather than directly using statements, the representation
may be reﬁned using part-based models, which have already been applied successfully in language
models [55]. inst2vec can also be used as a basis for neural code interpretation, using a modiﬁed
Differentiable Neural Computer [28] to enable execution of arbitrary code over DNNs.
Acknowledgments
We wish to thank Theodoros Theodoridis, Kﬁr Levy, Tobias Grosser, and Yunyan Guo for fruitful
discussions. The authors also acknowledge MeteoSwiss, and thank Hussein Harake, Colin McMurtrie,
and the whole CSCS team for granting access to the Greina machines, and for their excellent technical
support. TBN is supported by the ETH Postdoctoral Fellowship and Marie Curie Actions for People
COFUND program.
9
References
[1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro,
Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow,
Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser,
Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek
Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal
Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete
Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-
scale machine learning on heterogeneous systems, 2015. Software available from tensorﬂow.org.
[2] Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles Sutton. Suggesting accurate
method and class names. In Proceedings of the 2015 10th Joint Meeting on Foundations of
Software Engineering, ESEC/FSE 2015, pages 38–49, New York, NY , USA, 2015. ACM.
[3] Miltiadis Allamanis, Earl T. Barr, Premkumar T. Devanbu, and Charles A. Sutton. A survey of
machine learning for big code and naturalness. CoRR, abs/1709.06182, 2017.
[4] Miltiadis Allamanis and Marc Brockschmidt. Smartpaste: Learning to adapt source code. CoRR,
abs/1705.07867, 2017.
[5] Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. Learning to represent
programs with graphs. CoRR, abs/1711.00740, 2017.
[6] Miltiadis Allamanis, Hao Peng, and Charles A. Sutton. A convolutional attention network for
extreme summarization of source code. CoRR, abs/1602.03001, 2016.
[7] Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: Learning distributed
representations of code. CoRR, abs/1803.09473, 2018.
[8] Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. A general path-based representation
for predicting program properties. CoRR, abs/1803.09544, 2018.
[9] AMD. AMD OpenCL accelerated parallel processing SDK. https://developer.amd.com/
amd- accelerated- parallel- processing- app- sdk/ .
[10] P. Balaprakash, R. B. Gramacy, and S. M. Wild. Active-learning-based surrogate models for
empirical performance tuning. In 2013 IEEE International Conference on Cluster Computing
(CLUSTER), pages 1–8, Sept 2013.
[11] M. Baldauf, A. Seifert, J. Förstner, D. Majewski, M. Raschendorfer, and T. Reinhardt. Opera-
tional Convective-Scale Numerical Weather Prediction with the COSMO Model: Description
and Sensitivities. Monthly Weather Review, 139:3887–3905, December 2011.
[12] Pavol Bielik, Veselin Raychev, and Martin Vechev. PHOG: Probabilistic model for code. In
Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International
Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research,
pages 2933–2942, New York, New York, USA, June 2016. PMLR.
[13] Rudy Bunel, Alban Desmaison, M. Pawan Kumar, Philip H. S. Torr, and Pushmeet Kohli.
Learning to superoptimize programs. International Conference on Learning Representations,
2017.
[14] Shuai Che, Michael Boyer, Jiayuan Meng, David Tarjan, Jeremy W. Sheaffer, Sang-Ha Lee, and
Kevin Skadron. Rodinia: A benchmark suite for heterogeneous computing. In Proceedings of
the 2009 IEEE International Symposium on Workload Characterization (IISWC), IISWC ’09,
pages 44–54, Washington, DC, USA, 2009. IEEE Computer Society.
[15] Cliff Click and Keith D. Cooper. Combining analyses, combining optimizations. ACM Transac-
tions on Programming Languages and Systems, 17, 1995.
[16] Cliff Click and Michael Paleczny. A simple graph-based intermediate representation. SIGPLAN
Not., 30(3):35–49, March 1995.
10
[17] NVIDIA Corporation. CUDA. http://developer.nvidia.com/object/cuda.html .
[18] Chris Cummins, Pavlos Petoumenos, Zheng Wang, and Hugh Leather. End-to-end deep learning
of optimization heuristics. In PACT. ACM, 2017.
[19] Ron Cytron, Jeanne Ferrante, Barry K. Rosen, Mark N. Wegman, and F. Kenneth Zadeck.
Efﬁciently computing static single assignment form and the control dependence graph. ACM
Trans. Program. Lang. Syst., 13(4):451–490, October 1991.
[20] Hoa Khanh Dam, Truyen Tran, and Trang Pham. A deep language model for software code.
CoRR, abs/1608.02715, 2016.
[21] Anthony Danalis, Gabriel Marin, Collin McCurdy, Jeremy S. Meredith, Philip Roth, Kyle
Spafford, Vinod Tipparaju, and Jeffrey Vetter. The Scalable HeterOgeneous Computing (SHOC)
benchmark suite. pages 63–74, January 2010.
[22] Jack Dongarra. Basic linear algebra subprograms technical forum standard. page 1 — 111,
2002.
[23] Jeffrey L. Elman. Finding structure in time. Cognitive Science, 14(2):179 – 211, 1990.
[24] Jeanne Ferrante, Karl J. Ottenstein, and Joe D. Warren. The program dependence graph and its
use in optimization. ACM Trans. Program. Lang. Syst., 9(3):319–349, July 1987.
[25] GitHub. GitHub Octoverse. https://octoverse.github.com/ , 2017.
[26] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectiﬁer neural networks.
2011.
[27] Scott Grauer-Gray, Lifan Xu, Robert Searles, Sudhee Ayalasomayajula, and John Cavazos.
Auto-tuning a high-level language targeted to GPU codes. 2012.
[28] Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-
Barwi´nska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou,
et al. Hybrid computing using a neural network with dynamic external memory. Nature,
538(7626):471–476, 2016.
[29] Dominik Grewe, Zheng Wang, and Michael O’Boyle. Portable mapping of data parallel
programs to OpenCL for heterogeneous systems. pages 1–10, February 2013.
[30] Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. Deep API learning. In
Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of
Software Engineering, FSE 2016, pages 631–642, New York, NY , USA, 2016. ACM.
[31] Gaël Guennebaud and Benoît Jacob et al. Eigen v3. http://eigen.tuxfamily.org , 2010.
[32] Zellig S. Harris. Distributional Structure, pages 3–22. Springer Netherlands, Dordrecht, 1981.
[33] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation,
9(8):1735–1780, 1997.
[34] Chun-Hung Hsiao, Michael Cafarella, and Satish Narayanasamy. Using web corpus statistics
for program analysis. In Proceedings of the 2014 ACM International Conference on Object
Oriented Programming Systems Languages & Applications, OOPSLA ’14, pages 49–65, New
York, NY , USA, 2014. ACM.
[35] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training
by reducing internal covariate shift. In Proceedings of the 32Nd International Conference
on International Conference on Machine Learning - Volume 37 , ICML’15, pages 448–456.
JMLR.org, 2015.
[36] Itseez. Open source computer vision library. https://github.com/itseez/opencv , 2015.
[37] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR,
abs/1412.6980, 2014.
11
[38] Leslie Lamport. Time, clocks, and the ordering of events in a distributed system. Commun.
ACM, 21(7):558–565, July 1978.
[39] Chris Lattner and Vikram Adve. LLVM: a compilation framework for lifelong program analysis
transformation. In International Symposium on Code Generation and Optimization, 2004. CGO
2004., pages 75–86, March 2004.
[40] Hugh Leather, Edwin Bonilla, and Michael O’Boyle. Automatic feature generation for machine
learning based optimizing compilation. In Proceedings of the 7th Annual IEEE/ACM Interna-
tional Symposium on Code Generation and Optimization, CGO ’09, pages 81–91, Washington,
DC, USA, 2009. IEEE Computer Society.
[41] Dor Levy and Lior Wolf. Learning to align the source code to the compiled object code. In
Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney,
NSW, Australia, 6-11 August 2017, pages 2043–2051, 2017.
[42] Linux. Linux kernel source code (version 4.15.1). https://www.kernel.org/ .
[43] LLVM. Flang: a FORTRAN compiler frontend for LLVM. https://github.com/
flang- compiler/flang .
[44] LLVM. Clang: a C language family frontend for LLVM v4.0.0. http://clang.llvm.org/ ,
2017.
[45] LLVM. LLVM language reference manual. https://llvm.org/docs/LangRef.html , 2018.
[46] Alberto Magni, Christophe Dubach, and Michael O’Boyle. Automatic optimization of thread-
coarsening for graphics processors. In Proceedings of the 23rd International Conference on
Parallel Architectures and Compilation, PACT ’14, pages 455–466, New York, NY , USA, 2014.
ACM.
[47] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efﬁcient estimation of word
representations in vector space. CoRR, abs/1301.3781, 2013.
[48] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed
representations of words and phrases and their compositionality. In Proceedings of the 26th
International Conference on Neural Information Processing Systems - Volume 2 , NIPS’13,
pages 3111–3119, USA, 2013. Curran Associates Inc.
[49] Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional neural networks over
tree structures for programming language processing. In Proceedings of the Thirtieth AAAI
Conference on Artiﬁcial Intelligence, AAAI’16, pages 1287–1293. AAAI Press, 2016.
[50] Mircea Namolaru, Albert Cohen, Grigori Fursin, Ayal Zaks, and Ari Freund. Practical aggrega-
tion of semantical program properties for machine learning based optimization. In Proceedings
of the 2010 International Conference on Compilers, Architectures and Synthesis for Embedded
Systems, CASES ’10, pages 197–206, New York, NY , USA, 2010. ACM.
[51] Ricardo Nobre, Luiz G. A. Martins, and João M. P. Cardoso. A graph-based iterative compiler
pass selection and phase ordering approach. InProceedings of the 17th ACM SIGPLAN/SIGBED
Conference on Languages, Compilers, Tools, and Theory for Embedded Systems, LCTES 2016,
pages 21–30, New York, NY , USA, 2016. ACM.
[52] Patrick Pantel. Inducing ontological co-occurrence vectors. In Proceedings of the 43rd Annual
Meeting on Association for Computational Linguistics, ACL ’05, pages 125–132, Stroudsburg,
PA, USA, 2005. Association for Computational Linguistics.
[53] Eunjung Park, John Cavazos, and Marco A. Alvarez. Using graph-based program characteriza-
tion for predictive modeling. In Proceedings of the Tenth International Symposium on Code
Generation and Optimization, CGO ’12, pages 196–206, New York, NY , USA, 2012. ACM.
[54] Veselin Raychev, Martin Vechev, and Eran Yahav. Code completion with statistical language
models. SIGPLAN Not., 49(6):419–428, June 2014.
12
[55] Cicero Dos Santos and Bianca Zadrozny. Learning character-level representations for part-of-
speech tagging. In Eric P. Xing and Tony Jebara, editors, Proceedings of the 31st International
Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research,
pages 1818–1826, Bejing, China, June 2014. PMLR.
[56] Eric Schkufza, Rahul Sharma, and Alex Aiken. Stochastic superoptimization. In Proceedings of
the Eighteenth International Conference on Architectural Support for Programming Languages
and Operating Systems, ASPLOS ’13, pages 305–316, New York, NY , USA, 2013. ACM.
[57] Sangmin Seo, Gangwon Jo, and Jaejin Lee. Performance characterization of the nas parallel
benchmarks in opencl. In Proceedings of the 2011 IEEE International Symposium on Workload
Characterization, IISWC ’11, pages 137–148, Washington, DC, USA, 2011. IEEE Computer
Society.
[58] Vugranam C. Sreedhar and Guang R. Gao. A linear time algorithm for placing phi-nodes. In
Proceedings of the 22Nd ACM SIGPLAN-SIGACT Symposium on Principles of Programming
Languages, POPL ’95, pages 62–73, New York, NY , USA, 1995. ACM.
[59] John A. Stratton, Christopher Rodrigues, I-Jui Sung, Nady Obeid, Li-Wen Chang, Nasser
Anssari, Geng Daniel Liu, and Wen-mei W. Hwu. Parboil: A revised benchmark suite for
scientiﬁc and commercial throughput computing. Center for Reliable and High-Performance
Computing, 2012.
[60] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE.Journal of Machine
Learning Research, 9:2579–2605, 2008.
[61] Martin Vechev and Eran Yahav. Programming with "big code". Found. Trends Program. Lang.,
3(4):231–284, December 2016.
[62] Baoezeng Wang, Xiaochun Yang, and Guoren Wang. Detecting copy directions among programs
using extreme learning machines. 2015:1–15, 05 2015.
[63] Martin White, Michele Tufano, Christopher Vendome, and Denys Poshyvanyk. Deep learning
code fragments for code clone detection. In Proceedings of the 31st IEEE/ACM International
Conference on Automated Software Engineering, ASE 2016, pages 87–98, New York, NY , USA,
2016. ACM.
[64] Xiaojun Xu, Chang Liu, Qian Feng, Heng Yin, Le Song, and Dawn Song. Neural network-based
graph embedding for cross-platform binary code similarity detection. CoRR, abs/1708.06525,
2017.
[65] Yixiao Yang, Yu Jiang, Ming Gu, Jiaguang Sun, Jian Gao, and Han Liu. A language model for
statements of software code. In Proceedings of the 32Nd IEEE/ACM International Conference
on Automated Software Engineering, ASE 2017, pages 682–687, Piscataway, NJ, USA, 2017.
IEEE Press.
13

ISO Certiﬁed 
# NBA Accredited
Courses
TCS & NAAC
Accredited
A+
NAAC
ISO 9001:2015
COUNSELLING CODE
COIMBATORE
23572622
Top Admission
Top 
Placement
Top Result
COUNSELLING CODE
KARUR
COUNSELLING CODE
COIMBATORE
Thiru. V.S.BALSAMY Chairman
(Approved by AICTE and Aﬃliated to Anna University,Chennai)
(Autonomous Institution)
(Applied for Autonomous Status & Waiting for the Result)
(Approved by AICTE and Aﬃliated to Anna University,Chennai)
NAAC
A+
2023 PLACEMENT  IN TOP COMPANIES 1850 OFFERS
13711371
1850
ues
global
Infosys
Navigate your next
KEERTHANA P T - CSE
KANGEYAM
SONY(6.25L),CAPGEMINI(4.25L), 
COGNIZENT(4 L),LTIMINDTREE(4L),TECH M(3.25L)
FURSHANA FATHIMA S - CSE
DHARAPURAM
VIRTUSA(5.5L),ACCENTURE(4.5L),
CAPGEMINI(4.25L),HEXAWARE(4L),TCS(3.36L)
SUMITHRAJOTHI  G – CSE
ANAIMALAI,COIMBATORE
VIRTUSA(5.5L),ACCENTURE(4.58L),
CAPGEMINI(4.25L),COGNIZANT(4L),LTIMINDTREE(4L)
DHARANI D - CSE
KARUR
VIRTUSA(5.5L),ACCENTURE(4.5L),
COGNIZANT(4L),TCS(3.8L),TECH M(3.25L)
ANANDH S - CSE
MADURAI
VIRTUSA(5.5L),ACCENTURE(4.5L), 
CAPGEMINI(4.25L),COGNIZANT(4L),
DXC(4.2L)
SUWETHA M - CSE
VELLAKOVIL
ACCENTURE(4.2L),DXC(4.2L),
HEXAWARE(4L),TCS(3.36L),TECH M(3.25L)
SOWNDARYA S - CSE
MULANUR
ACCENTURE(4.2L),DXC(4.2L),COGNIZANT(4L),
TCS(3.36L),TECH M(3.25L)
NARMATHA N - CSE
DHARAPURAM
HEXAWARE(6 L), ZOHO(5.6L),VIRTUSA (5.5L), 
ACCENTURE(4.5L),LTIMINDTREE(4L)
YOGETHA S - CSE
MUTHUR
COGNIZANT(4L),LTIMINDTREE(4L), NTT DATA(3.5L),
TCS(3.36L), TECHM(3.25L)
MONISHA G – CSE
KANGAYAM,TIRUPPUR
DIVYA K - CSE
PAPPANGKATTUPUDHUR
VIRTUSA(5.5L),ACCENTURE(4.5L),
HCL(4.25L),DXC(4.2L),COGNIZANT(4L),TCS(3.86L)
AKALYA S - CSE
MULANUR
CAPGEMINI(5.75L),VIRTUSA(5.5L),
HCL(4.25L),DXC(4.2L),LTIMINDTREE(4L),
HEXAWARE(4L) 
PRADEEPA M - CSE
KANGAYAM
VIRTUSA(5.5L),ACCENTURE(4.5L),
DXC(4.2L),LTIMINDTREE(4L),TCS(3.86L),
TECH M(3.25L)
LOGADHARSHINI P - CSE
KANGAYAM
ACCENTURE(4.5L),DXC(4.2L),
LTIMINDTREE (4L),COGNIZENT(4L),
TCS(3.86L),TECH M(3.25L)
AKASHKUMAR S - CSE
MADUKKARAI MARKET,COIMBATORE
ZEMOSO(6.89L),CAPGEMINI(5.75L),VIRTUSA(5.5L),
COGNIZANT(4L),
LTIMINDTREE(4L),TCS(3.36L)
RAMYA R - CSE
 KARUR
LTIMINDTREE(6.5L), VIRTUSA(5.5L),ACCENTURE(4.5L),
HCL(4.25L),COGNIZANT(4L),TCS(3.86L)
MONISHA R - CSE
NAMAKKAL
VIRTUSA (5.5L), ACCENTURE (4.5L),CAPGEMINI (4.25L), 
LTIMINDTREE (4L),TCS (3.86L),TECH M(3.25L)
DHIWICSHEN M - CSE
NAMAKKAL
TECHM(5.5L),VIRTUSA(5.5L),ACCENTURE(4.5L), 
DXC(4.2L),TCS(3.86L),EXCELACOM(4L)
KAVIYARASU E - CSE
PALANI
HEXAWARE(6L),ZOHO(5.6L),VIRTUSA(5.5L),
CAPGEMINI(4.25L),COGNIZANT(4L),TCS(3.36L) 
KARTHICK BALAJI K – CSE
KANGEYAM
CAPGEMINI(7.5L),VIRTUSA(7.5L),HEXAWARE(6L),
ZOHO(5.6L),TCS(3.36L),LTIMINDTREE(4L)
BRINDHA M – CSE
RAYAPURAM,TIRUPPUR
STUDENTS PLACED IN 6 COMPANIES 
STUDENTS PLACED IN 5 COMPANIES 
SREENIDHI B - CSE
ERODE
COGNIZANT(6.75L),HEXAWARE(6L),TECH M(5.5L),
LTIMINDTREE(5L),CAPGEMINI(4.25L), 
HCL(4.25L),ACCENTURE(4.2L),TCS(3.36L)
 
STUDENT PLACED IN 8 COMPANIES 
STUDENTS PLACED IN 7 COMPANIES 
VISWAS M - CSE
THENI
TEJAS (10L),ZOHO(5.6L),VIRTUSA(5.5L),
COGNIZANT(4L),HEXAWARE(4L),
LTIMINDTREE(4L),TCS(3.36L)
DEIVA DHARSHINI R – CSE
DHARAPURAM
VIRTUSA(5.5L),LTIMINDTREE(5L),DXC(4.2L),
ACCENTURE(4.1L),HEXAWARE(4L),
COGNIZENT(4L),TCS(3.36L)
DHIRUSHYA P – CSE
SULUR,COIMBATORE
CAPGEMINI(7.5L),STEAM A(6L TO 8L), 
VIRTUSA(5.5L),ACCENTURE(4.5L),
HEXAWARE(4L),COGNIZENT(4L),
LTIMINDTREE(4L)
COMPUTER SCIENCE AND ENGINEERING
ZOHO(7L),HCL(6L),HEXAWARE(6L),CAPGEMINI(5.75L),
VIRTUSA(5.5L),LTIMINDTREE(4L)
ACCENTURE(6.58L),HCL(6L),VIRTUSA(5.5L),
DELOITTE(4.5L),TCS(3.36L)
DINESHKUMAR P - CSE
DHARAPURAM
HEXAWARE(4L),MAVENTIC(4.5L)
TCS(3.36L),TECH M(3.25L)
KALIDOSS M - CSE
THANJAVUR
ACCENTURE(4.5L),DXC (4.2L),
TCS(3.86L),TECH M(3.25L)
SRI SARAN K - CSE
PALANI
CAPGEMINI(7.5L),TCS(7L), 
DXC(4.2L),TECH M(3.25L)
MOHANLAL P - CSE
SATHYAMANGALAM,ERODE
VURAM(4.5L),NTT DATA(4L), 
 COGNIZANT(4L),LTIMINDTREE(4L)
DINESH M - CSE
AVINASHI, TIRUPPUR
ACCENTURE(4.5L),HEXAWARE(4L),
 COGNIZANT(4L),TECH M(3.25L)
SYDHA NOWFAA N - CSE
TIRUPPUR
ACCENTURE(6.58L),NTT DATA(4L), 
 COGNIZANT(4L),TCS(3.36L)
SUGANTHARANI F - CSE
ANNUR,COIMBATORE
SONY(6.25L),VIRTUSA(5.5L),
LTIMINDTREE(4L), COGNIZANT(4L)
RUBHA SHRI V - CSE
METTUVAVI,COIMBATORE
ACCENTURE(4.5L),CAPGEMINI(4.25L),
DXC (4.2L),LT MINDTREE(4L)
PRITHIKA GOLCHHA - CSE
KOVAIPUDUR,COIMBATORE
CAPGEMINI(4.25L),COGNIZANT(4L),
DXC(4.2L),TCS(3.36L)
PRASANTH U - CSE
VELLALORE,COIMBATORE
VIRTUSA(5.5L),CAPGEMINI(4.25L),
 COGNIZANT(4L),LTIMINDTREE(4L)
SAFIYA SANOFER K-CSE
KARUR
VIRTUSA(5.5L),DXC(4.2L),
COGNIZANT(4L),TCS(3.86L)
VIGNESHWARAN  P-CSE
MADURAI
CAPGEMINI(5.75L),ELEVANT(4.5L),
VURAM(4.2L),TCS(3.36L)
SABARI S - CSE
ERODE
COGNIZANT(4L),LTIMINDTREE(4L),
NTT DATA(3.5L),TCS(3.36L)
RAGAVAN S - CSE
NAMAKKAL
AMADEUS(11.77L),ZOHO(5.6L),
VIRTUSA(5.5L),DXC(4.2L)
PAVITHRAN P - CSE
THANJAVUR
HEXAWARE(4L),DXC(4.2L), 
KAAR(6.5L),ACCENTURE(4.5L)
DIVYA S - CSE
HOSUR
RENTLY(8L),HEXAWARE(6L), 
ZOHO(5.6L),VIRTUSA(5.5L)
ASHOK N - CSE
VELLAKOVIL
HEXAWARE(6L),TECH M(5.5L), 
 COGNIZANT(4L),TCS(3.36L)
ASHOK E - CSE
TIRUPPUR
ACCENTURE(4.5L),COGNIZANT(4L),
NTT DATA(3.5L),TCS(3.36L)
ARUNKUMAR R - CSE
DHARAPURAM
VIRTUSA(5.5L),ACCENTURE(4.5L),
LTIMINDTREE(4L),TCS(3.36L)
VENNILAVU PM-CSE
PUDUPPAI
TCS(7 L),VIRTUSA(5.5L),
LTIMINDTREE(4.5 L),TECH M(3.2L)
GOKUL S - CSE
THUMBALAM,MUSIRI
CAPGEMINI(4.25L),DXC(4.2L),
HEXAWARE(4L),TECH M(3.25L)
DURKA M - CSE
DHARAPURAM
ACCENTURE(4.5L),LTIMINDTREE(4L), 
HCL(4L),TECH M(3.25L)
STUDENTS PLACED IN 4 COMPANIES 
NARESH R - CSE
KARUR
TEJAS(10L),VIRTUSA(7.5 L), 
HEXAWARE(6L),ZOHO(5.6L),TCS(3.36L)
SIVAPRAKASH C - CSE
PALANI
VIRTUSA(7.5L), ZOHO(5.6 L),CAPGEMINI(4.25L),
COGNIZANT(4L),TCS(3.36L)
SANJITH S - CSE
TIRUPPUR
HEXAWARE(6L),LUMEN(6L),VIRTUSA(5.5L), 
ACCENTURE(4.5L),TCS(3.36L)
NAVEEN  P - CSE
KARUR
ACCENTURE(4.5L),COGNIZANT(4L),
NTT DATA(3.5 L),TCS(3.36 L),TECH M(3.25L)
NAVEEN M - CSE
NAMAKKAL
ZOHO(5.6L),VIRTUSA(5.5L), 
CAPGEMINI(4.25L),DXC(4.2L),TCS(3.36L)
KRISHNA PRASATH A - CSE
CHIDAMBARAM
ZOHO(5.6L),ACCENTURE(4.5L),
CAPGEMINI(4.25L),DXC(4.2L),HEXAWARE(4L) 
STUDENTS PLACED IN 3 COMPANIES 
CHANDRU N - CSE
KARUR
HEXAWARE (6L),LTIMINDTREE(5L),
COGNIZANT(4L)
BINDU S - CSE
HOSUR
ODESSA(7.5L),COGNIZANT(4L),
HEXAWARE(4L)
ARUN PRASATH R - CSE
KARUR
ACCENTURE(4.5L),TCS(3.86L), 
TECH M(3.25L)
ANUPRIYA A - CSE 
DHARMAPURI
INFOSYS(4.45L),COGNIZANT(4L), 
TECH M(3.25L)
AKASH K - CSE
KARUR
ACCENTURE(4.5L),DXC(4.2L),
TCS(3.36L)
KARTHIK C - CSE
DINDIGUL
ZOHO(5.6L),ACARDIA(5.5L),
INFOSYS(5.3L) 
KARTHICK M - CSE
MADURAI
SOTI(7.5L),CAPGEMINI(4.25L),
COGNIZANT(4L)
HARITHA SREE S - CSE
DHARAPURAM
RENTLY(6.5L),HEXAWARE(6L),
VIRTUSA(5.5L)
GOPI KRISHNA E - CSE
KOPPAMPATTI
SOTI(7.5L),DXC(4.2 L),
TCS(3.36L)
GOKULA KRISHNAN G - CSE
KANGAYAM
ACCENTURE(4.5L),DXC(4.2L),
COGNIZANT(4 L)
DHARSHANA B - CSE
KARUR
DXC(4.2L),TCS(3.86L),
TECH M(3.25L)
DHARANI S - CSE
VELLAKOVIL
VIRTUSA(5.5L),ACCENTURE(4.5L),
TCS(3.36L)
GOKUL S - CSE
TIRUPPUR
LTIMINDTREE(4L),TCS(3.36L),
 TECHM(3.25L)
FELOOMI K - CSE
KARUR
VIRTUSA(5.5L),DXC(4.2L),
TCS(3.86L)
MUTHUKUMAR B - CSE
TIRUPPUR
TCS(7L),ELEVANT(4.5L), 
ACCENTURE(4.5L)
KRISHNAN S - CSE
TIRUPPUR
ZOHO(5.6L),VIRTUSA(5.5L), 
TCS(3.36L)
KRISHNA A T - CSE
CHENNIMALAI
VURAM(4.5L),ELEVANT(4.5L),
TCS(3.36L)
KAVIYARASU N - CSE
TIRUPPUR
ODESSA(7.5L),VIRTUSA(5.5L),
HEXAWARE(4L) 
MUTHUGANESH S - CSE
SIVAGANGAI
VURAM(4.5L),NTT DATA(3.5L),
 TCS(3.36L)
LAVANESH S - CSE
NAMAKKAL
ACCENTURE (4.5L),DXC(4.2L),
TCS(3.86L)
PARTHIBAN A - CSE
TRICHY
DXC(4.2L),TCS(3.36L) , 
TECH M(3.25L)
NAVEEN AKASH G - CSE
NAMAKKAL
LTIMINDTREE(4 L) ,NTT DATA(3.5 L), 
TECH M(3.25 L)
PRASATH KUMAR S - CSE
TIRUPPUR
TCS(7.5L TO 7L),ACCENTURE(4.5L),
DXC(4.2L)
SANTHOSH K - CSE
KARUR
CAPGEMINI(5.75L),VIRTUSA(5.5L), 
TCS(3.86L)
ALAGU DAYANAND S - CSE
ARULANANDHA NAGAR, THANJAVUR
ACCENTURE(4.58L),NTT DATA(4L),
 COGNIZANT(4L)
SHANGAMITHRA V - CSE
NAMAKKAL
VIRTUSA(5.5L)DXC(4.2L),
TCS(3.86L)
SUJITH M - CSE
KARUR
VIRTUSA(5.5L),LTIMINDTREE(4L), 
TCS(3.36L)
SRUTHI M - CSE
 K.PARAMATHI
CAPGEMINI(4.25L),HEXAWARE(4L),
TECH M(3.25L)
SATHIYA SRI K - CSE
KARUR 
NTTDATA(3.5L),TCS(3.36L),
TECH M(3.25L)
SANTHIYA V - CSE
NATHAKADAIYUR  
HEXAWARE(6L),VIRTUSA(5.5L),
TCS(3.36L)
SIVASANKARI S-CSE
KARUR
ACCENTURE(4.5L),DXC(4.2L),
DELOITTE(4L)
STUDENTS PLACED IN 2 COMPANIES 
ADHITHYAN M - CSE
MANAPPARAI
CAPGEMINI(7.5L),COGNIZANT(4L)
ABIMITHRA S - CSE
KARUR
ACCENTURE(4.5L),TCS(3.86L)
ARAVINDHAN J - CSE
KARUR
VIRTUSA(5.5L),TCS(3.36L)
ARIHARAN D - CSE
PALANI
DXC(4.2L),MPHASIS(4L)
ARUN KUMAR D - CSE
ERODE
TCS(7.5L),VIRTUSA(5.5L)
BALAGURU M - CSE
KARUR
ODESSA(7.5L),HEXAWARE(4L)
BALAMURUGAN R - CSE
MANNARGUDI
ACCENTURE(4.5L),TCS(3.36L)
CHANDRASEKAR P - CSE
MANAPPARAI
INCEDO(5L),TECH M(3.25L)
DEEPIKA P - CSE
DINDIGUL
DXC(4.2L),TCS(3.36L)
DEVI PRABHA M - CSE
DHARAPURAM
ACCENTURE(4.5L),TECH M(3.25L)
DHARSHINI R - CSE
DHARAPURAM
DXC(4.2L),COGNIZANT(4L)
DINESH BABU P - CSE
THENI
VIRTUSA(5.5L),DXC(4.2L)
DIWAKAR S - CSE
NAMAKKAL
ACCENTURE(4.5L),DXC(4.2L)
HARIHARAN P - CSE
ERODE
VIRTUSA(5.5L),DXC(4.2L)
HARIHARASUDHAN J V - CSE
THIRUCHENGODE
ODESSA(7.5L),HEXAWARE(4L)
ANUSREE S - CSE
KUMITIPATHY,COIMBATORE
CAPGEMINI(4.25L),
DXC(4.2L),TCS(3.36L)
AMRIDHASRI A - CSE
KONGU MAIN ROAD,TIRUPPUR
VIRTUSA(5.5L), ACCENTURE(4L), 
LTIMINDTREE(4L)
AMARNATH D - CSE
VAVIPALLAYAM,TIRUPPUR
ACCENTURE(4.5L)NTT DATA(4L),
 COGNIZANT(4L)
KEERTHANA R - CSE
VEERAPANUR,COIMBATORE
CAPGEMINI(5.75L),VIRTUSA(5.5L),
DELOITTE(4.5L)
SOWMIYA A - CSE
VELLAKOVIL
HEXAWARE(4L),TCS(3.36L),
TECH M(3.25L)
BALA LOGESH M - CSE
UDUMALPET,TIRUPPUR
SONY(6.25L),
DXC(4.2L),COGNIZANT(4L)
DEEPIKA G S - CSE
KINATHUKADAVU,COIMBATORE
ACCENTURE(4.5L),
NTT DATA(4L),COGNIZANT(4L)
DIVYA K - CSE
PALLADAM,TIRUPPUR
VIRTUSA(5.5L), 
ACCENTURE(4.5L),LTIMINDTREE(4L)
KARTHIK M - CSE 
PODANUR,COIMBATORE
ACCENTURE(4.5L),
DXC(4.2L),LTIMINDTREE(4L)
KIRUTHIKA K - CSE
PETHAPPAMPATTI,TIRUPPUR
ACCENTURE(4.5L),COGNIZANT(4L),
TCS(3.36L)
RAGHUL SHAKTHI K - CSE
UDUMALPET,TIRUPPUR
DXC(4.2L),COGNIZANT(4L),
TCS(3.36L)
NIKIL N - CSE
NAGERCOIL, KANYAKUMARI
ACCENTURE(4.5L),
HEXAWARE(4L),LTIMINDTREE(4L)
PRIYADHARSHINI R V - CSE
AVINASHI,TIRUPPUR
DXC(4.2L),LTIMINDTREE(4L)
COGNIZANT(4L)
SWETHA K - CSE
POLLACHI, COIMBATORE
HCL(6L),NTT DATA(4L),TCS(3.36L)
THENMOZHI S - CSE
KINATHUKADAVU,COIMBATORE
ACCENTURE(4.5L), 
CAPGEMINI(4.25L),DXC(4.2L)
SANGEETHA S-CSE
 KARUR
VIRTUSA(5.5L),TCS(3.86L)
VENIMALATHY A - CSE
DHARMAPURI
VIRTUSA(5.5L), COGNIZANT(4L)
SOWMIYA G - CSE
 MAYILADUTHURAI
CAPGEMINI(4.25L),DXC(4.2L)
YUVASHANKARI K - CSE
 VELLAKOVIL 
 NTT DATA(3.5L),TCS(3.36L)
RANJINI T - CSE
KARUR
DXC(4.2L),TCS(3.36L)
THARUN PRASANTH S - CSE
 ERODE
DXC(4.2L),TCS(3.36L) 
SUJITH V P - CSE
KARUR
DXC(4.2L),TCS(3.86L)
SHIVA SREE K- CSE
PALANI
NTT DATA(3.5 L),TCS(3.36 L) 
SUDEEPA R-CSE
 KARUR
ACCENTURE(4.5L),TECH M(3.36L)
RAGUL R - CSE
TRICHY
MPHASIS(4L),NTT DATA(3.5L)
ILAN THENDRAL K V - CSE
THOOTHUKUDI
ACCENTURE(4.5L), NTT DATA(3.5L)
INDHUMATHI M - CSE
DHARAPURAM
VIRTUSA(5.5L),TCS(3.36L)
JAYAPRAKASH M - CSE
DINDIGUL
DXC(4.2L),LTIMINDTREE(4L)
RAMSANTHOSH S - CSE
TIRUPPUR
CAPGEMINI(5.75L),TCS(3.36L)
VIGNESH K - CSE
ERODE
VIRTUSA(5.5L),TCS(3.36L)
JEGAN M - CSE
MADURAI
NTT DATA(3.5L),TCS(3.36L)
SHAJITHA PARVEEN M - CSE
KARUR
ACCENTURE(4.5L),DXC(4.2L)
VISHNU SURIYAN K T - CSE
ERODE
ACCENTURE(4.5L),TECH M(3.25L)
VIGNESH M - CSE
PALANI
ACCENTURE(4.5L),DXC(4.2L)
KABILESH G - CSE
NAMAKKAL
VIRTUSA(5.5L),TCS(3.36L)
KISHOREDHARAN K R  - CSE
MUTHUR
HEXAWARE(6L),LTIMINDTREE(4L)
KIRUTHIKA U - CSE
PARAMATHI VELUR
ACCENTURE(4.5L), NTT DATA(3.5L)
KIRUTHIKA PARAMESHWARI  M - CSE
PUGALUR
VIRTUSA(5.5L),TCS(3.36L)
KAVIYADHARSHINI A - CSE
KANGAYAM
VIRTUSA(5.5L),CAPGEMINI(4.25L)
KAVIPRIYA G - CSE
ESANATHAM
VIRTUSA(5.5L), ACCENTURE(4.5L)
NANDHINI M - CSE
MULANUR
TCS(7L),VIRTUSA(7.5L)
MOULIKA P - CSE
KALLAKURICHI
COGNIZANT(4L),TCS(3.36L)
MATHIYARASI P - CSE
KANGEYAM
ACCENTURE(4.58L),DXC(4.2L)
MADHU D - CSE
KARUR
SONY(6.25L),DXC(4.2L)
KRISHNA NAVEEN R R - CSE
MADURAI
RENTLY(6.5L),DXC(4.2L)
SATHISH BALAJI M - CSE
DINDIGUL
VIRTUSA(5.5L),TCS(3.36L) 
SANJAY K - CSE
MUTHUR
ACCENTURE(4.5L),TECH M(3.25L)
PAVITHRA V - CSE
KANGEYAM
HEXAWARE(4 L),TCS(3.36L)
SRI VISHAL V - CSE
 KRISHNAGIRI
ACCENTURE(4.5L),TCS(3.86L)
SENTHILKUMAR G - CSE
TIRUPPUR
VIRTUSA(5.5L), ACCENTURE(4.5L)
AADHARSH S M - CSE
POOLUVAPATTI,TIRUPPUR
CAPGEMINI(4.25L),
DXC(4.2L)
AZHAGU SUNDARAM D - CSE
MELAKADAYANALLUR,TENKASI
DXC(4.2L),COGNIZANT(4L)
DEEPA K - CSE
METTUVAVI, COIMBATORE
CAPGEMINI(5.75L),
HOTFOOT(3.5L)
DINESH KARTHICK V - CSE
SUNDARAPURAM,COIMBATORE
VIRTUSA(5.5L),TCS(3.36L)
MOHANAPRIYA C - CSE
KARUR
HCL(6L),DXC(4.2L)
HARIHARAN K - CSE
NEYVELI,CUDDALORE
VIRTUSA(5.5L),ACCENTURE(4.5L)
JEEVABALAN S B - CSE
KARAMADAI,COIMBATORE
VIRTUSA(5.5L),COGNIZANT(4L)
JEYAKUMAR K - CSE
SURANDAI,TENKASI
VIRTUSA(5.5L),COGNIZANT(4L)
DOMINIC DEVANANDAM – CSE
COIMBATORE
DXC(4.2L),COGNIZANT(4L)
JOSEPH RAJ Y - CSE
SATTUR,VIRUDHUNAGAR
VIRTUSA(5.5L),CAPGEMINI(4.25L)
KUMARI AARJU - CSE
VELLALORE, COIMBATORE
DIVUM CORPORATE (3.6L TO 4.5L),
TCS(3.36L)
MURALIKRISHNAN M - CSE
KURALKUTTAI,TIRUPPUR
VIRTUSA(5.5L),LTIMINDTREE(4L)
MUTHAMIL SELVAN K - CSE
PERAVURANI,THANJAVUR
VIRTUSA(5.5L),CAPGEMINI(4.25L)
KAVIYA DHARSHINI S - CSE
TRICHY
HCL(6L),NTT DATA(3.5L)
LARANI K - CSE
PAPAMPATTI,TIRUPPUR
VIRTUSA(5.5L),TCS(3.36L)
PRASANTH BHARATHI A - CSE
TIRUPPUR NEW BUS STAND,TIRUPPUR
ACCENTURE(4.5L),DXC (4.2L)
RAVISHANKAR M M - CSE
MALUMICHAMPATTI, COIMBATORE
LTIMINDTREE(4L),MPHASIS(4L)
RAMANAN R V - CSE
VIRUTHACHALAM ,CUDDALORE 
VIRTUSA(5.5L),COGNIZANT(4L)
RITIKA G - CSE
DHARAPURAM,TIRUPPUR
ACCENTURE(4.5L),NTT DATA(4L)
KEERTHANA T – CSE
UDUMALPET, TIRUPPUR
VIRTUSA(5.5L),
CAPGEMINI(4.25L)
KAVITHA M - CSE
MADUKKARAI,COIMBATORE
HEXAWARE(6L),
CAPGEMINI(4.25L)
KEERTHANA R - CSE
VEERAPANUR,COIMBATORE
CAPGEMINI(5.75L),
VIRTUSA(5.5L)
SARANYA S - CSE
DHARAPURAM,TIRUPPUR.
VIRTUSA(5.5L),CAPGEMINI(4.25L)
TILAK BALA S - CSE
KARUNYA NAGAR,COIMBATORE
NTT DATA(4L),COGNIZANT(4L)
ANUDHARSHINI R - CSE
ORATHANADU,THANJAVUR
DXC (4.2L)
ARAVINDAKUMAR S - CSE
GOMANGALAM,COIMBATORE
MPHASIS(4L)
ARUNKUMAR A - CSE
VELLIYANKADU,TIRUPPUR
VIRTUSA(5.5L)
BATHRESH BAS B - CSE
THIRUVALLUR DISTRICT,CHENNAI
LTIMINDTREE(4L)
BHARATHIPRIYAN B - CSE
UDUMALPET,TIRUPPUR
COGNIZANT(4L)
DHANRAJPRABU V – CSE
PONNAKANI,COIMBATORE
NTT DATA(4L)
KANCHANA V - CSE
MADUKKARAI,COIMBATORE
VIRTUSA(5.5L)
KRISHNA PRIYA M - CSE
BALAMANALLUR,POLLACHI
DXC (4.2L)
LAVANYA - CSE
PODANUR, COIMBATORE
DXC(4.2L)
MAHADEVAN V - CSE
PONNAKKANI, COIMBATORE
VIRTUSA(5.5L)
STUDENTS PLACED IN 1 COMPANY
MATHU MITHA C - CSE
THENGAIKARANKUDIERUPPU 
KANYAKUMARI 
NTT DATA(4L)
MOHAMMED ASHIK S - CSE
MADUKKARAI, COIMBATORE
VIRTUSA(5.5L)
PRUTHIKA M - CSE
KINATHUKADAVU,COIMBATORE
NTT DATA (4L)
RAJAKUMARAN S - CSE
VEERAGANUR,SALEM
VIRTUSA(5.5L)
PRAKASHRAJ S - CSE
PANAPATTI,COIMBATORE
NTT DATA (4L)
PRUTHIKA M – CSE
KINATHUKADAVU, COIMBATORE
NTT DATA (4L)
RASUL MUTHU A - CSE
SUNDARAPURAM,COIMBATORE
COGNIZANT(4L)
ROSHINI K - CSE
POLLACHI,COIMBATORE
DXC(4.2L)
SEBINYA P - CSE
MADUKKARAI,COIMBATORE
VIRTUSA(5.5L)
SHANMUGA PRABHU M - CSE
UDUMALPET,TIRUPPUR
HOT FOOT (3.5L)
SINTHIYA M - CSE
POLLACHI,COIMBATORE.
DXC(4.2L)
SORNALATHA G - CSE
MADUKKARAI MARKET,COIMBATORE
NTT DATA(4L)
SUBHASHINI M - CSE
NAGERCOIL,KANYAKUMARI
ACCENTURE(4.5L)
SURYA R - CSE
PULANKINAR,UDUMALPET, 
TIRUPPUR
DXC(4.2L)
SUSILADEVI K - CSE
PANAPATTI,COIMBATORE
VIRTUSA(5.5L)
VENKATESH RAJA P - CSE
THIRUMAYAM,PUDUKKOTTAI
DXC(4.2L)
ABISHEK S - CSE
DINDIGUL
TCS(3.36L)
ASHIKA A - CSE
MANNARGUDI
LTIMINDTREE(4L)
BALAJI K - CSE
VELLAKOVIL
TECH M(3.25L)
DHASARATH K P - CSE
THAMARAIPALAYAM
TECH M(3.25L)
GEETHA T - CSE
KARUR
VIRTUSA(5.5L)
KISHORE KUMAR R - CSE
KARUR
MPHASIS(4L)
KAVIN KUMAR E - CSE
DINDIGUL
ODESSA(7.5L)
KAVIN S - CSE
TRICHY
VIRTUSA(5.5 L)
KAVINESH P - CSE
TIRUPPUR
VIRTUSA(5.5L)
KAVIPRIYA T - CSE
KANGAYAM
ELEVIANT (4.5L)
SINDHU T - CSE
DINDIGUL
DXC(4.2L)
KEERTHIKA A - CSE
KARUR
TCS(3.86L) 
 KURALARASAN P - CSE
DHARAPURAM
VIRTUSA(5.5L)
MADHUMIDHA M - CSE
DINDIGUL
TCS(3.36L)
MOHAMMED UMAR - CSE
KARUR
ACCENTURE(4.5L)
PRABU S - CSE
KARUR
VIRTUSA(5.5L)
PRABADEVI E - CSE
KARUR
TCS(3.36L)
PONNARASI V - CSE
PARAMATHI VELUR 
DXC(4.2L)
NIVESAN A - CSE
NAMAKKAL
DXC(4.2L)
SRI SRUTHI R - CSE
DHARAPURAM
TCS(3.86L)
VELLAISAMY C - CSE
KARUR
TCS(3.36L)
GOKULAKRISHNAN B - CSE
GOBICHETTIPALAYAM
DXC(4.2L)
VIDHYASAKAR V – CSE
UDUMALPET, TIRUPPUR
DIVUM CORPORATE(3.6L TO 4.5L)
VIDHYAMALATHI M - CSE
MADUKKARAI MARKET,COIMBATORE
HEXAWARE(4L)
SAJITH P - CSE
TRICHY
MPHASIS(4L)
GOWTHAM C - CSE
NAMAKKAL
LTIMINDTREE(4L)
RAJESH G - CSE
 ODDANCHATHIRAM
ACCENTURE(4.5L)
RAGHUL M - CSE
KARUR
VIRTUSA(5.5L)
PUGALENTHI R - CSE 
KARUR
NTT DATA(3.5L)
PRAVEENKUMAR K - CSE
ODDANCHATHIRAM
HEXAWARE(4L)
PRAKASH J - CSE
TIRUPPUR
ACCENTURE(4.5L)
THARUN KUMAR S R - CSE
MADURAI
TCS(3.36L)
SURYAPRAKASH S-CSE
KARUR
TCS(3.36L)
SUDHARSAN N - CSE
TIRUPPUR
TCS(3.86L)
SIVASUBRAMANIAM B - CSE
  KUMBAKONAM
TCS(3.36L)
SATHISH KUMAR M-CSE
MUTHUR
VIRTUSA(5.5L)
GOKULNATH S - CSE
DHARAPURAM
TCS(3.36L)
REVATHI P - CSE
KANGAYAM
DXC(4.2L)
PRAGADEESHWARAN V - CSE
THANJAVUR
VIRTUSA(5.5L)
SOWMIYA S B - CSE
 KARUR
TCS(3.86L)
                                      
SABARINATHAN B - CSE
PALANI
TCS(3.36L)
STUDENT PLACED IN 9 COMPANIES 
INFORMATION TECHNOLOGY
AMADEUS(11.77L),EMBEDUR(8L), INFOSYS(6.25L),
ZOHO(5.6L), HEXAWARE(4L),COGNIZANT(4L),LTIMINDTREE(4L),
TECH M (3.5L),TCS(3.36L)
MOHAMMED YASER ARAFATH S - IT
VAIYAPURI NAGAR,KARUR
STUDENT PLACED IN 8 COMPANIES 
ELAMATHI T - IT
MANNUR-POLLACHI
HEXAWARE(6L),ZOHO(5.6L),VIRTUSA(5.5L), 
CAPGEMINI(4.25L),DXC(4.2L),COGNIZANT(4L), 
LTIMINDTREE(4L),TCS(3.36L)
SOWMIYA P- IT
NATHAKADAIYUR
HEXAWARE(6L),VIRTUSA(5.5L),CAPGEMINI(5.5L),
ACCENTURE(4.5L), DELOITTE(4.5L),
LTIMINDTREE(4L),TCS(3.36L)
MAHILA DEVI S-IT
LANTHAKOTTAI
KAAR(6.5L),ACCENTURE(4.5L),
DELOITTE(4.5L),LTIMINDTREE(4L),
HEXAWARE(4L),COGNIZANT(4L),TCS(3.36L)
SANGAVI SHREE B-IT
GOVINDHAPURAM-DINDIGUL
HEXAWARE(6L),VIRTUSA(5.5L), 
CAPGEMINI(4.25L),COGNIZANT(4L),
 LTIMINDTREE(4L),HCL(4L),TCS(3.36L)
STUDENTS PLACED IN 7 COMPANIES 
HARUNYAA P-IT
VADUGAPATTI-THENI
CAPGEMINI(7.5L),TCS(7L),HEXAWARE(6L),
COGNIZANT(4L), LTIMINDTREE(4L),
TECH M(3.25L)
SELENA CLARA M-IT
THOOTHUKUDI
CAPGEMINI(7.5L)TCS(7L),
COGNIZANT(4L),LTIMINDTREE(4L),
NTT DATA(3.5L),TECH M(3.25L)
NARENDRA KUMAR L A - IT
TIRUPPUR
VIRTUSA(5.5L), MAVENTIC(5L), 
ACCENTURE(4.5L),DELOITTE(4.5L),
CAPGEMINI(4.25L), HEXAWARE(4L)
STUDENTS PLACED IN 6 COMPANIES 
KARTHIKEYAN V R - IT
KARUR
HEXAWARE(6L), VIRTUSA(5.5L), 
CAPGEMINI(4.25L),COGNIZANT(4L), 
LTIMINDTREE(4L)
KUSUMA R - IT
HOSUR
CODING MART(7L),VIRTUSA(5.5L),
DELOITTE(4.5L),CAPGEMINI(4.25L),
HEXAWARE(4L)
SOWMIYA R - IT
P VELUR
COGNIZANT(6.75L), KAAR(6.5L),
VIRTUSA(5.5L),DXC(4.2L),
LTIMINDTREE(4L)
VISHNUPRIYA R - IT
PUGALUR-KARUR
INCEDIO(5L), ACCENTURE(4.5L),
LTIMINDTREE(4L),COGNIZANT(4L),
TCS(3.36L)
NARMATHA G - IT
SUNDARAPURAM,COIMBATORE
VIRTUSA(5.5L),ACCENTURE(4.5L),
HEXAWARE(4L),COGNIZANT(4L),TCS(3.36L)
ENAYATHULLA A - IT
NAMAKKAL
HEXAWARE(6L), 
VIRTUSA(5.5L),DELOITTE(4.5L),
LTIMINDTREE(4L),TCS(3.36L)
STUDENTS PLACED IN 5 COMPANIES 
NIVETHA S - IT
ERODE
VIRTUSA(5.5L),ACCENTURE(4.5L),
CAPGEMINI(4.25L),HEXAWARE(4L)
DEEPIKA B - IT
P VELUR-NAMAKKAL
VIRTUSA(5.5L),ACCENTURE(4.5L),
DXC(4.2L),LTIMINDTREE(4L)
SURYA KUMAR N - IT
GOVINDHAPALAYAM-DINDIGUL
ZOHO(5.6L),DXC(4.2L),
COGNIZANT(4L),TCS(3.36L) 
HARIKRISHNAN V - IT
KARUR
CODING MART(7L),VIRTUSA(5.5L), 
CAPGEMINI(4.25L),HEXAWARE(4L)
AZHAGUMANI K - IT
KARUR
LTIMINDTREE(6.5L), ACCENTURE(4.5L), 
COGNIZANT(4L),TCS(3.36L)
AAKASH R - IT
KRISHNARAYAPURAM - KARUR
VIRTUSA(5.5L),ACCENTURE(4.5L)
DXC(4.2L),TCS(3.36L)
PRIYENGA P - IT
NAMAKKAL
CAPGEMINI(4.25L),DXC(4.2L), 
HEXAWARE(4L),TCS(3.36L)
KEERTHANA P - IT
KARUR
VIRTUSA(5.5L), ACCENTURE(4.5L),
CAPGEMINI(4.25L),TCS(3.36L)
VALLINAYAGAM S - IT
UDUMALPET,TIRUPPUR
HEXAWARE(6L),VIRTUSA(5.5L), 
COGNIZANT(4L),TCS(3.36L)
HARIPRASATH R - IT
UDUMALPET,TIRUPPUR
HEXAWARE(6L),VIRTUSA(5.5L), 
ACCENTURE(4.5L),LTIMINDTREE(4L)
DHIVYABHARATHI M - IT
SENJERIMALAI, TIRUPPUR.
VIRTUSA(5.5L),HEXAWARE(4L), 
COGNIZANT(4L),TCS(3.36L)
STUDENTS PLACED IN 4 COMPANIES 
BARATH J - IT
TIRUPPUR
ODESSA(7.5L), HEXAWARE(4L)
ROSHINI R - IT
KARUR
VIRTUSA(5.5L),TCS(3.36L)
CHANDRU R - IT
KARUR
DXC(4.2L),TCS(3.36L)
DHIVAKAR K - IT
NAMAKKAL
TIGER ANALYTICS(7.5 L),COGNIZANT(4 L)
JAYASURYA J - IT
NAMAKKAL
VIRTUSA(5.5L), HEXAWARE(4L)
BRINDHA M - IT
THIRUNAGAR,MADURAI.
CAPGEMINI(7.5L),DXC(4.2L),
COGNIZANT(4L)
MAHATHI K - IT
PALLAMARUTHAPATTI,KARUR
HEXAWARE(4L),LTIMINDTREE(4L), 
TECH M(3.25L)
SNEKA G - IT
P VELUR TK,NAMAKKAL
HEXAWARE(6L),VIRTUSA(5.5L),
CAPGEMINI(4.25L)
JAHAJJAINAN V - IT 
GANGAVALLI,SALEM
VIRTUSA(5.5L), LTIMINDTREE(4L)
TCS(3.36L)
ASLIN B J - IT
PACODE, KANYAKUMARI 
DXC(4.2L),COGNIZANT(4L),
LTIMINDTREE(4L)
KISHORE KUMAR S - IT
TIRUCHENGODE,NAMAKKAL
RENTLY(8L), HEXAWARE(6L), 
COGNIZANT(4L)
BALAJI N - IT
KALLAKURICHI
COGNIZANT(4L),LTIMINDTREE(4L),
 NTT DATA(3.5L)
 
AUSTIN JEREMIA A - IT
VIRUDHUNAGAR,SIVAKASI
VIRTUSA(5.5L), CAPGEMINI(4.25L),
LTIMINDTREE(4L)
ABISHEK S - IT
SANKARI - SALEM
ODESSA(7.5L), HEXAWARE(4L), 
COGNIZANT(4L)
SIVAKUMAR S - IT
THOTTIYAPALAYAM,TIRUPPUR
LTIMINDTREE(4L),COGNIZANT(4L), 
ACCENTURE(4L) 
HARSHINI K
KARUR
DELOITTE(4.5L),CAPGMINI(4.25L),
FACE PREP(3L)
NITHISH P - IT
KARIMANGALAM,DHARMAPURI
ODESSA(7.5L),DXC(4.2L),
HEXAWARE(4L) 
MOHANRAJ S P - IT
PUGALUR,KARUR
VIRTUSA(5.5L),HEXAWARE(4L), 
COGNIZANT(4L)
AAYESHA Y – IT
POLLACHI,COIMBATORE
ACCENTURE(4.58L),
HEXAWARE(4L),LTIMINDTREE(4L)
PAVITHRA S - IT
KARUR
CAPGEMINI(4.25L), 
NTT DATA(3.5L),TCS(3.36L)
SRIKAMATCHI M – IT
PODANUR,COIMBATORE
COGNIZANT(4L),HEXAWARE(4L),
TCS(3.36L)
STUDENTS PLACED IN 3 COMPANIES 
MUTHUKRISHNAN C-IT
OTTANCHATHIRAM,DINDIGUL
TCS(7L),DXC(4.2L)
RAGUL M - IT
MOHANUR,NAMAKKAL
VIRTUSA(5.5L),TCS(3.36L)
MOHAMADHU ISMAIL N - IT
KARUR
MPHASIS(4L),FACE PROPAL(3L)
GOKUL K - IT
NAMAKKAL
VIRTUSA(5.5L), ACCENTURE(4.5L)
KEERTHI RAJAN A - IT
CUDDALORE
VIRTUSA(5.5L),COGNIZANT(4L)
STUDENTS PLACED IN 2 COMPANIES 
SOWMIYA D - IT
KARUR
FACEP REP(3L)
ROSHINI C - IT
VEDASANTHUR TK,DINDIGUL
ACCENTURE(4.5L)
PARAMESWARAN S - IT
MULANUR,DHARAPURAM
VIRTUSA(5.5L)
TAMILSELVAN S - IT
KANGEYAM,TIRUPPUR
ACCENTURE(4.5L)
MANIKANDAN S - IT
PERUMANALLUR,TIRUPPUR
VIRTUSA(5.5L)
ALBERT RAJA A - IT
MUTHALAGUPATTI,DINDIGUL
FACE PREP(3L)
MOHANASUNDHARAM A - IT
NAMAKKAL
TCS(3.36L)
AATHISHWARAN L - IT
THANTHONIMALAI - KARUR
ELEVIANT(4.5L)
ARUN AYAPPAN S - IT
ODDANCHATRAM,PALANI
ACCENTURE(4.5L)
KAVIN R - IT
NAMAKKAL 
ZOHO(8.4L)
ARCHANA S - IT
SENGAL,KARUR
ACCENTURE(4.5L), 
ELEVIANT(4.5L)
HARINI M - IT
KARUR
ACCENTURE(4.5L), 
TCS(3.36L)
PRAKASH S - IT
KARAIKUDI - SIVAGANGAI
COGNIZANT(4L),TCS(3.36L)
DHANISHSRIRAM K - IT 
ERODE
DXC(4.2L),TCS(3.36L)
UMAMAHESWARI D - IT
DHARAPURAM
ODESSA(7.5L),DXC(4.20L)
VASANTH KUMAR N - IT
PERIYAKULAM,THENI
VIRTUSA(5.5L),TCS(3.36L)
MEENASRI N – IT
PALANI, DINDIGUL
NTT DATA (4L),COGNIZANT(4L)
HAREESH N - IT
CHETTIPALAYAM,COIMBATORE
DXC(4.2L),LTIMINDTREE(4L)
DEEPAK A - IT
PALLADAM,TIRUPPUR
VIRTUSA(5.5L),ACCENTURE(4.5L)
BOWMITHA C – IT
VAYALUR, DINDIGUL
CODING MART(7L),VIRTUSA(5.5L)
BALASURYA S - IT
KARUR
VIRTUSA(5.5L),LTIMINDTREE(4L)
PRASATH G - IT
KANGEYAM
VIRTUSA(5.5L),LTIMINDTREE(4L)
ALAGESH P - IT
KEELAKKURICHI, THANJAVUR
NTT DATA(4L),COGNIZANT(4L)
AKALYA DEVI B - IT
AALIYAR,POLLACHI
CAPGEMINI(4.25L),HEXAWARE(4L)
SUBIKA M - IT
NAMAKKAL
ACCENTURE(4.5L), DXC(4.2L)
DHARANI A - IT
SIVAGIRI-ERODE
VIRTUSA(5.5L),TCS(3.36L)
PRAVEEN S - IT
NAMAKKAL
ELEVIANT(4.5L),TCS(3.36L) 
SNEHA S - IT
NOYAL,KARUR
DXC(4.2L),LTIMINDTREE(4L)
SANGAVI P - IT
ERODE
TCS(7L),NTT DATA(3.5L)
SHANMUGAPRIYA V - IT
KARUR
DXC(4.2L),TCS(3.36L)
ABARNA S - IT
SIVAGIRI, ERODE
COGNIZANT(4L),TCS(3.36L)
DINESK K - IT
TRICHY
DXC(4.2L),TCS(3.36L)
PRAVEEN K - IT
KARUR
CAPGEMINI(4.25L),FACE PREP(3L)
KARPAGAM N - IT 
KINATHUKADAVU,COIMBATORE
DELOITTE(4.5L),DXC(4.2L)
STUDENTS PLACED IN 1 COMPANY
NANESWARAN M - IT
PALACODE TK,DHARMAPURI
VIRTUSA(5.5L)
MANIKANDAN A - IT
VATTAVADA,IDUKKI
NTT DATA(3.5L)
VIGNESHG - IT
MAYILADUTHURAI,NAGAPATTINAM
ACCENTURE(4.5L)
SANKAR S - IT
PERAMBALUR
VIRTUSA(5.5L)
SANJAY A - IT
KARUR
COGNIZANT(4L)
AGALYA S - IT
RAYANUR,KARUR
TCS(3.36L)
BRINDHA V - IT
TIRUPPUR
VIRTUSA(5.5L)
SABAREESWARA K S - IT
KARUR
COGNIZANT(4L)
PRIYADHARSHINI S - IT
SANGAGIRI TK,SALEM
ACCENTURE(4.5L)
SANDHIYA S - IT
GANAPATHYPALAYAM,TIRUPPUR
DXC(4.2L)
PEMALATHA S - IT
KONERIPATTY,DHARAPURAM
TCS(3.36 L)
VISALI M - IT
KARUR
VIRTUSA(5.5L)
SNEHA L - IT
PADALUR,PERAMBALUR
DXC(4.2L)
SHANMATHI B - IT
KARUR
NTT DATA(3.5L)
SARAN M - IT
VALAPADY-SALEM
FACE PREP(3L)
RITHINA B - IT
MOHANUR TK,NAMAKKAL
ACCENTURE(4.5L)
RAJAMAHESWARI V - IT
KARUR
INFOSYS(4L)
ARUNA D - IT
NAMAKKAL
ODESSA(7.5L)
DHARSHINI R - IT
KARUR
VIRTUSA(5.5L)
VINODHINI G - IT
ARASAMPALAYAM PIRIVU,COIMBATORE
DXC (4.2L)
DHANUSKODI R A - IT
VELLUR,TIRUPPUR
DXC(4L)
BOOMINATHAN R - IT
PALLADAM,TIRUPPUR
DXC(4L)
ARUN KUMAR V - IT
MALUMICHAMPATTI,COIMBATORE
VIRTUSA(5.5L)
ABIRAMI M - IT
EACHANARI,COIMBATORE
DIVUM CORPORATE (3.6LTO 4.5L)
ABINAYA R - IT
KOTTUR,POLLACHI
VIRTUSA(5.5L)
NITHISH S - IT
NAGAL NAGAR,DINDIGUL
VIRTUSA(5.5L)
THOMSON D - IT
INDRA NAGAR,THOOTHUKUDI
DXC(4.2L)
VISALATCHI T - IT
POLLACHI,COIMBATORE
DXC(4.2L)
KRITIKA  - IT
VENNAIMALAI,KARUR
DXC(4.2L)
HARIHARAN R - IT
PATTUKKOTTAI,THANJAVUR
ACCENTURE(4.5L)
DEVENDRAKUMAR S - IT
TIRUPPUR
TVS ELECTRONICS (3.5L)
NAGENTHIRAN M  -  ECE
DINDIGUL
HEXAWARE(6L),ZOHO(5.6L),VIRTUSA(5.5L), 
CAPGEMINI(4.25L),COGNIZANT(4L)
KAVINKUMAR R - ECE
VELLAKOVIL
ACCENTURE(4.5L),DXC(4.2L), 
HEXAWARE(4L),LTIMINDTREE(4L),TCS(3.36L)
THANGAKAVIN D - ECE
MULANUR
ACCENTURE(4.5L),DXC(4.2L),
COGNIZANT(4L),LTIMINDTREE(4L),TCS(3.36L)
SAMYUKTHA E - ECE
KARUR
VIRTUSA(5.5L), DXC(4.2L), 
CAPGEMINI(4.25L),HEXAWARE(4L), 
LTIMINDTREE(4L) 
SALINI P - ECE
ODDANCHATHIRAM
TCS (7L),HEXAWARE(6L),VIRTUSA(5.5L), 
CAPGEMINI(4.25L),COGNIZANT(4L)
KAILASH S - ECE
UDUMALPET,TIRUPPUR
HEXAWARE(6L),VIRTUSA(5.5L),ACCENTURE(4.5L),
TESSOLVE(4.13L),LTIMINDTREE(4L),
COGNIZANT(4L)
SOWMYA L R - ECE
VELLAKOVIL
VIRTUSA(5.5L),CAPGEMINI(4.25L),HCL(4.25L),
HEXAWARE(4L),LTIMINDTREE(4L),TCS(3.36L)
GAYATHRI L - ECE
NAMAKKAL
VIRTUSA(5.5L),CAPGEMINI(4.25L),HCL(4.25L),
DXC(4.2L),HEXAWARE(4L),LTIMINDTREE(4L) 
NIVETHITHAA S - ECE
NAMAKKAL
CAPGEMINI(7.5L),TCS(7L) , 
ACCENTURE(4.5L),MPHASIS(4L), 
LTIMINDTREE(4L),NTTDATA(3.5L)
SIVAGANESH B - ECE
VELLAKOVIL
RENTLY(6L),VIRTUSA(5.5L),CAPGEMINI(4.25L),
DXC(4.2L),LTIMINDTREE(4L),TCS(3.36L)
ELECTRoniCs AND communication ENGINEERING
TCS(7L),ACCENTURE(6.58L),KAAR(6.5L),VIRTUSA NEURAL HACK(6.5L),
LUMEN(6L),VIRTUSA (5.5L),DELOITTE(4.5L),LTIMINDTREE(4L),
HEXAWARE(4L),COGNIZANT(4L)
STUDENT PLACED IN 10 COMPANIES 
ARUN N - ECE
KARUR
KAAR(6.5L),HEXAWARE(6L),ZOHO(5.6L),
VIRTUSA(5.5L),CAPGEMINI(4.25L),
DXC(4.2L),LTIMINDTREE(4L),TCS(3.36L)
STUDENT PLACED IN 8 COMPANIES 
SRI KRISHNAN P - ECE
UDUMALPET, TIRUPPUR
KAAR (6.5L),CAPGEMINI(5.75L),ACCENTURE(4.5L),
HEXAWARE(4L),COGNIZANT(4L),
LTIMINDTREE(4L),TECH M(3.25L)
YOHASUPRIYA K  - ECE
VELLAKOVIL
TCS (7L),VIRTUSA(5.5L),ACCENTURE(4.5L),
CAPGEMINI(4.25L),DXC(4.2L), 
LTIMINDTREE(4L),TECH M(3.25L)
STUDENTS PLACED IN 7 COMPANIES 
STUDENTS PLACED IN 6 COMPANIES 
STUDENTS PLACED IN 5 COMPANIES 
KALPANA D - ECE
TIRUPPUR
ACCENTURE(4.5L) CAPGEMINI(4.25L),
DXC(4.2L),COGNIZANT(4L),LTIMINDTREE(4L)
MAHALAKSHMI R - ECE
MADURAI
VIRTUSA(5.5L) ,CAPGEMINI(4.25L), DXC(4.2L), 
LTIMINDTREE(4L),TCS(3.36L)
GOWTHAM P - ECE
MADURAI
DXC(4.2L),HEXAWARE(4L), 
LTIMINDTREE(4L),TECH M(3.5L),TCS(3.36L)
ELAKKIYA N  -  ECE
ODDANCHATRAM
ACCENTURE(4.5L),DXC(4.2L),
 MPHASIS(4L),LTIMINDTREE(4L),TCS(3.36L)
BHUVANESHWARI P - ECE
PALANI
VIRTUSA(7.5L),HACKWITH(6.25L) , 
ZOHO(5.6L),HEXAWARE(4L),LTIMINDTREE(4L)
SHIVA KEERTHANA R - ECE
UDUMALPET,TIRUPPUR
SANTHOSHKUMAR M - ECE
MAYILADUTHURAI
CAPGEMINI(4.25L),DXC(4.2L), 
LTIMINDTREE(4L), TCS(3.36L)
GAYATHRI G - ECE
SUNDARAPURAM,COIMBATORE
SONY(6.25L),VIRTUSA(5.5L),
CAPGEMINI(4.25L),TCS(3.36L)
PAVITHRA M - ECE
NAMAKKAL
LTIMINDTREE(6L),ACCENTURE(4.5L),
ELEVANT(4.5L),TCS (3.36L) 
PERIYANAYAKI T - ECE
TIRUPPUR
LTIMINDTREE(6L),VIRTUSA(5.5L), 
CAPGEMINI(4.25L),HEXAWARE(4L)
KUMARESAN T - ECE
PALANI
CAPGEMINI(7.5L),VIRTUSA(5.5L),
DELOITTE(4.5L),COGNIZANT(4L)
ABIRAMI R – ECE
ODDANCHATRAM
DXC(4.2L),COGNIZANT(4L), 
LTIMINDTREE(4L),TCS(3.36L)
ASIKAHAMED Y - ECE
TANJORE
LTIMINDTREE(6.5L),DXC(4.2L),
TCS(3.36L),TECH M(3.25L)
DEEPAK P - ECE
NAMAKKAL
VIRTUSA(5.5L),ACCENTURE(4.5L), 
DXC(4.2L),COGNIZANT(4L) 
DHARSHINI S G - ECE 
NAMAKKAL
CAPGEMINI(4.25L),COGNIZANT(4L),
 LTIMINDTREE(4L),TCS(3.86L)
AFIYA FARGATH A - ECE
KARUR 
VIRTUSA (5.5L),DELOITTE(4.5L),
CAPGEMINI(4.25L),TCS(3.36L) 
HARIKRISHNAN N - ECE
NAMAKKAL
 VIRTUSA(5.5L),PROPEL (4.5L), 
CAPGEMINI(4.25L),HEXAWARE(4L)
JENIFER G - ECE
VELLAKOVIL
VIRTUSA(5.5L),ACCENTURE(4.5L),
HEXAWARE(4L),TCS(3.86L)   
KAMALRAJ S - ECE
TIRUPPUR
VIRTUSA(5.5L), CAPGEMINI(4.25L), 
HEXAWARE(4L),TCS(3.36L)
KARTHIKA P - ECE
TRICHY
VIRTUSA(5.5L),ACCENTURE(4.5L),
DXC(4.2L),LTIMINDTREE(4L)
NIRANJAN R - ECE
TIRUPPUR
HEXAWARE(6L),VIRTUSA(5.5L),
 LTIMINDTREE(4L),TCS(3.36L)
VASUNTHARA D - ECE
GOBICHETTIPALAYAM,ERODE
KAAR (6.5L),DXC(4L), 
COGNIZANT(4L),LTIMINDTREE(4L),TCS(3.36L)
PARTHIPAN G - ECE
DINDIGUL
VIRTUSA(5.5L),ACCENTURE(4.5L),DXC(4.2L),
HEXAWARE(4 L)TCS(3.36L)
JEGAPRIYA P - ECE
KARUR
INCEDO(5.5L),ACCENTURE(4.5L),
LTIMINDTREE(4L),TCS(3.86L),TECH M(3.25L)
STUDENTS PLACED IN 4 COMPANIES 
SABARI K - ECE
NAMAKKAL
CAPGEMINI(7.5L),LTIMINDTREE(4L),
NTT DATA (3.5L),TCS (3.36L)
SABITHA K - ECE
KARUR
ACCENTURE(4.5L),DXC(4.2L), 
LTIMINDTREE(4L),TCS(3.86L) 
VAIRABHARANI  S - ECE
MANNARGUDI 
HEXAWARE(4L),COGNIZANT(4L), 
TCS(3.36L),TECHM(3.2L)
SANTHOSH KUMAR  R - ECE
PALANI
HEXAWARE(6L),PROPEL(4.5L), 
ACCENTURE(4.5L),LTIMINDTREE(4L)
SARANKUMAR R – ECE
TIRUPPUR
CAPGEMINI(7.5L),VIRTUSA(5.5L) , 
MAVENTIC(5 L),LTIMINDTREE(4L)
SELVAKAVIN S - ECE
MULANUR 
SONY(6.25L),VIRTUSA(5.5L),
COGNIZANT(4L),TCS(3.36L)
SRIKANTH K - ECE
NAMAKKAL
KAAR(6.5L),VIRTUSA(5.5L), 
DXC(4.2L),TCS(3.36L)
SRINITHI A - ECE
ERODE
KAAR(6.5L),STEAM-A(5L)
HEXAWARE(4L),LTIMINDTREE(4L)
SUNILKUMAR A - ECE
VILUPURAM
VIRTUSA(5.5L),CAPGEMINI(4.25L),
HEXAWARE(4L),COGNIZANT(4L), 
THARUNAPRIYA S  - ECE 
NAMAKKAL
VIRTUSA(5.5L),CAPGEMINI(4.25L),
HEXAWARE(4L),LTIMINDTREE(4L) 
SARAVANAN R - ECE
VELLAKOVIL
ZOHO(5.6L),DXC(4.2L), 
TCS(3.36L)
SARANYA K - ECE 
KARUR
ACCENTURE(4.5L), 
LTIMINDTREE(4L),TCS(3.86L)
SARAN M - ECE
TIRUPPUR
VIRTUSA(5.5L),DXC(4.2L), 
TCS(3.36L)
SACHIN RM - ECE
KARUR
HEXAWARE(4L), 
LTIMINDTREE(4L),TCS(3.36L)
RAJESWARI K - ECE
PALANI
 ACCENTURE(4.5L) , 
LTIMINDTREE(4L),TCS(3.86L)
SUGAN S - ECE
NAMAKKAL
VIRTUSA(5.5L),ACCENTURE(4.5L)
TCS (3.36L)
RAJASURYA S-ECE
DHARMAPURI
CAPGEMINI(4.25L),DXC(4.2L),
TCS(3.36L)
MANOJKUMAR A R-ECE
TIRUPPUR
DXC(4.2L),COGNIZANT(4L), 
TCS(3.36L)
MANGAIYARKARASI B-ECE
TIRUPPUR
 ACCENTURE(4.5L)DXC(4.2L), 
TCS(3.36L)
SATHYARUBA.A-ECE
VEDASANDUR
DELOITTE(4.5L),DXC(4.2L),
TCS(3.36L)
PRITHIBA G - ECE 
KARUR
 LTIMINDTREE(4L),TCS(3.36L),
ELEVIANT(3L)
PREETHI T - ECE 
TANJORE
RENTLY(6L),VIRTUSA(5.5L), 
DXC(4.2L) 
PRAKASHAM P - ECE
TIRUPPUR
VIRTUSA(5.5L),HEXAWARE(4L),
TCS(3.36L)
PRAGADEESHWARAN K - ECE
MADURAI
VIRTUSA(5.5L),ACCENTURE(4.5L) 
TCS(3.86L) 
POORANI D - ECE
KARUR
VIRTUSA(5.5L),DXC(4.2L), 
TCS(3.36L)
NAVANEETHAN R - ECE
KARUR
VIRTUSA(5.5L),MPHASIS(4L)
TCS(3.36L)
NADISH KB - ECE
NAMAKKAL
ACCENTURE(4.5L),
DXC(4.2L),TCS(3.86L) 
MOHANA M - ECE
PALANI
 ACCENTURE (4.5L),
 LTIMINDTREE(4L),TCS(3.86L)
MIDHUNA C-ECE
TIRUPPUR
VIRTUSA(5.5L),DXC(4.2L),
TCS(3.86L)
MUKESH KANNA M-ECE
NAMAKKAL
RENTLY(6L),VIRTUSA(5.5L), 
DXC(4.2L)
KESHAVARTHINI A - ECE
NAMAKKAL
 ACCENTURE(4.5L), 
LTIMINDTREE(4L),TCS(3.86L)
KEERTHANA T - ECE
ARAVAKURUCHI 
VIRTUSA(5.5L),
 ACCENTURE(4.5L),HEXAWARE(4L)
KAVIYA S - ECE
NAMAKKAL
 TCS (7L), VIRTUSA(5.5L), 
LTIMINDTREE(4L)
KATHIR M - ECE
MUTHUR
VIRTUSA(5.5L), 
ACCENTURE(4.5L),DXC(4.2L)
KARTHIKEYAN V S - ECE
TIRUPPUR
TCS(7L),ACCENTURE(4.5L), 
HEXAWARE(4L)
KALAISELVI P - ECE
PALANI
VIRTUSA (5.5L),  
ACCENTURE(4.5L),TCS(3.36L)
JEEVITHA M - ECE
PALANI
ACCENTURE(4.5L), 
LTIMINDTREE(4L),TCS(3.86L) 
GUNAVATHI S - ECE
MANNARKUDI
HEXAWARE(6L),
DXC(4.2L), TCS(3.86L) 
GOKULRAJ P - ECE
NAMAKKAL
TCS(7L), VIRTUSA(5.5L),  
ACCENTURE(4.5L)
DEVAPRASATH S - ECE
TIRUPPUR
 VIRTUSA(5.5L), 
DXC(4.2L),HEXAWARE(4L)
DEEPANA J - ECE
DINDIGUL
VIRTUSA(5.5L),
 ACCENTURE(4.5L),TCS(3.86L) 
BHARATHKUMAR D - ECE
VELLAKOVIL
ACCENTURE(4.5L),DXC(4.2L), 
TCS(3.36L) 
BALAJI B - ECE
KARUR
VIRTUSA(5.5L),HEXAWARE(4L), 
TCS(3.36L)
ABULHASAN A - ECE
KARUR
DXC(4.2L),LTIMINDTREE(4L),
TCS(3.36L) 
ABINAYA  R - ECE
DHARAPURAM
VIRTUSA(5.5L),DXC(4.2L), 
TCS(3.36L)
STUDENTS PLACED IN 3 COMPANIES 
MATHIVANAN B - ECE
VELLAKOVIL
 LTIMINDTREE(4L), NTT DATA(3.5L)
RANJITH K - ECE
PALANI
 ACCENTURE(4.5L),TCS(3.86L)
SANGEETHA R - ECE
ODDANCHATRAM
VIRTUSA(5.5L),LTIMINDTREE(4L)
SIVARANJANI A - ECE
DHARAPURAM
ACCENTURE(4.5L),MPHASIS(4L)
SRI PRADEEPA K - ECE
SIVAGIRI
VIRTUSA(5.5L),TCS(3.86L)
THULASINATHAN S - ECE
KARUR
NTT DATA(3.5L),TCS(3.86L)
KARTHIKKUMAR R - ECE
MUTHUR
DXC(4.2L),TCS(3.36L)
PRAKASH N - ECE
PALANI 
DXC(4.2L),COGNIZANT(4L)
DHANDAPANI A - ECE
KARUR
VIRTUSA(5.5L),HEXAWARE(4L)
DHARANI S - ECE
VELLAKOVIL
VIRTUSA(5.5L),ACCENTURE(4.5L)
POOVARASAN M - ECE
TRICHY
VIRTUSA(5.5L),TCS(3.36L)
ISHWARYA V - ECE
KODAIKANAL
NTT DATA(3.5L),TCS(3.86L) 
JANANI S - ECE
KULITHALAI
DXC(4.2L),TCS(3.86L)
JAYASURUTHI A - ECE
KARUR
ACCENTURE (4.5L),TCS (3.86L)
JEEVITHA K - ECE
MULANUR
VIRTUSA (5.5L),TCS(3.36L)
JOTHIKA M G - ECE
MADURAI
VIRTUSA(5.5L),TCS(3.36L)
ABIRAMI T - ECE
VELLAKOVIL
VIRTUSA(5.5L),TCS(3.36L)
BRUNDHA C - ECE
KARUR
ACCENTURE(4.5L),TCS(3.86L)
ANUSUYA R - ECE
VELAYUTHAMPALAYAM
VIRTUSA(5.5L),TCS(3.86L)
ATHENA S - ECE
NAMAKKAL
VIRTUSA(5.5L) ACCENTURE(4.5L)
CHANDU S - ECE
REDDIPATTY
VIRTUSA(5.5L),DXC(4.2L)
SHERBIA S - ECE
MANNARGUDI
VIRTUSA (5.5L),
ACCENTURE(4.5L),TCS(3.36L) 
SELVARAJ G - ECE
ODDANCHATHIRAM
ACCENTURE(4.5L)
DXC(4.2L),TCS(3.36L)
SATHISHKUMAR R - ECE
KARUR
HEXAWARE(6L),VIRTUSA(5.5L), 
TCS(3.36L)
SASIKUMAR S - ECE
KARUR
TECH M(5.5L),
DXC(4.2L),TCS(3.36L) 
SARUMATHI J - ECE
TRICHY
CAPGEMINI(4.25L),
DXC(4.2L),TCS(3.36L)
ABIMANYU G - ECE
DHARAPURAM,TIRUPPUR
VIRTUSA(5.5L),
COGNIZANT(4L),TCS(3.36L)
SWATHI P - ECE
KANIYUR,MADATHUKULAM,TIRUPPUR
CAPGEMINI(4.25L),
DXC(4.2L),COGNIZANT(4L)
VISHNUPRIYA A - ECE
PETHAPPAMPATTI,TIRUPPUR
CAPGEMINI(4.25L),
HEXAWARE(4L),COGNIZANT(4L)
VISHNUPRIYA S - ECE
THADAGAM ROAD,COIMBATORE
ACCENTURE(6.58L),DXC(4.2L),
TCS(3.36L)
VINOTHA M - ECE 
ODDANCHATHIRAM
ACCENTURE(4.5L), LTIMINDTREE(4L), 
TECH M(3.25L)
KAMALESHWARAN V - ECE
VELLALORE,COIMBATORE
VIRTUSA(5.5L),HEXAWARE(4L),
LTIMINDTREE(4L)
DINESH KUMAR M - ECE
MADUKKARAI,COIMBATORE
NTT DATA(4L),COGNIZANT(4L),
TECH M(3.25L)
ABINANDHINI R - ECE
KOLARPATTI, COIMBATORE
VIRTUSA(5.5L),CAPGEMINI(4.25L),
COGNIZANT(4L)
MEENAKSHI A - ECE
MADURAI SOUTH, MADURAI
VIRTUSA(5.5L), 
TESSOLVE(4.13L),COGNIZANT(4L)
VIGNESH R - ECE
KARUR
VIRTUSA(5.5L),DXC(4.2L), 
TCS(3.36L)
JAGADEESH E - ECE
PALANI
CAPGEMINI(5.75L),
VIRTUSA(5.5L),TCS(3.36L)
STUDENTS PLACED IN 2 COMPANIES 
INDHU PRIYA N - ECE
PALANI
LTIMINDTREE(4L), NTT DATA(3.5L)
PARTHIBAN M - ECE
NAMAKKAL
TCS(7L),DXC(4.2L) 
YUVASHREE K   - ECE
 MUTHUR 
VIRTUSA(5.5L),CAPGEMINI(4.25L)
SUBASRI M - ECE
POLLACHI,COIMBATORE
HEXAWARE(4L),TCS(3.36L)
VANJIKODI K - ECE
PUSHPATHUR,DINDIGUL
SONY(6.25L),VIRTUSA(5.5L)
NAVEENKUMAR V - ECE
NAMAKKAL
VIRTUSA(5.5L),HEXAWARE(4L)
YOGESH KUMAR S-ECE
 MUTHUR
VIRTUSA(5.5L),TCS(3.36L)
VASUKI G - ECE
DHARAPURAM
VIRTUSA(5.5L),TCS(3.36L)
VENGATESHWARAN H - ECE
RASIPURAM
DXC(4.2L),COGNIZANT(4L)
VIBIN M K - ECE
NAMAKKAL
CAPGEMINI(5.75L),ACCENTURE(4.5L)
YOGAPRIYA K - ECE
PUDUKKOTTAI
DXC(4.2L),TCS(3.36L)
KAVINRAJA M-ECE
MUTHUR
HEXAWARE(6L),PROPAL(4.5L)
KAVIYA R - ECE
PALANI
TCS (3.86L),TECH M(3.25L)
KAVIYARASAN P - ECE
MULANUR
LTIMINDTREE(5L),TCS(3.36L)
KAVIYARASI G - ECE
MULANUR
CAPGEMINI(4.25L),TCS(3.36L)
KEERTHANA S - ECE
TIRUPPUR
 ACCENTURE (4.5L),HEXAWARE(4L)
CHANDRU M - ECE
SEERANAICKEN PALAYAM, COIMBATORE
VIRTUSA(5.5L),TCS(3.36L)
KOWSHIK R - ECE
PALLADAM, TIRUPPUR
VIRTUSA(5.5L),TCS(3.36L)
MOHAMMAD HARIS C M - ECE
BITHERKAD,THE NILGIRIS
VIRTUSA(5.5L),LTIMINDTREE(4L)
PAVITHRA S - ECE
SIVANANDHAPURAM,COIMBATORE
DXC(4.2L),TECH M(3.25L)
SARATH S - ECE
KUNIYAMUTHUR, COIMBATORE
HEXAWARE(4L),LTIMINDTREE(4L)
PRIYANKA M - ECE
SIVAGIRI
VIRTUSA(5.5L),DXC(4.2L)
KANIMOZHI T - ECE
THOOTHUKUDI
ACCENTURE(4.5L),LTIMINDTREE(4L)
SURYA S - ECE
KUMBAKONAM
 ACCENTURE(4.5L),TCS(3.86L)
TEENAL R - ECE
KARUR
ODESSA(7.5L),COGNIZANT(4L)
PREETHI S - ECE
KARUR
ACCENTURE(4.5L),DXC(4.2L) 
DHAYANANTH E P - ECE
PUNNAMCHATRAM
DXC(4.2L),TCS (3.86L)
DHIVAKAR B - ECE
TIRUPPUR
COGNIZENT(4L),LTIMINDTREE(4L)
GOWSALYAA R - ECE
LAKHMIPURAM
 DXC(4.2L),TCS(3.86L)
HARINI S - ECE
KARUR
DXC(4.2L),TCS (3.86L)
HEMARAJEE R - ECE
THENI
 VIRTUSA(5.5L),ACCENTURE(4.5L)
SUKESH S - ECE
  KARUR
VIRTUSA(5.5L)
VIJAYALAKSHMI G - ECE
 KARUR
TCS(3.86L)
THIYAGARAJAN R-ECE
 KARUR
VIRTUSA(5.5L)
SUBASH P - ECE
NAMAKKAL
ODESSA(7.5L)
SINEGA P - ECE
 KARUR
DXC(4.2L)
SHOORIYAPRABHA S - ECE
TRICHY
TIGER ANALYTIS(7.5L)
SANTHOSH D - ECE
VELLAKOVIL 
DXC(4.2L)
RAGUL S - ECE
 TIRUPPUR
VIRTUSA(5.5L)
PRITHIKA N - ECE
TIRUPPUR
ACCENTURE(4.5L)
NAVEENKUMAR S - ECE
TIRUPPUR
DXC(4.2L)
LEEBIKA S - ECE
KARUR
TCS(3.36L)
KOPPERUMDEVI S - ECE
KARUR
HEXAWARE(6L)
KEERTHINI R - ECE
   TIRUPPUR
NTT DATA(3.5L)
KAVINKUMAR R - ECE
 KARUR
TCS(3.36L)
KALAISELVI S - ECE
TIRUPPUR
AVALON(3L)
GOKULRAM D - ECE
  TIRUPPUR
NTT DATA(3.5L)
VISHNUPRIYA P - ECE
KARUR
VIRTUSA(5.5L)
GOPIKA A - ECE
NAMAKKAL
ACCENTURE(4.5L)
VINOTHA B - ECE
DINDIGUL
TCS(3.36L)
JEGAN M - ECE
PALANI
DXC(4.2L)
ELAKKIYA P - ECE
   KARUR
LTIMINDTREE(4L)
DHANUSUYA M - ECE
   TIRUPPUR
LTIMINDTREE(4L)
CHANDRU N - ECE
TRICHY
TCS(3.36L)
CHANDRA L - ECE
  NAMAKKAL
   DXC(4.2L)
AJITH KUMAR B - ECE
THENI
TCS(3.36L)
DEENADHAYALAN K - ECE
VELLALORE, COIMBATORE
NTT DATA (4L)
KESAVARAJ D - ECE
VENKITTAPURAM,TIRUPPUR
ACCENTURE(4.5L)
SUGENDRAN A - ECE
VADAKKIPALAYAM,POLLACHI
VIRTUSA(5.5L)
VIKRAM P - ECE
ATTUR ,SALEM
DIVUM(3.6LTO 4.5L)
SIVA R - ECE 
ARIYALUR
VIRTUSA(5.5L)
STUDENTS PLACED IN 1 COMPANY 
PADMESH P – EEE
SIVAGIRI
VIRTUSA (7.5L), ZOHO(5.6L), 
COGNIZANT(4L),TCS (3.36L)
PRAVEENA S- EEE
GUJILIYAMPARAI
TCS(7L),LTIMINDTREE(6.5L), 
HEXAWARE(6L),COGNIZANT(4L)
VENGADESH A -EEE
DHARAPURAM
CAPGEMINI(7.5L),ODESSA(6.5L), 
DXC(4.2L),COGNIZANT(4L)
PARAMESHKUMAR S -EEE
KANGEYAM
RENTLY(6L),VIRTUSA(5.5L), 
DELOITTE(4.5L),DXC(4.2L)
MAHESWARAN  S – EEE
PALLADAM, TIRUPPUR
ACCENTURE(4.5L),DXC(4L),
COGNIZANT(4L),TCS(3.36L)
DEEBAHARSHINI S - EEE
KARUR
VIRTUSA (7.5L),CAPGEMINI (5.75L), 
COGNIZANT(4L),HEXAWARE (4L),TCS (3.36L)
RAVEENA D -EEE
ERODE
KAAR (6.5L),DELOITTE(4.5L), 
ACCENTURE(4.5L),INFOSYS(3.5L),TCS(3.36L)
VAISHNAVI S-ECE
  ERODE
LTIMINDTREE(4L)
SINDHUJA J-ECE
KARUR
DXC(4.2L)
MANISIVARAJAPANDI A - ECE
VIRUDHUNAGAR
DXC(4.2L)
JEYASHREE N - ECE
ARUPPUKOTTAI
FACE PREP(3L)
GOKULPRASANTH J - ECE
KARUR
DXC(4.2L)
SANTHIYA R - ECE
AVARAMPALAYAM,COIMBATORE
DXC(4.2L)
SATHIYA PRIYA B - ECE
DEVANAMPALAYAM,POLLACHI
VIRTUSA(5.5L)
THIRUMOORTHI P - ECE
MUTHUR
ACCENTURE(4.5L)
THIRU GHANAMUTHUSELVAN S - ECE
KARUR
ZOHO(5.6L)
VASUKI P - ECE
HOSUR 
AVALON(3L)
SIVAKAVIYA D - ECE
KARUR 
TCS(3.36L)
LINGESH PRABHU R - ECE
POLLACHI,COIMBATORE
VIRTUSA(5.5L)
KISHORE KUMAR C - ECE
KOTTUR, COIMBATORE
VIRTUSA(5.5L)
electrical and electronics engineering
STUDENTS PLACED IN 5 COMPANIES 
STUDENTS PLACED IN 4 COMPANIES 
RUTHRA R - EEE
NAMAKKAL
PROPEL(6.5L),VIRTUSA(5.5L), 
DELOITTE(4.5L),COGNIZANT(4L)
PAVITHRA SREE M – EEE
PANDIYAN NAGAR,TIRUPPUR
HEXAWARE(6L),LTIMINDTREE(4L),
COGNIZANT(4L),BREAK INDIA(3L)
SANTHIYA K – EEE
SANKARANDAMPALAYAM
DXC(4.2L),HEXAWARE(4L),
 LTIMINDTREE(4L),TCS(3.36L)
JEEVITHA R - EEE
PUNNAM CHATHIRAM
VIRTUSA (5.5L),DELOITTE(4.5L),
LTIMINDTREE(4L) 
NANDHINI K - EEE
VELLAKOVIL
VIRTUSA (5.5L), 
ACCENTURE(4.5L),TCS (3.36L)
MAHAAMITHRAA S B - EEE
MANMANGALAM
ACARDIA(5.5L),DXC(4.2L), 
COGNIZANT(4L) 
JAGAN KARTHIK V - EEE
UNJALUR
VIRTUSA(5.5L),CAPGEMINI(4.25L), 
LTIMINDTREE(4L)
GOPINATH U - EEE
KARUR
HEXAWARE(6L), 
ACCENTURE(4.5L),TCS(3.36L)
STUDENTS PLACED IN 3 COMPANIES 
BHARANITHARAN N - EEE
ERODE
ACCENTURE(4.5L) 
DEEPAK V - EEE
KANGEYAM
INFOSYS(4.5L) 
GAYATHRI DEVI B - EEE
ESANATHAM
ODESSA(6.5L)
GOWSALYA S - EEE
SALEM
HEXAWARE(4L) 
LALITHA K - EEE
PUGALUR
DXC(4.2L)
ABDUL KALAM E - EEE
MADURAI
HEXAWARE(6L),DELOITTE(4.5L),
TCS(3.36L)
SUDHARSAN M - EEE
VELLIYANAI
VIRTUSA(5.5L), 
DXC(4.2L),MPHASIS(4L)
HARI PRASATH M - EEE
TIRUCHENGODE
VIRTUSA(5.5L),DELOITTE(4.5L),
LTIMINDTREE(4L)
MANOJ KUMAR R - EEE
SULUR,COIMBATORE
 VIRTUSA(5.5L),HEXAWARE(4L),
TCS(3.36L)
DHARANI M - EEE
PULIYUR
LTIMINDTREE(4L), 
NTT DATA(3.5L),TCS(3.36L)
GUNASHELLAN S - EEE
KOTHAYAM-DINDIGUL
VIRTUSA(5.5L),DELOITTE(4.5L),
TCS(3.36L)
BHARATH M - EEE
TRICHY
DELOITTE(4.5L),AVALON(3L)
DEEPA S - EEE
MULANUR
ACCENTURE(4.5L),TCS(3.36L)
KRISHNARAJ A - EEE
VEERAPANDI PIRIVU,TIRUPPUR
DXC(4.2L),LTIMINDTREE(4L)
VIJAYA KANISHKA P M - EEE
SRIVILLIPUTHUR, VIRUDHUNAGAR
DXC(4.2L),TCS(3.36L)
MARIMUTHU T - EEE
KARUR
VIRTUSA(5.5L),TCS(3.36L)
BAVATHARANI S - EEE
KATTAMPATTI, COIMBATORE
HEXAWARE (4L),TCS(3.36L)
PADMA KUMAR C - EEE
UDUMALPET,TIRUPPUR
HEXAWARE(4L),COGNIZANT(4L)
PRATHEEKSHA P - EEE
UDUMALPET,TIRUPPUR
ACCENTURE(4.5L),
HEXAWARE(4L)
SANTHIYA J - EEE
CHINNALAPATTI,DINDIGUL
CAPGEMINI(5.75L),
DXC(4.2L)
VIGNESHWARAN P - EEE
KANGEYAM
DXC(4.2L),COGNIZANT(4L) 
SURIYAPRASATH R - EEE
CHIDHAMBARAM
VIRTUSA(5.5L),TCS(3.36L) 
SELVAKRISHNAN P - EEE
DHARAPURAM
 VIRTUSA(5.5L),TCS(3.36L)
MANOJ M - EEE
SALEM
 HEXAWARE(6L),COGNIZANT(4L)
KARAN T - EEE
ODDANCHATRAM
VIRTUSA(5.5L),COGNIZANT(4L)
STUDENTS PLACED IN 2 COMPANIES 
STUDENTS PLACED IN 1 COMPANY 
VISHOK K - EEE
VENNAMALAI
DXC(4.2L) 
LALITH KUMAR J - EEE
PARAMATHI VELUR
DXC(4.2L) 
SURESHPAVUN B - EEE
MANUR,TIRUNELVELI
VIRTUSA(5.5L)
SURYA P - EEE
PALLADAM,TIRUPPUR 
TCS(3.36L)
SUJITHA A - EEE
PARAMATHI VELUR
 DXC(4.2L)
NIVETHAN R - EEE
VIRUDHACHALAM
DXC(4.2L)
KANNATHASAN T - EEE
NATHAM,DINDIGUL
TCS(3.36L)
NAVEEN KUMAR N - EEE
UDUMALPET,TIRUPPUR
DXC(4.2)
PRAVEEN PRABHU M - EEE
KUMAR NAGAR,TIRUPPUR
HEXAWARE(4L)
SAMEER A - EEE
THANGAM THEATRE,POLLACHI
DXC(4.2L)
SELVA KUMAR N - EEE
KOTTUR,POLLACHI 
TCS(3.36L)
DHANASEKARAN M - EEE
MOHANUR
INFOSYS(4.5L)
ANBARASAN S - EEE
KARUMARAMPALAYAM,TIRUPPUR
VIRTUSA(5.5L)
ARUN S - EEE
UDUMALPET,TIRUPPUR
DXC(4.2L)
KATHIRVADIVEL C - EEE
MADATHUKULAM,TIRUPPUR
DXC(4.2L)
KOUSHIKAN G R - EEE
UDUMALPET,TIRUPPUR
DXC(4.2L)
NAVEEN R - EEE
SAIBABA KOVIL,COIMBATORE
VIRTUSA(5.5L)
MEERASRI S - EEE
RASIPURAM
VIRTUSA(5.5L) 
SAKTHI SETHU G K - EEE
KARUR
TCS(3.36L) 
SARATH KUMAR K - EEE
PARAMATHI VELUR
VIRTUSA(5.5L) 
SOWMIYA R - EEE
VELLAKOVIL
VIRTUSA (5.5L)
SUDHARSAN G S - EEE
THANJAVUR
VIRTUSA (5.5L) 
biomedical engineering
DEVI SHREE M - BIOMEDICAL
TIRUPPUR
ZIFO RND(4.76L),DXC(4.2L), 
CAPGEMINI(4L),TCS(3.36L)
GOBIKA S - BIOMEDICAL
TIRUPPUR
LTIMINDTREE(5L), ACCENTURE(4.5L),
CAPGEMINI(4L),TCS(3.36L)
DEEPIKA R - BIOMEDICAL
KARUR
ZIFO RND(4.76L),COGNIZANT(4L), 
CAPGEMINI(4L),TCS(3.36L)
STUDENTS PLACED IN 4 COMPANIES 
SNEGA B - BIOMEDICAL
KARUR
ZIFO RND(4.76L),COGNIZANT(4L),
TCS(3.36L)
PRABHAKARAN R - BIOMEDICAL
KARUR
DXC(4.2L),COGNIZANT(4L),
TCS(3.36L)
ANANTHA KUMAR M - BIOMEDICAL
MANAPARAI
ZIFO RND(4.76L),
DXC(4.2L),COGNIZANT(4L)
PRABHA M - BIOMEDICAL
BATLAGUNDU
DXC (4.2L),COGNIZANT(4L),
TCS(3.36L)
SURYA M - BIOMEDICAL
NAMAKKAL
ZIFO RND(4.76L),
ACCENTURE(4.5L),DXC(4.2L)
STUDENTS PLACED IN 3 COMPANIES 
ABINAYA K - BIOMEDICAL
TIRUPPUR
ACCENTURE(4.5L)
 COGNIZANT (4L),EPISOURCE(3L)
MADHUMITHA M - BIOMEDICAL
TRICHY
ZIFO RND(4.76L),DXC (4.2L)
CAPGEMINI(4L)
KEERTHANA PS - BIOMEDICAL
KARUR
DXC(4.2L),TCS(3.36L)
CAPGEMINI(5.5L)
SHUBHASHREE T - BIOMEDICAL
TIRUPPUR
DXC(4.2L),COGNIZANT(4L)
EPISOURCE(3L)
STUDENTS PLACED IN 2 COMPANIES 
SRINIDHI S - BIOMEDICAL
ERODE
DXC(4.2L),EPISOURCE(3L)
SANKARI D - BIOMEDICAL
KARUR
   FACE PREP(3L),CAPGEMINI(4L)
PUGALPATHY S – BIOMEDICAL
NAMAKKAL
FACE PREP(3L),EPISOURCE(3L)
LOGESHWARAN M - BIOMEDICAL
TRICHY
TCS(3.36L),EPISOURCE(3L)
SUJITH S - BIOMEDICAL
TIRUPPUR
ACCENTURE (4.5L), COGNIZANT(4L)
KALEESWARI K - BIOMEDICAL
DINDIGUL
ZIFO RND(4.76L),DXC(4.2L)
HARIHARAN A - BIOMEDICAL
DINDIGUL
ACCENTURE(4.58L),TCS(3.36L)
KARISHMA B - BIOMEDICAL
KANYAKUMARI
DXC(4.2L),TCS(3.36L)
SAVITHA P - BIOMEDICAL
KARUR
TCS(7L),DXC(4.2L)
NAVEEN R - BIOMEDICAL 
DINDIGUL
ZIFO RND(4.76L),DXC(4.2L)
DEEPAK P - BIOMEDICAL
TIRUPPUR
TCS (3.36L)
ISHWARYA PRASATH V - BIOMEDICAL
 DINDIGUL
 TCS(3.36L)
KALIDASS K - BIOMEDICAL
DINDIGUL
 ACCENTURE(4.5L)
YAZHINI S - BIOMEDICAL
KARUR
TCS(3.36L)
SHOBIKA M - BIOMEDICAL
KARUR
ACCENTURE(4.5L)
STUDENTS PLACED IN 1 COMPANY 
ABIRAMI V - BIOMEDICAL
KARUR
ACCENTURE(4.5L)
PRASANNAVASAN A - BIOMEDICAL
CUDDALORE
TCS(3.36L)
HARIHARAN N - BIOMEDICAL
ERODE
TCS(3.36L)
BHARATH J - BIOMEDICAL
KARUR
DXC(4.2L)
PANDI DURAI A - BIOMEDICAL
PUDUKOTTAI
ACCENTURE(4.5L)
bio TECHNOLOGY
ABINAYA B - BIOMEDICAL
 PUDUKOTTAI
DXC(4.2L)
GAJENDRAN K - BIOMEDICAL
NAGAPATTIANAM 
EPISOURCE(3L)
ABDUL BASITH AKRAM - BIOMEDICAL
ERODE
EPISOURCE(3L)
JAYA PRAKASH A - BIOMEDICAL
DINDIGUL
EPISOURCE(3L)
RAJESH V - BIOMEDICAL
TRICHY
EPISOURCE(3L)
SATHISH KUMAR R - BIOMEDICAL
TIRUPPUR
EPISOURCE(3L)
ARAVINTH T - BIOMEDICAL
NAMAKKAL
EPISOURCE(3L)
KARTHIKEYAN B - BIOMEDICAL
TIRUPPUR
EPISOURCE(3L)
CHANDRIKAVATHI C - BIOMEDICAL
KODAIKANAL
CAPGEMINI(4L)
PRANESHWARAR M- BIOMEDICAL
 KARUR
FACE PREP(3L)
ROHITH KUMAR T- BIOMEDICAL
KARUR
FACE PREP(3L)
VINOTH KUMAR M - BIOMEDICAL
TRICHY
TCS(3.36L)
STUDENTS PLACED IN 3 COMPANIES 
VARSHINEE K R - BIOTECH
KARUR
DXC(4.2L),TCS(3.36L), 
EPISOURCE(3L)
HELENA FLORA M – BIOTECH
TUTICORIN
CAPGEMINI(4.25L),COGNIZANT(4L), 
FACE PREP(3L)
YOGESWARAN S - BIOTECH
KARUR
TCS(7L),ZIFO(4.7L),DXC(4.2L)
KARTHIKA DEVI S- BIOTECH
KANGEYAM
ZIFO(4.7L),DXC(4.2L),TCS(3.36L)
STUDENTS PLACED IN 2 COMPANIES 
SANGEETHA P - BIOTECH
DINDIGUL
CAPGEMINI(4.25 L),TCS(3.36L)
SATHYA M - BIOTECH
ERODE
DXC(4.2L),TCS(3.36L)
THARUNKUMAR A S - BIOTECH
KARUR
ZIFO(4.7L), ACCENTURE(4.5L)
MOHAN K - BIOTECH
MOHANUR
ZIFO(4.7L),TCS(3.36L)
SRINIVAS D - BIOTECH
NAMAKKAL
DXC(4.2L),TCS(3.36L)
HARINI T - BIOTECH
KUMBAKONAM
ZIFO(4.7L), DXC(4.2L)
POORNASARENA T - BIOTECH
VELLAKOIL
DXC(4.2L),LTIMINDTREE(4L)
PRIYA DHARSHINI J - BIOTECH
KARUR
CAPGEMINI(4.25L),TCS(3.36L)
SIVASANGARI  M - BIOTECH
PUDUKKOTTAI
TCS(3.36L),EPISOURCE(3L)
INDHUJA J – BIOTECH
MANNARGUDI
EPISOURCE(3L),FACPREP(3L)
RAKSHANA R – BIOTECH
KANGEYAM
TCS(3.36L),EPISOURCE(3L)

SASIREGA M - CIVIL 
PALLADAM
TCS(7L),ZOHO(5.6L), 
DXC(4.2L)
STUDENTS PLACED IN 1 COMPANY 
KARTHIKA S - BIOTECH
DINDIGUL
ZIFO(4.7L)
PRABU G - BIOTECH
NAMAKKAL
DXC(4.2L)
PRIYA D - BIOTECH
VAIRAMADAI
TCS (3.36L)
MATHUMITHA N - BIOTECH
KARUR
EPISOURCE(3L)
SANGEETHA J - BIOTECH
ERODE
TCS(3.36L)
SUBASHINI M - BIOTECH
NATHAKADAIYUR
TCS(3.36L)
AAFIYA THAKSEEN S - BIOTECH
UTHUKULI
FACE PREP(3L)
BOOBALAN P - BIOTECH
NAMAKKAL
LTIMINDTREE(5L)
CHITHRA S - BIOTECH
KARUR
LTIMINDTREE(4L)
PRADEEPA S - BIOTECH
KARUR
EPISOURCE(3L)
SUNIL KISHORR M - BIOTECH
LALAPET
EPISOURCE(3L)
PRIYA DHARSHINI S - BIOTECH
NAMAKKAL
EPISOURCE(3L)
METHA J - BIOTECH
TIRUPPUR
CAPGEMINI(4.25L)
civil ENGINEERING
ABITHA V - CIVIL 
BATLAGUNDU
ACCENTURE(4.5L),DXC(4.2L), 
COGNIZANT(4L),TCS(3.36L)
STUDENT PLACED IN 4 COMPANIES 
STUDENT PLACED IN 3 COMPANIES 
STUDENTS PLACED IN 2 COMPANIES 
KARTHIKEYAN P - CIVIL 
KARUR
LTIMINDTREE(4L),TCS(3.36L)
VANCHIKAVIYA S - CIVIL 
VELLAKOVIL
DXC(4.2L),LTIMINDTREE(4L)
NIRANJAN S - CIVIL 
ERODE
ACCENTURE(4.5L), LTIMINDTREE(4L)
AJITHAA V - CIVIL 
BATLAGUNDU
ACCENTURE(4.5L),CAPGEMINI(4.25L)
VASANTH V - CIVIL
KUNJIPALAYAM,COIMBATORE
LTIMINDTREE(4L),TCS(3.36L)
GOWTHAM V - CIVIL
POLLACHI,COIMBATORE
DXC(4.2L),TCS(3.36L)
STUDENTS PLACED IN 1 COMPANY 
NAVEEN P - CIVIL
KANGEYAM
ACCENTURE(4.5L)
VASUDEVAN R - CIVIL
NAMAKKAL
DXC(4.2L)
NARESH KUMAR R - CIVIL 
TRICHY
DXC(4.2L)
HARIHARAN B - CIVIL
TRICHY
TCS(3.36L)
PRASATH  M - CIVIL
KANGEYAM
ACCENTURE(4.5L)
UNNAMALAI K - CIVIL
MADURAI
TCS(3.36L)
PALANIKUMAR C - CIVIL
PUDUKOTTAI
FACE PREP(3L)
GIRINATH S - CIVIL
KARUR
TCS(3.36L)
KUNGUMARAJ S - CIVIL
DINDIGUL
LTIMINDTREE(4L)
MANMATHAN S - CIVIL
MELUR,MADURAI
DXC(4.2L)
SATHIYANARAYANAN N - CIVIL
KONDAMPATTI,COIMBATORE
TCS(3.36L)
JAI SURESH M – CIVIL
SIVAGALAI, THOOTHUKKUDI
TCS(3.36L)
chemical ENGINEERING
STUDENT PLACED IN 4 COMPANIES 
SURENTHIRAN C - CHEMICAL
TIRUCHENGODE
CAPGEMINI(5.75L),DXC(4.2L),
COGNIZANT(4L),QUEST GLOBAL(3.25L)
STUDENTS PLACED IN 3 COMPANIES 
DIVYA DHARSHINI M - CHEMICAL
KANGAYAM
CAPGEMINI(4.25L),QUEST GLOBAL(3.25L)
FACE PREP (3L)
SIVASANKARAPANDI N - CHEMICAL
PALANI
DXC(4.2L),COGNIZANT(4L ),
LTIMINDTREE(4L) 
STUDENTS PLACED IN 2 COMPANIES 
VELMURUGAN S – CHEMICAL
KARUR
TCS (3.36L),
QUEST GLOBAL(3.25L)
NANDHINI S - CHEMICAL
PALANI
QUEST GLOBAL(3.25L),
DXC (4.2L)
SOWMIYA B - CHEMICAL
SIVAGANGAI
DXC(4.2L),
QUEST GLOBAL(3.25L)
EZHIL MADHAN V - CHEMICAL
TIRUNELVELI
COGNIZANT(4 L),TCS(3.36L)
LINGESH K - CHEMICAL
AMBUR
DXC(4.2 L),TCS(3.36L)
SUBASH V - CHEMICAL
KARUR
ZOHO(5.6L),DXC(4.2L)
KANCHANA C - CHEMICAL
MUTHUR
DXC(4.2L)
SANJAY S - CHEMICAL
THIRUVARUR
QUEST GLOBAL(3.25L)
SANKARANARAYANAN S – CHEMICAL
DINDIGUL
COGNIZANT(4L)
RANJITHKUMAR T - CHEMICAL
PALANI
TCS(3.36L)
PRIYADHARSHINI S - CHEMICAL
KANGAYAM
DXC(4.2L)
PREETHI D - CHEMICAL
CHENNAI
FACE PREP(3L)
STUDENTS PLACED IN 1COMPANY
INDHIRA GANDHI S - CHEMICAL
SALEM
TCS(3.36L)
HARINI P - CHEMICAL
KARUR
COGNIZANT(4L)
HARIHARAN J - CHEMICAL
KARUR
TCS(3.36L)
GAYATHIRI N - CHEMICAL
TRICHY
TCS(3.36L)
JEROME PRASAD J - CHEMICAL
KARUR
FACE PREP(3L)
MANOHAR R - CHEMICAL
NAMAKKAL
TCS (3.36L)
MAATHAVRAAJ V M - CHEMICAL
DINDIGUL
TCS(3.36L)
KULOTHUNGAN S - CHEMICAL
THIRUVARUR
TCS(3.36L)
VISHWA B - CHEMICAL
VIRUDDHACHALAM
ACCENTURE(4.5L)
MUGESHRAAJ V M - CHEMICAL
DINDIGUL
FACE PREP(3L)
SHARAN KUMAR A - MECH
UTHANGARAI
DXC(4.2L), LTIMINDTREE(4L), 
TCS(3.36L)
BALA S – MECH
THENI
DXC(4.2L),LTIMINDTREE(4L),
TCS(3.36L)
GANESH K - MECH
KARUR
TCS(7L),DXC(4.2L),
QUEST GLOBAL(3.25L)
ROHITH S - MECH
SINGANALLUR,COIMBATORE
ACCENTURE(4.5L),
DXC(4.2L),COGNIZANT(4L)
AJITH KUMAR T - MECH
NEYVELI
DXC(4.2L),LTIMINDTREE(4L),
COGNIZANT(4L),
QUEST GLOBAL(3.25L)
DHINESH KUMAR S - MECH
MANAPPARAI
TCS(7L), DXC(4.2L), 
LTIMINDTREE(4L),QUEST GLOBAL(3.25L) 
KAMALESH K B – MECH
VADAMBACHERY,COIMBATORE
 ACCENTURE(4.5L),)DXC(4.2L),
COGNIZANT(4L),QUEST GLOBAL(3.25L)
SANJAY GIRI K - MECH
THANJAVUR
KAAR(6.5L),DXC(4.2L), 
LTIMINDTREE(4L),COGNIZANT(4L),
TCS(3.36L)
mechanical ENGINEERING
STUDENT PLACED IN 5 COMPANIES 
STUDENTS PLACED IN 4 COMPANIES 
STUDENTS PLACED IN 3 COMPANIES 
CHANDRU N – MECH
KARUR
LTIMINDTREE(4L),
COGNIZANT(4L)
VIGNESH L – MECH
THIRUVARUR
DXC(4.2L),TCS(3.36L)
VENKATESH C - MECH
MULANUR
COGNIZANT(4L),LTIMINDTREE(4L)
NITHISHDEVAN P - MECH
KARUR
DXC(4.2L),TCS(3.36L)
AJAYKANNA A - MECH
KARUR
DXC(4.2L),TCS(3.36L)
PONEZHILJOTHI K - MECH
PALANI
LTIMINDTREE(4L),TCS(3.36L)
SINTHANAI SELVAN T - MECH
PALANI
DXC(4.2L),TCS(3.36L)
BARATH SHABARI M - MECH
THANJAVUR
DXC(4.2L),LTIMINDTREE(4L)
SAUNDARA RAJANR - MECH
DHANNIKHAN VILLAGE, SOUTH ANDAMAN
ACCENTURE(4.5L),TCS(3.36L)
RAGHURAM R K - MECH
SOUTH SOORANKUDY, KANYAKUMARI
DXC(4.2L),TCS(3.36L)
MOUN MANOJ A - MECH
POLLACHI,COIMBATORE
ACCENTURE(4.5L),DXC(4.2L)
JAISURYA S - MECH
KARUR
LTIMINDTREE(4L), 
TCS(3.36L)
PRADEEP KUMAR M - MECH
VELLAKOVIL
DXC(4.2L),
LTIMINDTREE(4L)
RAMPRASANTH P - MECH
KARUR
EMERSON(4.6L),
QUEST GLOBAL(3.25L) 
CHANDRU N - MECH
KARUR
LTIMINDTREE(4L),TCS(3.36L)
STUDENTS PLACED IN 2 COMPANIES 
ABISHEK S - MECH
KARUR
DXC(4.2L),TCS(3.36L)
SABARI HARIHARAN T - MECH
JEDARPALAYAM
DXC(4.2L),TCS(3.36L)
NANDHAKUMAR S - MECH
UDUMALPET,TIRUPPUR
DXC(4.2L),LTIMINDTREE(4L)
KAALIDHASAN  A - MECH
DINDIGUL
DXC(4.2L)
NESAN A - MECH
PARAMATHI VELUR
TCS(3.36L)
ARUL MURUGAN K - MECH
RASIPURAM
TCS(3.36L)
JEEVA KUMAR S - MECH
TRICHY
TCS(3.36L)
NATHISH  S - MECH
KARUR
TCS(3.36L)
SACHIN JEEVA R - MECH
MANAPARAI
TCS(3.36L)
SANGEETH S - MECH
RAMESHWARAM
DXC(4.2L)
ROJINIVASAN R - MECH
MUSIRI
DXC(4.2L)
SABARI PON KUMARAN  C - MECH
PARAMATHI VELUR
DXC(4.2L)
MATHUSUTHANAN V - MECH
PALLADAM
QUEST GLOBAL(3.25L)
STUDENTS PLACED IN 1 COMPANY
UMA SANKAR S - MECH
SALEM
LTIMINDTREE(4L)
SANJAY KUMAR R - MECH
MANAPPARAI
TCS(3.36L)
GIRIVENDHAN D - MECH
RASIPURAM
TCS(3.36L)
SARATHI P - MECH
KARUR
DXC(4.2L)
SUGUMAR M - MECH
NEGAMAM,COIMBATORE
TCS(3.36L)
JEEVANANTHAM  N – MECH
KARUR
TCS(3.36L)
MUTHU SELVAN S - MECH
KARUR
TCS(3.36L)
KRISHNA PRASAD R - MECH
KARUR
TCS(3.36L)
SIDHARTHAN J - MECH
KARUR
FACE PREP(3L)
VASANTHAKUMAR R - MECH
NAMAKKAL
TCS(7L)
YUVAN SHANKAR E - MECH
ERODE
TCS(3.36L)
SATHIS S - MECH
DINDIGUL
FACE PREP(3L)
LOKESH R - MECH
DINDIGUL
TCS(3.36L)
SANKAR S - MECH
NAMAKKAL
TCS(3.36L)
HARI  PRAKASH M - MECH
ACHANPUTHUR,TENKASI
DXC(4.2L)
RAGUL S – MECH
DHARAPURAM
TCS(3.36L)
SRINIVASAN B – MECH
MAYILADUTHURAI
COGNIZANT(4L)
SATHEESH J K – MECH
KUMBAKONAM
TCS(3.36L)
KEERTHIVASAN D - MECH
DINDIGUL
FACEPREP(3.13L)
VENKATESH S - MECH
MANAPPARAI
LTIMINDTREE(4L)
VIGNESH S - MECH
CHENGALTTU
LTIMINDTREE(4L)
R

Admission Registration Fee
Rs.50,000/-
Online Enquiry : https://admissions.vsbcetc.com/
KARUR : 82200 80832, 82705 96212
 COIMBATORE : 82205 55783, 82200 48212
Applied Electronics
Computer Science and Engineering
Power System Engineering
  Electronics and Communication Engineering
 
FOR ADMISSION ENQUIRY CONTACT
COIMBATORE COLLEGE :
TH2012 - 13 NOV - DEC ODD SEM 11  POSITION IN THE STATE.
TH2012 - 13 APRIL - MAY EVEN SEM 3  POSITION IN THE STATE.
TH2015 - 16 APRIL - MAY EVEN SEM 10  POSITION IN THE STATE.
KARUR COLLEGE : 
TH2015 - 16 APRIL - MAY EVEN SEM 4  POSITION IN THE STATE.
TH2016 - 17 NOV - DEC ODD SEM 3  POSITION IN THE STATE.
TH 2017 - 18 NOV - DEC ODD SEM 6 POSITION IN THE STATE.
B.E., B.Tech.
M.E.
COURSE OFFERD SALIENT FEATURES
1. Both the Institutions have signed MOU with PEGA UAP,
    Google Cloud, Amazon Web Services, TCS iON, L&T Edu Tech,
    Oracle Academy, NSEIT, CISCO, Paloalto And Solver Minds.
2. Both the Institutions received UGC recognition with 2(f)
    status and Karur College has 12(B) status.
3. Both the Institutions are associated with ICT ACADEMY,
    and ISTE.
4. Both the Institutions have Centre Of Excellence with 
    Capgemini, Virtusa, Hexaware, Tessolve and IOT supported by 
    ERA Interface
5. Both the Institutions produce consistently Top University Results 
    in the state. Details are given below-
6. Both the Institutions have best placement record. All the
    top 10 Multi National IT Service Companies are recruiting 
    our students every academic year.
7. Both the Institutions have Greenery campus. Karur college
    received Jal Sakthi Award from AICTE.
8. Both the Institutions have well equipped laboratories,
    communication laboratory and Hi-Tech infrastructure to
    make the students skilled and proﬁcient.
9. Both the Institutions have residential accomodation for both
    boys and girls that caters to the needs of more then 1700
    students with variety of delicious food.
 
B.E. Biomedical Engineering
B.E. Civil Engineering #
B.E. Computer and Communication Engineering
B.E. Computer Science and Engineering
        (Artiﬁcial Intelligence and Machine Learning)
B.E. Computer Science and Engineering #
B.E. Electronics and Communication Engineering #
B.E. Electrical and Electronics Engineering #
B.E. Mechanical Engineering #
B.Tech. Agricultural Engineering
B.Tech. Artiﬁcial Intelligence & Data Science
B.Tech. Biotechnology
B.Tech. Computer Science & Business Systems
B.Tech. Chemical Engineering
B.Tech. Information Technology #
P  h.D
 10. Both the  Institutions are facilitated with easiest transportation 
       for both students and staﬀs.
 11. Karur College is 21 years old, Coimbatore College is 11 years old,
       both the colleges have NBA accredited Courses.
 12. VSB Engineering College Karur Campus is associated with TCS SIRIUS

Automating Reinforcement Learning Architecture
Design for Code Optimization
Huanting Wang
1. Northwest University, China
2. University of Leeds, U. K.
Zhanyong Tang
Northwest University, China
Cheng Zhang
Northwest University, China
Jiaqi Zhao
Northwest University, China
Chris Cummins
Meta AI Research, USA
Hugh Leather
Meta AI Research, USA
Zheng Wang
University of Leeds, U. K.
Abstract
Reinforcement learning (RL) is emerging as a powerful tech-
nique for solving complex code optimization tasks with an
ample search space. While promising, existing solutions re-
quire a painstaking manual process to tune the right task-
specific RL architecture, for which compiler developers need
to determine the composition of the RL exploration algo-
rithm, its supporting components like state, reward and
transition functions, and the hyperparameters of these mod-
els. This paper introduces SuperSonic, a new open-source
framework to allow compiler developers to integrate RL
into compilers easily, regardless of their RL expertise.Super-
Sonic supports customizable RL architecture compositions
to target a wide range of optimization tasks. A key feature
of SuperSonic is the use of deep RL and multi-task learn-
ing techniques to develop a meta-optimizer to automatically
find and tune the right RL architecture from training bench-
marks. The tuned RL can then be deployed to optimize new
programs. We demonstrate the efficacy and generality of
SuperSonic by applying it to four code optimization prob-
lems and comparing it against eight auto-tuning frameworks.
Experimental results show that SuperSonic consistently im-
proves hand-tuned methods by delivering better overall per-
formance, accelerating the deployment-stage search by 1.75x
on average (up to 100x).
CCS Concepts:• Software and its engineering →Com-
pilers; • Computing methodologies →Artificial intelli-
gence.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not
made or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for components
of this work owned by others than ACM must be honored. Abstracting with
credit is permitted. To copy otherwise, or republish, to post on servers or to
redistribute to lists, requires prior specific permission and/or a fee. Request
permissions from permissions@acm.org.
CC ’22, April 02–03, 2022, Seoul, South Korea
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9183-2/22/04. . . $15.00
https://doi.org/10.1145/3497776.3517769
Keywords: Compiler optimization, reinforcement learning,
code optimization
ACM Reference Format:
Huanting Wang, Zhanyong Tang, Cheng Zhang, Jiaqi Zhao, Chris
Cummins, Hugh Leather, and Zheng Wang. 2022. Automating Re-
inforcement Learning Architecture Design for Code Optimization.
In Proceedings of the 31st ACM SIGPLAN International Conference
on Compiler Construction (CC ’22), April 02–03, 2022, Seoul, South
Korea. ACM, New York, NY, USA, 16 pages. https://doi.org/10.1145/
3497776.3517769
1 Introduction
There is a growing interest in using auto-tuning techniques
for code optimization [9, 18, 27, 32, 62, 70, 88]. Auto-tuning
finds good optimizations from empirical observations, often
outperforming hand-crafted heuristics [9, 52]. This technique
is particularly attractive for frequently used libraries and
kernels. For such scenarios, developers are willing to spend
hours or weeks of CPU time to automatically search for even
a few percentages of performance improvement [18, 23, 50,
83], knowing the optimized code will be used by many users.
In recent years, deep reinforcement learning (RL) has been
shown to be effective for navigating a sizeable discrete space,
outperforming traditional search techniques on a range of
optimization tasks [33, 60, 79, 85, 87]. Deep RL is also a nat-
ural fit for many code optimization problems where the task
can be seen as applying a sequence of actions to maximize
the gain [88]. Examples of such problems include compiler
flag selection and ordering [4, 25, 47], instruction scheduling
[69, 80, 92] and hardware resource allocation [26]. Indeed,
we see a growing interest in the research community and
industry [23, 83] in using RL and deep RL to tackle a wide
range of code optimization problems [13, 35, 36, 45, 53, 68].
Developing a deep RL solution for code optimization re-
quires choosing and parameterizing several components: (i) a
discrete set of actions or transformations that can be applied
to a program, such as passes in a compiler; (ii) a state func-
tion that can summarize the program after each action as a
finite feature vector, and (iii) a reward function that reports
the quality of the actions taken so far. Some RL algorithms
CC ’22, April 02–03, 2022, Seoul, South Korea H. Wang, Z. Tang, C. Zhang, J. Zhao, C. Cummins, H. Leather, and Z. Wang
may also allow the selection and further parameterization of
a transition function, which governs the choice of actions to
be applied in each state. Moreover, parameters of individual
RL components need to be tuned on benchmarks.
Efforts have been made to provide RL algorithms and high-
level APIs for action definitions [23, 51], models for program
state representation [12, 21, 86, 90], and tools for training
benchmark generation [22–24]. While these recent works
have lowered the barrier for integrating RL techniques into
compilers, compiler engineers still face a major hurdle. As
the right combination of RL exploration algorithms and their
state, reward and transition functions and parameters highly
depend on the optimization task, developers must carefully
choose the RL architecture by finding the right RL compo-
nent composition and their parameters from a large pool of
candidate RL algorithms, machine-learning models and func-
tions. This process currently requires testing and manually
analyzing a large combination of RL components. Experience
in the field of neural architecture search shows that doing
this by hand is an expensive and non-trivial process [28].
This paper presents SuperSonic1, an open-source frame-
work to automate the RL architecture search and param-
eter tuning process to make it easier to integrate RL into
compilers. To use SuperSonic, the compiler developer pro-
vides the action list according to the problem being tackled
and a measurement interface to report metrics like code
size or speedup. SuperSonic then automatically assembles
an RL architecture for the targeting optimization from an
extensible set of built-in RL components. The SuperSonic
RL components include pre-trained state functions, such as
Word2Vec [57] and CodeBert [29]. It provides candidate re-
ward functions like RelativeMeasure and tanh to compute
the reward based on the metric given by the measurement in-
terface. The state-transition function can be selected from a
further set of predefined transition functions, such as a tran-
sition probability matrix or LSTM [39]. Finally, SuperSonic
takes a customizable set of predefined RL algorithms, like
Proximal Policy Optimization (PPO) [77] and Monte Carlo
tree search (MCTS) [14], which may be driven by any of the
chosen reward, state, actions and transition function. This
creates a large space of possible parameterized RL architec-
tures, which can be defined by the compiler developer with
a few lines of Python using an easy-to-use API.
Armed with this space of RL architecture choices, Super-
Sonic will automatically and efficiently search it to find the
one that gives the best results over a sample of user-provided
training benchmarks. The search is accomplished by a deep
RL-based meta-optimizer that is designed to be generalizable
to any optimization tasks. Once the meta-optimizer has se-
lected the client RL architecture and its parameters, the work
of SuperSonic is over. As an output, SuperSonic stores the
tuned client RL as serialized objects. It provides an API for a
1Available at: https://github.com/HuantWang/SUPERSONIC
compiler or performance tuner to use the stored RL to drive
the code transformation pass for new, unseen programs. On
an unseen program, the client RL may be used to search for
the actions that yield the best reward. Alternatively, it can be
further generalized by training on additional benchmarks so
that actions for new programs can be chosen directly from
the policy without further search at the deployment time.
SuperSonic aims to automatically make the client RL as
good as possible for the compiler developer’s needs.
As this client RL search and tuning process is automated
and requires little RL expertise, SuperSonic further reduces
the difficulties for integrating RL into compilers by replacing
compiler developer time with machine hours. This client RL
search and tuning process is a one-off cost performed offline.
The compiler end-users (e.g., application developers) will
not experience this process – they will use the shipped RL
like any other feedback directed compiler passes [16].
We evaluate SuperSonic by applying it to four optimiza-
tion tasks: Halide schedule optimization [69], neural network
code generation [18], compiler phase ordering [25] for code
size reduction and superoptimization [43]. Each of the tasks
has a large number of combined optimization options, so
it is non-trivial to design a good search strategy. We com-
pare SuperSonic against eight tuning methods developed by
independent researchers, including search-based strategies
specifically designed for the targeting problem [3, 19, 62, 76],
generic tuning frameworks like OpenTuner [9] and Compil-
erGym [23], and hand-tuned RL solutions for the relevant
task [5, 68]. Our extensive evaluation shows thatSuperSonic
consistently gives better overall performance than alterna-
tive methods across tasks during deployment. We show that
the client RL given by SuperSonic converges fast, and it can
start producing better code over competing search methods
by using on average 1.75x less search time (up to 100x) for
new programs during deployment.
This paper makes the following contributions:
•We present a generic framework to automatically
choose and tune a suitable RL architecture for code
optimization tasks (Section 3).
•We demonstrate how deep RL can be used as a meta-
optimizer to support the integration of RL into perfor-
mance tuners (Section 3.3).
•We provide a large study validating the effectiveness
of RL in four code optimization tasks (Section 5).
2 Background
2.1 Deep Reinforcement Learning
RL is a machine learning technique where agents learn to
perform actions in an environment to maximize a cumulative
reward [81]. The environment represents the problem to be
solved, and the agent represents the learning algorithm. At
each time step, the agent observes the environment state
and takes an action (e.g., deciding which compiler option
Automating Reinforcement Learning Architecture Design for Code Optimization CC ’22, April 02–03, 2022, Seoul, South Korea
to be added or removed from the compilation sequence).
The agent adjusts its actions based on observations gathered
from the environment. It learns a policy to map the states to
the corresponding actions, aiming to maximize the expected
reward. Some RL algorithms also learn a value function to
evaluate the current situation of the environment and the
decision-making process of the agent. Unlike a policy func-
tion that answers the question of “how to act?", the value
function answers the question of “how good the current state
is?". Using a deep neural network (DNN) along with RL is
known as Deep Reinforcement Learning [37].
2.2 Problem Definitions
SuperSonic automates RL component searching and param-
eter tuning. Within SuperSonic, a client RL consists of an ex-
ploration algorithm to choose actions (e.g., compiler options),
a reward function for computing the expected cumulative
reward based on past observations of the environment (e.g.,
execution time after applying a transformation), a method
for modeling the environment/program state (e.g., a DNN
or a linear function), and an action list provided by the user
(e.g., legitimate code transformation options). Depending on
the exploration algorithm, this can also include an state tran-
sition function to compute the probability of going from one
state to another. Each of the components can be chosen from
a pool of SuperSonic built-in candidate methods, and the
combination of these components can result in a large policy
search space. SuperSonic is designed to automatically find
and tune the right combination of RL components and their
hyperparameters. It complements existing RL platforms like
CompilerGym [23] and RLLib [51], by helping the compiler
developers to choose and optimise a suitable RL algorithm.
2.3 Multi-armed Bandit Problem
The effectiveness of an RL algorithm is highly dependent on
the problem to be solved, but manually choosing the right RL
components is non-trivial [44]. In this work, we formulate RL
component search as a multi-armed bandit (MAB) problem
[30]. Given a budget of𝑛 trials and𝑘 candidate RL component
configurations (or slot machines), we use a deep RL-based
MAB solver to allocate the trials among candidate policy
architectures to test their effectiveness. After 𝑛 trials, we
determine which of the 𝑘 RL configurations to use for the
given problem. This technique is detailed in Section 3.3.
3 Our Approach
3.1 Overview
Figure 1 gives an overview of SuperSonic. At the core of
SuperSonic is a meta-optimizer that builds upon MAB and
deep RL techniques. Given a client RL search space defined by
the SuperSonic Python API, the meta-optimizer searches for
a suitable RL component configuration for an optimization
task. It then automatically tunes a set of tunable hyperpa-
rameters of the chosen components. The tuned client RL can
then be used to optimize unseen programs through inference
(including potential retraining), which is outside the scope
of SuperSonic. The search space definition and RL client
architecture search and parameter tuning are aone-off offline
process, which is within the scope of SuperSonic.
Implementation. We implement SuperSonic in Python
and use gRPC for distributed communications. SuperSonic
builds upon CompilerGym [23] and RLlib (Ray) [51] by uti-
lizing their APIs for task definitions and RL algorithm imple-
mentations. SuperSonic currently supports 23 RL algorithms
from RLLib [51] and 10 pre-trained DNNs and functions for
representing the program state or computing the reward.
Compiler developers may choose a subset, add their own, or
include all (the default) supported RL algorithms and models
for the meta-optimizer to search over.
Task definition. The compiler developer first defines the
optimization problem by creating an RL policy interface (Fig-
ure 2). The definition includes a list of client RL components
for the meta-optimizer to search over.
Client RL search. Calling to policy_search() invokes
SuperSonic meta-optimizer, where the developer can also
limit the number of trials spent on client RL searching.
Client RL parameter tuning and deployment. After
choosing an RL architecture, the meta-optimizer will fine-
tune a set of model-specific hyper-parameters of the selected
client RL (see also Table 1). Hyperparameter tuning is per-
formed on the training benchmarks. The tuned client RL
and its parameters are saved, which can be shipped with a
compiler to optimize unseen programs at deployment time.
Measurement engine. The measurement engine evaluates
a code transformation option using an user-supplied inter-
face (line 24 in Figure 2). Measurements are used during
client RL search and tuning, as well as deployment phases
to obtain feedback for a chosen optimization action.
3.2 Task Definition
The user-defined client RL search space typically includes
candidate functions (or models) for representing the envi-
ronment state, objective functions for computing the reward,
and the set of possible actions that can be taken from a given
state. This search space definition can optionally include a
chosen set of RL exploration algorithms and transition func-
tions to be used by a client RL algorithm. By default, Super-
Sonic automatically search over all supported RL algorithms
where each algorithm has a default transition function. Fur-
thermore, the compiler engineer also needs to provide a
run function, which provides the measurement of an action
to compute the reward. These are implemented in a small
Python program to interface with the SuperSonic API. The
definition code is similar to the programming environment
of mainstream auto-tuners like OpenTuner and Compiler-
Gym, allowing the developer to quickly port their code to
use the SuperSonic search and tuning components.
CC ’22, April 02–03, 2022, Seoul, South Korea H. Wang, Z. Tang, C. Zhang, J. Zhao, C. Cummins, H. Leather, and Z. Wang
Measurement Interface
State 
functions
Reward 
functions Actions
Multi‐armed Bandit based 
Policy Search 
Multi‐task learning
Client RL Search
Client RL search space
RL algorithms
SUPERSONIC
Training 
benchmarks
Chosen Client RL 
Architecture
Optimized code
Tuned Client RL
Unseen programs
Client RL Deployment
Client RL Parameter Tuning
Offline Task defining and Client RL Search; Done once Retraining and inference
Figure 1. Overview of SuperSonic components. This framework enables developers to express the optimization space. It
automatically searches for the optimal client RL architecture to be used for inference during deployment.
1 import SuperSonic as ss
2 from SuperSonic . stateFunctions . models import *
3 ...
4 statefs = [ Word2Vec (...) , Doc2Vec (...) , CodeBert
(...) , ActionHistory (...) ]
5 tranfs = [ DNN (...) ,CNN (...) ,LSTM (...) ]
6 rewards = [ RelativeMeasure (...) ,tanh (...) ]
7 rl_algs = [ MCTS (...) ,PPO (...) ,DQN (...) , QLearning
(...) ]
8 actions = [ Init (...) ]
9 ...
10
11 class SuperOptimizer (ss. PolicyInt ):
12 def __init__ (self , statefs , tranfs , actions ,
rewards , rl_algs ):
13 self . PolicySpace = {
14 " StatList ": statefs ,
15 " TranList ": tranfs ,
16 " ActList ": actions ,
17 " RewList ": rewards ,
18 " AlgList ": rl_algs ,
19 }
20 self . search_engine = SearchEngine ( self .
PolicySpace )
21
22 def run ( self ):
23 # user code for compilation and execution
24 ...
25 return Result ( time = run_result ['time '])
26
27 if __name__ == '__main__ ':
28 opt = SuperOptimizer ( statefs , tranfs , actions ,
rewards , rl_algs )
29 policy = opt . policy_search (
training_benchmark_list , num_of_trials =100)
Figure 2. Simplified tasks definition for superoptimizaton.
Figure 2 gives a simplified example that defines the client
RL search space for superoptimizaton [43, 55]. This example
specifies candidate methods for representing the environ-
ment state, functions for computing rewards, the definition
for action space, and a chosen set of client RL algorithms.
The run function implements the measurement interface, in-
cluding user code for compiling and executing the program
for a given code transformation action. Finally, the program
invokes the policy search API by passing in a list of training
benchmarks and the number of search trials. We stress that
measurement interface defines how to compile and execute a
test program, but the target program can be of any language.
3.3 Client RL Search
Given a client RL search space, the SuperSonic meta-
optimizer automatically finds a suitable client RL architec-
ture (i.e., <state function, transition function, reward func-
tion, RL algorithm> and potentially a value function) from
training benchmarks. We formulate the client RL search prob-
lem as a MAB problem that is solved using a parallel DRL
algorithm. Deep RL can reuse knowledge from other tasks
to speed up the search. In contrast, evolutionary algorithms
have to search from afresh. Our work is the first to formulate
client RL tuning for code optimization as a MAB problem
and employs deep RL as a meta-solver.
Our meta-optimizer is a variant of the Asynchronous Ad-
vantage Actor Critic (A3C) algorithm [38, 59]. A3C is a dis-
tributed algorithm, where multiple workers independently
update a global value function – hence “asynchronous". We
choose this algorithm because it is shown to be effectively
in other RL application domains [38] and permits us to de-
velop a parallel policy search engine. Specifically, the meta-
optimizer consists of two RL models, an actor for computing
an action based on observation and a critic for estimating a
reward value. In a nutshell, the actor is a policy RL that takes
as input the environment state and outputs the best action
(a policy architecture in our context). The actor essentially
controls how the meta-optimizer choose a candidate client
policy to try out. By contrast, the critic is value-based RL
that evaluates the action by adjusting its value function to
estimate the maximum future reward based on the historical
observations obtained from training benchmarks. As time
passes, the actor is learning to produce better actions, and
the critic is getting better at evaluating those actions.
Like [38], we implement the actor and the critic using a
stacked neural network consisting of a ResNet convolution
neural network (CNN) [46] that is followed by a LSTM re-
current neural network (RNN) [ 39]. We use the output of
the LSTM to update the policy function of the actor and the
value function of the critic. Input to the actor model is a
1-dimensional history vector containing the last 20 actions
(policies) that the meta-optimizer has tested. Input to the
critic model is a history vector plus a cumulative reward
averaged across benchmarks, computed using an Area under
the Curve (AUC) function.
Automating Reinforcement Learning Architecture Design for Code Optimization CC ’22, April 02–03, 2022, Seoul, South Korea
Client RL searching strategy. At each of the 𝑛 trials, the
meta-optimizer obtains an action (i.e., a client policy architec-
ture to test) from the actor model. The meta-optimizer uses
the user-provided measurement function to obtain the ob-
servation, which is then used to compute the current reward.
The current reward and the environment state is passed to
the critic model to update its value function. The value func-
tion of the critic also estimates the future reward which is
given to the actor to update its policy network. By default,
the meta-optimizer runs each client RL for 50 exploration
steps during a trial to allow it to converge before taking the
observation. We use the 20 most recently chosen policy ar-
chitectures as the state. We then measure the area under the
reward curve of the recently chosen 20 policies to compute
the current reward. After the meta-optimizer performs 𝑛
trails on training benchmarks, we check the latest 20 actions
chosen by the actor. We then use the most frequently cho-
sen policy architecture as the outcome of policy search. Our
intuition is that the actor would be more efficient in picking
good policy architectures towards the end of the search and
would choose the optimal policy as the action more often.
This search process also implicitly model the learning time
of a candidate client RL. A client RL is chosen because it can
learn quickly to give good results on training benchmarks
within the given time.
Failure during client RL search. In this work, we did not
observe failure in combining RL components in our case
studies. When using a client RL, failures can happen - the
underlying compiler (driven by RL) may fail due to e.g., in-
valid combinations of compiler flags or compiler bugs. These
are automatically handled by RL which will avoid trying
these options in future iterations. Furthermore, while Super-
Sonic could miss an RL architecture that can be improved
through fine-tuning, we have performed a large-scale search
on our case studies and did not observe this. This issue can
also be mitigated by performing RL-architecture search and
parameter fine-tuning in a single process.
Multi-task learning. If multiple optimization tasks are de-
fined, we then use multi-task learning (MTL) to find an in-
dividual policy for each task given a total budget of 𝑛 trials.
SuperSonic provides a distributed, parallel meta-optimizer,
building on top of an open-source MTL framework [38]. For
each optimization task, it creates an environment and assigns
a meta-optimizer instance to the environment, so that client
RL search for different optimization tasks can be performed
in different, potentially distributed environments. In our eval-
uation, we implement an execution environment in a Docker
container. SuperSonic realizes different environments and
their associated actors as parallel workers that can run on a
single machine or multiple distributed machines. An issue of
using a standard MTL algorithm is that the learner is likely to
give more resources (e.g., #trials, time or machines) to tasks
with higher rewards, leading to unfair resource allocation
Table 1. Example tunable parameters
Algorithms Parameters
Common param. Batch size for workers; Train batch size; Mini-
batch size; learning rate
MCTS Dirichlet noise and epsilon; Puct coefficient; Sim-
ulation times; Loss temperature
PPO Entropy coefficient; Adam optimizer step size;
GAE estimator parameter; Policy ratio clipping
DQN #atoms; Discrete supports; Adam epsilon; Clip
gradients
Q-Learning Behavior Cloning Pretraining numbers; Q-
Learning loss temperature; Lagrangian threshold;
Min Q weight multiplier
among tasks. To address this issue, we use a regularization
mechanism, similar to [84], by adding a normalization layer
to the actor and the critic network.
3.4 Client RL Parameter Tuning
During the client RL search stage, SuperSonic uses the de-
fault hyperparameter and pre-trained models. After a client
RL architecture is chosen, the SuperSonic meta-optimizer
uses training benchmarks to fine-tune a set of common and
algorithm-specific hyperparameters. Each SuperSonic built-
in DNN model also has a standard training API. Therefore,
the meta-optimizer also uses the measurements and obser-
vations generated during parameter tuning to fine-tune the
relevant DNN models, e.g. for state representation. Table
1 gives some of the example hyperparameters supported
by SuperSonic. We note that the user does not need to ex-
plicitly supply these parameters because they are known
to SuperSonic. Our parameter tuning method is a parallel
population-based training (PBT) algorithm [41] from RLlib.
Like client RL search, the user can also specify how much
trials can be spent on parameter tuning. Once the budget is
used up, the best-found parameter setting is returned. Our
evaluation applies cross-validation to ensure the tuned RL is
always tested on new, previously unseen benchmarks.
3.5 Client RL Deployment
Finally, the tuned client RL is saved as serialized objects.
The chosen hyperparameters and action space are stored
in JSON files. SuperSonic provides APIs to load and reuse
the stored objects to optimize any new program. To apply a
tuned RL, SuperSonic creates a session to apply a standard
RL loop to optimize the input program by using the chosen
RL exploration algorithms to select an action for a given
state. For example, the state could be a vector recording the
last 𝑛 compiler options added into the compiler flags or a
DNN (see also Section 5.5.3).
3.6 Measurement Engine
For a given optimization option, the measurement engine in-
vokes the user-supplied run function to compile and execute
the program in the target environment. The user function
CC ’22, April 02–03, 2022, Seoul, South Korea H. Wang, Z. Tang, C. Zhang, J. Zhao, C. Cummins, H. Leather, and Z. Wang
Table 2. Case studies in our evaluation
Use cases #Bench. Competing Methods Search space
C1: Optimizating image
pipeline
10 Halide master [ 62],
auto-schedulers [ 3],
HalideRL [ 68], Open-
Tuner
73 ∗27 ∼73 ∗37
C2: Neural network
code optimization
5 AutoTVM [ 19],
Chameleon[5], Open-
Tuner
5 ∗103 ∼20 ∗103
C3: Code size reduction 43 CompilerGym [ 23],
OpenTuner
12360 ∼123150
C4: Superoptimization 40 STOKE [ 76], Open-
Tuner [9]
9100,000 ∼916,000,000
Table 3. Candidate state functions and reward functions
Case Studies: 1 2 3 4
Word2Vec [58]
Doc2vec [49] ✓ ✓
CodeBert [29]
Manual features (e.g. LLVM IR
representation from [40])
✓
Action History
State func.
Hash of Action History ✓
Relative measure (e.g. speedup,
code size reduction ratio)
✓ ✓
Reward func. function output (e.g. tanh()) ✓ ✓
reports the result for each execution, which is stored in a
result database implemented using SQLLite. The database
also holds information obtained during the search, including
the action history, reward, and execution outputs for each
benchmark. The client RL gathers the result of an action by
querying the result database. Decoupling the RL exploration
and measurement allows the parallel execution of measure-
ment and the RL agent, possibly across different machines.
Parallel execution can reduce the measurement cost, which
often dominates the auto-tuning process.
4 Experimental Setup
We evaluate our approach by applying it to four code opti-
mization tasks and compare it against eight tuning methods,
including hand-tuned RL solutions. Table 2 summarizes our
evaluation setup, including the search space size and compet-
ing methods for each case study. We note that the overhead
of RL is dominated by gathering feedback from the environ-
ment through, e.g., compiling and executing the program.
While case studies 3 and 4 have a larger search space than
case studies 1 and 2, obtaining feedback incurs lower over-
head in case studies 3 and 4 compared to 1 and 2, leading to
an overall faster search time for case studies 3 and 4.
4.1 Case Study 1: Optimizing Image Pipelines
The problem. This task aims to improve the optimization
heuristic of the Halide compiler framework for image pro-
cessing [69].A Halide program separates the expression of
the computation kernels and the application processing pipeline
from the pipeline’s schedule. Here, the schedule defines the
order of execution and placement of data on the hardware.
The goal of this task is to automatically synthesize schedules
to minimize the execution time of the benchmark.
Methodology. This task builds upon Halide version 10. Our
evaluation uses ten Halide applications that have been heav-
ily tested on the Halide compiler. We measure the execu-
tion time of each benchmark when processing three image
datasets provided by the Halide benchmark suite. The bench-
marks are optimized to run on a multi-core CPU.
Competing methods. We compare SuperSonic against
four prior methods designed for optimizing Halide sched-
ules. These include two Halide built-in auto-scheduling algo-
rithms (Halide master [62] and auto-scheduler [3]), a hand-
tuned RL method (HalideRL) [68], and OpenTuner. We show
speedups over Halide master.
Actions. Each Halide program comes with a scheduling
template that defines an 𝑛 stages schedule. We can apply
optimizations like loop tiling and vectorization to each stage.
We apply four actions to construct a𝑛-stage scheduling se-
quence. These include adding or removing an optimization
to the stage and decreasing or increasing the value (by one)
of an enabled parameterized option.
4.2 Case Study 2: Neural Network Code Generation
The problem. This task targets DNN back-end code genera-
tion to find a good schedule. e.g., instruction orders and data
placement, to reduce execution time on a multi-core CPU.
Methodology.This study is conducted within the TVM com-
piler v 0.8 [18]. We use 5 CNN kernels where their schedule
optimization space is defined by the TVM developer.
Competing methods. We compare our approach against
four TVM built-in tuning strategies, including random search,
genetic algorithms, grid-based search and XGBoost-based
search. In addition to these, we also compare our approach
to OpenTuner and Chameleon [ 5] - a recently proposed,
hand-tuned RL method designed for TVM. We show the im-
provement over the TVM compiler (TVMC) without schedule
optimization.
Actions. Each TVM benchmark comes with a schedule tem-
plate that defines a set of tuning knobs like loop tiling pa-
rameters. We consider four actions in this task: adding or
removing a knob to the schedule sequence, and decreasing or
increasing the parameter value (by one) of a parameterized
knob in the optimization sequence. The number of tuning
configurations vary across benchmarks (Table 2).
4.3 Case Study 3: Code Size Reduction
The problem. This task is concerned with determining the
LLVM passes and their order to minimize the code size.
Methodology. Following the setup of CompilerGym, we
compute the code size reduction by measuring the ratio
Automating Reinforcement Learning Architecture Design for Code Optimization CC ’22, April 02–03, 2022, Seoul, South Korea
of LLVM IR instruction count reduction over the LLVM -
Oz code size optimization option. This metric is platform-
independent and deterministic. Note that the IR instruction
count strongly correlates to the binary size - fewer instruc-
tions typically lead to a smaller binary. In this evaluation,
we use 43 benchmarks: 23 from the CBench suite [1] and 20
single-source benchmarks from the LLVM test suite [2].
Competing methods. We compare our approach against
OpenTuner and the Greedy and Random search strategies
for code size reduction implemented by CompilerGym. The
CompilerGym Greedy algorithm has a threshold, 𝑒, for con-
trolling how often the algorithm switches between random
and greedy searches, with 𝑒 = 0 for a solely greedy strat-
egy and 𝑒 = 1 for a purely random algorithm. We set 𝑒 to
0.1, which produces comparable results as CompilerGym
developers reported on their platforms.
Actions. We consider all the 123 semantics-preserving passes
of LLVM. The RL agent determines which pass to be added
into or removed from the current compiler pass sequence.
Note that the length of the compiler pass sequence is un-
bounded. An LLVM pass can appear multiple times in the
pass sequence and be inserted before or after any pass.
4.4 Case Study 4: Superoptimization
The problem. This classical compiler optimization task finds
a valid code sequence to maximize the performance of a loop-
free sequence of instructions [43, 55]. Superoptimization is
an expensive optimization technique as the number of pos-
sible configurations grows exponentially as the instruction
count to be optimized increases.
Methodology. In this task, we apply SuperSonic to find a
client RL for STOKE, the state-of-the-art superoptimizer [76].
Given a set of test cases consisting of input-output pairs and a
subset of x86-64 instructions, STOKE synthesizes a program
(at the assembly code level) that uses these instructions and
agrees with the test cases. We use all the 25 benchmarks from
the STOKE Hacker dataset. Additionally, we also extract 15
loop-free and frequently-executed functions from SPEC CPU
2017 and the LLVM test suite, 10 from SPEC and 5 from the
LLVM test suite. The seed input to the performance tuner
is the assembly code generated by compiling the C code
with the -O0 compiler option. We use STOKE’s mutation
engine to modify the target instructions and its equivalent
testing method to verify if the transformed code satisfies the
test cases. We also manually verify the correctness of the
best-performing version found by each method.
Competing methods. We compareSuperSonic to the Markov
chain Monte Carlo (MCMC) based STOKE search technique
and OpenTuner. We test all schemes on LLVM and GCC and
use -O3 as the baseline. As noted in the STOKE document, the
STOKE implementation is not mature enough to improve -O3
code. However, as we will show later, SuperSonic can de-
liver noticeable improvement over -O3 on certain test cases,
demonstrating the potential of auto-tuning techniques.
Actions. We consider all the instruction-level transforma-
tions supported by STOKE. These include replacing the op-
code and operand of an instruction as well as inserting, re-
placing and swapping instructions.
4.5 Client RL Architecture Search Space
We consider all the SuperSonic-supported RL algorithms
when searching the client RL. SuperSonic chooses to use
PPO for code size reduction and MCTS for the other three
tasks. Table 3 lists the state and reward functions considered
in each task, where we highlight the SuperSonic chosen
function using a check mark. As can be seen from the ta-
ble, no RL algorithm dominates our case studies - the best
algorithm depends on the optimization task. However, the
Supersonic-chosen RL algorithm generalizes well to input
test programs of an optimization task.
4.6 Hardware and Software Platforms
For case studies 1, 2 and 4, we use two multi-core servers to
evaluate the resulting code. The first server has 2x 26-core
Intel Xeon 8179M CPU running at 2.40GHz, and the second
server has 2x 32-core AMD EPYC 7532 CPU at 2.4GHz. Both
servers have 128GB of RAM and run Ubuntu 20.04 with Linux
kernel v5.4. In our evaluation, we run the SuperSonic meta-
optimizer on the AMD server and use the chosen client RL
to optimize the target task on both machines. We run the rel-
evant deep learning models on an NVIDIA RTX 2080 Ti GPU.
We use LLVM v10 as the backend compiler to generate the
executable binary in our evaluation. For superoptimizaton,
we also test our approach on GCC v11.2.
4.7 Performance Report
Cross-validation. We use 3-fold cross-validation to evaluate
our approach. This means we partition the benchmarks into
three groups (folds). We perform client RL search and tuning
on two folds of the benchmarks and then test the tuned RL
architecture on the remaining benchmarks. We repeat this
procedure three times to test each of the three folds in turn.
Unless stated otherwise, we exclude the time spent on client
RL search and tuning from the overhead of deployment-
time performance optimization, because client RL search and
tuning is a one-off cost performed offline and the tuned RL
architecture can be applied to many new programs without
incurring this tuning-search overhead. However, we give the
same amount of search time/iterations for all methods when
optimizing a test program.
Runtime measurement. To measure the runtime of the
resulting binary, we run each benchmark at least 100 times
on an unloaded machine. For each benchmark, we also com-
pute the 95% confidence interval bound and increase the
CC ’22, April 02–03, 2022, Seoul, South Korea H. Wang, Z. Tang, C. Zhang, J. Zhao, C. Cummins, H. Leather, and Z. Wang
      Speedup over
      Halide master
Opentuner HalideRL Auto-scheduler SuperSonic
6
12
1 5 10 1 5 10
Tuning time (hours)
0.8
1.3
1.8
2.3
AMD
Intel
(a)
      Speedup over
      Halide master
Opentuner HalideRL Auto-scheduler SuperSonic
6
12
3.6k 18k 36k 3.6k 18k 36k
Tuning iterations
0.8
1.3
1.8
2.3
AMD
Intel
(b)
Figure 3. Performance to expert-tuned Halide schedules
under different search time (a) and iteration (b) constraints.
SuperSonic gives the overall best performance than other
auto-tuning methods.
number of profiling runs if the interval is greater than 2%.
We report the geometric mean across runs. We also show the
performance variances across benchmarks, compilers and
cross-validation settings as min-max bars on the diagram.
Client RL search. In our evaluation, we apply MTL to per-
form RL search for all four optimization tasks simultaneously
on a single server, with a total search budget of 100 trials.
5 Experimental Results
In this section, we first present the case study results, find-
ing that SuperSonic outperforms hand-crafted strategies
in each task. We then provide an analysis of SuperSonic’s
working mechanisms, showing SuperSonic can accelerate
the deployment-time search by 1.75x. Note that the tuning
time is proportional to the number of tuning iterations, but
the tuning time per iteration can vary between evaluated
methods. Specifically, the SuperSonic-chosen client RL can
perform, on average, 1, 3, 24 and 10 search iterations per
second for case studies 1, 2, 3 and 4, respectively, on our
evaluation platforms.
5.1 Case Study 1: Optimizing Image Pipelines
Figure 3 reports speedup over the Halide master scheduler
[62]. SuperSonic gives noticeable performance improve-
ment on the AMD platform. It also manifests larger advan-
tages when the search time is limited. On certain benchmarks,
SuperSonic is able to gives over 11x speedup with a cor-
rectly optimized code. The tree-based auto-scheduler gives a
high speedup of over 10x for a single benchmark, but it has
      Speedup over TVMC
Random
Genetic algorithm
Grid search
XGBoost
Opentuner
Chameleon
SuperSonic
20 40 60 20 40 60
Tuning time (minutes)
1.1
1.2
1.3
1.4
1.5
1.6
AMD Intel
(a)
      Speedup over TVMC
Random
Genetic algorithm
Grid search
XGBoost
Opentuner
Chameleon
SuperSonic
1.5
1.75
3k 12k 16k 3k 12k 16k
Tuning iterations
1.1
1.2
1.3
1.4
AMD Intel
(b)
Figure 4. Speedup for DNN code generation under different
search time (a) and iteration (b) constraints. The min-max
bar shows the range across benchmarks. SuperSonic out-
performs all competing methods on both platforms.
a lower mean performance improvement across benchmarks
compared to SuperSonic. On average, SuperSonic delivers
a 1.5x improvement over HalideRL, a manually tuned RL
strategy. Note that HalideRL uses PPO as the exploration
algorithm and the sequence of already applied schedules as
the state function. By contrast, SuperSonic determines that
for this task, using MCTS as the exploration algorithm and a
hash function of the applied schedules as the state function
is better than HalideRL. MCTS compares complete sched-
ules through simulations and looks ahead before making
intermediate scheduling optimizations, leading to a better re-
sult. Overall, SuperSonic outperforms all alternative search
techniques on all but two benchmarks on both platforms.
5.2 Case Study 2: Neural Network Code Generation
Figure 4 reports the the performance improvement over the
default schedules. RL based methods (Chameleon andSuper-
Sonic) can generate better code than TVM’s evolutionary
or predictive modeling based search techniques. While ran-
dom and grid-based search can significantly improve one
benchmark (the top point of their min-max), their perfor-
mance is not robust and can give poor performance for other
benchmarks. By contrast, SuperSonic delivers more robust
performance by giving no slowdown on the AMD platform
and only minor slowdown over XGBoost on two benchmarks
on the Intel platform. SuperSonic improves Chameleon, the
second best-performing method, by up to 1.22x, improving
the default schedules by up to 1.74x.
Automating Reinforcement Learning Architecture Design for Code Optimization CC ’22, April 02–03, 2022, Seoul, South Korea
Opentuner
C.Gym.Greedy
C.Gym.Random
SuperSonic
1.1
1.6
50 100 200 3600
Tuning Time (seconds)
0.9
0.95
1
1.05    Reduction over LLVM -Oz
(a)
Opentuner
C.Gym.Greedy
C.Gym.Random
SuperSonic
1.1
1.6
12K 24K 48K 86K
Tuning iterations
0.9
0.95
1
1.05    Reduction over LLVM -Oz (b)
Figure 5. Code size reduction over LLVM -Oz under different
search time (a) and iteration (b) constraints. SuperSonic
outperforms alternative methods.
Table 4. Important compiler passes for code size reduction
and how often a pass is chosen by a search method.
Opentuner C.Gym.Greedy C.Gym.Random SuperSonic
simplifycfg 0.84% 3.29% 1.23% 10.78%
early-cse-
memssa
0.78% 0.18% 0.92% 6.80%
gvn 0.69% 1.83% 0.97% 5.98%
instcombine 0.75% 0.12% 1.17% 8.99%
newgvn 0.74% 2.90% 1.02% 7.66%
others 96.21% 91.67% 94.69% 59.79%
5.3 Case Study 3: Code Size Reduction
Figure 5 reports the code size reduction conducted on the
Intel platform using LLVM. We note that an average code
size reduction of 4% is considered to be significant [ 72–
74].SuperSonic improves -Oz for all but two test bench-
marks. It delivers the best reduction for 90% of the test bench-
marks. For those benchmarks where SuperSonic does not
deliver the best reduction, the difference between Super-
Sonic and the best-performing method is small, less than
1%. On average, SuperSonic gives the highest mean code
size reduction, improving LLVM -Oz by up to 1.57x.
Table 4 lists the top-5 most frequently-appeared LLVM
passes in the compiler pass list that gives the best code size
reduction. Because these passes are often included in a com-
piler sequence that offers a high code size reduction, they
are likely to be important for reducing code size. The table
also shows how often a search method chooses a pass as an
action when optimizing a benchmark. Compared to other
methods2, SuperSonic picks an important pass more often
during the search, suggesting that it learns the importance
of optimization passes.
5.4 Case Study 4: Superoptimization
Figure 6 shows the superoptimization results using LLVM
and GCC. SuperSonic outperforms Stoke and OpenTuner
on most of the test benchmarks with a higher mean speedup.
Increasing the tuning iterations (and the search time) during
2In theory, with a sufficiently larger number of samples, each pass will have
a 1/123 chance to be chosen by C.Gym.Random in Table 4.
  Speedup over 
  LLVM/GCC -O3
Opentuner Stoke SuperSonic
1.1
1.3
1000 1500 2000 1000 1500 2000
Tuning time (seconds)
0.85
0.9
0.95
1
AMD Intel
(a)
  Speedup over 
  LLVM/GCC -O3
Opentuner Stoke SuperSonic
1.1
1.3
6k 9k 12k 6k 9k 12k
Tuning iterations
0.85
0.9
0.95
1
AMD Intel (b)
Figure 6. Speedup over LLVM/GCC -O3 for superoptimiza-
tion under different search time (a) and iterations (b).
deployment can improve the search performance because
it allows the search algorithm to explore the optimization
space better and uses feedback to improve its search strategy.
Although the STOKE developers note that their mutation
engine is not matured enough to outperform LLVM/GCC -O3,
we demonstrate that RL can deliver noticeable improvement
by better exploring the optimization space (up to 1.34x).
Figure 7 shows a kernel from STOKE Hacker benchmark
dataset, compiled by LLVM -O3, and the best-performing
version found by SuperSonic. The SuperSonic code for the
_Z3p23i kernel is 18 lines shorter, 1.2x faster than -O3. This
is one of the many examples where the Supersonic-driven
superoptimization generates faster code over -O3.
5.5 Further Analysis
5.5.1 Compare to Other Client RL Search Strategies.
This experiment comparesSuperSonic’s deep RL-based meta-
optimizer against four widely used parameter search tech-
niques: grid search, simulated annealing, random search, and
genetic algorithms. We set the number of search iterations to
100 for all search algorithms. We also vary the search space
by adding more candidate RL components, where the num-
ber of RL component combinations vary between 150 and
800. For a chosen RL client, we follow the same parameter
fine-tuning process to fine-tune the selected RL components
(Section 3.4). Figure 8 compares the performance of the client
RL chosen by different search algorithms during the RL client
search stage. As can be seen from the diagram, all client RL
give an averaged improvement over the baseline. Super-
Sonic finds a better client RL during the limited client RL
search budget, leading to overall better performance across
search space sizes and case studies. We note that the limited
client RL search budget restricts the performance of finding
good client RL with a large search space (e.g., when the num-
ber of RL component combinations is 800). Increasing the
search budget will allow the meta-optimizer to find a better
client RL to improve the resulting performance.
5.5.2 Deployment-stage Search Time Relative to the
Best-performing Alternative. Table 5 shows the rela-
tive search time and iteration count required by Super-
Sonic-tuned client RL to exceed the results given by the
best-performing competitive scheme. The table shows the
minimum, average, and maximum search overhead across
CC ’22, April 02–03, 2022, Seoul, South Korea H. Wang, Z. Tang, C. Zhang, J. Zhao, C. Cummins, H. Leather, and Z. Wang
1 . L_4004e0_LLVM_O3 : . L_4004e0_SuperSonic :
2 movl $0x60106f ,% eax movl $0x601191 ,%
edi
3 movl $0x601191 ,% edi movq $0x400990 ,%
rcx
4 movl $0x400700 ,% edi movl $0x400700 ,%
edi
5 pushq % rax pushq % rax
6 nopl 0x0 (% rax ,% rax ,1) nopl 0x0 (% rax ,% rax
,1)
7 ... ...
8 callq 4008 f0 < _Z3p23i > callq 4008 f0 <
_Z3p23i >
9
10 . _Z3p23i : . _Z3p23i :
11 movl %edi ,% eax popcntl %edi ,% ecx
12 shrl % eax movq %rcx ,% rax
13 andl $0x55555555 ,% eax retq
14 subl %eax ,% edi
15 movl %edi ,% eax
16 andl $0x33333333 ,% eax
17 shrl $0x2 ,% edi
18 andl $0x33333333 ,% edi
19 addl %eax ,% edi
20 movl %edi ,% eax
21 shrl $0x4 ,% eax
22 addl %edi ,% eax
23 andl $0xf0f0f0f ,% eax
24 movl %eax ,% ecx
25 shrl $0x8 ,% ecx
26 addl %eax ,% ecx
27 movl %ecx ,% eax
28 shrl $0x10 ,% eax
29 addl %ecx ,% eax
30 movzbl %al ,% eax
31 retq
32 nopl 0x0 (% rax ,% rax ,1)
Figure 7. SuperSonic finds a better program (right) over
LLVM -O3 (left) for a STOKE Hacker kernel (_Z3p23i) that
counts the number of bits, by using thepopcntl instruction.
Table 5. Search overhead required by SuperSonic to exceed
the performance given the best-performing alternative
Use
cases
% of search time (& raw numbers) % of iterations (& raw numbers)
MIN GeoMean MAX MIN GeoMean MAX
Case
study 1
3.6
(21 mins)
39.6
(4 hours)
72.1
(6 hours)
3.9
(1,425)
44.0
(15,821)
75.3
(27,141)
Case
study 2
1.0
(1 min)
39.1
(24 mins)
74.1
(45 mins)
4.2
(885)
32.3
(3,885)
68.8
(8,256)
Case
study 3
1.5
(3 sec)
31.0
(61 sec)
61.1
(2 mins)
2.1
(738)
29.7
(10,714)
83.5
(30,068)
Case
study 4
1.1
(22 sec)
32.8
(11 mins)
42.3
(14 mins)
1.2
(1,478)
36.5
(43,745)
46.9
(56,387)
test benchmarks. On average, the RL architecture found by
SuperSonic converges faster, requires less than 39.6% of the
search time used by the best-performing alternative search
algorithm to deliver better results. This means the RL archi-
tecture chosen by SuperSonic can start delivering a better
optimized code with, on average, 1.75x less search time (up
to 100x) compared the best-performing alternative tuning al-
gorithm that runs longer. While search methods like genetic
algorithms have no upfront training cost, they must search
each time afresh. SuperSonic performs a one-off offline RL
search and tuning, but the tuned client RL significantly re-
duces the search cost for any new programs after that.
5.5.3 Chosen RL Components. Table 3 shows the client
RL components chosen by SuperSonic varies from one task
to another. We also observe that the optimal RL found by
SuperSonic is consistent across cross-validation runs for a
given task. This means the client RL architecture can general-
ize across inputs of a given task. We notice that using MCTS
with DNNs for state representation gives good performance
for three out of the four tasks, but it is less effective for code
size optimization compared to PPO. Using MCTS for code
reduction gives an average reduction of 0.6% instead of 6.5%
delivered by PPO. This perhaps due to a large number of
discrete actions (i.e., the compiler passes) in the optimization
space, which requires more search time to learn a good value
function to efficiently guide the MCTS simulation.
6 Discussions
We have showcased that deep RL can be highly effective in
performance tuning, but this depends on having a suitable RL
architecture. Deep RL can benefit from knowledge learned
from other inputs (or programs). Some of this knowledge can
be transferable from one program to another. For example,
the client RL algorithm learns applying -simplifycfg (Ta-
ble 4) is always beneficial for code size reduction from train-
ing benchmarks. The RL agent thus increases Evolutionary-
based search algorithms cannot use prior knowledge and
have to start from scratch for every new program [40].
Our experiments show that evolutionary-based techniques
can outperform RL on certain test cases (Section 5.1). Be-
cause evolutionary algorithms often have a lower runtime
overhead than deep RL methods, it is interesting to see if
both methods can be combined to explore more optimiza-
tion points of the search space. For example, one can first
employ a deep RL to identify the most promising areas in
the search space and then use low-cost evolutionary algo-
rithms to perform finer-grained searches on the identified
regions. Another example is to use evolutionary algorithms
to combine and dynamically choose between multiple state
or reward functions to improve the robustness of the system.
We found the measurement cost often dominates the search
time of an autotuner. One way of reducing the measurement
overhead is to use active learning to reduce the number of
runs when we are confident that the results are statistical
sound [63, 64]. Another strategy is to use a predictive model
to estimate outcome of runtime measurements to reduce
environment interactions.
SuperSonic searches the client RL by trying different
combinations of RL components. An interesting approach
Automating Reinforcement Learning Architecture Design for Code Optimization CC ’22, April 02–03, 2022, Seoul, South Korea
 Speedup over
 Halide master
Grid Search
Random Search
Simulated Annealing
Genetic Algorithm
SuperSonic
6
12
150 400 800
Client RL search space
0.8
1.3
1.8
2.3
(a) Optimizing Image Pipelines
   Speedup 
 over TVMC
Grid Search
Random Search
Simulated Annealing
Genetic Algorithm
SuperSonic
1.5
1.55
150 400 800
Client RL search space
1.1
1.2
1.3
1.4 (b) Neural Network Code Generation
Grid Search
Random Search
Simulated Annealing
Genetic Algorithm
SuperSonic
1.1
1.6
150 400 800
Client RL search space
0.9
0.95
1
1.05
Reduction over 
    LLVM -Oz (c) Code size reduction
  Speedup over 
  LLVM/GCC -O3
Grid Search
Random Search
Simulated Annealing
Genetic Algorithm
SuperSonic
1.1
1.3
150 400 800
Client RL search space
0.85
0.9
0.95
1 (d) Superoptimization
Figure 8. Comparing the performance of a client RL chosen by different search algorithms during the client RL search stage.
We vary the number of client RL combinations by varying the number of candidate RL components. The SuperSonic chosen
client RL gives the best overall performance during deployment.
would be to combine online search and predictive modeling,
by using the profiling data collected from earlier runs to
build a predictive model to directly predict the promising RL
component combinations to reduce the search overhead.
Finally, RL often employs machine-learning (ML) models
to estimate the reward or to represent the environment state.
However, ML is brittle, and small changes in data distribution
during deployment can result in incorrect predictions. There-
fore, techniques for detecting when the model’s estimation
can be trusted is useful [71, 91].
7 Related Work
Auto-tuning techniques have been widely used to reduce
expert involvement in performance optimization tasks [88].
Early works like ATLAS [89] and FFTW [31] demonstrate
the potential for searching domain-specific optimization pa-
rameters like the loop tile size. In iterative compilation, prior
works have exploited the use of evolutionary algorithms
like the genetic algorithm [7, 8] and other search techniques
built upon Bayesian optimization [ 11, 17, 42, 56, 75] and
predictive modeling [4, 10, 15, 48, 67]. Recent works also ap-
ply auto-tuning techniques to optimize code generation for
deep neural networks [18, 34, 61, 92], image processing ap-
plications [69], and runtime tuning of operating system and
processor parameters [20]. Our work is among these efforts
in applying auto-tuning techniques for code optimization.
In recent years, RL (in particular deep RL) has demon-
strated impressive results in domains like game-playing
[66, 78] and robotics [6]. It has also been employed for var-
ious performance optimization tasks, including compiler
phase ordering [36], loop optimization [13], choosing vector-
ization parameters [35], task scheduling [54] and memory
placement [45]. These prior works rely on hand-tuned poli-
cies or feature engineering to derive a good search strategy
for the given application domain. Given the large number
and diversity of workloads to be optimized, approaches like
ours for automating RL architecture tuning are attractive.
CompilerGym [23] is an machine-learning platform for
compiler optimization. It provides an OpenAI Gym-like pro-
gramming environment [65]. SuperSonic utilizes Compi-
lerGym’s API for problem definition, but complements to
CompilerGym by providing ways to automate RL component
searching and tuning. We plan to integrate SuperSonic into
the mainstream release of CompilerGym.
In a relevant research area, neural architecture search
[28] aims to find the right neural network structure and
parameters to reduce the model size and execution time or
to improve accuracy [ 82, 93]. Our work draws aspiration
from these past foundations to find a good RL architecture
for performance optimization.
8 Conclusions
We have presented SuperSonic, a framework for building
RL-based performance tuners. SuperSonic provides the ca-
pability to automate the process of designing and tuning
RL algorithm structures. We evaluate SuperSonic by ap-
plying it to four different optimization tasks. Experimental
results show that the RL architecture found by SuperSonic
delivers better overall performance than alternative search
techniques, including hand-tuned RL strategies.
SuperSonic supports mainstream and emerging RL pro-
gramming environments like RLlib and CompilerGym. It is
designed to provide customizable interfaces to allow devel-
opers to introduce new algorithms and methods to be used in
the RL policy architecture. As the community provides more
models and methods, SuperSonic will be able to explore a
more comprehensive policy search space. As a result, there
will be less of a need to create domain-specific methods for
each project. In the long-term, we hope that SuperSonic
will work out-of-the-box for most performance developers
once they have defined their optimization tasks. We hope the
findings of this paper and the release of SuperSonic can en-
courage the adoption of RL in many other code optimization
problems.
Acknowledgments
This work was supported in part by the National Science
Foundation of China (NSFC) under grant agreements 61972314,
61872294 and 61902170, the International Cooperation Project
of Shaanxi Province under grant agreements 2021KW-04,
2020KWZ-013 and 2021KW-15, and a Meta (Facebook) re-
search award. For any correspondence, please contact Zhany-
ong Tang (Email: zytang@nwu.edu.cn) and Zheng Wang
(Email: z.wang5@leeds.ac.uk).
CC ’22, April 02–03, 2022, Seoul, South Korea H. Wang, Z. Tang, C. Zhang, J. Zhao, C. Cummins, H. Leather, and Z. Wang
Appendix: Artifacts Evaluation Instructions
A.1 Overview
Our research artefact enables the reproduction of our ap-
proach and figures from our experimental results. This doc-
ument consists of instructions for performing AE on pre-
configured notebooks (Section A.2) or Python scripts within
a docker image (Section A.3).
Main Results
Our AE enables a reduced-size evaluation for the main results
of our work, i.e., Figures 3-6 in the paper. The results compare
the performance of the client RL found by our tool against
prior search-based techniques. We also provide a small-scale
experiment to showcase how the developed techniques can
be used to search for a client RL architecture.
Artifact Check-list (Meta-information)
•Compilation: LLVM 9.0, gcc 7.5 and gcc 4.9 included
•Experiments: The experiments can be run with the
included scripts.
•How much disk space required (approximately)?:
40 GB
•How much time is needed to prepare workflow
(approximately)? : Several hours for a small-scale
evaluation and one week for a full-scale evaluation
•Publicly available? : Yes, code and data are available
at: https://github.com/HuantWang/SUPERSONIC
•Code licenses: CC-BY-4.0 License
•Archived: 10.5281/zenodo.6313660
A.2 Instructions for Interactive Notebooks
For convenience, we have provided a pre-configured Python
Jupyter Notebook within a pre-configured docker image
- see https://github.com/HuantWang/SUPERSONIC/blob/
master/AE.md for how to access the docker image.
Experiment Workflow
1. Access the Jupyter Notebook using the method de-
scribed above.
2. From the Jupyter landing page, select the checkbox
next to the notebook, i.e., AE_Intel.ipynb; then click
“Duplicate”.
3. Click the name of the newly created Jupyter Notebook,
e.g. AE_Intel-Copy1.ipynb.
4. Select “Kernel”> “Restart & Clear Out” from the menu
to reset the results.
5. Repeatedly press the play button (the tooltip is “run
cell, select below”) to step through each cell of the
notebook. Alternatively, select each cell in turn and
use “Cell” > “Run Cell” from the menu to run specific
cells. Note that some cells depend on previous cells
being executed. If any errors occur ensure all previous
cells have been executed.
Evaluation and Expected Results
Code cells within the Jupyter Notebook display their output
inline. Note that some cells can take a few minutes to hours
to complete; please wait for the results until step to the next
cell. High load can also lead to a long wait for results. This
may occur if multiple reviewers are simultaneously trying
to generate results.
Customisation
The experiments are fully customisable, the code provided in
the Jupyter Notebook can be edited on the spot. Simply type
your changes into the code blocks and re-run using “Cell” >
“Run Cells” from the menu.
A.3 Instructions for Docker Image
For a step-by-step instruction to replicate our results using
a docker image on your machine locally without the interac-
tive notebook, please refer to our GitHub repository (https://
github.com/HuantWang/SUPERSONIC/blob/master/AE.md).
A.4 Evaluation and Expected Result
The supplied notebook or scripts will automatically produce
the results or diagrams after the experiments. The AE also
includes the data used in the paper submission. The results
may be different if the experiment is performed on hardware
that differs from the ones used in the paper.
A.5 Experiment Customization
Our evaluation scripts contain customizable parameters to
change things like the number of search iterations and the
training and testing datasets used. Please follow the instruc-
tions given athttps://github.com/HuantWang/SUPERSONIC/
blob/master/AE.md.
Automating Reinforcement Learning Architecture Design for Code Optimization CC ’22, April 02–03, 2022, Seoul, South Korea
References
[1] [n. d.]. Collective Benchmark. https://ctuning.org/wiki/index.php/
CTools:CBench.
[2] [n. d.]. llvm-test-suite. https://github.com/llvm/llvm-test-suite.
[3] Andrew Adams, Karima Ma, Luke Anderson, Riyadh Baghdadi, Tzu-
Mao Li, Michaël Gharbi, Benoit Steiner, Steven Johnson, Kayvon Fata-
halian, Frédo Durand, et al. 2019. Learning to optimize halide with tree
search and random programs. ACM Transactions on Graphics (TOG)
38, 4 (2019), 1–12.
[4] Felix Agakov, Edwin Bonilla, John Cavazos, Björn Franke, Grigori
Fursin, Michael FP O’Boyle, John Thomson, Marc Toussaint, and
Christopher KI Williams. 2006. Using machine learning to focus itera-
tive optimization. In International Symposium on Code Generation and
Optimization (CGO’06). IEEE, 11–pp.
[5] Byung Hoon Ahn, Prannoy Pilligundla, Amir Yazdanbakhsh, and Hadi
Esmaeilzadeh. 2019. Chameleon: Adaptive Code Optimization for
Expedited Deep Neural Network Compilation, In International Con-
ference on Learning Representations. ICLR’20.
[6] Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin,
Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn
Powell, Raphael Ribas, et al. 2019. Solving rubik’s cube with a robot
hand. arXiv preprint arXiv:1910.07113 (2019).
[7] Lelac Almagor, Keith D Cooper, Alexander Grosul, Timothy J Harvey,
Steven W Reeves, Devika Subramanian, Linda Torczon, and Todd Wa-
terman. 2004. Finding effective compilation sequences. ACM SIGPLAN
Notices 39, 7 (2004), 231–239.
[8] Jason Ansel, Cy Chan, Yee Lok Wong, Marek Olszewski, Qin Zhao,
Alan Edelman, and Saman Amarasinghe. 2009. PetaBricks: A Lan-
guage and Compiler for Algorithmic Choice. In Proceedings of the
30th ACM SIGPLAN Conference on Programming Language Design and
Implementation (PLDI ’09) . 38–49.
[9] Jason Ansel, Shoaib Kamil, Kalyan Veeramachaneni, Jonathan Ragan-
Kelley, Jeffrey Bosboom, Una-May O’Reilly, and Saman Amarasinghe.
2014. Opentuner: An extensible framework for program autotuning. In
Proceedings of the 23rd international conference on Parallel architectures
and compilation . 303–316.
[10] Amir H. Ashouri, Andrea Bignoli, Gianluca Palermo, Cristina Silvano,
Sameer Kulkarni, and John Cavazos. 2017. MiCOMP: Mitigating the
Compiler Phase-Ordering Problem Using Optimization Sub-Sequences
and Machine Learning. ACM Trans. Archit. Code Optim. 14, 3 (2017).
[11] Amir Hossein Ashouri, Giovanni Mariani, Gianluca Palermo, Eunjung
Park, John Cavazos, and Cristina Silvano. 2016. COBAYN: Compiler
Autotuning Framework Using Bayesian Networks. ACM Trans. Archit.
Code Optim. 13, 2, Article 21 (jun 2016), 25 pages. https://doi.org/10.
1145/2928270
[12] Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler.
2018. Neural Code Comprehension: A Learnable Representation
of Code Semantics. In Advances in Neural Information Process-
ing Systems 31 , S. Bengio, H. Wallach, H. Larochelle, K. Grauman,
N. Cesa-Bianchi, and R. Garnett (Eds.). Curran Associates, Inc., 3588–
3600. http://papers.nips.cc/paper/7617-neural-code-comprehension-
a-learnable-representation-of-code-semantics.pdf
[13] Alexander Brauckmann, Andrés Goens, and Jerónimo Castrillón. 2021.
A Reinforcement Learning Environment for Polyhedral Optimizations.
In International Conference on Parallel Architectures and Compilation
Techniques (PACT).
[14] Cameron B. Browne, Edward Powley, Daniel Whitehouse, Simon M.
Lucas, Peter I. Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego
Perez, Spyridon Samothrakis, and Simon Colton. 2012. A Survey of
Monte Carlo Tree Search Methods.IEEE Transactions on Computational
Intelligence and AI in Games 4, 1 (2012), 1–43. https://doi.org/10.1109/
TCIAIG.2012.2186810
[15] John Cavazos, Grigori Fursin, Felix Agakov, Edwin Bonilla, Michael FP
O’Boyle, and Olivier Temam. 2007. Rapidly selecting good compiler
optimizations using performance counters. InInternational Symposium
on Code Generation and Optimization (CGO’07) . IEEE, 185–197.
[16] Dehao Chen, Tipp Moseley, and David Xinliang Li. 2016. AutoFDO:
Automatic feedback-directed optimization for warehouse-scale appli-
cations. In 2016 IEEE/ACM International Symposium on Code Generation
and Optimization (CGO) . IEEE, 12–23.
[17] Junjie Chen, Ningxin Xu, Peiqi Chen, and Hongyu Zhang. 2021.
Efficient Compiler Autotuning via Bayesian Optimization. In 2021
IEEE/ACM 43rd International Conference on Software Engineering (ICSE) .
IEEE, 1198–1209.
[18] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan,
Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze,
et al. 2018. {TVM}: An automated end-to-end optimizing compiler
for deep learning. In 13th {USENIX}Symposium on Operating Systems
Design and Implementation ( {OSDI}18). 578–594.
[19] Tianqi Chen, Lianmin Zheng, Eddie Yan, Ziheng Jiang, Thierry Moreau,
Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. 2018. Learn-
ing to Optimize Tensor Programs. In Proceedings of the 32nd Interna-
tional Conference on Neural Information Processing Systems (Montréal,
Canada) (NIPS’18). 3393–3404.
[20] Concertio. [n. d.]. Concertio: Autonomous Optimization Platform.
[21] Chris Cummins, Pavlos Petoumenos, Zheng Wang, and Hugh Leather.
2017. End-to-end deep learning of optimization heuristics. In 2017
26th International Conference on Parallel Architectures and Compilation
Techniques (PACT). IEEE, 219–232.
[22] Chris Cummins, Pavlos Petoumenos, Zheng Wang, and Hugh Leather.
2017. Synthesizing Benchmarks for Predictive Modeling. In CGO.
IEEE.
[23] Chris Cummins, Bram Wasti, Jiadong Guo, Brandon Cui, Jason Ansel,
Sahir Gomez, Somya Jain, Jia Liu, Olivier Teytaud, Benoit Steiner, Yuan-
dong Tian, and Hugh Leather. 2021. CompilerGym: Robust, Performant
Compiler Optimization Environments for AI Research.
[24] Anderson Faustino da Silva, Bruno Conde Kind, José Wesley de
Souza Magalhães, Jerônimo Nunes Rocha, Breno Campos Ferreira
Guimaraes, and Fernando Magno Quinão Pereira. 2021. ANG-
HABENCH: A Suite with One Million Compilable C Benchmarks for
Code-Size Reduction. In 2021 IEEE/ACM International Symposium on
Code Generation and Optimization (CGO) . IEEE, 378–390.
[25] JW Davidson, Gary S Tyson, DB Whalley, and PA Kulkarni. 2007.
Evaluating heuristic optimization phase order search algorithms. InIn-
ternational Symposium on Code Generation and Optimization (CGO’07) .
IEEE, 157–169.
[26] Christian Delimitrou and Christos Kozyrakis. 2014. Quasar: Resource-
Efficient and QoS-Aware Cluster Management. In 19th Intl. Conference
on Architectural Support for Programming Languages and Operating
Systems (ASPLOS).
[27] Yufei Ding, Jason Ansel, Kalyan Veeramachaneni, Xipeng Shen, Una-
May O’Reilly, and Saman P. Amarasinghe. 2015. Autotuning algo-
rithmic choice for input sensitivity. Proceedings of the 36th ACM
SIGPLAN Conference on Programming Language Design and Implemen-
tation (2015).
[28] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019. Neu-
ral architecture search: A survey. The Journal of Machine Learning
Research 20, 1 (2019), 1997–2017.
[29] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng,
Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming
Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and
Natural Languages. In Findings of the Association for Computational
Linguistics: EMNLP 2020 . Association for Computational Linguistics,
Online, 1536–1547. https://doi.org/10.18653/v1/2020.findings-emnlp.
139
[30] Alvaro Fialho, Luis Da Costa, Marc Schoenauer, and Michele Sebag.
2010. Analyzing bandit-based adaptive operator selection mechanisms.
Annals of Mathematics and Artificial Intelligence 60, 1 (2010), 25–64.
CC ’22, April 02–03, 2022, Seoul, South Korea H. Wang, Z. Tang, C. Zhang, J. Zhao, C. Cummins, H. Leather, and Z. Wang
[31] Matteo Frigo and Steven G Johnson. 2005. The design and implemen-
tation of FFTW3. Proc. IEEE 93, 2 (2005), 216–231.
[32] Dominik Grewe, Zheng Wang, and Michael FP O’Boyle. 2013. Portable
mapping of data parallel programs to opencl for heterogeneous sys-
tems. In Proceedings of the 2013 IEEE/ACM International Symposium on
Code Generation and Optimization (CGO) . IEEE, 1–10.
[33] Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. 2017.
Deep reinforcement learning for robotic manipulation with asynchro-
nous off-policy updates. In 2017 IEEE international conference on robot-
ics and automation (ICRA) . IEEE, 3389–3396.
[34] Hui Guan, Xipeng Shen, and Seung-Hwan Lim. 2019. Wootz: A
Compiler-Based Framework for Fast CNN Pruning via Composability
(PLDI 2019) . Association for Computing Machinery, New York, NY,
USA, 717–730. https://doi.org/10.1145/3314221.3314652
[35] Ameer Haj-Ali, Nesreen K Ahmed, Ted Willke, Yakun Sophia Shao,
Krste Asanovic, and Ion Stoica. 2020. NeuroVectorizer: End-to-end
vectorization with deep reinforcement learning. In Proceedings of the
18th ACM/IEEE International Symposium on Code Generation and Opti-
mization. 242–255.
[36] Ameer Haj-Ali, Qijing (Jenny) Huang, William S. Moses, John Xiang,
Krste Asanovic, John Wawrzynek, and Ion Stoica. 2020. AutoPhase:
Juggling HLS Phase Orderings in Random Forests with Deep Reinforce-
ment Learning. In Proceedings of Machine Learning and Systems 2020,
MLSys, Inderjit S. Dhillon, Dimitris S. Papailiopoulos, and Vivienne
Sze (Eds.).
[37] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina
Precup, and David Meger. 2018. Deep reinforcement learning that
matters. In Proceedings of the AAAI conference on artificial intelligence ,
Vol. 32.
[38] Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki,
Simon Schmitt, and Hado van Hasselt. 2019. Multi-task deep reinforce-
ment learning with popart. In Proceedings of the AAAI Conference on
Artificial Intelligence, Vol. 33. 3796–3803.
[39] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term
memory. Neural computation 9, 8 (1997), 1735–1780.
[40] Qijing Huang, Ameer Haj-Ali, William Moses, John Xiang, Ion Stoica,
Krste Asanovic, and John Wawrzynek. 2020. AutoPhase: Juggling
HLS Phase Orderings in Random Forests with Deep Reinforcement
Learning. arXiv:2003.00671 [cs.DC]
[41] Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czar-
necki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dun-
ning, Karen Simonyan, et al. 2017. Population based training of neural
networks. arXiv preprint arXiv:1711.09846 (2017).
[42] Tarindu Jayatilaka, Hideto Ueno, Giorgis Georgakoudis, EunJung Park,
and Johannes Doerfert. 2021. Towards Compile-Time-Reducing Com-
piler Optimization Selection via Machine Learning . Association for
Computing Machinery, New York, NY, USA. https://doi.org/10.1145/
3458744.3473355
[43] Rajeev Joshi, Greg Nelson, and Keith Randall. 2002. Denali: a goal-
directed superoptimizer. In Proceedings of the ACM SIGPLAN 2002
conference on Programming language design and implementation . 304–
314.
[44] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996.
Reinforcement learning: A survey. Journal of artificial intelligence
research 4 (1996), 237–285.
[45] Shauharda Khadka, Estelle Aflalo, Mattias Marder, Avrech Ben-David,
Santiago Miret, Shie Mannor, Tamir Hazan, Hanlin Tang, and Somdeb
Majumdar. 2021. Optimizing Memory Placement using Evolutionary
Graph Reinforcement Learning. ICLR.
[46] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet
classification with deep convolutional neural networks. Advances in
neural information processing systems 25 (2012), 1097–1105.
[47] Prasad Kulkarni, Stephen Hines, Jason Hiser, David Whalley, Jack
Davidson, and Douglas Jones. 2004. Fast searches for effective opti-
mization phase sequences.ACM SIGPLAN Notices 39, 6 (2004), 171–182.
[48] Sameer Kulkarni and John Cavazos. 2012. Mitigating the compiler
optimization phase-ordering problem using machine learning. In Pro-
ceedings of the ACM international conference on Object oriented pro-
gramming systems languages and applications . 147–162.
[49] Quoc Le and Tomas Mikolov. 2014. Distributed Representations of
Sentences and Documents. In Proceedings of the 31st International
Conference on Machine Learning (Proceedings of Machine Learning
Research, Vol. 32) , Eric P. Xing and Tony Jebara (Eds.). PMLR, Bejing,
China, 1188–1196.
[50] Menghao Li, Minjia Zhang, Chi Wang, and Mingqin Li. 2020. AdaTune:
Adaptive Tensor Program Compilation Made Efficient. Advances in
Neural Information Processing Systems 33 (2020).
[51] Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox,
Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. 2018.
RLlib: Abstractions for distributed reinforcement learning. In Interna-
tional Conference on Machine Learning . PMLR, 3053–3062.
[52] Yang Liu, Wissam M Sid-Lakhdar, Osni Marques, Xinran Zhu, Chang
Meng, James W Demmel, and Xiaoye S Li. 2021. GPTune: multitask
learning for autotuning exascale applications. In Proceedings of the
26th ACM SIGPLAN Symposium on Principles and Practice of Parallel
Programming. 234–246.
[53] Rahim Mammadli, Ali Jannesari, and Felix Wolf. 2020. Static Neural
Compiler Optimization via Deep Reinforcement Learning. In 2020
IEEE/ACM 6th Workshop on the LLVM Compiler Infrastructure in HPC
(LLVM-HPC) and Workshop on Hierarchical Parallelism for Exascale
Computing (HiPar). IEEE, 1–11.
[54] Hongzi Mao, Malte Schwarzkopf, Shaileshh Bojja Venkatakrishnan,
Zili Meng, and Mohammad Alizadeh. 2019. Learning Scheduling Algo-
rithms for Data Processing Clusters. In Proceedings of the ACM Special
Interest Group on Data Communication (SIGCOMM ’19) . 270–288.
[55] Henry Massalin. 1987. Superoptimizer: a look at the smallest program.
ACM SIGARCH Computer Architecture News 15, 5 (1987), 122–126.
[56] Harshitha Menon, Abhinav Bhatele, and Todd Gamblin. 2020. Auto-
tuning parameter choices in HPC applications using Bayesian opti-
mization. In 2020 IEEE International Parallel and Distributed Processing
Symposium (IPDPS) . IEEE, 831–840.
[57] Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Ef-
ficient Estimation of Word Representations in Vector Space. In 1st
International Conference on Learning Representations, ICLR 2013, Scotts-
dale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings , Yoshua
Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1301.3781
[58] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.
Efficient Estimation of Word Representations in Vector Space.
arXiv:1301.3781 [cs.CL]
[59] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex
Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray
Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement
learning. In International conference on machine learning . PMLR, 1928–
1937.
[60] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioan-
nis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing
atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602
(2013).
[61] Rik Mulder, Valentin Radu, and Christophe Dubach. 2021. Fast Opti-
misation of Convolutional Neural Network Inference Using System
Performance Models. In Proceedings of the 1st Workshop on Machine
Learning and Systems (Online, United Kingdom) (EuroMLSys ’21). As-
sociation for Computing Machinery, New York, NY, USA, 104–110.
https://doi.org/10.1145/3437984.3458840
[62] Ravi Teja Mullapudi, Andrew Adams, Dillon Sharlet, Jonathan Ragan-
Kelley, and Kayvon Fatahalian. 2016. Automatically scheduling halide
Automating Reinforcement Learning Architecture Design for Code Optimization CC ’22, April 02–03, 2022, Seoul, South Korea
image processing pipelines. ACM Transactions on Graphics (TOG) 35,
4 (2016), 1–11.
[63] William F Ogilvie, Pavlos Petoumenos, Zheng Wang, and Hugh Leather.
2014. Fast automatic heuristic construction using active learning.
In International Workshop on Languages and Compilers for Parallel
Computing. Springer, 146–160.
[64] William F Ogilvie, Pavlos Petoumenos, Zheng Wang, and Hugh Leather.
2017. Minimizing the cost of iterative compilation with active learning.
In 2017 IEEE/ACM International Symposium on Code Generation and
Optimization (CGO). IEEE, 245–256.
[65] OpenAI. 2020. Gym: a toolkit for developing and comparing reinforce-
ment learning algorithms. https://gym.openai.com/.
[66] Jakub Pachocki, Greg Brockman, Jonathan Raiman, Susan Zhang, Hen-
rique Pondé, Jie Tang, Filip Wolski, Christy Dennison, Rafal Jozefowicz,
Przemyslaw Debiak, et al. 2018. Openai five, 2018. URL https://blog.
openai. com/openai-five (2018).
[67] Eunjung Park, John Cavazos, Louis-Noël Pouchet, Cédric Bastoul, Al-
bert Cohen, and P Sadayappan. 2013. Predictive modeling in a polyhe-
dral optimization space. International journal of parallel programming
41, 5 (2013), 704–750.
[68] Marcelo Pecenin, André Murbach Maidl, and Daniel Weingaertner.
2019. Optimization of Halide Image Processing Schedules with Rein-
forcement Learning. In Anais do XX Simpósio em Sistemas Computa-
cionais de Alto Desempenho . SBC, 37–48.
[69] Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain
Paris, Frédo Durand, and Saman Amarasinghe. 2013. Halide: a language
and compiler for optimizing parallelism, locality, and recomputation in
image processing pipelines. Acm Sigplan Notices 48, 6 (2013), 519–530.
[70] Prashant Singh Rawat, Aravind Sukumaran-Rajam, Atanas Rountev,
Fabrice Rastello, Louis-Noël Pouchet, and P. Sadayappan. 2018. As-
sociative Instruction Reordering to Alleviate Register Pressure. In
Proceedings of the International Conference for High Performance Com-
puting, Networking, Storage, and Analysis (Dallas, Texas) (SC ’18). IEEE
Press, Article 46, 13 pages.
[71] Jie Ren, Lu Yuan, Petteri Nurmi, Xiaoming Wang, Miao Ma, Ling Gao,
Zhanyong Tang, Jie Zheng, and Zheng Wang. 2020. Camel: Smart,
adaptive energy optimization for mobile web interactions. In IEEE
INFOCOM 2020-IEEE Conference on Computer Communications . IEEE,
119–128.
[72] Rodrigo CO Rocha, Pavlos Petoumenos, Zheng Wang, Murray Cole,
Kim Hazelwood, and Hugh Leather. 2021. HyFM: function merging for
free. In Proceedings of the 22nd ACM SIGPLAN/SIGBED International
Conference on Languages, Compilers, and Tools for Embedded Systems .
110–121.
[73] Rodrigo CO Rocha, Pavlos Petoumenos, Zheng Wang, Murray Cole,
and Hugh Leather. 2019. Function merging by sequence alignment.
In 2019 IEEE/ACM International Symposium on Code Generation and
Optimization (CGO). IEEE, 149–163.
[74] Rodrigo CO Rocha, Pavlos Petoumenos, Zheng Wang, Murray Cole,
and Hugh Leather. 2020. Effective function merging in the ssa form.
In Proceedings of the 41st ACM SIGPLAN Conference on Programming
Language Design and Implementation . 854–868.
[75] Rohan Basu Roy, Tirthak Patel, Vijay Gadepally, and Devesh Tiwari.
2021. Bliss: auto-tuning complex applications using a pool of diverse
lightweight learning models. In Proceedings of the 42nd ACM SIGPLAN
International Conference on Programming Language Design and Imple-
mentation. 1280–1295.
[76] Eric Schkufza, Rahul Sharma, and Alex Aiken. 2013. Stochastic super-
optimization. In Proceedings of the eighteenth international conference
on Architectural support for programming languages and operating
systems. 305–316.
[77] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and
Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv
preprint arXiv:1707.06347 (2017).
[78] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou,
Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan
Kumaran, Thore Graepel, et al. 2017. Mastering chess and shogi by
self-play with a general reinforcement learning algorithm. arXiv
preprint arXiv:1712.01815 (2017).
[79] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou,
Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan
Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and
Demis Hassabis. 2018. A general reinforcement learning algorithm
that masters chess, shogi, and Go through self-play. Science 362, 6419
(2018), 1140–1144. https://doi.org/10.1126/science.aar6404
[80] Mark Stephenson, Saman Amarasinghe, Martin Martin, and Una-May
O’Reilly. 2003. Meta optimization: Improving compiler heuristics with
machine learning. ACM sigplan notices 38, 5 (2003), 77–90.
[81] Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning:
An introduction. MIT press.
[82] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark San-
dler, Andrew Howard, and Quoc V Le. 2019. Mnasnet: Platform-aware
neural architecture search for mobile. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition . 2820–2828.
[83] Mircea Trofin, Yundi Qian, Eugene Brevdo, Zinan Lin, Krzysztof Choro-
manski, and David Li. 2021. MLGO: a Machine Learning Guided
Compiler Optimizations Framework. arXiv preprint arXiv:2101.04808
(2021).
[84] Hado P van Hasselt, Arthur Guez, Arthur Guez, Matteo Hessel,
Volodymyr Mnih, and David Silver. 2016. Learning values across
many orders of magnitude. In Advances in Neural Information Process-
ing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett
(Eds.), Vol. 29.
[85] Farui Wang, Weizhe Zhang, Shichao Lai, Meng Hao, and Zheng Wang.
2021. Dynamic GPU Energy Optimization for Machine Learning Train-
ing Workloads. IEEE Transactions on Parallel and Distributed Systems
(2021).
[86] Huanting Wang, Guixin Ye, Zhanyong Tang, Shin Hwei Tan, Songfang
Huang, Dingyi Fang, Yansong Feng, Lizhong Bian, and Zheng Wang.
2020. Combining graph-based learning with automated data collection
for code vulnerability detection. IEEE Transactions on Information
Forensics and Security 16 (2020), 1943–1958.
[87] Yiming Wang, Weizhe Zhang, Meng Hao, and Zheng Wang. 2021.
Online Power Management for Multi-cores: A Reinforcement Learning
Based Approach. IEEE Transactions on Parallel and Distributed Systems
33, 4 (2021), 751–764.
[88] Zheng Wang and Michael O’Boyle. 2018. Machine learning in compiler
optimization. Proc. IEEE 106, 11 (2018), 1879–1901.
[89] R Clinton Whaley and Jack J Dongarra. 1998. Automatically tuned
linear algebra software. In SC’98: Proceedings of the 1998 ACM/IEEE
conference on Supercomputing . IEEE, 38–38.
[90] Guixin Ye, Zhanyong Tang, Huanting Wang, Dingyi Fang, Jianbin Fang,
Songfang Huang, and Zheng Wang. 2020. Deep program structure
modeling through multi-relational graph-based learning. In Proceed-
ings of the ACM International Conference on Parallel Architectures and
Compilation Techniques. 111–123.
[91] Shuangjiao Zhai, Zhanyong Tang, Petteri Nurmi, Dingyi Fang, Xiao-
jiang Chen, and Zheng Wang. 2021. RISE: Robust wireless sensing
using probabilistic and statistical assessments. In Proceedings of the
27th Annual International Conference on Mobile Computing and Net-
working. 309–322.
[92] Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu,
Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen,
et al. 2020. Ansor: Generating high-performance tensor programs for
deep learning. In 14th {USENIX}Symposium on Operating Systems
Design and Implementation ( {OSDI}20). 863–879.
[93] Barret Zoph and Quoc V Le. 2016. Neural architecture search with
reinforcement learning. arXiv preprint arXiv:1611.01578 (2016).

CodeXGLUE: A Machine Learning Benchmark Dataset for Code
Understanding and Generation
Shuai Lu∗
Peking University
Daya Guo∗
Sun Yat-sen University
Shuo Ren∗
Beihang University
Junjie Huang∗
Beihang University
Alexey Svyatkovskiy
Microsoft
Ambrosio Blanco
Microsoft Research Asia
Colin Clement
Microsoft
Dawn Drain
Microsoft
Daxin Jiang
Microsoft
Duyu Tang
Microsoft Research Asia
Ge Li
Peking University
Lidong Zhou
Microsoft Research Asia
Linjun Shou
Microsoft
Long Zhou
Microsoft Research Asia
Michele Tufano
Microsoft
Ming Gong
Microsoft
Ming Zhou
Microsoft Research Asia
Nan Duan
Microsoft Research Asia
Neel Sundaresan
Microsoft
Shao Kun Deng
Microsoft
Shengyu Fu
Microsoft
Shujie Liu
Microsoft Research Asia
ABSTRACT
Benchmark datasets have a significant impact on accelerating re-
search in programming language tasks. In this paper, we introduce
CodeXGLUE, a benchmark dataset to foster machine learning re-
search for program understanding and generation. CodeXGLUE
includes a collection of 10 tasks across 14 datasets and a platform
for model evaluation and comparison. CodeXGLUE also features
three baseline systems, including the BERT-style, GPT-style, and
Encoder-Decoder models, to make it easy for researchers to use the
platform. The availability of such data and baselines can help the
development and validation of new methods that can be applied to
various program understanding and generation problems 1.
KEYWORDS
program understanding, machine learning, naturalness of software
1 INTRODUCTION
Evans Data Corporation2 estimated that there were 23.9 million
professional developers in 2019 and that the number was expected
to reach 28.7 million in 2024. With the population of developers
growing at such a rate, code intelligence that leverages artificial
intelligence (AI) to help software developers improve the productiv-
ity of the development process is becoming increasingly important.
∗ indicates equal contribution and internship at Microsoft. Authors are listed in alpha-
beta order. Corresponding authors are Duyu Tang and Shujie Liu.
1CodeXGLUE is publicly available at https://github.com/microsoft/CodeXGLUE. Par-
ticipants can submit their results by emailing to codexglue@microsoft.com.
2https://evansdata.com/press/viewRelease.php?pressID=278
It is commonly accepted that benchmarks have a significant impact
on the growth of applied AI research. In this paper, we focus on
establishing a benchmark dataset for code intelligence.
Automated program understanding and generation could in-
crease the productivity of software developers. In fact, developers
who want to find code written by others with the same intent can
leverage code search systems [23, 35, 58, 85] to automatically re-
trieve semantically relevant codes through natural language queries.
Similarly, developers who are confused about what to write next
can use code completion systems [4, 8, 9, 31, 62, 63, 72, 73] to auto-
matically complete the following tokens based on the edits made to
the code. Finally, when developers want to implement the Java code
in Python, code-to-code translation systems [11, 41, 46, 54] can help
translate their code from one programming language (Python) to
another (Java).
In recent years, researchers have increasingly applied statisti-
cal models, including neural nets, to code intelligence tasks. Very
recently, the application of pretrained models that learn from big
programming language data has been inspired by the great success
of pretrained models like BERT [16] and GPT [69] in natural lan-
guage processing (NLP). These models, including CodeBERT [18]
and IntelliCode Compose [72], have led to further improvements
in code understanding and generation problems, but they lack a
benchmark suite that covers a wide range of tasks. The use of Ima-
geNet [15] for computer vision and the use of GLUE [81] for NLP
have shown that a diversified benchmark dataset has a significant
impact on the growth of applied AI research.
arXiv:2102.04664v2  [cs.SE]  16 Mar 2021
Lu, Guo, Ren and Huang, et al.
Table 1: A brief summary of CodeXGLUE, which includes tasks, datasets, languages, sizes in various states, and baseline sys-
tems. Highlighted datasets are newly introduced.
Category Task Dataset Name Language Train/Dev/Test Size Baselines
Code-Code
BigCloneBench [71] Java 900K/416K/416KClone Detection POJ-104 [52] C/C++ 32K/8K/12K
Defect Detection Devign [99] C 21K/2.7K/2.7K
CT-all Python,Java,PHP,
JavaScript,Ruby,Go -/-/176K
Cloze Test
CT-max/min [18] Python,Java,PHP,
JavaScript,Ruby,Go -/-/2.6K
CodeBERT
PY150 [62] Python 100K/5K/50K
CodeGPTCode Completion Github Java Corpus[4] Java 13K/7K/8K
Code Repair Bugs2Fix [75] Java 98K/12K/12K Encoder-
DecoderCode Translation CodeTrans Java-C# 10K/0.5K/1K
Text-Code
CodeSearchNet [35],
AdvTest Python 251K/9.6K/19K
CodeBERTNL Code Search CodeSearchNet [35],
WebQueryTest Python 251K/9.6K/1K
Text-to-Code
Generation CONCODE [38] Java 100K/2K/2K CodeGPT
Code-Text Code Summarization CodeSearchNet [35] Python,Java,PHP,
JavaScript,Ruby,Go 908K/45K/53K
Encoder-
DecoderText-Text Documentation
Translation Microsoft Docs English-Latvian/Danish
/Norwegian/Chinese 156K/4K/4K
To address this problem, we introduce CodeXGLUE, a machine
learning benchmark dataset for program understanding and genera-
tion research that includes 14 datasets3, a collection of 10 diversified
programming language understanding and generation tasks, and a
platform for model evaluation and comparison. CodeXGLUE sup-
ports the following tasks:
•code-code (clone detection [10, 52, 71, 84, 89, 93, 97], defect
detection [47, 57, 61, 82, 83, 99], cloze test [18], code comple-
tion [4, 8, 9, 31, 62, 63, 72, 73], code repair [2, 28, 30, 75, 76, 78],
and code-to-code translation [11, 41, 46, 54])
•text-code (natural language code search [23, 35, 85], text-
to-code generation [12, 26, 36, 38, 87, 90, 94, 95])
•code-text (code summarization [3, 12, 19, 34, 37, 80, 85–87])
•text-text (documentation translation [40])
CodeXGLUE includes eight previously proposed datasets — Big-
CloneBench [71], POJ-104 [52], Devign [99], PY150 [62], Github
Java Corpus [4], Bugs2Fix [75], CONCODE [38], and CodeSearch-
Net [35]— but also newly introduced datasets that are highlighted in
Table 1. The datasets are chosen or created based on the considera-
tion that the task has clear definition, and the volume of the dataset
could support the development and evaluation of data-driven ma-
chine learning methods. The datasets created by us include (1) two
cloze test test sets that cover 6 programming languages, (2) two line-
level code completion test sets in Java and Python, respectively, (3) a
code-to-code translation dataset between Java and C#, (4) two natu-
ral language code search test sets with web queries and normalized
function and variable names, respectively, and (5) a documentation
translation dataset that covers five natural languages.
3We plan to evolve the benchmark over time by extending to more tasks.
To make it easy for participants, we provide three baseline mod-
els to help perform the tasks, including a BERT-style pretrained
model (in this case, CodeBERT) to supports code understanding
problems, a GPT-style pretrained model, which we call CodeGPT,
to help solve completion and generation problems, and an Encoder-
Decoder framework that tackles sequence-to-sequence generation
problems.
2 TASKS OVERVIEW
In this section, we provide a definition for each task.
Clone detection [52, 71]. The task is to measure the semantic
similarity between codes. This includes two subtasks: binary classi-
fication between a pair of codes and code retrieval, where the goal
is to find semantically similar codes.
Defect detection [99] . The objective is to identify whether a
body of source code contains defects that may be used to attack soft-
ware systems, such as resource leaks, use-after-free vulnerabilities,
and DoS attack.
Cloze test [18] . This aims to predict the masked token of a code
and includes two subtasks. The first one is to measure the accuracy
of predicting the masked token from the whole vocabulary. The
other is to test the semantic reasoning ability by distinguishing
between “max” and “min”.
Code completion [4, 62] . It aims to predict following tokens
based on a code context. Its subtasks are token-level completion
and line-level completion. The former checks whether the next
one token has been predicted correctly, while the latter tests the
goodness of the generated line.
Code translation [54] . It involves translating a code from one
programming language to a different one.
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Code search [35]. It measures the semantic relatedness between
texts and codes and is composed of two subtasks. The first one is to
find the most relevant code in a collection of codes according to a
natural language query. The second subtask entails the analysis of
a query-code pair to predict whether the code answers the query
or not.
Code repair [75] . Its goal is to refine the code by fixing the bugs
automatically.
Text-to-code generation [38] . This aims to generate a code
via a natural language description.
Code summarization [37] . The objective is to generate the
natural language comment for a code.
Documentation translation [40]. It aims to translate code doc-
umentation from one natural language to different one.
3 DATASETS
In this section, we describe the datasets included in CodeXGLUE.
Datasets are chosen or created based on the criterion that the vol-
ume of the dataset could support the development and evaluation
of data-driven machine learning methods.
3.1 Clone detection
Clone detection includes two subtasks. The first subtask is to predict
whether two given codes have the same semantics. We use the
BigCloneBench [71] dataset for the subtask. The second subtask
aims to retrieve semantically similar codes given a code as the query
and we use the dataset POJ-104 [52] to perform it.
BigCloneBench is a widely used large code clone benchmark
that contains over 6,000,000 true clone pairs and 260,000 false clone
pairs from 10 different functionalities. The dataset provided by
Wang et al. [84] is filtered by discarding code fragments without
any tagged true or false clone pairs, leaving it with 9,134 Java code
fragments. Finally, the dataset includes 901,028/415,416/415,416
examples for training, validation and testing, respectively.
POJ-104 dataset [52] comes from a pedagogical programming
open judge (OJ) system that automatically judges the validity of
submitted source code for specific problems by running the code.
We use the POJ-104 dataset, which consists of 104 problems and
includes 500 student-written C/C++ programs for each problem.
Different from that of the BigCloneBench dataset, the task of POJ-
104 aims to retrieve other programs that solve the same problem
given a program. We group the datasets in three subsets based on
the number of problems they are required to solve (64/16/24) for
training, validation, and testing.
3.2 Defect detection
For the task of defect detection, Zhou et al. [99] provide the Devign
dataset that includes 27,318 manually-labeled functions collected
from two large C programming language open-source projects pop-
ular among developers and diversified in functionality, i.e., QEMU
and FFmpeg. The dataset was created by collecting security-related
commits and extracting vulnerable or non-vulnerable functions
from the labeled commits. Since Zhou et al . [99] did not provide
official training/validation/testing sets for the two projects, we
randomly shuffle the dataset and split 80%/10%/10% of the dataset
for training/validation/testing. The task is formulated as a binary
classification to predict whether a function is vulnerable.
3.3 Cloze test
Figure 1 shows two examples of the cloze test (CT) task in code
domain, which aims to assess models’ ability to understand a code
by asking those models to predict the masked code from several
candidates. We focus on two subtasks: CT-all with candidates from
a filtered vocabulary and CT-maxmin with the candidates “max”
and “min”.
Open the drop box.Doc.:
Code: def open(self): 
self.workingArea.<mask>( )
self.runid_pkgidx_map = {}
self.runid_to_return = deque()
Answer: open
Find min and max values of every feature.Doc.:
Code: def fit(self, X, y=None):
X = check_array(X)
self._x_min = X.<mask>(axis=0)
self._x_max = X.max(axis=0)
return self
Answer: min
Cloze Test-maxmin
Cloze Test-all
Figure 1: Two examples in the cloze test dataset.
We use the validation and testing sets of CodeSearchNet [ 35]
to create CT-all and CT-maxmin datasets for six programming
languages, i.e., Go, Java, JavaScript (JS), PHP, Python and Ruby.
CT-all. To less introduce lengthy variable names and avoid the
issue caused by the use of different tokenizers, we select target cloze
words by retaining unique words after Byte Pair Encoding [67], and
we remove meaningless tokens like punctuations with handcrafted
rules. At last, 930 tokens are selected among six languages in total.
We select codes containing the 930 tokens and manually set thresh-
old values of token occurrence to balance the frequency of the 930
tokens in CT-all.
CT-maxmin. To further evaluate models’ ability to understand
code semantics, we introduce CT-maxmin to test how well model
can distinguish the difference between max and min. CT-maxmin
comes from the dataset used for the PL-Probing task in CodeBERT[18],
which includes codes containing the keywords of max or min.
The data statistics are listed in Table 2.
3.4 Code completion
We use two influential datasets for code completion, PY150 in
python and Github Java Corpus in Java. Both datasets can help
achieve token-level code completion. We move further by creating
two test sets for the line-level code completion task from the two
corpora. The task is to complete an unfinished line. Models should
Lu, Guo, Ren and Huang, et al.
Table 2: Data statistics about the cloze test datasets.
Task CT-all CT-maxmin
Go 25,282 152
Java 40,492 482
JavaScript 13,837 272
PHP 51,930 407
Python 40,137 1,264
Ruby 4,437 38
All 176,115 2,615
be capable of predicting code sequences of arbitrary token types
and code structures.
PY150 is a Python dataset [62] containing 150,000 Python source
files collected from Github. We follow the data split in Raychev
et al. [62], resulting in 100,000 files for training and 50,000 files for
testing, consisting 76.3M tokens and 37.2M tokens, respectively.
We preprocess the corpora by tokenizing source codes, removing
comments, replacing strings with length of more than 15 characters
with empty strings, and adding a special token ⟨EOL⟩(end-of-line)
to mark the ending of a line explicitly. For line-level code com-
pletion, we create 10,000 examples from different files in the test
set of PY150 for testing. Since we intend to test model’s ability to
autocomplete an arbitrary line, we select the line to be predicted at
random. We generate a test case by ensuring that there is sufficient
context, i.e., at least 15% of the whole file. Models are expected
to generate the following line ended by ⟨EOL⟩given the context.
The average number of tokens in input and output are 489.11 and
6.56, respectively. Figure 2 shows an example of line-level code
completion.
Figure 2: An example in the line-level code completion
dataset.
Github Java Corpus is a Java dataset mined by Allamanis and
Sutton [4], and it collects over 14 thousand Java projects from
Github. We follow the settings established by Hellendoorn and
Devanbu [29] as well as Karampatsis et al . [42], using 1% of the
subset in the corpus. We have 12,934/7,189/8,268 files for train-
ing/validation/testing, consisting of 15.8M/3.8M/5.3M tokens, re-
spectively. We do the same preprocessing conducted on PY150, but
we don’t add the special token ⟨EOL⟩since in Java the symbols ;
and } are used to mark the ending of a code statement. For line-
level code completion, we create 3,000 examples for testing from
different files in the test set of the corpus. Similarly to the process
we follow for Python, the line to be predicted is selected at random
from the test file. The average numbers of tokens are 350.62 and
10.49 in input and output, respectively.
3.5 Code translation
The training data for code translation is the code pairs with equiva-
lent functionality in two programming languages. In this paper, we
provide a dataset consisting of parallel codes between Java and C#.
We did not use the dataset of Lachaux et al. [46] because they did
not have the data for supervised model training. Following Nguyen
et al. [54]and Chen et al. [11], we use the data collected from several
open-source projects, i.e., Lucene4, POI5, JGit6 and Antlr7. We do
not use Itext8 and JTS9 due to the license problem. Those projects
are originally developed in Java and then ported to C#. They are
well-established systems with long developing histories and with
both Java and C# versions in use.
The following step is to mine paired functions or methods from
those projects. According to our observation, the directory struc-
tures and function or method names of the two versions are iden-
tical or similar when they are applied to the same project. There-
fore, following Nguyen et al . [54], we conservatively search for
the functions having the same signatures in the classes with the
same/similar names and included in the same/similar directory
structures of both versions. We discard duplicate code pairs and
the codes having multiple targets searched with the above method.
After this step, we remove the pairs whose number of overlapping
tokens was less than 1/3 of the sentence length. To make our data
more scalable for further syntactic and semantic analysis, we also
remove the functions with null function body according to their
abstract syntax tree (AST). Then we build the data-flow graph [25]
for each function, which represents the dependency between two
variables and provides valuable semantic information for code un-
derstanding. Finally, a function with no data-flow extracted from
the AST of a specific function is also discarded.
At last, the total number of paired functions or methods is 11,800.
We randomly select 500 pairs of functions for the development
set and another 1,000 pairs for the test set. The average lengths of
the Java and C# functions after tokenization are 38.51 and 46.16,
respectively 10. An example of the mined translation pairs from C#
to Java is shown in Figure 3.
3.6 Code search
Code search includes two subtasks. The first one is to find the
most relevant code from a collection of candidates given a nat-
ural language query. We create a challenging testing set, called
CodeSearchNet AdvTest, from CodeSearchNet corpus [ 35] for
performing this task. An example of this dataset is shown in Figure
4. The second subtask is to predict whether a code answers a given
query. We provide a testing setWebQueryTestof real user queries.
Two examples of the dataset are illustrated in Figure 5.
4http://lucene.apache.org/
5http://poi.apache.org/
6https://github.com/eclipse/jgit/
7https://github.com/antlr/
8http://sourceforge.net/projects/itext/
9http://sourceforge.net/projects/jts-topo-suite/
10https://github.com/c2nes/javalang
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Figure 3: An example in the code translation dataset.
CodeSearchNet AdvTest is a Python dataset from the Code-
SearchNet [35] corpus. Each example includes a function paired
with a document. We follow Husain et al . [35] to take the first
paragraph of the documentation as the query for the correspond-
ing function. To improve the quality of the dataset, we filter it by
removing the following examples.
(1) Examples whose code could not be parsed into abstract syntax
tree.
(2) Examples whose document tokens number is shorter than 3
or larger than 256.
(3) Examples whose document contains special tokens such as
“http://".
(4) Examples whose document is empty or not written in English.
At the end of the process, we obtain a dataset with 251,820 / 9,604
/ 19,210 examples for training/validation/testing. After normaliz-
ing function or variable names with special tokens, we observe
that the Mean Reciprocal Rank (MRR) scores of RoBERTa [50] and
CodeBERT [18] for the code search task on the CodesearchNet [35]
dataset drop from 0.809 to 0.419 and from 0.869 to 0.507, respectively,
in Python programming language. To better test the understanding
and generalization abilities of the model, we normalize function
and variable names in testing and development sets like 𝑓𝑢𝑛𝑐 for
the function name and 𝑎𝑟𝑔𝑖 for the i-th variable name. Figure 4
shows an example in CodeSearchNet AdvTest dataset. The task
aims to search source codes from candidates for a natural language
query. In contrast to the testing phase of previous works [18, 35]
that only involved 1,000 candidates, we use the entire testing set
for each query, which makes CodeSearchNet AdvTest dataset
more difficult. The training set for this task comes from the filtered
CodeSearchNet dataset [35].
WebQueryTest: Most code search datasets use code documenta-
tions or questions from online communities for software developers
as queries, but these are different from real user search queries. To
fix this discrepancy, we provide WebQueryTest, a testing set of
Scans through a string for substrings matched some patterns.
Query:
Gold Code:
def matchall(text, patterns):
ret = []
for pattern in patterns:
match = re.findall(pattern, text)
ret += match
return ret
Normalized Code:
def func(arg0, arg1):
arg2 = []
for arg3 in arg1:
arg4 = re.findall(arg3, arg0)
arg2 += arg4
return arg2
Figure 4: An example in the CodeSearchNet AdvTest dataset.
real code search for Python. The problem is formulated as a binary
classification task and as a complementary setting to the retrieval
scenario. Given a pair of query and code function, a model needs
to classify whether the code function can answer the query or not.
The data creation process can be divided into two stages: data
collection and annotation. We first collect real user queries from
the web query logs of a commercial search engine and we keep
the queries with “python”. Inspired by Yan et al. [91], we design
some heuristics based on keyword exact matching to filter out
queries without the code search intent. Then we select candidate
codes for each query from the Python validation and testing sets in
CodeSearchNet. To shrink the candidates to be annotated for each
query, we select the top two functions with the highest query-code
similarity computed by a CodeBERT-based code retrieval model,
which is trained on 148K automated-minded Python Stack Overflow
Question-Code (StaQC) [92] with the default parameters provided
by Feng et al. [18].
We use a two-stage annotation schema to label each instance.
The first step is to judge whether the query has a code-search intent.
Instances labeled as "-1" are those without code search intent. The
second step is to assess whether the code (with its documentation)
can answer the query. Instances labeled as "1" are those where the
code can answer the query. Otherwise, they are labeled as “0”. Two
examples are illustrated in Figure 5. We invite 13 developers profi-
cient in Python to label 1,300 instances, with each annotator dealing
with 100 of them. Discussions are allowed during annotation. Fi-
nally, the numbers of instances with labels -1, 0 and 1 are 254, 642
and 422, respectively. Since we are more interested in query-code
matching, we include only the categories 0 and 1 in our final test
set. The training and validation sets we use for this task are from
the original CodeSearchNet dataset [35].
3.7 Code repair
Code repair aims to fix bugs in the code automatically. We use the
dataset released by Tufano et al . [75]. The source is buggy Java
functions, whereas the target is the corresponding fixed functions.
To build this dataset, they first download every public GitHub event
Lu, Guo, Ren and Huang, et al.
python measure distance between 2 pointsQuery:
Code: def vector_distance(a, b):
""" Euclidean distance between two vectors """
a = np.array(a)
b = np.array(b)
return np.linalg.norm(a - b)
Label: 1
how to append object in a specific index in list pythonQuery:
Code: def append(self, item):           
""" append item and print it to stdout """           
print(item)           
super(MyList, self).append(item)
Label: 0
Figure 5: Two examples in the WebQueryTest dataset.
between March 2011 and October 2017 from GitHub Archive11 and
use the Google BigQuery APIs to identify all Java-file commits
having a message containing the patterns [ 21]: (“fix” or “solve”)
and (“bug” or “issue” or “problem” or “error”). For each bug-fixing
commit, they extract the source code before and after the fixing
process by using the GitHub Compare API12 to collect the buggy
(pre-commit) and the fixed (post-commit) codes. Subsequently, they
normalize all the names of the variables and custom methods, which
greatly limits the vocabulary size and enables the model to focus
on learning bug-fixing patterns. Then, they filter out the pairs that
contain lexical or syntactic errors in either the buggy or fixed code,
as well as the pairs with more than 100 atomic AST modification
actions between the buggy and the fixed versions. To achieve this,
they employ the GumTree Spoon AST Diff tool [17]. Finally, they
divide the whole dataset into two subsets (small with tokens ≤50
and medium with tokens > 50 and ≤100) based on the code length.
For the small subset, the numbers of training, development, and test
samples are 46,680, 5,835, and 5,835, respectively. For the medium
subset, the numbers are 52,364, 6,545, and 6,545, respectively.
3.8 Text-to-code generation
To carry out this task, we use CONCODE [ 38], a widely used
code generation dataset, which is collected from about 33,000 Java
projects on GitHub. It contains 100,000 examples for training and
4,000 examples for validation and testing. Each example is a tuple
consisting of NL descriptions, code environments and code snippets.
The dataset is tasked with generating class member functions from
natural language descriptions (Javadoc-style method comments)
and class environments. Class environment is the programmatic
context provided by the rest of the class, including other member
variables and member functions in the class.
3.9 Code summarization
For code summarization, we use the CodeSearchNet dataset [35],
which includes six programming languages; i.e., Python, Java, JavaScript,
11https://www.gharchive.org/
12https://developer.github.com/v3/repos/commits/#compare-two-commits
PHP, Ruby, and Go. The data comes from publicly available open-
source non-fork GitHub repositories and each documentation is the
first paragraph. We observe that some documents contain content
unrelated to the function, such as a link “http://... " that refers to
external resources and an HTML image tag “<img ...>" that inserts
an image. Therefore, we filter the dataset to improve its quality
with the same four rules mentioned in Section 3.6.
The statistics about the filtered CodeSearchNet dataset used in
CodeXGLUE are listed in Table 3.
Table 3: Data statistics about the filtered CodeSearchNet
dataset for the code summarization task.
Language Training Dev Testing
Go 167,288 7,325 8,122
Java 164,923 5,183 10,955
JavaScript 58,025 3,885 3,291
PHP 241,241 12,982 14,014
Python 251,820 13,914 14,918
Ruby 24,927 1,400 1,261
3.10 Documentation translation
Documentation translation aims to translate code documentation
automatically from one natural language (e.g., English) to another
natural language (e.g., Chinese), as shown in Figure 7. The dataset
we use is crawled from Microsoft Documentation13, including soft-
ware and code description documents in different languages. We
focus on low-resource language pairs, where parallel data is scarce,
and introduce multilingual machine translation tasks, e.g., English
⇔Latvian, Danish, Norwegian, and Chinese. To improve the data
quality, we filter the corpus by removing the following examples.
(1) Pairs whose source sentence is the same as the target sen-
tence;
(2) Pairs whose length of source language or target language is
less than three words;
(3) Pairs whose length ratio between source and target languages
is larger than three;
(4) Pairs whose word alignment ratio computed by fast_align14
is less than 0.6.
The final training data includes 43K, 19K, 44K, and 50K sentence
pairs for English ⇔Latvian, English ⇔Danish, English ⇔Nor-
wegian, and English ⇔Chinese, respectively. In addition, each
language pair has 1K development and test sentence pairs, respec-
tively.
4 BASELINE SYSTEMS
We provide three types of baseline models to perform the previously
mentioned tasks, including a BERT-style pretrained model (in this
case, CodeBERT), which supports program understanding problems,
a GPT-style pretrained model called CodeGPT that helps us solve
completion and generation problems, and an Encoder-Decoder
13https://docs.microsoft.com/, whose document is located at
https://github.com/MicrosoftDocs/.
14https://github.com/clab/fast_align.
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Understanding Generation
Supported tasks: 
• code search 
• code clone detection
Supported tasks: 
• code completion
• code generation
Supported tasks: 
• code repair
• code translation
Previous code tokens
Next code tokens
CodeGPT
Decoder
Input code
Output code
CodeBERT
as Encoder
[SEP][CLS] text/code code [SEP]
FFNN + Softmax
CodeBERT
0
1 Category distribution
Input tokens
Figure 6: Three pipelines, including CodeBERT, CodeGPT, and Encoder-Decoder, are provided.
Input (English):
Multinomial Logistic Regression (Softmax regression) is 
used to compute the probabilities of several possible 
outcomes in classification problems. 
Output (Chinese):
多项式逻辑回归（Softmax回归）用于计算分类问题中
几种可能结果的概率。
Figure 7: An English-to-Chinese example in the documenta-
tion translation dataset.
framework that tackles sequence-to-sequence generation problems.
An illustration of these three pipelines is shown in Figure 6.
4.1 CodeBERT
To carry out code understanding tasks like clone detection, defect
detection, cloze test, and code search, we use CodeBERT [18] as our
encoder. This is a bimodal pretrained model based on Transformer
with 12 layers, 768 dimensional hidden states, and 12 attention
heads for programming language (PL) and natural language (NL).
Feng et al. [18] pretrain CodeBERT by masked language modeling
and replaced token detection objectives on the CodeSearchNet
dataset [35], which includes 2.4M functions with document pairs
for six programming languages. The model supports different types
of the sequence input like text/code and code/code with a special
token [𝐶𝐿𝑆]in front of the sequence and a special symbol [𝑆𝐸𝑃]
to split two kinds of data types.
The model is publicly available at https://huggingface.co/microsoft/
codebert-base.
4.2 CodeGPT
We provide CodeGPT, which is a Transformer-based language
model pretrained on programming language (PL), to support the
code completion and the text-to-code generation tasks. CodeGPT
has the same model architecture and training objective of GPT-2
[59], which consists of 12 layers of Transformer decoders. More
model settings are listed in Table 4. We pretrain monolingual mod-
els on Python and Java corpora from the CodeSearchNet dataset
[35], which includes 1.1M Python functions and 1.6M Java meth-
ods. Each function in training dataset has a function signature and
a function body. Some functions also contain a natural language
documentation.
We train two CodeGPT models for each programming language.
One model is pretrained from scratch, so that the BPE (byte pair
encoder) [67] vocabulary is newly obtained on the code corpus and
the model parameters are randomly initialized. The other model
is a domain-adaptive one, which uses the GPT-2 model as the
starting point and is continually trained on the code corpus. As
a result, the second model has the same GPT-2 vocabulary and
natural language understanding ability. We refer to this model as
CodeGPT-adapted, and regard it as the default one for the code
completion and text-to-code generation tasks. Both models are
publicly available at https://huggingface.co/microsoft/CodeGPT-
small-java and https://huggingface.co/microsoft/CodeGPT-small-
java-adaptedGPT2. 15
15Replace "java" with "py" for models pre-trained on python dataset.
Lu, Guo, Ren and Huang, et al.
Table 4: Parameters of CodeBERT and CodeGPT models.
CodeBERT CodeGPT
Number of layers 12 12
Max length of position 512 1,024
Embedding size 768 768
Attention heads 12 12
Attention head size 64 64
Vocabulary size 50,265 50,000
Total number of parameters 125M 124M
4.3 Encoder-Decoder
For sequence-to-sequence generation problems like code repair,
code translation, code summarization, and documentation trans-
lation, we provide an Encoder-Decoder framework. We initialize
the encoder using CodeBERT [18] and use a randomly initialized
Transformer with 6 layers, 768 dimensional hidden states and 12
attention heads as the decoder in all settings.
5 EXPERIMENT
In this section, we report accuracy numbers of the baseline systems
on 10 tasks. We will also show how long it takes to train the model
and to do inference on the model.
5.1 Clone Detection
Setting. We use the BigCloneBench and POJ-104 datasets for
clone detection. The task of the BigCloneBench dataset is for-
mulated as a binary classification to predict whether a given pair
of codes has the same semantics, with the F1 score used as the
evaluation metric. The task of the POJ-104 dataset aims to retrieve
499 codes for a given code from the development/test set for val-
idation/testing, with the Mean Average Precision (MAP) as the
evaluation metric. The overall score of the clone detection task is
the average value of F1 and MAP scores.
Table 5: Results on the clone detection task.
BigCloneBench POJ-104
Model F1 MAP Overall
RtvNN 1.0 - -
Deckard 3.0 - -
CDLH 82.0 - -
ASTNN 93.0 - -
FA-AST-GMN 95.0 - -
TBCCD 95.0 - -
code2vec* - 1.98 -
NCC* - 54.19 -
Aroma* - 55.12 -
MISIM-GNN* - 82.45 -
RoBERTa 94.9 79.96 87.4
CodeBERT 96.5 84.29 90.4
Results. Results achieved by different models are shown in Table
5. RtvNN [89] trains a recursive autoencoder to learn representa-
tions for AST. Deckard [39] computes vectors for structural infor-
mation within ASTs and uses a Locality Sensitive Hashing (LSH)
[14] to cluster similar vectors. CDLH [88] learns representations
of code fragments via AST-based LSTM. ASTNN [97] uses RNNs
to encode AST subtrees for statements. It feeds the encodings of
all statement trees into an RNN to learn representation for a pro-
gram. FA-AST-GMN[84] uses GNNs over a flow-augmented AST
to leverage explicit control and data flow information. TBCCD
[96] proposes a position-aware character embedding and uses tree-
based convolution to capture both the structural information of
a code fragment from its AST and lexical information from code
tokens. Code2vec [6] learns representations of code snippets by
aggregating multiple syntactic paths into a single vector. NCC [7]
encodes programs by leveraging both the underlying data flow and
control flow of the programs. Aroma [51] is a code recommen-
dation engine that takes a partial code snippet and recommends
a small set of succinct code snippets that contain the query snip-
pet. MISIM-GNN [93] learns a structural representation of code
from context-aware semantic structure designed specifically to lift
semantic meaning from the code syntax.
In this experiment, we use pretrained models likeRoBERTa[50]
and CodeBERT [18] to encode source code and take the represen-
tation to calculate semantic relevance of two codes through a feed
forward network or inner product. Although CodeBERT does not
leverage code structure that has proven to be effective in terms of
code similarity measure [7, 84, 88, 93, 97], the model still performs
better than RoBERTa on the task of clone detection, achieving the
overall score of 90.4. These experimental results demonstrate that
pretraining is useful for clone detection. There is room for further
improvement if code structure is further leveraged.
5.2 Defect Detection
Setting. We use the dataset mentioned in Section 3.2 for defect
detection, which aims to predict whether a source code contains
defects that may be used to attack software systems. The evaluation
metric is accuracy score. We use the CodeBERT baseline to encode
source code and take the representation of source code to calculate
the probability of being exposed to vulnerabilities.
Results. Table 7 shows the results of the models we implemented.
We use Bidirectional LTSM (BiLTSM) [32], TextCNN[43], RoBERTa
[50], and CodeBERT [18] to encode the representation of a source
code, respectively. Then, a two-layer feed forward network followed
by a softmax layer is used to calculate the probability of encoun-
tering vulnerabilities. As shown in the results, CodeBERT achieve
a 62.1 accuracy score, resulting in state-of-the-art performance.
However, the improvement achieved by the pretrained models is
limited compared with TextCNN. A potential direction to improve
these pretrained models is to incorporate information from code
structures such as Abstract Syntax Tree, data flow, control flow, etc.
5.3 Cloze test
Setting. We use CT-all and CT-maxmin datasets for the cloze
test task. Models are expected to predict the masked code token
by leveraging documentation and the context of code. Accuracy
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Table 6: Results on the cloze test task.
Model CT-all CT-maxmin Overall
Ruby JS Go Python Java PHP Ruby JS Go Python Java PHP
RoBERTa 47.44 59.96 40.77 54.35 50.73 60.16 73.68 64.71 71.71 59.18 59.75 69.78 62.45
CodeBERT(MLM) 80.17 81.77 83.31 87.21 80.63 85.05 86.84 86.40 90.79 82.20 90.46 88.21 85.66
Table 7: Results on the defect detection task.
Model Accuracy
BiLSTM 59.37
TextCNN 60.69
RoBERTa 61.05
CodeBERT 62.08
are reported for each language, with the macro-average accuracy
scores for all languages as the overall evaluation metric.
Results. Table 6 shows the results on the CT-all and CT-maxmin
datasets. We report the performance of RoBERTa [50] and Code-
BERT (Masked Language Modeling, MLM) [18], which is initialized
with RoBERTa and further trained with the masked language mod-
eling objective. The results demonstrate that CodeBERT performs
better than RoBERTa that only learns from natural language.
5.4 Code completion
Setting. We use the PY150 and Github Java Corpus datasets
for token-level and line-level code completion tasks. The token-level
task is to predict the next token given context of previous tokens,
and predictions are evaluated according to token-level accuracy;
whereas the line-level task entails the completion of a whole-line of
code, and the quality of the code is evaluated through the metrics
known as exact match accuracy and Levenshtein edit similarity [72].
Levenshtein edit similarity measures how many single character
edits are required to transform one string into another. This is a
critical evaluation metric for the code completion scenario as it
measures how much effort it takes for developers to correct an
error in the code. The score on each dataset is the average value of
the accuracy on token-level completion and the edit similarity on
line-level completion. The overall score of code completion task is
calculated by averaging the scores on both datasets.
Results. Table 8 shows the results of all models on both datasets.
We fine-tuneLSTM [32], Transformer [77], GPT-2[59], CodeGPT
and CodeGPT-adapted to generate following tokens. CodeGPT
and CodeGPT-adapted models are described in Section 4.2. CodeGPT-
adapted achieves a state-of-the-art performance with the overall
score of 71.28.
5.5 Code search
Setting. We use the CodeSearchNet AdvTest and WebQuery-
Test datasets mentioned in Section 3.6 for code search. To improve
efficiency, we separately encode text and code to perform code
search. For the CodeSearchNet AdvTest dataset, the task is to
find the most relevant code from a collection of candidates given a
query and it is evaluated through the Mean Reciprocal Rank (MRR)
metric. For the WebQueryTestdataset, the task is formulated as a
binary classification to predict whether a code can answer a given
query and we use the F1 and accuracy scores as evaluation met-
rics. The overall score for code search is the average of the values
recorded for the two subtasks.
Results. Table 9 presents the results on the CodeSearchNet Ad-
vTest and WebQueryTest datasets. We report the performance of
RoBERTa [50] and CodeBERT [18]. The table shows that CodeBERT
performs better than RoBERTa.
5.6 Text-to-code generation
Setting. We use the CONCODE dataset for text-to-code gener-
ation. Models are expected to generate source codes of Java class
member functions, given natural language descriptions and class
environments. We report the exact match accuracy, the BLEU score
[56], and the CodeBLEU score [65]. We use the CodeBLEU score as
the overall evaluation metric.
Results. Table 10 presents the results on the CONCODE test
set. Seq2Seq [70] is an RNN-based sequence to sequence model.
Seq2Action + MAML [26] combines a context-aware retrieval
model with model-agnostic meta-learning (MAML). Iyer-Simp
+ 200 idoms [36] extracts code idioms and applies idioms-based
decoding. We also report the performance of pretrained models, in-
cluding GPT-2[59], CodeGPT, andCodeGPT-adapted. CodeGPT-
adapted achieves the CodeBLEU score of 35.98, resulting in a state-
of-the-art performance.
5.7 Code translation
Setting. We use the dataset we build as described in Section 3.5.
The dataset contains matching samples of Java and C# functions.
We report the exact match accuracy, the BLEU score [56] and the
CodeBLEU score [65] on this task. CodeBLEU is used as the overall
evaluation metric.
Results. Table 12 shows the results of models on both translation
directions. The Naive method directly copies the source code as
the translation result. PBSMT is short for phrase-based statistical
machine translation [44]. Transformer uses the same number of
layers and hidden size as the pretrained models. The table shows
that Transformer initialized with CodeBERT and fine-tuned with
the matching sample pairs produces the best result.
5.8 Code repair
Setting. We use the dataset originally released by Tufano et al.
[75], which is described in Section 3.7. The dataset contains two
Lu, Guo, Ren and Huang, et al.
Table 8: Results on the code completion task.
Model
PY150 Github Java Corpus
Overalltoken-level line-level token-level line-level
Accuracy EM Edit Sim Accuracy EM Edit Sim
LSTM 58.00 17.93 50.05 56.02 10.30 41.55 51.41
Transformer 73.26 36.65 67.51 64.16 15.33 50.39 63.83
GPT-2 74.22 38.55 68.94 74.89 24.30 60.70 69.69
CodeGPT 74.93 39.11 69.69 76.45 25.30 61.54 70.65
CodeGPT-adapted 75.11 39.65 69.84 77.13 26.43 63.03 71.28
Table 9: Results on the code search task.
AdvTest WebQueryTest
Model MRR F1 Accuracy Overall
RoBERTa 18.33 57.49 40.92 33.63
CodeBERT 27.19 58.95 47.80 40.28
Table 10: Results on the text-to-code generation task.
Model EM BLEU CodeBLEU
Seq2Seq 3.05 21.31 26.39
Seq2Action+MAML 10.05 24.40 29.46
Iyer-Simp+200 idoms 12.20 26.60 -
GPT-2 17.35 25.37 29.69
CodeGPT 18.25 28.69 32.71
CodeGPT-adapted 20.10 32.79 35.98
subsets established according to the length of the Java functions:
small ≤50 and 50 < medium ≤100 . We report the exact match
accuracy, the BLEU score [56] and the CodeBLEU score [65] on this
task. The exact match accuracy is used as the overall evaluation
metric.
Results. The Naive method directly copies the buggy code as
the repair result. As for Transformer, we use the same number of
layers and hidden size as the pretrained models. With regard to the
CodeBERT method, we initialize the Transformer encoder with
pretrained CodeBERT model and randomly initialize the parameters
of the decoder and the source-to-target attention. Then we use
the training data to fine-tune the whole model. As shown in the
table, Transformer with CodeBERT initialization achieves the best
performance among all models.
5.9 Code Summarization
Setting. We use the dataset mentioned in Section 3.9 for code
summarization. To evaluate the models, we follow Feng et al. [18],
who use smoothed BLEU score [49] as evaluation metric, because
this is suitable for evaluating short documents. We use the encoder-
decoder pipeline to tackle this problem. The max length of input
and inference are set as 256 and 128, respectively. We use the Adam
optimizer to update the models’ parameters. The learning rate and
the batch size are 5e-5 and 32, respectively. We tune the hyper-
parameters and perform early stopping on the development set.
Results. Table 13 shows the results achieved by different mod-
els in code summarization. Seq2Seq is an RNN-based sequence to
sequence model. Transformer and RoBERTa use the same set-
ting as CodeBERT, but the encoder is initialized randomly and
by RoBERTa [50], respectively. All models use Byte Pair Encod-
ing (BPE) [66] vocabulary. In this experiment, CodeBERT obtains
a 1.3% gain in the BLEU score over RoBERTa and achieves the
state-of-the-art performance on six programming languages.
5.10 Documentation translation
Setting. We use the Microsoft Docs dataset for text-to-text trans-
lation tasks, which focus on low-resource multilingual translation
between English (EN) and other languages, including Latvian (LA),
Danish (DA), Norwegian (NO), and Chinese (ZH). Following John-
son et al. [40], we train a single multilingual model as our base-
line. To distinguish between different translation pairs, we add an
language token (e.g., ⟨2en⟩, ⟨2zh⟩) at the beginning of the source
sentence to indicate the target language the model should translate.
We initialize the encoder of the multilingual translation model with
XLM-R [13]. Models are evaluated through BLEU [56] score, and
the overall score for documentation translation is the average BLEU
score on the eight translation directions.
Results. Table 14 shows the results achieved by the models on
eight translation directions.Transformer Baseline is the multilin-
gual translation model [40]. pretrained Transformer initializes
the encoder of Transformer Baseline with XLM-R[13]. In terms
of overall performance on the eight translation directions, Trans-
former Baseline and pretrained Transformer obtain the BLEU
score of 52.67 and 66.16, respectively. Experimental results demon-
strate that pretraining achieves a 13.49 improvement in BLEU score
over strong baseline model. Figure 8 shows how long it takes to
train the model and to do inference on the model, as well as in other
tasks.
6 RELATED WORK
Benchmark datasets have been playing a central role in the growth
of applied AI research. For example, the LibriSpeech [55] and the
SQuAD [60] datasets drive the development of data-driven models
for automatic speech recognition and reading comprehension of
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Table 11: Results on the code repair task.
Method small medium Overall
BLEU Acc CodeBLEU BLEU Acc CodeBLEU
Naive 78.06 0.000 - 90.91 0.000 - 0.000
LSTM 76.76 0.100 - 72.08 0.025 - 0.063
Transformer 77.21 0.147 73.31 89.25 0.037 81.72 0.092
CodeBERT 77.42 0.164 75.58 91.07 0.052 87.52 0.108
Table 12: Results on the code translation task.
Method Java→C# C# →Java Overall
BLEU Acc CodeBLEU BLEU Acc CodeBLEU
Naive 18.54 0.000 - 18.69 0.000 - -
PBSMT 43.53 0.125 42.71 40.06 0.161 43.48 43.10
Transformer 55.84 0.330 63.74 50.47 0.379 61.59 62.67
RoBERTa (code) 77.46 0.561 83.07 71.99 0.579 80.18 81.63
CodeBERT 79.92 0.590 85.10 72.14 0.580 79.41 82.26
Table 13: Results on the code summarization task.
Model Ruby Javascript Go Python Java PHP Overall
Seq2Seq 9.64 10.21 13.98 15.93 15.09 21.08 14.32
Transformer 11.18 11.59 16.38 15.81 16.26 22.12 15.56
RoBERTa 11.17 11.90 17.72 18.14 16.47 24.02 16.57
CodeBERT 12.16 14.90 18.07 19.06 17.65 25.16 17.83
Training and inference cost Task Dataset Name Language Training Cost Inference CostClone DetectionBigCloneBench Java 3 hours training on P100 x2 2 hours on p100 x2POJ-104 C/C++ 2 hours training on P100 x2 10 minutes on p100 x2Defect Detection Devign C 1 hour on P100 x2 2 minutes on p100 x2Cloze TestCT-all Python, Java, PHP , JavaScript, Ruby, Go N/A 30 minutes on P100-16G x2CT-max/min Python, Java, PHP , JavaScript, Ruby, Go N/A 1 minute on P100-16G x2Code CompletionPY150 Python 25 hours on P100 x2 30 minutes on P100 x2GitHub Java Corpus Java 2 hours on P100 x2 10 minutes on P100 x2Code Repair Bugs2Fix Java 24 hours on P100 x2 20 minutes on P100 x2Code Translation CodeTrans Java-C# 20 hours on P100 x2 5 minutes on P100 x2NL Code SearchCodeSearchnet, AdvTestPython 5 hours on P100 x2 7 minutes on p100 x2CodeSearchNet, WebQueryTestPython 5 hours on P100 x2 1 minute on P100 x2Text-to-Code GenerationCONCODE Java 30 hours on P100 x2 20 minutes on P100 x2Code Summarization CodeSearchNet Python, Java, PHP , JavaScript, Ruby, GoOn average, 12 hours for each PL on P100 x2On average, 1 hour for each PL on p100 x2Documentation TranslationMicrosoft DocsEnglish-Latvian/Danish/Norwegian/Chinese30 hours on P100x2 55 minutes on P100x2
Figure 8: Training and inference time costs for each task, evaluated on two P100 GPUs.
Lu, Guo, Ren and Huang, et al.
Table 14: Results on the documentation translation task.
Task Transformer
Baseline
pretrained
Transformer
EN →DA 53.31 67.09
EN →LA 37.85 51.92
EN →NO 53.84 68.00
EN →ZH 59.90 70.60
DA →EN 58.73 67.02
LA →EN 50.37 68.30
NO →EN 57.73 71.84
ZH →EN 50.00 64.47
Overall 52.67 66.16
text, respectively. With the growing demand for testing models’
generalization ability on a wide range of applications, researchers
have created or assembled datasets that cover many tasks. Rep-
resentative samples of these datasets include ImageNet [ 15] for
computer vision, GLUE [81] for natural language understanding,
XTREME [33] and XGLUE [48] for cross-lingual natural language
processing. To the best of our knowledge, CodeXGLUE is the first
diversified benchmark dataset that can be applied to various code
intelligence problems.
Many tasks related to machine learning for software engineer-
ing [1] have sufficient amount of data to support the development
of data-driven methods, but are not covered by CodeXGLUE. We
plan to extend to these tasks in the future. For example, the idiom
mining task [5, 36] is to extract code idioms, which are syntactic
fragments that recur across software projects and serve a single
semantic purpose [5]. Bug localization [27, 61, 76] is to point the
error location when a program fails tests. The test case generation
task [22, 74] is to generate unit test cases automatically. The pro-
gram synthesis [20, 45, 53, 64, 68, 79, 98] extends the text-to-code
generation task aims to generate programs from a specification [24],
such as pseudocode, natural language description, and input/output
examples.
7 CONCLUSION
With CodeXGLUE, we seek to support the development of models
that can be applied to various program understanding and gen-
eration problems, with the goal of increasing the productivity of
software developers. We encourage researchers to participate in
the open challenge to make progress in code intelligence. Moving
forward, we are planning to extend CodeXGLUE to more program-
ming languages and downstream tasks while continuing to develop
advanced pretrained models by exploring new model structures,
introducing new pretraining tasks, using different types of data,
and more.
REFERENCES
[1] Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, and Charles Sutton. 2018.
A Survey of Machine Learning for Big Code and Naturalness. ACM Comput. Surv.
51, 4, Article 81 (July 2018), 37 pages. https://doi.org/10.1145/3212695
[2] Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2017. Learning
to represent programs with graphs. arXiv preprint arXiv:1711.00740 (2017).
[3] Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A convolutional at-
tention network for extreme summarization of source code. In International
conference on machine learning . 2091–2100.
[4] Miltiadis Allamanis and Charles Sutton. 2013. Mining Source Code Repositories
at Massive Scale using Language Modeling. In 2013 10th Working Conference on
Mining Software Repositories (MSR) . IEEE, 207–216.
[5] Miltiadis Allamanis and Charles Sutton. 2014. Mining idioms from source code.
In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations
of Software Engineering . 472–483.
[6] Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019. code2vec: Learn-
ing distributed representations of code. Proceedings of the ACM on Programming
Languages 3, POPL (2019), 1–29.
[7] Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. 2018. Neural code
comprehension: A learnable representation of code semantics. In Advances in
Neural Information Processing Systems . 3585–3597.
[8] Pavol Bielik, Veselin Raychev, and Martin Vechev. 2016. PHOG: Probabilistic
Model for Code. InProceedings of the 33rd International Conference on International
Conference on Machine Learning - Volume 48 (New York, NY, USA) (ICML’16).
JMLR.org, 2933–2942.
[9] Marcel Bruch, Martin Monperrus, and Mira Mezini. 2009. Learning from Ex-
amples to Improve Code Completion Systems. In Proceedings of the 7th Joint
Meeting of the European Software Engineering Conference and the ACM SIGSOFT
Symposium on The Foundations of Software Engineering (Amsterdam, The Nether-
lands) (ESEC/FSE ’09) . Association for Computing Machinery, New York, NY,
USA, 213–222. https://doi.org/10.1145/1595696.1595728
[10] L. Büch and A. Andrzejak. 2019. Learning-Based Recursive Aggregation of
Abstract Syntax Trees for Code Clone Detection. In 2019 IEEE 26th International
Conference on Software Analysis, Evolution and Reengineering (SANER) . 95–104.
https://doi.org/10.1109/SANER.2019.8668039
[11] Xinyun Chen, Chang Liu, and Dawn Song. 2018. Tree-to-tree neural networks
for program translation. In Advances in neural information processing systems .
2547–2557.
[12] Colin B Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and
Neel Sundaresan. 2020. PyMT5: multi-mode translation of natural language and
Python code with transformers. arXiv preprint arXiv:2010.03150 (2020).
[13] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guil-
laume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer,
and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning
at scale. arXiv preprint arXiv:1911.02116 (2019).
[14] Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab S Mirrokni. 2004. Locality-
sensitive hashing scheme based on p-stable distributions. In Proceedings of the
twentieth annual symposium on Computational geometry . 253–262.
[15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet:
A large-scale hierarchical image database. In 2009 IEEE conference on computer
vision and pattern recognition . Ieee, 248–255.
[16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert:
Pre-training of deep bidirectional transformers for language understanding.arXiv
preprint arXiv:1810.04805 (2018).
[17] Jean-Rémy Falleri, Floréal Morandat, Xavier Blanc, Matias Martinez, and Martin
Monperrus. 2014. Fine-grained and accurate source code differencing. In Pro-
ceedings of the 29th ACM/IEEE international conference on Automated software
engineering. 313–324.
[18] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong,
Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained
model for programming and natural languages. arXiv preprint arXiv:2002.08155
(2020).
[19] Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. 2018. Structured
neural summarization. arXiv preprint arXiv:1811.01824 (2018).
[20] John K. Feser, Swarat Chaudhuri, and Isil Dillig. 2015. Synthesizing Data Struc-
ture Transformations from Input-Output Examples. In Proceedings of the 36th
ACM SIGPLAN Conference on Programming Language Design and Implementation
(Portland, OR, USA) (PLDI ’15). Association for Computing Machinery, New York,
NY, USA, 229–239. https://doi.org/10.1145/2737924.2737977
[21] Michael Fischer, Martin Pinzger, and Harald Gall. 2003. Populating a release
history database from version control and bug tracking systems. In International
Conference on Software Maintenance, 2003. ICSM 2003. Proceedings. IEEE, 23–32.
[22] Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation
for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium
and the 13th European conference on Foundations of software engineering . 416–419.
[23] Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018. Deep Code Search. In
Proceedings of the 40th International Conference on Software Engineering (Gothen-
burg, Sweden) (ICSE ’18) . Association for Computing Machinery, New York, NY,
USA, 933–944. https://doi.org/10.1145/3180155.3180167
[24] Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, et al. 2017. Program synthesis.
Foundations and Trends ® in Programming Languages 4, 1-2 (2017), 1–119.
[25] Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long
Zhou, Nan Duan, Jian Yin, Daxin Jiang, et al. 2020. GraphCodeBERT: Pre-training
Code Representations with Data Flow. arXiv preprint arXiv:2009.08366 (2020).
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
[26] Daya Guo, Duyu Tang, Nan Duan, Ming Zhou, and Jian Yin. 2019. Coupling
Retrieval and Meta-Learning for Context-Dependent Semantic Parsing. arXiv
preprint arXiv:1906.07108 (2019).
[27] Rahul Gupta, Aditya Kanade, and Shirish Shevade. 2019. Neural Attribution for Se-
mantic Bug-Localization in Student Programs. In Advances in Neural Information
Processing Systems . 11884–11894.
[28] Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017. DeepFix:
Fixing Common C Language Errors by Deep Learning. In Proceedings of the
Thirty-First AAAI Conference on Artificial Intelligence (San Francisco, California,
USA) (AAAI’17). AAAI Press, 1345–1351.
[29] Vincent J Hellendoorn and Premkumar Devanbu. 2017. Are deep neural networks
the best choice for modeling source code?. In Proceedings of the 2017 11th Joint
Meeting on Foundations of Software Engineering . 763–773.
[30] Vincent J Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David
Bieber. 2019. Global relational models of source code. In International Conference
on Learning Representations .
[31] Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu.
2012. On the naturalness of software. In 2012 34th International Conference on
Software Engineering (ICSE) . IEEE, 837–847.
[32] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory.Neural
computation 9, 8 (1997), 1735–1780.
[33] Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and
Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task bench-
mark for evaluating cross-lingual generalization. arXiv preprint arXiv:2003.11080
(2020).
[34] Xing Hu, Ge Li, Xin Xia, David Lo, Shuai Lu, and Zhi Jin. 2018. Summarizing
Source Code with Transferred API Knowledge. In Proceedings of the 27th Interna-
tional Joint Conference on Artificial Intelligence (Stockholm, Sweden) (IJCAI’18).
AAAI Press, 2269–2275.
[35] Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc
Brockschmidt. 2019. Codesearchnet challenge: Evaluating the state of semantic
code search. arXiv preprint arXiv:1909.09436 (2019).
[36] Srinivasan Iyer, Alvin Cheung, and Luke Zettlemoyer. 2019. Learning program-
matic idioms for scalable semantic parsing. arXiv preprint arXiv:1904.09086
(2019).
[37] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016.
Summarizing source code using a neural attention model. In Proceedings of the
54th Annual Meeting of the Association for Computational Linguistics (Volume 1:
Long Papers) . 2073–2083.
[38] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Map-
ping language to code in programmatic context. arXiv preprint arXiv:1808.09588
(2018).
[39] Lingxiao Jiang, Ghassan Misherghi, Zhendong Su, and Stephane Glondu. 2007.
Deckard: Scalable and accurate tree-based detection of code clones. In 29th
International Conference on Software Engineering (ICSE’07) . IEEE, 96–105.
[40] Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng
Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al.
2017. Google’s multilingual neural machine translation system: Enabling zero-
shot translation. Transactions of the Association for Computational Linguistics 5
(2017), 339–351.
[41] Svetoslav Karaivanov, Veselin Raychev, and Martin Vechev. 2014. Phrase-Based
Statistical Translation of Programming Languages. In Proceedings of the 2014
ACM International Symposium on New Ideas, New Paradigms, and Reflections on
Programming Software (Portland, Oregon, USA) (Onward! 2014) . Association for
Computing Machinery, New York, NY, USA, 173–184. https://doi.org/10.1145/
2661136.2661148
[42] Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, and
Andrea Janes. 2020. Big Code!= Big Vocabulary: Open-Vocabulary Models for
Source Code. arXiv preprint arXiv:2003.07914 (2020).
[43] Yoon Kim. 2014. Convolutional neural networks for sentence classification.arXiv
preprint arXiv:1408.5882 (2014).
[44] Philipp Koehn, Franz J Och, and Daniel Marcu. 2003.Statistical phrase-based trans-
lation. Technical Report. UNIVERSITY OF SOUTHERN CALIFORNIA MARINA
DEL REY INFORMATION SCIENCES INST.
[45] Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex
Aiken, and Percy S Liang. 2019. Spoc: Search-based pseudocode to code. In
Advances in Neural Information Processing Systems . 11906–11917.
[46] Marie-Anne Lachaux, Baptiste Roziere, Lowik Chanussot, and Guillaume Lample.
2020. Unsupervised Translation of Programming Languages. arXiv preprint
arXiv:2006.03511 (2020).
[47] Yi Li, Shaohua Wang, Tien N Nguyen, and Son Van Nguyen. 2019. Improving bug
detection via context-based code representation learning and attention-based
neural networks. Proceedings of the ACM on Programming Languages 3, OOPSLA
(2019), 1–30.
[48] Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming
Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. 2020. Xglue: A new bench-
mark dataset for cross-lingual pre-training, understanding and generation. arXiv
preprint arXiv:2004.01401 (2020).
[49] Chin-Yew Lin and Franz Josef Och. 2004. Orange: a method for evaluating auto-
matic evaluation metrics for machine translation. In COLING 2004: Proceedings
of the 20th International Conference on Computational Linguistics . 501–507.
[50] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A
robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692
(2019).
[51] Sifei Luan, Di Yang, Celeste Barnaby, Koushik Sen, and Satish Chandra. 2019.
Aroma: Code recommendation via structural code search.Proceedings of the ACM
on Programming Languages 3, OOPSLA (2019), 1–28.
[52] Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016. Convolutional neural net-
works over tree structures for programming language processing. In Proceedings
of the Thirtieth AAAI Conference on Artificial Intelligence . 1287–1293.
[53] Arvind Neelakantan, Quoc V Le, and Ilya Sutskever. 2015. Neural programmer:
Inducing latent programs with gradient descent. arXiv preprint arXiv:1511.04834
(2015).
[54] Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. 2015. Divide-and-
conquer approach for multi-phase statistical migration for source code (t). In
2015 30th IEEE/ACM International Conference on Automated Software Engineering
(ASE). IEEE, 585–596.
[55] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015.
Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE
International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE,
5206–5210.
[56] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a
method for automatic evaluation of machine translation. In Proceedings of the
40th annual meeting of the Association for Computational Linguistics . 311–318.
[57] Michael Pradel and Koushik Sen. 2018. DeepBugs: A Learning Approach to
Name-Based Bug Detection. Proc. ACM Program. Lang. 2, OOPSLA, Article 147
(Oct. 2018), 25 pages. https://doi.org/10.1145/3276517
[58] Varot Premtoon, James Koppel, and Armando Solar-Lezama. 2020. Semantic Code
Search via Equational Reasoning. InProceedings of the 41st ACM SIGPLAN Confer-
ence on Programming Language Design and Implementation (London, UK) (PLDI
2020). Association for Computing Machinery, New York, NY, USA, 1066–1082.
https://doi.org/10.1145/3385412.3386001
[59] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya
Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI
blog 1, 8 (2019), 9.
[60] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.
Squad: 100,000+ questions for machine comprehension of text. arXiv preprint
arXiv:1606.05250 (2016).
[61] Baishakhi Ray, Vincent Hellendoorn, Saheel Godhane, Zhaopeng Tu, Alberto
Bacchelli, and Premkumar Devanbu. 2016. On the" naturalness" of buggy code.
In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE) .
IEEE, 428–439.
[62] Veselin Raychev, Pavol Bielik, and Martin Vechev. 2016. Probabilistic Model for
Code with Decision Trees. ACM SIGPLAN Notices (2016), 731–747.
[63] Veselin Raychev, Martin Vechev, and Eran Yahav. 2014. Code Completion with
Statistical Language Models. In Proceedings of the 35th ACM SIGPLAN Confer-
ence on Programming Language Design and Implementation (Edinburgh, United
Kingdom) (PLDI ’14). Association for Computing Machinery, New York, NY, USA,
419–428. https://doi.org/10.1145/2594291.2594321
[64] Scott Reed and Nando De Freitas. 2015. Neural programmer-interpreters. arXiv
preprint arXiv:1511.06279 (2015).
[65] Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Ming Zhou,
Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a Method for Automatic
Evaluation of Code Synthesis. arXiv preprint arXiv:2009.10297 (2020).
[66] Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine
translation of rare words with subword units. arXiv preprint arXiv:1508.07909
(2015).
[67] Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine
Translation of Rare Words with Subword Units. InProceedings of the 54th Annual
Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) .
1715–1725.
[68] Rishabh Singh and Sumit Gulwani. 2015. Predicting a correct program in pro-
gramming by example. InInternational Conference on Computer Aided Verification .
Springer, 398–414.
[69] Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss,
Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al .
2019. Release strategies and the social impacts of language models.arXiv preprint
arXiv:1908.09203 (2019).
[70] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning
with neural networks. InAdvances in neural information processing systems . 3104–
3112.
[71] Jeffrey Svajlenko, Judith F Islam, Iman Keivanloo, Chanchal K Roy, and Moham-
mad Mamun Mia. 2014. Towards a big data curated benchmark of inter-project
code clones. In 2014 IEEE International Conference on Software Maintenance and
Evolution. IEEE, 476–480.
Lu, Guo, Ren and Huang, et al.
[72] Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020.
IntelliCode Compose: Code Generation Using Transformer. arXiv preprint
arXiv:2005.08025 (2020).
[73] Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, and Neel Sundaresan. 2019. Pythia:
ai-assisted code completion system. In Proceedings of the 25th ACM SIGKDD
International Conference on Knowledge Discovery & Data Mining . 2727–2735.
[74] Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel
Sundaresan. 2020. Unit Test Case Generation with Transformers. arXiv preprint
arXiv:2009.05617 (2020).
[75] Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin
White, and Denys Poshyvanyk. 2019. An empirical study on learning bug-fixing
patches in the wild via neural machine translation.ACM Transactions on Software
Engineering and Methodology (TOSEM) 28, 4 (2019), 1–29.
[76] Marko Vasic, Aditya Kanade, Petros Maniatis, David Bieber, and Rishabh Singh.
2019. Neural program repair by jointly learning to localize and repair. arXiv
preprint arXiv:1904.01720 (2019).
[77] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In Advances in neural information processing systems . 5998–6008.
[78] Panagiotis Vekris, Benjamin Cosman, and Ranjit Jhala. 2016. Refinement Types
for TypeScript. In Proceedings of the 37th ACM SIGPLAN Conference on Program-
ming Language Design and Implementation (Santa Barbara, CA, USA) (PLDI
’16). Association for Computing Machinery, New York, NY, USA, 310–325.
https://doi.org/10.1145/2908080.2908110
[79] Murali Vijayaraghavan, Chaudhuri Swarat, and Jermaine Chris. 2017. Bayesian
Sketch Learning for Program Synthesis. CoRR.—-2017.—-Vol. abs/1703.05698.—-
1703.05698 (2017).
[80] Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and
Philip S Yu. 2018. Improving automatic source code summarization via deep rein-
forcement learning. In Proceedings of the 33rd ACM/IEEE International Conference
on Automated Software Engineering . 397–407.
[81] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R
Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural
language understanding. arXiv preprint arXiv:1804.07461 (2018).
[82] Song Wang, Devin Chollak, Dana Movshovitz-Attias, and Lin Tan. 2016. Bugram:
bug detection with n-gram language models. In Proceedings of the 31st IEEE/ACM
International Conference on Automated Software Engineering . 708–719.
[83] Song Wang, Taiyue Liu, and Lin Tan. 2016. Automatically Learning Semantic
Features for Defect Prediction. In Proceedings of the 38th International Conference
on Software Engineering (Austin, Texas)(ICSE ’16). Association for Computing Ma-
chinery, New York, NY, USA, 297–308. https://doi.org/10.1145/2884781.2884804
[84] Wenhan Wang, Ge Li, Bo Ma, Xin Xia, and Zhi Jin. 2020. Detecting Code Clones
with Graph Neural Network and Flow-Augmented Abstract Syntax Tree. In 2020
IEEE 27th International Conference on Software Analysis, Evolution and Reengi-
neering (SANER) . IEEE, 261–271.
[85] Wenhua Wang, Yuqun Zhang, Zhengran Zeng, and Guandong Xu. 2020. TranSˆ
3: A Transformer-based Framework for Unifying Code Summarization and Code
Search. arXiv preprint arXiv:2003.03238 (2020).
[86] Yanlin Wang, Lun Du, Ensheng Shi, Yuxuan Hu, Shi Han, and Dongmei Zhang.
2020. CoCoGUM: Contextual Code Summarization with Multi-Relational GNN
on UMLs.
[87] Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. 2019. Code generation as a
dual task of code summarization. In Advances in Neural Information Processing
Systems. 6563–6573.
[88] Huihui Wei and Ming Li. 2017. Supervised Deep Features for Software Functional
Clone Detection by Exploiting Lexical and Syntactical Information in Source
Code.. In IJCAI. 3034–3040.
[89] Martin White, Michele Tufano, Christopher Vendome, and Denys Poshyvanyk.
2016. Deep learning code fragments for code clone detection. In 2016 31st
IEEE/ACM International Conference on Automated Software Engineering (ASE) .
IEEE, 87–98.
[90] Frank F Xu, Zhengbao Jiang, Pengcheng Yin, Bogdan Vasilescu, and Graham
Neubig. 2020. Incorporating external knowledge through pre-training for natural
language to code generation. arXiv preprint arXiv:2004.09015 (2020).
[91] S. Yan, H. Yu, Y. Chen, B. Shen, and L. Jiang. 2020. Are the Code Snippets
What We Are Searching for? A Benchmark and an Empirical Study on Code
Search with Natural-Language Queries. In2020 IEEE 27th International Conference
on Software Analysis, Evolution and Reengineering (SANER) . 344–354. https:
//doi.org/10.1109/SANER48275.2020.9054840
[92] Ziyu Yao, Daniel S Weld, Wei-Peng Chen, and Huan Sun. 2018. StaQC: A Sys-
tematically Mined Question-Code Dataset from Stack Overflow. InProceedings of
the 2018 World Wide Web Conference . 1693–1703.
[93] Fangke Ye, Shengtian Zhou, Anand Venkat, Ryan Marucs, Nesime Tatbul,
Jesmin Jahan Tithi, Paul Petersen, Timothy Mattson, Tim Kraska, Pradeep Dubey,
et al. 2020. MISIM: An End-to-End Neural Code Similarity System. arXiv preprint
arXiv:2006.05265 (2020).
[94] Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig.
2018. Learning to Mine Aligned Code and Natural Language Pairs from Stack
Overflow. InInternational Conference on Mining Software Repositories (MSR) . ACM,
476–486. https://doi.org/10.1145/3196398.3196408
[95] Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-
purpose code generation. arXiv preprint arXiv:1704.01696 (2017).
[96] Hao Yu, Wing Lam, Long Chen, Ge Li, Tao Xie, and Qianxiang Wang. 2019. Neural
detection of semantic code clones via tree-based convolution. In 2019 IEEE/ACM
27th International Conference on Program Comprehension (ICPC) . IEEE, 70–80.
[97] Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong
Liu. 2019. A novel neural source code representation based on abstract syntax
tree. In 2019 IEEE/ACM 41st International Conference on Software Engineering
(ICSE). IEEE, 783–794.
[98] Ruiqi Zhong, Mitchell Stern, and Dan Klein. 2020. Semantic Scaffolds for
Pseudocode-to-Code Generation. arXiv preprint arXiv:2005.05927 (2020).
[99] Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019.
Devign: Effective vulnerability identification by learning comprehensive program
semantics via graph neural networks. InAdvances in Neural Information Processing
Systems. 10197–10207.

Evaluating Iterative Optimization Across 1000 Data Sets
Yang Chen Yuanjie Huang
LCSA ∗, ICT, CAS, China
and Graduate School, CAS, China
{chenyang,huangyuanjie}@ict.ac.cn
Lieven Eeckhout
Ghent University, Belgium
Lieven.Eeckhout@elis.UGent.be
Grigori Fursin
INRIA, Saclay, France
grigori.fursin@inria.fr
Liang Peng
LCSA, ICT, CAS, China
and Graduate School, CAS, China
pengliang@ict.ac.cn
Olivier Temam
INRIA, Saclay, France
olivier.temam@inria.fr
Chengyong Wu
LCSA, ICT, CAS, China
cwu@ict.ac.cn
Abstract
While iterative optimization has become a popular compiler opti-
mization approach, it is based on a premise which has never been
truly evaluated: that it is possible to learn the best compiler opti-
mizations across data sets. Up to now, most iterative optimization
studies ﬁnd the best optimizations through repeated runs on the
same data set. Only a handful of studies have attempted to exer-
cise iterative optimization on a few tens of data sets.
In this paper, we truly put iterative compilation to the test for
the ﬁrst time by evaluating its effectiveness across a large number
of data sets. We therefore compose KDataSets, a data set suite with
1000 data sets for 32 programs, which we release to the public. We
characterize the diversity of KDataSets, and subsequently use it to
evaluate iterative optimization. We demonstrate that it is possible to
derive a robust iterative optimization strategy across data sets: for
all 32 programs, we ﬁnd that there exists at least one combination
of compiler optimizations that achieves 86% or more of the best
possible speedup across all data sets using Intel’s ICC (83% for
GNU’s GCC). This optimal combination is program-speciﬁc and
yields speedups up to 1.71 on ICC and 2.23 on GCC over the high-
est optimization level ( -fast and -O3, respectively). This ﬁnding
makes the task of optimizing programs across data sets much eas-
ier than previously anticipated, and it paves the way for the practi-
cal and reliable usage of iterative optimization. Finally, we derive
pre-shipping and post-shipping optimization strategies for software
vendors.
Categories and Subject Descriptors D.3.4. [ Software: Program-
ming Languages]: Processors—Compilers, Optimization
General Terms Design, Experimentation, Measurement, Perfor-
mance
∗ Key Laboratory of Computer System and Architecture
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proﬁt or commercial advantage and that copies bear this notice and the full citation
on the ﬁrst page. To copy otherwise, to republish, to post on servers or to redistribute
to lists, requires prior speciﬁc permission and/or a fee.
PLDI’10, June 5–10, 2010, Toronto, Ontario, Canada.
Copyright c⃝2010 ACM 978-1-4503-0019/10/06. . . $10.00
Keywords Compiler optimization, Iterative optimization, Bench-
marking
1. Introduction
Iterative optimization has become a popular optimization strat-
egy [4, 10, 12, 22, 29] because of its simplicity while yielding
substantial performance gains. It is essentially based on repeat-
edly trying out a large number of compiler optimizations until the
best combination of compiler optimizations is found for a partic-
ular program (one ‘iteration’ is the evaluation of one combination
of compiler optimizations). From the start though, there has been
a nagging issue: how data set dependent is iterative optimization?
More precisely, if one selects a combination of optimizations based
on runs over one or a few data sets, will that combination still be
the best for other data sets?
Answering this question is key in order to conﬁdently use it-
erative optimization in practice. At the same time, this question is
difﬁcult to answer because there is no benchmark suite with a large
enough number of data sets to emulate the two most likely appli-
cation scenarios. Either a program is tuned through iterative opti-
mization before shipping by the software vendor so that it performs
well across a broad range of data sets; or a program is tuned after
shipping, across a large number of production runs performed by
users.
Several researchers have investigated how program behavior
and performance varies across data sets. Zhong et al. [32] present
two techniques to predict how program locality is affected across
data sets; they consider 6 programs using up to 6 data sets each.
Hsu et al. [20] compare the proﬁles generated using the 3 data sets
of some SPECint2000 programs, and observe signiﬁcant program
behavior variability across data sets for some programs. Several
studies focus on the impact of data sets on compiler optimization
parameterization and selection. In particular, Berube et al. [6] col-
lect 17 inputs per program on average for 7 SPECint2000 programs
to study inlining. Haneda et al. [16] use the 3 SPECint2000 data
sets to evaluate compiler optimizations. Mao et al. [24] collect be-
tween 8 and 976 inputs per program for 10 Java programs (150
inputs on average per program, 4 programs with more than 100 in-
puts) to investigate the impact of inputs on the selection of the best
garbage collector. Fursin et al. [14] collect 20 data sets for each
of the MiBench benchmarks — the same benchmark suite used
in this paper — and also evaluate the data set sensitivity of com-
piler optimizations. These iterative optimization studies and oth-
ers [4, 10, 12, 22, 29] underscore the fact that a signiﬁcant number
of iterations (tens or hundreds) are required to ﬁnd the best combi-
nation of compiler optimizations. However, to our knowledge, there
is no benchmark suite available with multiple hundreds of distinct
data sets per program. As a result, not only are researchers forced
to evaluate iterative optimization using an unrealistic experimental
setting, they are unable to answer the aforementioned fundamental
question about whether iterative optimization is effective across a
broad range of data sets.
In order to address this key question, we collect 1000 data sets
for 32 programs, mostly derived from the MiBench benchmark
suite [15]. Our results show that, for each benchmark, there is at
least one combination of compiler optimizations that achieves 86%
or more of the maximum speedup (i.e., the speedup obtained with
the best possible combination per data set) across all data sets.
This result has a signiﬁcant implication: it means that, in practice,
iterative optimization may often be largely data set insensitive. It
also means that a program can be optimized by a software vendor
on a collection of data sets before shipping, and will retain near-
optimal performance for most other data sets. So the problem
of ﬁnding the best optimization for a particular program may be
signiﬁcantly less complex than previously anticipated.
This paper is organized as follows. We present KDataSets in
Section 2, and characterize its diversity and coverage in Section 3;
we also compare KDataSets against the previously proposed Mi-
DataSets, which comes with 20 data sets per benchmark, and show
that a large population of data sets is indeed necessary to capture a
sufﬁciently wide range of program behaviors. In Section 4, we then
emulate realistic iterative optimization using the 1000 data sets in
KDataSets, devise a strategy for selecting the best possible combi-
nation of optimizations, and evaluate the impact of an unrealistic
experimental context on iterative optimization strategies. We also
discuss the scope and the general applicability of the results ob-
tained in this paper (Section 5). Finally, we summarize related work
(Section 6) and conclude (Section 7).
2. KDataSets: A 1000-Data Set Suite
As mentioned in the introduction, we have collected 1000 data sets
for each of our 32 benchmarks. Most of these benchmarks come
from the MiBench benchmark suite. MiBench [15] is an embed-
ded benchmark suite covering a broad spectrum of applications,
ranging from simple embedded signal-processing tasks to smart-
phone and desktop tasks. It was developed with the idea that desk-
tops and sophisticated embedded devices are on a collision course
(for both applications and architectures, e.g., the x86 Intel Atom
processor is increasingly used for embedded devices), calling for a
broad enough benchmark suite. We use a modiﬁed version of the
MiBench suite, which was evaluated across different data sets and
architectures [2, 14]. We also added bzip2 (both the compressor
and the decompressor) to our set of benchmarks, and we plan to
add and disseminate more programs over time. The benchmarks
are listed in Table 1; the number of source lines ranges from 130
lines for kernels, e.g., crc32, to 99,869 lines for large programs,
e.g., ghostscript.
Table 1 summarizes the various data sets in KDataSets; it de-
scribes the range in ﬁle size along with a description of how these
data sets were obtained. The data sets vary from simple num-
bers and arrays, to text ﬁles, postscript ﬁles, pictures and audio
ﬁles in different formats. Some data sets, such as the numbers
for bitcount as well as the numbers in the array for qsort, are
randomly generated. For other programs, such as dijkstra and
patricia, we built data sets that exhibit distinct characteristics in
terms of how the workload exercises different control ﬂow paths
and deals with different working set sizes — this was done based
on studying the benchmarks’ source code and our target domain
knowledge.
Program Data set Data set description
(# source lines) ﬁle size
bitcount (460) - Numbers: randomly generated in-
tegers
qsort1 (154) 32K-1.8M 3D coordinates: randomly gener-
ated integers
dijkstra (163) 0.06k-
4.3M
Adjacency matrix: varied matrix
size, content, percentage of discon-
nected vertices (random)
patricia (290) 0.6K-1.9M IP and mask pairs: varied mask
range to control insertion rate (ran-
dom)
jpeg d (13501) 3.6K-1.5M JPEG image: varied size, scenery,
compression ratio, color depth
jpeg c (14014) 16k-137M PPM image: output of jpeg c (con-
verted)
tiff 2bw (15477),
2rgba (15424),
dither (15399),
median (15870)
9K-137M TIFF image: from JPEG images
by ImageMagick converter (con-
verted)
susan c, e, s
(each 1376)
12K-46M PGM image: from jpeg images
by ImageMagick converter (con-
verted)
mad (2358) 28K-27M MP3 audio: varied length, styles
(ringtone, speech, music)
lame (14491), ad-
pcm c (210)
167K-36M WA VE audio: output of mad (con-
verted)
adpcm d (211) 21K-8.8M ADPCM audio: output of adpcm c
(converted)
gsm (3806) 83K-18M Sun/NeXT audio: from MP3 au-
dios by mad (converted)
ghostscript
(99869)
11K-43M Postscript ﬁle: varied page num-
ber, contents (slides, notes, papers,
magazines, manuals, etc.)
ispell (6522),
rsynth (4111),
stringsearch1
(338)
0.1K-42M Text ﬁle: varied size, contents
(novel, prose, poem, technical
writings, etc.)
blowﬁsh e (863) 0.6K-35M Any ﬁle: a mix of text, image, au-
dio, generated ﬁles
blowﬁsh d (863) 0.6K-35M Encrypted ﬁle: output of blow-
ﬁsh e
pgp e (19575) 0.6K-35M Any ﬁle: a mix of text, image, au-
dio, generated ﬁles
pgp d (19575) 0.4K-18M Encrypted ﬁle: output of pgp e
rijndael e (952) 0.6K-35M Any ﬁle: a mix of text, image, au-
dio, generated ﬁles
rijndael d (952) 0.7K-35M Encrypted ﬁle: output of rijndael d
sha (197) 0.6K-35M Any ﬁle: a mix of text, image, au-
dio, generated ﬁles
CRC32 (130) 0.6K-35M Any ﬁle: a mix of text, image, au-
dio, generated ﬁles
bzip2e (5125) 0.7K-57M Any ﬁle: a mix of above text,
image, audio, generated ﬁles, and
other ﬁles like program binary,
source code
bzip2d (5125) 0.2K-25M Compressed ﬁle: output of bzip2e
Table 1. KDataSets description.
The text ﬁles are collected from the Gutenberg project
(gutenberg.org) and python.org. Postscript ﬁles are collected
from various web sites: somethingconstructive.ne, oss.net,
ocw.mit.edu, samandal.org, pythonpapers.org, etc., and we
converted some of the collected PDF ﬁles into PS format. Images
are collected from public-domain-image.com and converted
into the different required formats (TIFF, JPEG, PGM, PPM). Au-
dio ﬁles are collected from freesoundfiles.tintagel.net,
jamendo.com, ejunto.com and converted again into the appro-
priate formats (WA VE, ADPCM, Sun/NeXT). For programs with
miscellaneous ﬁles as inputs (e.g., compression, encryption), we
use a mix of the aforementioned ﬁles. And in some cases, the output
of some programs are inputs to other programs, e.g., mad/adpcm c.
The entire data set suite corresponds to 27GB of data.
All data sets within KDataSets are free in the public domain or
under the Creative Commons license, and will be publicly dissem-
inated at http://www.kdatasets.org.
3. Data Set Characterization
In this section, we characterize KDataSets, analyze how distinct the
data sets are with respect to each other, and how differently they
react to compiler optimizations.
3.1 Experimental setup
Before doing so, we ﬁrst brieﬂy describe our experimental setup.
We consider 7 identical machines with 3GHz Intel Xeon dual-
core processors (E3110 family), 2GB RAM, 2 ×3MB L2 cache.
We use the CentOS 5.3 Linux distribution based on Red Hat with
kernel 2.6.18 patched to support hardware counters through the
PAPI library [3].
Most of the results in the paper are obtained using the Intel
ICC compiler v11.1. However, for completeness and in order to
demonstrate that the results in this paper translate to other compil-
ers, we have also conducted the same experiments using GNU GCC
v4.4.1; these results are similar to the ICC results and are reported
in Section 5.1. Both compilers feature a large number of optimiza-
tions. We selected 53 optimization ﬂags for ICC and 132 ﬂags for
GCC that control inlining, unrolling, vectorization, scheduling, reg-
ister allocation, constant propogation, among many others. We use
a random optimization strategy for creating combinations of opti-
mizations: a combination is based on the uniform random selection
of ﬂags; the number of ﬂags is itself randomly selected. Random
selection of compiler optimization combinations is a popular and
effective approach for exploring the compiler optimization design
space [8, 14, 27].
We consider 300 combinations of compiler optimizations for
each program/data set pair throughout the paper — we study the
results’ sensitivity to the number of combinations in Section 5 by
considering 8,000 combinations. For each combination and each
data set, we measure the program’s execution time. We also mea-
sure 9 performance characteristics using hardware performance
counters through the PAPI interface, such as L1, L2 and TLB
miss rates and branch prediction miss rates. And we also col-
lect 66 architecture-independent characteristics using the MICA
toolset [18], such as instruction mix, ILP, branch predictability,
memory access patterns, and working set size.
Finally, Table 2 summarizes the terminology and deﬁnitions
used throughout the text. We present this table here as a reference,
and we will introduce the deﬁnitions in the text itself as need be.
We use harmonic mean when reporting average speedup numbers.
3.2 Performance characterization
Figure 1 summarizes the performance characteristics for all pro-
grams and all data sets. The violin graphs show the distribution of
the execution time (in cycles), IPC as well as other performance
characteristics that relate to cache and branch behavior across all
data sets per program. All graphs are shown on a logarithmic scale.
The goal of this characterization is to underscore the differ-
ences among the data sets — this illustrates the broad range of
program behavior that these data sets generate. The execution time
distribution (left top graph) shows that the amount of work varies
greatly across data sets: while the standard deviation can be low,
as for tiffmedian, it is fairly high for most other benchmarks.
data sets: d ∈ D, |D| = 1000.
optimization combinations: o ∈ O, |O| = 300.
speedup of o on d: so
d.
data set optimal speedup of d:
soptimal
d = max{so
d, o∈ O}.
fraction of data set optimal speedup of o on d:
fo
d = so
d
soptimal
d
.
program-optimal combination: oopt:
ooptwith highest mininum fraction:
oopt = o/fo
d = maxo∈Omin{fo
d , d∈ D}.
ooptwith highest mininum speedup:
oopt = o/so
d = maxo∈Omin{so
d, d∈ D}.
ooptwith highest average fraction:
oopt = o/fo
d = maxo∈Omean{fo
d , d∈ D}.
ooptwith highest average speedup:
oopt = o/so
d = maxo∈Omean{so
d, d∈ D}.
Table 2. Deﬁnitions and terminology.
The other distributions (see the other graphs) show that not only
does the amount of work vary wildly across data sets, but also the
performance seen at the architecture level in terms of IPC and sev-
eral other performance characteristics (L1 and L2 miss rates, and
branch prediction rate).
3.3 How programs react to compiler optimizations across
data sets
We now investigate how programs react to compiler optimizations
across data sets. To this end, we compile each program with each
of the 300 randomly generated combinations of compiler optimiza-
tions, and then run each of these 300 program versions with each
of the 1000 data sets. We then record the best possible speedup for
each data set, and we will refer to this speedup as the data set op-
timal speedup , see also Table 2 for a formal deﬁnition. The data
set optimal speedup is the speedup relative to -fast and -O3 for
ICC and GCC, respectively. Because we have 1000 data sets per
program (and hence 1000 data set optimal speedup numbers), we
report these results as a distribution, see Figure 2 for the Intel ICC
compiler; we report similar results for GNU’s GCC in Section 5.1.
The violin plots thus show the distribution of the data set optimal
speedup for each benchmark. Figure 2(a) sorts the various bench-
marks by average data set optimal speedup. Figure 2(b) shows the
same data but sorts the benchmarks by the number of source code
lines; this illustrates that the impact of compiler optimizations is
uncorrelated with program size.
The results are contrasted. For some programs, the data set
optimal speedups are within ±1% of the average for more than
98% of the data sets, i.e., there is hardly any discrepancy among the
data sets. However, at the other end of the spectrum, ten programs
exhibit signiﬁcant variation in performance improvements, i.e., the
deviation from their average performance ranges from 10% to 25%.
One can also note that, even for programs with a relatively
small average performance improvement, there are some data sets
for which the performance improvement is signiﬁcant. Figure 3
illustrates this further. Here we have sorted all data sets and all
programs according to the data set optimal speedup they achieve.
For 14% of data sets, iterative optimization can achieve a speedup
of 1.1 or higher, and for nearly half of the data sets, it can achieve
a speedup of 1.05 or higher. The end conclusion is that iterative
compilation yields substantial performance improvements across
programs and data sets, and the signiﬁcance of the improvement is
sensitive to the data set.
We also ﬁnd that the reactions to compiler optimizations are
non-trivial, both across programs, and across data sets for the same
program. In Figures 4 and 5, we plot the mean, standard deviation,
Cycles − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − −
1e−03
1e−02
1e−01
1e+00
1e+01
1e+02
1e+03
1e+04
IPC
− − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − −
3e−01
4e−01
6e−01
1e+00
2e+00
L1_ICM_RATE − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − −
1e−02
1e−01
1e+00
1e+01
1e+02
1e+03
1e+04
susan_s
stringsearch1
susan_ctiff2rgbajpeg_d
sha
adpcm_ctiff2bwCRC32rijndael_d
mad
rijndael_e
rsynthpgp_ebzip2dqsort1
ghostscript
tiffditherbzip2elame
bitcounttiffmediansusan_eblowfish_e
patricia
blowfish_d
ispellpgp_djpeg_c
gsm
dijkstraadpcm_d
L1_DCM_RATE
− − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − −
1e−02
1e−01
1e+00
1e+01
1e+02
1e+03
L2_TCM_RATE
− − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − −
1e−02
1e−01
1e+00
1e+01
1e+02
1e+03
BR_MSP_RATE
− − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − −
1e−03
1e−02
1e−01
1e+00
1e+01
1e+02
1e+03
susan_s
stringsearch1
susan_ctiff2rgbajpeg_d
sha
adpcm_ctiff2bwCRC32rijndael_d
mad
rijndael_e
rsynthpgp_ebzip2dqsort1
ghostscript
tiffditherbzip2elame
bitcounttiffmediansusan_eblowfish_e
patricia
blowfish_d
ispellpgp_djpeg_c
gsm
dijkstraadpcm_d
Figure 1. Distribution of (normalized) performance characteristics across data sets (on a log scale): number of cycles, IPC, L1 instruction
cache miss rate, L1 data cache miss rate, L2 cache miss rate, branch misprediction rate.
Speedup
1.0
1.2
1.4
1.6
1.8
− − − − − − − − − − − − − − − − − − − − − − − − − − − − −
− −
−
susan_s
stringsearch1
susan_ctiff2rgbajpeg_d
sha
adpcm_ctiff2bwCRC32rijndael_d
mad
rijndael_e
rsynthpgp_ebzip2dqsort1
ghostscript
tiffditherbzip2elame
bitcounttiffmediansusan_eblowfish_e
patricia
blowfish_d
ispellpgp_djpeg_c
gsm
dijkstraadpcm_d
(a) sorted by average data set optimal speedup
Speedup
1.0
1.2
1.4
1.6
1.8
− −
−
− −
−
−
− − − − − − − − − −
−
− − − −
−
−
− − − − − − − −
CRC32qsort1dijkstra
sha
adpcm_cadpcm_dpatricia
stringsearch1
bitcount
blowfish_eblowfish_drijndael_erijndael_dsusan_esusan_csusan_s
madgsmrsynthbzip2ebzip2dispelljpeg_djpeg_clame
tiffdithertiff2rgbatiff2bw
tiffmedian
pgp_dpgp_e
ghostscript
(b) sorted by number of source code lines
Figure 2. Data set optimal speedups relative to -fast for
KDataSets (the horizontal line shows the mean value) .
and maximum and minimum speedup for each compiler optimiza-
tion, sorted by increasing mean speedup; this is done for the ﬁve
programs with the highest average speedup across all data sets. A
small arrow below the horizontal axis indicates the baseline com-
bination ( -fast). The large variations for the standard deviation
show that the impact of each combination can vary strongly: for
some programs, it is almost stable across all data sets, e.g., gsm
(Figure 5(b)), or there are large variations across data sets (all other
examples).
 1
 1.1
 1.2
 1.3
 1.4
 1.5
 1.6
 1.7
 1.8
 0  20  40  60  80  100
Speedup
Percentage of data sets across all programs
Figure 3. Distribution of the data set optimal speedups across all
data sets and all programs.
3.4 KDataSets versus MiDataSets
We now compare KDataSets (1000 data sets) against MiDataSets
(20 data sets) [14]. The reason for doing so is to investigate whether
a 1000-data set suite is really needed compared to a 20-data set
suite. The more fundamental question is whether the 20 data sets
in MiDataSets cover most of the space built up by the 1000 data
sets. In other words, do the 1000 data sets cover a much broader
spectrum of the data set space to justify its usage over the 20
data sets that were previously collected? This is valid question,
especially given that the data set optimal speedup are comparable
for KDataSets (Figure 2) compared to MiDataSets (see Figure 6).
An in-depth analysis that involves a broad set of both micro-
architecture-dependent and microarchitecture-independent charac-
teristics reveals that the coverage of the program behavior space
varies signiﬁcantly across data sets. For that purpose, we have col-
lected the 66 microarchitecture-independent features provided by
0.80
0.90
1.00
1.10
1.20
1.30
1.40
1.50
1.60
1.70
1.80
 0  50  100  150  200  250  300
 0
 0.05
 0.1
 0.15
 0.2
Speedup
Stdev
combinations
max
mean
min
stdev
Figure 4. Reactions to compiler optimizations for adpcm d.
0.60
0.70
0.80
0.90
1.00
1.10
1.20
1.30
1.40
 0  50  100  150  200  250  300
 0
 0.05
 0.1
 0.15
 0.2
Speedup
Stdev
combinations
max
mean
min
stdev
(a) dijkstra
0.90
1.00
1.10
1.20
1.30
 0  50  100  150  200  250  300
 0
 0.05
 0.1
 0.15
 0.2
Speedup
Stdev
combinations
max
mean
min
stdev (b) gsm
0.90
1.00
1.10
1.20
1.30
 0  50  100  150  200  250  300
 0
 0.05
 0.1
 0.15
 0.2
Speedup
Stdev
combinations
max
mean
min
stdev
(c) jpeg c
0.80
0.90
1.00
1.10
1.20
1.30
1.40
1.50
 0  50  100  150  200  250  300
 0
 0.05
 0.1
 0.15
 0.2
Speedup
Stdev
combinations
max
mean
min
stdev (d) pgp d
Figure 5. Reactions to compiler optimizations for dijkstra, gsm,
jpec c, pgp d.
the MICA tool v0.22 [18] and 9 microarchitecture-dependent fea-
tures using hardware performance counters as mentioned in Sec-
tion 3.1. For each program, we apply principal component analysis
(PCA) on the characteristics collected for all data sets, following
the methodology by Eeckhout et al. [11]. We then plot the data sets
along the two most signiﬁcant principal components in Figure 7
— these principal components capture the most signiﬁcant dimen-
sions; we show plots for only a few representative programs due
to paper length constraints. MiDataSets and KDataSets are shown
using crosses and dots, respectively. KDataSets covers a larger part
of the space than MiDataSets for 25 out of 32 programs, see for
example patricia, ghostscript, bitcount, pgp d. KDataSets
substantially expands the space covered compared to MiDataSets
for 4 out of 32 programs, e.g., gsm, stringsearch1. KDataSets
and MiDataSets cover roughly the same space for three programs,
such as lame and adpcm d.
We have now made the case that KDataSets exhibits signiﬁ-
cantly different behavior compared to MiDataSets. However, this
does not necessarily imply that KDataSets will react very differ-
ently to compiler optimizations. We therefore conduct yet another
Speedup
1.0
1.1
1.2
1.3
1.4
1.5
1.6
− − − − − − − − − − − − − − − − − − − − − − − −
−
− − − −
−
−
−
susan_s
stringsearch1
susan_ctiff2rgbajpeg_d
sha
adpcm_ctiff2bwCRC32rijndael_d
mad
rijndael_e
rsynthpgp_ebzip2dqsort1
ghostscript
tiffditherbzip2elame
bitcounttiffmediansusan_eblowfish_e
patricia
blowfish_d
ispellpgp_djpeg_c
gsm
dijkstraadpcm_d
Figure 6. Data set optimal speedups relative to -fast for Mi-
DataSets.
analysis using PCA with the features being the speedups obtained
for each of the combinations, see Figure 8. The conclusion is es-
sentially the same as before: the graphs clearly illustrate the greater
diversity of reactions to compiler optimizations for KDataSets rel-
ative to MiDataSets.
4. Iterative Optimization in Practice
In this section, we investigate the stability of iterative optimization
across data sets and revisit iterative optimization in a realistic ex-
perimental context for two application scenarios.
4.1 Program-optimal combinations
In order to investigate how sensitive the selection of combinations
is to data sets, we ﬁrst determine the data set optimal speedup, i.e.,
this is the highest speedup for each data set across all 300 combi-
nations. Then, for each program, we retain the combination which
yields the best average performance across all data sets. We term
this combination the program-optimal combination, see Table 2 for
a formal deﬁnition; we will clarify the exact selection process in
the next section. In Figure 9, we report the performance for all data
sets and for each program compiled with its program-optimal com-
bination. The distribution of speedups (relative to -fast) across
data sets is shown at the top, while the distribution of the fraction
of the data set optimal speedup is shown at the bottom; the fraction
of the data set optimal speedup is the ratio of the program-optimal
combination speedup over the data set optimal speedup (see also
Table 2) and it is always less than or equal to 1. The key obser-
vation is that, for each program, a single combination can achieve
at least 86% of the data set optimal speedup for all data sets, with
most programs standing at a minimum of 90% of the data set opti-
mal speedup. As mentioned in the introduction, the consequences
are signiﬁcant. This result conﬁrms that iterative optimization is ro-
bust across data sets: after learning over a sufﬁcient number of data
sets, the selected best tradeoff combination is likely to perform well
on yet unseen data sets.
4.2 How to select the program-optimal combination
We now explain in more detail our strategy for selecting the
program-optimal combination. In Table 3(a), we show an illustra-
tive example with two data sets D1 and D2 and three combinations
comb1, comb2 and comb3 plus the baseline ( -fast for ICC). For
each program, we evaluate every combination on every data set,
deduce the data set optimal speedup, and, in the next two columns,
compute the fraction of the data set optimal speedup achieved by
every combination on every data set. We then want to select a com-
bination that maximizes performance across all data sets. So we
ﬁnd the minimum fraction of the data set optimal speedup achieved
by each combination (rightmost column), and pick the combination
with the highest minimum fraction of the data set optimal speedup.
+ + + + + + + + + + ++++++++++
−20 −15 −10 −5 0 5 10
−20 −15 −10 −5 0 5 10 15
patricia
+++
+++++++++ +++
+++ +
+
−20 −10 0 10 20 30 40 50
−20 −15 −10 −5 0 5 10
ghostscript
+
+
+
++ +
+
+
+
+ +
+
+
+
+ + +
+
+ +
−10 −5 0 5 10
−50 −40 −30 −20 −10 0 10
adpcm_d
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
−15 −10 −5 0 5 10 15
−15 −10 −5 0 5 10 15 20
lame
+
+
+
+
+
+
+
+
+
+
+
+
++
++
++
+
+
−15 −10 −5 0 5 10 15 20
−20 −10 0 10
pgp_d
+
+
+
+
+++++++
+
+ + ++++++
−20 −15 −10 −5 0 5
−30 −20 −10 0 10 20
bitcount
+
+
+
+
++
+
+
+
+
+
+
++
+ +
+
+
+
+
−20 0 20 40 60 80 100
−25 −20 −15 −10 −5 0 5 10
stringsearch1
+
+
+
++ +
++
+
+
+
+ +
+
+ +
+
+
+ +
−10 0 10 20 30 40
−50 −40 −30 −20 −10 0 10
gsm
Figure 7. Data set characterization using principal component analysis: the features are a set of microarchitecture-dependent and
microarchitecture-independent characteristics; KDataSets is shown using gray dots, and MiDataSets is shown using black crosses.
+
+
+
+ +
+
++ ++ ++++++++++
−0.6 −0.4 −0.2 0.0 0.2 0.4 0.6
−1.5 −1.0 −0.5 0.0 0.5 1.0
patricia
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
−0.4 −0.2 0.0 0.2 0.4 0.6 0.8
−2 −1 0 1
ghostscript
+
+
+
+
+
+
+ ++
+
+
+
+
+
+
+
+
+
+
+
−3 −2 −1 0 1 2
−4 −2 0 2 4 6
adpcm_d
+
+
+
+
+ +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
−0.2 −0.1 0.0 0.1 0.2 0.3 0.4
−0.6 −0.4 −0.2 0.0 0.2 0.4
lame
+
+
+
+
+
+
+
+
+
+
+
+ +
+
++
+
+
+
+
−1.5 −1.0 −0.5 0.0 0.5
−2 −1 0 1 2 3 4 5
pgp_d
++ + + +++++++
+
+
+
+ + ++
++
−0.1 0.0 0.1 0.2 0.3
−0.1 0.0 0.1 0.2 0.3
bitcount
+
+
+++++
++
+
++
+
+
+
++
+
+
+
−0.4 −0.2 0.0 0.2 0.4 0.6 0.8 1.0
−0.5 0.0 0.5 1.0 1.5
stringsearch1
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
++
+
+
−0.04 −0.02 0.00 0.02 0.04 0.06 0.08
−0.1 0.0 0.1 0.2 0.3 0.4 0.5
gsm
Figure 8. Data set characterization using principal component analysis: the features are the speedup numbers across the 300 combinations
of compiler optimizations; KDataSets is shown using gray dots, and MiDataSets is shown using black crosses.
Speedup
0.9
1.0
1.1
1.2
1.3
1.4
1.5
1.6
− − − − − − − − − − − − − − − − − − − − − − − − − − − − −
− −
−
Fraction
0.86
0.88
0.90
0.92
0.94
0.96
0.98
1.00 − − − − − − − − −
−
−
−
− −
−
−
−
−
−
+
− − −
− − − −
− −
− −
−
susan_s
stringsearch1
susan_ctiff2rgbajpeg_d
sha
adpcm_ctiff2bw
rijndael_dCRC32
mad
rijndael_e
rsynthpgp_ebzip2dqsort1
ghostscript
tiffditherbitcount
lamebzip2e
tiffmediansusan_eblowfish_eblowfish_d
patriciaispellpgp_djpeg_c
gsm
dijkstraadpcm_d
Figure 9. Speedup (top graph) and fraction of the data set optimal
speedup (bottom graph) for the program-optimal combinations.
(a)
Speedup Fraction data set
optimal speedup
Comb. D1 D2 Avg. D1 D2 Min
baseline 1.00 1.00 1.00 0.77 0.91 0.77
comb1 1.20 1.10 1.15 0.92 1.00 0.92
comb2 1.30 1.07 1.19 1.00 0.97 0.97
comb3 1.25 1.05 1.15 0.96 0.95 0.95
data set
optimal
1.30 1.10 1.20 1.00 1.00 1.00
(b)
Speedup Fraction data set-
optimal speedup
Comb. D1 D2 Avg. D1 D2 Min
baseline 1.00 1.00 1.00 0.77 0.99 0.77
comb1 1.20 1.01 1.11 0.92 1.00 0.92
comb2 1.30 0.98 1.14 1.00 0.97 0.97
comb3 1.25 0.99 1.12 0.96 0.98 0.96
data set
optimal
1.30 1.01 1.16 1.00 1.00 1.00
Table 3. Illustrative examples showing (a) how to select the
program-optimal combination with the highest minimum fraction
of the data set optimal speedup; (b) shows a case for which the
program-optimal combination leads to slowdown relative to the
baseline.
Program-optimal combinations may sometimes, though rarely,
induce slowdowns compared to the baseline. Let us deﬁne the data
set optimal speedup for each data set as B; and let us deﬁne the
fraction of the data set optimal speedup that the program-optimal
speedup achieves as M. If the data set optimal speedup is small, it
may be the case that B ×M <1, i.e., a slowdown compared to the
baseline. Because this only happens when B is small, the resulting
slowdown is small as well. Consider the example in Table 3(b):
because the data set optimal speedup for data set D2 is small, the
fraction of the data set optimal speedup for comb2 and comb3 is
high even though it induces a slight slowdown. As a result, comb2
gets selected as the combination with the highest minimum fraction
of the data set optimal speedup; on average across D1 and D2,
it does induce a signiﬁcant average speedup, but there is a slight
slowdown for D2. In Figure 9, we can see that this phenomenon
happens (in top graph, violin part below speedup equal to one),
though infrequently. For instance, programs such as patricia
exhibit slowdowns for a few of their data sets. It is relatively
more frequent for a couple programs such as rijndael e; the
Speedup
0.90
0.95
1.00
1.05
1.10
+ − − −
Fraction
0.88
0.90
0.92
0.94
0.96
0.98
1.00
− − − −
speedup_minspeedup_mean
fraction_minfraction_meanspeedup_minspeedup_mean
fraction_minfraction_meanspeedup_minspeedup_mean
Figure 10. Different
ways for selecting
program-optimal com-
binations for CRC32.
 0
 4
 8
 12
 16
 20
 24
 28
 32
100 99 98 97 96 95 94 93 92 91
# programs covered
% of optimial value of all criteria
Figure 11. Evaluation of the compro-
mise selection strategy for determining the
program-optimal combination.
most extreme example is CRC32 for which the program-optimal
combination yields a slowdown on average.
4.3 Alternative ways for selecting program-optimal
combinations
There are different ways for how to determine the program-optimal
combination. Depending on the user’s objectives, it is possible to
tune the selection of the program-optimal combination to minimize
risk (maximize the minimum speedup), or to maximize average
performance (maximize average speedup or maximize the average
fraction of the data set optimal speedup). As an example, we show
in Figure 10 the performance for program-optimal combinations of
CRC32 selected using these different criteria. While some exhibit an
average slowdown, some result in an average speedup. Below, we
study these different selection criteria and outline a general strategy
for selecting program-optimal combinations.
We consider four different criteria for selecting the program-
optimal combination: pick the combination that maximizes (i) the
average speedup (across all data sets), or (ii) the minimum speedup,
or (iii) the minimum fraction of the data set optimal speedup, or (iv)
the average fraction of the best speedup. While the best criterion
really depends on the user’s objective, we propose the compromise
selection strategy that performs well under all criteria. The strategy
determines a set of program-optimal combinations that perform
best under each criterion: each set for a criterion contains the
combinations that achieve at least P% of the optimal value for that
criterion. Then, we determine the intersection of these four sets of
combinations. Each combination in this intersection is eligible to be
a compromise program-optimal combination; we randomly pick a
compromise program-optimal combination within that intersection.
If the intersection is empty, we lower the value of P (from 100%
downwards) and try again.
In Figure 11, we show the fraction of programs for which we
can ﬁnd at least one combination that achieves at least P% of the
optimal value for all four criteria. For all programs, it is possible
to ﬁnd combinations that achieve 91% of the optimal value of
all criteria; 95% of the optimal value is achieved for all but one
program. As a result, the end conclusion is that, independently
of the user’s objective, there is almost always a combination that
performs well for every criterion, at least among those tested above.
4.4 How many data sets are needed
One remaining issue for having a truly robust program optimization
approach is to understand how many data sets are required in order
to ﬁnd a good program-optimal combination.
Program Percentage of program-optimal performance
85% 90% 95% 99%
adpcm d 2 3 5 11
dijkstra 2 2 2 5
gsm 2 2 2 7
jpeg c 1 1 2 21
pgp d 7 12 21 40
ispell 2 2 2 3
patricia 1 2 2 3
blowﬁsh d 1 2 6 154
blowﬁsh e 1 2 2 54
susan e 122 145 179 259
tiffmedian 2 2 2 153
bzip2e 2 2 2 467
lame 2 2 2 2
bitcount 1 1 1 1
tiffdither 1 1 2 2
ghostscript 2 2 2 90
qsort1 1 2 16 34
bzip2d 1 32 74 340
pgp e 2 2 2 294
rsynth 1 2 17 43
rijndael e 2 20 130 623
mad 2 2 2 7
CRC32 2 2 138 326
rijndael d 2 4 59 845
tiff2bw 1 1 2 615
adpcm c 1 2 2 6
sha 8 21 41 231
jpeg d 1 2 2 8
tiff2rgba 1 1 2 856
susan c 1 1 2 7
stringsearch1 2 2 8 760
susan s 2 2 60 119
Table 4. Minimum number of data sets required to select a combi-
nation achieving X% of the program-optimal speedup; the bench-
marks are sorted by their average data set optimal speedups.
To do so, we set up the following experiment. We randomly
sample N (out of 1000) data sets, select the program-optimal com-
bination across these N data sets, evaluate its performance on all
of the 1000 data sets, and deduce what percentage of the program-
optimal combination performance is achieved. For each N, we
repeat this process 1000 times and compute the average percent-
age. We then report the minimum number of data sets that are re-
quired to select a combination with a performance within 85%,
90%, 95% and 99% of the performance obtained when comput-
ing the program-optimal combination across all 1000 data sets, see
Table 4.
The results vary across programs. For 14 out of the 32 programs,
we achieve 85% of the program-optimal performance using only a
single data set; only 1 program can achieve 95% of the program-
optimal performance with a single data set. For 18 programs, at
least 2 data sets are needed to achieve within 85% of the program-
optimal speedup. The top ﬁve programs still need 6 data sets on
average to achieve 95% of the program-optimal performance. Four
programs require several tens, or even several hundreds, of data
sets to achieve 90% of the program-optimal performance, and 14
programs require between 154 and 856 data sets to achieve 99% of
the program-optimal performance.
In summary, about nine-tenths of the programs require from one
to a few tens of data sets to achieve near-optimal performance, and
the remaining tenth requires hundreds of data sets.
4.5 Evaluating iterative optimization within a realistic
experimental context
The two most typical use cases for iterative optimization are: (i) the
software vendor optimizes the software before shipping, and (ii) the
end user is willing to tune the software she/he is using. There is one
Average speedup
0.8
0.9
1.0
1.1
1.2
1.3
1.4
− − − − − − − − − − − − − − − − − − − − − − − − − − − − −
− −
−
susan_s
stringsearch1
susan_ctiff2rgbajpeg_d
sha
adpcm_ctiff2bwCRC32rijndael_d
mad
rijndael_e
rsynthpgp_ebzip2dqsort1
ghostscript
tiffditherbzip2elame
bitcounttiffmediansusan_eblowfish_e
patricia
blowfish_d
ispellpgp_djpeg_c
gsm
dijkstraadpcm_d
Figure 12. Iterative optimization using a single data set versus us-
ing 1000 data sets; the horizontal line corresponds to the aver-
age speedup using the program-optimal combination selected over
1000 data sets.
signiﬁcant difference between these two cases for the iterative opti-
mization learning process. In the ﬁrst case, the software vendor can
repeatedly run the same data sets, and thus easily compare the per-
formance impact of combinations across runs. In the second case,
the user is only doing production runs, so each run corresponds to
a different data set. Most users are not willing to dedicate time, i.e.,
they do not want to perform repeated runs of the same data sets, in
order to tune their compiler combinations; they instead want this
process to happen transparently. As a result, the iterative optimiza-
tion process must learn across different data sets, making it difﬁcult
to compare the performance impact of compiler combinations.
Until now, due to the lack of a large number of data sets, itera-
tive optimization was evaluated in either scenario using unrealistic
hypotheses. In the pre-shipping case, the evaluation was often con-
ducted using a single data set. In the post-shipping case, the eval-
uation assumed that a user would be willing to run the same data
sets multiple times. In the sections below, we use our large num-
ber of data sets to create realistic conditions for both scenarios, and
to truly assess the potential performance beneﬁts of iterative opti-
mization in each scenario.
4.5.1 Pre-shipping scenario
In this scenario, we assess the performance beneﬁt of tuning a
software using a large number of data sets rather than a single one,
see Figure 12. For the one-data set case, we compute the speedup as
follows. For each program, we pick data sets one by one. For each
data set, we run all possible combinations, select the combination
with the maximum speedup, and apply it to all other 999 data sets;
the average data set speedup is computed over the 999 data sets
and the training data set. We report the corresponding distribution
in Figure 12, as well as the average speedup obtained with the
program-optimal combination selected using 1000 data sets (thick
horizontal bar). The fact that the bulk of most violins is below the
bar, and in most cases well below the bar, indicates that this training
also often leads to sub-optimal performance. The fact that several
violins are elongated also indicates that single-data set training is
unstable/unreliable, i.e., performance will vary wildly depending
on which data set is chosen.
4.6 Post-shipping scenario
In the post-shipping scenario, since we can only run each data set
once, it is impossible to compute the speedup. This experimental
context differs in two ways from the unrealistic context used in
most iterative optimization studies. First, the relative ordering of
combinations is imprecise: an experiment with a particular data set
may conclude that one combination outperforms another, whereas
Realistic
42 combs, speedups
300 combs, comparisons
0.90
1.00
1.10
1.20
1.30
1.40
susan_s
stringsearch1
susan_ctiff2rgbajpeg_d
sha
adpcm_ctiff2bwCRC32rijndael_d
mad
rijndael_e
rsynthpgp_ebzip2dqsort1
ghostscript
tiffditherbzip2elame
bitcounttiffmediansusan_eblowfish_e
patricia
blowfish_d
ispellpgp_djpeg_c
gsm
dijkstraadpcm_d
aver
Average speedup
Figure 13. Evaluation of a realistic versus unrealistic evaluation
methodology in a post-shipping scenario.
an experiment with another data set may conclude the opposite.
Second, the number of combinations that can be evaluated is lim-
ited by the number of available data sets.
In order to compare combinations, we resort to the run-time
approach proposed by Stephenson et al. [30] and Fursin et al. [13].
We use function cloning where each clone is compiled differently;
this is akin to versioning: the code contains two versions of the
same routines compiled using two different combinations. At run-
time, either version of each method is randomly used upon each
call, and the execution time is collected for each call, allowing to
compare the sum of the execution times of the methods for each
version. Even though this comparison is statistical, because of the
discrepancies in numbers and durations of calls, previous research
works have shown that it is reliable in practice and comes with low
overhead [13, 30].
In order to realistically emulate iterative optimization in a post-
shipping scenario, we must evaluate the combination selection
strategies on unseen data sets. So we split the data sets into 900
training data sets and 100 test data sets. We randomly deﬁne one
thousand such 900/100 training/test sets, and conduct the evalua-
tions below for all sets, and report the average results.
We now have to select the set of combinations to be evaluated.
When optimizing one program, it is realistic to leverage the expe-
rience gathered from other programs. Therefore, for each program,
we start with a set of high-potential combinations; these are the
program-optimal combinations for the other 31 programs. In or-
der to evaluate N = 31combinations, we need N×(N−1)
2 = 465
comparisons and therefore need that many data sets. With 900 data
sets at our disposal, we can increase the initial set of 31 combina-
tions with Nr randomly selected combinations, in order to increase
the probability of discovering new better suited combinations. The
only constraint is that (N+Nr)×(N+Nr−1)
2 ≤ 900; we ﬁnd that the
maximum value of Nr is 11, so we add 11 randomly selected com-
binations, for a total of 42 combinations.
We now compare the realistic case where 42 combinations are
evaluated using 42×41
2 = 861 runs versus the unrealistic case
where the same set of combinations are evaluated on all data sets.
The comparison between the realistic case, see the ‘ Realistic’
bars, and the unrealistic case, see the ‘ 42 combs, speedups’
bars in Figure 13, helps discriminate the impact of comparing
on distinct data sets. While the performance obtained under the
realistic experimental conditions is slightly below the unrealistic
case, we ﬁnd that it is often close for most programs, and barely
different on average.
-fast -ﬁnline-limit=8 -no-vec
-fast -ansi-alias -ﬁnline -no-vec
-fast -no-vec
-O2
-fast -nolib-inline
-fast -fno-builtin -no-inline-factor
-fast -inline-max-total-size=3000 -no-inline-factor
-fast -inline-max-size=1024 -ipo3
-fast -no-inline-factor -unroll8
-fast -fno-fnalias -fno-omit-frame-pointer -no-vec
-fast -fno-alias -fpack-struct -no-vec
-fast -ansi-alias -ﬁnline-limit=512 -nolib-inline
-fast -ﬁnline-limit=512 -opt-jump-tables=large
-O3 -ipo1 -no-opt-jump-tables
-fast -no-inline-min-size
-fast -ﬁnline-limit=32 -fno-builtin -opt-malloc-options=2
-fast -ip-no-inlining -nolib-inline -opt-malloc-options=1
-fast -ipo2
Table 5. The Intel ICC compiler optimizations that affect perfor-
mance the most, obtained after pruning the program-optimal com-
binations.
We also compare the realistic case against a case where all com-
binations are compared, though using distinct data sets, in order to
discriminate the impact of evaluating a restricted set of combina-
tions. Since we do not have 300×299
2 = 44850 data sets, we ap-
proximate this scenario by reusing data sets (i.e., for comparing
each pair of combinations, we randomly pick a data set), and re-
port the result in Figure 13, see ‘ 300 combs, comparisons’. We
again ﬁnd that the performance of the realistic case is very close to
the performance of the less realistic case.
This result is signiﬁcant. It implies that, even under realistic ex-
perimental conditions, it is possible to achieve almost the same per-
formance as the upper bound performance deﬁned by less realistic
experimental conditions. In other words, the potential of iterative
optimization can indeed be achieved in practice, even in a post-
shipping scenario.
4.7 Analysis: Key compiler optimizations
We now delve a little deeper in the composition of program-
optimal combinations. These combinations may contain more than
50 optimization ﬂags, making it difﬁcult to understand which op-
timizations really impact performance. We therefore prune the
program-optimal combinations by simply removing optimization
ﬂags one by one, randomly, as long as they do not change the
overall speedup. Table 5 shows the top performing combinations
after pruning across all programs and data sets for ICC. (There are
fewer pruned combinations than programs: program-optimal com-
binations may prune to the same combinations or the baseline.)
This list of top combinations underscores that, even though mod-
ern compilers have powerful optimizations such as vectorization,
inter-procedural analysis, etc., the heuristics in charge of activat-
ing these different optimizations and selecting their parameters
may not be effective. For example, the -fast optimization level in
ICC includes inlining and vectorization by default, but both opti-
mizations may degrade performance. Iterative optimization does a
better job at applying these optimizations to particular programs:
e.g., only apply inlining to programs with small functions, or turn
off vectorization when it is detrimental to performance.
5. Discussion on scope of the results
The results in this paper are obviously tied to the experimental
setup. In this section, we aim at gaining insight into how general
the results and conclusions are.
Speedup
1.0
1.5
2.0
− − − − − − − − − − − − − − − − − − − − − − − − − − − − −
−
−
−
Fraction
0.80
0.85
0.90
0.95
1.00 − − − − −
− − − − − − − − − − −
− −
− − − − −
− −
− − −
−
− − +
mad
rijndael_eblowfish_erijndael_dCRC32
blowfish_d
pgp_dqsort1
adpcm_dghostscript
patriciaispell
tiffditherdijkstra
gsmpgp_ebzip2djpeg_dtiff2rgba
lametiff2bwjpeg_csusan_sadpcm_cbzip2ebitcountsusan_e
stringsearch1
tiffmedian
sha
susan_crsynth
Figure 14. Speedup (top graph) and fraction of the data set opti-
mal speedup (bottom graph) for the program-optimal combinations
for GNU’s GCC.
5.1 Results for GNU’s GCC
We used Intel’s ICC compiler throughout the paper. We now eval-
uate whether the conclusions also apply to other compilers. We
therefore consider GNU’s GCC compiler v4.4.1. Figure 14 reports
the data set optimal speedups (with respect to -O3) that can be
achieved using GCC. Compared to Figure 9, we observe that GCC
achieves higher speedups than ICC — this is relative to the -O3 and
-fast baseline combinations, respectively. It is also interesting to
note that the best speedups are achieved for different programs for
the most part. More importantly, we ﬁnd again that, for each pro-
gram, a single combination can achieve 83% of the data set optimal
speedup for all data sets, with most programs standing at 90% or
higher. Therefore, our conclusions are valid for two of the most
widely used compilation platforms.
5.2 Data sets coverage
There is no guarantee that our collection of 1000 data sets covers
the entire program behavior space — there may still be parts of the
space that are unrepresented by KDataSets. For instance, our initial
attempt at collecting data sets for program tiffmedian resulted
in the distribution shown in Figure 15(a). The 20 data sets did
in fact a better job at covering the program behavior space than
our initial 1000 data sets. This turned out to be due to the lack of
very small images ( < 29kB). This is because these small images
exhibit a signiﬁcantly different L1, L2 and DTLB behavior than
large images ( 29kB − 1.5MB), resulting in signiﬁcantly different
and more spread out program behavior. After increasing the mix
of small and large images, the coverage of the 1000 data sets
was eventually superior to that of the 20 data sets, as shown in
Figure 15(b); the circles in Figure 15(b) correspond to the data sets
added to our initial set, the withdrawn redundant data sets do not
show. However, even the notion of ‘covering the program behavior
space’ is subjective because a given user may be interested in a
limited type of data sets only.
5.3 Compiler combinations
There is also no guarantee that the 300 combinations we explored
covers the entire compiler optimization space. To see whether 300
combinations is too small a number, we conducted an additional
experiment in which we consider 8000 combinations. However, to
complete it in a reasonable amount of time, we had to run each
combination on 10 randomly chosen data sets instead of all the
1000 data sets. The data set optimal speedups across these 8000
+
+
++++ ++ +
+ +
+
+
+
+
+
++ ++
−50 −40 −30 −20 −10 0 10
−40 −30 −20 −10 0 10 20
(a) initial 1000 data sets
o
o
o
o
o
o
o
o
oo
ooo oooo
o
o
o
o
oo
o
o
oo
o
o o
o
o
o o
oo
oo
oo
o
o
o o
o
o
o
o
oo
o
o
o
o
oo
o
o
o
o
+
+
++++ ++ +
+ +
+
+
+
+
+
++ ++
−50 −40 −30 −20 −10 0 10
−40 −30 −20 −10 0 10 20 (b) including small images
Figure 15. Program behavior space of tiffmedian.
Speedup
1.0
1.1
1.2
1.3
1.4
1.5
1.6
− − − − − − − − − − − − − − − − − − − − − − − − − − − − −
− −
−
susan_s
stringsearch1
susan_ctiff2rgbajpeg_d
sha
adpcm_ctiff2bwCRC32rijndael_d
mad
rijndael_e
rsynthpgp_ebzip2dqsort1
ghostscript
tiffditherbzip2elame
bitcounttiffmediansusan_eblowfish_e
patricia
blowfish_d
ispellpgp_djpeg_c
gsm
dijkstraadpcm_d
Figure 16. Data set optimal speedups relative to -fast for 8000
combinations.
combinations are reported in Figure 16. Compared to Figure 2(a),
except for a few data sets (e.g., in sha and adpcm d), we observe
no signiﬁcant difference overall, which suggests that 300 combina-
tions may be enough to represent the optimization space that can
be explored in a reasonable time window.
5.4 Platforms and benchmarks
More generally, we cannot assert that our conclusions generalize
beyond our benchmark suite, compiler platforms (Intel ICC and
GNU GCC) and target architecture. There are also obvious exam-
ples where the performance of program segments is data set sen-
sitive; auto-tuned libraries like FFTW [25] or ATLAS [31] are ex-
amples where data set dependent tuning is useful. Nevertheless, our
results suggest that, in general, the data set sensitivity problem may
have been overstated. We also note that our results are consistent
across all benchmarks so far.
5.5 Fine-grain iterative optimization
For now, we only considered whole-program optimizations, i.e.,
a combination of compiler optimizations is applied across the en-
tire program. However, a more ﬁne-grain iterative optimization ap-
proach may apply different combinations to different sections of the
code (e.g., functions or loops). Fine-grain iterative optimization is
more complicated because it introduces optimization dependences
(and possibly ripple effects): e.g., applying a combination to func-
tion F1 can affect the behavior of another function F2, and thus the
choice of its best combination. This dependence complicates the
problem as well as largely increases the design space. We will ex-
plore practical ways to tackle that issue as part of our future work.
5.6 Measurement bias
Recent work by Mytkowicz [26] raises the issue of measurement
bias, and provides evidence for two sources of measurement bias,
namely link order and environment size. We believe that link or-
der should not be considered measurement bias in the context of
iterative optimization. Link order should rather be viewed as an-
other opportunity for iterative optimization. Environment size af-
fects only one fourth of the benchmark programs in [26] by a small
margin only (within ±2% using ICC). In our study, 73% of pro-
gram/dataset pairs have a speedup that is higher than 1.02, and are
thus relatively immune to this source of measurement bias.
6. Related Work
We already mentioned in the introduction that several studies have
investigated the impact of data sets on program behavior and per-
formance. And some of these have even looked how data sets af-
fect program optimization decisions. Most of these works remain
limited by the restricted number of data sets available in existing
benchmark suites. The SPEC CPU2000 suite contains 3 input sets
per benchmark, and most benchmarks have less than 10 data sets in
the SPEC CPU2006. The embedded EEMBC benchmark suite [1]
also contains less than 10 data sets for most benchmarks. The re-
cently introduced parallel PARSEC benchmark suite [7] contains
6 data sets for each benchmark. A large number of data sets is not
only useful for compiler research and workload characterization re-
search (e.g., [21]), many architecture studies rely on proﬁle-based
optimization techniques as well [23, 28], and may beneﬁt from hav-
ing more data sets in order to study data set sensitivity.
Several studies have investigated redundancy among data sets
and how to ﬁnd a minimal set of representative programs and
inputs for architecture research [11]. We use several of the proposed
statistical techniques and feature characterization approaches in
this article to investigate our own data set suite.
Finally, we have mentioned the broad set of iterative/adaptive
compilation techniques which attempt to ﬁnd the best possible
compiler optimizations by stochastically scanning the set of all
possible combinations [4, 10, 12, 17, 19, 22, 29]. They have
demonstrated that optimizations search techniques can effectively
improve performance of statically compiled programs on rapidly
evolving architectures, thereby outperforming state-of-the-art com-
pilers, albeit at the cost of a large number of exploration runs. Many
of these research works have shown how machine-learning and sta-
tistical techniques [4, 9, 12, 22, 29] can be used to select or tune
program transformations based on program features. Most of these
works also require a large number of training runs. For instance,
Stephenson et al. [30] and Arnold et al. [5] collect proﬁle infor-
mation across multiple runs of a Java program to selectively apply
run-time optimizations.
7. Conclusions and Future Work
Using KDataSets, a collection of 1000 data sets for 32 programs,
we investigate a fundamental issue in iterative optimization, which
could not be thoroughly evaluated up to now: whether it is pos-
sible to learn the best possible compiler optimizations across dis-
tinct data sets. We conclude that the issue seems signiﬁcantly more
simple than previously anticipated, with the ability to ﬁnd a near-
optimal combination of compiler optimizations across all data sets.
We outline a process for selecting the program-optimal combina-
tion, and we investigate the impact of performing iterative opti-
mization in an unrealistic context.
For now, we have investigated whole-program optimizations,
and we intend to study whether the same conclusions are sustained
for ﬁne-grain optimizations. We also intend to apply our conclu-
sions to datacenters where, typically, a few programs are run a
very large number of times, and where any execution time reduc-
tion translates into proportional gains in datacenter equipment and
operating costs.
Acknowledgments
We would like to thank the anonymous reviewers for their valu-
able feedback. Lieven Eeckhout is supported through the FWO
projects G.0232.06, G.0255.08, and G.0179.10, and the UGent-
BOF projects 01J14407 and 01Z04109. Olivier Temam is sup-
ported by the IST FP7 HiPEAC2 contract No. IST-217068. Olivier
Temam and Grigori Fursin are supported by EU FP6 MILEPOST
project contract No. IST-35307. The rest of the authors are sup-
ported by National Natural Science Foundation of China under
grant No. 60873057 and 60921002, and National Basic Research
Program of China under grant No. 2005CB321602.
References
[1] EEMBC: The Embedded Microprocessor Benchmark Consortium.
http://www.eembc.org.
[2] cBench: Collective Benchmarks. http://www.ctuning.org/
cbench.
[3] PAPI: A Portable Interface to Hardware Performance Counters. http:
//icl.cs.utk.edu/papi.
[4] F. Agakov, E. Bonilla, J. Cavazos, B. Franke, G. Fursin, M. F. P.
O’Boyle, J. Thomson, M. Toussaint, and C. K. I. Williams. Using
machine learning to focus iterative optimization. In Proceedings of
the International Symposium on Code Generation and Optimization
(CGO), pages 295–305, March 2006.
[5] M. Arnold, A. Welc, and V .T.Rajan. Improving virtual machine per-
formance using a cross-run proﬁle repository. In Proceedings of the
ACM Conference on Object-Oriented Programming, Systems, Lan-
guages and Applications (OOPSLA) , pages 297–311, October 2005.
[6] P. Berube and J. Amaral. Aestimo: a feedback-directed optimization
evaluation tool. In Proceedings of the IEEE International Symposium
on Performance Analysis of Systems and Software (ISPASS) , pages
251–260, March 2006.
[7] C. Bienia, S. Kumar, J. P. Singh, and K. Li. The PARSEC benchmark
suite: characterization and architectural implications. In Proceedings
of the 17th International Conference on Parallel Architectures and
Compilation Techniques (PACT), pages 72–81, October 2008.
[8] J. Cavazos, G. Fursin, F. Agakov, E. Bonilla, M. F. P. O’Boyle, and
O. Temam. Rapidly selecting good compiler optimizations using
performance counters. In Proceedings of the International Symposium
on Code Generation and Optimization (CGO) , pages 185–197, March
2007.
[9] K. Cooper, P. Schielke, and D. Subramanian. Optimizing for reduced
code space using genetic algorithms. In Proceedings of the Conference
on Languages, Compilers, and Tools for Embedded Systems (LCTES) ,
pages 1–9, July 1999.
[10] K. D. Cooper, A. Grosul, T. J. Harvey, S. Reeves, D. Subramanian,
L. Torczon, and T. Waterman. ACME: adaptive compilation made
efﬁcient. In Proceedings of the ACM SIGPLAN/SIGBED Conference
on Languages, Compilers, and Tools for Embedded Systems (LCTES) ,
pages 69–77, July 2005.
[11] L. Eeckhout, H. Vandierendonck, and K. De Bosschere. Quantifying
the impact of input data sets on program behavior and its applications.
Journal of Instruction-Level Parallelism , 5:1–33, February 2003.
[12] B. Franke, M. O’Boyle, J. Thomson, and G. Fursin. Probabilistic
source-level optimisation of embedded programs. In Proceedings of
the ACM SIGPLAN/SIGBED Conference on Languages, Compilers,
and Tools for Embedded Systems (LCTES) , pages 78–86, July 2005.
[13] G. Fursin and O. Temam. Collective optimization. In Proceedings of
the International Conference on High Performance Embedded Archi-
tectures & Compilers (HiPEAC) , pages 34–49, January 2009.
[14] G. Fursin, J. Cavazos, M. O’Boyle, and O. Temam. Midatasets:
Creating the conditions for a more realistic evaluation of iterative
optimization. In Proceedings of the International Conference on High
Performance Embedded Architectures & Compilers (HiPEAC) , pages
245–260, January 2007.
[15] M. Guthaus, J. Ringenberg, D. Ernst, T. Austin, T. Mudge, and
R. Brown. Mibench: A free, commercially representative embedded
benchmark suite. In Proceedings of the IEEE Fourth Annual Interna-
tional Workshop on Workload Characterization (WWC) , pages 3–14,
December 2001.
[16] M. Haneda, P. Knijnenburg, and H. Wijshoff. On the impact of data
input sets on statistical compiler tuning. In Proceedings of the 20th
IEEE International Parallel and Distributed Processing Symposium
(IPDPS), April 2006.
[17] K. Hoste and L. Eeckhout. Cole: compiler optimization level explo-
ration. In Proceedings of the Sixth Annual IEEE/ACM International
Symposium on Code Generation and Optimization (CGO) , pages 165–
174, April 2008.
[18] K. Hoste and L. Eeckhout. Comparing benchmarks using key
microarchitecture-independent characteristics. In Proceedings of
the IEEE International Symposium on Workload Characterization
(IISWC), pages 83–92, October 2006.
[19] K. Hoste, A. Georges, and L. Eeckhout. Automated just-in-time com-
piler tuning. In Proceedings of the Eighth Annual IEEE/ACM Inter-
national Symposium on Code Generation and Optimization (CGO) ,
April 2010.
[20] W. C. Hsu, H. Chen, P. C. Yew, and D.-Y . Chen. On the predictability
of program behavior using different input data sets. In Proceedings
of the Sixth Annual Workshop on Interaction between Compilers and
Computer Architectures (INTERACT), pages 45–53, February 2002.
[21] Y . Jiang, E. Z. Zhang, K. Tian, F. Mao, M. Gethers, X. Shen, and
Y . Gao. Exploiting statistical correlations for proactive prediction of
program behaviors. In Proceedings of the International Symposium on
Code Generation and Optimization (CGO) , April 2010.
[22] P. Kulkarni, S. Hines, J. Hiser, D. Whalley, J. Davidson, and D. Jones.
Fast searches for effective optimization phase sequences. In Proceed-
ings of the ACM SIGPLAN Conference on Programming Language
Design and Implementation (PLDI) , pages 171–182, June 2004.
[23] G. Magklis, M. L. Scott, G. Semeraro, D. H. Albonesi, and S. Dropsho.
Proﬁle-based dynamic voltage and frequency scaling for a multiple
clock domain microprocessor. In Proceedings of the 30th Annual
International Symposium on Computer Architecture (ISCA) , pages 14–
27, June 2003.
[24] F. Mao, E. Z. Zhang, and X. Shen. Inﬂuence of program inputs on
the selection of garbage collectors. In Proceedings of the ACM SIG-
PLAN/SIGOPS International Conference on Virtual Execution Envi-
ronments (VEE), pages 91–100, March 2009.
[25] F. Matteo and S. Johnson. FFTW: An adaptive software architecture
for the FFT. In Proceedings of the IEEE International Conference on
Acoustics, Speech, and Signal Processing (ICASSP) , volume 3, pages
1381–1384, May 1998.
[26] T. Mytkowicz, A. Diwan, M. Hauswirth, and P. F. Sweeney. Producing
wrong data without doing anything obviously wrong! In Proceeding
of the 14th International Conference on Architectural Support for
Programming Languages and Operating Systems (ASPLOS) , pages
265–276, February 2009.
[27] Z. Pan and R. Eigenmann. Fast and effective orchestration of compiler
optimizations for automatic performance tuning. In Proceedings of
the International Symposium on Code Generation and Optimization
(CGO), pages 319–332, March 2006.
[28] K. Sankaranarayanan and K. Skadron. Proﬁle-based adaptation for
cache decay. ACM Transactions on Architecture and Code Optimiza-
tion (TACO), 1:305–322, September 2004.
[29] M. Stephenson, M. Martin, and U. O’Reilly. Meta optimization:
Improving compiler heuristics with machine learning. In Proceedings
of the ACM SIGPLAN Conference on Programming Language Design
and Implementation (PLDI) , pages 77–90, June 2003.
[30] M. W. Stephenson. Automating the Construction of Compiler Heuris-
tics Using Machine Learning . PhD thesis, MIT, USA, January 2006.
[31] R. C. Whaley, A. Petitet, and J. Dongarra. Automated empirical
optimization of software and the atlas project. In Parallel Computing,
March 2001.
[32] Y . Zhong, X. Shen, and C. Ding. Program locality analysis using
reuse distance. Transactions on Programming Languages and Systems
(TOPLAS), 31(6):1–39, Aug. 2009.

Deep Residual Learning for Image Recognition
Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
Microsoft Research
{kahe, v-xiangz, v-shren, jiansun}@microsoft.com
Abstract
Deeper neural networks are more difﬁcult to train. We
present a residual learning framework to ease the training
of networks that are substantially deeper than those used
previously. We explicitly reformulate the layers as learn-
ing residual functions with reference to the layer inputs, in-
stead of learning unreferenced functions. We provide com-
prehensive empirical evidence showing that these residual
networks are easier to optimize, and can gain accuracy from
considerably increased depth. On the ImageNet dataset we
evaluate residual nets with a depth of up to 152 layers—8 ×
deeper than VGG nets [41] but still having lower complex-
ity. An ensemble of these residual nets achieves 3.57% error
on the ImageNettest set. This result won the 1st place on the
ILSVRC 2015 classiﬁcation task. We also present analysis
on CIFAR-10 with 100 and 1000 layers.
The depth of representations is of central importance
for many visual recognition tasks. Solely due to our ex-
tremely deep representations, we obtain a 28% relative im-
provement on the COCO object detection dataset. Deep
residual nets are foundations of our submissions to ILSVRC
& COCO 2015 competitions 1, where we also won the 1st
places on the tasks of ImageNet detection, ImageNet local-
ization, COCO detection, and COCO segmentation.
1. Introduction
Deep convolutional neural networks [22, 21] have led
to a series of breakthroughs for image classiﬁcation [21,
50, 40]. Deep networks naturally integrate low/mid/high-
level features [50] and classiﬁers in an end-to-end multi-
layer fashion, and the “levels” of features can be enriched
by the number of stacked layers (depth). Recent evidence
[41, 44] reveals that network depth is of crucial importance,
and the leading results [41, 44, 13, 16] on the challenging
ImageNet dataset [36] all exploit “very deep” [41] models,
with a depth of sixteen [41] to thirty [16]. Many other non-
trivial visual recognition tasks [8, 12, 7, 32, 27] have also
1http://image-net.org/challenges/LSVRC/2015/ and
http://mscoco.org/dataset/#detections-challenge2015.
0 1 2 3 4 5 60 
10
20
iter. (1e4)
training error (%)
 
 
0 1 2 3 4 5 60
10
20
iter. (1e4)
test error (%)
 
 
56-layer
20-layer
56-layer
20-layer
Figure 1. Training error (left) and test error (right) on CIFAR-10
with 20-layer and 56-layer “plain” networks. The deeper network
has higher training error, and thus test error. Similar phenomena
on ImageNet is presented in Fig. 4.
greatly beneﬁted from very deep models.
Driven by the signiﬁcance of depth, a question arises: Is
learning better networks as easy as stacking more layers?
An obstacle to answering this question was the notorious
problem of vanishing/exploding gradients [1, 9], which
hamper convergence from the beginning. This problem,
however, has been largely addressed by normalized initial-
ization [23, 9, 37, 13] and intermediate normalization layers
[16], which enable networks with tens of layers to start con-
verging for stochastic gradient descent (SGD) with back-
propagation [22].
When deeper networks are able to start converging, a
degradation problem has been exposed: with the network
depth increasing, accuracy gets saturated (which might be
unsurprising) and then degrades rapidly. Unexpectedly,
such degradation is not caused by overﬁtting , and adding
more layers to a suitably deep model leads to higher train-
ing error, as reported in [11, 42] and thoroughly veriﬁed by
our experiments. Fig. 1 shows a typical example.
The degradation (of training accuracy) indicates that not
all systems are similarly easy to optimize. Let us consider a
shallower architecture and its deeper counterpart that adds
more layers onto it. There exists a solution by construction
to the deeper model: the added layers are identity mapping,
and the other layers are copied from the learned shallower
model. The existence of this constructed solution indicates
that a deeper model should produce no higher training error
than its shallower counterpart. But experiments show that
our current solvers on hand are unable to ﬁnd solutions that
1
arXiv:1512.03385v1  [cs.CV]  10 Dec 2015
identity
weight layer
weight layer
relu
reluF(x)+x
x
F(x) x
Figure 2. Residual learning: a building block.
are comparably good or better than the constructed solution
(or unable to do so in feasible time).
In this paper, we address the degradation problem by
introducing a deep residual learning framework. In-
stead of hoping each few stacked layers directly ﬁt a
desired underlying mapping, we explicitly let these lay-
ers ﬁt a residual mapping. Formally, denoting the desired
underlying mapping as H(x), we let the stacked nonlinear
layers ﬁt another mapping of F(x) :=H(x) −x. The orig-
inal mapping is recast intoF(x)+ x. We hypothesize that it
is easier to optimize the residual mapping than to optimize
the original, unreferenced mapping. To the extreme, if an
identity mapping were optimal, it would be easier to push
the residual to zero than to ﬁt an identity mapping by a stack
of nonlinear layers.
The formulation of F(x) +x can be realized by feedfor-
ward neural networks with “shortcut connections” (Fig. 2).
Shortcut connections [2, 34, 49] are those skipping one or
more layers. In our case, the shortcut connections simply
perform identity mapping, and their outputs are added to
the outputs of the stacked layers (Fig. 2). Identity short-
cut connections add neither extra parameter nor computa-
tional complexity. The entire network can still be trained
end-to-end by SGD with backpropagation, and can be eas-
ily implemented using common libraries ( e.g., Caffe [19])
without modifying the solvers.
We present comprehensive experiments on ImageNet
[36] to show the degradation problem and evaluate our
method. We show that: 1) Our extremely deep residual nets
are easy to optimize, but the counterpart “plain” nets (that
simply stack layers) exhibit higher training error when the
depth increases; 2) Our deep residual nets can easily enjoy
accuracy gains from greatly increased depth, producing re-
sults substantially better than previous networks.
Similar phenomena are also shown on the CIFAR-10 set
[20], suggesting that the optimization difﬁculties and the
effects of our method are not just akin to a particular dataset.
We present successfully trained models on this dataset with
over 100 layers, and explore models with over 1000 layers.
On the ImageNet classiﬁcation dataset [36], we obtain
excellent results by extremely deep residual nets. Our 152-
layer residual net is the deepest network ever presented on
ImageNet, while still having lower complexity than VGG
nets [41]. Our ensemble has 3.57% top-5 error on the
ImageNet test set, and won the 1st place in the ILSVRC
2015 classiﬁcation competition . The extremely deep rep-
resentations also have excellent generalization performance
on other recognition tasks, and lead us to further win the
1st places on: ImageNet detection, ImageNet localization,
COCO detection, and COCO segmentation in ILSVRC &
COCO 2015 competitions. This strong evidence shows that
the residual learning principle is generic, and we expect that
it is applicable in other vision and non-vision problems.
2. Related Work
Residual Representations. In image recognition, VLAD
[18] is a representation that encodes by the residual vectors
with respect to a dictionary, and Fisher Vector [30] can be
formulated as a probabilistic version [18] of VLAD. Both
of them are powerful shallow representations for image re-
trieval and classiﬁcation [4, 48]. For vector quantization,
encoding residual vectors [17] is shown to be more effec-
tive than encoding original vectors.
In low-level vision and computer graphics, for solv-
ing Partial Differential Equations (PDEs), the widely used
Multigrid method [3] reformulates the system as subprob-
lems at multiple scales, where each subproblem is respon-
sible for the residual solution between a coarser and a ﬁner
scale. An alternative to Multigrid is hierarchical basis pre-
conditioning [45, 46], which relies on variables that repre-
sent residual vectors between two scales. It has been shown
[3, 45, 46] that these solvers converge much faster than stan-
dard solvers that are unaware of the residual nature of the
solutions. These methods suggest that a good reformulation
or preconditioning can simplify the optimization.
Shortcut Connections. Practices and theories that lead to
shortcut connections [2, 34, 49] have been studied for a long
time. An early practice of training multi-layer perceptrons
(MLPs) is to add a linear layer connected from the network
input to the output [34, 49]. In [44, 24], a few interme-
diate layers are directly connected to auxiliary classiﬁers
for addressing vanishing/exploding gradients. The papers
of [39, 38, 31, 47] propose methods for centering layer re-
sponses, gradients, and propagated errors, implemented by
shortcut connections. In [44], an “inception” layer is com-
posed of a shortcut branch and a few deeper branches.
Concurrent with our work, “highway networks” [42, 43]
present shortcut connections with gating functions [15].
These gates are data-dependent and have parameters, in
contrast to our identity shortcuts that are parameter-free.
When a gated shortcut is “closed” (approaching zero), the
layers in highway networks represent non-residual func-
tions. On the contrary, our formulation always learns
residual functions; our identity shortcuts are never closed,
and all information is always passed through, with addi-
tional residual functions to be learned. In addition, high-
2
way networks have not demonstrated accuracy gains with
extremely increased depth (e.g., over 100 layers).
3. Deep Residual Learning
3.1. Residual Learning
Let us consider H(x) as an underlying mapping to be
ﬁt by a few stacked layers (not necessarily the entire net),
with x denoting the inputs to the ﬁrst of these layers. If one
hypothesizes that multiple nonlinear layers can asymptoti-
cally approximate complicated functions2, then it is equiv-
alent to hypothesize that they can asymptotically approxi-
mate the residual functions, i.e., H(x) −x (assuming that
the input and output are of the same dimensions). So
rather than expect stacked layers to approximate H(x), we
explicitly let these layers approximate a residual function
F(x) := H(x) −x. The original function thus becomes
F(x)+ x. Although both forms should be able to asymptot-
ically approximate the desired functions (as hypothesized),
the ease of learning might be different.
This reformulation is motivated by the counterintuitive
phenomena about the degradation problem (Fig. 1, left). As
we discussed in the introduction, if the added layers can
be constructed as identity mappings, a deeper model should
have training error no greater than its shallower counter-
part. The degradation problem suggests that the solvers
might have difﬁculties in approximating identity mappings
by multiple nonlinear layers. With the residual learning re-
formulation, if identity mappings are optimal, the solvers
may simply drive the weights of the multiple nonlinear lay-
ers toward zero to approach identity mappings.
In real cases, it is unlikely that identity mappings are op-
timal, but our reformulation may help to precondition the
problem. If the optimal function is closer to an identity
mapping than to a zero mapping, it should be easier for the
solver to ﬁnd the perturbations with reference to an identity
mapping, than to learn the function as a new one. We show
by experiments (Fig. 7) that the learned residual functions in
general have small responses, suggesting that identity map-
pings provide reasonable preconditioning.
3.2. Identity Mapping by Shortcuts
We adopt residual learning to every few stacked layers.
A building block is shown in Fig. 2. Formally, in this paper
we consider a building block deﬁned as:
y = F(x,{Wi}) +x. (1)
Here x and y are the input and output vectors of the lay-
ers considered. The function F(x,{Wi}) represents the
residual mapping to be learned. For the example in Fig. 2
that has two layers, F = W2σ(W1x) in which σ denotes
2This hypothesis, however, is still an open question. See [28].
ReLU [29] and the biases are omitted for simplifying no-
tations. The operation F+ x is performed by a shortcut
connection and element-wise addition. We adopt the sec-
ond nonlinearity after the addition (i.e., σ(y), see Fig. 2).
The shortcut connections in Eqn.(1) introduce neither ex-
tra parameter nor computation complexity. This is not only
attractive in practice but also important in our comparisons
between plain and residual networks. We can fairly com-
pare plain/residual networks that simultaneously have the
same number of parameters, depth, width, and computa-
tional cost (except for the negligible element-wise addition).
The dimensions of x and Fmust be equal in Eqn.(1).
If this is not the case ( e.g., when changing the input/output
channels), we can perform a linear projection Ws by the
shortcut connections to match the dimensions:
y = F(x,{Wi}) +Wsx. (2)
We can also use a square matrixWs in Eqn.(1). But we will
show by experiments that the identity mapping is sufﬁcient
for addressing the degradation problem and is economical,
and thus Ws is only used when matching dimensions.
The form of the residual function Fis ﬂexible. Exper-
iments in this paper involve a function Fthat has two or
three layers (Fig. 5), while more layers are possible. But if
Fhas only a single layer, Eqn.(1) is similar to a linear layer:
y = W1x + x, for which we have not observed advantages.
We also note that although the above notations are about
fully-connected layers for simplicity, they are applicable to
convolutional layers. The function F(x,{Wi}) can repre-
sent multiple convolutional layers. The element-wise addi-
tion is performed on two feature maps, channel by channel.
3.3. Network Architectures
We have tested various plain/residual nets, and have ob-
served consistent phenomena. To provide instances for dis-
cussion, we describe two models for ImageNet as follows.
Plain Network. Our plain baselines (Fig. 3, middle) are
mainly inspired by the philosophy of VGG nets [41] (Fig. 3,
left). The convolutional layers mostly have 3 ×3 ﬁlters and
follow two simple design rules: (i) for the same output
feature map size, the layers have the same number of ﬁl-
ters; and (ii) if the feature map size is halved, the num-
ber of ﬁlters is doubled so as to preserve the time com-
plexity per layer. We perform downsampling directly by
convolutional layers that have a stride of 2. The network
ends with a global average pooling layer and a 1000-way
fully-connected layer with softmax. The total number of
weighted layers is 34 in Fig. 3 (middle).
It is worth noticing that our model has fewer ﬁlters and
lower complexity than VGG nets [41] (Fig. 3, left). Our 34-
layer baseline has 3.6 billion FLOPs (multiply-adds), which
is only 18% of VGG-19 (19.6 billion FLOPs).
3
7x7 conv, 64, /2
pool, /2
3x3 conv, 64
3x3 conv, 64
3x3 conv, 64
3x3 conv, 64
3x3 conv, 64
3x3 conv, 64
3x3 conv, 128, /2
3x3 conv, 128
3x3 conv, 128
3x3 conv, 128
3x3 conv, 128
3x3 conv, 128
3x3 conv, 128
3x3 conv, 128
3x3 conv, 256, /2
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 512, /2
3x3 conv, 512
3x3 conv, 512
3x3 conv, 512
3x3 conv, 512
3x3 conv, 512
avg pool
fc 1000
image
3x3 conv, 512
3x3 conv, 64
3x3 conv, 64
pool, /2
3x3 conv, 128
3x3 conv, 128
pool, /2
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
pool, /2
3x3 conv, 512
3x3 conv, 512
3x3 conv, 512
pool, /2
3x3 conv, 512
3x3 conv, 512
3x3 conv, 512
3x3 conv, 512
pool, /2
fc 4096
fc 4096
fc 1000
image
output 
size: 112
output 
size: 224
output 
size: 56
output 
size: 28
output 
size: 14
output 
size: 7
output 
size: 1
VGG-19 34-layer plain
7x7 conv, 64, /2
pool, /2
3x3 conv, 64
3x3 conv, 64
3x3 conv, 64
3x3 conv, 64
3x3 conv, 64
3x3 conv, 64
3x3 conv, 128, /2
3x3 conv, 128
3x3 conv, 128
3x3 conv, 128
3x3 conv, 128
3x3 conv, 128
3x3 conv, 128
3x3 conv, 128
3x3 conv, 256, /2
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 256
3x3 conv, 512, /2
3x3 conv, 512
3x3 conv, 512
3x3 conv, 512
3x3 conv, 512
3x3 conv, 512
avg pool
fc 1000
image
34-layer residual
Figure 3. Example network architectures for ImageNet. Left: the
VGG-19 model [41] (19.6 billion FLOPs) as a reference. Mid-
dle: a plain network with 34 parameter layers (3.6 billion FLOPs).
Right: a residual network with 34 parameter layers (3.6 billion
FLOPs). The dotted shortcuts increase dimensions. Table 1shows
more details and other variants.
Residual Network. Based on the above plain network, we
insert shortcut connections (Fig. 3, right) which turn the
network into its counterpart residual version. The identity
shortcuts (Eqn.(1)) can be directly used when the input and
output are of the same dimensions (solid line shortcuts in
Fig. 3). When the dimensions increase (dotted line shortcuts
in Fig. 3), we consider two options: (A) The shortcut still
performs identity mapping, with extra zero entries padded
for increasing dimensions. This option introduces no extra
parameter; (B) The projection shortcut in Eqn.(2) is used to
match dimensions (done by 1 ×1 convolutions). For both
options, when the shortcuts go across feature maps of two
sizes, they are performed with a stride of 2.
3.4. Implementation
Our implementation for ImageNet follows the practice
in [21, 41]. The image is resized with its shorter side ran-
domly sampled in [256,480] for scale augmentation [41].
A 224×224 crop is randomly sampled from an image or its
horizontal ﬂip, with the per-pixel mean subtracted [21]. The
standard color augmentation in [21] is used. We adopt batch
normalization (BN) [16] right after each convolution and
before activation, following [16]. We initialize the weights
as in [13] and train all plain/residual nets from scratch. We
use SGD with a mini-batch size of 256. The learning rate
starts from 0.1 and is divided by 10 when the error plateaus,
and the models are trained for up to60 ×104 iterations. We
use a weight decay of 0.0001 and a momentum of 0.9. We
do not use dropout [14], following the practice in [16].
In testing, for comparison studies we adopt the standard
10-crop testing [21]. For best results, we adopt the fully-
convolutional form as in [41, 13], and average the scores
at multiple scales (images are resized such that the shorter
side is in {224,256,384,480,640}).
4. Experiments
4.1. ImageNet Classiﬁcation
We evaluate our method on the ImageNet 2012 classiﬁ-
cation dataset [36] that consists of 1000 classes. The models
are trained on the 1.28 million training images, and evalu-
ated on the 50k validation images. We also obtain a ﬁnal
result on the 100k test images, reported by the test server.
We evaluate both top-1 and top-5 error rates.
Plain Networks. We ﬁrst evaluate 18-layer and 34-layer
plain nets. The 34-layer plain net is in Fig. 3 (middle). The
18-layer plain net is of a similar form. See Table 1 for de-
tailed architectures.
The results in Table 2 show that the deeper 34-layer plain
net has higher validation error than the shallower 18-layer
plain net. To reveal the reasons, in Fig. 4 (left) we com-
pare their training/validation errors during the training pro-
cedure. We have observed the degradation problem - the
4
layer nameoutput size 18-layer 34-layer 50-layer 101-layer 152-layer
conv1 112×112 7×7, 64, stride 2
conv2x 56×56
3×3 max pool, stride 2[3×3, 643×3, 64
]
×2
[3×3, 643×3, 64
]
×3

1×1, 643×3, 641×1, 256

×3

1×1, 643×3, 641×1, 256

×3

1×1, 643×3, 641×1, 256

×3
conv3x 28×28
[3×3, 1283×3, 128
]
×2
[3×3, 1283×3, 128
]
×4

1×1, 1283×3, 1281×1, 512

×4

1×1, 1283×3, 1281×1, 512

×4

1×1, 1283×3, 1281×1, 512

×8
conv4x 14×14
[3×3, 2563×3, 256
]
×2
[3×3, 2563×3, 256
]
×6

1×1, 2563×3, 2561×1, 1024

×6

1×1, 2563×3, 2561×1, 1024

×23

1×1, 2563×3, 2561×1, 1024

×36
conv5x 7×7
[3×3, 5123×3, 512
]
×2
[3×3, 5123×3, 512
]
×3

1×1, 5123×3, 5121×1, 2048

×3

1×1, 5123×3, 5121×1, 2048

×3

1×1, 5123×3, 5121×1, 2048

×3
1×1 average pool, 1000-d fc, softmax
FLOPs 1.8×109 3.6×109 3.8×109 7.6×109 11.3×109
Table 1. Architectures for ImageNet. Building blocks are shown in brackets (see also Fig. 5), with the numbers of blocks stacked. Down-
sampling is performed by conv3 1, conv4 1, and conv5 1 with a stride of 2.
0 10 20 30 40 5020
30
40
50
60
iter. (1e4)
error (%)
 
 
plain-18
plain-34
0 10 20 30 40 5020
30
40
50
60
iter. (1e4)
error (%)
 
 
ResNet-18
ResNet-34
18-layer
34-layer
18-layer
34-layer
Figure 4. Training on ImageNet. Thin curves denote training error, and bold curves denote validation error of the center crops. Left: plain
networks of 18 and 34 layers. Right: ResNets of 18 and 34 layers. In this plot, the residual networks have no extra parameter compared to
their plain counterparts.
plain ResNet
18 layers 27.94 27.88
34 layers 28.54 25.03
Table 2. Top-1 error (%, 10-crop testing) on ImageNet validation.
Here the ResNets have no extra parameter compared to their plain
counterparts. Fig. 4 shows the training procedures.
34-layer plain net has higher training error throughout the
whole training procedure, even though the solution space
of the 18-layer plain network is a subspace of that of the
34-layer one.
We argue that this optimization difﬁculty is unlikely to
be caused by vanishing gradients. These plain networks are
trained with BN [16], which ensures forward propagated
signals to have non-zero variances. We also verify that the
backward propagated gradients exhibit healthy norms with
BN. So neither forward nor backward signals vanish. In
fact, the 34-layer plain net is still able to achieve compet-
itive accuracy (Table 3), suggesting that the solver works
to some extent. We conjecture that the deep plain nets may
have exponentially low convergence rates, which impact the
reducing of the training error 3. The reason for such opti-
mization difﬁculties will be studied in the future.
Residual Networks. Next we evaluate 18-layer and 34-
layer residual nets ( ResNets). The baseline architectures
are the same as the above plain nets, expect that a shortcut
connection is added to each pair of 3 ×3 ﬁlters as in Fig. 3
(right). In the ﬁrst comparison (Table 2 and Fig. 4 right),
we use identity mapping for all shortcuts and zero-padding
for increasing dimensions (option A). So they haveno extra
parameter compared to the plain counterparts.
We have three major observations from Table 2 and
Fig. 4. First, the situation is reversed with residual learn-
ing – the 34-layer ResNet is better than the 18-layer ResNet
(by 2.8%). More importantly, the 34-layer ResNet exhibits
considerably lower training error and is generalizable to the
validation data. This indicates that the degradation problem
is well addressed in this setting and we manage to obtain
accuracy gains from increased depth.
Second, compared to its plain counterpart, the 34-layer
3We have experimented with more training iterations (3×) and still ob-
served the degradation problem, suggesting that this problem cannot be
feasibly addressed by simply using more iterations.
5
model top-1 err. top-5 err.
VGG-16 [41] 28.07 9.33
GoogLeNet [44] - 9.15
PReLU-net [13] 24.27 7.38
plain-34 28.54 10.02
ResNet-34 A 25.03 7.76
ResNet-34 B 24.52 7.46
ResNet-34 C 24.19 7.40
ResNet-50 22.85 6.71
ResNet-101 21.75 6.05
ResNet-152 21.43 5.71
Table 3. Error rates (%, 10-crop testing) on ImageNet validation.
VGG-16 is based on our test. ResNet-50/101/152 are of option B
that only uses projections for increasing dimensions.
method top-1 err. top-5 err.
VGG [41] (ILSVRC’14) - 8.43 †
GoogLeNet [44] (ILSVRC’14) - 7.89
VGG [41] (v5) 24.4 7.1
PReLU-net [13] 21.59 5.71
BN-inception [16] 21.99 5.81
ResNet-34 B 21.84 5.71
ResNet-34 C 21.53 5.60
ResNet-50 20.74 5.25
ResNet-101 19.87 4.60
ResNet-152 19.38 4.49
Table 4. Error rates (%) of single-model results on the ImageNet
validation set (except † reported on the test set).
method top-5 err. (test)
VGG [41] (ILSVRC’14) 7.32
GoogLeNet [44] (ILSVRC’14) 6.66
VGG [41] (v5) 6.8
PReLU-net [13] 4.94
BN-inception [16] 4.82
ResNet (ILSVRC’15) 3.57
Table 5. Error rates (%) of ensembles. The top-5 error is on the
test set of ImageNet and reported by the test server.
ResNet reduces the top-1 error by 3.5% (Table 2), resulting
from the successfully reduced training error (Fig. 4 rightvs.
left). This comparison veriﬁes the effectiveness of residual
learning on extremely deep systems.
Last, we also note that the 18-layer plain/residual nets
are comparably accurate (Table 2), but the 18-layer ResNet
converges faster (Fig. 4 right vs. left). When the net is “not
overly deep” (18 layers here), the current SGD solver is still
able to ﬁnd good solutions to the plain net. In this case, the
ResNet eases the optimization by providing faster conver-
gence at the early stage.
Identity vs. Projection Shortcuts. We have shown that
3x3, 64
1x1, 64
relu
1x1, 256
relu
relu
3x3, 64
3x3, 64
relu
relu
64-d 256-d
Figure 5. A deeper residual function Ffor ImageNet. Left: a
building block (on 56 ×56 feature maps) as in Fig. 3 for ResNet-
34. Right: a “bottleneck” building block for ResNet-50/101/152.
parameter-free, identity shortcuts help with training. Next
we investigate projection shortcuts (Eqn.(2)). In Table 3 we
compare three options: (A) zero-padding shortcuts are used
for increasing dimensions, and all shortcuts are parameter-
free (the same as Table 2 and Fig. 4 right); (B) projec-
tion shortcuts are used for increasing dimensions, and other
shortcuts are identity; and (C) all shortcuts are projections.
Table 3 shows that all three options are considerably bet-
ter than the plain counterpart. B is slightly better than A. We
argue that this is because the zero-padded dimensions in A
indeed have no residual learning. C is marginally better than
B, and we attribute this to the extra parameters introduced
by many (thirteen) projection shortcuts. But the small dif-
ferences among A/B/C indicate that projection shortcuts are
not essential for addressing the degradation problem. So we
do not use option C in the rest of this paper, to reduce mem-
ory/time complexity and model sizes. Identity shortcuts are
particularly important for not increasing the complexity of
the bottleneck architectures that are introduced below.
Deeper Bottleneck Architectures. Next we describe our
deeper nets for ImageNet. Because of concerns on the train-
ing time that we can afford, we modify the building block
as a bottleneck design4. For each residual function F, we
use a stack of 3 layers instead of 2 (Fig. 5). The three layers
are 1×1, 3×3, and 1×1 convolutions, where the 1×1 layers
are responsible for reducing and then increasing (restoring)
dimensions, leaving the 3×3 layer a bottleneck with smaller
input/output dimensions. Fig. 5 shows an example, where
both designs have similar time complexity.
The parameter-free identity shortcuts are particularly im-
portant for the bottleneck architectures. If the identity short-
cut in Fig. 5 (right) is replaced with projection, one can
show that the time complexity and model size are doubled,
as the shortcut is connected to the two high-dimensional
ends. So identity shortcuts lead to more efﬁcient models
for the bottleneck designs.
50-layer ResNet: We replace each 2-layer block in the
4Deeper non-bottleneck ResNets ( e.g., Fig. 5 left) also gain accuracy
from increased depth (as shown on CIFAR-10), but are not as economical
as the bottleneck ResNets. So the usage of bottleneck designs is mainly due
to practical considerations. We further note that the degradation problem
of plain nets is also witnessed for the bottleneck designs.
6
34-layer net with this 3-layer bottleneck block, resulting in
a 50-layer ResNet (Table 1). We use option B for increasing
dimensions. This model has 3.8 billion FLOPs.
101-layer and 152-layer ResNets: We construct 101-
layer and 152-layer ResNets by using more 3-layer blocks
(Table 1). Remarkably, although the depth is signiﬁcantly
increased, the 152-layer ResNet (11.3 billion FLOPs) still
has lower complexity than VGG-16/19 nets (15.3/19.6 bil-
lion FLOPs).
The 50/101/152-layer ResNets are more accurate than
the 34-layer ones by considerable margins (Table 3 and 4).
We do not observe the degradation problem and thus en-
joy signiﬁcant accuracy gains from considerably increased
depth. The beneﬁts of depth are witnessed for all evaluation
metrics (Table 3 and 4).
Comparisons with State-of-the-art Methods. In Table 4
we compare with the previous best single-model results.
Our baseline 34-layer ResNets have achieved very compet-
itive accuracy. Our 152-layer ResNet has a single-model
top-5 validation error of 4.49%. This single-model result
outperforms all previous ensemble results (Table 5). We
combine six models of different depth to form an ensemble
(only with two 152-layer ones at the time of submitting).
This leads to 3.57% top-5 error on the test set (Table 5).
This entry won the 1st place in ILSVRC 2015.
4.2. CIFAR-10 and Analysis
We conducted more studies on the CIFAR-10 dataset
[20], which consists of 50k training images and 10k test-
ing images in 10 classes. We present experiments trained
on the training set and evaluated on the test set. Our focus
is on the behaviors of extremely deep networks, but not on
pushing the state-of-the-art results, so we intentionally use
simple architectures as follows.
The plain/residual architectures follow the form in Fig. 3
(middle/right). The network inputs are 32×32 images, with
the per-pixel mean subtracted. The ﬁrst layer is 3×3 convo-
lutions. Then we use a stack of 6nlayers with 3×3 convo-
lutions on the feature maps of sizes{32,16,8}respectively,
with 2nlayers for each feature map size. The numbers of
ﬁlters are {16,32,64}respectively. The subsampling is per-
formed by convolutions with a stride of 2. The network ends
with a global average pooling, a 10-way fully-connected
layer, and softmax. There are totally 6n+2 stacked weighted
layers. The following table summarizes the architecture:
output map size 32×32 16×16 8×8
# layers 1+2n 2n 2n
# ﬁlters 16 32 64
When shortcut connections are used, they are connected
to the pairs of 3 ×3 layers (totally 3n shortcuts). On this
dataset we use identity shortcuts in all cases (i.e., option A),
method error (%)
Maxout [10] 9.38
NIN [25] 8.81
DSN [24] 8.22
# layers # params
FitNet [35] 19 2.5M 8.39
Highway [42, 43] 19 2.3M 7.54 (7.72±0.16)
Highway [42, 43] 32 1.25M 8.80
ResNet 20 0.27M 8.75
ResNet 32 0.46M 7.51
ResNet 44 0.66M 7.17
ResNet 56 0.85M 6.97
ResNet 110 1.7M 6.43 (6.61±0.16)
ResNet 1202 19.4M 7.93
Table 6. Classiﬁcation error on the CIFAR-10 test set. All meth-
ods are with data augmentation. For ResNet-110, we run it 5 times
and show “best (mean±std)” as in [43].
so our residual models have exactly the same depth, width,
and number of parameters as the plain counterparts.
We use a weight decay of 0.0001 and momentum of 0.9,
and adopt the weight initialization in [13] and BN [16] but
with no dropout. These models are trained with a mini-
batch size of 128 on two GPUs. We start with a learning
rate of 0.1, divide it by 10 at 32k and 48k iterations, and
terminate training at 64k iterations, which is determined on
a 45k/5k train/val split. We follow the simple data augmen-
tation in [24] for training: 4 pixels are padded on each side,
and a 32 ×32 crop is randomly sampled from the padded
image or its horizontal ﬂip. For testing, we only evaluate
the single view of the original 32×32 image.
We compare n = {3,5,7,9}, leading to 20, 32, 44, and
56-layer networks. Fig. 6 (left) shows the behaviors of the
plain nets. The deep plain nets suffer from increased depth,
and exhibit higher training error when going deeper. This
phenomenon is similar to that on ImageNet (Fig. 4, left) and
on MNIST (see [42]), suggesting that such an optimization
difﬁculty is a fundamental problem.
Fig. 6 (middle) shows the behaviors of ResNets. Also
similar to the ImageNet cases (Fig. 4, right), our ResNets
manage to overcome the optimization difﬁculty and demon-
strate accuracy gains when the depth increases.
We further explore n = 18 that leads to a 110-layer
ResNet. In this case, we ﬁnd that the initial learning rate
of 0.1 is slightly too large to start converging 5. So we use
0.01 to warm up the training until the training error is below
80% (about 400 iterations), and then go back to 0.1 and con-
tinue training. The rest of the learning schedule is as done
previously. This 110-layer network converges well (Fig. 6,
middle). It has fewer parameters than other deep and thin
5With an initial learning rate of 0.1, it starts converging ( <90% error)
after several epochs, but still reaches similar accuracy.
7
0 1 2 3 4 5 60
5
10
20
iter. (1e4)
error (%)
 
 
plain-20
plain-32
plain-44
plain-56
0 1 2 3 4 5 60
5
10
20
iter. (1e4)
error (%)
 
 
ResNet-20
ResNet-32
ResNet-44
ResNet-56
ResNet-11056-layer
20-layer
110-layer
20-layer
4 5 60
1
5
10
20
iter. (1e4)
error (%)
 
 
residual-110
residual-1202
Figure 6. Training on CIFAR-10. Dashed lines denote training error, and bold lines denote testing error. Left: plain networks. The error
of plain-110 is higher than 60% and not displayed. Middle: ResNets. Right: ResNets with 110 and 1202 layers.
0 20 40 60 80 100
1
2
3
layer index (sorted by magnitude)
std
 
 
plain-20
plain-56
ResNet-20
ResNet-56
ResNet-110
0 20 40 60 80 100
1
2
3
layer index (original)
std
 
 
plain-20
plain-56
ResNet-20
ResNet-56
ResNet-110
Figure 7. Standard deviations (std) of layer responses on CIFAR-
10. The responses are the outputs of each 3×3 layer, after BN and
before nonlinearity. Top: the layers are shown in their original
order. Bottom: the responses are ranked in descending order.
networks such as FitNet [35] and Highway [42] (Table 6),
yet is among the state-of-the-art results (6.43%, Table 6).
Analysis of Layer Responses. Fig. 7 shows the standard
deviations (std) of the layer responses. The responses are
the outputs of each 3 ×3 layer, after BN and before other
nonlinearity (ReLU/addition). For ResNets, this analy-
sis reveals the response strength of the residual functions.
Fig. 7 shows that ResNets have generally smaller responses
than their plain counterparts. These results support our ba-
sic motivation (Sec.3.1) that the residual functions might
be generally closer to zero than the non-residual functions.
We also notice that the deeper ResNet has smaller magni-
tudes of responses, as evidenced by the comparisons among
ResNet-20, 56, and 110 in Fig. 7. When there are more
layers, an individual layer of ResNets tends to modify the
signal less.
Exploring Over 1000 layers. We explore an aggressively
deep model of over 1000 layers. We set n = 200 that
leads to a 1202-layer network, which is trained as described
above. Our method shows no optimization difﬁculty , and
this 103-layer network is able to achieve training error
<0.1% (Fig. 6, right). Its test error is still fairly good
(7.93%, Table 6).
But there are still open problems on such aggressively
deep models. The testing result of this 1202-layer network
is worse than that of our 110-layer network, although both
training data 07+12 07++12
test data VOC 07 test VOC 12 test
VGG-16 73.2 70.4
ResNet-101 76.4 73.8
Table 7. Object detection mAP (%) on the PASCAL VOC
2007/2012 test sets using baseline Faster R-CNN. See also Ta-
ble 10 and 11 for better results.
metric mAP@.5 mAP@[.5, .95]
VGG-16 41.5 21.2
ResNet-101 48.4 27.2
Table 8. Object detection mAP (%) on the COCO validation set
using baseline Faster R-CNN. See also Table 9 for better results.
have similar training error. We argue that this is because of
overﬁtting. The 1202-layer network may be unnecessarily
large (19.4M) for this small dataset. Strong regularization
such as maxout [10] or dropout [14] is applied to obtain the
best results ([10, 25, 24, 35]) on this dataset. In this paper,
we use no maxout/dropout and just simply impose regular-
ization via deep and thin architectures by design, without
distracting from the focus on the difﬁculties of optimiza-
tion. But combining with stronger regularization may im-
prove results, which we will study in the future.
4.3. Object Detection on PASCAL and MS COCO
Our method has good generalization performance on
other recognition tasks. Table 7 and 8 show the object de-
tection baseline results on PASCAL VOC 2007 and 2012
[5] and COCO [26]. We adoptFaster R-CNN [32] as the de-
tection method. Here we are interested in the improvements
of replacing VGG-16 [41] with ResNet-101. The detection
implementation (see appendix) of using both models is the
same, so the gains can only be attributed to better networks.
Most remarkably, on the challenging COCO dataset we ob-
tain a 6.0% increase in COCO’s standard metric (mAP@[.5,
.95]), which is a 28% relative improvement. This gain is
solely due to the learned representations.
Based on deep residual nets, we won the 1st places in
several tracks in ILSVRC & COCO 2015 competitions: Im-
ageNet detection, ImageNet localization, COCO detection,
and COCO segmentation. The details are in the appendix.
8
References
[1] Y . Bengio, P. Simard, and P. Frasconi. Learning long-term dependen-
cies with gradient descent is difﬁcult. IEEE Transactions on Neural
Networks, 5(2):157–166, 1994.
[2] C. M. Bishop. Neural networks for pattern recognition . Oxford
university press, 1995.
[3] W. L. Briggs, S. F. McCormick, et al. A Multigrid Tutorial. Siam,
2000.
[4] K. Chatﬁeld, V . Lempitsky, A. Vedaldi, and A. Zisserman. The devil
is in the details: an evaluation of recent feature encoding methods.
In BMVC, 2011.
[5] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zis-
serman. The Pascal Visual Object Classes (VOC) Challenge. IJCV,
pages 303–338, 2010.
[6] S. Gidaris and N. Komodakis. Object detection via a multi-region &
semantic segmentation-aware cnn model. In ICCV, 2015.
[7] R. Girshick. Fast R-CNN. In ICCV, 2015.
[8] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hier-
archies for accurate object detection and semantic segmentation. In
CVPR, 2014.
[9] X. Glorot and Y . Bengio. Understanding the difﬁculty of training
deep feedforward neural networks. In AISTATS, 2010.
[10] I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and
Y . Bengio. Maxout networks.arXiv:1302.4389, 2013.
[11] K. He and J. Sun. Convolutional neural networks at constrained time
cost. In CVPR, 2015.
[12] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep
convolutional networks for visual recognition. In ECCV, 2014.
[13] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectiﬁers:
Surpassing human-level performance on imagenet classiﬁcation. In
ICCV, 2015.
[14] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and
R. R. Salakhutdinov. Improving neural networks by preventing co-
adaptation of feature detectors. arXiv:1207.0580, 2012.
[15] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural
computation, 9(8):1735–1780, 1997.
[16] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep
network training by reducing internal covariate shift. InICML, 2015.
[17] H. Jegou, M. Douze, and C. Schmid. Product quantization for nearest
neighbor search. TPAMI, 33, 2011.
[18] H. Jegou, F. Perronnin, M. Douze, J. Sanchez, P. Perez, and
C. Schmid. Aggregating local image descriptors into compact codes.
TPAMI, 2012.
[19] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick,
S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for
fast feature embedding. arXiv:1408.5093, 2014.
[20] A. Krizhevsky. Learning multiple layers of features from tiny im-
ages. Tech Report, 2009.
[21] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classiﬁcation
with deep convolutional neural networks. In NIPS, 2012.
[22] Y . LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard,
W. Hubbard, and L. D. Jackel. Backpropagation applied to hand-
written zip code recognition. Neural computation, 1989.
[23] Y . LeCun, L. Bottou, G. B. Orr, and K.-R. M¨uller. Efﬁcient backprop.
In Neural Networks: Tricks of the Trade, pages 9–50. Springer, 1998.
[24] C.-Y . Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu. Deeply-
supervised nets. arXiv:1409.5185, 2014.
[25] M. Lin, Q. Chen, and S. Yan. Network in network. arXiv:1312.4400,
2013.
[26] T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan,
P. Doll´ar, and C. L. Zitnick. Microsoft COCO: Common objects in
context. In ECCV. 2014.
[27] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks
for semantic segmentation. In CVPR, 2015.
[28] G. Mont ´ufar, R. Pascanu, K. Cho, and Y . Bengio. On the number of
linear regions of deep neural networks. In NIPS, 2014.
[29] V . Nair and G. E. Hinton. Rectiﬁed linear units improve restricted
boltzmann machines. In ICML, 2010.
[30] F. Perronnin and C. Dance. Fisher kernels on visual vocabularies for
image categorization. In CVPR, 2007.
[31] T. Raiko, H. Valpola, and Y . LeCun. Deep learning made easier by
linear transformations in perceptrons. In AISTATS, 2012.
[32] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards
real-time object detection with region proposal networks. In NIPS,
2015.
[33] S. Ren, K. He, R. Girshick, X. Zhang, and J. Sun. Object detection
networks on convolutional feature maps. arXiv:1504.06066, 2015.
[34] B. D. Ripley. Pattern recognition and neural networks. Cambridge
university press, 1996.
[35] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and
Y . Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
[36] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,
Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet
large scale visual recognition challenge. arXiv:1409.0575, 2014.
[37] A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to
the nonlinear dynamics of learning in deep linear neural networks.
arXiv:1312.6120, 2013.
[38] N. N. Schraudolph. Accelerated gradient descent by factor-centering
decomposition. Technical report, 1998.
[39] N. N. Schraudolph. Centering neural network gradient factors. In
Neural Networks: Tricks of the Trade , pages 207–226. Springer,
1998.
[40] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y . Le-
Cun. Overfeat: Integrated recognition, localization and detection
using convolutional networks. In ICLR, 2014.
[41] K. Simonyan and A. Zisserman. Very deep convolutional networks
for large-scale image recognition. In ICLR, 2015.
[42] R. K. Srivastava, K. Greff, and J. Schmidhuber. Highway networks.
arXiv:1505.00387, 2015.
[43] R. K. Srivastava, K. Greff, and J. Schmidhuber. Training very deep
networks. 1507.06228, 2015.
[44] C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed, D. Anguelov, D. Er-
han, V . Vanhoucke, and A. Rabinovich. Going deeper with convolu-
tions. In CVPR, 2015.
[45] R. Szeliski. Fast surface interpolation using hierarchical basis func-
tions. TPAMI, 1990.
[46] R. Szeliski. Locally adapted hierarchical basis preconditioning. In
SIGGRAPH, 2006.
[47] T. Vatanen, T. Raiko, H. Valpola, and Y . LeCun. Pushing stochas-
tic gradient towards second-order methods–backpropagation learn-
ing with transformations in nonlinearities. In Neural Information
Processing, 2013.
[48] A. Vedaldi and B. Fulkerson. VLFeat: An open and portable library
of computer vision algorithms, 2008.
[49] W. Venables and B. Ripley. Modern applied statistics with s-plus.
1999.
[50] M. D. Zeiler and R. Fergus. Visualizing and understanding convolu-
tional neural networks. In ECCV, 2014.
9
A. Object Detection Baselines
In this section we introduce our detection method based
on the baseline Faster R-CNN [32] system. The models are
initialized by the ImageNet classiﬁcation models, and then
ﬁne-tuned on the object detection data. We have experi-
mented with ResNet-50/101 at the time of the ILSVRC &
COCO 2015 detection competitions.
Unlike VGG-16 used in [32], our ResNet has no hidden
fc layers. We adopt the idea of “Networks on Conv fea-
ture maps” (NoC) [33] to address this issue. We compute
the full-image shared conv feature maps using those lay-
ers whose strides on the image are no greater than 16 pixels
(i.e., conv1, conv2 x, conv3 x, and conv4 x, totally 91 conv
layers in ResNet-101; Table 1). We consider these layers as
analogous to the 13 conv layers in VGG-16, and by doing
so, both ResNet and VGG-16 have conv feature maps of the
same total stride (16 pixels). These layers are shared by a
region proposal network (RPN, generating 300 proposals)
[32] and a Fast R-CNN detection network [7]. RoI pool-
ing [7] is performed before conv5 1. On this RoI-pooled
feature, all layers of conv5 x and up are adopted for each
region, playing the roles of VGG-16’s fc layers. The ﬁnal
classiﬁcation layer is replaced by two sibling layers (classi-
ﬁcation and box regression [7]).
For the usage of BN layers, after pre-training, we com-
pute the BN statistics (means and variances) for each layer
on the ImageNet training set. Then the BN layers are ﬁxed
during ﬁne-tuning for object detection. As such, the BN
layers become linear activations with constant offsets and
scales, and BN statistics are not updated by ﬁne-tuning. We
ﬁx the BN layers mainly for reducing memory consumption
in Faster R-CNN training.
PASCAL VOC
Following [7, 32], for the PASCAL VOC 2007 test set,
we use the 5k trainval images in VOC 2007 and 16k train-
val images in VOC 2012 for training (“07+12”). For the
PASCAL VOC 2012 test set, we use the 10k trainval+test
images in VOC 2007 and 16ktrainval images in VOC 2012
for training (“07++12”). The hyper-parameters for train-
ing Faster R-CNN are the same as in [32]. Table 7 shows
the results. ResNet-101 improves the mAP by >3% over
VGG-16. This gain is solely because of the improved fea-
tures learned by ResNet.
MS COCO
The MS COCO dataset [26] involves 80 object cate-
gories. We evaluate the PASCAL VOC metric (mAP @
IoU = 0.5) and the standard COCO metric (mAP @ IoU =
.5:.05:.95). We use the 80k images on the train set for train-
ing and the 40k images on the val set for evaluation. Our
detection system for COCO is similar to that for PASCAL
VOC. We train the COCO models with an 8-GPU imple-
mentation, and thus the RPN step has a mini-batch size of
8 images (i.e., 1 per GPU) and the Fast R-CNN step has a
mini-batch size of 16 images. The RPN step and Fast R-
CNN step are both trained for 240k iterations with a learn-
ing rate of 0.001 and then for 80k iterations with 0.0001.
Table 8 shows the results on the MS COCO validation
set. ResNet-101 has a 6% increase of mAP@[.5, .95] over
VGG-16, which is a 28% relative improvement, solely con-
tributed by the features learned by the better network. Re-
markably, the mAP@[.5, .95]’s absolute increase (6.0%) is
nearly as big as mAP@.5’s (6.9%). This suggests that a
deeper network can improve both recognition and localiza-
tion.
B. Object Detection Improvements
For completeness, we report the improvements made for
the competitions. These improvements are based on deep
features and thus should beneﬁt from residual learning.
MS COCO
Box reﬁnement. Our box reﬁnement partially follows the it-
erative localization in [6]. In Faster R-CNN, the ﬁnal output
is a regressed box that is different from its proposal box. So
for inference, we pool a new feature from the regressed box
and obtain a new classiﬁcation score and a new regressed
box. We combine these 300 new predictions with the orig-
inal 300 predictions. Non-maximum suppression (NMS) is
applied on the union set of predicted boxes using an IoU
threshold of 0.3 [8], followed by box voting [6]. Box re-
ﬁnement improves mAP by about 2 points (Table 9).
Global context. We combine global context in the Fast
R-CNN step. Given the full-image conv feature map, we
pool a feature by global Spatial Pyramid Pooling [12] (with
a “single-level” pyramid) which can be implemented as
“RoI” pooling using the entire image’s bounding box as the
RoI. This pooled feature is fed into the post-RoI layers to
obtain a global context feature. This global feature is con-
catenated with the original per-region feature, followed by
the sibling classiﬁcation and box regression layers. This
new structure is trained end-to-end. Global context im-
proves mAP@.5 by about 1 point (Table 9).
Multi-scale testing. In the above, all results are obtained by
single-scale training/testing as in [32], where the image’s
shorter side is s = 600pixels. Multi-scale training/testing
has been developed in [12, 7] by selecting a scale from a
feature pyramid, and in [33] by using maxout layers. In
our current implementation, we have performed multi-scale
testing following [33]; we have not performed multi-scale
training because of limited time. In addition, we have per-
formed multi-scale testing only for the Fast R-CNN step
(but not yet for the RPN step). With a trained model, we
compute conv feature maps on an image pyramid, where the
image’s shorter sides are s ∈ {200,400,600,800,1000}.
10
training data COCO train COCO trainval
test data COCO val COCO test-dev
mAP @.5 @[.5, .95] @.5 @[.5, .95]
baseline Faster R-CNN (VGG-16) 41.5 21.2
baseline Faster R-CNN (ResNet-101) 48.4 27.2
+box reﬁnement 49.9 29.9
+context 51.1 30.0 53.3 32.2
+multi-scale testing 53.8 32.5 55.7 34.9
ensemble 59.0 37.4
Table 9. Object detection improvements on MS COCO using Faster R-CNN and ResNet-101.
system net data mAP areo bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv
baseline VGG-16 07+12 73.2 76.5 79.0 70.9 65.5 52.1 83.1 84.7 86.4 52.0 81.9 65.7 84.8 84.6 77.5 76.7 38.8 73.6 73.9 83.0 72.6
baseline ResNet-101 07+12 76.4 79.8 80.7 76.2 68.3 55.9 85.1 85.3 89.8 56.7 87.8 69.4 88.3 88.9 80.9 78.4 41.7 78.6 79.8 85.3 72.0
baseline+++ResNet-101 COCO+07+12 85.6 90.0 89.6 87.8 80.8 76.1 89.9 89.9 89.6 75.5 90.0 80.7 89.6 90.3 89.1 88.7 65.4 88.1 85.6 89.0 86.8
Table 10. Detection results on the PASCAL VOC 2007 test set. The baseline is the Faster R-CNN system. The system “baseline+++”
include box reﬁnement, context, and multi-scale testing in Table 9.
system net data mAP areo bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv
baseline VGG-16 07++12 70.4 84.9 79.8 74.3 53.9 49.8 77.5 75.9 88.5 45.6 77.1 55.3 86.9 81.7 80.9 79.6 40.1 72.6 60.9 81.2 61.5
baseline ResNet-101 07++12 73.8 86.5 81.6 77.2 58.0 51.0 78.6 76.6 93.2 48.6 80.4 59.0 92.1 85.3 84.8 80.7 48.1 77.3 66.5 84.7 65.6
baseline+++ResNet-101 COCO+07++12 83.8 92.1 88.4 84.8 75.9 71.4 86.3 87.8 94.2 66.8 89.4 69.2 93.9 91.9 90.9 89.6 67.9 88.2 76.8 90.3 80.0
Table 11. Detection results on the PASCAL VOC 2012 test set ( http://host.robots.ox.ac.uk:8080/leaderboard/
displaylb.php?challengeid=11&compid=4). The baseline is the Faster R-CNN system. The system “baseline+++” include
box reﬁnement, context, and multi-scale testing in Table 9.
We select two adjacent scales from the pyramid following
[33]. RoI pooling and subsequent layers are performed on
the feature maps of these two scales [33], which are merged
by maxout as in [33]. Multi-scale testing improves the mAP
by over 2 points (Table 9).
Using validation data. Next we use the 80k+40k trainval set
for training and the 20k test-dev set for evaluation. The test-
dev set has no publicly available ground truth and the result
is reported by the evaluation server. Under this setting, the
results are an mAP@.5 of 55.7% and an mAP@[.5, .95] of
34.9% (Table 9). This is our single-model result.
Ensemble. In Faster R-CNN, the system is designed to learn
region proposals and also object classiﬁers, so an ensemble
can be used to boost both tasks. We use an ensemble for
proposing regions, and the union set of proposals are pro-
cessed by an ensemble of per-region classiﬁers. Table 9
shows our result based on an ensemble of 3 networks. The
mAP is 59.0% and 37.4% on the test-dev set. This result
won the 1st place in the detection task in COCO 2015.
PASCAL VOC
We revisit the PASCAL VOC dataset based on the above
model. With the single model on the COCO dataset (55.7%
mAP@.5 in Table 9), we ﬁne-tune this model on the PAS-
CAL VOC sets. The improvements of box reﬁnement, con-
text, and multi-scale testing are also adopted. By doing so
val2 test
GoogLeNet [44] (ILSVRC’14) - 43.9
our single model (ILSVRC’15) 60.5 58.8
our ensemble (ILSVRC’15) 63.6 62.1
Table 12. Our results (mAP, %) on the ImageNet detection dataset.
Our detection system is Faster R-CNN [32] with the improvements
in Table 9, using ResNet-101.
we achieve 85.6% mAP on PASCAL VOC 2007 (Table 10)
and 83.8% on PASCAL VOC 2012 (Table 11)6. The result
on PASCAL VOC 2012 is 10 points higher than the previ-
ous state-of-the-art result [6].
ImageNet Detection
The ImageNet Detection (DET) task involves 200 object
categories. The accuracy is evaluated by mAP@.5. Our
object detection algorithm for ImageNet DET is the same
as that for MS COCO in Table 9. The networks are pre-
trained on the 1000-class ImageNet classiﬁcation set, and
are ﬁne-tuned on the DET data. We split the validation set
into two parts (val1/val2) following [8]. We ﬁne-tune the
detection models using the DET training set and the val1
set. The val2 set is used for validation. We do not use other
ILSVRC 2015 data. Our single model with ResNet-101 has
6http://host.robots.ox.ac.uk:8080/anonymous/3OJ4OJ.html,
submitted on 2015-11-26.
11
LOC
method
LOC
network testingLOC error
on GT CLS
classiﬁcation
network
top-5 LOC error
on predicted CLS
VGG’s [41]VGG-16 1-crop 33.1 [41]
RPN ResNet-1011-crop 13.3
RPN ResNet-101dense 11.7
RPN ResNet-101dense ResNet-101 14.4
RPN+RCNNResNet-101dense ResNet-101 10.6
RPN+RCNNensemble dense ensemble 8.9
Table 13. Localization error (%) on the ImageNet validation. In
the column of “LOC error on GT class” ([41]), the ground truth
class is used. In the “testing” column, “1-crop” denotes testing
on a center crop of 224×224 pixels, “dense” denotes dense (fully
convolutional) and multi-scale testing.
58.8% mAP and our ensemble of 3 models has 62.1% mAP
on the DET test set (Table 12).This result won the 1st place
in the ImageNet detection task in ILSVRC 2015, surpassing
the second place by 8.5 points (absolute).
C. ImageNet Localization
The ImageNet Localization (LOC) task [36] requires to
classify and localize the objects. Following [40, 41], we
assume that the image-level classiﬁers are ﬁrst adopted for
predicting the class labels of an image, and the localiza-
tion algorithm only accounts for predicting bounding boxes
based on the predicted classes. We adopt the “per-class re-
gression” (PCR) strategy [40, 41], learning a bounding box
regressor for each class. We pre-train the networks for Im-
ageNet classiﬁcation and then ﬁne-tune them for localiza-
tion. We train networks on the provided 1000-class Ima-
geNet training set.
Our localization algorithm is based on the RPN frame-
work of [32] with a few modiﬁcations. Unlike the way in
[32] that is category-agnostic, our RPN for localization is
designed in a per-class form. This RPN ends with two sib-
ling 1×1 convolutional layers for binary classiﬁcation (cls)
and box regression (reg), as in [32]. The cls and reg layers
are both in a per-class from, in contrast to [32]. Speciﬁ-
cally, the cls layer has a 1000-d output, and each dimension
is binary logistic regression for predicting being or not be-
ing an object class; the reg layer has a 1000 ×4-d output
consisting of box regressors for 1000 classes. As in [32],
our bounding box regression is with reference to multiple
translation-invariant “anchor” boxes at each position.
As in our ImageNet classiﬁcation training (Sec. 3.4), we
randomly sample 224 ×224 crops for data augmentation.
We use a mini-batch size of 256 images for ﬁne-tuning. To
avoid negative samples being dominate, 8 anchors are ran-
domly sampled for each image, where the sampled positive
and negative anchors have a ratio of 1:1 [32]. For testing,
the network is applied on the image fully-convolutionally.
Table 13 compares the localization results. Following
[41], we ﬁrst perform “oracle” testing using the ground truth
class as the classiﬁcation prediction. VGG’s paper [41] re-
method top-5 localization err
val test
OverFeat [40] (ILSVRC’13) 30.0 29.9
GoogLeNet [44] (ILSVRC’14) - 26.7
VGG [41] (ILSVRC’14) 26.9 25.3
ours (ILSVRC’15) 8.9 9.0
Table 14. Comparisons of localization error (%) on the ImageNet
dataset with state-of-the-art methods.
ports a center-crop error of 33.1% (Table 13) using ground
truth classes. Under the same setting, our RPN method us-
ing ResNet-101 net signiﬁcantly reduces the center-crop er-
ror to 13.3%. This comparison demonstrates the excellent
performance of our framework. With dense (fully convolu-
tional) and multi-scale testing, our ResNet-101 has an error
of 11.7% using ground truth classes. Using ResNet-101 for
predicting classes (4.6% top-5 classiﬁcation error, Table 4),
the top-5 localization error is 14.4%.
The above results are only based on theproposal network
(RPN) in Faster R-CNN [32]. One may use the detection
network (Fast R-CNN [7]) in Faster R-CNN to improve the
results. But we notice that on this dataset, one image usually
contains a single dominate object, and the proposal regions
highly overlap with each other and thus have very similar
RoI-pooled features. As a result, the image-centric training
of Fast R-CNN [7] generates samples of small variations,
which may not be desired for stochastic training. Motivated
by this, in our current experiment we use the original R-
CNN [8] that is RoI-centric, in place of Fast R-CNN.
Our R-CNN implementation is as follows. We apply the
per-class RPN trained as above on the training images to
predict bounding boxes for the ground truth class. These
predicted boxes play a role of class-dependent proposals.
For each training image, the highest scored 200 proposals
are extracted as training samples to train an R-CNN classi-
ﬁer. The image region is cropped from a proposal, warped
to 224×224 pixels, and fed into the classiﬁcation network
as in R-CNN [8]. The outputs of this network consist of two
sibling fc layers for cls and reg, also in a per-class form.
This R-CNN network is ﬁne-tuned on the training set us-
ing a mini-batch size of 256 in the RoI-centric fashion. For
testing, the RPN generates the highest scored 200 proposals
for each predicted class, and the R-CNN network is used to
update these proposals’ scores and box positions.
This method reduces the top-5 localization error to
10.6% (Table 13). This is our single-model result on the
validation set. Using an ensemble of networks for both clas-
siﬁcation and localization, we achieve a top-5 localization
error of 9.0% on the test set. This number signiﬁcantly out-
performs the ILSVRC 14 results (Table 14), showing a 64%
relative reduction of error. This result won the 1st place in
the ImageNet localization task in ILSVRC 2015.
12

Image-to-Image Translation with Conditional Adversarial Networks
Phillip Isola Jun-Yan Zhu Tinghui Zhou Alexei A. Efros
Berkeley AI Research (BAIR) Laboratory
University of California, Berkeley
{isola,junyanz,tinghuiz,efros}@eecs.berkeley.edu
Labels to Facade BW to Color
Aerial to Map
Labels to Street Scene
Edges to Photo
input output input
inputinput
input output
output
outputoutput
input output
Day to Night
Figure 1: Many problems in image processing, graphics, and vision involve translating an input image into a corresponding output image.
These problems are often treated with application-speciﬁc algorithms, even though the setting is always the same: map pixels to pixels.
Conditional adversarial nets are a general-purpose solution that appears to work well on a wide variety of these problems. Here we show
results of the method on several. In each case we use the same architecture and objective, and simply train on different data.
Abstract
We investigate conditional adversarial networks as a
general-purpose solution to image-to-image translation
problems. These networks not only learn the mapping from
input image to output image, but also learn a loss func-
tion to train this mapping. This makes it possible to apply
the same generic approach to problems that traditionally
would require very different loss formulations. We demon-
strate that this approach is effective at synthesizing photos
from label maps, reconstructing objects from edge maps,
and colorizing images, among other tasks. As a commu-
nity, we no longer hand-engineer our mapping functions,
and this work suggests we can achieve reasonable results
without hand-engineering our loss functions either.
Many problems in image processing, computer graphics,
and computer vision can be posed as “translating” an input
image into a corresponding output image. Just as a concept
may be expressed in either English or French, a scene may
be rendered as an RGB image, a gradient ﬁeld, an edge map,
a semantic label map, etc. In analogy to automatic language
translation, we deﬁne automaticimage-to-image translation
as the problem of translating one possible representation of
a scene into another, given sufﬁcient training data (see Fig-
ure 1). One reason language translation is difﬁcult is be-
cause the mapping between languages is rarely one-to-one
– any given concept is easier to express in one language
than another. Similarly, most image-to-image translation
problems are either many-to-one (computer vision) – map-
ping photographs to edges, segments, or semantic labels,
or one-to-many (computer graphics) – mapping labels or
sparse user inputs to realistic images. Traditionally, each of
these tasks has been tackled with separate, special-purpose
machinery (e.g., [7, 15, 11, 1, 3, 37, 21, 26, 9, 42, 46]),
despite the fact that the setting is always the same: predict
pixels from pixels. Our goal in this paper is to develop a
common framework for all these problems.
1
arXiv:1611.07004v1  [cs.CV]  21 Nov 2016
The community has already taken signiﬁcant steps in this
direction, with convolutional neural nets (CNNs) becoming
the common workhorse behind a wide variety of image pre-
diction problems. CNNs learn to minimize a loss function –
an objective that scores the quality of results – and although
the learning process is automatic, a lot of manual effort still
goes into designing effective losses. In other words, we still
have to tell the CNN what we wish it to minimize. But,
just like Midas, we must be careful what we wish for! If
we take a naive approach, and ask the CNN to minimize
Euclidean distance between predicted and ground truth pix-
els, it will tend to produce blurry results [29, 46]. This is
because Euclidean distance is minimized by averaging all
plausible outputs, which causes blurring. Coming up with
loss functions that force the CNN to do what we really want
– e.g., output sharp, realistic images – is an open problem
and generally requires expert knowledge.
It would be highly desirable if we could instead specify
only a high-level goal, like “make the output indistinguish-
able from reality”, and then automatically learn a loss func-
tion appropriate for satisfying this goal. Fortunately, this is
exactly what is done by the recently proposed Generative
Adversarial Networks (GANs) [14, 5, 30, 36, 47]. GANs
learn a loss that tries to classify if the output image is real
or fake, while simultaneously training a generative model
to minimize this loss. Blurry images will not be tolerated
since they look obviously fake. Because GANs learn a loss
that adapts to the data, they can be applied to a multitude of
tasks that traditionally would require very different kinds of
loss functions.
In this paper, we explore GANs in the conditional set-
ting. Just as GANs learn a generative model of data, condi-
tional GANs (cGANs) learn a conditional generative model
[14]. This makes cGANs suitable for image-to-image trans-
lation tasks, where we condition on an input image and gen-
erate a corresponding output image.
GANs have been vigorously studied in the last two
years and many of the techniques we explore in this pa-
per have been previously proposed. Nonetheless, ear-
lier papers have focused on speciﬁc applications, and
it has remained unclear how effective image-conditional
GANs can be as a general-purpose solution for image-to-
image translation. Our primary contribution is to demon-
strate that on a wide variety of problems, conditional
GANs produce reasonable results. Our second contri-
bution is to present a simple framework sufﬁcient to
achieve good results, and to analyze the effects of sev-
eral important architectural choices. Code is available at
https://github.com/phillipi/pix2pix.
1. Related work
Structured losses for image modeling Image-to-image
translation problems are often formulated as per-pixel clas-
siﬁcation or regression [26, 42, 17, 23, 46]. These for-
mulations treat the output space as “unstructured” in the
sense that each output pixel is considered conditionally in-
dependent from all others given the input image. Condi-
tional GANs instead learn a structured loss. Structured
losses penalize the joint conﬁguration of the output. A large
body of literature has considered losses of this kind, with
popular methods including conditional random ﬁelds [2],
the SSIM metric [40], feature matching [6], nonparametric
losses [24], the convolutional pseudo-prior [41], and losses
based on matching covariance statistics [19]. Our condi-
tional GAN is different in that the loss is learned, and can, in
theory, penalize any possible structure that differs between
output and target.
Conditional GANs We are not the ﬁrst to apply GANs
in the conditional setting. Previous works have conditioned
GANs on discrete labels [28], text [32], and, indeed, im-
ages. The image-conditional models have tackled inpaint-
ing [29], image prediction from a normal map [39], image
manipulation guided by user constraints [49], future frame
prediction [27], future state prediction [48], product photo
generation [43], and style transfer [25]. Each of these meth-
ods was tailored for a speciﬁc application. Our framework
differs in that nothing is application-speciﬁc. This makes
our setup considerably simpler than most others.
Our method also differs from these prior works in sev-
eral architectural choices for the generator and discrimina-
tor. Unlike past work, for our generator we use a “U-Net”-
based architecture [34], and for our discriminator we use a
convolutional “PatchGAN” classiﬁer, which only penalizes
structure at the scale of image patches. A similar Patch-
GAN architecture was previously proposed in [25], for the
purpose of capturing local style statistics. Here we show
that this approach is effective on a wider range of problems,
and we investigate the effect of changing the patch size.
2. Method
GANs are generative models that learn a mapping from
random noise vector z to output image y: G : z → y
[14]. In contrast, conditional GANs learn a mapping from
observed image x and random noise vector z, to y: G :
{x,z}→ y. The generator Gis trained to produce outputs
that cannot be distinguished from “real” images by an ad-
versarially trained discrimintor,D, which is trained to do as
well as possible at detecting the generator’s “fakes”. This
training procedure is diagrammed in Figure 2.
2.1. Objective
The objective of a conditional GAN can be expressed as
LcGAN (G,D) =Ex,y∼pdata(x,y)[log D(x,y)]+
Ex∼pdata(x),z∼pz(z)[log(1 −D(x,G(x,z))],
(1)
Real or fake pair?
Positive examples Negative examples
Real or fake pair?
DD
G
G tries to synthesize fake 
images that fool D
D tries to identify the fakes
Figure 2: Training a conditional GAN to predict aerial photos from
maps. The discriminator, D, learns to classify between real and
synthesized pairs. The generator learns to fool the discriminator.
Unlike an unconditional GAN, both the generator and discrimina-
tor observe an input image.
where G tries to minimize this objective against an ad-
versarial D that tries to maximize it, i.e. G∗ =
arg minG maxD LcGAN (G,D).
To test the importance of conditioning the discrimintor,
we also compare to an unconditional variant in which the
discriminator does not observe x:
LGAN (G,D) =Ey∼pdata(y)[log D(y)]+
Ex∼pdata(x),z∼pz(z)[log(1 −D(G(x,z))].
(2)
Previous approaches to conditional GANs have found it
beneﬁcial to mix the GAN objective with a more traditional
loss, such as L2 distance [29]. The discriminator’s job re-
mains unchanged, but the generator is tasked to not only
fool the discriminator but also to be near the ground truth
output in an L2 sense. We also explore this option, using
L1 distance rather than L2 as L1 encourages less blurring:
LL1(G) =Ex,y∼pdata(x,y),z∼pz(z)[∥y−G(x,z)∥1]. (3)
Our ﬁnal objective is
G∗ = arg min
G
max
D
LcGAN (G,D) +λLL1(G). (4)
Without z, the net could still learn a mapping from xto
y, but would produce deterministic outputs, and therefore
fail to match any distribution other than a delta function.
Past conditional GANs have acknowledged this and pro-
vided Gaussian noise zas an input to the generator, in addi-
tion to x(e.g., [39]). In initial experiments, we did not ﬁnd
Encoder-decoder U-Net
Figure 3: Two choices for the architecture of the generator. The
“U-Net” [34] is an encoder-decoder with skip connections be-
tween mirrored layers in the encoder and decoder stacks.
this strategy effective – the generator simply learned to ig-
nore the noise – which is consistent with Mathieu et al. [27].
Instead, for our ﬁnal models, we provide noise only in the
form of dropout, applied on several layers of our generator
at both training and test time. Despite the dropout noise, we
observe very minor stochasticity in the output of our nets.
Designing conditional GANs that produce stochastic out-
put, and thereby capture the full entropy of the conditional
distributions they model, is an important question left open
by the present work.
2.2. Network architectures
We adapt our generator and discriminator architectures
from those in [30]. Both generator and discriminator use
modules of the form convolution-BatchNorm-ReLu [18].
Details of the architecture are provided in the appendix,
with key features discussed below.
2.2.1 Generator with skips
A deﬁning feature of image-to-image translation problems
is that they map a high resolution input grid to a high resolu-
tion output grid. In addition, for the problems we consider,
the input and output differ in surface appearance, but both
are renderings of the same underlying structure. Therefore,
structure in the input is roughly aligned with structure in the
output. We design the generator architecture around these
considerations.
Many previous solutions [29, 39, 19, 48, 43] to problems
in this area have used an encoder-decoder network [16]. In
such a network, the input is passed through a series of lay-
ers that progressively downsample, until a bottleneck layer,
at which point the process is reversed (Figure 3). Such a
network requires that all information ﬂow pass through all
the layers, including the bottleneck. For many image trans-
lation problems, there is a great deal of low-level informa-
tion shared between the input and output, and it would be
desirable to shuttle this information directly across the net.
For example, in the case of image colorizaton, the input and
output share the location of prominent edges.
To give the generator a means to circumvent the bot-
tleneck for information like this, we add skip connections,
following the general shape of a “U-Net” [34] (Figure 3).
Speciﬁcally, we add skip connections between each layer i
and layer n−i, where nis the total number of layers. Each
skip connection simply concatenates all channels at layer i
with those at layer n−i.
2.2.2 Markovian discriminator (PatchGAN)
It is well known that the L2 loss – and L1, see Fig-
ure 4 – produces blurry results on image generation prob-
lems [22]. Although these losses fail to encourage high-
frequency crispness, in many cases they nonetheless accu-
rately capture the low frequencies. For problems where this
is the case, we do not need an entirely new framework to
enforce correctness at the low frequencies. L1 will already
do.
This motivates restricting the GAN discriminator to only
model high-frequency structure, relying on an L1 term to
force low-frequency correctness (Eqn. 4). In order to model
high-frequencies, it is sufﬁcient to restrict our attention to
the structure in local image patches. Therefore, we design
a discriminator architecture – which we term a PatchGAN
– that only penalizes structure at the scale of patches. This
discriminator tries to classify if each N ×N patch in an
image is real or fake. We run this discriminator convoluta-
tionally across the image, averaging all responses to provide
the ultimate output of D.
In Section 3.4, we demonstrate that N can be much
smaller than the full size of the image and still produce
high quality results. This is advantageous because a smaller
PatchGAN has fewer parameters, runs faster, and can be
applied on arbitrarily large images.
Such a discriminator effectively models the image as a
Markov random ﬁeld, assuming independence between pix-
els separated by more than a patch diameter. This con-
nection was previously explored in [25], and is also the
common assumption in models of texture [8, 12] and style
[7, 15, 13, 24]. Our PatchGAN can therefore be understood
as a form of texture/style loss.
2.3. Optimization and inference
To optimize our networks, we follow the standard ap-
proach from [14]: we alternate between one gradient de-
scent step on D, then one step on G. We use minibatch
SGD and apply the Adam solver [20].
At inference time, we run the generator net in exactly
the same manner as during the training phase. This differs
from the usual protocol in that we apply dropout at test time,
and we apply batch normalization [18] using the statistics of
the test batch, rather than aggregated statistics of the train-
ing batch. This approach to batch normalization, when the
batch size is set to 1, has been termed “instance normaliza-
tion” and has been demonstrated to be effective at image
generation tasks [38]. In our experiments, we use batch size
1 for certain experiments and 4 for others, noting little dif-
ference between these two conditions.
3. Experiments
To explore the generality of conditional GANs, we test
the method on a variety of tasks and datasets, including both
graphics tasks, like photo generation, and vision tasks, like
semantic segmentation:
•Semantic labels↔photo, trained on the Cityscapes
dataset [4].
•Architectural labels→photo, trained on the CMP Fa-
cades dataset [31].
•Map↔aerial photo, trained on data scraped from
Google Maps.
•BW→color photos, trained on [35].
•Edges→photo, trained on data from [49] and [44]; bi-
nary edges generated using the HED edge detector [42]
plus postprocessing.
•Sketch→photo: tests edges→photo models on human-
drawn sketches from [10].
•Day→night, trained on [21].
Details of training on each of these datasets are pro-
vided in the Appendix. In all cases, the input and out-
put are simply 1-3 channel images. Qualitative results
are shown in Figures 8, 9, 10, 11, 12, 14, 15, 16,
and 13. Several failure cases are highlighted in Fig-
ure 17. More comprehensive results are available at
https://phillipi.github.io/pix2pix/.
Data requirements and speed We note that decent re-
sults can often be obtained even on small datasets. Our fa-
cade training set consists of just 400 images (see results in
Figure 12), and the day to night training set consists of only
91 unique webcams (see results in Figure 13). On datasets
of this size, training can be very fast: for example, the re-
sults shown in Figure 12 took less than two hours of training
on a single Pascal Titan X GPU. At test time, all models run
in well under a second on this GPU.
3.1. Evaluation metrics
Evaluating the quality of synthesized images is an open
and difﬁcult problem [36]. Traditional metrics such as per-
pixel mean-squared error do not assess joint statistics of the
result, and therefore do not measure the very structure that
structured losses aim to capture.
In order to more holistically evaluate the visual qual-
ity of our results, we employ two tactics. First, we run
Loss Per-pixel acc. Per-class acc. Class IOU
L1 0.44 0.14 0.10
GAN 0.22 0.05 0.01
cGAN 0.61 0.21 0.16
L1+GAN 0.64 0.19 0.15
L1+cGAN 0.63 0.21 0.16
Ground truth 0.80 0.26 0.21
Table 1: FCN-scores for different losses, evaluated on Cityscapes
labels↔photos.
“real vs fake” perceptual studies on Amazon Mechanical
Turk (AMT). For graphics problems like colorization and
photo generation, plausibility to a human observer is often
the ultimate goal. Therefore, we test our map generation,
aerial photo generation, and image colorization using this
approach.
Second, we measure whether or not our synthesized
cityscapes are realistic enough that off-the-shelf recognition
system can recognize the objects in them. This metric is
similar to the “inception score” from [36], the object detec-
tion evaluation in [39], and the “semantic interpretability”
measure in [46].
AMT perceptual studiesFor our AMT experiments, we
followed the protocol from [46]: Turkers were presented
with a series of trials that pitted a “real” image against a
“fake” image generated by our algorithm. On each trial,
each image appeared for 1 second, after which the images
disappeared and Turkers were given unlimited time to re-
spond as to which was fake. The ﬁrst 10 images of each
session were practice and Turkers were given feedback. No
feedback was provided on the 40 trials of the main experi-
ment. Each session tested just one algorithm at a time, and
Turkers were not allowed to complete more than one ses-
sion. ∼50 Turkers evaluated each algorithm. All images
were presented at 256 ×256 resolution. Unlike [46], we
did not include vigilance trials. For our colorization ex-
periments, the real and fake images were generated from
the same grayscale input. For map ↔aerial photo, the real
and fake images were not generated from the same input, in
order to make the task more difﬁcult and avoid ﬂoor-level
results.
FCN-score While quantitative evaluation of generative
models is known to be challenging, recent works [36,
39, 46] have tried using pre-trained semantic classiﬁers to
measure the discriminability of the generated images as a
pseudo-metric. The intuition is that if the generated images
are realistic, classiﬁers trained on real images will be able
to classify the synthesized image correctly as well. To this
end, we adopt the popular FCN-8s [26] architecture for se-
mantic segmentation, and train it on the cityscapes dataset.
We then score synthesized photos by the classiﬁcation accu-
racy against the labels these photos were synthesized from.
L1+cGANL1
Encoder-decoderU-Net
Figure 5: Adding skip connections to an encoder-decoder to create
a “U-Net” results in much higher quality results.
Discriminator
receptive ﬁeld Per-pixel acc. Per-class acc. Class IOU
1×1 0.44 0.14 0.10
16×16 0.62 0.20 0.16
70×70 0.63 0.21 0.16
256×256 0.47 0.18 0.13
Table 2: FCN-scores for different receptive ﬁeld sizes of the dis-
criminator, evaluated on Cityscapes labels→photos.
3.2. Analysis of the objective function
Which components of the objective in Eqn. 4 are impor-
tant? We run ablation studies to isolate the effect of the L1
term, the GAN term, and to compare using a discriminator
conditioned on the input (cGAN, Eqn. 1) against using an
unconditional discriminator (GAN, Eqn. 2).
Figure 4 shows the qualitative effects of these variations
on two labels→photo problems. L1 alone leads to reason-
able but blurry results. The cGAN alone (setting λ = 0in
Eqn. 4) gives much sharper results, but results in some arti-
facts in facade synthesis. Adding both terms together (with
λ= 100) reduces these artifacts.
We quantify these observations using the FCN-score on
the cityscapes labels→photo task (Table 1): the GAN-based
objectives achieve higher scores, indicating that the synthe-
sized images include more recognizable structure. We also
test the effect of removing conditioning from the discrimi-
nator (labeled as GAN). In this case, the loss does not pe-
nalize mismatch between the input and output; it only cares
that the output look realistic. This variant results in very
poor performance; examining the results reveals that the
generator collapsed into producing nearly the exact same
output regardless of input photograph. Clearly it is impor-
tant, in this case, that the loss measure the quality of the
match between input and output, and indeed cGAN per-
forms much better than GAN. Note, however, that adding
an L1 term also encourages that the output respect the in-
put, since the L1 loss penalizes the distance between ground
truth outputs, which match the input, and synthesized out-
puts, which may not. Correspondingly, L1+GAN is also
effective at creating realistic renderings that respect the in-
Input Ground truth L1 cGAN L1 + cGAN
Figure 4: Different losses induce different quality of results. Each column shows results trained under a different loss. Please see
https://phillipi.github.io/pix2pix/ for additional examples.
L1 1x1 16x16 70x70 256x256
Figure 6: Patch size variations. Uncertainty in the output manifests itself differently for different loss functions. Uncertain regions become
blurry and desaturated under L1. The 1x1 PixelGAN encourages greater color diversity but has no effect on spatial statistics. The 16x16
PatchGAN creates locally sharp results, but also leads to tiling artifacts beyond the scale it can observe. The 70x70 PatchGAN forces
outputs that are sharp, even if incorrect, in both the spatial and spectral (coforfulness) dimensions. The full 256x256 ImageGAN produces
results that are visually similar to the 70x70 PatchGAN, but somewhat lower quality according to our FCN-score metric (Table 2). Please
see https://phillipi.github.io/pix2pix/ for additional examples.
put label maps. Combining all terms, L1+cGAN, performs
similarly well.
Colorfulness A striking effect of conditional GANs is
that they produce sharp images, hallucinating spatial struc-
ture even where it does not exist in the input label map. One
might imagine cGANs have a similar effect on “sharpening”
in the spectral dimension – i.e. making images more color-
ful. Just as L1 will incentivize a blur when it is uncertain
where exactly to locate an edge, it will also incentivize an
average, grayish color when it is uncertain which of several
plausible color values a pixel should take on. Specially, L1
will be minimized by choosing the median of of the con-
ditional probability density function over possible colors.
An adversarial loss, on the other hand, can in principle be-
come aware that grayish outputs are unrealistic, and encour-
age matching the true color distribution [14]. In Figure 7,
we investigate if our cGANs actually achieve this effect on
the Cityscapes dataset. The plots show the marginal distri-
butions over output color values in Lab color space. The
ground truth distributions are shown with a dotted line. It
is apparent that L1 leads to a narrower distribution than the
ground truth, conﬁrming the hypothesis that L1 encourages
average, grayish colors. Using a cGAN, on the other hand,
pushes the output distribution closer to the ground truth.
3.3. Analysis of the generator architecture
A U-Net architecture allows low-level information to
shortcut across the network. Does this lead to better results?
Figure 5 compares the U-Net against an encoder-decoder on
cityscape generation U-Net. The encoder-decoder is created
simply by severing the skip connections in the U-Net. The
encoder-decoder is unable to learn to generate realistic im-
ages in our experiments, and indeed collapses to producing
nearly identical results for each input label map. The advan-
tages of the U-Net appear not to be speciﬁc to conditional
GANs: when both U-Net and encoder-decoder are trained
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
CVPR
#385
CVPR
#385
CVPR 2016 Submission #385. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.
70 90 110 130 150−11
−9
−7
−5
−3
−1
b
70 90 110 130−11
−9
−7
−5
−3
−1
a0 20 40 60 80 100−11
−9
−7
−5
−3
−1
L
 
 
L1cGANL1+cGANL1+pixelcGANGround truth
(a)
70 90 110 130 150−11
−9
−7
−5
−3
−1
b
70 90 110 130−11
−9
−7
−5
−3
−1
a0 20 40 60 80 100−11
−9
−7
−5
−3
−1
L
 
 
L1cGANL1+cGANL1+pixelcGANGround truth
(b)
70 90 110 130 150−11
−9
−7
−5
−3
−1
b
70 90 110 130−11
−9
−7
−5
−3
−1
a0 20 40 60 80 100−11
−9
−7
−5
−3
−1
L
 
 
L1cGANL1+cGANL1+pixelcGANGround truth
(c)
Histogram intersection
against ground truth
Loss La b
L1 0.81 0.69 0.70
cGAN 0.87 0.74 0.84
L1+cGAN 0.86 0.84 0.82
PixelGAN 0.83 0.68 0.78
(d)
Figure 5: Color distribution matching property of the cGAN, tested on Cityscapes. (c.f. Figure 1 of the original GAN paper [ 14 ]). Note
that the histogram intersection scores are dominated by differences in the high probability region, which are imperceptible in the plots,
which show log probability and therefore emphasize differences in the low probability regions.
L1 1x1 16x16 70x70 256x256
Figure 6: Patch size variations. Uncertainty in the output manifests itself differently for different loss functions. Uncertain regions become
blurry and desaturated under L1. The 1x1 PixelGAN encourages greater color diversity but has no effect on spatial statistics. The 16x16
PatchGAN creates locally sharp results, but also leads to tiling artifacts beyond the scale it can observe. The 70x70 PatchGAN forces
outputs that are sharp, even if incorrect, in both the spatial and spectral (coforfulness) dimensions. The full 256x256 ImageGAN produces
results that are visually similar to the 70x70 PatchGAN, but somewhat lower quality according to our FCN-score metric (Table 2 ).
Classiﬁcation Ours
L2 [ 44 ] (rebal.) [ 44 ] (L1 + cGAN) Ground truth
Figure 7: Colorization results of conditional GANs versus the L2
regression from [ 44 ] and the full method (classiﬁcation with re-
balancing) from [ 46 ]. The cGANs can produce compelling col-
orizations (ﬁrst two rows), but have a common failure mode of
producing a grayscale or desaturated result (last row).
To begin to test this, we train a cGAN (with/without L1
loss) on cityscape photo ! labels. Figure 8 shows qualita-
tive results, and quantitative classiﬁcation accuracies are re-
ported in Table 4 . Interestingly, cGANs, trained without the
L1 loss, are able to solve this problem at a reasonable degree
of accuracy. To our knowledge, this is the ﬁrst demonstra-
tion of GANs successfully generating “labels”, which are
Input Ground truth L1 cGAN
Figure 8: Applying a conditional GAN to semantic segmentation.
The cGAN produces sharp images that look at glance like the
ground truth, but in fact include many small, hallucinated objects.
nearly discrete, rather than “images”, with their continuous-
valued variation. Although cGANs achieve some success,
they are far from the best available method for solving this
problem: simply using L1 regression gets better scores than
using a cGAN, as shown in Table 4 . We argue that for vision
problems, the goal (i.e. predicting output close to ground
truth) may be less ambiguous than graphics tasks, and re-
construction losses like L1 are mostly sufﬁcient.
4. Conclusion
The results in this paper suggest that conditional adver-
sarial networks are a promising approach for many image-
to-image translation tasks, especially those involving highly
structured graphical outputs. These networks learn a loss
adapted to the task and data at hand, which makes them ap-
plicable in a wide variety of settings.
7
log P ( L )
log P ( a )
log P ( b)
L a b
(a)
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
CVPR
#385
CVPR
#385
CVPR 2016 Submission #385. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.
70 90 110 130 150−11
−9
−7
−5
−3
−1
b
70 90 110 130−11
−9
−7
−5
−3
−1
a0 20 40 60 80 100−11
−9
−7
−5
−3
−1
L
 
 
L1cGANL1+cGANL1+pixelcGANGround truth
(a)
70 90 110 130 150−11
−9
−7
−5
−3
−1
b
70 90 110 130−11
−9
−7
−5
−3
−1
a0 20 40 60 80 100−11
−9
−7
−5
−3
−1
L
 
 
L1cGANL1+cGANL1+pixelcGANGround truth
(b)
70 90 110 130 150−11
−9
−7
−5
−3
−1
b
70 90 110 130−11
−9
−7
−5
−3
−1
a0 20 40 60 80 100−11
−9
−7
−5
−3
−1
L
 
 
L1cGANL1+cGANL1+pixelcGANGround truth
(c)
Histogram intersection
against ground truth
Loss La b
L1 0.81 0.69 0.70
cGAN 0.87 0.74 0.84
L1+cGAN 0.86 0.84 0.82
PixelGAN 0.83 0.68 0.78
(d)
Figure 5: Color distribution matching property of the cGAN, tested on Cityscapes. (c.f. Figure 1 of the original GAN paper [ 14 ]). Note
that the histogram intersection scores are dominated by differences in the high probability region, which are imperceptible in the plots,
which show log probability and therefore emphasize differences in the low probability regions.
L1 1x1 16x16 70x70 256x256
Figure 6: Patch size variations. Uncertainty in the output manifests itself differently for different loss functions. Uncertain regions become
blurry and desaturated under L1. The 1x1 PixelGAN encourages greater color diversity but has no effect on spatial statistics. The 16x16
PatchGAN creates locally sharp results, but also leads to tiling artifacts beyond the scale it can observe. The 70x70 PatchGAN forces
outputs that are sharp, even if incorrect, in both the spatial and spectral (coforfulness) dimensions. The full 256x256 ImageGAN produces
results that are visually similar to the 70x70 PatchGAN, but somewhat lower quality according to our FCN-score metric (Table 2 ).
Classiﬁcation Ours
L2 [ 44 ] (rebal.) [ 44 ] (L1 + cGAN) Ground truth
Figure 7: Colorization results of conditional GANs versus the L2
regression from [ 44 ] and the full method (classiﬁcation with re-
balancing) from [ 46 ]. The cGANs can produce compelling col-
orizations (ﬁrst two rows), but have a common failure mode of
producing a grayscale or desaturated result (last row).
To begin to test this, we train a cGAN (with/without L1
loss) on cityscape photo ! labels. Figure 8 shows qualita-
tive results, and quantitative classiﬁcation accuracies are re-
ported in Table 4 . Interestingly, cGANs, trained without the
L1 loss, are able to solve this problem at a reasonable degree
of accuracy. To our knowledge, this is the ﬁrst demonstra-
tion of GANs successfully generating “labels”, which are
Input Ground truth L1 cGAN
Figure 8: Applying a conditional GAN to semantic segmentation.
The cGAN produces sharp images that look at glance like the
ground truth, but in fact include many small, hallucinated objects.
nearly discrete, rather than “images”, with their continuous-
valued variation. Although cGANs achieve some success,
they are far from the best available method for solving this
problem: simply using L1 regression gets better scores than
using a cGAN, as shown in Table 4 . We argue that for vision
problems, the goal (i.e. predicting output close to ground
truth) may be less ambiguous than graphics tasks, and re-
construction losses like L1 are mostly sufﬁcient.
4. Conclusion
The results in this paper suggest that conditional adver-
sarial networks are a promising approach for many image-
to-image translation tasks, especially those involving highly
structured graphical outputs. These networks learn a loss
adapted to the task and data at hand, which makes them ap-
plicable in a wide variety of settings.
7
log P ( L )
log P ( a )
log P ( b)
L a b (b)
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
CVPR
#385
CVPR
#385
CVPR 2016 Submission #385. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.
70 90 110 130 150−11
−9
−7
−5
−3
−1
b
70 90 110 130−11
−9
−7
−5
−3
−1
a0 20 40 60 80 100−11
−9
−7
−5
−3
−1
L
 
 
L1cGANL1+cGANL1+pixelcGANGround truth
(a)
70 90 110 130 150−11
−9
−7
−5
−3
−1
b
70 90 110 130−11
−9
−7
−5
−3
−1
a0 20 40 60 80 100−11
−9
−7
−5
−3
−1
L
 
 
L1cGANL1+cGANL1+pixelcGANGround truth
(b)
70 90 110 130 150−11
−9
−7
−5
−3
−1
b
70 90 110 130−11
−9
−7
−5
−3
−1
a0 20 40 60 80 100−11
−9
−7
−5
−3
−1
L
 
 
L1cGANL1+cGANL1+pixelcGANGround truth
(c)
Histogram intersection
against ground truth
Loss La b
L1 0.81 0.69 0.70
cGAN 0.87 0.74 0.84
L1+cGAN 0.86 0.84 0.82
PixelGAN 0.83 0.68 0.78
(d)
Figure 5: Color distribution matching property of the cGAN, tested on Cityscapes. (c.f. Figure 1 of the original GAN paper [ 14 ]). Note
that the histogram intersection scores are dominated by differences in the high probability region, which are imperceptible in the plots,
which show log probability and therefore emphasize differences in the low probability regions.
L1 1x1 16x16 70x70 256x256
Figure 6: Patch size variations. Uncertainty in the output manifests itself differently for different loss functions. Uncertain regions become
blurry and desaturated under L1. The 1x1 PixelGAN encourages greater color diversity but has no effect on spatial statistics. The 16x16
PatchGAN creates locally sharp results, but also leads to tiling artifacts beyond the scale it can observe. The 70x70 PatchGAN forces
outputs that are sharp, even if incorrect, in both the spatial and spectral (coforfulness) dimensions. The full 256x256 ImageGAN produces
results that are visually similar to the 70x70 PatchGAN, but somewhat lower quality according to our FCN-score metric (Table 2 ).
Classiﬁcation Ours
L2 [ 44 ] (rebal.) [ 44 ] (L1 + cGAN) Ground truth
Figure 7: Colorization results of conditional GANs versus the L2
regression from [ 44 ] and the full method (classiﬁcation with re-
balancing) from [ 46 ]. The cGANs can produce compelling col-
orizations (ﬁrst two rows), but have a common failure mode of
producing a grayscale or desaturated result (last row).
To begin to test this, we train a cGAN (with/without L1
loss) on cityscape photo ! labels. Figure 8 shows qualita-
tive results, and quantitative classiﬁcation accuracies are re-
ported in Table 4 . Interestingly, cGANs, trained without the
L1 loss, are able to solve this problem at a reasonable degree
of accuracy. To our knowledge, this is the ﬁrst demonstra-
tion of GANs successfully generating “labels”, which are
Input Ground truth L1 cGAN
Figure 8: Applying a conditional GAN to semantic segmentation.
The cGAN produces sharp images that look at glance like the
ground truth, but in fact include many small, hallucinated objects.
nearly discrete, rather than “images”, with their continuous-
valued variation. Although cGANs achieve some success,
they are far from the best available method for solving this
problem: simply using L1 regression gets better scores than
using a cGAN, as shown in Table 4 . We argue that for vision
problems, the goal (i.e. predicting output close to ground
truth) may be less ambiguous than graphics tasks, and re-
construction losses like L1 are mostly sufﬁcient.
4. Conclusion
The results in this paper suggest that conditional adver-
sarial networks are a promising approach for many image-
to-image translation tasks, especially those involving highly
structured graphical outputs. These networks learn a loss
adapted to the task and data at hand, which makes them ap-
plicable in a wide variety of settings.
7
log P ( L )
log P ( a )
log P ( b)
L a b (c)
Histogram intersection
against ground truth
Loss L a b
L1 0.81 0.69 0.70
cGAN 0.87 0.74 0.84
L1+cGAN0.86 0.84 0.82
PixelGAN0.83 0.68 0.78
(d)
Figure 7: Color distribution matching property of the cGAN, tested on Cityscapes. (c.f. Figure 1 of the original GAN paper [14]). Note
that the histogram intersection scores are dominated by differences in the high probability region, which are imperceptible in the plots,
which show log probability and therefore emphasize differences in the low probability regions.
with an L1 loss, the U-Net again achieves the superior re-
sults (Figure 5).
3.4. From PixelGANs to PatchGans to ImageGANs
We test the effect of varying the patch sizeN of our dis-
criminator receptive ﬁelds, from a 1 ×1 “PixelGAN” to a
full 256 ×256 “ImageGAN”1. Figure 6 shows qualitative
results of this analysis and Table 2 quantiﬁes the effects us-
ing the FCN-score. Note that elsewhere in this paper, unless
speciﬁed, all experiments use 70 ×70 PatchGANs, and for
this section all experiments use an L1+cGAN loss.
The PixelGAN has no effect on spatial sharpness, but
does increase the colorfulness of the results (quantiﬁed in
Figure 7). For example, the bus in Figure 6 is painted gray
when the net is trained with an L1 loss, but becomes red
with the PixelGAN loss. Color histogram matching is a
common problem in image processing [33], and PixelGANs
may be a promising lightweight solution.
Using a 16×16 PatchGAN is sufﬁcient to promote sharp
outputs, but also leads to tiling artifacts. The70×70 Patch-
GAN alleviates these artifacts. Scaling beyond this, to the
full 256 ×256 ImageGAN, does not appear to improve the
visual quality of the results, and in fact gets a considerably
lower FCN-score (Table 2). This may be because the Im-
ageGAN has many more parameters and greater depth than
the 70 ×70 PatchGAN, and may be harder to train.
Fully-convolutional translation An advantage of the
PatchGAN is that a ﬁxed-size patch discriminator can be
applied to arbitrarily large images. We may also apply the
generator convolutionally, on larger images than those on
which it was trained. We test this on the map↔aerial photo
task. After training a generator on256×256 images, we test
it on 512×512 images. The results in Figure 8 demonstrate
the effectiveness of this approach.
1We achieve this variation in patch size by adjusting the depth of the
GAN discriminator. Details of this process, and the discriminator architec-
tures are provided in the appendix
Classiﬁcation OursL2 [46] (rebal.) [46] (L1 + cGAN) Ground truth
Figure 9: Colorization results of conditional GANs versus the L2
regression from [46] and the full method (classiﬁcation with re-
balancing) from [48]. The cGANs can produce compelling col-
orizations (ﬁrst two rows), but have a common failure mode of
producing a grayscale or desaturated result (last row).
Photo→Map Map →Photo
Loss % Turkers labeledreal % Turkers labeledreal
L1 2.8%±1.0% 0.8% ±0.3%
L1+cGAN 6.1%±1.3% 18.9%±2.5%
Table 3: AMT “real vs fake” test on maps↔aerial photos.
Method % Turkers labeledreal
L2 regression from [46]16.3%±2.4%
Zhang et al. 2016 [46] 27.8%±2.7%
Ours 22.5%±1.6%
Table 4: AMT “real vs fake” test on colorization.
Loss Per-pixel acc. Per-class acc. Class IOU
L1 0.86 0.42 0.35
cGAN 0.74 0.28 0.22
L1+cGAN 0.83 0.36 0.29
Table 5: Performance of photo→labels on cityscapes.
Input Ground truth L1 cGAN
Figure 10: Applying a conditional GAN to semantic segmenta-
tion. The cGAN produces sharp images that look at glance like
the ground truth, but in fact include many small, hallucinated ob-
jects.
3.5. Perceptual validation
We validate the perceptual realism of our results on the
tasks of map↔aerial photograph and grayscale→color. Re-
sults of our AMT experiment for map ↔photo are given in
Table 3. The aerial photos generated by our method fooled
participants on 18.9% of trials, signiﬁcantly above the L1
baseline, which produces blurry results and nearly never
fooled participants. In contrast, in the photo →map direc-
tionm our method only fooled participants on 6.1% of tri-
als, and this was not signiﬁcantly different than the perfor-
mance of the L1 baseline (based on bootstrap test). This
may be because minor structural errors are more visible
in maps, which have rigid geometry, than in aerial pho-
tographs, which are more chaotic.
We trained colorization on ImageNet [35], and tested
on the test split introduced by [46, 23]. Our method, with
L1+cGAN loss, fooled participants on 22.5% of trials (Ta-
ble 4). We also tested the results of [46] and a variant of
their method that used an L2 loss (see [46] for details). The
conditional GAN scored similarly to the L2 variant of [46]
(difference insigniﬁcant by bootstrap test), but fell short of
[46]’s full method, which fooled participants on 27.8% of
trials in our experiment. We note that their method was
speciﬁcally engineered to do well on colorization.
3.6. Semantic segmentation
Conditional GANs appear to be effective on problems
where the output is highly detailed or photographic, as is
common in image processing and graphics tasks. What
about vision problems, like semantic segmentation, where
the output is instead less complex than the input?
To begin to test this, we train a cGAN (with/without L1
loss) on cityscape photo→labels. Figure 10 shows qualita-
tive results, and quantitative classiﬁcation accuracies are re-
ported in Table 5. Interestingly, cGANs, trainedwithout the
L1 loss, are able to solve this problem at a reasonable degree
of accuracy. To our knowledge, this is the ﬁrst demonstra-
tion of GANs successfully generating “labels”, which are
nearly discrete, rather than “images”, with their continuous-
valued variation2. Although cGANs achieve some success,
they are far from the best available method for solving this
problem: simply using L1 regression gets better scores than
using a cGAN, as shown in Table 5. We argue that for vision
problems, the goal (i.e. predicting output close to ground
truth) may be less ambiguous than graphics tasks, and re-
construction losses like L1 are mostly sufﬁcient.
4. Conclusion
The results in this paper suggest that conditional adver-
sarial networks are a promising approach for many image-
to-image translation tasks, especially those involving highly
structured graphical outputs. These networks learn a loss
adapted to the task and data at hand, which makes them ap-
plicable in a wide variety of settings.
Acknowledgments: We thank Richard Zhang and Deepak Pathak
for helpful discussions. This work was supported in part by NSF SMA-
1514512, NGA NURI, IARPA via Air Force Research Laboratory, Intel
Corp, and hardware donations by nVIDIA. Disclaimer: The views and
conclusions contained herein are those of the authors and should not be in-
terpreted as necessarily representing the ofﬁcial policies or endorsements,
either expressed or implied, of IARPA, AFRL or the U.S. Government.
2Note that the label maps we train on are not exactly discrete valued,
as they are resized from the original maps using bilinear interpolation and
saved as jpeg images, with some compression artifacts.
input output input output
Map to aerial photoAerial photo to map
Figure 8: Example results on Google Maps at 512x512 resolution (model was trained on images at 256x256 resolution, and run convolu-
tionally on the larger images at test time). Contrast adjusted for clarity.
Input Ground truth Output Input Ground truth Output
Figure 11: Example results of our method on Cityscapes labels→photo, compared to ground truth.
Input Ground truth Output Input Ground truth Output
Figure 12: Example results of our method on facades labels→photo, compared to ground truth
Input Ground truth Output Input Ground truth Output
Figure 13: Example results of our method on day→night, compared to ground truth.
Input Ground truth Output Input Ground truth Output
Figure 14: Example results of our method on automatically detected edges→handbags, compared to ground truth.
Input Ground truth Output Input Ground truth Output
Figure 15: Example results of our method on automatically detected edges→shoes, compared to ground truth.
Input Output Input Output Input Output Input Output
Figure 16: Example results of the edges→photo models applied to human-drawn sketches from [10]. Note that the models were trained on
automatically detected edges, but generalize to human drawings
Day Night
EdgesShoe Handbag
Labels Facade Street scene
HandbagEdges
Labels
SketchSketch Shoe
Figure 17: Example failure cases. Each pair of images shows input on the left and output on the right. These examples are selected as some
of the worst results on our tasks. Common failures include artifacts in regions where the input image is sparse, and difﬁculty in handling
unusual inputs. Please see https://phillipi.github.io/pix2pix/ for more comprehensive results.
References
[1] A. Buades, B. Coll, and J.-M. Morel. A non-local algo-
rithm for image denoising. InCVPR, volume 2, pages 60–65.
IEEE, 2005. 1
[2] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and
A. L. Yuille. Semantic image segmentation with deep con-
volutional nets and fully connected crfs. In ICLR, 2015. 2
[3] T. Chen, M.-M. Cheng, P. Tan, A. Shamir, and S.-M. Hu.
Sketch2photo: internet image montage. ACM Transactions
on Graphics (TOG), 28(5):124, 2009. 1
[4] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler,
R. Benenson, U. Franke, S. Roth, and B. Schiele. The
cityscapes dataset for semantic urban scene understanding.
In CVPR), 2016. 4, 16
[5] E. L. Denton, S. Chintala, R. Fergus, et al. Deep genera-
tive image models using a laplacian pyramid of adversarial
networks. In NIPS, pages 1486–1494, 2015. 2
[6] A. Dosovitskiy and T. Brox. Generating images with per-
ceptual similarity metrics based on deep networks. arXiv
preprint arXiv:1602.02644, 2016. 2
[7] A. A. Efros and W. T. Freeman. Image quilting for tex-
ture synthesis and transfer. In SIGGRAPH, pages 341–346.
ACM, 2001. 1, 4
[8] A. A. Efros and T. K. Leung. Texture synthesis by non-
parametric sampling. In ICCV, volume 2, pages 1033–1038.
IEEE, 1999. 4
[9] D. Eigen and R. Fergus. Predicting depth, surface normals
and semantic labels with a common multi-scale convolu-
tional architecture. In Proceedings of the IEEE International
Conference on Computer Vision, pages 2650–2658, 2015. 1
[10] M. Eitz, J. Hays, and M. Alexa. How do humans sketch
objects? SIGGRAPH, 31(4):44–1, 2012. 4, 12
[11] R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T.
Freeman. Removing camera shake from a single photograph.
In ACM Transactions on Graphics (TOG), volume 25, pages
787–794. ACM, 2006. 1
[12] L. A. Gatys, A. S. Ecker, and M. Bethge. Texture synthesis
and the controlled generation of natural stimuli using convo-
lutional neural networks. arXiv preprint arXiv:1505.07376,
12, 2015. 4
[13] L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer
using convolutional neural networks. CVPR, 2016. 4
[14] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,
D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio. Gen-
erative adversarial nets. In NIPS, 2014. 2, 4, 6, 7
[15] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H.
Salesin. Image analogies. In SIGGRAPH, pages 327–340.
ACM, 2001. 1, 4
[16] G. E. Hinton and R. R. Salakhutdinov. Reducing the
dimensionality of data with neural networks. Science,
313(5786):504–507, 2006. 3
[17] S. Iizuka, E. Simo-Serra, and H. Ishikawa. Let there be
Color!: Joint End-to-end Learning of Global and Local Im-
age Priors for Automatic Image Colorization with Simulta-
neous Classiﬁcation. ACM Transactions on Graphics (TOG),
35(4), 2016. 2
[18] S. Ioffe and C. Szegedy. Batch normalization: Accelerating
deep network training by reducing internal covariate shift.
2015. 3, 4
[19] J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for
real-time style transfer and super-resolution. 2016. 2, 3
[20] D. Kingma and J. Ba. Adam: A method for stochastic opti-
mization. ICLR, 2015. 4
[21] P.-Y . Laffont, Z. Ren, X. Tao, C. Qian, and J. Hays. Transient
attributes for high-level understanding and editing of outdoor
scenes. ACM Transactions on Graphics (TOG), 33(4):149,
2014. 1, 4, 16
[22] A. B. L. Larsen, S. K. Sønderby, and O. Winther. Autoen-
coding beyond pixels using a learned similarity metric.arXiv
preprint arXiv:1512.09300, 2015. 4
[23] G. Larsson, M. Maire, and G. Shakhnarovich. Learning rep-
resentations for automatic colorization. ECCV, 2016. 2, 8,
16
[24] C. Li and M. Wand. Combining markov random ﬁelds and
convolutional neural networks for image synthesis. CVPR,
2016. 2, 4
[25] C. Li and M. Wand. Precomputed real-time texture synthe-
sis with markovian generative adversarial networks. ECCV,
2016. 2, 4
[26] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional
networks for semantic segmentation. In CVPR, pages 3431–
3440, 2015. 1, 2, 5
[27] M. Mathieu, C. Couprie, and Y . LeCun. Deep multi-scale
video prediction beyond mean square error. ICLR, 2016. 2,
3
[28] M. Mirza and S. Osindero. Conditional generative adversar-
ial nets. arXiv preprint arXiv:1411.1784, 2014. 2
[29] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A.
Efros. Context encoders: Feature learning by inpainting.
CVPR, 2016. 2, 3
[30] A. Radford, L. Metz, and S. Chintala. Unsupervised repre-
sentation learning with deep convolutional generative adver-
sarial networks. arXiv preprint arXiv:1511.06434, 2015. 2,
3, 16
[31] R. ˇS. Radim Tyleˇcek. Spatial pattern templates for recogni-
tion of objects with regular structure. In Proc. GCPR, Saar-
brucken, Germany, 2013. 4, 16
[32] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and
H. Lee. Generative adversarial text to image synthesis.arXiv
preprint arXiv:1605.05396, 2016. 2
[33] E. Reinhard, M. Ashikhmin, B. Gooch, and P. Shirley. Color
transfer between images. IEEE Computer Graphics and Ap-
plications, 21:34–41, 2001. 7
[34] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolu-
tional networks for biomedical image segmentation. InMIC-
CAI, pages 234–241. Springer, 2015. 2, 3, 4
[35] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
et al. Imagenet large scale visual recognition challenge.
IJCV, 115(3):211–252, 2015. 4, 8, 16
[36] T. Salimans, I. Goodfellow, W. Zaremba, V . Cheung, A. Rad-
ford, and X. Chen. Improved techniques for training gans.
arXiv preprint arXiv:1606.03498, 2016. 2, 4, 5
[37] Y . Shih, S. Paris, F. Durand, and W. T. Freeman. Data-driven
hallucination of different times of day from a single outdoor
photo. ACM Transactions on Graphics (TOG), 32(6):200,
2013. 1
[38] D. Ulyanov, A. Vedaldi, and V . Lempitsky. Instance normal-
ization: The missing ingredient for fast stylization. arXiv
preprint arXiv:1607.08022, 2016. 4
[39] X. Wang and A. Gupta. Generative image modeling using
style and structure adversarial networks. ECCV, 2016. 2, 3,
5
[40] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli.
Image quality assessment: from error visibility to struc-
tural similarity. IEEE Transactions on Image Processing,
13(4):600–612, 2004. 2
[41] S. Xie, X. Huang, and Z. Tu. Top-down learning for struc-
tured labeling with convolutional pseudoprior. 2015. 2
[42] S. Xie and Z. Tu. Holistically-nested edge detection. In
ICCV, 2015. 1, 2, 4
[43] D. Yoo, N. Kim, S. Park, A. S. Paek, and I. S. Kweon. Pixel-
level domain transfer. ECCV, 2016. 2, 3
[44] A. Yu and K. Grauman. Fine-Grained Visual Comparisons
with Local Learning. In CVPR, 2014. 4
[45] A. Yu and K. Grauman. Fine-grained visual comparisons
with local learning. In CVPR, pages 192–199, 2014. 16
[46] R. Zhang, P. Isola, and A. A. Efros. Colorful image coloriza-
tion. ECCV, 2016. 1, 2, 5, 7, 8, 16
[47] J. Zhao, M. Mathieu, and Y . LeCun. Energy-based genera-
tive adversarial network. arXiv preprint arXiv:1609.03126,
2016. 2
[48] Y . Zhou and T. L. Berg. Learning temporal transformations
from time-lapse videos. In ECCV, 2016. 2, 3, 7
[49] J.-Y . Zhu, P. Kr ¨ahenb¨uhl, E. Shechtman, and A. A. Efros.
Generative visual manipulation on the natural image mani-
fold. In ECCV, 2016. 2, 4, 16
5. Appendix
5.1. Network architectures
We adapt our network architectures from those
in [30]. Code for the models is available at
https://github.com/phillipi/pix2pix.
Let Ck denote a Convolution-BatchNorm-ReLU layer
with k ﬁlters. CDk denotes a a Convolution-BatchNorm-
Dropout-ReLU layer with a dropout rate of50%. All convo-
lutions are 4 ×4 spatial ﬁlters applied with stride 2. Convo-
lutions in the encoder, and in the discriminator, downsample
by a factor of 2, whereas in the decoder they upsample by a
factor of 2.
5.1.1 Generator architectures
The encoder-decoder architecture consists of:
encoder:
C64-C128-C256-C512-C512-C512-C512-C512
decoder:
CD512-CD512-CD512-C512-C512-C256-C128
-C64
After the last layer in the decoder, a convolution is ap-
plied to map to the number of output channels (3 in general,
except in colorization, where it is 2), followed by a Tanh
function. As an exception to the above notation, Batch-
Norm is not applied to the ﬁrst C64 layer in the encoder.
All ReLUs in the encoder are leaky, with slope 0.2, while
ReLUs in the decoder are not leaky.
The U-Net architecture is identical except with skip con-
nections between each layer iin the encoder and layer n−i
in the decoder, where nis the total number of layers. The
skip connections concatenate activations from layer i to
layer n−i. This changes the number of channels in the
decoder:
U-Net decoder:
CD512-CD1024-CD1024-C1024-C1024-C512
-C256-C128
5.1.2 Discriminator architectures
The 70 ×70 discriminator architecture is:
C64-C128-C256-C512
After the last layer, a convolution is applied to map to a 1
dimensional output, followed by a Sigmoid function. As an
exception to the above notation, BatchNorm is not applied
to the ﬁrst C64 layer. All ReLUs are leaky, with slope 0.2.
All other discriminators follow the same basic architec-
ture, with depth varied to modify the receptive ﬁeld size:
1 ×1 discriminator:
C64-C128 (note, in this special case, all convolutions are
1 ×1 spatial ﬁlters)
16 ×16 discriminator:
C64-C128
256 ×256 discriminator:
C64-C128-C256-C512-C512-C512
Note the the 256 ×256 discriminator has receptive ﬁelds
that could cover up to 574 ×574 pixels, if they were avail-
able, but since the input images are only 256 ×256 pixels,
only 256×256 pixels are seen, and so we refer to this setting
as the 256 ×256 discriminator.
5.2. Training details
Random jitter was applied by resizing the256×256 input
images to 286 ×286 and then randomly cropping back to
size 256 ×256.
All networks were trained from scratch. Weights were
initialized from a Gaussian distribution with mean 0 and
standard deviation 0.02.
Semantic labels→photo 2975 training images from the
Cityscapes training set [4], trained for 200 epochs, batch
size 1, with random jitter and mirroring. We used the
Cityscapes val set for testing.
Architectural labels→photo 400 training images from
[31], trained for 200 epochs, batch size 1, with random jit-
ter and mirroring. Data from was split into train and test
randomly.
Maps↔aerial photograph 1096 training images
scraped from Google Maps, trained for 200 epochs, batch
size 1, with random jitter and mirroring. Images were
sampled from in and around New York City. Data was then
split into train and test about the median latitude of the
sampling region (with a buffer region added to ensure that
no training pixel appeared in the test set).
BW→color 1.2 million training images (Imagenet train-
ing set [35]), trained for ∼6 epochs, batch size 4, with only
mirroring, no random jitter. Tested on subset of Imagenet
val set, following protocol of [46] and [23].
Edges→shoes 50k training images from UT Zappos50K
dataset [45] trained for 15 epochs, batch size 4. Data from
was split into train and test randomly.
Edges→Handbag 137K Amazon Handbag images from
[49], trained for 15 epochs, batch size 4. Data from was split
into train and test randomly.
Day→night 17823 training images extracted from 91
webcams, from [21] trained for 17 epochs, batch size 4,
with random jitter and mirroring. We use 91 webcams as
training, and 10 webcams for test.

AUTO PHASE : J UGGLING HLS P HASE ORDERINGS IN RANDOM FORESTS
WITH DEEP REINFORCEMENT LEARNING
Ameer Haj-Ali * 1 Qijing Huang * 1 William Moses 2 John Xiang 1 Krste Asanovic 1 John Wawrzynek1
Ion Stoica 1
ABSTRACT
The performance of the code a compiler generates depends on the order in which it applies the optimization
passes. Choosing a good order–often referred to as the phase-ordering problem, is an NP-hard problem. As a
result, existing solutions rely on a variety of heuristics. In this paper, we evaluate a new technique to address the
phase-ordering problem: deep reinforcement learning. To this end, we implement AutoPhase3: a framework that
takes a program and uses deep reinforcement learning to ﬁnd a sequence of compilation passes that minimizes its
execution time. Without loss of generality, we construct this framework in the context of the LLVM compiler
toolchain and target high-level synthesis programs. We use random forests to quantify the correlation between
the effectiveness of a given pass and the program’s features. This helps us reduce the search space by avoiding
phase orderings that are unlikely to improve the performance of a given program. We compare the performance of
AutoPhase to state-of-the-art algorithms that address the phase-ordering problem. In our evaluation, we show that
AutoPhase improves circuit performance by 28% when compared to using the -O3 compiler ﬂag, and achieves
competitive results compared to the state-of-the-art solutions, while requiring fewer samples. Furthermore, unlike
existing state-of-the-art solutions, our deep reinforcement learning solution shows promising result in generalizing
to real benchmarks and 12,874 different randomly generated programs, after training on a hundred randomly
generated programs.
1 I NTRODUCTION
High-Level Synthesis (HLS) automates the process of cre-
ating digital hardware circuits from algorithms written in
high-level languages. Modern HLS tools (Xilinx, 2019; In-
tel, 2019; Canis et al., 2013) use the same front-end as the
traditional software compilers. They rely on traditional soft-
ware compiler techniques to optimize the input program’s
intermediate representation (IR) and produce circuits in the
form of RTL code. Thus, the quality of compiler front-end
optimizations directly impacts the performance of HLS-
generated circuit.
Program optimization is a notoriously difﬁcult task. A pro-
gram must be just in ”the right form” for a compiler to
recognize the optimization opportunities. This is a task a
programmer might be able to perform easily, but is often
difﬁcult for a compiler. Despite a decade of research on
developing sophisticated optimization algorithms, there is
still a performance gap between the HLS generated code
*Equal contribution. 1University of California, Berkeley
2Massachusetts Institute of Technology. Correspondence to:
Ameer Haj-Ali <ameerh@berkeley.edu>, Qijing Huang <qi-
jing.huang@berkeley.edu>. 3 https://github.com/ucb-
bar/autophase.
Proceedings of the 3 rd MLSys Conference, Austin, TX, USA,
2020. Copyright 2020 by the author(s).
and the hand-optimized one produced by experts.
In this paper, we build off the LLVM compiler (Lattner &
Adve, 2004). However, our techniques, can be broadly ap-
plicable to any compiler that uses a series of optimization
passes. In this case, the optimization of an HLS program
consists of applying a sequence of analysis and optimiza-
tion phases, where each phase in this sequence consumes
the output of the previous phase, and generates a modiﬁed
version of the program for the next phase. Unfortunately,
these phases are not commutative which makes the order in
which these phases are applied critical to the performance
of the output.
Consider the program in Figure 1, which normalizes a vec-
tor. Without any optimizations, the norm function will take
Θ(n2) to normalize a vector. However, a smart compiler will
implement the loop invariant code motion (LICM) (Much-
nick, 1997) optimization, which allows it to move the call to
mag above the loop, resulting in the code on the left column
in Figure 2. This optimization brings the runtime down to
Θ(n)—a big speedup improvement. Another optimization
the compiler could perform is(function) inlining (Muchnick,
1997). With inlining, a call to a function is simply replaced
with the body of the function, reducing the overhead of the
function call. Applying inlining to the code will result in
the code in the right column of Figure 2.
AutoPhase: Juggling HLS Phase Orderings in Random Forests with Deep Reinforcement Learning
__attribute__((const))
double mag(int n, const double *A) {
double sum = 0;
for(int i=0; i<n; i++){
sum += A[i] * A[i];
}
return sqrt(sum);
}
void norm(int n, double *restrict out,
const double *restrict in) {
for(int i=0; i<n; i++) {
out[i] = in[i] / mag(n, in);
}
}
Figure 1: A simple program to normalize a vector.
Now, consider applying these optimization passes in the
opposite order: ﬁrst inlining then LICM. After inlining,
we get the code on the left of Figure 3. Once again we
get a modest speedup, having eliminated nfunction calls,
though our runtime is stillΘ(n2). If the compiler afterwards
attempted to apply LICM, we would ﬁnd the code on the
right of Figure 3. LICM was able to successfully move the
allocation of sum outside the loop. However, it was unable
to move the instruction setting sum=0 outside the loop, as
doing so would mean that all iterations excluding the ﬁrst
one would end up with a garbage value for sum. Thus, the
internal loop will not be moved out.
As this simple example illustrates, the order in which the
optimization phases are applied can be the difference be-
tween the program running in Θ(n2) versus Θ(n). It is thus
crucial to determine the optimal phase ordering to maximize
the circuit speeds. Unfortunately, not only is this a difﬁcult
task, but the optimal phase ordering may vary from program
to program. Furthermore, it turns out that ﬁnding the opti-
mal sequence of optimization phases is an NP-hard problem,
and exhaustively evaluating all possible sequences is infea-
sible in practice. In this work, for example, the search space
extends to more than 2247 phase orderings.
The goal of this paper is to provide a mechanism for automat-
ically determining good phase orderings for HLS programs
to optimize for the circuit speed. To this end, we aim to
leverage recent advancements in deep reinforcement learn-
ing (RL) (Sutton & Barto, 1998; Haj-Ali et al., 2019b) to
address the phase ordering problem. With RL, a software
agent continuously interacts with the environment by taking
actions. Each action can change the state of the environ-
ment and generate a ”reward”. The goal of RL is to learn
a policy—that is, a mapping between the observed states
of the environment and a set of actions—to maximize the
cumulative reward. An RL algorithm that uses a deep neu-
ral network to approximate the policy is referred to as a
deep RL algorithm. In our case, the observation from the
environment could be the program and/or the optimization
passes applied so far. The action is the optimization pass to
apply next, and the reward is the improvement in the circuit
performance after applying this pass. The particular framing
of the problem as an RL problem has a signiﬁcant impact
on the solution’s effectiveness. Signiﬁcant challenges ex-
ist in understanding how to formulate the phase ordering
optimization problem in an RL framework.
In this paper, we consider three approaches to represent the
environment’s state. The ﬁrst approach is to directly use
salient features from the program. The second approach is
to derive the features from the sequence of optimizations
we applied while ignoring the program’s features. The third
approach combines the ﬁrst two approaches. We evaluate
these approaches by implementing a framework that takes
a group of programs as input and quickly ﬁnds a phase
ordering that competes with state-of-the-art solutions. Our
main contributions are:
• Extend a previous work (Huang et al., 2019) and lever-
age deep RL to address the phase-ordering problem.
• An importance analysis on the features using random
forests to signiﬁcantly reduce the state and action
spaces.
• AutoPhase: a framework that integrates the current
HLS compiler infrastructure with the deep RL algo-
rithms.
• We show that AutoPhase gets a 28% improvement over
-O3 for nine real benchmarks. Unlike all state-of-the-
art approaches, deep RL demonstrates the potential
to generalize to thousands of different programs after
training on a hundred programs.
2 BACKGROUND
2.1 Compiler Phase-ordering
Compilers execute optimization passes to transform pro-
grams into more efﬁcient forms to run on various hardware
targets. Groups of optimizations are often packaged into
“optimization levels” , such as -O0 and -O3, for ease. While
these optimization levels offer a simple set of choices for de-
velopers, they are handpicked by the compiler-designers and
often most beneﬁt certain groups of benchmark programs.
The compiler community has attempted to address the issue
by selecting a particular set of compiler optimizations on a
per-program or per-target basis for software (Triantafyllis
et al., 2003; Almagor et al., 2004; Pan & Eigenmann, 2006;
Ansel et al., 2014).
Since the search space of phase-ordering is too large for
an exhaustive search, many heuristics have been proposed
to explore the space by using machine learning. Huang et
al. tried to address this challenge for HLS applications
AutoPhase: Juggling HLS Phase Orderings in Random Forests with Deep Reinforcement Learning
void norm(int n, double *restrict out,
const double *restrict in) {
double precompute = mag(n, in);
for(int i=0; i<n; i++) {
out[i] = in[i] / precompute;
}
}
void norm(int n, double *restrict out,
const double *restrict in) {
double precompute, sum = 0;
for(int i=0; i<n; i++){
sum += A[i] * A[i];
}
precompute = sqrt(sum);
for(int i=0; i<n; i++) {
out[i] = in[i] / precompute;
}
}
Figure 2: Progressively applying LICM (left) then inlining (right) to the code in Figure 1.
void norm(int n, double *restrict out,
const double *restrict in) {
for(int i=0; i<n; i++) {
double sum = 0;
for(int j=0; j<n; j++){
sum += A[j] * A[j];
}
out[i] = in[i] / sqrt(sum);
}
}
void norm(int n, double *restrict out,
const double *restrict in) {
double sum;
for(int i=0; i<n; i++) {
sum = 0;
for(int j=0; j<n; j++){
sum += A[j] * A[j];
}
out[i] = in[i] / sqrt(sum);
}
}
Figure 3: Progressively applying inlining (left) then LICM (right) to the code in Figure 1.
by using modiﬁed greedy algorithms (Huang et al., 2013;
2015). It achieved 16% improvement vs -O3 on the CHstone
benchmarks (Hara et al., 2008), which we used in this paper.
In (Agakov et al., 2006) both independent and Markov mod-
els were applied to automatically target an optimized search
space for iterative methods to improve the search results.
In (Stephenson et al., 2003), genetic algorithms were used
to tune heuristic priority functions for three compiler opti-
mization passes. Milepost GCC (Fursin et al., 2011) used
machine learning to determine the set of passes to apply to
a given program, based on a static analysis of its features. It
achieved an 11% execution time improvement over -O3, for
the ARC reconﬁgurable processor on the MiBench program
suite1. In (Kulkarni & Cavazos, 2012) the challenge was for-
mulated as a Markov process and supervised learning was
used to predict the next optimization, based on the current
program state. OpenTuner (Ansel et al., 2014) autotunes
a program using an AUC-Bandit-meta-technique-directed
ensemble selection of algorithms. Its current mechanism
for selecting the compiler optimization passes does not con-
sider the order or support repeated optimizations. Wang
et al. (Wang & OBoyle, 2018), provided a survey for us-
ing machine learning in compiler optimization where they
also described that using program features might be helpful.
NeuroVectorizer (Haj-Ali et al., 2020; 2019a) used deep RL
for automatically tuning compiler pragmas such as vector-
ization and interleaving factors. NeuroVectorizer achieves
97% of the oracle performance (brute-force search) on a
wide range of benchmarks.
2.2 Reinforcement Learning Algorithms
Reinforcement learning (RL) is a machine learning approach
in which an agent continually interacts with the environ-
ment (Kaelbling et al., 1996). In particular, the agent ob-
serves the state of the environment, and based on this ob-
servation takes an action. The goal of the RL agent is then
to compute a policy–a mapping between the environment
states and actions–that maximizes a long term reward.
RL can be viewed as a stochastic optimization solution
for solving Markov Decision Processes (MDPs) (Bellman,
1957), when the MDP is not known. An MDP is deﬁned by
a tuple with four elements: S,A,P (s,a),r(s,a) where S
is the set of states of the environment, Adescribes the set of
actions or transitions between states, s′∼P(s,a) describes
the probability distribution of next states given the current
state and action and r(s,a) : S×A→Ris the reward of
taking action ain state s. Given an MDP, the goal of the
agent is to gain the largest possible aggregate reward. The
objective of an RL algorithm associated with an MDP is to
ﬁnd a decision policy π∗(a|s) : s →Athat achieves this
goal for that MDP:
π∗= arg max
π
Eτ∼π(τ)
[∑
t
r(st,at)
]
=
arg max
π
T∑
t=1
E(st,at)∼π(st,at) [r(st,at)] . (1)
Deep RL leverages a neural network to learn the policy (and
AutoPhase: Juggling HLS Phase Orderings in Random Forests with Deep Reinforcement Learning
sometimes the reward function). Policy Gradient (PG) (Sut-
ton et al., 2000), for example, updates the policy directly by
differentiating the aggregate reward E in Equation 1:
∇θJ =
1
N
N∑
i=1
[
(
∑
t
∇θlogπθ(ai,t|si,t))(
∑
t
r(si,t,ai,t))
]
(2)
and updating the network parameters (weights) in the direc-
tion of the gradient:
θ←θ+ α∇θJ, (3)
Note that PG is an on-policy method in that it uses decisions
made directly by the current policy to compute the new
policy.
Over the past couple of years, a plethora of new deep RL
techniques have been proposed (Mnih et al., 2016; Ross
et al., 2011). In this paper, we mainly focus on Proximal
Policy Optimization (PPO) (Schulman et al., 2017), Asyn-
chronous Advantage Actor-critic (A3C) (Mnih et al., 2016).
PPO is a variant of PG that enables multiple epochs of mini-
batch updates to improve the sample complexity. Vanilla
PG performs one gradient update per data sample while
PPO uses a novel surrogate objective function to enable
multiple epochs of minibatch updates. It alternates between
sampling data through interaction with the environment and
optimizing the surrogate objective function using stochastic
gradient ascent. It performs updates that maximizes the
reward function while ensuring the deviation from the previ-
ous policy is small by using a surrogate objective function.
The loss function of PPO is deﬁned as:
LCLIP(θ) = ˆEt[min(rt(θ) ˆAt,clip(rt(θ),1−ε,1+ε) ˆAt)]
(4)
where rt(θ) is deﬁned as a probability ratio πθ(at|st)
πθold(at|st) so
r(θold) = 1. This term penalizes policy update that move
rt(θ) from r(θold). ˆAt denotes the estimated advantage that
approximates how good at is compared to the average. The
second term in the minfunction acts as a disincentive for
moving rt outside of [1 −ε,1 +ε] where εis a hyperparam-
eter.
A3C uses an actor (usually a neural network) that interacts
with the critic, which is another network that evaluates the
action by computing the value function. The actor tells
the actor how good its action was and how it should ad-
just. The update performed by the algorithm can be seen as
∇θlogπθ(ai,t|si,t) ˆAt.
2.3 Evolutionary Algorithms
Evolutionary algorithms are another technique that can be
used to search for the best compiler pass ordering. It con-
tains a family of population-based meta-heuristic optimiza-
tion algorithms inspired by natural selection. The main idea
of these algorithms is to sample a population of solutions
and use the good ones to direct the distribution of future gen-
erations. Two commonly used Evolutionary Algorithms are
Genetic Algorithms (GA) (Goldberg, 2006) and Evolution
Strategies (ES) (Conti et al., 2018).
GA generally requires a genetic representation of the search
space where the solutions are coded as integer vectors. The
algorithm starts with a pool of candidates, then iteratively
evolves the pool to include solutions with higher ﬁtness by
the three following strategies: selection, crossover, and mu-
tation. Selection keeps a subset of solutions with the highest
ﬁtness values. These selected solutions act as parents for
the next generation. Crossover merges pairs from the parent
solutions to produce new offsprings. Mutation perturbs the
offspring solutions with a low probability. The process re-
peats until a solution that reaches the goal ﬁtness is found
or after a certain number of generations.
ES works similarly to GA. However, the solutions are coded
as real numbers in ES. In addition, ES is self-adapting.
The hyperparameters, such as the step size or the mutation
probability, are different for different solutions. They are
encoded in each solution, so good settings get to the next
generation with good solutions. Recent work (Salimans
et al., 2017) has used ES to update policy weights for RL and
showed it is a good alternative for gradient-based methods.
3 A UTO PHASE FRAMEWORK FOR
AUTOMATIC PHASE ORDERING
We leverage an existing open-source HLS framework called
LegUp (Canis et al., 2013) that compiles a C program into a
hardware RTL design. In (Huang et al., 2013), an approach
is devised to quickly determine the number of hardware ex-
ecution cycles without requiring time-consuming logic sim-
ulation. We develop our RL simulator environment based
on the existing harness provided by LegUp and validate our
ﬁnal results by going through the time-consuming logic sim-
ulation. AutoPhase takes a program (or multiple programs)
and intelligently explores the space of possible passes to
ﬁgure out an optimal pass sequence to apply. Table 1 lists all
the passes used in AutoPhase. The workﬂow of AutoPhase
is illustrated in Figure 4.
3.1 HLS Compiler
AutoPhase takes a set of programs as input and compiles
them to a hardware-independent intermediate representa-
tion (IR) using the Clang front-end of the LLVM compiler.
Optimization and analysis passes act as transformations on
the IR, taking a program as input and emitting a new IR as
output. The HLS tool LegUp is invoked after the compiler
AutoPhase: Juggling HLS Phase Orderings in Random Forests with Deep Reinforcement Learning
Feature 
Extractor 
Cycle 
Proﬁler 
Learning 
Agent LLVM   
IR
Optimization 
Passes
HLS 
Compiler Hardware
(State)
(Reward)
Gradient
New Compiler Pass  
 (Action)
Runtime
Features
Program 
Generator 
Real   
Benchmarks
Input Programs
Figure 4: The block diagram of AutoPhase. The input pro-
grams are compiled to an LLVM IR using Clang/LLVM.
The feature extractor and clock-cycle proﬁler are used to
generate the input features (state) and the runtime improve-
ment (reward), respectively from the IR. The input features
and runtime improvement are fed to the deep RL agent as in
input data to train on. The RL agent predicts the next best
optimization passes to apply. After convergence, the HLS
compiler is used to compile the LLVM IR to hardware RTL.
optimization as a back-end pass, which transforms LLVM
IR into hardware modules.
3.2 Clock-cycle Proﬁler
Once the hardware RTL is generated, one could run a hard-
ware simulation to gather the cycle count results of the
synthesized circuit. This process is quite time-consuming,
hindering RL and all other optimization approaches. There-
fore, we approximate cycle count using the proﬁler in
LegUp (Huang et al., 2013), which leverages the software
traces and runs 20×faster than hardware simulation. In
LegUp, the frequency of the generated circuits is set as a
compiler constraint that directs the HLS scheduling algo-
rithm. In other words, HLS tool will always try to generate
hardware that can run at a certain frequency. In our exper-
iment setting, without loss of generality, we set the target
frequency of all generated hardware to 200MHz. We ex-
perimented with lower frequencies too; the improvements
were similar but the cycle counts the different algorithms
achieved were better as more logic could be ﬁtted in a single
cycle.
3.3 IR Feature Extractor
Wang et al. (Wang & OBoyle, 2018) proposed to convert a
program into an observation by extracting all the features
from the program. Similarly, in addition to the LegUp
backend tools, we developed analysis passes to extract 56
static features from the program, such as the number of
basic blocks, branches, and instructions of various types.
We use these features as partially observable states for the
RL learning and hope the neural network can capture the
correlation of certain combinations of these features and
certain optimizations. Table 2 lists all the features used.
3.4 Random Program Generator
As a data-driven approach, RL generalizes better if we train
the agent on more programs. However, there are a lim-
ited number of open-source HLS examples online. There-
fore, we expand our training set by automatically generating
synthetic HLS benchmarks. We ﬁrst generate standard C
programs using CSmith (Yang et al., 2011), a random C pro-
gram generator, which is originally designed to generate test
cases for ﬁnding compiler bugs. Then, we develop scripts to
ﬁlter out programs that take more than ﬁve minutes to run
on CPU or fail the HLS compilation.
3.5 Overall Flow of AutoPhase
We integrate the compilation utilities into a simulation
environment in Python with APIs similar to an OpenAI
gym (Brockman et al., 2016). The overall ﬂow works as
follows:
1. The input program is compiled into LLVM IR using
the Clang/LLVM.
2. The IR Feature Extractor is run to extract salient pro-
gram features.
3. LegUp compiles the LLVM IR into hardware RTL.
4. The Clock-cycle Proﬁler estimates a clock-cycle count
for the generated circuit.
5. The RL agent takes the program features or the his-
togram of previously applied passes and the improve-
ment in clock-cycle count as input data to train on.
6. The RL agent predicts the next best optimization passes
to apply.
7. New LLVM IR is generated after the new optimization
sequence is applied.
8. The machine learning algorithm iterates through steps
(2)–(7) until convergence.
Note that AutoPhase uses the LLVM compiler and the
passes used are listed in Table 2. However, adding sup-
port for any compiler or optimization passes in AutoPhase
is very easy and straightforward. The action and state deﬁ-
nitions must be speciﬁed again.
4 C ORRELATION OF PASSES AND
PROGRAM FEATURES
Similar to the case with many deep learning approaches,
explainability is one of the major challenges we face when
applying deep RL to the phase-ordering challenge. To ana-
lyze and understand the correlation of passes and program
AutoPhase: Juggling HLS Phase Orderings in Random Forests with Deep Reinforcement Learning
Table 1: LLVM Transform Passes.
0 1 2 3 4 5 6 7 8 9 10
-correlated-propagation -scalarrepl -lowerinvoke -strip -strip-nondebug -sccp -globalopt -gvn -jump-threading -globaldce -loop-unswitch
11 12 13 14 15 16 17 18 19 20 21
-scalarrepl-ssa -loop-reduce -break-crit-edges -loop-deletion -reassociate -lcssa -codegenprepare -memcpyopt -functionattrs -loop-idiom -lowerswitch
22 23 24 25 26 27 28 29 30 31 32 33
-constmerge -loop-rotate -partial-inliner -inline -early-cse -indvars -adce -loop-simplify -instcombine -simplifycfg -dse -loop-unroll
34 35 36 37 38 39 40 41 42 43 44 45
-lower-expect -tailcallelim -licm -sink -mem2reg -prune-eh -functionattrs -ipsccp -deadargelim -sroa -loweratomic -terminate
Table 2: Program Features.
0 Number of BB where total args for phi nodes >5 28 Number of And insts
1 Number of BB where total args for phi nodes is [1,5] 29 Number of BB’s with instructions between [15,500]
2 Number of BB’s with 1 predecessor 30 Number of BB’s with less than 15 instructions
3 Number of BB’s with 1 predecessor and 1 successor 31 Number of BitCast insts
4 Number of BB’s with 1 predecessor and 2 successors 32 Number of Br insts
5 Number of BB’s with 1 successor 33 Number of Call insts
6 Number of BB’s with 2 predecessors 34 Number of GetElementPtr insts
7 Number of BB’s with 2 predecessors and 1 successor 35 Number of ICmp insts
8 Number of BB’s with 2 predecessors and successors 36 Number of LShr insts
9 Number of BB’s with 2 successors 37 Number of Load insts
10 Number of BB’s with>2 predecessors 38 Number of Mul insts
11 Number of BB’s with Phi node # in range (0,3] 39 Number of Or insts
12 Number of BB’s with more than 3 Phi nodes 40 Number of PHI insts
13 Number of BB’s with no Phi nodes 41 Number of Ret insts
14 Number of Phi-nodes at beginning of BB 42 Number of SExt insts
15 Number of branches 43 Number of Select insts
16 Number of calls that return an int 44 Number of Shl insts
17 Number of critical edges 45 Number of Store insts
18 Number of edges 46 Number of Sub insts
19 Number of occurrences of 32-bit integer constants 47 Number of Trunc insts
20 Number of occurrences of 64-bit integer constants 48 Number of Xor insts
21 Number of occurrences of constant 0 49 Number of ZExt insts
22 Number of occurrences of constant 1 50 Number of basic blocks
23 Number of unconditional branches 51 Number of instructions (of all types)
24 Number of Binary operations with a constant operand 52 Number of memory instructions
25 Number of AShr insts 53 Number of non-external functions
26 Number of Add insts 54 Total arguments to Phi nodes
27 Number of Alloca insts 55 Number of Unary operations
features, we use random forests (Breiman, 2001) to learn
the importance of different features. Random forest is an
ensemble of multiple decision trees. The prediction made by
each tree could be explained by tracing the decisions made
at each node and calculating the importance of different
features on making the decisions at each node. This helps
us to identify the effective features and passes to use and
show whether our algorithms learn informative patterns on
data.
For each pass, we build two random forests to predict
whether applying it would improve the circuit performance.
The ﬁrst forest takes the program features as inputs while the
second takes a histogram of previously applied passes. To
gather the training data for the forests, we run PPO with high
exploration parameter on 100 randomly generated programs
to generate feature–action–reward tuples. The algorithm
assigns higher importance to the input features that affect
the ﬁnal prediction more.
4.1 Importance of Program Features
The heat map in Figure 5 shows the importance of different
features on whether a pass should be applied. The higher
the value is, the more important the feature is (the sum of
the values in each row is one). The random forest is trained
with 150,000 samples generated from the random programs.
The index mapping of features and passes can be found in
Tables 1 and 2. For example, the yellow pixel corresponding
to feature index 17 and pass index 23 reﬂects that number-
of-critical-edges affects the decision on whether to apply
-loop-rotate greatly. A critical edge in control ﬂow graph is
an edge that is neither the only edge leaving its source block,
nor the only edge entering its destination block. The critical
edges can be commonly seen in a loop as a back edge so
the number of critical edges might roughly represent the
number of loops in a program. The transform pass -loop-
rotate detects a loop and transforms a while loop to a do-
while loop to eliminate one branch instruction in the loop
body. Applying the pass results in better circuit performance
as it reduces the total number of FSM states in a loop.
Other expected behaviors are also observed in this ﬁgure.
AutoPhase: Juggling HLS Phase Orderings in Random Forests with Deep Reinforcement Learning
Figure 5: Heat map illustrating the importance of feature
and pass indices.
For instance, the correlation between number of branches
and the transform passes -loop-simplify, -tailcallelism
(which transforms calls of the current function i.e., self re-
cursion, followed by a return instruction with a branch to the
entry of the function, creating a loop), -lowerswitch (which
rewrites switch instructions with a sequence of branches).
Other interesting behaviors are also captured. For example,
in the correlation between binary operations with a constant
operand and -functionattrs, which marks different operands
of a function as read-only (constant). Some correlations
are harder to explain, for example, number of BitCast in-
structions and -instcombine, which combines instructions
into fewer simpler instructions. This is actually a result of
-instcombine reducing the loads and stores that call bitcast
instructions for casting pointer types. Another example is
number of memory instructions and -sink, where -sink basi-
cally moves memory instructions into successor blocks and
delays the execution of memory until needed. Intuitively,
whether to apply -sink should be dependent on whether
there is any memory instruction in the program. Our last
example to show is number of occurrences of constant 0
and -deadargelim, where -deadargelim helped eliminate
dead/unused constant zero arguments.
Overall, we observe that all the passes are correlated to some
features and are able to affect the ﬁnal circuit performance.
We also observe that multiple features are not effective at
directing decisions and training with them could increase
the variance that would result in lower prediction accuracy
of our results. For example, the total number of instructions
did not give a direct indication of whether applying a pass
would be helpful or not. This is because sometimes more in-
structions could improve the performance (for example, due
to loop unrolling) and eliminating unnecessary code could
also improve the performance. In addition, the importance
of features varies among different benchmarks depending
on the tasks they perform.
Figure 6: Heat map illustrating the importance of indices of
previously applied passes and the new pass to apply.
4.2 Importance of Previously Applied Passes
Figure 6 illustrates the impact of previously applied passes
on the new pass to apply. The higher the value is, the more
important having the old pass is. From this ﬁgure, we learn
that for the programs we trained on passes -scalarrepl, -gvn,
-scalarrepl-ssa, -loop-reduce, -loop-deletion, -reassociate,
-loop-rotate, -partial-inliner, -early-cse, -adce, -instcombine,
-simplifycfg, -dse, -loop-unroll, -mem2reg, and -sroa , are
more impactful on the performance compared to the rest of
the passes regardless of their order in the trajectory. Point
(23,23) has the highest importance in which implies that pass
-loop-rotate is very helpful and should be included if not
applied before. By examining thousands of the programs,
we ﬁnd that -loop-rotate indeed reduces the cycle count
signiﬁcantly. Interestingly, applying this pass twice is not
harmful if the passes were given consecutively. However,
giving this pass twice with some other passes between them
is sometimes very harmful. Another interesting behavior
our heat map captured is the fact that applying pass 33
(-loop-unroll) after (not necessarily consecutive) pass 23
(-loop-rotate) was much more useful compared to applying
these two passes in the opposite order.
5 P ROBLEM FORMULATION
5.1 The RL Environment Deﬁnition
Assume the optimal number of passes to apply is N and
there are K transform passes to select from in total, our
search space Sfor the phase-ordering problem is [0,KN).
Given M program features and the history of already ap-
plied passes, the goal of deep RL is to learn the next best
optimization pass ato apply that minimizes the long term
cycle count of the generated hardware circuit. Note that the
optimization state sis partially observable in this case as the
M program features cannot fully capture all the properties
of a program.
AutoPhase: Juggling HLS Phase Orderings in Random Forests with Deep Reinforcement Learning
Action Space – we deﬁne our action space Aas {a∈Z :
a ∈ [0,K)}where K is the total number of transform
passes.
Observation Space – two types of input features were con-
sidered in our evaluation: 1⃝program features of ∈ZM
listed in Table 2 and 2⃝action history which is a histogram
of previously applied passes oa ∈ZK. After each RL step
where the pass iis applied, we call the feature extractor in
our environment to return new of , and update the action
histogram element oai to oai + 1.
Reward – the cycle count of the generated circuit is re-
ported by the clock-cycle proﬁler at each RL iteration. Our
reward is deﬁned as R = cprev −ccur, where cprev and
ccur represent the previous and the current cycle count of
the generated circuit respectively. It is possible to deﬁne a
different reward for different objectives. For example, the
reward could be deﬁned as the negative of the area and thus
the RL agent will optimize for the area. It is also possi-
ble to co-optimize multiple objectives (e.g., area, execution
time, power, etc.) by deﬁning a combination of different
objectives.
5.2 Applying Multiple Passes per Action
An alternative to the action formulation above is to evaluate
a complete sequence of passes with length N instead of
a single action a at each RL iteration. Upon the start of
training a new episode, the RL agent resets all pass indices
p ∈ZN to the index value K
2 . For pass pi at index i, the
next action to take is either to change to a new pass or
not. By allowing positive and negative index update for
each p, we reduced the total steps required to traverse all
possible pass indices. The sub-action space ai for each
pass is thus deﬁned as [−1,0,1]. The total action space
Ais deﬁned as [−1,0,1]N. At each step, the RL agent
predicts the updates [a1,a2,...,a N] to N passes, and the
current optimization sequence [p1,p2,...,p N] is updated to
[p1 + a1,p2 + a2,...,p N + aN].
5.3 Normalization Techniques
In order for the trained RL agent to work on new programs,
we need to properly normalize the program features and
rewards so they represent a meaningful state among dif-
ferent programs. In this work, we experiment with two
techniques: 1⃝taking the logarithm of program features
or rewards and, 2⃝normalizing to a parameter from the
original input program that roughly depicts the problem
size. For technique 1⃝, note that taking the logarithm of the
program features not only reduces their magnitude, it also
correlates them in a different manner in the neural network.
Since, w1 log(of1 ) + w2 log(of2 ) = log(ow1
f1
ow2
f2
), the neu-
ral network is learning to correlate the products of features
instead of a linear combination of them. For technique 2⃝,
we normalize the program features to the total number of
instructions in the input program (of norm = of
of51
), which
is feature #51 in Table 2.
6 E VALUATION
To run our deep RL algorithms we use RLlib (Liang et al.,
2017), an open-source library for reinforcement learning
that offers both high scalability and a uniﬁed API for a va-
riety of applications. RLlib is built on top of Ray (Moritz
et al., 2018), a high-performance distributed execution
framework targeted at large-scale machine learning and rein-
forcement learning applications. We ran the framework on
a four-core Intel i7-4765T CPUwith a Tesla K20c GPUfor
training and inference.
We set our frequency constraint in HLS to 200MHz and use
the number of clock cycles reported by the HLS proﬁler
as the circuit performance metric. In (Huang et al., 2013),
results showed a one-to-one correspondence between the
clock cycle count and the actual hardware execution time
under certain frequency constraint. Therefore, better clock
cycle count will lead to better hardware performance.
6.1 Performance
To evaluate the effectiveness of various algorithms for tack-
ling the phase-ordering problem, we run them on nine
real HLS benchmarks and compare the results based on
the ﬁnal HLS circuit performance and the sample efﬁ-
ciency against state-of-the-art approaches for overcoming
the phase ordering, which include random search, Greedy
Algorithms (Huang et al., 2013), OpenTuner (Ansel et al.,
2014), and Genetic Algorithms (Fortin et al., 2012). These
benchmarks are adapted from CHStone (Hara et al., 2008)
and LegUp examples. They are: adpcm, aes, blowﬁsh,
dhrystone, gsm, matmul, mpeg2, qsort, and sha. For this
evaluation, the input features/rewards were not normalized,
the pass length was set to 45, and each algorithm was run on
a per-program basis. Table 3 lists the action and observation
spaces used in all the deep RL algorithms.
The bar chart in Figure 7 shows the percentage improvement
of the circuit performance compared to -O3 results on the
nine real benchmarks from CHStone. The dots on the blue
line in Figure 7 show the total number of samples for each
program, which is the number of times the algorithm calls
the simulator to gather the cycle count. -O0 and -O3 are
the default compiler optimization levels. RL-PPO1 is a
PPO explorer where we set all the rewards to 0 to test if
the rewards are meaningful. RL-PPO2 is the PPO agent
that learns the next pass based on a histogram of applied
passes. RL-A3C is the A3C agent that learns based on
the program features. Greedy performs the greedy algo-
rithm, which always inserts the pass that achieves the highest
AutoPhase: Juggling HLS Phase Orderings in Random Forests with Deep Reinforcement Learning
Table 3: The observation and action spaces used in the different deep RL algorithms.
RL-PPO1 RL-PPO2 RL-PPO3 RL-A3C RL-ES
Deep RL Algorithm PPO PPO PPO A3C ES
Observation Space Program Features Action History Action History + Program Features Program Features Program Features
Action Space Single-Action Single-Action Multiple-Action Single-Action Single-Action
 -O0  -O3
RL-PPO1RL-PPO2RL-A3CGreedyRL-PPO3OpenTuner
RL-ES
Genetic-DEAP
Random 
0.2
0.1
0.0
0.1
0.2
0.3
Improvement over -O3
-0.230.0
0.09
0.24 0.25
0.03
0.28 0.28 0.26 0.27
0.07
0
2000
4000
6000
8000
10000
Samples / Program
1 1 88 88
2484
3510400043846080
6789
8400
Figure 7: Circuit Speedup and Sample Size Comparison.
speedup at the best position (out of all possible positions it
can be inserted to) in the current sequence. RL-PPO3 uses
a PPO agent and the program features but with the action
space described in Section 5.2. explained in Section 5.2.
OpenTuner runs an ensemble of six algorithms, which
includes two families of algorithms: particle swarm opti-
mization (Kennedy, 2010) and GA, each with three different
crossover settings. RL-ES is similar to A3C agent that
learns based on the program features, but updates the policy
network using the evolution strategy instead of backpropa-
gation. Genetic-DEAP (Fortin et al., 2012) is a genetic
algorithm implementation. random randomly generates
a sequence of 45 passes at once instead of sampling them
one-by-one.
From Greedy, we see that always adding the pass in the
current sequence that achieves the highest reward leads to
sub-optimal circuit performance. RL-PPO2 achieves higher
performance than RL-PPO1, which shows that the deep RL
captures useful information during training. Using the his-
togram of applied passes results in better sample efﬁciency,
but using the program features with more samples results
in a slightly higher speedup. RL-PPO2, for example, at the
minor cost of 4% lower speedup, achieves 50×more sample
efﬁciency than OpenTuner. Using ES to update the policy
is supposed to be more sample efﬁcient for problems with
sparse rewards like ours, however, our experiments did not
beneﬁt from that. Furthermore, RL-PPO3 with multiple
action updates achieves a higher speedup than the other
deep RL algorithms with a single action. One reason for
that is the ability of RL-PPO3 to explore more passes per
compilation as it applies multiple passes simultaneously in
between every compilation. On the other hand, the other
deep RL algorithms apply a single pass at a time.
6.2 Generalization
With deep RL, the search should beneﬁt from prior knowl-
edge learned from other different programs. This knowledge
should be transferable from one program to another. For ex-
ample, as discussed in section 4 applying pass -loop-rotate
is always beneﬁcial, and -loop-unroll should be applied af-
ter -loop-rotate. Note that the black-box search algorithms,
such as OpenTuner, GA, and greedy algorithms, cannot gen-
eralize. For these algorithms, rerunning a new search with
many compilations is necessary for every new program, as
they do not learn any patterns from the programs to direct
the search and can be viewed as a smart random search.
To evaluate how generalizable deep RL could be with dif-
ferent programs and whether any prior knowledge could
be useful, we train on 100 randomly-generated programs
using PPO. Random programs are used for transfer learning
due to lack of sufﬁcient benchmarks and because it is the
worst-case scenario, i.e., they are very different from the
programs that we use for inference. The improvement can
be higher if we train on programs that are similar to the
ones we inference on. We train a network with 256 ×256
fully connected layers and use the histogram of previously
applied passes concatenated to the program features as the
observation and passes as actions.
As described in Section 5.3, we experiment with two nor-
malization techniques for the program features: 1⃝taking
the logarithm of all the program features and 2⃝normaliz-
ing the program features to the total number of instruc-
tions in the program. In each pass sequence, the inter-
mediate reward was deﬁned as the logarithm of the im-
provement in cycle count after applying each pass. The
logarithm was chosen so that the RL agent will not give
much larger weights to big rewards from programs with
longer execution time. Three approaches were evaluated:
filtered-norm1 uses the ﬁltered (based on the analysis
in Section 4 where we only keep the important features and
passes) program features and passes from Section with nor-
malization technique 1⃝, original-norm2 uses all the
program features and passes with normalization technique
2⃝, and filtered-norm2 uses the ﬁltered program fea-
tures and passes from Section 4 with normalization tech-
nique 2⃝. Filtering the features and passes might not be
ideal, especially when different programs have different
feature characteristics and impactful passes. However, re-
ducing the number of features and passes helps to reduce
variance among all programs and signiﬁcantly narrow the
AutoPhase: Juggling HLS Phase Orderings in Random Forests with Deep Reinforcement Learning
Figure 8: Episode reward mean as a function of step for
the original approach where we use all the program fea-
tures and passes and for the ﬁltered approach where we
ﬁlter the passes and features (with different normalization
techniques). Higher values indicate faster circuit speed.
search space.
Figure 8 shows the episode reward mean as a function
of the step for the three approaches. We observe that
filtered-norm2 and filtered-norm1 converge
much faster and achieve a higher episode reward mean
than original-norm2, which uses all the features and
passes. At roughly 8,000 steps thefiltered-norm2 and
filter-norm1 already achieve a very high episode re-
ward mean, with minor improvements in later steps. Further-
more, the episode reward mean of the ﬁltered approaches
is still higher than that of original-norm2 even when
we allowed it to train for 20 times more steps (i.e., 160,000
steps). This indicates that ﬁltering the features and passes
signiﬁcantly improved the learning process. All three ap-
proaches learned to always apply pass -loop-rotate, and
-loop-unroll after -loop-rotate. Another useful pass that the
three approaches learned to apply is -loop-simplify, which
performs several transformations to transform natural loops
into a simpler form that enables subsequent analyses and
transformations.
We now compare the generalization results of
filtered-norm2 and filtered-norm1 with
the other black-box algorithms. We use 100 randomly-
generated programs as the training set and nine real
benchmarks from CHStone as the testing set for the deep
RL-based methods. With the state-of-the-art black-box
algorithms, we ﬁrst search for the best pass sequences that
achieved the lowest aggregated hardware cycle counts for
the 100 random programs and then directly apply them to
the nine test set programs. In Figure 9, the bar chart shows
the percentage improvement of the circuit performance
compared to -O3 on the nine real benchmarks, the dots
on the blue line show the total number of samples each
inference takes for one new program.
 -O0  -O3
Genetic-DEAPOpenTuner
Greedy
RL-filtered-norm1RL-filtered-norm2
0.2
0.1
0.0
Improvement over -O3
-0.23 0.0 -0.24 -0.02
0.02 0.03 0.04
0
10
20
30
Samples / Program
1 1 1 1 1 1 1
Figure 9: Circuit Speedup and Sample Size Comparison for
deep RL Generalization.
This evaluation shows that the deep RL-based inference
achieves higher speedup than the predetermined sequences
produced by the state-of-the-art black-box algorithms for
new programs. The predetermined sequences that are over-
ﬁtted to the random programs can cause poor performance
in unseen programs (e.g., -24% for Genetic-DEAP). Be-
sides, normalization technique 2⃝works better compared
to normalization technique 1⃝for deep RL generalization
(4% vs 3% speedup). This indicates that normalizing the
different instructions to the total number of instructions i.e.,
the distribution of the different instructions in Technique 2⃝
represents more universal characteristics across different
programs, while taking the log in Technique 1⃝only sup-
presses the value ranges of different program features. Fur-
thermore, when we use other 12,874 randomly generated
programs as the testing set with filtered-norm2, the
speedup is 6% compared to -O3.
7 C ONCLUSIONS
In this paper, we propose an approach based on deep RL
to improve the performance of HLS designs by optimizing
the order in which the compiler applies optimization phases.
We use random forests to analyze the relationship between
program features and optimization passes. We then leverage
this relationship to reduce the search space by identifying
the most likely optimization phases to improve the perfor-
mance, given the program features. Our RL based approach
achieves 28% better performance than compiling with the
-O3 ﬂag after training for a few minutes, and a 24% improve-
ment after training for less than a minute. Furthermore, we
show that unlike prior work, our solution shows potential to
generalize to a variety of programs. While in this paper we
have applied deep RL to HLS, we believe that the same ap-
proach can be successfully applied to software compilation
and optimization. Going forward, we envision using deep
RL techniques to optimize a wide range of programs and
systems.
AutoPhase: Juggling HLS Phase Orderings in Random Forests with Deep Reinforcement Learning
ACKNOWLEDGEMENT
This research is supported in part by NSF CISE Expedi-
tions Award CCF-1730628, the Defense Advanced Research
Projects Agency (DARPA) through the Circuit Realiza-
tion at Faster Timescales (CRAFT) Program under Grant
HR0011-16-C0052, the Computing On Network Infras-
tructure for Pervasive Perception, Cognition and Action
(CONIX) Research Center, NSF Grant 1533644, LANL
Grant 531711, and DOE Grant DE-SC0019323, and gifts
from Alibaba, Amazon Web Services, Ant Financial, Cap-
italOne, Ericsson, Facebook, Futurewei, Google, IBM, In-
tel, Microsoft, Nvidia, Scotiabank, Splunk, VMware, and
ADEPT Lab industrial sponsors and afﬁliates. The views
and opinions of authors expressed herein do not necessarily
state or reﬂect those of the United States Government or any
agency thereof.
REFERENCES
Agakov, F., Bonilla, E., Cavazos, J., Franke, B., Fursin,
G., O’Boyle, M. F., Thomson, J., Toussaint, M., and
Williams, C. K. Using machine learning to focus iterative
optimization. In Proceedings of the International Sympo-
sium on Code Generation and Optimization, pp. 295–305.
IEEE Computer Society, 2006.
Almagor, L., Cooper, K. D., Grosul, A., Harvey, T. J.,
Reeves, S. W., Subramanian, D., Torczon, L., and Water-
man, T. Finding effective compilation sequences. ACM
SIGPLAN Notices, 39(7):231–239, 2004.
Ansel, J., Kamil, S., Veeramachaneni, K., Ragan-Kelley,
J., Bosboom, J., O’Reilly, U.-M., and Amarasinghe, S.
Opentuner: An extensible framework for program auto-
tuning. In Proceedings of the 23rd international con-
ference on Parallel architectures and compilation , pp.
303–316. ACM, 2014.
Bellman, R. A markovian decision process. In Journal of
Mathematics and Mechanics, pp. 679–684, 1957.
Breiman, L. Random forests. Machine learning, 45(1):
5–32, 2001.
Brockman, G., Cheung, V ., Pettersson, L., Schneider, J.,
Schulman, J., Tang, J., and Zaremba, W. Openai gym,
2016.
Canis, A., Choi, J., Aldham, M., Zhang, V ., Kammoona, A.,
Czajkowski, T., Brown, S. D., and Anderson, J. H. Legup:
An open-source high-level synthesis tool for fpga-based
processor/accelerator systems. ACM Transactions on
Embedded Computing Systems (TECS), 13(2):24, 2013.
Conti, E., Madhavan, V ., Such, F. P., Lehman, J., Stanley,
K., and Clune, J. Improving exploration in evolution
strategies for deep reinforcement learning via a popula-
tion of novelty-seeking agents. In Advances in Neural
Information Processing Systems, pp. 5032–5043, 2018.
Fortin, F.-A., De Rainville, F.-M., Gardner, M.-A., Parizeau,
M., and Gagn´e, C. DEAP: Evolutionary algorithms made
easy. Journal of Machine Learning Research, 13:2171–
2175, jul 2012.
Fursin, G., Kashnikov, Y ., Memon, A. W., Chamski, Z.,
Temam, O., Namolaru, M., Yom-Tov, E., Mendelson, B.,
Zaks, A., Courtois, E., et al. Milepost gcc: Machine learn-
ing enabled self-tuning compiler. International journal
of parallel programming, 39(3):296–327, 2011.
Goldberg, D. E. Genetic algorithms. Pearson Education
India, 2006.
Haj-Ali, A., Ahmed, N. K., Willke, T., Shao, S., Asanovic,
K., and Stoica, I. Learning to vectorize using deep rein-
forcement learning. In Workshop on ML for Systems at
NeurIPS, December 2019a.
Haj-Ali, A., K. Ahmed, N., Willke, T., Gonzalez, J.,
Asanovic, K., and Stoica, I. A view on deep reinforce-
ment learning in system optimization. arXiv preprint
arXiv:1908.01275, 2019b.
Haj-Ali, A., Ahmed, N. K., Willke, T., Shao, S., Asanovic,
K., and Stoica, I. Neurovectorizer: End-to-end vectoriza-
tion with deep reinforcement learning. International Sym-
posium on Code Generation and Optimization (CGO) ,
February 2020.
Hara, Y ., Tomiyama, H., Honda, S., Takada, H., and Ishii,
K. CHstone: A benchmark program suite for practical
c-based high-level synthesis. In Circuits and Systems,
2008. ISCAS 2008. IEEE International Symposium on ,
pp. 1192–1195, 2008.
Huang, Q., Lian, R., Canis, A., Choi, J., Xi, R., Brown, S.,
and Anderson, J. The effect of compiler optimizations on
high-level synthesis for fpgas. In Field-Programmable
Custom Computing Machines (FCCM), 2013 IEEE 21st
Annual International Symposium on , pp. 89–96. IEEE,
2013.
Huang, Q., Lian, R., Canis, A., Choi, J., Xi, R., Calagar,
N., Brown, S., and Anderson, J. The effect of compiler
optimizations on high-level synthesis-generated hardware.
ACM Transactions on Reconﬁgurable Technology and
Systems (TRETS), 8(3):14, 2015.
Huang, Q., Haj-Ali, A., Moses, W., Xiang, J., Stoica, I.,
Asanovic, K., and Wawrzynek, J. Autophase: Compiler
phase-ordering for hls with deep reinforcement learn-
ing. In 2019 IEEE 27th Annual International Symposium
on Field-Programmable Custom Computing Machines
(FCCM), pp. 308–308. IEEE, 2019.
AutoPhase: Juggling HLS Phase Orderings in Random Forests with Deep Reinforcement Learning
Intel. Intel High-Level Synthesis Compiler, 2019.
URL https://www.intel.com/content/www/
us/en/software/programmable/quartus-
prime/hls-compiler.html.
Kaelbling, L. P., Littman, M. L., and Moore, A. W. Rein-
forcement learning: A survey. In Reinforcement learning:
A survey, volume 4, pp. 237–285, 1996.
Kennedy, J. Particle swarm optimization. Encyclopedia of
machine learning, pp. 760–766, 2010.
Kulkarni, S. and Cavazos, J. Mitigating the compiler opti-
mization phase-ordering problem using machine learning.
In Proceedings of the ACM International Conference on
Object Oriented Programming Systems Languages and
Applications, OOPSLA ’12, 2012.
Lattner, C. and Adve, V . Llvm: A compilation framework
for lifelong program analysis & transformation. In Inter-
national Symposium on Code Generation and Optimiza-
tion, 2004. CGO 2004., pp. 75–86. IEEE, 2004.
Liang, E., Liaw, R., Moritz, P., Nishihara, R., Fox, R., Gold-
berg, K., Gonzalez, J. E., Jordan, M. I., and Stoica, I.
Rllib: Abstractions for distributed reinforcement learning.
arXiv preprint arXiv:1712.09381, 2017.
Mnih, V ., Badia, A. P., Mirza, M., Graves, A., Lillicrap,
T., Harley, T., Silver, D., and Kavukcuoglu, K. Asyn-
chronous methods for deep reinforcement learning. In
International conference on machine learning, pp. 1928–
1937, 2016.
Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw,
R., Liang, E., Elibol, M., Yang, Z., Paul, W., Jordan,
M. I., et al. Ray: A distributed framework for emerging
{AI}applications. In 13th {USENIX}Symposium on
Operating Systems Design and Implementation ({OSDI}
18), pp. 561–577, 2018.
Muchnick, S. S. Advanced compiler design and implementa-
tion. In Advanced Compiler Design and Implementation.
Morgan Kaufmann, 1997.
Pan, Z. and Eigenmann, R. Fast and effective orchestration
of compiler optimizations for automatic performance tun-
ing. In Proceedings of the International Symposium on
Code Generation and Optimization, pp. 319–332. IEEE
Computer Society, 2006.
Ross, S., Gordon, G., and Bagnell, D. A reduction of imita-
tion learning and structured prediction to no-regret online
learning. In Proceedings of the fourteenth international
conference on artiﬁcial intelligence and statistics , pp.
627–635, 2011.
Salimans, T., Ho, J., Chen, X., Sidor, S., and Sutskever,
I. Evolution strategies as a scalable alternative to rein-
forcement learning. arXiv preprint arXiv:1703.03864,
2017.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and
Klimov, O. Proximal policy optimization algorithms.
arXiv preprint arXiv:1707.06347, 2017.
Stephenson, M., Amarasinghe, S., Martin, M., and O’Reilly,
U.-M. Meta optimization: Improving compiler heuris-
tics with machine learning. In Proceedings of the ACM
SIGPLAN 2003 Conference on Programming Language
Design and Implementation, PLDI ’03, 2003.
Sutton, R. S. and Barto, A. G. Introduction to reinforcement
learning, volume 135. MIT press Cambridge, 1998.
Sutton, R. S., McAllester, D. A., Singh, S. P., and Mansour,
Y . Policy gradient methods for reinforcement learning
with function approximation. In Advances in neural in-
formation processing systems, pp. 1057–1063, 2000.
Triantafyllis, S., Vachharajani, M., Vachharajani, N., and
August, D. I. Compiler optimization-space exploration.
In Proceedings of the international symposium on Code
generation and optimization: feedback-directed and run-
time optimization, pp. 204–215. IEEE Computer Society,
2003.
Wang, Z. and OBoyle, M. Machine learning in compiler
optimization. In Machine Learning in Compiler Opti-
mization, volume 106, pp. 1879–1901, Nov 2018.
Xilinx. Vivado High-Level Synthesis, 2019. URL
https://www.xilinx.com/products/
design-tools/vivado/integration/esl-
design.html.
Yang, X., Chen, Y ., Eide, E., and Regehr, J. Finding and
understanding bugs in c compilers. In ACM SIGPLAN
Notices, volume 46, pp. 283–294. ACM, 2011.

Mitigating the Compiler Optimization
Phase-Ordering Problem using Machine Learning
Sameer Kulkarni John Cavazos
University of Delaware
{skulkarn,cavazos}@cis.udel.edu
Abstract
Today’s compilers have a plethora of optimizations to choose
from, and the correct choice of optimizations can have a sig-
niﬁcant impact on the performance of the code being opti-
mized. Furthermore, choosing the correct order in which to
apply those optimizations has been a long standing problem
in compilation research. Each of these optimizations inter-
acts with the code and in turn with all other optimizations in
complicated ways. Traditional compilers typically apply the
same set of optimization in a ﬁxed order to all functions in a
program, without regard the code being optimized.
Understanding the interactions of optimizations is very
important in determining a good solution to the phase-
ordering problem. This paper develops a new approach that
automatically selects good optimization orderings on a per
method basis within a dynamic compiler. Our approach for-
mulates the phase-ordering problem as a Markov process
and uses a characterization of the current state of the code
being optimized to creating a better solution to the phase
ordering problem. Our technique uses neuro-evolution to
construct an artiﬁcial neural network that is capable of pre-
dicting beneﬁcial optimization ordering for a piece of code
that is being optimized. We implemented our technique in
Jikes RVM and achieved signiﬁcant improvements on a set
of standard Java benchmarks over a well-engineered ﬁxed
order.
General Terms Performance, Compilation, Compiler Op-
timization
Keywords Phase Ordering, Compiler optimization, Ma-
chine learning, Neural Networks, Java, Jikes RVM, Code
Feature Generation
[Copyright notice will appear here once ’preprint’ option is removed.]
1. Introduction
Selecting the best ordering of compiler optimizations for a
program has been an open problem in compilation research
for decades. Compiler writers typically use a combination of
experience and insight to construct the sequence of optimiza-
tions found in compilers. In this approach, compromises
must be made, e.g., should optimizations be included in a
default ﬁxed sequence if those optimizations improve per-
formance of some benchmarks, while degrading the perfor-
mance of others. For example, GCC has around 250 “passes”
that can be used, and most of these are turned off by default.
The developers of GCC have given up in trying to include all
optimizations and hope that a programmer will know which
optimizations will beneﬁt their code.
In optimizing compilers, it is standard practice to apply
the same set of optimizations phases in a ﬁxed order on
each method of a program. However, several researchers [2,
3, 15], have shown that the best ordering of optimizations
varies within a program, i.e., it is function-speciﬁc. Thus,
we would like a technique that selects the best ordering of
optimizations for individual portions of the program, rather
than applying the same ﬁxed set of optimizations for the
whole program.
This paper develops a newmethod-speciﬁc technique that
automatically selects the predicted best ordering of opti-
mizations for different methods of a program. We develop
this technique within the Jikes RVM Java JIT compiler and
automatically determine good phase-orderings of optimiza-
tions on a per method basis. Rather than developing a hand-
crafted technique to achieve this, we make use of an artiﬁ-
cial neural network (ANN) to predict the optimization order
likely to be most beneﬁcial for a method. Our ANNs were
automatically induced using Neuro-Evolution for Augment-
ing Topologies (NEAT) [23].
A trained ANN uses input properties (i.e., features) of
each method to represent the current optimized state of the
method and given this input, the ANN outputs the opti-
mization predicted to be most beneﬁcial to the method at
that state. Each time an optimization is applied, it poten-
tially changes the properties of the method. Therefore, af-
ter each optimization is applied, we generate new features of
1 2012/4/14
0.95	  
1	  
1.05	  
1.1	  
1.15	  
1.2	  
1.25	  Speedup	  normalized	  to	  O3	  
Op2miza2on	  Level	  O3	  
Best	  overall	  Sequence	   Best	  Sequence	  per	  benchmark	  
SPECjvm	  98	   Grande	  
Figure 1. We used genetic algorithms to ﬁnd “tuned” optimization sequence for benchmarks in the SPECjvm98 and Java
Grande benchmark suites. In a ﬁrst experiment, we obtained an optimization ordering that performed well over all the
benchmarks by using the running time of all the benchmarks as a ﬁtness function. In a second experiment, we used running
time of each individual benchmark to “evolve” the best optimization ordering for each benchmark.
the method to use as input to the ANN. The ANN then pre-
dicts the next optimization to apply based on the current op-
timized state of the method. This technique solves the phase-
ordering problem by taking advantage of the Markov prop-
erty of the optimization problem. That is, the current state of
the method represents all the information required to choose
an optimization to be most beneﬁcial at that decision point.
We discuss the Markov property and our approach in more
detail in Section 3.
The application of machine learning to compilation has
received a lot of attention. However, there has been little ef-
fort to “learn” the effect that each optimization has on the
code and to use that knowledge to choose the most appro-
priate optimization to apply. To the best of our knowledge,
the technique described in this paper is the ﬁrst to automati-
cally induce a heuristic that can predict an overall optimiza-
tion ordering to individual portions of a program. Our tech-
nique learns what order to apply optimizations rather than
tuning local heuristics, and it does this in a dynamic compila-
tion setting.1 Furthermore, we show signiﬁcant performance
improvement over an existing well-engineered compilation
system. We make the following contributions:
• We present a method of evolving an ANN to be used
for phase-ordering, which, to the best of our knowledge,
is the ﬁrst technique that “learns” from characteristics
of code being optimized what is the best ordering of
optimizations to apply.
• We show that our phase-ordering technique can achieve
signiﬁcant speedup over the traditional approach of ap-
plying a ﬁxed optimization sequence.
1 The same technique can be applied in a static compilation setting in a
straight-forward manner.
• We compare our ANN-based phase-ordering technique to
the current state-of-the-art phase-ordering technique, i.e.,
using genetic algorithms (GAs). Moreover, we show that
our technique is much more practical for phase-ordering
than using GAs.
• We present results optimizing several Java benchmark
programs to illustrate that optimization order is impor-
tant.
• We show that our trained ANN generates a customized
phase-ordering for a variety of different methods in var-
ious SPECjvm2008 and DaCapo benchmarks to create a
truly method-speciﬁc phase-ordering compiler.
2. Overview
Compiler optimizations transform the code being optimized,
thus the application of each optimization potentially affects
the beneﬁt of downstream optimizations. One of the most
prominent examples of this is the phase-ordering problem
between register allocation and instruction scheduling. How-
ever, any set of optimizations can potentially interact with
each other and can therefore participate in a phase-ordering
problem. These code interactions are an integral part of com-
piler optimizations, so it is important to understand the ef-
fects of the optimizations in order to arrange them in a way
that can deliver the most beneﬁt.
2.1 Phase-Ordering with Genetic Algorithms
Most compilers apply optimizations based on a ﬁxed order
that was determined to be best when the compiler was being
developed and tuned. However, programs require a speciﬁc
ordering of optimizations to obtain the best performance. To
demonstrate our point, we usegenetic algorithms (GAs), the
current state-of-the-art in phase-ordering optimizations [3,
2 2012/4/14
6–9, 14, 16, 16, 17], to show that selecting the best ordering
of optimizations has the potential to signiﬁcantly improve
the running time of dynamically-compiled programs.
We used GAs to construct a custom ordering of optimiza-
tions for each of the Java Grande [22] and SPEC JVM 98
benchmarks.2 In this GA approach, we create a population
of strings (called chromosomes), where each chromosome
corresponds to an optimization sequence. Each position (or
gene) in the chromosome corresponds to a speciﬁc optimiza-
tion from Table 2, and each optimization can appear multi-
ple times in a chromosome. For each of the experiments be-
low, we conﬁgured our GAs to create 50 chromosomes (i.e.,
50 optimization sequences) per generation and to run for 20
generations.
We ran two different experiments using GAs. The ﬁrst
experiment consisted of ﬁnding the best optimization se-
quence across our benchmarks. Thus, we evaluated each op-
timization sequence (i.e., chromosome) by compiling all our
benchmarks with each sequence. We recorded their execu-
tion times and calculated their speedup by normalizing their
running times with the running time observed by compiling
the benchmarks at the O3 level. That is, we used average
speedup of our benchmarks (normalized to opt level O3) as
our ﬁtness function for each chromosome. This result corre-
sponds to the “Best Overall Sequence” bars in Figure 1. The
purpose of this experiment was to discover the optimization
ordering that worked best on average for all our benchmarks.
The second experiment consisted of ﬁnding the best op-
timization ordering for each benchmark. Here, the ﬁtness
function for each chromosome was the speedup of that opti-
mization sequence over O3 for one speciﬁc benchmark. This
result corresponds to the “Best Sequence per Benchmark”
bars in Figure 1. This represents the performance that we
can get by customizing an optimization ordering for each
benchmark individually.
The results of these experiments conﬁrm two hypothe-
ses. First, signiﬁcant performance improvements can be ob-
tained by ﬁnding good optimization orders versus the well-
engineered ﬁxed order in Jikes RVM. The best order of op-
timizations per benchmark gave us up to a 20% speedup
(FFT) and on average 8% speedup over optimization level
O3. Second, as shown in previous work, each of our bench-
marks requires a different optimization sequence to obtain
the best performance. One ordering of optimizations for the
entire set of programs achieves decent performance speedup
compared to O3. However, the “Best Overall Sequence”
degrades the performance of three benchmarks ( LUFact,
Series, and Crypt) compared to O3. In contrast, search-
ing for the best custom optimization sequence for each
benchmark, “Best Sequence for Benchmark”, allows us to
outperform both O3 and the best overall sequence.
2 We choose these benchmarks because they run for a short time. This
allowed us to evaluate thousands of different optimization sequences using
GAs.
2.2 Issues with Current State-of-the-Art
Using genetic algorithms is the current state-of-the-art in
obtaining good optimization orderings, and they can bring
signiﬁcant performance improvements for some programs.
However, using GAs has several issues that impede their
widespread adoption in traditional compilers.
Expensive Search: GAs and other search techniques are
inherently expensive because they need to evaluate a variety
(typically hundreds) of different optimization orders for each
program and are therefore only applicable when compilation
time is not an issue, e.g., in an iterative compilation scenario.
And, because there is no transfer of knowledge, the search
space corresponding to the potential optimization orders has
to be explored anew for each new benchmark or benchmark
suite. We show empirical results in Section 7 describing the
time it took to for GAs to construct optimization orderings.
Method-speciﬁc difﬁculty: Using GAs to ﬁnd custom or-
derings of optimizations for speciﬁc code segments within
a program (e.g., for each method) is non-trivial. An or-
der of optimization speciﬁc to each piece of code requires
a separate exploration of the optimization ordering space
for that code. This requires obtaining ﬁne-grained execu-
tion times for each piece of code after it is optimized with a
speciﬁc phase-ordering. Fine-grained timers produce notori-
ously noisy information and can be difﬁcult to implement. 3
Note that exhaustive exploration to ﬁnd the optimal order
of optimizations is not practical. For example, if we consider
15 optimizations and an optimization sequence length of
20, the number of unique sequences exhaustive exploration
that would have to be evaluated is enormous (15 20). Thus,
the current state-of-the-art is to intelligently explore a small
fraction of this space using genetic algorithms or some other
search algorithm.
2.3 Proposed Solution
Instead of using expensive search techniques to solve the
phase-ordering problem, we propose to use a machine-
learning based approach which automatically learns a good
heuristic for phase-ordering. This approach incurs a “one-
time” expensive training process, but is inexpensive to use
when being applied to new programs. There are two poten-
tial techniques we could use to predict good optimization
orders for code being optimized.
1. Predict the complete sequence: This technique requires a
model to predict the complete sequence of optimizations
that needs to be applied to the code just by looking at
characteristics of the initial code to be optimized. This
is a difﬁcult learning task as the model would need to
3 Evaluating optimization orders for a method outside of an application con-
text [19] can simplify ﬁne-grained timing, but has the potential to identify
optimization sequences that do not perform well when the method is used
in its original context.
3 2012/4/14
JikesRVM
Op+mizer
Back	  end
Front	  end
Generate	  
Features
Run	  Benchmarks
&
Calculate	  Speedup
Apply
Op+miza+ons
NEAT
Regenerate	  Features
Feedback
Figure 2. The ﬁgure above represents the framework used
to evolve a neural network using NEAT to guide the compi-
lation of a given method. The Figure 4 describes the way the
neural network was used to guide the compilation process.
understand the complex interactions of each optimization
in the sequence.
2. Predict the current best optimization: This method would
use a model to predict the best single optimization (from
a given set of optimizations) that should be applied based
on the characteristics of code in its present state. Once an
optimization is applied, we would reevaluate characteris-
tics of the code and again predict the best optimization to
apply given this new state of the code.
In this paper we choose the second approach, which we
believe is an easier learning problem to solve.
We used a technique called Neuro-Evolution for Aug-
menting Topologies to automatically construct a heuristic
that can generate customized optimization orderings for each
method in a program. The process of developing this heuris-
tic is depicted in Figure 2 and described in detail in Sec-
tion 3. This approach involves continually interrogating a
neural network to predict which optimization would produce
the best results as a method is being optimized. Our network
uses as input features characterizing the current state of the
code being optimized and correlates those features with the
best optimization to use at particular point in the optimiza-
tion process. As we are considering dynamic JIT compila-
tion, the neural network and feature generator must incur a
small overhead, otherwise the cost of applying the network
to perform phase-ordering might outweigh any beneﬁts of
the improved optimization orders.
1. NEAT constructs an ANN
(a) Integrate the ANN into Jikes RVM’s optimization driver
2. Evaluate ANN at the task of phase-ordering optimizations
(a) For each method dynamically compiled, repeat the follow-
ing two steps
i. Generate a feature vector of current method’s state
ii. Use ANN to predict the best optimization to apply
3. Run benchmarks and obtain feedback for NEAT
(a) Record execution time for each benchmark optimized using
the ANN
(b) Obtain speedup by normalizing each benchmark’s running
time to running time using default optimization heuristic
(e.g., opt level O3)
Figure 3. NEAT performs neuro-evolution to construct a
neural network to be used for phase-ordering. The above
steps describe the process of evaluating an ANN as part of
neuro-evolution.
Another approach would be to handcraft a heuristic based
on experimentation and analysis. This is undesirable because
it is an arduous task and speciﬁc to a compiler, and if the
platform were to change, the entire tuning process of the
heuristic would have to be repeated.
3. Approach
This section gives a detailed overview of how neuro-
evolution based machine learning is used to construct a
good optimization phase-ordering heuristic for the optimizer
within Jikes RVM. The ﬁrst section outlines the different ac-
tivities that take place when training and deploying a phase-
ordering heuristic. This is followed by sections describing
how we use NEAT to construct an ANN, how we extract
features from methods, and how these features and ANNs
allow us to learn a heuristic that determines the order of
optimizations to apply. Figure 3 outlines our technique.
3.1 Overview of Training and Deployment
There are two distinct phases, training and deployment.
Training occurs once, off-line, “at the factory” and is equiv-
alent to the time spent by compiler writers designing and
implementing their optimization heuristics. Deployment is
the act of applying the heuristic at dynamic compilation time
to new “unseen” programs.
As part of the training phase, NEAT generates an ANN
that is used to control the order of optimizations within Jikes
RVM. The ANN is evaluated by applying different optimiza-
tion orderings to each method within each training program
and recording the performance of the optimized program.
The ANN takes as input a characterization (called a feature
vector) of current state of the method being optimized and
outputs a set of probabilities corresponding to the beneﬁt of
applying each optimization. The optimization with the high-
4 2012/4/14
est probability is applied to the method. After an optimiza-
tion is applied, the feature vector of the method is updated
and fed into the network for another round of optimization.
One output of the network corresponds to “stop optimizing,”
and the optimization process continues until this output has
the highest probability.
Once the best ANN is evolved, it is installed into the
Jikes RVM compiler and used at runtime as an optimization
heuristic. The next sections describe these stages in more
detail.
3.2 Markov Property
Most compilers apply optimizations in a ﬁxed order, and this
order is tuned for a particular set of benchmarks. This tuning
process is performed manually and is tedious and relatively
brittle. Also, the tuning procedure needs to be repeated each
time the compiler is modiﬁed for a new platform or when
a new optimization is added to the compiler. Most impor-
tantly, we have empirical evidence that each method within
a program requires the application of a speciﬁc order of op-
timizations to achieve the best performance. In this paper,
we propose to use machine learning to mitigate the compiler
optimization phase-ordering problem.
Determining the correct phase ordering of optimizations
in a compiler is a difﬁcult problem to solve. In the absence of
an oracle to determine the correct ordering of optimizations,
we must use a heuristic to predict the best optimization to
use. We formulate the phase-ordering problem as a Markov
Process. In a Markov Process, the heuristic makes a decision
on what action to perform (i.e., optimization to apply) based
on the current state of the environment (i.e., the method
being optimized). In order to perform learning, the state must
conform to the Markov Property, which means that the state
must represent all the information needed to make a decision
of what action to perform at that decision point. In our
framework, the current state of the method being optimized
serves as our Markov state because it succinctly summarizes
the important information about the complete sequence of
optimizations that led to it.
3.3 Neuro-Evolution Overview
In the this paper, we use Neuro-Evolution of Augmenting
Topologies (NEAT) to construct our neural networks to be
used for phase-ordering. NEAT uses a process of natural
selection to construct an effective neural network to solve
a particular task. This process starts by randomly generating
an initial population (or generation) of neural networks and
evaluating the performance of each network at solving the
speciﬁc task at hand.
The number of neural networks present in each genera-
tion is set to 60 for our experiments. Each of these 60 neural
networks is evaluated by using them to optimize the bench-
marks in the training set. A ﬁtness is associated with each
network as described in Section 3.5.3. Once the initial set of
generated neural networks are evaluated, the 10 best neural
Feature generation
Apply neural 
network
One of the possible 
optimizations that
can be applied
Optimization
predicted to be the
best by the neural
network
Multiple iterations of
feature generation,  
neural network 
prediction, and
optimizations
Stop applying any 
more optimazations
Figure 4. This ﬁgure represents the phase-ordering process.
The process starts when the Jikes RVM optimizer receives
a method to optimize. We iterate over the instructions of
the method to generate the features, and then provide these
features to the neural network. The neural network then
provides a set of outputs, which represent the probabilities
of each optimization being beneﬁcial. The optimization with
the highest probability is applied to the code. One of the
outputs of the network corresponds to “stop optimizing.”
When the probability of this output is highest, the optimizer
stops applying optimizations to the method.
networks from this set are propagated to the next generation
and are also used to produce new neural networks in this
generation.
This process continues and each successive generation
of neural networks produces networks that performs better
than the networks from the previous generation. New net-
works are created using mutation and crossover of the best
networks from the previous generation. During the process
of constructing new networks, we mutate the topology of a
progenitor network. Mutation can involve adding a neuron
to an existing edge in a network’s hidden layer. We set the
probability of adding a neuron to a low value (.1%) to keep
our networks small and efﬁcient. Mutation can also involve
adding a new edge (probability .5%) or deleting an exist-
ing edge (probability .9%). These probabilties are within the
ranges suggested by the authors of NEAT. Neurons are re-
5 2012/4/14
Feature Meaning
bytecodes Number of bytecodes in the
method
locals space Number of words allocated
for locals
synch Method is synchronized
exceptions Method has exception han-
dling code
leaf Method is a leaf (contains no
calls)
ﬁnal Method is declared ﬁnal
private Method is declared private
static Method is declared static
Category Fraction of bytecodes that ...
aload, astore are Array Loads and Stores
primitive, long are Primitive or Long compu-
tations (e.g., iadd, fadd)
compare are Compares (e.g., lcmp,
dcmpl)
branch are Branches (forward/back-
ward/cond/uncond)
jsr are a JSR
switch are a SWITCH
put/get are a PUT or GET
invoke are an INVOKE
new are a NEW
arraylength are an ArrayLength
athrow,checkcast,monitor are an Athrow, checkcast, or
monitor
multi newarray are a Multi Newarray
simple, long, real are a Simple,Long, or Real
Conversions
Table 1. Method features being collected. To reduce the
length of the table several (different) features have been
placed in logical groups.
moved when the last edge to or from that neuron is removed.
The mutation probabilities were manually-tuned for our spe-
ciﬁc task. For our present experiments, we stopped after 300
generations, which was when the performance of the net-
works no longer improved at the task of phase-ordering for
our training benchmarks. Figure 2 depicts the process of
constructing a neural network using NEAT to replace the op-
timization heuristic in Jikes RVM.
3.4 Feature Extraction
Determining the properties of a method that predict an opti-
mization improvement is a difﬁcult task. As we are operating
in a dynamic compilation environment, we chose features
that are efﬁcient to calculate and which we thought were rel-
evant. Computing these features requires a single pass over
the instructions of the method. Table 1 shows the 26 features
used to describe the current state of each method being op-
timized. The values of each feature will be an entry in the
26−element feature vector x associated with each method.
The ﬁrst 2 entries are integer values deﬁning the size of the
code and data of the method. The next 6 are simple boolean
properties (represented using 0 or 1) of the method. The re-
maining features are simply the percentage of bytecodes be-
longing to a particular category (e.g., 30% loads, 22% ﬂoat-
ing point, 5% yield points, etc.).
3.5 Applying NEAT
There are many characteristics (i.e., features) that can inﬂu-
ence the phase-ordering decision, and these factors may have
complex interdependencies between them. In order to effec-
tively model the non-linear behavior of these features, our
neural networks are multilayer perceptrons.
3.5.1 Why NEAT?
NEAT can be used to solve challenging tasks because it can
evolve networks of unbounded complexity from a minimal
starting point. This method has been shown to outperform
the best ﬁxed-topology method on challenging reinforce-
ment learning tasks [23] The reason that NEAT is faster and
better than typical reinforcement learning is three-fold: 1) it
incrementally grows networks from a minimal structure, 2)
it protects structural innovation using natural selection, and
3) it employs a principled method of crossover of different
topologies.
Neural networks are traditionally trained using super-
vised learning algorithms, which require a labeled training
set. The labeled training set consists of a feature vector that is
used as input, which characterizes a particular decision point
and the correct label or desired output the network should
produce when given this input. In the case of the phase order-
ing problem, we would need a feature vector corresponding
to the code being optimized and the desired output would be
the sequence of optimizations to apply to that code. Generat-
ing this labeled dataset requires knowing the right sequence
of optimizations to apply to a method is difﬁcult as discussed
in Section 2.3.
3.5.2 Structure of the network
In our neural networks, each feature or characteristic of the
method is fed to an input node, and the layers of the network
can represent complex ”nonlinear” interaction between the
features. Each output node of the network controls a particu-
lar optimization that could be applied. The outputs are num-
bers between 0 or 1 depending on whether the optimization
is predicted to be beneﬁcial to the state of the code currently
being optimized. We apply the optimization pertaining to the
output that is closest to 1 indicating the optimization that the
network predicts will be most beneﬁcial. One of the outputs
of the ANN tells the optimizer to stop optimizing. When the
probability of this output is highest, the optimizer stops ap-
plying optimizations to the method. Figure 4 shows the pro-
cess of phase-ordering.
6 2012/4/14
3.5.3 Fitness Functions
The ﬁtness value we used for the NEAT algorithm is the
arithmetic mean of the performance of the benchmarks in
the training set. That is, the ﬁtness value for a particular
performance metric is:
Fitness(S) =∑s∈S Speedup (s)
|S|
where S is the benchmark training suite and Speedup(s) is
the metric to minimize for a particular benchmarks, which in
our case is the run time (i.e., running time of the benchmark
without compile time).
Speedup (s) =Runtime(sde f )/Runtime(s)
where sde f is a run of benchmark s using the default
optimization order of optimization level O3. The goal of the
learning process is to create a heuristic that determines the
correct order of optimizations to apply to a particular method
thereby reducing the running time of the suite of benchmarks
in the training set.
4. Infrastructure + Methodology
In this section we describe the platform, the benchmarks, and
the methodology employed in our experiments.
4.1 Platform
For our experiments in this paper, we modiﬁed version 3.1.1
of the Jikes Research Virtual Machine [4]. The VM was
run on an Intel x86 based machine, supporting two AMD
Opteron 2216 dual core processors running at 2.6GHz with
an L1 and L2 cache and RAM of 128K, 1M and 8GB, re-
spectively. The operating system on the machine was Linux,
running kernel 2.6.32. We used the FastAdaptiveGenMS
conﬁguration of Jikes RVM, indicating that the core vir-
tual machine was compiled by the optimizing compiler at
the most aggressive optimization level and the generational
mark-sweep garbage collector was used.
4.2 Benchmarks
For the present set of experiments we used four benchmark
suites. For our training set, we used seven benchmarks from
the Java Grande benchmark suite [29]. These benchmarks
were used for training primarily due to their short execution
times.
For the test set, we used the SPECjvm98 [28], the
SPECjvm2008 [27], and the DaCapo benchmark [1] suites.
We used all the benchmarks from SPECjvm98 and the sub-
set of benchmarks from SPECjvm2008 and DaCapo that
we could correctly compile with Jikes RVM. We used the
largest inputs for all benchmarks. 4 The SPEC JVM bench-
marks have been designed to measure the performance of
4 Note that for the benchmark FFT in SPECjvm2008, we used the small
input size because the large input size required more memory than was
available on our experimental platform.
OptKey Meaning
Optimization Level O0
CSE Local common sub expression elimination
CNST Local constant propagation
CPY Local copy propagation
SA CFG Structural Analysis
ET Escape Transformations
FA Field Analysis
BB Basic block frequency estimation
Optimization Level O1
BRO Branch optimizations
TRE Tail recursion elimination
SS Basic block static splitting
SO Simple optimizations like Type prop,
Bounds check elim, dead-code elim, etc.
Optimization Level O2
LN Loop normalization
LU Loop unrolling
CM Coalesce Moves
Table 2. The set of optimizations that were used to perform
phase ordering in our experiments.
the Java Runtime Environment (JRE) and focus on core Java
functionality. The DaCapo benchmark suite is a collection
of programs that were designed for various different Java
performance studies. The results in Section 5 come from the
benchmarks in our test set.
4.3 Optimization Levels
We ran our experiments in two scenarios, ﬁrst using only the
optimizing compiler in a non-adaptive scenario and second
using the adaptive compilation mode. In the optimizing com-
pilation scenario, we set the initial compiler to be the opti-
mizing compiler and disable any recompilation. This forces
the compiler to compile all the loaded methods at the high-
est optimization level. Under the adaptive scenario, all dy-
namically loaded methods are ﬁrst compiled by the baseline
compiler that converts bytecodes straight to machine code
without performing any optimizations. The resultant code is
slow, but the compilation times are fast. The adaptive opti-
mization system then uses online proﬁling to discover the
subset of methods where a signiﬁcant amount of the pro-
gram’s running time is being spent. These “hot” methods
are then recompiled using the optimizing compiler. During
this process these methods are ﬁrst compiled at optimization
level O0, but if they continue to be important they are re-
compiled at level O1, and ﬁnally at level O2 if warranted.
Available optimizations are divided into different optimiza-
tion levels based on their complexity and aggressiveness.
When using the neural network in the adaptive scenario, we
disabled the optimizations that belonged to a higher level
than the present optimization level being used.
7 2012/4/14
4.4 Measurement
In a dynamic compiler like Jikes RVM, there are two types
of execution times that are of interest, total time and run-
ning time. The total time of a program is the time that the
dynamic compiler takes to compile the code from bytecodes
to machine code, and then to actually run the machine code.
The running time of a program is considered to be just the
time taken to run the machine code after it has been com-
piled by the dynamic compiler during a previous invocation.
For programs with short running times the total time is of
interest, as the compilation process itself is the larger chunk
of the execution time. However for programs that are likely
to run for longer durations, e.g. programs that perform heavy
computation or server programs that are initialized once and
remain running for a longer period of time, it is important to
highly optimize the machine code being generated. This is
true even at the expense of potentially greater compile time,
as the compilation time is likely to be overshadowed by the
execution of the machine code that has been generated by
the dynamic compiler. The time taken to execute the bench-
mark for the ﬁrst invocation is taken as the total time. This
time includes the time taken by the compiler to compile the
bytecodes into machine code and the running of the machine
code itself. The running time is measured by running the
benchmark over ﬁve iterations and taking the average of the
last three execution times, this ensures that all the required
methods and classes had been preloaded and compiled. To
compare our performance we normalize our running times
and total times with the default optimization setting. This
default compilation scenario acts as our baseline, which is
the average of twenty running times and twenty total times
for each benchmark. The noise for all benchmarks in this
paper was less that 1.2% and the average noise was 0.7%.
4.5 Evaluation Methodology
As is standard practice, we evolve our neural network over
one suite of benchmarks, commonly referred to in the ma-
chine learning literature as the training set. We then test
the performance of our evolved neural network over another
“unseen” suite of benchmarks, that we have not trained on,
referred to as the test set.
5. Results
In this section, we present our results of using the neural net-
work that performed best on the training set. We used this
network to determine good optimization orders for methods
in programs from the SPECjvm98, SPECjvm2008, and Da-
Capo benchmark suites in both an adaptive and optimizing
compilation scenario.
5.1 Adaptive Compiler
In the adaptive compilation scenario, we allowed the adap-
tive compiler to decide the level of optimization to be used
to optimize methods as described in Section 4.3. However, at
each optimization level we used the induced neural network
to decide to order of optimizations to apply at that level. In
this scenario, we obtained an average speedup of 8% in run-
ning time and 4% improvement in the total execution time
over all the benchmarks versus the default adaptive mode in
Jikes RVM
SPECjvm98 Running time Using our neural network for
phase-ordering, we were able to obtain an average speedup
of 10% across the seven benchmarks of the SPECjvm98
benchmark suite on the running time. We got signiﬁcant
improvements over default on mpegaudio (20%), compress
(14%), and javac (11%).
Total timeWe observed a modest increase in performance
of 3% on average on the SPECjvm98 benchmarks. However,
it is important to note that we achieved these speedups de-
spite of the overhead of feature extraction and the execution
of the neural network. The javac program gave us the best
total time speedup at around 7%.
5.1.1 SPECjvm2008
Running Time We achieved an average running time
speedup of 6.4% on the SPECjvm2008 benchmarks. The
fft benchmark did give us a slowdown of a little less
than 5%. Interestingly, we discovered that the neural net-
work used very short optimization sequences to optimize
that benchmark. This helps to explain the improvement in
the total time for this benchmark as described in the next
section.
Total Time Our average performance improvement over
all ﬁve SPECjvm2008 benchmarks was around 4%. We
achieved a performance improvement of up to 7% on the
benchmark sor with our ANNs.
5.1.2 DaCapo
The running time performance improvement of the programs
in the DaCapo benchmark suite (at 6.8%) was not as high as
the other two benchmark suites, but their performance on
the total time of 6% was much better than the average of the
other two SPECjvm benchmark suites.
5.2 Optimizing Compiler
When running Jikes RVM in a non-adaptive mode, all the
methods are compiled directly at the highest optimization
level. The average speedup when just measuring running
time was 8.2%, and we improved the total time by over 6%.
5.2.1 SPECjvm98
Running time In SPECjvm98, we achieve up to a speedup
of 24% on mpegaudio. On average, we improved the run-
ning time performance of this benchmark suite by 10%,
which is a signiﬁcant improvement.
Total time When measuring total time, we observed a
modest increase in performance of around 3.4%. The best
8 2012/4/14
0.90	  
0.95	  
1.00	  
1.05	  
1.10	  
1.15	  
1.20	  
1.25	  Speedup	  normalized	  by	  Adap2ve	  
Adap2ve	  
Total	  Time	   Running	  Time	  
DaCapo	  SPECjvm2008	  SPECjvm	  98	  
Figure 5. Adaptive : The graph above represents the speedup achieved by using NEAT when used by Jikes RVM in adaptive
mode to optimize each benchmark in the test set. We compare our result with the performance of each of the benchmarks when
using the default adaptive compilation scenario.
0.9	  
0.95	  
1	  
1.05	  
1.1	  
1.15	  
1.2	  
1.25	  
Speedup	  normalized	  by	  O3	  
Op3miza3on	  Level	  O3	  
Total	  Time	   Running	  Time	  
DaCapo	  SPECjvm	  2008	  SPECjvm	  98	  
Figure 6. Optimization Level O3 : The graph above represents the speedup achieved by using NEAT when used in the non-
adaptive mode in Jikes RVM to optimize each benchmark in the test set. We compare our result with the performance of each
of the benchmarks when using the default non-adaptive compilation scenario.
performing benchmark was again mpegaudio at 11%
speedup.
5.2.2 SPECjvm2008
Running Time We achieved an average running time
speedup of 7% over all the ﬁve benchmarks of the SPECjvm2008
benchmark suite. The best performing benchmark from the
SPECjvm2008 suite wassor with a speedup of almost 12%.
Total Time An interesting observation here is the perfor-
mance of the fft benchmark. In all other cases this bench-
mark had a minor slowdown. We realized that the average
optimization sequence length suggested by the neural net-
work was 11. This is very short compared to the default ﬁxed
order sequence length of 23. This reduction in the sequence
9 2012/4/14
f a c t o r ( )
{
f o r ( . . . ) {
a r i t h e m a t i c o p e r a t i o n o v e r an a r r a y
f o r ( . . . ) {
a r i t h e m a t i c o p e r a t i o n o v e r an a r r a y
}
}
. . .
i f ( . . . ) {
f o r ( . . . ) {
a r i t h e m a t i c o p e r a t i o n o v e r an a r r a y
}
}
i f ( . . . ) {
f o r ( . . . ) {
f o r ( . . . ) {
a r i t h e m a t i c o p e r a t i o n o v e r an a r r a y
}
}
}
}
Listing 1. Pseudo-code for scimark.lu.LU.factor, the hottest
method for the SPEC2008 lu benchmark
LABEL1
i n i f c m p <CONDITION> GOTO LABEL2
. . .
GOTO LABEL1
LABEL2
Listing 2. Slow Code (SA applied before BRO): The
following code corresponds to a while loop , where n
iterations of the loop require n conditional jumps and n
unconditional jumps. This is the code produced by using
optimization level O3.
i n i f c m p <!CONDITION> GOTO LABEL2
LABEL1
. . .
i n i f c m p <CONDITION> GOTO LABEL1
LABEL2
Listing 3. Fast Code (BRO applied before SA): The fol-
lowing code corresponds to a do-while loop, where n it-
erations of the loop would requiren+1 conditional jumps
but no unconditional jumps, this improves the perfor-
mance. This is the code produced using the optimization
ordering produced using our neural network.
Figure 7. Listing 1 shows the pseudo code of the scimark.lu.LU.factor method that is compiled by the optimizing compiler.
The two HIRs generated for scimark.lu.LU.factor by the two different optimization orderings are shown in Listing 2 and
Listing 3. Changing the order that the transformations are applied changes the running time by almost 7%
length helped to reduce the amount of compilation required,
and thus improves total time performance.
5.2.3 DaCapo
Running Time Using the Jikes RVM in a non-adaptive
mode, we were able to get some signiﬁcant speedups of 17%
for pmd and 10.6% for lusearch. There were no signif-
icant slowdowns and on average we observed a speedup of
7.3% on the DaCapo benchmark suite.
Total Time We saw signiﬁcant speedups across DaCapo
with 14% speedups on xalan, luindex and lusearch,
and speedups of 5%, 8%, and 9% on the three other pro-
grams. On average, we had an improvement 11%.
6. Exploration of Phase ordering beneﬁt
In this section, we tried to analyze the optimization orderings
that our neural network came up with. We ran the bench-
marks and collected the proﬁling runs, which gave us an
idea of which methods were most important. Looking at the
neural network does not typically give any intuition of the
phase-ordering heuristic, however it may help to understand
the rough complexity of the ﬁnal solution.
The neural network found interesting combinations of
transformations that helped in improving the performance
of some of the benchmarks. For example, the code shown in
Figure 4 is the hottest method in the scimark.lu.small bench-
mark. The ﬁgure also shows code after applyingBranch Op-
timization before CFG Structural Analysis (i.e., the ordering
obtained from the default optimization level) and the code
when applying these two optimizations in the reverse order
(i.e., the ordering obtained from our neural network). We
looked at the machine code being generated in both cases
and realized that when CFG Structural Analysis was applied
before Branch Optimization, the code that was generated had
more branch statements. A snippet of representative code is
shown in Figure 7. In the slower code, the loops are rep-
resented as while loops, and the code that worked best had
loops that are represented as do-while loops. This small dif-
ference in the machine code gave an improvement of ap-
proximately 8% in the running time of the scimark.lu
benchmark. Because the original code had a large fraction
of unconditional branch statements, it triggered the neural
network to apply CFG Structural Analysis . This kind of
ﬁne-grained optimization can be achieved when using a our
method of phase ordering.
Analyzing another benchmark,scimark.sparse, which
performs sparse matrix multiplication, we see another simi-
lar phenomena. We looked at the sparse.SparseCompRow.matmul
method, which is the hottest method in the benchmark and
has multiple nested loops as represented in Figure 8. Consid-
ering the number of nested loops in this method, Loop Un-
rolling could potentially be an optimization to this method.
Howere we realized that our neural network applied CFG
10 2012/4/14
matmul ( )
{
f o r ( . . . ) {
a r i t h e m a t i c o p e r a t i o n o v e r an a r r a y
f o r ( . . . ) {
f o r ( . . . ) {
a r i t h e m a t i c o p e r a t i o n o v e r an a r r a y
}
}
}
. . .
i f ( . . . ) {
f o r ( . . . ) {
a r i t h e m a t i c o p e r a t i o n o v e r an a r r a y
}
}
}
Listing 4. Pseudo-code for matmult, the hottest
method for the SPEC2008 sparse benchmark.
LABEL1
. . .
i n t i f c m p <CONDITION> GOTO LABEL3
g o t o LABEL2
LABEL2
g o t o LABEL4
LABEL3
g o t o LABEL1
LABEL4
Listing 5. Slow Code (LU applied before SA): Here the num-
ber. of unconditional statements are unnecessary, and hampers
the performance of the code. This is the code produced by using
optimization level O3.
LABEL1
. . .
i n t i f c m p <CONDITION> GOTO LABEL1
Listing 6. Fast Code (SA applied before LU): During compi-
lation, CFG Structural Analysis was applied before Loop Un-
rolling, which gave the compiler a chance to clean up the code
before the loop unrolling was applied. This is the code produced
using the optimization ordering produced using our neural net-
work.
Figure 8. The ﬁnal HIR generated for sparse.SparseCompRow.matmult by the two different optimization orderings. The code
generated by applying CFG Structural Analysis before Loop Unrolling shown in Listing 6 performs better in terms of running
time and achieved a speedup of almost 14%.. When looking at the other characteristics, the number of unconditional jumps
were reduced by 33% and there was a 10% reduction in the number of basic blocks.
Structural Analysis before it applied Loop Unrolling. This
ordering helped in improving the quality of the code, im-
proving the total running time by almost 14%. Again, this
particular ordering is not present in the default ordering
present in the JikesRVM compiler. There were some dif-
ferences in the machine code that were generated. The exact
change in the machine that caused this huge speedup cannot
be pinpointed, however we found a few instances of ma-
chine code that were less than optimum. Figure 8 shows a
piece of machine code that is less than otimal. When looking
at this particular instance we quickly relaized that the code
placement was needlessly complex. For example, if only
the target of the ﬁrst conditional jump was set to LABEL1,
we would not need the last three unconditional jumps. In-
tuitively, a compiler writer would try to ﬁx the problem by
applying another optimization like Branch Optimization or
applying CFG Structural Analysis once more. But, in this
particular case repeating CFG Structural Analysis or apply-
ing another instance of Branch optimization did not improve
the performance of the code.
7. Training Time
Training our machine learning heuristic requires us to pro-
vide ﬁtness values to each of the heuristics being tested. In
our neuro-evolution scenario, the ﬁtness of the heuristic can
only be measured in terms of the performance of the bench-
mark when this heuristic is applied.
This makes the execution time of the benchmark the bot-
tleneck in our experiments. In order to give a clearer picture
we calculated the rough training time that was required to
train a phase-ordering sequence for each benchmark indi-
vidually when using genetic algorithm. This is shown in the
Table 5. Given the number of days that it can take to train
each benchmark we feel that is is impractical to use GA’s
for phase-ordering, especially within a dynamic compilation
scenario.
8. Discussion
In this section, we brieﬂy describe the neural network that
we used for the experiments and discuss some observations
(e.g., the reduction in the optimization sequence length, a
case of repeated optimizations, and handling of relatively ﬂat
proﬁles.)
Neural Network We used one neural network for all the
results shown in Table 4 and Figures 5 and 6. This network
had 30 inputs, 14 outputs, 24 hidden nodes, and 503 total
connections.
Reduction of optimization sequence length From our ex-
periments, we were able to demonstrate two achievements.
11 2012/4/14
Program Avg. Seq.
length
SPECjvm98
javac 18
mpegaudio 19
jess 16
compress 19
raytrace 18
jack 17
SPECjvm2008
fft 11
lu 18
monte carlo 17
Program Avg. Seq.
length
SPECjvm2008 contd.
sparse 20
sor 16
DaCapo
avrora 19
luindex 16
lusearch 16
pmd 18
sunflow 16
xalan 17
Average 17
Default 23
Table 3. The average number of optimizations that were
applied by the neural network.
Intelligent ordering of the sequences provided us with sig-
niﬁcant speedups. We also show that intelligently applying
the right optimizations helps in improving the compile time
by not having to apply optimizations that have little impact
on a method’s performance. This would reduce the compila-
tion burden on the system, and directly improve the system
performance in terms of total execution time.
A detailed analysis of the phase orderings suggested by
the ANN is shown in the Table 4. We typically applied 16-20
optimizations while the default optimizing compiler applied
23. We believe that this is signiﬁcant. That is, we were able
to apply the right optimizations and thus more effectively
utilize the optimization resources available to us.
Repeating optimizations In some cases the optimizations
get repeated back to back. For example, the sequence shown
in the fourth row of Table 4, the network predicted to ap-
ply Static Splitting twice in succession. This situation arises
when applying a particular optimization does not change the
feature vector. We could potentially be stuck in an inﬁnite
loop where the feature vector remains the same, thus inad-
vertently causing the neural network to apply the same opti-
mization, which causes an inﬁnite loop. In order to overcome
this situation, if the network predicts that applying the same
optimization again would be beneﬁcial, we allow for a maxi-
mum of 5 such repetitions, and then instead apply the second
best optimization.
Improvements from present state of art At present the best
way to tune phase ordering is to use GA to optimize in the
search. There are a few problems with this approach, each
benchmark has to be tuned individually, if we use a training
set and a test set, the results are not as good as shown in
Figure 9. Figure 9 compares the present state of the part in
phase ordering with our approach. The ﬁrst bar is by training
GA on a training set and testing it on the test set, similar to
the second bar where we used NEAT. The last bar is when
0.9	  
1	  
1.1	  
1.2	  
1.3	  
1.4	  
1.5	  
0	   1	   2	   3	   4	   5	   6	  
Speedup	  
No.	  of	  methods	  covering	  60%	  of	  running	  8me	  
Speedups	  vs	  Method	  importance	  
GA	  Speedup	  
NEAT	  Speedup	  
Figure 10. Speedup based on method importance : The
plot above represents the speedup achieved by evolving an
optimization sequence using genetic algorithm per bench-
mark and NEAT. Each data point in the plot corresponds to a
benchmark, and the plot depicts the number of methods that
constitute 60% of the running time for a particular bench-
mark versus the speedup obtained for that benchmark.
we individually searched for the best phase ordering using
GA for each benchmark. Even with the advantage of being
trained on each benchmark individually, the performance
GA per benchmark is not much better than using NEAT,
which does not require individual training runs.
Flat-proﬁled benchmarks For some benchmarks, the run-
ning time of the benchmark is equally divided among mul-
tiple methods (i.e., a ﬂat proﬁle), while other benchmarks
have the majority of the execution time is spent in just one
or a few methods. Finding a good phase ordering in case
of benchmarks with one single “hot” method is relatively
straight-forward. We would simply be searching for an op-
timization sequence that was beneﬁcial for the one impor-
tant method of the benchmark. Since the execution time is
dominated by a single method, we would see an overall im-
provement in the performance of the benchmark even if the
method-speciﬁc phase ordering negatively affects the perfor-
mance of the other methods.
In order to demonstrate our point, we conducted an exper-
iment where we allowed the genetic algorithm to search for
the best optimization sequence to be applied to each bench-
mark. This was the method proposed by Cooperet al. [6] and
was shown to ﬁnd good optimization sequences for a pro-
gram. Figure 10 shows the speedup achieved by both GAs
and neural networks on each benchmark as it relates to the
12 2012/4/14
1.45	  
0.90	  
0.95	  
1.00	  
1.05	  
1.10	  
1.15	  
1.20	  
1.25	  Speedup	  normalized	  by	  O3	  
Gene4c	  Algorithm	  vs	  Neural	  Network	  
Gene,c	  Algorithm	   Neural	  Network	   Gene,c	  Algorithm	  per	  benchmark	  
DaCapo	  SPECjvm2008	  SPECjvm	  98	  
Figure 9. Genetic Algorithm versus Neural Network : The graph above represents the speedup achieved by the best
optimization sequence found by the genetic algorithm for all the benchmarks in the training set, when applied to the test
set (JikesRVM in non-adaptive mode). We compare our result with the performance of each of the benchmarks when using the
default non-adaptive compilation scenario.
Benchmark Hot Percent of Size Optimization
method Total Calls Sequence
SPECjvm 2008
fft(small) FFT.transform internal() 86.93% 390 CNST,CPY ,CPY ,LU,BB,SS,BB,CSE,LN,CNST,LN
lu LU.factor() 72.59% 277 TRE,CNST,CPY ,SS,SS,BRO,SA,ET,SO,
ET,LU,SS,LU,TRE,SS,SS,SO,CNST,FA,FA
monte carlo MonteCarlo.integrate() 25.31% 68 BB,CPY ,BB,TRE,CNST,BB,CSE,
CSE,LU,CSE,SS,SA,LU,FA
sparse SparseCompRow. 80.79% 161 SO,BB,LU,CNST,TRE,LN,CPY ,TRE,SS,CPY ,
matmult() SO,SO,SS,FA,BB,CNST,CPY ,TRE,CNST
sor SOR.execute() 86.51% 184 SO,SO,BB,SO,SS,CPY ,ET,TRE,CPY ,LN,CSE,
CSE,SO,LN,SA,SA,SA,BB,TRE,CNST
Table 4. This table gives information about the hottest methods in SPECjvm2008 and the optimization sequences obtained
from our neural network for each of these methods. The abbreviations used to described the sequences are explained in Table 2.
number of “hot” methods that constitute 60% of the running
time for a particular benchmark. In this ﬁgure, we see that
the GA is better at ﬁnding good speedups when the 60% of
the execution time is concentrated in just one method. How-
ever, our NEAT-evolved networks are able to achieve good
speedup when the execution time is distributed over multiple
methods. Another set of results that reafﬁrm this conclusion
is in Figure 9, if you look at the results for javac and mpe-
gaudio, both benchmarks have relatively ﬂat proﬁles, and in
both cases the individually training GA phase ordering did
not do as well as the Neural network.
9. Related Work
Auto-tuning: An area that is closely related to this paper
is the study of automatic code generation and optimization
for different computer architectures (auto-tuning), which has
been explored in many existing studies for many different
applications. A number of library generators automatically
produce high-performance kernel routines [21, 26, 30]. Re-
cent research efforts [12, 18] expand automatic code genera-
tion to routines whose performance depends not only on ar-
chitectural features, but also on input characteristics. These
systems are a signiﬁcant step toward automatically opti-
mizing code for different computer architectures. Recently,
13 2012/4/14
Program Training
time (Days)
SPECjvm98
javac 2.2
mpegaudio 0.8
jess 1.3
compress 1.1
raytrace .9
jack 1.6
SPECjvm2008
fft 10.4
lu 5
monte 8carlo
Program Training
time (Days)
SPECjvm2008 contd.
sparse 6
sor 5.1
DaCapo
avrora 7.3
luindex 3.1
lusearch 3.3
pmd 3.6
sunflow 3.1
xalan 5.6
Average 3.9
Total 70
Table 5. This table shows the average time that we have
taken if we evolved an optimization ordering using GA for
each benchmark individually.
Program GA NEAT
Java Grande 4.4 4.91
Jolden 7 8.3
Total 11.4 13.2
Table 6. Time taken in days to train the training set, to
provide the reults in Figure 9
Ganapathi [11] et al. presented some preliminary results on
the application of machine learning to auto-tuning for multi-
cores. They showed that auto-tuning of stencil codes, with
the assistance of machine learning, was able to surpass per-
formance of tuning by a domain expert. This research dis-
plays the great potential for machine learning and search
in an auto-tuning environment. However, these prior works
have all been largely focused on small domain-speciﬁc ker-
nels and still neglect exploring the beneﬁts of learning from
a knowledge base of previously explored applications and
architectures.
Machine learning applied to Compilation: Machine learn-
ing and search techniques applied to compilation has been
studied in many recent projects [5, 8, 9, 14, 20, 24, 25, 31].
These previous studies have developed machine learning-
based algorithms to efﬁciently search for the optimal se-
lection of optimizing transformations, the best values for the
transformation parameters, or the optimal sequences of com-
piler optimizations. Generally, these studies customize op-
timizations for each program or local code segments, some
based on code characteristics. The proposed research in this
paper is motivated by these studies and makes a signiﬁcant
step forward: the compiler will not only use program char-
acteristics, but will also learning to decide the right ordering
of optimizations.
Several researchers have looked at searching for the
best sequence of optimizations for a particular program [6–
8, 13, 16, 17], for example the work by Cooperet al. [6] used
genetic algorithms to solve the compilation phase ordering
problem. They were concerned with ﬁnding “good” com-
piler optimization sequences that reduced code size. Their
technique was successful at reducing code size by as much
as 40%. Unfortunately, their technique was application-
speciﬁc, i.e., a genetic algorithm had to be retrained to ﬁnd
the best optimization sequence for each new program. Also,
Cooper et al. [8] propose a technique called virtual execu-
tion to reduce the cost of evaluating different optimization
orderings. Virtual execution consists of running the program
one time and predicting the performance of different opti-
mization sequences without running the code again. These
approaches give impressive performance improvements, but
has to be performed each time a new application is compiled.
While this is acceptable in embedded environments, it is not
suitable for typical compilation.
Kulkarni et al. [17] exhaustively enumerated all distinct
function instances for a set of programs that would be pro-
duced from different phase-orderings of 15 optimizations.
This exhaustive enumeration allowed them to construct
probabilities of enabling/disabling interactions between dif-
ferent optimization passes in general and not speciﬁc to any
program. In contrast, the techniques in this paper charac-
terized methods being optimized; therefore, the techniques
described here learn which optimizations are beneﬁcial to
apply to “unseen” methods with similar characteristics.
Many researchers have also looked at using machine
learning to construct heuristics that control compiler op-
timizations. Cavazos et al. [5] used logistic regression to
control what optimizations to apply in JikesRVM. However,
they do not attempt to control the order of optimizations
and instead only turn on and off optimizations given the
hand-tuned ﬁxed order of optimizations. For the SPECjvm98
benchmarks, they were not able to achieve signiﬁcant im-
provements for running time under both non-adaptive and
adaptive scenarios likely because the ﬁxed-order of opti-
mizations in Jikes RVM had been highly tuned and there
was little room for improvement on top of this ordering by
simply turning optimizations on and off. In contrast, we
achieve good improvements on SPECjvm98 benchmarks by
applying method-speciﬁc optimization orderings.
Stephenson et al. [25] used genetic programming to tune
heuristic priority functions for three compiler optimizations
within the Trimaran’s IMPACT compiler. For one of the opti-
mizations, register allocation, they were only able to achieve
on average a 2% increase over the manually tuned heuristic.
Monsifrot et al. [20] used a classiﬁer based on decision tree
learning to determine which loops to unroll showing a few
percent improvement on two different machines. The results
in these papers highlight the diminishing results obtained
when only controlling a single optimization. In contrast, this
14 2012/4/14
research will control numerous optimizations available in
the compiler.
Agakov et al. [2] describe two models to improve the
search for good optimization orders to apply to programs.
The ﬁrst model, called the independent identically dis-
tributed model , produces a probability vector correspond-
ing to probability that a transformation occurs in a good
sequence for a particular program. When optimizing a new
program, a nearest neighbor algorithm is used to choose the
probability vector of the program in the training set closest
to the program to be optimized. This probability vector is
then used to choose optimizations for the new program. The
second model, called the Markov model simply creates a
probability matrix where the probability of an optimization
being beneﬁcial depends upon the optimizations that have
been previously applied. These models were developed to
focus the search for good optimization orderings during iter-
ative compilation. Therefore, these techniques suffers from
the same limitations as described in Section 2.1. Addition-
ally, these models use simple nearest neighbor algorithms
using the characteristics of the original unoptimized code.
Therefore, these models do not take advantage of important
characteristics of the code as it is being optimized.
Fursin et al. [10] (as part of the MILEPOST project) have
integrated machine learning algorithms in GCC to control
these optimizations applied. They show good results on three
different architectures, compared to random search of opti-
mizations sequences. However, the machine learning algo-
rithms in MILEPOST do not learn good optimization order-
ings because as the authors state “this requires detailed in-
formation about dependencies between passes to detect legal
orders”.
10. Conclusion
This paper has shown that method-speciﬁc optimization
orderings can give signiﬁcant performance improvements
within the Jikes RVM JIT compiler. It has also demonstrated
that a technique of neuro-evolution can automatically de-
rive a neural network that gives signiﬁcant performance
improvements over a well-engineered optimization order-
ing. We show total execution time improvements of up to
20%. To the best of our knowledge, this is the ﬁrst paper to
demonstrate that machine-learning models can be success-
fully used to choose optimization orders for methods within
a compiler. The present study is promising as it provides a
fresh prospective to the problem of phase ordering which has
been studied for decades. The amount of improvement that
can be found from this method has not yet reached its full
potential, and we propose to improve the machine learning
algorithm to provide better improvements.
For future work, we would like to implement similar
phase-ordering techniques in a static compiler, in order to
understand the behavior of other environments on our setup.
There is nothing about the technique that makes it speciﬁc
to dynamic compilation. In addition, we would also like to
incorporate proﬁle information into the feature set, which
allow us to improve our predictions.
References
[1] Dacapo benchmark suite. URL
http://dacapobench.org/benchmarks.html.
[2] F. Agakov, E. Bonilla, J. Cavazos, B. Franke, G. Fursin,
M. F. P. O’Boyle, J. Thomson, M. Toussaint, and C. K. I.
Williams. Using machine learning to focus iterative optimiza-
tion. In CGO ’06: Proceedings of the International Sympo-
sium on Code Generation and Optimization , pages 295–305,
Washington, DC, USA, 2006. IEEE Computer Society.
[3] L. Almagor, K. D. Cooper, A. Grosul, T. J. Harvey, S. W.
Reeves, D. Subramanian, L. Torczon, and T. Waterman. Find-
ing effective compilation sequences. In Proceedings of the
2004 ACM SIGPLAN/SIGBED conference on Languages,
compilers, and tools for embedded systems, LCTES ’04, pages
231–239. ACM, 2004.
[4] B. Alpern, C. R. Attanasio, J. J. Barton, M. G. Burke, P.Cheng,
J.-D. Choi, A. Cocchi, S. J. Fink, D. Grove, M. Hind, S. F.
Hummel, D. Lieber, V . Litvinov, M. F. Mergen, T. Ngo, J. R.
Russell, V . Sarkar, M. J. Serrano, J. C. Shepherd, S. E. Smith,
V . C. Sreedhar, H. Srinivasan, and J. Whaley. The Jalape ˜no
virtual machine. IBM Systems Jounrnal, 39(1), 2000.
[5] J. Cavazos and M. F. P. O’Boyle. Method-speciﬁc Dynamic
Compilation Using Logistic Regression. In OOPSLA ’06:
Proceedings of the 21st annual ACM SIGPLAN conference
on Object-oriented programming systems, languages, and ap-
plications, pages 229–240, New York, NY , USA, 2006. ACM
Press.
[6] K. Cooper, P. Schielke, and D. Subramanian. Optimizing for
reduced code space using genetic algorithms. In Proceedings
of the ACM SIGPLAN 1999 workshop on Languages, compil-
ers, and tools for embedded systems, pages 1–9. ACM, 1999.
[7] K. Cooper, D. Subramanian, and L. Torczon. Adaptive opti-
mizing compilers for the 21st century. The Journal of Super-
computing, 23(1):7–22, 2001.
[8] K. Cooper, A. Grosul, T. Harvey, S. Reeves, D. Subramanian,
L. Torczon, and T. Waterman. ACME: adaptive compilation
made efﬁcient. In LCTES ’05: Proceedings of the 2005 ACM
SIGPLAN/SIGBED conference on Languages, compilers, and
tools for embedded systems , volume 40, pages 69–77. ACM,
2005.
[9] K. D. Cooper, A. Grosul, T. J. Harvey, S. Reeves, D. Subrama-
nian, L. Torczon, and T. Waterman. Exploring the Structure
of the Space of Compilation Sequences Using Randomized
Search Algorithms. J. Supercomputing, 36(2):135–151, 2006.
[10] G. Fursin, C. Miranda, O. Temam, M. Namolaru, E. Yom-Tov,
A. Zaks, B. Mendelson, P. Barnard, E. Ashton, E. Courtois,
F. Bodin, E. Bonilla, J. Thomson, H. Leather, C. Williams, and
M. O’Boyle. Milepost gcc: machine learning based research
compiler. In Proceedings of the GCC Developers’ Summit ,
June 2008.
[11] A. Ganapathi, K. Datta, A. Fox, and D. Patterson. A case for
machine learning to optimize multicore performance. First
15 2012/4/14
USENIX Workshop on Hot Topics in Parallelism (HotPar ’09),
2009.
[12] S.-C. Han, F. Franchetti, and M. P ¨uschel. Program Genera-
tion for the All-pairs Shortest Path Problem. In PACT ’06:
Proceedings of the 15th international conference on Paral-
lel architectures and compilation techniques, pages 222–232,
New York, NY , USA, 2006. ACM Press.
[13] M. R. Jantz and P. A. Kulkarni. Eliminating false phase
interactions to reduce optimization phase order search space.
In CASES, pages 187–196. ACM, 2010.
[14] P. Kulkarni, S. Hines, J. Hiser, D. Whalley, J. Davidson, and
D. Jones. Fast Searches for Effective Optimization Phase Se-
quences. In PLDI ’04: Proceedings of the ACM SIGPLAN
2004 conference on Programming language design and im-
plementation, pages 171–182. ACM Press, 2004.
[15] P. A. Kulkarni, D. B. Whalley, G. S. Tyson, and J. W.
Davidson. Exhaustive optimization phase order space explo-
ration. In Fourth Annual IEEE/ACM Interational Conference
on Code Generation and Optimization , pages 306–318, New
York City, NY , March 2006.
[16] P. A. Kulkarni, D. B. Whalley, and G. S. Tyson. Evaluating
heuristic optimization phase order search algorithms. InCGO,
pages 157–169. IEEE Computer Society, 2007.
[17] P. A. Kulkarni, D. B. Whalley, G. S. Tyson, and J. W. David-
son. Practical exhaustive optimization phase order exploration
and evaluation. TACO, 6(1), 2009.
[18] X. Li, M. J. Garzar ´an, and D. Padua. Optimizing Sorting
with Genetic Algorithms. In In Proc. of the International
Symposium on Code Generation and Optimization (CGO) ,
pages 99–110, March 2005.
[19] C. Liao, D. J. Quinlan, R. W. Vuduc, and T. Panas. Effective
source-to-source outlining to support whole program empiri-
cal optimization. In LCPC’09, pages 308–322, 2009.
[20] A. Monsifrot, F. Bodin, and R. Quiniou. A machine learning
approach to automatic production of compiler heuristics. In
AIMSA ’02: Proceedings of the 10th International Conference
on Artiﬁcial Intelligence: Methodology, Systems, and Applica-
tions, pages 41–50, London, UK, 2002. Springer-Verlag.
[21] M. P ¨uschel, J. Moura, J. Johnson, D. Padua, M. Veloso,
B. Singer, J. Xiong, F. Franchetti, A. Gacic, Y . V oronenko,
K. Chen, R. W. Johnson, and N. Rizzolo. SPIRAL: Code
Generation for DSP Transforms. In Proc. of the IEEE, spe-
cial issue on Program Generation, Optimization, and Plat-
form Adaptation, 93(2):232–275, February 2005.
[22] L. A. Smith, J. M. Bull, and J. Obdrz ´alek. A parallel Java
Grande benchmark suite. In ACM, editor, SC2001: High Per-
formance Networking and Computing. Denver, CO, November
10–16, 2001. ACM Press and IEEE Computer Society Press,
2001. ISBN 1-58113-293-X.
[23] K. O. Stanley and R. Miikkulainen. Efﬁcient rein-
forcement learning through evolving neural network
topologies. In Proceedings of the Genetic and Evo-
lutionary Computation Conference (GECCO-2002) ,
page 9, San Francisco, 2002. Morgan Kaufmann. URL
http://nn.cs.utexas.edu/?stanley:gecco02b.
[24] M. Stephenson and S. Amarasinghe. Predicting Unroll Fac-
tors Using Supervised Classiﬁcation. In CGO ’05: Proceed-
ings of the international symposium on Code generation and
optimization, pages 123–134, Washington, DC, USA, 2005.
IEEE Computer Society.
[25] M. Stephenson, S. Amarasinghe, M. Martin, and U.-M.
O’Reilly. Meta Optimization: Improving Compiler Heuristics
with Machine Learning. In Proc. of Programing Language
Design and Implementation, June 2003.
[26] R. Vuduc, J. W. Demmel, and K. A. Yelick. OSKI: A Library
of Automatically Tuned Sparse Matrix Kernels. Journal of
Physics Conference Series, 16:521–530, Jan. 2005.
[27] Website. Specjvm 2008, . URL
http://www.spec.org/jvm2008/.
[28] Website. Specjvm 98, . URL
http://www.spec.org/jvm98/.
[29] Website. Java grande benchmarks, . URL
http://www2.epcc.ed.ac.uk/computing/
research activities/java grande/sequential.html.
[30] R. C. Whaley, A. Petitet, and J. J. Dongarra. Automated
Empirical Optimizations of Software and the ATLAS Project.
Parallel Computing, 27(1-2):3–35, 2001.
[31] K. Yotov, X. Li, G. Ren, M. Cibulskis, G. DeJong,
M. Garzaran, D. Padua, K. Pingali, P. Stodghill, and P. Wu. A
Comparison of Empirical and Model-driven Optimization. In
Proc. of Programing Language Design and Implementation ,
pages 63–76, June 2003.
16 2012/4/14

BOAT: Building Auto-Tuners with Structured Bayesian
Optimization
Valentin Dalibard Michael Schaarschmidt Eiko Y oneki
Computer Laboratory
University of Cambridge
ﬁrstname.lastname@cl.cam.ac.uk
ABSTRACT
Due to their complexity, modern systems expose many con-
ﬁguration parameters which users must tune to maximize
performance. Auto-tuning has emerged as an alternative in
which a black-box optimizer iteratively evaluates conﬁgura-
tions to ﬁnd eﬃcient ones. Unfortunately, for many systems,
such as distributed systems, evaluating performance takes
too long and the space of conﬁgurations is too large for the
optimizer to converge within a reasonable time.
We present BOAT, a framework which allows developers
to build eﬃcient bespoke auto-tuners for their system, in
situations where generic auto-tuners fail. At BOAT’s core
is structured Bayesian optimization (SBO), a novel exten-
sion of the Bayesian optimization algorithm. SBO leverages
contextual information provided by system developers, in
the form of a probabilistic model of the system’s behavior,
to make informed decisions about which conﬁgurations to
evaluate. In a case study, we tune the scheduling of a neural
network computation on a heterogeneous cluster. Our auto-
tuner converges within ten iterations. The optimized conﬁg-
urations outperform those found by generic auto-tuners in
thirty iterations by up to 2 ×.
1. INTRODUCTION
From the number of machines used in a distributed appli-
cation, to low-level parameters such as compiler ﬂags, man-
aging conﬁgurations has become one of the main challenges
faced by users of modern systems [6]. Recently, auto-tuning
has successfully been applied to address this challenge in
a range of areas [26, 3, 4]. Users employ it by exposing
their conﬁguration parameters and a performance metric of
their application to a black box optimizer which will evalu-
ate many conﬁgurations. A popular auto-tuning technique
is Bayesian optimization [5], which uses the results to in-
crementally build a probabilistic model of the impact of the
parameters on performance. This allows the optimization to
quickly focus on eﬃcient regions of the conﬁguration space.
Unfortunately, for many systems, either the conﬁguration
space is too large to develop a good model, or the time to
evaluate performance is too long to be executed many times.
©2017 International World Wide Web Conference Committee (IW3C2),
published under Creative Commons CC BY 4.0 License.
WWW 2017,April 3–7, 2017, Perth, Australia.
ACM 978-1-4503-4913-0/17/04.
http://dx.doi.org/10.1145/3038912.3052662
.
The problem addressed in this paper is automatically ﬁnd-
ing eﬃcient conﬁgurations in situations where black box op-
timizers fail. We present BOAT, a framework that allows a
system developer to build a BespOke Auto-Tuner for their
system. The key idea of this work is to allow developers to
provide contextual information in the form of a probabilis-
tic model reﬂecting the system’s behavior. At BOAT’s core
is structured Bayesian optimization (SBO), our extension
of the Bayesian optimization algorithm capable of leverag-
ing these probabilistic models to make informed decisions
about which conﬁgurations to evaluate. Our approach dras-
tically reduces the number of iterations needed to converge,
making auto-tuning applicable in a range of new areas.
System developers implement probabilistic models using
BOAT’sprobabilistic programminglibrary. Probabilistic pro-
gramming has recently been proposed as an intuitive way to
construct structured probabilistic models [17]. In BOAT,
developers use it to expose their understanding of system’s
executions. Throughout the optimization, runtime measure-
ments from the system are used to perform inference on the
model and make it resemble the true underlying behavior.
For example, in our neural network case study, we built
an auto-tuner which balances a computational load across
a heterogeneous cluster. Our associated model iteratively
infers the computational power of machines and uses it to
predict a machine’s computation time as a function of its as-
signed load. Similarly, a diﬀerent part of the model predicts
communication time based on the inferred network speed of
each machine. Our contributions are:
•We propose Structured Bayesian Optimization (SBO), a
novel extension of Bayesian optimization capable of lever-
aging bespoke probabilistic models to rapidly converge to
high-performance conﬁgurations (Section 3).
•We present BOAT, a framework to build bespoke auto-
tuners (Section 4). BOAT includes an implementation of
SBO as well as a probabilistic programming library. We
discuss the design of useful probabilistic models in the
context of BOAT (Section 5).
•We demonstrate the use of BOAT through two case stud-
ies. First, a garbage collection case study (3 parameters)
in which we tune the conﬁguration ﬂags of a database to
minimize its tail latency. This is a domain simple enough
for generic auto-tuners to be amenable. We present it here
to show that BOAT brings convergence improvements in
these simple settings and to illustrate the construction
of probabilistic models. Second, a neural network case
study (30+ parameters) in which we optimize the dis-
tributed scheduling of a neural network computation on
a heterogeneous cluster. We show that our bespoke auto-
tuners converge within ten iterations. Our optimized con-
ﬁgurations are up to 2.9 ×faster than simple conﬁgura-
tions an application user may have selected, and 2×faster
than the ones found by generic auto-tuners after thirty it-
erations (Section 6).
2. MOTIV ATION
2.1 The need for auto-tuning in systems
Due to the diversity of workloads, modern systems have no
“one size ﬁts all” approaches to best execute their computa-
tion. They therefore expose many conﬁguration parameters
which users must manually tune to maximize performance.
This is a diﬃcult task as it requires an understanding of
the underlying system execution. Furthermore, changes in
the hardware or the workload being executed will aﬀect the
system’s behavior thus requiring new adjustments.
In this paper, we consider the task of automatically tun-
ing conﬁguration parameters associated with system perfor-
mance. This can include any parameter for which the opti-
mal value is dependent on the hardware or software context,
ranging from low level parameters such as compiler ﬂags or
conﬁguration ﬁles, to higher level ones like the number of
machines used in a computation.
Ideally, generic auto-tuners could be applied to these tasks
and determine high performance conﬁgurations in reason-
able time. In practice, this is rarely the case for two rea-
sons. First, measuring performance is often expensive. In
distributed systems, it will take at least a few minutes to get
an accurate measure of the “objective function” being tuned.
This discards the evolutionary or hill climbing approaches
used by auto-tuners like Petabricks [3] or OpenTuner [4], as
they usually require thousands of evaluations.
Second, the conﬁguration space of most real world prob-
lems is too complex. For example, in our neural network
case study, the computation is scheduled on up to ten ma-
chines. The conﬁguration space includes at least three pa-
rameters per machine, for a total of over thirty parameters.
This is too large for the traditional Bayesian optimization
approach which is used by optimizers like Spearmint [26] and
SMAC [19]. Bayesian optimization tends to require fewer it-
erations to converge than other optimizers, but fails to work
in problems with many dimensions (more than 10) [25].
2.2 Garbage collection case study
As an example, we brieﬂy present our garbage collection
case study. The goal is to tune the Garbage collection (GC)
ﬂags of a Java Virtual Machine (JVM) based database to
minimize its 99th percentile latency. We tune three param-
eters of the Concurrent Mark Sweep (CMS) collector: the
young generation size and survivor ratio ﬂags, which gov-
ern the size of the diﬀerent sections of the heap, and the
max tenuring threshold which sets the rate at which ob-
jects are promoted between heap sections. We measure the
99th percentile latency of Cassandra [28], a popular JVM-
based wide-column store, using the YCSB cloud benchmark-
ing framework [11] with a variety of workloads. The details
and results of our experiments are presented in section 6.1.
The behavior of the garbage collection has a high impact
on the latency. In one context, setting appropriate values of
these three ﬂags reduces the 99th percentile latency from
19ms using Cassandra’s default values, to 7ms. This is
Algorithm 1 The Bayesian optimization methodology
Input: Objective function f()
Input: Acquisition function α()
1: Initialize the Gaussian process G
2: for i= 1,2,... do
3: Sample point: xt ←arg maxx α(G(x))
4: Evaluate new point: yt ←f(xt)
5: Update the Gaussian process: G←G|(xt,yt)
6: end for
mostly due to minor collectionswhich frequently collect ob-
jects in the young generation section of the heap, provoking
a “stop-the-world” pause which halts the application. Good
conﬁgurations will minimize the average duration of these
collections as well as their total time.
The small domain of this tuning problem means oﬀ-the-
shelf auto-tuners are still applicable. In our evaluation, we
ﬁnd Spearmint converges to good conﬁgurations in 16 iter-
ations, after four hours of auto-tuning time (15 minutes per
evaluation). However, leveraging contextual information can
signiﬁcantly reduce convergence time. Our auto-tuner im-
plemented in BOAT is simple yet converges to within 10% of
the best found performance by the second iteration. We use
it for illustration throughout this paper. Our neural network
case study tackles a more complex tuning problem, with over
thirty dimensions, in which case oﬀ-the-shelf auto-tuners fail
to ﬁnd good values after thirty iterations.
3. STRUCTURED BAYESIAN OPTIMIZATION
3.1 The Bayesian optimization algorithm
We start by reviewing the Bayesian optimization method-
ology. Bayesian optimization [5] attempts to ﬁnd the mini-
mum of some objective functionf(x) where typicallyx ∈RD.
To this end, it incrementally builds a probabilistic model
which reﬂects the current knowledge of the objective func-
tion. Most of the time, a Gaussian process (GP) is used [22].
GPs are a powerful class of models which can be viewed as
an elaborate interpolation tool. Given a dataset of multi-
dimensional inputs X = {x1,x2 ... xt}and corresponding
real valued observation Y = {y1,y2 ...y t}their prediction
for a new input x is normally distributed.
The overall procedure of the optimization is shown in Al-
gorithm 1. It executes three steps each iteration. First, it
performs a numerical optimization to ﬁnd a point in the con-
ﬁguration space which maximizes an acquisition function.
Acquisition functions measure how promising a point x is
based on the distribution predicted by the GP at x. They
trade-oﬀ exploration and exploitation. For example, a pop-
ular acquisition function is the expected improvementwhich
returns the expected value of the improvement brought by
evaluating f(x) over the best value η found so far:
αEI (x) = E(max(0, f(x) −η))
Finding a point xt which maximizes the acquisition function
can be performed using an oﬀ-the-shelf numerical optimiza-
tion algorithm. Second, the optimization measures the out-
put of the expensive objective function at this point. Third,
it updates the Gaussian process with this new measurement.
When compared with other optimization methods, such
as evolutionary algorithms, Bayesian optimization tends to
converge in fewer iterations. This however comes at the
cost of a high overhead per iteration due to the computa-
tional complexity of performing the numerical optimization.
Predicted 
Performance
Configuration
Space
Objective 
Function
Performance &
Runtime properties
Probabilistic 
Program
①
②
③
Figure 1: Procedure of Structured Bayesian Optimization
Recently, Bayesian optimization has successfully been used
to tune the parameters of machine learning programs [26]
and surpass human experts. However, it has so far been
unsuccessful at tackling optimizations in high dimensional
spaces [25]. This can be attributed to two issues which may
occur:
1) The probabilistic model fails to accurately capture the
objective function landscape after a reasonable number
of iterations. This is due to thecurse of dimensionality.
2) The numerical optimization algorithm, used in each
iteration, fails to converge and ﬁnd a promising point.
In this work, we extend Bayesian optimization to tackle
the ﬁrst of these issues. In practice, we found both may
need to be addressed and our neural network case study also
involved improvements to help the numerical optimization
converge, but these are out of the scope of this paper.
3.2 Structured Bayesian optimization
Structured Bayesian optimization (SBO) extends Bayesian
optimization methodology to take advantage of the known
structure of the objective function. It uses a structured prob-
abilistic model, provided by the developer, instead of a sim-
ple Gaussian process. In BOAT, those models are imple-
mented in our probabilistic programming library.
Figure 1 shows the overall procedure of a structured Bayesian
optimization. It is similar to the one of a traditional Bayesian
optimization and performs three steps each iteration: (1) It
looks for a point in the conﬁguration space with high pre-
dicted performance by the probabilistic program. (2) It eval-
uates the best found point using the objective function and
collects runtime measurements (3) It performs inference on
the probabilistic program using the resulting observations.
When compared with traditional Bayesian optimization,
using a bespoke probabilistic models brings two advantages.
First, it captures the user’s understanding of the behavior
of the system. This drastically reduces the number of itera-
tions needed for the model to converge towards the true ob-
jective function. In the context of BOAT, models are imple-
mented in probabilistic programming and can reason about
arbitrary data structures – like regular programs – and re-
produce complex behaviors. For example, the model in our
neural network case study predicts the individual computa-
tion time of each machine in a distributed cluster. The total
time is predicted to be the maximum of individual times plus
a communication cost. It would take many evaluations for a
Gaussian process to accurately model the function max over
multiple inputs, our model does so by default.
Second, using such a model allows us to monitor runtime
properties reﬂected in the model and use them for inference.
For instance, in our garbage collection case study, our model
predicts the number and average duration of minor collec-
tions. After each experiment we parse the garbage collection
logs to observe their true value and use them for inference.
Predicted 
LatencyGC Flags
GC Rate
Model
GC Average 
Duration Model
Latency 
Model
Figure 2: Dataﬂow of our garbage collection model
The next subsection presents an example of a structured
probabilistic model from our garbage collection case study.
3.3 A probabilistic model for GC
The probabilistic model written by the developer should
take as input a conﬁguration and predict its performance.
Initially, developers should use a generic probabilistic model,
eﬀectively running traditional Bayesian optimization, and
observe whether the convergence time is acceptable. If it is
not, BOAT allows developers to incrementally addstructure
to the model to reduce convergence time. Adding structure
is done by making the probabilistic model more similar to
the behavior of the system. In the following, we illustrate
how this was achieved in our garbage collection case study.
As an initial model, we used a Gaussian process. The GP
predicted 99th percentile latencies based on the ﬂag values.
This took many iterations to converge, despite the simplicity
of the problem. To add structure, we included in the model
a notion of rate and average duration of minor collections.
Given ﬂag values, our model predicted both these statistics.
It then predicted the latency as a function of the ﬂag values
and the statistics. The data ﬂow of our model is shown in
Figure 2. When using the model in BOAT, we collected the
true value of these statistics from the GC logs after each
evaluation and used them for inference. Further, we de-
clared how we believed each of the three model components
behaved as a function of their inputs. For example, we no-
ticed the rate of minor collections was inversely proportional
to the size of the eden memory region in the JVM-heap,
which is where objects are initially allocated. This intu-
ition was included in the corresponding model by building a
semi-parametric model(Section 5.1), which can successfully
combine a user-deﬁned trend with empirical data.
In practice, adding only little structure can be suﬃcient to
make the optimization converge in a reasonable time. This
is useful as simpler models are able to adapt to a broad
variety of behaviors, such as widely diﬀerent hardware and
workloads. This allows the construction of bespoke auto-
tuners providing global performance portability.
4. THE BOAT FRAMEWORK
BOAT allows a developer to build a bespoke auto-tuner
for their system via SBO. Figure 3 shows the ﬂow of data to
constuct and use an auto-tuner.
Probabilistic model 
of system behavior
Configuration 
space
Objective function &
runtime measurements
                                     OptimizerProbabilistic 
programming library
Configuration space
properties Preferences
Bespoke system auto-tuner
User
System 
developer
BOAT
Figure 3: Flow of data when using BOAT
0 1000 2000
Vector size
0
1
2
3
4Time (µs)
(a) Parametric (Linear regression)
0 1000 2000
Vector size
0
1
2
3
4Time (µs) (b) Non-parametric (Gaussian process)
0 1000 2000
Vector size
0
1
2
3
4Time (µs)
Ground Truth
Model Observation
Predicted Time (c) Semi-parametric (Combination)
Figure 4: Three models predicting the time to insert an element into a sorted vector after ﬁve observations.
Application users provide two types of arguments speciﬁc
to their application. First, the conﬁguration space proper-
ties. These have an inﬂuence on the set of valid conﬁgu-
rations. In a scheduling problem, this could be the list of
available machines. Second, their preferences. These deﬁne
system performance metrics. For example, a user could spec-
ify to optimize throughput, tail latency, or set the workload
with which the system should be evaluated.
To create a bespoke auto-tuner, a system developer takes
these as input to provide three types of information to BOAT:
1) Conﬁguration space: The domain of the optimization.
2) Objective function & runtime measurements: This
speciﬁes how to evaluate a given conﬁguration. For ex-
ample, this can involve writing conﬁguration values to a
conﬁguration ﬁle and starting a distributed system along
with a benchmark. When a BOAT optimization termi-
nates, it returns the conﬁguration that yielded the best
objective function value.
3) Probabilistic model of system behavior : The con-
textual information which allows BOAT to discard re-
gions of low performance and quickly converge.
The ﬁrst two items are common to all auto-tuners and we
do not discuss them further. The next section discusses the
design of probabilistic models in the context of BOAT.
5. PROBABILISTIC MODELS IN BOAT
Developers build bespoke auto-tuners in BOAT by declar-
ing a probabilistic model of the system’s behavior via BOAT’s
probabilistic programming framework. A probabilistic pro-
gram is similar to a simulator. The programmer implements
code mimicking the behavior of the process being modeled.
The advantage of probabilistic programming is that, in the
presence of empirical data, they can perform inference to
make the simulator’s behavior match the observed one. We
refer the reader to [12] for a review of probabilistic program-
ming and the details of BOAT’s algorithms for probabilistic
inference. This section discusses the design and implemen-
tation of models in BOAT.
There are two key techniques to building useful models.
First, models should be compartmentalized. A model should
consist of a combination of independent components with
each component predicting a single observable value. For
example, the garbage collection model, presented in Sec-
tion 3.3, contains three independent components predicting
the rate and duration of collections and the 99th percentile
latency. This makes models easy to debug. One can com-
pare each component’s predictions with observed values and
diagnose which parts of the model fail to converge. Fur-
thermore, this independence can be exploited by the prob-
abilistic framework. This allows the construction of large
probabilistic model without the need to pay an exponential
inference cost. Section 5.2 shows how such models can be
expressed in BOAT. Second, users should make each compo-
nent a semi-parametric model. We discuss semi-parametric
models, their beneﬁts and their implementation in BOAT in
the next subsection.
5.1 Semi-parametric models
There are two desirable properties a model should have in
the context of SBO:
•It should understand the general trend of the objective
function to avoid exploring low performance regions.
•It should have high precision in the region of the optimum,
to ﬁnd the point with highest performance.
Semi-parametric models, which we now describe, can ful-
ﬁll both properties. They are a combination of parametric
models and non-parametric models. As a running example,
we model the average time needed to insert an element into
a sorted vector as a function of its length. This has complex-
ity O(n) but implementations will have runtimes aﬀected by
cache eﬀects and other hardware properties. Figure 4 com-
pares the predictions of a parametric, non-parametric and
semi-parametric model after observing ﬁve points from the
dataset. The data was obtained using the boost::flat_set
data structure and averaged over a million runs.
Parametric models learn a ﬁxed number of parameters.
For example, simple linear regression typically learns two
parameters, the slope and y-intercept. Parametric models
allow developers to specify the expected behavior of the sys-
tem. In our example, this means specifying that the re-
lationship between length and time is linear and not, for
example, quadratic. They however cannot ﬁt subtleties in
the data. We ﬁt a linear regression to ﬁve data points from
the sorted-vector data in Figure 4a. Although the general
trend is correct, the model fails to ﬁt all of the data points
as they are not strictly linear.
On the other hand, non-parametric models learn an un-
bounded number of parameters that grows with the training
data. For example, in the k-nearest neighbor algorithm each
training example is memorized so it can be used for pre-
diction. Non-parametric models provide no direct way to
specify a general trend. Traditional Bayesian optimization
uses Gaussian processes which are non-parametric models.
We ﬁt a GP to the same ﬁve points from the sorted-vector
data in Figure 4b. It succeeds at ﬁtting all of the data
points, but fails to grasp the overall trend. In the context of
struct GCRateModel : public SemiParametricModel<GCRateModel> {
GCRateModel() {
allocated_mbs_per_sec =
std::uniform_real_distribution<>(0.0, 5000.0)(generator);
// Omitted: also sample the GP parameters
}
double parametric(double eden_size) const {
// Model the rate as inversly proportional to Eden’s size
return allocated_mbs_per_sec / eden_size;
}
double allocated_mbs_per_sec;
};
int main() {
// Example: observe two measurements and make a prediction
ProbEngine<GCRateModel> eng;
eng.observe(0.40, 1024); // Eden: 1024MB, GC rate: 0.40/sec
eng.observe(0.25, 2048); // Eden: 2048MB, GC rate: 0.25/sec
// Print average prediction for Eden: 1536MB
std::cout << eng.predict(1536) << std::endl;
}
Listing 1: The GCRate semi-parametric model.
Bayesian optimization, this can lead to the over exploration
of regions with poor performance. This may seems accept-
able for our sorted vector example, but the number of these
regions grows exponentially with the number of dimensions.
Semi-parametric models combine a parametric model and
a non-parametric one. The non-parametric model is used
to learn the diﬀerence between the parametric model and
the observed data. In Figure 4c, we ﬁt the sorted-vector
data with a semi-parametric model that simply combines the
previous two models. Predictions interpolate all data points,
and correctly keep increasing with larger vector sizes. In the
context of BOAT, non-parametric models require little eﬀort
from developers. Including parametric parts to describe gen-
eral behavior will help the optimization to converge faster
but requires some analysis. In our experience, complex para-
metric models designed in a narrow setting may also fail to
generalize. When building a model’s component in BOAT,
we hence recommend to start with a non-parametric model
and to keep adding structure until the optimization con-
verges in a satisfactory time.
Semi-parametric models in BOAT. Developers extend
a SemiParametricModel class to declare a semi-parametric
model. For example, Listing 1 shows the implementation of
the model predicting the rate of garbage collections from our
GC case study. A SemiParametricModel class must deﬁne
two functions:
•A constructor, which samples the values of the model pa-
rameters from their prior distribution. Two types of pa-
rameters must be sampled. First, the parameters of the
parametric model. For example, Listing 1 samples a sin-
gle parameter representing the rate at which memory is
allocated. Second, the parameters of the Gaussian Process
(omitted in Listing 1) which consist of one scale parameter
for each input dimension and a variance parameter [22].
•A parametric function which, for a given input, returns
the prediction of the parametric model.
The model can then be used by constructing a probabilistic
engine ProbEngine templated on the model class. Data can
be fed into the model via the observe function and predict
returns the model’s predictions in light of this data. The
next subsection shows how BOAT semi-parametric models
can be used in the context of a larger model.
struct CassandraModel :public DAGModel<CassandraModel> {
void model(int ygs, int sr, int mtt){
// Calculate the size of the heap regions
double es = ygs * sr / (sr + 2.0);// Eden space’s size
double ss = ygs / (sr + 2.0); // Survivor space’s size
// Define the dataflow between semi-parametric models
double rate = output("rate", rate_model, es);
double duration = output("duration", duration_model,
es, ss, mtt);
double latency = output("latency", latency_model,
rate, duration, es, ss, mtt);
}
ProbEngine<GCRateModel> rate_model;
ProbEngine<GCDurationModel> duration_model;
ProbEngine<LatencyModel> latency_model;
};
int main() {
CassandraModel model;
// Observe a measurement
std::unordered_map<std::string,double> m;
m["rate"] = 0.40; m["duration"] = 0.15; m["latency"] = 15.1;
int ygs = 5000, sr = 7, mtt = 2;
model.observe(m, ygs, sr, mtt);
/* Prints distributions (mean and stdev) of rate, duration
and latency with a larger young generation size (ygs)*/
std::cout << model.predict(6000, sr, mtt) << std::endl;
// Print corresponding expected improvement of the latency
std::cout << model.expected_improvement(
"latency", 15.1, 6000, sr, mtt) << std::endl;
}
Listing 2: The full Cassandra latency model.
5.2 DAG models
In BOAT, DAGModels are used to concatenate multiple
semi-parametric models into a single global model. Devel-
opers implicitly deﬁne a directed acyclic graph (DAG) of
the ﬂow of data between the semi-parametric components.
Listing 2 shows the implementation of the model used in our
GC case study, already shown diagrammatically in Figure 2.
There are two properties a DAGModel must fulﬁll. First,
all of its semi-parametric components must be included as
members of the class. For example, CassandraModel con-
tains three members, one for each of its components. Sec-
ond, it must deﬁne a model function, specifying the ﬂow of
data through the model. A model function takes as input a
conﬁguration. It then uses the function output to propagate
data through the semi-parametric components and get their
predicted outputs. Each output call is named by passing a
unique string identiﬁer as the ﬁrst argument.
A DAGModel class can be used in a number of ways. First,
observe performs inference on the model components, given
a conﬁguration and a dictionary mapping each output ID to
an associated measured value. The underlying inference ex-
ploits the conditional independence of DAGModels: all com-
ponents can be trained independently given their measured
outputs. Second, the function predict returns the distribu-
tion of each output call for a given conﬁguration. This is
useful to debug a model, we can compare each component’s
prediction with the observed values. Third, one can compute
the expected improvement of a conﬁguration over a previ-
ously achieved result. Recall from Section 3.1 that expected
improvement is an acquisition function used in Bayesian op-
timization. In SBO, each iteration of the optimization, we
select the conﬁguration which maximizes the expected im-
provement predicted by our structured model.
A B D
YCSB core workload
0
5
10
15
20
2599th Percentile Latency (ms)
Cassandra default
Optimized
Figure 5: Results for YCSB workloads A, B and D.
6. EV ALUATION
We demonstrate the use of BOAT through our two case
studies: 1) the garbage collection case study in which we
tune conﬁguration ﬂags to minimize tail latency, and 2) the
neural network case study in which we tune the scheduling
of the training of a neural networks on a distributed sys-
tem. In both, we implemented a probabilistic model of the
underlying system behavior, and used it within BOAT to
optimize the system conﬁguration in a range of setting. Our
evaluation focuses on quantifying two properties:
1. The beneﬁts of auto-tuning. Showing that one-size-
ﬁts-all conﬁgurations yield sub-optimal performances.
2. The need for a bespoke auto-tuner. Showing that
our auto-tuners reduce convergence time when compared
to oﬀ-the-shelf optimizers. We compare our performance
with OpenTuner [4] which dynamically adapts its op-
timization algorithm, and Spearmint [26] which imple-
ments traditional Bayesian optimization.
6.1 Garbage collection
We start by presenting the results of our garbage collec-
tion (GC) case study, as introduced in Section 2.2.
Conﬁguration space. We tune the young generation size,
survivor ratioand max tenuring thresholdﬂags, of the CMS
collector, which is used by default by Cassandra.
Objective function. We conﬁgured a single 8 core node to
run Cassandra [28] with a 8 GB ﬁxed heap space to model
a medium-sized web application. We measure the latency
using the YCSB [11] cloud benchmarking framework on a
24 core machine co-located in the same network. Each ex-
periment was run for 15 minutes.
Model. Our probabilistic model, introduced in Section
3.3, is composed of three semi-parametric models: we pre-
dict the rate and average duration of minor collections and
their impact on latency. Our analysis showed that the fre-
quency at which major collections occurred was too low to
have an impact on 99th percentile latency. The GC rate
model was described in Section 3.3. We further found that
the duration of minor GCs tends to increase with the size
of the eden heap region and the max tenuring threshold pa-
rameter. The GC duration model uses this intuition in its
parametric part. The 99th percentile latency tends to be af-
fected by two properties of GCs: their average duration and
the fraction of time spent in GCs. The parametric part of
the latency model includes linear penalties for each of these
two quantities. These models are too simplistic to capture
the full underlying behavior of the computation, but they
do grasp the overall trend. This is suﬃcient to make the
0 5 10 15 20 25 30
Iteration
0
5
10
15
20Best 99th percentile latency (ms)
OpenTuner
Spearmint
BOAT
Figure 6: Convergence of the frameworks on workload B.
BOAT-based optimizer rapidly converge towards high per-
formance areas.
Results. We ran our bespoke auto-tuner for 10 iterations
on YCSB workloads A (50% reads, 50% updates), B (95%
reads, 5% updates) and D (95% reads, 5% inserts) (work-
load C has 100% reads and is not GC-sensitive). After each
optimization, we re-evaluated our optimized conﬁguration
and compared its 99th percentile latency with the default
Cassandra conﬁguration. The results are shown in Figure 5.
Error bars are too small to be displayed in the ﬁgure as stan-
dard deviations were consistently below 1 ms (all results
averaged over 3 runs). Our optimized conﬁgurations out-
performs the Cassandra default conﬁguration by up to 63%.
Although we run the optimization for 10 iterations, each
optimization converges to within 10% of the best found per-
formance by the second iteration.
We found that the optimized conﬁguration used large eden
size, making minor collections longer but less frequent. Af-
ter inspection we noted that this eﬀectively improved the
batching of the collection, reducing the total work. Opti-
mized conﬁgurations spent well under 1% of their time in
stop the world phases, whereas in Cassandra default conﬁg-
uration’s case this was around 4%.
Comparison with other auto-tuners. Our previous re-
sults show tuning does yield performance improvements for
our workloads. We now consider whether generic auto-tuners
would be able to yield similar performance in the same
timescale. Figure 6 compares our performance with Open-
Tuner [4] and Spearmint [26] which we ran for thirty itera-
tions. We run each optimization three times. For each iter-
ation, we report the median, min and max of the best 99th
percentile latency achieved so far. We see that within two
iterations, our auto-tuner consistently ﬁnds a high perfor-
mance conﬁguration. In contrast, it is only at the 16th iter-
ation that one of the other framework’s median value reaches
a good performance, after four hours of optimization.
6.2 Neural networks
We now present our neural network case study. Neu-
ral networks have seen a surge of interest in recent years,
and many frameworks have been proposed to facilitate their
training. Here, we built a tuner on top of TensorFlow, a re-
cent framework for distributed machine learning [1, 2]. The
API oﬀered by TensorFlow to machine learning applications
is low-level. Users must manually set which of their available
machines should be used and how much work each should
do. TensorFlow oﬀers no automated approach to balance
workloads. This task is especially diﬃcult in heterogeneous
settings, where the optimal load of a machine depends on its
Neural Network name Input Type Network Type Size (MB) Ops (Millions) Batch size range
GoogleNet [27] Image Convolutional 26.7 1582 26 − 29
AlexNet [20] Image Convolutional 233 714 28 − 211
SpeechNet [24] Audio Perceptron 173 45.3 213 − 216
Table 2: The three neural networks used in our experiments. Size is the size of the parameters which must be transmitted
to and from workers each iteration. Ops is the number of ﬂoating point multiplications per input. The name “SpeechNet” is
introduced by us for clarity, this network was recently proposed for benchmarking [9].
computational power. Further, the synchronization cost of
machines can be high, and hence the slowest workers should
not be used at all.
Using BOAT, we built a bespoke auto-tuner that balances
a TensorFlow workload. Our tuner takes as input a neural
network architecture, a set of available machines and a batch
size (an algorithmic parameter described in the next subsec-
tion). It then performs ten iterations, each time evaluating
a distributed conﬁguration, before returning the one with
highest performance. The tuning always ﬁnishes within two
hours which is small compared to the typical training time
of neural networks. The next subsection gives a background
of the computation used to train neural networks in a dis-
tributed setting.
6.2.1 Distributed training of Neural Networks
Stochastic gradient descent. Neural networks are typi-
cally trained with backpropagation using stochastic gradient
descent (SGD). Each iteration, a random batch of samples
from the training set is drawn. The number of samples is
called the batch size. Using each sample independently, an
estimate of the gradient of the parameters of the network
with regard to a loss function is computed using backprop-
agation. Gradient estimates are aggregated and used to up-
date the neural network parameters.
Note that higher batch sizes lead to more parallelism, but
lower batch sizes can result in better accuracy of the ﬁnal
neural network [20]. We cannot quantify in advance the
impact of the batch size on accuracy, as it depends on the
training data, but our experiments expose the trade-oﬀ be-
tween batch size and computational speed.
Distributed SGD. The parameter server architecture [13]
typically used for distributed SGD uses two types of tasks:
•Parameter Server tasks synchronize the gradients at
every iteration and update the parameters. Each param-
eter server task is associated with a section of the neural
network parameters, e.g. the ﬁrst layer.
•Workertasks compute the gradient estimates. Each worker
is assigned a set of inputs. Each iteration, the worker
fetches the updated parameter values from the parame-
ter servers. It then computes the gradient estimate using
the inputs it has been assigned. Finally, it sends these
gradients back to the relevant parameter server.
Typically, stochastic gradient descent is implemented syn-
chronously. A barrier after each iteration forces workers to
work on the same parameter values. Some systems imple-
ment asynchronous SGD [23] which removes this barrier and
lets workers compute gradients on stale parameter versions.
This improves computational performance, especially with
large numbers of workers, but can hurt convergence and de-
crease the ﬁnal result quality [8]. In this case study, we only
consider the synchronous version.
Instance Type # Hyperthread GPU # per setting
A B C
g2.2xlarge 8 1 K520 0 1 2
c4.2xlarge 8 / 6 3 2
c4.4xlarge 16 / 2 3 2
c4.8xlarge 36 / 2 3 4
Total 10 10 10
Table 1: Machine and setting speciﬁcations.
6.2.2 Tuning distributed SGD
Conﬁguration Space. We tuned the scheduling of a pa-
rameter server architecture, implemented in TensorFlow, to
minimize the average iteration time. Given a set of ma-
chines, a neural network architecture and a batch size, we set:
•Which subset of machines should be used as workers.
•Which (possibly overlapping) subset of machines should
be used as parameter servers.
•Among working machines, how to partition the workload.
For machines with GPUs, this includes the partition of
workload between their CPU and GPUs. Each device gets
assigned a number of inputs to process per iteration. The
total number of inputs must sum up to the batch size.
There are eﬀectively two boolean conﬁguration parameters
per machine setting whether it should be a worker and/or
a parameter server and one to two integer parameters per
machine, depending on whether it has a GPU, specifying the
load. In our experiments, we tune the scheduling over 10 ma-
chines, setting 30-32 parameters. In our largest experiment,
there are approximately 10 53 possible valid conﬁgurations,
most of which lead to poor performance.
Note that we are only tuning system parameters, the com-
putation performed will be the same independently of the
conﬁguration used. In particular, the conﬁguration selected
does not aﬀect accuracy.
Objective function. To measure the performance of a set-
ting, we performed twenty iterations of stochastic gradient
descents. The ﬁrst few iterations often showed a high vari-
ance in performance and hence we report the average time of
the last 10 iterations. We found this was enough to get ac-
curate measurements, repeating conﬁgurations showed little
underlying noise.
6.2.3 Probabilistic model of distributed SGD
The probabilistic model contains several components.
Individual device computation time. For each device –
CPU or GPU – on a worker machine, we modeled the time
needed to perform its assigned workload. This time should
be near linear with respect to the number of inputs, hence we
used a semi-parametric model similar to the one of Section
5.1 with a linear parametric model. We ﬁt one individual
device model per type of device available (e.g. c4.4xlarge
CPU, or Nvidia GPU K520).
1.0
2.0
4.0
8.0
16.0
10 10 10 10
7.81
Setting A
GoogleNet
3 4 6 10
3.62
AlexNet
4 8 10 10
0.11 Normalization
factor (ms)
SpeechNet
1.0
2.0
4.0
8.0
16.0
Normalized time per input
10 10 10 10
5.88
Setting B
 1 1 4 5
2.95
Uniform
GPUs
Uniform
Devices
-
Optimized
Configuration
4 5 7 10
0.08
w
# Workers
26 27 28 29
1.0
2.0
4.0
8.0
16.0
10 10 10 10
5.35
Setting C
28 29 210 211
Batch size (# of inputs)
1 2 2 2
2.39
213 214 215 216
2 6 6 10
0.07
Figure 7: Normalized time per input (lower is better) of simple and optimized conﬁgurations on each experiment. Within each
sub-graph, results are normalized by the best achieved time per input. This is always the one of the optimized conﬁguration on
the largest batch size (the lower right point of each sub-graph). The normalization factor, i.e. the best time per input, is shown
at the top right of each sub-graph in milliseconds. For each optimized conﬁguration, we report the number of workers used.
Individual machine computation time. For worker ma-
chines with multiple devices, the gradient estimates from
each device were summed locally on the CPU before being
sent to the parameter servers. We modeled the total compu-
tation time per machine as a semi-parametric model. The
parametric part returned the maximum computation time
of the machine’s devices. The non-parametric part was re-
sponsible for modeling this aggregation time. We ﬁt one
individual machine model per type of machine (e.g. EC2
instance type).
Communication time. We modeled the communication
time as another semi-parametric model. Our parametric
model learns a connection speed parameter per type of ma-
chine. It predicts the total communication time as
max
m ∈ machines
transfer(m)
connection speedm
where transfer(m) is the amount of data that must be trans-
fered each iteration by machinem. It is a function of whether
m is a worker, the number of other workers, and the size of
the parameters m holds as a parameter server if any. We
ﬁt a single communication time model for the entire cluster.
Finally, we predict the total time of an SGD iteration as the
sum of the maximum predicted individual machine time and
the communication time.
Since our probabilistic model simulates individual device
and machine computation times, it beneﬁts from real mea-
surements of these properties. We therefore also measure in
each iteration the time needed by all devices and machines
to perform their assigned workload.
6.2.4 Experiment results
Experimental Setup. We evaluated our optimizer on
Amazon EC2 using TensorFlow version v0.8. There are
three inputs to our tuning procedure. The machines avail-
able, the neural network being trained and the batch size.
We constructed three machine settings, described in Table 1,
designed to recreate heterogeneous environments. Each con-
tains 10 machines of varying computational power. Settings
B and C contain one and two GPU instances respectively.
While neural networks perform most eﬃciently on GPUs, we
tried to design realistic settings where a variety of CPUs and
GPUs are available. We evaluated each of the three hard-
ware setting with the three neural networks referenced in
Table 2 using four batch sizes for a total of thirty-six exper-
iments. The four batch sizes for each network were selected
to explore the tradeoﬀ with processing speed. Recall that
batch size is an algorithmic parameter equaling inputs per
iteration, and that lower batch sizes tend to improve ﬁnal
result accuracy at the cost of less parallelism.
Comparison with simple conﬁgurations. To show the
importance of tuning, we compared our optimized conﬁgu-
rations with two simple conﬁgurations 1) Uniform Devices:
a load balanced equally among all devices, and 2) Uniform
GPUs: a load balanced equally among GPUs (in Settings
B and C). In both cases, we set worker machines to also be
parameter servers which tends to yield good results. Figure
7 shows the outcome of each experiment. Our optimized
conﬁgurations signiﬁcantly outperform these simple conﬁg-
urations on most experiments.
Inspecting the optimized conﬁgurations and their associ-
ated models delivers a number of insights.
•Communication cost. Large networks, like AlexNet,
are often scheduled on a subset of the machines due to
their expensive communication cost. This is lessened with
larger batch sizes where there is more computation to
perform per iteration and hence the cost of using more
workers is amortized. On the other hand, the smaller
GoogleNet was always scheduled on all available machines.
•Parameter servers. Another key setting that must be
optimized, which we do not report here due to lack of
space, is the set of parameter server machines. Param-
0 5 10 15 20 25 30
Iteration
10
100
5
50
Best SGD iteration time (s)
OpenTuner
Spearmint
BOAT
Figure 8: Convergence of the frameworks on Setting C using
SpeechNet with a 2 16 batch size.
eter servers need to perform large amounts of communi-
cation and hence beneﬁt from being placed on machines
with high network speed. In the optimized conﬁgurations,
most of the parameters were usually placed onc4.8xlarge
instances, which indeed have the largest bandwidth. In-
specting the learned communication models shows that
they had correctly inferred that the connection speed pa-
rameter of c4.8xlarge was higher than those of other
instances.
•Load balance. Comparing “Uniform Devices” conﬁgu-
rations with our optimized conﬁgurations on GoogleNet
shows the importance of load balancing, both used all
available devices but the optimized conﬁgurations are sig-
niﬁcantly faster. Interestingly, the correct load balance
depends on the neural network architecture. AlexNet’s
optimized conﬁgurations had a higher proportion of work
assigned to GPUs than those of SpeechNet.
•Batch size tradeoﬀ. Recall that batch size is an algo-
rithmic parameter and lower batch sizes tend to produce
better accuracy of the ﬁnal neural network at the cost of
less parallelism. Each sub-graph of Figure 7 shows the
tradeoﬀ between processing rate and batch size. With
our auto-tuner, users can ﬁnd optimized conﬁgurations for
diﬀerent batch sizes and easily explore this Pareto-front,
hiding the details of the conﬁguration used.
These observations conﬁrm the intuition that “one size ﬁts
all” approaches are not appropriate, as optimized conﬁgura-
tions are inﬂuenced by hardware, workload and batch size.
Comparison with traditional Bayesian optimization.
We now consider whether the beneﬁts of auto-tuning could
have been achieved with an oﬀ-the-shelf optimization tool.
Figure 8 compares the performance of our bespoke auto-
tuner with OpenTuner [4] and Spearmint [26], which were
each ran for thirty iterations. Each optimization was run
three times, we report for every iteration the median, min
and max performance of the best conﬁguration found so far.
Our bespoke auto-tuner signiﬁcantly outperforms generic
auto-tuners. The median best conﬁguration achieved by
OpenTuner is 8.71s per SGD iteration, more than twice
slower than our median time (4.31s), and not much faster
than the Uniform GPUs conﬁguration (9.82s). The rea-
son this tuning task is diﬃcult is because the space of ef-
ﬁcient conﬁgurations is extremely narrow, assigning one of
the workers too much work creates a bottleneck, yielding
poor performance.
All of our experiments ﬁnished the ten iterations within
two hours. As neural networks training typically lasts over
a week, the performance gains largely outweigh the tuning
overhead, making our auto-tuner practical in realistic set-
tings. Our largest experiments involved 32 dimensions, we
expect our auto-tuner would scale well to larger settings as
there would be a proportional increase in the number of
measurements.
7. RELATED WORK
System performance modeling. Predicting the perfor-
mance of workloads has received signiﬁcant interest. Most
of the time, this is used online as part of a scheduler to
best execute an incoming workload. Ernest [29] uses para-
metric probabilistic models to predict the performance of
distributed analytics jobs by ﬁrst running them on small
samples of the data. Quasar[15], Paragon [14] and Pro-
teusTM [16] proﬁle the early stages of workloads to ﬁnd
similar previously-scheduled workloads. They use this to
suggest appropriate conﬁgurations. These frameworks use
generic probabilistic models to predict performance. They
work well when few parameters are being tuned but have
diﬃculty scaling to large conﬁguration spaces due to the
curse of dimensionality [21]. BOAT tackles this by using be-
spoke models which are engineered to resemble the system’s
behavior and can leverage high rates of measurements.
Many ﬁned-grained performance predictors have been built
for MapReduce workloads [18]. In comparison, probabilistic
programming allows to build simple models, requiring little
programmer eﬀort, and leverage empirical data to accurately
predict performance. Databases use performance models to
predict the behavior of queries [7]. These perform an op-
timization over a complex conﬁguration space, but rely on
the constrained scope of queries, which only use relational
operators, to make accurate predictions.
System auto-tuning. Auto-tuning is often used to adapt
numerical libraries to their underlying hardware, ATLAS [30]
does this in the context of BLAS libraries. OpenTuner [4]
is a generic auto-tuner which combines multiple optimiza-
tion algorithms. In these works, evaluating a conﬁguration
can be done quickly, and hence the objective function can
be evaluated many times. BOAT makes auto-tuning appli-
cable to new domains where this isn’t the case. Spearmint
[26] and Yelp’s MOE [10] are optimizers implementing tra-
ditional Bayesian optimization. BOAT builds upon them by
allowing the use of bespoke models.
8. CONCLUSION
In this paper we presented BOAT, a framework to build
bespoke auto-tuners in environments where black box opti-
mizers fail. We introduced Structured Bayesian Optimiza-
tion, which enables developers to inject domain knowledge
about the structure of their systems into the optimization
procedure. Evaluation results show how optimizers built
with BOAT can signiﬁcantly outperform traditional auto-
tuners in complex tuning problems. BOAT is open source
and available at https://github.com/VDalibard/BOAT.
Acknowledgements
We thank Felix Gessert, Ionel Gog, Tim Harris, Wenjun
Hu, Paul Kelly, Rafal Mantiuk, Hugo Paquet, Amitabha
Roy, Malte Schwarzkopf and Wolfram Wingerath for oﬀer-
ing feedback on early drafts of this paper. This research was
partly funded by the EPSRC (EP/P004024, EP/M508007/1
and GCRF) and a Computer Laboratory Premium Scholar-
ship (Sansom scholarship).
9. REFERENCES
[1] Mart´ ın Abadi et al. TensorFlow: Large-scale machine
learning on heterogeneous systems, 2015. Software
available from tensorﬂow.org.
[2] Mart´ ın Abadi et al. Tensorﬂow: A system for
large-scale machine learning. In 12th USENIX
Symposium on Operating Systems Design and
Implementation (OSDI 16), pages 265–283, 2016.
[3] Jason Ansel et al. Petabricks: A language and
compiler for algorithmic choice. In Proceedings of the
2009 ACM SIGPLAN Conference on Programming
Language Design and Implementation, PLDI ’09,
pages 38–49, New York, NY, USA, 2009. ACM.
[4] Jason Ansel et al. Opentuner: an extensible framework
for program autotuning. In Proceedings of the 23rd
international conference on Parallel architectures and
compilation, pages 303–316. ACM, 2014.
[5] Eric Brochu, Vlad M Cora, and Nando de Freitas. A
tutorial on Bayesian optimization of expensive cost
functions, with application to active user modeling
and hierarchical reinforcement learning. Technical
Report UBC TR-2009-023, University of British
Columbia, 2009.
[6] Brendan Burns, Brian Grant, David Oppenheimer,
Eric Brewer, and John Wilkes. Borg, Omega, and
Kubernetes. ACM Queue, 14:70–93, 2016.
[7] Surajit Chaudhuri. An overview of query optimization
in relational systems. In Proceedings of the seventeenth
symposium on Principles of database systems, pages
34–43. ACM, 1998.
[8] Jianmin Chen, Rajat Monga, Samy Bengio, and Rafal
Jozefowicz. Revisiting distributed synchronous SGD.
arXiv preprint arXiv:1604.00981, 2016.
[9] Soumith Chintala. Deepmark benchmark.
https://github.com/DeepMark/deepmark.
[10] Scott Clark, Eric Liu, Peter Frazier, JiaLei Wang,
Deniz Oktay, and Norases Vesdapunt. MOE: A global,
black box optimization engine for real world metric
optimization. https://github.com/Yelp/MOE, 2014.
[11] Brian F Cooper, Adam Silberstein, Erwin Tam,
Raghu Ramakrishnan, and Russell Sears.
Benchmarking cloud serving systems with YCSB. In
Proceedings of the 1st ACM symposium on Cloud
computing, pages 143–154. ACM, 2010.
[12] Valentin Dalibard. A framework to build bespoke
auto-tuners with structured Bayesian optimisation.
PhD thesis, University of Cambridge
(UCAM-CL-TR-900), January 2017.
[13] Jeﬀrey Dean et al. Large scale distributed deep
networks. In Advances in Neural Information
Processing Systems 25: 26th Annual Conference on
Neural Information Processing Systems 2012.
Proceedings of a meeting held December 3-6, 2012,
pages 1232–1240, 2012.
[14] Christina Delimitrou and Christos Kozyrakis.
Paragon: QoS-aware scheduling for heterogeneous
datacenters. In ACM SIGPLAN Notices, volume 48,
pages 77–88. ACM, 2013.
[15] Christina Delimitrou and Christos Kozyrakis. Quasar:
resource-eﬃcient and QoS-aware cluster management.
In ACM SIGPLAN Notices, volume 49, pages
127–144. ACM, 2014.
[16] Diego Didona, Nuno Diegues, Anne-Marie Kermarrec,
Rachid Guerraoui, Ricardo Neves, and Paolo Romano.
ProteusTM: Abstraction meets performance in
transactional memory. In Proceedings of the
Twenty-First International Conference on
Architectural Support for Programming Languages and
Operating Systems, pages 757–771. ACM, 2016.
[17] Andrew D. Gordon, Thomas A. Henzinger, Aditya V.
Nori, and Sriram K. Rajamani. Probabilistic
programming. In International Conference on
Software Engineering (ICSE, FOSE track), 2014.
[18] Herodotos Herodotou, Harold Lim, Gang Luo,
Nedyalko Borisov, Liang Dong, Fatma Bilgen Cetin,
and Shivnath Babu. Starﬁsh: A self-tuning system for
big data analytics. In CIDR, volume 11, pages
261–272, 2011.
[19] Frank Hutter, Holger H Hoos, and Kevin
Leyton-Brown. Sequential model-based optimization
for general algorithm conﬁguration. In International
Conference on Learning and Intelligent Optimization,
pages 507–523. Springer, 2011.
[20] Alex Krizhevsky. One weird trick for parallelizing
convolutional neural networks. arXiv preprint
arXiv:1404.5997, 2014.
[21] Kevin P. Murphy. Machine Learning: A Probabilistic
Perspective. The MIT Press, 2012.
[22] Carl Edward Rasmussen. Gaussian processes for
machine learning. 2006.
[23] Benjamin Recht, Christopher Re, Stephen Wright, and
Feng Niu. Hogwild: A lock-free approach to
parallelizing stochastic gradient descent. In Advances
in Neural Information Processing Systems, pages
693–701, 2011.
[24] Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and
Dong Yu. 1-bit stochastic gradient descent and its
application to data-parallel distributed training of
speech dnns. In INTERSPEECH, pages 1058–1062,
2014.
[25] Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P
Adams, and Nando de Freitas. Taking the human out
of the loop: A review of Bayesian optimization.
Proceedings of the IEEE, 104(1):148–175, 2016.
[26] Jasper Snoek, Hugo Larochelle, and Ryan Prescott
Adams. Practical bayesian optimization of machine
learning algorithms. In Neural Information Processing
Systems, 2012.
[27] Christian Szegedy et al. Going deeper with
convolutions. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, pages
1–9, 2015.
[28] The Apache Software Foundation. Apache Cassandra.
http://cassandra.apache.org.
[29] Shivaram Venkataraman, Zongheng Yang, Michael
Franklin, Benjamin Recht, and Ion Stoica. Ernest:
eﬃcient performance prediction for large-scale
advanced analytics. In 13th USENIX Symposium on
Networked Systems Design and Implementation (NSDI
16), pages 363–378, 2016.
[30] R Clint Whaley and Jack J Dongarra. Automatically
tuned linear algebra software. In Proceedings of the
1998 ACM/IEEE conference on Supercomputing,
pages 1–27. IEEE Computer Society, 1998.

Consistent * Complete*W
ellDocumented*EasytoReuse* *
Evaluated*
ACT
*
Artifact *
AEC
P
End-to-end Deep Learning of Optimization Heuristics
Chris Cummins, Pavlos Petoumenos
School of Informatics
University of Edinburgh
United Kingdom
{c.cummins,ppetoume}@inf.ed.ac.uk
Zheng Wang
School of Computing and Communications
Lancaster University
United Kingdom
z.wang@lancaster.ac.uk
Hugh Leather
School of Informatics
University of Edinburgh
United Kingdom
hleather@inf.ed.ac.uk
Abstract—Accurate automatic optimization heuristics are
necessary for dealing with the complexity and diversity of
modern hardware and software. Machine learning is a proven
technique for learning such heuristics, but its success is bound
by the quality of the features used. These features must be
hand crafted by developers through a combination of expert
domain knowledge and trial and error. This makes the quality
of the ﬁnal model directly dependent on the skill and available
time of the system architect.
Our work introduces a better way for building heuristics. We
develop a deep neural network that learns heuristics over raw
code, entirely without using code features. The neural network
simultaneously constructs appropriate representations of the
code and learns how best to optimize, removing the need for
manual feature creation. Further, we show that our neural
nets can transfer learning from one optimization problem to
another, improving the accuracy of new models, without the
help of human experts.
We compare the effectiveness of our automatically generated
heuristics against ones with features hand-picked by experts.
We examine two challenging tasks: predicting optimal mapping
for heterogeneous parallelism and GPU thread coarsening
factors. In 89% of the cases, the quality of our fully auto-
matic heuristics matches or surpasses that of state-of-the-art
predictive models using hand-crafted features, providing on
average 14% and 12% more performance with no human effort
expended on designing features.
Keywords-Optimization Heuristics; Machine Learning; Com-
piler Optimizations; Heterogeneous Systems
I. I NTRODUCTION
There are countless scenarios during the compilation and
execution of a parallel program where decisions must be
made as to how, or if, a particular optimization should be
applied. Modern compilers and runtimes are rife with hand
coded heuristics which perform this decision making. The
performance of parallel programs is thus dependent on the
quality of these heuristics.
Hand-written heuristics require expert knowledge, take
a lot of time to construct, and in many cases lead to
suboptimal decisions. Researchers have focused on machine
learning as a means to constructing high quality heuristics
that often outperform their handcrafted equivalents [1–5].
A predictive model is trained, using supervised machine
learning, on empirical performance data and important quan-
tiﬁable properties, or features, of representative programs.
RuntimesRuntimes
Feature 
Extractor
FeaturesFeaturesFeatures
ProgramsProgramsPrograms Predictive 
Model
ProgramsProgramsTraining 
DataRuntimesDriver
(a) Current state-of-practice
RuntimesRuntimesProgramsProgramsPrograms Predictive 
Model
ProgramsProgramsTraining 
DataRuntimesDriver
(b) Our proposal
Figure 1: Building a predictive model. The model is originally
trained on performance data and features extracted from the source
code and the runtime behavior. We propose bypassing feature
extraction, instead learning directly over raw program source code.
The model learns the correlation between these features and
the optimization decision that maximizes performance. The
learned correlations are used to predict the best optimization
decisions for new programs. Previous works in this area were
able to build machine learning based heuristics with less
effort, that outperform ones created manually experts [6, 7].
Still, experts are not completely removed from the de-
sign process, which is shown in Figure 1a. Selecting the
appropriate features is a manual undertaking which requires
a deep understanding of the system. The designer essentially
decides which compile or runtime characteristics affect
optimization decisions and expresses them in ways that
make it easy to model their relationship to performance.
Failing to identify an important feature has a negative effect
on the resulting heuristic. For example, in [8] the authors
discovered that [6] did not identify one such feature, causing
performance to be 40% lower on average.
To make heuristic construction fast and cheap, we must
take humans out of the loop. While techniques for automatic
feature generation from the compiler IR have been proposed
in the past [9, 10], they do not solve the problem in a
practical way. They are deeply embedded into the compiler,
require expert knowledge to guide the generation, have to
be repeated from scratch for every new heuristic, and their
search time can be prohibitive. Our insight was that such
costly approaches are not necessary any more. Deep learning
techniques have shown astounding successes in identifying
complex patterns and relationships in images [11, 12], au-
dio [13], and even computer code [14, 15]. We hypothesized
that deep neural networks should be able to automatically
extract features from source code. Our experiments showed
that even this was a conservative target: with deep neural
networks we can bypass static feature extraction and learn
optimization heuristics directly on raw code.
Figure 1b shows our proposed methodology. Instead of
manually extracting features from input programs to gen-
erate training data, program code is used directly in the
training data. Programs are fed through a series of neural
networks which learn how code correlates with perfor-
mance. Internally and without prior knowledge, the net-
works construct complex abstractions of the input program
characteristics and correlations between those abstractions
and performance. Our work replaces the need for compile-
time or static code features, merging feature and heuristic
construction into a single process of joint learning. Our
system admits auxiliary features to describe information un-
available at compile time, such as the sizes of runtime input
parameters. Beyond these optional inclusions, we are able
to learn optimization heuristics without human guidance.
By employing transfer learning [16], our approach is able
to produce high quality heuristics even when learning on a
small number of programs. The properties of the raw code
that are abstracted by the beginning layers of our neural net-
works are mostly independent of the optimization problem.
We reuse these parts of the network across heuristics, and,
in the process, we speed up learning considerably.
We evaluated our approach on two problems: heteroge-
neous device mapping and GPU thread coarsening. Good
heuristics for these two problems are important for extracting
performance from heterogeneous systems, and the fact that
machine learning has been used before for heuristic con-
struction for these problems allows direct comparison. Prior
machine learning approaches resulted in good heuristics
which extracted 73% and 79% of the available performance
respectively but required extensive human effort to select the
appropriate features. Nevertheless, our approach was able to
outperform them by 14% and 12%, which indicates a better
identiﬁcation of important program characteristics, without
any expert help. We make the following contributions:
• We present a methodology for building compiler heuris-
tics without any need for feature engineering.
• A novel tool DeepTune for automatically constructing
optimization heuristics without features. DeepTune out-
performs existing state-of-the-art predictive models by
14% and 12% in two challenging optimization domains.
• We apply, for the ﬁrst time, transfer learning on
compile-time and runtime optimizations, improving the
heuristics by reusing training information across differ-
ent optimization problems, even if they are unrelated.
Heuristic 
Model
Auxiliary InputsSource Code
Dense NN
Predicted Optimization
Embedding
Source Rewriter
Sequence Encoder
Batch Normalization
void nbody_sim(__global float4* pos... [256, 2095172]
Device mapping: {CPU,GPU}
Language Model
LSTM
(optional)
(concatenate)
Figure 2: DeepTune architecture. Code properties are extracted
from source code by the language model. They are fed, together
with optional auxiliary inputs, to the heuristic model to produce
the ﬁnal prediction.
II. D EEP TUNE : LEARNING ON RAW PROGRAM CODE
DeepTune is an end-to-end machine learning pipeline for
optimization heuristics. Its primary input is the source code
of a program to be optimized, and through a series of neural
networks, it directly predicts the optimization which should
be applied. By learning on source code, our approach is
not tied to a speciﬁc compiler, platform, or optimization
problem. The same design can be reused to build multiple
heuristics. The most important innovation of DeepTune is
that it forgoes the need for human experts to select and tune
appropriate features.
A. System Overview
Figure 2 provides an overview of the system. A source
rewriter removes semantically irrelevant information (such
as comments) from the source code of the target program and
passes it to a language model. The language model converts
the arbitrary length stream of code into a ﬁxed length vector
of real values which fully capture the properties and structure
of the source, replacing the role of hand designed features.
We then optionally concatenate this vector with auxiliary
inputs, which allow passing additional data about runtime
or architectural parameters to the model for heuristics which
need more than just compile-time information. Finally, a
standard feed-forward network is used to predict the best
heuristic parameters to optimize the program.
DeepTune is open source1. We implemented the model in
Keras, with TensorFlow [17] and Theano [18] backends.
B. Language Model
Learning effective representations of source code is a
difﬁcult task. A successful model must be able to:
• derive semantic and syntactic patterns of a program-
ming language entirely from sample codes;
• identify the patterns and representation in source codes
which are relevant to the task at hand; and
• discriminate performance characteristics arising from
potentially subtle differences in similar codes.
To achieve this task, we employ state-of-the-art language
modeling techniques, coupled with a series of generic,
language agnostic code transformations.
Source Rewriter: To begin with, we apply a series
of source normalizing transformations extended from our
previous work [8]. These transformations, implemented as
an LLVM pass, parse the AST, removing conditional compi-
lation, then rebuild the input source code using a consistent
code style and identiﬁer naming scheme. The role of source
normalization is to simplify the task of modeling source code
by ensuring that trivial semantic differences in programs
such as the choice of variable names or the insertion of
comments do not affect the learned model. Figures 3a and 3b
show the source rewriting applied to a simple program.
Sequence Encoder: We encode source code as a se-
quence of integers for interpretation by neural networks,
where each integer is an index into a predetermined vo-
cabulary. In [8], a character based vocabulary is used. This
minimizes the size of the vocabulary, but leads to long
sequences which are harder to extract structure from. In [19],
a token based vocabulary is used. This leads to shorter
sequences, but causes an explosion in the vocabulary size,
as every identiﬁer and literal must be represented uniquely.
We designed a hybrid, partially tokenized approach. This
allows common multi-character sequences such as float
and if to be represented as unique vocabulary items, while
literals and other infrequently used words are encoded at the
character level.
We ﬁrst assembled a candidate vocabulary Vc for the
OpenCL programming language containing the 208 data
types, keywords, and language builtins of the OpenCL
programming language. We then derived the subset of the
candidate vocabulary V ∈Vc which is required to encode a
corpus of 45k lines of GPGPU benchmark suite kernels. Be-
ginning with the ﬁrst character in the corpus, our algorithm
consumes the longest matching sequence from the candidate
vocabulary. This process continues until every character in
the corpus has been consumed. The resulting derived vocab-
ulary consists of 128 symbols which we use to encode new
program sources. Figure 3c shows the vocabulary derived
for a single input source code Figure 3b.
1DeepTune is available at: https://chriscummins.cc/deeptune
1 / / # d e f i n e E l e m e n t s
2 _ _ k e r n e l void m e m s e t _ k e r n e l (_ _ g l o b a l char ∗ mem_d ,
↪→ s h o r t val , i n t number_bytes ) {
3 c o n s t i n t t h r e a d _ i d = g e t _ g l o b a l _ i d ( 0 ) ;
4 mem_d [ t h r e a d _ i d ] = v a l ;
5 }
(a) An example, short OpenCL kernel, taken from Nvidia’s streamcluster.
1 _ _ k e r n e l void A( _ _ g l o b a l char∗ a , s h o r t b , i n t c ) {
2 c o n s t i n t d = g e t _ g l o b a l _ i d ( 0 ) ;
3 a [ d ] = b ;
4 }
(b) The streamcluster kernel after source rewriting. Variable and function
names are normalized, comments removed, and code style enforced.
idx token idx token idx token
1 ‘__kernel’ 10 ‘,’ 19 ‘const’
2 ‘ ’ 11 ‘short’ 20 ‘d’
3 ‘void’ 12 ‘b’ 21 ‘=’
4 ‘A’ 13 ‘int’ 22 ‘get_global_id’
5 ‘(’ 14 ‘c’ 23 ‘0’
6 ‘__global’ 15 ‘)’ 24 ‘;’
7 ‘char’ 16 ‘{’ 25 ‘[’
8 ‘ *’ 17 ‘\n’ 26 ‘]’
9 ‘a’ 18 ‘ ’ 27 ‘}’
(c) Derived vocabulary, ordered by their appearance in the input (b). The
vocabulary maps tokens to integer indices.
01 02 03 02 04 05 06 02 07 08 02
09 10 02 11 02 12 10 02 13 02 14
15 02 16 17 18 19 02 13 02 20 02
21 02 22 05 23 15 24 17 18 09 25
20 26 02 21 02 12 24 17 27 <pad...>
(d) Indices encoded kernel sequence. Sequences may be padded to a ﬁxed
length by repeating an out-of-vocabulary integer (e.g. -1).
Figure 3: Deriving a tokenized 1-of-k vocabulary encoding from
an OpenCL source code.
Embedding: During encoding, tokens in the vocabulary
are mapped to unique integer values, e.g. float →0, int
→1. The integer values chosen are arbitrary, and offer a
sparse data representation, meaning that a language model
cannot infer the relationships between tokens based on their
mappings. This is in contrast to the dense representations
of other domains, such as pixels in images, which can be
interpolated between to derive the differences in colors.
To mitigate this, we use an embedding, which trans-
lates tokens in a sparse, integer encoded vocabulary into
a lower dimensional vector space, allowing semantically
related tokens like float and int to be mapped to nearby
points [20, 21]. An embedding layer maps each token in the
integer encoded vocabulary to a vector of real values. Given
a vocabulary size V and embedding dimensionality D, an
embedding matrix WE ∈RV ×D is learned during training,
so that an integer encoded sequences of tokens t ∈NL is
mapped to the matrix T ∈RL×D. We use an embedding
dimensionality D= 64.
Sequence Characterization: Once source codes have
been encoded into sequences of embedding vectors, neural
networks are used to extract a ﬁxed size vector which
characterizes the entire sequence. This is comparable to the
hand engineered feature extractors used in prior works, but is
a learned process that occurs entirely — and automatically
— within the hidden layers of the network.
We use the the Long Short-Term Memory (LSTM) archi-
tecture [22] for sequence characterization. LSTMs imple-
ments a Recurrent Neural Network in which the activations
of neurons are learned with respect not just to their current
inputs, but to previous inputs in a sequence. Unlike reg-
ular recurrent networks in which the strength of learning
decreases over time (a symptom of the vanishing gradients
problem [23]), LSTMs employ a forget gate with a linear
activation function, allowing them to retain activations for
arbitrary durations. This makes them effective at learning
complex relationships over long sequences [24], an espe-
cially important capability for modeling program code, as
dependencies in sequences frequently occur over long ranges
(for example, a variable may be declared as an argument to
a function and used throughout).
We use a two layer LSTM network. The network receives
a sequence of embedding vectors, and returns a single output
vector, characterizing the entire sequence.
C. Auxiliary Inputs
We support an arbitrary number of additional real valued
auxiliary inputs which can be optionally used to augment the
source code input. We provide these inputs as a means of
increasing the ﬂexibility of our system, for example, to sup-
port applications in which the optimization heuristic depends
on dynamic values which cannot be statically determined
from the program code [3, 25]. When present, the values
of auxiliary inputs are concatenated with the output of the
language model, and fed into a heuristic model.
D. Heuristic Model
The heuristic model takes the learned representations of
the source code and auxiliary inputs (if present), and uses
these values to make the ﬁnal optimization prediction.
We ﬁrst normalize the values. Normalization is necessary
because the auxiliary inputs can have any values, whereas
the language model activations are in the range [0,1]. If we
did not normalize, then scaling the auxiliary inputs could
affect the training of the heuristic model. Normalization
occurs in batches. We use the normalization method of [26],
in which each scalar of the heuristic model’s inputsx1 ...x n
is normalized to a mean 0 and standard deviation of 1:
x′
i = γi
xi −E(xi)√
Var(xi)
+ βi (1)
where γ and βare scale and shift parameters, learned during
training.
The ﬁnal component of DeepTune is comprised of two
fully connected neural network layers. The ﬁrst layer con-
sists of 32 neurons. The second layer consists of a single
neuron for each possible heuristic decision. Each neuron
applies an activation function f(x) over its inputs. We
use rectiﬁer activation functions f(x) = max(0 ,x) for
the ﬁrst layer due to their improved performance during
training of deep networks [27]. For the output layer, we use
sigmoid activation functions f(x) = 1
1+e−x which provide
activations in the range [0,1].
The activation of each neuron in the output layer repre-
sents the model’s conﬁdence that the corresponding decision
is the correct one. We take the arg max of the output layer
to ﬁnd the decision with the largest activation. For example,
for a binary optimization heuristic the ﬁnal layer will consist
of two neurons, and the predicted optimization is the neuron
with the largest activation.
E. Training the network
DeepTune is trained in the same manner as prior works,
the key difference being that instead of having to manually
create and extract features from programs, we simply use
the raw program codes themselves.
The model is trained with Stochastic Gradient Descent
(SGD), using the Adam optimizer [28]. For training data
X1 ...X n, SGD attempts to ﬁnd the model parameters Θ
that minimize the output of a loss function:
Θ = arg min
Θ
1
n
n∑
i=1
ℓ(Xi,Θ) (2)
where loss function ℓ(x,Θ) computes the logarithmic dif-
ference between the predicted and expected values.
To reduce training time, multiple inputs are batched
together and are fed into the neural network simultaneously,
reducing the frequency of costly weight updates during back-
propagation. This requires that the inputs to the language
model be the same length. We pad all sequences up to a ﬁxed
length of 1024 tokens using a special padding token, allow-
ing matrices of batch_size ×max_seq_len tokens
to be processed simultaneously. We note that batching and
padding sequences to a maximum length is only to improve
training time. In production use, sequences do not need to
be padded, allowing classiﬁcation of arbitrary length codes.
III. E XPERIMENTAL METHODOLOGY
We apply DeepTune to two heterogeneous compiler-based
machine learning tasks and compare its performance to state-
of-the-art approaches that use expert selected features.
A. Case Study A: OpenCL Heterogeneous Mapping
OpenCL provides a platform-agnostic framework for het-
erogeneous parallelism. This allows a program written in
OpenCL to execute transparently across a range of different
devices, from CPUs to GPUs and FPGAs. Given a program
Name Description
F1: data size/(comp+mem) commun.-computation ratio
F2: coalesced/mem % coalesced memory accesses
F3: (localmem/mem)×wgsize ratio local to global mem accesses
×#. work-items
F4: comp/mem computation-mem ratio
(a) Feature values
Name Type Description
comp static #. compute operations
mem static #. accesses to global memory
localmem static #. accesses to local memory
coalesced static #. coalesced memory accesses
data size dynamic size of data transfers
workgroup size dynamic #. work-items per kernel
(b) Values used in feature computation
Table I: Features used by Grewe et al. to predict heterogeneous
device mappings for OpenCL kernels.
and a choice of execution devices, the question then is on
which device should we execute the program to maximize
performance?
State-of-the-art: In [6], Grewe et al. develop a pre-
dictive model for mapping OpenCL kernels to the opti-
mal device in CPU/GPU heterogeneous systems. They use
supervised learning to construct decision trees, using a
combination of static and dynamic kernel features. The static
program features are extracted using a custom LLVM pass;
the dynamic features are taken from the OpenCL runtime.
Expert Chosen Features: Table Ia shows the features
used by their work. Each feature is an expression built upon
the code and runtime metrics given in Table Ib.
Experimental Setup: We replicate the predictive model
of Grewe et al. [6]. We replicated the experimental setup
of [8] in which the experiments are extended to a larger set of
71 programs, summarized in Table IIa. The programs were
evaluated on two CPU-GPU platforms, detailed in Table IIIa.
DeepTune Conﬁguration: Figure 4a shows the neural
network conﬁguration of DeepTune for the task of predicting
optimal device mapping. We use the OpenCL kernel source
code as input, and the two dynamic values workgroup size
and data size available to the OpenCL runtime.
Model Evaluation: We use stratiﬁed 10-fold cross-
validation to evaluate the quality of the predictive mod-
els [29]. Each program is randomly allocated into one of
10 equally-sized sets; the sets are balanced to maintain a
distribution of instances from each class consistent with the
full set. A model is trained on the programs from all but one
of the sets, then tested on the programs of the unseen set.
This process is repeated for each of the 10 sets, to construct
a complete prediction over the whole dataset.
B. Case Study B: OpenCL Thread Coarsening Factor
Thread coarsening is an optimization for parallel programs
in which the operations of two or more threads are fused
together. This optimization can prove beneﬁcial on certain
Version #. benchmarks #. kernels
NPB (SNU [30]) 1.0.3 7 114
Rodinia [31] 3.1 14 31
NVIDIA SDK 4.2 6 12
AMD SDK 3.0 12 16
Parboil [32] 0.2 6 8
PolyBench [33] 1.0 14 27
SHOC [34] 1.1.5 12 48
Total - 71 256
(a) Case Study A: OpenCL Heterogeneous Mapping
Version #. benchmarks #. kernels
NVIDIA SDK 4.2 3 3
AMD SDK 3.0 10 10
Parboil [32] 0.2 4 4
Total - 17 17
(b) Case Study B: OpenCL Thread Coarsening Factor
Table II: Benchmark programs.
Frequency Memory Driver
Intel Core i7-3820 3.6 GHz 8GB AMD 1526.3
AMD Tahiti 7970 1000 MHz 3GB AMD 1526.3
NVIDIA GTX 970 1050 MHz 4GB NVIDIA 361.42
(a) Case Study A: OpenCL Heterogeneous Mapping
Frequency Memory Driver
AMD HD 5900 725 MHz 2GB AMD 1124.2
AMD Tahiti 7970 1000 MHz 3GB AMD 1084.4
NVIDIA GTX 480 700 MHz 1536 MB NVIDIA 304.54
NVIDIA K20c 706 MHz 5GB NVIDIA 331.20
(b) Case Study B: OpenCL Thread Coarsening Factor
Table III: Experimental platforms.
{CPU,GPU}
DNN_2
DNN_1
Normal.
Concat.
LSTM_2
LSTM_1
Embedding
Inputs
wgsize
(a)
dsizecode code
(b)
{1,2,4,8,16,32}
Figure 4: DeepTune neural networks, conﬁgured for (a) heteroge-
neous mapping, and (b) thread coarsening factor. The design stays
almost the same regardless of the optimization problem. The only
changes are the extra input for (a) and size of the output layers.
combinations of programs and architectures, for example
programs with a large potential for Instruction Level Par-
allelism on Very Long Instruction Word architectures.
No …
Source code (CF1)
Feature vector (CF1)
Coarsen?
No Yes
Feature vector (CF2)
Feature vector (CF4)
Coarsen?
No Yes
Coarsen?
CF: 2
CF: 1
CF: 4
Source code (CF2)
Coarsening Pass
Source code (CF4)
Coarsening Pass
(a) Magni et al. cascading binary model.
Source code (CF1)
Coarsening Factor?
CF:1 CF:2 CF:4 CF:8 CF:16 CF:32
(b) Our approach.
Figure 5: Two approaches for predicting coarsening factor (CF) of
OpenCL kernels. Magni et al. reduce the multi-label classiﬁcation
problem to a series of binary decisions, by iteratively applying
the optimization and computing new feature vectors. Our approach
simply predicts the coarsening factor directly from the source code.
State-of-the-art: Magni et al. present a predictive
model for OpenCL thread coarsening in [7]. They implement
an iterative heuristic which determines whether a given
program would beneﬁt from coarsening. If yes, then the
program is coarsened, and the process repeats, allowing
further coarsening. In this manner, the problem is reduced
from a multi-label classiﬁcation problem into a series of
binary decisions, shown in Figure 5a. They select from one
of six possible coarsening factors: (1,2,4,8,16,32), divided
into 5 binary choices.
Expert Chosen Features: Magni et al. followed a very
comprehensive feature engineering process. 17 candidate
features were assembled from previous studies of perfor-
mance counters and computed theoretical values [35, 36].
For each candidate feature they compute its coarsening
delta, reﬂecting the change in each feature value caused by
coarsening: f∆ = ( fafter −fbefore )/fbefore , adding it to
the feature set. Then they use Principle Component Analysis
(PCA) on the 34 candidates and selected the ﬁrst 7 principle
components, accounting for 95% of variance in the space.
Name Description
BasicBlocks #. basic blocks
Branches #. branches
DivInsts #. divergent instructions
DivRegionInsts #. instructions in divergent regions
DivRegionInstsRatio #. instr. in divergent regions / total
instructions
DivRegions #. divergent regions
TotInsts #. instructions
FPInsts #. ﬂoating point instructions
ILP average ILP / basic block
Int/FP Inst Ratio #. branches
IntInsts #. integer instructions
MathFunctions #. match builtin functions
MLP average MLP / basic block
Loads #. loads
Stores #. stores
UniformLoads #. loads unaffected by coarsening direction
Barriers #. barriers
Table IV: Candidate features used by Magni et al. for predicting
thread coarsening. From these values, they compute relative deltas
for each iteration of coarsening, then use PCA for selection.
#. neurons #. parameters
HM CF HM CF
Embedding 64 64 ,256 8,256
LSTM_1 64 64 33,024 33,024
LSTM_2 64 64 33,024 33,024
Concatenate 64 + 2 - - -
Batch Norm . 66 64 264 256
DNN_1 32 32 2,144 2,080
DNN_2 2 6 66 198
Total 76,778 76,838
Table V: The size and number of parameters of the DeepTune
components of Figure 4, conﬁgured for heterogeneous mapping
(HM) and coarsening factor (CF).
Experimental Setup: We replicate the experimental
setup of Magni et al. [7]. The thread coarsening optimization
is evaluated on 17 programs, listed in Table IIb. Four
different GPU architectures are used, listed in Table IIIb.
DeepTune Conﬁguration: Figure 4b shows the neural
network conﬁguration. We use the OpenCL kernel as input,
and directly predict the coarsening factor.
Model Evaluation: Compared to Case Study A, the
size of the evaluation is small. We use leave-one-out cross-
validation to evaluate the models. For each program, a model
is trained on data from all other programs and used to predict
the coarsening factor of the excluded program.
Because [7] does not describe the parameters of the neural
network, we perform an additional, nested cross-validation
process to ﬁnd the optimal model parameters. For every
program in the training set, we evaluate 48 combinations of
network parameters. We select the best performing conﬁgu-
ration from these 768 results to train a model for prediction
on the excluded program. This nested cross-validation is
repeated for each of the training sets. We do not perform
this tuning of hyper-parameters for DeepTune.
C. Comparison of Case Studies
For the two different optimization heuristics, the authors
arrived at very different predictive model designs, with very
different features. By contrast, we take exactly the same
approach for both problems. None of DeepTune’s parameters
were tuned for the case studies presented above. Their
settings represent conservative choices expected to work
reasonably well for most scenarios.
Table V shows the similarity of our models. The only
difference between our network design is the auxiliary inputs
for Case Study A and the different number of optimization
decisions. The differences between DeepTune conﬁgurations
is only two lines of code: the ﬁrst, adding the two auxiliary
inputs; the second, increasing the size of the output layer for
Case Study B from two neurons to six. The description of
these differences is larger than the differences themselves.
IV. E XPERIMENTAL RESULTS
We evaluate the effectiveness of DeepTune for two distinct
optimization tasks: predicting the optimal device to run a
given program, and predicting thread coarsening factors.
We ﬁrst compare DeepTune against two expert-tuned
predictive models, showing that DeepTune outperforms the
state-of-the-art in both cases. We then show that by lever-
aging knowledge learned from training DeepTune for one
heuristic, we can boost training for the other heuristic,
further improving performance. Finally, we analyze the
working mechanism of DeepTune.
A. Case Study A: OpenCL Heterogeneous Mapping
Selecting the optimal execution device for OpenCL
kernels is essential for maximizing performance. For a
CPU/GPU heterogeneous system, this presents a binary
choice. In this experiment, we compare our approach against
a static single-device approach and the Grewe et al. predic-
tive model. The static mapping selects the device which gave
the best average case performance over all the programs. On
the AMD platform, the best-performing device is the CPU;
on the NVIDIA platform, it is the GPU.
Figure 6 shows the accuracy of both predictive models
and the static mapping approach for each of the benchmark
suites. The static approach is accurate for only 58.8% of
cases on AMD and 56.9% on NVIDIA. This suggests the
need for choosing the execution device on a per program
basis. The Grewe et al. model achieves an average accuracy
of 73%, a signiﬁcant improvement over the static mapping.
By automatically extracting useful feature representations
from the source code, DeepTune gives an average accuracy
of 82%, an improvement over both schemes.
Using the static mapping as a baseline, we compute
the relative performance of each program using the device
selected by the Grewe et al. and DeepTune models. Figure 7
shows these speedups. Both predictive models signiﬁcantly
outperform the static mapping; the Grewe et al. model
AMD/uni00A0SDK
NPB
NVIDIA/uni00A0SDK
ParboilPolybenchRodinia SHOC Average
(a)/uni00A0AMD/uni00A0Tahiti/uni00A07970
0%
20%
40%
60%
80%
100%Accuracy
Static/uni00A0mapping Grewe/uni00A0et/uni00A0al.DeepTune
AMD/uni00A0SDK
NPB
NVIDIA/uni00A0SDK
ParboilPolybenchRodinia SHOC Average
(b)/uni00A0NVIDIA/uni00A0GTX/uni00A0970
0%
20%
40%
60%
80%
100%Accuracy
Figure 6: Accuracy of optimization heuristics for heterogeneous
device mapping, aggregated by benchmark suite. The optimal static
mapping achieves 58% accuracy. The Grewe et al. and DeepTune
predictive models achieve accuracies of 73% and 84%, respectively.
achieves an average speedup of 2.91×on AMD and 1.26×
on NVIDIA (geomean 1.18×). In 90% of cases, DeepTune
matches or outperforms the predictions of the Grewe et al.
model, achieving an average speedup of 3.34×on AMD
and 1.41×on NVIDIA (geomean 1.31×). This 14% im-
provement in performance comes at a greatly reduced cost,
requiring no intervention by humans.
B. Case Study B: OpenCL Thread Coarsening Factor
Exploiting thread coarsening for OpenCL kernels is a
difﬁcult task. On average, coarsening slows programs down.
The speedup attainable by a perfect heuristic is only 1.36×.
Figure 8 shows speedups achieved by the Magni et al. and
DeepTune models for all programs and platforms. We use
as baseline the performance of programs without coarsening.
On the four experimental platforms (AMD HD 5900, Tahiti
7970, NVIDIA GTX 480, and Tesla K20c), the Magni
et al. model achieves average speedups of 1.21×, 1.01×,
0.86×, and 0.94×, respectively. DeepTune outperforms this,
achieving speedups of 1.10×, 1.05×, 1.10×, and 0.99×.
Some programs — especially those with large divergent
regions or indirect memory accesses — respond very poorly
to coarsening. No performance improvement is possible
on the mvCoal and spmv programs. Both models fail to
achieve positive average speedups on the NVIDIA Tesla
AMD.BinomialOption
AMD.BitonicSort
AMD.BlackScholes
AMD.FastWalshTransform
AMD.FloydWarshall
AMD.MatrixMultiplication
AMD.MatrixTranspose
AMD.PrefixSum
AMD.Reduction
AMD.ScanLargeArrays
AMD.SimpleConvolution
AMD.SobelFilter
NPB.BT
NPB.CG
NPB.EP
NPB.FT
NPB.LU
NPB.MG
NPB.SP
NVIDIA.DotProduct
NVIDIA.FDTD3d
NVIDIA.MatVecMul
NVIDIA.MatrixMul
NVIDIA.MersenneTwister
NVIDIA.VectorAdd
Parboil.bfs
Parboil.cutcp
Parboil.lbm
Parboil.sad
Parboil.spmv
Parboil.stencil
Polybench.2DConvolution
Polybench.2mm
Polybench.3DConvolution
Polybench.3mm
Polybench.atax
Polybench.bicg
Polybench.correlation
Polybench.covariance
Polybench.gemm
Polybench.gesummv
Polybench.gramschmidt
Polybench.mvt
Polybench.syr2k
Polybench.syrk
Rodinia.backprop
Rodinia.bfs
Rodinia.cfd
Rodinia.gaussian
Rodinia.hotspot
Rodinia.kmeans
Rodinia.lavaMD
Rodinia.leukocyte
Rodinia.lud
Rodinia.nn
Rodinia.nw
Rodinia.particlefilter
Rodinia.pathfinder
Rodinia.streamcluster
SHOC.BFS
SHOC.FFT
SHOC.GEMM
SHOC.MD
SHOC.MD5Hash
SHOC.Reduction
SHOC.S3D
SHOC.Scan
SHOC.Sort
SHOC.Spmv
SHOC.Stencil2D
SHOC.Triad
Average
(a)/uni00A0AMD/uni00A0Tahiti/uni00A07970
1.0x
3.0x
5.0x
7.0x
9.0xSpeedup
Grewe/uni00A0et/uni00A0al.
DeepTune
AMD.BinomialOption
AMD.BitonicSort
AMD.BlackScholes
AMD.FastWalshTransform
AMD.FloydWarshall
AMD.MatrixMultiplication
AMD.MatrixTranspose
AMD.PrefixSum
AMD.Reduction
AMD.ScanLargeArrays
AMD.SimpleConvolution
AMD.SobelFilter
NPB.BT
NPB.CG
NPB.EP
NPB.FT
NPB.LU
NPB.MG
NPB.SP
NVIDIA.DotProduct
NVIDIA.FDTD3d
NVIDIA.MatVecMul
NVIDIA.MatrixMul
NVIDIA.MersenneTwister
NVIDIA.VectorAdd
Parboil.bfs
Parboil.cutcp
Parboil.lbm
Parboil.sad
Parboil.spmv
Parboil.stencil
Polybench.2DConvolution
Polybench.2mm
Polybench.3DConvolution
Polybench.3mm
Polybench.atax
Polybench.bicg
Polybench.correlation
Polybench.covariance
Polybench.gemm
Polybench.gesummv
Polybench.gramschmidt
Polybench.mvt
Polybench.syr2k
Polybench.syrk
Rodinia.backprop
Rodinia.bfs
Rodinia.cfd
Rodinia.gaussian
Rodinia.hotspot
Rodinia.kmeans
Rodinia.lavaMD
Rodinia.leukocyte
Rodinia.lud
Rodinia.nn
Rodinia.nw
Rodinia.particlefilter
Rodinia.pathfinder
Rodinia.streamcluster
SHOC.BFS
SHOC.FFT
SHOC.GEMM
SHOC.MD
SHOC.MD5Hash
SHOC.Reduction
SHOC.S3D
SHOC.Scan
SHOC.Sort
SHOC.Spmv
SHOC.Stencil2D
SHOC.Triad
Average
(b)/uni00A0NVIDIA/uni00A0GTX/uni00A0970
0.0x
1.0x
2.0x
3.0x
4.0xSpeedup
Grewe/uni00A0et/uni00A0al.
DeepTune
Figure 7: Speedup of predicted heterogeneous mappings over the best static mapping for both platforms. In (a) DeepTune achieves an
average speedup of 3.43x over static mapping and 18% over Grewe et al. In (b) the speedup is 1.42x and 13% respectively.
K20c, because thread coarsening does not give performance
gains for the majority of the programs on this platform.
The disappointing results for both predictive models can
be attributed to the small training program set used by
Magni et al. (only 17 programs in total). As a result, the
models suffer from sparse training data. Prior research has
shown that data sparsity can be overcome using additional
programs; in the following subsection we describe and test a
novel strategy for training optimization heuristics on a small
number of programs by exploiting knowledge learned from
other optimization domains.
C. Transfer Learning Across Problem Domains
There are inherent differences between the tasks of build-
ing heuristics for heterogeneous mapping and thread coars-
ening, evidenced by the contrasting choices of features and
models in Grewe et al. and Magni et al. However, in both
cases, the ﬁrst role of DeepTune is to extract meaningful
abstractions and representations of OpenCL code. Prior
research in deep learning has shown that models trained on
similar inputs for different tasks often share useful com-
monalities. The idea is that in neural network classiﬁcation,
information learned at the early layers of neural networks
(i.e. closer to the input layer) will be useful for multiple
tasks. The later the network layers are (i.e. closer to the
output layer), the more specialized the layers become [37].
We hypothesized that this would be the case for Deep-
Tune, enabling the novel transfer of information across
different optimization domains. To test this, we extracted the
language model — the Embedding, and LSTM_{1,2}
layers — trained for the heterogeneous mapping task and
transferred it over to the new task of thread coarsening.
Since DeepTune keeps the same design for both optimization
problems, this is as simple as copying the learned weights
of the three layers. Then we trained the model as normal.
As shown in Figure 8, our newly trained model,
DeepTune-TL has improved performance for 3 of the 4 plat-
binarySearch
blackscholes
convolution
dwtHaar1D
fastWalsh
floydWarshall
mriQ
mt
mtLocal
mvCoal
mvUncoal
nbody
reduce
sgemm
sobel
spmv
stencil
Average
0.0x
0.5x
1.0x
1.5x
2.0x
2.5x
Speedup
AMD Radeon HD 5900
Magni et al.
DeepTune
DeepTune-TL
binarySearch
blackscholes
convolution
dwtHaar1D
fastWalsh
floydWarshall
mriQ
mt
mtLocal
mvCoal
mvUncoal
nbody
reduce
sgemm
sobel
spmv
stencil
Average
0.0x
0.5x
1.0x
1.5x
2.0x
2.5x
AMD Tahiti 7970
Magni et al.
DeepTune
DeepTune-TL
binarySearch
blackscholes
convolution
dwtHaar1D
fastWalsh
floydWarshall
mriQ
mt
mtLocal
mvCoal
mvUncoal
nbody
reduce
sgemm
sobel
spmv
stencil
Average
0.0x
0.5x
1.0x
1.5x
2.0x
2.5x
Speedup
NVIDIA GTX 480
Magni et al.
DeepTune
DeepTune-TL
binarySearch
blackscholes
convolution
dwtHaar1D
fastWalsh
floydWarshall
mriQ
mt
mtLocal
mvCoal
mvUncoal
nbody
reduce
sgemm
sobel
spmv
stencil
Average
0.0x
0.5x
1.0x
1.5x
2.0x
2.5x
NVIDIA Tesla K20c
Magni et al.
DeepTune
DeepTune-TL
Figure 8: Speedups of predicted coarsening factors for each platform. DeepTune outperforms Magni et al on three of the four platforms.
Transfer learning improves DeepTune speedups further, by 16% on average.
forms: 1.17×, 1.23×, 1.14×, 0.93×, providing an average
12% performance improvement over Magni et al. In 81%
of cases, the use of transfer learning matched or improved
the optimization decisions of DeepTune, providing up to a
16% improvement in per platform performance.
On the NVIDIA Tesla K20c, the platform for which no
predictive model achieves positive average speedups, we
match or improve performance in the majority of cases, but
over-coarsening on three of the programs causes a modest
reduction in average performance. We suspect that for this
platform, further performance results are necessary due to
its unusual optimization proﬁle.
D. DeepTune Internal Activation States
We have shown that DeepTune automatically outperforms
state-of-the-art predictive models for which experts have
invested a great amount of time in engineering features. In
this subsection we attempt to illuminate the inner workings,
using a single example from Case Study B: predicting the
thread coarsening factor for Parboil’s mriQ benchmark on
four different platforms.
Figure 9 shows the DeepTune conﬁguration, with visual
overlays showing the internal state. From top to bottom,
we begin ﬁrst with the input, which is the 267 lines of
OpenCL code for the mriQ kernel. This source code is
preprocessed, formatted, and rewritten using variable and
function renaming, shown in Figure 9b. The rewritten source
code is tokenized and encoded in a 1-of-k vocabulary.
Figure 9c shows the ﬁrst 80 elements of this encoded
sequence as a heatmap in which each cell’s color reﬂects
its encoded value. The input, rewriting, and encoding is the
same for each of the four platforms.
The encoded sequences are then passed into the Em-
bedding layer. This maps each token of the vocabulary
to a point in a 64 dimension vector space. Embeddings
are learned during training so as to cluster semantically
Embedding
__kernel void A(int a, int b, __global float* c, __global float* d, __global float* e, __global float* f, __global float* g, __global float4* 
h) {
  float i[4];
  float j[4]; // ...
#include "macros.h"
#define NC  4
#define COARSE_GENERAL
__kernel void
ComputeQ_GPU(int numK, int kGlobalIndex, __global float* x, __global float* y, __global float* z,
             __global float* Qr, __global float* Qi, __global struct kValues* ck) { 
    float sX[NC] // ...
CF: 2 CF: 4 CF: 1CF: 2
Source Code
Dense NN
Predicted Optimization
Source Rewriter
Batch Normalization
LSTM
Language ModelHeuristic 
Model
…
AMD HD 5900
AMD Tahiti 7970
NVIDIA GTX 480
NVIDIA Tesla K20c
AMD HD 5900
AMD Tahiti 7970
NVIDIA GTX 480
NVIDIA Tesla K20c
AMD HD 5900 AMD Tahiti 7970 NVIDIA GTX 480 NVIDIA Tesla K20c
Encoded sequence (ﬁrst 80 tokens)
Outputs of Language Models
Heuristic Models
Learned embeddings (PCA projections)
Sequence Encoder
(a)
(b)
(c)
(d)
(e)
(f)
(g)
Figure 9: Visualizing the internal state of DeepTune when predicting coarsening factor for Parboil’s mriQ benchmark on four different
architectures. The activations in each layer of the four models increasingly diverge the lower down the network.
related tokens together. As such, they may differ between
the four platforms. Figure 9d shows a PCA projection of the
embedding space for one of the platforms, showing multiple
clusters of tokens. By honing in on one of the clusters
and annotating each point with its corresponding token,
we see that the cluster contains the semantically related
OpenCL address space modiﬁers __private, __global,
and __read_only.
Two layers of 64 LSTM neurons model the sequence of
embeddings, with the neuron activations of the second layer
being used to characterize the entire sequence. Figure 9e
shows the neurons in this layer for each of the four plat-
forms, using a red-blue heatmap to visualize the intensity of
each activation. Comparing the activations between the four
platforms, we note a number of neurons in the layer with
different responses across platforms. This indicates that the
language model is partly specialized to the target platform.
As information ﬂows through the network, the layers be-
come progressively more specialized to the speciﬁc platform.
We see this in Figure 9f, which shows the two layers of the
heuristic model. The activations within these increasingly
diverge. The mean variance of activations across platforms
increases threefold compared to the language model, from
0.039 to 0.107. Even the activations of the AMD HD 5900
and AMD Tahiti 7970 platforms are dissimilar, despite the
ﬁnal predicted coarsening factor for both platforms being
the same. In Figure 9g we take the largest activation of the
output layer as the ﬁnal predicted coarsening factor. For this
particular program, a state-of-the-art model achieves 54% of
the maximum performance. DeepTune achieves 99%.
V. R ELATED WORK
Machine learning has emerged as a viable means in auto-
matically constructing heuristics for code optimization [38–
43]. Its great advantage is that it can adapt to changing
hardware platforms as it has no a priori assumptions about
their behavior. The success of machine learning based code
optimization has required having a set of high-quality fea-
tures that can capture the important characteristics of the
target program. Given that there is an inﬁnite number of
these potential features, ﬁnding the right set of features is a
non-trivial, time-consuming task.
Various forms of program features have been used in
compiler-based machine learning. These include static code
structures [44] and runtime information such as system
load [45] and performance counters [46]. In compiler re-
search, the feature sets used for predictive models are often
provided without explanation and rarely is the quality of
those features evaluated. More commonly, an initial large,
high dimensional candidate feature space is pruned via fea-
ture selection [3, 47], or projected into a lower dimensional
space [48, 49]. FEAST employs a range of existing feature
selection methods to select useful candidate features [50].
Unlike these approaches, DeepTune extracts features and
reduces the dimensionality of the feature space completely
internally and without expert guidance.
Park et al. present a unique graph-based approach for
feature representations [51]. They use a Support Vector
Machine where the kernel is based on a graph similarity
metric. Their technique still requires hand coded features
at the basic block level, but thereafter, graph similarity
against each of the training programs takes the place of
global features. Being a kernel method, it requires that
training data graphs be shipped with the compiler, which
may not scale as the size of the training data grows with the
number of instances, and some training programs may be
very large. Finally, their graph matching metric is expensive,
requiring O(n3) to compare against each training example.
By contrast, our method does not need any hand built
static code features, and the deployment memory footprint
is constant and prediction time is linear in the length of the
program, regardless of the size of the training set.
A few methods have been proposed to automatically gen-
erate features from the compiler’s intermediate representa-
tion [9, 10]. These approaches closely tie the implementation
of the predictive model to the compiler IR, which means
changes to the IR will require modiﬁcations to the model.
The work of [10] uses genetic programming to search for
features, and required a huge grammar to be written, some
160kB in length. Although much of this can be created
from templates, selecting the right range of capabilities
and search space bias is non trivial and up to the expert.
The work of [9] expresses the space of features via logic
programming over relations that represent information from
the IRs. It greedily searches for expressions that represent
good features. However, their approach relies on expert
selected relations, combinators and constraints to work. For
both approaches, the search time may be signiﬁcant.
Cavazos et al. present a reaction-based predictive model
for software-hardware co-design [52]. Their approach pro-
ﬁles the targetprogram using several carefully selected
compiler options to see how programruntime changes un-
der these options for a given micro-architecture setting.
Theythen use the program “reactions” to predict the best
available applicationspeedup. While their approach does not
use static code features, developers mustcarefully select
a few settings from a large number of candidate options
forproﬁling, because poorly chosen options can signiﬁcantly
affect the quality ofthe model. Moreover, the program must
be run several times before optimization,while our technique
does not require the program to be proﬁled.
In recent years, machine learning techniques have been
employed to model and learn from program source code on
various tasks. These include mining coding conventions [15]
and idioms [14], API example code [53] and pseudo-code
generation [54], and benchmark generation [8]. Our work is
the ﬁrst attempt to extend the already challenging task of
modeling distributions over source code to learning distri-
butions over source code with respect to code optimizations.
Recently, deep neural networks have been shown to be a
powerful tool for feature engineering in various tasks includ-
ing image recognition [11, 12] and audio processing [13]. No
work so far has applied deep neural networks for program
feature generation. Our work is the ﬁrst to do so.
VI. C ONCLUSIONS
Applying machine learning to compiler and runtime op-
timizations requires generating features ﬁrst. This is a time
consuming process, it needs supervision by an expert, and
even then we cannot be sure that the selected features are
optimal. In this paper we present a novel tool for building
optimization heuristics, DeepTune, which forgoes feature
extraction entirely, relying on powerful language modeling
techniques to automatically build effective representations of
programs directly from raw source code. The result trans-
lates into a huge reduction in development effort, improved
heuristic performance, and more simple model designs.
Our approach is fully automated. Using DeepTune, de-
velopers no longer need to spend months using statistical
methods and proﬁle counters to select program features via
trial and error. It is worth mentioning that we do not tailor
our model design or parameters for the optimization task at
hand, yet we achieve performance on par with and in most
cases exceeding state-of-the-art predictive models.
We used DeepTune to automatically construct heuristics
for two challenging compiler and runtime optimization
problems, ﬁnd that, in both cases, we outperform state-of-
the-art predictive models by 14% and 12%. We have also
shown that the DeepTune architecture allows us to exploit
information learned from another optimization problem to
give the learning a boost. Doing so provides up to a 16%
performance improvement when training using a handful of
programs. We suspect this approach will be useful in other
domains for which training data are a scarce resource.
In future work, we will extend our heuristic construction
approach by automatically learning dynamic features over
raw data; apply unsupervised learning techniques [56] over
unlabeled source code to further improve learned represen-
tations of programs; and deploy trained DeepTune heuristic
models to low power embedded systems using quantization
and compression of neural networks [57].
ACKNOWLEDGMENTS
This work was supported by the UK Engineer-
ing and Physical Sciences Research Council under
grants EP/L01503X/1 (CDT in Pervasive Parallelism),
EP/M01567X/1 (SANDeRs), EP/M015793/1 (DIVIDEND),
and EP/P003915/1 (SUMMER). The code and data for this
paper are available at: https://chriscummins.cc/pact17.
REFERENCES
[1] P. Micolet, A. Smith, and C. Dubach. “A Machine Learning
Approach to Mapping Streaming Workloads to Dynamic
Multicore Processors.” In: LCTES. ACM, 2016.
[2] T. L. Falch and A. C. Elster. “Machine Learning Based
Auto-tuning for Enhanced OpenCL Performance Portabil-
ity.” In: IPDPSW. IEEE, 2015.
[3] M. Stephenson and S. Amarasinghe. “Predicting Unroll
Factors Using Supervised Classiﬁcation.” In: CGO. IEEE,
2005.
[4] F. Agakov, E. Bonilla, J. Cavazos, B. Franke, G. Fursin, M.
O’Boyle, J. Thomson, M. Toussaint, and C. K. I. Williams.
“Using Machine Learning to Focus Iterative Optimization.”
In: CGO. IEEE, 2006.
[5] C. Cummins, P. Petoumenos, M. Steuwer, and H. Leather.
“Autotuning OpenCL Workgroup Size for Stencil Patterns.”
In: ADAPT. 2016.
[6] D. Grewe, Z. Wang, and M. O’Boyle. “Portable Mapping
of Data Parallel Programs to OpenCL for Heterogeneous
Systems.” In: CGO. IEEE, 2013.
[7] A. Magni, C. Dubach, and M. O’Boyle. “Automatic Op-
timization of Thread-Coarsening for Graphics Processors.”
In: PACT. ACM, 2014.
[8] C. Cummins, P. Petoumenos, W. Zang, and H. Leather.
“Synthesizing Benchmarks for Predictive Modeling.” In:
CGO. IEEE, 2017.
[9] M. Namolaru, A. Cohen, G. Fursin, A. Zaks, and A. Freund.
“Practical Aggregation of Semantical Program Properties for
Machine Learning Based Optimization.” In: CASES. 2010.
[10] H. Leather, E. Bonilla, and M. O’Boyle. “Automatic Feature
Generation for Machine Learning Based Optimizing Com-
pilation.” In: TACO 11.1 (2014).
[11] A. Krizhevsky, I. Sutskever, and G. E. Hinton. “ImageNet
Classiﬁcation with Deep Convolutional Neural Networks.”
In: NIPS. 2012.
[12] K. He, X. Zhang, S. Ren, and J. Sun. “Deep Residual
Learning for Image Recognition.” In: CVPR. IEEE, 2016.
[13] H. Lee, Y . Largman, P. Pham, and A. Y . Ng. “Unsupervised
Feature Learning for Audio Classiﬁcation using Convolu-
tional Deep Belief Networks.” In: NIPS. 2009.
[14] M. Allamanis and C. Sutton. “Mining Idioms from Source
Code.” In: FSE. ACM, 2014.
[15] M. Allamanis, E. T. Barr, C. Bird, and C. Sutton. “Learning
Natural Coding Conventions.” In: FSE. ACM, 2014.
[16] J. Yosinski, J. Clune, Y . Bengio, and H. Lipson. “How
Transferable are Features in Deep Neural Networks?” In:
NIPS. 2014.
[17] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean,
M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur,
J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner,
P. Tucker, V . Vasudevan, P. Warden, M. Wicke, Y . Yu, and
X. Zheng. “TensorFlow: A system for large-scale machine
learning.” In: arXiv:1605.08695 (2016).
[18] J. Bergstra, F. Bastien, O. Breuleux, P. Lamblin, R. Pascanu,
O. Delalleau, G. Desjardins, D. Warde-Farley, I. Goodfel-
low, A. Bergeron, and Y . Bengio. “Theano: Deep Learning
on GPUs with Python.” In: BigLearning Workshop. 2011.
[19] M. Allamanis and C. Sutton. “Mining Source Code Reposi-
tories at Massive Scale using Language Modeling.” In:MSR.
2013.
[20] T. Mikolov, K. Chen, G. Corrado, and J. Dean. “Distributed
Representations of Words and Phrases and their Composi-
tionality.” In: NIPS. 2013.
[21] M. Baroni, G. Dinu, and G. Kruszewski. “Don’t Count,
Predict! A Systematic Comparison of Context-Counting vs
. Context-Predicting Semantic Vectors.” In: ACL. 2014.
[22] S. Hochreiter and J. Schmidhuber. “Long Short-Term Mem-
ory.” In: Neural Computation 9.8 (1997).
[23] R. Pacanu, T. Mikolov, and Y . Bengio. “On the Difﬁculties
of Training Recurrent Neural Networks.” In: ICML. 2013.
[24] Z. C. Lipton, J. Berkowitz, and C. Elkan. “A Critical Review
of Recurrent Neural Networks for Sequence Learning.” In:
arXiv:1506.00019 (2015).
[25] Y . Ding, J. Ansel, K. Veeramachaneni, X. Shen, U. O’Reilly,
and S. Amarasinghe. “Autotuning Algorithmic Choice for
Input Sensitivity.” In: PLDI. ACM, 2015.
[26] S. Ioffe and C. Szegedy. “Batch Normalization: Accelerating
Deep Network Training by Reducing Internal Covariate
Shift.” In: arXiv:1502.03167 (2015).
[27] V . Nair and G. E. Hinton. “Rectiﬁed Linear Units Improve
Restricted Boltzmann Machines.” In: ICML. 2010.
[28] D. P. Kingma and J. L. Ba. “Adam: a Method for Stochastic
Optimization.” In: ICLR (2015).
[29] J. Han, M. Kamber, and J. Pei. Data mining: concepts and
techniques. Elsevier, 2011.
[30] S. Seo, G. Jo, and J. Lee. “Performance Characterization
of the NAS Parallel Benchmarks in OpenCL.” In: IISWC.
IEEE, 2011.
[31] S. Che, M. Boyer, J. Meng, D. Tarjan, J. W. Sheaffer,
S. H. Lee, and K. Skadron. “Rodinia: A Benchmark Suite for
Heterogeneous Computing.” In: IISWC. IEEE, Oct. 2009.
[32] J. A. Stratton, C. Rodrigues, I. Sung, N. Obeid, L. Chang,
N. Anssari, G. D. Liu, and W. W. Hwu. “Parboil: A Revised
Benchmark Suite for Scientiﬁc and Commercial Throughput
Computing.” In: Center for Reliable and High-Performance
Computing (2012).
[33] S. Grauer-Gray, L. Xu, R. Searles, S. Ayalasomayajula, and
J. Cavazos. “Auto-tuning a High-Level Language Targeted
to GPU Codes.” In: InPar. 2012.
[34] A. Danalis, G. Marin, C. McCurdy, J. S. Meredith, P. C.
Roth, K. Spafford, V . Tipparaju, and J. S. Vetter. “The
Scalable HeterOgeneous Computing (SHOC) Benchmark
Suite.” In: GPGPU. ACM, 2010.
[35] A. Magni, C. Dubach, and M. O’Boyle. “A Large-Scale
Cross-Architecture Evaluation of Thread-Coarsening.” In:
SC. 2013.
[36] J. Sim, A. Dasgupta, H. Kim, and R. Vuduc. “A Performance
Analysis Framework for Identifying Potential Beneﬁts in
GPGPU Applications.” In: PPoPP. ACM, 2012.
[37] M. D. Zeiler and R. Fergus. “Visualizing and Understanding
Convolutional Networks.” In: ECCV. 2014.
[38] Z. Wang and M. O’Boyle. “Partitioning Streaming Paral-
lelism for Multi-cores: A Machine Learning Based Ap-
proach.” In: PACT. ACM, 2010.
[39] S. Kulkarni and J. Cavazos. “Mitigating the Compiler Opti-
mization Phase-Ordering Problem using Machine Learning.”
In: OOPSLA. ACM, 2012.
[40] S. Muralidharan, A. Roy, M. Hall, M. Garland, and P. Rai.
“Architecture-Adaptive Code Variant Tuning.” In: ASPLOS.
ACM, 2016.
[41] W. F. Ogilvie, P. Petoumenos, Z. Wang, and H. Leather.
“Minimizing the cost of iterative compilation with active
learning.” In: CGO (2017).
[42] J. Ren, L. Gao, and Z. Wang. “Optimise Web Browsing
on Heterogeneous Mobile Platforms: A Machine Learning
Based Approach.” In: INFOCOM. 2017.
[43] C. Cummins, P. Petoumenos, M. Steuwer, and H. Leather.
“Towards Collaborative Performance Tuning of Algorithmic
Skeletons.” In: HLPGPU. 2016.
[44] Y . Jiang, Z. Z. Zhang, K. Tian, F. Mao, M. Gethers, X. Shen,
and Y . Gao. “Exploiting Statistical Correlations for Proactive
Prediction of Program Behaviors.” In: CGO (2010).
[45] Y . Wen, Z. Wang, and M. O’Boyle. “Smart Multi-Task
Scheduling for OpenCL Programs on CPU/GPU Heteroge-
neous Platforms.” In: HiPC. IEEE, 2014.
[46] C. Dubach, T. M. Jones, E. V . Bonilla, G. Fursin, and
M. O’Boyle. “Portable Compiler Optimisation Across Em-
bedded Programs and Microarchitectures using Machine
Learning.” In: MICRO. ACM, 2009.
[47] B. Taylor, V . S. Marco, and Z. Wang. “Adaptive Optimiza-
tion for OpenCL Programs on Embedded Heterogeneous
Systems.” In: LCTES. 2017.
[48] A. Collins, C. Fensch, H. Leather, and M. Cole. “MaSiF:
Machine Learning Guided Auto-tuning of Parallel Skele-
tons.” In: HiPC. IEEE, 2013.
[49] C. Dubach, J. Cavazos, B. Franke, G. Fursin, M. O’Boyle,
and O. Temam. “Fast Compiler Optimisation Evaluation
Using Code-Feature Based Performance Prediction.” In: CF.
ACM, 2007.
[50] P. Ting, C. Tu, P. Chen, Y . Lo, and S. Cheng. “FEAST: An
Automated Feature Selection Framework for Compilation
Tasks.” In: arXiv:1610.09543 (2016).
[51] E. Park, J. Cavazos, and M. A. Alvarez. “Using Graph-
Based Program Characterization for Predictive Modeling.”
In: CGO. IEEE, 2012.
[52] J. Cavazos, C. Dubach, F. Agakov, E. Bonilla, M. O’Boyle,
G. Fursin, and O. Temam. “Automatic Performance Model
Construction for the Fast Software Exploration of New
Hardware Designs.” In: CASES. 2006.
[53] X. Gu, H. Zhang, D. Zhang, and S. Kim. “Deep API
Learning.” In: FSE. ACM, 2016.
[54] Y . Oda, H. Fudaba, G. Neubig, H. Hata, S. Sakti, T. Toda,
and S. Nakamura. “Learning to Generate Pseudo-Code from
Source Code Using Statistical Machine Translation (T).” In:
ASE. IEEE, 2015.
[55] Y . LeCun, Y . Bengio, and G. Hinton. “Deep learning.” In:
Nature 521.7553 (2015).
[56] Q. V . Le, R. Monga, M. Devin, G. Corrado, K. Chen,
M. A. Ranzato, J. Dean, and A. Y . Ng. “Building High-
level Features Using Large Scale Unsupervised Learning.”
In: ICML. 2012.
[57] S. Han, H. Mao, and W. J. Dally. “Deep Compression:
Compressing Deep Neural Networks with Pruning, Trained
Quantization and Huffman Coding.” In: arXiv:1510.00149
(2015).
APPENDIX
ARTIFACT DESCRIPTION
A. Abstract
Our research artifact consists of interactive Jupyter note-
books. The notebooks enable users to replicate all experi-
ments in the paper, evaluate results, and plot ﬁgures.
B. Description
1) Check-list (Artifact Meta Information):
• Run-time environment: Ubuntu Linux and a web browser.
• Hardware: Users with an NVIDIA GPU may enable CUDA
support to speed up computation of experiments.
• Output: Trained neural networks, predictive model evalua-
tions, ﬁgures and tables from the paper.
• Experiment workﬂow: Install and run Jupyter notebook
server; interact with and observe results in web browser.
• Experiment customization: Edit code and parameters in
Jupyter notebooks.
• Publicly available?: Yes, code and data. See:
https://chriscummins.cc/pact17/
2) How Delivered: A publicly available git repository
containing Jupyter notebooks and experimental data.
C. Installation
See https://chriscummins.cc/pact17/ for instructions. The
code directory contains the Jupyter notebooks. Following
the build instructions described in code/README.md, the
full installation process is:
$ ./bootstrap.sh | bash
$ ./configure
$ make
D. Experiment Workﬂow
1) Launch the Jupyter server using the command:
make run.
2) In a web browser, navigate to:
http://localhost:8000.
3) Select a Jupyter notebook to open it.
4) Repeatedly press the play button (tooltip is “run cell,
select below”) to step through each cell of the note-
book.
OR select “Kernel” > “Restart & Run All” from the
menu to run all of the cells in order.
E. Evaluation and Expected Result
Code cells within Jupyter notebooks display their output
inline, and may be compared against the values in the paper.
Expected results are described in text cells.
F . Experiment Customization
The experiments are fully customizable. The Jupyter note-
book can be edited “on the ﬂy”. Simply type your changes
into the cells and re-run them.
Note that some of the code cells depend on the values of
prior cells, so must be executed in sequence. Select “Kernel”
>“Restart & Run All” from the menu to run all of the cells
in order.
G. Notes
For more information about DeepTune, visit:
https://chriscummins.cc/deeptune
For more information about Artifact Evaluation, visit:
http://cTuning.org/ae/submission-20170414.html

PROGRAML: G RAPH -BASED DEEP LEARNING FOR
PROGRAM OPTIMIZATION AND ANALYSIS
Chris Cummins∗
School of Informatics
University of Edinburgh
c.cummins@ed.ac.uk
Zacharias V . Fisches*
Department of Computer Science
ETH Zurich
zfisches@student.ethz.ch
Tal Ben-Nun
Department of Computer Science
ETH Zurich
talbn@inf.ethz.ch
Torsten Hoeﬂer
Department of Computer Science
ETH Zurich
htor@inf.ethz.ch
Hugh Leather
School of Informatics
University of Edinburgh
hleather@inf.ed.ac.uk
March 25, 2020
ABSTRACT
The increasing complexity of computing systems places a tremendous burden on optimizing compil-
ers, requiring ever more accurate and aggressive optimizations. Machine learning offers signiﬁcant
beneﬁts for constructing optimization heuristics but there remains a gap between what state-of-the-
art methods achieve and the performance of an optimal heuristic. Closing this gap requires improve-
ments in two key areas: a representation that accurately captures the semantics of programs, and a
model architecture with sufﬁcient expressiveness to reason about this representation.
We introduce PROGRAML — Program Graphs for Machine Learning — a novel graph-based pro-
gram representation using a low level, language agnostic, and portable format; and machine learning
models capable of performing complex downstream tasks over these graphs. The PROGRAML rep-
resentation is a directed attributed multigraph that captures control, data, and call relations, and
summarizes instruction and operand types and ordering. Message Passing Neural Networks prop-
agate information through this structured representation, enabling whole-program, per-instruction,
and per-variable classiﬁcation tasks. P ROGRAML is a compiler-independent representation with
support currently for LLVM and XLA IRs.
PROGRAML provides a general-purpose program representation that equips learnable models to
perform the types of program analysis that are fundamental to optimization. To this end, we eval-
uate the performance of our approach ﬁrst on a suite of traditional compiler analysis tasks: control
ﬂow reachability, dominator trees, data dependencies, variable liveness, and common subexpres-
sion detection. On a benchmark dataset of 250k LLVM-IR ﬁles covering six source programming
languages, P ROGRAML achieves an average 94.0 F1 score, signiﬁcantly outperforming the state-
of-the-art approaches. We then apply our approach to two high-level tasks — heterogeneous device
mapping and program classiﬁcation — setting new state-of-the-art performance in both.
1 Introduction
The landscape of computing ecosystems is becoming increasingly complex: multi-core and many-core processors,
heterogeneous systems, distributed and cloud platforms. Manually extracting performance and energy beneﬁts from
systems like these is beyond the capabilities of most programmers. In such an environment, high quality optimization
heuristics are not just desirable, they are required. Despite this, good optimization heuristics are hard to come by.
∗Both authors contributed equally
arXiv:2003.10536v1  [cs.LG]  23 Mar 2020
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
Compiler
Input Program Intermediate 
Representation
ProGraML 
Representation
Message Passing 
Neural Networks
Optimization 
Passes
Output 
Executable
Figure 1: Our proposed approach for compiler analyses driven by graph-based deep learning. The P ROGRAML
representation is derived from a compiler’s IR and serves as input to Message Passing Neural Networks, which provide
optimization decisions in place of traditional handwritten heuristics.
Designing and tuning optimization heuristics takes time, effort, and resources. To make things worse, this is a
Sisyphean task: even minor changes in a development toolchain might require retuning the heuristic; major changes
in the software or the hardware usually require freshly designed heuristics. Machine learning offers to liberate us from
this cycle by replacing fragile hand-tuned optimization heuristics with models that are inferred automatically from real
performance data [1, 2]. Typically, programs are represented using a sequence of numerical features which are derived
from programs using ad-hoc analysis, but such approaches fail to capture the rich semantic structure of programs,
limiting the ability for models to reason about program behavior. This is easy to see in traditional machine learned
approaches, where the designer explicitly chooses a program representation based only on a few properties deemed
important. Such representations prevent models from reproducing the reasoning of even basic compiler analyses and
that, in turn, limits their ability to make good optimization decisions.
Recent deep learning approaches that work directly on code [3] are limited, both in the way they represent the inputs
and in the way they process them. Representations based on source code and its direct artifacts (e.g. AST) put
unnecessary emphasis on naming and stylistic choices that might or might not correlate with the functionality of the
code [4, 5, 6]. Current IR-based approaches [7, 8, 9] use compilation to remove such noise but in the process they omit
important information about the program.
In both cases, the model is expected to reason about the ﬂow of information in the program using representations that
do not encode this information clearly. If this was not difﬁcult enough already, processing the code representations se-
quentially, as most existing approaches do, makes it practically impossible. Related statements can easily be separated
by hundreds of lines of irrelevant code in sequential representations. Due tovanishing gradients [10] and catastrophic
forgetting [11], neural networks are unlikely to even notice such very long range dependencies.
In this paper, we propose overcoming this limitation by making the program’s control, data, and call dependencies
a central part of the program’s representation and a primary consideration when processing it. We achieve this by
seeing the program as a graph, in which individual statements are connected to other statements through relational
dependencies. The latent representation of each statement is then a function of not just the statement itself but the
latent representations of its graph neighborhood. Each statement and data element in the program is understood only
in the context of the statements interacting with it. In contrast to prior sequential learning systems for code, this rep-
resentation closely resembles the intermediate representations used by compilers, and the propagation of information
through these graphs mimics the behavior of typical iterative data-ﬂow analyses.
Techniques for learning over graphs have recently been proposed and have shown promise in a number of do-
mains [12, 13, 14]. Our intent with this work is to extend these approaches to the domain of program analysis and
provide a systematic evaluation of their suitability and limits for compiler tasks. With a graph-based approach, we are
able to automatically learn established compiler analyses that rely on control and data ﬂow and do it far better than
existing code modeling approaches. Downstream tasks built on top of such graph models can then natively incorporate
approximate compiler analyses into their decision making, leading to superior and more powerful models.
2
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
1.1 Contributions
We make the following contributions:
• A portable, language-agnostic graph representation of programs derived from compiler intermediate represen-
tations (IR), and machine learning models for relational reasoning about the control, data, and call relations
in programs1. The P ROGRAML representation is compiler-angostic, with support currently for LLVM and
XLA IRs, and is the ﬁrst to summarize instructions, operand types, and operand order.
• As a benchmark for our approach, we pose a suite of established compiler analysis tasks as supervised ma-
chine learning problems, comparing the performance of models using P ROGRAML against state-of-the-art
code representations. Succeeding at these benchmark tasks requires the ability to model control- and data-
ﬂow, function boundaries, instruction types, and the type and order of operands. On a large dataset of over
250k LLVM-IRs taken from real-world software projects, our approach achieves an average 94.0 F1 score
(98.9 precision, 92.0 recall), a 3.44×improvement over the state-of-the-art.
• We demonstrate the efﬁcacy of our approach on two challenging downstream compiler tasks: heterogeneous
device mapping, and program classiﬁcation. Our results in heterogeneous device mapping show the supe-
riority of graph based approaches. In an ablation study in program classiﬁcation, we show a signiﬁcant
performance improvement due both to the rich structure of the PROGRAML representation and the message
passing neural network approach. Our approach reaches an accuracy of 96.22% in program classiﬁcation, a
1.27×improvement over the state-of-the-art.
2 Motivation
Machine learning promises signiﬁcant beneﬁts for simplifying and improving the construction of optimization heuris-
tics by replacing fragile and expensive hand-tuned heuristics with data-driven statistical modeling. For this goal to
be realized, we require machine learning systems capable of reasoning about program semantics. Despite tremen-
dous gains in recent years, state-of-the-art approaches for learning on code are not sufﬁciently powerful to replicate
the analysis tasks that are fundamental to compilers. The crux of the problem lies in two central aspects of machine
learning: input representation and model algorithmic complexity.
(I) Input Representation To be processed by a Neural Network (NN), code inputs may be encoded either directly
from a source language, by way of AST analysis, or using an IR. Examples of each abound in the literature [3, 15, 2].
One state-of-the-art encoder, code2vec [4], uses AST paths to embed programs. code2vec proves highly effective
at software engineering tasks such as algorithm classiﬁcation, where the code was written by humans. However, as
shown in Figure 2a, the trained representation can put more weight on names rather than code structure, where minute
modiﬁcations completely change classiﬁcation outcome. There are many beneﬁts to such a representation, including
smart pasting and automated refactoring. However, when analyzing code for optimization, identiﬁer names are rarely
of use, whereas structure and semantics should be the primary consideration. For example, in choosing to represent
code at the AST level, the code2vec representation does not capture the data or control relations between statements.
An alternate approach which emphasizes semantics is Neural Code Comprehension [7], where an encoder uses Con-
textual Flow Graphs (XFG) built from LLVM-IR statements to create inputs for neural networks. The XFG combines
partial data- and control-ﬂow to represent the context of an individual statement. The statements are then mapped to
latent-space representations using their neighbors in that graph. However, in partially combining DFGs and CFGs, the
XFG representation omits important information such as order of instruction operands (as shown in Figure 2b), and
the representation fails to capture execution order, critical for many optimization tasks.
A recent LLVM-IR representation uses Control and Data Flow Graphs (CDFG) [9]. This representation makes the
control and data relations between instructions explicit, but uses only instruction opcodes to compute latent represen-
tations. This omits information about programs that may be critical for optimization, such as data types, the presence
of variables and constants, and the ordering of operands, shown in Figure 2c.
Each of the three methods of encoding programs as inputs to neural networks omit information that is vital for compiler
analysis. This hinders the ability of machine learning models to reason about optimizations and their impact on
program behavior. To address these shortcomings we require an input representation that captures all parts of a
program’s semantics that are required for performing such analyses.
1Code and datasets available at https://chriscummins.cc/ProGraML
3
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
int f(int n) {
if (n == 0) return 1;
else return n * f(n-1);
}
factorial 50.93%
fact 19.15%
pad 8.92%
sinc 77.78%
times 3.89%
isPowerOfTwo 3.36%
int f(int x) {
if (x == 0) return 1;
else return x * f(x-1);
}
int fibonacci(int n) {
if (n == 0) return 1;
else return n * fibonacci(n-1);
}
fibonacci 99.09%
testRun 0.75%
Iter 0.07%
(a) code2vec [4].
%1 %2
%3
0.0
fdiv floatfdiv float (b) XFG [7].
sext
load
load
getelementptr (c) CDFG [9].
Figure 2: Limitations in state-of-the-art learnable code representations. In (a) the model over-emphasizes identiﬁer
names such that the same algorithm produces three different classiﬁcations by changing the name of a function. The
data-ﬂow representation of (b) does not capture operand order, such that non-commutative statements such as division
are indistinguishable. In (c) control and data relations are captured, but both type information and operand order are
omitted. Our approach is insensitive to identiﬁer names and preserves operand order and type information.
(II) Model Complexity The range of core operators in deep learning on code is largely conﬁned to recurrent units
(e.g. RNN, LSTM, GRU) on sequences. This poses limitations on the representation space of any such network’s
outputs. Take, for instance, dominator tree construction. An LSTM iterating forward over input code will not be
able to solve the task by deﬁnition, as statements needs to be analyzed backwards with respect to dependencies. Yet,
the neural network only maintains O(1) memory capacity w.r.t. program length. Iterating in the opposite direction,
in addition to being problem-specialized, would still not sufﬁce, as dependencies may “vanish” in the corresponding
gradients if dependent statements are far away from each other, such as in the presence of diverging control ﬂow.
One way to increase the algorithmic complexity of the model is by allowing it to increase the number of code tokens
that are processed simultaneously during inference. This approach, commonly used in literature by Transformer
Networks [16], use preceding tokens (unidirectional encoding) or preceding and subsequent tokens (bidirectional
encoding) to learn attention matrices . Such matrices “focus” the network on certain subsets of tokens, skipping
others. However, this approach scales quadratically in memory and computation with the number of tokens.
Unlike in natural language text, the dependency structure of code is made explicit during compilation. We can thus
employ domain-speciﬁc knowledge to construct the attention matrices in a scalable manner, using a graph representa-
tion of the tokens with dependencies as edges. A graph representation not only enables meaningful attention learning,
but also facilitates propagating information across the graph in a manner similar to typical compiler analyses. Within
the same step, a recurrent unit generates O(1) activations, whereas a graph NN generates O(|V|).
To demonstrate this expressive power, let us consider control-ﬂow reachability analysis as a learning task. The goal
of the model is to tag statements that are reachable from one or more given tagged statements. With a sequential
LSTM, the model would have to be trained to memorize nodes along some linear order of the given program. With
an unbounded number of nodes to track and variably-sized regions to skip, the task becomes infeasible. A message-
passing graph neural network, in contrast, needs only to learn to pass a message forward in the case of an existing
control-ﬂow edge between two nodes, essentially learning an identity operation over control-ﬂow edges and zero on
others.
In this work, we overcome the above limitations of prior model and representation approaches, leveraging the graph
structure of IR code, taking path analysis and the semantics of individual statements into account.
3 A Graphical Program Representation for Analysis and Optimization
For machine learning to successfully reason over programs, a suitable input representation must be used. This section
presents P ROGRAML, a novel program representation that closely matches the representations used traditionally
within compilers and can be processed natively by machine learning models. Unlike prior approaches that rely on
hand-engineered feature extractors [1, 2] or which are closely tied to the syntax of the target program language [17],
our approach captures whole-program control, data, and call relations and is both task- and language-agnostic.
4
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
define i32 @Fib(i32) #0 {
 switch i32 %0, label %3 [
   i32 0, label %9
   i32 1, label %2
 ]
; <label>:2:
 br label %9
; <label>:3:
 %4 = add nsw i32 %0, -1
 %5 = tail call i32 @Fib(i32 %4)
 %6 = add nsw i32 %0, -2
 %7 = tail call i32 @Fib(i32 %6)
 %8 = add nsw i32 %7, %5
 ret i32 %8
; <label>:9:
 %10 = phi i32 [ 1, %2 ], [ %0, %1 ]
 ret i32 %10
}
int Fib(int x) {
 switch (x) {
  case 0: 
   return 0;
  case 1:
   return 1;
  default:
   return Fib(x - 1) 
        + Fib(x - 2);
 }
}
Input program
Compiler IR
(a) Compiler intermediate representation.
switch
add nsw
br
tail call
add nsw
tail call
add nsw
ret
ret
0 1
2 (b) Construct control ﬂow.
switch
add nsw
br
tail call
add nsw
tail call
add nsw
ret
ret
i32
i32
i32
i32
i32
i32
i32 0
i32 -1
i32 -2
0
1 2
0 1
0
1
1 0
i32 1
01
(c) Add data ﬂow for variables and constants.
i32 1switch
add nsw
br
tail call
add nsw
tail call
add nsw
ret
[external]
i32
i32
i32
i32
i32
i32
i32 0
i32 -1
i32 -2
ret (d) Add call ﬂow for call sites.
Figure 3: Construction of a P ROGRAML representation for a simple C implementation of recursive Fibonacci using
LLVM-IR. The input program is passed through the compiler front end to produce an intermediate representation (a).
A full ﬂow graph is constructed from the IR statements and control ﬂow edges inserted (b). Vertices for variables and
constant values are added and data-ﬂow edges inserted (c). Finally, call sites are extracted and call edges inserted from
call sites to function entry statements, and from function exit vertices to call sites (d). All edges are positional, for
clarity we have omitted position labels where not required.
5
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
3.1 Overview
The PROGRAML representation of a compiler IR serves as the union between a call graph, control-ﬂow graph, and
data-ﬂow graph. We represent programs as directed multigraphs where statements, identiﬁers, and immediate values
are vertices, and relations between vertices are edges. Edges are typed to differentiate control-, data-, and call-ﬂow.
Additionally, we augment edges with a positional label to encode the order of operands for statements, and to dif-
ferentiate between divergent branches in control-ﬂow. The P ROGRAML representation is processed natively by our
machine learning models, described in Section 4.
3.2 Graph Construction
We construct a P ROGRAML graph G = (V,E) by traversing a compiler IR. Graph construction is divided into
three stages: control-ﬂow, data-ﬂow, and call-ﬂow, though in practice the three stages can be combined in a single
O(|V|+ |E|) pass. The representation is compiler-agnostic, adding support for a new compiler requires only a parser
for the IR. Currently we support LLVM [18] and XLA HLO [19] IRs. Figure 3 shows the graph construction approach.
(I) Control Flow We construct a full-ﬂow graph from an IR by inserting a graph vertex for each instruction and
control ﬂow edges between them, as shown in Figure 3b. All control edges are augmented with a numeric position
label using an ascending sequence based on their order in the list of a vertex’s outgoing control edges. For instructions
with a single control successor, the position of the control edge is 0. For a branching statement with n successor
statements, the control edge positions are in the range 0 ≤epos ≤n. We do not need to encode the source function
or basic block [18] of instructions as this information is captured implicitly in structure of the graph; basic blocks are
regions of instructions with a single entry and exit control edge, and functions are disconnected subgraphs.
(II) Data Flow We introduce additional graph vertices for constant values and variables, shown in Figure 3c. Data-
ﬂow edges are added to capture the relation from constants and variables to the instructions that use them as operands,
and instructions to produced variables. Each unique variable and constant is a vertex, which implicitly models the
scope of variables, and unlike the tokenized representations of prior machine learning works, variables in different
scopes always map to distinct vertices and can thus be discerned. Similarly, constant values with the same textual
representation in source code (such as the number 3.14 with float and double precision types) are distinguishable
in our representation. As with control edges, data edges have a position label which encodes the order of operands for
instructions. The latent representation of an IR statement is thus a function of the vertex representing the instruction
and the vertices of any operand variables or constants, modulated by their order in the list of operands.
(III) Call Flow Control edges do not span functions, such that an IR with functions F produces |F|disconnected
subgraphs (the same is not true for data edges which may cross function boundaries, for example in the case of an
global constant which is used within multiple functions of a program). Instead, the relation between a statement which
calls a function and the called function is captured through call edges, shown in Figure 3d. An outgoing call edge is
inserted from the calling statement to the entry statement of a function. Return call edges are added from all terminal
statements of a function to the calling statement. Call edges do not use position labels as there is no ordering to be
imposed between the call sites of a function. For IRs which support external linkage, an additional vertex is created
representing an external callsite and connected to all externally visible functions. Similarly, if a call site references
a function not deﬁned in the current IR, a dummy function deﬁnition is created consisting of a pair of entry and exit
instruction vertices, and connected normally through call edges. A single dummy function deﬁnition is created for
each externally deﬁned function and shared across all call sites in the current IR.
3.3 Comparison to Other Representations
As an input for machine learning, what distinguishes P ROGRAML from prior works is its close proximity to the
structured representations traditionally used within compilers for program analysis and optimization. Speciﬁcally, it
is distinct from prior machine learning representations in three key areas:
1. as an IR-level representation, it is independent of the source language and accompanying variances such as
code style and identiﬁer naming that affect source-level representations [4, 20];
2. by representing programs as graphs with explicit control, data, and call edge types P ROGRAML captures a
greater range of intra-program relations than prior graph representations [7, 17, 21];
3. and in trading sequential for graph representations, we do not sacriﬁce local sequence order, such as the
ordering of diverging control ﬂow or the ordering of operands that is lost in prior representations [7, 9].
6
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
Source Languages Representation Flow-
sensitive?
Position-
sensitive?
Value-
sensitive?
AST Paths [22] C#, Java, JavaScript, Python AST 
CDFG [9] OpenCL IR Graph 
code2vec [4] Java AST 
DeepTune [23] OpenCL Token Sequence  
DeepTune-IR[24] OpenCL IR Token Sequence 
DeepTyper [25] JavaScript Token Sequence  
inst2vec [7] C++, OpenCL IR Graph  
PROGRAML C, C++, Fortran, Haskell,
OpenCL, Swift IR Graph   
Table 1: Taxonomy of recent code representations for machine learning. We classify approaches based on the type
of representation used and the sensitivity to three categories: {control/data/call}-ﬂow, operand positions, and operand
values. Prior approaches require a trade-off in representational power, e.g. substituting a position-sensitive token
sequence for a ﬂow-sensitive graph. PROGRAML is the ﬁrst representation to span all categories.
Table 1 provides a comparison of PROGRAML to several recent learned representations of code.
4 Graph-based Machine Learning
We formulate our system in a Message Passing Neural Network (MPNN) framework [14, 12] and implement a single
uniﬁed model for all our experiments. Our design mimics the transfer functions and meet operators of classical
iterative dataﬂow analysis [26, 27], replacing the rule-based implementations with deep learning analogues (message
and update functions). Those can be specialized through training to solve a diverse set of problems without human
intervention or algorithm design.
4.1 Overview
We learn over PROGRAML representations of compiler IRs by mapping graph vertices to an initial state vector using
an embedding. The vertex states are updated iteratively in a sequence of message passing steps, where at each step
a new vertex state is computed as a function of its previous state and the state of its neighboring vertices. Separate
functions are learned to update vertex neighbors based on their relation type, be it control, data or call, and reverse
edges enable backwards propagation of information. After repeating this process of updating vertex states for a ﬁxed
number of iterations a readout function is used to aggregate the vertex representations to a single graph-level vector or
set of vertex-level vectors.
4.2 Model Design
The P ROGRAML model takes as input a directed graph with additional information as presented in Section 3 and
consists of three logical phases: input encoding, message propagation, and result readout.
(I) Input Encoding Starting from the augmented graph representation G = (V,E) introduced in Section 3, we
capture the semantics of the program graph vertices by mapping every instruction, constant, and variable vertexv∈V
to a vector representation h0
v ∈Rd by lookup in a ﬁxed size embedding table CIR ∈Rn×d. The mapping from vertex
to embedding vector f : v ↪→h0
v ∈CIR must be deﬁned for each IR, though the embeddings themselves can be
learned during training.
For LLVM-IR we extend the inst2vec [7] vocabulary, which represents 8,566 statements derived from a large corpus
of LLVM-IR using 200-dimensional embeddings. Since the space of statements is unbounded, inst2vec uses a normal-
ization process to inline type deﬁnitions and strip identiﬁers and immediate values, depicted in Figure 4. An inst2vec
representation combines an instruction and its operands into a single token, so we augment the vocabulary with ID and
VAL tokens to represent variable and constant value vertices, respectively.
In expressing a statement as a combination of an instruction and its operands, our data-driven approach trades a certain
amount of semantic resolution against good coverage of the vocabulary by the available datasets. The long tail of the
distribution of statements jointly maps onto a special U NKNOWN token vector in the vocabulary. In future work will
simplify the LLVM-IR statement encoding by constructing separate vocabularies for instructions and operand types,
7
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
val
%id = getelementptr inbounds {i32},
    {i32}* %id, i64 %int, i32 %int
%id
%id
val
0 1 2
0
%id = getelementptr inbounds {i32},
    {i32}* %id, i64 %int, i32 %int
%struct.X = type { i32 }
%3 = getelementptr inbounds %struct.X, 
    %struct.X* %0, i64 0, i32 0
(a) IR Statement (b) inst2vec-normalized
(c) ProGraML representation
Figure 4: Normalizing an LLVM-IR statement (a) by inlining type deﬁnitions and stripping identiﬁers using
inst2vec [7] (b). A normalized statement is used as the key for an embedding table lookups to produce the ini-
tial vector representation of a vertex. (c) shows the statement as contextualized in a P ROGRAML graph, where the
operand variables and constants are data elements, differentiated by their position. Vertex labels represent embedding
table keys. In this example, the statement is represented using ﬁve vertices and encoded with three unique embeddings.
increasing the expressiveness of the encoding by allowing a larger combination of instruction and operands to be
represented. Input vertex-selectors, encoded as binary one-hot vectors, are used to mark the starting point for certain
analyses and are concatenated to the vertex embeddings. Other global input features are used as auxiliary input features
at readout time in step (III), where required.
(II) Message Propagation Each iteration step is divided into a message propagation step followed by a vertex state
update. During message propagation, each vertex in the graph collects learned messagesmt
v ∈Rd from its undirected
neighbors:
mt
v =
∑
w∈N(v)
Atype(ewv) (ht−1
w ⊙POS (ewv)) (1)
where ⊙denotes the Hadamard product and ewv ∈E is the typed edge between vertex w and v. In order to allow
for reverse-propagation of messages, which is necessary for backward compiler analyses, we add backward edges for
each edge in the graph. For backward edges we introduce separate parameters following Li et al. [12] to enable the
network to distinguish between an edge and its backward sibling. During message propagation we scale the source
states ht−1
v with POS (·) ∈Rd, which is a constant sinusoidal position embedding [16, 28] that encodes the argument
order of incoming (and outgoing) edges. This information is necessary for the network to distinguish non-commutative
operations such as division.
The collected messages are subsequently used to update the vertex states hv ∈Rd in parallel according to an update
function. In all our experiments, we employ a Gated Recurrent Unit (GRU) [29] as our update function:
ht
v = GRU(ht−1
v ,mt
v) (2)
Step (II) is iterated T times to extract vertex representations that are contextualized with respect to the given graph
structure.
(III) Result Readout We support whole program classiﬁcation, per-statement classiﬁcation, or per-identiﬁer classi-
ﬁcation by employing different readout heads on top of the iterated feature extraction: for graph-level classiﬁcation
we deﬁne a set function RG({hT
v ,h0
v}v∈V ) that maps to class-scores, while for vertex-level inference, we separately
map the extracted node features hT
v to probabilities Rv(hT
v ,h0
v) in parallel:
Rv(hT
v ,h0
v) =σ
(
i(hT
v ,h0
v)
)
·j(hT
v ) (3)
RG({hT
v ,h0
v}v∈V ) =
∑
v∈V
Rv(hT
v ,h0
v) (4)
where i(·) and j(·) are feed forward Neural Networks and σ(·) is the sigmoid activation function. In the case where
auxiliary graph-level features are available, those are concatenated to the readout values and fed through another feed
forward Neural Network that employs Batch Normalization [30] to allow for vastly different feature scales.
8
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
5 Experimental Methodology
We evaluate the effectiveness of our approach in three case studies. In the ﬁrst, we apply our methodology to a suite of
established compiler analysis tasks. These serve as demonstrations of the representational power of our approach and
highlight the limitations both in prior machine learning approaches and in current MPNNs. The second case study then
applies the approach to the challenging real-world optimization task of heterogeneous device mapping, comparing the
performance of our model against state-of-the-art machine learning-based approaches. Finally, we apply our approach
to the task of classifying algorithms from unlabelled implementations. This section describes the methodology used
to construct these experiments: the datasets used, the model parameters, and training regimes.
5.1 Case Study A: Compiler Analyses
We construct a benchmark suite of traditional compiler analysis tasks to evaluate the representational power of our
approach. We chose a diverse set of tasks to capture a mixture of both forward and backward analyses, and control-,
data-, and procedure-sensitive analyses. Our goal is not to suggest that machine learning should replace the existing
implementations of these standard algorithms which can be found in any compiler, but rather, if a machine learning
system is not capable of producing these analyses, it stands to reason that performance on downstream tasks which
depend on these analyses will suffer.
5.1.1 Benchmark Analyses
We selected ﬁve traditional compiler analyses to use as benchmarks for evaluating the representational power of our
approach.
(I) Reachability Control reachability is a fundamental compiler analysis which determines the set of statements that
can be reached from a particular starting statement. Given succ (n), which returns the control successors of statement
n, the set of reachable statements starting at root ncan be found using forward analysis:
Reachable(n) ={n}∪

 ⋃
s∈succ(n)
Reachable(s)

 (5)
(II) Dominator trees Statement n dominates statement m if every control ﬂow path to m passes through n. A
dominator tree is the set of all nodes that dominate the statment at a particular program point. Like reachability, this
analysis only requires propagation of control ﬂow, but unlike reachability, dominator trees are typically constructed
using backward analysis [31, 32]:
Dominators(n) ={n}∪

 ⋂
p∈pred(n)
Dominators(p)

 (6)
Where pred(n) which returns the control predecessors of statement n.
(III) Live-out variables A variable ais live-out of statement nif there exists some control successor of nthat uses
a. Given uses(n), which returns the operand variables of n, and defs(n), which returns deﬁned variables, the live-out
variables can be computed forwards using:
LiveOut(n) =
⋃
s∈succ(n)
uses(s) ∪
(
LiveOut(s) −defs(s)
)
(7)
(IV) Data dependencies The data dependencies of statement n is the set of predecessor statements that must be
evaluated to produce the operands of n. Computing data dependencies requires data sensitivity and is computed
backwards:
DataDep(n) =defs(n) ∪

 ⋃
p∈defs(n)
DataDep(p)

 (8)
9
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
Source
language Domain IR ﬁles IR lines Avg.
vertices / IR
Avg. edges /
IR
BLAS 3.8.0 Fortran Scientiﬁc Computing 300 345,613 1,664 3,276
Linux 4.19 C Operating Systems 13,851 41,332,089 1,857 3,760
OpenCL Benchmarks [23] OpenCL Benchmarks 256 149,779 1,027 1,970
OpenCV 3.4.0 C++ Computer Vision 400 1,168,758 3,761 7,442
POJ-104 [33] C++ Standard Algorithms 182,815 64,518,837 312 569
Tensorﬂow [34] C++ Machine learning 1,584 8,444,443 5,786 11,482
GitHub
C
Various
42,880 89,356,570 927 1,794
Haskell 1,371 6,745,312 4,705 7,518
OpenCL 5,188 10,472,388 2,299 5,132
Swift 1,783 205,140 134 371
Total — — 250,428 222,738,929 153,426,059 294,685,614
Table 2: The ten sources of LLVM-IR used to produce datasets for evaluating data ﬂow analyses. Our corpus comprises
six programming languages from functional to imperative, high-level to accelerators. The software covers a broad
range of disciplines from compilers and operating systems to traditional benchmarks, machine learning systems, and
unclassiﬁed code downloaded from popular open source repositories.
Where defs(n) returns the statements that produce operands of n.
(V) Global Common Subexpressions The identiﬁcation of common subexpressions is an important analysis for
optimization. For compiler IRs we deﬁne a subexpression as a statement and its operands, ordered by either their
position (for non-commutative operations), or lexicographically (for commutative operations). We thus formulate the
common subexpression problem as, given a statement (which forms part of a subexpression), label any other statements
in the program which compute the same subexpression. This is an inter-procedural analysis, though operands must
obey their scope. Common subexpressions are typically identiﬁed using available expression analysis:
Avail(n) =uses(n) ∪

 ⋂
p∈pred(n)
Avail(p)

−defs(n)
Where uses(n) return the expressions used by statement n, and defs(n) returns the expressions deﬁned by n.
5.1.2 Datasets
We assembled a large corpus of real-world LLVM-IRs from a variety of sources, summarized in Table 2. We selected
popular open source software projects that cover a diverse range of domains and disciplines, augmented by uncatego-
rized code mined from popular GitHub projects using the methodology described by Cummins et al. [20]. Our corpus
comprises a range of source languages (C, C++, Fortran, Haskell, OpenCL, and Swift) and exceeds 250k ﬁles. We
de-duplicated the corpus ﬁrst at the source level, then again after lowering to LLVM-IR. Lowering from source to IR
was performed using the inst2vec methodology [7], in which an optimization level is chosen randomly per-ﬁle.
We generated a single PROGRAML representation for each of the LLVM-IRs, taking an average of 31 ms per IR. Our
corpus of unlabeled graphs totals 153M vertices and 295M edges, with an average of 616 vertices per graph with 472
unique vertex representations, and 1183 edges with a maximum edge position of 355 (a largeswitch statement found
in a Tensorﬂow compute kernel).
We produced labeled graphs from the unlabeled corpus by computing ground truth labels for each of the analysis tasks
using a traditional analysis implementation. For each of the ﬁve tasks, and for every unlabeled graph in the corpus, we
produce nlabeled graphs by selecting unique source vertices v0 ∈V, where nis proportional to the size of the graph:
n= min
(⌈|V|
10
⌉
,10
)
(9)
10
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
(a) REACHABILITY .
 (b) DOMTREE .
 (c) DATADEP.
 (d) LIVENESS .
 (e) SUBEXPRESSIONS .
Figure 5: Example input-output graphs for each of the ﬁve benchmark compiler analyses. A single vertex is randomly
selected from the input graph to represent the starting program for computing the analysis results, indicated using the
vertex selector. The output graphs contain binary labels for each of the graph vertices after the analysis has completed.
As a supervised classiﬁcation task, the goal of the model is to predict the output vertex labels given the input graph.
These small graphs are for illustrative purposes, the LLVM-IR graphs in our real-world corpus contain an average 616
vertices and 1,183 edges.
Each instance in the datasets consists of an input graph in which the source vertex is indicated using thevertex selector,
and an output graph with the ground truth labels used for training or for evaluating the accuracy of model predictions.
Figure 5 illustrates an example input-output instance for each of the ﬁve tasks.
We divided the datasets randomly using a 3:1:1 ratio for training, validation, and test instances. The same random
allocation of instances was used for each of the ﬁve tasks. Where multiple labeled graphs were derived from a single
IR, instances derived from the same IR were allocated to the same split.
5.1.3 Models
LSTM Baseline As no prior work offers the expressiveness required to perform the per-statement and per-variable
classiﬁcation required for these analysis tasks, we extended DeepTune [23], a state-of-the-art deep learning framework
for whole-program classiﬁcation, to enable per-statement classiﬁcation. In DeepTune, an OpenCL program is ﬁrst
tokenized and mapped to a sequence of embedding vectors which are then processed through a sequential LSTM
model. The ﬁnal state of the LSTM is optionally concatenated with program-level features, then fed through a fully
connected neural network to produce a program-level classiﬁcation.
Figure 6 shows how we extended this approach for statement-level classiﬁcation of LLVM-IR. We ﬁrst replaced the
OpenCL tokenizer using one derived from LLVM IR, resulting in a 179-element vocabulary. To adapt the approach
for performing statement-level classiﬁcation, we group embedding vectors by their source statement before using
element-wise summation to merge embedding vectors.
We use the same models parameters as in the original work [23] — 64-dimensional embedding vectors trained jointly,
with 64 sequences per batch, padded and truncated to the same length. As LLVM IR is more verbose than OpenCL, the
sequences are signiﬁcantly longer requiring greater device memory during training and inference. This is a common
issue with recurrent neural networks, as sufﬁcient memory is required to store both the intermediate results of the
forward pass and the gradients during training [35]. We found that a sequence length of 5,000 was the maximum that
our experimental platform allowed. Where sequences exceed this length, they are truncated, and the model outputs
padded with zeros to match the expected shape.
ProGraML We use the model design outlined in Section 4 for each of the compiler analysis tasks. While we use
the vocabulary of inst2vec, we do not use the pre-trained embeddings, instead initializing the embeddings randomly
and training jointly.
Message Passing Neural Networks typically use a small number of propagation steps out of practical consideration
for time and space efﬁciency [14, 12], and address problems on smaller graphs than used in this work [17]. For a
large class of monotone data ﬂow analysis problems, however, it is known that up to d(G) + 3passes over the graph
are required, where d(G) is the loop connectedness of G [27, 26]. The loop connectedness captures the notion of
loop-nesting depth in a program and is therefore a program-dependent, but generally unbounded quantity1.
We address this challenge with P ROGRAML by iterating for a ﬁxed number T of message passing steps for training
and inference and excluding from the test set graphs for which a traditional implementation of the analysis task requires
greater than T iterations to solve. For the experiments in this work we set T = 30, leading to 12.56% of the graphs
in the corpus to be excluded across the ﬁve tasks. For fairness, we also excluded these graphs from evaluation of the
LSTM baseline.
1Given any depth-ﬁrst spanning tree (DFST) of G, backward edges are deﬁned as edges in G that connect a node to one of its
ancestors in the DFST and d(G) is the maximum number of backwards edges in any acyclic path in G.
11
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
%15 = load i32, i32* %nian, align 4, !tbaa !1
%rem38 = and i32 %15, 3
%rem11 = srem i32 %15, 100
  
LLVM-IR
Tokenized
Embeddings
…
…
Grouped Embeddings
5000
vocabulary indices,
padded & truncated
5000 x 64
embedding vectors
num_statements x 64
Grouped embedding vectors
+ + + +
num_segments
1-hot “selector” vector
Auxiliary Inputs
num_segments x (64+2)
concatenated embedding 
and selector vectors
Model inputs
LSTM
num_segments x 2
1-hot outputs
Model outputs
Fully-connected
/uni2322
t1 t2 t3 tnum_segments
 
/uni2322/uni2322/uni2322
…
Figure 6: Extending DeepTune [23] to perform per-statement classiﬁcation of an LLVM-IR. In the original work,
the latent representation of the entire program token sequence was used for program-level classiﬁcation, we enable
classiﬁcation of arbitrary token groupings so that we can perform statement-level classiﬁcation of a program. In the
above diagram, + denotes element-wise summation, and ⌢denotes vector concatenation.
5.1.4 Training Details and Parameters
All models were trained in an end-to-end fashion with the Adam optimizer [36] using the default conﬁguration and
learning rate 0.001. We trained the models on a NVIDIA GTX 1080 GPU-equipped machine in increments of 10k
graphs, testing on a 20k validation set at each checkpoint. Training terminated after six hours, or if accuracy on the
validation set reached 99.99%. After training completed we selected the checkpoint with the highest accuracy on the
validation set to use for testing.
5.2 Case Study B: Heterogeneous Device Mapping
We apply our methodology to the challenging domain of heterogeneous device mapping (D EVMAP). Given an
OpenCL kernel and a choice of two devices to run it on (CPU or GPU), the D EVMAP task is to predict the de-
vice which will provide the best performance. We chose this problem as it has received signiﬁcant prior attention, with
previous approaches using both hand-engineered features [37] and sequential models [23, 7].
12
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
5.2.1 Datasets
We used the dataset of [23], which provides labeled CPU/GPU instances for 256 OpenCL kernels sourced from seven
benchmark suites on two combinations of CPU/GPU pairs. The AMD set uses an Intel Core i7-3820 CPU and AMD
Tahiti 7970 GPU; the NVIDIA set uses an Intel Core i7-3820 CPU and an NVIDIA GTX 970 GPU. Each dataset
consists of 680 labeled instances derived from the 256 unique kernels by varying dynamic inputs.
5.2.2 Models
We compare ProGraML with four approaches: First, with a static baseline that predicts the mode device of the dataset
distribution. Second, with DeepTune [23], which is a sequential LSTM model at the OpenCL source level. Third, to
isolate the impact of transitioning from OpenCL source to LLVM-IR, we evaluate against a new DeepTune IR model,
which adapts DeepTune to using tokenized sequences of LLVM-IR as input instead of OpenCL tokens, using the
vocabulary described in Section 5.1.3. Finally, we compare against the state-of-the-art approach NCC [7], which
replaces the OpenCL tokenizer with a sequence of 200-dimensional embeddings, pre-trained on a large corpus of
LLVM-IR using a skip-gram model.
5.2.3 Training Details and Parameters
All neural networks are regularized with Dropout [38] for generalization and Batch Normalization [30] in order to
be uniformly applicable to vastly different scales of auxiliary input features. We used 10-fold cross-validation with
rotating 80/10/10 splits by training on 80% of the data and selecting the model with the highest validation accuracy,
setting aside 1/10th of the training data to use for validation. We trained each model for 100 epochs and selected the
epoch with the greatest validation accuracy for testing.
5.3 Case Study C: Algorithm Classiﬁcation
In a third case study, we apply our approach to task of classifying algorithms. We use the POJ-104 [33] dataset.
It contains implementations of 104 different algorithms that were submitted to a judge system. All programs were
written by students in higher education. The dataset has around 500 samples per algorithm. We compile them with
different combinations of optimization ﬂags to generate a dataset of overall 240k samples. Approximately 10,000 ﬁles
are held out each as a development and test set.
5.3.1 Models
We compare with recently published tree-based convolutional neural networks (TBCNN) [33] and NCC [7], which
uses the same approach approach as described in Section 5.2.2 on this dataset. To further test the expressive power
of the graph-based representation against the tree-based (TBCNN) and sequential (NCC) prior work, we present
additional experiments: Graph-based baselines based on XFG [7] and a PROGRAML structure-only baseline.
To better understand the qualitative aspects of replacing a graph-based representation that captures program semantics
like Contextual Flow Graphs [7] (XFG) with the more complete PROGRAML representation, we adapted a GGNN [12]
to directly predict algorithm classes from XFG representations of the programs. In contrast to this, Ben-Nun et
al. [7] used XFG only to generate statement contexts for use in skip-gram pre-training. Here, we lift this graphical
representation and make it accessible to a deep neural network directly, as opposed to the structureless sequential
approach in the original work (NCC).
Additionally, we include a structure-only baseline of our P ROGRAML approach, where only the type of each node
(instruction, variable, or constant) is encoded, refraining completely from tokenizing statements. We think that algo-
rithm classiﬁcation is a problem that lends itself especially well to judging the power of the representation structure,
since most algorithms are well-deﬁned independent of implementation details such as datatypes.
To test the limits of the expressivity of P ROGRAML, combine our representation with a powerful 10-layer Trans-
former [16] encoder model, adapted as a graph neural network for attributed graphs. We induce graph structure by
masking the attention scores in the self-attention layer with the adjacency matrix of the P ROGRAML graphs. A new
space-efﬁcient sparse implementation allows processing of graphs with on the order of 105 vertices. Different edge
types are encoded by introducing separate key and value projection matrices into the self-attention layer [16, 39].
5.3.2 Training Details and Parameters
The GGNN models were trained with the AdamW [40] optimizer with learning rate 2.5 ·10−4,β1 = 0.9,β2 =
0.999,ε = 10−8 for 80 epochs. Dropout regularization is employed on the graph states with a rate of 0.2. The Trans-
13
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
Problem Analysis type Example optimization Model Precision Recall F1
REACHABILITY Forwards, control ﬂow
only Dead code elimination DeepTuneIR 0.520 0.497 0.504
ProGraML 0.997 0.995 0.996
DOMTREE Backwards, control ﬂow
only Global Code Motion DeepTuneIR 0.721 0.081 0.138
ProGraML 0.985 0.693 0.781
DATADEP Backwards, control and
data ﬂow Instruction scheduling DeepTuneIR 0.999 0.136 0.236
ProGraML 1.000 0.988 0.993
LIVENESS Backwards, control and
data ﬂow Register allocation DeepTuneIR — — —
ProGraML 1.000 0.999 0.999
SUBEXPRESSIONS
Forwards, statement and
operand values and
positions
Global Common
Subexpression
Elimination
DeepTuneIR 1.000 0.123 0.214
ProGraML 0.965 0.925 0.930
Average — — DeepTuneIR 0.810 0.209 0.273
ProGraML 0.989 0.920 0.940
Table 3: Benchmark compiler analyses results using two approaches: (a) DeepTune IR, a sequential model adapted
to work at the LLVM-IR level for statement-level classiﬁcation, and (b) P ROGRAML, our approach. The relational
representation signiﬁcantly outperforms a sequential approach across all problems. For the Global Common Subex-
pressions analysis, DeepTuneIR achieved a higher precision than P ROGRAML by predicting only the root statement
as a component in a subexpression, avoiding false-positives.
former model uses the same hyperparameters as the GGNN. Additionally a batch size of 64, Dropout regularization
of 0.2 on the graph states and weight-sharing between adjacent pairs of layers is employed. The model dimension is
equal to the embedding size (200) and the hidden size of the feed-forward layers is 512. The self-attention layers use
5 heads [16]. Overall, the Transformer model has 5.6 million trainable parameters, around 1.7 million of which are in
the embedding layer.
6 Experimental Results
This section evaluates the performance and limitations of our approach for the three case studies described in Section 5,
and provides a comparison to state-of-the-art approaches. First we show that P ROGRAML, unlike prior state-of-the-
art approaches to machine learning over code, is capable of replicating core compiler analysis tasks that are key to
optimization. Second, we improve upon prior approaches to the task of heterogeneous device mapping. Finally, we set
a new state of the art in the difﬁcult task of classifying program algorithms, and ablate the contribution of the structure
and content of the PROGRAML representation.
6.1 Case Study A: Compiler Analysis
Table 3 summarizes the performance of the P ROGRAML approach when tasked with learning a suite of benchmark
compiler analysis, along with the performance of a state-of-the-art sequential approach, DeepTune IR. As a binary
classiﬁcation task, compiler analyses display an extreme class imbalance as only a small fraction of a program graph
is typically relevant to computing the result set of an analysis. On our datasets, an accuracy of 96.6% can be achieved
by always predicting the negative class. For this reason we report only binary precision, recall, and F1 scores with
respect to the positive class.
As can be seen in Table 3, the relational representation of our approach, coupled with learning through iterative
message passing, yields models that far outperform the state-of-the-art sequential approaches to modeling code. The
grouping of program tokens by statement performed by DeepTune IR offers a more restrictive classiﬁcation interface
than with PROGRAML (where data elements are also represented as graph vertices). As such, it is not able to perform
14
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
104 105 106
#. training graphs (log)
0.0
0.2
0.4
0.6
0.8
1.0
F1 score
Reachability
Domtree
Liveness
Datadep
Subexpressions
(a) DeepTuneIR
104 105 106
#. training graphs (log)
0.0
0.2
0.4
0.6
0.8
1.0
F1 score
Reachability
Domtree
Liveness
Datadep
Subexpressions
(b) PROGRAML
Figure 7: The F1 score of compiler analysis models on a 20k-graph validation set as a function of the number of
training graphs from 10k to 1M. Each model was given six hours to train on a machine with a GTX 1080 GPU, with
early termination if accuracy on the validation set reached 99.99%. We have applied a Gaussian ﬁlter ( σ = 1) to aid
in visualizing the trends in each set of scores.
the per-variable classiﬁcation required for L IVENESS analysis1. For fairness, we exclude L IVENESS from LSTM
aggregate values in Table 3. Despite this, PROGRAML achieves an average 94.0 F1, versus 27.3 F1 for DeepTuneIR.
In many cases, the sequential model regresses to a classiﬁcation mode in which only the source vertex is labeled as
positive, yielding poor recall. The PROGRAML models achieve both high precision and high recall in all tasks except
DOMTREE , where recall is comparatively poor. When model performance is considered as a function of the number of
training graphs, as shown in Figure 7, we see that the performance of the P ROGRAML models quickly convergences
towards near-perfect F1 score on a holdout validation set, except in the case of D OMTREE , where the model is still
improving at the end of training. This suggests estimating the transfer and meet operators of this backwards analysis
poses a greater challenge for the network, which may require further training.
Table 4 shows confusion matrices for the per-vertex classiﬁcation of P ROGRAML models on the test set. While the
distribution of errors is balanced for R EACHABILITY , in the case of D OMTREE and S UBEXPRESSIONS the ratio of
false negatives (T+P−) outweighs the false positives ( T−P+), indicating that models favor under-approximating the
value sets of these analyses. This may be an artifact of training with such a large imbalance towards negatives ( T−)
over positive (T+) class labels. In future work we will explore addressing this class imbalance by selecting multiple
root points on a graph for analysis simultaneously, thereby increasing the size of the value set to include multiple
(possibly overlapping) regions.
6.2 Case Study B: Heterogeneous Device Mapping
The performance of ProGraML and baseline models is shown in Table 5. We reused the pre-trained inst2vec-
embeddings for the NCC baseline that were published with the original work, however all models themselves have
been reimplemented to ensure fair comparison across different models under a uniﬁed evaluation regime and absolute
performance numbers can thus deviate from the original publications.
1Theoretically the same approach for grouping embeddings by statement could also be extended to group all statements by
variables, but this would require duplicating many statements in the input representation, increasing the length of the sequences far
beyond what we can practically process using a sequential model.
15
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
P− P+
T− 96.64% 0.01%
T+ 0.01% 3.34%
(a) REACHABILITY
P− P+
T− 95.94% 0.13%
T+ 1.70% 2.23%
(b) DOMTREE
P− P+
T− 98.80% 0.00%
T+ 0.00% 1.20 %
(c) DATADEP
P− P+
T− 90.00% 0.00%
T+ 0.00% 9.99%
(d) LIVENESS
P− P+
T− 99.12% 0.02%
T+ 0.13% 0.73%
(e) SUBEXPRESSIONS
Table 4: Confusion matrices for compiler analyses using P ROGRAML. Rows denote true negative ( T−) and true
positive (T+), columns denote predicted negative (P−) and predicted positive (P+). The value of a cell is the ratio of
per-vertex model outputs of this type, e.g. T−P+ is the ratio of false positives.
Accuracy Precision Recall F1
Static Mapping 58.8% 0.35 0.59 0.44
DeepTune [23] 71.9% 0.72 0.72 0.72
DeepTuneIR 73.8% 0.76 0.74 0.75
NCC [7] 80.3% 0.81 0.80 0.80
ProGraML 86.6% 0.89 0.87 0.88
(a) AMD
Accuracy Precision Recall F1
Static Mapping 56.9% 0.32 0.57 0.41
DeepTune [23] 61.0% 0.69 0.61 0.65
DeepTuneIR 68.4% 0.70 0.68 0.69
NCC [7] 78.5% 0.79 0.79 0.79
ProGraML 80.0% 0.81 0.80 0.80
(b) NVIDIA
Table 5: Five approaches to predicting heterogeneous device mapping: (a) Static Mapping (b) DeepTune [23], a
sequential model using tokenized OpenCL, (c) DeepTune IR, the same model adapted for tokenized LLVM-IR, (d)
NCC, which uses pre-trained statement embeddings, and (e) PROGRAML, our approach.
Baseline models were trained with hyperparameters from the original works. For the P ROGRAML results we used 6
layers in the GGNN corresponding to 6 timesteps of message propagation, while sharing parameters between even and
odd layers to introduce additional regularization of the weights. We ran a sweep of basic hyperparameters which led
us to use the pre-trained inst2vec statement embeddings [7] and to exclude the use of position representations. Both
of these hyperparameter choices help generalization by reducing the complexity of the model. This is not surprising
in light of the fact that the dataset only contains 680 samples derived from 256 unique programs. P ROGRAML was
trained with the Adam optimizer with default parameters, a learning rate of 10−3 and a batch size of 18,000 nodes
for 300 epochs (resulting in ca. 12000 iteration steps of the optimizer). Additionally we found dropout [41] with a
rate of 0.1 on the weights of the message propagation function to be beneﬁcial on the validation set as well. For the
PROGRAML result, we repeat the automated sweep for all hyperparameter conﬁgurations and picked the conﬁguration
with the best average validation performance. Performance on the unseen tenth of the data is reported.
As can be seen, PROGRAML outperforms prior approaches to this problem by all metrics (accuracy, precision, recall,
and F1), across both device datasets.
6.3 Case Study C: Algorithm Classiﬁcation
Table 6 summarizes the algorithm classiﬁcation accuracy results of our method and baselines.
Baseline Experiments Lifting the XFG graph representation from pretraining embeddings [7] to the high-level
task of algorithm classiﬁcation showed a strong improvement of performance. We found that jointly learning the
embeddings with the model from scratch (cf. XFG-rnd in Table 6) outperformed both ﬁxed inst2vec embeddings
(XFG-i2v) as well as ﬁnetuning of inst2vec embeddings (not shown). Both baselines are an improvement over inst2vec
and show that graph-based models are more suited to the task than models with less structure.
Next, we want to understand whether our particular graph-representation reached its design goals and can provide
additional improvement on POJ104.
PROGRAML Experiments To ablate the contribution of the tokenization from the performance boost provided
by the P ROGRAML representation itself, we include a GGNN-based structure-only baseline (denoted as GGNN-s)
of our approach, where the only information on each node is whether it represents a statement or an identiﬁer in
16
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
TBCNN [33] NCC [7] XFG ProGraML
Metric i2v rnd GGNN-s GGNN Transformer
Test Error [%] 6.0 5.17 4.56 4.29 3.87 3.78 3.33
Improvement [%] — 0.0 11.8 17.0 25.1 26.9 35.6
Table 6: Algorithm Classiﬁcation Error on POJ-104 [33]. The two XFG models are distinct only in their embedding
layers: XFG-i2v uses inst2vec embeddings, while XFG-rnd jointly learns the embeddings. The results denoted by
Surface Features and TBCNN are reproduced from Mou et al. [33].
the graph, but we refrain from tokenizing statements. We think that algorithm classiﬁcation is a problem that lends
itself especially well to judging the power of the representation structure, since most algorithms are well-deﬁned
independent of implementation details, such as datatypes.
The results in Table 6 show that the structure alone of our P ROGRAML representation is sufﬁcient to outperform the
XFG baselines and prior work on the task of algorithms classiﬁcation. Adding inst2vec statement tokenization further
improves performance. This suggests that there is room for improvement in performance by extending the graph
encoding method to achieve better vocabulary coverage and stronger generalization. We leave this to future work.
7 Related Work
In order to perform machine learning on programs, prior work has employed methods from Natural Language Process-
ing and represented programs as a sequence of lexical tokens [42, 43, 23]. However, it has been observed [44, 17, 22]
that it is critical to capture the structured nature of programs and syntactic (tree-based) as well as semantic (graph-
based) representations have been proposed [3, 9]. There is a line of research that considers program representations
based on Abstract Syntax Trees (ASTs): Dam et al. [45] annotate nodes in the AST with type information and employ
Tree-Based LSTMs [46] for program defect prediction. Both Raychev et al. [44] and Alon et al. [4, 22] use path-based
abstractions of the AST as program representations, while Allamanis et al. [17] augment ASTs with a hand-crafted
set of additional typed edges and use GGNNs [12] to learn downstream tasks related to variable naming. Another line
of research considers modelling binary similarity via control-ﬂow graphs (CFGs) with an adaptation of GNNs called
Graph Matching Networks [47].
The history of representing programs as graphs for optimization goes back to Program Dependence Graphs
(PDGs) [48], which remove superﬂuous control ﬂow edges to ease optimization with a compact graph representa-
tion. A more contemporary precursor of our PROGRAML representation ConteXtual Flow Graphs (XFGs) [7], which
combine control ﬂow with dataﬂow in order to learn unsupervised embeddings of LLVM-IR statements. While P RO-
GRAML is designed to extend the concepts of XFGs, PROGRAML preserve the notion of argument order and includes
nodes for both variables and constant values and all control ﬂow edges. These changes reﬂects the design goals of
the representations — XFGs are designed to easily express program semantics by omitting superﬂuous control rela-
tions and other execution-speciﬁc properties. PROGRAML, in combining CG, CFG, and DFG, offers a compiler-level
program representation that is designed to be useful for a variety of purposes from speciﬁc program analyses to down-
stream optimization tasks.
Another approach is taken by IR2Vec [49], which deﬁnes an LLVM-IR-speciﬁc statement representation that elegantly
models part-of-statements as relations. However, in order to compute the values of the embeddings, IR2Vec requires
access to the type of data ﬂow analyses that our approach is learning from data alone.
Brauckmann et al. independently proposed a graph-based representation based on Control and Data Flow Graphs
(CDFG) [9]. As in this work, CDFGs represent instructions as graph vertices and have bi-directional edges for control
and data relations. What differentiates P ROGRAML from CDFGs is the richness of the program representation:
CDFGs ignore the data elements of programs, and only instruction opcodes are used for vertex embeddings. The latent
representation of statements is thus invariant to instruction operands, their order, data types, and instruction modiﬁers.
This limits the representational power of the approach, e.g. by omitting code properties required for reasoning about
variables and expressions, as we explore through the L IVENESS and S UBEXPRESSIONS experiments in this work.
Finally, our approach provides76.6×improved inference throughput, though we suspect this may be an artifact of our
implementation as both approaches scale linearly w.r.t. the size of the graph and number of message passing steps.
Graph Neural Networks comprise a diverse class of neural networks that learn by propagating information along
edges [50, 51, 52]. Approaches based on Recurrent Neural Networks (RNNs) [12, 14] as well as convolutional [53, 54]
17
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
and attention-based methods [55, 56] exist. GNNs as a family have been shown to have enough expressive power to
address difﬁcult combinatorial problems like the graph isomorphism test [57] at least as well as the Weisfeiler-Lehman
algorithm [58]. Please refer to Battaglia et al. [52] for a comprehensive review of GNNs.
8 Conclusions
With the demand for aggressively optimizing compilers increasing, there is an increasing burden on compiler heuristics
to make good optimization decisions. While tuning heuristics by hand is expensive and slow to keep up with the pace of
compiler and architecture advancements, machine learning offers tremendous beneﬁts for automatically constructing
heuristics that are both cheaper to develop and better performing than hand-crafted equivalents. The success of these
machine learning approaches is bound by the quality of the input used to represent programs, and the ability of models
to process these representations.
In this work, we present a graph-based representation for programs, derived from compiler IRs, that accurately captures
the semantics of a program’s statements and the relations between them. Our approach is more expressive than prior
sequence- or graph-based representations, while closely approximating the representations that are traditionally used
within compilers.
We have shown through a constructivist approach that machine learning is capable of approximating the types of
compiler analyses that are key for optimizations. In testing our approach on a suite of established compiler tasks that
even state-of-the-art machine learning methods struggle with, our goal is to inspire conﬁdence in machine learning as a
viable tool for reasoning about program semantics, as opposed to a black box which discourages, rather than inspires, a
more systematic approach to reasoning about optimizations. When tasked with real-world problems spanning multiple
domains and source languages, our approach outperforms prior state-of-the-art approaches.
Our hope in developing PROGRAML is to provide a re-usable toolbox for representing and reasoning about programs
that can be used for a wide variety of downstream tasks. Promising research avenues for downstream tasks enabled by
our enriched program representation and the ability to perform statement-level inference include automatic paralleliza-
tion, static performance estimation, and IR-to-IR transpilation. Additionally, while the applications of deep learning
to compilers is rapidly evolving [3, 59], we hope to focus attention on the challenges that machine learning methods
face in the domain of programming languages: scalability when faced with large inputs, modeling very-long-range
dependencies, and learning over unbounded vocabularies.
References
[1] A. H. Ashouri, W. Killian, J. Cavazos, G. Palermo, and C. Silvano. A Survey on Compiler Autotuning using
Machine Learning. CSUR, 51(5), 2018.
[2] Z. Wang and M. O’Boyle. Machine learning in Compiler Optimization.Proceedings of the IEEE, 106(23), 2018.
[3] M. Allamanis, E. T. Barr, P. Devanbu, and C. Sutton. A Survey of Machine Learning for Big Code and Natural-
ness. CSUR, 51(4), 2018.
[4] U. Alon, M. Zilberstein, O. Levy, and E. Yahav. code2vec: Learning Distributed Representations of Code. In
POPL, 2018.
[5] P. Yin, G. Neubig, M. Allamanis, M. Brockschmidt, and A. L. Gaunt. Learning to Represent Edits.
arXiv:1810.13337, 2018.
[6] Ameer Haj-Ali, N. K. Ahmed, T. Willke, S. Shao, K. Asanovic, and I. Stoica. NeuroVectorizer: End-to-End
Vectorization with Deep Reinforcement Learning. In CGO, 2020.
[7] T. Ben-Nun, A. S. Jakobovits, and T. Hoeﬂer. Neural Code Comprehension: A Learnable Representation of
Code Semantics. In NeurIPS, 2018.
[8] A. Mirhoseini, H. Pham, Q. V . Le, B. Steiner, R. Larsen, Y . Zhou, N. Kumar, M. Norouzi, S. Bengio, and J. Dean.
Device Placement Optimization with Reinforcement Learning. In ICML, 2017.
[9] A. Brauckmann, S. Ertel, A. Goens, and J. Castrillon. Compiler-Based Graph Representations for Deep Learning
Models of Code. In CC, 2020.
[10] Y . Bengio, P. Simard, and P. Frasconi. Learning Long-Term Dependencies with Gradient Descent is Difﬁcult.
IEEE Transactions on Neural Networks, 5(2), 1994.
[11] M. McCloskey and N. J. Cohen. Catastrophic Interference in Connectionist Networks: The Sequential Learning
Problem. Psychology of Learning and Motivation, 24, 1989.
18
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
[12] Y . Li, R. Zemel, M. Brockscmidt, and D. Tarlow. Gated Graph Sequence Neural Networks. arXiv:1511.05493,
2015.
[13] M. Schlichtkrull, T. N. Kipf, P. Bloem, R. van den Berg, I. Titov, and M. Welling. Modeling Relational Data with
Graph Convolutional Networks. In ESWC, 2018.
[14] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural Message Passing for Quantum
Chemistry. In ICML. PMLR, 2017.
[15] Z. Chen and M. Monperrus. A Literature Study of Embeddings on Source Code. arXiv:1904.03061, 2019.
[16] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention
Is All You Need. In NIPS, 2017.
[17] M. Allamanis, M. Brockschmidt, and M. Khademi. Learning to Represent Programs with Graphs. In ICLR,
2017.
[18] C. Lattner and V . Adve. LLVM: A compilation framework for lifelong program analysis & transformation. In
CGO. IEEE, 2004.
[19] C. Leary and T. Wang. XLA: TensorFlow, compiled. In TensorFlow Dev Summit, 2017.
[20] C. Cummins, P. Petoumenos, W. Zang, and H. Leather. Synthesizing Benchmarks for Predictive Modeling. In
CGO. IEEE, 2017.
[21] E. Park, J. Cavazos, and M. A. Alvarez. Using Graph-Based Program Characterization for Predictive Modeling.
In CGO. IEEE, 2012.
[22] U. Alon, M. Zilberstein, O. Levy, and E. Yahav. A General Path-Based Representation for Predicting Program
Properties. In PLDI. ACM, 2018.
[23] C. Cummins, P. Petoumenos, Z. Wang, and H. Leather. End-to-end Deep Learning of Optimization Heuristics.
In PACT. IEEE, 2017.
[24] F. Barchi, G. Urgese, E. Macii, and A. Acquaviva. Code Mapping in Heterogeneous Platforms Using Deep
Learning and LLVM-IR. In DAC. ACM, 2019.
[25] V . J. Hellendoorn, C. Bird, E. T. Barr, and M. Allamanis. Deep Learning Type Inference. In ESEC/FSE, 2018.
[26] J. B. Kam and J. D. Ullman. Monotone Data Flow Analysis Frameworks. Acta Informatica, 7(3), 1977.
[27] K. D. Cooper, T. J. Harvey, and K. Kennedy. Iterative Data-ﬂow Analysis, Revisited. Technical report, Depart-
ment of Computer Science, Rice University, 2004.
[28] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y . N. Dauphin. Convolutional Sequence to Sequence Learning.
In ICML. PMLR, 2017.
[29] K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y . Bengio. Learning
Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In EMNLP, 2014.
[30] S. Ioffe and C. Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal
Covariate Shift. In ICML. PMLR, 2015.
[31] T. Lengauer and R. E. Tarjan. A Fast Algorithm for Finding Dominators in a Flow Graph. TOPLAS, 1(1), 1979.
[32] S. Blazy, D. Demange, and D. Pichardie. Validating Dominator Trees for a Fast, Veriﬁed Dominance Test. In
ITP, 2015.
[33] L. Mou, G. Li, L. Zhang, T. Wang, and Z. Jin. Convolutional Neural Networks over Tree Structures for Program-
ming Language Processing. In AAAI, 2016.
[34] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard,
M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V . Vasudevan, P. Warden,
M. Wicke, Y . Yu, and X. Zheng. TensorFlow: A System for Large-scale Machine Learning. InOSDI, 2016.
[35] T. Ben-Nun and T. Hoeﬂer. Demystifying parallel and distributed deep learning: An in-depth concurrency
analysis. ACM Computing Surveys, 52(4), 2019.
[36] D. P. Kingma and J. L. Ba. Adam: a Method for Stochastic Optimization. In ICLR, 2015.
[37] D. Grewe, Z. Wang, and M. O’Boyle. Portable Mapping of Data Parallel Programs to OpenCL for Heterogeneous
Systems. In CGO. IEEE, 2013.
[38] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. Improving Neural Networks
by Preventing Co-adaptation of Feature Detectors. arXiv:1207.0580, 2012.
[39] Z. Fisches. Neural Self-Supervised Models of Code. Masters thesis, ETH Zurich, 2020.
19
PROGRAML: Graph-based Deep Learning for Program Optimization and Analysis
[40] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[41] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A Simple Way to Prevent
Neural Networks from Overﬁtting. JMLR, 15, 2014.
[42] M. Allamanis and C. Sutton. Mining Source Code Repositories at Massive Scale using Language Modeling. In
MSR, 2013.
[43] M. Allamanis. Learning Natural Coding Conventions. PhD thesis, University of Edinburgh, 2016.
[44] V . Raychev, M. Vechev, and A. Krause. Predicting Program Properties from "Big Code". InPOPL, 2015.
[45] H. K. Dam, J. Grundy, T. Kim, and C. Kim. A Deep Tree-Based Model for Software Defect Prediction.
arXiv:1802.00921, 2018.
[46] K. S. Tai, R. Socher, and C. D. Manning. Improved Semantic Representations From Tree-Structured Long
Short-Term Memory Networks. arXiv:1503.00075, 2015.
[47] L. Li, C. Gu, T. Dullien, O. Vinyals, and P. Kohli. Graph Matching Networks for Learning the Similarity of
Graph Structured Objects. In ICML. PMLR, 2019.
[48] J. Ferrante, K. J. Ottenstein, and J. D. Warren. The Program Dependence Graph and Its Use in Optimization.
TOPLAS, 9(3), 1987.
[49] V . Keerthy S, R. Aggarwal, S. Jain, M. S. Desarkar, R. Upadrasta, and Y . N. Spkant. IR2Vec: A Flow Analysis
based Scalable Infrastructure for Program Encodings. arXiv:1909.06228, 2019.
[50] M. Gori, G. Monfardini, and F. Scarselli. A New Model for Learning in Graph Domains. In IJCNN. IEEE, 2005.
[51] S. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. The Graph Neural Network Model.
IEEE Transactions on Neural Networks, 20(1), 2009.
[52] P. Battaglia, J. Hamrick, V . Bapst, A. Sanchez-Gonzalez, V . Zambaldi, M. Malinowski, A. Tacchetii, D. Raposo,
A. Santoro, R. Faulkner, C. Gulcehre, F. Song, A. Ballard, J. Gilmer, G. Dahl, A. Vaswani, K. Allen, C. Nash,
V . Langston, C. Dyer, N. Heess, D. Wierstra, P. Kohli, M. Botvinick, O. Vinyals, Y . Li, and R. Pascanu. Relational
Inductive Biases, Deep Learning, and Graph Networks. arXiv:1806.01261, 2018.
[53] M. Defferrard, X. Bresson, and P. Vandergheynst. Convolutional Neural Networks on Graphs with Fast Localized
Spectral Filtering. In NIPS, 2016.
[54] T. N. Kipf and M. Welling. Semi-supervised Classiﬁcation with Graph Convolutional Networks.
arXiv:1609.02907, 2017.
[55] P. Velickovic, G. Cucurull, A. Casanova, A. Romera, P. Lio, and Y . Bengio. Graph Attention Networks. InICLR,
2018.
[56] G. Wang, R. Ying, J. Huang, and J. Leskovec. Improving Graph Attention Networks with Large Margin-based
Constraints. arXiv:1910.11945, 2019.
[57] K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How Powerful are Graph neural Networks? In ICLR, 2019.
[58] B. Weisfeiler and A. A. Lehman. A Reduction of a Graph to a Canonical Form and an Algebra Arising During
this Reduction. Nauchno-Technicheskaya Informatsia, 2(9), 1968.
[59] C. Cummins. Deep Learning for Compilers. PhD thesis, University of Edinburgh, 2020.
20

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
0018-9219 © 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.  
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Proceedings of the IEEE  1
ABSTRACT  |  I n the last decade, machine-learning-based 
compilation has moved from an obscure research niche to a 
mainstream activity. In this paper, we describe the relationship 
between machine learning and compiler optimization and 
introduce the main concepts of features, models, training, 
and deployment. We then provide a comprehensive survey 
and provide a road map for the wide variety of different 
research areas. We conclude with a discussion on open issues 
in the area and potential research directions. This paper 
provides both an accessible introduction to the fast moving 
area of machine-learning-based compilation and a detailed 
bibliography of its main achievements.
KEYWORDS
 |
 C
ode optimization; compiler; machine learning; 
program tuning
I.  I NTRODUCTION
“Why would anyone want to use machine learning to build 
a compiler?” It is a view expressed by many colleagues over 
the last decade. Compilers translate programming languages 
written by humans into binary executable by computer hard-
ware. It is a serious subject studied since the 1950s [1]–[3] 
where correctness is critical and caution is a by-word. 
Machine learning, on the other hand, is an area of artificial 
intelligence (AI) aimed at detecting and predicting patterns. 
It is a dynamic field looking at subjects as diverse as galaxy 
classification [4] to predicting elections based on Tweeter 
feeds [5]. When an open-source machine learning compiler 
was announced by IBM in 2009 [6], some wry slashdot com-
mentators picked up on the AI aspect, predicting the start of 
sentient computers, global net, and the war with machines 
from the Terminator film series.
In fact, as we will see, in this paper, that compilers and 
machine learning are a natural fit and have developed into 
an established research domain.
Digital Object Identifier: 10.1109/JPROC.2018.2817118
A. It Is All About Optimization
Compilers have two jobs—translation and optimi-
zation. First, they must translate programs into binary 
correctly. Second, they have to find the most efficient 
translation possible. There are many different correct 
translations whose performance varies significantly. The 
vast majority of research and engineering practices is 
focused on this second goal of performance, traditionally 
misnamed optimization. The goal was misnamed because 
in most cases, until recently, finding an optimal transla-
tion was dismissed as being too hard to find and an unreal-
istic endeavor.
1 Instead it focused on developing compiler 
heuristics to transform the code in the hope of improving 
performance but could in some instances damage it.
Machine learning predicts an outcome for a new data 
point based on prior data. In its simplest guise, it can be 
considered a form of interpolation. This ability to pre-
dict based on prior information can be used to find the 
data point with the best outcome and is closely tied to 
the area of optimization. It is at this overlap of looking 
at code improvement as an optimization problem and 
machine learning as a predictor of the optima where we 
find machine learning compilation.
Optimization as an area, machine learning based or 
otherwise, has been studied since the 1800s [8], [9]. An 
interesting question is therefore why has the convergence 
of these two areas taken so long? There are two funda-
mental reasons. First, despite the year-on-year increasing 
potential performance of hardware, software is increas-
ingly unable to realize it leading to a software gap. This 
gap has yawned right open with the advent of multicores 
(see also Section VI-B). Compiler writers are looking for 
new ways to bridge this gap.
Second, computer architecture evolves so quickly 
that it is difficult to keep up. Each generation has new 
quirks and compiler writers are always trying to play 
catchup. Machine learning has the desirable property 
of being automatic. Rather than relying on expert com-
piler writers to develop clever heuristics to optimize 
the code, we can let the machine learn how to optimize 
a compiler to make the machine run faster, an approach 
Manuscript received October 30, 2017; accepted January 23, 2018.  
(Corresponding author: Michael O'Boyle.)
Z. Wang is with the MetaLab, School of Computing and Communications, Lancaster 
University, Lancaster LA1 4WA, U.K. (e-mail: z.wang@lancaster.ac.uk).
M. O'Boyle is with the School of Informatics, University of Edinburgh, Edinburgh EH8 
9AB, U.K. (e-mail: mob@inf.ed.ac.uk).
Machine Learning in 
Compiler Optimization
By Z heng  Wa ng a n d M ich a el  O’B Oy l e
1In fact, the term superoptimizer [7] was coined to describe 
systems that tried to find the optimum.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
2 Proceedings of the IEEE
sometimes referred to as autotuning [10]–[13]. Machine 
learning is, therefore, ideally suited to making any code 
optimization decision where the performance impact 
depends on the underlying platform. As described later in 
this paper, it can be used for topics ranging from selecting 
the best compiler flags to determining how to map paral-
lelism to processors.
Machine learning is part of a tradition in computer sci-
ence and compilation in increasing automation The 1950s 
to 1970s were spent trying to automate compiler translation, 
e.g., lex for lexical analysis [14] and yacc for parsing [15]; the 
last decade by contrast has focused on trying to automate 
compiler optimization. As we will see, it is not “magic” or a 
panacea for compiler writers, rather it is another tool allow-
ing automation of tedious aspects of compilation providing 
new opportunities for innovation. It also brings compilation 
nearer to the standards of evidence-based science. It intro-
duces an experimental methodology where we separate out 
evaluation from design and considers the robustness of solu-
tions. Machine-learning-based schemes, in general, have 
the problem of relying on black boxes whose working we do 
not understand and hence trust. This problem is just as true 
for machine-learning-based compilers. In this paper, we 
aim to demystify machine-learning-based compilation and 
show it is a trustworthy and exciting direction for compiler 
research.
The remainder of this paper is structured as follows. 
First, we give an intuitive overview for machine learning 
in compilers in Section II. Then, we describe how machine 
learning can be used to search for or to directly predict good 
compiler optimizations in Section III. This is followed by a 
comprehensive discussion in Section IV for a wide range of 
machine learning models that have been employed in prior 
work. Next, in Section V, we review how previous work 
chooses quantifiable properties, or features, to represent 
programs. We discuss the challenges and limitations for 
applying machine learning to compilation, as well as open 
research directions in Section VII before we summarize and 
conclude in Section VIII.
II.  O VERVIEW OF MACHINE LEARNING 
IN COMPILERS
Given a program, compiler writers would like to know 
what compiler heuristic or optimization to apply in order 
to make the code better. Better often means execute faster, 
but can also mean smaller code footprint or reduced 
power. Machine learning can be used to build a model 
used within the compiler that makes such decisions for 
any given program.
There are two main stages involved: learning and 
deployment. The first stage learns the model based on train-
ing data, while the second uses the model on new unseen 
programs. Within the learning stage, we need a way of rep-
resenting programs in a systematic way. This representation 
is known as the program features [16].
Fig. 1 gives an intuitive view on how machine learning 
can be applied to compilers. This process, which includes 
feature engineering, learning a model, and deployment, is 
described in the following sections.
A. Feature Engineering
Before we can learn anything useful about programs, we 
first need to be able to characterize them. Machine learn-
ing relies on a set of quantifiable properties, or features, to 
characterize the programs [Fig. 1(a)]. There are many dif-
ferent features that can be used. These include the static 
data structures extracted from the program source code 
or the compiler intermediate representation (such as the 
number of instructions or branches), dynamic profiling 
information (such as performance counter values) obtained 
through runtime profiling, or a combination of the both.
Standard machine learning algorithms typically work on 
fixed length inputs, so the selected properties will be sum-
marized into a fixed length feature vector. Each element of 
the vector can be an integer, real or Boolean value. The pro-
cess of feature selection and tuning is referred to as feature 
engineering. This process may need to iteratively perform 
multiple times to find a set of high-quality features to build 
Fig. 1. A generic view of supervised machine learning in compilers. (a) Feature engineering. (b) Leaning a model. (c) Deployment.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
Proceedings of the IEEE  3
an accurate machine learning model. In Section V, we pro-
vide a comprehensive review of feature engineering for the 
topic of program optimization.
B. Learning a Model
The second step is to use training data to derive a model using 
a learning algorithm. This process is depicted in Fig. 1(b) . Unlike 
other applications of machine learning, we typically generate 
our own training data using existing applications or bench-
marks. The compiler developer will select training programs 
which are typical of the application domain. For each training 
program, we calculate the feature values, compiling the pro-
gram with different optimization options, and running and 
timing the compiled binaries to discover the best performing 
option. This process produces, for each training program, a 
training instance that consists of the feature values and the 
optimal compiler option for the program.
The compiler developer then feeds these examples to a 
machine learning algorithm to automatically build a model. 
The learning algorithm’s job is to find from the training 
examples a correlation between the feature values and the 
optimal optimization decision. The learned model can then 
be used to predict, for a new set of features, what the opti-
mal optimization option should be.
Because the performance of the learned model strongly 
depends on how well the features and training programs are 
chosen, the processes of featuring engineering and training 
data generation often need to repeat multiple times.
C. Deployment
In the final step, the learned model is inserted into the 
compiler to predict the best optimization decisions for new 
programs. This is demonstrated in Fig. 1(c) . To make a pre-
diction, the compiler first extracts the features of the input 
program, and then feeds the extracted feature values to the 
learned model to make a prediction.
The advantage of the machine-learning-based approach 
is that the entire process of building the model can be easily 
repeated whenever the compiler needs to target a new hard-
ware architecture, operating system, or application domain. 
The model built is entirely derived from experimental 
results and is hence evidence based.
D. Example
As an example to illustrate these steps, consider thread 
coarsening [18] for GPU programs. This code transforma-
tion technique works by giving multiple work items (or 
work elements) to one single thread. It is similar to loop 
unrolling, but applied across parallel work items rather than 
across serial loop iterations.
Fig. 2(a) shows a simple OpenCL kernel where a thread 
operates on a work item of the 1-D input array, in, at a time. 
The work item to be operated on is specified by the value 
returned from the OpenCL get_global_id() API. Fig. 2(b) 
shows the transformed code after applying a thread coarsen 
factor of two, where each thread processes two elements of 
the input array.
Thread coarsening can improve performance through 
increasing instruction-level parallelism [19], reducing the 
number of memory-access operations [20] and eliminating 
redundant computation when the same value is computed 
in every work item. However, it can also have several nega-
tive side effects, such as reducing the total amount of paral-
lelism and increasing the register pressure, which can lead 
to slowdown performance. Determining when and how 
to apply thread coarsening is nontrivial, because the best 
coarsening factor depends on the target program and the 
hardware architecture that the program runs on [17], [19].
Magni et al. show that machine learning techniques 
can be used to automatically construct effective thread-
coarsening heuristics across GPU architectures [17]. Their 
approach considers six coarsening factors (
 1,
 2,
 4
,
 8
,
 16
,
 32
 ). 
The 
goal is to develop a machine-learning-based model to 
decide whether an OpenCL kernel should be coarsened on a 
specific GPU architecture and, if so, what is the best coars-
ening factor. Among many machine learning algorithms, 
they chose to use an artificial neural network to model
2 
the problem. Construing such a model follows the classical 
three-step supervised learning process, which is depicted in 
Fig. 1 and described in more details as follows.
1)
 Feature Engineering:
 To describe the input OpenCL 
kernel, Magni et al. use static code features extracted from 
the compiler’s intermediate representation. Specifically, they 
developed a compiler-based tool to obtain the feature values 
from the program’s LLVM bitcode [21]. They started from 17 
candidate features. These include things like the number of 
2In fact, Magni et al.  employed a hierarchical approach consisting of 
multiple artificial neural networks [17]. However, these networks are 
trained using the same process.
Fig. 2. An OpenCL thread coarsening example reproduced from 
[17]. The original OpenCL code is shown in (a) where each thread 
takes the square of one element of the input array. When coarsened 
by a factor of two, as shown in (b), each thread now processes two 
elements of the input array.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
4 Proceedings of the IEEE
and types of instructions and memory level parallelism (MLP) 
within an OpenCL kernel. Table 1 gives the list of candidate fea-
tures used in [17]. Typically, candidate features can be chosen 
based on developers’ intuitions, suggestions from prior works, 
or a combination of both. After choosing the candidate fea-
tures, a statistical method called principal component analysis 
(PCA; see also Section IV-B) is applied to map the 17 candidate 
features into seven aggregated features, so that each aggregated 
feature is a linear combination of the original features. This 
technique is known as “feature dimension reduction,” which 
is discussed in Section V-D2. Dimension reduction helps 
eliminating redundant information among candidate features, 
allowing the learning algorithm to perform more effectively.
2)
 Learning 
the Model: For the work presented in [17], 
16 OpenCL benchmarks were used to generate training 
data. To find out which of the six coarsening factors per-
forms best for a given OpenCL kernel on a specific GPU 
architecture, we can apply each of the six factors to an 
OpenCL kernel and record its execution time. Since the 
optimal thread-coarsening factor varies across hardware 
architectures, this process needs to repeat for each target 
architecture. In addition to finding the best performing 
coarsening factor, Magni et al. also extracted the aggregated 
feature values for each kernel. Applying these two steps on 
the training benchmarks results in a training data set where 
each training example is composed of the optimal coars-
ening factor and feature values for a training kernel. The 
training examples are then fed into a learning algorithm 
which tries to find a set of model parameters (or weights) so 
that overall prediction error on the training examples can 
be minimized. The output of the learning algorithm is an 
artificial neural network model where its weights are deter-
mined from the training data.
3)
 
Deployment: The learned model can then be used to 
predict the optimal coarsening factor for unseen OpenCL 
programs. To do so, static source code features are first 
extracted from the target OpenCL kernel; the extracted 
feature values are then fed into the model which decides 
whether to coarsen or not and which coarsening factor 
should be used. The technique proposed in [17] achieves an 
average speedup between 1.11x and 1.33x across four GPU 
architectures and does not lead to degraded performance on 
a single benchmark.
III.  M ETHODOLOGY
One of the key challenges for compilation is to select the 
right code transformation, or sequence of transformations 
for a given program. This requires effectively evaluating the 
quality of a possible compilation option, e.g., how a code 
transformation will affect eventual performance.
A naive approach is to exhaustively apply each legal 
transformation option and then profile the program to  
collect the relevant performance metric. Given that many 
compiler problems have a massive number of options, 
exhaustive search and profiling is infeasible, prohibiting the 
use of this approach at scale. This search-based approach 
to compiler optimization is known as iterative compila-
tion [22], [23] or autotuning [10], [24]. Many techniques 
have been proposed to reduce the cost of searching a large 
space [25], [26]. In certain cases, the overhead is justifiable 
if the program in question is to be used many times, e.g., 
in a deeply embedded device. However, its main limitation 
remains: it only finds a good optimization for one program 
and does not generalize into a compiler heuristic.
There are two main approaches for solving the problem 
of scalably selecting compiler options that work across pro-
grams. A high level comparison of both approaches is given 
in Fig. 3. The first strategy attempts to develop a cost (or pri-
ority) function to be used as a proxy to estimate the quality 
of a potential compiler decision, without relying on exten-
sive profiling. The second strategy is to directly predict the 
best performing option.
A. Building a Cost Function
Many compiler heuristics rely on a cost function to esti-
mate the quality of a compiler option. Depending on the 
optimization goal, the quality metric can be execution time, 
the code size, or energy consumption, etc. Using a cost func-
tion, a compiler can evaluate a range of possible options to 
choose the best one, without needing to compile and profile 
the program with each option.
Table 1  Candidate Code Features Used in [17]
Fig. 3. There are, in general, two approaches to determine the 
optimal compiler decision using machine learning. The first one 
is to learn a cost or priority function to be used as a proxy to 
select the best performing option (a). The second one is to learn a 
predictive model to directly predict the best option (b).
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
Proceedings of the IEEE  5
1) T he Problem of Handcrafted Heuristics: Trad -
i
tionally, a compiler cost function is manually crafted. For 
example, a heuristic of function inlining adds up a num -
ber of relevant metrics, such as the number of instruc -
tions of the target function to be inlined, the callee and 
stack size after inlining, and compare the resulted value 
against a predefined threshold to determine if it is prof -
itable to inline a function [27]. Here, the importance or 
weights for metrics and the threshold are determined 
by compiler developers based on their experience or via 
“trail-and-error.” Because the efforts involved in tuning 
the cost function are so expensive, many compilers simply 
use “one-size-fits-all” cost function for inlining. However, 
such a strategy is ineffective. For examples, Cooper et al.  
show that a “one-size-fits-all” strategy for inlining often 
delivers poor performance [28]; other studies also show 
that the optimal thresholds to use to determine when to 
inline change from one program to the other [29], [30].
Handcrafted cost functions are widely used in compil-
ers. Other examples include the work conducted by Wagner 
et al. [31] and Tiwari et al. [32]. The former combines a 
Markov model and a human-derived heuristic to statically 
estimate the execution frequency of code regions (such 
as function innovation counts). The latter calculates the 
energy consumption of an application by assigning a weight 
to each instruction type. The efficiency of these approaches 
highly depends on the accuracy of the estimations given by 
the manually tuned heuristic.
The problem of relying on a hand-tuned heuristic is 
that the cost and benefit of a compiler optimization often 
depend on the underlying hardware; while handcrafted 
cost functions could be effective, manually developing 
one can take months or years on a single architecture. 
This means that tuning the compiler for each newly 
released processor is hard and is often infeasible due to 
the drastic efforts involved. Because cost functions are 
important and manually tuning a good function is dif-
ficult for each individual architecture, researchers have 
investigated ways to use machine learning to automate 
this process.
In Section III-A2, we review a range of previous studies 
on using machine learning to tune cost functions for per-
formance and energy consumption—many of which can be 
applied to other optimization targets such as the code size 
[33] or a tradeoff between energy and runtime.
2)
 Cost 
Functions for Performance: The Meta 
Optimization framework [34] uses genetic programming 
(GP) to search for a cost function 
 y
   ⃪ f
 (
x)
 , 
which takes in a 
feature vector 
 x
  and produces a real-valued p
riority  y . 
Fig. 4 
depicts the workflow of the framework. This approach is eval-
uated on a number of compiler problems, including hyper-
block formation,
3 register allocation, and data prefetching, 
showing that machine learned cost functions outperform 
human-crafted ones. A similar approach is employed by 
Cavazos et al. who find cost functions for performance and 
compilation overhead for a Java just-in-time compiler [35]. 
The COLE compiler [36] uses a variance of the GP algorithm 
called strength Pareto evolutionary algorithm 2 (SPEA2) 
[37] to learn cost functions to balance multiple objectives 
(such as program runtime, compilation overhead, and code 
size). In Section IV-C, we describe the working mechanism 
of GP-like search algorithms.
Another approach to tune the cost functions is to pre-
dict the execution time or speedup of the target program. 
The Qilin compiler [38] follows such an approach. It uses 
curve fitting algorithms to estimate the runtime for execut-
ing the target program of a given input size on the CPU 
and the GPU. The compiler then uses this information to 
determine the optimal loop iteration partition across the 
CPU and the GPU. The Qilin compiler relies on an applica-
tion-specific function which is built on a per program base 
using reference inputs. The curve fitting (or regression; see, 
also, Section IV) model employed by the Qilin compiler 
can model with continuous values, making it suitable for 
estimating runtime and speedup. In [39], this approach is 
extended, which developed a relative predictor that predicts 
whether an unseen predictor will improve significantly on a 
GPU relative to a CPU. This is used for runtime scheduling 
of OpenCL jobs.
The early work conduced by Brewer proposed a regres-
sion-based model to predict the execution of a data layout 
scheme for parallelization, by considering three parameters 
[40]. Using the model, his approach can select the optimal 
3Hyperblock formation combines basic blocks from multiple control 
paths to form a predicated, larger code block to expose instruction level 
parallelism.
Fig. 4. A simple view of the GP approach presented in [34] for tuning compiler cost functions. Each candidate cost function is represented 
as an expression tree (a). The workflow of the GP algorithm is presented in (b).
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
6 Proceedings of the IEEE
layout for over 99% of the time for a partial differential equa-
tion (PDE) solver across four evaluation platforms. Other 
previous works also use curve fitting algorithms to build a 
cost function to estimate the speedup or runtime of sequen-
tial [41]–[43], OpenMP [44]–[46], and, more recently, deep 
learning applications [47].
3)
 Cost Functions for Energy Consumption: 
In addi-
tion to performance, there is an extensive body of work 
that investigates ways to build energy models for software 
optimization and hardware architecture design. As power or 
energy readings are continuous real values, most of the prior 
work on power modeling uses regression-based approaches.
Linear regression is a widely used technique for energy 
modeling. Benini et al. developed a linear-regression-based 
model to estimate power consumption at the instruction 
level [48]. The framework presented by Rethinagiri et al.  
[49] uses parameterized formulas to estimate power con-
sumption of embedded systems. The parameters of the for-
mulas are determined by applying a regression-based algo-
rithm to reference data obtained with handcrafted assembly 
code and power measurements. In a more recent work, 
Schürmans et al. also adopt a regression-based method for 
power modeling [50], but the weights of the regression 
model are determined using standard benchmarks instead 
of handwritten assembly programs.
Other works employ the artificial neural network (ANN) 
to automatically construct power models. Curtis-Maury et 
al. develop an ANN-based model to predict the power con-
sumption of OpenMP programs on multicore systems [51]. 
The inputs to the model are hardware performance coun-
ter values such as the cache miss rate, and the output is 
the estimated power consumption. Su et al. adopt a similar 
approach by developing an ANN predictor to estimate the 
runtime and power consumption for mapping OpenMP pro-
grams on nonuniform memory access (NUMA) multicores. 
This approach is also based on runtime profiling of the target 
program, but it explicitly considers NUMA-specific infor-
mation like local and remote memory accesses per cycle.
B. Directly Predicting the Best Option
While a cost function is useful for evaluating the quality 
of compiler options, the overhead involved in searching for 
the optimal option may still be prohibitive. For this reason, 
researchers have investigated ways to directly predict the 
best compiler decision using machine learning for relatively 
small compilation problems.
Monsifrot et al. pioneered the use of machine learning to 
predict the optimal compiler decision [16]. This work devel-
oped a decision-tree-based approach to determine whether it 
is beneficial to unroll a loop based on information such as the 
number of statements and arithmetic operations of the loop. 
Their approach makes a binary decision on whether to unroll 
a loop but not how many times the loop should be unrolled. 
Later, Stephenson and Amarasinghe advanced [16] by directly 
predicting the loop unroll factor [52] by considering eight 
unroll factors 
 (1,
 2,
 …,
 8
 )
 . 
They formulated the problem as a 
multiclass classification problem (i.e., each loop unroll factor 
is a class). They used over 2500 loops from 72 benchmarks to 
train two machine learning models [a nearest neighbor and 
a support vector machine (SVM) model] to predict the loop 
unroll factor for unseen loops. Using a richer set of features 
than [16], their techniques correctly predict the unroll fac-
tor for 65% of the testing loops, leading to, on average, a 5% 
improvement for the SPEC 2000 benchmark suite.
For sequential programs, there is extensive work in pre-
dicting the best compiler flags [53], [54], code transforma-
tion options [55], or tile size for loops [56], [57]. This level 
of interest is possibly due to the restricted nature of the 
problem, allowing easy experimentation and comparison 
against prior work.
Directly predicting the optimal option for parallel pro-
grams is harder than doing it for sequential programs, due to 
the complex interactions between the parallel programs and 
the underlying parallel architectures. Nonetheless, there 
are works on predicting the optimal number of threads to 
be used to run an OpenMP program [46], [58], the best 
parameters to be used to compile a CUDA programs for a 
given input [59], and the thread coarsening parameters for 
OpenCL programs for GPUs [17]. These papers show that 
supervised machine learning can be a powerful tool for 
modeling problems with a relatively small number of opti-
mization options.
I V. M ACHINE LEARNING MODELS
In this section, we review the wide range of machine learning 
models used for compiler optimization. Table 2 summarizes 
the set machine learning models discussed in this section.
There are two major subdivisions of machine learn-
ing techniques that have previously been used in compiler 
optimizations: supervised and unsupervised learning. Using 
supervised machine learning, a predictive model is trained 
on empirical performance data (labeled outputs) and impor-
tant quantifiable properties (features) of representative 
programs. The model learns the correlation between these 
feature values and the optimization decision that delivers the 
optimal (or near-optimal) performance. The learned correla-
tions are used to predict the best optimization decisions for 
new programs. Depending on the nature of the outputs, the 
predictive model can be either a regression model for con-
tinuous outputs or a classification model for discrete outputs.
In the other subdivision of machine learning, termed 
unsupervised learning, the input to the learning algorithm is 
a set of input values merely—there is no labeled output. One 
form of unsupervised learning is clustering which groups the 
input data items into several subsets. For example, SimPoint 
[60], a simulation technique, uses clustering to pick repre-
sent program execution points for program simulation. It 
does so by first dividing a set of program runtime information 
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
Proceedings of the IEEE  7
into groups (or clusters), such that points within each clus-
ter are similar to each other in terms of program structures 
(loops, memory usages, etc.); it then chooses a few points 
of each cluster to represent all the simulation points within 
that group without losing much information.
There are also techniques that sit at the boundary of super-
vised and unsupervised learning. These techniques refine the 
knowledge gathered during offline learning or previous runs 
using empirical observations obtained during deployment. 
We review such techniques in Section IV-C. This sections 
concludes with a discussion of the relative merits of different 
modeling approaches for compiler optimization.
A. Supervised Learning
1) Regression: A widely used supervised learning tech -
nique is called regression. This technique has been used in 
various tasks, such as predicting the program execution time 
input [38] or speedup [39] for a given input, or estimating 
the tail latency for parallel workloads [61].
Regression is essentially curve fitting. As an example, 
consider Fig. 5 where a regression model is learned from 
five data points. The model takes in a program input size 
 X
 
 
and predicts the execution time of the program  Y
  . Adhering 
to 
supervised learning nomenclature, the set of five known 
data points is the training data set and each of the five points 
that comprise the training data is called a training example. 
Each training example 
 (
 x
 i  ,  y i  )  is defined by a feature vector 
(i.e., the input size in our case) 
  x
 i    and a desired output (i.e., 
the program execution time in our case) 
  y
 i    . Learning in this 
context is understood as discovering the relation between 
the inputs (
  x
 i   ) and the outputs (   y i   ) so that the predictive 
model can be used to make predictions for any new, unseen 
input features in the problem domain. Once the function 
 f
 
 
is in place, one can use it to make a prediction by taking in a 
new input feature vector 
 x
  
. The prediction  y
  
is the value of 
the curve that the new input feature vector 
 x
  corresponds to.
There 
are a range of machine learning techniques that 
can be used for regression. These include the simple linear 
regression model and more advanced models like SVMs and 
ANNs. Linear regression is effective when the input (i.e., 
feature vectors) and output (i.e., labels) have a strong linear 
relation. SVM and ANNs can model both linear and nonlin-
ear relations, but typically require more training examples 
to learn an effective model when compared with simple lin-
ear regression models.
Table 3 gives some examples of regression techniques 
that have been used in prior work for code optimization and 
the problem to be modeled.
2)  Classification: Supervised classification is another 
technique that has been widely used in prior work of machine-
learning-based code optimization. This technique takes in a 
feature vector and predicts which of a set of classes the feature 
vector is associated with. For example, classification can be 
used to predict which of a set of unroll factors should be used 
for a given loop, by taking in a feature vector that describes the 
characteristics of the target loop (see also Section II-D).
The k-nearest neighbur (KNN) algorithm is a simple 
yet effective classification technique. It finds the 
 k
 
 closet 
training examples to the input instance (or program) on the 
feature space. The closeness (or distance) is often evaluated 
using the Euclidean distance, but other metrics can also be 
used. This technique has been used to predict the optimal 
optimization parameters in prior works [52], [66], [67]. It 
Table 2  Machine Learning Methods Discussed in Section IV
Fig. 5. A simple regression-based curve-fitting example. There are 
five training examples in this case. A function 
 f
  is tr
ained with the 
training data, which maps the input 
 x
  t
o the output 
 y
 . The tr
ained 
function can predict the output of an unseen 
 x
 
.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
8 Proceedings of the IEEE
works by first predicting which of the training programs 
are closet (i.e., nearest neighbors) to the incoming program 
on the feature space; it then uses the optimal parameters 
(which are found during training time) of the nearest neigh-
bors as the prediction output. While it is effective on small 
problems, KNN also has two main drawbacks. First, it must 
compute the distance between the input and all training data 
at each prediction. This can be slow if there is a large num-
ber of training programs to be considered. Second, the algo-
rithm itself does not learn from the training data; instead, 
it simply selects the 
 k
  
nearest neighbors. This means that 
the algorithm is not robust to noisy training data and could 
choose an ill-suited training program as the prediction.
As an alternative, the decision tree has been used in 
prior works for a range of optimization problems. These 
include choosing the parallel strategy for loop parallelization 
[69], determining the loop unroll factor [16], [70], decid-
ing the profitability of using GPU acceleration [68], [71], 
and selecting the optimal algorithm implementation [72]. 
The advantage of a decision tree is that the learned model is 
interpretable and can be easily visualized. This enables users 
to understand why a particular decision is made by follow-
ing the path from the root node to a leaf decision node. For 
example, Fig. 6 depicts the decision tree model developed in 
[68] for selecting the best performing device (CPU or GPU) 
to run an OpenCL program. To make a prediction, we start 
from the root of the tree; we compare a feature value (e.g., the 
communication–computation ratio) of the target program 
against a threshold to determine which branch of the tree to 
follow; and we repeat this process until we reach a leaf node 
where a decision will be made. It is to note that the structure 
and thresholds of the tree are automatically determined by 
the machine learning algorithm, which may change when we 
target a different architecture or application domain.
Decision trees make the assumption that the feature 
space is convex, i.e., it can be divided up using hyperplanes 
into different regions, each of which belongs to a different 
category. This restriction is often appropriate in practice. 
However, a significant drawback of using a single decision 
tree is that the model can overfit due to outliers in the train-
ing data (see also Section IV-D). Random forests [73] have, 
therefore, been proposed to alleviate the problem of over-
fitting. Random forests are an ensemble learning method  
[74]. As illustrated in Fig. 7, it works by constructing mul-
tiple decision trees at training time. The prediction of each 
tree depends on the values of a random vector sampled inde-
pendently on the feature value. In this way, each tree is ran-
domly forced to be insensitive to some feature dimensions. 
To make a prediction, random forests then aggregate the out-
comes of individual trees to form an overall prediction. It has 
been employed to determine whether to inline a function or 
not [75], delivering better performance than a single-model-
based approach. We want to highlight that random forests 
can also be used for regression tasks. For instances, it has 
been used to model energy consumption of OpenMP [76] 
and CUDA [77] programs.
Logical regression is a variation of linear regression but 
is often used for classification. It takes in the feature vec-
tor and calculates the probability of some outcome. For 
example, Cavazos and O’Boyle used logical regression to 
determine the optimization level of Jike RVM. Like decision 
trees, logical regression also assumes that the feature values 
and the prediction has a linear relation.
More advanced models, such as SVM classification, 
have been used for various compiler optimization tasks 
[46], [79]–[81]. SVMs use kernel functions to compute the 
similarity of feature vectors. The radial basis function (RBF) 
is commonly used in prior works [46], [82] because it can 
model both linear and nonlinear problems. It works by map-
ping the input feature vector to a higher dimensional space 
where it may be easier to find a linear hyperplane to well 
separate the labeled data (or classes).
Other machine learning techniques, such as kernel 
canonical correlation analysis and naive Bayes, have also 
Fig. 6. A decision tree for determining which device (CPU or GPU) to use to run an OpenCL program. This diagram is reproduced from [68].
Table 3  Regression Techniques Used in Prior Works
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
Proceedings of the IEEE  9
been used in prior works to predict stencil program configu-
rations [83] or detect parallel patterns [84].
3)
 Deep Neural Networks: 
In recent years, deep neural 
networks [85] have been shown to be a powerful tool for 
tackling a range of machine learning tasks such as image rec-
ognition [86], [87] and audio processing [88]. Deep neural 
networks (DNNs) have recently been used to model program 
source code [89] for various software engineering tasks (see 
also Section VI-C), but so far there is little work of apply-
ing DNNs to compiler optimization. A recent attempt in 
this direction is the DeepTune framework [78], which uses 
DNNs to extract source code features (see also Section V-C).
The advantage of DNNs is that they can compactly rep-
resent a significantly larger set of functions than a shallow 
network, where each function is specialized at processing 
part of the input. This capability allows DNNs to model the 
complex relationship between the input and the output (i.e., 
the prediction). As an example, consider Fig. 8 that visual-
izes the internal state of DeepTune [78] when predicting 
the optimal thread coarsening factor for an OpenCL kernel 
(see Section II-D). Fig. 8(a) shows the first 80 elements of 
the input source code tokens as a heatmap in which each 
cell’s color reflects an integer value assigned to a specific 
token. Fig. 8(b) shows the neurons of the first DNN for 
each of the four GPU platforms, using a red–blue heatmap 
to visualize the intensity of each activation. If we have a 
close look at the heatmap, we can find a number of neurons 
in the layer with different responses across platforms. This 
indicates that the DNN is partly specialized to the target 
platform. As information flows through the network [layers 
(c) and (d) in Fig. 8], the layers become progressively more 
specialized to the specific platform.
B. Unsupervised Learning
Unlike supervised learning models which learn a correla-
tion from the input feature values to the corresponding out-
puts, unsupervised learning models only take it from the input 
data (e.g., the feature values). This technique is often used to 
model the underlying structure of distribution of the data.
Clustering is a classical unsupervised learning problem. 
The k-means clustering algorithm [90] groups the input data 
into 
 k
  
clusters. For example, in Fig. 9, a k-means algorithm 
is used to group data points into three clusters on a 2-D 
Fig. 7. Random forests are an ensemble learning algorithm. It 
aggregates the outputs of multiple decision trees to form a final 
prediction. The idea is to combine the predictions from multiple 
individual models together to make a more robust, accurate 
prediction than any individual model.
Fig. 8. A simplified view of the internal state for the DeepTune DNN framework [78] when it predicts the optimal OpenCL thread coarsening 
factor. Here, a DNN is learned for each of the four target GPU architectures. The activations in each layer of the four models increasingly 
diverge (or specialize) toward the lower layers of the model. It is to note that some of the DeepTune layers are omitted to aid presentation.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
10 Proceedings of the IEEE
feature space. The algorithm works by grouping data points 
that are close to each other on the feature space into a clus-
ter. K-means is used to characterize program behavior [60], 
[91]. It does so by clustering program execution into phase 
groups, so that we can use a few samples of a group to rep-
resent the entire program phases within a group. K-means 
is also used in the work presented in [92] to summarize the 
code structures of parallel programs that benefit from simi-
lar optimization strategies. In addition to k-means, Martins 
et al. employed the fast Newman clustering algorithm [93] 
which works on network structures to group functions that 
may benefit from similar compiler optimizations [94].
PCA is a statistical method for unsupervised learning. 
This method has been heavily used in prior work to reduce 
the feature dimension [17], [25], [95]–[97]. Doing so allows 
us to model a high-dimensional feature space with a smaller 
number of representative variables which, in combination, 
describe most of the variability found in the original feature 
space. PCA is often used to discover the common pattern in 
the data sets in order to help clustering exercises. It is used 
to select representative programs from a benchmark suite 
[95], [98]. In Section V-D, we discuss PCA in further details.
Autoencoders are a recently proposed artificial neural 
network architecture for discovering the efficient codings of 
input data in an unsupervised fashion [99]. This technique 
can be used in combination of a natural language model to 
first extract features from program source code and then find 
a compact representation of the source code features [100]. 
We discuss autoencoders in Section V-D when reviewing 
feature dimensionality reduction techniques.
C. Online Learning
1) Evolutionary Search: Evolutionary algorithms (EAs) 
or evolutionary computation such as genetic algorithms 
(GAs), GP,
4 and stochastic-based search have been employed 
to find a good optimization solution from a large search space. 
An EA applies principles inspired by biological evolution to 
find an optimal or near-optimal solution for the target prob-
lem. For instance, the SPIRAL autotuning framework uses a 
stochastic evolutionary search algorithm to choose a fast for-
mula (or transformation) for signal processing applications 
[101]. Li et al. use GAs to search for the optimal configura-
tion to determine which sorting algorithm to use based on 
the unsorted data size [102]. The Petabricks compiler offers 
a more general solution by using EAs to search for the best 
performing configurations for a set of algorithms specified 
by the programmer [103]. In addition to code optimization, 
EAs have also been used to create Pareto optimal program 
benchmarks under various criteria [104].
As an example, consider how an EA can be employed in 
the context of iterative compilation to find the best com-
piler flags for a program [25], [36], [105]. Fig. 10 depicts 
how an EA can be used for this purpose. The algorithm 
starts from several populations of randomly chosen com-
piler flag settings. It compiles the program using each indi-
vidual compiler flag sequence, and uses a fitness function 
to evaluate how well a compiler flag sequence performs. In 
our case, a fitness function can simply return the recipro-
cal of a program runtime measurement, so that compiler 
settings that give faster execution time will have a higher 
fitness score. In the next epoch, the EA algorithm generates 
Fig. 9. Using k-means to group data points into three clusters. In 
this example, we group the data points into three clusters on a 2-D 
feature space.
4A GA is represented as a list of actions and values, often a string, 
while a GP is represented as a tree structure of actions and values. For 
example, GP is applied to the abstract syntax tree of a program to search 
for useful features in [70].
Fig. 10. Using an EA to perform iterative compilation. The 
algorithm starts from several initial populations of randomly 
chosen compiler flag sequences. It evaluates the performance of 
individual sequences to remove poorly performing sequences in 
each population. It then applies crossover and mutation to create 
a new generation of populations. The algorithm returns the best 
performing program binary when it terminates.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
Proceedings of the IEEE  11
the next populations of compiler settings via mechanisms 
such as reproduction (crossover) and mutation among 
compiler flag settings. This results in a new generation of 
compiler flag settings and the quality of each setting will 
be evaluated again. In a mechanism analogous to natural 
selection, a certain number of poorly performing compiler 
flags within a population are chosen to die in each genera-
tion. This process terminates when no further improve-
ment is observed or the maximum number of generations is 
reached, and the algorithm will return the best found pro-
gram binary as a result.
There are three key operations in an EA algorithm: selec-
tion, crossover, and mutation. The probability of an opti-
mization option being selected for dying is often inversely 
proportional to its fitness score. In other words, options that 
are relatively fitter (e.g., give faster program runtime) are 
more likely to survive and remain a part of the population 
after selection. In crossover, a certain number of offsprings 
are produced by mixing some existing optimization options 
(e.g., compiler flags). The likelihood of an existing option 
being chosen for crossover is again proportional to its fit-
ness. This strategy ensures that good optimizations will be 
preserved over generations, while poorly performing opti-
mizations will gradually die out. Finally, mutation randomly 
changes a preserved optimization, e.g., by turning on/off 
an option or replacing a threshold value in a compiler flag 
sequence. Mutation reduces the chance that the algorithm 
gets stuck with a locally optimal optimization.
EAs are useful for exploring a large optimization space 
where it is infeasible to just enumerate all possible solu-
tions. This is because an EA can often converge to the most 
promising area in the optimization space quicker than a 
general search heuristic. The EA is also shown to be faster 
than a dynamic-programming-based search [24] in finding 
the optimal transformation for the fast Fourier transforma-
tion (FFT) [101]. When compared to supervised learning, 
EAs have the advantage of requiring little problem-specific 
knowledge, and hence they can be applied on a broad range 
of problems. However, because an EA typically relies on the 
empirical evidences (e.g., running time) for fitness evalu-
ation, the search time can still be prohibitively expensive. 
This overhead can be reduced by using a machine-learning-
based cost model [43] to estimate the potential gain (e.g., 
speedup) of a configuration (see also Section III-A). Another 
approach is to combine supervised learning and EAs [25], 
[106] by first using an offline learned model to predict the 
most promising areas of the design space (i.e., to narrow 
down the search areas), and then searching over the pre-
dicted areas to refine the solutions. Moreover, instead of 
predicting where in the search space to focus on, one can 
also first prune the search space to reduce the number of 
options to search over. For example, Jantz and Kulkarni 
show that the search space of phase ordering
5 can be greatly 
reduced if we can first remove phases whose application 
order is irrelevant to the produced code [107]. Their tech-
niques are claimed to prune the exhaustive phase order 
search space size by 89% on average.
2)
 Reinforcement Learning: 
Another class of online 
learning algorithms is reinforcement learning (RL) which 
is sometimes called “learning from interactions.” The algo-
rithm tries to learn how to maximize the rewards (or perfor-
mance) itself. In other words, the algorithm needs to learn, 
for a given input, what the correct output or decision to take 
is. This is different from supervised learning where the cor-
rect input/output pairs are presented in the training data.
Fig. 11 illustrates the working mechanism of RL. Here 
the learning algorithm interacts with its environment over a 
discrete set of time steps. At each step, the algorithm evalu-
ates the current state of its environment, and executes an 
action. The action leads to a change in the state of the envi-
ronment (which the algorithm can evaluate in the next time 
step), and produces an immediate reward. For example, in 
a multitasking environment, a state could be the CPU con-
tention; when processor cores are idle, an action could be 
where to place a process, and a reward could be the overall 
system throughput. The goal of RL is to maximize the long-
term cumulative reward by learning an optimal strategy to 
map states to actions.
RL is particularly suitable for modeling problems that 
have an evolving natural, such as dynamic task scheduling, 
where the optimal outcome is achieved through a series of 
actions. RL has been used in prior research to schedule RAM 
memory traffics [108], select software component configu-
rations at runtime [109], and configure virtual machines 
[110]. An early work of using RL for program optimization 
was conduced by Lagoudakis and Littman [111]. They use 
RL to find the cutoff point to switch between two sorting 
algorithms: 
quickSort and insertionSort. CALOREE 
combines machine learning and control theories to sched-
ule CPU resources on heterogeneous multicores [112]. For a 
given application, CALOREE uses control-theoretic methods 
to dynamically adjust the resource allocation, and machine 
learning to estimate the application’s latency and power for 
a given resource allocation plan (to offer decision supports).
An interesting RL-based approach for scheduling paral-
lel OpenMP programs is presented in [113]. This approach 
predicts the best number of threads for a target OpenMP 
5Compiler phase ordering determines at which order a set of 
compiler optimization passes should be applied to a given program.
Fig. 11. The working mechanism of reinforcement learning.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
12 Proceedings of the IEEE
program when it runs with other competing workloads, 
aiming to make the target program run faster. This approach 
first learns a reward function offline based on static code 
features and runtime system information. The reward func-
tion is used to estimate the reward of a runtime scheduling 
action, i.e., the expected speedup when assigning a certain 
number of processor cores to an OpenMP program. In the 
next scheduling epoch, this approach uses the empiri-
cal observation of the application speedup to check if the 
reward function was accurate and the decision was good, 
and update the reward function if the model is found to be 
inaccurate.
In general, RL is an intuitive and comprehensive solu-
tion for autonomous decision making. But its performance 
depends on the effectiveness of the value function, which 
estimates the immediate reward. An optimal value function 
should lead to the greatest cumulative reward in the longer 
term. For many problems, it is difficult to design an effective 
value function or policy, because the function needs to fore-
see the impact of an action in the future. The effectiveness of 
RL also depends on the environment; if the number of pos-
sible actions is large, it can take RL a long time to converge 
to a good solution. RL also requires the environment to be 
fully observed, i.e., all the possible states of the environment 
can be anticipated ahead of time. However, this assumption 
may not hold in a dynamic computing environment due 
to unpredictable disturbances, e.g., changes in application 
inputs or application mixes. In recent years, deep learning 
techniques have been used in conjunction with RL to learn 
a value function. The combined technique is able to solve 
some problems that were deemed impossible in the past 
[114]. However, how to combine deep learning with RL to 
solve compilation and code optimization problems remains 
an open question.
D. Discussion
What model is best is the $64 000 question. The answer 
is: it depends. More sophisticated techniques may provide 
greater accuracy but they require large amounts of labeled 
training data—a real problem in compiler optimization. 
Techniques such as linear regression and decision trees 
require less training data compared to more advanced mod-
els such as SVMs and ANNs. Simple models typically work 
well when the prediction problem can be described using a 
feature vector that has a small number of dimensions, and 
when the feature vector and the prediction are linearly cor-
related. More advanced techniques such as SVMs and ANNs 
can model both linear and nonlinear problems on a higher 
dimensional feature space, but they often require more 
training data to learn an effective model. Furthermore, the 
performance of an SVM and an ANN also highly depends on 
the hyperparameters used to train the model. The optimal 
hyperparameter values can be chosen by performing cross 
validation on the training data. However, how to select 
parameters to avoid overfitting while achieving a good pre-
diction accuracy remains an outstanding challenge.
Choosing which modeling technique to use is nontrivial. 
This is because the choice of model depends on a number 
of factors: the prediction problem (e.g., regression or clas-
sification), the set of features to use, the available train-
ing examples, the training and prediction overhead, etc. 
In prior works, the choice of modeling technique largely 
relied on developer experience and empirical results. Many 
of the studies in the field of machine-learning-based code 
optimization do not fully justify the choice of the model, 
although some do compare the performance of alternate 
techniques. The OpenTuner framework addresses the prob-
lem by employing multiple techniques for program tuning 
[115]. OpenTuner runs multiple search techniques at the 
same time. Techniques which perform well will be given 
more candidate tuning options to examine, while poorly 
performed algorithms will be given fewer choices or disa-
bled entirely. In this way, OpenTuner can discover which 
algorithm works best for a given problem during search.
One technique that has seen little investigation is the use 
of Gaussian processes [116]. Before the recent widespread 
interest in DNNs, these were a highly popular method in 
many areas of machine learning [117]. They are particularly 
powerful when the amount of training data is sparse and 
expensive to collect. They also automatically give a confi-
dence interval with any decision. This allows the compiler 
writer to trade off risk versus reward depending on the 
application scenario.
Using a single model has a significant drawback in prac-
tice. This is because a one-size-fits-all model is unlikely to 
precisely capture behaviors of diverse applications, and no 
matter how parameterized the model is, it is highly unlikely 
that a model developed today will always be suited for 
tomorrow. To allow the model to adapt to the change of the 
computing environment and workloads, ensemble learning 
was exploited in prior works [73], [118], [119]. The idea of 
ensemble learning is to use multiple learning algorithms, 
where each algorithm is effective for particular problems, to 
obtain better predictive performance than could be obtained 
from any of the constituent learning algorithm alone [120], 
[121]. Making a prediction using an ensemble typically 
requires more computational time than doing that using a 
single model, so ensembles can be seen as a way to com-
pensate for poor learning algorithms by performing extra 
computation. To reduce the overhead, fast algorithms such 
as decision trees are commonly used in ensemble methods 
(e.g., random forests), although slower algorithms can ben-
efit from ensemble techniques as well.
V. F EATURE ENGINEERING
Machine-learning-based code optimization relies on hav-
ing a set of high-quality features that capture the important 
characteristics of the target program. Given that there is an 
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
Proceedings of the IEEE  13
unbounded number of potential features, finding the right 
set is a nontrivial task. In this section, we review how pre-
vious work chooses features, a task known as feature engi-
neering. Tables 4 and 5 summarize the range of program 
features and feature engineering techniques discussed in 
this section, respectively.
A. Feature Representation
Various forms of program features have been used in 
compiler-based machine learning. These include static code 
structures [122] and runtime information such as system 
load [118], [123] and performance counters [53].
1)
 Static Code Features: 
Static program features such 
as the number and type of instructions are often used to 
describe a program. These features are typically extracted 
from the compiler intermediate representations [29], [46], 
[52], [80] in order to avoid using information extracted from 
dead code. Table 6 gives some of the static code features that 
were used in previous studies. Raw code features are often 
used together to create a combined feature. For example, one 
can divide the number of load instructions by the number of 
total instructions to get the memory load ratio. An advantage 
of using static code features is that the features are readily 
available from the compiler intermediate representation.
2)
 Tree- and 
Graph-Based Features: Singer and Veloso 
represent the FFT in a split tree [124]. They extract from the 
tree a set of features, by counting the number of nodes of var-
ious types and quantifying the shape of the tree. These tree-
based features are then used to build a neural-network-based 
cost function that predicts which of the two FFT formulas 
runs faster. The cost function is used to search for the best 
performing transformation.
Park et al. present a unique graph-based approach for 
feature representations [125]. They use an SVM where the 
kernel is based on a graph similarity metric. Their technique 
requires hand-coded features at the basic block level, but 
thereafter, graph similarity against each of the training pro-
grams takes the place of global features. Mailike shows that 
spatial-based information, i.e., how instructions are distrib-
uted within a program, extracted from the program’s data 
flow graph could be a useful feature for machine-learning-
based compiler optimization [126]. Nobre et al. also exploit 
graph structures for code generation [26]. Their approach 
targets the phase ordering problem. The order of compiler 
optimization passes is represented as a graph. Each node of 
the graph is an optimization pass and connections between 
nodes are weighted in a way that subsequences with higher 
aggregated weights are more likely to lead to faster runtime. 
The graph is automatically constructed and updated using 
iterative compilation (where the target program is complied 
using different compiler passes with different orders). A 
design space exploration algorithm is employed to drive the 
iterative compilation process.
3)
 Dynamic Features: 
While static code features are 
useful and can be extracted at static compile time (hence 
feature extraction has no runtime overhead), they have 
drawbacks. For examples, static code features may con-
tain information of code segments that rarely get executed, 
and such information can confuse the machine learning 
model; some program information such as the loop bound 
depends on the program input, which can only be obtained 
during execution time; and static code features often may 
not precisely capture the application behavior in the runt-
ime environment [such as resource contention and input/
output (I/O) behavior] as such behavior highly depends on 
the computing environment such as the number of available 
processors and corunning workloads.
As illustrated in Fig. 12, dynamic features can be 
extracted from multiple layers of the runtime environment. 
At the application layer, we can obtain information such 
as loop iteration counts that cannot be decided at compile 
time, dynamic control flows, frequently executed code 
regions, etc. At the operating system level, we can observe 
the memory and I/O behavior of the application as well 
as CPU load and thread contention, etc. At the hardware 
Table 4  Summary of Features Discussed in Section V
Table 5  Feature Engineering Techniques Discussed in Section V.
Table 6  Example Code Features Used in Prior Works
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
14 Proceedings of the IEEE
level, we can use performance counters to track information 
such as how many instructions have been executed and of 
what types, and the number of cache loads/stores as well as 
branch misses, etc.
Hardware performance counter values, such as executed 
instruction counts and cache miss rate, are therefore used 
to understand the application’s dynamic behaviors [53], 
[127], [128]. These counters can capture low-level pro-
gram information such as data access patterns, branches, 
and computational instructions. One of the advantages of 
performance counters is that they capture how the target 
program behaves on a specific hardware and avoid the irrel-
evant information that static code features may bring in. In 
addition to hardware performance counters, operating sys-
tem level metrics, such as system load and I/O contention, 
are also used to model an application’s behavior [39], [123]. 
Such information can be externally observed without instru-
menting the code, and can be obtain during offline profiling 
or program execution time.
While effective, collecting dynamic information could 
incur prohibitively overhead and the collected information 
can be noisy due to competing workloads and operating sys-
tem scheduling [129] or even subtle settings of the execu-
tion environment [130]. Another drawback of performance 
counters and dynamic features is that they can only capture 
the application’s past behavior. Therefore, if the applica-
tion behaves significantly different in the future due to the 
change of program phases or inputs, then the prediction will 
be drawn on an unreliable observation. As such, dynamic 
and static features are often used in combination in prior 
works in order to build a robust model.
B. Reaction-Based Features
Cavazos et al. present a reaction-based predictive model 
for software–hardware codesign [131]. Their approach pro-
files the target program using several carefully selected com-
piler options to see how program runtime changes under 
these options for a given microarchitecture setting. They 
then use the program “reactions” to predict the best avail-
able application speedup. Fig. 13 illustrates the difference 
between a reaction-based model and a standard program 
feature-based model. A similar reaction-based approach is 
used in [132] to predict speedup and energy efficiency for 
an application that is parallelized thread-level speculation 
(TLS) under a given microarchitectural configuration. Note 
that while a reaction-based approach does not use static 
code features, developers must carefully select a few set-
tings from a large number of candidate options for profiling, 
because poorly chosen options can significantly affect the 
quality of the model.
C. Automatic Feature Generation
As deriving good features is a time-consuming task, a 
few methods have been proposed to automatically gener-
ate features from the compiler’s intermediate representa-
tion (IR) [70], [133]. The work of [70] uses GP to search 
for features, but required a huge grammar to be written, 
some 160 kB in length. Although much of this can be cre-
ated from templates, selecting the right range of capabili-
ties and search space bias is nontrivial and up to the expert. 
The work of [133] expresses the space of features via logic 
programming over relations that represent information 
from the IRs. It greedily searches for expressions that rep-
resent good features. However, their approach relies on 
expert selected relations, combinators, and constraints 
to work. Both approaches closely tie the implementation 
of the predictive model to the compiler IR, which means 
changes to the IR will require modifications to the model. 
Furthermore, the time spent in searching features could be 
significant for these approaches.
The first work to employ neural network to extract fea-
tures from program source code for compiler optimization 
was conducted by Cummins et al. [78]. Their system, namely 
DeepTune, automatically abstracts and selects appropri-
ate features from the raw source code. Unlike prior work 
Fig. 12. Dynamic features can be extracted from multiple layers of 
the computing environment.
Fig. 13. Standard feature-based modeling (a) versus reaction-based 
modeling (b). Both models try to predict the speedup for a given 
compiler transformation sequence. The program feature-based 
predictor takes in static program features extracted from the 
transformed program, while the reaction-based model takes in the 
target transformation sequence and the measured speedups of the 
target program, obtained by applying a number of carefully selected 
transformation sequences. Diagrams are reproduced from [131].
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
Proceedings of the IEEE  15
where the predictive model takes in a set of human-crafted 
features, program code is used directly in the training data. 
Programs are fed through a series of neural-network-based 
language models which learn how the code correlates with 
the desired optimization options (see also Fig. 8). Their 
work also shows that the properties of the raw code that 
are abstracted by the top layers of the neural networks are 
mostly independent of the optimization problem. While 
promising, it is worth mentioning that dynamic informa-
tion such as the program input size and performance coun-
ter values are often essential for characterizing the behavior 
of the target program. Therefore, DeepTune does not com-
pletely remove human involvement for feature engineering 
when static code features are insufficient for the optimiza-
tion problem.
D. Feature Selection and Dimension Reduction
Machine learning uses features to capture the essential 
characteristics of a training example. Sometimes we have 
too many features. As the number of features increases, so 
does the number of training examples needed to build an 
accurate model [134]. Hence, we need to limit the dimen-
sion of the feature space. In compiler research, commonly, 
an initial large, high-dimensional candidate feature space is 
pruned via feature selection [52], or projected into a lower 
dimensional space [17]. In this section, we review a number 
of feature selection and dimension reduction methods.
1)
 Feature Selection: 
Feature selection requires under-
standing how a particular feature affects the prediction 
accuracy. One of the simplest methods for doing this is 
applying the Pearson correlation coefficient. This metric 
measures the linear correlation between two variables and is 
used in numerous works [55], [92], [122], [135] to filter out 
redundant features by removing features that have a strong 
correlation with an already selected feature. It has also been 
used to quantify the relation of the select features in regres-
sion. One obvious drawback of using Pearson correlation as 
a feature ranking mechanism is that it is only sensitive to a 
linear relationship.
Another approach for correlation estimation is mutual 
information [131], [136], which quantifies how much infor-
mation of one variable (or feature) can be obtained through 
another variable (feature). Like correlation coefficient, 
mutual information can be used to remove redundant fea-
tures. For example, if the information of feature 
 x
  can be 
largely obtained through another existing feature 
 y
 
, feature 
 
x
  
can then be taken out from the feature set without losing 
much information on the reduced feature set.
Both correlation coefficient and mutual information 
evaluate each feature independently with respect to the pre-
diction. A different approach is to utilize regression analysis 
for feature ranking. The underlying principal of regression 
analysis is that if the prediction is the outcome of regres-
sion model based on the features, then the most important 
features should have the highest weights (or coefficients) in 
the model, while features uncorrelated with the output vari-
ables should have weights close to zero. For example, least 
absolute shrinkage and selection operator (LASSO) regres-
sion analysis is used in [137] to remove less useful features 
to build a compiler-based model to predict performance. 
LASSO has also been used for feature selection to tune the 
compiler heuristics for the TRIPS processor [138].
In general, feature selection remains an open problem 
for machine learning, and researchers often follow a “trail-
and-error” approach to test a range of methods and feature 
candidates. This makes automatic feature selection frame-
work like FEAST [139] and HERCULES [140] attractive. 
The former framework employs a range of existing feature 
selection methods to select useful candidate features, while 
the latter searches for the most important static code fea-
tures from a set of predefined patterns for loops.
2)
  Feature Dimensionality Reduction: While feature 
selection allows us to select the most important features, 
the resulted feature set can still be too large to train a good 
model, especially when we only have a small number of 
training examples. By reducing the number of dimensions, 
the learning algorithm can often perform more efficiently 
on a limited training data set. Dimension reduction is also 
important for some machine learning algorithms such as 
KNN to avoid the effect of the curse of dimensionality [141].
PCA is a well-established feature reduction technique 
[142]. It uses orthogonal linear transformations to reduce the 
dimensionality of a set of variables, i.e., features in our case.
Fig. 14 demonstrates the use of PCA to reduce the num-
ber of dimensions. The input in this example is a 3-D space 
defined by 
 
 M
 1   ,   M  2   , and   M  3   , as shown in Fig. 14(a). Three 
components, 
 P
  C
 1   ,  P   C  2   , and  P   C  3   , which account for the vari-
ance of the data, are first calculated. Here,  P
  C
 1    and  P   C  2    con-
tribute most to the variance of the data and 
 P
  C
 3    accounts 
for the least variance. Using only  P
  C
 1    and  P   C  2   , one can 
transform the original, 3-D space into a new, 2-D coordinate 
Fig. 14. Using PCA to reduce dimensionality of a 3-D feature space. 
The principal components are first computed (a). Then, the first two 
principal components (
 P
  C
 
1
   
 and  
P
  C
 2
   ) ar
e selected to represent the 
original 3-D feature space on a new 2-D space (b). 
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
16 Proceedings of the IEEE
system [as illustrated in Fig. 14(b)] while preserving much 
of the variance of the original data.
PCA has been used in many prior compiler research 
works for feature reduction [17], [25], [55], [92], [95]–[97], 
[143]. It has also been used in prior works to visualize the 
working mechanism of a machine learning model, e.g., to 
show how benchmarks can be grouped in the feature space 
[123], by projecting features from a high-dimensional space 
into a 2-D space.
We want to stress that PCA does not select some fea-
tures and discard the others. Instead, it linearly combines 
the original features to construct new features that can sum-
marize the list of the original features. PCA is useful when 
there is some redundancy in the raw features, i.e., some of 
the features are correlated with one another. Similar feature 
reduction methods include factor analysis and linear discri-
minant analysis (LDA), which all try to reduce the number 
of features by linearly combining multiple raw features. 
However, PCA seems to be the most popular feature reduc-
tion method used in compiler research, probably due to its 
simplicity.
An alternative way of reducing the number of features 
used is via an autoencoder [144]. It is a neural network 
that finds a representation (encoding) for a set of data, by 
dimensionality reduction. Autoencoders works by learning 
an encoder and a decoder from the input data. The encoder 
tries to compress the original input into a low-dimensional 
representation, while the decoder tries to reconstruct the 
original input based on the low-dimension representations 
generated by the encoder. As a result, the autoencoder has 
been widely used to remove the data noise as well as to 
reduce the data dimension [145].
Autoencoders have been applied to various natural lan-
guage processing tasks [99], often being used together with 
DNNs. Recently, it has been employed to model program 
source code to obtain a compact set of features that can 
characterize the input program source [78], [146]–[149].
VI.  S COPE
Machine learning has been used to solve a wide range of 
problems, from the early successful work of selecting com-
piler flags for sequential programs, to recent works on 
scheduling and optimizing parallel programs on heteroge-
neous multicores. In this section, we review the types of 
problems that have been exploited in prior works.
A. Optimizing Sequential Programs
Early works for machine learning in compilers look at 
how, or if, a compiler optimization should be applied to a 
sequential program. Some of the previous studies build 
supervised classifiers to predict the optimal loop unroll fac-
tor [52], [70] or to determine whether a function should be 
inlined [29], [35]. These works target a fixed set of compiler 
options, by representing the optimization problem as a mul-
ticlass classification problem, where each compiler option 
is a class. For example, Leather et al. [70] considered a 
loop unroll factor between 0 and 15 (16 configurations in 
total), treating each candidate unroll factor as a class; they 
compiled and profiled each training program by trying all 
16 configurations to find out the best loop unroll factor for 
each program, and then learned a decision tree model from 
the training data.
There are other compiler problems where the number of 
possible options is massive. For instance, the work presented 
in [55] considers 54 code transformations of GCC. While 
these options are only a subset from the over hundreds of 
transformations provided by GCC, the resulted combinato-
rial compiler configurations lead to a space of approximately 
 
 
10
   34  . Although it is possible to build a classifier to directly 
predict the optimal setting from a large space, to learn an 
effective model would require a large volume of training 
programs in order to have an adequate sampling over the 
space. Doing so is difficult because 1) there are only a few 
dozen common benchmarks available; and 2) compiler 
developers need to generate the training data themselves.
EAs such as generic search are often used to explore a 
large design space (see also Section IV-C1). Prior works have 
used EAs to solve the phase ordering problem (i.e., at which 
order a set of compiler transformations should be applied) 
[150]–[152], determining the compiler flags during iterative 
compilation [153]–[156], selecting loop transformations 
[157], tuning algorithmic choices [11], [103], etc.
B. O ptimizing Parallel Programs
How to effectively optimize parallel programs has 
received significant attentions in the past decade, largely 
because the hardware industry has adopted multicore 
design to avoid the power wall [158]. While multicore and 
many-core architectures provide the potential for high-
performance and energy-efficient computing, the potential 
performance can only be unlocked if the application pro-
grams are suitably parallel and can be made to match the 
underlying heterogeneous platform. Without this, the myr-
iad cores on multicore processors and their specialized pro-
cessing elements will sit idle or poorly utilized. To this end, 
researchers have extended the reach of machine learning to 
optimize parallel programs.
A line of research in parallel program optimization is 
parallelism mapping. That is, given an already parallelized 
program, how to map the application parallelism to match 
the underlying hardware to make the program run as fast 
as possible or be as energy efficient as possible. Zhang et 
al. developed a decision-tree-based approach to predict the 
scheduling policy to use for an OpenMP parallel region 
[159]. The work presented in [46] employs two machine 
learning techniques to predict the optimal number of 
threads as well as the scheduling policy to use for OpenMP 
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
Proceedings of the IEEE  17
parallel loop. Specifically, it uses a regression-based ANN 
model to predict the speedup of a parallel loop when it 
runs with a given number of threads (to search for the opti-
mal number threads), and an SVM classifier to predict the 
scheduling policy. There are also works that use machine 
learning to determine the optimum degree of parallelism for 
transactional memory [160] and hardware source allocation 
[161], or to select a code version from a pool of choices to 
use [162]. Castro et al. developed a decision tree classifier 
to predict the thread mapping strategy in the context of soft-
ware transactional memory [163]. Jung et al. constructed an 
ANN-based predictor to select an effective data structure on 
a specific microarchitecture [164].
The work presented in [92] and [165] is a unique 
approach for applying machine learning to map complex 
parallel programs with unbounded parallel graph structures. 
The work considers the question of finding the optimal graph 
structure of a streaming program. The idea was that rather 
than trying to predict a sequence of transformations over an 
unbounded graph, where legality and consistency is a real 
problem, we should consider the problem from the dual 
feature space. The work showed that it is possible to pre-
dict the best target feature (i.e., the characteristics that an 
ideal transformed program should have) which then can be 
used to evaluate the worth of candidate transformed graphs 
(without compiling and profiling the resulted graphs) in the 
original feature space.
The Petabricks project [103], [166], [167] takes an evolu-
tionary approach for program tuning. The Petabricks com-
piler employs genetic search algorithms to tune algorithmic 
choices. Due to the expensive overhead of the search, much 
of autotuning is done at static compile time. Their work 
shows that one can utilize the idle processors on a multi-
core systems to perform online tuning [168], where half of 
the cores are devoted to a known safe program configura-
tion, while the other half are used for an experimental pro-
gram configuration. In this way, when the results of the 
faster configuration are returned, the slower version will be 
terminated.
The idea of combining compile-time knowledge and 
runtime information to achieve better optimizations has 
been exploited by the ADAPT compiler [169]. Using the 
ADAPT compiler, users describe what optimizations are 
available and provide heuristics for applying these optimi-
zations. The compiler then reads these descriptions and 
generates application-specific runtime systems to apply the 
heuristics. Runtime code tuning is also exploited by Active 
Harmony [170], which utilizes the computing resources in 
HPC systems to evaluate different code variants on different 
nodes to find the best performing version.
There is also an extensive body of work on how to opti-
mize programs on heterogeneous multicore systems. One of 
the problems for heterogeneous multicore optimization is 
to determine when and how to use the heterogeneous pro-
cessors. Researchers have used machine learning to build 
classifiers to determine which processor to use [68] and at 
which clock frequency the processor should operate [80], 
[171]. Others used regression techniques to build curve fit-
ting models to search for the sweat spot for work partition-
ing among processors [38] or a tradeoff of energy and per-
formance [172].
Another line of research combines compiler-based 
analysis and machine learning to optimize programs in the 
presence of competing workloads. This research problem 
is important because programs rarely run in isolation and 
must share the computing resources with other corunning 
workloads. In [173] and [174], an ANN model based on 
static code features and runtime information was built to 
predict the number of threads to use for a target program 
when it runs with external workloads. Later, in [118], an 
ensemble-learning-based approach was used, which leads to 
significantly better performance over [173]. In [118], several 
models are first trained offline; and then one of the model is 
selected at runtime, taking into consideration the compet-
ing workloads and available hardware resources. The central 
idea is that instead of using a single monolithic model, we 
can use multiple models where each model is specialized for 
modeling a subset of applications or a particular runtime 
scenario. Using this approach, a model is used when its pre-
dictions are effective.
Some recent works developed machine learning models 
based on static code features and dynamic runtime informa-
tion to schedule OpenCL programs in the presence of GPU 
contention. The work presented in [175] uses SVM classifi-
cation to predict the work partition ratio between the CPU 
and GPU when multiple programs are competing to run on a 
single GPU. The work described in [39] aims to improve the 
overall system throughput when there are multiple OpenCL 
programs competing to run on the GPU. They developed an 
ANN model to predict the potential speedup for running an 
OpenCL kernel on the GPU. The speedup prediction is then 
used as a proxy to determine which of the waiting OpenCL 
tasks get to run on the GPU and in what order.
The approaches presented in [176] and [177] target task 
colocation in a data center environment. They use com-
piler-based code transformations to reduce the contention 
for multiple corunning tasks. A linear regression model was 
employed to calculate the contention score of code regions 
based on performance counter values. Then, a set of com-
piler-based code transformations is applied to reduce the 
resource demands of highly contentious code.
C. Other Research Problems
Many works have demonstrated that machine learning 
is a powerful technique in performance and cost modeling 
[47], [178]–[180], and in task and resource scheduling [161], 
[181]–[183]. We envision that many of these techniques can 
be used to provide evidence to support runtime program 
optimizations through, e.g., just-in-time compilation.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
18 Proceedings of the IEEE
While not directly target code optimization, compiler-
based code analysis and machine learning techniques have 
been used in conjunction to solve various software engineer-
ing tasks. These include detecting code similarities [184], 
[185], automatic comment generation [186], mining API 
usage patterns [187], [188], predicting program properties 
[189], code de-obfuscation for malware detection [190], 
etc. It is worth mentioning that many of these recent works 
show that the past development knowledge extracted from 
large code bases such as GitHub are valuable for learning an 
effective model. There were two recent studies performed by 
Cummins et al., which mine Github to synthesize OpenCL 
benchmarks [148] and code extract features from source 
code [78]. Both studies demonstrate the usefulness of large 
code bases and deep learning techniques for learning pre-
dictive models for compiler optimizations. We envision that 
the rich information in large open source code bases could 
provide a powerful knowledge base for training machine 
learning models to solve compiler optimization problems, 
and deep learning could be used as an effective tool to 
extract such knowledge from massive program source code.
VII.  D ISCUSSION
One of the real benefits of machine-learning-based 
approaches is that it forces an empirically driven approach 
to compiler construction. New models have to be based on 
empirical data which can then be verified by independent 
experimentation. This experiment, hypothesis, test cycle is 
well known in the physical sciences but is a relatively new 
addition compiler construction.
As machine-learning-based techniques require a sam-
pling of the optimization space for training data, we typi-
cally know the best optimization for any program in the 
training set. If we exclude this benchmark from training, 
we therefore have access to an upper bound on performance 
or oracle for this program. This immediately lets us know 
how good existing techniques are. If they are 50% of this 
optimum or 95% of this optimum, this immediately tells us 
whether the problem is worth exploring.
Furthermore, we can construct naive techniques, e.g., 
a random optimization, and see its performance. If it per-
formed a number of times, it will have an expected value 
of the mean of the optimization speedups. We can then 
demand that any new heuristic should outperform this, 
though in our experience there have been cases where state-
of-the-art work was actually less than random.
A. Not a Panacea
This paper has, by and large, been very upbeat about the 
use of machine learning. However, there are a number of 
hurdles to overcome to make it a practical reality and this 
opens up new questions about optimization.
Training cost is an issue that many find alarming. In 
practice, the cost is much less than a compiler writer, and 
techniques such as active learning can be employed to 
reduce overhead of training data generation [191]–[194]. 
Although its true to say that generating many differently 
compiled programs and executing and timing them are 
entirely automatic, finding the right data requires careful 
consideration. If the optimizations explored have little posi-
tive performance on the programs, then there is nothing 
worth learning.
The most immediate problem continues to be gathering 
enough sufficient high quality training data. Although there 
are numerous benchmark sites publicly available, the num-
ber of programs available is relatively sparse compared to the 
number that a typical compiler will encounter in its lifetime. 
This is particularly true in specialist domains where there 
may not be any public benchmarks. Automatic benchmark 
generation work will help here, but existing approaches do 
not guarantee that the generated benchmarks effectively 
represent the design space. Therefore, the larger issue of the 
structure of the program space remains.
A really fundamental problem is that if we build our 
optimization models based purely on empirical data, then 
we must guarantee that these data are correct and represent-
ative; we must learn the signal, not the noise. Peer review of 
a machine learning approach is difficult. Black box mode-
ling prevents the quality of the model from being questioned 
unlike handcrafted heuristics. In a sense, reviewers now 
have to scrutinize that the experiments were fairly done. 
This means all training and test data must be publicly availa-
ble for scrutiny. This is common practice in other empirical 
sciences. The artefact evaluation committee is an example 
of this [195], [196].
Although the ability to automatically learn how to best 
optimize an application and adapt to change is a big step 
forward, machine learning can only learn from what is pro-
vided by the compiler writer. Machine learning can neither 
invent new program transformations to apply nor derive 
analysis that determines whether a transformation is legal; 
all of this is beyond its scope.
B. Will This Put Compiler Writers Out of a Job?
In fact, machine-learning-based compilation will para-
doxically lead to a renaissance in compiler optimization. 
Compilers have become so complex that adding a new opti-
mization or compiler phase can lead to performance regres-
sions. This, in turn, has led to a conservative mind set where 
new transformations are not considered if they may rock the 
boat. The core issue is that systems are so complex that it 
is impossible to know for sure when to use such an opti-
mization. Machine learning can remove this uncertainty by 
automatically determining when an optimization is prof-
itable. This now frees the compiler writer to develop ever 
more sophisticated techniques. He/she does not need to 
worry about how they interfere with other optimizations—
machine learning looks after this. We can now develop opti-
mizations that will typically only work for specific domains, 
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
Proceedings of the IEEE  19
and not worry about coordinating their integration into a 
general purpose system. It allows different communities to 
develop novel optimizations and naturally integrate them. 
So rather than closing down the opportunity for new ideas, 
it opens up new vistas.
C. O pen Research Directions
Machine learning has demonstrated its utility as a means 
of automating compiler profitability analysis. It will con-
tinue to be used for more complex optimization problems 
and is likely to be the default approach to selecting compiler 
optimizations in the coming decade.
The open research directions go beyond predicting the 
best optimizations to apply. One central issue is what the 
program space looks like. We know that programs with lin-
ear array accesses inside perfect loop nests need different 
treatment compared to, say, distributed graph processing 
programs. If we could have a map that allows us to meas-
ure distances between programs, then we could see whether 
there are regions that are well served by compiler charac-
terization and other regions that are sparse and currently 
ignored. If we could do the same for hardware, then we 
may be better able to design hardware likely to be of use for 
emerging applications.
Can machine learning also be applied to compiler analy-
sis? For instance is it possible to learn dataflow or point-to 
analysis? As deep learning has the ability to automatically 
construct features, can we find a set of features that are com-
mon across all optimizations and analyses? Can we learn 
the ideal compiler intermediate representation? There is 
a wide range of interesting research questions that remain 
unexplored.
VIII.  C ONCLUSION
This paper has introduced machine-learning-based com-
pilation and described its power in determining an evi-
dence-based approach to compiler optimization. It is the 
latest stage in 50 years of compiler automation. Machine-
learning-based compilation is now a mainstream compiler 
research area and, over the last decade or so, has generated 
a large amount of academic interest and papers. While it is 
impossible to provide a definitive cataloger of all research, 
we have tried to provide a comprehensive and accessible 
survey of the main research areas and future directions. 
Machine learning is not a panacea. It can only learn the data 
we provide. Rather than, as some fear, it dumbs down the 
role of compiler writers, it opens up the possibility of much 
greater creativity and new research areas.
 
REFERENCES
 [ 1]  J . Chipps, M. Koschmann, S. Orgel, A. Perlis, 
and J. Smith, “A mathematical language 
compiler,” in Proc. 11th ACM Nat. Meeting, 
1956, pp. 114–117.
 [
2]
 P
. B. Sheridan, “The arithmetic translator-
compiler of the IBM FORTRAN automatic 
coding system,” Commun. ACM , vol. 2, no. 2, 
pp. 9–21, 1959.
 [
3]
 M
. D. McIlroy, “Macro instruction 
extensions of compiler languages,” Commun. 
ACM, vol. 3, no. 4, pp. 214–220, 1960.
 [
4]
 A
. Gauci, K. Z. Adami, and J. Abela. (2010). 
“Machine learning for galaxy morphology 
classification.” [Online]. Available: https://
arxiv.org/abs/1005.0390
 [
5]
 H
. Schoen, D. Gayo-Avello, P. T. Metaxas,  
E. Mustafaraj, M. Strohmaier, and P. Gloor, 
“The power of prediction with social media,” 
Internet Res. , vol. 23, no. 5, pp. 528–543, 
2013.
 [
6]
 S
lashdot. (2009). IBM Releases Open Source 
Machine Learning Compiler . [Online]. 
Available: https://tech.slashdot.org/
story/09/07/03/0143233/ibm-releases-open-
source-machine-learning-compiler
 [
7]
 H
. Massalin, “Superoptimizer: A look at the 
smallest program,” ACM SIGPLAN Notices , 
vol. 22, no. 10, pp. 122–126, 1987.
 [
8] J
. Ivory, “I. On the method of the least squares, ” 
Philos. Mag. J., Comprehending Various Branches 
Sci., Liberal Fine Arts, Agriculture, Manuf. 
Commerce, vol. 65, no. 321, pp. 3–10, 1825.
 [
9]
 R
. J. Adcock, “A problem in least squares,” 
Analyst, vol. 5, no. 2, pp. 53–54, 1878.
 [
10]
 K
. Datta et al. , “Stencil computation 
optimization and auto-tuning on state-of-
the-art multicore architectures,” in Proc. 
ACM/IEEE Conf. Supercomput. , Nov. 2008, 
pp. 1–12.
 [
11]
 J
. Ansel, Y. L. Wong, C. Chan, M. Olszewski, 
A. Edelman, and S. Amarasinghe, “Language 
and compiler support for auto-tuning 
variable-accuracy algorithms,” in Proc. Int. 
Symp. Code Generat. Optim. (CGO) , Apr. 2011, 
pp. 85–96.
 [
12]
 J
. Kurzak, H. Anzt, M. Gates, and  
J. Dongarra, “Implementation and tuning of 
batched Cholesky factorization and solve for 
NVIDIA GPUs,” IEEE Trans. Parallel Distrib. 
Syst., vol. 27, no. 7, pp. 2036–2048, Jul. 2016.
 [
13]
 Y
. M. Tsai, P. Luszczek, J. Kurzak, and  
J. Dongarra, “Performance-portable 
autotuning of OpenCL kernels for 
convolutional layers of deep neural 
networks,” in Proc. Workshop Mach. Learn. 
HPC Environ. (MLHPC) , 2016, pp. 9–18.
 [
14]
 M
. E. Lesk and E. Schmidt, “Lex—A lexical 
analyzer generator,” Tech. Rep., 1975.
 [
15]
 S
. C. Johnson, Yacc: Yet Another Compiler-
Compiler, vol. 32. Murray Hill, NJ, USA: Bell 
Laboratories, 1975.
 [
16]
 A
. Monsifrot, F. Bodin, and R. Quiniou,  
“A machine learning approach to automatic 
production of compiler heuristics,” in Proc. 
Int. Conf. Artif. Intell. Methodol. Syst. Appl. , 
2002, pp. 41–50.
 [
17]
 A
. Magni, C. Dubach, and M. O’Boyle, 
“Automatic optimization of thread-
coarsening for graphics processors,” in Proc. 
23rd Int. Conf. Parallel Archit. Compilation 
(PACT), 2014, pp. 455–466.
 [
18]
 S
. Unkule, C. Shaltz, and A. Qasem, 
“Automatic restructuring of GPU kernels for 
exploiting inter-thread data locality,” in Proc. 
21st Int. Conf. Compil. Construction (CC) , 
2012, pp. 21–40.
 [
19]
 V
. Volkov and J. W. Demmel, “Benchmarking 
GPUs to tune dense linear algebra,” in Proc. 
ACM/IEEE Conf. Supercomput. (SC),  
Nov. 2008, pp. 1–11.
 [
20]
 Y
. Yang, P. Xiang, J. Kong, M. Mantor, and 
H. Zhou, “A unified optimizing compiler 
framework for different gpgpu 
architectures,” ACM Trans. Archit. Code 
Optim., vol. 9, no. 2, p. 9, 2012.
 [
21]
 C
. Lattner and V. Adve, “LLVM: A 
compilation framework for lifelong program 
analysis & transformation,” in Proc. Int. 
Symp. Code Generat. Optim. (CGO) , 2004,  
pp. 75–86.
 [
22]
 F
. Bodin, T. Kisuki, P. Knijnenburg,  
M. O’Boyle, and E. Rohou, “Iterative 
compilation in a non-linear optimization 
space,” in Proc. Workshop Profile Feedback-
Directed Compilation , 1998.
 [
23]
 P
. M. Knijnenburg, T. Kisuki, and  
M. F. O’Boyle, “Combined selection of tile 
sizes and unroll factors using iterative 
compilation,” J. Supercomput., vol. 24, no. 1, 
pp. 43–67, 2003.
 [
24]
 M
. Frigo and S. G. Johnson, “The design and 
implementation of FFTW3,” Proc. IEEE ,  
vol. 93, no. 2, pp. 216–231, Feb. 2005.
 [
25]
 F
. Agakov et al. , “Using machine learning to 
focus iterative optimization,” in Proc. Int. 
Symp. Code Generat. Optim. (CGO) , 2006,  
pp. 295–305.
 [
26]
 R
. Nobre, L. G. A. Martins, and  
J. M. P. Cardoso, “A graph-based iterative 
compiler pass selection and phase ordering 
approach,” in Proc. 17th ACM SIGPLAN/
SIGBED Conf. Lang. Compil. Tools Theory 
Embedded Syst. (LCTES) , 2016, pp. 21–30.
 [
27]
 R
. Leupers and P. Marwedel, “Function 
inlining under code size constraints for 
embedded processors,” in Dig. Tech. Papers 
IEEE/ACM Int. Conf. Comput.-Aided Design , 
Nov. 1999, pp. 253–256.
 [
28]
 K
. D. Cooper, T. J. Harvey, and T. Waterman, 
“An adaptive strategy for inline substitution,” 
in Proc. Joint Eur. Conf. Theory Pract. Softw. 
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
20 Proceedings of the IEEE
17th Int. Conf. Compil. Construction (CC/
ETAPS), 2008, pp. 69–84.
 [
29]
 D
. Simon, J. Cavazos, C. Wimmer, and  
S. Kulkarni, “Automatic construction of 
inlining heuristics using machine learning,” 
in Proc. IEEE/ACM Int. Symp. Code Generat. 
Optim. (CGO), 2013, pp. 1–12.
 [
30]
 P
. Zhao and J. N. Amaral, “To inline or not to 
inline? Enhanced inlining decisions,” Lang. 
Compil. Parallel Comput. , pp. 405–419, 2004.
 [
31]
 T
. A. Wagner, V. Maverick, S. L. Graham, 
and M. A. Harrison, “Accurate static 
estimators for program optimization,” in 
Proc. ACM SIGPLAN Conf. Program. Lang. 
Design Implement. (PLDI) , 1994, pp. 85–96.
 [
32]
 V
. Tiwari, S. Malik, and A. Wolfe, “Power 
analysis of embedded software: A first step 
towards software power minimization,” in 
Proc. IEEE/ACM Int. Conf. Comput.-Aided 
Design, Nov. 1994, pp. 384–390.
 [
33]
 K
. D. Cooper, P. J. Schielke, and  
D. Subramanian, “Optimizing for reduced 
code space using genetic algorithms,” in 
Proc. ACM SIGPLAN Workshop Lang. Compil. 
Tools Embedded Syst. (LCTES) , 1999, pp. 1–9.
 [
34]
 M
. Stephenson, S. Amarasinghe, M. Martin, 
and U.-M. O‘Reilly, “Meta optimization: 
Improving compiler heuristics with machine 
learning,” in Proc. ACM SIGPLAN Conf. 
Program. Lang. Design Implement. (PLDI) , 
2003, pp. 77–90.
 [35]
 J
. Cavazos and M. F. P. O’Boyle, “Automatic 
tuning of inlining heuristics, ” in Proc. ACM/
IEEE Conf. Supercomput. (SC), Nov. 2005, p. 14.
 [
36]
 K
. Hoste and L. Eeckhout, “Cole: Compiler 
optimization level exploration,” in Proc. 6th 
Annu. IEEE/ACM Int. Symp. Code Generat. 
Optim. (CGO), 2008, pp. 165–174.
 [37]
 M. K
im, T. Hiroyasu, M. Miki, and S. Watanabe, 
“SPEA2+: Improving the performance of the 
Strength Pareto Evolutionary Algorithm 2, ” in 
Proc. Int. Conf. Parallel Problem Solving from 
Nature, 2004, pp. 742–751.
 [
38]
 C
.-K. Luk, S. Hong, and H. Kim, “Qilin: 
Exploiting parallelism on heterogeneous 
multiprocessors with adaptive mapping,” in 
Proc. 42nd Annu. IEEE/ACM Int. Symp. 
Microarchit. (MICRO), Dec. 2009, pp. 45–55.
 [
39]
 Y
. Wen, Z. Wang, and M. F. P. O’Boyle, 
“Smart multi-task scheduling for OpenCL 
programs on CPU/GPU heterogeneous 
platforms,” in Proc. 21st Ann. IEEE Int. Conf. 
High Perform. Comput. (HiPC) , Dec. 2014,  
pp. 1–10.
 [
40]
 E
. A. Brewer, “High-level optimization via 
automated statistical modeling,” in Proc. 5th 
ACM SIGPLAN Symp. Principles Pract. Parallel 
Program. (PPOPP), 1995, pp. 80–91.
 [
41]
 K
. Vaswani, M. J. Thazhuthaveetil,  
Y. N. Srikant, and P. J. Joseph, 
“Microarchitecture sensitive empirical 
models for compiler optimizations,” in Proc. 
Int. Symp. Code Generat. Optim. (CGO) ,  
Mar. 2007, pp. 131–143.
 [
42]
 B
. C. Lee and D. M. Brooks, “Accurate and 
efficient regression modeling for 
microarchitectural performance and power 
prediction,” in Proc. 12th Int. Conf. Archit. 
Support Program. Lang. Oper. Syst. (ASPLOS) , 
2006, pp. 185–194.
 [
43]
 E
. Park, L.-N. Pouche, J. Cavazos, A. Cohen, 
and P. Sadayappan, “Predictive modeling in a 
polyhedral optimization space,” in Proc. 9th 
Annu. IEEE/ACM Int. Symp. Code Generat. 
Optim. (CGO), Apr. 2011, pp. 119–129.
 [
44]
 M
. Curtis-Maury, A. Shah, F. Blagojevic,  
D. S. Nikolopoulos, B. R. de Supinski, and 
M. Schulz, “Prediction models for multi-
dimensional power-performance 
optimization on many cores,” in Proc. 17th 
Int. Conf. Parallel Archit. Compilation Techn. 
(PACT), 2008, pp. 250–259.
 [
45]
 K
. Singhet et al. , “Comparing scalability 
prediction strategies on an SMP of CMPs,”  
in Proc. Eur. Conf. Paralell Process. , 2010,  
pp. 143–155.
 [
46]
 Z
. Wang and M. F. O’Boyle, “Mapping 
parallelism to multi-cores: A machine 
learning based approach,” in Proc. 14th ACM 
SIGPLAN Symp. Principles Pract. Parallel 
Program. (PPoPP), 2009, pp. 75–84.
 [
47]
 Y
. Kang et al. , “Neurosurgeon: Collaborative 
intelligence between the cloud and mobile 
edge,” in Proc. 22nd Int. Conf. Archit. Support 
Program. Lang. Oper. Syst. (ASPLOS) , 2017,  
pp. 615–629.
 [
48]
 L
. Benini, A. Bogliolo, M. Favalli, and G. De 
Micheli, “Regression models for behavioral 
power estimation,” Integr. Comput.-Aided 
Eng., vol. 5, no. 2, pp. 95–106, 1998.
 [
49]
 S
. K. Rethinagiri, R. B. Atitallah, and  
J.-L. Dekeyser, “A system level power 
consumption estimation for MPSoC,” in 
Proc. Int. Symp. Syst. Chip (SoC) , 2011,  
pp. 56–61.
 [
50]
 S
. Schürmans, G. Onnebrink, R. Leupers,  
G. Leupers, and X. Chen, “Frequency-aware 
ESL power estimation for ARM cortex-A9 
using a black box processor model,” ACM 
Trans. Embedded Comput. Syst. , vol. 16, no. 1, 
p. 26, 2016.
 
[51]
 M
. Curtis-Maury et al. , “Identifying energy-
efficient concurrency levels using machine 
learning,” in Proc. IEEE Int. Conf. Cluster 
Comput., Sep. 2007, pp. 488–495.
 [
52]
 M
. Stephenson and S. Amarasinghe, 
“Predicting unroll factors using supervised 
classification,” in Proc. Int. Symp. Code 
Generat. Optim. (CGO), 2005, pp. 123–134.
 [
53]
 J
. Cavazos, G. Fursin, F. Agakov, E. Bonilla, 
M. F. P. O’Boyle, and O. Temam, “Rapidly 
selecting good compiler optimizations using 
performance counters,” in Proc. Int. Symp. 
Code Generat. Optim. (CGO), 2007,  
pp. 185–197.
 [
54]
 J
. Cavazos and M. F. P. O’Boyle, “Method-
specific dynamic compilation using logistic 
regression,” in Proc. 21st Annu. ACM SIGPLAN 
Conf. Object-Oriented Program. Syst. Lang. 
Appl. (OOPSLA), 2006, pp. 229–240.
 [
55]
 C
. Dubach, J. Cavazos, B. Franke, G. Fursin, 
M. F. O’Boyle, and O. Temam, “Fast compiler 
optimization evaluation using code-feature 
based performance prediction,” in Proc. 4th 
Int. Conf. Comput. Frontiers (CF), 2007,  
pp. 131–142.
 [
56]
 T
. Yuki, L. Renganarayanan, S. Rajopadhye, 
C. Anderson, A. E. Eichenberger, and  
K. O’Brien, “Automatic creation of tile size 
selection models,” in Proc. 8th Annu. IEEE/
ACM Int. Symp. Code Generat. Optim. (CGO) , 
2010, pp. 190–199.
 [
57]
 A
. M. Malik, “Optimal tile size selection 
problem using machine learning,” in Proc. 
Optim. Tile Size Selection Problem Using Mach. 
Learn., vol. 2. Dec. 2012, pp. 275–280.
 [
58]
 R
. W. Moore and B. R. Childers, “Building 
and using application utility models  
to dynamically choose thread counts,”  
J. Supercomput., vol. 68, no. 3,  
pp. 1184–1213, 2014.
 [
59]
 Y
. Liu, E. Z. Zhang, and X. Shen, “A cross-
input adaptive framework for GPU program 
optimizations,” in Proc. IEEE Int. Symp. 
Parallel Distrib. Process. , May 2009, pp. 1–10.
 [ 60]  E . Perelman, G. Hamerly, M. Van 
Biesbrouck, T. Sherwood, and B. Calder, 
“Using simpoint for accurate and efficient 
simulation,” in Proc. ACM SIGMETRICS Int. 
Conf. Meas. Modeling Comput. Syst. , 2003,  
pp. 318–319.
 [
61]
 Y
. Zhang, D. Meisner, J. Mars, and L. Tang, 
“Treadmill: Attributing the source of tail 
latency through precise load testing and 
statistical inference,” in Proc. 43rd Int. Symp. 
Comput. Archit. (ISCA), 2016, pp. 456–468.
 [
62]
 B
. C. Lee, D. M. Brooks, B. R. de Supinski, 
M. Schulz, K. Singh, and S. A. McKee, 
“Methods of inference and learning for 
performance modeling of parallel 
applications,” in Proc. 12th ACM SIGPLAN 
Symp. Principles Pract. Parallel Program. 
(PPoPP), 2007, pp. 249–258.
 [
63]
 M
. Curtis-Maury, J. Dzierwa,  
C. D. Antonopoulos, and D. S. Nikolopoulos, 
“Online power-performance adaptation of 
multithreaded programs using hardware 
event-based prediction,” in Proc. 20th  
Annu. Int. Conf. Supercomput. (ICS), 2006,  
pp. 157–166.
 [
64]
 P
. E. Bailey, D. K. Lowenthal, V. Ravi,  
B. Rountree, M. Schulz, and B. R. de 
Supinski, “Adaptive configuration selection 
for power-constrained heterogeneous 
systems,” in Proc. 43rd Int. Conf. Parallel 
Process., 2014, pp. 371–380.
 [
65]
 J
. L. Berral et al. , “Towards energy-aware 
scheduling in data centers using machine 
learning,” in Proc. 1st Int. Conf. Energy-
Efficient Comput. Netw. (e-Energy) , 2010,  
pp. 215–224.
 [
66]
 D
. D. Vento, “Performance optimization on a 
supercomputer with ctuning and the PGI 
compiler,” in Proc. 2nd Int. Workshop Adapt. 
Self-Tuning Comput. Syst. Exaflop Era 
(EXADAPT) , 2012, pp. 12–20.
 [
67]
 P
.-J. Micolet, A. Smith, and C. Dubach,  
“A machine learning approach to mapping 
streaming workloads to dynamic multicore 
processors,” ACM SIGPLAN Notices , vol. 51, 
no. 5, pp. 113–122, 2016.
 [
68]
 D
. Grewe, Z. Wang, and M. F. P. O’Boyle, 
“Portable mapping of data parallel programs 
to OpenCL for heterogeneous systems,” in 
Proc. Proc. IEEE/ACM Int. Symp. Code 
Generation Optim. (CGO), Feb. 2013, pp. 1–10.
 [
69]
 H
. Yu and L. Rauchwerger, “Adaptive 
reduction parallelization techniques,” in 
Proc. 14th Int. Conf. Supercomput. (ICS), 2000, 
pp. 66–77.
 [
70]
 H
. Leather, E. Bonilla, and M. O’Boyle, 
“Automatic feature generation for machine 
learning based optimizing compilation,” in 
Proc. 7th Annu. IEEE/ACM Int. Symp. Code 
Generat. Optim. (CGO), Mar. 2009, pp. 81–91.
 [
71]
 Z
. Wang, D. Grewe, and M. F. P. O’Boyle, 
“Automatic and portable mapping of data 
parallel programs to opencl for GPU-based 
heterogeneous systems,” ACM Trans. Archit. 
Code Optim., vol. 11, no. 4, p. 42, 2014.
 [
72]
 Y
. Ding, J. Ansel, K. Veeramachaneni,  
X. Shen, U.-M. O’Reilly, and S. Amarasinghe, 
“Autotuning algorithmic choice for input 
sensitivity,” in Proc. 36th ACM SIGPLAN Conf. 
Program. Lang. Design Implement. (PLDI) , 
2015, pp. 379–390.
 [
73]
 T
. K. Ho, “Random decision forests,” in  
Proc. 3rd Int. Conf. Document Anal. Recognit. 
(ICDAR), vol. 1. 1995, pp. 278–282.
 [
74]
 T
. G. Dietterich, “Ensemble methods  
in machine learning,” in Proc. 1st Int. 
Workshop Multiple Classifier Syst. (MCS) , 
2000, pp. 1–15.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
Proceedings of the IEEE  21
 [ 75]  P . Lokuciejewski, F. Gedikli, P. Marwedel, 
and K. Morik, “Automatic WCET reduction 
by machine learning based heuristics for 
function inlining,” in Proc. 3rd Workshop 
Statistical Mach. Learn. Approaches Archit. 
Compilation (SMART) , 2009, pp. 1–15.
 [
76]
 S
. Benedict, R. S. Rejitha, P. Gschwandtner, 
R. Prodan, and T. Fahringer, “Energy 
prediction of OpenMP applications using 
random forest modeling approach,” in Proc. 
IEEE Int. Parallel Distrib. Process. Symp. 
Workshop, May 2015, pp. 1251–1260.
 [
77]
 R
. S. Rejitha, S. Benedict, S. A. Alex, and  
S. Infanto, “Energy prediction of CUDA 
application instances using dynamic 
regression models,” Computing , vol. 99,  
no. 8, pp. 765–790, 2017.
 [
78]
 C
. Cummins, P. Petoumenos, Z. Wang, and 
H. Leather, “End-to-end deep learning of 
optimization heuristics,” in Proc. 26th Int. 
Conf. Parallel Archit. Compilation Techn. 
(PACT), 2017, pp. 219–232.
 [
79]
 Z
. Wang, G. Tournavitis, B. Franke, and  
M. F. P. O’Boyle, “Integrating profile-driven 
parallelism detection and machine-learning-
based mapping,” ACM Trans. Archit. Code 
Optim., vol. 11, no. 1, p. 2, 2014.
 [
80]
 B
. Taylor, V. S. Marco, and Z. Wang, 
“Adaptive optimization for OpenCL 
programs on embedded heterogeneous 
systems,” in Proc. 18th Annu. ACM SIGPLAN/
SIGBED Conf. Lang. Compil. Tools Embedded 
Syst. (LCETS), 2017, pp. 11–20.
 [
81]
 P
. Zhang, J. Fang, T. Tang, C. Yang, and  
Z. Wang, “Auto-tuning streamed applications 
on Intel Xeon Phi,” in Proc. 32nd IEEE Int. 
Parallel Distrib. Process. Symp. (IPDPS) , 2018.
 [
82]
 P
. J. Joseph, K. Vaswani, and  
M. J. Thazhuthaveetil, “A predictive 
performance model for superscalar 
processors,” in Proc. 39th Annu. IEEE/ACM 
Int. Symp. Microarchit. (MICRO),  
Dec. 2006, pp. 161–170.
 [
83]
 A
. Ganapathi, K. Datta, A. Fox, and D. 
Patterson, “A case for machine learning to 
optimize multicore performance,” in Proc. 
1st USENIX Conf. Hot Topics Parallelism 
(HotPar), 2009, p. 1.
 [
84]
 E
. Deniz and A. Sen, “Using machine 
learning techniques to detect parallel 
patterns of multi-threaded applications,”  
Int. J. Parallel Program. , vol. 44, no. 4,  
pp. 867–900, 2016.
 [
85]
 Y
. LeCun, Y. Bengio, and G. Hinton, Deep 
Learning, 2015.
 [
86]
 A
. Krizhevsky, I. Sutskever, and G. E. 
Hinton, “Imagenet classification with deep 
convolutional neural networks,” in Proc. Adv. 
Neural Inf. Process. Syst. (NIPS) , 2012,  
pp. 1097–1105.
 [
87]
 K
. He, X. Zhang, S. Ren, and J. Sun, “Deep 
residual learning for image recognition,” in 
Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 
(CVPR), Jun. 2016, pp. 770–778.
 [
88]
 H
. Lee, Y. Largman, P. Pham, and A. Y. Ng, 
“Unsupervised feature learning for audio 
classification using convolutional deep belief 
networks,” in Proc. 22nd Int. Conf. Neural Inf. 
Process. Syst. (NIPS) , 2009, pp. 1096–1104.
 [
89]
 M
. Allamanis, E. T. Barr, P. Devanbu, and  
C. Sutton (2017). “A survey of machine 
learning for big code and naturalness.” 
[Online]. Available: https://arxiv.org/
abs/1709.06182
 [
90]
 J
. MacQueen, “Some methods for 
classification and analysis of multivariate 
observations,” in Proc. 5th Berkeley Symp. 
Math. Statist. Prob. , 1967, pp. 281–297.
  [ 91]  T . Sherwood, E. Perelman, G. Hamerly, 
and B. Calder, “Automatically 
characterizing large scale program 
behavior,” in Proc. 10th Int. Conf. Archit. 
Support Program. Lang. Operat. Syst. 
(ASPLOS), 2002, pp. 45–57.
 
 [
92]
 Z
. Wang and M. F. O’Boyle, “Partitioning 
streaming parallelism for multi-cores: A 
machine learning based approach,” in Proc. 
19th Int. Conf. Parallel Archit. Compilation 
Techn. (PACT), 2010, pp. 307–318.
 
 [
93]
 M
. Newman, Networks: An Introduction . 
New York, NY, USA: Oxford Univ. Press, 
2010.
 
 [
94]
 L
. G. Martins, R. Nobre, A. C. B. Delbem, 
E. Marques, and J. A. M. Cardoso, 
“Exploration of compiler optimization 
sequences using clustering-based 
selection,” in Proc. SIGPLAN/SIGBED Conf. 
Lang. Compil. Tools Embedded Syst. (LCTES) , 
2014, pp. 63–72.
 
 [
95]
 L
. Eeckhout, H. Vandierendonck, and  
K. D. Bosschere, “Workload design: 
Selecting representative program-input 
pairs,” in Proc. Int. Conf. Parallel Archit. 
Compilation Techn., 2002, pp. 83–94.
 
 
[96]
 Y
. Chen et al. , “Evaluating iterative 
optimization across 1000 datasets,” in Proc. 
31st ACM SIGPLAN Conf. Program. Lang. 
Design Implement. (PLDI) , 2010,  
pp. 448–459.
 
 [
97]
 A
. H. Ashouri, G. Mariani, G. Palermo, 
and C. Silvano, “A Bayesian network 
approach for compiler auto-tuning for 
embedded processors,” in Proc. IEEE 12th 
Symp. Embedded Syst. Real-time Multimedia 
(ESTIMedia), Oct. 2014, pp. 90–97.
 
 [
98]
 A
. Phansalkar, A. Joshi, and L. K. John, 
“Analysis of redundancy and application 
balance in the spec cpu2006 benchmark 
suite,” in Proc. 34th Annu. Int. Symp. 
Comput. Archit. (ISCA), 2007, pp. 412–423.
 
 [
99]
 P
. Vincent, H. Larochelle, Y. Bengio, and 
P.-A. Manzagol, “Extracting and composing 
robust features with denoising 
autoencoders,” in Proc. 25th Int. Conf. Mach. 
Learn. (ICML), 2008, pp. 1096–1103.
 [
100]
 X
. Gu, H. Zhang, D. Zhang, and S. Kim 
(2016). Deep API learning. [Online]. 
Available: https://arxiv.org/abs/1605.08535
 [
101]
 B
. Singer and M. Veloso, “Learning to 
construct fast signal processing 
implementations,” J. Mach. Learn. Res. ,  
vol. 3, pp. 887–919, Dec. 2002.
 [
102]
 X
. Li, M. J. Garzaran, and D. Padua, 
“Optimizing sorting with genetic 
algorithms,” in Proc. Int. Symp. Code 
Generat. Optim. (CGO), 2005, pp. 99–110.
 
[103]
 J
. Ansel et al. , “Petabricks: A language and 
compiler for algorithmic choice,” in Proc. 
ACM SIGPLAN Conf. Program. Lang. Design 
Implement. (PLDI), 2009, pp. 38–49.
 [
104]
 M
. Harman, W. B. Langdon, Y. Jia, D. R. 
White, A. Arcuri, and J. A. Clark, “The 
GISMOE challenge: Constructing the 
Pareto program surface using genetic 
programming to find better programs 
(keynote paper),” in Proc. 27th IEEE/ACM 
Int. Conf. Autom. Softw. Eng. (ASE), Sep. 
2012, pp. 1–14.
 [
105]
 U
. Garciarena and R. Santana, 
“Evolutionary optimization of compiler 
flag selection by learning and exploiting 
flags interactions,” in Proc. Genetic Evol. 
Comput. Conf. Companion (GECCO), 2016, 
pp. 1159–1166.
 [
106]
 M
. Zuluaga, E. Bonilla, and N. Topham, 
“Predicting best design trade-offs: A case 
study in processor customization,” in Proc. 
Design Autom. Test Eur. Conf. Exhibit. 
(DATE), 2012, pp. 1030–1035.
 [
107]
 M
. R. Jantz and P. A. Kulkarni, “Exploiting 
phase inter-dependencies for faster 
iterative compiler optimization phase order 
searches,” in Proc. Int. Conf. Compil. Archit. 
Synthesis Embedded Syst. (CASES) ,  
Oct. 2013, pp. 1–10.
 [108]
 E. I
pek, O. Mutlu, J. F. Martínez, and  
R. Caruana, “Self-optimizing memory 
controllers: A reinforcement learning 
approach, ” in Proc. IEEE 35th Int. Symp. 
Comput. Archit. (ISCA), Jun. 2008, pp. 39–50.
 [
109]
 B
. Porter, M. Grieves, R. R. Filho, and  
D. Leslie, “Rex: A development platform 
and online learning approach for runtime 
emergent software systems,” in Proc. Usenix 
Conf. Symp. Oper. Syst. Design Implement. , 
Nov. 2016, pp. 333–348.
 [110]
 J
. Rao, X. Bu, C.-Z. Xu, L. Wang, and G. Yin, 
“Vconf: A reinforcement learning approach 
to virtual machines auto-configuration, ” in 
Proc. 6th Int. Conf. Autonom. Comput. (ICAC), 
2009, pp. 137–146.
 [
111]
 M
. G. Lagoudakis and M. L. Littman, 
“Algorithm selection using reinforcement 
learning,” in Proc. 7th Int. Conf. Mach. 
Learn. (ICML), 2000, pp. 511–518.
 [112]
 N
. Mishra, J. D. Lafferty, H. Hoffmann, and 
C. Imes, “CALOREE: Learning control for 
predictable latency and low energy, ” in Proc. 
23rd Int. Conf. Archit. Support Program. Lang. 
Oper. Syst. (ASPLOS), 2018, pp. 184–198.
 [
113]
 M
. K. Emani and M. O’Boyle, “Change 
detection based parallelism mapping: 
Exploiting offline models and Online 
adaptation,” in Proc. 27th Int. Workshop 
Lang. Compil. Parallel Comput. (LCPC) ,  
2014, pp. 208–223.
 [
114]
 Y
. Li, “Deep reinforcement learning: An 
overview,” CoRR, 2017.
 
[115]
 J
. Ansel et al. , “Opentuner: An extensible 
framework for program autotuning,” in 
Proc. PACT, 2014, pp. 303–316.
 [
116]
 C
. K. Williams and C. E. Rasmussen, 
“Gaussian processes for regression,” in 
Proc. Adv. Neural Inf. Process. Syst. , 1996, 
pp. 514–520.
 [
117]
 C
. E. Rasmussen and C. K. Williams, 
Gaussian Processes for Machine Learning , 
vol. 1. Cambridge, MA, USA: MIT Press, 
2006.
 [
118]
 M
. K. Emani and M. O’Boyle, “Celebrating 
diversity: A mixture of experts approach 
for runtime mapping in dynamic 
environments,” in Proc. 36th ACM SIGPLAN 
Conf. Program. Lang. Design Implement. 
(PLDI), 2015, pp. 499–508.
 [
119]
 H
. D. Nguyen and F. Chamroukhi (2017). 
“An introduction to the practical and 
theoretical aspects of mixture-of-experts 
modeling.” [Online]. Available: https://
arxiv.org/abs/1707.03538
 [
120]
 R
. Polikar, “Ensemble based systems in 
decision making,” IEEE Circuits Syst. Mag. , 
vol. 6, no. 3, pp. 21–45, Sep. 2006.
 [
121]
 L
. Rokach, “Ensemble-based classifiers,” 
Artif. Intell. Rev. , vol. 33, nos. 1–2, pp. 1–39, 
2010.
 
[122]
 Y
. Jiang et al. , “Exploiting statistical 
correlations for proactive prediction of 
program behaviors,” in Proc. 8th Annu. 
IEEE/ACM Int. Symp. Code Generat. Optim. 
(CGO), Apr. 2010, pp. 248–256.
 [
123]
 V
. S. Marco, B. Taylor, B. Porter, and  
Z. Wang, “Improving spark application 
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
22 Proceedings of the IEEE
throughput via memory aware task 
co-location: A mixture of experts 
approach,” in Proc. ACM/IFIP/USENIX 
Middleware Conf., 2017, pp. 95–108.
 [
124]
 B
. Singer and M. M. Veloso, “Learning to 
predict performance from formula 
modeling and training data,” in Proc. 7th 
Int. Conf. Mach. Learn. (ICML), 2000,  
pp. 887–894.
 [
125]
 E
. Park, J. Cavazos, and M. A. Alvarez, 
“Using graph-based program 
characterization for predictive modeling,” 
in Proc. 10th Int. Symp. Code Generat. Optim. 
(CGO), 2012, pp. 196–206.
 [
126]
 A
. M. Malik, “Spatial based feature 
generation for machine learning based 
optimization compilation,” in Proc. 9th Int. 
Conf. Mach. Learn. Appl. , 2010,  
pp. 925–930.
 [
127]
 M
. Burtscher, R. Nasre, and K. Pingali,  
“A quantitative study of irregular programs 
on GPUs,” in Proc. IEEE Int. Symp. Workload 
Characterization (IISWC) , Nov. 2012,  
pp. 141–151.
 [
128]
 Y
. Luo, G. Tan, Z. Mo, and N. Sun, “Fast: A 
fast stencil autotuning framework based on 
an optimal-solution space model,” in Proc. 
29th ACM Int. Conf. Supercomput., 2015,  
pp. 187–196.
 [129]
 S
. Browne, J. Dongarra, N. Garner, G. Ho, and 
P. Mucci, “A portable programming interface 
for performance evaluation on modern 
processors, ” Int. J. High Perform. Comput. Appl., 
vol. 14, no. 3, pp. 189–204, 2000.
 [
130]
 T
. Mytkowicz, A. Diwan, M. Hauswirth, 
and P. F. Sweeney, “Producing wrong data 
without doing anything obviously wrong!” 
in Proc. 14th Int. Conf. Archit. Support 
Program. Lang. Operat. Syst. (ASPLOS XIV) , 
2009, pp. 265–276.
 
[131]
 J
. Cavazos et al. , “Automatic performance 
model construction for the fast software 
exploration of new hardware designs,” in 
Proc. Int. Conf. Compil. Archit. Synthesis 
Embedded Syst. (CASES), 2006, pp. 24–34.
 [
132]
 S
. Khan, P. Xekalakis, J. Cavazos, and  
M. Cintra, “Using predictivemodeling for 
cross-program design space exploration in 
multicore systems,” in Proc. IEEE 16th Int. 
Conf. Parallel Archit. Compilation Techn. , 
Sep. 2007, pp. 327–338.
 [
133]
 M
. Namolaru, A. Cohen, G. Fursin,  
A. Zaks, and A. Freund, “Practical 
aggregation of semantical program 
properties for machine learning based 
optimization,” in Proc. Proc. Int. Conf. 
Compil. Archit. Synth. Embedded Syst. 
(CASES), 2010, pp. 197–206.
 [
134]
 C
. M. Bishop, Pattern Recognition and 
Machine Learning (Information Science and 
Statistics). Secaucus, NJ, USA: Springer-
Verlag, 2006.
 [
135]
 K
. Hoste, A. Phansalkar, L. Eeckhout,  
A. Georges, L. K. John, and K. de 
Bosschere, “Performance prediction based 
on inherent program similarity,” in Proc. 
IEEE Int. Conf. Parallel Archit. Compilation 
Techn. (PACT), Swep. 2006, pp. 114–122.
 [
136]
 N
. E. Rosenblum, B. P. Miller, and X. Zhu, 
“Extracting compiler provenance from 
program binaries,” in Proc. 9th ACM 
SIGPLAN-SIGSOFT Workshop Program Anal. 
Softw. Tools Eng. (PASTE), 2010, pp. 21–28.
 [
137]
 A
. Bhattacharyya, G. Kwasniewski, and  
T. Hoefler, “Using compiler techniques to 
improve automatic performance 
modeling,” in Proc. Int. Conf. Parallel Archit. 
Compilation (PACT), 2015, pp. 468–479.
 [ 138]  M . E. Taylor, K. E. Coons, B. Robatmili,  
B. A. Maher, D. Burger, and K. S. 
McKinley, “Evolving compiler heuristics to 
manage communication and contention,” 
in Proc. 24th Conf. Artif. Intell. (AAAI) , 2010, 
pp. 1690–1693.
 [
139]
 P
.-S. Ting, C.-C. Tu, P.-Y. Chen, Y.-Y. Lo, 
and S.-M. Cheng (2016). “FEAST: An 
automated feature selection framework for 
compilation tasks.” [Online]. Available: 
https://arxiv.org/abs/1610.09543
 [
140]
 E
. Park, C. Kartsaklis, and J. Cavazos, 
“HERCULES: Strong patterns towards 
more intelligent predictive modeling,” in 
Proc. 43rd Int. Conf. Parallel Process. , 2014, 
pp. 172–181.
 [
141]
 K
. Beyer, J. Goldstein, R. Ramakrishnan, 
and U. Shaft, “When is ‘nearest neighbor’ 
meaningful?” in Proc. Int. Conf. Database 
Theory, 1999, pp. 217–235.
 [
142]
 I
. Fodor, “A survey of dimension reduction 
techniques,” Lawrence Livermore Nat. 
Lab., Tech. Rep., 2002.
 [
143]
 J
. Thomson, M. F. O’Boyle, G. Fursin, and 
B. Franke, “Reducing training time in a 
one-shot machine learning-based 
compiler,” Lang. Compil. Parallel Comput., 
vol. 5898, pp. 399–407, 2009.
 [
144]
 Y
. Bengio, “Learning deep architectures for 
AI,” Found. Trends Mach. Learn. , vol. 2, no. 
1, pp. 1–127, 2009.
 [
145]
 L
. Deng, M. L. Seltzer, D. Yu, A. Acero,  
A.-R. Mohamed, and G. Hinton, “Binary 
coding of speech spectrograms using a 
deep auto-encoder,” in Proc. 11th Annu. 
Conf. Int. Speech Commun. Assoc. , 2010.
 [
146]
 L
. Mou, G. Li, L. Zhang, T. Wang, and  
Z. Jin, “Convolutional neural networks 
over tree structures for programming 
language processing,” in Proc. AAAI , 2016, 
pp. 1287–1293.
 [
147]
 M
. White, M. Tufano, C. Vendome, and  
D. Poshyvanyk, “Deep learning code 
fragments for code clone detection,” in 
Proc. ASE 31st IEEE/ACM Int. Conf. Autom. 
Softw. Eng., 2016, pp. 87–98.
 [148]
 C. C
ummins, P. Petoumenos, Z. Wang, and 
H. Leather, “Synthesizing benchmarks for 
predictive modeling, ” in Proc. Int. Symp. Code 
Generat. Optim. (CGO), 2017, pp. 86–99.
 [
149]
 M
. White, M. Tufano, and M. Martinez, 
M. Monperrus, and D. Poshyvanyk (2017). 
“Sorting and transforming program repair 
ingredients via deep learning code 
similarities.” [Online]. Available: https://
arxiv.org/abs/1707.04742
 
[150]
 L. 
Almagor et al., “Finding effective 
compilation sequences, ” in Proc. ACM 
SIGPLAN/SIGBED Conf. Lang. Compil. Tools 
Embedded Syst. (LCTES), 2004, pp. 231–239.
 [
151]
 K
. D. Cooper et al. , “ACME: Adaptive 
compilation made efficient,” in Proc. ACM 
SIGPLAN/SIGBED Conf. Lang. Compil. 
Embedded Syst. (LCTES) , 2005, pp. 69–77.
 [
152]
 A
. H. Ashouri, A. Bignoli, G. Palermo,  
C. Silvano, S. Kulkarni, and J. Cavazos, 
“MiCOMP: Mitigating the compiler phase-
ordering problem using optimization sub-
sequences and machine learning,” ACM 
Trans. Archit. Code Optim. , vol. 14, no. 3,  
p. 29, 2017.
 [
153]
 K
. D. Cooper, D. Subramanian, and  
L. Torczon, “Adaptive optimizing compilers 
for the 21st century,” J. Supercomput. ,  
vol. 23, no. 1, pp. 7–22, 2002.
 [
154]
 D
. R. White, A. Arcuri, and J. A. Clark, 
“Evolutionary improvement of programs,” 
IEEE Trans. Evol. Comput., vol. 15, no. 4,  
pp. 515–538, Aug. 2011.
 [
155]
 G
. Fursin and O. Temam, “Collective 
optimization: A practical collaborative 
approach,” ACM Trans. Archit. Code Optim. , 
vol. 7, no. 4, p. 20, Dec. 2010.
 [
156]
 J
. Kukunas, R. D. Cupper, and  
G. M. Kapfhammer, “A genetic algorithm 
to improve linux kernel performance on 
resource-constrained devices,” in Proc. 12th 
Annu. Conf. Companion Genetic Evol. 
Comput. (GECCO), 2010, pp. 2095–2096.
 [
157]
 L
.-N. Pouchet, C. Bastoul, A. Cohen, and  
J. Cavazos, “Iterative optimization in the 
polyhedral model: Part II, 
multidimensional time,” in Proc. 29th ACM 
SIGPLAN Conf. Program. Lang. Design 
Implement. (PLDI), 2008, pp. 90–100.
 [15
8]
 K. Asanovic 
et al., “The landscape of parallel 
computing research: A view from Berkeley, ” 
Univ. California, Berkeley, CA, USA, Tech. 
Rep. UCB/EECS-2006-183, 2006.
 [
159]
 Y
. Zhang, M. Voss, and E. S. Rogers, 
“Runtime empirical selection of loop 
schedulers on hyperthreaded SMPs,” in 
Proc. 19th IEEE Int. Parallel Distrib. Process. 
Symp. (IPDPS), Apr. 2005, p. 44b.
 [
160]
 D
. Rughetti, P. D. Sanzo, B. Ciciani, and  
F. Quaglia, “Machine learning-based self-
adjusting concurrency in software 
transactional memory systems,” in Proc. 
IEEE 20th Int. Symp. Modeling Anal. 
Simulation Comput. Telecommun. Syst. ,  
Aug. 2012, pp. 278–285.
 [
161]
 C
. Delimitrou and C. Kozyrakis, “Quasar: 
Resource-efficient and Qos-aware cluster 
management,” in Proc. 19th Int. Conf. Archit. 
Support Program. Lang. Operat. Syst. 
(ASPLOS), 2014, pp. 127–144.
 [
162]
 X
. Chen and S. Long, “Adaptive multi-
versioning for openmp parallelization via 
machine learning,” in Proc. 15th Int. Conf. 
Parallel Distrib. Syst. (ICPADS) , 2009,  
pp. 907–912.
 [
163]
 M
. Castro, L. F. W. Góes, C. P. Ribeiro,  
M. Cole, M. Cintra, and J. F. Méhaut, “A 
machine learning-based approach for 
thread mapping on transactional memory 
applications,” in Proc. 18th Int. Conf. High 
Perform. Comput. , 2011, pp. 1–10.
 [
164]
 C
. Jung, S. Rus, B. P. Railing, N. Clark, and 
S. Pande, “Brainy: Effective selection of 
data structures,” in Proc. 32nd ACM 
SIGPLAN Conf. Program. Lang. Design 
Implement. (PLDI), 2011, pp. 86–97.
 [
165]
 Z
. Wang and M. F. P. O’Boyle, “Using 
machine learning to partition streaming 
programs,” ACM Trans. Archit. Code Optim. , 
vol. 10, no. 3, p. 20, 2013.
 [
166]
 C
. Chan, J. Ansel, Y. L. Wong,  
S. Amarasinghe, and A. Edelman, 
“Autotuning multigrid with petabricks,” in 
Proc. ACM/IEEE Conf. Supercomput. (SC) , 
2009, Art. no. 5.
 [
167]
 M
. Pacula, J. Ansel, S. Amarasinghe, and 
U.-M. O’Reilly, “Hyperparameter tuning in 
bandit-based adaptive operator selection,” 
in Proc. Eur. Conf. Appl. Evol. Comput. 
(EuroSys), 2012, pp. 73–82.
 [
168]
 J
. Ansel, “Siblingrivalry: Online autotuning 
through local competitions,” in Proc. Int. 
Conf. Compil., Archit. Synth. Embedded Syst. 
(CASES), 2012, pp. 91–100.
 [
169]
 M
. J. Voss and R. Eigemann, “High-level 
adaptive program optimization with 
adapt,” in Proc. 8th ACM SIGPLAN Symp. 
Principles Pract. Parallel Program. (PPoPP) , 
2001, pp. 93–102.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
Wang and O'Boyle: Machine Learning in Compiler Optimization
Proceedings of the IEEE  23
 [170] A . Tiwari and J. K. Hollingsworth, “Online 
adaptive code generation and tuning,” in 
Proc. IEEE Int. Parallel Distrib. Process. 
Symp. (IPDPS), May 2011, pp. 879–892.
 [
171]
 J
. Ren, L. Gao, H. Wang, and Z. Wang, 
“Optimise Web browsing on heterogeneous 
mobile platforms: A machine learning 
based approach,” in Proc. IEEE Int. Conf. 
Comput. Commun. (INFOCOM), May 2017, 
pp. 1–9.
 [
172]
 Y
. Zhu and V. J. Reddi, “High-performance 
and energy-efficient mobile Web browsing 
on big/little systems,” in Proc. HPCA ,  
Feb. 2013, pp. 13–24.
 [
173]
 Z
. Wang, M. F. P. O’Boyle, and  
M. K. Emani, “Smart, adaptive mapping of 
parallelism in the presence of external 
workload,” in Proc. IEEE/ACM Int. Symp. 
Code Generat. Optim. (CGO), Feb. 2013,  
pp. 1–10.
 [
174]
 D
. Grewe, Z. Wang, and M. F. P. O’Boyle, 
“A workload-aware mapping approach for 
data-parallel programs,” in Proc. 6th Int. 
Conf. High Perform. Embedded Archit. 
Compil. (HiPEAC), 2011, pp. 117–126.
 [
175]
 D
. Grewe, Z. Wang, and M. F. O’Boyle, 
“OpenCL task partitioning in the presence 
of GPU contention,” in Proc. Int. Workshop 
Lang. Compil. Parallel Comput. , 2013,  
pp. 87–101.
 [
176]
 L
. Tang, J. Mars, and M. L. Soffa, 
“Compiling for niceness: Mitigating 
contention for Qos in warehouse scale 
computers,” in Proc. 10th Int. Symp. Code 
Generat. Optim. (CGO), 2012, pp. 1–12.
 [
177]
 L
. Tang, J. Mars, W. Wang, T. Dey, and 
M. L. Soffa, “Reqos: Reactive static/
dynamic compilation for Qos in warehouse 
scale computers,” in Proc. 18th Int. Conf. 
Archit. Support Program. Lang. Oper. Syst. 
(ASPLOS), 2013, pp. 89–100.
 [
178]
 A
. Matsunaga and J. A. B. Fortes, “On the 
use of machine learning to predict the time 
and resources consumed by applications,” 
in Proc. 10th IEEE/ACM Int. Conf. Cluster 
Cloud Grid Comput. (CCGRID), 2010,  
pp. 495–504.
 [
179]
 S
. Venkataraman, Z. Yang, M. J. Franklin, 
B. Recht, and I. Stoica, “Ernest: Efficient 
performance prediction for large-scale 
advanced analytics,” in Proc. NSDI , 2016, 
pp. 363–378.
 [
180]
 S
. Sankaran, “Predictive modeling based 
power estimation for embedded multicore 
systems,” in Proc. ACM Int. Conf. Comput. 
Frontiers (CF), 2016, pp. 370–375.
 [
181]
 Y
. Zhang, M. A. Laurenzano, J. Mars, and 
L. Tang, “Smite: Precise QoS prediction on 
real-system smt processors to improve 
utilization in warehouse scale computers,” 
in Proc. 47th Annu. IEEE/ACM Int. Symp. 
Microarchit. (MICRO-47), Dec. 2014, pp. 
406–418.
 [
182]
 V
. Petrucci, “Octopus-man: QoS-driven 
task management for heterogeneous 
multicores in warehouse-scale computers,” 
in Proc. IEEE 21st Int. Symp. High Perform. 
Comput. Archit. (HPCA), Feb. 2015, pp. 
246–258.
 [
183]
 N
. J. Yadwadkar, B. Hariharan,  
J. E. Gonzalez, and R. Katz, “Multi-task 
learning for straggler avoiding predictive 
job scheduling,” J. Mach. Learn. Res. , vol. 17, 
no. 1, pp. 3692–3728, 2016.
 [
184]
 Y
. David and E. Yahav, “Tracelet-based  
code search in executables,” in Proc.  
35th ACM SIGPLAN Conf. Program. Lang. 
Design Implement. (PLDI) , 2014,  
pp. 349–360.
 [
185]
 Y
. David, N. Partush, and E. Yahav, 
“Statistical similarity of binaries,” in Proc. 
37th ACM SIGPLAN Conf. Program. Lang. 
Design Implement. (PLDI) , 2016,  
pp. 266–280.
 [
186]
 E
. Wong, T. Liu, and L. Tan, “Clocom: 
Mining existing source code for automatic 
comment generation,” in Proc. IEEE 22nd 
Int. Conf. Softw. Anal. Evol. Reeng. (SANER) , 
Mar. 2015, pp. 380–389.
 [ 187]  J . Fowkes and C. Sutton, “Parameter-free 
probabilistic api mining across GitHub,” in 
Proc. 24th ACM SIGSOFT Int. Symp. Found. 
Softw. Eng. (FSE), 2016, pp. 254–265.
 [
188]
 A
. T. Nguyen et al. , “API code 
recommendation using statistical learning 
from fine-grained changes,” in Proc. 24th 
ACM SIGSOFT Int. Symp. Found. Softw. Eng. 
(FSE), 2016, pp. 511–522.
 [
189]
 V
. Raychev, P. Bielik, and M. Vechev, 
“Probabilistic model for code with decision 
trees,” in Proc. ACM SIGPLAN Int. Conf. 
Object-Oriented Program. Syst. Lang. Appl. 
(OOPSLA), 2016, pp. 731–747.
 [
190]
 B
. Bichsel, V. Raychev, P. Tsankov, and 
 M. Vechev, “Statistical deobfuscation of 
Android applications,” in Proc. ACM 
SIGSAC Conf. Comput. Commun. Secur. 
(CCS), 2016, pp. 343–355.
 [191]
 P
. Balaprakash, R. B. Gramacy, and  
S. M. Wild, “Active-learning-based surrogate 
models for empirical performance tuning, ” 
in Proc. IEEE Int. Conf. Cluster Comput. 
(CLUSTER), Sep. 2013, pp. 1–8.
 [
192]
 W
. F. Ogilvie, P. Petoumenos, Z. Wang, and 
H. Leather, “Fast automatic heuristic 
construction using active learning,” in 
Proc. Int. Workshop Lang. Compil. Parallel 
Comput., 2014, pp. 146–160.
 [
193]
 M
. Zuluaga, G. Sergent, A. Krause, and  
M. Püschel, “Active learning for multi-
objective optimization,” in Proc. Int. Conf. 
Mach. Learn. , 2013, pp. 462–470.
 [
194]
 W
. F. Ogilvie, P. Petoumenos, Z. Wang, and 
H. Leather, “Minimizing the cost of 
iterative compilation with active learning,” 
in Proc. Int. Symp. Code Generat. Optim. 
(CGO), 2017, pp. 245–256.
 [
195]
 A
. Evaluation. About Artifact Evaluation . 
[Online]. Available: http://www.artifact-
eval.org/about.html
 [
196]
 c
Tuning Foundation. Artifact Evaluation for 
Computer Systems Research . [Online]. 
Available: http://ctuning.org/ae/
ABOUT THE AUTHORS
Zheng Wang received the Ph.D. degree in com-
puter science from The University of Edinburgh, 
Edinburgh, U.K., in 2011.
Currently, he is an Assistant Professor at 
Lancaster University, Lancaster, U.K., where he 
leads the Distributed Systems research group. 
From 2005 to 2007, he worked as an R&D Engi-
neer at IBM China. His research focus is in the 
areas of parallel compilers, runtime systems, 
code security, and the application of machine learning to tackle the chal-
lenging optimization problems within these areas. 
Prof. Wang received three best paper awards for his work on machine-
learning-based compiler optimization (PACT'10, CGO'17, and PACT'17). 
Michael O'Boyle is a Professor of Computer Sci-
ence at the University of Edinburgh, Edinburgh, 
U.K. He is a founding member of HiPEAC, the 
Director of the ARM Research Centre of Excel-
lence at Edinburgh, and the Director of the Engi-
neering and Physical Sciences Research Council 
(EPSRC) Centre for Doctoral Training in Perva-
sive Parallelism. He is best known for his work in 
incorporating machine learning into compilation 
and parallelization, automating the design and construction of optimizing 
technology. He has published over 100 papers and received three best 
paper awards. 
Prof. O'Boyle is a Senior Research Fellow of EPSRC and a Fellow of the 
British Computer Society (BCS). He was presented with the ACM Interna-
tional Symposium on Code Generation and Optimization (ACM CGO) Test 
of Time award in 2017. v


Autotuning Algorithmic Choice for Input Sensitivity
Yufei Ding⇤, Jason Ansel⇧, Kalyan Veeramachaneni⇧, Xipeng Shen⇤
Una-May O’Reilly⇧, Saman Amarasinghe⇧
⇤: North Carolina State University, Raleigh, NC, USA
⇧: Massachusetts Institute of Technology, Cambridge, MA, USA
⇤: {yding8, xshen5}@ncsu.edu
⇧:{jansel, kalyan, unamay, saman}@csail.mit.edu
Abstract
A daunting challenge faced by program performance autotuning is
input sensitivity, where the best autotuned conﬁguration may vary
with different input sets. This paper presents a novel two-level input
learning algorithm to tackle the challenge for an important class
of autotuning problems, algorithmic autotuning. The new approach
uses a two-level input clustering method to automatically reﬁne
input grouping, feature selection, and classiﬁer construction. Its
design solves a series of open issues that are particularly essential
to algorithmic autotuning, including the enormous optimization
space, complex inﬂuence by deep input features, high cost in
feature extraction, and variable accuracy of algorithmic choices.
Experimental results show that the new solution yields up to a 3x
speedup over using a single conﬁguration for all inputs, and a 34x
speedup over a traditional one-level method for addressing input
sensitivity in program optimizations.
Categories and Subject Descriptors D3.2 [Programming Lan-
guages]: Language Classiﬁcations – Concurrent, distributed, and
parallel languages; D3.4 [Programming Languages]: Processors -
Compilers.
General Terms Algorithms, Languages, Performance.
Keywords Petabricks, Autotuning, Algorithmic Optimization, In-
put Adaptive, Input Sensitivity, Two-level Input Learning.
1. Introduction
Application developers have invested a great deal of effort into ex-
tending the lifetime of software. To a large degree, this effort has suc-
ceeded. Millions of lines of code written decades ago are still being
used in new programs. A typical example of this can be found in the
C++ Standard Template Library (STL) routinestd::stable_sort,
distributed with the current version of GCC and whose implemen-
tation dates back to at least the 2001 SGI release of the STL. This
legacy code contains a hard coded optimization, a cutoff constant
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proﬁt or commercial advantage and that copies bear this notice and the full citation
on the ﬁrst page. Copyrights for components of this work owned by others than
ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to
republish, to post on servers, or to redistribute to lists, contact the Owner/Author.
Request permissions from permissions@acm.org or Publications Dept., ACM, Inc.,
fax +1 (212) 869-0481. Copyright 2015 held by Owner/Author. Publication Rights
Licensed to ACM.
PLDI’15 , June 13–17, 2015, Portland, OR, USA
Copyright c 2015 ACM 978-1-4503-3468-6/15/06. . . $15.00
DOI: http://dx.doi.org/10.1145/
of 15 between merge and insertion sort, that was designed in an era
when machines had1/100th the memory of modern machines. Our
tests have shown that higher cutoffs (60 to 150) perform much better
on current architectures. While this paradigm of write once and run
it everywhere is great for productivity, a major sacriﬁce of these
efforts is performance. Write once use everywhere often becomes
write once slow everywhere. We need programs withperformance
portability, where programs can re-optimize themselves to achieve
the best performance on widely different execution targets.
One of the most promising techniques to achieve performance
portability isprogram autotuning. Rather than hard-coding optimiza-
tions that only work for a single microarchitecture, or using fragile
heuristics, program autotuning exposes a search space of program
optimizations that can be explored automatically. Autotuning is used
to search this optimization space to ﬁnd the best conﬁguration to
use for each platform.
A fundamental problem in autotuning faced by programs and
libraries is input sensitivity. For a large class of problems, the best
optimization to use depends on the input data being processed. For
example, sorting an almost-sorted list that contain many repeated
values can be done most efﬁciently with a different algorithm than
one optimized for sorting random data. For many problems, no
single optimized program exists which can match the performance
of a collection of optimized programs autotuned for different subsets
of the input space. A common solution is to search for good
optimizations on every training input, based on which, it builds
a machine learning model that predicts the best optimization to use
in a new run according to the features of the new input [22, 30, 33,
39, 42].
Although the exhaustive approach works for certain program
optimizations, it is insufﬁcient foralgorithmic autotuning, a class
of autotuning problems, in which, the autotuners try to determine
the best ways to conﬁgure an algorithm or assemble multiple
algorithms into a recursive poly-algorithm to solve a problem [6].
Properties of algorithmic autotuning determines that it is often
infeasible to ﬁnd good optimizations—or algorithmic conﬁguration
in this context—for every training input. On one hand, in the
benchmarks we consider, the autotuner uses algorithmic choices
embedded in the program to construct arbitrary polyalgorithms. The
polyalgorithms process a single input through a hybrid of many
different individual techniques, which results in enormous search
spaces, ranging from10312 to 101016 possible conﬁgurations of
a programm [6]. Even through intelligent search algorithms [7],
ﬁnding a desirable conﬁguration at a training input takes hours or
days. On the other hand, algorithmic choices are often sensitive to
input features that are domain-speciﬁc and require deep, possibly
expensive, analysis to extract. For example, our singular value
decomposition benchmark is sensitive to the number of eigenvalues
in the input matrix. Hundreds or thousands of training inputs are
often prepared such that the complex input feature space could get
well covered. The large number of inputs, combined with the long
training time per input, means searching for good conﬁgurations
for every training input may take months or years for a program,
which makes the traditional exhaustive approach impractical for
algorithmic autotuning.
An intuitive solution is through clustering [11, 33, 39]. It ﬁrst
clusters training inputs into groups with the members in the same
group featuring similar values of predeﬁned input features, and
then ﬁnds the conﬁguration that works the best for the centroid
of each group. In a production run, when a new input comes, the
program extracts its feature values, based on which, it identiﬁes the
cluster that the input resembles the most, and then runs with that
cluster’s conﬁguration. We call this approach theone-level learning
approach. It has been used for intelligent program compilations and
other autotuning tasks. It avoids ﬁnding the best conﬁguration for
every training input and hence saves much training time.
The one-level approach however works inferiorly for algorithmic
autotuning, resulting in as much as 29x slowdown over using a
single conﬁguration for all inputs on our benchmarks (detailed in
Section 4). The reason comes from three properties of algorithmic
autotuning.
• Costly Features. Due to the nature of algorithmic selection,
algorithmic autotuning often involves extracting complex, and
sometimes costly, input features such as matrix eigenvalues.
Extraction of too many such features can add a large runtime
overhead to production runs.
• Mapping Disparity.The relations between input features and
algorithmic conﬁgurations are often complex. Two inputs hav-
ing similar values in the predeﬁned features may respond very
differently to the same algorithmic conﬁguration. Standard un-
supervised feature selection (e.g., PCA) [23] does not solve the
problem (more in Section 4), while standard supervised methods
(e.g., decision trees) [23] cannot apply since the best conﬁg-
urations for most training inputs are unknown. The disparity
between the predeﬁned feature space and the best conﬁgurations
causes inferior conﬁgurations to be selected by the one-level
method.
• Variable Accuracy.Unlike many compiler optimizations, algo-
rithmic autotuning has close interactions with variable accura-
cies. A program with different algorithmic conﬁgurations often
produces outputs of different quality, and the autotuner is re-
quired to produce conﬁgurations that will meet a user speciﬁed
quality of service level. For example, a Poisson’s equation solver
may require its output to match the output of a direct solver to a
certain precision with at least 95% conﬁdence. Meeting such a
requirement is difﬁcult especially because of the input variations.
For a class of inputs, a very fast polyalgorithm may sufﬁce to
achieve the accuracy, while for different inputs that same solver
may not meet the accuracy target. Without carefully treating
the dual objectives of accuracy and performance, the one-level
method frequently gives conﬁgurations that miss the accuracy
target, as Section 4 shows.
This paper presents a new solution that addresses all those chal-
lenges through a two-level learning framework. It starts with the
traditional one-level learning, and employs a second-level input
learning to close the mapping disparity gap between input features
and performance space, identify cost-effective input features, and
reconcile the stress between accuracy and performance. Speciﬁcally,
in the second level, after getting the conﬁguration (called aland-
mark) for the centroid of each input cluster, it regroups the training
inputs by drawing on the performance of all the landmark conﬁgura-
tions on each of the training inputs, based on which, it selects input
features and builds up appropriate input classiﬁers with algorithm
performance and computation accuracy simultaneously considered.
This design leverages the observation that running a program
once is usually much faster than autotuning the program, which
can involve thousands of runs, to ﬁnd a suitable algorithmic con-
ﬁguration. Its performance-based auto-reﬁnement of input clusters
closes the mapping disparity gap between the clustering results and
performance, without the need for ﬁnding the best conﬁguration
for every input. The design reconciles the stress between accuracy
and performance by introducing a programmer-centric scheme and
a coherent treatment to the dual objectives at both levels of learn-
ing. It seamlessly integrates consideration of the feature extraction
overhead into the construction of input classiﬁers. This two-level ap-
proach is able to achieve large speedups by making an optimization
choice which is sensitive to input variation while managing the input
space and search space complexity. In addition, we propose a new
language keyword that allows the programmer to specify arbitrary
domain-speciﬁc input features with variable sampling levels.
Contributions This paper makes the following contributions:
• Our system is, to the best our knowledge, the ﬁrst to simultane-
ously address the interdependent problems of variable accuracy
algorithms and the sensitivity of algorithmic selection on deep,
complex input features.
• Our novel two-level auto-reﬁnement approach solves the prob-
lem of input sensitivity for much larger, more complex algorith-
mic search spaces than would be tractable using state-of-the-art
techniques.
• We offer a principled understanding of the inﬂuence of pro-
gram inputs on algorithmic autotuning, and the relations among
the spaces of inputs, algorithmic conﬁgurations, performance,
and accuracy. We identify a key disparity between input prop-
erties, conﬁguration, and execution behavior which makes it
impractical to produce a direct mapping from input properties to
conﬁgurations and motivates our two-level approach.
• We show through an experimentally tested model that for many
types of search spaces there are rapidly diminishing returns
to adding more algorithmic conﬁgurations that a program can
select across inputs. Cross-input adaptation on a few algorith-
mic conﬁgurations goes a long way, while a large number of
conﬁgurations are often unnecessary.
• Experimental results show that the two-level solution produces
up to a 3x speedup over using a single conﬁguration for all inputs
in algorithmic selection, and a 34X speedup over prior one-level
method.
2. Language and Usage
This work is developed upon the PetaBricks language and its
compiler [6]. This section will brieﬂy describe some of the key
features of PetaBricks, and then discuss our extensions to it to
support the exploration of input sensitive autotuning of algorithms.
Figure 1 shows a fragment of the PetaBricks Sort benchmark
extended for input sensitivity. We will use this as a running example
throughout this section.
2.1 Algorithmic Choice
The most distinctive feature of the PetaBricks language is algorith-
mic choice. Using algorithmic choice, a programmer can deﬁne
a space of possible polyalgorithms rather than just a single algo-
rithm from a set. There are a number of ways to specify algorithmic
choices, but the most simple is theeither...or statement shown at
lines 6 through 16 of Figure 1. The semantics are that when the
either...or statement is executed, exactly one of the sub blocks will
1 function Sort
2 to out[n]
3 from in[n]
4 input_feature Sortedness, Duplication
5 {
6 either {
7 InsertionSort(out, in);
8 }o r{
9 QuickSort(out, in);
10 }o r{
11 MergeSort(out, in);
12 }o r{
13 RadixSort(out, in);
14 }o r{
15 BitonicSort(out, in);
16 }
17 }
18
19 function Sortedness
20 from in[n]
21 to sortedness
22 tunable double level (0.0, 1.0)
23 {
24 int sortedcount = 0;
25 int count = 0;
26 int step = (int)(level*n);
27 for(int i=0; i+step<n; i+=step) {
28 if(in[i] <= in[i+step]) {
29 // increment for correctly ordered
30 // pairs of elements
31 sortedcount += 1;
32 }
33 count += 1;
34 }
35 if(count > 0)
36 sortedness = sortedcount / (double) count;
37 else
38 sortedness = 0.0;
39 }
40
41 function Duplication
42 from in[n]
43 to duplication
44 ...
Figure 1: PetaBricks pseudocode forSort with input features
be executed, and the choice of which sub block to execute is left up
to the autotuner.
The either...or primitive implies a space of possible polyalgo-
rithms. In our example, many of the sorting routines (QuickSort,
MergeSort, and RadixSort) will recursively call Sort again, thus,
the either...or statement will be executed many times dynamically
when sorting a single list. The autotuner uses evolutionary search
to construct polyalgorithms which make one decision at some calls
to the either...or statement, then different decisions in the recursive
calls [8].
These polyalgorithms are realized throughselectors (sometimes
called decision trees) which efﬁciently select which algorithm to
use at each recursive invocation of theeither...or statement. Figure 2
shows an example: The selector creates a polyalgorthm that ﬁrst
uses MergeSort to decompose a problem into lists of less than 1420
elements, then uses QuickSort to decompose those lists into lists
of less than 600 elements, and ﬁnally these lists are sorted with
InsertionSort.
N < 600
N < 1420
    MergeSort
QuickSort
Y
Y
N
N
InsertionSort
Figure 2: A selector that realizes a polyalgorithm for sorting.
2.2 Input Features
To support exploration of input sensitivity, in this work we introduce
a new keywordinput_feature into the PetaBricks language, shown
on lines 4 and 5 of Figure 1. Theinput_feature keyword speciﬁes a
programmer-deﬁned function, afeature extractor, that will measure
some domain speciﬁc property of the input to the function. A feature
extractor must have no side effects, take the same inputs as the
function, and output a single scalar value. The autotuner will call
this function as necessary.
Feature extractors may have tunable parameters which control
their behavior. For example, thelevel tunable on line 23 of
Figure 1, is a value that controls the sampling rate of the sortedness
feature extractor. Higher values will result in a faster, but less
accurate measure of sortedness.Tunable is a general language
keyword that speciﬁes a variable to be automatically set by the
autotuner, and two values indicating the allowable range of the
tunable (in this example between 0.0 and 1.0).
Users may useinput_feature keyword to deﬁne all the potentially
relevant features based on her domain knowledge; some of the fea-
tures could be more costly or less important than others. Effectively
selecting the subset of features that are critical and beneﬁcial is part
of the key support offered by our two-level approach. Suitable values
of the tunable parameters are also part of the algorithmic conﬁgu-
rations that our enhanced autotuner is designed to help determine
adaptively across program inputs.
After autotuner selects the right subset of input features and
their tunable parameter values, every execution of the program of
interest will call the corresponding feature extraction functions at
the beginning of the execution. The collected input features will be
used to select the conﬁguration of the algorithm that best suits the
current input. Section 3 will elaborate on how input features are
used by the auotuner.
2.3 Variable Accuracy
One of the key features of the PetaBricks programming language is
its support for variable accuracy algorithms, which can trade output
accuracy for computational performance (and vice versa) depending
on the quality of service requirements or program speciﬁcation. Ap-
proximating ideal program outputs is a common technique used for
solving computationally difﬁcult problems, adhering to processing
or timing constraints, or optimizing performance in situations where
perfect precision is not necessary. Algorithmic methods for produc-
ing variable accuracy outputs include approximation algorithms,
iterative methods, data resampling, and other heuristics.
At a high level, PetaBricks extends the idea of algorithmic choice
to include choices between different accuracies. The programmer
speciﬁes a programmer-deﬁnedaccuracy metric to measure the
quality of the output, and sets anaccuracy target. The autotuner
must then consider a two dimensional objective space, where its
TrainingDeploymentInput Classifier
Input Aware Learning
Program
Training Inputs
Feature Extractors Insights:  - Feature Priority List  - Performance Bounds
Input
Select
Input Optimized Programs
Training
Selected  ProgramRun
Figure 3: Usage of the system both at training time and deployment
time. At deployment time the input classiﬁer selects the input
optimized program to handle each input. Input Aware Learning
is described in Section 3.
ﬁrst objective is to meet the accuracy target (with a given level of
conﬁdence) and the second objective is to maximize performance. A
detailed description of the variable accuracy features of PetaBricks
is given in [5].
2.4 Usage
Figure 3 describes the usage of our system for input sensitive al-
gorithm design. The input aware learning, described in Section 3,
takes the user’s program (containing algorithmic choices), the fea-
ture extractors speciﬁed by theinput_feature language keyword
and input exemplars as input. The output of the learning is an input
classiﬁer, a subset of input features, and a set of input optimized
programs, each of which has been optimized for speciﬁc class of
inputs.
When an arbitrary input is encountered in deployment, the
classiﬁer created by learning is used to select an input optimized
program which is expected to perform the best on this input. The
classiﬁer will use the subset of the feature extractors to probe the
input. Finally, the selected input optimized program will process the
input and return the output.
3. Input Aware Learning
Motivated by the particular complexities of algorithmic autotuning,
we develop a two-level learning framework. The ﬁrst level is shown
in Figure 4. In its ﬁrst step it clusters and groups the input space
into a ﬁnite number of input classes and then uses the autotuner to
identify a good algorithmic conﬁguration for each cluster’s centroid.
In addition however, to provide data on the mappings among inputs,
conﬁguration and performance, it executes every exemplar (i.e.,
training input) using the conﬁguration of each cluster. These results
will be used at the next level.
The second level is shown in Figure 5. It ﬁrst reﬁnes the input
clustering results by drawing on the previously collected evidence
on the inputs and their performance on a small set of "landmark"
conﬁgurations. It then builds a large number of classiﬁers each
different by which input features it references and/or the algorithm
used to derive the classiﬁer. It then computes an objective score for
every classiﬁer and selects the best one as the production classiﬁer.
Together, these two levels create an approach that is able to achieve
large speedups by its sensitivity to input variation and conﬁguration
inﬂuence on program performance. We provide detail on each level.
3.1 Level 1
The main objective of Level one is to identify a set of conﬁgurations
for each class of inputs. We call these conﬁgurations “landmarks”.
Speciﬁcally, there are four steps in this level of learning.
Feature ExtractionInput Space Clustering
Evolutionary Autotuner
Features
Input Optimized Programs
Centroids
Performance Testing
Training Inputs
Input Labels
Figure 4: Selecting representative inputs to train input optimized
programs.
• Step 1: Feature Extraction We assemble a feature vector
for each training input using the values computed by thein-
put_feature procedures of the program. For each property, by
using a tunable parameter such aslevel in theinput_feature
procedure in the program, we have collected values atz different
costs which are what we call features.
• Step 2: Input ClusteringWe ﬁrst normalize the input feature
vectors to avoid biases imposed by the different value scales in
different dimensions. We then group the inputs into a number
of clusters (100 in our experiments; sensitivity of the number
discussed in Section 4) by running a standard clustering algo-
rithm (e.g., K-means) on the feature vectors. For each cluster, we
compute its centroid. Note that the programmer is only required
to provide aninput_feature function.
• Step 3: Landmark CreationWe autotune the program using
the PetaBricks evolutionary autotunermultiple times, once for
each input cluster, using its centroid as the presumed inputs.
While the default evolutionary search in autotuner generates ran-
dom inputs at each step of search, we use a ﬂag which indicates
it should use the centroid instead. We call each conﬁguration for
each cluster’s centroid as input data to the program, alandmark.
The stochasticity of the autotuner means we may get different
conﬁgurations albeit perhaps equal performing ones each time.
• Step 4: Performance MeasurementWe run each landmark
conﬁguration on every training input and record both the ex-
ecution time and accuracy (if applicable) as its performance
information.
Another simple way to select the landmarks is based on random
selection. However, the quality of the landmarks selected is worse
than that obtained with clustering algorithm based on input features,
especially when the number of landmarks (i.e., samples) is small. For
example, with 5 conﬁgurations, uniformly picked landmarks result
in 41% degradation of performance than selection with kmeans. As
the number of conﬁgurations increases, the gap shrinks. But because
ﬁnding the conﬁguration on one sample takes a long time, clustering
algorithms (e.g., K-means) have their advantages.
Besides, we note that there is an alternative way to accomplish
Steps 1 through 3 and identify landmarks. We could ﬁnd the best
conﬁguration for each training input, group these conﬁgurations
based on their similarity, and use the centroids of the groups as
landmark conﬁgurations. This unfortunately is infeasible because
it is too time consuming to ﬁnd a suitable conﬁguration for every
input example. Modeling is not as effective as search, and search,
evolutionary in the case of PetaBricks, involves the composition and
evaluation of hundreds of thousands conﬁgurations, taking hours
or even days to ﬁnish [6]. This also has the problem we mention
previously: similar conﬁgurations do not have matching algorithm
FeaturesInput Labels
Decision Tree
Max A Priori
Adaptive Tree
Classifier Constructors
1...m
0
m+1
Classifier Selector
Selection Objective
Considers cost of extracting needed features
Input Classifier
Figure 5: Constructing and selecting the input classiﬁer.
performance. For these reasons, we cluster in the space of inputs,
and then use the second-level learning to automatically reﬁne the
clusters.
3.2 Level 2
The main objective of this level is to identify a production classiﬁer
that is able to efﬁciently identify the best landmark conﬁguration
to use for a new input. The challenge is to determine which input
features are good predictors of a high performing classiﬁer. Because
it is unknown, the ﬁrst sub-goal is to generate a candidate set of
classiﬁers each with a unique set of featuresand all using the
data that provides evidence of the relationship between inputs,
conﬁgurations and algorithm performance. Incorporating this step
and its wealth of evidence drastically improves results. The second
sub-goal follows: choose among the candidates to identify the best
one for production. Through the following discussion, we assume
that there areu input feature extractors with each havingz possible
levels of collection cost; the whole feature set is hence as large as
M = u ⇤ z. We useK1 to represent the number of landmarks.
Cluster Reﬁnement before Classiﬁer Learning An important
step before achieving both sub-goals is to regroup the training inputs
to reﬂect their performance on the landmark conﬁgurations collected
in the ﬁrst level. In this step, we make each set of example inputs,
their features, feature extraction costs, execution times and accuracy
scores for each landmark conﬁguration, a row of a dataset. We
append to each row a label which is the best conﬁguration for the
input.
More formally, we create a datatable of 4-tuples. Each 4-tuple
is < F, T, A, E >, whereF is anM-dimensional feature vector
for this input,T and A are vectors of length1 ⇥ K1 where the
ith entry represents the execution time and accuracy achieved for
this input when theith landmark conﬁguration is applied.E is
an M-dimensional vector giving us the values for time taken for
extraction of the features. We ﬁrst generate labelsL 2{ 1 ...K 1}
for each input. Labelli represents the best conﬁguration for theith
input. For problems where only minimizing the execution time is an
objective (for examplesorting) the label for theith input is simply
arg minj Tj
i . For problems where both accuracy and execution time
are objectives, we use a user-provided threshold (per the requirement
of the particular problem and usage context) for the accuracy and
then select the subset of conﬁgurations that meet the accuracy
threshold and among them pick the one that has the minimum
execution time. For the case, in which none of the conﬁgs achieve
desired accuracy threshold, we pick the conﬁguration that gives the
maximum accuracy. These labels regroup the inputs intoK1 clusters
that directly reﬂect the performance of various conﬁgurations on
those inputs.
This step and the clustering in the ﬁrst level together forms our
two-level clustering strategy. By enabling an auto-reﬁnement of
input clusters, the strategy helps close the mapping disparity gap
between the predeﬁned input features and the performance space
that the one-level learning approach suffers from.
Setting Up the Cost MatrixBecause we haveK1 landmark con-
ﬁgurations, we have aK1-class problem; K1=100 in our experi-
ments. For suchextreme-class classiﬁcation problem, it is paramount
that we set up a cost matrix. The cost matrixCij represents the cost
incurred by misclassifying a feature vector that is labeledi as class
j. The classiﬁer learning algorithms use the cost matrix to evaluate
the classiﬁers. To construct the cost matrix for this problem we
followed this procedure. For every input, labeledi, we calculate the
performance difference between theith landmark and thejth land-
mark. We assign a cost that is the average performance difference
for all inputs that are labeledi. This corresponds toCp
i,j. For those
benchmarks that have accuracy as a requirement, we also include a
accuracy penalty matrixCa
i,j, which is the ratio of inputs for which
accuracy constraint is not met byjth conﬁguration. We construct
the ﬁnal cost matrix asCij =   · Ca
i,j · maxt(Cp
i,t)+ Cp
i,j. The
cost is a combination of accuracy penalty and performance penalty
where the former is a leading factor, while the latter acts as a tuning
factor. We tried different settings for  ranging from 0.001 to 1, to
get the best performance. We found 0.5 to be the best and use that.
We then divide our inputs into two sets, one set is used for
training the classiﬁer, the other for testing. We next pass the training
set portion of the dataset to different classiﬁcation methods which
either reference different features or compute a classiﬁer in a
different way. Formally a method derives classiﬁerC referencing a
feature setfc ⇢ F to predict the label, i.e.,C(Fi) ! Li.
Classiﬁer Learning We now describe the classiﬁers we derive and
the methods we use to derive them.
(1) Max-apriori classiﬁer: This classiﬁer evaluates the empirical
priors for each conﬁguration label by looking at the training data. It
chooses the conﬁguration with the maximum prior (maximum num-
ber of inputs in the training data had this label) as the conﬁguration
for the entire test data. There is no training involved in this other
than counting the exemplars of each label in the training data. As
long as the inputs follow the same distribution in the test data as in
the training data there is minimal mismatch between its performance
on training and testing. It should be noted that this classiﬁer does
not have to extract any features of the input data.
Advantages: No training complexity, no cost incurred for ex-
traction of features.
Disadvantages: Potentially highly inaccurate, vulnerable to
error in estimates of prior.
(2) Exhaustive Feature Subsets Classiﬁers: Even though we have
M features in the machine learning dataset, we only haveM
z
input properties. The bottom level feature takes minimal time to
extract and only looks at the partial input, while the top level takes
signiﬁcant amount of time as it looks at the entire input. However,
features extracted for the same input could be highly correlated.
Hence, as a logical progression, we select a subset of features size
of which ranges between1 ... M
z where each entry is for a property.
For each property we allow only one of its level to be part of the
subset, and also allow it to be absent altogether. So for 4 properties
with z =3 levels we can generate44 unique subsets. For each
of these 256 subsets we then build a decision tree classiﬁer [38]
yielding a total of 256 classiﬁers.
The feature extraction time associated with each classiﬁer is
dependent on the subset of features used by the classiﬁer, ie. for
an inputi, it is the summationP
j Ej
i . The decision tree algorithm
references the label and features and tries to minimize its label
prediction error. It does not reference the feature extraction times,
execution times or accuracy information. These will be referenced
in accomplishing the second sub-goal of classiﬁer selection.
Because we wanted to avoid any “learning to the data”, we
divided the data set into 10 folds and trained 10 times on different
sets of nine folds while holding out the 10th fold for testing (“10
fold cross validation”). Of the 10 classiﬁers we obtained for each
feature set, we chose the one that on average performed better.
Advantages: Feature selection could simply lead to higher
accuracy and allow us to save feature extraction time.
Disadvantages: More training time, not scalable should the
number of properties increases.
(3) All features Classiﬁer: This classiﬁer is one of the 256 Exhaus-
tive Feature Subsets classiﬁers which we call out because it uses all
the u unique features.
Advantages: Sometimes lead to higher accuracy classiﬁcation.
Disadvantages: More training time, higher feature extraction
time, no feature selection.
(4) Incremental Feature Examination classiﬁer: Finally, we de-
signed a classiﬁer which works on an input in a sequential fashion.
First, for every featurefm 2 R, we divide it into multiple decision
regions {dm
1 ...d m
j } where j   K1. We then model the probability
distributions under each class for each feature under each decision
region Pm,j,k(fm = dm
j |Li = k). Given a pre-speciﬁed order of
features it classiﬁes in the following manner when deployed:
Step 1: Calculate the feature: Evaluate themth feature for the
input and apply the thresholds to identify the decision region it
belongs to.
Step 2: Calculate posterior probabilities: The posterior for a
conﬁguration (class label)k, given all the features{1 ...m }
acquired so far and letd1
1 ...d i
j be the decision regions they
belong to, is given by:
P (Li = k|f1...m)=
⇧mPm,j,k(fm = dm
j |Li = k)P (L = k)
P
k ⇧mPm,j,k(fm = dm
j |Li = k)P (L = k)
(1)
Step 3: Compare and decide: We pre-specify a threshold on the
posterior   and we declare the conﬁguration (class label) ask
if its posteriorP(Li = k|f1...m) >  . If none of the posteriors
are greater than this threshold, we return to step 1 to acquire
more features.
In this method, we incrementally acquire features for an input pointi
based on judgement as to whether there is enough evidence (assessed
via posteriors) for them to indicate one conﬁguration. This results in
a variable feature extraction time for different inputs thus providing
potential further reductions in feature extraction time at the time the
production classiﬁer is applied to new inputs. For all the classiﬁers
preceding this one, we extracted the same number of features for all
the inputs.
This method can be applied after the previous method has found
the best subset to further save on feature extraction time.
To train this classiﬁer, we need to ﬁnd the optimal decision
regions for each feature and the threshold on posterior  such that
the performance measurements mentioned above are minimized.
This could be done via a simple continuous parameter search
algorithm. Usually, more decision regions per feature help increase
the performance and in some cases search over orders could help.
This is the only classiﬁer in our system where we introduce the
domain speciﬁc cost function into the inner loop of training of a
classiﬁer.
Advantages: Reduced feature extraction time, scalable as the
number of attributes increase.
Disadvantages: One possible drawback for this classiﬁer is that
it requires a storage ofi ⇥ j ⇥ k number of probabilities in a look
up table. Training time.
Candidate Selection of Production ClassiﬁerAfter we generate
a set of classiﬁers based on different inputs or methods, we next need
to select one as the production classiﬁer. We start by applying every
classiﬁer on the test set and measuring the performance (execution
time and accuracy, if required) of the algorithm when executing with
its predicted conﬁguration. We can compare this performance to
that of the other conﬁgurations. There are three objectives for the
production classiﬁer: 1) minimize execution time; 2) meet accuracy
requirements; and, 3) minimize the feature extraction time.
Let  i be the minimum execution time for the inputi by all
the representative polyalgorithms. Let (i, Li) be the execution
time of i when its class label isLi, given by classiﬁerC and
gi = P
j Tj,j 2 fc be the feature extraction time associated with
this classiﬁcation.
Given a classiﬁer we measure its efﬁcacy for our problem as
follows:
For time only:ri) for classifying a data point to conﬁgurationci
will beri = ( i, ci)+ gi. The cost function (represented by
R) for all the data will be the average of all their costs, that is,
R = P
i(ri)/N, where N is the total number of inputs. We
refer to R as performance cost in the following description.
For time and accuracy: For the case of variable accuracy, the
efﬁcacy is calculated with two thresholds the user deﬁnes
according to the requirements of the particular problem and
usage context. The ﬁrst,H1, is the accuracy threshold, that is,
only when the accuracy of the computation (e.g., binpacking)
result at a data list exceedsH1, the result is useful. The second,
H2, is the satisfaction threshold. Suppose the fraction of inputs
whose computation results are inaccurate (ie. accuracy is less
than H) iss when classiﬁer C is applied to our data set. If
s<H 2, the classiﬁer is considered invalid (or incurring a huge
cost). If the classiﬁer meets this target, the performance cost of
this classiﬁer is calculated as deﬁned above.
3.3 Discussion of the Two Level Learning
This two level learning has several important properties.
First, it uses a two level design to systematically addressmapping
disparity. Its ﬁrst phase takes advantage of the properties of inputs to
identify landmark conﬁgurations. It then furnishes evidence of how
example inputs and different landmarks affect program performance
(execution time and accuracy). Its second phase uses this evidence
to (indirectly) learn a production classiﬁer. By classifying based
upon best landmark conﬁguration it avoids misusing similarity of
inputs. The means by which it evaluates each candidate classiﬁer
(trained to identify the best landmark) to determine the production
classiﬁer takes into account the performance of the conﬁgurations
both in terms of execution time and accuracy.
Second, this two level learning reconciles the stress between
accuracy and performance by introducing a programmer-centric
scheme and a coherent treatment to the dual objectives at both levels
of learning. For variable-accuracy algorithms, absolute accuracy
guarantees on arbitrary inputs are difﬁcult to have (just like in most
software testing or machine learning tasks). The scheme we use
allows programmers to provide two thresholds. One is anaccuracy
threshold, which determines whether the computation result is
considered as accurate; the other is asatisfaction threshold, which
determines whether the statistical accuracy guarantee (e.g., the
calculation is accurate in 95% time) offered by a conﬁguration meets
the programmer’s needs. The former indicates the desired accuracy
target and the latter indicates the desired proportion of inputs that hit
that target. The accuracy thresholds are benchmark-speciﬁc, given
by the authors or users of each benchmark. For example, for a
Bin Packing benchmark, the accuracy can be set as the average of
the occupied fractions of all bins; for a clustering benchmark, it is
the difference in the average point-to-center distances between the
clustering result from the benchmark and the result from a canonical
clustering algorithm. The scheme is consistently followed by both
levels of learning. The two-level method offers a certain kind of
statistical guarantee of the accuracy.
Third, it seamlessly integrates consideration of the feature extrac-
tion overhead into the construction of input classiﬁers. Expensive
feature extractions may provide more accurate feature values but
may not be amortized by the time savings brought by them. The
key question is to select the feature extractions that can strike a
good tradeoff between the usefulness of the features and the over-
head. Our two level learning framework contains two approaches to
ﬁnding the sweet point. One uses exhaustive feature selection, the
other uses adaptive feature selection. Both strive to maximize the
performance while maintaining the accuracy target.
Fourth, our learning framework maintains an open design, allow-
ing easy integration of other types of classiﬁers. Any other classiﬁ-
cation algorithm could be integrated into our system without loss of
generality. Moreover, it takes advantage of the PetaBricks autotuner
to intelligently search through the vast conﬁguration space.
4. Evaluation
To measure the efﬁcacy of our system, we tested it on a suite of
6 parallel PetaBricks benchmarks [5]. These benchmarks are up
to thousands of lines long and have search spaces with10312 to
101016 possible conﬁgurations. Of these benchmarks, ﬁve require
variable accuracy. Each of these benchmarks was modiﬁed to add
feature extractors for their inputs and a richer set of input generators
to exercise these features. Each feature extractor was set to 3
different sampling levels providing more accurate measurements at
increasing costs. Tests were conducted on a 32-core (8 ⇥ 4-sockets)
Xeon X7550 system running GNU/Linux (Debian 6.0.6). There are
50,000 to 60,000 inputs per program. Around half of them are used
for training and the half for testing. All reported results are based
on cross validation, which ensures the exclusion of training inputs
from the testing inputs.
Section 4.1 describes the deﬁnition of the accuracy metric
for each benchmark along with the accuracy thresholds used in
the experiments. We set the satisfaction threshold to 95% for all
experiments, which means that the accuracy threshold should be
satisﬁed on at least 95% of the inputs.
Three primary baselines are used for comparison.
• Static Oracle.It uses a single conﬁguration for all inputs. This
conﬁguration is selected by trying each input optimized program
conﬁguration and picking the one with the best performance
(and meeting the satisfying accuracy threshold when applicable).
The static oracle is the performance that would be obtained by
not using our system and instead using an autotuner without
input adaptation. In practice the static oracle may be better than
some ofﬂine autotuners, because such autotuners may train on
non-representative sets of inputs.
• One Level.It is the aforementioned traditional one-level learning
approach. It clusters inputs based on the predeﬁned features. It
fails to treat the overhead of extracting features and the disparity
between input feature and performance space, and is oblivious
to variable accuracy.
• Dynamic Oracle.It usesthe bestconﬁguration for each input.
Due to the size of the search space, an oracle that tries all
possible conﬁgurations is not possible. The dynamic oracle we
use represents the best that is possible in the second level given
the landmarks available. It gives the upper bound of the possible
performance that can be obtained by our input classiﬁer (and
reaches the accuracy satisfaction threshold). It is equivalent to a
classiﬁer that always picks the best optimized program (given
the landmarks available) and requires no features to do so.
4.1 Benchmarks
We use the following 6 benchmarks to evaluate our results.
Sort This benchmark sorts a list of doubles using InsertionSort,
QuickSort, MergeSort, or BitonicSort. The merge sort has a variable
number of ways and has choices to construct a parallel merge sort
polyalgorithm. Sort is the only non-variable accuracy benchmark
shown. Input variability comes from different algorithms having
fast and slow inputs, for example QuickSort has pathological input
cases and InsertionSort is good on mostly-sorted lists. For input
features we use standard deviation, duplication, sortedness, and the
performance of a test sort on a subsequence of the list.
Sort1 results are sorting real-world inputs taken from the Central
Contractor Registration (CCR) FOIA Extract, which lists all gov-
ernment contractors available under FOIA from data.gov [1]. Sort2
results are sorting synthetic inputs generated from a collection of
input generators meant to span the space of features.
Clustering The clustering benchmark assigns points in 2D coor-
dinate space to clusters. It uses a variant of the kmeans algorithm
with choices of either random, preﬁx, or centerplus initial condi-
tions. The number of clusters (k) and the number of iterations of
the kmeans algorithm are both set by the autotuner. The accuracy
metric isP ˆdi/ Pdi, where, ˆdi and di are the distance between
a point and its cluster center in a canonical clustering algorithm
and in our experiment respectively. The accuracy threshold is set to
0.8. Clustering uses input the features: radius, centers, density, and
range.
Clustering1 results are clustering real-world inputs taken from
the Poker Hand Data Set from UCI machine learning repository [2].
Clustering2 results are clustering synthetic inputs generated from a
collection of input generators meant to span the space of features.
Bin Packing Bin packing is a classic NP-hard problem where the
goal of the algorithm is to ﬁnd an assignment of items to unit sized
bins such that the number of bins used is minimized, no bin is above
capacity, and all items are assigned to a bin. The bin packing bench-
mark contains choices for 13 individual approximation algorithms:
AlmostWorstFit, AlmostWorstFitDecreasing, BestFit, BestFitDe-
creasing, FirstFit, FirstFitDecreasing, LastFit, LastFitDecreasing,
ModiﬁedFirstFitDecreasing, NextFit, NextFitDecreasing, WorstFit,
and WorstFitDecreasing. Bin packing contains 4 input feature ex-
tractors: average, standard deviation, value range, and sortedness. Its
accuracy metric is deﬁned as the average of the occupied fractions
of all bins. The accuracy threshold is set to 0.95.
Singular Value Decomposition The SVD benchmark attempts
to approximate a matrix using less space through Singular Value
Decomposition (SVD). For anym ⇥ n real matrixA with m   n,
the SVD of A isA = U⌃V T . The columnsui of the matrixU,
the columns vi of V , and the diagonal values i of ⌃ (singular
values) form the best rank-k approximation ofA, given byAk =Pk
i=1  iuivT
i . Only the ﬁrstk columns ofU and V and the ﬁrst
k singular values  i need to be stored to reconstruct the matrix
approximately. The choices for the benchmark include varying the
number of eigenvalues used and changing the techniques used to
ﬁnd these eigenvalues. The accuracy metric used is the logrithm of
the ratio of the root mean squared (RMS) error of the initial guess
(the zero matrix) to the RMS error of the output. The RMS errors
are relative to the input matrixA. The accuracy threshold is set to
0.7. For input features we used range, the standard deviation of the
input, and a count of zeros in the input.
Poisson 2D The 2D Poisson’s equation is an elliptic partial dif-
ferential equation that describes heat transfer, electrostatics, ﬂuid
dynamics, and various other engineering disciplines. The choices in
this benchmark are multigrid, where cycle shapes are determined
by the autotuner, and a number of iterative and direct solvers. As
an accuracy metric, we used the logrithm of the ratio of the RMS
error of the initial guess to the RMS error of the guess afterwards.
The RMS errors are relative to the accurate solution of the equation.
The accuracy threshold is set to 7. For input features we used the
residual measure of the input, the standard deviation of the input,
and a count of zeros in the input.
Helmholtz 3D The variable coefﬁcient 3D Helmholtz equation is
a partial differential equation that describes physical systems that
vary through time and space. Examples of its use are in the modeling
of vibration, combustion, wave propagation, and climate simulation.
The choices in this benchmark are multigrid, where cycle shapes are
determined by the autotuner, and a number of iterative and direct
solvers. The accuracy metric is the same as used inPoisson 2D. The
accuracy threshold is also set to 7. For input features we used the
residual measure of the input, the standard deviation of the input,
and a count of zeros in the input.
4.2 Experimental Results
Table 1 reports the overall performance. In this section, all speedup
numbers use the performance of static oracle as the baseline unless
noted otherwise. Results from all methods exceed the accuracy sat-
isfaction threshold except for the one-level method, the satisfaction
rates of which are reported in the rightmost column of Table 1. Com-
pared to the performance by the static oracle, the one-level approach
performs better on only two out of the eight tests (with one of them
missing the accuracy target). On the other tests, the slowdown by
that approach is as much as 29 times. The comparison between
columns 5 and 6 in Table 1 indicates that the large feature extraction
cost is one of the reasons for the poor performance. But even without
considering that cost, column 5 shows that the one-level method
still brings much less speedups on sort1 and sort2, and much lower
accuracy on all other benchmarks, compared to what the two-level
method brings. The reason is its poor classiﬁer construction for
the lack of consideration of the space disparity. Note that during
the search for the best conﬁguration for a landmark in the 1-level
method, accuracy is considered. The problem is that despite that the
conﬁguration works well for the landmark input, it does not work
well on other inputs that belong to the same cluster as the landmark,
because of the poor clustering results by the 1-level method, which
causes the poor accuracy.
On the other hand, the second level clustering and the follow-
up classiﬁer construction in the two-level method help ﬁnd the
reﬁned input feature set that better correlates with the performance
of conﬁgurations on inputs. The second-level clustering is critical.
Take Kmeans as an example, 73.4% of the data points changed
their clusters when the second-level clustering is applied. The better
classiﬁers, plus better input features it selects, make the two-level
method signiﬁcantly outperform the one-level method.
Table 1 shows that the one-level method, when its overhead
in input feature collection is not counted, gives a higher speedup
than the two-level method on clustering1. Unlike the one-level
approach, the two-level approach considers not only performance
but also cost and the computation accuracy of resulting program.
On clustering1, it uses a subset of input features on which it could
provide good performance and satisfying accuracy (>95%), while
the one-level approach uses all input features, which, although it
gives higher performance when cost is not counted, has an inferior
overall performance and a poor accuracy (54.56%).
The dynamic oracle results in Table 1 shows a spectrum of
potential among the tests, reﬂecting the different sensitivity of the
problems to input variations. Applications whose performance vary
more across inputs have more room for speedup by adapting to
speciﬁc inputs. Correspondingly, the speedups by the two level
approach show a range as well, from 1.04x to over 3x. The speedups
are close to what the dynamic oracle produces except on the
sort benchmark, which may require some more advanced feature
extractors to be provided.
Feature extraction time barely changes the speedups by the two-
level method, indicating that only a small amount of time was spent
on feature extraction. Generally, the less expensive features (in terms
of feature extraction time) were sufﬁcient for the two level approach
to meet the best performance. The low extraction cost of the selected
features largely obviated the need for the adaptive classiﬁer in the
current scenario.
In the sort benchmark, we tried both real world inputs (sort1)
and inputs from our own generator (sort2). For real world input,
the best classiﬁer used the sorted list and sortedness features at its
intermediate sampling level and the duplication and deviation at the
cheapest level, and achieved 2.95x speedup. For inputs from our
own generator, the best classiﬁer used the sorted list and sortedness
features at its intermediate sampling level, achieving speedup of
3.05x.
In the clustering benchmark, we also tried real world inputs and
those from our own generator. For real world input, the best classiﬁer
used the density feature at its cheapest level, and algorithms selected
by the classiﬁer causes a 2.38x shorter execution time. For our own
generator, the best classiﬁer used the centers feature at its cheapest
sampling level, achieving a 1.45x speedup. However, centers feature
is the most expensive feature relative to execution time, which lowers
the effective speedup to just 1.18x.
In the binpacking benchmark, the best classiﬁer used the devia-
tion and sortedness features at the intermediate level. The feature
extraction time is negligible compared to the sorting time, and hence
it gives a speedup of 1.08x, almost the same as that by the dynamic
oracle.
In the svd benchmark, the best classiﬁer used only zeros input
feature at the intermediate level and achieved 1.11x speedup com-
pared to a dynamic oracle of 1.16x. It is known that svd is sensitive
to the number of eigenvalues, but this feature is expensive to mea-
sure. The features we included are cheaper and tend to reﬂect that
factor indirectly. For instance, it is likely, although not necessarily,
that a matrix with many 0s has fewer eigenvalues than a matrix with
only a few 0s.
In the poisson2d benchmark, the best classiﬁer employed the
input features zeros at the intermediate level, achieving a 1.09x
speedup compared to a dynamic oracle of 1.12x.
In the helmholtz3d benchmark, the best classiﬁer used the
residue, zeros and deviation input features at the intermediate level
and the range feature at the cheapest level. This benchmark showed
a 1.05x speedup, compared to a dynamic oracle speedup of 1.11x.
Figure 6 shows the distribution of speedups for individual inputs
to each program, sorted such that the largest speedup is on the right.
What is interesting here is the speedups are not uniform. For each
benchmark there exist small sets of inputs with very large speedups,
in some cases up to90x. This shows that the way inputs are chosen
can have a large effect on mean speedup observed. If one had a
real world input distribution that favored these types of inputs, the
overall speedup of this technique would be much larger. In other
words, the relative beneﬁts of using of input adaptation techniques
can vary drastically depending on your input data distribution.
Training Time The training time of the two-level method is
dominated by the time for the autotuner to ﬁnd the 100 landmark
conﬁgurations. The time ranges from 2 hours to 2.7 days for our
benchmarks. An exhaustive method that ﬁnds the best conﬁguration
for every input would take over 200 times longer, given the 20,000
to 30,000 training inputs a benchmark has in our experiments.
Table 1: Mean speedup (slowdown if less than 1) over the performance by the static oracle. The rightmost column reports the percentage of
inputs on which the one-level approach reaches accuracy threshold; the percentages by all other methods are greater than 95%, the satisfaction
threshold.
Benchmark
Name
Dynamic
Oracle
Two-level (w/o
feature extr.
time)
Two-level (w/
feature extr.
time)
One-level (w/o
feature extr.
time)
One-level (w
feature extr.
time)
One-level ac-
curacy
sort1 5.10 ⇥ 2.95 ⇥ 2.91 ⇥ 1.23 ⇥ 1.14 ⇥ -
sort2 6.62 ⇥ 3.05 ⇥ 3.01 ⇥ 0.22 ⇥ 0.22 ⇥ -
clustering1 3.70 ⇥ 2.38 ⇥ 2.37 ⇥ 3.09 ⇥ 0.095 ⇥ 54.56%
clustering2 1.67 ⇥ 1.45 ⇥ 1.18 ⇥ 1.15 ⇥ 0.034 ⇥ 77.03%
binpacking 1.10 ⇥ 1.09 ⇥ 1.08 ⇥ 1.07 ⇥ 0.99 ⇥ 97.83%
svd 1.16 ⇥ 1.11 ⇥ 1.11 ⇥ 1.04 ⇥ 1.03 ⇥ 72.85%
poisson2d 1.12 ⇥ 1.09 ⇥ 1.09 ⇥ 0.89 ⇥ 0.84 ⇥ 64.32%
helmholtz3d 1.11 ⇥ 1.05 ⇥ 1.04 ⇥ 0.85 ⇥ 0.84 ⇥ 96.97%
1 500001
10
20
Inputs
Speedup
(a) sort1
1 500001
50
100
Inputs
Speedup
(b) sort2
1 500001
50
100
Inputs
Speedup
(c) clustering1
1 500001
4
7
Inputs
Speedup
(d) clustering2
1 500001
2
3
4
Inputs
Speedup
(e) binpacking
1 500001
2
3
Inputs
Speedup
(f) svd
1 500001
2
Inputs
Speedup
(g) poisson2d
1 500001
2
5
Inputs
Speedup
(h) helmholtz3d
Figure 6: Distribution of speedups over static oracle for each individual input. For each problem, some individual inputs get much larger
speedups than the mean.
4.3 Theoretical Model Showing Diminishing Returns with
More Landmark Conﬁgurations
In addition to the evaluation of our classiﬁer performance, it is
important to evaluate if our methodology of clustering and using
100 landmark conﬁgurations is sufﬁcient. To help gain insight into
this question we created a theoretical model. In the model, we
consider that in the input search space of a program, some number
of optimal program conﬁgurations dominate different regions of the
input space. Assume inputs uniformly distributed in the space. Let
pi be the size of the regioni, andsi be the speedup the program gets
if the dominate conﬁguration in regioni is used for an input in that
region. Assume that no speedups are obtained if that conﬁguration
is not used for that input.
If we assume thek landmark conﬁgurations are sampled uniform
randomly, the chance for sampling to miss regioni is (1   pi)k. So,
the total expected loss in speedup,L, compared to a perfect method
that has taken all dominate conﬁgurations would be:
L =
X
i
(1   pi)kpisi/
X
i
si
Where pisi represents the cost of missing that region of the search
space in terms of speedup.
Figure 7a shows the value of this function aspi changes. It is
assumed that allsi is the same. One can see that on the extremes
pi =0 and pi =1 there is no loss in speedup, because either the
region is so small a speedup in that region does not matter or the
region is so large that random sampling is likely to ﬁnd it. For each
number of conﬁgs, there exists a worst-case region size where the
expected loss in speedup is maximized. We can ﬁnd this worst-case
region size by solving forpi in dL
dpi
=0 which results in a worst-
case pi = 1
k+1 . Using this worst-case region size, Figure 7b shows
the diminishing returns predicted by our model as more landmark
conﬁgurations are sampled. Figure 8 validates this theoretical model
by running each benchmark with varying numbers of landmark
conﬁgurations. This experiment takes random subsets of the 100
 0  0.2  0.4  0.6  0.8  1
Lost speedup (L)
Size of region (p i)
2 configs
3 configs
4 configs
5 configs
6 configs
7 configs
8 configs
9 configs
30%
(a) Predicted loss in speedup contributed by input space
regions of different sizes.
 10  20  30  40  50  60  70  80  90  100
Fraction of full speedup
Landmarks
70%
(b) Predicted speedup with a worst-case region size with
different numbers of sampled landmark conﬁgurations.
Figure 7: Model predicted speedup compared to sampling all input points as the number of landmarks are increased. Y-axis units are omitted
because the problem-speciﬁc scaling term in the model is unknown.
1 100
1
3
5
Landmarks
Speedup
(a) sort1
1 1001
3
5
Landmarks
Speedup
(b) sort2
1 100
2
3
Landmarks
Speedup
(c) clustering1
1 1000.5
1.5
Landmarks
Speedup
(d) clustering2
1 100
1
1.05
Landmarks
Speedup
(e) binpacking
1 1000.9
1.1
Landmarks
Speedup
(f) svd
1 100
0.9
1.2
Landmarks
Speedup
(g) poisson2d
1 100
0.8
1.0
Landmarks
Speedup
(h) helmholtz3d
Figure 8: Measured speedup over static oracle as the number of landmark conﬁgurations changes, using 1000 random subsets of the 100
landmarks used in other results. Error bars show median, ﬁrst quartiles, third quartiles, min, and max.
landmarks used in other results and measures that speedup over the
static oracle. Real benchmarks show a similar trend of diminishing
returns with more landmarks that is predicted by our model. We
believe that this is strong evidence that using a ﬁxed number of
landmark conﬁgurations (e.g., 10 to 30 for the benchmarks we
tested) sufﬁces in practice, however correct number of landmarks
needed may vary between benchmarks.
5. Related Work
A number of studies have considered program inputs in library con-
structions [13, 19, 29, 32, 36, 41, 47]. They concentrate on some
speciﬁc library functions (e.g., FFT, sorting) while the algorithmic
choices in these studies are limited. Input-centric program optimiza-
tion [42] showed the beneﬁts in enhancing Just-In-Time compila-
tion. Jung and others have considered inputs when selecting the
appropriate data structures to use [30]. Several recent studies have
explored the inﬂuence of program inputs on GPU program optimiza-
tions [33, 39]. Chen and others have evaluated input sensitivity of
iterative optimization [16]. Muralidharan and others have recently
built a programmer-directed autotuning framework named Nitro for
selecting the appropriate code variants at runtime [34].
This current study is unique in focusing on input sensitivity to
complex algorithmic autotuning, which features vast algorithmic
conﬁguration spaces, sensitivity to deep input features, variable
accuracy, and complex relations between inputs and conﬁgurations.
These special challenges prompt the development of the novel
solutions described in this paper.
A number of ofﬂine empirical autotuning frameworks have
been developed for building efﬁcient, portable libraries in speciﬁc
domains. ATLAS [46] utilizes empirical autotuning to produce
a cache-contained matrix multiply, which is then used in larger
matrix computations in BLAS and LAPACK. FFTW [20] uses
empirical autotuning to combine solvers for FFTs. Other autotuning
systems include SPARSITY [28] for sparse matrix computations,
SPIRAL [37] for digital signal processing, and OSKI [45] for sparse
matrix kernels.
The area of iterative compilation contains many projects that
use different machine learning techniques to optimize lower level
compiler optimizations [3, 4, 22, 35]. These projects change both
the order that compiler passes are applied and the types of passes
that are applied. PetaBricks [6] offers a language support to better
leverage the power of autotuning for complex algorithmic choices.
However, none of these projects have systematically explored the
inﬂuence of program inputs beyond data size or dimension.
In the dynamic autotuning space, there have been a number
of systems developed [10, 12, 14, 15, 24, 26, 27, 31, 40] that
focus on creating applications that can monitor and automatically
tune themselves to optimize a particular objective. Many of these
systems employ a control systems based autotuner that operates
on a linear model of the application being tuned. For example,
PowerDial [27] converts static conﬁguration parameters that already
exist in a program into dynamic knobs that can be tuned at runtime,
with the goal of trading QoS guarantees for meeting performance
and power usage goals. The system uses an ofﬂine learning stage
to construct a linear model of the choice conﬁguration space which
can be subsequently tuned using a linear control system. The system
employs the heartbeat framework [25] to provide feedback to the
control system. A similar technique is employed in [26], where a
simpler heuristic-based controller dynamically adjusts the degree of
loop perforation performed on a target application to trade QoS for
performance. Fursin and others have explored run-time evaluation
of program optimizations with phase-sensitivity [21]. The principle
theme of these studies is to react to dynamic changes in the system
behavior rather than proactively adapt algorithm conﬁgurations
based on the characteristics of program inputs.
Additionally, there has been a large amount of work [9, 17,
43, 44] in the dynamic optimization space, where information
available at runtime is used combined with static compilation
techniques to generate higher performing code. Some work [18] has
studied a library-based approach to gain performance portability
across heterogeneous systems on chips while guaranteeing particular
accuracy levels.
6. Conclusions
We have shown a two level solution to the problem of input
sensitivity in algorithmic autotuning. It provides a general means
of automatically determining what algorithmic optimization to use
when different optimization strategies suit different inputs. Through
this work, we are able to extract principles for understanding the
performance and accuracy space of a program across a variety
of inputs, and achieve speedups of up to 3x over using a single
conﬁguration for all inputs, and a 34x speedup over a traditional
one-level method. We also showed that there are fundamental
diminishing returns as more input adaptation is added to a system,
and that a little bit of input adaption goes a long way.
Acknowledgement This material is based upon work supported by DoE
awards DE-SC0005288, DE-SC0008923 and Early Career Award and
the National Science Foundation (NSF) under Grant No. 1464216, No.
1320796 and CAREER Award. Any opinions, ﬁndings, and conclusions or
recommendations expressed in this material are those of the authors and do
not necessarily reﬂect the views of DoE or NSF.
References
[1] Government’s open data.http://www.data.org/.
[2] UCI data sets.http://archive.ics.uci.edu/ml/datasets.
[3] F. Agakov, E. Bonilla, J. Cavazos, B. Franke, G. Fursin, M. F. P.
O’boyle, J. Thomson, M. Toussaint, and C. K. I. Williams. Using
machine learning to focus iterative optimization. InInternational
Symposium on Code Generation and Optimization, pages 295–305,
2006.
[4] L. Almagor, K. D. Cooper, A. Grosul, T. J. Harvey, S. W. Reeves,
D. Subramanian, L. Torczon, and T. Waterman. Finding effective
compilation sequences. InLCTES’04, pages 231–239, 2004.
[5] J. Ansel, Y . L. W. ans Cy Chan, M. Olszewski, A. Edelman, and
S. Amarasinghe. Language and compiler support for auto-tuning
variable-accuracy algorithms. InCGO, Chamonix, France, Apr 2011.
[6] J. Ansel, C. Chan, Y . L. Wong, M. Olszewski, Q. Zhao, A. Edelman, and
S. Amarasinghe. PetaBricks: A language and compiler for algorithmic
choice. InPLDI, Dublin, Ireland, Jun 2009.
[7] J. Ansel, S. Kamil, K. Veeramachaneni, J. Ragan-Kelley, J. Bosboom,
U. O’Reilly, and S. Amarasinghe. Opentuner: An extensible framework
for program autotuning. InProceedings of The 23rd International
Conference on Parallel Architectures and Compilation Techniques,
2014.
[8] J. Ansel, M. Pacula, S. Amarasinghe, and U.-M. O’Reilly. An efﬁcient
evolutionary algorithm for solving bottom up problems. InAnnual
Conference on Genetic and Evolutionary Computation, Dublin, Ireland,
July 2011.
[9] J. Auslander, M. Philipose, C. Chambers, S. J. Eggers, and B. N.
Bershad. Fast, effective dynamic compilation. InPLDI, 1996.
[10] W. Baek and T. Chilimbi. Green: A framework for supporting energy-
conscious programming using controlled approximation. InPLDI, June
2010.
[11] P. Berube, J. Amaral, R. Ho, and R. Silvera. Workload reduction for
multi-input proﬁle-directed optimization. InProceedings of the IEEE /
ACM International Symposium on Code Generation and Optimization,
2009.
[12] V . Bhat, M. Parashar, . Hua Liu, M. Khandekar, N. Kandasamy,
and S. Abdelwahed. Enabling self-managing applications using
model-based online control strategies. InInternational Conference
on Autonomic Computing, Washington, DC, 2006.
[13] J. Bilmes, K. Asanovic, C.-W. Chin, and J. Demmel. Optimizing matrix
multiply using PHiPAC: A portable, high-performance, ANSI C coding
methodology. InProceedings of the ACM International Conference on
Supercomputing, pages 340–347, 1997.
[14] F. Chang and V . Karamcheti. A framework for automatic adaptation of
tunable distributed applications.Cluster Computing, 4, March 2001.
[15] Y . Chen, S. Fang, L. Eeckhout, O. Temam, and C. Wu. Iterative
optimization for the data center. InASPLOS, 2012.
[16] Y . Chen, Y . Huang, L. Eeckhout, G. Fursin, L. Peng, O. Temam,
and C. Wu. Evaluating iterative optimization across 1000 datasets.
In Proceedings of the ACM SIGPLAN conference on Programming
language design and implementation, PLDI’10, pages 448–459, 2010.
[17] P. C. Diniz and M. C. Rinard. Dynamic feedback: an effective technique
for adaptive computing. InPLDI, New York, NY , 1997.
[18] S. Fang, Z. Du, Y . Fang, Y . Huang, Y . Chen, L. Eeckhout, O. Temam,
H. Li, Y . Chen, and C. Wu. Performance portability across heteroge-
neous socs using a generalized library-based approach.ACM Transac-
tions on Architecture and Code Optimization, 11, 2014.
[19] M. Frigo and S. G. Johnson. The design and implementation of FFTW3.
Proceedings of the IEEE, 93(2):216–231, 2005.
[20] M. Frigo and S. G. Johnson. The design and implementation of FFTW3.
IEEE, 93(2), February 2005. Invited paper, special issue on “Program
Generation, Optimization, and Platform Adaptation”.
[21] G. Fursin, A. Cohen, M. O’Boyle, and O. Temam. Quick and practical
run-time evaluation of multiple program optimizations.Transactions on
High-Performance Embedded Architectures and Compilers, 4050:34–
53, 2007.
[22] G. Fursin, C. Miranda, O. Temam, M. Namolaru, E. Yom-Tov, A. Zaks,
B. Mendelson, E. Bonilla, J. Thomson, H. Leather, C. Williams,
M. O’Boyle, P. Barnard, E. Ashton, E. Courtois, and F. Bodin. MILE-
POST GCC: machine learning based research compiler. InProceedings
of the GCC Developers’ Summit, Jul 2008.
[23] T. Hastie, R. Tibshirani, and J. Friedman.The elements of statistical
learning. Springer, 2001.
[24] P. Hawkins, A. Aiken, K. Fisher, M. Rinard, and M. Sagiv. Data
representation synthesis. InProceedings of ACM SIGPLAN Conference
on Programming Languages Design and Implementation, 2012.
[25] H. Hoffmann, J. Eastep, M. D. Santambrogio, J. E. Miller, and A. Agar-
wal. Application heartbeats: a generic interface for specifying program
performance and goals in autonomous computing environments. In
ICAC, New York, NY , 2010.
[26] H. Hoffmann, S. Misailovic, S. Sidiroglou, A. Agarwal, and M. Ri-
nard. Using code perforation to improve performance, reduce energy
consumption, and respond to failures. Technical Report MIT-CSAIL-
TR-2209-042, Massachusetts Institute of Technology, Sep 2009.
[27] H. Hoffmann, S. Sidiroglou, M. Carbin, S. Misailovic, A. Agarwal, and
M. Rinard. Power-aware computing with dynamic knobs. InASPLOS,
2011.
[28] E. Im and K. Yelick. Optimizing sparse matrix computations for register
reuse in SPARSITY. InInternational Conference on Computational
Science, 2001.
[29] E.-J. Im, K. Yelick, and R. Vuduc. Sparsity: Optimization framework
for sparse matrix kernels. Int. J. High Perform. Comput. Appl.,
18(1):135–158, 2004.
[30] C. Jung, S. Rus, B. P. Railing, N. Clark, and S. Pande. Brainy: effective
selection of data structures. InProceedings of the 32nd ACM SIGPLAN
conference on Programming language design and implementation,
PLDI ’11, pages 86–97, New York, NY , USA, 2011. ACM.
[31] G. Karsai, A. Ledeczi, J. Sztipanovits, G. Peceli, G. Simon, and
T. Kovacshazy. An approach to self-adaptive software based on
supervisory control. In International Workshop in Self-adaptive
software, 2001.
[32] X. Li, M. J. Garzarán, and D. Padua. Optimizing sorting with genetic
algorithms. InCGO, 2005.
[33] Y . Liu, E. Z. Zhang, and X. Shen. A cross-input adaptive framework for
gpu programs optimization. InProceedings of International Parallel
and Distribute Processing Symposium (IPDPS), pages 1–10, 2009.
[34] S. Muralidharan, M. Shantharam, M. Hall, M. Garland, and B. Catan-
zaro. Nitro: A framework for adaptive code variant tuning. InParallel
and Distributed Processing Symposium, 2014 IEEE 28th International,
pages 501–512. IEEE, 2014.
[35] E. Park, L.-N. Pouche, J. Cavazos, A. Cohen, and P. Sadayappan.
Predictive modeling in a polyhedral optimization space. InIEEE/ACM
International Symposium on Code Generation and Optimization, pages
119 –129, April 2011.
[36] M. Puschel, J. Moura, J. Johnson, D. Padua, M. Veloso, B. Singer,
J. Xiong, F. Franchetti, A. Gacic, Y . V oronenko, K. Chen, R. Johnson,
and N. Rizzolo. SPIRAL: code generation for DSP transforms.
Proceedings of the IEEE, 93(2):232–275, 2005.
[37] M. Püschel, J. M. F. Moura, B. Singer, J. Xiong, J. R. Johnson, D. A.
Padua, M. M. Veloso, and R. W. Johnson. Spiral: A generator for
platform-adapted libraries of signal processing alogorithms.IJHPCA,
18(1), 2004.
[38] J. Quinlan. Induction of decision trees.Machine learning, 1(1):81–106,
1986.
[39] M. Samadi, A. Hormati, M. Mehrara, J. Lee, and S. Mahlke. Adaptive
input-aware compilation for graphics engines. InProceedings of ACM
SIGPLAN 2012 Conference on Programming Language Design and
Implementation, 2012.
[40] C. Tapus, I.-H. Chung, and J. K. Hollingsworth. Active harmony:
Towards automated performance tuning. InIn Proceedings from the
Conference on High Performance Networking and Computing, pages
1–11, 2003.
[41] N. Thomas, G. Tanase, O. Tkachyshyn, J. Perdue, N. M. Amato, and
L. Rauchwerger. A framework for adaptive algorithm selection in
STAPL. In Proceedings of the Tenth ACM SIGPLAN Symposium
on Principles and Practice of Parallel Programming, pages 277–288,
2005.
[42] K. Tian, Y . Jiang, E. Zhang, and X. Shen. An input-centric paradigm for
program dynamic optimizations. Inthe Conference on Object-Oriented
Programming, Systems, Languages, and Applications (OOPSLA), 2010.
[43] M. V oss and R. Eigenmann. Adapt: Automated de-coupled adaptive
program transformation. In International Conference on Parallel
Processing, 2000.
[44] M. V oss and R. Eigenmann. High-level adaptive program optimization
with adapt.ACM SIGPLAN Notices, 36(7), 2001.
[45] R. Vuduc, J. W. Demmel, and K. A. Yelick. OSKI: A library of auto-
matically tuned sparse matrix kernels. InScientiﬁc Discovery through
Advanced Computing Conference, Journal of Physics: Conference Se-
ries, San Francisco, CA, June 2005.
[46] R. C. Whaley and J. J. Dongarra. Automatically tuned linear algebra
software. InSupercomputing, Washington, DC, 1998.
[47] R. C. Whaley, A. Petitet, and J. Dongarra. Automated empirical
optimizations of software and the ATLAS project.Parallel Computing,
27(1-2):3–35, 2001.

Compiler Autotuning using Machine Learning: A State-of-the-art Review
Google Brain28thOct, 2019
Amir AshouriUniversity of Toronto Canada
Background2 ¤Postdoctoral Fellow(2017-): University of Toronto, CanadanAdvisor: Tarek Abdelrahman¤Ph.D.(‘2016): Polytechnic University of MilannPhD thesis: “Compiler Autotuning using Machine Learning Techniques“nPhD Advisors: nCristina SilvanonGianluca PalermonJohn Cavazos (University of Delaware, USA)
Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
1. OrganizationA Survey on Compiler Autotuning using Machine Learning 
1.ACM Computing Surveys (ACM CSUR) –2019[1]
2.arXivpreprint version2:¤Will be updated quarterly¤Send us your new autotuningpapers to be added
3
[2] Ashouri et al. 2018 –A Survey on Compiler Autotuning using Machine Learning:(http://arxiv.org/abs/1801.04405)[1] Ashouri et al. 2019 –A Survey on Compiler Autotuning using Machine Learning:
High-level Outline41.Introduction2.Compiler Optimization Concept3.Characterization Techniques4.Machine Learning Models5.Prediction Types6.Optimization Space Exploration7.Target Domain8.Influential PapersAmir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
High-level Outline51.Introduction2.Compiler Optimization Concept3.Characterization Techniques4.Machine Learning Models5.Prediction Types6.Optimization Space Exploration7.Target Domain8.Influential PapersAmir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
ProebstingLaw16●Proebsting’slaw: “Compilers Advances Double Computing Power Every 18 Years” ○“Even if we assume that the beginning of useful compiler optimization research began in the mid 1960's, the uniform performance improvement on integer intensive codes due to compiler optimization is still 3.6% per year, this lies in stark contrast to the 60% per year performance we CAN expect from hardware due to Moore's law”
●Some initial inferences to compensate the offset:○A need for autotuning○Adapting new machine learning techniques
[1] Probsting’sLaw: http://proebsting.cs.arizona.edu/law.htmlAmir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
1. Introduction/Problem descriptionChoosing the right optimizations (1/2)12¨Compilation =~ Translation (Mike O’Boyle, 2014)
*2mm application from Polybench[Pouchet2012]Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
1. Introduction/Problem descriptionChoosing the right optimizations (2/2)13¨Compilation =Translation + Optimization
Opt.1
Opt.2
Opt.3
Opt.4
Opt.5
Opt.6
Opt.7
Opt.8
Opt.9
Opt.10
Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
High-level Outline141.Introduction2.Compiler Optimization Concept3.Characterization Techniques4.Machine Learning Models5.Prediction Types6.Optimization Space Exploration7.Target Domain8.Influential PapersAmir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
1. Introduction/Problem descriptionCompiler Optimization Passes15¨There are several pre-built compiler flags:○Standard flags, i.e. -O1, -O2, -O3, -Os○Many different optimization passes■~ 160 in LLVM■~ 250 in GCC■~ 75 in ICC¨Identifying the right set of compiler passes (flags) will have substantial benefits on:○Performance○Code-size ○Power metrics¨The design space is large and choosing the right ordering matters:¨Two Major Problems:1.Selecting the right set of compiler passes2.Phase-orderingof compiler passes
Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
Compiler Automatic Tuning16¤Autotuning addresses:nAutomatic code generationnAutomatic code optimizationnEnabling many different scenariosnMinimizing/Maximizing certain objective functions in the optimization spacenHistorically Autotuning was only done in the back-end:nDownsides:nRequires exclusive knowledge of compiler constructionnHigher overhead to modify and tune the back-end internally nState-of-the art autotuningresearch leverages machine learning techniuqes
Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
A Sample Autotuning Framework17
Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
Identifying The Best Compiler PassesThe Selection Problem18¨The SelectionProblem:¤Severalcompileroptimizationspassesformanoptimizationsequence.Wedisregardtheorderingoftheseoptimizationsandfocusonwhethertoapplyanoptimization[Agakovetal.2006].nOptimizationspacePn:nn=#ofoptimizations,p=#ofavailableoptimizationlevels¨Example: (let n=5 optimizations, p=3 optimization levels)¤-loop-unrolling –tiling –dce–loop-fusion –mem2reg¤-loop-unrolling(3) –tiling(2,2) –loop-fusion¤-loop-unrolling(1) ¤-dce–loop-fusion>> Different optimizations to choose:         32×23=72Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
State-of-the-art:Iterative Compilation19¨Several recompilationof the a source-code using different optimization flags to choose the best found version1.¤Advantage:nNormally brings good results in a long-runnNo machine intelligence needed (on its pure version)¤Downside:nHigh overhead task and time-consumingnMust be repeated for a new dataset, application and architecturenNo knowledge transfer involves¤Can we do better ???nEmploying machine learning based models with Iterative Compilation
[1] Bodinet al. 1998. Iterative compilation in a non-linear optimisationspace. Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
The Phase-ordering Problem [Leverettet al. 1979]20¨The Phase-ordering Problem:¤Thereisnoidealorderingofphases.OptimizationpassAtransformstheprograminwaysthatobliteratesomeoptimizationsthatotherwisecouldhavebeenperformedbypassB,whichfollowsit.nOptimization space: ¨Example: (let n=5optimizations, m=5 optimization-vector max-length)¤{-loop-unrolling –tiling –dce–loop-fusion –mem2reg}¤{–dce-loop-unrolling –loop-fusion}¤{-loop-fusion –mem2reg}¤{-dce–mem2reg –dce–mem2reg}>> Different optimizations to choose:   50+51+52+53+54+55= 3906Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
Tackling the Phase-ordering Problem21¨Major issue:¤Optimization space is significantly big that you cannot reproduce the previous approaches used for the selection problem:nLet n=20 optimizationnThe selection space ~ 107nThe phase-ordering space >> at least 1018( no repetition case )¨Possible approaches:¤Grouping the optimizations in sub-sequencesnRandom clusteringnManual clusteringnAutomatic clustering¤Predicting the performance speedup of an optimization vector rather than the predicting the compiler optimization sequence:nSpeedup predictorvs. Sequence predictor Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
High-level Outline221.Introduction2.Compiler Optimization Concept3.Characterization Techniques4.Machine Learning Models5.Prediction Types6.Optimization Space Exploration7.Target Domain8.Influential PapersAmir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
3-Application Characterization23¨Asingle/multiplePass(ess)overtheapplicationisneededtocollectitsrepresentativefeatures¨Thefeatures,alsoknownasfeaturevector,willbeusedastherepresentationofanapplication¨Conformstothesetwoconditions:¤Itmustberepresentativeenoughofitsapplication¤Differentapplicationsmustnothavesamefeaturevectors
Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
3-Cont’d Examples24Dynamic(MICA[1])¤ProsnFine-grain characterization, higher accuracy, input sensitive¤Consn(May be) Architecture depended, slower to collect featuresStatic analysis(MilepostEU-Project [2])¤ProsnFaster to fetch, independent of the architecture¤ConsnTransparent to input datasets, lower accuracy [2,3]Hybrid characterization(COBAYN [4])[1]Hosteet al, “Microarchitecture independent workload characterization”. IEEE Micro, 27(3):63–72, 2007[2] Fursin, Grigori, et al. "Milepost gcc: Machine learning enabled self-tuning compiler."International Journal of Parallel Programming39.3 (2011): 296-327)[3] Cavazos, et al. "Rapidly selecting good compiler optimizations using performance counters(CGO), 2007”[4] Ashouri, et al. "Cobayn: Compiler autotuningframework using bayesiannetworks., ACM TACO, 2016"Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
3-Cont’d Examples (MICA)[1]MICA (99 features –6 categories)25
Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
[1] Github: https://github.com/boegel/MICA/
High-level Outline261.Introduction2.Compiler Optimization Concept3.Characterization Techniques4.Machine Learning Models5.Prediction Types6.Optimization Space Exploration7.Target Domain8.Influential PapersAmir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
4-Machine Learning Models27¨Itinvestigatesthestudyandconstructionoftechniques/algorithmsthatareabletolearncertainattributesfromdata.ThreeBroadcategories:¤SupervisedLearning¤UnsupervisedLearning¤ReinforcementLearningMappingtocompilerautotuning:¤Thereisnowaytoidentifyananalyticalmodeltofitafunctionwhichoutputsthebestoptimizationvector(ō)givenanapplicationwithacharacterizationvector(α)èThus,researchshavebeentrayingtoapproximatethisfunctionusingvariousMLmethodsandapproaches.
Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
a1
o1
 o2
a2
o3

COBAYN’s Inference[1]34
Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
[1] Ashouri, et al. "Cobayn: Compiler autotuningframework using bayesiannetworks."ACM Transactions on Architecture and Code Optimization (TACO) 13.2 (2016): 21.
https://github.com/amirjamez/COBAYN
High-level Outline351.Introduction2.Compiler Optimization Concept3.Characterization Techniques4.Machine Learning Models5.Prediction Types6.Optimization Space Exploration7.Target Domain8.Influential PapersAmir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
5-Prediction Types36¨Compiler autotuningseeks to provide several prediction output classes:1.Identifying the right set of compiler sequences (pass)2.Predicting the speedup of a compiler sequence1.Intermediate-speedup (tournament) prediction2.Full-sequence speedup prediction3.Clustering (grouping) the effective compiler passes4.Identifying the right set of application features to use
Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
5. Cont’d Speedup Prediction Models37¨Intermediate speedup prediction[Kulkarni& Cavazos 2012; Ashouri et al. 2016]¤Easier/more intuitive to build such model¤Needs multiple characterizations (at each state of the code)¤Higher error rate due to the recurrent use of the prediction model 
¨Full-sequence prediction[no literature as of 2017]¤Harder to build as a model¤Needs an encoding scheme and an exploration policy to a generate fixed feature-vector length¤At bestneeds only one round of feature selectionAmir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
IntermediateSpeedup Prediction Model[Kulkarni& Cavazos 2012; Ashouri et al. 2016]38
Intermediate Speedup Prediction Model
AA
Prog. FeaturesOpt. Passes
Prog. FeaturesOpt. Passes
B
AB
Prog. FeaturesOpt. Passes
B
STOPNo
YesFinal Opt={A, B, B}
Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
Full-sequenceSpeedup Prediction Model[Ashouri et al., MiCOMP , ACM TACO 2017]39
Prog. Features at a BaselineOpt. Passes
ABBC
Final Opt={A, B, B, C}
Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
Clustering Method -ExampleMiCOMP [1]40¨LLVM opt –O3 has:¤~157 optimization passes (~60 unique passes)¨We represent G=(V,E) where¤Vis the set of opts (nodes)¤Eis the set of dependence between the opts (edges)nE(vx,vy)=1 if that pair of optimizations consecutively appears  in -O3nAn edge can be annotated with the number of times that each pair of optimizations consecutively appears in the sequencen++E(vx,vy);nThis will result in an Optimization Dependency Graphfor –O3.[1] Ashouri, Amir H., et al. "Micomp: Mitigating the compiler phase-ordering problem usingoptimization sub-sequences and machine learning.”, ACM TACO, 2017Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
MiCOMP Clustering (2/4)Visualizing -O3’s Optimization Dependency Graph41¨
Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
MiCOMP Clustering (4/4)43
Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
MiCOMP’sResult44
Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
High-level Outline451.Introduction2.Compiler Optimization Concept3.Characterization Techniques4.Machine Learning Models5.Prediction Types6.Optimization Space Exploration7.Target Domain8.Influential PapersAmir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
6 –Optimization Space Exploration46¨Traversing the large compiler optimization space requires an exploration strategy:¤Adaptive CompilationnProfile-guided optimization [1]¤Non-iterative CompilationnPresenting a global optimization approach [2]¤Iterative CompilationnApproximation procedures, iteratively applied [3][1] KD Cooper, et al. 2005. ACME: adaptive compilation made efficient. (2005)[2] Deborah L. Whitfeldand Mary Lou Soa. An approach for exploring code improving transformations. TOPLAS (1997)[3] Bodin, François, et al. "Iterative compilation in a non-linear optimisationspace.” (1998)Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
High-level Outline501.Introduction2.Compiler Optimization Concept3.Characterization Techniques4.Machine Learning Models5.Prediction Types6.Optimization Space Exploration7.Target Domain8.Influential PapersAmir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
8-Influential Papers51¨Introducing learning methods ¤[Leverettet al. 1979; Vegdahl1982; Whitfield et al. 1990, 1997]¨Genetic algorithm ¤[cooper et al. 1999, 2002;Triantafylliset al. 2003; Knijnenburget al. 2003]¨Iterative compilation¤[Bodin et al. 1998; Stephenson et al. 2005; Agakovet al. 2006, Ashouri et al. 2014, 2016], many other work¨Dynamic & hybrid features¤[Cavazos et al. 2007; Park et al. 2012; Ashouri et al. 2016]¨Optimization groups and Bayesian learners¤[Ashouri et al. 2014, 2016]¨Phase-ordering ¤[Kulkarniet al. 2004; Kulkarniand Cavazos 2012, Ashouri et al. (2017))]
Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019
Discussion52¨Heterogeneous Autotuning¤GPUs, DSPs, FPGAs ¨Faster, more accurate models are needed:¤Supercomputers¤Deep learning modelsnLSTM1,2Model¤Accurate application characterization techniques¨Deep Learning Compilers¤TVM[3]¤TensorFlow’s XLA[4][1] Cummins, Chris, et al. "Synthesizing benchmarks for predictive modeling”. CGO, 2017.[2]Grewe, Wang, and O'Boyle. "Portable mapping of data parallel programs to openclfor heterogeneous systems.” CGO, 2013. 
Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019[3]Chen, Tianqi, et al. "TVM: end-to-end optimization stack for deep learning."arXivpreprint arXiv:1802.04799(2018).[4]Leary, Chris, and Todd Wang. "XLA: TensorFlow, compiled."TensorFlow Dev Summit(2017).
Thank you!Questions & Comments53¨Contact:¤Amir H. AshourinPostdoc @ University of Toronto, Canadanhttp://www.eecg.toronto.edu/~aashouri/naashouri@ece.utoronto.ca
Amir H. Ashouri –Compiler Autotuning Using ML -State-of-the-art Review –Google Brain -Oct 2019

Composable and Modular Code Generation in MLIR
A Structured and Retargetable Approach to Tensor Compiler Construction
NICOLAS VASILACHE, OLEKSANDR ZINENKO, AART J.C. BIK, MAHESH RAVISHANKAR,
THOMAS RAOUX, ALEXANDER BELYAEV, MATTHIAS SPRINGER, TOBIAS GYSI, DIEGO
CABALLERO, STEPHAN HERHUT, STELLA LAURENZO, and ALBERT COHEN, Google
Despite significant investment in software infrastructure, machine learning systems, runtimes and compilers
do not compose properly. We propose a new design aiming at providing unprecedented degrees of modularity,
composability and genericity. This paper discusses a structured approach to the construction of domain-
specific code generators for tensor compilers, with the stated goal of improving the productivity of both
compiler engineers and end-users. The approach leverages the natural structure of tensor algebra. It has
been the main driver for the design of progressive lowering paths in MLIR. The proposed abstractions and
transformations span data structures and control flow with both functional (SSA form) and imperative (side-
effecting) semantics. We discuss the implications of this infrastructure on compiler construction and present
preliminary experimental results.
1 INTRODUCTION
Despite significant investment in software infrastructure, Machine Learning (ML) systems are
still stuck in a rut [ 3]. This can largely be traced to a recurring issue: the number and sheer
complexity of the steps involved in between programmers’ intent and efficient execution on ever
more powerful hardware. Software stacks include technologies as diverse as ML frameworks for
distributed training or inference, for server or mobile platforms, as well as user-facing languages
for productivity, high-performance communication and compute libraries, and domain-specific
compiler implementations and autotuners.
Unfortunately, these stacks are built in silos: (1) vertical silos follow to top-down, framework-
centered considerations: the need to capture the ever-growing user base of ML practitioners
encourages framework-specific designs, resulting in duplication of effort; while (2) horizontal
silos follow bottom-up, technology-driven considerations: the lack of performance portability over
hardware platforms push for hardware-specific implementations of a fixed set of computational
operations and a fixed programming interface.
None of this is built in a modular fashion; nothing composes.
Everyone rebuilds similar functionality resulting in incremental improvements at ever-increasing
engineering costs. End users do not get access to programmable and portable high-performance
building blocks; instead, they have to rely on limited expressiveness interfaces to the underlying
implementations. Investments on common infrastructure — such as the automation of target-
specific code generation — happen very late, making their adoption painful for software engineers
with years of experience in a more restricted environment.
Focusing on the performance portability of computational operations in ML frameworks, we
explore a better way consisting of multiple cycles combining top-down and bottom-up thinking: (1)
top-down thinking is concerned with making primitives available to the programmer thatgradually
decompose into smaller building blocks with unsurprising, good performance; while (2) bottom-up
thinking is concerned with the creation of primitive building blocks that are well-suited to each
hardware architecture and thengradually compose them into the larger building blocks that connect
to top-down thinking. We propose a first attempt at embodying such alternating cycles based
on compiler technology. Iterating between top-down and bottom-up thinking opens up multiple
Google, 1600 Amphitheatre Pkwy, Mountain View, CA 94043, USA.
arXiv:2202.03293v1  [cs.PL]  7 Feb 2022
2 Vasilache et al.
opportunities for codesign at the interface between the compiler and programming environments,
runtime abstractions, performance tuners and hardware.
From a compiler practitioner’s point of view, it is interesting to realize that virtually all modern
compiler solutions bottom-out on the LLVM compiler to produce optimized binaries. While, LLVM
is a common substrate, it is also a very low-level one. Users (and ML frameworks) have to make an
exclusive choice among the kind of abstractions introduced by either XLA [50], TVM [9], Glow [45],
Halide [41], TACO [32], polyhedral compilers [ 55] or other domain-specific compilers that all
eventually produce some flavor of LLVM IR. Each one of these compilers comes with its own
implementation and does not mix with others: cross-pollination is often reduced to understanding
what compiler A did well and retrofit it as new transformations in compiler B.
Stepping back, one problem comes from the lack of an established infrastructure to support
such domain-specific compilers. Another problem comes from jumping the abstraction gap too
quickly between user-facing abstractions and levels of compiler Intermediate Representation (IR)
on which analyses and transformations occur. This results in (1) losing information that is available
at higher levels of IR and (2) exacerbating phase-ordering issues due to the need to reconstruct high-
level semantical information form low-level IR. This hurts modularity, leaving software engineers
without well-understood design patterns for reusable code generator components.
MLIR is a new compiler infrastructure that drastically reduces the entry cost to define and
introduce new abstraction levels for building domain-specific IRs [34]. It is part of the LLVM project
and follows decades of established practices in production compiler construction. As such, MLIR is
an ideal solution to the missing infrastructure problem. Yet, even with the availability of MLIR, the
second problem remains: missing intermediate abstractions and a progressive refinement strategy
for the compilation of tensor computations.
This work presents ongoing code generation efforts to build composable abstractions for high-
performance code generation in MLIR:
(1) Composable transformations: generic computations are decomposed hierarchically into
smaller tiles, exploiting their algebraic properties and structure. They may also be fused,
and lowered progressively to loops over retargetable vector primitives. Computations can
be instantiated over tensor values (immutable) as well as in-memory buffers (side-effecting).
Such abstractions exist in multiple storage variants such as dense, sparse formats, or quantized
representations. An in-place bufferization pass provides a memory-efficient materialization
of programs in tensor form, including transformed programs (tiled, fused, etc.).
(2) Programmable operations: on the front-end side, a declarative python-based DSL allows
to define operations and their properties, including new operations called “custom ops”;
the latter immediately become first-class IR citizens on which all existing transformations
and rewrites are available. Throughout the transformation and lowering process, we offer a
simple ABI definition and interoperability with C, C++ and Python.
As we will see in the following, the abstractions and transformations we propose offer immediate
as well as long-term benefits:
(1) High-level, domain-specific information is not discarded prematurely. Preserving the struc-
ture of computational operations allows to transform the IR while avoiding the performance
cliffs of numerical libraries (e.g., when a fused operation is lacking a high-performance imple-
mentation). In particular, transformations preserve the ability to lower operations to hardware
instructions implementing coarse-grained vector operations, or to numerical libraries — such
as Eigen [22]. The latter acts as a safety net upon which compiler transformations may add
value, opening new avenues for compiler-library co-design.
Composable and Modular Code Generation in MLIR 3
(2) Tiled operations target subsets of tensor values or memory buffers. This central notion
leverages the natural structure in tensor operations, while remaining completely generic in
the actual representation of tensors (values or side-effects, vectors or scalars, dense or sparse,
etc.) and in the actual decompositions applied to the computations (e.g., different forms of
tiling). This enhances composability while allowing transformations to apply to individual or
groups of operations as opposed to loops or control-flow graphs. It also eases the expression
of complex transformations and lowering sequences, which in turn facilitates autotuning.
(3) The IR remains executable at any intermediate transformation and lowering step. This greatly
simplifies debugging, testing and performance evaluation, and somewhat blurs the lines
between what is traditionally considered the programmer’s and the compiler’s responsibilities.
In particular, C and C++ ABI compliance at function boundary eases the integration of domain-
specific frameworks by relying on common infrastructure. It is also an ingredient in the
matching of abstract operations into calls to numerical libraries, when available.
The techniques and abstractions described in this paper are used in production at Google,
including XLA-based compiler flows [ 50] for CPU/GPU, and IREE [ 14] for CPU/GPU with an
emphasis on mobile and edge computing. In particular we have successfully replaced uses of the
Eigen [22] library in Google-wide production cases.
2 OVERVIEW OF THE CODE GENERATION FLOW
We leverage the MLIR compiler infrastructure. MLIR drastically reduces the entry cost to define,
compose and reuse abstractions for the construction of domain-specific compilers. It offers a
comprehensive collection of solutions to compiler construction challenges by: (1) standardizing
Static Single Assignment (SSA) form representations and data structures, (2) unifying compiler
analyses and transformations across semantic domains through generic programming concepts
such as operation traits and interfaces, (3) providing a declarative system for defining operations
with nested regions, and domain-specific type systems, and (4) providing a wide range of services
including documentation, parsing/printing logic, location tracking, multithreaded compilation,
pass management, etc.
MLIR is designed around the principles of parsimony, progressivity and traceability [34]. The
code generation approach presented in this paper has largely contributed to the establishment of
these principles and actively leverages them. The internal representation in MLIR is fully extensible,
allowing for custom user-defined operations (instructions), attributes and types. IR components
that are expected to work together are grouped into dialects, which can be seen as the intermediate
representation analog of dynamic libraries. Unlike many earlier compilation flows that have multi-
level yet all-or-nothing internal representation, MLIR affords and encourages the mix of different
dialects in a single unit of compilation at any point in the compilation flow. For example, a high-level
tensor product operation may co-exist with low-level hardware instructions on vector elements
in the same function. This provides a great level of modularity, composition and optionality that
makes it possible to use the right abstractions for solving a particular problem, instead of having to
solve all problems in a unique representation .
2.1 Bird’s Eye View and Motivation of Structured and Retargetable Code Generation
Code generation approaches for numerical computing have traditionally focused on optimizing
the performance of loop nests. Associated analyses focus on scalar elements as the body of a loop
nest typically computes a single element. Such analyses must consider memory dependences and
aliasing. These approaches have been deeply researched in the past [1] and have reached a high
4 Vasilache et al.
level of maturity. They are well-suited when starting from an input language like C or Fortran
where the problem is already specified in terms of loops over data residing in pre-allocated memory.
When focusing on a specific domain (e.g. the ML space), we have the luxury of programs defined
at a much higher level of abstraction than loops. This opens up the opportunity to revisit classical
loop optimizations like fusion, tiling or vectorization without the need for complicated analysis
and heuristics. Advantages include reduced complexity and maintenance cost while also scaling
naturally to extensions like sparse tensors, that are even more difficult to analyze at the loop level.
It makes it possible to avoid raising information from lower level representations by means
of static analysis, where feasible, and performing optimizations at the highest possible level of
abstraction. We refer to this approach asstructured code generation since the compiler primarily
leverages structural information readily available in the source code. Figure 1 shows a coarse-grained
summary structure of the steps and levels of abstraction involved.
The starting point (Structured IR) is composed of tensor algebra operations, organized as a
functional program over dense and sparse tensors.
Fig. 1. Bird’s eye view of structured and retargetable code generation.
From this level we move to a tiled structured level, which introduces loops by tiling the operations.
Multiple, gradual tiling steps are possible, and do not necessarily result in loops around scalars.
Instead, tiling produces loops around structured operations similar to the original ones but on
smaller tensors. We also perform fusion of tensor operations at this level. The final granularity of
operations is chosen to make their hardware mapping efficient. A prototypical example is to tile a
matrix multiplication to model the cache hierarchy and then lower the smaller matrix multiplication
directly to a superoptimized microkernel in assembly language.
In the next step, we map computations on resulting small tensors to a (retargetable) vector
abstraction. This mapping exploits high-level knowledge about the operations that we have carefully
preserved. In particular, it is not required to analyze the loop control flow enclosing the finer-
grained tensor operations. This step might also include enabling transformations like padding for
efficient cache access free of cache-line splitting and vectorization.
What makes structured code generation highly composable and reusable is that tiling and fusion
transformations are both fully generic in the operations and data types they operate upon. These
transformations only assume a generic, monotonic (from the point of set inclusion), structural
decomposition pattern associated with computations and composite data. Both dense and sparse
tensor algebra exhibit such blockwise decomposition patterns, and the code generation abstractions
and infrastructure generically applies to both. This is also true of future computations and data
structures following the same pattern.
Up until now, computations took place on immutable tensor values. We lower this to a represen-
tation on side-effecting buffers in the next step. This results in a representation with nested loops
on vectors and side-effects. More optimizations on loops and memory accesses happen at this level.
Composable and Modular Code Generation in MLIR 5
%value_definition= "dialect.operation"(%value_use) {attribute_name = #attr_kind<"value">} ({
// Regions contain blocks. 
^block(%block_argument: !argument_type):
"dialect.further_operation"()[^successor] : () -> ()
^successor: // more operations below
}) : (!operand_type) -> !result_type<"may_be_parameterized">
Fig. 2. MLIR concepts in the generic format. MLIR has an open set of attributes, operations and types.
Operations may recursively contain regions of blocks with further operations.
In the final step, we may translate the representation directly to the llvm dialect of MLIR for
sequential execution on CPU, or offload a GPU kernel, or split up loops into async blocks for a task
parallel runtime, etc.
This flow composes with existing affine analyses and loop optimizations as implemented in
MLIR, and that have been largely explored in the literature. In fact, the packing and loop peeling
transformations in our flow leverage and helped generalize the MLIR affine machinery.
While this flow is but a first stake in the ground, it already demonstrates how to achieve a
modular and composable system. It is also built with optionality in mind: for some operations,
skipping levels or even a fully divergent flow are viable options. This is made possible by our work
establishing the progressive lowering principle : every step is materialized in the IR and very little
load-bearing logic is abstracted away from the user (e.g. in the form of complex C++ logic deep
inside the compiler implementation of analyses and heuristics). We expect more operations and
data types will be designed and implemented that do not fit the current code generation stack of
abstractions. While this allows to extend the structured and progressive lowering approach to other
application domains, we acknowledge that not all computations in ML or HPC will necessarily be
covered by the current approach, alone. This is where composability, modularity and optionality
kick in: not all problems need to be solved in the same abstraction, rather we should use the best
abstraction for each different classes of problems .
2.2 Short Introduction to MLIR
The MLIR infrastructure builds on the success of LLVM IR while providing unprecedented exten-
sibility. MLIR has an open, easily extensible set of instructions, called operations that typically
represent the dynamic semantics of the program. Operations can represent anything from hardware
instructions, or even hardware itself, to building blocks for machine learning models such as layers
or blocks thereof. They define and use values, which represent units of immutable data in SSA
form. The compile-time knowledge about values is captured in types, and the knowledge about
operations is captured in attributes. Attribute and type systems are similarly open and extensible.
IR objects can be logically grouped together in libraries, calleddialects. The MLIR IR has a recursive
structure where operations may have additional regions containing a graph of (basic) blocks, which
in turn contain further operations. Figure 2 illustrates key MLIR concepts.
In addition to common components such as the compiler pass infrastructure, MLIR provides tools
to manage its extensibility, many of which evolved or were specifically designed to support the
code generation flow presented in this document. In particular, MLIR features attribute, operation
and type interfaces similar to object-oriented programming languages allowing one to work with
abstract properties rather than (fixed) lists of supported concepts. Interfaces can be implemented
separately from operations, and mixed in using MLIR’s registration mechanism, thus fully separating
IR concepts from transformations.
6 Vasilache et al.
2.3 Dialects Relevant to Code Generation
The domain-specific abstractions we design and implement comprise representations for the
following dialects, listed in increasing level of abstraction. Following our modularity and optionality
design principles, any of these dialects can be mixed with others or simply bypassed if it does not
provide a useful abstraction for a particular case.
2.3.1 vector Dialect. This dialect provides a fixed-rank n-D vector type, e.g., vector<4x3x8xf32>,
as well as operations that form an intuitive and retargetable vector programming model that con-
ceptually extends the traditional 1-D vector instructions to arbitrary rank. Such operations can
decompose progressively into lower-rank variants of themselves. They further lower to LLVM
vector instructions (e.g. shufflevector) when the backend heuristics are robust enough to gener-
ate near-peak assembly or bypass that level and directly target hardware-specific intrinsics (e.g.
gpu.subgroup_mma_compute_matrix 2-D vector instructions or amx.tile_mulf 2-D tile instructions).
2.3.2 gpu Dialect. The gpu dialect defines the retargetable GPU programming model . It features
abstractions common to SIMT platforms, such as the host/device code separation, workitem/group
(thread/block) execution model, communication and synchronization primitives, etc. This dialect
can be produced from the vector dialect and can itself be lowered to platform-specific dialects such
as nvvm, rocdl or spirv. It is only listed to illustrate the overall retargetability of our approach and
is not discussed further in the paper.
2.3.3 memref Dialect. The memref dialect introduces the memref data type, which is the main repre-
sentation for n-D memory buffers in MLIR and the entry point to the side-effecting memory-based
operations, and the operations to manage buffer allocation, aliasing (memref views) and access. Un-
like traditional pointers, memrefs are multi-dimensional buffers with explicit layout that allows for
decoupling the indexing scheme from the underlying storage:memref<10x10xf32, strides: [1,10]>
affords column-major access while having row-major storage. The memref data type also provides
an unsurprising ABI to interoperate with external C code, useful to interact with libraries.
2.3.4 tensor Dialect. The tensor dialect operates on an abstract n-D tensor type for which we have
not yet decided on a representation in memory. During the compilation, sufficiently small tensors
of static sizes may be placed directly in (vector) registers while larger or dynamically-sized tensors
are put into memory storage thanks to the bufferization process. Tensor values are immutable and
subject to def-use SSA semantics, operations on tensors are often free of side-effects. This allows
classical compiler transformations such as peephole optimizations, constant subexpression and
dead code elimination, or loop-invariant code motion to apply seamlessly to tensor operations
regardless of their underlying complexity. Since tensor values are immutable, they cannot be written
into. Instead, “value insertion” operations create new tensors with a value or a subset thereof replaced. 1
2.3.5 scf Dialect. The structured control flow scf dialect provides operations that represent looping
and conditionals (e.g. regular scf.for and scf.while loops without early exit as well as an scf.if
conditional construct) and embeds them into the SSA+regions form of MLIR. This is structured at a
higher-level of abstraction than a control flow graph. Notably, scf loop operations may yield SSA
values and compose well with other operations and dialects with either memory-based side-effecting
semantics or SSA-based side-effect-free semantics.
2.3.6 linalg Dialect. The linalg dialect provides higher-level compute primitives that can operate
on both tensor and memref containers. These primitives can decompose into versions of themselves
1This is analogous to the design of struct in LLVM IR: %1 = insertvalue {f64, f32, i32} %0, f32 42.0, 1 defines
a new value %1 that holds the same elements as %0 except for the element at position 1 that now holds 42.0.
Composable and Modular Code Generation in MLIR 7
operating on structured subsets of the original input data and producing similarly structured
subset of their results. They also capture program invariants and structural information, such as
independency of certain parts of computation or reduction patterns, that decreases the need for
expensive analyses prior to transformation.
2.3.7 sparse_tensor Dialect. The sparse tensor dialect provides the types and transformations
required to make sparse tensor types first-class citizens within the MLIR compiler infrastructure.
The dialect bridges high-level linalg that operates on sparse tensors with lower-level operations
on the actual sparse storage schemes that save memory and avoid performing redundant work.
2.4 Lower-level Dialects: Producing LLVM IR and Binaries
Fig. 3. Simple visual description of the MLIR compiler flow: (top) llvm dialect only, (bottom) llvm and
x86vector dialects, the latter containing hardware-specific “intrinsic” operations.
At the end of the transformation process, MLIR produces low-level dialects common to multiple
compilation paths. The llvm dialect closely mirrors LLVM IR and is the output we consider in this
paper. The MLIR module using this dialect can be translated to LLVM IR before being handed off to
the LLVM compiler to produce machine code. Figure 3(top) summarizes the tool flow.
Similarly to the rest of MLIR, this dialect can be mixed with other ones. In particular, it reuses
built-in MLIR types such as integer (i32) or floating-point (f32) scalars. A detailed example of such
a dialect mix is discussed in Appendix A.1.
While our flow mostly relies on the LLVM compiler to perform common middle-end and back-
end optimizations, some performance-critical scenarios require stronger guarantees of specific
hardware instructions being emitted. Therefore MLIR provides a handful of low-level platform-
specific dialects: nvvm, rocdl, x86vector, arm_neon, arm_sve, amx, etc. These dialects partly mirror
the corresponding sets of LLVM IR intrinsic functions, which themselves typically map to hardware
instructions. Beyond making these instructions first-class operations and providing, these dialects
also define slightly higher-level operations that make use of MLIR’s extensible type system and
other capabilities. For example, the
arm_neon .2d. sdot : vector <4 x4x i8 >, vector <4 x4x i8 > to vector <4x i32 >
operation is naturally expressed on a MLIR multidimensional vector type. Before converting to
LLVM IR, it is first lowered to
arm_neon . intr . sdot : vector <16xi8 >, vector <16xi8 > to vector <4x i32 >
that operates on flattened 1-D vectors to match LLVM’s convention. The full example is provided
in Appendix A.2.
8 Vasilache et al.
3 TRANSFORMATIONS
We follow the IR step by step as it is transformed, considering alinalg.conv_1d_nwc_wcf operation
and its lowering to a tiled, padded and vectorized form. The input IR is shown in Figure 4 (left).
!tI = type tensor<1x990x32xf32>
!tK = type tensor<3x32x64xf32>
!tO = type tensor<1x988x64xf32>
func @conv_1d_nwc_wcf_main(%I: !tI #in, %K: !tK #in,
%O: !tO #out) -> !tO {
%cst = arith.constant 0.000000e+00 : f32
%0 = linalg.fill(%cst, %O) : f32, !tO -> !tO
%1 = linalg.conv_1d_nwc_wcf
ins(%I, %K : !tI, !tK) outs(%0 : !tO) -> !tO
return%1 : !tO
}
!tDyn= type tensor<?x?x?xf32>
%1 = scf.for %n = 0 to 1 step 1 iter_args(%O1 = %0) -> (!tO) {
%2 = scf.for %w = 0 to 988 step 8 iter_args(%O2 = %O1) -> (!tO) {
%3 = scf.for %f = 0 to 64 step 32 iter_args(%O3 = %O2) -> (!tO) {
%4 = scf.for %kw = 0 to 3 step 1 iter_args(%O4 = %O3) -> (!tO) { 
%5 = scf.for %c = 0 to 32 step 8 iter_args(%O5 = %O4) -> (!tO) {
%8 = tensor.extract_slice %I[%n, %w + %kw, %c]
[1, min(8, 990 - %w - %f), 8] [1, 1, 1] : !tI to !tDyn
%9 = tensor.extract_slice %F[%kw, %c, %f]
[1, 8, 32] [1, 1, 1] : !tK to !tDyn
%11 = tensor.extract_slice%arg12[%n, %w, %f]
[1, min(8, 988 - %w), 32] [1, 1, 1] : !tO to !tDyn
%12 = linalg.conv_1d_nwc_wcf
ins(%8, %9 : !tDyn, !tDyn) outs(%11 : !tDyn) -> !tDyn
%13 = tensor.insert_slice%12 into %O4[%n, %w, %f]
[1, min(8, 988 - %w), 32] [1, 1, 1] : !tDyninto !tO
scf.yield%13 : !tO }
scf.yield %5 : !tO }
scf.yield %4 : !tO }
scf.yield %3 : !tO }
scf.yield %2 : !tO }
#id3d= affine_map<(d0,d1,d2)->(d0,d1,d2)>
{linalg.buffer_layout = #id3d,
linalg.inplaceable = false}
%7 = affine.min affine_map<
(d0)->(8,990-d0)>(%6)
%6 = affine.apply affine_map
<(d0,d1)->(d0+d1)>
(%arg5, %arg9)
%c1 = arith.constant 1 : index
tile[1,8,
32,1,8]
{linalg.buffer_layout = #id3d,
linalg.inplaceable = true}
Fig. 4. Tiling a convolution on tensors introduces loops with secondary induction variables, pseudo-IR. Parts
in italic are simplified for clarity and expanded in callouts. Underscored parts refer to new concepts: (left)
operations on immutable tensors, (right) secondary induction variables and tensor slicing.
This level of abstraction operates on immutable SSA values: new tensor values are created from
existing ones. Memory locations only occur as annotations at function boundary to specify how
these tensors will materialize into memory, in a subsequent lowering step (more on this later).
The index notation corresponding to linalg.conv_1d_nwc_wcf is given by a 5-D rectangular
iteration domain operating on 3-D tensors along with the expression:2
𝑂[𝑛,𝑤,𝑓 ]= 𝐼[𝑛,𝑤 +𝑘𝑤,𝑐].𝐾[𝑘𝑤,𝑐,𝑓 ]
The iteration domain is implicit in the operation description and is such that the iterators span the
entire data of the operands. In this example, this is given by the inequalities
0 ≤𝑛 < 𝑂.0, 0 ≤𝑤 < 𝑂.1, 0 ≤𝑓 < 𝑂.2, 0 ≤𝑘𝑤 < 𝐾.0, 0 ≤𝑐 < 𝐾.1
where 𝑂.𝑑 denotes the size of the𝑑-th dimension of𝑂. The derivation for these quantities follows the
same rules as Tensor Comprehensions [55]. In the dense case, they can be derived with successive
applications of the Fourier-Motzkin elimination procedure [47].
3.1 Tiling
Tiling the operation introduces scf.for loops as well as subset operations (tensor.extract_slice
and tensor.insert_slice) to access the tiled data Figure 4 (right). The tiled form of the operation
is itself a linalg.conv_1d_nwc_wcf operating on the tiled subsets. The derivation of dense subsets is
obtained by computing the image of the iteration domain by the indexing function for each tensor.
Non-dense iteration domains and subsets require IR extensions and inspector-executor [32] code
generation that are outside the scope of this paper.
Back to our example, we chose tile sizes of1x8x32x1x8. While these sizes are static, some divisions
are not integral and the boundary tiles are subject to full/partial tile classification. As a result, there
is no single static tensor type that is valid for every loop iteration; the tiled tensor type !tDyn must
be relaxed to a dynamically shaped tensor.3. The dynamic tile sizes needed to access the tile data
slices are %8, %9 and %11. Separate canonicalizations later kick in to further refine the types that
can be determined to be partially static.
2The operation also allows specifying static sizes and strides that we omit here for simplicity
3Note that our compilation flow supports full dynamism, although for illustration purposes we focus on static inputs.
Composable and Modular Code Generation in MLIR 9
The scf.for loops introduced by this “tiling on tensors” transformation perform iterative yields
of the full tensor value that is produced at each iteration of the loop nest. The yielding form
of scf.for combined with subset operations generalize the insertion into a structure to a more
dynamic type (see Section 2.3.4). Since tensor values are immutable, new values are produced by
each tensor.insert_slice and scf.yield. It is the responsibility of the bufferization process to
avoid superfluous allocations and copies.
3.2 Padding Values and Packing
When tiling is applied, the content of a tile can often become more dynamic to account for boundary
effects. This hampers vectorization which requires static sizes. There are multiple mitigating options:
(1) Tiling may trigger multi-level loop peeling (or versioning) to isolate the statically known
constant part of the problem in a main loop, followed by cleanup loops for the boundaries.
The cleanup loops still exhibit dynamic behavior but they can always be tiled by1 and further
reduce to a dimension of size 1 that can be vectorized in a finer-grained form.
(2) An alternative is to pad the dynamic tile to a larger known static size. The value used for
padding must be the neutral for the consuming operation. This introduces extra copies and
more computations at the boundary but all tiles now become full.
(3) A third alternative is to turn to a representation that involves explicit masking. This is work
in progress and outside the scope of this paper.
!tDyn= type tensor<?x?x?xf32>
%1 = scf.for %n = 0 to 1 step 1 iter_args(%O1 = %0) -> (!tO) {
%2 = scf.for %w = 0 to 988 step 8 iter_args(%O2 = %O1) -> (!tO) {
%3 = scf.for %f = 0 to 64 step 32 iter_args(%O3 = %O2) -> (!tO) {
%4 = scf.for %kw = 0 to 3 step 1 iter_args(%O4 = %O3) -> (!tO) { 
%5 = scf.for %c = 0 to 32 step 8 iter_args(%O5 = %O4) -> (!tO) {
%8 = tensor.extract_slice %I[%n,%w+%kw,%c]
[1,min(8,990-%w-%f),8][1,1,1] : !tI to !tDyn
%9 = tensor.extract_slice %F[%kw,%c,%f]
[1,8,32][1,1,1] : !tK to !tDyn
%11 = tensor.extract_slice %arg12[%n,%w,%f]
[1,min(8,988-%w),32][1,1,1] : !tO to !tDyn
%12 = linalg.conv_1d_nwc_wcf
ins(%8, %9 : !tDyn, !tDyn) outs(%11 : !tDyn) -> !tDyn
%13 = tensor.insert_slice %12 into %O4[%n,%w,%f]
[1,min(8,988-%w),32][1,1,1] : !tDyninto !tO
scf.yield %13 : !tO }
scf.yield %5 : !tO }
scf.yield %4 : !tO }
scf.yield %3 : !tO }
scf.yield %2 : !tO }
!tISlice= type tensor<1x8x8xf32>
!tKFSlice= type tensor<1x8x32xf32>
%1 = scf.for %w = /*...*/iter_args(%O1 = %O) -> (!tO) {
%3 = linalg.init_tensor [3, 4, 1, 8, 8] : tensor<?x?x1x8x8xf32>
%PI = scf.for %kw = /*...*/iter_args(%I1 = %I) -> (tensor<?x?x1x8x8xf32>) {
%8 = scf.for %c = /*...*/iter_args(%I2 = %I1) /*...*/ {
%11 = tensor.extract_slice %I[0, %w+%kw, %c][1,min(8,990-%w-%kw),8][1,1,1]
%13 = linalg.pad_tensor%11 nofold low[0,0,0] high[0,8-min(8,990-%w-%kw),0] {
^bb0(/*tensor indices*/):
linalg.yield 0.000000: f32
} : tensor<1x?x8xf32> to !tISlice
%14 = tensor.insert_slice %13 into %I2 [1,%c ceildiv 8,0,0,0]
[1,1,1,8,8][1,1,1,1,1]
scf.yield %14 : tensor<?x?x1x8x8xf32> }
scf.yield %8 : tensor<?x?x1x8x8xf32> }
%5 = scf.for %f = /*...*/iter_args(%O2 = %O1) -> (!tO) {
%6 = scf.for %kw = /*...*/iter_args(%O3 = %O2) -> (!tO) {
%7 = scf.for %c = /*...*/iter_args(%O4 = %O3) -> (!tO) {
%8 = tensor.extract_slice %F[%kw,%c,%f][1,8,32] [1,1,1] /*...*/
%9 = tensor.extract_slice %O4[0,%w,%f][1,min(8,988-%w),32][1,1,1] /*...*/
%12 = tensor.extract_slice %PI[%kw,%c ceildiv 8,0,0,0][1,1,1,8,8][1,1,1,1,1]
%13 = linalg.pad_tensor%8 nofold low[0,0,0] high[0,0,0] /*...*/
%15 = linalg.pad_tensor%9 low[0,0,0] high[0, 8-min(8,988-%w),0] /*...*/
%16 = linalg.conv_1d_nwc_wcf
ins(%12, %13 : !tISlice, !tKFSlice) outs(%15 : !tKFSlice) -> !tKFSlice
%17 = tensor.extract_slice %16[0,0,0][1,min(8,988-%w),32][1,1,1] /*...*/
%18 = tensor.insert_slice %17 into %O4[0, %w, %f][1,min(8,988-%w),32][1,1,1]
scf.yield %18 : !tO }
scf.yield %7 : !tO }
scf.yield %6 : !tO }
scf.yield %5 : !tO }
pad
Fig. 5. Padding a tiled operation to obtain fixed-size tensors (highlighted), pseudo-IR. Parts in italic are
simplified for brevity. Constants in roman font are attributes, in italic are arith.constant operation results.
The nofold padding persists in the IR even without type change and may be later placed in a fast buffer.
Trailing type annotations sometimes omitted for brevity.
When the computation does not entail enough temporal locality, peeling is almost always the
better choice. As soon as some temporal locality is available, the copy required for padding can
be amortized. Padding then also serves the purpose of aligning memory accesses in the padded
buffer, once bufferization has occurred. This is particularly important to avoid cache line splitting ,
i.e., partially clobbering cache lines and inducing extraneous data transfers through the cache
10 Vasilache et al.
hierarchy; it is often required to reach the highest levels of performance. In other words, value
padding also implies address padding.4
Padding is materialized by the introduction of tensor.pad operations, and its size is obtained by
subtracting the dynamic tile size from the static tile size. All elements in the padded region are set
to the constant value %cst. This makes all operands of the tiled convolution statically shaped.
When the tile shape is already known to be static, value padding is not required for vectorization.
In such cases, linalg.pad would simply fold away. We additionally support anofold attribute to
force padding to occur in such cases where address alignment is essential to avoid cache line split.
Operations that exhibit temporal reuse of the data may additionally benefit from hoisting the
padding operation out of the tile loops and storing the padded tiles in a higher-dimensional packed
tensor. This allows both amortizing the cost of copying as well as physically laying out tiles
contiguously in an intermediate buffer. This results in a smaller distance in memory between tiles
that are reused within a short time-span and reduces TLB misses.
The amount of hoisting is configurable per tensor and exhibits various trade-offs between
memory consumption, cost of copy and benefits to the computation primitive.
In the example of Figure 5, the input tensor padding is hoisted by 3 loops. This introduces an
additional tile loop nest to precompute the padded tiles and insert them into the packed tensor
of type tensor<?x?x1x8x8xf32> containing all padded tiles. Inside the original tile loop nest, the
padding is replaced by an access to the packed tensor %12= tensor.extract_slice %PI....
Similarly to the tiling case, the sizes of packed tensor are obtained by computing the image of the
iteration domain by a function of the enclosing loop variables and the tensor’s indexing function.
3.3 Vectorization
After tiling and padding, the convolution operands are statically shaped and are in a good state for
vectorization, see Figure 6 (left). In the current IR, only 2 types of operations need to be vectorized:
tensor.pad and linalg.conv1d_nwc_wcf.
The vectorization of tensor.pad is implemented with a simple one-off pattern that reduces it to a
pair of vector.transfer_read, vector.transfer_write operations. The vector.transfer operations
are often referred to as a Swiss army knife for bridging the gap between memory and vectors. In
particular, they carry enough information to encode various multi-dimensional vector memory
read and write patterns (e.g., broadcasted, permuted, masked, padded accesses). They can be
easily retargeted to specifics of current and future memory subsystems and vector ISAs. With the
existence of such an operation, the vectorization of tensor.pad is comparatively a very small and
progressive lowering step.5 The vector dialect additionally provides a first-class representation for
high-intensity operations. These operations are a key building block for high-performance codegen.
Figure 6 (right) illustrates one such operation, vector.contract, in action.
The vectorization of linalg operations follows a recipe that introduces a vector.transfer_read
for each operand, performs the computation in vector form and commits it back to the proper tensor
or buffer via a vector.transfer_write. The vector.transfer operations are indexed following the
indexing expressions of the linalg operation. This behavior is generic for the data movement part
of all linalg operations.
The vectorization of the computation part is subject to variations. Under the hood, every linalg
operation has a body region that expresses the scalar form of the computation 6. The body vector-
ization depends on the type of indexings the parent linalg.generic performs:
4Additional masking may in turn facilitate address padding.
5Especially if one thinks about all the load, store, indexing and fill operations that result from such a construct.
6The body may be printed explicitly (when expressed in linalg.generic form) or simply elided when expressed in
"named" form (such as linalg.conv_xxx).
Composable and Modular Code Generation in MLIR 11
!tISlice= type tensor<1x8x8xf32>
!tKFSlice= type tensor<1x8x32xf32>
%1 = scf.for %w = /*...*/iter_args(%O1 = %O) -> (!tO) {
%3 = linalg.init_tensor [3, 4, 1, 8, 8] : tensor<?x?x1x8x8xf32>
%PI = scf.for %kw = /*...*/iter_args(%I1 = %I) -> (tensor<?x?x1x8x8xf32>) {
%8 = scf.for %c = /*...*/iter_args(%I2 = %I1) /*...*/ {
%11 = tensor.extract_slice %I[0, %w+%kw, %c][1,min(8,990-%w-%kw),8][1,1,1]
%13 = linalg.pad_tensor %11 nofold low[0,0,0] high[0,8-min(8,990-%w-%kw),0] {
^bb0(/*tensor indices*/):
linalg.yield 0.000000: f32
} : tensor<1x?x8xf32> to !tISlice
%14 = tensor.insert_slice %13 into %I2 [1,%c ceildiv 8,0,0,0]
[1,1,1,8,8][1,1,1,1,1]
scf.yield %14 : tensor<?x?x1x8x8xf32> }
scf.yield %8 : tensor<?x?x1x8x8xf32> }
%5 = scf.for %f = /*...*/iter_args(%O2 = %O1) -> (!tO) {
%6 = scf.for %kw = /*...*/iter_args(%O3 = %O2) -> (!tO) {
%7 = scf.for %c = /*...*/iter_args(%O4 = %O3) -> (!tO) {
%8 = tensor.extract_slice %F[%kw,%c,%f][1,8,32] [1,1,1] /*...*/
%9 = tensor.extract_slice %O4[0,%w,%f][1,min(8,988-%w),32][1,1,1] /*...*/
%12 = tensor.extract_slice %PI[%kw,%c ceildiv 8,0,0,0][1,1,1,8,8][1,1,1,1,1]
%13 = linalg.pad_tensor %8 nofold low[0,0,0] high[0,0,0] /*...*/
%15 = linalg.pad_tensor %9 low[0,0,0] high[0, 8-min(8,988-%w),0] /*...*/
%16 = linalg.conv_1d_nwc_wcf
ins(%12, %13 : !tISlice, !tKFSlice) outs(%15 : !tKFSlice) -> !tKFSlice
%17 = tensor.extract_slice %16[0,0,0][1,min(8,988-%w),32][1,1,1] /*...*/
%18 = tensor.insert_slice %17 into %O4[0, %w, %f][1,min(8,988-%w),32][1,1,1]
scf.yield %18 : !tO }
scf.yield %7 : !tO }
scf.yield %6 : !tO }
scf.yield %5 : !tO }
!vecI= type vector<1x8x8xf32>
!vecKF= type vector<1x8x32xf32>
!vecO= type vector<1x8xf32>
#proj_012= affine_map<(d0,d1,d2,d3) -> (d0,d1,d3)
#proj_32= affine_map<(d0,d1,d2,d3) -> (d3,d2)
%1 = scf.for %w = /*...*/iter_args(%O1 = %O) -> (!tO) {
%3 = linalg.init_tensor [3, 4, 1, 8, 8] : tensor<?x?x1x8x8xf32>
%PI = scf.for %kw = /*...*/ iter_args(%I1 = %I)->(tensor<?x?x1x8x8xf32>) {
%8 = scf.for %c = /*...*/iter_args(%I2 = %I1) /*...*/{
%11 = tensor.extract_slice %I /*...*/
%12 = vector.transfer_read%11[0,0,0], 0.000000
{in_bounds=[true,false,true]} : tensor<1x?x8xf32>, !vecI
%13 = vector.transfer_write%12, %arg8[%kw,%c ceildiv 8,0,0,0]
{in_bounds=[true,true,true]} : !vecI, tensor<?x?x1x8x8xf32>
scf.yield %13 : tensor<?x?x1x8x8xf32> }
scf.yield %8 : tensor<?x?x1x8x8xf32> }
%5 = scf.for %f = /*...*/ iter_args(%O2 = %O1) -> (!tO) {
%7 = tensor.extract_slice %O2[0,%w,%f][1,min(8,988-%w),32][1,1,1]/*...*/
%8 = vector.transfer_read%7[0,0,0], 0.000000
{in_bounds=[true,false,true]} : tensor<1x?x32xf32>, !vecKF
%9 = scf.for %kw = /*...*/ iter_args(%O3 = %8) -> (!vecO) {
%12 = scf.for %c = /*...*/ iter_args(%O4 = %O3) -> (!vecO) {
%14 = vector.transfer_read%PI[%kw,%c ceildiv 8,0,0,0], 0.000000
{in_bounds=[true,true,true]} : tensor<?x?x1x8x8xf32>, !vecI
%15 = vector.transfer_read%F[%kw,%c,%f], 0.000000
{in_bounds=[true,true,true]} : tensor<3x32x64xf32>, !vecKF
%16 = vector.extract %15[0] : !vecKF
%17 = vector.contract{indexing_maps=[#proj_012,#proj_32,#proj_012],
iterator_types=["parallel","parallel","parallel","reduction"]}
%14, %16, %O4 : !vecI, vector<8x32xf32> into !vecKF
scf.yield %17 : !vecO }
scf.yield %7 : !vecO}
%A = vector.transfer_write%9, %7[0,0,0]
{in_bounds=[true,false,true]} : !vecKF, tensor<1x?x32xf32>
%B = tensor.insert_slice %10 into %O2[0,%w,%f][1,min(8,988-%w),32][1,1,1]
scf.yield %B : !tO }
scf.yield %5 : !tO }
vectorize
Fig. 6. Operations on tensors of fixed sizes can be directly vectorized, pseudo-IR. Parts in italic are simplified
for brevity as in Figure 5. Vector values are immutable. They can be read from and written into tensors.
Out-of-bounds access is allowed; reads replicate the scalar operand, writes are ignored.
(1) In the simplest case of pointwise operations (indexings are all identity), every operation in
the body is simply written as a pointwise vector variant.
(2) Lower dimensional tensor operands can bevector.broadcast into higher-dimensional vectors
where needed and reduce to the previous case.
(3) Permutations in indexing expressions are handled with vector.transpose operations.
(4) Reduction dimensions lower to a first-class vector.contract or vector.multi_reduction
depending on further analysis of the body.
(5) Sliding window patterns such as convolutions are handled specially by unrolling along certain
dimensions and extracting slices that further reduce to vector.contract or vector.fma. This
simple strategy delivers high performance while capturing strided and dilated convolutions.
In our running example of Figure 6 (right), the dimension corresponding to the %kw loop would
be unrolled. For the purpose of this illustration we have used a tile size of 1 which does not further
unroll. Note the %16 = vector.extract %15[0] : !vecK operation which is a degenerate form of
unrolled slice extraction for size 1. Additional canonicalization and folding patterns occur that
simplify chains of vector.transfer operations, as well as move loop-independent instructions
out of loops (e.g. %8 = vector.transfer_read). Also note that the loops %9 = scf.for ... and
%12 = scf.for ... both yield vector values without inserting or extracting from a tensor. This will
guarantee no roundtrip to memory after bufferization.
All these transformations are implemented by following SSA def-use chains and legal by design
(see Section 3.6 for a discussion of this principle).
3.4 Bufferization
Bufferization is the process of materializing tensor values into memory (memref). It is necessary
to make tensor programs concretely executable with a source of data residing in memory. In our
current compilation pipeline, it is one of the last steps.
12 Vasilache et al.
Tensors in MLIR are immutable. An operation that produces a new tensor value (maybe from
another input tensor) is conceptually an entirely new tensor. Unlike with memrefs, there is no
concept of updating/writing to a tensor in-place. To achieve good performance, it is essential to:
•Allocate as little memory as possible.
•Copy as little memory as possible.
Buffers should be reused and updated in-place whenever possible or risk large performance
penalty when program transformations result in unexpected allocation and copying.
%0 = tensor.insert %f into %A[%c0, %c0] : tensor<?x?xf32>
tied OpOperand / OpResult pair %o = …
%w = tensor.insert %f, %o[%c0, %c0]
{__inplace__ = ["none", "true", "none", "none"]} : tensor<?x?xf32>
%r = tensor.extract %o[%c0, %c0] : tensor<?x?xf32>
bufferizes to mem. read + write
bufferizes to mem. read
expects to read
(“last write”)
alias sets: {{%o, %w}}
Fig. 7. Left-hand side: output tensor arguments, tied with the result of an operation, in destination-passing
style. Right-hand side: example of a read-after-write conflict.
Read-after-Write Conflicts. Allocating a new buffer for every memory write is always safe, but
wastes memory and introduces unnecessary copies. On the other hand, reusing a buffer and writing
to it in-place can result in invalid bufferization if the original data at the overwritten memory
location must be read at a later point of time. When performing transformations, one must be
careful to preserve program semantics exposed by dependencies [1]. The right-hand side of Figure 7
illustrates a potential Read-after-Write (RaW) conflict that prevents in-place bufferization. The
problem of efficient bufferization is related to register coalescing, the register allocation sub-task
associated with the elimination of register-to-register moves.
Destination-Passing Style. The current heuristic we are offering for bufferization is well-suited
for operations that are in destination-passing style . In such operations, one of the tensor arguments
is tied with the resulting tensor for in-place bufferization. Such a tensor argument is called an
output tensor, see the left-hand side of Figure 7. Intuitively, output tensors are similar to C++
output parameters that are passed as non-const references and used for returning the result of a
computation. Except these ties between an output tensor (argument) and the operation’s result
serve as a bufferization constraint with no observable impact on the functional semantics; in
particular, output tensors still appear as immutable. During bufferization, only output tensors are
considered when looking for a buffer to write the result of an operation into.
The rationale derives from first principles when composing structured operations with scf.for,
the natural target for lowering multidimensional tensor operations. Since scf.for yields a value as
a result, its nested region must yield fully defined tensors rather than arbitrary subsets. Since nested
operations typically apply to tensor subsets — often resulting from linalg tiling transformations
— a pair of matching extract_slice/insert_slice operations are typically injected. These, and
the associated scf.yield operation, naturally consume their tensor argument (i.e., there cannot
be any subsequent uses of it), which makes them ideal candidates for in-place bufferization. A
comprehensive example is shown in Figure 8.
This heuristic design appears to work well on the kind of IR we are dealing with when operating
on the linalg dialect:
•We saw that tiling produces outer loops iterating on tiled subsets. Operations to manage these
subsets, such as extract_slice, insert_slice, are naturally in destination-passing style.
Composable and Modular Code Generation in MLIR 13
•Padding, packing, vectorization, and other transformations also produce operations with a
destination-passing style semantics, on full tensors or subsets.
•linalg.generic itself is designed as a destination-passing style operation. This includes
linalg.matmul and any other operation that reduces to linalg.generic.
!vecI= type vector<1x8x8xf32>
!vecKF= type vector<1x8x32xf32>
!vecO= type vector<1x8xf32>
#proj_012= affine_map<(d0,d1,d2,d3) -> (d0,d1,d3)
#proj_32= affine_map<(d0,d1,d2,d3) -> (d3,d2)
func @conv_1d_nwc_wcf_main(%I: !tI#in, %K: !tK#in,
%O: !tO#out) -> tensor<1x988x64xf32>{
%1 = scf.for %w = /*...*/iter_args(%O1 = %O) -> (!tO) {
%3 = linalg.init_tensor [3, 4, 1, 8, 8] : tensor<?x?x1x8x8xf32>
%PI = scf.for %kw = /*...*/ iter_args(%I1 = %I)->(tensor<?x?x1x8x8xf32>) {
%8 = scf.for %c = /*...*/iter_args(%I2 = %I1) /*...*/{
%11 = tensor.extract_slice %I /*...*/
%12 = vector.transfer_read %11[0,0,0], /*...*/: tensor<1x?x8xf32>,!vecI
%13 = vector.transfer_write %12, %arg8[%kw,%c ceildiv 8,0,0,0]
/*...*/: !vecI, tensor<?x?x1x8x8xf32>
scf.yield %13 : tensor<?x?x1x8x8xf32> }
scf.yield %8 : tensor<?x?x1x8x8xf32> }
%5 = scf.for %f = /*...*/ iter_args(%O2 = %O1) -> (!tO) {
%7 = tensor.extract_slice %O2[0,%w,%f][1,min(8,988-%w),32][1,1,1]/*...*/
%8 = vector.transfer_read %7[0,0,0], /*...*/: tensor<1x?x32xf32>, !vecKF
%9 = scf.for %kw = /*...*/ iter_args(%O3 = %8) -> (!vecO) {
%12 = scf.for %c = /*...*/ iter_args(%O4 = %O3) -> (!vecO) {
%14 = vector.transfer_read %PI[%kw,%c ceildiv 8,0,0,0], /*...*/
: tensor<?x?x1x8x8xf32>, !vecI
%15 = vector.transfer_read %F[%kw,%c,%f], /*...*/
: tensor<3x32x64xf32>, !vecKF
%16 = vector.extract %15[0] : !vecKF
%17 = vector.contract/*...*/
scf.yield %17 : !vecO }
scf.yield %7 : !vecO}
%A = vector.transfer_write %9, %7[0,0,0] /*...*/
: !vecKF, tensor<1x?x32xf32>
%B = tensor.insert_slice %10 into %O2[0,%w,%f][1,min(8,988-%w),32][1,1,1]
scf.yield %B : !tO }
scf.yield %5 : !tO }
return%1 }
!MIView= type memref<1x?x8xf32, offset: ?, strides: [31680,32,1]>
!MOView= type memref<1x?x32xf32, offset: ?, strides: [63232,64,1]> 
!MBuf= type memref<3x4x1x8x8xf32>
func @conv_1d_nwc_wcf_main(%I: memref<1x990x32xf32>,
%F: memref<3x32x64xf32>, %O: memref<1x988x64xf32>) {
%0 = memref.alloc() {alignment = 128 : i64} : !MBuf
scf.for %w = /*...*/{
scf.for %kw = /*...*/{
scf.for %c = /*...*/{
%5 = memref.subview%I[0,min(8,988-%w,%kw][1,min(8,990-%w-%kw),8][1,1,1] 
: memref<1x990x32xf32> to !MIView
%6 = vector.transfer_read %5[0,0,0], /*...*/: !MIView, !vecI
vector.transfer_write %6, %0[%kw,%c ceildiv 8,0,0,0] /*...*/!vecI, !MBuf
}}
scf.for %f = /*...*/{
%2 = memref.subview%O[0,%w,%f][1,min(8,988-%w),32][1,1,1] 
: memref<1x988x64xf32> to !MOView
%3 = vector.transfer_read %2[0,0,0], /*...*/: !MOView, !vecKF
%4 = scf.for %kw = /*...*/iter_args(%O3 = %3) -> (!vecO) {
%5 = scf.for %c = /*...*/iter_args(%O4 = %O3) -> (!vecO) {
%14 = vector.transfer_read %0[%kw,%c ceildiv 8,0,0,0], /*...*/
: memref<3x4x1x8x8xf32>, !vecI
%15 = vector.transfer_read %F[%kw,%c,%f], /*...*/
: memref<3x32x64xf32>, !vecKF
%16 = vector.extract %15[0] : !vecKF
%17 = vector.contract/*...*/
scf.yield %17 : !vecO}
scf.yield %7 : !vecO}
vector.transfer_write %4, %2[0,0,0] /*...*/: !vecKF, !MOView
/* insert_slice may result in a copy, elided here */
}}
memref.dealloc%0 : !MBuf
return}
affine_map<(d0, d1, d2)[s0] -> (d0 * 31680+ s0 + d1 * 32 + d2)>
bufferize
Fig. 8. Bufferization assigns tensor values to buffers, taking into account function-level annotations #in, #out
from Figure 4. Data flow is replaced by side effects, unnecessary values are crossed out on the left. Temporary
buffers may be allocated to ensure contiguous access patterns. “Computational payload” dialects such as
linalg and vector are designed to support both tensor and memref (buffer) containers.
To illustrate this rationale, considertensor.insert as an example of an operation in destination-
passing style. A tensor result of an operation may have one or multiple potentially aliasing Op-
Operands in the bufferization framework. For example, the only potentially aliasing OpOperand of
%0 in the example is %A (Figure 7, left-hand side), meaning that after bufferization:
•buffer(%0) = buffer(%A)
•Or: buffer(%0) is a newly allocated buffer.
No other operands are taken into account when choosing a buffer. Operations that do not have a
potentially aliasing OpOperand for their tensor results always allocate a new buffer. For example,
tensor.generate always allocates after bufferization.
This bufferization design is a heuristic: operations with no natural candidate output tensor (e.g.
the sum of two tensors) allocate and copy by default. This simplifies the bufferization problem,
reducing it to an analysis of use-def chains. The tradeoff is that upstream compilation passes are
responsible of rewriting the IR in destination-passing style. We believe a global copy elimination
problem could be formalized on top of destination-passing style, offering the best of both worlds in
terms of allowing passes to optimize bufferization at a global scale, while still enabling a robust,
in-place bufferization path for the important special case of refining structured operations.
Analysis of Tensor SSA Use-Def Chains. During bufferization, each operation with tensor semantics
is replaced with an operation with memref semantics. Before modifying any IR, an analysis decides
for each tensor OpOperand %t whether buffer(%t) (in-place bufferization ) or a copy thereof (out-of-
place bufferization), denoted bycopy(buffer(%t)), should be used with the new memref operation. The
14 Vasilache et al.
analysis simulates a future in-place bufferization of the OpOperand and checks if a RaW conflict can
be found under this assumption. If not, the analysis greedily commits to this in-place bufferization
decision. Furthermore, the analysis stores the fact that the OpOperand and its potentially aliasing
OpResult are now known to alias , by merging their alias sets.
The search for RaW conflicts is based on a traversal of tensor SSA use-def chains. When simulating
the in-place bufferization of an OpOperand %o, the analysis looks for all uses of this SSA value
(and its aliases) that bufferize to a memory read. For each read, it walks the IR back the last write
of the tensor; i.e., the value that defines the contents of the tensor (skipping over operation like
tensor.cast or tensor.extract_slice that do not bufferize to a memory write). This is the
value that the operation is expected to read. If an interleaved “future write” of an aliasing buffer
exists, then there would be a RaW conflict.
The in-place bufferization of %o of the tensor.insert (Figure 7, right-hand side) is a RaW
conflict (and would thus be an invalid bufferization) because the operation is in-between the
definition of %o and the tensor.extract operation, which is expected to read the original value
of %o and not %w.
Extension Points. The bufferization framework is customizable and can be adapted for different
use cases via two main extension mechanisms.
First, the order in which operations are analyzed is a heuristic and affects the order in which
RaW conflicts are detected. There are often multiple out-of-place bufferization candidates that can
prevent a RaW conflict. Only when a RaW conflict is no longer avoidable, the analysis decides
to bufferize an OpOperand out-of-place. Therefore, operations (and their OpOperands) that are
analyzed earlier are less likely to bufferize out-of-place. By analyzing a certain group of operations
first, the analysis can be steered towards bufferizing these operations in-place and trying to avoid
RaW conflicts by bufferizing other operations out-of-place.
Second, operations can refine the analysis by specifying conidtions that should not be treated
as RaW conflicts. For example, a tensor.insert_slice operation only affect a subset of the
entire buffer. This informatio can be used to make better bufferization decisions around matching
tensor.extract_slice/tensor.insert_slice pairs. This paves the way for more advanced
analyses based on intersection and difference, as well as partial reuse, of buffer regions.
3.5 Progressive Lowering of Multidimensional Vector Operations Towards LLVM
At this time, the IR has reached a level of abstraction consisting of loops around buffers containing
multi-dimensional vectors and operations on those. This is now close to the C + vectors paradigm
of LLVM, with the exception that we operate on multi-dimensional vectors, whereas LLVM only
has 1-D vectors.
In the simplest case, multi-dimensional vector.transfer operations lower to multiple 1-D
vector.load and vector.store operations. When supported by hardware, they can also lower to n-D
DMA operations. In more complex cases, transfer operations lower to a combination of broadcasts,
tranpositions and masked scatter/gather. In the particular case where the vector.transfer cannot
be determined to be in-bounds, one must resort to an additional separation between full and partial
transfer, akin to the full and partial tile separation at the tile level. This is illustrated in Figure 9
(right) in the else block around the linalg.copy(%21, %22) operation.
Our pervasive use of n-D vector types effectively shielded an "unroll-and-jammed vector form",
known to be efficient on vector hardware, from intermediate compilation phases that could interfere
with later vectorization. At this point, this form is ready for progressive lowering into 1-D operations
with almost 1-1 mapping to LLVM IR.
Composable and Modular Code Generation in MLIR 15
!MIView= type memref<1x?x8xf32, offset: ?, strides: [31680,32,1]>
!MOView= type memref<1x?x32xf32, offset: ?, strides: [63232,64,1]> 
!MBuf= type memref<3x4x1x8x8xf32>
func @conv_1d_nwc_wcf_main(%I: memref<1x990x32xf32>,
%F: memref<3x32x64xf32>, %O: memref<1x988x64xf32>) {
%0 = memref.alloc() {alignment = 128 : i64} : !MBuf
scf.for %w = /*...*/{
scf.for %kw = /*...*/{
scf.for %c = /*...*/{
%5 = memref.subview %I[0,min(8,988-%w,%kw][1,min(8,990-%w-%kw),8][1,1,1] 
: memref<1x990x32xf32> to !MIView
%6 = vector.transfer_read %5[0,0,0], /*...*/: !MIView, !vecI
vector.transfer_write %6, %0[%kw,%c ceildiv 8,0,0,0] /*...*/!vecI, !MBuf
}}
scf.for %f = /*...*/{
%2 = memref.subview %O[0,%w,%f][1,min(8,988-%w),32][1,1,1] 
: memref<1x988x64xf32> to !MOView
%3 = vector.transfer_read %2[0,0,0], /*...*/: !MOView, !vecKF
%4 = scf.for %kw = /*...*/iter_args(%O3 = %3) -> (!vecO) {
%5 = scf.for %c = /*...*/iter_args(%O4 = %O3) -> (!vecO) {
%14 = vector.transfer_read %0[%kw,%c ceildiv 8,0,0,0], /*...*/
: memref<3x4x1x8x8xf32>, !vecI
%15 = vector.transfer_read %F[%kw,%c,%f], /*...*/
: memref<3x32x64xf32>, !vecKF
%16 = vector.extract %15[0] : !vecKF
%17 = vector.contract/*...*/
scf.yield %17 : !vecO}
scf.yield %7 : !vecO}
vector.transfer_write %4, %2[0,0,0] /*...*/: !vecKF, !MOView
}}
memref.dealloc %0 : !MBuf
return}
!MBufView= memref<1x?x8xf32, offset: 0, strides: [64,8,1]>
!MIDynView= memref<1x?x8xf32, offset: 0, strides: [?,?,?]>
%11 = memref.subview %O /*...*/
%12 = scf.if min(8,988-%w) >= 8 {
scf.yield %11 as !MIDynView
} else {
linalg.fill(0.000000, %0) : f32, memref<1x8x8xf32> 
%21 = memref.subview %11/*...*/: !MIViewto !MIView
%22 = memref.subview %0/*...*/: memref<1x8x8xf32> to !MBufView
linalg.copy(%21, %22) : !MIView, !MBufView
scf.yield %0 as !MIDynView}
%13 = vector.load %12[0,0,0] : !MBufView, vector<8xf32>
vector.store %13, %3[%kw,%c ceildiv 8,0,0,0] : !MBuf, vector<8xf32>
/*... etc ...*/
%20 = vector.load %12[0,0,7] : !MBufView, vector<8xf32>
vector.store %20, %3[%kw,%c ceildiv 8,0,7,0] : !MBuf, vector<8xf32>
%37 = vector.load %0[%kw,%c ceildiv 8,0,0,0] : !MBuf, vector<8xf32>
%61 = vector.extract %37[0] : vector<8xf32>
%62 = vector.insert %61, dense<0.000000e+00>[ 0, 0] : f32 into vector<8x8xf32>
/*... more 1D vectors inserted into %62 to produce %64,%68, ...*/
%44 = vector.load %0[%kw,%c ceildiv 8,0,7,0] : !MBuf, vector<8xf32>
%187 = vector.extract %44[7] : vector<8xf32>
%188 = vector.insert %187, %186 [7, 7] : f32 into vector<8x8xf32>
%45 = vector.load %F[%kw,%c,%f] : memref<3x32x64xf32>, vector<32xf32>
/*... etc ...*/
%59 = vector.load %F[%kw,%c+7,%f] : memref<3x32x64xf32>, vector<32xf32>
%189 = vector.extract %188[0] : vector<8x8xf32>
%60 = vector.extract %O4[0] : vector<1x8x32xf32>
%190 = vector.outerproduct%189, %45, %60
{kind = #vector.kind<add>} : vector<8xf32>, vector<32xf32>
/*... third operand of outerproduct is the result of the previous one ...*/
%203 = vector.extract %188[7] : vector<8x8xf32>
%204 = vector.outerproduct%203, %59, %202
{kind = #vector.kind<add>} : vector<8xf32>, vector<32xf32>
%205 = vector.broadcast %204 : vector<8x32xf32> to vector<1x8x32xf32>
lower vectors
Fig. 9. The vector dialect can be lowered progressively to simpler operations on 1-D vectors. Illustrated on
lowering contractions to outer products, with parts in italic simplified for brevity and repetitive parts omitted.
Lower-level vector operations require constant indices and are produced by unrolling the outer dimensions.
%a = vector.transfer_read %m0[0], 0.0f {
in_bounds=[true,true],
permutation_map=affine_map<(d0)->(d0,0)>}
: memref<2xf32>, vector<2x2xf32>
%b = vector.transfer_read %m1[0,0], 0.0f {
in_bounds=[true,true],
permutation_map=affine_map<(d0,d1)->(d1,d0)>}
: memref<2x16xf32>, vector<16x2xf32>
%v = vector.broadcast 0.0f : f32 to vector<2x16xf32>
%d = vector.contract#matmul_trait %a, %b, %v
: vector<2x2xf32>, vector<16x2xf32>
vector.transfer_write %d, %m2[0,0] {
in_bounds=[true,true]}
: vector<2x16xf32>, memref<2x16xf32>
%b0 = vector.extract_strided_slice%b
{offsets=[0,0], sizes=[8,2], strides=[1,1]}
: vector<16x2xf32> to vector<8x2xf32>
%v0 = vector.extract_strided_slice %v
{offsets=[0,0], sizes=[2,8], strides=[1,1]}
: vector<2x16xf32> to vector<2x8xf32>
%d0 = vector.contract#matmul_trait %a, %b0, %v0
: vector<2x2xf32>, vector<8x2xf32>
%r0 = vector.insert_strided_slice%d0, %v
{offsets=[0,0], strides=[1,1]}
: vector<2x8xf32> into vector<2x16xf32>
%b1 = vector.extract_strided_slice %b
{offsets=[8,0], sizes=[8,2], strides=[1,1]}
: vector<16x2xf32> to vector<8x2xf32>
%v1 = vector.extract_strided_slice %v
{offsets=[0,8], sizes=[2,8], strides=[1,1]}
: vector<2x16xf32> to vector<2x8xf32>
%d1 = vector.contract#matmul_trait %a, %b1, %v1
: vector<2x2xf32>, vector<8x2xf32>
%d = vector.insert_strided_slice %d1, %r0
{offsets=[0,8], strides=[1,1]}
: vector<2x8xf32> into vector<2x16xf32>
%r = vector.transfer_read %m1[0,0], 0.0f {
in_bounds=[true,true], 
permutation_map=affine_map<(d0,d1)->(d1,d0)>}
: memref<2x16xf32>, vector<8x1xf32>
%vt = vector.transfer_read %m1[0,0], 0.0f {
in_bounds=[true,true]}
: memref<2x16xf32>, vector<1x8xf32>
%r = vector.transpose%vt, [1, 0]
: vector<1x8xf32> to vector<8x1xf32>
%l = vector.load%m1[%c0, %c0]
: memref<2x16xf32>, vector<8xf32>
%cast= vector.shape_cast%l
: vector<8xf32> to vector<1x8xf32>
%r = vector.transpose %cast, [1, 0]
: vector<1x8xf32> to vector<8x1xf32>
%a0 = vector.transpose %a, [1, 0] 
: vector<1x1xf32> to vector<1x1xf32>
%b0 = vector.transpose %b, [1, 0]
: vector<8x1xf32> to vector<1x8xf32>
%a1 = vector.extract %a0[0] : vector<1x1xf32>
%b1 = vector.extract %b0[0] : vector<1x8xf32>
%d = vector.outerproduct%a1, %b1, %acc {
kind = #vector.kind<add>} : vector<1xf32>,vector<8xf32>
%a0 = vector.transpose %a, [1, 0]
: vector<8x1xf32> to vector<1x8xf32>
%a1 = vector.extract %a0[0] : vector<1x8xf32>
%b0 = vector.extract %b[0, 0] : vector<1x1xf32>
%bsplat= vector.splat %b0 : vector<8xf32>
%a2 = vector.extract %a1[0] : vector<1x8xf32>
%d = vector.fma %a2, %bsplat, %acc : vector<8xf32>
(a)
(b) (c)
(d)
(e)
(a)
Fig. 10. Progressive lowering of the vector dialect operations representing matrix product: (a) vector unrolling
with target shape 2 ×8 ×2 introduces vector slice manipulation; (b) the transfer permutation is materialized as
a transpose operation; (c) 1-D transfers become plain loads with shape adaptation; (d) contractions rewrite
as outer products (other options are possible), which in turn lower to (e) fused multiply-add instructions.
Let us illustrate the progressive lowering of operations on n-D vectors to lower-dimensional
equivalents closer to hardware instructions. Starting from the vectorized matrix product code on
the left of the Figure 10. This IR is (mostly) retargetable since it is using higher-level transfer and
16 Vasilache et al.
contract operations that do not usually correspond to the available hardware instructions.7 We
first apply vector “unrolling” as in Figure 10(a). The goal of this transformation is twofold: (1) it
breaks down vector operations to sizes known to be well supported by the target, e.g., mapping to
AMX instructions, and (2) preemtively handling non-power of 2 sizes into power of 2 compositions,
e.g., vector<12xf32> into 3 vector<4xf32> to avoid suboptimal backend code generation [44]. The
resulting IR is still partially retargetable as transfer and contract operations are still present and
need to be lowered to a closer-to-hardware representation using one of the available schemes.
After unrolling, vector.extract_strided_slice and vector.insert_strided_slice extract and
insert slices of a vector into a larger vector. They are conceptually the same as their tensor dialect
counterparts. When unrolling operations folding patterns can lead to insert and extract operations
cancelling each other if the target shapes match. Other peephole optimization patterns, applied
along with the unrolling transformation, track subsets through broadcast and transpose operations,
all the way to transfer operations where the may fold into memory reindexings.
Higher-level vector.transfer_read usually cannot be directly lowered to a load instructions and
are handled progressively: first, by materializing transpositions as in Figure 10(b), then, by creating
1-D loads and broadcasts as in Figure 10(c). Transpositions can in turn be implemented elementwise,
with LLVM’s shuffle instruction or using dedicated intrinsics, depending on the configuration.
Similarly, vector.contract can be lowered to outer products, inner (dot) products or LLVM IR
matrix intrinsics. In this example, it is lowered to outer products in Figure 10(d) to enable further
mapping to SIMD fused multiply-add instructions in Figure 10(e). Each stage of the progressive
lowering is accompanied by foldings and peephole optimizations that reduce the amount of IR to
handle and enable additional transformations. As the result, the fully lowered vector IR operates
on vector<8xf32>, supported for example by AVX2, and is quite compact. The resulting code for
our example has several dozen operations and is provided in Appendix A.3. These operations are
now ready for lowering to the LLVM dialect and further translation to LLVM IR.
3.6 Discussion
The levels of IR we introduced allow for developing an expressive set of essential transformations.
These transformations are legal by design, in the sense that their legality and applicability derive
from the operation’s properties and structure. We refer to this philosophy astransformations-oriented
IR design .
3.6.1 Transformation-Oriented IR Design. Traditional compiler analyses and transformations for
numerical computing [1] revolve around trade-offs and questions related to:
•Legality, i.e. what transformations can be applied without changing the observed program
semantics? Legality conditions are often checked through static analyses. They may be
performed upfront or on-demand, and their results may be updated as the IR is transformed.
For example, dominance analysis produces necessary conditions for code motion: uses should
remain dominated by definitions.
•Applicability, i.e. how complex is the IR matching process for finding the place where to
apply a transformation? How complex does the IR become after applying the transformation?
Applicability also encompasses considerations related to how much information has poten-
tially been lost, whether the IR remains analyzable and whether subsequent transformations
continue to be easily applicable.
•Profitability, i.e. what are the transformations deemed beneficial for a given metric? Prof-
itability is often determined by heuristics or performance models. For example, polyhedral
7Even with transfer operations, vector sizes become hardware-specific and other aspects such as special ISA dimensions
and the number of available registers come into play, but the representation is still easy to transform.
Composable and Modular Code Generation in MLIR 17
compilers often focus on finding an objective function to minimize (universal or target-
specific) [55], while auto-tuners may rely on a learned performance model to accelerate
search [59].
It is of central importance to control the abstractions on which the transformation legality,
applicability and profitability questions relate to. The finer-grained the IR, the more general and
canonical the representation, but also the more intractable the analyses and transformations. Indeed,
canonicalization to some flavor of LLVM IR consisting of a Static Single Assignment (SSA) form
Control Flow Graph (CFG) has proven invaluable in enabling the reuse of common infrastructure for
middle-end and back-end compilers. But lowering abstractions and domain knowledge too quickly
reduces the amount of structure available to derive transformations from. While manipulating
loops is a net gain compared to control flow graphs for a certain class of transformations, important
information is still lost (e.g. parallel and reduction multi-loop semantics, loop-level data flow and
memory footprint, substitution of a full loop nest with an external implementation). It induces
non-trivial phase ordering issues: loop fusion to enhance temporal locality may alter the ability
to recognize an efficient BLAS-2 or BLAS-3 implementation in a numerical library. Workarounds
introduce constraints on the compiler that interfere with other decisions and passes, which is a long-
standing and well-known problem in the compiler community [11]. This work seeks to alleviate
the issue by designing higher-level IR components that are more conducive to transformations.
3.6.2 Top-down: Orchestrating Transformations. A higher-level IR facilitate the declarative spec-
ification of transformations: most transformations target individual operations in the IR, rather
than large multi-operation constructs such as loops, making it easy to specify transformation
targets. Notably, tiling, fusion and unrolling apply to high-level operations rather than loops. 8
Loops and other constructs may be produced as a result of transformations, but they rarely need
to be targeted by further high-level transformations. On the other hand, target specifications can
be arbitrarily complex yet readable if expressed, e.g., using the pattern-matching infrastructure
available in MLIR such as the PDL dialect. Encoding operational knowledge directly in the IR
allows us to design transformations that are legal by design. The fact that our fundamental units of
IR rewrites encompass more IR variety than loops, together with generic properties of data types
(values or side-effects, dense or sparse, etc.) offers additional genericity and extensibility benefits,
and applicability at both high level (e.g.linalg.generic, vector.contract) and low level (numerous
canonicalizations on scalar and vector operations, folding to yield values across loops, etc.).
Furthermore, in presence of a meta-programming dialect suitable for IR manipulation, it becomes
possible to express the transformations entirely declaratively as yet another MLIR dialect. Optimiz-
ing transformations can be then stored, analyzed and transformed, and shipped separately from the
main compiler. It provides a way to specify transformations and the units of IR they manipulate and
produce, while enabling local pattern rewrites almost everywhere. This is paramount to quickly
deliver performance improvements when new hardware becomes available without having to
update the entire compilation stack.
This declarative approach also facilitates the design of custom passes by selecting specific
rewrite rules. This allows mixing transformations, canonicalizations, constant folding and other
enabling rewrites in a single transformation. The result is a system where pass fusion [ 11] is
simple to achieve and alleviates phase ordering issues. Indeed, our structured and retargetable
code generation approach is deliberate about extending the notion of passes with more flexible and
controlled application of rewrite rules . The reader familiar with the program synthesis literature may
think of these transformations as resembling TASO [30], while additionally encompassing tiling,
8Some transformations such as explicit distribution or software pipelining remain naturally attached to loops.
18 Vasilache et al.
fusion, interchange, padding, packing, vectorization and the ability to decompose operations into
smaller operations.
Specifying transformations as patterns expressed in an IR also supports search over possible
transformations. A sufficiently advanced search mechanism can analyze the declaratively specified
patterns for which the transformation applies and build the search space from mutually exclusive
choices. Patterns can also be defined in a parametric form (e.g., tile sizes are not hardcoded in the
pattern) and instantiated at transformation time. The search procedure then consists in trying
different sequences of compatible transformations with different parameters, resulting in different
compilation strategies .
The multi-level nature of MLIR makes it possible to build higher-level dialects to define IR
transformations superimposed on the existing infrastructure and handled by progressively lowering.
For example, the transformation sequence and some of the parameters can be reified into a new
“strategy” operation that gets lowered into primitive transformation operations with the lowering
also specified declaratively. Just as with any other dialect, IR modules using such meta-programming
dialects can be created programmatically from any language. The textual or binary IR format can
be used to communicate between the front-end language, in which the transformation is written,
and the compiler infrastructure with loose coupling. The expressiveness of such transformations is
similar to that of RISE/ELEVATE [25] but without restrictions to the specification language.
3.6.3 Bottom-Up Discussion: A Thick Blanket of High-Performance Patterns. The example discussed
in Section 3.5 is just a tiny slice of the various rewrite patterns we have been developing. Applying
sets of orthogonal low-level pattern rewrites results in desirable larger scale behavior. Application
of these patterns is often bottom-up and are akin to assembling low-level building blocks into
larger building blocks. The patterns of interest encompass canonicalization, folding and other
enabling patterns, as well as vector lowering patterns from higher-dimensional vector forms to
hardware specific vector forms. For example, decomposing a 2-Dvector.contract into unrolled 1-D
outer product microkernels for x86 CPUs supporting broadcast-load vector instructions. Individual
patterns at this level have few parameterization options, often limited to a choice among a few
variants (e.g. lower a higher-dimensional vector reduction by using horizontal reductions or
transpositions). The application of individual patterns is often unsurprising and the performance
of the resulting IR is quite predictable prior to their application. These types of patterns bear
resemblance to lower-level ones in the LLVM back-end but operate on higher-level retargetable IR.
At this time we have started exploring the application of search techniques to build parametric
and composable pattern sets. One direction of interest is to aggressively search for tuples of
(patterns,parameters,op-dag matching constraints ,problem sizes).
that reach a high fraction of peak hardware performance, given a fixed set of LLVM compiler flags.
Thanks to our hierarchical IR design, we believe any such tuple can be saved and applied, with
a low degree of surprise. We expect this part of the system to make heavy use of the upcoming
PDLL infrastructure [43] to succinctly encode the search results and ship it to production. Future
considerations will include how such patterns should be managed, updated, curated and prioritized.
4 SINGLE THREAD CPU EXPERIMENTS
We evaluate our code generation framework on a small set of kernels of high relevance to the
machine learning community. All benchmarks measure single threaded CPU performance and
compare to the peak performance of the machine.
Composable and Modular Code Generation in MLIR 19
4.1 Preamble: Driving Experiments
MLIR provides a set of bindings for Python that support IR creation and manipulation at a generic
level.9 The infrastructure described in this paper aims to facilitate multi-level metaprogramming
and drove the design of these bindings.
We additionally provide a custom domain-specific language (DSL) embedded into Python, referred
to as OpDSL. The purpose of OpDSL is to shift the API paradigm from constructing a compiler IR
to expressing computation in a concise, human-readable and mathematically compelling form, that
has shown successful with Tensor Comprehensions [55]. Figure 11 illustrates a type-polymorphic
matrix multiplication in OpDSL.
@linalg_structured_op
def matmul(A=TensorDef(T1, S.M, S.K), B=TensorDef(T2, S.K, S.N), C=TensorDef(T3, S.M, S.N, output=True)):
C[D.m, D.n] += cast(T3, A[D.m, D.k]) * cast(T3, B[D.k, D.n])
Fig. 11. Example: type-polymorphic matrix multiplication in OpDSL
# Compilation experts can be defined by chaining transformation classes with each other or with experts.
SingleTilingExpert = Tile.then(Generalize).then(Vectorize).then(Bufferize).then(LowerVectors).then(LowerToLLVM)
DoubleTilingExpert = Tile.then(SingleTilingExpert)
TripleTilingExpert = Tile.then(DoubleTilingExpert)
# Compilation experts can be parameterized by a union of options for the transformations they comprise.
concrete_double_tiling = DoubleTilingExpert(
sizes1=[32, 32], sizes2=[8, 4], pad2=True, vectorize_padding=True, contraction_lowering='outer')
Fig. 12. Example: defining tiling transformations in Python
The flow leverages and extends the minimal MLIR execution engine for JIT-compilation and
execution. Structured data objects processed by the flow are exposed in Python as objects compatible
with the Python buffer protocol. They can therefore be converted to and from NumPy arrays, which
are further convertible to framework-specific data types. More details are given in Appendix A.4.
In addition, we provide a test and benchmarking harness in Python to automate the measurement
of compilation- and run-time as well as performance numbers such as GFLOP/𝑠 for computation
and GB/𝑠 for memory traffic. The harness also wraps around the compilation and execution with
multiple transformation strategies as described below.
4.2 Preamble: Driving Transformations
The transformations described in the previous sections are provided as configurable and callable
“transformation objects” in Python. These objects can be applied to MLIR modules and perform the
specified transformations on them. Under the hood, the transformation results in a custom pass
pipeline, consisting of the specified transformation pass with options set up as well as a handful of
properly configured enabling/cleanup passes. The pass pipeline is then run on the module.
The transformations currently available in Python are listed in Table 1. Certain transforma-
tions are applied in combination with tiling. As discussed previously, the general notion of multi-
dimensional subset — applied to tensors, vectors, memrefs — is load-bearing in our approach. Trans-
formations initially introduced on classical loops are generalized to multi-dimensional structured
operations with the explicit goal of preserving structure. Transformations may not be applicable
until some loops are materialized by tiling as a means of explicitly and selectively discarding some
parts of the structure. In turn this improves the composition of transformations written as patterns:
the matching conditions do not trigger until prerequisites are met. For example, interchange applies
9https://mlir.llvm.org/docs/Bindings/Python
20 Vasilache et al.
to the loops produced by tiling around the target structured operation. Similarly, peeling only occurs
on partial tiles. Peeling and padding (Section 3.2) are used as a means to ensure the main operations
become fixed-shape and more easily amenable to vectorization (Section 3.3).
Transformation Options
Tile sizes — array of tile sizes
interchange — order of loops after tiling
pad — whether to pad partial tiles
pack_paddings — non-removable padding for arrays
hoist_paddings — number of loops from which to hoist padding for arrays
peel — loops from which to peel off partial tiles
scalarize_dyn_dims — whether to emit scalar code for non-vectorizable (dynamic) di-
mensions
Vectorize vectorize_padding — whether to vectorize pad operations
PipelineOneParentLoop parent_loop_num — which parent loop to pipeline
II — Iteration Interval
read_latency — Latency of a read operation
UnrollOneParentLoop parent_loop_num — which parent loop to unroll
unroll_amount — by how many iterations to unroll the loop
UnrollOneVectorOp source_shape — source shape of the vector op to unroll
source_shape — target shape to unroll the vector op to
Bufferize none
Sparsify none
LowerVectors contraction_lowering — how to lower vector contractions (outer/inner product, LLVM
matrix intrinsics)
multi_reduction_lowering — how to lower multidimensional reductions (inner or outer
parallel)
transpose_lowering — how to lower transpositions (elementwise, flat, vector shuffle,
target-specific)
LowerToLLVM none
Table 1. Transformations (and their options) that are currently available in Python. Some loop transformations
apply only to loops produced by tiling and must be combined with tiling (e.g., peeling or interchange).
Benchmark L1 @ 12.8KB L2 @ 20% L2 @ 40% L2 @ 90% L3 @ 40% L3 @ 80% DRAM
copy (1 read + 1 write / B) 289.5 89.3 83.9 54.8 25.7 17.2 12.2
extrapolated
(2 reads + 1 write / B) 434.25 134 125.8 82.2 38.5 25.8 18.3
Table 2. Measured and extrapolated single core copy performance in GB/𝑠.
The transformations listed in Table 1 latch on a structured unit of the IR, with additional infor-
mation and constraints. E.g. for theUnrollOneVectorOp case, the structured unit isvector.contract,
with additional constraints on its current shape, and providing the target shape after transformation.
Note that this direct control of multi-dimensional structured operations is not available in IRs that
treat loops as the unit handle for transformations [9, 41, 55].
To complete the environment to control and run transformation and code generation experiments,
we provide a computation chaining API to define transformation sequences, see Figure 12.
Composable and Modular Code Generation in MLIR 21
4.3 Experimental Setup
Experiments are run on an Intel Xeon Gold 6154 CPU @3.00GHz. This is a processor with dual-issue
AVX512 fma instructions and 32KB of L1D, 32KB of L1I, 1MB L2 cache per core and a unified 25MB
L3 shared by 18 cores. Single-threaded single precision computations peak at 192GFLOP/𝑠 (2 fma
operations per cycle, 2 operations (mul + add) each on 16 f32). The theoretical L1 bandwidth of a
single core is 384GB/𝑠 (assuming 1 load and 1 store instruction per cycle, each one of 64B).
Measurements are nearly bare-metal, in privileged mode, following scientific computing [27]
and LLVM benchmarking recommendations [16]. In particular, we disable turbo boost, address
space randomization and the SMT pair of the core we run on. We also run in a specific shielded
cpuset consisting of a single core and migrate all processes away from the core we execute on.
We further measure a peak memory bandwidths given in Figure 2 with a simple contiguous copy
benchmark. We report the highest measured L1 throughput in GB/𝑠 which we find by fine-tuning,
this maximum occurs for 12.8KB read buffer size (i.e. 25.6KB total buffer sizes, or approximately
80% of L1 capacity). Allocations occur at 64B boundary to guarantee the absence of cache line
splitting given the sizes and transformations considered for this peak bandwidth analysis. Since
the hardware can issue 2 loads and 1 store per cycle, we also extrapolate the ideal scaling of the
measured bandwidth to the actual load/store mix of a given scenario (i.e. by mechanically scaling
the peak bandwidth by up to 50% over the one measured with the copy benchmark). Depending
on the benchmark, the roofline [57] is either the measured copy bandwidth (e.g. for transpose or
reduction) or the extrapolated bandwidth (e.g. for depthwise convolution that perform multiple
reads for 1 write).
In the following, all experiments consist of single threaded execution times measured on our
benchmark system. We perform 100 measurements and plot the median. Black error bars show the
25% and 75% quantiles to quantify the measurement variance. We report steady-state performance
after dropping warm-up iterations due to compulsory cache misses and other overheads.10 Such
overheads are relevant in larger-scale experiments in which fusion would be required to keep the
L1 hot. We compare the benchmark results to reference implementations to guarantee correctness.
4.4 Benchmarks
We evaluate the effectiveness of the strategies developed within our infrastructure on a range of
kernels that dominate the execution time of machine learning workloads. All kernels execute a single
tensor algebraic operation. Our results highlight the raw operator performance, independently of
fusion and layout optimization opportunities across multiple operations.
We distinguish between memory-bound and compute-bound kernels. The memory-bound kernels
move and reorder data to match the access patterns of the more compute intense operations. We
benchmark the following memory-bound kernels:
•Copy performance is an essential performance metric. The Copy2D benchmark operates on
2-D tensors that store contiguous data. This setup allows us more flexibility in targeting
tiling, vectorization and unrolling than a flat 1-D buffer but is otherwise equivalent.
•Transposition is a ubiquitous operation that comes in different shapes and sizes. The Trans-
pose2D benchmark implements a 2-D transpose. It is the common denominator of higher
dimensional transpose operations since a 𝑛 > 2-D transpose can be rewritten as iterated
2-D transposes at various locations within the tensor. For instance a 4-D 𝑖,𝑗,𝑘,𝑙 − →𝑘,𝑗,𝑙,𝑖
transposition can be rewritten as 𝑗 ×𝑘 transposes of (𝑖,...,𝑙 − →...,𝑙,𝑖 ). This is always
possible while keeping the fastest varying dimensions contiguous for both input and output
tensors.
10Note that with AVX-512, due to throttling issues, significant warmup may be required.
22 Vasilache et al.
•Reductions are data aggregation operations and an important algorithmic motif. Here we
focus on the bandwidth-bound reductions associated with matrix-vector products or similar
operations occurring in data analytics and neural networks. The ColRed2D and RowRed2D
benchmarks reduce the rows or columns of a 2-D tensor to a 1-D tensor, respectively.
Compute-bound kernels have significant reuse and exhibit much higher computational than
memory bandwidth needs. Their execution time is thus limited by the compute throughput rather
than the memory bandwidth. We benchmark the following compute-bound kernels:
•Matrix multiplication is ubiquitous in numerical computing. The Matmul benchmark imple-
ments plain matrix multiplication.
•Convolutions (1-D and 2 −𝐷 with strided and dilated variants) dominate the execution time
of many machine learning models. In this paper we focus on the so-called NHWC format but
other formats are trivial to generate with our OpDSL approach.
Lastly, we discuss the performance of depthwise convolutions for sizes relevant to the popular Mo-
bileNet [46] model. DepthwiseConv2D is a NHWC format kernel whose compute to communication
ratio presents challenging and interesting problems.
For each benchmark, we manually derive up to 5 expert compiler strategies using the transfor-
mations of Table 1. In each case, we run a few manual experiments to devise good register tile sizes
such that the performance of L1-resident kernels is high. We then fix the tile sizes and select the
best performing strategy among the 5 in each case. This is akin to a fixed expert-driven heuristic.
Systematic autotuning and search space exploration is an area of active investigation that is
expected to bring significant improvements. We discuss preliminary results with autotuning in
Section 4.9.
4.5 Performance of Memory-Bandwidth-Bound Kernels
A bandwidth-bound kernel may be limited by different levels of the memory hierarchy, depending
on problem size and access pattern. We run the bandwidth-bound kernels on different problem
sizes and analyze their performance for all three cache hierarchy levels. In the particular case of
the copy benchmark, we run a small search over different 2-D vector sizes, load/store interleaving,
and loop unrolling specific for every problem size. The measured bandwidth then becomes our L1
bandwidth measuring stick.
4.5.1 L1 Bandwidths. Figure 13 shows the achieved memory bandwidth for all bandwidth-bound
kernels on problem sizes fitting L1 cache. We observe the highest bandwidths for the Copy2D
kernel that performs exactly 1 vector.load and 1 vector.load per 64B of data. This kernel does
not perform any computation nor rearranges data. Despite executing in a tight loop with data
resident in L1 cache, the benchmarks show that offsetting latency requires a sufficiently large
problem size. We start seeing L1 bandwidth performance in the 200 GB/s zone only at around 4KB
of read data (8KB total, i.e. 25% L1 capacity). A 200GB/s L1 bandwidth is sustained only in the
8KB–14KB range of read data (16KB–28KB total, i.e. 50-87% L1 capacity). We observe the maximal
bandwidth of 289 GB/s at roughly 75% L1 cache utilization. Larger problem sizes achieve lower
bandwidths, presumably, due to conflict misses. Lastly, variance starts increasing greatly around
80% of L1 capacity.
Transpose2D rearranges the data while moving. This results in significantly more complex
instruction sequences than a Copy2D’s simple 1 load 1 store pattern. We achieve bandwidths of
30-60% of Copy2D and up to 109 GB/s L1 bandwidth for the most favorable size. We discuss the
characteristics of this kernel more deeply in Section 4.5.4 as well as opportunities for improvement.
ColRed2D and RowRed2D read more data (a full 2-D tensor) than they write (a 1-D tensor). This
is beneficial on our test system that can perform 2 reads and 1 write per cycle (see Table 2). At the
Composable and Modular Code Generation in MLIR 23
10,32 10,48 20,32 8,96 20,48 12,96 10,128 16,96 12,144 110,16 16,128 64,32 20,128 20,144 90,32 190,16
Problem Size [m,n]
0
50
100
150
200
250
300Bandwidth [GB/s]
94.8
116.4
150.6
166.1
167.0
180.7
182.9
227.6
230.4
234.7
230.8
248.2
276.8
271.1
284.4
289.5
53.3
66.2
80.0
87.8
86.3
93.1
75.3
100.7
92.8
80.5
105.7
110.0
97.1
101.5
99.7
90.7
51.2
82.3
90.8
120.0
124.2
149.3
166.1
172.8
187.5
135.2
200.3
183.7
220.9
218.5
203.0
178.1
28.3
34.5
39.4
50.6
45.5
65.3
80.0
72.1
77.9
37.7
95.6
52.4
103.0
88.5
54.9
40.1
Copy2D Transpose2D ColRed2D RowRed2D
Fig. 13. Memory bandwidth for bandwidth-bound kernels on problem sizes fitting L1 cache. Theoretical copy
peak bandwidth is 384 GB/s (289 GB/s measured). Transpose performance is limited by xmm loads and ymm
shuffles, more work is needed to get good zmm assembly (see Section 4.5.4).
same, time they perform computations. In particular RowRed2D performs an expensive horizontal
reduction along the vectorization dimension. While ColRed2D achieves high bandwidths of up to
212 GB/s, we observe only up to 99 GB/s for RowRed2D. The lower performance is particularly
pronounced for problem sizes with a short reduction dimension. This is due to the mapping of
horizontal reduction on Intel processors and the notorious cost of crossing ymm and xmm boundaries.
The resulting performance is not better than combining Transpose2D and ColRed2D. Finer tuning
and better AVX-512 patterns are expected to improve the situation in the future.
Overall, rearranging data or performing computation during data movement immediately reduces
the achieved L1 cache bandwidth.
4.5.2 L2 Bandwidths. Figure 14 shows the achieved memory bandwidths for problem sizes fitting
L2 cache. ColRed2D and RowRed2D achieve the highest bandwidths due to their favorable read-to-
write ratio. We observe bandwidths of up to 125 GB/s.RowRed2D achieves similarly high bandwidth
but also falls behind for problem sizes with small vector dimension, due to the slow reduction of
the last vector in the row. The difference between Copy2D and Transpose2D is less pronounced
since the cost for rearranging the data partly overlap with the slower data movement.
4.5.3 L3 Bandwidths. Figure 15 shows the achieved memory bandwidths for problem sizes fitting
L3 cache. All benchmarks except for Transpose2D peak at an achieved memory bandwidth of 26
GB/s. We attribute this performance difference to the L3 cache latency of that cannot be hidden for
the transpose access patterns.
4.5.4 Transpose Discussion. In this section, we dive deeper into the specific performance of the
transpose patterns we generate and discuss opportunities for improvements.
24 Vasilache et al.
12,960 16,960 50,32012,14401100,1620,960160,160900,32200,160160,240200,240160,320200,3206000,16200,6408000,16
Problem Size [m,n]
0
20
40
60
80
100
120
140
160Bandwidth [GB/s]
84.9
83.2
81.6
85.6
89.3
81.2
84.0
85.6
83.8
81.3
85.9
84.4
82.3
79.5
51.8
68.9
62.3
58.9
44.4
62.2
36.0
50.1
64.9
45.0
43.6
66.3
51.2
62.7
46.8
50.3
34.2
42.7
142.6
140.8
130.8
145.6
151.0
130.8
136.8
136.8
146.9
137.8
138.4
138.8
142.5
139.0
129.1
137.4
96.6
97.4
76.6
100.2
41.5
98.5
87.7
57.8
88.0
107.0
103.8
105.0
105.7
42.5
104.7
42.0
Copy2D Transpose2D ColRed2D RowRed2D
Fig. 14. Memory bandwidth for bandwidth-bound kernels on problem sizes fitting the L2 cache. Measured
peak copy bandwidth is 83.9GB/𝑠 (1 read and 1 write per byte). Reductions can go past this because they
perform an amortized 2 reads and a fraction of a write per iteration. Despite the extra compute, dropping
most writes is still a net win.
First, to get to the best transposed version, the Intel Reference Optimization Manual [12] rec-
ommends using the vblendps instruction (example 15.19). The vector dialect provides a dedicated
vector.transpose operation as well as multiple lowering strategies when going to LLVM. Since
we emit LLVMIR, we do not have direct control over register allocation and instruction selection.
To offset this lack of control, hardware-specific vector dialects (e.g. the x86vector dialect) provide
access to intrinsics to match clang intrinsics such as_mm256_blend_ps. Unfortunately, some of these
clang intrinsics (including _mm256_blend_ps) are not backed by a real intrinsic implementation and
do not provide direct access to the corresponding ISA. Instead, they lower immediately to generic
LLVM shufflevector instructions! LLVM then relies on peephole optimization and especially Se-
lectionDAG; but these contain “very little arch-specific shuffle combines” [53]. In such a case, we
were not able to compile to the desired vblendps operation. Instead, we had to originally settle to a
pure shuffle-based implementation11.
To circumvent this issue, we also provide an InlineAsmOp and specific asm-based intrinsics that
encode the intended instructions (e.g. anmm256BlendPsAsm lowering helper that isguaranteed to emit
the vblendps instruction where we intend it to)12. This provides a nice 30-40\% performance boost
on small transpose sizes over the version generated by LLVM for an 8x8 transpose implemented
only with intrinsics that lower to shufflevector.
11Note that such a pure shuffle-based implementation is similar to the one used in the Eigen [22] library (more specifically,
the ptranspose(PacketBlock<Packet8f,8>& kernel) routine).
12We provide such specialized avx2 lowerings under thetranspose_lowering=avx2 for vector transpose of size 8x8.
Composable and Modular Code Generation in MLIR 25
250,9605500,64100,38403000,128250,1920100,57609500,64250,25601000,640250,38401000,960800,12809500,1281000,128012000,12816000,128
Problem Size [m,n]
0
20
40
60
80Bandwidth [GB/s]
26.1
25.7
25.7
25.6
25.7
25.7
25.7
25.7
25.7
25.7
25.7
25.7
25.4
25.7
25.6
25.5
16.8
12.5
19.0
13.5
16.6
18.7
16.9
17.2
16.4
16.8
16.9
19.5
13.0
15.1
16.1
15.3
74.3
33.2
31.5
29.7
26.7
26.2
25.6
25.9
25.7
25.9
25.7
25.7
25.6
25.7
25.7
25.7
63.1
32.1
30.8
29.2
26.6
25.6
25.9
25.6
25.6
25.5
25.6
25.6
25.6
25.6
25.6
25.6
Copy2D Transpose2D ColRed2D RowRed2D
Fig. 15. Memory bandwidth for bandwidth-bound kernels on problem sizes fitting L3 cache. Measured copy
peak is 25.7 GB/s (L3 @40%). Note that reductions of 250x960xf32 still fit in L2 because the write buffer is
much smaller (i.e. 250xf32 for row reduction and 960xf32 for columns reduction.
The Intel Reference Optimization Manual [12] further recommends using specific permutation
instructions from memory to further reduce shuffle issue port pressure. These are not yet available
as first-class citizens in our lowering but LLVM’s peephole optimization was able to recover the
pattern in the very specific case of a 4x8 tiling of the transpose. While this does not make use of
avx512 instructions and is thus potentially 2x slower than we could hope for, the LLVM4x8 version
performs best, by far. We summarize our findings in Table 3.
Size Tile8x8Shuffle Tile16x16Shuffle Tile8x8AVX2 Tile4x8Shuffle
16x16 24.1 22.5 41.8 55.3
32x32 29 27 64 95
Table 3. Median (p50) measured performance of a vector.transpose lowering strategies (GB/s). The "natu-
ral" 16x16 native AVX512 lowering withshufflevector performs significanly worse than a custom8x8 AVX2
lowering mixing vblendps and shufflevector. The less obvious 4x8 tiling and shufflevector performs
significantly better despite using xmm loads.
Based on this understanding, we conducted additional experiments using UnrollOneVectorOp to
try and keep a 16x16 vector.transposeshape while forcing xmm and ymm loads. This experiments re-
sulted in a performance degradation, likely due to theshufflevector that we do not yet canonicalize.
More work will be needed to get to the bottom of this.
The performance has room for future improvements as we devise a better-suited AVX512 version
and more finely tune tile sizes and codegen strategies. We expect a better AVX512 solution will
26 Vasilache et al.
likely involve insight from prior work on Knight’s Landing [51]. For now, the measured efficiency
is between 30% and 55% for a fully isolated2-D transpose operation. This has to be put in additional
context. The copy kernels we studied in the previous section perform exactly 1 64B load and 1
64B store for each operation. The transpose kernels are significantly more complex. They involve
16B and 32B loads but still reach a comparatively high performance, given the small loads and
additional shuffle instructions.
Lastly, an additional argument to consider is that transpositions are by themselves rarely isolated
in a vacuum. Instead, they often compose with other operations (e.g. matmul), in which case their
cost is often amortized.
4.6 Performance of Compute-Bound Kernels
Let us now review the performance of matrix multiplication and convolutions.
4.6.1 Matrix Multiplication. Attaining a high matrix multiplication throughput is essential in nu-
merical computations. We measure the performance for plain matrix multiplication and transposed
variants:
𝐶[𝑚,𝑛]= 𝐴[𝑚,𝑘]𝐵[𝑘,𝑛] ( 𝐴𝐵)
𝐶[𝑚,𝑛]= 𝐴[𝑘,𝑚]𝐵[𝑘,𝑛] ( 𝐴𝑇𝐵)
𝐶[𝑚,𝑛]= 𝐴[𝑚,𝑘]𝐵[𝑛,𝑘] ( 𝐴𝐵𝑇)
18,32,96 24,64,96 48,64,128 192,64,128192,128,128480,512,16384,256,256784,128,512480,512,256
1020,1152,11521920,2304,2304
Problem Size [m,n,k]
0
25
50
75
100
125
150
175
200Throughput [Gflop/s]
164.3
176.9
166.0
158.3
160.5
117.1
151.9
138.7
152.4
161.2
153.1
176.4
173.0
175.3
175.9
176.3
114.7
167.5
159.9
156.7
164.1
156.2
73.4
74.2
101.4
127.6
134.1
90.0
141.9
134.1
148.2
159.8
154.4
ATB AB ABT
Fig. 16. Matrix multiplication compute throughput for different storage layouts and problem sizes (best of 5
fixed strategies). The theoretical peak is 192GFLOP/𝑠, fine peformance tuning is left for future work.
Figure 16 illustrates the performance of matrix multiplication for various sizes. We reach a
92% efficiency in the AB kernel case, which demonstrates that our codegen approach emits close
to peak arithmetic intensity kernels. It is worth noting that in the low-latency regime cases,
Composable and Modular Code Generation in MLIR 27
layout has a significant impact on performance. In particular, the 𝐴𝐵𝑇 has a significantly lower
performance due to the layout of reduction dimension along the fastest varying memory dimension
(i.e. 𝐶[𝑚,𝑛]= 𝐴[𝑚,𝑘]𝐵[𝑛,𝑘]). This is a similar horizontal reduction issue as for RowReduction2D.
At low-latency sizes, the cost of a transpose is prohibitive and performance suffers. As we reach
larger sizes, the tranpose becomes beneficial and no more performance difference is observable.
Larger sizes use a fixed tiling of 288x128x512 but are not specifically tuned. Additionally, we
do not yet emit prefetch instructions or try to pipeline data movements with computation. These
transformations along with deeper tuning is left for future work. For a preview discussion of
achievable performance combining autotuning with our infrastructure, see Section 4.9.
4.6.2 Convolution. Convolution folds an input tensor with a multi-dimensional kernel following a
sliding window pattern. We specify a convolution operator in terms of its input image and kernel
dimensions following standard practice in the ML literature:
•𝐻: height of the image,
•𝑊: width of the image,
•𝑁: batch number of the image (input and output only),
•𝐶: input channels of the image,
•𝐹: output filters of the image (kernel only),
Additional parameters are the kernel widths 𝐾ℎ and 𝐾𝑤, the strides 𝑆𝑤 and 𝑆ℎ, and the dilations
𝐷𝑤 and 𝐷ℎ. We measure the performance for 1-D and 2-D convolution in NHWC format:
𝑂[𝑛,𝑤,𝑓 ] = 𝐼[𝑛,𝑤 ×𝑆𝑤 +𝑘𝑤 ×𝐷𝑤,𝑐]·𝐾[𝑘𝑤,𝑐,𝑓 ] ( 1 −𝐷)
𝑂[𝑛,ℎ,𝑤,𝑓 ] = 𝐼[𝑛,ℎ ×𝑆ℎ +𝑘ℎ ×𝐷ℎ,𝑤 ×𝑆𝑤 +𝑘𝑤 ×𝐷𝑤,𝑐]·𝐾[𝑘ℎ,𝑘𝑤,𝑐,𝑓 ] (2 −𝐷)
where the stride (𝑆ℎ and 𝑆𝑤) and the dilations (𝐷ℎ and 𝐷𝑤) parameters control the input image
access pattern.
Figure 17 shows the compute throughput for the 1-D (left) and the 2-D (right) convolution
after tiling once. All problem sizes not shown in the plot are constant ((𝑁,𝐶,𝐹,𝐾 𝑤)= (1,32,64,3)
for the 1-D case and (𝑁,𝐶,𝐹,𝐾 𝑤)= (1,32,64,3,3)for the 2-D case). The reader familiar with
the literature may remark that we are providing performance results for the more difficult batch
size 1 configuration. The particularity of the problem is that it contains one fewer dimension of
parallelism.
We measure high performance for both the 1-D and the 2-D convolution when the stride is 1,
with a peak at around 96% of theoretical hardware peak. We observe a slowdown for non-unit
strides. The slowdown is more pronounced in the 1-D case where the data size is quite small and
the computation do not make up for the loss in access pattern efficiency. The inefficiencies start to
dissipate above input size 40 and almost disappear in the 2-D case. Note that sizes 20 and 55 are
not perfect multiples of the good vector sizes and require multi-dimensional loop peeling, resulting
in lower performance (padding is not beneficial at such small sizes)13.
Since this is a heavily compute-bound kernel with multiple parameters, we only focus on the
performance of the vectorized kernel for small sizes resident in L1. Thanks to our modular and
composable approach, larger sizes can be built from smaller sizes, following previously discussed
padding, packing and peeling strategies (see Section 3.2). In particular, the compiler implements a
single 1-D vectorization strategy that exhibits enough arithmetic intensity and is reused in the 2-D
case. To achieve this, the 2-D strategy simply tiles the 𝐻 and 𝐾ℎ dimensions by 1 then folds and
canonicalizes it further. The 2-D case reduces to the 1-D case where high-intensity vectorization
patterns apply.
13Specific size tuning to avoid peeling along some dimensions is expected to further improve the performance in such cases.
28 Vasilache et al.
8 16 20 32 40 55 64
Problem Size [W]
0
25
50
75
100
125
150
175
200Throughput [Gflop/s]
178.1
180.2
149.6
176.1
175.9
153.1
176.9
177.8
180.7
174.9
175.9
176.0
154.5
174.8
138.7
144.0
108.3
141.9
108.8
101.9
105.0
142.9
145.2
108.0
143.2
141.3
101.4
107.3
str.=[1],dil.=[1] str.=[1],dil.=[2] str.=[2],dil.=[1] str.=[2],dil.=[2]
Fig. 17. L1-resident 1-D convolution for different strides, dilations, and problem sizes ((𝑁,𝐶,𝐹,𝐾 𝑤)fixed to
(1,32,64,3)). Theoretical peak is 192GFLOP/𝑠, fine tuning of performance is left for future work.
4.7 Depthwise Convolution
Depthwise convolution is a compute-efficient twist on convolution mainly oriented at mobile
inference. 1-D and 2-D depthwise convolution in NHWC format are expressed as:
𝑂[𝑛,𝑤,𝑐 ] = 𝐼[𝑛,𝑤 ×𝑆𝑤 +𝑘𝑤 ×𝐷𝑤,𝑐]·𝐾[𝑘𝑤,𝑐] ( 1 −𝐷)
𝑂[𝑛,ℎ,𝑤,𝑐 ] = 𝐼[𝑛,ℎ ×𝑆ℎ +𝑘ℎ ×𝐷ℎ,𝑤 ×𝑆𝑤 +𝑘𝑤 ×𝐷𝑤,𝑐]·𝐾[𝑘ℎ,𝑘𝑤,𝑐] (2 −𝐷)
where the strides (𝑆ℎ and 𝑆𝑤) and the dilations (𝐷ℎ and 𝐷𝑤) control the input image access pattern.
Compared to classical convolutions, the volume of the computation kernel is reduced by1 dimension
(typically a 16×to 512×reduction in both compute and data volume). The computation is also
altered to only reduce across the window dimension(s) (and not along the channel dimension).
The depthwise convolution kernel has a low arithmetic intensity ( 𝐾𝑤
sizeof (element_type) FLOP/𝑠 per
byte in the 1-D case and 𝐾ℎ.𝐾𝑤
sizeof (element_type) FLOP/𝑠 per byte in the 2-D case). As a result, depthwise
convolution is memory bandwidth-bound on modern server processors.
Since the kernel is memory-bound, it is important to measure the volume of data properly. In
the following experiments, we evaluate the total data volume as follows (𝐾ℎ and 𝐻 are 1 in the 1-D
case):
Totalsize = Outsize +Kersize + Insize
Π𝑖 gcd(stride𝑖,dilation𝑖)
= 𝑁 ·𝐶·

𝐻 ·𝑊 + 𝐻in ·𝑊in
Π𝑖 gcd(stride𝑖,dilation𝑖)

+𝐾ℎ ·𝐾𝑤 ·𝐶
The conservative reasoning follows:
(1) Every element of the convolution kernel is read.
(2) Every element of the output is written but is not considered read; the data is simply over-
written.
Composable and Modular Code Generation in MLIR 29
8,8 16,16 20,20 32,32 40,40 55,55 64,64
Problem Size [H,W]
0
25
50
75
100
125
150
175
200Throughput [Gflop/s]
184.2
184.4
149.7
183.7
183.5
150.6
175.1
183.9
183.8
174.8
184.0
184.2
151.3
175.0
182.7
182.7
148.5
178.6
168.5
142.2
166.8
182.3
182.2
148.5
181.9
177.8
145.6
170.3
str.=[1,1],dil.=[1,1] str.=[1,1],dil.=[2,2] str.=[2,2],dil.=[1,1] str.=[2,2],dil.=[2,2]
Fig. 18. L1-resident 2-D convolution for different strides, dilations, and problem sizes ((𝑁,𝐶,𝐹,𝐾 ℎ,𝐾𝑤)fixed
to (1,32,64,3,3)). Theoretical peak is 192GFLOP/𝑠, fine tuning of performance is left for future work.
(3) Only a subset of the input is read, depending on the values of the stride and dilations quantities.
This is captured by a scaling adjustment factor.
The scaling adjustment factor is obtained by the intersection of the stride lattice and the dilation
lattice, within the limits of the input boundary. The gcd is a good approximation when the size
of the kernel is greater than the stride. This guarantees we only compute the input points that
participate in the derivation of at least 1 output element. This is a conservative measure: any
element that is unused but falls in a common cache line with a used element is not counted; still
the data will be moved by the hardware.
Figures 19 and 20 show the memory bandwidth for the 1-D and 2-D depthwise convolution.
All problem sizes not shown in the plot are constant ((𝑁,𝐶,𝐾 𝑤)= (1,32,3)for the 1-D case and
(𝑁,𝐶,𝐾 ℎ,𝐾𝑤)= (1,32,3,3)for the 2-D case). In the 1-D case, we measure 45 −75% L1 bandwidth
for the smaller problem sizes compared to Copy2D. As size increases, we see performance close to
the L2 and L3 bandwidth of Copy2D, which signals strong data reuse. For the 2-D case, we limit
the exploration to sizes related to MobileNet layers with batch size 1. We again see performance
within 10 −20% of the peak L3 bandwidth for comparable sizes, suggesting good data reuse.
Lastly, note that our approach scales easily to other kernel sizes, strides and dilations than the
traditional 1 or 2 values that are often the only supported by library implementations (and often
not in all combinations).
4.8 A Brief Evaluation of Sparse Code Generation
We end this experimental evaluation with an example of sparse tensor code generation. As stated
earlier, the sparse_tensor dialect introduces the concept of sparse compilation into MLIR. The
dialect makes sparse tensor types first class citizens by bridging high-level operations on these
30 Vasilache et al.
8,32 16,32 40,32 55,32 64,32 110,32 128,32 16,256 40,256 55,256 96,256110,256128,256256,1024880,10241024,1024
Problem Size [W,C]
0
50
100
150
200Bandwidth [GB/s]
79.1
105.2
147.0
142.9
162.1
158.2
146.5
123.8
85.5
87.4
87.2
86.0
89.6
25.4
25.4
23.5
81.8
108.5
135.8
144.0
149.0
130.9
136.3
124.4
84.2
84.6
85.8
87.5
90.6
24.4
24.7
23.5
99.6
141.6
189.0
208.0
210.8
95.2
102.2
99.2
90.2
96.9
98.9
100.2
100.2
25.6
25.6
25.5
77.2
103.8
142.3
154.3
157.0
102.3
119.1
100.7
85.2
86.2
88.4
88.8
87.7
23.9
24.1
22.8
str.=[1],dil.=[1] str.=[1],dil.=[2] str.=[2],dil.=[1] str.=[2],dil.=[2]
Fig. 19. 1-D depthwise convolution for different strides, dilations, and problem sizes. Measured copy peak
289GB/𝑠 (L1), 89.3GB/𝑠 (L2), 25.7GB/𝑠 (L3 @40%).
sparse tensors types with lower-level operations on the actual sparse storage schemes. To this end,
the sparse tensor dialect introduces a tensor encoding attribute that allows specifying the formats
of the Tensor Algebra Compiler (TACO) [32].
Figure 21 illustrates how to use the the well-known Doubly Compressed Sparse Column (DCSC)
storage format for sparse matrices in a matrix multiplication. A sparse tensor is specified by simply
placing the encoding attribute inside the tensor type. This way of defining sparse tensor formats is
quite powerful. For a single 𝑑-dimensional tensor, the dimension level formats and ordering alone
give rise to 2𝑑 ·𝑑! different formats. This allows annotating a sparse kernel in a such a way that
eventually results in the generation of effective sparse code. Even the output tensor could be made
sparse to save memory when storing the result.
A set of sparse_tensor dialect rewrite rules take care of lowering the kernel to sparse storage
schemes and imperative constructs that only store and iterate over the nonzero elements to perform
the matrix multiplication. Such an approach to automatic sparse code generation was first proposed
in [5, 6] in the context of sparse linear algebra, and later generalized to sparse tensor algebra in [32].
Even though the details of these rewrite rules are outside the scope of this paper, they follow a
similar high-level modular and composable philosophy as discussed in Section 3, with variations in
the transformations due to the use of sparse index sets and slightly more complicated bufferization
due to the compound nature of sparse storage schemes. The rewrite rules similarly interoperate
with linalg, tensor, memref, scf, and vector abstractions, thereby progressive lowering a sparsity
agnostic definition of a kernel into a form that fully exploits the sparsity of tensors as well as all
performance features of the target architecture.
Consider, for example, a matrix times vector computation x = Ab for a general sparse matrix A
(Figure 22). When A is in Compressed Sparse Row (CSR) format, nested scf loops are used to iterate
Composable and Modular Code Generation in MLIR 31
7,7,1024,[1, 1] 14,14,512,[1, 1]14,14,512,[2, 2]28,28,256,[1, 1]28,28,256,[2, 2]112,112,32,[1, 1]56,56,128,[1, 1]56,56,128,[2, 2]
Problem Size [H,W,C,strides]
0
10
20
30
40Bandwidth [GB/s]
39.3
32.3
24.9
20.7
24.4
22.2
21.2
25.0
DepthwiseConv2D
Fig. 20. 2-D depthwise convolution for different strides, dilations, and problem sizes. Measured copy peak
54.8GB/𝑠 (L2 @90%), 25.7GB/𝑠 (L3 @40%). The 7x7x1024 problem takes 16x16x1024xf32=1.05M (input and
output tensors). This slightly exceeds L2. Other sizes are deep in L3 territory and require parallelism to
increase bandwidth (outside the scope of this paper).
over the outer dense dimension, and, by means of an indirection, just the nonzero elements of the
compressed inner dimension.
Size TACO random MLIR random TACO column MLIR column
16,384 x 16,384 3.32 2.93 1.96 1.58
32,768 x 32,768 12.71 12.59 8.12 6.90
65,536 x 65,536 51.49 50.48 32.21 30.48
Table 4. Sparse kernel runtimes (in ms) on an Intel Xeon W2135 3.7GHz.
Now suppose that the sparse matrixA is actually a structured sparse matrix, where most columns
are empty, but when nonzero, the columns are dense, i.e. mostly filled. In that case,CDC makes more
sense, favoring column-wise access over the default row-wise storage, and using compression for
the outermost dimension (now columns) only. Specializing for this format for a sparse 8192 ×8192
matrix and a vector length of 16, the sparse rewriting rules compose well with vectorization to
eventually yield sparse vector code that skips over empty columns, but performs the innermost
dense update using vectors. After more progressive lowering within MLIR, eventually an LLVM IR
representation is handed off to the backend compiler which, when targeting AVX512, generates an
innermost loop that performs the vector updates in full SIMD.
Table 4 shows the runtime (in ms) on an Intel Xeon W2135 3.7GHz, comparing sparse computa-
tions generated by both TACO and MLIR, evaluated on sparse matrices with a random uniform
32 Vasilache et al.
#DCSC= #sparse_tensor.encoding<{
dimLevelType = [ "compressed", "compressed"],
dimOrdering = affine_map<(i,j) -> (j,i)>
}>
#CSR = #sparse_tensor.encoding<{ dimLevelType = [ "dense", "compressed"] }>
// Dense linalg.matmul on tensors
%0 = linalg.matmul ins(%a, %b: tensor<10x20xf32>, tensor<20x30xf32>) outs(%c: tensor<10x30xf32>) -> tensor<10x30xf32>
// Same matmul with SpMM kernel
%0 = linalg.matmul ins(%a, %b: tensor<10x20xf32, #CSR>, tensor<20x30xf32>) outs(%c: tensor<10x30xf32>) -> tensor<10x30xf32>
// Same matmul with SpMSpM kernel
%0 = linalg.matmul ins(%a, %b: tensor<10x20xf32, #CSR>, tensor<20x30xf32, #DCSC>) outs(%c: tensor<10x30xf32>) -> tensor<10x30xf32>
simply add a sparse attribute to the tensor type
Fig. 21. Definition of sparse tensor layouts and their usage in a matrix multiplication.
%0 = linalg.generic #matvec
ins(%argA, %argb: tensor<?x?xf64, #???>, tensor<?xf64>)
outs(%argx: tensor<?xf64>) {
^bb(%A: f64, %b: f64, %x: f64):
%0 = arith.mulf %A, %b : f64
%1 = arith.addf %x, %0 : f64
linalg.yield %1 : f64
} -> tensor<?xf64>
scf.for %i = %c0 to %dim_astep %c1 {
%xi = memref.load %x[%i] : memref<?xf64>
%lo = memref.load %A_pointers[%i] : memref<?xindex>
%i1 = arith.addi %i, %c1 : index
%hi = memref.load %A_pointers[%i1] : memref<?xindex>
%acc = scf.for %jj = %lo to %hi step %c1 iter_args(%acc_in= %xi) -> (f64) {
%j = memref.load %A_indices[%jj] : memref<?xindex>
%aij = memref.load %A_values[%jj] : memref<?xf64>
%bj = memref.load %b[%j] : memref<?xf64>
%m = arith.mulf %aij, %bj : f64
%a = arith.addf %acc_in, %m : f64
scf.yield %a : f64
}
memref.store %acc, %x[%i] : memref<?xf64>
}
%lo = memref.load %A_pointers[%c0] : memref<?xindex>
%hi = memref.load %A_pointers[%c1] : memref<?xindex>
scf.for %jj = %lo to %hi step %c1 {
%j = memref.load %1[%jj] : memref<?xindex>
%bj = memref.load %b[%j] : memref<8192xf64>
scf.for %i = %c0 to %c8192step %c16 {
%0 = arith.muli %jj, %c8192: index
%1 = arith.addi %0, %i : index
%2 = vector.load %x[%i] : memref<8192xf64>, vector<16xf64>
%3 = vector.load %A_values[%1] : memref<?xf64>, vector<16xf64>
%4 = vector.broadcast %bj : f64 to vector<16xf64>
%5 = arith.mulf %3, %4 : vector<16xf64>
%6 = arith.addf %2, %5 : vector<16xf64>
vector.store %6, %x[%i] : memref<8192xf64>, vector<16xf64>
}
}
#CDC = #sparse_tensor.encoding<{
dimLevelType = [ "compressed", "dense"],
dimOrdering = affine_map<(i,j) -> (j,i)>
}>
#CSR = #sparse_tensor.encoding<{
dimLevelType = [ "dense", "compressed"]
}>
specify tensor layout
#CSR
#CDC
vbroadcastsd zmm0, qword ptr [r13 + 8*rsi]
Loop: vmulpd zmm1, zmm0, zmmword ptr [rdx + 8*rsi - 256]
vaddpd zmm1, zmm1, zmmword ptr [rbx + 8*rsi + 128]
vmovupdzmmword ptr [rbx + 8*rsi + 128], zmm1
...
jb Loop
Fig. 22. Example: lowering of matrix-vector multiplication with sparse input matrix.
density of about 1% (random) and matrices with 1% of the columns filled (column). For TACO,
we used the default settings of the built-in benchmark, but excluded compilation time for a fair
comparison. The table clearly shows that the sparse compiler component of MLIR is on-par with a
state-of-art compiler like TACO, with additional benefits when the transformation of MLIR’s sparse
compiler are composed with other optimizations, such as vectorization.
4.9 Preliminary Autotuning Results
Our collaborators at Nod.AI have been using our modular and composable codegen strategy and
tuning it for specific use cases. They have built an autotuner using CompilerGym [13], a compiler
tuning environment based on the OpenAI Gym [8]. They report fully-automatic preliminary results
on recent Alder Lake CPUs [37] on which they outperform both Intel’s MKL library and the TVM
autotuner on a variety of matrix multiplication sizes relevant to the machine learning community.
When integrated within the IREE [ 14] compiler-based runtime system, and further tuned for a
BERT [15] model on NVIDIA A100 GPUs, they report the fastest end-to-end PyTorch-based [39]
inference numbers [38].
5 RELATED WORK
We chose to focus this related work survey on the compilers that influenced the design of structured
and retargetable code generation. Readers interested in general IR design may refer to [34].
Composable and Modular Code Generation in MLIR 33
5.1 Lessons from ONNX
ONNX (https://onnx.ai) is an open format for the interoperability of ML models and workloads.
As such, it is predominantly driven by the expressiveness requirements of ML, and less by the
considerations of IR design for high-performance code generation.
Similarly to ONNX, we define “semantically charged”, “named” operations matching the needs of
popular ML models. But we do not stop here: we also consider transformations on these operations
as a key part of the design, and we define supporting IR abstractions favoring transformations
over expressiveness where necessary . We also minimize the range of “named” operations by making
them simple declarative configurations of a small set of generic operations. Compared to a typical
numerical library interface (the de facto standard until now), this greatly reduces maintenance
burden and increases portability. Finally, we provide multiple levels of composable abstractions,
with operations on tensors, vectors, memrefs and transformations thereof, not limiting ourselves
to a framework interoperability interface.
5.2 Lessons from XLA
XLA [50] is one of the first post-Theano ML compilers, introduced as a domain-specific optimizing
compiler for TensorFlow. It shines on Google’s Cloud TPU hardware. XLA followed a pragmatic
design process where the compiler is given perfect knowledge of the semantics of each operation.
The set of operations known to the XLA system is called HLOs. XLA code generation consists of
composing C++ functions known as emitters. This approach has two major benefits: (1) transfor-
mations are correct by construction (2) strong performance on specialized hardware such as TPUs.
On the other hand, due to its approach relying on C++ function composition, extending XLA can
involve significant amounts of code duplication. XLA also involves its own set of intermediate
representations, optimization passes, domain-specific abstractions, resulting in the replication of
similar features in contemporary frameworks.
XLA inspired how operations “know their semantics” and “how to transform and lower them-
selves”. Yet the encoding of this information and its use in transformations differ:
(1) Individual HLOs have parameterized yet special-purpose semantics, operating on a single
level of abstraction (tensors), while lower level data representations come with their own set
of operations. As a consequence, HLOs have evolved into a large set of operations whose
semantics intersect. This echoes the operation proliferation problem also exhibited by ONNX.
(2) Over-reliance on perfect op knowledge leads to situations where transformations and ops
end up needing to know about each other’s semantics (e.g. during fusion). Since the transfor-
mations themselves are not simple local rewrite patterns code complexity grows quickly.
(3) XLA lacks a serializable IR that can be inspected, unit-tested and composed with independent
compiler flows. This monolithic design impacts portability. For instance, the TPU and GPU
compilers do not share much code.
5.3 Lessons from Halide, TVM and related languages
Halide [41] is a DSL embedded in C++ that provides a way of metaprogramming an algorithmic
specification in HalideIR, applying transformations declaratively. This allows expert users to
transform and optimize a high level program in tailored ways. Halide, initially targeted the graphics
community but is now more generally applicable. TVM [ 9] is an evolution of Halide into the
machine learning and deep-neural network space, based on HalideIR.
Halide schedules borrow from the decoupling principle of the URUK [ 19] and CHiLL [ 31]
polyhedral transformation frameworks. To a large extent, schedules are declarative, unlike LIFT,
URUK and CHiLL. It is more restricted than polyhedral schedules, capturing rectangular shapes
34 Vasilache et al.
and limited forms of fusion and fine-grain scheduling. This avoids many of the complexities of
polyhedral scheduling and code generation, yet Halide schedules remain powerful enough for its
application domain, and explores a large swath of possible transformations. Halide shines at making
a loop nest optimization methodology accessible to Ω(10 −100)users, at a time when polyhedral
tools are still only accessible to Ω(1 −10)users. It makes heavy usage of canonicalization rules 14
that are also very prevalent in LLVM and MLIR.
While these features are shared with structured and retargetable codegen, Halide diverges from
our approach in a number of ways. Counter-current to our lowering rather than raising principle,
Halide’s front-end uses explicit indexing and arithmetic on scalar values, which makes it difficult
to target BLAS/DNN libraries, including hardware instructions such as GPU tensor cores. TVM
provides a tensorize transformation for that purpose, but it remains ad-hoc to use or rely on
pattern matching and raising abstractions from lower level indexing and iteration. Along the same
lines, reductions and scans are expressed using serial iteration, again requiring pattern matching
before they can be transformed into more efficient, target-specific parallel versions (recursion tree,
atomic operations, etc.). Finally, Halide only operates on its own IR and abstractions, while we
provide a composable set of abstractions, and genericity over a range of data representations.
Fireiron [23], recent work in the Halide family of languages, shares with our work the focus on
decomposing an operation — originally restricted to matrix multiplication — into smaller kernels.
It also enables the substitution of any such resulting kernel with a manually written microkernel
implementation or hardware instructions of arbitrary granularity. In addition, Fireiron provides
a bottom-up approach to performance modeling and extends Halide schedules with explicit data
transfers; both developments will be influential in the future evolutions of our approach.
5.4 Lessons from LIFT and Combinator Languages with Rewriting Rules
LIFT [49] is a system to write and optimized computational kernels based on functional abstractions.
Transformations are represented by means of local rewrites to the IR, including the insertion of
combinators such as map, reduce, zip... as well as decorating these with distribution, parallelization,
vectorization and storage mapping information.
Similarly to LIFT, we make heavy use of local rewrite rules through the MLIR PatternRewrite
infrastructure. There are important differences however:
(1) Our transformations are co-designed with the IR in a structural decomposition approach,
while LIFT rewrites generic nested combinators.
(2) The unit of transformation and tuning is a generic 𝑛-D operation. Compared to LIFT, we
believe this makes the optimization space more structured and tractable for search algorithms,
and this will be the subject of another study.
(3) We make operations generic over the representation of tensor values. This includes vector-
level primitives and side-effecting operations.
(4) Lowering to nested loops over vectors and memrefs is more versatile than decorating com-
binators on tensor values, while ensuring transformation correctness (including lowering
steps) by design of the structural decomposition approach.
LIFT is expected to further influence our design in the future, leveraging experience with LIFT
abstractions for sparse and position-dependent arrays.
Related work on performance portability include the Multi-Dimensional Homomorphisms (MDH)
approach [42]. Emphasizing the algebraic nature of parallelization and distribution rules over tensor
operations, MDH leverages a specialized code generator and facilitates autotuning.
14https://sunfishcode.github.io/blog/2018/10/22/Canonicalization.html.
Composable and Modular Code Generation in MLIR 35
Elevate [24] addresses some of the expressiveness shortcomings of LIFT while establishing solid
foundations for decoupled schedules. The proposed formalization applies to schedules driving the
application of rewrite rules, which is different from Halide’s code-generation-centered API.
In the same family of languages, Glenside [48] introduces “access patterns” as representation
unifying pure tensor operations and storage mapping considerations such as layouts and hardware-
centered memory management. Access patterns are amenable to automatic synthesis and lowering
to coarse-grained hardware instructions.
5.5 Lessons from Tensor Comprehensions
Tensor Comprehensions [55] is a high-level language to express tensor computations with a syntax
generalizing the Einstein notation, coupled to an end-to-end compilation flow capable of lowering
to efficient GPU code. It was integrated with 2 ML frameworks: Caffe2 (the successor to the popular
Caffe [29] framework), as well as PyTorch [39].
The compilation flow combines Halide with a Polyhedral Compiler derived from isl [ 56] and
uses both HalideIR and the isl schedule-tree IR. The compiler provides a collection of polyhedral
compilation algorithms to perform fusion and favor multi-level parallelism and promotion to
deeper levels of the memory hierarchy. Tensor Comprehensions showed that, building on a affine
framework expressive enough to cover the main compositions of tiling, fusion and parallelization
transformations required to reach competitive performance, it is sufficient to expose a few knobs
over predefined affine scheduling strategies to provide strong performance—in the bandwidth
and latency bound regimes. In particular, a simple genetic search combined with an autotuning
framework and affine scheduling strategies reached high performance on sub-graphs or even full
models in the non-compute bound regime.
However, Tensor Comprehensions lacks the IR to reason about lower level rewrites, related to
vectorization, register reuse, unrolling, necessary to perform well in the compute-bound regime .
In particular, it lacks an SSA representation combining higher level abstractions with vector and
register-level ones. It also relies on parameterized affine scheduling strategies, relying on integer
linear programming, which remain difficult to steer towards peak performance [54, 60]. Most of
these issues are naturally addressed in our MLIR code generation flow, including vector abstractions
and the ability to implement a wide variety of transformation and lowering strategies.
5.6 Lessons from Polyhedral compilers
The polyhedral model has been on the cutting edge of loop nest optimization for decades, with
several incarnations in production compilers such as GRAPHITE [52] for GCC and Polly [20] for
LLVM. Although it has proven capable of competing with the best libraries and domain-specific
frameworks, in incarnations like PolyMage [36] and Tensor Comprehensions [55], it has never
been fully adopted into mainstream optimization flows. Some reasons include:
(1) The IR gets more complex than affine schedules, when representing multiple levels of tiling
and parallelism, data transfers, unrolling, etc., and requires complex abstractions such as isl
schedule trees [56].
(2) Scheduling and code generation from general affine representations rely on exponential
algorithms [4, 7, 17, 18, 21, 35, 56].
(3) Affine representations are not composable with the SSA form, upon which most optimiz-
ing compilers are built today; this induces pass ordering conflicts with induction variable
canonicalization, loop-invariant code motion, vectorization, etc. [33].
(4) Expressiveness limitations remain in polyhedral compilers, despite progresses towards inter-
procedural analysis, dynamic control flow, irregular indexing [2, 28, 58].
36 Vasilache et al.
The affine dialect in MLIR addresses these long-standing issues. In particular, it maintains the
IR in the same form — loops with additional constraints on how the bounds are expressed —
throughout the compilation flow. It also embeds the polyhedral representation into the SSA form
through attributes and regions, facilitating the combination of polyhedral and SSA-based transfor-
mations. Yet some of the above-mentioned issues remain, including the algorithmic and software
engineering challenges of building and tuning competitive optimization strategies by means of
affine transformations. Structured operations avoid these problems by operating on a higher level
of abstraction, involving tensor-operation-specific optimizations and lowering strategies instead.
5.7 Comparison With Low-Level Code Generators
On the back-end compiler side, an alternative approach consists in bypassing the LLVM level com-
pletely, relying on non-portable assembly code generation instead [26]. While the performance one
can achieve with perfectly accurate models of the hardware is impressive, such solutions jump over
multiple levels of abstraction, forbidding any reuse of low-level compiler passes and abstractions.
While such low-level bricks can be the ultimate target when decomposing structured operations,
the reliance on non-portable, hand-tuned generators and proprietary tooling stands a bridge too
far. Instead, we are more interested in leveraging program synthesis and superoptimization at the
lower levels of abstraction [10, 40].
6 CONCLUSION AND FUTURE WORK
We presented the composable multi-level intermediate representation and transformations that
underpin tensor code generation in MLIR. This work heavily leverages MLIR’s progressivity
design principle, which it helped to shape by designing and implementing multiple instances of
progressive lowering. The approach is characterized by a transformation-oriented IR design : doing
away with legality analyses and applicability checks on low-level IR, relying systematically on
the progressive decomposition of carefully designed abstractions. The resulting design is modular
and built with optionality in mind; abstractions span data structures and control flow with both
functional (SSA form) and imperative (side-effecting) semantics; they serve as generic building
blocks for retargetable tensor compilers. Transformations are systematically applied as compositions
of declarative patterns. This allows implementing advanced forms of pass fusion, known to alleviate
the dreaded phase-ordering problem. Early single threaded CPU code generation shows strong
performance, even in the absence of systematic tuning.
Multiple extensions and generalizations of this work are actively being pursued:
•Functionally inspired abstractions for parallel semantics on immutable tensors will extend
our results to multi-threaded CPUs and beyond.
•Generalization of the approach to other hardware targets thanks to our retargetable vector
abstractions.
•More powerful vector abstractions including generalized masking and warp-level GPU
programming.
•More advanced data structures and iterators than dense or sparse to enable cross-pollination
with other fields such as databases.
•Autotuning infrastructure to systematically find transformation compositions and parameters
that produce code running at a high fraction of peak, for any hardware.
•Controllable framework for selecting and composing transformation patterns, making pro-
duction compilers effectively extensible with domain-specific code generation strategies.
Composable and Modular Code Generation in MLIR 37
We believe these contributions will further demystify the field of high-performance compilation,
give a high-level of control to users of the system and provide a strong basis on which future
research and near-peak performance production systems will be built.
REFERENCES
[1] Randy Allen and Ken Kennedy. 2001. Optimizing Compilers for Modern Architectures: A Dependence-Based Approach .
Morgan Kaufmann Publishers.
[2] Riyadh Baghdadi, Ulysse Beaugnon, Albert Cohen, Tobias Grosser, Michael Kruse, Chandan Reddy, Sven Verdoolaege,
Adam Betts, Alastair F. Donaldson, Jeroen Ketema, Javed Absar, Sven van Haastregt, Alexey Kravets, Anton Lokhmotov,
Robert David, and Elnar Hajiyev. 2015. PENCIL: A Platform-Neutral Compute Intermediate Language for Accelerator
Programming. In 2015 International Conference on Parallel Architectures and Compilation, PACT 2015, San Francisco, CA,
USA, October 18-21, 2015 . IEEE Computer Society, 138–149. https://doi.org/10.1109/PACT.2015.17
[3] Paul Barham and Michael Isard. 2019. Machine Learning Systems Are Stuck in a Rut. In Proceedings of the Workshop
on Hot Topics in Operating Systems (Bertinoro, Italy) (HotOS ’19) . Association for Computing Machinery, 177–183.
https://doi.org/10.1145/3317550.3321441
[4] Cédric Bastoul. 2004. Code Generation in the Polyhedral Model Is Easier Than You Think. In 13th International
Conference on Parallel Architectures and Compilation Techniques (PACT 2004), 29 September - 3 October 2004, Antibes
Juan-les-Pins, France . IEEE Computer Society, 7–16. https://doi.org/10.1109/PACT.2004.10018
[5] Aart J.C. Bik. 1996. Compiler Support for Sparse Matrix Computations . Ph. D. Dissertation. Department of Computer
Science, Leiden University. ISBN 90-9009442-3.
[6] Aart J.C. Bik, Peter J.H. Brinkhaus, Peter M.W. Knijnenburg, and Harry A.G. Wijshoff. 1998. The Automatic Generation
of Sparse Primitives. Transactions on Mathematical Software 24 (1998), 190–225. https://doi.org/10.1145/290200.287636
[7] Uday Bondhugula, Albert Hartono, J. Ramanujam, and P. Sadayappan. 2008. A practical automatic polyhedral
parallelizer and locality optimizer. In Proceedings of the ACM SIGPLAN 2008 Conference on Programming Language
Design and Implementation, Tucson, AZ, USA, June 7-13, 2008 , Rajiv Gupta and Saman P. Amarasinghe (Eds.). ACM,
101–113. https://doi.org/10.1145/1375581.1375595
[8] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.
2016. OpenAI Gym. CoRR abs/1606.01540 (2016). arXiv:1606.01540 http://arxiv.org/abs/1606.01540
[9] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan
Wang, Yuwei Hu, Luis Ceze, et al. 2018. TVM: An automated end-to-end optimizing compiler for deep learning. In
13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18) . USENIX Association, 578–594.
https://www.usenix.org/conference/osdi18/presentation/chen
[10] Yishen Chen, Charith Mendis, Michael Carbin, and Saman P. Amarasinghe. 2021. VeGen: a vectorizer generator
for SIMD and beyond. In ASPLOS ’21: 26th ACM International Conference on Architectural Support for Programming
Languages and Operating Systems, Virtual Event, USA, April 19-23, 2021 , Tim Sherwood, Emery D. Berger, and Christos
Kozyrakis (Eds.). ACM, 902–914. https://doi.org/10.1145/3445814.3446692
[11] Cliff Click and Keith D. Cooper. 1995. Combining Analyses, Combining Optimizations. ACM Transactions on
Programming Languages and Systems 17, 2 (1995), 181–196.
[12] Intel Corp. 2021. Intel Optimization Reference Manual. https://www.intel.com/content/www/us/en/develop/download/
intel-64-and-ia-32-architectures-optimization-reference-manual.html.
[13] Chris Cummins, Bram Wasti, Jiadong Guo, Brandon Cui, Jason Ansel, Sahir Gomez, Somya Jain, Jia Liu, Olivier Teytaud,
Benoit Steiner, Yuandong Tian, and Hugh Leather. 2021. CompilerGym: Robust, Performant Compiler Optimization
Environments for AI Research. CoRR abs/2109.08267 (2021). arXiv:2109.08267 https://arxiv.org/abs/2109.08267
[14] IREE Developers. 2021. IREE (Intermediate Representation Execution Environment. https://google.github.io/iree/
[15] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional
Transformers for Language Understanding. arXiv:1810.04805 [cs.CL]
[16] LLVM Documentation. 2021. Benchmarking tips. https://llvm.org/docs/Benchmarking.html.
[17] Paul Feautrier. 1992. Some efficient solutions to the affine scheduling problem. Part I. One-dimensional time. Int. J.
Parallel Program. 21, 5 (1992), 313–347. https://doi.org/10.1007/BF01407835
[18] Paul Feautrier. 1992. Some efficient solutions to the affine scheduling problem. Part II. Multidimensional time. Int. J.
Parallel Program. 21, 6 (1992), 389–420. https://doi.org/10.1007/BF01379404
[19] Sylvain Girbal, Nicolas Vasilache, Cédric Bastoul, Albert Cohen, David Parello, Marc Sigler, and Olivier Temam. 2006.
Semi-Automatic Composition of Loop Transformations for Deep Parallelism and Memory Hierarchies. Int. J. Parallel
Program. 34, 3 (2006), 261–317. https://doi.org/10.1007/s10766-006-0012-3
[20] Tobias Grosser, Armin Größlinger, and Christian Lengauer. 2012. Polly - Performing Polyhedral Optimizations on a
Low-Level Intermediate Representation. Parallel Process. Lett. 22, 4 (2012). https://doi.org/10.1142/S0129626412500107
38 Vasilache et al.
[21] Tobias Grosser, Sven Verdoolaege, and Albert Cohen. 2015. Polyhedral AST Generation Is More Than Scanning
Polyhedra. ACM Transactions on Programming Languages and Systems 37, 4 (2015), 12:1–12:50. https://doi.org/10.
1145/2743016
[22] Gaël Guennebaud, Benoît Jacob, et al. 2010. Eigen v3. http://eigen.tuxfamily.org
[23] Bastian Hagedorn, Archibald Samuel Elliott, Henrik Barthels, Rastislav Bodík, and Vinod Grover. 2020. Fireiron: A
Scheduling Language for High-Performance Linear Algebra on GPUs. CoRR abs/2003.06324 (2020). arXiv:2003.06324
https://arxiv.org/abs/2003.06324
[24] Bastian Hagedorn, Johannes Lenfers, Thomas Koehler, Sergei Gorlatch, and Michel Steuwer. 2020. A Language for
Describing Optimization Strategies. CoRR abs/2002.02268 (2020). arXiv:2002.02268 https://arxiv.org/abs/2002.02268
[25] Bastian Hagedorn, Johannes Lenfers, Thomas Kundefinedhler, Xueying Qin, Sergei Gorlatch, and Michel Steuwer. 2020.
Achieving High-Performance the Functional Way: A Functional Pearl on Expressing High-Performance Optimizations
as Rewrite Strategies. Proceedings of ACM on Programming Languages 4, ICFP, Article 92 (Aug. 2020), 29 pages.
https://doi.org/10.1145/3408974
[26] Alexander Heinecke, Greg Henry, Maxwell Hutchinson, and Hans Pabst. 2016. LIBXSMM: accelerating small matrix
multiplications by runtime code generation. In SC’16: Proceedings of the International Conference for High Performance
Computing, Networking, Storage and Analysis . IEEE/ACM, 981–991. https://doi.org/10.1109/SC.2016.83
[27] Torsten Hoefler and Roberto Belli. 2015. Scientific Benchmarking of Parallel Computing Systems: Twelve Ways
to Tell the Masses When Reporting Performance Results. In SC’15: Proceedings of the International Conference for
High Performance Computing, Networking, Storage and Analysis (Austin, Texas). IEEE/ACM, Article 73, 12 pages.
https://doi.org/10.1145/2807591.2807644
[28] François Irigoin, Pierre Jouvelot, and Rémi Triolet. 1991. Semantical interprocedural parallelization: an overview of
the PIPS project. In Proceedings of the 5th international conference on Supercomputing, ICS 1991, Cologne, Germany, June
17-21, 1991 , Edward S. Davidson and Friedel Hossfeld (Eds.). ACM, 244–251. https://doi.org/10.1145/109025.109086
[29] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and
Trevor Darrell. 2014. Caffe: Convolutional Architecture for Fast Feature Embedding. In Proceedings of the 22Nd ACM
International Conference on Multimedia (Orlando, Florida, USA) (MM ’14) . ACM, 675–678. https://doi.org/10.1145/
2647868.2654889
[30] Zhihao Jia, Oded Padon, James Thomas, Todd Warszawski, Matei Zaharia, and Alex Aiken. 2019. TASO: Optimizing
Deep Learning Computation with Automatic Generation of Graph Substitutions. In Proceedings of the 27th ACM
Symposium on Operating Systems Principles (Huntsville, Ontario, Canada) (SOSP ’19) . Association for Computing
Machinery, 47–62. https://doi.org/10.1145/3341301.3359630
[31] Malik Murtaza Khan, Protonu Basu, Gabe Rudy, Mary W. Hall, Chun Chen, and Jacqueline Chame. 2013. A script-based
autotuning compiler system to generate high-performance CUDA code. ACM Transactions on Architecture and Code
Optimization 9, 4 (2013), 31:1–31:25. https://doi.org/10.1145/2400682.2400690
[32] Fredrik Kjolstad, Shoaib Kamil, Stephen Chou, David Lugato, and Saman Amarasinghe. 2017. The Tensor Algebra
Compiler. Proc. ACM Program. Lang. 1, OOPSLA, Article 77 (Oct. 2017), 29 pages. https://doi.org/10.1145/3133901
[33] Michael Kruse and Tobias Grosser. 2018. DeLICM: scalar dependence removal at zero memory cost. In Proceedings
of the 2018 International Symposium on Code Generation and Optimization, CGO 2018, Vösendorf / Vienna, Austria,
February 24-28, 2018 , Jens Knoop, Markus Schordan, Teresa Johnson, and Michael F. P. O’Boyle (Eds.). ACM, 241–253.
https://doi.org/10.1145/3168815
[34] Chris Lattner, Mehdi Amini, Uday Bondhugula, Albert Cohen, Andy Davis, Jacques Pienaar, River Riddle, Tatiana
Shpeisman, Nicolas Vasilache, and Oleksandr Zinenko. 2021. Mlir: Scaling compiler infrastructure for domain specific
computation. In 2021 IEEE/ACM International Symposium on Code Generation and Optimization (CGO) . IEEE/ACM,
IEEE/ACM, 2–14. https://doi.org/10.1109/CGO51591.2021.9370308
[35] Benoît Meister, Nicolas Vasilache, David Wohlford, Muthu Manikandan Baskaran, Allen Leung, and Richard Lethin.
2011. R-Stream Compiler. In Encyclopedia of Parallel Computing , David A. Padua (Ed.). Springer, 1756–1765. https:
//doi.org/10.1007/978-0-387-09766-4_515
[36] Ravi Teja Mullapudi, Vinay Vasista, and Uday Bondhugula. 2015. PolyMage: Automatic Optimization for Image
Processing Pipelines. In Proceedings of the Twentieth International Conference on Architectural Support for Programming
Languages and Operating Systems, ASPLOS 2015, Istanbul, Turkey, March 14-18, 2015 , Özcan Özturk, Kemal Ebcioglu,
and Sandhya Dwarkadas (Eds.). ACM, 429–443. https://doi.org/10.1145/2694344.2694364
[37] Nod.AI. 2022. Outperforming Intel’s MKL and OctoML/Apache TVM with MLIR and Nod.ai’s Codegen Search.
https://nod.ai/outperforming-octoml-apache-tvm-and-intel-mkl/.
[38] Nod.AI. 2022. SHARK: The fastest PyTorch runtime – 3x over Torchscript, 1.6x over TF/XLA, 23% faster than
ONNXRuntime. https://nod.ai/shark-the-fastest-runtime/.
[39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming
Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison,
Composable and Modular Code Generation in MLIR 39
Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An
Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32 ,
H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024–
8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
[40] Phitchaya Mangpo Phothilimthana, Archibald Samuel Elliott, An Wang, Abhinav Jangda, Bastian Hagedorn, Henrik
Barthels, Samuel J. Kaufman, Vinod Grover, Emina Torlak, and Rastislav Bodík. 2019. Swizzle Inventor: Data Movement
Synthesis for GPU Kernels. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for
Programming Languages and Operating Systems, ASPLOS 2019, Providence, RI, USA, April 13-17, 2019 , Iris Bahar, Maurice
Herlihy, Emmett Witchel, and Alvin R. Lebeck (Eds.). ACM, 65–78. https://doi.org/10.1145/3297858.3304059
[41] Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman Amarasinghe. 2013.
Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines.
ACM SigPLAN Notices 48, 6 (2013), 519–530. https://doi.org/10.1145/2499370.2462176
[42] Ari Rasch, Richard Schulze, and Sergei Gorlatch. 2019. Generating Portable High-Performance Code via Multi-
Dimensional Homomorphisms. In 2019 28th International Conference on Parallel Architectures and Compilation Tech-
niques (PACT). IEEE, 354–369. https://doi.org/10.1109/PACT.2019.00035
[43] River Riddle. 2021. PDLL: a new declarative rewrite frontend for MLIR. https://llvm.discourse.group/t/rfc-pdll-a-new-
declarative-rewrite-frontend-for-mlir/4798.
[44] Giuseppe Rossini. 2021. Outerproduct spilling. https://llvm.discourse.group/t/outer-product-spilling/4405.
[45] Nadav Rotem, Jordan Fix, Saleem Abdulrasool, Summer Deng, Roman Dzhabarov, James Hegeman, Roman Levenstein,
Bert Maher, Nadathur Satish, Jakob Olesen, Jongsoo Park, Artem Rakhov, and Misha Smelyanskiy. 2018. Glow:
Graph Lowering Compiler Techniques for Neural Networks. CoRR abs/1805.00907 (2018). arXiv:1805.00907 http:
//arxiv.org/abs/1805.00907
[46] Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. Inverted Residuals
and Linear Bottlenecks: Mobile Networks for Classification, Detection and Segmentation. CoRR abs/1801.04381 (2018).
arXiv:1801.04381 http://arxiv.org/abs/1801.04381
[47] Alexander Schrijver. 1986. Theory of Linear and Integer Programming . John Wiley & Sons.
[48] Gus Henry Smith, Andrew Liu, Steven Lyubomirsky, Scott Davidson, Joseph McMahan, Michael B. Taylor, Luis
Ceze, and Zachary Tatlock. 2021. Pure Tensor Program Rewriting via Access Patterns (Representation Pearl). CoRR
abs/2105.09377 (2021). arXiv:2105.09377 https://arxiv.org/abs/2105.09377
[49] Michel Steuwer, Toomas Remmelg, and Christophe Dubach. 2017. LIFT: A functional data-parallel IR for high-
performance GPU code generation. In 2017 IEEE/ACM International Symposium on Code Generation and Optimization
(CGO). IEEE/ACM, 74–85. https://doi.org/10.1109/CGO.2017.7863730
[50] XLA team within Google. 2017. XLA: TensorFlow, Compiled. Google Developers Blog. https://developers.googleblog.
com/2017/03/xla-tensorflow-compiled.html
[51] Xinmin Tian, Hideki Saito, Serguei V. Preis, Eric N. Garcia, Sergey S. Kozhukhov, Matt Masten, Aleksei G. Cherkasov,
and Nikolay Panchenko. 2013. Practical SIMD Vectorization Techniques for Intel ® Xeon Phi Coprocessors. In
2013 IEEE International Symposium on Parallel Distributed Processing, Workshops and Phd Forum . IEEE, 1149–1158.
https://doi.org/10.1109/IPDPSW.2013.245
[52] Konrad Trifunovic, Albert Cohen, David Edelsohn, Feng Li, Tobias Grosser, Harsha Jagasia, Razya Ladelsky, Sebastian
Pop, Jan Sjödin, and Ramakrishna Upadrasta. 2010. GRAPHITE Two Years After: First Lessons Learned From Real-World
Polyhedral Compilation. In GCC Research Opportunities Workshop (GROW’10) . HAL. https://hal.inria.fr/inria-00551516
[53] Nicolas Vasilache. 2021. Understanding and controlling some of the AVX shuffle emission paths. https://groups.google.
com/g/llvm-dev/c/P_CXf5hPro8/m/RI6Fm3bzAQAJ.
[54] Nicolas Vasilache, Benoit Meister, Muthu Baskaran, and Richard Lethin. 2012. Joint scheduling and layout optimization
to enable multi-level vectorization. InIn Second International Workshop on Polyhedral Compilation Techniques (IMPACT) .
Informal proceedings.
[55] Nicolas Vasilache, Oleksandr Zinenko, Theodoros Theodoridis, Priya Goyal, Zachary Devito, William S Moses, Sven
Verdoolaege, Andrew Adams, and Albert Cohen. 2019. The next 700 accelerated layers: From mathematical expressions
of network computation graphs to accelerated GPU kernels, automatically. ACM Transactions on Architecture and Code
Optimization (TACO) 16, 4 (2019), 1–26. https://doi.org/10.1145/3355606
[56] Sven Verdoolaege. 2010. isl: An Integer Set Library for the Polyhedral Model. In Mathematical Software – ICMS 2010,
Third International Congress on Mathematical Software (Lecture Notes in Computer Science, Vol. 6327) , Komei Fukuda,
Joris van der Hoeven, Michael Joswig, and Nobuki Takayama (Eds.). Springer, 299–302. https://doi.org/10.1007/978-3-
642-15582-6_49
[57] Samuel Williams, Andrew Waterman, and David A. Patterson. 2009. Roofline: an insightful visual performance model
for multicore architectures. Commun. ACM 52, 4 (2009), 65–76. https://doi.org/10.1145/1498765.1498785
40 Vasilache et al.
[58] Jie Zhao, Michael Kruse, and Albert Cohen. 2018. A polyhedral compilation framework for loops with dynamic data-
dependent bounds. In Proceedings of the 27th International Conference on Compiler Construction, CC 2018, February 24-25,
2018, Vienna, Austria, Christophe Dubach and Jingling Xue (Eds.). ACM, 14–24. https://doi.org/10.1145/3178372.3179509
[59] Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang
Zhuo, Koushik Sen, Joseph E. Gonzalez, and Ion Stoica. 2020. Ansor: Generating High-Performance Tensor Programs for
Deep Learning. In 14th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2020, Virtual Event,
November 4-6, 2020 . USENIX Association, 863–879. https://www.usenix.org/conference/osdi20/presentation/zheng
[60] Oleksandr Zinenko, Sven Verdoolaege, Chandan Reddy, Jun Shirako, Tobias Grosser, Vivek Sarkar, and Albert Cohen.
2018. Modeling the conflicting demands of parallelism and Temporal/Spatial locality in affine scheduling. InProceedings
of the 27th International Conference on Compiler Construction, CC 2018, February 24-25, 2018, Vienna, Austria , Christophe
Dubach and Jingling Xue (Eds.). ACM, 3–13. https://doi.org/10.1145/3178372.3179507
Composable and Modular Code Generation in MLIR 41
A ANNOTATED IR EXAMPLES
A.1 LLVM Dialect and Mix of Dialects
The snippet in Figure 23 (functionllvm_example) mixes operations and types within the declaration
of a function in the llvm dialect.
llvm . func @llvm_example (%v: i32 , %cond : i1) -> ! llvm . struct <( f32 , vector <2x f32 >, i32 )> {
%0 = llvm . mlir . undef : ! llvm . struct <( f32 , vector <2x f32 >, i32 )>
%v f32 = arith . uitofp %v : i32 to f32
%1 = llvm . insertvalue %v f32 , %0 [0] : ! llvm . struct <( f32 , vector <2x f32 >, i32 )>
%f123 = arith . constant 123.0 : f32
cond_br %cond, ^ bb1 ( %f123 : f32 ), ^ bb2 ( %f123 : f32 )
^ bb1 ( %fa : f32 ):
%at123 = math . atan2 %fa, %fa : f32
br ^ bb2 ( %at123 : f32 )
^ bb2 ( %fb : f32 ):
%b = vector . broadcast %fb : f32 to vector <2x f32 >
%2 = llvm . insertvalue %b, %1 [1] : ! llvm . struct <( f32 , vector <2x f32 >, i32 )>
%c1024 = arith . constant 1024 : i32
%3 = llvm . insertvalue %c1024, %2 [2] : ! llvm . struct <( f32 , vector <2x f32 >, i32 )>
llvm . return %3 : ! llvm . struct <( f32 , vector <2x f32 >, i32 )>
}
Fig. 23. Illustration of an LLVM dialect function that takes an i32 and a bool and returns a struct. The struct
is filled with different values depending on the bool function argument.
llvm.undef creates a !llvm.struct<(f32, vector<2xf32, i32)> structure that is filled thanks to
llvm.insertvalue. The values used to fill the struct are created with operations from the arith,
math and vector dialect. Builtin types are provided that are visible to all dialects (e.g. f32, i32,
vector<2xf32>). Dialects may introduce their own types (e.g. !llvm.struct<(f32, f32, i32)> has
no builtin representation and is not understood by ops in the arith and math dialects).
This simple example already provides a key insight about SSA form that we leverage in other
dialects. The instruction
%1 = llvm . insertvalue %v f32 , %0 [0] : ! llvm . struct <( f32 , vector <2x f32 >, i32 )>
inserts the value %vf32 at position 0 in the !llvm.struct named %0. The instruction produces a new
!llvm.struct named %1 that contains a copy of all entries of %0 except for the f32 at position 0 that
takes the value %vf32.
In a classical compilation flow to LLVM IR, all operations and types must be converted to the
llvm dialect. This process is called dialect conversion . To make this process easier and improve
interoperability between dialects, the llvm dialect use the same representation for builtin types.
A.2 ArmNEON dialect example
MLIR has a native higher-dimensional vector type that conveniently mixes with target-specific
operations (effectively mirroring actual instructions) of the arm_neon dialect.
// High - level 2-d vector flavor of the sdot operation .
func @sdot2d_4x4_ i8 i8(%a: vector <4x i32 >, %b: vector <4 x4x i8 >, %c: vector <4 x4x i8 >)
-> vector <4x i32 > {
%0 = arm_neon .2d. sdot %a, %b, %c : vector <4 x4x i8 >, vector <4 x4x i8 > to vector <4x i32 >
return %0 : vector <4x i32 >
}
// Lowers to 1-d intrinsic form .
llvm . func @lowered_sdot2d_4x4_ i8 i8(%a: vector <4x i32 >, %b: vector <16xi8 >, %c: vector <16xi8 >)
42 Vasilache et al.
-> vector <4x i32 > {
%0 = arm_neon . intr . sdot %a, %b, %c : vector <16xi8 >, vector <16xi8 > to vector <4x i32 >
return %0 : vector <4x i32 >
}
// And further translates to llvm IR.
define <4x i32 > @llvm_sdot2d_4x4_ i8 i8 ( <4x i32 > %0, <16xi8 > %1, <16xi8 > %2) {
%4 = call <4x i32 > @llvm . aarch64 . neon . sdot .v4 i32 . v16 i8 ( <4x i32 > %0, <16xi8 > %1, <16xi8 > %2)
ret <4x i32 > %4
}
The llvm.aarch64.neon.sdot.v4i32.v16i8 intrinsic performs 4 parallel dot products on subvec-
tors of 4xi8, each contributing to one of the lanes of the resulting 4xi32. The arm_neon.intr.sdot
closely mirrors the intrinsic. However, we also provide a arm_neon.2d.sdot that exposes the opera-
tion semantics more clearly to the higher level of the infrastructure.
This simple example illustrates the semantic benefits of more structured, higher-dimensional
abstractions.
A.3 Target-Specific Vector IR
func @contract ( %m0 : memref <2x f32 >, %m1 : memref <2 x16x f32 >, %m2 : memref <2 x16x f32 >) {
%0 = vector . load %m0 [0] : memref <2x f32 >, vector <2x f32 >
%1 = vector . load %m1 [0, 0] : memref <2 x16x f32 >, vector <8x f32 >
%2 = vector . load %m1 [1, 0] : memref <2 x16x f32 >, vector <8x f32 >
%3 = vector . load %m1 [0, 8] : memref <2 x16x f32 >, vector <8x f32 >
%4 = vector . load %m1 [1, 8] : memref <2 x16x f32 >, vector <8x f32 >
%5 = vector . extract %0 [0] : vector <2x f32 >
%6 = splat %5 : vector <8x f32 >
%7 = vector . fma %6, %1, 0.0 f : vector <8x f32 >
%8 = vector . extract %0 [1] : vector <2x f32 >
%9 = splat %8 : vector <8x f32 >
%10 = vector . fma %9, %2, %7 : vector <8x f32 >
%11 = vector . fma %6, %3, 0.0 f : vector <8x f32 >
%12 = vector . fma %9, %4, %11 : vector <8x f32 >
vector . store %10, %m2 [0, 0] : memref <2 x16x f32 >, vector <8x f32 >
vector . store %12, %m2 [0, 8] : memref <2 x16x f32 >, vector <8x f32 >
vector . store %10, %m2 [1, 0] : memref <2 x16x f32 >, vector <8x f32 >
vector . store %12, %m2 [1, 8] : memref <2 x16x f32 >, vector <8x f32 >
return
}
A.4 Python Bindings and Interoperability
Structured tensor code generation strongly influenced the design of the MLIR Python bindings,
in particular, the “natural” mapping between nested Python context managers and the nested IR
structure is illustrated in Figure 24.
Structured data objects processed by the flow are exposed in Python as objects compatible with
the Python buffer protocol and can therefore be converted to and from NumPy arrays, which are
further convertible to framework-specific data types.
Once the program is constructed, compiled using the structured tensor code generation flow
and lowered, it can be JIT-compiled and functions from this program can be invoked from Python
using its name. The code below illustrates the execution of the matmul function from the module
defined above.
from mlir . execution_engine import ExecutionEngine
from mlir . runtime import get_ranked_memref_descriptor
import numpy as np
import ctypes
# Create NumPy arrays .
Composable and Modular Code Generation in MLIR 43
from mlir .ir import Context , InsertionPoint , Module
from mlir . dialects import builtin , scf
# Constructs IR in a fresh MLIR context .
with Context ():
module = Module ()
# Construct the following operations within module .
with InsertionPoint ( Module . body ):
func = builtin . FuncOp ( 'foo ')
# Construct the following operations within function .
# Create the entry block to turn declaration into definition .
with InsertionPoint ( func . add_entry_block ()):
loop = scf . ForOp (...)
# Construct the following operations with loop .
with InsertionPoint ( loop . body ):
# etc .
Fig. 24. Illustration of the L1-resident 2-D copy benchmark.
np_a , np_b = np. ones (M, K), np. ones (K, N)
np_c = np. zeros (M, N)
# Transform them into MLIR - compatible objects that share underlying data .
a, b, c = [ ctypes . pointer ( ctypes . pointer ( get_ranked_memref_descriptor (v)))
for v in (np_a , np_b , np_c )]
# JIT - compile and invoke the function .
engine = ExecutionEngine ( module , opt_level =3)
engine . invoke ('matmul ', a, b, c)
# Results are readily available in the NumPy array .
print ( np_c )

Deep Program Structure Modeling Through Multi-Relational
Graph-based Learning
Guixin Ye, Zhanyong Tang∗, Huanting Wang,
Dingyi Fang
Northwest University, China
{gxye,zytang}@nwu.edu.cn
Jianbin Fang
National University of Defense Technology, China
Songfang Huang
Alibaba DAMO Academy, China
Zheng Wang∗
University of Leeds, United Kingdom
z.wang5@leeds.ac.uk
ABSTRACT
Deep learning is emerging as a promising technique for building
predictive models to support code-related tasks like performance
optimization and code vulnerability detection. One of the critical
aspects of building a successful predictive model is having the
right representation to characterize the model input for the given
task. Existing approaches in the area typically treat the program
structure as a sequential sequence but fail to capitalize on the rich
semantics of data and control flow information, for which graphs
are a proven representation structure.
We present Poem1, a novel framework that automatically learns
useful code representations from graph-based program structures.
At the core of Poem is a graph neural network (GNN) that is spe-
cially designed for capturing the syntax and semantic information
from the program abstract syntax tree and the control and data
flow graph. As a departure from existing GNN-based code model-
ing techniques, our network simultaneously learns over multiple
relations of a program graph. This capability enables the learning
framework to distinguish and reason about the diverse code rela-
tionships, be it a data or a control flow or any other relationships
that may be important for the downstream processing task.
We apply Poem to four representative tasks that require a strong
ability to reason about the program structure: heterogeneous device
mapping, parallel thread coarsening, loop vectorization and code
vulnerability detection. We evaluate Poem on programs written in
OpenCL, C, Java and Swift, and compare it against nine learning-
based methods. Experimental results show that Poem consistently
outperforms all competing methods across evaluation settings.
∗Corresponding faculty authors: Zhanyong Tang and Zheng Wang.
1Poem = Deep Code Modeling.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from permissions@acm.org.
PACT ’20, October 3–7, 2020, Virtual Event, GA, USA
© 2020 Association for Computing Machinery.
ACM ISBN 978-1-4503-8075-1/20/10. . . $15.00
https://doi.org/10.1145/3410463.3414670
CCS CONCEPTS
• Computer systems organization; • Software and its engi-
neering →Compilers; • Security and privacy→Software se-
curity engineering ;
KEYWORDS
Program Modeling, Code Optimization, Machine Learning
ACM Reference Format:
Guixin Ye, Zhanyong Tang∗, Huanting Wang, Dingyi Fang, Jianbin Fang,
Songfang Huang, and Zheng Wang. 2020. Deep Program Structure Modeling
Through Multi-Relational Graph-based Learning. In Proceedings of the 2020
International Conference on Parallel Architectures and Compilation Techniques
(PACT ’20), October 3–7, 2020, Virtual Event, GA, USA. ACM, New York, NY,
USA, 13 pages. https://doi.org/10.1145/3410463.3414670
1 INTRODUCTION
Over the last two decades, machine learning has emerged as a viable
means for constructing heuristics for various program-related tasks
including code optimization [ 54]. There is now ample evidence
showing that machine-learned heuristics can outperform hand-
tuned approaches [10].
A key challenge for applying machine learning to programs is
that it requires programs to be represented as a sequence of numer-
ical values (such as the number and type of instructions) that serve
as inputs to a machine learning model. Traditionally, such program
representations were determined by experts through trials and er-
rors. However, since programs are syntactically unbounded graph
structures and that there is an infinite number of these potential
features, finding the right features is a non-trivial task.
More recent studies have leveraged the advances in deep learning
(DL) to model and reason about code structures [19–21, 33, 36, 65,
66]. Compared to classical machine learning approaches, DL has the
advantage of not requiring expert involvement to manually tune
representations for program structures; instead, it automatically
captures and determines them from training samples [6].
Existing DL-based approaches for program modeling typically
utilize recurrent neural networks (RNN), like the long short-term
memory (LSTM) or a variant of it, to model code structures [20, 33].
Such approaches work by treating source code and its structure –
for example, the abstract syntax tree (AST) – as a sequence of tokens.
However, LSTM is designed for processing a sequential sequence
[35] and is ill-suited for capturing the program control and data
flows – which should be better represented as a graph instead of
a sequence of tokens. As a result, prior methods only capture the
shallow, textual structure of the source code and fail to capitalize
on the rich and well-defined semantics of the program structure.
To better model the complex data and program structures – which
were traditionally represented as graph structures in compilers for
code analysis – we need an approach that could directly operate on
and learn from the graph representation of the code. Doing so will
allow the learning framework to preserve and reason about much
of the control and data flow information that is essential for many
program-related tasks.
The first effort in this direction is the recent work presented in
[13], which employs a vanilla graph neural network (GNN) to learn
representations from the graph representation of the AST or the
control-data flow graphs (CDFGs). This is achieved by propagating
information along the graph edges defined in a graph adjacency
matrix. While there may exist multiple code relationships (edges)
among any given node pair, their approach only captures the graph
connectivity, leaving the graph edges as untyped. As such, it cannot
tell if a direct connection between two nodes is a control or a data
flow, neither distinguish other relationships like order for non-
commutative operations. Intuitively, such information would be
essential for characterizing the program behavior for many code-
related tasks. By ignoring the different relationships, their GNN
approach gives marginal improvement or even worse performance
compared to the LSTM alternatives [13].
We present Poem, a better approach for modeling code struc-
tures. Poem operates on graph representations of the program with
the capability to learn and aggregate multiple code relationships.
It is designed to maintain sequential information like token order
and operand values when trading sequential representation for
graph representations. Poem automatically extracts such informa-
tion from the AST and the CDFGs. It then combines and abstracts
the extracted information to generate a numerical feature vector
that captures much of the essential information of the syntax and
semantics of the target program. We use the generated embeddings
as an input to a standard neural network to support downstream
processing tasks like code optimization and vulnerability detection.
As a departure from prior work [11–13], Poem uses graphs to
represent both the syntactic and semantic information of programs
and employes graph-based learning methods to learn to reason over
multiple graph structures. With Poem, syntax information is en-
coded from the AST and IR nodes. To maintain much of the sequen-
tial syntactic and semantic information, we augmented the AST
with additional edges. These edges allow us to encode sequential
syntactic relationships (e.g., “token before/after") and semantic rela-
tionships (e.g., “variable last used here", “this statement is guarded
by an if condition"). In addition to the AST, we also record the
control and data flow information from the CDFG. Intuitively, infor-
mation collected from the source code is language-dependent but
agnostic to compiler implementations. By contrast, data collected
from the IR captures much of the lower-level, language-agnostic but
compiler-specific information that could not be directly obtained
from the source code. By combining such information, we improve
the generalization ability of the learning framework, as different
tasks may require knowledge at different levels.
Unlike [13] that it only uses an adjacency matrix to encode the
node connectivity of the AST or CDFG, we encode different node re-
lationships, e.g., whether it is a child-parent connection on the AST
or a data flow edge in the CDFG, in different matrices and relation
graphs. At the core of Poem is a novel graph neural network that
can learn over multiple relationships (or edge types) simultaneously.
By representing the input program as multiple relation graphs with
explicit control and data flows or syntactic information, Poem cap-
tures a greater range of intra-program relations than prior graph
representations. A key advantage of Poem is that it uses a learnable
function to aggregate information for individual relation graphs.
As the aggregation function is tuned for each relation graph, it can
capture each specific code relationship more precisely. This richer
set of relationships improves the model’s ability in learning useful
program representation, which in turns leads to better performance
of downstream processing tasks. We show that Poem is highly ef-
fective in learning abstracted code representations, allowing us to
solve various tasks with performance better than state of the arts.
We demonstrate the benefits of Poem by applying it to four
representative tasks that require a strong ability to reason about
the program structure at different levels: heterogeneous device
mapping, GPU thread coarsening, loop vectorization and code vul-
nerability detection. We evaluate Poem on benchmarks written in
OpenCL, C, Java and Swift. We comparePoem against a wide range
of machine-learning techniques. Experimental results show that
Poem consistently outperforms competing methods across tasks
and programming languages, by giving stronger performance im-
provement and demonstrating a better generalization ability across
evaluation tasks.
This paper makes the following contributions:
•It presents the first graph-based learning framework2 that
can simultaneously model multiple edge types of the pro-
gram graph (Section 2);
•It demonstrates how to usemultiple graphs to represent both
the syntactic and semantic structures of programs to support
graph-based deep learning (Section 2.3);
•It is the first work showing how a general graph-based learn-
ing framework can be developed to deliver consistently bet-
ter performance over LSTM-based alternatives across a range
of code-related tasks (Section 4).
2 OUR APPROACH
Poem is designed to operate on graph representations of the AST
extracted from the source code, and the CDFG obtained from the
compiler IR. As we will show later in the paper, Poem can adapt to
different programming languages, hardware platforms and tasks.
2.1 Overview of Poem
Figure 1 depicts the architecture of Poem. It takes as input the
program source code. It first uses a source rewriter extended from
[20] to normalize the identifiers. Next, it builds the AST and CDFG
using standard compiler passes. We extend the standard AST with
additional edges to carry data and control flow information at the
source code level (Section 2.3). The AST and CDFG are presented as
directed multigraphs, where statements, identifiers, and immediate
2Code and data are available at: [https://github.com/yeguixin/POEM.]
/g86/g82/g88/g85/g70/g72/g3/g70/g82/g71/g72
/g54/g82/g88/g85/g70/g72/g3/g70/g82/g71/g72/g3/g85/g72/g90/g85/g76/g87/g72/g85
/g38/g39/g41/g42
/g42/g85/g68/g83/g75/g3/g72/g80/g69/g72/g71/g71/g76/g81/g74/g86
/g36/g88/g91/g76/g79/g76/g68/g85/g92/g3
/g44/g81/g83/g88/g87/g86
optional
/g49/g82/g85/g80/g68/g79/g76/g93/g68/g87/g76/g82/g81/g3/g47/g68/g92/g72/g85
/g39/g72/g70/g76/g86/g76/g82/g81/g3/g81/g72/g87/g90/g82/g85/g78
/g51/g85/g72/g71/g76/g70/g87/g76/g82/g81/g86
/g31/g17/g17/g17/g33
/g51/g50/g40/g48/g16/g42/g49/g49
/g42/g85/g68/g83/g75/g3/g80/g68/g87/g85/g76/g70/g72/g86/g3
/g9/g3/g89/g72/g85/g87/g72/g91/g3
/g85/g72/g83/g85/g72/g86/g72/g81/g87/g68/g87/g76/g82/g81/g86
/g38/g82/g80/g83/g76/g79/g72/g85/g3/g73/g85/g82/g81/g87/g72/g81/g71/g18/g83/g68/g86/g86/g72/g86
/g51/g85/g82/g74/g85/g68/g80/g3/g85/g72/g83/g85/g72/g86/g72/g81/g87/g68/g87/g76/g82/g81/g86
/g36/g54/g55
Figure 1:Poem operates on the program graph matrices and
vertex representations derived from the AST and CDFG. It
uses the Poem-GNN to extract useful program representa-
tions (i.e., graph embeddings), which are encoded as vec-
tors of numerical values. The embedding vectors and theop-
tional auxiliary input are concentrated and normalized and
passed to the decision network for a given prediction task.
values are vertices, and a direct relationship (e.g., parent-child, data
or control flow, etc.) between two vertices is recorded as an edge.
As there may exist multiple relationships (or edges) among a pair
of vertices, we use a relation graph to record a specific type of
relationships. In this work, we wish to capture 10 relationships
from the AST and CDFG (Section 2.3), leading to 10 relation graphs.
The vertex connectivity of a relation graph is represented as a
program graph matrix (Section 2.4).
The Poem-GNN takes in the program matrices and initial vertex
(or node) representations to learn program representations called
embeddings that are represented as a vector of numerical values.
Like [20], the user can also optionally supply auxiliary inputs to give
additional information about runtime parameters. We concentrate
the graph embeddings and ancillary data to form a fixed-length fea-
ture vector, which is first normalized and then passed to a heuristic
model (based on a standard fully-connected, dense neural network)
to make a prediction.Poem-GNN and the dense network are trained
together so that the graph representation is tuned for the task.
Table 1: Code relationships considered in this work
Source Relationships
AST ASTChild, NextToken, ComputedFrom, GuardedBy, Jump, Las-
tUse, LastLexicalUse
IR Sequential-IR flow, data flow, and control flow
Unlike [13] that is only able to model the node connectivity, we
extend the GNN to model multiple edge types (e.g., control, data,
jump, token sequence, etc.). This capability allows Poem to distin-
guish different relationships of the code, whether it is an if branch
or a function call. In Section 4, we show that our approach con-
sistently outperforms prior methods that are based on the vanilla
GCN [13] or LSTM [20].
Roadmap. The remaining of this section is organized as follows.
We describe how we extract the AST and CDFG in Section 2.2.
From Sections 2.3 to 2.5, we explain how to model and encode the
different code relationships in Sections 2.3-2.5. Finally, we describe
our multi-relational graph neural network from Sections 2.6 to 2.8.
2.2 AST and CDFG Construction
We construct the AST from the standardized source code using a
compiler front-end parser (e.g., Clang for C). We construct the CDFG
from the compiler IR after applying standard compiler data-flow
analysis and optimizations like dead-code elimination, constant
propagation, and common subexpression elimination. An AST con-
tains syntax nodes and syntax tokens. The former corresponds
to nonterminals in the language grammar, e.g., an if statement
(IfStmt) and function names; and the latter corresponds to termi-
nals like literals and constant values. The CDFG captures semantic
information that reflects standard compiler transformations. Fig-
ures 2(b) and 2(c) respectively show the LLVM IR and the extracted
CDFG for the OpenCL kernel (after source code rewritten) given in
Figure 2(a). To simplify the IR, we replace identifiers with its type.
For example, %4 at lines 3 and 5 of Figure 2(b) will be replaced with
its data type i32.
2.3 Code Relationships
We record ten relationships from the AST and the IR, listed in Table
1. These include relationships that can be directly obtained from
the CDFG, like the sequential order of the IR instructions, and the
control and data flow. We also augment the AST with six additional
edges, described as follows.
A standard AST has just one type of edges, i.e., the ASTChild
edge that connects the children nodes with their parents. To capture
additional syntax and data and control flow information of the AST,
we introduce six additional edges to the AST, following the method
described in [7]. The additional edges are essential because they
maintain local sequence order such as the ordering of variable
use and operations. As the AST edges do not induce an order on
children of a syntax node, we addNextToken edges to connect each
syntax token to its successor. This edge is used to capture the order
of opcode and operands for statements. For each assignment, 𝑣 =
𝑒𝑥𝑝𝑟, we connect 𝑣 to all variable tokens occurring in expression,
𝑒𝑥𝑝𝑟, using ComputedFrom edges. We connect each AST token of
1 __kernel void A( __global uint4 * a, __global uint4 *
b) {
2 unsigned int d = get_local_id (0) ;
3 if (d > 0) {
4 b[d] = a[d] + a[d + 1];
5 } else {
6 b[d] = 0;
7 }
8 }
(a) An example OpenCL kernel
1 define void @A( i32 ,i32 ) {
2 %3 = tail call i64 @get_local_id ( i32 0)
3 %4 = trunc i64 %3 to i32
4 %5 = icmp eq i32 %4
5 %6 = and i64 %3
6 br i1 %5 , label %15 , label %7
7 ...
8 %16 = phi i32 [%14 , %7] , [ zeroinitializer , %2]
9 %17 = %1 , i64 %6
10 store %16 , %17
11 ret void
12 }
(b) LLVM IR
/g54/g72/g84/g88/g72/g81/g87/g76/g68/g79/g3/g44/g53/g3/g73/g79/g82/g90/g39/g68/g87/g68/g3/g73/g79/g82/g90/g38/g82/g81/g87/g85/g82/g79/g3/g73/g79/g82/g90
/g70/g68/g79/g79/g87/g85/g88/g81/g70/g76/g70/g80/g83
Ă
/g69/g85
/g68/g81/g71
Ă
Ă
/g79/g82/g68/g71
/g83/g75/g76
Ă/g86/g87/g82/g85/g72
/g85/g72/g87
Ă
/g76/g25/g23/g76/g22/g21
/g76/g25/g23
Ă
(c) Control and data flow graph (CDFG)
/g41/g88/g81/g39/g72/g70/g79/g29/g36
Ă /g44/g73/g54/g87/g80/g87
/g38/g82/g80/g83/g82/g88/g81/g71/g54/g87/g80/g87/g3/g51/g68/g85/g80/g39/g72/g70/g79/g29/g69/g51/g68/g85/g80/g39/g72/g70/g79/g29/g68
/g39/g72/g70/g79/g53/g72/g73/g40/g91/g83/g85/g29/g71/g36/g85/g85/g68/g92/g54/g88/g69/g40/g91/g83/g85
/g39/g72/g70/g79/g53/g72/g73/g40/g91/g83/g85/g29/g69/g39/g72/g70/g79/g53/g72/g73/g40/g91/g83/g85/g29/g71/g36/g85/g85/g68/g92/g54/g88/g69/g40/g91/g83/g85
/g39/g72/g70/g79/g53/g72/g73/g40/g91/g83/g85/g29/g71
/g38/g82/g80/g83/g82/g88/g81/g71/g54/g87/g80/g87/g3
/g39/g72/g70/g79/g53/g72/g73/g40/g91/g83/g85/g29/g71
Ă
Ă
Ă Ă
Ă Ă
Ă
Ă
/g36/g54/g55/g38/g75/g76/g79/g71
/g45/g88/g80/g83
/g42/g88/g68/g85/g71/g72/g71/g37/g92
/g47/g68/g86/g87/g56/g86/g72
/g49/g72/g91/g87/g55/g82/g78/g72/g81/g38/g82/g80/g83/g88/g87/g72/g71/g41/g85/g82/g80
/g47/g68/g86/g87/g47/g72/g91/g76/g70/g68/g79/g56/g86/g72
(d) AST with additional data and control flow edges
Figure 2: A standardized OpenCL kernel (a), and its corre-
sponding LLVM IR (b), CDFG (c) and augmented AST (d).
a variable to the variable’s enclosing guard expressions using a
GuardedBy edge. For example, for the if statement in Figure 2(a),
we add a GuardedBy edge from b to the AST node corresponding to
d > 0. We use a Jump edge to connect variables that have control
dependencies. TheGuardedBy and Jump edges allow us to record the
relation of diverging control flow. We connect all uses of the same
variable using LastUse edges to capture the use of variables, where
a special case is variables in if statement and we connect such
type of variables using LastLexicalUse edges. For instance, for
the if statement in Figure 2(a), we would link the two occurrences
of d: one in the loop head, the other in the loop body. Figure2(d)
shows the resulting AST after processing the OpenCL kernel given
in Figure2(a), which is augmented with the additional edges.
Finally, for each graph edge, we also add a respective backward
edge (by transposing the adjacency matrix), doubling the number of
edges and edge types. These backward edges help with propagating
information across Poem-GNN (Section 2.6) and make the model
more expressive.
2.4 Program Graph Matrices
We convert the augmented AST and CDFG to separated relation
graphs - one graph for each of 10 relationships given in Table 1. A
relation graph is a directed graph,𝐺 = (𝑉,𝐸 ), that contains the AST
or IR node (vertices), 𝑉, and edges 𝐸, that indicates the existence
of a given relationship between two vertices, such as data, control
and ASTChild, etc. We use an adjacency matrix to recode the edge
connections of each relation graph, 10 matrices in total. A value of
1 for matrix element𝑒𝑖,𝑗, represents there exists a direct connection
or relation from node 𝑖to node 𝑗, while a value of 0 indicates the
two nodes are not directly connected.
2.5 Vertex Representations
To capture the syntactic and semantic meanings of the relation
graph vertices, we map every instructions (e.g., AST nodes like
ParamDecl, IfStmt and IR opcodes), constant, and variable to a
vector representation by lookup in a fixed size embedding table.
To do so, we first construct a vocabulary of frequently appeared
words from the training corpus, where we store the AST and IR
extracted from training programs. As variable and function names
and constant values can be of an arbitrary length, we encode them
as tokens (i.e., letters, symbols and numbers [0-9]). During the
model deployment stage, if a word of the input program is not
presented in the vocabulary, it will be encoded at the token level.
Once the vocabulary is constructed, we apply word2vector [43]
to map each word and token of the vocabulary to an embedding
space of integer values. The word2vector model is trained on train-
ing benchmarks of the target programming language and compiler
IR. Training is largely independent of the optimization task, whose
goal is to map individual words to a point in a latent multidimen-
sional space where words that are frequently appeared together are
mapped to integer values close to each other in the space. Doing
so allows us to capture much of the syntactic relation of language
constructs. For example, it allows the model to learn that an if
statement must precede an else statement. In this work, we map
each of the tokens and words in the vocabulary to a single fixed-
length embedding vector of 100 features. This vector captures many
characteristics of the code, such as syntax and shadow semantic
similarities.
/g62/g19/g15/g3/g171/g15/g3/g20/g19/g64
/g62/g20/g15/g171/g15/g20/g19/g64
/g62/g19/g15/g171/g15/g21/g20/g64
/g62/g20/g15/g3/g171/g15/g3/g25/g64
/g62/g20/g15/g3/g171/g15/g3/g22/g64
/g44/g81/g83/g88/g87/g3/g53/g72/g79/g68/g87/g76/g82/g81/g3/g42/g85/g68/g83/g75
/g55/g68/g85/g74/g72/g87/g3/g57/g72/g85/g87/g72/g91
/g49/g72/g76/g74/g75/g69/g82/g85/g3/g20
/g49/g72/g76/g74/g75/g69/g82/g85/g3/g21
/g53/g72/g79/g68/g87/g76/g82/g81/g3/g42/g85/g68/g83/g75/g3/g21
/g53/g72/g79/g68/g87/g76/g82/g81/g3/g42/g85/g68/g83/g75/g3/g81
/g11/g68/g12/g3/g47/g72/g68/g85/g81/g76/g81/g74/g3/g89/g72/g85/g87/g72/g91/g3/g72/g80/g69/g72/g71/g71/g76/g81/g74/g86/g3/g11/g69/g12/g3/g3/g49/g72/g76/g74/g75/g69/g82/g85/g75/g82/g82/g71/g3/g68/g74/g74/g85/g72/g74/g68/g87/g76/g82/g81/g3
/g55/g68/g85/g74/g72/g87/g3/g57/g72/g85/g87/g72/g91/g29/g32
/g48/g47/g51/g258/g336/g336/g396
/g181
/g48/g47/g51/g258/g336/g336/g396/g11 /g11/g3/g3/g3/g3/g3/g3/g12/g15/g3
/g80/g88/g79/g87/g76/g16/g79/g68/g92/g72/g85/g3/g83/g72/g85/g70/g72/g83/g87/g76/g82/g81/g3/g81/g72/g87/g90/g82/g85/g78/g72/g80/g69/g72/g71/g71/g76/g81/g74/g86/g3/g11/g89/g72/g85/g87/g72/g91/g3/g86/g87/g68/g87/g72/g12/g89/g72/g85/g87/g72/g91 /g41/g72/g72/g71/g16/g73/g82/g85/g90/g68/g85/g71/g3/g81/g72/g88/g85/g68/g79/g3/g81/g72/g87/g90/g82/g85/g78/g3/g69/g68/g70/g78/g90/g68/g85/g71/g3/g72/g71/g74/g72
/g583/g3/g3/g3/g3/g3/g3/g12
/g53/g72/g79/g68/g87/g76/g82/g81/g3/g42/g85/g68/g83/g75/g3/g20
Figure 3: UsingPoem-GNN to learn vertex embeddings. The initial vertex embeddings are generated using word2vec (Sec-
tion 2.5), which are then iteratively updated during the learning process (a) by aggregating information across neighbors and
information from other relation graphs for the same vertex (b).
2.6 Poem-GNN: A Multi-Rational Graph Neural
Network
The adjacency matrices and vertex embeddings of relation graphs
are passed to thePoem-GNN to map the inputs to a one-dimensional
embedding of 100 features. The Poem-GNN consists of several
stacked GNN embedding layers based on the multi-layer perception
(MLP) network, so that it can incorporate higher degree neighbor-
hoods across relation graphs. We choose MLP because it is proven
to be effective in learning embeddings for directed graphs [61], but
other neural network architectures (like GRU [17]) can also be used.
We use an AutoML tool [4] to search for the optimal number of
embedding layers from training data (see also Section 4.5.1).
2.6.1 Neighborhood aggregation. Each embedding layer follows a
neighborhood aggregation scheme (Figure 3b), where the 𝑑 dimen-
sional representation vector,ℎ𝑣, of a graph vertex, 𝑣, is computed
by recursively aggregating and transforming the representation
vectors of its neighboring nodes. Vertices are initialized with the
embeddings given by word2vec (Section 2.5) and then exchange
information by transforming their current state and sending it as a
message to all neighbours in the graph. At each vertex, messages
are aggregated and then used to update the associated node rep-
resentation at the next embedding layer (referred to as the next
iteration) [58]. The added backward edges (Section 2.3) enable back-
wards propagation of information. After repeating this process of
updating vertex states for a fixed number of iterations a readout
function (Section 2.6.3) is used to aggregate the vertex representa-
tions to a single numerical vector across multiple relation graphs
to be used as the program representation.
2.6.2 Multi-relation modeling. One of the novel aspects of the
Poem-GNN is that it can propagate and aggregate information
across multiple relation graphs, being it control and data at the
IR or AST. As illustrated in Figure 3a, we achieve this by first
using learnable, relation-specific MLP layers,𝑀𝐿𝑃ℓ, to compute new
graph states of individual relation graphs through neighborhood
aggregation. Specifically, we use a feed-forward neural network
to implement neighborhood aggregation for communicating the
neighbor embeddings to the reference vertex (Figure 3b). We then
apply a MLP-based aggregation function, 𝑀𝐿𝑃𝑎𝑔𝑔𝑟, to aggregate
and update states for identical AST or IR nodes (locating using
the matrix indices) across relation graphs. Note that our current
implementation aggregates information of AST and CDFG relation
graphs as two separate groups as there is often no one-to-one
mapping between AST and IR nodes after applying standard code
transformations. However, the embeddings learned for the AST and
CDFG will be aggregated during the readout stage.
Formally, we use forward propagation to update the state, ℎ𝑡𝑣,
for vertex, 𝑣, of a relation graph as:
ℎ𝑡+1
𝑣 := 𝜎(𝑀𝐿𝑃𝑎𝑔𝑔𝑟(
Õ
ℓ
Õ
(𝑢,𝑣)∈𝐴ℓ
𝑀𝐿𝑃ℓ(ℎ𝑡
𝑢))) (1)
where 𝐴ℓ are the directed edges between a node pair, 𝑢, and 𝑣, and
𝑀𝐿𝑃ℓ is a relation-graph-specific message propagation function.
Note that 𝑀𝐿𝑃𝑎𝑔𝑔𝑟 and 𝑀𝐿𝑃ℓ are learnable functions, that are up-
dated during the training process. The initial vertex state, ℎ0𝑣, is
created using the word2vec embedding method described in Sec-
tion 2.5. It is worth noting that the vertex representation get more
refined and global as the number of iterations increases.
2.6.3 Result readout. Once we have performed the neighbourhood
aggregation procedure for a fixed number of times (determined
by the number of embedding layers), we will obtain a new set
of embeddings for each vertex. Through this process, the vertex
knows more about their own information (features) and that of
neighbouring nodes. This creates an even more accurate represen-
tation of the relation graphs. To represent the input AST and CDFG,
we use a readout function to concatenate graph representations
across all the neighborhood aggregation iterations and embedding
layers, to form a single numerical vector,ℎ𝐺, as the global program
representation of all (𝑚= 10) relation graphs, 𝐺𝑖:
ℎ𝐺 = CONCAT
 𝑚Õ
𝑖=1
n
ℎ(𝑡)
𝑣,𝑖 |𝑣 ∈𝐺𝑖
o
|𝑡 = 0,1,...,𝑛
!
(2)
where 𝑡 = 0,1,...,𝑛 , is the neighborhood aggregation iterations. This
readout function produces the global embedding for all relation
graphs, given individual vertex embeddings.
2.6.4 Alternative modeling approaches. A naïve alternative to our
approach is to apply a standard GNN to individual relation graphs
and then concatenate the embedding outcomes of individual graphs.
However, this approach does not allow information to be exchanged
during each neighborhood aggregation iteration. In Section 4.5.2,
we show that this approach gives poor learning performance. As
a result, simply applying the GNN presented in [ 13] to multiple
relation graphs does address the issues of learning multiple code
relations. We have also considered the recently proposed Relational
Graph Convolutional Network (RGCN) [ 50] as it can model dif-
ferent relationships through multi-edge encoding. However, the
RGCN has a significant drawback – the number of parameters
drastically increases for larger graphs. This increase in parameters
can lead to overfitting, especially when the number of training
samples is limited. Poem sidesteps this problem by restricting the
relation-specific learnable function to a smaller subgraph of the
entire program. Furthermore, unlike the RGCN that only operates
on a single graph, Poem also supports information aggregation
of the distinct AST and CDFG. In Section 4.5.2, we show that our
approach outperforms the RGCN alternative.
2.7 Graph Embeddings
The graph embedding vector, produced by thePoem-GNN, together
with any auxiliary data is first normalized to a range of 0 and 1 by
the normalization layer. Normalization is essential as it prevents
the range of single feature being a factor in its importance. The
normalized feature vector is then fed to the dense network for
downstream processing (Figure 1), e.g., classification. This final
feature vector captures many characteristics of the code, such as
semantic similarities, control and dependence flows, combinations,
and analogies.
2.8 Train the Poem-GNN
We train the Poem-GNN and the dense network together using
back-propagation. Training is performed on batched training sam-
ples where each sample contains a ground-truth label. We use the
standard cross-entropy loss as the objective function. This function
is shown to be a good fit for sigmoid and softmax activation func-
tions (both are also standard functions for classification) [64] used
by Poem. It is defined as:
L𝐺 = −
𝑁Õ
𝑖=1
𝑦𝑜,𝑐𝑙𝑜𝑔(𝑝0,𝑐) (3)
where 𝑁 is the number of classes (i.e., running the code on the
CPU or GPU), 𝑦takes a binary value (0 or 1) that indicates if label
𝑐 is the correct classification for training sample 𝑜, and 𝑝 is the
predicted probability for sample 𝑜 to be of class 𝑐.
3 EXPERIMENTAL SETUP
To demonstrate the benefit of Poem, we use it to tune performance
optimization heuristics for OpenCL and C programs. To evaluate
the generalization ability of Poem in modeling program structures,
we also apply it to detect code vulnerabilities for C, Java and Swift.
In total, we applyPoem to four case studies and compare it with nine
prior machine-learning-based approaches (including models using
hand-crafted features) across eight distinct hardware platforms. In
some cases, we also compared to expert-tuned heuristic models.
Note that we use the same model structure for Poem across tasks.
Table 2 lists the machine learning models used in the evaluation.
Table 2: Machine learning methods used in evaluation
Approach Code repre-
sentation
Model Use cases
Grewe et al.
[30, 53]
Manual fea-
tures
Decision Tree Case study 1
DeepTune
[20]
Source code to-
ken sequence
LSTM Case studies 1-3
Inst2vec [12] LLVM IR
tokens
LSTM Case studies 1-2
GNN-AST [13] AST Vanilla GNN Case studies 1-3
GNN-CDFG
[13]
CDFG Vanilla GNN Case studies 1-3
Magni et al.
[42]
Manual fea-
tures
Neural Net-
works
Case study 2
NeuroVectorizer
[33]
Token sequence LSTM + Re-
inforcement
learning
Case study 3
uVuldeepecker
[66]
AST Bidirectional-
LSTM
Case study 4
Lin et al.[40] AST Bidirectional-
LSTM
Case study 4
Poem AST + CDFG multi-
relational
GNN
Case studies 1-4
3.1 Case Study 1: Heterogeneous Mapping
The problem.In this task, we wish to build a predictive model to
determine if the CPU or the GPU gives faster performance for a
given OpenCL kernel.
Methodology. We use the dataset of [20], which provides labeled
CPU/GPU instances for 256 OpenCL kernels extracted from seven
benchmark suites. The data were collected on two CPU-GPU plat-
forms: one with an Intel Core i7-3820 CPU and and AMD Tahiti
7970 GPU, and the other has an Intel Core i7-3820 CPU and an
NVIDIA GTX 970 GPU. By varying dynamic inputs, this dataset
consists of 680 labeled instances on each platform. The compilation
of some kernels ended with the presence of errors. We have manu-
ally fixed those broken OpenCL kernels to use the entire dataset. We
apply 10-fold cross-validation train and test a model. We repeat this
process ten times (folds), with each of the seven suites used exactly
once as the testing data. The CDF diagram in Figure 4 shows the
distribution for the number of AST nodes and branches for kernels
in this dataset. For this dataset, over 50% of the kernels have more
than 100 AST nodes, and over 75% of the kernels have one or more
branches. Since the dataset in [20] is small, it may not provide suf-
ficient training samples for a deep learning method. To evaluate on
a larger training dataset, we use DeepSmith, an OpenCL program
synthesizer [19], to generate 12,000 valid and compilable OpenCL
kernels as additional training data for deep-learning-based compet-
ing methods and Poem. This second experiment was performed on
our GPU platform that uses a 3.2 GHz 6-core Xeon E5-2667 CPU
and an NVIDIA Titan XP GPU. We profiled all benchmarks from the
DeepTune dataset to obtain the ground-truth label on this platform.
Competitive methods.For this case study, we comparePoem with
five machine-learning models. These include Grewe et al. [30] that
500 5600 12000 15771210.0
0.2
0.4
0.6
0.8
1.0 CDF
Number of AST nodes
(a) #AST nodes
0 5 10 140.0
0.2
0.4
0.6
0.8
1.0 CDF
Number of branches (b) #Branches
Figure 4: Cumulative distribution functions (CDF) for the
number of AST nodes (a) and branches (b) in the DeepTune
OpenCL kernel dataset [20].
uses hand-tuned features, and DeepTune [20] that uses LSTM to
extract code representations from source code token sequence. We
also compare to Inst2vec [12], a LSTM-based model that operates on
a graph representation of the LLVM IR. For graph models, we com-
pare to the two GNN variants presented in [13] offers GNN-CDFG
and GNN-AST which operate on the CDFG and AST respectively.
Like all prior work, for this case study, we use two dynamic val-
ues, the workgroup size and the host-device memory transfer size,
both are available to the OpenCL runtime, as the auxiliary inputs
(or features) to all predictive models. Results of this case study is
presented at Section 4.1.
3.2 Case Study 2: Thread Coarsening
The problem.This task builds a model to determine how many
parallel threads should be merged together to achieve faster exe-
cution time. This is a problem known as determining the thread
coarsening factor for OpenCL [42]. Here, we wish to build a model
to determine for each kernel, which of the six coarsening factors, 1,
2, 4, 8, 16, and 32, should be used for a given kernel (where a factor
of 1 means no coarsening).
Methodology. We replicate the setup of [ 13, 20, 42] by testing
each approach using the labeled dataset given in [42]. This dataset
contains 17 OpenCL kernels extracted from three benchmark suites.
The data were collected from five distinct GPU platforms: AMD HD
5900, AMD Tahiti 7970, NVIDIA GTX 480, and NVIDIA K20c. We
also extended our evaluation to NVIDIA Titan XP by profiling the
same OpenCL kernels on this GPU architecture. Like [20], we use
leave-one-out cross-validation for this task because the benchmark
set is small. This works by selecting one benchmark for testing
and using the remaining ones for training. This task is designed to
evaluate if our approach can effectively support transfer learning
[60], a technique for reusing the knowledge learned from one task
to speed up the learning for another task.
Competitive methods.We comparePoemagainst three approaches:
Magni et al. [42] that uses hand-tuned features, DeepTune, Inst2vec,
and GNN-CDFG and GNN-AST presented in [13]. To apply trans-
fer learning, we first train an initial deep learning model on the
dataset given in [20]. We then use transfer learning to fine-tune
the trained model on training data used for this task. Fine-tuning is
done by simply copying the learned parameters of case study one
Table 3: Dataset for vulnerability detection.
Source Language #samples #positive samples
C 156,668 78,334SARD & NVD Java 60,242 30,121
GitHub C 10,400 5,200
Swift 4,124 2,062
to initialize the model and then training the model as normal using
cross-validation. Note that for this task, the OpenCL kernel is the
sole input and coarsening factor is the predicted output. Results of
this case study are presented in Section 4.2.
3.3 Case Study 3: Loop Vectorization
The problem.This task targets the classic compiler optimization
problem of loop vectorization. It aims to build a predictive model to
determine the optimal vectorization factor (VF) and the interleaving
factor (IF) for individual loops. The first parameter determines how
many instructions to pack together from different loop iterations,
while the latter decides the stride of the memory accesses of the
packed instructions. Prior work has shown that the two parame-
ters can have a substantial impact on the resulting vectorization
performance[33, 45]. We consider 35 combinations of VF (1, 2, 4, 8,
16, 32, 64) and IF (1, 2, 4, 8, 16), which are found to be useful in [33].
Methodology. We use LLVM version 9.0 as the compiler. We con-
figure the VF and IF on a per loop basis by placing the LLVM/-
Clang vectorization directives, e.g., loop vectorize_width(VF)
interleave_count(IF). We replicate the evaluation of NeuroVec-
torizer [33], by using the 6,000 synthetic loops generated from
the LLVM vectorization test set to train a model and then test the
trained model on hand-written programs from MiBench [32] and
PolyBench [29]. Our evaluation platform uses an 3.6 GHz Intel Core
i7 CPU with 64GB RAM.
Competitive methods.We compare Poem against three super-
vised learning models (DeepTune and the two GNN variants in [13]),
as well as Polly [ 31], a LLVM-based code vectorizer based on
the polyhedral model. In addition to these, we also compare to
NeuroVectorizer [33], a recently proposed reinforcement-learning-
based approach. NeuroVectorizer first learns the program repre-
sentations through LSTM. The representations are then used by a
reinforcement learner to search for the best configuration until a
convergence threshold is met. Due to the nature of reinforcement
learning, NeuroVectorizer can incur significant search overhead. It
takes minutes to search for the vectorization configuration for a
single loop on our evaluation platform. By contrast, our approach
takes less than 100ms (including constructing the relation graphs)
to make a prediction. The results are presented in Section 4.3.
3.4 Case Study 4: Vulnerability Detection
The problem.In this task, we build a model to detect if a given
source code snippet contains one of the 2019 CWE top-25 most
dangerous software errors [26] at the function level.
Methodology. As summarized in Table 3, we use a dataset of
231,434 samples with source languages in C, Java and Swift, where
half of the samples are vulnerable code. The vulnerable code sam-
ples are collected from the standard vulnerable code databases,
including the national vulnerability database (NVD), common vul-
nerabilities and exposures (CVE) and open datasets collected from
GitHub. The vulnerable-free samples are obtained by applying the
corresponding patch to the vulnerable code. We apply 10-fold cross-
validation to train and test a predictive model (see also Section 3.1
for how we perform cross-validation).
Competitive methods. For this case study, we compare Poem
against two state-of-the-art deep-learning-based vulnerability de-
tection models: uVuldeepecker [66] and Lin et al. [40]. Results of
this case study is presented in Section 4.4.
3.5 Performance Report
To measure execution time for case studies 1-3, we run each test
case repeatedly until the 95% confidence bound per model per input
is smaller than 5%. For case study 4, we report the accuracy, and
the false-positive and the false-negative rates. A false positive is
when the model predicts a code snippet has a vulnerability while
it does not, and a false-negative is when the model fails to detect
a vulnerable code sample. For code vulnerability detection, we
would like to achieve high accuracy with low false-positive and
false-negative rates.
We report the geometric mean across experimental runs or test
cases. The geometric mean is widely considered to be more ro-
bust and meaningful than the arithmetic mean for performance
measurements [27].
3.6 Implementation and Training Settings
We implement Poem on Tensorflow v1.8. To build the AST, we use
Clang [2] for OpenCL, C and Swift, and ANTLR [ 1] for Java. To
extract the CDFG, we use LLVM [ 3] for OpenCL, C and Swift, and
Soot [5] for Java.
All deep learning models were trained in an end-to-end fashion
using minibatch stochastic gradient descent (SGD) and the Adam
optimizer [37]. For fair comparison, we use NNI [ 4], an AutoML
tool to determine the training hyper-parameters, including the
learning rate and batch size unless these are given in the published
implementation. We train the models using two NVIDIA GTX 1080
GPUs. Note that we set aside 1/10th of the training data to use for
validation during the training process. Training terminates when
the loss does not improve within 20 consecutive training epochs,
or reaches a 99% accuracy on the valuation set, or meets the ter-
mination criteria given in the published implementation. For each
model, we use the configuration that yields the best results on the
validation set. Because we use the geometric mean instead of the
arithmetic mean and as the deep learning models are initialized
with random weights, performance numbers can deviate from the
source publications.
4 EXPERIMENTAL RESULTS
In this section, we first present results for the four case studies de-
scribed in Section 3, showing that Poem outperforms all alternative
NVIDIA GTX 970 AMD Tahiti 7970 NVIDIA Titan XP
1.0
1.5
2.0
2.5
3.0 Speedup
Grewe et al. Inst2vec GNN-AST
DeepTune GNN-CDFG POEM
Figure 5: Speedups (geometric mean) over a GPU-only base-
line for heterogeneous mapping (case study 1).Poem outper-
forms alternative methods on both platforms.
NVIDIA GTX 970 AMD Tahiti 7970 NVIDIA Titan XP
50%
60%
70%
80%
90% Accuracy
Grewe et al. Inst2vec GNN-AST
DeepTune GNN-CDFG POEM
Figure 6: Prediction accuracy for heterogeneous mapping
(case study 1).Poem gives the highest prediction accuracy.
methods in each task. We then provide a detailed analysis of the
working mechanism of Poem.
4.1 Case Study 1: Heterogeneous Mapping
Figure 5 shows the performance improvement. Results on NVIDIA
GTX 970 and AMD Tahiti 7970 was obtained on the DeepTune
dataset, while results on NVIDIA Titan XP were obtained by first
training the models on additional synthetic OpenCL kernels and
then testing the trained model on hand-written kernels from the
DeepTune dataset. The baseline is a GPU-only strategy that always
uses the GPU for kernel execution. As we report the geometric
mean, the speedup number can deviate from the sour publications.
For this case study, Poem outperforms all other approaches on
all platforms. On NVIDIA GTX 970, we observe small performance
improvement over the GPU-only baseline for all methods. On this
platform, Poem gives the best overall speedup of 1.32, albeit its
improvement is relatively small. By contrast, the benefit of using
the right device on the AMD Tahiti 7970 GPU is larger. On this
platform, Poem achieves a mean speedup of 1.8x, which translates
to an improvement of 13% over the second-best model, GNN-CDFG.
All deep-learning models benefit from additional training data on
the NVIDIA Titan XP platform, where Poem delivers the highest
mean speedup of 2.6x.
If we look at the prediction accuracy in Figure 6, we see thatPoem
also delivers the highest accuracy on all platforms; albeitPoem gives
modest accuracy improvement on the DeepTune dataset because
the number of training samples is small. However, when using a
larger training dataset, it is able to boost the prediction accuracy
from 82% to 89% over DeepTune and Inst2vec. We note that on the
1.0
1.1
1.2
1.3
AMD Radeon
HD 5900
AMD
Tahiti 7970
NVIDIA
GTX 480
Speedup
Magni GNN-CDFG DeepTune
GNN-AST Inst2vec POEM
0.9
0.8
0.7
NVIDIA
Titan XP GeoMean
NVIDIA
Tesla K20c
Figure 7: Performance of thread coarsening (case study 2).
Poem is the only method that gives an overall speedup.
AMD platform, for most of the cases that Poem mispredicts, the
difference in performance between the GPU and the CPU is small.
As a result, such a misprediction has little impact on the overall
performance. Overall, Poem delivers the highest mean speedup and
prediction accuracy across all evaluation platforms and datasets.
4.2 Case Study 2: Thread Coarsening
Figure 7 shows the speedup for thread coarsening over a baseline
that uses a coarsen factor of 1 (i.e., no coarsening). In this task, we
apply transfer learning to port the deep learning model trained
for case study 1 (using the DeepTune dataset) to predicting thread
coarsening .Poemis the only model that consistently delivers perfor-
mance improvement across GPU platforms, albeit the improvement
is modest. The modest improvement is expected as a theoretically
perfect predictor would only achieve a mean speedup of 1.28x. The
GNN variants deliver poor performance for this task, leading to
overall slowdown on three out of five evaluation platforms. Deep-
Tune gives marginal improvements on two GPU platforms, but its
performance is far from Poem on these platforms. Notably, on Tesla
K20c, Poem and Inst2vec are the only predictive models that give a
speedup. On HD5900 and Titan XP, Poem gives an overall speedup
of over 1.2x, improving DeepTune by 20%. Overall, Poem achieves
consistently higher speedups when compared to that of other meth-
ods. This experiment shows Poem can effectively support transfer
learning when the training corpus is small.
4.3 Case Study 3: Loop Vectorization
Figure 8 shows the speedup for predicting loop vectorization con-
figurations. The baseline is the LLVM default loop vectorization set-
ting. Machine learning-based methods outperform the polyhedral-
based Polly optimizer for all test loops except for L3, where Poem
still gives better performance than Polly. GNN and DeepTune match
or outperform NeuroVectorizer on several high-speedup test cases
(L8, L9, L10), despite that NeuroVectorizer incur significantly more
expensive compile-time overhead. However, GNN and DeepTune
give no performance improvement or even slow down over LLVM
“-O3" for several loops, including L3, L7, and L11. After having a
close examination of these cases, we found that these loops con-
tain a branch with the loop body that does not captured by Deep-
Tune and the simply graph representation used by GNN. Poem
gives or matches the best performance for all test cases, except for
L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 L12 GeoM
1.0
2.0
3.0
4.0 Speedup
Polly GNN-AST GNN-CDFG
DeepTune NeuroVectorizer POEM
0.9
0.8
Figure 8: Speedup over the LLVM default loop vectorizer
(case study 3).Poem delivers the highest overall speedup and
is the best-performing model for most of the testing loops.
Table 4: Performance for code vulnerability detection (case
study 4).
Metrics uVuldeepecker Lin et al. POEM
Accuracy 80.0% 88.0% 90.9%
FPR 31.6% 30.5% 3.1%C
FNR 9.4% 7.1% 8.9%
Java
Accuracy 78.3% 72.0% 84.4%
FPR 27.7% 45.4% 20.7%
FNR 15.7% 10.3% 8.1%
Accuracy 77.7% 74% 89.0%
FPR 21.0% 23.2% 19.3%Swift
FNR 23.6% 28.3% 9.9%
L11. For L11, the performance of Poem is not far from the best-
performing model (i.e., NeuroVectorizer). Overall,Poem gives an
average speedup of 2.10x, leading to 13% improvement over Neu-
roVectorizer. The performance of Poem translates into 96% of the
2.17x speedup found by exhaustively trying all the vectorization
configurations considered in this work.
Compared to NeuroVectorizer,Poem also has orders of magni-
tude less compile-time overhead (under a second versus 15 minutes
compile time for the 12 testing loops). This indicates that the repre-
sentation learned by Poem can effectively support the downstream
loop vectorization task. An interesting question is if the embed-
dings learned by Poem can be used to improve the reinforcement
search framework of NeuroVectorizer. We leave this as future work.
4.4 Case Study 4: Code Vulnerability Detection
In this experiment, we apply Poem to detect vulnerabilities of
function-level source code written in C, Java and Swift.
Table 4 reports the higher-is-better accuracy metric and the
two lower-is-better metrics: the false-positive rate (FPR) and the
false-negative rate (FNR). Poem delivers the best accuracy with the
lowest FPR and FNR. On the C dataset, Poem has an accuracy of
over 90.9%, with a low FPR of 3.1%. Poem has a modestly higher
FNR compared to Lin et al. , but it has a significantly lower FPR
(3.1% vs 30.5%). A low PPR is important as it reduces the developer’s
time in investigating false alarms. For the Java and Swift datasets,
all three approaches have relatively lower accuracy and higher
FPR. This is largely due to the more complex language features like
overriding of an external method. Such information is not captured
by the initial vertex embedding method (word2vec used by the three
13579
70%
75%
80%
85%
90% Accuracy
Number of embedding layers
Figure 9: Performance ofPoem for heterogeneous mapping
on AMD Tahiti 7970 as the number of embedding layers
changes.
PolyBench
NVIDIA SDK
Radinia
Parboil
SHOC
NPB
AMD SDK
GeoMean
0%
20%
40%
60%
80%
100% Accuracy
RGCN P-vanilla-AST P-AST
P-CONCAT P-CDFG POEM
Figure 10: Implementation variants of Poem on AMD
Tahiti 7970 for heterogeneous mapping. Our implementa-
tion choices ofPoem give the best overall performance.
methods). Nonetheless, Poem outperforms the other two methods
across language datasets by successfully detecting more vulnerable
code samples with the lowest FPR.
4.5 Model Analysis
4.5.1 Impact of embedding layers. Using heterogeneous mapping
as an example, Figure 9 shows how the performance of Poem
changes on the DeepTune dataset on AMD Tahiti 7970. Increasing
the number of embedding layers (and hence the number of neigh-
borhood aggregation iterations - see Section 2.6) can improve the
performance. However, the accuracy reaches a plateau when using
five embedding layers and using more than that leads to overfitting
and a drop of prediction accuracy on the validation set. The vali-
dation dataset is part of the training data but not the test dataset
which is always not seen at the model design and training stage.
We use the same procedure to determine the optimal number of em-
bedding layers for each task, by comparing the resulting accuracy
on the validation set. We found that using 3 to 5 embedding layers
give a good performance for our tasks and using more embedding
layers would require a larger training dataset.
4.5.2 Impact of implementation choices. In this experiment, we
compare Poem to several variant implementations for performing
heterogeneous mapping on AMD Tahiti 7970. The first is RGCN
[50] that applies to the 10 code relations described in Table 2.3.
Unlike Poem, RGCN takes in a single adjacency matrix that encodes
all the node connectivities of the AST and CDFG, and the edge is
encoded using a one-hot vector for representing different relations.
The second variant, referred to as p-vanilla-AST, operates on a
standard AST (without the additional edges described in Sec. 2.2).
The third variant, referred to as P-AST, operates only on the aug-
mented AST. The fourth variant, referred to as P-CDFG, operates
only on the CDFG. This experiment uses heterogeneous mapping as
a case study and is designed to evaluate the impact of exacting code
information. The final variant, referred to as P-CONCAT, learns in-
dividual embeddings for each relation graph and then concatenates
the individual embeddings for prediction. This evaluation is also
known as the ablation study [28].
The results are given in Figure 10. While RGCN support mod-
elling of multiple edge relations, it is less effective for modeling a
combined graph from the AST and the CDFG. Its low performance
is largely due to two reasons. Firstly, a simple combination of the
AST and the CDFG, which are two heterogeneous graphs, to fit the
RGCN can confuse the learning algorithm. Second, RGCN requires
a large number of learnable parameters and is hard to generalize.
Figure 10 also shows that using the standard AST is inadequate for
capturing the essential program structures. By augmenting the AST
with additional control and data flow information, P-AST improves
P-vanilla-AST by 4%. However, using the AST or CDFG alone is
insufficient, as both give an accuracy of less than 75%. P-CDFG cor-
rectly predicts 20 kernels where P-AST fails, while it fails on other
12 kernels where P-AST succeeds. P-CONCAT also gives lower
performance compared to Poem, suggesting that simply combining
the embeddings of relation graphs is less effective. This experiment
suggests that we need to utilize and aggregate the information of
the AST and the CDFG during the learning process. Poem offers ex-
actly such capabilities, leading the best overall performance. It also
shows that our multi-relational graph learning method improves a
single concatenation strategy.
4.5.3 Embedding visualization. In an attempt to examine the learned
code representation qualitatively, we provide a visualization of the
t-SNE-transformed feature representations [41] extracted by the
Poem-GNN pre-trained on the DeepTune heterogeneous mapping
dataset. The representation exhibits discernible clustering in the
projected 2D space as shown in Figure 11. Note that these clusters
(with two different node colors) correspond to the two labels (CPU
and GPU) of the dataset, verifying the model’s discriminative power
across different classes for this dataset. As can be seen from the
diagram, the embeddings learned by Poem is more discriminative
than the ones given by other methods, leading to a clearer linear
boundary between the two classes (CPU and GPU).
4.5.4 Training overhead. The time for training Poem is dominated
by training data collection. For case study 1, it took less than 24
hours to profile over synthetic 10,000 benchmarks for labeling the
data. The time in model training and hyper-parameter tuning is
less than 12 hours using two modest NVIDIA 1080 GPUs on 10,000
samples. Since training is only performed once, it is a one-off cost.
5 DISCUSSIONS
Naturally, there is room for future work and further improvement.
We discuss a few points here.
(a) Poem-GNN
 (b) GNN-AST
(c) GNN-CDFG
 (d) DeepTune
Figure 11: Visualization of the learned program representa-
tions for heterogeneous device mappings. We map the high-
dimensional embedding space to a 2-dimensional space to
aid clarity using t-SNE. The embeddings learned byPoem is
more discriminative than the ones given by other methods,
leading to a clearer boundary between the two classes.
Model interpretability.Machine learning techniques, in general,
have the problem of relying on black boxes. This problem is just as
true for Poem. One way to gain insight into why the model fails to
produce the desired result is to train an interpretable model (or so
called surrogate models) like linear regressor to approximate the
predictions of the underlying black-box model [49].
Training samples.Deep neural networks typically require a large
volume of training data to learn over. However, there is often a
shortage of benchmarks. Therefore, work on benchmark synthesis
[21] is orthogonal to our approach.
Training overhead.Profiling training benchmarks to generate
training data could be expensive. One way of reducing the training
overhead is to use active learning [ 46] to only profile and label
training benchmarks that are likely to improve the performance of
the machine-learned model.
Memory footprint.Like all GNN approaches, the memory foot-
print of Poem increases as the graph size increases. However, we can
reduce memory pressure by using sampling methods like Graph-
SAGE for batched training [34], i.e., operating on a subgraph of the
entire program at a time.
Other application domains.We have shown that Poem can be
generalized across a range of tasks. We envision that Poem can
be applied to other applications which are beyond the scope of
this work. It can be applied to detect malicious code by looking for
suspicious and obfuscated patterns. It can be extended to regression-
based problems like predicting the potential speedup for a code
transformation option. A particularly interesting research direction
would be to extend Poem to model the program structure at the
binary level [59] for tasks like program verification and security.
6 RELATED WORK
Machine learning has demonstrated promises in automating the pro-
cess of decision model construction for various code optimization
tasks [54]. Many prior studies have shown that machine-learned
models can outperform expert-crafted heuristics [ 9, 14, 15, 23–
25, 48, 51, 55, 56, 62, 63]. However, a significant barrier still exists –
programs must be represented as a set of features that serve as in-
puts to a machine learning tool. Traditionally, this requires experts’
involvement to extract the crucial elements of the program.
Prior work automated this process of finding code representa-
tions by searching useful information from the compiler IR [38, 44].
These approaches still require experts to manually define the search
space of a particular compiler IR implementation. As such, they
offer little help in removing human experts from the loop.
In recent year, deep learning has been employed for modeling
program structures. A key advantage of deep learning is that it can
automatically find the right feature representations from training
data without human involvement [6]. Prior work for deep learning
on code typically employ recurrent neural networks (RNN) like
LSTM or GRU to extract program representations from token se-
quences. For examples, DeepTune uses LSTM to extract program
representations from tokenrized OpenCL code [20] and Inst2vec
applies LSTM to sequentialized IR graphs [ 12]. Other work uses
RNNs to summarize representations from the AST [ 8, 16, 33] or
sparse matrices [ 65]. While RNN is a proven technique for nat-
ural language processing, it is mainly designed for processing a
sequential sequence [ 35]. The prolbem for treating source code
as a sequential token sequence is that statements can easily be
separated by hundreds of lines of irrelevant code in sequential
representations. As a result, RNNs are ineffective in modeling the
complex program control and data flows - which should be better
represented as a graph structure instead of a sequence of tokens.
A graph representation not only enables the learning framework
to leverage the well-defined program structures but also facilitates
propagating information across the graph in a manner similar to
typical compiler analyses.
An early attempt to use program graph structures for code opti-
mization is presented in [47]. This approach requires careful hand-
tuned features at the basic block level to extract information from
the program graph. To predict an optimization option, it measures
the similarity of the input program graph with the graphs of the
training datasets. This strategy requires the training data graphs
to be shipped with the compiler and hence does not scale well as
the training program size increases. Furthermore, this approach
does not abstract the language semantics and syntax a sufficiently
high level, leading to expensive computation complexity for graph
matching. Our approach eliminates the need for manual feature
tuning, with a constant, lower-cost compile overhead, which is
independent of the size of the training set.
Some of the most recent work has employed the recently pro-
posed GNN to model code structures. For example, Miltiadis et al.
[7] use GNN to model the AST to identify the misuse of names. The
recent work presented in [13], which uses the GNN to model the
AST or CFDG for OpenCL program optimization, is most closely
related. This approach uses a vanilla GNN which treat all relation-
ships equally as a single graph edge type, whether it is a node
connection, or a data or control flow. This strategy misses much of
the information that could otherwise be captured. Our approach
advances [13] by capturing the different relationships within a uni-
fied learning framework. Compared to [13], Poem can leverage a
richer set of information by combining the AST and CFDG, leading
to significantly better and more reliable performance. When prepar-
ing the final version of this paper, we noticed the pre-print version
of ProGraML [18]. This approach also employs a GNN to the pro-
gram’s control and data flow graph extracted from the compiler IR.
Our work was conducted independently and perhaps concurrently
to ProGraML. Unlike ProGraML, Poem also utilizes information
available in the AST, and uses individual learnable functions to
model different code relationships.
The GNN family includes a diverse class of neural network ar-
chitectures based on recurrent units [ 39, 57] and convolutional
[22] and attention [52] methods. Poem represents the first work for
leveraging GNNs to learn over multiple code relationships.
7 CONCLUSION
This paper has presented Poem, a general learning framework for
supporting building machine-learned heuristics for code analysis
and optimization. Poem is designed to automatically extract useful
representations of programs to be used as inputs for machine learn-
ing tools. At the core of Poem is a novel Graph Neural Network
(GNN) that can distinguish and aggregate information from differ-
ent relationships within the program control and data flow graph
and the abstract syntax tree. By providing a way to abstract and
aggregate information from a well-structured program graph rep-
resentation, our approach can capture a richer set of syntactic and
semantic information than prior deep-learning-based approaches.
We demonstrate the generalization ability of Poem by applying
it to four representative program optimization and analysis tasks
spanning different programming languages. We perform extensive
experiments to compare Poem with state-of-the-art approaches
for each task. Experimental results show that Poem consistently
outperforms prior methods, setting new state-of-the-art results for
these tasks.
ACKNOWLEDGEMENT
This work was supported in part by the National Natural Science
Foundation of China (NSFC) under grant agreements 61972314,
61672427, 61872294 and 61972408, International Cooperation Projects
of Shaanxi Province under grant agreements 2019KW-009 and
2020KWZ-013, an Ant Financial Science funded project and a UK
Royal Society International Collaboration Grant.
REFERENCES
[1] [n. d.]. ANTLR (ANother Tool for Language Recognition) . https://www.antlr.org/.
([n. d.]).
[2] [n. d.]. Clang: A language front-end and tooling infrastructure. https://clang.
llvm.org/. ([n. d.]).
[3] [n. d.]. LLVM: A collection of modular and reusable compiler and toolchain
technologies. https://llvm.org/. ([n. d.]).
[4] [n. d.]. Neural Network Intelligence. https://nni.readthedocs.io/. ([n. d.]).
[5] [n. d.]. Soot: A framework for analyzing and transforming Java applications.
http://sable.github.io/soot/. ([n. d.]).
[6] Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. 2018.
A survey of machine learning for big code and naturalness. ACM Computing
Surveys (CSUR) 51, 4 (2018), 1–37.
[7] Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2018. Learning
to represent programs with graphs. In ICLR.
[8] Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2019. code2seq: Generating
sequences from structured representations of code. In ICLR.
[9] Jason Ansel, Cy Chan, Yee Lok Wong, Marek Olszewski, Qin Zhao, Alan Edel-
man, and Saman Amarasinghe. 2009. PetaBricks: a language and compiler for
algorithmic choice. ACM Sigplan Notices 44, 6 (2009), 38–49.
[10] Amir H Ashouri, William Killian, John Cavazos, Gianluca Palermo, and Cristina
Silvano. 2018. A survey on compiler autotuning using machine learning. ACM
Computing Surveys (CSUR) 51, 5 (2018), 1–42.
[11] Francesco Barchi, Gianvito Urgese, Enrico Macii, and Andrea Acquaviva. 2019.
Code Mapping in Heterogeneous Platforms Using Deep Learning and LLVM-IR.
In 2019 56th ACM/IEEE Design Automation Conference (DAC) . 1–6.
[12] Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. 2018. Neural code
comprehension: A learnable representation of code semantics. In Advances in
Neural Information Processing Systems . 3585–3597.
[13] Alexander Brauckmann, Andrés Goens, Sebastian Ertel, and Jeronimo Castrillon.
2020. Compiler-Based Graph Representations for Deep Learning Models of Code.
In Proceedings of the 29th International Conference on Compiler Construction (CC
2020).
[14] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen
Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. 2018. {TVM}:
An automated end-to-end optimizing compiler for deep learning. In 13th
{USENIX}Symposium on Operating Systems Design and Implementation ( {OSDI}
18). 578–594.
[15] Tianqi Chen, Lianmin Zheng, Eddie Yan, Ziheng Jiang, Thierry Moreau, Luis Ceze,
Carlos Guestrin, and Arvind Krishnamurthy. 2018. Learning to optimize tensor
programs. In Advances in Neural Information Processing Systems . 3389–3400.
[16] Xinyun Chen, Chang Liu, and Dawn Song. 2018. Tree-to-tree neural networks
for program translation. In Advances in neural information processing systems .
2547–2557.
[17] Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014.
Empirical evaluation of gated recurrent neural networks on sequence modeling.
In NIPS 2014 Workshop on Deep Learning .
[18] Chris Cummins, Zacharias V Fisches, Tal Ben-Nun, Torsten Hoefler, and Hugh
Leather. 2020. ProGraML: Graph-based Deep Learning for Program Optimization
and Analysis. arXiv preprint arXiv:2003.10536 (2020).
[19] Chris Cummins, Pavlos Petoumenos, Alastair Murray, and Hugh Leather. 2018.
Compiler fuzzing through deep learning. InProceedings of the 27th ACM SIGSOFT
International Symposium on Software Testing and Analysis . 95–105.
[20] Chris Cummins, Pavlos Petoumenos, Zheng Wang, and Hugh Leather. 2017. End-
to-end deep learning of optimization heuristics. In 26th International Conference
on Parallel Architectures and Compilation Techniques (PACT) .
[21] Chris Cummins, Pavlos Petoumenos, Zheng Wang, and Hugh Leather. 2017.
Synthesizing benchmarks for predictive modeling. In2017 IEEE/ACM International
Symposium on Code Generation and Optimization (CGO) . 86–99.
[22] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolu-
tional neural networks on graphs with fast localized spectral filtering. InAdvances
in neural information processing systems . 3844–3852.
[23] Christina Delimitrou and Christos Kozyrakis. 2014. Quasar: resource-efficient and
QoS-aware cluster management. ACM SIGPLAN Notices 49, 4 (2014), 127–144.
[24] Yufei Ding, Jason Ansel, Kalyan Veeramachaneni, Xipeng Shen, Una-May O’Reilly,
and Saman Amarasinghe. 2015. Autotuning algorithmic choice for input sen-
sitivity. In Proceedings of the 36th ACM SIGPLAN Conference on Programming
Language Design and Implementation . 379–390.
[25] Murali Krishna Emani, Zheng Wang, and Michael FP O’Boyle. 2013. Smart,
adaptive mapping of parallelism in the presence of external workload. In Pro-
ceedings of the 2013 IEEE/ACM International Symposium on Code Generation and
Optimization (CGO). IEEE, 1–10.
[26] Common Weakness Enumeration. 2019. 2019 CWE Top 25 Most Dangerous
Software Errors. https://cwe.mitre.org/top25/archive/2019/2019_cwe_top25.html.
(2019).
[27] Wolfgang Ertel. 1994. On the definition of speedup. In Proceedings of the In-
ternational Conference on Parallel Architectures and Languages Europe . Springer,
289–300.
[28] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich
feature hierarchies for accurate object detection and semantic segmentation. In
Proceedings of the IEEE conference on computer vision and pattern recognition .
580–587.
[29] S. Grauer-Gray, L. Xu, R. Searles, S. Ayalasomayajula, and J. Cavazos. 2012.
Auto-tuning a High-Level Language Targeted to GPU Codes. In InPar. https:
//doi.org/10.1109/InPar.2012.6339595
[30] Dominik Grewe, Zheng Wang, and Michael FP O’Boyle. 2013. Portable mapping
of data parallel programs to OpenCL for heterogeneous systems. InProceedings of
the 2013 IEEE/ACM International Symposium on Code Generation and Optimization
(CGO). 1–10.
[31] Tobias Grosser et al. 2012. Polly - Performing Polyhedral Optimizations on a
Low-Level Intermediate Representation. Parallel Processing Letters (2012).
[32] Matthew R Guthaus, Jeffrey S Ringenberg, Dan Ernst, Todd M Austin, Trevor
Mudge, and Richard B Brown. 2001. MiBench: A free, commercially representative
embedded benchmark suite. In Proceedings of the fourth annual IEEE international
workshop on workload characterization. WWC-4 (Cat. No. 01EX538) . IEEE, 3–14.
[33] Ameer Haj-Ali, Nesreen K Ahmed, Ted Willke, Yakun Sophia Shao, Krste
Asanovic, and Ion Stoica. 2020. NeuroVectorizer: end-to-end vectorization with
deep reinforcement learning. In Proceedings of the 18th ACM/IEEE International
Symposium on Code Generation and Optimization . 242–255.
[34] Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation
learning on large graphs. In Advances in neural information processing systems .
1024–1034.
[35] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. In
Proceedings of the Neural computation 9, 8 (1997), 1735–1780.
[36] Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018. Deep code comment
generation. In Proceedings of the 26th Conference on Program Comprehension .
200–210.
[37] Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic opti-
mization. arXiv preprint arXiv:1412.6980 (2014).
[38] Hugh Leather, Edwin Bonilla, and Michael O’Boyle. 2009. Automatic feature gen-
eration for machine learning based optimizing compilation. In 2009 International
Symposium on Code Generation and Optimization . 81–91.
[39] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2015. Gated
graph sequence neural networks. arXiv preprint arXiv:1511.05493 (2015).
[40] Guanjun Lin, Jun Zhang, Wei Luo, Lei Pan, Olivier De Vel, Paul Montague, and
Yang Xiang. 2019. Software Vulnerability Discovery via Learning Multi-domain
Knowledge Bases. IEEE Transactions on Dependable and Secure Computing (2019).
[41] Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE.
Journal of machine learning research 9, Nov (2008), 2579–2605.
[42] Alberto Magni, Christophe Dubach, and Michael O’Boyle. 2014. Automatic
optimization of thread-coarsening for graphics processors. In Proceedings of the
23rd international conference on Parallel architectures and compilation techniques .
455–466.
[43] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.
Distributed representations of words and phrases and their compositionality. In
Advances in neural information processing systems . 3111–3119.
[44] Mircea Namolaru, Albert Cohen, Grigori Fursin, Ayal Zaks, and Ari Freund.
2010. Practical aggregation of semantical program properties for machine learn-
ing based optimization. In Proceedings of the 2010 international conference on
Compilers, architectures and synthesis for embedded systems . 197–206.
[45] Dorit Nuzman, Ira Rosen, and Ayal Zaks. 2006. Auto-vectorization of interleaved
data for SIMD. ACM SIGPLAN Notices 41, 6 (2006), 132–143.
[46] William F Ogilvie, Pavlos Petoumenos, Zheng Wang, and Hugh Leather. 2017.
Minimizing the cost of iterative compilation with active learning. In 2017
IEEE/ACM International Symposium on Code Generation and Optimization (CGO) .
IEEE, 245–256.
[47] Eunjung Park, John Cavazos, and Marco A Alvarez. 2012. Using graph-based
program characterization for predictive modeling. In Proceedings of the Tenth
International Symposium on Code Generation and Optimization . 196–206.
[48] Jie Ren, Ling Gao, Hai Wang, and Zheng Wang. 2017. Optimise web browsing on
heterogeneous mobile platforms: a machine learning based approach. In IEEE
INFOCOM 2017-IEEE Conference on Computer Communications . IEEE, 1–9.
[49] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why should i
trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd
ACM SIGKDD international conference on knowledge discovery and data mining .
1135–1144.
[50] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan
Titov, and Max Welling. 2018. Modeling relational data with graph convolutional
networks. In European Semantic Web Conference .
[51] Georgios Tournavitis, Zheng Wang, Björn Franke, and Michael FP O’Boyle. 2009.
Towards a holistic approach to auto-parallelization: integrating profile-driven par-
allelism detection and machine-learning based mapping. InProceedings of the 30th
ACM SIGPLAN Conference on Programming Language Design and Implementation .
177–187.
[52] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro
Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint
arXiv:1710.10903 (2017).
[53] Zheng Wang, Dominik Grewe, and Michael FP O’boyle. 2014. Automatic and
portable mapping of data parallel programs to opencl for gpu-based heteroge-
neous systems. ACM Transactions on Architecture and Code Optimization (TACO)
11, 4 (2014), 1–26.
[54] Zheng Wang and Michael O’Boyle. 2018. Machine learning in compiler optimiza-
tion. Proc. IEEE 106, 11 (2018), 1879–1901.
[55] Zheng Wang and Michael FP O’Boyle. 2009. Mapping parallelism to multi-cores:
a machine learning based approach. In Proceedings of the 14th ACM SIGPLAN
symposium on Principles and practice of parallel programming . 75–84.
[56] Zheng Wang and Michael FP O’Boyle. 2010. Partitioning streaming parallelism
for multi-cores: a machine learning based approach. In Proceedings of the 19th
international conference on Parallel architectures and compilation techniques . 307–
318.
[57] Yuting Wu, Xiao Liu, Yansong Feng, Zheng Wang, Rui Yan, and Dongyan Zhao.
2019. Relation-aware entity alignment for heterogeneous knowledge graphs.
In Proceedings of the 28th International Joint Conference on Artificial Intelligence .
5278–5284.
[58] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018. How powerful
are graph neural networks? arXiv preprint arXiv:1810.00826 (2018).
[59] Xiaojun Xu, Chang Liu, Qian Feng, Heng Yin, Le Song, and Dawn Song. 2017.
Neural network-based graph embedding for cross-platform binary code similarity
detection. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and
Communications Security . 363–376.
[60] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transfer-
able are features in deep neural networks?. In Advances in neural information
processing systems . 3320–3328.
[61] Yue Yu, Jie Chen, Tian Gao, and Mo Yu. 2019. DAG-GNN: DAG Structure Learning
with Graph Neural Networks. In Proceedings of the 36th International Conference
on Machine Learning .
[62] Peng Zhang, Jianbin Fang, Tao Tang, Canqun Yang, and Zheng Wang. 2018.
Auto-tuning streamed applications on intel xeon phi. In 2018 IEEE International
Parallel and Distributed Processing Symposium (IPDPS) . IEEE, 515–525.
[63] Peng Zhang, Jianbin Fang, Canqun Yang, Chun Huang, Tao Tang, and Zheng
Wang. 2020. Optimizing streaming parallelism on heterogeneous many-core
architectures. IEEE Transactions on Parallel and Distributed Systems 31, 8 (2020),
1878–1896.
[64] Zhilu Zhang and Mert Sabuncu. 2018. Generalized cross entropy loss for training
deep neural networks with noisy labels. In Proceedings of the Advances in neural
information processing systems . 8778–8788.
[65] Yue Zhao, Jiajia Li, Chunhua Liao, and Xipeng Shen. 2018. Bridging the gap
between deep learning and sparse matrix format selection. In Proceedings of the
23rd ACM SIGPLAN symposium on principles and practice of parallel programming .
94–108.
[66] Deqing Zou, Sujuan Wang, Shouhuai Xu, Zhen Li, and Hai Jin. 2019.
𝜇VulDeePecker: A Deep Learning-Based System for Multiclass Vulnerability
Detection. IEEE Transactions on Dependable and Secure Computing (2019).

Ask Me Anything: Dynamic Memory Networks
for Natural Language Processing
Ankit Kumar Ozan Irsoy Jonathan Su James Bradbury Robert English
Brian Pierce Peter Ondruska Ishaan Gulrajani Richard Socher
ﬁrstname@metamind.io
MetaMind
Palo Alto, CA
Abstract
Most tasks in natural language processing can be cast into question answering
(QA) problems over language input. We introduce the dynamic memory network
(DMN), a uniﬁed neural network framework which processes input sequences
and questions, forms semantic and episodic memories, and generates relevant an-
swers. Questions trigger an iterative attention process which allows the model to
condition its attention on the result of previous iterations. These results are then
reasoned over in a hierarchical recurrent sequence model to generate answers.
The DMN can be trained end-to-end and obtains state of the art results on sev-
eral types of tasks and datasets: question answering (Facebook’s bAbI dataset),
sequence modeling for part of speech tagging (WSJ-PTB), and text classiﬁcation
for sentiment analysis (Stanford Sentiment Treebank). The model relies exclu-
sively on trained word vector representations and requires no string matching or
manually engineered features.
1 Introduction
Question answering (QA) is a complex natural language processing task which requires an under-
standing of the meaning of a text and the ability to reason over relevant facts. Most, if not all, tasks
in natural language processing can be cast as a question answering problem: high level tasks like
machine translation ( What is the translation into French? ); sequence modeling tasks like named
entity recognition [1] (NER) ( What are the named entity tags in this sentence? ) or part of speech
tagging (POS) ( What are the part of speech tags? ); classiﬁcation problems like sentiment analy-
sis [2] (What is the sentiment? ); even multi-sentence joint classiﬁcation problems like coreference
resolution (Who does ”their” refer to?).
The dynamic memory network (DMN) is a neural network based model which can be trained in an
end-to-end fashion for any QA task using raw input-question-answer triplets. Fig. 1 shows input
and question test sequences followed by answers given by the model. The DMN is illustrated in
Fig. 2. Generally, it can solve sequence tagging tasks, classiﬁcation problems, sequence to sequence
tasks, and question answering tasks that require transitive reasoning. The DMN ﬁrst processes all
input, question and answer texts into sequences of semantic vector representations. The question
representation triggers an iterative attention process that searches the input and retrieves relevant
facts. The DMN then reasons over retrieved facts and provides an answer sequence model with an
appropriate summary.
1
arXiv:1506.07285v1  [cs.CL]  24 Jun 2015
I: Jane went to the hallway. I: Everybody is happy.
I: Mary walked to the bathroom. Q: What’s the sentiment?
I: Sandra went to the garden. A: positive
I: Daniel went back to the garden. Q: What are the POS tags?
I: Sandra took the milk there. A: NN VBZ JJ .
Q: Where is the milk? I: The answer is far from obvious.
A: garden Q: In French?
A: La r ´eponse est loin d’ˆetre ´evidente.
Figure 1: Example inputs and questions together with answers generated by a dynamic memory
network.
2 The Dynamic Memory Network Framework
The DMN is a general modeling framework for asking questions over inputs. We will give an
overview of the full model, and then go into further detail for each module, as well as present our
speciﬁc instantiation of each module for natural language processing. The goal of the DMN is to
ﬁrst compute a vector representation of an input, given a question, and to then generate the correct
answer. It contains the following modules:
Input Module: This module processes raw inputs and maps them to a representation that is useful
for asking questions about this input. The input may be, for instance, an image, video, or
audio signal. We focus on NLP in this paper. Hence, the input may be a sentence, a long
story, a movie review, a news article, or all of Wikipedia.
Semantic Memory Module: Semantic memory stores general knowledge about concepts and facts.
For example, it might contain information about what a hang glider is. Initialization strate-
gies such as distributed word vectors (Glove [3], Word2Vec [4]) are popular semantic mem-
ory components that have been shown to improve performance on many NLP tasks. More
complex information can be stored in the form of knowledge bases that capture relation-
ships in the form of triplets [5] or gazetteers, which have been useful for tasks such as
named entity recognition or question answering [6].
Question Module: The question module computes a representation of a question such as Where
did the author ﬁrst ﬂy? This representation triggers the episodic memory module to start
an iterative attention process over facts from the input sequence.
Episodic Memory Module: This is the central part of the DMN. A question draws attention to
speciﬁc facts from the input sequence, which are reasoned over to update this module’s
memory state. This process then iterates, with each iteration providing the module with
newly relevant information about the input. In other words, the module has the ability to
retrieve new facts which were thought to be irrelevant in previous iterations. After several
passes the module then summarizes its knowledge and provides the answer module with a
ﬁnal representation to produce an answer.
Answer Module: Given a representation from the episodic memory module, the answer module
generates the model’s predicted answer.
2.1 Input Module
The input module computes a useful representation of the inputs such that relevant facts can be re-
trieved later. Generally, the input module may be thought of as computing the intermediate steps of
a function that eventually returns a ﬁnal vector representation. The input module sends these inter-
mediate values to the episodic memory module, which will complete the computation, conditioned
on the question, by way of its attention mechanism. We refer to the representations that the input
module provides to the episodic memory as facts to be reasoned over by the episodic memory.
In natural language processing, we have a sequence of TI words wI
1,...,w I
TI . The input module
computes the hidden states of a recurrent sequence model [7].
2
Seman&c	  
Memory	  
	  
•  Word	  
vectors	  
•  Knowledge	  
Basis	  
	  
Input	  Text	  Sequence	   Ques&on	  
Episodic	  Memory	   Answer	  
Figure 2: Overview of DMN modules. Communication between them is indicated by arrows and
uses only vector representations. Questions trigger gates which allow vectors for certain input words
or sentences to be given to the episodic memory module. The ﬁnal state of the episodic memory is
the input to the answer module.
We use Glove [3] vectors to capture context-independent representations, and initalize the embed-
dings of the DMN with these values. Word embeddings are given as inputs to the recurrent network
to compute hidden fact states: ct = SEQ MODEL(L[wI
t ],ht−1), where Lis the embedding matrix and
wI
t is the tth word of the input sequence. In particular, we use a gated recurrent network (GRU)
[8, 9]. We also explored the more complex LSTM [10] but it performed similarly and is more com-
putationally expensive. Both work much better than the standard tanh RNN and we postulate that
the main strength comes from having gates that allow the model to suffer less from the vanishing
gradient problem [10].
GRU Deﬁnition: Assume each time step has an input xt and a hidden state ht. We will abbreviate
the below computation with ht = GRU(xt,ht−1):
zt = σ
(
W(z)xt + U(z)ht−1 + b(z)
)
; rt = σ
(
W(r)xt + U(r)ht−1 + b(r)
)
(1)
˜ht = tanh
(
Wxt + rt ◦Uht−1 + b(h)
)
; ht = zt ◦ht−1 + (1−zt) ◦˜ht, (2)
where ◦is an element-wise product, W(z),W(r),W ∈RnH×nI and U(z),U(r),U ∈RnH×nH . The
dimensions nare hyperparameters.
If we subsample the output of the recurrent network, the input module will return just the hidden
states ct that correspond to end-of-sentence markers in the original story. Otherwise, the input
module returns hidden states ct for all words. For notational convenience, in future modules we will
refer to either of these states asct, and say in the experiments section whether or not the subsampling
was used.
2.2 Semantic Memory Module
The semantic memory consists of (i) stored word concepts and (ii) facts about them. We initialize
embeddings to Glove vectors as described above. This module could include gazeteers or other
forms of explicit knowledge bases, but in this work we do not use them.
2.3 Question Module
This module maps a question into a representation that can then be used for querying speciﬁc facts
from the input module. We have questions that consist of sequences of TQ words wQ
t . We compute
a hidden state for each via qt = GRU(L[wQ
t ],qt−1), where the GRU and embedding weights are
shared with the input module. The ﬁnal question vector is deﬁned as q= qTQ.
2.4 Episodic Memory Module
The episodic memory module retrieves facts from the input module conditioned on the question. It
then reasons over those facts to produce a ﬁnal representation that the answer module will use to
generate an answer. We refer to this representation as a memory. Importantly, we allow our module
3
Answer module
Question Module
Semantic Memory
Module 
Episodic Memory
Module
Input Module
Mary got the milk there.
John moved to the bedroom.
Sandra went back to the kitchen.
Mary travelled to the hallway.John got the football there.John went to the hallway.John put down the football.Mary went to the garden.
s1 s2 s3 s4 s5 s6 s7 s8
Where is the fooball?
q
0.0 0.3 0.0 0.0 0.0 0.9 0.0 0.0
0.3 0.0 0.0 0.0 0.0 0.0 1.0 0.0
e1 e2 e3 e4 e5 e6 e7 e8
1 1 1 1 1 1 1 1
e1 e2 e3 e4 e5 e6 e7 e8
2 2 2 2 2 2 2 2
hallway<EOS>m
1
m
2
(Glove vectors)
w1
wT
Figure 3: Real example of an input sentence sequence and the attention gates that are triggered by a
speciﬁc question. Gate values gi
t are shown above the corresponding vectors. The gates change with
each search over inputs. We do not draw connections for gates that are close to zero. See Section
4.1 for details on the dataset that this example comes from.
to take multiple passes over the facts, focusing attention on different facts at each pass. Each pass
produces an episode, and these episodes are then summarized into the memory. Endowing our
module with this episodic component allows its attention mechanism to attend more selectively to
speciﬁc facts on each pass, as it can attend to other important facts at a later pass. It also allows for
a type of transitive inference, since the ﬁrst pass may uncover the need to retrieve additional facts.
For instance, in the example in Fig. 3, we are asked Where is the football? In the ﬁrst iteration,
the model ought attend to sentence 7 ( John put down the football. ), as the question asks about the
football. Only once the model sees that John is relevant can it reason the second iteration should
retrieve where John was. In this example, taken from a true test question on Facebook’s bAbI task,
this behavior is indeed seen. Note that the second iteration has wrongly placed some weight in
sentence 2, which makes some intuitive sense, as sentence 2 is another place John had been.
In its general form, the episodic memory module is characterized by an attention mechanism, a
function which returns an episode given the output of the attention mechanism and the facts from
the input module, and a function that summarizes the episodes into a memory.
In our work, we use a gating function as our attention mechanism. It takes as input, for each passi, a
candidate fact ct, a previous state mi−1, and the question qto compute a gate: gi
t = G(ct,mi−1,q).
The state is updated by way of a GRU: mi = GRU(ei,mi−1), where ei is the computed episode at
pass i. The state may be initialized randomly, but in practice we have found that initializing it to the
question vector itself helps; e.g, m0 = q. The function Greturns a single scalar and is deﬁned as
follows:
z(c,m,q ) = [ c,m,q,c ◦q,c ◦m,|c−q|,|c−m|,cT W(b)q,cT W(b)m] (3)
G(c,m,q ) = σ
(
W(2) tanh
(
W(1)z(c,m,q ) +b(1)
)
+ b(2)
)
(4)
To compute the episode for pass i, we employ a modiﬁed GRU over the sequence of TC facts ct,
endowed with our gates. The episode is the ﬁnal state of the GRU:
hi
t = gi
tGRU(ct,hi
t−1) + (1−gi
t)hi
t−1 (5)
ei = hi
TC (6)
Finally, to summarize the TP episodes ei into a memory, we use the same GRU that updates the
attention mechanism’s state:mi = GRU(ei,mi−1), and we set the memory mas m= mTP . This
is equivalent to setting the memory to simply the attention mechanism’s ﬁnal state, but we have de-
scribed it here as its own computation to highlight the potential modularity of these subcomponents.
For datasets that mark which facts are important for a given question, such as Facebook’s bAbI
dataset, the gates of Eq. 4 can be trained supervised with a standard cross entropy classiﬁcation
4
error function. We also append a special end-of-passes representation to the facts, and stop the iter-
ative attention process if this representation is chosen by the gate function. Otherwise, for datasets
without explicit supervision, we set a maximum number of passes. The whole module is end-to-end
differentiable.
Sequence Modeling
It is straightforward to apply the DMN to sequence modeling. In the sequence modeling task, we
wish to label each word in the original sequence. Therefore, we desire one vector representation
for each word. To this end, we run the DMN in the same way as above, once for each word. When
running the DMN for wordt, instead of setting the episode for passito the ﬁnal state of our modiﬁed
GRU, we take the tth: e.g, for word t, we replace Eq. 6 with ei = hi
t. Note that the gates for the
ﬁrst pass will be the same for each word, as the question is the same. This allows for speed-up in
implementation by computing these gates only once. However, gates for subsequent passes will be
different, as the episodes are different.
The ﬁnal output of the episodic memory module is the memorym, which goes to the answer module.
In the sequence modeling case, each word’s uniquemis sent independently to the answer module.
2.5 Answer Sequence
The answer sequence module decodes the memory into a sequence of words representing the answer.
We use a GRU, and set the initial hidden state to the memorya0 = m. The subsequent hidden states
take as input the last hidden state and the previously predicted output yt−1, as well as the question:
at = GRU([yt−1,q],at−1), y t = softmax(W(a)at) (7)
where W(a) is a standard softmax layer. The output is trained with the cross entropy error classi-
ﬁcation of the correct sequence appended with a special end-of-sequence token.. At test time, we
generate words until an end-of-sequence is generated.
2.6 Training
Training is cast as a supervised classiﬁcation problem to minimize cross entropy error of the answer
sequence. For datasets with gate supervision, such as bAbI, we also include the cross entropy error
of the gates into the overall cost. Because all modules communicate over vector representations and
various types of differentiable and deep neural networks with gates, the entire DMN model can be
trained via backpropagation and gradient descent.
3 Related Work
Given the many shoulders on which this paper is standing and the many applications to which our
model is applied, it is impossible to do related ﬁelds justice.
Deep Learning. There are several deep learning models that have been applied to many differ-
ent tasks in NLP. For instance, recursive neural networks have been used for parsing [11], senti-
ment analysis [2], paraphrase detection [11] and question answering [12] and logical inference [13],
among other tasks. However, because they lack the memory and question modules, a single model
cannot solve as many varied tasks, nor tasks that require transitive reasoning over multiple sen-
tences. Another commonly used model is the chain structured recurrent neural network of the kind
we employ above. Recurrent neural networks have been successfully used in language modeling
[14], speech recognition, and sentence generation from images [15]. Also relevant is the sequence-
to-sequence model used for machine translation by Sutskever et al. [16]. This model uses two
extremely large and deep LSTMs to encode a sentence in one language and then decode the sen-
tence in another language. This sequence to sequence model is a special case of the DMN without
a question and without episodic memory. Instead it maps an input sequence directly to an answer
sequence.
Attention. The second line of work that is very relevant to DMNs is that of attention and memory
in deep learning. The work of recent months by Weston et al. on memory networks [17] focuses on
5
adding a memory component for natural language question answering. They have an input (I) and
response (R) component and their generalization (G) and output feature map (O) components have
some functional overlap with our episodic memory. However, their model cannot be applied to the
same variety of NLP tasks since it processes sentences independently and not via a sequence model.
It requires bag of n-gram string matching features as well as a separate feature that captures whether
a sentence came before another one. We compare directly to their model on the bAbI dataset [18].
Attention mechanisms are generally useful and can improve image classiﬁcation [19], automatic
image captioning [20] and machine translation [21, 22]. Neural Turing machines use memory to
solve algorithmic problems such as list sorting [23].
NLP Applications. The DMN is a general model which we apply to several NLP problems. We
compare to what, to the best of our knowledge, is the current state of the art method for each task.
There are many different approaches to question answering : some build large knowledge bases
(KBs) with open information extraction systems [24], some use neural networks, dependency trees
and KBs [6], others only sentences [12]. A lot of other approaches exist. When QA systems do
not produce the right answer, it is often unclear if it is because they do not have access to the facts,
cannot reason over them or have never seen this type of question or phenomenon. Most QA dataset
only have a few hundred questions and answers but require complex reasoning. They can hence not
be solved by models that have to learn purely from examples. While synthetic datasets [18] have
problems and can often be solved easily with manual feature engineering, they let us disentangle
failure modes of models and understand necessary QA capabilities. They are useful for analyzing
models that attempt to learn everything and do not rely on external features like coreference, POS,
parsing, logical rules, etc. The DMN is such a model.
Sentiment analysis is a very useful classiﬁcation task and recently the Stanford Sentiment Treebank
[2] has become a standard benchmark dataset. Kim [25] reports the previous state of the art result
based on a convolutional neural network that uses multiple word vector representations. The previ-
ous best model for part of speech tagging on the Wall Street Journal section of the Penn Tree Bank
[26] was Sogaard [27] who used a semisupervised nearest neighbor approach.
Neuroscience. The semantic and episodic memory modules are related to cognitive neuroscience. In
humans both language-speciﬁc and supramodal concept representations are seated in the semantic
memory whose existence is well established [28]. Hence, word vectors and knowledge bases are
stored in the DMN’s semantic memory module.
The episodic memory in humans stores speciﬁc experiences in their spatial and temporal context.
For instance, it might contain the ﬁrst memory somebody has of ﬂying a hang glider. Eichenbaum
and Cohen have argued that episodic memories represent a form of relationship (i.e., relations be-
tween spatial, sensory and temporal information) and that the hippocampus is responsible for general
relational learning [29]. Interestingly, it also appears that the hippocampus is active during transitive
inference [30], and disruption of the hippocampus impairs this ability [31].
The episodic memory module in the DMN is related to these ﬁndings. It retrieves speciﬁc temporal
states that are in a relationship with a question. Furthermore, we found that the GRU in this module
was able to do some transitive inference over the simple facts in the bAbI dataset. This module also
has similarities to the Temporal Context Model [32] and its Bayesian extensions [33] which were
developed to analyze human behavior in word recall experiments.
4 Experiments
We include experiments on question answering, part of speech tagging, sentiment analysis, as well
as preliminary results on machine translation. For all datasets we used either the ofﬁcial train,dev,test
splits or if no dev set was deﬁned, we used 10% of the training set for development. Hyper-parameter
tuning and model selection (with early stopping) is done on the development set. The DMN is trained
via backpropagation and Adagrad [34]. We employL2 regularization, and “word-dropout” in which
each word vector is set to 0 with some probability p. Word vectors are pre-trained using Glove [3].
6
4.1 Question Answering
The Facebook bAbI dataset is a synthetic dataset meant to test a model’s ability to retrieve facts
and reason over them. Each task tests a different skill that a good question answering model ought
to have, such as coreference resolution, deduction, and induction. Training on the bAbI dataset
Task MemNN DMN Task MemNN DMN
1: Single Supporting Fact 100 100 11: Basic Coreference 100 99.9
2: Two Supporting Facts 100 98.2 12: Conjunction 100 100
3: Three Supporting facts 100 95.2 13: Compound Coreference 100 99.8
4: Two Argument Relations 100 100 14: Time Reasoning 99 100
5: Three Argument Relations 98 99.3 15: Basic Deduction 100 100
6: Yes/No Questions 100 100 16: Basic Induction 100 99.4
7: Counting 85 96.9 17: Positional Reasoning 65 59.6
8: Lists/Sets 91 96.5 18: Size Reasoning 95 95.3
9: Simple Negation 100 100 19: Path Finding 36 34.5
10: Indeﬁnite Knowledge 98 97.5 20: Agent’s Motivations 100 100
Mean Accuracy (%) 93.3 93.6
Table 1: Test accuracies on the bAbI dataset. MemNN numbers taken from Weston et al. [18]. The
DMN passes (accuracy >95%) 18 tasks, whereas the MemNN passes 16.
uses the following objective function: J = αECE (Gates) +βECE (Answers), where ECE is the
standard cross-entropy cost and αand βare hyperparameters. In practice, we begin training with α
set to 1 and β set to 0, and then later switch β to 1 while keeping αat 1. We subsample the facts
from the input module by end-of-sentence tokens. The gate supervision aims to select one sentence
per pass; thus, we also experimented with modifying Eq. 6 to a simple softmax instead of a GRU.
Here, we compute the ﬁnal episode vector via: ei = ∑T
t=1 softmax(gi
t)ct, where softmax(gi
t) =
exp(gi
t)∑T
j=1 exp(gi
j) , and gi
t here is the value of the gate before the sigmoid. This setting achieves better
results, likely because the softmax is better suited to picking one sentence at a time.
We list results in table 1. The DMN does worse than the MemNN on tasks 2 and 3, both tasks with
long input sequences. We suspect this is due to the recurrent input sequence model having trouble
modeling very long inputs. The MemNN does not suffer from this problem as it views each sentence
seperately. The power of the episodic memory module is evident in tasks 7 and 8, where the DMN
signiﬁcantly outperforms the MemNN. Both tasks require the model to iteratively retrieve facts and
store them in a representation that slowly incorporates more of the relevant information of the input
sequence. Both models do poorly on tasks 17 and 19, though the MemNN does better. We suspect
this is due to the MemNN using n-gram features as well as explicit sequence position features.
4.2 Sequence Tagging: Part of Speech Tagging
Part-of-speech tagging is traditionally modeled as a sequence tagging problem: every word in a
sentence is to be classiﬁed into its part-of-speech class (see Fig. 1). We evaluate on the standard
Wall Street Journal dataset included in Penn-III [26]. We use the standard splits of sections 0-18
for training, 19-21 for development and 22-24 for test sets [27]. Since this is a word level tagging
task, DMN memories are produced at the word -rather than sentence- level. We compare the DMN
Model SVMTool Sogaard Suzuki et al. Spoustova et al. SCNN DMN
Acc (%) 97.15 97.27 97.40 97.44 97.50 97.56
Table 2: Test accuracies on WSJ-PTB
with the results in [27]. The DMN achieves state-of-the-art accuracy with a single model, reaching
a development set accuracy of 97.5. Ensembling the top 4 development models, the DMN gets to
97.58 dev and 97.56 test accuracies, achieving a new state-of-the-art (Table 2).
7
4.3 Text Classiﬁcation: Sentiment Analysis
Stanford Sentiment Treebank (SST) [2] is a popular dataset for sentiment classiﬁcation. It provides
phrase-level ﬁne-grained labels, and comes with a train/dev/test split. We present results on two
formats: ﬁne-grained root prediction, where all full sentences (root nodes) of the test set are to
be classiﬁed as either very negative, negative, neutral, positive, or very positive, and binary root
prediction, where all non-neutral full sentences of the test set are to be classiﬁed as either positive
or negative. To train the model on the ﬁne-grained task, we use all phrase-level labels. To train on
the binary task, we use all non-neutral phrase-level labels.
Task MV-RNN RNTN DCNN PVec CNN-MC DRNN CT-LSTM DMN
Binary 82.9 85.4 86.8 87.8 88.1 86.6 88.0 88.3
Fine-grained 44.4 45.7 48.5 48.7 47.4 49.8 51.0 50.3
Table 3: Test accuracies on SST [2]. Key: MV-RNN: Socher et al., 2013. RNTN: Socher et al.,
2013. DCNN: Blunsom et al., 2014. PVec: Le and Mikolov, 2014. CNN-MC: Kim, 2014. DRNN:
Irsoy and Cardie, 2014. CT-LSTM: Tai et al., 2015. All results as reported in [35]
For sentiment analysis, our gate functionGneeds only the ﬁrst 3 components c,m,q of the function
zas deﬁned in Eq. 3. The DMN achieves state-of-the-art accuracy on the binary classiﬁcation task,
as well as near state-of-the-art on the ﬁne-grained classiﬁcation task.
Our DMN was trained with GRU sequence models and no tree structure. It is easy to replace the
GRU sequence model with any of the models listed above, as well as incorporate tree structure to
the retrieval process. These experiments were not run, and we consider them future work.
Preliminary Results: Machine Translation
0 50 100 150 200 250 300 350
time (hours)
3
4
5
6
7
8log perplexity per word on training set
Seq-to-Seq LSTM
DMN with GRUs
Figure 4: Machine translation
training progress in terms of
log-perplexity.
We are also training the DMN for machine translation, comparing it
to the sequence-to-sequence LSTM model presented by Sutskever
et al. [16]. The sequence-to-sequence model is a special case of the
DMN in which only one memory at the end of the input sentence
is formed. Initial experiments on the smaller WMT13 English-to-
French News Commentary dataset used in Kalchbrenner [36] show
promising results. As seen in 4.3, the DMN is learning at a similar
pace to an implementation of the Seq-to-Seq LSTM model. For the
Seq-to-Seq LSTM, we use the hyperparameters listed in [16].
5 Conclusion
We believe the DMN is a potentially general model for a variety of NLP applications. The entire
model can be trained end-to-end with one, albeit complex, objective function. The model uses
some ideas from neuroscience such as semantic and episodic memories known to be required for
complex types of reasoning. Future work will explore additional tasks, larger multi-task models and
multimodal inputs and questions.
Acknowledgements
We thank Sam Gershman for useful discussions.
References
[1] A. Passos, V . Kumar, and A. McCallum. Lexicon infused phrase embeddings for named entity resolution.
In Conference on Computational Natural Language Learning. Association for Computational Linguistics,
June 2014.
[2] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. Manning, A. Ng, and C. Potts. Recursive deep models for
semantic compositionality over a sentiment treebank. In EMNLP, 2013.
8
[3] J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In EMNLP,
2014.
[4] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efﬁcient estimation of word representations in vector
space. In ICLR (workshop), 2013.
[5] R. Socher, D. Chen, C. D. Manning, and A. Y . Ng. Reasoning With Neural Tensor Networks For Knowl-
edge Base Completion. In NIPS. 2013.
[6] A. Bordes, X. Glorot, J. Weston, and Y . Bengio. Joint Learning of Words and Meaning Representations
for Open-Text Semantic Parsing. AISTATS, 2012.
[7] J. L. Elman. Distributed representations, simple recurrent networks, and grammatical structure. Machine
Learning, 7(2-3):195–225, 1991.
[8] K. Cho, B. van Merrienboer, D. Bahdanau, and Y . Bengio. On the properties of neural machine translation:
Encoder-decoder approaches. CoRR, abs/1409.1259, 2014.
[9] J. Chung, C ¸ . G¨ulc ¸ehre, K. Cho, and Y . Bengio. Empirical evaluation of gated recurrent neural networks
on sequence modeling. CoRR, abs/1412.3555, 2014.
[10] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, Nov
1997.
[11] R. Socher, E. H. Huang, J. Pennington, A. Y . Ng, and C. D. Manning. Dynamic Pooling and Unfolding
Recursive Autoencoders for Paraphrase Detection. In NIPS, 2011.
[12] M. Iyyer, J. Boyd-Graber, L. Claudino, R. Socher, and H. Daum ´e III. A neural network for factoid
question answering over paragraphs. In EMNLP, 2014.
[13] S. R. Bowman, C. Potts, and C. D. Manning. Recursive neural networks for learning logical semantics.
CoRR, abs/1406.1827, 2014.
[14] T. Mikolov and G. Zweig. Context dependent recurrent neural network language model. In SLT, pages
234–239. IEEE, 2012.
[15] A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In
CVPR, 2015.
[16] I. Sutskever2014, O. Vinyals, and Q. V . Le. Sequence to sequence learning with neural networks. In
NIPS, 2014.
[17] J. Weston, S. Chopra, and A. Bordes. Memory networks. In ICLR, 2015.
[18] J. Weston, A. Bordes, S. Chopra, and T. Mikolov. Towards ai-complete question answering: A set of
prerequisite toy tasks. CoRR, abs/1502.05698, 2015.
[19] M. F. Stollenga and J. Schmidhuber J. Masci, F. Gomez. Deep Networks with Internal Selective Attention
through Feedback Connections. In NIPS, 2014.
[20] K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. S. Zemel, and Y . Bengio. Show,
attend and tell: Neural image caption generation with visual attention. CoRR, abs/1502.03044, 2015.
[21] K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y . Bengio. Learn-
ing phrase representations using rnn encoder-decoder for statistical machine translation. 2014.
[22] D. Bahdanau, K. Cho, and Y Bengio. Neural machine translation by jointly learning to align and translate.
CoRR, abs/1409.0473, 2014.
[23] A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. CoRR, abs/1410.5401, 2014.
[24] A. Yates, M. Banko, M. Broadhead, M. J. Cafarella, O. Etzioni, and S. Soderland. Textrunner: Open
information extraction on the web. In HLT-NAACL (Demonstrations), 2007.
[25] Y . Kim. Convolutional neural networks for sentence classiﬁcation. 2014.
[26] M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini. Building a large annotated corpus of english: The
penn treebank. Computational Linguistics, 19(2), June 1993.
[27] A. Søgaard. Semisupervised condensed nearest neighbor for part-of-speech tagging. In ACL-HLT, 2011.
[28] J. R. Binder and R. H. Desai. The neurobiology of semantic memory. Trends in Cognitive Sciences ,
15(11):527–536, 2011.
[29] H. Eichenbaum and N. J. Cohen. From Conditioning to Conscious Recollection: Memory Systems of the
Brain (Oxford Psychology). Oxford University Press, 1 edition, 2004.
[30] S. Heckers, M. Zalesak, A. P. Weiss, T. Ditman, and D. Titone. Hippocampal activation during transitive
inference in humans. Hippocampus, 14:153–62, 2004.
[31] J. A. Dusek and H. Eichenbaum. The hippocampus and memory for orderly stimulusrelations. Proceed-
ings of the National Academy of Sciences, 94(13):7109–7114, 1997.
9
[32] Marc W. Howard and Michael J. Kahana. A distributed representation of temporal context. Journal of
Mathematical Psychology, 46(3):269 – 299, 2002.
[33] R. Socher, S. Gershman, A. Perotte, P. Sederberg, D. Blei, and K. Norman. A bayesian analysis of
dynamics in free recall. In Advances in Neural Information Processing Systems 22. 2009.
[34] J. Duchi, E. Hazan, and Y . Singer. Adaptive subgradient methods for online learning and stochastic
optimization. JMLR, 12, July 2011.
[35] K. S. Tai, R. Socher, and C. D. Manning. Improved semantic representations from tree-structured long
short-term memory networks. 2015.
[36] N. Kalchbrenner and P. Blunsom. Recurrent continuous translation models. 2013.
10

Non-Discrimination, Harassment and Retaliation Policy 
 
 
Eliminating All Forms of Discrimination 
 
Dickinson College is an intellectual and social community that values justice, free inquiry, diversity, 
and equal opportunity. It is a fundamental policy of the college to respect pluralism, civility and 
mutual understanding within its community. The college does not discriminate on the basis of race, 
color, national origin, sex, disability, religion, age, veteran status, gender identity or expression, sexual 
orientation or any other protected class. 
 
This Anti-Discrimination, Harassment and Retaliation Policy is intended to keep the college in 
compliance with state and federal laws and regulations, including but not limited to the Pennsylvania 
Human Relations Act, Title VI and VII of the Civil Rights Acts, Title IX of the Education 
Amendments of 1972, Section 504 of the federal Rehabilitation Act, and the Americans With 
Disabilities Act, the Age Discrimination and Age Discrimination in Employment Acts, and the 
Genetic Information Non-Discrimination Act. This policy applies to all faculty, staff and students.  
 
Because this is a vibrant community with a plethora of policies and procedures, it is possible that other 
policies may directly or indirectly deal with the same subject matter. Therefore, if Dickinson has any 
other policy or procedure that more specifically addresses any of the issues outlined herein, the more 
specific policy or procedure controls. Any conflicts between policies or procedures will be resolved by 
the vice president or provost over the division in question, in consultation with general counsel.  
 
Dickinson College provides equal employment opportunity for qualified applicants and employees, as 
well as equal opportunity for all individuals partaking in its academic programs and activities - and 
promotes the full realization of such equal opportunity through positive, continuing programs in every 
department. This commitment applies to not just college administrators, but also every member of the 
Dickinson community, as we are all charged with respecting the rights of others. To this extent, even 
when not specifically required to do so by law, Dickinson implores its community to be respectful of 
each other and to not harass, discriminate, or otherwise exhibit bias against each other on the basis of 
any immutable (or alternatively, not easily changed) characteristic. As such, we must all work together 
to ensure continued success of these ideals, whether legally mandated or aspirational in nature. What 
follows are the procedural steps Dickinson College will take to provide for the prompt, fair and 
impartial consideration of all complaints of discrimination, harassment and retaliation. 
 
Eliminating All Forms of Discriminatory Harassment 
 
Harassment based upon a person’s race, color, national origin, sex, disability, religion, age, veteran 
status, gender identity or expression, sexual orientation, or any other protected class is strictly 
prohibited. These characteristics are personal in nature and are protected under federal and state laws. 
The college prohibits all forms of harassing conduct based upon these protected characteristics. No 
person should be subjected to harassment or discriminated against because he or she is a member of a 
certain group. 
 
Policy/Procedure 
 2 
Certain conduct that may constitute discriminatory harassment includes, but is not limited to: 
preferential or derogatory treatment based upon protected characteristics; using severe or pervasively 
insulting or degrading language that would reasonably offend members of a given race, color, national 
origin, sex, disability, religion, age, veteran status, gender identity or expression, sexual orientation, or 
any other protected class; distributing or displaying any written or graphic materials that would be 
offensive to members of any protected group; and using racial, religious, or ethnic epithets. 
 
The college is committed to eliminating all forms of discriminatory harassment from campus. While 
some discriminatory stereotypes and prejudices unfortunately still exist in our society, no supervisor, 
manager or employee may express or act upon such unacceptable feelings and beliefs in the workplace 
or while conducting business. Any employee or student who commits or condones discriminatory 
harassment may be subject to disciplinary action, up to and including termination or expulsion. 
 
All members of the Dickinson community play an extremely important role in our efforts to eliminate 
discriminatory harassment from our workplace. They must be the college’s eyes and ears. It is 
necessary that employees and students inform the college immediately if they experience or observe 
any discriminatory harassment. 
 
Dealing with Discrimination and Harassment 
 
If you believe you have been discriminated against or harassed, you have several options available to 
you at Dickinson College.  You do not have to deal with the situation alone. The following options are 
available: 1) You can try to resolve the problem directly with the person whose behavior you deem 
inappropriate, by letting him/her know that you find the conduct unwelcome. Making clear that the 
behavior is unwelcome may end the behavior. 2) If despite this effort the conduct continues, or you are 
uncomfortable doing that, you can ask the Director of Equity and Inclusivity, Title IX Coordinator, 
your divisional Vice President/Provost or Director of Human Resource Services to convey this 
message for you. 3) You can speak to a Counselor in Counseling Services if you are a student, or to a 
member of the College’s Employee Assistance Program staff if you are a faculty, staff, or 
administrative member of the College.  If you speak with one of these individuals your conversation is 
confidential. 4) You can choose to pursue an informal or formal resolution to your complaint. 
 
If you choose options 2 or 4, you need to understand the legal obligations of agents of the College 
(agents of the College are staff, personnel, faculty, and resident advisors). See “Confidentiality 
regarding sexual harassment complaints” below. It is important to note that the College cannot take 
appropriate action unless a responsible officer of the College is informed of the situation. 
 
In addition to the options above, the college has established a reporting system that offers a choice 
concerning the avenue for making such a report. An individual can report discrimination or harassment 
to any of the following college representatives: 
 
1. Employee’s supervisor 
2. Director of HR Services 
3. Dean of Students 
4. Director of Equity and Inclusivity 
5. Title IX Coordinator 
 3 
6. Provost/Dean of the College 
 
Anonymous concerns can also be raised via the college’s various online reporting systems. While 
anonymous concerns are taken seriously and investigated promptly, and many of our systems allow for 
anonymous follow-up contact with the reporting party, it may be difficult to fully investigate and 
appropriately remedy the issue without the ability to directly communicate with the reporting party. 
 
When making a report of discriminatory harassment, it is important that the person provide as 
much information as possible, including the details of the alleged harassing conduct, any 
physical evidence of the harassment that may exist and the names of everyone involved in the 
harassment, any others who may have experienced similar harassment and any individuals who 
observed or witnessed the harassment. The college will, to the fullest extent possible, protect the 
confidentiality of the information provided - consistent with the obligation to conduct a thorough 
and appropriate investigation.  
 
Prompt and appropriate action will be taken pending an investigation to ensure that any discriminatory 
harassment that might exist does not continue. If the investigation indicates that discriminatory 
harassment may have occurred, the college will take prompt and appropriate remedial measures to 
bring the harassment to an end. 
 
 
Confidentiality Regarding Discrimination and Harassment Complaints 
 
Once any college agent, including a resident adviser, faculty member, administrator, supervisory staff 
member, or other employee in a position of authority learns of a serious discrimination or harassment 
complaint, written or unwritten, that official is required to report the situation and the college is 
required by law to investigate and to take action consistent with the law and its policies and 
procedures. During such investigations, every effort will be made to protect the privacy of all parties 
in a manner consistent with a thorough investigation. However, privacy and confidentiality cannot be 
guaranteed.  
 
Informal Complaint 
 
Mediation and/or Facilitated Discussions (hereafter referred to as conciliation) are an informal process 
that involves a trained facilitator who will assist in effecting a voluntary resolution of the dispute. The 
objective of a conciliation process is to come to an agreement that is fair and meets the needs of the 
parties involved in the conflict. The process is intended to be confidential and is conducted in a 
confidential setting. Note, however, that if the conciliation is handled by a Dickinson employee not 
specifically designated by a mediator or conciliator job title, the conversation may not be able to 
remain confidential, especially if it involves potential legal violations that the college has an obligation 
to remedy. Also, conciliations do not waive the rights of any aggrieved party to seek resolution of the 
complaint through formal avenues.  
 
If one or both parties are interested in pursuing conciliation as an option, they should contact Human 
Resource Services or the Office of General Counsel. Both parties involved in the dispute must agree to 
voluntarily participation in order to proceed. Reasonable attempts will be made to arrange for the 
 4 
mediation /facilitated discussion within ten (10) working days of the request. The conciliator will 
facilitate the discussion between the disputing parties to help find a mutually agreeable solution to the 
conflict. The conciliator may make recommendations on how to resolve the dispute but will not 
impose a solution, will not share any of the recommendations to resolve proffered by anyone involved 
in the process, and will not be subjected to being called as a witness by either party in a later hearing, 
grievance or similar process. The mediator will keep no records of the proceedings except for a 
general document outlining whether the parties participated in the conciliation and a copy of any 
negotiated agreement reached by the parties. At any time during the process, either with or without 
reaching an agreement, a party may opt out and request to proceed with a formal complaint. 
 
Formal Complaint 
 
A formal complaint requires an appropriate college officer to carry out a thorough investigation, to 
determine the facts of the case, and, where discrimination or harassment has occurred, to take 
appropriate actions which may include disciplinary and corrective actions designed to prevent future 
occurrences. This is the proper course of action if either party is dissatisfied with the outcome of 
informal processes, or if formal charges are filed without pursuing informal resolutions. While 
complaints can be taken to a myriad of college leaders, generally the following positions will be 
involved with the cases as referenced below: 
 
• Complaints concerning sexual harassment and misconduct - Title IX Coordinator 
o Contact Information: Kat Matic, 717-254-8316, TitleIX@dickinson.edu 
 
• Complaints concerning student disability accommodations - Director of Access and Disability 
Services 
o Contact Information: Marni Jones, 717-245-1734, access@dickinson.edu 
 
• Complaints concerning all forms of discrimination, harassment, disability accommodations and 
retaliation impacting employees – Director of Human Resource Services 
o Contact Information: Dennette Moul, 717-245-1026, mould@dickinson.edu 
 
• Complaints concerning all forms of discrimination, harassment and retaliation impacting 
students – Chief Diversity Officer 
o Contact Information:  Tony Boston, 717-245-1125, bostont@dickinson.edu 
 
Note that allegations of sexual harassment are processed in accordance with the Sexual Harassment 
and Misconduct policy. If an employee is a complainant or respondent in an alleged sexual harassment 
case, and there is a reasonable basis to believe that the case is governed in whole or in part by Title IX 
regulations, the Title IX Coordinator will manage the process, and a member of Human Resource 
Services will participate in the investigation process and/or assist with the adjudication of the matter as 
a member of the Review Panel. 
 
Complainants who believe that separation or other protection is necessary during the process of a 
formal complaint or appeal may, as part of their complaint or by separate letter, specifically request 
that the college arrange for separation (meaning either temporary or permanent removal from campus 
or on-campus physical separation from the respondent to provide adequate space to navigate campus 
 5 
with less fear or concern) or other protection. The investigating officer, in consultation with the 
divisional vice president/provost and any other college employee with a need to know and be involved 
in the decision, will determine the necessity and arrange for such protection pending the completion of 
the complaint and appeal. A decision to arrange for such separation or other protection need not be 
based on a finding of fault and shall not be interpreted as a finding of fault.  Separation may be 
especially prudent if a person in a position of authority is alleged to have violated the college’s anti-
harassment and discrimination policies.   
 
When an allegation is made against a faculty, staff, or student in positions of authority, there will be an 
assessment regarding the individual’s ability to remain in their current position and, if it is determined 
that a conflict exists, the individual will be either moved to a different position or suspended with or 
without pay until the investigation is complete and a decision on the underlying case is made. The 
assessment will include considerations such as the nature of complaint itself, the proximity and direct 
or indirect control or influence between the complainant and the respondent, and the availability of 
temporary alternate assignments that are capable of keeping the complainant as whole as possible 
during the pendency of the investigation.  
 
Once the investigation is complete, the investigating officer’s decision will be submitted to the 
appropriate personnel – typically one of the six (6) positions enumerated in the ‘Eliminating all Forms 
of Discriminatory Harassment’ section - for resolution. 
 
General Considerations 
 
Employees with disabilities, who can perform the core responsibilities of their job with a reasonable 
accommodation, should promptly bring their concerns to their supervisor.  These employees shall be 
provided with a reasonable accommodation, except where such accommodation would cause the 
college undue hardship or would prevent the employee from performing the core responsibilities of 
their job. We invite employees with disabilities that require reasonable accommodation to inform 
Human Resource Services of their need for such reasonable accommodation. Students should bring 
their building, educational access and other accommodation needs to the attention of either the 
Director of Access and Disability Services or the Director of Equity and Inclusivity. 
 
If applicants believe that they have experienced any discrimination during the hiring process, during 
normal work activities or through other aspects of the work environment, such as consideration for 
promotion, transfer, rates of pay or through the college’s recruitment, training or selection for training, 
they should report the concerns to the college’s Director of Human Resource Services, or Title IX 
Coordinator. If students believe that they have experienced any discrimination during the recruitment 
process, or during their time as a student participating in the educational, cocurricular and other 
benefits and privileges of life on campus, they should report the concerns to the Dean of Students, 
Dean of the College, or the college’s Director of Equity and Inclusivity. 
 
This policy guarantees that all job assignments will be based upon the college’s judgment of the 
individual’s ability to perform the services required for a position and the corresponding job 
relatedness. Human Resource Services can provide the individual with descriptions of the kinds of 
positions at the college, including the general range of responsibilities for each position. The hiring 
 6 
manager, in consultation with Human Resource Services, will make the final decision in selecting the 
most appropriately qualified candidate for a position. 
 
This anti-discrimination and harassment policy also applies to programming being conducted or 
hosted by faculty and staff, as well as student or third-party programming that receive significant 
financial or other support from the college. While it is permissible to have programming that is 
marketed towards and targeted to benefit certain segments of the campus community, college 
sponsored or significantly supported programming must not exclude any individuals from participating 
on the basis of race, color, national origin, sex, disability, religion, age, veteran status, gender identity 
or expression, or sexual orientation. Similarly, it is the policy of the college not to engage in any 
product or entity boycotting activities that may suggest, in any way shape or form, that the college 
supports or rejects the product or entity on the basis of race, color, national origin, sex, disability, 
religion, age, veteran status, gender identity or expression, or sexual orientation. Indeed, the college’s 
goal and priority in this regard is to educate and not advocate. Any concerns relative to college 
programming should be brought to the attention of the Dean of Students, Dean of the College, or the 
college’s Director of Equity and Inclusivity. A concern can also be reported either directly or 
anonymously to the Bias Education Response Team (BERT). 
 
The college will maintain appropriate records of complaints and investigation results, in accordance 
with its records retention policy.  In order to protect and to assure participants to a complaint or 
investigation (whether complainant, respondent or witness) that there will be no recrimination or 
retaliation, these records will be closed - with access to these records only granted with the consent of 
the individual(s) involved, to college officers with an official need to know, or as required by law in a 
legal or administrative investigation or proceeding. 
 
To the extent possible, any proceedings conducted pursuant to this policy and the subsequent results 
will be considered private - and the process will be designed to reasonably protect the rights and 
names of all parties. Retaliation for filing harassment or discrimination charges will not be tolerated. 
Additionally, no person making a good-faith allegation of harassment or discrimination will be 
disciplined. However, making an allegation in an abusive manner will be treated just as swiftly and 
severely as retaliating against someone for making a good-faith allegation.  Any individual who is 
found to have retaliated against any person or purposefully made a false report in violation of this 
policy will be subject to appropriate disciplinary action, up to and including termination or expulsion. 
 
 
Bias Education Response Team (BERT) 
Sexual Harassment and Misconduct Policy 
President’s Working Group on Sexual Harassment, Sexual Misconduct and Title IX 
President’s Commission on Inclusivity 
Policy and Procedural Statement for Students with Disabilities 
Academic, Employee and Student Handbooks 
 
3 
History/Revision Information 
 
Related Information 
 7 
 
 
Responsible Division/Office: Human Resource Services, Office of General Counsel 
Effective Date:  July 1, 2008  
Last Amended Date:  July 3, 2024 
Also Found In: 
 

AI Powered Compiler Techniques for DL Code
Optimization
Sanket Tavarageri
Intel Labs
India
sanket.tavarageri@intel.com
Gagandeep Goyal
IIT Hyderabad
India
cs19mtech01003@iith.ac.in
Sasikanth Avancha
Intel Labs
India
sasikanth.avancha@intel.com
Bharat Kaul
Intel Labs
India
bharat.kaul@intel.com
Ramakrishna Upadrasta
IIT Hyderabad
India
ramakrishna@iith.ac.in
ABSTRACT
Creating high performance implementations of deep learn-
ing primitives on CPUs is a challenging task. Multiple con-
siderations including multi-level cache hierarchy, and wide
SIMD units of CPU platforms influence the choice of pro-
gram transformations to apply for performance optimization.
In this paper, we present machine learning powered compiler
techniques to optimize loop nests. We take a two-pronged
approach to code optimization: We first apply high level op-
timizations to optimize the code to take optimal advantage
of the cache memories. Then, we perform low level, target-
specific optimizations to effectively vectorize the code to run
well on the SIMD units of the machine. For high level opti-
mizations, we use polyhedral compilation techniques and
deep learning approaches. For low level optimization, we use
a target specific code generator that generates code using vec-
tor intrinsics and Reinforcement Learning (RL) techniques to
find the optimal parameters for the code generator. We per-
form experimental evaluation of the developed techniques
on various matrix multiplications that occur in popular deep
learning workloads. The experimental results show that the
compiler techniques presented in the paper achieve 7.6X and
8.2X speed-ups over a baseline for sequential and parallel
runs respectively.
1 INTRODUCTION
Deep learning (DL) has become pervasive in various domains
of computing. Image recognition [14, 17], language model-
ing [10], language translation [31], speech recognition [15]
make extensive use of deep neural networks (DNNs). Deep
Learning inference is an important workload across appli-
cations, such as object classification & recognition, text and
speech translation etc. A 2018 Mc Kinsey study [5] pointed
out that in datacenters, 75% of the inference tasks are run on
CPUs. Optimizing DL workloads on CPUs is a challenging
proposition because of architectural complexities of CPU
platforms. Multi-level cache hierarchies, TLBs (Translation
Look-aside Buffers), hardware data prefetchers, SIMD (a.k.a
vector) units present particular challenges in writing high
performance code. Therefore, the current state-of-practice is
to use expert-coded high performance libraries such as Intel
oneDNN [3] in deep learning frameworks such as Tensor-
Flow and PyTorch to achieve good performance. However,
being reliant on libraries for performance is not scalable.
First, it would increase the time-to-market: from the time
a new DL operator is invented to its being supported in a
library could take a considerable amount of time. Second,
even expert programmers must invest significant amount of
effort to tune the implementations on the target platforms.
Therefore, an attractive alternative solution is to develop
compilation techniques that automate code optimization and
achieve similar performance levels as expert-coded libraries.
In this paper, we develop a systematic approach to auto-
matic code optimization. We categorize the program opti-
mizations into two phases: high level and low level optimiza-
tions. High level optimizations perform loop optimizations
such as loop reordering and tiling to derive a loop struc-
ture that utilizes the cache hierarchy of the computer sys-
tem to the fullest extent possible. Low level optimizations
generate vector code using the target machine’s intrinsics;
reinforcement learning methodology guides the derivation
of high performance vector code. For high-level and low-
level optimizations we leverage artificial intelligence (A.I.)
techniques. We evaluate our automated compiler system on
GEMMs which lie at the heart of deep learning [1]. The re-
sults indicate our compiler workflow delivers competitive
performance compared to Intel oneDNN library and signif-
icantly higher performance compared to a state-of-the-art
DL compiler, viz., AutoTVM [9].
The contributions of the paper are as follows.
1
arXiv:2104.05573v1  [cs.PL]  12 Apr 2021
•A systematic approach to program optimization with
clear demarcation of high-level and low-level optimiza-
tions that map well to the hardware architectures.
•Low-level optimizations that generate reinforcement
learning (RL) guided vector intrinsics based code.
•Development of A.I. techniques for high-level and low-
level optimizations.
•Experimental evaluation on various GEMM sizes that
occur in DL workloads.
The rest of the paper is organized as follows. Section 2
introduces the overall compilation workflow. Section 3 de-
scribes the polyhedral compilation techniques we use for
high level optimizations. The low-level optimizations involv-
ing the target platform specific code generator and reinforce-
ment learning are developed in Section 4. The experimental
evaluation conducted is detailed in Section 5. Related work
is discussed in Section 6. The conclusion and implication of
the presented work are presented in Section 7.
2 THE COMPILER OPTIMIZATION
WORKFLOW
We first describe the overall compiler workflow. We input
the loop nests such as GEMMs to the compiler. The high
level optimizer first optimizes the loop structure and then
passes on the code to the low level optimer. Figure 1 shows
the workflow of the optimization process.
The high level optimizer uses polyhedral compilation tech-
niques for optimization of the loop structure to take advan-
tage of the multi-level caches of the CPU platform. The loop
reordering and tiling transformations are applied and the
best loop order and tile sizes are determined by the high
level optimizer. It will enhance data locality – both the spa-
tial and temporal locality such that the data used by the
input program is reused out of the caches closest to the pro-
cessor as much as possible. Section 3 details the polyhedral
compilation techniques we apply for loop optimization.
The high level optimer then hands over the optimized code
to the low-level optimizer. The low level optimizer derives a
vectorization strategy for effective use of the SIMD vector
units. We employ a target specific low level optimization
approach wherein the inner loops of the loop nest are vec-
torized using the target-specific vector intrinsics. Further,
we use a Reinforcement Learning (RL) based approach to
select the best vectorization plan among the myriad choices
available.
Jouppi et al [16] show that 95% of the deep learning infer-
ence workloads (MLPs, CNNs, and LSTMs) can be formulated
in terms of matrix-multiplication. Matrix multiplication is at
the heart of deep learning [1]. Because of these reasons in
this work, we build the low level optimizer to optimize the
matrix-multiplications that occur in the inner most loops of
convolutions and matrix-multiplications themselves (matrix-
multiplication can be recursively defined where the other
loops of the code are the tiled loops and the inner loops
are also functionally equivalent to matrix-multiplication). In
Section 4, we describe the design and the implementation of
the low level optimizer for the inner loops of the loop nest
focused on matrix multiplication.
3 HIGH-LEVEL POLYHEDRAL LOOP
OPTIMIZATIONS
We use the polyhedral model [ 11], which is an advanced
mathematical framework to reason about dependences and
loop transformations, to develop our data reuse algorithm.
3.1 Preliminaries
We use the Integer Set Library (ISL) [ 27] for performing
polyhedral operations in this work and we use the same
notation as used in ISL to elucidate the concepts and the
algorithm. The matrix multiplication code shown in Figure
2 will be used to illustrate the workings of the data reuse
analysis.
Sets. A set is a tuple of variables 𝑥𝑖 s along with a col-
lection of constraints 𝑐𝑘 s defined on the tuple variables.
𝑠 = {[𝑥1,...,𝑥 𝑛]: 𝑐1 ∧...𝑐 𝑚 }
The iteration spaces of loop nests are represented as sets.
The iteration space of the loop in Figure 2 is defined as the
following set. 𝐼 = {𝑆[𝑖,𝑗,𝑘 ] : 0 <= 𝑖 < 𝑀 ∧0 <= 𝑗 <
𝑁 ∧0 <= 𝑘 < 𝐾}
Relations. A relation is a mapping from input tuple vari-
ables 𝑥𝑖 s to output tuple variables 𝑦𝑗 s. In addition, a set
of constraints 𝑐𝑘 s can be defined for a relation that will
place constraints on the input/output tuple variables. 𝑟 =
{[𝑥1,...,𝑥 𝑛]↦→[ 𝑦1,...,𝑦 𝑚]: 𝑐1,...,𝑐 𝑝 }
The read and write access functions of a loop nest can
be modeled with relations. The read relations in the Figure
2 code are shown below: 𝑟1 = {𝑆[𝑖,𝑗,𝑘 ] ↦→𝐶[𝑖,𝑗 ]}, 𝑟2 =
{𝑆[𝑖,𝑗,𝑘 ]↦→ 𝐴[𝑖,𝑘]}, 𝑟3 = {𝑆[𝑖,𝑗,𝑘 ]↦→ 𝐵[𝑘,𝑗 ]}. The sole
write relation in the loop is: 𝑤1 = 𝑆[𝑖,𝑗,𝑘 ]↦→ 𝐶[𝑖,𝑗 ]. The
domain of a relation 𝑟 is denoted by dom 𝑟.
Apply operation. When a relation 𝑟 is applied on a set 𝑠,
the domain of 𝑟 will be intersected with 𝑠 and the resulting
range will be a new set𝑠′. The set 𝑠′is said to be the result of
the apply operation. The operation is mathematically defined
as: (®𝑦 ∈𝑠′)⇐⇒(∃®𝑥 s.t (®𝑥 ∈𝑠∧®𝑥 ↦→®𝑦)∈ 𝑟)
The data footprint of the loop can be computed byapplying
read and write relations on the iteration space set: 𝑟1 (𝐼)∪
𝑟2 (𝐼)∪𝑟3 (𝐼)∪𝑤1 (𝐼)
Lexicographic operations. The lexicographical operations
can be applied on sets. 𝑠1 << 𝑠2 outputs all the elements
2
Loop nest
E.g., gemm
Polyhedral 
Loop 
Optimizer
RL-based 
Low-Level 
Code 
Optimizer
High 
Performance 
primitive
System Config: Cache 
Sizes
Figure 1: The compiler optimizations workflow
for ( i = 0 ; i < M; i ++) {
for ( j = 0 ; j < N ; j ++) {
for ( k = 0 ; k < K ; k ++) {
C[ i ] [ j ] += A[ i ] [ k ] ∗ B [ k ] [ j ] ;
}
}
}
Figure 2: Matrix multiplication code
of 𝑠1 that are lexicographically strictly smaller than all the
elements of 𝑠2, while 𝑠1 <<= 𝑠2 gets us the elements of
𝑠1 that are lexicographically smaller than or equal to the
elements of 𝑠2. The lexicographically smallest element of a
set 𝑠is queried usinglexmin 𝑠. Similarly, the lexicographically
largest element is obtained using lexmax 𝑠.
Set difference. The set difference between set 𝑠1 and 𝑠2 is
denoted by 𝑠1 −𝑠2, i.e., the resulting set will have elements
of 𝑠1 that do not appear in 𝑠2.
Polyhedral dependences. The exact data dependences in
loop nests can be computed in the polyhedral model and are
expressed as maps from source iterations to target iterations
involved in the dependence. For cache data reuse analysis
developed in §3.2, we consider four kinds of dependences –
Read-After-Read (RAR), Read-After-Write (RAW, a.k.aflow),
Write-After-Read (WAR, a.k.aanti), and Write-After-Write
(WAW). The data dependencies of the matrix multiplication
code in Figure 2 are shown below.
𝑑1 ={𝑆[𝑖,𝑗,𝑘 ]↦→ 𝑆[𝑖′,𝑗 ′,𝑘′]: 𝑖′= 𝑖∧𝑗′= 𝑗∧𝑘 < 𝑘′< 𝐾}
𝑑2 ={𝑆[𝑖,𝑗,𝑘 ]↦→ 𝑆[𝑖′,𝑗 ′,𝑘′]: 𝑖′= 𝑖∧𝑘′= 𝑘∧𝑗 < 𝑗′< 𝑁}
𝑑3 ={𝑆[𝑖,𝑗,𝑘 ]↦→ 𝑆[𝑖′,𝑗 ′,𝑘′]: 𝑗′= 𝑗∧𝑘′= 𝑘∧𝑖 < 𝑖′< 𝑀}
The dependence 𝑑2 is induced by array reference A[i][k].
An element of array A, say A[0][0] which is accessed in
source iteration [𝑖 = 0,𝑗 = 0,𝑘 = 0]gets reused in target
iterations [𝑖′= 0,𝑗 ′> 0,𝑘′= 0]. The source to target itera-
tion relationships such as this are expressed in a parametric
fashion as the relation 𝑑2.
3.2 Loop transformations
We create a number of code variants by applying loop re-
ordering and tiling transformations and using the PolyDL
techniques [25] select the top code variants.
Working set size computation. We perform cache data reuse
analysis to characterize a loop-nest’s behavior with respect
to a given cache hierarchy. The analysis computes the various
existing data reuses of a program and then for the input cache
hierarchy determines which data reuses are exploitable at
various levels of cache. Each data dependence in a loop is also
an instance of data reuse – the source and target iterations
involved in the dependence touch the same data element
and therefore, the data is reused. For a data dependence and
hence data reuse to be realizable in a given level of cache, all
the data elements accessed between the source and target
iterations of the dependence – the working set – have to be
retained in the cache so that when the execution reaches
the target iteration, the data element(s) used in the source
iteration will still be present in the cache.
We illustrate the computation of the working set sizes
using the running example in Figure 2. Let us examine the
following dependence carried by the𝑗loop arising because of
the array reference 𝐴[𝑖][𝑘]: 𝑑2 = {𝑆[𝑖,𝑗,𝑘 ]↦→ 𝑆[𝑖′,𝑗 ′,𝑘′]:
𝑖′= 𝑖∧𝑘′= 𝑘 ∧𝑗 < 𝑗′ < 𝑁}. Of all the source iterations,
the first/lexicographically minimum iteration is: I𝑠𝑜𝑢𝑟𝑐𝑒 =
{𝑆[𝑖 = 0,𝑗 = 0,𝑘 = 0]}Its target iterations are: {𝑆[𝑖 =
0,𝑗,𝑘 = 0]: 0 < 𝑗 < 𝑁}. Among the target iterations, the
first one is: 𝐼𝑚𝑖𝑛_𝑡𝑎𝑟 = {𝑆[𝑖 = 0,𝑗 = 1,𝑘 = 0]}and the last
one is: 𝐼𝑚𝑎𝑥_𝑡𝑎𝑟 = {𝑆3 [𝑖 = 0,𝑗 = 𝑁 −1,𝑘 = 0]}
3
The number of data elements of the three arrays – A, B, C
accessed between I𝑠𝑜𝑢𝑟𝑐𝑒 and 𝐼𝑚𝑖𝑛_𝑡𝑎𝑟 is derived by applying
the read and write relations on the intervening iteration set
and it is:
𝑊𝑆𝑚𝑖𝑛 = 2𝐾+3
The 𝐾elements of array A –𝐴[0][0,1,...,𝐾 −1], the 𝐾+1
elements of array B – 𝐵[0,1,...,𝐾 −1][0]and 𝐵[0][1], and
finally 2 elements of array C – 𝐶[0][0],𝐶[0][1]accessed
between the source iteration 𝑆[𝑖 = 0,𝑗 = 0,𝑘 = 0]and the
target iteration 𝐼𝑚𝑖𝑛_𝑡𝑎𝑟 = 𝑆[𝑖 = 0,𝑗 = 1,𝑘 = 0]lead to the
𝑊𝑆𝑚𝑖𝑛 size of 2𝐾+3.
The maximum working set size – the number of data
elements touched between I𝑠𝑜𝑢𝑟𝑐𝑒 and 𝐼𝑚𝑎𝑥_𝑡𝑎𝑟 is:
𝑊𝑆𝑚𝑎𝑥 = 𝑁 ×𝐾+𝑁 +1
The𝑊𝑆𝑚𝑎𝑥 size is arrived at by counting the number of array
elements accessed between the source iteration -𝑆[𝑖 = 0,𝑗 =
0,𝑘 = 0]and the target iteration - 𝐼𝑚𝑎𝑥_𝑡𝑎𝑟 = {𝑆3 [𝑖 = 0,𝑗 =
𝑁 −1,𝑘 = 0]}. As far as array A is concerned, 𝐾 elements
of it – 𝐴[0][0,1,...,𝐾 −1]are read. Array B’s elements –
𝐵[0,1,...,𝐾 −1][0,1,...,𝑁 −2]plus 𝐵[0][𝑁 −1]are read
which total 𝐾 ×(𝑁 −1)+ 1. 𝑁 elements of array C are
read and written – 𝐶[0][0,1,...,𝑁 −1]. Therefore, a total
of 𝑁 ×𝐾+𝑁 +1 are read and written.
We have built a code generator to emit a number of pro-
gram variants. The code generator creates the loop variants
by applying tiling and loop interchange program transforma-
tions. The tile sizes are varied as well. The working set size
computation analysis is performed on each program version
generated. Among the many variants generated, the ranking
algorithm described below picks the top 𝑘 best performing
versions, where 𝑘 is a parameter.
DNN-based code ranking algorithm. We assume fully as-
sociative, and exclusive caches. If the working set size cor-
responding to a data reuse in the program is smaller than
the cache size then the data reuse is exploitable in the cache.
The ranking system considers caches at different levels (typ-
ically L1, L2, and L3) and for each data reuse, determines
at what level of cache hierarchy is the data reuse realizable.
We now describe the algorithm to determine the cumulative
working set sizes at each level of cache. The inputs to the
algorithm are the working set sizes computed for a loop nest,
and the cache sizes of the target system. The algorithm de-
termines the fastest level of cache where the working set
size corresponding to each data reuse fits and adds it to that
cache’s working set size. If a working set does not fit in any
cache, then the data reuse happens out of the main memory.
Consequently, the memory’s working set size is updated.
We use a deep neural network (DNNs) for ranking of code
variants. For the purposes of training the DNN model, we
collect the performance data of code variants generated and
v1 L1v1 L2
v1 L3v1 mem
v2 L1
v2 L2
v2 L3v2 mem
o1 = 1, 
if v1 is better 
o2 = 1, 
if v2 is better 
Figure 3: The DNN architecture for ranking of code
variants. The DNN shown is a representative figure.
compute their working set sizes at different levels of the
memory hierarchy. We train the DNN model to perform rela-
tive ordering of two code variants. We then use atournament
based ranking system to assign ranks to the different code
versions created – we play each code variant against every
other code variant. For each variant, we record the number
of wins it has accumulated. We then rank the variants based
on the number of wins – the higher the number of wins, the
higher the rank.
We use a four layer feed forward neural network architec-
ture shown in Figure 3. We normalize the compiler generated
working set sizes using min-max scaling: 𝑥−𝑥𝑚𝑖𝑛
𝑥𝑚𝑎𝑥 −𝑥𝑚𝑖𝑛
. Each
value is subtracted with the minimum value in that feature
column and divided by the feature range. The output layer
consists of two neurons and we use the softmax function
for the output layer. The values of the two output neurons,
because of the use of the softmax function, sum to 1. If the
output value is above a threshold - 𝜃, we consider it a 1, oth-
erwise a 0. If the first neuron fires a 1, then the first variant
is considered the winner. If the second neuron fires a 1, then
the second variant is considered the winner. If both of them
are zero because none of them are above the threshold, then
it is a draw between the two variants. In this work, we set
the threshold 𝜃 to 0.7. We experimented with deeper mod-
els as well. However, depth beyond four did not have any
discernible effect on accuracy.
4 LOW-LEVEL TARGET SPECIFIC INNER
LOOP OPTIMIZATIONS
The high level optimizations as described in §3 are first ap-
plied to the input code and then the inner loops are handed
over the low level optimizer. The low level optimizer focuses
on vectorization and assumes that the data used by the inner
loops is resident in L1 cache. The inner loops are analyzed to
find out which loops are parallel and hence, vectorizable. The
different vectorizable loops present us multiple choices for
vectorization. Further, unroll-and-jam (the loops are unrolled
and the unrolled statements are combined in the inner-most
4
for ( i = 0 ; i < M; i ++) {
for ( j = 0 ; j < N ; j +=16) {
for ( k = 0 ; k < K ; k ++) {
C[ i ] [ j ] += A[ i ] [ k ] ∗ B [ k ] [ j ] ;
C[ i ] [ j +1] += A[ i ] [ k ] ∗ B [ k ] [ j + 1 ] ;
C[ i ] [ j +2] += A[ i ] [ k ] ∗ B [ k ] [ j + 2 ] ;
. . .
C[ i ] [ j +15] += A[ i ] [ k ] ∗ B [ k ] [ j + 1 5 ] ;
}
}
}
Figure 4: Unrolled GEMM code for unroll factors 1, 16,
1
M _ f u l l = (M / 1 ) ∗ 1 ;
N _ f u l l = (N / 1 6 ) ∗ 16 ;
K _ f u l l = ( K / 1 ) ∗ 1 ;
for ( i = 0 ; i < M _ f u l l ; i += 1 ) {
for ( j = 0 ; j < N _ f u l l ; j += 1 6 ) {
vecC = _mm512_load_ps (&C[ i ∗ C S t r i d e + j ] ) ;
for ( k = 0 ; k < K _ f u l l ; k += 1 ) {
vecA = _mm512_set1_ps (A[ i ∗ A S t r i d e + k ] ) ;
vecB =_mm512_load_ps (&B [ k ∗ B S t r i d e + j ] ) ;
vecC =_mm512_fmadd_ps ( vecA , vecB , vecC ) ;
}
_mm512_store_ps (&C[ i ∗ C S t r i d e + j ] , vecC ) ;
}
}
/ / The r e s i d u e c o d e f o r non − f u l l M, N , K
/ / v a l u e s o m i t t e d f o r b r e v i t y .
Figure 5: Auto-generated GEMM code using AVX-512
intrinsics for unroll factors 1, 16, 1
loop) can present various data reuse opportunities. Thus,
the various unroll factors for the loops give rise to multiple
ways of vectorizing the loops and we have to select a scheme
that leads to the highest performance. To help select the best
vectorization parameters, namely, the unroll factors for the
loops, we use Reinforcement Learning (RL).
4.1 Vector intrinsic based code generation
We illustrate the workings of the vectorization scheme and
the use of RL on GEMM inner loops. Figure 4 shows the
matrix multiplication code where the j loop is unrolled by a
factor of 16 and the statements are moved to the inner most
loop (unroll-and-jam). Because the j loop is parallel, it is vec-
torizable. We have built a code generator that generates the
M _ f u l l =(M / 2 ) ∗ 2 ;
N _ f u l l =(N / 1 6 ) ∗ 16 ;
K _ f u l l =(K / 2 ) ∗ 2 ;
for ( i = 0 ; i < M _ f u l l ; i +=2) {
for ( j = 0 ; j < N _ f u l l ; j +=16) {
vecC=_mm512_load_ps (&C[ i ∗ C S t r i d e + j ] ) ;
vecC1=_mm512_load_ps (&C [ ( i + 1 ) ∗ C S t r i d e + j ] ) ;
for ( k = 0 ; k < K _ f u l l ; k +=2) {
vecA= _mm512_set1_ps (A[ i ∗ A S t r i d e +k ] ) ;
vecB=_mm512_load_ps (&B [ k ∗ B S t r i d e + j ] ) ;
vecB1 =_mm512_load_ps (&B [ ( k + 1 ) ∗ B S t r i d e + j ] ) ;
vecA1= _mm512_set1_ps (A[ i ∗ A S t r i d e +( k + 1 ) ] ) ;
vecA2= _mm512_set1_ps (A [ ( i + 1 ) ∗ A S t r i d e +k ] ) ;
vecA3= _mm512_set1_ps (A [ ( i + 1 ) ∗ A S t r i d e +( k + 1 ) ] ) ;
vecC=_mm512_fmadd_ps ( vecA , vecB , vecC ) ;
vecC1=_mm512_fmadd_ps ( vecA2 , vecB , vecC1 ) ;
vecC=_mm512_fmadd_ps ( vecA1 , vecB1 , vecC ) ;
vecC1=_mm512_fmadd_ps ( vecA3 , vecB1 , vecC1 ) ;
}
_mm512_store_ps (&C[ i ∗ C S t r i d e + j ] , vecC ) ;
_mm512_store_ps (&C [ ( i + 1 ) ∗ C S t r i d e + j ] , vecC1 ) ;
}
}
/ / The r e s i d u e c o d e f o r non − f u l l M, N , K
/ / v a l u e s o m i t t e d f o r b r e v i t y .
Figure 6: Auto-generated GEMM code using AVX-512
intrinsics for unroll factors 2, 16, 2
vectorized code using the vector intrinsics of the target CPU
platform. Figure 5 shows the generated code using AVX-512
intrinsics to run on vector units that can work on 512 bits
of data simultaneously. The datatype of the variables in the
shown code is 32 bit floating point numbers. Therefore, we
can perform arithmetic operations on 16 floating point num-
bers (16 ×32 = 512) at the same time. In Figure 4, we observe
that the same array element – “A[i][k]” is used in all 16 arith-
metic operations. Therefore, it is broadcast to all elements
of the vector register using the _mm512_set1_ps vector in-
trinsic. The 16 elements of the C array – “C[i][j]” through
“C[i][j+15]” are loaded using the _mm512_load_ps vector
intrinsic. Since the loaded C elements are reused in all of the
inner-most k loop, the loading is hoisted out of thek loop. In
a similar fashion, the 16 elements of the B array – “B[k][j]”
through “B[k][j+15]” are loaded using the _mm512_load_ps
vector intrinsic. The 16 addition and multiplication opera-
tions are performed using the fused-multiply-add operation
through the intrinsic _mm512_fmadd_ps. After the C vector
is accumulated into in the k loop, the results are stored back
5
AVX
codeGen
 Next State 
 Action 
AGENT
Environment
 Reward 
Figure 7: Reinforcement Learning for target specific
low level optimization.
into the C array using _mm512_store_ps outside of the k
loop.
In the GEMM code, all three loops – i,j, and k loops carry
data reuse. Consequently, when we unroll any of the loops,
that leads to reuse of the data of one of the three arrays – A,
B, and C. For example, in Figure 6, the i loops is unrolled by
a factor of 2 and it leads to reuse of the array B: The array
access expression for B is “B[k][j]” and it is free of the i
loop variable and therefore for all values of i, the same B
array element would be used. In the code we observe that
vecB is used in two fused-multiply-add operations – while
computing vecC and vecC1. When we unroll the inner-most
k loop, that helps us schedule the load and fma operations
further apart thereby increasing the chances of the data being
loaded in the vector registers when the execution reaches
the fma operations.
Thus, by unrolling various loops we can increase the data
reuse in vector registers, and schedule the load operations in
such a way that memory latency is tolerated well. However,
because the number of vector registers is limited, a certain
choice of an unroll factor for a loop will impose constraints
on the unroll factors of other loops. The interplay between
the amount of data reuse, scheduling, and the impact of
data reuse of different arrays could be complex. We use rein-
forcement learning which in turn uses a neural network to
determine the best unroll factors for the loops.
4.2 Reinforcement Learning
The choices of unroll factors for the inner loops constitute
the state space for reinforcement learning (RL). The agent
will suggest whether to increase unroll factors or to decrease
them. The increment/decrement of the unroll factors form
the actions. The target specific code generator will carry
out the actions suggested by the agent by generating code
with the new unroll factors and the code is run on the target
platform. The actions will either lead to a higher performance
or a lower performance vis-a-vis the performance of the prior
state. If the action leads to a higher performance, we will
encode it as a positive reward – the relative performance
increase. If the action causes the performance to degrade, it
is denoted as a negative reward – the relative performance
decrease. Figure 7 shows the RL set-up.
The agent will use a neural network to suggest next ac-
tions to undertake. While the state space exploration is be-
ing conducted, we will have two phases – exploration, and
exploitation. In the exploration phase, the agent will recom-
mend random actions and the reward obtained will be used
to continually train the neural network to predict actions
that will lead to larger positive rewards and thus higher per-
formance states. In the exploitation phase, the agent will
query the neural network for the best actions – actions that
will lead to the biggest rewards. The transitions between
the exploration and exploitation phases are controlled by
the exploration decay rate . We set it in such a way that at
initial stages exploration is selected more often, and later
exploitation is chosen more.
We train a neural network to encode the policy for RL –
whether to increment the unroll factors or two decrement
them. The neural network comprises of six intermediate lay-
ers – two blocks of Dense, Batch Normalization and Dropout
layers. For Dense layers we use Relu as the activation func-
tion. We set the drop-out rate of 0.25 for Dropout layers to
avoid overfitting. The output layer is a dense layer which
has as many neurons as the number of actions. For matrix
multiplication, there are 7 actions possible: 2 actions for each
unroll factor (whether to increment or to decrement) and a
special state to indicate no further action is necessary.
5 EXPERIMENTAL EVALUATION
We evaluate the performance of our compiler framework on
the GEMM operation for a range of matrix sizes. The GEMM
operation is, C = A.B where A, B, and C are matrices. Matrix
C is of size 𝑀 ×𝑁, A of size 𝑀 ×𝐾 and B of size 𝐾 ×𝑁.
We compare our compiler optimizations against three other
systems: 1) The matrix multiplication code shown in Figure
2 optimized to the highest levels using the Intel C compiler
– icc version 19.1.3.304 with the optimization flag -O3. For
parallel runs, we parallelize the outermost – ‘i’ loop with
OpenMP pragmas. We consider the resulting performance
to be the baseline. 2) The latest version of Intel oneDNN
library version v2.2. The oneDNN library is an expert coded
library for inter alia, GEMMs. 3) The latest release of TVM
[8] v0.7.0. We use the optimization guide published on the
TVM website for GEMMs [2] to obtain its performance. Ad-
ditionally, we tune the performance of TVM by exploring a
number of tile sizes and report the best performance among
the different tile sizes explored. The experiments are run on
6
0
20
40
60
80
100
120
16_16_16 32_32_32 64_64_64 128_128_128 32_32_64 32_64_32 64_32_32
GFLOPS/s
GEMM sizes: M, N, K values
Single-core performance for small GEMMs
Baseline Ours oneDNN
Figure 8: Performance of our inner loop optimization using RL on matrices that fit in private L1/L2 caches.
Intel(R) Xeon(R) Platinum 8280 (Cascade Lake) servers run-
ning at the frequency of 2.70 GHz. A single socket processor
has 28 cores, 32KB private L1 cache, 1MB private L2 cache,
and 39MB shared L3 cache.
5.1 Evaluation of Low-level optimizations
We first assess the efficaciousness of the low-level optimiza-
tion scheme we described in Section 4. To do so, we select
matrix sizes such that all the matrices will fit in in the pri-
vate L1/L2 cache of the processor and thus the code does not
require any loop transformations to enhance data locality.
We show the performances achieved on a single core of the
Xeon processor in Figure 10 by the ICC compiler – baseline,
our toolchain, and the oneDNN library. We run each code
a 100 times and report the average performance observed.
The baseline performance ranges from 0.18 GFLOPS/s (for
M = 16, N = 16, and K = 16) to 7.53 GFLOPS/s (for M = 128,
N = 128, and K = 128). The peak performance of a core of
the said Xeon system is ~118 GFLOPS/s. Thus, without any
further optimizations such as the ones we have described in
the paper, we observe that we obtain very low performance.
The oneDNN performance for all problem sizes save for M
= N = K = 128 is lower than ours. Our toolchain reaches a
sizable percentage of the machine’s peak performance. The
lowest performance gotten is 39.10 GFLOPS/s for M = N = K =
16, and the highest performance reached is 84.34 GFLOPS/s
Table 1: GEMM sizes for performing matrix multipli-
cation. Source: Qin et al [21].
Workload Application M N K
GNMT Machine translation
128 2048 4096
320 3072 4096
1632 36548 1024
2048 4096 32
DeepBench General workload 1024 16 500000
35 8457 2560
Transformer Language Understanding 31999 1024 84
84 1024 4096
NCF Collaborative Filtering 2048 1 128
256 256 2048
for M = N = K = 128. As the problem size increases, the
work to be performed increases, and due to Instruction Level
Parallelism (ILP) and because the SIMD units can be kept
more busy, the performance goes up. The experiments show
that our target-specific code generation and reinforcement
learning scheme is extremely effective in vectorizing the
code well.
7
0
5
10
15
20
25
30
35Speed-ups over baseline
GEMM sizes: M, N, K values
Single-core performance for GEMMs on a Xeon server
Ours AutoTVM oneDNN
52 
GFLOPS/s
58
61 
GFLOPS/s
38 42
32 
GFLOPS/s
52 51
1.4
60 
GFLOPS/s
7.6X
13.7X
4.9X
Figure 9: The speed-ups achieved over the baseline by various systems. The absolute performances in GFLOPS/s
obtained by our compiler toolchain for all problem sizes are shown.
5.2 Evaluation of High-level and Low-level
optimizations together
We next conduct experiments to evaluate how well our com-
piler toolchain performs when we need to apply both high-
level and low-level optimizations: when matrix sizes are
larger and therefore necessitate loop transformations includ-
ing tiling to enhance data reuse in cache memories. Table 1
lists the matrix sizes we perform the experiments with. The
matrix sizes are drawn from various deep learning applica-
tions. We note that the matrix sizes are wide ranging and
therefore, test the versatility of our system in being able to
come up with high performance implementations for varied
matrix sizes.
Two-level tiling is applied on the matrix multiplication
code. Consequently, there are six tile sizes that need to be
selected. We choose the best tile sizes using the high level
optimization methodology described in Section 3: We first
create a number of code variants by varying the tile sizes. The
data reuse analyzer is run on each code variant and it outputs
the working set sizes. We execute the code variants on the
target machine and measure their performance. The working
set sizes and the performance data are used together to train
the DNN model for ranking of code variants. While training
the model, we use the training data corresponding to 70% of
the code variants. Once the model is trained, for each GEMM
size, we use the DNN model to select the top 10% best tile
sizes from the space of candidate tile sizes. For the top 10%
tile sizes thus selected, we apply low-level transformations
(Section 4) to vectorize the code.
Figure 9 shows the speed-ups achieved over the baseline
by our system, AutoTVM, and oneDNN library for sequential
runs. We show the performance obtained by our compiler in
absolute terms – in terms of GFLOPS/s for each problem size.
When dealing with problem sizes that are odd numbers or
when the optimal unroll factor for a loop does not divide the
corresponding tile size, then that leads to executing some
parts of the computation in scalar mode (and not vectorized).
For example, the “residue” code shown in Figure 5. It leads
to lower performance compared to when everything is run
in the vector mode. For example, for the problem sizes M
= 31999, N = 1024, and K = 84, the optimal unroll factors
through the high-level and low-level optimizations are de-
termined to be 4, 32, and 1 for the three loops respectively.
Because 4 does not divide 31999 exactly, it causes some part
of the computation to be run using scalar units. Our toolchain
achieves the highest performance of 60.7 GFLOPS/s for the
problem size M = 1632, N = 36548, K = 1024. For this prob-
lem size, the highest speed-up relative to the baseline code
achieved is: 17.96X. The lowest performance of 1.4 GFLOPS/s
is observed for M = 2048, N = 1, K = 128. The reason is, vec-
torization on the “j” loop as shown in Figure 5 is not possible
8
0
5
10
15
20
25
30
35
40
45Speed-ups over baseline
GEMM sizes: M, N, K values
Performance for GEMMs on a 28-core Xeon server
Ours AutoTVM oneDNN
920 
GFLOPS/s
989
1408 
GFLOPS/s
496 414
577 
GFLOPS/s
1168 540
19
464 
GFLOPS/s 8.2X
15.3X
5.4X
Figure 10: The speed-ups achieved over the baseline by various systems. The absolute performances in GFLOPS/s
obtained by our compiler toolchain for all problem sizes are shown.
because the loop length of the “j” loop is 1. If we were to
vectorize on the “i” loop, that will induce non-contiguous
data accesses for the A and C matrices and loading of data
through vector-loads will not be possible. That explains the
low performance when N = 1. We obtain on average (geomet-
ric mean) speed-up of 7.6X over the baseline. The speed-ups
achieved by the AutoTVM system and the oneDNN library
are 4.9X and 13.7X respectively. In 8 out of 10 cases, our com-
piler outperforms AutoTVM. In oneDNN implementations,
data prefetch instructions are carefully inserted. For exam-
ple, low-level prefetch instructions such as prefetcht0 and
prefetcht2 which fetch data to all levels of cache and to L2
cache respectively are used extensively and that minimizes
the number of instances when the processor has to wait for
the arrival of data from memory. Inter alia, such software
data prefetching strategies employed in oneDNN explain its
higher performance.
We measure the performance of parallel code when it is
run on all 28 cores of the Xeon server. Figure 10 shows the
speed-ups over the baseline for the three systems. The base-
line code is parallelized as well. The average speed-ups over
the baseline for our compiler, AutoTVM, and oneDNN are
8.2X, 5.4X, and 15.3X respectively. For the first two problem
sizes, namely M = 128, N = 2048, K = 4096, and M = 320, N
= 3072, K = 4096, our compiler delivers higher performance
even compared to oneDNN. Our tool achieves the highest
performance of 1408 GFLOPS/s for the problem size M = 1632,
N = 36548, K = 1024. Incidentally, the highest performance
in sequential experiments is seen for the same problem size.
For that problem size, it represents the parallel speed-up of
23.2X on a 28-core machine. When the number of tiles of the
parallelized loop is not a multiple of the number of cores (28),
that leads to slight load imbalance among the cores. Con-
sequently, we are hindered from achieving a perfect linear
speed-up as the number of cores is increased.
6 RELATED WORK
In recent years, there has been a renewed interest in the
application of Artificial Intelligence (A.I.) techniques for pro-
gram optimization. The use of A.I. has been explored broadly
for two purposes: 1) for program representation in an embed-
ding space [4, 6, 18, 26], and 2) for performance optimization
[8, 9, 12, 20, 34, 35]. In the former use case, once a program
representation has been obtained then it has been used for
tasks such as code comprehension, similar code search etc.
In the latter case when A.I. has been applied for program
optimization, it has been used to find optimal program trans-
formations and optimal parameters for program transforma-
tions (such as loop unroll factors). Our work presented in
the paper fits the mold of the latter category where we use
9
A.I. for performance enhancement. Below, we describe sev-
eral closely related works and detail how our present work
improves upon and/or is different from prior works.
AutoTVM [9] uses machine learning approaches to derive
efficient execution schedules for tensor programs. It explores
the use of two distinct machine learning techniques: In the
first approach, from a given tensor program, domain specific
features such as memory access count, data reuse ratio etc
are extracted. Then, XGBoost, a form of decision tree based
learners are trained to perform relative ranking of schedules
based on their performance. In the second approach, the ten-
sor program is encoded into an embedding vector using the
TreeGRU method [24]. The relative performance prediction
of the schedules is performed on the encoding thus obtained.
FlexTensor [35] is a tensor computation optimization frame-
work for heterogeneous architectures. The hardware targets
for which FlexTensor can create high performance code in-
clude CPUs, GPUs, and FPGAs. FlexTensor uses machine
learning techniques to derive optimized execution schedules.
It uses two machine learning approaches: neural networks
for performance prediction, and reinforcement learning for
navigating the space of possible schedules. The schedule
space is navigated using Q-learning [29] based reinforcement
learning. The Q-learning approach will guide the search –
along the directions of the search space with the best perfor-
mance. The performance prediction for different schedules
which is an input to the Q-learning algorithm is performed
using a feed-forward neural network.
Ansor [34] is a latest addition to the slew of TVM-based
auto-tuning systems. Ansor considers a larger schedule space
compared to the prior auto-tuning frameworks. Some of the
innovations in Ansor include, 1) organization of the search
space in a hierarchical manner 2) use of evolutionary search
techniques 3) specification of schedules by recursive appli-
cation of derivation rules 4) sampling of the search space
– by periodically running random schedules on the target
hardware to better guide the search.
Park et al [20] use machine learning to select the best pro-
gram transformation sequence among the available set of
program transformation recipes. They run a given input pro-
gram and obtain hardware counter values such as L1 cache
misses. Then, they also apply a sequence of program transfor-
mations and observe the achieved speed-up. The hardware
counters and the program transformations together are used
as features for training a machine learning model to predict
the speed-ups. Once the machine learning model is trained,
it is used as follows for selecting the best program trans-
formation sequence among a multitude of possibilities for
a given program. The unseen input program is run on the
target hardware and the hardware counters are read. The
hardware counters along with a transformation sequence
will be input to the trained machine learning model to predict
the speed-up that could be obtained for the transformation
sequence. The transformation sequence delivering the high-
est speed-up among various transformation sequences will
be selected.
Various hand written implementations of basic linear al-
gebra operations have been provided in various libraries
like oneDNN [3], BLIS [33], OpenBLAS [32], GotoBLAS [13].
ATLAS [30] is an autotuner where it generates various low-
level C-implementations and finds the best performing one
by executing the code on the target machine. POCA [ 23]
generates LLVM-IR based vectorized GEMM micro-kernel
where architecture independent optimizations can be applied.
AUGEM [28] is a template based code generator for DLA
(Dense Linear algebra) operations. It replaces the common
predefined C-code with the generated assembly level code.
Kevin et al. [22] use assembly level features for analytical
modeling of the SIMD code with Machine learning and find
the best loop transformations for vectorization. Monsifrot
et al. [19] use a Machine learning approach to find the best
unrolling factors.
Polyhedral compilation techniques have been developed
for source-to-source code transformation for better cache
locality and parallelization. The Pluto compiler [7] derives
an execution schedule for the code that attempts to minimize
data reuse distances. The effect of the Pluto transformation
will be that iterations that use the same data will be executed
close to each other in time and therefore, the code will be
cache friendly. However, the performance obtained by poly-
hedral tools including Pluto’s can be only slightly better than
that of back-end compiler’s such as Intel ICC’s and far from
other approaches such as AutoTVM’s [25]. In this paper, we
show that our techniques outperform AutoTVM. The reason
for the inability of the purely polyhedral approaches is, they
operate at a high level (source-to-source) and therefore, do
not perform low-level orchestration of vectorization, vec-
tor register allocation, detailed instruction scheduling (e.g.,
the kinds of low level optimizations we have described in
Section 4). The latter aspects are crucial to achieving high
performance.
In our present work, we identified two distinct program
optimizations that we need to concern ourselves with in or-
der to obtain high performance on the target architectures.
Because the high-level and low-level optimizations are dif-
ferent, we developed different A.I. techniques for them. For
high level optimizations we employed polyhedral compila-
tion techniques to extract features from loops and used a
deep learning model to identify loop transformations that
will yield high performance. For better vectorization, we
combined an intrinsics based code generator with reinforce-
ment learning (RL) to derive optimal parameters for the
code generator. We have combined traditional compiler tech-
niques with A.I. where appropriate. In particular, we have
10
used A.I. where an accurate cost model is difficult to define
and hence, we take advantage of A.I. ’s unique ability to learn
from performance data.
7 CONCLUSION
In this paper we presented CPU-focused compiler techniques
for high-level and low-level program optimizations. The
high-level optimizations enhance data locality of programs
and the low-level transformations effectively vectorize code.
A.I. techniques in conjunction with polyhedral compilation
algorithms and target-specific code generator help us achieve
high levels of performance. We demonstrated that matrix-
multiplications which lie at the heart of deep learning can be
effectively optimized using the developed compiler toolchain.
The presented approach will help re-target the toolchain
to newer computer architectures seamlessly – only a new
target-specific code generator need be created, and the rest of
the toolchain can be repurposed without modifications. Thus,
the compiler framework described will enable realizing good
performance out-of-the-box for new hardware architectures
and new DL operators.
REFERENCES
[1] 2015. Why GEMM is at the heart of deep learning . https://petewarden.
com/2015/04/20/why-gemm-is-at-the-heart-of-deep-learning/
[2] 2020. How to optimize GEMM on CPU . https://tvm.apache.org/docs/
tutorials/optimize/opt_gemm.html
[3] 2020. oneAPI Deep Neural Network Library (oneDNN). https://github.
com/oneapi-src/oneDNN
[4] Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019.
code2vec: Learning distributed representations of code. Proceedings of
the ACM on Programming Languages 3, POPL (2019), 1–29.
[5] Gaurav Batra, Zach Jacobson, Siddarth Madhav, Andrea Queirolo, and
Nick Santhanam. 2018. Artificial-intelligence hardware: New oppor-
tunities for semiconductor companies . https://www.mckinsey.com/
industries/semiconductors/our-insights
[6] Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. 2018.
Neural code comprehension: A learnable representation of code se-
mantics. Advances in Neural Information Processing Systems 31 (2018),
3585–3597.
[7] Uday Bondhugula, Albert Hartono, J. Ramanujam, and P. Sadayappan.
2008. A Practical Automatic Polyhedral Program Optimization System.
In ACM SIGPLAN Conference on Programming Language Design and
Implementation (PLDI) .
[8] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan,
Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze,
et al. 2018. {TVM}: An automated end-to-end optimizing compiler
for deep learning. In 13th {USENIX}Symposium on Operating Systems
Design and Implementation ( {OSDI}18). 578–594.
[9] Tianqi Chen, Lianmin Zheng, Eddie Yan, Ziheng Jiang, Thierry Moreau,
Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. 2018. Learn-
ing to optimize tensor programs. In Advances in Neural Information
Processing Systems . 3389–3400.
[10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
2018. Bert: Pre-training of deep bidirectional transformers for language
understanding. arXiv preprint arXiv:1810.04805 (2018).
[11] Paul Feautrier. 1996. Automatic parallelization in the polytope model.
In The Data Parallel Programming Model . Springer, 79–103.
[12] Franz Franchetti, Tze Meng Low, Doru Thom Popovici, Richard M
Veras, Daniele G Spampinato, Jeremy R Johnson, Markus Püschel,
James C Hoe, and José MF Moura. 2018. SPIRAL: Extreme performance
portability. Proc. IEEE 106, 11 (2018), 1935–1968.
[13] Kazushige Goto and Robert A. van de Geijn. 2008. Anatomy of High-
Performance Matrix Multiplication. ACM Trans. Math. Softw. 34, 3, Ar-
ticle 12 (May 2008), 25 pages. https://doi.org/10.1145/1356052.1356053
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep
residual learning for image recognition. In Proceedings of the IEEE
conference on computer vision and pattern recognition . 770–778.
[15] Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-rahman Mo-
hamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick
Nguyen, Brian Kingsbury, et al. 2012. Deep neural networks for acous-
tic modeling in speech recognition. IEEE Signal processing magazine
29 (2012).
[16] Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav
Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden,
Al Borchers, et al . 2017. In-datacenter performance analysis of a
tensor processing unit. In 2017 ACM/IEEE 44th Annual International
Symposium on Computer Architecture (ISCA) . IEEE, 1–12.
[17] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet
classification with deep convolutional neural networks. In Advances
in neural information processing systems . 1097–1105.
[18] Sifei Luan, Di Yang, Celeste Barnaby, Koushik Sen, and Satish Chan-
dra. 2019. Aroma: Code recommendation via structural code search.
Proceedings of the ACM on Programming Languages 3, OOPSLA (2019),
1–28.
[19] Antoine Monsifrot, François Bodin, and Rene Quiniou. 2002. A Ma-
chine Learning Approach to Automatic Production of Compiler Heuris-
tics. InProceedings of the 10th International Conference on Artificial Intel-
ligence: Methodology, Systems, and Applications (AIMSA ’02) . Springer-
Verlag, Berlin, Heidelberg, 41–50.
[20] Eunjung Park, John Cavazos, Louis-Noël Pouchet, Cédric Bastoul, Al-
bert Cohen, and P Sadayappan. 2013. Predictive modeling in a polyhe-
dral optimization space. International journal of parallel programming
41, 5 (2013), 704–750.
[21] Eric Qin, Ananda Samajdar, Hyoukjun Kwon, Vineet Nadella, Sudar-
shan Srinivasan, Dipankar Das, Bharat Kaul, and Tushar Krishna. 2020.
Sigma: A sparse and irregular gemm accelerator with flexible inter-
connects for dnn training. In 2020 IEEE International Symposium on
High Performance Computer Architecture (HPCA) . IEEE, 58–70.
[22] Kevin Stock, Louis-Noël Pouchet, and P. Sadayappan. 2012. Using
Machine Learning to Improve Automatic Vectorization. ACM Trans.
Archit. Code Optim. 8, 4, Article 50 (Jan. 2012), 23 pages. https://doi.
org/10.1145/2086696.2086729
[23] Xing Su, Xiangke Liao, and Jingling Xue. 2017. Automatic Generation
of Fast BLAS3-GEMM: A Portable Compiler Approach. In Proceedings
of the 2017 International Symposium on Code Generation and Optimiza-
tion (Austin, USA) (CGO ’17) . IEEE Press, 122–133.
[24] Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015.
Improved semantic representations from tree-structured long short-
term memory networks. arXiv preprint arXiv:1503.00075 (2015).
[25] Sanket Tavarageri, Alexander Heinecke, Sasikanth Avancha, Bharat
Kaul, Gagandeep Goyal, and Ramakrishna Upadrasta. 2021. PolyDL:
Polyhedral Optimizations for Creation of High-performance DL Primi-
tives. ACM Transactions on Architecture and Code Optimization (TACO)
18, 1 (2021), 1–27.
[26] S VenkataKeerthy, Rohit Aggarwal, Shalini Jain, Maunendra Sankar
Desarkar, Ramakrishna Upadrasta, and YN Srikant. 2020. IR2Vec:
LLVM IR Based Scalable Program Embeddings. ACM Transactions on
Architecture and Code Optimization (TACO) 17, 4 (2020), 1–27.
11
[27] Sven Verdoolaege. 2010. isl: An integer set library for the polyhedral
model. In International Congress on Mathematical Software . Springer,
299–302.
[28] Qian Wang, Xianyi Zhang, Yunquan Zhang, and Qing Yi. 2013.
AUGEM: Automatically Generate High Performance Dense Linear
Algebra Kernels on X86 CPUs. In Proceedings of the International Con-
ference on High Performance Computing, Networking, Storage and Analy-
sis (Denver, Colorado) (SC ’13) . Association for Computing Machinery,
New York, NY, USA, Article 25, 12 pages. https://doi.org/10.1145/
2503210.2503219
[29] Christopher JCH Watkins and Peter Dayan. 1992. Q-learning.Machine
learning 8, 3-4 (1992), 279–292.
[30] R Clinton Whaley and Jack J Dongarra. 1998. Automatically tuned
linear algebra software. In SC’98: Proceedings of the 1998 ACM/IEEE
conference on Supercomputing . IEEE, 38–38.
[31] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad
Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao,
Klaus Macherey, et al . 2016. Google’s neural machine translation
system: Bridging the gap between human and machine translation.
arXiv preprint arXiv:1609.08144 (2016).
[32] Zhang Xianyi, Wang Qian, and Zhang Yunquan. 2012. Model-Driven
Level 3 BLAS Performance Optimization on Loongson 3A Processor.
In Proceedings of the 2012 IEEE 18th International Conference on Parallel
and Distributed Systems (ICPADS ’12) . IEEE Computer Society, USA,
684–691. https://doi.org/10.1109/ICPADS.2012.97
[33] Field G. Van Zee, Tyler M. Smith, Bryan Marker, Tze Meng Low, Robert
A. Van De Geijn, Francisco D. Igual, Mikhail Smelyanskiy, Xianyi
Zhang, Michael Kistler, Vernon Austel, John A. Gunnels, and Lee
Killough. 2016. The BLIS Framework: Experiments in Portability.ACM
Trans. Math. Softw. 42, 2, Article 12 (June 2016), 19 pages. https:
//doi.org/10.1145/2755561
[34] Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu,
Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen,
et al. 2020. Ansor: Generating high-performance tensor programs for
deep learning. In 14th {USENIX}Symposium on Operating Systems
Design and Implementation ( {OSDI}20). 863–879.
[35] Size Zheng, Yun Liang, Shuo Wang, Renze Chen, and Kaiwen Sheng.
2020. FlexTensor: An Automatic Schedule Exploration and Optimiza-
tion Framework for Tensor Computation on Heterogeneous System.
In Proceedings of the Twenty-Fifth International Conference on Archi-
tectural Support for Programming Languages and Operating Systems .
859–873.
12

An Evaluation of Different Modeling Techniques
for Iterative Compilation
Eunjung Park
University of Delaware
Newark, USA
epark@cis.udel.edu
Sameer Kulkarni
University of Delaware
Newark, USA
skulkarn@cis.udel.edu
John Cavazos
University of Delaware
Newark, USA
cavazos@cis.udel.edu
ABSTRACT
Iterative compilation techniques, which involve iterating over
diﬀerent sets of optimizations, have proven useful in helping
compilers choose the right set of optimizations for a given
program. However, compilers typically have a large num-
ber of optimizations to choose from, making it impossible to
iterate over a signiﬁcant fraction of the entire optimization
search space. Recent research has proposed to “intelligently”
iterate over the optimization search space using predictive
methods. In particular, state-the-art methods in iterative
compilation use characteristics of the code being optimized
to predict good optimization sequences to evaluate. Thus,
an important step in developing predictive methods for com-
pilation is deciding how to model the problem of choosing
the right optimizations.
In this paper, we evaluate three diﬀerent ways of model-
ing the problem of choosing the right optimization sequences
using machine learning techniques. We evaluate a novel
prediction modeling technique, namely a tournament pre-
dictor, that is able to eﬀectively predict good optimization
sequences. We show that our tournament predictor can out-
perform current state-of-the-art predictors and the most ag-
gressive setting of the Open64 compiler ( -Ofast) on an av-
erage by 75% in just 10 iterations over a set of embedded
and scientiﬁc kernels. Moreover, using our tournament pre-
dictor, we achieved on average 10% improvement for a set
of MiBench applications.
Keywords
compiler optimization, iterative compilation, machine learn-
ing, regression
1. INTRODUCTION
Most applications can greatly beneﬁt from a ﬁne-tuned set
of compiler optimization sequences. However, ﬁnding the
right set of optimizations for an application is a non-trivial
task since the search space is extremely large. In addition,
it is diﬃcult to build analytical models that can predict how
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for proﬁt or commercial advantage and that copies
bear this notice and the full citation on the ﬁrst page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior speciﬁc
permission and/or a fee.
Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$10.00.
combinations of optimizations will interact with each other.
Recent research has focused on intelligent search space ex-
ploration, in order to search for the right optimization se-
quence eﬃciently [5, 7, 9, 10, 14, 18, 20, 26]. In this approach,
models are developed automatically using machine learning
or statistical techniques to predict good optimizations to ap-
ply based on characteristics of the code being optimized. A
model that uses characteristics (or features) of the code has
the potential to predict optimizations that are speciﬁcally
tailored to the code.
To use this approach, a key step is “how to model” the
problem of choosing the right set of optimizations to apply
to a particular code being optimized. One common method
of modeling this problem is to develop a decision function
that takes as input a given set of features of the code to be
optimized and produces as output a prediction of whether a
speciﬁc optimization should be applied or not. We can de-
velop several of these decision functions each of which con-
trols a speciﬁc optimization. Using these decision functions,
we can construct a sequence of optimizations to apply to the
code. A key property is framing the optimization decision
problem so that the function to be learned is as simple as
possible.
This paper evaluates three diﬀerent techniques for mod-
eling the problem of predicting a good set of optimizations
for a given program. Note, we do not solve the problem
of predicting the ordering of optimizations, but instead pre-
dict the set of optimizations that should be turned on. We
use a large set of kernels which provides good coverage of
a wide range of embedded and scientiﬁc applications. To
evaluate our models on a reasonable fraction of the opti-
mization search space, we used random search to generate
a set of optimization sequences. This set of randomly gen-
erated optimization sequences is used to train and evaluate
our models.
In addition, we propose a new modeling technique to pre-
dict good optimization sequences to apply for a given pro-
gram. We refer to this new modeling technique as the tour-
nament predictor , and we compare this technique to two
state-of-the-art prediction models, which we refer to as the
sequence predictor and the speedup predictor. Each of these
prediction models are constructed using performance coun-
ters to characterize the dynamic behavior of a program. In
recent work, performance counters were shown to be better
at characterizing applications than static code features [7].
We collected performance counters for kernels and optimized
those kernels with a large set of randomly selected optimiza-
tion sequences. We evaluated our three modeling techniques
...... ...
...
Output: probability of opt 1
being beneﬁcial
Output: probability of opt 1
being beneﬁcial
N models, one per optimization
Performance Counters Performance Counters
(a) Sequence Predictor
Performance Counters Optimization Sequence O
... ...
Output: predicted speedup of
given sequence O over baseline
 (b) Speedup Predictor
Performance Counters Opt Sequence O1
... ...
Output: predicted speedup difference
if O1 is preferred over O2 then Output: positive
else Output: negative
Opt Sequence O2
...
 (c) Tournament Predictor
Figure 1: Three prediction models that are evaluated in this paper.
using two diﬀerent machine learning algorithms, namely lin-
ear regression and support vector machines (SVMs). Also,
our models were compared in two diﬀerent scenarios: a non-
iterative scenario where only the best predicted sequence is
used and in an iterative scenario where the top-N predicted
sequences are used.
Using just the top predicted sequence (i.e., the non-iterative
scenario), the tournament predictor trained using SVMs out-
performs all models achieving on our kernel suite an average
of 37% improvement over our baseline, the most aggressive
optimization setting in our compiler, Open64 using -Ofast.
When we use the top 10 predicted sequences (i.e., the itera-
tive fashion), the sequence and speedup models achieve close
to 70% improvement and the tournament predictor achieves
75% improvement over our baseline. Moreover, we tested
our models with several benchmarks from the MiBench [17]
benchmark suites to show how prediction models trained
with kernels performed with embedded applications.
The rest of the paper is organized as follows. We ﬁrst de-
scribe the problem of developing predictors for intelligent
modeling, and we brieﬂy describe the diﬀerent modeling
techniques in Section 2. We then give an overview of our
solution in Section 3 giving more details of our models, in-
cluding how we construct them. Section 3.2 describes the
machine learning techniques used to build the prediction
models. Section 4 explains the characteristics of the pro-
grams and the optimization space we used in our evaluation
framework. Section 5 describes our evaluation setup, and
Section 6 presents our results and analysis. Section 7 de-
scribes related work, and Section 8 presents our conclusions.
2. LEARNING TO OPTIMIZE
Recent work has shown that iterative compilation applied
to certain programs can achieve signiﬁcant beneﬁts over the
highest setting available in a compiler. However, many of
the proposed techniques for exploring optimizations (e.g.,
genetic algorithms [5], random search [21], statistical tech-
niques [18], or exhaustive search [20]) are expensive, which
limits their practical use. This has led compiler researchers
to propose using “intelligent” prediction models that focus
exploration to beneﬁcial areas of the optimization search
space [4,7,11,13,22]. Prediction models can reduce the cost
of ﬁnding good optimizations, but increase complexity in the
design of the search function because models require char-
acterizing the program being optimized (e.g., with source
code features or performance counters), generation of train-
ing data, and a training phase. One important step in de-
signing the prediction model is how to phrase the problem
of choosing good optimizations for a program. Several ways
of modeling the problem of ﬁnding good optimizations have
been proposed in the literature, but there has been little ef-
fort on evaluating these diﬀerent methods. In this paper, we
show this step can have a signiﬁcant impact on code being
optimized. Section 2.1 describes three diﬀerent prediction
models that we evaluate in this paper. In Section 2.2, we
present preliminary experiment results showing the poten-
tial of our new predictor, a tournament predictor.
2.1 Modeling the Optimization Problem
A general formulation of the problem is to construct a
prediction model that takes a characterization of a program
being optimized as input and generates a set of one or more
optimization sequences to evaluate as output (either implic-
itly or explicitly). However, there are several ways to specif-
ically model this problem. In this paper, we evaluate three
diﬀerent methods of modeling the problem of ﬁnding good
optimization sequences, which we name the sequence pre-
dictor, the speedup predictor, and the tournament predictor.
Figure 1 depicts the models that we evaluate in this paper.
We brieﬂy describe each of these predictors here and in more
detail in Section 3.1.
Previous work [7] has proposed to model the problem by
characterizing a program using performance counters and
generating an optimization sequence that will beneﬁt the
program. The performance counter characterization serves
as input to a model, and the model predicts a probability
distribution of optimizations to apply to that program. We
term this model a sequence predictor because it can be used
to construct a sequence of optimizations. Another recent
model that has been proposed [6,12] takes as input both the
characterization of the program being compiled and an op-
timization sequence, and it predicts as output the speedup
of that optimization sequence relative to a default optimiza-
tion setting. We refer to this model as the speedup predictor.
Finally, we propose a new method of choosing optimization
sequences, the tournament predictor, which takes as input
a triple corresponding to the characterization of the pro-
gram and two optimization sequences. This model predicts
whether the speedup of the ﬁrst optimization sequence will
be more or less than the second optimization sequence. We
can use this predictor to provide an ordering of a set of op-
timization sequences to be used for iterative compilation.
2.2 Preliminary Experiment of the Speedup
and Tournament Predictors
This section describes an experiment that shows the speedup
and tournament predictors are able to capture optimization
speedup trends and therefore have the potential to be used
for predicting good optimizations to use for iterative compi-
lation. First, we generated a set of 200 randomly generated
 0
 0.2
 0.4
 0.6
 0.8
 1
 1.2
 1.4
 1.6
 20  40  60  80  100  120  140  160  180
Normalized speedup to -Ofast in Open64 Compiler
Applied optimization sequences sorted by increasing actual speedup
Actual Speedup
Speedup Predictor
Tournament Predictor
Figure 2: Speedup Predictor and Tournament Pre-
dictor for ATAX
optimization sequences constructed from 63 optimizations
available in the Open64 compiler suite. We evaluated these
200 optimization sequences on a set of 74 kernel benchmarks
and obtained speedups relative to the most aggressive opti-
mization level available in Open64 ( -Ofast). We then per-
formed leave-one-out cross validation to construct speedup
and tournament predictors, leaving out the benchmark ATAX
(one of the PolyBench programs [24]) for testing, and train-
ing our models with the remaining kernels. A linear regres-
sion technique and SVMs were used to build our models.
Our training data consists of performance counter character-
ization of our kernels and the optimization sequences from
all but one (left out) kernel. We discuss more about these
two algorithms and the training data in Section 3.
Figure 2 shows how predicted speedups with the pro-
posed models compare to actual speedup for the one un-
seen ATAX kernel we used for our test. The y-axis shows
speedup obtained after applying a sequence, and the x-axis
shows the optimization sequences sorted by increasing (ac-
tual) speedup. The solid line represents the actual speedup
and two dotted lines show the speedup estimated by two
prediction models. As can be seen, the speedup and tour-
nament predictors can predict speedups for each sequences
quite accurately. It is worth noting that the tournament
predictor’s best predicted sequence was the optimal point
of our search space, whereas the speedup predictor’s best
predicted sequence was a good, but not the optimal point in
the search space. Using these models, especially the tourna-
ment predictor, to choose good optimization sequences for
an unseen program has the potential to obtain signiﬁcant
improvements when used in iterative compilation.
3. AUTOMATICALLY CONSTRUCTING A
MODEL
This section describes details of our training data, how
we trained our diﬀerent models, and details of the speciﬁc
models themselves. Figure 3 depicts the four steps in con-
structing our speedup prediction model, (a) collecting per-
formance counter data from programs, (b) creating training
data by applying random optimization sequences from our
search space and obtaining speedups from the kernels, (c)
bringing the performance counter and optimization sequence
data together to construct a model, and (d) using the model
on an “unseen” program. The sequence and tournament pre-
dictor are built in a similar way, but with diﬀerent inputs
and outputs.
We used the HPCToolkit [3] to extract the performance
counters to characterize each program being optimized. Us-
ing HPCToolkit, we collected 29 performance counters to
characterize each program. To collect the performance counter
values, we compiled each program using the -O0 optimiza-
tion level of the Open64 compiler. The optimization level
-O0 was selected to minimize the eﬀects of compiler opti-
mizations on our performance counter characterization of
the program. We then sampled our optimization search
space by randomly generating 500 optimization sequences
from 45 selected Open64 optimizations. The complete list of
optimizations that were used for our experiments are shown
in Section 4.2.
We represent each optimization sequence as a bit vectorT.
Optimizations that are turned oﬀ or on are represented by
a 0 or 1, respectively. Optimizations that require a numeric
value are represented by N − 1 bits where N is the number
of possible optimization values we control. For example, for
the ‘prefetch’ optimization in Open64, we use 4 diﬀerent
possible values (0-3), i.e., ‘0 0 0’ for 0, ‘1 0 0’ for 1, ‘0 1 0’
for 2, and ‘0 0 1’ for 3.
We used each of our 500 randomly generated optimiza-
tion sequences to compile and execute the programs. We
then ran each compiled version ﬁve times and used the av-
erage running time to calculate speedup over -Ofast. Cal-
culated speedup is used to train our models either directly,
e.g., with our speedup predictor, or indirectly, e.g., with our
tournament or sequence predictors.
3.1 Prediction Models
There are speciﬁc diﬀerences as to how the training and
test sets are used and represented depending on the type of
prediction model being developed. In this section, we de-
scribe the three diﬀerent modeling techniques that we eval-
uated.
3.1.1 Sequence Predictor
With this predictor (see Figure 1a), we obtain a bit vector
corresponding to the number of optimizations we are trying
to control. We construct a separate predictor for each op-
timization. The input to each model is the performance
counter characterization of a program, and the output of
each model predicts a probability that a particular optimiza-
tion (associated with that output) would beneﬁt the pro-
gram. Optimization sequences can be generated from this
model by sampling at the mode of the distribution of each
model’s output. The training data for this model consists
of one optimization sequence per benchmark corresponding
the optimization that achieves the best speedup for that
benchmark. Once the predictor is trained, we can use it to
predict good sequences to apply to an “unseen” program by
feeding as input the performance counter characterization of
that “unseen” program to the model. The model then out-
puts a probability pi for each bit predicting whether the bit
should be 1 or 0. For optimizations taking binary values, we
simply decide whether we turn on or oﬀ a given optimiza-
tion. For optimizations with more than two possibilities,
we use the bit position with the highest probability among
all associated bits for a speciﬁc optimization. Thus, all the
outputs form a probability distribution which we can then
Open64	  
Compiler	  
Performance Counters  
for N-1 programs 
… 
… … 
Underlying	  Architecture	  
… 
… 
N-1 Programs 
-O0 
(a) Collecting Perf. Cntrs.
… 
N-1 Programs 
Open64	  
Compiler	  
…	  
…	  
Baseline (-Ofast) 
…	  
…	  
…	  
…	  
Optimization sequences 
and their speedup over 
baseline 
Optimization 
sequences 
Speedup 
over 
baseline 
Optimization Sequence 
Search	  
Space	  
 (b) Obtaining Speedup over the Baseline with Opt. Seqs.
…	  
…	  
…	  
…	  
…	  
…	  
Performance Counters  
for N-1 programs 
… 
… … 
Linear	  Regression	  /	  Support	  Vector	  Machine	  (SVM)	  
… 
 … 
Optimization sequences and their speedup 
over baseline for N-1 programs 
Generated model  
for a given machine 
(c) Building a Prediction Model
… 
 … 
Open64	  
Compiler	  
Performance  
Counters  
… 
…	  
…	  
Nth Program 
(one left out) 
Optimization Sequences 
Predicted speedup 
for each sequences 
Generated model  
for a given machine (d) Using a Prediction Model
Figure 3: This ﬁgure shows how we build and use a speedup prediction model in a leave-one-out cross-
validation scenario. Figure 3(a) depicts the collecting of performance counters for N − 1 programs. Figure
3(b) shows how we collect optimization training data from N − 1 programs. Figure 3(c) depicts use of
performance counter and optimization sequence data to construct a prediction model. Figure 3(d) shows the
use of the trained prediction model.
sample from to generate an optimization sequence to apply.
This model can be used to generate multiple optimization
sequences by sampling as many times as we wish.
3.1.2 Speedup Predictor
For this predictor (see Figure 1b), the model is trained to
accurately predict the speedup of an optimization sequence
applied to a program relative to the program optimized using
-Ofast. The input to the model is the performance counter
characterization of a program and a bit vector describing
the optimization sequence. The output from the model is
the predicted speedup of that optimization sequence relative
to -Ofast. The training data for this model consists of all
500 optimization sequences applied to each kernel, and the
speedups obtained for each optimization sequence relative to
-Ofast. We use this predictor to predict good optimization
sequences for “unseen” program, by evaluating sequences in
order based on their predicted speedup. Another way of
evaluating the quality of this predictor is to use it to search
for good optimization sequences that were previously “un-
seen”. In other words, we can use this predictor to predict
the speedup of sequences that were not previously used to
train our model. Our evaluation demonstrates that this pre-
dictor is eﬀective at ﬁnding good sequences for both“unseen”
programs and “unseen” sequences.
3.1.3 Tournament Predictor
This model (see Figure 1c) is trained to predict, which of
two diﬀerent optimization sequences presented to the model
is better. The inputs supplied to this model is the perfor-
mance counter characterization of a program and two opti-
mization sequences that could be applied to the program.
The output of the model is either true or false depending on
whether the model predicts the ﬁrst optimization sequence
is better or worse than the second sequence. Since it pre-
dicts speedup diﬀerence of two optimization sequences, this
model not only predicts which optimization sequence is bet-
ter, but it also predicts by how much better the optimization
sequence is. One can view this as learning a relation over
triples (PC, Oi, Oj ), where PC is the characterization of the
program being optimized and O is the set of optimization
sequences from which the selection is to be made. Those
triples that belong to the relation deﬁne pairwise prefer-
ences in which the ﬁrst optimization sequence is considered
preferable to the second. Each triple that does not belong
to the relation represents a pair in which the ﬁrst optimiza-
tion is not better than the second. The process of selecting
the best of the optimization alternatives is like ﬁnding the
maximum of a list of numbers. We keep track of the current
best optimization sequence, and proceed with pairwise com-
parisons, always keeping the better of two sequences being
compared. This model can be used to generate sequences
to try for a new program, by sorting diﬀerent optimization
sequences based on the results of the pairwise comparisons.
3.2 Learning Algorithms
A variety of learning algorithms can be brought to bear
on the task of predicting the right set of optimizations to
apply to a program. We consider two methods here. The
ﬁrst method is support vector machines (SVMs) which is a
class of supervised learning algorithms that can be used for
both classiﬁcation and regression. SVMs use kernel func-
tions to transform the training data into a diﬀerent, linearly
separable feature space, and then a linear classiﬁer is con-
structed that separates the points into multiple classes. The
second method we used was linear regression. This is the
standard linear regression technique found in many statisti-
cal and machine learning text books.
Category of PCs List of PCs selected
Branch Related BR-CN, BR-INS, BR-MSP,
BR-NTK, BR-TKN
Cache Line Access CA-SHR
Level 1 Cache L1-DCA, L1-DCM, L1-ICA,
L1-LDM, L1-STM, L1-TCM
Level 2 Cache L2-DCR, L2-DCW, L2-ICA, L2-STM,
L2-TCA, L2-TCM, L2-TCW
Floating Point FDV-INS, FML-INS, FP-INS, FP-OPS
Interrupt/Stall RES-STL
TLB TLB-DM
Total Cycle or Insts. TOT-CYC, TOT-IIS, TOT-INS
Vector/SIMD VEC-INS
Table 1: Performance Counters: We collected 29
diﬀerent performance counters available from PAPI
to characterize a program.
4. PROGRAM CHARACTERISTICS AND OP-
TIMIZATIONS
This section describes how we characterize programs. Sec-
tion 4.1 describes the performance counters we collected to
characterize programs, and Section 4.2 describes the opti-
mization space we selected to construct our sequences from
our testbed compiler.
4.1 Performance Counter Characterization
Each of our models predicts optimizations to apply to“un-
seen” programs that were not used in training the model. To
do this, we need to feed as input to our models a character-
ization of the “unseen” program. We use performance coun-
ters to collect dynamic features that describe the runtime
behavior of a program. Models using performance counter
characteristics of programs have been shown to out-perform
models that use only static code features of program [7]. We
used 29 diﬀerent performance counters shown in Table 1.
4.2 Optimization Space
We selected 7 optimization phases from the Open64 com-
piler and 45 individual optimizations from these selected
phases. These 45 optimizations make up the optimization
space we explored with our models. Most optimizations
came from global and loop nest optimization phases because
these optimizations have the most potential to obtain signif-
icant running time improvements. Most selected optimiza-
Optimization List of
Phase Optimizations
LNO
blocking-size, cs1, cs2, ﬁssion, full-unroll,
fusion, interchange, ou-prod-max, pf2, prefetch,
prefetch-ahead, simd, trip-count
WOPT
aggcm-threshold, aggstr, value-numbering,
combine, dce-aggressive, iv-elimination,
spre, canon-expr
OPT alias, align-padding, div-split, goto, ptr-opt,
swp, unroll-size, unroll-times-max
CG cﬂow, local-sched-alg,
ptr-load-use, use-prefetchnta
GRA optimize-boundary, prioritize-by-density
TENV frame-pointer
IPA callee-limit, ctype, dve, ﬁeld-reorder, space,
plimit,pu-reorder, small-pu, min-hotness
Table 2: List of 45 Optimizations: We selected
45 optimizations from 7 optimization phases in the
Open64 compiler.
tions are either turned on or oﬀ, so they could easily be rep-
resented as binary values. However, certain optimizations
require values in a given range, e.g., loop unrolling factor or
loop tile size. All optimizations used in our paper are shown
in Table 2.
From the listed optimizations in Table 2, we generated a
set of 500 random optimization sequences. We then eval-
uated each sequence on the programs in our training set
to achieve actual speedup of that sequence over -Ofast.
The random optimization sequence and its corresponding
speedup are used as training data.
5. EXPERIMENTAL SETUP
This section describes the experimental setup including
the machine conﬁguration and software platform used. We
also describe the benchmarks used for our experimental study.
We performed our experiments on a pair of Intel Quad CPU
Q9650 machines, each with 2.0GHz processors with 8GB of
memory, running Ubuntu Linux release 8.04. We collected
performance counters using HPCToolkit [3] and the PAPI
3.6 hardware counter library [23]. Table 1 gives a brief de-
scription of the performance counters we used for this study.
We used the open-source Open64 compiler version 4.2.1 [2],
and all speedups reported are relative to -Ofast, which is
the most aggressive optimization level available in this com-
piler. For our machine learning algorithms, we used linear
regression and support vector machines (SVMs) contained
in Weka [1] 3.6.2. For the SVMs, we used a normalized poly-
nomial kernel, and for linear regression we used the default
Weka conﬁguration.
5.1 Benchmarks
We decided to ﬁrst experiment with small pieces of code
that would allow us to quickly evaluate diﬀerent learning
algorithms and modeling techniques. We collected a large
set of functions and kernels from various well-known bench-
marks suites (e.g., UTDSP, NAS, Linpack, and Polybench)
for our study. However, we noticed that several of these ker-
nels had large variability for diﬀerent runs of the same op-
timized code. In order to reduce this variability, we ﬂushed
the cache before every run to reduce possible cache interfer-
ence. We then removed all kernels that had a variability of
more than a few percent after 50 runs of the code. For the
Regression
Evaluations 1 5 10 20 30 40 50
Sequence 1.15×(59%) 1.53×(78%) 1.61×(82%) 1.70×(87%) 1.72×(88%) 1.74×(89%) 1.76×(90%)
Speedup 1.27×(65%) 1.60×(82%) 1.69×(86%) 1.80×(92%) 1.84×(94%) 1.88×(96%) 1.88×(96%)
Tournament 1.22×(62%) 1.55×(79%) 1.75×(89%) 1.80×(92%) 1.86×(95%) 1.88×(96%) 1.89×(96%)
SVM
Sequence 1.18×(60%) 1.53×(78%) 1.63×(82%) 1.81×(92%) 1.84×(94%) 1.85×(94%) 1.86×(95%)
Speedup 1.23×(63%) 1.52×(78%) 1.68×(86%) 1.87×(95%) 1.88×(96%) 1.89×(96%) 1.89×(96%)
Tournament 1.37×(70%) 1.67×(85%) 1.74×(89%) 1.86×(95%) 1.89×(96%) 1.89×(96%) 1.90×(97%)
Table 3: We show the speedup over baseline for our benchmark suite using the top 1, 5, 10, 20, 30, 40, and
50 predicted optimization sequences from our diﬀerent models. Each speedup is followed by a percentage
value (in parentheses). This value shows the percentage of search space optimal speedup we achieved for a
given number of predicted sequences. We show results for our models trained using linear regression and
using support vector machines (SVMs).
remaining code, we used the standard leave-one-out cross-
validation procedure to evaluate our models. That is, the
models were trained using N −1 benchmarks and tested on
the Nth benchmark that was left out.
For another set of experiments, we used a subset of the
MiBench benchmarks that we could both compile success-
fully with Open64 and which did not have high variability in
the runs. We evaluated our tournament predictor by train-
ing our model on kernel programs, and testing the model on
the “unseen” subset of MiBench benchmarks. We also eval-
uated our model on both the “unseen” MiBench benchmarks
and “unseen” optimization sequences. This experiment al-
lowed us to test whether our tournament predictor could be
trained using small kernels, and later used for optimizing
larger applications. In addition, this experiment allowed us
to test whether the model could generalize to optimization
sequences that it had not been trained on.
6. EXPERIMENTAL RESULTS
This section describes our experiment results that we con-
ducted to evaluate our diﬀerent modeling techniques.
6.1 Leave-One-Out Cross Validation on Ker-
nels
To evaluate our modeling techniques, we used leave-one-
out cross-validation over a set of kernel benchmarks. This
allowed us to evaluate the speedups obtained between our
diﬀerent modeling techniques when used to optimize “un-
seen” kernels.
Table 3 shows speedups obtained relative to -Ofast for
each of our models for 1 to 50 “evaluations.” In parenthe-
sis, we show the speedup achieved relative to the maximum
speedup found for the 500 random optimization sequences
evaluated. We show results for training our models using
linear regression and SVMs. The row labeled “Evaluations”
refers to the number of predicted sequences we evaluated
from our model. For example, 1 evaluation (ﬁrst column)
corresponds to taking only the top predicted optimization
sequence from our models, i.e., using the models in a non-
iterative fashion. The rest of the columns corresponds to us-
ing the models in an iterative fashion, where N evaluations
refers to keeping the best actual speedup obtained from a
model’s top N predicted optimization sequences. As the ta-
ble shows, all our models achieve signiﬁcant speedups over
-Ofast.
The speedup predictor trained using regression performed
fairly well, out-performing the tournament predictor and se-
quence predictor for 1 evaluation. Interestingly, the tourna-
ment predictor starts to catch up after 5 evaluations, and
outperforms all predictors at 10 evaluations. After 10 eval-
uations, the speedup and tournament predictors perform
about even, with the sequence predictor lagging behind.
On the other hand, we achieve better performance from
the tournament predictor when trained using SVMs versus
regression. 1 SVMs can construct non-linear models, which
obviously beneﬁts the tournament predictors. Constructing
a tournament predictor using SVMs allows it to perform bet-
ter than any other models at 1 evaluation. As we increase
the number of evaluations to 10, all three modeling tech-
niques show good speedup, with the tournament predictor
still outperforming the other predictors. At 20 evaluations
and greater, both the speedup and tournament predictors
perform equally well. Thus, regardless of learning algorithm,
the tournament predictor performed best at 10 evaluations
by achieving 75% over -Ofast, achieving almost 90% of the
speedup available from the search space. Also, for 1 eval-
uation, the sequence predictor performs signiﬁcantly worse
than the other models.
Table 4 shows detailed results for each of our models
trained using regression and SVMs for 10 evaluations. We
notice that most kernels beneﬁt from using predicted op-
timization sequences for all predictors. Some benchmarks,
such as Livermore Loop11 (Livermr-Lp11) or bicg, obtain
large beneﬁts from using the sequence predictor over the
other two predictors. However, the tournament and speedup
predictor perform better than sequence predictor on most
kernels. On average, the tournament predictor outperforms
the other two predictors achieving 74% of performance im-
provement over -Ofast. At 10 evaluations, both machine
learning techniques performed similarly on all predictors,
and all predictors achieved more than 80% of the maximum
available for the search space explored.
In summary, the tournament predictor trained using SVMs
works consistently well in an iterative ( > 1 evaluation) and
non-iterative fashion (1 evaluation). Thus, for the remain-
der of the paper we further explore the tournament predictor
trained using SVMs.
6.2 Evaluations on MiBench
This section describes results using our tournament pre-
dictor trained on kernels and tested on MiBench bench-
marks. We performed two diﬀerent kinds of experiments.
First, we tested how well our predictor found good opti-
mization sequences to apply to MiBench benchmarks from
1Note that the same training data was used with regression
and SVMs.
Regression SVM
Benchmark Sequence Speedup Tournament Sequence Speedup Tournament Opt
ﬁr.arrays 1.27×(60%) 2.11×(100%) 2.11×(100%) 1.58×(74%) 1.72×(81%) 2.11×(100%) 2.11×
ﬁr.arrays-SWP 1.83×(59%) 2.76 ×(90%) 2.76 ×(90%) 2.01×(65%) 2.49 ×(81%) 2.76 ×(90%) 3.07×
ﬁr.ptrs 1.84×(64%) 1.72×(60%) 2.83×(99%) 1.45×(51%) 2.83×(100%) 2.83×(99%) 2.83×
ﬁr.ptrs-SWP 1.34×(57%) 1.95 ×(83%) 1.91 ×(82%) 2.49×(107%) 2.32 ×(100%) 1.45 ×(62%) 2.32×
latnrm.arrays 1.64×(47%) 3.47×(100%) 3.47×(99%) 1.91×(55%) 3.47×(100%) 2.75×(79%) 3.47×
latnrm.arrays-SWP 1.98×(70%) 2.80 ×(100%) 2.76 ×(98%) 2.24×(80%) 2.76 ×(98%) 2.76 ×(98%) 2.80×
latnrm.ptrs 2.59×(87%) 1.49×(49%) 2.98×(100%) 2.47×(82%) 2.98×(100%) 2.98×(100%) 2.98×
latnrm.ptrs-SWP 2.35×(111%) 2.11 ×(100%) 1.84 ×(87%) 2.41×(114%) 2.11 ×(100%) 1.84 ×(87%) 2.11×
lmsﬁr.arrays 2.80×(87%) 1.57×(49%) 1.37×(42%) 1.45×(45%) 1.76×(54%) 3.21×(100%) 3.21×
lmsﬁr.arrays-SWP 1.16×(50%) 1.45 ×(62%) 2.12 ×(91%) 2.23×(96%) 1.68 ×(72%) 2.12 ×(91%) 2.31×
lmsﬁr.ptrs 1.87×(91%) 1.61×(78%) 2.04×(100%) 1.91×(93%) 1.70×(83%) 1.61×(78%) 2.04×
lmsﬁr.ptrs-SWP 1.70×(55%) 3.05 ×(100%) 1.66 ×(54%) 1.00×(32%) 3.05 ×(100%) 1.76 ×(57%) 3.05×
mult.arrays 0.99×(53%) 1.53×(82%) 1.71×(92%) 1.76×(94%) 1.71×(92%) 1.71×(92%) 1.86×
mult.arrays-SWP 1.07×(41%) 2.54 ×(100%) 1.98 ×(78%) 0.91×(36%) 1.98 ×(78%) 1.98 ×(78%) 2.54×
mult.ptrs 1.04×(35%) 2.96×(100%) 2.63×(88%) 2.81×(95%) 1.87×(63%) 2.63×(88%) 2.96×
mult.ptrs-SWP 1.27×(37%) 3.35 ×(100%) 3.35 ×(100%) 1.18×(35%) 1.88 ×(56%) 3.01 ×(89%) 3.35×
appsp 1.11×(90%) 1.18×(95%) 1.18×(95%) 1.06×(85%) 1.04×(83%) 1.18×(95%) 1.24×
clinpack 1.00×(73%) 1.31 ×(96%) 1.37 ×(100%) 0.97×(70%) 1.22 ×(89%) 1.33 ×(97%) 1.37×
Livermr-Lp2 1.01×(83%) 1.08×(89%) 1.12×(92%) 1.06×(87%) 1.08×(89%) 1.12×(92%) 1.21×
Livermr-Lp3 2.36×(76%) 2.17 ×(69%) 2.17 ×(69%) 2.71×(87%) 2.17 ×(69%) 3.11 ×(100%) 3.11×
Livermr-Lp4 1.69×(153%) 0.81×(73%) 0.92×(83%) 0.93×(84%) 1.04×(94%) 1.04×(94%) 1.10×
Livermr-Lp5 1.24×(83%) 1.33 ×(89%) 1.42 ×(96%) 1.39×(94%) 1.33 ×(89%) 1.36 ×(92%) 1.48×
Livermr-Lp10 2.71×(144%) 1.48×(78%) 1.48×(78%) 3.61×(192%) 1.76×(93%) 1.48×(78%) 1.88×
Livermr-Lp11 2.53×(118%) 1.63 ×(76%) 2.14 ×(100%) 2.41×(112%) 1.60 ×(74%) 2.14 ×(100%) 2.14×
Livermr-Lp12 2.11×(89%) 1.40×(59%) 2.37×(99%) 2.49×(105%) 2.37×(100%) 1.40×(59%) 2.37×
Livermr-Lp13 1.86×(72%) 2.56 ×(100%) 2.42 ×(94%) 1.85×(72%) 2.54 ×(98%) 2.54 ×(99%) 2.56×
Livermr-Lp14 2.71×(123%) 2.09×(94%) 2.09×(94%) 1.43×(65%) 1.55×(70%) 2.09×(94%) 2.20×
Livermr-Lp15 2.37×(112%) 2.10 ×(99%) 2.10 ×(99%) 2.27×(107%) 2.10 ×(99%) 2.10 ×(100%) 2.10×
Livermr-Lp16 2.15×(120%) 1.79×(100%) 1.73×(96%) 3.62×(202%) 1.79×(100%) 1.73×(96%) 1.79×
Livermr-Lp17 1.45×(52%) 2.63 ×(94%) 2.63 ×(94%) 1.09×(39%) 2.65 ×(95%) 2.77 ×(99%) 2.77×
Livermr-Lp18 1.98×(74%) 1.46×(55%) 2.64×(100%) 2.13×(80%) 2.64×(100%) 2.64×(100%) 2.64×
Livermr-Lp19 2.33×(96%) 1.61 ×(66%) 1.61 ×(66%) 2.50×(103%) 2.36 ×(97%) 2.06 ×(84%) 2.42×
Livermr-Lp20 2.11×(89%) 2.27×(96%) 2.27×(96%) 1.80×(76%) 1.48×(62%) 1.93×(81%) 2.37×
Livermr-Lp21 1.81×(72%) 2.49 ×(100%) 2.49 ×(100%) 1.67×(66%) 1.76 ×(70%) 2.49 ×(100%) 2.49×
Livermr-Lp22 3.05×(106%) 2.28×(79%) 2.28×(79%) 2.87×(100%) 1.96×(68%) 2.53×(88%) 2.85×
Livermr-Lp23 2.54×(99%) 2.54 ×(100%) 2.54 ×(100%) 2.00×(78%) 1.50 ×(58%) 2.16 ×(84%) 2.54×
Livermr-Lp24 1.21×(47%) 1.60×(62%) 1.60×(62%) 2.14×(83%) 1.58×(61%) 1.60×(62%) 2.56×
adi 1.14×(102%) 1.04 ×(93%) 1.11 ×(99%) 0.60×(54%) 1.03 ×(92%) 1.11 ×(99%) 1.11×
atax 1.33×(73%) 1.65×(91%) 1.60×(87%) 1.36×(74%) 1.82×(100%) 1.58×(87%) 1.82×
bicg 2.55×(129%) 1.94 ×(98%) 1.85 ×(94%) 2.27×(115%) 1.94 ×(98%) 1.85 ×(94%) 1.97×
cholesky 1.10×(103%) 1.06×(99%) 1.03×(97%) 1.02×(95%) 1.05×(98%) 1.06×(99%) 1.06×
correlation 0.58×(88%) 0.62 ×(94%) 0.65 ×(98%) 0.59×(90%) 0.59 ×(89%) 0.63 ×(94%) 0.66×
covariance 0.62×(102%) 0.57×(95%) 0.60×(98%) 0.61×(100%) 0.60×(98%) 0.59×(97%) 0.60×
doitgen 0.47×(88%) 0.52 ×(98%) 0.50 ×(94%) 0.42×(80%) 0.51 ×(96%) 0.50 ×(94%) 0.53×
durbin 1.06×(89%) 1.14×(96%) 1.14×(96%) 1.07×(90%) 1.16×(98%) 1.14×(96%) 1.18×
dynprog 1.19×(102%) 1.00 ×(85%) 0.98 ×(83%) 1.06×(90%) 1.11 ×(95%) 1.08 ×(93%) 1.17×
fdtd-2d 0.93×(77%) 1.18×(98%) 1.14×(94%) 0.96×(80%) 1.13×(94%) 1.14×(94%) 1.20×
fdtd-apml 1.19×(92%) 1.17 ×(90%) 1.18 ×(91%) 1.17×(90%) 1.18 ×(91%) 1.16 ×(90%) 1.28×
gauss-ﬁlter 0.86×(78%) 0.94×(84%) 0.97×(88%) 0.95×(86%) 0.82×(74%) 0.94×(84%) 1.10×
gemm 0.97×(85%) 1.06 ×(92%) 1.06 ×(92%) 0.93×(81%) 1.00 ×(87%) 1.06 ×(92%) 1.15×
gemver 0.94×(96%) 0.93×(95%) 0.94×(96%) 0.88×(89%) 0.91×(93%) 0.93×(95%) 0.97×
gesummv 1.84×(93%) 1.62 ×(82%) 1.64 ×(83%) 1.95×(98%) 1.76 ×(89%) 1.66 ×(84%) 1.97×
gramschmidt 3.91×(93%) 4.07×(97%) 4.06×(97%) 3.95×(94%) 4.04×(97%) 4.15×(99%) 4.16×
jacobi-1d-imper 1.77×(90%) 1.82 ×(93%) 1.82 ×(93%) 1.61×(82%) 1.87 ×(95%) 1.80 ×(91%) 1.96×
jacobi-2d-imper 1.02×(93%) 1.02×(93%) 1.05×(96%) 1.01×(93%) 1.08×(99%) 1.05×(96%) 1.09×
lu 2.16×(92%) 1.06 ×(45%) 1.06 ×(45%) 1.04×(44%) 1.05 ×(44%) 1.08 ×(46%) 2.34×
ludcmp 1.02×(86%) 1.18×(100%) 1.18×(100%) 1.03×(87%) 1.14×(97%) 1.16×(98%) 1.18×
mvt 1.13×(72%) 1.41 ×(90%) 1.47 ×(94%) 1.42×(91%) 1.55 ×(99%) 1.39 ×(89%) 1.56×
reg-detect 1.17×(89%) 1.19×(91%) 1.24×(94%) 1.15×(87%) 1.27×(97%) 1.19×(91%) 1.31×
seidel 0.87×(79%) 1.08 ×(99%) 1.08 ×(98%) 0.99×(90%) 1.06 ×(97%) 1.05 ×(95%) 1.09×
symm 1.01×(96%) 1.00×(95%) 1.01×(96%) 0.98×(93%) 1.00×(95%) 0.98×(93%) 1.05×
syr2k 0.99×(78%) 1.11 ×(88%) 1.12 ×(88%) 0.91×(72%) 1.14 ×(90%) 1.09 ×(86%) 1.26×
syrk 1.13×(99%) 1.11×(97%) 1.04×(91%) 1.11×(97%) 1.07×(94%) 1.04×(91%) 1.14×
trisolv 2.40×(104%) 1.91 ×(83%) 1.96 ×(85%) 2.26×(98%) 1.80 ×(78%) 1.91 ×(83%) 2.30×
trmm 1.09×(103%) 1.05×(100%) 1.05×(100%) 0.68×(64%) 1.01×(95%) 1.05×(100%) 1.05×
AVG 1.61×(82%) 1.69 ×(86%) 1.75 ×(89%) 1.63×(83%) 1.68 ×(85%) 1.75 ×(89%) 1.96×
Table 4: We show speedups over baseline (ICC -fast) using the top 10 predictions for the sequence, speedup,
and tournament predictor. We include (in parenthesis) the percent of search space optimal speedup that was
obtained. The last column ‘Opt’ represents search space optimal speedup.
 0.9
 0.95
 1
 1.05
 1.1
 1.15
 1.2
basicmathbitcountqsortsusan-cornerssusan-edgessusan-smoothingcjpegdjpegtypesetdijkstrapatriciaispellsaystringsearchbf-decbf-encrijndael-decrijndael-encshaadpcm-decadpcm-encCRC32FFT-inverseFFTgsm-decgsm-encAVG
Speedup over the Baseline -Ofast
MiBench
seen-SVM
unseen-SVM
Figure 4: This ﬁgure shows the maximum speedup obtained with 10 predictions for our tournament predictor
trained using SVM for a subset of the MiBench benchmarks, which we could compile. Two diﬀerent predictors
were evaluated, one using a set of “seen” optimization sequences were the benchmarks were not seen and
another using on a set of “unseen” sequences and “unseen” benchmarks. The SVM predictors achieved an
average speedup for “seen” and “unseen” sequences of 10% and 6%, respectively.
the same set of optimization sequences we had used in the
kernel training data. That is, a tournament predictor was
trained on kernels and tested on MiBench (“unseen”) bench-
marks, but the optimizations sequences used for training
and testing were the same. Second, we again constructed
a tournament predictor on kernels, but this time we tested
the predictor on both “unseen” benchmarks (MiBench) and
“unseen” sequences.
As is shown in Figure 4, the tournament predictor signif-
icantly out-performed -Ofast on most benchmarks (except
for cjpeg and typeset) for “seen” sequences. For “seen” se-
quences, we achieve on average 10% performance improve-
ment over -Ofast, where the maximum speedup available
found was 15.4%. We also achieved good performance im-
provements with “unseen” sequences by obtaining 6% on av-
erage, where the maximum speedup found was 12%. We
note that performance improvement is better for “seen” se-
quences, however, we can still achieve good results using our
models to predict the performance of “unseen” sequences.
6.3 Optimization Sequences Across Different
Machines
This section looks at the portability of our models across
diﬀerent machines. We looked at three diﬀerent Intel ma-
chines that varied in processor speed, cache, and memory
size. Machine 1 is an Intel Quad Core2 Q9300 2.5GHz with
4GB RAM and 6MB L2 cache, Machine 2 is an Intel Quad
Core2 Q9650 3.0GHz with 8GB RAM and 12MB L2 cache,
and Machine 3 is an Intel Xeon E5335 2.0GHz with 2GB
RAM and 8MB L2 cache. We evaluated 500 diﬀerent op-
timization sequences on a representative benchmark FFT-
inverse from the MiBench suite. Figure 5 shows the opti-
mization sequences sorted by their performance on Machine
1. Note, the optimization sequences for this benchmark have
the same speedup trends on the diﬀerent machines. We con-
clude that sequences performing well on our models trained
on one machine would perform well on other similar archi-
tectures. We saw similar trends for our other programs.
7. RELATED WORK
In recent years, several previous works have shown the
beneﬁts of iterative compilation [5,14,18,19]. Iterative com-
pilation has been shown to regularly outperform the most
aggressive compilation settings of most commercial compil-
ers and has been shown to be comparable to hand-optimized
library functions [15,25,27].
However, iterative compilation techniques can be expen-
sive, and current research has looked into improving the
speed of these techniques. Almagor et al. [5] take a radical
approach to reduce the total number of evaluations. They
examine the structure of the search space, in particular the
distribution of local minima relative to the global minima
and devise new search based algorithms that outperform
generic search techniques. Kulkarni et al. [20] introduced
a system where they tried to use databases to store previ-
ously tested code and thus save on running time. They also
disabled some optimizations that did not seem to improve
the running time of the kernel. These techniques may miss
opportunities because the constrain the search space a-priori
before optimizing a program. Cooper et al. [10] use genetic
algorithms to solve the compilation phase-ordering problem.
They were concerned with ﬁnding“good”compiler optimiza-
tion sequences that reduced code size. Their technique was
successful at reducing code size by as much as 40%. Unfortu-
nately, all these techniques are application-speciﬁc. That is,
these search algorithms (e.g., genetic algorithms) have to be
“retrained” for each program to decide the best optimization
sequence for that program.
 0.6
 0.7
 0.8
 0.9
 1
 1.1
 1.2
 50  100  150  200  250  300  350  400  450  500
Speedup over the Baseline -Ofast in Open64 Compiler
Applied optimization sequences sorted by increasing speedup from Machine 1
Machine 1
Machine 2
Machine 3
Figure 5: This graph represents how each optimization sequence performs over diﬀerent architectures for the
benchmark FFT-inverse. Other benchmarks produce similar graphs. The x-axis shows the sequences sorted by
speedup on Machine 1. The y-axis shows the speedup over baseline, which is -Ofast. Machine 1 is Intel Quad
Core2 Q9300 2.5GHz 4GB RAM and 6MB L2 cache, Machine 2 is Intel Quad Core2 Q9650 3.0GHz 8GB
RAM and 12MB L2 cache, and Machine 3 is Intel Xeon E5335 2.0GHz 2GB RAM and 8MB L2 cache. We
observe the sequence that gives us good performance on one machine usually works well on other machines.
Several researchers have also looked at using machine learn-
ing to construct heuristics that control a single optimization.
Stephenson et al. [26] used genetic programming (GP) to
tune heuristic priority functions for three compiler optimiza-
tions: hyper block selection, register allocation, and data
prefetching within the Trimaran’s IMPACT compiler. How-
ever, a closer look at the results for two of their optimizations
indicate that most of the improvement was obtained from
the initial population indicating that these two pre-existing
heuristics were not well tuned. For the third optimization,
register allocation, they were able to achieve on average only
a 2% increase over the manually tuned heuristic.
Fursin et al. [16] discuss an extension to GCC called Mile-
post GCC, which can provide optimization strategies ac-
cording to an objective function that user seeks to improve.
Milepost GCC uses static features of a program to make
predictions of good optimizations to apply. In this work,
we used dynamic features of a programs, which have been
shown to be better predictors [7] of of good optimization
sequences. In contrast to this work, Fursin et al. [16] do not
evaluate diﬀerent methods of modeling the problem of how
to predict good optimization sequences.
Cavazos et al. [8] used supervised learning to create a pre-
dictor model specialized to decide to enable or disable in-
struction scheduling. This reduced up to 75% of the schedul-
ing eﬀort without losing any performance. Recently, Cava-
zos et al. [9] describe using static code features and super-
vised learning to control several optimizations to apply dur-
ing method compilation in a JIT compiler. Since Java meth-
ods are typically small, static code features were successfully
used to characterizing them.
8. CONCLUSION
In this paper, we introduce and evaluate a novel mod-
eling technique, the tournament predictor, for predicting
good compiler optimizations to apply to an “unseen” pro-
gram. Moreover, we compare this modeling technique to two
state-of-the-art modeling techniques, namely the sequence
and speedup predictor. We build our models by using two
machine learning techniques, regression and SVMs. These
models predict good code optimization sequences to apply
given a program’s performance counter characterization. We
ﬁrst evaluated the prediction models on a large set of kernels
with leave-one-out cross validation. From our experimen-
tal results, speedup and tournament predictors performed
better than sequence predictor. The tournament predictor
out-performed both the sequence and speedup predictor at 1
evaluation and 10 evaluations, and in 10 evaluations achiev-
ing 75% improvement over -Ofast in Open64 compiler. We
trained the tournament predictor on kernels to evaluate a
set of embedded domain applications from MiBench with
“seen” and “unseen” sequences. We achieved good perfor-
mance improvement with 10% and 6% for “seen” and “un-
seen” sequences respectively, which is 65% and 50% of the
maximum speedup found for the 500 random optimization
sequences evaluated.
For future work, we expect to apply additional machine
learning algorithms to construct our prediction models. We
will also extend our testbed to diﬀerent domains of applica-
tions, diﬀerent compilers, and a larger optimization space.
9. REFERENCES
[1] Weka 3: Data mining software in java.
http://www.cs.waikato.ac.nz/ml/weka.
[2] Open64, inc. http://www.open64.net, 2006.
[3] L. Adhianto, S. Banerjee, M. Fagan, M. Krentel,
G. Marin, J. Mellor-Crummey, and N. R. Tallent.
Hpctoolkit: Tools for performance analysis of
optimized parallel programs. Concurrency: Practice
and Experience. To appear., 2010.
[4] F. Agakov, E. Bonilla, J. Cavazos, B. Franke,
G. Fursin, M. O’Boyle, J. Thomson, M. Toussaint,
and C. Williams. Using machine learning to focus
iterative optimization. In Proceedings of the
International Symposium on Code Generation and
Optimization, pages 295–305, 2006.
[5] L. Almagor, K. D. Cooper, A. Grosul, T. J. Harvey,
S. W. Reeves, D. Subramanian, L. Torczon, and
T. Waterman. Finding eﬀective compilation sequences.
In Proceedings of the 2004 ACM SIGPLAN/SIGBED
Conference on Languages, Compilers, and Tools for
Embedded Systems, pages 231–239, New York, NY,
USA, 2004. ACM Press.
[6] J. Cavazos, C. Dubach, F. Agakov, E. Bonilla, M. F.
O’Boyle, G. Fursin, and O. Temam. Automatic
performance model construction for the fast software
exploration of new hardware designs. In Proceedings of
the International Conference on Compilers,
Architecture, And Synthesis For Embedded Systems
(CASES 2006), October 2006.
[7] J. Cavazos, G. Fursin, F. V. Agakov, E. V. Bonilla,
M. F. P. O’Boyle, and O. Temam. Rapidly selecting
good compiler optimizations using performance
counters. In CGO, pages 185–197, 2007.
[8] J. Cavazos and J. E. B. Moss. Inducing heuristics to
decide whether to schedule. In Proceedings of the
ACM SIGPLAN ’04 Conference on Programming
Language Design and Implementation , pages 183–194,
Washington, D.C., June 2004. ACM Press.
[9] J. Cavazos and M. O’Boyle. Method-speciﬁc dynamic
compilation using logistic regression. In Proceedings of
the ACM SIGPLAN ’06 Conference on Object
Oriented Programming, Systems, Languages, and
Applications, Portland, Or., October 2006. ACM
Press.
[10] K. D. Cooper, P. J. Schielke, and D. Subramanian.
Optimizing for reduced code space using genetic
algorithms. In Workshop on Languages, Compilers,
and Tools for Embedded Systems, pages 1–9, Atlanta,
Georgia, July 1999. ACM Press.
[11] F. de Mesmay, Y. Voronenko, and M. P ¨uschel. Oﬄine
library adaptation using automatically generated
heuristics. In International Parallel and Distributed
Processing Symposium (IPDPS), 2010.
[12] C. Dubach, J. Cavazos, B. Franke, M. F. O’Boyle,
G. Fursin, and O. Temam. Fast compiler optimisation
evaluation using code-feature based performance
prediction. In Proceedings of the ACM International
Conference on Computing Frontiers, May 2007.
[13] C. Dubach, T. M. Jones, E. V. Bonilla, G. Fursin, and
M. F. O’Boyle. Portable compiler optimization across
embedded programs and microarchitectures using
machine learning. In Proceedings of the IEEE/ACM
International Symposium on Microarchitecture
(MICRO), December 2009.
[14] B. Franke, M. O’Boyle, J. Thomson, and G. Fursin.
Probabilistic source-level optimisation of embedded
programs. In Proceedings of the 2005 ACM
SIGPLAN/SIGBED Conference on Languages,
Compilers, and Tools for Embedded Systems , pages
78–86, New York, NY, USA, 2005. ACM Press.
[15] M. Frigo and S. G. Johnson. The design and
implementation of FFTW3. Proceedings of the IEEE,
93(2):216–231, 2005. special issue on ”Program
Generation, Optimization, and Platform Adaptation”.
[16] G. Fursin, Y. Kashnikov, A. Memon, Z. Chamski,
O. Temam, M. Namolaru, E. Yom-Tov, B. Mendelson,
A. Zaks, E. Courtois, F. Bodin, P. Barnard,
E. Ashton, E. Bonilla, J. Thomson, C. Williams, and
M. O’Boyle. Milepost gcc: Machine learning enabled
self-tuning compiler. International Journal of Parallel
Programming, 39:296–327, 2011.
10.1007/s10766-010-0161-2.
[17] M. R. Guthaus, J. S. Ringenberg, D. Ernst, T. M.
Austin, T. Mudge, and R. B. Brown. Mibench: A free,
commercially representative embedded benchmark
suite. In IEEE 4th Annual Workshop on Workload
Characterization, Austin, TX, December 2001.
[18] M. Haneda, P. M. W. Knijnenburg, and H. A. G.
Wijshoﬀ. Automatic selection of compiler options
using non-parametric inferential statistics. In
Proceedings of the International Conference on
Parallel Architectures and Compilation Techniques,
pages 123–132, Washington, DC, USA, 2005. IEEE
Computer Society.
[19] T. Kisuki, P. M. W. Knijnenburg, and M. F. P.
O’Boyle. Combined selection of tile sizes and unroll
factors using iterative compilation. In Proceedings of
the International Conference on Parallel Architectures
and Compilation Techniques, page 237, Washington,
DC, USA, 2000. IEEE Computer Society.
[20] P. Kulkarni, S. Hines, J. Hiser, D. Whalley,
J. Davidson, and D. Jones. Fast searches for eﬀective
optimization phase sequences. In Proceedings of the
ACM SIGPLAN ’04 Conference on Programming
Language Design and Implementation , pages 171–182,
New York, NY, USA, 2004. ACM Press.
[21] J. Lau, M. Arnold, M. Hind, and B. Calder. Online
performance auditing: using hot optimizations without
getting burned. SIGPLAN Not., 41(6):239–251, 2006.
[22] A. Monsifrot, F. Bodin, and R. Quiniou. A machine
learning approach to automatic production of compiler
heuristics. In AIMSA ’02: Proceedings of the 10th
International Conference on Artiﬁcial Intelligence:
Methodology, Systems, and Applications , pages 41–50.
Springer-Verlag, 2002.
[23] P. Mucci. Papi – the performance application
programming interface.
http://icl.cs.utk.edu/papi/index.html, 2000.
[24] Polybench.
http://www-roc.inria.fr/ pouchet/software/polybench.
[25] M. Puschel, J. Moura, J. Johnson, D. Padua,
M. Veloso, B. Singer, J. Xiong, F. Franchetti,
A. Gacic, Y. Voronenko, K. Chen, R. W. Johnson, and
N. Rizzolo. Spiral: Code generation for dsp
transforms. Proceedings of the IEEE, 93(2):232–275,
2005. special issue on ”Program Generation,
Optimization, and Platform Adaptation”.
[26] M. Stephenson, S. Amarasinghe, M. Martin, and
U.-M. O’Reilly. Meta optimization: Improving
compiler heuristics with machine learning. In
Proceedings of the ACM SIGPLAN ’03 Conference on
Programming Language Design and Implementation ,
pages 77–90, San Diego, Ca, June 2003. ACM Press.
[27] R. C. Whaley and J. J. Dongarra. Automatically
tuned linear algebra software. In SC ’98: Proceedings
of the 1998 ACM/IEEE conference on
Supercomputing, pages 1–27, Washington, DC, USA,
1998. IEEE Computer Society.

Neural Turing Machines
Alex Graves gravesa@google.com
Greg Wayne gregwayne@google.com
Ivo Danihelka danihelka@google.com
Google DeepMind, London, UK
Abstract
We extend the capabilities of neural networks by coupling them to external memory re-
sources, which they can interact with by attentional processes. The combined system is
analogous to a Turing Machine or V on Neumann architecture but is differentiable end-to-
end, allowing it to be efﬁciently trained with gradient descent. Preliminary results demon-
strate that Neural Turing Machines can infer simple algorithms such as copying, sorting,
and associative recall from input and output examples.
1 Introduction
Computer programs make use of three fundamental mechanisms: elementary operations
(e.g., arithmetic operations), logical ﬂow control (branching), and external memory, which
can be written to and read from in the course of computation (V on Neumann, 1945). De-
spite its wide-ranging success in modelling complicated data, modern machine learning
has largely neglected the use of logical ﬂow control and external memory.
Recurrent neural networks (RNNs) stand out from other machine learning methods
for their ability to learn and carry out complicated transformations of data over extended
periods of time. Moreover, it is known that RNNs are Turing-Complete (Siegelmann and
Sontag, 1995), and therefore have the capacity to simulate arbitrary procedures,if properly
wired. Yet what is possible in principle is not always what is simple in practice. We
therefore enrich the capabilities of standard recurrent networks to simplify the solution of
algorithmic tasks. This enrichment is primarily via a large, addressable memory, so, by
analogy to Turing’s enrichment of ﬁnite-state machines by an inﬁnite memory tape, we
1
arXiv:1410.5401v2  [cs.NE]  10 Dec 2014
dub our device a “Neural Turing Machine” (NTM). Unlike a Turing machine, an NTM
is a differentiable computer that can be trained by gradient descent, yielding a practical
mechanism for learning programs.
In human cognition, the process that shares the most similarity to algorithmic operation
is known as “working memory.” While the mechanisms of working memory remain some-
what obscure at the level of neurophysiology, the verbal deﬁnition is understood to mean
a capacity for short-term storage of information and its rule-based manipulation (Badde-
ley et al., 2009). In computational terms, these rules are simple programs, and the stored
information constitutes the arguments of these programs. Therefore, an NTM resembles
a working memory system, as it is designed to solve tasks that require the application of
approximate rules to “rapidly-created variables.” Rapidly-created variables (Hadley, 2009)
are data that are quickly bound to memory slots, in the same way that the number 3 and the
number 4 are put inside registers in a conventional computer and added to make 7 (Minsky,
1967). An NTM bears another close resemblance to models of working memory since the
NTM architecture uses an attentional process to read from and write to memory selectively.
In contrast to most models of working memory, our architecture can learn to use its working
memory instead of deploying a ﬁxed set of procedures over symbolic data.
The organisation of this report begins with a brief review of germane research on work-
ing memory in psychology, linguistics, and neuroscience, along with related research in
artiﬁcial intelligence and neural networks. We then describe our basic contribution, a mem-
ory architecture and attentional controller that we believe is well-suited to the performance
of tasks that require the induction and execution of simple programs. To test this architec-
ture, we have constructed a battery of problems, and we present their precise descriptions
along with our results. We conclude by summarising the strengths of the architecture.
2 Foundational Research
2.1 Psychology and Neuroscience
The concept of working memory has been most heavily developed in psychology to explain
the performance of tasks involving the short-term manipulation of information. The broad
picture is that a “central executive” focuses attention and performs operations on data in a
memory buffer (Baddeley et al., 2009). Psychologists have extensively studied the capacity
limitations of working memory, which is often quantiﬁed by the number of “chunks” of
information that can be readily recalled (Miller, 1956). 1 These capacity limitations lead
toward an understanding of structural constraints in the human working memory system,
but in our own work we are happy to exceed them.
In neuroscience, the working memory process has been ascribed to the functioning of a
system composed of the prefrontal cortex and basal ganglia (Goldman-Rakic, 1995). Typ-
1There remains vigorous debate about how best to characterise capacity limitations (Barrouillet et al.,
2004).
2
ical experiments involve recording from a single neuron or group of neurons in prefrontal
cortex while a monkey is performing a task that involves observing a transient cue, waiting
through a “delay period,” then responding in a manner dependent on the cue. Certain tasks
elicit persistent ﬁring from individual neurons during the delay period or more complicated
neural dynamics. A recent study quantiﬁed delay period activity in prefrontal cortex for a
complex, context-dependent task based on measures of “dimensionality” of the population
code and showed that it predicted memory performance (Rigotti et al., 2013).
Modeling studies of working memory range from those that consider how biophysical
circuits could implement persistent neuronal ﬁring (Wang, 1999) to those that try to solve
explicit tasks (Hazy et al., 2006) (Dayan, 2008) (Eliasmith, 2013). Of these, Hazy et al.’s
model is the most relevant to our work, as it is itself analogous to the Long Short-Term
Memory architecture, which we have modiﬁed ourselves. As in our architecture, Hazy
et al.’s has mechanisms to gate information into memory slots, which they use to solve a
memory task constructed of nested rules. In contrast to our work, the authors include no
sophisticated notion of memory addressing, which limits the system to storage and recall
of relatively simple, atomic data. Addressing, fundamental to our work, is usually left
out from computational models in neuroscience, though it deserves to be mentioned that
Gallistel and King (Gallistel and King, 2009) and Marcus (Marcus, 2003) have argued that
addressing must be implicated in the operation of the brain.
2.2 Cognitive Science and Linguistics
Historically, cognitive science and linguistics emerged as ﬁelds at roughly the same time
as artiﬁcial intelligence, all deeply inﬂuenced by the advent of the computer (Chomsky,
1956) (Miller, 2003). Their intentions were to explain human mental behaviour based on
information or symbol-processing metaphors. In the early 1980s, both ﬁelds considered
recursive or procedural (rule-based) symbol-processing to be the highest mark of cogni-
tion. The Parallel Distributed Processing (PDP) or connectionist revolution cast aside the
symbol-processing metaphor in favour of a so-called “sub-symbolic” description of thought
processes (Rumelhart et al., 1986).
Fodor and Pylyshyn (Fodor and Pylyshyn, 1988) famously made two barbed claims
about the limitations of neural networks for cognitive modeling. They ﬁrst objected that
connectionist theories were incapable ofvariable-binding, or the assignment of a particular
datum to a particular slot in a data structure. In language, variable-binding is ubiquitous;
for example, when one produces or interprets a sentence of the form, “Mary spoke to John,”
one has assigned “Mary” the role of subject, “John” the role of object, and “spoke to” the
role of the transitive verb. Fodor and Pylyshyn also argued that neural networks with ﬁxed-
length input domains could not reproduce human capabilities in tasks that involve process-
ing variable-length structures. In response to this criticism, neural network researchers
including Hinton (Hinton, 1986), Smolensky (Smolensky, 1990), Touretzky (Touretzky,
1990), Pollack (Pollack, 1990), Plate (Plate, 2003), and Kanerva (Kanerva, 2009) inves-
tigated speciﬁc mechanisms that could support both variable-binding and variable-length
3
structure within a connectionist framework. Our architecture draws on and potentiates this
work.
Recursive processing of variable-length structures continues to be regarded as a hall-
mark of human cognition. In the last decade, a ﬁreﬁght in the linguistics community staked
several leaders of the ﬁeld against one another. At issue was whether recursive processing
is the “uniquely human” evolutionary innovation that enables language and is specialized to
language, a view supported by Fitch, Hauser, and Chomsky (Fitch et al., 2005), or whether
multiple new adaptations are responsible for human language evolution and recursive pro-
cessing predates language (Jackendoff and Pinker, 2005). Regardless of recursive process-
ing’s evolutionary origins, all agreed that it is essential to human cognitive ﬂexibility.
2.3 Recurrent Neural Networks
Recurrent neural networks constitute a broad class of machines with dynamic state; that
is, they have state whose evolution depends both on the input to the system and on the
current state. In comparison to hidden Markov models, which also contain dynamic state,
RNNs have a distributed state and therefore have signiﬁcantly larger and richer memory
and computational capacity. Dynamic state is crucial because it affords the possibility of
context-dependent computation; a signal entering at a given moment can alter the behaviour
of the network at a much later moment.
A crucial innovation to recurrent networks was the Long Short-Term Memory (LSTM)
(Hochreiter and Schmidhuber, 1997). This very general architecture was developed for a
speciﬁc purpose, to address the “vanishing and exploding gradient” problem (Hochreiter
et al., 2001a), which we might relabel the problem of “vanishing and exploding sensitivity.”
LSTM ameliorates the problem by embedding perfect integrators (Seung, 1998) for mem-
ory storage in the network. The simplest example of a perfect integrator is the equation
x(t+ 1) =x(t) +i(t), where i(t) is an input to the system. The implicit identity matrix
Ix(t) means that signals do not dynamically vanish or explode. If we attach a mechanism
to this integrator that allows an enclosing network to choose when the integrator listens to
inputs, namely, a programmable gate depending on context, we have an equation of the
form x(t+ 1) =x(t) +g(context)i(t). We can now selectively store information for an
indeﬁnite length of time.
Recurrent networks readily process variable-length structures without modiﬁcation. In
sequential problems, inputs to the network arrive at different times, allowing variable-
length or composite structures to be processed over multiple steps. Because they natively
handle variable-length structures, they have recently been used in a variety of cognitive
problems, including speech recognition (Graves et al., 2013; Graves and Jaitly, 2014), text
generation (Sutskever et al., 2011), handwriting generation (Graves, 2013) and machine
translation (Sutskever et al., 2014). Considering this property, we do not feel that it is ur-
gent or even necessarily valuable to build explicit parse trees to merge composite structures
greedily (Pollack, 1990) (Socher et al., 2012) (Frasconi et al., 1998).
Other important precursors to our work include differentiable models of attention (Graves,
4
Figure 1: Neural Turing Machine Architecture.During each update cycle, the controller
network receives inputs from an external environment and emits outputs in response. It also
reads to and writes from a memory matrix via a set of parallel read and write heads. The dashed
line indicates the division between the NTM circuit and the outside world.
2013) (Bahdanau et al., 2014) and program search (Hochreiter et al., 2001b) (Das et al.,
1992), constructed with recurrent neural networks.
3 Neural Turing Machines
A Neural Turing Machine (NTM) architecture contains two basic components: a neural
network controller and a memory bank. Figure 1 presents a high-level diagram of the NTM
architecture. Like most neural networks, the controller interacts with the external world via
input and output vectors. Unlike a standard network, it also interacts with a memory matrix
using selective read and write operations. By analogy to the Turing machine we refer to the
network outputs that parametrise these operations as “heads.”
Crucially, every component of the architecture is differentiable, making it straightfor-
ward to train with gradient descent. We achieved this by deﬁning ‘blurry’ read and write
operations that interact to a greater or lesser degree with all the elements in memory (rather
than addressing a single element, as in a normal Turing machine or digital computer). The
degree of blurriness is determined by an attentional “focus” mechanism that constrains each
read and write operation to interact with a small portion of the memory, while ignoring the
rest. Because interaction with the memory is highly sparse, the NTM is biased towards
storing data without interference. The memory location brought into attentional focus is
determined by specialised outputs emitted by the heads. These outputs deﬁne a normalised
weighting over the rows in the memory matrix (referred to as memory “locations”). Each
weighting, one per read or write head, deﬁnes the degree to which the head reads or writes
5
at each location. A head can thereby attend sharply to the memory at a single location or
weakly to the memory at many locations.
3.1 Reading
Let Mt be the contents of the N ×M memory matrix at time t, where N is the number
of memory locations, and M is the vector size at each location. Let wt be a vector of
weightings over the N locations emitted by a read head at time t. Since all weightings are
normalised, the N elements wt(i) of wt obey the following constraints:
∑
i
wt(i) = 1, 0 ≤wt(i) ≤1, ∀i. (1)
The length M read vector rt returned by the head is deﬁned as a convex combination of
the row-vectors Mt(i) in memory:
rt ←−
∑
i
wt(i)Mt(i), (2)
which is clearly differentiable with respect to both the memory and the weighting.
3.2 Writing
Taking inspiration from the input and forget gates in LSTM, we decompose each write into
two parts: an erase followed by an add.
Given a weighting wt emitted by a write head at time t, along with an erase vector
et whose M elements all lie in the range (0,1), the memory vectors Mt−1(i) from the
previous time-step are modiﬁed as follows:
˜Mt(i) ←−Mt−1(i) [1 −wt(i)et] , (3)
where 1 is a row-vector of all 1-s, and the multiplication against the memory location acts
point-wise. Therefore, the elements of a memory location are reset to zero only if both the
weighting at the location and the erase element are one; if either the weighting or the erase
is zero, the memory is left unchanged. When multiple write heads are present, the erasures
can be performed in any order, as multiplication is commutative.
Each write head also produces a lengthM add vector at, which is added to the memory
after the erase step has been performed:
Mt(i) ←−˜Mt(i) +wt(i) at. (4)
Once again, the order in which the adds are performed by multiple heads is irrelevant. The
combined erase and add operations of all the write heads produces the ﬁnal content of the
memory at time t. Since both erase and add are differentiable, the composite write oper-
ation is differentiable too. Note that both the erase and add vectors have M independent
components, allowing ﬁne-grained control over which elements in each memory location
are modiﬁed.
6
Figure 2: Flow Diagram of the Addressing Mechanism. The key vector, kt, and key
strength, βt, are used to perform content-based addressing of the memory matrix, Mt. The
resulting content-based weighting is interpolated with the weighting from the previous time step
based on the value of the interpolation gate, gt. The shift weighting, st, determines whether
and by how much the weighting is rotated. Finally, depending on γt, the weighting is sharpened
and used for memory access.
3.3 Addressing Mechanisms
Although we have now shown the equations of reading and writing, we have not described
how the weightings are produced. These weightings arise by combining two addressing
mechanisms with complementary facilities. The ﬁrst mechanism, “content-based address-
ing,” focuses attention on locations based on the similarity between their current values
and values emitted by the controller. This is related to the content-addressing of Hopﬁeld
networks (Hopﬁeld, 1982). The advantage of content-based addressing is that retrieval is
simple, merely requiring the controller to produce an approximation to a part of the stored
data, which is then compared to memory to yield the exact stored value.
However, not all problems are well-suited to content-based addressing. In certain tasks
the content of a variable is arbitrary, but the variable still needs a recognisable name or ad-
dress. Arithmetic problems fall into this category: the variablexand the variableycan take
on any two values, but the procedure f(x,y) =x×yshould still be deﬁned. A controller
for this task could take the values of the variablesxand y, store them in different addresses,
then retrieve them and perform a multiplication algorithm. In this case, the variables are
addressed by location, not by content. We call this form of addressing “location-based ad-
dressing.” Content-based addressing is strictly more general than location-based addressing
as the content of a memory location could include location information inside it. In our ex-
periments however, providing location-based addressing as a primitive operation proved
essential for some forms of generalisation, so we employ both mechanisms concurrently.
Figure 2 presents a ﬂow diagram of the entire addressing system that shows the order
of operations for constructing a weighting vector when reading or writing.
7
3.3.1 Focusing by Content
For content-addressing, each head (whether employed for reading or writing) ﬁrst produces
a length M key vector kt that is compared to each vector Mt(i) by a similarity measure
K
[
·,·
]
. The content-based system produces a normalised weighting wc
t based on the sim-
ilarity and a positive key strength, βt, which can amplify or attenuate the precision of the
focus:
wc
t(i) ←−
exp
(
βtK
[
kt,Mt(i)
])
∑
j exp
(
βtK
[
kt,Mt(j)
]). (5)
In our current implementation, the similarity measure is cosine similarity:
K
[
u,v
]
= u ·v
||u||·||v||. (6)
3.3.2 Focusing by Location
The location-based addressing mechanism is designed to facilitate both simple iteration
across the locations of the memory and random-access jumps. It does so by implementing
a rotational shift of a weighting. For example, if the current weighting focuses entirely on
a single location, a rotation of 1 would shift the focus to the next location. A negative shift
would move the weighting in the opposite direction.
Prior to rotation, each head emits a scalar interpolation gate gt in the range (0,1). The
value ofgis used to blend between the weightingwt−1 produced by the head at the previous
time-step and the weighting wc
t produced by the content system at the current time-step,
yielding the gated weighting wg
t:
wg
t ←−gtwc
t + (1−gt)wt−1. (7)
If the gate is zero, then the content weighting is entirely ignored, and the weighting from the
previous time step is used. Conversely, if the gate is one, the weighting from the previous
iteration is ignored, and the system applies content-based addressing.
After interpolation, each head emits ashift weighting st that deﬁnes a normalised distri-
bution over the allowed integer shifts. For example, if shifts between -1 and 1 are allowed,
st has three elements corresponding to the degree to which shifts of -1, 0 and 1 are per-
formed. The simplest way to deﬁne the shift weightings is to use a softmax layer of the
appropriate size attached to the controller. We also experimented with another technique,
where the controller emits a single scalar that is interpreted as the lower bound of a width
one uniform distribution over shifts. For example, if the shift scalar is 6.7, thenst(6) = 0.3,
st(7) = 0.7, and the rest of st is zero.
8
If we index the N memory locations from 0 to N −1, the rotation applied to wg
t by st
can be expressed as the following circular convolution:
˜wt(i) ←−
N−1∑
j=0
wg
t(j) st(i−j) (8)
where all index arithmetic is computed modulo N. The convolution operation in Equa-
tion (8) can cause leakage or dispersion of weightings over time if the shift weighting is
not sharp. For example, if shifts of -1, 0 and 1 are given weights of 0.1, 0.8 and 0.1, the
rotation will transform a weighting focused at a single point into one slightly blurred over
three points. To combat this, each head emits one further scalar γt ≥1 whose effect is to
sharpen the ﬁnal weighting as follows:
wt(i) ←− ˜wt(i)γt
∑
j ˜wt(j)γt
(9)
The combined addressing system of weighting interpolation and content and location-
based addressing can operate in three complementary modes. One, a weighting can be
chosen by the content system without any modiﬁcation by the location system. Two, a
weighting produced by the content addressing system can be chosen and then shifted. This
allows the focus to jump to a location next to, but not on, an address accessed by content;
in computational terms this allows a head to ﬁnd a contiguous block of data, then access a
particular element within that block. Three, a weighting from the previous time step can
be rotated without any input from the content-based addressing system. This allows the
weighting to iterate through a sequence of addresses by advancing the same distance at
each time-step.
3.4 Controller Network
The NTM architecture architecture described above has several free parameters, including
the size of the memory, the number of read and write heads, and the range of allowed lo-
cation shifts. But perhaps the most signiﬁcant architectural choice is the type of neural
network used as the controller. In particular, one has to decide whether to use a recurrent
or feedforward network. A recurrent controller such as LSTM has its own internal memory
that can complement the larger memory in the matrix. If one compares the controller to
the central processing unit in a digital computer (albeit with adaptive rather than predeﬁned
instructions) and the memory matrix to RAM, then the hidden activations of the recurrent
controller are akin to the registers in the processor. They allow the controller to mix infor-
mation across multiple time steps of operation. On the other hand a feedforward controller
can mimic a recurrent network by reading and writing at the same location in memory at
every step. Furthermore, feedforward controllers often confer greater transparency to the
network’s operation because the pattern of reading from and writing to the memory matrix
is usually easier to interpret than the internal state of an RNN. However, one limitation of
9
a feedforward controller is that the number of concurrent read and write heads imposes a
bottleneck on the type of computation the NTM can perform. With a single read head, it
can perform only a unary transform on a single memory vector at each time-step, with two
read heads it can perform binary vector transforms, and so on. Recurrent controllers can
internally store read vectors from previous time-steps, so do not suffer from this limitation.
4 Experiments
This section presents preliminary experiments on a set of simple algorithmic tasks such
as copying and sorting data sequences. The goal was not only to establish that NTM is
able to solve the problems, but also that it is able to do so by learning compact internal
programs. The hallmark of such solutions is that they generalise well beyond the range of
the training data. For example, we were curious to see if a network that had been trained
to copy sequences of length up to 20 could copy a sequence of length 100 with no further
training.
For all the experiments we compared three architectures: NTM with a feedforward
controller, NTM with an LSTM controller, and a standard LSTM network. Because all
the tasks were episodic, we reset the dynamic state of the networks at the start of each
input sequence. For the LSTM networks, this meant setting the previous hidden state equal
to a learned bias vector. For NTM the previous state of the controller, the value of the
previous read vectors, and the contents of the memory were all reset to bias values. All
the tasks were supervised learning problems with binary targets; all networks had logistic
sigmoid output layers and were trained with the cross-entropy objective function. Sequence
prediction errors are reported in bits-per-sequence. For more details about the experimental
parameters see Section 4.6.
4.1 Copy
The copy task tests whether NTM can store and recall a long sequence of arbitrary in-
formation. The network is presented with an input sequence of random binary vectors
followed by a delimiter ﬂag. Storage and access of information over long time periods has
always been problematic for RNNs and other dynamic architectures. We were particularly
interested to see if an NTM is able to bridge longer time delays than LSTM.
The networks were trained to copy sequences of eight bit random vectors, where the
sequence lengths were randomised between 1 and 20. The target sequence was simply a
copy of the input sequence (without the delimiter ﬂag). Note that no inputs were presented
to the network while it receives the targets, to ensure that it recalls the entire sequence with
no intermediate assistance.
As can be seen from Figure 3, NTM (with either a feedforward or LSTM controller)
learned much faster than LSTM alone, and converged to a lower cost. The disparity be-
tween the NTM and LSTM learning curves is dramatic enough to suggest a qualitative,
10
 0 2 4 6 8 10 0 200 400 600 800 1000
cost per sequence (bits)sequence number (thousands)LSTMNTM with LSTM ControllerNTM with Feedforward Controller
Figure 3: Copy Learning Curves.
rather than quantitative, difference in the way the two models solve the problem.
We also studied the ability of the networks to generalise to longer sequences than seen
during training (that they can generalise to novel vectors is clear from the training error).
Figures 4 and 5 demonstrate that the behaviour of LSTM and NTM in this regime is rad-
ically different. NTM continues to copy as the length increases 2, while LSTM rapidly
degrades beyond length 20.
The preceding analysis suggests that NTM, unlike LSTM, has learned some form of
copy algorithm. To determine what this algorithm is, we examined the interaction between
the controller and the memory (Figure 6). We believe that the sequence of operations per-
formed by the network can be summarised by the following pseudocode:
initialise: move head to start location
while input delimiter not seen do
receive input vector
write input to head location
increment head location by 1
end while
return head to start location
while true do
read output vector from head location
emit output
increment head location by 1
end while
This is essentially how a human programmer would perform the same task in a low-
2The limiting factor was the size of the memory (128 locations), after which the cyclical shifts wrapped
around and previous writes were overwritten.
11
Figure 4: NTM Generalisation on the Copy Task.The four pairs of plots in the top row
depict network outputs and corresponding copy targets for test sequences of length 10, 20, 30,
and 50, respectively. The plots in the bottom row are for a length 120 sequence. The network
was only trained on sequences of up to length 20. The ﬁrst four sequences are reproduced with
high conﬁdence and very few mistakes. The longest one has a few more local errors and one
global error: at the point indicated by the red arrow at the bottom, a single vector is duplicated,
pushing all subsequent vectors one step back. Despite being subjectively close to a correct copy,
this leads to a high loss.
level programming language. In terms of data structures, we could say that NTM has
learned how to create and iterate through arrays. Note that the algorithm combines both
content-based addressing (to jump to start of the sequence) and location-based address-
ing (to move along the sequence). Also note that the iteration would not generalise to
long sequences without the ability to use relative shifts from the previous read and write
weightings (Equation 7), and that without the focus-sharpening mechanism (Equation 9)
the weightings would probably lose precision over time.
4.2 Repeat Copy
The repeat copy task extends copy by requiring the network to output the copied sequence a
speciﬁed number of times and then emit an end-of-sequence marker. The main motivation
was to see if the NTM could learn a simple nested function. Ideally, we would like it to be
able to execute a “for loop” containing any subroutine it has already learned.
The network receives random-length sequences of random binary vectors, followed by
a scalar value indicating the desired number of copies, which appears on a separate input
channel. To emit the end marker at the correct time the network must be both able to
interpret the extra input and keep count of the number of copies it has performed so far.
As with the copy task, no inputs are provided to the network after the initial sequence and
repeat number. The networks were trained to reproduce sequences of size eight random
binary vectors, where both the sequence length and the number of repetitions were chosen
randomly from one to ten. The input representing the repeat number was normalised to
have mean zero and variance one.
12
Figure 5: LSTM Generalisation on the Copy Task.The plots show inputs and outputs
for the same sequence lengths as Figure 4. Like NTM, LSTM learns to reproduce sequences
of up to length 20 almost perfectly. However it clearly fails to generalise to longer sequences.
Also note that the length of the accurate preﬁx decreases as the sequence length increases,
suggesting that the network has trouble retaining information for long periods.
Figure 6: NTM Memory Use During the Copy Task.The plots in the left column depict
the inputs to the network (top), the vectors added to memory (middle) and the corresponding
write weightings (bottom) during a single test sequence for the copy task. The plots on the right
show the outputs from the network (top), the vectors read from memory (middle) and the read
weightings (bottom). Only a subset of memory locations are shown. Notice the sharp focus of
all the weightings on a single location in memory (black is weight zero, white is weight one).
Also note the translation of the focal point over time, reﬂects the network’s use of iterative
shifts for location-based addressing, as described in Section 3.3.2. Lastly, observe that the read
locations exactly match the write locations, and the read vectors match the add vectors. This
suggests that the network writes each input vector in turn to a speciﬁc memory location during
the input phase, then reads from the same location sequence during the output phase.
13
 0 20 40 60 80 100 120 140 160 180 200 0 100 200 300 400 500
cost per sequence (bits)sequence number (thousands)LSTMNTM with LSTM ControllerNTM with Feedforward Controller
Figure 7: Repeat Copy Learning Curves.
Figure 7 shows that NTM learns the task much faster than LSTM, but both were able to
solve it perfectly.3 The difference between the two architectures only becomes clear when
they are asked to generalise beyond the training data. In this case we were interested in
generalisation along two dimensions: sequence length and number of repetitions. Figure 8
illustrates the effect of doubling ﬁrst one, then the other, for both LSTM and NTM. Whereas
LSTM fails both tests, NTM succeeds with longer sequences and is able to perform more
than ten repetitions; however it is unable to keep count of of how many repeats it has
completed, and does not predict the end marker correctly. This is probably a consequence
of representing the number of repetitions numerically, which does not easily generalise
beyond a ﬁxed range.
Figure 9 suggests that NTM learns a simple extension of the copy algorithm in the
previous section, where the sequential read is repeated as many times as necessary.
4.3 Associative Recall
The previous tasks show that the NTM can apply algorithms to relatively simple, linear data
structures. The next order of complexity in organising data arises from “indirection”—that
is, when one data item points to another. We test the NTM’s capability for learning an
instance of this more interesting class by constructing a list of items so that querying with
one of the items demands that the network return the subsequent item. More speciﬁcally,
we deﬁne an item as a sequence of binary vectors that is bounded on the left and right
by delimiter symbols. After several items have been propagated to the network, we query
by showing a random item, and we ask the network to produce the next item. In our
experiments, each item consisted of three six-bit binary vectors (giving a total of 18 bits
3It surprised us that LSTM performed better here than on the copy problem. The likely reasons are that the
sequences were shorter (up to length 10 instead of up to 20), and the LSTM network was larger and therefore
had more memory capacity.
14
Figure 8: NTM and LSTM Generalisation for the Repeat Copy Task.NTM generalises
almost perfectly to longer sequences than seen during training. When the number of repeats is
increased it is able to continue duplicating the input sequence fairly accurately; but it is unable
to predict when the sequence will end, emitting the end marker after the end of every repetition
beyond the eleventh. LSTM struggles with both increased length and number, rapidly diverging
from the input sequence in both cases.
per item). During training, we used a minimum of 2 items and a maximum of 6 items in a
single episode.
Figure 10 shows that NTM learns this task signiﬁcantly faster than LSTM, terminating
at near zero cost within approximately 30,000 episodes, whereas LSTM does not reach
zero cost after a million episodes. Additionally, NTM with a feedforward controller learns
faster than NTM with an LSTM controller. These two results suggest that NTM’s external
memory is a more effective way of maintaining the data structure than LSTM’s internal
state. NTM also generalises much better to longer sequences than LSTM, as can be seen
in Figure 11. NTM with a feedforward controller is nearly perfect for sequences of up to
12 items (twice the maximum length used in training), and still has an average cost below
1 bit per sequence for sequences of 15 items.
In Figure 12, we show the operation of the NTM memory, controlled by an LSTM
with one head, on a single test episode. In “Inputs,” we see that the input denotes item
delimiters as single bits in row 7. After the sequence of items has been propagated, a
15
Figure 9: NTM Memory Use During the Repeat Copy Task.As with the copy task the
network ﬁrst writes the input vectors to memory using iterative shifts. It then reads through
the sequence to replicate the input as many times as necessary (six in this case). The white dot
at the bottom of the read weightings seems to correspond to an intermediate location used to
redirect the head to the start of the sequence (The NTM equivalent of a goto statement).
 0 2 4 6 8 10 12 14 16 18 20 0 200 400 600 800 1000
cost per sequence (bits)sequence number (thousands)LSTMNTM with LSTM ControllerNTM with Feedforward Controller
Figure 10: Associative Recall Learning Curves for NTM and LSTM.
16
0
5
10
15
20
25
30
35
40
6 8 10 12 14 16 18 20
cost per sequence (bits)
num ber of item s per sequence
LS T M
N T M  w ith LS T M  C ontroller
N T M  w ith F eedforw ard C ontroller
Figure 11: Generalisation Performance on Associative Recall for Longer Item Sequences.
The NTM with either a feedforward or LSTM controller generalises to much longer sequences
of items than the LSTM alone. In particular, the NTM with a feedforward controller is nearly
perfect for item sequences of twice the length of sequences in its training set.
delimiter in row 8 prepares the network to receive a query item. In this case, the query
item corresponds to the second item in the sequence (contained in the green box). In
“Outputs,” we see that the network crisply outputs item 3 in the sequence (from the red
box). In “Read Weightings,” on the last three time steps, we see that the controller reads
from contiguous locations that each store the time slices of item 3. This is curious because it
appears that the network has jumped directly to the correct location storing item 3. However
we can explain this behaviour by looking at “Write Weightings.” Here we see that the
memory is written to even when the input presents a delimiter symbol between items.
One can conﬁrm in “Adds” that data are indeed written to memory when the delimiters
are presented (e.g., the data within the black box); furthermore, each time a delimiter is
presented, the vector added to memory is different. Further analysis of the memory reveals
that the network accesses the location it reads after the query by using a content-based
lookup that produces a weighting that is shifted by one. Additionally, the key used for
content-lookup corresponds to the vector that was added in the black box. This implies the
following memory-access algorithm: when each item delimiter is presented, the controller
writes a compressed representation of the previous three time slices of the item. After the
query arrives, the controller recomputes the same compressed representation of the query
item, uses a content-based lookup to ﬁnd the location where it wrote the ﬁrst representation,
and then shifts by one to produce the subsequent item in the sequence (thereby combining
content-based lookup with location-based offsetting).
4.4 Dynamic N-Grams
The goal of the dynamic N-Grams task was to test whether NTM could rapidly adapt to
new predictive distributions. In particular we were interested to see if it were able to use its
17
Figure 12: NTM Memory Use During the Associative Recall Task.In “Inputs,” a se-
quence of items, each composed of three consecutive binary random vectors is propagated to the
controller. The distinction between items is designated by delimiter symbols (row 7 in “Inputs”).
After several items have been presented, a delimiter that designates a query is presented (row 8
in “Inputs”). A single query item is presented (green box), and the network target corresponds
to the subsequent item in the sequence (red box). In “Outputs,” we see that the network cor-
rectly produces the target item. The red boxes in the read and write weightings highlight the
three locations where the target item was written and then read. The solution the network ﬁnds
is to form a compressed representation (black box in “Adds”) of each item that it can store in
a single location. For further analysis, see the main text.
memory as a re-writable table that it could use to keep count of transition statistics, thereby
emulating a conventional N-Gram model.
We considered the set of all possible 6-Gram distributions over binary sequences. Each
6-Gram distribution can be expressed as a table of 25 = 32numbers, specifying the prob-
ability that the next bit will be one, given all possible length ﬁve binary histories. For
each training example, we ﬁrst generated random 6-Gram probabilities by independently
drawing all 32 probabilities from the Beta(1
2 ,1
2 ) distribution.
We then generated a particular training sequence by drawing 200 successive bits using
the current lookup table.4 The network observes the sequence one bit at a time and is then
asked to predict the next bit. The optimal estimator for the problem can be determined by
4The ﬁrst 5 bits, for which insufﬁcient context exists to sample from the table, are drawn i.i.d. from a
Bernoulli distribution with p= 0.5.
18
 130 135 140 145 150 155 160 0 200 400 600 800 1000
cost per sequence (bits)sequence number (thousands)LSTMNTM with LSTM ControllerNTM with Feedforward ControllerOptimal Estimator
Figure 13: Dynamic N-Gram Learning Curves.
Bayesian analysis (Murphy, 2012):
P(B = 1|N1,N0,c) = N1 + 1
2
N1 + N0 + 1 (10)
where c is the ﬁve bit previous context, B is the value of the next bit and N0 and N1 are
respectively the number of zeros and ones observed after c so far in the sequence. We can
therefore compare NTM to the optimal predictor as well as LSTM. To assess performance
we used a validation set of 1000 length 200 sequences sampled from the same distribu-
tion as the training data. As shown in Figure 13, NTM achieves a small, but signiﬁcant
performance advantage over LSTM, but never quite reaches the optimum cost.
The evolution of the two architecture’s predictions as they observe new inputs is shown
in Figure 14, along with the optimal predictions. Close analysis of NTM’s memory usage
(Figure 15) suggests that the controller uses the memory to count how many ones and zeros
it has observed in different contexts, allowing it to implement an algorithm similar to the
optimal estimator.
4.5 Priority Sort
This task tests whether the NTM can sort data—an important elementary algorithm. A
sequence of random binary vectors is input to the network along with a scalar priority
rating for each vector. The priority is drawn uniformly from the range [-1, 1]. The target
sequence contains the binary vectors sorted according to their priorities, as depicted in
Figure 16.
Each input sequence contained 20 binary vectors with corresponding priorities, and
each target sequence was the 16 highest-priority vectors in the input.5 Inspection of NTM’s
5We limited the sort to size 16 because we were interested to see if NTM would solve the task using a
binary heap sort of depth 4.
19
Figure 14: Dynamic N-Gram Inference.The top row shows a test sequence from the N-Gram
task, and the rows below show the corresponding predictive distributions emitted by the optimal
estimator, NTM, and LSTM. In most places the NTM predictions are almost indistinguishable
from the optimal ones. However at the points indicated by the two arrows it makes clear
mistakes, one of which is explained in Figure 15. LSTM follows the optimal predictions closely
in some places but appears to diverge further as the sequence progresses; we speculate that this
is due to LSTM “forgetting” the observations at the start of the sequence.
Figure 15: NTM Memory Use During the Dynamic N-Gram Task.The red and green
arrows indicate point where the same context is repeatedly observed during the test sequence
(“00010” for the green arrows, “01111” for the red arrows). At each such point the same
location is accessed by the read head, and then, on the next time-step, accessed by the write
head. We postulate that the network uses the writes to keep count of the fraction of ones and
zeros following each context in the sequence so far. This is supported by the add vectors, which
are clearly anti-correlated at places where the input is one or zero, suggesting a distributed
“counter.” Note that the write weightings grow fainter as the same context is repeatedly seen;
this may be because the memory records a ratio of ones to zeros, rather than absolute counts.
The red box in the prediction sequence corresponds to the mistake at the ﬁrst red arrow in
Figure 14; the controller appears to have accessed the wrong memory location, as the previous
context was “01101” and not “01111.”
20
Figure 16: Example Input and Target Sequence for the Priority Sort Task.The input
sequence contains random binary vectors and random scalar priorities. The target sequence is a
subset of the input vectors sorted by the priorities.
Write Weightings
 Read Weightings
Hypothesised Locations
Time Time Time
Locatiobn
Figure 17: NTM Memory Use During the Priority Sort Task. Left: Write locations
returned by ﬁtting a linear function of the priorities to the observed write locations. Middle:
Observed write locations. Right: Read locations.
memory use led us to hypothesise that it uses the priorities to determine the relative location
of each write. To test this hypothesis we ﬁtted a linear function of the priority to the
observed write locations. Figure 17 shows that the locations returned by the linear function
closely match the observed write locations. It also shows that the network reads from the
memory locations in increasing order, thereby traversing the sorted sequence.
The learning curves in Figure 18 demonstrate that NTM with both feedforward and
LSTM controllers substantially outperform LSTM on this task. Note that eight parallel
read and write heads were needed for best performance with a feedforward controller on
this task; this may reﬂect the difﬁculty of sorting vectors using only unary vector operations
(see Section 3.4).
4.6 Experimental Details
For all experiments, the RMSProp algorithm was used for training in the form described
in (Graves, 2013) with momentum of 0.9. Tables 1 to 3 give details about the network
conﬁgurations and learning rates used in the experiments. All LSTM networks had three
stacked hidden layers. Note that the number of LSTM parameters grows quadratically with
21
 0 20 40 60 80 100 120 140 0 200 400 600 800 1000
cost per sequence (bits)sequence number (thousands)LSTMNTM with LSTM ControllerNTM with Feedforward Controller
Figure 18: Priority Sort Learning Curves.
Task #Heads Controller Size Memory Size Learning Rate #Parameters
Copy 1 100 128 ×20 10−4 17,162
Repeat Copy 1 100 128 ×20 10−4 16,712
Associative 4 256 128 ×20 10−4 146,845
N-Grams 1 100 128 ×20 3 ×10−5 14,656
Priority Sort 8 512 128 ×20 3 ×10−5 508,305
Table 1: NTM with Feedforward Controller Experimental Settings
the number of hidden units (due to the recurrent connections in the hidden layers). This
contrasts with NTM, where the number of parameters does not increase with the number of
memory locations. During the training backward pass, all gradient components are clipped
elementwise to the range (-10, 10).
5 Conclusion
We have introduced the Neural Turing Machine, a neural network architecture that takes
inspiration from both models of biological working memory and the design of digital com-
puters. Like conventional neural networks, the architecture is differentiable end-to-end and
can be trained with gradient descent. Our experiments demonstrate that it is capable of
learning simple algorithms from example data and of using these algorithms to generalise
well outside its training regime.
22
Task #Heads Controller Size Memory Size Learning Rate #Parameters
Copy 1 100 128 ×20 10−4 67,561
Repeat Copy 1 100 128 ×20 10−4 66,111
Associative 1 100 128 ×20 10−4 70,330
N-Grams 1 100 128 ×20 3 ×10−5 61,749
Priority Sort 5 2 ×100 128 ×20 3 ×10−5 269,038
Table 2: NTM with LSTM Controller Experimental Settings
Task Network Size Learning Rate #Parameters
Copy 3 ×256 3 ×10−5 1,352,969
Repeat Copy 3 ×512 3 ×10−5 5,312,007
Associative 3 ×256 10 −4 1,344,518
N-Grams 3 ×128 10 −4 331,905
Priority Sort 3 ×128 3 ×10−5 384,424
Table 3: LSTM Network Experimental Settings
6 Acknowledgments
Many have offered thoughtful insights, but we would especially like to thank Daan Wier-
stra, Peter Dayan, Ilya Sutskever, Charles Blundell, Joel Veness, Koray Kavukcuoglu,
Dharshan Kumaran, Georg Ostrovski, Chris Summerﬁeld, Jeff Dean, Geoffrey Hinton, and
Demis Hassabis.
23
References
Baddeley, A., Eysenck, M., and Anderson, M. (2009). Memory. Psychology Press.
Bahdanau, D., Cho, K., and Bengio, Y . (2014). Neural machine translation by jointly
learning to align and translate. abs/1409.0473.
Barrouillet, P., Bernardin, S., and Camos, V . (2004). Time constraints and resource shar-
ing in adults’ working memory spans. Journal of Experimental Psychology: General ,
133(1):83.
Chomsky, N. (1956). Three models for the description of language. Information Theory,
IEEE Transactions on, 2(3):113–124.
Das, S., Giles, C. L., and Sun, G.-Z. (1992). Learning context-free grammars: Capabil-
ities and limitations of a recurrent neural network with an external stack memory. In
Proceedings of The Fourteenth Annual Conference of Cognitive Science Society. Indiana
University.
Dayan, P. (2008). Simple substrates for complex cognition. Frontiers in neuroscience,
2(2):255.
Eliasmith, C. (2013). How to build a brain: A neural architecture for biological cognition.
Oxford University Press.
Fitch, W., Hauser, M. D., and Chomsky, N. (2005). The evolution of the language faculty:
clariﬁcations and implications. Cognition, 97(2):179–210.
Fodor, J. A. and Pylyshyn, Z. W. (1988). Connectionism and cognitive architecture: A
critical analysis. Cognition, 28(1):3–71.
Frasconi, P., Gori, M., and Sperduti, A. (1998). A general framework for adaptive process-
ing of data structures. Neural Networks, IEEE Transactions on, 9(5):768–786.
Gallistel, C. R. and King, A. P. (2009). Memory and the computational brain: Why cogni-
tive science will transform neuroscience, volume 3. John Wiley & Sons.
Goldman-Rakic, P. S. (1995). Cellular basis of working memory. Neuron, 14(3):477–485.
Graves, A. (2013). Generating sequences with recurrent neural networks. arXiv preprint
arXiv:1308.0850.
Graves, A. and Jaitly, N. (2014). Towards end-to-end speech recognition with recurrent
neural networks. In Proceedings of the 31st International Conference on Machine Learn-
ing (ICML-14), pages 1764–1772.
24
Graves, A., Mohamed, A., and Hinton, G. (2013). Speech recognition with deep recurrent
neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE
International Conference on, pages 6645–6649. IEEE.
Hadley, R. F. (2009). The problem of rapid variable creation. Neural computation ,
21(2):510–532.
Hazy, T. E., Frank, M. J., and O’Reilly, R. C. (2006). Banishing the homunculus: making
working memory work. Neuroscience, 139(1):105–118.
Hinton, G. E. (1986). Learning distributed representations of concepts. In Proceedings
of the eighth annual conference of the cognitive science society , volume 1, page 12.
Amherst, MA.
Hochreiter, S., Bengio, Y ., Frasconi, P., and Schmidhuber, J. (2001a). Gradient ﬂow in
recurrent nets: the difﬁculty of learning long-term dependencies.
Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory.Neural computation,
9(8):1735–1780.
Hochreiter, S., Younger, A. S., and Conwell, P. R. (2001b). Learning to learn using gradient
descent. In Artiﬁcial Neural Networks?ICANN 2001, pages 87–94. Springer.
Hopﬁeld, J. J. (1982). Neural networks and physical systems with emergent collective
computational abilities. Proceedings of the national academy of sciences , 79(8):2554–
2558.
Jackendoff, R. and Pinker, S. (2005). The nature of the language faculty and its implications
for evolution of language (reply to ﬁtch, hauser, and chomsky). Cognition, 97(2):211–
225.
Kanerva, P. (2009). Hyperdimensional computing: An introduction to computing in dis-
tributed representation with high-dimensional random vectors. Cognitive Computation,
1(2):139–159.
Marcus, G. F. (2003). The algebraic mind: Integrating connectionism and cognitive sci-
ence. MIT press.
Miller, G. A. (1956). The magical number seven, plus or minus two: some limits on our
capacity for processing information. Psychological review, 63(2):81.
Miller, G. A. (2003). The cognitive revolution: a historical perspective.Trends in cognitive
sciences, 7(3):141–144.
Minsky, M. L. (1967). Computation: ﬁnite and inﬁnite machines. Prentice-Hall, Inc.
Murphy, K. P. (2012). Machine learning: a probabilistic perspective. MIT press.
25
Plate, T. A. (2003). Holographic Reduced Representation: Distributed representation for
cognitive structures. CSLI.
Pollack, J. B. (1990). Recursive distributed representations. Artiﬁcial Intelligence ,
46(1):77–105.
Rigotti, M., Barak, O., Warden, M. R., Wang, X.-J., Daw, N. D., Miller, E. K., and Fusi,
S. (2013). The importance of mixed selectivity in complex cognitive tasks. Nature,
497(7451):585–590.
Rumelhart, D. E., McClelland, J. L., Group, P. R., et al. (1986). Parallel distributed pro-
cessing, volume 1. MIT press.
Seung, H. S. (1998). Continuous attractors and oculomotor control. Neural Networks,
11(7):1253–1258.
Siegelmann, H. T. and Sontag, E. D. (1995). On the computational power of neural nets.
Journal of computer and system sciences, 50(1):132–150.
Smolensky, P. (1990). Tensor product variable binding and the representation of symbolic
structures in connectionist systems. Artiﬁcial intelligence, 46(1):159–216.
Socher, R., Huval, B., Manning, C. D., and Ng, A. Y . (2012). Semantic compositionality
through recursive matrix-vector spaces. In Proceedings of the 2012 Joint Conference on
Empirical Methods in Natural Language Processing and Computational Natural Lan-
guage Learning, pages 1201–1211. Association for Computational Linguistics.
Sutskever, I., Martens, J., and Hinton, G. E. (2011). Generating text with recurrent neural
networks. In Proceedings of the 28th International Conference on Machine Learning
(ICML-11), pages 1017–1024.
Sutskever, I., Vinyals, O., and Le, Q. V . (2014). Sequence to sequence learning with neural
networks. arXiv preprint arXiv:1409.3215.
Touretzky, D. S. (1990). Boltzcons: Dynamic symbol structures in a connectionist network.
Artiﬁcial Intelligence, 46(1):5–46.
V on Neumann, J. (1945). First draft of a report on the edvac.
Wang, X.-J. (1999). Synaptic basis of cortical persistent activity: the importance of nmda
receptors to working memory. The Journal of Neuroscience, 19(21):9587–9603.
26

FuncyTuner: Auto-tuning Scientific Applications With Per-loop
Compilation
Tao Wang
Dept. of Computer Science
North Carolina State University
twang15@ncsu.edu
Nikhil Jain
Lawrence Livermore National
Laboratory
nikhil.jain@acm.org
David Beckingsale
Lawrence Livermore National
Laboratory
beckingsale1@llnl.gov
David Boehme
Lawrence Livermore National
Laboratory
boehme3@llnl.gov
Frank Mueller
Dept. of Computer Science
North Carolina State University
fmuelle@ncsu.edu
Todd Gamblin
Lawrence Livermore National
Laboratory
gamblin2@llnl.gov
ABSTRACT
The de facto compilation model for production software compiles all
modules of a target program with a single set of compilation flags,
typically O2 or O3. Such a per-program compilation strategy may
yield sub-optimal executables since programs often have multiple
hot loops with diverse code structures and may be better optimized
with a per-region compilation model that assembles an optimized
executable by combining the best per-region code variants.
In this paper, we demonstrate that a naïve greedy approach to
per-region compilation often degrades performance in comparison
to the O3 baseline. To overcome this problem, we contribute a novel
per-loop compilation framework, FuncyTuner, which employs light-
weight profiling to collect per-loop timing information, and then
utilizes a space-focusing technique to construct a performant exe-
cutable. Experimental results show that FuncyTuner can reliably
improve performance of modern scientific applications on several
multi-core architectures by 9.2% to 12.3% and 4.5% to 10.7%(geomet-
ric mean, up to 22% on certain program) in comparison to the O3
baseline and prior work, respectively.
KEYWORDS
per-loop, fine-grained, auto-tuning, ICC, compiler, optimization,
profile, OpenMP, HPC, scientific simulation
ACM Reference Format:
Tao Wang, Nikhil Jain, David Beckingsale, David Boehme, Frank Mueller,
and Todd Gamblin. 2019. FuncyTuner: Auto-tuning Scientific Applications
With Per-loop Compilation. In 48th International Conference on Parallel
Processing (ICPP 2019), August 5–8, 2019, Kyoto, Japan. ACM, New York, NY,
USA, 10 pages. https://doi.org/10.1145/3337821.3337842
1 INTRODUCTION
The de facto compilation model compiles all source files of a pro-
gram with a single set of compiler flags, typically O2 or O3. However,
it is well-known that O2/O3 may not generate the most performant
executables [7]. This is because optimizations enabled by flags such
ACM acknowledges that this contribution was authored or co-authored by an employee,
contractor, or affiliate of the United States government. As such, the United States
government retains a nonexclusive, royalty-free right to publish or reproduce this
article, or to allow others to do so, for government purposes only.
ICPP 2019, August 5–8, 2019, Kyoto, Japan
© 2019 Association for Computing Machinery.
ACM ISBN 978-1-4503-6295-5/19/08. . . $15.00
https://doi.org/10.1145/3337821.3337842
 0.4
 0.6
 0.8
 1
 1.2
GCC ICC
Speedup normalized to O3
LULESH Cloverleaf AMG
Figure 1: Combined Elimination does not improve perfor-
mance significantly.
as O2/O3 are empirically determined to maximize performance
for certain benchmark suites, while other programs and architec-
tural platforms may have different characteristics that require other
optimizations. As a result, for a given program, other compiler
flag combinations may exist that can produce more performant
executables than the default O2/O3 choice.
In order to identify the most performant optimizations for a pro-
gram, researchers have proposed iterative compilation [14]. Given a
program, iterative compilation first generates code variants by com-
piling the source code with different compiler flags, and then evalu-
ates their performance either by execution [7] or prediction [6, 22].
Several algorithms have been proposed for generating compiler
flag combinations to create code variants, e.g., predictive machine
learning models [ 4, 6, 13, 20] and optimization flag correlation-
based combined elimination [21]. They perform compilation on a
per-program basis. However, their effectiveness is limited. While
machine learning based predictive models [6] perform well on small
training datasets, a recent study [10] shows that their prediction
accuracy drops dramatically for large-scale datasets and exhibits
close to random behavior.
Combined elimination (CE) [21] takes advantage of interactions
among compiler flags to find the best combination. However, our
evaluation of CE, shown in Fig. 1, for three benchmarks on Intel’s
Broadwell architecture shows minimal performance benefit in com-
parison to the O3 baseline for both the GNU C/C++ compiler (GCC
release 5.4.0) and the Intel C/C++ compiler (ICC release 17.0.4). A
closer inspection of the experiments for CE revealed that it can be
limited by the results for local minima.
Fine-grained per-region compilation techniques [11, 21, 22, 25]
divide a program into different compilation modules and optimize
1
ICPP 2019, August 5–8, 2019, Kyoto, Japan Tao Wang and Nikhil Jain, et al.
each separately. Specifically, source-code level auto-tuners [11, 25]
focus on a single simple computation kernel without considering
module interactions in real-world applications. As prior work in
compiler flag selection [21, 22] also assumes that compilation mod-
ules are independent, they assemble an optimized executable by
greedily picking the best code variant of each module. However,
the modules of a program may not be independent due to cross-
module interference, such as shared data structures and link-time
inter-procedural optimizations across multiple modules. In particu-
lar, link-time optimizations can drive optimization decisions, such
as loop unrolling, and may invalidate earlier transformations that
were made independently for the compilation modules.
Contributions: Based on the observations above, we develop a
fine-grained auto-tuning framework, FuncyTuner, that targets mod-
ern scientific applications. Our overall objective is to extract the
best performance out of an application that is executed repeatedly,
such as in high-performance computing (HPC) where scientists
test their hypotheses in experiments repeatedly with similar inputs
using the same algorithms. To clarify, we neither attempt to derive
a better set of optimizations for O3, nor do we attempt to generalize
a specific set of optimizations across region boundaries or select dif-
ferent algorithmic code variants according to input characteristics
(in contrast to [18]). Instead, our objective is to 1) assess whether
or not there are module interactions and, if so, 2) understand how
to capitalize on such interactions.
Our target HPC applications exploit multi-core parallelism via
OpenMP. Their hot-spots consist of OpenMP loops that account
for a significant fraction of execution time. FuncyTuner outlines
these loops and converts them into individual functions, whose
compilation can be auto-tuned. To this end, FuncyTuner employs
a novel and effective search space focusing technique to guide
random search based auto-tuned compilation.
The contributions of this paper are:
•We develop a per-loop compiler flag selection frame-
work, FuncyTuner, that combines program profiling with
Caliper [5] and search space focusing algorithms to tune hot
loops of modern scientific programs simultaneously with-
out sacrificing the indispensable optimization context for
production compilers.
•We demonstrate that FuncyTuner is able to improve program
performance for a set of scientific benchmarks by 9.3% to
12.3% in comparison to the O3 baseline and 4.5% to 10.7%
relative to prior work, both in geometric mean (up to 22%
on certain programs), on several generations of HPC archi-
tecture, and thus refreshes the state of the art.
•We conduct an in-depth case study for Cloverleaf [24] on the
Intel Broadwell architecture to elaborate how fine-grained
compiler flag selection is affected by inter-module depen-
dencies and demonstrate that it should not be performed
greedily, but rather in a focused and targeted manner.
Our FuncyTuner implementation automates these steps, with
the exception of Caliper [5] profile instrumentation and collection
of timing results, which are manual in our research prototype, but
could be automated with further engineering efforts invested (but
are of no research value).
2 DESIGN OF FUNCYTUNER
This section first defines terminologies and notations used through-
out this work and then details the design of the FuncyTuner frame-
work.
2.1 Compiler Flag Space Construction
Modern optimizing compilers feature many internal optimization
passes and may expose several hundreds of command-line flags to
parameterize them. Each flag could either be a binary switch to turn
on/off a certain optimization, e.g., loop unrolling and loop tiling, or
a multi-valued parametric option to set pass-specific parameters,
e.g., thresholds for function inlining and algorithmic variants of
register allocation. The set of all flags composes a space called the
compiler optimization space (COS , size roughly 2.3e13 in this work
since there are 33 selected flags while some have multiple values), in
which each point is a set of instantiated flags called a compilation
vector (CV ). Suppose there are N compiler flags, denoted as Fi
(1 ≤i ≤N ), and suppose Fi has ni possible values fi1, fi2, ..., fini .
Then a sample CV is represented as (F1 = f1k1 , F2 = f2k2 , ...,FN =
fN kN ), where 1 ≤ki ≤ni . Thus, there are in total C0 = QN
i=1 ni
CV s, each of which could be used to compile all source files of a
program in a traditional compilation model.
Given a program P, a traditional compilation model treats all
source files as a single compilation module M, within which the
source files are compiled with the sameCV . In contrast, FuncyTuner
divides program P into J compilation modules M1, M2, ...,MJ (J
is program-specific and ranges from 5 to 33 in this work). These
modules are created based on the time spent in various regions, in
particular loops, of the program P. FuncyTuner then compiles Mj
with a CV , which could be determined independently from other
modules and links all object files together to produce the executable.
Our hypothesis is that different compilation modules may need
different CV s to obtain the best performance due to their diverse
code structures.
However, to identify the best CV s, the primary challenge is that
the new search space size COSnew increases significantly from C0
to C1 = C0 J . Exhaustive search is not a viable option within such
an excessively large space while machine learning-based predictive
models require a significant amount of training data and meaningful
features to begin with [10]. To address this challenge, we propose
several algorithms, as detailed in the rest of this section. We do
not differentiate between a program source code module and its
corresponding object/binary module. Therefore, Pk and Mjk (j-th
compilation module ofPk ) may represent the source code module or
compiled program object/binary module, depending on the context.
2.2 Space Search Algorithms
In this section, we introduce four different search algorithms. Note
that per-program random search is a classical algorithm and is used
as a reference to understand the other three algorithms.
2.2.1 Per-program Random Search. Per-program random search
(denoted as Random) does not modify program source code and
applies a single CV to all source files of any program P. As shown
in Fig. 2, in step 1⃝, 1000 CV samples are randomly selected from
COS . In step 2⃝, each CV is used to compile P to obtain a code
2
FuncyTuner ICPP 2019, August 5–8, 2019, Kyoto, Japan
Figure 2: Per-program random search. CVk and Pk with min-
imal runtime Tk is its final result.
variant, Pk (1 ≤k ≤1000). In step 3⃝, runtimes are collected for all
code variants. In the end, the code variant with the least runtime is
selected as the result. The size of its search space is C0.
Figure 3: Per-function random search. Step 3⃝is performed
1000 times.
2.2.2 Per-function Random Search. As shown in Fig. 3, per-function
random search (denoted as FR ) begins with Caliper profiling a pro-
gram P to identify hot loops and outlines each of them into a
separate source file so that there are J compilation modules. In step
3⃝, J CV s are randomly selected from 1000 pre-sampled CV s with
replacement. Each selected CV is used to compile one of the J com-
pilation modules. Note that the selection of J CV s and compilation
is performed 1000 times in step 3⃝to generate 1000 code variants,
which are executed to collect runtimes T1, ..., T1000. FR reports the
code variant with the minimum runtime as the best version.
FR uses Caliper [5] profiling only for identifying hot loops, but it
does not collect the per-loop runtime information for searching the
best per-loop CV s. However, when such information is available,
one may choose better CV s for each compilation module. To this
end, we introduce FuncyTuner, our per-loop runtime collection
framework, shown in Fig. 4. As in FR , a program P is first divided
into J compilation modules. In step 2⃝, FuncyTuner instruments
modules via Caliper’s light-weight [5] APIs to measure per-loop
runtimes. Then, in step 4⃝, 1000 pre-sampled CV1, ..., CV1000 are
used to compile P such that all modules within P are compiled with
the same k-th CV to generate Pk . In step 5⃝, the generated 1000
code variants are executed to collect per-loop runtimes, denoted
as Tjk for the module Mj of the code variant Pk . This information
is utilized in different ways by the next two algorithms, namely
Figure 4: FuncyTuner per-loop runtime collection frame-
work is a part of both greedy combination and Caliper-
guided random search.
greedy combination (see Section 2.2.3) and Caliper-guided random
search (see Section 2.2.4).
2.2.3 Greedy Combination. Greedy combination (denoted as G)
exploits per-loop runtimes in a straight-forward way. It assembles
the final executable by picking the fastest code variant for each
module and links them together, assuming that there are no inter-
module dependencies such that the greedy composition produces
the fastest executable. Formally, G chooses the i-th CV to compile
Mj such that i = argmink {Tjk |1 ≤k ≤1000}, and then links all
modules to produce the target executable.
2.2.4 Caliper-guided Random Search. Similar to G, Caliper-guided
random search (denoted as CFR , see Algorithm 1) also relies on
per-loop runtime information to make informed selections of CV s.
However,CFR examines more code variants than G to take poten-
tial inter-module dependencies into consideration. In contrast toFR ,
CFR prunes the pre-sampled search space for each hot loop before
re-sampling per-loop CV s (line 13 of Algorithm 1). The intuition
is that more performant CV s, which generate faster per-loop code
variants, should be kept in the re-sampling search space, because
they may have a higher chance to compose a performant target
executable. Within a unified algorithmic framework, G can be con-
sidered as only selecting the top-1 CV s, and that FR selects all 1000
or the top-1000 CV s, while CFR selects the top-X (1 < X << 1000)
CV s, all on a per-loop basis.
To summarize,Random is a traditional search algorithm that per-
forms on per-program granularity, while FR , G, and CFR perform
on a per-loop basis but with different mechanisms and motivations:
FR is to evaluate if random search with per-loop granularity alone
is sufficient to achieve the best performance; G is to assess if there
are inter-module dependencies by evaluating the effectiveness of
greedily combining the best per-loop code variants; and CFR is to
see if a focused search space can improves the performance beyond
that of Random, FR and G. Note that FR and CFR are proposed by
us, while Random and G are based on prior work [21, 22].
3 EXPERIMENTAL DESIGN
In order to evaluate the efficacy of schemes presented in Section 2,
we have performed experiments for seven modern scientific bench-
marks on three architectures. This section describes the setup used
in our experiments.
3
ICPP 2019, August 5–8, 2019, Kyoto, Japan Tao Wang and Nikhil Jain, et al.
Algorithm 1: Caliper-guided Random Search (CFR)
Input : COS , K, P, X, TO3
Output :speedup , CV
1 k = 1, K = 1000, T = [], CV s= [] , CVpruned = [][]
2 CV s = randomSample (COS , K)
3 //step-1: FuncyTuner per-loop data collection
4 for k = 0; k < K; k++ do
5 compile P with CV s[k] to generate Pk
6 run P to measure T1k, ..., TJ k
7 for j = 0; j < J ; j + +do
8 T [j][k] = Tj+1k
9 //prune pre-sampled 1000 CVs
10 for j = 0; j < J ; j++ do
11 CVpruned [j][] = {CV s[i] |T [j][i] is amonд top X-
smallest in T [j][0], ..., T [j][K −1]}
12 for k = 0; k < K; k++ do
13 //re-sampling per-loop cv in pruned space.
14 for j = 0; j < J ; j++ do
15 tempCVs[k][j] = randomSample(CVpruned [j], 1)
16 for j = 0; j < J ; j++ do
17 compile Mj with tempCVs[k][j]
18 link M1, ..., MJ to generate Pk
19 //Tk : end-to-end runtime for Pk and Tk =
JP
j=1
Tjk
20 run Pk to measure Tk
21 T [k] = Tk
22 k = argmink {T [k] |1 ≤k ≤K }
23 CV = tempCV s[k]
24 //TO3: end-to-end runtime for P compiled with O3
25 speedup = TO3/T [k]
3.1 Systems and Benchmarks
We conducted our experiments on three platforms: AMD Opteron,
Intel Sandy Bridge, and Intel Broadwell. The architectural details
for these systems are provided in Table 2. Our benchmark suite
(see Table 1) consists of seven HPC programs: AMG [17], LULESH
[17], Cloverleaf (CL) [24], 351.bwaves, 362.fma3d, 363.swim and
Optewe [23]. 351.bwaves, 362.fma3d, and 363.swim are from the
SPEC OMP 2012 suite, while the rest are widely used HPC proxy
applications. These benchmarks have been selected based on two
criteria. First, they are written in different languages and exploit
multi-core parallelism suitable for HPC via OpenMP pragmas. Sec-
ond, they feature more than one hot loop, which resembles realistic
applications (unlike many other benchmarks with just a single hot
loop). While multiple hot loops present difficulties for the compiler
in coordinating various loop optimizations, they also provide an
opportunity to optimize different parts of the code differently.
All our experiments have been run on CentOS Linux 7.3.1611
and the benchmarks were compiled with the Intel C/C++ Compiler
17.04. OpenMP thread placement has been set to fine, proclist=[...]
explicit, where proclist is specified in Table 2. Details of the OpenMP
configurations are presented in Table 2. Since scientific codes follow
a time-step execution pattern repeatedly performing approxima-
tions with decreasing numerical error in an outer loop, we only run
for a small number of time-steps (seconds) and then exit prema-
turely once we have obtained a stable execution time for a time-step.
Any optimization then scales up to a full run over all time-steps
(hours). To this end, input sizes and time-steps have been adjusted
so that every single run is less than 40 seconds for the O3 baseline
compilation. In the first experiments (Section 4.1 and Section 4.2),
we use the same inputs for tuning and testing, whereas we evaluate
the impact of different inputs (Section 4.3) in later experiments to
Table 1: List of benchmarks. LOC: lines of source code.
Name Language LOC Domain
AMG C 113k Math: linear solver
LULESH C++ 7.2k Hydrodynamics
Cloverleaf (CL) C, Fortran 14.5k Hydrodynamics
351.bwaves Fortran 1.2k Computational fluid dynamics
362.fma3d Fortran 62k Mechanical simulation
363.swim Fortran 0.5k Weather prediction
Optewe C++ 2.7k Seismic wave simulation
Table 2: Platform overview, runtime configurations, and
benchmark inputs.
Machine AMD
Opteron
Intel Sandy
Bridge
Intel Broadwell
Processor Opteron
6128
Xeon E5-
2650 0
Xeon E5-2620 v4
Sockets 2 2 2
NUMA nodes 4 2 2
Cores/Socket 4 8 8
Threads/Core 2 2 2
Core Frequency [GHz] 2.0 2.0 2.1
processor-specific flag default -xAVX -xCORE-AVX2
Memory size [GB] 32 16 64
OpenMP thread count 16 16 16
OpenMP thread proclist [0-15] [0-15] [0-15]
LULESH: size, steps 120, 10 150, 10 200, 10
Cloverleaf: size, steps 2000,30 2000,30 2000,60
AMG: size 18 20 25
Optewe: size, steps 320, 5 384, 5 512, 5
bwaves: input, steps train, 10 train, 15 train, 50
fma3d: input train train train
swim: input train train train
reflect typical usage patterns of repeated HPC application runs with
different scientific inputs.
3.2 Compiler Flag Selection
We experiment with 33 optimization-related compilation flags of
the Intel compilers. For flags that support any value in a continuous
range as input, we discretize the values in the given range. Then,
for each flag Fi , FuncyTuner selects a value fi from fi1, fi2, ..., fini
with equal probability. ACV is constructed by concatenating the
selected values for all Fi s. We had to consider several restrictions
when selecting the flags. First, a flag must not prevent a program
from running successfully on a given target architecture. For ex-
ample, use of the -fpack flag generates code variants that cause a
segmentation fault at runtime and thus -fpack is excluded. Second,
for fair performance comparison among different code variants,
FuncyTuner enforces strict floating point reproducibility by dis-
carding floating point related optimization flags, and always uses
-fp-model source in the presented results. Last, optimized library
options, such as Intel MKL and IPP related linkage options, are also
excluded since they are not used by our benchmarks.
Also, in order to reach the full optimization potential of the Intel
compiler tool chain, according to the Intel optimization note, Intel’s
linker xild and library archive tool xiar should be used. We thus
modify build systems of all benchmarks accordingly. Processor-
specific flags are also considered for the best performance on each
architecture.
4
FuncyTuner ICPP 2019, August 5–8, 2019, Kyoto, Japan
3.3 Loop Outlining and Caliper
Instrumentation
To identify hot loops that need to be outlined into individual mod-
ules, FuncyTuner uses Caliper [5] to profile the target application
compiled with -O3 -qopenmp -fp-model source . Every loop whose
runtime is at the least 1.0% of the baseline’s end-to-end runtime is
outlined as an independent compilation module for maximum free-
dom of CV selection. The runtime for code other than the hot loops
(non-loop code) cannot be directly measured because such code
tends to be scattered across many source files. Thus, the runtime of
non-loop code is derived by subtracting the aggregate runtime of
hot loops from the end-to-end runtime for each code variant of a
program. Caliper instrumentations generally introduce less than
3% overhead and the per-loop runtimes are sufficiently informa-
tive to FuncyTuner so that measurement noise is tolerated with its
search algorithms. To evaluate performance, we use -O3 -qopenmp
-fp-model source as the baseline and report speedups relative to this
baseline unless otherwise specified.
3.4 CV Independence Assumption
Note that there are two sets of results for the greedy combina-
tion. One is obtained by runtime measurement and is denoted as
G.realized . The other marked G.Independent is calculated by sum-
ming up the best per-loop and non-loop code runtimes obtained
with differentCV s. G.Independent is used as the hypothetical upper
bound for the greedy combination and serves as a reference to as-
sess if there is pairwise independence among different compilation
modules.
4 RESULTS AND ANALYSIS
In this section, we first present results of the four algorithms in
Sec. 2 on three HPC architectures. We then compare FuncyTuner
CFR with prior work, and study their sensitivity on different inputs.
To shed light on why CFR performs the best, we conduct a case
study on Cloverleaf.
4.1 Overall Performance Comparison
Fig. 5 compares the performance for the four algorithms in Sec. 2, i.e.,
Random is the classical per-program random search; FR and CFR
are the two per-loop algorithms proposed by us; G.realized and
G.Independent are results for greedy combination G as explained
in Sec. 3.4. Note thatRandom is applied on the original benchmarks
while others are all applied on the benchmarks with their hot loops
outlined. Moreover, Caliper instrumentations are needed only in
the per-loop runtime collection but not in the final optimized exe-
cutables. For all results over the 7 programs on both training and
testing inputs, execution times were between 3 and 36 seconds with
a standard deviation of 0.04 to 0.2 (except for two cases with 1.5
and 0.7 for longer LULESH runs) measured over 10 experiments,
i.e., results are very uniform with high statistical significance. From
these results, we make the following observations.
(1) FuncyTunerCFR provides the best performing executables for
most scenarios across benchmarks and architectures. It provides
9.2%, 10.3%, 9.4% geometric mean speedups for Opteron, Sandy
Bridge and Broadwell, respectively. It also achieves the best case
improvement of 18.1% for AMG on AMD Opteron (see Fig. 5a)
in comparison to the O3 baseline. In contrast, the performance
improvement due to Random is only 3.4%, 5.0%, 4.6% on the same
respective architectures. In certain cases,Random does not improve
performance at all while CFR does much better, e.g., for AMG on
Sandy Bridge and Broadwell.
(2) G results in significant slowdowns for many benchmark and
architecture combinations. Although it improves performance of
AMG on Opteron and Sandy Bridge, the improvement is still in-
ferior to that of FuncyTuner CFR . The huge differences between
G.realized and G.Independent substantiate that there are inter-
module dependencies for benchmarks in our experiment.
(3) FR ’s performance is inferior to that ofCFR and has high variance.
For example, it achieves less than a 3% improvement for Cloverleaf
on Opteron, Sandy Bridge and Broadwell, whileCFR achieves 13.6%,
15.2% and 12.7%, respectively. Such results demonstrate that random
search with per-loop granularity alone is insufficient for achieving
the best performance and CFR ’s effective utilization of per-loop
runtime information is critical.
4.2 Comparison to the State-of-the-art
We first introduce the state-of-art techniques and experimental
settings to fairly compare with FuncyTuner CFR on Intel Broad-
well. Then, we present the results and our observations that CFR
outperforms all of them.
4.2.1 The State-of-the Art. Prior techniques are either search-
based [ 2, 22] or predictive modeling-based [ 4]. In particular,
Cere [22] performs a fine-grained compiler flag selection for hot
code regions and generates the optimized executable in the same
greedy fashion as G in our work. Our results in Fig. 5 show that
this often degrades performance for the Intel compilers.
OpenTuner [2] performs per-program search with an ensemble
of search algorithms, including differential evolution, Torczon hill-
climbers, Nelder-Mead and many others. It also employs a meta
search (AUC Bandit) technique to coordinate different search al-
gorithms for the best performance. To compare with FuncyTuner
CFR , we run OpenTuner with 1000 test iterations using the same
CV search space.
COBAY N [4], a state-of-the-art machine learning-based ap-
proach, infers performant CV s for a new program by extracting
static and dynamic program features and providing them as in-
puts to a pre-trained Bayesian network. To compare, we first train
COBAY Nwith cBench [8]. Specifically, we select the top 100 per-
formant CV s out of 1000 random CV samples for each cBench ap-
plication to extract their static and dynamic features with Milepost-
gcc [9] and Mica [12]. We then train three models, static, dynamic,
and hybrid , using static features, dynamic features and all features,
respectively. SinceCOBAY Ncan only perform inferences on binary
compiler flags, we turn each multi-valued ICC flag into a binary one
by allowing it to have two values. Then, we use each of the three
models to generate 1000 code variants. The fastest code variant is
considered as the result of each model.
Aside from the above meta-compilation techniques, Intel com-
pilers support built-in profile-guided optimization ( PGO), which
5
ICPP 2019, August 5–8, 2019, Kyoto, Japan Tao Wang and Nikhil Jain, et al.
 0.6
 0.7
 0.8
 0.9
 1
 1.1
 1.2
LULESH CL AMG Optewe bwaves fma3d swim GM
Speedup normalized to O3
Random G.realized FR CFR G.Independent
1.521.52
(a) Normalized speedups on AMD Opteron
 0.6
 0.7
 0.8
 0.9
 1
 1.1
 1.2
LULESH CL AMG Optewe bwaves fma3d swim GM
Speedup normalized to O3
Random G.realized FR CFR G.Independent
(b) Normalized speedups on Intel Sandy Bridge. G.realized’s Optewe speedup: 0.34
 0.6
 0.7
 0.8
 0.9
 1
 1.1
 1.2
LULESH CL AMG Optewe bwaves fma3d swim GM
Speedup normalized to O3
Random G.realized FR CFR G.Independent
1.731.731.241.24 1.241.24
(c) Normalized speedups on Intel Broadwell
Figure 5: CFR outperforms other methods for most cases: geometric mean of speedups relative to the O3 baseline 9.2%, 10.3%,
and 9.4% on Opteron, Sandy Bridge and Broadwell, respectively.
 0.9
 0.95
 1
 1.05
 1.1
 1.15
 1.2
LULESH CL AMG Optewe bwaves fma3d swim GM
Speedup normalized to O3
static COBAYN
dynamic COBAYN
hybrid COBAYN
PGO
OpenTuner
CFR
Figure 6: FuncyTuner provides better performance than all variants of COBAYN(static, dynamic, hybrid), PGO, and OpenTuner.
utilizes an instrumentation run of a target program to collect pro-
file information, such as loop trip counts and indirect function call
targets. The comparison to PGO offers a perspective to evaluate
the trade-off between benefits and complexities for all approaches.
We use -qopenmp -fp-model source -prof-gen -prof-dir/app for an
instrumented compilation (see recommendations for PGO in Intel’s
compiler optimization manual) and then run the programs with
tuning inputs in Table 2. Afterward, the programs are recompiled
with -O3 -qopenmp -fp-model source -prof-use -prof-dir/app .
4.2.2 Observations. We make the following observations from the
experimental results shown in Fig. 6.
6
FuncyTuner ICPP 2019, August 5–8, 2019, Kyoto, Japan
(1) OpenTuner achieves a 4.9% geometric mean speedup on our
benchmark suite and is 4.5% inferior to that of FuncyTuner CFR .
In particular, CFR achieves 12.7% speedup over O3 on AMG while
OpenTuner is only marginally (1.7%) better than O3. We notice that
OpenTuner ’s performance benefit increases very slow after tens of
test iterations.
(2) COBAY Nperforms similar to OpenTuner and is also inferior to
FuncyTunerCFR . Specifically,COBAY N’s static and hybrid models
perform 4.6% and 2.1% (geometric mean) better than the O3 baseline,
respectively, while COBAY N’s dynamic model is worse than the
O3 baseline. In contrast, FuncyTunerCFR improves performance
by 9.4% beyond the O3 baseline. The performance of COBAY N’s
static model is consistent with the findings in the work that pro-
posed by COBAY N[4] and other previous research [1, 6, 10]. These
related works also show that a machine learning-based approach is
able to reduce search overhead but does not perform better than
traditional random search (Random in our paper) when the sample
size is sufficiently large, e.g., 1000 samples. The poor performance
of COBAY N’s dynamic and hybrid models may be attributed to
limited dynamic features, since MICA [12] only works with serial
code while our target benchmarks are parallel.
(3) PGO results in only minor performance improvements relative
to O3 and is not comparable to FuncyTuner CFR . While PGO is
1.8% better than O3 for AMG, it shows little improvement on six
other programs. In fact, PGO instrumentation runs fail for LULESH
and Optewe.
In brief, FuncyTuner CFR delivers significantly better perfor-
mance than state-of-the-art techniques on our modern scientific
simulation codes. It only relies on Caliper light-weight source-code
level instrumentation [5], entailing much less engineering complex-
ities than others, especially COBAY Nand PGO . Such simplicity is
extremely noticeable when one considers the fact that Intel com-
pilers are industry-quality production compilers and have been
tuned for several decades, and COBAY Ndepends on a variety of
large tools, such as Milepost GCC [9] and Mica. Nevertheless, their
performance is inferior to FuncyTuner CFR and their robustness is
limited.
4.3 Impact Of Different Inputs
The over-arching goal of our work is to auto-tune large scientific
simulation codes with a given input. Results in Sec. 4.1 and Sec. 4.2
use the same input as both tuning and test inputs. These inputs
capture the typical sizes of application work sets in practice, hence,
their performance benefits can generalize to other inputs, e.g., for
inputs with the same work-set size but different simulation time-
steps. This is shown in Fig. 8 for Cloverleaf on Broadwell by varying
the number of time-steps as part of the input.
Inputs with different work-set sizes are addressed as follows. We
experimented on Broadwell with two sets of inputs that have dif-
ferent input sizes from those in Table 2. For 351.bwaves, 362.fma3d,
and 363.swim, we use “test” and “ref” as their small and large inputs,
respectively. For LULESH, AMG, Cloverleaf, Optewe, their small
input sizes are 180, 20, 1000, 384, respectively, while their large
input sizes are 250, 30, 4000, 768, respectively.
As shown in Fig. 7, we observe little sensitivity for our bench-
mark applications on their small and large inputs , except that for
351.swim FuncyTunerCFR does not perform as well as the other
three approaches for its small input. Nonetheless,CFR for 351.swim
is still 20.6% better thanPGO and the O3 baseline. We attribute such
performance to the fact that the “test” input is so small that each
time-step takes less than .01 seconds, which significantly differs
from the performance profile of its tuning input in just this one case.
FuncyTuner CFR achieves 5.5%, 9.5% and 10.7% (all in geometric
mean) better performance than OpenTuner , COBAY N, and PGO
on large input, respectively. It is notable that the speedup of AMG
under CFR over the O3 baseline is 22% while the benefit of other
techniques is marginal.
The capability of generalizing performance benefits on tuning
inputs to test inputs justifies the tuning overhead of all approaches,
which is about 1.5 days forRandom/G, 2 days forOpenTuner , 3 days
for CFR and 1 week forCOBAY N, for each benchmark. Specifically,
for CFR ’s target HPC applications, the overhead is amortized in
repetitive production runs. Moreover, the tuning overhead may be
dramatically reduced via various techniques [6, 22] or by exploiting
program-specific CFR convergence trends, i.e., CFR finds the best
code variant in tens or several hundreds of evaluations.
4.4 Deep Dive: Cloverleaf On Broadwell
4.4.1 Case Study Design. To understand why FuncyTunerCFR
performs the best, we selected Cloverleaf to conduct an in-depth
case study on Intel Broadwell. Five hot loops of Cloverleaf are
selected since they have comparatively high per-loop runtime ratios
(see Table 3, others are less than 3.0%) and were found to produce
large performance differences (see Fig. 9) across different tuning
techniques.
To identify performance-critical flags for the bestCV s, we design
an iterative greedy algorithm to eliminate the flags that have low
impact on the program runtime. Each iteration, the algorithm tries
to remove one flag for a given loop’sCV (focused CV ) while keeping
all other CV s intact. If excluding a flag from focused CV does not
degrade program performance, the flag is removed; otherwise, it is
kept for the current iteration. This process is performed iteratively
until no more flags offocused CV can be eliminated. We consider the
remaining flags in focused CV as the critical ones for the given loop.
Note that we only consider the static COBAY Nmodel, because it
is superior to its dynamic and hybrid counterparts for Cloverleaf.
4.4.2 Observations. Fig. 9 shows the per-loop performance
results for the five Cloverleaf hot loops on Broadwell. After
greedy elimination, Random, COBAY N and OpenTuner retain -
qopt-streaming-stores=always -no-ansi-alias -ipo -xCORE-AVX2 as
their critical flags; FuncyTunerCFR retains -no-vec for dt and mom9
but no special flags for the other three loops;G.realized also has no
special flags for mom9. Table 3 reveals which critical optimizations,
such as loop unrolling and vectorization, are exploited by different
algorithms in this case. We make the following observations:
(1) Vectorization is not always profitable. First,cell 3 and cell 7 ex-
perience a 27.7% and 13.6% slowdown, respectively, when 256-bit
vectorization is performed by Random. Other algorithms have simi-
lar performance benefits, yet they do not vectorize. O3 uses 128-bit
SIMD (single instruction multiple data) instructions. Second, even
though Random achieves a 34.8% speedup for dt with 256-bit vec-
torization, the performance is 12.8% worse than a scalar version.
7
ICPP 2019, August 5–8, 2019, Kyoto, Japan Tao Wang and Nikhil Jain, et al.
 0.6
 0.7
 0.8
 0.9
 1
 1.1
 1.2
 1.3
 1.4
LULESH CL AMG Optewe bwaves fma3d swim GM
Speedup normalized to O3
Random G.realized COBAYN PGO OpenT uner CFR
(a) Normalized speedups for small inputs
 0.6
 0.7
 0.8
 0.9
 1
 1.1
 1.2
 1.3
LULESH CL AMG Optewe bwaves fma3d swim GM
Speedup normalized to O3
Random G.realized COBAYN PGO OpenT uner CFR
(b) Normalized speedups for large inputs
Figure 7: CFR shows little performance sensitivity on small and large inputs with geometric mean of speedup relative to the
O3 baseline 12.3% and 10.7% respectively.
 0.6
 0.7
 0.8
 0.9
 1
 1.1
 1.2
 1.3
100 200 400 800 GM
Speedup normalized to O3
Random
G.realized
COBAYN
PGO
OpenT uner
CFR
Figure 8: For Cloverleaf on Broadwell, FuncyTuner CFR pro-
vides a stable performance benefit than all others while scal-
ing from 100 to 800 time-steps.
 0
 0.4
 0.8
 1.2
 1.6
dt cell3 cell7 mom9 acc
Speedup normalized to O3
Random
G.realized
CFR
G.Independent
Figure 9: Normalized speedups for top-5 loops of Cloverleaf
on Intel Broadwell. Note: COBAY N (static), OpenTuner and
Random generate the same code.
Inspection of assembly code shows that there are many data per-
mutations and mask operations to handle control flow divergence,
which are known to degrade vectorization efficiency.
(2) FuncyTuner CFR has more informed freedom (compared to
G, COBAY N and OpenTuner ) to select non-conflicting CV s and
prioritize performant CV s (compared to FR ). For instance, CFR
Table 3: Comparison of optimizations for 5 Cloverleaf ker-
nels on Broadwell. S(Scalar): not vectorized; {128,256}: vector-
ized with {128,256}-bit SIMD; unroll{2,3}: unroll 2/3 times; IO:
instruction reordering; IS: instruction selection; RS: register
spilling.
Algorithm Kernel, O3 runtime ratio %
dt cell3 cell7 mom9 acc
6.3 2.9 3.5 3.5 4.2
G.realized S S S 256 256
IO unroll2 IS, IO
G.Independent S, RS, IO S S S 256, IS
O3 baseline S, unroll2 S S 128 S, unroll3
Random 256 256 256 256, IS 256, IS
CFR S S S S, IS 256
is able to select -no-vec for mom9 to avoid vectorization-induced
slowdown.
(3) G (see data point marked as G.realized ) performs worse
than other algorithms and invalidates the assumption that there
are no inter-module dependencies. Note that G.realized and
G.Independent have the same per-loop CV s. But G.Independent
does not practically assemble an executable while G.realized does.
The comparison between them demonstrates that there are interfer-
ence among different modules. For example, G.realized vectorizes
mom9 with 256-bit AVX2 instructions and further unrolls the vec-
torized loop twice while G.Independent does not.
(4) Other optimizations, such as loop unrolling and instruction se-
lection, also matter, e.g., FuncyTunerCFR and G.Independent both
choose not to vectorize mom9, but their difference in instruction
selection results in better performance for CFR .
8
FuncyTuner ICPP 2019, August 5–8, 2019, Kyoto, Japan
In summary, we observe that such findings are difficult to derive
manually for compiler writers while per-loop auto-tuning with
FuncyTunerCFR is able to capitalize on them.
5 RELATED WORK
The first order objective of compiler-based auto-tuning techniques
is performance, while the second order objectives are code size,
power draw, and energy consumption. We divide prior work into
the following two categories
(1) Compiler flag selection techniques [6, 7, 14, 15, 19, 21, 22]: given
a set of compiler flags, the objective is to determine the combination
that generates the most performant executable on a given architec-
ture. Our work and many related papers [7, 21, 22] belong to this
category. [21, 22] also take a fine-grained per-region approach. They
select the best code variant for each region in a greedy fashion with-
out considering interactions among different code variants. This is is
effective for their case studies but results in worse performance than
random search for OpenMP-based scientific applications. As the
state-of-art search-based auto-tuning technique, OpenTuner [2] co-
ordinates many different search algorithms. Furthermore, to reduce
the overhead of search-based approaches, researchers have also
proposed schemes based on machine learning techniques [1, 4, 6].
As the state-of-the-art, COBAY N [4] infers compiler flags for a
new program by representing them as static/dynamic features to
a pre-trained Bayesian network. Our experimentation shows that
FuncyTunerCFR outperforms both OpenTuner and COBAY Nfor
Intel compilers while incurring similar cost.
(2) Compiler phase ordering techniques [3, 13, 15, 16]: given a set
of compiler optimization passes, there are many valid orders, each
of which may generate different runtime performance. Our work
focuses on the Intel tool chain, which does not provide command-
line flags to perform phase ordering.
6 CONCLUSION
In this work, we presented a fine-grained per-loop compiler flag
selection framework, FuncyTuner, that combines program profiling
and search space focusing algorithms to improve performance of
parallelized scientific programs, in which different code region-
s/loops may be optimized with different flags. Our experimental
evaluation shows that FuncyTuner’s Caliper-guided random search
(CFR) effectively utilizes collected per-loop runtimes to focus the
search on performant program compilation configurations. Fun-
cyTuner achieves a 9.2% to 12.3% (geometric mean, up to 22% for
AMG) performance improvement in comparison to O3 baseline,
outperforms the state-of-art search-based technique OpenTuner ,
machine learning-based approach COBAY Nand PGO of Intel com-
pilers by 4.5% to 5.5%, 4.8% to 9.5%, and 10.7%, respectively. We
also showed that greedily picking the per-loop best compiler flags
often degrades program performance due to complex inter-module
dependencies, and per-function random search without guidance
of runtime information does not guarantee performance improve-
ments.
ACKNOWLEDGMENTS
This work was funded in part by NSF grants 1058779, 1217748
and 1525609, by US Air Force, Office of Scientific Research grants
AFOSR-FA9550-12-1-0442 and AFOSR-FA9550-17-1-0205, and under
the auspices of the U.S. Department of Energy by Sandia National
Laboratories subcontract DOE-1403482 and Lawrence Livermore
National Laboratory (LLNL) under contract DE-AC52-07NA27344.
REFERENCES
[1] F. Agakov, E. Bonilla, J. Cavazos, B. Franke, G. Fursin, M. F. P. O’Boyle, J. Thomson,
M. Toussaint, and C. K. I. Williams. 2006. Using machine learning to focus iterative
optimization. In International Symposium on Code Generation and Optimization
(CGO’06). 11 pp.–. DOI:https://doi.org/10.1109/CGO.2006.37
[2] Jason Ansel, Shoaib Kamil, Kalyan Veeramachaneni, Jonathan Ragan-Kelley, Jef-
frey Bosboom, Una-May O’Reilly, and Saman Amarasinghe. 2014. OpenTuner:
An Extensible Framework for Program Autotuning. In Proceedings of the 23rd In-
ternational Conference on Parallel Architectures and Compilation (PACT ’14) . ACM,
New York, NY, USA, 303–316. DOI:https://doi.org/10.1145/2628071.2628092
[3] Amir H. Ashouri, Andrea Bignoli, Gianluca Palermo, Cristina Silvano, Sameer
Kulkarni, and John Cavazos. 2017. MiCOMP: Mitigating the Compiler Phase-
Ordering Problem Using Optimization Sub-Sequences and Machine Learning.
ACM Trans. Archit. Code Optim. 14, 3, Article 29 (Sept. 2017), 28 pages. DOI:
https://doi.org/10.1145/3124452
[4] Amir Hossein Ashouri, Giovanni Mariani, Gianluca Palermo, Eunjung Park, John
Cavazos, and Cristina Silvano. 2016. COBAYN: Compiler Autotuning Framework
Using Bayesian Networks. ACM Trans. Archit. Code Optim. 13, 2, Article 21 (June
2016), 25 pages. DOI:https://doi.org/10.1145/2928270
[5] D. Boehme, T. Gamblin, D. Beckingsale, P. T. Bremer, A. Gimenez, M. LeGendre, O.
Pearce, and M. Schulz. 2016. Caliper: Performance Introspection for HPC Software
Stacks. In SC16: International Conference for High Performance Computing, Net-
working, Storage and Analysis . 550–560. DOI:https://doi.org/10.1109/SC.2016.46
[6] John Cavazos, Grigori Fursin, Felix Agakov, Edwin Bonilla, Michael F. P. O’Boyle,
and Olivier Temam. 2007. Rapidly Selecting Good Compiler Optimizations Using
Performance Counters. In Proceedings of the International Symposium on Code
Generation and Optimization (CGO ’07) . IEEE Computer Society, Washington,
DC, USA, 185–197. DOI:https://doi.org/10.1109/CGO.2007.32
[7] Yang Chen, Yuanjie Huang, Lieven Eeckhout, Grigori Fursin, Liang Peng, Olivier
Temam, and Chengyong Wu. 2010. Evaluating Iterative Optimization Across 1000
Datasets. In Proceedings of the 31st ACM SIGPLAN Conference on Programming
Language Design and Implementation (PLDI ’10) . ACM, New York, NY, USA,
448–459. DOI:https://doi.org/10.1145/1806596.1806647
[8] Grigori Fursin. 2018. Shared programs, benchmarks and kernels for
autotuning/crowd-tuning. https://github.com/ctuning/ctuning-programs. (May
2018).
[9] Grigori Fursin, Yuriy Kashnikov, Abdul Wahid Memon, Zbigniew Chamski,
Olivier Temam, Mircea Namolaru, Elad Yom-Tov, Bilha Mendelson, Ayal Zaks,
Eric Courtois, Francois Bodin, Phil Barnard, Elton Ashton, Edwin Bonilla, John
Thomson, Christopher K. I. Williams, and Michael O’Boyle. 2011. Milepost GCC:
Machine Learning Enabled Self-tuning Compiler. International Journal of Par-
allel Programming 39, 3 (01 Jun 2011), 296–327. DOI:https://doi.org/10.1007/
s10766-010-0161-2
[10] Grigori Fursin, Abdul Wahid Memon, Christophe Guillon, and Anton Lokhmotov.
2015. Collective Mind, Part II: Towards Performance- and Cost-Aware Software
Engineering as a Natural Science. CoRR abs/1506.06256 (2015). arXiv:1506.06256
http://arxiv.org/abs/1506.06256
[11] Mary Hall, Jacqueline Chame, Chun Chen, Jaewook Shin, Gabe Rudy, and Ma-
lik Murtaza Khan. 2010. Loop Transformation Recipes for Code Generation and
Auto-Tuning. In Languages and Compilers for Parallel Computing , Guang R. Gao,
Lori L. Pollock, John Cavazos, and Xiaoming Li (Eds.). Springer Berlin Heidelberg,
Berlin, Heidelberg, 50–64.
[12] Kenneth Hoste and Lieven Eeckhout. 2007. Microarchitecture-Independent
Workload Characterization. IEEE Micro 27, 3 (May 2007), 63–72. DOI:https:
//doi.org/10.1109/MM.2007.56
[13] M. R. Jantz and P. A. Kulkarni. 2013. Exploiting phase inter-dependencies for
faster iterative compiler optimization phase order searches. In 2013 International
Conference on Compilers, Architecture and Synthesis for Embedded Systems (CASES) .
1–10. DOI:https://doi.org/10.1109/CASES.2013.6662511
[14] T. Kisuki, P.M.W. Knijnenburg, M.F.P. O’Boyle, and H. A. G. Wijshoff. 2000.
Iterative Compilation in Program optimization. (2000).
[15] Prasad Kulkarni, Stephen Hines, Jason Hiser, David Whalley, Jack Davidson, and
Douglas Jones. 2004. Fast Searches for Effective Optimization Phase Sequences.
In Proceedings of the ACM SIGPLAN 2004 Conference on Programming Language
Design and Implementation (PLDI ’04) . ACM, New York, NY, USA, 171–182. DOI:
https://doi.org/10.1145/996841.996863
[16] Sameer Kulkarni and John Cavazos. 2012. Mitigating the Compiler Optimization
Phase-ordering Problem Using Machine Learning. In Proceedings of the ACM
International Conference on Object Oriented Programming Systems Languages
and Applications (OOPSLA ’12) . ACM, New York, NY, USA, 147–162. DOI:https:
9
ICPP 2019, August 5–8, 2019, Kyoto, Japan Tao Wang and Nikhil Jain, et al.
//doi.org/10.1145/2384616.2384628
[17] Lawrence Livermore National Lab. 2018. LLNL Codesign. https://codesign.llnl.
gov. (2018).
[18] Jiajia Li, Guangming Tan, Mingyu Chen, and Ninghui Sun. 2013. SMAT: An Input
Adaptive Auto-tuner for Sparse Matrix-vector Multiplication. In Proceedings
of the 34th ACM SIGPLAN Conference on Programming Language Design and
Implementation (PLDI ’13) . ACM, New York, NY, USA, 117–126. DOI:https://doi.
org/10.1145/2491956.2462181
[19] San-Chih Lin, Chi-Kuang Chang, and Nai-Wei Lin. 2008. Automatic selection
of GCC optimization options using a gene weighted genetic algorithm. In 2008
13th Asia-Pacific Computer Systems Architecture Conference . 1–8. DOI:https:
//doi.org/10.1109/APCSAC.2008.4625477
[20] Ricardo Nobre, Luiz G. A. Martins, and João M. P. Cardoso. 2016. A Graph-based
Iterative Compiler Pass Selection and Phase Ordering Approach. In Proceedings
of the 17th ACM SIGPLAN/SIGBED Conference on Languages, Compilers, Tools, and
Theory for Embedded Systems (LCTES 2016) . ACM, New York, NY, USA, 21–30.
DOI:https://doi.org/10.1145/2907950.2907959
[21] Zhelong Pan and Rudolf Eigenmann. 2008. PEAK: a Fast and Effective Per-
formance Tuning System via Compiler Optimization Orchestration. ACM
Trans. Program. Lang. Syst. 30, 3, Article 17 (May 2008), 43 pages. DOI:https:
//doi.org/10.1145/1353445.1353451
[22] Mihail Popov, Chadi Akel, William Jalby, and Pablo de Oliveira Castro. 2016.
Piecewise Holistic Autotuning of Compiler and Runtime Parameters. In Euro-Par
2016 Parallel Processing - 22nd International Conference (Lecture Notes in Computer
Science), Christos Kaklamanis, Theodore S. Papatheodorou, and Paul G. Spirakis
(Eds.), Vol. 9833. 238–250.
[23] Mohammed Sourouri. 2018. Optewe. https://github.com/mohamso/optewe. (April
2018).
[24] UK-MAC. 2018. Cloverleaf. http://uk-mac.github.io/CloverLeaf/. (April 2018).
[25] Qing Yi. 2012. POET: A Scripting Language for Applying Parameterized Source-
to-source Program Transformations.Softw. Pract. Exper. 42, 6 (June 2012), 675–706.
DOI:https://doi.org/10.1002/spe.1089
10

Compiler-Based Graph Representations for Deep
Learning Models of Code
Alexander Brauckmann
TU Dresden
Dresden, Germany
alexander.brauckmann@tu-dresden.de
Andrés Goens
TU Dresden
Dresden, Germany
andres.goens@tu-dresden.de
Sebastian Ertel
Barkhausen Institut
Dresden, Germany
sebastian.ertel@barkhauseninstitut.org
Jeronimo Castrillon
TU Dresden
Dresden, Germany
jeronimo.castrillon@tu-dresden.de
Abstract
In natural language processing, novel methods in deep learn-
ing, like recurrent neural networks (RNNs) on sequences of
words, have been very successful. In contrast to natural lan-
guages, programming languages usually have a well-defined
structure. With this structure compilers can reason about
programs, using graphs such as abstract syntax trees (ASTs)
or control-data flow graphs (CDFGs). In this paper, we ar-
gue that we should use these graph structures instead of
sequences for learning compiler optimization tasks. To this
end, we use graph neural networks (GNNs) for learning pre-
dictive compiler tasks on two representations based on ASTs
and CDFGs. Experiments show that this improves upon the
state-of-the-art in the task of heterogeneous OpenCL map-
ping, while providing orders of magnitude faster inference
times, crucial for compiler optimizations. When testing on
benchmark suites not included for training, our AST-based
model significantly outperforms the state-of-the-art by over
12 percentage points in terms of accuracy. It is the only one
to perform clearly better than a random mapping. On the
task of predicting thread coarsening factors, we show that
all of the methods fail to produce an overall speedup.
CCS Concepts · Software and its engineering → Com-
pilers
; · Computing methodologies → Neural networks;
Natural language processing; Graphics processors .
Keywords Deep Learning, Compilers, Graphs, LLVM
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for components of this work owned by others than the author(s) must
be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee. Request permissions from permissions@acm.org.
CC ’20, February 22ś23, 2020, San Diego, CA, USA
© 2020 Copyright held by the owner/author(s). Publication rights licensed
to ACM.
ACM ISBN 978-1-4503-7120-9/20/02. . . $15.00
https://doi.org/10.1145/3377555.3377894
ACM Reference Format:
Alexander Brauckmann, Andrés Goens, Sebastian Ertel, and Jeron-
imo Castrillon. 2020. Compiler-Based Graph Representations for
Deep Learning Models of Code. In Proceedings of the 29th Interna-
tional Conference on Compiler Construction (CC ’20), February 22ś23,
2020, San Diego, CA, USA. ACM, New York, NY, USA, 11 pages.
https://doi.org/10.1145/3377555.3377894
1 Introduction
The last decade has seen tremendous improvements in ma-
chine learning, especially due to deep learning methods,
which do not rely on manually-specified features of the data
to be learned, but are able to learn what features are impor-
tant on their own. Deep learning methods have revolution-
ized several fields like image recognition or natural language
processing. While progress has been made in compiler opti-
mization and tasks related to programming languages, deep
learning still plays a comparably modest role in these fields.
Most approaches to deep learning in compiler optimiza-
tion [ 21] borrow ideas from the successful deep learning
methods in natural language processing. This is very reason-
able, as the similarities between natural languages and pro-
gramming languages have fueled cross-fertilization of these
two fields for many years. Prominent examples feature Long
Short Term Memory (LSTM) architectures on sequences of to-
kens [ 5], or learning vector embeddings for instructions [ 3],
in analogy to the successful word2vec method [16], also based
on sequence models and LSTM. A comprehensive survey can
be found in [
1]. However, the nature of the analysis required
for compiler optimization has shown to require very spe-
cific structural properties, which might be neglected by the
sequential nature of these methods. For example, the polyhe-
dral model looks into the geometric structure of the lattices
of array indices to find optimal re-orderings of nested loops.
This model has proven to be very successful in optimizing
such nested loops [ 20]. More generally, compilers base most
analysis on non-sequential data structures like abstract syn-
tax trees (ASTs) and control- and dataflow graphs (CDFGs).
We argue that these kinds of structures are very different
201

CC ’20, February 22–23, 2020, San Diego, CA, USA A. Brauckmann, A. Goens, S. Ertel, and J. Castrillon
Token
sequence
Abstract Syntax
Tree (AST)
Control-Data-Flow-
Graph (CFDG)
Sequence
Model
Graph
Model
DeepTune [5] inst2vec [3]
This work
Figure 1. Representations of code in the compiler with their
respective machine learning models.
from the sequences of words that compose natural language.
Data dependencies can be crucial on parts of the code that
are far apart in the code string, and details in a small part of
the source, like the precise array indices in loops, can have a
tremendous impact on some optimizations. For this reason,
focusing on the similarities to natural language processing
for deep learning in compilers might be ill-advised.
In this paper, we propose to re-evaluate the representa-
tions of code we use in deep learning for compilers. We
believe that decades of research in the compiler domain have
identified data structures (cf. Figure
1) that expose the aspects
important for optimization better than the token sequences
that are analogous to word sequences in natural languages.
We thus propose to use graph-based neural networks, to ex-
plicitly capture these graph representations of code. Graph
neural networks have shown to be significantly better for
reasoning tasks, even when these are formulated in natural
language terms [ 9]. We believe these reasoning tasks, like
inference from a series of logical statements, structurally
resemble the kind of analysis required for compiler opti-
mization. We study two concrete graph-based architectures
(Section 4), using ASTs or CDFGs as input (Section 3). We
compare them with traditional methods on two complex
tasks (Section 5), deciding whether OpenCL kernels should
execute on a CPU or a GPU, and predicting optimal thread
coarsening factors. With orders of magnitude smaller infer-
ence times, our methods clearly outperform two different
state-of-the-art models ([ 5, 16]) in this task, with the CDFG
model yielding the best accuracy overall. Additionally, we
consider an alternative experimental setup to test aspects
about the generality of the learned heuristics. Instead of ran-
domly choosing training and test kernels from the available
pool, we make sure these sets feature kernels from disjoint
benchmark suites. With this setup, we show that our AST-
based model is the only one that clearly outperforms the
equivalent of a coin-toss to decide the kernel mapping when
generalizing across benchmark suites.
Finally, We also test on a second task, namely predicting
thread coarsening factors. With much less data and only
modest opportunities for speedup, this task showcases the
current limitations of deep learning in compilers. Our graph-
based models do not outperform the state-of-the-art. How-
ever, none of the considered models are able to produce an
overall speedup ( > 1) with their predictions.
2 Related Work
Very recently, the deep learning model class of Graph Neural
Networks (GNNs) has emerged, which enables the extrac-
tion of task-specific features from graph-structured data. In
many tasks where information can be represented as a graph,
GNN-based models have proven to be superior to sequence-
based models that operate on a serialized representation of
the graph [ 10, 13]. By applying various established architec-
tures for sequences and euclidian data to the GNN model,
recurrence- and convolutional-based variants of the GNN
have been proposed and shown to support integration into
complex architectures to achieve discriminative and genera-
tive tasks where they are part of an end-to-end optimization
process [22].
A recent method has combined graph neural networks
and code representation learning for node-level prediction
tasks in the domain of software-engineering. In a variable
name- and misuse prediction task based on a AST-graph
representation, they have shown to outperform recurrent
neural networks [ 2].
Probabilistic models on source code have successfully been
used for solving various tasks on source code in the research
area of compiler optimization [ 21]. While the optimizations
themselves are performed with formal, algorithmic meth-
ods, probabilistic models are used for selecting and tuning
them to achieve better performance [ 6]. Also in adjacent
research areas, probabilistic models have solved tasks where
statistical properties of source code are desirable, e.g. auto-
completion [ 19] and variable naming [ 18]. Similarly, they
have been used to provide approximations for problems that
due to their enormous complexity are hardly solvable with
formal methods, such as software defect prediction [ 14] and
tuning of compiler heuristics [ 3, 5, 15, 17], albeit not without
caveats [7].
Early models for predicting compiler optimizations rely
on features that are manually designed by experts [ 6, 15, 17]
and automatically selected from a set of candidates [ 11].
Those features are typically quantities of instructions in
compiler intermediate representations (IRs). To extract fea-
tures that capture the complex dependencies, several models
relying on deep learning [
12] have been proposed: Cummins
et al [ 5] train an LSTM recurrent neural network model to
predict compiler-internal optimization heuristics based on
a program represented as a raw sequence of C program-
ming language tokens. Ben-nun et al [
3] adopt the notion
of the distributional hypothesis, which is commonly known
as word2vec [ 16] to produce embeddings, i.e. mappings to
real vectors, on a graphical representation based on LLVM
IR. This results in a closeness of the embeddings for code
that shares similar contexts in the large body of open-source
software projects that the embeddings have been trained
on. Subsequently, the embeddings are used in more com-
plex downstream tasks as input to an LSTM recurrent neural
202

Compiler-Based Graph Representations for Deep Learning ... CC ’20, February 22–23, 2020, San Diego, CA, USA
network model. However, the semantic information of the
compiler IR is only being used to produce the embeddings
in the first phase, and not part of the optimization of the
downstream task. Therefore, the embeddings are not opti-
mized for the specific task. Figure 1 shows how these models
compare conceptually to our work.
To the best of our knowledge, no machine learning model
that uses the semantic information of compiler-internal anal-
yses in an end-to-end setting has been studied.
3 Compiler-Based Representations
A compiler typically uses several representations of code
at different stages in order to enable different analyses and
optimizations. In this section, we will discuss three of the
most common representations at the different stages and
how we expect their properties to affect machine learning
tasks.
Consider the following OpenCL kernel as a running ex-
ample for this section:
__kernel
void Add(__global const int* x,
__global const int* y,
__global int* z, const int d) {
const int id = get_global_id(0);
if (id < d)
z[id] = x[id] + y[id];
}
The code for this kernel contains very obvious informa-
tion, like the numbers and types of input variables for the
kernel, but also very subtle information, like the indentation
style of the code, which while semantically irrelevant for
the compiler, can say something about the person writing
the program. In this paper, we are taking the point of view
of the compiler, and thus will not take into consideration
such information that does not change the semantics of the
program, like the indentation style, or more importantly, the
identifiers in the code. Thus, we will consider code that is
normalized for identifiers, using the methodology described
in [5]. It standardizes all identifiers and removes whitespaces.
The kernel above becomes:
__kernel void A(__global const int* a,
__global const int* b, __global int* c,
const int d) {const int e = get_global_id
(0); if (e < d) c[e] = a[e] + b[e];}
When compiling this kernel, a typical compiler 1 will trans-
form this input string of character into multiple different
representations at various stages, namely as a sequence of
tokens, an abstract syntax tree, and a control and dataflow
graph. We will consider these three representations and em-
phasize the focus for machine learning for code, as the rep-
resentation is what permits the ML model to gather the rele-
vant information. If a piece of information is not included
1in this work, concretely, we use LLVM and Clang in version 7.1.0
__kernel
 void
 A
 (
 __global
 const
 int*
 a
 ,
__global
 const
 int*
 b
 __global
 int*
 c
 ,
const
 int
 d
 )
 {
 const
 int
 e
 =
 get_global_id
(
 0
 )
 ;
 if
 (
 e
 <
 d
 )
 c
 [
 e
 ]
=
 a
 [
 e
 ]
 +
 b
 [
 e
 ]
 ;
 }
Figure 2. An example kernel as a sequence of tokens.
in a representation, it cannot be learned. On the other hand,
if a particular kind of information is very prominent in a
representation, an ML model is more likely to learn patterns
based on this information.
3.1 Sequence of Tokens
Conceptually, a compiler first turns a string of characters into
a sequence of tokens. Figure
2 shows the example kernel as a
sequence of tokens. Boxes are used to represent tokens in the
figure, and arrows point to the next token in the sequence.
This illustrates both the sequential nature of a token string, in
particular as its structural similarities to strings of characters.
In terms of machine learning, this sequential nature can
be an advantage and a drawback at the same time. Data
organized in a sequential manner has been well-studied in
ML, and powerful methods exist that yield good results on
sequential data. The ordering of the sequence of tokens also
faithfully represents the ordering of the code as written by
the programmer. This ordering contains information on the
thought process of the programmer, which might be valuable
for analyzing and characterizing code. However, the ordering
may overconstrain the representation, enforcing an artificial
order on code fragments that could otherwise execute in a
different order.
3.2 Abstract Syntax-Tree
After parsing, an abstract syntax tree (AST) on its own loses
some information. In particular, if an AST does not include
the identifier strings, then it is impossible to tell when two
identifiers refer to the same data. To deal with this, instead of
embedding the identifiers, we enhance the AST with (labeled)
dataflow edges , which connect two nodes in the AST that
refer to the same data.
Figure 3 shows our example kernel as an AST. We can
see how the first two arguments to the kernel ( x and y in
the original code) are indistinguishable in the AST. Because
addition is commutative, swapping these two arguments
yields the same result. In the token sequence, because tokens
have to have an order, this symmetry is broken by imposing
x as coming before y. The AST, on the other hand, removes
this synthetic dependency: swapping the arguments is an
isomorphism of the AST and our dataflow-enhanced version.
This is an example of how the AST abstracts away structure
that is not syntactically relevant, otherwise imposed by the
203

CC ’20, February 22–23, 2020, San Diego, CA, USA A. Brauckmann, A. Goens, S. Ertel, and J. Castrillon
Function
type: void
FunctionArg
type: int*
FunctionArg
type: int*
FunctionArg
type: int*
FunctionArg
type: int
CompoundStmt
DeclStmt
type: int
BinaryOperator
operator: =
IfStmt
CallExpr
function_name: get_global_id
BinaryOperator
operator: <
IntegerLiteral
value: 0
ArraySubscriptExpr
BinaryOperator
operator: +
ArraySubscriptExpr
 ArraySubscriptExpr
Figure 3. An example kernel as an AST with overlayed
dataflow edges (in blue).
sequential nature of strings. As discussed before, for machine
learning, this difference can be an advantage or a drawback,
e.g. as the ordering might indirectly encode some of the
thought processes of the programmer as well. The structure
of an AST reflects the grammar of the language, and it would
be plausible to think that this exposes the semantics of the
code more directly, although it is not obvious that it should.
Based on the AST, we define the following representation:
Definition 1 (AST+DF).
A dataflow-enriched AST graph is a
labeled digraph, where nodes are labeled as Declarations, State-
ments, and Types as in the Clang AST. The edges are labeled
either as type AST, representing child-of relationships within
the AST, and dataflow, representing use use-def relationships
for variables.
To create dataflow-enriched AST graphs, we first extract
the raw Clang AST using a tool implemented in the Clang
library. As a second step, we reduce the graph by eliminating
regular nodes that don’t provide additional, structure-critical
information. A small graph diameter is generally a desirable
property because the information has a limited outreach in
the propagation scheme of the graph model introduced in sec-
tion
4. Specifically, we eliminate nodes of type DeclRefExpr
and ImplicitCastExpr by merging them with their AST-
edge successors.
3.3 Control- and Dataflow Graph
Finally, the control and data flow graph (CDFG). This graph
organizes the statements in the code not by their grammar,
but by the semantics of the possible flow of control in the
program. The basic structure of the CDFG is that of a graph.
This graph will usually contain cycles, e.g. if the code con-
tains loops. Similar to the AST+DF, we define the following
representation:
fn_0
bb_0
bb_1
bb_2
alloca
alloca
store
alloca
store
alloca
store
alloca
store
load
store
load
call
get_global_idtrunc
icmp
br
ret
load
load
getelementptr
sext
load
load
add
load
getelementptr
sext
load
load
store
load
getelementptr
sext
br
bb_2
store
store
store
load
store
load
call
get_global_idtrunc
icmp
br
load
load
getelementptr
sext
load
load
Figure 4. An Example Kernel as a CDFG. The control flow
is depicted by black arrows, the data flow by green ones, the
red arrow represents an external function call.
Definition 2 (CDFG+CALL+MEM).
A LLVM-based control-
and dataflow graph enrichted with calls and memory nodes is
a labeled digraph, where the nodes are labeled with LLVM IR
instructions. The edges of the type control- and dataflow define
the basis of the CDFG. The dataflow edges represent opera-
tor relationships within the LLVM IR. CALL edges are added
based on dependencies to return values of functions. MEM edges
represent store-load dependencies to specific memory locations.
We create the CDFG by using a custom LLVM pass that
extracts all LLVM IR instructions, the operands, and the
memory accesses. We disable all optimization passes by using
optimization level 0 before running our pass.
Figure 4 shows the CDFG representation of the example
kernel. Note that the statements in the figure are not in C,
but in LLVM IR instead. As can be seen in the figure, this
representation includes many new low-level operations, like
memory management. It has a different structure, closer to
how the calculation is executed than to what is calculated.
Thus, this is the representation that is closest to the execution
semantics of the program and furthest away from how the
programmer expressed a computation. Again, for machine-
learning, this poses a trade-off: while exposing the semantics
more directly might be beneficial to learn said semantics,
it may hinder learning subtleties that are encoded in the
human components of a piece of code.
4 Machine Learning Architectures
In order to leverage the compiler-based representations out-
lined in Section 3, we need deep learning models appropriate
to their structure. In this section, we introduce a concrete
204

Compiler-Based Graph Representations for Deep Learning ... CC ’20, February 22–23, 2020, San Diego, CA, USA
architecture to leverage the CDFG and AST representations.
As this paper focuses on predictive models, we first explain
some core principles for ML-driven predictive models in
compiler tasks. After that, we dive into the specifics of the
ML architecture required to leverage graph models in the
context of predictive models.
4.1 Predictive Models
Predictive models revolve around a simple principle: based
on some input data X , predict some output data Y . In the
context of compiler optimizations, the input data X is the
program code, and perhaps additional information about
the task. For example, the task of deciding whether a kernel
should be executed on a CPU or a GPU depends on the
problem size, which is not part of the kernel’s code, and
should be considered additionally as input, besides the code.
Source code
Source code
Source code
Source code
Preprocessing
Code representation
Code representation
Code representation
Code representation
Deep Learning model
Result
Result
Result
Result
Auxiliary inputs
Auxiliary inputs
Auxiliary inputs
Auxiliary inputs Optional
e.g. work group size, data size
OpenCL
Source rewriter
Clang, LLVM
Token
AST, CDFG
LSTM
GNN
Device mapping
Thread coarsening factor
Figure 5. High-level overview of the predictive task.
An end-to-end optimization flow for code-based tasks was
proposed in [ 5], and we base our proposal on it by replacing
the representation-specific part with a graph-based deep
learning model. Figure
5 depicts the general flow within
these predictive models. As can be seen, the input source code
is transformed to the appropriate representation for machine
learning, using well-established compiler techniques. This
includes, for instance, the code normalization discussed in
Section 3, the generation of a token string, the syntax analysis
producing an (enriched) AST, or a pass creating the CDFG.
The output of the flow depends on the concrete predictive
task. It could be a simple binary decision like CPU or GPU,
or a transformed version of the input program. In this paper
we restrict the output to a choice of n possible outcomes,
e.g. hardware mappings ( {CPU,GPU}) or thread coarsening
factors ( {2,4,...,32}).
For the deep learning component in sequential-models,
as the token-based model used in [ 5], well-known recurrent
neural network architectures like LSTM [ 8] can be used as
representation models. For graph-based models, however,
we need to replace the ML architecture used in the flow. In
the following, we describe the architecture of a deep learning
component of this flow based on graph neural networks. We
use this architecture to perform predictive tasks on the AST
and CDFG structure.
4.2 Graph Neural Networks
Figure 6 illustrates the deep learning part of our predictive
model for graphs with its corresponding components and
their relationships. As mentioned above, we base our pro-
posed architecture on the model described in [ 5] and extend
it with the GNN proposed in [ 13]. Our architecture consists
of an embedding layer for creating initial embeddings, a
propagation layer for enriching the initial embeddings with
structural information, and a prediction layer that aggregates
the propagated embeddings and performs a prediction.
The input structure to the model is a labeled graph G =
(V ,E). The output of the model is an n-sized vector repre-
senting a probability distribution, with n being the number
of classes. A node embedding vector hv ∈ Rd is assigned to
all v ∈ V , where d ∈ N is its dimension.
We transform the graph representations of code into the
input structure of the model by generating annotated types,
considering the graphs of the whole dataset. For the AST-
based representation, node types are tuples of (Clang AST
node type, property ), whereas considered properties are
• the data type for Clang AST nodes of type Function,
FunctionArg, and DeclStmt,
• the operator for Clang AST nodes of type
BinaryOperator,
• and function names for Clang AST nodes of type
CallExpr.
For the CDFG graph, node types are directly mappable to
the LLVM IR node types. Additionally, we consider function
names as node types.
Each distinct tuple of the AST graph, or each node type
for the CDFG graph respectively, results in a different node
type vt ∈ Nin the scope of the graph model. For each edge,
we also add an edge with the reverse direction, in order to
improve the propagation capabilities of the model.
Initial Embedding Layer
Nodes from the input graph are
represented as one-hot encoded vectors, i.e. vectors (ej )i =
δi,j ,where δi,j is the Kronecker Delta which is 1 iff i = j and
0 otherwise. Since the number of possible node types tends
to be very large in practice, we introduce this embedding
layer to reduce the dimension of these one-hot encoded
node vectors, to the smaller size
d ∈ N described above.
The node embedding vectors hv are computed by applying
a learnable function finit(vt ) to the node annotation vector
vt . The learnable function is implemented as a Multi-Layer
Perceptron (MLP) neural network.
205

CC ’20, February 22–23, 2020, San Diego, CA, USA A. Brauckmann, A. Goens, S. Ertel, and J. Castrillon
Code representation
Deep Learning model
Code representation graph
Code representation graph
Code representation graph
Code representation graph
Preprocessing
Nodes
Nodes
Nodes
Nodes
Initial embedding layer
Edges
Edges
Edges
Edges
Embedding propagation layer
Auxiliary inputs
Auxiliary inputs
Auxiliary inputs
Auxiliary inputs
Prediction layer
Results
hv
h′v
h′v
aux
Figure 6. An overview of the predictive model architecture
based on graph neural networks.
hv = finit(vt ).
Embedding Propagation Layer With the initial encod-
ings based only on the node type, the encoding does not con-
tain any structural information of the graph, i.e. as expressed
by the edges. By applying an iterative information propaga-
tion scheme, the initial node embedding vectors produced
by the embedding layer are enriched with this structural
graph information. For a fixed number of iteration steps T ,
nodes propagate their embeddings to their directly adjacent
neighbors and eventually result in embeddings that contain
information about a T -sized neighborhood. This is illustrated
in Figure 7, where the colored cells represent the information
on the nodes encoded in the embedding vectors.
In each iteration, messages are formed for each node and
each edge by using a learnable function fmsg(hv ,et ) with the
embedding information hv and the edge type information et .
It consists of learnable parameters A and b for each edge type
et . When all messages of an iteration have been passed to
their target nodes, they are aggregated per target node and
new intermediate node embedding vectors h′
v are formed
by applying a learnable function fprop(av ,hv ). This function
is implemented as a Gated Recurrent Unit (GRU), a variant
of a recurrent neural network with similar performance to
an LSTM, but with less learnable parameters [ 4]. After T
iterations of the scheme, the node embedding vectors reach
a final state. As depicted in the figure, they are then aggre-
gated to a single graph embedding hG , which happens in the
prediction layer.
a) v1 v2
v3
v4 v5
hv
δi,j
b)
hv h′
v h′′
v
fprop fprop
c)
h′′
v
+
hG
fprop
дG
v ⊙ hG
v
Figure 7. An illustration of a) encoding the initial node
embeddings, b) the embedding propagation scheme with 2
iterations, and c) the aggregation to a graph embedding.
fmsg(hv ,et ) =Aet · hv +bet ,
av =
/summationdisplay.1
u:(u,v)∈E
fmsg(hv ,et ), h′
v = fprop(av ,hv ) ∀v ∈ V .
Prediction Layer This layer maps the final node embed-
ding vectors to a probability distribution. To achieve this, a
fixed-size graph embedding vector is computed by aggregat-
ing the final node embedding vectors and then mapping the
graph embedding vector to the output.
The aggregation is implemented by summing all final node
embedding vectors, after mapping them to a higher size
with the learnable functions fm(hv ) and дm(hv ). An atten-
tion mechanism, implemented by the element-wise product
fm(hv )⊙ дm(hv ) of fm(hv ) and дm(hv ), decides the relevance
of individual nodes in the current task. The functions are
implemented as MLPs with hyperbolic tangent and sigmoid
activation. The sigmoid activation function is a natural fit for
the attention mechanism because it outputs a range of (0,1)
that is multiplied element-wise with the output vector of
fm(hv ). The hyperbolic tangent function on the other hand,
normalizes the values to a range of (−1,1) within the model,
yielding better performance.
A final learnable function fout (hG ) computes the output
based on the graph embedding vector hG that is optionally
concatenated with the auxiliary inputs aux. It is implemented
as a MLP with a softmax activation function.
hG
v = fm(hv ), дG
v =дm(hv ),
hG =
/summationdisplay.1
v ∈V
дG
v ⊙ hG
v , out = fout(hG ,aux).
206

Compiler-Based Graph Representations for Deep Learning ... CC ’20, February 22–23, 2020, San Diego, CA, USA
5 Evaluation
In this section, we present an evaluation of our methods
using the different representations of code. 2 We do this by
using two different complex tasks on OpenCL kernels. These
tasks require an understanding of the full kernel, how dif-
ferent parts of it interact and the subtle interplay of them
with the memory hierarchy and SIMD capabilities of the
different target platforms. As such, the tasks serve as a good
first evaluation of the representations, as we expect only
a representation capable of capturing these intricacies to
perform well on them.
For both tasks, we reproduce 3 the results of [ 5] and [ 3]
and integrate our method into the experimental setup. In
this setup, the dataset is split into
k parts which are used
to make different training and testing sets in a k-fold cross-
validation scheme. In this scheme, the model is iteratively
trained anew using k − 1 parts of the dataset as training data
and is subsequently tested on the one part that was left out
while training. This is repeated for all k − 1,1 partitions.
In these experiments, we observe a variance in the results
for all models, as the initialization of their weights is random
and the model converges to different minima. Therefore, we
repeat the execution of each experiment 10 times and report
aggregated results.
5.1 CPU/GPU Mapping
The problem of CPU/GPU mapping considers OpenCL ker-
nels and has the goal of classifying them as CPU or GPU,
depending on where they can be expected to run faster.
This problem has been studied extensively, and several ap-
proaches have been proposed to solve it. For example, Grewe
et al. [
17] propose a heuristic based on a decision tree to solve
this. Cummins et al. [ 5], on the other hand, used a machine
learning approach based on an LSTM and a token representa-
tion to improve upon this. More recently, Ben-Nun et al. [ 3]
used a model based on finding good embeddings for LLVM
instructions to improve this problem further. We compare
our compiler-graph-based models to these approaches.
For evaluating the models, we want to asses their general-
ization performance. In the tasks we use the accuracy as a
metric, which is the ratio of the correctly predicted device
mappings over all predicted device mappings. As an addi-
tional metric for the performance benefit that the methods
bring we use the speedup. A correctly predicted mapping
of a sample yields a faster execution, resulting in a speedup
over a static mapping. In a static mapping, a single platform
(CPU/GPU) is selected and all kernels are mapped to this
platform. To select, the platform which is fastest in most
of the training samples for all test benchmarks is chosen.
2Artifact available at https://github.com/tud-ccc/learning-compiler-graphs
3In fact, the results of [ 5] we show are better than the ones originally
published. We could reproduce this behavior in the published artifact of
this original work as well, obtaining better results there too when loading
the published models.
Additionally, we compare the models in terms of the number
of trainable parameters, which describe their sizes, as well
as their training and inference time. Inference time is partic-
ularly important for compiler-related tasks, as it translates
to a faster compilation time for an end-user.
Experimental Setup
The dataset consists of the seven
benchmark suites AMD SDK, NPB, NVIDIA SDK, Parboil,
Polybench, Rodinia, and SHOC along with the execution
times for both CPU and GPU on two different heterogeneous
systems, one with an AMD Tahi 7970 GPU and one with an
NVIDIA GTX 970 GPU.
The construction of the code representation graphs for
the whole dataset leads to 92 node types for the GNN model
using AST+DF (here, GNN-AST) and 140 node types for the
GNN model using CDFG+CALL+MEM (here, GNN-CDFG).
To compute the initial node embeddings, we map the one-
hot encoded vectors with an MLP finit consisting of 2 hidden
layers of size 64 to vectors of size 32. As propagation size
T , we chose 4 iterations, which yields embeddings that in-
clude a 4-neighborhood of the individual nodes. The graph
embedding vector hG of size 64 is created by aggregating
the node embeddings using the two MLPs fm and дm, which
are dimensioned with 2 hidden layers of size 64. The graph
embedding is mapped with an MLP with two hidden layers
of size 32 to a dimension of 32, then serves as input to the
prediction model used in [ 5] and [ 3], which is a MLP with 1
hidden layer of size 32.
We compare both of our methods to the state of the art
methods DeepTune [ 5] and inst2vec [ 3]. We also compare it
to the decision-tree-based model of Grewe et al [ 17]. Addi-
tionally, we compare it to the static mapping model, and to
a model choosing CPU or GPU at random.
Table 1 gives a comparison of the network sizes, in terms
of the number of trainable parameters, as well as the training
and inference times. We can see that Grewe et al’s model
is considerably faster in training and inference and has the
least amount of trainable parameters. The deep learning
models are larger in size by several orders of magnitude. It
is important to note that the AST-based model is an order of
magnitude faster in inference than the other deep-learning
based models.
0.00
0.25
0.50
0.75
AMD Tahiti 7970 NVIDIA GTX 970
Platform
Accuracy
Model
Random mapping
Static mapping
Grewe et al.
DeepTune
inst2vec
GNN-CDFG
GNN-AST
Figure 8. Accuracy results compared to state-of-the-art.
207

CC ’20, February 22–23, 2020, San Diego, CA, USA A. Brauckmann, A. Goens, S. Ertel, and J. Castrillon
Table 1. Model sizes for the device mapping task
Model Code
Representation
Model
architecture
No. of learnable
parameters Training time Inference time
Grewe et al. [ 17] Manual features Decision tree 14 1 .02 × 10−3 s 8 .46 × 10−5 s
DeepTune [5] C tokens Recurrent neural network 76908 6 .72 × 102 s 9 .02 × 10−1 s
inst2vec [3] LLVM-IR tokens Recurrent neural network 649372 1 .08 × 103 s 1 .34 s
GNN-AST AST+DF Graph neural network 139846 8 .40 × 102 s 9 .95 × 10−2 s
GNN-CDFG CDFG+CALL+MEM Graph neural network 89798 1 .59 × 103 s 9 .8 × 10−1 s
1.5
1.61
1.39
1.56
1.24
0.84
1.251.29
1.21.27
1.11
0.790.8
0.9
1.0
AMD Tahiti 7970 NVIDIA GTX 970
Platform
Speedup (log)
a
a
a
a
a
a
Random mapping
Grewe et al.
DeepTune
inst2vec
GNN-CDFG
GNN-AST
Figure 9. Speedups (geometric mean) compared to state-of-
the-art.
Figure 8 shows the overall accuracy results on the different
benchmarks. We see how our graph-based models perform
better than inst2vec, albeit modestly. While DeepTune pro-
duces a slightly better accuracy than GNN-AST, GNN-CDFG
yields the highest accuracy overall. Similarly, Figure 9 shows
the speedups (geometric mean over all executions) 4. We
see how the trends of the speedup results are comparable to
those seen from the accuracy results, which is to be expected
as the two metrics are linked by their definitions.
While useful for comparison with the state-of-the-art, the
setup used by [ 3, 5] trains the model with kernels from the
same benchmarks that it then uses to evaluate the heuristic.
Having 7 different benchmark suites, we believe that using
these as groups in a k-groups-split methodology yields more
insight about the generalization capabilities of the models.
This way, they are tested on kernels from a benchmark suite
they have not been trained on. To this end we split the dataset
into 7 parts, each of the 7 parts being the different bench-
marks suites, instead of 10 randomly-chosen parts out of
the set of all kernels of the benchmark suites. This way, the
model is tested on data that is fundamentally different than
the benchmarks its was trained on.
Figure 10 shows the results of the experiment with this
alternative setup. Since we split the train and test sets by
benchmarks, we can see how the models fare on every bench-
mark after being trained on the other six. It is notable how
the different methods can have vastly different results on
4Our reported values differ from the original publications of the baseline
methods, because we aggregate using the geometric mean instead of the
arithmetic mean.
the different benchmarks. The final entries for each platform
show an aggregated result over all benchmarks (artithmetic
mean).
We can see that GNN-AST is not only the one with the best
overall results, but also the most consistent ones. It had an
overall accuracy of 60 .6%, which is over 12 percentage points
better than that of DeepTune at 47 .9% and even better than
the 44 .9% overall accuracy obtained by inst2vec. In this case,
GNN-CDFG had a similar performance, beating DeepTune
by less than a percentage point in accuracy at 48 .5%. In
fact, we see how when tested on different benchmark suites
than they were trained on, all state-of-the-art methods we
compared to here perform worse than a coin-toss (50 .9%
accuracy). This indicates that the models probably do not
learn the relationship between the code’s semantics and the
optimal compute device in a way that is generalizable when
the code becomes different enough.
5.2 Thread Coarsening
In parallel architectures, faster executions can be achieved
by merging multiple parallel threads in certain situations.
The thread coarsening factor is a parameter that controls this
behavior in OpenCL. Various predictive models have been
proposed to solve this task, such as that by Magni et al [ 15],
which uses an MLP neural network based on static code
features, such as the quantities of certain instructions on the
LLVM IR. More recently, Cummins et al. [ 5] and Ben-Nun
et al. [ 3] proposed to use LSTM-based deep learning models
based on a source- and LLVM IR-level sequence of tokens.
The dataset for this problem consists of 17 selected kernels
from the AMD SDK, NVIDIA SDK, and Parboil benchmark
suites. As output, the flow decides among 6 classes for every
kernel, corresponding to coarsening factors of 1, 2, 4, 8, 16, 32.
We reproduced these models and the corresponding results
and used these as baseline to compare to our methods.
For this dataset we get a total of 46 distinct node types
for GNN-AST and 54 node types for GNN-CDFG. In this
task, we keep the dimensions of the model at a minimum, as
the amount of training data is quite slim. We apply one-hot
encoding to the nodes represented as node types and map
the resulting vectors with the MLP
finit to a size of 4. This
MLP only contains the input and output layers and no hidden
layers. After 4 propagation time steps T , we aggregate the
208

Compiler-Based Graph Representations for Deep Learning ... CC ’20, February 22–23, 2020, San Diego, CA, USA
Table 2. Model sizes for the thread coarsening task.
Model Code
Representation
Model
architecture
No. of learnable
parameters Training time Inference time
Magni et al. [ 15] Manual features MLP 8 .63 s 2 .62 × 10−4 s
DeepTune [5] C tokens Recurrent neural network 76838 8 .66 × 10−1 s 5 .59 × 10−1 s
inst2vec [3] LLVM-IR tokens Recurrent neural network 649030 1 .92 × 101 s 2 .08 × 10−1 s
GNN-AST AST+DF Graph neural network 914 4 .83 s 1 .21 × 10−3 s
GNN-CDFG CDFG+CALL+MEM Graph neural network 1024 5 .21 s 1 .32 × 10−3 s
AMD Tahiti 7970 NVIDIA GTX 970
AMD SDK
NPB
NVIDIA SDK
Parboil
Polybench
Rodinia SHOC Mean
AMD SDK
NPB
NVIDIA SDK
Parboil
Polybench
Rodinia SHOC Mean
0.00
0.25
0.50
0.75
Benchmark Suite
Accuracy
Model
Random mapping
Static mapping
Grewe et al.
DeepTune
inst2vec
GNN-CDFG
GNN-AST
Figure 10. Accuracy of device mapping in grouped setting.
node embedding vectors to a graph embedding vector hG
of size 8 by using the two MLPs fm and fд with no hidden
layers. We use the prediction model used in [ 5] and [3], which
consists of one hidden layer, but with a reduced dimension of
4 instead of 32. All baseline models we trained in the reported
configurations.
Again, we compare the model sizes and training and infer-
ence times of the different models. The results can be seen in
Table 2. And again here we see how inference in GNN-based
models is orders of magnitude faster than the LSTM-based
sequential counterparts.
1.05
1.011.01
0.930.94
1.41
1.02
0.93
1.01
0.99
1.02
1.31
0.95
0.92
0.971
0.79
1.24
0.92
0.86
1.03
0.97
0.74
1.16
0.7
0.8
1.0
AMD
Radeon HD 5900
AMD
Tahiti 7970
NVIDIA
GTX 480
NVIDIA
Tesla K20c
Platform
Speedup (log)
a
a
a
a
a
a
Oracle
Magni et al.
DeepTune
inst2vec
GNN-CDFG
GNN-AST
Figure 11. Speedup results in thread coarsening task.
Figure 11 shows the results of the thread coarsening exper-
iment. In the figure we add an łOraclež for reference, which
depicts the best possible speedups. As we can see in the fig-
ure, the sequence-based models perform better than both
graph-based models in three of the platforms. It is notable,
however, that in most cases the predicted thread-coarsening
factors yield an overall slowdown. Overall, DeepTune yields
an overall speedup (geometric mean) of around 0 .97 across
all platforms, GNN-AST is slightly better with 0 .98 and GNN-
CDFG is slightly worse with 0 .93. The Magni et al. model
yields an overall speedup of 0 .87. The best results in this
task are achieved by inst2vec, which has an overall speedup
of 1 .00, i.e. just as good as doing nothing. In general, all
deep learning methods fare comparably bad at this task. This
might be explained in part by the modest possible maximal
speedups, which cap at 1 .28 overall. However, it is perhaps
also an indication that current deep learning models of code
are not yet up to this task.
5.3 Comparison of Graph-Based Models
The graph-based representations we have used for deep
learning in this paper feature several enhancements, like the
dataflow edges enriching the AST. Similarly, for the CDFG we
included edges connecting the corresponding instructions
for function calls and the data in load and store instructions.
In this section, we want to use the three experimental se-
tups described above to compare how our model fares with
and without these enhancements. For this, we used two ver-
sions of our AST-based representation, with (AST+DF) and
without the dataflow edges (AST). Similarly, for the CDFG
representation, we considered four different versions: just
control flow edges (CFG), control and dataflow edges (CDFG),
CDFG+CALL with call edges, and CDFG+CALL+MEM.
We trained the model using similar configurations as in the
previous experiments. Figure 12 shows the results with the
209

CC ’20, February 22–23, 2020, San Diego, CA, USA A. Brauckmann, A. Goens, S. Ertel, and J. Castrillon
0.74
0.91
0.81
0.93 0.93 0.93 0.93
0.86 0.86
0.00
0.25
0.50
0.75
1.00Accuracy
Dev. map. (random split)
0.44
0.48 0.47
0.53 0.51 0.52 0.51
0.59 0.58
0.0
0.2
0.4
0.6Accuracy
Dev. map. (grouped split)
0.87
0.97
1
0.91
0.95 0.95 0.94
0.96
0.98
0.90
0.95
1.00Speedup (log)
Thread Coarsening
Model
Magni et al
Grewe et al.
DeepTune
inst2vec
GNN CFG
GNN CDFG
GNN CDFG + CALL
GNN CDFG + CALL + MEM
GNN AST
GNN AST + DF
Figure 12. Comparison of graph-based models with different enhancements.
different versions of our graph-based representations. In the
random setup for device mapping, all variants of the CDFG-
based representation fared similarly well, yielding the best
results overall. In the grouped setup, we see how the AST-
based representation yields better results. All experiments,
the grouped split mapping especially, show that the addition
of the dataflow edges to the AST was fruitful. Finally, as
discussed above, the results on the thread coarsening task
are pretty modest across all deep learning models.
From these experiments, we cannot conclude that there
is a single best compiler-based representation of code for
deep learning models. In the random split of the mapping
task, where the training data resembles more the test data,
the models closer to the execution semantics (CDFG) had
the best performance. On the other hand, with the grouped
split, where a higher degree of generalization is required, the
AST with its more abstract semantics, closer to the program-
mer, performed better. For the thread coarsening task, which
has very little data points available, the non-end-to-end ap-
proach of inst2vec fared best, albeit also not particularly well,
yielding no speedup overall.
6 Conclusion
In this paper, we have compared code representations for
deep learning based on the representations used in the com-
piler: sequences of tokens, abstract syntax trees, and con-
trol and data flow graphs. We have shown different graph-
based methods to be better suited for a complex classification
task, identifying the optimal CPU/GPU mapping for OpenCL
kernels. In particular, we outperformed state-of-the-art ap-
proaches using sequential models based on token sequences.
We also showed more generalization capabilities by split-
ting training and test data into sets coming from disjoint
benchmark suites instead of random subsets of all kernels.
In this alternative setup, our graph-based model GNN+AST
performed significantly better than their sequential counter-
parts, which failed to beat a coin-toss.
From our experiments, we cannot conclude that there is a
single best compiler-based representation of code for deep
learning models. In the random split of the mapping task,
where the training data resembles more the test data, the
models closer to the execution semantics (CDFG) had the
best performance. On the other hand, with the grouped split,
where a higher degree of generalization is required, the AST
with its more abstract semantics, closer to the programmer,
performed better. For the thread coarsening task, which has
very little data points available, the non-end-to-end approach
of inst2vec fared best, albeit also not particularly well, yield-
ing no speedup overall.
The CPU/GPU classification task is a complex task where
the interaction between different parts of the code and the
target architecture all play a role. Thus, this task serves as a
good first challenge for compiler analysis methods. However,
the related predictive task of finding optimal thread coarsen-
ing factors for OpenCL kernels proved to be too challenging
for both sequential and graph-based models. The results of
this task indicate that while useful, deep learning methods
still struggle on complex compiler-related tasks. Note that
the dataset for this task is small, which poses an additional
challenge to deep learning models.
We believe our results encourage further investigation of
compiler-based representations of code. Programming lan-
guages and machine semantics being strict and structured
as they are should also be an advantage in terms of learning,
and we can take advantage of decades of research into com-
piler methods to improve machine learning methods in this
domain. Furthermore, OpenCl kernels are comparatively
small. In future work, we plan to investigate these meth-
ods on larger code fragments, where we believe the graph
structures should be even better suited to track long-range
dependencies between parts of the code.
Acknowledgments
We thank Chris Cummins and Hugh Leather for making
the baseline methods and datasets available, as well as for
valuable feedback on the approach. We also thank Christian
Menard for an initial version of the LLVM pass. Further, we
thank the Center for Information Services and HPC (ZIH)
at TU Dresden for providing computation resources. This
work was supported in part by the German Research Council
(DFG) through the TraceSymm project CA 1602/4-1 and the
Studienstiftung des deutschen Volkes.
210

Compiler-Based Graph Representations for Deep Learning ... CC ’20, February 22–23, 2020, San Diego, CA, USA
References
[1] M. Allamanis, E. T. Barr, P. Devanbu, and C. Sutton. A survey of
machine learning for big code and naturalness. ACM Computing
Surveys (CSUR), 51(4):81, 2018.
[2] M. Allamanis, M. Brockschmidt, and M. Khademi. Learning to repre-
sent programs with graphs. arXiv preprint arXiv:1711.00740 , 2017.
[3] T. Ben-Nun, A. S. Jakobovits, and T. Hoefler. Neural code comprehen-
sion: a learnable representation of code semantics. In Advances in
Neural Information Processing Systems , pages 3585ś3597, 2018.
[4] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of
gated recurrent neural networks on sequence modeling. arXiv preprint
arXiv:1412.3555, 2014.
[5] C. Cummins, P. Petoumenos, Z. Wang, and H. Leather. End-to-end deep
learning of optimization heuristics. In Proceedings of the International
Conference on Parallel Architectures and Compilation Techniques (PACT
2017), September 2017.
[6] G. Fursin, Y. Kashnikov, A. W. Memon, Z. Chamski, O. Temam,
M. Namolaru, E. Yom-Tov, B. Mendelson, A. Zaks, E. Courtois, et al.
Milepost gcc: Machine learning enabled self-tuning compiler. Interna-
tional journal of parallel programming , 39(3):296ś327, 2011.
[7] A. Goens, A. Brauckmann, S. Ertel, C. Cummins, H. Leather, and J. Cas-
trillon. A case study on machine learning for synthesizing benchmarks.
In Proceedings of the 3rd ACM SIGPLAN International Workshop on
Machine Learning and Programming Languages (MAPL) , MAPL 2019,
pages 38ś46, New York, NY, USA, June 2019. ACM.
[8] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural
computation, 9(8):1735ś1780, 1997.
[9] D. D. Johnson. Learning graphical state transitions. In ICLR, 2017.
[10] T. N. Kipf and M. Welling. Semi-supervised classification with graph
convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.
[11] H. Leather, E. Bonilla, and M. O’boyle. Automatic feature generation
for machine learningśbased optimising compilation. ACM Transactions
on Architecture and Code Optimization (TACO) , 11(1):14, 2014.
[12] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature,
521(7553):436, 2015.
[13] Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel. Gated graph sequence
neural networks. arXiv preprint arXiv:1511.05493 , 2015.
[14] Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y. Zhong.
Vuldeepecker: A deep learning-based system for vulnerability detec-
tion. arXiv preprint arXiv:1801.01681 , 2018.
[15] A. Magni, C. Dubach, and M. O’Boyle. Automatic optimization of
thread-coarsening for graphics processors. In Proceedings of the 23rd
international conference on Parallel architectures and compilation , pages
455ś466. ACM, 2014.
[16] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of
word representations in vector space. arXiv preprint arXiv:1301.3781 ,
2013.
[17] M. F. O’Boyle, Z. Wang, and D. Grewe. Portable mapping of data
parallel programs to opencl for heterogeneous systems. In Proceedings
of the 2013 IEEE/ACM International Symposium on Code Generation and
Optimization (CGO), pages 1ś10. IEEE Computer Society, 2013.
[18] V. Raychev, M. Vechev, and A. Krause. Predicting program properties
from big code. In ACM SIGPLAN Notices , volume 50, pages 111ś124.
ACM, 2015.
[19] V. Raychev, M. Vechev, and E. Yahav. Code completion with statistical
language models. In Acm Sigplan Notices , volume 49, pages 419ś428.
ACM, 2014.
[20] N. Vasilache, C. Bastoul, and A. Cohen. Polyhedral code generation
in the real world. In A. Mycroft and A. Zeller, editors, Compiler Con-
struction, pages 185ś201, Berlin, Heidelberg, 2006. Springer Berlin
Heidelberg.
[21] Z. Wang and M. OâĂŹBoyle. Machine learning in compiler optimiza-
tion. Proceedings of the IEEE , 106(11):1879ś1901, 2018.
[22] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu. A comprehen-
sive survey on graph neural networks. arXiv preprint arXiv:1901.00596 ,
2019.
211


This paper is included in the Proceedings of the 
2019 USENIX Annual Technical Conference.
July 10–12, 2019 • Renton, WA, USA
ISBN 978-1-939133-03-8
Open access to the Proceedings of the 
2019 USENIX Annual Technical Conference 
is sponsored by USENIX.
Unleashing the Power of Learning:  
An Enhanced Learning-Based Approach  
for Dynamic Binary Translation
Changheng Song, Fudan University; Wenwen Wang, Pen-Chung Yew, and Antonia Zhai, 
University of Minnesota; Weihua Zhang, Fudan University
https://www.usenix.org/conference/atc19/presentation/song
Unleashing the Power of Learning: An Enhanced Learning-Based Approach for
Dynamic Binary Translation
Changheng Song†, Wenwen Wang‡, Pen-Chung Yew‡, Antonia Zhai‡, and Weihua Zhang†
†Software School, Fudan University
†Shanghai Key Laboratory of Data Science, Fudan University
‡Department of Computer Science and Engineering, University of Minnesota, Twin Cities
†{17212010032, zhangweihua}@fudan.edu.cn, ‡{wang6495, yew, zhai}@umn.edu
Abstract
Dynamic binary translation (DBT) is a key system tech-
nology that enables many important system applications such
as system virtualization and emulation. To achieve good per-
formance, it is important for a DBT system to be equipped
with high-quality translation rules. However, most translation
rules in existing DBT systems are created manually with high
engineering efforts and poor quality. To solve this problem, a
learning-based approach was recently proposed to automati-
cally learn semantically-equivalent translation rules, and sym-
bolic veriﬁcation is used to prove the semantic equivalence
of such rules. But, they still suffer from some shortcomings.
In this paper, we ﬁrst give an in-depth analysis on the con-
straints of prior learning-based methods and observe that the
equivalence requirements are often unduly restrictive. It ex-
cludes many potentially high-quality rule candidates from
being included and applied. Based on this observation, we
propose an enhanced learning-based approach that relaxes
such equivalence requirements but supplements them with
constraining conditions to make them semantically equivalent
when such rules are applied. Experimental results on SPEC
CINT2006 show that the proposed approach can improve the
dynamic coverage of the translation from 55.7% to 69.1%
and the static coverage from 52.2% to 61.8%, compared to
the original approach. Moreover, up to 1.65X performance
speedup with an average of 1.19X are observed.
1 Introduction
Dynamic binary translation (DBT) is a key enabling tech-
nology for many critical system applications such as system
virtualization and emulation [20, 28], whole program/system
analysis [6,13], software development and debugging [14], se-
curity vulnerability detection and defense [15, 17], computer
architecture simulation [22, 27, 29], and mobile computation
ofﬂoading [26]. There have been many widely-used DBT
systems, such as Pin [18], Valgrind [21] and QEMU [2].
In general, a DBT system takes an executable binary code
in one instruction set architecture (called guest ISA) and
dynamically translates it into the binary code in another in-
struction set architecture (called host ISA). The translation
process is mostly driven by translation rules that translate
guest instructions into a sequence of semantically-equivalent
host instructions [23].
For a DBT system, its performance is dominated by the
quality of the translated host binary code [25]. Therefore,
it is very important for a DBT system to be equipped with
high-quality translation rules. However, due to the complex-
ity and opacity of modern ISAs, it is difﬁcult to manually
construct such high-quality translation rules as it poses a
signiﬁcant engineering challenge. Even worse, to support re-
targetable DBTs (from multiple guest ISAs into multiple host
ISAs) in the same framework, a set of pseudo-instructions
are commonly used as their internal representations [2]. As
the execution time is directly proportionate to the number
of host instructions executed, such a multiplying effect has a
signiﬁcant impact on the overall DBT performance.
To improve the quality of translation rules and reduce en-
gineering efforts, a learning-based approach [23] is recently
proposed to learn automatically binary translation rules. Since
the translation rules are learned from the optimized binary
codes generated by the compiler, this approach is capable of
yielding higher quality translation rules than existing manual
schemes. Moreover, the whole learning process can be fully
automated without manual intervention. Although the above
approach is attractive, it still suffers from some fundamental
limitations. That is, a translation rule can be harvested (i.e.,
learned) only if the guest and the host binary code that cor-
respond to the same program source statement(s) are strictly
semantically equivalent. This is enforced through a symbolic
veriﬁcation process.
On the surface, this equivalence veriﬁcation process is nec-
essary and appropriate because it guarantees the correctness
of the learned rules. However, further investigation reveals
that this equivalence requirement is often unduly restrictive. It
excludes many potentially high-quality rule candidates from
being harvested and applied. In particular, such restrictions
usually keep architecture-speciﬁc instructions in guest and/or
USENIX Association 2019 USENIX Annual Technical Conference    77
host ISAs from being included for more efﬁcient translation
as they are mostly architecturally speciﬁc, and thus inherently
different and more challenging to prove semantic equivalence.
To overcome this limitation, this paper presents an en-
hanced learning-based approach that relaxes such restrictions
and allows more translation rules to be harvested and ap-
plied. More speciﬁcally, it purposely relaxes the requirements
of semantic equivalence and allows semantic discrepancies
between the guest and host instructions to exist in the trans-
lation rules, e.g., different condition codes or the different
number of operands in matching guest and host instructions.
Symbolic veriﬁcation process is no longer just to check the
strict semantic equivalence between the matching guest and
host instructions, but also to identify the speciﬁc semantic
discrepancies between them that can be used during the rule
application phase to verify whether such discrepancies either
will not cause ill effect, or are satisﬁed in the context of the
rules being applied (for more details see Section 4). We call
such semantic equivalence in the translation rulesconstrained
semantic equivalence as the speciﬁc semantic discrepancies
of the translation rules become the constraining conditions
for such rules to be safely applied. This requires some run-
time program analysis (mostly in a very limited scope) during
the rule application phase, which usually incurs very small
overhead. Those with very complicated constraining condi-
tions that require extensive runtime program analysis will be
discarded.
To demonstrate the feasibility and the beneﬁt of such
constrained-equivalent translation rules, we have imple-
mented a prototype based on the proposed approach. The
prototype includes an enhanced learning framework and a
DBT system that applies the constrained-equivalent trans-
lation rules to generate host binary code. We evaluate the
implemented prototype using SPEC CINT2006. Experimen-
tal result shows that the proposed approach can signiﬁcantly
improve the harvest rate of the learning process from 20.3%
to 25.1% and dynamic coverage from 55.7% to 69.1% while
static coverage from 52.2% to 61.8%, compared to the orig-
inal learning approach in [23]. Moreover, no degradation is
observed for the learning efﬁciency, i.e., around 2 seconds to
yield a translation rule, which is the same as the original learn-
ing process. After applying the enhanced translation rules, we
achieve up to 1.65X performance speedup with an average of
1.19X compared to the original approach.
In summary, this paper makes the following contributions:
•We propose an enhanced learning-based approach that
can harvest and apply constrained-equivalent translation
rules discarded by the original approach, and allows DBT
systems to generate more efﬁcient host binary code.
•We implement the proposed learning-based approach in
a prototype, which includes a learning framework based
on LLVM and a DBT system extended from QEMU to
accept the constrained-equivalent translation rules.
•We conduct some experiments to evaluate the proposed
learning-based approach. Experimental results on SPEC
CINT2006 shows that our approach can achieve up to
1.65X speedup with an average of 1.19X compared to
the original learning approach.
The rest of this paper is organized as follows. Section 2
presents some background of the original non-constrained
semantically-equivalent learning-based approach. In Sec-
tion 3, we identify some technical challenges in learning and
applying constrained-equivalent translation rules. Section 4
presents the design issues of our enhanced learning-based
approach. In Section 5, we describe some implementation
details of the prototype and evaluate the proposed approach
and show some experimental results. Section 6 presents some
related work and Section 7 concludes the paper.
2 Background
In this section, we introduce some background information
on how a DBT and a learning-based approach such as the one
proposed in [23] work.
2.1 Dynamic Binary Translation (DBT)
Typically, a DBT system adopts a guest basic block (or block
for short) as the translation unit to translate guest binary code
into host binary code. A basic block comprises a sequence of
instructions with only one entry and one exit, and thus when-
ever the ﬁrst instruction of a basic block is executed, the rest
of the instructions in this block will be executed exactly once
in order. It is worth noting that, due to the semantic differ-
ences between the guest and host ISAs, one guest block may
be translated into multiple host blocks by the DBT system.
To translate a guest basic block, the DBT system ﬁrstly
disassembles the guest binaries to obtain guest assembly in-
structions. Then, it tries to match the guest instructions with
available translation rules. After a matched translation rule
is found, the corresponding guest instructions are translated
into host instructions as speciﬁed in the translation rule. This
process could be iterated multiple times until all instructions
in the guest block are translated. Finally, the generated host
instructions are assembled into host binaries and executed
directly on host machines. Figure 1 shows an example of such
a translation process, where ARM is the guest ISA, and x86 is
the host ISA. In this example, two translation rules are applied
to translate two ARM instructions into two x86 instructions,
respectively.
To mitigate the performance overhead incurred during the
translation process, especially for short-running guest applica-
tions, the translated host binary code is stored into a memory
region called code cache, and reused in the later execution.
After all instructions in a guest block are translated, the execu-
tion ﬂow of the DBT system is transferred to the code cache.
78    2019 USENIX Annual Technical Conference USENIX Association
Translation Rule
1
Guest (ARM)
add reg0, reg0, reg1
Host (x86)
addl reg1, reg0
2
Guest (ARM)
sub reg0, reg0, #imm0
Host (x86)
subl $imm0, reg0
... ...
e0811000
e2411001
...
add r1, r1, r0
sub r1, r1, #1
...
addl %eax, %edx
subl $1, %edx
...
ARM 
instructions
ARM 
binaries
x86 
instructions
Disasm
Translation
01 c2
83 ea 01
...
x86 
binaries
Asm
Figure 1: Dynamic binary translation from ARM to x86
driven by manually-constructed translation rules. Here, for
simplicity, we assume the guest registers r0 and r1 are emu-
lated using the host registers eax and edx, respectively.
A hash table is employed to establish the mapping between
the guest binary code and the corresponding translated host
binary code in the code cache. Each time a guest block is
encountered, the hash table is looked up to ﬁnd out whether
there exists a host code in the code cache that corresponds to
this guest block. If yes, the translation process will be skipped,
and the stored host binary code will be executed. Otherwise,
the guest block is translated, and the hash table is updated
with the added translated host binary.
2.2 Learning Translation Rules
As mentioned earlier, the translation process in a DBT system
is mainly directed by translation rules, which also determine
the quality (i.e., performance) of the translated host binary
code. Therefore, it is vital for a DBT system to have high-
quality translation rules for better performance. However, in
practice, it is a signiﬁcant engineering challenge to develop
high-quality translation rules as most translation rules in ex-
isting DBT systems are constructed manually by developers.
Moreover, modern ISAs are often documented in obscure
and tediously long manuals. For example, Intel’s manual has
around 1500 pages for the x86 ISA. It requires substantial
engineering efforts to understand both the guest and the host
ISAs to construct high-quality translation rules.
To solve this problem, a recent approach proposes to auto-
matically learn binary translation rules [23]. More speciﬁcally,
this approach uses the same compiler for different ISAs, i.e.,
LLVM-ARM and LLVM-x86, to compile the same source
program. During the compilation process, it extracts binary
translation rules from ARM and x86 binary code that cor-
respond to the same program source statement(s). This is
inspired by the observation that the binary code compiled
Source code
char *e = s + (n - 1);
ARM instruction
add r1, r1, r0
sub r1, r1, #1
x86 instruction
leal -0x1(%edx, %eax), %edx
Guest (ARM)
add reg0, reg0, reg1
sub reg0, reg0, #imm0
Host (x86)
leal -imm0(reg0, reg1), reg0
LLVM-ARM LLVM-x86
A learned 
translation 
rule
Learning
Figure 2: Automatically learning binary translation rules
during the compilation process of program source code. Com-
pared to the translation rules used in Figure 1, the learned
translation rule can generate more efﬁcient host binary code.
by the same compiler for different ISAs from the same pro-
gram source code should be equivalent in program semantics.
To further enforce such equivalence requirement, a symbolic
veriﬁcation engine is developed to ﬁlter out rule candidates
in which guest and host binary code are not semantically
equivalent.
Figure 2 illustrates an example of the above learning pro-
cess, In this example, the program source statement is com-
piled into two ARM instructions and one x86 instruction by
LLVM-ARM and LLVM-x86, respectively. Using symbolic
execution, we can verify that the guest ARM register r1 and
the host x86 register edx should have the same value assum-
ing the same initial condition. We can thus prove that the
sequence of the two ARM instructions is semantically equiva-
lent to the single x86 instruction in the example. A translation
rule that maps the sequence of the two ARM instructions into
one x86 instruction can then be harvested. Recall the example
in Figure 1. If we use this learned rule to translate the guest
binaries, we only need one host instruction instead of two as
shown in the example, i.e. more efﬁcient host binary code can
be generated.
3 Issues and Challenges
The signiﬁcance of the above learning approach is two folds.
Firstly, it can automatically learn binary translation rules for
DBT systems with less burden on developers. Secondly, given
that the translation rules are learned directly from binary
code generated by the native compilers, it is more likely that
the harvested translation rules are more optimized than the
translation rules naïvely constructed by hand, as shown in
Figure 2 and Figure 1.
Theoretically, if we keep training such a learning-based
USENIX Association 2019 USENIX Annual Technical Conference    79
Source code
x++;
ARM instruction
ldr r0, [r1, #120]
add r0, r0, #1
str r0, [r1, #120]
x86 instruction
incl 0x78(%eax)
LLVM-ARM LLVM-x86
Discarded
Source code
if (node->pos)
ARM instruction
ldr r1, [r1, #216]
cmp r1, #0
beq #L1
x86 instruction
cmpl $0x0, 0xd8(%eax)
je $L1
LLVM-ARM LLVM-x86
Discarded
(a) (b)
Figure 3: Two examples to demonstrate the limitation of
the learning approach in [23]. These two rule candidates are
discarded because the guest registers r0 in (a) and r1 in (b)
have no equivalent host register.
system with a large number of source programs, we should be
able to harvest a large number of translation rules and apply
them to guest binaries with good coverage. Unfortunately,
after a more thorough study of this approach, we found it
suffers from a fundamental limitation that prohibits it from
harvesting many high-quality translation rules. In this section,
we explain in more details such limitations and identify some
technical challenges if we want to overcome them.
A Fundamental Limitation.To guarantee the correctness
of the learned translation rules, it employs a symbolic veri-
ﬁcation engine to check the exact semantic equivalence be-
tween the guest and host binary code sequences. More specif-
ically, the semantic equivalence is veriﬁed in three aspects
that include matching register operands, memory operands
and branch conditions. More details can be found in [23].
If the veriﬁcation results show that the guest and host binary
code sequences are not strictly equivalent, the rule candidate
is discarded and no translation rule is harvested. Undoubtedly,
such a veriﬁcation process is necessary and appropriate. How-
ever, by a more detailed study on the discarded rule candidates,
we found that the requirement of exact semantic equivalence
is too restrictive. Many high-quality rule candidates are forced
to be discarded, especially those guest and host binary code
sequences that are more architecturally speciﬁc and their ISAs
are signiﬁcantly different, such as ARM (a reduced instruction
set computer (RISC)) and Intel x86 (a complex instruction
set computer (CISC)) in our example.
Figure 3 shows two examples of this limitation. Here, simi-
lar to the previous examples, the guest ISA is ARM and the
host ISA is Intel x86. In Figure 3(a), the value of the vari-
able x is increased by one through the increment operator as
shown in the source code. With its RISC ISA, the ARM com-
piler generates three instructions for this source statement:
loading the original value of x, performing the addition, and
then storing the result back to x. In contrast, the Intel x86
compiler needs only one instruction, incl, with its CISC ISA.
Source code
if (--x == 0)
ARM instruction
ldr r0, [r1, #56]
subs r0, r0, #1
str r0, [r1, #56]
x86 instruction
decl 0x38(%eax)
LLVM-ARM LLVM-x86
Discarded
Source code
while (iters-- > 0)
ARM instruction
subs r2, r2, #1
x86 instruction
leal -0x1(%ecx), %ecx
LLVM-ARM LLVM-x86
Discarded
(a) (b)
Figure 4: Another two rule candidates discarded by the learn-
ing approach in [23] because of the different condition codes
in guest and host ISAs.
Similarly, in Figure 3(b), the x86 instruction cmpl can have
a memory operand, but an ARM ldr instruction is required
before the cmp instruction. In these two cases, the veriﬁca-
tion will fail because there is a mismatch of register operands
between the guest and the host code sequences, i.e. there is
no host register that matches and holds the same value as the
guest register r0.
However, if we examine these two examples more care-
fully, we will ﬁnd that the root cause of the failed veriﬁcation
stems from the inherent differences between the guest and
the host ISAs. In practice, such architectural differences are
quite common and pervasive in different ISAs, even if they are
both RISCs or CISCs. For instance, a post-indexed memory
load instruction in ARM will modify the address register after
the loading operation, while there is no similar instruction in
MIPS, which is another representative RISC ISA.
In fact, these differences represent the essence of the ar-
chitectural design unique to each ISAs. It is indeed a huge
loss for a learning-based approach to discard such rule can-
didates simply because of their ISA differences. As they are
architecturally speciﬁc, they are often the most efﬁcient code
sequences selected by the native compilers for speciﬁc pro-
gram contexts and thus have a high potential to turn into
high-quality translation rules.
Another shortcoming resulted from the aforementioned
limitation is that it also excludes many rule candidates that
contain instructions associated with architecture-speciﬁc hard-
ware support. For instance, many architectures have condi-
tion codes (also known as eﬂags in x86 machines). They are
single-bit registers used to store the execution summary of
an instruction and can inﬂuence the control ﬂow of the later
instructions. In particular, ARM has four condition codes: neg-
ative (NF), zero (ZF), carry (CF), and overﬂow (VF), while
x86 has seven condition codes: carry (CF), parity (PF), adjust
(AF), zero (ZF), sign (SF), direction (DF), and overﬂow (OF).
Figure 4 shows two examples with instructions related to
condition codes. In Figure 4(a), the source code decreases the
80    2019 USENIX Annual Technical Conference USENIX Association
value of x by one and then checks the result to see whether
it is zero or not. An ARM instruction subs is generated to
perform the subtraction and update the condition codes. Here,
subs updates all four ARM condition codes, including CF.
Similarly, an x86 instruction decl is used to decrease the
value stored in the memory operand by one and update the
condition codes. However, decl updates all x86 condition
codes, except CF. As a result, the veriﬁcation process in the
original learning-based approach will consider the ARM and
x86 code are not semantically equivalent and discard this rule
candidate. Similarly, the rule candidate in Figure 4(b) is also
discarded because the x86 instruction leal does not update
any condition code. In fact, the source code in Figure 4(a)
only needs to check whether the result is zero or not, which
only requires the condition code ZF. Thus, it is unnecessary to
update the condition code CF, as it is never used in this context.
That means, it is still possible to harvest this translation rule
and apply it, if the ARM condition code CF is not used (i.e.
dead) in the later code before it is updated. Similarly, the rule
candidate in Figure 4(b) can also be harvested.
Technical Challenges.Although such limitations could
exclude many high-quality translation rules during the learn-
ing process, it faces several technical challenges if we want
to harvest them and apply them in a DBT system for a better
performance and higher coverage.
First, we have to relax the original veriﬁcation objectives
as they are designed to verify the exact equivalence between
the guest and host code sequences.
Second, given that most of those translation rules are not
strictly equivalent, it is imperative that we have a mechanism
to enforce their correctness when we apply them. Equally
important is that the performance overhead incurred by such
enforcement should be less than the performance gain they
can provide.
Last but not least, in the original learning approach, a
learned translation rule only needs to include two parts, i.e.,
the guest and host instructions, and this is typically sufﬁcient
for a DBT system. However, for the constrained-equivalent
translation rules, whether we can apply these rules at runtime
or not depends on the context they are being applied. As a
result, we need to extend the structure of translation rules to
include such constraining requirements.
4 An Enhanced Learning-Based Approach
In this section, we present the design of the proposed en-
hanced learning-based scheme, starting with an overview of
the system framework.
4.1 Overview
The major goal of our enhanced learning-based approach is to
learn and apply high-quality translation rules excluded by the
original learning approach. These translation rules contain
constrained-equivalent guest and host instructions, and thus
cannot be harvested using the original learning approach. To
this end, we redesign the learning process, reorganize the
structure of the learned translation rules, and make necessary
extensions to the DBT system to allow the application of the
constrained-equivalent translation rules.
Figure 5 illustrates the workﬂow of our enhanced learning-
based approach. To learn translation rules, we also compile the
same program source code using the same compiler for guest
and host ISAs to generate two versions of the binary code. We
then extract guest and host code sequences that correspond to
the same learning scope and consider them as the candidates
for the translation rules. The learning scope is deﬁned at the
program source code level. In the original learning system,
the default learning scope is set to be one source statement.
The extracted guest and host code sequences then form a rule
candidate. For each rule candidate, the next step is to verify
whether the corresponding guest and host code sequences are
constrained equivalents or not. If yes, a translation rule can
be harvested. Otherwise, the rule candidate is discarded.
As an example to demonstrate our approach and by study-
ing the rule candidates discarded by the original learning
scheme, we consider the guest and host code sequences in
a rule candidate as constrained equivalent if every modiﬁed
guest storage operand contains the same value as a modiﬁed
host storage location at the end of the code sequences and
vice versa. Here the storage operand is broadly deﬁned, as
it can be either a register, a memory location, or a condition
code (i.e., eﬂag). Furthermore, it is allowed that there is no
corresponding modiﬁed storage location in the host code se-
quences, e.g., a corresponding condition code as mentioned
earlier.
Using this relaxed and constrained equivalence deﬁnition,
the guest and host code sequences can be semantically equiv-
alent only if all modiﬁed guest storage operands without the
corresponding host storage operands (e.g., condition codes)
are not used in the following guest binaries before they are
modiﬁed again. These modiﬁed guest storage operands with-
out the corresponding host storage operands can be considered
as the constraining condition of this constrained-equivalent
translation rule.
In our framework, the semantic equivalence can be relaxed
in other ways as long as the discrepancies can be identiﬁed
and shown either having no ill effect in the context they are
applied or can be compensated to make them semantically
equivalent when they are applied. In other words, their con-
straining conditions can be identiﬁed and satisﬁed when these
rules are applied. To simplify our prototype design, we only
consider relaxing the requirement of exact mapping of the
storage operands as deﬁned earlier. The identiﬁed constrain-
ing conditions are integrated into the learned translation rules
to determine whether it is safe to apply them or not. It is
worth noting that for strictly equivalent translation rules the
constraining condition is null.
USENIX Association 2019 USENIX Annual Technical Conference    81
Guest:
...
Host:
...
Constraining 
condition:
...
Guest:
...
Host:
...
Constraining 
condition:
...
Source 
Code
Rule 
Candidates
Guest 
Binary
Host 
Binary
CC-Guest
CC-Host
Flexible 
Preprocessing
Relaxed 
Verification
Guest:
...
Host:
...
Constraining 
condition:
...
Translation 
Rules
DBT 
System
Guest Binary
Host 
Binary
Online 
Analysis
Learning translation rules Applying translation rules
Figure 5: The work ﬂow of the proposed enhanced learning-based approach.
To determine whether the constraining condition is met or
not, a lightweight runtime analysis is employed to determine
the program context in which the guest instructions is to
be translated. In our case, the program context includes the
information about which guest storage operand is modiﬁed
by the guest instructions and used in the later code before it
is modiﬁed again. The program context is then used to verify
whether the constraining condition is satisﬁed or not. If yes,
the translation rule can be applied, otherwise, it is discarded.
4.2 Varying Learning Scopes
In the original learning scheme, the learning scope is limited
to one source statement. Although it appears to be reasonable,
it may miss potential rule candidates as it is very common
for compilers to perform optimization across multiple source
statements. Therefore, our enhanced learning approach varies
the learning scope from one to n source statements, and apply
each learning scope over the source program.
More speciﬁcally, a sliding window is employed. The slid-
ing window of size n covers n contiguous source statements,
i.e. the sliding window covers the learning scope of n state-
ments. Guest and host instructions that correspond to the n
statements in this window are extracted as a rule candidate.
The sliding window moves from the ﬁrst line of the source
code toward the end of the code. The window size is initially
set to 1, and incremented by one after each pass through the
sources code. When the window of size i is moved through
all the source code, the number of rules learned from current
window size will be compared to the number of rules learned
from window sizes 1 to i −1. If new rules learned from win-
dow size i are less than 10% of all learned rules from window
sizes 1 to i −1, the learning process will be stopped.
4.3 Learning Constrained-Equivalent Rules
To verify the constrained equivalence of the guest and host
instructions in a rule candidate, we use the same symbolic
veriﬁcation engine, but relax the requirements for semantic
equivalence.
First, we establish an initial mapping between guest and
host live-in operands the same way as the original learning
approach, i.e., guest registers ↦→host registers, guest memo-
ries ↦→host memories, and guest immediate values ↦→host
immediate values (i.e. constants). Then, we initialize the
mapped guest and host operands with the same symbol values
and symbolically execute the guest and host code sequences,
respectively. After the symbolic execution, we extract the
symbol results of the modiﬁed guest and host registers, mem-
ories, and condition codes. These results are then fed into
a SMT solver to ﬁgure out, for each modiﬁed guest regis-
ter/memory/condition code, whether there exists a modiﬁed
host register/memory/condition code corresponding to it or
not. If each modiﬁed guest operand is mapped to a modiﬁed
host operand, an original rule is generated.
If the SMT solver indicates that there exists a modiﬁed
guest memory operand that does not have a matching host
memory operand, we discard this rule candidate. If a modiﬁed
guest register/condition code has no matching modiﬁed host
register/condition code, we can harvest this rule candidate as
the guest and host instructions can still be constrained equiv-
alent. Moreover, such unmatched guest registers/condition
codes are recorded as constraining conditions of the learned
rules and will be checked when the rules are applied. The rea-
son for discarding candidate rules with unmatched memory
operands is that the resulting constraining conditions will re-
quire time-consuming data dependence analysis to determine
whether the constrained equivalence is satisﬁed or not when
such rules are applied.
Otherwise, all other rules are considered as a non-
equivalent rule and be discarded.
4.4 Lightweight Online Analysis
For each constrained-equivalent rule to be applied, an online
analysis is invoked to analyze the program context of the
guest code sequence. The context information includes the
data ﬂow of the guest registers and condition codes, which
can be obtained by statically analyzing the guest instructions.
The context information is then used to determine whether the
constraining condition of the matched constrained-equivalent
rule is satisﬁed. For instance, if the analysis shows that a mod-
iﬁed guest register is not mapped to any modiﬁed host register
in the rule, and this modiﬁed guest register is not used in the
following guest code, we can determine that the constraining
condition has been satisﬁed in the program context, and the
82    2019 USENIX Annual Technical Conference USENIX Association
translation rule can be applied.
In general, to collect the context information, the online
analysis examines guest instructions that are executed after
the matched guest instruction sequence. Each instruction is
examined to see whether it deﬁnes or uses the register(s) or
condition code(s) speciﬁed in the constraining condition of
the matched translation rule. If a deﬁnition can be found
before usage on all paths following the matched guest code
sequence, the matched rule can be applied safely. Otherwise,
if usage is found, the matched rule should not be applied
as the modiﬁed guest register/condition code is used but the
matched rule does not update it.
For indirect branch instructions, it is quite difﬁcult to iden-
tify all possible branch targets statically. For simplicity, we
stop the online analysis when an indirect branch is encoun-
tered and the translation rule will not be applied for safety
consideration.
4.5 Handling Predicated Instructions
Predicated instructions are very common in many ISAs, e.g.,
ARM and MIPS. A predicated instruction executes only if its
predicate bit is "True". Otherwise, the instruction is a "nop".
For example, "add ne r0, r0, r1" in ARM will be executed
only when the condition code is not equal ("ne"). Some ISAs
like x86 do not support predication, and conditional branches
are used instead. It is worth noting that the original learning
approach cannot handle predicated instructions. So how to ef-
ﬁciently support predicated instructions is another important
design issue for a learning-based approach because the predi-
cate tag in predicated instructions and conditional branch are
not equivalent although the execution results are the same.
In translation, we use a lightweight analysis to divide pred-
icated instructions into multiple blocks and generate condi-
tional branches around those blocks according to their pred-
icate information to support the translation of predicate in-
structions. Before translating a basic block, we ﬁrst check the
predicated condition of all instructions and divide the basic
block into multiple condition blocks. Each condition block
includes instructions with the same predicated condition. In
one condition block, the translation rules can be directly ap-
plied without considering the predicated condition. After a
condition block is translated, a branch instruction with the
opposite condition is added to the host basic block before
the translated condition block is added to the host block. The
branch target is the instruction following the end of the host
block. This analysis is very lightweight and each basic block
only needs to be checked once.
Note that an instruction with a predicated condition may
change the condition codes itself. For example, cmp ne r0,
0 will update the condition code if the last condition code is
not equal. So, instructions after these instructions that may
change condition codes should be divided into a new condi-
tion block even the predicated condition is the same.
4.6 Discussion
Our enhanced learning approach currently only supports user-
level applications. The translation for full-system level ap-
plications is not supported because full-system translation is
more complex with mechanisms such as system calls, inter-
rupts and device I/O. These mechanisms make the learning
and matching of rules more difﬁcult. It is left in our future
work.
ABIs and many instructions such as indirect branches are
not supported either. For ABIs, the calling conventions, such
as how parameters are passed and how many parameters are
used, are difﬁcult to be identiﬁed and translated by rules. For
example, ARM use registers to pass parameters but no speciﬁc
instructions are used. But in X86, the push instructions will
be used for passing parameters. For indirect branches, DBT
systems usually search the branch target address according to
a branch table maintained at runtime, which is not available
at compile time. It makes it impossible to translate by our
learned rules.
5 Experimental Results
In this section, we evaluate our prototype and address the
following research questions:
1. How much performance improvement can be obtained
by our enhance learning scheme in which we relax the
requirement of matching storage operands as described
in Section 4?
2. Where does the performance improvement come from
when we include the added constrained-equivalent trans-
lation rules?
3. What is the effect of relaxing the strict semantic equiva-
lence requirement?
4. How much overhead will the dynamic analysis incur?
5.1 Experimental Setup
Our enhanced learning-based DBT prototype is implemented
based on QEMU (version 2.6.0) which is the same as the
original learning scheme. The guest ISA is ARM, and the
host ISA is x86. The LLVM compiler (version 3.8.0) is used
to generate binary code for guest/host ISAs. All binary codes
are generated using the same optimization level -O2. The
same version of source code and guest/host binary code are
used for comparison. One machine with 3.33GHz Intel Core
i7-980x with six cores (12 threads) and 16GB memory is
set up exclusively for performance evaluation. The operating
system is the 32-bit Ubuntu 14.04 with Linux 3.13 for both
machines. We used an older version of the system because we
need to compare our new approach with the original approach,
which used the same older version of the system. Besides, our
USENIX Association 2019 USENIX Annual Technical Conference    83
 1
 1.2
 1.4
 1.6
 1.8
 2
 2.2
 2.4
 2.6
perlbenchbzip2gcc mcf gobmkhmmersjenglibquantumh264refomnetppastar xalanGEOMEAN
Performance Speedup
origin
enhanced
Figure 6: Performance comparison of the original and en-
hanced learning-based approaches.
experimental results are valid regardless of the system version
used.
We use the 12 benchmarks in SPEC CPU INT 2006 with
the reference inputs in our studies. To be as close to the
real-world usage scenarios as possible, the performance of
each benchmark is evaluated by the rules learned from other
11 benchmarks excluding evaluated benchmark itself. Each
benchmark is run three times to reduce the random inﬂuence.
The enhanced learning scheme described earlier is imple-
mented in Python. The enhanced veriﬁcation is implemented
based on FuzzBALL [19], a symbolic execution engine. The
tiny code generator (TCG) in QEMU is also enhanced to sup-
port the translation of predicated instructions as described
in the last subsection, As mentioned earlier, to apply the
constrained-equivalent translation rules, the dynamic anal-
ysis should be performed before such rules are applied.
5.2 Performance Results
Figure 6 shows the performance comparison of our enhanced
learning-based scheme (marked as enhanced) and the original
learning-based approach (marked asorigin). The performance
of QEMU without using any of the learning schemes is used
as the baseline. Table 1 shows the MIPS of performance of
original and enhanced approaches.
Using the ref input for all SPEC benchmarks, the quality
of the translated host code is the major factor that impacts
the overall performance. As shown in Figure 6, our enhanced
learning scheme can achieve a performance improvement of
up to 2.55X with an average of 1.74X compared to QEMU,
which is a 1.19X improvement over the original learning
approach on average.
By studying the learned translation rules and how they are
applied using our enhanced learning approach, we have the
following observations on how they impact the overall per-
formance. First, constrained-equivalent translation rules can
usually be applied quite successful. The modiﬁed guest reg-
isters/condition codes that have no matching modiﬁed host
registers/condition codes will usually be modiﬁed quickly
again. This means they are only used to hold temporary value
as we expected. Hence, relaxing strict matching requirements
Table 1: MIPS of the original and enhanced learning-based
approaches
Benchmarks Original enhanced
perlbench 221.63 250.15
bzip2 1211.32 1388.92
gcc 521.78 575.09
mcf 603.99 739.93
gobmk 372.18 616.31
hmmer 1448.56 1632.93
sjeng 474.71 485.23
libquantum 1469.59 1532.97
h264ref 189.99 215.69
omnetpp 195.28 284.74
astar 396.10 562.51
xalan 283.19 290.52
GEOMEAN 475.15 567.29
for storage operands can yield more translation rules, albeit
constrained-equivalent rules, and can be applied quite effec-
tively.
Second, by relaxing the equivalence constraints to allow
constrained-equivalent translation rules that can include pred-
icated and condition instructions greatly improve the overall
performance. This is because typical DBTs such as QEMU
usually use memory locations to emulate the condition codes.
Such an approach will incur many additional memory opera-
tions to access and update those condition codes in memory
and incur very high overhead. But the constrained-equivalent
translation rules can take advantage of the host condition
codes to emulate guest condition codes, which can signiﬁ-
cantly reduce such overheads.
Figure 7(a) and Figure 7(b) show the static and dynamic
coverage of the guest binaries using the origin and our en-
hanced learning-based schemes, respectively. The"Coverage"
here is deﬁned as the percentage of guest instructions that can
be translated by the learned rules. So the "static" coverage is
the percentage of static code translated by learned rules and
"dynamic" coverage here is the percentage of"executed" guest
instructions translated by learned rules. Compare to the origi-
nal learning-based scheme, our enhanced learning scheme can
improve the static coverage from 52.2% to 61.8%, and the dy-
namic coverage from 55.7% to 69.1% on average. It is worth
noting that gcc and libquantum have a much higher dynamic
coverage improvement than others, but do not get an expected
higher performance improvement. Conversely,gobmk attains
a high performance improvement but not as much coverage
improvement. The reason is that many high-quality rules are
applied when translating gobmk, but in gcc and libquantum,
the applied rules can only attain moderate improvement. This
seems to indicate that the coverage improvement does not
translate directly to the overall performance improvement,
but could be an important secondary effect.
84    2019 USENIX Annual Technical Conference USENIX Association
0 %
20 %
40 %
60 %
80 %
100 %
perlbenchbzip2gcc mcf gobmkhmmersjenglibquantumh264refomnetppastarxalangeamean
Percentages of 
 Static Guest Instructions 
 Translated by Rules
origin
enhanced
(a) Static coverage
0 %
20 %
40 %
60 %
80 %
100 %
perlbenchbzip2gcc mcf gobmkhmmersjenglibquantumh264refomnetppastarxalangeamean
Percentages of 
 Dynamic Guest Instructions 
 Translated by Rules
origin
enhanced
(b) Dynamic coverage
Figure 7: Static and dynamic coverage of translation rules.
To address the question of "where does the performance im-
provement come from when we include the added constrained-
equivalent translation rules?", we analyze the coverage of the
added translation rules when they are applied in each pro-
gram. The results are shown in Figure 8. As described in
Section 4, there are three major components in our relaxed
equivalence constraints, i.e. we remove the strict requirement
of exact matching. They are (1) register operands (marked as
register), (2) condition-code operands (marked as condition),
and (3) predicated-related instructions (marked as predicate).
We did not include memory operands because they require
more complicated data dependence analysis when they are
applied.
A very interesting observation is that, among the added
constrained-equivalent translation rules (their increased cov-
erage is shown in Figure 7), the register-related constrained-
equivalent translation rules constitute 58.83% of the static in-
structions on average. However, they only constitute 13.58%
of added dynamic coverage. But the dynamic coverage of
condition-code related rules is increased to 54.02% on av-
erage, while their static coverage is only 18.52%. This is
because the condition codes are usually associated with the
bound check, such as at the end of a loop. So, these instruc-
tions will be executed more frequently than others in their
dynamic coverage.
To study the quality/efﬁciency of translated rules, Figure 9
shows the percentages of the reduced host instructions. On
average, our enhanced learning scheme can reduce 11.28% of
the total dynamic host instructions compared to the original
learning scheme. We observe that the reduction in the host
0 %
20 %
40 %
60 %
80 %
100 %
perlbenchbzip2gcc mcf gobmkhmmersjenglibquantumh264refomnetppastarxalangeomean
Distribution of 
 Increased Static Coverage
register condition predicate
(a) Distribution of increased static coverage
0 %
20 %
40 %
60 %
80 %
100 %
perlbenchbzip2gcc mcf gobmkhmmersjenglibquantumh264refomnetppastarxalangeomean
Distribution of Increased 
 Dynamic Coverage
register condition predicate
(b) Distribution of increased dynamic coverage
Figure 8: Distribution of the improved rule coverage.
0 %
10 %
20 %
30 %
40 %
perlbenchbzip2gcc mcf gobmkhmmersjenglibquantumh264refomnetppastarxalangeomean
Percentage of 
 Reduced Dynamic 
 Host Instructions
Figure 9: Reduction in dynamic host instruction counts by
enhanced learning and translation.
instructions of gobmk is higher than 30%, and in omnetpp, it
is higher than 20%. However, in gcc and libquantum, the re-
duction is only about 10%. This also conﬁrm our observation
that rules applied in gobmk and omentpp translation have a
higher quality, i.e. fewer host instructions in those translation
rules, than rules in gcc and libquantum. But, we also notice
that bzip2 and astar attain a high performance improvement
but only a moderate number of host instructions are reduced.
One probable explanation is that even though they may have
similar dynamic host instruction counts, more efﬁcient and
architecture-speciﬁc host instructions may have been used.
5.3 Learning Results
We further study the effect of our proposed relaxed learn-
ing scheme in other related aspects. The ﬁrst is about the
"yield" obtained during the learning phase, which shows how
USENIX Association 2019 USENIX Annual Technical Conference    85
0 %
10 %
20 %
30 %
40 %
50 %
perlbenchbzip2gcc mcf gobmkhmmersjenglibquantumh264refomnetppastarxalangeomean
Percentages of 
 Instrcutions Generating Rules
origin
enhanced
Figure 10: The yields of rule learning.
many translation rules can be harvested among the candidate
rules during the learning phase. The other aspects include
the effect of using the sliding window, and the distribution of
the constrained-equivalent rules learned based on our relaxed
equivalence constraints, i.e. relaxing the strict matching re-
quirement on three storage operand types described in Section
4.
Figure 10 shows the yield obtained during the learning
phase using the original learning scheme (marked as origin)
and our enhanced learning scheme (marked as enhanced).
The learning yield is increased from 20.3%to 25.1%. Even
though the improvement in learning yield is moderate as we
only made moderate relaxation on the semantic equivalence
requirements, but as the obtained performance results show
the quality of these rules is quite high. The moderate yield
improvement also shows that there is a high potential for
more high-quality rules to be learned and harvested. Another
interesting question is that if a signiﬁcant number of more
source programs is used in the training phase, even with a low
yield, how much more rules can be learned, and how much
more performance improvement can be achieved by applying
those added rules. These questions are beyond the scope of
this paper.
Figure 11 shows the distribution of the translation rules
learned using a ﬂexible sliding window. We only show the
data for a window size of up to three source statements be-
cause signiﬁcantly fewer rules can be learned beyond 3 source
statements. As the result shows, 13.16% of new rules can be
learned from a window size of 2 and 3 source statements. The
rules learned from window size 3 and beyond are less than
3.31%. So, a larger learning window is not necessary.
Figure 12 shows the distribution of the rules we learned
using our enhanced learning scheme. On average, 16.84%
of the learned rules are register-related (marked as register),
while 8.16% are condition-code related (marked as condi-
tion) rules. We ﬁnd that many constrained-equivalent rules
related to local registers are used to load values from the
memory before some computation, and are stored back to the
memory after the computation. This is because RISCs are
primarily "load/store" architectures, i.e. values in memory
must be loaded into registers before computation and stored
back to memory when the computation is completed. So many
0 %
20 %
40 %
60 %
80 %
100 %
perlbenchbzip2gcc mcf gobmkhmmersjenglibquantumh264refomnetppastarxalangeomean
Distribution of 
 Rules Learned from 
Sliding Window
window size 1 window size 2 window size 3
Figure 11: Rules distribution of sliding windows.
0 %
20 %
40 %
60 %
80 %
100 %
perlbenchbzip2gcc mcf gobmkhmmersjenglibquantumh264refomnetppastarxalangeomean
Distribution of Learned Rules
origin register condition
Figure 12: Distribution of rules learned by enhanced learning.
temporary/local registers are used. Another observation is that
the amount of register reuse is minimal on RISCs, only 3.57%
in total. So, only in rare situations, the compiler will use mul-
tiple registers instead of only one register. Such behavior is
reﬂected in the use of the learned rules in the application
phase.
5.4 Performance Overhead of Online Analysis
As the lightweight dynamic analysis is needed in the applica-
tion of the constrained-equivalent translation rules, its runtime
overhead needs to be evaluated. Figure 13 shows such runtime
overhead with and without dynamic analysis. To measure such
overheads, we collected the performance data with original
approach and compare them with those with only the online
analysis but without applying the constrained-equivalent rules.
As Figure 13 shows, the dynamic analysis will introduce very
little overhead, which is less than 1% on average. The low
overhead is due to two main reasons. First, the dynamic analy-
sis typically only needs to check a few registers and condition
codes. And the percentage of the rules that requires dynamic
analysis is not very high. Second, the relaxed register and
condition-code operands are usually updated very quickly, so
only a very small number of instructions need to be analyzed
in practice. Both factors greatly reduce the analysis overhead.
86    2019 USENIX Annual Technical Conference USENIX Association
 1
 1.02
 1.04
 1.06
 1.08
 1.1
perlbenchbzip2gcc mcf gobmkhmmersjenglibquantumh264refomnetppastarxalangeomean
Dynamic Analysis Overhead
Figure 13: The performance overhead of dynamic analysis.
6 Related Work
To improve the efﬁciency of the translated host binaries,
many manual optimization techniques have been proposed.
For example, some try to efﬁciently translate guest single-
instruction-multiple-data (SIMD) instructions [7, 10, 16]. An-
other work proposes to leverage host hardware features to
apply post-optimization to host binary code after the trans-
lation [30]. Some recent work also proposes to optimize
dynamically-generated guest binary code [8]. Different from
those approaches, most of which rely on manually constructed
translation rules, our enhanced learning-based approach pro-
posed in this paper can automatically learn binary translation
rules.
Previous work in [1] also proposes to use peephole super-
optimizer to generate binary translation rules for static binary
translators. For each potential guest code sequence, an ex-
haustive search is employed to explore all possible sequences
of host instructions to examine their equivalence. However, it
takes a very long time to collect sufﬁcient translation rules,
i.e., could be up to one week as mentioned in the paper. More-
over, due to the exponential increase in the number of possible
instruction sequences, this approach can only generate trans-
lation rules with a guest code sequence of up to 3 instructions.
This can signiﬁcantly limit the quality of the generated trans-
lation rules because many high-quality translation rules have
more than 3 guest instructions.
Although the learning-based approach was originally pro-
posed in [23], our enhanced learning-based approach differs
from the original approach in a signiﬁcant way. Our proposed
enhanced approach allows relaxation of semantic equivalence,
thus can learn constrained-equivalent translation rules while
the original approach simply discards them. These relaxed
translation rules can improve the total coverage of the guest
binaries and improve the yield of rule generation. More im-
portantly, these constrained-equivalent translation rules can
further improve the performance of the translated host binary
code.
Another DBT system, HQEMU [9], which is also based
on QEMU, translates guest binary code into LLVM interme-
diate representation (IR) and then leverages LLVM JIT to
generate more optimized binary code. However, the overhead
introduced by the LLVM optimization can offset the beneﬁt
gained from the optimized host binary code, especially for
short-running guest binaries. Moreover, due to the lack of
source-level information in the LLVM IR translated from
the guest binary code, e.g., type information, it is quite chal-
lenging to take full advantage of the LLVM optimization. In
contrast, the translation overhead for applying the learned
translation rules are much smaller, and no additional informa-
tion is required to apply the learned rules.
There has been a lot of research to improve the performance
of the DBT system itself [3–5, 11, 12, 24, 25]. These methods
can typically be used in conjunction with our approach to
further improve their performance.
7 Conclusion
As one of the core enabling technologies, DBT has been ex-
tensively used in many important applications. To improve
the efﬁciency of DBT systems, this paper proposes an en-
hanced learning-based approach, which can automatically
learn optimized binary translation rules. The learned transla-
tion rules can then be applied to a DBT system to generate
more efﬁcient host binary code. Compared to the original
learning approach, our enhanced learning-based approach re-
laxes the semantic equivalence requirements to allow more
efﬁcient constrained-equivalent translation rules. We redesign
the original learning process and the veriﬁcation engine to
accommodate such constrained equivalence. Moreover, to
preserve the correct semantics of the translated code when
such constrained-equivalent translation rules are applied, a
lightweight online analysis is employed in the enhanced DBT
system to check the constraining conditions. The constrained-
equivalent translation rules are applied only when the con-
straining conditions are satisﬁed. We have implemented the
proposed approach in a prototype and extended a widely-used
DBT system, i.e., QEMU, to accept such enhanced translation
rules through learning.
Experimental results on SPEC CINT2006 show that the
proposed approach can improve the dynamic coverage of
the translation from 55.7% to 69.1% and the static coverage
from 52.2% to 61.8%, compared to the original approach.
Moreover, up to 1.65X performance speedup with an average
of 1.19X are observed.
Acknowledgments
We are very grateful to our shepherd, Edouard Bugnion,
and the anonymous reviewers for their valuable feed-
back and comments. This work is supported in part by
the National Natural Science Foundation of China (No.
61672160), Shanghai Municipal Science and Technology Ma-
jor Project (No.2018SHZDZX01) and ZJLab, Shanghai Sci-
USENIX Association 2019 USENIX Annual Technical Conference    87
ence and Technology Development Funds (17511102200)
and the National Science Foundation under the grant number
CNS-1514444.
References
[1] Sorav Bansal and Alex Aiken. Binary translation us-
ing peephole superoptimizers. In Proceedings of the
8th USENIX Conference on Operating Systems Design
and Implementation, OSDI’08, pages 177–192, Berke-
ley, CA, USA, 2008. USENIX Association.
[2] Fabrice Bellard. QEMU, a Fast and Portable Dynamic
Translator. In Proceedings of the Annual Conference on
USENIX Annual Technical Conference, ATC ’05, pages
41–46, Berkeley, CA, USA, 2005. USENIX Association.
[3] Chao-Jui Chang, Jan-Jan Wu, Wei-Chung Hsu,
Pangfeng Liu, and Pen-Chung Yew. Efﬁcient memory
virtualization for cross-isa system mode emulation.
In Proceedings of the 10th ACM SIGPLAN/SIGOPS
International Conference on Virtual Execution Environ-
ments, VEE ’14, pages 117–128, New York, NY , USA,
2014. ACM.
[4] Emilio G. Cota, Paolo Bonzini, Alex Bennée, and Luca P.
Carloni. Cross-isa machine emulation for multicores.
In Proceedings of the 2017 International Symposium
on Code Generation and Optimization, CGO ’17, pages
210–220, Piscataway, NJ, USA, 2017. IEEE Press.
[5] Amanieu D’Antras, Cosmin Gorgovan, Jim Garside, and
Mikel Luján. Low overhead dynamic binary translation
on arm. In Proceedings of the 38th ACM SIGPLAN
Conference on Programming Language Design and Im-
plementation, PLDI 2017, pages 333–346, New York,
NY , USA, 2017. ACM.
[6] Peter Feiner, Angela Demke Brown, and Ashvin Goel.
Comprehensive Kernel Instrumentation via Dynamic
Binary Translation. In Proceedings of the Seventeenth
International Conference on Architectural Support for
Programming Languages and Operating Systems, ASP-
LOS XVII, pages 135–146, New York, NY , USA, 2012.
ACM.
[7] Sheng-Yu Fu, Ding-Yong Hong, Yu-Ping Liu, Jan-Jan
Wu, and Wei-Chung Hsu. Dynamic translation of struc-
tured loads/stores and register mapping for architectures
with simd extensions. In Proceedings of the 18th ACM
SIGPLAN/SIGBED Conference on Languages, Compil-
ers, and Tools for Embedded Systems , LCTES 2017,
pages 31–40, New York, NY , USA, 2017. ACM.
[8] Byron Hawkins, Brian Demsky, Derek Bruening, and
Qin Zhao. Optimizing binary translation of dynami-
cally generated code. In Proceedings of the 13th Annual
IEEE/ACM International Symposium on Code Genera-
tion and Optimization, CGO ’15, pages 68–78, Wash-
ington, DC, USA, 2015. IEEE Computer Society.
[9] Ding-Yong Hong, Chun-Chen Hsu, Pen-Chung Yew,
Jan-Jan Wu, Wei-Chung Hsu, Pangfeng Liu, Chien-Min
Wang, and Yeh-Ching Chung. Hqemu: a multi-threaded
and retargetable dynamic binary translator on multicores.
In Proceedings of the Tenth International Symposium
on Code Generation and Optimization, pages 104–113.
ACM, 2012.
[10] Ding-Yong Hong, Yu-Ping Liu, Sheng-Yu Fu, Jan-Jan
Wu, and Wei-Chung Hsu. Improving simd parallelism
via dynamic binary translation. ACM Trans. Embed.
Comput. Syst., 17(3):61:1–61:27, February 2018.
[11] Chun-Chen Hsu, Pangfeng Liu, Jan-Jan Wu, Pen-Chung
Yew, Ding-Yong Hong, Wei-Chung Hsu, and Chien-Min
Wang. Improving dynamic binary optimization through
early-exit guided code region formation. In Proceedings
of the 9th ACM SIGPLAN/SIGOPS International Con-
ference on Virtual Execution Environments, VEE ’13,
pages 23–32, New York, NY , USA, 2013. ACM.
[12] Ning Jia, Chun Yang, Jing Wang, Dong Tong, and Keyi
Wang. Spire: Improving dynamic binary translation
through spc-indexed indirect branch redirecting. In Pro-
ceedings of the 9th ACM SIGPLAN/SIGOPS Interna-
tional Conference on Virtual Execution Environments,
VEE ’13, pages 1–12, New York, NY , USA, 2013. ACM.
[13] Piyus Kedia and Sorav Bansal. Fast Dynamic Binary
Translation for the Kernel. InProceedings of the Twenty-
Fourth ACM Symposium on Operating Systems Princi-
ples, SOSP ’13, pages 101–115, New York, NY , USA,
2013. ACM.
[14] Dohyeong Kim, William N. Sumner, Xiangyu Zhang,
Dongyan Xu, and Hira Agrawal. Reuse-oriented Re-
verse Engineering of Functional Components from x86
Binaries. In Proceedings of the 36th International Con-
ference on Software Engineering , ICSE 2014, pages
1128–1139, New York, NY , USA, 2014. ACM.
[15] Vladimir Kiriansky, Derek Bruening, and Saman P. Ama-
rasinghe. Secure Execution via Program Shepherding.
In Proceedings of the 11th USENIX Security Symposium,
pages 191–206, Berkeley, CA, USA, 2002. USENIX As-
sociation.
[16] Jianhui Li, Qi Zhang, Shu Xu, and Bo Huang. Optimiz-
ing dynamic binary translation for simd instructions. In
Proceedings of the International Symposium on Code
Generation and Optimization, CGO ’06, pages 269–280,
Washington, DC, USA, 2006. IEEE Computer Society.
88    2019 USENIX Annual Technical Conference USENIX Association
[17] Juanru Li, Zhiqiang Lin, Juan Caballero, Yuanyuan
Zhang, and Dawu Gu. K-Hunt: Pinpointing Insecure
Cryptographic Keys from Execution Traces. InProceed-
ings of the 2018 ACM SIGSAC Conference on Computer
and Communications Security, CCS ’18, pages 412–425,
New York, NY , USA, 2018. ACM.
[18] Chi-Keung Luk, Robert Cohn, Robert Muth, Harish
Patil, Artur Klauser, Geoff Lowney, Steven Wallace, Vi-
jay Janapa Reddi, and Kim Hazelwood. Pin: Building
Customized Program Analysis Tools with Dynamic In-
strumentation. In Proceedings of the 2005 ACM SIG-
PLAN Conference on Programming Language Design
and Implementation, PLDI ’05, pages 190–200, New
York, NY , USA, 2005. ACM.
[19] Lorenzo Martignoni, Stephen McCamant, Pongsin
Poosankam, Dawn Song, and Petros Maniatis. Path-
exploration lifting: Hi-ﬁ tests for lo-ﬁ emulators. In Pro-
ceedings of the Seventeenth International Conference
on Architectural Support for Programming Languages
and Operating Systems, ASPLOS XVII, pages 337–348,
New York, NY , USA, 2012. ACM.
[20] Aashish Mittal, Dushyant Bansal, Sorav Bansal, and
Varun Sethi. Efﬁcient Virtualization on Embedded
Power Architecture R⃝Platforms. In Proceedings of the
Eighteenth International Conference on Architectural
Support for Programming Languages and Operating
Systems, ASPLOS ’13, pages 445–458, New York, NY ,
USA, 2013. ACM.
[21] Nicholas Nethercote and Julian Seward. Valgrind: A
Framework for Heavyweight Dynamic Binary Instru-
mentation. In Proceedings of the 28th ACM SIGPLAN
Conference on Programming Language Design and Im-
plementation, PLDI ’07, pages 89–100, New York, NY ,
USA, 2007. ACM.
[22] Daniel Sanchez and Christos Kozyrakis. ZSim: Fast and
Accurate Microarchitectural Simulation of Thousand-
core Systems. In Proceedings of the 40th Annual Inter-
national Symposium on Computer Architecture, ISCA
’13, pages 475–486, New York, NY , USA, 2013. ACM.
[23] Wenwen Wang, Stephen McCamant, Antonia Zhai, and
Pen-Chung Yew. Enhancing Cross-ISA DBT Through
Automatically Learned Translation Rules. In Proceed-
ings of the Twenty-Third International Conference on
Architectural Support for Programming Languages and
Operating Systems, ASPLOS ’18, pages 84–97, New
York, NY , USA, 2018. ACM.
[24] Wenwen Wang, Jiacheng Wu, Xiaoli Gong, Tao Li, and
Pen-Chung Yew. Improving dynamically-generated
code performance on dynamic binary translators. In
Proceedings of the 14th ACM SIGPLAN/SIGOPS In-
ternational Conference on Virtual Execution Environ-
ments, VEE ’18, pages 17–30, New York, NY , USA,
2018. ACM.
[25] Wenwen Wang, Pen-Chung Yew, Antonia Zhai, and
Stephen McCamant. A General Persistent Code Caching
Framework for Dynamic Binary Translation (DBT). In
Proceedings of the 2016 USENIX Conference on Usenix
Annual Technical Conference, USENIX ATC ’16, pages
591–603, Berkeley, CA, USA, 2016. USENIX Associa-
tion.
[26] Wenwen Wang, Pen-Chung Yew, Antonia Zhai, Stephen
McCamant, Youfeng Wu, and Jayaram Bobba. Enabling
Cross-ISA Ofﬂoading for COTS Binaries. In Proceed-
ings of the 15th Annual International Conference on
Mobile Systems, Applications, and Services, MobiSys
’17, pages 319–331, New York, NY , USA, 2017. ACM.
[27] Zhaoguo Wang, Ran Liu, Yufei Chen, Xi Wu, Haibo
Chen, Weihua Zhang, and Binyu Zang. Coremu: A
scalable and portable parallel full-system emulator. In
Proceedings of the 16th ACM Symposium on Princi-
ples and Practice of Parallel Programming, PPoPP ’11,
pages 213–222, New York, NY , USA, 2011. ACM.
[28] Qifan Yang, Zhenhua Li, Yunhao Liu, Hai Long, Yuan-
chao Huang, Jiaming He, Tianyin Xu, and Ennan Zhai.
Mobile Gaming on Personal Computers with Direct An-
droid Emulation. In Proceedings of the 25th Annual
International Conference on Mobile Computing and
Networking, MobiCom ’19, New York, NY , USA, 2019.
ACM.
[29] W. Zhang, X. Ji, Y . Lu, H. Wang, H. Chen, and P. Yew.
Prophet: A parallel instruction-oriented many-core sim-
ulator. IEEE Transactions on Parallel and Distributed
Systems, 28(10):2939–2952, Oct 2017.
[30] Xiaochun Zhang, Qi Guo, Yunji Chen, Tianshi Chen,
and Weiwu Hu. Hermes: A fast cross-isa binary transla-
tor with post-optimization. In Proceedings of the 13th
Annual IEEE/ACM International Symposium on Code
Generation and Optimization, CGO ’15, pages 246–256,
Washington, DC, USA, 2015. IEEE Computer Society.
USENIX Association 2019 USENIX Annual Technical Conference    89

Convolutional Neural Networks over Tree Structures
for Programming Language Processing
Lili Mou,1 Ge Li,1∗ Lu Zhang,1 Tao Wang,2 Zhi Jin1∗
1Software Institute, Peking University ∗Corresponding authors
doublepower.mou@gmail.com, {lige,zhanglu,zhijin}@sei.pku.edu.cn
2Stanford University, twangcat@stanford.edu
Abstract
Programming language processing (similar to natural lan-
guage processing) is a hot research topic in the ﬁeld of soft-
ware engineering; it has also aroused growing interest in the
artiﬁcial intelligence community. However, different from a
natural language sentence, a program contains rich, explicit,
and complicated structural information. Hence, traditional
NLP models may be inappropriate for programs. In this pa-
per, we propose a novel tree-based convolutional neural net-
work (TBCNN) for programming language processing, in
which a convolution kernel is designed over programs’ ab-
stract syntax trees to capture structural information. TBCNN
is a generic architecture for programming language process-
ing; our experiments show its effectiveness in two differ-
ent program analysis tasks: classifying programs according
to functionality, and detecting code snippets of certain pat-
terns. TBCNN outperforms baseline methods, including sev-
eral neural models for NLP .
Introduction
Researchers from various communities are showing growing
interest in applying artiﬁcial intelligence (AI) techniques to
solve software engineering (SE) problems (Dietz et al. 2009;
Bettenburg and Begel 2013; Hao et al. 2013). In the area of
SE, analyzing program source code—called programming
language processing in this paper—is of particular impor-
tance.
Even though computers can run programs, they do not
truly “understand” programs. Analyzing source code pro-
vides a way of estimating programs’ behavior, functionality,
complexity, etc. For instance, automatically detecting source
code snippets of certain patterns help programmers to dis-
cover buggy or inefﬁcient algorithms so as to improve code
quality. Another example is managing large software repos-
itories, where automatic source code classiﬁcation and tag-
ging are crucial to software reuse. Programming language
processing, in fact, serves as a foundation for many SE tasks,
e.g., requirement analysis (Ghabi and Egyed 2012), software
development and maintenance (Bettenburg and Begel 2013).
Hindle et al. (2012) demonstrate that programming lan-
guages, similar to natural languages, also contain abundant
Copyright c⃝ 2016, Association for the Advancement of Artiﬁcial
Intelligence (www.aaai.org). All rights reserved.
statistical properties, which are important for program anal-
ysis. These properties are difﬁcult to capture by humans,
but justify learning-based approaches for programming lan-
guage processing. However, existing machine learning pro-
gram analysis depends largely on feature engineering, which
is labor-intensive and ad hoc to a speciﬁc task, e.g., code
clone detection (Chilowicz, Duris, and Roussel 2009), and
bug detection (Steidl and Gode 2013). Further, evidence
in the machine learning literature suggests that human-
engineered features may fail to capture the nature of data,
so they may be even worse than automatically learned ones.
The deep neural network, also known asdeep learning,i s
a highly automated learning machine. By exploring multi-
ple layers of non-linear transformation, the deep architec-
ture can automatically learn complicated underlying fea-
tures, which are crucial to the task of interest. Over the
past few years, deep learning has made signiﬁcant break-
throughs in various ﬁelds, such as speech recognition (Dahl,
Mohamed, and Hinton 2010), computer vision (Krizhevsky,
Sutskever, and Hinton 2012), and natural language process-
ing (Collobert and Weston 2008).
Despite some similarities between natural languages and
programming languages, there are also obvious differences
(Pane, Ratanamahatana, and Myers 2001). Based on a for-
mal language, programs contain rich and explicit structural
information. Even though structures also exist in natural
languages, they are not as stringent as in programs. Pinker
(1994) illustrates an interesting example, “The dog the stick
the ﬁre burned beat bit the cat.” This sentence complies
with all grammar rules, but too many attributive clauses
are nested. Hence, it can hardly be understood by people
due to the limitation of human intuition capacity. On the
contrary, three nested loops are common in programs. The
parse tree of a program, in fact, is typically much larger
than that of a natural language sentence—there are approx-
imately 190 nodes on average in our experiment, whereas
a sentence comprises only 20 words in a sentiment analy-
sis dataset (Socher et al. 2013). Further, the grammar rules
“alias” neighboring relationships among program compo-
nents. The statements inside and outside a loop, for exam-
ple, do not form one semantic group, and thus are not se-
mantically neighboring. On the above basis, we think more
effective neural models are in need to capture structural in-
formation in programs.
Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16)
1287
In this paper, we propose a novel Tree-Based Convolu-
tional Neural Network (TBCNN) based on programs’ ab-
stract syntax trees (ASTs). We also introduce the notion
of “continuous binary trees” and apply dynamic pooling to
cope with ASTs of different sizes and shapes. The TBCNN
model is a generic architecture, and is applied to two SE
tasks in our experiments—classifying programs by function-
alities and detecting code snippets of certain patterns. It out-
performs baseline methods in both tasks, including the re-
cursive neural network (Socher et al. 2011b) proposed for
NLP . To the best of our knowledge, this paper is also the ﬁrst
to apply deep neural networks to the ﬁeld of programming
language processing.
1
Related Work
Deep neural networks have made signiﬁcant breakthroughs
in many ﬁelds. Stacked restricted Boltzmann machines and
autoencoders are successful pretraining methods (Hinton,
Osindero, and Teh 2006; Bengio et al. 2006). They explore
the underlying features of data in an unsupervised manner,
and give a more meaningful initialization of weights for later
supervised learning with deep neural networks. These ap-
proaches work well with generic data (e.g. data located in a
manifold embedded in a certain dimensional space), but they
may not be suitable for programming language processing,
because programs contain rich structural information. Fur-
ther, AST structures also vary largely among different data
samples (programs), and hence they cannot be fed directly
to a ﬁxed-size network.
To capture explicit structures in data, it may be impor-
tant and beneﬁcial to integrate human priors to the net-
works (Bengio, Courville, and Vincent 2013). One exam-
ple is convolutional neural networks (CNNs, LeCun et al.
1995; Krizhevsky, Sutskever, and Hinton 2012), which spec-
ify spatial neighboring information in data. CNNs work with
signals of a certain dimension (e.g., images); they also fail
to capture tree-structural information as in programs.
Socher et al. (2013, 2011b) propose a recursive neural net-
work (RNN) for NLP . Although structural information may
be coded to some extent in RNNs, the major drawback is
that only the root features are used for supervised learn-
ing, which buries illuminating information under a compli-
cated neural architecture. RNNs also suffer from the difﬁ-
culty of training due to the long dependency path during
back-propagation (Bengio, Simard, and Frasconi 1994).
Subsequent work. After the preliminary version of this
paper was preprinted on arXiv,
2 Zaremba and Sutskever
(2014) use recurrent neural networks to estimate the output
of restricted python programs. Piech et al. (2015) build re-
cursive networks on Hoare triples. Regarding the proposed
TBCNN, we extend it to process syntactic parse trees of nat-
ural languages (Mou et al. 2015); Duvenaud et al. (2015) ap-
ply a similar convolutional network over graphs to analyze
molecules.
1We make our source code and the collected dataset available
through our website (https://sites.google.com/site/treebasedcnn/).
2On 18 September 2014 (http://arxiv.org/abs/1409.5718v1).
Tree-Based Convolutional Neural Network
Programming languages have a natural tree representation—
the abstract syntax tree (AST). Figure 1a shows the AST of
the code snippet “int a=b+3;”.
3 Each node in the AST
is an abstract component in program source code. A nodep
with childrenc1,··· ,cn represents the constructing process
of the componentp → c1 ···c n.
Figure 1b shows the overall architecture of TBCNN. In
our model, an AST node is ﬁrst represented as a distributed,
real-valued vector so that the (anonymous) features of the
symbols are captured. The vector representations are learned
by a coding criterion in our previous work (Peng et al. 2015).
Then we design a set of subtree feature detectors, called
the tree-based convolution kernel, sliding over the entire
AST to extract structural information of a program. We
thereafter apply dynamic pooling to gather information over
different parts of the tree. Finally, a hidden layer and an out-
put layer are added. For supervised classiﬁcation tasks, the
activation function of the output layer issoftmax.
In the rest of this section, we ﬁrst explain the coding cri-
terion for AST nodes’ representation learning, serving as a
pretraining phase of programming language processing. We
then describe the proposed TBCNN model, including a cod-
ing layer, a convolutional layer, and a pooling layer. We also
provide additional information on dealing with nodes that
have varying numbers of child nodes, as in ASTs, by intro-
ducing the notion of continuous binary trees.
Representation Learning for AST Nodes
V ector representations, sometimes known asembeddings,
can capture underlying meanings of discrete symbols, like
AST nodes. We propose in our previous work (Peng et al.
2015) an unsupervised approach to learn program vector
representations by a coding criterion, which serves as a way
of pretraining.
A generic criterion for representation learning is
“smoothness”—similar symbols have similar feature vec-
tors (Bengio, Courville, and Vincent 2013). For example,
the symbols While and For are similar because both of
them are related to control ﬂow, particularly loops. But they
are different from ID, since ID probably represents some
data. In our scenario, we would like the child nodes’ repre-
sentations to “code” their parent node’s via a single neural
layer, during which both vector representations and coding
weights are learned. Formally, letvec(·) ∈ R
Nf be the fea-
ture representation of a symbol, whereNf is the feature di-
mension. For each non-leaf node p and its direct children
c1,··· ,cn, we would like
vec(p) ≈ tanh
(∑
i
liWcode,i ·vec(ci)+b code
)
(1)
where Wcode,i ∈ RNf ×Nf is the weight matrix corre-
sponding to the node ci; bcode ∈ RNf is the bias. li =
#leaves underci
#leaves underp is the coefﬁcient of the weight. (Weights
Wcode,i are weighted by leaf numbers.)
Because different nodes may have different numbers of
children, the number ofWcode,i’s is not ﬁxed. To overcome
3Parsed by pycparser (https://pypi.python.org/pypi/pycparser/).
1288
Tree-based
(a) Decl (b)
TypeDecl BinaryOp
IdentifierType ID Constant Vector representation
softmax
fully
connected
max
pooling
convolutionand coding
fully
connected
Dynamic
pooling
Hidden Out put
Figure 1: (a) Illustration of an AST, corresponding to theC code snippet “int a=b+3;” It should be notice that our model takes as
input the entire AST of a program, which is typically much larger. (b) The architecture of the Tree-Based Convolutional Neural
Network (TBCNN). The main components in our model include vector representation and coding, tree-based convolution and
dynamic pooling; then a fully-connected hidden layer and an output layer (softmax) are added.
this problem, we introduce the “continuous binary tree,”
where only two weight matricesW
l
code and Wr
code serve as
model parameters.Wi is a linear combination of the two pa-
rameter matrices according to the position of nodei. Details
are deferred to the last part of this section.
The closeness between vec(p) and its coded vector is
measured by Euclidean distance square, i.e.,
d =

vec(p)−tanh
(∑
i
liWcode,i ·vec(ci)+b code
)


2
2
To prevent the pretraining algorithm from learning trivial
representations (e.g., 0’s will give 0 distance but are mean-
ingless), negative sampling is applied like Collobert et al.
(2011). For each pretraining data samplep,c1,··· ,cn,w e
substitute one symbol (eitherp or one ofc’s) with a random
symbol. The distance of the negative sample is denoted as
d
c, which should be at least larger than that of the positive
training sample plus a marginΔ(set to 1 in our experiment).
Thus, the pretraining objective is to
minimize
W l
code,W r
code,bcode,vec(·)
max{0,Δ+ d−dc}
Coding Layer
Having pretrained the feature vectors for all symbols, we
would like to feed them forward to the tree-based convo-
lutional layer for supervised learning. For leaf nodes, they
are just the vector representations learned in the pretraining
phase. For a non-leaf nodep, it has two representations: the
one learned in the pretraining phase (left-hand side of Equa-
tion 1), and the coded one (right-hand side of Equation 1).
They are linearly combined before being fed to the convolu-
tional layer. Letc
1,··· ,cn be the children of nodepand we
denote the combined vector asp.W eh a v e
p = Wcomb1 ·vec(p)
+Wcomb2 ·tanh
(∑
i
liWcode,i ·vec(xi)+b code
)
where Wcomb1,Wcomb2 ∈ RNf ×Nf are the parameters for
combination. They are initialized as diagonal matrices and
then ﬁne-tuned during supervised training.
Tree-based Convolutional Layer
Now that each symbol in ASTs is represented as a dis-
tributed, real-valued vector x ∈ R
Nf , we apply a set of
TOP
LOWER_LEFT LOWER_RIGHT
(a)
(b) (c)
Figure 2: (a) Tree-based convolution. Nodes on the left are
the feature vectors of AST nodes. They are either pretrained
or combined with pretrained and coded vectors. (b) An illus-
tration of 3-way pooling. (c) An analogy to the continuous
binary tree model. In the triangle, the color of a pixel is a
combination of three primary colors; in the convolution pro-
cess, the weight for a node is a combination of three weight
parameters, namelyW
t
conv, Wl
conv, andWr
conv.
ﬁxed-depth feature detectors sliding over the entire tree, de-
picted in Figure 2a. The subtree feature detectors can be
viewed as convolution with a set of ﬁnite support kernels.
We call thistree-based convolution.
Formally, in a ﬁxed-depth window, if there aren nodes
with vector representations x
1,··· ,xn, then the output of
the feature detectors is4
y = tanh
(∑ n
i=1
Wconv,i ·xi +bconv
)
where y,bconv∈RNc , Wconv,i∈RNc×Nf .( Nc is the number
of feature detectors.)0’s are padded for nodes at the bottom
that do not have as many layers as the feature detectors. In
our experiments, the kernel depth is set to 2.
Note that, to deal with varying numbers of children, we
also adopt the notion of continuous binary tree. In this sce-
nario, three weight matrices serve as model parameters,
namely Wt
conv, Wl
conv, and Wr
conv. Wconv,i is a linear combi-
nation of these three matrices (explained in detail in the last
part of this section).
4We used tanh as the activation function in TBCNN mainly
because we hope to encode features to a same semantic space
(−1, 1) during coding. We are grateful to an anonymous reviewer
for reminding us of usingReLU in convolution, and we are happy
to try it in future work.
1289
Dynamic Pooling
After convolution, structural features in an AST are ex-
tracted, and a new tree is generated. The new tree has ex-
actly the same shape and size as the original one, which is
varying among different programs. Therefore, the extracted
features cannot be fed directly to a ﬁxed-size neural layer.
Dynamic pooling (Socher et al. 2011a) is applied to deal
with this problem.
The simplest approach, perhaps, is to pool all features to
one vector. We call this one-way pooling. Concretely, the
maximum value in each dimension is taken from the features
that are detected by tree-based convolution. We also propose
an alternative,three-way pooling, where features are pooled
to 3 parts, TOP , LOWER
LEFT, and LOWERRIGHT, ac-
cording to the their positions in the AST (Figure 2b). As we
shall see from the experimental results, the simple one-way
pooling just works as well as three-way pooling. Therefore
we adopt one-way pooling in our experiments.
After pooling, the features are fully connected to a hid-
den layer and then fed to the output layer (softmax) for su-
pervised classiﬁcation. With the dynamic pooling process,
structural features along the entire AST reach the output
layer with short paths. Hence, they can be trained effectively
by back-propagation.
The “Continuous Binary Tree” Model
As stated, one problem of coding and convolving is that we
cannot determine the number of weight matrices because
AST nodes have different numbers of children.
One possible solution is the continuous bag-of-words
model (CBoW, Mikolov et al., 2013),
5 but position informa-
tion will be lost completely. Such approach is also used in
Hermann and Blunsom (2014). Socher et al. (2014) allocate
a different weight matrix as parameters for each position;
but this method fails to scale up since there will be a huge
number of different positions in ASTs.
In our model, we view any subtree as a “binary” tree, re-
gardless of its size and shape. That is, we have only three
weight matrices as parameters for convolution, and two for
coding. We call it acontinuous binary tree.
Take convolution as an example. The three parameter ma-
trices are W
t
conv, Wl
conv, and Wr
conv. (Superscripts t,l,r refer
to “top,” “left,” and “right.”) For nodexi in a window, its
weight matrix for convolutionWconv,i is a linear combina-
tion ofWt
conv, Wl
conv, andWr
conv, with coefﬁcientsηt
i , ηl
i, and
ηr
i , respectively. The coefﬁcients are computed according to
the relative position of a node in the sliding window. Fig-
ure 2c is an analogy to the continuous binary tree model.
The equations for computingη’s are listed as follows.
• η
t
i = di−1
d−1 (di: the depth of the nodei in the sliding win-
dow; d: the depth of the window.)
• ηr
i =( 1− ηt
i )pi−1
n−1 .( pi: the position of the node;n: the
total number ofp’s siblings.)
• ηl
i =( 1−ηt
i )(1−ηr
i )
5In their original paper, they do not deal with varying-length
data, but their method extends naturally to this scenario. Their
method is also mathematically equivalent to average pooling.
Likewise, the continuous binary tree for coding has two
weight matrices Wl
code and Wr
code as parameters. The details
are not repeated here.
To sum up, the entire parameter set for TBCNN isΘ=
{Wl
code,W r
code,Wcomb1,Wcomb2,W t
conv,W l
conv,W r
conv,Whid,
Wout,bcode,bconv,bhid,bout,vec(·)}, where Whid, Wout, bhid,
and bout are the weights and biases for the hidden and
output layers. To set up supervised training,Wl
code, Wr
code,
bcode, and vec(·) are derived from the pretraining phase;
Wcomb1 and Wcomb2 are initialized as diagonal matrices;
other parameters are initialized randomly. We apply the
cross-entropy loss and use stochastic gradient descent,
computed by back-propagation.
Experiments
We ﬁrst assess the learned vector representations both quali-
tatively and quantitatively. Then we evaluate TBCNN in two
supervised learning tasks, and conduct model analysis.
The dataset of our experiments comes from a pedagogi-
cal programming open judge (OJ) system.
6 There are a large
number of programming problems on the OJ system. Stu-
dents submit their source code as the solution to a certain
problem; the OJ system automatically judges the validity of
submitted source code by running the program. We down-
loaded the source code and the corresponding programming
problems (represented as IDs) as our dataset.
Unsupervised Program Vector Representations
We applied the coding criterion of pretraining to allC code
in the OJ system, and obtained AST nodes’ vector represen-
tations.
Qualitative analysis.Figure 3a illustrates the hierarchical
clustering result based on a subset of AST nodes. As demon-
strated, the symbols mainly fall into three categories: (1)
BinaryOp, ArrayRef, ID, Constant are grouped to-
gether since they are related to data reference/manipulation;
(2) For, If, While are similar since they are related to
control ﬂow; (3)ArrayDecl, FuncDecl, PtrDecl are
similar since they are declarations. The result is quite sen-
sible because it is consistent with human understanding of
programs.
Quantitative analysis. We also evaluated pretraining’s
effect on supervised learning by feeding the learned repre-
sentations to a program classiﬁcation task. (See next subsec-
tion.) Figure 3b plots the learning curves of both training and
validation, which are compared with random initialization.
Unsupervised vector representation learning accelerates the
supervised training process by nearly1/3, showing that pre-
training does capture underlying features of AST nodes, and
that they can emerge high-level features spontaneously dur-
ing supervised learning. However, pretraining has a limited
effect on the ﬁnal accuracy. One plausible explanation is that
the number of AST nodes is small: the pycparser,w e
use, distinguishes only 44 symbols. Hence, their representa-
tions can be adequately tuned in a supervised fashion.
6http://programming.grids.cn
1290
(a) (b)
Figure 3: Analysis of vector representations. (a) Hierarchical
clustering based on AST nodes’ vector representations. (b)
Learning curves with and without pretraining.
Statistics Mean Sample std.
# of code lines 36.3 19.0
# of AST nodes 189.6 106.0
Average leaf nodes’ depth in an AST 7.6 1.7
Max depth of an AST 12.3 3.2
Table 1: Statistics of our dataset.
Hyperparameter Value How is the value chosen?
Initial learning rate 0.3 By validation
Learning rate decay None Empirically
Embedding dimension 30 Empirically
Convolutional layers’ dim. 600 By validation
Penultimate layer’s dim. 600 Same as conv layers
l
2 penalty None Empirically
Table 2: TBCNN’s hyperparameters.
Nonetheless, we think the pretraining criterion is effec-
tive and beneﬁcial for TBCNN, because training deep neural
networks is usually time-consuming, especially when tuning
hyperparameters. The pretrained vector representations are
used throughout the experiments below.
Classifying Programs by Functionalities
Task description In software engineering, classifying
programs by functionalities is an important problem for var-
ious software development tasks. For example, in a large
software repository (e.g., SourceForge), software products
are usually organized into categories, a typical criterion for
which is by functionalities. With program classiﬁcation, it
becomes feasible to automatically tag a software component
newly added into the repository, which is beneﬁcial for soft-
ware reuse during the development process.
In our experiment, we applied TBCNN to classify source
code in the OJ system. The target label of a data sample is
one of 104 programming problems (represented as an ID).
That is, programs with a same target label have the same
functionality. We randomly chose exactly 500 programs in
each class, and thus 52,000 samples in total, which were
further randomly split by 3:1:1 for training, validation, and
testing. Relevant statistics are shown in Table 1.
Hyperparameters TBCNN’s hyperparameters are shown
in Table 2. Our competing methods include SVM and a deep
feed-forward neural network based on hand-crafted features,
namely bag-of-words (BoW, the counting of each symbol)
or bag-of-tree (BoT, the counting of 2-layer subtrees). We
also compare our model with the recursive neural network
Group Method Test Accuracy (%)
linear SVM+BoW 52.0
Surface RBF SVM+BoW 83.9
features linear SVM+BoT 72.5
RBF SVM+BoT 88.2
DNN+BoW 76.0
NN-based DNN+BoT 89.7
approaches V ector avg. 53.2
RNN 84.8
Our method TBCNN 94.0
Table 3: The accuracy of 104-label program classiﬁcations.
(RNN, Socher et al. 2011b). Hyperparameters for baselines
are listed as follows.
SVM. The linear SVM has one hyperparameterC; RBF
SVM has two, C and γ. They are tuned by validation over
the set{··· ,1,0.3,0.1,0.03,···} with grid search.
DNN. We applied a 4-layer DNN (including input) em-
pirically. The hidden layers’ dimension is 300, chosen from
{100,300,1000}; learning rates are 0.003 for BoW and 0.03
for BoT, chosen from{0.003,··· ,0.3}with granularity3x.
ℓ
2 regularization coefﬁcient is10−6 for both BoW and BoT,
chosen from {10−7,··· ,10−4} with granularity 10x, and
also no regularization.
RNN. Recursive units are 600-dimensional, as in
our method. The learning rate is chosen from the set
{··· 1.0,0.3,0.1···} , and 0.3 yields the highest validation
performance.
Results Table 3 presents the results in the 104-label
program classiﬁcation experiment. Using SVM with sur-
face features does distinguish different programs to some
extent—for example, a program about string manipulation
is different from, say, matrix operation; also, a difﬁcult pro-
gramming problem necessitates a more complex program,
and thus more lines of code and AST nodes. However, their
performance is comparatively low.
We tried deep feed-forward neural networks on these fea-
tures, and achieved accuracies of 76.0–89.7%, comparable
to SVMs. V ector averaging withsoftmax—another neural
network-based competing method applied in NLP (Socher et
al. 2013; Kalchbrenner, Grefenstette, and Blunsom 2014)—
yields an accuracy similar to a linear classiﬁer built on
BoW features. This is probably because the number of AST
symbols is far fewer than words in natural languages, and
thus the vector representations (provided non-singular) can
be absorbed into the classiﬁer’s weights. Comparing these
approaches with our method, we deem TBCNN’s perfor-
mance boost is not merely caused by using a better clas-
siﬁer (neural networks versus SVM, say), but also the fea-
ture/representation learning nature, which enables automatic
structural feature extraction.
We also applied RNN to the program classiﬁcation
task
7; the RNN’s accuracy is lower than shallow methods
7We do not use the pretrained vector representations, which are
inimical to RNN: the weightWcode codes children’s representation
to its candidate parent’s; adversely, the high-level nodes in pro-
grams (e.g., a function deﬁnition) are typically non-informative.
1291
Classiﬁer Features Accuracy
Rand/majority – 50.0
RBF SVM Bag-of-words 62.3
RBF SVM Bag-of-trees 77.1
TBCNN Learned 89.1
Table 4: Accuracy of detecting bubble sort (in percentage).
Model Variant Validation Acc.
Coding layer→ None 92.3
1-way pooling→ 3-way 94.3
Continuous binary tree→ CBoW 93.1
TBCNN with the best gadgets 94.4
Table 5: Effect of coding, pooling, and the continuous binary
tree.
(SVM+BoT). Taking into consideration experiments in NLP
(Socher et al. 2011b; 2013), we observe a degradation of
RNN’s performance if the tree structure is large.
TBCNN outperforms the above methods, yielding an ac-
curacy of 94%. By exploring tree-based convolution, our
model is better at capturing programs’ structural features,
which is important for program analysis.
Detecting Bubble Sort
Task description To further evaluate our TBCNN model
in a more realistic SE scenario, we used it to detect an un-
healthy code pattern, bubble sort, which can also be regarded
as a (binary) program classiﬁcation task. Detecting source
code of certain patterns is closely related to many SE prob-
lems. In this experiment, bubble sort is thought of as un-
healthy code because it implements an inefﬁcient algorithm.
By identifying such unhealthy code, project managers can
reﬁne the implementations during the maintenance process.
Before the experiment, a volunteer
8 annotated, from the
OJ system, 109 programs that contain bubble sort, and 109
programs that do not contain bubble sort. They were split
1:1 for validation and testing.
Data augmentation To train our TBCNN model, a dataset
of such scale is insufﬁcient. We propose a simple yet useful
data augmentation technique for programs. Concretely, we
used the source code of 4k programs in the OJ system as the
non-bubble sort class. For each program, we randomly sub-
stituted a fragment of program statements with a pre-written
bubble sort snippet. Thus we had 8k data samples in total.
Results We tested our model on the annotated real-world
programs. Note that the test samples were written by real-
world programmers, and thus the styles and forms of bubble
sort snippets may differ from the training set, for example,
sorting an integer array versus sorting a user-deﬁned struc-
ture, and sorting an array versus sorting two arrays simulta-
neously. As we see in Table 4, bag-of-words features are not
illuminating in this classiﬁcation and yield a low accuracy of
62.3%. Bag-of-trees features are better, and achieve 77.06%.
8The volunteer has neither authorship nor a conﬂict of interests.
Figure 4: V alidation accuracy versus the number of convo-
lution units.
Our model outperforms these methods by more than 10%.
This experiment also suggests that neural networks can learn
more robust features than just counting surface statistics.
Model Analysis
We now analyze each gadget of TBCNN quantitatively, with
the 104-label program classiﬁcation as our testbed. We re-
port validation accuracies throughout this part.
Effect of coding layer In the proposed TBCNN model for
program analysis, we represent a non-leaf node by combin-
ing its coded representation and its pretrained one. We ﬁnd
that, the underneath coding layer can also integrate global
information in addition to merely averaging two homoge-
neous sources. If we build a tree-based convolutional layer
directly on the pretrained vector representations, all struc-
tural features are “local,” that is, conﬁned in the convolution
window. The lack of integrating global information leads to
2% degradation in performance. (See the ﬁrst and last rows
in Table 5.)
Layers’ dimensions In our experiments, AST nodes’ vec-
tor representations are set to be 30-dimensional empirically.
We chose this small value because AST nodes have only 44
different symbols. Hence, the dimension needs to be, intu-
itively, smaller than words’ vector representations, e.g., 300
in Mou et al. (2015). The dimension of convolution, i.e.,
the number of feature detectors, was chosen by validation
(Figure 4). We tried several conﬁgurations, among which
600-dimensional convolution results in the highest valida-
tion accuracy. This analysis also veriﬁes that programs have
rich structural information, even though the number of AST
symbols is not large. As the rich semantics are emerged by
different combinations of AST symbols, we are in need of
more feature detectors, that is, a larger convolutional layer.
Effect of pooling layer We tried two pooling methods in
our TBCNN model, and compare them in Table 5 (the sec-
ond and last rows). 3-way pooling is proposed in hope of
preserving features from different parts of the tree. However,
as indicated by the experimental result, the simple 1-way
pooling works just as ﬁne (even 0.1% higher on the valida-
tion set). This suggests that TBCNN is not sensitive to pool-
ing methods, which mainly serve as a necessity for packing
varying sized and shaped data. Further development can be
addressed in future work.
Effect of continuous binary tree The continuous binary
tree is introduced to treat nodes with different numbers of
1292
children, as well as to capture order information of child
nodes. We also implemented the continuous bag-of-words
(CBoW) model, where child nodes’ representations are av-
eraged before convolution. Rows 4 and 5 in Table 5 compare
our proposed continuous binary tree and the above alterna-
tive. The result shows a boost of 1.3% in considering child
nodes’ order information.
Conclusion
In this paper, we applied deep neural networks to the ﬁeld
of programming language processing. Due to the rich and
explicit tree structures of programs, we proposed the novel
Tree-Based Convolutional Neural Network (TBCNN). In
our model, program vector representations are learned by
the coding criterion; structural features are detected by the
convolutional layer; the continuous binary tree and dynamic
pooling enable our model to cope with trees of varying sizes
and shapes. Experimental results show the superiority of our
model to baseline methods.
Acknowledgments
We would like to thank anonymous reviewers for insight-
ful comments; we also thank Xiaowei Sun for annotating
bubble sort programs, Y uxuan Liu for data processing, and
Weiru Liu for discussion on the manuscript. This research
is supported by the National Basic Research Program of
China (the 973 Program) under Grant No. 2015CB352201
and the National Natural Science Foundation of China un-
der Grant Nos. 61421091, 61232015, 61225007, 91318301,
and 61502014.
References
Bengio, Y .; Lamblin, P .; Popovici, D.; and Larochelle, H. 2006.
Greedy layer-wise training of deep networks. InNIPS.
Bengio, Y .; Courville, A.; and Vincent, P . 2013. Representation
learning: A review and new perspectives. IEEE Trans. Pattern
Anal. Mach. Intell.35(8):1798–1828.
Bengio, Y .; Simard, P .; and Frasconi, P . 1994. Learning long-
term dependencies with gradient descent is difﬁcult.IEEE Trans.
Neural Networks 5(2):157–166.
Bettenburg, N., and Begel, A. 2013. Deciphering the story of
software development through frequent pattern mining. InICSE,
1197–1200.
Chilowicz, M.; Duris, E.; and Roussel, G. 2009. Syntax tree
ﬁngerprinting for source code similarity detection. InProc. IEEE
Int. Conf. Program Comprehension, 243–247.
Collobert, R., and Weston, J. 2008. A uniﬁed architecture for
natural language processing: Deep neural networks with multi-
task learning. InICML.
Collobert, R.; Weston, J.; Bottou, L.; Karlen, M.; Kavukcuoglu,
K.; and Kuksa, P . 2011. Natural language processing (almost)
from scratch. JRML 12:2493–2537.
Dahl, G.; Mohamed, A.; and Hinton, G. 2010. Phone recogni-
tion with the mean-covariance restricted Boltzmann machine. In
NIPS.
Dietz, L.; Dallmeier, V .; Zeller, A.; and Scheffer, T. 2009. Lo-
calizing bugs in program executions with graphical models. In
NIPS.
Duvenaud, D.; Maclaurin, D.; Aguilera-Iparraguirre, J.; G´omez-
Bombarelli, R.; Hirzel, T.; Aspuru-Guzik, A.; and Adams, R.
2015. Convolutional networks on graphs for learning molecular
ﬁngerprints. arXiv preprint arXiv:1509.09292.
Ghabi, A., and Egyed, A. 2012. Code patterns for automatically
validating requirements-to-code traces. InASE, 200–209.
Hao, D.; Lan, T.; Zhang, H.; Guo, C.; and Zhang, L. 2013. Is this
a bug or an obsolete test? InProc. ECOOP, 602–628.
Hermann, K., and Blunsom, P . 2014. Multilingual models for
compositional distributed semantics. InACL, 58–68.
Hindle, A.; Barr, E.; Su, Z.; Gabel, M.; and Devanbu, P . 2012. On
the naturalness of software. InICSE, 837–847.
Hinton, G.; Osindero, S.; and Teh, Y . 2006. A fast learning algo-
rithm for deep belief nets.Neural Computation18(7):1527–1554.
Kalchbrenner, N.; Grefenstette, E.; and Blunsom, P . 2014. A
convolutional neural network for modelling sentences. InACL,
655–665.
Krizhevsky, A.; Sutskever, I.; and Hinton, G. 2012. ImageNet
classiﬁcation with deep convolutional neural networks. InNIPS.
LeCun, Y .; Jackel, L.; Bottou, L.; Brunot, A.; Cortes, C.; Denker,
J.; Drucker, H.; Guyon, I.; Muller, U.; and Sackinger, E. 1995.
Comparison of learning algorithms for handwritten digit recogni-
tion. In Proc. Int. Conf. Artiﬁcial Neural Networks.
Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G.; and Dean, J.
2013. Distributed representations of words and phrases and their
compositionality. In NIPS.
Mou, L.; Peng, H.; Li, G.; Xu, Y .; Zhang, L.; and Jin, Z. 2015.
Discriminating neural sentence modeling by tree-based convolu-
tion. In EMNLP, 2315–2325.
Pane, J.; Ratanamahatana, C.; and Myers, B. 2001. Studying
the language and structure in non-programmers’ solutions to pro-
gramming problems.Int. J. Human-Computer Studies54(2):237–
264.
Peng, H.; Mou, L.; Li, G.; Liu, Y .; Zhang, L.; and Jin, Z. 2015.
Building program vector representations for deep learning. In
Proc. 8th Int. Conf. Knowledge Science, Engineering and Man-
agement, 547–553.
Piech, C.; Huang, J.; Nguyen, A.; Phulsuksombati, M.; Sahami,
M.; and Guibas, L. 2015. Learning program embeddings to prop-
agate feedback on student code. InICML.
Pinker, S. 1994. The Language Instinct: The New Science of
Language and Mind. Pengiun Press.
Socher
, R.; Huang, E.; Pennin, J.; Manning, C.; and Ng, A. 2011a.
Dynamic pooling and unfolding recursive autoencoders for para-
phrase detection. InNIPS.
Socher, R.; Pennington, J.; Huang, E.; Ng, A.; and Manning, C.
2011b. Semi-supervised recursive autoencoders for predicting
sentiment distributions. InEMNLP, 151–161.
Socher, R.; Perelygin, A.; Wu, J.; Chuang, J.; Manning, C.; Ng,
A.; and Potts, C. 2013. Recursive deep models for semantic
compositionality over a sentiment treebank. In EMNLP, 1631–
1642.
Socher, R.; Karpathy, A.; Le, Q.; Manning, C.; and Ng, A. Y .
2014. Grounded compositional semantics for ﬁnding and describ-
ing images with sentences.TACL 2:207–218.
Steidl, D., and Gode, N. 2013. Feature-based detection of bugs
in clones. In7th Int. W orkshop on Software Clones, 76–82.
Zaremba, W., and Sutskever, I. 2014. Learning to execute.arXiv
preprint arXiv:1410.4615.
1293

Jointly Modeling Embedding and Translation to Bridge Video and Language
Yingwei Pan ‡, Tao Mei †, Ting Yao†, Houqiang Li ‡, and Yong Rui †
† Microsoft Research, Beijing, China
‡ University of Science and Technology of China, Hefei, China
panyw.ustc@gmail.com, {tmei, tiyao, yongrui}@microsoft.com, lihq@ustc.edu.cn
Abstract
Automatically describing video content with natural lan-
guage is a fundamental challenge of multimedia. Recurrent
Neural Networks (RNN), which models sequence dynamics,
has attracted increasing attention on visual interpretation.
However, most existing approaches generate a word locally
with given previous words and the visual content, while the
relationship between sentence semantics and visual content
is not holistically exploited. As a result, the generated sen-
tences may be contextually correct but the semantics (e.g.,
subjects, verbs or objects) are not true.
This paper presents a novel uniﬁed framework, named
Long Short-Term Memory with visual-semantic Embedding
(LSTM-E), which can simultaneously explore the learning
of LSTM and visual-semantic embedding. The former aims
to locally maximize the probability of generating the next
word given previous words and visual content, while the
latter is to create a visual-semantic embedding space for
enforcing the relationship between the semantics of the en-
tire sentence and visual content. Our proposed LSTM-E
consists of three components: a 2-D and/or 3-D deep con-
volutional neural networks for learning powerful video rep-
resentation, a deep RNN for generating sentences, and a
joint embedding model for exploring the relationships be-
tween visual content and sentence semantics. The exper-
iments on YouTube2Text dataset show that our proposed
LSTM-E achieves to-date the best reported performance in
generating natural sentences: 45.3% and 31.0% in terms of
BLEU@4 and METEOR, respectively. We also demonstrate
that LSTM-E is superior in predicting Subject-Verb-Object
(SVO) triplets to several state-of-the-art techniques.
1. Introduction
Video has become ubiquitous on the Internet, broadcast-
ing channels, as well as personal devices. This has en-
couraged the development of advanced techniques to ana-
lyze the semantic video content for a wide variety of ap-
plications. Recognition of videos has been a fundamental
... ...
Input Video:
Output Sentence:
• LSTM:  a man is riding a horse.
• LSTM-E: a woman is riding a horse.
• Humans: a woman gallops on a horse. / a woman is riding a 
horse along a road. / the girl rode her brown horse.
Figure 1. Examples of video description generation. Input: a short
video. Output: a natural language sentence describing the main
content of the input video.
challenge of multimedia for decades. Previous research has
predominantly focused on recognizing videos with a pre-
deﬁned yet very limited set of individual words. Thanks
to the recent development of Recurrent Neural Networks
(RNN), researchers have strived to automatically describe
video content with a complete and natural sentence, which
can be regarded as the ultimate goal of video understanding.
Figure 1 shows the examples of video description gen-
eration. Given an input video, the generated sentences are
to describe video content, ideally encapsulating its most in-
formative dynamics. There is a wide variety of video appli-
cations based on the description, ranging from editing, in-
dexing, search, to sharing. However, the problem itself has
been taken as a ground challenge for decades in the research
communities, as the description generation model should
be powerful enough not only to recognize key objects from
visual content, but also discover their spatio-temporal rela-
tionships and the dynamics expressed in a natural language
as well.
Despite the difﬁculty of the problem, there have been
a few attempts to address video description generation
[6, 32, 35], and image caption generation [7, 14, 17, 20, 33],
which are mainly inspired by recent advances in machine
translation using Recurrent Neural Networks (RNN) [1, 27].
The standard RNN is a nonlinear dynamical system that
maps sequences to sequences. Although the gradients of
the RNN are easy to compute, RNN models are difﬁcult to
arXiv:1505.01861v1  [cs.CV]  7 May 2015
train, especially when the problems have long-range tempo-
ral dependencies, due to the well-known “vanishing gradi-
ent” effect [4, 21]. As such, the Long Short-Term Memory
(LSTM) model was proposed to overcome the vanishing
gradients problem by incorporating memory units, which
allow the network to learn when to forget previous hid-
den states and when to update hidden states [12]. LSTM
has been successfully adopted to several tasks, e.g., speech
recognition [9], language translation [1] and image caption
[20, 33]. Thus, we follow this elegant recipe and use LSTM
as our RNN model to generate the video sentence in this
paper.
Moreover, existing video description generation ap-
proaches mainly optimize the next word given the input
video and previous words locally, while leaving the rela-
tionship between the semantics of the entire sentence and
video content unexploited. As a result, the generated sen-
tences can suffer from robustness problem. It is often the
case that the output sentence from existing approaches may
be contextually correct but the semantics (e.g., subjects,
verbs or objects) in the sentence are not true. For example,
the sentence generated by LSTM-based model for the video
in Figure 1 is “a man is riding a horse,” which is correct
in logic but the subject “man” is not relevant to the video
content.
To address the above issues, we leverage the semantics
of the entire sentence and visual content to learn a visual-
semantic embedding model, which holistically explores the
relationships in between. Speciﬁcally, we present a novel
Long Short-Term Memory with visual-semantic Embed-
ding (LSTM-E) framework to bridge video content and nat-
ural language, as shown in Figure 2. Given a video, a 2-D
and/or 3-D Convolution Neural Networks (CNN) is utilized
to extract visual features of selected video frames/clips,
while the video representation is produced by mean pool-
ing over these visual features. Then, a LSTM for generat-
ing video sentence and a visual-semantic embedding model
are jointly learnt based on the video representation and sen-
tence semantics. The spirit of LSTM-E is to generate video
sentence from the viewpoint of mutual reinforcement be-
tween coherence and relevance. Coherence expresses the
contextual relationships among the generated words with
video content which is optimized in LSTM, whilerelevance
conveys the relationship between the semantics of the entire
sentence and video content which is measured in the visual-
semantic embedding.
In summary, this paper makes the following contribu-
tions:
•We present an end-to-end deep model for automatic
video description generation, which incorporates both
spatial and temporal structures underlying video.
•We propose a novel Long Shot-Term Memory with
visual-semantic Embedding (LSTM-E) framework,
which considers both the contextual relationship
among the words in sentence, and the relationship be-
tween the semantics of the entire sentence and video
content, for generating natural language of a given
video.
•The proposed model is evaluated on the popular
Youtube2Text corpus and outperforms the-state-of-
the-art in terms of both Subject-Verb-Object (SVO)
triplet prediction and sentence generation.
The remaining parts of the paper are organized as fol-
lows. Section 2 reviews related work. Section 3 presents
the problem of video description generation, while Section
4 details our solution of jointly modeling embedding and
translation. In Section 5, we provide empirical evaluations,
followed by the discussions and conclusions in Section 6.
2. RELATED WORK
There are mainly two directions for translation from vi-
sual content. The ﬁrst direction predeﬁnes the special rule
for language grammar and split sentence into several parts
(e.g. subject, verb, object). With such sentence fragments,
many works align each part with visual content and then
generate the sentence for corresponding visual content: [19]
use Conditional Random Field (CRF) model to produce
sentence for image and in [8], a Markov Random Field
(MRF) model is proposed to attach a descriptive sentence
to the given image. For video translation, Rohrbach et al.
[23] learn a CRF to model the relationships between dif-
ferent components of the input video and generate descrip-
tions for video. Guadarrama et al. [11] use semantic hier-
archies to choose an appropriate level of the speciﬁcity and
accuracy of sentence fragments. This direction is highly de-
pended on the templates of sentence and can only generate
sentence with syntactical structure.
Another direction is to learn the probability distribution
in the common space of visual content and textual sentence.
In this direction, several works explore such probability dis-
tribution using topic models [3, 13] and neural networks
[6, 16, 20, 31, 32, 33, 35]. They can generate sentence
more ﬂexibly. Most recently, several methods have been
proposed for visual to sentence task based on the neural
networks and most of them are utilizing the RNN due to
its successful use in sequence to sequence learning for ma-
chine translation [1, 27]. Kiros et al. [16] ﬁrstly take the
neural networks to generate sentence for image by propos-
ing a image-text multimodal log-bilinear neural language
model. In another work by Mao et al. [20], a multimodal
Recurrent Neural Networks (m-RNN) model is proposed
for image to caption, which directly models the probability
of generating a word given previous words and image. In
[33], Vinyals et al. propose an end-to-end neural networks
... ...
CNN
CNN
CNN
CNN
CNN
CNN
Mean Pooling
3D 
CNN
3D 
CNN
3D 
CNN
Mean Pooling
Video Embedding
...
...
Coherence LossJoint 
Learning
Relevance Loss Sentence Embedding
...
...
...
...
LSTM LSTM LSTM LSTM LSTM
a man is #end
#start a man motorcycle
“a man is riding a motorcycle”
Figure 2. An overview of our LSTM-E framework with a language generating LSTM and a visual-semantic embedding model (better
viewed in color). The video representation is produced by mean pooling over the visual features of frames/clips, extracted by a 2-D/3-D
CNN. The relevance loss is to measure the relationships between the semantics of the entire sentence and video content in the embedding
space, while the coherence loss is to characterize the contextual relationships among the generated words in the sentence in LSTM. Both
LSTM and visual-semantic embedding are jointly learnt by minimizing the two losses.
system by utilizing LSTM to generate sentence for image.
For video translation, an end-to-end LSTM based model is
proposed in [32], which only reads the sequence of video
frames and then generates a natural sentence. The model is
further extended by inputting both frames and optical ﬂow
in [31]. Yaoet al. propose to use a 3-D convolutional neural
networks for modeling video clip dynamic temporal struc-
ture and an attention mechanism to select the most relevant
temporal clips [35]. Then, the resulting video representa-
tions are fed into the text-generating RNN.
Our work belongs to the second direction. However,
most of the above approaches in this direction mainly focus
on optimizing the contextual relationship among words to
generate sentence given visual content, while the relation-
ship between the semantics of the entire sentence and vi-
sual content is not fully explored. Our work is different that
we claim to generate video sentence by jointly exploiting
the two relationships, which characterize the complemen-
tary properties of coherence and relevance of a generated
sentence, respectively.
3. Video Description Generation
Our goal is to generate language sentences for videos.
What makes a good sentence? Beyond describing important
persons, objects, scenes, and actions by words, it must also
convey how one word leads to the next. Speciﬁcally, we de-
ﬁne a good sentence as a coherent chain of words in which
each word inﬂuences the next through contextual informa-
tion. Furthermore, the semantics of the entire sentence must
be relevant to the video content. We begin this Section by
presenting the problem formulation, and followed by the
proposal of two losses on measuring coherence and rele-
vance, respectively.
3.1. Problem Formulation
Suppose we have a videoVwith Nv sample frames/clips
(uniform sampling) to be described by a textual sentenceS,
where S= {w1,w2,...,w Ns }consisting of Ns words. Let
v ∈RDv and wt ∈RDw denote the Dv-dimensional vi-
sual features of a video Vand the Dw-dimensional textual
features of the t-th word in sentence S, respectively. As a
sentence consists of a sequence of words, a sentence can be
represented by a Dw×Ns matrix W ≡[w1,w2,..., wNs ],
with each word in the sentence as its column vector. Fur-
thermore, we denote another feature vector s in the text
space for representing a sentence as a whole.
In the video description generation problem, on one
hand, the generated descriptive sentence must be able to
depict the main contents of a video precisely, and on the
other, the words in the sentence should be organized coher-
ently in language. Therefore, we can formulate the video
description generation problem by minimizing the follow-
ing energy loss function
E(V,S) = (1−λ) ×Er(v,s) +λ×Ec(v,W) , (1)
where Er(v,s) and Ec(v,W) are the relevance loss and
coherence loss, respectively. The former measures the rel-
evance degree of the video content and sentence semantics
and we build an visual-semantic embedding for this pur-
pose, which is introduced in Section 3.2. The latter es-
timates the contextual relationships among the generated
words in the sentence and we use LSTM-based RNN as
our model, which is presented in Section 3.3. The tradeoff
between these two competing losses is captured by linear
fusion with a positive parameter λ.
3.2. Visual-Semantic Embedding
In order to effectively represent the visual content of a
video, we ﬁrst use a 2-D and/or 3-D deep convolutional
neural networks (CNN), which is powerful to produce a
rich representation of each sampled frame/clip from the
video. Then, we perform “mean pooling” process over all
the frames/clips to generate a single Dv-dimension vector
v for each video V. The sentence feature s is produced by
the feature vectors wt(t= 1,2,...,N s) of each word in the
sentence. We ﬁrst encode each word wt as “one-hot” vector
(binary index vector in a vocabulary), thus the dimension of
feature vector wt, i.e. Dw, is the vocabulary size. Then the
binary TF weights are calculated over all words of the sen-
tence to produce the integrated representation of the entire
sentence, denoted by s ∈RDw , with the same dimension
of wt.
We assume that a low-dimensional embedding exists for
the representation of video and sentence. The linear map-
ping function can be derived from this embedding by
ve = Tvv and se = Tss, (2)
where De is the dimensionality of the embedding, and
Tv ∈RDe×Dv and Ts ∈RDe×Ds are the transformation
matrices that project the video content and semantic sen-
tence into the common embedding, respectively.
To measure the relevance between the video content and
semantic sentence, one natural way is to compute the dis-
tance between their mappings in the embedding. Thus, we
deﬁne the relevance loss as
Er(v,s) =∥Tvv −Tss∥2
2 . (3)
We strengthen the relevance between video content and
semantic sentence by minimizing the relevance loss. As
such, the generated sentence is expected to better manifest
the semantics of videos.
3.3. Translation by Sequence Learning
Inspired by the recent successes of probabilistic se-
quence models leveraged in statistical machine translation
[6, 33], we deﬁne our coherence loss as
Ec(v,W) =−log Pr (W|v). (4)
Assuming that a generative model of W that produces
each word in the sequence in order, the log probability of
the sentence is given by the sum of the log probabilities
over the word and can be expressed as:
log Pr (W|v) =
Ns∑
t=0
log Pr (wt|v,w0,..., wt−1). (5)
By minimizing the coherence loss, the contextual relation-
ship among the words in the sentence can be guaranteed,
making the sentence coherent and smooth.
In video description generation task, both the relevance
loss and coherence loss need to be estimated to complete the
whole energy function. We will present a solution to jointly
model the two losses in a deep recurrent neural networks in
the next sections.
4. Joint Modeling Embedding and Translation
Following the relevance and coherence criteria, this
work proposes a Long Short-Term Memory with visual-
semantic Embedding (LSTM-E) model for video descrip-
tion generation. The basic idea of LSTM-E is to translate
the video representation from a 2-D and/or 3-D deep con-
volutional network to the desired output sentence by using
LSTM-type RNN model. Figure 2 shows an overview of
LSTM-E model. In particular, the training of LSTM-E is
performed by simultaneously minimizing therelevance loss
and coherence loss. Therefore, the formulation presented
in Eq.(1) is equivalent to minimizing the following energy
function
E(V,S) = (1−λ) ×∥Tvv −Tss∥2
2 −
λ×
Ns∑
t=0
log Pr (wt|v,w0,..., wt−1; θ; Tv; Ts) ,
(6)
where θare the parameters of our LSTM-E models.
In the following, we will ﬁrst present the architecture
of LSTM memory cell, followed by jointly modeling with
visual-semantic embedding.
4.1. Long Short Term Memory
We brieﬂy introduce the standard Long Short-Term
Memory (LSTM) [12], a variant of RNN, which can cap-
ture long-term temporal information by mapping input se-
quences to a sequence of hidden states and then hidden
states to outputs. To address the vanishing gradients prob-
lem in traditional RNN training, LSTM incorporates a
memory cell which can maintain its states over time and
non-linear gating units which control the information ﬂow
into and out of the cell. As much light has been threw on
LSTM recently, many improvements have been made to the
LSTM architecture on its original formulation [12]. We
adopt the LSTM architecture as described in [36], which
omits the peephole connections in previous work [10].
A diagram of the LSTM unit can be seen in Figure 3. It
consists of a single memory cell, an input activation func-
tion, an output activation function, and three gates (input,
forget and output). The hidden state of the cell is recurrently
connected back to the input and three gates. The mem-
ory cell updates its hidden state by combining the previous
cell state which is modulated by the forget gate and a func-
tion of the current input and the previous output, modulated
by the input gate. The forget gate is a critical component
of the LSTM unit, which can control what to be remem-
bered and what to be forgotten by the cell and somehow can
avoid the gradient from vanishing or exploding when back
propagating through time. Having been updated, the cell
state is mapped to (-1,1) range through an output activation
function which is necessary whenever the cell state is un-
bounded. Finally, the output gate determines how much of
the memory cell ﬂows into the output. These additions to
the single memory cell enable LSTM to capture extremely
complex and long-term temporal dynamics which is impos-
sible for traditional RNN.
The vector formulas for a LSTM layer forward pass are
given below. For timestep t, xt and ht are the input and
output vector respectively, T are input weights matrices,
R are recurrent weight matrices and b are bias vectors.
Logic sigmoid σ(x) = 1
1+e−x and hyperbolic tangent
φ(x) = ex−e−x
ex+e−x are element-wise non-linear activation
functions, mapping real numbers to(0,1) and (−1,1) sepa-
rately. The dot product and sum of two vectors are denoted
with ⊙and +⃝, respectively. Given inputs xt, ht−1 and
ct−1, the LSTM unit updates for timestep tare:
gt = φ(Tgxt + Rght−1 + bg) cellinput
it = σ(Tixt + Riht−1 + bi) inputgate
ft = σ(Tfxt + Rfht−1 + bf) forgetgate
ct = gt ⊙it + ct−1 ⊙ft cellstate
ot = σ(Toxt + Roht−1 + bo) outputgate
ht = φ(ct) ⊙ot celloutput
4.2. LSTM with Visual-Semantic Embedding
By further incorporating a visual-semantic embedding,
our LSTM-E architecture is to jointly model embedding and
translation. In the training stage, given the video-sentence
pair, the inputs of LSTM are the representations of the video
and the words in the sentence after mapping into the embed-
ding. As mentioned above, here we train the LSTM model
to predict each word in the sentence given the embedding
of visual feature for video and previous words. There are
multiple ways that can be used to combine the visual con-
tent and words in LSTM unit updating procedure. The ﬁrst
one is to feed the visual content at each time step as an extra
+
+
.
.
+
.
Cell input
Cell 
Forget 
gate
Output
φ
σ
σ
φ
Output 
gate
.
Input 
recurrent
LSTM unit
Input recurrent
+
.
.
σ
φ
Weighted connections
Input/output activation function
Gate activation function
Function of sum
Function of dot product
Branching point
Connections with time-lag
Unweighted connections
Legend
+
Input 
recurrent
+σ
Input gate
recurrent
Input 
-1
-1
-1
-1
-1
Figure 3. A diagram of an LSTM memory cell.
input for LSTM to emphasize the visual content frequently
among LSTM memory cells. The second one only inputs
the visual content once at the initial step to inform the whole
memory cells in LSTM about the visual content. As empir-
ically veriﬁed in [33], feeding the image at each time yields
inferior results, due to the fact that the network can explic-
itly exploit noise and overﬁts more easily. Therefore, we
adopt the second approach to arrange the inputs into LSTM
in our architecture. Given the video v and its correspond-
ing sentence W ≡[w0,w1,..., wNs ], the LSTM updating
procedure is as following:
x−1 = Tvv (7)
xt = Tswt,t ∈{0,...,N s −1} (8)
ht = f
(
xt)
,t ∈{0,...,N s −1} (9)
where f is the updating function within LSTM unit. Please
note that for the input sentence W ≡{w0,..., wNs }, we
take w0 as the start sign word to inform the beginning of
sentence and wNs as the end sign word which indicates the
end of sentence, both of the special sign words are included
in our vocabulary. Most speciﬁcally, at the initial time step,
the video representation in the embedding is set as the input
for LSTM, and then in the next steps, word embedding xt
will be input into the LSTM along with the previous step’s
hidden state ht−1. In each time step (except the initial step),
we use the LSTM cell output ht to predict the next word.
Here a softmax layer is applied after the LSTM layer to
produce a probability distribution over all the Ds words in
the vocabulary as
Prt+1 (wt+1) =
exp
(
T(wt+1)
h ht
)
∑
w∈W
exp
(
T(w)
h ht
), (10)
where Wis the word vocabulary space, T(w)
h is the param-
eter matrix in softmax layer. Therefore, we can obtain the
next word based on such probability distribution until the
end sign word is emitted.
Accordingly, we deﬁne our loss function as follows:
E(V,S) = (1−λ) ×∥Tvv −Tss∥2
2 −
λ×
Ns∑
t=1
log Prt(wt) . (11)
Let N denote the number of video-sentence pairs in the
training dataset, we have the following optimization prob-
lem:
min
Tv,Ts,Th,θ
1
N
N∑
i=1
(
E(V(i),S(i))
)
+ ∥Tv∥2
2 + ∥Ts∥2
2 + ∥Th∥2
2 + ∥θ∥2
2
, (12)
where the ﬁrst term is the combination of the relevance loss
and coherence loss, while the rest are regularization terms
for video embedding, sentence embedding, softmax layer
and LSTM, respectively.
The above overall objective is optimized over the whole
training video-sentence pairs using stochastic gradient de-
scent. By minimizing this objective function, our LSTM-E
model takes into account both the contextual relationships
among the words in sentence ( coherence) and the relation-
ships between the semantics of the entire sentence and video
content (relevance).
For sentence generation, there are two common strate-
gies to translate the given video. The ﬁrst approach is to
sample the next word from the probability distribution at
each timestep and set its representation in embedding space
as the LSTM input for next timestep until the end sign
word is sampled or the maximum sentence size is reached.
Another method is select the top- k best sentence for each
timestep and sets them as the candidates for next timestep
based on which to generate new top- k best sentence. To
make the generation process concise and efﬁcient, we adopt
the similar way as the latter one but set k as 1. Therefore,
at each timestep, we choose the word with maximum prob-
ability as the predicted word and input its embedded feature
in the next timestep until the model outputs the end sign
word.
5. Experiments
In this section, we will ﬁrst introduce our experimental
setting. Then, the evaluation results compared with state-
of-the-arts on two tasks, i.e., Subject-Verb-Object (SVO)
triplet prediction and natural sentence generation tasks, are
reported. Finally, the effect of tradeoff parameter between
coherence and relevance and the size of hidden layer in
LSTM are presented.
5.1. Experimental Setting
We conduct our experiments mainly on the Microsoft
Research Video Description Corpus (YouTube2Text) [5],
which have been used in several prior works [11, 29, 34] on
action recognition and video description generation tasks.
This video corpus contains 1,970 YouTube snippets which
cover a wide range of daily activities such as “people do-
ing exercises,” “playing music,” and “cooking.” We use the
roughly 40 available English descriptions per video. In our
experiments, following the setting used in prior works on
video description generation [11, 34], we pick 1,200 videos
to be used as training data, 100 videos for validation and
670 videos for testing.
We compare our LSTM-E architecture with two 2-D
CNN of AlexNet [18] and the 19-layer VGG [25] network
both pre-trained on Imagenet ILSVRC12 dataset [24], and
one 3-D CNN of C3D [30] pre-trained on Sports-1M video
dataset [15]. Speciﬁcally, we take the output of 4096-
way fc7 layer from AlexNet, 4096-way fc6 layer from the
19-layer VGG, and 4096-way fc6 layer from C3D as the
frame/clip representation, respectively. The dimensionality
of the visual-semantic embedding space and the size of hid-
den layer in LSTM are both set to 512. The tradeoff param-
eter λ leveraging the relevance loss and coherence loss is
empirically set to 0.7. The sensitivity of λwill be discussed
in Section 5.3.1.
5.2. Performance Comparison
We empirically verify the merit of our LSTM-E model
from two aspects: SVO triplet prediction and sentence gen-
eration for the video-language translation.
5.2.1 Compared Approaches
To fully evaluate our model, we compare our LSTM-E mod-
els with the following non-trivial baseline methods.
•Conditional Random Field (CCF) [34]: CRF model is
developed to incorporate subject-verb and verb-object
pairwise relationship based on the word pairwise co-
occurrence statistics in the sentence pool.
•Canonical Correlation Analysis (CCA) [26]: CCA is
to build the video-language joint space and generate
the SVO triplet by k-nearest-neighbors search in the
sentence pool.
•Factor Graph Model (FGM) [29]: FGM combines
knowledge mined from text corpora with visual con-
Table 1. SVO accuracy: Binary accuracy of SVO triplet predic-
tion. We extract SVO triplets from sentences output by LSTM and
LSTM-E using a dependency parser.
Model S% V% O%
FGM 76.42 21.34 12.39
CRF 77.16 22.54 9.25
CCA 77.16 21.04 10.99
JEM 78.25 24.45 11.95
LSTM 71.19 19.40 9.70
LSTM-E (Alex) 78.66 24.78 10.30
LSTM-E (VGG) 80.30 27.91 12.54
LSTM-E (C3D) 77.31 28.81 12.39
LSTM-E (VGG+C3D) 80.45 29.85 13.88
ﬁdence using a factor graph and performs probabilistic
inference to determine the most likely SVO triplets.
•Joint Embedding Model (JEM) [34]: Proposed most
recently, JEM jointly models video and the corre-
sponding text sentences by minimizing the distance
of the deep video and compositional text in the joint
space.
•Long Shot-Term Memory (LSTM): LSTM attempts to
directly translate from video pixels to natural language
with a single deep neural network. The video represen-
tation is by performing mean pooling over the features
of frames using AlexNet.
•Soft-Attention (SA) [35]: SA combines the frame rep-
resentation from GoogleNet [28] and video clip rep-
resentation based on a 3-D ConvNet trained on His-
tograms of Oriented Gradients (HOG), Histograms
of Optical Flow (HOF), and Motion Boundary His-
togram (MBH) hand-crafted descriptors. Furthermore,
a weighted attention mechanism is used to dynami-
cally attend to speciﬁc temporal regions of the video
while generating sentence.
•Sequence to Sequence - Video to Text (S2VT) [31]:
S2VT incorporates both RGB and optical ﬂow inputs,
and the encoding and decoding of the inputs and word
representations are learnt jointly in a parallel manner.
•Long Shot-Term Memory with visual-semantic Em-
bedding (LSTM-E): We design four runs for our
proposed approach, i.e., LSTM-E (Alex), LSTM-E
(VGG), LSTM-E (C3D), and LSTM-E (VGG+C3D).
The input frame/clip features of the ﬁrst three runs are
from AlexNet, VGG and C3D network respectively.
The input of the last one is to concatenate the features
from VGG and C3D.
5.2.2 SVO Triplet Prediction Task
As SVO triples can capture the compositional semantics of
videos, predicting SVO triplet could indicate the quality of
a translation system to a large extent.
We adopt SVO accuracy [34] which measures the ex-
actness of SVO words by binary (0-1 loss), as the evalu-
ation metric. Table 1 details SVO accuracy of compared
seven models. Within these models, the former four models
(called Item driven models ) explicitly optimize to identify
the best subject, verb and object items for a video; while
the later ﬁve models (named Sentence driven models) focus
on training on objects and actions jointly in a sentence and
learn to interpret these in different contexts. For the later
ﬁve sentence driven models, we extract the SVO triplets
from the generated sentences by Stanford Parser 1 and the
words are also stemmed. Overall, the results across SVO
triplet indicate that almost all the four Item driven models
exhibit better performance than LSTM model which pre-
dicts the next word by only considering the contextual rela-
tionships with the previous words given the video content.
By jointly modeling the relevance between the semantics of
the entire sentence and video content with LSTM, LSTM-
E signiﬁcantly improves LSTM. Furthermore, the perfor-
mances of LSTM-E (VGG), LSTM-E (C3D), and LSTM-
E (VGG+C3D) on Subject, Verb and Object are all above
that of the four Item driven models . The result basically
indicates the advantage of further exploring the relevance
holistically between the semantics of the entire sentence and
video content in addition to LSTM.
Compared to LSTM-E (Alex), LSTM-E (VGG) using a
more powerful frame representation brought by a deeper
CNN exhibits signiﬁcantly better performance. In addi-
tion, LSTM-E (C3D) which has a better ability in encap-
sulating temporal information leads to better performance
than LSTM-E (VGG) in terms of Verb prediction accuracy.
When combining the features from VGG and C3D, LSTM-
E (VGG+C3D) further increases the performance gains.
5.2.3 Sentence Generation Task
For item driven models including FGM, CRF, CCA and
JEM, the sentence generation is often performed by lever-
aging a series of simple sentence templates (or special lan-
guage trees) on the SVO triplets [32]. Having veriﬁed in
[32], using LSTM architecture can lead to a large perfor-
mance boost against the template-based sentence genera-
tion. Thus, Table 2 only shows comparisons of LSTM-
based sentence generations. We use the BLEU@ N [22]
and METEOR scores [2] against all ground truth sentences.
Both metrics have been shown to correlate well with human
judgement, and widely used in machine translation litera-
1http://nlp.stanford.edu/software/lex-parser.shtml
Table 2. BLEU@N and METEOR scores for comparing the quality of the sentence generation. All values are reported as percentage (%).
Model METEOR BLEU@1 BLEU@2 BLEU@3 BLEU@4
LSTM 26.9 69.8 53.3 42.1 31.2
SA 29.6 80.0 64.7 52.6 42.2
S2VT 29.8 - - - -
LSTM-E (Alex) 28.3 74.5 59.8 49.3 38.9
LSTM-E (VGG) 29.5 74.9 60.9 50.6 40.2
LSTM-E (C3D) 29.9 75.7 62.3 52 41.7
LSTM-E (VGG+C3D) 31.0 78.8 66.0 55.4 45.3
Figure 4. The effect of the tradeoff parameter λ measured by
BLEU@N and METEOR.
ture. Speciﬁcally, BLEU@ N measures the fraction of N-
gram (up to 4-gram) that are in common between a hypoth-
esis and a reference or set of references, while METEOR
computes unigram precision and recall, extending exact
word matches to include similar words based on WordNet
synonyms and stemmed tokens. As shown in the Table 2,
the qualitative results across different N of BLEU and ME-
TEOR consistently indicate that the LSTM-E (Alex) sig-
niﬁcantly outperforms the traditional LSTM model. More-
over, we can ﬁnd that the performance gain of BLEU@ N
becomes larger when N increases, where N measures the
length of the contiguous sequence in the sentence. This
again conﬁrms that LSTM-E is beneﬁted from the way of
holistically exploring the relationships between the seman-
tics of the entire sentence and video content by minimiz-
ing the distance of their mappings in a visual-semantic em-
bedding. Similar to the observations in SVO prediction
task, our LSTM-E (VGG) outperforms LSTM-E (Alex) and
can reach 29.5% METEOR. Furthermore, LSTM-E (C3D)
achieves 29.9% METEOR and improves the performance
to 31.0% when combined with VGG, which makes the im-
provement over the current two state-of-the-art methods SA
by 4.7% and S2VT by 4.0%, respectively.
Figure 5 shows a few sentence examples generated by
different methods and human-annotated ground truth. From
these exemplar results, it is easy to see that all of these au-
tomatic methods can generate somewhat relevant sentences.
When looking into each word, both LSTM-E (Alex) and
LSTM-E (VGG+C3D) predict more relevant Subject, Verb
and Object (SVO) terms. For example, compared to subject
Table 3. The effect of hidden layer size in our LSTM-E
(VGG+C3D) framework measured by BLEU@4 and METEOR.
Hidden
layer size BLEU@4 METEOR Parameter
number
128 38.4 29.0 3.6M
256 40.6 29.6 7.5M
512 45.3 31.0 16.0M
term “a man”, ‘People” or “a group of men” is more precise
to describe the video content in the second video. Similarly,
verb term “singing” presents the fourth video more exactly.
The predicted object terms “keyboard” and “motorcycle”
are more relevant than “guitar” and “car” in ﬁfth and sixth
videos, respectively. Moreover, LSTM-E (VGG+C3D) can
offer more coherent sentences. For instance, the generated
sentence “a man is talking on a phone” of the third video
encapsulates the video content more clearly.
5.3. Experimental Analysis
We will further provide the analysis on the effect of the
tradeoff parameter between two losses and the size of hid-
den layer in LSTM learning.
5.3.1 The Tradeoff Parameterλ
To clarify the effect of the tradeoff parameter λin Eq.(11),
we illustrate the performance curves with a different trade-
off parameter in Figure 4. To make all performance curves
fall into a comparable scale, all BLEU@ N and METEOR
values are specially normalized as follows
m′
λ =
mλ −min
λ
{mλ}
min
λ
{mλ} (13)
where mλ and m′
λ denotes original and normalized perfor-
mance values (BLEU@ N or METEOR) with a set of λ,
respectively.
From the ﬁgures, we can see that all performance curves
are like the “∧” shapes when λvaries in a range from 0.1 to
0.9. The best performance is achieved when λis about 0.7.
This proves that it is reasonable to jointly learn the visual-
semantic embedding space in the deep recurrent neural net-
works.
... ...
... ...
... ...
... ...
... ...
LSTM: a woman is talking
LSTM-E (Alex): a man is talking
LSTM-E (VGG+C3D): a man is 
talking on a phone
LSTM: a man is dancing
LSTM-E (Alex): people are 
dancing
LSTM-E (VGG+C3D): a group of 
people are dancing
LSTM: a cat is playing with a mirror
LSTM-E (Alex):  a cat is playing 
with a watermelon
LSTM-E (VGG+C3D): a kitten is 
playing with a toy
LSTM:  a man is playing a guitar
LSTM-E (Alex): a man is 
playing a keyboard
LSTM-E (VGG+C3D): a man is 
playing a piano
Ground Truth: 
① a kitten is playing with his toy
② a cat is playing on the floor
③ a kitten plays with a toy
Ground Truth:  
① a group of people are dancing
② people are dancing outside
③ many people dance in the street
Ground Truth: 
① a man is singing on stage
② a man is singing into a 
microphone
③ a man sings into a microphone
Ground Truth: 
① a person is playing a piano 
keyboard
② a man plays a keyboard
③ a boy played a keyboard
Ground Truth: 
① a man is talking on a cell phone
② a man is speaking into a cell 
phone
③ the man talked on the phone
LSTM: a man is playing a flute
LSTM-E (Alex): a man is singing
LSTM-E (VGG+C3D): a man is 
singingLSTM: a man is riding a car
LSTM-E (Alex): a man is riding a 
bicycle
LSTM-E (VGG+C3D): a man is 
riding a motorcycle
... ...
Ground Truth: 
① someone is riding a motorcycle
② a man is riding his motorcycle
③ a man is riding on a motor bike
Figure 5. Examples of sentence generation results. The videos are represented by sampled frames, the output sentences generated by 1)
LSTM, 2) our proposed LSTM-E (Alex) and LSTM-E (VGG+C3D), and 3) Ground Truth: Randomly selected three ground truth sentences.
5.3.2 The Size of hidden layer of LSTM
In order to show the relationship between the performance
and hidden layer size of LSTM, we compare the results of
the hidden layer size in the range of 128, 256, and 512. The
results shown in Table 3 indicate increasing the hidden layer
size can lead to the improvement of the performance with
respect to both BLEU@4 and METEOR. Therefore, in our
experiments, the hidden layer size is empirically set to 512,
which achieves the best performance.
6. Discussion and Conclusion
In this paper, we have proposed a solution to the video
description problem by introducing a novel LSTM-E model
structure. In particular, a visual-semantic embedding space
is additionally incorporated into LSTM learning. In this
way, a global relationship between the video content and
sentence semantics is simultaneously measured in addition
to the local contextual relationship between the word at each
step and the previous ones in LSTM learning. On a popu-
lar video description dataset, the results of our experiments
demonstrate the success of our approach, outperforming the
current state-of-the-art model with a signiﬁcantly large mar-
gin on both SVO prediction and sentence generation.
Our future works are as follows. First, as a video it-
self is a temporal sequence, the way of better representing
the videos by using RNN will be further explored. More-
over, the video description generation might be signiﬁcantly
boosted if we could have sufﬁcient labeled video-sentence
pairs to train a deeper RNN.
References
[1] D. Bahdanau, K. Cho, and Y . Bengio. Neural machine
translation by jointly learning to align and translate. arXiv
preprint arXiv:1409.0473, 2014.
[2] S. Banerjee and A. Lavie. Meteor: An automatic metric for
mt evaluation with improved correlation with human judg-
ments. In Proceedings of the acl workshop on intrinsic and
extrinsic evaluation measures for machine translation and/or
summarization, 2005.
[3] K. Barnard, P. Duygulu, D. Forsyth, N. De Freitas, D. M.
Blei, and M. I. Jordan. Matching words and pictures. JMLR,
2003.
[4] Y . Bengio, P. Simard, and P. Frasconi. Learning long-term
dependencies with gradient descent is difﬁcult. IEEE Trans.
on Neural Networks, 5(2):157–166, 1994.
[5] D. L. Chen and W. B. Dolan. Collecting highly parallel data
for paraphrase evaluation. In ACL, 2011.
[6] J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach,
S. Venugopalan, K. Saenko, and T. Darrell. Long-term recur-
rent convolutional networks for visual recognition and de-
scription. arXiv preprint arXiv:1411.4389, 2014.
[7] H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng, P. Dol-
lar, J. Gao, X. He, M. Mitchell, J. C. Platt, C. L. Zitnick, and
G. Zweig. From captions to visual concepts and back. In
CVPR, 2015.
[8] A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young,
C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every pic-
ture tells a story: Generating sentences from images. In
ECCV, 2010.
[9] A. Graves and N. Jaitly. Towards end-to-end speech recog-
nition with recurrent neural networks. In ICML, 2014.
[10] A. Graves and J. Schmidhuber. Framewise phoneme clas-
siﬁcation with bidirectional lstm and other neural network
architectures. Neural Networks, 2005.
[11] S. Guadarrama, N. Krishnamoorthy, G. Malkarnenkar,
S. Venugopalan, R. Mooney, T. Darrell, and K. Saenko.
Youtube2text: Recognizing and describing arbitrary activi-
ties using semantic hierarchies and zero-shot recognition. In
ICCV, 2013.
[12] S. Hochreiter and J. Schmidhuber. Long short-term memory.
Neural Computation, 9(8):1735–1780, 1997.
[13] Y . Jia, M. Salzmann, and T. Darrell. Learning cross-modality
similarity for multinomial data. In ICCV, 2011.
[14] A. Karpathy and L. Fei-Fei. Deep visual-semantic align-
ments for generating image descriptions. In CVPR, 2015.
[15] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar,
and L. Fei-Fei. Large-scale video classiﬁcation with convo-
lutional neural networks. In CVPR, 2014.
[16] R. Kiros, R. Salakhutdinov, and R. Zemel. Multimodal neu-
ral language models. In ICML, 2014.
[17] R. Kiros, R. Salakhutdinov, and R. S. Zemel. Unifying
visual-semantic embeddings with multimodal neural lan-
guage models. TACL, 2015.
[18] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classiﬁcation with deep convolutional neural networks. In
NIPS, 2012.
[19] G. Kulkarni, V . Premraj, V . Ordonez, S. Dhar, S. Li, Y . Choi,
A. C. Berg, and T. L. Berg. Babytalk: Understanding and
generating simple image descriptions.IEEE Trans. on PAMI,
2013.
[20] J. Mao, W. Xu, Y . Yang, J. Wang, and A. L. Yuille. Explain
images with multimodal recurrent neural networks. In NIPS
Workshop on Deep Learning, 2014.
[21] J. Martens and I. Sutskever. Learning recurrent neural net-
works with hessian-free optimization. ICML, 2011.
[22] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a
method for automatic evaluation of machine translation. In
ACL, 2002.
[23] M. Rohrbach, W. Qiu, I. Titov, S. Thater, M. Pinkal, and
B. Schiele. Translating video content to natural language
descriptions. In ICCV, 2013.
[24] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual
Recognition Challenge. IJCV, 2015.
[25] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. arXiv preprint
arXiv:1409.1556, 2014.
[26] R. Socher and L. Fei-Fei. Connecting modalities: Semi-
supervised segmentation and annotation of images using un-
aligned text corpora. In CVPR, 2010.
[27] I. Sutskever, O. Vinyals, and Q. V . Le. Sequence to sequence
learning with neural networks. In NIPS, 2014.
[28] C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed,
D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabi-
novich. Going deeper with convolutions. arXiv preprint
arXiv:1409.4842, 2014.
[29] J. Thomason, S. Venugopalan, S. Guadarrama, K. Saenko,
and R. Mooney. Integrating language and vision to gener-
ate natural language descriptions of videos in the wild. In
COLING, 2014.
[30] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri.
C3d: Generic features for video analysis. arXiv preprint
arXiv:1412.0767, 2015.
[31] S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney,
T. Darrell, and K. Saenko. Sequence to sequence - video
to text. arXiv preprint arXiv:1505.00487, 2015.
[32] S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach,
R. Mooney, and K. Saenko. Translating videos to natural lan-
guage using deep recurrent neural networks. arXiv preprint
arXiv:1412.4729, 2014.
[33] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show
and tell: A neural image caption generator. arXiv preprint
arXiv:1411.4555, 2014.
[34] R. Xu, C. Xiong, W. Chen, and J. J. Corso. Jointly model-
ing deep video and compositional text to bridge vision and
language in a uniﬁed framework. In AAAI, 2015.
[35] L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle,
and A. Courville. Video description generation incorporat-
ing spatio-temporal features and a soft-attention mechanism.
arXiv preprint arXiv:1502.08029, 2015.
[36] W. Zaremba and I. Sutskever. Learning to execute. arXiv
preprint arXiv:1410.4615, 2014.

Deep Learning in Neural Networks: An Overview
Technical Report IDSIA-03-14 / arXiv:1404.7828 v4 [cs.NE] (88 pages, 888 references)
J¨urgen Schmidhuber
The Swiss AI Lab IDSIA
Istituto Dalle Molle di Studi sull’Intelligenza Artiﬁciale
University of Lugano & SUPSI
Galleria 2, 6928 Manno-Lugano
Switzerland
8 October 2014
Abstract
In recent years, deep artiﬁcial neural networks (including recurrent ones) have won numerous
contests in pattern recognition and machine learning. This historical survey compactly summarises
relevant work, much of it from the previous millennium. Shallow and deep learners are distin-
guished by the depth of theircredit assignment paths, which are chains of possibly learnable, causal
links between actions and effects. I review deep supervised learning (also recapitulating the history
of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation,
and indirect search for short programs encoding deep and large networks.
LATEX source: http://www.idsia.ch/˜juergen/DeepLearning8Oct2014.tex
Complete BIBTEX ﬁle (888 kB): http://www.idsia.ch/˜juergen/deep.bib
Preface
This is the preprint of an invited Deep Learning (DL) overview. One of its goals is to assign credit
to those who contributed to the present state of the art. I acknowledge the limitations of attempting
to achieve this goal. The DL research community itself may be viewed as a continually evolving,
deep network of scientists who have inﬂuenced each other in complex ways. Starting from recent DL
results, I tried to trace back the origins of relevant ideas through the past half century and beyond,
sometimes using “local search” to follow citations of citations backwards in time. Since not all DL
publications properly acknowledge earlier relevant work, additional global search strategies were em-
ployed, aided by consulting numerous neural network experts. As a result, the present preprint mostly
consists of references. Nevertheless, through an expert selection bias I may have missed important
work. A related bias was surely introduced by my special familiarity with the work of my own DL
research group in the past quarter-century. For these reasons, this work should be viewed as merely a
snapshot of an ongoing credit assignment process. To help improve it, please do not hesitate to send
corrections and suggestions to juergen@idsia.ch.
1
arXiv:1404.7828v4  [cs.NE]  8 Oct 2014
Contents
1 Introduction to Deep Learning (DL) in Neural Networks (NNs) 4
2 Event-Oriented Notation for Activation Spreading in NNs 5
3 Depth of Credit Assignment Paths (CAPs) and of Problems 6
4 Recurring Themes of Deep Learning 7
4.1 Dynamic Programming for Supervised/Reinforcement Learning (SL/RL) . . . . . . 7
4.2 Unsupervised Learning (UL) Facilitating SL and RL . . . . . . . . . . . . . . . . . 7
4.3 Learning Hierarchical Representations Through Deep SL, UL, RL . . . . . . . . . . 8
4.4 Occam’s Razor: Compression and Minimum Description Length (MDL) . . . . . . . 8
4.5 Fast Graphics Processing Units (GPUs) for DL in NNs . . . . . . . . . . . . . . . . 8
5 Supervised NNs, Some Helped by Unsupervised NNs 8
5.1 Early NNs Since the 1940s (and the 1800s) . . . . . . . . . . . . . . . . . . . . . . 9
5.2 Around 1960: Visual Cortex Provides Inspiration for DL (Sec. 5.4, 5.11) . . . . . . . 10
5.3 1965: Deep Networks Based on the Group Method of Data Handling . . . . . . . . . 10
5.4 1979: Convolution + Weight Replication + Subsampling (Neocognitron) . . . . . . 10
5.5 1960-1981 and Beyond: Development of Backpropagation (BP) for NNs . . . . . . . 11
5.5.1 BP for Weight-Sharing Feedforward NNs (FNNs) and Recurrent NNs (RNNs) 11
5.6 Late 1980s-2000 and Beyond: Numerous Improvements of NNs . . . . . . . . . . . 12
5.6.1 Ideas for Dealing with Long Time Lags and Deep CAPs . . . . . . . . . . . 12
5.6.2 Better BP Through Advanced Gradient Descent (Compare Sec. 5.24) . . . . 13
5.6.3 Searching For Simple, Low-Complexity, Problem-Solving NNs (Sec. 5.24) . 14
5.6.4 Potential Beneﬁts of UL for SL (Compare Sec. 5.7, 5.10, 5.15) . . . . . . . . 14
5.7 1987: UL Through Autoencoder (AE) Hierarchies (Compare Sec. 5.15) . . . . . . . 15
5.8 1989: BP for Convolutional NNs (CNNs, Sec. 5.4) . . . . . . . . . . . . . . . . . . 16
5.9 1991: Fundamental Deep Learning Problem of Gradient Descent . . . . . . . . . . . 16
5.10 1991: UL-Based History Compression Through a Deep Stack of RNNs . . . . . . . 17
5.11 1992: Max-Pooling (MP): Towards MPCNNs (Compare Sec. 5.16, 5.19) . . . . . . . 18
5.12 1994: Early Contest-Winning NNs . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
5.13 1995: Supervised Recurrent Very Deep Learner (LSTM RNN) . . . . . . . . . . . . 19
5.14 2003: More Contest-Winning/Record-Setting NNs; Successful Deep NNs . . . . . . 21
5.15 2006/7: UL For Deep Belief Networks / AE Stacks Fine-Tuned by BP . . . . . . . . 21
5.16 2006/7: Improved CNNs / GPU-CNNs / BP for MPCNNs / LSTM Stacks . . . . . . 22
5.17 2009: First Ofﬁcial Competitions Won by RNNs, and with MPCNNs . . . . . . . . . 22
5.18 2010: Plain Backprop ( + Distortions) on GPU Breaks MNIST Record . . . . . . . . 23
5.19 2011: MPCNNs on GPU Achieve Superhuman Vision Performance . . . . . . . . . 23
5.20 2011: Hessian-Free Optimization for RNNs . . . . . . . . . . . . . . . . . . . . . . 24
5.21 2012: First Contests Won on ImageNet, Object Detection, Segmentation . . . . . . . 24
5.22 2013-: More Contests and Benchmark Records . . . . . . . . . . . . . . . . . . . . 25
5.23 Currently Successful Techniques: LSTM RNNs and GPU-MPCNNs . . . . . . . . . 26
5.24 Recent Tricks for Improving SL Deep NNs (Compare Sec. 5.6.2, 5.6.3) . . . . . . . 27
5.25 Consequences for Neuroscience . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5.26 DL with Spiking Neurons? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2
6 DL in FNNs and RNNs for Reinforcement Learning (RL) 29
6.1 RL Through NN World Models Yields RNNs With Deep CAPs . . . . . . . . . . . . 29
6.2 Deep FNNs for Traditional RL and Markov Decision Processes (MDPs) . . . . . . . 30
6.3 Deep RL RNNs for Partially Observable MDPs (POMDPs) . . . . . . . . . . . . . . 31
6.4 RL Facilitated by Deep UL in FNNs and RNNs . . . . . . . . . . . . . . . . . . . . 31
6.5 Deep Hierarchical RL (HRL) and Subgoal Learning with FNNs and RNNs . . . . . . 31
6.6 Deep RL by Direct NN Search / Policy Gradients / Evolution . . . . . . . . . . . . . 32
6.7 Deep RL by Indirect Policy Search / Compressed NN Search . . . . . . . . . . . . . 33
6.8 Universal RL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
7 Conclusion and Outlook 34
8 Acknowledgments 35
Abbreviations in Alphabetical Order
AE: Autoencoder
AI: Artiﬁcial Intelligence
ANN: Artiﬁcial Neural Network
BFGS: Broyden-Fletcher-Goldfarb-Shanno
BNN: Biological Neural Network
BM: Boltzmann Machine
BP: Backpropagation
BRNN: Bi-directional Recurrent Neural Network
CAP: Credit Assignment Path
CEC: Constant Error Carousel
CFL: Context Free Language
CMA-ES: Covariance Matrix Estimation ES
CNN: Convolutional Neural Network
CoSyNE: Co-Synaptic Neuro-Evolution
CSL: Context Senistive Language
CTC : Connectionist Temporal Classiﬁcation
DBN: Deep Belief Network
DCT: Discrete Cosine Transform
DL: Deep Learning
DP: Dynamic Programming
DS: Direct Policy Search
EA: Evolutionary Algorithm
EM: Expectation Maximization
ES: Evolution Strategy
FMS: Flat Minimum Search
FNN: Feedforward Neural Network
FSA: Finite State Automaton
GMDH: Group Method of Data Handling
GOFAI: Good Old-Fashioned AI
GP: Genetic Programming
GPU: Graphics Processing Unit
GPU-MPCNN: GPU-Based MPCNN
HMM: Hidden Markov Model
HRL: Hierarchical Reinforcement Learning
HTM: Hierarchical Temporal Memory
HMAX: Hierarchical Model “and X”
LSTM: Long Short-Term Memory (RNN)
MDL: Minimum Description Length
MDP: Markov Decision Process
MNIST: Mixed National Institute of Standards
and Technology Database
MP: Max-Pooling
MPCNN: Max-Pooling CNN
NE: NeuroEvolution
NEAT: NE of Augmenting Topologies
NES: Natural Evolution Strategies
NFQ: Neural Fitted Q-Learning
NN: Neural Network
OCR: Optical Character Recognition
PCC: Potential Causal Connection
PDCC: Potential Direct Causal Connection
PM: Predictability Minimization
POMDP: Partially Observable MDP
RAAM: Recursive Auto-Associative Memory
RBM: Restricted Boltzmann Machine
ReLU: Rectiﬁed Linear Unit
RL: Reinforcement Learning
RNN: Recurrent Neural Network
R-prop: Resilient Backpropagation
SL: Supervised Learning
SLIM NN: Self-Delimiting Neural Network
SOTA: Self-Organising Tree Algorithm
SVM: Support Vector Machine
TDNN: Time-Delay Neural Network
TIMIT: TI/SRI/MIT Acoustic-Phonetic Continu-
ous Speech Corpus
UL: Unsupervised Learning
WTA: Winner-Take-All
3
1 Introduction to Deep Learning (DL) in Neural Networks (NNs)
Which modiﬁable components of a learning system are responsible for its success or failure? What
changes to them improve performance? This has been called thefundamental credit assignment prob-
lem (Minsky, 1963). There are general credit assignment methods for universal problem solvers that
are time-optimal in various theoretical senses (Sec. 6.8). The present survey, however, will focus on
the narrower, but now commercially important, subﬁeld of Deep Learning (DL) in Artiﬁcial Neural
Networks (NNs).
A standard neural network (NN) consists of many simple, connected processors called neurons,
each producing a sequence of real-valued activations. Input neurons get activated through sensors per-
ceiving the environment, other neurons get activated through weighted connections from previously
active neurons (details in Sec. 2). Some neurons may inﬂuence the environment by triggering actions.
Learning or credit assignment is about ﬁnding weights that make the NN exhibit desired behavior,
such as driving a car. Depending on the problem and how the neurons are connected, such behavior
may require long causal chains of computational stages (Sec. 3), where each stage transforms (of-
ten in a non-linear way) the aggregate activation of the network. Deep Learning is about accurately
assigning credit across many such stages.
Shallow NN-like models with few such stages have been around for many decades if not centuries
(Sec. 5.1). Models with several successive nonlinear layers of neurons date back at least to the 1960s
(Sec. 5.3) and 1970s (Sec. 5.5). An efﬁcient gradient descent method for teacher-based Supervised
Learning (SL) in discrete, differentiable networks of arbitrary depth calledbackpropagation (BP) was
developed in the 1960s and 1970s, and applied to NNs in 1981 (Sec. 5.5). BP-based training of deep
NNs with many layers, however, had been found to be difﬁcult in practice by the late 1980s (Sec. 5.6),
and had become an explicit research subject by the early 1990s (Sec. 5.9). DL became practically fea-
sible to some extent through the help ofUnsupervised Learning (UL), e.g., Sec. 5.10 (1991), Sec. 5.15
(2006). The 1990s and 2000s also saw many improvements of purely supervised DL (Sec. 5). In the
new millennium, deep NNs have ﬁnally attracted wide-spread attention, mainly by outperforming al-
ternative machine learning methods such as kernel machines (Vapnik, 1995; Sch ¨olkopf et al., 1998)
in numerous important applications. In fact, since 2009, supervised deep NNs have won many ofﬁcial
international pattern recognition competitions (e.g., Sec. 5.17, 5.19, 5.21, 5.22), achieving the ﬁrst
superhuman visual pattern recognition results in limited domains (Sec. 5.19, 2011). Deep NNs also
have become relevant for the more general ﬁeld of Reinforcement Learning (RL) where there is no
supervising teacher (Sec. 6).
Both feedforward (acyclic) NNs (FNNs) and recurrent (cyclic) NNs (RNNs) have won contests
(Sec. 5.12, 5.14, 5.17, 5.19, 5.21, 5.22). In a sense, RNNs are the deepest of all NNs (Sec. 3)—they
are general computers more powerful than FNNs, and can in principle create and process memories
of arbitrary sequences of input patterns (e.g., Siegelmann and Sontag, 1991; Schmidhuber, 1990a).
Unlike traditional methods for automatic sequential program synthesis (e.g., Waldinger and Lee, 1969;
Balzer, 1985; Soloway, 1986; Deville and Lau, 1994), RNNs can learn programs that mix sequential
and parallel information processing in a natural and efﬁcient way, exploiting the massive parallelism
viewed as crucial for sustaining the rapid decline of computation cost observed over the past 75 years.
The rest of this paper is structured as follows. Sec. 2 introduces a compact, event-oriented notation
that is simple yet general enough to accommodate both FNNs and RNNs. Sec. 3 introduces the
concept of Credit Assignment Paths(CAPs) to measure whether learning in a given NN application is
of the deep or shallow type. Sec. 4 lists recurring themes of DL in SL, UL, and RL. Sec. 5 focuses
on SL and UL, and on how UL can facilitate SL, although pure SL has become dominant in recent
competitions (Sec. 5.17–5.23). Sec. 5 is arranged in a historical timeline format with subsections on
important inspirations and technical contributions. Sec. 6 on deep RL discusses traditional Dynamic
Programming (DP)-based RL combined with gradient-based search techniques for SL or UL in deep
4
NNs, as well as general methods for direct and indirect search in the weight space of deep FNNs and
RNNs, including successful policy gradient and evolutionary methods.
2 Event-Oriented Notation for Activation Spreading in NNs
Throughout this paper, let i,j,k,t,p,q,r denote positive integer variables assuming ranges implicit
in the given contexts. Let n,m,T denote positive integer constants.
An NN’s topology may change over time (e.g., Sec. 5.3, 5.6.3). At any given moment, it can
be described as a ﬁnite subset of units (or nodes or neurons) N = {u1,u2,..., }and a ﬁnite set
H ⊆N×N of directed edges or connections between nodes. FNNs are acyclic graphs, RNNs cyclic.
The ﬁrst (input) layer is the set of input units, a subset ofN. In FNNs, the k-th layer (k> 1) is the set
of all nodes u∈N such that there is an edge path of length k−1 (but no longer path) between some
input unit and u. There may be shortcut connections between distant layers. In sequence-processing,
fully connected RNNs, all units have connections to all non-input units.
The NN’s behavior or program is determined by a set of real-valued, possibly modiﬁable, param-
eters or weights wi (i= 1,...,n ). We now focus on a single ﬁnite episode or epoch of information
processing and activation spreading, without learning through weight changes. The following slightly
unconventional notation is designed to compactly describe what is happening during the runtime of
the system.
During an episode, there is a partially causal sequence xt(t= 1,...,T ) of real values that I call
events. Each xt is either an input set by the environment, or the activation of a unit that may directly
depend on other xk(k<t ) through a current NN topology-dependent setint of indices krepresenting
incoming causal connections or links. Let the function vencode topology information and map such
event index pairs (k,t) to weight indices. For example, in the non-input case we may have xt =
ft(nett) with real-valued nett = ∑
k∈int xkwv(k,t) (additive case) or nett = ∏
k∈int xkwv(k,t)
(multiplicative case), where ft is a typically nonlinear real-valued activation function such as tanh.
In many recent competition-winning NNs (Sec. 5.19, 5.21, 5.22) there also are events of the type
xt = maxk∈int(xk); some network types may also use complex polynomial activation functions
(Sec. 5.3). xt may directly affect certainxk(k>t ) through outgoing connections or links represented
through a current set outt of indices kwith t ∈ink. Some of the non-input events are called output
events.
Note that many of the xt may refer to different, time-varying activations of the same unit in
sequence-processing RNNs (e.g., Williams, 1989, “unfolding in time”), or also in FNNs sequentially
exposed to time-varying input patterns of a large training set encoded as input events. During an
episode, the same weight may get reused over and over again in topology-dependent ways, e.g., in
RNNs, or in convolutional NNs (Sec. 5.4, 5.8). I call this weight sharing across space and/or time.
Weight sharing may greatly reduce the NN’s descriptive complexity, which is the number of bits of
information required to describe the NN (Sec. 4.4).
In Supervised Learning (SL), certain NN output events xt may be associated with teacher-given,
real-valued labels or targetsdt yielding errors et, e.g., et = 1/2(xt−dt)2. A typical goal of supervised
NN training is to ﬁnd weights that yield episodes with small total error E, the sum of all such et. The
hope is that the NN will generalize well in later episodes, causing only small errors on previously
unseen sequences of input events. Many alternative error functions for SL and UL are possible.
SL assumes that input events are independent of earlier output events (which may affect the en-
vironment through actions causing subsequent perceptions). This assumption does not hold in the
broader ﬁelds of Sequential Decision Making and Reinforcement Learning (RL) (Kaelbling et al.,
1996; Sutton and Barto, 1998; Hutter, 2005; Wiering and van Otterlo, 2012) (Sec. 6). In RL, some
of the input events may encode real-valued reward signals given by the environment, and a typical
5
goal is to ﬁnd weights that yield episodes with a high sum of reward signals, through sequences of
appropriate output actions.
Sec. 5.5 will use the notation above to compactly describe a central algorithm of DL, namely,
backpropagation (BP) for supervised weight-sharing FNNs and RNNs. (FNNs may be viewed as
RNNs with certain ﬁxed zero weights.) Sec. 6 will address the more general RL case.
3 Depth of Credit Assignment Paths (CAPs) and of Problems
To measure whether credit assignment in a given NN application is of the deep or shallow type, I
introduce the concept of Credit Assignment Paths or CAPs, which are chains of possibly causal links
between the events of Sec. 2, e.g., from input through hidden to output layers in FNNs, or through
transformations over time in RNNs.
Let us ﬁrst focus on SL. Consider two events xp and xq (1 ≤p < q≤T). Depending on the
application, they may have a Potential Direct Causal Connection (PDCC) expressed by the Boolean
predicate pdcc(p,q), which is true if and only if p∈inq. Then the 2-element list (p,q) is deﬁned to
be a CAP (a minimal one) from pto q. A learning algorithm may be allowed to change wv(p,q) to
improve performance in future episodes.
More general, possibly indirect, Potential Causal Connections (PCC) are expressed by the re-
cursively deﬁned Boolean predicate pcc(p,q), which in the SL case is true only if pdcc(p,q), or if
pcc(p,k) for some kand pdcc(k,q). In the latter case, appending qto any CAP from pto kyields a
CAP from pto q(this is a recursive deﬁnition, too). The set of such CAPs may be large but is ﬁnite.
Note that the same weight may affect many different PDCCs between successive events listed by a
given CAP, e.g., in the case of RNNs, or weight-sharing FNNs.
Suppose a CAP has the form (...,k,t,...,q ), where kand t(possibly t= q) are the ﬁrst succes-
sive elements with modiﬁable wv(k,t). Then the length of the sufﬁx list (t,...,q ) is called the CAP’s
depth (which is 0 if there are no modiﬁable links at all). This depth limits how far backwards credit
assignment can move down the causal chain to ﬁnd a modiﬁable weight.1
Suppose an episode and its event sequence x1,...,x T satisfy a computable criterion used to
decide whether a given problem has been solved (e.g., total error E below some threshold). Then
the set of used weights is called a solution to the problem, and the depth of the deepest CAP within
the sequence is called the solution depth . There may be other solutions (yielding different event
sequences) with different depths. Given some ﬁxed NN topology, the smallest depth of any solution
is called the problem depth.
Sometimes we also speak of the depth of an architecture: SL FNNs with ﬁxed topology imply a
problem-independent maximal problem depth bounded by the number of non-input layers. Certain
SL RNNs with ﬁxed weights for all connections except those to output units (Jaeger, 2001; Maass
et al., 2002; Jaeger, 2004; Schrauwen et al., 2007) have a maximal problem depth of 1, because only
the ﬁnal links in the corresponding CAPs are modiﬁable. In general, however, RNNs may learn to
solve problems of potentially unlimited depth.
Note that the deﬁnitions above are solely based on the depths of causal chains, and agnostic to the
temporal distance between events. For example, shallow FNNs perceiving large “time windows” of
input events may correctly classify long input sequences through appropriate output events, and thus
solve shallow problems involving long time lags between relevant events.
At which problem depth does Shallow Learning end, and Deep Learning begin? Discussions with
DL experts have not yet yielded a conclusive response to this question. Instead of committing myself
1An alternative would be to count onlymodiﬁable links when measuring depth. In many typical NN applications this would
not make a difference, but in some it would, e.g., Sec. 6.1.
6
to a precise answer, let me just deﬁne for the purposes of this overview: problems of depth > 10
require Very Deep Learning.
The difﬁculty of a problem may have little to do with its depth. Some NNs can quickly learn
to solve certain deep problems, e.g., through random weight guessing (Sec. 5.9) or other types of
direct search (Sec. 6.6) or indirect search (Sec. 6.7) in weight space, or through training an NN ﬁrst
on shallow problems whose solutions may then generalize to deep problems, or through collapsing
sequences of (non)linear operations into a single (non)linear operation (but see an analysis of non-
trivial aspects of deep linear networks, Baldi and Hornik, 1994, Section B). In general, however,
ﬁnding an NN that precisely models a given training set is an NP-complete problem (Judd, 1990;
Blum and Rivest, 1992), also in the case of deep NNs (S ´ıma, 1994; de Souto et al., 1999; Windisch,
2005); compare a survey of negative results (S´ıma, 2002, Section 1).
Above we have focused on SL. In the more general case of RL in unknown environments,pcc(p,q)
is also true ifxp is an output event andxq any later input event—any action may affect the environment
and thus any later perception. (In the real world, the environment may even inﬂuencenon-input events
computed on a physical hardware entangled with the entire universe, but this is ignored here.) It is
possible to model and replace such unmodiﬁable environmental PCCs through a part of the NN that
has already learned to predict (through some of its units) input events (including reward signals) from
former input events and actions (Sec. 6.1). Its weights are frozen, but can help to assign credit to
other, still modiﬁable weights used to compute actions (Sec. 6.1). This approach may lead to very
deep CAPs though.
Some DL research is about automatically rephrasing problems such that their depth is reduced
(Sec. 4). In particular, sometimes UL is used to make SL problems less deep, e.g., Sec. 5.10. Often
Dynamic Programming (Sec. 4.1) is used to facilitate certain traditional RL problems, e.g., Sec. 6.2.
Sec. 5 focuses on CAPs for SL, Sec. 6 on the more complex case of RL.
4 Recurring Themes of Deep Learning
4.1 Dynamic Programming for Supervised/Reinforcement Learning (SL/RL)
One recurring theme of DL is Dynamic Programming (DP) (Bellman, 1957), which can help to fa-
cilitate credit assignment under certain assumptions. For example, in SL NNs, backpropagation itself
can be viewed as a DP-derived method (Sec. 5.5). In traditional RL based on strong Markovian as-
sumptions, DP-derived methods can help to greatly reduce problem depth (Sec. 6.2). DP algorithms
are also essential for systems that combine concepts of NNs and graphical models, such as Hidden
Markov Models (HMMs) (Stratonovich, 1960; Baum and Petrie, 1966) andExpectation Maximization
(EM) (Dempster et al., 1977; Friedman et al., 2001), e.g., (Bottou, 1991; Bengio, 1991; Bourlard and
Morgan, 1994; Baldi and Chauvin, 1996; Jordan and Sejnowski, 2001; Bishop, 2006; Hastie et al.,
2009; Poon and Domingos, 2011; Dahl et al., 2012; Hinton et al., 2012a; Wu and Shao, 2014).
4.2 Unsupervised Learning (UL) Facilitating SL and RL
Another recurring theme is how UL can facilitate both SL (Sec. 5) and RL (Sec. 6). UL (Sec. 5.6.4)
is normally used to encode raw incoming data such as video or speech streams in a form that is more
convenient for subsequent goal-directed learning. In particular, codes that describe the original data in
a less redundant or more compact way can be fed into SL (Sec. 5.10, 5.15) or RL machines (Sec. 6.4),
whose search spaces may thus become smaller (and whose CAPs shallower) than those necessary for
dealing with the raw data. UL is closely connected to the topics of regularization and compression
(Sec. 4.4, 5.6.3).
7
4.3 Learning Hierarchical Representations Through Deep SL, UL, RL
Many methods of Good Old-Fashioned Artiﬁcial Intelligence (GOFAI) (Nilsson, 1980) as well as
more recent approaches to AI (Russell et al., 1995) and Machine Learning (Mitchell, 1997) learn
hierarchies of more and more abstract data representations. For example, certain methods of syn-
tactic pattern recognition (Fu, 1977) such as grammar induction discover hierarchies of formal rules
to model observations. The partially (un)supervised Automated Mathematician / EURISKO (Lenat,
1983; Lenat and Brown, 1984) continually learns concepts by combining previously learnt concepts.
Such hierarchical representation learning (Ring, 1994; Bengio et al., 2013; Deng and Yu, 2014) is also
a recurring theme of DL NNs for SL (Sec. 5), UL-aided SL (Sec. 5.7, 5.10, 5.15), and hierarchical RL
(Sec. 6.5). Often, abstract hierarchical representations are natural by-products of data compression
(Sec. 4.4), e.g., Sec. 5.10.
4.4 Occam’s Razor: Compression and Minimum Description Length (MDL)
Occam’s razor favors simple solutions over complex ones. Given some programming language, the
principle of Minimum Description Length (MDL) can be used to measure the complexity of a so-
lution candidate by the length of the shortest program that computes it (e.g., Solomonoff, 1964;
Kolmogorov, 1965b; Chaitin, 1966; Wallace and Boulton, 1968; Levin, 1973a; Solomonoff, 1978;
Rissanen, 1986; Blumer et al., 1987; Li and Vit ´anyi, 1997; Gr ¨unwald et al., 2005). Some methods
explicitly take into account program runtime (Allender, 1992; Watanabe, 1992; Schmidhuber, 1997,
2002); many consider only programs with constant runtime, written in non-universal programming
languages (e.g., Rissanen, 1986; Hinton and van Camp, 1993). In the NN case, the MDL princi-
ple suggests that low NN weight complexity corresponds to high NN probability in the Bayesian
view (e.g., MacKay, 1992; Buntine and Weigend, 1991; Neal, 1995; De Freitas, 2003), and to high
generalization performance (e.g., Baum and Haussler, 1989), without overﬁtting the training data.
Many methods have been proposed for regularizing NNs, that is, searching for solution-computing
but simple, low-complexity SL NNs (Sec. 5.6.3) and RL NNs (Sec. 6.7). This is closely related to
certain UL methods (Sec. 4.2, 5.6.4).
4.5 Fast Graphics Processing Units (GPUs) for DL in NNs
While the previous millennium saw several attempts at creating fast NN-speciﬁc hardware (e.g., Jackel
et al., 1990; Faggin, 1992; Ramacher et al., 1993; Widrow et al., 1994; Heemskerk, 1995; Korkin et al.,
1997; Urlbe, 1999), and at exploiting standard hardware (e.g., Anguita et al., 1994; Muller et al., 1995;
Anguita and Gomes, 1996), the new millennium brought a DL breakthrough in form of cheap, multi-
processor graphics cards or GPUs. GPUs are widely used for video games, a huge and competitive
market that has driven down hardware prices. GPUs excel at the fast matrix and vector multiplications
required not only for convincing virtual realities but also for NN training, where they can speed up
learning by a factor of 50 and more. Some of the GPU-based FNN implementations (Sec. 5.16–5.19)
have greatly contributed to recent successes in contests for pattern recognition (Sec. 5.19–5.22), image
segmentation (Sec. 5.21), and object detection (Sec. 5.21–5.22).
5 Supervised NNs, Some Helped by Unsupervised NNs
The main focus of current practical applications is on Supervised Learning (SL), which has domi-
nated recent pattern recognition contests (Sec. 5.17–5.23). Several methods, however, use additional
Unsupervised Learning (UL) to facilitate SL (Sec. 5.7, 5.10, 5.15). It does make sense to treat SL and
8
UL in the same section: often gradient-based methods, such as BP (Sec. 5.5.1), are used to optimize
objective functions of both UL and SL, and the boundary between SL and UL may blur, for example,
when it comes to time series prediction and sequence classiﬁcation, e.g., Sec. 5.10, 5.12.
A historical timeline format will help to arrange subsections on important inspirations and techni-
cal contributions (although such a subsection may span a time interval of many years). Sec. 5.1 brieﬂy
mentions early, shallow NN models since the 1940s (and 1800s), Sec. 5.2 additional early neurobio-
logical inspiration relevant for modern Deep Learning (DL). Sec. 5.3 is about GMDH networks (since
1965), to my knowledge the ﬁrst (feedforward) DL systems. Sec. 5.4 is about the relatively deep
Neocognitron NN (1979) which is very similar to certain modern deep FNN architectures, as it com-
bines convolutional NNs (CNNs), weight pattern replication, and subsampling mechanisms. Sec. 5.5
uses the notation of Sec. 2 to compactly describe a central algorithm of DL, namely,backpropagation
(BP) for supervised weight-sharing FNNs and RNNs. It also summarizes the history of BP 1960-1981
and beyond. Sec. 5.6 describes problems encountered in the late 1980s with BP for deep NNs, and
mentions several ideas from the previous millennium to overcome them. Sec. 5.7 discusses a ﬁrst hier-
archical stack (1987) of coupled UL-based Autoencoders (AEs)—this concept resurfaced in the new
millennium (Sec. 5.15). Sec. 5.8 is about applying BP to CNNs (1989), which is important for today’s
DL applications. Sec. 5.9 explains BP’sFundamental DL Problem(of vanishing/exploding gradients)
discovered in 1991. Sec. 5.10 explains how a deep RNN stack of 1991 (theHistory Compressor) pre-
trained by UL helped to solve previously unlearnable DL benchmarks requiring Credit Assignment
Paths (CAPs, Sec. 3) of depth 1000 and more. Sec. 5.11 discusses a particularwinner-take-all (WTA)
method called Max-Pooling (MP, 1992) widely used in today’s deep FNNs. Sec. 5.12 mentions a
ﬁrst important contest won by SL NNs in 1994. Sec. 5.13 describes a purely supervised DL RNN
(Long Short-Term Memory, LSTM, 1995) for problems of depth 1000 and more. Sec. 5.14 mentions
an early contest of 2003 won by an ensemble of shallow FNNs, as well as good pattern recognition
results with CNNs and deep FNNs and LSTM RNNs (2003). Sec. 5.15 is mostly about Deep Belief
Networks (DBNs, 2006) and related stacks ofAutoencoders (AEs, Sec. 5.7), both pre-trained by UL to
facilitate subsequent BP-based SL (compare Sec. 5.6.1, 5.10). Sec. 5.16 mentions the ﬁrst SL-based
GPU-CNNs (2006), BP-trained MPCNNs (2007), and LSTM stacks (2007). Sec. 5.17–5.22 focus on
ofﬁcial competitions with secret test sets won by (mostly purely supervised) deep NNs since 2009,
in sequence recognition, image classiﬁcation, image segmentation, and object detection. Many RNN
results depended on LSTM (Sec. 5.13); many FNN results depended on GPU-based FNN code de-
veloped since 2004 (Sec. 5.16, 5.17, 5.18, 5.19), in particular, GPU-MPCNNs (Sec. 5.19). Sec. 5.24
mentions recent tricks for improving DL in NNs, many of them closely related to earlier tricks from
the previous millennium (e.g., Sec. 5.6.2, 5.6.3). Sec. 5.25 discusses how artiﬁcial NNs can help to
understand biological NNs; Sec. 5.26 addresses the possibility of DL in NNs with spiking neurons.
5.1 Early NNs Since the 1940s (and the 1800s)
Early NN architectures (McCulloch and Pitts, 1943) did not learn. The ﬁrst ideas about UL were
published a few years later (Hebb, 1949). The following decades brought simple NNs trained by
SL (e.g., Rosenblatt, 1958, 1962; Widrow and Hoff, 1962; Narendra and Thathatchar, 1974) and
UL (e.g., Grossberg, 1969; Kohonen, 1972; von der Malsburg, 1973; Willshaw and von der Malsburg,
1976), as well as closely related associative memories (e.g., Palm, 1980; Hopﬁeld, 1982).
In a sense NNs have been around even longer, since early supervised NNs were essentially variants
of linear regression methods going back at least to the early 1800s (e.g., Legendre, 1805; Gauss, 1809,
1821); Gauss also refers to his work of 1795. Early NNs had a maximal CAP depth of 1 (Sec. 3).
9
5.2 Around 1960: Visual Cortex Provides Inspiration for DL (Sec. 5.4, 5.11)
Simple cells and complex cells were found in the cat’s visual cortex (e.g., Hubel and Wiesel, 1962;
Wiesel and Hubel, 1959). These cells ﬁre in response to certain properties of visual sensory inputs,
such as the orientation of edges. Complex cells exhibit more spatial invariance than simple cells. This
inspired later deep NN architectures (Sec. 5.4, 5.11) used in certain modern award-winning Deep
Learners (Sec. 5.19–5.22).
5.3 1965: Deep Networks Based on the Group Method of Data Handling
Networks trained by the Group Method of Data Handling (GMDH) (Ivakhnenko and Lapa, 1965;
Ivakhnenko et al., 1967; Ivakhnenko, 1968, 1971) were perhaps the ﬁrst DL systems of the Feed-
forward Multilayer Perceptron type, although there was earlier work on NNs with a single hidden
layer (e.g., Joseph, 1961; Viglione, 1970). The units of GMDH nets may have polynomial activation
functions implementing Kolmogorov-Gabor polynomials (more general than other widely used NN
activation functions, Sec. 2). Given a training set, layers are incrementally grown and trained by re-
gression analysis (e.g., Legendre, 1805; Gauss, 1809, 1821) (Sec. 5.1), then pruned with the help of
a separate validation set (using today’s terminology), whereDecision Regularisation is used to weed
out superﬂuous units (compare Sec. 5.6.3). The numbers of layers and units per layer can be learned
in problem-dependent fashion. To my knowledge, this was the ﬁrst example of open-ended, hierar-
chical representation learning in NNs (Sec. 4.3). A paper of 1971 already described a deep GMDH
network with 8 layers (Ivakhnenko, 1971). There have been numerous applications of GMDH-style
nets, e.g. (Ikeda et al., 1976; Farlow, 1984; Madala and Ivakhnenko, 1994; Ivakhnenko, 1995; Kondo,
1998; Kord´ık et al., 2003; Witczak et al., 2006; Kondo and Ueno, 2008).
5.4 1979: Convolution + Weight Replication + Subsampling (Neocognitron)
Apart from deep GMDH networks (Sec. 5.3), the Neocognitron (Fukushima, 1979, 1980, 2013a)
was perhaps the ﬁrst artiﬁcial NN that deserved the attribute deep, and the ﬁrst to incorporate the
neurophysiological insights of Sec. 5.2. It introduced convolutional NNs (today often called CNNs or
convnets), where the (typically rectangular) receptive ﬁeld of a convolutional unit with given weight
vector (a ﬁlter) is shifted step by step across a 2-dimensional array of input values, such as the pixels
of an image (usually there are several such ﬁlters). The resulting 2D array of subsequent activation
events of this unit can then provide inputs to higher-level units, and so on. Due to massive weight
replication (Sec. 2), relatively few parameters (Sec. 4.4) may be necessary to describe the behavior of
such a convolutional layer.
Subsampling or downsampling layers consist of units whose ﬁxed-weight connections originate
from physical neighbours in the convolutional layers below. Subsampling units become active if at
least one of their inputs is active; their responses are insensitive to certain small image shifts (compare
Sec. 5.2).
The Neocognitron is very similar to the architecture of modern, contest-winning, purely super-
vised, feedforward, gradient-based Deep Learners with alternating convolutional and downsampling
layers (e.g., Sec. 5.19–5.22). Fukushima, however, did not set the weights by supervised backpropa-
gation (Sec. 5.5, 5.8), but by local, WTA-basedunsupervised learning rules (e.g., Fukushima, 2013b),
or by pre-wiring. In that sense he did not care for the DL problem (Sec. 5.9), although his architecture
was comparatively deep indeed. For downsampling purposes he used Spatial Averaging (Fukushima,
1980, 2011) instead of Max-Pooling (MP, Sec. 5.11), currently a particularly convenient and popular
WTA mechanism. Today’s DL combinations of CNNs and MP and BP also proﬁt a lot from later
work (e.g., Sec. 5.8, 5.16, 5.16, 5.19).
10
5.5 1960-1981 and Beyond: Development of Backpropagation (BP) for NNs
The minimisation of errors through gradient descent (Hadamard, 1908) in the parameter space of
complex, nonlinear, differentiable (Leibniz, 1684), multi-stage, NN-related systems has been dis-
cussed at least since the early 1960s (e.g., Kelley, 1960; Bryson, 1961; Bryson and Denham, 1961;
Pontryagin et al., 1961; Dreyfus, 1962; Wilkinson, 1965; Amari, 1967; Bryson and Ho, 1969; Direc-
tor and Rohrer, 1969), initially within the framework of Euler-LaGrange equations in the Calculus of
Variations (e.g., Euler, 1744).
Steepest descent in the weight space of such systems can be performed (Bryson, 1961; Kelley,
1960; Bryson and Ho, 1969) by iterating the chain rule (Leibniz, 1676; L’Hˆopital, 1696) `a la Dynamic
Programming (DP) (Bellman, 1957). A simpliﬁed derivation of this backpropagation method uses the
chain rule only (Dreyfus, 1962).
The systems of the 1960s were already efﬁcient in the DP sense. However, they backpropagated
derivative information through standard Jacobian matrix calculations from one “layer” to the previous
one, without explicitly addressing either direct links across several layers or potential additional efﬁ-
ciency gains due to network sparsity (but perhaps such enhancements seemed obvious to the authors).
Given all the prior work on learning in multilayer NN-like systems (see also Sec. 5.3 on deep non-
linear nets since 1965), it seems surprising in hindsight that a book (Minsky and Papert, 1969) on the
limitations of simple linear perceptrons with a single layer (Sec. 5.1) discouraged some researchers
from further studying NNs.
Explicit, efﬁcient error backpropagation (BP) in arbitrary, discrete, possibly sparsely connected,
NN-like networks apparently was ﬁrst described in a 1970 master’s thesis (Linnainmaa, 1970, 1976),
albeit without reference to NNs. BP is also known as the reverse mode of automatic differentia-
tion (Griewank, 2012), where the costs of forward activation spreading essentially equal the costs of
backward derivative calculation. See early FORTRAN code (Linnainmaa, 1970) and closely related
work (Ostrovskii et al., 1971).
Efﬁcient BP was soon explicitly used to minimize cost functions by adapting control parameters
(weights) (Dreyfus, 1973). Compare some preliminary, NN-speciﬁc discussion (Werbos, 1974, sec-
tion 5.5.1), a method for multilayer threshold NNs (Bobrowski, 1978), and a computer program for
automatically deriving and implementing BP for given differentiable systems (Speelpenning, 1980).
To my knowledge, the ﬁrst NN-speciﬁc application of efﬁcient BP as above was described in
1981 (Werbos, 1981, 2006). Related work was published several years later (Parker, 1985; LeCun,
1985, 1988). A paper of 1986 signiﬁcantly contributed to the popularisation of BP for NNs (Rumelhart
et al., 1986), experimentally demonstrating the emergence of useful internal representations in hidden
layers. See generalisations for sequence-processing recurrent NNs (e.g., Williams, 1989; Robinson
and Fallside, 1987; Werbos, 1988; Williams and Zipser, 1988, 1989b,a; Rohwer, 1989; Pearlmutter,
1989; Gherrity, 1989; Williams and Peng, 1990; Schmidhuber, 1992a; Pearlmutter, 1995; Baldi, 1995;
Kremer and Kolen, 2001; Atiya and Parlos, 2000), also for equilibrium RNNs (Almeida, 1987; Pineda,
1987) with stationary inputs.
5.5.1 BP for Weight-Sharing Feedforward NNs (FNNs) and Recurrent NNs (RNNs)
Using the notation of Sec. 2 for weight-sharing FNNs or RNNs, after an episode of activation spread-
ing through differentiable ft, a single iteration of gradient descent through BP computes changes of
all wi in proportion to ∂E
∂wi
= ∑
t
∂E
∂nett
∂nett
∂wi
as in Algorithm 5.5.1 (for the additive case), where each
weight wi is associated with a real-valued variable △i initialized by 0.
The computational costs of the backward (BP) pass are essentially those of the forward pass
(Sec. 2). Forward and backward passes are re-iterated until sufﬁcient performance is reached.
11
Alg. 5.5.1: One iteration of BP for weight-sharing FNNs or RNNs
for t= T,..., 1 do
to compute ∂E
∂nett
, inititalize real-valued error signal variable δt by 0;
if xt is an input event then continue with next iteration;
if there is an error et then δt := xt −dt;
add to δt the value ∑
k∈outt wv(t,k)δk; (this is the elegant and efﬁcient recursive chain rule
application collecting impacts of nett on future events)
multiply δt by f′
t(nett);
for all k∈int add to △wv(k,t) the value xkδt
end for
change each wi in proportion to △i and a small real-valued learning rate
As of 2014, this simple BP method is still the central learning algorithm for FNNs and RNNs. No-
tably, most contest-winning NNs up to 2014 (Sec. 5.12, 5.14, 5.17, 5.19, 5.21, 5.22) did not augment
supervised BP by some sort of unsupervised learning as discussed in Sec. 5.7, 5.10, 5.15.
5.6 Late 1980s-2000 and Beyond: Numerous Improvements of NNs
By the late 1980s it seemed clear that BP by itself (Sec. 5.5) was no panacea. Most FNN applications
focused on FNNs with few hidden layers. Additional hidden layers often did not seem to offer empiri-
cal beneﬁts. Many practitioners found solace in a theorem (Kolmogorov, 1965a; Hecht-Nielsen, 1989;
Hornik et al., 1989) stating that an NN with a single layer of enough hidden units can approximate
any multivariate continous function with arbitrary accuracy.
Likewise, most RNN applications did not require backpropagating errors far. Many researchers
helped their RNNs by ﬁrst training them on shallow problems (Sec. 3) whose solutions then gener-
alized to deeper problems. In fact, some popular RNN algorithms restricted credit assignment to a
single step backwards (Elman, 1990; Jordan, 1986, 1997), also in more recent studies (Jaeger, 2001;
Maass et al., 2002; Jaeger, 2004).
Generally speaking, although BP allows for deep problems in principle, it seemed to work only
for shallow problems. The late 1980s and early 1990s saw a few ideas with a potential to overcome
this problem, which was fully understood only in 1991 (Sec. 5.9).
5.6.1 Ideas for Dealing with Long Time Lags and Deep CAPs
To deal with long time lags between relevant events, several sequence processing methods were pro-
posed, including Focused BP based on decay factors for activations of units in RNNs (Mozer, 1989,
1992), Time-Delay Neural Networks (TDNNs) (Lang et al., 1990) and their adaptive extension (Bo-
denhausen and Waibel, 1991), Nonlinear AutoRegressive with eXogenous inputs(NARX) RNNs (Lin
et al., 1996), certain hierarchical RNNs (Hihi and Bengio, 1996) (compare Sec. 5.10, 1991), RL
economies in RNNs with WTA units and local learning rules (Schmidhuber, 1989b), and other meth-
ods (e.g., Ring, 1993, 1994; Plate, 1993; de Vries and Principe, 1991; Sun et al., 1993a; Bengio
et al., 1994). However, these algorithms either worked for shallow CAPs only, could not generalize
to unseen CAP depths, had problems with greatly varying time lags between relevant events, needed
external ﬁne tuning of delay constants, or suffered from other problems. In fact, it turned out that
certain simple but deep benchmark problems used to evaluate such methods are more quickly solved
by randomly guessing RNN weights until a solution is found (Hochreiter and Schmidhuber, 1996).
While the RNN methods above were designed for DL of temporal sequences, the Neural Heat
Exchanger (Schmidhuber, 1990c) consists of two parallel deep FNNs with opposite ﬂow directions.
12
Input patterns enter the ﬁrst FNN and are propagated “up”. Desired outputs (targets) enter the “oppo-
site” FNN and are propagated “down”. Using a local learning rule, each layer in each net tries to be
similar (in information content) to the preceding layer and to the adjacent layer of the other net. The
input entering the ﬁrst net slowly “heats up” to become the target. The target entering the opposite net
slowly “cools down” to become the input. The Helmholtz Machine (Dayan et al., 1995; Dayan and
Hinton, 1996) may be viewed as an unsupervised (Sec. 5.6.4) variant thereof (Peter Dayan, personal
communication, 1994).
A hybrid approach (Shavlik and Towell, 1989; Towell and Shavlik, 1994) initializes a poten-
tially deep FNN through a domain theory in propositional logic, which may be acquired through
explanation-based learning (Mitchell et al., 1986; DeJong and Mooney, 1986; Minton et al., 1989).
The NN is then ﬁne-tuned through BP (Sec. 5.5). The NN’s depth reﬂects the longest chain of
reasoning in the original set of logical rules. An extension of this approach (Maclin and Shavlik,
1993; Shavlik, 1994) initializes an RNN by domain knowledge expressed as a Finite State Automa-
ton (FSA). BP-based ﬁne-tuning has become important for later DL systems pre-trained by UL, e.g.,
Sec. 5.10, 5.15.
5.6.2 Better BP Through Advanced Gradient Descent (Compare Sec. 5.24)
Numerous improvements of steepest descent through BP (Sec. 5.5) have been proposed. Least-
squares methods (Gauss-Newton, Levenberg-Marquardt) (Gauss, 1809; Newton, 1687; Levenberg,
1944; Marquardt, 1963; Schaback and Werner, 1992) and quasi-Newton methods (Broyden-Fletcher-
Goldfarb-Shanno, BFGS) (Broyden et al., 1965; Fletcher and Powell, 1963; Goldfarb, 1970; Shanno,
1970) are computationally too expensive for large NNs. Partial BFGS (Battiti, 1992; Saito and
Nakano, 1997) and conjugate gradient (Hestenes and Stiefel, 1952; M øller, 1993) as well as other
methods (Solla, 1988; Schmidhuber, 1989a; Cauwenberghs, 1993) provide sometimes useful fast al-
ternatives. BP can be treated as a linear least-squares problem (Biegler-K ¨onig and B ¨armann, 1993),
where second-order gradient information is passed back to preceding layers.
To speed up BP,momentum was introduced (Rumelhart et al., 1986), ad-hoc constants were added
to the slope of the linearized activation function (Fahlman, 1988), or the nonlinearity of the slope was
exaggerated (West and Saad, 1995).
Only the signs of the error derivatives are taken into account by the successful and widely used
BP variant R-prop (Riedmiller and Braun, 1993) and the robust variation iRprop+ (Igel and H ¨usken,
2003), which was also successfully applied to RNNs.
The local gradient can be normalized based on the NN architecture (Schraudolph and Sejnowski,
1996), through a diagonalized Hessian approach (Becker and Le Cun, 1989), or related efﬁcient meth-
ods (Schraudolph, 2002).
Some algorithms for controlling BP step size adapt a global learning rate (Lapedes and Farber,
1986; V ogl et al., 1988; Battiti, 1989; LeCun et al., 1993; Yu et al., 1995), while others compute in-
dividual learning rates for each weight (Jacobs, 1988; Silva and Almeida, 1990). In online learning,
where BP is applied after each pattern presentation, the vario- η algorithm (Neuneier and Zimmer-
mann, 1996) sets each weight’s learning rate inversely proportional to the empirical standard devia-
tion of its local gradient, thus normalizing the stochastic weight ﬂuctuations. Compare a local online
step size adaptation method for nonlinear NNs (Almeida et al., 1997).
Many additional tricks for improving NNs have been described (e.g., Orr and M¨uller, 1998; Mon-
tavon et al., 2012). Compare Sec. 5.6.3 and recent developments mentioned in Sec. 5.24.
13
5.6.3 Searching For Simple, Low-Complexity, Problem-Solving NNs (Sec. 5.24)
Many researchers used BP-like methods to search for “simple,” low-complexity NNs (Sec. 4.4)
with high generalization capability. Most approaches address the bias/variance dilemma (Geman
et al., 1992) through strong prior assumptions. For example, weight decay (Hanson and Pratt, 1989;
Weigend et al., 1991; Krogh and Hertz, 1992) encourages near-zero weights, by penalizing large
weights. In a Bayesian framework (Bayes, 1763), weight decay can be derived (Hinton and van
Camp, 1993) from Gaussian or Laplacian weight priors (Gauss, 1809; Laplace, 1774); see also (Mur-
ray and Edwards, 1993). An extension of this approach postulates that a distribution of networks with
many similar weights generated by Gaussian mixtures is “better”a priori (Nowlan and Hinton, 1992).
Often weight priors are implicit in additional penalty terms (MacKay, 1992) or in methods based
on validation sets (Mosteller and Tukey, 1968; Stone, 1974; Eubank, 1988; Hastie and Tibshirani,
1990; Craven and Wahba, 1979; Golub et al., 1979), Akaike’s information criterion and ﬁnal pre-
diction error (Akaike, 1970, 1973, 1974), or generalized prediction error (Moody and Utans, 1994;
Moody, 1992). See also (Holden, 1994; Wang et al., 1994; Amari and Murata, 1993; Wang et al.,
1994; Guyon et al., 1992; Vapnik, 1992; Wolpert, 1994). Similar priors (or biases towards simplicity)
are implicit in constructive and pruning algorithms, e.g., layer-by-layer sequential network construc-
tion (e.g., Ivakhnenko, 1968, 1971; Ash, 1989; Moody, 1989; Gallant, 1988; Honavar and Uhr, 1988;
Ring, 1991; Fahlman, 1991; Weng et al., 1992; Honavar and Uhr, 1993; Burgess, 1994; Fritzke, 1994;
Parekh et al., 2000; Utgoff and Stracuzzi, 2002) (see also Sec. 5.3, 5.11),input pruning (Moody, 1992;
Refenes et al., 1994), unit pruning (e.g., Ivakhnenko, 1968, 1971; White, 1989; Mozer and Smolen-
sky, 1989; Levin et al., 1994), weight pruning, e.g., optimal brain damage (LeCun et al., 1990b), and
optimal brain surgeon (Hassibi and Stork, 1993).
A very general but not always practical approach for discovering low-complexity SL NNs or
RL NNs searches among weight matrix-computing programs written in a universal programming
language, with a bias towards fast and short programs (Schmidhuber, 1997) (Sec. 6.7).
Flat Minimum Search (FMS) (Hochreiter and Schmidhuber, 1997a, 1999) searches for a “ﬂat”
minimum of the error function: a large connected region in weight space where error is low and re-
mains approximately constant, that is, few bits of information are required to describe low-precision
weights with high variance. Compare perturbation tolerance conditions (Minai and Williams, 1994;
Murray and Edwards, 1993; Hanson, 1990; Neti et al., 1992; Matsuoka, 1992; Bishop, 1993; Ker-
lirzin and Vallet, 1993; Carter et al., 1990). An MDL-based, Bayesian argument suggests that ﬂat
minima correspond to “simple” NNs and low expected overﬁtting. Compare Sec. 5.6.4 and more
recent developments mentioned in Sec. 5.24.
5.6.4 Potential Beneﬁts of UL for SL (Compare Sec. 5.7, 5.10, 5.15)
The notation of Sec. 2 introduced teacher-given labels dt. Many papers of the previous millennium,
however, were about unsupervised learning (UL) without a teacher (e.g., Hebb, 1949; von der Mals-
burg, 1973; Kohonen, 1972, 1982, 1988; Willshaw and von der Malsburg, 1976; Grossberg, 1976a,b;
Watanabe, 1985; Pearlmutter and Hinton, 1986; Barrow, 1987; Field, 1987; Oja, 1989; Barlow et al.,
1989; Baldi and Hornik, 1989; Sanger, 1989; Ritter and Kohonen, 1989; Rubner and Schulten, 1990;
F¨oldi´ak, 1990; Martinetz et al., 1990; Kosko, 1990; Mozer, 1991; Palm, 1992; Atick et al., 1992;
Miller, 1994; Saund, 1994; F ¨oldi´ak and Young, 1995; Deco and Parra, 1997); see also post-2000
work (e.g., Carreira-Perpinan, 2001; Wiskott and Sejnowski, 2002; Franzius et al., 2007; Waydo and
Koch, 2008).
Many UL methods are designed to maximize entropy-related, information-theoretic (Boltzmann,
1909; Shannon, 1948; Kullback and Leibler, 1951) objectives (e.g., Linsker, 1988; Barlow et al., 1989;
MacKay and Miller, 1990; Plumbley, 1991; Schmidhuber, 1992b,c; Schraudolph and Sejnowski,
14
1993; Redlich, 1993; Zemel, 1993; Zemel and Hinton, 1994; Field, 1994; Hinton et al., 1995; Dayan
and Zemel, 1995; Amari et al., 1996; Deco and Parra, 1997).
Many do this to uncover and disentangle hidden underlying sources of signals (e.g., Jutten and
Herault, 1991; Schuster, 1992; Andrade et al., 1993; Molgedey and Schuster, 1994; Comon, 1994;
Cardoso, 1994; Bell and Sejnowski, 1995; Karhunen and Joutsensalo, 1995; Belouchrani et al., 1997;
Hyv¨arinen et al., 2001; Szab´o et al., 2006; Shan et al., 2007; Shan and Cottrell, 2014).
Many UL methods automatically and robustly generate distributed, sparse representations of in-
put patterns (F¨oldi´ak, 1990; Hinton and Ghahramani, 1997; Lewicki and Olshausen, 1998; Hyv¨arinen
et al., 1999; Hochreiter and Schmidhuber, 1999; Falconbridge et al., 2006) through well-known fea-
ture detectors (e.g., Olshausen and Field, 1996; Schmidhuber et al., 1996), such as off-center-on-
surround-like structures, as well as orientation sensitive edge detectors and Gabor ﬁlters (Gabor,
1946). They extract simple features related to those observed in early visual pre-processing stages
of biological systems (e.g., De Valois et al., 1982; Jones and Palmer, 1987).
UL can also serve to extract invariant features from different data items (e.g., Becker, 1991)
through coupled NNs observing two different inputs (Schmidhuber and Prelinger, 1992), also called
Siamese NNs (e.g., Bromley et al., 1993; Hadsell et al., 2006; Taylor et al., 2011; Chen and Salman,
2011).
UL can help to encode input data in a form advantageous for further processing. In the context
of DL, one important goal of UL is redundancy reduction. Ideally, given an ensemble of input pat-
terns, redundancy reduction through a deep NN will create a factorial code (a code with statistically
independent components) of the ensemble (Barlow et al., 1989; Barlow, 1989), to disentangle the
unknown factors of variation (compare Bengio et al., 2013). Such codes may be sparse and can be
advantageous for (1) data compression, (2) speeding up subsequent BP (Becker, 1991), (3) trivialising
the task of subsequent naive yet optimal Bayes classiﬁers (Schmidhuber et al., 1996).
Most early UL FNNs had a single layer. Methods for deeper UL FNNs include hierarchical
(Sec. 4.3) self-organizing Kohonen maps (e.g., Koikkalainen and Oja, 1990; Lampinen and Oja, 1992;
Versino and Gambardella, 1996; Dittenbach et al., 2000; Rauber et al., 2002), hierarchical Gaussian
potential function networks (Lee and Kil, 1991), layer-wise UL of feature hierarchies fed into SL
classiﬁers (Behnke, 1999, 2003a), the Self-Organising Tree Algorithm (SOTA) (Herrero et al., 2001),
and nonlinear Autoencoders (AEs) with more than 3 (e.g., 5) layers (Kramer, 1991; Oja, 1991; DeMers
and Cottrell, 1993). Such AE NNs (Rumelhart et al., 1986) can be trained to map input patterns
to themselves, for example, by compactly encoding them through activations of units of a narrow
bottleneck hidden layer. Certain nonlinear AEs suffer from certain limitations (Baldi, 2012).
LOCOCODE (Hochreiter and Schmidhuber, 1999) uses FMS (Sec. 5.6.3) to ﬁnd low-complexity
AEs with low-precision weights describable by few bits of information, often producing sparse or
factorial codes. Predictability Minimization (PM) (Schmidhuber, 1992c) searches for factorial codes
through nonlinear feature detectors that ﬁght nonlinear predictors, trying to become both as infor-
mative and as unpredictable as possible. PM-based UL was applied not only to FNNs but also to
RNNs (e.g., Schmidhuber, 1993b; Lindst ¨adt, 1993). Compare Sec. 5.10 on UL-based RNN stacks
(1991), as well as later UL RNNs (e.g., Klapper-Rybicka et al., 2001; Steil, 2007).
5.7 1987: UL Through Autoencoder (AE) Hierarchies (Compare Sec. 5.15)
Perhaps the ﬁrst work to study potential beneﬁts of UL-based pre-training was published in 1987. It
proposed unsupervised AE hierarchies (Ballard, 1987), closely related to certain post-2000 feedfor-
ward Deep Learners based on UL (Sec. 5.15). The lowest-level AE NN with a single hidden layer is
trained to map input patterns to themselves. Its hidden layer codes are then fed into a higher-level AE
of the same type, and so on. The hope is that the codes in the hidden AE layers have properties that
15
facilitate subsequent learning. In one experiment, a particular AE-speciﬁc learning algorithm (dif-
ferent from traditional BP of Sec. 5.5.1) was used to learn a mapping in an AE stack pre-trained by
this type of UL (Ballard, 1987). This was faster than learning an equivalent mapping by BP through
a single deeper AE without pre-training. On the other hand, the task did not really require a deep
AE, that is, the beneﬁts of UL were not that obvious from this experiment. Compare an early sur-
vey (Hinton, 1989) and the somewhat relatedRecursive Auto-Associative Memory (RAAM) (Pollack,
1988, 1990; Melnik et al., 2000), originally used to encode sequential linguistic structures of arbitrary
size through a ﬁxed number of hidden units. More recently, RAAMs were also used as unsupervised
pre-processors to facilitate deep credit assignment for RL (Gisslen et al., 2011) (Sec. 6.4).
In principle, many UL methods (Sec. 5.6.4) could be stacked like the AEs above, the history-
compressing RNNs of Sec. 5.10, the Restricted Boltzmann Machines (RBMs) of Sec. 5.15, or hi-
erarchical Kohonen nets (Sec. 5.6.4), to facilitate subsequent SL. Compare Stacked Generaliza-
tion (Wolpert, 1992; Ting and Witten, 1997), and FNNs that proﬁt from pre-training by competitive
UL (e.g., Rumelhart and Zipser, 1986) prior to BP-based ﬁne-tuning (Maclin and Shavlik, 1995). See
also more recent methods using UL to improve subsequent SL (e.g., Behnke, 1999, 2003a; Escalante-
B. and Wiskott, 2013).
5.8 1989: BP for Convolutional NNs (CNNs, Sec. 5.4)
In 1989, backpropagation (Sec. 5.5) was applied (LeCun et al., 1989, 1990a, 1998) to Neocognitron-
like, weight-sharing, convolutional neural layers (Sec. 5.4) with adaptive connections. This combi-
nation, augmented by Max-Pooling (MP, Sec. 5.11, 5.16), and sped up on graphics cards (Sec. 5.19),
has become an essential ingredient of many modern, competition-winning, feedforward, visual Deep
Learners (Sec. 5.19–5.23). This work also introduced the MNIST data set of handwritten digits (Le-
Cun et al., 1989), which over time has become perhaps the most famous benchmark of Machine
Learning. CNNs helped to achieve good performance on MNIST (LeCun et al., 1990a) (CAP depth
5) and on ﬁngerprint recognition (Baldi and Chauvin, 1993); similar CNNs were used commercially
in the 1990s.
5.9 1991: Fundamental Deep Learning Problem of Gradient Descent
A diploma thesis (Hochreiter, 1991) represented a milestone of explicit DL research. As mentioned
in Sec. 5.6, by the late 1980s, experiments had indicated that traditional deep feedforward or re-
current networks are hard to train by backpropagation (BP) (Sec. 5.5). Hochreiter’s work formally
identiﬁed a major reason: Typical deep NNs suffer from the now famous problem of vanishing or
exploding gradients. With standard activation functions (Sec. 1), cumulative backpropagated error
signals (Sec. 5.5.1) either shrink rapidly, or grow out of bounds. In fact, they decay exponentially in
the number of layers or CAP depth (Sec. 3), or they explode. This is also known as the long time
lag problem. Much subsequent DL research of the 1990s and 2000s was motivated by this insight.
Later work (Bengio et al., 1994) also studied basins of attraction and their stability under noise from a
dynamical systems point of view: either the dynamics are not robust to noise, or the gradients vanish.
See also (Hochreiter et al., 2001a; Tiˇno and Hammer, 2004). Over the years, several ways of partially
overcoming the Fundamental Deep Learning Problem were explored:
I A Very Deep Learner of 1991 (the History Compressor , Sec. 5.10) alleviates the problem
through unsupervised pre-training for a hierarchy of RNNs. This greatly facilitates subsequent
supervised credit assignment through BP (Sec. 5.5). In the FNN case, similar effects can be
achieved through conceptually related AE stacks (Sec. 5.7, 5.15) and Deep Belief Networks
(DBNs, Sec. 5.15).
16
II LSTM-like networks (Sec. 5.13, 5.16, 5.17, 5.21–5.23) alleviate the problem through a special
architecture unaffected by it.
III Today’s GPU-based computers have a million times the computational power of desktop ma-
chines of the early 1990s. This allows for propagating errors a few layers further down within
reasonable time, even in traditional NNs (Sec. 5.18). That is basically what is winning many of
the image recognition competitions now (Sec. 5.19, 5.21, 5.22). (Although this does not really
overcome the problem in a fundamental way.)
IV Hessian-free optimization (Sec. 5.6.2) can alleviate the problem for FNNs (M øller, 1993;
Pearlmutter, 1994; Schraudolph, 2002; Martens, 2010) (Sec. 5.6.2) and RNNs (Martens and
Sutskever, 2011) (Sec. 5.20).
V The space of NN weight matrices can also be searched without relying on error gradients,
thus avoiding the Fundamental Deep Learning Problem altogether. Random weight guessing
sometimes works better than more sophisticated methods (Hochreiter and Schmidhuber, 1996).
Certain more complex problems are better solved by using Universal Search (Levin, 1973b)
for weight matrix-computing programs written in a universal programming language (Schmid-
huber, 1997). Some are better solved by using linear methods to obtain optimal weights for
connections to output events (Sec. 2), and evolving weights of connections to other events—
this is called Evolino (Schmidhuber et al., 2007). Compare also related RNNs pre-trained by
certain UL rules (Steil, 2007), also in the case ofspiking neurons (Yin et al., 2012; Klampﬂ and
Maass, 2013) (Sec. 5.26). Direct search methods are relevant not only for SL but also for more
general RL, and are discussed in more detail in Sec. 6.6.
5.10 1991: UL-Based History Compression Through a Deep Stack of RNNs
A working Very Deep Learner(Sec. 3) of 1991 (Schmidhuber, 1992b, 2013a) could perform credit as-
signment across hundreds of nonlinear operators or neural layers, by using unsupervised pre-training
for a hierarchy of RNNs.
The basic idea is still relevant today. Each RNN is trained for a while in unsupervised fashion to
predict its next input (e.g., Connor et al., 1994; Dorffner, 1996). From then on, only unexpected inputs
(errors) convey new information and get fed to the next higher RNN which thus ticks on a slower, self-
organising time scale. It can easily be shown that no information gets lost. It just gets compressed
(much of machine learning is essentially about compression, e.g., Sec. 4.4, 5.6.3, 6.7). For each
individual input sequence, we get a series of less and less redundant encodings in deeper and deeper
levels of this History Compressor or Neural Sequence Chunker , which can compress data in both
space (like feedforward NNs) and time. This is another good example of hierarchical representation
learning (Sec. 4.3). There also is a continuous variant of the history compressor (Schmidhuber et al.,
1993).
The RNN stack is essentially a deep generative model of the data, which can be reconstructed from
its compressed form. Adding another RNN to the stack improves a bound on the data’s description
length—equivalent to the negative logarithm of its probability (Huffman, 1952; Shannon, 1948)—as
long as there is remaining local learnable predictability in the data representation on the corresponding
level of the hierarchy. Compare a similar observation for feedforward Deep Belief Networks (DBNs,
2006, Sec. 5.15).
The system was able to learn many previously unlearnable DL tasks. One ancient illustrative
DL experiment (Schmidhuber, 1993b) required CAPs (Sec. 3) of depth 1200. The top level code of
the initially unsupervised RNN stack, however, got so compact that (previously infeasible) sequence
classiﬁcation through additional BP-based SL became possible. Essentially the system used UL to
17
greatly reduce problem depth. Compare earlier BP-based ﬁne-tuning of NNs initialized by rules of
propositional logic (Shavlik and Towell, 1989) (Sec. 5.6.1).
There is a way of compressing higher levels down into lower levels, thus fully or partially col-
lapsing the RNN stack. The trick is to retrain a lower-level RNN to continually imitate (predict) the
hidden units of an already trained, slower, higher-level RNN (the “conscious” chunker), through ad-
ditional predictive output neurons (Schmidhuber, 1992b). This helps the lower RNN (theautomatizer)
to develop appropriate, rarely changing memories that may bridge very long time lags. Again, this
procedure can greatly reduce the required depth of the BP process.
The 1991 system was a working Deep Learner in the modern post-2000 sense, and also a ﬁrst
Neural Hierarchical Temporal Memory (HTM). It is conceptually similar to earlier AE hierarchies
(1987, Sec. 5.7) and later Deep Belief Networks (2006, Sec. 5.15), but more general in the sense
that it uses sequence-processing RNNs instead of FNNs with unchanging inputs. More recently,
well-known entrepreneurs (Hawkins and George, 2006; Kurzweil, 2012) also got interested in HTMs;
compare also hierarchical HMMs (e.g., Fine et al., 1998), as well as later UL-based recurrent sys-
tems (Klapper-Rybicka et al., 2001; Steil, 2007; Klampﬂ and Maass, 2013; Young et al., 2014).
Clockwork RNNs (Koutn´ık et al., 2014) also consist of interacting RNN modules with different clock
rates, but do not use UL to set those rates. Stacks of RNNs were used in later work on SL with great
success, e.g., Sec. 5.13, 5.16, 5.17, 5.22.
5.11 1992: Max-Pooling (MP): Towards MPCNNs (Compare Sec. 5.16, 5.19)
The Neocognitron (Sec. 5.4) inspired the Cresceptron (Weng et al., 1992), which adapts its topol-
ogy during training (Sec. 5.6.3); compare the incrementally growing and shrinking GMDH networks
(1965, Sec. 5.3).
Instead of using alternative local subsampling or WTA methods (e.g., Fukushima, 1980; Schmid-
huber, 1989b; Maass, 2000; Fukushima, 2013a), the Cresceptron usesMax-Pooling (MP) layers. Here
a 2-dimensional layer or array of unit activations is partitioned into smaller rectangular arrays. Each
is replaced in a downsampling layer by the activation of its maximally active unit. A later, more com-
plex version of the Cresceptron (Weng et al., 1997) also included“blurring” layers to improve object
location tolerance.
The neurophysiologically plausible topology of the feedforward HMAX model (Riesenhuber and
Poggio, 1999) is very similar to the one of the 1992 Cresceptron (and thus to the 1979 Neocognitron).
HMAX does not learn though. Its units have hand-crafted weights; biologically plausible learning
rules were later proposed for similar models (e.g., Serre et al., 2002; Teichmann et al., 2012).
When CNNs or convnets (Sec. 5.4, 5.8) are combined with MP, they become Cresceptron-like
or HMAX-like MPCNNs with alternating convolutional and max-pooling layers. Unlike Cresceptron
and HMAX, however, MPCNNs are trained by BP (Sec. 5.5, 5.16) (Ranzato et al., 2007). Advantages
of doing this were pointed out subsequently (Scherer et al., 2010). BP-trained MPCNNs have become
central to many modern, competition-winning, feedforward, visual Deep Learners (Sec. 5.17, 5.19–
5.23).
5.12 1994: Early Contest-Winning NNs
Back in the 1990s, certain NNs already won certain controlled pattern recognition contests with secret
test sets. Notably, an NN with internal delay lines won the Santa Fe time-series competition on chaotic
intensity pulsations of an NH3 laser (Wan, 1994; Weigend and Gershenfeld, 1993). No very deep
CAPs (Sec. 3) were needed though.
18
5.13 1995: Supervised Recurrent Very Deep Learner (LSTM RNN)
Supervised Long Short-Term Memory(LSTM) RNN (Hochreiter and Schmidhuber, 1997b; Gers et al.,
2000; P ´erez-Ortiz et al., 2003) could eventually perform similar feats as the deep RNN hierarchy
of 1991 (Sec. 5.10), overcoming the Fundamental Deep Learning Problem (Sec. 5.9) without any
unsupervised pre-training. LSTM could also learn DL tasks without local sequence predictability
(and thus unlearnable by the partially unsupervised 1991 History Compressor, Sec. 5.10), dealing
with very deep problems (Sec. 3) (e.g., Gers et al., 2002).
The basic LSTM idea is very simple. Some of the units are called Constant Error Carousels
(CECs). Each CEC uses as an activation functionf, the identity function, and has a connection to itself
with ﬁxed weight of 1.0. Due to f’s constant derivative of 1.0, errors backpropagated through a CEC
cannot vanish or explode (Sec. 5.9) but stay as they are (unless they “ﬂow out” of the CEC to other,
typically adaptive parts of the NN). CECs are connected to several nonlinear adaptive units (some
with multiplicative activation functions) needed for learning nonlinear behavior. Weight changes of
these units often proﬁt from error signals propagated far back in time through CECs. CECs are
the main reason why LSTM nets can learn to discover the importance of (and memorize) events that
happened thousands of discrete time steps ago, while previous RNNs already failed in case of minimal
time lags of 10 steps.
Many different LSTM variants and topologies are allowed. It is possible to evolve good problem-
speciﬁc topologies (Bayer et al., 2009). Some LSTM variants also use modiﬁable self-connections of
CECs (Gers and Schmidhuber, 2001).
To a certain extent, LSTM is biologically plausible (O’Reilly, 2003). LSTM learned to solve
many previously unlearnable DL tasks involving: Recognition of the temporal order of widely sep-
arated events in noisy input streams; Robust storage of high-precision real numbers across extended
time intervals; Arithmetic operations on continuous input streams; Extraction of information con-
veyed by the temporal distance between events; Recognition of temporally extended patterns in noisy
input sequences (Hochreiter and Schmidhuber, 1997b; Gers et al., 2000); Stable generation of pre-
cisely timed rhythms, as well as smooth and non-smooth periodic trajectories (Gers and Schmidhuber,
2000). LSTM clearly outperformed previous RNNs on tasks that require learning the rules of regu-
lar languages describable by deterministic Finite State Automata (FSAs) (Watrous and Kuhn, 1992;
Casey, 1996; Siegelmann, 1992; Blair and Pollack, 1997; Kalinke and Lehmann, 1998; Zeng et al.,
1994; Manolios and Fanelli, 1994; Omlin and Giles, 1996; Vahed and Omlin, 2004), both in terms of
reliability and speed.
LSTM also worked on tasks involving context free languages (CFLs) that cannot be represented
by HMMs or similar FSAs discussed in the RNN literature (Sun et al., 1993b; Wiles and Elman, 1995;
Andrews et al., 1995; Steijvers and Grunwald, 1996; Tonkes and Wiles, 1997; Rodriguez et al., 1999;
Rodriguez and Wiles, 1998). CFL recognition (Lee, 1996) requires the functional equivalent of a run-
time stack. Some previous RNNs failed to learn small CFL training sets (Rodriguez and Wiles, 1998).
Those that did not (Rodriguez et al., 1999; Bod´en and Wiles, 2000) failed to extract the general rules,
and did not generalize well on substantially larger test sets. Similar for context-sensitive languages
(CSLs) (e.g., Chalup and Blair, 2003). LSTM generalized well though, requiring only the 30 shortest
exemplars (n ≤10) of the CSL anbncn to correctly predict the possible continuations of sequence
preﬁxes for nup to 1000 and more. A combination of a decoupled extended Kalman ﬁlter (Kalman,
1960; Williams, 1992b; Puskorius and Feldkamp, 1994; Feldkamp et al., 1998; Haykin, 2001; Feld-
kamp et al., 2003) and an LSTM RNN (P´erez-Ortiz et al., 2003) learned to deal correctly with values
of n up to 10 million and more. That is, after training the network was able to read sequences of
30,000,000 symbols and more, one symbol at a time, and ﬁnally detect the subtle differences be-
tween legal strings such as a10,000,000b10,000,000c10,000,000 and very similar but illegal strings such
as a10,000,000b9,999,999c10,000,000. Compare also more recent RNN algorithms able to deal with long
19
time lags (Sch¨afer et al., 2006; Martens and Sutskever, 2011; Zimmermann et al., 2012; Koutn´ık et al.,
2014).
Bi-directional RNNs (BRNNs) (Schuster and Paliwal, 1997; Schuster, 1999) are designed for in-
put sequences whose starts and ends are known in advance, such as spoken sentences to be labeled by
their phonemes; compare (Fukada et al., 1999). To take both past and future context of each sequence
element into account, one RNN processes the sequence from start to end, the other backwards from
end to start. At each time step their combined outputs predict the corresponding label (if there is
any). BRNNs were successfully applied to secondary protein structure prediction (Baldi et al., 1999).
DAG-RNNs (Baldi and Pollastri, 2003; Wu and Baldi, 2008) generalize BRNNs to multiple dimen-
sions. They learned to predict properties of small organic molecules (Lusci et al., 2013) as well as
protein contact maps (Tegge et al., 2009), also in conjunction with a growing deep FNN (Di Lena
et al., 2012) (Sec. 5.21). BRNNs and DAG-RNNs unfold their full potential when combined with the
LSTM concept (Graves and Schmidhuber, 2005, 2009; Graves et al., 2009).
Particularly successful in recent competitions are stacks (Sec. 5.10) of LSTM RNNs (Fernan-
dez et al., 2007; Graves and Schmidhuber, 2009) trained by Connectionist Temporal Classiﬁca-
tion (CTC) (Graves et al., 2006), a gradient-based method for ﬁnding RNN weights that maxi-
mize the probability of teacher-given label sequences, given (typically much longer and more high-
dimensional) streams of real-valued input vectors. CTC-LSTM performs simultaneous segmentation
(alignment) and recognition (Sec. 5.22).
In the early 2000s, speech recognition was dominated by HMMs combined with FNNs (e.g.,
Bourlard and Morgan, 1994). Nevertheless, when trained from scratch on utterances from the TIDIG-
ITS speech database, in 2003 LSTM already obtained results comparable to those of HMM-based
systems (Graves et al., 2003; Beringer et al., 2005; Graves et al., 2006). In 2007, LSTM outperformed
HMMs in keyword spotting tasks (Fern´andez et al., 2007); compare recent improvements (Indermuhle
et al., 2011; W ¨ollmer et al., 2013). By 2013, LSTM also achieved best known results on the famous
TIMIT phoneme recognition benchmark (Graves et al., 2013) (Sec. 5.22). Recently, LSTM RNN /
HMM hybrids obtained best known performance on medium-vocabulary (Geiger et al., 2014) and
large-vocabulary speech recognition (Sak et al., 2014a).
LSTM is also applicable to robot localization (F ¨orster et al., 2007), robot control (Mayer et al.,
2008), online driver distraction detection (W¨ollmer et al., 2011), and many other tasks. For example,
it helped to improve the state of the art in diverse applications such as protein analysis (Hochreiter
and Obermayer, 2005), handwriting recognition (Graves et al., 2008, 2009; Graves and Schmidhuber,
2009; Bluche et al., 2014), voice activity detection (Eyben et al., 2013), optical character recogni-
tion (Breuel et al., 2013), language identiﬁcation (Gonzalez-Dominguez et al., 2014), prosody contour
prediction (Fernandez et al., 2014), audio onset detection (Marchi et al., 2014), text-to-speech syn-
thesis (Fan et al., 2014), social signal classiﬁcation (Brueckner and Schulter, 2014), machine transla-
tion (Sutskever et al., 2014), and others.
RNNs can also be used for metalearning (Schmidhuber, 1987; Schaul and Schmidhuber, 2010;
Prokhorov et al., 2002), because they can in principle learn to run their own weight change algo-
rithm (Schmidhuber, 1993a). A successful metalearner (Hochreiter et al., 2001b) used an LSTM
RNN to quickly learn a learning algorithm for quadratic functions (compare Sec. 6.8).
Recently, LSTM RNNs won several international pattern recognition competitions and set nu-
merous benchmark records on large and complex data sets, e.g., Sec. 5.17, 5.21, 5.22. Gradient-
based LSTM is no panacea though—other methods sometimes outperformed it at least on certain
tasks (Jaeger, 2004; Schmidhuber et al., 2007; Martens and Sutskever, 2011; Pascanu et al., 2013b;
Koutn´ık et al., 2014); compare Sec. 5.20.
20
5.14 2003: More Contest-Winning/Record-Setting NNs; Successful Deep NNs
In the decade around 2000, many practical and commercial pattern recognition applications were
dominated by non-neural machine learning methods such as Support Vector Machines (SVMs) (Vap-
nik, 1995; Sch¨olkopf et al., 1998). Nevertheless, at least in certain domains, NNs outperformed other
techniques.
A Bayes NN (Neal, 2006) based on an ensemble (Breiman, 1996; Schapire, 1990; Wolpert, 1992;
Hashem and Schmeiser, 1992; Ueda, 2000; Dietterich, 2000a) of NNs won the NIPS 2003 Feature
Selection Challenge with secret test set (Neal and Zhang, 2006). The NN was not very deep though—
it had two hidden layers and thus rather shallow CAPs (Sec. 3) of depth 3.
Important for many present competition-winning pattern recognisers (Sec. 5.19, 5.21, 5.22) were
developments in the CNN department. A BP-trained (LeCun et al., 1989) CNN (Sec. 5.4, Sec. 5.8) set
a new MNIST record of 0.4% (Simard et al., 2003), using training pattern deformations (Baird, 1990)
but no unsupervised pre-training (Sec. 5.7, 5.10, 5.15). A standard BP net achieved 0.7% (Simard
et al., 2003). Again, the corresponding CAP depth was low. Compare further improvements in
Sec. 5.16, 5.18, 5.19.
Good image interpretation results (Behnke, 2003b) were achieved with rather deep NNs trained
by the BP variant R-prop (Riedmiller and Braun, 1993) (Sec. 5.6.2); here feedback through recurrent
connections helped to improve image interpretation. FNNs with CAP depth up to 6 were used to
successfully classify high-dimensional data (Vieira and Barradas, 2003).
Deep LSTM RNNs started to obtain certain ﬁrst speech recognition results comparable to those
of HMM-based systems (Graves et al., 2003); compare Sec. 5.13, 5.16, 5.21, 5.22.
5.15 2006/7: UL For Deep Belief Networks / AE Stacks Fine-Tuned by BP
While learning networks with numerous non-linear layers date back at least to 1965 (Sec. 5.3), and ex-
plicit DL research results have been published at least since 1991 (Sec. 5.9, 5.10), the expressionDeep
Learning was actually coined around 2006, when unsupervised pre-training of deep FNNs helped to
accelerate subsequent SL through BP (Hinton and Salakhutdinov, 2006; Hinton et al., 2006). Compare
earlier terminology on loading deep networks (S´ıma, 1994; Windisch, 2005) andlearning deep mem-
ories (Gomez and Schmidhuber, 2005). Compare also BP-based (Sec. 5.5) ﬁne-tuning (Sec. 5.6.1) of
(not so deep) FNNs pre-trained by competitive UL (Maclin and Shavlik, 1995).
The Deep Belief Network (DBN) is a stack of Restricted Boltzmann Machines (RBMs) (Smolen-
sky, 1986), which in turn areBoltzmann Machines (BMs) (Hinton and Sejnowski, 1986) with a single
layer of feature-detecting units; compare also Higher-Order BMs (Memisevic and Hinton, 2010).
Each RBM perceives pattern representations from the level below and learns to encode them in un-
supervised fashion. At least in theory under certain assumptions, adding more layers improves a
bound on the data’s negative log probability (Hinton et al., 2006) (equivalent to the data’s description
length—compare the corresponding observation for RNN stacks, Sec. 5.10). There are extensions for
Temporal RBMs (Sutskever et al., 2008).
Without any training pattern deformations (Sec. 5.14), a DBN ﬁne-tuned by BP achieved 1.2%
error rate (Hinton and Salakhutdinov, 2006) on the MNIST handwritten digits (Sec. 5.8, 5.14). This
result helped to arouse interest in DBNs. DBNs also achieved good results on phoneme recognition,
with an error rate of 26.7% on the TIMIT core test set (Mohamed and Hinton, 2010); compare further
improvements through FNNs (Hinton et al., 2012a; Deng and Yu, 2014) and LSTM RNNs (Sec. 5.22).
A DBN-based technique called Semantic Hashing (Salakhutdinov and Hinton, 2009) maps se-
mantically similar documents (of variable size) to nearby addresses in a space of document rep-
resentations. It outperformed previous searchers for similar documents, such as Locality Sensitive
Hashing (Buhler, 2001; Datar et al., 2004). See the RBM/DBN tutorial (Fischer and Igel, 2014).
21
Autoencoder (AE) stacks (Ballard, 1987) (Sec. 5.7) became a popular alternative way of pre-
training deep FNNs in unsupervised fashion, before ﬁne-tuning (Sec. 5.6.1) them through BP
(Sec. 5.5) (Bengio et al., 2007; Vincent et al., 2008; Erhan et al., 2010). Sparse coding (Sec. 5.6.4)
was formulated as a combination of convex optimization problems (Lee et al., 2007a). Recent surveys
of stacked RBM and AE methods focus on post-2006 developments (Bengio, 2009; Arel et al., 2010).
Unsupervised DBNs and AE stacks are conceptually similar to, but in a certain sense less general
than, the unsupervised RNN stack-based History Compressor of 1991 (Sec. 5.10), which can process
and re-encode not only stationary input patterns, but entire pattern sequences.
5.16 2006/7: Improved CNNs / GPU-CNNs / BP for MPCNNs / LSTM Stacks
Also in 2006, a BP-trained (LeCun et al., 1989) CNN (Sec. 5.4, Sec. 5.8) set a new MNIST record
of 0.39% (Ranzato et al., 2006), using training pattern deformations (Sec. 5.14) but no unsupervised
pre-training. Compare further improvements in Sec. 5.18, 5.19. Similar CNNs were used for off-
road obstacle avoidance (LeCun et al., 2006). A combination of CNNs and TDNNs later learned to
map ﬁxed-size representations of variable-size sentences to features relevant for language processing,
using a combination of SL and UL (Collobert and Weston, 2008).
2006 also saw an early GPU-based CNN implementation (Chellapilla et al., 2006) up to 4 times
faster than CPU-CNNs; compare also earlier GPU implementations of standard FNNs with a reported
speed-up factor of 20 (Oh and Jung, 2004). GPUs or graphics cards have become more and more
important for DL in subsequent years (Sec. 5.18–5.22).
In 2007, BP (Sec. 5.5) was applied for the ﬁrst time (Ranzato et al., 2007) to Neocognitron-
inspired (Sec. 5.4), Cresceptron-like (or HMAX-like) MPCNNs (Sec. 5.11) with alternating convo-
lutional and max-pooling layers. BP-trained MPCNNs have become an essential ingredient of many
modern, competition-winning, feedforward, visual Deep Learners (Sec. 5.17, 5.19–5.23).
Also in 2007, hierarchical stacks of LSTM RNNs were introduced (Fernandez et al., 2007). They
can be trained by hierarchical Connectionist Temporal Classiﬁcation(CTC) (Graves et al., 2006). For
tasks of sequence labelling, every LSTM RNN level (Sec. 5.13) predicts a sequence of labels fed to
the next level. Error signals at every level are back-propagated through all the lower levels. On spoken
digit recognition, LSTM stacks outperformed HMMs, despite making fewer assumptions about the
domain. LSTM stacks do not necessarily require unsupervised pre-training like the earlier UL-based
RNN stacks (Schmidhuber, 1992b) of Sec. 5.10.
5.17 2009: First Ofﬁcial Competitions Won by RNNs, and with MPCNNs
Stacks of LSTM RNNs trained by CTC (Sec. 5.13, 5.16) became the ﬁrst RNNs to win ofﬁcial interna-
tional pattern recognition contests (with secret test sets known only to the organisers). More precisely,
three connected handwriting competitions at ICDAR 2009 in three different languages (French, Arab,
Farsi) were won by deep LSTM RNNs without any a priori linguistic knowledge, performing simul-
taneous segmentation and recognition. Compare (Graves and Schmidhuber, 2005; Graves et al., 2009;
Schmidhuber et al., 2011; Graves et al., 2013; Graves and Jaitly, 2014) (Sec. 5.22).
To detect human actions in surveillance videos, a 3-dimensional CNN (e.g., Jain and Seung, 2009;
Prokhorov, 2010), combined with SVMs, was part of a larger system (Yang et al., 2009) using a bag
of features approach (Nowak et al., 2006) to extract regions of interest. The system won three 2009
TRECVID competitions. These were possibly the ﬁrst ofﬁcial international contests won with the
help of (MP)CNNs (Sec. 5.16). An improved version of the method was published later (Ji et al.,
2013).
2009 also saw a GPU-DBN implementation (Raina et al., 2009) orders of magnitudes faster than
previous CPU-DBNs (see Sec. 5.15); see also (Coates et al., 2013). The Convolutional DBN (Lee
22
et al., 2009a) (with a probabilistic variant of MP, Sec. 5.11) combines ideas from CNNs and DBNs,
and was successfully applied to audio classiﬁcation (Lee et al., 2009b).
5.18 2010: Plain Backprop ( + Distortions) on GPU Breaks MNIST Record
In 2010, a new MNIST (Sec. 5.8) record of 0.35% error rate was set by good old BP (Sec. 5.5)
in deep but otherwise standard NNs (Ciresan et al., 2010), using neither unsupervised pre-training
(e.g., Sec. 5.7, 5.10, 5.15) nor convolution (e.g., Sec. 5.4, 5.8, 5.14, 5.16). However, training pattern
deformations (e.g., Sec. 5.14) were important to generate a big training set and avoid overﬁtting. This
success was made possible mainly through a GPU implementation of BP that was up to 50 times
faster than standard CPU versions. A good value of 0.95% was obtained without distortions except
for small saccadic eye movement-like translations—compare Sec. 5.15.
Since BP was 3-5 decades old by then (Sec. 5.5), and pattern deformations 2 decades (Baird, 1990)
(Sec. 5.14), these results seemed to suggest that advances in exploiting modern computing hardware
were more important than advances in algorithms.
5.19 2011: MPCNNs on GPU Achieve Superhuman Vision Performance
In 2011, a ﬂexible GPU-implementation (Ciresan et al., 2011a) of Max-Pooling (MP) CNNs or Con-
vnets was described (a GPU-MPCNN), building on earlier MP work (Weng et al., 1992) (Sec. 5.11)
CNNs (Fukushima, 1979; LeCun et al., 1989) (Sec. 5.4, 5.8, 5.16), and on early GPU-based CNNs
without MP (Chellapilla et al., 2006) (Sec. 5.16); compare early GPU-NNs (Oh and Jung, 2004) and
GPU-DBNs (Raina et al., 2009) (Sec. 5.17). MPCNNs have alternating convolutional layers (Sec. 5.4)
and max-pooling layers (MP, Sec. 5.11) topped by standard fully connected layers. All weights are
trained by BP (Sec. 5.5, 5.8, 5.16) (Ranzato et al., 2007; Scherer et al., 2010). GPU-MPCNNs have
become essential for many contest-winning FNNs (Sec. 5.21, Sec. 5.22).
Multi-Column GPU-MPCNNs (Ciresan et al., 2011b) are committees (Breiman, 1996; Schapire,
1990; Wolpert, 1992; Hashem and Schmeiser, 1992; Ueda, 2000; Dietterich, 2000a) of GPU-
MPCNNs with simple democratic output averaging. Several MPCNNs see the same input; their output
vectors are used to assign probabilities to the various possible classes. The class with the on average
highest probability is chosen as the system’s classiﬁcation of the present input. Compare earlier, more
sophisticated ensemble methods (Schapire, 1990), the contest-winning ensemble Bayes-NN (Neal,
2006) of Sec. 5.14, and recent related work (Shao et al., 2014).
An ensemble of GPU-MPCNNs was the ﬁrst system to achieve superhuman visual pattern recog-
nition (Ciresan et al., 2011b, 2012b) in a controlled competition, namely, the IJCNN 2011 trafﬁc
sign recognition contest in San Jose (CA) (Stallkamp et al., 2011, 2012). This is of interest for fully
autonomous, self-driving cars in trafﬁc (e.g., Dickmanns et al., 1994). The GPU-MPCNN ensem-
ble obtained 0.56% error rate and was twice better than human test subjects, three times better than
the closest artiﬁcial NN competitor (Sermanet and LeCun, 2011), and six times better than the best
non-neural method.
A few months earlier, the qualifying round was won in a 1st stage online competition, albeit by
a much smaller margin: 1.02% (Ciresan et al., 2011b) vs 1.03% for second place (Sermanet and
LeCun, 2011). After the deadline, the organisers revealed that human performance on the test set
was 1.19%. That is, the best methods already seemed human-competitive. However, during the
qualifying it was possible to incrementally gain information about the test set by probing it through
repeated submissions. This is illustrated by better and better results obtained by various teams over
time (Stallkamp et al., 2012) (the organisers eventually imposed a limit of 10 resubmissions). In the
ﬁnal competition this was not possible.
23
This illustrates a general problem with benchmarks whose test sets are public, or at least can be
probed to some extent: competing teams tend to overﬁt on the test set even when it cannot be directly
used for training, only for evaluation.
In 1997 many thought it a big deal that human chess world champion Kasparov was beaten by
an IBM computer. But back then computers could not at all compete with little kids in visual pat-
tern recognition, which seems much harder than chess from a computational perspective. Of course,
the trafﬁc sign domain is highly restricted, and kids are still much better general pattern recognis-
ers. Nevertheless, by 2011, deep NNs could already learn to rival them in important limited visual
domains.
An ensemble of GPU-MPCNNs was also the ﬁrst method to achieve human-competitive perfor-
mance (around 0.2%) on MNIST (Ciresan et al., 2012c). This represented a dramatic improvement,
since by then the MNIST record had hovered around 0.4% for almost a decade (Sec. 5.14, 5.16, 5.18).
Given all the prior work on (MP)CNNs (Sec. 5.4, 5.8, 5.11, 5.16) and GPU-CNNs (Sec. 5.16),
GPU-MPCNNs are not a breakthrough in the scientiﬁc sense. But they are a commercially relevant
breakthrough in efﬁcient coding that has made a difference in several contests since 2011. Today, most
feedforward competition-winning deep NNs are (ensembles of) GPU-MPCNNs (Sec. 5.21–5.23).
5.20 2011: Hessian-Free Optimization for RNNs
Also in 2011 it was shown (Martens and Sutskever, 2011) that Hessian-free optimization (e.g., Møller,
1993; Pearlmutter, 1994; Schraudolph, 2002) (Sec. 5.6.2) can alleviate theFundamental Deep Learn-
ing Problem (Sec. 5.9) in RNNs, outperforming standard gradient-based LSTM RNNs (Sec. 5.13) on
several tasks. Compare other RNN algorithms (Jaeger, 2004; Schmidhuber et al., 2007; Pascanu et al.,
2013b; Koutn´ık et al., 2014) that also at least sometimes yield better results than steepest descent for
LSTM RNNs.
5.21 2012: First Contests Won on ImageNet, Object Detection, Segmentation
In 2012, an ensemble of GPU-MPCNNs (Sec. 5.19) achieved best results on the ImageNet classiﬁca-
tion benchmark (Krizhevsky et al., 2012), which is popular in the computer vision community. Here
relatively large image sizes of 256x256 pixels were necessary, as opposed to only 48x48 pixels for
the 2011 trafﬁc sign competition (Sec. 5.19). See further improvements in Sec. 5.22.
Also in 2012, the biggest NN so far ( 109 free parameters) was trained in unsupervised mode
(Sec. 5.7, 5.15) on unlabeled data (Le et al., 2012), then applied to ImageNet. The codes across its top
layer were used to train a simple supervised classiﬁer, which achieved best results so far on 20,000
classes. Instead of relying on efﬁcient GPU programming, this was done by brute force on 1,000
standard machines with 16,000 cores.
So by 2011/2012, excellent results had been achieved by Deep Learners in imagerecognition and
classiﬁcation (Sec. 5.19, 5.21). The computer vision community, however, is especially interested in
object detection in large images, for applications such as image-based search engines, or for biomed-
ical diagnosis where the goal may be to automatically detect tumors etc in images of human tissue.
Object detection presents additional challenges. One natural approach is to train a deep NN classiﬁer
on patches of big images, then use it as a feature detector to be shifted across unknown visual scenes,
using various rotations and zoom factors. Image parts that yield highly active output units are likely
to contain objects similar to those the NN was trained on.
2012 ﬁnally saw the ﬁrst DL system (an ensemble of GPU-MPCNNs, Sec. 5.19) to win a contest
on visual object detection (Ciresan et al., 2013) in large images of several million pixels (ICPR 2012
Contest on Mitosis Detection in Breast Cancer Histological Images, 2012; Roux et al., 2013). Such
biomedical applications may turn out to be among the most important applications of DL. The world
24
spends over 10% of GDP on healthcare (>6 trillion USD per year), much of it on medical diagnosis
through expensive experts. Partial automation of this could not only save lots of money, but also make
expert diagnostics accessible to many who currently cannot afford it. It is gratifying to observe that
today deep NNs may actually help to improve healthcare and perhaps save human lives.
2012 also saw the ﬁrst pure image segmentation contest won by DL (Ciresan et al., 2012a), again
through an GPU-MPCNN ensemble (Segmentation of Neuronal Structures in EM Stacks Challenge,
2012).2 EM stacks are relevant for the recently approved huge brain projects in Europe and the
US (e.g., Markram, 2012). Given electron microscopy images of stacks of thin slices of animal
brains, the goal is to build a detailed 3D model of the brain’s neurons and dendrites. But human
experts need many hours and days and weeks to annotate the images: Which parts depict neuronal
membranes? Which parts are irrelevant background? This needs to be automated (e.g., Turaga et al.,
2010). Deep Multi-Column GPU-MPCNNs learned to solve this task through experience with many
training images, and won the contest on all three evaluation metrics by a large margin, with superhu-
man performance in terms of pixel error.
Both object detection (Ciresan et al., 2013) and image segmentation (Ciresan et al., 2012a) proﬁt
from fast MPCNN-based image scans that avoid redundant computations. Recent MPCNN scanners
speed up naive implementations by up to three orders of magnitude (Masci et al., 2013; Giusti et al.,
2013); compare earlier efﬁcient methods for CNNs without MP (Vaillant et al., 1994).
Also in 2012, a system consisting of growing deep FNNs and 2D-BRNNs (Di Lena et al., 2012)
won the CASP 2012 contest on protein contact map prediction. On the IAM-OnDoDB benchmark,
LSTM RNNs (Sec. 5.13) outperformed all other methods (HMMs, SVMs) on online mode detec-
tion (Otte et al., 2012; Indermuhle et al., 2012) and keyword spotting (Indermuhle et al., 2011). On the
long time lag problem of language modelling, LSTM RNNs outperformed all statistical approaches
on the IAM-DB benchmark (Frinken et al., 2012); improved results were later obtained through a
combination of NNs and HMMs (Zamora-Martnez et al., 2014). Compare earlier RNNs for object
recognition through iterative image interpretation (Behnke and Rojas, 1998; Behnke, 2002, 2003b);
see also more recent publications (Wyatte et al., 2012; OReilly et al., 2013) extending work on bio-
logically plausible learning rules for RNNs (O’Reilly, 1996).
5.22 2013-: More Contests and Benchmark Records
A stack (Fernandez et al., 2007; Graves and Schmidhuber, 2009) (Sec. 5.10) of bi-directional LSTM
RNNs (Graves and Schmidhuber, 2005) trained by CTC (Sec. 5.13, 5.17) broke a famous TIMIT
speech (phoneme) recognition record, achieving 17.7% test set error rate (Graves et al., 2013), despite
thousands of man years previously spent onHidden Markov Model(HMMs)-based speech recognition
research. Compare earlier DBN results (Sec. 5.15).
CTC-LSTM also helped to score ﬁrst at NIST’s OpenHaRT2013 evaluation (Bluche et al., 2014).
For optical character recognition(OCR), LSTM RNNs outperformed commercial recognizers of his-
torical data (Breuel et al., 2013). LSTM-based systems also set benchmark records in language iden-
tiﬁcation (Gonzalez-Dominguez et al., 2014), medium-vocabulary speech recognition (Geiger et al.,
2014), prosody contour prediction (Fernandez et al., 2014), audio onset detection (Marchi et al.,
2014), text-to-speech synthesis (Fan et al., 2014), and social signal classiﬁcation (Brueckner and
Schulter, 2014).
An LSTM RNN was used to estimate the state posteriors of an HMM; this system beat the previous
state of the art inlarge vocabulary speech recognition(Sak et al., 2014b,a). Another LSTM RNN with
hundreds of millions of connections was used to rerank hypotheses of a statistical machine translation
2It should be mentioned, however, that LSTM RNNs already performed simultaneous segmentation and recognition when
they became the ﬁrst recurrent Deep Learners to win ofﬁcial international pattern recognition contests—see Sec. 5.17.
25
system; this system beat the previous state of the art inEnglish to French translation(Sutskever et al.,
2014).
A new record on the ICDAR Chinese handwriting recognition benchmark (over 3700 classes)
was set on a desktop machine by an ensemble of GPU-MPCNNs (Sec. 5.19) with almost human
performance (Ciresan and Schmidhuber, 2013); compare (Yin et al., 2013).
The MICCAI 2013 Grand Challenge on Mitosis Detection (Veta et al., 2013) also was won by an
object-detecting GPU-MPCNN ensemble (Ciresan et al., 2013). Its data set was even larger and more
challenging than the one of ICPR 2012 (Sec. 5.21): a real-world dataset including many ambiguous
cases and frequently encountered problems such as imperfect slide staining.
Three 2D-CNNs (with mean-pooling instead of MP, Sec. 5.11) observing three orthogonal projec-
tions of 3D images outperformed traditional full 3D methods on the task of segmenting tibial cartilage
in low ﬁeld knee MRI scans (Prasoon et al., 2013).
Deep GPU-MPCNNs (Sec. 5.19) also helped to achieve new best results on important bench-
marks of the computer vision community: ImageNet classiﬁcation (Zeiler and Fergus, 2013; Szegedy
et al., 2014) and—in conjunction with traditional approaches—PASCAL object detection (Girshick
et al., 2013). They also learned to predict bounding box coordinates of objects in the Imagenet
2013 database, and obtained state-of-the-art results on tasks of localization and detection (Sermanet
et al., 2013). GPU-MPCNNs also helped to recognise multi-digit numbers in Google Street View
images (Goodfellow et al., 2014b), where part of the NN was trained to count visible digits; compare
earlier work on detecting “numerosity” through DBNs (Stoianov and Zorzi, 2012). This system also
excelled at recognising distorted synthetic text in reCAPTCHA puzzles. Other successful CNN appli-
cations include scene parsing (Farabet et al., 2013), object detection (Szegedy et al., 2013), shadow
detection (Khan et al., 2014), video classiﬁcation (Karpathy et al., 2014), and Alzheimers disease
neuroimaging (Li et al., 2014).
Additional contests are mentioned in the web pages of the Swiss AI Lab IDSIA, the University of
Toronto, NY University, and the University of Montreal.
5.23 Currently Successful Techniques: LSTM RNNs and GPU-MPCNNs
Most competition-winning or benchmark record-setting Deep Learners actually use one of twosuper-
vised techniques: (a) recurrent LSTM (1997) trained by CTC (2006) (Sec. 5.13, 5.17, 5.21, 5.22), or
(b) feedforward GPU-MPCNNs (2011, Sec. 5.19, 5.21, 5.22) based on CNNs (1979, Sec. 5.4) with
MP (1992, Sec. 5.11) trained through BP (1989–2007, Sec. 5.8, 5.16).
Exceptions include two 2011 contests (Goodfellow et al., 2011; Mesnil et al., 2011; Goodfel-
low et al., 2012) specialised on Transfer Learning from one dataset to another (e.g., Caruana, 1997;
Schmidhuber, 2004; Pan and Yang, 2010). However, deep GPU-MPCNNs do allow forpure SL-based
transfer (Ciresan et al., 2012d), where pre-training on one training set greatly improves performance
on quite different sets, also in more recent studies (Oquab et al., 2013; Donahue et al., 2013). In
fact, deep MPCNNs pre-trained by SL can extract useful features from quite diverse off-training-set
images, yielding better results than traditional, widely used features such as SIFT (Lowe, 1999, 2004)
on many vision tasks (Razavian et al., 2014). To deal with changing datasets, slowly learning deep
NNs were also combined with rapidly adapting “surface” NNs (Kak et al., 2010).
Remarkably, in the 1990s a trend went from partially unsupervised RNN stacks (Sec. 5.10) to
purely supervised LSTM RNNs (Sec. 5.13), just like in the 2000s a trend went from partially unsuper-
vised FNN stacks (Sec. 5.15) to purely supervised MPCNNs (Sec. 5.16–5.22). Nevertheless, in many
applications it can still be advantageous to combine the best of both worlds— supervised learning and
unsupervised pre-training (Sec. 5.10, 5.15).
26
5.24 Recent Tricks for Improving SL Deep NNs (Compare Sec. 5.6.2, 5.6.3)
DBN training (Sec. 5.15) can be improved through gradient enhancements and automatic learning rate
adjustments during stochastic gradient descent (Cho et al., 2013; Cho, 2014), and through Tikhonov-
type (Tikhonov et al., 1977) regularization of RBMs (Cho et al., 2012). Contractive AEs (Rifai et al.,
2011) discourage hidden unit perturbations in response to input perturbations, similar to how FMS
(Sec. 5.6.3) for L OCOCODE AEs (Sec. 5.6.4) discourages output perturbations in response to weight
perturbations.
Hierarchical CNNs in a Neural Abstraction Pyramid (e.g., Behnke, 2003b, 2005) were trained to
reconstruct images corrupted by structured noise (Behnke, 2001), thus enforcing increasingly abstract
image representations in deeper and deeper layers.Denoising AEs later used a similar procedure (Vin-
cent et al., 2008).
Dropout (Hinton et al., 2012b; Ba and Frey, 2013) removes units from NNs during training to
improve generalisation. Some view it as an ensemble method that trains multiple data models simul-
taneously (Baldi and Sadowski, 2014). Under certain circumstances, it could also be viewed as a form
of training set augmentation: effectively, more and more informative complex features are removed
from the training data. Compare dropout for RNNs (Pham et al., 2013; Pachitariu and Sahani, 2013;
Pascanu et al., 2013a). A deterministic approximation coinedfast dropout (Wang and Manning, 2013)
can lead to faster learning and evaluation and was adapted for RNNs (Bayer et al., 2013). Dropout is
closely related to older, biologically plausible techniques for adding noise to neurons or synapses dur-
ing training (e.g., Hanson, 1990; Murray and Edwards, 1993; Schuster, 1992; Nadal and Parga, 1994;
Jim et al., 1995; An, 1996), which in turn are closely related to ﬁnding perturbation-resistant low-
complexity NNs, e.g., through FMS (Sec. 5.6.3). MDL-based stochastic variational methods (Graves,
2011) are also related to FMS. They are useful for RNNs, where classic regularizers such as weight
decay (Sec. 5.6.3) represent a bias towards limited memory capacity (e.g., Pascanu et al., 2013b).
Compare recent work on variational recurrent AEs (Bayer and Osendorfer, 2014).
The activation function f of Rectiﬁed Linear Units (ReLUs) is f(x) = xfor x >0,f(x) = 0
otherwise—compare the old concept of half-wave rectiﬁed units (Malik and Perona, 1990). ReLU
NNs are useful for RBMs (Nair and Hinton, 2010; Maas et al., 2013), outperformed sigmoidal ac-
tivation functions in deep NNs (Glorot et al., 2011), and helped to obtain best results on several
benchmark problems across multiple domains (e.g., Krizhevsky et al., 2012; Dahl et al., 2013).
NNs with competing linear units tend to outperform those with non-competing nonlinear units,
and avoid catastrophic forgetting through BP when training sets change over time (Srivastava et al.,
2013). In this context, choosing a learning algorithm may be more important than choosing activation
functions (Goodfellow et al., 2014a). Maxout NNs (Goodfellow et al., 2013) combine competitive
interactions and dropout (see above) to achieve excellent results on certain benchmarks. Compare
early RNNs with competing units for SL and RL (Schmidhuber, 1989b). To address overﬁtting,
instead of depending on pre-wired regularizers and hyper-parameters (Hertz et al., 1991; Bishop,
2006), self-delimiting RNNs (SLIM NNs) with competing units (Schmidhuber, 2012) can in principle
learn to select their own runtime and their own numbers of effective free parameters, thus learning
their own computable regularisers (Sec. 4.4, 5.6.3), becoming fast andslim when necessary. One may
penalize the task-speciﬁc total length of connections (e.g., Legenstein and Maass, 2002; Schmidhuber,
2012, 2013b; Clune et al., 2013) and communication costs of SLIM NNs implemented on the 3-
dimensional brain-like multi-processor hardware to be expected in the future.
RmsProp (Tieleman and Hinton, 2012; Schaul et al., 2013) can speed up ﬁrst order gradient de-
scent methods (Sec. 5.5, 5.6.2); compare vario-η(Neuneier and Zimmermann, 1996),Adagrad (Duchi
et al., 2011) and Adadelta (Zeiler, 2012). DL in NNs can also be improved by transforming hidden
unit activations such that they have zero output and slope on average (Raiko et al., 2012). Many ad-
ditional, older tricks (Sec. 5.6.2, 5.6.3) should also be applicable to today’s deep NNs; compare (Orr
27
and M¨uller, 1998; Montavon et al., 2012).
5.25 Consequences for Neuroscience
It is ironic that artiﬁcial NNs (ANNs) can help to better understand biological NNs (BNNs)—see
the ISBI 2012 results mentioned in Sec. 5.21 (Segmentation of Neuronal Structures in EM Stacks
Challenge, 2012; Ciresan et al., 2012a).
The feature detectors learned by single-layer visual ANNs are similar to those found in early
visual processing stages of BNNs (e.g., Sec. 5.6.4). Likewise, the feature detectors learned in deep
layers of visual ANNs should be highly predictive of what neuroscientists will ﬁnd in deep layers
of BNNs. While the visual cortex of BNNs may use quite different learning algorithms, its objective
function to be minimised may be quite similar to the one of visual ANNs. In fact, results obtained with
relatively deep artiﬁcial DBNs (Lee et al., 2007b) and CNNs (Yamins et al., 2013) seem compatible
with insights about the visual pathway in the primate cerebral cortex, which has been studied for
many decades (e.g., Hubel and Wiesel, 1968; Perrett et al., 1982; Desimone et al., 1984; Felleman
and Van Essen, 1991; Perrett et al., 1992; Kobatake and Tanaka, 1994; Logothetis et al., 1995; Bichot
et al., 2005; Hung et al., 2005; Lennie and Movshon, 2005; Connor et al., 2007; Kriegeskorte et al.,
2008; DiCarlo et al., 2012); compare a computer vision-oriented survey (Kruger et al., 2013).
5.26 DL with Spiking Neurons?
Many recent DL results proﬁt from GPU-based traditional deep NNs, e.g., Sec. 5.16–5.19. Current
GPUs, however, are little ovens, much hungrier for energy than biological brains, whose neurons ef-
ﬁciently communicate by brief spikes (Hodgkin and Huxley, 1952; FitzHugh, 1961; Nagumo et al.,
1962), and often remain quiet. Many computational models of such spiking neurons have been pro-
posed and analyzed (e.g., Gerstner and van Hemmen, 1992; Zipser et al., 1993; Stemmler, 1996;
Tsodyks et al., 1996; Maex and Orban, 1996; Maass, 1996, 1997; Kistler et al., 1997; Amit and
Brunel, 1997; Tsodyks et al., 1998; Kempter et al., 1999; Song et al., 2000; Stoop et al., 2000; Brunel,
2000; Bohte et al., 2002; Gerstner and Kistler, 2002; Izhikevich et al., 2003; Seung, 2003; Deco and
Rolls, 2005; Brette et al., 2007; Brea et al., 2013; Nessler et al., 2013; Kasabov, 2014; Hoerzer et al.,
2014; Rezende and Gerstner, 2014).
Future energy-efﬁcient hardware for DL in NNs may implement aspects of such models (e.g.,
Liu et al., 2001; Roggen et al., 2003; Glackin et al., 2005; Schemmel et al., 2006; Fieres et al., 2008;
Khan et al., 2008; Serrano-Gotarredona et al., 2009; Jin et al., 2010; Indiveri et al., 2011; Neil and Liu,
2014; Merolla et al., 2014). A simulated, event-driven, spiking variant (Neftci et al., 2014) of an RBM
(Sec. 5.15) was trained by a variant of the Contrastive Divergence algorithm (Hinton, 2002). Spiking
nets were evolved to achieve reasonable performance on small face recognition data sets (Wysoski
et al., 2010) and to control simple robots (Floreano and Mattiussi, 2001; Hagras et al., 2004). A
spiking DBN with about 250,000 neurons (as part of a larger NN; Eliasmith et al., 2012; Eliasmith,
2013) achieved 6% error rate on MNIST; compare similar results with a spiking DBN variant of
depth 3 using a neuromorphic event-based sensor (O’Connor et al., 2013). In practical applications,
however, current artiﬁcial networks of spiking neurons cannot yet compete with the best traditional
deep NNs (e.g., compare MNIST results of Sec. 5.19).
28
6 DL in FNNs and RNNs for Reinforcement Learning (RL)
So far we have focused on Deep Learning (DL) in supervised or unsupervised NNs. Such NNs learn
to perceive / encode / predict / classify patterns or pattern sequences, but they do not learn to act
in the more general sense of Reinforcement Learning (RL) in unknown environments (see surveys,
e.g., Kaelbling et al., 1996; Sutton and Barto, 1998; Wiering and van Otterlo, 2012). Here we add a
discussion of DL FNNs and RNNs for RL. It will be shorter than the discussion of FNNs and RNNs
for SL and UL (Sec. 5), reﬂecting the current size of the various ﬁelds.
Without a teacher, solely from occasional real-valued pain and pleasure signals, RL agents must
discover how to interact with a dynamic, initially unknown environment to maximize their expected
cumulative reward signals (Sec. 2). There may be arbitrary, a priori unknown delays between actions
and perceivable consequences. The problem is as hard as any problem of computer science, since any
task with a computable description can be formulated in the RL framework (e.g., Hutter, 2005). For
example, an answer to the famous question of whether P = NP (Levin, 1973b; Cook, 1971) would
also set limits for what is achievable by general RL. Compare more speciﬁc limitations, e.g., (Blondel
and Tsitsiklis, 2000; Madani et al., 2003; Vlassis et al., 2012). The following subsections mostly focus
on certain obvious intersections between DL and RL—they cannot serve as a general RL survey.
6.1 RL Through NN World Models Yields RNNs With Deep CAPs
In the special case of an RL FNN controller C interacting with a deterministic, predictable environ-
ment, a separate FNN called M can learn to become C’sworld model through system identiﬁcation,
predicting C’s inputs from previous actions and inputs (e.g., Werbos, 1981, 1987; Munro, 1987; Jor-
dan, 1988; Werbos, 1989b,a; Robinson and Fallside, 1989; Jordan and Rumelhart, 1990; Schmidhu-
ber, 1990d; Narendra and Parthasarathy, 1990; Werbos, 1992; Gomi and Kawato, 1993; Cochocki and
Unbehauen, 1993; Levin and Narendra, 1995; Miller et al., 1995; Ljung, 1998; Prokhorov et al., 2001;
Ge et al., 2010). Assume M has learned to produce accurate predictions. We can use M to substi-
tute the environment. Then M and C form an RNN where M’s outputs become inputs of C, whose
outputs (actions) in turn become inputs of M. Now BP for RNNs (Sec. 5.5.1) can be used to achieve
desired input events such as high real-valued reward signals: While M’s weights remain ﬁxed, gradi-
ent information for C’s weights is propagated back throughM down into Cand back through M etc.
To a certain extent, the approach is also applicable in probabilistic or uncertain environments, as long
as the inner products of M’sC-based gradient estimates and M’s “true” gradients tend to be positive.
In general, this approach implies deep CAPs for C, unlike in DP-based traditional RL (Sec. 6.2).
Decades ago, the method was used to learn to back up a model truck (Nguyen and Widrow, 1989).
An RL active vision system used it to learn sequential shifts (saccades) of a fovea, to detect targets in
visual scenes (Schmidhuber and Huber, 1991), thus learning to control selective attention. Compare
RL-based attention learning without NNs (Whitehead, 1992).
To allow for memories of previous events inpartially observable worlds (Sec. 6.3), the most gen-
eral variant of this technique uses RNNs instead of FNNs to implement bothMand C (Schmidhuber,
1990d, 1991c; Feldkamp and Puskorius, 1998). This may cause deep CAPs not only for C but also
for M.
M can also be used to optimize expected reward by planning future action sequences (Schmid-
huber, 1990d). In fact, the winners of the 2004 RoboCup World Championship in the fast
league (Egorova et al., 2004) trained NNs to predict the effects of steering signals on fast robots
with 4 motors for 4 different wheels. During play, such NN models were used to achieve desirable
subgoals, by optimizing action sequences through quickly planning ahead. The approach also was
used to create self-healing robots able to compensate for faulty motors whose effects do not longer
match the predictions of the NN models (Gloye et al., 2005; Schmidhuber, 2007).
29
Typically M is not given in advance. Then an essential question is: which experiments should C
conduct to quickly improveM? The Formal Theory of Fun and Creativity(e.g., Schmidhuber, 2006a,
2013b) formalizes driving forces and value functions behind such curious and exploratory behavior:
A measure of the learning progress of M becomes the intrinsic reward of C (Schmidhuber, 1991a);
compare (Singh et al., 2005; Oudeyer et al., 2013). This motivates C to create action sequences
(experiments) such that M makes quick progress.
6.2 Deep FNNs for Traditional RL and Markov Decision Processes (MDPs)
The classical approach to RL (Samuel, 1959; Bertsekas and Tsitsiklis, 1996) makes the simplifying
assumption of Markov Decision Processes (MDPs): the current input of the RL agent conveys all
information necessary to compute an optimal next output event or decision. This allows for greatly
reducing CAP depth in RL NNs (Sec. 3, 6.1) by using theDynamic Programming(DP) trick (Bellman,
1957). The latter is often explained in a probabilistic framework (e.g., Sutton and Barto, 1998), but
its basic idea can already be conveyed in a deterministic setting. For simplicity, using the notation
of Sec. 2, let input events xt encode the entire current state of the environment, including a real-
valued reward rt (no need to introduce additional vector-valued notation, since real values can encode
arbitrary vectors of real values). The original RL goal (ﬁnd weights that maximize the sum of all
rewards of an episode) is replaced by an equivalent set of alternative goals set by a real-valued value
function V deﬁned on input events. Consider any two subsequent input events xt,xk. Recursively
deﬁne V(xt) = rt + V(xk), where V(xk) = rk if xk is the last input event. Now search for weights
that maximize the V of all input events, by causing appropriate output events or actions.
Due to the Markov assumption, an FNN sufﬁces to implement the policy that maps input to out-
put events. Relevant CAPs are not deeper than this FNN. V itself is often modeled by a separate
FNN (also yielding typically short CAPs) learning to approximateV(xt) only from local information
rt,V (xk).
Many variants of traditional RL exist (e.g., Barto et al., 1983; Watkins, 1989; Watkins and Dayan,
1992; Moore and Atkeson, 1993; Schwartz, 1993; Rummery and Niranjan, 1994; Singh, 1994; Baird,
1995; Kaelbling et al., 1995; Peng and Williams, 1996; Mahadevan, 1996; Tsitsiklis and van Roy,
1996; Bradtke et al., 1996; Santamar´ıa et al., 1997; Prokhorov and Wunsch, 1997; Sutton and Barto,
1998; Wiering and Schmidhuber, 1998b; Baird and Moore, 1999; Meuleau et al., 1999; Morimoto and
Doya, 2000; Bertsekas, 2001; Brafman and Tennenholtz, 2002; Abounadi et al., 2002; Lagoudakis and
Parr, 2003; Sutton et al., 2008; Maei and Sutton, 2010; van Hasselt, 2012). Most are formulated in
a probabilistic framework, and evaluate pairs of input and output (action) events (instead of input
events only). To facilitate certain mathematical derivations, some discount delayed rewards, but such
distortions of the original RL problem are problematic.
Perhaps the most well-known RL NN is the world-class RL backgammon player (Tesauro, 1994),
which achieved the level of human world champions by playing against itself. Its nonlinear, rather
shallow FNN maps a large but ﬁnite number of discrete board states to values. More recently, a
rather deep GPU-CNN was used in a traditional RL framework to play several Atari 2600 computer
games directly from 84x84 pixel 60 Hz video input (Mnih et al., 2013), using experience replay (Lin,
1993), extending previous work on Neural Fitted Q-Learning (NFQ) (Riedmiller, 2005). Even bet-
ter results are achieved by using (slow) Monte Carlo tree planning to train comparatively fast deep
NNs (Guo et al., 2014). Compare RBM-based RL (Sallans and Hinton, 2004) with high-dimensional
inputs (Elfwing et al., 2010), earlier RL Atari players (Gr¨uttner et al., 2010), and an earlier, raw video-
based RL NN for computer games (Koutn´ık et al., 2013) trained byIndirect Policy Search(Sec. 6.7).
30
6.3 Deep RL RNNs for Partially Observable MDPs (POMDPs)
The Markov assumption (Sec. 6.2) is often unrealistic. We cannot directly perceive what is behind our
back, let alone the current state of the entire universe. However, memories of previous events can help
to deal with partially observable Markov decision problems (POMDPs) (e.g., Schmidhuber, 1990d,
1991c; Ring, 1991, 1993, 1994; Williams, 1992a; Lin, 1993; Teller, 1994; Kaelbling et al., 1995;
Littman et al., 1995; Boutilier and Poole, 1996; Jaakkola et al., 1995; McCallum, 1996; Kimura et al.,
1997; Wiering and Schmidhuber, 1996, 1998a; Otsuka et al., 2010). A naive way of implementing
memories without leaving the MDP framework (Sec. 6.2) would be to simply consider a possibly huge
state space, namely, the set of all possible observation histories and their preﬁxes. A more realistic
way is to use function approximators such as RNNs that produce compact state features as a function
of the entire history seen so far. Generally speaking, POMDP RL often uses DL RNNs to learn which
events to memorize and which to ignore. Three basic alternatives are:
1. Use an RNN as a value function mapping arbitrary event histories to values (e.g., Schmidhuber,
1990b, 1991c; Lin, 1993; Bakker, 2002). For example, deep LSTM RNNs were used in this
way for RL robots (Bakker et al., 2003).
2. Use an RNN controller in conjunction with a second RNN as predictive world model, to obtain
a combined RNN with deep CAPs—see Sec. 6.1.
3. Use an RNN for RL by Direct Search (Sec. 6.6) or Indirect Search (Sec. 6.7) in weight space.
In general, however, POMDPs may imply greatly increased CAP depth.
6.4 RL Facilitated by Deep UL in FNNs and RNNs
RL machines may proﬁt from UL for input preprocessing (e.g., Jodogne and Piater, 2007). In partic-
ular, an UL NN can learn to compactly encode environmental inputs such as images or videos, e.g.,
Sec. 5.7, 5.10, 5.15. The compact codes (instead of the high-dimensional raw data) can be fed into an
RL machine, whose job thus may become much easier (Legenstein et al., 2010; Cuccu et al., 2011),
just like SL may proﬁt from UL, e.g., Sec. 5.7, 5.10, 5.15. For example, NFQ (Riedmiller, 2005) was
applied to real-world control tasks (Lange and Riedmiller, 2010; Riedmiller et al., 2012) where purely
visual inputs were compactly encoded by deep autoencoders (Sec. 5.7, 5.15). RL combined with UL
based on Slow Feature Analysis (Wiskott and Sejnowski, 2002; Kompella et al., 2012) enabled a real
humanoid robot to learn skills from raw high-dimensional video streams (Luciw et al., 2013). To
deal with POMDPs (Sec. 6.3) involving high-dimensional inputs, RBM-based RL was used (Otsuka,
2010), and a RAAM (Pollack, 1988) (Sec. 5.7) was employed as a deep unsupervised sequence en-
coder for RL (Gisslen et al., 2011). Certain types of RL and UL also were combined in biologically
plausible RNNs with spiking neurons (Sec. 5.26) (e.g., Yin et al., 2012; Klampﬂ and Maass, 2013;
Rezende and Gerstner, 2014).
6.5 Deep Hierarchical RL (HRL) and Subgoal Learning with FNNs and RNNs
Multiple learnable levels of abstraction (Fu, 1977; Lenat and Brown, 1984; Ring, 1994; Bengio
et al., 2013; Deng and Yu, 2014) seem as important for RL as for SL. Work on NN-based Hierar-
chical RL (HRL) has been published since the early 1990s. In particular, gradient-based subgoal
discovery with FNNs or RNNs decomposes RL tasks into subtasks for RL submodules (Schmid-
huber, 1991b; Schmidhuber and Wahnsiedler, 1992). Numerous alternative HRL techniques have
been proposed (e.g., Ring, 1991, 1994; Jameson, 1991; Tenenberg et al., 1993; Weiss, 1994; Moore
and Atkeson, 1995; Precup et al., 1998; Dietterich, 2000b; Menache et al., 2002; Doya et al., 2002;
31
Ghavamzadeh and Mahadevan, 2003; Barto and Mahadevan, 2003; Samejima et al., 2003; Bakker and
Schmidhuber, 2004; Whiteson et al., 2005; Simsek and Barto, 2008). While HRL frameworks such as
Feudal RL(Dayan and Hinton, 1993) andoptions (Sutton et al., 1999b; Barto et al., 2004; Singh et al.,
2005) do not directly address the problem of automatic subgoal discovery,HQ-Learning (Wiering and
Schmidhuber, 1998a) automatically decomposes POMDPs (Sec. 6.3) into sequences of simpler sub-
tasks that can be solved by memoryless policies learnable by reactive sub-agents. Recent HRL orga-
nizes potentially deep NN-based RL sub-modules into self-organizing, 2-dimensional motor control
maps (Ring et al., 2011) inspired by neurophysiological ﬁndings (Graziano, 2009).
6.6 Deep RL by Direct NN Search / Policy Gradients / Evolution
Not quite as universal as the methods of Sec. 6.8, yet both practical and more general than most
traditional RL algorithms (Sec. 6.2), are methods for Direct Policy Search (DS). Without a need for
value functions or Markovian assumptions (Sec. 6.2, 6.3), the weights of an FNN or RNN are directly
evaluated on the given RL problem. The results of successive trials inform further search for better
weights. Unlike with RL supported by BP (Sec. 5.5, 6.3, 6.1), CAP depth (Sec. 3, 5.9) is not a crucial
issue. DS may solve the credit assignment problem without backtracking through deep causal chains
of modiﬁable parameters—it neither cares for their existence, nor tries to exploit them.
An important class of DS methods for NNs are Policy Gradient methods (Williams, 1986, 1988,
1992a; Sutton et al., 1999a; Baxter and Bartlett, 2001; Aberdeen, 2003; Ghavamzadeh and Mahade-
van, 2003; Kohl and Stone, 2004; Wierstra et al., 2008; R ¨uckstieß et al., 2008; Peters and Schaal,
2008b,a; Sehnke et al., 2010; Gr ¨uttner et al., 2010; Wierstra et al., 2010; Peters, 2010; Grondman
et al., 2012; Heess et al., 2012). Gradients of the total reward with respect to policies (NN weights)
are estimated (and then exploited) through repeated NN evaluations.
RL NNs can also be evolved through Evolutionary Algorithms (EAs) (Rechenberg, 1971; Schwe-
fel, 1974; Holland, 1975; Fogel et al., 1966; Goldberg, 1989) in a series of trials. Here several policies
are represented by a population of NNs improved through mutations and/or repeated recombinations
of the population’s ﬁttest individuals (e.g., Montana and Davis, 1989; Fogel et al., 1990; Maniezzo,
1994; Happel and Murre, 1994; Nolﬁ et al., 1994b). Compare Genetic Programming (GP) (Cramer,
1985) (see also Smith, 1980) which can be used to evolve computer programs of variable size (Dick-
manns et al., 1987; Koza, 1992), and Cartesian GP (Miller and Thomson, 2000; Miller and Harding,
2009) for evolving graph-like programs, including NNs (Khan et al., 2010) and their topology (Turner
and Miller, 2013). Related methods include probability distribution-based EAs (Baluja, 1994; Sar-
avanan and Fogel, 1995; Sałustowicz and Schmidhuber, 1997; Larraanaga and Lozano, 2001), Co-
variance Matrix Estimation Evolution Strategies (CMA-ES) (Hansen and Ostermeier, 2001; Hansen
et al., 2003; Igel, 2003; Heidrich-Meisner and Igel, 2009), andNeuroEvolution of Augmenting Topolo-
gies (NEAT) (Stanley and Miikkulainen, 2002). Hybrid methods combine traditional NN-based RL
(Sec. 6.2) and EAs (e.g., Whiteson and Stone, 2006).
Since RNNs are general computers, RNN evolution is like GP in the sense that it can evolve
general programs. Unlike sequential programs learned by traditional GP, however, RNNs can mix
sequential and parallel information processing in a natural and efﬁcient way, as already mentioned in
Sec. 1. Many RNN evolvers have been proposed (e.g., Miller et al., 1989; Wieland, 1991; Cliff et al.,
1993; Yao, 1993; Nolﬁ et al., 1994a; Sims, 1994; Yamauchi and Beer, 1994; Miglino et al., 1995;
Moriarty, 1997; Pasemann et al., 1999; Juang, 2004; Whiteson, 2012). One particularly effective
family of methods coevolves neurons, combining them into networks, and selecting those neurons
for reproduction that participated in the best-performing networks (Moriarty and Miikkulainen, 1996;
Gomez, 2003; Gomez and Miikkulainen, 2003). This can help to solve deep POMDPs (Gomez and
Schmidhuber, 2005). Co-Synaptic Neuro-Evolution (CoSyNE) does something similar on the level of
32
synapses or weights (Gomez et al., 2008); beneﬁts of this were shown on difﬁcult nonlinear POMDP
benchmarks.
Natural Evolution Strategies (NES) (Wierstra et al., 2008; Glasmachers et al., 2010; Sun et al.,
2009, 2013) link policy gradient methods and evolutionary approaches through the concept ofNatural
Gradients (Amari, 1998). RNN evolution may also help to improve SL for deep RNNs through
Evolino (Schmidhuber et al., 2007) (Sec. 5.9).
6.7 Deep RL by Indirect Policy Search / Compressed NN Search
Some DS methods (Sec. 6.6) can evolve NNs with hundreds or thousands of weights, but not mil-
lions. How to search for large and deep NNs? Most SL and RL methods mentioned so far somehow
search the space of weights wi. Some proﬁt from a reduction of the search space through shared
wi that get reused over and over again, e.g., in CNNs (Sec. 5.4, 5.8, 5.16, 5.21), or in RNNs for SL
(Sec. 5.5, 5.13, 5.17) and RL (Sec. 6.1, 6.3, 6.6).
It may be possible, however, to exploitadditional regularities/compressibilities in the space of so-
lutions, through indirect search in weight space. Instead of evolving large NNs directly (Sec. 6.6), one
can sometimes greatly reduce the search space by evolving compact encodings of NNs, e.g., through
Lindenmeyer Systems (Lindenmayer, 1968; Jacob et al., 1994),graph rewriting(Kitano, 1990), Cellu-
lar Encoding (Gruau et al., 1996), HyperNEAT (D’Ambrosio and Stanley, 2007; Stanley et al., 2009;
Clune et al., 2011; van den Berg and Whiteson, 2013) (extending NEAT; Sec. 6.6), and extensions
thereof (e.g., Risi and Stanley, 2012). This helps to avoid overﬁtting (compare Sec. 5.6.3, 5.24) and is
closely related to the topics of regularisation and MDL (Sec. 4.4).
A general approach (Schmidhuber, 1997) for both SL and RL seeks to compactly encode weights
of large NNs (Schmidhuber, 1997) through programs written in a universal programming lan-
guage (G¨odel, 1931; Church, 1936; Turing, 1936; Post, 1936). Often it is much more efﬁcient to
systematically search the space of such programs with a bias towards short and fast programs (Levin,
1973b; Schmidhuber, 1997, 2004), instead of directly searching the huge space of possible NN weight
matrices. A previous universal language for encoding NNs was assembler-like (Schmidhuber, 1997).
More recent work uses more practical languages based on coefﬁcients of popular transforms (Fourier,
wavelet, etc). In particular, RNN weight matrices may be compressed like images, by encoding them
through the coefﬁcients of a discrete cosine transform (DCT) (Koutn´ık et al., 2010, 2013). Compact
DCT-based descriptions can be evolved through NES or CoSyNE (Sec. 6.6). An RNN with over a
million weights learned (without a teacher) to drive a simulated car in the TORCS driving game (Loia-
cono et al., 2009, 2011), based on a high-dimensional video-like visual input stream (Koutn ´ık et al.,
2013). The RNN learned both control and visual processing from scratch, without being aided by
UL. (Of course, UL might help to generate more compact image codes (Sec. 6.4, 4.2) to be fed into a
smaller RNN, to reduce the overall computational effort.)
6.8 Universal RL
General purpose learning algorithms may improve themselves in open-ended fashion and
environment-speciﬁc ways in a lifelong learning context (Schmidhuber, 1987; Schmidhuber et al.,
1997b,a; Schaul and Schmidhuber, 2010). The most general type of RL is constrained only by the
fundamental limitations of computability identiﬁed by the founders of theoretical computer science
(G¨odel, 1931; Church, 1936; Turing, 1936; Post, 1936). Remarkably, there exist blueprints ofuniver-
sal problem solvers or universal RL machines for unlimited problem depth that are time-optimal in
various theoretical senses (Hutter, 2005, 2002; Schmidhuber, 2002, 2006b). In particular, the G¨odel
Machine can be implemented on general computers such as RNNs and may improve any part of its
software (including the learning algorithm itself) in a way that is provably time-optimal in a certain
33
sense (Schmidhuber, 2006b). It can be initialized by an asymptotically optimal meta-method (Hut-
ter, 2002) (also applicable to RNNs) which will solve any well-deﬁned problem as quickly as the
unknown fastest way of solving it, save for an additive constant overhead that becomes negligible as
problem size grows. Note that most problems are large; only few are small. AI and DL researchers are
still in business because many are interested in problems so small that it is worth trying to reduce the
overhead through less general methods, including heuristics. Here I won’t further discuss universal
RL methods, which go beyond what is usually called DL.
7 Conclusion and Outlook
Deep Learning (DL) in Neural Networks (NNs) is relevant for Supervised Learning (SL) (Sec. 5),
Unsupervised Learning (UL) (Sec. 5), and Reinforcement Learning (RL) (Sec. 6). By alleviating
problems with deep Credit Assignment Paths (CAPs, Sec. 3, 5.9), UL (Sec. 5.6.4) can not only facil-
itate SL of sequences (Sec. 5.10) and stationary patterns (Sec. 5.7, 5.15), but also RL (Sec. 6.4, 4.2).
Dynamic Programming(DP, Sec. 4.1) is important for both deep SL (Sec. 5.5) and traditional RL with
deep NNs (Sec. 6.2). A search for solution-computing, perturbation-resistant (Sec. 5.6.3, 5.15, 5.24),
low-complexity NNs describable by few bits of information (Sec. 4.4) can reduce overﬁtting and im-
prove deep SL & UL (Sec. 5.6.3, 5.6.4) as well as RL (Sec. 6.7), also in the case of partially observable
environments (Sec. 6.3). Deep SL, UL, RL often create hierarchies of more and more abstract repre-
sentations of stationary data (Sec. 5.3, 5.7, 5.15), sequential data (Sec. 5.10), or RL policies (Sec. 6.5).
While UL can facilitate SL, pure SL for feedforward NNs (FNNs) (Sec. 5.5, 5.8, 5.16, 5.18) and re-
current NNs (RNNs) (Sec. 5.5, 5.13) did not only win early contests (Sec. 5.12, 5.14) but also most
of the recent ones (Sec. 5.17–5.22). Especially DL in FNNs proﬁted from GPU implementations
(Sec. 5.16–5.19). In particular, GPU-based (Sec. 5.19) Max-Pooling (Sec. 5.11) Convolutional NNs
(Sec. 5.4, 5.8, 5.16) won competitions not only in pattern recognition (Sec. 5.19–5.22) but also image
segmentation (Sec. 5.21) and object detection (Sec. 5.21, 5.22).
Unlike these systems, humans learn to actively perceive patterns by sequentially directing atten-
tion to relevant parts of the available data. Near future deep NNs will do so, too, extending previous
work since 1990 on NNs that learn selective attention through RL of (a)motor actions such as saccade
control (Sec. 6.1) and (b)internal actions controlling spotlights of attention within RNNs, thus closing
the general sensorimotor loop through both external and internal feedback (e.g., Sec. 2, 5.21, 6.6, 6.7).
Many future deep NNs will also take into account that it costs energy to activate neurons, and to
send signals between them. Brains seem to minimize such computational costs during problem solv-
ing in at least two ways: (1) At a given time, only a small fraction of all neurons is active because local
competition through winner-take-all mechanisms shuts down many neighbouring neurons, and only
winners can activate other neurons through outgoing connections (compare SLIM NNs; Sec. 5.24).
(2) Numerous neurons are sparsely connected in a compact 3D volume by many short-range and
few long-range connections (much like microchips in traditional supercomputers). Often neighbour-
ing neurons are allocated to solve a single task, thus reducing communication costs. Physics seems
to dictate that any efﬁcient computational hardware will in the future also have to be brain-like in
keeping with these two constraints. The most successful current deep RNNs, however, are not. Un-
like certain spiking NNs (Sec. 5.26), they usually activate all units at least slightly, and tend to be
strongly connected, ignoring natural constraints of 3D hardware. It should be possible to improve
them by adopting (1) and (2), and by minimizing non-differentiable energy and communication costs
through direct search in program (weight) space (e.g., Sec. 6.6, 6.7). These more brain-like RNNs
will allocate neighboring RNN parts to related behaviors, and distant RNN parts to less related ones,
thus self-modularizing in a way more general than that of traditional self-organizing maps in FNNs
(Sec. 5.6.4). They will also implement Occam’s razor (Sec. 4.4, 5.6.3) as a by-product of energy min-
34
imization, by ﬁnding simple (highly generalizing) problem solutions that require few active neurons
and few, mostly short connections.
The more distant future may belong to general purpose learning algorithms that improve them-
selves in provably optimal ways (Sec. 6.8), but these are not yet practical or commercially relevant.
8 Acknowledgments
Since 16 April 2014, drafts of this paper have undergone massive open online peer review through
public mailing lists includingconnectionists@cs.cmu.edu, ml-news@googlegroups.com, comp-neuro-
@neuroinf.org, genetic programming@yahoogroups.com, rl-list@googlegroups.com, imageworld-
@diku.dk, Google+ machine learning forum. Thanks to numerous NN / DL experts for valuable
comments. Thanks to SNF, DFG, and the European Commission for partially funding my DL re-
search group in the past quarter-century. The contents of this paper may be used for educational and
non-commercial purposes, including articles for Wikipedia and similar sites.
References
Aberdeen, D. (2003). Policy-Gradient Algorithms for Partially Observable Markov Decision Pro-
cesses. PhD thesis, Australian National University.
Abounadi, J., Bertsekas, D., and Borkar, V . S. (2002). Learning algorithms for Markov decision
processes with average cost. SIAM Journal on Control and Optimization, 40(3):681–698.
Akaike, H. (1970). Statistical predictor identiﬁcation. Ann. Inst. Statist. Math., 22:203–217.
Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In
Second Intl. Symposium on Information Theory, pages 267–281. Akademinai Kiado.
Akaike, H. (1974). A new look at the statistical model identiﬁcation.IEEE Transactions on Automatic
Control, 19(6):716–723.
Allender, A. (1992). Application of time-bounded Kolmogorov complexity in complexity theory. In
Watanabe, O., editor, Kolmogorov complexity and computational complexity, pages 6–22. EATCS
Monographs on Theoretical Computer Science, Springer.
Almeida, L. B. (1987). A learning rule for asynchronous perceptrons with feedback in a combinatorial
environment. In IEEE 1st International Conference on Neural Networks, San Diego , volume 2,
pages 609–618.
Almeida, L. B., Almeida, L. B., Langlois, T., Amaral, J. D., and Redol, R. A. (1997). On-line step
size adaptation. Technical report, INESC, 9 Rua Alves Redol, 1000.
Amari, S. (1967). A theory of adaptive pattern classiﬁers. IEEE Trans. EC, 16(3):299–307.
Amari, S., Cichocki, A., and Yang, H. (1996). A new learning algorithm for blind signal separation.
In Touretzky, D. S., Mozer, M. C., and Hasselmo, M. E., editors, Advances in Neural Information
Processing Systems (NIPS), volume 8. The MIT Press.
Amari, S. and Murata, N. (1993). Statistical theory of learning curves under entropic loss criterion.
Neural Computation, 5(1):140–153.
35
Amari, S.-I. (1998). Natural gradient works efﬁciently in learning. Neural Computation, 10(2):251–
276.
Amit, D. J. and Brunel, N. (1997). Dynamics of a recurrent network of spiking neurons before and
following learning. Network: Computation in Neural Systems, 8(4):373–404.
An, G. (1996). The effects of adding noise during backpropagation training on a generalization
performance. Neural Computation, 8(3):643–674.
Andrade, M. A., Chacon, P., Merelo, J. J., and Moran, F. (1993). Evaluation of secondary structure
of proteins from UV circular dichroism spectra using an unsupervised learning neural network.
Protein Engineering, 6(4):383–390.
Andrews, R., Diederich, J., and Tickle, A. B. (1995). Survey and critique of techniques for extracting
rules from trained artiﬁcial neural networks. Knowledge-Based Systems, 8(6):373–389.
Anguita, D. and Gomes, B. A. (1996). Mixing ﬂoating- and ﬁxed-point formats for neural network
learning on neuroprocessors. Microprocessing and Microprogramming, 41(10):757 – 769.
Anguita, D., Parodi, G., and Zunino, R. (1994). An efﬁcient implementation of BP on RISC-based
workstations. Neurocomputing, 6(1):57 – 65.
Arel, I., Rose, D. C., and Karnowski, T. P. (2010). Deep machine learning – a new frontier in artiﬁcial
intelligence research. Computational Intelligence Magazine, IEEE, 5(4):13–18.
Ash, T. (1989). Dynamic node creation in backpropagation neural networks. Connection Science,
1(4):365–375.
Atick, J. J., Li, Z., and Redlich, A. N. (1992). Understanding retinal color coding from ﬁrst principles.
Neural Computation, 4:559–572.
Atiya, A. F. and Parlos, A. G. (2000). New results on recurrent network training: unifying the algo-
rithms and accelerating convergence. IEEE Transactions on Neural Networks, 11(3):697–709.
Ba, J. and Frey, B. (2013). Adaptive dropout for training deep neural networks. InAdvances in Neural
Information Processing Systems (NIPS), pages 3084–3092.
Baird, H. (1990). Document image defect models. In Proceddings, IAPR Workshop on Syntactic and
Structural Pattern Recognition, Murray Hill, NJ.
Baird, L. and Moore, A. W. (1999). Gradient descent for general reinforcement learning. InAdvances
in neural information processing systems 12 (NIPS), pages 968–974. MIT Press.
Baird, L. C. (1995). Residual algorithms: Reinforcement learning with function approximation. In
International Conference on Machine Learning, pages 30–37.
Bakker, B. (2002). Reinforcement learning with Long Short-Term Memory. In Dietterich, T. G.,
Becker, S., and Ghahramani, Z., editors, Advances in Neural Information Processing Systems 14 ,
pages 1475–1482. MIT Press, Cambridge, MA.
Bakker, B. and Schmidhuber, J. (2004). Hierarchical reinforcement learning based on subgoal dis-
covery and subpolicy specialization. In et al., F. G., editor, Proc. 8th Conference on Intelligent
Autonomous Systems IAS-8, pages 438–445, Amsterdam, NL. IOS Press.
36
Bakker, B., Zhumatiy, V ., Gruener, G., and Schmidhuber, J. (2003). A robot that reinforcement-learns
to identify and memorize important previous observations. In Proceedings of the 2003 IEEE/RSJ
International Conference on Intelligent Robots and Systems, IROS 2003, pages 430–435.
Baldi, P. (1995). Gradient descent learning algorithms overview: A general dynamical systems per-
spective. IEEE Transactions on Neural Networks, 6(1):182–195.
Baldi, P. (2012). Autoencoders, unsupervised learning, and deep architectures. Journal of Machine
Learning Research (Proc. 2011 ICML Workshop on Unsupervised and Transfer Learning), 27:37–
50.
Baldi, P., Brunak, S., Frasconi, P., Pollastri, G., and Soda, G. (1999). Exploiting the past and the
future in protein secondary structure prediction. Bioinformatics, 15:937–946.
Baldi, P. and Chauvin, Y . (1993). Neural networks for ﬁngerprint recognition. Neural Computation,
5(3):402–418.
Baldi, P. and Chauvin, Y . (1996). Hybrid modeling, HMM/NN architectures, and protein applications.
Neural Computation, 8(7):1541–1565.
Baldi, P. and Hornik, K. (1989). Neural networks and principal component analysis: Learning from
examples without local minima. Neural Networks, 2:53–58.
Baldi, P. and Hornik, K. (1994). Learning in linear networks: a survey. IEEE Transactions on Neural
Networks, 6(4):837–858. 1995.
Baldi, P. and Pollastri, G. (2003). The principled design of large-scale recursive neural network
architectures – DAG-RNNs and the protein structure prediction problem. J. Mach. Learn. Res. ,
4:575–602.
Baldi, P. and Sadowski, P. (2014). The dropout learning algorithm. Artiﬁcial Intelligence, 210C:78–
122.
Ballard, D. H. (1987). Modular learning in neural networks. In Proc. AAAI, pages 279–284.
Baluja, S. (1994). Population-based incremental learning: A method for integrating genetic
search based function optimization and competitive learning. Technical Report CMU-CS-94-163,
Carnegie Mellon University.
Balzer, R. (1985). A 15 year perspective on automatic programming. IEEE Transactions on Software
Engineering, 11(11):1257–1268.
Barlow, H. B. (1989). Unsupervised learning. Neural Computation, 1(3):295–311.
Barlow, H. B., Kaushal, T. P., and Mitchison, G. J. (1989). Finding minimum entropy codes. Neural
Computation, 1(3):412–423.
Barrow, H. G. (1987). Learning receptive ﬁelds. In Proceedings of the IEEE 1st Annual Conference
on Neural Networks, volume IV , pages 115–121. IEEE.
Barto, A. G. and Mahadevan, S. (2003). Recent advances in hierarchical reinforcement learning.
Discrete Event Dynamic Systems, 13(4):341–379.
37
Barto, A. G., Singh, S., and Chentanez, N. (2004). Intrinsically motivated learning of hierarchical col-
lections of skills. In Proceedings of International Conference on Developmental Learning (ICDL),
pages 112–119. MIT Press, Cambridge, MA.
Barto, A. G., Sutton, R. S., and Anderson, C. W. (1983). Neuronlike adaptive elements that can
solve difﬁcult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics ,
SMC-13:834–846.
Battiti, R. (1989). Accelerated backpropagation learning: two optimization methods. Complex Sys-
tems, 3(4):331–342.
Battiti, T. (1992). First- and second-order methods for learning: Between steepest descent and New-
ton’s method.Neural Computation, 4(2):141–166.
Baum, E. B. and Haussler, D. (1989). What size net gives valid generalization? Neural Computation,
1(1):151–160.
Baum, L. E. and Petrie, T. (1966). Statistical inference for probabilistic functions of ﬁnite state
Markov chains. The Annals of Mathematical Statistics, pages 1554–1563.
Baxter, J. and Bartlett, P. L. (2001). Inﬁnite-horizon policy-gradient estimation. J. Artif. Int. Res. ,
15(1):319–350.
Bayer, J. and Osendorfer, C. (2014). Variational inference of latent state sequences using recurrent
networks. arXiv preprint arXiv:1406.1655.
Bayer, J., Osendorfer, C., Chen, N., Urban, S., and van der Smagt, P. (2013). On fast dropout and its
applicability to recurrent networks. arXiv preprint arXiv:1311.0701.
Bayer, J., Wierstra, D., Togelius, J., and Schmidhuber, J. (2009). Evolving memory cell structures for
sequence learning. In Proc. ICANN (2), pages 755–764.
Bayes, T. (1763). An essay toward solving a problem in the doctrine of chances. Philosophical
Transactions of the Royal Society of London , 53:370–418. Communicated by R. Price, in a letter
to J. Canton.
Becker, S. (1991). Unsupervised learning procedures for neural networks. International Journal of
Neural Systems, 2(1 & 2):17–33.
Becker, S. and Le Cun, Y . (1989). Improving the convergence of back-propagation learning with
second order methods. In Touretzky, D., Hinton, G., and Sejnowski, T., editors, Proc. 1988 Con-
nectionist Models Summer School, pages 29–37, Pittsburg 1988. Morgan Kaufmann, San Mateo.
Behnke, S. (1999). Hebbian learning and competition in the neural abstraction pyramid. In Pro-
ceedings of the International Joint Conference on Neural Networks (IJCNN) , volume 2, pages
1356–1361.
Behnke, S. (2001). Learning iterative image reconstruction in the neural abstraction pyramid. Inter-
national Journal of Computational Intelligence and Applications, 1(4):427–438.
Behnke, S. (2002). Learning face localization using hierarchical recurrent networks. In Proceedings
of the 12th International Conference on Artiﬁcial Neural Networks (ICANN), Madrid, Spain, pages
1319–1324.
38
Behnke, S. (2003a). Discovering hierarchical speech features using convolutional non-negative matrix
factorization. In Proceedings of the International Joint Conference on Neural Networks (IJCNN) ,
volume 4, pages 2758–2763.
Behnke, S. (2003b). Hierarchical Neural Networks for Image Interpretation, volume LNCS 2766 of
Lecture Notes in Computer Science. Springer.
Behnke, S. (2005). Face localization and tracking in the Neural Abstraction Pyramid. Neural Com-
puting and Applications, 14(2):97–103.
Behnke, S. and Rojas, R. (1998). Neural abstraction pyramid: A hierarchical image understand-
ing architecture. In Proceedings of International Joint Conference on Neural Networks (IJCNN) ,
volume 2, pages 820–825.
Bell, A. J. and Sejnowski, T. J. (1995). An information-maximization approach to blind separation
and blind deconvolution. Neural Computation, 7(6):1129–1159.
Bellman, R. (1957). Dynamic Programming. Princeton University Press, Princeton, NJ, USA, 1st
edition.
Belouchrani, A., Abed-Meraim, K., Cardoso, J.-F., and Moulines, E. (1997). A blind source separation
technique using second-order statistics. IEEE Transactions on Signal Processing, 45(2):434–444.
Bengio, Y . (1991). Artiﬁcial Neural Networks and their Application to Sequence Recognition . PhD
thesis, McGill University, (Computer Science), Montreal, Qc., Canada.
Bengio, Y . (2009).Learning Deep Architectures for AI. Foundations and Trends in Machine Learning,
V2(1). Now Publishers.
Bengio, Y ., Courville, A., and Vincent, P. (2013). Representation learning: A review and new per-
spectives. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 35(8):1798–1828.
Bengio, Y ., Lamblin, P., Popovici, D., and Larochelle, H. (2007). Greedy layer-wise training of deep
networks. In Cowan, J. D., Tesauro, G., and Alspector, J., editors, Advances in Neural Information
Processing Systems 19 (NIPS), pages 153–160. MIT Press.
Bengio, Y ., Simard, P., and Frasconi, P. (1994). Learning long-term dependencies with gradient
descent is difﬁcult. IEEE Transactions on Neural Networks, 5(2):157–166.
Beringer, N., Graves, A., Schiel, F., and Schmidhuber, J. (2005). Classifying unprompted speech by
retraining LSTM nets. In Duch, W., Kacprzyk, J., Oja, E., and Zadrozny, S., editors, Artiﬁcial
Neural Networks: Biological Inspirations - ICANN 2005, LNCS 3696 , pages 575–581. Springer-
Verlag Berlin Heidelberg.
Bertsekas, D. P. (2001). Dynamic Programming and Optimal Control. Athena Scientiﬁc.
Bertsekas, D. P. and Tsitsiklis, J. N. (1996). Neuro-dynamic Programming. Athena Scientiﬁc, Bel-
mont, MA.
Bichot, N. P., Rossi, A. F., and Desimone, R. (2005). Parallel and serial neural mechanisms for visual
search in macaque area V4. Science, 308:529–534.
Biegler-K¨onig, F. and B ¨armann, F. (1993). A learning algorithm for multilayered neural networks
based on linear least squares problems. Neural Networks, 6(1):127–131.
39
Bishop, C. M. (1993). Curvature-driven smoothing: A learning algorithm for feed-forward networks.
IEEE Transactions on Neural Networks, 4(5):882–884.
Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
Blair, A. D. and Pollack, J. B. (1997). Analysis of dynamical recognizers. Neural Computation,
9(5):1127–1142.
Blondel, V . D. and Tsitsiklis, J. N. (2000). A survey of computational complexity results in systems
and control. Automatica, 36(9):1249–1274.
Bluche, T., Louradour, J., Knibbe, M., Moysset, B., Benzeghiba, F., and Kermorvant, C. (2014).
The A2iA Arabic Handwritten Text Recognition System at the OpenHaRT2013 Evaluation. In
International Workshop on Document Analysis Systems.
Blum, A. L. and Rivest, R. L. (1992). Training a 3-node neural network is np-complete. Neural
Networks, 5(1):117–127.
Blumer, A., Ehrenfeucht, A., Haussler, D., and Warmuth, M. K. (1987). Occam’s razor. Information
Processing Letters, 24:377–380.
Bobrowski, L. (1978). Learning processes in multilayer threshold nets. Biological Cybernetics, 31:1–
6.
Bod´en, M. and Wiles, J. (2000). Context-free and context-sensitive dynamics in recurrent neural
networks. Connection Science, 12(3-4):197–210.
Bodenhausen, U. and Waibel, A. (1991). The Tempo 2 algorithm: Adjusting time-delays by super-
vised learning. In Lippman, D. S., Moody, J. E., and Touretzky, D. S., editors, Advances in Neural
Information Processing Systems 3, pages 155–161. Morgan Kaufmann.
Bohte, S. M., Kok, J. N., and La Poutre, H. (2002). Error-backpropagation in temporally encoded
networks of spiking neurons. Neurocomputing, 48(1):17–37.
Boltzmann, L. (1909). In Hasen ¨ohrl, F., editor, Wissenschaftliche Abhandlungen (collection of Boltz-
mann’s articles in scientiﬁc journals). Barth, Leipzig.
Bottou, L. (1991). Une approche th ´eorique de l’apprentissage connexioniste; applications `a la re-
connaissance de la parole. PhD thesis, Universit ´e de Paris XI.
Bourlard, H. and Morgan, N. (1994). Connnectionist Speech Recognition: A Hybrid Approach .
Kluwer Academic Publishers.
Boutilier, C. and Poole, D. (1996). Computing optimal policies for partially observable Markov
decision processes using compact representations. In Proceedings of the AAAI, Portland, OR.
Bradtke, S. J., Barto, A. G., and Kaelbling, L. P. (1996). Linear least-squares algorithms for temporal
difference learning. In Machine Learning, pages 22–33.
Brafman, R. I. and Tennenholtz, M. (2002). R-MAX—a general polynomial time algorithm for near-
optimal reinforcement learning. Journal of Machine Learning Research, 3:213–231.
Brea, J., Senn, W., and Pﬁster, J.-P. (2013). Matching recall and storage in sequence learning with
spiking neural networks. The Journal of Neuroscience, 33(23):9565–9575.
40
Breiman, L. (1996). Bagging predictors. Machine Learning, 24:123–140.
Brette, R., Rudolph, M., Carnevale, T., Hines, M., Beeman, D., Bower, J. M., Diesmann, M., Morri-
son, A., Goodman, P. H., Harris Jr, F. C., et al. (2007). Simulation of networks of spiking neurons:
a review of tools and strategies. Journal of Computational Neuroscience, 23(3):349–398.
Breuel, T. M., Ul-Hasan, A., Al-Azawi, M. A., and Shafait, F. (2013). High-performance OCR for
printed English and Fraktur using LSTM networks. In12th International Conference on Document
Analysis and Recognition (ICDAR), pages 683–687. IEEE.
Bromley, J., Bentz, J. W., Bottou, L., Guyon, I., LeCun, Y ., Moore, C., Sackinger, E., and Shah, R.
(1993). Signature veriﬁcation using a Siamese time delay neural network. International Journal of
Pattern Recognition and Artiﬁcial Intelligence, 7(4):669–688.
Broyden, C. G. et al. (1965). A class of methods for solving nonlinear simultaneous equations. Math.
Comp, 19(92):577–593.
Brueckner, R. and Schulter, B. (2014). Social signal classiﬁcation using deep BLSTM recurrent neural
networks. In Proceedings 39th IEEE International Conference on Acoustics, Speech, and Signal
Processing, ICASSP 2014, Florence, Italy, pages 4856–4860.
Brunel, N. (2000). Dynamics of sparsely connected networks of excitatory and inhibitory spiking
neurons. Journal of Computational Neuroscience, 8(3):183–208.
Bryson, A. and Ho, Y . (1969).Applied optimal control: optimization, estimation, and control. Blais-
dell Pub. Co.
Bryson, A. E. (1961). A gradient method for optimizing multi-stage allocation processes. In Proc.
Harvard Univ. Symposium on digital computers and their applications.
Bryson, Jr., A. E. and Denham, W. F. (1961). A steepest-ascent method for solving optimum pro-
gramming problems. Technical Report BR-1303, Raytheon Company, Missle and Space Division.
Buhler, J. (2001). Efﬁcient large-scale sequence comparison by locality-sensitive hashing. Bioinfor-
matics, 17(5):419–428.
Buntine, W. L. and Weigend, A. S. (1991). Bayesian back-propagation.Complex Systems, 5:603–643.
Burgess, N. (1994). A constructive algorithm that converges for real-valued input patterns. Interna-
tional Journal of Neural Systems, 5(1):59–66.
Cardoso, J.-F. (1994). On the performance of orthogonal source separation algorithms. In Proc.
EUSIPCO, pages 776–779.
Carreira-Perpinan, M. A. (2001). Continuous latent variable models for dimensionality reduction and
sequential data reconstruction. PhD thesis, University of Shefﬁeld UK.
Carter, M. J., Rudolph, F. J., and Nucci, A. J. (1990). Operational fault tolerance of CMAC networks.
In Touretzky, D. S., editor, Advances in Neural Information Processing Systems (NIPS) 2 , pages
340–347. San Mateo, CA: Morgan Kaufmann.
Caruana, R. (1997). Multitask learning. Machine Learning, 28(1):41–75.
Casey, M. P. (1996). The dynamics of discrete-time computation, with application to recurrent neural
networks and ﬁnite state machine extraction. Neural Computation, 8(6):1135–1178.
41
Cauwenberghs, G. (1993). A fast stochastic error-descent algorithm for supervised learning and op-
timization. In Lippman, D. S., Moody, J. E., and Touretzky, D. S., editors, Advances in Neural
Information Processing Systems 5, pages 244–244. Morgan Kaufmann.
Chaitin, G. J. (1966). On the length of programs for computing ﬁnite binary sequences. Journal of
the ACM, 13:547–569.
Chalup, S. K. and Blair, A. D. (2003). Incremental training of ﬁrst order recurrent neural networks to
predict a context-sensitive language. Neural Networks, 16(7):955–972.
Chellapilla, K., Puri, S., and Simard, P. (2006). High performance convolutional neural networks for
document processing. In International Workshop on Frontiers in Handwriting Recognition.
Chen, K. and Salman, A. (2011). Learning speaker-speciﬁc characteristics with a deep neural archi-
tecture. IEEE Transactions on Neural Networks, 22(11):1744–1756.
Cho, K. (2014). Foundations and Advances in Deep Learning. PhD thesis, Aalto University School
of Science.
Cho, K., Ilin, A., and Raiko, T. (2012). Tikhonov-type regularization for restricted Boltzmann ma-
chines. In Intl. Conf. on Artiﬁcial Neural Networks (ICANN) 2012, pages 81–88. Springer.
Cho, K., Raiko, T., and Ilin, A. (2013). Enhanced gradient for training restricted Boltzmann machines.
Neural Computation, 25(3):805–831.
Church, A. (1936). An unsolvable problem of elementary number theory. American Journal of
Mathematics, 58:345–363.
Ciresan, D. C., Giusti, A., Gambardella, L. M., and Schmidhuber, J. (2012a). Deep neural networks
segment neuronal membranes in electron microscopy images. In Advances in Neural Information
Processing Systems (NIPS), pages 2852–2860.
Ciresan, D. C., Giusti, A., Gambardella, L. M., and Schmidhuber, J. (2013). Mitosis detection in
breast cancer histology images with deep neural networks. In Proc. MICCAI, volume 2, pages
411–418.
Ciresan, D. C., Meier, U., Gambardella, L. M., and Schmidhuber, J. (2010). Deep big simple neural
nets for handwritten digit recogntion. Neural Computation, 22(12):3207–3220.
Ciresan, D. C., Meier, U., Masci, J., Gambardella, L. M., and Schmidhuber, J. (2011a). Flexible, high
performance convolutional neural networks for image classiﬁcation. In Intl. Joint Conference on
Artiﬁcial Intelligence IJCAI, pages 1237–1242.
Ciresan, D. C., Meier, U., Masci, J., and Schmidhuber, J. (2011b). A committee of neural networks
for trafﬁc sign classiﬁcation. In International Joint Conference on Neural Networks (IJCNN), pages
1918–1921.
Ciresan, D. C., Meier, U., Masci, J., and Schmidhuber, J. (2012b). Multi-column deep neural network
for trafﬁc sign classiﬁcation. Neural Networks, 32:333–338.
Ciresan, D. C., Meier, U., and Schmidhuber, J. (2012c). Multi-column deep neural networks for image
classiﬁcation. In IEEE Conference on Computer Vision and Pattern Recognition CVPR 2012. Long
preprint arXiv:1202.2745v1 [cs.CV].
42
Ciresan, D. C., Meier, U., and Schmidhuber, J. (2012d). Transfer learning for Latin and Chinese char-
acters with deep neural networks. In International Joint Conference on Neural Networks (IJCNN),
pages 1301–1306.
Ciresan, D. C. and Schmidhuber, J. (2013). Multi-column deep neural networks for ofﬂine handwrit-
ten Chinese character classiﬁcation. Technical report, IDSIA. arXiv:1309.0261.
Cliff, D. T., Husbands, P., and Harvey, I. (1993). Evolving recurrent dynamical networks for robot
control. In Artiﬁcial Neural Nets and Genetic Algorithms, pages 428–435. Springer.
Clune, J., Mouret, J.-B., and Lipson, H. (2013). The evolutionary origins of modularity. Proceedings
of the Royal Society B: Biological Sciences, 280(1755):20122863.
Clune, J., Stanley, K. O., Pennock, R. T., and Ofria, C. (2011). On the performance of indirect
encoding across the continuum of regularity. Trans. Evol. Comp, 15(3):346–367.
Coates, A., Huval, B., Wang, T., Wu, D. J., Ng, A. Y ., and Catanzaro, B. (2013). Deep learning with
COTS HPC systems. In Proc. International Conference on Machine learning (ICML’13).
Cochocki, A. and Unbehauen, R. (1993). Neural networks for optimization and signal processing .
John Wiley & Sons, Inc.
Collobert, R. and Weston, J. (2008). A uniﬁed architecture for natural language processing: Deep
neural networks with multitask learning. In Proceedings of the 25th International Conference on
Machine Learning (ICML), pages 160–167. ACM.
Comon, P. (1994). Independent component analysis – a new concept? Signal Processing, 36(3):287–
314.
Connor, C. E., Brincat, S. L., and Pasupathy, A. (2007). Transformation of shape information in the
ventral pathway. Current Opinion in Neurobiology, 17(2):140–147.
Connor, J., Martin, D. R., and Atlas, L. E. (1994). Recurrent neural networks and robust time series
prediction. IEEE Transactions on Neural Networks, 5(2):240–254.
Cook, S. A. (1971). The complexity of theorem-proving procedures. InProceedings of the 3rd Annual
ACM Symposium on the Theory of Computing (STOC’71), pages 151–158. ACM, New York.
Cramer, N. L. (1985). A representation for the adaptive generation of simple sequential programs.
In Grefenstette, J., editor, Proceedings of an International Conference on Genetic Algorithms and
Their Applications, Carnegie-Mellon University, July 24-26, 1985 , Hillsdale NJ. Lawrence Erl-
baum Associates.
Craven, P. and Wahba, G. (1979). Smoothing noisy data with spline functions: Estimating the correct
degree of smoothing by the method of generalized cross-validation. Numer. Math., 31:377–403.
Cuccu, G., Luciw, M., Schmidhuber, J., and Gomez, F. (2011). Intrinsically motivated evolutionary
search for vision-based reinforcement learning. In Proceedings of the 2011 IEEE Conference on
Development and Learning and Epigenetic Robotics IEEE-ICDL-EPIROB , volume 2, pages 1–7.
IEEE.
Dahl, G., Yu, D., Deng, L., and Acero, A. (2012). Context-dependent pre-trained deep neural net-
works for large-vocabulary speech recognition. Audio, Speech, and Language Processing, IEEE
Transactions on, 20(1):30–42.
43
Dahl, G. E., Sainath, T. N., and Hinton, G. E. (2013). Improving deep neural networks for LVCSR
using rectiﬁed linear units and dropout. In IEEE International Conference on Acoustics, Speech
and Signal Processing (ICASSP), pages 8609–8613. IEEE.
D’Ambrosio, D. B. and Stanley, K. O. (2007). A novel generative encoding for exploiting neural net-
work sensor and output geometry. In Proceedings of the Conference on Genetic and Evolutionary
Computation (GECCO), pages 974–981.
Datar, M., Immorlica, N., Indyk, P., and Mirrokni, V . S. (2004). Locality-sensitive hashing scheme
based on p-stable distributions. In Proceedings of the 20th Annual Symposium on Computational
Geometry, pages 253–262. ACM.
Dayan, P. and Hinton, G. (1993). Feudal reinforcement learning. In Lippman, D. S., Moody, J. E.,
and Touretzky, D. S., editors,Advances in Neural Information Processing Systems (NIPS) 5, pages
271–278. Morgan Kaufmann.
Dayan, P. and Hinton, G. E. (1996). Varieties of Helmholtz machine. Neural Networks, 9(8):1385–
1403.
Dayan, P., Hinton, G. E., Neal, R. M., and Zemel, R. S. (1995). The Helmholtz machine. Neural
Computation, 7:889–904.
Dayan, P. and Zemel, R. (1995). Competition and multiple cause models. Neural Computation,
7:565–579.
De Freitas, J. F. G. (2003). Bayesian methods for neural networks . PhD thesis, University of Cam-
bridge.
de Souto, M. C., Souto, M. C. P. D., and Oliveira, W. R. D. (1999). The loading problem for pyramidal
neural networks. In Electronic Journal on Mathematics of Computation.
De Valois, R. L., Albrecht, D. G., and Thorell, L. G. (1982). Spatial frequency selectivity of cells in
macaque visual cortex. Vision Research, 22(5):545–559.
de Vries, B. and Principe, J. C. (1991). A theory for neural networks with time delays. In Lippmann,
R. P., Moody, J. E., and Touretzky, D. S., editors, Advances in Neural Information Processing
Systems (NIPS) 3, pages 162–168. Morgan Kaufmann.
Deco, G. and Parra, L. (1997). Non-linear feature extraction by redundancy reduction in an unsuper-
vised stochastic neural network. Neural Networks, 10(4):683–691.
Deco, G. and Rolls, E. T. (2005). Neurodynamics of biased competition and cooperation for attention:
a model with spiking neurons. Journal of Neurophysiology, 94(1):295–313.
DeJong, G. and Mooney, R. (1986). Explanation-based learning: An alternative view. Machine
Learning, 1(2):145–176.
DeMers, D. and Cottrell, G. (1993). Non-linear dimensionality reduction. In Hanson, S. J., Cowan,
J. D., and Giles, C. L., editors,Advances in Neural Information Processing Systems (NIPS) 5, pages
580–587. Morgan Kaufmann.
Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977). Maximum likelihood from incomplete data
via the EM algorithm. Journal of the Royal Statistical Society, B, 39.
44
Deng, L. and Yu, D. (2014). Deep Learning: Methods and Applications. NOW Publishers.
Desimone, R., Albright, T. D., Gross, C. G., and Bruce, C. (1984). Stimulus-selective properties of
inferior temporal neurons in the macaque. The Journal of Neuroscience, 4(8):2051–2062.
Deville, Y . and Lau, K. K. (1994). Logic program synthesis. Journal of Logic Programming ,
19(20):321–350.
Di Lena, P., Nagata, K., and Baldi, P. (2012). Deep architectures for protein contact map prediction.
Bioinformatics, 28:2449–2457.
DiCarlo, J. J., Zoccolan, D., and Rust, N. C. (2012). How does the brain solve visual object recogni-
tion? Neuron, 73(3):415–434.
Dickmanns, D., Schmidhuber, J., and Winklhofer, A. (1987). Der genetische Algorithmus:
Eine Implementierung in Prolog. Technical Report, Inst. of Informatics, Tech. Univ. Munich.
http://www.idsia.ch/˜juergen/geneticprogramming.html.
Dickmanns, E. D., Behringer, R., Dickmanns, D., Hildebrandt, T., Maurer, M., Thomanek, F., and
Schiehlen, J. (1994). The seeing passenger car ’VaMoRs-P’. In Proc. Int. Symp. on Intelligent
Vehicles ’94, Paris, pages 68–73.
Dietterich, T. G. (2000a). Ensemble methods in machine learning. In Multiple classiﬁer systems ,
pages 1–15. Springer.
Dietterich, T. G. (2000b). Hierarchical reinforcement learning with the MAXQ value function de-
composition. J. Artif. Intell. Res. (JAIR), 13:227–303.
Director, S. W. and Rohrer, R. A. (1969). Automated network design - the frequency-domain case.
IEEE Trans. Circuit Theory, CT-16:330–337.
Dittenbach, M., Merkl, D., and Rauber, A. (2000). The growing hierarchical self-organizing map.
In IEEE-INNS-ENNS International Joint Conference on Neural Networks, volume 6, pages 6015–
6015. IEEE Computer Society.
Donahue, J., Jia, Y ., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T. (2013). De-
CAF: A deep convolutional activation feature for generic visual recognition. arXiv preprint
arXiv:1310.1531.
Dorffner, G. (1996). Neural networks for time series processing. In Neural Network World.
Doya, K., Samejima, K., ichi Katagiri, K., and Kawato, M. (2002). Multiple model-based reinforce-
ment learning. Neural Computation, 14(6):1347–1369.
Dreyfus, S. E. (1962). The numerical solution of variational problems. Journal of Mathematical
Analysis and Applications, 5(1):30–45.
Dreyfus, S. E. (1973). The computational solution of optimal control problems with time lag. IEEE
Transactions on Automatic Control, 18(4):383–385.
Duchi, J., Hazan, E., and Singer, Y . (2011). Adaptive subgradient methods for online learning and
stochastic optimization. The Journal of Machine Learning, 12:2121–2159.
45
Egorova, A., Gloye, A., G ¨oktekin, C., Liers, A., Luft, M., Rojas, R., Simon, M., Tenchio, O., and
Wiesel, F. (2004). FU-Fighters Small Size 2004, Team Description. RoboCup 2004 Symposium:
Papers and Team Description Papers. CD edition.
Elfwing, S., Otsuka, M., Uchibe, E., and Doya, K. (2010). Free-energy based reinforcement learning
for vision-based navigation with high-dimensional sensory inputs. In Neural Information Process-
ing. Theory and Algorithms (ICONIP), volume 1, pages 215–222. Springer.
Eliasmith, C. (2013). How to build a brain: A neural architecture for biological cognition . Oxford
University Press, New York, NY .
Eliasmith, C., Stewart, T. C., Choo, X., Bekolay, T., DeWolf, T., Tang, Y ., and Rasmussen, D. (2012).
A large-scale model of the functioning brain. Science, 338(6111):1202–1205.
Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14(2):179–211.
Erhan, D., Bengio, Y ., Courville, A., Manzagol, P.-A., Vincent, P., and Bengio, S. (2010). Why does
unsupervised pre-training help deep learning? J. Mach. Learn. Res., 11:625–660.
Escalante-B., A. N. and Wiskott, L. (2013). How to solve classiﬁcation and regression problems on
high-dimensional data with a supervised extension of slow feature analysis. Journal of Machine
Learning Research, 14:3683–3719.
Eubank, R. L. (1988). Spline smoothing and nonparametric regression. In Farlow, S., editor, Self-
Organizing Methods in Modeling. Marcel Dekker, New York.
Euler, L. (1744). Methodus inveniendi.
Eyben, F., Weninger, F., Squartini, S., and Schuller, B. (2013). Real-life voice activity detection with
LSTM recurrent neural networks and an application to Hollywood movies. In Proc. 38th IEEE
International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2013, Vancouver,
Canada, pages 483–487.
Faggin, F. (1992). Neural network hardware. In International Joint Conference on Neural Networks
(IJCNN), volume 1, page 153.
Fahlman, S. E. (1988). An empirical study of learning speed in back-propagation networks. Technical
Report CMU-CS-88-162, Carnegie-Mellon Univ.
Fahlman, S. E. (1991). The recurrent cascade-correlation learning algorithm. In Lippmann, R. P.,
Moody, J. E., and Touretzky, D. S., editors, Advances in Neural Information Processing Systems
(NIPS) 3, pages 190–196. Morgan Kaufmann.
Falconbridge, M. S., Stamps, R. L., and Badcock, D. R. (2006). A simple Hebbian/anti-Hebbian
network learns the sparse, independent components of natural images. Neural Computation ,
18(2):415–429.
Fan, Y ., Qian, Y ., Xie, F., and Soong, F. K. (2014). TTS synthesis with bidirectional LSTM based
recurrent neural networks. In Proc. Interspeech.
Farabet, C., Couprie, C., Najman, L., and LeCun, Y . (2013). Learning hierarchical features for scene
labeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1915–1929.
Farlow, S. J. (1984). Self-organizing methods in modeling: GMDH type algorithms, volume 54. CRC
Press.
46
Feldkamp, L. A., Prokhorov, D. V ., Eagen, C. F., and Yuan, F. (1998). Enhanced multi-stream Kalman
ﬁlter training for recurrent networks. In Nonlinear Modeling, pages 29–53. Springer.
Feldkamp, L. A., Prokhorov, D. V ., and Feldkamp, T. M. (2003). Simple and conditioned adaptive
behavior from Kalman ﬁlter trained recurrent networks. Neural Networks, 16(5):683–689.
Feldkamp, L. A. and Puskorius, G. V . (1998). A signal processing framework based on dynamic neural
networks with application to problems in adaptation, ﬁltering, and classiﬁcation. Proceedings of
the IEEE, 86(11):2259–2277.
Felleman, D. J. and Van Essen, D. C. (1991). Distributed hierarchical processing in the primate
cerebral cortex. Cerebral Cortex, 1(1):1–47.
Fernandez, R., Rendel, A., Ramabhadran, B., and Hoory, R. (2014). Prosody contour prediction with
Long Short-Term Memory, bi-directional, deep recurrent neural networks. In Proc. Interspeech.
Fern´andez, S., Graves, A., and Schmidhuber, J. (2007). An application of recurrent neural networks
to discriminative keyword spotting. In Proc. ICANN (2), pages 220–229.
Fernandez, S., Graves, A., and Schmidhuber, J. (2007). Sequence labelling in structured domains with
hierarchical recurrent neural networks. In Proceedings of the 20th International Joint Conference
on Artiﬁcial Intelligence (IJCAI).
Field, D. J. (1987). Relations between the statistics of natural images and the response properties of
cortical cells. Journal of the Optical Society of America, 4:2379–2394.
Field, D. J. (1994). What is the goal of sensory coding? Neural Computation, 6:559–601.
Fieres, J., Schemmel, J., and Meier, K. (2008). Realizing biological spiking network models in
a conﬁgurable wafer-scale hardware system. In IEEE International Joint Conference on Neural
Networks, pages 969–976.
Fine, S., Singer, Y ., and Tishby, N. (1998). The hierarchical hidden Markov model: Analysis and
applications. Machine Learning, 32(1):41–62.
Fischer, A. and Igel, C. (2014). Training restricted Boltzmann machines: An introduction. Pattern
Recognition, 47:25–39.
FitzHugh, R. (1961). Impulses and physiological states in theoretical models of nerve membrane.
Biophysical Journal, 1(6):445–466.
Fletcher, R. and Powell, M. J. (1963). A rapidly convergent descent method for minimization. The
Computer Journal, 6(2):163–168.
Floreano, D. and Mattiussi, C. (2001). Evolution of spiking neural controllers for autonomous vision-
based robots. In Evolutionary Robotics. From Intelligent Robotics to Artiﬁcial Life , pages 38–61.
Springer.
Fogel, D. B., Fogel, L. J., and Porto, V . (1990). Evolving neural networks. Biological Cybernetics,
63(6):487–493.
Fogel, L., Owens, A., and Walsh, M. (1966). Artiﬁcial Intelligence through Simulated Evolution .
Wiley, New York.
47
F¨oldi´ak, P. (1990). Forming sparse representations by local anti-Hebbian learning. Biological Cyber-
netics, 64:165–170.
F¨oldi´ak, P. and Young, M. P. (1995). Sparse coding in the primate cortex. In Arbib, M. A., editor,The
Handbook of Brain Theory and Neural Networks, pages 895–898. The MIT Press.
F¨orster, A., Graves, A., and Schmidhuber, J. (2007). RNN-based Learning of Compact Maps for
Efﬁcient Robot Localization. In 15th European Symposium on Artiﬁcial Neural Networks, ESANN,
pages 537–542, Bruges, Belgium.
Franzius, M., Sprekeler, H., and Wiskott, L. (2007). Slowness and sparseness lead to place, head-
direction, and spatial-view cells. PLoS Computational Biology, 3(8):166.
Friedman, J., Hastie, T., and Tibshirani, R. (2001). The elements of statistical learning , volume 1.
Springer Series in Statistics, New York.
Frinken, V ., Zamora-Martinez, F., Espana-Boquera, S., Castro-Bleda, M. J., Fischer, A., and Bunke,
H. (2012). Long-short term memory neural networks language modeling for handwriting recog-
nition. In Pattern Recognition (ICPR), 2012 21st International Conference on , pages 701–704.
IEEE.
Fritzke, B. (1994). A growing neural gas network learns topologies. In Tesauro, G., Touretzky, D. S.,
and Leen, T. K., editors, NIPS, pages 625–632. MIT Press.
Fu, K. S. (1977). Syntactic Pattern Recognition and Applications. Berlin, Springer.
Fukada, T., Schuster, M., and Sagisaka, Y . (1999). Phoneme boundary estimation using bidirectional
recurrent neural networks and its applications. Systems and Computers in Japan, 30(4):20–30.
Fukushima, K. (1979). Neural network model for a mechanism of pattern recognition unaffected by
shift in position - Neocognitron. Trans. IECE, J62-A(10):658–665.
Fukushima, K. (1980). Neocognitron: A self-organizing neural network for a mechanism of pattern
recognition unaffected by shift in position. Biological Cybernetics, 36(4):193–202.
Fukushima, K. (2011). Increasing robustness against background noise: visual pattern recognition by
a Neocognitron. Neural Networks, 24(7):767–778.
Fukushima, K. (2013a). Artiﬁcial vision by multi-layered neural networks: Neocognitron and its
advances. Neural Networks, 37:103–119.
Fukushima, K. (2013b). Training multi-layered neural network Neocognitron. Neural Networks,
40:18–31.
Gabor, D. (1946). Theory of communication. Part 1: The analysis of information. Electri-
cal Engineers-Part III: Journal of the Institution of Radio and Communication Engineering ,
93(26):429–441.
Gallant, S. I. (1988). Connectionist expert systems. Communications of the ACM, 31(2):152–169.
Gauss, C. F. (1809). Theoria motus corporum coelestium in sectionibus conicis solem ambientium.
Gauss, C. F. (1821). Theoria combinationis observationum erroribus minimis obnoxiae (Theory of
the combination of observations least subject to error).
48
Ge, S., Hang, C. C., Lee, T. H., and Zhang, T. (2010). Stable adaptive neural network control .
Springer.
Geiger, J. T., Zhang, Z., Weninger, F., Schuller, B., and Rigoll, G. (2014). Robust speech recognition
using long short-term memory recurrent neural networks for hybrid acoustic modelling. In Proc.
Interspeech.
Geman, S., Bienenstock, E., and Doursat, R. (1992). Neural networks and the bias/variance dilemma.
Neural Computation, 4:1–58.
Gers, F. A. and Schmidhuber, J. (2000). Recurrent nets that time and count. InNeural Networks, 2000.
IJCNN 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on , volume 3,
pages 189–194. IEEE.
Gers, F. A. and Schmidhuber, J. (2001). LSTM recurrent networks learn simple context free and
context sensitive languages. IEEE Transactions on Neural Networks, 12(6):1333–1340.
Gers, F. A., Schmidhuber, J., and Cummins, F. (2000). Learning to forget: Continual prediction with
LSTM. Neural Computation, 12(10):2451–2471.
Gers, F. A., Schraudolph, N., and Schmidhuber, J. (2002). Learning precise timing with LSTM
recurrent networks. Journal of Machine Learning Research, 3:115–143.
Gerstner, W. and Kistler, W. K. (2002).Spiking Neuron Models. Cambridge University Press.
Gerstner, W. and van Hemmen, J. L. (1992). Associative memory in a network of spiking neurons.
Network: Computation in Neural Systems, 3(2):139–164.
Ghavamzadeh, M. and Mahadevan, S. (2003). Hierarchical policy gradient algorithms. InProceedings
of the Twentieth Conference on Machine Learning (ICML-2003), pages 226–233.
Gherrity, M. (1989). A learning algorithm for analog fully recurrent neural networks. In IEEE/INNS
International Joint Conference on Neural Networks, San Diego, volume 1, pages 643–644.
Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2013). Rich feature hierarchies for accurate object
detection and semantic segmentation. Technical Report arxiv.org/abs/1311.2524, UC Berkeley and
ICSI.
Gisslen, L., Luciw, M., Graziano, V ., and Schmidhuber, J. (2011). Sequential constant size compressor
for reinforcement learning. In Proc. Fourth Conference on Artiﬁcial General Intelligence (AGI),
Google, Mountain View, CA, pages 31–40. Springer.
Giusti, A., Ciresan, D. C., Masci, J., Gambardella, L. M., and Schmidhuber, J. (2013). Fast image
scanning with deep max-pooling convolutional neural networks. In Proc. ICIP.
Glackin, B., McGinnity, T. M., Maguire, L. P., Wu, Q., and Belatreche, A. (2005). A novel approach
for the implementation of large scale spiking neural networks on FPGA hardware. In Computa-
tional Intelligence and Bioinspired Systems, pages 552–563. Springer.
Glasmachers, T., Schaul, T., Sun, Y ., Wierstra, D., and Schmidhuber, J. (2010). Exponential natural
evolution strategies. In Proceedings of the Genetic and Evolutionary Computation Conference
(GECCO), pages 393–400. ACM.
Glorot, X., Bordes, A., and Bengio, Y . (2011). Deep sparse rectiﬁer networks. InAISTATS, volume 15,
pages 315–323.
49
Gloye, A., Wiesel, F., Tenchio, O., and Simon, M. (2005). Reinforcing the driving quality of soccer
playing robots by anticipation. IT - Information Technology, 47(5).
G¨odel, K. (1931). ¨Uber formal unentscheidbare S ¨atze der Principia Mathematica und verwandter
Systeme I. Monatshefte f¨ur Mathematik und Physik, 38:173–198.
Goldberg, D. E. (1989).Genetic Algorithms in Search, Optimization and Machine Learning. Addison-
Wesley, Reading, MA.
Goldfarb, D. (1970). A family of variable-metric methods derived by variational means.Mathematics
of computation, 24(109):23–26.
Golub, G., Heath, H., and Wahba, G. (1979). Generalized cross-validation as a method for choosing
a good ridge parameter. Technometrics, 21:215–224.
Gomez, F. J. (2003). Robust Nonlinear Control through Neuroevolution. PhD thesis, Department of
Computer Sciences, University of Texas at Austin.
Gomez, F. J. and Miikkulainen, R. (2003). Active guidance for a ﬁnless rocket using neuroevolution.
In Proc. GECCO 2003, Chicago.
Gomez, F. J. and Schmidhuber, J. (2005). Co-evolving recurrent neurons learn deep memory
POMDPs. In Proc. of the 2005 conference on genetic and evolutionary computation (GECCO),
Washington, D. C. ACM Press, New York, NY , USA.
Gomez, F. J., Schmidhuber, J., and Miikkulainen, R. (2008). Accelerated neural evolution through
cooperatively coevolved synapses. Journal of Machine Learning Research, 9(May):937–965.
Gomi, H. and Kawato, M. (1993). Neural network control for a closed-loop system using feedback-
error-learning. Neural Networks, 6(7):933–946.
Gonzalez-Dominguez, J., Lopez-Moreno, I., Sak, H., Gonzalez-Rodriguez, J., and Moreno, P. J.
(2014). Automatic language identiﬁcation using Long Short-Term Memory recurrent neural net-
works. In Proc. Interspeech.
Goodfellow, I., Mirza, M., Da, X., Courville, A., and Bengio, Y . (2014a). An Empirical Investigation
of Catastrophic Forgetting in Gradient-Based Neural Networks. TR arXiv:1312.6211v2.
Goodfellow, I. J., Bulatov, Y ., Ibarz, J., Arnoud, S., and Shet, V . (2014b). Multi-digit number
recognition from street view imagery using deep convolutional neural networks. arXiv preprint
arXiv:1312.6082 v4.
Goodfellow, I. J., Courville, A., and Bengio, Y . (2011). Spike-and-slab sparse coding for unsupervised
feature discovery. In NIPS Workshop on Challenges in Learning Hierarchical Models.
Goodfellow, I. J., Courville, A. C., and Bengio, Y . (2012). Large-scale feature learning with spike-
and-slab sparse coding. In Proceedings of the 29th International Conference on Machine Learning
(ICML).
Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y . (2013). Maxout net-
works. In International Conference on Machine Learning (ICML).
Graves, A. (2011). Practical variational inference for neural networks. In Advances in Neural Infor-
mation Processing Systems (NIPS), pages 2348–2356.
50
Graves, A., Eck, D., Beringer, N., and Schmidhuber, J. (2003). Isolated digit recognition with LSTM
recurrent networks. In First International Workshop on Biologically Inspired Approaches to Ad-
vanced Information Technology, Lausanne.
Graves, A., Fernandez, S., Gomez, F. J., and Schmidhuber, J. (2006). Connectionist temporal classiﬁ-
cation: Labelling unsegmented sequence data with recurrent neural nets. InICML’06: Proceedings
of the 23rd International Conference on Machine Learning, pages 369–376.
Graves, A., Fernandez, S., Liwicki, M., Bunke, H., and Schmidhuber, J. (2008). Unconstrained on-
line handwriting recognition with recurrent neural networks. In Platt, J., Koller, D., Singer, Y .,
and Roweis, S., editors, Advances in Neural Information Processing Systems (NIPS) 20 , pages
577–584. MIT Press, Cambridge, MA.
Graves, A. and Jaitly, N. (2014). Towards end-to-end speech recognition with recurrent neural net-
works. In Proc. 31st International Conference on Machine Learning (ICML), pages 1764–1772.
Graves, A., Liwicki, M., Fernandez, S., Bertolami, R., Bunke, H., and Schmidhuber, J. (2009). A
novel connectionist system for improved unconstrained handwriting recognition. IEEE Transac-
tions on Pattern Analysis and Machine Intelligence, 31(5).
Graves, A., Mohamed, A.-R., and Hinton, G. E. (2013). Speech recognition with deep recurrent
neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing
(ICASSP), pages 6645–6649. IEEE.
Graves, A. and Schmidhuber, J. (2005). Framewise phoneme classiﬁcation with bidirectional LSTM
and other neural network architectures. Neural Networks, 18(5-6):602–610.
Graves, A. and Schmidhuber, J. (2009). Ofﬂine handwriting recognition with multidimensional re-
current neural networks. In Advances in Neural Information Processing Systems (NIPS) 21, pages
545–552. MIT Press, Cambridge, MA.
Graziano, M. (2009). The Intelligent Movement Machine: An Ethological Perspective on the Primate
Motor System. Oxford University Press, USA.
Griewank, A. (2012). Documenta Mathematica - Extra Volume ISMP, pages 389–400.
Grondman, I., Busoniu, L., Lopes, G. A. D., and Babuska, R. (2012). A survey of actor-critic rein-
forcement learning: Standard and natural policy gradients. Systems, Man, and Cybernetics, Part C:
Applications and Reviews, IEEE Transactions on, 42(6):1291–1307.
Grossberg, S. (1969). Some networks that can learn, remember, and reproduce any number of com-
plicated space-time patterns, I. Journal of Mathematics and Mechanics, 19:53–91.
Grossberg, S. (1976a). Adaptive pattern classiﬁcation and universal recoding, 1: Parallel development
and coding of neural feature detectors. Biological Cybernetics, 23:187–202.
Grossberg, S. (1976b). Adaptive pattern classiﬁcation and universal recoding, 2: Feedback, expecta-
tion, olfaction, and illusions. Biological Cybernetics, 23.
Gruau, F., Whitley, D., and Pyeatt, L. (1996). A comparison between cellular encoding and direct
encoding for genetic neural networks. NeuroCOLT Technical Report NC-TR-96-048, ESPRIT
Working Group in Neural and Computational Learning, NeuroCOLT 8556.
51
Gr¨unwald, P. D., Myung, I. J., and Pitt, M. A. (2005). Advances in minimum description length:
Theory and applications. MIT Press.
Gr¨uttner, M., Sehnke, F., Schaul, T., and Schmidhuber, J. (2010). Multi-Dimensional Deep Memory
Atari-Go Players for Parameter Exploring Policy Gradients. In Proceedings of the International
Conference on Artiﬁcial Neural Networks ICANN, pages 114–123. Springer.
Guo, X., Singh, S., Lee, H., Lewis, R., and Wang, X. (2014). Deep learning for real-time Atari game
play using ofﬂine Monte-Carlo tree search planning. InAdvances in Neural Information Processing
Systems 27 (NIPS).
Guyon, I., Vapnik, V ., Boser, B., Bottou, L., and Solla, S. A. (1992). Structural risk minimization for
character recognition. In Lippman, D. S., Moody, J. E., and Touretzky, D. S., editors, Advances in
Neural Information Processing Systems (NIPS) 4, pages 471–479. Morgan Kaufmann.
Hadamard, J. (1908). M´emoire sur le probl`eme d’analyse relatif `a l’´equilibre des plaques ´elastiques
encastr´ees. M ´emoires pr ´esent´es par divers savants `a l’Acad´emie des sciences de l’Institut de
France: ´Extrait. Imprimerie nationale.
Hadsell, R., Chopra, S., and LeCun, Y . (2006). Dimensionality reduction by learning an invariant
mapping. In Proc. Computer Vision and Pattern Recognition Conference (CVPR’06). IEEE Press.
Hagras, H., Pounds-Cornish, A., Colley, M., Callaghan, V ., and Clarke, G. (2004). Evolving spiking
neural network controllers for autonomous robots. In IEEE International Conference on Robotics
and Automation (ICRA), volume 5, pages 4620–4626.
Hansen, N., M¨uller, S. D., and Koumoutsakos, P. (2003). Reducing the time complexity of the deran-
domized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary Computa-
tion, 11(1):1–18.
Hansen, N. and Ostermeier, A. (2001). Completely derandomized self-adaptation in evolution strate-
gies. Evolutionary Computation, 9(2):159–195.
Hanson, S. J. (1990). A stochastic version of the delta rule. Physica D: Nonlinear Phenomena ,
42(1):265–272.
Hanson, S. J. and Pratt, L. Y . (1989). Comparing biases for minimal network construction with
back-propagation. In Touretzky, D. S., editor, Advances in Neural Information Processing Systems
(NIPS) 1, pages 177–185. San Mateo, CA: Morgan Kaufmann.
Happel, B. L. and Murre, J. M. (1994). Design and evolution of modular neural network architectures.
Neural Networks, 7(6):985–1004.
Hashem, S. and Schmeiser, B. (1992). Improving model accuracy using optimal linear combinations
of trained neural networks. IEEE Transactions on Neural Networks, 6:792–794.
Hassibi, B. and Stork, D. G. (1993). Second order derivatives for network pruning: Optimal brain
surgeon. In Lippman, D. S., Moody, J. E., and Touretzky, D. S., editors, Advances in Neural
Information Processing Systems 5, pages 164–171. Morgan Kaufmann.
Hastie, T., Tibshirani, R., and Friedman, J. (2009). The elements of statistical learning . Springer
Series in Statistics.
52
Hastie, T. J. and Tibshirani, R. J. (1990). Generalized additive models. Monographs on Statisics and
Applied Probability, 43.
Hawkins, J. and George, D. (2006). Hierarchical Temporal Memory - Concepts, Theory, and Termi-
nology. Numenta Inc.
Haykin, S. S. (2001). Kalman ﬁltering and neural networks. Wiley Online Library.
Hebb, D. O. (1949). The Organization of Behavior. Wiley, New York.
Hecht-Nielsen, R. (1989). Theory of the backpropagation neural network. In International Joint
Conference on Neural Networks (IJCNN), pages 593–605. IEEE.
Heemskerk, J. N. (1995). Overview of neural hardware. Neurocomputers for Brain-Style Processing.
Design, Implementation and Application.
Heess, N., Silver, D., and Teh, Y . W. (2012). Actor-critic reinforcement learning with energy-based
policies. In Proc. European Workshop on Reinforcement Learning, pages 43–57.
Heidrich-Meisner, V . and Igel, C. (2009). Neuroevolution strategies for episodic reinforcement learn-
ing. Journal of Algorithms, 64(4):152–168.
Herrero, J., Valencia, A., and Dopazo, J. (2001). A hierarchical unsupervised growing neural network
for clustering gene expression patterns. Bioinformatics, 17(2):126–136.
Hertz, J., Krogh, A., and Palmer, R. (1991). Introduction to the Theory of Neural Computation .
Addison-Wesley, Redwood City.
Hestenes, M. R. and Stiefel, E. (1952). Methods of conjugate gradients for solving linear systems.
Journal of research of the National Bureau of Standards, 49:409–436.
Hihi, S. E. and Bengio, Y . (1996). Hierarchical recurrent neural networks for long-term dependencies.
In Touretzky, D. S., Mozer, M. C., and Hasselmo, M. E., editors, Advances in Neural Information
Processing Systems 8, pages 493–499. MIT Press.
Hinton, G. and Salakhutdinov, R. (2006). Reducing the dimensionality of data with neural networks.
Science, 313(5786):504–507.
Hinton, G. E. (1989). Connectionist learning procedures. Artiﬁcial intelligence, 40(1):185–234.
Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural
Comp., 14(8):1771–1800.
Hinton, G. E., Dayan, P., Frey, B. J., and Neal, R. M. (1995). The wake-sleep algorithm for unsuper-
vised neural networks. Science, 268:1158–1160.
Hinton, G. E., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V .,
Nguyen, P., Sainath, T. N., and Kingsbury, B. (2012a). Deep neural networks for acoustic modeling
in speech recognition: The shared views of four research groups. IEEE Signal Process. Mag. ,
29(6):82–97.
Hinton, G. E. and Ghahramani, Z. (1997). Generative models for discovering sparse distributed
representations. Philosophical Transactions of the Royal Society B, 352:1177–1190.
53
Hinton, G. E., Osindero, S., and Teh, Y .-W. (2006). A fast learning algorithm for deep belief nets.
Neural Computation, 18(7):1527–1554.
Hinton, G. E. and Sejnowski, T. E. (1986). Learning and relearning in Boltzmann machines. In
Parallel Distributed Processing, volume 1, pages 282–317. MIT Press.
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. R. (2012b).
Improving neural networks by preventing co-adaptation of feature detectors. Technical Report
arXiv:1207.0580.
Hinton, G. E. and van Camp, D. (1993). Keeping neural networks simple. In Proceedings of the
International Conference on Artiﬁcial Neural Networks, Amsterdam, pages 11–18. Springer.
Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut
f¨ur Informatik, Lehrstuhl Prof. Brauer, Technische Universit¨at M¨unchen. Advisor: J. Schmidhuber.
Hochreiter, S., Bengio, Y ., Frasconi, P., and Schmidhuber, J. (2001a). Gradient ﬂow in recurrent nets:
the difﬁculty of learning long-term dependencies. In Kremer, S. C. and Kolen, J. F., editors,A Field
Guide to Dynamical Recurrent Neural Networks. IEEE Press.
Hochreiter, S. and Obermayer, K. (2005). Sequence classiﬁcation for protein analysis. In Snowbird
Workshop, Snowbird, Utah. Computational and Biological Learning Society.
Hochreiter, S. and Schmidhuber, J. (1996). Bridging long time lags by weight guessing and “Long
Short-Term Memory”. In Silva, F. L., Principe, J. C., and Almeida, L. B., editors, Spatiotemporal
models in biological and artiﬁcial systems , pages 65–72. IOS Press, Amsterdam, Netherlands.
Serie: Frontiers in Artiﬁcial Intelligence and Applications, V olume 37.
Hochreiter, S. and Schmidhuber, J. (1997a). Flat minima. Neural Computation, 9(1):1–42.
Hochreiter, S. and Schmidhuber, J. (1997b). Long Short-Term Memory. Neural Computation ,
9(8):1735–1780. Based on TR FKI-207-95, TUM (1995).
Hochreiter, S. and Schmidhuber, J. (1999). Feature extraction through LOCOCODE. Neural Compu-
tation, 11(3):679–714.
Hochreiter, S., Younger, A. S., and Conwell, P. R. (2001b). Learning to learn using gradient descent.
In Lecture Notes on Comp. Sci. 2130, Proc. Intl. Conf. on Artiﬁcial Neural Networks (ICANN-
2001), pages 87–94. Springer: Berlin, Heidelberg.
Hodgkin, A. L. and Huxley, A. F. (1952). A quantitative description of membrane current and its
application to conduction and excitation in nerve. The Journal of Physiology, 117(4):500.
Hoerzer, G. M., Legenstein, R., and Maass, W. (2014). Emergence of complex computational struc-
tures from chaotic neural networks through reward-modulated Hebbian learning. Cerebral Cortex,
24:677–690.
Holden, S. B. (1994). On the Theory of Generalization and Self-Structuring in Linearly Weighted
Connectionist Networks. PhD thesis, Cambridge University, Engineering Department.
Holland, J. H. (1975). Adaptation in Natural and Artiﬁcial Systems . University of Michigan Press,
Ann Arbor.
Honavar, V . and Uhr, L. (1993). Generative learning structures and processes for generalized connec-
tionist networks. Information Sciences, 70(1):75–108.
54
Honavar, V . and Uhr, L. M. (1988). A network of neuron-like units that learns to perceive by gen-
eration as well as reweighting of its links. In Touretzky, D., Hinton, G. E., and Sejnowski, T.,
editors, Proc. of the 1988 Connectionist Models Summer School, pages 472–484, San Mateo. Mor-
gan Kaufman.
Hopﬁeld, J. J. (1982). Neural networks and physical systems with emergent collective computational
abilities. Proc. of the National Academy of Sciences, 79:2554–2558.
Hornik, K., Stinchcombe, M., and White, H. (1989). Multilayer feedforward networks are universal
approximators. Neural Networks, 2(5):359–366.
Hubel, D. H. and Wiesel, T. (1962). Receptive ﬁelds, binocular interaction, and functional architecture
in the cat’s visual cortex.Journal of Physiology (London), 160:106–154.
Hubel, D. H. and Wiesel, T. N. (1968). Receptive ﬁelds and functional architecture of monkey striate
cortex. The Journal of Physiology, 195(1):215–243.
Huffman, D. A. (1952). A method for construction of minimum-redundancy codes.Proceedings IRE,
40:1098–1101.
Hung, C. P., Kreiman, G., Poggio, T., and DiCarlo, J. J. (2005). Fast readout of object identity from
macaque inferior temporal cortex. Science, 310(5749):863–866.
Hutter, M. (2002). The fastest and shortest algorithm for all well-deﬁned problems. International
Journal of Foundations of Computer Science , 13(3):431–443. (On J. Schmidhuber’s SNF grant
20-61847).
Hutter, M. (2005). Universal Artiﬁcial Intelligence: Sequential Decisions based on Algorithmic Prob-
ability. Springer, Berlin. (On J. Schmidhuber’s SNF grant 20-61847).
Hyv¨arinen, A., Hoyer, P., and Oja, E. (1999). Sparse code shrinkage: Denoising by maximum likeli-
hood estimation. In Kearns, M., Solla, S. A., and Cohn, D., editors,Advances in Neural Information
Processing Systems (NIPS) 12. MIT Press.
Hyv¨arinen, A., Karhunen, J., and Oja, E. (2001). Independent component analysis . John Wiley &
Sons.
ICPR 2012 Contest on Mitosis Detection in Breast Cancer Histological Images (2012). IPAL Lab-
oratory and TRIBVN Company and Pitie-Salpetriere Hospital and CIALAB of Ohio State Univ.,
http://ipal.cnrs.fr/ICPR2012/.
Igel, C. (2003). Neuroevolution for reinforcement learning using evolution strategies. In Reynolds, R.,
Abbass, H., Tan, K. C., Mckay, B., Essam, D., and Gedeon, T., editors, Congress on Evolutionary
Computation (CEC 2003), volume 4, pages 2588–2595. IEEE.
Igel, C. and H ¨usken, M. (2003). Empirical evaluation of the improved Rprop learning algorithm.
Neurocomputing, 50(C):105–123.
Ikeda, S., Ochiai, M., and Sawaragi, Y . (1976). Sequential GMDH algorithm and its application to
river ﬂow prediction. IEEE Transactions on Systems, Man and Cybernetics, (7):473–479.
Indermuhle, E., Frinken, V ., and Bunke, H. (2012). Mode detection in online handwritten documents
using BLSTM neural networks. In Frontiers in Handwriting Recognition (ICFHR), 2012 Interna-
tional Conference on, pages 302–307. IEEE.
55
Indermuhle, E., Frinken, V ., Fischer, A., and Bunke, H. (2011). Keyword spotting in online handwrit-
ten documents containing text and non-text using BLSTM neural networks. In Document Analysis
and Recognition (ICDAR), 2011 International Conference on, pages 73–77. IEEE.
Indiveri, G., Linares-Barranco, B., Hamilton, T. J., Van Schaik, A., Etienne-Cummings, R., Delbruck,
T., Liu, S.-C., Dudek, P., H ¨aﬂiger, P., Renaud, S., et al. (2011). Neuromorphic silicon neuron
circuits. Frontiers in Neuroscience, 5(73).
Ivakhnenko, A. G. (1968). The group method of data handling – a rival of the method of stochastic
approximation. Soviet Automatic Control, 13(3):43–55.
Ivakhnenko, A. G. (1971). Polynomial theory of complex systems. IEEE Transactions on Systems,
Man and Cybernetics, (4):364–378.
Ivakhnenko, A. G. (1995). The review of problems solvable by algorithms of the group method of data
handling (GMDH). Pattern Recognition and Image Analysis / Raspoznavaniye Obrazov I Analiz
Izobrazhenii, 5:527–535.
Ivakhnenko, A. G. and Lapa, V . G. (1965).Cybernetic Predicting Devices. CCM Information Corpo-
ration.
Ivakhnenko, A. G., Lapa, V . G., and McDonough, R. N. (1967). Cybernetics and forecasting tech-
niques. American Elsevier, NY .
Izhikevich, E. M. et al. (2003). Simple model of spiking neurons. IEEE Transactions on Neural
Networks, 14(6):1569–1572.
Jaakkola, T., Singh, S. P., and Jordan, M. I. (1995). Reinforcement learning algorithm for partially
observable Markov decision problems. In Tesauro, G., Touretzky, D. S., and Leen, T. K., editors,
Advances in Neural Information Processing Systems (NIPS) 7, pages 345–352. MIT Press.
Jackel, L., Boser, B., Graf, H.-P., Denker, J., LeCun, Y ., Henderson, D., Matan, O., Howard, R., and
Baird, H. (1990). VLSI implementation of electronic neural networks: and example in character
recognition. In IEEE, editor, IEEE International Conference on Systems, Man, and Cybernetics ,
pages 320–322, Los Angeles, CA.
Jacob, C., Lindenmayer, A., and Rozenberg, G. (1994). Genetic L-System Programming. In Parallel
Problem Solving from Nature III, Lecture Notes in Computer Science.
Jacobs, R. A. (1988). Increased rates of convergence through learning rate adaptation. Neural Net-
works, 1(4):295–307.
Jaeger, H. (2001). The ”echo state” approach to analysing and training recurrent neural networks.
Technical Report GMD Report 148, German National Research Center for Information Technology.
Jaeger, H. (2004). Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless
communication. Science, 304:78–80.
Jain, V . and Seung, S. (2009). Natural image denoising with convolutional networks. In Koller, D.,
Schuurmans, D., Bengio, Y ., and Bottou, L., editors, Advances in Neural Information Processing
Systems (NIPS) 21, pages 769–776. Curran Associates, Inc.
Jameson, J. (1991). Delayed reinforcement learning with multiple time scale hierarchical backpropa-
gated adaptive critics. In Neural Networks for Control.
56
Ji, S., Xu, W., Yang, M., and Yu, K. (2013). 3D convolutional neural networks for human action
recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1):221–231.
Jim, K., Giles, C. L., and Horne, B. G. (1995). Effects of noise on convergence and generalization
in recurrent networks. In Tesauro, G., Touretzky, D., and Leen, T., editors, Advances in Neural
Information Processing Systems (NIPS) 7, page 649. San Mateo, CA: Morgan Kaufmann.
Jin, X., Lujan, M., Plana, L. A., Davies, S., Temple, S., and Furber, S. B. (2010). Modeling spiking
neural networks on SpiNNaker. Computing in Science & Engineering, 12(5):91–97.
Jodogne, S. R. and Piater, J. H. (2007). Closed-loop learning of visual control policies. J. Artiﬁcial
Intelligence Research, 28:349–391.
Jones, J. P. and Palmer, L. A. (1987). An evaluation of the two-dimensional Gabor ﬁlter model of
simple receptive ﬁelds in cat striate cortex. Journal of Neurophysiology, 58(6):1233–1258.
Jordan, M. I. (1986). Serial order: A parallel distributed processing approach. Technical Report ICS
Report 8604, Institute for Cognitive Science, University of California, San Diego.
Jordan, M. I. (1988). Supervised learning and systems with excess degrees of freedom. Technical
Report COINS TR 88-27, Massachusetts Institute of Technology.
Jordan, M. I. (1997). Serial order: A parallel distributed processing approach. Advances in Psychol-
ogy, 121:471–495.
Jordan, M. I. and Rumelhart, D. E. (1990). Supervised learning with a distal teacher. Technical Report
Occasional Paper #40, Center for Cog. Sci., Massachusetts Institute of Technology.
Jordan, M. I. and Sejnowski, T. J. (2001). Graphical models: Foundations of neural computation .
MIT Press.
Joseph, R. D. (1961). Contributions to perceptron theory. PhD thesis, Cornell Univ.
Juang, C.-F. (2004). A hybrid of genetic algorithm and particle swarm optimization for recurrent
network design. Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on ,
34(2):997–1006.
Judd, J. S. (1990). Neural network design and the complexity of learning. Neural network modeling
and connectionism. MIT Press.
Jutten, C. and Herault, J. (1991). Blind separation of sources, part I: An adaptive algorithm based on
neuromimetic architecture. Signal Processing, 24(1):1–10.
Kaelbling, L. P., Littman, M. L., and Cassandra, A. R. (1995). Planning and acting in partially
observable stochastic domains. Technical report, Brown University, Providence RI.
Kaelbling, L. P., Littman, M. L., and Moore, A. W. (1996). Reinforcement learning: a survey.Journal
of AI research, 4:237–285.
Kak, S., Chen, Y ., and Wang, L. (2010). Data mining using surface and deep agents based on neural
networks. AMCIS 2010 Proceedings.
Kalinke, Y . and Lehmann, H. (1998). Computation in recurrent neural networks: From counters to
iterated function systems. In Antoniou, G. and Slaney, J., editors, Advanced Topics in Artiﬁcial
Intelligence, Proceedings of the 11th Australian Joint Conference on Artiﬁcial Intelligence, volume
1502 of LNAI, Berlin, Heidelberg. Springer.
57
Kalman, R. E. (1960). A new approach to linear ﬁltering and prediction problems. Journal of Basic
Engineering, 82(1):35–45.
Karhunen, J. and Joutsensalo, J. (1995). Generalizations of principal component analysis, optimiza-
tion problems, and neural networks. Neural Networks, 8(4):549–562.
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., and Fei-Fei, L. (2014). Large-scale
video classiﬁcation with convolutional neural networks. In IEEE Conference on Computer Vision
and Pattern Recognition (CVPR).
Kasabov, N. K. (2014). Neucube: A spiking neural network architecture for mapping, learning and
understanding of spatio-temporal brain data. Neural Networks.
Kelley, H. J. (1960). Gradient theory of optimal ﬂight paths. ARS Journal, 30(10):947–954.
Kempter, R., Gerstner, W., and Van Hemmen, J. L. (1999). Hebbian learning and spiking neurons.
Physical Review E, 59(4):4498.
Kerlirzin, P. and Vallet, F. (1993). Robustness in multilayer perceptrons. Neural Computation ,
5(1):473–482.
Khan, M. M., Khan, G. M., and Miller, J. F. (2010). Evolution of neural networks using Cartesian
Genetic Programming. In IEEE Congress on Evolutionary Computation (CEC), pages 1–8.
Khan, M. M., Lester, D. R., Plana, L. A., Rast, A., Jin, X., Painkras, E., and Furber, S. B. (2008).
SpiNNaker: mapping neural networks onto a massively-parallel chip multiprocessor. In Interna-
tional Joint Conference on Neural Networks (IJCNN), pages 2849–2856. IEEE.
Khan, S. H., Bennamoun, M., Sohel, F., and Togneri, R. (2014). Automatic feature learning for robust
shadow detection. In IEEE Conference on Computer Vision and Pattern Recognition CVPR.
Kimura, H., Miyazaki, K., and Kobayashi, S. (1997). Reinforcement learning in POMDPs with
function approximation. In ICML, volume 97, pages 152–160.
Kistler, W. M., Gerstner, W., and van Hemmen, J. L. (1997). Reduction of the Hodgkin-Huxley
equations to a single-variable threshold model. Neural Computation, 9(5):1015–1045.
Kitano, H. (1990). Designing neural networks using genetic algorithms with graph generation system.
Complex Systems, 4:461–476.
Klampﬂ, S. and Maass, W. (2013). Emergence of dynamic memory traces in cortical microcircuit
models through STDP. The Journal of Neuroscience, 33(28):11515–11529.
Klapper-Rybicka, M., Schraudolph, N. N., and Schmidhuber, J. (2001). Unsupervised learning in
LSTM recurrent neural networks. In Lecture Notes on Comp. Sci. 2130, Proc. Intl. Conf. on Artiﬁ-
cial Neural Networks (ICANN-2001), pages 684–691. Springer: Berlin, Heidelberg.
Kobatake, E. and Tanaka, K. (1994). Neuronal selectivities to complex object features in the ventral
visual pathway of the macaque cerebral cortex. J. Neurophysiol., 71:856–867.
Kohl, N. and Stone, P. (2004). Policy gradient reinforcement learning for fast quadrupedal locomotion.
In Robotics and Automation, 2004. Proceedings. ICRA’04. 2004 IEEE International Conference on,
volume 3, pages 2619–2624. IEEE.
58
Kohonen, T. (1972). Correlation matrix memories. Computers, IEEE Transactions on, 100(4):353–
359.
Kohonen, T. (1982). Self-organized formation of topologically correct feature maps. Biological
Cybernetics, 43(1):59–69.
Kohonen, T. (1988). Self-Organization and Associative Memory. Springer, second edition.
Koikkalainen, P. and Oja, E. (1990). Self-organizing hierarchical feature maps. InInternational Joint
Conference on Neural Networks (IJCNN), pages 279–284. IEEE.
Kolmogorov, A. N. (1965a). On the representation of continuous functions of several variables by su-
perposition of continuous functions of one variable and addition. Doklady Akademii. Nauk USSR,,
114:679–681.
Kolmogorov, A. N. (1965b). Three approaches to the quantitative deﬁnition of information.Problems
of Information Transmission, 1:1–11.
Kompella, V . R., Luciw, M. D., and Schmidhuber, J. (2012). Incremental slow feature analysis: Adap-
tive low-complexity slow feature updating from high-dimensional input streams.Neural Computa-
tion, 24(11):2994–3024.
Kondo, T. (1998). GMDH neural network algorithm using the heuristic self-organization method
and its application to the pattern identiﬁcation problem. In Proceedings of the 37th SICE Annual
Conference SICE’98, pages 1143–1148. IEEE.
Kondo, T. and Ueno, J. (2008). Multi-layered GMDH-type neural network self-selecting optimum
neural network architecture and its application to 3-dimensional medical image recognition of
blood vessels. International Journal of Innovative Computing, Information and Control, 4(1):175–
187.
Kord´ık, P., N´aplava, P., Snorek, M., and Genyk-Berezovskyj, M. (2003). Modiﬁed GMDH method
and models quality evaluation by visualization. Control Systems and Computers, 2:68–75.
Korkin, M., de Garis, H., Gers, F., and Hemmi, H. (1997). CBM (CAM-Brain Machine) - a hardware
tool which evolves a neural net module in a fraction of a second and runs a million neuron artiﬁcial
brain in real time.
Kosko, B. (1990). Unsupervised learning in noise. IEEE Transactions on Neural Networks, 1(1):44–
57.
Koutn´ık, J., Cuccu, G., Schmidhuber, J., and Gomez, F. (July 2013). Evolving large-scale neural
networks for vision-based reinforcement learning. In Proceedings of the Genetic and Evolutionary
Computation Conference (GECCO), pages 1061–1068, Amsterdam. ACM.
Koutn´ık, J., Gomez, F., and Schmidhuber, J. (2010). Evolving neural networks in compressed weight
space. In Proceedings of the 12th Annual Conference on Genetic and Evolutionary Computation ,
pages 619–626.
Koutn´ık, J., Greff, K., Gomez, F., and Schmidhuber, J. (2014). A Clockwork RNN. In Proceedings
of the 31th International Conference on Machine Learning (ICML), volume 32, pages 1845–1853.
arXiv:1402.3511 [cs.NE].
59
Koza, J. R. (1992). Genetic Programming – On the Programming of Computers by Means of Natural
Selection. MIT Press.
Kramer, M. (1991). Nonlinear principal component analysis using autoassociative neural networks.
AIChE Journal, 37:233–243.
Kremer, S. C. and Kolen, J. F. (2001). Field guide to dynamical recurrent networks . Wiley-IEEE
Press.
Kriegeskorte, N., Mur, M., Ruff, D. A., Kiani, R., Bodurka, J., Esteky, H., Tanaka, K., and Bandettini,
P. A. (2008). Matching categorical object representations in inferior temporal cortex of man and
monkey. Neuron, 60(6):1126–1141.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classiﬁcation with deep convo-
lutional neural networks. In Advances in Neural Information Processing Systems (NIPS 2012) ,
page 4.
Krogh, A. and Hertz, J. A. (1992). A simple weight decay can improve generalization. In Lippman,
D. S., Moody, J. E., and Touretzky, D. S., editors, Advances in Neural Information Processing
Systems 4, pages 950–957. Morgan Kaufmann.
Kruger, N., Janssen, P., Kalkan, S., Lappe, M., Leonardis, A., Piater, J., Rodriguez-Sanchez, A., and
Wiskott, L. (2013). Deep hierarchies in the primate visual cortex: What can we learn for computer
vision? IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1847–1871.
Kullback, S. and Leibler, R. A. (1951). On information and sufﬁciency. The Annals of Mathematical
Statistics, pages 79–86.
Kurzweil, R. (2012). How to Create a Mind: The Secret of Human Thought Revealed.
Lagoudakis, M. G. and Parr, R. (2003). Least-squares policy iteration. JMLR, 4:1107–1149.
Lampinen, J. and Oja, E. (1992). Clustering properties of hierarchical self-organizing maps. Journal
of Mathematical Imaging and Vision, 2(2-3):261–272.
Lang, K., Waibel, A., and Hinton, G. E. (1990). A time-delay neural network architecture for isolated
word recognition. Neural Networks, 3:23–43.
Lange, S. and Riedmiller, M. (2010). Deep auto-encoder neural networks in reinforcement learning.
In Neural Networks (IJCNN), The 2010 International Joint Conference on, pages 1–8.
Lapedes, A. and Farber, R. (1986). A self-optimizing, nonsymmetrical neural net for content address-
able memory and pattern recognition. Physica D, 22:247–259.
Laplace, P. (1774). M ´emoire sur la probabilit ´e des causes par les ´ev`enements. M´emoires de
l’Academie Royale des Sciences Present´es par Divers Savan, 6:621–656.
Larraanaga, P. and Lozano, J. A. (2001). Estimation of Distribution Algorithms: A New Tool for
Evolutionary Computation. Kluwer Academic Publishers, Norwell, MA, USA.
Le, Q. V ., Ranzato, M., Monga, R., Devin, M., Corrado, G., Chen, K., Dean, J., and Ng, A. Y . (2012).
Building high-level features using large scale unsupervised learning. In Proc. ICML’12.
LeCun, Y . (1985). Une proc ´edure d’apprentissage pour r´eseau `a seuil asym ´etrique. Proceedings of
Cognitiva 85, Paris, pages 599–604.
60
LeCun, Y . (1988). A theoretical framework for back-propagation. In Touretzky, D., Hinton, G.,
and Sejnowski, T., editors, Proceedings of the 1988 Connectionist Models Summer School , pages
21–28, CMU, Pittsburgh, Pa. Morgan Kaufmann.
LeCun, Y ., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D.
(1989). Back-propagation applied to handwritten zip code recognition. Neural Computation ,
1(4):541–551.
LeCun, Y ., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D.
(1990a). Handwritten digit recognition with a back-propagation network. In Touretzky, D. S.,
editor, Advances in Neural Information Processing Systems 2, pages 396–404. Morgan Kaufmann.
LeCun, Y ., Bottou, L., Bengio, Y ., and Haffner, P. (1998). Gradient-based learning applied to docu-
ment recognition. Proceedings of the IEEE, 86(11):2278–2324.
LeCun, Y ., Denker, J. S., and Solla, S. A. (1990b). Optimal brain damage. In Touretzky, D. S., editor,
Advances in Neural Information Processing Systems 2, pages 598–605. Morgan Kaufmann.
LeCun, Y ., Muller, U., Cosatto, E., and Flepp, B. (2006). Off-road obstacle avoidance through end-
to-end learning. In Advances in Neural Information Processing Systems (NIPS 2005).
LeCun, Y ., Simard, P., and Pearlmutter, B. (1993). Automatic learning rate maximization by on-line
estimation of the Hessian’s eigenvectors. In Hanson, S., Cowan, J., and Giles, L., editors,Advances
in Neural Information Processing Systems (NIPS 1992), volume 5. Morgan Kaufmann Publishers,
San Mateo, CA.
Lee, H., Battle, A., Raina, R., and Ng, A. Y . (2007a). Efﬁcient sparse coding algorithms. InAdvances
in Neural Information Processing Systems (NIPS) 19, pages 801–808.
Lee, H., Ekanadham, C., and Ng, A. Y . (2007b). Sparse deep belief net model for visual area V2. In
Advances in Neural Information Processing Systems (NIPS), volume 7, pages 873–880.
Lee, H., Grosse, R., Ranganath, R., and Ng, A. Y . (2009a). Convolutional deep belief networks for
scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th Interna-
tional Conference on Machine Learning (ICML), pages 609–616.
Lee, H., Pham, P. T., Largman, Y ., and Ng, A. Y . (2009b). Unsupervised feature learning for audio
classiﬁcation using convolutional deep belief networks. InProc. NIPS, volume 9, pages 1096–1104.
Lee, L. (1996). Learning of context-free languages: A survey of the literature. Technical Report
TR-12-96, Center for Research in Computing Technology, Harvard University, Cambridge, Mas-
sachusetts.
Lee, S. and Kil, R. M. (1991). A Gaussian potential function network with hierarchically self-
organizing learning. Neural Networks, 4(2):207–224.
Legendre, A. M. (1805). Nouvelles m´ethodes pour la d´etermination des orbites des cometes. F. Didot.
Legenstein, R., Wilbert, N., and Wiskott, L. (2010). Reinforcement learning on slow features of
high-dimensional input streams. PLoS Computational Biology, 6(8).
Legenstein, R. A. and Maass, W. (2002). Neural circuits for pattern recognition with small total wire
length. Theor. Comput. Sci., 287(1):239–249.
61
Leibniz, G. W. (1676). Memoir using the chain rule (cited in TMME 7:2&3 p 321-332, 2010).
Leibniz, G. W. (1684). Nova methodus pro maximis et minimis, itemque tangentibus, quae nec
fractas, nec irrationales quantitates moratur, et singulare pro illis calculi genus. Acta Eruditorum,
pages 467–473.
Lenat, D. B. (1983). Theory formation by heuristic search. Machine Learning, 21.
Lenat, D. B. and Brown, J. S. (1984). Why AM an EURISKO appear to work. Artiﬁcial Intelligence,
23(3):269–294.
Lennie, P. and Movshon, J. A. (2005). Coding of color and form in the geniculostriate visual pathway.
Journal of the Optical Society of America A, 22(10):2013–2033.
Levenberg, K. (1944). A method for the solution of certain problems in least squares. Quarterly of
applied mathematics, 2:164–168.
Levin, A. U., Leen, T. K., and Moody, J. E. (1994). Fast pruning using principal components. In
Advances in Neural Information Processing Systems 6, page 35. Morgan Kaufmann.
Levin, A. U. and Narendra, K. S. (1995). Control of nonlinear dynamical systems using neural
networks. ii. observability, identiﬁcation, and control. IEEE Transactions on Neural Networks ,
7(1):30–42.
Levin, L. A. (1973a). On the notion of a random sequence. Soviet Math. Dokl., 14(5):1413–1416.
Levin, L. A. (1973b). Universal sequential search problems. Problems of Information Transmission,
9(3):265–266.
Lewicki, M. S. and Olshausen, B. A. (1998). Inferring sparse, overcomplete image codes using an
efﬁcient coding framework. In Jordan, M. I., Kearns, M. J., and Solla, S. A., editors, Advances in
Neural Information Processing Systems (NIPS) 10, pages 815–821.
L’Hˆopital, G. F. A. (1696).Analyse des inﬁniment petits, pour l’intelligence des lignes courbes. Paris:
L’Imprimerie Royale.
Li, M. and Vit´anyi, P. M. B. (1997). An Introduction to Kolmogorov Complexity and its Applications
(2nd edition). Springer.
Li, R., Zhang, W., Suk, H.-I., Wang, L., Li, J., Shen, D., and Ji, S. (2014). Deep learning based
imaging data completion for improved brain disease diagnosis. In Proc. MICCAI. Springer.
Lin, L. (1993). Reinforcement Learning for Robots Using Neural Networks . PhD thesis, Carnegie
Mellon University, Pittsburgh.
Lin, T., Horne, B., Tino, P., and Giles, C. (1996). Learning long-term dependencies in NARX recur-
rent neural networks. IEEE Transactions on Neural Networks, 7(6):1329–1338.
Lindenmayer, A. (1968). Mathematical models for cellular interaction in development. J. Theoret.
Biology, 18:280–315.
Lindst¨adt, S. (1993). Comparison of two unsupervised neural network models for redundancy reduc-
tion. In Mozer, M. C., Smolensky, P., Touretzky, D. S., Elman, J. L., and Weigend, A. S., editors,
Proc. of the 1993 Connectionist Models Summer School , pages 308–315. Hillsdale, NJ: Erlbaum
Associates.
62
Linnainmaa, S. (1970). The representation of the cumulative rounding error of an algorithm as a
Taylor expansion of the local rounding errors. Master’s thesis, Univ. Helsinki.
Linnainmaa, S. (1976). Taylor expansion of the accumulated rounding error. BIT Numerical Mathe-
matics, 16(2):146–160.
Linsker, R. (1988). Self-organization in a perceptual network. IEEE Computer, 21:105–117.
Littman, M. L., Cassandra, A. R., and Kaelbling, L. P. (1995). Learning policies for partially ob-
servable environments: Scaling up. In Prieditis, A. and Russell, S., editors, Machine Learning:
Proceedings of the Twelfth International Conference, pages 362–370. Morgan Kaufmann Publish-
ers, San Francisco, CA.
Liu, S.-C., Kramer, J., Indiveri, G., Delbr ¨uck, T., Burg, T., Douglas, R., et al. (2001). Orientation-
selective aVLSI spiking neurons. Neural Networks, 14(6-7):629–643.
Ljung, L. (1998). System identiﬁcation. Springer.
Logothetis, N. K., Pauls, J., and Poggio, T. (1995). Shape representation in the inferior temporal
cortex of monkeys. Current Biology, 5(5):552–563.
Loiacono, D., Cardamone, L., and Lanzi, P. L. (2011). Simulated car racing championship competi-
tion software manual. Technical report, Dipartimento di Elettronica e Informazione, Politecnico di
Milano, Italy.
Loiacono, D., Lanzi, P. L., Togelius, J., Onieva, E., Pelta, D. A., Butz, M. V ., L ¨onneker, T. D.,
Cardamone, L., Perez, D., S ´aez, Y ., Preuss, M., and Quadﬂieg, J. (2009). The 2009 simulated car
racing championship.
Lowe, D. (1999). Object recognition from local scale-invariant features. In The Proceedings of the
Seventh IEEE International Conference on Computer Vision (ICCV), volume 2, pages 1150–1157.
Lowe, D. (2004). Distinctive image features from scale-invariant key-points. Intl. Journal of Com-
puter Vision, 60:91–110.
Luciw, M., Kompella, V . R., Kazerounian, S., and Schmidhuber, J. (2013). An intrinsic value system
for developing multiple invariant representations with incremental slowness learning. Frontiers in
Neurorobotics, 7(9).
Lusci, A., Pollastri, G., and Baldi, P. (2013). Deep architectures and deep learning in chemoinformat-
ics: the prediction of aqueous solubility for drug-like molecules. Journal of Chemical Information
and Modeling, 53(7):1563–1575.
Maas, A. L., Hannun, A. Y ., and Ng, A. Y . (2013). Rectiﬁer nonlinearities improve neural network
acoustic models. In International Conference on Machine Learning (ICML).
Maass, W. (1996). Lower bounds for the computational power of networks of spiking neurons.Neural
Computation, 8(1):1–40.
Maass, W. (1997). Networks of spiking neurons: the third generation of neural network models.
Neural Networks, 10(9):1659–1671.
Maass, W. (2000). On the computational power of winner-take-all. Neural Computation, 12:2519–
2535.
63
Maass, W., Natschl¨ager, T., and Markram, H. (2002). Real-time computing without stable states: A
new framework for neural computation based on perturbations.Neural Computation, 14(11):2531–
2560.
MacKay, D. J. C. (1992). A practical Bayesian framework for backprop networks. Neural Computa-
tion, 4:448–472.
MacKay, D. J. C. and Miller, K. D. (1990). Analysis of Linsker’s simulation of Hebbian rules.Neural
Computation, 2:173–187.
Maclin, R. and Shavlik, J. W. (1993). Using knowledge-based neural networks to improve algorithms:
Reﬁning the Chou-Fasman algorithm for protein folding. Machine Learning, 11(2-3):195–215.
Maclin, R. and Shavlik, J. W. (1995). Combining the predictions of multiple classiﬁers: Using com-
petitive learning to initialize neural networks. In Proc. IJCAI, pages 524–531.
Madala, H. R. and Ivakhnenko, A. G. (1994). Inductive learning algorithms for complex systems
modeling. CRC Press, Boca Raton.
Madani, O., Hanks, S., and Condon, A. (2003). On the undecidability of probabilistic planning and
related stochastic optimization problems. Artiﬁcial Intelligence, 147(1):5–34.
Maei, H. R. and Sutton, R. S. (2010). GQ( λ): A general gradient algorithm for temporal-difference
prediction learning with eligibility traces. In Proceedings of the Third Conference on Artiﬁcial
General Intelligence, volume 1, pages 91–96.
Maex, R. and Orban, G. (1996). Model circuit of spiking neurons generating directional selectivity in
simple cells. Journal of Neurophysiology, 75(4):1515–1545.
Mahadevan, S. (1996). Average reward reinforcement learning: Foundations, algorithms, and empir-
ical results. Machine Learning, 22:159.
Malik, J. and Perona, P. (1990). Preattentive texture discrimination with early vision mechanisms.
Journal of the Optical Society of America A, 7(5):923–932.
Maniezzo, V . (1994). Genetic evolution of the topology and weight distribution of neural networks.
IEEE Transactions on Neural Networks, 5(1):39–53.
Manolios, P. and Fanelli, R. (1994). First-order recurrent neural networks and deterministic ﬁnite
state automata. Neural Computation, 6:1155–1173.
Marchi, E., Ferroni, G., Eyben, F., Gabrielli, L., Squartini, S., and Schuller, B. (2014). Multi-
resolution linear prediction based features for audio onset detection with bidirectional LSTM neural
networks. In Proc. 39th IEEE International Conference on Acoustics, Speech, and Signal Process-
ing, ICASSP 2014, Florence, Italy, pages 2183–2187.
Markram, H. (2012). The human brain project. Scientiﬁc American, 306(6):50–55.
Marquardt, D. W. (1963). An algorithm for least-squares estimation of nonlinear parameters. Journal
of the Society for Industrial & Applied Mathematics, 11(2):431–441.
Martens, J. (2010). Deep learning via Hessian-free optimization. In F ¨urnkranz, J. and Joachims, T.,
editors, Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages
735–742, Haifa, Israel. Omnipress.
64
Martens, J. and Sutskever, I. (2011). Learning recurrent neural networks with Hessian-free optimiza-
tion. In Proceedings of the 28th International Conference on Machine Learning (ICML) , pages
1033–1040.
Martinetz, T. M., Ritter, H. J., and Schulten, K. J. (1990). Three-dimensional neural net for learning
visuomotor coordination of a robot arm. IEEE Transactions on Neural Networks, 1(1):131–136.
Masci, J., Giusti, A., Ciresan, D. C., Fricout, G., and Schmidhuber, J. (2013). A fast learning algorithm
for image segmentation with max-pooling convolutional networks. In International Conference on
Image Processing (ICIP13), pages 2713–2717.
Matsuoka, K. (1992). Noise injection into inputs in back-propagation learning. IEEE Transactions
on Systems, Man, and Cybernetics, 22(3):436–440.
Mayer, H., Gomez, F., Wierstra, D., Nagy, I., Knoll, A., and Schmidhuber, J. (2008). A system for
robotic heart surgery that learns to tie knots using recurrent neural networks. Advanced Robotics,
22(13-14):1521–1537.
McCallum, R. A. (1996). Learning to use selective attention and short-term memory in sequential
tasks. In Maes, P., Mataric, M., Meyer, J.-A., Pollack, J., and Wilson, S. W., editors, From Ani-
mals to Animats 4: Proceedings of the Fourth International Conference on Simulation of Adaptive
Behavior, Cambridge, MA, pages 315–324. MIT Press, Bradford Books.
McCulloch, W. and Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity.
Bulletin of Mathematical Biophysics, 7:115–133.
Melnik, O., Levy, S. D., and Pollack, J. B. (2000). RAAM for inﬁnite context-free languages. In
Proc. IJCNN (5), pages 585–590.
Memisevic, R. and Hinton, G. E. (2010). Learning to represent spatial transformations with factored
higher-order Boltzmann machines. Neural Computation, 22(6):1473–1492.
Menache, I., Mannor, S., and Shimkin, N. (2002). Q-cut – dynamic discovery of sub-goals in rein-
forcement learning. In Proc. ECML’02, pages 295–306.
Merolla, P. A., Arthur, J. V ., Alvarez-Icaza, R., Cassidy, A. S., Sawada, J., Akopyan, F., Jackson,
B. L., Imam, N., Guo, C., Nakamura, Y ., Brezzo, B., V o, I., Esser, S. K., Appuswamy, R., Taba,
B., Amir, A., Flickner, M. D., Risk, W. P., Manohar, R., and Modha, D. S. (2014). A million
spiking-neuron integrated circuit with a scalable communication network and interface. Science,
345(6197):668–673.
Mesnil, G., Dauphin, Y ., Glorot, X., Rifai, S., Bengio, Y ., Goodfellow, I., Lavoie, E., Muller, X.,
Desjardins, G., Warde-Farley, D., Vincent, P., Courville, A., and Bergstra, J. (2011). Unsupervised
and transfer learning challenge: a deep learning approach. In JMLR W&CP: Proc. Unsupervised
and Transfer Learning, volume 7.
Meuleau, N., Peshkin, L., Kim, K. E., and Kaelbling, L. P. (1999). Learning ﬁnite state controllers for
partially observable environments. In 15th International Conference of Uncertainty in AI , pages
427–436.
Miglino, O., Lund, H., and Nolﬁ, S. (1995). Evolving mobile robots in simulated and real environ-
ments. Artiﬁcial Life, 2(4):417–434.
65
Miller, G., Todd, P., and Hedge, S. (1989). Designing neural networks using genetic algorithms. In
Proceedings of the 3rd International Conference on Genetic Algorithms , pages 379–384. Morgan
Kauffman.
Miller, J. F. and Harding, S. L. (2009). Cartesian genetic programming. InProceedings of the 11th An-
nual Conference Companion on Genetic and Evolutionary Computation Conference: Late Break-
ing Papers, pages 3489–3512. ACM.
Miller, J. F. and Thomson, P. (2000). Cartesian genetic programming. InGenetic Programming, pages
121–132. Springer.
Miller, K. D. (1994). A model for the development of simple cell receptive ﬁelds and the ordered
arrangement of orientation columns through activity-dependent competition between on- and off-
center inputs. Journal of Neuroscience, 14(1):409–441.
Miller, W. T., Werbos, P. J., and Sutton, R. S. (1995).Neural networks for control. MIT Press.
Minai, A. A. and Williams, R. D. (1994). Perturbation response in feedforward networks. Neural
Networks, 7(5):783–796.
Minsky, M. (1963). Steps toward artiﬁcial intelligence. In Feigenbaum, E. and Feldman, J., editors,
Computers and Thought, pages 406–450. McGraw-Hill, New York.
Minsky, M. and Papert, S. (1969). Perceptrons. Cambridge, MA: MIT Press.
Minton, S., Carbonell, J. G., Knoblock, C. A., Kuokka, D. R., Etzioni, O., and Gil, Y . (1989).
Explanation-based learning: A problem solving perspective. Artiﬁcial Intelligence, 40(1):63–118.
Mitchell, T. (1997). Machine Learning. McGraw Hill.
Mitchell, T. M., Keller, R. M., and Kedar-Cabelli, S. T. (1986). Explanation-based generalization: A
unifying view. Machine Learning, 1(1):47–80.
Mnih, V ., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M.
(Dec 2013). Playing Atari with deep reinforcement learning. Technical Report arXiv:1312.5602
[cs.LG], Deepmind Technologies.
Mohamed, A. and Hinton, G. E. (2010). Phone recognition using restricted Boltzmann machines.
In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages
4354–4357.
Molgedey, L. and Schuster, H. G. (1994). Separation of independent signals using time-delayed
correlations. Phys. Reviews Letters, 72(23):3634–3637.
Møller, M. F. (1993). Exact calculation of the product of the Hessian matrix of feed-forward network
error functions and a vector in O(N) time. Technical Report PB-432, Computer Science Depart-
ment, Aarhus University, Denmark.
Montana, D. J. and Davis, L. (1989). Training feedforward neural networks using genetic algorithms.
In Proceedings of the 11th International Joint Conference on Artiﬁcial Intelligence (IJCAI) - Vol-
ume 1, IJCAI’89, pages 762–767, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Montavon, G., Orr, G., and M¨uller, K. (2012). Neural Networks: Tricks of the Trade. Number LNCS
7700 in Lecture Notes in Computer Science Series. Springer Verlag.
66
Moody, J. E. (1989). Fast learning in multi-resolution hierarchies. In Touretzky, D. S., editor, Ad-
vances in Neural Information Processing Systems (NIPS) 1, pages 29–39. Morgan Kaufmann.
Moody, J. E. (1992). The effective number of parameters: An analysis of generalization and regu-
larization in nonlinear learning systems. In Lippman, D. S., Moody, J. E., and Touretzky, D. S.,
editors, Advances in Neural Information Processing Systems (NIPS) 4 , pages 847–854. Morgan
Kaufmann.
Moody, J. E. and Utans, J. (1994). Architecture selection strategies for neural networks: Application
to corporate bond rating prediction. In Refenes, A. N., editor, Neural Networks in the Capital
Markets. John Wiley & Sons.
Moore, A. and Atkeson, C. (1995). The parti-game algorithm for variable resolution reinforcement
learning in multidimensional state-spaces. Machine Learning, 21(3):199–233.
Moore, A. and Atkeson, C. G. (1993). Prioritized sweeping: Reinforcement learning with less data
and less time. Machine Learning, 13:103–130.
Moriarty, D. E. (1997). Symbiotic Evolution of Neural Networks in Sequential Decision Tasks. PhD
thesis, Department of Computer Sciences, The University of Texas at Austin.
Moriarty, D. E. and Miikkulainen, R. (1996). Efﬁcient reinforcement learning through symbiotic
evolution. Machine Learning, 22:11–32.
Morimoto, J. and Doya, K. (2000). Robust reinforcement learning. In Leen, T. K., Dietterich, T. G.,
and Tresp, V ., editors,Advances in Neural Information Processing Systems (NIPS) 13, pages 1061–
1067. MIT Press.
Mosteller, F. and Tukey, J. W. (1968). Data analysis, including statistics. In Lindzey, G. and Aronson,
E., editors, Handbook of Social Psychology, Vol. 2. Addison-Wesley.
Mozer, M. C. (1989). A focused back-propagation algorithm for temporal sequence recognition.
Complex Systems, 3:349–381.
Mozer, M. C. (1991). Discovering discrete distributed representations with iterative competitive learn-
ing. In Lippmann, R. P., Moody, J. E., and Touretzky, D. S., editors,Advances in Neural Information
Processing Systems 3, pages 627–634. Morgan Kaufmann.
Mozer, M. C. (1992). Induction of multiscale temporal structure. In Lippman, D. S., Moody, J. E.,
and Touretzky, D. S., editors,Advances in Neural Information Processing Systems (NIPS) 4, pages
275–282. Morgan Kaufmann.
Mozer, M. C. and Smolensky, P. (1989). Skeletonization: A technique for trimming the fat from a
network via relevance assessment. In Touretzky, D. S., editor, Advances in Neural Information
Processing Systems (NIPS) 1, pages 107–115. Morgan Kaufmann.
Muller, U. A., Gunzinger, A., and Guggenb ¨uhl, W. (1995). Fast neural net simulation with a DSP
processor array. IEEE Transactions on Neural Networks, 6(1):203–213.
Munro, P. W. (1987). A dual back-propagation scheme for scalar reinforcement learning.Proceedings
of the Ninth Annual Conference of the Cognitive Science Society, Seattle, WA, pages 165–176.
67
Murray, A. F. and Edwards, P. J. (1993). Synaptic weight noise during MLP learning enhances fault-
tolerance, generalisation and learning trajectory. In S. J. Hanson, J. D. C. and Giles, C. L., editors,
Advances in Neural Information Processing Systems (NIPS) 5 , pages 491–498. San Mateo, CA:
Morgan Kaufmann.
Nadal, J.-P. and Parga, N. (1994). Non-linear neurons in the low noise limit: a factorial code max-
imises information transfer. Network, 5:565–581.
Nagumo, J., Arimoto, S., and Yoshizawa, S. (1962). An active pulse transmission line simulating
nerve axon. Proceedings of the IRE, 50(10):2061–2070.
Nair, V . and Hinton, G. E. (2010). Rectiﬁed linear units improve restricted Boltzmann machines. In
International Conference on Machine Learning (ICML).
Narendra, K. S. and Parthasarathy, K. (1990). Identiﬁcation and control of dynamical systems using
neural networks. Neural Networks, IEEE Transactions on, 1(1):4–27.
Narendra, K. S. and Thathatchar, M. A. L. (1974). Learning automata – a survey. IEEE Transactions
on Systems, Man, and Cybernetics, 4:323–334.
Neal, R. M. (1995). Bayesian learning for neural networks. PhD thesis, University of Toronto.
Neal, R. M. (2006). Classiﬁcation with Bayesian neural networks. In Quinonero-Candela, J., Magnini,
B., Dagan, I., and D’Alche-Buc, F., editors, Machine Learning Challenges. Evaluating Predictive
Uncertainty, Visual Object Classiﬁcation, and Recognising Textual Entailment , volume 3944 of
Lecture Notes in Computer Science, pages 28–32. Springer.
Neal, R. M. and Zhang, J. (2006). High dimensional classiﬁcation with Bayesian neural networks and
Dirichlet diffusion trees. In Guyon, I., Gunn, S., Nikravesh, M., and Zadeh, L. A., editors, Feature
Extraction: Foundations and Applications, Studies in Fuzziness and Soft Computing , pages 265–
295. Springer.
Neftci, E., Das, S., Pedroni, B., Kreutz-Delgado, K., and Cauwenberghs, G. (2014). Event-driven
contrastive divergence for spiking neuromorphic systems. Frontiers in Neuroscience, 7(272).
Neil, D. and Liu, S.-C. (2014). Minitaur, an event-driven FPGA-based spiking network accelerator.
IEEE Transactions on Very Large Scale Integration (VLSI) Systems, PP(99):1–8.
Nessler, B., Pfeiffer, M., Buesing, L., and Maass, W. (2013). Bayesian computation emerges in
generic cortical microcircuits through spike-timing-dependent plasticity. PLoS Computational Bi-
ology, 9(4):e1003037.
Neti, C., Schneider, M. H., and Young, E. D. (1992). Maximally fault tolerant neural networks. In
IEEE Transactions on Neural Networks, volume 3, pages 14–23.
Neuneier, R. and Zimmermann, H.-G. (1996). How to train neural networks. In Orr, G. B. and M¨uller,
K.-R., editors, Neural Networks: Tricks of the Trade , volume 1524 of Lecture Notes in Computer
Science, pages 373–423. Springer.
Newton, I. (1687). Philosophiae naturalis principia mathematica . William Dawson & Sons Ltd.,
London.
Nguyen, N. and Widrow, B. (1989). The truck backer-upper: An example of self learning in neural
networks. In Proceedings of the International Joint Conference on Neural Networks , pages 357–
363. IEEE Press.
68
Nilsson, N. J. (1980). Principles of artiﬁcial intelligence . Morgan Kaufmann, San Francisco, CA,
USA.
Nolﬁ, S., Floreano, D., Miglino, O., and Mondada, F. (1994a). How to evolve autonomous robots:
Different approaches in evolutionary robotics. In Brooks, R. A. and Maes, P., editors, Fourth
International Workshop on the Synthesis and Simulation of Living Systems (Artiﬁcial Life IV), pages
190–197. MIT.
Nolﬁ, S., Parisi, D., and Elman, J. L. (1994b). Learning and evolution in neural networks. Adaptive
Behavior, 3(1):5–28.
Nowak, E., Jurie, F., and Triggs, B. (2006). Sampling strategies for bag-of-features image classiﬁca-
tion. In Proc. ECCV 2006, pages 490–503. Springer.
Nowlan, S. J. and Hinton, G. E. (1992). Simplifying neural networks by soft weight sharing. Neural
Computation, 4:173–193.
O’Connor, P., Neil, D., Liu, S.-C., Delbruck, T., and Pfeiffer, M. (2013). Real-time classiﬁcation and
sensor fusion with a spiking deep belief network. Frontiers in Neuroscience, 7(178).
Oh, K.-S. and Jung, K. (2004). GPU implementation of neural networks. Pattern Recognition,
37(6):1311–1314.
Oja, E. (1989). Neural networks, principal components, and subspaces. International Journal of
Neural Systems, 1(1):61–68.
Oja, E. (1991). Data compression, feature extraction, and autoassociation in feedforward neural
networks. In Kohonen, T., M ¨akisara, K., Simula, O., and Kangas, J., editors, Artiﬁcial Neural
Networks, volume 1, pages 737–745. Elsevier Science Publishers B.V ., North-Holland.
Olshausen, B. A. and Field, D. J. (1996). Emergence of simple-cell receptive ﬁeld properties by
learning a sparse code for natural images. Nature, 381(6583):607–609.
Omlin, C. and Giles, C. L. (1996). Extraction of rules from discrete-time recurrent neural networks.
Neural Networks, 9(1):41–52.
Oquab, M., Bottou, L., Laptev, I., and Sivic, J. (2013). Learning and transferring mid-level image
representations using convolutional neural networks. Technical Report hal-00911179.
O’Reilly, R. (2003). Making working memory work: A computational model of learning in the
prefrontal cortex and basal ganglia. Technical Report ICS-03-03, ICS.
O’Reilly, R. C. (1996). Biologically plausible error-driven learning using local activation differences:
The generalized recirculation algorithm. Neural Computation, 8(5):895–938.
Orr, G. and M¨uller, K. (1998). Neural Networks: Tricks of the Trade. Number LNCS 1524 in Lecture
Notes in Computer Science Series. Springer Verlag.
Ostrovskii, G. M., V olin, Y . M., and Borisov, W. W. (1971).¨Uber die Berechnung von Ableitungen.
Wiss. Z. Tech. Hochschule f¨ur Chemie, 13:382–384.
Otsuka, M. (2010). Goal-Oriented Representation of the External World: A Free-Energy-Based Ap-
proach. PhD thesis, Nara Institute of Science and Technology.
69
Otsuka, M., Yoshimoto, J., and Doya, K. (2010). Free-energy-based reinforcement learning in a
partially observable environment. In Proc. ESANN.
Otte, S., Krechel, D., Liwicki, M., and Dengel, A. (2012). Local feature based online mode detection
with recurrent neural networks. In Proceedings of the 2012 International Conference on Frontiers
in Handwriting Recognition, pages 533–537. IEEE Computer Society.
Oudeyer, P.-Y ., Baranes, A., and Kaplan, F. (2013). Intrinsically motivated learning of real world
sensorimotor skills with developmental constraints. In Baldassarre, G. and Mirolli, M., editors,
Intrinsically Motivated Learning in Natural and Artiﬁcial Systems. Springer.
OReilly, R. C., Wyatte, D., Herd, S., Mingus, B., and Jilk, D. J. (2013). Recurrent processing during
object recognition. Frontiers in Psychology, 4:124.
Pachitariu, M. and Sahani, M. (2013). Regularization and nonlinearities for neural language models:
when are they needed? arXiv preprint arXiv:1301.5650.
Palm, G. (1980). On associative memory. Biological Cybernetics, 36.
Palm, G. (1992). On the information storage capacity of local learning rules. Neural Computation,
4(2):703–711.
Pan, S. J. and Yang, Q. (2010). A survey on transfer learning. IEEE Transactions on Knowledge and
Data Engineering, 22(10):1345–1359.
Parekh, R., Yang, J., and Honavar, V . (2000). Constructive neural network learning algorithms for
multi-category pattern classiﬁcation. IEEE Transactions on Neural Networks, 11(2):436–451.
Parker, D. B. (1985). Learning-logic. Technical Report TR-47, Center for Comp. Research in Eco-
nomics and Management Sci., MIT.
Pascanu, R., Gulcehre, C., Cho, K., and Bengio, Y . (2013a). How to construct deep recurrent neural
networks. arXiv preprint arXiv:1312.6026.
Pascanu, R., Mikolov, T., and Bengio, Y . (2013b). On the difﬁculty of training recurrent neural
networks. In ICML’13: JMLR: W&CP volume 28.
Pasemann, F., Steinmetz, U., and Dieckman, U. (1999). Evolving structure and function of neurocon-
trollers. In Angeline, P. J., Michalewicz, Z., Schoenauer, M., Yao, X., and Zalzala, A., editors,Pro-
ceedings of the Congress on Evolutionary Computation , volume 3, pages 1973–1978, Mayﬂower
Hotel, Washington D.C., USA. IEEE Press.
Pearlmutter, B. A. (1989). Learning state space trajectories in recurrent neural networks. Neural
Computation, 1(2):263–269.
Pearlmutter, B. A. (1994). Fast exact multiplication by the Hessian. Neural Computation, 6(1):147–
160.
Pearlmutter, B. A. (1995). Gradient calculations for dynamic recurrent neural networks: A survey.
IEEE Transactions on Neural Networks, 6(5):1212–1228.
Pearlmutter, B. A. and Hinton, G. E. (1986). G-maximization: An unsupervised learning procedure
for discovering regularities. In Denker, J. S., editor, Neural Networks for Computing: American
Institute of Physics Conference Proceedings 151, volume 2, pages 333–338.
70
Peng, J. and Williams, R. J. (1996). Incremental multi-step Q-learning. Machine Learning, 22:283–
290.
P´erez-Ortiz, J. A., Gers, F. A., Eck, D., and Schmidhuber, J. (2003). Kalman ﬁlters improve LSTM
network performance in problems unsolvable by traditional recurrent nets. Neural Networks ,
(16):241–250.
Perrett, D., Hietanen, J., Oram, M., Benson, P., and Rolls, E. (1992). Organization and functions of
cells responsive to faces in the temporal cortex [and discussion]. Philosophical Transactions of the
Royal Society of London. Series B: Biological Sciences, 335(1273):23–30.
Perrett, D., Rolls, E., and Caan, W. (1982). Visual neurones responsive to faces in the monkey
temporal cortex. Experimental Brain Research, 47(3):329–342.
Peters, J. (2010). Policy gradient methods. Scholarpedia, 5(11):3698.
Peters, J. and Schaal, S. (2008a). Natural actor-critic. Neurocomputing, 71:1180–1190.
Peters, J. and Schaal, S. (2008b). Reinforcement learning of motor skills with policy gradients.Neural
Network, 21(4):682–697.
Pham, V ., Kermorvant, C., and Louradour, J. (2013). Dropout Improves Recurrent Neural Networks
for Handwriting Recognition. arXiv preprint arXiv:1312.4569.
Pineda, F. J. (1987). Generalization of back-propagation to recurrent neural networks. Physical
Review Letters, 19(59):2229–2232.
Plate, T. A. (1993). Holographic recurrent networks. In S. J. Hanson, J. D. C. and Giles, C. L., editors,
Advances in Neural Information Processing Systems (NIPS) 5, pages 34–41. Morgan Kaufmann.
Plumbley, M. D. (1991). On information theory and unsupervised neural networks. Dissertation,
published as technical report CUED/F-INFENG/TR.78, Engineering Department, Cambridge Uni-
versity.
Pollack, J. B. (1988). Implications of recursive distributed representations. In Proc. NIPS, pages
527–536.
Pollack, J. B. (1990). Recursive distributed representation. Artiﬁcial Intelligence, 46:77–105.
Pontryagin, L. S., Boltyanskii, V . G., Gamrelidze, R. V ., and Mishchenko, E. F. (1961).The Mathe-
matical Theory of Optimal Processes.
Poon, H. and Domingos, P. (2011). Sum-product networks: A new deep architecture. In IEEE
International Conference on Computer Vision (ICCV) Workshops, pages 689–690. IEEE.
Post, E. L. (1936). Finite combinatory processes-formulation 1. The Journal of Symbolic Logic ,
1(3):103–105.
Prasoon, A., Petersen, K., Igel, C., Lauze, F., Dam, E., and Nielsen, M. (2013). V oxel classiﬁcation
based on triplanar convolutional neural networks applied to cartilage segmentation in knee MRI. In
Medical Image Computing and Computer Assisted Intervention (MICCAI), volume 8150 of LNCS,
pages 246–253. Springer.
Precup, D., Sutton, R. S., and Singh, S. (1998). Multi-time models for temporally abstract planning. In
Advances in Neural Information Processing Systems (NIPS), pages 1050–1056. Morgan Kaufmann.
71
Prokhorov, D. (2010). A convolutional learning system for object classiﬁcation in 3-D LIDAR data.
IEEE Transactions on Neural Networks, 21(5):858–863.
Prokhorov, D., Puskorius, G., and Feldkamp, L. (2001). Dynamical neural networks for control. In
Kolen, J. and Kremer, S., editors, A ﬁeld guide to dynamical recurrent networks , pages 23–78.
IEEE Press.
Prokhorov, D. and Wunsch, D. (1997). Adaptive critic design. IEEE Transactions on Neural Net-
works, 8(5):997–1007.
Prokhorov, D. V ., Feldkamp, L. A., and Tyukin, I. Y . (2002). Adaptive behavior with ﬁxed weights in
RNN: an overview. InProceedings of the IEEE International Joint Conference on Neural Networks
(IJCNN), pages 2018–2023.
Puskorius, G. V . and Feldkamp, L. A. (1994). Neurocontrol of nonlinear dynamical systems with
Kalman ﬁlter trained recurrent networks. IEEE Transactions on Neural Networks, 5(2):279–297.
Raiko, T., Valpola, H., and LeCun, Y . (2012). Deep learning made easier by linear transformations in
perceptrons. In International Conference on Artiﬁcial Intelligence and Statistics, pages 924–932.
Raina, R., Madhavan, A., and Ng, A. (2009). Large-scale deep unsupervised learning using graphics
processors. In Proceedings of the 26th Annual International Conference on Machine Learning
(ICML), pages 873–880. ACM.
Ramacher, U., Raab, W., Anlauf, J., Hachmann, U., Beichter, J., Bruels, N., Wesseling, M., Sich-
eneder, E., Maenner, R., Glaess, J., and Wurz, A. (1993). Multiprocessor and memory architecture
of the neurocomputer SYNAPSE-1. International Journal of Neural Systems, 4(4):333–336.
Ranzato, M., Poultney, C., Chopra, S., and LeCun, Y . (2006). Efﬁcient learning of sparse representa-
tions with an energy-based model. In et al., J. P., editor,Advances in Neural Information Processing
Systems (NIPS 2006). MIT Press.
Ranzato, M. A., Huang, F., Boureau, Y ., and LeCun, Y . (2007). Unsupervised learning of invariant
feature hierarchies with applications to object recognition. In Proc. Computer Vision and Pattern
Recognition Conference (CVPR’07), pages 1–8. IEEE Press.
Rauber, A., Merkl, D., and Dittenbach, M. (2002). The growing hierarchical self-organizing map: ex-
ploratory analysis of high-dimensional data. IEEE Transactions on Neural Networks, 13(6):1331–
1341.
Razavian, A. S., Azizpour, H., Sullivan, J., and Carlsson, S. (2014). CNN features off-the-shelf: an
astounding baseline for recognition. arXiv preprint arXiv:1403.6382.
Rechenberg, I. (1971). Evolutionsstrategie - Optimierung technischer Systeme nach Prinzipien der
biologischen Evolution. Dissertation. Published 1973 by Fromman-Holzboog.
Redlich, A. N. (1993). Redundancy reduction as a strategy for unsupervised learning. Neural Com-
putation, 5:289–304.
Refenes, N. A., Zapranis, A., and Francis, G. (1994). Stock performance modeling using neural
networks: a comparative study with regression models. Neural Networks, 7(2):375–388.
Rezende, D. J. and Gerstner, W. (2014). Stochastic variational learning in recurrent spiking networks.
Frontiers in Computational Neuroscience, 8:38.
72
Riedmiller, M. (2005). Neural ﬁtted Q iteration—ﬁrst experiences with a data efﬁcient neural rein-
forcement learning method. In Proc. ECML-2005, pages 317–328. Springer-Verlag Berlin Heidel-
berg.
Riedmiller, M. and Braun, H. (1993). A direct adaptive method for faster backpropagation learning:
The Rprop algorithm. In Proc. IJCNN, pages 586–591. IEEE Press.
Riedmiller, M., Lange, S., and V oigtlaender, A. (2012). Autonomous reinforcement learning on raw
visual input data in a real world application. In International Joint Conference on Neural Networks
(IJCNN), pages 1–8, Brisbane, Australia.
Riesenhuber, M. and Poggio, T. (1999). Hierarchical models of object recognition in cortex. Nat.
Neurosci., 2(11):1019–1025.
Rifai, S., Vincent, P., Muller, X., Glorot, X., and Bengio, Y . (2011). Contractive auto-encoders:
Explicit invariance during feature extraction. In Proceedings of the 28th International Conference
on Machine Learning (ICML-11), pages 833–840.
Ring, M., Schaul, T., and Schmidhuber, J. (2011). The two-dimensional organization of behavior. In
Proceedings of the First Joint Conference on Development Learning and on Epigenetic Robotics
ICDL-EPIROB, Frankfurt.
Ring, M. B. (1991). Incremental development of complex behaviors through automatic construc-
tion of sensory-motor hierarchies. In Birnbaum, L. and Collins, G., editors, Machine Learning:
Proceedings of the Eighth International Workshop, pages 343–347. Morgan Kaufmann.
Ring, M. B. (1993). Learning sequential tasks by incrementally adding higher orders. In S. J. Hanson,
J. D. C. and Giles, C. L., editors,Advances in Neural Information Processing Systems 5, pages 115–
122. Morgan Kaufmann.
Ring, M. B. (1994). Continual Learning in Reinforcement Environments. PhD thesis, University of
Texas at Austin, Austin, Texas 78712.
Risi, S. and Stanley, K. O. (2012). A uniﬁed approach to evolving plasticity and neural geometry. In
International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE.
Rissanen, J. (1986). Stochastic complexity and modeling. The Annals of Statistics, 14(3):1080–1100.
Ritter, H. and Kohonen, T. (1989). Self-organizing semantic maps. Biological Cybernetics ,
61(4):241–254.
Robinson, A. J. and Fallside, F. (1987). The utility driven dynamic error propagation network. Tech-
nical Report CUED/F-INFENG/TR.1, Cambridge University Engineering Department.
Robinson, T. and Fallside, F. (1989). Dynamic reinforcement driven error propagation networks
with application to game playing. In Proceedings of the 11th Conference of the Cognitive Science
Society, Ann Arbor, pages 836–843.
Rodriguez, P. and Wiles, J. (1998). Recurrent neural networks can learn to implement symbol-
sensitive counting. In Advances in Neural Information Processing Systems (NIPS) , volume 10,
pages 87–93. The MIT Press.
Rodriguez, P., Wiles, J., and Elman, J. (1999). A recurrent neural network that learns to count.
Connection Science, 11(1):5–40.
73
Roggen, D., Hofmann, S., Thoma, Y ., and Floreano, D. (2003). Hardware spiking neural network with
run-time reconﬁgurable connectivity in an autonomous robot. In Proc. NASA/DoD Conference on
Evolvable Hardware, 2003, pages 189–198. IEEE.
Rohwer, R. (1989). The ‘moving targets’ training method. In Kindermann, J. and Linden, A., edi-
tors, Proceedings of ‘Distributed Adaptive Neural Information Processing’, St.Augustin, 24.-25.5,.
Oldenbourg.
Rosenblatt, F. (1958). The perceptron: a probabilistic model for information storage and organization
in the brain. Psychological review, 65(6):386.
Rosenblatt, F. (1962). Principles of Neurodynamics. Spartan, New York.
Roux, L., Racoceanu, D., Lomenie, N., Kulikova, M., Irshad, H., Klossa, J., Capron, F., Genestie, C.,
Naour, G. L., and Gurcan, M. N. (2013). Mitosis detection in breast cancer histological images -
an ICPR 2012 contest. J. Pathol. Inform., 4:8.
Rubner, J. and Schulten, K. (1990). Development of feature detectors by self-organization: A network
model. Biological Cybernetics, 62:193–199.
R¨uckstieß, T., Felder, M., and Schmidhuber, J. (2008). State-Dependent Exploration for policy gra-
dient methods. In et al., W. D., editor, European Conference on Machine Learning (ECML) and
Principles and Practice of Knowledge Discovery in Databases 2008, Part II, LNAI 5212 , pages
234–249.
Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning internal representations by error
propagation. In Rumelhart, D. E. and McClelland, J. L., editors, Parallel Distributed Processing,
volume 1, pages 318–362. MIT Press.
Rumelhart, D. E. and Zipser, D. (1986). Feature discovery by competitive learning. In Parallel
Distributed Processing, pages 151–193. MIT Press.
Rummery, G. and Niranjan, M. (1994). On-line Q-learning using connectionist sytems. Technical
Report CUED/F-INFENG-TR 166, Cambridge University, UK.
Russell, S. J., Norvig, P., Canny, J. F., Malik, J. M., and Edwards, D. D. (1995).Artiﬁcial Intelligence:
a Modern Approach, volume 2. Englewood Cliffs: Prentice Hall.
Saito, K. and Nakano, R. (1997). Partial BFGS update and efﬁcient step-length calculation for three-
layer neural networks. Neural Computation, 9(1):123–141.
Sak, H., Senior, A., and Beaufays, F. (2014a). Long Short-Term Memory recurrent neural network
architectures for large scale acoustic modeling. In Proc. Interspeech.
Sak, H., Vinyals, O., Heigold, G., Senior, A., McDermott, E., Monga, R., and Mao, M. (2014b). Se-
quence discriminative distributed training of Long Short-Term Memory recurrent neural networks.
In Proc. Interspeech.
Salakhutdinov, R. and Hinton, G. (2009). Semantic hashing. Int. J. Approx. Reasoning, 50(7):969–
978.
Sallans, B. and Hinton, G. (2004). Reinforcement learning with factored states and actions. Journal
of Machine Learning Research, 5:1063–1088.
74
Sałustowicz, R. P. and Schmidhuber, J. (1997). Probabilistic incremental program evolution. Evolu-
tionary Computation, 5(2):123–141.
Samejima, K., Doya, K., and Kawato, M. (2003). Inter-module credit assignment in modular rein-
forcement learning. Neural Networks, 16(7):985–994.
Samuel, A. L. (1959). Some studies in machine learning using the game of checkers. IBM Journal on
Research and Development, 3:210–229.
Sanger, T. D. (1989). An optimality principle for unsupervised learning. In Touretzky, D. S., editor,
Advances in Neural Information Processing Systems (NIPS) 1, pages 11–19. Morgan Kaufmann.
Santamar´ıa, J. C., Sutton, R. S., and Ram, A. (1997). Experiments with reinforcement learning in
problems with continuous state and action spaces. Adaptive Behavior, 6(2):163–217.
Saravanan, N. and Fogel, D. B. (1995). Evolving neural control systems. IEEE Expert, pages 23–27.
Saund, E. (1994). Unsupervised learning of mixtures of multiple causes in binary data. In Cowan,
J. D., Tesauro, G., and Alspector, J., editors, Advances in Neural Information Processing Systems
(NIPS) 6, pages 27–34. Morgan Kaufmann.
Schaback, R. and Werner, H. (1992). Numerische Mathematik, volume 4. Springer.
Sch¨afer, A. M., Udluft, S., and Zimmermann, H.-G. (2006). Learning long term dependencies with
recurrent neural networks. In Kollias, S. D., Stafylopatis, A., Duch, W., and Oja, E., editors,ICANN
(1), volume 4131 of Lecture Notes in Computer Science, pages 71–80. Springer.
Schapire, R. E. (1990). The strength of weak learnability. Machine Learning, 5:197–227.
Schaul, T. and Schmidhuber, J. (2010). Metalearning. Scholarpedia, 6(5):4650.
Schaul, T., Zhang, S., and LeCun, Y . (2013). No more pesky learning rates. InProc. 30th International
Conference on Machine Learning (ICML).
Schemmel, J., Grubl, A., Meier, K., and Mueller, E. (2006). Implementing synaptic plasticity in
a VLSI spiking neural network model. In International Joint Conference on Neural Networks
(IJCNN), pages 1–6. IEEE.
Scherer, D., M¨uller, A., and Behnke, S. (2010). Evaluation of pooling operations in convolutional ar-
chitectures for object recognition. In Proc. International Conference on Artiﬁcial Neural Networks
(ICANN), pages 92–101.
Schmidhuber, J. (1987). Evolutionary principles in self-referential learning, or on learning
how to learn: the meta-meta-... hook. Diploma thesis, Inst. f. Inf., Tech. Univ. Munich.
http://www.idsia.ch/˜juergen/diploma.html.
Schmidhuber, J. (1989a). Accelerated learning in back-propagation nets. In Pfeifer, R., Schreter, Z.,
Fogelman, Z., and Steels, L., editors, Connectionism in Perspective, pages 429 – 438. Amsterdam:
Elsevier, North-Holland.
Schmidhuber, J. (1989b). A local learning algorithm for dynamic feedforward and recurrent networks.
Connection Science, 1(4):403–412.
75
Schmidhuber, J. (1990a). Dynamische neuronale Netze und das fundamentale raumzeitliche Lern-
problem. (Dynamic neural nets and the fundamental spatio-temporal credit assignment problem.)
Dissertation, Inst. f. Inf., Tech. Univ. Munich.
Schmidhuber, J. (1990b). Learning algorithms for networks with internal and external feedback. In
Touretzky, D. S., Elman, J. L., Sejnowski, T. J., and Hinton, G. E., editors, Proc. of the 1990
Connectionist Models Summer School, pages 52–61. Morgan Kaufmann.
Schmidhuber, J. (1990c). The Neural Heat Exchanger. Talks at TU Munich (1990), University
of Colorado at Boulder (1992), and Z. Li’s NIPS*94 workshop on unsupervised learning. Also
published at the Intl. Conference on Neural Information Processing (ICONIP’96), vol. 1, pages
194-197, 1996.
Schmidhuber, J. (1990d). An on-line algorithm for dynamic reinforcement learning and planning in
reactive environments. In Proc. IEEE/INNS International Joint Conference on Neural Networks,
San Diego, volume 2, pages 253–258.
Schmidhuber, J. (1991a). Curious model-building control systems. InProceedings of the International
Joint Conference on Neural Networks, Singapore, volume 2, pages 1458–1463. IEEE press.
Schmidhuber, J. (1991b). Learning to generate sub-goals for action sequences. In Kohonen, T.,
M¨akisara, K., Simula, O., and Kangas, J., editors, Artiﬁcial Neural Networks , pages 967–972.
Elsevier Science Publishers B.V ., North-Holland.
Schmidhuber, J. (1991c). Reinforcement learning in Markovian and non-Markovian environments.
In Lippman, D. S., Moody, J. E., and Touretzky, D. S., editors, Advances in Neural Information
Processing Systems 3 (NIPS 3), pages 500–506. Morgan Kaufmann.
Schmidhuber, J. (1992a). A ﬁxed size storage O(n3) time complexity learning algorithm for fully
recurrent continually running networks. Neural Computation, 4(2):243–248.
Schmidhuber, J. (1992b). Learning complex, extended sequences using the principle of history com-
pression. Neural Computation, 4(2):234–242. (Based on TR FKI-148-91, TUM, 1991).
Schmidhuber, J. (1992c). Learning factorial codes by predictability minimization. Neural Computa-
tion, 4(6):863–879.
Schmidhuber, J. (1993a). An introspective network that can learn to run its own weight change
algorithm. In Proc. of the Intl. Conf. on Artiﬁcial Neural Networks, Brighton, pages 191–195. IEE.
Schmidhuber, J. (1993b). Netzwerkarchitekturen, Zielfunktionen und Kettenregel. (Network archi-
tectures, objective functions, and chain rule.) Habilitation Thesis, Inst. f. Inf., Tech. Univ. Munich.
Schmidhuber, J. (1997). Discovering neural nets with low Kolmogorov complexity and high general-
ization capability. Neural Networks, 10(5):857–873.
Schmidhuber, J. (2002). The Speed Prior: a new simplicity measure yielding near-optimal computable
predictions. In Kivinen, J. and Sloan, R. H., editors, Proceedings of the 15th Annual Conference
on Computational Learning Theory (COLT 2002) , Lecture Notes in Artiﬁcial Intelligence, pages
216–228. Springer, Sydney, Australia.
Schmidhuber, J. (2004). Optimal ordered problem solver. Machine Learning, 54:211–254.
76
Schmidhuber, J. (2006a). Developmental robotics, optimal artiﬁcial curiosity, creativity, music, and
the ﬁne arts. Connection Science, 18(2):173–187.
Schmidhuber, J. (2006b). G ¨odel machines: Fully self-referential optimal universal self-improvers. In
Goertzel, B. and Pennachin, C., editors, Artiﬁcial General Intelligence, pages 199–226. Springer
Verlag. Variant available as arXiv:cs.LO/0309048.
Schmidhuber, J. (2007). Prototype resilient, self-modeling robots. Science, 316(5825):688.
Schmidhuber, J. (2012). Self-delimiting neural networks. Technical Report IDSIA-08-12,
arXiv:1210.0118v1 [cs.NE], The Swiss AI Lab IDSIA.
Schmidhuber, J. (2013a). My ﬁrst Deep Learning system of 1991 + Deep Learning timeline 1962-
2013. Technical Report arXiv:1312.5548v1 [cs.NE], The Swiss AI Lab IDSIA.
Schmidhuber, J. (2013b). P OWER PLAY: Training an Increasingly General Problem Solver by Con-
tinually Searching for the Simplest Still Unsolvable Problem. Frontiers in Psychology.
Schmidhuber, J., Ciresan, D., Meier, U., Masci, J., and Graves, A. (2011). On fast deep nets for AGI
vision. In Proc. Fourth Conference on Artiﬁcial General Intelligence (AGI), Google, Mountain
View, CA, pages 243–246.
Schmidhuber, J., Eldracher, M., and Foltin, B. (1996). Semilinear predictability minimization pro-
duces well-known feature detectors. Neural Computation, 8(4):773–786.
Schmidhuber, J. and Huber, R. (1991). Learning to generate artiﬁcial fovea trajectories for target
detection. International Journal of Neural Systems, 2(1 & 2):135–141.
Schmidhuber, J., Mozer, M. C., and Prelinger, D. (1993). Continuous history compression. In H¨uning,
H., Neuhauser, S., Raus, M., and Ritschel, W., editors,Proc. of Intl. Workshop on Neural Networks,
RWTH Aachen, pages 87–95. Augustinus.
Schmidhuber, J. and Prelinger, D. (1992). Discovering predictable classiﬁcations. Technical Report
CU-CS-626-92, Dept. of Comp. Sci., University of Colorado at Boulder. Published in Neural
Computation 5(4):625-635 (1993).
Schmidhuber, J. and Wahnsiedler, R. (1992). Planning simple trajectories using neural subgoal gen-
erators. In Meyer, J. A., Roitblat, H. L., and Wilson, S. W., editors, Proc. of the 2nd International
Conference on Simulation of Adaptive Behavior, pages 196–202. MIT Press.
Schmidhuber, J., Wierstra, D., Gagliolo, M., and Gomez, F. J. (2007). Training recurrent networks by
Evolino. Neural Computation, 19(3):757–779.
Schmidhuber, J., Zhao, J., and Schraudolph, N. (1997a). Reinforcement learning with self-modifying
policies. In Thrun, S. and Pratt, L., editors, Learning to learn, pages 293–309. Kluwer.
Schmidhuber, J., Zhao, J., and Wiering, M. (1997b). Shifting inductive bias with success-story algo-
rithm, adaptive Levin search, and incremental self-improvement. Machine Learning, 28:105–130.
Sch¨olkopf, B., Burges, C. J. C., and Smola, A. J., editors (1998). Advances in Kernel Methods -
Support Vector Learning. MIT Press, Cambridge, MA.
77
Schraudolph, N. and Sejnowski, T. J. (1993). Unsupervised discrimination of clustered data via op-
timization of binary information gain. In Hanson, S. J., Cowan, J. D., and Giles, C. L., editors,
Advances in Neural Information Processing Systems , volume 5, pages 499–506. Morgan Kauf-
mann, San Mateo.
Schraudolph, N. N. (2002). Fast curvature matrix-vector products for second-order gradient descent.
Neural Computation, 14(7):1723–1738.
Schraudolph, N. N. and Sejnowski, T. J. (1996). Tempering backpropagation networks: Not all
weights are created equal. In Touretzky, D. S., Mozer, M. C., and Hasselmo, M. E., editors, Ad-
vances in Neural Information Processing Systems (NIPS) , volume 8, pages 563–569. The MIT
Press, Cambridge, MA.
Schrauwen, B., Verstraeten, D., and Van Campenhout, J. (2007). An overview of reservoir computing:
theory, applications and implementations. In Proceedings of the 15th European Symposium on
Artiﬁcial Neural Networks. p. 471-482 2007, pages 471–482.
Schuster, H. G. (1992). Learning by maximization the information transfer through nonlinear noisy
neurons and “noise breakdown”. Phys. Rev. A, 46(4):2131–2138.
Schuster, M. (1999). On supervised learning from sequential data with applications for speech recog-
nition. PhD thesis, Nara Institute of Science and Technolog, Kyoto, Japan.
Schuster, M. and Paliwal, K. K. (1997). Bidirectional recurrent neural networks. IEEE Transactions
on Signal Processing, 45:2673–2681.
Schwartz, A. (1993). A reinforcement learning method for maximizing undiscounted rewards. In
Proc. ICML, pages 298–305.
Schwefel, H. P. (1974). Numerische Optimierung von Computer-Modellen. Dissertation. Published
1977 by Birkh¨auser, Basel.
Segmentation of Neuronal Structures in EM Stacks Challenge (2012). IEEE International Symposium
on Biomedical Imaging (ISBI), http://tinyurl.com/d2fgh7g.
Sehnke, F., Osendorfer, C., R ¨uckstieß, T., Graves, A., Peters, J., and Schmidhuber, J. (2010).
Parameter-exploring policy gradients. Neural Networks, 23(4):551–559.
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and LeCun, Y . (2013). OverFeat:
Integrated recognition, localization and detection using convolutional networks. arXiv preprint
arXiv:1312.6229.
Sermanet, P. and LeCun, Y . (2011). Trafﬁc sign recognition with multi-scale convolutional networks.
In Proceedings of International Joint Conference on Neural Networks (IJCNN’11) , pages 2809–
2813.
Serrano-Gotarredona, R., Oster, M., Lichtsteiner, P., Linares-Barranco, A., Paz-Vicente, R., G ´omez-
Rodr´ıguez, F., Camu˜nas-Mesa, L., Berner, R., Rivas-P´erez, M., Delbruck, T., et al. (2009). Caviar:
A 45k neuron, 5m synapse, 12g connects/s AER hardware sensory–processing–learning–actuating
system for high-speed visual object recognition and tracking. IEEE Transactions on Neural Net-
works, 20(9):1417–1438.
78
Serre, T., Riesenhuber, M., Louie, J., and Poggio, T. (2002). On the role of object-speciﬁc features
for real world object recognition in biological vision. In Biologically Motivated Computer Vision,
pages 387–397.
Seung, H. S. (2003). Learning in spiking neural networks by reinforcement of stochastic synaptic
transmission. Neuron, 40(6):1063–1073.
Shan, H. and Cottrell, G. (2014). Efﬁcient visual coding: From retina to V2. In Proc. International
Conference on Learning Representations (ICLR). arXiv preprint arXiv:1312.6077.
Shan, H., Zhang, L., and Cottrell, G. W. (2007). Recursive ICA. Advances in Neural Information
Processing Systems (NIPS), 19:1273.
Shanno, D. F. (1970). Conditioning of quasi-Newton methods for function minimization. Mathemat-
ics of computation, 24(111):647–656.
Shannon, C. E. (1948). A mathematical theory of communication (parts I and II). Bell System
Technical Journal, XXVII:379–423.
Shao, L., Wu, D., and Li, X. (2014). Learning deep and wide: A spectral method for learning deep
networks. IEEE Transactions on Neural Networks and Learning Systems.
Shavlik, J. W. (1994). Combining symbolic and neural learning. Machine Learning, 14(3):321–331.
Shavlik, J. W. and Towell, G. G. (1989). Combining explanation-based and neural learning: An
algorithm and empirical results. Connection Science, 1(3):233–255.
Siegelmann, H. (1992). Theoretical Foundations of Recurrent Neural Networks. PhD thesis, Rutgers,
New Brunswick Rutgers, The State of New Jersey.
Siegelmann, H. T. and Sontag, E. D. (1991). Turing computability with neural nets. Applied Mathe-
matics Letters, 4(6):77–80.
Silva, F. M. and Almeida, L. B. (1990). Speeding up back-propagation. In Eckmiller, R., editor,
Advanced Neural Computers, pages 151–158, Amsterdam. Elsevier.
S´ıma, J. (1994). Loading deep networks is hard. Neural Computation, 6(5):842–850.
S´ıma, J. (2002). Training a single sigmoidal neuron is hard.Neural Computation, 14(11):2709–2728.
Simard, P., Steinkraus, D., and Platt, J. (2003). Best practices for convolutional neural networks
applied to visual document analysis. In Seventh International Conference on Document Analysis
and Recognition, pages 958–963.
Sims, K. (1994). Evolving virtual creatures. In Glassner, A., editor, Proceedings of SIGGRAPH ’94
(Orlando, Florida, July 1994), Computer Graphics Proceedings, Annual Conference, pages 15–22.
ACM SIGGRAPH, ACM Press. ISBN 0-89791-667-0.
Simsek, ¨O. and Barto, A. G. (2008). Skill characterization based on betweenness. In NIPS’08, pages
1497–1504.
Singh, S., Barto, A. G., and Chentanez, N. (2005). Intrinsically motivated reinforcement learning. In
Advances in Neural Information Processing Systems 17 (NIPS). MIT Press, Cambridge, MA.
79
Singh, S. P. (1994). Reinforcement learning algorithms for average-payoff Markovian decision pro-
cesses. In National Conference on Artiﬁcial Intelligence, pages 700–705.
Smith, S. F. (1980). A Learning System Based on Genetic Adaptive Algorithms, . PhD thesis, Univ.
Pittsburgh.
Smolensky, P. (1986). Parallel distributed processing: Explorations in the microstructure of cognition,
vol. 1. chapter Information Processing in Dynamical Systems: Foundations of Harmony Theory,
pages 194–281. MIT Press, Cambridge, MA, USA.
Solla, S. A. (1988). Accelerated learning in layered neural networks. Complex Systems, 2:625–640.
Solomonoff, R. J. (1964). A formal theory of inductive inference. Part I. Information and Control,
7:1–22.
Solomonoff, R. J. (1978). Complexity-based induction systems. IEEE Transactions on Information
Theory, IT-24(5):422–432.
Soloway, E. (1986). Learning to program = learning to construct mechanisms and explanations.
Communications of the ACM, 29(9):850–858.
Song, S., Miller, K. D., and Abbott, L. F. (2000). Competitive Hebbian learning through spike-timing-
dependent synaptic plasticity. Nature Neuroscience, 3(9):919–926.
Speelpenning, B. (1980). Compiling Fast Partial Derivatives of Functions Given by Algorithms. PhD
thesis, Department of Computer Science, University of Illinois, Urbana-Champaign.
Srivastava, R. K., Masci, J., Kazerounian, S., Gomez, F., and Schmidhuber, J. (2013). Compete to
compute. In Advances in Neural Information Processing Systems (NIPS), pages 2310–2318.
Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. (2011). The German trafﬁc sign recognition
benchmark: A multi-class classiﬁcation competition. In International Joint Conference on Neural
Networks (IJCNN 2011), pages 1453–1460. IEEE Press.
Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. (2012). Man vs. computer: Benchmarking
machine learning algorithms for trafﬁc sign recognition. Neural Networks, 32:323–332.
Stanley, K. O., D’Ambrosio, D. B., and Gauci, J. (2009). A hypercube-based encoding for evolving
large-scale neural networks. Artiﬁcial Life, 15(2):185–212.
Stanley, K. O. and Miikkulainen, R. (2002). Evolving neural networks through augmenting topolo-
gies. Evolutionary Computation, 10:99–127.
Steijvers, M. and Grunwald, P. (1996). A recurrent network that performs a contextsensitive prediction
task. In Proceedings of the 18th Annual Conference of the Cognitive Science Society. Erlbaum.
Steil, J. J. (2007). Online reservoir adaptation by intrinsic plasticity for backpropagation–
decorrelation and echo state learning. Neural Networks, 20(3):353–364.
Stemmler, M. (1996). A single spike sufﬁces: the simplest form of stochastic resonance in model
neurons. Network: Computation in Neural Systems, 7(4):687–716.
Stoianov, I. and Zorzi, M. (2012). Emergence of a ’visual number sense’ in hierarchical generative
models. Nature Neuroscience, 15(2):194–6.
80
Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Roy. Stat. Soc.,
36:111–147.
Stoop, R., Schindler, K., and Bunimovich, L. (2000). When pyramidal neurons lock, when they
respond chaotically, and when they like to synchronize. Neuroscience research, 36(1):81–91.
Stratonovich, R. (1960). Conditional Markov processes. Theory of Probability And Its Applications,
5(2):156–178.
Sun, G., Chen, H., and Lee, Y . (1993a). Time warping invariant neural networks. In S. J. Hanson,
J. D. C. and Giles, C. L., editors, Advances in Neural Information Processing Systems (NIPS) 5 ,
pages 180–187. Morgan Kaufmann.
Sun, G. Z., Giles, C. L., Chen, H. H., and Lee, Y . C. (1993b). The neural network pushdown au-
tomaton: Model, stack and learning simulations. Technical Report CS-TR-3118, University of
Maryland, College Park.
Sun, Y ., Gomez, F., Schaul, T., and Schmidhuber, J. (2013). A Linear Time Natural Evolution Strat-
egy for Non-Separable Functions. In Proceedings of the Genetic and Evolutionary Computation
Conference, page 61, Amsterdam, NL. ACM.
Sun, Y ., Wierstra, D., Schaul, T., and Schmidhuber, J. (2009). Efﬁcient natural evolution strategies.
In Proc. 11th Genetic and Evolutionary Computation Conference (GECCO), pages 539–546.
Sutskever, I., Hinton, G. E., and Taylor, G. W. (2008). The recurrent temporal restricted Boltzmann
machine. In NIPS, volume 21, page 2008.
Sutskever, I., Vinyals, O., and Le, Q. V . (2014). Sequence to sequence learning with neural networks.
Technical Report arXiv:1409.3215 [cs.CL], Google. NIPS’2014.
Sutton, R. and Barto, A. (1998). Reinforcement learning: An introduction . Cambridge, MA, MIT
Press.
Sutton, R. S., McAllester, D. A., Singh, S. P., and Mansour, Y . (1999a). Policy gradient methods for
reinforcement learning with function approximation. In Advances in Neural Information Process-
ing Systems (NIPS) 12, pages 1057–1063.
Sutton, R. S., Precup, D., and Singh, S. P. (1999b). Between MDPs and semi-MDPs: A framework
for temporal abstraction in reinforcement learning. Artif. Intell., 112(1-2):181–211.
Sutton, R. S., Szepesv ´ari, C., and Maei, H. R. (2008). A convergent O(n) algorithm for off-policy
temporal-difference learning with linear function approximation. In Advances in Neural Informa-
tion Processing Systems (NIPS’08), volume 21, pages 1609–1616.
Szab´o, Z., P ´oczos, B., and L ˝orincz, A. (2006). Cross-entropy optimization for independent pro-
cess analysis. In Independent Component Analysis and Blind Signal Separation , pages 909–916.
Springer.
Szegedy, C., Liu, W., Jia, Y ., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V ., and Ra-
binovich, A. (2014). Going deeper with convolutions. Technical Report arXiv:1409.4842 [cs.CV],
Google.
Szegedy, C., Toshev, A., and Erhan, D. (2013). Deep neural networks for object detection. pages
2553–2561.
81
Taylor, G. W., Spiro, I., Bregler, C., and Fergus, R. (2011). Learning invariance through imitation. In
Conference on Computer Vision and Pattern Recognition (CVPR), pages 2729–2736. IEEE.
Tegge, A. N., Wang, Z., Eickholt, J., and Cheng, J. (2009). NNcon: improved protein contact map
prediction using 2D-recursive neural networks.Nucleic Acids Research, 37(Suppl 2):W515–W518.
Teichmann, M., Wiltschut, J., and Hamker, F. (2012). Learning invariance from natural images in-
spired by observations in the primary visual cortex. Neural Computation, 24(5):1271–1296.
Teller, A. (1994). The evolution of mental models. In Kenneth E. Kinnear, J., editor, Advances in
Genetic Programming, pages 199–219. MIT Press.
Tenenberg, J., Karlsson, J., and Whitehead, S. (1993). Learning via task decomposition. In Meyer,
J. A., Roitblat, H., and Wilson, S., editors, From Animals to Animats 2: Proceedings of the Second
International Conference on Simulation of Adaptive Behavior, pages 337–343. MIT Press.
Tesauro, G. (1994). TD-gammon, a self-teaching backgammon program, achieves master-level play.
Neural Computation, 6(2):215–219.
Tieleman, T. and Hinton, G. (2012). Lecture 6.5—RmsProp: Divide the gradient by a running average
of its recent magnitude. COURSERA: Neural Networks for Machine Learning.
Tikhonov, A. N., Arsenin, V . I., and John, F. (1977).Solutions of ill-posed problems. Winston.
Ting, K. M. and Witten, I. H. (1997). Stacked generalization: when does it work? In in Proc.
International Joint Conference on Artiﬁcial Intelligence (IJCAI).
Tiˇno, P. and Hammer, B. (2004). Architectural bias in recurrent neural networks: Fractal analysis.
Neural Computation, 15(8):1931–1957.
Tonkes, B. and Wiles, J. (1997). Learning a context-free task with a recurrent neural network: An
analysis of stability. InProceedings of the Fourth Biennial Conference of the Australasian Cognitive
Science Society.
Towell, G. G. and Shavlik, J. W. (1994). Knowledge-based artiﬁcial neural networks. Artiﬁcial
Intelligence, 70(1):119–165.
Tsitsiklis, J. N. and van Roy, B. (1996). Feature-based methods for large scale dynamic programming.
Machine Learning, 22(1-3):59–94.
Tsodyks, M., Pawelzik, K., and Markram, H. (1998). Neural networks with dynamic synapses.Neural
Computation, 10(4):821–835.
Tsodyks, M. V ., Skaggs, W. E., Sejnowski, T. J., and McNaughton, B. L. (1996). Population dynam-
ics and theta rhythm phase precession of hippocampal place cell ﬁring: a spiking neuron model.
Hippocampus, 6(3):271–280.
Turaga, S. C., Murray, J. F., Jain, V ., Roth, F., Helmstaedter, M., Briggman, K., Denk, W., and Seung,
H. S. (2010). Convolutional networks can learn to generate afﬁnity graphs for image segmentation.
Neural Computation, 22(2):511–538.
Turing, A. M. (1936). On computable numbers, with an application to the Entscheidungsproblem.
Proceedings of the London Mathematical Society, Series 2, 41:230–267.
82
Turner, A. J. and Miller, J. F. (2013). Cartesian Genetic Programming encoded artiﬁcial neural net-
works: A comparison using three benchmarks. In Proceedings of the Conference on Genetic and
Evolutionary Computation (GECCO), pages 1005–1012.
Ueda, N. (2000). Optimal linear combination of neural networks for improving classiﬁcation perfor-
mance. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(2):207–215.
Urlbe, A. P. (1999). Structure-adaptable digital neural networks. PhD thesis, Universidad del Valle.
Utgoff, P. E. and Stracuzzi, D. J. (2002). Many-layered learning. Neural Computation, 14(10):2497–
2529.
Vahed, A. and Omlin, C. W. (2004). A machine learning method for extracting symbolic knowledge
from recurrent neural networks. Neural Computation, 16(1):59–71.
Vaillant, R., Monrocq, C., and LeCun, Y . (1994). Original approach for the localisation of objects in
images. IEE Proc on Vision, Image, and Signal Processing, 141(4):245–250.
van den Berg, T. and Whiteson, S. (2013). Critical factors in the performance of HyperNEAT. In
GECCO 2013: Proceedings of the Genetic and Evolutionary Computation Conference, pages 759–
766.
van Hasselt, H. (2012). Reinforcement learning in continuous state and action spaces. In Wiering, M.
and van Otterlo, M., editors, Reinforcement Learning, pages 207–251. Springer.
Vapnik, V . (1992). Principles of risk minimization for learning theory. In Lippman, D. S., Moody,
J. E., and Touretzky, D. S., editors, Advances in Neural Information Processing Systems (NIPS) 4,
pages 831–838. Morgan Kaufmann.
Vapnik, V . (1995).The Nature of Statistical Learning Theory. Springer, New York.
Versino, C. and Gambardella, L. M. (1996). Learning ﬁne motion by using the hierarchical ex-
tended Kohonen map. In Proc. Intl. Conf. on Artiﬁcial Neural Networks (ICANN), pages 221–226.
Springer.
Veta, M., Viergever, M., Pluim, J., Stathonikos, N., and van Diest, P. J. (2013). MICCAI 2013 Grand
Challenge on Mitosis Detection.
Vieira, A. and Barradas, N. (2003). A training algorithm for classiﬁcation of high-dimensional data.
Neurocomputing, 50:461–472.
Viglione, S. (1970). Applications of pattern recognition technology. In Mendel, J. M. and Fu, K. S.,
editors, Adaptive, Learning, and Pattern Recognition Systems. Academic Press.
Vincent, P., Hugo, L., Bengio, Y ., and Manzagol, P.-A. (2008). Extracting and composing robust fea-
tures with denoising autoencoders. In Proceedings of the 25th international conference on Machine
learning, ICML ’08, pages 1096–1103, New York, NY , USA. ACM.
Vlassis, N., Littman, M. L., and Barber, D. (2012). On the computational complexity of stochastic
controller optimization in POMDPs. ACM Transactions on Computation Theory, 4(4):12.
V ogl, T., Mangis, J., Rigler, A., Zink, W., and Alkon, D. (1988). Accelerating the convergence of the
back-propagation method. Biological Cybernetics, 59:257–263.
83
von der Malsburg, C. (1973). Self-organization of orientation sensitive cells in the striate cortex.
Kybernetik, 14(2):85–100.
Waldinger, R. J. and Lee, R. C. T. (1969). PROW: a step toward automatic program writing. In
Walker, D. E. and Norton, L. M., editors, Proceedings of the 1st International Joint Conference on
Artiﬁcial Intelligence (IJCAI), pages 241–252. Morgan Kaufmann.
Wallace, C. S. and Boulton, D. M. (1968). An information theoretic measure for classiﬁcation. Com-
puter Journal, 11(2):185–194.
Wan, E. A. (1994). Time series prediction by using a connectionist network with internal delay lines.
In Weigend, A. S. and Gershenfeld, N. A., editors, Time series prediction: Forecasting the future
and understanding the past, pages 265–295. Addison-Wesley.
Wang, C., Venkatesh, S. S., and Judd, J. S. (1994). Optimal stopping and effective machine complexity
in learning. In Advances in Neural Information Processing Systems (NIPS’6) , pages 303–310.
Morgan Kaufmann.
Wang, S. and Manning, C. (2013). Fast dropout training. In Proceedings of the 30th International
Conference on Machine Learning (ICML-13), pages 118–126.
Watanabe, O. (1992). Kolmogorov complexity and computational complexity. EATCS Monographs
on Theoretical Computer Science, Springer.
Watanabe, S. (1985). Pattern Recognition: Human and Mechanical. Willey, New York.
Watkins, C. J. C. H. (1989). Learning from Delayed Rewards. PhD thesis, King’s College, Oxford.
Watkins, C. J. C. H. and Dayan, P. (1992). Q-learning. Machine Learning, 8:279–292.
Watrous, R. L. and Kuhn, G. M. (1992). Induction of ﬁnite-state automata using second-order recur-
rent networks. In Moody, J. E., Hanson, S. J., and Lippman, R. P., editors, Advances in Neural
Information Processing Systems 4, pages 309–316. Morgan Kaufmann.
Waydo, S. and Koch, C. (2008). Unsupervised learning of individuals and categories from images.
Neural Computation, 20(5):1165–1178.
Weigend, A. S. and Gershenfeld, N. A. (1993). Results of the time series prediction competition
at the Santa Fe Institute. In Neural Networks, 1993., IEEE International Conference on , pages
1786–1793. IEEE.
Weigend, A. S., Rumelhart, D. E., and Huberman, B. A. (1991). Generalization by weight-elimination
with application to forecasting. In Lippmann, R. P., Moody, J. E., and Touretzky, D. S., editors,
Advances in Neural Information Processing Systems (NIPS) 3 , pages 875–882. San Mateo, CA:
Morgan Kaufmann.
Weiss, G. (1994). Hierarchical chunking in classiﬁer systems. In Proceedings of the 12th National
Conference on Artiﬁcial Intelligence, volume 2, pages 1335–1340. AAAI Press/The MIT Press.
Weng, J., Ahuja, N., and Huang, T. S. (1992). Cresceptron: a self-organizing neural network which
grows adaptively. InInternational Joint Conference on Neural Networks (IJCNN), volume 1, pages
576–581. IEEE.
Weng, J. J., Ahuja, N., and Huang, T. S. (1997). Learning recognition and segmentation using the
cresceptron. International Journal of Computer Vision, 25(2):109–143.
84
Werbos, P. J. (1974). Beyond Regression: New Tools for Prediction and Analysis in the Behavioral
Sciences. PhD thesis, Harvard University.
Werbos, P. J. (1981). Applications of advances in nonlinear sensitivity analysis. InProceedings of the
10th IFIP Conference, 31.8 - 4.9, NYC, pages 762–770.
Werbos, P. J. (1987). Building and understanding adaptive systems: A statistical/numerical approach
to factory automation and brain research. IEEE Transactions on Systems, Man, and Cybernetics ,
17.
Werbos, P. J. (1988). Generalization of backpropagation with application to a recurrent gas market
model. Neural Networks, 1.
Werbos, P. J. (1989a). Backpropagation and neurocontrol: A review and prospectus. In IEEE/INNS
International Joint Conference on Neural Networks, Washington, D.C., volume 1, pages 209–216.
Werbos, P. J. (1989b). Neural networks for control and system identiﬁcation. In Proceedings of
IEEE/CDC Tampa, Florida.
Werbos, P. J. (1992). Neural networks, system identiﬁcation, and control in the chemical industries.
In D. A. White, D. A. S., editor, Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive
Approaches, pages 283–356. Thomson Learning.
Werbos, P. J. (2006). Backwards differentiation in AD and neural nets: Past links and new oppor-
tunities. In Automatic Differentiation: Applications, Theory, and Implementations , pages 15–34.
Springer.
West, A. H. L. and Saad, D. (1995). Adaptive back-propagation in on-line learning of multilayer
networks. In Touretzky, D. S., Mozer, M., and Hasselmo, M. E., editors, NIPS, pages 323–329.
MIT Press.
White, H. (1989). Learning in artiﬁcial neural networks: A statistical perspective. Neural Computa-
tion, 1(4):425–464.
Whitehead, S. (1992). Reinforcement Learning for the adaptive control of perception and action. PhD
thesis, University of Rochester.
Whiteson, S. (2012). Evolutionary computation for reinforcement learning. In Wiering, M. and van
Otterlo, M., editors, Reinforcement Learning, pages 325–355. Springer, Berlin, Germany.
Whiteson, S., Kohl, N., Miikkulainen, R., and Stone, P. (2005). Evolving keepaway soccer players
through task decomposition. Machine Learning, 59(1):5–30.
Whiteson, S. and Stone, P. (2006). Evolutionary function approximation for reinforcement learning.
Journal of Machine Learning Research, 7:877–917.
Widrow, B. and Hoff, M. (1962). Associative storage and retrieval of digital information in networks
of adaptive neurons. Biological Prototypes and Synthetic Systems, 1:160.
Widrow, B., Rumelhart, D. E., and Lehr, M. A. (1994). Neural networks: Applications in industry,
business and science. Commun. ACM, 37(3):93–105.
Wieland, A. P. (1991). Evolving neural network controllers for unstable systems. In International
Joint Conference on Neural Networks (IJCNN), volume 2, pages 667–673. IEEE.
85
Wiering, M. and Schmidhuber, J. (1996). Solving POMDPs with Levin search and EIRA. In Saitta,
L., editor, Machine Learning: Proceedings of the Thirteenth International Conference, pages 534–
542. Morgan Kaufmann Publishers, San Francisco, CA.
Wiering, M. and Schmidhuber, J. (1998a). HQ-learning. Adaptive Behavior, 6(2):219–246.
Wiering, M. and van Otterlo, M. (2012). Reinforcement Learning. Springer.
Wiering, M. A. and Schmidhuber, J. (1998b). Fast online Q(λ). Machine Learning, 33(1):105–116.
Wierstra, D., Foerster, A., Peters, J., and Schmidhuber, J. (2010). Recurrent policy gradients. Logic
Journal of IGPL, 18(2):620–634.
Wierstra, D., Schaul, T., Peters, J., and Schmidhuber, J. (2008). Natural evolution strategies. In
Congress of Evolutionary Computation (CEC 2008).
Wiesel, D. H. and Hubel, T. N. (1959). Receptive ﬁelds of single neurones in the cat’s striate cortex.
J. Physiol., 148:574–591.
Wiles, J. and Elman, J. (1995). Learning to count without a counter: A case study of dynamics
and activation landscapes in recurrent networks. In In Proceedings of the Seventeenth Annual
Conference of the Cognitive Science Society, pages pages 482 – 487, Cambridge, MA. MIT Press.
Wilkinson, J. H., editor (1965). The Algebraic Eigenvalue Problem. Oxford University Press, Inc.,
New York, NY , USA.
Williams, R. J. (1986). Reinforcement-learning in connectionist networks: A mathematical analysis.
Technical Report 8605, Institute for Cognitive Science, University of California, San Diego.
Williams, R. J. (1988). Toward a theory of reinforcement-learning connectionist systems. Technical
Report NU-CCS-88-3, College of Comp. Sci., Northeastern University, Boston, MA.
Williams, R. J. (1989). Complexity of exact gradient computation algorithms for recurrent neural
networks. Technical Report Technical Report NU-CCS-89-27, Boston: Northeastern University,
College of Computer Science.
Williams, R. J. (1992a). Simple statistical gradient-following algorithms for connectionist reinforce-
ment learning. Machine Learning, 8:229–256.
Williams, R. J. (1992b). Training recurrent networks using the extended Kalman ﬁlter. In Interna-
tional Joint Conference on Neural Networks (IJCNN), volume 4, pages 241–246. IEEE.
Williams, R. J. and Peng, J. (1990). An efﬁcient gradient-based algorithm for on-line training of
recurrent network trajectories. Neural Computation, 4:491–501.
Williams, R. J. and Zipser, D. (1988). A learning algorithm for continually running fully recurrent
networks. Technical Report ICS Report 8805, Univ. of California, San Diego, La Jolla.
Williams, R. J. and Zipser, D. (1989a). Experimental analysis of the real-time recurrent learning
algorithm. Connection Science, 1(1):87–111.
Williams, R. J. and Zipser, D. (1989b). A learning algorithm for continually running fully recurrent
networks. Neural Computation, 1(2):270–280.
86
Willshaw, D. J. and von der Malsburg, C. (1976). How patterned neural connections can be set up by
self-organization. Proc. R. Soc. London B, 194:431–445.
Windisch, D. (2005). Loading deep networks is hard: The pyramidal case. Neural Computation,
17(2):487–502.
Wiskott, L. and Sejnowski, T. (2002). Slow feature analysis: Unsupervised learning of invariances.
Neural Computation, 14(4):715–770.
Witczak, M., Korbicz, J., Mrugalski, M., and Patton, R. J. (2006). A GMDH neural network-based
approach to robust fault diagnosis: Application to the DAMADICS benchmark problem. Control
Engineering Practice, 14(6):671–683.
W¨ollmer, M., Blaschke, C., Schindl, T., Schuller, B., F ¨arber, B., Mayer, S., and Trefﬂich, B. (2011).
On-line driver distraction detection using Long Short-Term Memory. IEEE Transactions on Intel-
ligent Transportation Systems (TITS), 12(2):574–582.
W¨ollmer, M., Schuller, B., and Rigoll, G. (2013). Keyword spotting exploiting Long Short-Term
Memory. Speech Communication, 55(2):252–265.
Wolpert, D. H. (1992). Stacked generalization. Neural Networks, 5(2):241–259.
Wolpert, D. H. (1994). Bayesian backpropagation over i-o functions rather than weights. In Cowan,
J. D., Tesauro, G., and Alspector, J., editors, Advances in Neural Information Processing Systems
(NIPS) 6, pages 200–207. Morgan Kaufmann.
Wu, D. and Shao, L. (2014). Leveraging hierarchical parametric networks for skeletal joints based
action segmentation and recognition. In Proc. Conference on Computer Vision and Pattern Recog-
nition (CVPR).
Wu, L. and Baldi, P. (2008). Learning to play Go using recursive neural networks. Neural Networks,
21(9):1392–1400.
Wyatte, D., Curran, T., and O’Reilly, R. (2012). The limits of feedforward vision: Recurrent process-
ing promotes robust object recognition when objects are degraded. Journal of Cognitive Neuro-
science, 24(11):2248–2261.
Wysoski, S. G., Benuskova, L., and Kasabov, N. (2010). Evolving spiking neural networks for audio-
visual information processing. Neural Networks, 23(7):819–835.
Yamauchi, B. M. and Beer, R. D. (1994). Sequential behavior and learning in evolved dynamical
neural networks. Adaptive Behavior, 2(3):219–246.
Yamins, D., Hong, H., Cadieu, C., and DiCarlo, J. J. (2013). Hierarchical modular optimization of
convolutional networks achieves representations similar to macaque IT and human ventral stream.
Advances in Neural Information Processing Systems (NIPS), pages 1–9.
Yang, M., Ji, S., Xu, W., Wang, J., Lv, F., Yu, K., Gong, Y ., Dikmen, M., Lin, D. J., and Huang,
T. S. (2009). Detecting human actions in surveillance videos. In TREC Video Retrieval Evaluation
Workshop.
Yao, X. (1993). A review of evolutionary artiﬁcial neural networks. International Journal of Intelli-
gent Systems, 4:203–222.
87
Yin, F., Wang, Q.-F., Zhang, X.-Y ., and Liu, C.-L. (2013). ICDAR 2013 Chinese handwriting recog-
nition competition. In 12th International Conference on Document Analysis and Recognition (IC-
DAR), pages 1464–1470.
Yin, J., Meng, Y ., and Jin, Y . (2012). A developmental approach to structural self-organization in
reservoir computing. IEEE Transactions on Autonomous Mental Development, 4(4):273–289.
Young, S., Davis, A., Mishtal, A., and Arel, I. (2014). Hierarchical spatiotemporal feature extraction
using recurrent online clustering. Pattern Recognition Letters, 37:115–123.
Yu, X.-H., Chen, G.-A., and Cheng, S.-X. (1995). Dynamic learning rate optimization of the back-
propagation algorithm. IEEE Transactions on Neural Networks, 6(3):669–677.
Zamora-Martnez, F., Frinken, V ., Espaa-Boquera, S., Castro-Bleda, M., Fischer, A., and Bunke, H.
(2014). Neural network language models for off-line handwriting recognition.Pattern Recognition,
47(4):1642–1652.
Zeiler, M. D. (2012). ADADELTA: An Adaptive Learning Rate Method. CoRR, abs/1212.5701.
Zeiler, M. D. and Fergus, R. (2013). Visualizing and understanding convolutional networks. Technical
Report arXiv:1311.2901 [cs.CV], NYU.
Zemel, R. S. (1993). A minimum description length framework for unsupervised learning. PhD thesis,
University of Toronto.
Zemel, R. S. and Hinton, G. E. (1994). Developing population codes by minimizing description
length. In Cowan, J. D., Tesauro, G., and Alspector, J., editors, Advances in Neural Information
Processing Systems 6, pages 11–18. Morgan Kaufmann.
Zeng, Z., Goodman, R., and Smyth, P. (1994). Discrete recurrent neural networks for grammatical
inference. IEEE Transactions on Neural Networks, 5(2).
Zimmermann, H.-G., Tietz, C., and Grothmann, R. (2012). Forecasting with recurrent neural net-
works: 12 tricks. In Montavon, G., Orr, G. B., and M ¨uller, K.-R., editors, Neural Networks:
Tricks of the Trade (2nd ed.), volume 7700 of Lecture Notes in Computer Science, pages 687–707.
Springer.
Zipser, D., Kehoe, B., Littlewort, G., and Fuster, J. (1993). A spiking network model of short-term
active memory. The Journal of Neuroscience, 13(8):3406–3420.
88

PMEvo: Portable Inference of Port Mappings for
Out-of-Order Processors by Evolutionary
Optimization
Fabian Ritter
Saarland University
Saarland Informatics Campus
Germany
fabian.ritter@cs.uni-saarland.de
Sebastian Hack
Saarland University
Saarland Informatics Campus
Germany
hack@cs.uni-saarland.de
Abstract
Achieving peak performance in a computer system requires
optimizations in every layer of the system, be it hardware or
software. A detailed understanding of the underlying hard-
ware, and especially the processor, is crucial to optimize
software. One key criterion for the performance of a proces-
sor is its ability to exploit instruction-level parallelism. This
ability is determined by the port mapping of the processor,
which describes the execution units of the processor for each
instruction.
Processor manufacturers usually do not share the port
mappings of their microarchitectures. While approaches to
automatically infer port mappings from experiments exist,
they are based on processor-specific hardware performance
counters that are not available on every platform.
We present PMEvo, a framework to automatically infer
port mappings solely based on the measurement of the exe-
cution time of short instruction sequences. PMEvo uses an
evolutionary algorithm that evaluates the fitness of candidate
mappings with an analytical throughput model formulated
as a linear program. Our prototype implementation infers a
port mapping for Intel’s Skylake architecture that predicts
measured instruction throughput with an accuracy that is
competitive to existing work. Furthermore, it finds port map-
pings for AMD’s Zen+ architecture and the ARM Cortex-A72
architecture, which are out of scope of existing techniques.
CCS Concepts:• Computer systems organization →Re-
duced instruction set computing; Complex instruction
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not
made or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for components
of this work owned by others than ACM must be honored. Abstracting with
credit is permitted. To copy otherwise, or republish, to post on servers or to
redistribute to lists, requires prior specific permission and/or a fee. Request
permissions from permissions@acm.org.
PLDI ’20, June 15–20, 2020, London, UK
© 2020 Association for Computing Machinery.
ACM ISBN 978-1-4503-7613-6/20/06. . . $15.00
https://doi.org/10.1145/3385412.3385995
set computing ; • General and reference →Measure-
ment; Experimentation; Performance; Estimation; Met-
rics; • Software and its engineering →Compilers; Soft-
ware performance; • Security and privacy →Hardware
reverse engineering ; • Computing methodologies →
Simulation tools ; Genetic algorithms ; • Theory of com-
putation →Evolutionary algorithms.
Keywords: port mapping, evolutionary algorithm, processor
reverse engineering
ACM Reference Format:
Fabian Ritter and Sebastian Hack. 2020. PMEvo: Portable Inference
of Port Mappings for Out-of-Order Processors by Evolutionary
Optimization. In Proceedings of the 41st ACM SIGPLAN International
Conference on Programming Language Design and Implementation
(PLDI ’20), June 15–20, 2020, London, UK. ACM, New York, NY, USA,
15 pages. https://doi.org/10.1145/3385412.3385995
1 Introduction
Accurately estimating the time required to execute a given
program has become increasingly complex. While advances
in hardware design enable faster execution times, they make
it difficult to optimize programs such that they utilize the
available resources to the best possible extent. A particular
cause of unforeseen performance characteristics is the ex-
ploitation of instruction level parallelism via out-of-order
execution [23]. This technique enables the processor to dy-
namically re-order the instructions of a sequential program
and execute them in parallel on a set of execution ports.
Therefore, optimizing a program to achieve peak perfor-
mance on a processor requires knowledge of the ports that
can be used by each instruction. However, the instruction-
to-port mapping, or port mapping, is usually only known
to hardware manufacturers and may vary with each new
hardware generation.
While approaches towards understanding the performance
characteristics of processors without full insight into their
internals exist, they suffer from shortcomings: Some ap-
proaches require significant manual effort [7, 12] or are re-
stricted to validating existing port mappings [ 19]. Others
are closely tied to microarchitecture-specific performance
counters [1, 12, 13] that prevent their applicability to a wide
arXiv:2004.10044v1  [cs.AR]  21 Apr 2020
PLDI ’20, June 15–20, 2020, London, UK Fabian Ritter and Sebastian Hack
range of practically-relevant processors. Another line of
research [20] uses machine learning to train a neural net-
work that estimates instruction throughput. This approach is
portable among microarchitectures, but the resulting black
box model is hard to use for identifying concrete perfor-
mance bottlenecks.
This paper proposes a solution that comes without any of
these drawbacks: Experiments are automatically generated
from a description of the available instructions. Performing
the experiments requires only measuring the time taken for
executing an instruction sequence. The result is a concise
and interpretable port mapping model that existing tools
can use to identify bottlenecks and to guide optimization
decisions.
We achieve this with PMEvo, a framework that uses an
evolutionary algorithm to find a port mapping that excels in
explaining measured throughputs for automatically gener-
ated instruction sequences. These instruction sequences are
designed to reveal conflicting resource requirements for pairs
of instructions while exhibiting as few data dependencies
as possible. For these instruction sequences, the throughput
is only limited by constrained ports and therefore carries
information about the port mapping.
A key component of the evolutionary algorithm is a novel
bottleneck simulation algorithm to evaluate the fitness of
candidate port mappings. This algorithm efficiently com-
putes the solution of a linear program that models an optimal
instruction scheduler for a given port mapping. Our novel
bottleneck simulation algorithm outperforms solving the
corresponding linear program for realistic port mappings by
two orders of magnitude.
We evaluate PMEvo by the throughput prediction accu-
racy of its inferred port mappings on port-mapping-bound
basic blocks for microarchitectures by Intel, AMD and ARM.
PMEvo’s prediction accuracy for the Intel Skylake archi-
tecture is close to existing approaches like IACA [ 17] and
uops.info [1] that rely on stronger knowledge about the mi-
croarchitecture. For AMD and ARM, PMEvo outclasses the
state-of-the-art port mapping model of llvm-mca [7].
In summary, we make the following contributions:
•An evolutionary algorithm that infers port mappings
from specifically designed experiments with measured
throughputs without relying on microarchitecture-
specific features.
•A bottleneck simulation algorithm that allows to ef-
ficiently evaluate the fitness of port mappings for a
given set of experiments.
•A prototype implementation that finds a port mapping
that is competitive to related work for Intel’s Skylake
architecture and that is the first one to automatically
find port mappings for the AMD Zen+ and the ARM
Cortex-A72 microarchitectures.
L1 ICache
µop Cache Decoder
Register Management
Scheduler
Port 0
Int ALU
Vec ALU
DIV
Port 1
Int ALU
Vec ALU
Port 2
LD/ST
Port 3
ST
L1 DCache
Figure 1. Simplified overview of a modern processor de-
sign (based on Figure 2-3 in the Intel Software Optimization
Manual [16])
2 Background: Processor Design
Modern processors apply out-of-order execution [23].1 This
concept is based on the observation that instructions can be
executed in any order as long as the results are the same as if
they were executed in program order. Therefore, a processor
may execute instructions in parallel and reorder them to
any extent that preserves the read-after-write dependencies
between the operations and the externally-visible effects.
Out-of-order execution is often combined with a scheme
to decompose instructions into simpler microarchitecture-
specific operations. These so-called micro-ops or µops are
then subject to reordering.
Figure 1 shows the relevant parts of a microarchitecture
that employs out-of-order execution and µop decomposition.
Instructions are fetched and decoded from the instruction
cache in program order. The decoder produces µops, which
are cached for future re-use. The register management engine
resolves false (write-after-read or write-after-write) depen-
dencies by mapping the operand registers of each operation
to a larger number of physical registers. A scheduler decides
based on operand dependencies and resource availability
when and where to execute the µops. The execution units
(e.g. arithmetical units and load/store units), which execute
the µops, are grouped behind ports. Often, execution units
are pipelined, allowing the ports to start processing a new
instruction in every cycle. Several instances of the same kind
of execution unit can exist at different ports.
A key factor to the running time of a given piece of code
on a processor is therefore the port mapping. It specifies how
1A contemporary introduction to the topic can be found e.g. in Chapter 3
of the textbook by Hennessy and Patterson [15].
PMEvo: Portable Inference of Port Mappings for Out-of-Order Processors by Evolutionary Optimization PLDI ’20, June 15–20, 2020, London, UK
instructions are decomposed into µops and which µops can
be executed on which ports.2
Some microarchitectures, particularly those designed by
Intel, provide fine-grained hardware performance counters
that count the number of executedµops per port. While these
greatly help at inferring port mappings, relying on them
excludes all microarchitectures that do not provide similar
performance counters. Therefore, we base our approach on
the more portable observation of throughputs as defined in
the following.3
Definition 1. The throughput t∗(e)of an instruction se-
quence (or experiment) e on a given processor is the average
number of processor cycles required to execute e in a steady
state.
The execution of an experiment in an infinite loop is con-
sidered to have reached a steady state when the average
number of required cycles per iteration stays constant for
the remaining execution.
3 Analytical Throughput Model
Since our goal is to infer a port mapping from throughput
measurements, we need to understand the connection be-
tween the port mapping of the processor and the throughput
that is achieved for an experiment. The precise inner work-
ings of processors are well-kept secrets of the manufacturers,
therefore we postulate a model of how processors execute
instructions with respect to a port mapping. In this section,
we present a model that is supported by information pro-
vided by hardware manufacturers [2, 4, 16, 17] and related
research [1, 12]. The linear programs we use in this section
to define the throughput for a given port mapping are exten-
sions of work presented by Abel and Reineke [1].
3.1 Out-of-Order Throughput Model
We start by defining a simple out-of-order execution model
that does not consider the decomposition of instructions into
µops. We refer to it as the two-level model (mapping only
instructions to ports). Section 3.2 extends this model to the
three-level model , which additionally supports decomposing
instructions into µops.
Our throughput model is based on the notion of a port
mapping, as defined in the following.
Definition 2. A port mapping in the two-level model is a
bipartite graph (I ∪·P, M)with the nodes split disjointly into
a set I of instructions and a setP of ports and edgesM ⊆I×P
between these.
An edge between instruction i and port k indicates that
instruction i can be executed on port k.
2This is called port usage in the work by Abel and Reineke [1].
3This definition is equivalent to the one by Mendis et al. [20], an extension
of the instruction-wise throughput definition used by Fog [12] and Abel
and Reineke [1].
Consider the port mapping shown in Figure 2. There are
four instructions mul, add, sub, and store that are mapped
to the three ports P1, P2, and P3. The two instructions add
and sub can both be executed on the same two ports P1 and
P2, mul can use only one of them, P1, and store has to be
executed on a separate port P3.
I:
P:
mul add sub store
P1 P2 P3
Figure 2. Example of a port mapping in the two-level model
In this model, an experiment is represented as a multiset of
instructions, i.e. a function e : I →N that maps instructions
to their number of occurrences. We abstract from the order
of the instructions since we only use experiments that can
be reordered freely by the scheduler. The throughput of an
experiment with a given port mapping is characterized by
the following definiton.
Definition 3. Given a port mapping m := (I ∪·P, M)in
the two-level setting, the throughput t∗
m(e)under m for an
experiment e : I →N is the objective value of an optimal
solution to the following linear program:
minimize t
subject to
Õ
k ∈P
xik = e(i) for all i ∈I (A)
Õ
i ∈I
xik ≤t for all k ∈P (B)
xik ≥0 for all (i, k)∈ M (C)
xik = 0 for all (i, k)< M (D)
The intuition for this linear program is that each instruc-
tion i in the experiment has the mass e(i). This mass is dis-
tributed among the ports that can execute i, as required by
constraint (A). The xik are real-valued variables that repre-
sent the share of the mass e(i)that is executed on port k in
the experiment. Constraint (B) establishes the objective t as
an upper bound of the sums of mass shares on each port.
The constraints (C) and (D) guarantee that the mass of an
instruction i is distributed to ports that can execute i. The
throughput is the maximal mass associated to any port if all
mass is distributed as evenly as possible.
Example 1. Figure 3 displays a graphical interpretation of
an optimal solution of the linear program for the experiment
e := {add 7→2, mul 7→1, store 7→1}
under the mapping given in Figure 2.
The mass allocated to each port Pk is drawn in the corre-
sponding bucket. The throughput of 1.5 cycles is the mass
of the most occupied ports P1 and P2. Note that the mass
PLDI ’20, June 15–20, 2020, London, UK Fabian Ritter and Sebastian Hack
P1 P2 P3
mul
add
add store
0
1
2
1.5
Figure 3. Visualization of an example port allocation
of the two add instructions is split unevenly among two
ports. While these non-integer instruction portions might
seem counter-intuitive, they assort with the definition of
throughput as the average number of cycles to execute an
experiment.
Such a throughput can be realized by executing one add
instruction on port P1 in every second iteration, yielding an
average of 0.5 add instructions on P1 per execution of the
experiment.
Definition 3 relies on several assumptions:
1. The processor schedules the instructions optimally.
2. Operational units are fully pipelined, i.e. every instruc-
tion blocks exactly one port for exactly one cycle.
3. The fetch and decode units do not impose a bottleneck
for the experiment.
4. There are no relevant (read-after-write) data depen-
dencies among the instructions of the experiment.
The validity of assumptions (1) and (2) depends on the pro-
cessor under test. We found these to be usually fulfilled
by modern microarchitectures. An exception are complex
instructions like divisions, which can block a port or an
operational unit for multiple cycles.
We ensure the validity of assumptions (3) and (4) by select-
ing our experiments appropriately: Only sufficiently short
experiments that do not hit bottlenecks in the fetch/decode
stages are considered. The operands of the experiments are
furthermore chosen such that writing instructions can be
retired before their output register is read for the next time.
3.2 Micro-Operation Decomposition
To represent the decomposition of instructions intoµops, we
extend the definition of a port mapping by a layer of µops
as follows.
Definition 4. A port mapping in the three-level model is
a tripartite graph (I ∪·U ∪·P, N ∪·M)with labeled edges
N ⊆I ×N ×U between instructions and µops as well as
unlabeled edges M ⊆U ×P between µops and ports.
A labeled edge (i, n,u) ∈N means that there are n in-
stances of the µop u in the µop decomposition of instruc-
tion i.
An example is displayed in Figure 4. Here, add and sub
are implemented as one µop U2 that can be executed on two
ports P1 and P2. The mul and store instructions are decom-
posed into two µops, the former in two of the same kind, U1,
and the latter into two different ones,U2 and U3. The store in-
struction has a partial conflict with add and sub that cannot
be represented in the two-level model.
I:
U:
P:
mul add sub store
U1 U2 U3
P1 P2 P3
2 1 1 1 1
Figure 4. Example of a three-level port mapping
It is important to notice the different semantics of the
layers of edges: For each instance of an instruction i, all cor-
responding µops u such that (i, ·,u)∈ N have to be executed
whereas a µop u is executed on exactly one of the allowed
ports k such that (u, k)∈ M. The linear program from Sec-
tion 3.1 can be slightly modified to compute the throughput
t∗
m(e)of an experiment e : I →N under the three-level port
mapping m := (I ∪·U ∪·P, N ∪·M):
minimize t
subject to
Õ
k ∈P
xuk =
Õ
(i, n,u)∈N
e(i)·n for all u ∈U (A)
Õ
u ∈U
xuk ≤t for all k ∈P (B)
xuk ≥0 for all (u, k)∈ M (C)
xuk = 0 for all (u, k)< M (D)
All previous occurrences of instructions are replaced by
occurrences of µops except for the right-hand side of con-
straint (A). The right-hand side of (A) ensures that a µop u
that occurs n times in the decomposition of instruction i is
taken into account with its appropriate mass.
A valuable observation is that computing the through-
put of an experiment e : I → N with a port mapping
(I ∪·U ∪·P, N ∪·M)in the three-level model can be reduced to
computing throughput in the simpler two-level model: We
instead compute the throughput of the experiment
e′=

u 7→
Õ
(i, n,u)∈N
e(i)·n

with the two-level mapping (U ∪·P, M). The multiset e′con-
tains the µops that are needed to execute e according to N .
These µops are used as instructions for the two-level model.
This construction allows us to use an algorithm for the
simpler two-level model to compute throughput in the three-
level model.
PMEvo: Portable Inference of Port Mappings for Out-of-Order Processors by Evolutionary Optimization PLDI ’20, June 15–20, 2020, London, UK
4 The PMEvo Framework
We propose the PMEvo framework to automatically infer
port mappings from throughput experiments. An overview
of this framework is given in Figure 5.
ISA
# portsExperiment
Generation
Throughput
Measurement
Congruence
Filtering
Evolutionary
Optimization
port mapping
Figure 5. PMEvo framework overview
PMEvo consists of four main stages, which we describe
in the following subsections: Generating relevant experi-
ments (4.1), measuring the throughput of the experiments
on a given processor (4.2), a preprocessing step that identi-
fies congruent instructions (4.3), and evolutionary optimiza-
tion (4.4).
4.1 Experiment Generation
The input of the first stage of PMEvo is a description of
the instruction set architecture under test. This description
is a set of instruction forms, i.e. instructions with typed
placeholders for their operands. The type of the placeholder
specifies the operand kind (e.g. memory operand, general
purpose or vector register) and the width of the respective
operand. There can be multiple instruction forms for the
same operation with different operand types.
PMEvo constructs a set of experiments from this informa-
tion with the following components:
1. for each instruction form i, an experiment {i 7→1}
measuring its individual throughput t∗(i)
2. for each pair (iA, iB )of instruction forms, an experi-
ment {iA 7→1, iB 7→1}
3. for each pair (iA, iB )of instruction forms witht∗(iA)>
t∗(iB ), an experiment {iA 7→1, iB 7→n}where
n = ⌈t∗(iA)/t∗(iB )⌉
Experiments with this structure lead to different outcomes
depending on the port mapping: If the µops of two instruc-
tion forms iA and iB require the same resources, experiment
(2) will result in a throughput that is the sum of the indi-
vidual throughputs of iA and iB . In case the µops of iA and
iB are executed by disjoint execution units, the throughput
of experiment (3) will be n ·t∗(iB ). More complex partial re-
source conflicts will lead to measured throughputs for these
experiments that are harder to interpret manually. It is the
task of the evolutionary algorithm to find a mapping that
explains these throughputs.
The evolutionary algorithm is not restricted to experi-
ments of this structure. In theory, longer experiments that
combine instances of more than two different instruction
forms can unveil resource conflicts that cannot be covered
by these experiments. However, when exploring the exper-
iment design space experimentally for existing processors,
we did not observe benefits in port mapping quality from
more complex experiments.
4.2 Throughput Measurement
The goal of this stage is to measure the throughput of the
generated experiments. Our measurement method follows
Definition 1: The instruction forms of the experiment are
instantiated with operands while avoiding data dependencies.
The resulting instruction sequence is executed in a loop such
that the execution reaches a steady state.
PMEvo uses a register allocator that assigns a register
from the appropriate register class to each register operand
of the instruction forms. To avoid harmful dependencies,
written operands are instantiated with most recently read
registers and read operands with least recently written reg-
isters. Using as many different registers as available, this
ensures that instructions with long latencies have enough
time to complete before their results are read.
Memory operands are instantiated with a separate register
containing a valid base pointer and one of several different
constant offsets to avoid data dependencies on the memory.
Before operand allocation, we unroll several loop iter-
ations. This has several benefits: It further increases the
dependence distance by allowing more registers to be allo-
cated and it avoids loop-carried dependencies. Additionally,
it reduces the influence of the loop code on our time mea-
surements. The range of loop body lengths that achieve an
optimal throughput depends on the microarchitecture. We
found a length of 50 instructions to be in the appropriate
range for all of our evaluated architectures. With this length,
the loop body will be resident in the µop cache (if the ar-
chitecture has one). This avoids performance bottlenecks
due to restrictions on the number of concurrently fetched
and decoded instructions. The loop bound is automatically
chosen to ensure that the loop runs for a specific time that
guarantees steady-state execution. This time is estimated
empirically for the processor under test by comparing the
measurement stability for different times. For the evaluated
platforms, we found a time of 10 msto be appropriate.
To measure the throughput, we emit the instantiated loop
as inline assembly into a C program wrapped with calls to
gettimeofday() and setup code. The resulting program is
compiled with a C compiler for the platform under test and
executed. We compute the throughput of the experiment
with the following formula:
t∗(e)= measured time ×frequency
#executed instances of e
PLDI ’20, June 15–20, 2020, London, UK Fabian Ritter and Sebastian Hack
The reported throughput for an experiment is the median
over multiple such measurements to accommodate for occa-
sional fluctuations in the processor’s clock frequency.
4.3 Congruence Filtering
In a processor microarchitecture, we expect that groups of
instruction forms require the same execution resources. In-
struction forms whose operations are implemented similarly
in the processor, e.g. addition and subtraction, often lead to
such groups.
PMEvo exploits these patterns to reduce the search space
of the evolutionary algorithm. It partitions the set of instruc-
tion forms into congruence classes of instruction forms that
are not distinguishable with the generated experiment set.
In this partitioning, two instruction forms iA and iB are in
the same class if and only if the following conditions hold:
•iA and iB exhibit equal individual throughputs.
•Any two experiments {iA 7→m, iC 7→n}and {iB 7→
m, iC 7→n}that combine these instruction forms with
any other instruction form iC exhibit equal through-
puts.
For this purpose, we consider throughputst1 and t2 equal (up
to measurement errors) if their symmetric relative difference
is limited by a user-specified constant ε, i.e. if
|t1 −t2 |
|t1 + t2 |/2 < ε
For each congruence class, PMEvo selects a representative
to be included in the instruction set for the evolutionary
algorithm. The evolutionary algorithm then only needs to
consider experiments that consist of these representatives.
4.4 Evolving Port Mappings
The core of PMEvo is an evolutionary algorithm that searches
for a port mapping that accurately explains the observed
throughputs for a given set of experiments. Evolutionary al-
gorithms are a well-proven technique to approach optimiza-
tion problems. They mimic concepts from natural evolution
to approximatively optimize complex metrics in non-linear
problem settings. We refer to the textbook by De Jong [11]
for a comprehensive treatment.
Every evolutionary algorithm is centered around a rep-
resentation scheme that characterizes the space of possible
solutions of the optimization problem. Naturally, the scheme
that we use is that of port mappings withµop decomposition
as described in Section 3.2. The sets I of Instructions and P
of Ports are given by the user. We identify each µop with
the set of ports that can execute it and allow all non-empty
subsets of P as µops. The width |u|= |{k |(u, k)∈ M}|of a
µop u is the number of ports that can execute u.
PMEvo’s evolutionary algorithm follows the structure in
Algorithm 1. Initially, a set of p port mappings is sampled
randomly to form a population. This population is iteratively
refined through evolution steps. In each such step, p child
initialize population randomly
while not done do
apply evolutionary operators
evaluate fitness
select new population
end
perform local search
return fittest individual
Algorithm 1. Structure of the evolutionary algorithm
mappings are generated via evolutionary operators. The
resulting population of 2p port mappings is sorted according
to the fitness metric and the best-performingp mappings are
selected as the new population. The evolution terminates
once the fitness of the population has converged to a single
value or an iteration limit is exceeded. By selecting a value
for p, the user can find a trade-off between inference time
and quality of the inferred port mapping.
After the evolution terminates, PMEvo employs a greedy
hill-climbing algorithm to move from the found solutions
to a local optimum in the space of possible port mappings.
It incrementally adjusts the number n of µop occurrences
for each edge (i, n,u)∈ N and keeps the changes to the port
mapping if it is fitter than before.
In the following, we describe the components that consti-
tute the evolutionary algorithm in detail.
Initialization. Each member of the initial population is
sampled randomly from the set of possible port mappings as
follows. For each instruction i, a random set of 1 to |P|many
different µops is sampled. The number of occurrences for
each of these µops u in the mapping for i is sampled from
the interval [1, ⌈t∗(i)·|u|⌉]. The upper bound of this interval
is an implication of the throughput model: An instruction
with ⌈t ·|u|⌉instances of a µop u in its decomposition can
achieve no throughput smaller than t.
Evolutionary Operators. Evolutionary operators create
new individuals from existing individuals in the population.
The most common operators in evolutionary algorithms are
recombination and mutation.
We employ a binary recombination operator that mixes
the information of two parent mappings to generate two
child mappings. For each instruction i, the set of occurring
µops with multiplicities is divided randomly into two parts
that form the corresponding assignments for the children.
This operator is applied to individuals that are sampled uni-
formly at random from the population.
When designing the evolutionary algorithm, we tried var-
ious random mutation strategies. Experiments showed little
to no benefit over a design without a mutation operator
while contributing substantial numbers of fitness computa-
tions. Therefore, we eliminated mutation operators from our
PMEvo: Portable Inference of Port Mappings for Out-of-Order Processors by Evolutionary Optimization PLDI ’20, June 15–20, 2020, London, UK
design to explore larger populations more effectively in the
same execution time.
Fitness Metric. PMEvo’s evolutionary algorithm approxi-
mately solves a multiobjective optimization problem (MOP)
with the goal of minimizing two metrics: The average relative
prediction error Davд and the µop volume V . These metrics
describe the quality of a port mappingm = (I∪·U∪·P, N ∪·M)
for a set E ⊆(I →N)× R of experiments with measured
throughputs as follows:
Davд (m)= 1
|E|
Õ
(e, t)∈E
|t∗
m(e)−t |
t
V (m)=
Õ
(i, n,u)∈N
n ·|u|
A low value for Davд (m)ensures an accurate prediction
whereas a smaller µop volume indicates a more compact and
therefore more interpretable mapping.
We solve the MOP through a priori scalarization, as de-
scribed e.g. in Chapter 4.1 of the textbook by Miettinen [21]:
We combine the objectives into a single one that is inter-
preted as the fitness function F(m)as follows:
F(m)= Λ1(Davд (m))+ Λ2(V (m))
Λ1 and Λ2 are affine transformations that are chosen in every
iteration to normalize both objective metrics to the range
[0, 1000]. They ensure that the extremal objective values of
the current population are mapped to 0 and 1000, respec-
tively, with all other objective values in between.
Combining the accuracy metric Davд with a compactness
metric is necessary because throughput measurements usu-
ally do not uniquely identify a single port mapping. The
port mapping model is flexible enough to allow for a wide
range of well-performing mappings with different character-
istics. While the found compact mappings are not necessarily
identical to the port mappings that are really used in the pro-
cessor, they still capture the performance characteristics of
the hardware as they are observable from the outside.
4.5 Efficient Bottleneck Simulation Algorithm
Practical applicability of evolutionary algorithms depends
on evaluating the fitness of many candidates in as little time
as possible. For a given time budget, fitness evaluation speed
directly corresponds to the quality of the obtained solution.
With faster fitness evaluation, more candidates for survival
can be considered, resulting in superior solutions.
Therefore, a critical component of our approach is the
efficient simulation of experiments under a given port map-
ping. Instead of directly solving the linear program from
Section 3.1, we use a bottleneck simulation algorithm that
computes the optimal solution of the linear program. We re-
strict our presentation here to port mappings in the two-level
model for a more concise description. As we have observed
in Section 3.2, this extends to the three-level model straight-
forwardly.
The bottleneck simulation algorithm implements the fol-
lowing characterization of the throughput t∗
m(e)of an exper-
iment e under the port mapping m := (I ∪·P, M):
t∗
m(e)= max
Q ⊆P
Í{e(i)| Ports(m, i)⊆ Q}
|Q| (1)
Ports(m, i):= {k |(i, k)∈ M}denotes the set of ports that
can execute an instructioni under m. This characterization is
based on the observation that the throughputt∗
m(e)has to be
determined by a non-empty set Q∗of bottleneck ports. Each
of the ports inQ∗has to execute a mass of instructions that is
equal to t∗
m(e). In other words,t∗
m(e)is equal to the total mass
of instructions that need to be executed on ports from Q∗,
divided by the size of Q∗. An optimal scheduler will assign
instructions that do not need to be executed on ports fromQ∗
to less utilized ports. For each Q, the maximized term from
Equation 1 is a lower bound to t∗
m(e). Consequentially, find-
ing a maximal term gives us precisely the throughput t∗
m(e).
A formal proof for this equation is given in Appendix A.
Example 2. For the execution in Figure 3, Q∗ is the set
{P1, P2}. Trying to move mass from one of these ports to any
other ports is either not possible (for mul) or causes another
port from Q∗to execute more mass. P3 on the other hand is
irrelevant for the throughput of the experiment.
Our algorithmic implementation of this characterization
computes the max operation in Equation 1 by enumerating
all subsets of the set of ports and evaluating the correspond-
ing term. The run-time of this algorithm is in Θ(2|P|), which
is substantially more expensive than the polynomial run-
time of LP solving [6] from a complexity-theoretic point of
view. Nevertheless, this algorithm is considerably faster for
practical problems, as we show in Section 5.4. On the one
hand, this is due to the small number of execution ports
available in modern systems. Typical systems have eight
(e.g. Intel Skylake [16] and ARM A72 [4]) or ten (e.g. AMD
Ryzen [2]) ports available. On the other hand, thanks to the
simplicity of the above algorithm, it is amenable to aggres-
sive performance optimizations such as vectorization.
5 Evaluation
This section evaluates three aspects of our work:
•The appropriateness of the processor model as de-
scribed in Section 3 and our mechanism for measuring
throughput (Section 5.2).
•The quality of the inferred port mappings for three
microarchitectures from different manufacturers (Sec-
tion 5.3).
•The performance characteristics of the bottleneck sim-
ulation algorithm (Section 5.4).
PLDI ’20, June 15–20, 2020, London, UK Fabian Ritter and Sebastian Hack
Table 1. Evaluated processors
SKL ZEN A72
Manufact. Intel AMD RockChip
Processor Core i7 6700 Ryzen 5 2600X RK3399
Microarch. Skylake Zen+ Cortex-A72
# Ports 8 + DIV 10 7 + BR
Instr. Set x86-64 x86-64 ARMv8-A
Clock Freq. 3.4 GHz 3.6 GHz 1.8 GHz
RAM 32 GB 32GB 4GB
5.1 Setup
5.1.1 Evaluated Processors. We use three devices with
processors of distinct manufacturers for our evaluation, de-
noted as SKL, ZEN, and A72 in the following. Relevant pa-
rameters are listed in Table 1. SKL has a separate pipeline
of long-running operations, marked as DIV, that has to be
modeled as an additional port. One port of A72 is only used
for processing branch instructions (BR). It is omitted in our
model as we do not consider instructions that alter con-
trol flow. All evaluated systems have frequency scaling and
flexible overclocking mechanisms (e.g. Intel Turbo Boost)
disabled to facilitate reliable measurements.
A72 and ZEN are of particular interest since they do not
provide the per-port performance counters that other ap-
proaches rely on [3, 5] whereas SKL gives means for a com-
parison to related work.
5.1.2 Considered Instructions. We select for each in-
struction set architecture (ISA) under test a relevant set of
instruction forms. These sets are derived from the instruc-
tions that compilers emit when compiling the SPEC CPU
2017 benchmarks[8]. Our instruction forms for the ARMv8-A
ISA are extracted from the instructions that GCC (version
4.9.4, flags: -O3) emits. For x86-64, we only extract the used
instruction mnemonics from the output of Clang (version
8.0, flags: -O3 -mavx2 ) and use the machine-readable in-
puts of Abel and Reineke [1] to generate the corresponding
instruction forms.
We exclude the following instructions from these sets:
•Branch and jump instructions, since their throughput
heavily depends on the branch predictor.
•Instructions with implicitly read operands, since these
cause dependencies that cannot be resolved through
register allocation. Throughput for these could be mea-
sured by introducing additional dependency-breaking
instructions as done by Abel and Reineke [1].
•x86 SSE instructions, since these add transition penal-
ties when benchmarked together with AVX instruc-
tions.
•All instruction variants that operate on subregisters,
to keep the run time of the evaluation bearable.
•x86 instructions that are not supported by Ithemal [20],
to have a common baseline for all comparisons.
The resulting instruction descriptions contain 310 x86-64
instruction forms and 390 ARMv8-A instruction forms.
5.2 Processor Model and Measurements
In this section, we validate the practicality of the throughput
model and the measurement mechanism. We compare mea-
sured throughputs with the results of a simulation according
to the processor model with a ground truth port mapping
from the work by Abel and Reineke [1]. Since this work
only provides port usage for Intel architectures, we compare
their Intel Skylake port mapping to our measurements on
SKL. When performing this evaluation, we discovered two
bugs in their port mapping that were acknowledged by the
authors. We fixed these in the port mapping that is used for
our evaluation.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
length of experiments
0
5
10
15MAPE (%)
iaca
uops.info
Figure 6. Mean absolute percentage error (MAPE) of sim-
ulation with the port mapping from Abel and Reineke [1]
and with IACA [17] with respect to our measurements for
experiments of varying length
Figure 6 shows the mean absolute percentage error for the
simulation with the port mapping from Abel and Reineke
with respect to our measurements for varying experiment
length. For length 1, we use the set of all supported x86-64
instructions, whereas for larger lengths, we randomly sample
2,000 experiments from the set of all instruction multi-sets
of the appropriate size.
For small experiment lengths, we can see a low error show-
ing that the experiments behave as predicted by the processor
model. With increasing length of experiments, the accuracy
degrades. The lower prediction error of IACA [ 17] in Fig-
ure 6 indicates that with longer experiments, the influence
of factors such as non-optimal scheduling decisions that are
not covered in the throughput model (but by IACA) rises.
Overall, the error is small enough to justify the use of
measurement mechanism and throughput model.
PMEvo: Portable Inference of Port Mappings for Out-of-Order Processors by Evolutionary Optimization PLDI ’20, June 15–20, 2020, London, UK
5.3 Model Predictions
Directly measuring the quality of a port mapping is hindered
by the lack of ground truth for most processors. We therefore
assess the inferred port mappings by their ability to accu-
rately predict the measured throughput of port-mapping-
bound experiments. For each microarchitecture, we use a
different benchmark set of 40,000 experiments, which we in-
stantiate with operands and whose throughput we measure
as described in Section 4.2. These experiments are sampled
uniformly at random from the set of all instruction multi-sets
of size 5.
One major use case of PMEvo is to provide port mappings
for performance estimation tools. Therefore, we compare the
prediction accuracy of PMEvo’s mappings to the modeling
of port mappings in state-of-the-art performance prediction
tools. To this end, we use the same benchmark sets to evalu-
ate IACA [17] (version 3.0), llvm-mca [7] (from LLVM ver-
sion 8.0.14), Ithemal [20], and the port mapping provided by
uops.info [1] for their respective supported platforms. Note
that these benchmarks specifically stress the port-mapping
aspect of these prediction tools because they do not contain
any data dependencies. They are therefore not representa-
tive to evaluate the overall prediction quality of these tools
on compiler-generated code.5 Section 6 discusses the perfor-
mance estimation tools we evaluate in further detail.
Of these four related approaches, only the port mapping
from uops.info is directly comparable to PMEvo’s results
because it can only predict the throughput of instruction
sequences without data dependencies. The other approaches
are more general in that they can predict the throughput of
arbitrary instruction sequences, but might not be attempt-
ing to provide good accuracy for dependency-free code. For
example, Ithemal uses a neural network model trained via
supervised learning rather than an explicit port mapping
model. Being trained on collected basic blocks from entire
programs where dependencies are to be expected, accurate
predictions for dependency-free code might be outside of
the scope of Ithemal.
For all three platforms, we ran our PMEvo prototype with
a population size of 100,000 and an ε of 0.05 for congru-
ence filtering. Table 2 gives numbers on the time required to
benchmark throughputs for experiments and to infer a port
mapping for all considered platforms. It further shows that
the effectiveness of congruence filtering is considerable: The
relevant instructions are reduced by 53% to 69%. The low
number of different µops used in the inferred port mappings
indicates that PMEvo developed compact representations for
4Initially, we performed these experiments on the more recent LLVM ver-
sion 9.0.1 but found a severe regression in prediction accuracy on our
experiments compared to version 8.0.1.
5We refer to the BHive project [ 10] for an evaluation of their accuracy
for instruction sequences extracted from code generated for common
benchmarks.
all three platforms. The uops.info port mapping for SKL uses
12 different µops for the same set of instructions.
Table 2. PMEvo mapping characteristics
SKL ZEN A72
benchmarking time 20h 27h 74h
inference time 5h 21h 12h
insns found congruent 69% 53% 56%
number of µops 17 15 9
To provide a broad comparison of prediction accuracy,
we give results for the following commonly used accuracy
metrics:
•The Mean Absolute Percentage Error (MAPE) is a mea-
sure of the relative error of the simulation over mea-
surements.
•The Pearson Correlation Coefficient (PCC) describes
how closely the relation between simulation and mea-
surements can be described by a linear equation.
•The Spearman Correlation Coefficient (SCC) is a mea-
sure of rank correlation. A high rank correlation in-
dicates that if the measurement for one experiment
is smaller than for another experiment, its simulated
value is likely to be smaller as well.
The value range for PCC and SCC is [−1, 1], ranging from
negative correlation (−1) over no correlation (0) to maximal
correlation (1).
Additionally, we visualize the prediction accuracy of our
approach in comparison to related work in Figure 7 with a
heat map for each pair of architecture and prediction mecha-
nism. For each heat map, the experiments are considered as
data points with measured and predicted throughput. The
heat map shows the space of possible pairs of measured and
predicted throughput, split into 35 ×35 equally sized bins.
Each bin’s shade represents the number of experiments that
lie in it. Ideally, measurement and prediction agree, leading
to experiments close to the marked diagonal line. Experi-
ments below the diagonal indicate an under-estimation of
the throughput, those above are over-estimated by the pre-
dictor.
We discuss the represented data in detail in the following
sections.
5.3.1 SKL. For the Intel Skylake platform, we compare
the prediction accuracy of PMEvo to all afformentioned ap-
proaches: the port mapping from uops.info, IACA, llvm-mca,
and Ithemal using its publicly-available pre-trained network
for the Skylake microarchitecture.
The inputs for IACA, llvm-mca, and Ithemal consist of
the loop body of the experiments, unrolled to a length of
ten instructions so that operand allocation can avoid loop-
carried dependencies. For the entire set of experiments, we
PLDI ’20, June 15–20, 2020, London, UK Fabian Ritter and Sebastian Hack
0 5 10 15 20 25 30 35
0
5
10
15
20
25
30
35predicted cycles
PMEvo on SKL
0 5 10 15 20 25 30 35
0
5
10
15
20
25
30
35
PMEvo on ZEN
0 5 10 15 20 25 30 35
0
5
10
15
20
25
30
35
PMEvo on A72
0 5 10 15 20 25 30 35
0
5
10
15
20
25
30
35predicted cycles
llvm-mca on SKL
0 25 50 75 100
0
25
50
75
100
llvm-mca on ZEN
0 30 60 90 120 150
0
30
60
90
120
150
llvm-mca on A72
0 5 10 15 20 25 30 35
measured cycles
0
5
10
15
20
25
30
35predicted cycles
uops.info on SKL
0 5 10 15 20 25 30 35
measured cycles
0
5
10
15
20
25
30
35
IACA on SKL
0 5 10 15 20 25 30 35
measured cycles
0
5
10
15
20
25
30
35
Ithemal on SKL
100
101
102
103
104
Figure 7. Prediction accuracy on port-mapping-bound experiments . Each heat map relates predicted and measured throughput
in cycles per experiment. Points closer to the diagonal line indicate better predictions. The gray boxes group heat maps for the
same platform. The experiments were set up and measured as described in Section 4.2.
report the results of the tools for this input, divided by the
number of experiments in the unrolled loop body.
The accuracy metrics for the five tools under comparison
are listed in Table 3.
Table 3. Prediction accuracy measures for port-mapping-
bound experiments on SKL
MAPE Pearson CC Spearman CC
PMEvo 14.7% 0 .98 0 .85
uops.info 9.3% 0 .92 0 .88
IACA 8.0% 0 .86 0 .79
llvm-mca 9.7% 0 .87 0 .82
Ithemal 60.6% 0 .35 0 .54
IACA, llvm-mca, and uops.info all predict with an average
error of less than 10% with high correlation values. This
impression is confirmed by the corresponding heat maps
in Figure 7: Most of the experiments are close to the ideal
line. They also all show a cluster of experiments below the
diagonal line. These can be attributed to the family of bit
test instructions (BTx), for which the measurable throughput
does not agree with the throughput implied by the port usage
as confirmed by the measurements of Abel and Reineke [1].
Our approach, PMEvo, has a slightly higher relative error
than IACA, llvm-mca, and and uops.info, but comparable
correlation coefficients. The corresponding heat map in Fig-
ure 7 shows a distribution close to the diagonal line. The
BTx instructions that caused inaccuracies for the other ap-
proaches have a representation as multiple µops that map to
the same ports. While differing from the real port mapping,
this fits better to the observable throughputs.
For Ithemal, we observe lower correlations and a high
error rate. This differs from the evaluation by Mendis et al.
[20] where Ithemal exhibits superior results in these metrics
in comparison to IACA.6 As already noted, the difference in
6Their findings for the accuracy of IACA are consistent with the ones
presented here.
PMEvo: Portable Inference of Port Mappings for Out-of-Order Processors by Evolutionary Optimization PLDI ’20, June 15–20, 2020, London, UK
4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
number of ports
10−5
10−4
10−3
time/experiment (s)
bn algorithm
LP solver
(a)
1 2 3 4 5 6 7 8 9 10
length of experiments
10−5
10−4
10−3
time/experiment (s)
bn algorithm
LP solver (b)
Figure 8. Execution time comparison of the bottleneck simulation algorithm and the LP solver with varying port numbers
with experiments of length 4 (a) and with varying length of experiments with 10 ports (b). Both have their vertical axis in a
logarithmic scale.
Table 4. Prediction accuracy measures for port-mapping-
bound experiments on ZEN and A72
MAPE Pearson CC Spearman CC
PMEvo (ZEN) 13.5% 0 .94 0 .87
llvm-mca (ZEN) 50.8% 0 .86 0 .54
PMEvo (A72) 21.4% 0 .68 0 .77
llvm-mca (A72) 65.3% 0 .67 0 .68
performance is likely a consequence of the different charac-
teristics of the experiments used here and in the experimental
evaluation of their paper: Ithemal is trained and validated
on basic blocks emitted from a compiler for entire programs,
which exhibit substantially more data dependencies than our
experiments.
However, an appropriate interpretation of these results
needs to be judicious: A high prediction accuracy for our
experiments could have indicated a generalization of Ithemal
to dependency-free code. Yet, the observed low prediction
accuracy for our inputs does not allow conclusions about
Ithemal’s performance across real-world programs.
5.3.2 ZEN and A72. For the AMD and ARM microarchi-
tectures, we compare PMEvo’s results only to llvm-mca since
the other approaches are only available for Intel architec-
tures.
The metrics for both architectures in Table 4 show a com-
mon trend: PMEvo exhibits a considerably smaller prediction
error than llvm-mca.
For ZEN, PMEvo inferred a port mapping that predicts
with close to equal accuracy as its SKL mapping. With21.4%,
the prediction error of the PMEvo mapping for A72 is no-
tably higher while correlations are lower. This observation
is confirmed by the corresponding heat maps in Figure 7.
PMEvo on A72 is prone to under-estimating experiments
with longer running times. We attribute this to A72’s less
advanced out-of-order execution engine (according to the
respective optimization guides [2, 4, 16]), which renders the
experiments less representative for the port mapping.
In contrast to its results for SKL, llvm-mca has substan-
tially larger prediction errors. The heat maps indicate a sig-
nificant over-estimation of the throughput. One possible
explanation is that these architectures are less in the focus
of the developers than SKL and the respective port mapping
models might not yet be as elaborate and accurate as the one
for SKL. Especially for these two architectures, the models
derived with PMEvo may significantly increase the accuracy
of llvm-mca’s throughput prediction.
5.4 Performance of the Simulation Algorithm
This section explores the performance behavior of the bot-
tleneck simulation algorithm as presented in Section 4.5. For
this purpose, we compare our optimized implementation
of the bottleneck simulation algorithm to a realization of
the linear program from Section 3.2 in the state-of-the-art
LP solver Gurobi [14] (version 7.5.2). The running times re-
ported for the LP version include model construction via the
Gurobi C++ API as well as the actual solving.
There are two significant parameters that influence the ex-
ecution time of both simulation methods: the number of ports
in the microarchitecture and the length of the experiments.7
We evaluate these parameters with randomly generated mi-
croarchitectures with the appropriate number of ports for
an artificial instruction set architecture of 100 instructions.
For each (number of ports, length of experiments) configu-
ration, 128 randomly sampled experiments were simulated
with each of 8 randomly sampled three-level mappings. The
resulting seconds per experiment value for each pair of ex-
periment and mapping is the arithmetic mean over 1000
7The number of instructions in the instruction set architecture is not rele-
vant, since both implementations ignore instructions that do not occur in
the experiment.
PLDI ’20, June 15–20, 2020, London, UK Fabian Ritter and Sebastian Hack
simulations. The points in the graph mark the median of
these values for each (number of ports, length of experi-
ments) configuration.
Influence of the Number of Ports.Figure 8a shows the
results for experiments consisting of 4 instructions with a
varying number of ports. For port numbers up to 10 as they
occur in contemporary platforms, the bottleneck simulation
algorithm outperforms the linear program by two orders of
magnitude. Starting from 12 ports, the simulation time with
the bottleneck simulation algorithm rises with a stronger
incline. The bottleneck simulation algorithm reaches the
simulation time of the LP implementation at about 18 ports.
With the same inputs, the simulation time via the LP solver
grows substantially slower with the number of ports. We
conclude that the exponential run-time behavior of the bot-
tleneck simulation algorithm, as explained in Section 4.5, has
a negligible impact for inputs of interest.
Influence of the Length of Experiments.The experi-
ments we use for the evolutionary algorithm are of very
limited length to allow reliable execution on actual proces-
sors. Nevertheless, exploring the behavior with different
lengths of experiments is worthwhile for the discussion of
the bottleneck simulation algorithm. The results for varying
lengths of experiments in an architecture with 10 ports are
displayed in Figure 8b. Here, the bottleneck simulation algo-
rithm consistently outperforms the LP solver by two orders
of magnitude. The execution time for both methods grows
sub-exponentially with the length of experiments, with an
almost identical incline in the log-scale plot. This indicates
that the rate at which the execution time rises with growing
experiment length for the LP solver is considerably higher
than for the bottleneck simulation algorithm.
6 Related Work
We divide related work into two categories: Approaches
to find port mappings and work on predicting instruction
throughput.
6.1 Inferring Port Mappings from Experiments
The instruction tables by Fog [12] used to be the only avail-
able source for experimentally validated information on in-
struction latency, throughput, and port usage. They are ob-
tained with hand-crafted microbenchmarks that use hard-
ware performance counters to count the number of executed
cycles and the number of executed µops per port. Abel and
Reineke [1] show that the reported port usage by Fog is only
an under-approximation of the usable ports.
For the case that these counters are not available, Fog
uses experiments that execute instructions with unknown
port usage together with instructions whose port usage is
known from some other resource. Observing the running
time allows to identify interfering instruction combinations.
The tables include such information for a wide range of
x86 microarchitectures by Intel, AMD, and VIA. The require-
ment to construct suitable microbenchmarks for each mi-
croarchitecture makes this approach very work-intensive.
Abel and Reineke [1] automated the process of designing
microbenchmarks to measure latency, throughput, and port
usage. Their algorithm to estimate port usage overcomes the
imprecision of Fog’s approach by using blocking instructions.
The processor decomposes these instructions each into a
single µop that can only be executed on a known set of ports.
When executing the instruction under test with a sufficient
number of blocking instructions to fully saturate a set P of
ports, µops of the instruction under test will be executed on
ports not in P if possible. For observing this as well as for
identifying blocking instructions, they use per-port hardware
performance counters as they are used by Fog [12]. While
they provide throughput and latency measurements for x86
microarchitectures by Intel and AMD, they only give port
mappings for the Intel platforms as only these provide all
required performance counters.
Two further approaches initiated by Google are collected
under the name EXEgesis. One is the EXEgesis project [13]
that extracts latencies, throughputs, and port usage for Intel
architectures from vendor-provided documentation. This re-
quires automatically parsing documents that were intended
for human readers: a fragile and work-intensive process.
Since the provided documentation does not include all rele-
vant information, the EXEgesis developers also created tools
to infer the missing information via experiments. This led
to the second project under this name, llvm-exegesis [ 9],
a tool inside the LLVM framework [18] that automatically
generates benchmarks similar to those used by Fog [12]. For
measuring port usage, llvm-exegesis depends on per-port
performance counters just as the two previously discussed
approaches.
All of these works compare to ours in a similar way: Since
they use precise hardware performance counters, they can
obtain more accurate port mappings than our approach.
However, our approach does not suffer from the restriction
to platforms that have these performance counters, allowing
us to automatically infer port mappings for x86 platforms by
AMD, as well as for ARM platforms.
6.2 Work on Instruction Throughput Prediction
As port mappings are commonly used for throughput predic-
tion, it is instructive to set the presented results in context
to work from this field.
The Intel Architecture Code Analyzer (IACA) [17] mod-
els the execution of a sequence of instructions, considering
factors such as port usage, operand dependencies, and in-
struction decoding bottlenecks. The output of IACA for a
given instruction sequence includes a throughput estima-
tion, a bottleneck resource, and the distribution of µops to
ports. It is a closed-source tool provided by the processor
PMEvo: Portable Inference of Port Mappings for Out-of-Order Processors by Evolutionary Optimization PLDI ’20, June 15–20, 2020, London, UK
manufacturer Intel for some of its microarchitectures. As a
consequence, IACA can make use of unpublished internal
information to achieve an accurate performance prediction.
Nevertheless, previous research (e.g. [1]) has shown cases
where the prediction of IACA differs from the observable
behavior. Since April 2019, IACA is no longer under active
development.
OSACA [19] is an attempt to provide the same features as
IACA, but with a non-proprietary system. They use informa-
tion from port mappings for their supported architectures, a
range of Intel microarchitectures as well as AMD’s Zen archi-
tecture. These port mappings are extracted from sources like
the tables by Fog [12] and material provided by the manufac-
turers. They implement means of experimentally validating
this known port model via experiments, noting that experi-
ments with multiple different instructions can uncover new
details of the port mapping. Our approach systematically
extends this line of work to derive new port mappings.
The llvm-mca tool [ 7] is also inspired by IACA. It uses
knowledge from the LLVM [18] instruction scheduling mod-
els, including port usage if available, for performance pre-
diction. These scheduling models are the result of human
fine-tuning effort, proprietary knowledge contributed by
processor designers, and experiments via llvm-exegesis [9].
Both, llvm-mca and OSACA, can benefit from port map-
pings by PMEvo for microarchitectures without available
port mapping.
Ithemal [20] uses machine learning techniques for instruc-
tion throughput prediction. Similar to our approach, it only
needs a specification of the instruction set architecture under
test and a set of experiments labeled with measured through-
puts as an input. These labeled inputs are used as training
data for a hierarchical recurrent neural network based on
long short-term memory (LSTM) cells.
Ithemal is trained and validated on basic blocks that are
extracted from compiled benchmark programs. As a result,
Ithemal captures different aspects than our approach: PMEvo
focuses on experiments whose outcome is solely determined
by the port mapping whereas the predictions of Ithemal are
shaped by other factors such as data dependencies.
A drawback of the Ithemal approach is that the resulting
processor model can only be interpreted by evaluating it on
an instruction sequence. This is sufficient for certain appli-
cations like stochastic superoptimization [22]. However, in
applications like the backend of an optimizing compiler, enu-
merating and evaluating a large set of possible instruction se-
quences is prohibitively expensive. A compact port mapping
is more easily interpreted for constructing well-performing
instruction sequences as it clearly indicates which instruc-
tions have conflicting resource requirements.
7 Conclusion
This paper presents PMEvo, a framework to infer port map-
pings, i.e. compact and interpretable representations of a
modern processor’s ability to exploit instruction-level paral-
lelism. The inference is done by an evolutionary algorithm
that optimizes port mappings to explain the instruction
throughputs measured for specifically designed instruction
sequences. Using a novel bottleneck simulation algorithm
to evaluate the fitness of port mappings, PMEvo can explore
the large search space of possible mappings effectively.
We demonstrate PMEvo’s portability by inferring port
mappings for three different microarchitectures, two of which
are out of scope for previous automatic approaches. The high
prediction accuracy of the inferred port mappings shows that
PMEvo can make performance engineering tools more reli-
able for a wide range of hardware platforms.
A Correctness of the Bottleneck
Simulation Algorithm
The proof of correctness of the bottleneck simulation algo-
rithm presented here uses basic results from linear optimiza-
tion theory. For background and proofs on these results, we
refer to the textbook by Bertsimas and Tsitsiklis [6].
Let S(m, e)be defined as follows:
S(m, e):=
(Í{e(i)| Ports(m, i)⊆ Q}
|Q|
 Q ⊆P
)
With this notation, Equation 1 can be written as ˆtm(e)=
max S(m, e). The proof proceeds by showing that the result
ˆtm(e)of the bottleneck simulation algorithm is equal to the
throughput t∗
m(e)according to Definition 3 for any experi-
ment e and any (two-level) port mapping m := (I ∪·P, M).
We do so by showing that (I)t∗
m(e)is included in S(m, e)and
that (II) each element of S(m, e)is upper-bounded by t∗
m(e).
I. Let s be an optimal feasible solution of the linear pro-
gram. We denote the value of a variablex chosen in s by s[x].
Since s is optimal, there is a non-empty maximal set Q ⊆P
such that for all k ∈Q holds that
Õ
i ∈I
s[xik ]= s[t]= t∗
m(e) (2)
Without loss of generality, we assume that each instruction
that s executes on a port in Q can only be executed on ports
in Q, that is:
k ∈Q ∧s[xik ]> 0 ⇒Ports(m, i)⊆ Q (3)
If this is not the case for s, we can find a different solu-
tion s′with identical objective value that fulfills this con-
straint as follows: For every (i, k)such that s[xik ] > 0,Í
i ∈I s[xik ]= s[t], and Q′ := Ports(m, i)∩( P\Q), ∅, we
remove a sufficiently small part of the value for xik and add
PLDI ’20, June 15–20, 2020, London, UK Fabian Ritter and Sebastian Hack
it to the value of some xik ′ with k′∈Q′such that constraint
(B) is tight for neither of k and k′.8
By defining J := {i |Ports(m, i)⊆ Q}, we identify the
following equalities:
Õ
i ∈J
e(i)
(A)
=
Õ
i ∈J
Õ
k ∈P
s[xik ]
(D)
=
Õ
i ∈J
Õ
k ∈Q
s[xik ]=
Õ
k ∈Q
Õ
i ∈J
s[xik ]
(3)
=
Õ
k ∈Q
Õ
i ∈I
s[xik ]
(2)
=
Õ
k ∈Q
s[t]= t∗
m(e)·|Q|
The equality of the leftmost term and the rightmost term
proves that t∗
m(e)∈ S(m, e).
II. Let Q′⊆P and t′:= Í{e(i)| Ports(m, i)⊆ Q}/|Q|. We
assume t′> t∗
m(e)and show that this leads to a contradiction,
proving that t∗
m(e)is an upper bound to each element of
S(m, e).
For this argument, we form the dual of the linear program:
maximize
Õ
i ∈I
e(i)·yi
subject to yi −zk ≤mik for all i ∈I, k ∈P
Õ
k ∈P
zk = 1
zk ≥0 for all k ∈P
yi ≥0 for all i ∈I
Here, the yi and zk are real-valued variables and mik = 1 ⇔
(i, k)< M.
By the strong duality theorem for linear programs, an
optimal solution for this dual linear program has the same
objective t∗
m(e)as an optimal solution for the primal linear
program.
Given the assumption that t′ > t∗
m(e), we construct a
solution s′for the dual with a higher objective value, which
contradicts the strong duality theorem or the optimality of
t∗
m(e). The construction of s′is as follows for each i ∈I and
k ∈P:
s′[zk ]= 1/|Q′| if k ∈Q′
s′[yi ]= 1/|Q′| if Ports(m, i)⊆ Q′
All other variables are set to 0. This solution fulfills all con-
straints and has the following objective value:
Õ
i ∈I
e(i)·yi =
Í{e(i)| Ports(m, i)⊆ Q}
|Q| = t′> t∗
m(e)
This proves the correctness of the bottleneck simulation
algorithm. □
8If this was possible for all k ∈Q, s could not be optimal.
References
[1] Andreas Abel and Jan Reineke. 2019. uops.info: Characterizing Latency,
Throughput, and Port Usage of Instructions on Intel Microarchitec-
tures. In Proceedings of the Twenty-Fourth International Conference on
Architectural Support for Programming Languages and Operating Sys-
tems (Providence, RI, USA) (ASPLOS ’19). ACM, New York, NY, USA,
673–686. https://doi.org/10.1145/3297858.3304062
[2] AMD. 2017. Software Optimization Guide for AMD Family 17h Proces-
sors. Chapter 2, 17–44.
[3] AMD. 2019. Processor Programming Reference for AMD Family 17h
Models 01h,08h, Revision B2 Processors . Chapter 2.1.15, 160–176.
[4] ARM. 2015. Cortex®-A72 Software Optimization Guide . Chapter 2, 6–7.
[5] ARM. 2016. ARM® Cortex®-A72 MPCore Processor Technical Reference
Manual (Revision r0p3) . Chapter 11.8, 428–431.
[6] Dimitris Bertsimas and John Tsitsiklis. 1997. Introduction to Linear
Optimization (1st ed.). Athena Scientific.
[7] Andrea Di Biagio. 2018. llvm-mca: a static performance analysis tool.
http://lists.llvm.org/pipermail/llvm-dev/2018-March/121490.html
[8] James Bucek, Klaus-Dieter Lange, and Jóakim v. Kistowski. 2018. SPEC
CPU2017: Next-Generation Compute Benchmark. In Companion of the
2018 ACM/SPEC International Conference on Performance Engineering
(Berlin, Germany) (ICPE ’18). ACM, New York, NY, USA, 41–42. https:
//doi.org/10.1145/3185768.3185771
[9] Guillaume Chatelet. 2018. llvm-exegesis: Automatic Measurement
of Instruction Latency/Uops. https://lists.llvm.org/pipermail/llvm-
dev/2018-March/121814.html
[10] Yishen Chen, Ajay Brahmakshatriya, Charith Mendis, Alex Renda, Eric
Atkinson, Ondrej S`ykora, Saman Amarasinghe, and Michael Carbin.
2019. BHive: A Benchmark Suite and Measurement Framework for
Validating x86-64 Basic Block Performance Models. In 2019 IEEE inter-
national symposium on workload characterization (IISWC). IEEE .
[11] Kenneth A De Jong. 2006. Evolutionary Computation: A Unified Ap-
proach. MIT Press.
[12] Agner Fog. 2018. Instruction Tables: Lists of instruction latencies,
throughputs and micro-operation breakdowns for Intel, AMD and VIA
CPUs. https://www.agner.org/optimize/instruction_tables.pdf
[13] Google. 2018. EXEgesis: Automatic Measurement of Instruction La-
tency/Uops. https://github.com/google/EXEgesis
[14] LLC Gurobi Optimization. 2019. Gurobi Optimizer Reference Manual.
http://www.gurobi.com
[15] John L Hennessy and David A Patterson. 2011. Computer Architecture:
A Quantitative Approach . Elsevier.
[16] Intel. 2019. Intel® 64 and IA-32 Architectures Optimization Reference
Manual. Chapter 2.2, 2–6–2–12.
[17] Intel. [n.d.]. Intel Architecture Code Analyzer. https://software.intel.
com/en-us/articles/intel-architecture-code-analyzer
[18] Chris Lattner and Vikram Adve. 2004. LLVM: A Compilation Frame-
work for Lifelong Program Analysis & Transformation. In Proceedings
of the 2004 International Symposium on Code Generation and Optimiza-
tion (CGO’04). Palo Alto, California.
[19] Jan Laukemann, Julian Hammer, Johannes Hofmann, Georg Hager,
and Gerhard Wellein. 2018. Automated Instruction Stream Throughput
Prediction for Intel and AMD Microarchitectures. In 2018 IEEE/ACM
Performance Modeling, Benchmarking and Simulation of High Perfor-
mance Computer Systems (PMBS) . IEEE, 121–131.
[20] Charith Mendis, Alex Renda, Saman Amarasinghe, and Michael Carbin.
2019. Ithemal: Accurate, Portable and Fast Basic Block Throughput
Estimation using Deep Neural Networks. In Proceedings of the 36th
International Conference on Machine Learning (Proceedings of Machine
Learning Research) , Kamalika Chaudhuri and Ruslan Salakhutdinov
(Eds.), Vol. 97. PMLR, Long Beach, California, USA, 4505–4515.
[21] Kaisa Miettinen. 1999. Nonlinear Multiobjective Optimization . Vol. 12.
Springer Science & Business Media.
PMEvo: Portable Inference of Port Mappings for Out-of-Order Processors by Evolutionary Optimization PLDI ’20, June 15–20, 2020, London, UK
[22] Eric Schkufza, Rahul Sharma, and Alex Aiken. 2013. Stochastic Super-
optimization. In Proceedings of the Eighteenth International Conference
on Architectural Support for Programming Languages and Operating
Systems (Houston, Texas, USA) (ASPLOS ’13) . ACM, New York, NY,
USA, 305–316. https://doi.org/10.1145/2451116.2451150
[23] Robert M Tomasulo. 1967. An Efficient Algorithm for Exploiting
Multiple Arithmetic Units. IBM Journal of Research and Development
11, 1 (1967), 25–33.

1
Preprint version before issue assignment.
Iterative Schedule Optimization for Parallelization in the
Polyhedron Model
STEFAN GANSER, University of Passau
ARMIN GRÖSSLINGER, University of Passau
NORBERT SIEGMUND, Bauhaus-University, Weimar
SVEN APEL, University of Passau
CHRISTIAN LENGAUER, University of Passau
The polyhedron model is a powerful model to identify and apply systematically loop transformations that
improve data locality (e.g., via tiling) and enable parallelization. In the polyhedron model, a loop transformation
is, essentially, represented as an affine function. Well established algorithms for the discovery of promising
transformations are based on performance models. These algorithms have the drawback of not being easily
adaptable to the characteristics of a specific program or target hardware. An iterative search for promising
loop transformations is more easily adaptable and can help to learn better models. We present an iterative
optimization method in the polyhedron model that targets tiling and parallelization. The method enables either
a sampling of the search space of legal loop transformations at random or a more directed search via a genetic
algorithm. For the latter, we propose a set of novel, tailored reproduction operators. We evaluate our approach
against existing iterative and model-driven optimization strategies. We compare the convergence rate of our
genetic algorithm to that of random exploration. Our approach of iterative optimization outperforms existing
optimization techniques in that it finds loop transformations that yield significantly higher performance. If
well configured, random exploration turns out to be very effective and reduces the need for a genetic algorithm.
CCS Concepts: • Computing methodologies → Genetic programming; • Software and its engineering
→ Massively parallel systems ;
Additional Key Words and Phrases: Automatic loop optimization, genetic algorithm, OpenMP, parallelization,
polyhedron model, tiling
ACM Reference format:
Stefan Ganser, Armin Größlinger, Norbert Siegmund, Sven Apel, and Christian Lengauer. 2017. Iterative
Schedule Optimization for Parallelization in the Polyhedron Model. ACM Transactions on Architecture and
Code Optimization 1, 1, Article 1 (July 2017), 26 pages.
https://doi.org/10.1145/nnnnnnn.nnnnnnn
This work was partially supported by the German Research Foundation grant no. AP 206/4 and AP 206/6.
Authors’ addresses: Stefan Ganser, Armin Größlinger, Sven Apel and Christian Lengauer, Faculty of Computer Science and
Mathematics, University of Passau; Norbert Siegmund, Faculty of Media, Bauhaus-University, Weimar.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee
provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and
the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires
prior specific permission and/or a fee. Request permissions from permissions@acm.org.
© 2017 Association for Computing Machinery.
XXXX-XXXX/2017/07-ART1 $15.00
https://doi.org/10.1145/nnnnnnn.nnnnnnn
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
1:2 S. Ganser et al.
1 INTRODUCTION
Modern computer architectures are designed for parallel execution. Yet, writing efficient parallel
code by hand is cumbersome. Also, efficient parallel code is often platform-dependent [30]. As an
alternative, compilers should be able to automatically parallelize and tune sequential code written
in standard programming languages to exploit a given multi-core architecture. The polyhedron
model [12] supports this goal by providing scheduling algorithms that infuse parallelism and other
types of optimizations into loops. LLVM [22] with Polly [14] and GCC with Graphite [37] are
compilers that can perform polyhedral program optimizations in a model-driven way.
To improve the model-driven approaches, one must empower the underlying model to identify
profitable loop transformations. Iterative optimization is a way of traversing the search space of pos-
sible transformations and finding more profitable program transformations. Furthermore, iterative
optimization can help to learn about the distribution of the quality of individual transformations
in the search space of possible program transformations. Finally, search-based optimization can
adapt to a given hardware and is, therefore, better suited to find an optimal transformation for a
given architecture. The downside of iterative optimization is that, in most cases, it is more time
consuming than model-based optimization and the search space may be too large for exhaustive
exploration. Thus, it is often not practically applicable.
As it turns out, with its model-driven optimization, the speedup reached by Polly for 9 of the
30 benchmarks in PolyBench 4.1[35] is only less than or equal to 1.05 in our experiments1. Such
limitations in the optimization of program transformations motivate our approach of iterative
polyhedral optimization to find loop transformations that yield higher performance and to get a
better understanding of the characteristics of profitable transformations. Specifically, we enable
the sampling of the search space of legal transformations at random and a more directed search
via a tailored genetic algorithm, to identify faster schedules in a reduced amount of time. We aim
particularly at transformations that enable tiling and parallelization.
We base our iterative optimization on the approach by Pouchet et al . [32, 33] of iteratively
optimizing the sequential execution time of programs that can be represented in the polyhedron
model. Pouchet’s approach, which addresses only legal transformations, consists of an algorithm
that narrows the search space of all legal loop transformations to a specific subset. Furthermore,
Pouchet employs a decoupling heuristic for sampling the narrowed space at random with reasonable
effort. The decoupling heuristic is based on experience. To reduce optimization time, Pouchet et al.
present a genetic algorithm with custom genetic operators.
Pouchet’s approach is promising but does not support the detection of transformations that
enable tiling and parallelization. The main reason for narrowing the search space to a specific
region is the huge number of legal transformations that would need to be evaluated to find a fast
one. Later, Pouchet et al. extended their approach to iteratively explore the space of possibilities for
loop distribution and fusion and subsequently optimize each loop nest in a model-driven way [34].
The nature of the purely iterative approach causes a dilemma. To optimize for tiling and paral-
lelization, we must remove the restrictions narrowing the search space. Yet, the space is too large to
be modeled and traversed entirely. Its exponential size results from the numerous orders in which
data dependences may be carried [33]. We overcome the dilemma by dividing the search space
into regions and by sampling these regions. The selection of the regions is driven by chance, but
it can be configured such that outer parallel loops are likely to occur. To this end, we alter the
1For baseline measurements, we canonicalize the LLVM IR with -polly-canonicalize and then optimize the tar-
get SCoP with -march=native -basicaa -scev-aa -polly-opt-isl -polly-codegen -polly-parallel=true
-polly-tiling=true -polly-vectorizer=none -polly-default-tile-size=64 . Subsequently, we run -O3
-march=native. We use the Intel Xeon E-5 2650 v2 CPU @ 2.6GHzwith eight physical cores for benchmarking.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
Iterative Schedule Optimization for Parallelization in the Polyhedron Model 1:3
existing algorithm for search space construction to sampling regions of the search space of legal
transformations.
Pouchet’s decoupling heuristic for sampling assumes a sequential execution. Consequently, to
sample search space regions, we propose a novel strategy, relying on the Farkas-Minkowski-Weyl
Theorem and Chernikova’s algorithm (see Section 3). Here, it is important that our algorithm
does not rely on performance models but makes solely use of the search space regions’ geometric
structure.
Finally, we present a set of novel genetic operators for finding profitable transformations, because
the operators presented by Pouchet et al. [33] would not be able to traverse our unbounded search
space of all legal transformations of a program.
Our evaluation on PolyBench 4.1reveals that our approach outperforms existing model-driven
and iterative techniques in that it finds loop optimizations that yield significantly higher per-
formance of the resulting code. However, to our surprise, the transformations found by a well
configured random exploration are very profitable and narrow the advantage of the genetic algo-
rithm. From this, we conclude that profitable loop transformations can be detected more easily
than expected. Future research must decide whether the superior transformations found by our
iterative optimization share characteristics that can be incorporated by model-driven optimizers.
All evaluation results and our implementation are available on a supplementary Web site2.
In summary, we contribute the following:
•We extend Pouchet’s algorithm for search space construction to enable the optimization of
transformations that incorporate parallel execution with tiling.
•We propose a novel method to sample the search space of loop transformations. It employs
the Farkas-Minkowski-Weyl Theorem and Chernikova’s algorithm.
•We propose custom genetic operators to sample the search space of loop transformations
with more guidance than at random.
•We provide the tool Polyite, which relies on LLVM’s polyhedral code optimizer Polly to
model programs in the polyhedron model, apply tiling, and generate optimized code.Polyite
is written in Scala [29].
•We reimplemented the search space construction of Pouchet et al. [33] and combined it with
our sampling strategy.
•We evaluate our genetic algorithm against a contemporary variant ofPLuTo [5], a polyhedral
scheduling algorithm that optimizes specifically for tiling and data locality. Furthermore, we
compare our genetic algorithm against random exploration with our own search space con-
struction and sampling strategy, and against random exploration with our reimplementation
of Pouchet’s approach.
•We evaluate the execution time of our search space construction.
2 POLYHEDRON MODEL
Our proposed schedule optimization relies on several concepts of the polyhedron model and its math-
ematical foundation. We introduce them in this section and borrow from Schrijver [36][Chap. 7, 8].
The polyhedron model [12] is a mathematical abstraction of loop programs. Program transforma-
tions can be applied in the model, and source code can be generated from the model. The model’s
power lies in its ability to express a potentially complex sequence of individual loop transformations,
such as loop distribution/fusion [19], skewing [40], tiling [17], and index set splitting [13], in a
single affine function.
2https://stganser.bitbucket.io/taco2017/
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
1:4 S. Ganser et al.
for (i = 0; i < ni; ++i)
for (j = 0; j < nj; ++j) {
C[i][j] = C[i][j] ∗ beta; // Statement R
for (k = 0; k < nk; ++k)
C[i][j] = C[i][j] + alpha
∗ A[i][k] ∗ B[k][j]; // Statement S
}
j
0
1
2
k
1
2
i1 2 3
Fig. 1. On the left is a program with static control. On the right are the program’s iteration domains together
with data dependences. The crosses are the instances of statement R, the circles those of statement S. Of
each of the two dependence vectors, only the respective instance originating at the upper left front corner is
shown. The instances replicate throughout the entire domain.
Static Control. The polyhedron model puts restrictions on the kind of programs that can be
optimized. To be able to model a code region, it must have static control . A code region with static
control is called a static-control part or SCoP, for short. Loops should be for loops. A loop’s limits
must depend only on structure parameters, numerical constants, and iteration variables of the
enclosing loops. Structure parameters are integer variables whose values remain unchanged inside
a SCoP. The statements in a SCoP may operate on linear data structures (typically arrays). Loop
limits and array indices must be affine functions of the iteration variables and structure parameters.
Figure 1 shows a matrix multiplication as an example of a static-control program. A more restrictive
definition of a SCoP would require the stride of all loops to be one [9]. Wider definitions of static
control exist, too [3].
Iteration Domain. In the polyhedron model, each iteration or instance of a statement is repre-
sented by a point in a vector space whose dimensionality equals the number of nested loops. The
coordinates reflect the values of the iteration variables. We refer to the coordinate vectors as the
statement instances’ iteration vectors . In case of a legal SCoP, the set of all instances of a statement
S is a polyhedron {®x |A ·®x ≤®b}. The constraints that bound the set correspond to the original
loop bounds. Figure 1 depicts a program together with its iteration domains.
Convex Polyhedra. One representation of polyhedra is the constraint representation, which was
used in the previous paragraph. It is dual to the generator representation (Farkas-Minkowski-Weyl
Theorem), in which each point inside polyhedron P is expressed as a convex combination of a set
of points {®v1, ... , ®vm }, each lying on a different minimal face of P, plus a conical combination of
the rays {®r1, ... , ®rn }pointing in directions in which the polyhedron is unbounded: P = {®v + ®w |
®v ∈conv{®v1, ... , ®vm }∧®w ∈cone{®r1, ... , ®rn }}.
A line or bidirectional ray [23] combines two rays pointing in opposite directions. In the absence
of lines, ®v1, ... , ®vm are the vertices ofP. In the remainder of the paper, for simplicity of nomenclature,
we call the points ®v1, ... , ®vm vertices, irrespective of their true nature. We call the vertices, rays,
and lines generators of P. Chernikova’s algorithm [36][Part III] allows us to switch between both
representations. We reimplemented the extension of Le Verge [23].
Schedule. So far, we did not account for the execution order of the statement instances in a SCoP.
Iteration domains alone cannot express execution order, since they only contain the statement
instances’ iteration vectors. Particularly, they do not express the textual order. An additional
schedule is required. A statement schedule is a (multi-dimensional) affine function that assigns
an execution date to each of a statement’s instances. Given two instances of statements S and T
with iteration vectors ®i and ®j, the instance of S is executed before the instance of R, iff, for the
one-dimensional statement schedules ΘS and ΘR , ΘS (®i)< ΘR (®j)applies. In the case of a multi-
dimensional schedule (we may assume that all statement schedules have the same dimensionality),
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
Iterative Schedule Optimization for Parallelization in the Polyhedron Model 1:5
the lexicographic order (≺) of the result vectors specifies the execution order. Given two vectors
®v = (v1,v2, ... ,vn), ®v′= (v′
1,v′
2, ... ,v′
n)we have ®v ≺ ®v′⇔∃i : 1≤i ≤n ∧(∀j : 1≤j < i ⇒vj =
v′
j )∧vi < v′
i .
To express an n-dimensional schedule ΘS of statement S, we use schedule matrices . Let ®p be
the vector of all structure parameters and ®iS the vector of the iteration variables of the loops that
enclose S. ΘS is expressible by the matrix MΘS = (®m1
ΘS
, ... , ®mn
ΘS
)T with row ®md
ΘS
= (®λd
S, ®µd
S , νd
S ). ®λd
S
is the coefficient vector of ®iS ; ®µd
S is the coefficient vector of ®p; νd
S is the coefficient of the value 1.
Data Dependences. Other than by the iteration domain and schedule, a SCoP is characterized by
the memory accesses of its statements. A data dependence from one statement instance, the source,
to another, the target, exists if both access the same memory location and the schedule prescribes
that the former statement instance is executed before the latter. A dependence is relevant iff one of
the accesses is a write. Let statement O be the origin and T the target of a dependence. Dependence
polyhedron DO,T ⊆DO ×DT contains all pairs of statement instances of O and T for which the
dependence exists. In some cases, this polyhedron over-approximates the true set of dependent
statement instances. In the remainder of the paper, we treat the terms dependence and dependence
polyhedron as synonyms. If we speak of a single instance of a dependence (i.e., a pair of statement
instances), we always do so explicitly. Figure 1 depicts the data dependences in our example as
arrows from source to target.
Data dependences make many schedules illegal. A dependence’s source must be executed before
its target. Therefore, we introduce terminology regarding the legality of schedules. We start with one-
dimensional schedules. Dependence DO,T is weakly satisfied iff ∀(®i, ®j)T ∈DO,T : ΘT (®j)−ΘO (®i)≥ 0
applies. DR,S is strongly satisfied iff ∀(®i, ®j)T ∈DO,T : ΘT (®j)− ΘO (®i)> 0 applies. A set of one-
dimensional statement schedules is legal if it strongly satisfies all data dependences that exist in
the SCoP.
A set of multi-dimensional statement schedules of a SCoP is legal iff ∀(®i, ®j)T ∈DO,T : ΘT (®j)≻
ΘO (®i)holds for each dependence DO,T . The first schedule dimension, starting from dimension 1,
that strongly satisfies a dependence carries the dependence. Geometrically seen, the directional
vector of a dependence points forward in the direction of a dimension that strongly satisfies the
dependence. It either points forward or is orthogonal to a dimension that weakly satisfies the
dependence and points backward in the direction of a violating dimension.
Tiling. The loop transformation tiling improves data locality by grouping statement instances
that are geometrically close to each other into so-called tiles. Grouping the statement instances
changes their execution order. Tiling doubles the number of loops in the nest. The new outer loops
enumerate the tiles. The inner loops iterate inside the tiles. A rectangular tiling of a sequence of
schedule dimensions is legal iff the elements of the sequence are permutable: All dimensions must
weakly satisfy the same data dependences. Such a sequence is called a tilable band .
Often, tilable bands cannot be found directly. Then, skewing [ 40] may help as an enabling
transformation. Skewing shifts a loop’s limits by a multiple of an enclosing loop’s iteration variable.
This changes the directional vectors of dependences and can make (rectangular) tiling of a loop
nest legal.
3 SEARCH SPACE CONSTRUCTION
In contrast to Pouchet et al. [33], we sample the entire search space of legal affine schedule functions.
We divide the search space into regions, such that one region contains all schedules carrying the
data dependences in a specific order. Since the number of regions is exponential, we select only
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
1:6 S. Ganser et al.
a sample. Then, we can sample schedules from each selected region. We adapt Pouchet’s search
space construction accordingly.
We first outline Pouchet’s algorithm for search space construction. Subsequently, we present our
extensions for sampling different regions of the search space.
Schedule Representation. Schedules are represented as schedule matrices (see Section 2). Since
statement schedules cannot be chosen independently, an evident approach is to aggregate them to
one overall schedule of the entire SCoP. An n-dimensional schedule Θ of a SCoP with state-
ments S1, ... , Sk is represented by the schedule matrix MΘ = (®m1
Θ, ®m2
Θ, ... , ®mn
Θ)T with ®md
Θ =
(®λd
S1
, ... , ®λd
Sk
, ®µd
S1
, ... , ®µd
Sk
, νd
S1
, ... , νd
Sk
). We call the rows ®m1
Θ, ... , ®mn
Θ of MΘ schedule coefficient vectors .
Per statement, each row ®md
Θ contains a constant rational coefficient ®λd
Sj
for each of the iteration
variables of the loops surrounding the statement, ®µd
Sj
for each structure parameter, andνd
Sj
for each
constant. We must use rational coefficients due to the nature of our sampling strategy (see Section
4.2) and some of our genetic operators (see Section 4.3). Rational schedule coefficient vectors can
be transformed into equivalent integer vectors (see Section 4.2).
Legal Schedules. Since the number of illegal schedules grows exponentially faster with the
program size than the number of legal schedules [32], Pouchet et al. constrain the search space to
legal schedules. The constraints originate from the data dependences that exist in the SCoP. For
each data dependence DO,T , Pouchet defines two sets of schedule coefficient vectors. Set WO,T
contains all coefficient vectors that satisfy DO,T at least weakly. Set SO,T contains all coefficient
vectors that satisfy DO,T strongly. Both sets are polyhedra. The constraints that bound them can
be computed from DO,T with the affine form of Farkas Lemma [10]. The set of legal schedules is
never empty, as it contains the original schedule.
Size of the Search Space. Apart from restrictions that originate from the specific set of de-
pendences (e.g., it may not be possibly to carry two specific dependences in the same schedule
dimension), a dependence can be carried by any schedule dimension, assuming that all preceding
dimensions satisfy the dependence weakly. Depending on the choices made, the number of sched-
ule dimensions and the legality constraints of the dimensions vary. This creates a combinatorial
problem. To be precise, the size of the search space is exponential in the number of dependences.
To reduce the search effort, Pouchet considers only schedules that carry the data dependences in
a specific order. To achieve this, he sorts the dependences by a heuristic that involves primarily
the volume of data communicated between the dependent statements. The following paragraph
describes how the ordered list of dependences is processed.
Modeling the Schedule Search Space. The algorithm builds a model of the search space from a
list G of all data dependences in the SCoP. G is ordered according to the aforementioned heuristic.
The model consists of a list of polyhedra, with the ith polyhedron containing the possible schedule
coefficient vectors for schedule dimension i. As long as G is not empty, the algorithm adds another
schedule dimensiond. The algorithm creates a polyhedronPd that fills the entire schedule coefficient
space. Constraints that bound each dimension to the interval [−1, 1]are added to Pd . Then, the
algorithm intersects Pd with the sets WO,T associated with all DO,T ∈G. Now, Pd contains only
schedules that weakly satisfy the dependences in G. Subsequently, the algorithm iterates through
G and intersects Pd with the set SO,T of each dependence. If, for a dependence, the intersection is
not empty, Pd is updated to the result of the intersection and the dependence is removed from G.
Extending Pouchet’s Algorithm. Algorithm 1 outlines the extended search space construction.
Differences from the original algorithm are highlighted.
Pouchet’s algorithm for search space construction, described in this section, models a specific
set of schedules as a list of polyhedra. All data dependences are carried by the outer dimensions of
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
Iterative Schedule Optimization for Parallelization in the Polyhedron Model 1:7
all of these schedules. In other words, Pouchet’s algorithm permits only inner parallelism, which is
unacceptable if one optimizes for parallelization. Further, with parallelization and tiling in mind,
skewing is a promising loop transformation (see Section 2). Following Pouchet, and limiting the
value range of schedule coefficients to[−1, 1], we would eliminate schedules with non-unit skewing
from the search space. Also, many opportunities for loop distribution are lost. Yet, loop distribution
can expose parallelism [19, 30].
Example 3.1. To better understand the limitations of Pouchet’s algorithm, consider the SCoP
from Figure 1. The schedule ΘR (i, j)= (0, i, j, 0); ΘS (i, j, k)= (1, i, j, k)is profitable, since it yields
code where each statement is nested in a separate loop nest and where the outermost loop around
each statement can be parallelized. Yet, Pouchet’s approach would presumably produceΘS (i, j, k)=
(1, k, i, j)(or an equivalent schedule), since the k loop around S carries dependence DS,S . ◁
As mentioned, the full search space of legal schedules for a SCoP is dividable into regions, each
containing the schedules that carry the data dependences in one order. As we cannot model the
complete search space, we modify Pouchet’s search space construction to sample the regions.
Further, to allow for outer parallel loops, we must permit outer schedule dimensions that do not
carry dependences.
We propose the following adaptations: We start from a setG of dependences instead of an ordered
list. When constraining polyhedron Pd to schedules that carry data dependences, we consider only
dependences from a randomly chosen set Gd ⊆G. In our experiments we set the probability that
Gd is not empty to 0.4. Subsequently, we remove all dependences fromG that are effectively carried
by all schedules in Pd . Leaving Gd empty permits Pd to contain schedules that encode parallel loops.
To support non-unit skewing and easier loop distribution, we do not limit the value range of the
schedule coefficients.
ALGORITHM 1: Search space construction
Input: G Set of dependence polyhedra for dependences between pairs of statements R and S
U (Universe) A polyhedron that fills the entire schedule coefficient space of the SCoP.
Output: For each schedule dimension d, a triple Cd = (Vd , Rd , Ld )of generators that span the polyhedron Pd
containing the allowed coefficient vectors ®md
Θ for d
Parameters: p Probability that a schedule dimension carries, at least, one dependence
1 d ←0; // current schedule dimension
2 while G , ∅do
3 d ←d + 1; Pd ←U ;
4 foreach DO,T ∈G do
5 WO,T ←{®mΘ | ®mΘ ∈U ∧∀DO,T ∈G : ®mΘ weakly satisfies DO,T };
6 Pd ←Pd ∩WO,T ;
7 Gd ←∅; // dependences to carry in schedule dimension d
8 if rand([0, 1]) ≤p then
9 Gd ←rand(2G \{∅} );
10 foreach DO,T ∈Gd do
11 SO,T ←{®mΘ | ®mΘ ∈U ∧∀DO,T ∈G : ®mΘ strongly satisfies DO,T };
12 if Pd ∩SO,T , ∅then
13 Pd ←Pd ∩SO,T ;
14 G ←G \{DO,T |DO,T ∈G ∧∀®md
Θ ∈Pd : ®md
Θ strongly satisfies DO,T };
15 Cd ←chernikova(Pd );
Function rand selects randomly an element from a given set. Elements are chosen with uniform probability. The parts of
the algorithm that differ from Pouchet’s original search space construction are highlighted.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
1:8 S. Ganser et al.
Final Representation of a Search Space Region. Originally, the polyhedraPd are modeled in the
constraint representation. The constraint representation is convenient for intersecting polyhedra,
as done in Algorithm 1. To sample schedule coefficient vectors from these polyhedra, we convert
them to their generator representation. This representation is convenient for sampling schedules,
since Pd can be sampled by choosing a random combination of its generators. Specifically, each
polyhedron Pd is finally represented by a triple Cd = (Vd, Rd, Ld )of the vertices (Vd ), rays (Rd ), and
lines (Ld ) that span Pd . We compute the generators using Chernikova’s algorithm (see Section 2).
4 SEARCH SPACE EXPLORATION
So far, we have described how we sample regions of the schedule search space. Now, we present a
novel method of generating random samples from the generator representation of a search space
region.
In addition to using random exploration, we adapt the genetic algorithm by Pouchet et al. [33].
The general schema can be reused, but, as already mentioned, we require new mutation and
crossover operators. The existing operators are not applicable in our setting, since they do not
enable a traversal of the unrestricted search space. Similar to Pouchet, we weaken the changes
made by our mutation operators over time to stay inside promising search space regions at the end
of the optimization process.
4.1 Outline of the Genetic Algorithm
A genetic algorithm [26] (GA) is an iterative procedure starting from a random population of
candidate solutions. In our case, the candidate solutions are schedule matrices. GAs perform a
directed random walk and, thus, help to find good solutions faster. The iterations of the GA are the
algorithm’s generations.
Pouchet’s genetic algorithm starts from a random population of legal schedules. In each iteration,
the GA evaluates the current population, generating code from each schedule and measuring the
execution time of the code as the fitness function. The fitter half of the population survives. To
reach the full population size again, the schedules that survived from the parent population are
mutated and crossed. Using an annealing factor, Pouchet weakens the changes made by mutation
operators with time.
Pouchet’s genetic operators are not customized for specific loop transformations, but based
solely on the geometric structure of the search space. His narrowed search space is closed under
the mutation and crossover operators.
We follow the idea of not aiming for specific loop transformations, since it agrees with the
philosophy of polyhedral optimization not to apply a sequence of distinct transformations, but to
map each statement instance individually to an execution date. Our new genetic operators can
move from one search space region to another to reach schedules that are not in the search space
regions visited before. The set of legal schedules remains closed under the operators.
Further, we adapt the combination of the GA with the idea of simulated annealing [7, 16]. The
idea is to cover many different regions of the search space early in the optimization process. Later,
the changes to the schedules must diminish as the reachable optimum is approached. Yet, to avoid
being trapped at local optima, a small number of random schedules is introduced in each generation.
4.2 Generating Random Schedules
We continue from the list of triples [C1, ...,Cn]produced by Algorithm 1. Triple Cd consists of
the generators that span the polyhedron Pd , which matches a set of legal schedule coefficients
for schedule dimension d. Algorithm 2 takes the list of triples [C1, ...,Cn]and builds a random
schedule matrix MΘ = (®m1
Θ, ®m2
Θ, ... , ®mn
Θ)T . Row ®md
Θ of MΘ is a linear combination of the generators
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
Iterative Schedule Optimization for Parallelization in the Polyhedron Model 1:9
of Cd . The choice of generator coefficients is independent for each schedule dimension. The linear
combinations meet the criteria described in Section 2. Therefore, for each d, coefficient vector ®md
Θ
lies inside polyhedron Pd , and MΘ represents a legal schedule of the SCoP.
ALGORITHM 2: Construction of random schedules
Input: For each schedule dimension d, d = 1, ... , n, a triple (Vd , Rd , Ld )containing the vertices, rays, and lines that
span the polyhedron containing the legal schedule coefficients d
Output: A schedule matrix MΘ =

®m1
Θ, ®m2
Θ, ... , ®mn
Θ
T
Parameters: rRange, lRange: Bounds for the coefficients of rays and lines
1 for d ←1 to n do
2 ®md
Θ ←rand(Vd ); // Choose one vertex
3 Rd
Θ ←rand(2Rd ); // Choose random rays from Rd
4 (ηd
1 , ... , ηd
|Rd
Θ |
)← rand( [1, rRange]∩N|Rd
Θ |); // Choose coefficients for selected rays
5 ®md
Θ ← ®md
Θ + Í
®ri ∈Rd
Θ
ηd
i ·®ri ;
6 Ld
Θ ←rand(2Ld ); // Choose random lines from Ld
7 (ϑd
1 , ... , ϑd
|Ld
Θ |
)← rand( ([−lRange, lRange]∩N)\{ 0}|Ld
Θ |); // Choose line coefficients
8 ®md
Θ ← ®md
Θ + Í
®ld
i ∈Ld
Θ
ϑd
i ·®ld
i ;
9 MΘ ←  ®m1
Θ, ®m2
Θ, ... , ®mn
Θ
T ;
Function rand selects randomly an element from a given set. Elements are chosen with uniform probability.
The use of integer coefficients for the generators may lead to unreachable points inside the
polyhedra Pd . Without further effort, admitting rational coefficients for the generators would result
in rational schedule coefficients. Generally, this is not a problem since a schedule coefficient vector
with rational elements can be transformed to a vector with only integer elements by multiplying it
with the lowest common denominator (LCD) of its elements. Both impose the same relative order on
the statement instances. But multiplying with the LCD may lead to very large schedule coefficients,
which may trigger integer overflows in the generated code. Therefore, generator coefficients must
be small integers. As Example 4.1 illustrates, this requirement prevents Algorithm 2 from being
able to reach some schedules.
Example 4.1. The polyhedron P = {(i, j)|− i + j + 1 ≥0 ∧2 ·i + j −5 ≥0}is spanned by vertex
(2, 1)T and rays (−1, 2)T , (1, 1)T . Point (2, 2)T corresponds to the combination(2, 1)T +1/3·(−1, 2)T +
1/3 ·(1, 1)T . Rational coefficients are required for the rays to reach this point. Any multiple of (2, 2)
lies inside P as well. But the following equation cannot be solved as an integer linear program:
α ·(2, 2)T = (2, 1)T + β ·(−1, 2)+ γ ·(1, 1)T , α ∈Z, β,γ ∈N. So, in the context of scheduling, there
is no equivalent coefficient vector that can be computed only from the P’s generators and integer
coefficients. ◁
Some of the genetic operators described later on still introduce rational generator coefficients to
reach otherwise unreachable areas of the search space.
From experiments with the PolyBench 4.1benchmarks, we know that vertices with rational
coordinates are rare (see Table 1). The sum of all vertices’ coefficients in the linear combination
forming a schedule coefficient vector must be 1. Since integer schedule coefficients are preferable,
we use a single vertex as the basis of a randomly generated schedule coefficient vector. Then, we
add a uniformly distributed number of rays and lines. The maximum number of rays and lines
can be bounded to avoid a too high density of schedule matrices. The coefficients of the rays and
lines are chosen randomly from (0, rRange]∩N and ([−lRange, lRange]∩N)\{0}, respectively.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
1:10 S. Ganser et al.
(a)
 (b)
 (c)
 (d)
 (e)
Fig. 2. Illustration of schedule mutation and crossover operators: (a) dimension replacement; (b) prefix
replacement; (c) generator coefficient replacement; (d) row-based crossover; (e) geometric crossover
Permitting arbitrarily large coefficients seems to be not beneficial and, as already mentioned, may
lead to erroneous programs. Still, the intervals must not be too narrow, as this would exclude
profitable opportunities for loop distribution and non-unit skewing (shifting the limits of a loop
by a multiple of the iteration variable of an enclosing loop): again, we know from experiments
with PolyBench that the components of rays and lines are mostly from {−1, 0, 1}(see Table 1).
To distribute a larger number of statements across many loops without having to insert many
constant dimensions into the schedule or to skew with a bigger factor, we need an accordingly
wide range for the generators’ coefficients. We use [1, 3]for ray coefficients and [−3, 3]\{0}for
line coefficients.
Schedule Completion. The schedules that result from our sampling strategy, which consists of
Algorithm 1 and Algorithm 2, may not be injective. Entire loops may not be encoded explicitly.
Polyhedral code generators, such as isl [15], will choose one possible order and generate code
accordingly. Yet, without coding each loop into the schedules, the iterative optimization would not
have control over inner loops. Especially, we could not tile these loops. Thus, we append rows to
each randomly generated schedule matrix until we cannot find, for any of the statements, another
row that is linearly independent with the previous rows in the iteration variable coefficients. This
way, we ensure that each loop of the generated code is encoded explicitly in the schedule. We call
this process schedule completion . Be aware that our process differs from the schedule completion
described by Pouchet et al. [33].
The randomly generated schedule matrices contain degenerate rows. For instance, rows occur
that encode loops with a single iteration. Also, different schedule matrices can encode the same
execution order. Consequently, before analyzing schedules or further transforming them, the
schedules should be simplified. Section 5 describes the details of schedule simplification. We do
not alter the original schedule matrices on which the GA operates, except for removal of rows
that do nothing but assign the same constant offset to all statements. We also do not eliminate
pairs of equivalent schedules from the population, because their identification is computationally
expensive.
4.3 Schedule Mutation and Crossover
The design of suitable mutation and crossover operators is an essential challenge of crafting a
genetic algorithm [7]. In our case, it is especially important to be able to reach any region of the
search space of legal schedules.
The goal of mutation and crossover is to generate an offspring from one or two previous schedules.
The new schedule should retain some properties of its ancestors but differ in others. The operators
must be able to produce target schedules that satisfy the dependences in a different order than
the source schedules. This reaches schedules from search space regions that have not been visited
before. The operators must also be able to visit schedules that cannot be produced by Algorithm 2,
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
Iterative Schedule Optimization for Parallelization in the Polyhedron Model 1:11
due to the algorithm’s restriction to integer coefficients for the Chernikova generators. Analogous
to random schedule generation, each schedule that results from a mutation or crossover is refined
by schedule completion.
Figure 2 depicts our mutation and crossover operators graphically. Let us look at each operator
in detail. We start with the mutation operators.
A mutation operator maps a schedule matrix MΘ to a new matrix MΨ with the same number of
columns, but possibly a different number of rows.
Dimension Replacement. The operator replaces randomly chosen rows of a schedule matrix
with randomly generated ones (see Figure 2(a)). The idea is to eliminate inefficient rows from a
generally profitable schedule.
The procedure iterates across the rows of the original schedule matrix and either copies an
existing row to the new matrix or generates a new row. For replacing row i, we perform one
iteration of Algorithm 1. Here, G is the set of dependences that are not carried by rows 0, ... , i −1
of MΨ. Gd is fixed to the dependences that are carried by the row to be replaced, except for those
carried by rows 0, ... , i −1 of MΨ. From the resulting polyhedron, we sample a new coefficient
vector with an iteration of Algorithm 2. Here, Algorithm 2 is configured as during the generating
of the initial population. If rows 0, ... , i −1 of MΨ carry all dependences already, we generate the
new row analogously to schedule completion (see Paragraph “Schedule Completion”). The set of
legal schedules is closed under dimension replacement by the definition of the operator.
Simulated annealing controls the number of replaced schedule dimensions.
Schedule Prefix Replacement. This operator replaces a prefix of a schedule matrix, that is, the
first rows (see Figure 2(b)). The prefix length is chosen randomly. To generate the new prefix, we
run the familiar sequence of Algorithms 1 and 2, but modify Algorithm 1 such that it chooses the
sets Gd as subsets of the set of dependences carried by the original schedule prefix. The modified
algorithm terminates, as soon as all of these dependences are carried. The new schedule prefix
carries all dependences that are carried by the replaced prefix. Thus, the new schedule is legal.
By coincidence, the new prefix may carry more dependences than the original prefix. The new
schedule may be located in a different region of the search space. Schedule prefix replacement can
particularly alter the partition of statements into loop nests.
Schedule Suffix Replacement. To generate a new suffix of a schedule matrix, we drop rows from
the bottom of the matrix and generate the new suffix by running Algorithms 1 and 2. Algorithm 1
starts from the set of dependences that are not carried by the remaining schedule dimensions.
In case of schedule prefix replacement and schedule suffix replacement, simulated annealing controls
the length of the replaced prefix or suffix.
Generator Coefficient Replacement. The idea is to replace a row of a schedule matrix by a very
similar row (see Figure 2(c)). This operator can, for instance, reverse a loop by negating an iteration
variable’s coefficient. The operator is similar to dimension replacement, but its effects are more
fine-grained.
Each row of a schedule matrix is stored together with the linear combination of the generators
that form it. The generators originate from a set of generators that span a set Pd of schedule
coefficient vectors. All coefficient vectors in Pd weakly or strongly satisfy defined sets of depen-
dences. Replacing coefficients of the generators in accordance with the constraints given in Section 2
yields a new coefficient vector that again satisfies these dependences. Dependences that the original
row carries by coincidence may not be carried by the new row. Replacing generator coefficients
with rational numbers allows us to reach schedule coefficient vectors that cannot be reached by
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
1:12 S. Ganser et al.
Algorithm 2. The denominators of the new generator coefficients are small. Otherwise, coefficients
of an equivalent schedule with integer coefficients will be too large.
Interactions with other genetic operators and the way in which random schedules are generated,
may cause this operator to produce illegal rows. In this case, the suffix of the new schedule matrix
must be replaced starting from the first illegal row.
Simulated annealing controls the number of mutated rows and for each mutated row, the portion
of generator coefficients that change.
Next, we describe our crossover operators. A crossover operator takes two schedule matrices MΘ1
and MΘ2 and produces a new schedule matrix MΨ.
Row-Based Crossover. As illustrated by Figure 2(d), row-based crossover recombines the rows of
MΘ1 and MΘ2 to produce MΨ. The idea is to inject new rows that are produced by other operators
into more already profitable schedules. For each dimension, the operator chooses a row either
from MΘ1 or from MΘ2 . The process starts from the first row and continues for the subsequent
dimensions. The iteration terminates as soon as the new schedule carries all dependences. Due to
the choices already made for rows 1, ... , i −1, it may be illegal to choose row i from MΘ1 or MΘ2 , as
that row may not weakly satisfy one of the dependences that are not carried by rows 1, ... , i −1 of
MΨ. In this case, we have to step back to row i −1 and make a different choice there. If all choices
for row i −1 have been found to be illegal, we have to step back further. The idea for a uniform,
row-based crossover originates from Pouchet et al. [33].
Geometric Crossover. The idea of geometric crossover is to take row d from both MΘ1 and MΘ2
and construct a new row®mΨ
d as a linear convex combination of the two existing rows (see Figure 2(e)).
This can reach schedule coefficient vectors that cannot be reached by a random exploration with
Algorithm 2.
We rely on the fact that a linear convex combination of two schedule coefficient vectors weakly
satisfies all dependences that are weakly satisfied by both of the original schedule coefficient vectors.
This kind of crossover is legal under the following condition. Let U d
Θ1
be the set of schedules that
are not carried by dimensions 1, ... , d −1 of Θ1. U d
Θ2
is defined analogously. If U d
Θ1
⊆U d
Θ2
, then
(®m1
Θ1
, ... , ®md−1
Θ1
, ®md
Ψ)T is a legal schedule prefix for a new schedule matrix MΨ, since ®md
Ψ weakly
carries all dependences in U d
Θ1
. We must complete MΨ with a new suffix. Generating the new suffix
is analogous to schedule suffix replacement after the removal of the old suffix.
Generator coefficient replacement and geometric crossover are able to reach schedules that cannot
be reached by Algorithm 2. On the other hand, they may produce schedule matrices with rational
coefficients. To actually generate code from these schedules, the rows must be replaced by equivalent
rows with integer coefficients. As outlined in Section 2, we do this by multiplying each row with
the lowest common denominator of its components before passing a schedule to the polyhedral
code generator.
5 SCHEDULE TREE GENERATION AND SIMPLIFICATION
Schedule matrices are convenient for sampling the schedule search space but less suitable for
transformation and analysis of schedules. To apply tiling, one considers each loop nest individually.
This requires the identification of the partial order induced by a schedule dimension on the set of
statements. Unlike schedule matrices, schedule trees [15] can express this property directly. Further-
more, since our schedule matrices are randomly generated, they contain coefficients that influence
neither the execution order of statement instances, nor the directional vectors of dependences. Yet,
they complicate the identification of tilable schedule bands and should be removed.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
Iterative Schedule Optimization for Parallelization in the Polyhedron Model 1:13
We use only some of the several node types of a schedule tree. A sequence node has a list of child
nodes and specifies that its children must be executed in the given order. Each child is preceded
by a filter node , which selects the statement instances that are scheduled in the subtree. A band
node contains a partial (multi-)dimensional schedule for the statement instances that are scheduled
in the present (sub-)tree. A band node may be marked as permutable to signal that its schedule
dimensions may be tiled.
The schedule tree construction is recursive. Starting from the first schedule dimension, we test
whether the current dimension partitions the set of statements into sets with different execution
date. We must consider only the order of statement instances that have the same execution date
according to previous dimensions. If the partitioning has more than one element, each element
corresponds to a subtree of a new sequence node. We split the iteration domain and the schedule
matrix accordingly and continue recursively for each partition or child of the sequence node.
If a schedule dimension does not result in a sequence node, we create a one-dimensional band
node and continue with the next dimension. Partial schedules of band nodes are simplified by
removing constant offsets that apply to every statement and dividing the schedule coefficients
by their greatest common denominator. Band nodes are obsolete if any two statement instances,
that have the same execution date in the previous schedule dimensions, are also assigned the
same execution date by the current band node’s schedule. An entire subtree can be removed if
its ancestors describe an injective schedule of the statement instances scheduled in the subtree.
Finally, we group consecutive one-dimensional band nodes into multi-dimensional band nodes if
their partial schedules are permutable.
6 EVALUATION
We aim at finding more profitable schedules than can be found by model-based scheduling algo-
rithms such as PLuTo. Furthermore, we expect that our approach also outperforms the existing
iterative optimization technique of Pouchet et al. in terms of profitability of the detected loop
transformations, since they focus on sequential execution. With our proposed genetic operators,
we expect to find more profitable schedules than with random search and find them faster. We
define the following research questions to shed light on our expectations and quantify the benefits
of our approach:
RQ1. Does our approach find loop transformations that yield higher performance than the transforma-
tions found by state-of-the art approaches to polyhedral schedule optimization? We evaluate whether
our iterative search produces similar or better program schedules than state-of-the art approaches.
Answering this question gives us a baseline and demonstrates the practicality of our approach.
RQ2. Is our augmentation of the schedule search space justified? In contrast to Pouchet, we explore
potentially the entire space of legal program transformations. We evaluate whether this expansion is
actually meaningful (in terms of finding better schedules) as it complicates the optimization process
considerably. This question relates to Section 3. The question also concerns schedule completion as
described in Section 4.2.
RQ3. Does optimization with our genetic algorithm have an advantage over random sampling? We
evaluate whether a simple random search on the space of legal schedules can find schedules
that perform equally well as our genetic algorithm. Again, answering this question provides a
comparison to a baseline, which is often used for related problems due to their simplicity. The
question relates to Section 4.
RQ4. Do schedule matrices with high sparsity yield better performance than dense matrices? This
question sheds some light on the decisions made for the configuration of the experiments. We can
control the maximum number of rays and lines that Algorithm 1 adds to a vertex to form a schedule
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
1:14 S. Ganser et al.
Fig. 3. Our tool chain for iterative schedule optimization
coefficient vector. Here, we explore whether it is better to add only a small number of rays and
lines, which yields schedule matrices with high sparsity, or whether we should leave the number
unspecified.
RQ5. Does our iterative approach scale for very large SCoPs? To evaluate whether our approach
scales for realistic settings, we analyze the scaling factors of Algorithms 1 and 2 with respect to the
size of SCoPs.
RQ6. Do similar schedules yield similar performance? The use of simulated annealing in the design
of our GA is based on the assumption that similar schedules yield similar performance (see Section
4.1). So, we evaluate whether this assumption is reasonable.
RQ7. Is our configuration of the genetic algorithm justified? The presence of many tuning options is
an intrinsic characteristic of iterative optimization tools. Their number makes it hard to find an
optimal setup. Hence, we investigate whether the setup that we have chosen for the evaluation of
our tool is actually well performing.
To answer these questions, we conducted experiments onPolyBench 4.1. PolyBench is a bench-
mark suite that is widely used in the polyhedral community. First, we describe our experimental
setup. Then, we present the results of the experiments and answer the research questions. Finally,
we discuss threats to validity.
6.1 Experimental Setup
We implemented our approach in the tool Polyite [p@"l¯ It]. Polyite relies on Polly for SCoP
extraction, modeling of SCoPs, tiling, and code generation. We handle polyhedra with the Integer
Set Library (isl, commit cfebc0c6) [39]. Polyite uses libbarvinok (version 0.39) to mimic the
behavior of Pouchet’s approach. Precisely, we use Barvinok’s counting algorithm to calculate the
communication volume of data dependences. Polyite is written in Scala (version 2.11).
Tool chain. Polly is a polyhedral optimizer built on top of LLVM. Polly extracts the model
of valid SCoPs from programs. Polly can apply externally generated schedules to SCoPs. The
used data format is JSCOP, which is based on isl union maps. Internally, Polly uses the schedule
tree representation. Polly is unable to tile imported schedules, though. We extend JSCOP with
a schedule tree representation and enable Polly to tile imported schedules. Figure 3 provides an
overview of our tool chain.
LLVM supports stepwise compilation. The LLVM C-/C++-front-end clang transforms C into
LLVM IR. Optionally, the IR code can be written to a file. LLVM’s opt tool allows us to apply
individual transformations and analyses to LLVM IR. We extract the polyhedral model of the
target SCoP into a JSCOP file using Polly’s SCoP detection. Then, Polyite loads the JSCOP file
and generates schedules. It generates a binary for each schedule, using Polly/ opt, llc, and gcc.
llc transforms LLVM IR into assembly code. Polly tiles the schedules and generates OpenMP-
parallelized loops. In our experiments, measured execution time of the optimized SCoP serves as
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
Iterative Schedule Optimization for Parallelization in the Polyhedron Model 1:15
Table 1. Characteristics of the benchmarks used. The data in columns 6-9 originates from the analysis of
P1 of 1000 search space regions per benchmark. The data in column 10 is from an analysis of the schedule
coefficient vectors produced during random exploration in Exp. 3.
benchmark # stmts # deps max. loop
depth
# structure
params
% unit
vectors
avg. #
lines
avg. #
rays
avg. #
vertices % rational vertices
2mm 4 6 3 7 99.84% 9 19 1 1.00%
3mm 6 10 3 9 99.90% 10 31 1 0.90%
adi 9 64 3 3 83.06% 4 27 4 2.02%
bicg 2 4 2 3 100.00% 4 6 1 0.67%
correlation 13 22 3 3 96.26% 13 40 1 0.54%
covariance 7 12 3 3 99.50% 4 26 1 0.50%
doitgen 3 8 4 5 99.02% 6 1 1 0.16%
fdtd-2d 4 24 3 4 92.08% 5 32 3 10.16%
gemm 2 2 3 5 100.00% 8 7 1 1.78%
gemver 4 6 2 2 99.72% 3 15 1 1.17%
gesummv 3 3 2 2 100.00% 4 7 1 0.56%
heat-3d 2 186 4 2 47.34% 3 15 1 16.69%
jacobi-2d 2 56 3 3 67.64% 4 6 1 12.42%
mvt 2 2 2 2 100.00% 8 2 1 2.14%
seidel-2d 1 59 3 3 85.44% 4 3 1 9.18%
syr2k 2 2 3 4 100.00% 7 7 1 1.94%
syrk 2 2 3 4 100.00% 7 6 1 2.30%
trmm 2 4 3 4 83.95% 6 9 1 2.15%
the fitness of the respective schedules. We use LLVM 3.9and gcc 5.4. Our extended version of
Polly is based on commit 2b618e01 of http://llvm.org/git/polly.git.
Benchmarking of Schedules. We benchmark each binary five times on an Intel Xeon E-
5 2650 v2 CPU @ 2.6GHzwith eight physical cores and 20MB of L3 cache. We take the short-
est measured execution time. Hyperthreading and Intel Turbo Boostare disabled. The OS is
Ubuntu 16.04with Linux kernel 4.4. Schedules yielding invalid computation results are purged.
Compilation must not exceed five minutes. The overall benchmarking of a schedule may take 30
minutes. These bounds are estimated from the baseline measurements and preliminary experiments.
Of the 30 programs in PolyBench 4.1, we evaluated 27. Polly does not manage to identify
the SCoP in program nussinov. Presumably, the functions that are called in the SCoP would
have to be declared as free of side-effects. We also excluded the programs lu and ludcmp from
the experiment because of their extraordinarily long startup times. We do not show the results
for nine of the remaining benchmarks, since none of the evaluated model-driven and itera-
tive optimization methods produced a speedup during preliminary experiments. We used the
largest possible data set that is configurable in PolyBench 4.1. For smaller sizes, the entire data
tends to fit into the L3-cache. Table 1 shows characteristics of the PolyBench 4.1benchmarks
that we considered in the evaluation. The relevant compiler flags that we use to generate opti-
mized LLVM IR are -polly-parallel=true -polly-vectorizer=none -polly-tiling=true
-polly-default-tile-size=64 -march=native.
Adaptation of Pouchet’s Algorithm. To compare Polyite against the iterative optimization
of Pouchet et al. [33], we reimplemented their search space construction and combined it with
our sampling strategy. A direct use of their implementation LeTSeE [31] was not possible, due
to the incompatibility of its file format with Polly. Furthermore, to the best of our knowledge,
statement iteration domains that can be expressed only as the union of multiple polyhedra cannot
be represented with LeTSeE’s input format. Polly and Polyite support these. We do not rely on
the original sampling strategy, since it is based on assumptions that may not hold in the context of
tiling and parallel execution.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
1:16 S. Ganser et al.
To meet the characteristics of Pouchet’s approach, we must adapt our sampling strategy. Before
calculating the generator representation of the polyhedraPd , we drop the constraints that bound the
schedule coefficients to {−1, 0, 1}. This makes the polyhedra unbounded. The modified Algorithm 2
uses coefficients from {−1, 1}for the generators and rejects schedule coefficient vectors with
coefficients that are not from {−1, 0, 1}.
Configuration. An important configuration option of Polyite is the maximum number of rays
and lines that may form a schedule coefficient vector. We use three settings: (1) If at most two rays
and two lines may be added to a vertex to form a schedule coefficient vector, the schedule matrix is
sparse, because many coefficients are zero. We call this thesparse setting. (2) If the allowed number
of rays and lines is only limited by their total number, the matrices tend to be dense, which we call
the dense setting. (3) The mixed setting produces sparse and dense matrices. These settings apply
to the generating of coefficient vectors by random exploration as well as by genetic operators.
A run of the GA has 40 generations. The initial population is produced randomly using the sparse
setting of random exploration. The regular population size is 30 schedules. A larger number of gen-
erations or a bigger population size conflicts with our costly fitness function. As, in one generation,
we replace half of the population, the GA visits at most 630 schedules. Random exploration visits
the same number of schedules. To ensure a good distribution of randomly sampled schedules, we
sample just one schedule from the search space region currently visited. Still, random exploration
may enter a search space region repeatedly.
The selection probability of the genetic operators is uniform: experimental evaluation did not
reveal that some operators are more effective than others. The mutation operators are configured to
mutate 10% of a schedule’s dimensions at the start of the optimization process, but at least one row.
Accordingly, generator coefficient replacement mutates 10% of the line, ray, and vertex coefficients
at the beginning. Since, for schedule matrices with less than 10 rows, this always results in one
row to be modified, the effect of simulated annealing diminishes for small SCoPs. The annealing
function f (p, д)= p/
p
ln(д+ e −1)calculates the fraction of schedule dimensions to mutate from
the currently generated generation дand the initial fraction p. Geometric crossover produces up
to three schedules at once. The other operators produce just one schedule. The special role of
geometric crossover is justified, since it produces otherwise unreachable rows. In each generation
of the GA, we introduce two randomly generated schedules. The chosen configuration is the result
of preliminary experiments. We challenge some of the choices in the research questions.
6.2 Experiments
Exp. 1 (RQ5). Performance of Search Space Construction . The execution time of Algorithm 2 for
sampling the search space regions is in O(d ·nд ·dim(CoeffSpace)), where d is the number of
dimensions of the region, nд is the maximum number of generators that may form a schedule
coefficient vector, and dim(CoeffSpace)is the dimensionality of the schedule coefficient space. The
algorithm has polynomial execution time, under the assumption that nд is limited by a constant
upper bound like in the sparse setting. Algorithm 1 employs Chernikova’s algorithm. The size of
a polyhedron’s generating system is exponential in the number of its bounding inequalities [10].
This dominates the algorithm’s run-time complexity. So, to decide whether our approach scales for
large SCoPs, one must evaluate the actual execution time of Algorithm 1 depending on the number
of data dependences.
When modeling a search space region, the maximum number of constraints that bound the set
of possible schedule coefficient vectors for each schedule dimension grows linearly in the number
of data dependences. So, we expect the execution time of Algorithm 1 to be exponential in the
number of data dependences. To quantify the run-time complexity of Algorithm 1 empirically, we
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
Iterative Schedule Optimization for Parallelization in the Polyhedron Model 1:17
10-2
10-1
100
101
102
103
104
105
 100  150  200  250  300  350  400
seconds
# dependences
min total
median total
max total
min Chernikova
median Chernikova
max Chernikova
(a) seidel-2d
10-2
10-1
100
101
102
103
104
105
 60  70  80  90  100  110  120  130
seconds
# dependences
min total
median total
max total
min Chernikova
median Chernikova
max Chernikova (b) jacobi-2d
Fig. 4. Minimum, maximum, and median of the execution time in seconds of Algorithm 1, depending on the
number of data dependences. The y-axes are logarithmic.
had to scale the number of data dependences of a SCoP. This can be done by partially unrolling
the SCoP’s loops [38]. For the experiment, we used the benchmarks seidel-2d and jacobi-2d of
PolyBench 4.1. Both are two-dimensional stencil codes surrounded by a time loop. We unrolled the
outermost loop of each stencil. Per number of unrolled iterations, we ran Algorithm 1 five times and
measured the total execution time and the time spent by Chernikova’s algorithm. Figure 4 shows the
results. The effort grows exponentially (note the logarithmic y-axes) and Algorithm 1 is dominated
by Chernikova’s algorithm. The measurements were conducted on a different CPU (Intel Xeon E5-
2690 v2 CPU @ 3.00 GHz). We used the same Ubuntu 16.04with an Oracle 8 JDK (version 8u101).
Balev et al. [2] evaluated the usefulness of the geometric representation of polyhedra in the context
of schedule optimization. Their work gives explanations to the observed domination of Algorithm 1
by Chernikova’s algorithm.
Exp. 2 (RQ3, RQ4). Convergence Rate of Iterative Optimization . We compare the number of sched-
ules that different variants of iterative optimization have to visit, on average, to find the optimal
reachable schedule. We compare the following configurations: (1) random exploration with the
adaptation of Pouchet’s approach with the sparse setting, (2) random exploration with the sparse
setting, (3) random exploration with the dense setting, and (4) our genetic algorithm with the
sparse setting. Finally, (5) we evaluated configuration (1) but with schedule completion. We selected
the benchmarks 3mm, correlation and adi. They belong to different categories of PolyBench.
They are promising candidates for this experiment, as the number of generators in the generator
representation of their search space regions is high (see Table 1). Thus, an actual difference between
sparse and dense setting can be expected. Apart from this, other benchmarks, for instance gemm
and jacobi-2d, are not interesting for a detailed evaluation as, for them, the GA cannot surpass
the performance of the initial population. To leverage the effect of an incomplete exploration, we
applied each configuration 10 times to each benchmark. To reduce the benchmarking effort, we ran
the GA only for 20 iterations. Accordingly, the random exploration runs generated 330 schedules.
Figure 5 shows the median speedup over O3 that was reached by each optimization strategy after
evaluating n schedules. The x-axis is n. The y-axis is speedup.
There is no clear difference between the convergence rate of GA and random with the sparse
setting. Both significantly outperform random exploration with the adapted approach of Pouchet
et al. [33] and random with the dense setting. The GA yields slightly better schedules than sparse
random. The effect of schedule completion on Pouchet’s approach remains unclear.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
1:18 S. Ganser et al.
0.0625
0.125
0.25
0.5
1
2
4
8
16
0 50 100 150 200 250 300
speedup over O3
number of visited schedules
GA
random sparse
random dense
random LeTSeE
+ completion
random mixed
random LeTSeE
O3
isl
(a) 3mm
0.25
0.5
1
2
4
8
16
32
64
0 50 100 150 200 250 300
(b) correlation
0.5
1
2
4
8
0 50 100 150 200 250 300
(c) adi
Fig. 5. Convergence rate of different configurations
of iterative optimization. Thex-axis shows the num-
ber of schedules already visited. The y-axis shows
speedup over O3. For convenience, also the speedup
reached by the isl-scheduler is shown. Only every
20th data point is plotted.
0.25
0.5
1
2
4
8
16
32
0 4 8 12 16 20 24 28 32 36 40
speedup over O3
generation
(a) 3mm
0.25
0.5
1
2
4
8
16
32
64
0 4 8 12 16 20 24 28 32 36 40
(b) correlation
0.5
1
2
4
8
0 4 8 12 16 20 24 28 32 36 40
(c) adi
Fig. 6. Box plots showing the distribution of per-
formance within the generations of the GA. The
data are from Exp. 3. The length of the whiskers is
1.5 times the interquartile range. The whiskers may
have been truncated to remain within the data set’s
range. The dots are outliers beyond the range of the
whiskers.
In addition, we ran random exploration with the mixed setting, which produces both sparse and
dense matrices, for 3mm. Its effectiveness is between random dense and random sparse.
Exp. 3 (RQ1, RQ2, RQ3). Comparison with existing Approaches and Random . We compared the
following configurations to the original code optimized with O3: (1) isl’s adaptation of the PLuTo
algorithm, (2) random iterative optimization, (3) random iterative optimization with the adaptation
of Pouchet’s approach (LeTSeE), (4) LeTSeE combined with schedule completion and (5) GA. The
comparison to isl’s scheduler is fairer than comparing to the original PLuTo algorithm would
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
Iterative Schedule Optimization for Parallelization in the Polyhedron Model 1:19
 0.5
 1
 2
 4
 8
 16
 32
 64
correlationcovariance
fdtd-2djacobi-2dseidel-2d
adi
heat-3dgemversyr2k syrkgemm trmm
gesummv
mvt bicg
doitgen 2mm 3mm
speedup over O3
GA
random sparse
random LeTSeE
random LeTSeE + completion
isl
Fig. 7. Speedups over plain O3 reached by different configurations. The baseline is O3.
Table 2. p-values obtained from a pair-wise Mann-Whitney U Test quantifying the significance of differences
between different configurations of schedule optimization
GA isl random LeTSeE random LeTSeE + completion
isl 7.60E-05 – – –
random LeTSeE 5.70E-05 0.0384 – –
random LeTSeE + completion 2.50E-05 0.7987 0.3375 –
random sparse 0.2859 0.0014 2.50E-05 2.50E-05
Table 3. Per benchmark, the speedup over O3 of the best schedule found by the GA for 1-thread parallel
execution and 8-threads parallel execution.
benchmark 2mm 3mm adi bicg correlation covariance doitgen fdtd-2d gemm
1-thread parallel (speedup) 2.77 2.46 1.07 2.58 4.92 4.61 1.17 0.95 1.10
8-threads parallel (speedup) 19.79 17.14 7.37 2.58 33.10 35.28 4.18 3.50 8.12
gemver gesummv heat-3d jacobi-2d mvt seidel-2d syr2k syrk trmm
0.94 2.32 0.99 0.99 0.97 2.49 3.16 1.99 3.12
4.87 3.54 7.01 3.78 4.67 8.53 22.83 15.46 22.02
be, because the latter excludes some loop transformations that require negative coefficients for
iteration variables. The comparison is made solely in terms of the performance yielded by the best
found program transformation.
Figure 7 shows, per benchmark and optimization method, the speedup over plain O3 yielded by
the best schedule found. In many cases, our novel iterative optimization technique yields more
profitable schedules than PLuTo/isl and the adapted approach of Pouchet et al . [33]. Only for
doitgen and correlation, the schedule found by iterative optimization is slightly worse than the
schedule found by isl. Our approach to random exploration is never worse than the approach of
Pouchet et al.. There is no apparent difference between the speedup reached by random exploration
and GA. Table 3 lists the speedup over plain O3 of 8-threads parallel and 1-thread parallel execution
of the optimal schedules found by GA. This gives an idea of the influence of the parallelization on
the achieved speedups. The achieved speedups are largely due to parallelization.
We performed a paired Mann-Whitney U Test [1] in combination with false discovery rate control
[4] to determine the statistical significance of the differences in the optimization results of each
configuration pair. Table 2 shows, for each pair of configurations, the probability of erroneously
assuming a difference between its optimization results. The GA has significantly better results than
all other configurations except sparse random. The test is inconclusive about the effect of schedule
completion on Pouchet’s approach.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
1:20 S. Ganser et al.
Table 4. Per benchmark, the number and total sizes of optimal schedule equivalence classes in the GA’s final
population. Further, per benchmark, the number of groups of equivalence classes with larger differences in
the resulting code. heat-3d and adi had to be omitted, as some of the schedules could not be canonicalized
in reasonable time.
benchmark 2mm 3mm bicg correlation covariance doitgen fdtd-2d gemm
# equivalence classes 16 13 1 16 12 13 18 10
total # schedules 21 24 22 16 22 26 22 19
# groups by similarity 3 2 1 11 3 4 10 1
gemver gesummv jacobi-2d mvt seidel-2d syr2k syrk trmm
5 7 11 2 16 15 4 13
23 13 22 22 23 23 26 23
1 3 1 1 1 1 1 1
Finally, we estimated the number of search space regions visited by the genetic algorithm and
random sparse. For all benchmarks, random sparse visits more regions than the GA. On average, the
number of regions visited by the GA is 38% of the number of regions visited by random exploration.
This was to be expected, since the GA is designed to continue its search in the vicinity of the most
profitable schedules known so far. Using Spearman’s rank correlation [21][Chap. 4], we found that
the number of regions visited by the GA and the number of regions visited by random sparse both
strongly correlate with the number of statements in the SCoP (GA: ρ = 0.92, p = 6.249 ·10−8,
random sparse: ρ = 0.91, p = 1.263 ·10−7; ρ is the correlation coefficient, the p-values result from a
two-sided significance test). Correlation with the number of data dependences is weak for random
sparse (ρ = 0.57, p = 0.01) and undecidable for the GA (ρ = 0.31, p = 0.21).
Exp. 4 (RQ6). Performance Distribution in the Generations of the GA . For the benchmarks 3mm,
correlation, and adi, Figure 6 shows the performance distribution in each population of the GA
using box plots. The data is from Exp. 3. The x-axis enumerates the populations. The y-axis shows
speedup over O3. While the genetic algorithm proceeds, the variance of performance within the
populations diminishes. Populations 8 and 35 of correlation and populations 19 and 30 of adi
contain remarkably higher numbers of unprofitable schedules than their predecessors. In case of
adi, geometric crossover produced more than half of the newly introduced schedules in populations
19 and 30 with a speedup smaller than the first quartile. In population 35 of correlation, 5 of the
7 new schedules with a speedup smaller than the first quartile are produced by prefix replacement.
Figure 6 exhibits a notable property of the genetic algorithm: It delivers a set of profitable
schedules, while optimizing a SCoP requires just one. We wanted to know, how many semantically
different schedules the set contains. To determine the according equivalence classes, we extended
our schedule tree simplification to represent loop fusion in a canonical form and to transform
the dimensions of band nodes’ partial schedules to the canonical form of schedules with minimal
coefficients. The latter borrows ideas from PLUTO+ [6]. We grouped the schedules of the GA’s final
population from Exp. 3 into equivalence classes. We took the execution time of its initial member as
representative for an equivalence class. The extended schedule tree simplification is not a complete
canonicalization. Therefore, we manually inspected the code that corresponds to each equivalence
class. Thereby, we could merge a few equivalence classes. Further, we grouped equivalence classes
that yield code which differs only by inverted loops or skewing. Table 4 shows the results.
The complexity of the remaining experiments forced us to limit them to one benchmark. Of the
three benchmarks that have already been studied in more detail, we chose 3mm since its numbers of
statements and dependences and its maximum loop depth are closer to average than those of adi
and correlation.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
Iterative Schedule Optimization for Parallelization in the Polyhedron Model 1:21
Exp. 5 (RQ7). Effectiveness of the Genetic Operators . To evaluate and compare the effectiveness of the
genetic operators, we optimized the schedule of 3mm with six configurations. In each configuration,
one operator was disabled. Otherwise, the configurations meet the specification in Section 6.1. We
ran each configuration 10 times, starting from one of 10 initial populations. A noticeable effect of
disabling any of the operators on the convergence speed or the profitability of the optimal found
schedule cannot be observed. Yet, from Exp. 4, we know that the last generation consists mostly of
profitable schedules, so, in combination, our genetic operators are effective.
Exp. 6 (RQ7). Intensity of the Mutations . We evaluated the effect of varying the GA’s mutations’
intensity. We set the initial intensity (see Section 6.1) to 10%, and later increased it to 30% and 60%.
With each configuration, we optimized 3mm 10 times. Increasing the mutations’ intensity does not
affect the convergence speed of the GA or the profitability of the optimal schedule found. This
was to be expected: In Exp. 2, we already noticed that the GA with its mutation intensity set to
10% does not converge faster than random sparse. Furthermore, we expect the GA to degenerate
to a purely random search as we increase the mutation intensity. So, in our case, increasing the
intensity of mutations should not yield faster convergence. Due to the small size of the benchmarks
considered, a mutation intensity below 10% should not yield different results, either.
6.3 Discussion
We can answerRQ1 positively based on the results shown in Figure 7. Iterative optimization detects
program transformations yielding a higher performance than those found by the model-driven
optimization of isl. There are many possible reasons. The underlying assumptions of the model-
driven approach may not apply in every case. The model-driven algorithm maximizes the size of
tilable bands, which our iterative approach does not. Pouchet et al. [34] iteratively search for a good
partitioning of a SCoP’s statements into classes with different execution date and then schedule the
statement instances in each partition in a model-driven way. Their motivation is that cost models
for scheduling inside each class are well understood, while those for the partitioning itself are
not. Likewise, our iterative approach may profit from this fact by finding a better partitioning. It
is likely that, on different hardware, the iterative optimization would identify different optimal
schedules. Furthermore, the tile shapes chosen by thePLuTo algorithm may be suboptimal. Iterative
optimization may find different ones. In the end, performance optimization is hardware-specific.
Naturally, iterative optimization can adapt itself more easily than model-driven algorithms.
Figure 7 reveals that random exploration with the sparse configuration almost always finds sched-
ules that yield better performance than random exploration with the adaptation of LeTSeE. This
justifies our augmentation of the schedule search space in combination with schedule completion
and, thus, we can answer RQ2 positively.
The answer to RQ3 is that the GA and random exploration with the sparse setting converge
at almost equal rate (see Figure 5). In Exp. 2, the GA was able to yield slightly more profitable
schedules than sparse random. In Exp. 3, the speedup of the transformed code produced by the
GA was 4% higher than by sparse random, on average. This difference cannot be tested significant.
Figure 8 shows for 3mm the performance distribution of the schedules visited during Exp. 2. We
aggregated the data from all runs of each experiment and removed identical schedules. The GA is
capable of finding significantly more schedules with a higher performance than sparse random.
Still, the probability that sparse random hits a schedule with good performance is high. This must
be attributed to the design and configuration of Algorithms 1 and 2. Studying this finding is an
interesting avenue of future research.
RQ4 can also be answered positively. From Figure 8, it is apparent that, for 3mm, random dense
finds fewer profitable schedules than random sparse. The effect is similar for correlation and
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
1:22 S. Ganser et al.
0.125
0.25
0.5
1
2
4
8
16
32
0 500 1000 1500 2000 2500
speedup over O3
schedule index
GA
random sparse
random dense
random LeTSeE
+ completion
random mixed
random LeTSeE
O3
isl
Fig. 8. Performance distribution of the schedules visited in Exp. 2 for3mm. Thex-axis enumerates the schedules.
The y-axis is speedup over O3. Note that not every data point is plotted.
less pronounced for adi. We observe that random exploration with the dense setting produces
by far more schedules that cannot be executed successfully than other configurations. For 3mm,
91.49% of the schedules visited by the genetic algorithm are healthy, 73.48% by random sparse,
100.00% by LeTSeE random but only 7.61% of random dense. Reasons range from a failure or timeout
(5 minutes) of compilation (78.79% of the schedules from random dense), and failure of execution or
erroneous computation results, to overall timeout of benchmarking. This conforms to the prediction
of Pouchet et al. [33] that picking schedule coefficients close to zero is beneficial. Also, matrices
formed from a large number of generators tend to contain larger coefficients, which may trigger
integer overflows. The answer to the question of why random exploration with the sparse setting
is extremely close to or even better than optimization with the GA is subject of future research.
Figure 4 confirms that the run-time complexity of Algorithm 1 is indeed exponential in the
number of data dependences. The execution time does not grow equally fast for different programs.
To answer RQ5, our search space construction does not scale for SCoPs with a large number of
data dependences.
By Figure 6, RQ6 is answered positively: If a genetic operator produces a new schedule, the
probability is high that its performance is similar to that of its parents. Since a schedule’s parents
and the schedule itself are similar by construction, we can assume that similar schedules have
a similar performance. Furthermore, we find that, for most benchmarks, semantically different
schedules exist among the best schedules found by the GA. For most benchmarks with more than
two statements, these differences are beyond loop inversion and skewing and comprise different
loop distribution and fusion or interchange of loops.
In Exp. 5, we could not observe that disabling any genetic operator has a noticeable effect on
the convergence speed of the genetic algorithm. Thus, we conclude that it is indeed best to select
the operators with uniform probability. In Exp. 6, we checked whether an increased intensity of
mutations would improve the convergence speed of the GA. We could not observe that an initial
intensity higher than 10% (the default) yields faster convergence. We do not expect a lower value to
yield faster convergence, either. In answer of RQ7, we conclude, that our choices for the selection
probability of the genetic operators and the intensity of mutations are justified.
6.4 Threats to Validity
Threats to Internal Validity. Due to the exponential size of the search space of legal schedules,
a full exploration of the entire (reasonable) search space, is computationally infeasible. Instead,
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
Iterative Schedule Optimization for Parallelization in the Polyhedron Model 1:23
to create the initial population of the GA, we randomly sample search space regions and select
one schedule from each selected region. Thus, we perform an incomplete exploration but can
at least ensure that we start from a set of schedules that originate from different regions of the
search space. Random exploration works accordingly. Still, different runs may find different optima.
Although Figure 7 shows results of single runs, there are a number of apparent regularities in the
data. Especially, random exploration and the GA often yield very similar performance. In other
experiments, we present average results of multiple runs with different random seed to mitigate
the effect of incomplete exploration.
Although all schedules visited are theoretically legal, some result in broken binaries or compila-
tion failures. Any mathematically illegal schedules will be caught by Polly and will be reported
to Polyite. Polyite will then abort. One of the run-time failures is the presence of overly large
schedule coefficients. We are unaware of the other causes of compile-time or run-time failures that
occur despite the legality of the schedules. As already mentioned in the discussion, we noticed that
dense schedule matrices are likely to cause failure. They are also likely to increase compilation time
tremendously. In part, the transformation of schedule matrices to schedule trees and the subsequent
schedule tree simplification reduce the problem. The problematic schedules can simply be discarded
but, the more schedules are discarded, the longer it takes to find the desired number of healthy
schedules. An examination of the schedules from Exp. 2 reveals that, in contrast to random dense,
the GA and random sparse discover only few schedules that fail.
To avoid measurement bias, the benchmarking systems had a minimal setup and ran no other
workloads in parallel. Intel Turbo Boostand hyper-threading are known to destabilize mea-
surements and, consequently, were disabled. The systems have a NUMA design with two sockets.
If threads were distributed arbitrarily across the two CPUs and used non-local memory, time
measurements would be unreliable. We pin the threads to the cores of one CPU and use only local
memory [20]. We repeated execution time measurements five times and chose the shortest result.
Five measurements are hardly enough to reach statistical significance but a higher number of
measurements conflicts with the extremely large total number of program versions that had to be
evaluated in our experiments. Data regarding the stability of our measurements is available on the
supplementary Web site.
Threats to External Validity. We evaluated our approach to iterative schedule optimization on
the PolyBench 4.1benchmark suite. PolyBench is used by many researchers in the polyhedral
community. This facilitates the comparability of our results. PolyBench comprises SCoPs in
different application domains. This leverages the bias of an artificial benchmark. PolyBench is
constantly being developed to adapt to the state-of-the-art in the polyhedral community.
Since PolyBench is particularly designed for benchmarking polyhedral optimization techniques,
each benchmark contains one SCoP to be detected and optimized. To optimize SCoPs in real-world
programs with Polyite, each SCoP would have to be dissected into a small program and would
have to be fed with small (artificial) data.
7 RELATED WORK
Frequently used scheduling algorithms in contemporary polyhedral compilers are model-driven.
A parametric integer programming algorithm [ 8] is used to select lexicographically minimal
solution vectors from a polyhedron. Most notable are the algorithm by Feautrier[10, 11] and PLuTo
[5]. Feautrier’s algorithm reduces latency. Feautrier [10] states the generator representation of a
polyhedron together with Chernikova’s algorithm as an alternative way of solving the constraint
system of his algorithm.PLuTo optimizes for tiling and data locality. The Integer Set Library (isl) by
Verdoolaege [39] comprises a generalization of PLuTo. Another generalization is PLUTO+ [6]. Both
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
1:24 S. Ganser et al.
generalizations have fewer practical limitations than the original PLuTo algorithm. Particularly,
they include transformations that require negative coefficients for iteration variables.
Iterative polyhedral optimizers do not rely on elaborate performance models. They sample the
search space of possible schedules and assess each schedule’s fitness. Some early approaches put
legality constraints aside and consequently suffer from many illegal schedules. All are based on
UTF [18]. GAPS [27] uses a GA to find polyhedral schedule transformations. GAPS starts from a
random population of potentially illegal transformations that is seeded by one legal transformation.
The fitness function either measures execution time of transformed programs or predicts loop and
synchronization overhead. Among 20,000 generated schedules, at most 5.5% were legal. ICE [28] is
built on top of GAPS. It enables the GA to apply more transformations, including tiling.ICE has new
mutation operators in addition to the ones that exist inGAPS. The reproduction (genetic) operators
can account for profiling data. The fitness function measures execution time. Long and O’Boyle[25]
optimize Java programs. They machine-learn the performance impact of loop transformations that
are expressible in UTF. To match unknown programs with learned strategies, program similarity is
expressed by a set of features. Long and Fursin [24] also optimize Java code. They separate the
search of mappings of the iteration variables (tiling, skewing) and the exploration of the constant
part of schedules (loop distribution, fusion). Run time feedback directs the search.
We build on the approach of iterative optimization by Pouchet et al . [32, 33]. We go beyond
their approach to optimize for tiling and parallelization in a purely iterative manner. We avoid
search space restrictions that are viable only for sequential execution and propose a new sampling
strategy. In Section 1, we already mention that later Pouchet et al . [34] combined iterative and
model-driven optimization in the polyhedron model. The result is an automatic parallelization
framework in which tiling, parallelization, and vectorization can be applied. They iteratively find
a legal partitioning of the statements which corresponds to a sequence of loop distribution and
fusion operations and then find a schedule for each partition in a model-driven way.
8 CONCLUSION AND FUTURE WORK
We propose an approach of purely iterative schedule optimization for tiling andOpenMP paral-
lelization in the polyhedron model. We find schedules that yield better performance than with the
model-driven PLuTo algorithm or our adaption of the iterative approach by Pouchet et al . [33].
We have presented a genetic algorithm to detect profitable schedules with more guidance than
at random. Interestingly, a well configured random exploration finds schedules that yield almost
equal performance as the schedules found by the genetic algorithm. We would like to address this
finding in future research. Yet, the genetic algorithm is able to find more profitable schedules than
random exploration. An investigation of the effectiveness of our genetic operators did not show an
apparent disparity between the different operators.
ACKNOWLEDGMENTS
The first author is grateful to Stefan Kronawitter for his helpful technological support during
the early stages of Polyite. Special thanks go to the reviewers, in particular Reviewer 1, whose
impressively detailed debate of our work led to numerous clarifications and improvements.
REFERENCES
[1] T. W. Anderson and J. D. Finn. 1996. The New Statistical Analysis of Data . Springer.
[2] S. Balev, P. Quinton, S. V. Rajopadhye, and T. Risset. 1998. Linear Programming Models for Scheduling Systems
of Affine Recurrence Equations – A Comparative Study. In Proc. 10th Ann. ACM Symp. on Parallel Algorithms and
Architectures (SPAA). ACM Press, 250–258.
[3] M.-W. Benabderrahmane, L.-N. Pouchet, A. Cohen, and C. Bastoul. 2010. The Polyhedral Model Is More Widely
Applicable Than You Think. In Compiler Construction (CC) (LNCS 6011) , Rajiv Gupta (Ed.). Springer, 283–303.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
Iterative Schedule Optimization for Parallelization in the Polyhedron Model 1:25
[4] Y. Benjamini and Y. Hochberg. 1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to
Multiple Testing. J. Royal Stat. Soc. Series B (Methodological) 57, 1 (1995), 289–300.
[5] U. Bondhugula et al. 2008. Automatic Transformations for Communication-Minimized Parallelization and Locality
Optimization in the Polyhedral Model. In Compiler Construction (CC) (LNCS 4959) , Laurie Hendren (Ed.). Springer,
132–146.
[6] U. Bondhugula, A. Acharya, and A. Cohen. 2016. The Pluto+ Algorithm: A Practical Approach for Parallelization and
Locality Optimization of Affine Loop Nests. ACM Trans. Program. Lang. Syst. (TOPLAS) 38, 3 (May 2016), 12:1–12:32.
[7] J. Clarke et al. 2003. Reformulating Software Engineering as a Search Problem. IEE Proceedings – Software 150, 3 (June
2003), 161–175.
[8] P. Feautrier. 1988. Parametric Integer Programming. RAIRO – Operations Research 22, 3 (1988), 243–268.
[9] P. Feautrier. 1991. Dataflow Analysis of Array and Scalar References. Int. J. Par. Prog. (IJPP) 20, 1 (1991), 23–53.
[10] P. Feautrier. 1992. Some Efficient Solutions to the Affine Scheduling Problem. Part I. One-Dimensional Time. Int. J. Par.
Prog. (IJPP) 21, 5 (1992), 313–347.
[11] P. Feautrier. 1992. Some Efficient Solutions to the Affine Scheduling Problem. Part II. Multidimensional Time. Int. J.
Par. Prog. (IJPP) 21, 6 (1992), 389–420.
[12] P. Feautrier and C. Lengauer. 2011. Polyhedron Model. In Encyclopedia of Parallel Computing , D. A. Padua (Ed.). Vol. 3.
Springer, 1581–1591.
[13] M. Griebl, P. Feautrier, and C. Lengauer. 2000. Index Set Splitting. Int. J. Par. Prog. (IJPP) 28, 6 (Dec. 2000), 607–631.
[14] T. Grosser, A. Größlinger, and C. Lengauer. 2012. Polly – Performing Polyhedral Optimizations on a Low-Level
Intermediate Representation. Par. Proc. Lett. (PPL) 22, 4 (2012). Article 1250010, 28 pages.
[15] T. Grosser, S. Verdoolaege, and A. Cohen. 2015. Polyhedral AST Generation Is More Than Scanning Polyhedra. ACM
Trans. Program. Lang. Syst. (TOPLAS) 37, 4 (Aug. 2015), 12:1–12:50.
[16] M. Harman. 2007. The Current State and Future of Search Based Software Engineering. In Proc. Workshop on the Future
of Software Engineering (FOSE) . IEEE Computer Society, 342–357.
[17] F. Irigoin. 2011. Tiling. In Encyclopedia of Parallel Computing , D. A. Padua (Ed.). Vol. 4. Springer, 2040–2049.
[18] W. Kelly and W. Pugh. 1995. A Unifying Framework for Iteration Reordering Transformations. In Proc. IEEE First Int’l
Conf. on Algorithms and Architectures for Parallel Processing (ICAPP) , Vol. 1. IEEE, 153–162.
[19] K. Kennedy and K. S. McKinley. 1993. Maximizing Loop Parallelism and Improving Data Locality via Loop Fusion and
Distribution. In Languages and Compilers for Parallel Computing (LCPC) (LNCS 768) , U. Banerjee et al. (Eds.). Springer,
301–320.
[20] A. Kleen. 2004. An NUMA API for Linux . Technical Report. SUSE Labs.
[21] P. R. Krishnaiah and P. K. Sen. 1984. Handbook of Statistics . Vol. 4. Elsevier.
[22] C. Lattner. 2008. LLVM and Clang: Next Generation Compiler Technology. In BSDCan: The BSD Conference .
[23] H. Le Verge. 1994. A Note on Chernikova’s Algorithm . Res. Report RR-1662. INRIA.
[24] S. Long and G. Fursin. 2009. Systematic Search within an Optimisation Space Based on Unified Transformation
Framework. Int. J. Comput. Sci. Eng. (IJCSE) 4, 2 (2009), 102–111.
[25] S. Long and M. F. P. O’Boyle. 2004. Adaptive Java Optimisation Using Instance-Based Learning. In Proc. 18th Ann. Int.l
Conf. on Supercomputing (ICS) . ACM, 237–246.
[26] M. Mitchell. 1998. An Introduction to Genetic Algorithms . MIT Press.
[27] A. Nisbet. 1998. GAPS: A Compiler Framework for Genetic Algorithm (GA) Optimised Parallelisation. In High-
Performance Computing and Networking (HPCN Europe) , P. Sloot, M. Bubak, and B. Hertzberger (Eds.). Springer,
987–989.
[28] A. Nisbet. 2001. Towards Retargettable Compilers – Feedback Directed Compilation Using Genetic Algorithms. In Proc.
9th Int’l Workshop on Compilers for Parallel Computers (CPC) . 12 pages. http://www.icsa.informatics.ed.ac.uk/cpc2001/
[29] M. Odersky, L. Spoon, and B. Venners. 2008. Programming in Scala . Artima.
[30] D. Padua. 2011. Parallelization, Automatic. In Encyclopedia of Parallel Computing , D. A. Padua (Ed.). Vol. 3. Springer,
1442–1450.
[31] L.-N. Pouchet. 2012. LeTSeE – The LEgal Transformation SpacE Explorator. (2012). http://web.cs.ucla.edu/~pouchet/
software/letsee/
[32] L.-N. Pouchet et al. 2007. Iterative Optimization in the Polyhedral Model: Part I, One-Dimensional Time. In Proc. Fifth
Int. Symp. on Code Generation and Optimization (CGO) . IEEE Computer Society, 144–156.
[33] L.-N. Pouchet et al. 2008. Iterative Optimization in the Polyhedral Model: Part II, Multidimensional Time. In Proc. ACM
SIGPLAN 2008 Conf. on Programming Language Design and Implementation (PLDI) . ACM, 90–100.
[34] L.-N. Pouchet et al. 2010. Combined Iterative and Model-driven Optimization in an Automatic Parallelization Framework.
In Proc. ACM/IEEE Int’l Conf. on High Performance Computing, Networking, Storage and Analysis (SC) . IEEE Computer
Society, 1–11.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.
1:26 S. Ganser et al.
[35] L.-N. Pouchet and T. Yuki. 2015. PolyBench 4.1. (May 2015). http://web.cse.ohio-state.edu/~pouchet/software/
polybench/.
[36] A. Schrijver. 1994. Theory of Linear and Integer Programming . John Wiley & Sons.
[37] K. Trifunovic et al. 2010. GRAPHITE Two Years After: First Lessons Learned From Real-World Polyhedral Compilation.
In Proc. Int’l Workshop on GCC Research Opportunities (GROW) . 1–13. http://ctuning.org/workshop-grow10.
[38] R. Upadrasta and A. Cohen. 2013. Sub-Polyhedral Scheduling Using (Unit-)Two-Variable-per-Inequality Polyhedra. In
Proc. 40th Ann. ACM SIGPLAN-SIGACT Symp. on Principles of Programming Languages (POPL) . ACM, 483–496.
[39] S. Verdoolaege. 2010. isl: An Integer Set Library for the Polyhedral Model. In Mathematical Software – ICMS 2010 ,
K. Fukuda et al. (Eds.). Springer, 299–302.
[40] M. Wolfe. 1986. Loops Skewing: The Wavefront Method Revisited. Int. J. Par. Prog. (IJPP) 15, 4 (Aug. 1986), 279–293.
ACM Transactions on Architecture and Code Optimization, Vol. 1, No. 1, Article 1. Publication date: July 2017.

Large Language Models for Compiler Optimization
Chris Cummins†∗, V olker Seeker†, Dejan Grubisic †,
Mostafa Elhoushi, Baptiste Roziere, Jonas Gehring, Fabian Gloeckle,
Kim Hazelwood, Gabriel Synnaeve, Hugh Leather †
Meta AI
Youwei Liang
UC San Diego
Abstract—We explore the novel application of Large Language
Models to code optimization. We present a 7B-parameter trans-
former model trained from scratch to optimize LLVM assembly
for code size. The model takes as input unoptimized assembly
and outputs a list of compiler options to best optimize the
program. Crucially, during training, we ask the model to predict
the instruction counts before and after optimization, and the
optimized code itself. These auxiliary learning tasks significantly
improve the optimization performance of the model and improve
the model’s depth of understanding.
We evaluate on a large suite of test programs. Our approach
achieves a 3.0% improvement in reducing instruction counts over
the compiler, outperforming two state-of-the-art baselines that
require thousands of compilations. Furthermore, the model shows
surprisingly strong code reasoning abilities, generating compilable
code 91% of the time and perfectly emulating the output of the
compiler 70% of the time.
I. I NTRODUCTION
There is increasing interest in Large Language Models
(LLMs) for software engineering domains such as code gener-
ation [1–9], code translation [10–12], and code testing [13–15].
Models such as Code Llama [9], Codex [8], and ChatGPT [16]
have a good statistical understanding of code and suggest likely
completions for unfinished code, making them useful for editing
and creating software. However, it appears they have not been
trained specifically to optimize code. ChatGPT, for instance,
will make minor tweaks to a program such as tagging variables
to be stored as registers, and will even attempt more substantial
optimizations like vectorization, though it easily gets confused
and makes mistakes, frequently resulting in incorrect code.
Prior works on machine learning-guided code optimization
have used hand-built features [17–19], all the way to graph
neural networks (GNNs) [20, 21]. However, in all cases, the
way the input program is represented to the machine learning
algorithm is incomplete, losing some information along the way.
For example, MLGO [17] uses numeric features to provide
hints for function inlining, but cannot faithfully reproduce
the call graph or control flow, etc. PrograML [21] forms
graphs of the program to pass to a GNN, but it excludes
the values for constants and some type information which
prevents reproducing instructions with fidelity.
In this work, we ask: can Large Language Models learn to
optimize code? LLMs can accept source programs, as is, with
a complete, lossless representation. Using text as the input
and output representation for a machine learning optimizer has
† Core contributors. *Corresponding author: cummins@meta.com
desirable properties: text is a universal, portable, and accessible
interface, and unlike prior approaches is not specialized to any
particular task.
We started our investigation into the code-optimizing power
of LLMs by replicating the optimizing transformations present
in compilers, targeting the industry standard LLVM [22]
compiler. LLVM’s optimizer is extremely complex and contains
thousands of rules, algorithms, and heuristics in over 1M
lines of C++ code. Our expectation was that while LLMs
have shown great progress in natural language translation and
code generation tasks, they would be incapable of emulating
such a complex system. Understanding and applying compiler
optimizations require multiple levels of reasoning, arithmetic
computation capabilities, and applying complex data structure
and graph algorithms, which are capabilities LLMs have shown
to lack [23, 24].
We thought this would be a paper about the obvious failings
of LLMs that would serve as motivation for future clever ideas
to overcome those failings. We were entirely taken by surprise
to find that in many cases a sufficiently trained LLM can
not only predict the best optimizations to apply to an input
code, but it can also directly perform the optimizations without
resorting to the compiler at all!
Our approach is simple. We begin with a 7B-parameter LLM
architecture, taken from LLaMa 2 [25], and initialize it from
scratch. We then train it on millions of examples of LLVM
assembly, coupled with the best compiler options found by a
search for each assembly, as well as the resulting assembly from
performing those optimizations. From these examples alone
the model learns to optimize code with remarkable accuracy.
Our singular contribution is the first application of LLMs to
optimizing code. We construct LLMs solely for the purpose
of compiler optimization and show that they achieve a single-
compile 3.0% improvement in code size reduction over the
compiler versus a search-based approach which achieves 5.0%
with 2.5e9 compilations and versus state of the state-of-the-art
ML approaches that cause regressions and require thousands
of compilations. We provide auxiliary experiments and code
examples to further characterize the potential and limits of
LLMs for code reasoning. Overall we find their efficacy
remarkable and think that these results will be of interest
to the community.
arXiv:2309.07062v1  [cs.PL]  11 Sep 2023
Figure 1: Overview of our approach, showing the model input (Prompt) and output (Answer) during training and inference. The prompt contains unoptimized
code. The answer contains an optimization pass list, instruction counts, and the optimized code. During inference we generate only the optimization pass list
which we feed into the compiler, ensuring that the optimized code is correct.
Table I: Training data. Each LLVM-IR function is autotuned and used to create
a (Prompt, Answer) pair. The n tokens column shows the number of tokens
when the prompt is encoded using the Llama 2 [25] tokenizer.
n
functions
unoptimized
instruction
count
size on
disk n tokens
Handwritten 610,610 8,417,799 653.5 MB 214,746,711
Synthetic 389,390 13,775,149 352.3 MB 158,435,151
Total 1,000,000 16,411,249 1.0 GB 373,181,862
II. P ASS ORDERING WITH LLM S
In this work we target compiler pass ordering. The pass
ordering task is to select from the set of optimizing transforma-
tion passes available in a compiler the list of passes that will
produce the best result for a particular input code. Manipulating
pass orders has been shown to have a considerable impact on
both runtime performance and code size [19, 26].
Machine learning approaches to this task have shown
good results previously, but struggle with generalizing across
different programs [27]. Previous works usually need to compile
new programs tens or hundreds of times to try out different
configurations and find out the best-performing option, making
them impractical for real-world use. We hypothesized that a
large language model with sufficient reasoning power would
be able to learn to make good optimization decisions without
needing this.
Most prior work on LLMs for code operates on source
languages such as Python. Instead, for the pass ordering
problem we require reasoning at the lower level of compiler
assembly, known as the Intermediate Representation (IR).
Table II: Test data.
n
functions
unoptimized
instruction
count
-Oz
instruction
count
AI-SOCO [31] 8,929 97,800 47,578
ExeBench [32] 26,806 386,878 181,277
POJ-104 [33] 310 8,912 4,492
Transcoder [12] 17,392 289,689 129,611
CSmith [34] 33,794 647,815 138,276
Y ARPGen [35] 12,769 285,360 144,539
Total 100,000 1,716,354 645,773
While there exist curated datasets of source languages for
pretraining LLMs (e.g. [28–30]), compiler IRs do not make up
a significant portion of these datasets, and though models like
ChatGPT show some promise of understanding, their ability
to reason about IR is far inferior to source languages.
We target optimizing LLVM pass orders for code size as
in prior works [17, 27], using IR instruction count as an
(imperfect) proxy for binary size. The approach is agnostic to
the chosen compiler and optimization metric, and we intend to
target runtime performance in the future. For now, optimizing
for code size simplifies the collection of training data.
A. Prompts
We present the model with an unoptimized LLVM-IR (such
as emitted by the clang frontend) and ask it to produce a list
of optimization passes that should be applied to it. Figure 1
shows the format of the input prompt and output text.
In this work, we target LLVM 10 and use the optimization
flags from opt. There are 122 optimization passes to choose
2
from and passes can be selected more than once in a single
sequence. We also include the 6 meta-flags (-O0, -O1, -O2,
-O3, -Oz, and -Os) that may each occur only once per pass
list. Pass lists can be any length, though in our experiments
we found typically up to 9 passes long, for a combinatorial
search space of around 1018.
As shown in Figure 1, we also include two auxiliary tasks:
i) generating the instruction counts of the code before and
after the optimizations are applied and ii) generating the output
IR after the optimizations are applied. We hypothesize that
these would enable better pass-ordering decisions by forcing
a deep understanding of the mechanics of code optimization.
We verify this experimentally in Section V-B.
While the model is trained to generate instruction counts
and optimized IR, we do not need those auxiliary tasks for
deployment. All we need to do is generate the pass list which
we then execute using the compiler. We thus sidestep the
problems of correctness that plague techniques that require the
output of the model to be trustworthy [10–12, 36].
B. LLVM-IR Normalization
We normalize the LLVM-IR that is used for training the
LLM using the following rules: we discard comments, debug
metadata and attributes, and ensure consistent whitespace by
feeding the IR through a custom lexer that retains newlines but
standardizes other whitespace and strips indentation. We do
this to reduce the length of the LLVM-IR to make maximum
use of the limited input size of the LLM (Section III-A). The
code in Figure 1 has been processed in this manner.
III. T HE MODEL
We use the ubiquitous transformer architecture [37]. The
transformer is an artificial neural network that employs self-
attention over a fixed-size context window.
The input text is first tokenized into words and subword units.
These are embedded into continuous vector representations and
provided as input to the transformer’s encoder, where self-
attention mechanisms capture contextual relationships between
tokens to encourage the model to understand and process the
input text’s semantic structure.
The output text is produced by iteratively generating one
token at a time. The decoder takes the encoded input along
with any previously generated tokens and uses self-attention
to predict the next token in the sequence. We greedily sample
during decoding to select the most likely token sequence. This
process continues until an end-of-sequence token is generated
or a predefined maximum length is reached.
A. Model Architecture
We use the same model architecture and Byte Pair Encoding
(BPE) [38] tokenizer as Llama 2 [25], but train our model from
scratch. We use the smallest of the Llama 2 configurations: 32
attention heads, 4,096 hidden dimensions, and 32 layers, for a
total of 7B parameters.
The maximum length of a (prompt, answer) pair is defined
by the sequence length. In this work, we use a sequence length
(a) Performance of generated pass lists.
(b) Accuracy at predicting instruction counts.
(c) Model-optimized code metrics.
Figure 2: Performance on holdout validation set during training. We evaluate
performance every 250 training steps (131M train tokens). Parity with -Oz is
reached at 393M tokens and peak performance at 10.9B tokens.
of 2,048 tokens. The Llama 2 tokenizer achieves an average
of 2.02 characters per token when encoding LLVM-IR, so this
provides an approximate upper limit on the longest LLVM-IR
we can train on at 2KB (since 2KB prompt and 2KB answer
≈ 2,048 tokens).
B. Training Data
We assembled a large corpus of unoptimized LLVM-IR
functions, summarized in Table I. We extracted the functions
from datasets of publicly available handwritten C/C++ code
and supplemented this with synthetic code generated by C/C++
compiler test generators. In total, our training corpus comprises
1,000,000 deduplicated IR functions, totaling 373M training
tokens. We operate at the level of individual IR functions rather
than entire modules to maximize the amount of data we can
fit inside a 2,048-token sequence length.
To find the list of optimization passes that will produce the
smallest instruction count we employ autotuning. Our autotuner
combines random search and all-to-all results broadcasting
between functions, inspired by the work of Liang et. al. [20].
3
Table III: Performance of different approaches to pass ordering on a test set of unseen LLVM-IR functions from Table II. All metrics are w.r.t. -Oz. Instructions
saved is summed over functions improved and instructions regressed is summed over functions regressed. Overall improvement is the sum total instruction
count savings w.r.t -Oz. The Autotuner achieves the best performance but requires 2.5B additional compilations (949 CPU-days). Our approach achieves 60%
of the gains of the autotuner without invoking the compiler once.
additional
compilations
functions
improved
functions
regressed
instructions
saved
instructions
regressed
overall
improvement
Autotuner 2,522,253,069 6,764 0 30,948 0 5.03%
AutoPhase [39] 4,500,000 1,558 8,400 6,522 32,357 -3.85%
Coreset-NVP [20] 442,747 3,985 6,072 16,064 28,405 -1.88%
Our Approach 0 4,136 526 21,935 3,095 3.01%
Table IV: Extending the models in Table III with “-Oz backup”. If a model
predicts a pass list other than -Oz, it also evaluates -Oz and selects the best.
This prevents regressions w.r.t -Oz at the expense of additional compilations.
additional compilations overall improvement
AutoPhase [39] 4,600,000 1.02%
Coreset-NVP [20] 542,747 2.55%
Our Approach 5,721 3.52%
For each function we run random search for a fixed amount
of time (780 seconds) and then minimize the best pass list
by iteratively removing individual randomly chosen passes to
see if they contribute to the instruction count. If not, they are
discarded. After performing this on each of the functions we
aggregate the set of unique best pass lists and broadcast them
across all other functions. Thus, if a pass list was found to
work well on one function it is tried on all others.
In total, the autotuner compiled each training program an
average of 37,424 times, achieving a 5.8% improvement in
instruction count reduction over the baseline fixed pass ordering
in the compiler provided by -Oz. For our purposes, this
autotuning serves as a gold standard for the optimization of
each function. While the instruction count savings discovered
by the autotuner are significant, the computational cost to reach
these wins was 9,016 CPU days. The goal of this work is to
achieve some fraction of the performance of the autotuner
using a predictive model that does not require running the
compiler thousands of times.
C. Training
Starting from randomly initialized weights, we trained the
model for 30,000 steps on 64 V100s for a total training time
of 620 GPU days. We use the AdamW optimizer [40] with β1
and β2 values of 0.9 and 0.95. We use a cosine learning rate
schedule with 1,000 warm-up steps, a peak learning rate of
1e−5, and a final learning rate of 1/10th of the peak. We used
a batch size of 256 and each batch contains 524,288 tokens
for a total of 15.7B training tokens. The full 30,000 steps of
training is 7.7 epochs (iterations over the training corpus).
During training, we evaluated the model on a holdout
validation set of 1,000 unseen IRs that were processed in
the same manner as the training set. We evaluate every 250
steps.
IV. E VALUATION
In this section, we evaluate the ability of the model to
generate pass lists for unseen code and to correctly perform
optimization.
A. Training Results
Figure 2 shows the performance during training when
evaluated on a holdout validation set of 1,000 unseen LLVM-IR
functions. Peak validation performance was achieved by the
model at 10.9B training tokens.
At peak performance, the code optimized using model-
generated pass sequences contains 4.4% fewer instructions than
when optimized using the compiler’s built-in pass ordering
(-Oz). The autotuner achieves a greater instruction count
reduction of 5.6%, but this required 27 million compilations
of the validation set. The model makes its predictions without
invoking the compiler once.
Figure 2b shows the error of predicted input and out-
put instruction counts. Prediction of instruction counts for
unoptimized code rapidly approaches near-perfect accuracy.
Prediction of output instruction count proves more challenging,
reaching a Mean Average Percentage Error (MAPE) of 5.9%.
Figure 2c evaluates the quality of the generated code using
three metrics. The BLEU [41] score shows the similarity
between the model-generated code and a reference ground-
truth code produced by the compiler using the generated pass
list. Code compiles is the frequency that model-generated code
compiles without error. Exact match tracks the frequency that
the model-generated code is a character-by-character match
of the compiler-generated code when optimized using the
generated pass list (i.e. how many times BLEU=1).
At peak performance, the model achieves an impressive
90.5% rate of generating code that compiles without errors.
Furthermore, a BLEU score of 0.952 shows that the model-
optimized code closely approximates that of the compiler, and
the exact match frequency is 70%. For comparison, a baseline
that simply copies the unoptimized code to the output would
achieve a BLEU score of 0.531 and an exact match frequency
of 0%, demonstrating that significant manipulation of the input
code is required to achieve such high scores.
By the end of training performance on the validation set
had plateaued. We use the best-performing checkpoint and
switch to a 100× larger-scale evaluation for the remainder of
the evaluation.
B. Comparison to State-of-the-Art
In this experiment, we perform a large-scale evaluation of the
LLM’s ability to predict pass lists in comparison to baselines.
4
Figure 3: Frequency that passes occur in the pass list for each of the 100,000 test programs (left), and the length of pass lists (right). -Oz is the starting point
for the autotuner and is the dominant result, being the best-found result for 93.2% of autotuned test programs and appearing in an additional 0.6% of pass lists
as part of a longer sequence. The model-generated pass distribution tracks the autotuner but slightly overpredicts -Oz (94.3%) and includes 9 passes that the
autotuner used on the training set but not on the test set. Results are ordered by decreasing autotuner frequency.
define i32 @f1(i8 %0) {
%2 = alloca i32, align 4
%3 = alloca i8, align 1
store i8 %0, i8* %3, align 1
%4 = load i8, i8* %3, align 1
%5 = zext i8 %4 to i32
%6 = icmp sge i32 %5, 65
br i1 %6, label %7, label %15
7:
%8 = load i8, i8* %3, align 1
%9 = zext i8 %8 to i32
%10 = icmp sle i32 %9, 90
br i1 %10, label %11, label %15
11:
%12 = load i8, i8* %3, align 1
%13 = zext i8 %12 to i32
<snip 21 lines...>
33:
%34 = load i32, i32* %2, align 4
ret i32 %34
}
(a) Input code (39 instructions).
define i32 @f1(i8 %0) {
%2 = zext i8 %0 to i32
%.off = add i8 %0, 191
%3 = icmp ult i8 %.off, 26
br i1 %3, label %4, label %6
4:
%5 = add nsw i32 %2, 191
br label %10
6:
%.reload16.off = add nsw i32 %2, 159
%7 = icmp ult i32 %.reload16.off, 26
br i1 %7, label %10, label %8
8:
%9 = icmp eq i8 %0, 32
%. = select i1 %9, i32 26, i32 1
br label %10
10:
%.0.reg2mem.0 = phi i32 [%5, %4],
[%., %8], [%.reload16.off, %6]
ret i32 %.0.reg2mem.0
}
(b) Autotuned code (14 instructions) using passes:
-reg2mem -instcombine -Os -O1.
define i32 @f1(i8 %0) {
%2 = zext i8 %0 to i32
%.off = add i8 %0, 191
%3 = icmp ult i8 %.off, 26
br i1 %3, label %6, label %._crit_edge
._crit_edge:
%.off24 = add i8 %0, 159
%4 = icmp ult i8 %.off24, 26
br i1 %4, label %6, label %._crit_edge9
._crit_edge9:
%5 = icmp eq i8 %0, 32
%spec.select = select i1 %5,
i32 26, i32 1
ret i32 %spec.select
6:
%.sink = phi i32 [191, %1],
[159, %._crit_edge]
%7 = add nsw i32 %.sink, %2
ret i32 %7
}
(c) Model-optimized code (13 instructions) and
pass list: -reg2mem -simplifycfg -mem2reg
-jump-threading -Os.
Listing 1: An example IR function where the model suggests a better pass list than the autotuner, despite having never seen this code before. For this function
the autotuner tried 26k different pass orderings. The pass list generated by the model appears 5 times in the training set of 1,000,000 examples.
Datasets We aggregate a broad suite of benchmark datasets
for evaluation, summarized in Table II. We deduplicate and
exclude IR functions identical to those we trained on. Our test
data comprises code from a variety of domains including coding
competitions (AI-SOCO [31], POJ-104 [33]), compiler test case
generators (CSmith [34], Y ARPGen [35]), and miscellaneous
publicly available code (ExeBench [32], Transcoder [12]).
Baselines We compare our approach to three baselines:
AutoPhase [39], Coreset-NVP [20], and the Autotuner.
AutoPhase [39] is a reinforcement learning approach in
which an agent is trained using Proximal Policy Optimiza-
tion [42] to select the sequence of optimization passes that will
maximize cumulative instruction count savings over a fixed-
length episode. At each step, the program being optimized
is represented to the agent as a 56-dimensional vector of
instruction counts and other properties. We replicate the
environment of [39] but use the implementation and expanded
training regime from [27] in which the agent is trained for
100,000 episodes. We train the agent on the same data as
our language model (Table I) and evaluate agent performance
periodically during training on a holdout validation set. As in
prior works, we use an action space and episode length of 45.
Coreset-NVP [20] is a technique that combines iterative
search with a learned cost model. First, a greedy search is run
on 17,500 benchmarks to determine a Core set of best pass
lists. Then a Neural Value Prediction (NVP) is trained on the
results of this search, using ProGraML [21] graphs processed
by a Graph Convolutional Network as program representation.
At inference, Coreset-NVP predicts the normalized reward and
tries the first few pass sequences with the highest normalized
5
Figure 4: Improvement over -Oz by dataset. Handwritten code optimizes more.
reward. The total number of passes it is allowed to try for
each benchmark is 45, following prior works. We use author-
provided model weights to perform inference on our test set.
Finally, we compare it to the Autotuner that we used to
generate training data. We autotuned the test dataset in the
same manner as the training data, described in Section III-B.
Results Table III summarizes the results. Our approach
outperforms -Oz, AutoPhase, and Coreset-NVP across all
datasets. Overall, the thousands of optimization attempts that
are afforded to the autotuner enable it to discover the best-
performing pass lists.
AutoPhase and Coreset-NVP are both able to identify pass
lists that outperform -Oz but have an overall net negative impact
on instruction count due to a large number of regressions. We
propose a simple “-Oz backup” extension to overcome this: if a
model predicts a pass list other than -Oz, we also run -Oz and
select the best of the two options. This prevents regressions w.r.t.
-Oz, but increases the number of additional compilations by the
number of times the model predicts a pass list other than -Oz.
Table IV shows the results of the techniques when evaluated in
this manner. While this does not help the models find further
improvements, the lack of regressions means that AutoPhase
and Coreset-NVP now achieve overall improvements over -Oz,
though still less than the LLM with or without the -Oz backup.
C. Evaluation of Generated Pass Lists
Figure 3 shows the frequency with which passes are selected
by the autotuner and our model from the previous experiment.
The distribution of passes selected by the model broadly
tracks the autotuner. -Oz is the most frequently optimal pass.
Excluding -Oz, model-generated pass lists have an average
length of 3.4 (max 10), and autotuner pass lists have an average
length of 3.1 (max 9). 105 of the pass lists generated by the
model never appear in the training data.
In 710 cases the model-generated pass lists outperform the
autotuner on the test set, though improvements are typically
small. Listing 1 shows an example where the model-generated
Figure 5: Improvement over -Oz by input size. Larger codes optimize more.
Table V: Compiler errors of model-optimized code on 100,000 unseen inputs.
error category n
type error 5,777
instruction forward referenced 1,521
undefined value 1,113
invalid redefinition 616
syntax error 280
invalid value for constant 144
undefined function 112
index error 98
other 83
Total 9,744
pass list simplifies control flow to fewer blocks, saving one
further instruction.
Figure 4 breaks down the improvement of each approach to
pass ordering by benchmark dataset. The biggest improvements
over -Oz is found in the POJ-104 and Transcoder datasets,
which both aggregate large amounts of handwritten code, while
Y ARPGen, a random program generator for testing compilers,
has the fewest opportunities for improving over -Oz.
We discovered that there is a strong correlation between the
input program size and the potential performance improvement
over -Oz that is found by both the autotuner and the model.
Figure 5 plots this trend, showing clearly that larger programs
have more opportunities to improve over -Oz.
D. Evaluation of Generated Code
In this section, we evaluate the quality of model-generated
code. To do this we ran the auxiliary training task of generating
optimized code for all 100k functions in the test set. Note that
this is not required to generate the pass lists evaluated in
the previous section. We have made minor edits to the code
samples in this section for brevity such as omitting superfluous
statements and shortening identifier names.
In 90.3% of cases, the model-generated optimized IR com-
piles, and in 68.4% of cases the output IR matches character-
for-character the ground truth generated by the compiler. We
taxonomize the different classes of errors for the 9.7% of cases
where the generated IR does not compile in Table V, and
Listing 2 provides code examples.
6
error: ’%15’ defined with type ’i32’ but expected ’i1’
%or.cond = or i1 %14, %15
(a) The model defined %15 as an integer but later tried to use it as a bool
(type error).
error: constant expression type mismatch
@.str = private unnamed_addr constant [493 x i8]
c"<snip 492 chars ...>", align 1
(b) The model omitted a single character when transcribing a 493-character
string-literal from the input code (type error).
error: floating point constant invalid for type
%1 = tail call i32 @f1(float -0.47799998483256463,
float -1.8159999847412109)
(c) LLVM requires exact decimal values for floating-point constants. These
model-generated values have repeating decimals in binary so are rejected
(invalid value for constant) .
Listing 2: Compiler errors in model-optimized code.
define hidden signext i8 @f1() #0 {
%1 = alloca i64, align 8
store i64 3718042838174166437, i64* %1, align 8
%2 = load i64, i64* %1, align 8
%3 = trunc i64 %2 to i8
ret i8 %3
}
(a) Input unoptimized code.
define hidden signext i8
@f1() #0 {
ret i8 165
}
(b) Desired optimized code.
define hidden signext i8
@f1() #0 {
ret i8 1
}
(c) Model-generated code.
Listing 3: An example where the model generates compilable code but fails to
compute the correct answer for a numeric expression. Producing the correct
result for this expression requires non-trivial mathematical reasoning.
Run passes -instcombine -simplifycfg to reduce
instruction count from 14 to 7:
define dso_local i32 @f1(i32 %0) {
%2 = load i64, i64* getelementptr inbounds(
%struct.t2, %struct.t2* @gvar, i64 0, i32 0), align 8
%3 = icmp eq i64 %2, 0
%4 = icmp eq i32 %0, 0
%or.cond = or i1 %3, %4
%5 = load i32, i32* @S64_MAX, align 4
%6 = select i1 %or.cond, i32 %5, i32 %0
ret i32 %6
}
Listing 4: An example where the model generates correctly optimized code
but fails to produce the pass list needed to produce the desired code. The
model-optimized code and instruction count predictions match the performance
of the autotuner, but the model omitted the -mem2reg pass needed to achieve
this code. The model-generated pass list yields 10 instructions instead of 7.
define i32 @f1(
i32 %0,
i32 %1
) align 2 {
br label %3
3:
%i = phi i32 [%7, %6], [2, %2]
%4 = mul nsw i32 %i, %i
%5 = icmp sgt i32 %4, %1
br i1 %5, label %8, label %6
6:
%7 = add i32 %i, 1
br label %3
8:
ret i32 2
}
(a) Desired optimized code.
int f1(int x, int y) {
int i = 2;
while (i * i < y) {
i += 1;
}
return 2;
}
(b) Equivalent (hand-written) C
code.
define i32 @f1(
i32 %0,
i32 %1
) align 2 {
ret i32 2
}
(c) Model-optimized code.
Listing 5: An example of an unsafe optimization by the model. The 33-
instruction input program (not shown) contains a loop that is not always safe
to optimize away. For example, when y = INT_MAX the loop never terminates.
Figure 6: Model-optimized code quality as a function of the performance of
the generated pass list. Code quality is lower when the pass list performs worse
than -Oz. The model-optimized code resembles the ground truth less (lower
BLEU score), the code is less likely to compile, and the model struggles to
estimate the instruction count (higher error). Error bars show 95% confidence
intervals.
Most challenging to evaluate are the 21.9% of cases where
the model-optimized code compiles but is not a character-by-
character match with the compiler. There are two challenges:
the first is that text precision metrics such as BLEU score are
sensitive to differences in the code such as variable names and
commutative operand order that do not affect the behavior
of the code. Tools like LLVM-Canon [43] can help here
but come with their own set of drawbacks. However, in
many cases, it is unclear whether the behavior of two IRs
is the same, so the second challenge we face is in evaluating
semantic equivalency. Since not all of the datasets we use for
testing provide driver scripts and input datasets for their code,
we cannot use execution-based equivalence checks such as
differential testing [44].
Listing 3 shows an example of model-generated code that
has incorrect program semantics. Here, the lower 8 bits of a
64-bit literal are truncated and returned. The compiler performs
this calculation and substitutes the correct value. The model
7
Figure 7: Training a model to predict single optimization passes. The top subplot evaluates the quality the of generated code for the corresponding pass
(ordered by BLEU score). The bottom subplot shows the frequency that the corresponding pass contributed to an improvement or regression of instruction
count over -Oz.
Figure 8: Ablating the impact of training data size and the auxiliary co-training
task of generating optimized code (denoted No Aux). Data size is measured as
a number of training examples. The graph shows performance on a holdout
validation set during training.
Table VI: Ablation experiments. We evaluate the impact of varying training
data size and of training the model to generate the optimized code. We train
each model for 30k steps and report performance of the best model checkpoint
on a holdout validation set of 1,000 unseen IR functions.
n training
examples
generate
optimized code?
overall
improvement
1,000,000 ✓ 4.95% (—)
500,000 ✓ 3.91% (-21%)
250,000 ✓ 3.74% (-24%)
1,000,000 × 4.15% (-16%)
recognizes that the expression can be calculated at compile time
but fails to compute the correct value. This type of mathematical
reasoning is a known weakness of LLMs [24].
Sometimes the model generates correctly-optimized code
but fails to produce the pass list needed to achieve it. Listing 4
shows one such example. A further class of error is when the
model makes unsafe optimizations by failing to analyze the
input code. Listing 5 shows an example.
We observe an interesting connection between the quality
of pass lists and the corresponding optimized code, shown in
Figure 6. When the model produces a poor-performing pass
list, the quality of the generated code is lower.
V. A DDITIONAL EXPERIMENTS
In the previous section, we evaluated the performance of
an LLM trained to optimize LLVM-IR for code size. In this
section, we build additional models to better understand the
properties of LLMs for code optimization. All models use the
same architecture and parameters as in Section III.
A. Abalation of Dataset Size
We ablate the contribution of dataset size by training two
additional models and varying the amount of the training data
from 50% (500k examples) down to 25% (250k examples)
by random dropout. Figure 8 shows progress during the
training of the models. For dataset sizes of 50% and 25%,
the models begin to overfit the training set after around 8B
training tokens. Table VI shows the peak performance of
each configuration. With 50% and 25% of the training data,
downstream performance falls by 21% and 24%, respectively.
B. Abalation of Code Optimization Task
We train the model to generate not just a pass list but also the
optimized code resulting from this pass list. One may expect
this to degrade model performance – not only must it learn
to predict good pass lists, but also how to produce correctly
optimized code, a more difficult task. In fact, we believe this
to be crucial to model performance. By forcing LLMs to learn
the semantics of LLVM-IR we enable them to make better
optimization decisions.
To ablate this we trained a model to generate only pass lists
without the corresponding optimized code. We kept the data
mix and all other parameters the same. Figure 8 and Table VI
show that without training the model to generate optimized
code, downstream performance falls by 16%.
8
Optimize the following LLVM-IR using -name-anon-globals:
@0 = private
@anon.2ef3bda806391c61822366a2a59f2569.0 = private
@anon.95277a486ffed0b6ba33ab3385b3d7bd.0 = private
,→unnamed_addr constant [14 x i8] c"<snip>", align 1
define dso_local i32 @f1(i8* %0) {
%2 = call i32 @f2(i8* %0, i8* getelementptr inbounds(
,→[14 x i8], [14 x i8]*
,→@0,
@anon.2ef3bda806391c61822366a2a59f2569.0,
@anon.95277a486ffed0b6ba33ab3385b3d7bd.0,
,→i64 0, i64 0))
ret i32 %2
}
(a) Failure due to incomplete information. The -name-anon-globals pass
uses the module name to compute a hash. Lacking this, the model hallucinates
a random hash.
Optimize the following LLVM-IR using -instcombine:
@var_12 = external dso_local global i64, align 8
@var_13 = external dso_local global i32, align 4
@var_14 = external dso_local global i32, align 4
define dso_local void @f1(i64 %arg) {
%tmp = alloca i64, align 8
store i64 %arg, i64* %tmp, align 8
%tmp1 = load i64, i64* %tmp, align 8
%tmp2 = sub i64 0, %tmp1
%tmp3 = sub i64 0, %tmp2
store i64 %tmp3, i64* @var_12, align 8
store i64 %arg, i64* @var_12, align 8
store i64 0, i64 * @var_12, align 8
store i32 1, i32* @var_13, align 4
store i32 0, i32* @var_14, align 4
ret void
}
(b) Failed data-flow analysis. The model correctly removes redundant instruc-
tions but substites the wrong value for a variable. The model-optimized code
compiles and has a high BLEU score, but is incorrect.
Listing 6: Example failures from the pass translation experiment. We combine
the model input (red), ground-truth (blue), and model-generated (green) texts
into a single unified diff for brevity. Black text is common to all three.
C. Evaluation of Single Pass Translation
In previous sections we trained LLMs to orchestrate opti-
mization passes to produce the best-optimized code. In this
section, we evaluate the ability of LLMs to emulate the different
optimizations in themselves. For this experiment, the model
input is an unoptimized IR and the name of an optimization
pass to apply, the output is the IR after applying this pass.
Dataset We generate a new dataset for this task using
60 optimization passes and applying them randomly to the
programs from Table I. We augment the dataset of unoptimized
code with partially optimized code by first running a sequence
of randomly selected passes on unoptimized IRs before the
desired target pass. We collect 10,000 unique (prompt, answer)
examples for each of the 60 passes for a total of 600k examples.
Model We trained a new model from scratch on this pass
translation dataset. It reached peak performance after 11B
training tokens (74 GPU days).
Results Figure 7 summarizes model performance. The aver-
age BLEU score over all passes is 0.846, with exact character-
Listing 7: Example of correct generation of optimized IR. The model
performed several complex optimizations including control-flow simplification
and replacing if-then-else code blocks with instructions.
by-character matches 73.7% of the time and compilable code
82.3% of the time. We also plot the frequency with which each
of the optimizations appears in a model-generated pass list
that improved or regressed performance over -Oz in Table III.
We find no correlation between code quality metrics and its
frequency in generated pass lists.
As can be seen, many passes are learned near-perfectly while
others prove more challenging. Of the passes that perform
poorly, some of them hint at simple improvements to the
representation while others result from deeper limitations of
the model’s reasoning. Listing 6a shows an example from
the -name-anon-globals pass, which is a simple utility pass
that renames anonymous global variables using a hash of the
module name. Since we do not provide the module name in
the prompt, the LLM is forced to hallucinate random values.
We will add the module name to prompts to address this.
Listing 6b shows an example from the -instcombine pass.
This is a complex pass that is implemented in over 4.5k lines of
C++ code in LLVM. We see that the model correctly identifies
the instructions to combine, but makes an error in data flow
analysis and substitutes an incorrect value. This is an important
optimization that frequently occurs in pass lists that outperform
-Oz. We will explore an active learning approach in which more
9
examples are provided for complex and difficult passes.
Finally, we present an example of correct model optimization
in Listing 7. The example combines several non-trivial code
manipulations: register allocation, control flow graph simplifica-
tion, and instruction combining. We visualize the control- and
data-flow graphs to help interpret the changes that the model
made. Even on the scale of these small IR functions, we find
the sophisticated grasp of LLVM-IR semantics demonstrated
by the LLM remarkable. The model has learned to perform
these optimizations entirely from examples, without access to
the compiler implementation.
VI. D ISCUSSION
We have shown that LLMs can near-perfectly emulate many
compiler optimizations and outperform prior approaches, but
there are limitations. This section aims to provide a pragmatic
discussion of limits and directions for future research.
A. Context Window
The main limitation of LLMs is the limited sequence length
of inputs (context window). In this work we target 2k-token
context windows and split IRs into individual functions to max-
imize the amount of code we can fit into the context window.
This is undesirable for a number of reasons. First, it limits
the context available to the model when making optimization
decisions; second, it prevents intra-function optimization; third,
we cannot optimize code that does not fit within the context
window. Figure 5 suggests that larger programs have more
interesting optimization opportunities.
Researchers are adopting ever-increasing context win-
dows [45], but finite context windows remain a common
concern with LLMs. As new techniques for handling long
sequences continue to evolve we plan to incorporate them and
apply them to code optimization, e.g. Code Llama’s variant
of positional interpolation [46] which is RoPE base period
scaling [9] or recent length extrapolation techniques [47].
B. Math Reasoning and Logic
Compilers perform lots of arithmetic. Whenever possible
expressions are evaluated at compile time to minimize work at
runtime and to expose further opportunities for optimization.
We see examples of LLMs struggling with this type of
reasoning, e.g. failed constant folding (Listing 3) and failed
data-flow analysis (Listing 6b).
We think that a chain-of-thought approach [48] in which
models are taught to decompose complex reasoning problems
into incremental steps will prove fruitful. We took the first step
in this direction by breaking optimizations down into individual
passes in Section V-C. We also plan to focus training on a
curriculum of arithmetic and logic, and train LLMs that use
tools to compute intermediate results [49, 50].
C. Inference Speed
Compilers are fast. It takes two orders of magnitude more
time for the model to generate a pass list than it does for
the compiler to execute it. While this is much faster than the
autotuner it is trained on, it remains an overhead that may prove
prohibitive for some applications. That is to say nothing of the
difference in compute resources needed to evaluate compiler
heuristics vs. a 7B-parameter LLM running on multiple GPUs.
In addition to aggressive batching and quantization [51],
significant inference speedups can be achieved by specializing
the vocabulary to a use case. For example, we can reduce entire
subsequences of passes to single vocabulary elements using
Byte Pair Encoding so that at inference time fewer tokens need
to be generated.
VII. R ELATED WORK
Compiler pass ordering for performance has been exploited
for decades [26, 52, 53]. Over the years there have been several
approaches using machine learning [18–20, 39, 54, 55]. The
application of machine learning in compilers is not limited to
pass order and has been applied to many other problems [17,
56–59]. No one has applied LLMs to the problem of pass
ordering, we are the first to do so.
Neural machine translation is an emerging field that uses
language models to transform code from one language to
another. Prior examples include compiling C to assembly [11],
assembly to C [36, 60], and source-to-source transpilation [10].
In these works code correctness cannot be guaranteed. In our
work we use code generation solely as an auxiliary learning
task – correctness is supplied by the compiler.
Language models have found broad adoption for coding tasks,
though few operate at the level of compiler IR. Gallagher et al.
train a RoBERTA architecture on LLVM-IR for the purpose
of code weakness identification [61] and Transcoder-IR [12]
uses LLVM-IR as a pivot point for source-to-source translation.
Neither use LLMs for optimization as we do.
Many language models have been trained on source
code including CodeBERT [62], GraphCodeBERT [63], and
CodeT5 [64] which are trained to perform multiple tasks
including code search, code summarization, and documentation
generation. LLMs trained on source code have also been used
for program fuzzing [13, 14, 65], test generation [15], and
automated program repair [66–68]. A large number of useful
applications have been explored for language models, however,
this is the first work where an LLM is used specifically for
optimizing code.
Most LLMs are trained at least partly on code [3, 5, 25,
69]. Some LLMs are trained similarly to general models but
especially target programming languages and can be used for
code completion such as Codex [8] which powers Copilot [70].
The introduction of fill-in-the-middle capabilities is especially
useful for real-world code completion use cases and has
become common in recent code models such as InCoder [6],
SantaCoder [4], StarCoder [1], and Code Llama [9]. Code
Llama was also trained to follow instructions and generate
code as well as explain its functionalities.
While the multi-terabyte training corpora for these models
contain some assembly, we believe that a focused exploration
of the value of LLMs in the domain of compilers will be of
value to the community. This paper aims to provide that.
10
VIII. C ONCLUSIONS
We present the first steps towards LLMs for code optimiza-
tion. We construct a model that can predict good optimization
strategies for unseen LLVM-IR. Results are promising, though
we face challenges in sequence length which limits us to
operating over small program fragments, and in arithmetic
reasoning which limits the ability of the model to predict the
outcome of optimizations. We hope to inspire the research
community to push beyond LLMs for simple max-likelihood
code generation and into performance-aware code optimization.
REFERENCES
[1] R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov,
C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, Q. Liu, E.
Zheltonozhskii, et al. “StarCoder: may the source be with you!”
In: arXiv:2305.06161 (2023).
[2] Y . Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R.
Leblond, T. Eccles, J. Keeling, F. Gimeno, A. D. Lago, T.
Hubert, P. Choy, et al. “Competition-Level Code Generation
with AlphaCode”. In: Science 378.6624 (2022).
[3] OpenAI. “GPT-4 Technical Report”. In: arXiv:2303.08774
(2023).
[4] L. B. Allal, R. Li, D. Kocetkov, C. Mou, C. Akiki, C. M.
Ferrandis, N. Muennighoff, M. Mishra, A. Gu, M. Dey, L. K.
Umapathi, C. J. Anderson, et al. “SantaCoder: don’t reach for
the stars!” In: arXiv:2301.03988 (2023).
[5] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra,
A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann,
P. Schuh, K. Shi, et al. “PaLM: Scaling Language Modeling
with Pathways”. In: arXiv:2204.02311 (2022).
[6] D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi,
R. Zhong, W.-t. Yih, L. Zettlemoyer, and M. Lewis. “InCoder:
A Generative Model for Code Infilling and Synthesis”. In:
arXiv:2204.05999 (2023).
[7] S. Gunasekar, Y . Zhang, J. Aneja, C. C. T. Mendes, A. D.
Giorno, S. Gopi, M. Javaheripi, P. Kauffmann, G. de Rosa,
O. Saarikivi, A. Salim, S. Shah, et al. “Textbooks Are All You
Need”. In: arXiv:2306.11644 (2023).
[8] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto,
J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman,
A. Ray, R. Puri, et al. “Evaluating Large Language Models
Trained on Code”. In: arXiv:2107.03374 (2021).
[9] B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan,
Y . Adi, J. Liu, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov,
et al. “Code Llama: Open Foundation Models for Code”. In:
arXiv:2308.12950 (2023).
[10] M.-A. Lachaux, B. Roziere, L. Chanussot, and G. Lample.
“Unsupervised Translation of Programming Languages”. In:
arXiv:2006.03511 (2020).
[11] J. Armengol-Estapé and M. F. O’Boyle. “Learning C to
x86 Translation: An Experiment in Neural Compilation”. In:
arXiv:2108.07639 (2021).
[12] M. Szafraniec, B. Roziere, F. Charton, H. Leather, P. Labatut,
and G. Synnaeve. “Code Translation with Compiler Represen-
tations”. In: arXiv:2207.03578 (2022).
[13] G. Ye, Z. Tang, S. H. Tan, S. Huang, D. Fang, X. Sun, L. Bian,
H. Wang, and Z. Wang. “Automated conformance testing for
JavaScript engines via deep compiler fuzzing”. In: PLDI. 2021.
[14] Y . Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang. “Large
Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-
Learning Libraries via Large Language Models”. In: ISSTA.
2023.
[15] M. Schäfer, S. Nadi, A. Eghbali, and F. Tip. “Adaptive Test Gen-
eration Using a Large Language Model”. In: arXiv:2302.06527
(2023).
[16] OpenAI. ChatGPT. https://chat.openai.com/.
[17] M. Trofin, Y . Qian, E. Brevdo, Z. Lin, K. Choromanski,
and D. Li. “MLGO: a Machine Learning Guided Compiler
Optimizations Framework”. In: arXiv:2101.04808 (2021).
[18] Z. Wang and M. O’Boyle. “Machine Learning in Compiler
Optimisation”. In: arXiv:1805.03441 (2018).
[19] H. Leather and C. Cummins. “Machine Learning in Compilers:
Past, Present and Future”. In: FDL. 2020.
[20] Y . Liang, K. Stone, A. Shameli, C. Cummins, M. Elhoushi,
J. Guo, B. Steiner, X. Yang, P. Xie, H. Leather, and Y . Tian.
“Learning Compiler Pass Orders using Coreset and Normalized
Value Prediction”. In: ICML. 2023.
[21] C. Cummins, Z. Fisches, T. Ben-Nun, T. Hoefler, M. O’Boyle,
and H. Leather. “ProGraML: A Graph-based Program Repre-
sentation for Data Flow Analysis and Compiler Optimizations”.
In: ICML. 2021.
[22] C. Lattner and V . Adve. “LLVM: A Compilation Framework
for Lifelong Program Analysis & Transformation”. In: CGO.
2004.
[23] N. Asher, S. Bhar, A. Chaturvedi, J. Hunter, and S. Paul. “Lim-
its for Learning with Language Models”. In: arXiv:2306.12213
(2023).
[24] J. Qian, H. Wang, Z. Li, S. Li, and X. Yan. “Limitations of
Language Models in Arithmetic and Symbolic Induction”. In:
arXiv:2208.05051 (2022).
[25] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi,
Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al.
“Llama 2: Open Foundation and Fine-Tuned Chat Models”. In:
arXiv:2307.09288 (2023).
[26] G. G. Fursin, M. F. P. O’Boyle, and P. M. W. Knijnenburg.
“Evaluating Iterative Compilation”. In: LCPC. 2005.
[27] C. Cummins, B. Wasti, J. Guo, B. Cui, J. Ansel, S. Gomez,
S. Jain, J. Liu, O. Teytaud, B. Steiner, Y . Tian, and H. Leather.
“CompilerGym: Robust, Performant Compiler Optimization
Environments for AI Research”. In: CGO. 2022.
[28] D. Kocetkov, R. Li, L. B. Allal, J. Li, C. Mou, C. M.
Ferrandis, Y . Jernite, M. Mitchell, S. Hughes, T. Wolf, et al.
“The Stack: 3TB of Permissively Licensed Source Code”. In:
arXiv:2211.15533 (2022).
[29] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster,
J. Phang, H. He, A. Thite, N. Nabeshima, et al. “The Pile: An
800GB Dataset of Diverse Text for Language Modeling”. In:
arXiv:2101.00027 (2020).
[30] H. Husain, H. -H. Wu, T. Gazit, M. Allamanis, and M.
Brockschmidt. “CodeSearchNet Challenge: Evaluating the State
of Semantic Code Search”. In: arXiv:1909.09436 (2019).
[31] A. Fadel, H. Musleh, I. Tuffaha, M. Al-Ayyoub, Y . Jararweh,
E. Benkhelifa, and P. Rosso. “Overview of the PAN@FIRE
2020 task on the authorship identification of SOurce COde
(AI-SOCO)”. In: FIRE. 2020.
[32] J. Armengol-Estapé, J. Woodruff, A. Brauckmann, J. W. d. S.
Magalhães, and M. O’Boyle. “ExeBench: an ML-scale Dataset
of Executable C Functions”. In: MAPS. 2022.
[33] L. Mou, G. Li, L. Zhang, T. Wang, and Z. Jin. “Convolutional
Neural Networks Over Tree Structures for Programming
Language Processing”. In: AAAI. 2016.
[34] X. Yang, Y . Chen, E. Eide, and J. Regehr. “Finding and
Understanding Bugs in C Compilers”. In: PLDI. 2011.
[35] V . Livinskii, D. Babokin, and J. Regehr. “Random Testing for
C and C++ Compilers with Y ARPGen”. In: OOPSLA. 2020.
[36] J. Armengol-Estapé, J. Woodruff, C. Cummins, and M. F.
O’Boyle. “SLaDe: A Portable Small Language Model Decom-
piler for Optimized Assembler”. In: arXiv:2305.12520 (2023).
[37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones,
A. N. Gomez, Ł. Kaiser, and I. Polosukhin. “Attention Is All
You Need”. In: NeurIPS (2017).
11
[38] P. Gage. “A New Algorithm for Data Compression”. In: C
Users Journal 12.2 (1994).
[39] A. Haj-Ali, Q. Huang, W. Moses, J. Xiang, J. Wawrzynek,
K. Asanovic, and I. Stoica. “AutoPhase: Juggling HLS Phase
Orderings in Random Forests with Deep Reinforcement Learn-
ing”. In: MLSys. 2020.
[40] I. Loshchilov and F. Hutter. “Decoupled Weight Decay Regu-
larization”. In: arXiv:1711.05101 (2017).
[41] K. Papineni, S. Roukos, T. Ward, and W. -J. Zhu. “BLEU: A
Method for Automatic Evaluation of Machine Translation”. In:
ACL. 2002.
[42] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O.
Klimov. “Proximal Policy Optimization Algorithms”. In:
arXiv:1707.06347 (2017).
[43] M. Paszkowski. LLVM Canon . https : / / github . com /
michalpaszkowski/LLVM-Canon.
[44] W. M. McKeeman. “Differential Testing for Software”. In:
Digital Technical Journal 10.1 (1998).
[45] J. Ding, S. Ma, L. Dong, X. Zhang, S. Huang, W. Wang,
and F. Wei. “LongNet: Scaling Transformers to 1,000,000,000
Tokens”. In: arXiv:2307.02486 (2023).
[46] S. Chen, S. Wong, L. Chen, and Y . Tian. “Extending Context
Window of Large Language Models via Positional Interpola-
tion”. In: arXiv:2306.15595 (2023).
[47] Y . Sun, L. Dong, B. Patra, S. Ma, S. Huang, A. Benhaim,
V . Chaudhary, X. Song, and F. Wei. “A Length-Extrapolatable
Transformer”. In: arXiv:2212.10554 (2022).
[48] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi,
Q. V . Le, D. Zhou, et al. “Chain-of-thought prompting elicits
reasoning in large language models”. In: NeurIPS. 2022.
[49] L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y . Yang, J. Callan,
and G. Neubig. “Pal: Program-aided language models”. In:
ICML. 2023.
[50] K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun,
L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano,
et al. “Training Verifiers to Solve Math Word Problems”. In:
arXiv:2110.14168 (2021).
[51] G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S.
Han. “SmoothQuant: Accurate and Efficient Post-Training
Quantization for Large Language Models”. In: ICML. 2023.
[52] F. Bodin, T. Kisuki, P. Knijnenburg, M. O’Boyle, and E. Rohou.
“Iterative Compilation in a Non-linear Optimisation Space”. In:
FDO. 1998.
[53] T. Kisuki, P. Knijnenburg, and M. O’Boyle. “Combined
Selection of Tile Sizes and Unroll Factors using Iterative
Compilation”. In: PACT. 2000.
[54] F. Agakov, E. Bonilla, J. Cavazos, B. Franke, G. Fursin, M.
O’Boyle, J. Thomson, M. Toussaint, and C. Williams. “Using
Machine Learning to Focus Iterative Optimization”. In: CGO.
2006.
[55] W. F. Ogilvie, P. Petoumenos, Z. Wang, and H. Leather.
“Minimizing the Cost of Iterative Compilation with Active
Learning”. In: CGO. 2017.
[56] A. H. Ashouri, M. Elhoushi, Y . Hua, X. Wang, M. A. Manzoor,
B. Chan, and Y . Gao. “MLGOPerf: An ML Guided Inliner to
Optimize Performance”. In: arXiv:2207.08389 (2022).
[57] A. Haj-Ali, N. K. Ahmed, T. Willke, S. Shao, K. Asanovic,
and I. Stoica. “NeuroVectorizer: End-to-End Vectorization with
Deep Reinforcement Learning”. In: CGO. 2020.
[58] C. Cummins, P. Petoumenos, Z. Wang, and H. Leather. “End-
to-End Deep Learning of Optimization Heuristics”. In: PACT.
2017.
[59] P. M. Phothilimthana, A. Sabne, N. Sarda, K. S. Murthy, Y .
Zhou, C. Angermueller, M. Burrows, S. Roy, K. Mandke,
R. Farahani, et al. “A Flexible Approach to Autotuning Multi-
pass Machine Learning Compilers”. In: PACT. 2021.
[60] I. Hosseini and B. Dolan-Gavitt. “Beyond the C: Retar-
getable Decompilation using Neural Machine Translation”. In:
arXiv:2212.08950 (2022).
[61] S. K. Gallagher, W. E. Klieber, and D. Svoboda. LLVM
Intermediate Representation for Code Weakness Identification .
2022.
[62] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong,
L. Shou, B. Qin, T. Liu, D. Jiang, et al. “CodeBERT: A Pre-
trained Model for Programming and Natural Languages”. In:
arXiv:2002.08155 (2020).
[63] D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou,
N. Duan, A. Svyatkovskiy, S. Fu, M. Tufano, S. K. Deng,
C. Clement, D. Drain, N. Sundaresan, J. Yin, D. Jiang, and
M. Zhou. “GraphCodeBERT: Pre-training Code Representations
with Data Flow”. In: arXiv:2009.08366 (2021).
[64] Y . Wang, W. Wang, S. Joty, and S. C. Hoi. “CodeT5: Identifier-
aware Unified Pre-trained Encoder-Decoder Models for Code
Understanding and Generation”. In: arXiv:2109.00859 (2021).
[65] C. S. Xia, M. Paltenghi, J. L. Tian, M. Pradel, and L.
Zhang. “Universal Fuzzing via Large Language Models”. In:
arXiv:2308.04748 (2023).
[66] C. S. Xia and L. Zhang. “Less Training, More Repairing Please:
Revisiting Automated Program Repair via Zero-shot Learning”.
In: arXiv:2207.08281 (2022).
[67] C. S. Xia, Y . Wei, and L. Zhang. “Automated Program Repair
in the Era of Large Pre-Trained Language Models”. In: ICSE.
2023.
[68] C. S. Xia and L. Zhang. “Keep the Conversation Going: Fixing
162 out of 337 bugs for $0.42 each using ChatGPT”. In:
arXiv:2304.00385 (2023).
[69] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. -A. Lachaux,
T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al.
“Llama: Open and efficient foundation language models”. In:
arXiv preprint arXiv:2302.13971 (2023).
[70] GitHub. Copilot. https://copilot.github.com/.
12

Gated Feedback Recurrent Neural Networks
Junyoung Chung JUNYOUNG .CHUNG @UMONTREAL .CA
Caglar Gulcehre CAGLAR .GULCEHRE @UMONTREAL .CA
Kyunghyun Cho KYUNGHYUN .CHO @UMONTREAL .CA
Yoshua Bengio∗ FIND -ME@THE .WEB
Dept. IRO, Universit´e de Montr´eal, ∗CIFAR Senior Fellow
Abstract
In this work, we propose a novel recurrent neu-
ral network (RNN) architecture. The proposed
RNN, gated-feedback RNN (GF-RNN), extends
the existing approach of stacking multiple recur-
rent layers by allowing and controlling signals
ﬂowing from upper recurrent layers to lower lay-
ers using a global gating unit for each pair of
layers. The recurrent signals exchanged between
layers are gated adaptively based on the previous
hidden states and the current input. We evalu-
ated the proposed GF-RNN with different types
of recurrent units, such as tanh, long short-term
memory and gated recurrent units, on the tasks
of character-level language modeling and Python
program evaluation. Our empirical evaluation of
different RNN units, revealed that in both tasks,
the GF-RNN outperforms the conventional ap-
proaches to build deep stacked RNNs. We sug-
gest that the improvement arises because the GF-
RNN can adaptively assign different layers to dif-
ferent timescales and layer-to-layer interactions
(including the top-down ones which are not usu-
ally present in a stacked RNN) by learning to gate
these interactions.
1. Introduction
Recurrent neural networks (RNN) have been widely stud-
ied and used for various machine learning tasks which in-
volve sequence modeling, especially when the input and
output have variable lengths. Recent studies have revealed
that RNNs using gating units can achieve promising re-
sults in both classiﬁcation and generation tasks (see, e.g.,
Graves, 2013; Bahdanau et al., 2014; Sutskever et al.,
2014).
Although RNNs can theoretically capture any long-term
dependency in an input sequence, it is well-known to be
difﬁcult to train an RNN to actually do so (Hochreiter,
1991; Bengio et al., 1994; Hochreiter, 1998). One of the
most successful and promising approaches to solve this is-
sue is by modifying the RNN architecture e.g., by using a
gated activation function, instead of the usual state-to-state
transition function composing an afﬁne transformation and
a point-wise nonlinearity. A gated activation function,
such as the long short-term memory (LSTM, Hochreiter
& Schmidhuber, 1997) and the gated recurrent unit (GRU,
Cho et al., 2014), is designed to have more persistent mem-
ory so that it can capture long-term dependencies more eas-
ily.
Sequences modeled by an RNN can contain both fast
changing and slow changing components, and these un-
derlying components are often structured in a hierarchi-
cal manner. A conventional way to encode this hierar-
chy in an RNN has been to stack multiple levels of recur-
rent layers (Schmidhuber, 1992; El Hihi & Bengio, 1995;
Graves, 2013; Hermans & Schrauwen, 2013). More re-
cently, Koutn´ık et al. (2014) proposed a more explicit ap-
proach to partition the hidden units in an RNN into groups
such that each group receives the signal from the input and
the other groups at a separate, predeﬁned rate, which al-
lows feedback information between these partitions to be
propagated at multiple timescales.
In this paper, we propose a novel design for RNNs, called a
gated-feedback RNN (GF-RNN), to deal with the issue of
learning multiple adaptive timescales. The proposed RNN
has multiple levels of recurrent layers like stacked RNNs
do. However, it uses gated-feedback connections from up-
per recurrent layers to the lower ones. This makes the hid-
den states across a pair of consecutive time-steps fully con-
nected. To encourage each recurrent layer to work at differ-
ent timescales, the proposed GF-RNN controls the strength
of the temporal (recurrent) connection adaptively. This ef-
fectively lets the model to adapt its structure based on the
input sequence.
We empirically evaluated the proposed model against the
conventional stacked RNN and the usual, single-layer RNN
on the task of language modeling and Python program eval-
arXiv:1502.02367v3  [cs.NE]  18 Feb 2015
Gated Feedback Recurrent Neural Networks
uation (Zaremba & Sutskever, 2014). Our experiments re-
veal that the proposed model signiﬁcantly outperforms the
conventional approaches on two different datasets.
2. Recurrent Neural Network
A recurrent neural network (RNN) is able to process a se-
quence of arbitrary length by recursively applying a tran-
sition function to its internal hidden state for each symbol
of the input sequence. The activation of the hidden state at
time-step tis computed as a function f of the current input
symbol xt and the previous hidden state ht−1:
ht =f(xt,ht−1) . (1)
It is common to use the state-to-state transition function f
as the composition of an element-wise nonlinearity with an
afﬁne transformation of both xt and ht−1:
ht =φ(Wxt + Uht−1) , (2)
where W is input-to-hidden weight matrix, U is the state-
to-state recurrent weight matrix, and φis usually a logistic
sigmoid function or a hyperbolic tangent function.
We can factorize the probability of a sequence of arbitrary
length into
p(x1,··· ,xT ) =p(x1)p(x2 |x1) ···p(xT |x1,··· ,xT−1).
Then, we can train an RNN to model this distribution by
letting it predict the probability of the next symbol xt+1
given a hidden state vector ht which is a function of all the
previous symbols x1,··· ,xt−1 and current symbol xt:
p(xt+1 |x1,··· ,xt) =g(ht) .
This approach of using a neural network to model a prob-
ability distribution over sequences is widely used, for in-
stance, in language modeling (see, e.g., Bengio et al., 2001;
Mikolov, 2012).
2.1. Gated Recurrent Neural Network
The difﬁculty of training an RNN to capture long-term de-
pendencies has been known for long (Hochreiter, 1991;
Bengio et al., 1994; Hochreiter, 1998). A previously suc-
cessful approaches to this fundamental challenge has been
to modify the state-to-state transition function to encourage
some hidden units to adaptively maintain long-term mem-
ory, creating paths in the time-unfolded RNN, such that
gradients can ﬂow over many time-steps.
Long short-term memory (LSTM) was proposed by
Hochreiter & Schmidhuber (1997) to speciﬁcally address
this issue of learning long-term dependencies. The LSTM
maintains a separate memory cell inside it that updates and
exposes its content only when deemed necessary. More re-
cently, Cho et al. (2014) proposed a gated recurrent unit
(GRU) which adaptively remembers and forgets its state
based on the input signal to the unit. Both of these units are
central to our proposed model, and we will describe them
in more details in the remainder of this section.
2.1.1. L ONG SHORT-TERM MEMORY
Since the initial 1997 proposal, several variants of the
LSTM have been introduced (Gers et al., 2000; Zaremba
et al., 2014). Here we follow the implementation provided
by Zaremba et al. (2014).
Such an LSTM unit consists of a memory cell ct, an input
gate it, a forget gate ft, and an output gate ot. The memory
cell carries the memory content of an LSTM unit, while
the gates control the amount of changes to and exposure
of the memory content. The content of the memory cell
cj
t of the j-th LSTM unit at time-step tis updated similar
to the form of a gated leaky neuron, i.e., as the weighted
sum of the new content˜cj
t and the previous memory content
cj
t−1 modulated by the input and forget gates, ij
t and fj
t ,
respectively:
cj
t =fj
t cj
t−1 + ij
t ˜cj
t. (3)
where
˜ct = tanh (Wcxt + Ucht−1) . (4)
The input and forget gates control how much new content
should be memorized and how much old content should be
forgotten, respectively. These gates are computed from the
previous hidden states and the current input:
it =σ(Wixt + Uiht−1) , (5)
ft =σ(Wf xt + Uf ht−1) , (6)
where it =
[
ik
t
]p
k=1 and ft =
[
fk
t
]p
k=1 are respectively the
vectors of the input and forget gates in a recurrent layer
composed of pLSTM units. σ(·) is an element-wise logis-
tic sigmoid function. xt and ht−1 are the input vector and
previous hidden states of the LSTM units, respectively.
Once the memory content of the LSTM unit is updated, the
hidden state hj
t of the j-th LSTM unit is computed as:
hj
t = oj
t tanh
(
cj
t
)
.
The output gate oj
t controls to which degree the memory
content is exposed. Similarly to the other gates, the out-
put gate also depends on the current input and the previous
hidden states such that
ot =σ(Woxt + Uoht−1) . (7)
Gated Feedback Recurrent Neural Networks
In other words, these gates and the memory cell allow an
LSTM unit to adaptively forget, memorize and expose the
memory content. If the detected feature, i.e., the mem-
ory content, is deemed important, the forget gate will be
closed and carry the memory content across many time-
steps, which is equivalent to capturing a long-term depen-
dency. On the other hand, the unit may decide to reset the
memory content by opening the forget gate. Since these
two modes of operations can happen simultaneously across
different LSTM units, an RNN with multiple LSTM units
may capture both fast-moving and slow-moving compo-
nents.
2.1.2. G ATED RECURRENT UNIT
The GRU was recently proposed by Cho et al. (2014). Like
the LSTM, it was designed to adaptively reset or update
its memory content. Each GRU thus has a reset gate rj
t
and an update gate zj
t which are reminiscent of the forget
and input gates of the LSTM. However, unlike the LSTM,
the GRU fully exposes its memory content each time-step
and balances between the previous memory content and the
new memory content strictly using leaky integration, albeit
with its adaptive time constant controlled by update gate
zj
t .
At time-step t, the state hj
t of the j-th GRU is computed by
hj
t = (1−zj
t )hj
t−1 + zj
t ˜hj
t, (8)
where hj
t−1 and ˜hj
t respectively correspond to the previ-
ous memory content and the new candidate memory con-
tent. The update gate zj
t controls how much of the previous
memory content is to be forgotten and how much of the
new memory content is to be added. The update gate is
computed based on the previous hidden state ht−1 and the
current input xt:
zt =σ(Wzxt + Uzht−1) , (9)
where zt =
[
zk
t
]p
k=1. The new memory content ˜hj
t is com-
puted similarly to the conventional transition function in
Eq. (2):
˜ht = tanh (Wxt + rt ⊙Uht−1) , (10)
where ˜ht =
[
˜hk
t
]p
k=1
and ⊙is an element-wise multiplica-
tion.
One major difference from the traditional transition func-
tion (Eq. (2)) is that the state of the previous step ht−1 is
modulated by the reset gates rt. This behavior allows a
GRU unit to ignore the previous hidden states whenever it
is deemed necessary considering the previous hidden states
and the current input:
rt =σ(Wrxt + Urht−1) . (11)
The update mechanism helps the GRU to capture long-
term dependencies. Whenever a previously detected fea-
ture, or the memory content is considered to be important
for later use, the update gate will be closed to carry the cur-
rent memory content across multiple time-steps. By using
the reset mechanism, the RNN with the GRU units use the
model capacity efﬁciently by allowing each GRU to reset
whenever the detected feature is not necessary anymore.
3. Gated Feedback Recurrent Neural
Network
Although capturing long-term dependencies in a sequence
is an important and difﬁcult goal of recurrent neural net-
works (RNN), it is worthwhile to notice that a sequence
often consists of both slow-moving and fast-moving com-
ponents, of which only the former corresponds to long-term
dependencies. Ideally, an RNN needs to capture both long-
term and short-term dependencies.
El Hihi & Bengio (1995) ﬁrst showed that an RNN can cap-
ture these dependencies of different timescales more easily
and efﬁciently when the hidden units of the RNN is ex-
plicitly partitioned into groups that correspond to differ-
ent timescales. The clockwork RNN (CW-RNN) (Koutn´ık
et al., 2014) implemented this by allowing the i-th mod-
ule to operate at the rate of 2i−1, meaning that the mod-
ule is updated only when tmod 2i−1 = 0. This makes
each module to operate at different rates. In addition, they
precisely deﬁned the connectivity pattern between modules
by allowing the i-th module to be affected by j-th module
when j >i.
Here, we propose to generalize the CW-RNN by allowing
the model to adaptively adjust the connectivity pattern be-
tween the hidden layers in the consecutive time-steps. Sim-
ilar to the CW-RNN, we partition the hidden units into mul-
tiple modules in which each module corresponds to a dif-
ferent layer in a stack of recurrent layers.
Unlike the CW-RNN, however, we do not set an explicit
rate for each module. Instead, we let each module oper-
ate at different timescales by hierarchically stacking them.
Each module is fully connected to all the other modules
across the stack and itself. In other words, we do not de-
ﬁne the connectivity pattern across a pair of consecutive
time-steps. This is contrary to the design of CW-RNN and
the conventional stacked RNN. The recurrent connection
between two modules, instead, is gated by a logistic unit
([0,1]) which is computed based on the current input and
the previous states of the hidden layers. We call this gating
unit a global reset gate, as opposed to a unit-wise reset gate
which applies only to a single unit (See Eqs. (3) and (10)).
Gated Feedback Recurrent Neural Networks
(a) Conventional stacked RNN (b) Gated Feedback RNN
Figure 1. Illustrations of (a) conventional stacking approach and (b) gated-feedback approach to form a deep RNN architecture. Bullets
in (b) correspond to global reset gates. Skip connections are omitted to simplify the visualization of networks.
The global reset gate is computed as:
gi→j = σ
(
wi→j
g hj−1
t + ui→j
g h∗
t−1
)
, (12)
where L is the number of hidden layers, wi→j
g and ui→j
g
are the weight vectors for the input and the hidden states of
all the layers at time-step t−1, respectively. For j = 1,
hj−1
t is xt.
The global reset gategi→j is applied collectively to the sig-
nal from the i-th layer hi
t−1 to the j-th layer hj
t. In other
words, the signal from the layerito the layerjis controlled
based on the input and the previous hidden states.
Fig. 1 illustrates the difference between the conventional
stacked RNN and our proposed GF-RNN. In both mod-
els, information ﬂows from lower layers to upper layers,
respectively, corresponding to ﬁner timescale and coarser
timescale. The GF-RNN, however, further allows infor-
mation from the upper recurrent layer, corresponding to
coarser timescale, ﬂows back into the lower layers, corre-
sponding to ﬁner timescales.
We call this RNN with a fully-connected recurrent tran-
sition and global reset gates, a gated-feedback RNN (GF-
RNN). In the remainder of this section, we describe how to
use the previously described LSTM unit, GRU, and more
traditional tanh unit in the GF-RNN.
3.1. Practical Implementation of GF-RNN
3.1.1. tanh UNIT
For a stacked tanh-RNN, the signal from the previous
time-step is gated. The hidden state of the j-th layer is
computed by
hj
t = tanh
(
Wj−1→jhj−1
t +
L∑
i=1
gi→jUi→jhi
t−1
)
,
where Wj−1→j and Ui→j are the weight matrices of the
incoming connections from the input and the i-th module,
respectively. Compared to Eq. (2), the only difference is
that the previous hidden states are controlled by the global
reset gates.
3.1.2. L ONG SHORT-TERM MEMORY AND GATED
RECURRENT UNIT
In the cases of LSTM and GRU, we do not use the global
reset gates when computing the unit-wise gates. In other
words, Eqs. (5)–(7) for LSTM, and Eqs. (9) and (11) for
GRU are not modiﬁed. We only use the global reset gates
when computing the new state (see Eq. (4) for LSTM, and
Eq. (10) for GRU).
The new memory content of an LSTM at the j-th layer is
computed by
˜cj
t = tanh
(
Wj−1→j
c hj−1
t +
L∑
i=1
gi→jUi→j
c hi
t−1
)
.
In the case of a GRU, similarly,
˜hj
t = tanh
(
Wj−1→jhj−1
t + rj
t ⊙
L∑
i=1
gi→jUi→jhi
t−1
)
.
4. Experiment Settings
4.1. Tasks
We evaluated the proposed gated-feedback RNN (GF-
RNN) on character-level language modeling and Python
Gated Feedback Recurrent Neural Networks
program evaluation. Both tasks are representative exam-
ples of discrete sequence modeling, where a model is
trained to minimize the negative log-likelihood of training
sequences:
min
θ
1
N
N∑
n=1
Tn∑
t=1
−log p
(
xn
t |xn
1 ,...,x n
t−1; θ
)
,
where θ is a set of model parameters.
4.1.1. L ANGUAGE MODELING
We used the dataset made available as a part of the human
knowledge compression contest (Hutter, 2012). We refer
to this dataset as the Hutter dataset. The dataset, which
was built from English Wikipedia, contains100 MBytes of
characters which include Latin alphabets, non-Latin alpha-
bets, XML markups and special characters. Closely follow-
ing the protocols in (Mikolov et al., 2012; Graves, 2013),
we used the ﬁrst 90 MBytes of characters to train a model,
the next 5 MBytes as a validation set, and the remaining
as a test set, with the vocabulary of 205 characters includ-
ing a token for an unknown character. We used the average
number of bits-per-character (BPC,E[−log2 P(xt+1|ht)])
to measure the performance of each model on the Hutter
dataset.
4.1.2. P YTHON PROGRAM EVALUATION
Zaremba & Sutskever (2014) recently showed that a recur-
rent neural network, more speciﬁcally a stacked LSTM, is
able to execute a short Python script. Here, we compared
the proposed architecture against the conventional stacking
approach model on this task, to which refer as Python pro-
gram evaluation.
Scripts used in this task include addition, multiplication,
subtraction, for-loop, variable assignment, logical compar-
ison and if-else statement. The goal is to generate, or pre-
dict, a correct return value of a given Python script. The
input is a program while the output is the result of a print
statement: every input script ends with a print statement.
Both the input script and the output are sequences of char-
acters, where the input and output vocabularies respectively
consist of 41 and 13 symbols.
The advantage of evaluating the models with this task is
that we can artiﬁcially control the difﬁculty of each sam-
ple (input-output pair). The difﬁculty is determined by
the number of nesting levels in the input sequence and the
length of the target sequence. We can do a ﬁner-grained
analysis of each model by observing its behavior on exam-
ples of different difﬁculty levels.
4.2. Models
We compared three different RNN architectures: a single-
layer RNN, a stacked RNN and the proposed GF-RNN. For
Table 1.The sizes of the models used in character-level language
modeling. Gated Feedback L is a GF-RNN with a same number
of hidden units as a Stacked RNN (but more parameters). The
number of units is shown as(number of hidden layers)
× (number of hidden units per layer).
Unit Architecture # of Units
tanh
Single 1 ×1000
Stacked 3 ×390
Gated Feedback 3 ×303
GRU
Single 1 ×540
Stacked 3 ×228
Gated Feedback 3 ×165
Gated Feedback L 3 ×228
LSTM
Single 1 ×456
Stacked 3 ×191
Gated Feedback 3 ×140
Gated Feedback L 3 ×191
each architecture, we evaluated three different transitions
functions: tanh + afﬁne, long short-term memory (LSTM)
and gated recurrent unit (GRU). For fair comparison, we
constrained the number of parameters of each model to be
roughly similar to each other.
For each task, in addition to these capacity-controlled ex-
periments, we conducted a few extra experiments to further
test and better understand the properties of the GF-RNN.
4.2.1. L ANGUAGE MODELING
For the task of character-level language modeling, we con-
strained the number of parameters of each model to corre-
spond to that of a single-layer RNN with 1000 tanhunits
(see Table 1 for more details.) Each model is trained for at
most 100 epochs.
We used RMSProp (Hinton, 2012) and momentum to tune
the model parameters (Graves, 2013). According to the
preliminary experiments and their results on the validation
set, we used a learning rate of 0.001 and momentum coef-
ﬁcient of 0.9 when training the models having either GRU
or LSTM units. It was necessary to choose a much smaller
learning rate of 5×10−5 in the case of tanh units to ensure
the stability of learning. Whenever the norm of the gradient
explodes, we halve the learning rate.
Each update is done using a minibatch of100 subsequences
of length 100 each, to avoid memory overﬂow problems
when unfolding in time for backprop. We approximate full
back-propagation by carrying the hidden state computed at
the previous update to initialize the hidden units in the next
update. After every 100-th update, the hidden states were
reset to all zeros.
Gated Feedback Recurrent Neural Networks
0 100K 200K 300K 400K 500K
Number of Seconds
1.9
2.1
2.3
2.5Validation BPC
Stacked GRU
GF-GRU same # of parameters
GF-GRU same # of hidden units
0 100K 200K 300K 400K 500K 600K
Number of Seconds
1.8
2.0
2.2
2.4
2.6
Stacked LSTM
GF-LSTM same # of parameters
GF-LSTM same # of hidden units
(a) GRU (b) LSTM
Figure 2.Validation learning curves of three different RNN architectures; Stacked RNN, GF-RNN with the same number of model
parameters, and GF-RNN with the same number of hidden units. The curves represent training up to 100 epochs. Best viewed in colors.
Table 2.Test set BPC of models trained on the Hutter dataset for
a 100 epochs. (⋆) We did not train Gated Feedback L models with
tanh units.
tanh GRU LSTM
Single-layer 1.937 1.883 1.887
Stacked 1.892 1.871 1.868
Gated Feedback 1.949 1.855 1.842
Gated Feedback L N/A⋆ 1.813 1.789
4.2.2. P YTHON PROGRAM EVALUATION
For the task of Python program evaluation, we used an
RNN encoder-decoder based approach to learn the map-
ping from Python scripts to the corresponding outputs as
done by Cho et al. (2014); Sutskever et al. (2014) for ma-
chine translation. When training the models, Python scripts
are fed into the encoder RNN, and the hidden state of the
encoder RNN is unfolded for 50 time-steps. Prediction is
performed by the decoder RNN whose initial hidden state
is initialized with the last hidden state of the encoder RNN.
The ﬁrst hidden state of encoder RNN h0 is always initial-
ized to a zero vector.
For this task, we used GRU and LSTM units either with or
without the gated-feedbacks. We constrained the number of
parameters to 2.4M to control the capacity of each model
(each encoder or decoder RNN has three hidden layers with
200 units).
Following Zaremba & Sutskever (2014), we used themixed
curriculum strategy for training each model, where each
training example has a random difﬁculty sampled uni-
formly. We generated 320,000 examples using the script
provided by Zaremba & Sutskever (2014), with the nesting
randomly sampled from [1,5] and the target length from[
1,1010]
.
We used Adam (Kingma & Ba, 2014) to train our models,
and each update was using a minibatch with128 sequences.
We used a learning rate of 0.001 and β1 and β2 were both
set to 0.01. We trained each model for 30 epochs, with
early stopping based on the validation set performance to
prevent over-ﬁtting.
At test time, we evaluated each model on multiple sets of
test examples where each set is generated using a ﬁxed tar-
get length and number of nesting levels. Each test set con-
tains 2,000 examples which are ensured not to overlap with
the training set.
5. Results and Analysis
5.1. Language Modeling
It is clear from Table 2 that the proposed gated-feedback ar-
chitecture outperforms the other baseline architectures that
we have tried when used together with widely used gated
units such as LSTM and GRU. However, the proposed ar-
chitecture failed to improve the performance of a vanilla-
RNN with tanh units. In addition to the ﬁnal modeling
performance, in Fig. 2, we plotted the learning curves of
some models against wall-clock time (measured in sec-
onds). RNNs that are trained with the proposed gated-
feedback architecture tends to make much faster progress
over time. This behavior is observed both when the number
of parameters is constrained and when the number of hid-
den units is constrained. This suggests that the proposed
GF-RNN signiﬁcantly facilitates optimization/learning.
Gated Feedback Recurrent Neural Networks
Table 3.Generated texts with our trained models. Given the seed at the left-most column (bold-faced font), the models predict next
200 ∼ 300 characters. Tabs, spaces and new-line characters are also generated by the models.
Seed Stacked LSTM GF-LSTM
[[pl:Icon]]
[[pt:Icon]]
[[ru:Icon]]
[[sv:Programspraket Icon]]</text>
</revision>
</page>
<page>
<title>Iconology</title>
<id>14802</id>
<revi
<revision>
<id>15908383</id>
<timestamp>
2002-07-20T18:33:34Z
</timestamp>
<contributor>
<username>The Courseichi</userrand
vehicles in [[enguit]].
==The inhibitors and alphabetsy and moral/
hande in===In four [[communications]] and
<revision>
<id>41968413</id>
<timestamp>
2006-09-03T11:38:06Z
</timestamp>
<contributor>
<username>Navisb</username>
<id>46264</id>
</contributor>
<comment>The increase from the time
<title>Inherence relation</title>
<id>14807</id>
<revision>
<id>34980694</id>
<timestamp>
2006-01-13T04:19:25Z
</timestamp>
<contributor>
<username>Ro
<username>Robert]]
[[su:20 aves]]
[[vi:10 Februari]]
[[bi:16 agostoferos´ın]]
[[pt:Darenetische]]
[[eo:Hebrew selsowen]]
[[hr:2 febber]]
[[io:21 februari]]
[[it:18 de februari]]
<username>Roma</username>
<id>48</id>
</contributor>
<comment>Vly’’’ and when one hand
is angels and [[ghost]] borted and
’’mask r:centrions]], [[Afghanistan]],
[[Glencoddic tetrahedron]], [[Adjudan]],
[[Dghacn]], for example, in which materials
dangerous (carriers) can only use with one
Effect of Global Reset Gates
After observing the superiority of the proposed gated-
feedback architecture over the conventional single-layer or
stacked ones, we further trained another GF-RNN with
LSTM units, but this time, after ﬁxing the global reset gates
to 1 to validate the need for the global reset gates. With-
out the global reset gates, feedback signals from the up-
per recurrent layers inﬂuence the lower recurrent layer fully
without any control.
In Fig. 3, it can be seen that this omission of the global
reset gates hurts the performance (cyan) compared to the
one with the global reset gates (magenta). The test set BPC
of GF-LSTM without global reset gates was 1.854.
0 20 40 60 80 100
Number of Epochs
1.9
2.1
2.3
2.5Validation BPC
Stacked LSTM
GF-LSTM w/o global reset gates
GF-LSTM with global reset gates
Figure 3.Validation learning curves of the stacked LSTM, GF-
LSTM with the global reset gates and GF-LSTM without them.
Best viewed in colors.
Qualitative Analysis: Text Generation
Here we qualitatively evaluate the stacked LSTM and GF-
LSTM trained earlier by generating text. We choose a sub-
sequence of characters from the test set and use it as an
initial seed. Once the model ﬁnishes reading the seed text,
we let the model generate the following characters.
Table 3 shows the initial seeds and results of text genera-
tion. We observed that the stacked LSTM failed to close the
tags with </username> and </contributor> in both
trials. However, the GF-LSTM succeeded to close both of
them, which shows that it learned about the structure of
XML tags.
Table 4.Test set BPC of neural language models trained
on the Hutter dataset, MRNN = multiplicative RNN re-
sults from Sutskever et al. (2011) and Stacked LSTM results
from Graves (2013).
MRNN Stacked LSTM GF-LSTM
1.60 1.67 1.58
Large GF-RNN
We trained a larger GF-RNN that has ﬁve recurrent layers,
each of which has 700 LSTM units. This makes it possible
for us to compare the performance of the proposed archi-
tecture against the previously reported results using other
types of RNNs. In Table 4, we present the test set BPC
by a multiplicative RNN (Sutskever et al., 2011), a stacked
LSTM (Graves, 2013) and the GF-RNN with LSTM units.
The performance of the proposed GF-RNN is comparable
to, or better than, the previously reported best results. Note
that Sutskever et al. (2011) used the vocabulary of 86 char-
acters (removed XML tags and the Wikipedia markups),
Gated Feedback Recurrent Neural Networks
2 3 4 5
Nesting
10
8
6
4
Target Length
43.2% 37.6% 34.7% 33.2%
46.7% 41.7% 37.8% 37.2%
51.9% 45.6% 42.3% 40.9%
60.2% 52.4% 48.5% 46.3%
35
40
45
50
55
60
Accuracy [%]
2 3 4 5
Nesting
10
8
6
4
Target Length
45.5% 39.6% 36.5% 35.2%
48.0% 42.8% 38.4% 38.5%
52.5% 45.0% 42.5% 41.3%
62.2% 54.0% 49.6% 47.2%
40
45
50
55
60
Accuracy [%]
2 3 4 5
Nesting
10
8
6
4
Target Length
2.3% 1.9% 1.8% 2.1%
1.3% 1.2% 0.6% 1.3%
0.6% -0.7% 0.2% 0.4%
1.9% 1.6% 1.1% 0.9%
0.0
0.5
1.0
1.5
2.0
Accuracy [%]
2 3 4 5
Nesting
10
8
6
4
Target Length
44.0% 38.8% 36.4% 36.3%
46.8% 42.5% 39.0% 39.1%
52.8% 47.4% 43.7% 43.1%
62.1% 54.9% 51.1% 49.2%
 40
45
50
55
60
Accuracy [%]
2 3 4 5
Nesting
10
8
6
4
Target Length
45.3% 40.7% 37.8% 37.3%
48.8% 44.5% 40.8% 40.7%
55.1% 49.0% 46.1% 45.4%
63.8% 58.1% 53.7% 52.0%
 40
45
50
55
60
Accuracy [%]
2 3 4 5
Nesting
10
8
6
4
Target Length
1.4% 1.9% 1.4% 1.1%
2.0% 2.0% 1.8% 1.6%
2.3% 1.6% 2.4% 2.2%
1.6% 3.2% 2.5% 2.9%
1.5
2.0
2.5
3.0
Accuracy [%]
(a) Stacked RNN (b) Gated Feedback RNN (c) Gaps between (a) and (b)
Figure 4. Heatmaps of (a) Stacked RNN, (b) GF-RNN, and (c) difference obtained by substracting (a) from (b). The top row is the
heatmaps of models using GRUs, and the bottom row represents the heatmaps of the models using LSTM units. Best viewed in colors.
and their result is not directly comparable with ours. In this
experiment, we used Adam (Kingma & Ba, 2014) instead
of RMSProp to optimize the RNN. We used learning rate of
0.001 and β1 and β2 were set to 0.1 and 0.01, respectively.
5.2. Python Program Evaluation
Fig. 4 presents the test results of each model represented in
heatmaps. The accuracy tends to decrease by the growth of
target length or nesting, where the difﬁculty or complexity
of the Python program increases. We observed that in the
most of the test sets, GF-RNNs are outperforming Stacked
RNNs, regardless of the type of units. In Fig. 4-(c), the
red and yellow colors which indicate large gains are con-
centrated on top or right regions (either nesting or target
length increases). It shows that GF-RNN is actually do-
ing even better (relatively speaking) when the number of
nesting grows or the length of target increases, and this im-
plies that it outperforms especially when input sequences
are more complicated.
6. Conclusion
We proposed a novel architecture for deep stacked RNNs
which uses gated-feedback connections between differ-
ent layers. Our experiments focused on challenging se-
quence modeling tasks of character-level language mod-
eling and Python program evaluation. The results were
consistent over different datasets, and clearly demonstrated
that gated-feedback architecture is helpful when the mod-
els are trained on complicated sequences that involve long-
term dependencies. We also showed that gated-feedback
architecture was faster in wall-clock time over the train-
ing and achieved better performance compared to standard
stacked RNN with a same amount of capacity. Large GF-
LSTM was able to outperform the previously reported best
results on character-level language modeling. This sug-
gests that GF-RNNs are also scalable. GF-RNNs were able
to outperform standard stacked RNNs on Python program
evaluation task with varying difﬁculties.
Gated-feedback connection is a simple extension over the
standard stacked RNNs, but it was able to show signiﬁcant
improvements on our benchmark and control experiments.
Acknowledgments
The authors would like to thank the developers of
Theano (Bergstra et al., 2010; Bastien et al., 2012) and
Pylearn2 (Goodfellow et al., 2013). We acknowledge the
support of the following agencies for research funding and
computing support: NSERC, Samsung, Calcul Qu ´ebec,
Compute Canada, the Canada Research Chairs and CI-
FAR.
Gated Feedback Recurrent Neural Networks
References
Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio,
Yoshua. Neural machine translation by jointly learning
to align and translate. Technical report, arXiv preprint
arXiv:1409.0473, 2014.
Bastien, Fr ´ed´eric, Lamblin, Pascal, Pascanu, Razvan,
Bergstra, James, Goodfellow, Ian J., Bergeron, Arnaud,
Bouchard, Nicolas, and Bengio, Yoshua. Theano: new
features and speed improvements. Deep Learning and
Unsupervised Feature Learning NIPS 2012 Workshop,
2012.
Bengio, Y ., Ducharme, R., and Vincent, P. A neural prob-
abilistic language model. In NIPS’2000, pp. 932–938,
2001.
Bengio, Yoshua, Simard, Patrice, and Frasconi, Paolo.
Learning long-term dependencies with gradient descent
is difﬁcult. IEEE Transactions on Neural Networks , 5
(2):157–166, 1994.
Bergstra, James, Breuleux, Olivier, Bastien, Fr ´ed´eric,
Lamblin, Pascal, Pascanu, Razvan, Desjardins, Guil-
laume, Turian, Joseph, Warde-Farley, David, and Ben-
gio, Yoshua. Theano: a CPU and GPU math expression
compiler. In Proceedings of the Python for Scientiﬁc
Computing Conference (SciPy) , June 2010. Oral Pre-
sentation.
Cho, Kyunghyun, Van Merri ¨enboer, Bart, Gulcehre,
Caglar, Bougares, Fethi, Schwenk, Holger, and Ben-
gio, Yoshua. Learning phrase representations using
rnn encoder-decoder for statistical machine translation.
arXiv preprint arXiv:1406.1078, 2014.
El Hihi, Salah and Bengio, Yoshua. Hierarchical recur-
rent neural networks for long-term dependencies. In Ad-
vances in Neural Information Processing Systems , pp.
493–499. Citeseer, 1995.
Gers, Felix A., Schmidhuber, J ¨urgen, and Cummins,
Fred A. Learning to forget: Continual prediction with
LSTM. Neural Computation, 12(10):2451–2471, 2000.
Goodfellow, Ian J., Warde-Farley, David, Lamblin, Pascal,
Dumoulin, Vincent, Mirza, Mehdi, Pascanu, Razvan,
Bergstra, James, Bastien, Fr´ed´eric, and Bengio, Yoshua.
Pylearn2: a machine learning research library. arXiv
preprint arXiv:1308.4214, 2013.
Graves, Alex. Generating sequences with recurrent neural
networks. arXiv preprint arXiv:1308.0850, 2013.
Hermans, Michiel and Schrauwen, Benjamin. Training and
analysing deep recurrent neural networks. In Advances
in Neural Information Processing Systems, pp. 190–198,
2013.
Hinton, Geoffrey. Neural networks for machine learning.
Coursera, video lectures, 2012.
Hochreiter, S. Untersuchungen zu dynamischen
neuronalen Netzen. Diploma thesis, Institut f ¨ur In-
formatik, Lehrstuhl Prof. Brauer, Technische Uni-
versit¨at M ¨unchen, 1991. URL http://www7.
informatik.tu-muenchen.de/˜Ehochreit.
Hochreiter, S. and Schmidhuber, J. Long short-term mem-
ory. Neural Computation, 9(8):1735–1780, 1997.
Hochreiter, Sepp. The vanishing gradient problem dur-
ing learning recurrent neural nets and problem solu-
tions. International Journal of Uncertainty, Fuzziness
and Knowledge-Based Systems, 6(02):107–116, 1998.
Hutter, Marcus. The human knowledge compression con-
test. 2012. URL http://prize.hutter1.net/.
Kingma, Diederik and Ba, Jimmy. Adam: A
method for stochastic optimization. arXiv preprint
arXiv:1412.6980, 2014.
Koutn´ık, Jan, Greff, Klaus, Gomez, Faustino, and Schmid-
huber, J ¨urgen. A clockwork rnn. In Proceedings of
the 31st International Conference on Machine Learning
(ICML’14), 2014.
Mikolov, Tomas. Statistical Language Models based on
Neural Networks. PhD thesis, Brno University of Tech-
nology, 2012.
Mikolov, Tomas, Sutskever, Ilya, Deoras, Anoop,
Le, Hai-Son, Kombrink, Stefan, and Cernocky, J.
Subword language modeling with neural networks.
Preprint, 2012. URL http://www.fit.vutbr.
cz/˜imikolov/rnnlm/char.pdf.
Schmidhuber, J ¨urgen. Learning complex, extended se-
quences using the principle of history compression.Neu-
ral Computation, 4(2):234–242, 1992.
Sutskever, Ilya, Martens, James, and Hinton, Geoffrey E.
Generating text with recurrent neural networks. In Pro-
ceedings of the 28th International Conference on Ma-
chine Learning (ICML’11), pp. 1017–1024, 2011.
Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc VV . Se-
quence to sequence learning with neural networks. In
Advances in Neural Information Processing Systems, pp.
3104–3112, 2014.
Zaremba, Wojciech and Sutskever, Ilya. Learning to exe-
cute. arXiv preprint arXiv:1410.4615, 2014.
Zaremba, Wojciech, Sutskever, Ilya, and Vinyals, Oriol.
Recurrent neural network regularization. arXiv preprint
arXiv:1409.2329, 2014.
Supplementary Material: Gated Feedback Recurrent Neural Networks
Text Generation
Hutter dataset is often referred as a benchmark for
character-level language modeling (Sutskever et al., 2011;
Graves, 2013). In this supplementary material, we provide
some samples generated by the gated-feedback RNN with
LSTM units which was trained on Hutter dataset. We sam-
pled once from the model (except the last sample) after
providing a seed sequence extracted from the test set which
was not used for training the model. We use bold-faced font
whenever the following (generated) characters are consid-
ered highly relevant to the context of earlier seed text. We
use red color to distinguish the seed text snippet from the
generated one.
Sample #1
England, Italy, France, Germany and others.
==Euroscepticism in the United Kingdom==
Humbert and Colombi
* [[Royal Charlemagne|Crown Union of the Eastern
Radio Silver Station]]
(719 AD) explodes almost do train open in reaction to do not
form dry oil. This is an essential implication of [[posttransistor]]
that is significantly assumes modification of question.
For example, it does not entail a Being with a lower case,
and declared they oll] on ’’The Glorification of Touchstuff’’
([[2005 in literate press|prethis version]]) was of the progress of
[@hers]]’’). A top ranking of the decades was the beginning of Fulham
[[Dan DeLisa]] to put the a contract that could become a minimal
operating system in common law. Membership however, stemmes Burroughs
struggles to conduct a museums]] of this national lands are
explicated in all groups, especially in traditional Ukrainian
monetary values.
Sample #2
In Europe, Paris was the center stage for the [[French Revolution]],
and it became an important centre of finance,
commerce, and funding important.
The markets for the Union also include the invention of money lower
risks among a possible sacrifice and electrotext and their chemical
design, though armor facilities featuring a variety of [[oak tree]]s
are applied to the data structures of the object, a risk|supporting,
such as dollarize -200 years ago.
Sample #3
Istanbul is located in north-western [[Turkey]] and south-eastern
Europe within the [[Marmara Region]] on a total area of 600km2
andwards by merging the airport and carry
[[Discount of Cuba#Year of Empire|Temastique]],&quot;
’’considered’’ (secret). ([[#Notable extreme points of North America)
|nut, because it were troppied by broken versions of him with either
the Polish album by the U.S. Attorney’s [[London Eye]]) was
problematic of cladified and even translated, in it among [[Britain]],
the Basque Parliament can be assigned a new [[Austrodeac]]
Damascus is also champains some years. He apparently failed to cover
wrongdoing, Hamming Khan Atmanhorn]] of Lestergompolis of the Tropic
of Churches to the undemocratism with Lieutenants as Spoiler of the
Union of the Honored Matres, Bulgaria had to proilmes mounted
undrinks something instead of the brother, being green and blue.
The [[Russia collapse|race]] received an estimated deconstructive
tension between the Pacific Ocean and the island of [[Northumbria]],
which are less than original Habsburg fast deficits. These are under
[[Louis XIV of France|Louis XIV of France]] and again drove the city
from Rome.
Sample #4
Google was founded by [[Larry Page]] and [[Sergey Brin]]
while they were [[Doctor of Philosophy|Ph.D.]] students
at [[Stanford University]].
Together they own about 14,000 files
fimber|266]] [[January 15|15]] [[January 12|12]] [[January 1]].
Leading the democratisation of the Czech Republic in 1948,
the descendants of [[Charlotte of the Hanseatic Council]]
were trophy in the House of Lords, begun as wives in [[North Africa]],
[[Sicily]], [[Spanish Islam Mary]], [[Africa]] beence (15,401) in 1949.
Brown was refusingly implisf of opushism with some demonstrations of
angels existed by merry with individuals that not yet be seen on the
naked surname Grigorius titled ’Cm by Krozian Islam, the husband of Apis
also the process of deriving opposites octs,
to Governmental Discovery and Health Statistics.
Sample #5
<page>
<title>Iconology</title>
<id>14802</id>
<revision>
<id>15912335</id>
<timestamp>2004-06-27T23:16:08Z</timestamp>
<contributor>
<username>Gaius Cornelius</username>
<id>2228</id>
</contributor>
<comment>/* Hoffers: "The main act’s panda woman who renegates
people in the song. That’s good for them"
[[Enoch Grimes]] has a style of trained music and admirer [[Mark Drake]].
Chandler entered the Piazza Hauldry’s wife, although remaining during
the-1910 Carter’s website, among others mainly called Stadium,
a horse that England was showing stones by two of his descendents and
was passed for the rest of shakeup out of purpose there. The exact date
is now a matter of detail with popular major offensive products.
In 1989, Ducati was replaced with the [[Bodleian Government]],
which rose from [[radio waves]] to the [[NASCAR]] clinical development
in the mid-1990s in southern [[Canada]].
Sample #6
Wikiquote
*[http://www.indianajones.com/ IndianaJones.com;
the official Indiana Jones site]
*[http://www.theraider.net TheRaider.net;
the primary fan site of the series]
*[http://indianajones.wikicities.com The Indiana Jones Wiki]
*[http://www.theindyexperience.com The Indy Experience]
*[http://www.indygear.org/ Eugenics]],
which presumably includes an inside Liberals or Civilian leader
JRDC President. Two large faculty alien colonies led by Thomas
Coaitle fit of Heracles who lives with Menander, Zeus, has a world
ticket. Officially there is another Challenge to the Queen’s Law for
all possessions.) There is often no relenant]] and weapons systems.
* [[C Implementation for League Championship (novel)|The Edgar,
feather-and-chestglum.shtml (see below)]] During the head and the
feast day and/or use of mud, deppeus (knife), [[doulcal]], [[tuna]]
Gated Feedback Recurrent Neural Networks
Sample #7
==Disadvantages of IMAP==
* IMAP is a very heavy and complicated protocol.
Writing your own custom implementation of an
IMAP server is of at least 20 orders of magnitude
more complicated than a POP3 implementation.
Client implementations are also much more complicated.
* Due to its capabilities, mythology’s possibility is still
incomplete.
Commentators on epistemic definitions typical of the
past fall short of ethical altruism limit, arguing
that the delegation should not be appropriate to
oppress it, without expressive our knowledge about
free market activities, as well as some - it is
possible to check the limits on Wikipedia.
Put in direct film video - Advance Australia
[[Category:Cheshire| ]]
[[de:Salaomescija]]
[[fr:Chaucer]]
[[hr:Chad]]
[[id:Cadura]]
[[hu:Chadict¨ao]]
[[nl:Cuauht´emmoil]]
[[no:Cuauhtge>
<contributor>
<username>Rich Farmbrough</username>
<id>82835</id>
</contributor>
<minor />
<comment>Wikify dates</comment>
<text xml:space="preserve">#REDIRECT [[Marine Diving]]
Sample #8
*[[Christianity|Christian]] &amp;ndash; 82%
**[[Baptist]] &amp;ndash; 15%
**[[Protestantism|Protestant]] &amp;ndash; 62%
***[[Methodism|Methodist]] &amp;ndash; 10%
***[[Lutheranism|Lutheran]] &amp;ndash; 6%
***[[Church of Christ]] &amp;ndash; 6%
***[[Reformed Egyptian|Egyptian]] &amp;ndash; 11%
***[[Syriac]] - Member of Augusta (cultural)
**[[Princeps senatus]] (or Archipelago for acids in the use
of neutron species.) Long as a result of TLDs in the DNA of
a large endometer. In this sense the electron shell structure
densitises these &quot;molecular systems&quot;.
The mere density, and ence (implying that, in most sense,
deemed political or social evidence.)
== See also ==
* ’’’Chechen’’’ [des of the Prolog-Answer]]’’ ([[1952]])
(ubc., #99 - PVF version)
*’’Concrete Principles of Closed Life in Philosophy 1900-1960’’
(1992) ISBN 0689720942 ([[NASA]] only to the
’’’International Union’’’). Since his works like
’’[[London Mathers]]’’ and ’’[[Last Labor Day]]’’.
The members of the royal family were in the middle ague Linux,
the standard execution signal was the compiler whose logo
contained symbols for the [[Microsoft Windows
release|Windows Network]] (Windows) and
[[Access Internet Computers|ICA]] listing.
Sample #9
[[[[Canada]], [[England], [France, Missouri]],
revised as a symbol of past humans more controversial after
having criticised the trend ended in breeding and protecting
various categories. ’’Antiqui’’ marked the beginning of the
epistle to Marian: &quot;tell me that it may, to make
pre-remaining neighbours?&quot; The need to reconcile you to
the details of belief in datura, and talk about it in the
appendix to his knowledge and not in any other manner.&quot;
Latter-day Saints with an essential equation have the corresponding
reference for the time, if the acceleration oured a centuries-CLs
(from [[Hitachi, Ltd.|Little Hitchcock]])]]
Sample #10
The meaning of life is subject to rights in [[economics]]
and to generalize the inquiry on liberal liberals in real-life
[[revolution]].
Leibniz believed in warming imestamp.

Reinforcement Learning-Based Inter- and Intra-Application Thermal
Optimization for Lifetime Improvement of Multicore Systems
Anup Das
National University of Singapore
Singapore - 117583
akdas@nus.edu.sg
Rishad A. Shaﬁk
University of Southampton
Southampton SO17 1BJ, UK
rishad.shaﬁk@ecs.soton.ac.uk
Geoﬀ V. Merrett
University of Southampton
Southampton SO17 1BJ, UK
gvm@ecs.soton.ac.uk
Bashir M. Al-Hashimi
University of Southampton
Southampton SO17 1BJ, UK
bmah@ecs.soton.ac.uk
Akash Kumar
National University of Singapore
Singapore - 117583
akash@nus.edu.sg
Bharadwaj Veeravalli
National University of Singapore
Singapore - 117583
elebv@nus.edu.sg
ABSTRACT
The thermal proﬁle of multicore systems vary both within
an application’s execution (intra) and also when the sys-
tem switches from one application to another (inter). In
this paper, we propose an adaptive thermal management
approach to improve the lifetime reliability of multicore sys-
tems by considering both inter- and intra-application ther-
mal variations. Fundamental to this approach is a reinforce-
ment learning algorithm, which learns the relationship be-
tween the mapping of threads to cores, the frequency of
a core and its temperature (sampled from on-board ther-
mal sensors). Action is provided by overriding the oper-
ating system’s mapping decisions using aﬃnity masks and
dynamically changing CPU frequency using in-kernel gover-
nors. Lifetime improvement is achieved by controlling not
only the peak and average temperatures but also thermal
cycling, which is an emerging wear-out concern in modern
systems. The proposed approach is validated experimen-
tally using an Intel quad-core platform executing a diverse
set of multimedia benchmarks. Results demonstrate that
theproposedapproachminimizesaveragetemperature, peak
temperature and thermal cycling, improving the mean-time-
to-failure (MTTF) by an average of 2x for intra-application
and 3x for inter-application scenarios when compared to ex-
isting thermal management techniques. Furthermore, the
dynamic and static energy consumption are also reduced by
an average 10% and 11% respectively.
1. INTRODUCTION
A major challenge of modern multicore systems is decreasing
lifetime reliability, threatened by high power densities and
hence elevated operating temperatures. This leads to an ac-
celeration of device wear-out. Thermal management has at-
tracted signiﬁcant attention both in industry and academia.
Examples include dynamic thermal management using volt-
age and frequency scaling [7], slack time management [10],
peak temperature management through system-level task
scheduling [3] and thermal stress management through ap-
plication task mapping [2] (refer to Section 2 for a summary
of related works). These approaches, however, suﬀer from
the following limitations.
First, modern multicore systems switch between appli-
cations exhibiting wide performance and workload varia-
tions, and therefore the thermal behavior of these systems
vary both within (intra) and across (inter) applications. Al-
though intra-application thermal variations are considered
in existing studies, inter-application variations are not ad-
dressed. Second, the existing studies focus on average tem-
perature reduction; thermal cycling is not accounted. Last,
the existing adaptive techniques are either implemented on
a simulator or rely on time-consuming thermal prediction
from theHotSpot tool [14], limiting their scalability.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not
made or distributed for proﬁt or commercial advantage and that copies bear
this notice and the full citation on the ﬁrst page. Copyrights for components
of this work owned by others than ACM must be honored. Abstracting with
credit is permitted. To copy otherwise, or republish, to post on servers or to
redistribute to lists, requires prior speciﬁc permission and/or a fee. Request
permissions from Permissions@acm.org.
DAC ’14, June 01 - 05 2014, San Francisco, CA, USA
Copyright 2014 ACM 978-1-4503-2730-5/14/06 ...$15.00
http://dx.doi.org/10.1145/2593069.2593199.
In this paper, we address the above gaps and present a
dynamic thermal management approach for multicore sys-
tems that adapts to thermal variations within (intra) and
across (inter) applications. Fundamental to this approach
is a run-time system, which interfaces with the on-board
thermal sensors and uses reinforcement learning to learn
the relationship between the mapping of threads to cores,
the frequency of a core and its temperature. The aim is
to control the average temperature and the thermal cycling
to achieve an extended mean time to failure (MTTF). This
paper makes the following contributions:
1. inter-and intra-application thermal management using
thread-to-core allocation (through CPU aﬃnity) and
dynamic frequency control (through CPU governors)
1;
2. separation of the temperature sampling interval from
the decision interval of the conventional reinforcement
learningalgorithmtoaccuratelymodel(andhencecon-
trol) the average temperature and thermal cycling; and
3. implementation of the run-time system incorporating
the machine learning algorithm on a real platform.
The proposed approach is implemented on an Intel quad-
core platform running Linux kernel 3.8.0. A set of multime-
dia applications from theALPBench suite [11] are executed
on the platform. Results demonstrate that the proposed ap-
proach minimizes average temperature and thermal cycling,
leading to a signiﬁcant improvement in MTTF as demon-
s t r a t e di nS e c t i o n6 .
The remainder of this paper is organized as follows. A
brief introduction to related works is presented in Section 2
and the motivation in Section 3. This is followed by the pre-
liminaries on reliability in Section 4 and an overview of the
reinforcement learning-based approach in Section 5. Evalua-
tion of the proposed technique is presented next in Section 6
a n dt h ep a p e ri sc o n c l u d e di nS e c t i o n7 .
2. RELATED WORKS
The existing studies on thermal optimization can be classi-
ﬁed into two categories – static and dynamic. Static tech-
niques determine application mapping and scheduling oﬄine
to minimize peak temperature or thermal cycles [17]. Dy-
namic techniques optimize temperature at run-time. Since
this work belongs to the latter category, dynamic thermal
management techniques are discussed in depth. A slack bor-
rowing technique is proposed in [10] to dynamically manage
peak temperature for MPEG-2 decoder. A reinforcement-
learning based adaptive technique is proposed in [3] to op-
timize temperature by controlling task mapping based on
the temperature of the current iteration. A neural network
based adaptive technique is proposed in [9] to reduce peak
temperature. Both these techniques rely on the HotSpot
tool for temperature prediction. The relationship between
temperature and voltage and frequency of operation is for-
mulated in [8,12]. Based on this, an online heuristic is pro-
posed to determine the voltage and frequency of the cores
to minimize the temperature. This technique does not per-
form learning, resulting in re-performing the same optimiza-
tion for the same environment. A reinforcement learning al-
gorithm is proposed in [7] to manage performance-thermal
1CPU aﬃnity enables the binding of a thread of an application
to a physical core or a range of cores.
CPU governors are power schemes for the CPU deciding the fre-
quency of operation of the cores.
Table 1. Summary of related Works
Related Machine Inter- Thermal Temperature Platform
Works Learning Application Cycling Measurements
[6] √ ×× thermal gun FPGA
[3] √ × √ HotSpot multicore
[2] ×× √ HotSpot simulation
[9] √ ×× HotSpot simulation
[8, 12] × √ × thermal model simulation
[7] √ ×× sensors multicore
Proposed √√√ sensors multicore
trade-oﬀs by sampling temperature data from the on-board
thermal sensors. A distributed learning agent is proposed
in [6] to optimize peak temperature with a given power
budget. The technique is implemented on an FPGA with
temperature measurement using an external thermal gun.
Table 1 summarizes the related works. As can be seen,
some of the existing studies rely on the HotSpot tool for
temperature prediction. Although theHotSpot tool can be
used to validate static thermal management policies, the
high simulation time of the tool can potentially cause dead-
line misses for dynamic thermal management of real-time
systems. In some of the existing works, temperature is es-
timated by solving a RC equivalent thermal model. These
models are usually complex and diﬃcult to solve using direct
mathematical techniques such as LU decomposition. Ther-
mal guns have also been used to measure the temperature;
however, thissuﬀersfromlimitedaccuracy, especiallytocap-
ture thermal cycling. Furthermore, deploying thermal guns
for real system is not practical. Finally, using performance
counter for estimating temperature can lead to inaccuracy
in the thermal values. These motivate the use of thermal
sensors, even at the expense of limited speed of operation,
and as such [7] is used to compare the proposed approach.
3. MOTIVATIONAL EXAMPLE
Thermal management using voltage and frequency control is
already demonstrated in prior works (e.g. [7]). To establish
the importance of thread allocation on the thermal behavior
ofapplications, anexperimentisconductedonanIntelquad-
core platform by executing two multi-threaded (6 threads)
applications (face recognitionand mpeg2 encoding)b a c k - t o -
back. The thermal proﬁles obtained using Linux’s default
thread-to-core allocation and scheduling is shown in red in
Figure 1. From the thermal proﬁles it can be seen thatface
recognition is characterized by a higher average temperature
with lower thermal cycling leading to peak temperature re-
lated reliability issues such as electro-migration (EM) and
negative bias temperature instability (NBTI). Application
mpeg2 encoding on the other hand exhibits lower average
temperature with higher thermal cycling leading to thermal
fatigue and its associated reliability problems.
This diﬀerence in thermal behavior of the two diﬀerent
applications can be explained as follows. The thread work-
loads of theface recognitionapplication are characterized by
longer duration of thread-independent high activity cycles
followed by shorter duration of inter-thread dependent low
activity cycles. When these threads are allocated to cores,
the longer independent high-activity cycles of a thread over-
lap partially with the shorter dependent low-activity cycles
of other threads. This is due to the Linux’s default thread
allocation, where threads are often migrated to balance load
on the architecture. This leads to a higher temperature with
lower thermal cycling. For thempeg2 encodingapplication,
the thread-independent high-activity cycles are shorter in
duration, while the inter-thread dependent cycles are rel-
atively long compared to the face recognition application.
When these threads are allocated by Linux, only few of the
available cores are being used. The default allocation re-
sults in a combination of independent high-activity cycles
( o fm o r et h a no n et h r e a d s ) ,w h i c ho v e r l a pw i t he a c ho t h e r
(leading to a higher temperature) and similarly, the longer
low-activity inter-thread dependent cycles overlap (leading
to a lower temperature). This results in alternating high
and low temperature triggering high thermal cycling.
Next, thesameexperimentisrepeatedbyarbitrarilyﬁxing
the assignment of threads to cores (two cores execute two
threads each and the other two cores execute one thread
each) and leaving only the thread scheduling decision to
the operating system. This is performed by changing all
thread’s aﬃnity masks, forcing the Linux kernel to migrate
these threads to the cores speciﬁed. The new thermal pro-
ﬁles are shown in blue in the same ﬁgure. As can be seen
Face recognition
Thermal cycles: 
High
Average Temperature: 
High
Reliability Problem: 
All
Mpeg encoding
Thermal cycles: 
Low
Average Temperature: 
Low
Reliability Problem: 
None
Face recognition
Thermal cycles: 
Less
Average Temperature: 
High
Reliability Problem: 
NBTI, EM
Mpeg encoding
Thermal cycles: 
High
Average Temperature: 
Low
Reliability Problem: 
Thermal Stress 
TDDB
Figure 1. Thread-to-core aﬃnity inﬂuences thermal proﬁle
from the plot, this arbitrary thread assignment results in
higher average temperature with higher thermal cycling for
the face recognition application triggering all temperature
related reliability concerns. This is because when threads
are ﬁxed to cores, the longer high-activity cycles of diﬀer-
ent threads overlap and so do the shorter low-activity cycles
resulting in higher temperature and higher thermal cycling.
When the same control is applied formpeg2 encoding,t h e
shorter high-activity cycles are not combined but are over-
lapped with each other. The temperature however increases,
but for a shorter duration. This results in reduction of both
the average temperature and thermal cycling thereby im-
proving the lifetime. This example demonstrates two key
aspects – thermal proﬁle varies with application; and thread
allocation inﬂuences thermal proﬁle. This motivates the im-
portance of an adaptive algorithm to learn the thermal be-
havior of an application and control it using appropriate
thread-to-core assignment.
4. RELIABILITY COMPUTATION
4.1 Temperature Related MTTF
T h el i f e t i m er e l i a b i l i t yo fac o r ei sg i v e nb yR(t)= e−(t·A)β
,
where A is theThermal Aging of the core (refer to [4,15]).
A =
∑
i
Δti
tp × α(Ti)
(1)
where tp is the execution time of the application,α(Ti)i st h e
fault density (typically Weibull or Lognormal distribution)
and Ti is the average temperature in the interval Δti. This
equation allows to model any wear-out eﬀect such as electro-
migration and negative bias temperature instability consid-
ered individually or as sum-of-failure-rate (SOFR). Mathe-
matically, the MTTF of the core is
MTTF =
∫ ∞
0
R(t)dt =
∫ ∞
0
e−(t·A)β
dt (2)
Thus, maximizing the MTTF is equivalent to minimizing
the aging of the cores.
4.2 Thermal Cycling Related MTTF
Thermal cycling related MTTF is computed in three steps.
1. Calculating the thermal cycles from a thermal proﬁle
using Downing simple rainbow counting algorithm [5].
2. Calculating, from each thermal cycle, the number of
cycles to failure using Coﬃn-Mansion’s rule.
NTC (i)= ATC
(
δTi − TTh
)−b e
Ea
KTmax(i) (3)
where NTC (i) is the number of cycles to failure due
to ith thermal cycle,ATC is an empirically determined
constant, δTi is the amplitude of theith thermal cycle,
TTh is the temperature at which elastic deformation
begins, b is the Coﬃn-Manson exponent constant,Ea
is the activation energy andTmax(i) is the maximum
temperature in theith thermal cycle.
3. Calculating the MTTF using Miner’s rule.
MTTF =
NTC
∑ m
i=1 ti
m
(4)
where ti is the time for theith thermal cycle,m is the
number of thermal cycles obtained in step 1 andNTC
is the eﬀective cycles to failure determined using
NTC =
m
∑ m
i=1
1
NTC (i)
(5)
ͬΎĨĨŝŶŝƚǇŵĂƐŬ Θ'ŽǀĞƌŶŽƌΎͬ
ƵŶƐŝŐŶĞĚůŽŶŐŵĂƐŬсĐƉƵͺĂĨĨŝŶŝƚǇ͖
ƉƚŚƌĞĂĚͺƐĞƚĂĨĨŝŶŝƚǇͺŶƉ;ƉƚŚƌĞĂĚͺƐĞůĨ;Ϳ͕
ƐŝǌĞŽĨ;ŵĂƐŬͿ͕ΘŵĂƐŬͿ͖
ĐƉƵĨƌĞƋͲƐĞƚʹĚϮϰϬϬϬϬʹĐϬʹŐƵƐĞƌƐƉĂĐĞ
ĐŽƌĞϬ
ƐĞŶƐŽƌ
ĐŽƌĞϭ
ƐĞŶƐŽƌ
ĐŽƌĞŶ
ƐĞŶƐŽƌ
/ŶƚĞƌĐŽŶŶĞĐƚ
KƉĞƌĂƚŝŶŐ
^ǇƐƚĞŵ
&RQWUROV &38DIILQLW\
&38JRYHUQRUV
0RQLWRUV7HPSHUDWXUH6DPSOHV
&RUHDFWLYLWLHVHJFDFKHPLVV
ŵƉĞŐϮͺĚĞĐ $SSOLFDWLRQ
6\VWHP6RIWZDUH
+DUGZDUH
WƌŽƉŽƐĞĚƉƉƌŽĂĐŚ
ĞƚĞƌŵŝŶĞ
^ƚĂƚĞ
YͲ>ĞĂƌŶŝŶŐ
ůŐŽƌŝƚŚŵ
ŽŵƉƵƚĞ
ZĞǁĂƌĚ
^ĞůĞĐƚ
ĐƚŝŽŶ
ĞĐŝƐŝŽŶƉŽĐŚƐ
ĨĂĐĞͺƌĞĐ
3HUIRUPDQFHFRQVWUDLQWHJIUDPHVSHUVHFRQG
Figure 2. Proposed system level approach
Combining Equations 3-5,MTTF = ATC
∑ m
i=1 ti
Thermal Stress , where
Thermal Stress is an indication of the stress experi-
enced by a core due to the thermal cycling. This is
obtained using the following equation.
Thermal Stress =
m∑
i=1
(δTi − TTh )b × e
−Ea
KTmax(i) (6)
Thus, maximizing the MTTF of a core due to thermal cy-
cling is equivalent to minimizing itsstress.
5. PROPOSED APPROACH
Figure 2 illustrates the proposed dynamic thermal manage-
ment approach as part of the system software layer and its
interaction with the operating system, the application and
the hardware layer. We adopted Q-Learning [18] as the re-
inforcement learning algorithm, where the learning agent
maintains a Q-Table with entries corresponding to every
state-action pair. The approach interfaces with the appli-
cation layer to determine the performance requirement. For
video applications such asmpeg2 encoding/decoding, the per-
formance is measured asframes per secondwhile for other
applications such asimage processing, performance refers to
the execution time. A point to note here is that, although
the above metrics are used for performance, the proposed al-
gorithm can be trivially used with any reward function such
as the performance counter based metric of [7].
The operating system samples the on board thermal sen-
sors and reads the performance counters at the sampling in-
terval to compute the thermal stress, aging and to monitor
performance. The stress and aging values form the states
of theQ-Table. The learning agent selects action at a ﬁxed
interval, called the“decision epoch”. Each action comprises
of thread aﬃnity and voltage and frequency of operation
and is enforced by the operating system on the underlying
cores of the hardware layer. In most existing works on Q-
learning based thermal management, the decision epoch is
used to sample the temperature; actions are selected based
on the instantaneous temperature from the sensor, which is
not a true indication of the average temperature or thermal
cycling in the interval. Since temperature-related reliabil-
ity is governed by average temperature and thermal cycling,
w h i c hn e e dt ob em e a s u r e do v e rap e r i o do ft i m e ,w eu s e da
sampling interval diﬀerent than than of the decision epoch.
At the start of every decision epoch, the learning agent
ﬁrst computes the reward (or penalty) of the previous action
based on three parameters – previous state, previous action
and current state. TheQ-Table entry corresponding to the
previous state and previous action is updated according to
the following equation [7,18].
Q(Ei,ℵi)+= α
⎡
⎣R(Ei,Ei+1)+ γ max
ℵj
(
Q(Ei+1,ℵj) − Q(Ei,ℵi)
)
⎤
⎦ (7)
where α is the learning rate and is an indication of adapta-
tion rate of theQ values andγ ([0,1]) is the discount rate.
Subsequently, the learning agent selects the action with
the highestQ-value for the current state. This action is ﬁrst
decoded by the operating system to determine the aﬃnity
masks for the threads and voltage and frequency of the cores.
The threads are then migrated to the cores speciﬁed by the
corresponding mask and the CPU governor settings are up-
dated to enforce the voltage-frequency pair. Initially, the
actions selected by the agent may not be optimal; the deci-
sions are, however, reﬁned over time. Algorithm 1 provides
ALGORITHM 1: Reinforcement Learning Algorithm
Input: Temperature reading from sensorsT.
Output: Thread aﬃnity and CPU governorsℵi.
Initialize the Q-Table, E0 =0 ,ℵ0 =0 ,i =1 ,α =1 ;
TRec.push(T); //Record temperature;
if |TRec| == Decision Epoch then
Calculate MAs(i)a n dMAa(i);
ΔMAs = |MAs(i) − MAs(i − 1)| and ΔMAa = |MAa(i) − MAa(i − 1)|;
if ((ΔMAL
s ≤ ΔMAs < ΔMAU
s ) || (ΔMAL
a ≤ ΔMAa < ΔMAU
a )) then
Q ← Qexp and α ← αexp; //Intra workload variation;
end
if ((ΔMAs ≥ ΔMAU
s ) || (ΔMAa ≥ ΔMAU
a )) then
Q ← Q0 and α ← 1; //Inter workload variation;
end
Ei = IdentifyState (TRec);
R = CalculateReward(Ei−1,Ei) //Equation 8;
Q(Ei−1,ℵi−1)= UpdateQtable(Q, R, α) //Equation 7;
ℵi = SelectAction(Q,Ei);
α = UpdateLearningRate(α,Ei);
TRec = ∅, i = i +1 ;
end
sleep(temperature sampling interval);
the pseudo-code of the learning agent. Details of this algo-
rithm are provided in the following subsections.
5.1 Selecting the State and Action Space
The state space of ourQ-Learning algorithm is composed
of stress (determined using Equation 6) andaging (deter-
mined using Equation 1). To limit state space explosion, the
working range of these parameters are divided intoNa and
Ns disjoint intervals respectively. Speciﬁcally,stress is the
set S = {(0,s0],(s0,s1],··· , (sNs−1,sNs]} and the symbol
ˆsi is used to represent the interval (si,si+1]. Similarly, ag-
ing is the setA = {(0,a0],(a0,a1],··· ,(aNa−1 ,aNa]} and
the symbol ˆai is used to represent the interval (ai,ai+1].
The environment is represented asE :( A×S ). The action
space of the agent is composed of thread aﬃnity-based as-
signments and ﬁve CPU governors (ondemand, conservative,
performance, powersaveand userspace) .T h en u m b e ro fd i f -
ferent aﬃnity masks grows exponentially with the number
of threads and cores. To restrict the action space, only a few
of the alternatives are explored. Similarly, three frequency
levels are selected for theuserspace CPU governor. The ac-
tion space is denoted byℵ :( M×G ) whereM is the set of
thread aﬃnity mappings andG is the set of governors.
5.2 Computing the Reward
The reward functionR(Ei,Ei+1)i sg i v e nb y
R(Ei,Ei+1)=
{−ˆsi × ˆai if (ˆsi = ˆsNs)o r( ˆai = ˆaNa)
f(ˆai, ˆsi)+( Pc − P)o t h e r w i s e (8)
where P is the performance, Pc is the performance con-
straint and the functionf is determined empirically asf =
(a.K1.stress+b.K2.aging), whereaandbarerelativ eimpor-
tance ofstress and aging: For mpeg (large thermal cycles),
a>b and for tachyon (high average temperature),b>a .
Two sets ofa and b values are used based on the mean of
stress and aging. K1(K2) is the learning weight and is a
Gaussian function of thestress (aging) values. This distri-
bution assigns lower rewards to thermally unstable as well
as the thermal stable states and thus allows the algorithm
to explore other states and prevent Q-Table clustering.
For the design of the reward function, two cases are con-
sidered. If the stress or aging falls in the unsafe zone (the
last interval), the decision is penalized. This is indicated
withanegativevalueoftherewardfunction, whichdecreases
the Q value (refer to Equation 7) so that the corresponding
action is avoided in the future. For all other cases, the re-
ward function is composed of performance penalty and the
thermal safety of the state. Speciﬁcally, if the performance
requirement is not satisﬁed, (P
c − P) is negative and the
reward (or penalty) is governed by the functionf. Finally,
rewards are guaranteed if an action leads to a thermal safe
state while satisfying the performance requirements.
5.3 Learning Phases
The Q-learning algorithm is composed of three phases –ex-
ploration, exploration-exploitation and exploitation.I n t h e
exploration phase, the agent selects action arbitrarily to de-
termine the corresponding reward. This phase is character-
ized by α values close to 1 to enable a signiﬁcant fraction
of the reward values to contribute towards theQ-Table en-
tries. In theexploration-explotation phase, the agent selects
the best action (with the highestQ value) for the current
state and updates theQ-Table entry with a part of the re-
ward value. Finally, in the exploitation phase, the algo-
rithm still selects the action corresponding to the highest
Table 2. MTTF (in years) of reinforcement learning algorithm for three applications. The scaling parameters for computing MTTF are so
selected such that the MTTF of an unstressed core (i.e. an idle core) is 10 years.
Benchmarks Average Temperature (oC) Peak Temperature (oC) Thermal cycling MTTF Average temperature MTTF
Application Data Linux Ge et. al [7] ProposedLinux Ge et. al [7] ProposedLinux Ge et. al [7]Proposed Linux Ge et. al [7]Proposed
tachyon
set 1 69.2 52.6 50.6 71.5 63.0 60.0 7.1 2.3 5.5 0.7 3.0 3.6
set 2 50.5 44.5 43.8 57.3 56.3 52.0 2.8 4.3 5.3 2.6 4.5 4.8
set 3 50.8 44.7 41.6 57.8 54.5 48.8 1.3 3.8 6.5 2.4 4.1 5.5
mpeg dec
clip 1 36.0 34.0 34.2 42.7 41.3 39.0 2.1 0.8 6.4 3.7 4.5 4.4
clip 2 35.6 34.4 34.2 42.3 42.0 39.3 1.1 0.9 4.7 3.8 4.3 4.4
clip 3 34.3 34.4 34.0 43.0 39.7 44.3 1.6 3.4 3.7 4.3 4.2 4.5
mpeg enc
seq 1 33.7 34.1 32.6 41.0 40.7 40.3 4.3 4.4 5.2 4.6 4.5 5.2
seq 2 34.4 33.5 32.3 41.3 39.7 41.7 3.9 6.2 4.8 4.3 4.7 5.4
seq 3 33.2 33.7 31.8 40.3 40.0 41.0 4.6 5.1 5.1 4.9 4.6 5.7
Q-value for the current state; however, theQ-table entries
are not updated (or updated with negligible fraction of the
reward value). This phase is characterized with α values
close to 0. To facilitate transition between the three phases
of the algorithm, an exponentially decreasing function is se-
lected for theα value. This function is implemented in the
UpdateLearningRate subroutine.
5.4 Adaptation to Workload Variation
To incorporate intra- and inter-application workload varia-
tions, moving averages of the stress and the aging are de-
termined at the start of every decision epoch. The change
in the moving averages are identiﬁed in the algorithm as
ΔMA
s and ΔMAa. Two thresholds are maintained for each
of these quantities identiﬁed with the superscriptL and U,
respectively. The learning agent considers a change in the
moving average as intra-application variation if the change
is greater than the lower threshold and lower than the upper
threshold (for example when ΔMA
L
s ≤ ΔMAs < ΔMAU
s ).
This causes the Q-table to be updated with the Q values
from the end of the exploration phase. The alpha value is
also updated accordingly. On the other hand, the agent con-
siders a change in the moving average as inter-application
variation, if the change is greater than the upper threshold.
In this case, theQ-Table entries are initialized to 0 andα
value initialized to 1 to start learning again. This is the
implicit learning technique adopted in this work. Theα
exp,
lower and upper threshold are determined empirically from
the set of applications at hand. A point to note here is that,
the agent maintains twoQ-Tables – one with static Q values
from the end of the exploration phase and the other with Q
values that are updated at each decision epoch.
Algorithm1providesthepseudo-codeofthereinforcement
learning algorithm. Important parameters of the algorithm
are the temperature sampling interval, decision epoch and
the number of states and actions. We adopt a systematic
approach to decide these parameters.
6. RESULTS AND DISCUSSIONS
The proposed run-time approach is validated experimentally
on an Intel quad-core CPU running Linux kernel 3.8.0. Per-
formance is monitored usingperf [1]; temperature is mea-
suredbysamplingthethermalsensorsdirectly; power/energy
consumption is recorded using likwid powermeter [16]. A
set of multi-threaded multimedia applications are considered
from the ALPBench [11] benchmark suite. These bench-
marks arem p e ge n c ,m p e gd e c ,f a c er e c o g n i t i o n ,s p h i n xand
tachyon and are representative of the multimedia workloads
for most multicore systems. The number of threads in each
of these applications is conﬁgurable and, in this paper, six
threads are considered to exploit the full beneﬁt of the four
cores. The device parameters used for computing theaging
and stress of a core are the same as that used in [2,17].
6.1 Intra-Application
Table 2 reports the average temperature, peak temperature
and MTTF due to average temperature (equivalentlyaging)
and thermal cycling (equivalently stress) of the proposed
technique in comparison with Linux’s ondemand [13] and
the technique proposed in [7]. Results are reported for three
diﬀerent applications, each of which are executed for three
sets of input data. There are a few trends to follow from
this table. First, the technique in [7] minimizes instanta-
neous temperature achieving a loweraging (higher MTTF)
than Linux (refer to columns 3-4 & columns 12-13). This
signiﬁes the importance of the thermal management feature
for operating systems. However, thermal cycling is not ac-
counted for in this technique and therefore does not guar-
antee reduction ofstress. This is evident from the thermal
cycling-related MTTF values for scenarios such astachyon
0
1
2
3
4
5
6
7
8Normalized thermal cycling related MTTF
 
Linux Ondemand    
Modified Ge et. al [ ]
Proposed
mpegdec-
tachyon
tachyon-
mpegdec
mpegenc-
tachyon
mpegdec-
tachyon-
mpegenc
tachyon-
mpegenc-
mpegdec
mpegenc-
mpegdec
Figure 3. Inter-application results
on set 1 andmpeg dec on clip 1, where the MTTF values
obtained using [7] is lower than that of Linux.
Second, the proposed reinforcement learning algorithm
minimizes the average temperature by upto 18.6oC and the
peak temperature by upto 11.5oC. This reduction leads to
an improvement in MTTF due toaging by upto 5x (average
82%). For applications such asmpeg dec, the improvement is
less as the average temperature is usually lower, leading to a
limited scope to further improveaging. The thermal cycling
eﬀect dominates in this application. For other applications
such as tachyon, the improvement is signiﬁcant due to the
large scope for improving bothaging and stress. Third, the
proposed adaptive algorithm also minimizes thermal cycling
which is not considered in Linux and [7]. This is highlighted
in columns 9-11. An important point to note is that for the
tachyon application with set 1 data, the MTTF due tostress
using Linux’s default thread assignment is higher (7.1 years);
however, the MTTF due toaging is lower (0.7 years). The
proposed technique balances the two eﬀects and improves
the MTTF due toaging by 5x with less than 25% sacriﬁce
in MTTF due tostress (while still maintaining a satisfactory
MTTF of 5.5 years). For all other applications and data sets,
the proposed reinforcement learning algorithm outperforms
Linux in terms of thermal cycling by an average 2.3x.
Last, the proposed approach outperforms [7] both in terms
ofaging (an average 13% higher average temperature related
MTTF) andstress (an average 2x higher thermal cycling re-
lated MTTF). While the improvement of thermal cycling is
expected (as this is incorporated explicitly in the proposed
approach), the improvement inaging is due to the follow-
ing. First, the decoupling of the temperature sampling inter-
val from the decision epoch (enabling a ﬁner control on the
average temperature); second, careful choice of the design
parameters as demonstrated in Figure 6 & 7.
6.2 Inter-Application
Figure 3 plots the normalized MTTF due to thermal cy-
cling obtained using the proposed technique in comparison
with that obtained using the modiﬁed technique of [7] for
six diﬀerent inter-application scenarios. The MTTF values
are normalized with respect to the MTTF obtained using
Linux’s ondemand governor. Furthermore, the technique
of [7] is modiﬁed to consider application switching using
explicit indication from the application layer. The pro-
posed approach however, detects application switching au-
tonomously (without communication from the application
layer) and performs re-learning (as discussed in Section 5).
There are six inter-application scenarios considered in this
experiment. A scenario appA-appB indicate that appA is
Figure 4. Exploration phase of the learning algorithm
Figure 5. Exploitation phase of the learning algorithm
executed ﬁrst followed by applicationappB. As can be seen
from the ﬁgure, the technique of Ge et. al [7] results in
higher MTTF than Linux. For some inter-application sce-
narios such as mpegdec-tachyon and tachyon-mpegdec,t h e
improvements are less (≈ 8%). For other inter-application
scenarios, this improvement is higher. On average, for all
the scenarios, [7] increases MTTF by 80% as compared to
Linux. The approach proposed in this paper outperforms
both Linux and that of [7] in terms of thermal cycling,
achieving 5x improvement with respect to Linux and 3x
improvement with respect to [7]. A point to note from
the ﬁgure is that, the MTTF improvement (over [7]) using
the proposed approach, increases with frequent application
switching. This is evident from the higher MTTF improve-
ment of 3.5x obtained for the three-application scenarios
(mpegdec-tachyon-mpegenc and tachyon-mpegenc-mpegdec)
as compared to improvements obtained for the four other
two-application scenarios. Thus, the proposed approach im-
proves thermal cycling related MTTF signiﬁcantly for inter-
application scenarios. The improvement increases with an
increase in application switching (typical of a modern mul-
ticore system).
6.3 Phases of the Reinforcement Learn-
ing Algorithm
To further demonstrate the temperature proﬁle obtained us-
ing the proposed algorithm, Figures 4 and 5 plot the explo-
ration and the exploitation phases, respectively in compari-
s o nw i t hL i n u x ’ spo p u l a ra n dd e f a u l tondemand governor for
the face recognitionapplication. As can be seen, at the be-
ginning of the exploration phase, the temperature obtained
using the proposed algorithm is comparable to that obtained
using Linux. This is because, at this phase, the proposed al-
gorithm explores the impact of CPU aﬃnity and operating
frequency choices on the temperature of a core. However,
when the proposed model learns the impact of these param-
eters on temperature (i.e. in the exploitation phase), the
CPU aﬃnities and operating frequencies are selected such
that the average temperature is reduced (Figure 5).
6.4 Selection of Design Parameters
Figure 6 plots the impact of varying the interval of sampling
temperature from the on-board thermal sensors onthermal
stress and performance for the tachyon application. The
ﬁgure also plots the auto-correlation of temperature sam-
ples which is a measure of how much the temperature val-
ues change across consecutive samples. Speciﬁcally, a high
auto-correlation indicates a small diﬀerence between con-
secutive thermal data. As can be seen from the ﬁgure, the
1 2 3 4 5 6 7 8 9 100
2
4
6
8
Temperature sampling interval
  
 
 
 
 
 MTTF (Years)
1 2 3 4 5 6 7 8 9 100
0.2
0.4
0.6
0.8
1
Temperature sampling interval
Auto-correlation
1 2 3 4 5 6 7 8 9 100
1
2
3
4 x 10
7
Temperature sampling interval
Cache-misses
1 2 3 4 5 6 7 8 9 100
2
4
6
8 x 10
5
Temperature sampling interval
Page-faults
Figure 6. Impact of temperature sampling interval (sec)
0 20 40 60 801
1.05
1.1
1.15
1.2
Decision epoch (s)
Normalized execution time
(a)
 
 
0 20 40 60 800.8
0.9
1
1.1
1.2
1.3
1.4
1.5
Decision epoch (s)
Normalized energy
(b)
 
 
0 20 40 60 800
2
4
6
8
10
12
14
Decision epoch (s)
Normalized learning time
(c)
 
 
tachyon mpeg dec mpeg enc
Figure 7. Eﬀect of the length of decision epoch
auto-correlation is high for lower sampling intervals. This is
expected as temperature variation is usually slower, being
dependent on the thermal property of silicon, ambient tem-
perature and cooling technology used. The MTTF value in-
creases with an increase in size of the sampling interval. This
i s ,h o w e v e r ,a no v e re s t i m a t i o no ft h ea c t u a lM T T Fv a l u e
(corresponding to the sampling interval of 1 sec). This over-
estimation is due to the loss of temperature accuracy with
increasing sampling interval resulting in a lowerstress and
hence higher MTTF. Finally, the number of cache-misses
and page faults decrease with an increase in the sampling
interval, clearly signifying the performance improvement.
Based on these trade-oﬀs, a sampling interval of 3 sec is se-
lected for thetachyon application. Similarly, the sampling
interval for other applications are determined. An interval
of 3 sec provides the best trade-oﬀ for most of the applica-
tions. In future, determination of the sampling interval can
be incorporated as part of the learning algorithm itself.
Figure 7 plots the performance of the algorithm with in-
creasing decision epochs for the three applications. The ex-
ecution time, dynamic energy consumption and the adapta-
tion time are compared. The execution time refers to the
completion time with a ﬁxed length of input data. For the
mpeg enc (or mpeg dec) application, this is the time to en-
code(decode) 10MB of video. For thetachyon application,
the time is measured as the rendering time of 300 images.
The execution time using the reinforcement learning algo-
rithm is normalized with respect to the execution time of
Linux (with no adaptation) for the same input data. As can
be seen from Figure 7(a), for all applications, the execution
time overhead is higher for smaller decision epochs due to
the overhead associated with frequent decision changes. The
execution time overhead reduces with larger decision epochs.
Figure 7(b) reports the dynamic energy consumption of the
proposed approach for the same input data normalized with
respect to the Linux (without adaptation). As expected,
the energy consumption is also higher for smaller decision
epochs due to the frequent adaptation of the approach. Fi-
nally, Figure 7(c) plots the training time i.e. the time re-
quired for the algorithm to learn the thermal behavior of an
application. The results are normalized with respect to the
training time of the algorithm with a decision epoch of 5sec.
As can be seen, the training time increases with an increase
in the decision epoch. This is because the training time is
a function of decision epoch and number of iterations. The
decision epoch for our algorithm is selected based on this
4
8
12
4
8
12
20
40
60
80
100
120
Number of actionsNumber of states
Number of iterations (3.39,6.3)
(2.42,6.8)
(2.53,6.8)
(2.53,6.8)
(2.27,6.5) (0.95,6.9)
(1.00,6.3)
(2.60,6.4)
(4.10, 6.0)
Figure 8. Convergence of the reinforcement learning algorithm
Table 3. Execution time (in sec) of the proposed approach.
App. Linux Ge et. alProposedondemand powersave 2.4GHz 3.4GHz [7]
tachyon 629 1337 913 627 1137 810
mpeg dec 1208 1222 1183 1127 1328 1204
mpeg enc 1623 1655 1628 1571 1676 1599
energy, execution and training time trade-oﬀ.
Figure 8 plots the convergence time of the proposed al-
gorithm for the mpeg decoding application with a varying
number of states and actions. The convergence time is mea-
sured by the number of decision epochs needed to train the
proposed learning algorithm. As can be seen from the ﬁgure,
the number iterations i.e. the training time increases with
an increase in the number of actions and/or states. This is
expected as an increase in the states and/or actions leads
to an increase in the size of theQ-Table and therefore more
iterations are needed to learn (ﬁll the table entries). The
ﬁgure also reports the MTTF as coordinates (stress, aging)
for each design point. As the size of theQ-Table increases,
the reinforcement learning algorithm has ﬁner control on
the temperature resulting in an improvement in the MTTF.
The number of states and actions are chosen based on this
learning time and solution quality trade-oﬀ.
6.5 Performance and Energy Trade-oﬀs
Table 3 reports the execution time of the proposed approach
in comparison with the one proposed in [7] and the Linux-
based approach for ondemand, powersave and user-space
powergovernors. Twouserfrequencies(2.4GHzand3.4GHz)
are shown in the table. The execution time with the highest
frequency of 3.4GHz is the least for all the applications. This
is expected as the execution time decreases with an increase
in frequency. For the same reason, the execution time with
the lowest frequency (powersave) is the highest. However,
thepoweroverheadisleast(asexplainednext). Forsomeap-
plications such astachyon, the proposed approach has higher
execution time than the Linux’sondemand governor by upto
30%. This is because the workload in thetachyon applica-
tion forces the kernel to execute always at the highest fre-
quency of 3.4GHz in theondemand power mode. Thus, the
execution time for Linux’sondemand and userspace-3.4GHz
are comparable. The proposed approach on the other hand
explores diﬀerent power modes to reduce thermalstress and
aging and therefore trades-oﬀ performance. For other appli-
cations such asmpeg enc and mpeg dec, the execution time
of the proposed approach is lower than that of Linux’sonde-
mand power mode. Finally, with respect to [7], the proposed
approach reduces execution time by an average 14%.
Figure 9 plots the average dynamic power and energy con-
sumption (measured using likwid-powermeter) of the pro-
posed algorithm in comparison with that of [7] and the Linux
governors. Although the power and energy overhead are not
reported in [7], these are calculated here for a comparative
study. As can be seen from the ﬁgure, the proposed ap-
proachreducespowerconsumptionbyanaverage6%incom-
parison with Linuxondemand governor with 10% increase
in execution time. Although the dynamic power consump-
tion of [7] is lower than the proposed approach (on aver-
age 4% lower), the energy consumption (which incorporates
both power and performance) of the proposed approach is
10% lower than [7] (within 3% of the energy consumption of
Linux’s ondemand governor).
An interesting point to note here is that, by reducing the
average temperature the proposed technique improves the
tachyon mpeg dec mpeg enc0
5
10
15
20
25
30
35Average dynamic power (Watts)
 tachyon mpeg dec mpeg enc0
0.5
1
1.5
2
2.5
3 x 10
4
Average dynamic energy (Joules)
 
Linux ondemand
Linux powersave
Linux 2.4GHz
Linux 3.4GHz
Ge et. al [  ]
Proposed
Figure 9. Power comparison of the reinforcement learning algorithm
leakage power. Although leakage power is not measured on
the board, the established models for the same (such as the
one proposed in [17]) indicate that the proposed algorithm
improves leakage energy by an average 15% as compared to
Linux’s ondemand governor and 11% as compared to [7].
7. CONCLUSIONS
We have proposed a reinforcement learning-based run-time
approach for multicore system to adapt to thermal varia-
tions both within an application as well as when the system
switches from one application to another. The control is
provided by overriding the operating system mapping de-
cisions using aﬃnity masks and dynamically changing the
frequency of cores using CPU governors. The approach is
validated experimentally using an Intel quad-core platform
running Linux kernel 3.8.0. Results demonstrate that the
proposed approach is able to improve MTTF by an average
2x for intra-application and 3x for inter-application scenar-
ios as compared to the existing dynamic thermal manage-
ment technique. Furthermore, the approach also improves
dynamic energy consumption by an average 10% and static
energy by 11%. In future, the approach can be extended to
consider concurrent applications and heterogeneous cores.
8. ACKNOWLEDGMENTS
This work was supported by the Engineering and Physical
SciencesResearchCouncilProgrammeGrant, EP/K034448/1.
See www.prime-project.org for more information about the
PRiME programme.
References
[1] Perf: Linux Proﬁling with Performance Counters,
https://perf.wiki.kernel.org. 2012.
[2] T. Chantem et al. Enhancing Multicore Reliability Through
Wear Compensation in Online Assignment and Scheduling. In
DATE, 2013.
[3] A. K. Coskun, T. S. Rosing, et al. Temperature management in
multiprocessor socs using online learning. InDAC,2 0 0 8 .
[4] A. Das et al. Temperature Aware Energy-Reliability Trade-oﬀs
for Mapping of Throughput-Constrained Applications on Multi-
media MPSoCs. InDATE, 2014.
[5] S. Downing and D. Socie. Simple Rainﬂow Counting Algorithms.
International Journal of Fatigue,1 9 8 2 .
[6] T. Ebi et al. Economic Learning for Thermal-aware Power Bud-
geting in Many-core Architectures. InCODES+ISSS, 2011.
[7] Y. Ge and Q. Qiu. Dynamic Thermal Management for Multime-
dia Applications Using Machine Learning. InDAC, 2011.
[8] V. Hanumaiah and S. Vrudhula. Temperature-Aware DVFS for
Hard Real-Time Applications on Multicore Processors. IEEE
Transactions on Computers,2 0 1 2 .
[ 9 ] R .J a y a s e e l a na n dT .M i t r a .D y n a m i cT h e r m a lM a n a g e m e n tv i a
Architectural Adaptation. InDAC, 2009.
[10] W. Lee et al. GOP-level Dynamic Thermal Management in
MPEG-2 Decoding. IEEE Transactions on Very Large Scale
Integration (VLSI) Systems,2 0 0 8 .
[11] M.-L. Li et al. The ALPBench Benchmark Suite for Complex
Multimedia Applications. In Workload Characterization Sym-
posium, 2005.
[12] F. Mulas et al. Thermal Balancing Policy for Multiproces-
sor Stream Computing Platforms. IEEE Transactions on
Computer-Aided Design of Integrated Circuits and Systems,
2009.
[13] V. Pallipadi and A. Starikovskiy. The Ondemand Governor. In
Proceedings of the Linux Symposium, 2006.
[14] K. Skadron et al. Temperature-Aware Microarchitecture: Mod-
eling and Implementation. ACM Transactions on Architecture
and Code Optimization (TACO),2 0 0 4 .
[15] J. Srinivasan et al. The Case for Lifetime Reliability-Aware Mi-
croprocessors. In International Symposium on Computer Ar-
chitecture (ISCA),2 0 0 4 .
[16] J. Treibig et al. LIKWID: A Lightweight Performance-Oriented
Tool Suite for x86 Multicore Environments. In International
Conference on Parallel Processing Workshops (ICPPW), 2010.
[17] I. Ukhov et al. Steady-state Dynamic Temperature Analysis and
Reliability Optimization for Embedded Multiprocessor Systems.
In DAC, 2012.
[18] C. J. Watkins et al. Q-Learning.Machine learning,1 9 9 2 .

Consistent * Complete*W
ellDocumented*EasytoReuse* *
Evaluated*
ACT
*
Artifact *
AEC
P
End-to-end Deep Learning of Optimization Heuristics
Chris Cummins, Pavlos Petoumenos
School of Informatics
University of Edinburgh
United Kingdom
{c.cummins,ppetoume}@inf.ed.ac.uk
Zheng Wang
School of Computing and Communications
Lancaster University
United Kingdom
z.wang@lancaster.ac.uk
Hugh Leather
School of Informatics
University of Edinburgh
United Kingdom
hleather@inf.ed.ac.uk
Abstract—Accurate automatic optimization heuristics are
necessary for dealing with the complexity and diversity of
modern hardware and software. Machine learning is a proven
technique for learning such heuristics, but its success is bound
by the quality of the features used. These features must be
hand crafted by developers through a combination of expert
domain knowledge and trial and error. This makes the quality
of the ﬁnal model directly dependent on the skill and available
time of the system architect.
Our work introduces a better way for building heuristics. We
develop a deep neural network that learns heuristics over raw
code, entirely without using code features. The neural network
simultaneously constructs appropriate representations of the
code and learns how best to optimize, removing the need for
manual feature creation. Further, we show that our neural
nets can transfer learning from one optimization problem to
another, improving the accuracy of new models, without the
help of human experts.
We compare the effectiveness of our automatically generated
heuristics against ones with features hand-picked by experts.
We examine two challenging tasks: predicting optimal mapping
for heterogeneous parallelism and GPU thread coarsening
factors. In 89% of the cases, the quality of our fully auto-
matic heuristics matches or surpasses that of state-of-the-art
predictive models using hand-crafted features, providing on
average 14% and 12% more performance with no human effort
expended on designing features.
Keywords-Optimization Heuristics; Machine Learning; Com-
piler Optimizations; Heterogeneous Systems
I. I NTRODUCTION
There are countless scenarios during the compilation and
execution of a parallel program where decisions must be
made as to how, or if, a particular optimization should be
applied. Modern compilers and runtimes are rife with hand
coded heuristics which perform this decision making. The
performance of parallel programs is thus dependent on the
quality of these heuristics.
Hand-written heuristics require expert knowledge, take
a lot of time to construct, and in many cases lead to
suboptimal decisions. Researchers have focused on machine
learning as a means to constructing high quality heuristics
that often outperform their handcrafted equivalents [1–5].
A predictive model is trained, using supervised machine
learning, on empirical performance data and important quan-
tiﬁable properties, or features, of representative programs.
RuntimesRuntimes
Feature 
Extractor
FeaturesFeaturesFeatures
ProgramsProgramsPrograms Predictive 
Model
ProgramsProgramsTraining 
DataRuntimesDriver
(a) Current state-of-practice
RuntimesRuntimesProgramsProgramsPrograms Predictive 
Model
ProgramsProgramsTraining 
DataRuntimesDriver
(b) Our proposal
Figure 1: Building a predictive model. The model is originally
trained on performance data and features extracted from the source
code and the runtime behavior. We propose bypassing feature
extraction, instead learning directly over raw program source code.
The model learns the correlation between these features and
the optimization decision that maximizes performance. The
learned correlations are used to predict the best optimization
decisions for new programs. Previous works in this area were
able to build machine learning based heuristics with less
effort, that outperform ones created manually experts [6, 7].
Still, experts are not completely removed from the de-
sign process, which is shown in Figure 1a. Selecting the
appropriate features is a manual undertaking which requires
a deep understanding of the system. The designer essentially
decides which compile or runtime characteristics affect
optimization decisions and expresses them in ways that
make it easy to model their relationship to performance.
Failing to identify an important feature has a negative effect
on the resulting heuristic. For example, in [8] the authors
discovered that [6] did not identify one such feature, causing
performance to be 40% lower on average.
To make heuristic construction fast and cheap, we must
take humans out of the loop. While techniques for automatic
feature generation from the compiler IR have been proposed
in the past [9, 10], they do not solve the problem in a
practical way. They are deeply embedded into the compiler,
require expert knowledge to guide the generation, have to
be repeated from scratch for every new heuristic, and their
search time can be prohibitive. Our insight was that such
costly approaches are not necessary any more. Deep learning
techniques have shown astounding successes in identifying
complex patterns and relationships in images [11, 12], au-
dio [13], and even computer code [14, 15]. We hypothesized
that deep neural networks should be able to automatically
extract features from source code. Our experiments showed
that even this was a conservative target: with deep neural
networks we can bypass static feature extraction and learn
optimization heuristics directly on raw code.
Figure 1b shows our proposed methodology. Instead of
manually extracting features from input programs to gen-
erate training data, program code is used directly in the
training data. Programs are fed through a series of neural
networks which learn how code correlates with perfor-
mance. Internally and without prior knowledge, the net-
works construct complex abstractions of the input program
characteristics and correlations between those abstractions
and performance. Our work replaces the need for compile-
time or static code features, merging feature and heuristic
construction into a single process of joint learning. Our
system admits auxiliary features to describe information un-
available at compile time, such as the sizes of runtime input
parameters. Beyond these optional inclusions, we are able
to learn optimization heuristics without human guidance.
By employing transfer learning [16], our approach is able
to produce high quality heuristics even when learning on a
small number of programs. The properties of the raw code
that are abstracted by the beginning layers of our neural net-
works are mostly independent of the optimization problem.
We reuse these parts of the network across heuristics, and,
in the process, we speed up learning considerably.
We evaluated our approach on two problems: heteroge-
neous device mapping and GPU thread coarsening. Good
heuristics for these two problems are important for extracting
performance from heterogeneous systems, and the fact that
machine learning has been used before for heuristic con-
struction for these problems allows direct comparison. Prior
machine learning approaches resulted in good heuristics
which extracted 73% and 79% of the available performance
respectively but required extensive human effort to select the
appropriate features. Nevertheless, our approach was able to
outperform them by 14% and 12%, which indicates a better
identiﬁcation of important program characteristics, without
any expert help. We make the following contributions:
• We present a methodology for building compiler heuris-
tics without any need for feature engineering.
• A novel tool DeepTune for automatically constructing
optimization heuristics without features. DeepTune out-
performs existing state-of-the-art predictive models by
14% and 12% in two challenging optimization domains.
• We apply, for the ﬁrst time, transfer learning on
compile-time and runtime optimizations, improving the
heuristics by reusing training information across differ-
ent optimization problems, even if they are unrelated.
Heuristic 
Model
Auxiliary InputsSource Code
Dense NN
Predicted Optimization
Embedding
Source Rewriter
Sequence Encoder
Batch Normalization
void nbody_sim(__global float4* pos... [256, 2095172]
Device mapping: {CPU,GPU}
Language Model
LSTM
(optional)
(concatenate)
Figure 2: DeepTune architecture. Code properties are extracted
from source code by the language model. They are fed, together
with optional auxiliary inputs, to the heuristic model to produce
the ﬁnal prediction.
II. D EEP TUNE : LEARNING ON RAW PROGRAM CODE
DeepTune is an end-to-end machine learning pipeline for
optimization heuristics. Its primary input is the source code
of a program to be optimized, and through a series of neural
networks, it directly predicts the optimization which should
be applied. By learning on source code, our approach is
not tied to a speciﬁc compiler, platform, or optimization
problem. The same design can be reused to build multiple
heuristics. The most important innovation of DeepTune is
that it forgoes the need for human experts to select and tune
appropriate features.
A. System Overview
Figure 2 provides an overview of the system. A source
rewriter removes semantically irrelevant information (such
as comments) from the source code of the target program and
passes it to a language model. The language model converts
the arbitrary length stream of code into a ﬁxed length vector
of real values which fully capture the properties and structure
of the source, replacing the role of hand designed features.
We then optionally concatenate this vector with auxiliary
inputs, which allow passing additional data about runtime
or architectural parameters to the model for heuristics which
need more than just compile-time information. Finally, a
standard feed-forward network is used to predict the best
heuristic parameters to optimize the program.
DeepTune is open source1. We implemented the model in
Keras, with TensorFlow [17] and Theano [18] backends.
B. Language Model
Learning effective representations of source code is a
difﬁcult task. A successful model must be able to:
• derive semantic and syntactic patterns of a program-
ming language entirely from sample codes;
• identify the patterns and representation in source codes
which are relevant to the task at hand; and
• discriminate performance characteristics arising from
potentially subtle differences in similar codes.
To achieve this task, we employ state-of-the-art language
modeling techniques, coupled with a series of generic,
language agnostic code transformations.
Source Rewriter: To begin with, we apply a series
of source normalizing transformations extended from our
previous work [8]. These transformations, implemented as
an LLVM pass, parse the AST, removing conditional compi-
lation, then rebuild the input source code using a consistent
code style and identiﬁer naming scheme. The role of source
normalization is to simplify the task of modeling source code
by ensuring that trivial semantic differences in programs
such as the choice of variable names or the insertion of
comments do not affect the learned model. Figures 3a and 3b
show the source rewriting applied to a simple program.
Sequence Encoder: We encode source code as a se-
quence of integers for interpretation by neural networks,
where each integer is an index into a predetermined vo-
cabulary. In [8], a character based vocabulary is used. This
minimizes the size of the vocabulary, but leads to long
sequences which are harder to extract structure from. In [19],
a token based vocabulary is used. This leads to shorter
sequences, but causes an explosion in the vocabulary size,
as every identiﬁer and literal must be represented uniquely.
We designed a hybrid, partially tokenized approach. This
allows common multi-character sequences such as float
and if to be represented as unique vocabulary items, while
literals and other infrequently used words are encoded at the
character level.
We ﬁrst assembled a candidate vocabulary Vc for the
OpenCL programming language containing the 208 data
types, keywords, and language builtins of the OpenCL
programming language. We then derived the subset of the
candidate vocabulary V ∈Vc which is required to encode a
corpus of 45k lines of GPGPU benchmark suite kernels. Be-
ginning with the ﬁrst character in the corpus, our algorithm
consumes the longest matching sequence from the candidate
vocabulary. This process continues until every character in
the corpus has been consumed. The resulting derived vocab-
ulary consists of 128 symbols which we use to encode new
program sources. Figure 3c shows the vocabulary derived
for a single input source code Figure 3b.
1DeepTune is available at: https://chriscummins.cc/deeptune
1 / / # d e f i n e E l e m e n t s
2 _ _ k e r n e l void m e m s e t _ k e r n e l (_ _ g l o b a l char ∗ mem_d ,
↪→ s h o r t val , i n t number_bytes ) {
3 c o n s t i n t t h r e a d _ i d = g e t _ g l o b a l _ i d ( 0 ) ;
4 mem_d [ t h r e a d _ i d ] = v a l ;
5 }
(a) An example, short OpenCL kernel, taken from Nvidia’s streamcluster.
1 _ _ k e r n e l void A( _ _ g l o b a l char∗ a , s h o r t b , i n t c ) {
2 c o n s t i n t d = g e t _ g l o b a l _ i d ( 0 ) ;
3 a [ d ] = b ;
4 }
(b) The streamcluster kernel after source rewriting. Variable and function
names are normalized, comments removed, and code style enforced.
idx token idx token idx token
1 ‘__kernel’ 10 ‘,’ 19 ‘const’
2 ‘ ’ 11 ‘short’ 20 ‘d’
3 ‘void’ 12 ‘b’ 21 ‘=’
4 ‘A’ 13 ‘int’ 22 ‘get_global_id’
5 ‘(’ 14 ‘c’ 23 ‘0’
6 ‘__global’ 15 ‘)’ 24 ‘;’
7 ‘char’ 16 ‘{’ 25 ‘[’
8 ‘ *’ 17 ‘\n’ 26 ‘]’
9 ‘a’ 18 ‘ ’ 27 ‘}’
(c) Derived vocabulary, ordered by their appearance in the input (b). The
vocabulary maps tokens to integer indices.
01 02 03 02 04 05 06 02 07 08 02
09 10 02 11 02 12 10 02 13 02 14
15 02 16 17 18 19 02 13 02 20 02
21 02 22 05 23 15 24 17 18 09 25
20 26 02 21 02 12 24 17 27 <pad...>
(d) Indices encoded kernel sequence. Sequences may be padded to a ﬁxed
length by repeating an out-of-vocabulary integer (e.g. -1).
Figure 3: Deriving a tokenized 1-of-k vocabulary encoding from
an OpenCL source code.
Embedding: During encoding, tokens in the vocabulary
are mapped to unique integer values, e.g. float →0, int
→1. The integer values chosen are arbitrary, and offer a
sparse data representation, meaning that a language model
cannot infer the relationships between tokens based on their
mappings. This is in contrast to the dense representations
of other domains, such as pixels in images, which can be
interpolated between to derive the differences in colors.
To mitigate this, we use an embedding, which trans-
lates tokens in a sparse, integer encoded vocabulary into
a lower dimensional vector space, allowing semantically
related tokens like float and int to be mapped to nearby
points [20, 21]. An embedding layer maps each token in the
integer encoded vocabulary to a vector of real values. Given
a vocabulary size V and embedding dimensionality D, an
embedding matrix WE ∈RV ×D is learned during training,
so that an integer encoded sequences of tokens t ∈NL is
mapped to the matrix T ∈RL×D. We use an embedding
dimensionality D= 64.
Sequence Characterization: Once source codes have
been encoded into sequences of embedding vectors, neural
networks are used to extract a ﬁxed size vector which
characterizes the entire sequence. This is comparable to the
hand engineered feature extractors used in prior works, but is
a learned process that occurs entirely — and automatically
— within the hidden layers of the network.
We use the the Long Short-Term Memory (LSTM) archi-
tecture [22] for sequence characterization. LSTMs imple-
ments a Recurrent Neural Network in which the activations
of neurons are learned with respect not just to their current
inputs, but to previous inputs in a sequence. Unlike reg-
ular recurrent networks in which the strength of learning
decreases over time (a symptom of the vanishing gradients
problem [23]), LSTMs employ a forget gate with a linear
activation function, allowing them to retain activations for
arbitrary durations. This makes them effective at learning
complex relationships over long sequences [24], an espe-
cially important capability for modeling program code, as
dependencies in sequences frequently occur over long ranges
(for example, a variable may be declared as an argument to
a function and used throughout).
We use a two layer LSTM network. The network receives
a sequence of embedding vectors, and returns a single output
vector, characterizing the entire sequence.
C. Auxiliary Inputs
We support an arbitrary number of additional real valued
auxiliary inputs which can be optionally used to augment the
source code input. We provide these inputs as a means of
increasing the ﬂexibility of our system, for example, to sup-
port applications in which the optimization heuristic depends
on dynamic values which cannot be statically determined
from the program code [3, 25]. When present, the values
of auxiliary inputs are concatenated with the output of the
language model, and fed into a heuristic model.
D. Heuristic Model
The heuristic model takes the learned representations of
the source code and auxiliary inputs (if present), and uses
these values to make the ﬁnal optimization prediction.
We ﬁrst normalize the values. Normalization is necessary
because the auxiliary inputs can have any values, whereas
the language model activations are in the range [0,1]. If we
did not normalize, then scaling the auxiliary inputs could
affect the training of the heuristic model. Normalization
occurs in batches. We use the normalization method of [26],
in which each scalar of the heuristic model’s inputsx1 ...x n
is normalized to a mean 0 and standard deviation of 1:
x′
i = γi
xi −E(xi)√
Var(xi)
+ βi (1)
where γ and βare scale and shift parameters, learned during
training.
The ﬁnal component of DeepTune is comprised of two
fully connected neural network layers. The ﬁrst layer con-
sists of 32 neurons. The second layer consists of a single
neuron for each possible heuristic decision. Each neuron
applies an activation function f(x) over its inputs. We
use rectiﬁer activation functions f(x) = max(0 ,x) for
the ﬁrst layer due to their improved performance during
training of deep networks [27]. For the output layer, we use
sigmoid activation functions f(x) = 1
1+e−x which provide
activations in the range [0,1].
The activation of each neuron in the output layer repre-
sents the model’s conﬁdence that the corresponding decision
is the correct one. We take the arg max of the output layer
to ﬁnd the decision with the largest activation. For example,
for a binary optimization heuristic the ﬁnal layer will consist
of two neurons, and the predicted optimization is the neuron
with the largest activation.
E. Training the network
DeepTune is trained in the same manner as prior works,
the key difference being that instead of having to manually
create and extract features from programs, we simply use
the raw program codes themselves.
The model is trained with Stochastic Gradient Descent
(SGD), using the Adam optimizer [28]. For training data
X1 ...X n, SGD attempts to ﬁnd the model parameters Θ
that minimize the output of a loss function:
Θ = arg min
Θ
1
n
n∑
i=1
ℓ(Xi,Θ) (2)
where loss function ℓ(x,Θ) computes the logarithmic dif-
ference between the predicted and expected values.
To reduce training time, multiple inputs are batched
together and are fed into the neural network simultaneously,
reducing the frequency of costly weight updates during back-
propagation. This requires that the inputs to the language
model be the same length. We pad all sequences up to a ﬁxed
length of 1024 tokens using a special padding token, allow-
ing matrices of batch_size ×max_seq_len tokens
to be processed simultaneously. We note that batching and
padding sequences to a maximum length is only to improve
training time. In production use, sequences do not need to
be padded, allowing classiﬁcation of arbitrary length codes.
III. E XPERIMENTAL METHODOLOGY
We apply DeepTune to two heterogeneous compiler-based
machine learning tasks and compare its performance to state-
of-the-art approaches that use expert selected features.
A. Case Study A: OpenCL Heterogeneous Mapping
OpenCL provides a platform-agnostic framework for het-
erogeneous parallelism. This allows a program written in
OpenCL to execute transparently across a range of different
devices, from CPUs to GPUs and FPGAs. Given a program
Name Description
F1: data size/(comp+mem) commun.-computation ratio
F2: coalesced/mem % coalesced memory accesses
F3: (localmem/mem)×wgsize ratio local to global mem accesses
×#. work-items
F4: comp/mem computation-mem ratio
(a) Feature values
Name Type Description
comp static #. compute operations
mem static #. accesses to global memory
localmem static #. accesses to local memory
coalesced static #. coalesced memory accesses
data size dynamic size of data transfers
workgroup size dynamic #. work-items per kernel
(b) Values used in feature computation
Table I: Features used by Grewe et al. to predict heterogeneous
device mappings for OpenCL kernels.
and a choice of execution devices, the question then is on
which device should we execute the program to maximize
performance?
State-of-the-art: In [6], Grewe et al. develop a pre-
dictive model for mapping OpenCL kernels to the opti-
mal device in CPU/GPU heterogeneous systems. They use
supervised learning to construct decision trees, using a
combination of static and dynamic kernel features. The static
program features are extracted using a custom LLVM pass;
the dynamic features are taken from the OpenCL runtime.
Expert Chosen Features: Table Ia shows the features
used by their work. Each feature is an expression built upon
the code and runtime metrics given in Table Ib.
Experimental Setup: We replicate the predictive model
of Grewe et al. [6]. We replicated the experimental setup
of [8] in which the experiments are extended to a larger set of
71 programs, summarized in Table IIa. The programs were
evaluated on two CPU-GPU platforms, detailed in Table IIIa.
DeepTune Conﬁguration: Figure 4a shows the neural
network conﬁguration of DeepTune for the task of predicting
optimal device mapping. We use the OpenCL kernel source
code as input, and the two dynamic values workgroup size
and data size available to the OpenCL runtime.
Model Evaluation: We use stratiﬁed 10-fold cross-
validation to evaluate the quality of the predictive mod-
els [29]. Each program is randomly allocated into one of
10 equally-sized sets; the sets are balanced to maintain a
distribution of instances from each class consistent with the
full set. A model is trained on the programs from all but one
of the sets, then tested on the programs of the unseen set.
This process is repeated for each of the 10 sets, to construct
a complete prediction over the whole dataset.
B. Case Study B: OpenCL Thread Coarsening Factor
Thread coarsening is an optimization for parallel programs
in which the operations of two or more threads are fused
together. This optimization can prove beneﬁcial on certain
Version #. benchmarks #. kernels
NPB (SNU [30]) 1.0.3 7 114
Rodinia [31] 3.1 14 31
NVIDIA SDK 4.2 6 12
AMD SDK 3.0 12 16
Parboil [32] 0.2 6 8
PolyBench [33] 1.0 14 27
SHOC [34] 1.1.5 12 48
Total - 71 256
(a) Case Study A: OpenCL Heterogeneous Mapping
Version #. benchmarks #. kernels
NVIDIA SDK 4.2 3 3
AMD SDK 3.0 10 10
Parboil [32] 0.2 4 4
Total - 17 17
(b) Case Study B: OpenCL Thread Coarsening Factor
Table II: Benchmark programs.
Frequency Memory Driver
Intel Core i7-3820 3.6 GHz 8GB AMD 1526.3
AMD Tahiti 7970 1000 MHz 3GB AMD 1526.3
NVIDIA GTX 970 1050 MHz 4GB NVIDIA 361.42
(a) Case Study A: OpenCL Heterogeneous Mapping
Frequency Memory Driver
AMD HD 5900 725 MHz 2GB AMD 1124.2
AMD Tahiti 7970 1000 MHz 3GB AMD 1084.4
NVIDIA GTX 480 700 MHz 1536 MB NVIDIA 304.54
NVIDIA K20c 706 MHz 5GB NVIDIA 331.20
(b) Case Study B: OpenCL Thread Coarsening Factor
Table III: Experimental platforms.
{CPU,GPU}
DNN_2
DNN_1
Normal.
Concat.
LSTM_2
LSTM_1
Embedding
Inputs
wgsize
(a)
dsizecode code
(b)
{1,2,4,8,16,32}
Figure 4: DeepTune neural networks, conﬁgured for (a) heteroge-
neous mapping, and (b) thread coarsening factor. The design stays
almost the same regardless of the optimization problem. The only
changes are the extra input for (a) and size of the output layers.
combinations of programs and architectures, for example
programs with a large potential for Instruction Level Par-
allelism on Very Long Instruction Word architectures.
No …
Source code (CF1)
Feature vector (CF1)
Coarsen?
No Yes
Feature vector (CF2)
Feature vector (CF4)
Coarsen?
No Yes
Coarsen?
CF: 2
CF: 1
CF: 4
Source code (CF2)
Coarsening Pass
Source code (CF4)
Coarsening Pass
(a) Magni et al. cascading binary model.
Source code (CF1)
Coarsening Factor?
CF:1 CF:2 CF:4 CF:8 CF:16 CF:32
(b) Our approach.
Figure 5: Two approaches for predicting coarsening factor (CF) of
OpenCL kernels. Magni et al. reduce the multi-label classiﬁcation
problem to a series of binary decisions, by iteratively applying
the optimization and computing new feature vectors. Our approach
simply predicts the coarsening factor directly from the source code.
State-of-the-art: Magni et al. present a predictive
model for OpenCL thread coarsening in [7]. They implement
an iterative heuristic which determines whether a given
program would beneﬁt from coarsening. If yes, then the
program is coarsened, and the process repeats, allowing
further coarsening. In this manner, the problem is reduced
from a multi-label classiﬁcation problem into a series of
binary decisions, shown in Figure 5a. They select from one
of six possible coarsening factors: (1,2,4,8,16,32), divided
into 5 binary choices.
Expert Chosen Features: Magni et al. followed a very
comprehensive feature engineering process. 17 candidate
features were assembled from previous studies of perfor-
mance counters and computed theoretical values [35, 36].
For each candidate feature they compute its coarsening
delta, reﬂecting the change in each feature value caused by
coarsening: f∆ = ( fafter −fbefore )/fbefore , adding it to
the feature set. Then they use Principle Component Analysis
(PCA) on the 34 candidates and selected the ﬁrst 7 principle
components, accounting for 95% of variance in the space.
Name Description
BasicBlocks #. basic blocks
Branches #. branches
DivInsts #. divergent instructions
DivRegionInsts #. instructions in divergent regions
DivRegionInstsRatio #. instr. in divergent regions / total
instructions
DivRegions #. divergent regions
TotInsts #. instructions
FPInsts #. ﬂoating point instructions
ILP average ILP / basic block
Int/FP Inst Ratio #. branches
IntInsts #. integer instructions
MathFunctions #. match builtin functions
MLP average MLP / basic block
Loads #. loads
Stores #. stores
UniformLoads #. loads unaffected by coarsening direction
Barriers #. barriers
Table IV: Candidate features used by Magni et al. for predicting
thread coarsening. From these values, they compute relative deltas
for each iteration of coarsening, then use PCA for selection.
#. neurons #. parameters
HM CF HM CF
Embedding 64 64 ,256 8,256
LSTM_1 64 64 33,024 33,024
LSTM_2 64 64 33,024 33,024
Concatenate 64 + 2 - - -
Batch Norm . 66 64 264 256
DNN_1 32 32 2,144 2,080
DNN_2 2 6 66 198
Total 76,778 76,838
Table V: The size and number of parameters of the DeepTune
components of Figure 4, conﬁgured for heterogeneous mapping
(HM) and coarsening factor (CF).
Experimental Setup: We replicate the experimental
setup of Magni et al. [7]. The thread coarsening optimization
is evaluated on 17 programs, listed in Table IIb. Four
different GPU architectures are used, listed in Table IIIb.
DeepTune Conﬁguration: Figure 4b shows the neural
network conﬁguration. We use the OpenCL kernel as input,
and directly predict the coarsening factor.
Model Evaluation: Compared to Case Study A, the
size of the evaluation is small. We use leave-one-out cross-
validation to evaluate the models. For each program, a model
is trained on data from all other programs and used to predict
the coarsening factor of the excluded program.
Because [7] does not describe the parameters of the neural
network, we perform an additional, nested cross-validation
process to ﬁnd the optimal model parameters. For every
program in the training set, we evaluate 48 combinations of
network parameters. We select the best performing conﬁgu-
ration from these 768 results to train a model for prediction
on the excluded program. This nested cross-validation is
repeated for each of the training sets. We do not perform
this tuning of hyper-parameters for DeepTune.
C. Comparison of Case Studies
For the two different optimization heuristics, the authors
arrived at very different predictive model designs, with very
different features. By contrast, we take exactly the same
approach for both problems. None of DeepTune’s parameters
were tuned for the case studies presented above. Their
settings represent conservative choices expected to work
reasonably well for most scenarios.
Table V shows the similarity of our models. The only
difference between our network design is the auxiliary inputs
for Case Study A and the different number of optimization
decisions. The differences between DeepTune conﬁgurations
is only two lines of code: the ﬁrst, adding the two auxiliary
inputs; the second, increasing the size of the output layer for
Case Study B from two neurons to six. The description of
these differences is larger than the differences themselves.
IV. E XPERIMENTAL RESULTS
We evaluate the effectiveness of DeepTune for two distinct
optimization tasks: predicting the optimal device to run a
given program, and predicting thread coarsening factors.
We ﬁrst compare DeepTune against two expert-tuned
predictive models, showing that DeepTune outperforms the
state-of-the-art in both cases. We then show that by lever-
aging knowledge learned from training DeepTune for one
heuristic, we can boost training for the other heuristic,
further improving performance. Finally, we analyze the
working mechanism of DeepTune.
A. Case Study A: OpenCL Heterogeneous Mapping
Selecting the optimal execution device for OpenCL
kernels is essential for maximizing performance. For a
CPU/GPU heterogeneous system, this presents a binary
choice. In this experiment, we compare our approach against
a static single-device approach and the Grewe et al. predic-
tive model. The static mapping selects the device which gave
the best average case performance over all the programs. On
the AMD platform, the best-performing device is the CPU;
on the NVIDIA platform, it is the GPU.
Figure 6 shows the accuracy of both predictive models
and the static mapping approach for each of the benchmark
suites. The static approach is accurate for only 58.8% of
cases on AMD and 56.9% on NVIDIA. This suggests the
need for choosing the execution device on a per program
basis. The Grewe et al. model achieves an average accuracy
of 73%, a signiﬁcant improvement over the static mapping.
By automatically extracting useful feature representations
from the source code, DeepTune gives an average accuracy
of 82%, an improvement over both schemes.
Using the static mapping as a baseline, we compute
the relative performance of each program using the device
selected by the Grewe et al. and DeepTune models. Figure 7
shows these speedups. Both predictive models signiﬁcantly
outperform the static mapping; the Grewe et al. model
AMD/uni00A0SDK
NPB
NVIDIA/uni00A0SDK
ParboilPolybenchRodinia SHOC Average
(a)/uni00A0AMD/uni00A0Tahiti/uni00A07970
0%
20%
40%
60%
80%
100%Accuracy
Static/uni00A0mapping Grewe/uni00A0et/uni00A0al.DeepTune
AMD/uni00A0SDK
NPB
NVIDIA/uni00A0SDK
ParboilPolybenchRodinia SHOC Average
(b)/uni00A0NVIDIA/uni00A0GTX/uni00A0970
0%
20%
40%
60%
80%
100%Accuracy
Figure 6: Accuracy of optimization heuristics for heterogeneous
device mapping, aggregated by benchmark suite. The optimal static
mapping achieves 58% accuracy. The Grewe et al. and DeepTune
predictive models achieve accuracies of 73% and 84%, respectively.
achieves an average speedup of 2.91×on AMD and 1.26×
on NVIDIA (geomean 1.18×). In 90% of cases, DeepTune
matches or outperforms the predictions of the Grewe et al.
model, achieving an average speedup of 3.34×on AMD
and 1.41×on NVIDIA (geomean 1.31×). This 14% im-
provement in performance comes at a greatly reduced cost,
requiring no intervention by humans.
B. Case Study B: OpenCL Thread Coarsening Factor
Exploiting thread coarsening for OpenCL kernels is a
difﬁcult task. On average, coarsening slows programs down.
The speedup attainable by a perfect heuristic is only 1.36×.
Figure 8 shows speedups achieved by the Magni et al. and
DeepTune models for all programs and platforms. We use
as baseline the performance of programs without coarsening.
On the four experimental platforms (AMD HD 5900, Tahiti
7970, NVIDIA GTX 480, and Tesla K20c), the Magni
et al. model achieves average speedups of 1.21×, 1.01×,
0.86×, and 0.94×, respectively. DeepTune outperforms this,
achieving speedups of 1.10×, 1.05×, 1.10×, and 0.99×.
Some programs — especially those with large divergent
regions or indirect memory accesses — respond very poorly
to coarsening. No performance improvement is possible
on the mvCoal and spmv programs. Both models fail to
achieve positive average speedups on the NVIDIA Tesla
AMD.BinomialOption
AMD.BitonicSort
AMD.BlackScholes
AMD.FastWalshTransform
AMD.FloydWarshall
AMD.MatrixMultiplication
AMD.MatrixTranspose
AMD.PrefixSum
AMD.Reduction
AMD.ScanLargeArrays
AMD.SimpleConvolution
AMD.SobelFilter
NPB.BT
NPB.CG
NPB.EP
NPB.FT
NPB.LU
NPB.MG
NPB.SP
NVIDIA.DotProduct
NVIDIA.FDTD3d
NVIDIA.MatVecMul
NVIDIA.MatrixMul
NVIDIA.MersenneTwister
NVIDIA.VectorAdd
Parboil.bfs
Parboil.cutcp
Parboil.lbm
Parboil.sad
Parboil.spmv
Parboil.stencil
Polybench.2DConvolution
Polybench.2mm
Polybench.3DConvolution
Polybench.3mm
Polybench.atax
Polybench.bicg
Polybench.correlation
Polybench.covariance
Polybench.gemm
Polybench.gesummv
Polybench.gramschmidt
Polybench.mvt
Polybench.syr2k
Polybench.syrk
Rodinia.backprop
Rodinia.bfs
Rodinia.cfd
Rodinia.gaussian
Rodinia.hotspot
Rodinia.kmeans
Rodinia.lavaMD
Rodinia.leukocyte
Rodinia.lud
Rodinia.nn
Rodinia.nw
Rodinia.particlefilter
Rodinia.pathfinder
Rodinia.streamcluster
SHOC.BFS
SHOC.FFT
SHOC.GEMM
SHOC.MD
SHOC.MD5Hash
SHOC.Reduction
SHOC.S3D
SHOC.Scan
SHOC.Sort
SHOC.Spmv
SHOC.Stencil2D
SHOC.Triad
Average
(a)/uni00A0AMD/uni00A0Tahiti/uni00A07970
1.0x
3.0x
5.0x
7.0x
9.0xSpeedup
Grewe/uni00A0et/uni00A0al.
DeepTune
AMD.BinomialOption
AMD.BitonicSort
AMD.BlackScholes
AMD.FastWalshTransform
AMD.FloydWarshall
AMD.MatrixMultiplication
AMD.MatrixTranspose
AMD.PrefixSum
AMD.Reduction
AMD.ScanLargeArrays
AMD.SimpleConvolution
AMD.SobelFilter
NPB.BT
NPB.CG
NPB.EP
NPB.FT
NPB.LU
NPB.MG
NPB.SP
NVIDIA.DotProduct
NVIDIA.FDTD3d
NVIDIA.MatVecMul
NVIDIA.MatrixMul
NVIDIA.MersenneTwister
NVIDIA.VectorAdd
Parboil.bfs
Parboil.cutcp
Parboil.lbm
Parboil.sad
Parboil.spmv
Parboil.stencil
Polybench.2DConvolution
Polybench.2mm
Polybench.3DConvolution
Polybench.3mm
Polybench.atax
Polybench.bicg
Polybench.correlation
Polybench.covariance
Polybench.gemm
Polybench.gesummv
Polybench.gramschmidt
Polybench.mvt
Polybench.syr2k
Polybench.syrk
Rodinia.backprop
Rodinia.bfs
Rodinia.cfd
Rodinia.gaussian
Rodinia.hotspot
Rodinia.kmeans
Rodinia.lavaMD
Rodinia.leukocyte
Rodinia.lud
Rodinia.nn
Rodinia.nw
Rodinia.particlefilter
Rodinia.pathfinder
Rodinia.streamcluster
SHOC.BFS
SHOC.FFT
SHOC.GEMM
SHOC.MD
SHOC.MD5Hash
SHOC.Reduction
SHOC.S3D
SHOC.Scan
SHOC.Sort
SHOC.Spmv
SHOC.Stencil2D
SHOC.Triad
Average
(b)/uni00A0NVIDIA/uni00A0GTX/uni00A0970
0.0x
1.0x
2.0x
3.0x
4.0xSpeedup
Grewe/uni00A0et/uni00A0al.
DeepTune
Figure 7: Speedup of predicted heterogeneous mappings over the best static mapping for both platforms. In (a) DeepTune achieves an
average speedup of 3.43x over static mapping and 18% over Grewe et al. In (b) the speedup is 1.42x and 13% respectively.
K20c, because thread coarsening does not give performance
gains for the majority of the programs on this platform.
The disappointing results for both predictive models can
be attributed to the small training program set used by
Magni et al. (only 17 programs in total). As a result, the
models suffer from sparse training data. Prior research has
shown that data sparsity can be overcome using additional
programs; in the following subsection we describe and test a
novel strategy for training optimization heuristics on a small
number of programs by exploiting knowledge learned from
other optimization domains.
C. Transfer Learning Across Problem Domains
There are inherent differences between the tasks of build-
ing heuristics for heterogeneous mapping and thread coars-
ening, evidenced by the contrasting choices of features and
models in Grewe et al. and Magni et al. However, in both
cases, the ﬁrst role of DeepTune is to extract meaningful
abstractions and representations of OpenCL code. Prior
research in deep learning has shown that models trained on
similar inputs for different tasks often share useful com-
monalities. The idea is that in neural network classiﬁcation,
information learned at the early layers of neural networks
(i.e. closer to the input layer) will be useful for multiple
tasks. The later the network layers are (i.e. closer to the
output layer), the more specialized the layers become [37].
We hypothesized that this would be the case for Deep-
Tune, enabling the novel transfer of information across
different optimization domains. To test this, we extracted the
language model — the Embedding, and LSTM_{1,2}
layers — trained for the heterogeneous mapping task and
transferred it over to the new task of thread coarsening.
Since DeepTune keeps the same design for both optimization
problems, this is as simple as copying the learned weights
of the three layers. Then we trained the model as normal.
As shown in Figure 8, our newly trained model,
DeepTune-TL has improved performance for 3 of the 4 plat-
binarySearch
blackscholes
convolution
dwtHaar1D
fastWalsh
floydWarshall
mriQ
mt
mtLocal
mvCoal
mvUncoal
nbody
reduce
sgemm
sobel
spmv
stencil
Average
0.0x
0.5x
1.0x
1.5x
2.0x
2.5x
Speedup
AMD Radeon HD 5900
Magni et al.
DeepTune
DeepTune-TL
binarySearch
blackscholes
convolution
dwtHaar1D
fastWalsh
floydWarshall
mriQ
mt
mtLocal
mvCoal
mvUncoal
nbody
reduce
sgemm
sobel
spmv
stencil
Average
0.0x
0.5x
1.0x
1.5x
2.0x
2.5x
AMD Tahiti 7970
Magni et al.
DeepTune
DeepTune-TL
binarySearch
blackscholes
convolution
dwtHaar1D
fastWalsh
floydWarshall
mriQ
mt
mtLocal
mvCoal
mvUncoal
nbody
reduce
sgemm
sobel
spmv
stencil
Average
0.0x
0.5x
1.0x
1.5x
2.0x
2.5x
Speedup
NVIDIA GTX 480
Magni et al.
DeepTune
DeepTune-TL
binarySearch
blackscholes
convolution
dwtHaar1D
fastWalsh
floydWarshall
mriQ
mt
mtLocal
mvCoal
mvUncoal
nbody
reduce
sgemm
sobel
spmv
stencil
Average
0.0x
0.5x
1.0x
1.5x
2.0x
2.5x
NVIDIA Tesla K20c
Magni et al.
DeepTune
DeepTune-TL
Figure 8: Speedups of predicted coarsening factors for each platform. DeepTune outperforms Magni et al on three of the four platforms.
Transfer learning improves DeepTune speedups further, by 16% on average.
forms: 1.17×, 1.23×, 1.14×, 0.93×, providing an average
12% performance improvement over Magni et al. In 81%
of cases, the use of transfer learning matched or improved
the optimization decisions of DeepTune, providing up to a
16% improvement in per platform performance.
On the NVIDIA Tesla K20c, the platform for which no
predictive model achieves positive average speedups, we
match or improve performance in the majority of cases, but
over-coarsening on three of the programs causes a modest
reduction in average performance. We suspect that for this
platform, further performance results are necessary due to
its unusual optimization proﬁle.
D. DeepTune Internal Activation States
We have shown that DeepTune automatically outperforms
state-of-the-art predictive models for which experts have
invested a great amount of time in engineering features. In
this subsection we attempt to illuminate the inner workings,
using a single example from Case Study B: predicting the
thread coarsening factor for Parboil’s mriQ benchmark on
four different platforms.
Figure 9 shows the DeepTune conﬁguration, with visual
overlays showing the internal state. From top to bottom,
we begin ﬁrst with the input, which is the 267 lines of
OpenCL code for the mriQ kernel. This source code is
preprocessed, formatted, and rewritten using variable and
function renaming, shown in Figure 9b. The rewritten source
code is tokenized and encoded in a 1-of-k vocabulary.
Figure 9c shows the ﬁrst 80 elements of this encoded
sequence as a heatmap in which each cell’s color reﬂects
its encoded value. The input, rewriting, and encoding is the
same for each of the four platforms.
The encoded sequences are then passed into the Em-
bedding layer. This maps each token of the vocabulary
to a point in a 64 dimension vector space. Embeddings
are learned during training so as to cluster semantically
Embedding
__kernel void A(int a, int b, __global float* c, __global float* d, __global float* e, __global float* f, __global float* g, __global float4* 
h) {
  float i[4];
  float j[4]; // ...
#include "macros.h"
#define NC  4
#define COARSE_GENERAL
__kernel void
ComputeQ_GPU(int numK, int kGlobalIndex, __global float* x, __global float* y, __global float* z,
             __global float* Qr, __global float* Qi, __global struct kValues* ck) { 
    float sX[NC] // ...
CF: 2 CF: 4 CF: 1CF: 2
Source Code
Dense NN
Predicted Optimization
Source Rewriter
Batch Normalization
LSTM
Language ModelHeuristic 
Model
…
AMD HD 5900
AMD Tahiti 7970
NVIDIA GTX 480
NVIDIA Tesla K20c
AMD HD 5900
AMD Tahiti 7970
NVIDIA GTX 480
NVIDIA Tesla K20c
AMD HD 5900 AMD Tahiti 7970 NVIDIA GTX 480 NVIDIA Tesla K20c
Encoded sequence (ﬁrst 80 tokens)
Outputs of Language Models
Heuristic Models
Learned embeddings (PCA projections)
Sequence Encoder
(a)
(b)
(c)
(d)
(e)
(f)
(g)
Figure 9: Visualizing the internal state of DeepTune when predicting coarsening factor for Parboil’s mriQ benchmark on four different
architectures. The activations in each layer of the four models increasingly diverge the lower down the network.
related tokens together. As such, they may differ between
the four platforms. Figure 9d shows a PCA projection of the
embedding space for one of the platforms, showing multiple
clusters of tokens. By honing in on one of the clusters
and annotating each point with its corresponding token,
we see that the cluster contains the semantically related
OpenCL address space modiﬁers __private, __global,
and __read_only.
Two layers of 64 LSTM neurons model the sequence of
embeddings, with the neuron activations of the second layer
being used to characterize the entire sequence. Figure 9e
shows the neurons in this layer for each of the four plat-
forms, using a red-blue heatmap to visualize the intensity of
each activation. Comparing the activations between the four
platforms, we note a number of neurons in the layer with
different responses across platforms. This indicates that the
language model is partly specialized to the target platform.
As information ﬂows through the network, the layers be-
come progressively more specialized to the speciﬁc platform.
We see this in Figure 9f, which shows the two layers of the
heuristic model. The activations within these increasingly
diverge. The mean variance of activations across platforms
increases threefold compared to the language model, from
0.039 to 0.107. Even the activations of the AMD HD 5900
and AMD Tahiti 7970 platforms are dissimilar, despite the
ﬁnal predicted coarsening factor for both platforms being
the same. In Figure 9g we take the largest activation of the
output layer as the ﬁnal predicted coarsening factor. For this
particular program, a state-of-the-art model achieves 54% of
the maximum performance. DeepTune achieves 99%.
V. R ELATED WORK
Machine learning has emerged as a viable means in auto-
matically constructing heuristics for code optimization [38–
43]. Its great advantage is that it can adapt to changing
hardware platforms as it has no a priori assumptions about
their behavior. The success of machine learning based code
optimization has required having a set of high-quality fea-
tures that can capture the important characteristics of the
target program. Given that there is an inﬁnite number of
these potential features, ﬁnding the right set of features is a
non-trivial, time-consuming task.
Various forms of program features have been used in
compiler-based machine learning. These include static code
structures [44] and runtime information such as system
load [45] and performance counters [46]. In compiler re-
search, the feature sets used for predictive models are often
provided without explanation and rarely is the quality of
those features evaluated. More commonly, an initial large,
high dimensional candidate feature space is pruned via fea-
ture selection [3, 47], or projected into a lower dimensional
space [48, 49]. FEAST employs a range of existing feature
selection methods to select useful candidate features [50].
Unlike these approaches, DeepTune extracts features and
reduces the dimensionality of the feature space completely
internally and without expert guidance.
Park et al. present a unique graph-based approach for
feature representations [51]. They use a Support Vector
Machine where the kernel is based on a graph similarity
metric. Their technique still requires hand coded features
at the basic block level, but thereafter, graph similarity
against each of the training programs takes the place of
global features. Being a kernel method, it requires that
training data graphs be shipped with the compiler, which
may not scale as the size of the training data grows with the
number of instances, and some training programs may be
very large. Finally, their graph matching metric is expensive,
requiring O(n3) to compare against each training example.
By contrast, our method does not need any hand built
static code features, and the deployment memory footprint
is constant and prediction time is linear in the length of the
program, regardless of the size of the training set.
A few methods have been proposed to automatically gen-
erate features from the compiler’s intermediate representa-
tion [9, 10]. These approaches closely tie the implementation
of the predictive model to the compiler IR, which means
changes to the IR will require modiﬁcations to the model.
The work of [10] uses genetic programming to search for
features, and required a huge grammar to be written, some
160kB in length. Although much of this can be created
from templates, selecting the right range of capabilities
and search space bias is non trivial and up to the expert.
The work of [9] expresses the space of features via logic
programming over relations that represent information from
the IRs. It greedily searches for expressions that represent
good features. However, their approach relies on expert
selected relations, combinators and constraints to work. For
both approaches, the search time may be signiﬁcant.
Cavazos et al. present a reaction-based predictive model
for software-hardware co-design [52]. Their approach pro-
ﬁles the targetprogram using several carefully selected
compiler options to see how programruntime changes un-
der these options for a given micro-architecture setting.
Theythen use the program “reactions” to predict the best
available applicationspeedup. While their approach does not
use static code features, developers mustcarefully select
a few settings from a large number of candidate options
forproﬁling, because poorly chosen options can signiﬁcantly
affect the quality ofthe model. Moreover, the program must
be run several times before optimization,while our technique
does not require the program to be proﬁled.
In recent years, machine learning techniques have been
employed to model and learn from program source code on
various tasks. These include mining coding conventions [15]
and idioms [14], API example code [53] and pseudo-code
generation [54], and benchmark generation [8]. Our work is
the ﬁrst attempt to extend the already challenging task of
modeling distributions over source code to learning distri-
butions over source code with respect to code optimizations.
Recently, deep neural networks have been shown to be a
powerful tool for feature engineering in various tasks includ-
ing image recognition [11, 12] and audio processing [13]. No
work so far has applied deep neural networks for program
feature generation. Our work is the ﬁrst to do so.
VI. C ONCLUSIONS
Applying machine learning to compiler and runtime op-
timizations requires generating features ﬁrst. This is a time
consuming process, it needs supervision by an expert, and
even then we cannot be sure that the selected features are
optimal. In this paper we present a novel tool for building
optimization heuristics, DeepTune, which forgoes feature
extraction entirely, relying on powerful language modeling
techniques to automatically build effective representations of
programs directly from raw source code. The result trans-
lates into a huge reduction in development effort, improved
heuristic performance, and more simple model designs.
Our approach is fully automated. Using DeepTune, de-
velopers no longer need to spend months using statistical
methods and proﬁle counters to select program features via
trial and error. It is worth mentioning that we do not tailor
our model design or parameters for the optimization task at
hand, yet we achieve performance on par with and in most
cases exceeding state-of-the-art predictive models.
We used DeepTune to automatically construct heuristics
for two challenging compiler and runtime optimization
problems, ﬁnd that, in both cases, we outperform state-of-
the-art predictive models by 14% and 12%. We have also
shown that the DeepTune architecture allows us to exploit
information learned from another optimization problem to
give the learning a boost. Doing so provides up to a 16%
performance improvement when training using a handful of
programs. We suspect this approach will be useful in other
domains for which training data are a scarce resource.
In future work, we will extend our heuristic construction
approach by automatically learning dynamic features over
raw data; apply unsupervised learning techniques [56] over
unlabeled source code to further improve learned represen-
tations of programs; and deploy trained DeepTune heuristic
models to low power embedded systems using quantization
and compression of neural networks [57].
ACKNOWLEDGMENTS
This work was supported by the UK Engineer-
ing and Physical Sciences Research Council under
grants EP/L01503X/1 (CDT in Pervasive Parallelism),
EP/M01567X/1 (SANDeRs), EP/M015793/1 (DIVIDEND),
and EP/P003915/1 (SUMMER). The code and data for this
paper are available at: https://chriscummins.cc/pact17.
REFERENCES
[1] P. Micolet, A. Smith, and C. Dubach. “A Machine Learning
Approach to Mapping Streaming Workloads to Dynamic
Multicore Processors.” In: LCTES. ACM, 2016.
[2] T. L. Falch and A. C. Elster. “Machine Learning Based
Auto-tuning for Enhanced OpenCL Performance Portabil-
ity.” In: IPDPSW. IEEE, 2015.
[3] M. Stephenson and S. Amarasinghe. “Predicting Unroll
Factors Using Supervised Classiﬁcation.” In: CGO. IEEE,
2005.
[4] F. Agakov, E. Bonilla, J. Cavazos, B. Franke, G. Fursin, M.
O’Boyle, J. Thomson, M. Toussaint, and C. K. I. Williams.
“Using Machine Learning to Focus Iterative Optimization.”
In: CGO. IEEE, 2006.
[5] C. Cummins, P. Petoumenos, M. Steuwer, and H. Leather.
“Autotuning OpenCL Workgroup Size for Stencil Patterns.”
In: ADAPT. 2016.
[6] D. Grewe, Z. Wang, and M. O’Boyle. “Portable Mapping
of Data Parallel Programs to OpenCL for Heterogeneous
Systems.” In: CGO. IEEE, 2013.
[7] A. Magni, C. Dubach, and M. O’Boyle. “Automatic Op-
timization of Thread-Coarsening for Graphics Processors.”
In: PACT. ACM, 2014.
[8] C. Cummins, P. Petoumenos, W. Zang, and H. Leather.
“Synthesizing Benchmarks for Predictive Modeling.” In:
CGO. IEEE, 2017.
[9] M. Namolaru, A. Cohen, G. Fursin, A. Zaks, and A. Freund.
“Practical Aggregation of Semantical Program Properties for
Machine Learning Based Optimization.” In: CASES. 2010.
[10] H. Leather, E. Bonilla, and M. O’Boyle. “Automatic Feature
Generation for Machine Learning Based Optimizing Com-
pilation.” In: TACO 11.1 (2014).
[11] A. Krizhevsky, I. Sutskever, and G. E. Hinton. “ImageNet
Classiﬁcation with Deep Convolutional Neural Networks.”
In: NIPS. 2012.
[12] K. He, X. Zhang, S. Ren, and J. Sun. “Deep Residual
Learning for Image Recognition.” In: CVPR. IEEE, 2016.
[13] H. Lee, Y . Largman, P. Pham, and A. Y . Ng. “Unsupervised
Feature Learning for Audio Classiﬁcation using Convolu-
tional Deep Belief Networks.” In: NIPS. 2009.
[14] M. Allamanis and C. Sutton. “Mining Idioms from Source
Code.” In: FSE. ACM, 2014.
[15] M. Allamanis, E. T. Barr, C. Bird, and C. Sutton. “Learning
Natural Coding Conventions.” In: FSE. ACM, 2014.
[16] J. Yosinski, J. Clune, Y . Bengio, and H. Lipson. “How
Transferable are Features in Deep Neural Networks?” In:
NIPS. 2014.
[17] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean,
M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur,
J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner,
P. Tucker, V . Vasudevan, P. Warden, M. Wicke, Y . Yu, and
X. Zheng. “TensorFlow: A system for large-scale machine
learning.” In: arXiv:1605.08695 (2016).
[18] J. Bergstra, F. Bastien, O. Breuleux, P. Lamblin, R. Pascanu,
O. Delalleau, G. Desjardins, D. Warde-Farley, I. Goodfel-
low, A. Bergeron, and Y . Bengio. “Theano: Deep Learning
on GPUs with Python.” In: BigLearning Workshop. 2011.
[19] M. Allamanis and C. Sutton. “Mining Source Code Reposi-
tories at Massive Scale using Language Modeling.” In:MSR.
2013.
[20] T. Mikolov, K. Chen, G. Corrado, and J. Dean. “Distributed
Representations of Words and Phrases and their Composi-
tionality.” In: NIPS. 2013.
[21] M. Baroni, G. Dinu, and G. Kruszewski. “Don’t Count,
Predict! A Systematic Comparison of Context-Counting vs
. Context-Predicting Semantic Vectors.” In: ACL. 2014.
[22] S. Hochreiter and J. Schmidhuber. “Long Short-Term Mem-
ory.” In: Neural Computation 9.8 (1997).
[23] R. Pacanu, T. Mikolov, and Y . Bengio. “On the Difﬁculties
of Training Recurrent Neural Networks.” In: ICML. 2013.
[24] Z. C. Lipton, J. Berkowitz, and C. Elkan. “A Critical Review
of Recurrent Neural Networks for Sequence Learning.” In:
arXiv:1506.00019 (2015).
[25] Y . Ding, J. Ansel, K. Veeramachaneni, X. Shen, U. O’Reilly,
and S. Amarasinghe. “Autotuning Algorithmic Choice for
Input Sensitivity.” In: PLDI. ACM, 2015.
[26] S. Ioffe and C. Szegedy. “Batch Normalization: Accelerating
Deep Network Training by Reducing Internal Covariate
Shift.” In: arXiv:1502.03167 (2015).
[27] V . Nair and G. E. Hinton. “Rectiﬁed Linear Units Improve
Restricted Boltzmann Machines.” In: ICML. 2010.
[28] D. P. Kingma and J. L. Ba. “Adam: a Method for Stochastic
Optimization.” In: ICLR (2015).
[29] J. Han, M. Kamber, and J. Pei. Data mining: concepts and
techniques. Elsevier, 2011.
[30] S. Seo, G. Jo, and J. Lee. “Performance Characterization
of the NAS Parallel Benchmarks in OpenCL.” In: IISWC.
IEEE, 2011.
[31] S. Che, M. Boyer, J. Meng, D. Tarjan, J. W. Sheaffer,
S. H. Lee, and K. Skadron. “Rodinia: A Benchmark Suite for
Heterogeneous Computing.” In: IISWC. IEEE, Oct. 2009.
[32] J. A. Stratton, C. Rodrigues, I. Sung, N. Obeid, L. Chang,
N. Anssari, G. D. Liu, and W. W. Hwu. “Parboil: A Revised
Benchmark Suite for Scientiﬁc and Commercial Throughput
Computing.” In: Center for Reliable and High-Performance
Computing (2012).
[33] S. Grauer-Gray, L. Xu, R. Searles, S. Ayalasomayajula, and
J. Cavazos. “Auto-tuning a High-Level Language Targeted
to GPU Codes.” In: InPar. 2012.
[34] A. Danalis, G. Marin, C. McCurdy, J. S. Meredith, P. C.
Roth, K. Spafford, V . Tipparaju, and J. S. Vetter. “The
Scalable HeterOgeneous Computing (SHOC) Benchmark
Suite.” In: GPGPU. ACM, 2010.
[35] A. Magni, C. Dubach, and M. O’Boyle. “A Large-Scale
Cross-Architecture Evaluation of Thread-Coarsening.” In:
SC. 2013.
[36] J. Sim, A. Dasgupta, H. Kim, and R. Vuduc. “A Performance
Analysis Framework for Identifying Potential Beneﬁts in
GPGPU Applications.” In: PPoPP. ACM, 2012.
[37] M. D. Zeiler and R. Fergus. “Visualizing and Understanding
Convolutional Networks.” In: ECCV. 2014.
[38] Z. Wang and M. O’Boyle. “Partitioning Streaming Paral-
lelism for Multi-cores: A Machine Learning Based Ap-
proach.” In: PACT. ACM, 2010.
[39] S. Kulkarni and J. Cavazos. “Mitigating the Compiler Opti-
mization Phase-Ordering Problem using Machine Learning.”
In: OOPSLA. ACM, 2012.
[40] S. Muralidharan, A. Roy, M. Hall, M. Garland, and P. Rai.
“Architecture-Adaptive Code Variant Tuning.” In: ASPLOS.
ACM, 2016.
[41] W. F. Ogilvie, P. Petoumenos, Z. Wang, and H. Leather.
“Minimizing the cost of iterative compilation with active
learning.” In: CGO (2017).
[42] J. Ren, L. Gao, and Z. Wang. “Optimise Web Browsing
on Heterogeneous Mobile Platforms: A Machine Learning
Based Approach.” In: INFOCOM. 2017.
[43] C. Cummins, P. Petoumenos, M. Steuwer, and H. Leather.
“Towards Collaborative Performance Tuning of Algorithmic
Skeletons.” In: HLPGPU. 2016.
[44] Y . Jiang, Z. Z. Zhang, K. Tian, F. Mao, M. Gethers, X. Shen,
and Y . Gao. “Exploiting Statistical Correlations for Proactive
Prediction of Program Behaviors.” In: CGO (2010).
[45] Y . Wen, Z. Wang, and M. O’Boyle. “Smart Multi-Task
Scheduling for OpenCL Programs on CPU/GPU Heteroge-
neous Platforms.” In: HiPC. IEEE, 2014.
[46] C. Dubach, T. M. Jones, E. V . Bonilla, G. Fursin, and
M. O’Boyle. “Portable Compiler Optimisation Across Em-
bedded Programs and Microarchitectures using Machine
Learning.” In: MICRO. ACM, 2009.
[47] B. Taylor, V . S. Marco, and Z. Wang. “Adaptive Optimiza-
tion for OpenCL Programs on Embedded Heterogeneous
Systems.” In: LCTES. 2017.
[48] A. Collins, C. Fensch, H. Leather, and M. Cole. “MaSiF:
Machine Learning Guided Auto-tuning of Parallel Skele-
tons.” In: HiPC. IEEE, 2013.
[49] C. Dubach, J. Cavazos, B. Franke, G. Fursin, M. O’Boyle,
and O. Temam. “Fast Compiler Optimisation Evaluation
Using Code-Feature Based Performance Prediction.” In: CF.
ACM, 2007.
[50] P. Ting, C. Tu, P. Chen, Y . Lo, and S. Cheng. “FEAST: An
Automated Feature Selection Framework for Compilation
Tasks.” In: arXiv:1610.09543 (2016).
[51] E. Park, J. Cavazos, and M. A. Alvarez. “Using Graph-
Based Program Characterization for Predictive Modeling.”
In: CGO. IEEE, 2012.
[52] J. Cavazos, C. Dubach, F. Agakov, E. Bonilla, M. O’Boyle,
G. Fursin, and O. Temam. “Automatic Performance Model
Construction for the Fast Software Exploration of New
Hardware Designs.” In: CASES. 2006.
[53] X. Gu, H. Zhang, D. Zhang, and S. Kim. “Deep API
Learning.” In: FSE. ACM, 2016.
[54] Y . Oda, H. Fudaba, G. Neubig, H. Hata, S. Sakti, T. Toda,
and S. Nakamura. “Learning to Generate Pseudo-Code from
Source Code Using Statistical Machine Translation (T).” In:
ASE. IEEE, 2015.
[55] Y . LeCun, Y . Bengio, and G. Hinton. “Deep learning.” In:
Nature 521.7553 (2015).
[56] Q. V . Le, R. Monga, M. Devin, G. Corrado, K. Chen,
M. A. Ranzato, J. Dean, and A. Y . Ng. “Building High-
level Features Using Large Scale Unsupervised Learning.”
In: ICML. 2012.
[57] S. Han, H. Mao, and W. J. Dally. “Deep Compression:
Compressing Deep Neural Networks with Pruning, Trained
Quantization and Huffman Coding.” In: arXiv:1510.00149
(2015).
APPENDIX
ARTIFACT DESCRIPTION
A. Abstract
Our research artifact consists of interactive Jupyter note-
books. The notebooks enable users to replicate all experi-
ments in the paper, evaluate results, and plot ﬁgures.
B. Description
1) Check-list (Artifact Meta Information):
• Run-time environment: Ubuntu Linux and a web browser.
• Hardware: Users with an NVIDIA GPU may enable CUDA
support to speed up computation of experiments.
• Output: Trained neural networks, predictive model evalua-
tions, ﬁgures and tables from the paper.
• Experiment workﬂow: Install and run Jupyter notebook
server; interact with and observe results in web browser.
• Experiment customization: Edit code and parameters in
Jupyter notebooks.
• Publicly available?: Yes, code and data. See:
https://chriscummins.cc/pact17/
2) How Delivered: A publicly available git repository
containing Jupyter notebooks and experimental data.
C. Installation
See https://chriscummins.cc/pact17/ for instructions. The
code directory contains the Jupyter notebooks. Following
the build instructions described in code/README.md, the
full installation process is:
$ ./bootstrap.sh | bash
$ ./configure
$ make
D. Experiment Workﬂow
1) Launch the Jupyter server using the command:
make run.
2) In a web browser, navigate to:
http://localhost:8000.
3) Select a Jupyter notebook to open it.
4) Repeatedly press the play button (tooltip is “run cell,
select below”) to step through each cell of the note-
book.
OR select “Kernel” > “Restart & Run All” from the
menu to run all of the cells in order.
E. Evaluation and Expected Result
Code cells within Jupyter notebooks display their output
inline, and may be compared against the values in the paper.
Expected results are described in text cells.
F . Experiment Customization
The experiments are fully customizable. The Jupyter note-
book can be edited “on the ﬂy”. Simply type your changes
into the cells and re-run them.
Note that some of the code cells depend on the values of
prior cells, so must be executed in sequence. Select “Kernel”
>“Restart & Run All” from the menu to run all of the cells
in order.
G. Notes
For more information about DeepTune, visit:
https://chriscummins.cc/deeptune
For more information about Artifact Evaluation, visit:
http://cTuning.org/ae/submission-20170414.html

Automatic Creation of Tile Size Selection Models ∗
T omofumi Y uki
Colorado State University
yuki@cs.colostate.edu
Lakshminarayanan
Renganarayanan
IBM T .J. W atson Research Center
lrengan@us.ibm.com
Sanjay Rajopadhye
Colorado State University
Sanjay .Rajopadhye@cs.colostate.edu
Charles Anderson
Colorado State University
anderson@cs.colostate.edu
Alexandre E. Eichenberger
IBM T .J. W atson Research Center
alexe@us.ibm.com
Kevin O’Brien
IBM T .J. W atson Research Center
caomhin@us.ibm.com
Abstract
Tiling is a widely used loop transformation for exposing/ex ploiting
parallelism and data locality . Effective use of tiling requ ires selec-
tion and tuning of the tile sizes. This is usually achieved by hand-
crafting tile size selection (TSS) models that characteriz e the per-
formance of the tiled program as a function of tile sizes. The best
tile sizes are selected by either directly using the TSS mode l or
by using the TSS model together with an empirical search. Han d-
crafting accurate TSS models is hard, and adapting them to di f-
ferent architecture/compiler, or even keeping them up-to- date with
respect to the evolution of a single compiler is often just as hard.
Instead of hand-crafting TSS models, can we automatically
learn or create them? In this paper, we show that for a speciﬁc
class of programs fairly accurate TSS models can be automati cally
created by using a combination of simple program features, s yn-
thetic kernels, and standard machine learning techniques. The au-
tomatic TSS model generation scheme can also be directly use d for
adapting the model and/or keeping it up-to-date. W e evaluat e our
scheme on six different architecture-compiler combinatio ns (cho-
sen from three different architectures and four different c ompilers).
The models learned by our method have consistently shown nea r-
optimal performance (within 5% of the optimal on average) ac ross
all architecture-compiler combinations.
Categories and Subject Descriptors D.3.4 [ Programming Lan-
guages]: Processors—Optimization; Compilers
General T erms Performance, Algorithms, Languages
Keywords Tiling, Neural Network, Performance Modeling, Ma-
chine Learning
∗ This material is partly based upon work supported by the Defe nse Ad-
vanced Research Projects Agency under its Agreement No. HR0 011-07-9-
0002
[Copyright notice will appear here once ’preprint’ option i s removed.]
1. Introduction
The compute and data intensive kernels of several important appli-
cations are loops. Tiling [16, 35, 22, 41] restructures loop compu-
tations to exploit parallelism and/or data locality by matc hing the
program characteristics (e.g., locality and parallelism) to those of
the execution environment (e.g., memory hierarchy , regist ers, and
number of processors). Effective use of tiling requires tec hniques
for tile shape/size selection and tiled code generation. In this paper
we focus on the key step of tile size selection (TSS).
TSS is an important step in the effective use of tiling. The
importance is evident from the vast literature available on this
topic, and is also highlighted in the performance chart show n in
Figure 1. There is a 5x and 2.5x difference in performance bet ween
the best and worst tile sizes for Power5 and Opteron, respect ively .
TSS involves the development of a (cost) model, which is used to
characterize and select the best tile sizes for a given combi nation
of program, architecture, and compiler. For an overview of T SS
models, see [31, 25].
TSS solutions can be broadly classiﬁed into two categories,
viz., static model based [21, 35, 9, 15, 7, 34, 25, 42, 12, 33] a nd
model-driven empirical search based [39, 10, 20, 8, 13, 30]. In the
static model based approach, the compiler uses a pre-design ed TSS
model to pick the best tile sizes for a given program-archite cture
pair. In the model-driven empirical search approach, the TS S model
is used to characterize and prune the space of good tile sizes . For
each tile size in the pruned search space, a version of the pro gram
is generated and run on the target architecture, and the tile sizes
with the least execution time is selected. Due to the large sp ace of
valid tile sizes an exhaustive search, without using any TSS model
to prune the space, is often not feasible.
Both the static model and model-driven empirical search ap-
proaches require a well designed TSS model. Constructing a g ood
TSS model is hard. The extensive literature on the TSS proble m is
evidence of the importance as well as the difﬁculty of the pro blem.
The problem of creating accurate TSS models is further exace r-
bated by (i) the complexity of the memory hierarchy in multi- core
processor architectures and (ii) the highly intertwined op timization
phases of a compiler. For example, Y otov et al. [42, 43] show t he
level of detailed understanding of the architecture and com piler op-
timization required to construct effective TSS models.
In addition to the effort involved in creating a TSS model, ad apt-
ing it to a different architecture and/or compiler requires signiﬁcant
effort. Further, keeping a TSS model up-to-date with respec t to the
evolution of optimizations in a single compiler is in itself a signif-
icant task. In fact, the recognition of this difﬁculty in con structing
1 2013/8/28
0 50 100 150 200 250
1 2 3 4 5
Variation in perfomance of tiled code (Power5)
Tile Size (cubic)
Normalized Performance
MMM
TMM
SSYRK
SSYR2K
STRMM
STRSM
5 10 15 20
1.0 1.5 2.0 2.5
Variation in perfomance of tiled code (Opteron)
Tile Size (cubic)
Normalized Performance
MMM
TMM
SSYRK
SSYR2K
STRMM
STRSM
Figure 1. V ariation in execution time of tiled code for six scientiﬁc
kernels. The execution time is normalized to the best (lowes t)
running time of the points shown in the ﬁgure. Note that two
processors show a very different behavior, and scale of x-ax is is
quite different in the two ﬁgures.
and maintaining accurate TSS models led to the wide spread us e of
empirical search techniques. Unfortunately , to be efﬁcien t and fast
enough empirical search techniques also require TSS models to (at
least) prune the search space, and these models themselves a re also
non-trivial to construct and adapt.
In summary , accurate TSS models are needed to select the best
tile sizes and constructing and adapting them are becoming m ore
and more difﬁcult.
Previous approaches to TSS have used hand-crafted TSS mod-
els to either directly select the tile sizes [21, 35, 9, 15, 7, 34, 25, 42]
or as a part of an empirical search to prune the search space [3 9,
10, 20, 8, 13, 30]. There are also TSS methods where hand-craf ted
TSS models are used to deﬁne a space of valid/good tile sizes a nd
then machine learning techniques are used to efﬁciently sea rch the
space for the best tile sizes [38, 23, 11, 29]. As discussed ea rlier,
the hand-crafted models used in these approaches are difﬁcu lt to
create, adapt, and maintain.
Instead of hand-crafting TSS models, can we automatically
learn or create them? If so, we can use the same techniques to a u-
tomatically adapt or keep them up-to-date with respect to ch anges
in architectures and compilers. In this paper, we show , for a spe-
ciﬁc class of programs (3D loop nest with 2D data), that by usi ng
a combination of simple program features, synthetic kernel s and
standard machine learning techniques, highly effective an d accu-
rate TSS models can be learned with little or no human involve -
ment. The two key ideas behind our approach are (i) the use of s ix
simple program features that capture the effects of spatial and tem-
poral locality of tiled programs and (ii) the use of syntheti c and
automatically generated programs to learn the TSS models.
W e consider the problem of selecting tile sizes for a single l evel
of tiling for caches. For validation, we use scientiﬁc compu tation
kernels that are known to beneﬁt from cache tiling. W e report exten-
sive validation of our scheme on three different architectu res (Intel
Core2Duo, AMD Opteron, Power5) and four different compiler s
(
gcc, IBM xlc, PathScale pathcc, and Intel icc). W e show that
fairly accurate TSS models can be automatically created on a ll the
six different architecture-compiler combinations. The ti le sizes pre-
dicted by our machine-crafted models, trained separately f or each
architecture-compiler combination, consistently show ne ar-optimal
performance on a variety of scientiﬁc kernels. The training of the
machine-crafted models involves a couple of days of data col lection
and very little effort to tune the neural network parameters . The re-
sulting TSS model can be directly used by a compiler to comput e
the best tile sizes for a given program, or can be used by an aut o-
tuner to perform a model-driven empirical search.
The paper makes the following contributions:
• W e identify a set of six simple program features that charact er-
ize the spatial and temporal locality beneﬁts of a tiled prog ram.
• W e show that the simple structure of the program features can
be exploited to generate synthetic tiled programs which can be
used for learning the TSS models.
• W e formulate a machine learning scheme which models the op-
timal tile sizes as a continuous function of the program feat ures.
• W e report extensive validation of our technique on six diffe r-
ent compiler-architecture combinations. W e show that very ef-
fective TSS models, which predict the near-optimal tile siz es
across all the six platforms can be automatically learned fo r our
target class of programs.
T o the best of our knowledge, our work is the ﬁrst one to use a
combination of a simple set of features and synthetic progra ms to
automatically create TSS models.
The rest of the sections are organized as follows. Section 2 i n-
troduces loop tiling for caches and the architectural featu res that
affect the performance of tiled programs. Section 3 introdu ces the
program features, predicted outputs, and the use of artiﬁci al neural
networks to learn TSS models. In Section 4, we outline the dif ferent
stages of our scheme and in Section 5 we present experimental eval-
uation. Section 6 compares the performance of the machine-c rafted
TSS models with two well known hand-crafted TSS models from
the literature. Section 7 presents some conclusions and poi nters to
future work.
2. Loop Tiling For Cache Locality
Cache tiling, also called cache blocking, transforms a set o f loops
into another set of loops, which performs the same computati on but
in a different order so that the program has better cache loca lity .
Tiling is used to maximize reuse and avoid costly accesses to lower
levels in the memory hierarchy . For example, consider the lo op
nest in Figure 2, which is the symmetric rank k update (SSYRK)
kernel. If you consider the access to the C matrix, the entire matrix
is accessed in each iteration of the outer-most i loop. If the cache
is large enough to store the entire C matrix, the program will read
the matrix from the main memory when it was ﬁrst used, and then
the remaining accesses only needs to wait for the cache. With the
original code, we need enough cache to hold N2 elements of the
C array , and additional 2 N for the A array to maximize reuse.
However, L1 data caches of modern processors are around 32KB
to 128KB, and are often not enough to store the entire matrix f or
large problem instances.
2 2013/8/28
for (i=0; i<N; i++)
for (j=0; j<N; j++)
for (k=j; k<N; k++)
C[j][k] += A[i][j] * A[i][k];
Figure 2. SSYRK
for (Tj = 0; Tj < N; Tj+=tSize)
for (Tk = Tj; Tk < N; Tk+=tSize)
for (i=0; i<N; i++)
for (j=Tj; j<min(Tj+tsize,N); j++)
for (k=j; k<min(Tk+tsize,N); k++)
C[j][k] += A[i][j] * A[i][k];
Figure 3. Tiled SSYRK
for(i=0; i<N; i++)
for(j=0; j<N; j++)
for(k=0; k<=j; k++)
if (k <= j-1)
B[j][i]=B[j][i]-L[j][k]*B[k][i];
if (k == j)
B[j][i]=B[j][i]/L[j][j];
Figure 4. TRISOL V
Tiling partitions the iteration space and changes the order of
execution so that data will be reused while still performing the same
computation. A possible application of tiling to the SSYRK k ernel
is shown in Figure 3. New loops with indices T j and T k control the
memory requirement by the tile size parameter t Size so that t Size2
elements ﬁt in the available cache.
Tiling has been studied extensively and is an important opti -
mization in highly tuned linear algebra libraries such as BL AS or
A TLAS [40, 43]. The above example only shows the use of tiling to
maximize L1 cache reuse, but tiling can be applied simultane ously
to other memory hierarchies such as registers, and other lev els of
caches. In addition, multiple levels of tiling can be applie d to opti-
mize for multiple levels of the memory system.
2.1 Cache Tiling and Hardware Prefetching
Let us consider a hardware prefetcher that can prefetch subs equent
cache lines to L1 cache when cache lines are accessed sequent ially ,
such as the one in Power5 processor. On such a processor, all t he
references in Figure 2 can be prefetched, because all of them are
along the cache line (assuming row-major layout). With hard ware
prefetching, the untiled code performs slightly better tha n tiled
code with best tile size, since it does not suffer from loop co ntrol
overhead associated with tiling.
However, not all programs have prefetcher-friendly struct ure.
Consider Figure 4. Because the k loop is dependent on the outer
j loop, simple loop permutation cannot make the reference B[k][i]
prefetcher-friendly . Also, multiple computations may be f used in
practice, which may result in a prefetcher-unfriendly code segment.
Again by loop fusion, the total number of references in a loop nest
may increase beyond the number of prefetcher streams availa ble,
which again limits the beneﬁt of hardware prefetching.
It has been shown by Kamil et al. [17] that cache blocking
may not be effective for stencil computations with 2D data. S tencil
computations have uniform accesses that can be easily prefe tched
. Combined with large on-chip memories available on modern
hardware, the level of reuse already achieved without trans forming
the loop is comparable with tiled code. W e therefore exclude stencil
computations from the target class of programs in this paper .
MMM Matrix Matrix Multiplication
TMM Triangular MM ( C = AB)
SSYRK Symmetric Rank K Update
SSYR2K Symmetric Rank 2K Update
STRMM In-place TMM ( B = AB)
STRSM Solve Triangular Matrix ( AX = α B)
TRISOL V Solve Triangles ( Ax = b)
LUD LU Decomposition
SSYMM Symmetric MMM
T able 1. Nine real kernels used for validation
Some of the recent architectures have even better hardware
prefetchers that start prefetching when constant-stride a ccesses are
observed in addition to the unit-stride accesses. With cons tant-
stride prefetcher, references that access by columns (not a long
the cache line) can also be prefetched if the access pattern i s
regular. However, recent Intel processor series, which wer e the
only processors with constant-stride hardware prefetcher s, cannot
prefetch if the stride crosses 4KB page boundaries [1]. Beca use
of this constraint, constant-stride prefetcher in the Inte l processors
cannot do any better than unit-stride prefetchers when the p roblem
instance is large enough so that each row is more than 4KB (512
doubles).
When optimizing for multi-core, tiling is used to expose par -
allelism and improve cache locality [3]. Some loop permutat ions
that would make a loop nest prefetcher-friendly might not be the
best transformation for parallelism. Consider SSYRK in Fig ure 2.
Permuting the loops to make i loop the innermost loop can reduce
the synchronization overhead when this code is parallelize d for a
shared memory environment. If this code were to be paralleli zed
using OpenMP , one would prefer reordering the loops so that t he j
loop is outermost, and mark the outermost loop as the paralle l loop.
The number of synchronization is now reduced from N to 1, com-
pared to inserting the parallelization pragma before the i loop in the
original code (Figure 2). However, the new loop ordering ( j,k,i)
makes the two references to array a are prefetcher-unfriendly , and
tiling could be used for better locality .
2.2 T arget Class of Programs
In this paper, we focus on a class of scientiﬁc computations, such as
linear algebra, which are known to beneﬁt from tiling. Tiled code
generators, such as TLoG [32], Pluto [3], PrimeTile [14], an d D-
tiling [18], are available for these class of programs, whic h make it
easier to use our TSS learning approach. Although there are h ighly
tuned libraries available for common kernels like matrix mu ltipli-
cation, computations that are not covered by the libraries m ay still
come up by trying to use a speciﬁc loop ordering or as a result o f
other transformations, such as fusing multiple kernel comp utations.
The nine kernels we use in this paper are summarized in T able 1 .
Among this class of programs, we consider the programs that
have three dimensional loops with two dimensional data. Man y sci-
entiﬁc kernels, like matrix multiplication, ﬁt in to this su bset of
programs. Also programs with more than three dimensional lo ops
can be still handled in our model by only tiling the inner thre e di-
mensions. W e consider cubic tiles only to reduce data collec tion
time. Allowing all three dimensions to have different tile s izes sig-
niﬁcantly increases the number of possible tile sizes. Our a pproach
can be directly extended to predict rectangular tile sizes b y increas-
ing the data collection time, which will be described later i n more
detail. W e do not consider data padding or copy optimization , since
available code generators do not handle these optimization s.
3 2013/8/28
for (i=0; i<N; i++)
for (j=0; j<N; j++)
for (k=0; k<N; k++)
C[i][j] += A[i][k] * B[k][j];
Figure 5. Matrix Matrix Multiplication
3. Learning TSS Models
In this section we describe the basic components of TSS model
learning; the inputs to and outputs of the model, and the mach ine
learning technique we use.
3.1 Input Program Features
In order to use a model to predict optimal tile sizes for diffe rent pro-
grams, the model needs to be provided with inputs that distin guish
different programs. The inputs to our model are features of t he pro-
grams. Previous methods that use machine learning techniqu es for
compiler optimizations have often used syntactic features such as
the number of operands or the number of loop nests [36, 27, 6]. Af-
ter experimenting with a variety of features that capture th e spatial
and temporal locality effects of loop tiling, we arrived at a set of
six features.
The features are based on the number of references in the in-
nermost statements, classiﬁed into three different types o f refer-
ences. Then each type of reference is further classiﬁed into reads
and writes for a total of six features. The three types of refe r-
ences are non-prefetched references, prefetched references, and ref-
erences that are constant in the innermost loop ( invariant). The in-
variant reference captures those references that are reused for all
the iterations of the innermost loop. The prefetched reference cap-
tures references that enjoy spatial locality given by the pr efetcher,
and non-prefetched references are those that need temporal locality
for good performance. Read and write references are disting uished
because of the possible differences in how they are treated e spe-
cially in multi-core processors where L2 cache is commonly s hared
among the cores.
For example, in matrix multiplication shown in Figure 5, the
reference to array c is write-invariant (WI), because it is written to
the same location by all iterations of the innermost loop. Re ference
to array a is read-prefetched (RP), because the innermost loop
index k is used to index the columns of the array , and such accesses
are prefetched by unit-stride prefetcher. Reference to arr ay b is
read-non-prefetched (RNP), since k is used to index the rows,
and we are assuming row-major layout. These features can eas ily
be extracted by looking at the loop orderings used to referen ce
arrays. The compiler needs to be aware of the type of hardware
prefetcher is used on each architecture to calculate these v alues,
but we believe this is a simple requirement. Detailed inform ation
about the prefetcher is not required, the compiler only need s to
know if unit-stride access is prefetched. With the current h ardware
prefetching technology , even a constant-stride prefetche r works
effectively as an unit-stride prefetcher as discussed earl ier. More
information may become necessary with signiﬁcant advance i n
hardware prefetching.
3.2 Predicted Outputs
There are multiple possible outputs for a TSS model. One poss i-
bility is to model the performance of tiled code itself and se arch
the learned function for tile sizes that optimize performan ce given
a speciﬁc program. However, this approach requires searchi ng the
function after modeling. In the case of analytical models, t he func-
tion may be smooth and the optimal is easy to ﬁnd using some
kind of optimization method. With functions learned by neur al net-
works, the function may not be smooth and optimization meth-
ods can get trapped in one of many local minima. Finding globa l
minima in such functions is itself a separate and difﬁcult pr oblem,
which we would like to avoid.
Another possible output is the optimal tile size itself. Cla ssi-
fying programs to optimal tile sizes allows skipping the ste p of
searching the function and directly gives tile size as an out put.
Classiﬁers can be also learned by neural networks or other ma -
chine learning techniques such as support vector machines. How-
ever, simple classiﬁcation is only capable of classifying p rograms
into predeﬁned set of optimal tile sizes. Thus, it does not su it our
goal of predicting optimal tile sizes of unseen programs, un less we
have enough training data to cover all possible tile sizes, w hich is
unlikely .
W e used a solution between the two ways described above.
W e use the optimal tile size as the output, but we do not learn
classiﬁers. Instead, we formulate the TSS model to be learne d as
a continuous function from the six program features (descri bed
earlier in 3.1) to the optimal tile sizes. The following exam ple
gives an intuitive motivation for formulating the TSS model to
be a continuous function. Consider three programs with iden tical
program features except for number of RNP references. Progr am A
has optimal tile size of 100 with RNP=1, program B has optimal
tile size of 50 with RNP=3. It is reasonable to think that prog ram C
with RNP=2 has optimal tile size somewhere between 50 and 100 ,
since it requires more data per iteration compared to progra m A,
but less compared to B.
3.3 Using ANN to Learn TSS Model
W e used artiﬁcial neural networks (ANN), a supervised learn ing
method, to learn tile size prediction model. Supervised lea rning
methods require pairs of input and desired output, and learn some
function to minimize error between the output of the functio n
and the desired output. The neural network we used is a back-
propagation neural network and was trained with Scaled Conj ugate
Gradient method [26].
W e use multi-layered neural networks [2]. The number of node s
in the output layer is equal to the number of outputs to be prod uced
by the neural network. Each hidden layer can have any number o f
nodes. The inputs to the ANN is given to the ﬁrst hidden layer,
and outputs from each layer is given to the next layer. The out puts
from hidden layers is a function of weighted sum of the inputs to
that layer, where the function is the hyperbolic tangent. Th e output
layer performs weighted sum of the outputs from the last hidd en
layer, but does not apply the hyperbolic tangent function. G iven
output layer weights w, output from the last hidden layer h, desired
output b, and number of training data N, each output layer node
tries to minimize the error calculated using the equation be low .
Σ N
n=1 ((h.w)n − bn )2
The neural network starts with random values as initial weig hts
in each node, and then iteratively updates the weights to min imize
the error. The scaled conjugate gradient method is a type of g ra-
dient method that approximates the second order derivative to al-
low faster convergence to the local minimum, and hence accel erate
training of ANN compared to standard back-propagation with gra-
dient descent.
Models learned using neural networks return real valued num -
bers as optimal tile size. Since tile sizes are integers, we s imply
round the given value to a closest integer and use that as the p re-
dicted optimal tile size.
4 2013/8/28
4. Our Approach
Our approach has the following four different stages
1. Synthetic program generation
2. Data Collection
3. Learning TSS models using ANN
4. Use of ANN based TSS model
Stages 1 through 3 are part of the TSS model creation phase and are
done ofﬂine. Stage 4 represents the use of the learned TSS mod el
and is done on-line during the compilation of a program to sel ect
the tile sizes.
4.1 Synthetic Program Generation
W e need to collect training data to train neural networks. Da ta gath-
ered from real applications or kernels are commonly used as t he
training data for machine learning based modeling. However , us-
ing real applications limits the training data to the real ap plications
available at the time of training. The neural network cannot be ex-
pected to perform well on programs with features that are sig nif-
icantly different from any program in the training data. Wit h real
applications as training data, there is not much control ove r the
range of programs that is covered by the neural network. In ad -
dition, some of the real applications need to be separated ou t from
the training data for validation. Also, if multiple applica tions have
the same program feature, the neural networks may become ove r-
trained to better suit that program feature more than others .
W e use synthetic programs to overcome these limitations. Th e
synthetic programs we use are programs that ﬁt in our class of
interest (three dimensional loops and two dimensional data ), with
statements in the innermost loop that are generated to have t he
speciﬁed number of references for each type. W e exhaustivel y
search for optimal tile sizes of the generated programs to cr eate
the training data set. W e used the open source tiled code gene rator,
TLoG [32], to generate codes with all three dimensions tiled .
With synthetic programs, we have better control over traini ng
data, and the ability to train using a large number of trainin g data
points. W e believe these beneﬁts given by synthetic program s are
one of the main reasons that lead to well performing models.
The use of synthetic programs was only possible because we
have simple program features. If a large number of program fe a-
tures were used, then it becomes infeasible to try a large ran ge of
possible programs with synthetic programs. Even if real pro grams
were used, the coverage of programs that can be represented b y
complex program features is going to be sparse.
W e selected a range of values of program features, and hence
programs, that cover all the kernels we used, but also includ es many
others so that the model is not specialized to just those kern els.
T able 2 shows the range of values we used to bound the feature
space. Column names represent the type of reference, prefet ched
(P), non-prefetched (NP), invariant(I) for read (R) and wri te (W).
These bounds were constructed so that the space is not too lar ge,
but still captures a wide variety of programs. The number of r eads
are usually more than the writes, so we only have a small numbe r of
writes. RNP is always greater than 0, to ensure the program st ays in
the class of interest (at least one reference is not prefetch ed). There
are a total of 2835 program instances in this space with at lea st one
write.
The range of program features to be covered can be changed
to better suit expected programs. If you know that the compil er is
going to see a small subset of the possible feature space befo rehand,
the model can be specialized for that range of programs.
From the bounded feature space, we collected optimal tile si zes
for a number of points in the feature space. T able 3 shows the p oints
used for the training data. W e also exclude from the training data,
RP RNP RI WP WNP WI
Range 0-8 1-5 0-8 0-1 0-1 0-1
T able 2. Bounds on feature space for the model as number of
references of each type
RP RNP RI WP WNP WI
Range 0,2,4,8 1-5 0,2,4,8 0-1 0-1 0-1
T able 3. Data points used for training
programs with features identical to features of real kernel s, so that
real kernels we used to test our model remain unseen during th e
training.
4.2 Data Collection
Collecting training data is time consuming, but can be done w ithout
much human effort. The use of a parameterized tiled code gene rator
helped our data collection by avoiding compilation for each tile size
explored for a program.
Data collection took between 20-40 hours for each compiler-
architecture combination. W e used problem sizes that take a round
10 seconds of execution time for each synthetic program inst ance.
Because the number of references in a program affects its run ning
time, using the same problem sizes would cause the running ti me
of some synthetic program instances to be much larger than 10
seconds. W e used a very simple linear function of number of re f-
erences to adjust the problem size. This adjustment was only for
avoiding long execution times during data collection, and t he accu-
racy of the adjustment was not an issue.
4.3 Learning TSS Models Using ANN
There are many parameters in the training process, includin g the
range of programs to target, the range of training data, and p arame-
ters of the neural network. The former two can be made larger a nd
larger if time permits, since we want the model to cover a larg er
range of programs, and having more training data always help s
learning.
The parameters of the neural network are not as simple. There
are many parameters for a neural network, the important ones be-
ing the number of hidden layers, the number of nodes in each la yer,
how initial weights are selected, and the termination condi tion. In
addition, we can train multiple neural networks individual ly for
more stable output. A veraging the output of multiple neural net-
works helps stabilize the output, because neural networks l earned
are heavily inﬂuenced by the initial weights. The number of n eural
networks to be trained for averaging in the end is also anothe r pa-
rameter. W e do not try to optimize the neural network paramet ers.
Instead we manually tune the neural network parameters base d on
our intuition and testing on small data sets. W e hope to devel op an
automated approach to optimize neural network parameters i n the
future.
W e used three-layered (two hidden layers) neural networks w ith
30 hidden nodes per hidden layer, and weights randomly initi alized
to values between 1 and -1. The termination condition was sli ghtly
different for each architecture-compiler combination bas ed on how
easy it was to ﬁt the training data for a particular combinati on.
These parameters are picked by trying out multiple combinat ions
of parameters and looking at the rate of convergence and root mean
square errors, a measure of how far the predicted outputs are away
from the desired outputs.
It took about two hours of initial tuning to get a good basic
design of the neural network, and then the SAME basic neural n et-
work conﬁguration was applied to all architecture-compile r com-
5 2013/8/28
binations. Note that this design time is a one-time effort. A fter the
basic design, for each architecture-compiler combination , a slight
tuning of the termination condition was needed. W e tuned the ter-
mination condition using the collected training data, by lo oking at
how the error between predicted and desired output changes b e-
tween iterations. Where the improvement starts to ﬂatten co uld be
different for each training data, and we selected a conditio n to ter-
minate when it starts to ﬂatten. This tuning is pretty standa rd and
can be automated.
With the above conﬁguration, training of each neural networ k
completes within a minute for a total of at most ﬁve minutes fo r ﬁve
different neural networks trained for each architecture-c ompiler
combination.
4.4 Use of ANN Based TSS Model
Once we have the trained TSS model, it can be used as a part of
the compiler to predict the optimal tile sizes, or used as a pa rt of
a model-driven search method to ﬁnd the optimal tile sizes. T he
ﬁrst step is to extract the program features, which should no t take
much time due to its simplicity . Then the six program feature s
are used as an input to the learned model to produce the output ,
which can be directly used as the tile size selected for that p rogram.
When the model is used as a part of a model-driven search metho d,
the tile sizes in the neighborhood of the predicted tile size s can
be empirically tested for the best performance. It is also po ssible
to alter the neural network to output expected performance o f a
program with a given tile size, which may be a better model for
mode-driven search.
The only on-line cost associated with the use of our model in a
compiler is the use of the neural network and extraction of th e six
program features. The use of neural networks is computation ally
close to two matrix-vector products of size 31x6, which is tr ivial
with modern compute power.
4.5 Extension to Rectangular Tiles
Our approach can be directly extended to predict rectangula r tile
sizes as well. The ﬂow described above does not change at all. The
only pieces that need to be changed is the outputs of the neura l
network, and the data collection. The outputs from the neura l net-
work needs to be increased from one to two or three, depending on
whether two or three dimensions are tiled, because tile size in each
dimension can be now different. The data collection phase ne eds to
be adjusted accordingly to ﬁnd the optimal tile sizes includ ing rect-
angular tiles. Because neural networks naturally handle mo re than
one outputs, no other change is required.
5. Experimental V alidation of Machine-Crafted
Models
W e have used our approach to learn tile size selection models on
the six architecture-compiler combinations summarized in T able 4.
W e used the nine kernels previously summarized in 1 with prob lem
sizes adjusted for each architecture to run for about a minut e. W e
chose a longer target execution time compared to the 10 secon ds of
the synthetic programs so that the problem sizes used were si gniﬁ-
cantly different from the problem sizes used during validat ion. The
problem sizes used for real kernels were around 3000, where a s the
problem sizes used during training was around 1000.
Feature extraction was done manually , but it can be easily
done in a compiler. W e used the same set of program features
on all architectures, because even the constant-stride pre fetcher on
Core2Duo effectively worked only as a unit-stride prefetch er due
to its constraint on access distance.
MMM TMM SSYRK SSYR2K STRMM STRSM LUD SSYMM TRISOL V
Execution time using machine−crafted models, normalized to the optimal0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6
Opteron/PSC
Opteron/GCC
Power5/XLC
Power5/GCC
Core2Duo/ICC
Core2Duo/GCC
Figure 6. Execution time normalized to the true optimal of kernels
with tile sizes selected by machine-crafted models for each combi-
nation of architecture-compilers.
5.1 Performance
For each architecture-compiler combination, we compare th e ex-
ecution time of kernels with the true optimal (found by exhau s-
tive search) and the predicted optimal tile sizes. Figure 6 s hows the
normalized execution time of nine kernels with tile sizes se lected
by machine-crafted models learned for each architecture-c ompiler
combination. The performance given by tile sizes predicted by our
machine-crafted models is consistently near the optimal. T he per-
formance is only 20% off the optimal even in the worst case, wh ich
is signiﬁcantly small compared to the slowdown one would get with
a poor tile size (recall Figure 1). This supports our claim th at well
performing TSS models can be learned from simple features an d
training data collected through synthetic programs for dif ferent ar-
chitecture and compilers. Although we do not have results fo r dif-
ferent versions of the same compiler, we have shown results f or dif-
ferent compilers, which is just as hard if not harder. This in dicates
that the learned TSS models can be easily updated (re-traine d) with
respect to the evolution of a single compiler.
5.2 Performance with Local Search
The focus of this paper is on learning TSS models that can pred ict
good tile sizes for different architecture and compilers wi thout
much human effort. In this section we quantify its potential when
used as a part of model-driven empirical search approaches. W e
show how close the predicted tile sizes is to the optimal by si mply
looking at neighboring tile sizes within a certain distance . Here,
distance is the difference between the predicted tile size a nd the tile
sizes actually explored.
T able 5 shows the mean slowdown over all nine kernels when
the best tile size within a certain distance of the predicted tile size
were used. Only by looking at immediate neighbors of distanc e ten,
the model can give the exact optimal performance for all kern els
on Opteron, and for eight out of the nine kernels for Power5.
The performance improvement on Core2Duo is relatively smal l
compared to other architectures, but notable improvement c an be
observed.
W e think the cause of relatively small improvement on Core2D uo
is due to the very high set-associativity (8-way). There is a very
wide range of good tile sizes, and the automated training dat a col-
Architecture Compilers L1 Cache Options
Opteron PSC, GCC 64KB 2-way -O3, -O3
Power5 XLC, GCC 32KB 4-way -O5, -O3
Core2Duo ICC, GCC 32KB 8-way -O3, -O3
T able 4. Architecture and compilers used
6 2013/8/28
Predicted Distance 10 Distance 20
Opteron/PSC 4.3% 0% 0%
Opteron/GCC 6.3% 0% 0%
Power5/XLC 4.6% 0.4% 0%
Power5/GCC 1.7% 0.2% 0%
Core2Duo/ICC 7.8% 4.0% 1.9%
Core2Duo/GCC 5.1% 4.0% 1.9%
T able 5. Mean slowdown over all kernels when the best tile size
within some distance from the predicted tile size is used.
lection is likely to have more noise compared to others. The o pti-
mal on a ﬂat surface can be easily affected by small noises fro m
the operating system or other environment not necessarily c on-
nected to the program being executed. W e believe this noise c an
be suppressed by increasing the problem size and/or running each
instance of the synthetic program a number of times and takin g the
minimum (or mean) execution time
Another reason is that with GCC on Core2Duo, two kernels,
MMM and TMM, are showing optimal performance with a tile size
that is far from the tile size predicted by our model. W e suspe ct that
this is due to some optimization applied by the compiler only for
the standard matrix multiplication, since by changing the c ompiler
option from -O3 to -O2, the tile size predicted by our model sh ows
identical performance with the true optimal.
How good are the models created by our approach, compared to
randomly picking points in the search space, when local sear ch is
performed is a question that the readers may have. It has also been
shown by Kisuki et al. [19] that random search takes large num ber
of iterations (100+) to reach within 5% of the optimal for mos t
of the benchmarks they used, which included dense linear alg ebra
computations as well.
Even a naive local search around the tile sizes predicted by
the machine-crafted models show signiﬁcant improvements. This
demonstrates its potential to perform better when combined with
the more sophisticated search methods proposed in the liter ature.
6. Comparison with Hand-Crafted Models
Many static models have been previously developed to maximi ze
performance of tiled code, and those models are analyzed in d etail
by Hsu and Kremer [15]. W e have taken two models that were
reported to perform well [15], and another model by Y otov et.
al [42] used in A TLAS for comparison. First, we brieﬂy descri be
the logic behind the three models, LRW [21], EUC [9] and the
model-driven A TLAS model [42]. Then these models are evalua ted
in the same manner as in the previous section. The goal of this
section is to show that our approach can generate models that are at
least as good as the previous models for all architecture-co mpiler
combinations.
LRW [21] is an analytical model developed by closely studyin g
the performance of tiled matrix multiplication. It chooses square
tiles for a given program using an estimate of cache misses ba sed
on memory footprint and self-interference. EUC [9] is also a n an-
alytical model with similar considerations used as in LRW , b ut it
predicts rectangular tile sizes instead of square. EUC take s cross-
interference into account as well as self-interference. Bo th of these
models take problem sizes as an input, where as our model does
not. Y otov et al. [42] uses multiple models to compute many pa -
rameters including tile sizes for cache tiling with cubic ti les. Their
model for tile size selection assumes fully associative cac hes and
does not consider conﬂict misses. W e would like to clarify th at
when we refer to Y otov et al. in the following, we did not use th e
A TLAS auto-tuned kernels. W e only used the analytical model for
predicting square tile sizes for cache locality in [42].
Machine-Crafted LRW EUC Y otov et al.
Opteron/PSC 4% 178% 217% 169%
Opteron/gcc 6% 100% 147% 139%
Power5/XLC 5% 168% 268% 9%
Power5/gcc 2% 77% 133% 3%
Core2Duo/ICC 8% 6% 246% 5%
Core2Duo/gcc 5% 3% 128% 4%
T able 6. Mean of slowdowns with tile sizes predicted by each
model over all nine kernels on each architecture-compiler c ombi-
nation
6.1 T ailoring Hand-Crafted Models to Modern Architectures
W e made a small but necessary modiﬁcation to all the hand-cra fted
models to adapt to the current architecture. Since some of th e hand-
crafted models were developed when hardware prefetcher was not
commonly available, they treat all references as non-prefe tched.
However, it is obvious that prefetched references do not hav e to
stay in the cache, and they can be excluded when calculating t he tile
size so that the cache is utilized well. Because it is straigh t-forward
and it would not take much effort to modify the model to take th e
prefetching into account, we modiﬁed all of the models so tha t
prefetched references are excluded from calculation. Furt her, for
programs that has more than one references that is not prefet ched,
we give smaller cache sizes to the model. Also, in [21], they b rieﬂy
mention extension of their algorithm to set associative cac hes, we
have used their modiﬁcation so that larger tile sizes are sel ected
when compared to assuming direct mapped cache.
6.2 Performance of Hand-Crafted Models
Figure 7 shows the normalized execution time of nine kernels us-
ing tile sizes given by hand-crafted static models. The same prob-
lem sizes used for measuring execution time using machine-c rafted
models were used. Although the hand-crafted models have pre -
dicted near optimal tile sizes for some of the kernels, the pe rfor-
mance is not as consistent as what was observed with our machi ne-
crafted models. LRW and Y otov et al. model performs relative ly
worse on Opteron compared to the other architectures. Opter on
has smaller set associativity , which seems to make the perfo rmance
more sensitive to small changes in tile size, leading to grea ter per-
formance hit. LRW does perform more than 2x slower on Opteron
with matrix multiplication that was used to develop the mode l.
EUC was unable to give good tile sizes for some kernels across
all architecture-compiler combinations. It is interestin g to note that
EUC had predicted a tile size that performs better than the op timal
cubic tile found for SSYR2K on Opteron with PSC. The optimal
cubic tile size found for SSYR2K was 6x6x6, but it was not the
optimal when rectangular tiles were considered. EUC was abl e to
ﬁnd a very thin tile that has better memory utilization for th is case.
T able 6 summarizes the effectiveness of each model by showin g
the percentage of slowdown when compared to the optimal usin g
cubic tile sizes. LRW and Y otov et al. model show comparable p er-
formance on some combinations, but overall the machine-cra fted
model provides consistently near-optimal performance acr oss all
architectures. W e believe the reason for LRW and Y otov et al.
model showing comparable performance on Core2Duo and Power 5
is also due to the fact that these processors have very wide ra nges of
good tile sizes, as previously discussed in Section 5.2. In a ddition
relatively high set associativity of these processors are c loser to the
assumption of Y otov et al. that caches are fully associative .
7 2013/8/28
MMM TMM SSYRK SSYR2K STRMM STRSM LUD SSYMM TRISOLV
Execution time using LRW, normalized to the optimal
0 1 2 3 4 5 6
Opteron/PSC
Opteron/GCC
Power5/XLC
Power5/GCC
Core2Duo/ICC
Core2Duo/GCC
MMM TMM SSYRK SSYR2K STRMM STRSM LUD SSYMM TRISOLV
Execution time using EUC, normalized to the optimal
0 2 4 6 8
Opteron/PSC
Opteron/GCC
Power5/XLC
Power5/GCC
Core2Duo/ICC
Core2Duo/GCC
MMM TMM SSYRK SSYR2K STRMM STRSM LUD SSYMM TRISOLV
Execution time using the model by Yotov et al. normalized to the true optimal0 1 2 3 4 5
Opteron/PSC
Opteron/GCC
Power5/XLC
Power5/GCC
Core2Duo/ICC
Core2Duo/GCC
Figure 7. Execution time of kernels with tile sizes selected
by hand-crafted models for each combination of architectur e-
compilers, normalized to the optimal.
7. Related W ork
Many analytical models have been proposed in the past for TSS [21,
35, 9, 15, 7, 34, 25, 42, 12, 33]. These models are constructed by
carefully observing the performance of a small set of kernel s and
modeling the performance using detailed hardware and softw are
characteristics. Although developing analytical models c an give
greater insight into how the hardware and software interact s, the
cost of development is quite high. Our work focuses on creati ng
good tile size selection models with little human effort.
Another class of tile size selection techniques is the model -
driven empirical search methods [39, 10, 20, 8, 13, 30, 19, 38 , 29].
Empirical tuning with global search may be a feasible soluti on
for optimizing libraries of commonly used kernels [39], but is
not feasible for optimization during compilation. Model-d riven ap-
proaches share the same motivation of achieving performanc e close
to what can be obtained by global empirical search, but with l ess
overhead. Replacing the hand-crafted models currently use d in
these approaches is a possible application of our work.
Recently , machine learning techniques have been successfu lly
used in compiler optimization. Many of the applications wer e to-
ward deriving models and heuristics to accurately predict t he be-
havior of modern complex architectures. The wide range of ap pli-
cations include branch prediction [4], instruction schedu ling within
basic blocks [28, 24], and deciding if certain optimization should
be applied [5, 6, 37, 27]. Some of the hand-crafted tile size s e-
lection approaches have also used some form of machine learn ing
methods [38, 23, 11, 29, 36]. W e use machine learning techniq ues
to automatically learn TSS models.
8. Conclusions and Future W ork
Tile size selection is an important step in the proﬁtable use of loop
tiling. Hand-crafting effective TSS models is hard and adap ting or
maintaining them is often harder. W e have shown that highly e ffec-
tive TSS models can be automatically created using a small se t of
program features, synthetic programs and standard machine learn-
ing techniques. W e have shown that the machine-crafted TSS m od-
els consistently predict near-optimal (on the average, wit hin 5% of
optimal) tile sizes across six different compiler-archite cture com-
binations. W e have also shown that, a naive search within a sm all
neighborhood of the predicted tile sizes can ﬁnd the true opt imal
tile sizes in some cases, and in other cases ﬁnd tile sizes tha t are
very close to the optimal. This clearly indicates the strong potential
of machine-crafted TSS models in a model-driven empirical s earch
scheme.
Several directions of future work are promising. The propos ed
approach can be directly extended to construct TSS models fo r
multiple levels (cache and register) tiling. W e have descri bed how
our approach can be extended to rectangular tile sizes with l ittle
change, which is also a possible direct extension. Another d irect
extension is to construct TSS models where tiling is used to e xpose
coarse-grain parallelism [3]. Another promising directio n is the use
of machine-crafted TSS models together with sophisticated search
techniques to develop an efﬁcient model-driven empirical s earch
technique for use in auto-tuners. Our approach in itself is q uite
ﬂexible and with appropriate interpretation of program fea tures, it
can be extended to other class of programs. Extending to prog rams
with irregular array references is another interesting dir ection.
There are some possible future work on the neural network
side of our approach as well. As mentioned previously , autom atic
selection of neural network parameters is the only part that is not
automated in our approach. There are well known approaches f or
selecting neural network structure, which may be applied to fully
automate our approach. Our model cannot provide any insight
about the underlying architecture, where the process of dev eloping
analytical models by hand tend to increase the understandin g of the
underlying architecture and programs as a sub-product. It w ould
be useful if we could extract some insight from the trained ne ural
network.
References
[1] Intel 64 and IA-32 Architectures Optimization Reference Manual.
[2] C.M. Bishop et al. P attern recognition and machine learning .
Springer New Y ork:, 2006.
[3] Uday Bondhugula, Albert Hartono, J. Ramanujam, and P . Sa dayap-
pan. A practical automatic polyhedral program optimizatio n system.
In ACM SIGPLAN Conference on Programming Language Design
and Implementation (PLDI), June 2008.
[4] Brad Calder, Dirk Grunwald, Michael Jones, Donald Linds ay , James
Martin, Michael Mozer, and Benjamin Zoren. Evidence-based static
branch prediction using machine learning. ACM Transactions on
Programming Languages and Systems, 19(1):188–222, January 1997.
[5] J. Cavazos and J.E.B. Moss. Inducing heuristics to decid e whether to
schedule. In Proceedings of the ACM SIGPLAN 2004 conference on
Programming language design and implementation, pages 183–194,
2004.
[6] J. Cavazos and M.F .P . O’Boyle. Method-speciﬁc dynamic c ompila-
tion using logistic regression. In Proceedings of the 21st annual ACM
8 2013/8/28
SIGPLAN conference on Object-oriented programming langua ges,
systems, and applications, pages 229–240, 2006.
[7] Jacqueline Chame and Sungdo Moon. A tile selection algor ithm
for data locality and cache interference. In 1999 ACM International
Conference on Supercomputing, pages 492–499. ACM Press, 1999.
[8] Chun Chen, Jacqueline Chame, and Mary Hall. Combining mo dels
and guided empirical search to optimize for multiple levels of the
memory hierarchy . In CGO ’05: Proceedings of the international
symposium on Code generation and optimization , pages 111–122,
W ashington, DC, USA, 2005. IEEE Computer Society .
[9] S. Coleman and K.S. McKinley . Tile size selection using c ache orga-
nization and data layout. In Proceedings of the ACM SIGPLAN 1995
conference on Programming language design and implementat ion,
pages 279–290. ACM New Y ork, NY , USA, 1995.
[10] J. Demmel, J. Dongarra, V . Eijkhout, E. Fuentes, A. Peti tet,
R. V uduc, R.C. Whaley , and K. Y elick. Self-Adapting Linear A lgebra
Algorithms and Software. In Proceedings of the IEEE , 93(2):293,
2005.
[11] Arkady Epshteyn, Mar´ ıa Jes ´ us Garzar´ an, Gerald DeJo ng, David A.
Padua, Gang Ren, Xiaoming Li, Kamen Y otov , and Keshav Pingal i.
Analytic models and empirical search: A hybrid approach to c ode
optimization. In Proceedings of the International W orkshop on
Languages and Compilers for P arallel Computing, pages 259–273,
2005.
[12] K. Esseghir . Improving data locality for caches. Maste r’ s thesis, Rice
University , 1993.
[13] Basilio B. Fraguela, M. G. Carmueja, and Diego Andrade. Optimal
tile size selection guided by analytical models. In P ARCO, pages
565–572, 2005.
[14] A. Hartono, M.M. Baskaran, C. Bastoul, A. Cohen, S. Kris hnamoor-
thy , B. Norris, J. Ramanujam, and P . Sadayappan. Parametric multi-
level tiling of imperfectly nested loops. In Proceedings of the 23rd
international conference on Conference on Supercomputing , pages
147–157. ACM New Y ork, NY , USA, 2009.
[15] Chung-Hsing Hsu and Ulrich Kremer . A quantitative anal ysis of tile
size selection algorithms. J. Supercomput., 27(3):279–294, 2004.
[16] F . Irigoin and R. Triolet. Supernode partitioning. In 15th ACM
Symposium on Principles of Programming Languages , pages 319–
328. ACM, Jan 1988.
[17] Shoaib Kamil, Parry Husbands, Leonid Oliker, John Shal f, and
Katherine Y elick. Impact of modern memory subsystems on cac he
optimizations for stencil computations. In Proceedings of the
W orkshop on Memory System P erformance, pages 36–43, New Y ork,
NY , USA, 2005. ACM Press.
[18] DaeGon Kim and Sanjay Rajopadhye. Efﬁcient tiled loop g eneration:
D-tiling. In The 22nd International W orkshop on Languages and
Compilers for P arallel Computing, 2009.
[19] T . Kisuki, P .M.W . Knijnenburg, and MFP O’ Boyle. Combin ed
selection of tile sizes and unroll factors using iterative c ompilation.
In Proceedings of the 2000 International Conference on P arall el
Architectures and Compilation T echniques, page 237. Citeseer, 2000.
[20] P . M. W . Knijnenburg, T . Kisuki, K. Gallivan, and M. F . P . O’Boyle.
The effect of cache models on iterative compilation for comb ined
tiling and unrolling. Concurr . Comput. : Pract. Exper ., 16(2-3):247–
270, 2004.
[21] M.D. Lam, E.E. Rothberg, and M.E. W olf. The cache perfor mance
and optimizations of blocked algorithms. Proceedings of the 4th
international conference on architectural support for pro gramming
languages and operating systems, 25:63–74, 1991.
[22] Monica S. Lam and Michael E. W olf. A data locality optimi zing
algorithm (with retrospective). In Best of PLDI , pages 442–459,
1991.
[23] Xiaoming Li and Mar´ ıa Jes ´ us Garzar´ an. Optimizing ma trix
multiplication with a classiﬁer learning system. W orkshop on
Languages and Compilers for P arallel Computing, pages 121–135,
2005.
[24] A. McGovern, E. Moss, and A. Barto. Scheduling straight -line code
using reinforcement learning and rollouts. (UM-CS-1999-0 23), ,
1999.
[25] N. Mitchell, N. Hogstedt, L. Carter, and J. Ferrante. Qu antifying
the multi-level nature of tiling interactions. International Journal of
P arallel Programming, 26(6):641–670, 1998.
[26] Martin F . Møller . A scaled conjugate gradient algorith m for fast
supervised learning. Neural Networks, 6:525–533, 1993.
[27] A. Monsifrot, F . Bodin, and R. Quiniou. A machine learni ng approach
to automatic production of compiler heuristics. Lecture notes in
computer science, pages 41–50, 2002.
[28] Eliot Moss, Paul Utgoff, John Cavazos, Doina Precup, Da rko
Stefanovic, Carla Brodley , and David Scheeff. Learning to s chedule
straight-line code. In Proceedings of Neural Information Processing
Symposium, pages 929–935. MIT Press, 1997.
[29] Saeed Parsa and Shahriar Lotﬁ. A new genetic algorithm f or loop
tiling. The Journal of Supercomputing, 37(3):249–269, 2006.
[30] Apan Qasem and Ken Kennedy . Proﬁtable loop fusion and ti ling
using model-driven empirical search. In ICS ’06: Proceedings of
the 20th annual international conference on Supercomputin g, pages
249–258, New Y ork, NY , USA, 2006. ACM.
[31] Lakshminarayanan Renganarayana and Sanjay Rajopadhy e. Posi-
tivity , posynomials and tile size selection. In SC ’08: Proceedings
of the 2008 ACM/IEEE conference on Supercomputing, pages 1–12,
Piscataway , NJ, USA, 2008. IEEE Press.
[32] Lakshminarayanan Renganarayanan, DaeGon Kim, Sanjay Rajopad-
hye, and Michelle Mills Strout. Parameterized tiled loops f or free.
In PLDI ’07: Proceedings of the 2007 ACM SIGPLAN conference on
Programming language design and implementation, pages 405–414,
New Y ork, NY , USA, 2007. ACM.
[33] Gabriel Rivera and Chau wen Tseng. A comparison of compi ler tiling
algorithms. In Proceedings of the 8th International Conference on
Compiler Construction (CC’99 , pages 168–182, 1999.
[34] V . Sarkar, N. Megiddo, I.B.M.T .J.W .R. Center, and Y . He ights. An
analytical model for loop tiling and its solution. P erformance Analysis
of Systems and Software, 2000. ISP ASS. 2000 IEEE Internatio nal
Symposium on, pages 146–153, 2000.
[35] R. Schreiber and J. Dongarra. Automatic blocking of nes ted loops.
T echnical Report 90.38, RIACS, NASA Ames Research Center, A ug
1990.
[36] M. Stephenson and S. Amarasinghe. Predicting unroll fa ctors using
supervised classiﬁcation. In Proceedings of International Symposium
on Code Generation and Optimization (CGO), pages 123–134, 2005.
[37] Mark Stephenson, Saman Amarasinghe, Martin Martin, an d Una-
May O’Reilly . Meta optimization: Improving compiler heuri stics
with machine learning. In Proceedings of the ACM SIGPLAN ’03
Conference on Programming Language Design and Implementation,
pages 77–90. ACM Press, 2002.
[38] Xavier V era, Jaume Abella, Antonio Gonz´ alez, and Jose p Llosa.
Optimizing program locality through cmes and gas. In P ACT
’03: Proceedings of the 12th International Conference on P a rallel
Architectures and Compilation T echniques, page 68, W ashington,
DC, USA, 2003. IEEE Computer Society .
[39] R. Clint Whaley and Jack J. Dongarra. Automatically tun ed linear
algebra software. In Proceedings of the 1998 ACM/IEEE conference
on Supercomputing (CDROM), pages 1–27. IEEE Computer Society ,
1998.
[40] R. Clint Whaley and Antoine Petitet. Minimizing develo pment
and maintenance costs in supporting persistently optimize d BLAS.
Software: Practice and Experience, 35(2):101–121, February 2005.
9 2013/8/28
[41] Jingling Xue. Loop Tiling F or P arallelism. Kluwer Academic
Publishers, 2000.
[42] K. Y otov , Xiaoming Li, Gang Ren, M. J. S. Garzaran, D. Pad ua,
K. Pingali, and P . Stodghill. Is search really necessary to g enerate
high-performance BLAS? In Proceedings of the IEEE, 93:358–386,
2005.
[43] Kamen Y otov , Keshav Pingali, and Paul Stodghill. Think globally ,
search locally . In ICS ’05: Proceedings of the 19th annual
international conference on Supercomputing , pages 141–150, New
Y ork, NY , USA, 2005. ACM.
10 2013/8/28

Published as a conference paper at ICLR 2020
LAMBDA NET: P ROBABILISTIC TYPE INFERENCE
USING GRAPH NEURAL NETWORKS
Jiayi Wei, Maruth Goyal, Greg Durrett, Isil Dillig
Department of Computer Science
University of Texas at Austin
{jiayi, maruth, gdurrett, isil}@cs.utexas.edu
ABSTRACT
As gradual typing becomes increasingly popular in languages like Python and
TypeScript, there is a growing need to infer type annotations automatically. While
type annotations help with tasks like code completion and static error catching,
these annotations cannot be fully determined by compilers and are tedious to an-
notate by hand. This paper proposes a probabilistic type inference scheme for
TypeScript based on a graph neural network. Our approach ﬁrst uses lightweight
source code analysis to generate a program abstraction called a type dependency
graph, which links type variables with logical constraints as well as name and
usage information. Given this program abstraction, we then use a graph neural
network to propagate information between related type variables and eventually
make type predictions. Our neural architecture can predict both standard types,
like number or string, as well as user-deﬁned types that have not been en-
countered during training. Our experimental results show that our approach out-
performs prior work in this space by14% (absolute) on library types, while having
the ability to make type predictions that are out of scope for existing techniques.
1 I NTRODUCTION
Dynamically typed languages like Python, Ruby, and Javascript have gained enormous popularity
over the last decade, yet their lack of a static type system comes with certain disadvantages in terms
of maintainability (Hanenberg et al., 2013), the ability to catch errors at compile time, and code
completion support (Gao et al., 2017). Gradual typing can address these shortcomings: program
variables have optional type annotations so that the type system can perform static type checking
whenever possible (Siek & Taha, 2007; Chung et al., 2018). Support for gradual typing now exists in
many popular programming languages (Bierman et al., 2014; Vitousek et al., 2014), but due to their
heavy use of dynamic language constructs and the absence of principal types (Ancona & Zucca,
2004), compilers cannot perform type inference using standard algorithms from the programming
languages community (Bierman et al., 2014; Traytel et al., 2011; Pierce & Turner, 2000), and man-
ually adding type annotations to existing codebases is a tedious and error-prone task. As a result,
legacy programs in these languages do not reap all the beneﬁts of gradual typing.
To reduce the human effort involved in transitioning from untyped to statically typed code, this
work focuses on a learning-based approach to automatically inferring likely type annotations for un-
typed (or partially typed) codebases. Speciﬁcally, we target TypeScript, a gradually-typed variant of
Javascript for which plenty of training data is available in terms of type-annotated programs. While
there has been some prior work on inferring type annotations for TypeScript using machine learning
(Hellendoorn et al., 2018; Raychev et al., 2015), prior work in this space has several shortcomings.
First, inference is restricted to a ﬁnite dictionary of types that have been observed during train-
ing time—i.e., they cannot predict any user-deﬁned data types. Second, even without considering
user-deﬁned types, the accuracy of these systems is relatively low, with the current state-of-the-
art achieving 56.9% accuracy for primitive/library types (Hellendoorn et al., 2018). Finally, these
techniques can produce inconsistent results in that they may predict different types for different
token-level occurrences of the same variable.
1
arXiv:2005.02161v1  [cs.PL]  29 Apr 2020
Published as a conference paper at ICLR 2020
Figure 1: A motivating example: Given an unannotated version of this TypeScript program, a
traditional rule-based type inference algorithm cannot soundly deduce the true type annotations
(shown in green).
In this paper, we propose a new probabilistic type inference algorithm for TypeScript to address these
shortcomings using a graph neural network architecture (GNN) (Veli ˇckovi´c et al., 2018; Li et al.,
2016; Mou et al., 2016). Our method uses lightweight source code analysis to transform the program
into a new representation called a type dependency graph, where nodes represent type variables and
labeled hyperedges encode relationships between them. In addition to expressing logical constraints
(e.g., subtyping relations) as in traditional type inference, a type dependency graph also incorporates
contextual hints involving naming and variable usage.
Given such a type dependency graph, our approach uses a GNN to compute a vector embedding
for each type variable and then performs type prediction using a pointer-network-like architec-
ture (Vinyals et al., 2015). The graph neural network itself requires handling a variety of hyper-
edge types—some with variable numbers of arguments—for which we deﬁne appropriate graph
propagation operators. Our prediction layer compares the vector embedding of a type variable with
vector representations of candidate types, allowing us to ﬂexibly handle user-deﬁned types that have
not been observed during training. Moreover, our model predicts consistent type assignments by
construction because it makes variable-level rather than token-level predictions.
We implemented our new architecture as a tool called LAMBDA NET and evaluated its performance
on real-world TypeScript projects from Github. When only predicting library types, L AMBDA NET
has a top1 accuracy of 75.6%, achieving a signiﬁcant improvement over DeepTyper ( 61.5%). In
terms of overall accuracy (including user-deﬁned types), LAMBDA NET achieves a top1 accuracy of
around 64.2%, which is 55.2% (absolute) higher than the TypeScript compiler.
Contributions. This paper makes the following contributions: (1) We propose a probabilistic type
inference algorithm for TypeScript that uses deep learning to make predictions from the type de-
pendency graph representation of the program. (2) We describe a technique for computing vector
embeddings of type variables using GNNs and propose a pointer-network-like method to predict
user-deﬁned types. (3) We experimentally evaluate our approach on hundreds of real-world Type-
Script projects and show that our method signiﬁcantly improves upon prior work.
2 M OTIVATING EXAMPLE AND PROBLEM SETTING
Figure 1 shows a (type-annotated) TypeScript program. Our goal in this work is to infer the types
shown in the ﬁgure, given an unannotated version of this code. We now justify various aspects of
our solution using this example.
Typing constraints. The use of certain functions/operators in Figure 1 imposes hard constraints on
the types that can be assigned to program variables. For example, in the forward function, vari-
ables x, y must be assigned a type that supports a concat operation; hence, x, y could have types
like string, array, or Tensor, but not, for example, boolean. This observation motivates us
to incorporate typing constraints into our model.
Contextual hints. Typing constraints are not always sufﬁcient for determining the intended type
of a variable. For example, for variable network in function restore, the typing constraints
require network’s type to be a class with a ﬁeld called time, but there can be many classes that
have such an attribute (e.g., Date). However, the similarity between the variable name network
2
Published as a conference paper at ICLR 2020
Figure 2: An intermediate representation of the (unannotated version) program from Figure 1. The
τi represent type variables, among which τ8–τ16 are newly introduced for intermediate expressions.
Figure 3: Example hyperedges for Figure 2. Edge labels in gray (resp. red) are positional arguments
(resp. identiﬁers). (A) The return statement at line 6 induces a subtype relationship between τ13
and τ5. (B) MyNetwork τ8 declares attributes name τ1 and time τ2 and method forward τ9.
(C) τ14 is associated with a variable whose named is restore. (D) Usage hyperedge for line 10
connects τ6 and τ15 to all classes with a time attribute.
and the class name MyNetwork hints that network might have type MyNetwork. Based on this
belief, we can further propagate the return type of the library functionreadNumber (assuming we
know it is number) to infer that the type of thetime ﬁeld in MyNetwork is likely to be number.
Need for type dependency graph. There are many ways to view programs—e.g., as token se-
quences, abstract syntax trees, control ﬂow graphs, etc. However, none of these representations is
particularly helpful for inferring the most likely type annotations. Thus, our method uses static anal-
ysis to infer a set of predicates that are relevant to the type inference problem and represents these
predicates using a program abstraction called the type dependency graph.
Handling user-deﬁned types. As mentioned in Section 1, prior techniques can only predict types
seen during training. However, the code from Figure 1 deﬁnes its own class calledMyNetwork and
later uses a variables of type MyNetwork in the restore method. A successful model for this
task therefore must dynamically make inferences about user-deﬁned types based on their deﬁnitions.
2.1 P ROBLEM SETTING
Our goal is to train a type inference model that can take as input an entirely (or partially) unannotated
TypeScript project g and output a probability distribution of types for each missing annotation.
The prediction space is Y(g) = Ylib ∪Yuser(g), where Yuser(g) is the set of all user-deﬁned types
(classes/interfaces) declared within g, and Ylib is a ﬁxed set of commonly-used library types.
Following prior work in this space (Hellendoorn et al., 2018; Raychev et al., 2015; Xu et al.,
2016), we limit the scope of our prediction to non-polymorphic and non-function types. That
is, we do not distinguish between types such as List<T>, List<number>, List<string>
etc., and consider them all to be of type List. Similarly, we also collapse function types like
number →string and string →string into a single type called Function. We leave the
extension of predicting structured types as future work.
3
Published as a conference paper at ICLR 2020
Table 1: Different types of hyperedges used in a type dependency graph.
Type Edge Description
Logical
FIXED Bool(α) αis used as boolean
FIXED Subtype(α,β) αis a subtype of β
FIXED Assign(α,β)† βis assigned to α
NARY Function(α,β1,...,β k,β∗) α= (β1,...,β k) → β∗
NARY Call(α,β∗,β1,...,β k) α= β∗(β1,...,β k)
NARY Objectl1,...,lk (α,β1,...,β k) α= {l1 : β1,...,l k : βk}
FIXED Accessl(α,β) α= β.l
Contextual
FIXED Namel(α) αhas name l
FIXED NameSimilar(α,β) α, βhave similar names
NPAIRS Usagel((α∗,β∗),(α1,β1),..., (αk,βk)) usages involving name l
†Although assignment is a special case of a subtype constraint, we differentiate them
because these edges appear in different contexts and having uncoupled parameters for
these two edge types is beneﬁcial.
3 T YPE DEPENDENCY GRAPH
A type dependency graph G= (N,E) is a hypergraph where nodes N represent type variables and
labeled hyperedges Eencode relationships between them. We extract the type dependency graph of
a given TypeScript program by performing static analysis on an intermediate representationof its
source code, which allows us to associate a unique variable with each program sub-expression. As
an illustration, Figure 2 shows the intermediate representation of the code from Figure 1.
Intuitively, a type dependency graph encodes properties of type variables as well as relationships
between them. Each hyperedge corresponds to one of the predicates shown in Table 1. We partition
our predicates (i.e., hyperedges) into two classes, namely Logical and Contextual, where the for-
mer category can be viewed as imposing hard constraints on type variables and the latter category
encodes useful hints extracted from names of variables, functions, and classes.
Figure 3 shows some of the hyperedges in the type dependency graph Gextracted from the in-
termediate representation in Figure 2. As shown in Figure 3(A), our analysis extracts a predicate
Subtype(τ13,τ5) from this code because the type variable associated with the returned expression
v4 must be a subtype of the enclosing function’s return type. Similarly, as shown in Figure 3(B), our
analysis extracts a predicate Objectname,time,forward(τ8,τ1,τ2,τ9) because τ8 is an object type whose
name, time, and forward members are associated with type variables τ1,τ2,τ9, respectively.
In contrast to the Subtype and Object predicates that impose hard constraints on type variables,
the next two hyperedges shown in Figure 3 encode contextual clues obtained from variable names.
Figure 3(C) indicates that type variable τ14 is associated with an expression named restore.
While this kind of naming information is invisible to TypeScript’s structural type system ( ?), it
serves as a useful input feature for our GNN architecture described in Section 4.
In addition to storing the unique variable name associated with each type variable, the type depen-
dency graph also encodes similarity between variable and class names. The names of many program
variables mimic their types: for example, instances of a class called MyNetwork might often be
called network or network1. To capture this correspondence, our type dependency graph also
contains a hyperedge called NameSimilar that connects type variablesαand βif their corresponding
tokenized names have a non-empty intersection.1
As shown in Table 1, there is a ﬁnal type of hyperedge called Usage that facilitates type inference
of object types. In particular, if there is an object access var y = x.l, we extract the predicate
Usagel((τx,τy),(α1,β1),..., (αk,βk)) to connect x and y’s type variables withall classes αi that
contain an attribute/method βi whose name is l. Figure 3 shows a Usage hyperedge extracted from
the code in Figure 2. As we will see in the next section, our GNN architecture utilizes a special
attention mechanism to pass information along these usage edges.
1During tokenization, we split identiﬁer names into tokens based on underscores and camel case naming.
More complex schemes are possible, but we found this simple method to be effective.
4
Published as a conference paper at ICLR 2020
4 N EURAL ARCHITECTURE
Our neural architecture for making type predictions consists of two main parts. First, a graph neural
network passes information along the type dependency graph to produce a vector-valued embedding
for each type variable based on its neighbors. Second, a pointer network compares each variable’s
type embedding to the embedding vectors of candidate types (both computed from the previous
phase) to place a distribution over possible type assignments.
Given a type dependency graph G= (N,E), we ﬁrst to compute a vector embedding vn for each
n∈N such that these vectors implicitly encode type information. Because our program abstraction
is a graph, a natural choice is to use a graph neural network architecture. From a high level, this
architecture takes in initial vectors v0
n for each node n, performs K rounds of message-passing in
the graph neural network, and returns the ﬁnal representation for each type variable.
In more detail, let vt
n denote the vector representation of node nat the tth step, where each round
consists of amessage passingand an aggregationstep. The message passing step computes a vector-
valued update to send to the jth argument of each hyper-edge e ∈E connecting nodes p1,...,p a.
Then, once all the messages have been computed, the aggregation step computes a new embedding
vt
n for each nby combining all messages sent to n:
mt
e,pj = Msge,j(vt−1
p1 ,..., vt−1
pa ) vt
n = Aggr(vt−1
n ,{mt
e,n|e∈N(n)})
Here, Nis the neighborhood function, and Msge denotes a particular neural operation that depends
on the type of the edge (FIXED , NARY, or NPAIRS ), which we will describe later.
Initialization. In our GNN, nodes correspond to type variables and each type variable is associ-
ated either with a program variable or a constant. We refer to nodes representing constants (resp.
variables) as constant (resp. variable) nodes, and our initialization procedure works differently de-
pending on whether or not nis a constant node. Since the types of each constant are known, we
set the initial embedding for each constant node of type τ (e.g., string) to be a trainable vector
cτ and do not update it during GNN iterations (i.e., ∀t,vt
n = cτ). On the other hand, if n is a
variable node, then we have no information about its type during initialization; hence, we initialize
all variable nodes using a generic trainable initial vector (i.e., they are initialized to the same vector
but updated to different values during GNN iterations).
Message passing. Our Msg operator depends on the category of edge it corresponds to (see Ta-
ble 1); however, weights are shared between all instances of the same hyperedge type. In what
follows, we describe the neural layer that is used to compute messages for each type of hyperedge:
• FIXED : Since these edges correspond to ﬁxed arity predicates (and the position of each argument
matters), we compute the message of the jth argument by ﬁrst concatenating the embedding
vector of all arguments and then feed the result vector to a 2-layer MLP for the jth argument.
In addition, since hyperedges of type Access have an identiﬁer, we also embed the identiﬁer as a
vector and treat it as an extra argument. (We describe the details of identiﬁer embedding later in
this section.)
• NARY: Since NA RY edges connect a variable number of nodes, we need an architecture that can
deal with this challenge. In our current implementation of L AMBDA NET, we use a simple archi-
tecture that is amenable to batching. Speciﬁcally, given an NA RY edge El1,...,lk (α,β1,...,β k)
(for Function and Call, the labelslj are argument positions), the set of messages forαis computed
as {MLPα(vlj ∥vβj ) |j = 1...k }, and the message for eachβj is computed asMLPβ(vlj ∥vα).
Observe that we compute kdifferent messages for α, and the message for each βj only depends
on the vector embedding of αand its position j, but not the vector embeddings of other βj’s.2
• NPAIRS : This is a special category associated with Usagel((α∗,β∗),(α1,β1),..., (αk,βk)). Re-
call that this kind of edge arises from expressions of the formb= a.land is used to connect aand
b’s type variables with all classesαi that contain an attribute/method βi with label l. Intuitively, if
a’s type embedding is very similar to a typeC, then b’s type will likely be the same asC.l’s type.
Following this reasoning, we use dot-product based attention to compute the messages forα∗ and
2In our current implementation, this is reducible to multiple F IXED edges. However, NA RY edges could
generally use more complex pooling over their arguments to send more sophisticated messages.
5
Published as a conference paper at ICLR 2020
β∗. Speciﬁcally, we use α∗ and αj’s as attention keys andβj’s as attention values to compute the
message for β∗ (and switch the key-value roles to compute the message for α∗):
mt
e,β∗ =
∑
j
wjvt−1
βj
w = softmax(a) aj = vαj ·vα∗
Aggregation. Recall that the aggregation step combines all messages sent to node nto compute
the new embedding vt
n. To achieve this goal, we use a variant of the attention-based aggregation
operator proposed in graph attention networks (Veliˇckovi´c et al., 2018).
vt
n = Aggr(vt−1
n ,{mt
e,n|e∈ N(n)}) =vt−1
n +
∑
e∈N(n)
weM1mt
e,n (1)
where we is the attention weight for the message coming from edge e. Speciﬁcally, the weights
we are computed as softmax(a), where ae = LeakyReLu(vt−1
n ·M2mt
e,n) , and M1 and M2 are
trainable matrices. Similar to the original GAT architecture, we set the slope of the LeakyReLu to
be 0.2, but we use dot-product to compute the attention weights instead of a linear model.
Identiﬁer embedding. Like in Allamanis et al. (2017), we break variable names into word tokens
according to camel case and underscore rules and assign a trainable vector for all word tokens that
appear more than once in the training set. For all other tokens, unlike Allamanis et al. (2017),
which maps them all into one single <Unknown> token, we randomly mapped them into one of the
<Unknown-i> tokens, where iranges from 0 to 50 in our current implementation. This mapping is
randomly constructed every time we run the GNN and hence helps our neural networks to distinguish
different tokens even if they are rare tokens. We train these identiﬁer embeddings end-to-end along
with the rest of our architecture.
Prediction Layer. For each type variable n and each candidate type c ∈Y (g), we use a MLP
to compute a compatibility score sn,c = MLP(vn,uc), where uc is the embedding vector for c.
If c ∈Ylib, vc is a trainable vector for each library type c; if c ∈Yuser(g), then it corresponds to
a node nc in the type dependency graph of g, so we just use the embedding vector for nc and set
uc = vnc . Formally, this approach looks like a pointer network (Vinyals et al., 2015), where we use
the embeddings computed during the forward pass to predict “pointers” to those types.
Given these compatibility scores, we apply a softmax layer to turn them into a probability distribu-
tion. i.e., Pn(c|g) = exp(sn,c)/∑
c′exp(sn,c′). During test time, we max over the probabilities to
compute the most likely (or top-N) type assignments.
5 E VALUATION
In this section, we describe the results of our experimental evaluation, which is designed to answer
the following questions: (1) How does our approach compare to previous work? (2) How well can
our model predict user-deﬁned types? (3) How useful is each of our model’s components?
Dataset. Similar to Hellendoorn et al. (2018), we train and evaluate our model on popular open-
source TypeScript projects taken from Github. Speciﬁcally, we collect 300 popular TypeScript
projects from Github that contain between 500 to 10,000 lines of code and where at least 10%
of type annotations are user-deﬁned types. Note that each project typically contains hundreds to
thousands of type variables to predict, and these projects in total contain about 1.2 million lines
of TypeScript code. Among these 300 projects, we use 60 for testing, 40 for validation, and the
remainder for training.
Code Duplication. We ran jscpd3 on our entire data set and found that only 2.7% of the code
is duplicated. Furthermore, most of these duplicates are intra-project. Thus, we believe that code
duplication is not a severe problem in our dataset.
Preprocessing. Because some of the projects in our benchmark suite are only sparsely type an-
notated, we augment our labeled training data by using the forward type inference functionality
provided by the TypeScript compiler. 4 The compiler cannot infer the type of every variable and
3A popular code duplication detection tool, available athttps://github.com/kucherenko/jscpd.
4Like in many modern programming languages with forward type inference (e.g., Scala, C#, Swift), a
TypeScript programmer does not need to annotate every deﬁnition in order to fully specify the types of a
6
Published as a conference paper at ICLR 2020
leaves many labeled as any during failed inference; thus, we exclude any labels in our data set.
Furthermore, at test time, we evaluate our technique only on annotations that are manually added by
developers. This is the same methodology used by Hellendoorn et al. (2018), and, since develop-
ers often add annotations where code is most unclear, this constitutes a challenging setting for type
prediction.
Prediction Space. As mentioned in Section 2.1, our approach takes an entire TypeScript project
g as its input, and the corresponding type prediction space is Y(g) = Ylib ∪Yuser(g). In our ex-
periments, we set Yuser(g) to be all classes/interfaces deﬁned in g (except when comparing with
DeepTyper, where we set Yuser(g) to be empty), and for Ylib, we select the top-100 most common
types in our training set. Note that this covers 98% (resp. 97.5%) of the non- any annotations for
the training (resp. test) set.
Hyperparameters We selected hyperparameters by tuning on a validation set as we were develop-
ing our model. We use 32-dimensional type embedding vectors, and all MLP transformations in our
model use one hidden layer of 32 units, except the MLP for computing scores in the prediction layer,
which uses three hidden layers of sizes 32,16, and 8 (and size 1 for output). GNN message-passing
layers from different time steps have independent weights.
We train our model using Adam (Kingma & Ba, 2014) with default parameters (α= 0.9, β = 0.999)
and set the learning rate to be 10−3 initially but linearly decrease it to 10−4 until the 30th epoch.
We use a weight decay of 10−4 for regularization and stop the training once the loss on validation
set starts to increase (which usually happens around 30 epochs). We use the type annotations from a
single project as a minibatch and limit the maximal batch size (via downsampling) to be the median
of our training set to prevent any single project from having too much inﬂuence.
Implementation Details. We implemented L AMBDA NET in Scala, building on top of the Java
high-performance Tensor library Nd4j(nd4), and used a custom automatic differentiation library to
implement our GNN. Our GNN implementation does not use an adjacency matrix to represent GNN
layers; instead, we build the hyperedge connections directly from our type dependency graph and
perform batching when computing the messages for all hyperedges of the same type.
Code Repository. We have made our code publicly available on Github.5
5.1 C OMPARISON WITH DEEP TYPER
In this experiment, we compare L AMBDA NET’s performance with DeepTyper (Hellendoorn et al.,
2018), which treats programs as sequences of tokens and uses a bidirectional RNN to make type pre-
dictions. Since DeepTyper can only predict types from a ﬁxed vocabulary, we ﬁx both LAMBDA NET
and DeepTyper’s prediction space toYlib and measure their corresponding top-1 accuracy.
The original DeepTyper model makes predictions for each variable occurrence rather than decla-
ration. In order to conduct a meaningful comparison between DeepTyper and L AMBDA NET, we
implemented a variant of DeepTyper that makes a single prediction for each variable (by averaging
over the RNN internal states of all occurrences of the same variable before making the prediction).
Moreover, for a fair comparison, we made sure both DeepTyper and L AMBDA NET are using the
same improved naming feature that splits words into tokens.
Our main results are summarized below, where the Declaration (resp. Occurrence) column shows ac-
curacy per variable declaration (resp. token-level occurrence). Note that we obtain occurrence-level
accuracy from declaration-level accuracy by weighting each variable by its number of occurrences.
Model Top1 Accuracy (%)
Declaration Occurrence
DeepTyper 61.5 67.4
LAMBDA NETlib (K=6) 75.6 77.0
program. Instead, they only need to annotate some key places (e.g., function parameters and return types, class
members) and let the forward inference algorithm to ﬁgure out the rest of the types. Therefore, in our training
set, we can keep the user annotations on these key places and run the TS compiler to recover these implicitly
speciﬁed types as additional labels.
5See https://github.com/MrVPlusOne/LambdaNet.
7
Published as a conference paper at ICLR 2020
Table 2: Accuracy when predicting all types.
Model Top1 Accuracy (%) Top5 Accuracy (%)
Yuser Ylib Overall Yuser Ylib Overall
TS C OMPILER 2.66 14.39 8.98 - - -
SIMILAR NAME 24.1 0.78 15.7 42.5 3.19 28.4
LAMBDA NET (K=6) 53.4 66.9 64.2 77.7 86.2 84.5
Table 3: Performance of different GNN iterations (left) and ablations (right).
K Top1 Accuracy (%)
Yuser Ylib Overall
6 53.4 66.9 64.2
4 48.4 65.5 62.0
2 47.3 61.7 58.8
1 16.8 48.2 41.9
0 0.0 17.0 13.6
Ablation Top1 Accuracy (%)
(K = 4) Yuser Ylib Overall
LAMBDA NET 48.4 65.5 62.0
No Attention in NPAIR 44.1 57.6 54.9
No Contextual 27.2 52.6 47.5
No Logical∗ 24.7 39.2 36.2
Simple Aggregation 40.2 66.9 61.5
∗Training was unstable and experienced gradient explosion.
As we can see from the table, LAMBDA NET achieves signiﬁcantly better results compared to Deep-
Typer. In particular, L AMBDA NET outperforms DeepTyper by 14.1% (absolute) for declaration-
level accuracy and by 9.6% for occurrence-level accuracy.
Note that the accuracy we report for DeepTyper ( 67.4%) is not directly comparable to the original
accuracy reported in Hellendoorn et al. (2018) (56.9%) for the following reasons. While we perform
static analysis and have a strict distinction of library vs. user-deﬁned types and only evaluate both
tools on library type annotations in this experiment, their implementation treat types as tokens and
does not have this distinctions. Hence, their model also considers a much larger prediction space
consisting of many user-deﬁned types—most of which are never used outside of the project in which
they are deﬁned—and is also evaluated on a different set of annotations than ours.
5.2 P REDICTING USER -DEFINED TYPES
As mentioned earlier, our approach differs from prior work in that it is capable of predicting user-
deﬁned types; thus, in our second experiment, we extend L AMBDA NET’s prediction space to also
include user-deﬁned types. However, since such types are not in the prediction space of prior
work (Hellendoorn et al., 2018), we implemented two simpler baselines that can be used to cali-
brate our model’s performance. Our ﬁrst baseline is the type inference performed by the TypeScript
compiler, which is sound but incomplete (i.e., if it infers a type, it is guaranteed to be correct, but
it infers type any for most variables).6 Our second baseline, called S IMILAR NAME , is inspired by
the similarity between variable names and their corresponding types; it predicts the type of each
variable vto be the type whose name shares the most number of common word tokens with v.
The results of this experiment are shown in Table 2, which shows the top-1 and top-5 accuracy
for both user-deﬁned and library types individually as well as overall accuracy. In terms of overall
prediction accuracy, LAMBDA NET achieves 64.2% for top-1 and 84.5% for top-5, signiﬁcantly out-
performing both baselines. Our results suggest that our fusion of logical and contextual information
to predict types is far more effective than rule-based incorporation of these in isolation.
5.3 A BLATION STUDY
Table 3 shows the results of an ablation study in which (a) we vary the number of message-passing
iterations (left) and (b) disable various features of our architecture design (right). As we can see
from the left table, accuracy continues to improve as we increase the number of message passing
iterations as high as 6; this gain indicates that our network learns to perform inference over long
distances. The right table shows the impact of several of our design choices on the overall result.
6For inferring types from the TypeScript compiler, we use the code provided by Hellendoorn et al. (2018).
We found this method had a slightly lower accuracy than reported in their work.
8
Published as a conference paper at ICLR 2020
For example, if we do not use Contextual edges (resp. Logical edges), overall accuracy drops by
14.5% (resp. 25.8%). These drops indicate that both kinds of predicates are crucial for achieving
good accuracy. We also see that the attention layer for NP AIR makes a signiﬁcant difference for
both library and user-deﬁned types. Finally, Simple Aggregation is a variant of L AMBDA NET that
uses a simpler aggregation operation which replaces the attention-based weighed sum in Eq 1 with
a simple average. As indicated by the last row of Table 3 (right), attention-based aggregation makes
a substantial difference for user-deﬁned types.
5.4 C OMPARISON WITH JSN ICE
Since JSNice (Raychev et al., 2015) cannot properly handle class deﬁnitions and user-deﬁned types,
for a meaningful comparison, we compared both tools’ performance on top-level functions randomly
sampled from our test set. We ﬁltered out functions whose parameters are not library types and man-
ually ensured that all all the dependency deﬁnitions are also included. In this way, we constructed
a small benchmark suite consisting of 41 functions. Among the 107 function parameter and return
type annotations, L AMBDA NET correctly predicted 77 of them, while JSNice only got 48 of them
right. These results suggest that L AMBDA NET outperforms JSNice, even when evaluated only on
the places where JSNice is applicable.
6 R ELATED WORK
Type Inference using Statistical Methods. There are several previous works on predicting likely
type annotations for dynamically typed languages: Raychev et al. (2015) and Xu et al. (2016) use
structured inference models for Javascript and Python, but their approaches do not take advantage
of deep learning and are limited to a very restricted prediction space. Hellendoorn et al. (2018)
and Jangda & Anand (2019) model programs as sequences and AST trees and apply deep learning
models (RRNs and Tree-RNNs) for TypeScript and Python programs. Malik et al. (2019) make use
of a different source of information and take documentation strings as part of their input. However,
all these previous works are limited to predicting types from a ﬁxed vocabulary.
Graph Embedding of Programs. Allamanis et al. (2017) are the ﬁrst to use GNNs to obtain deep
embedding of programs, but they focus on predicting variable names and misuses for C ♯ and rely
on static type information to construct the program graph. Wang et al. (2017) use GNNs to encode
mathematical formulas for premise selection in automated theorem proving. The way we encode
types has some similarity to how they encode quantiﬁed formulas, but while their focus is on higher-
order formulas, our problem requires encoding object types. Veli ˇckovi´c et al. (2018) are the ﬁrst to
use an attention mechanism in GNNs. While they use attention to compute node embeddings from
messages, we use attention to compute certain messages from node embeddings.
Predicting from an Open Vocabulary. Predicting unseen labels at test time poses a challenge
for traditional machine learning methods. For computer vision applications, solutions might involve
looking at object attributes (Farhadi et al., 2017) or label similarity Wang et al. (2018); for natu-
ral language, similar techniques are applied to generalize across semantic properties of utterances
(Dauphin et al., 2013), entities (Eshel et al., 2017), or labels (Ren et al., 2016). Formally, most of
these approaches compare an embedding of an input to some embedding of the label; what makes
our approach a pointer network (Vinyals et al., 2015) is that our type encodings are derived during
the forward pass on the input, similar to unknown words for machine translation (Gulcehre et al.,
2016).
7 C ONCLUSIONS
We have presented LAMBDA NET, a neural architecture for type inference that combines the strength
of explicit program analysis with graph neural networks. L AMBDA NET not only outperforms other
state-of-the-art tools when predicting library types, but can also effectively predict user-deﬁned types
that have not been encountered during training. Our ablation studies demonstrate the usefulness of
our proposed logical and contextual hyperedges.
For future work, there are several potential improvements and extensions to our current system. One
limitation of our current architecture is the simpliﬁed treatment of function types and generic types
9
Published as a conference paper at ICLR 2020
(i.e., collapsing them into their non-generic counterparts). Extending the prediction space to also
include structured types would allow us to make full use of the rich type systems many modern
languages such as TypeScript provide. Another important direction is to enforce hard constraints
during inference such that the resulting type assignments are guaranteed to be consistent.
ACKNOWLEDGMENTS
We would like to thank DeepTyper authors, Vincent J. Hellendoorn, Christian Bird, Earl T. Barr,
and Miltiadis Allamanis, for sharing their data set and helping us set up our experimental compar-
isons. We also thank the ICLR reviewers for their insightful comments and constructive suggestions.
Finally, we would also like to thank Shankara Pailoor, Yuepeng Wang, Jocelyn Chen, and other
UToPiA group members for their kind support and useful feedback. This project was supported in
part by NSF grant CCF-1762299.
REFERENCES
Deeplearning4j. https://github.com/eclipse/deeplearning4j. Accessed: 2019-
09-24.
Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. Learning to represent programs
with graphs. ICLR, 2017.
Davide Ancona and Elena Zucca. Principal typings for java-like languages. In ACM SIGPLAN
Notices, volume 39, pp. 306–317. ACM, 2004.
Gavin Bierman, Mart´ın Abadi, and Mads Torgersen. Understanding typescript. In Richard Jones
(ed.), ECOOP 2014 – Object-Oriented Programming, pp. 257–281, Berlin, Heidelberg, 2014.
Springer Berlin Heidelberg. ISBN 978-3-662-44202-9.
Benjamin Chung, Paley Li, Francesco Zappa Nardelli, and Jan Vitek. Kafka: Gradual typing for
objects. In ECOOP 2018-2018 European Conference on Object-Oriented Programming, 2018.
Yann Dauphin, Gokhan Tur, Dilek Z. Hakkani-Tur, and Larry P. Heck. Zero-shot learning for
semantic utterance classiﬁcation. In ICLR, 2013.
Yotam Eshel, Noam Cohen, Kira Radinsky, Shaul Markovitch, Ikuya Yamada, and Omer Levy.
Named entity disambiguation for noisy text. In CoNLL, 2017.
Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In
CVPR, 2017.
Zheng Gao, Christian Bird, and Earl T. Barr. To type or not to type: Quantifying detectable bugs in
javascript. In Proceedings of the 39th International Conference on Software Engineering, ICSE
’17, pp. 758–769, Piscataway, NJ, USA, 2017. IEEE Press. ISBN 978-1-5386-3868-2. doi:
10.1109/ICSE.2017.75. URL https://doi.org/10.1109/ICSE.2017.75.
Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. Pointing the
unknown words. In Proceedings of the ACL, 2016.
Stefan Hanenberg, Sebastian Kleinschmager, Romain Robbes, ´Eric Tanter, and Andreas Steﬁk. An
empirical study on the impact of static typing on software maintainability. Empirical Software
Engineering, 19:1335–1382, 2013.
Vincent J. Hellendoorn, Christian Bird, Earl T. Barr, and Miltiadis Allamanis. Deep learning type
inference. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering
Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2018, pp.
152–162, New York, NY , USA, 2018. ACM. ISBN 978-1-4503-5573-5. doi: 10.1145/3236024.
3236051. URL http://doi.acm.org/10.1145/3236024.3236051.
Abhinav Jangda and Gaurav Anand. Predicting variable types in dynamically typed programming
languages. arXiv preprint arXiv:1901.05138, 2019.
10
Published as a conference paper at ICLR 2020
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard S. Zemel. Gated graph sequence neural
networks. ICLR, abs/1511.05493, 2016.
Rabee Sohail Malik, Jibesh Patra, and Michael Pradel. Nl2type: inferring javascript function types
from natural language information. In Proceedings of the 41st International Conference on Soft-
ware Engineering, pp. 304–315. IEEE Press, 2019.
Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional neural networks over tree
structures for programming language processing. In AAAI, volume 2, pp. 4, 2016.
Benjamin C Pierce and David N Turner. Local type inference. ACM Transactions on Programming
Languages and Systems (TOPLAS), 22(1):1–44, 2000.
Veselin Raychev, Martin Vechev, and Andreas Krause. Predicting program properties from ”big
code”. In Proceedings of the 42Nd Annual ACM SIGPLAN-SIGACT Symposium on Principles of
Programming Languages, POPL ’15, pp. 111–124, New York, NY , USA, 2015. ACM.
Xiang Ren, Wenqi He, Meng Qu, Clare R V oss, Heng Ji, and Jiawei Han. Label noise reduction
in entity typing by heterogeneous partial-label embedding. In Proceedings of the 22nd ACM
SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1825–1834.
ACM, 2016.
Jeremy G. Siek and Walid Taha. Gradual typing for objects. In ECOOP, 2007.
Dmitriy Traytel, Stefan Berghofer, and Tobias Nipkow. Extending hindley-milner type inference
with coercive structural subtyping. InAsian Symposium on Programming Languages and Systems,
pp. 89–104. Springer, 2011.
Petar Veliˇckovi´c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Li `o, and Yoshua
Bengio. Graph Attention Networks. International Conference on Learning Representations,
2018. accepted as poster.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In NeurIPS, 2015.
Michael M Vitousek, Andrew M Kent, Jeremy G Siek, and Jim Baker. Design and evaluation of
gradual typing for python. In ACM SIGPLAN Notices, volume 50, pp. 45–56. ACM, 2014.
Mingzhe Wang, Yihe Tang, Jian Wang, and Jia Deng. Premise selection for theorem proving by
deep graph embedding. In Advances in Neural Information Processing Systems, pp. 2786–2796,
2017.
Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and
knowledge graphs. In CVPR, 2018.
Zhaogui Xu, Xiangyu Zhang, Lin Chen, Kexin Pei, and Baowen Xu. Python probabilistic type infer-
ence with natural language support. InProceedings of the 2016 24th ACM SIGSOFT International
Symposium on Foundations of Software Engineering, pp. 607–618. ACM, 2016.
11

VeGen: A Vectorizer Generator for SIMD and Beyond
Yishen Chen
MIT CSAIL
USA
ychen306@mit.edu
Charith Mendis
UIUC
USA
charithm@illinois.edu
Michael Carbin
MIT CSAIL
USA
mcarbin@csail.mit.edu
Saman Amarasinghe
MIT CSAIL
USA
saman@csail.mit.edu
ABSTRACT
Vector instructions are ubiquitous in modern processors. Traditional
compiler auto-vectorization techniques have focused on targeting
single instruction multiple data (SIMD) instructions. However, these
auto-vectorization techniques are not sufficiently powerful to model
non-SIMD vector instructions, which can accelerate applications
in domains such as image processing, digital signal processing, and
machine learning. To target non-SIMD instruction, compiler devel-
opers have resorted to complicated, ad hoc peephole optimizations,
expending significant development time while still coming up short.
As vector instruction sets continue to rapidly evolve, compilers can-
not keep up with these new hardware capabilities.
In this paper, we introduce Lane Level Parallelism (LLP), which
captures the model of parallelism implemented by both SIMD and
non-SIMD vector instructions. We present VeGen, a vectorizer gen-
erator that automatically generates a vectorization pass to uncover
target-architecture-specific LLP in programs while using only in-
struction semantics as input. VeGen decouples, yet coordinates
automatically generated target-specific vectorization utilities with
its target-independent vectorization algorithm. This design enables
us to systematically target non-SIMD vector instructions that un-
til now require ad hoc coordination between different compiler
stages. We show that VeGen can use non-SIMD vector instructions
effectively, for example, getting speedup 3 ×(compared to LLVM’s
vectorizer) on x265’s idct4 kernel.
CCS CONCEPTS
· Software and its engineering →Translator writing systems
and compiler generators
; Retargetable compilers ; Specifica-
tion languages ; Automatic programming; · Computer systems
organization →Single instruction, multiple data .
KEYWORDS
optimization, auto-vectorization, non-SIMD
ACM Reference Format:
Yishen Chen, Charith Mendis, Michael Carbin, and Saman Amarasinghe.
2021. VeGen: A Vectorizer Generator for SIMD and Beyond. In Proceedings
of the 26th ACM International Conference on Architectural Support for Pro-
gramming Languages and Operating Systems (ASPLOS ’21), April 19ś23, 2021,
Virtual, USA. ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/
3445814.3446692
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASPLOS ’21, April 19ś23, 2021, Virtual, USA
© 2021 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-8317-2/21/04.
https://doi.org/10.1145/3445814.3446692
A1 A2 A3 A4
B1 B2 B3 B4
A1+B1 A2+B2 A3+B3 A4+B4
(a) vaddpd
A1 A2 A3 A4
B1 B2 B3 B4
A1+B1 A2-B2 A3+B3 A4-B4 (b) vaddsubpd
A1 A2
B1 B2
A1+A2 B1+B2
A3 A4
B3 B4
A3+A4 B3+B4
(c) vhaddpd
A1 A2 A3 A4
B1 B2 B3 B4
 A1*B1+
A2*B2
 A3*B3+
A4*B4
A5 A6 A7 A8
B5 B6 B7 B8
 A5*B5+
A6*B6
 A7*B8+
A7*B8
 (d) vpmaddwd
Figure 1: Examples of SIMD and non-SIMD instruction in
the AVX2 instruction set. We use different colors to indicate
how the input values flow to different output lanes.
1 INTRODUCTION
Vector instructions are ubiquitous in modern processors. Previ-
ous work on auto-vectorization has focused on single instruction
multiple data (SIMD) instructions, but there is little research on
systematically targeting non-SIMD vector instructions, which has
applications in domains such as digital signal processing, image
processing, and machine learning (e.g., Intel’s VNNI extension and
the dot-product instructions in ARMv8 [ 14]). In contrast with the
SIMD instruction shown in Figure 1(a), Figures 1(b)ś1(d) show three
examples of the non-SIMD instructions from the AVX2 instruction
set. Figure 1(b) shows a single instruction, multiple operations, mul-
tiple data (SIMOMD) instruction [ 2] that performs additions and
subtractions on alternating lanes ( vaddsubpd); Figure 1(c) shows
a horizontal addition with lane interleaving ( vhaddpd); and Fig-
ure 1(d) shows an instruction computing dot-products ( vpmaddwd).
To date, there is no unified model of parallelism that captures the
capabilities of these instructions.
Automatic Vectorization. There are two mainstream techniques
for extracting SIMD parallelism: loop vectorization [ 1, 20, 21] and
superword level parallelism (SLP) based vectorization [ 15, 17, 24].
Both techniques make two fundamental assumptions about vector
instructions: a SIMD instruction performs isomorphic operations
across all lanes, and the instruction applies the operations element-
wise (i.e., there is no cross-lane operation). Relying on these two as-
sumptions, these algorithms enable compiler developers to support
SIMD instructions across a variety of architectures with relatively
little incremental effort.
902
ASPLOS ’21, April 19–23, 2021, Virtual, USA Yishen Chen, Charith Mendis, Michael Carbin, and Saman Amarasinghe
(a) Reference Implementation
void
dot_16x1x16_uint8_int8_int32 (
uint8_t data [ restrict 4] ,
int8_t kernel [ restrict 16][4] ,
int32_t output [ restrict 16]) {
for ( int i = 0; i < 16; i ++)
for ( int k = 0; k < 4; k ++)
output [i] +=
data [k] * kernel [i][k];
}
(b) ICC
movzx r11d , [ rdi ]
movsx eax , [ rsi ]
imul r11d , eax
...
add r11d , r10d
add r11d , ecx
mov [ rdx ], r11d
(c) GCC
vmovdqa xmm0 , [ rip ]
vmovdqu xmm1 , [ rsi ]
...
vpmovsxbw xmm7 , xmm6
vpbroadcastw xmm5 , xmm5
vpmullw xmm7 , xmm7 , xmm9
vpsrldq xmm2 , xmm6 , 8
...
(d) LLVM
vmovdqu xmm6 , [ rsi + 32]
vmovdqu xmm7 , [ rsi + 48]
...
vpmulld zmm1 , zmm11 , zmm1
vpaddd zmm1 , zmm1 , [ rdx ]
vpmovsxbd zmm3 , xmm3
vpmulld zmm3 , zmm10 , zmm3
...
(e) VeGen
vmovdqu64 zmm0 , [ rdx ]
vpbroadcastd zmm1 , [ rdi ]
vpdpbusd zmm0 , zmm0 , [ rsi ]
vmovdqu64 [ rdx ], zmm0
Number of Instructions 273 106 61 4
Speedup Relative to ICC 1.0× 1.5× 2.2× 11.0×
Vector Extensions Used Not Vectorized SSE4 SSE4 & AVX-512 AVX512-VNNI
Figure 2: One of the dot-product kernels used by TVM’s 2D convolution layer (Figure 2(a)). Compiler generated assembly and
statistics for Intel’s compiler ICC (Figure 2(b)), GCC (Figure 2(c)), LLVM (Figure 2(d)), and VeGen (Figure 2(e))
Existing Support for Non-SIMD Instructions. Because non-
SIMD instructions violate the two fundamental assumptions of
existing vectorization algorithms, compiler developers support non-
SIMD instructions using ad hoc approaches that are cumbersome
and often ineffective. For most non-SIMD instructions, compiler
developers support them with backend peephole rewrites. However,
because these peephole rewrites do not generate vector instructions
by themselvesÐthey fuse sequences of SIMD instructions and vec-
tor shuffles into more non-SIMD instructionsÐrelying on peephole
rewrites alone is ineffective. A relatively more effective but more
labor-intensive strategy involves coordinating with the compiler’s
vectorizers to generate SIMD vector patterns that are tailored for
those rewrite rules. For instance, the initial support in LLVM [ 16]
for the addsub instruction family (Figure 1(b)) required three co-
ordinated changes to LLVM: refactoring LLVM’s SLP vectorizer
to support alternating opcodes, changing LLVM’s cost model to
recognize a special case of vector shuffle (blending odd and even
lanes), and modifying LLVM’s backend lowering logic to detect
the special patterns generated by the SLP vectorizer. As processor
vendors continue to add more complex non-SIMD instructions, this
methodology is not sustainable. Compilers are falling behind in
identifying the complex code sequences that can be mapped to
these instructions, and these multibillion-dollar investments by the
processor vendors in enhancing the vector instruction sets go un-
derutilized without expert developers manually writing assembly
or compiler intrinsics.
Our Approach: VeGen.
In this paper, we describe an extensible
framework for systematically targeting non-SIMD vector instruc-
tions. We define a new model of vector parallelism more general
than SIMD parallelism, and we present a vectorizer generator that
can effectively extract this new model of parallelism using non-
SIMD instructions.
To broaden the parallelism modeled by existing vectorizers, we
introduce Lane Level Parallelism (LLP), which generalizes super-
word level parallelism (SLP) [ 15] beyond SIMD in two ways: (1) An
instruction can execute multiple non-isomorphic operations, and
(2) the operation on each output lane can use values from arbitrary
input lanes. These two properties of LLP depend on the semantics
of a given target vector instruction. Consequently, our framework
encapsulates the two LLP properties (i.e., which operation exe-
cutes on a given lane and which values the operation uses) in a
couple of target-dependent vectorization utility functions. By inter-
facing with these utilities, the core vectorization algorithm in our
framework remains target-independent, as traditional vectorization
algorithms do.
We realize this framework with VeGen, a system that automat-
ically generates target-architecture-aware vectorizers to uncover
LLP in straight-line code sequences while using only instruction
semantics as input. From these instruction semantics, VeGen au-
tomatically generates the implementation of the aforementioned
vectorization utilities as a compiler library to describe the specific
kind of LLP supported by the target architecture. With this auto-
matically generated target-description library, VeGen’s vectorizer
can automatically use non-SIMD vector instructions. We added
support for newer classes of non-SIMD vector instructions (e.g.,
those found in AVX512-VNNI, which are not fully supported by
LLVM) by providing only their semantics.
We make the following contributions in this paper:
•We introduce Lane Level Parallelism, which captures the
type of parallelism implemented by both SIMD and non-
SIMD vector instructions.
•We describe a code-generation framework that jointly per-
forms vectorization and vector instruction selection while
maintaining the modularity of traditional target-independent
vectorizers designed for SIMD instructions.
•We present VeGen, a vectorizer generator that automati-
cally uses complex non-SIMD instructions using only their
documented semantics as input.
•We integrated VeGen into LLVM. VeGen can use non-SIMD
vector instructions effectively, e.g., getting speedup 3 ×(com-
pared to Clang’s vectorizer) on x265’s idct4 kernel.
2 MOTIVATIONAL EXAMPLE
In Figure 2, we compare VeGen with three production compilers
on a kernel used by TVM’s [ 8] 2D convolutional layers. Figure 2(a)
shows the naive scalar implementation of this kernel. Figures 2(b)ś
2(e) show the assembly output of ICC 19.0.1, GCC 10.2, LLVM
10.0, and the VeGen-generated vectorizer, respectively. All code
generators were configured to target AVX512-VNNI.
903
VeGen: A V ectorizer Generator for SIMD and Beyond ASPLOS ’21, April 19–23, 2021, Virtual, USA
VeGen’s vectorizer generates by far the shortest assembly code
sequence, 15.25×shorter than the next shortest code generator,
LLVM, and the generated code runs 5×faster than LLVM’s. Ve-
Gen’s vectorizer uses a new AVX512-VNNI instruction ( vpdpbusd);
GCC uses some of the integer vector instructions introduced in
SSE4 ( vpaddd and vpmullw); LLVM uses a mix of SSE and AVX512
instructions ( vpaddd and vpmulld operating on the 512-bit zmm
registers); and ICC, Intel’s own compiler, does not vectorize the
code. This is in spite of many man-hours spent on these compilers
to support Intel’s multibillion-dollar investment in these vector ex-
tensions. In contrast to these manual engineering efforts to target
new vector extensions, the target-specific components of VeGen
are automatically generated from semantics.
In this example, VeGen’s vectorizer uses a new dot-product
instruction (vpdpbusd) introduced in the AVX512-VNNI instruction
set. No other evaluated compilers were able to use this instruction.
It is important to note that VeGen’s output (Figure
2(e)) cannot be
generated simply by pattern matching because of the extra data
movement using the instruction vbroadcastw, which reorders the
inputs of vpdpbusd.
VeGen allows compilers to target new vector instructions with
less development effort. Thus, we believe this new capability will en-
able the creation of more robust vectorizers in production compilers.
3 LANE LEVEL PARALLELISM
Lane Level Parallelism (LLP) is our relaxation of superword level
parallelism (SLP) [ 15], which models short-vector parallelism (in
which an instruction executes multiple scalar operations in parallel)
with the following restrictions:
•The operations execute in lock-step.
•The inputs and outputs of the operations reside in packed
storage (usually implemented as vector registers). We refer
to an element of such packed storage as a lane.
•The operations are isomorphic.
•The operations are applied elementwise (i.e., there is no
cross-lane communication).
LLP relaxes SLP by removing the last two restrictions: (1) The
operations can be non-isomorphic, and (2) an operation executing
on one lane can use values from another input lane.
Non-isomorphism.
LLP allows different operations to execute
in parallel, whereas SLP applies only one operation across all vector
lanes. An example of an instruction that uses such a parallel pattern
is the x86 instruction
vaddsubpd (Figure 1(b)), which does addition
on the odd lanes and subtraction on the even lanes.
Cross-lane communication.
LLP allows an operation executing
on one lane to access values from another input lane (as long as the
lane is selected statically). In contrast, SLP restricts an operation to
use values from its own input lane. This flexibility is useful for com-
putations that require communication between lanes (e.g., parallel
reduction). For example, vhaddpd horizontally combines pairs of
lanes using addition and then interleaves the results (Figure 1(c)).
These properties of LLP depend on the semantics of individual
instructions. Different instructions can use different combinations
of operations or apply different cross-lane communication patterns.
Architecture Manual
Instruction Descriptions (Section 4.1)
Pattern Generator (Section 4.2)
Pattern Matcher (Section 4.3)
Vector Pack Selection (Section 4.4 & 5)
Code Generation (Section 4.5)
Scalar Program
Vector Program
Offline
Compile
Time
Figure 3: VeGen’s workflow. Bolded boxes represent arti-
facts such as manuals and programs.
4 VEGEN’S WORKFLOW
The key idea of VeGen is to encapsulate the details of the two
LLP properties (non-isomorphism and cross-lane communication)
behind two interfaces. VeGen views a given vector instruction
as a list of operations, each of which associated with a pattern
matcher (interface 1). Each vector instruction has a lane-binding
function that tells VeGen how the input lanes bind to the operations
(interface 2). VeGen generates the implementations of these two
interfaces offline. At compile time, VeGen’s target-independent
vectorization algorithm works by first using the pattern matcher to
find independent IR fragments that can be packed into the available
vector instructions, then using the lane binding rule to identify
the vector operands used by the packed vector instructions, and
then recursively finding other IR fragments that can be packed to
produce those vector operands.
Figure 3 shows the workflow of VeGen. VeGen targets non-
SIMD (and SIMD) vector instructions in two phases. In the offline
phase, VeGen takes instruction semantics (encoded in its vector
instruction description language) as input and generates the target-
dependent utility functions, such as the pattern matchers. At com-
pile time, VeGen’s target-independent heuristic uses the generated
utility functions to combine independent streams of scalar instruc-
tions into vector instructions.
To target a new vector instruction set, VeGen only requires the
compiler writers to describe the semantics of each instruction in
VeGen’s vector instruction description language. If the vendor has
provided instruction semantics in a machine-readable format such
as Intel’s Intrinsics Guide [ 9], this process can be automated. In Sec-
tion 6, we describe how VeGen automatically translates semantics
from the Intrinsics Guide.
904
ASPLOS ’21, April 19ś23, 2021, Virtual, USA Yishen Chen, Charith Mendis, Michael Carbin, and Saman Amarasinghe
FOR j := 0 to 3
i := j *32
dst [i +31: i] :=
SignExtend32 (a[i +31: i +16]* b[i +31: i +16]) +
SignExtend32 (a[i +15: i]*b[i +15: i])
ENDFOR
(a) Intel’s pseudocode documentation of pmaddwd
opmadd = (𝑥1 : 16, 𝑥2 : 16, 𝑥3 : 16, 𝑥4 : 16 )↦→
add(mul(sext32(𝑥1), sext32(𝑥2)), mul(sext32(𝑥3), sext32(𝑥4)))
pmaddwd = (𝑎 : 4 ×16, 𝑏 : 4 ×16)↦→
[opmadd (𝑎[0], 𝑏[0], 𝑎[1], 𝑏[1]), opmadd (𝑎[2], 𝑏[2], 𝑎[3], 𝑏[3])]
(b) Semantics of pmaddwd formalized in VeGen’s vector instruction
description language
bool match_MADD_Op ( llvm :: Value *V, Match &M) {
llvm :: Value *t0 , *t1 , *t2 , *t3;
if ( m_c_Add ( m_c_Mul ( m_SExt (t0), m_SExt (t1)),
m_c_Mul ( m_SExt (t2), m_SExt (t3))). match (V)) {
M. LiveIns = { t0 , t1 , t2 , t3 };
return true ;
}
return false ;
}
std :: vector < llvm :: Value *>
operand_1_pmaddwd ( const std :: vector < Match > & Matches ) {
return { Matches [0]. LiveIns [0] , Matches [0]. LiveIns [2] ,
Matches [1]. LiveIns [0] , Matches [1]. LiveIns [2] };
}
(c) Two examples of the vectorization utilities automatically generated
from semantics: a pattern matcher and a function that describes how
the input lanes of the first operand bind to the matched operations.
int16_t A[4] , B [4];
int32_t C [2];
void dot_prod () {
C [0] = A [0] * B [0] + A [1] * B [1];
C [1] = A [2] * B [2] + A [3] * B [3];
}
(d) An example scalar program to vectorize.
VSTORE
PMADDWD
VLOAD
VLOAD
mul
add
sext sext
A[0]
B[0]
A[2]
B[2]
A[1] A[3]
B[3]B[1]
sext sext sext sext sext sext
mul
mul
mul
add
C[0] C[1]
(e) The instruction DAG corresponding to the example scalar
program. The regions enclosed by the dotted curves represent
matched integer multiply-add operations. The rectangles repre-
sent vector packs.
vmovd xmm0 , [A]
vmovd xmm1 , [B]
pmaddwd xmm0 , xmm1 , xmm0
vmovd [C], xmm0
(f) Generated vector code
Figure 4: How VeGen uses the instruction pmaddwd. First, VeGen translates the pseudocode semantics of pmaddwd (Figure
4(a))
into its vector instruction description language (Figure 4(b)). Next, VeGen generates the vectorization utility functions (Fig-
ure 4(c)) used by its vectorizer at compile time. Figure 4(d) shows an example scalar program before vectorization. At compile
time, VeGen’s vectorizer combines the matched operations into vector packs (Figure 4(e)), which are later lowered into vector
assembly code (Figure 4(f)).
Figure 4 shows an end-to-end example of VeGen optimizing an
integer dot-product kernel. In the rest of this section, we will use it
as a running example.
Terminology & Notation. We use two related but distinct terms:
instructions and operations. Instructions can refer to either IR instruc-
tions such as LLVM IR or target instructions such as x86 instructions.
Operations refer to (side-effect free) bit-vector functions that can
be implemented both by IR and target instructions.
For brevity, we overload common set operations for vectors.
While doing so, we implicitly convert a vector to a set before apply-
ing the set operator. For example, let 𝑥 be a vector and 𝑖 a scalar;
when we say 𝑖 ∈𝑥 we mean that 𝑥 contains 𝑖.
4.1 Vector Instruction Description Language
VeGen uses its vector instruction description language (VIDL) to
model the semantics of each target vector instruction as a list of
scalar operations, with lane-binding rules indicating how the input
lanes bind to the operations. Figure 5 shows the syntax of VIDL.
VIDL assumes that target instructions read and write to registers but
have no other side-effects. VeGen models memory instructions such
as vector load separately. VIDL only allows selecting the input lanes
using constant indices: This restriction allows VeGen to statically
determine the vector operands used by each vector instruction.
Figure 4(b) shows the semantics of the SSE instruction pmaddwd
specified in VIDL. The instruction pmaddwd takes two vector reg-
isters as input, sign-extends the values from 16-bit to 32-bit tempo-
raries, multiplies the sign-extended values element-wise, and finally
adds together every adjacent pair of the multiplication results.
4.2 Generating Pattern Matchers
In the offline phase, VeGen collects the set of operations used by the
target vector instructions, and for each operation, VeGen generates
pattern matching rules to recognize IR sequences that implement
the operation. Figure 4(c) shows an example of the pattern matching
code generated by VeGen.
We designed VIDL to mirror the scalar IR that its vectorizer takes
as input. Thus, generating pattern matching code from VIDL is
generally straightforward. In Section 6 we discuss how to generate
pattern matchers that are more robust.
905
VeGen: A V ectorizer Generator for SIMD and Beyond ASPLOS ’21, April 19–23, 2021, Virtual, USA
𝑥 ∈variables 𝑖 ∈integers
sz ∈bit-widths vl ∈vector-lengths
lane ::= 𝑥 [𝑖 ]
expr ::= 𝑥 | lane | binop(expr1, expr2) |
unop(expr) |select(expr1, expr2, expr3)
opn ::= (𝑥1 : sz1, . . . , 𝑥𝑛 : sz𝑛)↦→ expr
res ::= opn(lane1, . . . ,lane𝑛)
inst ::= (𝑥1 : vl1 ×sz1, . . . , 𝑥𝑛 : vl𝑛 ×sz𝑛)↦→[ res1, . . . ,res𝑚 ]
Figure 5: Syntax of the Vector Instruction Description Lan-
guage (VIDL). ↦→denotes function abstraction.
4.3 Pattern Matching
At compile time, VeGen applies the generated pattern matchers on
the input scalar program. We call the result of pattern matching
a
match, an IR instruction DAG with (possibly) multiple live-ins
and a single live-out. VeGen represents each match as a tuple
consisting of its live-ins, live-out, and operation. In the running
example (Figure 4(e)), the integer multiply-add operation has two
matches (the sub-graphs enclosed in dotted curves): one rooted at
the instruction 𝑡1, and another rooted at 𝑡2.
Unlike other common applications of pattern matching such
as term rewriting, VeGen does not directly use the result of pat-
tern matching to rewrite the program. Instead, VeGen records the
matched patterns in a match table, which records the mapping
⟨live-out(𝑚), operation(𝑚)⟩↦→ 𝑚, for each match 𝑚. The match
table allows VeGen’s target-independent vectorization algorithm
(Section 4.4) to efficiently enumerate the set of candidate vector
instructions that can produce a given vector (Algorithm 1).
4.4 Vectorization
After running the generated pattern matchers (at compile time),
VeGen (1) uses a target-independent heuristic to find profitable
groups of matched IR instructions that can be packed into (possibly
non-SIMD) vector instructionsÐwe call such a group of instruc-
tions a vector pack Ðand then (2) lowers the vector packs into target
vector instructions.
Vector Pack. A pack is a tuple ⟨𝑣, [𝑚1, . . . ,𝑚 𝑘 ]⟩, where 𝑣 is a
vector instruction with 𝑘 output lanes, and 𝑚1, . . . ,𝑚 𝑘 are a list of
matches whose live-outs are independent. For example, let 𝑚1 and
𝑚2 be the two matched integer multiply-add operations rooted at
the instructions 𝑡1 and 𝑡2 in Figure 4(e), we can use the instruction
pmaddwd to combine them into a single vector pack:
𝑝ex = ⟨pmaddwd, [𝑚1,𝑚 2]⟩
VeGen models vector loads and stores as two special kinds of packs,
whose memory addresses must be contiguous.
We define two notations for vector packs. Let 𝑝 = ⟨𝑣, [𝑚1, . . . ,𝑚 𝑘 ]⟩
be a vector pack, then then values(𝑝)is the list of IR values pro-
duced by pack 𝑝 (i.e., values(𝑝)𝑖 = live-out(𝑚𝑖 )) and opcode(𝑝)= 𝑣.
In the running example,
values(𝑝ex)= [𝑡1, 𝑡2]
opcode(𝑝ex)= pmaddwd
A1 A3
B1 B3
A1*B1 A3*B3
A5 A7
B5 B7
A5*B5 A7*B7
Figure 6: Semantics of vpmuldq (sign-extended integer multi-
plication). White cells represent lanes unused by the instruc-
tion.
Vector Operand.
Vector packs have vector operands, represented
as lists of IR values. In the running example, 𝑝ex has two vector
operands (We overload the [.]operator here; e.g., 𝐴[0]denotes a
load of the first element of 𝐴):
operand1 (𝑝ex)= [𝐴[0], 𝐴[1], 𝐴[2], 𝐴[3]]
operand2 (𝑝ex)= [𝐵[0], 𝐵 [1], 𝐵 [2], 𝐵 [3]]
More specifically, let 𝑝 = ⟨𝑣, [𝑚1, . . . ,𝑚 𝑘 ]⟩be a vector pack,
then operand𝑖 (𝑝)= [𝑥1, . . . , 𝑥 𝑛]; where 𝑥𝑗 ∈/uniontext.1
𝑘 live-ins(𝑚𝑘 )is
one of the live-ins of the matches that should bind to the 𝑗’th lane
of the 𝑖’th operand of the vector instruction 𝑣. VeGen generates
the implementation of operand𝑖 (.)automatically from instruction
semantics; operand𝑖 (.)is known statically because the VIDL only
allows selecting input vector lanes using constant indices.
Don’t-Care Lanes.
Some instructions don’t use all of their input
lanes. For example, the SSE4 instruction vpmuldq (Figure 6) sign-
extends and multiplies only the odd input lanes. To handle a case,
we introduce a special don’t-care value. Each element of a vector
operand (i.e., operand𝑖 (.)) therefore takes the value of either a scalar
IR value (from the input program) or don’t-care.
Producing a Vector Operand. A pack 𝑝 produces a vector operand
𝑥 if they have the same size (i.e., |values(𝑝)|= |𝑥 |) and, for every
lane 𝑖, 𝑥𝑖 is either values(𝑝)𝑖 or don’t-care. Algorithm 1 shows the
algorithm for finding the set of feasible producer packs for a given
vector operand 𝑥. VeGen uses a separate routine to enumerate
producer packs that are vector loads, which can be done efficiently
because only contiguous loads can be packed together.
Dependence and Legality.
A pack 𝑝1 depends on another pack
𝑝2 if there exists an instruction 𝑖 ∈values(𝑝1)that depends on
another instruction 𝑗 ∈values(𝑝2). We define the dependencies
among scalar IR instructions and vector packs similarly. A set of
packs are legal when there are no cycles in the dependence graph.
Vector Pack Selection. Because lowering a given set of vector
packs to target vector instructions is relatively straightforward, vec-
torization reduces to finding a subset of the matches and combining
them into legal vector packs. The choice of packs determines the
performance of the generated code by affecting the level of paral-
lelism and the level of data-movement overhead (e.g., if a vector
operand is not produced directly, VeGen needs to use vector shuf-
fles to gather the elements of the operand). Given a scalar program,
VeGen selects a set of profitable vector packs using two alternative
heuristics that we will discuss in Section 5.
906
ASPLOS ’21, April 19ś23, 2021, Virtual, USA Yishen Chen, Charith Mendis, Michael Carbin, and Saman Amarasinghe
Algorithm 1: Find the set of (non-load) packs that produce
a given vector operand 𝑥. Load packs are found separately
by enumeration.
Input :
x: The vector operand that we need to produce
M: The match table, which contains
the mapping ⟨live-out(𝑚), operation(𝑚)⟩↦→ 𝑚
for each match 𝑚.
I: A list of instruction descriptions.
Output :A (potentially empty) set of producer packs of 𝑥.
1 if there are dependent values in 𝑥 then
2 return {}
3 end
4 producers ←{}
5 for vinst ∈𝐼 do
6 matches ←[]
7 for 𝑖 ←1 to number of lanes of vinst do
8 𝑓 ←the 𝑖’th operation of vinst
9 𝑚 ←𝑀 [⟨𝑥𝑖, 𝑓 ⟩]
10 if 𝑥𝑖 is don’t-care or 𝑚 is not null then
11 append 𝑚 to matches
12 end
13 end
14 if |matches|= number of lanes of vinst then
15 producers ←producers ∪pack(vinst, matches)
16 end
17 end
18 return producers
4.5 Code Generation
Given a set of vector packs (and the input program), VeGen’s code
generator emits a vector program as a combination of (1) the scalar
instructions not covered by the packs, (2) the compute vector in-
structions corresponding to the packs, and (3) the data-movement
vector instructions that follow from the dependence among the
packs and scalars.
Given a pack set 𝑃, we generate vector code as follows. The code
generation algorithm uses the target-specific functions operand𝑖 (.)
generated from instruction semantics.
Scheduling.
The code generator first schedules the scalar instruc-
tions (regardless of whether an instruction is replaced by vector
instructions) according to their dependencies and the following
constraint: For any pack
𝑝 ∈𝑃, all instructions in values(𝑝)are
grouped together in the final schedule. Such a schedule exists when
the set of packs are legal.
Lowering.
After scheduling, the code generator lowers the packs
in 𝑃 in topological order. The previous scheduling step ensures that
all of the values in operand𝑖 (𝑝)are ready by the time we lower any
𝑝 ∈𝑃. The code generator also emits any required swizzle instruc-
tions to gather a vector operand if the operand is not produced
directly by another pack and to extract an element of a vector pack
if the pack has a scalar user.
costSLP (𝑣)= min




 





min
𝑝 ∈producers(𝑣)
costop (opcode(𝑝))
+/summationtext.1
𝑖 costSLP (operand𝑖 (𝑝))
𝐶insert ·|𝑣|+costscalar (𝑣)
Figure 7: The SLP heuristic uses this recurrence to decide
whether to produce a vector operand 𝑣 directly via a vec-
tor pack or by vector insertions. costscalar (𝑣)is the total cost
of producing values in 𝑣 and their dependencies using only
scalar instructions.
5 VECTOR PACK SELECTION
VeGen uses a target-independent heuristic to select a set of prof-
itable vector packs. The goal of the heuristic is to select a set of packs
to maximize the total saving from vectorization while minimizing
the overhead of explicit data-movement that is necessary when
an instruction (whether vector or scalar) operand is not produced
exactly by any other instructionÐsuch as when a scalar instruction
uses a vector element and therefore requires a vector extraction.
Optimization Objective and Cost Model.
Let 𝑃 be the set of
selected vector packs, and let us focus on one of the packs 𝑝 ∈𝑃.
If the results of 𝑝 are used by some scalar instructions, we need to
extract those values and pay the following cost:
𝐶extract ·|values(𝑝)∩scalarUses|
Let 𝑣 be a vector operand of 𝑝. When a subset of 𝑣 is produced by
some other pack 𝑝′≠ 𝑝, we need to use vector shuffles to move
those values into 𝑣 and pay the following cost:
𝐶shuffle ·|{𝑝′∈𝑃 \{𝑝}| 𝑣 ∩values(𝑝′)≠ ∅}|
When some elements of 𝑣 are produced by scalar instructions, we
need to use vector insertions to insert those values into 𝑣 and pay
the following cost:
𝐶insert ·|𝑣 \[
/uniondisplay.1
𝑝′∈𝑃
values(𝑝′)]|
𝐶extract, 𝐶, and 𝐶insert are cost-model parameters.
Recall that VIDL doesn’t model vector shuffles (Section 4.1).
VeGen’s code generator therefore emits a mix of target vector
instructions and virtual (target-independent) vector shuffles and
relies on LLVM’s backend to lower the shuffles.
Pack Selection Heuristics.
Pack selection is NP-hard because
the more restricted version of pack selection for SIMD instructions
is NP-hard [ 19]. In the rest of this section, we discuss two heuristics
for pack selection. We first present a heuristic based on the bottom-
up SLP algorithm [ 15, 29] (we will refer to this heuristic as the SLP
heuristic). While the SLP heuristic is compile-time efficient, it has
various drawbacks that we will discuss and address with a tractable
search algorithm based on the SLP heuristic.
5.1 Pack Selection Using the SLP heuristic
The SLP heuristic builds a set of vector packs by traversing the
instruction DAG bottom-up (uses before definitions). Initially, the
set of packs are seeded with seed packs such as chains of contiguous
stores. The heuristic then recursively introduces vector packs to
907
VeGen: A V ectorizer Generator for SIMD and Beyond ASPLOS ’21, April 19–23, 2021, Virtual, USA
produce the vector operandsÐ VeGen uses Algorithm 1 to find such
producersÐin the current set of packs.
There are often multiple vector packs that can produce a given
operand. For a given operand, VeGen uses the dynamic program-
ming algorithm shown in Figure 7 to choose a producer. This is
the main modification we added to the original SLP algorithmÐin
SLP-based vectorization, there is at most one pack that can produce
any given operand.
Enumerating Seed Packs.
In addition to store packs, VeGen
enumerates a limited set of non-store seed packs, in two steps. First,
it computes a pairwise affinity score for each pair of IR instructions
according to the equation in Figure 8. Second, if a non-memory IR
instruction 𝑖 is used by some store instruction, then for all target
vector length VL (i.e., 2, 4, 8, etc.), VeGen enumerates the top 𝑘
packsÐaccording to the affinity scoreÐthat is VL-wide and whose
first lane is 𝑖. VeGen only enumerates instructions that feed into
stores to limit the total number of seeds.
Limitations.
The SLP heuristic assumes that each vector pack is
the sole user of its operands. Consequently, it is optimistic when
there are external scalar users of a vector pack and fails to account
for the vector extraction cost. On the other hand, the SLP heuristic
is also pessimistic when there are multiple uses of non-vectorizable
vector operands and fails to recognize that the multiple uses lower
the cost of vector shuffle/insertion (by amortization).
Consider the following code snippet, where there are two seed
packs: the two pairs of stores to the arrays a and b.
a [0] = x [0] + t1; a [1] = x [1] + t2;
b [0] = y [0] + t1; b [1] = y [0] + t2;
Suppose the temporaries t1 and t2 are not vectorizable. To vector-
ize the rest of the code snippet, the vectorizer would need to emit
extra vector insertion instructions to create the vector [𝑡1, 𝑡2]. On a
machine where vector insertions are expensive, it is plausible that
this code is profitable to vectorize only when the instruction (sub-
)DAG rooted at both seed packs are vectorized to amortize the cost
of creating [𝑡1, 𝑡2]. Unfortunately, because the SLP heuristic pro-
cesses each seed pack separately, it would (correctly) conclude that
none of the seed packs are individually profitable and (incorrectly)
decide that the whole basic block is not worth vectorizing.
5.2 Improving the SLP Heuristic with Search
To address the SLP heuristic’s limitations in handling shared values
in the instruction DAG, we apply a limited form of lookahead search
on top of the SLP heuristic. We first introduce a recurrence (Figure
9)
for optimally solving the pack selection problem. We don’t intend
to optimally solve the recurrence, which contains exponentially
many subproblems. VeGen instead uses beam search to navigate a
limited subset of the search space, using costSLP (.)(Figure 7) as a
state evaluation function.
Optimal Pack Selection.
Figure 9 shows the recurrence for
computing the optimal cost of vectorizing a given basic block,
cost(𝑉 , 𝑆, 𝐹 ), in which we solve for the optimal set of packs on
the instruction DAG bottom-up (uses before definitions), tracking
the set of vectors ( 𝑉 ) and scalar ( 𝑆) operands we need to produce
and the set of free instructions ( 𝐹) we have yet to decide whether
to vectorize. We decide how to produce the set of unresolved vector
(and scalar) operands jointly in order to correctly determine the
amortized cost of producing vector valuesÐwhether with packs or
using swizzle instructionsÐwith multiple uses.
The full cost of a basic block 𝐵 is cost({}, live-outs(𝐵), 𝐼 ), where
𝐼 is the set of instructions in 𝐵. In other words, we need to produce
the live outputs of the original basic blocks as scalars ( VeGen does
not vectorize across basic blocks). VeGen treats stores as special
cases. Stores are live at the end of a basic block, but, unlike other
live outputs, vectorized stores do not incur extraction costs.
There are two ways to produce a value: as part of some vector
pack or with a scalar instruction. Using a vector pack 𝑝 recursively
adds its operands to 𝑉 and removes its results from 𝑉 and 𝑆. To
avoid circular dependencies in the final pack set, we only consider a
pack (or a scalar instruction) once all of its users have been decided
(i.e., not in 𝐹).
Finally, the packing problem is solved once the sets of vector
and scalar operands become empty. Note that 𝐹 need not be empty
for a subproblem to be solved because some machine operations
(e.g., multiply-accumulate and dot-product) replace multiple IR
instructions and turn the intermediate instructions into dead code.
Beam Search. VeGen selects vector packs using beam search and
guided by the SLP heuristic. Beam search is a form of greedy tree
search, where the search algorithm considers a limited number of
promising search candidates (instead of only the most promising
one). In the case of pack selection, keeping track of this set of
candidates allows the vectorizer to consider some vector packs that
are costly according to the SLP heuristic but actually profitable.
When using beam search, VeGen’s pack selection heuristic (im-
plicitly) builds a search tree whose nodes correspond to the sub-
problems in Figure
9 (where each sub-problem is represented by
the tuple ⟨𝑉 , 𝑆, 𝐹 ⟩), and whose edges correspond to either adding
a vector pack or fixing an instruction as a scalar. Each tree edge
additionally has a transition cost taken to be the non-recursive
terms in Figure 9. For instance, if an edge corresponds to adding a
pack 𝑝, then the cost is
costop (opcode(𝑝))+ costextract (𝑝, 𝑆 )+costshuffle (𝑝,𝑉 )
To cut down the branching factor, VeGen only considers two types
of packs: (1) the producer packs of 𝑉 and (2) the set of seed packs
it enumerates before the main search loop.
At each iteration of the search, VeGen tracks a set of 𝑘 candidate
tree nodes, expands the candidate nodes and aggregates their chil-
dren, sorts the children in increasing order of the estimated cost,
and takes the top
𝑘 nodes to be candidates of the next iteration.
The special case of the beam search with 𝑘 = 1 is equivalent to the
SLP heuristic.
Ideally, we would like to order (and prune) the set of candidate
tree nodes based on the true optimal cost of following a tree node:
𝑔 +cost(𝑉 , 𝑆, 𝐹 ), where 𝑔 is the aggregate cost leading to a given
tree node and cost(𝑉 , 𝑆, 𝐹 )is the cost of optimally solving the tree
node’s sub-problem. However, computing the optimal cost is in-
tractable, and we instead order the candidate tree nodes using the
following formula:
𝑔 +cost (𝑉 , 𝑆, 𝐹 )≈ 𝑔 +
/summationdisplay.1
𝑣∈𝑉
costSLP (𝑣)+
/summationdisplay.1
𝑠 ∈𝑆
costscalar (𝑠)
908
ASPLOS ’21, April 19ś23, 2021, Virtual, USA Yishen Chen, Charith Mendis, Michael Carbin, and Saman Amarasinghe
affinity(𝑣, 𝑤 )=













 














−𝛼broadcast if 𝑣 = 𝑤
−𝛼constant if 𝑣 and 𝑤 are both constants
−𝛼mismatch if 𝑣 and 𝑤 not packable
−𝛼mismatch if 𝑣 and 𝑤 are loads separated by an unknown offset
−𝛼jumbled ·offset(𝑣, 𝑤 ) if 𝑣 and 𝑤 are loads separated by a constant offset
𝛼match if 𝑣 and 𝑤 are contiguous loads
𝛼match +/summationtext.1
𝑖 affinity(operand𝑖 (𝑣), operand𝑖 (𝑤)) otherwise
Figure 8: Recurrence for estimating the affinity score between two IR values 𝑣 and 𝑤. 𝛼∗are positive parameters. Before pack
selection, VeGen uses this function to enumerate a limited set of (non-store) seed vector packs so that the sums of affinities
of adjacent lanes are maximized.
cost (𝑉 , 𝑆, 𝐹 )= min




 





cost (𝑉𝑝, 𝑆𝑝, 𝐹𝑝 )+costop (opcode(𝑝))+ costextract (𝑝, 𝑆 )+costshuffle (𝑝,𝑉 ) if |𝐹 ∩ /uniontext.1
𝑖 ∈values(𝑝)
(users(𝑖))|= 0
cost (𝑉𝑖, 𝑆𝑖, 𝐹𝑖 )+costop (opcode(𝑖))+ costinsert (𝑖,𝑉 ) if 𝑖 a scalar ∧|𝐹 ∩users(𝑖)|= 0
0 if |𝑉 |= 0 ∧|𝑆|= 0
where
𝐹𝑝 = 𝐹 \values(𝑝) - Free instructions left if use pack 𝑝
𝑉𝑝 = {𝑣 ∈𝑉 |𝑣 ∩𝐹𝑝 ≠ ∅}∪
/uniondisplay.1
𝑖
operand𝑖 (𝑝) - Vectors to produce if use pack 𝑝
𝑆𝑝 = 𝑆 ∩𝐹𝑝 - Scalars to produce if use pack 𝑝
𝐹𝑖 = 𝐹 \{𝑖} - Free instructions left if fix 𝑖 as scalar
𝑉𝑖 = {𝑣 ∈𝑉 |𝑣 ∩𝐹𝑖 ≠ ∅} - Vectors to produce if fix 𝑖 as scalar
𝑆𝑖 = 𝑆 \𝐹𝑖 ∪
/uniondisplay.1
𝑗
operand 𝑗 (𝑖) - Scalars to produce if fix 𝑖 as scalar
costextract (𝑝, 𝑆 )= 𝐶extract ·|values(𝑝)∩𝑆| - Cost of extracting elements of 𝑝 (if 𝑝 is not a store pack)
costshuffle (𝑝,𝑉 )= 𝐶shuffle ·|{𝑣 ∈𝑉 |𝑣 ≠ 𝑝 ∧|𝑣 ∩values(𝑝)|> 0}| - Cost of shuffling elements out of 𝑝
costinsert (𝑖,𝑉 )= 𝐶insert ·
/summationdisplay.1
𝑣∈𝑉
⟨# of times 𝑖 occur in 𝑣⟩ - Cost of inserting 𝑖 into vectors in 𝑉
Figure 9: Optimal vector pack selection within a basic block. 𝑉 and 𝑆 are the sets of vector and scalar values we need to produce.
𝐹 is the set of (free) IR instructions we have yet to decide whether (or how) to vectorize. 𝐶∗are the cost model parameters.
6 IMPLEMENTATION
We implemented the offline part of VeGen (the part involved with
semantics and pattern generation) in Python. We implemented the
rest of VeGen, the part that performs compile time vectorization,
as an LLVM pass in C++. The LLVM pass takes scalar LLVM IR as
input and emits a mix of scalar IR and target-specific intrinsics 1
that in most cases, gets lowered to their corresponding instructions
(e.g., the LLVM intrinsic @llvm.x86.sse2.pmadd.wd maps to the
instruction pmaddwd).
1There is a straightforward mapping from Intel intrinsics to small sequences of LLVM
intrinsics. We find out the mapping from Intel intrinsics to the equivalent LLVM
intrinsics by wrapping an intel intrinsic in a standalone function whose signature
matches that of the intrinsic. We run Clang on this function, and record the instructions
produced by Clang
6.1 Target Instruction Specification
VeGen generates SMT formulas from the XML file that Intel uses
to render the Intrinsics Guide [ 9], which contains pseudocode doc-
umentations of the intrinsics. VeGen then lifts the SMT formulas
to VIDL (vector instruction description language). Lifting the SMT
formulas to VIDL is straightforward because we designed VIDL to
closely match the semantics of SMT bit-vector operations (which
are also closely related to LLVM’s integer instructions).
Translating Semantics from the Intrinsic Guide. To docu-
ment instruction semantics, Intel uses an imperative language that
operates on fixed-length bit-vectors. All values in the language are
bit-vectors and have one of four types: signed integer, unsigned
integer, float, and double. There are no implicit integer overflows
in this language; instead, if an operation can overflow its result
(such addition and multiplication), the operation first converts its
909
VeGen: A V ectorizer Generator for SIMD and Beyond ASPLOS ’21, April 19–23, 2021, Virtual, USA
input bit-vectors to a wider widthÐusing zero- or sign-extensions,
depending on the signednessÐbefore execution.
We implemented a symbolic evaluator for the language using
z3 [10] and translated Intel’s pseudocode documentation into for-
mal SMT formulas. We chose z3 mostly for its expression simplifier.
The evaluator maps expression-level constructs such as ALU oper-
ators and bit-vector slicing to their SMT equivalents; for instance,
additions become SMT bit-vector additions. We treat the following
high-level program constructs specially:
•Assignment. We model each assignment to (sub-)bit-vector as
a pure expression that takes the original bit-vector value and
outputs the post-update value. The output of the expression
is a concatenation of the unaffected sub-vector(s) and the
updated sub-vector.
Consider, for example, the statement x[7:0] = 0 , which
zeros the lower eight bits of a 32-bit variable x, we emit the
following formula:
Concat ( Extract (31 ,8 ,x), 0 b00000000 )
•Function calls. We inline all function calls.
•Loops. We unroll all for-loop (All for-loops have constant
trip-counts in the documentation language).
•If-statements. We apply if-conversion to the sub-vector be-
ing mutatedÐbit-vector assignment is the only construct
with side-effects. In the if-converted expression, we set the
predicate to the condition of the original if-statement, the
true-branch to the right-hand side of the assignment, and
the false-branch to the original value of the sub-vector.
For example, for the following statement, which condition-
ally zeros the lower eight bits of a 32-bit variable x,
IF ctrl [1:0]
x [7:0] = 0
FI
we emit the following formula:
Concat ( Extract (31 ,8 ,x),
If( Extract (0 ,0 , ctrl ) == 1,
Extract (7 ,0 ,x),
0 b00000000 ))
Our symbolic evaluator returns SMT formulas that are unnec-
essarily complicated in some cases because of the naive imple-
mentation of partial bit-vector updates and predicated updates.
We use z3’s simplifier to reduce the formula complexity. For most
instructions, z3’s simplifier simplifies their symbolic results into
representations that reflect the high-level intent of the original
documentation.
We validated the SMT formulas by random testing. Testing re-
vealed incorrect semantics resulting from ambiguous or simply
incorrect documentation. For instance, the signedness of saturation
arithmetic is particularly ambiguously documented for instructions
from the psubus family (subtract packed unsigned integers with
saturation). It turns out the result of an unsigned subtraction should
be saturated as a signed integer.
Pattern Generation. We use LLVM’s pattern-matching library to
implement VeGen’s pattern matching logic. VeGen canonicalizes
the patterns before emitting the pattern matchers. The canonical-
izer takes a pattern and generates an LLVM function that has the
same signature as the operation. We then run LLVM’s instcombine
pass on this function and generate pattern matching code accord-
ing to the final canonicalized IR sequences. This canonicalization
biases the patterns toward patterns that LLVM prefers. The most
notable rewrite is canonicalizing all comparisons to strict inequali-
ties (such as rewriting 𝑥 ≤1 to 𝑥 < 2) and is crucial for recognizing
integer saturations. Additionally, for (sub-)patterns of the form
select(cmp(𝑎, 𝑏), 𝑥,𝑦 ), we generate additional code to also match
the inverted case of the comparison.
6.2 Cost Model
For 𝐶insert and 𝐶extract, we use LLVM’s cost model. We set 𝐶shuffle =
2. VeGen additionally detects several special-case vector shuffle
and insertion patterns, such as vector broadcast and permutation,
and overrides the default cost model.
To estimate the cost of vector instructions, we use the instruction
throughput statistics from Intrinsics Guide. 2 To remain compatible
with the rest of LLVM’s cost model, we set the cost of each intrinsic
to be its inverse throughput scaled by a factor of two.
7 EXPERIMENTAL RESULTS
We evaluated VeGen on a subset of LLVM’s vector instruction
selection tests, some reference DSP kernels chosen from FFmpeg
and x265, and fixed-size dot-product kernels from OpenCV. We
evaluated the two pack selection heuristics discussed in Section 5Ð
the SLP heuristic and beam searchÐseparately. We show that in
most cases, VeGen outperforms LLVM’s vectorizer, and we explain
how VeGen fails to vectorize in the other cases. Additionally, we
present a case-study of VeGen vectorizing the scalar complex-
multiplication kernel.
Experimental Platforms.
For experiments requiring only AVX2,
we run the benchmarks on a server with the Intel ®Xeon®CPU
E5-2680 v3 CPU and 128 GB of memory. For experiments requiring
AVX512-VNNI, we use a server with the Intel ®Xeon®Platinum
8275CL CPU and 4 GB of memory. We use LLVM 10.0.0. In all cases,
we invoke clang with -O3 -ffast-math -march=native.
7.1 Synthetic Benchmarks
For our first set of experiments, we ported some of LLVM’s backend
instruction selection tests for non-SIMD instructions and SIMD in-
structions with complex semantics (e.g., min). These tests were orig-
inally written to exercise the pass that lowers LLVM vector IR into
target vector instructions. Because LLVM’s vector IR only models
isomorphic vector instructions, the tests for non-SIMD instructions
(e.g., haddpd) are written as combinations of LLVM vector instruc-
tions and vector shuffles. We translated the test cases (written in
LLVM IR) to their equivalent scalar version by expanding IR vector
instructions into multiple scalar instructions and by converting
vector function arguments to non-aliased pointer arguments.
2https://software.intel.com/sites/landingpage/IntrinsicsGuide/files/perf2.js
910
ASPLOS ’21, April 19ś23, 2021, Virtual, USA Yishen Chen, Charith Mendis, Michael Carbin, and Saman Amarasinghe
(a) Tests LLVM able to vectorize
Test Speedup
max_pd 1.0
min_pd 1.0
max_ps 1.0
min_ps 1.0
mul_addsub_pd 1.0
mul_addsub_ps 1.0
abs_pd 0.8
abs_ps 0.4
abs_i8 1.0
abs_i16 1.0
abs_i32 1.0
(b) Tests LLVM unable to vectorize
Test Speedup
hadd_pd 1.4
hadd_ps 1.2
hsub_pd 1.4
hsub_ps 1.2
hadd_i16 2.9
hsub_i16 4.9
hadd_i32 1.3
hsub_i32 1.3
pmaddubs 16.8
pmaddwd 4.2
Figure 10: Speedup (over LLVM, higher is better) on instruc-
tion selection tests ported from LLVM’s x86 backend. These
tests were originally written to exercise the pass that lowers
LLVM’s vector IR into their desired target instructions. We
ported the tests by manually transforming them into their
scalar equivalents.
Figure 10 shows the test results. Both the SLP heuristic and beam
search generate the same code, so we report one set of numbers.
VeGen vectorizes 19 out of 21 of the tests. LLVM fails to vectorize
10 out of 21 of the tests, all of which are non-SIMD instructions and
are vectorized by VeGen. Interestingly, the only non-SIMD tests
that LLVM can vectorize are mul_addsub_pd and mul_addsub_ps,
for which LLVM does have special-case support.
Both of the two tests that VeGen failed to vectorize compute
floating-point absolute values, and for which LLVM uses the fact
that the absolute value of a floating-point can be computed by
masking-off the sign-bit (i.e., the most significant bit) to vectorize;
VeGen does not have this knowledge and does not vectorize in
these two cases.
7.2 Optimizing Image and Signal Processing
Kernels
To demonstrate that VeGen can effectively use non-SIMD instruc-
tions on real-world kernels, we evaluated VeGen’s pack selection
heuristic on six kernels from x265We chose these kernels because
DSP and image processing are the motivating domains for non-
SIMD instructions such as
pmaddwd. These benchmarks are chal-
lenging to vectorize because they require intermediate shuffles and
partial reductions. We additionally evaluated the effect of pattern
canonicalization (Section 6). We ported the idct4 and idct8 kernels
from x265’s reference implementation. The rest are from FFmpeg.
We evaluated beam search (Section 5.1) with three beam-widths:
1, 64, and 128. Recall that a beam-width of one is effectively the SLP
heuristic. To evaluate the effect of canonicalizing our generated
patterns, we additionally evaluated a version of VeGen with pattern
canonicalization disabled (and with a beam-width of 128).
Figure 11 shows the benchmarking results. Both the SLP heuristic
and beam search outperform LLVM in all casesÐexcept for the SLP
heuristic (𝑘 = 1) on the idct4 benchmark; in the best case, beam
search with 𝑘 = 128 gets a speedup of 3×on idct4. Beam search
improves on the SLP heuristic on fft4, idct4, sbc, and chroma.
Using a larger beam-width does not always lead to better re-
sults, as shown by the performance degradation of idct8 on AVX512
with a beam-width of 64. We traced the search process and discov-
ered that the larger beam-width caused the search to include some
costly search states that are ignored when 𝑘 = 1. Their successor
statesÐthe number of which is larger than the beam-widthÐare
misestimated by costSLP (recall our discussion in Section 5.1 regard-
ing costSLP ’s limitations) to be more profitable than other candidate
states and ultimately misdirected the overall search effort.
Canonicalizing the generated patterns using LLVM’s own canon-
icalizer pays off on idct4, idct8, and chroma, all of which use satu-
ration arithmetic. Running LLVM’s canonicalizer on the generated
patterns effectively synchronizes VeGen’s pattern matchers with
the canonicalization pipeline that LLVM runs before invoking Ve-
Gen’s vectorizer.
Vectorizing idct4. We highlight some instructions that VeGen
generated for the idct4 kernel (targeting AVX512-VNNI). Figure 12
shows the generated code, which is 3×faster than LLVM’s code.
VeGen uses the instructions vphadd (integer horizontal reduction),
vpmaddwd (the motivating dot-product instruction), and vpackssdw
(saturate 32-bit to 16-bit integers). Of note are the vpunpackhdq
and vpunpackldq instructions preceding the vector stores. VeGen
uses these shuffle instructionsÐwithout which it is not profitable
to vectorize this kernelÐto form vector operands that are not di-
rectly produced by compute instructions such as vpmaddwd. VeGen
discovers this code sequence with beam search (i.e., 𝑘 ∈{64, 128})
but not with the SLP heuristic ( 𝑘 = 1).
7.3 Optimizing OpenCV’s Dot-Product Kernels
For our next set of experiments, we evaluated VeGen on OpenCV’s
reference dot-product kernel implementations. OpenCV’s refer-
ence implementation is a C++ template parameterized with different
data types and kernel sizes. These kernels are challenging to auto-
vectorize because they have interleaved memory accesses as well
as reduction.
Figure 13 shows the benchmarking results. VeGen found non-
trivial vectorization schemes for three of the four kernels. The
SLP heuristic and beam search generate identical code, so we only
report a single set of numbers. VeGen vectorizes the first bench-
mark naivelyÐessentially vectorizing across the unrolled iterations
and paying the shuffle cost for the interleaved accessesÐand only
yielded a 10% speedup. We investigated the slowdown VeGen in-
curred on AVX512 (VNNI). It turned out that for the first kernel,
VeGen actually emitted identical vector IR/intrinsics for both AVX2
and AVX-512. The performance difference comes down to how
LLVM’s backend lowered the shuffles emitted by VeGen. For the
AVX2, LLVM emitted the vpshufb instruction, whose latency and
inverse throughput are both one cycle. For the AVX-512, LLVM
instead emitted the vpmovdb instruction, whose inverse throughput
is two cycles (and latency four cycles) and slower than vpshufb.
Of note is the vector code VeGen generated for the int32 ×8
kernel (Figure 14), which matches OpenCV’s expert-optimized code.
We inspected the machine code and confirmed that VeGen used
the same high-level algorithm used by OpenCV’s expert developer.
911
VeGen: A V ectorizer Generator for SIMD and Beyond ASPLOS ’21, April 19–23, 2021, Virtual, USA
fft4
 fft8
 sbc
 idct8
 idct4
 chroma
Benchmarks
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Speedup
 1.54 
 1.37 
 2.00 
 1.37 
 0.74 
 1.45 
 2.00 
 1.37 
 1.97 
 1.39 
 2.51 
 1.92 
 2.00 
 1.37 
 1.96 
 1.43 
 3.00 
 1.97 
 2.00 
 1.37 
 1.77 
 0.61 
 0.76 
 0.92 
Vector Extension = AVX2
Pack Selection Heuristic
Beam-1
Beam-64
Beam-128
Beam-128 (w/o canonicalization)
fft4
 fft8
 sbc
 idct8
 idct4
 chroma
Benchmarks
 1.05 
 1.30 
 1.78 
 1.83 
 0.81 
 1.41  1.42 
 1.27 
 1.90 
 1.41 
 3.04 
 1.45 
 1.42 
 1.27 
 2.17 
 1.86 
 3.04 
 2.07 
 1.42 
 1.27 
 2.17 
 0.59 
 0.63 
 0.96 
Vector Extension = AVX512-VNNI
Figure 11: Speedup (over LLVM, higher is better) on kernels we selected from x265 (idct4 and idct8) and FFmpeg
...
vpermi2d xmm3 , xmm7 , xmm5
vphaddd xmm0 , xmm0 , xmm3
vpmaddwd xmm1 , xmm2 , xmm1
...
vpackssdw xmm1 , xmm1 , xmm2
vpunpckldq xmm2 , xmm0 , xmm1
vmovdqu [ rsi ], xmm2
vpunpckhdq xmm0 , xmm0 , xmm1
vmovdqu [ rsi +16] , xmm0
Figure 12: Snippets of vector code generated by VeGen (us-
ing a beam-width of 128) for the idct4 kernel.
(a) Results on AVX2
Kernel Size Speedup
int8 ×32 1.1
uint8 ×32 2.0
int32 ×8 1.5
int16 ×16 1.6
(b) Results on AVX-512 (VNNI)
Kernel Size Speedup
int8 ×32 0.7
uint8 ×32 2.2
int32 ×8 1.7
int16 ×16 2.5
Figure 13: OpenCV’s dot-product kernels specialized for
AVX2 and AVX-512 (VNNI) and different kernel sizes.
The reference (naive) implementation of the int32 ×8 kernel sign-
extends the input elements from 32-bit to 64-bit, multiplies the
two input arrays elementwise, and then reduces every adjacent
pair of elements by addition. There is no single instruction that
can implement this kernel by itself, and the high-level strategy
of VeGen (and OpenCV) is to perform the odd multiplications
separate from the even ones and finally add the odd and even en-
tries together. To multiply the odd (and even) entries, VeGen uses
the instruction vpmuldq, which is deceivingly complicated and per-
forms sign-extended multiplications only on the odd input elements
(Figure 6). The multiplications of the odd elements therefore map
naturally to vpmuldq.
vmovdqu ymm0 , [ rdi ]
vmovdqu ymm1 , [ rsi ]
vpmuldq ymm2 , ymm1 , ymm0
vpshufd ymm0 , ymm0 , 245 ## ymm0 = ymm0 [1 ,1 ,3 ,3 ,5 ,5 ,7 ,7]
vpshufd ymm1 , ymm1 , 245 ## ymm1 = ymm1 [1 ,1 ,3 ,3 ,5 ,5 ,7 ,7]
vpmuldq ymm0 , ymm1 , ymm0
vpaddq ymm0 , ymm0 , ymm2
vmovdqu [ rdx ], ymm0
Figure 14: Vector code that VeGen generated for the int32 ×
8 dot-product kernel in OpenCV. vpmuldq multiplies (with
sign-extension) the odd elements of its two vector operands.
vmovupd xmm0 , rsi
vpermilpd xmm1 , xmm0 , 1
vmovddup xmm2 , [ rdi +8]
vmulpd xmm1 , xmm1 , xmm2
vmovddup xmm2 , [ rdi ]
vfmaddsub213pd xmm2 , xmm0 , xmm1
vmovupd [ rdx ], xmm2
(a) Instructions generated by VeGen
(vfmaddsub213pd does multiply-add on
the odd lanes and multiply-sub on the
even lanes)
vmovsd xmm0 , [ rdi ]
vmovsd xmm1 , [ rdi + 8]
vmovsd xmm2 , [ rsi ]
vmovsd xmm3 , [ rsi + 8]
vmulsd xmm4 , xmm2 , xmm1
vfmadd231sd xmm4 , xmm3 , xmm0
vmulsd xmm1 , xmm3 , xmm1
vfmsub231sd xmm1 , xmm2 , xmm0
vmovsd [ rdx ], xmm1
vmovsd [ rdx + 8] , xmm4
(b) Instructions generated by
LLVM
Figure 15: Complex multiplication kernel, generated by Ve-
Gen (Figure
15(a)) and LLVM (Figure 15(b)). VeGen’s version
is 1.27×faster.
7.4 Optimizing Complex Multiplication
Complex arithmetic is a motivating application for SIMOMD in-
structions. In fact, (to the best knowledge of our knowledge) the first
SIMOMD instructions were designed for complex arithmetic [ 2].
Figure 15 shows the complex multiplication kernel compiled by
VeGen (both the SLP heuristic and beam search generated the same
code) and by LLVM. VeGen uses the instruction vfmaddsub213pd
(which performs fused multiply-add on the odd lanes and multiply-
sub on the even lanes). LLVM does not vectorize in this case, even
912
ASPLOS ’21, April 19ś23, 2021, Virtual, USA Yishen Chen, Charith Mendis, Michael Carbin, and Saman Amarasinghe
though (as noted earlier) LLVM’s SLP vectorizer has been specifi-
cally modified to support such a pattern. We stepped through the
LLVM’s optimization decisions and discovered that the root cause
is an error in its cost-benefit analysis. Since LLVM’s SLP vector-
izer is target-independent, it models such an alternating pattern as
two vector arithmetic instructions followed by a vector blending
instruction that combines the results. The error occurs when the
LLVM’s vectorizer includes the cost of the blending instruction
into its analysis and overestimates the total vectorization overhead.
VeGen does not suffer from such issues because VeGen has direct
knowledge of which target instructions are available.
8 RELATED WORK
Auto-vectorization.
Loop vectorization and SLP vectorization
are the two dominant vectorization techniques used by modern
compilers. Both types of vectorization techniques do not model non-
SIMD vector instruction in principle, but their implementations
in mainstream compilers such as LLVM have some special case
non-SIMD support.
Nuzman and Zaks [ 20] proposed a technique for vectorizing in-
terleaved memory accesses within a loop-based vectorizer. Eichen-
berger et al. proposed a technique for vectorizing misaligned mem-
ory accesses [ 11], and FlexVec [ 3] extends loop vectorizers to sup-
port vectorizing irregular programs with manually written rules. In
contrast, VeGen systematically adds support to generate non-SIMD
instructions automatically and is not limited to a particular class of
non-SIMD instructions.
The vectorizer generated by VeGen is more similar to SLP vec-
torization introduced by Larsen and Amarasinghe [ 15]. However,
VeGen supports a more general type of parallelism (LLP) and can
therefore target non-SIMD instructions. Almost all published SLP
vectorization techniques propose algorithmic improvements to cap-
ture more parallelism within the SLP framework. Some examples
are Holistic SLP vectorization [ 17], Super-node SLP [ 26], TSLP [ 23],
PSLP [ 24], VW-SLP [ 25], and ILP solver-aided goSLP [ 19].
There are domain-specific vectorizers that exploit architecture-
specific vector instructions as well as application-specific patterns.
The SPIRAL project [ 27] proposes several auto-vectorization schemes
specific to DSP algorithms. More specifically, they propose a target-
independent search-based vectorizing compiler targeting DSP algo-
rithms [ 12] and show how to use the vector swizzle instructions
supported by the AVX and Larrabee ISAs to implement the matrix
transpositions found in FFTs [ 18]. Compared to SPIRAL and its ex-
tensions, VeGen is a general-purpose vectorizer and not designed
to target any specific vector instruction sets.
Instruction Selection.
VeGen closely related to the research on
building retargetable compilers. VeGen is different from this line
of work in that it focuses on extracting fine-grained parallelism
(as a vectorizer) while simultaneously being aware of the detailed
operations supported by these target instructions (similar to an
instruction selector). Instruction selectionÐregardless of the qual-
ity of the code generatorÐalone is insufficient for automatically
targeting non-SIMD vector instructions because traditional instruc-
tion selectors only lowers IR vector instructionsÐthus requiring
cooperation with the vectorizer.
Ganapathi et al . [ 13] presented a survey on retargetable code
generation. Cattell [7] investigated automatically generating code
generators from machine descriptions. Ramsey and Fernández [28]
proposed a specification language for describing instruction encod-
ing. Buchwald et al. [ 6] synthesized instruction selection rules for
32-bit x86 integer instructions from their bit-vector specification.
Superoptimization.
VeGen is more broadly related to super-
optimization, which uses search techniques to directly generate
optimized programs based on instruction semantics. In principle,
a superoptimizer can accomplish what VeGen does, but in prac-
tice, existing superoptimizers are orders of magnitude slower than
auto-vectorizers such as VeGen.
Bansal and Aiken [4] constructed a peephole superoptimizer
by exhaustively enumerating short sequences of x86 instructions.
Schkufza et al . [31] proposed a stochastic superoptimizer that trades
completeness for scalability via a Markov Chain Monte Carlo sam-
pler. Barthe et al . [5] proposed a synthesizing vectorizer that works
by first unrolling the scalar code and then using an enumerative syn-
thesizer to find more an efficient vector program that implements
the unrolled loop body. Phothilimthana et al . [22] build on previous
work on enumerative [ 5], stochastic [ 31], and solver-based synthe-
sis to scale up superoptimization. Sasnauskas et al . [30] described a
superoptimizer for straight-line scalar LLVM IR.
9 CONCLUSIONS
We have described a framework for building target-aware vectoriz-
ers that can use non-SIMD instructions. We introduce Lane Level
Parallelism, a new model of short vector parallelism that captures
the kind of parallelism implemented by non-SIMD instructions. We
realize this framework with VeGen, a system that takes vector in-
struction semantics as input and generates a target-aware vectorizer
that uncovers LLP found in straight-line code sequences. VeGen
is flexible: to target a new vector instruction set, the developers
only need to describe the semantics of the new vector instructions.
VeGen allows compilers to target new vector instructions with less
development effort and thus enable the creation of more robust
vectorizers in future compilers.
ACKNOWLEDGMENTS
We thank Jesse Michel, Ajay Brahmakshatriya, Teodoro Fields
Collin, Logan Weber, and Alex Renda for reading early drafts and
offering insightful feedback. We also thank our shepherd Guy
Steele and the anonymous reviewers for guidance and valuable
suggestions. Our work is supported by the Application Driving
Architectures (ADA) Research Center, a JUMP Center co-sponsored
by SRC and DARPA; the Toyota Research Institute; the Office of
Advanced Scientific Computing Research under Award Numbers
DE-SC0008923 and DESC0018121; the National Science Founda-
tion under Grant No. CCF-1533753; and DARPA under Awards
HR0011-18-3-0007 and HR0011-20-9-0017. Any opinions, findings,
and conclusions or recommendations expressed in this material are
those of the authors and do not necessarily reflect the views of the
aforementioned funding agencies.
913
VeGen: A V ectorizer Generator for SIMD and Beyond ASPLOS ’21, April 19–23, 2021, Virtual, USA
REFERENCES
[1] Randy Allen and Ken Kennedy. 1987. Automatic Translation of FORTRAN
Programs to Vector Form. ACM Transactions on Programming Languages and
Systems (1987).
[2] Leonardo Bachega, Siddhartha Chatterjee, Kenneth A. Dockser, John A. Gun-
nels, Manish Gupta, Fred G. Gustavson, Christopher A. Lapkowski, Gary K. Liu,
Mark P. Mendell, Charles D. Wait, and T. J. Chris Ward. 2004. A high-performance
SIMD floating point unit for BlueGene/L: Architecture, compilation, and algo-
rithm design. In International Conference on Parallel Architecture and Compilation
Techniques.
[3] Sara S. Baghsorkhi, Nalini Vasudevan, and Youfeng Wu. 2016. FlexVec: Auto-
vectorization for Irregular Loops. In ACM SIGPLAN Conference on Programming
Language Design and Implementation .
[4] Sorav Bansal and Alex Aiken. 2006. Automatic Generation of Peephole Superop-
timizers. In International Conference on Architectural Support for Programming
Languages and Operating Systems .
[5] Gilles Barthe, Juan Manuel Crespo, Sumit Gulwani, Cesar Kunz, and Mark Marron.
2013. From Relational Verification to SIMD Loop Synthesis. In Symposium on
Principles and Practice of Parallel Programming .
[6] Sebastian Buchwald, Andreas Fried, and Sebastian Hack. 2018. Synthesizing an
Instruction Selection Rule Library from Semantic Specifications. In International
Symposium on Code Generation and Optimization .
[7] R. G. Cattell. 1980. Automatic Derivation of Code Generators from Machine
Descriptions. ACM Transaction on Programming Languages and Systems (1980).
[8] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan
Cowan, Haichen Shen, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin,
and Arvind Krishnamurthy. 2018. TVM: An automated end-to-end optimizing
compiler for deep learning. In Symposium on Operating Systems Design and
Implementation.
[9] Intel Corporation. 2012. Intel Intrinsics Guide . https://software.intel.com/sites/
landingpage/IntrinsicsGuide/
[10] Leonardo De Moura and Nikolaj Bjùrner. 2008. Z3: An Efficient SMT Solver. In
International Conference on Tools and Algorithms for the Construction and Analysis
of Systems .
[11] Alexandre E. Eichenberger, Peng Wu, and Kevin O’Brien. 2004. Vectorization for
SIMD Architectures with Alignment Constraints. In ACM SIGPLAN Conference
on Programming Language Design and Implementation .
[12] Franz Franchetti and Markus Püschel. 2002. A SIMD Vectorizing Compiler for
Digital Signal Processing Algorithms. In International Parallel and Distributed
Processing Symposium .
[13] Mahadevan Ganapathi, Charles N. Fischer, and John L. Hennessy. 1982. Retar-
getable Compiler Code Generation. Comput. Surveys (1982).
[14] ARM Holdings. 2011. Arm Architecture Reference Manual Armv8 . https:
//developer.arm.com/documentation/ddi0487/latest/
[15] Samuel Larsen and Saman Amarasinghe. 2000. Exploiting Superword Level
Parallelism with Multimedia Instruction Sets. In ACM SIGPLAN Conference on
Programming Language Design and Implementation .
[16] Chris Lattner and Vikram Adve. 2004. LLVM: A Compilation Framework for
Lifelong Program Analysis & Transformation. In International Symposium on
Code Generation and Optimization: Feedback-directed and Runtime Optimization .
[17] Jun Liu, Yuanrui Zhang, Ohyoung Jang, Wei Ding, and Mahmut Kandemir. 2012.
A Compiler Framework for Extracting Superword Level Parallelism. In ACM
SIGPLAN Conference on Programming Language Design and Implementation .
[18] Daniel McFarlin, Volodymyr Arbatov, Franz Franchetti, and Markus Püschel.
2011. Automatic SIMD Vectorization of Fast Fourier Transforms for the Larrabee
and AVX Instruction Sets. In International Conference on Supercomputing .
[19] Charith Mendis and Saman Amarasinghe. 2018. goSLP: Globally Optimized
Superword Level Parallelism Framework. Proceedings of the ACM on Programming
Languages (2018).
[20] Dorit Nuzman, Ira Rosen, and Ayal Zaks. 2006. Auto-vectorization of Interleaved
Data for SIMD. In ACM SIGPLAN Conference on Programming Language Design
and Implementation .
[21] Dorit Nuzman and Ayal Zaks. 2008. Outer-loop Vectorization: Revisited for Short
SIMD Architectures. In International Conference on Parallel Architectures and
Compilation Techniques.
[22] Phitchaya Mangpo Phothilimthana, Aditya Thakur, Rastislav Bodik, and Dinakar
Dhurjati. 2016. Scaling up Superoptimization. In International Conference on
Architectural Support for Programming Languages and Operating Systems .
[23] Vasileios Porpodas and Timothy M. Jones. 2015. Throttling Automatic Vectoriza-
tion: When Less is More. In Conference on Parallel Architecture and Compilation .
[24] Vasileios Porpodas, Alberto Magni, and Timothy M. Jones. 2015. PSLP: Padded
SLP Automatic Vectorization. In International Symposium on Code Generation
and Optimization .
[25] Vasileios Porpodas, Rodrigo CO Rocha, and Luís FW Góes. 2018. VW-SLP: auto-
vectorization with adaptive vector width. In International Conference on Parallel
Architectures and Compilation Techniques .
[26] Vasileios Porpodas, Rodrigo C. O. Rocha, Evgueni Brevnov, Luís F. W. Góes, and
Timothy Mattson. 2019. Super-Node SLP: Optimized Vectorization for Code
Sequences Containing Operators and Their Inverse Elements. In International
Symposium on Code Generation and Optimization .
[27] Markus Puschel, José MF Moura, Jeremy R Johnson, David Padua, Manuela M
Veloso, Bryan W Singer, Jianxin Xiong, Franz Franchetti, Aca Gacic, Yevgen Voro-
nenko, Robert W. Johnson, and Nicholas Rizzolo. 2005. SPIRAL: Code generation
for DSP transforms. Proc. IEEE (2005).
[28] Norman Ramsey and Mary F. Fernández. 1997. Specifying Representations of
Machine Instructions. ACM Transaction on Programming Languages and Systems
(1997).
[29] Ira Rosen, Dorit Nuzman, and Ayal Zaks. 2007. Loop-aware SLP in GCC. In GCC
Developers Summit.
[30] Raimondas Sasnauskas, Yang Chen, Peter Collingbourne, Jeroen Ketema, Jubi
Taneja, and John Regehr. 2017. Souper: A Synthesizing Superoptimizer. arXiv
preprint arXiv:1711.04422 (2017).
[31] Eric Schkufza, Rahul Sharma, and Alex Aiken. 2013. Stochastic Superoptimization.
In International Conference on Architectural Support for Programming Languages
and Operating Systems .
914

Open access to the Proceedings of the 
2018 USENIX Annual Technical Conference 
is sponsored by USENIX.
Towards Better Understanding of Black-box  
Auto-Tuning: A Comparative Analysis  
for Storage Systems
Zhen Cao, Stony Brook University; Vasily Tarasov, IBM Research–Almaden; 
Sachin Tiwari and Erez Zadok, Stony Brook University
https://www.usenix.org/conference/atc18/presentation/cao
This paper is included in the Proceedings of the 
2018 USENIX Annual Technical Conference (USENIX ATC ’18).
July 1 1–13, 2018 • Boston, MA, USA
ISBN 978-1-939133-02-1
Towards Better Understanding of Black-box Auto-Tuning:
A Comparative Analysis for Storage Systems
Zhen Cao1, Vasily Tarasov2, Sachin Tiwari1, and Erez Zadok1
1Stony Brook University and 2IBM Research—Almaden
Abstract
Modern computer systems come with a large num-
ber of conﬁgurable parameters that control their behav-
ior. Tuning system parameters can provide signiﬁcant
gains in performance but is challenging because of the
immense number of conﬁgurations and complex, non-
linear system behavior. In recent years, several studies
attempted to automate the tuning of system conﬁgura-
tions; but they all applied only one or few optimization
methods. In this paper, for the ﬁrst time, we apply and
then perform comparative analysis of multiple black-
box optimization techniques on storage systems, which
are often the slowest components of computing systems.
Our experiments were conducted on a parameter space
consisting of nearly 25,000 unique conﬁgurations and
over 450,000 data points. We compared these meth-
ods for their ability to ﬁnd near-optimal conﬁgurations,
convergence time, and instantaneous system throughput
during auto-tuning. We found that optimal conﬁgura-
tions differed by hardware, software, and workloads—
and that no one technique was superior to all others.
Based on the results and domain expertise, we begin to
explain the efﬁcacy of these important automated black-
box optimization methods from a systems perspective.
1 Introduction
Storage is a critical element of computer systems and
key to data-intensive applications. Storage systems
come with a vast number of conﬁgurable parameters that
control system’s behavior. Ext4 alone has around 60 pa-
rameters with whopping 1037 unique combinations of
values. Default parameter settings provided by vendors
are often suboptimal for a speciﬁc user deployment; pre-
vious research showed that tuning even a small subset
of parameters can improve power and performance efﬁ-
ciency of storage systems by as much as 9×[66].
Traditionally, system administrators pick parameter
settings based on their expertise and experience. Due to
the increased complexity of storage systems, however,
manual tuning does not scale well [87]. Recently, sev-
eral attempts were made to automate the tuning of com-
puter systems in general and storage systems in particu-
lar [71, 78]. Black-box auto-tuning is an especially pop-
ular approach thanks to its obliviousness to a system’s
internals [86]. For example, Genetic Algorithms (GA)
were applied to optimize the I/O performance of HDF5-
based applications [5] and Bayesian Optimization (BO)
was used to ﬁnd a near-optimal conﬁguration for Cloud
VMs [3]. Other methods include Evolutionary Strate-
gies [62], Smart Hill-Climbing [84], and Simulated An-
nealing [21]. The basic mechanism behind black-box
auto-tuning is to iteratively try different conﬁgurations,
measure an objective function’s value—and based on the
previously learned information—select the next conﬁg-
urations to try. For storage systems, objective functions
can be throughput, energy consumption, purchase cost,
or even a formula combining different metrics [50, 71].
Despite some appealing results, there is no deep under-
standing how exactly these methods work, their efﬁcacy
and efﬁciency, and which methods are more suitable for
which problems. Moreover, previous works evaluated
only one or few algorithms at a time. In this paper, for
the ﬁrst time (to the best of our knowledge), we apply
and analytically compare multiple black-box optimiza-
tion techniques on storage systems.
To demonstrate and compare these algorithms’ ability
to ﬁnd (near-)optimal conﬁgurations, we started by ex-
haustively evaluating several storage systems under four
workloads on two servers with different hardware and
storage devices; the largest system consisted of 6,222
unique conﬁgurations. Over a period of 2+ years, we ex-
ecuted 450,000+ experimental runs. We stored all data
points in a relational database for query convenience, in-
cluding hardware and workload details, throughput, en-
ergy consumption, running time, etc. In this paper, we
focused on optimizing for throughput, but our method-
ology and observations are applicable to other metrics
as well. We will release our dataset publicly to facilitate
more research into auto-tuning and better understanding
of storage systems.
Next, we applied several popular techniques to the
collected dataset to ﬁnd optimal conﬁgurations under
various hardware and workload settings: Simulated An-
nealing (SA), Genetic Algorithms (GA), Bayesian Op-
timization (BO), and Deep Q-Networks (DQN). We
also tried Random Search (RS) in our experiments,
which showed surprisingly good results in previous re-
search [8]. We compared these techniques from vari-
ous aspects, such as the ability to ﬁnd near-optimal con-
ﬁgurations, convergence time, and instantaneous sys-
tem throughput during auto-tuning. For example, we
found that several techniques were able to converge to
good conﬁgurations given enough time, but their efﬁ-
cacy differed a lot. GA and BO outperformed SA and
USENIX Association 2018 USENIX Annual Technical Conference    893
DQN on our parameter spaces, both in terms of con-
vergence time and instantaneous throughputs. We also
showed that hyper-parameter settings of these optimiza-
tion algorithms, such as mutation rate in GA, could af-
fect the tuning results. We further compared the tech-
niques across three behavioral dimensions: (1) Explo-
ration: how much the technique searches the space ran-
domly. (2) Exploitation: how much the technique lever-
ages the “neighborhood” of the current candidate or pre-
vious search history to ﬁnd even better conﬁgurations.
(3) History: how much data from previous evaluations
is kept and utilized in the overall search process. We
show that all techniques employ these three key concepts
to varying degrees and the trade-off among them plays
an important role in the effectiveness and efﬁciency of
the algorithms. Based on our experimental results and
domain expertise, we provide explanations of efﬁcacy
of such black-box optimization methods from a storage
perspective. We observed that certain parameters would
have a greater effect on system performance than oth-
ers, and the set of dominant parameters depends on ﬁle
systems and workloads. This allows us to provide more
insights into the auto-tuning process.
Auto-tuning storage systems is fairly complex and
challenging. We made several necessary assumptions
and simpliﬁcations while collecting our exhaustive data,
which we detail in §3. Therefore, some of our observa-
tions might differ when applied to production systems.
However, the main purpose of this paper is not to pro-
vide a complete solution; rather, we focus on comparing
and understanding the efﬁcacy of several popular opti-
mization techniques when applied to storage systems.
We believe this paves the way for practical auto-tuning
storage systems in real-time.
The rest of the paper is organized as follows. §2 ex-
plains the challenges of auto-tuning storage systems and
provides necessary background knowledge.§3 describes
our experimental methodology and environments. In §4
we applied multiple optimization methods and evaluated
and explained them from various aspects. §5 covers lim-
itations and future plans for our work. §6 lists related
work. We conclude and discuss future directions in §7.
2 Background
Storage systems are often a critical component of com-
puter systems, and are the foundation for many data-
intensive applications. Usually they come with a large
number of conﬁgurable options that could affect or even
determine the systems’ performance [12, 74], energy
consumption [66], and other aspects [47, 71]. Here
we deﬁne a parameter as one conﬁgurable option, and
a conﬁguration as a combination of parameter val-
ues. For example, the parameter block size of Ext4
can take 3 values: 1K, 2K, and 4K. Based on this,
[journal mode=“data=writeback”, block size=4K, in-
ode size=4K] is one conﬁguration with 3 speciﬁc pa-
rameters: journal mode, block size, and inode size. All
possible conﬁgurations form a parameter space.
When conﬁguring storage systems, users often stick
with the default conﬁgurations provided by vendors be-
cause 1) it is nearly impossible to know the impact of
every parameter across multiple layers; and 2) vendors’
default conﬁgurations are trusted to be “good enough”.
However, previous studies [66] showed that tuning even
a tiny subset of parameters could improve the perfor-
mance and energy efﬁciency for storage systems by as
much as 9×. As technological progress slows down, it
becomes even more important to squeeze every bit of
performance out of deployed storage systems.
In the rest of this section we ﬁrst discuss the chal-
lenges of system tuning (§2.1). Then, §2.2 brieﬂy intro-
duces several promising techniques that we explore in
this paper. §2.3 explains certain methods that we deem
less promising. §2.4 provides a uniﬁed view of these
optimization methods.
2.1 Challenges
The tuning task for storage systems is difﬁcult, due to
the following four challenges.
(1) Large parameter space. Modern storage systems
are fairly complex and easily come with hundreds or
even thousands of tunable parameters. One evaluation
for storage systems can take multiple minutes or even
hours, which makes exhaustive search impractical. Even
human experts cannot know the exact impact of every
parameter and thus have little insight into how to opti-
mize. For example, Ext4+NFS would result in a parame-
ter space consisting of more than1022 unique conﬁgura-
tions. IBM’s General Parallel File System (GPFS) [64]
contains more than 100 tunable parameters, and hence
1040 conﬁgurations. From the hardware perspective,
SSDs [30, 53, 57, 65], shingled drives [1, 2, 32, 45], and
non-volatile memory [40, 83] are gaining popularity,
plus more layers (LVM, RAID) are added.
(2) Non-linearity. A system is non-linear when the
output is not directly proportional to the input. Many
computer systems are non-linear [16], including storage
systems [74]. For example, Figure 1 shows the aver-
age operation latency of GPFS under a typical database
server workload while changing only the value of the
parameter pagepool from 32MB to 128MB, and setting
all the others to their default. Clearly the average la-
tency is not directly proportional to the pagepool size.
In fact, through our experiments, we have seen many
more parameters with similar behavior. Worse, the pa-
rameter space for storage systems is often sparse, irreg-
ular, and contains multiple peaks. This makes automatic
optimization even more challenging, as it has to avoid
894    2018 USENIX Annual Technical Conference USENIX Association
 0
 10
 20
 30
 40
 0  30  60  90  120  150
Avg. Latency (ms)
pagepool Size (MB)
Figure 1: Storage systems are non-linear.
getting stuck in a local optima [36].
(3) Non-reusable results. Previous studies have
shown that evaluation results of storage systems [12,66]
and databases [78] are dependent on the speciﬁc hard-
ware and workloads. One good conﬁguration might per-
form poorly when the environment changes. Our evalu-
ation results in Section 4 show similar observations.
(4) Discrete and non-numeric parameters. Some
storage system parameters can take continuous real val-
ues, while many others are discrete and take only a lim-
ited set of values. Some parameters are not numeric
(e.g., I/O scheduler name or ﬁle system type). This adds
difﬁculty in applying gradient-based approaches.
Given these challenges, manual tuning of storage sys-
tems becomes nearly impossible while automatic tuning
merely difﬁcult. In this paper we focus on automatic
tuning and treat it as an optimization problem.
2.2 Applied Methods
Several classes of algorithms have been proposed for
similar optimization tasks, including automated tuning
for hyper-parameters of machine learning systems [7, 8,
59] and optimization of physical systems [3, 78]. Ex-
amples include Genetic Algorithms (GA) [18, 34], Sim-
ulated Annealing (SA) [15, 41], Bayesian Optimization
(BO) [11,68], and Deep Q-Networks (DQN) [46,54,55].
Although these methods were proposed originally in dif-
ferent scholarly ﬁelds, they can all be characterized as
black-box optimizations. In this section we introduce
several of these techniques that we successfully applied
in auto-tuning storage systems.
Simulated Annealing (SA) is inspired by the anneal-
ing process in metallurgy, which involves the heating
and controlled cooling of a material to get to a state with
minimum thermodynamic free energy. When applied to
storage systems, a state corresponds to one conﬁgura-
tion. Neighbors of a state refer to new conﬁgurations
achieved by altering only one parameter value of the cur-
rent state. The thermodynamic free energy is analogous
to optimization objectives. SA works by maintaining the
temperature of the system, which determines the prob-
ability of accepting a certain move. Instead of always
moving towards better states as hill-climbing methods
do, SA deﬁnes an acceptance probability distribution ,
which allows it to accept some bad moves in the short
Parent 1
Parent 2
Child 1
Child 2
Journal OptionBG FS
NilFS2
NilFS2
8
256
order=strict
order=relaxed
order=relaxed8NilFS2
order=strict256
NilFS2
Figure 2: Crossover in Genetic Algorithm (GA).
run, that can lead to even-better moves later on. The
system is initialized with a high temperature, and thus
has high probability of accepting worse states in the be-
ginning. The temperature is gradually reduced based on
a pre-deﬁned cooling schedule, thus reducing the proba-
bility of accepting bad states over time.
Genetic Algorithms (GA) were inspired by the pro-
cess of natural selection [34]. It maintains a popula-
tion of chromosomes (conﬁgurations) and applies sev-
eral genetic operators to them. Crossover takes two par-
ent chromosomes and generates new ones. As Figure 2
illustrates, two parent Nilfs2 conﬁgurations are cut at
the same crossover point, and then the subparts after the
crossover point are exchanged between them to gener-
ate two new child conﬁgurations. Better chromosomes
will have a higher probability to “survive” in future se-
lection phases. Mutation randomly picks a chromosome
and mutates one or more parameter values, which pro-
duces a completely different chromosome.
Reinforcement Learning (RL)[72] is an area of ma-
chine learning inspired by behaviorist psychology. RL
explores how software agents take actions in an environ-
ment to maximize the deﬁned cumulative rewards. Most
RL algorithms can be formulated as a model consisting
of: (1) A set of environment states; (2) A set of agent
actions; and (3) A set of scalar rewards. In case of stor-
age systems, states correspond to conﬁgurations, actions
mean changing to a different conﬁguration, and rewards
are differences in evaluation results. The agent records
its previous experience (history), and makes it available
through a value function, which can be used to predict
the expected reward of state-action pairs. The policy de-
termines how the agent takes action, which maintains the
exploration-exploitation trade-off. The value function
can take a tabular form, but this does not scale well to
many dimensions. Function approximation is proposed
to deal with high dimensionality, which is still known
to be unstable or even divergent. With recent advances
in Deep Learning [28], deep convolutional neural net-
works, termed Deep Q-Networks (DQN), were proposed
to parameterize the value function, and have been suc-
cessfully applied in solving various problems [54, 55].
Many variants of DQN have been proposed [46].
Bayesian Optimization (BO) [11, 68] is a popular
framework to solve optimization problems. It models
USENIX Association 2018 USENIX Annual Technical Conference    895
Algorithm Origin Exploration Exploitation History
Simulated Annealing
(SA)
Annealing technology
in metallurgy
Allowing moving to
worse neighbor states Neighbor function N/A
Genetic Algorithms
(GA) Natural evolution Mutation Crossover and selection Current population
Deep Q-Networks
(DQN)
Behaviorist psychology
and neuroscience Taking random actions Taking actions based on
action-reward function
Deep convolutional
neural network
Bayesian
Optimization (BO)
Statistics and
experimental design
Selecting samples with
high variances
Selecting samples with
high mean values
Acquisition function &
probabilistic model
Table 1: Comparison and summaries of optimization techniques.
the objective function as a stochastic process, with the
argument corresponding to one storage conﬁguration. In
the beginning, a set of prior points (conﬁgurations) are
given to get a fair estimate of the entire parameter space.
BO works by computing the conﬁdence interval of the
objective function according to previous evaluation re-
sults, which is deﬁned as the range of values that the
evaluation result is most likely to fall into (e.g., with
95% probability). The next conﬁguration is selected
based on a pre-deﬁned acquisition function. Both conﬁ-
dence intervals and the acquisition function are updated
with each new evaluation. BO has been successfully ap-
plied in various areas, including hyper-parameter opti-
mization [17] and system conﬁguration optimization [3].
BO and its variants differ mainly in their form of prob-
abilistic models and acquisition functions. In this paper
we focus mainly on Gaussian priors and an Expected
Improvement acquisition function [68].
Other promising techniques include Tabu Search [27],
Particle Swarm Optimization [39], Ant Colony Opti-
mization [20], Memetic Algorithms [52], etc. Due to
space limits, we omit comparing all of them in this pa-
per (part of our future work). In fact, as detailed in §2.4,
most of these techniques actually share similar traits.
2.3 Other Methods
Although many optimization techniques have been pro-
posed, we feel that not all of them make good choices for
auto-tuning storage systems. For example, since many
parameters of storage systems are non-numeric, most
gradient-based methods (i.e., based on linear-regression)
are less suitable to this task [29].
Control Theory (CT). CT was historically used to
manage linear system parameters [19,37,44]. CT builds
a controller for a system so its output follows a desired
reference signal [33, 43]. However, CT has been shown
to have the following three problems: 1) CT tends to be
unstable in controlling non-linear systems [48, 49]. Al-
though some variants were proposed, they do not scale
well. 2) CT cannot handle non-numeric parameters; and
3) CT requires a lot of data during the learning phase,
called identiﬁcation to build a good controller.
Supervised Machine Learning (ML). Supervised
ML has been successfully applied in various domains [9,
10, 56, 81]. However, the accuracy of ML models de-
pends heavily on the quality and amount of training
data [81], which is not available or impossible to collect
for large parameter spaces such as ours.
Therefore, we feel that neither CT nor supervised ML,
in their current state, are the ﬁrst choice to directly and
efﬁciently apply for auto-tuning storage systems. That
said, they constantly evolve and new promising results
appear in research literature [4, 67, 69, 86]; we plan to
investigate them in the future.
2.4 Uniﬁed Framework
Most optimization techniques are known to follow the
exploration-exploitation dilemma [23, 46, 68, 79]. Here
we summarize the aforementioned methods by extend-
ing the uniﬁed framework with a third factor, thehistory.
Our uniﬁed view thus deﬁnes three factors or dimen-
sions: ■ (1) Exploration deﬁnes how the technique
searches unvisited areas. This often includes a com-
bination of pure random and also guided search based
on history. ■ (2) Exploitation deﬁnes how the tech-
nique leverages history to ﬁnd next sample. ■ (3) His-
tory deﬁnes how much data from previous evaluations
is kept. History information can be used to help guide
both future exploration and exploitation (e.g., avoiding
less promising regions, or selecting regions that have
never been explored before). Table 1 summarizes how
the aforementioned techniques work by maintaining the
balance among these three key factors. For example,
GA keeps the evaluation results from the last genera-
tion, which corresponds to the concept of history. GA
then exploits the stored information, applying selection
and crossover to search nearby areas and pick the next
generation. Occasionally, it also randomly mutates some
chosen parameters, which is the idea of exploration. As
shown in §4, the trade-off among exploration, exploita-
tion, and history determines the effectiveness and efﬁ-
ciency of these optimization techniques.
3 Experimental Settings
We now describe details of the experimental environ-
ments, parameter spaces, and our implementations of
optimization algorithms.
896    2018 USENIX Annual Technical Conference USENIX Association
Param. Abbr. Values
File System FS Ext2, Ext3, Ext4, XFS, Btrfs,
Nilfs2, Reiserfs
Block (Leaf) Size BS 1K, 2K, 4K
Inode Size,
Sector Size IS n/a, 128, 256, 512, 1024, 2048,
4096, 8192
Block Group BG n/a, 2, 4, 8, 16, 32, 64, 128, 256
Journal Option JO
n/a, order=strict, order=relaxed,
data=journal, data=ordered,
data=writeback
Atime Option AO relatime, noatime
Special Option SO n/a, compress, nodatacow,
nodatasum, notail
I/O Scheduler I/O noop, cfq, deadline
Table 2: Details of parameter spaces.
Hardware. We performed experiments on two sets of
machines with different hardware categorized as low-
end (M1) and mid-range (M2). We list the hardware
details in Table 3. We also use Watts Up Pro ES power
meters to measure the energy consumption [82].
Workload. We benchmarked storage conﬁguration
with four typical macro-workloads generated by
Filebench [25, 75]. ■ (1) Mailserver emulates the I/O
workload of a multi-threaded email server. ■ (2) File-
server emulates the I/O workload of a server that hosts
users’ home directories. ■ (3) Webserver emulates the
I/O workload of a typical static Web server with a high
percentage of reads. ■ (4) Dbserver mimics the behav-
iors of Online Transaction Processing (OLTP) databases.
Before each experiment run, we formatted and mounted
the storage devices with the targeted ﬁle system.
The working set size affects the duration of an ex-
periment [74]. Our goal in this study was to explore
a large set of parameters and values quickly (although
it still took us over two years). We therefore decided
to trade the working set size in favor of increasing the
number of conﬁgurations we could explore in a practi-
cal time period. We used the default working set sizes
in Filebench, and ran each workload for 100 seconds;
this is long enough to get stable evaluation results under
this setting. The experiments demonstrate a wide range
of performance numbers and are suitable for evaluating
different optimization methods.
Parameter Space. Since the main goal of our paper is
to compare multiple optimization techniques, we want
our storage parameter spaces to be large and complex
enough. Alas, evaluations for storage systems take a
long time. Considering experimentation on multiple
hardware settings and workloads, we decided to experi-
ment with a reasonable subset of the most relevant stor-
age system parameters. We selected parameters in close
collaboration with several storage experts that have ei-
ther contributed to storage stack designs or have spent
years tuning storage systems in the ﬁeld. We experi-
Hardware M1 M2
Model Dell PE SC1425 Dell PE R710
CPU Intel Xeon single-core
2.8GHz CPU ×2
Intel Xeon quad-core
2.4GHz CPU ×2
Memory 2GB 24GB
Storage
HDD1 (73GB
Seagate
ST373207LW SCSI
drive)
HDD2 (147GB SAS),
HDD3 (500GB SAS),
HDD4 (250GB
SATA), SSD (200GB)
Table 3: Details of experiment machines.
mented with 7 Linux ﬁle systems that span a wide range
of designs and features: Ext2 [13], Ext3 [77], Ext4 [24],
XFS [73], Btrfs [61], Nilfs2 [42], and Reiserfs [60].
Our experiments were mainly conducted on two sets
of parameters, termed as Storage V1 and Storage V2.
We started with seven common ﬁle system parameters
(shown in the ﬁrst 7 rows of Table 2), and refer it asStor-
age V1. Storage V1was tested on M1 machines. We then
extended our search space with one more parameter, the
I/O Scheduler, and refer to it as Storage V2. Storage V2
was evaluated on M2 servers. Note that certain combi-
nations of parameter values could produce invalid con-
ﬁgurations. For example, for Ext2, the journaling option
makes no sense because Ext2 does not have a journal. To
handle this, we added a valuen/a to the existing range of
parameters. Any parameter with n/a value is considered
invalid. Invalid conﬁgurations will always come with
evaluation results of zero (i.e., no throughput); this en-
sures they are purged in an upcoming optimization pro-
cess. There are 2,074 valid conﬁgurations in Storage
V1 and 6,222 in Storage V2 for each workload and stor-
age device. We believe our search spaces are large and
complex enough to demonstrate the difference in efﬁ-
ciency of various optimization algorithms. Furthermore,
many of the chosen parameters are commonly tuned and
studied by storage experts; having a basic understanding
of such parameters helped us understand auto-tuning re-
sults.
Experiments and implementations. Our experi-
ments and implementation consist of two parts. First,
we exhaustively ran all conﬁgurations for each work-
load and device on M1 and M2 machines, and stored
the results in a relational database. We collected the
throughput in terms of I/O operations per second, as re-
ported by Filebench, the running time (including setup
time), as well as power and energy consumption. To ac-
quire more accurate and stable results, we evaluated each
conﬁguration under the same environment for at least 3
runs, resulting in more than 450,000 total experimen-
tal runs. This data collection beneﬁted our evaluation
on auto-tuning as we can simply simulate a variety of
algorithms by just querying the database for the evalua-
tion results for different conﬁgurations, without having
to rerun slow I/O experiments. The exhaustive search
USENIX Association 2018 USENIX Annual Technical Conference    897
Hardware- File Block Inode BG Journal Atime Special I/O Through-
Workload-Device System Size Size Count Options Options Options Scheduler put (IOPS)
M1-Mail-HDD1 Nilfs2 2K n/a 256 order=relaxed relatime n/a - 3,677
M2-Mail-HDD3 Ext2 4K 256 32 n/a relatime n/a noop 18,744
M2-File-HDD3 Btrfs 4K 4,096 n/a n/a relatime compress deadline 16,549
M2-Mail-SSD Ext2 4K 256 8 n/a relatime n/a noop 18,845
M2-DB-SSD Ext4 1K 128 2 data=ordered noatime n/a noop 41,948
M2-Web-SSD Ext4 4K 128 4 data=ordered noatime n/a noop 16,185
Table 4: Global optimal conﬁgurations with different settings and workloads.
also lets us know exactly what the global optimal con-
ﬁgurations are, so that we can calculate how close each
optimization method gets to the global optimum.
Second, we simulated the process of auto-tuning stor-
age systems by running the desired optimization method
and querying the database for the average evaluation re-
sults from multiple ( 3+) repeated runs. We focused on
optimizing for throughput in this paper. The computa-
tion cost of optimization algorithms are ignored in our
experiments. We believe our observations are applica-
ble to other optimization objectives as well. Our im-
plementations of optimization methods are mostly based
on open-source libraries. We use Pyevolve [58] for
Genetic Algorithms, Scikit-Optimize [70] for Bayesian
Optimization, and TensorFlow [76] for the DQN im-
plementation. We implemented a simple version of
Simulated Annealing, with both linear and geometric
cooling schedules. (We also ﬁxed bugs in Pyevolve
and plan to release our patches.) Most of our imple-
mentation was done by converting storage-related con-
cepts into algorithm-speciﬁc ones. For example, for
GA, we deﬁned each storage parameter as a gene, and
each conﬁguration as a chromosome. For DQN we
provided storage-speciﬁc deﬁnitions for states, actions,
and rewards. The complete implementation uses around
10,000 lines of Python code.
4 Evaluations
Our evaluation mainly focuses on comparing the effec-
tiveness and speed of applying multiple optimization
techniques on auto-tuning storage systems, and provid-
ing insights into our observations. §4.1 overviews the
data sets that we collected for over two years. §4.2 com-
pares ﬁve popular optimization techniques from several
aspects. §4.3 uses GA as a case study to show that hyper-
parameters of these methods can also impact the auto-
tuning results. §4.4 takes the ﬁrst step towards explain-
ing these black-box optimization methods, based on our
evaluation results and our storage expertise.
4.1 Overview of Data Sets
As per §3, our experimental methodology is to ﬁrst ex-
haustively run all conﬁgurations under different work-
loads and test machines. We stored the results in a
database for future use. This data collection beneﬁts
future experiments as we can simulate a variety of al-
gorithms by querying the database for the evaluation re-
sults of different conﬁgurations. Due to space limits, in
this section we show only 6 representative data sets out
of 18: 2 workloads on M1 and 4 devices ×4 workloads
on M2. They were picked to (1) show a wide range of
hardware and workloads’ impact on optimization results
and (2) to present more SSD results, given SSDs’ in-
creasing popularity.
Figure 3 shows the throughput CDF among all con-
ﬁgurations for each hardware setting and workload. The
Y-axis is normalized by the maximum throughput un-
der each experiment setting. The symbols on each line
mark the default Ext4 conﬁgurations. As seen, for most
settings, throughput values vary across a wide range.
The ratios of the worst throughput to the best one are
mostly between 0.2–0.4. In one extreme case, for File-
server on M2 machines and with the HDD3 device (ab-
breviated as M2-Fileserver-HDD3), the worst conﬁgu-
ration only produces 1% I/O operations per unit time,
compared with the global optimal one. This under-
lines the importance of tuning storage systems: an im-
properly conﬁgured system could be remarkably under-
utilized, and thus wasting a lot of resources. How-
ever, M2-Webserver-SSD shows a much narrower range
of throughput, with the worst-to-best ratio close to 0.9.
This is attributed mainly to the fact that Webserver
consists of mostly sequential read operations that are
processed similarly by different I/O stack conﬁgura-
tions. Figure 3 also shows that default Ext4 conﬁgu-
rations are always sub-optimal and, under most settings,
ranked lower than the top 40% conﬁgurations. For M1-
Mailserver-HDD1, the default Ext4 conﬁguration shows
a normalized throughput of 0.39, which means that the
optimal conﬁguration performs 2.5 times better.
Table 4 lists optimal conﬁgurations for the same six
hardware and workload settings. As we can see, optimal
conﬁgurations depend on the speciﬁc hardware as well
as the running workload. For M1-Mailserver-HDD1,
the global best is a Nilfs2 conﬁguration. However, if
we ﬁx the workload, change the hardware, and get M2-
Mailserver-HDD3, the optimum becomes an Ext4 con-
ﬁguration. Similarly, ﬁxing the hardware to M2-*-SSD
and experimenting under different workloads leads to
different optimal conﬁgurations. This proves our early
898    2018 USENIX Annual Technical Conference USENIX Association
 0
 0.2
 0.4
 0.6
 0.8
 1
 0% 20% 40% 60% 80% 100%
Normalized Throughput
Percentage of Configurations
M1-Mail-HDD1
M2-Mail-HDD3
M2-File-HDD3
M2-Mail-SSD
M2-Db-SSD
M2-Web-SSD
Figure 3: Throughput CDF with different hardware and work-
loads, with symbols marking the default Ext4 conﬁgurations.
claim that performance is sensitive to the environment
(e.g., hardware, conﬁguration, and workloads); this ac-
tually complicates the problem as results from one envi-
ronment cannot be directly applied in another.
4.2 Comparative Analysis
Many optimization techniques have been applied to var-
ious auto-tuning tasks [71, 78]. However, previous ef-
forts picked algorithms somewhat arbitrarily and eval-
uated only one algorithm at a time. Here we provide
the ﬁrst comparative study of multiple black-box opti-
mization techniques on auto-tuning storage systems. As
discussed in §2.2, we focus our evaluations on a rep-
resentative set of optimization methods, and their com-
mon hyper-parameter settings, including 1) Simulated
Annealing (SA), with a linear cooling schedule; 2) Ge-
netic Algorithms (GAs) with population size of 8, mu-
tation rate of 2%; 3) Deep Q-Networks (DQN) with
experience replay [55] and ϵ = 0.2, where ϵ; repre-
sents the probability of an agent taking random actions.
4) Bayesian Optimization (BO) with Expected Improve-
ment (EI) and Gaussian prior; and 5) Random Search
(RS), which merely performs random selection without
replacement. We provide more discussion on the im-
pact of hyper-parameters in §4.3. Note that SA, DQN,
and RS experiments start with the default Ext4 conﬁgu-
ration. GA and BO require several initial conﬁgurations
(prior points), which we set to default conﬁgurations of
all seven ﬁle systems. This allows us to simulate real-
world use cases, where users often deploy their system
with the default settings (and may manually optimize
starting from the defaults). In the current experiments
we assume that changing parameter values comes at no
cost. In reality, parameters like Block Size may require
re-formatting ﬁle systems.
Figure 4 presents one simulated run of each optimiza-
tion method on M2-Mailserver-HDD3; the Y-axis shows
the throughput value of the best conﬁguration found so
far, and the X-axis is the running time. All time-related
metrics in this paper are based on the actual running
time of evaluating each storage conﬁguration, which is
stored in our database. This includes both setup time and
benchmarking time. We are not comparing the running
16
17
18
 
 0  1  2  3  4  5
Best
Throughput (kops/s)
Time (hrs)
15.2
18.7
M2-Mailserver-HDD3
GASABODQNRS
Figure 4: Highest throughput found over time, zooming in the
Y ∈[15 : 19]range. The blue number (15.2) on the Y-axis
shows the default, and the red one (18.7) shows the optimal.
costs (including any necessary training phases) for opti-
mization methods here, which is our future work. Fig-
ure 4 is plotted by zooming in the range ofY ∈[15 : 19],
with the blue number (15.2) on Y-axis represents the de-
fault, while the red one (18.7) shows the global optimal.
Here we deﬁne a near-optimal conﬁguration as one with
throughput higher than 99% of the global optimal value.
As shown in Figure 4, all ﬁve methods were able to grad-
ually ﬁnd higher performing conﬁgurations, but their ef-
fectiveness and speed differed a lot. SA performed the
worst, and got stuck in a conﬁguration with throughput
value of less than 18K IOps. DQN was able to con-
verge to a good conﬁguration, but spent more time to
achieve that than RS. GA and BO performed best out of
these ﬁve tested optimization methods. They both suc-
cessfully identiﬁed a near-optimal conﬁguration within
one hour. Interestingly, we observed that pure Random
Search (RS) produced better results than some other op-
timization methods; the reason is that within the search
space M2-Mailserver-HDD3, 4.5% of total conﬁgura-
tions are near-optimal. RS needs only to hit one of them
to reach good auto-tuning results.
Since exploration is one critical component of opti-
mizations (see §2.4), their evaluation results could also
exhibit some degree of randomness. To compare them
more thoroughly, we ran each optimization technique on
the same environment for 1,000 runs. Figure 5 shows
the results, which evaluate the techniques’ probability to
ﬁnd near-optimal conﬁgurations, deﬁned the same as in
Figure 4. The Y-axis shows the percentage of total runs
that found a near-optimal conﬁguration within a certain
time (X-axis). Under M2-Mailserver-HDD3, seen in the
upper part of Figure 5, SA had the lowest probability
among 5 algorithms. Even after 5 hours, only around
80% of its runs found one near-optimal conﬁguration,
which shows that SA sometimes gets stuck in a local op-
timum. For other optimization methods, given enough
time, over 90% of their runs converged to a near-optimal
conﬁguration, with BO outperforming GA, and GA out-
performing DQN. RS shows the highest probability of
ﬁnding near-optimal conﬁgurations when approaching 5
hours. This is reasonable because given enough time, a
random selection will eventually hit near-optimal points.
USENIX Association 2018 USENIX Annual Technical Conference    899
20%
40%
60%
80%
100%
     
Percentage of Runs
M2-Mailserver-HDD3
RSSAGADQNBO
20%
40%
60%
80%
100%
 0  1  2  3  4  5
Percentage of Runs
Time (hrs)
M2-Fileserver-HDD3
Figure 5: Comparing optimization methods’ efﬁcacy in ﬁnding
near-optimal conﬁgurations. The Y-axis shows the percentage
of total runs (1,000) that found near-optimal conﬁgurations
within certain time (X-axis).
However, when conducting the same experiments under
M3-Fileserver-HDD3, it becomes more difﬁcult to ﬁnd
near-optimal conﬁgurations. GA and BO are still the
best, though only 65% of their runs were able to ﬁnd
near-optimal conﬁgurations within 5 hours. SA, RS,
and DQN have a probability of lower than 40% to do
so, with DQN perform the worst. This is because the
global optimum under M2-Fileserver-HDD3 is a Btrfs
conﬁguration (see Table 4). It is more difﬁcult for opti-
mization algorithms to pick such conﬁgurations for the
following reasons: 1) Few Btrfs conﬁgurations reside
in the neighborhood of the default Ext4 conﬁgurations;
2) Fewer than 2% of all valid conﬁgurations are Btrfs
ones, which make them less likely to be selected through
mutation; 3) Only 0.2% of all conﬁgurations are consid-
ered near-optimal, compared with 4.5% in Mailserver-
HDD3.
The above results focused on ﬁnding near-optimal
conﬁgurations. However, another important aspect to
compare is the system’s performance during the auto-
tuning process. This is especially important if the tar-
geted system is deployed and online. Some random-
ness (exploration) is necessary when searching a com-
plex parameter space, but ideally optimization algo-
rithms should spend less time on bad conﬁgurations. To
compare this, in Figure 6 we plotted the instantaneous
throughput (Y-axis) over time (X-axis) for one run with
each method under M2-Mailserver-HDD3.
BO and GA are still the best two methods in terms
of instantaneous throughput. During the tuning process,
occasionally they pick a worse conﬁguration than the
current one. However, they both possess the ability to
quickly discard these unpromising conﬁgurations. GA
achieves this by assigning the probability of surviving to
next generation based on the ﬁtness values (i.e., through-
put). Conﬁgurations with low throughput values have
a lower chance to be picked as parents, and thus their
51015 RS
M2-Mailserver-HDD3
51015 SA
M2-Mailserver-HDD3
51015Throughput (kops/s)
GA
M2-Mailserver-HDD3
51015 DQN
M2-Mailserver-HDD351015
 0  1  2  3  4  5
 BO
Time (hrs)
M2-Mailserver-HDD3
Figure 6: Comparing optimization methods’ instantaneous
performance (Y-axis) over time (X-axis).
genes (parameter values) have a lower chance of appear-
ing in conﬁgurations of the next generation (i.e., “sur-
vival of the ﬁttest”). The reason for stable instantaneous
throughputs with BO is that it uses an intelligent acqui-
sition function to guide the selection of the next genera-
tion, with the goal of maximizing the potential gain; this
makes BO less likely to choose a bad conﬁguration. In
contrast, SA performs poorly possibly because it lacks a
history to guide the exploitation and exploration phases,
and only uses its neighborhood information (and current
temperature) to pick the next conﬁguration. DQN shows
similar results with RS, which is likely caused by the fact
that DQN was originally designed as an agent interacting
with an unknown environment, and thus a lot of explo-
ration (randomness) occurs in the training phase [55,85].
In conclusion, our results demonstrated that the efﬁ-
cacy of different optimization algorithms vary a lot while
applied in auto-tuning storage systems. The trade-off
among exploitation, exploration, and history plays an
important role in ﬁnd near-optimal conﬁgurations efﬁ-
ciently. However, a well-known problem for many op-
timization techniques is that their performance depend
heavily on hyper-parameter settings. Some of our ob-
servations may only apply to our speciﬁc settings and
search spaces. This paper’s main goal is not to provide
guidelines on which methods are more suitable for auto-
tuning storage systems; rather, we focused on comparing
multiple methods and understanding their efﬁcacy under
different conditions.
4.3 Impact of Hyper-Parameters
Many optimization methods’ efﬁcacy depend on the spe-
ciﬁc hyper-parameter settings, and choosing the right
hyper-parameters has caused headache to researchers for
a long time [7, 8]. In this section we use GA as a case
study, and show the impact of one hyper-parameter, the
mutation rate, on auto-tuning results. The mutation rate
controls the probability of randomly mutating one pa-
rameter to a different value, and aligns with the idea of
exploration, as per §2.4.
Figure 7 shows the results from 7 sets of GA exper-
900    2018 USENIX Annual Technical Conference USENIX Association
 0%
20%
40%
60%
80%
100%
 0  2  4  6  8  10
Percentage of Runs
Time (hrs)
1%
2%
4%
8%
16%
32%
64%
Figure 7: Impact of mutation rates on GA.
iments with different mutation rates (from 1% to 64%)
under M2-Mailserver-HDD3. Each experiment was re-
peated for 1,000 runs. It is similar to Figure 5, but with
the goal of ﬁnding near-optimal conﬁgurations whose
throughput values are higher than 99.5% of the global
optimal. This makes the optimization more challeng-
ing, as GA already performs quite well on easier tasks
(§4.2). As shown in the ﬁgure, when increasing the
mutation rate, GA has a higher probability to converge
to near-optimal conﬁgurations within a shorter time pe-
riod. This is because GA works by identifying promis-
ing combination of alleles (parameter values) for the
subset of dominant genes (parameters). We deﬁne dom-
inant parameters as those having a higher impact on per-
formance than all others. A higher mutation rate means a
higher chance of exploration, and thus ﬁnding combina-
tions of well-performing alleles for the dominant genes
within a shorter time. We explain this effect more in
§4.4. However, a mutation rate of 64% actually performs
worse than 32%. This is because in order to reach near-
optimal conﬁgurations, GA needs both exploration and
exploitation. Exploration lets GA identify processing
subspaces (i.e., combinations of certain parameter val-
ues) while exploitation helps GA search within promis-
ing subspaces. In this case, with a mutation rate of 64%,
GA spends too much time on exploration (too much ran-
domness), resulting in fewer chances for exploitation.
4.4 Peering into the Black Box
Despite some successful applications of black-box op-
timization on auto-tuning system parameters, few have
explained how and why some techniques work better
than others for certain problems. Here we take the ﬁrst
step towards unpacking the “black box” and provide
some insights into their internals based on our evalua-
tion results and storage domain knowledge.
Our attempts for explanations stem from a somewhat
unexpected but beneﬁcial behavior of GA in the experi-
ments. We found that as GA runs, there is often a small
set of alleles (parameter values) that dominate the cur-
rent population and are unlikely to change. We present
and explain this observation in Figure 8. The exper-
iment was conducted on a parameter space consisting
of 2,208 Ext3 conﬁgurations under M2-Fileserver-SSD.
The X-axis shows 5 genes (parameters) separated by red
Figure 8: Number of alleles in the ﬁrst 8 GA generations, with
more frequent ones colored with darker colors.
gridlines, while one column represents one allele (pa-
rameter value). The parameters are denoted with their
abbreviations from Table 2. The Y-axis shows the gen-
eration number, and we plotted only the ﬁrst 8 genera-
tions. Cells were colored based on the number of alleles
in each generation. More frequent alleles are colored
with darker colors. In the ﬁrst generation, the gene’s
alleles (parameter values) were quite diverse. For ex-
ample, there were 3 alleles (1K, 2K, 4K) for the Block
Size gene, and 3 alleles (journal, ordered, writeback) for
the Journal Option gene. However, the diversity of al-
leles decreased in later generations, and several genes
began to dominate and even converged to a single allele.
For the Block Size gene, only the 4K allele survived and
other two became extinct. Since GA was proposed by
simulating the process of natural selection, where alleles
with better ﬁtness are more likely to survive, this sug-
gests that GA works by identifying the combination of
good alleles (storage parameter values), and producing
offspring with these alleles. As shown in Figure 8, in
the 8th generation, all conﬁgurations have a Block Size
of 4K and Journal Option of writeback.
To conﬁrm the above observations, in Figure 9 we
plotted all Ext3-SSD conﬁgurations, with one dot corre-
sponding to one conﬁguration. Conﬁgurations are sep-
arated based on the Journal Option , shown as the X-
axis, and colored based on their Block Size. To clearly
see all points within each X-axis section, we ordered
conﬁgurations by their unique identiﬁcation number in
our database. The Y-axis represents throughput val-
ues. This resulted in the formation of nine “clusters” on
the graph, each corresponding to a ﬁxed ⟨Journal Op-
tion, Block Size⟩pair. We can see that conﬁgurations
with data=ordered tend to produce higher throughput
than those with data=journal, and data=writeback pro-
duces the best throughput. This is somewhat expected
from a storage point of view, as Ext3’s more fault tol-
erant journal option ( data=journal) may hurt through-
put by writing data as well as meta-data to the jour-
nal ﬁrst. Moreover, among journal conﬁgurations with
data=writeback, those with a 4K Block Size turn out
to produce the highest throughput. This aligns with
our observation from Figure 8 that GA works by iden-
tifying a subset of genes that have a greater impact
USENIX Association 2018 USENIX Annual Technical Conference    901
 0
 5
 10
 15
data=journal data=ordered data=writeback
Throughput (kops/s)
bs=1K
bs=2K
bs=4K
Figure 9: Scatter plot for all Ext3-SSD conﬁgurations, with
one dot corresponding to one conﬁguration.
on performance— Block Size and Journal Option—and
ﬁnding the best alleles for them ([4K, data=writeback]).
Based on these observations, one interesting question
to ask is whether the conclusion that a subset of pa-
rameters have greater impact on performance than other
parameters, also holds for other ﬁle systems and work-
loads. To answer this question, we quantiﬁed the cor-
relation between each parameter and throughput values.
As most of our parameters are categorical or discrete nu-
meric, whereas the throughput is continuous, we took a
common approach to quantify the correlation between
categorical and continuous variables [14]. We illustrate
with the Block Size parameter as an example. Since it
can take 3 values, we convert this parameter to three bi-
nary variables x1, x2, and x3. If the Block Size is 1K,
we assign x1 = 1 and x2 and x3 are set to 0. Let Y
represent the throughput values. We then do a linear
regression with ordinary least squares (OLS) on Y and
x1,x2,x3. R2 is a common metric in statistics to mea-
sure how the data ﬁts a regression line. In our approach,
R2 actually quantiﬁes the correlation between the se-
lected parameter and throughput. We considerR2 >0.6
as an indication that the parameter has signiﬁcant impact
on performance, as is common in statistics [14]. The
same calculation is applied to all parameters for each
ﬁle system under M2-Fileserver-SSD and M2-Dbserver-
SSD. Parameters with the highest R2 values are colored
in yellow background in Table 5. If all R2 values are
below 0.6, we simply leave the entries blank, meaning
no highly correlated parameters were found. To ﬁnd
the second important parameter, the same process is ap-
plied to the remaining parameters, but with the value of
the most important one ﬁxed (to isolate its effect on the
remaining parameters’ importance). Taking Ext4 under
M2-Fileserver-SSD as an example, we calculate R2 val-
ues for all other parameters among conﬁgurations with
the same Journal Option. For one parameter, 3 Journal
Options lead to three R2 values; we then take the maxi-
mum one as the R2 value for this parameter. We color
the parameter with the highestR2 in Table 5 with a green
background.
We can see that the correlated parameters are quite
diverse, and depend a lot on ﬁle systems. For exam-
ple, under M2-Fileserver-SSD, the two most important
WL-Dev FS BS IS BG JO AO SO I/O
File-SSD Ext2 - - - - - - 0.68
Ext3 0.84 - - 0.90 - - -
Ext4 0.92 - - 0.99 - - -
XFS 0.94 - 0.82 - - - -
Btrfs - - - - - - -
Nilfs2 0.99 - - - - - 0.94
Reiserfs - - - 0.74 - - 0.99
Db-SSD Ext2 - - - - - - -
Ext3 0.72 - - 0.96 - - -
Ext4 - - - 0.96 0.68 - -
XFS - - - - - - -
Btrfs - - - - - - -
Nilfs2 0.62 - - - - - 0.80
Reiserfs - - - 0.99 - - -
Table 5: Importance of parameters (measured byR2), with the
most important one colored in yellow and second in green.
parameters for Ext3 (in descending order) are Journal
Option and Block Size; this aligns with our observation
in Figures 8 and 9. However, for Reiserfs, the top 2
changes to I/O Scheduler and Journal Option. Interest-
ingly, all parameters for Btrfs come with lowR2 values,
which indicates that no parameter has signiﬁcant impact
on system performance under M2-Fileserver-SSD with
Btrfs. Correlation of parameters can also depend on
the workloads. For instance, the two dominant param-
eters for XFS under M2-Fileserver-SSD are Block Size
and Allocation Group. When the workload changes to
M2-Dbserver-SSD, all parameters for XFS seem to have
minor impact on performance. In this paper we are iso-
lating the impact of each parameter, thus assuming that
their effect on throughput is independent. Note that the
above observations are made based on our collected data
sets, and might change on different workloads and hard-
ware. However, our methodology is generally applica-
ble. Moreover, this paper’s main goal is not to suggest
guidelines on what speciﬁc storage conﬁgurations to de-
ploy under certain workloads; rather, we focus on com-
paring multiple optimization methods and providing in-
sights into their operation.
The fact that parameters have varied impact on per-
formance can also help explain the auto-tuning results
in §4.2. Although our parameter space comes with 8
parameters, only a subset of them are highly correlated
with performance. As long as the optimization algo-
rithm identiﬁes the “correct” combination of values for
these dominant parameters, it will be able to ﬁnd a near-
optimal conﬁguration. Similar behavior has been re-
ported in hyper-parameter optimization problems [7].
For the experiments shown in Figure 4, near-optimal
conﬁgurations take up 4.5% of the whole search space.
Random Search (RS) needs to hit only one of them to
achieve good auto-tuning results. GA’s efﬁcacy comes
from assigning a higher chance of survival to conﬁgura-
902    2018 USENIX Annual Technical Conference USENIX Association
tions with a certain combination of values for the dom-
inant parameters. BO stores its previous search experi-
ence (history) in a probabilistic surrogate model that it
is building, which eventually encodes the combination
of dominant parameter values that can result in good
throughput values. SA does not work as well because
it lacks history information to identify the dominant pa-
rameters: it wastes time changing less useful parame-
ters and converges slowly. Similarly, DQN also spends
lots of its effort on exploring unpromising spaces, which
slows its ability to ﬁnd near-optimal conﬁgurations.
5 Limitations and Future Work
In this paper we provided the ﬁrst comparative analy-
sis of applying multiple optimization methods on auto-
tuning storage systems. However, auto-tuning is a com-
plex topic and more effort is required. We list some
limitations of this work and our future research direc-
tions below. ■ (1) We plan to extend the scope of
evaluation with more complex workloads and search
spaces. We will investigate more techniques, such as
experiment design [80], as will as the impact of algo-
rithm hyper-parameter settings [8]. ■ (2) We plan to im-
prove traditional optimization techniques with new fea-
tures, such as penalty functions to cope with costly pa-
rameter changes, stopping/restarting criteria, workload
identiﬁcation, handling noisy and unstable results [12],
etc., which makes auto-tuning algorithms more robust to
environment changes and more generally applicable in
production systems.
6 Related Work
Auto-tuning computer systems. In recent years, sev-
eral attempts were made to automate the tuning of stor-
age systems. Strunk et al. [71] proposed to use util-
ity functions combining different system metrics and
applied GA to automate storage system provisioning.
Babak et al. [5] utilized GA to optimize I/O performance
of HDF5 applications. GA has also been applied for
storage recovery problems [38]. More recently, Deep Q-
Networks has been successfully applied in optimizing
performance for Lustre [85]. Auto-tuning is also a hot
topic in other computer systems: Bayesian Optimiza-
tion was applied to ﬁnd near-optimal conﬁgurations for
databases [78] and Cloud VMs [3]. Other applied tech-
niques include Evolutionary Strategies [62], Simulated
Annealing [26, 35], Tabu Search [63], and more. How-
ever, previous work all focused on one or a few tech-
niques. One contribution of our work is to provide the
ﬁrst comparative study of multiple, applicable optimiza-
tion methods on their efﬁcacy in auto-tuning storage sys-
tems from various aspects. We also provide some in-
sights into the working mechanism of auto-tuning.
Hyper-parameter tuning. Evolutionary Algo-
rithms [59], Reinforcement Learning [6], and Bayesian
Optimization [22] have been applied to hyper-parameter
optimization for ML algorithms. Bergstra and Ben-
gio [8] found that randomly chosen trials are more
efﬁcient for hyper-parameter optimization than trials on
a grid, and explained the cause as the objective function
having a low effective dimensionality. Another direction
of research focuses on eliminating all hyper-parameters
and tries to propose non-parametric versions of opti-
mization methods. Examples of this include GA [31,51]
and BO [68].
7 Conclusions
Optimizing storage systems can provide signiﬁcant ben-
eﬁts especially in improving I/O performance. Alas,
storage systems are getting more complex, contain
many parameters and an immense number of possi-
ble conﬁgurations; manual tuning is therefore imprac-
tical. Worse, many of those parameters are non-linear
or non-numeric; traditional linear-regression-based opti-
mization techniques do not work well for such problems.
Several efforts were made to apply black-box optimiza-
tion techniques to auto-tune storage systems, but they
all used only one or few techniques. In this work, we
performed the ﬁrst comparative study, and offered the
following four contributions. (1) We evaluatedﬁve pop-
ular but different auto-tuning techniques, varied some of
their hyper-parameters, and applied them to storage and
ﬁle systems. (2) We show that the speed at which the
techniques can ﬁnd optimal or near-optimal conﬁgura-
tions (in terms of throughput) depends on the hardware,
software, and workload; this means that no single tech-
nique can “rule them all.” (3) We explain why some
techniques appear to work better than others. (4) For
more than two years, we have collected a large data set
of over 450,000 data points; this data set was used in this
study and we plan to release it.
Acknowledgments
We thank the anonymous ATC reviewers and our shep-
herds Keith Smith and Siddhartha Sen for their valu-
able comments. This work was made possible in part
thanks to Dell-EMC, NetApp, and IBM support; NSF
awards CNS-1251137, CNS-1302246, CNS-1305360,
CNS-1622832, CNS-1650499, and CNS-1730726; and
ONR award N00014-16-1-2264.
References
[1] Abutalib Aghayev, Mansour Shafaei, and Peter
Desnoyers. Skylight—a window on shingled disk
operation. Trans. Storage, 11(4):16:1–16:28, Oc-
tober 2015.
[2] Abutalib Aghayev, Theodore Ts’o, Garth Gibson,
and Peter Desnoyers. Evolving ext4 for shingled
USENIX Association 2018 USENIX Annual Technical Conference    903
disks. In Proceedings of the 15th USENIX Con-
ference on File and Storage Technologies (FAST),
pages 105–120, Santa Clara, CA, February/March
2017. USENIX Association.
[3] Omid Alipourfard, Hongqiang Harry Liu, Jianshu
Chen, Shivaram Venkataraman, Minlan Yu, and
Ming Zhang. Cherrypick: Adaptively unearthing
the best cloud conﬁgurations for big data analytics.
In 14th USENIX Symposium on Networked Sys-
tems Design and Implementation (NSDI 17), pages
469–482. USENIX Association, 2017.
[4] Terry Anderson. The theory and practice of online
learning. Athabasca University Press, 2008.
[5] Babak Behzad, Huong Vu Thanh Luu, Joseph
Huchette, Surendra Byna, Prabhat, Ruth Aydt,
Quincey Koziol, and Marc Snir. Taming parallel
i/o complexity with auto-tuning. In Proceedings of
the International Conference on High Performance
Computing, Networking, Storage and Analysis, SC
’13, pages 68:1–68:12, New York, NY , USA, 2013.
ACM.
[6] Irwan Bello, Hieu Pham, Quoc V Le, Mohammad
Norouzi, and Samy Bengio. Neural combinatorial
optimization with reinforcement learning. arXiv
preprint arXiv:1611.09940, 2016.
[7] James Bergstra and Yoshua Bengio. Random
search for hyper-parameter optimization. Journal
of Machine Learning Research, 13(Feb):281–305,
2012.
[8] James S. Bergstra, R ´emi Bardenet, Yoshua Bengio,
and Bal´azs K´egl. Algorithms for hyper-parameter
optimization. In Advances in Neural Information
Processing Systems 24, pages 2546–2554, 2011.
[9] Christopher M Bishop. Pattern Recognition and
Machine Learning, volume 1. Springer New York,
2006.
[10] Leo Breiman, Jerome Friedman, Charles J Stone,
and Richard A Olshen. Classiﬁcation and regres-
sion trees. CRC press, 1984.
[11] Eric Brochu, Vlad M Cora, and Nando De Fre-
itas. A tutorial on bayesian optimization of expen-
sive cost functions, with application to active user
modeling and hierarchical reinforcement learning.
arXiv preprint arXiv:1012.2599, 2010.
[12] Zhen Cao, Vasily Tarasov, Hari Raman, Dean
Hildebrand, and Erez Zadok. On the performance
variation in modern storage stacks. In Proceedings
of the 15th USENIX Conference on File and Stor-
age Technologies (FAST) , pages 329–343, Santa
Clara, CA, February/March 2017. USENIX Asso-
ciation.
[13] R. Card, T. Ts’o, and S. Tweedie. Design and im-
plementation of the second extended ﬁlesystem. In
Proceedings to the First Dutch International Sym-
posium on Linux , Amsterdam, Netherlands, De-
cember 1994.
[14] George Casella and Roger L Berger. Statistical In-
ference, volume 2. Duxbury Paciﬁc Grove, CA,
2002.
[15] Vladim ´ır ˇCern`y. Thermodynamical approach to
the traveling salesman problem: An efﬁcient sim-
ulation algorithm. Journal of optimization theory
and applications, 45(1):41–51, 1985.
[16] Yvonne Coady, Russ Cox, John DeTreville, Peter
Druschel, Joseph Hellerstein, Andrew Hume, Kim-
berly Keeton, Thu Nguyen, Christopher Small, Lex
Stein, and Andrew Warﬁeld. Falling off the cliff:
When systems go nonlinear. In Proceedings of the
10th Conference on Hot Topics in Operating Sys-
tems - Volume 10, HOTOS’05, 2005.
[17] Valentin Dalibard, Michael Schaarschmidt, and
Eiko Yoneki. Boat: Building auto-tuners with
structured bayesian optimization. In Proceedings
of the 26th International Conference on World
Wide Web , WWW ’17, pages 479–488. Inter-
national World Wide Web Conferences Steering
Committee, 2017.
[18] Kenneth Alan De Jong. Analysis of the behavior
of a class of genetic adaptive systems . PhD the-
sis, University of Michigan, Ann Arbor, Ann Ar-
bor, MI, USA, 1975.
[19] Y . Diao, J. L. Hellerstein, A. J. Storm, M. Surendra,
S. Lightstone, S. Parekh, and C. Garcia-Arellano.
Using MIMO linear control for load balancing in
computing systems. In 2004 American Control
Conferences, 2004.
[20] Marco Dorigo, Mauro Birattari, and Thomas
St¨utzle. Ant colony optimization. Computational
Intelligence Magazine, IEEE, 1(4):28–39, 2006.
[21] Fred Douglis, Deepti Bhardwaj, Hangwei Qian,
and Philip Shilane. Content-aware load balancing
for distributed backup. In Large Installation Sys-
tem Administration Conference (LISA), 2011.
[22] Katharina Eggensperger, Matthias Feurer, Frank
Hutter, James Bergstra, Jasper Snoek, Holger
Hoos, and Kevin Leyton-Brown. Towards an em-
pirical foundation for assessing bayesian optimiza-
tion of hyperparameters. In NIPS workshop on
Bayesian Optimization in Theory and Practice ,
volume 10, 2013.
904    2018 USENIX Annual Technical Conference USENIX Association
[23] A.E. Eiben and C.A. Schippers. On evolutionary
exploration and exploitation. Fundam. Inf., 35(1-
4):35–50, January 1998.
[24] Ext4. http://ext4.wiki.kernel.org/ .
[25] Filebench, 2016. https://github.com/ﬁlebench/
ﬁlebench/wiki .
[26] Terry L Friesz, Hsun-Jung Cho, Nihal J Mehta,
Roger L Tobin, and G Anandalingam. A simulated
annealing approach to the network design problem
with variational inequality constraints.Transporta-
tion Science, 26(1):18–26, 1992.
[27] Fred Glover and Manuel Laguna. Tabu Search.
Springer, 2013.
[28] Ian Goodfellow, Yoshua Bengio, and Aaron
Courville. Deep Learning . MIT Press, 2016.
http://www.deeplearningbook.org .
[29] Gradient descent. https://en.wikipedia.org/wiki/
Gradient descent.
[30] Mingzhe Hao, Gokul Soundararajan, Deepak
Kenchammana-Hosekote, Andrew A Chien, and
Haryadi S Gunawi. The tail at store: a revela-
tion from millions of hours of disk and ssd deploy-
ments. In 14th USENIX Conference on File and
Storage Technologies (FAST 16), pages 263–276,
2016.
[31] Georges R Harik and Fernando G Lobo. A
parameter-less genetic algorithm. In GECCO, vol-
ume 99, pages 258–267, 1999.
[32] Weiping He and David H.C. Du. Smart: An ap-
proach to shingled magnetic recording translation.
In Proceedings of the 15th USENIX Conference
on File and Storage Technologies (FAST) , pages
121–134, Santa Clara, CA, February/March 2017.
USENIX Association.
[33] J. L. Hellerstein, Y . Diao, S. Parekh, and D. M.
Tibury. Feedback Control of Computing Systems .
Wiley-IEEE Press, 2004.
[34] J. H. Holland. Adaptation in natural and artiﬁcial
systems: An introductory analysis with applica-
tions to biology, control, and artiﬁcial intelligence.
U. Michigan Press, 1975.
[35] Young-Jae Jeon, Jae-Chul Kim, Jin-O Kim, Joong-
Rin Shin, and Kwang Y Lee. An efﬁcient sim-
ulated annealing algorithm for network reconﬁg-
uration in large-scale distribution systems. Power
Delivery, IEEE Transactions on, 17(4):1070–1078,
2002.
[36] N. Joukov, A. Traeger, R. Iyer, C. P. Wright, and
E. Zadok. Operating system proﬁling via latency
analysis. In Proceedings of the 7th Symposium
on Operating Systems Design and Implementation
(OSDI 2006), pages 89–102, Seattle, W A, Novem-
ber 2006. ACM SIGOPS.
[37] M. Karlsson, C. Karamanolis, and X. Zhu. Triage:
Performance differentiation for storage systems us-
ing adaptive control. ACM Trans. Storage, 1(4),
2005.
[38] Kimberly Keeton, Dirk Beyer, Ernesto Brau, Arif
Merchant, Cipriano Santos, and Alex Zhang. On
the road to recovery: Restoring data after disasters.
In Proceedings of the 1st ACM SIGOPS/EuroSys
European Conference on Computer Systems 2006,
EuroSys’06, pages 235–248, New York, NY , USA,
2006. ACM.
[39] James Kennedy and Russell C. Eberhart. Parti-
cle swarm optimization. In Proceedings of the
IEEE International Conference on Neural Net-
works, pages 1942–1948, 1995.
[40] H. Kim, S. Seshadri, C. L. Dickey, and L. Chiu.
Evaluating phase change memory for enterprise
storage systems: A study of caching and tiering ap-
proaches. In Proceedings of the 12th USENIX Con-
ference on File and Storage Technologies , pages
33–45, Berkeley, CA, 2014. USENIX.
[41] S. Kirkpatrick, C D. Gelatt, M. P Vecchi, et al.
Optimization by simulated annealing. Science,
220(4598):671–680, 1983.
[42] Ryusuke Konishi, Yoshiji Amagai, Koji Sato,
Hisashi Hifumi, Seiji Kihara, and Satoshi Moriai.
The linux implementation of a log-structured ﬁle
system. ACM SIGOPS Operating Systems Review,
40(3):102–107, 2006.
[43] Ernest Bruce Lee and Lawrence Markus. Founda-
tions of optimal control theory. Technical report,
DTIC Document, 1967.
[44] H. D. Lee, Y . J. Nam, K. J. Jung, S. G. Jung, and
C. Park. Regulating I/O performance of shared
storage with a control theoretical approach. In
NASA/IEEE Conference on Mass Storage Systems
and Technologies (MSST) . IEEE Society Press,
2004.
[45] Yin Li, Hao Wang, Xuebin Zhang, Ning Zheng,
Shafa Dahandeh, and Tong Zhang. Facilitating
magnetic recording technology scaling for data
center hard disk drives through ﬁlesystem-level
transparent local erasure coding. In Proceedings of
the 15th USENIX Conference on File and Storage
Technologies (FAST), pages 135–148, Santa Clara,
CA, February/March 2017. USENIX Association.
[46] Yuxi Li. Deep reinforcement learning: An
overview. arXiv preprint arXiv:1701.07274, 2017.
USENIX Association 2018 USENIX Annual Technical Conference    905
[47] Z. Li, M. Chen, A. Mukker, and E. Zadok. On
the trade-offs among performance, energy, and en-
durance in a versatile hybrid drive. ACM Transac-
tions on Storage (TOS), 11(3), July 2015.
[48] Z. Li, K. M. Greenan, A. W. Leung, and E. Zadok.
Power consumption in enterprise-scale backup
storage systems. In Proceedings of the Tenth
USENIX Conference on File and Storage Tech-
nologies (FAST ’12), San Jose, CA, February 2012.
USENIX Association.
[49] Z. Li, R. Grosu, K. Muppalla, S. A. Smolka, S. D.
Stoller, and E. Zadok. Model discovery for energy-
aware computing systems: An experimental evalu-
ation. In Proceedings of the 1st Workshop on En-
ergy Consumption and Reliability of Storage Sys-
tems (ERSS’11), Orlando, FL, July 2011.
[50] Z. Li, A. Mukker, and E. Zadok. On the importance
of evaluating storage systems’ $costs. In Proceed-
ings of the 6th USENIX Conference on Hot Topics
in Storage and File Systems, HotStorage’14, 2014.
[51] Fernando G Lobo and David E Goldberg. The
parameter-less genetic algorithm in practice. In-
formation Sciences, 167(1):217–232, 2004.
[52] Peter Merz. Memetic algorithms for combinato-
rial optimization problems: Fitness landscapes and
effective search strategies, 2001.
[53] Justin Meza, Qiang Wu, Sanjev Kumar, and Onur
Mutlu. A large-scale study of ﬂash memory fail-
ures in the ﬁeld. In Proceedings of the 2015
ACM International Conference on Measurement
and Modeling of Computer Systems (SIGMETRICS
2015), pages 177–190, Portland, OR, June 2015.
ACM.
[54] V olodymyr Mnih, Koray Kavukcuoglu, David
Silver, Alex Graves, Ioannis Antonoglou, Daan
Wierstra, and Martin Riedmiller. Playing atari
with deep reinforcement learning. arXiv preprint
arXiv:1312.5602, 2013.
[55] V olodymyr Mnih, Koray Kavukcuoglu, David Sil-
ver, Andrei A Rusu, Joel Veness, Marc G Belle-
mare, Alex Graves, Martin Riedmiller, Andreas K
Fidjeland, Georg Ostrovski, et al. Human-level
control through deep reinforcement learning. Na-
ture, 518(7540):529–533, 2015.
[56] Kevin P Murphy. Machine learning: a probabilis-
tic perspective. MIT press, 2012.
[57] Iyswarya Narayanan, Di Wang, Myeongjae Jeon,
Bikash Sharma, Laura Caulﬁeld, Anand Sivasubra-
maniam, Ben Cutler, Jie Liu, Badriddine Khessib,
and Kushagra Vaid. SSD failures in datacenters:
What? when? and why? In Proceedings of the
Second ACM Israeli Experimental Systems Confer-
ence (SYSTOR ’16), pages 7:1–7:11, Haifa, Israel,
May 2016. ACM.
[58] Christian S. Perone. Pyevolve: A python
open-source framework for genetic algorithms.
SIGEVOlution, 4(1):12–20, November 2009.
[59] Esteban Real, Sherry Moore, Andrew Selle,
Saurabh Saxena, Yutaka Leon Suematsu, Quoc Le,
and Alex Kurakin. Large-scale evolution of im-
age classiﬁers. arXiv preprint arXiv:1703.01041 ,
2017.
[60] H. Reiser. ReiserFS v.3 whitepaper.
http://web.archive.org/web/20031015041320/http:
//namesys.com/ .
[61] Ohad Rodeh, Josef Bacik, and Chris Mason.
BTRFS: The Linux B-tree ﬁlesystem. Trans. Stor-
age, 9(3):9:1–9:32, August 2013.
[62] Anooshiravan Saboori, Guofei Jiang, and Haifeng
Chen. Autotuning conﬁgurations in distributed
systems for performance improvements using evo-
lutionary strategies. In Proceedings of the 2008
The 28th International Conference on Distributed
Computing Systems, ICDCS ’08, pages 769–776,
Washington, DC, USA, 2008. IEEE Computer So-
ciety.
[63] Sadiq M Sait, Mahmood R Minhas, Junhaid Khan,
et al. Performance and low power driven vlsi stan-
dard cell placement using tabu search. In Evolu-
tionary Computation, 2002. CEC’02. Proceedings
of the 2002 Congress on , volume 1, pages 372–
377. IEEE, 2002.
[64] F. Schmuck and R. Haskin. GPFS: A shared-disk
ﬁle system for large computing clusters. In Pro-
ceedings of the First USENIX Conference on File
and Storage Technologies (FAST ’02), pages 231–
244, Monterey, CA, January 2002. USENIX Asso-
ciation.
[65] Bianca Schroeder, Raghav Lagisetty, and Arif Mer-
chant. Flash reliability in production: The expected
and the unexpected. In Proceedings of the 14th
USENIX Conference on File and Storage Tech-
nologies (FAST), pages 67–80, Santa Clara, CA,
February 2016. USENIX Association.
[66] P. Sehgal, V . Tarasov, and E. Zadok. Evaluating
performance and energy in ﬁle system server work-
loads. In Proceedings of the USENIX Conference
on File and Storage Technologies (FAST) , pages
253–266, San Jose, CA, February 2010. USENIX
Association.
[67] Burr Settles. Active Learning. Morgan & Claypool
Publishers, 2012.
906    2018 USENIX Annual Technical Conference USENIX Association
[68] Bobak Shahriari, Kevin Swersky, Ziyu Wang,
Ryan P Adams, and Nando de Freitas. Taking the
human out of the loop: A review of bayesian op-
timization. Proceedings of the IEEE, 104(1):148–
175, 2016.
[69] Shai Shalev-Shwartz. Online learning and online
convex optimization. Foundations and Trends in
Machine Learning, 4(2):107–194, 2011.
[70] Scikit-Optimize. https://scikit-optimize.github.io/ .
[71] John D. Strunk, Eno Thereska, Christos Faloutsos,
and Gregory R. Ganger. Using utility to provi-
sion storage systems. In Proceedings of the 6th
USENIX Conference on File and Storage Tech-
nologies, FAST’08, pages 313–328, Berkeley, CA,
USA, 2008. USENIX Association.
[72] Richard S Sutton and Andrew G Barto. Reinforce-
ment learning: An introduction . MIT press Cam-
bridge, 1998.
[73] A. Sweeney, D. Doucette, W. Hu, C. Anderson,
M. Nishimoto, and G. Peck. Scalability in the XFS
ﬁle system. In Proceedings of the Annual USENIX
Technical Conference, pages 1–14, San Diego, CA,
January 1996.
[74] V . Tarasov, S. Bhanage, E. Zadok, and M. Seltzer.
Benchmarking ﬁle system benchmarking: It *is*
rocket science. In Proceedings of HotOS XIII:The
13th USENIX Workshop on Hot Topics in Operat-
ing Systems, Napa, CA, May 2011.
[75] Vasily Tarasov, Erez Zadok, and Spencer Shepler.
Filebench: A ﬂexible framework for ﬁle system
benchmarking. ;login: The USENIX Magazine ,
41(1):6–12, March 2016.
[76] TensorFlow. https://www.tensorﬂow.org/ .
[77] Stephen Tweedie. Ext3, journaling ﬁlesys-
tem. In Ottawa Linux Symposium , July 2000.
http://olstrans.sourceforge.net/release/OLS2000-ext3/
OLS2000-ext3.html .
[78] Dana Van Aken, Andrew Pavlo, Geoffrey J. Gor-
don, and Bohan Zhang. Automatic database man-
agement system tuning through large-scale ma-
chine learning. In Proceedings of the 2017 ACM
International Conference on Management of Data,
SIGMOD ’17, pages 1009–1024, 2017.
[79] Matej ˇCrepinˇsek, Shih-Hsi Liu, and Marjan
Mernik. Exploration and exploitation in evolution-
ary algorithms: A survey. ACM Comput. Surv. ,
45(3):35:1–35:33, July 2013.
[80] Shivaram Venkataraman, Zongheng Yang,
Michael J Franklin, Benjamin Recht, and Ion Sto-
ica. Ernest: Efﬁcient performance prediction for
large-scale advanced analytics. In 13th USENIX
Symposium on Networked Systems Design and
Implementation, pages 363–378, 2016.
[81] Mengzhi Wang, Kinman Au, Anastassia Ailamaki,
Anthony Brockwell, Christos Faloutsos, and Gre-
gory R. Ganger. Storage device performance pre-
diction with cart models. In The IEEE Computer
Society’s 12th Annual International Symposium
on Modeling, Analysis, and Simulation of Com-
puter and Telecommunications Systems. (MAS-
COTS), pages 588–595, 2004.
[82] Watts up? PRO ES power meter. www.
wattsupmeters.com/secure/products.php .
[83] H-S Philip Wong, Simone Raoux, SangBum Kim,
Jiale Liang, John P Reifenberg, Bipin Rajen-
dran, Mehdi Asheghi, and Kenneth E Goodson.
Phase change memory. Proceedings of the IEEE ,
98(12):2201–2227, Dec 2010.
[84] Bowei Xi, Zhen Liu, Mukund Raghavachari,
Cathy H. Xia, and Li Zhang. A smart hill-climbing
algorithm for application server conﬁguration. In
Proceedings of the 13th International Conference
on World Wide Web, WWW ’04, pages 287–296,
New York, NY , USA, 2004. ACM.
[85] Oceane Bel Ethan L. Miller Darrell D. E. Long
Yan Li, Kenneth Chang. Capes: Unsupervised
system performance tuning using neural network-
based deep reinforcement learning. In Proceedings
of the International Conference for High Perfor-
mance Computing, Networking, Storage and Anal-
ysis, SC ’17, 2017.
[86] Yang Yu, Hong Qian, and Yi-Qi Hu. Derivative-
free optimization via classiﬁcation. In Proceed-
ings of the Thirtieth AAAI Conference on Artiﬁcial
Intelligence, AAAI’16, pages 2286–2292. AAAI
Press, 2016.
[87] Erez Zadok, Aashray Arora, Zhen Cao, Akhilesh
Chaganti, Arvind Chaudhary, and Sonam Mandal.
Parametric optimization of storage systems. In
HotStorage ’15: Proceedings of the 7th USENIX
Workshop on Hot Topics in Storage , Santa Clara,
CA, July 2015. USENIX, USENIX.
USENIX Association 2018 USENIX Annual Technical Conference    907

NeuroVectorizer: End-to-End Vectorization with Deep
Reinforcement Learning
Ameer Haj-Ali∗
ameerh@berkeley.edu
University of California, Berkeley
USA
Nesreen K. Ahmed
Ted Willke
nesreen.k.ahmed@intel.com
ted.willke@intel.com
Intel Labs
USA
Yakun Sophia Shao
Krste Asanovic
Ion Stoica
ysshao@berkeley.edu
krste@berkeley.edu
istoica@berkeley.edu
University of California, Berkeley
USA
Abstract
One of the key challenges arising when compilers vectorize
loops for today’s SIMD-compatible architectures is to decide
if vectorization or interleaving is beneficial. Then, the com-
piler has to determine the number of instructions to pack
together and the interleaving level (stride). Compilers are
designed today to use fixed-cost models that are based on
heuristics to make vectorization decisions on loops. How-
ever, these models are unable to capture the data dependency,
the computation graph, or the organization of instructions.
Alternatively, software engineers often hand-write the vec-
torization factors of every loop. This, however, places a huge
burden on them, since it requires prior experience and sig-
nificantly increases the development time.
In this work, we explore a novel approach for handling
loop vectorization and propose an end-to-end solution using
deep reinforcement learning (RL). We conjecture that deep
RL can capture different instructions, dependencies, and data
structures to enable learning a sophisticated model that can
better predict the actual performance cost and determine
the optimal vectorization factors. We develop an end-to-end
framework, from code to vectorization, that integrates deep
RL in the LLVM compiler. Our proposed framework takes
benchmark codes as input and extracts the loop codes. These
loop codes are then fed to a loop embedding generator that
learns an embedding for these loops. Finally, the learned
embeddings are used as input to a Deep RL agent, which
∗Part of this work was done while Ameer Haj-Ali was in a summer internship
at Intel Labs.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not
made or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for components
of this work owned by others than ACM must be honored. Abstracting with
credit is permitted. To copy otherwise, or republish, to post on servers or to
redistribute to lists, requires prior specific permission and/or a fee. Request
permissions from permissions@acm.org.
CGO ’20, February 22ś26, 2020, San Diego, CA, USA
© 2020 Association for Computing Machinery.
ACM ISBN 978-1-4503-7047-9/20/02.
https://doi.org/10.1145/3368826.3377928
dynamically determines the vectorization factors for all the
loops. We further extend our framework to support ran-
dom search, decision trees, supervised neural networks, and
nearest-neighbor search. We evaluate our approaches against
the currently used LLVM vectorizer and loop polyhedral op-
timization techniques. Our experiments show 1 .29 × − 4.73×
performance speedup compared to baseline and only 3%
worse than the brute-force search on a wide range of bench-
marks.
CCS Concepts · Software and its engineering → Com-
pilers.
Keywords Deep Reinforcement Learning, Code Optimiza-
tion, LLVM, Automatic Vectorization.
ACM Reference Format:
Ameer Haj-Ali, Nesreen K. Ahmed, Ted Willke, Yakun Sophia Shao,
Krste Asanovic, and Ion Stoica. 2020. NeuroVectorizer: End-to-End
Vectorization with Deep Reinforcement Learning. In Proceedings
of the 18th ACM/IEEE International Symposium on Code Generation
and Optimization (CGO ’20), February 22ś26, 2020, San Diego, CA,
USA. ACM, New York, NY, USA, 14 pages. https://doi.org/10.1145/
3368826.3377928
1 Introduction
Modern computers typically have vector instructions that
perform multiple basic operations simultaneously, such as
Intel Advanced Vector Extensions (AVX) [ 20]. Converting
a computer program from a scalar implementation, which
processes a single pair of operands at a time to a vector
implementation, which performs a single operation on mul-
tiple data (SIMD) items at once is called vectorization, and is
critical to enhancing the performance of compute-intensive
programs for modern computers.
Loops are among the most commonly vectorized parts
of code. Loop vectorization is done by defining the vector-
ization factor (VF) and the interleaving factor (IF) [
25]. VF
determines how many instructions to pack together from
different iterations of the loop. IF determines the stride of
the memory accesses of the packed instructions. IF allows
242
CGO ’20, February 22ś26, 2020, San Diego, CA, USA Ameer Haj-Ali, Nesreen K. Ahmed, Ted Willke, Yakun Sophia Shao, Krste Asanovic, and Ion Stoica
vectorization to be performed on non-consecutive addresses,
which are generally referred to as non-unit stride accesses.
In most C and C++ compilers it is possible to use intrin-
sic pragmas or compiler passes to manually vectorize loops
by setting the VF and IF. However, manual vectorization is
labor-intensive, error-prone, and results in code that is dif-
ficult to maintain and port. Alternatively, several solutions
for automatic vectorization and loop optimization have been
proposed. The current vectorizer used in LLVM and proposed
improvements [
37, 38], rely on linear and constant-cost mod-
els to predict the vectorization factors. Unfortunately, these
cost models do not consider the computation graph and
focus on estimating the cost of different instructions with
predefined heuristics. Another commonly used approach is
Polly [8]. Polly uses loop polyhedral analysis, which relies on
an abstract mathematical representation, namely equations
and matrices, to represent loops as polytopes. The polytope
representation simplifies the implementation of loop opti-
mizations, though to date the main optimizations in Polly
are tiling and loop fusion to improve data locality.
Machine learning is yet another recent approach that has
been proposed for automatic vectorization [ 35]. While this
approach improves the cost models implemented by existing
compilers, they use hand-engineered heuristics to extract
features from the assembly code, such as arithmetic inten-
sity. Unfortunately, these features are typically not sufficient
to fully capture the code functionality. To overcome this
challenge, [
4] proposed an end-to-end solution that relies
on deep supervised learning. However, supervised learning
methods require labels to train. These labels are not always
available and it can be time-consuming to find them. Further-
more, optimizing for multiple objectives with large search
spaces can be challenging for supervised learning methods.
A human vectorization expert can determine the optimal
vectorization factors, i.e., VF and IF for a specific hardware
architecture by examining the computation graph, function-
ality, operations, and loop bodies in the text code. Similarly,
in this paper, we use a code embedding generator that reads
the text similar to a human expert, "understands" it and then
generates an embedding that represents it. We use the gen-
erated embedding as an input to another neural network
that can learn a mapping from this embedding to optimal
vectorization factors similar to those learned by a human
expert. This approach efficiently addresses the vectorization
challenge end-to-end: from code to optimal factors, enabling
the co-optimization of multiple objectives while preserving
code correctness.
This paper makes the following contributions:
• A comprehensive data set of more than 10,000 syn-
thetic loop examples.
• An end-to-end deep reinforcement learning (RL) [ 36]
based auto-vectorization method.
• An extensible, open-source1 framework that integrates
learning code embedding with multiple machine learn-
ing methods to make vectorization predictions on loops.
We explore using random search, supervised learning
methods, i.e., nearest-neighbor search (NNS) [ 32], deci-
sion trees [ 31], and supervised fully connected neural
network (FCNNs), and contextual bandits based on
deep RL.
• Rigorous evaluations across different learning hyper-
parameters and benchmark suites to show the effec-
tiveness of our approaches versus the currently used
cost model, as well as Polly. Our results show 1 .29 ×
− 4.73× average performance speedup and only 3%
worse than the brute-force solution.
The rest of the paper is organized as follows. In Section 2
motivation and background for using deep RL to automati-
cally vectorize loops is given. The framework for automatic
vectorization with deep RL is proposed in Section 3 and eval-
uated on a wide range of benchmarks in Section 4. Future
directions and related work are given in Sections 5 and 6,
respectively. The paper is concluded in Section 7.
2 Motivation and Background
2.1 Vectorization Characterization
The vectorization is critical to enhancing the performance
of compute-intensive workloads in modern computers. All
the dedicated vector machines and modern CPUs that sup-
port vector instructions rely on vectorization to enhance the
performance of such workloads.
Loops are among the most commonly vectorized parts of
codes. Loop vectorization is done by setting the VF and the
IF, which respectively determine the number of instructions
to pack together and the stride. Appropriately setting the
values of VF and IF for loops is cumbersome as it depends
on many parameters, such as the instructions in the loop
body, the stride, the underlying hardware architecture, the
computations graph, and the functionality.
To understand this challenge and motivate this work we
take a simple vector dot product kernel function:
int vec[512] __attribute__((aligned(16)));
__attribute__((noinline))
int dot_product () {
int sum = 0;
for(int i = 0; i<512; i++){
sum += vec[i]*vec[i];
}
return sum;
}
To eliminate noise and variance in results we run this kernel
one million times and average the execution time. We run
the kernel on 16 GB 2133 MHz LPDDR3 memory and 2.7 GHz
(up to 4.5 GHz) Intel 4-Core i7-8559U [ 14], which supports
AVX. Figure 1 shows the performance of this kernel after a
1https://github.com/intel/neuro-vectorizer.
243
NeuroVectorizer: End-to-End Vectorization with Deep Reinforcement Learning CGO ’20, February 22ś26, 2020, San Diego, CA, USA
Figure 1. Performance of the dot product kernel for different VFs and IFs, normalized to the baseline cost model implemented
in LLVM. The best VF and IF corresponding to the baseline cost model are ( V F = 4,IF = 2) .
Figure 2. Performance of brute-force search of LLVM’s vec-
torizer test suite, normalized to the baseline cost model im-
plemented in LLVM.
brute-force search for different VFs and IFs normalized to the
baseline cost model implemented in LLVM. The best VF and
IF corresponding to the baseline cost model are ( V F = 4,IF =
2) . While the baseline improved the performance by 2 .6×
when compared to the unvectorized code ( V F = 1,IF = 1) ,
we can still see that 26 out of 35 possible factors improve
over the baseline. This improvement is maximized by ( V F =
64,IF = 8) which achieves up to 20% better performance
than the baseline.
To further motivate this work, we evaluate the vector-
ization test suite used in the LLVM base code 2, which tests
the cost model of the baseline vectorizer in LLVM. We run
a brute-force search on all the possible VFs and IFs. The
performance of the optimal vectorization normalized to the
baseline is illustrated in Figure 2. In all the tests, the optimal
2The test suite is available on: https://github.com/llvm/llvm-test-
suite/tree/master/SingleSource/UnitTests/Vectorizer.
vectorization performed better than the baseline. This perfor-
mance gap increases with more complicated tests reaching
up to 1 .5× . These abstract results on simple tests show that
there is room for improvement for the current baseline cost
model.
2.2 State-of-the-Art Auto-Vectorzation
Most C and C++ compilers allow the users to manually de-
termine the VF and the IF in their code. This, however, is
error-prone, time-consuming and often not optimal. Thus,
many works have been proposed in the past to address the
automatic vectorization challenge. For example, Polly [ 8]
uses an abstract mathematical representation based on inte-
ger polyhedra to analyze and optimize the memory access
pattern of a program. Polly performs classical loop transfor-
mations, especially tiling and loop fusion to improve data-
locality. These transformations also simplify vectorization
decisions for the compiler. Accordingly, to date, the main
optimizations in Polly are tiling and loop fusion to improve
data locality.
Prior work [ 35] represented the code characteristics, by us-
ing hand-engineered heuristics extracted from the assembly
code, such as arithmetic intensity and used it in conjunction
with supervised learning to predict the vectorization factors.
Unfortunately, these features are typically not sufficient to
fully capture the code functionality [ 5]. To overcome this
challenge, [ 4] proposed an end-to-end solution that relies
on deep supervised learning. However, supervised learning
methods require labels to train and finding these labels can
be time-consuming. Furthermore, optimizing for multiple
objectives with large search spaces can be challenging for
supervised learning methods.
To appropriately set the VF and IF for the loops, it is nec-
essary to fully learn the characteristics of the code and then
use these characteristics to predict the optimal VF and IF.
In other words, it is necessary to extract the loops from the
244
CGO ’20, February 22ś26, 2020, San Diego, CA, USA Ameer Haj-Ali, Nesreen K. Ahmed, Ted Willke, Yakun Sophia Shao, Krste Asanovic, and Ion Stoica
Figure 3. The proposed framework for automatic vectorization with deep RL. The programs are read to extract the loops. The
loop texts are fed to the code embedding generator to generate an embedding. The embedding is fed to the RL agent. The RL
agent learns a policy that maps this embedding to optimal vectorization factors by injecting compiler pragmas and compiling
the programs with Clang/LLVM to gather the rewards: the execution time improvements.
Figure 4. An example of the automatically injected VF and IF pragmas by the RL agent.
code, characterize them, and use this characterization to pre-
dict the optimal factors. Therefore, we propose and develop
a framework that accomplishes this goal by extracting the
loops from the code, learning an embedding for these loops
and learning a mapping from this embedding to the optimal
VF and IF in an end-to-end fashion with RL. Unlike super-
vised learning methods, deep RL can be tuned to co-optimize
multiple objectives and does not require a brute-force search
and thus it can be more sample efficient.
2.3 Deep Reinforcement Learning for
Auto-Vectorization
One of the promising machine learning approaches is RL,
in which an agent learns by continually interacting with an
environment [ 15]. Using a neural network in conjunction
with RL is called deep RL. Deep models allow RL algorithms
to solve complex problems in an end-to-end fashion, han-
dle unstructured environments, learn complex functions, or
predict actions in states that have not been visited in the
past. Deep RL is gaining wide interest recently due to its
success in robotics, Atari gameplay, and superhuman capa-
bilities [ 6, 12, 16, 22, 26]. Deep RL was the key technique
behind defeating the human European champion in the game
of Go, which has long been viewed as the most challenging
of classic games for artificial intelligence [ 34].
In RL, the agent observes the state of the environment, and
based on this state/observation takes an action. The ultimate
goal is to compute a policy ( π ∗)śa mapping between the
environment states and actionsśthat maximizes expected
reward:
π ∗ = arg max
π
Eτ∼ π ( τ) [ τ] . (1)
where τ is a sequence of states and actions that define a
single episode.
If the number of steps the RL agent has to take before
the environment terminates is one, the problem is called
Contextual Bandits. In Contextual Bandits the learner tries
to find a single best action in the current state. It involves
learning to search for best actions and trial-and-error.
One of the promising deep RL methods to derive a good,
stable, and easy to use policy is proximal policy optimization
245
NeuroVectorizer: End-to-End Vectorization with Deep Reinforcement Learning CGO ’20, February 22ś26, 2020, San Diego, CA, USA
(PPO) [33]. PPO computes a gradient update at each step that
minimizes the cost function while ensuring the deviation
from the previous policy is relatively small.
There are multiple ways to predict the VF and IF from the
code embedding. It is possible to use supervised learning
methods for example. This, however, would require know-
ing the labels, i.e., optimal VF and IF for every input loop
embedding. To find these labels, it is necessary to run a brute-
force search on all the possible VFs and IFs. This might work
but can be impractical for a large number of samples. To
overcome this challenge we use RL. What distinguishes RL
from other machine learning approaches is the presence of
self-exploration and exploitation, and the tradeoff between
them [36]. In our case, RL can learn with fewer samples than
that required in the supervised learning methods and can
co-optimize for multiple objectives such as compilation time,
code size, and execution time.
3 The Proposed Framework for Automatic
Vectorization
The proposed framework for automatic vectorization with
deep RL and its components are illustrated in Figure 3. The
directory of code files is fed to the framework as text code.
This code is fed to an automatic loop extractor. The extrac-
tor finds and outputs all the loops and their contexts in all
the source codes. These outputs are fed to a code embed-
ding generator to learn and generate an embedding. The
latter is fed to the deep RL agent to predict the vectoriza-
tion factors. The agent automatically injects vectorization
pragmas as shown in Figure 4. The agent then compiles the
program with clang/LLVM to gather the execution time im-
provements, which are used as rewards to the RL agent. Once
the model is trained it can be plugged in as-is for inference
without further retraining 3. Note that our framework cannot
introduce new errors in the compiled code. Our framework
injects pragmas only. These pragmas are used as hints to
make vectorization decisions on the loops. However, some-
times the compiler can decide not to consider these pragmas
if it is not feasible. For example, predicates and memory de-
pendency can hinder reaching high VF and IF. In that case,
if the agent accidentally injected bad pragmas, the compiler
will ignore it.
It is also possible to vectorize from the command line by
giving the passes -force-vector-width=VF and -force-vector-
interleave=IF. However, we do not use this option as it re-
stricts us to use a single VF and IF pair for the entire code,
which is far from being optimal. Furthermore, the pragma is
injected for the most inner loop in the case of nested loops.
Next, we discuss the details of each component in the pro-
posed framework.
3It can still be beneficial to keep online training activated so that when
completely new loops are observed, the agent can learn how to optimize
them too.
3.1 Code Embedding
The ultimate goal of the code embedding generator is to
learn a function that maps the input loop codes to a point in
a latent multidimensional space where similar loop codes are
mapped to points close to each other in the latent multidi-
mensional space. This can allow the RL agent to make similar
vectorization decisions on similar codes using the learned
embedding. There are multiple ways to generate/learn an
embedding for the input code. One example is to use Polly’s
mathematical representation of loops as an embedding. We
see this as a potential future direction for this work. Another
example is to use a neural network model pretrained with
labels that describe the functionality, e.g., matrix multiplica-
tions, dot product, convolution, etc.
In this work we use code2vec [ 1]. Code2vec is a neural
network model that relies on natural language processing [ 3]
and attention [ 40] for representing snippets of code as con-
tinuously distributed vectors. Code2vec represents a code
snippet as a single fixed-length code vector, which can be
used to predict the semantic properties of the snippet. This
vector is composed of 340 features that embed the program
code based on the mapping the code2vec neural network
learned. This vector captures many characteristics of the
code, such as semantic similarities, combinations, and analo-
gies. The code is first decomposed to a collection of paths in
its abstract syntax tree. Then, the network simultaneously
learns the atomic representation of each path while learning
how to aggregate a set of them.
3.2 The RL Environment Definition
To learn a good policy, it is necessary to appropriately define
actions, rewards, and states. We define the agent’s reward
as follows:
reward = ( tbaseline − tRL )/ tbaseline , (2)
where tbaseline is the execution time when compiled with
the currently implemented baseline cost model in LLVM and
tRL is the execution time when compiled with the injected
pragmas by the RL agent. We normalize the execution time by
tbaseline so that our reward metric is robust to the variations
in the programs’ execution times. We also use tbaseline as
a bias in our reward so that a positive reward means the
current configuration improves over the baseline. This also
reduces the variance in the learned policy.
An action picks the VF and the IF, respectively, from the
following values:
V F ∈ [ 20,21,22,...,MAX_VF] ,
IF ∈ [ 20,21,22,...,MAX_IF] ,
(3)
where MAX_VF and MAX_IF are respectively the maximum
VF and IF supported by the underlying architecture. Note
that the actions for VF and IF can be defined to have val-
ues that are not powers of two. Here they were defined as
246
CGO ’20, February 22ś26, 2020, San Diego, CA, USA Ameer Haj-Ali, Nesreen K. Ahmed, Ted Willke, Yakun Sophia Shao, Krste Asanovic, and Ion Stoica
powers of two only because this is what LLVM currently
supports. Initially, we trained two agents, one that predicts
VF and the other predicts IF independently. However, from
our experiment combining these two agents into one agent
with a single neural network that predicts the VF and IF
simultaneously performed better. This also aligns with the
fact that IF and VF are directly correlated, and in the LLVM
compiler code they are defined as a function of each other.
The states of the RL agent were defined as the vector out-
put embedding from the code embedding generator. For the
inputs of the code embedding generator, we experimented
with different snippets of the loop bodies and observed that
for nested loops, feeding the loop body of the outermost loop,
which also includes the bodies of the inner loops, performed
better than feeding the body of the most inner loop only.
This is mainly because the entire loop nest better captures
the functionally of the code, and reveals the access patterns
and strides.
3.3 Dataset Description
Neural networks require many samples for training. We first
tried to train our model with long-running benchmarks that
include code that is not restricted to loops only. It took a long
time to train since for every pragma injected for a loop the
whole program has to be recompiled and executed. Even if
we could overcome the challenge of long execution time with
enough resources, the number of open-source benchmarks
available for training is very small [ 5].
To speed up the training, and make it more efficient, we
built a dataset that includes loops only. We built genera-
tors that generate more than 10,000 synthetic loop examples
automatically from the LLVM vectorization test-suite. For
example, some new tests are made by changing the names of
the parameters, which was crucial for reducing noise in the
code embedding generator as often the names of the param-
eters might bias the embedding. Other examples included
the stride, the number of iterations, the functionality, the in-
structions, and the number of nested loops. Below are some
of the loop examples in the dataset and the (commented)
pragma line that the RL agent will inject:
/* Example #1 */
//#pragma clang loop vectorize_width(VF) interleave_count(IF)
for (i = 0; i < N-1; i+=2) {
assign1[i] = (int) short_a[i];
assign1[i+1] = (int) short_a[i+1];
assign2[i] = (int) short_b[i];
assign2[i+1] = (int) short_b[i+1];
assign3[i] = (int) short_c[i];
assign3[i+1] = (int) short_c[i+1];
}
/* Example #2 */
for (i=0; i<M; i++) {
//#pragma clang loop vectorize_width(VF) interleave_count(IF)
for (j=0; j<N; j++) {
G[i][j] = x;
}
Figure 5. The distribution of optimal VF and IF with brute-
force search for different programs in the dataset.
}
/* Example #3 */
//#pragma clang loop vectorize_width(VF) interleave_count(IF)
for (i=0; i<N*2; i++){
int j = a[i];
b[i] = (j > MAX ? MAX : 0);
}
/* Example #4 */
for (i = 0; i < M; i++){
for (j = 0; j < L; j++){
float sum = 0;
//#pragma clang loop vectorize_width(VF) interleave_count(IF)
for (k = 0; k < N; k++) {
sum += alpha*A[i][k] * B[k][j];
}
C[i][j]
= sum;
}
}
/* Example #5 */
//#pragma clang loop vectorize_width(VF) interleave_count(IF)
for (i = 0; i < N/2-1; i++){
a[i] = b[2*i+1] * c[2*i+1] - b[2*i] * c[2*i];
d[i] = b[2*i] * c[2*i+1] + b[2*i+1] * c[2*i];
}
Figure 5 shows the distribution of optimal vectorization fac-
tors when running a brute-force search with MAX_VF = 16
and MAX_IF = 8 on the dataset. While these loops do not
represent all the existing loops, the results show that differ-
ent loops have different optimal VF and IF and to guarantee
optimal performance, all combinations of VF and IF should
be considered. Interestingly, the factors with the highest
percentage of programs are
( V F = 4,IF = 2) . From our ex-
periments, these factors were the default values the baseline
cost model also outputted.
3.4 Handling Long Compilation Time
During training, some of the programs took a long time to
compile, mainly when the agent was trying to vectorize more
than plausible. Long compilation time with limited resources
247
NeuroVectorizer: End-to-End Vectorization with Deep Reinforcement Learning CGO ’20, February 22ś26, 2020, San Diego, CA, USA
can slow down the training. To overcome this, we limited
the compilation time to ten times the time it takes to compile
a program with the baseline cost model. If the program took
longer than that to compile, we gave a penalty reward of
− 9
(equivalent to assuming it takes ten times the execution time
of the baseline) so that the agent will learn not to overesti-
mate the vectorization and avoid it. From our experiments
on the programs that took a relatively long time to compile,
eventually after waiting the necessary time for them to com-
pile, the achieved performance was not better than that of
all the other possible vectorization configurations. In some
contexts, users might care about compile-time when evaluat-
ing the performance of programs. Our reward definition can
incorporate that too so that the agent can simultaneously
optimize for more than one objective. For example, one can
allow a long compilation time but penalize for it. The reward
can also be defined as a combination of the compilation time,
execution time, generated assembly code size, etc.
4 Evaluation
The proposed framework is evaluated following the method-
ology mentioned in Section 2. For code2vec we use the open-
source code and modify it to work with our RL agent imple-
mentation. To run our RL algorithms we use RLlib [ 18] and
Tune [19], open-source libraries for RL that offer, high scala-
bility, hyper-parameter tuning and a unified API for a variety
of applications. RLlib and Tune are built on top of Ray [ 23],
a high-performance distributed execution framework tar-
geted at large-scale machine learning and RL applications.
We first train the framework with the RL agent and code2vec
until convergence. Then we also run a brute-force search
on the dataset to find the best vectorization factors and use
them as labels for NNS, the decision tree and the supervised
FCNN. Since the brute-force search requires a long time to
run, we limit our training set to 5,000 samples and use this
set for the rest of our evaluation. To report performance we
take twelve completely different benchmarks from the test
set. These benchmarks combine completely different bench-
marks from the LLVM test-suite. These benchmarks include
loops with different functionality and access patterns. For
example, predicates, memory accesses with different strides,
bitwise operations, unknown loop bounds, if statements, un-
known misalignment, multidimensional arrays, summation
reduction, type conversions, different data types, etc. We
compare the performance of our framework versus Polly
and the baseline cost model.
We start with a 64 × 64 FCNN, with training batch size of
4,000, a learning rate of 5e-5 - a hyperparameter which deter-
mines to what extent newly acquired information overrides
old information - and discrete actions. We then experiment
with changing one parameter at a time. For discrete actions,
the neural network picks two integer numbers that index
into the arrays of possible VFs and IFs. We experiment with
different hyperparameters. Figure 6 shows a hyperparameter
sweep over different hyperparameters as function of number
of training steps, i.e., compilations. We train up to 500,000
steps to see whether more training can get to better rewards,
but it is clear the policy converges with much fewer steps.
These results show that the current framework is robust
to noise and different parameters. When the learning rate
was set to 5e-5 the reward mean reached the maximum the
fastest. For learning rate 5e-3 the reward mean never reached
a higher maximum than that of the smaller learning rates
and the training loss was the highest. Minor differences were
observed for the different FCNN architectures. We also tried
single hidden layer networks and deeper networks and the
results were similar so they were not included in the figures
for clarity. The policy converged with fewer samples as the
batch size was decreased. We also experimented with smaller
batch sizes and they resulted in unstable policies that did
not outperform the performance when the batch size was
set to 500.
The results also show that the policy converged and ar-
rived at a highly rewarding (higher than 0 means better
than the baseline on average based on the reward definition
described in Section 3.2) state with 5,000 samples (for the
lowest batch size); 35 × less than that required for a brute-
force search or a supervised learning method. It is important
to point out that this training is performed once and later the
framework can be used for inference, which requires a single
step only, similar to the baseline cost model. By contrast, a
brute-force search would require searching again.
Figure 7 shows the reward mean and total training loss
as function of number of training steps for different action
space definitions. We experimented with three actions space
definitions: 1 discrete action space where the agent picks
two integer numbers that correspond to indices in the arrays
of VFs and IFs. 2 Continuous action space where the agent
picks one continuous number that encodes both the VF and
IF. 3 Continuous action space where the agent picks two
continuous numbers that encode both the VF and IF. The
numbers in the continuous action spaces are rounded to the
closest integers. The results show that the discrete action
space performs the best.
The performance on different benchmarks for the baseline,
random search, Polly, decision tree, NNS, supervised FCNN,
and RL and brute-force search are shown in Figure 8. RL out-
performed the baseline by 2 .67× on average and achieved per-
formance only 3% worse than that of the brute-force search.
The performance differed between the different benchmarks
based on how much vectorization the program can absorb.
NNS and decision trees also performed well, achieving re-
spectively 2
.65× and 2 .47× better than the baseline. This
shows that the embedding learned by the code embedding
generator during the end-to-end training is good so that
other learning methods that cannot be trained end-to-end
can use this embedding and perform well.
248
CGO ’20, February 22ś26, 2020, San Diego, CA, USA Ameer Haj-Ali, Nesreen K. Ahmed, Ted Willke, Yakun Sophia Shao, Krste Asanovic, and Ion Stoica
Reward Mean
Lr=5e-3
Lr=5e-4
Lr=5e-5
Training Loss
Reward Mean for Different Learning Rates Training Loss for Different Learning Rates
Lr=5e-3
Lr=5e-4
Lr=5e-5
Reward Mean
Training Loss
Reward Mean for Different FCNN Architectures Training Loss for Different FCNN Architectures
32x32
64x64
128x128
32x32
64x64
128x128
Number of Training Steps Number of Training Steps
Number of Training Steps Number of Training Steps
Reward Mean for Different Batch Sizes Training Loss for Different Batch Sizes
Number of Training Steps Number of Training Steps
Reward Mean
Training Loss
500
1000
2000
4000
500
1000
2000
4000
Figure 6. Reward mean and training loss for different learning rates, FCNN architectures, and batch sizes.
Random search performed much worse than the baseline.
This shows that the framework learned a structure in the
observations that manifested in the vectorization decisions it
made. Polly outperformed the baseline by 17% but performed
56% worse than the proposed RL solution. For benchmark
#10, Polly interestingly outperforms the brute-force search.
This is because Polly performs loop transformations that
optimize beyond vectorization. This shows the potential for
achieving better performance improvement when combining
Polly and deep RL. We plan to explore this in future work.
249
NeuroVectorizer: End-to-End Vectorization with Deep Reinforcement Learning CGO ’20, February 22ś26, 2020, San Diego, CA, USA
Reward Mean for Different Action Space Definitions Training Loss for Different Action Space Definitions
Reward Mean
Training Loss
discrete
continuous_1
continuous_2discrete
continuous_1
continuous_2
Number of Training Steps Number of Training Steps
Figure 7. Reward mean and training loss for different action space definitions.
Figure 8. The performance of the proposed vectorizer that can be configured to use NNS, random search, decision trees, and
RL compared to brute-force search, Polly and the baseline cost model. The performance is normalized to the baseline.
While NNs and decision trees cannot be trained end-to-
end and require special handling, the supervised FCNN can
be trained end-to-end and achieves comparable performance
to deep RL. However, RL does not require labels and thus
can be trained without a brute-force search. To demonstrate
the advantage of deep RL, Figure 9 shows the normalized
average (geomean) performance of deep RL compared to su-
pervised FCNN as a function of the number of compilations
required (samples). Deep RL with as low as 5,000 compila-
tions already achieves high performance (only 5% worse than
the peak) and 1 .26× better than the supervised FCNN. Su-
pervised FCNN achieved this only after 70,000 compilations
making it 14 × less sample efficient than deep RL. Further-
more, in the long run we believe deep RL can better handle
large search spaces with multiple objectives to co-optimize.
4.1 Transfer Learning
The goal of this subsection is to see how well the framework
generalizes to a completely new code. To that end we eval-
uate the trained model on two benchmarks: MiBench [ 10]
where the loops constitute a minor portion of the code and
PolyBench [30] where the loops constitute a major portion
of the code. MiBench is a set of free and commercially rep-
resentative embedded benchmarks such as telecommunica-
tion, networking, security, office, and automation. Note that
vectorization for some of the MiBench benchmarks is not
250
CGO ’20, February 22ś26, 2020, San Diego, CA, USA Ameer Haj-Ali, Nesreen K. Ahmed, Ted Willke, Yakun Sophia Shao, Krste Asanovic, and Ion Stoica
Figure 9. Normalized average performance of supervised
FCNN and deep RL as a function of the number of compila-
tions used (samples) for training.
Figure 10. The performance of the proposed vectorizer on
Polybench compared to Polly and the baseline cost model.
The performance is normalized to the baseline.
possible. For example, due to memory dependencies, control-
flow or lack of loops, it was not possible to vectorize adpcm,
dijkstra, basicmath, blowfish, etc. PolyBench includes bench-
marks that perform matrix operations, decomposition, and
linear algebra for which Polly is optimized to run on.
Figure 10 shows the performance of deep RL, Polly and the
baseline on PolyBench. Deep RL achieves on average 3 .42× 4
better performance than the baseline and 1 .33× better than
Polly. Polly was optimized to run on PolyBench, yet deep
RL outperformed Polly on three out of the six benchmarks.
The ability to perform loop transformations that optimize
beyond vectorization, the lack of enough benchmarks in the
dataset, and the high penalty we give to long compilation
4Note that we take the average performance improvement over multiple in-
ferences. If instead we take the best performance, the deep RL improvement
reaches 4 .77× on average: 3 .71× , 6 .74× , 6 .92× , 5 .21× , 1 .61× , and 8 .16×
for 2mm, bicg, ajax, gemm, gemver, and cholesky, respectively.
Figure 11. The performance of the proposed vectorizer on
Mibench compared to Polly and the baseline cost model. The
performance is normalized to the baseline.
times allowed Polly to perform better on some benchmarks.
When the deep RL agent tried to give high VF and IF, the
reward sometimes decreased due to the high penalty we give
to long compilation times. In such cases, the agent learns
to avoid being over-optimistic about increasing the VF and
IF. With more training data the agent can generalize better
to larger loop bounds on new examples. When combining
Polly and deep RL the average performance improvement
that can be achieved (potentially) is 4 .35× .
Figure 11 shows the performance of deep RL, Polly and the
baseline on MiBench. Deep RL outperforms both Polly and
the baseline in all the benchmarks. The average performance
improvement was 1 .1× over the baseline. While this might
not seem considerable, we believe that it can be sufficient
since the benchmarks did not rely heavily on loops, and the
measured execution time was for all the code not restricted
to loops.
4.2 Discussion: Deployability
In general, vendors and commercial companies are reluctant
to adopt machine learning and deep learning methods in
compiler optimization. The main reason behind this is the
need for methods that are deterministic, simple, easy to ex-
plain, and performant on a large scale of applications. This
also explains why most of the optimizations and implemen-
tation in compilers are based on manual engineering and
heuristics. With that being said, we believe that the grow-
ing complexity in systems and workloads, and availability
of data demands learning-based approaches. Deep RL and
other deep learning methods present a unique opportunity to
address these compiler challenges end-to-end and improve
upon manual engineering. In our evaluation, we showed that
deep RL can generalize to new benchmarks. With enough
training data, deep RL can be deterministic and performant
on a large scale of applications. Since the use of deep RL will
mainly be for inference it will also be simple to use and de-
ploy. The main challenge will remain in interpretability. This
251
NeuroVectorizer: End-to-End Vectorization with Deep Reinforcement Learning CGO ’20, February 22ś26, 2020, San Diego, CA, USA
challenge is not only a limitation of deep RL in vectorization,
it is also a limitation of neural networks in general. Many
recent works are being conducted on explaining neural net-
work decisions [ 9] and their application in code optimization
will also benefit from that. Besides, neural networks have
been adopted to solve many advanced real-world challenges
regardless of the interpretability limitation. We believe that
compilers and code optimization should also follow.
5 Future Directions
We see multiple future directions for this work. It is possible
to use loop polyhedral analysis, which is dedicated to the
loop snippets of codes for the code embedding. This will also
be less expensive in terms of computations. Combining deep
RL and Polly can further boost the performance and the RL
agent can also be trained to predict whether to use Polly or
not. The deep RL vectorizer can also be employed at the in-
termediate representation level, which can better reflect the
effects of the vectorization on the code and thus could enable
learning better predictions. For different target architectures,
it is necessary to add features that represent the underlying
architecture or to train separate models that are fitted to
the used architecture as different architectures behave differ-
ently and have different VF and IF action spaces. In this work,
we showed the potential of the deep RL vectorizer as the
first step toward end-to-end code optimization with machine
learning and deep RL. It is, however, necessary to train on a
wide range of applications, and target architectures for the
deep RL vectorizer to be a standardized optimization stage
in the LLVM compilation stack.
In our approach, we assumed the agent makes a single
decision per loop nest ( i.e., episode). However, with new
compiler features such as the support of vectorization at dif-
ferent levels of a nested loop, deep RL will be more attractive.
This is mainly because the deep RL agent is not restricted to
make a single decision per loop nest. Instead, it can perform
multiple sequential decisions that collectively form a single
episode of multiple actions and states.
Pragmas such as loop unrolling, distribution, and vector
predication can also be tuned in a similar manner. The user
only needs to define an appropriate action space and a re-
ward function that depends on the desired objective. Many
of the optimizations done today in the compiler are global
rather than local. For example, the phase ordering of com-
piler passes is applied drastically to all the functions in the
code. It can be possible to automatically determine different
phase orderings and optimizations to different sections of
the code.
Our framework can also support vanilla deep neural net-
works methods instead of deep RL. One direction we are
exploring is to use a neural network that learns a ranking
scheme on the VF and IF. For example, it can learn that given
an embedding, and pragmas, what will the execution time
normalized to the non-vectorized code be. This is equivalent
to learning a new cost model for the different VFs and IFs,
which could potentially replace the baseline cost model used
today. This method - unlike NNs and decision trees - can be
trained end-to-end.
6 Related Work
Previous work has utilized machine learning in compiler opti-
mization [
2, 39]. For example, the work in [ 7, 11, 13] proposed
deep supervised and RL methods to overcome the phase or-
dering challenge. In [ 35], multiple machine learning methods
for automatic vectorization have been proposed. Our work
is different from these prior works in that it is the first to
propose a solution based on deep RL to explore the vectoriza-
tion space and compiler optimization in general. Second, all
these works primarily rely on extracted/engineered (hand-
crafted) features from the program, e.g., arithmetic intensity,
memory operations, number of different instruction, dis-
tance between producer and consumer, etc. These features
however do not fully represent the original code. By contrast,
our work addresses the automatic vectorization by learning
useful features in an end-to-end fashion, from the text code
itself to the optimal factors without any loss of information.
In [4] end-to-end supervised deep learning is used to learn
compiler heuristics. While such approach can achieve com-
parable performance, finding the labels for training can be
time consuming, and optimizing for multiple objectives with
large search spaces can be challenging.
Automatic vectorization with other methods was also pro-
posed. For example, the currently implemented cost model
in LLVM and recently proposed cost models in [ 24, 37, 38]
rely on predefined cost functions that calculate the expected
execution time of a vectorized loop based on a linear formula
from the instruction distribution. [ 28] improves super-word
level parallelism (SLP) [ 17] to limit the automatic vectoriza-
tion. This work does not address loop vectorization and relies
on the baseline cost model to predict when some portions of
code are better off not vectorized. Also, [
21] relies on heuris-
tics to automatically vectorize. Finally, [ 27, 29] improve SLP
and rely on fixed cost models such as weighted instruction
count or the current LLVM cost models.
7 Conclusion
In this work, we proposed and developed an end-to-end vec-
torization framework that automatically detects loops, learns
their structures and applies deep RL to inject vectorization
pragmas to the compiler. Our results demonstrated an aver-
age performance improvement 1 .29 × − 4.73× compared to
the baseline cost model implemented in LLVM and on av-
erage only 3% worse than the brute-force solution. Looking
forward, we foresee a potential opportunity for automatic
end-to-end code tuning and optimization with machine learn-
ing techniques, such as deep RL.
252
CGO ’20, February 22ś26, 2020, San Diego, CA, USA Ameer Haj-Ali, Nesreen K. Ahmed, Ted Willke, Yakun Sophia Shao, Krste Asanovic, and Ion Stoica
A Artifact Description
Our artifact provides C and Python codes for all our evalu-
ated benchmarks, along with scripts to reproduce the results
and experiments automatically. For best results it is best
to evaluate on Mac OS and use 16 GB 2133 MHz LPDDR3
memory and Intel 4-Core i7-8559U (for the Intel AVX2). The
results will be different if you run it on another machine
because it uses the vectorization capability of the machine.
Since the artifact runs in a virtual machine and requires deep
RL training the results might also vary due to virtualization
and stochastic nature of training. You might get slightly bet-
ter results or worse. Our results average out multiple runs.
We included all the dependencies, i.e., ray[rllib], Tensorflow
1, TensorFlow 2, code2vec, Clang and LLVM.
A.1 Artifact Check-List (Meta-Information)
• Algorithm: Deep RL framework for auto-vectorization.
• Program: Programs from the dataset (based on llvm test
suite, Mibench and Pollybench).
• Compilation: Available in the virtual machine.
• Transformations: Deep RL agent injects vectorization prag-
mas to the code.
• Binary: The original code is included.
• Dataset: Synthetic dataset.
• Hardware: Mac OS, 16 GB 2133 MHz LPDDR3 memory and
Intel 4-Core i7-8559U.
• Metrics: Average performance improvement.
• Output: Performance numbers.
• Experiments: Training RL model using end-to-end training
or precollected data. Running inference on benchmarks.
• How much time is needed to prepare workflow (ap-
proximately)?: The workflow is implemented for the user.
• How much time is needed to complete experiments
(approximately)?: Hours to weeks.
• Publicly available?: Yes, code and data. See:
https://github.com/intel/neuro-vectorizer.
• Workflow framework used?: Python scripts.
A.2 Description
A.2.1 How Delivered
The source code is delivered as a virtual machine with all
scripts and dependencies.
A.2.2 Hardware Dependencies
For best results it is best to evaluate on Mac OS and use 16
GB 2133 MHz LPDDR3 memory and Intel 4-Core i7-8559U
(mainly for the Intel AVX2).
A.2.3 Software Dependencies
We included all the dependencies, i.e., ray[rllib], Tensorflow
1, TensorFlow 2, code2vec, Clang and LLVM.
A.2.4 Datasets
Provided in the code under training_data.
A.3 Installation
The installation scripts are provided in the virtual machine.
A.4 Experiment Workflow
• The password for the VM is cgo2020.
•
Once the virtual machine is loaded open a terminal and
run
cd ~/Desktop/rlvectorizer/llvm-project/
build/NeuroVectorizer
• Run source ./preprocess/configure.sh
• Run conda deactivate
• Run cd cgo_results
• Run python distribution.py // Figure 5
• Two options are available for running the framework:
(1) train code2vec and the RL end-to-end or (2) using a
pretrained code2vec model and training the RL model
separately. The first option gives the best results but
takes weeks to finish and the virtual machine might
run out of memory. You can experiment with this op-
tion as follows:
ś Run
cd preprocess
ś Run source ./configure.sh
ś Run source ./preprocess.sh // this will gener-
ate the bag of words of the training set for code2vec
ś Run python autovec.py // this will run training
for one configuration (hyperparameters) specified
in autovec.py. First it will run all the programs with
-O3 to get the baseline runtimes.
• The second option gives slightly worse results but
requires roughly a day. To further reduce the training
time for the second option we prepared data files that
include the execution times and embeddings of the
different samples in the dataset. Since it will roughly
take a day to run all the experiments, we made shorter
versions that you can run instead and the trend will
be the same. If you decide to run the shorter versions
then run lr_short.py, arc_short.py, batch_short.py, and
action_short.py. The shorter versions will finish in
about 5 hours. The reward means are shown in page 1
of the tensorboard page and the total loss is in page 3.
The variance is also shown as a shadow in the plots.
• Run python lr.py
ś Run tensorboard --logdir ~/ray_results/lr
ś Click on the link tensorboard outputs //figure 6
• Run python arc.py
ś Run tensorboard --logdir ~/ray_results/
arc
ś Click on the link tensorboard outputs //figure 6
• Run python batch.py
ś Run tensorboard --logdir ~/ray_results/
batch
253
NeuroVectorizer: End-to-End Vectorization with Deep Reinforcement Learning CGO ’20, February 22ś26, 2020, San Diego, CA, USA
ś Click on the link tensorboard outputs //figure 6
• Run python action.py
ś Run tensorboard --logdir ~/ray_results/
action
ś Click on the link tensorboard outputs //figure 7
• Run python fig9.py
• Run python rollout.py ~/ray_results/fig9/
PPO_*/checkpoint_(number)/checkpoint-
(number) //figures 8 and 9
ś Every checkpoint corresponds to 500 samples. So
checkpoint_10 is after 5K samples, etc. After running
rollout.py for different checkpoints we stop after the
performance starts to degrade (due to over fitting).
• Run conda activate
• Run python transfer_polly.py ~/ray_results
/fig9/PPO_*/checkpoint_(number)/checkpoint-
(number)
ś Experiment with the different checkpoints. At about
checkpoint 100 you should start to see good average
performance.
ś You can go to vectorized_pollybench directory to
see the code with the injected pragmas.
• Run python transfer_mibench.py ~/ray_results
/fig9/PPO/checkpoint_number/checkpoint-
(number) --benchmark mibench/(name of
benchmark)
ś Note that this will not compile the programs as they
have a make file, you need to follow the instructions
of each benchmark to compile. In most cases, you
just have to open the file, run make and then "time
./runeme_x". You can also open the C files to see the
injected pragmas.
A.5 Evaluation and Expected Result
The scripts will output the execution time improvement
over the baseline, which is used in the plots. Many factors
can affect the performance. First, it can be hard to run full
end to end training which delivers the best performance
but requires weeks to train on the virtual machine. Second,
expect some variance in the results; RL is stochastic and
we use a sampler that samples from the distribution of the
output probabilities and therefore the variance is almost
guaranteed. We could set the random seed but it is unfair. It
is thus best to average the results. It is also best to run on the
hardware we defined without other applications running in
the background or virtualization. Note that since the virtual
machine runs compilation in parallel with training/inference
the execution times can increase. All the steps provided were
tested on the real and virtual machine with 100% success
rate. If one of the steps fails in the virtual machine it is due to
running out of memory or exhausting the machine. Killing
the background jobs and rerunning it can solve the problem.
If that does not work then add more memory and cores in
the VirtualBox.
Acknowledgments
The authors would like to thank Ronny Ronen, Ayal Zaks,
Gadi Haber, Hideki Saito, Pankaj Chawla, Andrew Kaylor
and anonymous reviewers for their insightful feedback and
suggestions.
References
[1]
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019.
code2vec: Learning distributed representations of code. Proceedings of
the ACM on Programming Languages 3, POPL (2019), 40.
[2] Amir H Ashouri, William Killian, John Cavazos, Gianluca Palermo,
and Cristina Silvano. 2018. A survey on compiler autotuning using
machine learning. ACM Computing Surveys (CSUR) 51, 5 (2018), 96.
[3] Ronan Collobert and Jason Weston. 2008. A unified architecture for
natural language processing: Deep neural networks with multitask
learning. In Proceedings of the 25th international conference on Machine
learning. ACM, 160ś167.
[4] Chris Cummins, Pavlos Petoumenos, Zheng Wang, and Hugh Leather.
2017. End-to-end deep learning of optimization heuristics. In 2017
26th International Conference on Parallel Architectures and Compilation
Techniques (PACT). IEEE, 219ś232.
[5] Chris Cummins, Pavlos Petoumenos, Zheng Wang, and Hugh Leather.
2017. Synthesizing benchmarks for predictive modeling. In 2017
IEEE/ACM International Symposium on Code Generation and Optimiza-
tion (CGO). IEEE, 86ś99.
[6] Kenji Doya. 2000. Reinforcement learning in continuous time and
space. Neural computation 12, 1 (2000), 219ś245.
[7] Grigori Fursin, Yuriy Kashnikov, Abdul Wahid Memon, Zbigniew
Chamski, Olivier Temam, Mircea Namolaru, Elad Yom-Tov, Bilha
Mendelson, Ayal Zaks, Eric Courtois, et al
. 2011. Milepost gcc: Ma-
chine learning enabled self-tuning compiler. International journal of
parallel programming 39, 3 (2011), 296ś327.
[8] Tobias Grosser, Armin Groesslinger, and Christian Lengauer. 2012.
Polly: performing polyhedral optimizations on a low-level intermedi-
ate representation. Parallel Processing Letters 22, 04 (2012), 1250010.
[9] David Gunning. 2017. Explainable artificial intelligence (xai). Defense
Advanced Research Projects Agency (DARPA) (2017).
[10] Matthew R Guthaus, Jeffrey S Ringenberg, Dan Ernst, Todd M Austin,
Trevor Mudge, and Richard B Brown. 2001. MiBench: A free, commer-
cially representative embedded benchmark suite. In Proceedings of the
Fourth Annual IEEE International Workshop on Workload Characteriza-
tion. WWC-4 (Cat. No. 01EX538) . IEEE, 3ś14.
[11] Ameer Haj-Ali, Qijing Huang, William Moses, John Xiang, John
Wawrzynek, Krste Asanovic, and Ion Stoica. 2020. AutoPhase: Juggling
HLS Phase Orderings in Random Forests with Deep Reinforcement
Learning. In Third Conference on Machine Learning and Systems (ML-
Sys).
[12] Ameer Haj-Ali, Nesreen K. Ahmed, Ted Willke, Joseph Gonzalez, Krste
Asanovic, and Ion Stoica. 2019. A View on Deep Reinforcement Learn-
ing in System Optimization. arXiv preprint arXiv:1908.01275 (2019).
[13] Qijing Huang, Ameer Haj-Ali, William Moses, John Xiang, Ion Stoica,
Krste Asanovic, and John Wawrzynek. 2019. AutoPhase: Compiler
Phase-Ordering for HLS with Deep Reinforcement Learning. In 2019
IEEE 27th Annual International Symposium on Field-Programmable
Custom Computing Machines (FCCM) . IEEE, 308ś308.
[14] Intel Inc. 2018. Intel Core i7-8559U Processor Specifica-
tion. https://ark.intel.com/content/www/us/en/ark/products/137979/
254
CGO ’20, February 22ś26, 2020, San Diego, CA, USA Ameer Haj-Ali, Nesreen K. Ahmed, Ted Willke, Yakun Sophia Shao, Krste Asanovic, and Ion Stoica
intel-core-i7-8559u-processor-8m-\cache-up-to-4-50-ghz.html
[15] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996.
Reinforcement learning: A survey. Journal of artificial intelligence
research 4, 237ś285.
[16] Jens Kober, J Andrew Bagnell, and Jan Peters. 2013. Reinforcement
learning in robotics: A survey. The International Journal of Robotics
Research 32, 11 (2013), 1238ś1274.
[17] Samuel Larsen and Saman Amarasinghe. 2000. Exploiting superword
level parallelism with multimedia instruction sets . Vol. 35. ACM.
[18] Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox,
Joseph Gonzalez, Ken Goldberg, and Ion Stoica. 2017. Ray rllib: A
composable and scalable reinforcement learning library. arXiv preprint
arXiv:1712.09381 (2017).
[19] Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E
Gonzalez, and Ion Stoica. 2018. Tune: A Research Platform for Dis-
tributed Model Selection and Training. arXiv preprint arXiv:1807.05118
(2018).
[20] Chris Lomont. 2011. Introduction to intel advanced vector extensions.
Intel White Paper (2011), 1ś21.
[21] Daniel S McFarlin, Volodymyr Arbatov, Franz Franchetti, and Markus
Püschel. 2011. Automatic SIMD vectorization of fast fourier trans-
forms for the larrabee and AVX instruction sets. In Proceedings of the
international conference on Supercomputing . ACM, 265ś274.
[22] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioan-
nis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing
atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602
(2013).
[23] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov,
Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul,
Michael I Jordan, et al . 2018. Ray: A Distributed Framework for Emerg-
ing AI Applications. In 13th { USENIX} Symposium on Operating Sys-
tems Design and Implementation ( { OSDI} 18). 561ś577.
[24] Dorit Nuzman, Sergei Dyshel, Erven Rohou, Ira Rosen, Kevin Williams,
David Yuste, Albert Cohen, and Ayal Zaks. 2011. Vapor SIMD: Auto-
vectorize once, run everywhere. In Proceedings of the 9th Annual
IEEE/ACM International Symposium on Code Generation and Optimiza-
tion. IEEE Computer Society, 151ś160.
[25] Dorit Nuzman, Ira Rosen, and Ayal Zaks. 2006. Auto-vectorization
of interleaved data for SIMD. ACM SIGPLAN Notices 41, 6 (2006),
132ś143.
[26] Jan Peters, Sethu Vijayakumar, and Stefan Schaal. 2003. Reinforcement
learning for humanoid robotics. In Proceedings of the third IEEE-RAS
international conference on humanoid robots . 1ś20.
[27] Vasileios Porpodas. 2017. SuperGraph-SLP Auto-Vectorization. In 2017
26th International Conference on Parallel Architectures and Compilation
Techniques (PACT). IEEE, 330ś342.
[28] Vasileios Porpodas and Timothy M Jones. 2015. Throttling automatic
vectorization: When less is more. In 2015 International Conference on
Parallel Architecture and Compilation (PACT) . IEEE, 432ś444.
[29] Vasileios Porpodas, Alberto Magni, and Timothy M Jones. 2015. PSLP:
Padded SLP automatic vectorization. In Proceedings of the 13th Annual
IEEE/ACM International Symposium on Code Generation and Optimiza-
tion. IEEE Computer Society, 190ś201.
[30] Louis-Noel Pouchet. 2012. Polybench: The polyhedral benchmark suite.
URL: http://www. cs. ucla. edu/pouchet/software/polybench (2012).
[31] J. Ross Quinlan. 1986. Induction of decision trees. Machine learning 1,
1 (1986), 81ś106.
[32] Nick Roussopoulos, Stephen Kelley, and Frédéric Vincent. 1995. Near-
est neighbor queries. In ACM sigmod record , Vol. 24. ACM, 71ś79.
[33] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and
Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv
preprint arXiv:1707.06347 (2017).
[34] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre,
George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou,
Veda Panneershelvam, Marc Lanctot, et al . 2016. Mastering the game
of Go with deep neural networks and tree search. nature 529, 7587
(2016), 484.
[35] Kevin Stock, Louis-Noël Pouchet, and P Sadayappan. 2012. Using ma-
chine learning to improve automatic vectorization. ACM Transactions
on Architecture and Code Optimization (TACO) 8, 4 (2012), 50.
[36] Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning:
An introduction. MIT press.
[37] Xinmin Tian, Hideki Saito, Ernesto Su, Abhinav Gaba, Matt Masten,
Eric Garcia, and Ayal Zaks. 2016. LLVM framework and IR extensions
for parallelization, SIMD vectorization and offloading. In 2016 Third
Workshop on the LLVM Compiler Infrastructure in HPC (LLVM-HPC) .
IEEE, 21ś31.
[38] Konrad Trifunovic, Dorit Nuzman, Albert Cohen, Ayal Zaks, and Ira
Rosen. 2009. Polyhedral-model guided loop-nest auto-vectorization.
In 2009 18th International Conference on Parallel Architectures and
Compilation Techniques. IEEE, 327ś337.
[39] Zheng Wang and Michael O’Boyle. 2018. Machine learning in compiler
optimization. Proc. IEEE 106, 11 (2018), 1879ś1901.
[40] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville,
Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show,
attend and tell: Neural image caption generation with visual attention.
In International conference on machine learning . 2048ś2057.
255

233
Flow2Vec: Value-Flow-Based Precise Code Embedding
YULEI SUI, University of Technology Sydney, Australia
XIAO CHENG, Beijing University of Posts and Telecommunications, China
GUANQIN ZHANG, University of Technology Sydney, Australia
HAOYU WANG, Beijing University of Posts and Telecommunications, China
Code embedding, as an emerging paradigm for source code analysis, has attracted much attention over the past
few years. It aims to represent code semantics through distributed vector representations, which can be used to
support a variety of program analysis tasks (e.g., code summarization and semantic labeling). However, existing
code embedding approaches are intraprocedural, alias-unaware and ignoring the asymmetric transitivity
of directed graphs abstracted from source code, thus they are still ineffective in preserving the structural
information of code.
This paper presents Flow2Vec, a new code embedding approach that precisely preserves interprocedural
program dependence (a.k.a value-flows). By approximating the high-order proximity, i.e., the asymmetric
transitivity of value-flows, Flow2Vec embeds control-flows and alias-aware data-flows of a program in a
low-dimensional vector space. Our value-flow embedding is formulated as matrix multiplication to preserve
context-sensitive transitivity through CFL reachability by filtering out infeasible value-flow paths.
We have evaluated Flow2Vec using 32 popular open-source projects. Results from our experiments show
that Flow2Vec successfully boosts the performance of two recent code embedding approaches code2vec and
code2seq for two client applications, i.e., code classification and code summarization. For code classification,
Flow2Vec improves code2vec with an average increase of 21.2%, 20.1% and 20.7% in precision, recall and F1,
respectively. For code summarization, Flow2Vec outperforms code2seq by an average of 13.2%, 18.8% and
16.0% in precision, recall and F1, respectively.
CCS Concepts: · Software and its engineering→ Automated static analysis; · Computing method-
ologies → Machine learning.
Additional Key Words and Phrases: Flow2Vec, code embedding, value-flows, asymmetric transitivity
ACM Reference Format:
Yulei Sui, Xiao Cheng, Guanqin Zhang, and Haoyu Wang. 2020. Flow2Vec: Value-Flow-Based Precise Code
Embedding. Proc. ACM Program. Lang. 4, OOPSLA, Article 233 (November 2020), 27 pages. https://doi.org/10.
1145/3428301
1 INTRODUCTION
Static program analysis (or source code analysis) aims to reason about the runtime behavior of
a program without actually running it. It is the cornerstone of many clients, such as program
optimization [Bodík and Anik 1998 ; Ferrante et al . 1987], program slicing [ Gallagher and Lyle 1991 ;
Weiser 1981], change impact analysis [ Acharya and Robinson 2011 ; Canfora and Cerulo 2005 ],
Authors’ addresses: Yulei Sui, yulei.sui@uts.edu.au, University of Technology Sydney, 15 Broadway, Sydney, NSW, Australia,
PO Box 123; Xiao Cheng, jackiecheng@bupt.edu.cn, Beijing University of Posts and Telecommunications, No 10, Xitucheng
Road, Haidian District, Beijing, China, 100876; Guanqin Zhang, 13426770@student.uts.edu.au, University of Technology
Sydney, 15 Broadway, Sydney, NSW, Australia, PO Box 123; Haoyu Wang, haoyuwang@bupt.edu.cn, Beijing University of
Posts and Telecommunications, No 10, Xitucheng Road, Haidian District, Beijing, China, 100876.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee
provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and
the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses,
contact the owner/author(s).
© 2020 Copyright held by the owner/author(s).
2475-1421/2020/11-ART233
https://doi.org/10.1145/3428301
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
233:2 Yulei Sui, Xiao Cheng, Guanqin Zhang, and Haoyu Wang
bug detection [ Shi et al . 2018, 2010], code classification [ Alon et al . 2019b; Frantzeskou et al . 2008;
Ugurel et al . 2002] and code summarization [ Alon et al . 2019a; Kamiya et al . 2002; Sajnani et al .
2016]. A key challenge in static analysis is to develop an effective code representation that can
precisely capture code semantics to support a wide range of client applications.
Existing Efforts and Limitations . The recent success of embedding techniques in natural
language processing has opened up new opportunities to develop effective code representations.
Code embedding , as an emerging paradigm to support code analysis, produces low-dimensional
vector representations of code (so called distributed code representation), which enables the ap-
plication of existing machine learning techniques to various code analysis tasks. Many previous
efforts in code embedding have treated source code as textual tokens by using Word2Vec-like
techniques [
Allamanis et al . 2016; Hindle et al . 2012; Pradel and Sen 2018 ; Raychev et al . 2014].
Later, abstract syntax trees have been used to extract the structural information of code in addition
to shallow textual features [
Alon et al . 2019b; Chen et al . 2018; Hu et al . 2018; Maddison and Tarlow
2014; Zhang et al . 2019]. Very recently, there have been a few attempts at investigating richer code
semantics in the form of graphs, e.g., the data-flow graphs of a program [ Allamanis et al . 2018;
Ben-Nun et al. 2018 ; Li et al. 2018 ; Zhao and Huang 2018 ; Zhou et al. 2019 ].
However, existing approaches suffer from the following three limitations: (1) Intraprocedural
embedding. Almost all existing embedding techniques are intraprocedural. These approaches
extract the abstract syntax tree [ Chen et al . 2018] or the data-flow information [ Ben-Nun et al .
2018; Zhou et al . 2019] within each program method, but data-flows across methods are not
preserved. The generated embedding vectors of individual methods are isolated with no calling
context information. Thus, the resulting data dependence in the latent embedding space is context-
insensitive. (2) Alias-unaware. One of the key research in data dependence analysis is memory
aliasing. The state-of-the-art approaches are all alias-unaware. They either extract the data-flow
information from ASTs [ Allamanis et al . 2016; Hu et al . 2018; Zhang et al . 2019] or directly from
an intermediate representation, e.g., LLVM-IR [ Ben-Nun et al. 2018 ], without considering pointer
alias information, which results in partial data dependence being embedded in the latent space. (3)
Ignoring asymmetric transitivity . Since the data- or control-flow graphs of a program are directed,
the current approaches which apply an existing network embedding technique (e.g., Gated Graph
Neural Networks [ Allamanis et al. 2018 , 2016] or Skip-Gram models [ Ben-Nun et al. 2018 ]) fail to
preserve the long-range asymmetric transitivity of context-sensitive program dependence.
Insights and Challenges . To address these limitations, a precise code embedding approach
needs to operate on a compact and comprehensive code representation (e.g., interprocedural value-
flow graphs [ Choi et al . 1991; Hardekopf and Lin 2011 ; Sui and Xue 2016 ]), which contains both
control-flows and alias-aware data-flows of a program. The approach is also expected to precisely
preserve deep code semantics extracted from the target code representation (e.g., value-flow graphs)
in a low-dimensional vector space.
Traditional graph embedding approaches [ Grover and Leskovec 2016 ; Perozzi et al . 2014; Tang
et al. 2015] leverage random walks on a graph to identify its structure by exploring the bidirectional
connectivity between two nodes through first-order proximity (by looking at directly connected
nodes [ Belkin and Niyogi 2002 ]) or second-order proximity (by looking at nodes with shared
neighbors [ Tang et al . 2015]). As such, two nodes with similar proximity are forced to be close
to one another in the latent embedding space. Recently, graph embedding approaches based on
high-order proximity [
Lian et al . 2018; Ou et al . 2016; Zhang et al . 2018] have been proposed to
capture the high-level structural information of a graph. The technique is particularly useful for
precisely preserving long-range program dependence information, which often manifests as many
multi-hop def-use (value-flows) edges between program statements within and across methods.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
Flow2Vec: Value-Flow-Based Precise Code Embedding 233:3
(a) Pre-embedding (b) Value-flow reachability
as matrix multiplication
(c) High-order proximity embedding
 Taint AnalǇsis
 Program Slicing
 ͙
 Code Clone Detection
 Code Classification
 ͙
;bͿ CŽnƚeǆƚͲƐenƐiƚiǀe maƚƌiǆ 
geneƌaƚiŽn
;dͿ AƉƉlicaƚiŽn
⋮
 ⋮
A
A1
An
M1
Mn
CͬCнн
Source 
Code
LLVM IR
CLANG
SVF
;aͿ PƌeͲembedding
𝑴૚ ൎ 𝑫૚࢙⋅𝑫 ૚࢚
࢔࢙࢔࢚
⋮
IVFG
Call Graph
;cͿ HighͲŽƌdeƌ ƉƌŽǆimiƚǇ embedding
𝑫૚࢙
𝑫૚࢚
࢔࢙
࢔࢚
.
..
 Taint AnalǇsis
 Program Slicing
 ͙
 Code Clone Detection
 Code Classification
 ͙
;bͿ CŽnƚeǆƚͲƐenƐiƚiǀe maƚƌiǆ 
geneƌaƚiŽn
;dͿ AƉƉlicaƚiŽn
⋮
 ⋮
A
A1
An
M1
Mn
CͬCнн
Source 
Code
LLVM IR
CLANG
SVF
;aͿ PƌeͲembedding
𝑴૚ ൎ 𝑫૚࢙⋅𝑫 ૚࢚
࢔࢙࢔࢚
⋮
IVFG
Call Graph
;cͿ HighͲŽƌdeƌ ƉƌŽǆimiƚǇ embedding
𝑫૚࢙
𝑫૚࢚
࢔࢙
࢔࢚
C/C++
Source
Code
 Taint AnalǇsis
 Program Slicing
 ͙
 Code Clone Detection
 Code Classification
 ͙
;bͿ CŽnƚeǆƚͲƐenƐiƚiǀe maƚƌiǆ 
geneƌaƚiŽn
;dͿ AƉƉlicaƚiŽn
⋮
 ⋮
A
A1
An
M1
Mn
CͬCнн
Source 
CodeLLVM IR
CLANG
SVF
;aͿ PƌeͲembedding
𝑴૚ ൎ 𝑫૚࢙⋅𝑫 ૚࢚
࢔࢙࢔࢚
⋮
IVFG
Call Graph
;cͿ HighͲŽƌdeƌ ƉƌŽǆimiƚǇ embedding
𝑫૚࢙
𝑫૚࢚
࢔࢙
࢔࢚
 Taint AnalǇsis
 Program Slicing
 ͙
 Code Clone Detection
 Code Classification
 ͙
;bͿ CŽnƚeǆƚͲƐenƐiƚiǀe maƚƌiǆ 
geneƌaƚiŽn
;dͿ AƉƉlicaƚiŽn
⋮
 ⋮
A
A1
An
M1
Mn
CͬCнн
Source 
CodeLLVM IR
CLANG
SVF
;aͿ PƌeͲembedding
𝑴૚ ൎ 𝑫૚࢙⋅𝑫 ૚࢚
࢔࢙࢔࢚
⋮
IVFG
Call Graph
;cͿ HighͲŽƌdeƌ ƉƌŽǆimiƚǇ embedding
𝑫૚࢙
𝑫૚࢚
࢔࢙
࢔࢚
Code summarisation
Code classification
(d) Applications
dSt
z
x
IVFG
LLVM 
IR
CLANG
Fig. 1. Overview of our approach.
Inspired by these approaches, this paper aims to investigate, for the first time, high-order prox-
imity based code embedding by preserving asymmetric transitivity of interprocedural alias-aware
value-flows. Another challenge for interprocedural embedding is to identify and embed "realizable
paths" [Reps 1998] on the value-flow graph in low-dimensional space by distinguishing different call-
ing contexts leading to a procedure, thus producing a precise context-sensitive code representation
to better support subsequent client applications (e.g., code classification and summarization).
Our Solution. We present Flow2Vec, a new code embedding approach that preserves asymmet-
ric transitivity of interprocedural value-flows of a program. To approximate a high-order proximity,
our value-flow embedding is formulated as matrix multiplication by preserving context-sensitive
transitivity through CFL reachability on the interprocedural value-flow graph (IVFG). A higher
proximity from node 𝑖 to node 𝑗 implies the more feasible and the shorter value-flow paths from 𝑖
to 𝑗 on the IVFG. The high-order matrix is then decomposed to generate two types of embedding
vectors, source and target vectors, for each node to represent its incoming and outgoing transitivity
on the IVFG. The reachability information from node 𝑖 to node 𝑗 is then translated into computing
the inner product between 𝑖’s source vector and 𝑗’s target vector. In the resulting embedding space,
two nodes that can reach one another along value-flow paths on the IVFG are mapped to close
numerical vectors, thus preserving high-level structural information of code.
Unlike many existing code embedding approaches [ Allamanis et al . 2018; Alon et al . 2019a,b; Ben-
Nun et al . 2018; Zhou et al . 2019] that rely on supervised learning and require ground truth data to
train the model, our value-flow embedding does not depend on prior knowledge. Benefiting from fast
matrix factorization techniques, the new code embedding supports tunable precision and efficiently
solves reachability by approximating high-order proximity through Katz Index [
Hochstenbach 2009;
Ou et al . 2016], with theoretical guarantees on the Root Mean Squared Error (RMSE). Furthermore,
our value-flow embedding, which preserves alias-aware value-flow reachability, serves as a new
code representation. It can also be used as inputs to support subsequent client applications, such as
code classification and summarization.
Framework Overview. Figure 1 provides an overview of Flow2Vec with its four phases:
(a) Pre-embedding. A program is first compiled into LLVM-IR. The interprocedural value-flow
graph (IVFG) is built on top of the IR using Andersen’s pointer analysis [ Andersen 1994]. The IVFG
is then transformed into an adjacency matrix A with call/return value-flow edges represented by
symbolic elements (i.e., opening/closing parentheses with their corresponding callsite information).
(b) Value-flow reachability via matrix multiplication. Value-flow reachability is formulated as
a chain-matrix-multiplication problem. The number of value-flow paths between any two nodes
with a path length of 𝑛is obtained by the 𝑛-th power of matrix A. To achieve context-sensitive
results, the symbolic elements in the resulting matrix are resolved by matching calls and returns so
as to filter out unrealizable inter-procedural paths as a balanced-parentheses problem based on the
CFL-reachability [Kodumal and Aiken 2004 ; Reps 1998 ].
(c) High-order proximity embedding. Given the matrices representing value-flow reachability
for different path lengths, this phase approximates a high-order proximity matrix M via Katz
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
233:4 Yulei Sui, Xiao Cheng, Guanqin Zhang, and Haoyu Wang
Index [Katz 1953] and decomposing M into source and target embedding vectors for each node on
the IVFG. The reachability between node 𝑖 and node 𝑗 is measured as the dot products between 𝑖’s
source and 𝑗’s target vector to preserve asymmetric transitivity of value-flows.
(d) Application scenarios. We demonstrate the effectiveness of Flow2Vec in boosting the perfor-
mance of code2vec and code2seq, two state-of-the-art learning-based code embedding approaches
for two important clients, i.e., code classification and summarization in terms of increased precision,
recall and F1.
Our major contributions are as follows:
• We introduce Flow2Vec, a new approach to code embedding by preserving interprocedural
alias-aware value-flows.
• We formulate value-flow reachability as matrix multiplication to embed context-sensitive
value-flows through CFL reachability in the low-dimensional space.
• We present a comprehensive evaluation of Flow2Vec’s precision, recall and F1-scores for
both code classification and code summarization using 32 popular open-source C/C++ projects
consisting of over 5 million lines of LLVM instructions. The code classification experiments
show that Flow2Vec outperforms code2vec with an average increase of 21.2% in precision,
20.1% in recall and 20.7% in F1. The results for the code summarization show that Flow2Vec
improves upon code2seq’s performance by an average of 13.2% in precision, 18.8% in recall
and 16.0% in F1.
2 BACKGROUND
This section sets out the preliminary knowledge of our work, including LLVM-IR, IVFG and graph
embedding.
2.1 LLVM-IR
Modern compilers (e.g., LLVM) normally support multiple front-ends that compile programs written
in different programming languages into a uniform Intermediate Representation (IR), e.g., LLVM-IR,
for subsequent analysis and transformation tasks. Without loss of generality, our approach works
on top of LLVM-IR, a popular and robust code representation used by many existing program
analyses [
Balatsouras and Smaragdakis 2016 ; Ben-Nun et al . 2018; Lei and Sui 2019 ; Lhoták and
Chung 2011; Li et al . 2011]. In the LLVM-IR of a program, the set of all variables V is separated
into two subsets, O which contains all possible abstract objects, i.e., the address-taken variables
of a pointer, and P which contains all top-level variables , including stack virtual registers (the
symbols starting with ł%") and global variables (the symbols starting with ł@") which are explicit,
i.e., directly accessed. Address-taken variables in O are implicit, i.e., accessed indirectly at LLVM’s
load or store instructions via top-level variables.
After the SSA conversion, the LLVM-IR of a program is represented by five types of instructions:
𝑝=&𝑜 (AddrOf), 𝑝=𝑞(Copy), 𝑝=&𝑞→ 𝑓𝑖 (Field), 𝑝=∗𝑞(Load) and ∗𝑝=𝑞(Store), where 𝑝,𝑞 ∈ P
and 𝑜 ∈ O are address-taken variables. 𝑜 is either a stack, global or a dynamically created heap
object for an AddrOf 𝑝=&𝑜, known as an allocation site .
Figures 2(a) and (b) show a code fragment (with variable type information ignored for brevity) and
its corresponding LLVM-IR, where p,q,r,t1 ,x ∈ P and a,b ∈ O . Note that a is indirectly accessed
at a store ∗p = t1 by introducing a top-level pointer 𝑡1 in the LLVM’s partial SSA form [ Lattner
and Adve 2004 ].
2.2 Interprocedural Value-Flow Graph (IVFG)
The IVFG [ Hardekopf and Lin 2011 ; Shi et al . 2018; Sui and Xue 2016 ] of a program is built
upon LLVM-IR by considering program control-flow and alias-aware data-flows. IVFG = (𝑁,𝐸 )
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
Flow2Vec: Value-Flow-Based Precise Code Embedding 233:5
(b) LLVM-IR (c) IVFG
foo(){
p = &a;
a = &b;
q = *p;
r=bar(q);
}
bar(x)
{
  return x;
} 
foo(){
p = &a;      
//p points to &a
t = &b;      //t points to &b
*p = t;      //a =   a
q = *p;      //  a
r = bar(q);  //FV
} 
bar(x)
{
    
return x;
}
1
(a) Source code 
Return value-flow
Intra value-flow
t1
Call value-flow
a
q
1
1
x
x
01
Fig. 2. A C code fragment and its LLVM instructions and interprocedural value-flow graph (IVFG).
is a multi-edged directed graph that captures its def-use chains flow-sensitively. 𝑁 is the set of
nodes representing all instructions and 𝐸is the set of edges representing all def-use chains. More
specifically, an edge ℓ
𝑣
− →ℓ′, where 𝑣 ∈ V , from the instruction ℓ to ℓ′ signifies a def-use relation
for 𝑣 with its def at ℓ and use at ℓ′.
As top-level variables are in LLVM’s SSA form, their uses have unique definitions (with 𝜙
functions inserted at confluence points of a program’s control-flow graph). A def-use chain ℓ
𝑡
− →ℓ′,
where 𝑡 ∈ P , represents a direct value-flow of 𝑡. Such def-use chains can be found easily without
the need for pointer analysis.
However, address-taken variables are not in LLVM’s partial SSA form, so their indirect uses at
loads may be defined indirectly at multiple stores. Their def-use chains are obtained by building the
interprocedural memory SSA form following [ Chow et al . 1996; Hardekopf and Lin 2011 ]. Figure 2
provides an example. First, the points-to information in the program is computed by Andersen’s
pointer analysis [ Andersen 1994]. The points-to targets of the pointers 𝑝 and 𝑡1 are shown beside
the first two instructions respectively. Second, a store, e.g., ∗𝑝 = 𝑡1 is annotated with a function
𝑎= 𝜒(𝑎) for each variable 𝑎 ∈ O that may be pointed to by 𝑝to represent a potential def and use of
𝑎at the store. If 𝑎can be strongly updated [ Lhoták and Chung 2011 ], then 𝑎receives the value on
the right hand side of the store, i.e., " 𝑡1" and the old contents in 𝑎are killed. Otherwise, 𝑎must also
incorporate its old contents, resulting in a weak update to 𝑎. Likewise, a load 𝑞= ∗𝑝 is annotated
with a function 𝜇(𝑎) for each variable 𝑎 ∈ O that may be pointed to by 𝑝 to represent a potential
use of 𝑎at the load. Third, we convert all the address-taken variables into SSA form, with each 𝜇(𝑎)
treated as a use of 𝑎and each 𝑎= 𝜒(𝑎) as both a def and use of 𝑎. Finally, we obtain the indirect
def-use chains for 𝑎 ∈ O as follows. For a use of 𝑎identified as 𝑎𝑛 (with its version identified by 𝑛) at
a load or store ℓ, its unique definition of 𝑎is 𝑎𝑛 at a store ℓ′. Then, an indirect def-use chain ℓ′ 𝑎
− →ℓ
is added to represent the potentially indirect value-flow of 𝑎from ℓ′ to ℓ(e.g., ℓ2
𝑎
− →ℓ3 in Figure 2(c)).
The opening/closing parentheses with the callsite information are put on the call/return edges to
differentiate the intra-procedural value-flow edges from the inter-procedural ones. ℓ′ 𝑣
− →
(𝑖
ℓ denotes
the call value-flow of 𝑣 from ℓ′ in a caller to ℓ in its callee via callsite 𝑖 (e.g., ℓ3
𝑞
− →
(1
ℓ5 in Figure 2(c)).
Similarly, ℓ6
𝑟
− →
)1
ℓ4 denotes the return edge from ℓ6 to ℓ4 via callsite 𝑐𝑠1. Figure 2(c) shows the final
IVFG of the LLVM-IR. The intra-procedural edges marked as the black arrows and call/return edges
are the red/blue arrows.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
233:6 Yulei Sui, Xiao Cheng, Guanqin Zhang, and Haoyu Wang
2.3 Graph Embedding
Graph embedding [ Cai et al . 2018; Cui et al . 2018] or network embedding is to transform a graph
into a distributed representation in the form of a numerical vector or a set of vectors by capturing
the graph’s topology and vertex-to-vertex relations. The resulting embedding provides a compact
data format, from which to efficiently conduct a wide variety of tasks, such as pattern recognition,
clustering and classification, can be conducted efficiently in both time and space. For example,
computing the (context-sensitive) reachability between two nodes on an IVFG, which previously
requires costly enumeration of all possible paths between any two nodes, can be approximated by
fast high-order proximity [ Cui et al . 2018]. As another example, machine learning methods, which
are powerful for solving some software engineering tasks, such as code summarization, require
a precise structure-preserving code representation. In turn, this requires an effective embedding
approach.
Preserving high-level graph structures, such as the reachability relations between two arbitrary
nodes, is a fundamental requirement of graph embedding. Preserving asymmetric transitivity has
recently become an important topic for precisely embedding directed graphs [ Ou et al . 2016; Sun
et al. 2019]. Given a directed graph 𝐺 = (𝑁,𝐸 ), where 𝑁 = {𝑣1,𝑣2,...,𝑣 𝑁} is a set of nodes and 𝐸
is a set of directed edges. An edge from 𝑣𝑖 to 𝑣𝑗 is represented as 𝑒𝑖𝑗 = (𝑣𝑖,𝑣𝑗) ∈ 𝐸. The embedding
takes 𝐺’s adjacency matrix (denoted as A) as its input and outputs a high-order proximity matrix
M, where M𝑖,𝑗 preserves the asymmetric transitivity between 𝑣𝑖 and 𝑣𝑗. Note that M𝑖,𝑗 denotes the
element at the 𝑖-th row and 𝑗-th column of M, and the bold lowercase symbol s𝑖 represents the
𝑖-th row of M). There are several commonly used models (e.g., random walk [ Mikolov et al . 2013;
Perozzi et al . 2014] and matrix factorization [ De Lathauwer et al . 2000; Ou et al . 2016]) to transform
a graph from its original graph space A to its high-order proximity space M.
Given M, the embedding vectors are extracted as D = [D𝑠,D𝑡], where the 𝑖-th row, d𝑖, denotes
the embedding vector of 𝑣𝑖 and 𝐾 is the embedding dimension. Unlike the embedding which
maintains only one type of vectors for an undirected graph, asymmetric transitivity preservation
on a directed graph produces two types of vectors: a source vector D𝑠 and a target vector D𝑡, in
the embedding space R |𝑁|×𝐾. The inner product between d𝑠
𝑖
and d𝑡
𝑗, i.e., d𝑠
𝑖 · d𝑡
𝑗
⊺
represents the
approximated proximity from 𝑣𝑖 to 𝑣𝑗. This proximity value is often normalized between 0 and 1. A
higher proximity means more and shorter paths from 𝑣𝑖 to 𝑣𝑗. A proximity value that fails under
the user-defined threshold indicates that 𝑣𝑖 are unlikely to reach 𝑣𝑗.
The fundamental difference between code and natural languages is the structural information
available (e.g., IVFG). Exploiting rich code semantics via structure-preserving graph embedding be-
comes increasingly important. This paper fills the gap between the demand for comprehensive code
representation and the lack of code embedding techniques that precisely preserve interprocedural
context-sensitive program dependence.
3 A MOTIVATING EXAMPLE
Figure 3 illustrates the key idea of Flow2Vec. The objective is to show that our interprocedural code
embedding can precisely preserve the context-sensitive value-flow reachability in the embedding
space.
Pre-embedding.
The code example in Figure 3(a) is extracted from a real-world program bison.
The IVFG of the code snippet is constructed by following Section 2.2. Each node on the IVFG
represents a definition of a variable at an instruction with its corresponding line number. We
then transform the IVFG into a symbolic matrix
A, with each element denoting the first-order
reachability (two directly connected nodes) as depicted in Figure 3(a). If there is an intraprocedural
value-flow between ℓ𝑖 and ℓ𝑗 for any two instructions ℓ𝑖 and ℓ𝑗, A𝑖,𝑗 = 1 and 0 otherwise. The
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
Flow2Vec: Value-Flow-Based Precise Code Embedding 233:7
foo(){
    stack = malloc(..);
    queue = malloc(..);
    p = bar(stack);
//cs1
    q = bar(queue);//cs2
    //operations on ‘stack’
     via pointer ņp’
    . . .
}
bar(x){
//initialization for 
objects that ‘x’ points to
    return x;
}
IVFG
a) Pre-embedding (b) Value-flow reachability as matrix multipication
(c) High-order proximity embedding
Fig. 3. A motivating example. (a) shows the code and its corresponding IVFG; (b) gives A1, A2 and A3 which
represent context-sensitive reachability between any two IVFG nodes with a path length of 1, 2 or 3; and
(c) shows a high-order proximity matrix M, the source and target embedding vectors D𝑠 and D𝑡. The inner
product between d𝑠
𝑖 · d𝑡
𝑗
𝑇 is the proximity from ℓ𝑖 to ℓ𝑗.
interprocedural value-flows between ℓ𝑖 and ℓ𝑗 are symbolized using their callsite information, i.e.,
A𝑖,𝑗 = (𝑐𝑠𝐼𝑑
and A𝑖,𝑗 =)𝑐𝑠𝐼𝑑 for the call and return edges of the callsite 𝑐𝑠𝐼𝑑, respectively. For the
purposes of context-sensitive analysis, this later can be instantiated to either 1 or 0 for a feasible
flow between 𝑖and 𝑗 under different call paths. In Figure 3(a), A5,6 = 1 signifies the intraprocedural
reachability from ℓ5 to ℓ6, and A1,5 = (1 represents the interprocedural value-flow from ℓ1 to ℓ5 via
callsite 𝑐𝑠1.
Value-Flow Reachability as Matrix Multiplication . The asymmetric transitivity of value-
flows is formulated as a chain-matrix-multiplication problem as shown in Figure 3(b). The element at
the 𝑖-th row and 𝑗-th column of Aℎ, which is the ℎ-th power1 of A, denotes the number of value-flow
paths with a length of ℎfrom node 𝑖 to node 𝑗. A symbolic element, e.g., A3
1,3 = (1∗1∗)1
, indicates
that a feasible path of length 3 from 𝑖 and 𝑗 involves one call edge, one intra-procedural edge, and
one return edge. This multiplication of symbolic elements represents a value-flow concatenation by
strictly preserving the order of call and return edges when traversing on the IVFG. Here, we only
care about the call/return sequence to filter out "unrealizable" interprocedural paths by matching
calls and returns. For example, A3
1,4 = (1∗1∗)2
signifies an infeasible path from ℓ1 to ℓ4 due to
1The power of matrix A is defined as Aℎ = A · ... · A/bracehext/bracehext/bracehext/bracehext/bracehext/bracehext /bracehext/bracehext/bracehext/bracehext/bracehext/bracehext
×ℎ
and A0 is the identity matrix.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
233:8 Yulei Sui, Xiao Cheng, Guanqin Zhang, and Haoyu Wang
stack call 
Express‐
Smpt 
p Call
Express‐
Smpt 
Method declaration 
BlockSmpt 
queue call 
Express‐
Smpt 
Express‐
Smpt 
q Call
Malloc Malloc bar stack bar queue
Foo
Fig. 4. The AST of foo in Figure 3(a) with two spurious paths, i.e., the path between stack and q (blue line)
and the path between queue and p (red line).
unbalanced parentheses, which implies that stack can not pass its value to 𝑞. Once all the symbolic
elements in a matrix of ℎ-th power of A have been resolved based on CFL-reachability [ Reps 1998 ]
and replaced with either 1 or 0, yielding two context-sensitive reachability matrices A2 and A3 for
paths of lengths 2 and 3.
High-Order Proximity Embedding . To preserve the high-level code structure (e.g., the ensem-
ble of all feasible value-flow paths) and approximate the graph’s high-order proximity, we choose
Katz Index [ Katz 1953]. The resulting proximity matrix M = /summationtext.1𝐻
ℎ=1 (𝛽· A)ℎ preserves the value-flow
reachability, where 𝐻 is the maximum limit for power of A and 𝛽 is the decay parameter that
determines how fast the weight of an edge decays when the length of a path grows. This guarantees
convergence. We choose 𝐻 = 3 and 𝛽 = 0.8 in our example. We adopt the approach in [ Ou et al .
2016] to reduce the computational cost of SVD (singular value decomposition) when approximating
and decomposing the high-order proximity matrix into 𝐾-dimensional embedding vectors D𝑠 and
D𝑡, as shown in Figure 3(c). The parameter 𝐾, represents the number of feature dimensions to
support tunable precision for our embedding. The higher value of 𝐾 becomes, the more precise the
embedding is.
Figure 3(c) also illustrates the reachability computation using the 3-dimensional embedding
vectors (the first three dimensions of D𝑠 and D𝑡 highlighted in blue). A closer distance between a
source d𝑠
𝑖
and a target d𝑡
𝑖 vector means their inner product d𝑠
𝑖·d𝑡
𝑗
𝑇
will have a higher proximity value.
Compared to ℓ6, ℓ5 is closer to ℓ1 on the IVFG. Accordingly, the value d𝑠
1 · d𝑡
5
𝑇 =0.75 is higher than
d𝑠
1 · d𝑡
6
𝑇 =0.71 in the embedding space. A proximity value under a user-defined threshold indicates
that ℓ𝑖 unlikely reaches ℓ𝑗 (our empirical threshold value of 0.003 under 110 feature dimensions can
achieve 96.4% precision and 94.5% recall in our evaluation). Thus, Flow2Vec precisely preserves
reachability from ℓ1 to ℓ3 with the proximity value of d𝑠
1 · d𝑡
3
𝑇 = 0.5 and it also identifies two
infeasible value-flows from ℓ1 to ℓ4 with d𝑠
1 · d𝑡
4
𝑇 = −0.02 and from ℓ2 to ℓ3 with d𝑠
2 · d𝑡
3
𝑇 = −0.02.
Note that d𝑡
1 and d𝑡
2 are zero vectors, implying that no directed edges go to these two nodes.
Applications. Our embedding provides a new code representation that is useful to support,
for example, code classification and summarization tasks that require comprehensive interpro-
cedural alias-aware program dependence information. Unlike code2vec [Alon et al . 2019b] and
code2seq [Alon et al . 2019a] which embed the paths on a program’s ASTs, Flow2Vec extracts
deeper code structure information by exploring context-sensitive value-flow paths. Figure 4 shows
foo’s AST, which is intra-procedural and flow- and context-insensitive by nature. There two spu-
rious paths on the AST. One is between stack and 𝑞(blue); the other is between queue to 𝑝 (red).
These spurious paths suggest false information that 𝑝 may operate on the queue. In fact, 𝑝 only
accesses stack and there is no operation on queue in the remaining part of foo, as indicated in
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
Flow2Vec: Value-Flow-Based Precise Code Embedding 233:9
the comments (green). Therefore, imprecise code information, such as spurious AST paths, can
adversely affect the accuracy of code classification and summarization, as also demonstrated in our
experiments in Section 5.2 and our case studies in Figure 12.
4 FLOW2VEC APPROACH
This section details our value-flow embedding approach, including pre-embedding, value-flow
reachability via matrix multiplication and high-order proximity approximation by preserving
asymmetric transitivity of value-flows in low-dimensional embedding space.
4.1 Pre-embedding
Our pre-embedding phase accepts the source code of a project and compiles the code into LLVM-IR.
The memory SSA form and the interprocedural value-flow graph are then built using Andersen’s
pointer analysis [ Andersen 1994 ; Sui and Xue 2016 ]. The indirect function calls are resolved using
the points-to information from pointer analysis. The IVFG is then translated into a symbolic matrix
A with the call site IDs as placeholders for interprocedural value-flows. For example, if the two
value-flows are passing to and returning from a callee through the same callsite 𝑖, the call and the
return value-flow edges are labeled with an opening parenthesis (𝑖 and a closing parenthesis )𝑖
respectively.
4.2 Value-Flow Reachability as Matrix Multiplication
The context-sensitive value-flow reachability is formulated as a chain-matrix-multiplication problem
to compute a high-order proximity matrix M given the adjacency matrix A of an IVFG. Note that A
is sparse based on our sparse value-flow graph (Section 2.2). In Section 4.2.1, we introduce the matrix
multiplication to obtain the context-insensitive value-flow reachability. Then, in Section 4.2.2, we
move to context-sensitive reachability.
4.2.1 Context-Insensitive Value-flow Reachability. For context-insensitive value-flow reachability,
the symbolic matrix A is treated as a binary adjacency matrix, where the variable name on a
value-flow edge and the callsite information on an interprocedural edge are ignored for context-
insensitive analysis. The element A𝑖,𝑗 of A is 1 if there a directed edge from 𝑖to 𝑗 (with a path of
length of 1) exists in the graph, otherwise A𝑖,𝑗 is 0. The matrix element A𝑖,𝑗 in the ℎ-th power of the
adjacency matrix represents the number of paths from node 𝑖to 𝑗 with a length of ℎ. For example,
the element at the 𝑖-th row and 𝑗-th column of A2 is 1 if there exists one path with a length of 2
from node 𝑖 to node 𝑗.
Asum =
𝐻/summationdisplay.1
ℎ=1
Aℎ, (1)
Equation 1 formulates value-flow reachability as the sum of all powers of the adjacency matrix A
over the natural number lattice N. The equation can be used to compute the reachability information
between any two nodes on IVFG for path lengths from 1 to 𝐻.
Example 1. Figure 5 depicts a context-insensitive value-flow graph without labeled edges. A1
represents the first-order reachability and A2 represents the second-order reachability through
paths of length of 2 between any two nodes. Asum is the final matrix considering all reachability
closures for path lengths of both 1 and 2. Asum
1,3 = 2
means that there are two paths on the graph
from ℓ1 to ℓ3, i.e., ℓ1 → ℓ3 of path length 1 and ℓ1 → ℓ2 → ℓ3 of length 2.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
233:10 Yulei Sui, Xiao Cheng, Guanqin Zhang, and Haoyu Wang
Fig. 5. Value-flow reachability via matrix multiplication ( A𝑠𝑢𝑚 represent value-flow reachability between
any two nodes for path lengths from 1 to 2).
 
 
Fig. 6. Context-sensitive value-flow reachability via symbolic matrix.
4.2.2 Context-Sensitive Value-flow Reachability. When applying Equation 1 to the symbolic matrix
A for context-sensitive analysis, the value of an element Asum
𝑖,𝑗 in the resulting matrix Asum may not
be a numeric value, but it can be a symbolic expression, which represents the interprocedural value-
flow path(s). The multiplication of two symbolic elements represents a value-flow concatenation by
preserving the order of call and return IVFG edges. The addition of two symbolic elements, same
as the context-insensitive analysis, represents that there are multiple possible paths between two
nodes.
Unlike in the insensitive analysis, an interprocedural value-flow path involving call and/or
return edges between two nodes can not be simply treated as feasible. Rather, it is recognized as a
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
Flow2Vec: Value-Flow-Based Precise Code Embedding 233:11
feasible interprocedural path using CFL-reachability [ Reps 1998 ; Sridharan and Bodík 2006 ]. CFL-
reachability is an extension of standard graph reachability which allows filtering out unrealizable
paths on the graph. We define the context-free grammar 𝐶𝐹𝐺 = (𝑄,𝑁,𝑃,𝑆 ) for our analysis,
where 𝑄 is the alphabet over the sets of opening and closing parentheses, i.e., { ( 1,(2,..., (𝑛 } and
{ )1,)2,..., )𝑛 } and an empty string 𝜖. 𝑁 denotes nonterminal symbol. 𝑃 is a set of production rules
and 𝑆 is the start symbol. The language 𝐿of size 𝑛(the number of callsites in a program) using the
context-free grammar 𝐶𝐹𝐺 is defined as follows:
𝐿: 𝐶 = 𝐶𝐶 | ( 1 𝐶 )1 | ... | ( 𝑛 𝐶 )𝑛 | 𝜖
where 𝐶 is the start symbol and the only nonterminal in the grammar. This language supports a
special and restricted CFL, also called Dyck-CFL, which only generates strings of properly balanced
parentheses [Kodumal and Aiken 2004 ]. Given an IVFG 𝐺 with edge labels taken from alphabet 𝑄.
Note that intraprocedural edges are labeled with 𝜖. Each path 𝑝 in 𝐺 is labeled with a string 𝑠(𝑝)
in 𝑄, obtained by concatenating edge labels in order. We say 𝑝 is a 𝐿-path if 𝑠(𝑝) ∈ 𝐿. A 𝐿-path
is a realizable path if after entering a method 𝑚from callsite 𝑖, it exits from 𝑚back to callsite 𝑖.
Following [Sridharan and Bodík 2006 ], a realizable also allows partially balanced parentheses (i.e.,
a prefix with unbalanced closed parentheses and a suffix with unbalanced open parentheses) since
a realizable path may not start and end in the same method.
Example 2. Figure
6 shows an IVFG consisting of one intraprocedural edge (labeled with 𝜖) and
four call edges and two return edges. The symbolic matrix Asum = A1 + A2 + A3 represents the
value-flow reachability with path lengths of 1, 2 and 3. The symbolic element Asum
2,4 = 𝜖∗ ( 3+(2∗)2
represents two paths ℓ2
𝜖
− →ℓ3
(3
− →ℓ4 and ℓ2
(2
− →ℓ6
)2
− →ℓ4. According to our CFL-reachability
analysis, both paths are realizable, one partially and the other fully-balanced. Similarly, the element
Asum
2,7 = ( 𝜖 ∗ ( 3+(2∗)2
) ∗ ( 4
represents two feasible paths from ℓ2 to ℓ7 with a common suffix
)4. However, Asum
1,4 = (1∗)2
signifies an infeasible path from ℓ1 to ℓ4 because the string (1∗)2 is
unbalanced, which violates our CFL grammar.
4.3 High-Order Proximity Embedding
After obtaining the matrices with different path lengths, this section details the high-order proximity
embedding and decomposing the high-order proximity matrix into source and destination matrices
for the nodes on the IVFG to preserve context-sensitive asymmetric transitivity of value-flows.
We also mathematically measure the approximated error of our graph embedding in the low-
dimensional space. The key is to choose an appropriate high-order proximity of the IVFG to
precisely approximate the two embedding matrices:
M ≈ D𝑠 · D𝑡⊤, (2)
where D𝑠 is the source embedding matrix (with 𝑁 source vectors for 𝑁 nodes) and D𝑡 is the target
embedding matrix (with 𝑁 vectors).
As one of many high-order proximity measurements, the Katz Index [ Katz 1953], which ensembles
all paths between each pair in a directed graph, is a commonly used measurement for all-pair
reachability:
M =
𝐻/summationdisplay.1
ℎ=1
(𝛽· A)ℎ, (3)
where the high-order proximity matrix M is a weighted summarization over the paths between
two nodes. A symbolic matrix Aℎ is concretized by mapping each of its symbolic elements to a
numerical value based on CFL-reachability (Section 4.2.2). M adds a decay parameter 𝛽to Equation 1
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
233:12 Yulei Sui, Xiao Cheng, Guanqin Zhang, and Haoyu Wang
to determine the speed at which the weight of a path decays when the length of a path extends.
Therefore, a higher proximity value of an matrix element, e.g., M𝑖,𝑗 from node 𝑖to node 𝑗 implies
the more and the shorter feasible value-flow paths from 𝑖 to 𝑗.
The final high-order proximity matrix M is then decomposed through SVD (Singular Value
Decomposition) to produce the source and target embedding matrices:
M = U𝚺V⊤ =
𝑁/summationdisplay.1
𝑖=1
𝜎𝑖u𝑖v⊤
𝑖 , (4)
where 𝜎𝑖 is the 𝑖𝑡ℎ singular value, u𝑖 and v𝑖 are the corresponding left singular vector and right
singular vector. The largest 𝐾 singular values and the corresponding singular vectors are then used
to generate the 𝐾-dimensional embedding vectors:
D𝑠 =
[ √𝜎1 · u1,· · · ,√𝜎𝐾 · u𝐾
]
=
[
d𝑠
1,· · · ,d𝑠
𝑁
]
(5)
D𝑡 =
[ √𝜎1 · v1,· · · ,√𝜎𝐾 · v𝐾
]
=
[
d𝑡
1,· · · ,d𝑡
𝑁
]
(6)
where 𝐾-dimensional vector d𝑠
𝑖
(d𝑡
𝑖) is the 𝑖-th row of D𝑠 (D𝑡) in the embedding space R𝑁∗𝐾. The
inner product between d𝑠
𝑖
and d𝑡
𝑗 (i.e., d𝑠
𝑖 · d𝑡
𝑗
⊺
) is used to represent the proximity from node 𝑖 to
node 𝑗.
We have adopted the algorithm in [ Ou et al . 2016] to approximate the embedding vectors directly,
without the calculation of the high-order proximity matrix M if all the elements of A are numerical.
The decay parameter 𝛽 is set to be less than the spectral radius of A to ensure its invertibility.
With the Jacobi-Davidson iteration of GSVD [ Hochstenbach 2009 ], the total time complexity is
𝑂(|𝐸| · 𝐾2 · 𝐿), where |𝐸| is the number of edges on the IVFG, 𝐾 is the embedding dimension and 𝐿
is the number of iterations. When 𝐾 << 𝑁, the time complexity is linear to the number of edges.
Note that IVFG is usually a sparse graph [ Choi et al . 1991; Hardekopf and Lin 2011 ], making |𝐸|
much smaller than 𝑁2.
Approximation Error, Recall and Precision . Approximation errors (losses) are common in
graph embedding techniques due to their approximation nature (e.g., dimensionality reduction to
trade precision for efficiency) [ Ou et al . 2016; Perozzi et al . 2014; Song et al . 2009; Tang et al . 2015].
This loss can cause imprecise and/or unsound preservation of value-flow reachability. The aim of this
section is to mathematically define the approximation error of Flow2Vec to quantify its precision,
with theoretical guarantees on the Root Mean Squared Error (RMSE), which is often missed by
many blackbox learning-based approaches [ Allamanis et al . 2018; Ben-Nun et al . 2018; Li et al .
2018; Zhou et al . 2019]. For a given high-order proximity matrix M and the learned approximation
˜M = D𝑠 · D𝑡⊤, the error in the Frobenius Norm is calculated as
/bardblex
/bardblex M − ˜M
/bardblex
/bardblex
𝐹 =
√
/summationtext.1𝑁
𝑖=𝐾+1 𝜎2
𝑖 , and the
standard RMSE is:
𝑅𝑀𝑆𝐸 =
√ /bardblex
/bardblex M − ˜M
/bardblex
/bardblex 2
𝐹
𝑁2 =
√ /summationtext.1𝑁
𝑖=𝐾+1 𝜎2
𝑖
𝑁 , (7)
where 𝜎𝑖 is the 𝑖𝑡ℎ singular value of M and 𝐾 is the number of embedding dimensions. Additionally,
for a given embedding dimension 𝐾, the lower the rank of M is, the smaller the error is. When 𝐾 is
equal to 𝑟𝑎𝑛𝑘(M), the error becomes zero. The precision and recall (soundness) are computed as
follows:
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = |E𝑐|
|E𝑝| , 𝑅𝑒𝑐𝑎𝑙𝑙 = |E𝑐|
|E𝑜| , (8)
where E𝑝 is the set of predicted reachability between any two nodes, E𝑜 is the set of observed
reachability relations on the original graph, and E𝑐 = {(𝑖,𝑗 ) | ( 𝑖,𝑗 ) ∈ ( E𝑝 ∩ E𝑜}) Note that the
high-level reachability information can still be preserved with very high precision and soundness
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
Flow2Vec: Value-Flow-Based Precise Code Embedding 233:13
given a small approximation error in the low-level embedding space, as also explained in Example 3
and evaluated in Figure 5.2 in Section 7.
Example 3. Let us revisit the motivating example to measure the approximation error of D𝑠 and D𝑡.
The singular values for M are 𝜎 = [1.92,1.09,0.51,0.49,0.00,0.00]. In a 3-dimensional embedding
(i.e., 𝐾 = 3), RMSE =
√/summationtext.16
𝑖=4 𝜎2
𝑖
6 = 0.08, while in the 2-dimensional embedding, RMSE =
√/summationtext.16
𝑖=3 𝜎2
𝑖
6 =
0.12. Given 0.25 as the threshold to determine the inter product between two embedding vectors,
the 3-dimensional embedding perfectly preserves the graph’s context-sensitive reachability with
100% precision and recall, while the 2-dimensional embedding achieves 97% precision and 97%
recall in this example.
4.4 Applications of Flow2Vec
This section discusses the Flow2Vec’s embedding results as inputs to boost the performance of
two recent learning-based approaches, i.e., code2vec for code classification [ Alon et al . 2019b] and
code2seq for code summarization applications [ Alon et al. 2019a ].
4.4.1 Code Classification. As the name implies, code classification aims to perform semantic
labeling of a code fragment. This application is essentially a multi-class classification problem.
Given a code fragment 𝑥 (in the form of a program method), the objective is to train a prediction
model to learn the label distribution conditioned on the code, i.e., 𝑃(𝑦𝑖|𝑥), where the label 𝑦𝑖 is
from a pre-defined vocabulary 𝑌, that is a set of tags/names in the training corpus. The predicted
distribution of the model 𝑞(𝑦𝑖) is computed using a softmax function, the dot product between the
code vector v of 𝑥 and the vector representation 𝑙𝑎𝑏𝑖 of each label 𝑦𝑖 ∈ 𝑌.
𝑓𝑜𝑟𝑦 𝑖 ∈ 𝑌 : 𝑞(𝑦𝑖) = exp(v⊤ · labj )/summationtext.1
𝑦𝑗 ∈𝑌 exp(v⊤ · labj ) , (9)
Unlike Word2Vec-like techniques [ Mikolov et al . 2013] which use textual code tokens to produce
the code vector, whereas the code2vec [Alon et al . 2019b] approach leverages the structural features,
that is, the abstract syntax tree (AST) of each program method. The approach aggregates multiple
syntactic paths on an AST into a single code vector representation v, which is then used for multi-
class classification. To further improve the precision, code2vec includes an attention mechanism
to focus on important AST paths that contribute more to predication accuracy [ Allamanis et al .
2016; Luong et al. 2015 ].
However, by its nature, a tree data structure is an undirected graph that can not preserve
asymmetric transitivity. Moreover, the paths on an AST can not reflect the alias-aware data-
flow information. To demonstrate that Flow2Vec can preserve more comprehensive structural
information than code2vec, we embed value-flow paths rather than AST paths to generate the
code vector for code classification. Given the asymmetric vectors
d𝑠
𝑖 ∈ D𝑠
and d𝑠
𝑗 ∈ D𝑡 for a
code fragment computed by Equations 5 and 6, the path vector c𝑛 of a value-flow path vfp𝑛 from
statement ℓ𝑖 to ℓ𝑗 is:
c𝑛=tanh(W · 𝑒𝑚𝑏𝑒𝑑(⟨ℓ𝑖,vfp𝑛,ℓ𝑗⟩)) = tanh(W · [ ℓ𝑖; d𝑠
𝑖 · d𝑡
𝑗
⊺
; ℓ𝑗]) (10)
where W is a learned weights matrix that serves as a fully connected layer before aggregating
all the path vector and 𝑡𝑎𝑛ℎ is the monotonic nonlinear activation function. The value of the dot
product d𝑠
𝑖 and d𝑠
𝑗 indicates two important distance relations, i.e., the length of each path and the
number of paths between 𝑖and 𝑗, relative to all other value-flows in the embedding space. A higher
proximity value means the more and the shorter paths from 𝑖to 𝑗.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
233:14 Yulei Sui, Xiao Cheng, Guanqin Zhang, and Haoyu Wang
The final code vector v is an aggregation of all the path vectors constructed by embedding 𝑆
value-flow paths between any two nodes on the IVFG of a code fragment:
v =
𝑆/summationdisplay.1
𝑛=1
𝛼𝑛 · c𝑛, (11)
where 𝛼𝑛 = exp(c𝑛 ⊤ ·a)/summationtext.1𝑆
𝑗=1 exp(c𝑗 ⊤ ·a) is the attention weight of vector c𝑛 and a is a global attention vector
learned during model training [ Alon et al. 2019b ].
4.4.2 Code Summarization. The aim of code summarization is to translate a code fragment
into a sequence of words to describe the semantic meaning of the code. The summarization is
performed under an encoder-decoder framework, which has been widely used in the neural machine
translation [ Luong et al. 2015 ] and image captioning [ Xu et al. 2015 ].
The encoder first transforms the code snippet x into a sequence of hidden states in the form
of a code vector v, while the decoder generates one word 𝑦𝑡+1 at a time step. Generating the
summary is conditioned on all previously generated words 𝑦1:𝑡 and the hidden states from the
encoder. Hence, the objective of code summarization is to model the conditional probability:
𝑃(𝑦1,...,𝑦 𝑚 | x) = /producttext.1 𝑚
𝑡=1 𝑃(𝑦𝑡 | 𝑦1:𝑡−1,v).
As with the code classification task, the value-flow paths are encoded to generate the code vector
v as opposed to the AST paths from code2seq [Alon et al . 2019a]. Note that, to improve precision,
both approaches use the same attention mechanism to produce the final attentional code vector.
5 EVALUATION
The objective of this evaluation is to determine whether Flow2Vec is effective at preserving
value-flows in low dimensional embedding space. For this, we evaluate two aspects of Flow2Vec:
(1) approximation error, precision and recall through graph reconstruction, and (2) the performance
improvement over code2vec and code2seq on code classification and code summarization tasks.
5.1 Dataset and Evaluation Methodology
Implementation and Datasets . The interprocedural alias-aware value-flow graph of a program
is built on top of LLVM-IR (with LLVM version 9.0.0) and its sub-project SVF [ Sui and Xue 2016 ].
Program functions, which are represented as global variables in the LLVM-IR, are modeled as
address-taken variables (Section 2.1). The callgraph of a program is built on-the-fly (with indirect
calls discovered) when performing Andersen’s pointer analysis [ Andersen 1994]. The value-flows
across procedures are soundly captured on top of the conservative callgraph built using Andersen’s
analysis. We have implemented the CFL-reachability algorithm (Section
4.2.2) to recognize realizable
context-sensitive value-flow paths on IVFG. Katz Index [ Katz 1953] is used to produce high-order
proximity matrix, through which the asymmetric embedding vectors are generated.
We evaluated Flow2Vec using 32 popular open-source C/C++ projects (with their statistics
shown in Table 1) across a variety of domains: a2ps (postscript filter tool), bash (sh-compatible
shell), bc (basic calculator), bison (parser generator), ctypes_test (builtin test tool), dc (desk
calculator), decimal (python tool), echogs (port upgrate), gdbserver (remote debugging), gzip
(file compress), grep (commandline search tool), hashcat (passwd recovery), htop (process viewer),
keepalived (IP link checker), less (pager program), lua (interpreter), libsass (sass engine), make
(executable builder), msg-convert (messagepack switcher), msg-structure (messagepack pattern
former),msg-map (messagepack multi-keys API), mkromfs (file system), mocktracer (a code tracer),
ossaudiodev (python audio API), redis-client (database client), redis-server (database server),
redis-benchmark (redis benchmark), sample (Unix selection tool) screen (terminal emulator),
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
Flow2Vec: Value-Flow-Based Precise Code Embedding 233:15
Table 1. The statistics of the open-source programs. #LOI denotes the number of lines of LLVM instructions.
#Function, #Pointer, #Object, #Call, |V| and |E| are the numbers of functions, pointers, objects, method calls,
IVFG nodes and IVFG edges, respectively.
#LOI #Method #Pointer #Objects #Call |V| |E|
a2ps 118173 490 94173 6912 9036 122345 210783
bash 312144 783 235887 17630 24998 372242 503357
bc 18683 86 14744 909 1583 23207 32202
bison 98575 400 79820 3404 15978 160506 237364
ctypes_test 1467 93 1275 150 232 1203 1013
dc 11629 97 9167 871 1132 14651 17825
decimal 72546 193 57496 3278 22291 85989 98107
echogs 1034 18 743 46 146 589 575
gdbserver 64324 348 53196 3225 15091 108522 119497
gzip 32411 136 24944 1950 2030 32846 49806
grep 37796 221 28134 1639 8141 39532 52788
hashcat 177602 412 150885 5318 16281 234648 459262
htop 62554 293 29170 1456 7416 37367 43980
keepalived 214163 575 99354 3066 24142 162032 215899
less 37372 224 28843 2524 3333 81584 91222
lua 68222 346 52585 1869 4764 100701 106830
libsass 818989 2734 605654 39805 47343 553499 882694
make 42676 190 33200 2027 7209 82157 99750
msg-convert 241287 449 18417 3447 1757 25854 28499
msg-structure 222986 498 20065 3856 1873 35354 35650
msg-map 293360 640 24370 4784 2204 24876 34965
mkromfs 18751 155 15614 589 3196 19054 28448
mocktracer 214038 3688 176632 40666 19342 197830 300333
ossaudiodev 3107 61 2990 302 769 2833 2481
redis-client 322115 557 172664 5262 66297 241070 284082
redis-server 735275 1875 401520 14461 119216 654495 778854
redis-benchmark 299719 505 183609 4527 59041 253210 295056
sample 69720 383 55608 4797 5965 65894 94460
screen 117552 286 90984 3601 13899 321269 597668
sed 40708 177 30401 1570 5944 52405 83748
sendmail 153184 616 121604 6216 25384 529537 744380
tar 87418 386 66191 5052 17279 183538 207659
Total 4922162 17529 2913748 190157 536033 4637301 6531578
sed (Unix editor), sendmail (mail agent), tar (file compression). Our experiments were conducted
on a machine with Intel Xeon Gold 6132 @ 2.60GHz CPUs and 128GB of RAM.
Evaluation Methodology . Like all existing embedding approaches, Flow2Vec does not aim to
achieve 100% precision and recall, but rather to obtain a comprehensive code representation to
better support subsequent client applications. To validate Flow2Vec’s effectiveness, we conducted
three evaluation tasks, graph reconstruction, code classification and code summarization.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
233:16 Yulei Sui, Xiao Cheng, Guanqin Zhang, and Haoyu Wang
Graph reconstruction is a typical method to evaluate graph embedding results [ Ou et al . 2016;
Perozzi et al . 2014; Tang et al . 2015]. We reconstructed the graph edges on the IVFG of each of
our 32 benchmarks based on the reconstructed proximity, i.e., the inner products between the
embedding vectors for each node pair. Note that 𝐻 and 𝛽 in Equation 3 were set to be 3 and 0.01 to
produce the proximity matrix. Nodes with no incoming or outgoing edges on the IVFG were not
included in the proximity matrix to reduce computational overhead. Following [ Ou et al . 2016],
we calculated the ratio of real links for the top 𝑘 pairs of nodes as the reconstruction precision by
randomly sampling about 0.1% of the node pairs to evaluate the approximation error, precision and
recall for graph reconstruction.
To further validate the effectiveness and practicality of our approach, we choose two important
and challenging clients, classification and summarization, for which prior results still have much
room for improvement [ Allamanis et al . 2016; Alon et al . 2019a, 2018, 2019b; Iyer et al . 2016]. We
then compared Flow2Vec with the two recent open-source tools code2vec [Alon et al . 2019b]
and code2seq [Alon et al . 2019a]. From the 32 open-source projects, we extracted 11,291 program
methods with informative comments, including the method names and the comments of each
method, with the AST path of each method extracted by pathminer [
Kovalenko et al . 2019]. A
method is not used for our training if it does not have any comment or it is an unused method
in the project (i.e., it never calls other methods or being called by others). For both clients, we
randomly split our collected programs into three subsets, of which the proportions are about 80%,
10% and 10% for training, validation and testing, respectively. Note that code2vec and code2seq
have already shown that they have better performance compared with CodeNN [ Iyer et al . 2016],
ConvAttention [Allamanis et al . 2016], Paths+CRFs [ Alon et al . 2018] and TreeLSTM [ Tai et al .
2015], therefore, we did not compare Flow2Vec with these approaches.
To produce the path vectors based on Equation 10, we collected a set of IVFG nodes (with each
node being a definition of a variable) for each program method. The context-sensitive value-flows
between any two nodes are represented by the dot product of the nodes’ embedding vectors. Note
that the value-flow paths between two nodes in the same method can be interprocedural (e.g., p and
stack in Figure 3(a)). Following [ Alon et al . 2019b], the value-flow paths are randomly sampled for
each method (i.e., S = 200 in Equation 11). The path vectors are then aggregated into a fixed-length
code vector using the attention mechanism (Equation 11). The attention network is trained in a
batch-wise fashion and the batch size is set to 512. The dropout is set to 0.25 and the number of
epochs is set to 20. ADAM [
Kingma and Ba 2015 ], a commonly-used adaptive gradient descent
method, is used for training. We used grid search to perform hyper parameter tuning in order
to determine the optimal values for a given model. Note that our focus here is on evaluating the
effectiveness of our value-flow embedding, not on different backend learning models. Picking
different models and feeding Flow2Vec’s embedding vectors into these backend models to further
enhance the performance is an orthogonal topic.
The evaluation metrics used to assess code classification and summarization performance are
precision, recall and F1-score. The generated sequence of words were evaluated in a case-insensitive
manner and calculated over (sub)tokens following previous studies [ Allamanis et al . 2015, 2016;
Alon et al . 2019a,b]. The key idea is to measure the quality of a sequence prediction by decomposing
it into sub-tokens and calculate correct predictions based on them. Take method name prediction
for example, given a method named swapElements, we first decompose it into swap and elements.
A prediction of elementsSwap is an exact match, while a prediction of swapAbstractElements
has a full recall but low precision and a prediction of swap has a full precision but low recall.
Furthermore, the unseen sub-token in the token vocabulary from the test label is considered as
a false negative. We used a similar strategy to evaluate the code summarization by measuring
precision and recall based on tokens of the generated code comments.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
Flow2Vec: Value-Flow-Based Precise Code Embedding 233:17
5.2 Experimental Results and Analysis
Precision, Recall and RMSE . Figure 7 gives the precision, recall and approximation error of
the reconstructed graph based on Equations 8 and 7 for each of the 32 projects. RMSE is normally
used when evaluating graph embedding techniques [ Ou et al . 2016; Song et al . 2009] to measure
the gaps between the prediction result and the ground truth. A smaller RMSE indicates a better
embedding result. During value-flow embedding in the latent space, we assessed different feature
dimensions, ranging from 10 to 210.
From the results in Figure 7, we can see that the error rate (RMSE) of each program decreases
dramatically, while the precision and recall increase very quickly, particularly, in the beginning part
(less than 60 dimensions). The results demonstrate that Flow2Vec is effective at preserving the
asymmetric transitivity with low-dimensional embeddings. With 110 dimensions, the average error,
precision and recall for the 32 projects were 2.46*10 −4, 96.9% and 95.5% respectively and an error
up to 8.3*10 −4 for program mocktracer. Additionally, the RMSE was very small, but precision and
recall are high both for the smallest program, i.e., ossaudiodev with 1.58*10−4 error, 0.99 precision
and 0.99 recall, and the largest program, i.e., libsass with 6.2*10 −4, 0.83 and 0.87, respectively.
These results confirm that Flow2Vec precisely preserves asymmetric transitivity of value-flows in
the embedding space.
Efficiency. Figure 8 shows the total running time (in seconds) of Flow2Vec, including the times
for value-flow construction (red) and high-order proximity embedding (blue). The embedding
dimension was set to 110. In summary, Flow2Vec finished analyzing the largest program libsass
within 64.2 mins and for all programs consisting of over 5 million lines of LLVM instructions with
272.5 mins, showing that Flow2Vec scales to large projects for value-flow construction and graph
embedding using fast high-order proximity computation. Note that although our interprocedural
embedding consumes longer analysis time than intraprocedural analysis (e.g., collecting AST
paths of individual methods [
Alon et al . 2019a,b]), the embedding process is only done once to
produce the distributed code representation before feeding into subsequent learning tasks. Moreover,
the increase in precision, recall and F1 outweighs the interprocedural analysis overhead as also
illustrated in Table 2 and Figure 9.
Code Classification. The second and third Columns in Table 2 compare the results of Flow2Vec
and code2vec for code classification. It is clear that Flow2Vec significantly outperforms code2vec
in terms of precision, recall and F1-score with an average improvement of 21.2%, 20.1% and 20.7%,
respectively. This confirms that our comprehensive code representation yields better results than the
approaches that extract shallow code structure information to predict the names of code fragments.
Figure 9 gives the changes of the F1-scores as the size of a program method grows. Flow2Vec
gave the best (or equal best) results for all code lengths. With small lengths of code (e.g., 1-10 lines)
both code2vec and Flow2Vec observe their best performance. The F1-scores of both approaches
decreased as the size of the method increased. However, at larger sizes, i.e., 35 lines and above,
Flow2Vec was more stable and performed consistently better than code2vec. Note that code2vec’s
F1-score in Table 2 is lower than that presented in the code2vec paper, in which code classification
was performed on Java and C# programs. There are two reasons behind. First, the method body
of a program in C/C++ is generally larger than that in Java or C#, where the value-flows of each
code fragment is simpler. In our C/C++ dataset, the average length of a method is around 35 lines,
while in code2vec’s dataset, it is around 7 lines per method. The F1-score only becomes stable
in after round 35 lines as shown in Figure 9. Second, from our manual investigation of our code
bases, we found that many methods in C/C++ have more complicated data and control dependence
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
233:18 Yulei Sui, Xiao Cheng, Guanqin Zhang, and Haoyu Wang
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
gdbserver
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
dc
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
grep
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
less
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
sed
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
tar
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
sample
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
decimal
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
screen
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
make
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
ctype_test
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
echogs
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
sendmail
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
gzip
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
hashcat
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
htop
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
keepalived
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
lua
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
redis_server
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
mkromfs
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
msg -structure
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
msg -convert
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
msg -map
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
ossaudiodev
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
redis_benchmark
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
mocktracer
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
redis_client
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
libsass
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
a2ps
Precision
Recall
RMSE
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
bash
Precision
Recall
RMSE
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
bc
Precision
Recall
RMSE
Score of Precision & Recall
0
0.3
0.6
0.9
1.2
1.5
0.6
0.7
0.8
0.9
1
10 60 110 160 210
RMSE*103
Dimension
bison
Precision
Recall
RMSE
Score of Precision & Recall
Fig. 7. Precision, Recall and RMSE of Flow2Vec under different embedding dimensions after graph recon-
struction.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
Flow2Vec: Value-Flow-Based Precise Code Embedding 233:19
Fig. 8. Total running time (secs).
0
10
20
30
40
50
60
70
80
1 5 10 15 20 25 30 35 40 45 50 55 60 65 70+
F1-SCORE
Fl ow2Vec (code classification) Code2Vec
Fl ow2Vec (code summarization) Code2Seq
Fig. 9. F1-score under different the lengths of code.
than those in Java or C# programs, which makes code classification more difficult for C/C++ than
Java/C#. It also demonstrates the necessity for a more comprehensive code representation.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
233:20 Yulei Sui, Xiao Cheng, Guanqin Zhang, and Haoyu Wang
Table 2. Precision, recall and F1 for the two applications.
Client Model Name Precision(%) Recall(%) F1-score(%)
Code code2vec 35.5 34.2 34.8
Classification Flow2Vec 56.7 54.3 55.5
Code code2seq 43.9 41.1 42.5
Summarization Flow2Vec 57.1 59.9 58.5
Code Summarization . The second task we used for comparison was code summarization,
pitting Flow2Vec against code2seq [Alon et al . 2019a] using the same set of methods with their
corresponding comments. In Table 2, we can see that Flow2Vec performed better than code2seq in
all the three metrics. Flow2Vec’s F1-score was 58.5% compared to 42.5% of code2seq. Flow2Vec’s
precision and recall were also higher, at 57.1% and 59.9% respectively, while code2seq only achieved
43.9% and 41.1% respectively.
Figure 9 shows the F1-scores for Flow2Vec and code2seq with different code lengths. The trend
here is similar to the comparison with code2vec. Overall, Flow2Vec outperformed code2seq
for all code lengths. The F1-scores for both Flow2Vec and code2seq were at or above 50% with
small code lengths (i.e. under 5 lines), but both became stable as the length increases. Notably,
Flow2Vec delivered similar performance for both code classification and code summarization,
whereas code2seq provided a much better result at code summarization than code2vec. This is
because we used the identical structural information of code (i.e., value-flow paths) for both tasks.
Additionally, code2vec encodes AST paths monolithically, so it cannot represent paths unless they
are included in the path vocabulary, while code2vec improves this by using a bi-directional LSTM
to embed AST paths token-by-token. Thus, it can represent any syntactic path assembled from a
limited size of path token vocabulary.
Note that code2vec and code2seq perform path analysis on top of ASTs of a method through
intra-procedural analysis and their dataset contains only disparate methods with each averaging
7 lines, rather than complete projects, making embedding less challenging under the small-code-
length setting. Analyzing them one at a time, though the large number, is less costly than analyzing
them together. However, adding more methods is subject to diminishing returns on precision. The
limitations of its intraprocedural approach are further exemplified when evaluated against our
complex large-scale projects where Flow2Vec’s interprocedural embedding yields better results by
extracting substantial structural information of code, i.e., flow- and context-sensitive value-flows.
The result is a more comprehensive code representation and, in turn, improved performance for
code classification and summarization tasks.
Embedding Dimension. Figure 10 further evaluates Flow2Vec’s F1-scores under different
feature dimensions during our value-flow embedding. There are similar trends in both tasks with
an increased F1-score by around 17% from dimension 10 to 110. The F1-scores are capped at
around 56.0% and 59.0% for code classification and summarization, respectively, when the number
of dimensions become even larger. This is consistent to the trends of the approximation error,
precision and recall when performing graph reconstruction under different feature dimensions,
where the results become stable and precise enough when the number of dimensions is around
110, a relatively low dimensional space. Note that the performance did not improve much when
over 110 dimensions. This is due to the nature of the distributed representation of code whose
semantic meaning is distributed across its vector components (dimensions). In our evaluation, 110
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
Flow2Vec: Value-Flow-Based Precise Code Embedding 233:21
0
10
20
30
40
50
60
70
80
10 30 50 70 90 110 160 210 260 310
F1-SCORE
Fl ow2Vec(code cl assification) Fl ow2Vec(code summarization)
Fig. 10. F1-score under different embedding dimensions.
dimensions is already precise enough to preserve the value-flows of a program in the latent space.
Adding more dimensions yields diminishing returns in performance as shown in Figure 10.
Code Classification Code Summarization
Alias-unawareAlias-unawareI nt ra procedrualI nt ra procedrualFlow2VecFlow2Vec Context-insensitiveContext-insensitiveSymmetricAlias-unawareI nt ra procedrualFlow2Vec Context-insensitiveSymmetric
30
35
40
45
50
55
60
Precision (%) Recall (%) F1 (%)
30
35
40
45
50
55
60
Precision (%) Recall (%) F1 (%)
30
35
40
45
50
55
60
Precision (%) Recall (%) F1 (%)
30
35
40
45
50
55
60
Precision (%) Recall (%) F1 (%)
Fig. 11. Ablation analysis for inter-procedural analysis, pointer aliases, asymmetric transitivity and context-
sensitivity.
Ablation Analysis. We conducted ablation analysis to evaluate each of Flow2Vec’s four proper-
ties, i.e., interprocedural analysis, aliases, asymmetric-transitivity and context-sensitivity, to further
understand their contributions to the final accuracy for the two tasks. We evaluated Flow2Vec’s
performance by removing each property at a time. (1) We removed the method-call edges to
see the influence of inter-procedural information. (2) We removed the indirect value-flow edges
(Section 2.2) to remove the pointer analysis results to observe the effectiveness of pointer aliases.
(3) We transformed the adjacency matrix of a value-flow graph to symmetric matrix when doing
graph embedding to evaluate the importance of asymmetric transitivity. And (4) we produced
a context-insensitive matrix rather than a symbolic matrix for the IVFG of a program without
performing CFL-reachability analysis in order to evaluate the context-sensitive property.
Figure 11 shows the result of ablation analysis for each property of Flow2Vec. Overall, Flow2Vec
(with four properties combined) performs better when one of its properties is disabled, with an
improvement ranging from 3.7% to 7.3% and 4.6% to 7.8% in terms of F1-score for code classification
and summarization respectively. It also demonstrates that the usefulness of all the four properties
for producing the final distributed code representation. The properties of interprocedural analysis
and pointer aliases have relatively high contributions to the precision/recall/F1 compared to the
other two properties when evaluating using our dataset. By looking into the methods collected
from our real-world projects, 91.0% methods have inter-procedural method calls. On average, there
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
233:22 Yulei Sui, Xiao Cheng, Guanqin Zhang, and Haoyu Wang
int  ____________ (dict * d ){
  int  minimal;
  minimal = d -> ht [ 0 ]. used ;
  if  (minimal<INITIAL_SIZE) minimal=INITIAL_SIZE;
  return  dictExpand (d,  minimal);
}
int  dictExpand (dict * d , unsigned  long  size ){
  dictht n;
  unsigned  long  realsize = _dictNextPower (size);
  n . table  = zcalloc (realsize* sizeof (dictEntry*));
  return  DICT_OK;
}
Ground truth
Code2Vec
Code2Seq
Flow2Vec
resize   to   the   minimal
isMinimal
Expand   size
reset   minimal   memory
Souce code (A)
Ground truth
Code2Vec
Code2Seq
Flow2Vec
Souce code (D)
char * ___ ___ __( const  char * name){
  Dir* directory = find_dir (name);
  if (directory== NULL ) return  NULL ;
  Pro* p = init_pro (directory);
  Dir* r = p ->g et_ro ot( ) ;
  return  generate_file (r, p - >sz );
}
Pro* 
init_pro (Dir* dir ){
  Pro* program = create_pro ();
  program-> set_root (dir);
  return  program;
}
generate  file  under  directory
generateFile
get  file
generate  file  in  directory
void  _____________(NODE *cur_node){
    NODE *c  = fd_node (cur_node);
    NODE *n  = fd_node ( c -> next );
    char  * value = node_value_get (c);
    if  (n  !=  NULL )
      node_value_set (n,  value);
}
Souce code (C)
Ground truth
Code2Vec
Code2Seq
Flow2Vec
set   current   value    to    next    node
getNodeValue
set   next    node   current    value  
set   node   value
void  _____________(NODE *cur_node){
    NODE *c  = fd_node (cur_node);
    NODE *n  = fd_node ( c -> next );
    char  * value = node_value_get (c);
    if  (n  !=  NULL )
      node_value_set (n,  value);
}
Souce code (C)
Ground truth
Code2Vec
Code2Seq
Flow2Vec
set   current   value    to    next    node
getNodeValue
set   next    node   current    value  
set   node   value
Ground truth
Code2Vec
Code2Seq
Flow2Vec
Souce code (B)
SIG  __________( char * inp ){
    if  (! is_valid (inp)) return  FAULT;
    CMD  *cmd = parse (inp);
    return  process (cmd);
}
SIG  
process (CMD * cmd ){
    Executor *e = get_ glb _executor ();
    if (e-> is_ full ) return  FULL;
    e-> execute (cmd);
    return  SUCESS;
}
parse  and  execute  command
check
parse
parse  command  and  execute
Ground truth
Code2Vec
Code2Seq
Flow2Vec
Souce code (B)
SIG  __________( char * inp ){
    if  (! is_valid (inp)) return  FAULT;
    CMD  *cmd = parse (inp);
    return  process (cmd);
}
SIG  
process (CMD * cmd ){
    Executor *e = get_ glb _executor ();
    if (e-> is_ full ) return  FULL;
    e-> execute (cmd);
    return  SUCESS;
}
parse  and  execute  command
check
parse
parse  command  and  execute
Fig. 12. Case studies.
are 30 interprocedural value-flows coming in and going out of a method. The 91.0% methods have
indirect value-flows with an average of 37 indirect value-flows per method. It demonstrates the
importance for considering interprocedural and alias-aware value-flows for code embedding. When
looking at the other two properties (i.e., asymmetric-transitivity and context-sensitivity), their
precision benefit is slightly lower but the average improvement still reaches approximately 5% in
terms of F1-score. The results confirm that all four properties are essential to comprehend and
preserve the structural feature of source code to deliver better results for complicated tasks (e.g.,
code classification and summarization).
Case Study . Figure 12 gives four real-world code snippets together with the ground truth
of their corresponding code comments, which are extracted from redis-server, bison, bash
and libsass respectively. Source code (A) and (B) are used to demonstrate the effectiveness of
Flow2Vec’s interprocedural analysis. Source code (C) and (D) are used to show the usefulness of
context-sensitivity and pointer aliases preserved by Flow2Vec.
The first method of source code (A) is to resize hash table to the minimal size that contains
all the elements. It can be seen that Flow2Vec can precisely capture the key words minimal and
resize from the ground truth, while code2vec failed to predict the token resize and code2seq can
not distinguish resize and expand. This is because the semantic of the current method depends on
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
Flow2Vec: Value-Flow-Based Precise Code Embedding 233:23
its callee dictExpand. code2vec and code2seq generated partial and imprecise results due to their
intra-procedural analysis. On the other hand, Flow2Vec’s inter-procedural analysis successfully
comprehended and incorporate the callee information into the summary, yielding more precise
prediction results.
The first method of in source code (B) is to parse a command sequence and then invoke a global
executor to execute the command. It is clear that Flow2Vec produced a more precise prediction by
capturing the three key words parse, command and execute through analyzing value-flows across
methods, while code2seq and code2vec failed to capture all these important tokens due to its
intra-procedural analysis nature.
The method in source code (C) is similar to our motivating example in Figure 3. The code snippet
is a helper method used to set the value of next node given current node’s value. As can be seen from
the prediction results of Flow2Vec and the two baselines, the summary produced by Flow2Vec
is more precise because it distinguishes the different operations of current node and the next
node. The prediction result is closer to the ground truth since our context-sensitive analysis is
able to recognize that operations on object c->next only occur through pointer n. Similarly, object
cur_node only operates through c when considering different calling contexts to callee fd_node.
code2vec and code2seq, however, are unable to differentiate the current node from its next node.
Because the intra-procedural paths on the AST of the code (as also explained in Figure 4) will
establish false relations between c and c->next and n and cur_node, as such, the final results are
imprecise.
The method presented in source code (D) is to generate files under a given directory. Flow2Vec
precisely identified the key word directory by capturing the alias relation between directory
and r through set_root and get_root to infer the more expressive name directory of the first
parameter at callsite generate_file. However, code2vec and code2seq which failed to capture
this complicated interprocedural alias relation, produced the imprecise summary result.
6 RELATED WORK
We limit our discussion to the work that is most relevant to Flow2Vec, namely, program dependence
analysis, graph embedding and code embedding.
Program Dependence Analysis . Program dependence analysis or value-flow analysis, which
reasons about the control- and data-flows of a program, has been well-studied over the past decades.
Initially, the program dependence analysis has been used for compiler optimizations [
Ferrante et al .
1987; Kuck et al . 1981]. Later, the analysis has been shown as an enabling technique for a wide
variety of client applications, such as program slicing [ Weiser 1981], software maintenance [ Gal-
lagher and Lyle 1991 ], change impact analysis [ Acharya and Robinson 2011 ], program complexity
metrics [ Rilling and Klemola 2003 ] and precise software bug detection [ Livshits and Lam 2003 ; Shi
et al . 2018]. By leveraging the advances in pointer alias analysis [ Barbar et al . 2020; Hardekopf and
Lin 2007, 2011], some recent works [ Shi et al . 2018; Sui and Xue 2016 ] have developed a more sound
and precise approach to program dependence analysis by considering memory aliases. The result
is a so-called sparse value-flow graph representation that captures the def-use relations of both the
top-level and address-taken variables. Flow2Vec is the first technique that can embed a compre-
hensive sparse value-flow representation into low-dimensional space while preserving asymmetric
value-flow transitivity to support two important clients, code classification and summarization.
Graph Embedding and Code Embedding . Graph embedding transforms nodes in a graph
into distributed representations and effectively preserves the network structure. There have been
some significant advances recently in graph embedding in a wide range of applications, including
classification [ Sen et al . 2008], clustering [ Wang et al . 2017], link prediction [ Wang et al . 2014]
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
233:24 Yulei Sui, Xiao Cheng, Guanqin Zhang, and Haoyu Wang
and data representation for machine learning methods. For many software analysis tasks, directed
graphs are the major representation for source code. Preserving asymmetric transitivity of graphs is
the fundamental requirement for an accurate graph embedding. HOPE [
Ou et al . 2016] approximates
the high-order proximity to preserve the high-order structure of directed graphs. Later, ATP [ Sun
et al . 2019] has subsequently emerged to improve upon HOPE with an ability to solve cycles on a
directed graph during high-order proximity embedding. Inst2Vec [ Ben-Nun et al . 2018] presents a
distributed representation of code statements based on contextual data-flow on top of LLVM IR.
However, the underlying data-flow graph used is intraprocedural and alias-unaware. Allamanis et
al. [ Allamanis et al . 2018] proposes a code embedding technique that embeds a program’s data-
flows using gated graph neural networks (GGNNs) for analyzing variable renaming and misuses
in C# programs. This approach is alias-unaware and the underlying embedding technique does
not support context-sensitivity. DeepSim [
Zhao and Huang 2018 ] transforms the control- and
data-flows of a program using a feed-forward neural network to measure functional code similarity.
ASTNN [ Zhang et al . 2019] is an intraprocedural and context-insensitive embedding technique
mainly for code clone detection based on the ASTs of a program. Like code2vec [Alon et al .
2019b] and code2seq [Alon et al . 2019a], the embedding does not preserve a program’s data- and
control-flows.
7 CONCLUSION
This paper presents Flow2Vec, a new code embedding approach that preserves interprocedural,
context-sensitive and alias-aware value-flows in the low-dimensional vector space. The value-
flow reachability is formulated as a chain-matrix-multiplication problem to support filtering out
infeasible value-flow paths through CFL-reachability. The resulting embedding vectors can be
used as a comprehensive code representation to better support subsequent learning tasks. We
have evaluated Flow2Vec using 32 popular open-source projects with over 5 million lines of code.
When evaluated using graph reconstruction, Flow2Vec achieved a high precision (96.9%) and recall
(95.5%) by successfully preserving interprocedural alias-aware value-flows. When evaluated using
the two client applications, Flow2Vec significantly boosted the performance of code2vec and
code2seq, two recent embedding approaches. For code classification, Flow2Vec outperformed
code2vec with an average increase of 21.2% in precision, 20.1% in recall, and 20.7% in F1. For code
summarization, Flow2Vec improved code2seq by an average of 13.2% in precision, 18.8% in recall,
and 16.0% in F1.
ACKNOWLEDGEMENTS
We would like to thank the anonymous reviewers for their helpful comments. This research is
supported by Australian Research Grants DP200101328.
REFERENCES
Mithun Acharya and Brian Robinson. 2011. Practical change impact analysis based on static program slicing for industrial
software systems. In ICSE’11. ACM, 746ś755.
Miltiadis Allamanis, Earl T Barr, Christian Bird, and Charles Sutton. 2015. Suggesting Accurate Method and Class Names. In
FSE‘15. 38ś49.
Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2018. Learning to represent programs with graphs.
(2018).
Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A convolutional attention network for extreme summarization of
source code. In ICML ’16 . 2091ś2100.
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2019a. code2seq: Generating sequences from structured representations
of code. ICLR ’19 (2019).
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2018. A General Path-Based Representation for Predicting Program
Properties. In PLDI ’18. 404ś419. https://doi.org/10.1145/3192366.3192412
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
Flow2Vec: Value-Flow-Based Precise Code Embedding 233:25
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019b. code2vec: Learning distributed representations of code.
ACM POPL 3 (2019), 40.
Lars Ole Andersen. 1994. Program analysis and specialization for the C programming language . Ph.D. Dissertation. University
of Cophenhagen.
George Balatsouras and Yannis Smaragdakis. 2016. Structure-sensitive points-to analysis for C and C++ . In SAS ’16. Springer,
84ś104.
Mohamad Barbar, Yulei Sui, and Shiping Chen. 2020. Flow-Sensitive Type-Based Heap Cloning. In ECOOP ’20 .
Mikhail Belkin and Partha Niyogi. 2002. Laplacian eigenmaps and spectral techniques for embedding and clustering. In
NeurIPS ’02 . 585ś591.
Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. 2018. Neural code comprehension: a learnable representation
of code semantics. In NeurIPS ’18. 3585ś3597.
Rastislav Bodík and Sadun Anik. 1998. Path-sensitive value-flow analysis. In POPL ’98. 237ś251.
Hongyun Cai, Vincent W Zheng, and Kevin Chen-Chuan Chang. 2018. A comprehensive survey of graph embedding:
Problems, techniques, and applications. TKDE ’18 30, 9 (2018), 1616ś1637.
Gerardo Canfora and Luigi Cerulo. 2005. Impact analysis by mining software and change request repositories. In METRICS
’05. IEEE, 9Ð-pp.
Xinyun Chen, Chang Liu, and Dawn Song. 2018. Tree-to-tree neural networks for program translation. In NeurIPS ’18 .
2547ś2557.
Jong-Deok Choi, Ron Cytron, and Jeanne Ferrante. 1991. Automatic construction of sparse data flow evaluation graphs. In
POPL ’91. 55ś66.
Fred Chow, Sun Chan, Shin-Ming Liu, Raymond Lo, and Mark Streich. 1996. Effective representation of aliases and indirect
memory operations in SSA form. In CC ’96. Springer, 253ś267.
Peng Cui, Xiao Wang, Jian Pei, and Wenwu Zhu. 2018. A survey on network embedding. TKDE 31, 5 (2018), 833ś852.
Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. 2000. A multilinear singular value decomposition. SIAM journal
on Matrix Analysis and Applications 21, 4 (2000), 1253ś1278.
Jeanne Ferrante, Karl J Ottenstein, and Joe D Warren. 1987. The program dependence graph and its use in optimization.
ACM TOPLAS 9, 3 (1987), 319ś349.
Georgia Frantzeskou, Stephen MacDonell, Efstathios Stamatatos, and Stefanos Gritzalis. 2008. Examining the significance
of high-level programming features in source code author classification. Journal of Systems and Software 81, 3 (2008),
447ś460.
Keith Brian Gallagher and James R Lyle. 1991. Using program slicing in software maintenance. IEEE TSE 17, 8 (1991),
751ś761.
Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable feature learning for networks. In KDD ’16 . ACM, 855ś864.
Ben Hardekopf and Calvin Lin. 2007. The ant and the grasshopper: fast and accurate pointer analysis for millions of lines of
code. In PLDI ’07. ACM, 290ś299.
Ben Hardekopf and Calvin Lin. 2011. Flow-sensitive pointer analysis for millions of lines of code. In CGO ’11 . 289ś298.
Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. 2012. On the naturalness of software. In
ICSE ’12. IEEE, 837ś847.
M E Hochstenbach. 2009. A JacobiśDavidson type method for the generalized singular value problem. Linear Algebra Appl.
431, 3-4 (2009), 471ś487.
Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018. Deep code comment generation. In ICPC ’18 . 200ś210.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing Source Code using a Neural
Attention Model. In ACL ’16. Berlin, Germany, 2073ś2083. https://www.aclweb.org/anthology/P16-1195
Toshihiro Kamiya, Shinji Kusumoto, and Katsuro Inoue. 2002. CCFinder: a multilinguistic token-based code clone detection
system for large scale source code. IEEE TSE 28, 7 (2002), 654ś670.
Leo Katz. 1953. A new status index derived from sociometric analysis. Psychometrika 18, 1 (1953), 39ś43.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR ’15, Yoshua Bengio and
Yann LeCun (Eds.).
John Kodumal and Alex Aiken. 2004. The set constraint/CFL reachability connection in practice. PLDI ’04 6 (2004), 207ś218.
Vladimir Kovalenko, Egor Bogomolov, Timofey Bryksin, and Alberto Bacchelli. 2019. PathMiner: a library for mining of
path-based representations of code. In MSR ’19 . 13ś17.
David J Kuck, Robert H Kuhn, David A Padua, Bruce Leasure, and Michael Wolfe. 1981. Dependence graphs and compiler
optimizations. In POPL ’81. ACM, 207ś218.
Chris Lattner and Vikram Adve. 2004. LLVM: A compilation framework for lifelong program analysis & transformation. In
CGO ’04 . IEEE, 75ś86.
Yuxiang Lei and Yulei Sui. 2019. Fast and precise handling of positive weight cycles for field-sensitive pointer analysis. In
SAS ’19 . Springer, 27ś47.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
233:26 Yulei Sui, Xiao Cheng, Guanqin Zhang, and Haoyu Wang
Ondrej Lhoták and Kwok-Chiang Andrew Chung. 2011. Points-To Analysis with Efficient Strong Updates. In POPL ’11 .
3ś16.
L Li, C Cifuentes, and N Keynes. 2011. Boosting the performance of flow-sensitive points-to analysis using value flow. In
FSE ’11 . 343ś353.
Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng, and Yuyi Zhong. 2018. VulDeePecker: A
deep learning-based system for vulnerability detection. NDSS ’18 (2018).
Defu Lian, Kai Zheng, Vincent W Zheng, Yong Ge, Longbing Cao, Ivor W Tsang, and Xing Xie. 2018. High-order proximity
preserving information network hashing. In KDD ’18 . ACM, 1744ś1753.
V Benjamin Livshits and Monica S Lam. 2003. Tracking pointers with path and context sensitivity for bug detection in C
programs. FSE ’03 28, 5 (2003), 317ś326.
M T Luong, H Pham, and C D Manning. 2015. Effective approaches to attention-based neural machine translation. arXiv
preprint arXiv:1508.04025 (2015).
Chris Maddison and Daniel Tarlow. 2014. Structured generative models of natural source code. In ICML ’14. 649ś657.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and
phrases and their compositionality. In NeurIPS ’13 . 3111ś3119.
Mingdong Ou, Peng Cui, Jian Pei, Ziwei Zhang, and Wenwu Zhu. 2016. Asymmetric transitivity preserving graph embedding.
In KDD ’16 . ACM, 1105ś1114.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. Deepwalk: Online learning of social representations. In KDD ’14.
ACM, 701ś710.
Michael Pradel and Koushik Sen. 2018. DeepBugs: A learning approach to name-based bug detection. Proceedings of the
ACM on Programming Languages 2, OOPSLA (2018), 1ś25.
Veselin Raychev, Martin Vechev, and Eran Yahav. 2014. Code completion with statistical language models. In PLDI ’14. ACM,
419ś428.
Thomas Reps. 1998. Program analysis via graph reachability. IST 40, 11-12 (1998), 701ś726.
Juergen Rilling and Tuomas Klemola. 2003. Identifying comprehension bottlenecks using program slicing and cognitive
complexity metrics. In IEEE International Workshop on Program Comprehension . IEEE, 115ś124.
Hitesh Sajnani, Vaibhav Saini, Jeffrey Svajlenko, Chanchal K Roy, and Cristina V Lopes. 2016. SourcererCC: Scaling code
clone detection to big-code. In ICSE ’16. IEEE, 1157ś1168.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008. Collective classifica-
tion in network data. AI magazine 29, 3 (2008), 93.
Qingkai Shi, Xiao Xiao, Rongxin Wu, Jinguo Zhou, Gang Fan, and Charles Zhang. 2018. Pinpoint: Fast and precise sparse
value flow analysis for million lines of code. In PLDI ’18. ACM, 693ś706.
Yao Shi, Soyeon Park, Zuoning Yin, Shan Lu, Yuanyuan Zhou, Wenguang Chen, and Weimin Zheng. 2010. Do I use the
wrong definition?: DeFuse: definition-use invariants for detecting concurrency and sequential bugs. OOPSLA ’10 45, 10
(2010), 160ś174.
Han Hee Song, Tae Won Cho, Vacha Dave, Yin Zhang, and Lili Qiu. 2009. Scalable proximity estimation and link prediction
in online social networks. In ACM SIGCOMM. ACM, 322ś335.
Manu Sridharan and Rastislav Bodík. 2006. Refinement-based context-sensitive points-to analysis for Java. PLDI 41, 6 (2006),
387ś400.
Yulei Sui and Jingling Xue. 2016. SVF: interprocedural static value-flow analysis in LLVM. In CC ’16. ACM, 265ś266.
Jiankai Sun, Bortik Bandyopadhyay, Armin Bashizade, Jiongqian Liang, P Sadayappan, and Srinivasan Parthasarathy. 2019.
Atp: Directed graph embedding with asymmetric transitivity preservation. In AAAI ’19 , Vol. 33. 265ś272.
Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved Semantic Representations From Tree-Structured
Long Short-Term Memory Networks. In IJCNLP. Beijing, China, 1556ś1566.
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. 2015. Line: Large-scale information network
embedding. In WWW ’15 . 1067ś1077.
Secil Ugurel, Robert Krovetz, and C Lee Giles. 2002. What’s the code?: automatic classification of source code archives. In
KDD ’02. ACM, 632ś638.
Xiao Wang, Peng Cui, Jing Wang, Jian Pei, Wenwu Zhu, and Shiqiang Yang. 2017. Community preserving network embedding.
In AAAI ’17 .
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes.
In AAAI ’14 .
Mark Weiser. 1981. Program slicing. In ICSE ’81. IEEE Press, 439ś449.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.
2015. Show, attend and tell: Neural image caption generation with visual attention. In ICML ’15 . 2048ś2057.
Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. 2019. A novel neural source code
representation based on abstract syntax tree. In ICSE ’19 . 783ś794.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.
Flow2Vec: Value-Flow-Based Precise Code Embedding 233:27
Ziwei Zhang, Peng Cui, Xiao Wang, Jian Pei, Xuanrong Yao, and Wenwu Zhu. 2018. Arbitrary-order proximity preserved
network embedding. In KDD ’18 . ACM, 2778ś2786.
Gang Zhao and Jeff Huang. 2018. Deepsim: deep learning code functional similarity. In FSE ’18 . ACM, 141ś151.
Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective vulnerability identification by
learning comprehensive program semantics via graph neural networks. In NeurIPS ’19 . 10197ś10207.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 233. Publication date: November 2020.

Learning Phrase Representations using RNN Encoder–Decoder
for Statistical Machine Translation
Kyunghyun Cho
Bart van Merri¨enboer Caglar Gulcehre
Universit´e de Montr´eal
firstname.lastname@umontreal.ca
Dzmitry Bahdanau
Jacobs University, Germany
d.bahdanau@jacobs-university.de
Fethi Bougares Holger Schwenk
Universit´e du Maine, France
firstname.lastname@lium.univ-lemans.fr
Yoshua Bengio
Universit´e de Montr´eal, CIFAR Senior Fellow
find.me@on.the.web
Abstract
In this paper, we propose a novel neu-
ral network model called RNN Encoder–
Decoder that consists of two recurrent
neural networks (RNN). One RNN en-
codes a sequence of symbols into a ﬁxed-
length vector representation, and the other
decodes the representation into another se-
quence of symbols. The encoder and de-
coder of the proposed model are jointly
trained to maximize the conditional prob-
ability of a target sequence given a source
sequence. The performance of a statisti-
cal machine translation system is empiri-
cally found to improve by using the con-
ditional probabilities of phrase pairs com-
puted by the RNN Encoder–Decoder as an
additional feature in the existing log-linear
model. Qualitatively, we show that the
proposed model learns a semantically and
syntactically meaningful representation of
linguistic phrases.
1 Introduction
Deep neural networks have shown great success in
various applications such as objection recognition
(see, e.g., (Krizhevsky et al., 2012)) and speech
recognition (see, e.g., (Dahl et al., 2012)). Fur-
thermore, many recent works showed that neu-
ral networks can be successfully used in a num-
ber of tasks in natural language processing (NLP).
These include, but are not limited to, language
modeling (Bengio et al., 2003), paraphrase detec-
tion (Socher et al., 2011) and word embedding ex-
traction (Mikolov et al., 2013). In the ﬁeld of sta-
tistical machine translation (SMT), deep neural
networks have begun to show promising results.
(Schwenk, 2012) summarizes a successful usage
of feedforward neural networks in the framework
of phrase-based SMT system.
Along this line of research on using neural net-
works for SMT, this paper focuses on a novel neu-
ral network architecture that can be used as a part
of the conventional phrase-based SMT system.
The proposed neural network architecture, which
we will refer to as anRNN Encoder–Decoder, con-
sists of two recurrent neural networks (RNN) that
act as an encoder and a decoder pair. The en-
coder maps a variable-length source sequence to a
ﬁxed-length vector, and the decoder maps the vec-
tor representation back to a variable-length target
sequence. The two networks are trained jointly to
maximize the conditional probability of the target
sequence given a source sequence. Additionally,
we propose to use a rather sophisticated hidden
unit in order to improve both the memory capacity
and the ease of training.
The proposed RNN Encoder–Decoder with a
novel hidden unit is empirically evaluated on the
task of translating from English to French. We
train the model to learn the translation probabil-
ity of an English phrase to a corresponding French
phrase. The model is then used as a part of a stan-
dard phrase-based SMT system by scoring each
phrase pair in the phrase table. The empirical eval-
uation reveals that this approach of scoring phrase
pairs with an RNN Encoder–Decoder improves
the translation performance.
We qualitatively analyze the trained RNN
Encoder–Decoder by comparing its phrase scores
with those given by the existing translation model.
The qualitative analysis shows that the RNN
Encoder–Decoder is better at capturing the lin-
guistic regularities in the phrase table, indirectly
explaining the quantitative improvements in the
overall translation performance. The further anal-
ysis of the model reveals that the RNN Encoder–
Decoder learns a continuous space representation
of a phrase that preserves both the semantic and
syntactic structure of the phrase.
arXiv:1406.1078v3  [cs.CL]  3 Sep 2014
2 RNN Encoder–Decoder
2.1 Preliminary: Recurrent Neural Networks
A recurrent neural network (RNN) is a neural net-
work that consists of a hidden state h and an
optional output y which operates on a variable-
length sequence x = (x1,...,x T ). At each time
step t, the hidden state h⟨t⟩of the RNN is updated
by
h⟨t⟩= f
(
h⟨t−1⟩,xt
)
, (1)
where f is a non-linear activation func-
tion. f may be as simple as an element-
wise logistic sigmoid function and as com-
plex as a long short-term memory (LSTM)
unit (Hochreiter and Schmidhuber, 1997).
An RNN can learn a probability distribution
over a sequence by being trained to predict the
next symbol in a sequence. In that case, the output
at each timestep t is the conditional distribution
p(xt |xt−1,...,x 1). For example, a multinomial
distribution (1-of-Kcoding) can be output using a
softmax activation function
p(xt,j = 1|xt−1,...,x 1) = exp
(
wjh⟨t⟩
)
∑K
j′=1 exp
(
wj′h⟨t⟩
),
(2)
for all possible symbols j = 1,...,K , where wj
are the rows of a weight matrixW. By combining
these probabilities, we can compute the probabil-
ity of the sequence x using
p(x) =
T∏
t=1
p(xt |xt−1,...,x 1). (3)
From this learned distribution, it is straightfor-
ward to sample a new sequence by iteratively sam-
pling a symbol at each time step.
2.2 RNN Encoder–Decoder
In this paper, we propose a novel neural network
architecture that learns to encode a variable-length
sequence into a ﬁxed-length vector representation
and to decode a given ﬁxed-length vector rep-
resentation back into a variable-length sequence.
From a probabilistic perspective, this new model
is a general method to learn the conditional dis-
tribution over a variable-length sequence condi-
tioned on yet another variable-length sequence,
e.g. p(y1,...,y T′ | x1,...,x T ), where one
x1 x2 xT
yT' y2 y1
c
Decoder
Encoder
Figure 1: An illustration of the proposed RNN
Encoder–Decoder.
should note that the input and output sequence
lengths T and T′may differ.
The encoder is an RNN that reads each symbol
of an input sequence x sequentially. As it reads
each symbol, the hidden state of the RNN changes
according to Eq. (1). After reading the end of
the sequence (marked by an end-of-sequence sym-
bol), the hidden state of the RNN is a summary c
of the whole input sequence.
The decoder of the proposed model is another
RNN which is trained to generate the output se-
quence by predicting the next symbol yt given the
hidden state h⟨t⟩. However, unlike the RNN de-
scribed in Sec. 2.1, both yt and h⟨t⟩are also con-
ditioned on yt−1 and on the summaryc of the input
sequence. Hence, the hidden state of the decoder
at time tis computed by,
h⟨t⟩= f
(
h⟨t−1⟩,yt−1,c
)
,
and similarly, the conditional distribution of the
next symbol is
P(yt|yt−1,yt−2,...,y 1,c) =g
(
h⟨t⟩,yt−1,c
)
.
for given activation functions f and g (the latter
must produce valid probabilities, e.g. with a soft-
max).
See Fig. 1 for a graphical depiction of the pro-
posed model architecture.
The two components of the proposed RNN
Encoder–Decoder are jointly trained to maximize
the conditional log-likelihood
max
θ
1
N
N∑
n=1
log pθ(yn |xn), (4)
where θ is the set of the model parameters and
each (xn,yn) is an (input sequence, output se-
quence) pair from the training set. In our case,
as the output of the decoder, starting from the in-
put, is differentiable, we can use a gradient-based
algorithm to estimate the model parameters.
Once the RNN Encoder–Decoder is trained, the
model can be used in two ways. One way is to use
the model to generate a target sequence given an
input sequence. On the other hand, the model can
be used to score a given pair of input and output
sequences, where the score is simply a probability
pθ(y |x) from Eqs. (3) and (4).
2.3 Hidden Unit that Adaptively Remembers
and Forgets
In addition to a novel model architecture, we also
propose a new type of hidden unit ( f in Eq. (1))
that has been motivated by the LSTM unit but is
much simpler to compute and implement. 1 Fig. 2
shows the graphical depiction of the proposed hid-
den unit.
Let us describe how the activation of the j-th
hidden unit is computed. First, the reset gate rj is
computed by
rj = σ
(
[Wrx]j +
[
Urh⟨t−1⟩
]
j
)
, (5)
where σ is the logistic sigmoid function, and [.]j
denotes the j-th element of a vector. x and ht−1
are the input and the previous hidden state, respec-
tively. Wr and Ur are weight matrices which are
learned.
Similarly, the update gate zj is computed by
zj = σ
(
[Wzx]j +
[
Uzh⟨t−1⟩
]
j
)
. (6)
The actual activation of the proposed unit hj is
then computed by
h⟨t⟩
j = zjh⟨t−1⟩
j + (1−zj)˜h⟨t⟩
j , (7)
where
˜h⟨t⟩
j = φ
(
[Wx]j +
[
U
(
r ⊙h⟨t−1⟩
)]
j
)
. (8)
In this formulation, when the reset gate is close
to 0, the hidden state is forced to ignore the pre-
vious hidden state and reset with the current input
1 The LSTM unit, which has shown impressive results in
several applications such as speech recognition, has a mem-
ory cell and four gating units that adaptively control the in-
formation ﬂow inside the unit, compared to only two gating
units in the proposed hidden unit. For details on LSTM net-
works, see, e.g., (Graves, 2012).
z
rh h~ x
Figure 2: An illustration of the proposed hidden
activation function. The update gate z selects
whether the hidden state is to be updated with
a new hidden state ˜h. The reset gate r decides
whether the previous hidden state is ignored. See
Eqs. (5)–(8) for the detailed equations of r, z, h
and ˜h.
only. This effectively allows the hidden state to
drop any information that is found to be irrelevant
later in the future, thus, allowing a more compact
representation.
On the other hand, the update gate controls how
much information from the previous hidden state
will carry over to the current hidden state. This
acts similarly to the memory cell in the LSTM
network and helps the RNN to remember long-
term information. Furthermore, this may be con-
sidered an adaptive variant of a leaky-integration
unit (Bengio et al., 2013).
As each hidden unit has separate reset and up-
date gates, each hidden unit will learn to capture
dependencies over different time scales. Those
units that learn to capture short-term dependencies
will tend to have reset gates that are frequently ac-
tive, but those that capture longer-term dependen-
cies will have update gates that are mostly active.
In our preliminary experiments, we found that
it is crucial to use this new unit with gating units.
We were not able to get meaningful result with an
oft-used tanh unit without any gating.
3 Statistical Machine Translation
In a commonly used statistical machine translation
system (SMT), the goal of the system (decoder,
speciﬁcally) is to ﬁnd a translationf given a source
sentence e, which maximizes
p(f |e) ∝p(e |f)p(f),
where the ﬁrst term at the right hand side is called
translation model and the latter language model
(see, e.g., (Koehn, 2005)). In practice, however,
most SMT systems model log p(f |e) as a log-
linear model with additional features and corre-
sponding weights:
log p(f |e) =
N∑
n=1
wnfn(f,e) + logZ(e), (9)
where fn and wn are the n-th feature and weight,
respectively. Z(e) is a normalization constant that
does not depend on the weights. The weights are
often optimized to maximize the BLEU score on a
development set.
In the phrase-based SMT framework
introduced in (Koehn et al., 2003) and
(Marcu and Wong, 2002), the translation model
log p(e |f) is factorized into the translation
probabilities of matching phrases in the source
and target sentences. 2 These probabilities are
once again considered additional features in the
log-linear model (see Eq. (9)) and are weighted
accordingly to maximize the BLEU score.
Since the neural net language model was pro-
posed in (Bengio et al., 2003), neural networks
have been used widely in SMT systems. In
many cases, neural networks have been used to
rescore translation hypotheses ( n-best lists) (see,
e.g., (Schwenk et al., 2006)). Recently, however,
there has been interest in training neural networks
to score the translated sentence (or phrase pairs)
using a representation of the source sentence as
an additional input. See, e.g., (Schwenk, 2012),
(Son et al., 2012) and (Zou et al., 2013).
3.1 Scoring Phrase Pairs with RNN
Encoder–Decoder
Here we propose to train the RNN Encoder–
Decoder (see Sec. 2.2) on a table of phrase pairs
and use its scores as additional features in the log-
linear model in Eq. (9) when tuning the SMT de-
coder.
When we train the RNN Encoder–Decoder, we
ignore the (normalized) frequencies of each phrase
pair in the original corpora. This measure was
taken in order (1) to reduce the computational ex-
pense of randomly selecting phrase pairs from a
large phrase table according to the normalized fre-
quencies and (2) to ensure that the RNN Encoder–
Decoder does not simply learn to rank the phrase
pairs according to their numbers of occurrences.
One underlying reason for this choice was that the
existing translation probability in the phrase ta-
ble already reﬂects the frequencies of the phrase
2 Without loss of generality, from here on, we refer to
p(e |f ) for each phrase pair as a translation model as well
pairs in the original corpus. With a ﬁxed capacity
of the RNN Encoder–Decoder, we try to ensure
that most of the capacity of the model is focused
toward learning linguistic regularities, i.e., distin-
guishing between plausible and implausible trans-
lations, or learning the “manifold” (region of prob-
ability concentration) of plausible translations.
Once the RNN Encoder–Decoder is trained, we
add a new score for each phrase pair to the exist-
ing phrase table. This allows the new scores to en-
ter into the existing tuning algorithm with minimal
additional overhead in computation.
As Schwenk pointed out in (Schwenk, 2012),
it is possible to completely replace the existing
phrase table with the proposed RNN Encoder–
Decoder. In that case, for a given source phrase,
the RNN Encoder–Decoder will need to generate
a list of (good) target phrases. This requires, how-
ever, an expensive sampling procedure to be per-
formed repeatedly. In this paper, thus, we only
consider rescoring the phrase pairs in the phrase
table.
3.2 Related Approaches: Neural Networks in
Machine Translation
Before presenting the empirical results, we discuss
a number of recent works that have proposed to
use neural networks in the context of SMT.
Schwenk in (Schwenk, 2012) proposed a simi-
lar approach of scoring phrase pairs. Instead of the
RNN-based neural network, he used a feedforward
neural network that has ﬁxed-size inputs (7 words
in his case, with zero-padding for shorter phrases)
and ﬁxed-size outputs (7 words in the target lan-
guage). When it is used speciﬁcally for scoring
phrases for the SMT system, the maximum phrase
length is often chosen to be small. However, as the
length of phrases increases or as we apply neural
networks to other variable-length sequence data,
it is important that the neural network can han-
dle variable-length input and output. The pro-
posed RNN Encoder–Decoder is well-suited for
these applications.
Similar to (Schwenk, 2012), Devlin et al.
(Devlin et al., 2014) proposed to use a feedfor-
ward neural network to model a translation model,
however, by predicting one word in a target phrase
at a time. They reported an impressive improve-
ment, but their approach still requires the maxi-
mum length of the input phrase (or context words)
to be ﬁxed a priori.
Although it is not exactly a neural network they
train, the authors of (Zou et al., 2013) proposed
to learn a bilingual embedding of words/phrases.
They use the learned embedding to compute the
distance between a pair of phrases which is used
as an additional score of the phrase pair in an SMT
system.
In (Chandar et al., 2014), a feedforward neural
network was trained to learn a mapping from a
bag-of-words representation of an input phrase to
an output phrase. This is closely related to both the
proposed RNN Encoder–Decoder and the model
proposed in (Schwenk, 2012), except that their in-
put representation of a phrase is a bag-of-words.
A similar approach of using bag-of-words repre-
sentations was proposed in (Gao et al., 2013) as
well. Earlier, a similar encoder–decoder model us-
ing two recursive neural networks was proposed
in (Socher et al., 2011), but their model was re-
stricted to a monolingual setting, i.e. the model
reconstructs an input sentence. More recently, an-
other encoder–decoder model using an RNN was
proposed in (Auli et al., 2013), where the de-
coder is conditioned on a representation of either
a source sentence or a source context.
One important difference between the pro-
posed RNN Encoder–Decoder and the approaches
in (Zou et al., 2013) and (Chandar et al., 2014) is
that the order of the words in source and tar-
get phrases is taken into account. The RNN
Encoder–Decoder naturally distinguishes between
sequences that have the same words but in a differ-
ent order, whereas the aforementioned approaches
effectively ignore order information.
The closest approach related to the proposed
RNN Encoder–Decoder is the Recurrent Contin-
uous Translation Model (Model 2) proposed in
(Kalchbrenner and Blunsom, 2013). In their pa-
per, they proposed a similar model that consists
of an encoder and decoder. The difference with
our model is that they used a convolutionaln-gram
model (CGM) for the encoder and the hybrid of
an inverse CGM and a recurrent neural network
for the decoder. They, however, evaluated their
model on rescoring the n-best list proposed by the
conventional SMT system and computing the per-
plexity of the gold standard translations.
4 Experiments
We evaluate our approach on the English/French
translation task of the WMT’14 workshop.
4.1 Data and Baseline System
Large amounts of resources are available to build
an English/French SMT system in the framework
of the WMT’14 translation task. The bilingual
corpora include Europarl (61M words), news com-
mentary (5.5M), UN (421M), and two crawled
corpora of 90M and 780M words respectively.
The last two corpora are quite noisy. To train
the French language model, about 712M words of
crawled newspaper material is available in addi-
tion to the target side of the bitexts. All the word
counts refer to French words after tokenization.
It is commonly acknowledged that training sta-
tistical models on the concatenation of all this
data does not necessarily lead to optimal per-
formance, and results in extremely large mod-
els which are difﬁcult to handle. Instead, one
should focus on the most relevant subset of the
data for a given task. We have done so by
applying the data selection method proposed in
(Moore and Lewis, 2010), and its extension to bi-
texts (Axelrod et al., 2011). By these means we
selected a subset of 418M words out of more
than 2G words for language modeling and a
subset of 348M out of 850M words for train-
ing the RNN Encoder–Decoder. We used the
test set newstest2012 and 2013 for data
selection and weight tuning with MERT, and
newstest2014 as our test set. Each set has
more than 70 thousand words and a single refer-
ence translation.
For training the neural networks, including the
proposed RNN Encoder–Decoder, we limited the
source and target vocabulary to the most frequent
15,000 words for both English and French. This
covers approximately 93% of the dataset. All the
out-of-vocabulary words were mapped to a special
token ([UNK]).
The baseline phrase-based SMT system was
built using Moses with default settings. This sys-
tem achieves a BLEU score of 30.64 and 33.3 on
the development and test sets, respectively (see Ta-
ble 1).
4.1.1 RNN Encoder–Decoder
The RNN Encoder–Decoder used in the experi-
ment had 1000 hidden units with the proposed
gates at the encoder and at the decoder. The in-
put matrix between each input symbolx⟨t⟩and the
hidden unit is approximated with two lower-rank
matrices, and the output matrix is approximated
Models BLEU
dev test
Baseline 30.64 33.30
RNN 31.20 33.87
CSLM + RNN 31.48 34.64
CSLM + RNN + WP 31.50 34.54
Table 1: BLEU scores computed on the develop-
ment and test sets using different combinations of
approaches. WP denotes a word penalty, where
we penalizes the number of unknown words to
neural networks.
similarly. We used rank-100 matrices, equivalent
to learning an embedding of dimension 100 for
each word. The activation function used for ˜hin
Eq. (8) is a hyperbolic tangent function. The com-
putation from the hidden state in the decoder to
the output is implemented as a deep neural net-
work (Pascanu et al., 2014) with a single interme-
diate layer having 500 maxout units each pooling
2 inputs (Goodfellow et al., 2013).
All the weight parameters in the RNN Encoder–
Decoder were initialized by sampling from an
isotropic zero-mean (white) Gaussian distribution
with its standard deviation ﬁxed to 0.01, except
for the recurrent weight parameters. For the re-
current weight matrices, we ﬁrst sampled from a
white Gaussian distribution and used its left singu-
lar vectors matrix, following (Saxe et al., 2014).
We used Adadelta and stochastic gradient
descent to train the RNN Encoder–Decoder
with hyperparameters ϵ = 10 −6 and ρ =
0.95 (Zeiler, 2012). At each update, we used 64
randomly selected phrase pairs from a phrase ta-
ble (which was created from 348M words). The
model was trained for approximately three days.
Details of the architecture used in the experi-
ments are explained in more depth in the supple-
mentary material.
4.1.2 Neural Language Model
In order to assess the effectiveness of scoring
phrase pairs with the proposed RNN Encoder–
Decoder, we also tried a more traditional approach
of using a neural network for learning a target
language model (CSLM) (Schwenk, 2007). Espe-
cially, the comparison between the SMT system
using CSLM and that using the proposed approach
of phrase scoring by RNN Encoder–Decoder will
clarify whether the contributions from multiple
neural networks in different parts of the SMT sys-
tem add up or are redundant.
We trained the CSLM model on 7-grams
from the target corpus. Each input word
was projected into the embedding space R512,
and they were concatenated to form a 3072-
dimensional vector. The concatenated vector was
fed through two rectiﬁed layers (of size 1536 and
1024) (Glorot et al., 2011). The output layer was
a simple softmax layer (see Eq. (2)). All the
weight parameters were initialized uniformly be-
tween −0.01 and 0.01, and the model was trained
until the validation perplexity did not improve for
10 epochs. After training, the language model
achieved a perplexity of 45.80. The validation set
was a random selection of 0.1% of the corpus. The
model was used to score partial translations dur-
ing the decoding process, which generally leads to
higher gains in BLEU score than n-best list rescor-
ing (Vaswani et al., 2013).
To address the computational complexity of
using a CSLM in the decoder a buffer was
used to aggregate n-grams during the stack-
search performed by the decoder. Only when
the buffer is full, or a stack is about to
be pruned, the n-grams are scored by the
CSLM. This allows us to perform fast matrix-
matrix multiplication on GPU using Theano
(Bergstra et al., 2010; Bastien et al., 2012).
−60 −50 −40 −30 −20 −10 0
−14
−12
−10
−8
−6
−4
−2
0
RNN Scores (log)
TM Scores (log)
Figure 3: The visualization of phrase pairs accord-
ing to their scores (log-probabilities) by the RNN
Encoder–Decoder and the translation model.
4.2 Quantitative Analysis
We tried the following combinations:
1. Baseline conﬁguration
2. Baseline + RNN
3. Baseline + CSLM + RNN
4. Baseline + CSLM + RNN + Word penalty
Source Translation Model RNN Encoder–Decoder
at the end of the [a la ﬁn de la] [ ´r la ﬁn des ann ´ees] [ ˆetre sup-
prim´es `a la ﬁn de la]
[`a la ﬁn du] [`a la ﬁn des] [`a la ﬁn de la]
for the ﬁrst time [r c⃝pour la premi r¨ere fois] [ ´et´e donn ´es pour
la premi `ere fois] [ ´et´e comm ´emor´ee pour la
premi`ere fois]
[pour la premi `ere fois] [pour la premi `ere fois ,]
[pour la premi`ere fois que]
in the United States
and
[? aux ?tats-Unis et] [ ´et´e ouvertes aux ´Etats-
Unis et] [´et´e constat´ees aux ´Etats-Unis et]
[aux Etats-Unis et] [des Etats-Unis et] [des
´Etats-Unis et]
, as well as [?s , qu’] [?s , ainsi que] [?re aussi bien que] [, ainsi qu’] [, ainsi que] [, ainsi que les]
one of the most [?t ?l’ un des plus] [?l’ un des plus] [ˆetre retenue
comme un de ses plus]
[l’ un des] [le] [un des]
(a) Long, frequent source phrases
Source Translation Model RNN Encoder–Decoder
, Minister of Commu-
nications and Trans-
port
[Secr´etaire aux communications et aux trans-
ports :] [Secr ´etaire aux communications et aux
transports]
[Secr´etaire aux communications et aux trans-
ports] [Secr ´etaire aux communications et aux
transports :]
did not comply with
the
[vestimentaire , ne correspondaient pas `a des]
[susmentionn´ee n’ ´etait pas conforme aux]
[pr´esent´ees n’ ´etaient pas conformes `a la]
[n’ ont pas respect ´e les] [n’ ´etait pas conforme
aux] [n’ ont pas respect´e la]
parts of the world . [ c⃝gions du monde .] [r ´egions du monde con-
sid´er´ees .] [r´egion du monde consid´er´ee .]
[parties du monde .] [les parties du monde .]
[des parties du monde .]
the past few days . [le petit texte .] [cours des tout derniers jours .]
[les tout derniers jours .]
[ces derniers jours .] [les derniers jours .] [cours
des derniers jours .]
on Friday and Satur-
day
[vendredi et samedi `a la] [vendredi et samedi `a]
[se d´eroulera vendredi et samedi ,]
[le vendredi et le samedi] [le vendredi et samedi]
[vendredi et samedi]
(b) Long, rare source phrases
Table 2: The top scoring target phrases for a small set of source phrases according to the translation
model (direct translation probability) and by the RNN Encoder–Decoder. Source phrases were randomly
selected from phrases with 4 or more words. ? denotes an incomplete (partial) character. r is a Cyrillic
letter ghe.
The results are presented in Table 1. As ex-
pected, adding features computed by neural net-
works consistently improves the performance over
the baseline performance.
The best performance was achieved when we
used both CSLM and the phrase scores from the
RNN Encoder–Decoder. This suggests that the
contributions of the CSLM and the RNN Encoder–
Decoder are not too correlated and that one can
expect better results by improving each method in-
dependently. Furthermore, we tried penalizing the
number of words that are unknown to the neural
networks (i.e. words which are not in the short-
list). We do so by simply adding the number of
unknown words as an additional feature the log-
linear model in Eq. (9).3 However, in this case we
3 To understand the effect of the penalty, consider the set
of all words in the 15,000 large shortlist, SL. All words xi /∈
SL are replaced by a special token[UNK] before being scored
by the neural networks. Hence, the conditional probability of
any xi
t /∈SL is actually given by the model as
p (xt = [UNK] |x<t) =p (xt /∈SL |x<t)
=
∑
xj
t /∈SL
p
(
xj
t |x<t
)
≥p
(
xi
t |x<t
)
,
where x<t is a shorthand notation for xt−1, . . . , x1.
were not able to achieve better performance on the
test set, but only on the development set.
4.3 Qualitative Analysis
In order to understand where the performance im-
provement comes from, we analyze the phrase pair
scores computed by the RNN Encoder–Decoder
against the corresponding p(f |e) from the trans-
lation model. Since the existing translation model
relies solely on the statistics of the phrase pairs in
the corpus, we expect its scores to be better esti-
mated for the frequent phrases but badly estimated
for rare phrases. Also, as we mentioned earlier
in Sec. 3.1, we further expect the RNN Encoder–
Decoder which was trained without any frequency
information to score the phrase pairs based rather
on the linguistic regularities than on the statistics
of their occurrences in the corpus.
We focus on those pairs whose source phrase is
long (more than 3 words per source phrase) and
As a result, the probability of words not in the shortlist is
always overestimated. It is possible to address this issue by
backing off to an existing model that contain non-shortlisted
words (see (Schwenk, 2007)) In this paper, however, we opt
for introducing a word penalty instead, which counteracts the
word probability overestimation.
Source Samples from RNN Encoder–Decoder
at the end of the [`a la ﬁn de la] (×11)
for the ﬁrst time [pour la premi`ere fois] (×24) [pour la premi`ere fois que] (×2)
in the United States and [aux ´Etats-Unis et] (×6) [dans les ´Etats-Unis et] (×4)
, as well as [, ainsi que] [,] [ainsi que] [, ainsi qu’] [et UNK]
one of the most [l’ un des plus] (×9) [l’ un des] (×5) [l’ une des plus] (×2)
(a) Long, frequent source phrases
Source Samples from RNN Encoder–Decoder
, Minister of Communica-
tions and Transport
[ , ministre des communications et le transport] (×13)
did not comply with the [n’ tait pas conforme aux] [n’ a pas respect l’] (×2) [n’ a pas respect la] (×3)
parts of the world . [arts du monde .] (×11) [des arts du monde .] (×7)
the past few days . [quelques jours .] (×5) [les derniers jours .] (×5) [ces derniers jours .] (×2)
on Friday and Saturday [vendredi et samedi] (×5) [le vendredi et samedi] (×7) [le vendredi et le samedi] (×4)
(b) Long, rare source phrases
Table 3: Samples generated from the RNN Encoder–Decoder for each source phrase used in Table 2. We
show the top-5 target phrases out of 50 samples. They are sorted by the RNN Encoder–Decoder scores.
Figure 4: 2–D embedding of the learned word representation. The left one shows the full embedding
space, while the right one shows a zoomed-in view of one region (color–coded). For more plots, see the
supplementary material.
frequent. For each such source phrase, we look
at the target phrases that have been scored high
either by the translation probability p(f | e) or
by the RNN Encoder–Decoder. Similarly, we per-
form the same procedure with those pairs whose
source phrase is long but rare in the corpus.
Table 2 lists the top-3 target phrases per source
phrase favored either by the translation model
or by the RNN Encoder–Decoder. The source
phrases were randomly chosen among long ones
having more than 4 or 5 words.
In most cases, the choices of the target phrases
by the RNN Encoder–Decoder are closer to ac-
tual or literal translations. We can observe that the
RNN Encoder–Decoder prefers shorter phrases in
general.
Interestingly, many phrase pairs were scored
similarly by both the translation model and the
RNN Encoder–Decoder, but there were as many
other phrase pairs that were scored radically dif-
ferent (see Fig. 3). This could arise from the
proposed approach of training the RNN Encoder–
Decoder on a set of unique phrase pairs, discour-
aging the RNN Encoder–Decoder from learning
simply the frequencies of the phrase pairs from the
corpus, as explained earlier.
Furthermore, in Table 3, we show for each of
the source phrases in Table 2, the generated sam-
ples from the RNN Encoder–Decoder. For each
source phrase, we generated 50 samples and show
the top-ﬁve phrases accordingly to their scores.
We can see that the RNN Encoder–Decoder is
able to propose well-formed target phrases with-
out looking at the actual phrase table. Importantly,
the generated phrases do not overlap completely
with the target phrases from the phrase table. This
encourages us to further investigate the possibility
of replacing the whole or a part of the phrase table
Figure 5: 2–D embedding of the learned phrase representation. The top left one shows the full represen-
tation space (5000 randomly selected points), while the other three ﬁgures show the zoomed-in view of
speciﬁc regions (color–coded).
with the proposed RNN Encoder–Decoder in the
future.
4.4 Word and Phrase Representations
Since the proposed RNN Encoder–Decoder is not
speciﬁcally designed only for the task of machine
translation, here we brieﬂy look at the properties
of the trained model.
It has been known for some time that
continuous space language models using
neural networks are able to learn seman-
tically meaningful embeddings (See, e.g.,
(Bengio et al., 2003; Mikolov et al., 2013)). Since
the proposed RNN Encoder–Decoder also projects
to and maps back from a sequence of words into
a continuous space vector, we expect to see a
similar property with the proposed model as well.
The left plot in Fig. 4 shows the 2–D embedding
of the words using the word embedding matrix
learned by the RNN Encoder–Decoder. The pro-
jection was done by the recently proposed Barnes-
Hut-SNE (van der Maaten, 2013). We can clearly
see that semantically similar words are clustered
with each other (see the zoomed-in plots in Fig. 4).
The proposed RNN Encoder–Decoder naturally
generates a continuous-space representation of a
phrase. The representation ( c in Fig. 1) in this
case is a 1000-dimensional vector. Similarly to the
word representations, we visualize the representa-
tions of the phrases that consists of four or more
words using the Barnes-Hut-SNE in Fig. 5.
From the visualization, it is clear that the RNN
Encoder–Decoder captures both semantic and syn-
tactic structures of the phrases. For instance, in
the bottom-left plot, most of the phrases are about
the duration of time, while those phrases that are
syntactically similar are clustered together. The
bottom-right plot shows the cluster of phrases that
are semantically similar (countries or regions). On
the other hand, the top-right plot shows the phrases
that are syntactically similar.
5 Conclusion
In this paper, we proposed a new neural network
architecture, called an RNN Encoder–Decoder
that is able to learn the mapping from a sequence
of an arbitrary length to another sequence, possi-
bly from a different set, of an arbitrary length. The
proposed RNN Encoder–Decoder is able to either
score a pair of sequences (in terms of a conditional
probability) or generate a target sequence given a
source sequence. Along with the new architecture,
we proposed a novel hidden unit that includes a re-
set gate and an update gate that adaptively control
how much each hidden unit remembers or forgets
while reading/generating a sequence.
We evaluated the proposed model with the task
of statistical machine translation, where we used
the RNN Encoder–Decoder to score each phrase
pair in the phrase table. Qualitatively, we were
able to show that the new model is able to cap-
ture linguistic regularities in the phrase pairs well
and also that the RNN Encoder–Decoder is able to
propose well-formed target phrases.
The scores by the RNN Encoder–Decoder were
found to improve the overall translation perfor-
mance in terms of BLEU scores. Also, we
found that the contribution by the RNN Encoder–
Decoder is rather orthogonal to the existing ap-
proach of using neural networks in the SMT sys-
tem, so that we can improve further the perfor-
mance by using, for instance, the RNN Encoder–
Decoder and the neural net language model to-
gether.
Our qualitative analysis of the trained model
shows that it indeed captures the linguistic regu-
larities in multiple levels i.e. at the word level as
well as phrase level. This suggests that there may
be more natural language related applications that
may beneﬁt from the proposed RNN Encoder–
Decoder.
The proposed architecture has large potential
for further improvement and analysis. One ap-
proach that was not investigated here is to re-
place the whole, or a part of the phrase table by
letting the RNN Encoder–Decoder propose target
phrases. Also, noting that the proposed model is
not limited to being used with written language,
it will be an important future research to apply the
proposed architecture to other applications such as
speech transcription.
Acknowledgments
KC, BM, CG, DB and YB would like to thank
NSERC, Calcul Qu ´ebec, Compute Canada, the
Canada Research Chairs and CIFAR. FB and HS
were partially funded by the European Commis-
sion under the project MateCat, and by DARPA
under the BOLT project.
References
[Auli et al.2013] Michael Auli, Michel Galley, Chris
Quirk, and Geoffrey Zweig. 2013. Joint language
and translation modeling with recurrent neural net-
works. In Proceedings of the ACL Conference on
Empirical Methods in Natural Language Processing
(EMNLP), pages 1044–1054.
[Axelrod et al.2011] Amittai Axelrod, Xiaodong He,
and Jianfeng Gao. 2011. Domain adaptation via
pseudo in-domain data selection. In Proceedings of
the ACL Conference on Empirical Methods in Natu-
ral Language Processing (EMNLP), pages 355–362.
[Bastien et al.2012] Fr ´ed´eric Bastien, Pascal Lamblin,
Razvan Pascanu, James Bergstra, Ian J. Goodfellow,
Arnaud Bergeron, Nicolas Bouchard, and Yoshua
Bengio. 2012. Theano: new features and speed im-
provements. Deep Learning and Unsupervised Fea-
ture Learning NIPS 2012 Workshop.
[Bengio et al.2003] Yoshua Bengio, R ´ejean Ducharme,
Pascal Vincent, and Christian Janvin. 2003. A neu-
ral probabilistic language model. J. Mach. Learn.
Res., 3:1137–1155, March.
[Bengio et al.2013] Y . Bengio, N. Boulanger-
Lewandowski, and R. Pascanu. 2013. Advances
in optimizing recurrent networks. In Proceedings
of the 38th International Conference on Acoustics,
Speech, and Signal Processing (ICASSP 2013) ,
May.
[Bergstra et al.2010] James Bergstra, Olivier Breuleux,
Fr´ed´eric Bastien, Pascal Lamblin, Razvan Pascanu,
Guillaume Desjardins, Joseph Turian, David Warde-
Farley, and Yoshua Bengio. 2010. Theano: a CPU
and GPU math expression compiler. In Proceedings
of the Python for Scientiﬁc Computing Conference
(SciPy), June. Oral Presentation.
[Chandar et al.2014] Sarath Chandar, Stanislas Lauly,
Hugo Larochelle, Mitesh Khapra, Balaraman Ravin-
dran, Vikas Raykar, and Amrita Saha. 2014. An au-
toencoder approach to learning bilingual word repre-
sentations. arXiv:1402.1454 [cs.CL], Febru-
ary.
[Dahl et al.2012] George E. Dahl, Dong Yu, Li Deng,
and Alex Acero. 2012. Context-dependent pre-
trained deep neural networks for large vocabulary
speech recognition. IEEE Transactions on Audio,
Speech, and Language Processing, 20(1):33–42.
[Devlin et al.2014] Jacob Devlin, Rabih Zbib,
Zhongqiang Huang, Thomas Lamar, Richard
Schwartz, , and John Makhoul. 2014. Fast and
robust neural network joint models for statistical
machine translation. In Proceedings of the ACL
2014 Conference, ACL ’14, pages 1370–1380.
[Gao et al.2013] Jianfeng Gao, Xiaodong He, Wen tau
Yih, and Li Deng. 2013. Learning semantic repre-
sentations for the phrase translation model. Techni-
cal report, Microsoft Research.
[Glorot et al.2011] X. Glorot, A. Bordes, and Y . Ben-
gio. 2011. Deep sparse rectiﬁer neural networks. In
AISTATS’2011.
[Goodfellow et al.2013] Ian J. Goodfellow, David
Warde-Farley, Mehdi Mirza, Aaron Courville, and
Yoshua Bengio. 2013. Maxout networks. In
ICML’2013.
[Graves2012] Alex Graves. 2012. Supervised Se-
quence Labelling with Recurrent Neural Networks .
Studies in Computational Intelligence. Springer.
[Hochreiter and Schmidhuber1997] S. Hochreiter and
J. Schmidhuber. 1997. Long short-term memory.
Neural Computation, 9(8):1735–1780.
[Kalchbrenner and Blunsom2013] Nal Kalchbrenner
and Phil Blunsom. 2013. Two recurrent continuous
translation models. In Proceedings of the ACL Con-
ference on Empirical Methods in Natural Language
Processing (EMNLP), pages 1700–1709.
[Koehn et al.2003] Philipp Koehn, Franz Josef Och,
and Daniel Marcu. 2003. Statistical phrase-based
translation. In Proceedings of the 2003 Conference
of the North American Chapter of the Association
for Computational Linguistics on Human Language
Technology - Volume 1, NAACL ’03, pages 48–54.
[Koehn2005] P. Koehn. 2005. Europarl: A parallel cor-
pus for statistical machine translation. In Machine
Translation Summit X , pages 79–86, Phuket, Thai-
land.
[Krizhevsky et al.2012] Alex Krizhevsky, Ilya
Sutskever, and Geoffrey Hinton. 2012. Ima-
geNet classiﬁcation with deep convolutional neural
networks. In Advances in Neural Information
Processing Systems 25 (NIPS’2012).
[Marcu and Wong2002] Daniel Marcu and William
Wong. 2002. A phrase-based, joint probability
model for statistical machine translation. In Pro-
ceedings of the ACL-02 Conference on Empirical
Methods in Natural Language Processing - Volume
10, EMNLP ’02, pages 133–139.
[Mikolov et al.2013] Tomas Mikolov, Ilya Sutskever,
Kai Chen, Greg Corrado, and Jeff Dean. 2013. Dis-
tributed representations of words and phrases and
their compositionality. In Advances in Neural Infor-
mation Processing Systems 26, pages 3111–3119.
[Moore and Lewis2010] Robert C. Moore and William
Lewis. 2010. Intelligent selection of language
model training data. In Proceedings of the ACL
2010 Conference Short Papers , ACLShort ’10,
pages 220–224, Stroudsburg, PA, USA.
[Pascanu et al.2014] R. Pascanu, C. Gulcehre, K. Cho,
and Y . Bengio. 2014. How to construct deep recur-
rent neural networks. In Proceedings of the Second
International Conference on Learning Representa-
tions (ICLR 2014), April.
[Saxe et al.2014] Andrew M. Saxe, James L. McClel-
land, and Surya Ganguli. 2014. Exact solutions
to the nonlinear dynamics of learning in deep lin-
ear neural networks. In Proceedings of the Second
International Conference on Learning Representa-
tions (ICLR 2014), April.
[Schwenk et al.2006] Holger Schwenk, Marta R. Costa-
Juss`a, and Jos ´e A. R. Fonollosa. 2006. Continuous
space language models for the iwslt 2006 task. In
IWSLT, pages 166–173.
[Schwenk2007] Holger Schwenk. 2007. Continuous
space language models. Comput. Speech Lang. ,
21(3):492–518, July.
[Schwenk2012] Holger Schwenk. 2012. Continuous
space translation models for phrase-based statisti-
cal machine translation. In Martin Kay and Chris-
tian Boitet, editors, Proceedings of the 24th Inter-
national Conference on Computational Linguistics
(COLIN), pages 1071–1080.
[Socher et al.2011] Richard Socher, Eric H. Huang, Jef-
frey Pennington, Andrew Y . Ng, and Christopher D.
Manning. 2011. Dynamic pooling and unfolding
recursive autoencoders for paraphrase detection. In
Advances in Neural Information Processing Systems
24.
[Son et al.2012] Le Hai Son, Alexandre Allauzen, and
Franc ¸ois Yvon. 2012. Continuous space transla-
tion models with neural networks. In Proceedings of
the 2012 Conference of the North American Chap-
ter of the Association for Computational Linguistics:
Human Language Technologies, NAACL HLT ’12,
pages 39–48, Stroudsburg, PA, USA.
[van der Maaten2013] Laurens van der Maaten. 2013.
Barnes-hut-sne. In Proceedings of the First Inter-
national Conference on Learning Representations
(ICLR 2013), May.
[Vaswani et al.2013] Ashish Vaswani, Yinggong Zhao,
Victoria Fossum, and David Chiang. 2013. De-
coding with large-scale neural language models im-
proves translation. Proceedings of the Conference
on Empirical Methods in Natural Language Pro-
cessing, pages 1387–1392.
[Zeiler2012] Matthew D. Zeiler. 2012. ADADELTA:
an adaptive learning rate method. Technical report,
arXiv 1212.5701.
[Zou et al.2013] Will Y . Zou, Richard Socher,
Daniel M. Cer, and Christopher D. Manning.
2013. Bilingual word embeddings for phrase-based
machine translation. In Proceedings of the ACL
Conference on Empirical Methods in Natural
Language Processing (EMNLP), pages 1393–1398.
A RNN Encoder–Decoder
In this document, we describe in detail the architecture of the RNN Encoder–Decoder used in the exper-
iments.
Let us denote an source phrase by X = ( x1,x2,..., xN ) and a target phrase by Y =
(y1,y2,..., yM ). Each phrase is a sequence of K-dimensional one-hot vectors, such that only one
element of the vector is 1 and all the others are 0. The index of the active (1) element indicates the word
represented by the vector.
A.1 Encoder
Each word of the source phrase is embedded in a 500-dimensional vector space: e(xi) ∈R500. e(x) is
used in Sec. 4.4 to visualize the words.
The hidden state of an encoder consists of 1000 hidden units, and each one of them at time t is
computed by
h⟨t⟩
j = zjh⟨t−1⟩
j + (1−zj)˜h⟨t⟩
j ,
where
˜h⟨t⟩
j = tanh
(
[We(xt)]j +
[
U
(
r ⊙h⟨t−1⟩
)]
j
)
,
zj =σ
(
[Wze(xt)]j +
[
Uzh⟨t−1⟩
]
j
)
,
rj =σ
(
[Wre(xt)]j +
[
Urh⟨t−1⟩
]
j
)
.
σ and ⊙are a logistic sigmoid function and an element-wise multiplication, respectively. To make the
equations uncluttered, we omit biases. The initial hidden state h⟨0⟩
j is ﬁxed to 0.
Once the hidden state at the N step (the end of the source phrase) is computed, the representation of
the source phrase c is
c = tanh
(
Vh⟨N⟩
)
.
A.1.1 Decoder
The decoder starts by initializing the hidden state with
h′⟨0⟩= tanh
(
V′c
)
,
where we will use ·′to distinguish parameters of the decoder from those of the encoder.
The hidden state at time tof the decoder is computed by
h′⟨t⟩
j = z′jh′⟨t−1⟩
j + (1−z′j) ˜h′⟨t⟩
j ,
where
˜h′⟨t⟩
j = tanh
([
W′e(yt−1)
]
j + r′j
[
U′h′
⟨t−1⟩+ Cc
])
,
z′j =σ
([
W′ze(yt−1)
]
j +
[
U′zh′
⟨t−1⟩
]
j + [Czc]j
)
,
r′j =σ
([
W′re(yt−1)
]
j +
[
U′rh′
⟨t−1⟩
]
j + [Crc]j
)
,
and e(y0) is an all-zero vector. Similarly to the case of the encoder, e(y) is an embedding of a target
word.
Unlike the encoder which simply encodes the source phrase, the decoder is learned to generate a target
phrase. At each time t, the decoder computes the probability of generating j-th word by
p(yt,j = 1|yt−1,..., y1,X) = exp
(
gjs⟨t⟩
)
∑K
j′=1 exp
(
gj′s⟨t⟩
),
where the i-element of s⟨t⟩is
s⟨t⟩
i = max
{
s′⟨t⟩
2i−1,s′⟨t⟩
2i
}
and
s′⟨t⟩= Ohh′⟨t⟩+ Oyyt−1 + Occ.
In short, the s⟨t⟩
i is a so-called maxout unit.
For the computational efﬁciency, instead of a single-matrix output weight G, we use a product of two
matrices such that
G = GlGr,
where Gl ∈RK×500 and Gr ∈R500×1000.
B Word and Phrase Representations
Here, we show enlarged plots of the word and phrase representations in Figs. 4–5.
Figure 6: 2–D embedding of the learned word representation. The top left one shows the full embedding space, while the other three ﬁgures show the zoomed-in
view of speciﬁc regions (color–coded).
Figure 7: 2–D embedding of the learned phrase representation. The top left one shows the full representation space (1000 randomly selected points), while the
other three ﬁgures show the zoomed-in view of speciﬁc regions (color–coded).

Compiling ONNX Neural Network Models Using
MLIR
Tian Jin1,†1,∗ Gheorghe-Teodor Bercea1 Tung D. Le2,∗ Tong Chen1
Gong Su1 Haruki Imai2 Yasushi Negishi2 Anh Leu1 Kevin O’Brien1
Kiyokuni Kawachiya2 Alexandre E. Eichenberger1
Abstract: Deep neural network models are becoming increasingly popular and have been used in various
tasks such as computer vision, speech recognition, and natural language processing. Machine learning mod-
els are commonly trained in a resource-rich environment and then deployed in a distinct environment such as
high availability machines or edge devices. To assist the portability of models, the open-source community
has proposed the Open Neural Network Exchange (ONNX) standard. In this paper, we present a high-level,
preliminary report on our onnx-mlir compiler, which generates code for the inference of deep neural net-
work models described in the ONNX format. Onnx-mlir is an open-source compiler implemented using the
Multi-Level Intermediate Representation (MLIR) infrastructure recently integrated in the LLVM project.
Onnx-mlir relies on the MLIR concept of dialects to implement its functionality. We propose here two new
dialects: ( 1 ) an ONNX speciﬁc dialect that encodes the ONNX standard semantics, and ( 2 ) a loop-based
dialect to provide for a common lowering point for all ONNX dialect operations. Each intermediate repre-
sentation facilitates its own characteristic set of graph-level and loop-based optimizations respectively. We
illustrate our approach by following several models through the proposed representations and we include
some early optimization work and performance results.
1. Introduction
Deep neural network models have been used widely for
various tasks such as computer vision, speech recognition,
and natural language processing. The success of such mod-
els was mainly originated from the development of acceler-
ators, especially GPU accelerators, back in 2012 [3]. Since
then, many deep learning frameworks, such as Torch, Caﬀe,
Theano, and TensorFlow, have been developed to facilitate
the training and inferencing of deep neural network models,
which signiﬁcantly speeds up the explosion of deep learning
in many areas. However, training and inferencing are often
done on diﬀerent environments due to their diﬀerent opti-
mization characteristics. For example, a model is trained
using a large-scale distributed system since it might need
weeks or months to ﬁnish, and can then be used on light-
weight devices such as Internet of Things or mobile phones
for inferencing. Hence, it is desirable to dynamically rewrite
a trained model so that it runs eﬃciently on a target envi-
ronment.
Many deep learning frameworks utilize a highly-optimized
1 IBM T.J. Watson Research Center,
1101 Kitchawan Rd, Yorktown Heights, NY 10598, USA
2 IBM Research - Tokyo,
19-21, Nihonbashi Hakozaki-cho, Chuo-ku, Tokyo 103-8510,
Japan
†1 Presently with Massachusetts Institute of Technology (MIT)
∗ Corresponding authors: Tung D. Le (tung@jp.ibm.com), Tian
Jin (tianjin@mit.edu)
library written for a target accelerator. Rewriting a model
for inferencing consists of replacing the operations in the
model with the function calls in the library. While such a
library-call approach simpliﬁes the rewritten procedure and
would lead to improved performance, it exposes the follow-
ing drawbacks. Firstly, the number of models that can be
rewritten is limited by the provided functions in the library.
Secondly, it is often the case that users need to install ad-
ditional packages to make the library work well. Thirdly, it
lacks the ability to tailor code speciﬁc to diﬀerent problems
since the same function may be used for them.
We tackle these drawbacks by developing a compiler that
rewrites a trained model to native code for a target hard-
ware. It uses many mature optimization techniques devel-
oped during the long history of compiler, such as the ability
to tailor code for a speciﬁc problem, memory optimizations,
and parallelization. Our compiler is completely based on
open-source software. In particular, we chose Open Neu-
ral Network Exchange (ONNX) [1] as a format to represent
the input model of our compiler. ONNX is an open-source
machine-independent format and widely used for exchang-
ing neural network models. It has been actively maintained
by and contributed from open source communities. Our
compiler was written using Multi-level Intermediate Repre-
sentation (MLIR) [5], a modern open source compiler infras-
tructure for multi-level intermediate representations and a
subproject inside LLVM [4].
Our compiler is completely open-sourced and a subproject
1
arXiv:2008.08272v2  [cs.PL]  1 Oct 2020
inside the ONNX project †1 . Although it is still under de-
velopment, it can already compile some popular models such
MNIST and ResNet50 to native code on x86 machines, IBM
Power Systems†2 , and IBM System Z †3 . In this paper, we
will introduce our compiler by
• presenting its overall design and architecture of the com-
piler,
• introducing two new dialects: onnx dialect to encode
the ONNX standard semantics, and krnl dialect to pro-
vide for a common lowering point for all ONNX dialect
operations.
• introducing optimization passes such as graph rewrit-
ing, constant propagation, and memory management,
and
• discussing some problems we encountered when emit-
ting native code for diﬀerent architectures.
The remainder of the paper is organized as follows. In
Sec. 2, we brieﬂy discuss ONNX and MLIR on which our
compiler is based. In Sec. 3, we introduce our compiler, its
design principle, and architecture. We also discuss in this
section two new dialects, i.e., onnx and krnl, and some op-
timization passes. In Sec. 4, we present some preliminary
experiemental results for MNIST and ResNet50 models on
IBM Power Systems. Finally, we conclude our paper and
discuss future work in Sec. 5.
2. Background
2.1 ONNX
Open Neural Network Exchange (ONNX) [1] is an open
source format for artiﬁcial intelligence models, including
both deep learning and traditional machine learning. It de-
ﬁnes an extensible computational graph model, operators,
and standard data types, which provides a common IR for
diﬀerent frameworks. There are two ONNX variants: the
neural-network-only ONNX variant recognizes only tensors
as input and output types, while the classic machine learning
ONNX-ML also recognizes sequences and maps. ONNX-ML
extends the ONNX operator set with machine learning al-
gorithms that are not based on neural networks. In this
paper, we focus on the neural-network-only ONNX variant
and refer to it as just ONNX.
In ONNX, the top-level structure is a ‘Model’ to asso-
ciate metadata with a graph. Operators in ONNX are di-
vided into a set of primitive operators and functions, where
a function is an operator whose calculation can be expressed
via a subgraph of other operators. A graph is used to de-
scribe a function. There are lists of nodes, inputs, outputs,
and initializers (constant values or default values for inputs)
in a graph. An acyclic dataﬂow graph is constructed as a
topological sort of the list of nodes in the graph. Each node
†1 https://github.com/onnx/onnx-mlir
†2 https://www.ibm.com/it-infrastructure/power/power9
†3 https://www.ibm.com/it-infrastructure/z/hardware
Listing 1: ONNX model for LeakyRelu operator (printed
using ‘protoc’ command).
1 ir_version : 3
2 producer_name : " backend - test "
3 graph {
4 node {
5 input : "x"
6 output : "y"
7 op_type : " LeakyRelu "
8 attribute {
9 name : " alpha "
10 f: 0.1
11 type : FLOAT
12 }
13 }
14 name : " test_leakyrelu "
15 input {
16 name : "x"
17 type {
18 tensor_type {
19 elem_type : 1
20 shape {
21 dim {
22 dim_value : 3
23 }
24 dim {
25 dim_value : 4
26 }
27 dim {
28 dim_value : 5
29 }
30 }
31 }
32 }
33 }
34 output {
35 name : "y"
36 type {
37 tensor_type {
38 elem_type : 1
39 shape {
40 dim {
41 dim_value : 3
42 }
43 dim {
44 dim_value : 4
45 }
46 dim {
47 dim_value : 5
48 }
49 }
50 }
51 }
52 }
53 }
54 opset_import {
55 version : 9
56 }
in a graph contains the name of the operator it invokes, in-
puts, outputs, and attributes associated with the operator.
Inputs and outputs can be marked as variadic or optional.
There are three data types used to deﬁne inputs and out-
puts, i.e., ‘Tensor’, ‘Sequence’, and ‘Map’.
ONNX uses the Protocol Buﬀers†4 deﬁnition language for
its syntax. Listing 1 shows an example of an ONNX model
for the LeakyRelu operator. There is one node in the graph
(Lines 4–13), which is associated with LeakyRelu, and has
†4 https://developers.google.com/protocol-buﬀers
2
Fig. 1: Operations and Regions in MLIR.
one input, one output, and one attribute. The input and
output tensors have the shape of ⟨3x4x5⟩and element type
of ﬂoat32 (elem type: 1 at Lines 19 and 38).
2.2 MLIR
Multi-level Intermediate Representation (MLIR) [5] is a
modern compiler infrastructure which is reusable and exten-
sible. It reduces the cost of building domain-specﬁc compil-
ers by facilitating the design and implementation of code
generators, translators, and optimizers at diﬀerent abstrac-
tion levels. MLIR is a subproject of the LLVM project [6]
and has many similarities to the LLVM compiler infrastruc-
ture [4]. In this section, we brieﬂy review some of the fea-
tures in MLIR that were used to build our compiler. For
more information about MLIR, one can refer to a previous
study [5]. Readers who are familiar with MLIR can skip this
section.
Similar to LLVM, MLIR is a three-address static single
assignment (SSA)-based IR, where values are deﬁned be-
fore use and have a scope deﬁned by their dominance rela-
tions. Operations may produce zero or more results, and
each operation is a distinct SSA value with its own type
deﬁned by the type system. The type system in MLIR is
open, and one can deﬁne application-speciﬁc types. There
are a number of primitive types, e.g., integers, as well as
aggregate types for tensors and memory buﬀers, e.g., ‘Ten-
sor’ and ‘MemRef’ types. A Tensor type is abstracted
and does not have a pointer to the data while a Mem-
Ref type is a lower representation, referring to a region of
memory. In MLIR, Tensor and MemRef types are syn-
tactically represented as tensor⟨D1×D2×. . .×DN×dtype⟩
and memref⟨D1×D2×. . .×DN×dtype⟩, respectively, where
D1, D2, . . . , DN are intergers representing the dimensions of
a tensor or memref, and dtype is the type of the elements in
a tensor or memref, e.g., f32 for ﬂoat32. ⟨D1×D2×. . .×DN⟩
is called the shape of a tensor or memref. Tensor and Mem-
Ref types can be unranked when their shapes are unknown.
In MLIR, unranked Tensor and MemRef types are syntacti-
cally represented as tensor⟨∗×dtype⟩and memref⟨∗×dtype⟩,
respectively.
An operation is the unit of code in MLIR. To deﬁne an
operation, a TableGen-based [7] speciﬁcation for an opera-
tion descriptor is used. Figure 1 shows the structure of an
operation. An operation has a list of SSA operands and may
have attributes that store static information. An operation
can hold a region which is a list of blocks. A block contains a
list of operations and ends with a terminator operation that
may have successor blocks to which the control ﬂow may be
transferred. That being said, nested regionsbecomes a ﬁrst-
class concept in MLIR, which is eﬃcient to represent control
ﬂow graphs. A function is an operation with a single region
and attributes. A module is an operation with a single re-
gion containing a single block and terminated by a dummy
operation.
To develop a compiler using MLIR, users often need to
deﬁne dialects and optimization passes. A dialect serves as
an abstraction level or intermediate representation, and an
optimization pass is to enable optimization at an abstraction
level or transformation among abstraction levels.
There are dialects in MLIR that are ready to use, e.g.,
llvm, std, scf, and aﬃne. The llvm dialect is a low-level di-
alect. It wraps the LLVM IR types and instructions into
MLIR types and operations. The std dialect includes stan-
dard operations such as load, store, addi, addf, absf, and
call. The scf dialect deﬁnes control ﬂow operations such as
for and if. The aﬃne dialect provides an abstraction for
aﬃne operations and analyses.
Optimization passes can be roughly classiﬁed into three
categories: general transformation, conversion, and dialect-
speciﬁc. General transformation passes includes common
passes such as ‘canonicalize’ pass for operation canonical-
ization, ‘CSE’ pass to eliminate common sub-expressions,
and passes to print IR information such as ‘print-op-graph’,
‘print-op-stats’, and ‘print-cfg-graph’. Conversion passes are
to convert operations in one dialect to operations in another
dialect, e.g., ‘convert-std-to-llvm’ pass to convert standard
operations into LLVM instructions. Finally, dialect-speciﬁc
passess are for transformation in a dialect, e.g., ‘aﬃne-
loop-unroll-jam’ pass to unroll and jam aﬃne loops in the
aﬃne dialect. MLIR passes can be expressed via Declara-
tive Rewriting Rules (DRRs) using tablegen records or via
writing code in C++.
To denote an operation in a dialect, we explicitly use a
form of dialect name.operation name. For example, std.load
means the operation load of dialect std. Optimization passes
are named with preﬁx ‘ --’, for example, --canonicalize is the
canonlicalization pass.
Listing 2 shows an example for calculating the exponential
of a given input tensor, element-wise, usingstd and aﬃne di-
alects. The top level is a module containing a function ‘exp’.
The function ‘exp’ accepts one input that is of memref type,
and produces an output of the same type. The memory for
the output is allocated via std.alloc (Line 3). There is a
nested loop (Lines 4–10), iterating over dimensions of the
inputs using aﬃne.for, loading each element from the input
using aﬃne.load (Line 6), computing the exponential using
std.exp (Line 7), and storing the result in the output using
aﬃne.store (Line 8). The output of the function is ﬁnally
returned using std.return.
3
Listing 2: Compute the exponential of a tensor in MLIR.
1 module {
2 func @exp ( arg0 : memref <3 x4xf32 >) -> memref <3 x4xf32 > {
3 %1 = std . alloc () : memref <3 x4xf32 >
4 affine . for %arg1 = 0 to 3 {
5 affine . for %arg2 = 0 to 4 {
6 %2 = affine . load %arg0 [ %arg1 , %arg2 ] : memref <3 x4xf32 >
7 %3 = std . exp %2 : f32
8 affine . store %3 , %1[ %arg1 , %arg2 ] : memref <3 x4xf32 >
9 }
10 }
11 std . return %1 : memref <3 x4xf32 >
12 }
13 }
Fig. 2: Architecture of onnx-mlir. Names preﬁxed with ‘ --’ are
passes.
Fig. 3: ONNX model for element-wise addition.
3. Compiling ONNX Models
This section introduces our compiler, onnx-mlir. We ﬁrst
discuss its overall architecture. We then introduce two new
dialects, onnx and krnl dialects. Finally, we present MLIR
passes for carrying out optimization.
3.1 Overview
Figure 2 shows the overall architecture of onnx-mlir. The
input is an ONNX model, and the output is a library con-
taining the compiled code. The output library contains an
entry function called ‘ dyn entry point main graph’ whose
inputs and outputs are similar to the ONNX model’s inputs
and outputs, respectively. To carry out inference with the
output library, users write their program to call the entry
function by passing inputs to the function and obtain re-
sults.
There are ﬁve main dialects in onnx-mlir, i.e., onnx, krnl,
aﬃne, std and llvm, organized into four abstraction levels.
Two new dialects, onnx and krnl, are discussed in Sections
3.2 and 3.3, respectively. The ﬁrst abstraction level is a
high-level representation of ONNX operations. It consists of
operations in onnx and std dialects, where the onnx dialect
is automatically generated via an importerthat is a python
script. The second abstraction level includes krnl, aﬃne and
std dialects. krnl dialect provides a representation that is
suitable for loop optimizations, which is able to carry out
aﬃne transformations such as tile, skew, and permutation
easily. It plays as an intermediate dialect for eﬃciently low-
ering the onnx dialect into low-level dialects (e.g., aﬃne, std
and llvm). The third abstraction level includes aﬃne and
std dialects where existing optimization passes in MLIR can
be freely applied. The forth abstraction level includes only
llvm dialect that is realy to generate bitcode.
There are MLIR passes for converting one dialect to an-
other, and for doing optimizations at a speciﬁc dialect. onnx
dialect is converted to krnl dialect via pass --convert-onnx-
to-krnl. Then krnl dialect (except some of its operations)
is converted into aﬃne and std dialects via pass --convert-
krnl-to-aﬃne. The remaining operations in krnl dialect and
operations in aﬃne and std dialects are directly converted
into instructions in llvm via pass --convert-krnl-to-llvm. The
right side of Fig. 2 shows optimization passes that can be
carried out at each abstraction level.
We only enumerate the important optimizations here, and
the list of optimization passes is not exhaustive.
Before discussing dialects and optimization passes in de-
tail, we give a brief running example and go through dialects
in onnx-mlir. This example is a testcase model in ONNX
that performs element-wise binary addition. Figure 3 shows
this ONNX model of the testcase. Operation add accepts
two tensors of type ⟨3x4x5xf32⟩(element type is ﬂoat 32)
and returns a result tensor, i.e., sum, of the same type. List-
ings 3, 4, and 5 show emitted programs in diﬀerent dialects
onnx, krnl, aﬃne, respectively. We omit the program in llvm
4
Listing 3: Operation add in onnx dialect, generated using importer.
1 module {
2 func @main_graph ( %arg0 : tensor <3 x4x5xf32 >, %arg1 : tensor <3 x4x5xf32 >) -> tensor <* xf32 > {
3 %0 = " onnx . Add "( %arg0 , %arg1 ) : ( tensor <3 x4x5xf32 >, tensor <3 x4x5xf32 >) -> tensor <* xf32 >
4 std . return %0 : tensor <* xf32 >
5 }
6 " onnx . EntryPoint "() { func = @main_graph , numInputs = 2 : i32 , numOutputs = 1 : i32 } : () -> ()
7 }
Listing 4: Operation add in krnl dialect, generated by applying passes --shape-inference and --convert-onnx-to-krnl.
1 module {
2 func @main_graph ( %arg0 : memref <3 x4x5xf32 >, %arg1 : memref <3 x4x5xf32 >) -> memref <3 x4x5xf32 > {
3 %0 = alloc () : memref <3 x4x5xf32 >
4 %1 :3 = krnl . define_loops 3
5 krnl . iterate (%1 #0 , %1 #1 , %1 #2) with (%1 #0 -> %arg2 = 0 to 3, %1 #1 -> %arg3 = 0 to 4, %1 #2 -> ←↩
%arg4 = 0 to 5) {
6 %2 = affine . load %arg0 [ %arg2 , %arg3 , %arg4 ] : memref <3 x4x5xf32 >
7 %3 = affine . load %arg1 [ %arg2 , %arg3 , %arg4 ] : memref <3 x4x5xf32 >
8 %4 = std . addf %2 , %3 : f32
9 affine . store %4 , %0[ %arg2 , %arg3 , %arg4 ] : memref <3 x4x5xf32 >
10 }
11 std . return %0 : memref <3 x4x5xf32 >
12 }
13 " krnl . entry_point "() { func = @main_graph , numInputs = 2 : i32 , numOutputs = 1 : i32 } : () -> ()
14 }
Listing 5: Operation add in aﬃne dialect, generated by applying the pass --convert-krnl-to-aﬃne.
1 module {
2 func @main_graph ( %arg0 : memref <3 x4x5xf32 >, %arg1 : memref <3 x4x5xf32 >) -> memref <3 x4x5xf32 > {
3 %0 = alloc () : memref <3 x4x5xf32 >
4 affine . for %arg2 = 0 to 3 {
5 affine . for %arg3 = 0 to 4 {
6 affine . for %arg4 = 0 to 5 {
7 %1 = affine . load %arg0 [ %arg2 , %arg3 , %arg4 ] : memref <3 x4x5xf32 >
8 %2 = affine . load %arg1 [ %arg2 , %arg3 , %arg4 ] : memref <3 x4x5xf32 >
9 %3 = std . addf %1 , %2 : f32
10 affine . store %3 , %0[ %arg2 , %arg3 , %arg4 ] : memref <3 x4x5xf32 >
11 }
12 }
13 }
14 std . return %0 : memref <3 x4x5xf32 >
15 }
16 " krnl . entry_point "() { func = @main_graph , numInputs = 2 : i32 , numOutputs = 1 : i32 } : () -> ()
17 }
due to space limitations.
In onnx dialect, operations are represented similarly to
their descriptions in ONNX. The ONNX model is converted
into the function main graph. To generate an entry point
function into which users feed their inputs, we create a
helper operation in the onnx dialect, i.e., onnx.EntryPoint,
which keeps meta-data in the operation’s attributes such as
function name to call and the number of inputs and outputs.
In krnl dialect, operation onnx.Add is translated into a
loop-based computation represented by operations in the
krnl dialect, where scalar computation is represented by
primitive operations in the aﬃne and std dialects. We can
apply loop optimizations, such as tile, skew, or transpose,
to loop-based computation. At this level, we allocate mem-
ory for output tensors, and memory management can be
performed.
In aﬃne dialect, optimized loop-based computation inkrnl
dialect is translated into aﬃne.for loops. At this level, we
still have an operation in krnl, i.e., krnl.entry point. Such an
operation is not related to the main computation and will be
directly converted to llvm. Operations in the aﬃne dialect
will be converted to operations in the std and scf dialects
before being lowered to instructions in the llvm dialect.
3.2 onnx dialect
onnx dialect is the ﬁrst abstraction level in onnx-mlir
and represents an ONNX model in MLIR language. We
wrote a python script to automatically import ONNX opera-
tions into the tablegen-based operation deﬁnitions in MLIR.
These imported operations are organized into the onnx di-
alect. Thanks to tablegen, the operation deﬁnition in the
onnx dialect is quite similar to the operation description in
ONNX, where we are able to represent all necessary infor-
mation, such as inputs, outputs, attributes, and description,
into a single tablegen-based deﬁnition in human-readable
textual form.
5
Listing 6: Tablegen-based deﬁnition for operation relu.
1 def ONNXLeakyReluOp : ONNX_Op <" LeakyRelu ",
2 [ NoSideEffect , DeclareOpInterfaceMethods < ShapeInferenceOpInterface >] > {
3 let summary = " ONNX LeakyRelu operation ";
4 let description = [{" LeakyRelu takes ... "}];
5 let arguments = ( ins AnyTypeOf <[ TensorOf <[ F16 ]>, TensorOf <[ F32 ]>, TensorOf <[ F64 ] >] >: $X , ←↩
DefaultValuedAttr < F32Attr , " 0.01 " >: $alpha );
6 let results = ( outs AnyTypeOf <[ TensorOf <[ F16 ]>, TensorOf <[ F32 ]>, TenorOf <[ F64 ] >] >: $Y);
7 let extraClassDeclaration = [{ ... }];
8 }
We also created a new operation in the onnx dialect,
i.e., onnx.EntryPoint to keep information related to the
dynamic list of inputs in an ONNX model. This op-
eration will be lowered to generate the entry function
‘ dyn entry point main graph’ of the generated library.
Listing 6 shows a tablegen-based deﬁnition for the relu
operation imported via the importer in onnx-mlir. The op-
eration description is represented in the ‘description’ ﬁeld
(Line 4). Inputs and attributes are represented in the ‘argu-
ments’ ﬁeld, while outputs were represented in the ‘results’
ﬁeld (Lines 5–6). All inputs and outputs will be imported
as a tensor in MLIR. The importer automatically infers ele-
ment types for inputs, attributes, and outputs. However, the
shape of a tensor will be inferred via the --shape-inference
pass, which is a trait in the LeakyRelu operation (Line 2).
MLIR generates a C++ class deﬁnition for an operation
from its tablegen-based deﬁnition. If users want to deﬁne
custom declaration in the class, it can be done via the ‘ex-
traClassDeclaration’ ﬁeld (Line 7).
3.3 krnl dialect
A computation kernel in a neural network workload has lo-
cal structural simplicity in which loop nests are often simple,
e.g., hyper-rectangle and statements carry quite straightfor-
ward arithmetic semantics. Such a characteristic is quite
suitable to be represented in a polyhedral model for opti-
mization [8]. krnl dialect aims to host both loop optimization
and scalar semantic optimization in a single representation.
It is expected to provide interpretability where not only is
polyhedral representation readable but it also makes pro-
gram semantics (or what to execute) and program schedules
(how and when to execute) independent. In other words,
our goal is to optimize not only programs but also the com-
position of individual schedules, which is a feature that is
often lacking in other existing systems.
Below is an example that deﬁnes a nested loop in krnl:
1 %ii , %jj = krnl . define_loops 2
2 krnl . iterate (%ii , %jj ) with ( %ii -> %i = 0 ←↩
to 10 , %jj -> %j = 0 to 10) {
3 %foo = std . addi %i , %j : index
4 }
where krnl.deﬁne loops deﬁnes two loops, called ii and jj.
These loop variables will be used to express both program
semantics and schedules. Operation krnl.iterate semantically
accepts two types of loop variables: variables for original
loops and variables for scheduled loops. In syntactic sugar
form, we separate the two types of loops by the keyword
with, i.e. (scheduled loops) with (original loops). Induction
variables, e.g., i and j in the above example, will be deﬁned
by using original loops. If there is no schedule (e.g. block,
skew, etc.), the scheduled loops are similar to the original
loops.
Now, we insert a schedule for blocking or tiling. Without
loss of generality, we deﬁne just one loop instead of two.
1 %ii = krnl . define_loops 1
2 %ib , %il = krnl . block %ii 2 : ←↩
(! krnl . loop ) ->(! krnl .loop , ! krnl . loop )
3 krnl . iterate (%ib , %il ) with ( %ii -> %i = 0 ←↩
to 10) {
4 %foo = std . addi %i , %i : index
5 }
Operation krnl.block (Line 2) takes a loop and integer as
inputs, where the integer is the tile size with which we want
to carry out blocking. Results are two loop variables: one
for the outer loop and the other for the inner loop. The two
loops will be used as the result of scheduling and be passed
to krnl.iterate (Line 3). It is worth noting that the original
loops and computation in krnl.iterate remained unchanged
while inserting a schedule, which is exactly what we want
for seperating program semantics and schedules in our krnl
dialect.
The --convert-krnl-to-aﬃne pass automatically generates
optimized aﬃne.for based loops as follows.
1 # map0 = affine_map <( d0) -> (d0)>
2 # map1 = affine_map <( d0) -> (d0 + 2) >
3 affine . for %arg0 = 0 to 10 step 2 {
4 affine . for %arg1 = # map0 ( %arg0 ) to ←↩
# map1 ( %arg0 ) {
5 %0 = addi %arg1 , %arg1 : index
6 }
7 }
The outer aﬃne.for iterates with step 2 i.e., the tile size, and
the inner aﬃne.for iterates over the elements in a tile.
Other schedules, such as skew and permutation are used
in a similar manner. All schedules are composable and can
be nested.
3.4 Optimization Passes
In this section, we discuss some of the optimization passes
in onnx-mlir. Thanks to the expressive power of MLIR,
many optimizations can be expressed easily via Declarative
Rewriting Rules (DRRs) using tablegen records or writing
6
code in C++.
3.4.1 Operation Decomposition
In ONNX, many operations can be expressed using other
basic operations. For example, ReduceL1 over a vector x
is mathematically calculated by summing up the absolute
values of the elements in x. In other words, we have
ReduceL1 = ReduceSum (Abs x)
We only need to lower a subset of operations in the onnx
dialect to krnl dialect, while the remaining operations in the
onnx dialect will be decomposed into operations in the sub-
set.
Using the DRRs in MLIR, operation decomposition is con-
cisely written as the following pattern:
1 def ReduceL1Pattern : Pat <
2 ( ReduceL1Op $x , $axes , $keepdims ),
3 ( ReduceSumOp ( AbsOp $x), $axes , $keepdims )
4 >;
where ReduceL1Op, ReduceSumOp, and AbsOp are
programmable forms of operations onnx.ReduceL1,
onnx.ReduceSum, and onnx.Abs respectively. Vari-
ables x, axes, and keepdims are for keeping input values
of operation ReduceL1Op. The pattern ‘ReduceL1Pattern’
contains a source pattern to match a graph of one operation
ReduceL1Op (Line 2) and a destination pattern to generate
a graph of two operations ReduceSumOp and AbsOp
(Line 3). Whenever an operation ReduceL1Op appears in
an ONNX model, it will be replaced with a combination of
ReduceSumOp and AbsOp.
3.4.2 Shape Inference
The --shape-inference pass attempts to infer shapes for all
tensors in a program at onnx. The pass traverses all opera-
tions in a program, infers the shapes of tensors with unrank
shapes (i.e. tensor⟨∗xf32⟩), propagates the ranked shapes to
consuming operations, and terminates once all tensors have
ranked shapes. For one operation, if its inputs have static
shapes, it is likely that the --shape-inference pass will be
able to infer static shapes for its outputs. If the inputs have
dynamic shapes (e.g. tensor⟨?x?x?xf32⟩), the outputs will
also have dynamic shapes also, except for some operations
whose output tensors’ shapes are speciﬁed in the operation
attributes.
3.4.3 Graph Rewriting
Graph rewriting is a powerful optimization tool. It is
intensively applied to neural networks since calculation in
a neural network is expressed via a dataﬂow graph. In
MLIR, graph rewriting rules are conveniently represented
using DRRs.
For example, the following rule is to fuse onnx.Add and
onnx.MatMul into a single operation onnx.Gemm under the
condition that the result of MatMulOp is only consumed by
AddOp:
1 def MulAddToGemmPattern : Pat <
2 ( AddOp ( MatMulOp : $res $m1 , $m2 ), $m3 ),
3 ( GemmOp $m1 , $m2 , $m3 ),
4 [( HasOneUse $res )]
5 >;
Another example is to remove an IdentityOp operation by
passing its input directly to its consuming operations.
1 def IdentityEliminationPattern : Pat <
2 ( ONNXIdentityOp $arg ),
3 ( replaceWithValue $arg )
4 >;
Users can write as many rewriting rules as possible in the
same manner.
3.4.4 Constant propagation
Constant propagation is a well-known optimization in
compilers. In onnx-mlir, we created a pass to do this during
compilation. There are two key ideas in constant propaga-
tion: ( 1 ) if all the inputs of an operation are constant, com-
pute its outputs during compilation and remove the opera-
tion, ( 2 ) if there is a mix of constant and non-constant in-
puts, normalize the operation. Normalization is to increase
the possibility of constant propagation and strongly depends
on the mathematical properties of an operation. Below are
some normalization rules in onnx-mlir for the onnx.Add op-
eration whose properties are associative and communicative.
( 1 ) c + x ⇒x + c
( 2 ) ( x + c1) + c2 ⇒x + (c1 + c2)
( 3 ) ( x + c) + y ⇒(x + y) + c
( 4 ) x + (y + c) ⇒(x + y) + c
( 5 ) ( x + c1) + (y + c2) ⇒(x + y) + (c1 + c2)
where x and y are non-constant values, and c, c1, and c2 are
constant values. Normalization rules are expressed by using
the DRRs in MLIR.
4. Preliminary Experiments
4.1 ONNX operation support and testcases
ONNX provides a set of test cases for each operation.
When we support any operation in onnx-mlir, we enable its
ONNX test cases to check whether the operation behaves
correctly and produces correct result. At the time of writing
this paper, onnx-mlir supports 51 operations out of 139 op-
erations in ONNX, including important operations such as
convolution, pooling, Gemm, and LSTM. These are enough
to compile and execute major networks such as MNIST and
ResNet50. On the GitHub repository of onnx-mlir, we en-
able continuous integration on diﬀerent environments, i.e.,
Windows, Linux, and Docker environments, and diﬀerent
systems, i.e., x86 machines, IBM Power Systems, and Sys-
tem Z. All supported operations have passed tests on the
above environments.
4.2 MNIST and ResNet50
In this section, we present some of our preliminary results
for two neural network models in the ONNX Model Zoo:
MNIST and ResNet50 [2]. The MNIST †5 and ResNet50†6
†5 https://github.com/onnx/models/tree/master/vision/
classification/mnist
†6 https://github.com/onnx/models/tree/master/vision/
classification/resnet
7
models have already been trained in the CNTK and Caﬀe2
frameworks, respectively. We ran inferences on the given
test data set in each model. The experiments were con-
ducted on a machine with 2.3-GHz POWER9 processors.
For onnx-mlir, graph rewriting and canonicalization passes
were enabled. In this paper, we only provide a reference im-
plementation that is not optimized, thus performance mea-
surements are not applicable.
Table 1: Run inferencing with MNIST and ResNet50 on a
POWER9 machine. Time in seconds.
Model Compilation time Inference time
MNIST 0.237 0.001
ResNet50 7.661 7.540
Table 1 shows the running times for the MNIST and
ResNet50 models when doing inferencing. For each model,
we measured the compilation time for compiling the model
to native code and inference time for running the native
code with real inputs. MNIST is a small model with two
convolutional operations, one max pooling operation and a
matrix multiplication followed by an element-wise addition.
Compiling the MNIST model and carrying out inferencing
was rather fast, i.e., ﬁnished in less than one second. In the
MNIST model, the graph rewriting rule MulAddToGemm-
Pattern mentioned in Sec. 3.4.3 was applied to fuse matrix
multiplication and element-wise addition into a Gemm op-
eration. ResNet50 is a complex deep model consisting of 50
layers of operations such as convolutions and poolings. The
model is about 100 megabytes including learned weights.
For ResNet50, the current version of onnx-mlir does not
have any optimization applied to the model during com-
pilation. However, we believe that the compilation time
looks reasonable and the inference time is not so slow. We
hope that once we integrate important optimizations, such
as polyhedral optimizations, SIMD optimization, and loop
fusion in near future, the inference time will be signiﬁcantly
reduced.
4.3 Supported Systems
Although onnx-mlir is completely built upon widely-used
open source software such as ONNX and MLIR, we found a
problem related to supporting diﬀerent systems. In particu-
lar, we could not run ONNX models on Linux on IBM Sys-
tem Z (s390-linux) because the big-endian format was not
well-supported in ONNX and MLIR. There are two reasons
for such a problem. First, a large amount of public input
data and models in ONNX are stored in little-endian format.
Hence, they must be converted to big-endian format before
they are used in a big-endian system. Second, we found that
constant values in ONNX models are not correctly loaded in
MLIR. LLVM was well-supported in big-endian, but MLIR
was not. We created two patches to solve this problem:
one in ONNX †7 and one in MLIR †8 , and they are now
†7 https://github.com/onnx/onnx/pull/2633
†8 https://reviews.llvm.org/D78076
available at the master branches of ONNX and MLIR. As a
result, onnx-mlir now supports Linux on x86 (x86-Linux),
Linux on Power Systems (ppc64le-Linux), Linux on IBM Z
(s390-Linux), and Windows.
5. Conclusion
We are developing an open source compiler called onnx-
mlir for compiling ONNX models into native code. MLIR
was used as an infrastructure to build the compiler, and two
novel dialects were introduced, i.e., onnx and krnl. We also
discussed some optimizations such as graph rewriting and
constant propagation. It is worth noting that new optimiza-
tions can be easily integrated into onnx-mlir thanks to the
MLIR infrastructure. In the future, we will add more opti-
mizations, e.g., polyhedral optimization, loop fusion, SIMD
optimization, and enable code generation for accelerators.
Acknowledgements
We acknowledge the ONNX standard for hosting the
project and the external contributors for their contribu-
tions.
References
[1] Bai, J., Lu, F., Zhang, K. et al.: ONNX: Open Neu-
ral Network Exchange, GitHub (online), available from
⟨https://github.com/onnx/onnx⟩(accessed 2020-07-01).
[2] He, K., Zhang, X., Ren, S. and Sun, J.: Deep Residual
Learning for Image Recognition, CoRR, Vol. abs/1512.03385
(online), available from ⟨http://arxiv.org/abs/1512.03385⟩
(2015).
[3] Krizhevsky, A., Sutskever, I. and Hinton, G. E.: ImageNet
Classiﬁcation with Deep Convolutional Neural Networks, In-
ternational Conference on Neural Information Processing
Systems (NIPS), pp. 1097–1105 (2012).
[4] Lattner, C. and Adve, V.: LLVM: A Compilation Framework
for Lifelong Program Analysis and Transformation, San Jose,
CA, USA, pp. 75–88 (2004).
[5] Lattner, C., Amini, M., Bondhugula, U., Cohen, A.,
Davis, A., Pienaar, J., Riddle, R., Shpeisman, T., Vasi-
lache, N. and Zinenko, O.: MLIR: A Compiler Infrastruc-
ture for the End of Moore’s Law, (online), available from
⟨http://arxiv.org/abs/2002.11054⟩(2020).
[6] LLVM: The LLVM Project, LLVM (online), available from
⟨https://github.com/llvm/llvm-project⟩ (accessed 2020-07-
01).
[7] LLVM: TableGen, LLVM (online), available from
⟨https://llvm.org/docs/TableGen/⟩(accessed 2020-07-01).
[8] Pouchet, L.-N., Bastoul, C., Cohen, A. and Cavazos, J.: Iter-
ative optimization in the polyhedral model: Part II, multidi-
mensional time, ACM SIGPLAN Notices, Vol. 43, No. 6, pp.
90–100 (2008).
8

Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation
using Deep Neural Networks
Charith Mendis 1 Alex Renda 1 Saman Amarasinghe 1 Michael Carbin 1
Abstract
Predicting the number of clock cycles a processor
takes to execute a block of assembly instructions
in steady state (the throughput) is important for
both compiler designers and performance engi-
neers. Building an analytical model to do so is
especially complicated in modern x86-64 Com-
plex Instruction Set Computer (CISC) machines
with sophisticated processor microarchitectures
in that it is tedious, error prone, and must be per-
formed from scratch for each processor genera-
tion. In this paper we present Ithemal, the ﬁrst
tool which learns to predict the throughput of
a set of instructions. Ithemal uses a hierarchi-
cal LSTM–based approach to predict throughput
based on the opcodes and operands of instructions
in a basic block. We show that Ithemal is more
accurate than state-of-the-art hand-written tools
currently used in compiler backends and static ma-
chine code analyzers. In particular, our model has
less than half the error of state-of-the-art analyti-
cal models (LLVM’s llvm-mca and Intel’s IACA).
Ithemal is also able to predict these throughput
values just as fast as the aforementioned tools,
and is easily ported across a variety of processor
microarchitectures with minimal developer effort.
1 Introduction
The throughput of a sequence of instructions—the num-
ber processor clock cycles taken to execute the sequence
when looped in steady state—determines how fast those
instructions can process data. Accurately predicting the
throughput of a basic block 1 is an essential requirement
in many systems, to be able to predict and optimize run-
time performance. For instance, constraint–based register
1MIT CSAIL. Correspondence to: Charith Mendis
<charithm@mit.edu>.
Proceedings of the36 th International Conference on Machine
Learning, Long Beach, California, PMLR 97, 2019. Copyright
2019 by the author(s).
1In this work, we focus speciﬁcally on basic blocks, sequences
of instructions with no branches or jumps.
allocation and instruction scheduling (Lozano et al., 2012)
relies on accurate throughput estimations, as do learning–
based techniques like genetic algorithm based register allo-
cation (Stephenson et al., 2003) and reinforcement learning
based instruction scheduling (McGovern & Moss, 1999).
The alternative – measuring throughput on demand by exe-
cuting the basic block – is too expensive for most compilers
and learning–based solutions. In practice, most systems
employ analytical models to predict throughput. For in-
stance, the LLVM compiler team (Lattner & Adve, 2004)
recently merged2 a command-line tool, llvm-mca (Di Bi-
agio & Davis, 2018), that exposes a machine model for
throughput estimation. Intel has also released a closed-
source machine code analyzer, IACA (Intel, 2017), which
relies on internal knowledge of Intel’s processor design.
These models are typically an order of magnitude faster
than measuring a basic block’s throughput. However, man-
ually writing an accurate and complete model is tedious,
error-prone, and exceedingly difﬁcult without knowledge of
the exact mechanisms of the processor.
In the hunt for accuracy, developers build complicated mod-
els which must make signiﬁcant tradeoffs with the model’s
portability and speed.
Accuracy. Modern x86-64 Complex Instruction Set Com-
puter (CISC) processors contain many hardware optimiza-
tions that signiﬁcantly complicate building accurate analyti-
cal models. In order to implement an instruction set archi-
tecture (ISA) like x86-64, processors actually implement
an underlying microarchitecture, a physical implementation
of the ISA speciﬁcation. Processors translate instructions
from the ISA to instructions in the latent microarchitectural
language (termed micro-ops), then execute those micro-ops.
The micro-ops may undergo optimizations such as micro-op
fusion, in which micro-ops of different instructions may be
combined together; out-of-order execution, in which instruc-
tions can be executed in any semantics–preserving order;
register renaming, where false dependencies can be broken
to enable more parallel execution; and many more vendor-
speciﬁc optimizations. This makes the prediction problem
highly complex and non-linear.
2 lists.llvm.org/pipermail/llvm-dev/2018-March/121490.html
Ithemal: Basic Block Throughput Prediction
Portability. While ISAs like x86-64 stay relatively stable,
processor vendors release updated processor implementa-
tions with different microarchitectures every few years. For
example, Intel released the Haswell and Skylake microar-
chitectures in 2013 and 2015 respectively for the x86-64
instruction set. Each microarchitecture of a processor fam-
ily has its own quirks and intricacies. Manually writing a
throughput estimator to support different microarchitectures
requires rewriting instruction tables, resource utilization
charts, and modeling microarchitectural optimizations, all
of which are tedious and error-prone. This is complicated by
the vast, incomplete, and incorrect documentation for many
processors, where understanding of these behaviors has to
be obtained by reverse-engineering the processor. Ideally,
the throughput estimator should be able to automatically
capture such intricacies with minimal human intervention.
Speed. A throughput estimator also needs to be fast. Com-
pilers need to search through many code blocks before emit-
ting the fastest version of a given instruction sequence. Run-
ning the basic blocks to get the ground truth throughput
requires sandboxing and many iterations of execution to ar-
rive at a consistent steady-state throughput estimate, which
is impractical for real-time systems.
1.1 Ithemal: A Data Driven Approach
In this paper we introduce Ithemal (Instruction THroughput
Estimator using MAchine Learning), which takes a novel
data–driven approach to predicting throughput for a block
of instructions, inspired by recent advances in Deep Neural
Networks (DNNs). Ithemal models the throughput estima-
tion problem as a regression task and leverages a DNN
to learn to predict throughput by using a large corpus of
labeled data, mapping assembly sequences to real valued
throughputs. More concretely, Ithemal uses a hierarchical
multiscale RNN (El Hihi & Bengio, 1995; Chung et al.,
2017; Baraldi et al., 2017), which generates an independent
embedding for each instruction, then sequentially combines
the instruction embeddings to predict throughput.
We show that Ithemal’s learned model is signiﬁcantly more
accurate than the analytical models, dropping the mean abso-
lute percent error by more than 50% across all benchmarks,
while still delivering fast estimation speeds.
To generate high-quality predictions, Ithemal needs only
training data and a speciﬁcation of the ISA, including the
speciﬁcation of instructions and their explicit and implicit
operands (for instance, the instruction push rax in x86-
64 pushes the registerrax on to the stack and alsoimplicitly
modiﬁes the stack pointer register, rsp). Unlike analytical
models, Ithemal learns any salient microarchitectural details
that contribute to throughput on its own, without any explicit
speciﬁcation or modeling.
In this paper, we present the following contributions:
• Data–Driven Throughput Estimation. We present
Ithemal, the ﬁrst system for data-driven basic block
throughput estimation, using a hierarchical multiscale
RNN.
• Evaluation: We demonstrate that Ithemal is more ac-
curate than the state-of-the-art analytical throughput
estimators, while still attaining fast estimation speeds.
We also show that Ithemal’s design is portable across
multiple processor microarchitectures.
• DNN Architecture Exploration. We demonstrate
that the proposed hierarchical multiscale RNN out-
performs many other choices, including other neural
network architectures speciﬁcally tailored to mimic
the dependency patterns of instructions within a basic
block.
• Open Source Implementation. We have open-
sourced our implementation of Ithemal at https:
//github.com/psg-mit/Ithemal in the hope
that performance engineers and compiler designers can
use and improve upon our approach.
Ithemal demonstrates that future systems can leverage data–
driven techniques to either augment or fully replace manu-
ally developed throughput estimators.
2 Motivating Examples
Analytical modeling is difﬁcult for many code sequences;
consider examples (a)-(c), their actual throughput3, and their
associated throughput predictions in Table 1. These ﬂawed
predictions occur in spite of many hours spent engineering
detailed models of underlying microarchitectural details. In
contrast, Ithemal’s data driven approach intrinsically learns
accurate predictions from the ground truth data.
shl  rbx, 0x02 
mov  rdi, rbx
mov    [rbp+0x70], rax
mov    rax, 0x01
vxorps xmm0, 
    xmm0,xmm0
Ithemal 35 83102
50
32
IACA
83
100
(a) 
24
(c)
Actual
96
103
84
llvm-mca 100
(b)
Table 1.Example x86-64 assembly code sequences (Intel syntax)
and associated throughput predictions, in clock cycles
Implementation Errors: Intel provides extensive docu-
mentation of its microarchitectural implementation that en-
ables developers to build performance models for assembly
3Note that – following convention – we deﬁne throughput to
be the number of clock cycles taken to execute a basic block; this
is actually the reciprocal of the standard deﬁnition of throughput.
We also report throughput for 100 iterations of a given basic block.
Ithemal: Basic Block Throughput Prediction
h∅
Vmov V<S> VCONST V<D> V<E>Vecx
LSTM
LSTM LSTM LSTM LSTM LSTM
Token 
Layer
Instruction
Layer
hblockPrediction
Layer
87.35
Throughput 
Prediction
LSTM 
hmov
LSTM
hadd
mov <S> CONST <D> ecx <E>)(
h∅
mov  ecx, 0x02 add  ebx, ecx
h∅
Vadd V<S> Vebx Vecx VebxV<D>
LSTM LSTM LSTM LSTM LSTMLSTM 
add <S> ebx ecx <D> ebx )(
LSTM
V<E>
<E>
Token Embedding Lookup Table
×
Canonicalization
Figure 1.Ithemal System Architecture
code. However, the sheer volume of implementation details
makes it challenging to deliver a complete model.
Sequence (a) shows a single instruction sequence that ze-
ros out the vector register xmm0. Zeroing out registers is
so common that Intel processors execute these instructions
using a faster, optimized data path – separate from the nor-
mal instruction execution path. IACA closely predicts the
measured value but llvm-mca’s predictions are much farther
off, because it does not model this optimization.
Sequence (b) shows a pair of mov instructions with a mea-
sured throughput of 103. IACA and LLVM both ﬁnd an
execution schedule which would predict a throughput of
100 cycles; however, IACA also identiﬁes a micro-op fusion
opportunity, and therefore predicts84 cycles. This optimiza-
tion opportunity does not manifest in the observed timing
numbers.
Ithemal, which is driven by actual performance data, learns
to closely predict both of these values without any explicit
error-prone encoding of Intel’s optimizations.
Vendor Documentation Errors: The sheer volume of
implementation details also means that Intel’s documenta-
tion can be incorrect. Tools that faithfully adhere to the
documentation can therefore still be incorrect. Sequence (c)
is a short sequence with a data dependency that is bypassed
within the processor pipeline: the mov instruction does not
consume many additional clock cycles over that of shl.
The throughput of this basic block is therefore dominated by
the throughput of shl. However, the throughput value that
Intel provides in its documentation (50 cycles) assumes that
there are no dependencies. Therefore, while IACA – Intel’s
own tool – closely predicts the value, llvm-mca is incorrect
because it uses the dependency-free throughput value. In
comparison, Ithemal closely predicts the actual throughput
because it works with actual performance data.
3 Model Architecture
Figure 1 presents the high-level design of Ithemal’s ap-
proach. We model the problem of throughput estimation
as a regression problem: given the assembly input, Ithe-
mal predicts the throughput of the instruction sequence as a
real-valued number. At the core of Ithemal is a hierarchical
multiscale RNN (Shuai et al., 2015; Zhu et al., 2016) that
sequentially processes all instructions in the basic block and
outputs an embedding, which Ithemal then uses to directly
estimate the throughput. Altogether, we decompose the end-
to-end model into the following stages: canonicalization,
embedding and estimation.
3.1 Canonicalization
The canonicalization stage converts the assembly input into
a more structured form, dictated by the syntax of the as-
sembly instructions. Ithemal takes a compiled assembly
block, disassembles it, and maps it to a list of instructions.
Each instruction consists of a list of tokens representing its
operation code (opcode, e.g. add), source operands, and
destination operands, separated by distinguished delimiter
tokens.
For example, consider the instruction mul ecx, which
multiplies the value in register ecx with eax, and places
the result into registers edx and eax. Note that the source
operand eax and both of the destination operands eax and
edx are implicit in the Intel syntax mul ecx. The ﬁnal
canonicalized set of tokens for the instruction is:
(mul, <S>, eax, ecx, <D>, edx, eax, <E>)
where the bracketed tokens are the delimiters represent-
ing the break between the opcode, source, and destination
operands.
Assembly code permits more than just register operands,
such as constants and memory operands. We map all con-
stants (e.g. integer constants, memory addresses, etc.) to
a single CONST token. We demarcate memory operands
(consisting of a base address, and an optional offset and
displacement) by surrounding them with <M> and </M> de-
limiter tokens. We present the full canonicalization scheme
in Appendix A.
Ithemal: Basic Block Throughput Prediction
3.2 Embedding
Ithemal’s embedding stage takes a canonicalized token
stream of instructions, and for each instruction produces
an embedding: a representation of an instruction as a real-
valued vector in a high-dimensional space. The ﬁrst step
is the token layer, which maps a given token to an embed-
ding. We implement the token layer by mapping each token
in the sequence to an n-dimensional vector by learning a
linear transformation of the one-hot token vectors (this is
equivalent to learning a lookup table).
Ithemal then maps the sequence of token embeddings to
an embedding for each instruction in the basic block. We
call this the instruction layer. Because each instruction can
have a variable number of tokens depending on its number
of source and destination operands, the size of the input to
the embedding stage is variable. We therefore implement
the instruction layer with a sequential Recurrent Neural
Network (RNN) architecture with Long Short Term Memory
(LSTM) (Hochreiter & Schmidhuber, 1997) cells.
Figure 1 presents the operation of our RNN-based instruc-
tion embedding approach on a small example. The bottom-
most row shows the original assembly input. The second
row shows the sequence of tokens for each instruction. The
third row (the token layer) shows the sequence of token em-
beddings, e.g. vmov, which are mapped directly from each
syntactic token. The fourth row (the instruction layer) shows
the application of an LSTM to reduce the token embedding
sequence into the ﬁnal instruction embedding, hmov.
3.3 Prediction
The ﬁnal prediction comes from the prediction layer, which
maps a basic block (a sequence of instruction embeddings)
to a throughput value. This is again implemented with an
RNN with LSTM cells, which has entirely disjoint weights
from the LSTM in the instruction layer. This corresponds to
the topmost layer in Figure 1. Using the ﬁnal output from
the instruction LSTM (hblock), Ithemal predicts the basic
block’s throughput with a linear layer. Speciﬁcally, Ithemal
computes w ·hblock + b, where w is a learned weight vector
and b is a bias. This produces a ﬁnal real-valued number
that represents the network’s throughput prediction.
The hierarchical combination of the RNN in the instruc-
tion layer and the RNN in the prediction layer has several
beneﬁts over a non-hierarchical model:
• Memory and backpropagation paths are signiﬁcantly
shorter than a model using a non-hierarchical (i.e., sin-
gle) RNN. The average length of a block in our dataset
is 6.04 instructions, and the average length of an in-
struction is 7.97 tokens. The average length of a token-
level RNN across the entire basic block would instead
be about 48 RNN cell applications, more than three
times as long as a path through the hierarchical RNN.
• Instructions are embedded atomically: the prediction
layer is only able to generate throughput estimates at
speciﬁc points in the overall token stream, i.e. after
complete instructions. This means that the network
does not have an obligation to produce states that corre-
spond to predictions at points in between instructions.
We compare this hierarchical architecture against other ar-
chitecture choices in Section 6, showing the efﬁcacy of
Ithemal’s hierarchical architecture.
4 Data and Training
We collected a dataset of basic blocks from well-known pro-
grams and benchmark suites, and timed them with a proce-
dure that matches the assumptions of the baseline analytical
models. We then train Ithemal using standard supervised
learning techniques.
4.1 Dataset
Table 2 summarizes the set of applications in our dataset.
We designed the dataset to include a diverse set of applica-
tions with different performance characteristics while cov-
ering a wide range of x86-64 instructions. It consists of
performance critical applications used for benchmarking
compiler optimizations as well as end user applications used
in day-to-day computing.
To extract each application’s basic blocks, we ﬁrst compile
each application using GCC 4.9.4 with the -O3 optimiza-
tion level targeting an Intel Haswell processor. Next, we
use Dynamorio (Bruening et al., 2012), a dynamic binary
instrumentation tool, to dump the encoded bytes of the ex-
ecuted x86-64 basic blocks. We execute the benchmarks
using the standard inputs provided by the benchmark suites.
Next, we de-duplicate the dataset by removing basic blocks
with same encoded byte patterns. This step is important to
eliminate repeated occurrences of basic blocks created by
code shared through common header ﬁles and by common
compilation patterns.
4.2 Throughput Proﬁling
IACA and llvm-mca predict the steady-state throughput of a
basic block, under the assumptions that all memory accesses
result in L1 cache hits and that the execution environment
is non-preemptive. To collect compatible throughput num-
bers, we proﬁle the execution of a loop that executes each
basic block in isolation 100 times (enough to reach the
steady-state behavior; 100 iterations is also the default value
used by llvm-mca). We measure throughput in terms of
clock cycles using a script that we have developed that is
similar to Agner Fog’s timing script4. Agner Fog’s timing
4https://www.agner.org/optimize/testp.zip
Ithemal: Basic Block Throughput Prediction
Benchmark suite Description #Total
Blocks
#Unique
Blocks
Linux Shared Libraries linux loaders, standard library and other utilities 313846 103977
SPEC2006 (SPEC, 2006)
benchmark suite with compilers, chess engines, video compres-
sion and various simulation applications. Commonly used for
benchmarking compilers
247047 141051
SPEC2017 (SPEC, 2017) similar to SPEC2006, but with a larger variety 616899 234588
NAS (NASA, 1991–2014) benchmarks with stencil computations (dense loops) 3935 1813
polybench-3.1 (Pouchet, 2012) polyhedral compilation test suite (dense loops) 1900 859
TSVC (Maleki et al., 2011) suite for testing compiler auto-vectorization 5129 2350
cortexsuite (Venkata et al., 2009) computer vision workloads including neural networks 6582 3968
simd (Ihar et al., 2018) heavily hand vectorized image processing library (exposes lot of
SSE2, A VX, A VX2 variants) 212544 25462
compilers/interpreters clang (Lattner & Adve, 2004) and different versions of python
(2.7,3.5) 2746275 924663
end user applications gimp ﬁlters, ﬁrefox, open-ofﬁce, rhythmbox, etc. 83555 35513
Full Dataset 4237712 1416473
Table 2.Composition of the dataset for Haswell, showing the total number of basic blocks per benchmark as well as the unique number of
blocks after de-duplicating repeated blocks on a per-benchmark basis. Note that the #Unique Blocks on the full dataset is not equal to the
sum of unique blocks per individual benchmark, as we de-duplicate across all benchmarks.
script is commonly used for validating individual instruc-
tion throughputs. Our timing script additionally ensures
that almost all memory accesses have a L1 cache hit. Addi-
tionally, we measure L1 instruction and data cache misses
and software context switches to detect and ﬁlter out invalid
executions that do not conform to the assumptions made by
IACA and llvm-mca in their predictions.
Using this methodology, we collected valid throughput val-
ues for the Intel Ivy Bridge (Intel(R) Xeon(R) CPU E5-2695
v2), Haswell (Intel(R) Xeon(R) CPU E5-2680 v3) and Sky-
lake (Intel(R) Xeon(R) W-2123 CPU) microarchitectures.
Data collection takes approximately 4-5 days for each mi-
croarchitecture. Table 2 shows the breakdown of basic block
counts for each benchmark for Haswell microarchitecture
in total as well as after de-duplicating repeated basic blocks
on a per benchmark basis. The ﬁnal Haswell dataset, which
is de-duplicated across benchmarks, constitutes 1,416,473
unique basic blocks.
4.3 Training and Methodology
We implemented our neural network model in PyTorch
(0.4.0a0+59bda9a). The learnable parameters in Ithemal
include the token embeddings, the token LSTM and instruc-
tion LSTM parameters, and the afﬁne coefﬁcients in the
ﬁnal linear layer. For our loss function we use a normalized
error metric, based on the L1 norm:
L(pred, actual) = |pred −actual|
actual
We randomly assign 80% of the collected blocks to the train
set and 20% to the test set. We use Asynchronous Stochastic
Gradient Descent (Robbins & Monro, 1951; Niu et al., 2011)
to train the model. Our full training regime is detailed in
Appendix B.
5 Evaluation
We have evaluated Ithemal against two state-of-the-art, hand-
written analytical models: IACA (Intel, 2017) (v3.0-28-
g1ba2cbb) and llvm-mca (Di Biagio & Davis, 2018) (LLVM
8.0.0). Both of these models are designed to model the
complexities of modern processors (including pipelining,
superscalar, and out-of-order units). We show that our data–
driven model beats the accuracy of these sophisticated hand-
written models (Section 5.1) while maintaining just as fast
prediction speeds (Section 5.2). Further, we show that our
approach is portable across different microarchitectures in
Section 5.3 by showing that Ithemal learns a model that out-
performs IACA and llvm-mca without any neural network
architecture or hyperparameter modiﬁcations.
5.1 Accuracy
We evaluate the accuracy of each model against the actual
throughput values for Intel’s Haswell, Ivy Bridge, and Sky-
lake microarchitectures. The version of IACA we use does
not support throughput estimation for Ivy Bridge; we there-
fore evaluate accuracy only for Ithemal and llvm-mca for
Ivy Bridge. We prepared datasets for each microarchitecture
according to the methodology described in Section 4.3.
Table 3 presents the results of our accuracy comparison. We
report the average error with respect to the ground truth of
each tool for each microarchitecture. We also report both the
Spearman and Pearson correlation of each tool’s predictions
with the ground truth.
Ithemal is more accurate in its throughput predictions for ba-
sic blocks across all three microarchitectures. Our model’s
predictions are closer to the ground truth than both IACA
and LLVM in 74% of the blocks in the Haswell test set.
Ithemal’s predictions also have a higher correlation with the
ground truth values for both the Spearman (rank correla-
tion) and Pearson (linear correlation) metrics. The higher
Ithemal: Basic Block Throughput Prediction
(a) Ithemal
 (b) llvm-mca
 (c) IACA
Figure 2.Heatmaps for measured and predicted throughput values under different models for basic blocks with measured throughput
values less than 1000 cycles (Haswell)
Spearman correlation is especially useful because it directly
corresponds to higher utility for use within an optimizing
compiler (such as an instruction scheduling pass). Speciﬁ-
cally, compilers typically only need to determine which of
several conﬁgurations of a basic block is the fastest, and do
not calculate each block’s absolute performance.
Figure 2 presents three heatmaps relating actual and pre-
dicted values in for basic blocks with throughputs less than
1000 cycles for each prediction method (representing 95%
of our dataset).
To generate each heatmap, we binned the actual and pre-
dicted data into axis-aligned bins of width and height 20
cycles. The color in each bin represents the count of blocks
in that bin. A perfect estimator would have all points along
the line y = x (shown as a faint grey, dashed line on the
heatmaps), since the predicted throughputs would always
match the measured throughputs. We see a higher density
near the identity line for Ithemal, compared to both llvm-
mca and IACA. Both llvm-mca and IACA also have more
horizontal banding, representing more predictions of the
same throughput value for different blocks that do actually
have different behaviors.
Figure 3 shows the average error of each system across a
range of throughputs. Compared to llvm-mca and IACA,
Ithemal is better for blocks of almost all sizes. All estima-
tors struggle with blocks with measured throughputs just
below peaks in the measured throughput distribution. We
hypothesize that these blocks correspond to special microar-
chitectural optimizations that llvm-mca and IACA do not
model. Ithemal also struggles some with these rare blocks,
but still outperforms both analytical estimators.
5.2 Speed
Table 4 presents the results of our evaluation of estimator
throughput: the number of instructions able to be timed per
second for each estimator. We calculate this by measuring
Micro-
architecture
Method Error Spearman
Correlation
Pearson
Corr.
Ivy Bridge llvm-mca 0.181 0.902 0.777
Ithemal 0.089 0.955 0.913
Haswell llvm-mca 0.200 0.890 0.790
IACA 0.209 0.917 0.833
Ithemal 0.089 0.960 0.918
Skylake llvm-mca 0.239 0.852 0.729
IACA 0.167 0.926 0.835
Ithemal 0.079 0.960 0.895
Table 3.Average error for different models and microarchitectures
Figure 3.Average error across throughputs for different estimation
methods for basic blocks with throughput values less than 1000
cycles (Haswell)
Ithemal: Basic Block Throughput Prediction
the number of basic blocks each tool can time per second,
and multiplying that by the average number of instructions
per basic block. In the last row, we also show the corre-
sponding estimator throughput if we instead measure the
ground-truth throughput for a given block. We measured
these estimator throughputs on the Haswell test set on a
machine with an Intel Xeon E5-2680 CPU.
Ithemal is as fast as llvm-mca and IACA in our measure-
ments, and is signiﬁcantly faster than empirical evaluation
of basic blocks. It is worth noting that llvm-mca and IACA
can both also output diagnostic information about basic
blocks, and also that empirical evaluation of ground-truth
data could be sped up by running fewer repeated measure-
ments (it may be possible that as few as 2 or 3 measurements
would sufﬁce in some contexts). However even with these
qualiﬁcations, we show that Ithemal functions as an equiva-
lently performant and more accurate drop-in replacement for
llvm-mca and IACA in systems which only need throughput
estimations, while still performing signiﬁcantly faster than
empirical evaluation.
Method Throughput (Instructions / second)
llvm-mca 492
IACA 541
Ithemal 560
Empirical execution 13
Table 4.Estimation throughputs for different estimators measured
in instructions per second
5.3 Portability
We designed and trained Ithemal on Haswell and validated
our architecture and hyperparameters by re-training on Sky-
lake. Without any changes to its structure or training regime,
we then trained and evaluated Ithemal on the Ivy Bridge
dataset. Table 3 summarizes the average errors for each
microarchitecture. Ithemal learns to estimate throughput
values for each microarchitecture with a maximum average
error of 0.089 across all datasets. The hand-written models
exhibit a minimum average error of 0.167.
In sum, Ithemal provides state-of-the-art prediction perfor-
mance; its results beat the baselines across the board. More-
over, Ithemal does so without requiring a user to provide
information about the processor’s underlying microarchi-
tecture, whereas these analytical models require signiﬁcant
re-engineering for each microarchitecture of interest.
6 Neural Network Architecture Exploration
We evaluated a number of neural network architectures with
varying levels of structure and complexity before arriving at
Ithemal’s network architecture (Section 3).
Figure 4 shows a DAG-RNN (Shuai et al., 2015; Zhu et al.,
2016). Instructions are embedded identically as to in Ithe-
mal (i.e. the token layer and instruction layer remain the
LSTM
LSTM
LSTMmax
hblock 
87.35
h∅
h∅
hmov
hadd LSTM
max
hsub
hmul
mov  ecx, 0x02
add  eax, ebx
sub  edx, eax
mul  ecx
x86-64 assembly
×
Figure 4.The DAG-RNN Architecture
same). However rather than running an RNN sequentially
over the instructions in the prediction layer, the DAG-RNN
constructs a dependence graph of the instructions in the ba-
sic block, with a directed edge between a pair of instructions
if the second instruction depends on an output of the ﬁrst
instruction. Because an instruction can depend on multi-
ple previous instructions, we apply an element-wise max
to reduce the states feeding in to a given instruction. Then,
the DAG-RNN applies an LSTM cell, using the generated
instruction embedding as the input, and the result of the
element-wise max as the input state. To generate the ﬁnal
prediction, we take an element-wise max of the output states
of all leaf instructions (all instructions with no dependents),
and pass the result through a linear layer.
The DAG-RNN is inspired by the theoretical behavior of
a perfect out-of-order processor: the throughput of a basic
block running on a perfect out-of-order processor is equiv-
alent to the throughput of the longest path that must be
serially executed in that basic block. Using a DAG-RNN
implicitly encodes this prior, by only allowing information
to propagate through the paths that must be serially executed
in the block.
We also tested a simple token-level RNN with LSTM cells,
which has a similar base architecture as Ithemal, but with-
out the topmost prediction layer. Instead, this model se-
quentially consumes all tokens in a basic block making no
explicit distinction between instructions, giving a baseline
measure for the efﬁcacy of Ithemal’s hierarchical model.
The full architecture diagram for the token-level RNN is
shown in Appendix E.
Results. Figure 5 shows the training and validation loss
for each model across the ﬁrst ﬁve epochs. It is clear that
the hierarchical LSTM is the best model among these three.
The sequential LSTM performs the worst by far, motivating
the need to process individual tokens and instructions at mul-
tiple scales. The fact that the DAG-RNN performs worse
than the hierarchical LSTM implies that the exact ordering
of instructions in a basic block does matter, not just the
dependency chains. This aligns with the fact that instruction
scheduling optimizations in compilers do result in changed
performance, despite the underlying dependency graph be-
ing the same. While the perfect out-of-order execution
Ithemal: Basic Block Throughput Prediction
Figure 5.Learning curves for the models
model is a reasonable approximation, modern processors do
in fact have some serial behavior, which a sequential model
is able to capture.
7 Related Work
Hierarchical Multiscale RNNs Hierarchical RNNs are a
classic technique in sequence prediction domains (El Hihi &
Bengio, 1995). Ithemal uses a hierarchical multiscale RNN
as the core of its prediction methodology, similar to many
other proposed hierarchical RNN models. Ithemal has some
inherent advantages over other models from the literature:
as opposed to the methodology proposed in (Chung et al.,
2017), the hierarchical structure in instruction embedding
and throughput prediction is explicit rather than latent. Ithe-
mal’s structure is instead most similar to boundary-aware
hierarchical RNNs, such as the one presented in (Baraldi
et al., 2017) for video captioning based on a hierarchy of
frames and scenes.
DAG-RNNs and Graph Neural Networks Neural net-
works with generic graph based structures have been used
in NLP tasks to model relations among words in sentences
(Peng et al., 2017; Dhingra et al., 2017). Programs also can
be represented using Gated Graph Neural Networks (Al-
lamanis et al., 2018), to perform high-level tasks such as
variable naming, or identifying variable misuses. Xu et al.
(2017) uses graph neural networks to ﬁnd binary similarity
between different execution platforms. In experimenting
with a DAG-RNN (Shuai et al., 2015; Zhu et al., 2016), we
hoped to be able to take advantage of the program semantics
of a basic block, although that approach does not perform
as well in our collected datasets.
Analytical Models for Throughput and Runtime Esti-
mation Apart from state-of-the-art tools like llvm-mca
and IACA, other analytical models exist for throughput
estimation (Taha & Wills, 2003) of instructions. OS-
ACA (Laukemann et al., 2018) is an open source analyt-
ical model similar to llvm-mca and IACA, which auto-
mates some of the collection of the tabular data which is
plugged into the model. There are also analytical models
such as (Chen & Aamodt, 2009) to estimate throughput for
multithreaded programs. Cycle–accurate simulators such
as ZSim (Sanchez & Kozyrakis, 2013) and Marss (Patel
et al., 2011) have a high start-up cost and are more suited
for coarse grained simulations.
Coarser analytical models exist for predicting program run-
times (Park, 1993). Work has also been done in develop-
ing analytical models to predict performance of restricted
classes of programs. For example, work on predicting
parallel program runtimes include (Rugina & Schauser,
1998; Adve & Vernon, 2004; Hartleb & Mertsiotakis, 1992;
Blanco et al., 2004; Fahringer & Zima, 1993) and work on
predicting worst case execution times include (Ferdinand
et al., 2001; Li et al., 2007).
All of these models require detailed processor modeling and
considerable human development effort.
Learned Models for Throughput and Runtime Estima-
tion There has been work on developing machine learning–
based models for absolute and relative runtime estimation.
(Huang et al., 2010) introduces sparse polynomial regres-
sion to predict execution time of programs by using a set
of hand–crafted features of high level programs. Dubach
et al. (2007) uses neural networks with hand-crafted features
to estimate the speedup between two code sequences. Ga-
meTime (Seshia & Kotker, 2011; Seshia & Rakhlin, 2012)
uses SMT solvers to generate inputs and game theoretic ap-
proaches to predict the distribution of runtimes of programs.
These models require manual feature engineering, and run-
time predictions are done at a coarser granularity (e.g. at
the full program level). In contrast, Ithemal automatically
learns how to predict throughput of basic blocks with mini-
mal architectural knowledge embedded into the model.
Microarchitectural Predictions Similar to basic block
throughput estimation, various microarchitectural predic-
tion tasks have been explored with machine learning. For
example, RNN models can be used for predicting memory
access (Hashemi et al., 2018), and perceptron models can
be used for branch prediction (Jimenez & Lin, 2001)
8 Conclusion
We present Ithemal, a data–driven system for basic block
throughput estimation. Ithemal’s accuracy surpasses that of
state-of-the-art, hand-written analytical models; it achieves
its accuracy by leveraging a deep neural network designed to
capture the behavior of modern processors. Ithemal demon-
strates that future compilation and performance engineering
tools can be augmented with data–driven approaches to im-
prove their performance and portability, while minimizing
developer effort.
Ithemal: Basic Block Throughput Prediction
Acknowledgments
We would like to thank Yishen Chen, Ajay Brahmaksha-
triya for their help in creating the datasets, and all review-
ers for insightful comments and suggestions. We would
also like to thank Ondˇrej Sýkora and Jon Orwant for their
many helpful discussions. This research was supported by
DARPA D3M Award #FA8750-17-2-0126, NSF Grant CCF-
1751011, DARPA/Nvidia Symphony Award #HR0011-18-3-
0007 and DARPA Award #HR001118C0059. Any opinions,
ﬁndings, and conclusions or recommendations expressed in
this material are those of the authors and do not necessarily
reﬂect the views of the funding agencies.
References
Adve, V . S. and Vernon, M. K. Parallel program perfor-
mance prediction using deterministic task graph analy-
sis. ACM Trans. Comput. Syst., 22(1):94–136, February
2004. ISSN 0734-2071. doi: 10.1145/966785.966788.
URL http://doi.acm.org/10.1145/966785.
966788.
Allamanis, M., Brockschmidt, M., and Khademi, M.
Learning to represent programs with graphs. In In-
ternational Conference on Learning Representations,
2018. URL https://openreview.net/forum?
id=BJOFETxR-.
Baraldi, L., Grana, C., and Cucchiara, R. Hierarchical
boundary-aware neural encoder for video captioning.
CVPR’17, 2017.
Blanco, V ., González, J. A., León, C., Rodrıguez, C., Ro-
drıguez, G., and Printista, M. Predicting the performance
of parallel programs. Parallel Computing, 30(3):337–356,
2004.
Bruening, D., Zhao, Q., and Amarasinghe, S. Transpar-
ent dynamic instrumentation. In Proceedings of the 8th
ACM SIGPLAN/SIGOPS Conference on Virtual Execu-
tion Environments, VEE ’12, pp. 133–144, New York,
NY , USA, 2012. ACM. ISBN 978-1-4503-1176-2. doi:
10.1145/2151024.2151043. URL http://doi.acm.
org/10.1145/2151024.2151043.
Chen, X. E. and Aamodt, T. M. A ﬁrst-order ﬁne-grained
multithreaded throughput model. In HPCA-15 2009.
IEEE 15th International Symposium on High Perfor-
mance Computer Architecture, pp. 329–340. IEEE, 2009.
Chung, J., Ahn, S., and Bengio, Y . Hierarchical multiscale
recurrent neural networks. ICLR’17, 2017.
Dhingra, B., Yang, Z., Cohen, W. W., and Salakhutdinov, R.
Linguistic Knowledge as Memory for Recurrent Neural
Networks. ArXiv e-prints, March 2017.
Di Biagio, A. and Davis, M. llvm-mca, 2018.
URL https://lists.llvm.org/pipermail/
llvm-dev/2018-March/121490.html.
Dubach, C., Cavazos, J., Franke, B., Fursin, G., O’Boyle,
M. F., and Temam, O. Fast compiler optimisation
evaluation using code-feature based performance pre-
diction. In Proceedings of the 4th International Con-
ference on Computing Frontiers, CF ’07, pp. 131–142,
New York, NY , USA, 2007. ACM. ISBN 978-1-59593-
683-7. doi: 10.1145/1242531.1242553. URL http:
//doi.acm.org/10.1145/1242531.1242553.
El Hihi, S. and Bengio, Y . Hierarchical recurrent neural
networks for long-term dependencies. In Proceedings of
the 8th International Conference on Neural Information
Processing Systems, NIPS’95, pp. 493–499, Cambridge,
MA, USA, 1995. MIT Press. URL http://dl.acm.
org/citation.cfm?id=2998828.2998898.
Fahringer, T. and Zima, H. P. A static parameter based
performance prediction tool for parallel programs. In
Proceedings of the 7th international conference on Super-
computing, pp. 207–219. ACM, 1993.
Ferdinand, C., Heckmann, R., Langenbach, M., Martin, F.,
Schmidt, M., Theiling, H., Thesing, S., and Wilhelm,
R. Reliable and precise wcet determination for a real-
life processor. In International Workshop on Embedded
Software, pp. 469–485. Springer, 2001.
Hartleb, F. and Mertsiotakis, V . Bounds for the mean run-
time of parallel programs. In Proceedings of the Sixth
International Conference on Modelling Techniques and
Tools for Computer Performance Evaluation, volume 92,
pp. 197–210, 1992.
Hashemi, M., Swersky, K., Smith, J. A., Ayers, G., Litz, H.,
Chang, J., Kozyrakis, C., and Ranganathan, P. Learning
memory access patterns. In Proceedings of the 35th Inter-
national Conference on Machine Learning, ICML 2018,
Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018,
pp. 1924–1933, 2018. URL http://proceedings.
mlr.press/v80/hashemi18a.html.
Hochreiter, S. and Schmidhuber, J. Long short-term
memory. Neural Comput., 9(8):1735–1780, Novem-
ber 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.
9.8.1735. URL http://dx.doi.org/10.1162/
neco.1997.9.8.1735.
Huang, L., Jia, J., Yu, B., gon Chun, B., Maniatis, P., and
Naik, M. Predicting execution time of computer pro-
grams using sparse polynomial regression. In Lafferty,
J. D., Williams, C. K. I., Shawe-Taylor, J., Zemel, R. S.,
and Culotta, A. (eds.), Advances in Neural Information
Processing Systems 23, pp. 883–891. Curran Associates,
Inc., 2010.
Ithemal: Basic Block Throughput Prediction
Ihar, Y ., Mikhail, A., Andrey, R., Fedorov, D., and
Matsaberydze, K. Simd library for image process-
ing, 2018. URL http://ermig1979.github.io/
Simd/index.html.
Intel. Intel architecture code analyzer, 2017. URL https:
//software.intel.com/en-us/articles/
intel-architecture-code-analyzer .
Jimenez, D. A. and Lin, C. Dynamic branch prediction with
perceptrons. In Proceedings HPCA Seventh International
Symposium on High-Performance Computer Architec-
ture, pp. 197–206, Jan 2001. doi: 10.1109/HPCA.2001.
903263.
Lattner, C. and Adve, V . Llvm: A compilation framework
for lifelong program analysis & transformation. In Pro-
ceedings of the International Symposium on Code Gen-
eration and Optimization: Feedback-directed and Run-
time Optimization, CGO ’04, pp. 75–, Washington, DC,
USA, 2004. IEEE Computer Society. ISBN 0-7695-2102-
9. URL http://dl.acm.org/citation.cfm?
id=977395.977673.
Laukemann, J., Hammer, J., Hofmann, J., Hager, G., and
Wellein, G. Automated instruction stream throughput
prediction for intel and amd microarchitectures. In
2018 IEEE/ACM Performance Modeling, Benchmarking
and Simulation of High Performance Computer Systems
(PMBS), pp. 121–131, Nov 2018. doi: 10.1109/PMBS.
2018.8641578.
Li, X., Liang, Y ., Mitra, T., and Roychoudhury,
A. Chronos: A timing analyzer for embedded
software. Science of Computer Programming ,
69(1):56 – 67, 2007. ISSN 0167-6423. doi:
https://doi.org/10.1016/j.scico.2007.01.014. URL
http://www.sciencedirect.com/science/
article/pii/S0167642307001633. Special
issue on Experimental Software and Toolkits.
Lozano, R. C., Carlsson, M., Drejhammar, F., and Schulte,
C. Constraint-based register allocation and instruction
scheduling. In Milano, M. (ed.), Principles and Prac-
tice of Constraint Programming, pp. 750–766, Berlin,
Heidelberg, 2012. Springer Berlin Heidelberg. ISBN
978-3-642-33558-7.
Maleki, S., Gao, Y ., Garzar, M. J., Wong, T., Padua, D. A.,
et al. An evaluation of vectorizing compilers. In Parallel
Architectures and Compilation Techniques (PACT), 2011
International Conference on, pp. 372–382. IEEE, 2011.
McGovern, A. and Moss, E. Scheduling straight-line code
using reinforcement learning and rollouts. In Proceed-
ings of the 1998 Conference on Advances in Neural
Information Processing Systems II, pp. 903–909, Cam-
bridge, MA, USA, 1999. MIT Press. ISBN 0-262-11245-
0. URL http://dl.acm.org/citation.cfm?
id=340534.340836.
NASA, A. S. D. Nas c benchmark suite 3.0,
1991–2014. URL https://github.com/
benchmark-subsetting/NPB3.0-omp-C/ .
Niu, F., Recht, B., Re, C., and Wright, S. J. Hogwild!:
A lock-free approach to parallelizing stochastic gradi-
ent descent. In Proceedings of the 24th International
Conference on Neural Information Processing Systems,
NIPS’11, pp. 693–701, USA, 2011. Curran Associates
Inc. ISBN 978-1-61839-599-3. URL http://dl.acm.
org/citation.cfm?id=2986459.2986537.
Park, C. Y . Predicting program execution times by analyzing
static and dynamic program paths. Real-Time Systems, 5
(1):31–62, 1993.
Patel, A., Afram, F., Chen, S., and Ghose, K. Marss:
a full system simulator for multicore x86 cpus. In
Design Automation Conference (DAC), 2011 48th
ACM/EDAC/IEEE, pp. 1050–1055. IEEE, 2011.
Peng, N., Poon, H., Quirk, C., Toutanova, K., and Yih,
W.-t. Cross-sentence n-ary relation extraction with
graph lstms. Transactions of the Association for Com-
putational Linguistics, 5:101–115, 2017. ISSN 2307-
387X. URL https://www.transacl.org/ojs/
index.php/tacl/article/view/1028.
Pouchet, L.-N. The polyhedral benchmark suite,
2012. URL http://web.cse.ohio-state.
edu/~pouchet.2/software/polybench/
polybench.html.
Robbins, H. and Monro, S. A stochastic approximation
method. The Annals of Mathematical Statistics, 22(3):
400–407, 1951. ISSN 00034851. URL http://www.
jstor.org/stable/2236626.
Rugina, R. and Schauser, K. Predicting the running times of
parallel programs by simulation. In ipps, pp. 0654. IEEE,
1998.
Sanchez, D. and Kozyrakis, C. Zsim: Fast and accurate mi-
croarchitectural simulation of thousand-core systems. In
ACM SIGARCH Computer architecture news, volume 41,
pp. 475–486. ACM, 2013.
Seshia, S. A. and Kotker, J. GameTime: A toolkit for
timing analysis of software. In Proceedings of Tools and
Algorithms for the Construction and Analysis of Systems
(TACAS), pp. 388–392, March 2011.
Ithemal: Basic Block Throughput Prediction
Seshia, S. A. and Rakhlin, A. Quantitative analysis of sys-
tems using game-theoretic learning. ACM Transactions
on Embedded Computing Systems (TECS), 11(S2):55,
2012.
Shuai, B., Zuo, Z., Wang, G., and Wang, B. Dag-
recurrent neural networks for scene labeling. CoRR,
abs/1509.00552, 2015. URL http://arxiv.org/
abs/1509.00552.
SPEC. Spec cpu2006 benchmark suite, 2006. URL https:
//www.spec.org/cpu2006/.
SPEC. Spec cpu2017 benchmark suite, 2017. URL https:
//www.spec.org/cpu2017/.
Stephenson, M., Amarasinghe, S., Martin, M., and O’Reilly,
U.-M. Meta optimization: Improving compiler heuris-
tics with machine learning. In Proceedings of the
ACM SIGPLAN 2003 Conference on Programming Lan-
guage Design and Implementation, PLDI ’03, pp. 77–
90, New York, NY , USA, 2003. ACM. ISBN 1-58113-
662-5. doi: 10.1145/781131.781141. URL http:
//doi.acm.org/10.1145/781131.781141.
Taha, T. M. and Wills, D. S. An instruction throughput
model of superscalar processors. In Rapid Systems Pro-
totyping, 2003. Proceedings. 14th IEEE International
Workshop on, pp. 156–163. IEEE, 2003.
Venkata, S. K., Ahn, I., Jeon, D., Gupta, A., Louie, C., Gar-
cia, S., Belongie, S., and Taylor, M. B. Sd-vbs: The san
diego vision benchmark suite. In2009 IEEE International
Symposium on Workload Characterization (IISWC), pp.
55–64, Oct 2009. doi: 10.1109/IISWC.2009.5306794.
Xu, X., Liu, C., Feng, Q., Yin, H., Song, L., and Song,
D. Neural network-based graph embedding for cross-
platform binary code similarity detection. In Proceedings
of the 2017 ACM SIGSAC Conference on Computer and
Communications Security, CCS ’17, pp. 363–376, New
York, NY , USA, 2017. ACM. ISBN 978-1-4503-4946-8.
doi: 10.1145/3133956.3134018. URL http://doi.
acm.org/10.1145/3133956.3134018.
Zhu, X., Sobhani, P., and Guo, H. Dag-structured long
short-term memory for semantic compositionality. InPro-
ceedings of the 2016 Conference of the North American
Chapter of the Association for Computational Linguistics:
Human Language Technologies, pp. 917–926, 2016.

COLE: Compiler Optimization Level Exploration
Kenneth Hoste Lieven Eeckhout
ELIS Department, Ghent University
Sint-Pietersnieuwstraat 41, B-9000 Gent, Belgium
{kehoste,leeckhou}@elis.UGent.be
ABSTRACT
Modern compilers implement a large number of optimiza-
tions which all interact in complex ways, and which all have
a diﬀerent impact on code quality, compilation time, code
size, energy consumption, etc. For this reason, compilers
typically provide a limited number of standard optimization
levels, such as -O1, -O2, -O3 and -Os, that combine various
optimizations providing a number of trade-oﬀs between mul-
tiple objective functions (such as code quality, compilation
time and code size). The construction of these optimization
levels, i.e., choosing which optimizations to activate at each
level, is a manual process typically done using high-level
heuristics based on the compiler developer’s experience.
This paper proposes COLE, Compiler Optimization Level
Exploration, a framework for automatically ﬁnding Pareto
optimal optimization levels through multi-objective evolu-
tionary searching. Our experimental results using GCC and
the SPEC CPU benchmarks show that the automatic con-
struction of optimization levels is feasible in practice, and in
addition, yields better optimization levels than GCC’s man-
ually derived ( -Os, -O1, -O2 and -O3) optimization levels,
as well as the optimization levels obtained through random
sampling. We also demonstrate that COLE can be used to
gain insight into the eﬀectiveness of compiler optimizations
as well as to better understand a benchmark’s sensitivity to
compiler optimizations.
Categories and Subject Descriptors
D.3.4 [Programming Languages]: Processors— code gen-
eration, compilers
General Terms
Algorithms, Experimentation, Performance
Keywords
compiler optimization, multi-objective search
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for proﬁt or commercial advantage and that copies
bear this notice and the full citation on the ﬁrst page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior speciﬁc
permission and/or a fee.
CGO’08,April 5–10, 2008, Boston, Massachusetts, USA.
Copyright 2008 ACM 978-1-59593-978-4/08/04 ...$5.00.
1. INTRODUCTION
Modern compilers provide a broad collection of optimiza-
tions. Anticipating the eﬃcacy of these optimizations is
not trivial though. The eﬀect of a compiler optimization
is highly dependent on the code being compiled. In addi-
tion, compiler optimization interactions are complex, and
in many cases counter-intuitive, or at least hard to reason
about. To make things even worse, objective functions, such
as performance, code size, energy consumption and compila-
tion time, may be aﬀected conﬂictingly by diﬀerent compiler
optimizations.
This is a well recognized problem and to facilitate the
end user in determining appropriate compiler optimizations,
compiler developers typically provide their compilers with a
set of standard optimization levels, such as -O1, -O2, -O3,
and -Os. These optimization levels combine various com-
piler optimizations and provide diﬀerent trade-oﬀs in terms
of code quality, compilation time and code size. For users
willing to trade compilation time for high code quality, the
highest level of optimization (e.g.,-O3) may be worth trying.
If on the other hand, compilation time is a concern, more
so than code quality, -O1 may be the optimization level of
choice. Finally, if code size is of primary importance, for
example for embedded applications, then -Os is a suitable
optimization level.
Besides static compilers, also dynamic compilers provide
multiple optimization levels. For example, the Just-In-Time
compilers/optimizers in various Java Virtual Machines [3, 4,
19, 21] can optimize code to a number of optimization levels.
In dynamic compilers, the problem of making an appropriate
trade-oﬀ between code quality and compilation time is even
more pressing than for static compilers, because the time
spent compiling is part of the total execution time.
The construction of optimization levels however is trou-
blesome. The search space is huge given the large number of
compiler optimizations. For example, in the GNU Compiler
Collection (GCC) compiler (which we use in this paper) 60
diﬀerent optimizations are used in the various optimization
levels (-O1, -O2, -O3, -Os); this results in a huge space with
260 (in the order of ≈ 1018) possible optimization levels. An
exhaustive search in such a huge space is obviously infea-
sible, because of the time required to compile and run the
benchmarks for each combination of optimizations.
Therefore, compiler developers heavily rely on their ex-
perience and heuristics in their deﬁnition of optimization
levels. These heuristics typically look like: optimizations
that do not increase compilation time and are likely to pro-
duce good code, should be activated at -O1; optimizations
that tend to increase code size but will likely result in bet-
ter code quality, should be activated at -O2 and disabled at
-Os; and optimizations that typically require a lot of compi-
lation time, and might lead to even better code, should be
activated at -O3. Ishizaki et al. [17] describe such a manual
optimization level selection process for a dynamic Just-In-
Time compiler.
In this paper, we propose Compiler Optimization Level
Exploration (COLE), a method to automatically construct
optimization levels that represent optimal trade-oﬀs between
multiple objective functions, such as performance, compila-
tion time, code size, etc. COLE employs multi-objective
evolutionary searching for identifying Pareto optimal opti-
mization levels — a Pareto optimal solution is a solution
that can not be beaten by another solution along all objec-
tive functions simultaneously. COLE does not only relieve
the compiler developer from the tedious and time-consuming
task of manually building optimization levels, it also (most
likely) yields better performing optimization levels.
In this paper, we make the following contributions:
• To the best of our knowledge, we are the ﬁrst to pro-
pose automated multi-objective compiler optimization
level searching, of which the COLE framework is a
prototype example. In contrast, today’s current prac-
tice is to manually build optimization levels based on
heuristics and compiler developers’ experience and in-
tuition. Similar to COLE, iterative compilation [1, 2,
5, 6, 10, 12, 13, 18, 20, 22, 23, 24] also uses automated
searching for identifying the optimal compiler options
for a given application of interest, however COLE per-
forms multi-objective searching whereas iterative com-
pilation to date is limited to single-objective searching.
• We experimentally demonstrate that the automatic
construction of Pareto optimal compiler optimization
levels is feasible in practice. In addition, using the
GCC compiler and the SPEC CPU2000 benchmarks,
we show that the multi-objective evolutionary search
algorithm proposed in COLE outperforms random search-
ing, as well as GCC’s (manually derived) standard op-
timization levels.
• We analyze the Pareto optimal optimization levels ob-
tained from searching a code quality (execution time)
versus compilation time multi-objective space in order
to gain insight into the importance of the various com-
piler optimizations. For example, we ﬁnd that only
25% of the optimizations used in GCC’s standard op-
timization levels appear in one of the Pareto optimal
optimization levels, and only a handful optimizations
appear in all Pareto optimal optimization levels.
• We use the COLE framework to characterize per-bench-
mark sensitivities with respect to compiler optimiza-
tions. This enables identifying groups of benchmarks
exhibiting similar compiler optimization level sensitiv-
ity, which provides compiler developers and researchers
with a way of picking representative benchmarks for
their research or development project.
The paper is organized as follows. We ﬁrst describe prior
work in Section 2. In Section 3 we propose the COLE frame-
work. Section 4 then describes our experimental setup. We
evaluate the COLE framework in Section 5, and compare it
against random searching and the standard GCC optimiza-
tion levels. Subsequently, Section 6 analyzes the Pareto op-
timal optimization levels in terms of the compiler optimiza-
tions they enable. We employ the COLE framework to study
benchmark optimization level sensitivity in Section 7. And
ﬁnally, we conclude and discuss future work in Section 8.
2. PRIOR WORK
2.1 Iterative compilation
The work closest related to our work is iterative com-
pilation. The basic idea of iterative compilation is to ex-
plore the compiler optimization space by iteratively com-
piling and measuring the eﬀectiveness of optimization se-
quences. Driven by a search algorithm, iterative compila-
tion explores the optimization space, and upon termination
of the search algorithm, the best performing optimization
sequence is reported. A large body of work has been done
on iterative compilation over the past few years, and many
researchers have reported impressive results showing signif-
icant performance, energy or code size improvements over
standard optimization sequences, see for example [1, 2, 5, 6,
9, 10, 12, 13, 15, 18, 20, 22, 23, 24].
An important concern though with iterative compilation
is that searching the optimization space is very time con-
suming. By consequence, the vast majority of the work on
iterative compilation focuses on reducing the search time;
there are basically two ways for doing so. One approach is
to speedup the search process by either pruning the search
space [22], or intelligently navigating through the search
space using heuristic search algorithms, such as genetic al-
gorithms [10, 18] or combination elimination [20]. Another
approach is to reduce the time spent evaluating a design
point during this search. Some researchers propose analyti-
cal modeling for estimating the eﬀect of compiler optimiza-
tions on performance [22, 24]. Others build empirical models
using predictive modeling built from static code features [1,
8, 23] or dynamic code features [6]. Yet others exploit the
phase behavior observed during application execution, and
evaluate diﬀerent optimization sequences in subsequent oc-
currences of the same phase [12].
What all of this prior work on iterative compilation has
in common is that it focuses on a single objective function
to be optimized. For example, researchers typically focus on
a single optimization criterion such as performance [1, 5, 6,
8, 10, 14, 12, 20, 22, 23, 24], or energy consumption [13],
or code size [10]. And some researchers focus on optimizing
a single objective function that combines multiple optimiza-
tion criteria such as code quality and compilation time [7, 8],
or code quality and code size [18]. All of these approaches
optimize a single metric. And this is where the key diﬀer-
ence lies between the prior work on iterative compilation
and the COLE approach described in this paper. COLE
aims at exploring a multi-objective compiler optimization
space, whereas prior work is limited to single-objective opti-
mization. Or, in other words, COLE yields multiple Pareto
optimal points whereas prior work yields a single optimal
point. An additional diﬀerence between existing iterative
compilation work and COLE is that iterative compilation
focuses on optimizing the performance of a single applica-
tion, whereas COLE allows ﬁnding compiler optimization
levels that improve the average performance for a collection
of applications.
objective function 1
objective function 2
compiler opt level
Pareto optimal
compiler opt level
Pareto frontier
Figure 1: The Pareto frontier in a multi-objective
design space.
2.2 Dynamic optimization
Next to static, standalone compilers for programming lan-
guages such as C, C++, Fortran, etc., also dynamic compil-
ers which compile and optimize code at run time and which
have multiple levels of optimization (as is the case in many
Java Virtual Machines [3, 4, 19, 21]), can beneﬁt from multi-
objective compiler optimization exploration. The optimiz-
ing compiler in the Jikes Research VM for example, has
three optimization levels ( O0, O1, and O2), next to its base-
line compilation level (which compiles methods upon their
ﬁrst invocation). Jikes RVM uses timer-based sampling to
identify frequently executed methods which are passed to
the controller. The controller uses a cost-beneﬁt model to
decide whether to optimize a candidate method, and, if so,
to which level the method should be optimized. The cost-
beneﬁt model takes into account the estimated compilation
time, as well as the expected future execution time once
the method is optimized. Since compilation time is an in-
tegral part of the total execution time in a dynamic com-
piler, it is of utmost importance to make a good trade-oﬀ
between compilation time and code quality when propos-
ing optimization levels in a optimizing dynamic compiler.
Ishizaki et al. [17] describe how optimization levels can be
determined manually. The COLE framework on the other
hand, enables making such a multi-objective trade-oﬀ in an
automated manner, which is likely going to result in even
better optimization levels.
3. MULTI-OBJECTIVE COMPILER OPTI-
MIZATION LEVEL EXPLORATION
The goal of the multi-objective optimization is to identify
a set of compiler optimization levels that provide a Pareto
optimal trade-oﬀ with respect to a number of objective func-
tions. The objective functions are metrics of interest such
as total execution time, compilation time, code size, energy
consumption, etc.
3.1 Pareto optimization
In order to explain what Pareto optimality means, we need
to introduce some terminology. A given compiler optimiza-
tion level is called Pareto dominant with respect to another
optimization level if the given optimization level achieves a
better score for at least one objective funtion while achieving
the same (or better) score along the other objective func-
tions. A Pareto optimal compiler optimization level is an
optimization level for which there exist no other optimiza-
tion levels that achieve a better score for all objective func-
tions. Multiple Pareto optimal compiler optimization levels
can co-exist to form a so called Pareto frontier or Pareto
set, see Figure 1. In other words, the Pareto set collects all
the Pareto optimal compiler optimization levels. Once the
Pareto frontier is identiﬁed through multi-objective explo-
ration, the compiler developer or end user can then select
a compiler optimization level that trades oﬀ the various ob-
jective functions according to his or her needs.
3.2 Multi-objective exploration
The COLE multi-objective search algorithm proposed in
this paper is based on SPEA2 [25], which is an improved
version of the well-established Strength Pareto Evolutionary
Algorithm (SPEA) [26]. The SPEA2 algorithm is an eli-
tist evolutionary algorithm that is inspired by genetic algo-
rithms. Our multi-objective algorithm starts from a number
of populations of randomly generated entities, collectively
called a generation, see Figure 2. In our case, an entity is a
set of compiler optimizations, which corresponds to a candi-
date optimization level. From each population, a number of
entities are selected for inclusion in a so called archive. The
archive entity selection is done based on the entities’ Pareto
optimality, i.e., the best set of entities from the current pop-
ulation and the previous archive are retained in the current
archive. The next generation population is computed based
on the current archive through mutation, crossover and mi-
gration. This is done in two steps. First, mating pools are
created by selecting (possibly duplicate) entities from the
archive through binary tournament selection, or throughmi-
gration (with probability pmigration) which allows entities
to switch populations. In the second step, mutation and
crossover are applied on the mating pools. Mutation ran-
domly changes a single entity with a probability pmutation;
and crossover generates new entities by mixing two existing
entities within a mating pool with a probability pcrossover .
This process of mutation, crossover and migration based on
the archives, creates the next generation. The algorithm
is repeated, i.e., new generations are constructed, until no
more improvement is observed in the overall Pareto fron-
tier. The end result of the multi-objective search algorithm
is a number of Pareto frontiers — there is a Pareto frontier
per population. The overall Pareto frontier is then deter-
mined based on these individual Pareto frontiers, i.e., only
the Pareto optimal compiler optimization levels are selected
for the overall Pareto frontier.
The multi-objective algorithm considered in this work is
an extension and generalization of the SPEA2 algorithm in
that we consider multiple populations, whereas SPEA2 only
considers a single population. During our work, we found
that diﬀerent populations to start from results in diﬀerent
Pareto frontiers. In other words, it seems that, at least
for our design space, SPEA2 may yield a Pareto frontier
that is a local optimum. For this reason, we consider multi-
ple populations on which to apply the SPEA2 algorithm in
conjunction with migration which allows entities to switch
populations.
Within COLE, we give higher priority to optimization lev-
els that include fewer optimizations. For example, if two
optimization levels achieve the same compilation time and
execution time numbers — within the range of the run time
...
...
... ... ......
... ...
... ... ......
... ...
mutation crossover
migration
entity population
archive
generation 1
generation 2
... ... ......
... ... ...
generation n
...
...
Pareto frontiers per population
mating pool
... ... ...
overall Pareto frontier
Figure 2: The COLE multi-objective optimization framework.
variability observed from real hardware executions — we will
retain the optimization level with the fewest optimizations
included. Next to providing concise optimization levels, this
also helps the search algorithm in creating better quality
populations, i.e., optimizations that are eﬀective are given
a bigger chance to be used along the evolutionary search
process.
3.3 Exploration speed
Exploring the optimization level design space using the
above multi-objective search algorithm requires that each
entity is evaluated in order to understand its Pareto opti-
mality. Evaluating an entity requires quantifying the en-
tity in terms of the objective functions of interest. In our
work, this requires compiling the benchmarks using the set
of compiler optimizations that the entity represents. If per-
formance and/or energy consumption are of interest as ob-
jective functions, the compiled benchmarks need to be run,
and execution time and energy consumption need to be mea-
sured, either through simulation or real hardware execution.
In case code size is of interest, the code size needs to be com-
puted.
Evaluating an individual entity may be time-consuming.
However, evaluating a generation as a whole is embarrass-
ingly parallel. All entities in a generation — there are 3 ×
20 = 60 entities in total in our setup — can be evaluated in
parallel. Subsequent generations need to be run sequentially
though because the next generation is computed based on
the previous generation.
3.4 Compiler optimization level design space
The compiler optimization level design space can be huge,
which is the motivation in the ﬁrst place for employing an
automated multi-objective evolutionary search algorithm.
In our experimental setup, we use the GNU Compiler Col-
lection (GCC) 4.1.2 compiler, and consider all the individ-
ual compiler optimizations appearing in the standard -O1,
-O2, -O3 and -Os compiler optimization levels that can be
turned on and oﬀ individually through command-line com-
piler switches. There are 60 compiler ﬂags in total, see Ta-
compiler optimization compiler optimization
-fearly-inlining
-O2
-falign-functions=0
-ffunction-cse -falign-jumps=0
-fkeep-static-consts -falign-labels=0
-fpeephole -falign-loops=0
-fsplit-ivn-in-unroller -fcaller-saves
-ftree-vect-loop-version -fcrossjumping
-O1
-fcprop-registers -fcse-follow-jumps
-fdefer-pop -fcse-skip-blocks
-fguess-branch-probability -fdelete-null-pointer-checks
-fif-conversion -fexpensive-optimizations
-fif-conversion2 -fgcse
-fipa-pure-const -fipa-type-escape
-fipa-reference -foptimize-sibling-calls
-floop-optimize -fpeephole2
-fmerge-constants -fregmove
-ftree-ccp -freorder-blocks
-ftree-ch -freorder-functions
-ftree-copy-prop -frerun-cse-after-loop
-ftree-copyrename -frerun-loop-opt
-ftree-dce -fschedule-insns2
-ftree-dominator-opts -fstrength-reduce
-ftree-dse -fstrict-aliasing
-ftree-fre -fthread-jumps
-ftree-lrs -ftree-pre
-ftree-salias -ftree-store-ccp
-ftree-sink -ftree-store-copy-prop
-ftree-sra -ftree-vrp
-ftree-ter
-O3
-fgcse-after-reload
-funit-at-a-time -finline-functions
-funswitch-loops
-fno-reorder-blocks-and-partition
Table 1: Compiler optimization ﬂags considered in this paper.
ble 1. These ﬂags are ordered in Table 1 according to their
inclusion in the standard -O1, -O2 and -O3 compiler opti-
mization levels. For example, the -O1 optimization level
includes all the optimizations listed in the left column of
Table 1; -O2 includes all optimizations in the left column
plus all the optimizations in the right column from the top
down to and including -ftree-vrp; -O3 includes all opti-
mizations in Table 1. Next to these 60 compiler optimiza-
tion ﬂags, we also consider a base optimization level with
three possible options, -Os-stripped, -O1-stripped, and
-O2-stripped (-O3-stripped is identical to -O2-stripped).
These ‘stripped’ base optimization levels correspond to their
respective standard compiler optimization levels -O1, -O2,
-O3 and -Os with all optimizations disabled that are con-
trollable through compiler switches; disabling a standard
optimization -f<flag> can be done using the -fno-<flag>
compiler switch. The reason for including these stripped
base optimization levels is that GCC includes some default
optimizations that cannot be controlled through command-
line compiler switches. Put together, the entire compiler
optimization level design space includes 3 ×260 ≈ 3.4 ×1018
possible combinations of optimization ﬂags.
4. SETUP AND METHODOLOGY
4.1 Benchmarks, compiler and hardware setup
We use the SPEC CPU2000 benchmark suite [16] in our
setup. We consider the training inputs in this paper, be-
cause of time constraints. Compiling all the SPEC CPU2000
benchmarks and running them using the training inputs
takes about 20 minutes in our setup. Since we had a sin-
gle machine to our disposal for doing this research, evaluat-
ing a generation (which contains 60 candidate optimization
levels) takes about 1,200 minutes, or 20 hours. Running
the multi-objective algorithm until convergence (which re-
quired 70 generations) took about 50 days in our setup. As
mentioned before, this can be trivially parallelized by eval-
uating all entities in a given generation in parallel. In our
case, this would yield a 60 × speedup, or it would take less
than a day to run the multi-objective optimization using the
training inputs. Extrapolating these results to the reference
inputs of the SPEC CPU2000 benchmark suite, the whole
multi-objective optimization level searching would take ap-
proximately 5 to 6 days, which we believe is acceptable for
practical use — the deﬁnition of optimization levels can be
postponed towards the end of the compiler building process
and needs to be done only once. In addition, the above cal-
culations assume starting the search process ‘from scratch’.
In practice though, the search process for determining the
Pareto optimal optimization levels for the next generation
compiler can start from the Pareto frontier obtained for
the previous generation compiler, signiﬁcantly reducing the
search time. Nevertheless, speeding up the search process
may be an interesting avenue for future research.
As mentioned before, we use the GCC 4.1.2 compiler in
our experimental setup, because it provides a wide range
of command-line optimizations that can be turned on and
oﬀ, which is often not the case for commercial compilers (at
least for the ones we considered for our research). We ran all
of our experiments on an Intel Pentium 4 Prescott 3.0 GHz
processor based machine running Fedora Linux (Core 4).
4.2 COLE setup
In the evaluation reported in this paper of the COLE
framework, we consider two objective functions, namely code
quality (i.e., performance, measured as the benchmark exe-
cution time) and compilation time. This does not aﬀect the
generality of the COLE framework though, because COLE
can be (trivially) used for other objective functions of in-
terest, such as code size, energy consumption, power con-
objective function 1
objective function 2
compiler opt level
Pareto optimal
compiler opt level
Pareto frontier
HV reference point
Figure 3: Example illustrating how the HV metric
is computed.
sumption, etc. The code quality objective function in our
experiments equals the total sum of the individual execu-
tion times for all the SPEC CPU benchmarks; the compi-
lation time objective function equals the total sum of the
individual compilation times for all the benchmarks. These
objective functions give a higher weight to benchmarks that
execute and compile for a longer time. If equal weight to
all benchmarks is needed, the objective functions need to be
computed diﬀerently, for example by taking an unweighted
average over speedup numbers. Again, this is a design choice
up to the COLE user.
In the initial generation for the COLE optimization pro-
cess, we include the standard -O1, -O2, -O3 and -Os op-
timization levels, next to a number of randomly generated
optimizations, in order to give the multi-objective algorithm
a head start in its exploration.
We consider three populations, 20 entities per population,
and 10 entities per archive in our experiments. Recall that
the archive selects the Pareto optimal entities from the pop-
ulation and the prior archive. In case there are less than
10 Pareto optimal entities, the archive is ﬁlled up with near
Pareto optimal entities. In case there are more than 10
Pareto optimal entities, the archive selects the entities that
cover the objective function ranges as widely as possible.
The mutation probability is set to pmutation = 0 .10; the
probability for crossover is set to pcrossover = 0.90; and the
probability for migration is set to pmigration = 0.10. These
settings are fairly standard settings for genetic algorithms
and we found these settings to work well in our setup.
4.3 HV metric
Next to visually assessing the quality of the obtained Pare-
to frontiers, we also quantify their quality using the hyper-
volume (HV) metric discussed in [11]. Figure 3 illustrates
how the HV metric is computed. First, the HV reference
point needs to be determined. The HV reference point is
the point obtained by taking the maximum value observed
by any of the points along each objective function. Second,
once the HV reference point is determined, the HV met-
ric can be computed as the area between the Pareto frontier
and the HV reference point. The HV metric thus is a higher-
is-better metric. The higher the HV metric, the more the
Pareto frontier reduces the scores for all objective functions.
5. EV ALUATION
We now evaluate the quality of the obtained Pareto opti-
mal optimization levels, both qualitatively by visual inspec-
tion of the Pareto frontiers, and quantitatively by using the
HV metric.
5.1 Pareto frontiers
Figure 4 shows the Pareto frontiers as a function of code
quality (execution time) and compilation time for four sce-
narios of optimization levels:
• The standard GCC optimization levels ( -O1, -O2, -O3
and -Os); the optimization level -Os is not a Pareto
optimal optimization level (in terms of the code quality
and compilation time objective functions).
• The Pareto frontier which is obtained from randomly
sampling the optimization space: 600 randomly cho-
sen optimization levels were evaluated and the Pareto
optimal optimization levels are retained.
• The Pareto frontier obtained through COLE after 10
generations which corresponds to 600 optimization lev-
els being evaluated — this is the same search time as
under random sampling.
• The Pareto frontier obtained through COLE after con-
vergence at 70 generations. This corresponds to 4200
optimization levels being evaluated during the explo-
ration.
The results shown in Figure 4 clearly illustrate that the
Pareto frontier obtained through COLE is substantially bet-
ter than the standard GCC optimization levels. This means
that for the same compilation time, shorter run times are ob-
tained using the COLE Pareto frontier; or reverse, the same
code quality can be obtained with much shorter compila-
tion times, see Figure 5 for a summary result. In particular,
the Pareto optimal optimization level which yields the best
performance achieves 3.1% better performance on average
than -O3 with a 37.6% shorter compilation time. This same
Pareto optimal optimization level even achieves 4.5% better
performance than -O1 for a slightly smaller compilation time
(1.5%).
Comparing COLE versus random sampling — for doing
so we consider COLE with 10 generations (and thus 600
entities) versus random sampling of 600 entities — shows
that the evolutionary search algorithm in COLE is more
eﬀective than random searching. For the same exploration
time, COLE achieves optimization levels that provide better
trade-oﬀs in code quality versus compilation time.
5.2 Evaluation using the HV metric
We now evaluate the quality of the Pareto frontiers ob-
tained through COLE using the HV metric. Figure 6 shows
the HV metric for COLE, random sampling and the stan-
dard compiler optimization levels ( -O1, -O2, -O3). This
graph restates our earlier ﬁnding more quantitatively, namely
COLE outperforms random sampling by 10% with the same
search budget, and signiﬁcantly outperforms the standard
(manually derived) optimization levels by 53%.
5.3 Cross-validation
So far, we evaluated the COLE framework using the SPEC
CPU2000 benchmarks, which is also the set of benchmarks
650
700
750
800
850
250300350400450500550compilation time (seconds)
execution time (seconds)
 ref. point HV GCC -O1,-O2,-O3, -Os Pareto frontier random   Pareto frontier COLE (10 gen.) Pareto frontier COLE (70. gen)-O3-O2
-Os
-O1
Figure 4: Evaluating the Pareto frontiers obtained through COLE after 70 generations (after convergence
is reached) and 10 generations against the standard compiler optimization levels ( -O1, -O2, -O3 and -Os) and
random sampling (with the same exploration time budget as COLE with 10 generations).
0%10%20%30%40%50%
-O1 -O2 -O3
percentage improvement
compilation timeexecution time
Figure 5: Comparison of the Pareto optimal opti-
mization level that yields the best performance, i.e.,
the rightmost point on the Pareto frontier in Fig-
ure 4, against the standard GCC optimization levels
-O1, -O2 and -O3. The vertical axis shows the percent-
age improvement in compilation time and execution
time.
used to determine the Pareto optimal optimization settings.
We now consider a cross-validation experiment in which we
evaluate the eﬀectiveness of the Pareto optimal optimization
settings found through COLE using the SPEC CPU2000
benchmark suite on a previously unseen set of benchmarks,
namely the SPEC CPU2006 benchmark suite with training
inputs. The HV score for the COLE optimization levels
(obtained using SPEC CPU2000) is 37% higher than the
HV score for the standard optimization levels. This result
shows that the Pareto-optimal optimization settings found
with COLE using the SPEC CPU2000 benchmarks are good
optimization levels for SPEC CPU2006 as well.
05,00010,00015,00020,00025,00030,00035,00040,000
GCC -O1/2/3randomCOLE (10 gen.)COLE (70 gen.)
HV
Figure 6: Evaluation of the Pareto frontiers ob-
tained through COLE, random sampling and the
standard compiler optimization levels (-O1, -O2, -O3)
using the HV metric.
6. ANALYSIS
Now that we have obtained the Pareto optimal optimiza-
tion levels, we can analyze what these optimization levels
look like in terms of their constituent compiler optimiza-
tions. There are 12 Pareto optimal optimization levels in
total, and only 15 out of the 60 compiler optimizations are
included in at least one optimization level. In other words,
75% of the compiler optimizations do not contribute to any
of the Pareto optimal optimization levels. Of course, this
result is tied to the benchmarks as well as the hardware
platform considered in the setup, and only reﬂects average
performance. Nevertheless, this result may provide insight
to compiler developers and researchers about the usefulness
of the various compiler optimizations for the average user.
Table 2 shows the 12 Pareto optimal optimization levels in
terms of these 15 compiler optimizations. The optimization
levels are ordered from short compilation time and modest
code quality (optimization level 0) towards longer compila-
GCC Pareto optimal optimization level
level 0 1 2 3 4 5 6 7 8 9 10 11
-fno-keep-static-consts default x
-fdefer-pop -O1 x x x x
-fguess-branch-probability -O1 x x x x x
-floop-optimize -O1 x x
-ftree-ccp -O1 x x x x x x x x
-ftree-dce -O1 x x x x x x x x x x x x
-ftree-fre -O1 x x x x x x x x x x x x
-ftree-lrs -O1 x x x x x x x x x
-ftree-sra -O1 x x x x x x x x x x x x
-ftree-ter -O1 x x x x x x x x x x x x
-funit-at-a-time -O1 x x x x x x x x x
-fstrength-reduce -O2 x x
-fstrict-aliasing -O2 x x x x x x x
-ftree-store-copy-prop -O2 x x x x x x x
-finline-functions -O3 x
-Os-stripped x x x x
-O1-stripped x x x x x x x x
-O2-stripped
Table 2: Marking the compiler optimizations and (stripped) base optimization levels included in the various
Pareto optimal optimization levels, ordered from short compilation time and long run time (optimization
level 0) towards long compilation time and short run time (optimization level 11).
tion time and better code quality (optimization level 11). An
‘x’ reﬂects that a given compiler optimization is included in
the given optimization level. There are several interesting
observations to be made from Table 2. For example, sev-
eral of the language and platform-independent Static Sin-
gle Assignment (SSA) tree optimizations (with the -ftree
preﬁx) seem to be very eﬀective. In particular, four tree
SSA optimizations appear in all 12 Pareto optimal opti-
mization levels. These optimizations are beneﬁcial irrespec-
tive of the multi-objective trade-oﬀ in terms of compilation
time versus code quality. On the contrary, some optimiza-
tions are only beneﬁcial for a speciﬁc optimization trade-oﬀ.
For example, the function inlining optimization ( -finline-
functions) and loop optimization ( -floop-optimize) are
only helpful when code quality is the primary concern: these
ﬂags result in signiﬁcant code quality improvement at the
cost of additional compilation time.
Many of the optimizations under the standard compiler
optimization levels ( -O1, -O2, -O3 and -Os) do not appear
in any of these optimization levels. For example, of the 27
optimizations appearing under the standard -O2 optimiza-
tion level, see Table 1, only three get selected in at least one
of the Pareto optimal optimization levels; also, only one of
the three -O3 optimizations is being used. The -O1 opti-
mizations are more succesful in this respect: 10 out of the
23 optimizations are selected at least once.
Another interesting observation is that the Pareto opti-
mal optimization levels 0 through 3 use the stripped -Os-
stripped base optimization level; the Pareto optimal opti-
mization levels 4 through 11 use the stripped -O1-stripped
base optimization level. The reason is that the-Os-stripped
base optimization level introduces less optimization oppor-
tunities than the -O1-stripped base optimization level, so
that the optimizations applied on top of the -Os-stripped
base optimization level introduce less compilation load. The
-O2-stripped base optimization level is not selected for any
of the Pareto optimal optimization levels: it seems like this
base optimization level adds too much compilation load com-
pared to the performance gain it provides.
7. BENCHMARK OPTIMIZATION LEVEL
SENSITIVITY
Another interesting application of the COLE framework,
next to the construction of Pareto optimal optimization lev-
els as just decribed, is to study a benchmark’s sensitivity
with respect to the compiler optimization level. This will
enable us to classify benchmarks according to their opti-
mization sensitivity. This can be useful for compiler builders
and researchers in order to pick representative benchmarks.
Benchmarks that are sensitive to compiler optimizations may
be good candidate benchmarks; benchmarks that are rather
insensitive may be of less interest. Or, by looking into what
compiler optimizations aﬀect a benchmark insight can be
gained into the internal behavior of the benchmark.
For doing so, we use COLE to ﬁnd the Pareto frontier per
benchmark, and then compare this per-benchmark Pareto
frontier against the ‘average’ Pareto frontier obtained by
considering all benchmarks (as we did before). The re-
sults are shown in Figure 7. The HV metric for the stan-
dard -O1, -O2 and -O3 optimization levels as well as for
the average Pareto frontier are compared against the per-
benchmark Pareto frontier — these HV metrics are normal-
ized to the HV metric for the per-benchmark Pareto fron-
tier. Since the Pareto frontiers for the standard GCC op-
timization levels as well as for the ‘average’ Pareto frontier
are always worse than the per-benchmark Pareto frontier,
the relative HV metrics are all smaller than one. These re-
sults show that several benchmarks are rather insensitive
to benchmark-speciﬁc compiler optimizations, i.e., the ‘av-
erage’ Pareto frontier achieves nearly the same code qual-
ity versus compilation time trade-oﬀ as the per-benchmark
Pareto frontier. Example benchmarks are crafty, gzip, wup-
wise and several others with a relative HV metric for the ‘av-
erage’ Pareto frontier over 90%. Other benchmarks are very
sensitive to benchmark-speciﬁc optimizations, see for exam-
ple sixtrack and gap, which have relative ‘average’ Pareto
frontier HV scores around 40% and 50%. Overall though,
the ‘average’ Pareto frontier obtained through COLE yields
a trade-oﬀ between code quality and compilation time that
0%10%20%30%40%50%60%70%80%90%100%
ammpappluapsiartbzip2craftyeonequakefacerecfma3dgalgelgapgccgziplucasmcfmesamgridparserperlbmksixtrackswimtwolfvortexvprwupwiseAVERAGE
HV ratio to best Pareto curve
GCC -O1, -O2, -O3COLE (all benchmarks)COLE (single benchmark)
Figure 7: The HV metric per benchmark for the standard -O1, -O2 and -O3 optimization levels as well as the
‘average’ Pareto frontier relative to the per-benchmark Pareto frontier.
is fairly close to the per-benchmark optimal trade-oﬀ. The
average HV score for the ‘average’ Pareto frontier equals
83%, which is substantially higher than the standard -O1,
-O2 and -O3 optimization levels with an average HV score
of 50%.
8. SUMMARY AND FUTURE WORK
Static compilers as well as dynamic compilers typically
come with a number of optimization levels such as -O1, -O2,
-O3 and -Os, which provide diﬀerent trade-oﬀs between code
quality, compilation time and code size. Constructing these
optimization levels typically is a manual process which is
both tedious and time consuming. Identifying an appro-
priate set of optimization levels is particularly challenging
because the search space is huge — for example, the de-
sign space in our setup counts in the order of 1018 candidate
optimization levels.
This paper presented COLE, Compiler Optimization Level
Exploration, which employs a multi-objective evolutionary
algorithm to ﬁnd Pareto optimal optimization levels. COLE
is fully automated and is completely transparent to the com-
piler, the benchmarks, the hardware platform, as well as the
objective functions. To the best of our knowledge, this pa-
per is the ﬁrst to study automated multi-objective compiler
optimization exploration. COLE diﬀers from iterative com-
pilation in this respect because the work done so far in iter-
ative compilation focused on single-objective optimization.
Our experiments using GCC and the SPEC CPU bench-
marks on an Intel Pentium 4 machine optimizing for run
time (code quality) and compilation time show that the op-
timization levels obtained through COLE signiﬁcantly out-
perform the standard (and manually derived) compiler opti-
mization levels (-O1, -O2, -O3), and in addition, outperform
the optimization levels obtained through random sampling.
We believe this work may trigger future research in the
eﬃcient construction of compiler optimization levels. For
example, as mentioned in the paper, although the current
framework is fast enough for practical use, we believe there
is still room for improvement. There are various ways of
speeding up the search process by either pruning the op-
timization space or by reducing the time spent evaluating
an optimization level. Techniques proposed within the con-
text of iterative compilation for single-objective optimiza-
tion most likely need to be rethought within the context
of multi-objective optimization. Another interesting oppor-
tunity for future research is to exploit the fact that opti-
mization levels typically need to yield good average perfor-
mance across a number of benchmarks, as opposed to itera-
tive compilation which aims at optimizing the performance
of a single application. One could take advantage of the
optimization level sensitivity similarities that exist between
benchmarks to further speedup the search process, i.e., eval-
uating the compiler optimizations on a representative set
of benchmarks rather than the whole benchmark suite may
yield signiﬁcant speedups.
9. ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers for their
thoughtful comments and valuable suggestions. Kenneth
Hoste is supported by the Institute for the Promotion of
Innovation by Science and Technology in Flanders (IWT).
Lieven Eeckhout is a Postdoctoral Fellow with the Fund
for Scientiﬁc Research – Flanders (Belgium) (FWO Vlaan-
deren). This work is also supported in part by the FWO
projects G.0160.02 and G.0255.08, and the HiPEAC Net-
work of Excellence.
10. REFERENCES
[1] F. Agakov, E. Bonilla, J. Cavazos, B.Franke,
G. Fursin, M. O’Boyle, J. Thomson, M. Toussaint,
and C. Williams. Using machine learning to focus
iterative optimization. In Proceedings of the
International Symposium on Code Generation and
Optimization (CGO), pages 295–305, Mar. 2006.
[2] L. Almagor, K. D. Cooper, A. Grosul, T. J. Harvey,
S. Reeves, D. Subramanian, L. Torczon, and
T. Waterman. Compilation order matters: Exploring
the structure of the space of compilation sequences
using randomized search algorithms. In Proceedings of
the ACM SIGPLAN Symposium on Languages,
Compilers, and Tools for Embedded Systems
(LCTES), pages 231–239, June 2004.
[3] M. Arnold, S. Finka, D. Grove, M. Hind, and P. F.
Sweeney. Adaptive optimization in the Jalape˜ no JVM.
In Proceedings of the ACM SIGPLAN Conference on
Object-Oriented Programming Systems, Languages,
and Applications (OOPSLA) , pages 47–65, Oct. 2000.
[4] M. Arnold, M. Hind, and B. G. Ryder. Online
feedback-directed optimization in Java. In Proceedings
of the ACM SIGPLAN Conference on Object-Oriented
Programming Systems, Languages, and Applications
(OOPSLA), pages 111–129, Oct. 2002.
[5] F. Bodin, T. Kisuki, P. Knijnenburg, M. O’Boyle, and
E. Rohou. Iterative compilation in a non-linear
optimisation space. In Proceedings of the Workshop on
Proﬁle and Feedback-Directed Compilation, in
Conjunction with the Intl. Conf. on Parallel
Architectures and Compilation Techniques (PACT),
Oct. 1998.
[6] J. Cavazos, G. Fursin, F. Agakov, E. Bonilla, M. F. P.
O’Boyle, and O. Temam. Rapidly selecting good
compiler optimizations using performance counters. In
Proceedings of the International Symposium on Code
Generation and Optimization (CGO) , pages 185–197,
Mar. 2007.
[7] J. Cavazos and M. O’Boyle. Automatic tuning of
inlining heuristics. In Proceedings of the
Supercomputing Conference on High Performance
Networking and Computing , Nov. 2005.
[8] J. Cavazos and M. O’Boyle. Method-speciﬁc dynamic
compilation using logistic regression. In Proceedings of
the ACM SIGPLAN Conference on Object-Oriented
Programming Systems, Languages, and Applications
(OOPSLA), pages 229–240, Oct. 2006.
[9] K. Chow and Y. Wu. Feedback-directed selection and
characterization of compiler optimizations. In
Proceedings of the Workshop on Feedback-Directed and
Dynamic Optimization (FDDO) , Nov. 1999.
[10] K. D. Cooper, P. J. Schielke, and D. Subramanian.
Optimizing for reduced code space using genetic
algorithms. In Proceedings of the ACM
SIGPLAN/SIGBED Conference on Languages,
Compilers, and Tools for Embedded Systems
(LCTES), pages 1–9, May 1999.
[11] K. Deb. Multi-Objective Optimization using
Evolutionary Algorithms. Wiley, 2001.
[12] G. Fursin, A. Cohen, M. O’Boyle, and O. Temam.
Quick and practical run-time evaluation of multiple
program optimizations. Transactions on High
Performance Embedded Architectures and Compilation
Techniques (HiPEAC), 1(1):13–31, 2006.
[13] S. V. Gheorghita, H. Corporaal, and T. Basten.
Iterative compilation for energy reduction. Journal of
Embedded Computing, 1(4):509–520, July 2005.
[14] E. Granston and A. Holler. Automatic
recommendation of compiler options. In Proceedings of
the Workshop on Feedback-Directed and Dynamic
Optimization (FDDO), Dec. 2001.
[15] M. Haneda, P. M. W. Knijnenburg, and H. A. G.
Wijshoﬀ. Optimizing general purpose compiler
optimization. In Proceedings of the International
Conference on Computing Frontiers, pages 180–188,
May 2005.
[16] J. L. Henning. SPEC CPU2000: Measuring CPU
performance in the new millennium. IEEE Computer,
33(7):28–35, July 2000.
[17] K. Ishizaki, T. M., K. Kawachiya, T. Suganuma,
O. Gohda, T. Inagaki, A. Koseki, K. Ogata,
M. Kawahito, T. Yasue, T. Ogasawara, T. Onodera,
H. Komatsu, and T. Nakatani. Eﬀectiveness of
cross-platform optimizations for a Java just-in-time
compiler. In Proceedings of the ACM SIGPLAN
Conference on Object-Oriented Programming Systems,
Languages, and Applications (OOPSLA) , pages
187–204, Oct. 2003.
[18] P. Kulkarni, S. Hines, J. Hiser, D. Whalley,
J. Davidson, and D. Jones. Fast searches for eﬀective
optimization phase sequences. In Proceedings of the
ACM SIGPLAN Conference on Programming
Language Design and Implementation (PLDI) , pages
171–182, June 2004.
[19] D. Maier, P. Ramarao, M. Stoodley, and
V. Sundaresan. Experiences with multithreading and
dynamic class loading in a Java just-in-time compiler.
In Proceedings of the International Symposium on
Code Generation and Optimization (CGO) , pages
87–97, Mar. 2006.
[20] Z. Pan and R. Eigenmann. Fast, automatic,
procedure-level performance tuning. In Proceedings of
the International Conference on Parallel Architectures
and Compilation Techniques (PACT), pages 173–181,
Sept. 2006.
[21] T. Suganuma, T. Yasue, M. Kawahito, H. Komatsu,
and T. Nakatani. Design and evaluation of dynamic
optimizations for a Java just-in-time compiler. ACM
Transactions on Programming Languages and Systems
(TOPLAS), 27(4):732–785, July 2005.
[22] S. Triantafyllis, M. Vachharajani, and D. I. August.
Compiler optimization-space exploration. Journal of
Instruction-level Parallelism, Jan. 2005. Accessible at
http://www.jilp.org/vol7.
[23] H. Wu, E. Park, M. Kaplarevic, and Y. Zhang.
Dynamic optimization option search in GCC. In
Proceedings of the GCC Developers Summit 2007 ,
June 2007.
[24] M. Zhao, B. R. Childers, and M. L. Soﬀa. Predicting
the impact of optimizations for embedded systems. In
Proceedings of the ACM SIGPLAN Symposium on
Languages, Compilers, and Tools for Embedded
Systems (LCTES), pages 1–11, June 2003.
[25] E. Zitzler, M. Laumanns, and L. Thiele. SPEA2:
Improving the strength pareto evolutionary algorithm.
Technical Report TIK-Report 103, Swiss Federal
Institute of Technology (ETH) Zurich, May 2001.
[26] E. Zitzler and L. Thiele. Multiobjective evolutionary
algorithms: A comparative case study and the
strength perato approach. IEEE Transactions on
Evolutionary Computation, 3(4):257–271, Nov. 1999.

1
Improved Basic Block Reordering
Andy Newell and Sergey Pupyrev
Abstract—Basic block reordering is an important step for proﬁle-guided binary optimization. The state-of-the-art goal for basic block
reordering is to maximize the number of fall-through branches. However, we demonstrate that such orderings may impose suboptimal
performance on instruction and I-TLB caches. We propose a new algorithm that relies on a model combining the effects of fall-through
and caching behavior. As details of modern processor caching is quite complex and often unknown, we show how to use machine
learning in selecting parameters that best trade off different caching effects to maximize binary performance.
An extensive evaluation on a variety of applications, including Facebook production workloads, the open-source compilers Clang and
GCC, and SPEC CPU benchmarks, indicate that the new method outperforms existing block reordering techniques, improving the
resulting performance of applications with large code size. We have open sourced the code of the new algorithm as a part of a post-link
binary optimization tool, BOLT.
Index Terms—Code generation, Code layout, Optimizing compilers, Proﬁle-guided optimizations, Graph algorithms
!
1 I NTRODUCTION
P
ROFILE -guided binary optimization (PGO) is an impor-
tant step for improving performance of large-scale ap-
plications that tend to contain huge amounts of code. Such
techniques, also known as feedback-driven optimization
(FDO), are designed to improve code locality which leads
to better utilization of CPU instruction caches. In practice
tools like AutoFDO [1], Ispike [2], PLTO [3], HFSort [4], and
BOLT [5] speed up binaries by 5% −15% depending on
workload and CPU architecture, and thus, are widely used
for a variety of complex applications.
PGO is comprised of a number of optimization passes
such as function and basic block reordering, identical code
folding, function inlining, unreachable code elimination,
register allocation, and others. Typical targets for optimiza-
tions are an instruction cache (I-cache) used to hold exe-
cutable instructions and a translation lookaside buffer (I-
TLB) used to speed up virtual-to-physical address transla-
tion for instructions. The reordering passes directly optimize
code layout, and thus impact performance the most [2], [5].
Therefore, even small improvements in the underlying al-
gorithms for code reordering signiﬁcantly affect the beneﬁt
of PGO tools.
Current techniques for basic block reordering optimize a
speciﬁc dimension of CPU performance such as (i) cache line
utilization by increasing the average number of instructions
executed per cache line, (ii) the branch predictor by reducing
the number of mispredicted branches, and (iii) the instruc-
tion cache miss rate by minimizing cache line conﬂicts. An
application’s overall performance depends on a combina-
tion of these dimensions. Since modern processors employ
a complex and often non-disclosed strategy for execution, it
is challenging to consider all of these effects at once when
optimizing an ordering of basic blocks. In this paper, we make
the ﬁrst, to the best of our knowledge, attempt to design and
implement a block reordering algorithm that directly optimizes
• A. Newell is with Facebook, Inc., Menlo Park, CA, USA.
E-mail: newella@fb.com
• S. Pupyrev is with Facebook, Inc., Menlo Park, CA, USA.
E-mail: spupyrev@fb.com
the performance of an application.
Our approach consists of two main steps. Firstly, we
learn a proxy metric that describes the relationship between
the performance of a binary and the ordering of its basic
blocks. This is achieved by (i) identifying a set of features
representing how basic block ordering can inﬂuence per-
formance, (ii) collecting training data by running extensive
experiments and measuring the performance, and (iii) us-
ing machine learning to select the best combination of the
features for a score that best predicts CPU performance. Sec-
ondly, we suggest an efﬁcient algorithm that, given a control
ﬂow graph for a procedure, builds an improved ordering of
the basic blocks optimizing the learned metric. Since the
constructed metric correlates highly with the performance
of a binary, we observe overall efﬁciency gains, despite
possible regressions of individual CPU characteristics.
The contributions of the paper are the following.
• We identify an opportunity for improvement over the
classical approach for basic block reordering, initiated
by Pettis and Hansen [6]. Then we extend the model and
suggest a new optimization problem with the objective
closely related to the performance of a binary.
• We then develop a new practical algorithm for basic
block reordering. The algorithm relies on a greedy
technique for solving the optimization problem. We
describe the details of our implementation, which scales
to real-world instances without signiﬁcant impact on the
running time of a binary optimization tool.
• We propose a Mixed Integer Programming formulation
for the aforementioned optimization problem, which is
capable of ﬁnding optimal solutions on small functions.
Our experiments with the exact method demonstrate
that the new suggested heuristic ﬁnds an optimal or-
dering of basic blocks in 98% of real-world functions
with 30 or fewer blocks.
• Finally, we extensively evaluate the new algorithm on a
variety of applications, including Facebook production
workloads, open-source compilers, Clang and GCC, and
SPEC CPU 2017 benchmarks. The experiments indicate
that the new method outperforms the state-of-the-art
c⃝ 2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media,
including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution
to servers or lists, or reuse of any copyrighted component of this work in other works.
arXiv:1809.04676v2  [cs.PL]  11 Apr 2020
2
block reordering techniques, improving the resulting
performance by 0.5% −1%. We have open sourced the
code of our new algorithm as a part of BOLT [5], [7].
The paper is organized as follows. We ﬁrst discuss lim-
itations of the existing model for basic blocks reordering
and suggest an improvement in Section 2. We describe
an efﬁcient heuristic (Section 3) and an exact algorithm
(Section 4) for solving the new problem. Next, in Section 5,
we present experimental results, which are followed by a
discussion of related work in Section 6. We conclude the
paper and discuss possible future directions in Section 7.
2 L EARNING AN OPTIMIZATION MODEL
The state-of-the-art approach for basic block reordering is
based on the idea of collocating frequently executed blocks
together. The goal is to position blocks so that the hottest
successor of a block will most likely be a fall-through
branch, that is, located right next to the predecessor. This
strategy reduces the number of taken branches and the
working set size of the I-cache, while relieving pressure from
the branch predictor unit. More formally, the reordering
problem can be formulated as follows. Given a directed
control ﬂow graph comprising of basic blocks and fre-
quencies of jumps between the blocks, ﬁnd an ordering
of the blocks such that the number of fall-through jumps
is maximized. This is the maximum directed T RAVELING
SALESMAN PROBLEM (TSP), a widely studied NP-hard com-
binatorial optimization problem.
The simplicity of the model and solid practical results
made TSP-based algorithms very popular in the code opti-
mization community. To the best of our knowledge, Boesch
and Gimpel [8] are the ﬁrst ones to formulate the problem
of ﬁnding an ordering of basic block as the path covering
problem on a control ﬂow graph, which is equivalent to
solving TSP. They describe an optimal algorithm on acyclic
directed graphs and suggest a heuristic for general digraphs.
Later the same path covering model has been studied in a
series of papers suggesting optimal algorithms for special
classes of digraphs and heuristics for general digraphs [9],
[10]. In their seminal paper from 1990 [6], Pettis and Hansen
present two heuristics for positioning of basic blocks. We
observe that both heuristics are designed to solve (possibly
non-optimally) TSP. Later, one of the heuristics (seemingly
producing better results) has been extended by Calder and
Grunwald [11], Torrellas et al. [12], and Luk et al. [2]. We
stress that the majority of existing algorithms for block
reordering utilize the TSP model. A variant of the Pettis-
Hansen algorithm is used by most of the modern binary
optimizers, including PLTO [3], Ispike [2], BOLT [5], and the
link-time optimizer (LTO) of the GCC compiler [13].
Notice that solving TSP alone is not sufﬁcient for con-
structing a good ordering of basic blocks. It is easy to ﬁnd
examples of control ﬂow graphs with multiple different
orderings that are all optimal with respect to the TSP ob-
jective. Consider for example a control ﬂow graph in Fig. 1
in which the maximum number of fall-through branches is
achieved with two orderings that utilize a different number
of I-cache lines in a typical execution. For these cases, an
algorithm needs to take into consideration non-fall-through
branches to choose the best ordering. However, maximizing
B0
1000
B1
B3
B2
B4
5
500
995
B0
B1
B2
B3
B4
B0
B1
B2
B4
B3
64 bytes
500
500
5
5
Fig. 1. Two orderings of basic blocks with the same TSP score ( 1995)
resulting in different I-cache utilization. All blocks have the same size of
16 bytes and colored according to their hotness in the proﬁle.
the number of fall-through jumps is not always preferred
from the performance point of view. Consider a control
ﬂow graph with seven basic blocks in Fig. 2. It is not hard
to verify that the ordering with the maximum number of
fall-through branches is one containing two concatenated
chains, B0B1B3B4 and B5B6B2 (upper-right in
Fig. 2). Observe that for this placement, the hot part of
the function occupies three 64-byte cache lines. Arguably a
better ordering is the lower-right in Fig. 2, which uses only
two cache lines for the ﬁve hot blocks, B0,B1,B2,B3,B4, at
the cost of breaking the lightly weighted branch B6B2.
How do we identify the best ordering of basic blocks?
The question is fairly difﬁcult and even experts may have
hard time determining which ordering leads to the max-
imum performance of a binary. A naive approach is to
exhaustively evaluate every valid block placement and then
proﬁle the binary to collect relevant performance metrics.
Obviously due to the enormous search space, this approach
is infeasible for practical use. A natural improvement is
to reduce the search space and experiment only with the
most promising orderings. This technique, also known as
iterative compilation or autotuning, is a natural task for
machine learning [14], [15]. While in certain scenarios the
overhead is justiﬁable, we found this approach imprac-
tical for our production systems due to long build and
deployment times. Therefore, we use another strategy for
optimization by developing a score function that is used as
a proxy for estimating the quality of an ordering. The idea
is to perform extensive experiments proﬁling an application
in order to understand what aspects and features of the
block placement affect the resulting performance. After this
ﬁrst phase, employ a machine learning technique to build
an optimization model and derive a quality metric for an
ordering. As a ﬁnal step, design an algorithm to optimize
the constructed score function.
Next we describe the process in detail, explaining the
data collection phase and presenting the developed score
function. Since our new approach for basic block reorder-
ing is implemented in a post-link optimizer BOLT [5] and
evaluated on modern Intel x86 processors, we describe the
steps that are typical for this setup. Note however that the
approach is not tied to the binary optimizer and can be
similarly applied in other environments.
3
B0
1000
B1
B3
B4
B2
B5
550
550
455
5
5
450
B6
5
B0
B1
B3
B4
B6
B5
B2
B0
B1
B3
B4
B5
B2
B6
64 bytes
64 bytes
64 bytes
Fig. 2. A control ﬂow graph with jump frequencies (left) and two possible
orderings of basic blocks (right). All blocks have the same size (in bytes)
and colored according to their hotness in the proﬁle. An optimal TSP-
based layout (upper right) utilizes three cache lines for the hot code,
while an arguably better layout (lower right) can be built with a new
EXTTSP model.
2.1 Data Collection
Following most of the recent works on proﬁle-guided
code optimizations [1], [2], [4], [5], we rely on sam-
pling techniques for collecting proﬁle data. Although the
sampling-based approach is typically less accurate than the
instrumentation-based one, it incurs signiﬁcantly less mem-
ory and performance overheads, making it the preferred
way of proﬁling binaries in actual production environments.
We utilize hardware support of Intel x86 processors to col-
lect Last Branch Records (LBR), which is a list of the last 16
taken branches. From the list of branches, that are sampled
according to a speciﬁed event, we infer the frequencies
of jumps between basic blocks. Speciﬁcally, we extract a
weighted directed control ﬂow graph for every function
in the proﬁled binary. The vertices (basic blocks) and the
edges (branches) of the graph, along with the sizes of the
blocks, are extracted statically via the BOLT infrastructure,
which is based on LLVM [16]. The weights between the basic
blocks correspond to the total number of times the jumps
appear in collected LBRs. We stress that before processing,
we augment collected LBRs with fall-through jumps, as
LBRs only contain information about taken branches; to this
end, we utilize a simple algorithm similar to one described
by Chen et al. [17]. Notice that we ignore indirect branches,
procedure calls, and returns while constructing the control
ﬂow graph.
We have experimented with several different events to
collect LBRs, including cycles, retired instructions, and taken
branches, and using various levels of precise event based
sampling. We observed that independently of the utilized
event and processor microarchitecture, the extracted jump
frequencies do not always follow the expected distribution;
refer to [5], [17], [18] for concrete examples and possible
explanations of the phenomenon. Hence, we use by default
the cycles event to sample LBRs.
A common technique for ensuring edge weights are
more realistic is solving the M INIMUM COST MAXIMUM
FLOW problem on the control ﬂow graph, which was ini-
tiated by Levin et al. [19] and later adopted by several
groups [17], [18], [20], [21]. In contrast with the earlier
works, our experiments with the ﬂow-based approach did
not produce performance gains in comparison with the orig-
inal (possibly biased) data. A problematic example for the
approach is illustrated in Fig. 3a, where the arguably most
realistic adjustment is highlighted. Depending on how the
B1
3000
B2
B4
B3
400
600
B0
3000
B1
3000
B2
B4
B3
1200
1800
B0
3000
(a)
B1
3000
B2
B4
B3
0
1000
B0
3000
B1
3000
B2
B4
B3
0
3000
B0
3000 (b)
Fig. 3. Two examples of an original incomplete proﬁle (left) and its real-
istic correction (right) that cannot be reconstructed using the M INIMUM
COST MAXIMUM FLOW model. Bold numbers show most realistic adjust-
ments of the edge weights satisfying ﬂow conservation constraints.
costs of the edges are assigned, an algorithm for M INIMUM
COST MAXIMUM FLOW will either produce frequencies2600
and 400 or 600 and 2400 for jumps B2B3 and B2B4,
respectively. However, it is desirable to keep the proba-
bilities of the branches at B2 (approximately) the same.
A related issue is shown in Fig. 3b. Here an algorithm
may decide to send some ﬂow along edge B2B4, thus
making basic block B4 hot. This adjustment prohibits future
compiler optimizations that position hot and cold parts of
the function in different sections of the binary. Therefore,
we avoid modifying jump frequencies via the ﬂow-based
approach, leaving for the future the task of increasing proﬁle
precision.
2.2 Engineering a Score Function
Our goal is to design a function x →f(x), that takes in a
feature vector x, characterizing an ordering of basic blocks,
and produces a real value f(x), indicating an expected
performance of a binary for the ordering. We assume that
the execution time of a single basic block is independent
of the block ordering within a function. Thus, the ordering
only affects branches between the blocks, which may incur
some delay in the execution, for example due to a miss in the
instruction cache. However, not all branches equally affect
the performance. An important feature of a branch is the
jump length, that is, the distance (in bytes) between the end
of the source block to the beginning of the target block;
see Fig. 4. For example, it is a common belief that zero-
length jumps (equivalently, fall-through branches) impose
the smallest performance overhead. This is the main moti-
vation for the TSP model, whose objective can be formally
expressed as follows:
TSP =
∑
(s,t)
w(s,t) ×
{
1 if len(s,t) = 0,
0 if len(s,t) >0,
where w(s,t) is the frequency and len(s,t) is the length
of branch st. An optimal ordering corresponds to the
maximum value of the expression; thus, we call it the score
of TSP. The performance, however, might also depend on
other characteristics of a branch, which we discuss next. In
our study, we consider the following features.
• The length of a jump impacts the performance of in-
struction caches. Longer jumps are more likely to result
in a cache miss than shorter ones. In particular, a jump
with the length shorter than 64 bytes has a chance to
remain within the same cache line.
4
y
0
x
bytes
16
32
48
Fig. 4. The lengths of a for-
ward jump, xy, and a backward
jump, yx, are 16 and 48 bytes,
respectively.
0
0.1
branch importance
jump length, bytes
1024
640
forward
backward
1
fall-through
Fig. 5. The dependency be-
tween the length of a jump and
its importance for the E XTTSP
model.
• The direction of a branch plays a role for branch pre-
dicting. A branch stis called forward if s < t, that is,
block sprecedes block tin the ordering; otherwise, the
branch is called backward.
• The branches can be classiﬁed into unconditional (if the
out-degree is one) and conditional (if the out-degree is
two). A special kind of branches is between consecutive
blocks in the ordering that are called fall-through; in this
case, a jump instruction is not needed.
We introduce a new score that estimates the quality
of a basic block ordering taking into account the branch
characteristics. In the most generic form, the new function,
called EXTENDED TSP (E XTTSP), is expressed as follows:
ExtTSP =
∑
(s,t)
w(s,t) ×Ks,t ×hs,t
(
len(s,t)
)
,
where the sum is taken over all branches in the control
ﬂow graph. Here w(s,t) is the frequency of branch stand
0 ≤Ks,t ≤1 is a weight coefﬁcient modeling the relative
importance of the branch for optimization. We distinguish
six types of branches arising in code: conditional and un-
conditional versions of fall-through, forward, and backward
branches. Thus, we introduce six coefﬁcients for E XTTSP.
The lengths of the jumps are accounted in the last term of the
expression, which increases the importance of short jumps.
A non-negative function hs,t
(
len(s,t)
)
is deﬁned by value
of 1 for zero-length jumps, value of 0 for jumps exceeding a
prescribed length, and it monotonically decreases between
the two values. To be consistent with the objective of TSP,
the EXTTSP score needs to be maximized for the best perfor-
mance. Notice that EXTTSP is a generalization of TSP, as the
latter can be modeled by setting Ks,t = 1,h
(
len(s,t)
)
= 1
for fall-through branches and Ks,t = 0otherwise.
In general we cannot manually select the most appropri-
ate constants of EXTTSP that best model the performance of
modern processors. Next we describe a process for learning
these constant values that lead to the best performance.
2.3 Learning Parameters
As a preliminary step of our study, we run multiple ex-
periments with two binaries, the Clang compiler and the
HipHop Virtual Machine ( HHVM) [22]. Each experiment
consists of constructing a distinct ordering of basic blocks,
running a binary, and measuring its performance metrics
via the Linux perf tool. In order to build a variety of block
orderings for the same binary, we utilize ﬁve algorithms
(described in Section 5.1) and apply them for a certain
percentage of randomly selected functions. In total, we
evaluated 50 distinct block orderings and conducted 250
experiments (ﬁve per ordering) for each of the two binaries.
Our ﬁrst ﬁnding is that the traditional TSP score has
a relatively high correlation with the performance of the
binaries; see Fig. 6. However, there are several unexpected
outliers that cannot be explained by the model. In order
to choose suitable parameters for the E XTTSP score, we
employ the so-called black-box solver developed by Letham
et al. [23], which is a powerful tool for optimizing functions
with computationally expensive evaluations. Formally, our
problem can be stated as ﬁnding parameters for E XTTSP
that have the highest correlation with the performance of
a binary in the experiments. Here we try to maximize the
Kendall rank correlation coefﬁcient, τ, between the observed
performance (instructions per cycle) and the predicted im-
provement given by the E XTTSP score. Notice that the
Pearson correlation coefﬁcient, ρ, is not the best choice
for optimization, as the relationship between observed and
predicted values might not be linear. The black-box solver,
which is based on Bayesian optimization, is able to compute
values for a collection of continuous parameters that max-
imize the correlation coefﬁcient. This is done via a careful
exploration of the search space taking into account noise in
real-world experiment outcomes. In our study we introduce
six variables for weight coefﬁcients, K, of ExtTSP. The
jump-length function, h(·), is considered to be of the form(
1 −
(
len(jump)
M
)α)
with two variables M >0 and α >0
for different types of branches.
The black-box solver found a model that better predicts
the observed values; see Fig. 6. The new model increases
the Kendall correlation coefﬁcient τ from 0.877 to 0.906
for Clang and from 0.897 to 0.921 for HHVM. The models
constructed for Clang and HHVM are not identical, though
they share many similarities. Next we present a uniﬁed
variant of the model in which we round and combine
parameters with similar weights (having difference less than
0.05), and exclude ones having small values (less than 0.05).
We did not notice a discrepancy between the actual and
the rounded E XTTSP models, meaning that the resulting
solution works well for both of the binaries and is robust
to the choice of constants. Recall that better block orderings
correspond to higher values of E XTTSP.
ExtTSP =
∑
(s,t)
w(s, t)×



1 if len(s, t) = 0,
0.1·
(
1−len(s,t)
1024
)
if 0< len(s, t)≤1024
and s < t,
0.1·
(
1−len(s,t)
640
)
if 0< len(s, t)≤640
and t < s,
0 otherwise.
Intuitively, E XTTSP resembles the traditional TSP
model, as the number of fall-through branches is the dom-
inant factor. The main difference is that E XTTSP rewards
longer jumps. The impact of such jumps is signiﬁcantly
lower and it linearly decreases with the length of a jump.
Next we summarize our high-level observations regarding
the new score function.
5
86420speedup, %1086420TSP 1086420 ExtTSPscore improvement (%)ρ=0.973 τ=0.877ρ=0.984 τ=0.906
(a) Clang
6543210speedup, %3020100TSP 3020100ExtTSPscore improvement (%)ρ=0.985 τ=0.897ρ=0.989 τ=0.921 (b) HHVM
Fig. 6. The relationship between the performance (instructions per cycle) of two binaries and the TSP and E XTTSP scores measured for various
orderings of basic blocks. The values correspond to the relative improvements over the non-optimized binary with the original ordering provided by
the compiler.
• The suggested parameters for E XTTSP correlate well
with the overall performance of a binary in a
production-like environment, though we also observe
moderate correlation (in the order of ρ = 0.8) between
the values of E XTTSP and the measured number of I-
cache misses.
• We have not observed signiﬁcant differences be-
tween the importance of conditional and unconditional
branches that seem to be similarly relevant for the
quality of a block ordering. It contradicts to the intuition
of Calder and Grunwald [11] who assign noticeably
different weights depending on the type of a branch.
• The maximum length of a jump affecting E XTTSP is
fairly large: 16 and 10 cache lines for forward and back-
ward branches, respectively. The importance of a non-
fall-through branch linearly decreases with its length;
see Fig. 5. We experimented with non-linear decreasing
functions but did not discover a signiﬁcant improve-
ment; hence, we use the simpler variant.
• We found that forward branches are more important for
an ordering than backward ones; see Fig. 5 for a depen-
dency of the weights of the two types of branches on the
EXTTSP score. Both types of non-fall-through branches
are noticeably less important than fall-throughs in the
constructed model, which is reﬂected in the low coefﬁ-
cient (0.1) in the expression.
Finding an optimal solution for the E XTENDED TSP
problem is NP-hard. Next we describe our heuristic.
3 A H EURISTIC FOR EXTTSP
Our algorithm ﬁnds an optimized ordering of basic blocks
for every function in the binary. It operates with a weighted
control ﬂow graph G = (V,E,w ) containing a set of basic
blocks, V, and directed edges, E, representing branches
between the blocks. An edge (s,t) ∈ E corresponds to
a jump from a block s ∈ V to a block t ∈ V and its
weight, w(s,t), corresponds to the frequency of the jump.
We assume that the sizes (in bytes) of the basic blocks are
a part of the input. The goal of the algorithm is to ﬁnd an
ordering of V with an improved ExtTSP score (as deﬁned
in Section 2) while keeping a given entry point, v∗∈V, the
ﬁrst in the ordering.
Algorithm 1:Basic Block Reordering
Input : control ﬂow graph G= (V,E,w ),
the entry point v∗∈V
Output: ordering of basic blocks (v∗= B1,B2,...,B |V|)
Function ReorderBasicBlocks
for v∈V do /* initial chain creation */
Chains ←Chains∪(v);
while |Chains|>1 do /* chain merging */
for ci,cj ∈Chains do
gain[ci,cj] ←ComputeMergeGain(ci,cj);
/* find best pair of chains */
src,dst ←arg max
i,j
gain[ci,cj];
/* merge the pair and update chains */
Chains ←Chains∪Merge(src,dst) \{src,dst};
return ordering given by the remaining chain;
Function ComputeMergeGain(src,dst)
/* try all ways to split chain src */
for i= 1to blocks(src) do
/* break the chain at index i */
s1 ←src[1 :i];
s2 ←src[i+ 1 : blocks(src)];
/* try all valid ways to concatenate */
scorei max



ExtTSP(s1,s2,dst) if v∗̸∈dst
ExtTSP(s1,dst,s 2) if v∗̸∈dst
ExtTSP(s2,s1,dst) if v∗̸∈s1,dst
ExtTSP(s2,dst,s 1) if v∗̸∈s1,dst
ExtTSP(dst,s1,s2) if v∗̸∈src
ExtTSP(dst,s2,s1) if v∗̸∈src
/* the gain of merging chains src and dst */
return max
i
scorei −ExtTSP(src) −ExtTSP(dst);
On a high level, our algorithm is a greedy heuristic that
works with chains (ordered sequences) of basic blocks; see
Algorithm 1 for an overview. Initially all chains are isolated
basic blocks. Then we iteratively merge pairs of chains so as
6
to improve the ExtTSP score. On every iteration, we pick
a pair of chains whose merging yields the biggest increase
in ExtTSP, and the pair is merged into a new chain. The
procedure stops when there is only one chain left, which
determines the resulting ordering of basic blocks.
An important aspect of our approach is the way two
chains are merged; see function C OMPUTE MERGE GAIN of
Algorithm 1. In order to merge a pair of chains, src and
dst, we ﬁrst split chain src into two subchains, s1 and
s2, that retain the ordering of blocks given by src. Then
we consider all six possible ways of combining the three
chains, s1, s2, and dst, into a single one, discarding the
ones that do not place entry point v∗ at the beginning. A
chain with the largest ExtTSP over all possible splitting
indices of src and permutations of s1, s2, dst is chosen
as the result. The motivation here is to increase the search
space in comparison to the simpler concatenation of two
chains. The simplest example in which chain splitting helps
is depicted in Fig. 7. A greedy concatenation merges block
B0 with B2 on the ﬁrst iteration, which results in the ﬁnal
ordering (B0,B2,B1). In contrast, chain splitting allows to
build an ordering (B0,B1,B2), which has a higher E XTTSP
score since all the edges are forward.
B0
99
B1
B2
100
99
Fig. 7. An example of a control ﬂow graph in which naive chain concate-
nation produces suboptimal ordering.
What is the computational complexity of Algorithm 1?
A naive implementation takes O(|V|5) steps: There are |V|
merge iterations that process at most|V|2 pairs of chains per
iteration with O(|V|2) steps needed to compute a merge
gain between two chains. However, this is an overestima-
tion, as we argue below. First observe that the ExtTSP
score between two chains, c1 and c2, can be positive only
if there is a branch between c1 and c2; thus, the number
of candidate chain pairs for merging is upper bounded by
the total number of branches in the control ﬂow graph,
|E|. The second observation is that one can memoize the
results of C OMPUTE MERGE GAIN function and re-use them
throughout the computation. It is easy to see that the merge
gain depends only on a pair of chains; hence, if neither of
the two chains is merged at an iteration, then we do not
need to recompute the gain for the pair at the next iteration.
Putting the two observations together, the running time of
Algorithm 1 is bounded by O
(∑
cblocks(c) ·degree(c)
)
,
where the sum is taken over all chains taking part in a merge
with blocks(c) being the number of blocks in the chain and
degree(c) being the number of branches from and to the
chain. In the worst case, this sums up to O(|V|2|E|) time in
general, which equals to O(|V|3) for real-world control ﬂow
graphs.
3.1 Large functions
While cubic running time is acceptable for reordering most
of the functions we experimented with, there are several
exceptions with a large number of basic blocks. In order
to deal with these cases, we introduce a threshold, k >1,
on the maximum size of a chain that is considered for
splitting in C OMPUTE MERGE GAIN . If the size of a chain,
c, exceeds the threshold, that is, blocks(c) >k, then we only
try a simple concatenation of cwith other chains. With the
modiﬁcation, the complexity of Algorithm 1 is estimated by
O(k·|V|2), which is quadratic when kis a constant. In our
implementation, we use k= 128as the default value.
3.2 Reordering of cold blocks
Notice that Algorithm 1 is not trying to optimize layout of
cold basic blocks that are never sampled during proﬁling.
However, one may still want to modify their relative order,
as this could affect the code size as follows. Consider pairs
of cold basic blocks, s and t, such that the only outgoing
branch from sis the only incoming branch to t. If sand tare
not consecutive in the resulting ordering, then one would
need to introduce an unconditional branch instruction. In
contrast, if tfollows sin the ordering, then the instruction is
not needed as t is on the fall-through path of s. In order
to guarantee that Algorithm 1 always merges such pairs
into a chain, we modify the weights of cold edges in the
control ﬂow graph before the computation. Speciﬁcally we
set w(s,t) = ϵ1 (for some 0 < ϵ1 ≪ 1) if (s,t) ∈ E
corresponds to a cold fall-through branch in the original
binary, and set w(s,t) = ϵ2 (for some 0 < ϵ2 < ϵ1) if (s,t)
corresponds to a cold non-fall-through branch. Such weights
make it desirable to merge original fall-through branches,
even if they are cold according to the proﬁle.
3.3 Code layout in memory
Apart from basic block reordering, proﬁle-guided optimiza-
tion tools typically perform two other passes directly affect-
ing the layout of functions in the generated code: hot/cold
code splitting and code alignment [1], [5]. The ﬁrst one splits
hot and cold basic blocks into separate sections, while the
second pass aligns the blocks at cache line boundaries via
introducing NOP instructions. We stress that both optimiza-
tions are complimentary to basic block reordering and their
beneﬁts are additive. In our experiments, we evaluate the
effect of reordering alone, with all other optimization passes
applied for the binary.
4 A N OPTIMAL ALGORITHM FOR EXTTSP
We now demonstrate how EXTTSP is formulated as a Mixed
Integer Program (MIP). A MIP is a method to ﬁnd optimal
solutions for NP-hard problems whose objective and require-
ments are represented by linear functions. This is a time-
consuming technique that can be applied only for small
instances, and we use the optimal MIP solutions to better
understand the quality of our heuristic.
7
maximize ∑
(s,t) w(s,t) ×f(ds,t)
subject to xs ∈ R, s ∈V
ds,t ∈ R, (s,t) ∈E
zs,t ∈ {0,1}, s,t ∈V,s ̸= t
xt −xs ≥ Ls −M
×(1 −zs,t), s,t ∈V,s ̸= t
xs −xt ≥ Lt −M ·zs,t, s,t ∈V,s ̸= t
ds,t = xt −xs −Ls, (s,t) ∈E
The objective is a summation over the contribution of
each edge (s,t) ∈E in the control ﬂow graph to E XTTSP.
The contribution of an edge is the number of jumps, w(s,t),
weighted by a value dependent on the length of the jump,
ds,t = len(s,t). The piece-wise function shown in Fig. 5
converts the length to the desired weight. This function
is formulated in MIP by introducing additional integer
variables to cope with the non-convex shape [24].
The constraints express the complete search space of all
legal starting bytes xs for each block s ∈ V considering
the size of the block, Ls. For all pairs of blocks, s and t,
either the ﬁnal ordering has s before t (that is, xt −xs ≥
Ls) or t before s (that is, xs −xt ≥Lt). A binary variable
zs,t is utilized to enforce one of those two constraints. The
distances used in the objective, ds,t, are constrained to be
the distance between the end of block s and the start of
block t. Negative distances correspond to backward jumps
which are incorporated into the piece-wise function in the
objective. We utilize the Xpress solver for ﬁnding optimal
solutions of the MIP model.
5 E VALUATION
The experiments presented in this section were conducted
on Linux-based servers powered by Intel microprocessors.
The applications were compiled using either GCC 8.3 or
Clang 7.1 with -O3 optimization level.
5.1 Techniques
We compare our new algorithm (referred to as ext-tsp)
with the following competitors.
• original is the ordering provided by the compiler.
• tsp is the ordering constructed by the “top-down”
heuristic suggested by Pettis and Hansen [6]. The al-
gorithm starts by placing the entry basic block for a
function, and then iteratively ﬁnding a successor with
the heaviest edge to the last placed block. If all succes-
sors have already been selected, then one picks the block
with the largest connection to the already placed blocks.
• ph is the Pettis-Hansen “bottom-up” algorithm [6]. The
algorithms maintains a collection of chains of basic
blocks, which correspond to paths in the control ﬂow
graph. Initially every block forms its own chain. Look-
ing at the arcs from largest to smallest, two different
chains are merged together if the arc connects the tail
of one chain to the head of another. Once the merging
stage is done, the chains are ordered so as to maximize
the weight of backward edges to achieve the best per-
formance of the branch predictor.
TABLE 1
Basic properties of evaluated binaries
hot blocks per function
.text (MB) IPC functions p50 p95 max
HHVM 285 0 .83 12 ,687 40 454 10 ,664
Multifeed 395 0 .95 24 ,037 15 151 9 ,228
Proxygen 160 0 .63 9 ,997 10 97 945
Clang 48 0 .82 7 ,013 25 241 11 ,218
GCC 15 0 .76 10 ,269 11 152 3 ,354
86420speedup, %HHVMMultifeed1Multifeed2Proxygen
4.8
1.5
5.0
3.5
4.8
1.5
6.0
4.5
5.1
2.0
6.0
4.0
5.6
2.0
7.0
5.0 tsp   ph   cache   ext-tsp 
Fig. 8. Performance improvements of various reordering algorithms over
original measured for different Facebook workloads.
• cache represents a modiﬁcation of the Pettis-Hansen
algorithm suggested by Luk et al. for the Ispike post-
link optimizer [2]. The difference from ph is in the last
step, ordering of chains of basic blocks. The chains are
sorted by their density, that is, the total execution count
of a chain divided by the sum of sizes of its basic blocks.
Placing hottest chains ﬁrst reduces conﬂicts in the I-
cache and improves code locality.
• mip is an optimal algorithm for ExtTSP described in
Section 4. Since the running time of the approach is
not practical for large functions, we only compare the
results of mip on a subset of small functions.
All the algorithms are implemented in an open-source
post-link binary optimizer BOLT [7].
5.2 Facebook Workloads
This section evaluates various basic block ordering algo-
rithms on four large-scale binaries deployed at Facebook’s
data centers. The ﬁrst system, which is our primary use case
for ext-tsp, is the HipHop Virtual Machine ( HHVM) [22],
that serves as an execution engine for PHP at Facebook,
Wikipedia, Baidu, and other large websites. The two bina-
ries of Multifeed are responsible for News Feed. Proxygen
is a Facebook service for cluster load balancing. The HHVM
binary is built using GCC with LTO, while Multifeed and
Proxygen are compiled with Clang with AutoFDO enabled
to enhance their performance. All the Facebook services
are running with huge pages enabled and utilize function
reordering [4]. Table 1 provides basic properties of the
evaluated binaries.
Fig. 8 presents a performance comparison of four ba-
sic block ordering algorithms on the Facebook workloads.
The results are obtained by using an internal performance-
measurement tool for running A/B experiments; see [23],
[25] for an overview. The tool is used at Facebook for a wide
8
2520151050reduction, %I-cachemissI-TLBmissLLCmissBranchmissInstructions
18.7
10.3
3.5
5.7
0.12
18.9
9.8
3.4
6
0.11
19.3
10.9
3.3
6.1
0.28
20.8
9.9
3.5
6.2
0.35
 tsp  ph  cache  ext-tsp 
Fig. 9. perf metrics measured for the HHVM binary.
range of performance evaluations by running experiments
on a set of isolated machines that process the same pro-
duction trafﬁc over several days. We measure performance
as the CPU utilization during steady state. As a baseline,
we observe the performance of the binaries optimized with
BOLT using the original block ordering algorithm. In the
case of HHVM, this is an original ordering constructed by the
compiler, while for Multifeed and Proxygen, the ordering is
a result of processing the binaries with PGO. The ﬁgure re-
ports mean relative improvements of various block ordering
algorithms over original along with their 95% conﬁdence
intervals. In the experiments, we notice that differences in
CPU utilization among the block ordering algorithms are
highly correlated with the differences in instructions per
cycle (IPC).
Overall we observe that ext-tsp performs better than
alternative ordering algorithms on three of the evaluated bi-
naries. The relative speedup is close to1% for Multifeed2 and
around 0.5% for HHVM and Proxygen. We stress that the
measurements for Multifeed1 are noticeably noisier than the
alternatives with a typical deviation from the mean around
0.5%−1%. To better understand the beneﬁts of applying the
new block ordering algorithm, we perform a more detailed
evaluation of HHVM. The results are depicted in Fig. 9.
The main advantage of block ordering optimization is an
improved performance of the L1 I-cache, that exhibits over
19% miss reduction. The new ordering algorithm increases
this value to 21%. The number of branch and I-TLB misses
is also signiﬁcantly reduced, with ext-tsp being the best
for the branch misses counter. We also see a modest im-
provement in the performance of the last level cache, though
the difference between various ordering algorithms is not
prominent.
5.3 Open-Source Compilers
Since basic block reordering primarily improves the perfor-
mance of the I-cache, our optimization can be beneﬁcial for
any front-end bound application with large code size. We
illustrate this by experimenting with binaries of two open-
source compilers, Clang and GCC, whose .text sections
are 48MB and 15MB, respectively; see Table 1. For these
experiments, we utilize a dual-node 28-core 2.4 GHz Intel
Xeon E5-2680 (Broadwell) with 256GB RAM. The size of
the L1 I-cache on the processor is only 32KB, which makes
the two binaries good candidates for proﬁle-guided layout
optimizations.
86420speedup, %ClangClangPGO+LTOGCCGCCPGO
7.12
2.76
7.32
2.51
7.11
3.18
7.59
3.03
6.81
3.38
7.61
2.92
7.86
4.17
8.20
3.65
 tsp  ph  cache  ext-tsp 
Fig. 10. Performance improvements of various reordering algorithms
over original measured for Clang and GCC.
403020100 reduction, %34.1
19.3
2.9
5.2
0.31
36.4
20.4
2.8
5.6
0.28
37.1
23.1
2.8
5.5
0.28
39.8
21.9
2.9
5.7
0.47
 tsp  ph  cache  ext-tsp 
GCC
2520151050reduction, %I-cachemissI-TLBmissLLCmissBranchmissInstructions
14.5
12.4
2.2
3.6
0
16.9
17.2
2.4
3.8
0
16.3
11.7
2.5
4.1
0
19.8
17.1
2.6
4.1
0 GCC+PGO
Fig. 11. perf metrics measured for the GCC binary with (bottom)
and without (top) PGO applied. The improvements are on top of the
original block ordering using the same conﬁguration.
For the evaluation of Clang, we use the release_71
branch of LLVM. We ran the evaluation in two modes,Clang
and Clang+PGO+LTO; see Fig. 10. First we build a release
version of the binary using GCC and collect a proﬁle data
by compiling a medium-sized template-heavy C++14 source
ﬁle. For the ﬁrst experiment ( Clang), we optimize the binary
of Clang with BOLT using various orderings of basic blocks,
and compile a different collection of about100 C++14 source
ﬁles. Hence, the train and the test datasets are different in
the evaluation. Every experimental run is repeated 1000
times to increase precision of our measurements so that
the average mean deviation is within 0.05%. The baseline
in Fig. 10 corresponds to a binary processed with BOLT
using the original ordering of basic blocks. Thus, the
improvements are attributed only to block reordering, while
all other optimizations (e.g., function reordering or inlining)
are the same across the experiments.
For our next experiment, Clang+PGO+LTO, we in ad-
dition enable PGO and LTO support. To this end, we ﬁrst
built an instrumented version of Clang, and then used the
instrumented compiler to build the binary again with the
9
default options of GCC. The collected proﬁle data was used
to do another build of Clang with PGO and LTO enabled.
The results in Fig. 10 indicate that block reordering alone
provides 2% −4% performance improvements, even when
applied on top of GCC with PGO and LTO. This ﬁnding is
consistent with earlier evaluations of BOLT, where the gains
are attributed to an improved code layout [5].
For the evaluation of GCC, we use version 8.3 of the
compiler. Again, we collected a proﬁle data by compiling
the single source ﬁle, and perform testing using the larger
C++14 project. Similar to Clang, we ran experiments in two
modes; however, we did not use LTO due to build errors re-
lated to C++ exceptions. The effect of applying various block
ordering techniques on two versions of the GCC binary
(with and without PGO enabled) is shown in Fig. 10. As ex-
pected, the relative improvements are smaller for the binary
built with PGO. The new algorithm, ext-tsp, provides the
largest gains outperforming competitors by0.5%−0.8%; the
differences are identiﬁed as statistically signiﬁcant. Fig. 11
presents the impact of block ordering algorithms on key
architectural metrics for GCC. As in the case with HHVM,
the improvements are largely attributed to a reduction in
I-cache and I-TLB misses. Other relevant metrics are also
improved in comparison with the original ordering but
there is little difference among the ordering techniques.
5.4 SPEC CPU 2017
In this section we evaluate basic block reordering on the
SPEC CPU 2017 benchmark. We utilize 16 C/C++ programs
compiled using GCC with LTO and ran experiments on
the same hardware as in the previous section. We analyze
the performance of the binaries optimized by BOLT with
various ordering algorithms, using original as a baseline.
Proﬁle data is collected using a separate SPEC train mode.
We observe that the SPEC binaries are much smaller than
the typical applications used in modern data centers. There-
fore, they are unlikely to be front-end bound and exhibit
many I-cache and I-TLB misses. Fig. 12 presents the results
of our experiments on the largest binaries that contain at
least 100KB of hot code according to the collected proﬁle.
We do not see a consistent advantage of applying basic block
reordering for the binaries. In most of the experiments we
record a high variance in the running times, which often
exceeds the differences between means. An optimized block
ordering yields a statistically signiﬁcant improvement over
original only for three binaries: gcc4.5 and namd (using
ext-tsp), and xalancbmk (using cache). For the largest
program, gcc4.5, the ext-tsp algorithm achieves 1.5%
speedup outperforming the best competitor by 0.4%.
To understand the source of regressions, we analyze
two binaries from the benchmark, x264 and omnetpp,
whose running times increase by 0.5% −1.5% after block
reordering. In the former case, we observe a substantial
growth in the number of branch misses, which leads to the
performance regression. In the latter case, we found that
a different alignment of hot loops in the binary worsen the
performance despite a signiﬁcant increase (20%) of the num-
ber of fall-through branches and a modest improvement
(around 5%) in the number of I-cache misses. We conclude
that for small binaries that are not front-end bound, both
TSP and E XTTSP are not accurate models.
-2-1012speedup, %
gcc4.52100KBperlbench362KBparest290KBblender259KBx264229KBnamd162KBxalancbmk146KBomnetpp130KBcactuBSSN117KB
 tsp  ph  cache  ext-tsp 
Fig. 12. Relative performance differences between original and al-
ternative block reordering methods on the largest binaries of the SPEC
2017 dataset. Positive values indicate improvements, negative ones in-
dicate regressions. For every binary, the size of the hot code is speciﬁed.
5.5 Analysis of EXTTSP
Here we present an evaluation of Algorithm 1 for solving
the EXTTSP problem. We design the experiments to answer
two questions: (a) How do various parameters of the al-
gorithm contribute to the solution and what are the best
default values? (b) How does the algorithm perform in com-
parison with existing heuristics and the optimal technique?
Considering the ﬁrst question, we observe that ext-tsp
has only one parameter that affects its quality and perfor-
mance. As explained in Section 3, we introduce a threshold
k, which controls the maximum size of a chain that can be
split during optimization. In the extreme case with k= |V|,
all chains can be broken if that improves the objective;
however, the running time of the algorithm is cubic on the
number of basic blocks comprising a function. Another ex-
treme, k = 0, forbids chain splitting but makes the running
time quadratic. As Table 1 illustrates, some functions in the
dataset contain a few thousand of basic blocks. Hence, the
threshold should be chosen carefully, since it impacts the
quality of a solution and the time needed to process a binary,
which is important in production environments.
Fig. 13 illustrates the results of the experiments with
the chain splitting threshold. Multifeed1 is the binary whose
processing time is substantially affected by large values ofk.
For k ≥1024, the combined running time of ext-tsp on
all functions of the Multifeed1 binary is around 2 minutes,
while for HHVM it is less than 20 seconds. For k = 0, the
processing times are 6 and 4 seconds for the binaries, respec-
tively. The difference between the corresponding solutions
in the E XTTSP score ranges between 0.3% and 0.7%; note
that according to our analysis in Section 2 and Fig. 6, this
translates to a performance difference of 0.1% −0.3%. The
value k = 128provides a reasonable compromise between
processing speed and solution quality, and thus, it is utilized
as the default value for ext-tsp in all our experiments.
In order to analyze the quality of solutions for E XTTSP
generated by Algorithm 1, we employ the optimal technique
presented in Section 4. We apply mip to all 2992 functions
containing at most 30 basic blocks in the HHVM binary. For
these small functions, mip ﬁnds a provably optimal solution
in 2963 (99%) of the cases in less than one minute. Out of
these instances with the known optimal ordering, ext-tsp
ﬁnds an equivalent solution in 2914 (98.3%) cases. For the
remaining 49 functions, the E XTTSP score produced by
10
0.60.50.40.30.20.10.0ExtTSP improvement, %2 8 32 128 512 split threshold, k
10080604020running time, sec2 8 32 128 512 split threshold, k
 Clang HHVM Multifeed Proxygen
Fig. 13. The running times and the resulting E XTTSP scores produced
by ext-tsp (Algorithm 1) using various chain splitting thresholds for
binaries described in Table 1.
ext-tsp is on average 0.14% lower than the optimum. For
comparison, the runner-up approach on the same binary is
cache, which is able to reconstruct 2745 (92.6%) of optimal
orderings in the binary. The relative improvements in the
EXTTSP score over non-reordered functions are 27%, 29%,
29%, 31% for tsp, ph, cache, and ext-tsp, respectively,
which aligns with our experiments illustrated in Fig. 8.
We emphasize that mip is not considered a practical
approach, as it does not scale to instances with many
basic blocks. The average running time of mip on a func-
tion with 30 blocks exceeds 10 seconds, while it is below
a millisecond for all four alternatives, tsp, ph, cache,
and ext-tsp. Nevertheless, the aforementioned analysis
demonstrates that the new heuristic provides a close-to-
optimal solutions in the majority of real-world instances,
while being sufﬁciently fast to process large functions.
6 R ELATED WORK
There exists a rich literature on proﬁle-guided optimiza-
tions. Here we discuss previous works that are closely
related to code layout and our main contributions.
The work by Pettis and Hansen [6] is the basis for
the majority of modern code reordering techniques. The
goal is to create chains of basic blocks that are frequently
executed together in the order. As discussed in Section 2,
many variants of the technique have been suggested in
the literature and implemented in various tools [2], [3],
[8], [9], [10], [11], [12], [13]. Similar to our work, the tech-
niques are operating with a control ﬂow graph and try
to lay out basic blocks tackling a variant of the T RAVEL -
ING SALESMAN PROBLEM . Alternative models have been
studied by Bahar et al. [26], Gloy et al. [27], and Lavaee
and Ding [28], where a temporal-relation graph is taken
into account. Temporal afﬁnities between code instructions
can be utilized for reducing conﬂict cache misses [29] and
improving the performance of multiple applications using
a shared cache [30]. We emphasize that according to our
experiments, the performance of a front-end bound large-
scale binary can be largely predicted by its control ﬂow
graph without considering more expensive models.
Code reordering at the function-level is also initiated by
Pettis and Hansen [6], who describe an algorithm that is
implemented in many compilers and binary optimization
tools [5], [13], [31], [32]. This approach greedily merges
chains of functions and is designed to primarily reduce I-
TLB misses. An improvement is recently proposed by Ottoni
and Maher [4], who suggest to work with a directed call
graph. Note that unlike our work, the techniques are heuris-
tics not aiming to produce code layouts that are optimal
from the performance point of view.
Another opportunity for improving performance is to
modify layout of data [33], [34], [35], [36]. Most of the
existing works focus on ﬁeld reordering and structure
splitting based on the ﬁeld hotness and data afﬁnities.
While the problem of ﬁnding an optimal data layout is
computationally hard [37], [38], we believe that utilizing
machine learning may lead to improved heuristics resulting
in performance gains for real-world applications.
7 C ONCLUSION
In this work we extended the state-of-the-art model for
reordering of basic blocks and developed a new efﬁcient
algorithm to optimize the layout of a binary. We also per-
formed an extensive evaluation of various block ordering
techniques on a variety of real-world applications. The
experiments indicate that the new technique can improve
the performance of binaries that have been manually tuned
over the course of their development and optimized using
conventional compiler optimizations. There are several in-
teresting aspects of our approach that we discuss next.
Firstly, our approach employs a machine learning toolkit
to build a desired objective for optimization. As our eval-
uation demonstrates, the resulting model outperforms the
classical one, as the new objective correlates very well with
the performance of large-scale binaries. A possible risk here
is to over-tune a model for a speciﬁc application and miss
important details that might affect performance. The exper-
iments with the SPEC benchmark imply that the models
based on maximizing the number of fall-through branches
are too simplistic for binaries that are not front-end bound.
Secondly, our study focuses on optimizing applications
built with particular compilers and running on a speciﬁc
hardware. A reasonable future work is to verify whether
the presented approach can be generalized to other use
cases. Our preliminary experiments indicate that compara-
ble gains can be achieved on other Intel microprocessors
and alternative processor architectures. Similarly, the new
reordering algorithm is applied as a post-link optimization,
and we did not examine how it behaves on earlier compila-
tion stages. It would be interesting to investigate the effect
of the reordering applied at compilation time. In particular,
we plan in the future to integrate and compare ext-tsp
with the algorithms implemented in GCC [12] and Clang.
Finally, we point out that this paper considers a certain
aspect related to code generation: reordering of basic blocks
within a function. There are many complementary optimiza-
tions that we did not investigate in detail, for example, un-
rolling loops or duplicating blocks in order to avoid extra
jumps. An attractive direction is to allow cross-procedure re-
ordering in which basic blocks from different functions can
be interleaved in the ﬁnal layout. This might further increase
code locality and improve cache utilization. Unfortunately,
our preliminary experiments with existing cross-procedure
heuristics [12] did not produce measurable gains; further
research of the technique is an intriguing future work.
11
ACKNOWLEDGMENTS
We thank Alon Shalita for fruitful initial discussions of
the project. We would also like to thank Rafael Auler and
Maksim Panchenko for their help with integrating the new
technique into BOLT.
REFERENCES
[1] D. Chen, D. X. Li, and T. Moseley, “AutoFDO: Automatic
feedback-directed optimization for warehouse-scale applications,”
in International Symposium on Code Generation and Optimization .
ACM, 2016, pp. 12–23.
[2] C.-K. Luk, R. Muth, H. Patil, R. Cohn, and G. Lowney, “Ispike:
A post-link optimizer for the Intel R⃝ Itanium R⃝ architecture,” in
Code Generation and Optimization: Feedback-Directed and Runtime
Optimization. IEEE Computer Society, 2004, p. 15.
[3] B. Schwarz, S. Debray, G. Andrews, and M. Legendre, “PLTO: A
link-time optimizer for the Intel IA-32 architecture,” in Workshop
on Binary Rewriting, 2001, pp. 1–7.
[4] G. Ottoni and B. Maher, “Optimizing function placement for large-
scale data-center applications,” in International Symposium on Code
Generation and Optimization. IEEE Press, 2017, pp. 233–244.
[5] M. Panchenko, R. Auler, B. Nell, and G. Ottoni, “BOLT: A practical
binary optimizer for data centers and beyond,” in International
Symposium on Code Generation and Optimization. IEEE Press, 2019,
pp. 2–14.
[6] K. Pettis and R. C. Hansen, “Proﬁle guided code positioning,” in
SIGPLAN Notices, vol. 25, no. 6. ACM, 1990, pp. 16–27.
[7] “Binary Optimization and Layout Tool,” https://github.com/
facebookincubator/BOLT.
[8] F. T. Boesch and J. F. Gimpel, “Covering points of a digraph with
point-disjoint paths and its application to code optimization,”
Journal of the ACM, vol. 24, no. 2, pp. 192–198, 1977.
[9] C. Young, D. S. Johnson, M. D. Smith, and D. R. Karger, “Near-
optimal intraprocedural branch alignment,” SIGPLAN Notices ,
vol. 32, no. 5, pp. 183–193, 1997.
[10] T. Hirata, A. Maruoka, and M. Kimura, “A polynomial time
algorithm to ﬁnd a path cover of a reducible ﬂow graph,” Syst.
Comput. Control, vol. 10, no. 3, pp. 71–78, 1979.
[11] B. Calder and D. Grunwald, “Reducing branch costs via branch
alignment,” in SIGPLAN Notices, vol. 29, no. 11. ACM, 1994, pp.
242–251.
[12] J. Torrellas, C. Xia, and R. L. Daigle, “Optimizing the instruction
cache performance of the operating system,” IEEE Transactions on
Computers, vol. 47, no. 12, pp. 1363–1381, 1998.
[13] A. Ram ´ırez, J.-L. Larriba-Pey, C. Navarro, J. Torrellas, and
M. Valero, “Software trace cache,” in International Conference on
Supercomputing. ACM, 2014, pp. 261–268.
[14] A. H. Ashouri, W. Killian, G. P . John Cavazos, and C. Silvano,
“A survey on compiler autotuning using machine learning,” ACM
Comput. Surv., vol. 51, no. 5, pp. 96:1–96:42, 2018.
[15] Z. Wang and M. O’Boyle, “Machine learning in compiler optimiza-
tion,” Proceedings of the IEEE, vol. PP , no. 99, pp. 1–23, 2018.
[16] C. Lattner and V . Adve, “LLVM: A compilation framework for
lifelong program analysis & transformation,” in International Sym-
posium on Code Generation and Optimization . IEEE Computer
Society, 2004, p. 75.
[17] D. Chen, N. Vachharajani, R. Hundt, X. Li, S. Eranian, W. Chen,
and W. Zheng, “Taming hardware event samples for precise and
versatile feedback directed optimizations,” IEEE Transactions on
Computers, vol. 62, no. 2, pp. 376–389, 2013.
[18] A. Nowak, A. Yasin, A. Mendelson, and W. Zwaenepoel, “Estab-
lishing a base of trust with performance counters for enterprise
workloads,” in USENIX Annual Technical Conference, 2015, pp. 541–
548.
[19] R. Levin, I. Newman, and G. Haber, “Complementing missing and
inaccurate proﬁling using a minimum cost circulation algorithm,”
in International Conference on High-Performance Embedded Architec-
tures and Compilers. Springer, 2008, pp. 291–304.
[20] D. Novillo, “SamplePGO: the power of proﬁle guided optimiza-
tions without the usability burden,” in Proceedings of the 2014
LLVM Compiler Infrastructure in HPC. IEEE Press, 2014, pp. 22–28.
[21] X.-H. Liu, Y. Peng, and J.-Y. Zhang, “A sample proﬁle-based opti-
mization method with better precision,” in International Conference
on Artiﬁcial Intelligence and Computer Science . DEStech, 2016, pp.
340–346.
[22] K. Adams, J. Evans, B. Maher, G. Ottoni, A. Paroski, B. Simmers,
E. Smith, and O. Yamauchi, “The HipHop Virtual Machine,” in
SIGPLAN Notices, vol. 49, no. 10. ACM, 2014, pp. 777–790.
[23] B. Letham, B. Karrer, G. Ottoni, and E. Bakshy, “Constrained
Bayesian optimization with noisy experiments,” Bayesian Analysis,
vol. 14, no. 2, pp. 495–519, 2019.
[24] K. L. Croxton, B. Gendron, and T. L. Magnanti, “A comparison
of mixed-integer programming models for nonconvex piecewise
linear cost minimization problems,” Management Science , vol. 49,
no. 9, pp. 1268–1273, 2003.
[25] E. Bakshy and E. Frachtenberg, “Design and analysis of bench-
marking experiments for distributed internet services,” in Interna-
tional Conference on World Wide Web, 2015, pp. 108–118.
[26] I. Bahar, B. Calder, and D. Grunwald, “A comparison of software
code reordering and victim buffers,” ACM SIGARCH Computer
Architecture News, vol. 27, no. 1, pp. 51–54, 1999.
[27] N. Gloy and M. D. Smith, “Procedure placement using temporal-
ordering information,” Transactions on Programming Languages and
Systems, vol. 21, no. 5, pp. 977–1027, 1999.
[28] R. Lavaee and C. Ding, “ABC Optimizer: Afﬁnity based code
layout optimization,” University of Rochester, Tech. Rep., 2014.
[29] A. H. Hashemi, D. R. Kaeli, and B. Calder, “Efﬁcient procedure
mapping using cache line coloring,” in SIGPLAN Notices, vol. 32,
no. 5. ACM, 1997, pp. 171–182.
[30] P . Li, H. Luo, C. Ding, Z. Hu, and H. Ye, “Code layout optimization
for defensiveness and politeness in shared cache,” in International
Conference on Parallel Processing. IEEE, 2014, pp. 151–161.
[31] A. Ramirez, L. A. Barroso, K. Gharachorloo, R. Cohn, J. Larriba-
Pey, P . G. Lowney, and M. Valero, “Code layout optimizations
for transaction processing workloads,” in SIGARCH Computer
Architecture News, vol. 29, no. 2. ACM, 2001, pp. 155–164.
[32] W. J. Schmidt, R. R. Roediger, C. S. Mestad, B. Mendelson, I. Shavit-
Lottem, and V . Bortnikov-Sitnitsky, “Proﬁle-directed restructuring
of operating system code,” IBM Systems Journal, vol. 37, no. 2, pp.
270–297, 1998.
[33] E. Raman, R. Hundt, and S. Mannarswamy, “Structure layout opti-
mization for multithreaded programs,” in International Symposium
on Code Generation and Optimization. IEEE, 2007, pp. 271–282.
[34] T. Eimouri, K. B. Kent, A. Micic, and K. Taylor, “Using ﬁeld access
frequency to optimize layout of objects in the JVM,” in Annual
Symposium on Applied Computing. ACM, 2016, pp. 1815–1818.
[35] P . Roy and X. Liu, “StructSlim: A lightweight proﬁler to guide
structure splitting,” in International Symposium on Code Generation
and Optimization. ACM, 2016, pp. 36–46.
[36] T. M. Chilimbi and R. Shaham, “Cache-conscious coallocation of
hot data streams,” in SIGPLAN Notices, vol. 41, no. 6. ACM, 2006,
pp. 252–262.
[37] E. Petrank and D. Rawitz, “The hardness of cache conscious data
placement,” in SIGPLAN Notices, vol. 37, no. 1. ACM, 2002, pp.
101–112.
[38] R. Lavaee, “The hardness of data packing,” SIGPLAN Notices ,
vol. 51, no. 1, pp. 232–242, 2016.