Perhaps the sensor has intrinsic nuclease activity that is essential for DNA recognition.

However, chimeric oligonucleotides with p-thio linkages at the ends and phosphodiester bonds in the center are fully active (D.B.S. and R.M., unpublished data).

Alterna- tively, the phosphodiester backbone may provide a sequence-independent binding site for the sensor and allow it to distinguish between DNA and RNA.


We found that intracellular DNA potently activates the IFN-b gene without detectable activation of NF-kB and MAP kinases.

Classical studies of this promoter estab- lished that robust transcriptional activation requires the coordinated assembly of several transcription fac- tors termed the enhanceosome (Maniatis et al., 1998).

These factors include IRF3/7, NF-kB p50/p65, and the MAP kinase-activated AP-1 (Atf2/c-Jun).

The vast ma- jority of studies defining the assembly of the enhanceo- some were done with RNA viruses, and the recent char- acterization of the RIG-I-dependent response to RNA viruses validates the classical model (Kato et al., 2005).

We suggest that ISD recognition might activate IFN-b transcription by using some but not all components of the enhanceosome.

Numerous studies have shown that overexpression of IRF3 alone robustly activates the IFN-b promoter, presumably without any direct con- tribution of NF-kB or MAP kinases, and multimerization of IRF3 binding sites renders an ectopic promoter virus inducible (Fujita et al., 1987).

We emphasize that live DNA viruses may activate these pathways via other li- gands or effects on cells, such as recognition of viral RNA transcripts or a global stress response to viral rep- lication.

Identification of the sensor and unique elements of the ISD signaling pathway will allow dissection of the components of the response to DNA viruses and intra- cellular bacteria.


We used microarray analysis to directly compare the gene expression patterns activated by ISD versus TLR9 and RIG-I/MDA5 and identified a unique but over- lapping response to intracellular DNA.

Together, RIG-I, MDA5, and the ISD pathway may be sufficient to recog- nize a majority of viruses based on the type and structure of nucleic acids exposed inside the cell.

Importantly, just as RNA viruses encode proteins that antagonize the RIG-I and MDA5 pathways (Andrejeva et al., 2004; Foy et al., 2005; Meylan et al., 2005), it is almost cer- tain that DNA viruses encode proteins that interfere with the ISD response.

Examination of these proteins will illuminate the specific mechanisms of DNA virus detection.


Finally, the existence of this pathway raises an impor- tant question of self/nonself discrimination: how does a sequence-independent sensor avoid recognition of the more than 10 9 base pairs of genomic DNA in all nu- cleated cells?

Exclusion from the nucleus is an attractive explanation, but the breakdown of the nuclear envelope during cell division suggests that other mechanisms might prevent autoreactivity.

Perhaps genomic DNA, with its myriad modifications and association with chro- matin components, is specifically marked as ''self.

'' This would suggest that a failure to mark the genome as ''self'' would result in activation of the sensor.

Such self-recognition could be beneficial in sensing genomic instability in severely damaged or neoplastic cells and may contribute to the important role for type I IFNs in tumor immunosurveillance (Dunn et al., 2005).

Alterna- tively, a loss of cell-autonomous tolerance to genomic DNA would result in IFN-dependent autoimmunity.


While these proposed functions are entirely speculative, we suggest that this pathway is responsible for the TLR- independent, type I IFN-mediated pathology observed in DNase II-deficient embryos, where liver macrophages become engorged with undigestable nuclei from eryth- rocyte precursors (Okabe et al., 2005).

Further charac- terization of this pathway will provide valuable insights into the mechanisms of nucleic acid recognition and host resistance to pathogens.


Mice, Cells, and Antibodies C57Bl/6 mice and IFNaR1 2/2 mice were purchased from Jackson Laboratories (Bar Harbor, ME).

MyD88 2/2 , Trif 2/2 , and TLR9 2/2 mice were kindly provided by Dr. Shizuo Akira.

RIP2 2/2 mice and MEFs have been described (Kobayashi et al., 2002).

Ku70 2/2 and DNA-PK 2/2 bone marrow and Atm 2/2 MEFs were generously pro- vided by Dr. Frederick Alt.

IRF3 2/2 MEFs were kindly provided by Dr. Tadatsugu Taniguchi.

All animals were maintained in accordance with guidelines of the Yale Institutional Animal Use and Care Com- mittee.

RAW264.7 cells stably expressing an NF-kB-luciferase re- porter were generated by standard procedures.


Antibodies to MHC Class II (I-A b ), CD86, CD40, CD11c, and p38a were from BD Biosciences (San Diego, CA).

Antibodies to SAPK/JNK and p44/42 MAPK (Erk1/2) were from Cell Signaling Technologies (Beverly, MA).

Antibodies to IFN-a were from HyCult Biotechnology (the Neth- erlands) and PBL laboratories (Piscataway, NJ), and monoclonal an- tibody to murine IFN-b was from Seikagaku America (East Falmouth, MA).


Calf thymus genomic DNA was from Sigma (St. Louis, MO).

Poly I:C was from Amersham (Piscataway, NJ).

All oligonucleotides were synthesized at the Yale W.M. Keck facility, with p-thio linkages rep- resented as lowercase: ISD (sense), TACAGATCTACTAGTGATC TATGACTGATCTGTACATGATCTACA; CpG, tccatgacgttcctgacgtt.


Equimolar amounts of ISD and their antisense oligos were an- nealed in PBS at 75ºC for 30 min before cooling to room tempera- ture.

ISD-reverse was the exact reverse sequence of that listed above.


Macrophages were infected with Listeria monocytogenes strain 10403S at a moi of 2 bacteria per cell.

Gentamycin (50 mg/ml) was added to cells 1.5 hr postinfection to kill extracellular bacteria.

For in vivo infections, mice were injected intravenously with 5000 L. monocytogenes.

Apoptotic thymocytes from MyD88/Trif DKO mice were prepared by a 4.5 hr treatment with 10 mM dexametha- sone.

Washed cells were loaded with recombinant LLO (100 ng per 10 6 cells) for 15 min in ice-cold PBS, washed, and added to MyD88/Trif DKO peritoneal-derived macrophages at a ratio of 5:1.


Listeria extracts were prepared from a pelleted 5 ml culture freeze- thawed in liquid nitrogen, resuspended in 400 ml PBS containing 1 mg/ml lysozyme (Roche), and sonicated for 2 min using a VirSonic 101 Digital 600 Cell Disrupter (VirTis, Gardiner, NY).

Remaining debris was removed by centrifugation, and cleared extracts were adjusted to contain 2 mM MgCl 2 , 50 mM KCl, and 20 mM Tris-Cl (pH 8). 100 ml aliquots were treated separately with DNase I (Invitrogen, Carlsbad, CA; 100 units/ml) or RNase (Qiagen, Valencia, CA; 100 mg/ml) for 45 min at 37ºC. EDTA was added to 2.5 mM and samples were heated for 10 min at 70ºC. Extracts were again cleared by a 5 min centrifugation, and 5-10 ml were complexed with Lipofectamine 2000 (Invitrogen) and transfected as described below.


Legionella pneumophila strains and opsonizing antibody were generously pro- vided by Dr. Craig Roy.

ISD were complexed with Lipofectamine 2000 (Invitrogen) at a ratio of 1 ml lipofectamine to 1 mg DNA.

Cells in 12-well plates were stimulated with 5 mg ISD.

Calcium phosphate transfections were done in the presence of 50 mM chloroquine by standard techniques.

CpG was added directly to the media at 3 mM. E. coli LPS (Sigma) was used at 100 ng/ml.


For analysis of IFN-a production, supernatants were harvested 18 hr poststimula- tion and used for a sandwich ELISA.

For quantitative RT-PCR, cells were harvested into RNA-Bee (Teltest, Friendswood, TX).

RNA was reverse transcribed with Superscript II (Invitrogen), and cDNAs were used for PCR with Quantitect SYBR Green reagents (Qiagen) on a Stratagene MX3000 bioanalyzer (La Jolla, CA).

The abundance of each cytokine mRNA was normalized to HPRT expression and com- pared to untreated cells to calculate the relative induction.


cDCs or macrophages were stimulated for 4 hr and harvested into RNA Bee.

Sample preparation and hybridization to Affymetrix Mouse genome 430 2.0 arrays were performed at the Yale W.M. Keck facility.

The complete data set has been submitted to the GEO database (http://www.ncbi.nlm.nih.gov/geo; submission #GSE2197).


Whole-cell lysates from macrophages stimulated for the indicated times were separated by SDS-PAGE, transferred to PVDF mem- branes, and probed with antibodies specific for the indicated pro- teins.

Gel shift analysis was performed with standard techniques.


Induction of type I interferons (IFNs) by viruses and other pathogens is crucial for innate immunity, and it is mediated by the activation of pattern-recognition receptors, such as Toll-like receptors and cytosolic receptors such as RIG-I and MDA5.

The type I IFN induction is primarily controlled at the gene transcriptional level, wherein a family of transcription factors, interferon regulatory factors (IRFs), plays central roles.

Here, we summarize the recent studies on IRFs, providing a paradigm of how genes are ingeniously regulated during immune responses.

We also consider some evolutional aspects on the IFN-IRF system.


The IFN Genes: A Brief Overview IFNs, so called because of their activity to interfere (inhibit) virus replication in a cell (see Vilcek, 2006, in this issue), are divided into at least three distinct types: types I, II, and III [1].

Type I IFNs are composed of various genes including IFN-a, -b [2], and others, such as IFN-u, -3, and -k [1].

In humans and mice, the IFN-a genes are composed of more than 13 subfamily genes (13 in humans and 14 in mice), whereas only a single IFN-b gene is found [3]; all of these IFN genes are clustered in one locus on the same chromosome (on chromosome 9 in humans and on chromosome 4 in mice).

Type II IFN is referred to as IFN-g, the gene which exists in a single copy; this gene is structurally unrelated to type I IFNs and is typically induced in cells of the immune system such as T cells or natural killer (NK) cells [4].

Recently, some new IFN gene members, namely IFN-l1, -l2, and -l3 (also known as interleukin-29 [IL-29], IL-28A, and IL-28B, respectively) genes, have been identified and classified as type III IFN [1].

Type III IFN genes are induced in virally infected cells, and these IFN genes may also be induced by mechanisms similar to those for IFN-a and -b genes.

The mechanisms of signal transduction by distinct types of IFNs are described elsewhere [4, 1, 5].

Type I IFN production is primarily controlled at the gene transcription level; hence, the regulatory mechanism underlying IFN gene transcription has been a focus of extensive studies even beyond the context of immunity, offering a paradigm on how quiescent genes are "switched on and off" in mammalian cells.

In particular, mechanisms of IFN-b gene induction have been extensively studied over the past few decades.

Indeed, as described below, the IFN-b gene induction occurs as a highly ordered process, and it is regulated by multiple transcription factors.


Type I interferons (IFNs) are a family of cytokines specialized to coordinate immunity to viruses and other intracellular infections.

In the past several years, many of the receptors and signaling pathways that link pathogen detection to induction of type I IFNs have been identified and characterized.

An integrated picture has emerged in which type I IFNs have essential functions in several seemingly disparate processes: they restrict viral spread by engaging machinery that ultimately cripples and kills infected cells, yet they are also positively linked to the activation and expansion of lymphocytes that are important for control of intracellular infections.

These advances highlight the context-specific actions of type I IFNs and clarify the multiple points at which they are integrated into both innate and adaptive immunity.


Viruses are the most abundant and diverse pathogens on earth.

One look at the genome of any organism confirms that viruses have been around for a very long time.

It is estimated that up to 40% of the human genome is derived from fragments of ancestral viral sequence, and approximately 1% of the genome consists of intact endogenous retroviruses [1].

The impact of viruses on our genes cannot be overstated: viruses are likely responsible for generating much of the variation that fueled selection of new traits and the evolution of our species, including exon shuffling, gene duplication, and diversification of noncoding regulatory elements.


Viruses can also be formidable pathogens and pose a unique challenge for the host’s detection systems.

It is now well established that immune recognition focuses on highly conserved, invariant structures of microbes that are distinct from the host [2].

Unlike bacteria and fungi, which contain easily distinguishable microbe-specific structures, viruses are made entirely of host-derived components.

With no microbe-specific “pattern” to latch onto, the receptors that detect viruses have instead evolved to recognize the one feature that is shared without exception among all viruses: a genome composed of nucleic acids.

Importantly, nucleic acid recognition is in principle an imperfect way for cells to distinguish self from non-self for obvious reasons, yet most (if not all) ways for a cell to signal the presence of viral infection are based on it.

The receptors that accomplish this are generally very successful at identifying viral nucleic acids in the midst of an abundance of host-derived RNA and DNA.

Because there is little structural basis for discrimination between viral and host nucleic acids, it appears that pattern recognition in this case is aided by the compartmentalization of antiviral sensors and by activities of host nucleases that help eliminate self nucleic acids from the extracellular fluids and from lysosomes [3][4][5].

Additional mechanisms that help such discrimination undoubtedly exist and await elucidation; insight into this fundamental question is attainable now that we know the identity of many of the receptors involved, as discussed below.


All organisms are susceptible to viral infections and all have a dedicated system to detect and prevent nucleic acid parasitism.

These systems range from restriction enzymes and antiphage defense in prokaryotes to the multilayered and complex networks of antiviral responses in higher metazoans.

Each of these systems directs responses that vary according to the organism’s physiology and needs.

Jawed vertebrates, which evolved an adaptive immune system of T and B cells, also developed the type I interferon (IFN) cytokine family that is dedicated to signaling the presence of intracellular infection and facilitating communication among the cells that provide defense against viruses and intracellular bacteria.

All vertebrates examined to date have a gene that encodes IFN-b and at least two that encode IFN-a, together with natural killer (NK) cells, T cells, and B cells; there is no known organism that has one of these components without all of the others.

The preservation of type I IFNs throughout the vertebrate lineage suggests an intimate functional linkage to adaptive immunity that has become increasingly apparent with recent experimental insights.


In this review, we will discuss the major receptor systems that detect viral infection and activate type I IFNs in vertebrates, with an emphasis on how these systems complement each other in host defense.

We will attempt to integrate important recent developments into the well-established framework of IFN function that is discussed in the accompanying reviews [6][7][8] and other papers [9][10][11][12].

These exciting advances clarify the mechanisms by which IFNs are activated, provide insight into their multiple and context-specific roles in antiviral immunity, and may suggest new directions for therapeutics and vaccine development.


Two Pathways for Type I IFN Induction: Toll-like Receptors Prior to the year 2000, no mammalian receptors that linked viral recognition to activation of type I IFNs had been identified.

After several years of remarkable progress, it is now clear that two complementary receptor systems account for most virus detection.

A general principle that has emerged is that all of these receptors detect nucleic acids.

Furthermore, these receptors fall into two categories: one class of receptors (exemplified by RIG-I and MDA5 proteins) is expressed ubiquitously and is localized to the cell’s cytosol where it detects viral nucleic acids produced upon infection.


The second class of receptors (members of the Toll-like receptor [TLR] family) detects viral nucleic acids in endosomes and only in specialized cell types.

This latter mode of recognition does not require that the IFN-producing cells are infected themselves and hence need not be ubiquitous.

The Toll-like receptors (TLRs) have been the subject of intense investigation as prototypical signaling receptors that activate immunity in response to microbial stimuli [13].

Several of the mammalian TLRs recognize different types of nucleic acids, and together they provide enough coverage to detect most types of viruses.


TLR3 recognizes double-stranded RNA, a common feature of both DNA and RNA viruses that until recently was thought to be the principal (or even only) form of viral nucleic acid recognized by the immune system [14].

TLR7 and TLR8 detect single-stranded RNA (ssRNA) and viruses that contain ssRNA genomes [15][16][17].

TLR9 detects unmethylated CpG motifs in DNA that are common in DNA viruses and also bacteria [18][19][20].


The only known exception to the rule that all receptors that activate type I IFNs detect nucleic acids is TLR4, which detects lipopolysaccharides (LPS) derived from gram-negative bacteria and activates type I IFNs in conventional dendritic cells (DCs) and macrophages through a signaling pathway that utilizes the adaptor protein Trif [21][22].

It is completely unclear at present why the TLR4-Trif pathway links recognition of bacterial cell walls to cytokines that generally signal the presence of a viral infection.

By comparison, TLR2 also recognizes cell wall components (peptidoglycans) but does not activate IFNs [23][24].

It is likely that the TLR4-Trif axis arose to harness some of the functions of IFNs in enhancing T cell differentiation and NK cell function, as discussed below.


TLRs are primarily expressed on key sentinel cells of the innate immune system: macrophages and dendritic cells (DCs).

TLR3 is expressed by conventional DCs, whereas TLRs 7 and 9 are expressed by both conventional and plasmacytoid dendritic cells (pDCs).

pDCs have a specialized signaling pathway that links activation of TLRs 7 and 9 to the production of copious amounts of type I IFNs, and they are primarily responsible for the systemic concentrations of IFNs detected during many experimental viral infections [25][26].

One feature shared by the four TLRs that sense nucleic acids is their intracellular localization: unlike all other TLRs, which are located at the cell surface, TLRs 3, 7, and 9 signal from within an acidified endosomal compartment.

Consequently, inhibitors of endosomal acidification block the function of these three TLRs but none of the others [27][28][29].

This intracellular localization is important for the ability of these TLRs to discriminate between self and non-self nucleic acids: relocating TLR9 to the cell surface abolishes its ability to respond to virus-encapsulated DNA but enables recognition of self-derived genomic DNA in the extracellular milieu [3].


A key feature of TLR-dependent nucleic acid recognition is directly tied to their intracellular location: TLRs sample material entering cells from the outside and thus do not detect the presence of infection from within.

This material often consists of viral particles that become uncoated or degraded upon endosomal acidification, thus exposing their nucleic acids for recognition.

Recent evidence indicates that TLR3 also samples apoptotic cells for viral dsRNA as they are engulfed and degraded in the phagosome [30].

Considering the strong link between viral infection and apoptosis in vertebrates and the role of TLRs in antigen presentation, this mechanism would provide an efficient means for viral antigens within dead cells derived from any tissue to be presented to T cells by uninfected DCs, a process known as crosspresentation.

The separation of viral infection from viral recognition in DCs has another important consequence: most viruses encode proteins that potently antagonize type I IFN expression and signaling within infected cells, but uninfected DCs would be refractory to such inhibition.


Two Pathways for Type I IFN Induction: Cytosolic Receptors
Although some TLRs can activate IFNs in a subset of specialized cells, almost all nucleated cells respond to viral infection by producing type I IFNs.

Fujita and colleagues advanced our understanding of this more broadly expressed virus-detection system by identifying the caspase recruitment domain (CARD)-containing RNA helicases retinoic acid inducible gene-I (RIG-I) and melanoma differentiation antigen 5 (MDA5) as sensors that link detection of dsRNA to activation of type I IFNs [31].

Both RIG-I and MDA5 use the CARD-containing adaptor protein MAVS (also known as IPS-1, VISA, or CARDIF) to activate the IFN-b gene [32][33][34][35].

The details of the RNA-activated signaling pathways have been reviewed elsewhere [36][37], and recent developments will also be discussed in the accompanying review [6].


Knockout mice lacking RIG-I and MDA5 have dramatic phenotypes when challenged with RNA viruses in vitro and in vivo.

RIG-I-deficient cells fail to activate type I IFN production when challenged with several classes of viruses, including vesicular stomatitis virus (VSV), Newcastle disease virus, Sendai virus, influenza virus, and Japanese encephalitis virus [38][39].

Interestingly, pDC-derived type I IFN production in response to these viruses requires MyD88-dependent TLR signaling but not RIG-I, highlighting the fact that TLRs and cytosolic sensors can independently activate type I IFNs in vivo [38].

MDA5 is the primary cytosolic receptor for the RNA polymer poly I:C and is required for detection of the picornavirus encephalomyocarditis virus (EMCV) [40][39].

Together, these studies establish that RIG-I and MDA5 detect different viral RNA structures, although the precise nature of these structures and the biochemical basis for this differential recognition remains unknown.


The generation of MAVS-deficient mice demonstrated that both RIG-I and MDA5 use this adaptor protein to activate type I IFNs [41][42].

Consequently, the phenotype of MAVS-deficient mice resembles the sum of RIG-I and MDA5 deficiency when challenged with RNA viruses that are detected by these helicases.

Importantly, MAVS-deficient mice are highly susceptible to infection with these viruses despite an intact TLR-dependent recognition system.

This interesting observation formally demonstrates that although the two receptor systems can independently activate the type I IFN response, they are not functionally redundant.

As discussed below, this also implies that type I IFNs are not sufficient for antiviral defense and that these cytosolic receptors may also control essential antiviral functions in an IFN-independent manner.


In addition to intracellular recognition of viral dsRNA, which has been known to exist for many years, foreign DNA is also recognized by an intracellular sensor that triggers type I IFN production [43][44][45].

This DNA-recognition system presumably evolved to detect DNA derived from DNA viruses during infection.

Thus, similar to RIG-I and MDA5, DNA recognition is a cell-autonomous and, presumably ubiquitous virus-recognition system.

The sensor for intracellular viral DNA is currently unknown.

Interestingly, however, the signaling pathway induced by this sensor is distinct from RIG-I and MDA5 signaling in that it does not result in activation of NF-kB transcription factor or MAP kinases, but does require IRF3 for the induction of interferon genes [45].

Consistent with the different downstream signaling pathway, DNA-induced interferon production is MAVS independent [41][42].

The reason why dsRNA and DNA trigger different signaling pathways presumably has to do with the induction of distinct effector responses to RNA and DNA viruses.

These two types of viruses differ dramatically with regards to their size and infection strategies, and so it is not surprising that different antiviral responses need to be induced to interfere with stages of the infection cycle unique to RNA or DNA viruses.


Interestingly, a potent type I interferon response is also induced by several intracellular bacterial infections.

This has been most clearly demonstrated in the case of Listeria monocytogenes [46], although other intracellular bacteria like Shigella flexneri and Legionella pneumophila induce interferon responses as well [47][45].

It appears that the induction of IFN-b in Listeria-infected macrophages may be mediated by the DNA-recognition pathway [45].

Consistent with this possibility, IFN-b induction by Listeria is TLR and nucleotide binding oligomerization domain (NOD) protein independent, but requires invasion of the host cell’s cytosol [48][46][49].

In the case of Listeria, IFN-b production and type I IFN-activated signaling in lymphocytes makes the host more susceptible to infection [50][51][48].

It is important to note that although type I IFNs appear to be detrimental to the host during Listeria infection, they are likely only one facet of the response activated by the sensor(s) that detect cytosolic Listeria; this pathway probably induces other cytokines and effector mechanisms, some of which might be beneficial to the host.

Nonetheless, this finding is a dramatic example of context-dependent effects of type I IFNs in the course of infection, which appears to be a common theme in IFN biology, as discussed below.

In the case of other bacteria that invade the host cytosol, it remains to be seen whether IFNs have a protective or detrimental effect at different stages of infection.


In summary, two distinct IFN-inducing systems are now known to exist: one is restricted to endosomes of specialized sentinel cells and mediated by TLRs, and the other is a ubiquitous cytosolic viral detection system mediated by RIG-I and MDA5 and a DNA sensor(s).

In principle, any type of virus would be vulnerable to detection by both a TLR and a cytosolic sensor; this breadth of coverage is remarkable considering the staggering diversity of viruses and the comparably tiny number of receptors charged with sensing them.

It would be interesting to determine whether retroviruses, which generate both RNA and DNA during their infectious cycle, are detected by sensors of both of these nucleic acids or whether one type of receptor predominates.

Based on the recent insights into the receptor systems that sense nucleic acids and the generation of knockout mice lacking specific components of each pathway, we are now able to evaluate the relative contributions of each of these systems to antiviral immunity.

What is beginning to emerge from these studies is a more integrated picture of the different ways in which type I IFNs facilitate antiviral defense through both cell-intrinsic and non-cell-autonomous mechanisms.


Cell-Intrinsic Antiviral Defense: The “Antiviral State”
The activity of type I IFNs was originally described almost 50 years ago as a soluble factor produced by cells treated with inactivated, nonreplicating viruses that blocked subsequent infection with live virus [52][53][54].

Interestingly, one group used the RNA influenza virus whereas the other used the DNA vaccinia virus, implying the existence of cellular sensors of both types of viral nucleic acids even in the first reports describing type I IFNs.

As indicated in these initial studies, type I IFNs produced by infected cells act in an autocrine and paracrine manner to signal the presence of viral infection.

By signaling through the type I IFN receptor, IFNs activate the inducible expression of hundreds of genes that together establish the “antiviral state” [7][12].

Antiviral state is a generic term and perhaps belies the fact that we still have only an elementary understanding of what exactly it is.

Indeed, there are entire families of IFN-inducible genes with completely unknown function, including several large GTPases and a series of proteins with tetratricopeptide repeat domains (IFITs) [55].

Although the function of each of these and other IFN-inducible genes awaits elucidation, it is safe to say that they act to cripple every stage of the viral life cycle.


Some IFN-inducible proteins have broad antiviral effects.

For example, 2'-5' oligoadenylate synthases (OAS) are activated by viral dsRNA and produce 2'-5' oligoadenylates, which in turn activate the latent nuclease, RNase L, resulting in the degradation of viral RNA transcripts as well as host RNAs [12].

Another well-studied IFN-induced antiviral effector is PKR, a member of the eukaryotic initiation factor 2a (eIF2a) kinase family.

Activation of PKR by dsRNA leads to eIF2a phosphorylation with a consequent blockade of translation of most cellular and viral mRNAs.

Interestingly, translation of mRNAs containing upstream open reading frames in their 5' UTRs is enhanced upon eIF2a phosphorylation.

Most known examples of proteins encoded by such mRNAs are restricted to the unfolded protein response and nutrient deprivation or stress conditions that activate other members of the eIF2a kinase family [56].

Conceivably, these proteins may indirectly contribute to the antiviral state.

However, one would expect that some of the bona fide antiviral effectors, including perhaps type I IFNs themselves, may also be translated preferentially when PKR is activated and eIF2a phosphorylated.


In addition to the relatively nonspecific antiviral activities of PKR and OAS-RNaseL systems, certain IFN-inducible genes may also exhibit specificity for distinct classes of viruses as implied by the existence of distinct sensors and signaling pathways that detect DNA or RNA within infected cells.

Recent studies demonstrate that there are indeed IFN-inducible genes that are able to intercept specific types of viruses.

The MX and guanylate binding proteins are large, IFN-inducible GTPases of the dynamin family.

The MX proteins have been shown to have a protective effect against two RNA viruses, influenza and VSV [57][58].

Although detailed mechanisms of MX and GBP functions await elucidation, their structural similarity with dynamin suggests that these proteins may interfere with viral assembly and trafficking in the cell.


Perhaps the best example of virus-specific functions of IFN-inducible genes and the most promising recent development in the search for novel HIV therapeutics is the recently discovered APOBEC3 family of cytidine deaminases.

It was established several years ago that HIV deficient in the virion infectivity factor (vif) is only able to form infectious viral particles in certain cell lines (termed permissive) and not in human T cells and several other cell lines (termed nonpermissive) [59][60].

The phenotype of the nonpermissive cells was interesting: they supported a single round of virus production, but the newly formed virions were unable to infect even permissive cells.

Fusion of permissive and nonpermissive cells established that the nonpermissive phenotype was dominant and suggested that a cellular factor expressed only in nonpermissive cells could suppress infectivity of vif-deficient HIV [61].

Malim and colleagues used subtractive cDNA analysis to identify CEM15 (also known as APOBEC3G) as this factor and showed that it is incorporated into vif-deficient, but not wild-type, virions [62].

Virion-incorporated APOBEC3G deaminates cytosine to uracil in reverse-transcribed retroviral DNA, generating catastrophic mutations and rendering viral DNA susceptible to cleavage by uracil DNA glycosylase and apurinic-apyrimidinic (AP) endonuclease enzymes [63][64].

Interestingly, the deaminase activity of APOBEC3G is not required for its antiretroviral function [65], and cellular (non-virus-incorporated) APOBEC3G potently blocks vif+ HIV infection in resting CD4+ T cells [66], suggesting that the regulation of APOBEC3G function may be more complex than initially suggested.

Nonetheless, these findings provide a remarkable example of IFN-inducible genes targeting a specific class of viruses.


Although the effector proteins induced by type I IFNs are clearly an essential component of the antiviral state, mounting evidence suggests that these cannot mediate antiviral immunity on their own and instead cooperate with numerous other signaling pathways to activate the full complement of antiviral functions.

For example, MAVS-deficient mice infected with VSV suffer severe mortality despite a strong systemic type I IFN response provided by TLR-dependent recognition in pDCs [42].

Moreover, fibroblasts lacking Fas-associated death domain protein (FADD), an important signaling molecule in the RIG-I and MDA5 pathways, are unable to curtail VSV replication in vitro, even when pretreated with recombinant type I IFNs [67].

Thus, although the paracrine function of type I IFNs is a very efficient way for infected cells to disseminate an alarm signal, these findings suggest that IFN-inducible effectors on their own are not sufficient to confer immunity and highlight the other key roles that type I IFNs play in coordinating the antiviral response.


Cell-Intrinsic Antiviral Defense: Type I IFNs and Apoptosis
It is well known that treatment of cells with type I IFNs sensitizes them to apoptosis upon subsequent viral infection [12].

The rationale for this response is fairly simple: because viruses require host cell machinery to replicate, elimination of an infected cell would shut down this machinery and thus prevent viral spread.

Many viruses have evolved mechanisms to interfere with host cell apoptosis, reflecting the importance of this process in antiviral defense [68].


How do interferons sensitize cells to apoptosis?

Several overlapping mechanisms likely exist.

PKR is required for apoptosis of cells in response to polyI:C transfection, which activates MDA5 [69].

Paradoxically, PKR-deficient fibroblasts are far more sensitive to apoptosis when infected with VSV; this apoptotic response in PKR-deficient cells is correlated with an inability to curtail VSV replication and can be reversed by pretreatment of the cells with type I IFNs [69].

The tumor suppressor p53 also may play an important role in virus-induced apoptosis.

p53 is interferon inducible, p53-deficient cells are defective in VSV-induced apoptosis, and p53-deficient mice are highly susceptible to in vivo infection with VSV [70].

It is also possible that signaling from the cytosolic nucleic acid receptors might cooperate with type I IFN signaling to activate apoptosis in infected cells.

The observation that the RIG-I and MDA5 signaling adaptor MAVS is localized to the outer membrane of mitochondria suggests the interesting possibility that this pathway may interact with Bcl2 family members, which modulate apoptosis from the mitochondrial membrane [34].


The strong link between type I IFNs and cell-intrinsic apoptosis may be an important clue to the evolutionary significance of why the type I IFN system exists in vertebrates and not in any other class of organisms.


Perhaps it is because vertebrates are unique in the animal kingdom in that they use apoptosis as a potent form of host defense.


When would the benefits of apoptosis (removal of an infected cell) outweigh the costs (loss of an important cell) enough to make it a useful form of defense? Only when tissue renewal can prevent impairment of physiological functions caused by irreplaceable cell loss.


Most cells of adult invertebrate animals are postmitotic, which would make the deliberate removal of even a few key cells in invertebrates highly deleterious.


The few examples of renewable tissue in invertebrates, such as the tentacles of octopi and starfish and the segments of earthworms, are clearly secondary adaptations and occur by de novo regeneration instead of by continuous replacement of individual cells over time.


In contrast, most cells in vertebrate animals are renewable and continually replenished throughout adult life.


There are exceptions to this general rule, including certain postmitotic neuronal populations, and the rate of cell renewal varies widely among tissues, with epithelial cells and hepatocytes at the faster end of the spectrum and tissues like heart muscle at the slower end.


It would be interesting to determine whether a cell’s utilization of the type I IFN system for apoptosis is directly proportional to its “renewability.”


This would imply that a postmitotic, irreplaceable neuron would respond to type I IFNs quite differently than an easily renewable epithelial cell; even the signaling downstream of the widely distributed cytosolic nucleic acid sensors might vary considerably depending on the type of infected cell.


A deeper understanding of this specificity in the type I IFN system would have important implications for clinical treatment of viruses with cell type-specific tropism.


Interestingly, the other class of organisms that uses programmed cell death for defense also has renewable tissues: plants activate a hypersensitivity response to eliminate host cells surrounding sites of infection and prevent pathogen spread.


Instead of type I IFNs, plants use inducible production of other types of rapidly diffusing soluble mediators to relay an alarm signal among their noncirculating cells [71].


Non-Cell-Autonomous Roles of Type I IFNs
In addition to cell-intrinsic effects of type I IFNs that confer the antiviral state, the IFN system is linked to a variety of effector responses of the innate and adaptive immune systems.

One characteristic shared by IFN-regulated effector responses is that their activation ultimately results in the elimination of infected cells.

Thus, two major effector cell populations regulated by type I IFNs are natural killer (NK) cells and cytotoxic T cells (CTLs).

In addition, type I IFNs have been shown to facilitate cross-presentation by DCs for presentation of viral antigens to CD8+ T cells [72].

Cytotoxic responses mediated by NK cells and CTLs are designed to eliminate infected cells, so it appears that a common feature of IFN activity is the defense against intracellular infections.


The mechanisms whereby type I IFNs contribute to NK and CTL responses are diverse and include production of chemokines that recruit the cytotoxic cells to the site of infection and induction of cytokines that positively regulate cytotoxic cell numbers and activities.

For example, type I IFNs induce production of IL-15, which plays a critical role in proliferation and maintenance of NK cells and memory CD8+ T cells [73][74].

IFNs also directly activate NK cells to enhance their cytotoxic activity [75][73].


A number of studies have demonstrated that type I IFNs also play an essential role in the differentiation and function of effector CD8+ T cells.

Murali-Krishna and colleagues explored the role of type I IFNs in the CD8+ T cell response to infection with lymphocyte choriomeningitis virus (LCMV), a well-characterized model of CD8+ T cell function and memory [76].

They found that CD8+ T cells lacking the type I IFN receptor were profoundly impaired in their ability to expand and differentiate into effector CTLs, demonstrating that type I IFNs provide a nonredundant costimulatory signal in vivo.


A study by Taniguchi and colleagues suggests a more context-specific role for type I IFNs in CD8+ T cell differentiation.

Knockout mice lacking interferon regulatory factor 7 (IRF7), a transcription factor required for TLR9-mediated type I IFN production in pDCs, were unable to mount a CD8+ T cell response to protein antigens delivered with the TLR ligand CpG DNA as an adjuvant [77].

In contrast, CD8+ T cell expansion and function in IRF7-deficient mice was normal when TLR2 ligands were used as the adjuvant.

Acute depletion of pDCs in wild-type mice gave similar results and selectively impaired CD8+ T cell responses when protein antigen was administered with TLR9 ligands but not when the same antigen was delivered with TLR2 ligands.

TLR2 is expressed by conventional DCs and does not activate type I IFNs, so it is reasonable that the CD8+ T cells that emerge after TLR2 activation would not require IFNs.


However, TLR9 can also activate conventional DC maturation and production of inflammatory cytokines such as IL-12, yet this conventional DC-derived pathway is insufficient to compensate for loss of pDC-derived, TLR9-activated type I IFNs in IRF7-deficient mice.

One interesting question raised by these findings is whether pDCs are the relevant antigen-presenting cells after protein and CpG DNA immunization or whether they provide type I IFNs in trans to CD8+ T cells encountering peptide-MHC complexes on a different antigen-presenting cell.

Another question with implications for vaccine development is whether this selective requirement for pDCs and type I IFNs revealed with TLR9 ligands extends to the RNA ligands for TLR7.

If so, this would imply that TLR activation in pDCs drives the differentiation of a specialized type of CD8+ effector T cells that is distinct from those that encounter antigens presented by conventional DCs.


Together with the in vivo studies that use LCMV infection, it now appears that there are at least two types of CD8+ T cells: those that require type I IFNs for expansion and function and those that do not.

Thus, vaccines aiming to elicit protective CD8+ T cell memory against viral infection should target the pDC-mediated TLR7 and TLR9 response, whereas vaccines against other pathogens like gram-positive bacteria should be designed to expand the type I IFN-independent subset of CD8+ T cells.

Because these two types of effector CTLs are generated by different pathways, it is likely that type I IFN-dependent memory CTLs may not be efficient against a pathogen that does not trigger type I IFN production; conversely, the IFN-independent CTLs would be unable to provide protective immunity to viral infection.


Interestingly, the effect of type I IFNs on CD8+ T cells is dependent on the stage of CD8+ T cell differentiation.

Whereas type I IFNs limit nonspecific CD8+ T cell expansion in a STAT1-dependent manner, they promote antigen-specific expansion and IFN-g production by CTLs in a STAT4-dependent fashion [78].

Thus, the relative abundance of STAT1 and STAT4 in CD8+ T cells determines the outcome of their exposure to IFNs.

This context-dependent effect of type I IFNs is characteristic of their activity, which we will discuss in the next section.


Context-Specific Actions of Type I IFNs
An emerging principle in type I IFN biology is the realization that type I IFNs are part of a contextual language that can mean different things to different cells or even within the same cell depending on the presence of other cues.

A simplified model that summarizes some of the contextual meanings of type I IFNs discussed above is depicted in Figure 1.

A cell encountering type I IFNs in the absence of other cues will activate the classical antiviral state by upregulating expression of interferon-inducible genes (Figure 1A).

These genes include nucleic acid sensors like RIG-I and MDA5; increased abundance of these sensors would make cells surrounding an infected cell more sensitive to detection of virus.

Within an infected cell, type I IFN signaling can pair with cell-autonomous detection of viral nucleic acids (Figure 1B).

It is likely that these two pathways collaborate to distinguish an infected cell from its uninfected neighbors in at least two ways.

First, these two pathways may synergize to activate cell-intrinsic apoptosis in infected cells.

Second, they may result in modulation of cell-surface ligands that instruct NK cells and CTLs to kill an infected cell that cannot commit apoptosis on its own.

Importantly, the type I IFNs in this case need not be autocrine and could be supplied in trans by pDC-derived systemic IFNs; however, cell-intrinsic nucleic acid recognition must be a prerequisite for apoptosis and modulation of NK and CTL ligands.


Paradoxically, in CD8+ T cells receiving appropriate T cell receptor and costimulatory signals, type I IFN signaling is essential for the massive expansion and differentiation of virus-specific effector CTLs (Figure 1C).

A similar situation exists in NK cells, where type I IFN signaling collaborates with activating NK receptors to enhance NK cell-mediated cytotoxicity.

It is understandable that virus-specific CTLs would be placed under the control of cytokines that specifically announce the presence of virus.

But what would happen if the CD8+ T cells were themselves infected? This model suggests that cell-intrinsic viral detection would revert infected CD8+ T cells to the state depicted in Figure 1B and implies that cell-intrinsic nucleic acid detection is dominant over activation and differentiation cues when paired with type I IFN signals.

This cell-autonomous antiviral state would arrest the cell cycle, block host cell mRNA translation, and drive cells to commit apoptosis, thus preventing the unwanted scenario of infected cells increasing their numbers.

Importantly, “bystander” activation of CD8+ T cells by type I IFNs in the absence of a TCR and costimulatory signal would revert them to a conventional antiviral state.


Finally, type I IFN signaling in DCs, when paired with a TLR signal, would favor crosspresentation of antigens within the TLR-containing phagosome (Figure 1D).

During crosspresentation, viral antigens from infected cells or viral particles engulfed by DCs access the MHC class I loading machinery.

The fact that type I IFNs both facilitate this presentation of virus-derived peptides and also are essential for differentiation of the CD8+ T cells specific for these crosspresented antigens may provide a dual safeguard against inappropriate expansion of non-antigen-specific CTLs.


There is important clinical relevance to the contextual language of type I IFNs.

For example, vaccines designed to combine a TLR signal with type I IFNs would likely have a different effect than vaccines that also activate a cytosolic nucleic acid sensor.

Additionally, viruses are known to target both type I IFN receptor signaling as well as the signaling machinery that links cytosolic nucleic acid recognition to activation of type I IFNs [10][79].

Based on the pairing of these two pathways in infected cells, therapies that would restore both of these functions would be far more effective than those that restore either IFN signaling or nucleic acid detection.

In the case of hepatitis C virus, inhibition of the virus-encoded NS3/4A protease that cleaves and inactivates MAVS would reactivate both functions in infected cells [80][33].

As we continue to develop our knowledge of the different pathways that cooperate with type I IFNs to achieve diverse outcomes, we will be in a position to better understand the subtleties of this contextual language and use it for therapeutic benefit.


Conclusions and Perspectives
Several years of remarkable progress have illuminated the many different points at which type I IFNs are integrated into host defense.

With this progress, a number of basic principles that govern type I IFN function have emerged.

Two general features unite the receptors that activate production of type I IFNs.

First, these receptors coordinate the immune response against intracellular pathogens, including viruses and probably cytosolic bacteria.

Second, nucleic acid recognition appears to underlie most of an organism’s ability to activate type I IFNs.

We can say this with fair certainty for RNA virus detection based on the recent characterization of the RIG-I- and MDA5-signaling pathways.

Whether the same principle holds true for recognition of DNA viruses and intracellular bacteria remains to be seen.

If true, then the source of type I IFNs that play a pathogenic role in many autoimmune settings must be an aberrant recognition of self nucleic acids by these sensors, a cost of this imperfect recognition strategy that detects pathogen components that are also present in the host (the role of type I IFNs in autoimmunity will be discussed in an accompanying review [8]).


The sensors that activate type I IFNs probably have other important functions as well.

This is certainly true for TLRs expressed by DCs, which are integrated into a number of key processes including antigen presentation (and crosspresentation) and inducible expression of other molecules that initiate the adaptive immune response.

The cell-intrinsic cytosolic sensors might also have diverse functions beyond activation of type I interferons, including induction of apoptosis and modulation of ligands that flag infected cells for elimination by NK cells and CTLs.

An important area of future research will be the precise elucidation of these IFN-independent functions and all of the signaling pathways activated by these receptors.


In conclusion, the pace of recent advances in our understanding of how type I IFNs operate in immunity suggests that major breakthroughs in vaccine development and clinical treatment of viral infections are more attainable now than ever before.

The discovery of many of the receptors that activate type I IFNs along with the details of antiviral signaling pathways provides a fertile new ground for development of strategies to enhance or intercept these responses in the event of pathogen infection or autoimmunity, respectively.


Indirect comparison and network meta-analysis (NMA) are standard approaches for producing pooled estimates of rela- tive treatment effects for multiple treatments from multiple randomized controlled trials, where the treatments of interest may not have all been compared in the same randomized controlled trial, but instead form a connected network of treat- ment comparisons.

1-6 For example, we may have a scenario (illustrated in Figure 1) where trials of treatments A vs B and A vs C are available, but there are no trials comparing B vs C. An indirect comparison for the B vs C treatment effect can be obtained via the common comparator A. NMA is the generalization of indirect comparisons to more complex networks of treatment comparisons.

Health technology assessments, such as those conducted by the National Institute for Health and Care Excellence in the United Kingdom, require companies to submit evidence on the clinical and cost effectiveness of their treatment compared with other relevant treatments.

Indirect comparisons and NMA are routinely used as part of the health technology assessment process, since the available trial evidence may not necessarily include a single head-to-head study of all of the relevant treatment options, and is likely to instead consist of several studies each comparing a subset of treatments.


Because the statistical methods used for indirect comparisons and NMA respect the randomization in the included tri- als, results are not affected by any differences between trials in the distribution of prognostic variables-those that affect absolute outcomes regardless of treatment.

However, bias will be introduced if there are any differences between trials in the distribution of effect modifying variables-those that alter the relative effect of treatment as measured on a given scale.

If individual patient data (IPD) are available from every study then an IPD network meta-regression is the "gold-standard" approach to adjusting for differences in effect modifiers.

7-11 However, this level of data availability is rare in health tech- nology appraisal: it is much more common for a submitting company to have IPD from their own study or studies, and only published aggregate data (AgD) from their competitors'.

Several methods have been proposed in this scenario where there are studies with IPD and studies with AgD, aiming to use available IPD to produce a "population-adjusted" indirect comparison.


Matching-adjusted indirect comparison (MAIC) 12,13 and simulated treatment comparison (STC) 13,14 are popula- tion adjustment methods designed specifically for a two-study indirect comparison scenario.

In this scenario there is a single IPD study comparing treatments A and B, and a single AgD study comparing treatments A and C (Figure 1).

MAIC is a weighting approach, and STC is based on regression adjustment.

The relative effect of treat- ment B vs treatment A in the aggregate study population is predicted, based on weighting (for MAIC) or regression (for STC), and used to construct a population-adjusted indirect comparison with the aggregate relative effect of C vs A.


Multilevel network meta-regression (ML-NMR) is an extension of the NMA framework to incorporate both individual and aggregate data from a connected network formed by any number of studies and treatments.

15,16 In contrast to previous approaches based on NMA where mean covariate values are "plugged in" to an individual-level model for those studies with AgD, 17-21 ML-NMR instead integrates an individual-level model over the covariate distribution in each AgD study, thus avoiding aggregation bias when the model is nonlinear.


Population adjustment methods are increasingly being used in health technology appraisal submissions and in the wider literature, 22-24 yet to date have largely only been investigated in limited simulation studies designed to demonstrate performance under ideal scenarios where they would be expected to perform well.

ML-NMR, as the most recently pub- lished of the three methods, has yet to be examined in a simulation study.

In this article, we aim to assess the performance of ML-NMR, MAIC, and STC population adjustment methods in a range of ideal and nonideal scenarios under various failures of assumptions using a simulation study.

We base our simulations on the two-study setting presented in Figure 1.

The simulation scenarios are designed to probe the underlying assumptions and properties of the methods, 22,23 and are motivated by recent reviews of technology appraisals submitted to the National Institute of Health and Care Excellence in the United Kingdom and of applications in the literature.

22,25 We begin in Section 2 by describing the methods of standard indirect comparison, MAIC, STC, and ML-NMR.


This is followed in Section 3 by a motivating example of population adjustment applied to a network of plaque psoriasis treat- ments.

Section 4 then provides an overview of the assumptions, properties, and issues often encountered in practical applications of population adjustment methods, based on recent methodological 22,23 and applied 22,25 reviews.

Motivated by the example and these reviews, we set out the simulation study design in Section 5, and report the results in Section 6.

We conclude with a discussion in Section 7.


Indirect comparisons follow the same general form, regardless of the method used to estimate them.

In a population P, the relative effect d BC(P ) of treatment C vs treatment B is determined from the difference in the relative effects of B and C against treatment A:


Differences between methods are due to how the relative effects d AB(P ) and d AC(P ) are estimated, and in which population.

Equation (1) describes an anchored indirect comparison, since the comparison is made via the common com- parator A. We focus on anchored indirect comparisons in this article.

However, it is also possible to produce unanchored comparisons, without reliance on a common comparator, under much stronger assumptions (see Section 4.1 and the discussion).


A standard indirect comparison, without any population adjustment, is performed following the Bucher method.

1 The relative effects are assumed to be constant across populations, so that the relative effect of B vs A in the population of the AB study is the same in the population of the AC study, d AB(AB) = d AB(AC) = d AB , and similarly for the C vs. A effect d AC(AB) = d AC(AC) = d AC , and the C vs B effect d BC(AB) = d BC(AC) = d BC .

The standard indirect comparison is then d BC =d AC −d AB (2a) with standard error s BC = √ s 2 AC + s 2 AB , (2b) whered AB andd AC are the relative effects estimated from the AB and AC studies respectively with standard errors s AB and s AC .

Standard indirect comparisons will be biased if there are differences between studies in the distribution of effect-modifying covariates.


For STC, a regression model is fitted to the IPD in the AB study, with linear predictor g( ik(AB) ) = (AB) + x T ik(AB) 1 + (x T ik(AB) 2,B + B )I(k = B), (3) where g(⋅) is a suitable link function (eg, the logit function for logistic regression) transforming the expected outcome ik(AB) for individual i on treatment k in the AB study with covariate vector x ik(AB) , (AB) is the intercept, 1 are coefficients for prognostic variables, 2,B are coefficients for effect modifier interactions with treatment B, and B is the individual level effect of treatment B vs A at the reference level of the covariates.

The predicted average treatment effect of B vs A in the AC population is then given byd AB(AC) = x T (AC)̂2,B +̂B, (4) where x (AC) is the vector of mean covariate values in the AC study population.

A population-adjusted indirect comparison between treatments B and C in the AC study population is then estimated aŝ d BC(AC) =d AC(AC) −d AB(AC) (5a) with standard error s BC(AC) = √ s 2 AC(AC) + s 2 AB(AC) , (5b) whered AC(AC) is the relative effect estimate from the AC study with standard error s AC(AC) , and s AB(AC) is the standard error ofd AB(AC) from the regression.

STC produces estimates which are specific to the AC study population.

However, making the shared effect modifier assumption (which is investigated in simulation scenario c, Section 6.3) means that the estimated C vs B relative effect is constant, d BC(AB) = d BC(AC) = d BC , and thus applicable in any target population.

STC is designed with this two-study indirect comparison scenario in mind, and does not easily generalize to larger networks of studies or treatments.


MAIC estimates weights w ik for each individual in the AB study, which match the moments of the reweighted covariate distributions to those in the AC study.

12 Weights are estimated using the method of moments 12 or entropy balancing, 26-28 which have been shown to be equivalent.

29 Given the weights, the predicted mean outcomes on treatment k = A, B in the AC population are estimated by taking a weighted average of the outcomes y ik(AB) of the N k(AB) individuals on treatment k in the AB populationŷ k(AC) = ∑N k(AB) i=1 y ik(AB) w ik ∑N k(AB) i=1 w ik , (6) and the predicted average treatment effect of B vs A in the AC population is then given bŷ


The population-adjusted indirect comparison between treatments B and C in the AC study population is again given by (5).

Typically both the mean and variance of each continuous covariate are matched between the two study populations (only proportions are matched for discrete covariates), and the standard error s AB(AC) is estimated using either bootstrap- ping or robust sandwich estimators.

12,22 MAIC produces estimates which are specific to the AC study population.

Again however, making the shared effect modifier assumption means that the estimated C vs B relative effect is constant across populations.

Like STC, MAIC is designed with this two-study indirect comparison scenario in mind, and does not easily generalize to larger networks of studies or treatments.


ML-NMR extends the network meta-analysis framework to synthesize evidence from a connected network of studies where some studies provide IPD and some provide AgD. The general ML-NMR model is 16 Individual level:


Aggregate level:


The individual- and aggregate-level data are given appropriate likelihood distributions Ind (⋅) and Agg (⋅).

ijk and jk (x ijk ) are the conditional mean outcome and linear predictor for an individual i in trial j on treatment k with covariate vector x ijk .

•jk is the marginal mean outcome on treatment k in trial j. g(⋅) is a suitable link function, and denotes the support of x. The coefficients j are study-specific baselines, 1 are coefficients for prognostic variables, and 2,k are coefficients for effect modifiers specific to each treatment k. The effect of the kth treatment (at the individual level), k , is defined with respect to the reference treatment A, and we set A = 0 and 2,A = 0.

The shared effect modifier assumption is not required for ML-NMR if sufficient data are available to identify the model parameters (eg, in larger networks); however, in a two-study indirect comparison scenario we make the shared effect modifier assumption setting 2,B = 2,C to identify the model.


Numerical integration may be used to evaluate the integral (8d) for the aggregate-level model.

The quasi-Monte Carlo integration approach described by Phillippo et al 15,16 is flexible, efficient, and widely applicable regardless of the model form or complexity.

However, this approach typically requires assumptions regarding the forms of the marginal distribu- tions and the covariate correlations in the AgD study or studies, as these are rarely reported in publications (Section 4.5).

These assumptions are investigated in simulation scenarios g, h, and i (Section 6.6).

Using ML-NMR, the population-adjusted average treatment effects in each study population P = AB, AC are given by d ab(P) = x T (P) ( 2,b − 2,a ) + b − a , (9) where a, b ∈ {A, B, C} and x (P) is the vector of covariate means in population P.


Unlike MAIC or STC, ML-NMR extends naturally to larger networks: Equation (8) applies equally to networks with any number of treatments and any number of AgD or IPD studies.

Moreover, through Equation (9) we can produce population-adjusted treatment effects for any target population with given covariate values, not just the AgD AC study population.

Notably the ML-NMR model (8) reduces to standard AgD NMA when no covariates are included in the model, and reduces to IPD network meta-regression when IPD are available from every study.

16 We now demonstrate the application of MAIC, STC, and ML-NMR to a real example.


Figure 2 displays the treat- ment network formed by four phase 3 trials comparing six treatments for moderate-to-severe plaque psoriasis.

In UNCOVER-1, UNCOVER-2, and UNCOVER-3, patients were randomized to receive placebo, etanercept (in UNCOVER-2 and UNCOVER-3 only), ixekizumab every 2 weeks (Q2W), or ixekizumab every 4 weeks (Q4W).

30,31 In FIXTURE, patients were randomized to receive placebo, secukinumab 150 mg, secukinumab 300 mg, or etanercept.

32 IPD were only avail- able from UNCOVER-1, UNCOVER-2, and UNCOVER-3.


We analyze a binary outcome, success/failure to achieve 75% improvement on the Psoriasis Area and Severity Index (PASI) scale (PASI 75) at 12 weeks, on the standardized mean dif- ference (SMD) scale (ie, using a probit link function).

Five clinically relevant covariates considered to be potential effect modifiers are available for individuals in the UNCOVER trials and as summary statistics from the FIXTURE trial, with some differences between studies in the distribution of these covariates: mean body surface area covered ranges from 26.0% to 34.4%; mean duration of psoriasis ranges from 16.5 to 19.6 years; the proportion of individuals having previous PBO ETN a b c F I G U R E 2 The UNCOVER 30,31 and FIXTURE 32 trials form a network of six treatments.


Shading indicates comparisons made in: (a) UNCOVER-2 and UNCOVER-3; (b) UNCOVER-1; (c) FIXTURE.

The thick and thin lines represent availability of IPD and AgD on a comparison, respectively.

ETN, etanercept; IXE, ixekizumab; PBO, placebo; SEC, secukinumab.

IXE and SEC were each investigated with two different dosing regimens.

Reproduced from Phillippo et al 16 [Color figure can be viewed at wileyonlinelibrary.com]


systemic treatment ranges from 57.1% to 71.3%; the proportion of individuals with psoriatic arthritis ranges from 14.7% to 26.3%; and mean weight ranges from 83.3 to 92.2 kg.

The within-study variation in each covariate is much greater than the variation between studies, with standard deviations between 16.5% and 18.9% for body surface area, 11.9 to 12.5 years for duration of psoriasis, and 20.8 to 23.8 kg for weight.

Full details are given by Phillippo et al. 16 MAIC has previously been used to estimate a population-adjusted indirect comparison between ixekizumab Q2W and secukinumab 300 mg (the approved dosages), adjusting for these baseline covariates, via the common etanercept arms in UNCOVER-2 and UNCOVER-3 and FIXTURE.

16,33 We also estimate this comparison using STC.

However, for both MAIC and STC, data from UNCOVER-1 could not be included as this study did not include an etanercept arm.

Furthermore, comparisons made using MAIC and STC are valid only for the population of the study with AgD (here FIXTURE) unless the shared effect modifier assumption is made (Sections 2.2 and 2.3).


Using ML-NMR allows all of the available data to be incorporated and to produce population-adjusted indirect com- parisons between any pair of treatments in any chosen target population.

We use quasi-Monte Carlo numerical integration to implement the aggregate-level model, integrating the individual-level model (8b) over the covariate distribution in the FIXTURE trial.

16 The forms of the marginal distributions for each covariate in the FIXTURE trial are chosen based on theoretical properties and on the observed distributions in the UNCOVER trials: weight and duration are given a Gamma distribution to account for skewness, and body surface area as a percentage is given a scaled logit-Normal distribution.

Previous systemic treatment and psoriatic arthritis are binary covariates.

Correlations between covariates in the FIXTURE study are assumed to match those observed in the UNCOVER trials since these are not reported, and are accounted for in the numerical integration using a Gaussian copula.

16 We then fit a ML-NMR model to the PASI 75 outcomes, including interaction terms for the five potential effect modi- fiers.

We take a Bayesian approach implemented in Stan, 34 placing a noninformative N(0, 100 2 ) prior distribution on each parameter.

With only two AgD secukinumab treatment arms available, it is not possible to identify a model with five dis- tinct effect modifier interactions and a treatment effect for each secukinumab dose.

However, since secukinumab and ixekizumab share modes of action as interleukin-17A blockers, we assume that the effect modifier interaction parame- ters are common between these treatments across all doses (the shared effect modifier assumption) to identify the model.

Four parallel chains were run for 2000 iterations each (the first 1000 iterations were discarded as warm-up).

Convergence was assessed using the potential scale reduction factorR, 35 and effective sample sizes and Monte Carlo standard errors guided the choice of number of iterations.

There were no divergent transitions.

Full details of the analysis are given by Phillippo et al. 16 The treatment contrast estimates for MAIC, STC, and ML-NMR are summarized in Table 1.

There are small differences in the estimated average treatment effects in each population, for example, etanercept appears slightly more effective relative to placebo in the FIXTURE study population than in the UNCOVER study populations.

Using MAIC, the com- parison between ixekizumab Q2W and secukinumab 300 mg in the FIXTURE population is estimated as a SMD of 0.28 (0.00, 0.56) in favor of ixekizumab Q2W; with STC, this comparison estimate is 0.25 (−0.02, 0.52); with ML-NMR, we estimate 0.34 (0.10, 0.58).

The standard indirect comparison estimate is 0.37 (0.12, 0.63).

The point estimates are similar between the three population adjustment approaches, but ML-NMR has reduced uncertainty compared with MAIC and STC due to incorporating all available information from the studies.

Phillippo et al 16 also fitted a standard random effects NMA to this data, finding that ML-NMR also reduced uncertainty compared with the NMA.

For example, the random effects NMA estimate for the comparison between ixekizumab Q2W and secukinumab 300 mg was 0.45 (−0.02, 0.92).

Since the differences in effect modifiers between trials are small, the possible bias in a standard indirect comparison or TA B L E 1 ML-NMR estimated SMD contrasts and 95% credible intervals for each treatment against placebo, plus the contrast targeted MAIC and STC, in each study population This example raises a number of issues which we will investigate in the simulation study.

First, we made explicit assumptions regarding the joint covariate distribution in the FIXTURE study for ML-NMR, and MAIC and STC also make implicit assumptions about the joint covariate distribution (see Section 4.5).

The shared effect modifier assumption was required to identify the ML-NMR model, and is also necessary if the results of MAIC and STC are to be applicable to a target population other than the FIXTURE study.

While this assumption seems reasonable in this example since ixekizumab and secukinumab are in the same class of treatments, this may not always be the case.

For all three population adjustment methods we assume that all effect modifiers have been suitably adjusted for; however, there may be unobserved or omitted effect modifiers.

Finally, while the differences in covariate distributions between study populations were relatively minor, in many cases there may be more substantial differences (see Section 4.7), which raises questions regarding sufficient between-study overlap and validity of extrapolation, particularly with continuous covariates.

We would like to investigate the potential impact of each of these assumptions, particularly as they are often unverifiable (although some validation may be possible in larger networks using ML-NMR, see Section 7).


We now provide a more detailed overview of the assumptions, properties, and issues often encountered in practical applications of population adjustment methods, based on recent methodological 22,23 and applied 22,25 reviews.


Anchored indirect comparisons make use of a common comparator treatment to compare relative effects (Figure 1).

That is, in a population P, the relative effect d BC(P ) of treatment C vs treatment B is determined from the difference in the relative effects of B and C against treatment A following Equation (1).

Standard indirect comparisons make the con- stancy of relative effects assumption, which requires that there are no differences in the distribution of any effect modifiers between studies, so that the relative effects in (1) are constant across study populations.

Anchored population adjust- ment methods relax this assumption to conditional constancy of relative effects, that is, all of the effect modifiers with respect to the chosen comparison scale (eg, log odds ratios) that are imbalanced between studies have been suitably adjusted for.

22,23 Unanchored indirect comparisons may be performed when there is no common comparator between randomized controlled trials or if a comparison involves single arm studies, under much stronger assumptions.

22,23 We focus on an anchored scenario in the simulation study (see Section 5.2), but expect the results and conclusions to apply widely to unanchored comparisons also.

We discuss this further in Section 7.


For anchored indirect comparisons, we are concerned with identifying and adjusting for effect-modifying covariates, so that the conditional constancy of relative effects assumption holds.

Potential effect modifiers should be identified prior to analysis; 22,23 however, in practical applications there has often been no justification of effect modifier status, or any attempt to check for effect modifiers that are missing from the model (either omitted or unobserved).

22,25 In the plaque psoriasis example we adjusted for five covariates that had been selected in a previous analysis by expert opinion, 33 and assumed that relative effects were constant given these (Section 3).

We therefore assess the performance of the three population adjustment methods when all effect modifiers are accounted for and when there is a missing effect modifier, across all simulation scenarios.

We also compare performance against a standard indirect comparison that makes no adjustment for effect modifiers.


When considering noncollapsible measures of treatment effects such as odds ratios or hazard ratios, care must be taken to ensure that compatible estimates that have been adjusted in the same way are combined.

This is because, for such effect measures, adjustment for a covariate moves the treatment effect estimate away from the null and adjusted and unadjusted estimands do not typically coincide, even without interaction or effect modification.

36,37 Standard indirect comparison and MAIC typically combine unadjusted treatment effects (eg, crude log odds ratios).

However, STC as commonly implemented 22 combines an adjusted treatment effect estimate (from the STC model) with an unadjusted esti- mate from the AC study, which may lead to bias.

An adjusted estimate from the AC study may be used instead, but it is unlikely that this estimate has been adjusted in the same manner as the STC model, if an adjusted estimate is even available.

ML-NMR correctly combines adjusted and unadjusted effect measures at each level of the model, through the integration in Equation (8d).

Network meta-regression with full IPD (the "gold-standard" approach) combines adjusted treatment effects from each study, so we consider adjusted population-average treatment effects as the target estimands of interest in this simulation study (Section 5.3).

We discuss this issue and the interpretation of our results further in Section 7.


When effect modification is present, relative effect estimates are population specific.

What is required for health technol- ogy assessment are estimates in the population that represents the target population for making decisions.

22 For MAIC and STC, the resulting population-adjusted estimates are valid in the population of the AC study-which is often not repre- sentative of the decision at hand, as found by the review of technology appraisals.

25 The shared effect modifier assumption is required to generalize the resulting relative effect estimate to other target populations, and means that the effect mod- ifiers of the active treatments are the same and interact with each treatment in the same way.

22,23 In the plaque psoriasis example (Section 3), this assumption was required for the MAIC and STC estimates to be applicable to a target popula- tion other than the FIXTURE study.

This assumption may also be required to identify the ML-NMR model in smaller networks, as demonstrated in the plaque psoriasis example where effect modifier coefficients were shared between ixek- izumab and secukinumab.

While this assumption seems reasonable in this example since ixekizumab and secukinumab are in the same class of treatments, this may not always be the case.

We therefore investigate the impact of making this assumption when it does not hold.

Using ML-NMR in a larger network may allow this assumption to be assessed or relaxed; however, this is not possible using any of the approaches in the two-study scenario, where the assumption is unverifiable.


The joint distribution of covariates is not typically published, instead only marginal covariate summaries are available (eg, mean and standard deviation for continuous covariates, proportions for categorical covariates).

However, for ML-NMR the covariate joint distribution in the AgD studies is required in order to perform the integration step.

Thus when imple- menting ML-NMR, assumptions regarding the marginal distributional forms and the correlation structure are typically required to construct a joint distribution from the marginal summaries.

16 In the plaque psoriasis example, we used IPD from the UNCOVER studies to guide the choice of marginal distributional forms and to compute a correlation matrix, to construct a joint distribution for the FIXTURE study (Section 3).

This is a potential limitation of ML-NMR, and we thus wish to assess the performance of ML-NMR when incorrect assumptions regarding the covariate joint distribution are made.

MAIC and STC typically ignore the correlations between covariates 22,23 which may in turn affect the perfor- mance of these methods, for example, if correlations differ between studies, and we also assess this in the simulation study.


In practical applications of population adjustment methods, sample sizes of the included studies are variable, ranging from several thousand down to less than a hundred participants.

22,25 We therefore assess performance of the methods at a range of sample sizes in this simulation study.


An important property of population adjustment methods is that they require sufficient overlap between the population of the IPD study and the AgD study.

22,23 As a weighting method MAIC cannot extrapolate, and thus requires that the AgD study population is contained within the IPD study population in order to effect an adjustment.

However, considerable reductions in effective sample size after weighting in MAIC are often seen in practice.

In the two reviews of applications, the average reduction in effective sample size was 70% in technology appraisals 25 and 80% in the applied literature, 22 and in some cases was as high as 98%.

This indicates that in many cases there may be poor overlap between the study populations.


The issue of overlap is also related to the choice of covariates to include in the adjustment.

As the number of covariates increases the effective overlap between populations must decrease, a phenomenon that we observe later in the results of the simulation study (Section 6.1).

To date, no simulation studies have investigated the performance of population adjustment methods as the level of overlap decreases.

We therefore assess all methods at a range of levels of between-study overlap.


Regression-based approaches such as STC and ML-NMR are not restricted to scenarios with sufficient overlap, but any resulting extrapolation requires an additional assumption about the validity of predictions beyond the range of the IPD.

We therefore also assess performance when extrapolation outside of the range of the IPD study is invalid.


This simulation study plan follows the ADEMP framework, 38 which breaks down the simulation study into five key elements: aims, data-generating mechanisms, estimands, methods, and performance measures.

The following sections are devoted to describing each of these elements in turn.


The simulation study aims to assess the performance of ML-NMR, MAIC, and STC alongside standard indirect compari- son in a range of ideal and nonideal scenarios under various failures of assumptions.

The primary concerns are bias and efficiency of the estimators, along with coverage.


For this simulation study, we will consider a binary outcome, generated under a logit (log odds ratio) model.

The basic data structure involves three treatments (A, B, C) investigated in two studies, AB and AC (Figure 1).

We will simulate two continuous covariates which modify the effect of treatments B and C. IPD are available for the AB study as a binary outcome and covariate information for each individual.

Only AgD are available for the AC study, given as an overall event count and summary covariate information.

For the AC study, individual outcomes will be simulated and then aggregated.


The underlying model is of the form y ijk ∼ Bern( ijk ) logit( ijk ) = j + q(x T ijk − m T j ) k + k x ijk ∼ j , where outcomes y ijk for individual i in study j receiving treatment k are generated from a Bernoulli distribution with individual event probability ijk .

We define an outcome model on the logit scale with study intercept j , covariate vector x ijk and coefficients k , and treatment effects k .

The covariates are generated following the joint distribution j .

The function q(⋅) defines a (potentially nonlinear) relationship between the covariates and outcome.

Covariates are centered in the regression model against the mean in each study m j , so that the model coefficients are more easily interpreted as log odds and log odds ratios at the mean.


We set A = 0, B = −2, C = −1.5, AB = 1, AC = 1.5.

The values of k will be set depending on the scenarios set out below, but always A = 0 because A is the reference treatment.


In the AB trial, the two covariates will be generated with means m X 1 (AB) = 1 and m X 2 (AB) = 0.5, and standard devia- tions X 1 (AB) = 0.5 and X 2 (AB) = 0.1 (as either a Normal or Gamma covariate, see g and h below).

The distributions of the covariates in the AC trial are set to achieve the required overlap (see e below).


We shall consider varying the following parameters (where the reference levels for each parameter are bold and underlined below):


a. Overall sample size N in AB and AC trials (100, 500, 1000).

We will use 1:1 randomization within each study.


b. Strength of effect modification.

Modifying the treatment effect log odds ratio by 0.1, 0.5 per AB covariate SD, that is, k = 0.1 X(AB) or 0.5 X(AB) .

This corresponds to roughly ±10% and ±50% modification of treatment effects within ±2 SD of the mean covariate value m X(AB) in the AB study.


c. Shared effect modification.

Effect modifier coefficients are shared ( B = C ) or not between treatments B and C. d. Strength of correlation between covariates in each study ( (AB) = (AC) = 0, 0.25, 0.5).


e. Between-study overlap.

Full overlap (AC contained entirely within AB), and 50%, 100% of AC population outside of AB.

Set using a proxy parameter (see below).


f. Covariate-outcome relationship.

Linear, nonlinear beyond the range of the AB study.


g. Distribution of covariates in AB.

Consider Normal covariates and Gamma covariates.

h. Distribution of covariates in AC.


i. Correlation structures.

Same in both studies ( (AB) = (AC) ), different in each study ( (AB) ≠ (AC) ).


For the purposes of computation time and reporting, we will not consider a full-factorial design of all possible data-generating mechanisms.

Instead, each parameter will be varied independently.

However, we will consider a facto- rial examination of scenarios e and f (investigating validity of extrapolation), and of g, h, and i (investigating validity of assumptions regarding the joint covariate distribution in the AgD study).


F I G U R E 3 The function q(⋅) is approximately linear within the range of the AB study, but attenuates outside this range [Color figure can be viewed at wileyonlinelibrary.com]


For scenario e, we wish to vary the overlap between the study populations, that is, the proportion of the AgD AC population lying within the IPD AB population (see Appendix A).

We vary a proxy parameter , where approximately = 0 corresponds to no overlap, = 0.5 to 50% overlap, and = 1 to full overlap.

Using , we then define the mean and standard deviation of each covariate in the AC population as


noting that these are set to differ from the AB population even when = 1.


For scenario f, the nonlinear relationship will be specified with a sigmoid function, parametrized so that y ≈ x within ±2 SD of the mean covariate value in the AB study, but then attenuating outside this range (Figure 3):


The estimands of interest will be adjusted population-average relative effects between each treatment, as would be estimated by a "gold standard" correctly specified IPD network meta-regression, with target population:


Represented by the AC trial population, denoted d AB(AC) , d AC(AC) , d BC(AC) ;


Represented by the AB trial population, denoted d AB(AB) , d AC(AB) , d BC(AB) .


ML-NMR estimates all six of these relative effects (see Section 2.4).

MAIC and STC only estimate d AB(AC) , d BC(AC) , and d BC(AB) (see Sections 2.3 and 2.2), and so we only report results for these relative effects.

For comparison, we report the same three relative effects for standard indirect comparison.

For MAIC, STC, and standard indirect comparison, d AB(AB) and d AC(AC) are taken to be the observed relative effects in the AB and AC studies respectively and are thus unchanged between these methods.


We compare ML-NMR, MAIC, STC, and standard indirect comparison, as described above in Section 2.

For ML-NMR, MAIC, and STC we will consider models with a full set of effect modifiers or missing one effect modifier.

For standard indirect comparison, no effect modifiers are adjusted for.


For MAIC, we match both the mean and variance of the covariates between the study populations and estimate the standard error s AB(AC) using bootstrapping.

22 For STC, we fit a logistic regression model to the IPD in the AB study, Covariate distributions and correlation structures 6.6 B.6 with linear predictor given by (3) and a logit link function.

For both MAIC and STC, making the shared effect modifier assumption (which is investigated in scenario c, Section 6.3) means that the estimated B vs C relative effect is constant across populations, d BC(AB) = d BC(AC) = d BC .


For ML-NMR, we use a logit link function, a Bernoulli individual-level likelihood Ind ( ijk ) = Bern( ijk ), and a Bino- mial aggregate-level likelihood Agg ( •jk ) = Bin(N jk , •jk ).

In this two-study scenario, we require the shared effect modifier assumption to identify the model: we set 2,B = 2,C .

This assumption is investigated in scenario c (Section 6.3).

We evalu- ate the integral (8d) for the aggregate-level model using quasi-Monte Carlo numerical integration, with a Gaussian copula to account for the correlations between covariates.

In the AgD study we only have marginal covariate information (here, means and standard deviations); therefore, the joint covariate distribution f (AC) (x) in the AgD study is assumed to have the same form of marginal distributions and covariate correlations as in the IPD study, an assumption that we investigate in scenarios g, h, and i (Section 6.6).


We implement ML-NMR in a Bayesian framework using Stan. 34 We use vague N(0, 100 2 ) prior distributions for all parameters, and summarize uncertainty in the parameter estimates with standard errors and credible intervals from the posterior distribution.


To assess the performance of each method, we compute the bias, empirical standard error, model standard error, and coverage probability in each simulation scenario.

Bias is the difference between the expected value of an estimator and the truth, estimated as the average difference between the repeated estimates and the truth.

An unbiased estimator (bias of zero) is desirable.

The empirical standard error is the true variability of the estimator, estimated as the observed standard error of the repeated estimates.

The model standard error is the average standard error reported by a method over all repetitions (taken on the variance scale).

We thus desire both that the empirical standard error is small (the estimator is precise) and that the empirical standard error is well estimated by the model standard error (so that uncertainty is appropriately quantified).

The coverage probability is the probability that the confidence or credible intervals contain the true value, estimated as the proportion of repetitions with intervals that include the true value, which we wish to be at the nominal level (here 95%).


The simulation study was run in R 3.4.1, 39 using the package simsalapar 40 to facilitate parallelization, error handling, and saving of results.

Each scenario was replicated 2000 times to achieve Monte Carlo standard errors (on the log odds ratio scale) below 0.035 for bias and 0.025 for empirical standard error, for all methods (other than a small number of scenarios for which MAIC was highly unstable).


Due to the size of simulation study, additional figures (including coverage plots) and tabulated summaries of results can be found in Appendix B. For reference, Table 2 summarizes the simulation scenarios and lists the corresponding results sections and appendices.


In scenario a, the sample sizes of the two studies were varied between 100, 500, and 1000.

Figure 4A shows the bias in the population-average contrast estimates for each method, adjusting for all effect modifiers, along with 95% Monte Carlo confidence intervals.

To aid comparison between methods, the points are colored by contrast, with lighter shades for the AB population and darker for the AC population.

Figure 4B shows the corresponding empirical and model standard errors.

Coverage zip plots 38 for the d BC(AC) contrast estimate are shown in Figure B1.

Table B1 provides the results in tabular format.


We see that ML-NMR and STC are comparable in terms of both bias and standard error, largely eliminating the bias that the Bucher method (standard indirect comparison) incurs, with similar empirical standard errors that are well esti- mated by the model standard errors.

For the smallest sample size (100), the model standard errors slightly underestimate the empirical standard errors.

Some small bias remains in each of the estimates for ML-NMR and STC, and is most pro- nounced at the smallest sample size; this is likely due to the small sample bias inherent to logistic regression and the low average number of events on treatment B (13.6 from 50 individuals).

41 As expected, ML-NMR produces more precise esti- mates (ie, with lower standard error) for contrasts between treatments where there is direct evidence than those based on indirect evidence (d AB(AB) and d AC(AC) are more precisely estimated than the remaining contrasts).

ML-NMR and STC also perform similarly in terms of coverage, achieving nominal coverage for sample sizes 500 and 1000.

Both methods display slight undercoverage at the smallest sample size (100), 90.6% (89.3, 91.8) for ML-NMR, and 93.6% (92.6, 94.7) for STC, perhaps due to increased small-sample bias.

The standard indirect comparison is always below nominal coverage, and coverage drops off severely with increasing sample size as a biased estimator is more precisely estimated (note the rightward skew to the zip, which agrees with Figure 4A).


MAIC does not perform as well as ML-NMR or STC in this scenario.

The reference level of between-study overlap is set at 50%, but this means that any reweighting scheme such as MAIC cannot hope to eliminate the bias since extrapolation is not possible.

As a result, MAIC provides a lesser bias reduction than ML-NMR and STC for sample sizes 500 and 100, and for the smallest sample size actually substantially increases the bias compared with a standard indirect comparison.

Empirical standard errors for MAIC are larger than for ML-NMR and STC, and the MAIC model standard error is only a good estimate at the largest sample size.

For sample size 500, the MAIC model standard errors (derived through boot- strapping) are extremely unstable, as the weights are highly dependent on a small number of individuals, and were only successfully obtained 47% of the time.

For sample size 100, estimation failed entirely 23% of the time, and no attempts to obtain model standard errors were successful.

Coverage for sample size 500 is at the nominal value, despite remaining bias and unstable standard errors.

For sample size 1000, coverage of 90.7% (89.4, 91.9) is below the nominal value, as the standard errors have stabilized but the estimator remains biased.


When one of the two effect modifiers is not adjusted for, none of the methods are able to eliminate bias from the estimates (Figure 5A, also Table B2).

ML-NMR and STC show slightly reduced standard errors compared with the models with the full set of effect modifiers (Figure 5B).

Again, the empirical standard errors are well estimated by the model standard errors for the larger sample sizes, with a slight underestimation for the smallest sample size.

MAIC has markedly reduced standard errors compared with adjustment for both effect modifiers, and the bootstrap model standard errors are now stable for the larger sample sizes.

This is because the overlap between studies is necessarily lower as the number of covariates increases.


However, MAIC is still unstable for the smallest sample size.

Coverage of all three population adjustment methods begins to drop from the nominal level as sample size increases, down to around 92% for sample size 1000 (Figure B2).

This is due to the bias remaining in the estimates, as further evidenced by the rightward skew to the zip plots.

However, coverage is still well above that of the standard indirect comparison, as bias is reduced.


Similar results to Figure 5 are seen across the remaining scenarios when an effect modifier is missing; here onward, we discuss any notable differences in the main text and refer to Appendix B for figures and tabulated results.


In scenario b, the strength of effect modification was varied from weak (0.1 change in log odds ratio per covariate standard deviation in the AB study) to strong (0.5 change in log odds ratio per covariate standard deviation in the AB study), and was the same for both treatments B and C (so the shared effect modifier assumption holds).

Figure 6A shows the bias in the population-average contrast estimates for each method, adjusting for all effect modifiers, along with 95% Monte Carlo confidence intervals.

Figure 6B shows the corresponding empirical and model standard errors.

( See also Table B3.) Both F I G U R E 4 Bias, A, and standard errors, B, for the population-average contrast estimates for scenario a, along with 95% Monte Carlo confidence intervals.

Each method (other than Bucher) adjusts for the full set of effect modifiers.

Sample size is varied between 100, 500, and 1000.

The points are colored by contrast, with lighter shades for the AB population and darker for the AC population [Color figure can be viewed at wileyonlinelibrary.com]


F I G U R E 5 Bias, A, and standard errors, B, for the population-average contrast estimates for scenario a, along with 95% Monte Carlo confidence intervals.

One of the two effect modifiers was not adjusted for.


ML-NMR and STC have successfully removed the bias from the standard indirect comparison, and display very similar standard errors.

The standard errors are largely unaffected by the strength of the effect modification, although the model standard errors for ML-NMR and STC slightly underestimate the empirical standard errors when the effect modification is strong.

MAIC has also removed most of the bias, though some remains due to lack of overlap between the two studies, and appears to do better when the effect modification is stronger.

Empirical standard errors for MAIC are higher than for ML-NMR and STC, and again the bootstrap model standard errors are extremely unstable.

Coverage for all three population adjustment methods is at the nominal level (Figure B3), although there is a small drop for ML-NMR and STC to 93.0% (91.9, 94.2) and 93.9% (92.9, 94.9) when the effect modification is stronger due to the slight underestimation of the standard error.

Coverage for the standard indirect comparison drops to zero when the effect modification is strong, as the incurred bias is so large.


When one of the two effect modifiers is not adjusted for, all population adjustment methods fail to remove the bias (Figure B4a, also Table B4).

As expected, the amount of bias remaining is larger when the missing effect modifier is stronger.

The standard errors of all population adjustment methods are reduced compared with adjustment for all effect modifiers, and again do not differ by the strength of effect modification (Figure B4b).

The coverage for all population adjustment methods drops further when the missing effect modifier is stronger (Figure B5), due to the increased residual bias.


In scenario c the shared effect modifier assumption is broken, so that treatment B is subject to weak effect modification, while treatment C is subject to strong effect modification and vice versa.

We expect to see that all population adjustment methods are capable of producing unbiased estimates in the AC population, but that extrapolation into other populations is biased, and indeed this is what occurs (Figure 7A).

Standard errors (Figure 7B) and coverage (Figure B6) are largely unchanged from scenario b (see also Table B5).

Again, ML-NMR and STC show slight underestimation of the empirical standard error when there is strong effect modification, leading to a small drop in coverage down to around 93%.


The increase in bias for a standard indirect comparison when treatment B is strongly modified compared with weakly modified (Figure 7A) is due to the focus on the AC population.

Since a standard indirect comparison does not adjust for population differences, the estimates of d AB , d AC , and d BC are the same regardless of population.

However, when the target of inference is a comparison in the AC population, the estimate of d AC(AC) is unbiased and bias in the estimate of d AB(AC) (and thus d BC(AC) ) is driven by the strength of effect modification of treatment B. If the target of inference was instead a comparison in the AB population, this pattern would be reversed (bias now being driven by the strength of effect modification of treatment C).

A similar pattern, driven by the same mechanism, is observed in scenario b (see Figure 4A).


When one of the two effect modifiers is not adjusted for, further bias is introduced into the estimates, in all target populations (Figure B7a, see also Table B6).

This bias is generally larger for estimates involving extrapolation of treatment effects affected by strong unobserved effect modifiers, for example, d AC(AB) and d BC(AB) when C is strongly modified by a missing effect modifier.

Again, standard error is reduced when fewer effect modifiers are adjusted for (Figure B7b).

Coverage for d BC(AC) is close to nominal level for all population adjustment methods when the missing effect modifier for B is weak as the incurred bias is small, but drops severely when the missing effect modifier for B is strong as the incurred bias is large (Figure B8).


In scenario d, the correlation between covariates is varied between 0, 0.25, and 0.5, and the correlation is the same in both study populations.

Figure 8A shows that ML-NMR and STC are unaffected by the correlation between covari- ates, achieving bias removal regardless of the correlation.

Standard errors are similarly unaffected (Figure 8B), and coverage is at the nominal level (Figure B10).

Table B7 shows the results in tabular format.