For MAIC, both bias and standard error are reduced as the correlation between covariates increases.

This is because, as the correlation increases, the effective number of covariates decreases and the overlap between study populations increases (Figure B9).

The sim- ulation parameter is only a proxy for overlap and does not account for correlation between covariates, so the true F I G U R E 6 Bias, A, and standard errors, B, for the population-average contrast estimates for scenario b, along with 95% Monte Carlo confidence intervals.

Strength of effect modification is varied from weak (0.1 change in log odds ratio per covariate standard deviation in the AB study) to strong (0.5 change in log odds ratio per covariate standard deviation in the AB study).

Each method (other than Bucher) adjusts for the full set of effect modifiers.

The points are colored by contrast, with lighter shades for the AB population and darker for the AC population [Color figure can be viewed at wileyonlinelibrary.com]


F I G U R E 7 Bias, A, and standard error, B, for the population-average contrast estimates for scenario c, along with 95% Monte Carlo confidence intervals.

The shared effect modifier assumption is broken, so that treatment B is subject to weak effect modification, while treatment C is subject to strong effect modification and vice versa.

The points are colored by contrast, with lighter shades for the AB population and darker for the AC population [Color figure can be viewed at wileyonlinelibrary.com] overlap changes with the correlation despite holding constant at 0.5.

Since MAIC cannot eliminate the bias, cover- age drops as the standard error decreases (as the correlation increases), down to 93.4% (92.3, 94.6) when the correlation is 0.5.


When one of the two effect modifiers is not adjusted for, all population adjustment methods produce biased esti- mates (Figure B11a, see also Table B8).

However, as we expect, the amount of bias reduces as the correlation between the observed and missing effect modifiers increases.

If we were to continue simulations with correlations tending closer to 1, the bias due to a missing effect modifier would disappear entirely.

Again, standard error is reduced when fewer effect modifiers are adjusted for (Figure B11b).

Coverage for all population adjustment methods was slightly below the nominal level (around 93%) due to the remaining bias (Figure B12).


In scenarios e and f, the between-study overlap and covariate-outcome relationship are varied jointly.

The between-study overlap is varied between 0 (no overlap, all of AC population outside of the AB population), 0.5 (approximately 50% of AC outside of AB), and 1 (full overlap, all of AC within AB), and the covariate-outcome relationship is either linear or nonlinear (Equation (10)).


When the covariate-outcome relationship is linear, both ML-NMR and STC produce unbiased estimates (Figure 9A, Table B9).

MAIC is unable to produce any estimates when there is no overlap between study populations, and remains biased when the overlap is 0.5.

Only when the study populations overlap completely does MAIC produce unbiased estimates.


When the covariate-outcome relationship is nonlinear, ML-NMR and STC (set to fit a linear relationship) are both biased when there is no overlap between the study populations.

There is no discernible bias when the overlap is 0.5 or 1, as Equation (10) behaves linearly within the range of the AB population.

However, prediction into another target population with a greater difference in covariate values from the AB and AC populations would still result in bias.

Again, MAIC cannot produce estimates when there is no overlap between study populations.


Standard errors for all population adjustment methods decrease as the level of overlap increases (Figure 9B).

Empirical standard errors are well estimated by model standard errors for ML-NMR and STC; however, MAIC only produced stable bootstrap model standard errors when the overlap was 1.


For the population adjustment methods, nominal coverage is achieved when the covariate-outcome relationship is linear (Figure B13).

With a nonlinear relationship, coverage drops slightly to 93.0% (91.9, 94.2) for ML-NMR and to 93.4% (92.4, 94.5) when there is no overlap, but is unaffected at 0.5 and full overlap as there is no discernible bias.

MAIC achieves nominal coverage for 0.5 and full overlap.


When one of the two effect modifiers is not adjusted for, all population adjustment methods produce biased estimates (Figure B14a, Table B10).

As expected, the bias due to the missing effect modifier reduces as the overlap between studies increases, since the difference in the missing effect modifier between the study populations becomes smaller.

Due to this, coverage shows the opposite relationship with overlap, dropping as the overlap decreases (Figure B15).

Again, standard error is reduced when fewer effect modifiers are adjusted for (Figure B14b).


In scenarios g, h, and i, the covariate distributions and correlation structures in each study population are varied jointly.

The covariate distributions are set to either Normal or Gamma in each of the study populations, and the correlation between covariates in the AC population was varied between 0, 0.25, and 0.5.

The correlation between covariates in the AB population was held constant at 0.25.


We are particularly interested in the performance of ML-NMR in this scenario, since ML-NMR makes the assumption that distributional form and correlation structure are the same in the AgD studies as in the IPD.

Scenarios g, h, and i break this assumption.

Despite this, ML-NMR is not seen to incur any bias when the distributional form and/or correlation structure are different in each population (Figure B16).

The empirical standard errors, and their estimation by the model standard errors, are also unaffected (Figure B17).

The bias and standard error of MAIC are slightly smaller when the covariates in the AB population are Gamma distributed compared to Normally distributed; however, this is due to the skew of the Gamma distribution slightly increasing the true overlap between the study populations (compare row 1 with F I G U R E 8 Bias, A, and standard errors, B, for the population-average contrast estimates for scenario d, along with 95% Monte Carlo confidence intervals.

The correlation between covariates is varied between 0, 0.25, and 0.5.

Each method (other than Bucher) adjusts for the full set of effect modifiers.

The points are colored by contrast, with lighter shades for the AB population and darker for the AC population [Color figure can be viewed at wileyonlinelibrary.com]


F I G U R E 9 Bias, A, and standard errors, B, the population-average contrast estimates for scenarios e and f, along with 95% Monte Carlo confidence intervals.

The between-study overlap and covariate-outcome relationship are varied jointly.

The points are colored by contrast, with lighter shades for the AB population and darker for the AC population [Color figure can be viewed at wileyonlinelibrary.com] 3, and 2 with 4, in Figure B18).

Nominal coverage is achieved by all population adjustment methods, regardless of the covariate distributions or correlation structure (Figure B19).

See Table B11 for a table of these results.


Once again, when one of the two effect modifiers is not adjusted for, all population adjustment methods produce biased estimates (Figure B20) but standard error is reduced (Figure B21).

Coverage is slightly reduced for all population adjustment methods, to around 94%, as a result of the bias (Figure B19).

The results are tabulated in Table B12.


In this simulation study, we have investigated the performance of ML-NMR in comparison with current population adjustment methods (MAIC and STC) in a wide range of scenarios, including varying sample sizes, strength of effect modification, overlap between studies, and joint covariate distributions.

Scenarios were chosen to be realistic, reflecting the issues raised by the applied example and the findings of two recent reviews of applications.

22,25 As well as validating the performance of the different methods when assumptions were met, we were particularly interested in how the meth- ods fared when assumptions were broken in realistic ways.

We investigated conditional constancy of relative effects (ie, no missing effect modifiers), the shared effect modifier assumption, validity of extrapolation beyond the IPD study popu- lation, and (for ML-NMR) correctly specifying the form of the marginal distributions and correlations between covariates in the AgD study.


ML-NMR and STC performed very similarly throughout the simulation study.

This is to be expected, as the two meth- ods are both regression adjustment methods.

Both methods were seen to perform well when the requisite assumptions were met, largely eliminating the bias incurred by a standard indirect comparison and estimating standard errors well.

ML-NMR offers additional advantages over STC, including the ability to synthesize a larger network of treatments, and to produce estimates in any target population.

ML-NMR makes explicit assumptions about the covariate distributions and correlation structure in the aggregate population in order to derive the aggregate likelihood through integration, typically that they are the same as in the IPD population.

However, the performance of ML-NMR was not found to be sensitive to this assumption (see scenarios g, h, and i, Section 6.6).

MAIC and STC typically ignore covariate correlations (Section 4.5), but the performance of these methods was also not seen to be sensitive to the implicit assumptions that this invokes.


Serious questions must be raised about the use of MAIC for population adjustment.

MAIC performed poorly in all simulation scenarios, and in some cases even increased the bias compared with a standard indirect comparison.

Boot- strap error estimation was also extremely unstable, except for the largest sample sizes.

The issues with MAIC stem from the fact that it is a reweighting method, and therefore cannot extrapolate.

As such, bias can only be completely removed when the population of the AgD study is entirely contained within the population of the IPD study (see scenario e, Section 6.5).

However, when the two study populations overlap entirely there is unlikely to be much bias in a standard indirect comparison as the imbalance in effect modifiers is necessarily small, and thus population adjustment methods may not be needed.

Furthermore, this means that MAIC is only valid from at most one company's perspective of the indi- rect comparison-that of the company whose study (for which they have IPD) has a broader distribution of covariates.

MAIC analysis from the other company's perspective (who has IPD on the other study, with a more restricted covariate distribution) will result in estimates that are biased, possibly by more than a standard indirect comparison (as well as being produced for a different target population).

If the two study populations do not overlap at all, then no MAIC analyses are possible.

The simulation study therefore raises serious questions regarding whether MAIC is fit for purpose.

Previous sim- ulation studies investigating MAIC have not observed this issue regarding overlap.

26,27,42-45 For some studies, 26,27 this is because they were not designed to vary the overlap of continuous covariates between studies, instead basing simulations on scenarios with good overlap where, as we found, MAIC works well. For others, 44,45 this is because they were focused on binary covariates, where issues only arise when covariate proportions are close to zero or one in the IPD study.

( The remaining studies 42,43 were available only as conference abstracts, and full details of the simulation scenarios were not available.) It is particularly worrying that, in practical applications of MAIC, study sample sizes are typically small and there is often a lack of overlap between studies (as evidenced by large reductions in effective sample size) 25 -situations where we have shown MAIC to perform poorly.

Given the severe issues with MAIC surrounding covariate overlap, we find it unlikely that there are realistic scenarios where MAIC may consistently outperform STC and ML-NMR.


In this simulation study, we focused on continuous covariates only.

However, the underlying assumptions made by each of the methods are unchanged whether covariates are discrete or continuous, and so we expect the conclusions of this simulation study to apply regardless of the type of covariates.

Continuous covariates were chosen for this simulation study since issues with between-study overlap can arise in much less extreme situations with continuous covariates than with discrete covariates.


In the binary covariate case, issues only appear as the covariate proportions in the IPD study approach 0 or 1, at which point no methods will work (there is no information to estimate the interaction term for ML-NMR or STC, and no individuals to reweight for MAIC).

When covariate proportions are not close to 0 or 1 there can be no overlap issues-estimates may be imprecise if groups are small, but they should not be biased.

We would expect the performance of all methods with binary covariates to be similar to that seen in the full overlap scenario in this simulation study, except when proportions are close to 0 or 1 (when no methods will work).

This also aligns with the results of previous simulation studies based on binary covariates, 44,45 which have observed no bias with nonextreme covariate proportions.


The continuous covariate scenario is of greater practical concern, as there is greater potential for methods to break down.

Practical applications often involve a mix of discrete and continuous covariates, as demonstrated in the plaque psoriasis example (Section 3).

As we have seen with continuous covariates bias can occur for MAIC even when the overlap is moderate, and ML-NMR and STC must be concerned with the validity of extrapolation (Section 6.5, scenarios e and f).


We have not considered including correlations between covariates in the MAIC matching in this simulation study, as these are typically unavailable and we are unaware of any practical applications where this has been done (see Section 4.5).

Nevertheless, it is possible to also match covariate correlations to the aggregate study in MAIC if these are known or perhaps imputed from the IPD study.

However, we did not find any notable impact of correlations on the performance of MAIC (or indeed ML-NMR or STC) in the scenarios considered, except through the impact on effective overlap (see scenarios d, g, h, and i, Sections 6.4 and 6.6).


For this simulation study the estimands of interest are adjusted population-average treatment effects in a given study population, as would be estimated by a "gold-standard" IPD network meta-regression with a correctly specified regression model.

Due to the noncollapsibility of the log odds ratio, these estimands differ from the unadjusted population-average treatment effects (ie, the unadjusted crude log odds ratios in each study population).

MAIC, as it is typically implemented (see Section 2.3), targets this latter estimand since unadjusted log odds ratios (after weighting) are combined.

This mis- match of estimands has the potential to introduce additional bias into our evaluation of MAIC; however, this does not seem to be the case in the scenarios considered here, as MAIC is seen to estimate an unbiased treatment effect when there is full overlap between study populations (Section 6.5).


The typical implementation of STC which we have used 22 is flawed for noncollapsible effect measures since it combines an adjusted effect estimate (d AB(AC) from the STC model) with an unadjusted effect estimate (d AC(AC) from the AC trial).

We assessed this version of STC, since it seems to be the most prevalent in the literature.

Moreover, a corrected version of STC combining two adjusted estimates requires that the aggregate AC study reports a d AC(AC) treatment effect estimate that has been adjusted in the same manner as the STC regression model, which seems unlikely to be available in practice.

Our simulations did not show any noticeable bias for STC when the requisite assumptions were met, despite the flawed combination of estimands; however, other simulation studies have demonstrated bias for STC due to this issue.

45 Both MAIC and STC (and indeed standard indirect com- parisons and NMA) may incur bias if incompatible estimates are combined.

ML-NMR correctly combines adjusted and unadjusted effect measures at each level of the model through the integration in Equation (8d).

Furthermore, ML-NMR can produce estimates of any population-average quantity of interest, including unadjusted treatment effects, through integration over the corresponding covariate distribution.


Although MAIC and STC are designed with a two-study indirect comparison scenario in mind, it is commonplace for larger networks of treatments and studies to be available in practice (as demonstrated in the plaque psoriasis example, Section 3).

In the review of applied literature, 73% of applications involved multiple IPD or AgD studies and/or addi- tional treatments, 22 and 56% of technology appraisals had larger networks present.

25 ML-NMR extends naturally to larger networks of studies and treatments, as we describe in Section 2.4.

While this simulation study focused on the two-study scenario for comparison with MAIC, STC, and standard indirect comparison, we expect the conclusions regard- ing ML-NMR to extend to larger networks since the underlying assumptions remain the same.

Further research could explore the properties of ML-NMR in larger networks in more detail.


Although ML-NMR offers the possibility to analyze larger networks of studies and treatments, with larger networks of evidence there is increased potential for heterogeneity and inconsistency-and also the possibility to check for such issues.

When ML-NMR is used in larger networks, Phillippo et al 16 suggest performing standard checks for heterogeneity and inconsistency.

46 The presence of residual heterogeneity or inconsistency may indicate a failure of assumptions, such as unobserved effect modifiers or an invalid shared effect modifier assumption.


The ML-NMR model described in Section 2.4 assumes that aggregate studies provide outcome data on each arm (eg, event counts or mean outcomes).

However, this can be extended to incorporate contrast-based aggregate data (eg, log odds ratios or mean differences) by using a Normal likelihood for studies providing evidence as contrasts (multivariate Normal for multi-arm studies), with linear predictor given by Equation (9).

15 This is analogous to the usual approach to handling contrast-based data in NMA.


MAIC and STC require the shared effect modifier assumption in order to produce estimates outside of the AC study population.

We also made this assumption for ML-NMR to identify the model in the two-study scenario.

This assumption may be relaxed for ML-NMR in larger networks if sufficient data are available, but this is data-intensive.


If additional information is available from aggregate studies such as reported regression coefficients or subgroup analyses, this can be used to aid estimation of a ML-NMR model.

For example, if regression coefficients and their covariance matrix are reported then these may be used as an informative prior distribution, and factorial subgroup analyses may be incorporated through a simple extension of the numerical integration approach.

15 However, further research is necessary to consider scenarios where regression coefficients are only univariate or reported without correlations, or where multiple univariate subgroup analyses are reported.


Moreover, reporting bias may be introduced if "significant" regression coefficients or subgroup analyses are selectively reported, and care should be taken to only consider evidence from prespecified analyses.


Practical applications of population adjustment include a large majority of "unanchored" comparisons, involving single arm studies or where there is no common comparator.

22,25 In this case, the expected outcomes Y B(P ) and Y C(P ) on each treatment are compared directly


Standard model fit and model comparison techniques may be employed for the ML-NMR model described in Section 2.4, such as the Deviance Information Criterion.

53 With more general individual-level likelihoods where the aggregate likelihood has no closed form (eg, for time-to-event data), alternative model comparison techniques are required, 15 and this is an area for further research.

However, we recommend that (at least for technology assess- ment purposes) applicable guidance on covariate selection is followed; that is, effect-modifying covariates should be prespecified.

22 As regression methods, ML-NMR and STC are able to extrapolate beyond the range of the IPD, producing estimates even when there is no overlap between study populations.

However, when extrapolation occurs, estimates will only be unbiased if such extrapolation is valid.

For example, in scenarios e and f (Section 6.5), ML-NMR and STC produced biased estimates when there was no overlap between populations and the true covariate-outcome relationship was nonlinear outside of the range of the IPD, but only a linear relationship was accounted for (Figure 3).


Notably, all population adjustment methods are susceptible to bias (and as a result, undercoverage) when an effect modifier is missing from the adjustment, and thus the conditional constancy of relative effects assumption is invalid.

This highlights the necessity of careful and considered selection of potential effect modifiers prior to analysis.

22,23 When all effect modifiers have been identified and included in the adjustment model, ML-NMR and STC are both robust techniques for obtaining population-adjusted indirect comparisons.

ML-NMR offers additional advantages over MAIC and STC, including extending naturally to incorporate larger networks of evidence and producing estimates in any target population of interest, 16 making this an attractive choice for population adjustment in a wide variety of scenarios.


In this commentary, we highlight the importance of: (1) carefully considering and clarifying whether a marginal or conditional treatment effect is of interest in a population-adjusted indirect treatment comparison; and (2) developing distinct methodologies for estimating the different measures of effect.

The appropriateness of each methodology depends on the preferred target of inference.

The extensive simulation study presented by Phillippo, Dias, Ades and Welton (PDAW) [1] is a timely and important addition to the evaluation of population adjustment methods for indirect treatment comparisons, a very relevant and current topic.

The authors investigate the statistical performance of matching-adjusted indirect comparison (MAIC), simulated treatment comparison (STC), and a novel method that they have recently proposed [2][3] named multilevel network meta-regression (ML-NMR), in an “anchored” scenario with a common comparator.

The use of these methods in health technology assessment (HTA), both in published literature and in submissions for reimbursement, and their acceptability by national HTA bodies, has increased across diverse therapeutic areas [4].

Their article is an important piece of scholarship because the performance of MAIC, STC and ML-NMR is assessed in a very wide range of settings, which reflect typical scenarios in health technology appraisals and include realistic “non-ideal” scenarios under several failures of assumptions.

The results of the simulation study are interesting, complement currently available guidance [5][6], and inform the circumstances under which population-adjusted indirect comparisons should be applied.

An important limitation of MAIC and STC is that they are only applicable to pairwise indirect comparisons, and not easily generalizable to larger network structures.

This is due to the methods being developed in the context of a two-study scenario, often seen in HTA submissions, where there is one AB study with individual patient data (IPD) and another AC study with only published aggregate data (AgD).

In this very sparse network, indirect comparisons are vulnerable to bias induced by effect-modifier imbalances.

In larger network structures, multiple pairwise MAICs or STCs do not necessarily generate a consistent set of relative effect estimates for all treatments because the indirect comparisons are made in the AgD study populations, due to the nature of these population adjustment methods.

ML-NMR is a timely addition because it extends population adjustment methods to synthesize scenarios where larger networks are required, with the two-study scenario as a special case.

The conclusions of the simulation study are that STC and ML-NMR are successful and robust in eliminating bias in the two-study scenario, provided the methods’ assumptions hold.

On the other hand, MAIC appears to perform poorly in nearly all simulation scenarios.

In general, MAIC provides much smaller bias reduction than ML-NMR and STC, and remains biased in relatively favorable settings where we would expect MAIC to perform well.

Despite the merits of PDAW, we believe that further clarification of the treatment effect targeted by the simulation study, and of the estimands targeted by the different methods, is required.

This is related to the conflation of marginal and conditional (log) odds ratios and the non-collapsibility of this measure of effect.

Section 5.3 of PDAW states that the estimands of interest are “adjusted population-average relative effects between each treatment”.

There are two estimands of interest in the comparison of MAIC, STC and ML-NMR.

Using the notation introduced by PDAW, these are: dAB(AC), the population-average treatment effect of B vs. A in the AC population, and dBC(AC) the population-average treatment effect of C vs. B in the AC population, implicitly assumed to be the target population if the shared effect modifier assumption is not made.

For ease of exposition, we focus on the former.

By population-average treatment effects, we believe that the authors refer to marginal treatment effects.

That is, dAB(AC) represents the average difference in the outcome between two identical AC populations, except that in one population all subjects have received treatment B, while in the other population all subjects have received treatment A, in this case taking the difference on the log odds ratio scale.

In health technology assessment, a marginal treatment effect is typically of interest when decision-making is made at the population level.

Because the goal is to evaluate the impact of a health technology on the target population for the decision problem, one is interested in the average effect, at the population level, of moving everyone in the target population from treatment A to B. Despite this, the target estimand specified by the data-generating mechanism in Section 5.2 of PDAW is, in fact, calibrated at a different hierarchical level.

This estimand is a conditional treatment effect, which has a different interpretation than the marginal or population-average treatment effect: a subject-specific interpretation at the individual level.

This is because the treatment effects (γk in Section 5.2 of PDAW) are coefficients of the logistic regression, conditional on the effects of the other covariates that have also been included in the model.

That is, dAB(AC) would represent the average effect, at the unit level, of changing an individual’s treatment from B to A, i.e., the average treatment effect conditioned on the average combination of covariates in AC.

The (log) odds ratio is an effect measure that is not collapsible over covariates.

Therefore, the marginal and conditional effects may not coincide, even if there is covariate balance (i.e., independence between the covariates and treatment) and the absence of confounding.

This is because the covariate effects do not cancel out in the computation of the conditional log odds ratio, due to the non-linearity of the logit link function of the outcome model.

For other link functions, e.g. the identity or log links, the conditional treatment effect does not vary across covariate strata (because the covariate effects cancel out in the computation), and marginal and conditional treatment effects are equal in the absence of confounding.

Therefore, the article by PDAW assesses the performance of population adjustment methods for anchored indirect comparisons, when the target estimand, dAB(AC), is a conditional measure of effect.

The article does not evaluate the performance of population adjustment methods, where the target estimand is a marginal or population-average treatment effect.

As seen in the next section, the population adjustment methods considered in the simulation study target different estimand types.

This is an important issue that requires further discussion because the appropriateness of each method depends on the target estimand.

MAIC is a method based on propensity score weighting, where the estimate dAB(AC) targets a marginal log odds ratio, rather than a conditional log odds ratio.

This is because the outcome model is a univariable weighted regression of outcome on treatment assignment, such that dAB(AC) is the fitted treatment coefficient of this weighted regression.

This coefficient estimates a relative effect between subjects that have the same distribution of covariates (corresponding to the AC population), assuming a reasonably large sample size and proper randomization in the AB trial, such that (chance) imbalances in baseline characteristics across treatment groups are insignificant.

Because the simulation study by PDAW targets a conditional treatment effect, the questions raised about the use of MAIC only apply in this context, where the method is biased due to a mismatch in estimands.

It is unsurprising that MAIC produces biased results, because it targets a marginal or population-average treatment effect, where the target estimand of the simulation study is a conditional or unit-specific treatment effect.

These do not coincide because the measure of effect, the log odds ratio, is non-collapsible.

If the target of the simulation study is a marginal treatment effect, then we would expect MAIC to be unbiased under no failures of assumptions.

In this case, MAIC should also produce unbiased estimates for dBC(AC) because it typically combines marginal treatment effects.

Evaluating the performance of MAIC with respect to the other methods, where the target estimand is a marginal or population-average effect, would be informative and impactful.

True values of the marginal or population-average effects can be calculated by simulating a very large AgD population (sufficiently large to minimize sampling variability) and fitting a simple univariable regression of outcome on treatment.

The treatment coefficient of the regression represents the expected difference in the counterfactual outcomes on the log odds ratio scale if all members of the target population are under active treatment versus if all members of the population are under the common comparator.

Under correct model specification, MAIC has produced unbiased estimates in recent simulation studies [7][8].

The authors argue that this is because the studies “were focused on binary covariates, where issues only arise when covariate proportions are close to zero or one in the IPD study” and “were not designed to vary the overlap of continuous covariates between studies, instead basing simulations on scenarios with good overlap”.

In our opinion, the difference in the performance of MAIC across simulation studies cannot be attributed solely to differences in overlap.

As explained earlier, PDAW consider a sufficiently broad set of scenarios, and MAIC is biased even in relatively favorable conditions with reasonable numbers of “overlapping” subjects (effective sample sizes).

Very recently, after submission and revision of the article by PDAW, we have updated one of the highlighted simulation studies [7] to include smaller sample sizes and to vary the overlap of continuous covariates.

The levels of overlap now correspond to percentage reductions in effective sample size that are representative of, and have been inspired by, applications of population adjustment in health technology appraisals.

Even though overlap issues can arise more easily with continuous than with binary covariates, MAIC remains unbiased under correct model specification.

The study features survival outcomes and a Cox proportional hazards model, which are likely less susceptible to small sample bias than the binary outcomes and logistic regression considered by PDAW.

However, we believe that bias is not observed because the target estimand in the study, the indirect effect dBC(AC) (using the notation of PDAW), is a marginal treatment effect.

While MAIC uses weighting for population adjustment, STC and ML-NMR are methods based on regression adjustment, which typically targets conditional estimation.

The estimate dAB(AC) targets a conditional treatment effect, because it is derived from a multivariable regression model containing as predictors both the treatment plus all effect modifiers and, potentially, prognostic variables considered for adjustment.

The treatment coefficient of the outcome model is adjusted, controlling for or holding constant all other covariate effects.

Therefore, such relative effect has a conditional interpretation.

Indeed, ML-NMR is a generalization of the “gold standard” correctly specified IPD network meta-regression, which also targets a conditional treatment effect.

As a result, it is unsurprising that STC and ML-NMR are adequate estimators of the conditional log odds ratio and produce similar unbiased results where there are no failures in assumptions.

Conversely, if the target estimand dAB(AC) was a marginal log odds ratio, we would expect STC and ML-NMR to be biased because these target conditional log odds ratios.

Unlike ML-NMR, STC has the additional issue of combining incompatible estimates.

As commonly implemented, STC combines a conditional treatment effect estimate dAB(AC) with a marginal treatment effect estimate dAC(AC) in Equation 5a of PDAW, dBC(AC) = dAC(AC) − dAB(AC).

This is because marginal treatment effects are typically reported in the clinical trial setting, e.g. derived from the coefficient of a simple regression of the clinical outcome of interest on treatment assignment.

Clinical trial publications are increasingly reporting conditional or adjusted treatment effects.

However, the reported conditional estimate for the AC study is unlikely to be compatible with the conditional effect estimate dAB(AC) because these will have been adjusted across different covariate sets and/or using different model specifications.

Because odds ratios are non-collapsible, conditional odds ratios differ across different sets of adjustment factors and model specifications, even if the covariates and treatment assignment are independent.

As mentioned by PDAW, adjusting for a greater number of covariates drives the estimated conditional treatment effect further away from the null, as these account for the some unexplained outcome heterogeneity if they are prognostic of outcome.

A conditional estimate from the AC study is only compatible if it is derived fitting the non-centered version of the model in Equation 3 of PDAW, g(θik(AC)) = μ(AC) + x⊤ik(AC)β1 + (x⊤ik(AC)β2,K + γC)1(k = C), including the same explanatory variables in the same manner, to the AC subject-level data.

Such data are unavailable and it is unlikely that the estimated treatment coefficient from this model is available in the clinical trial publication.

ML-NMR is not susceptible to the compatibility issues of STC, with compatible conditional effect measures being combined at each level of the model through the integration in Equation 8d of PDAW.

Therefore, ML-NMR is the gold standard for estimating conditional treatment effects, as STC may produce bias if the target estimand is dBC(AC) and incompatible estimates are compared in Equation 5a of PDAW, as observed in a recent simulation study [7].

In that study, the true marginal treatment effect has been set to zero by setting the true unit-level conditional treatment effects for all subjects to zero, implying a “null”-like simulation setup.

As both the true marginal and conditional effects coincide by design, ML-NMR would likely produce unbiased estimates of dBC(AC) in that setup.

We believe that this commentary is important because it highlights the importance of: (1) carefully considering and clarifying whether a marginal or conditional treatment effect is of interest in an indirect treatment comparison; and (2) developing distinct methodologies for estimating the different measures of effect.

Then, the analyst can decide between each class of methodologies according to the preferred target of inference.

Any method is susceptible to bias if targeting the wrong measure of effect.

Methods like MAIC are valid for population-based inference, but not “fit for purpose” when inference is at the individual level.

Similarly, methods like ML-NMR are valid for inference at the individual level, but not designed for population-based inference.

Note that, throughout this discussion, we have focused on non-collapsible measures of treatment effect, where marginal and conditional estimates may be very different and the conflation of these may lead to bias.

Collapsibility does not hold for most measures of interest in population-adjusted indirect comparisons, such as (log) hazard ratios or (log) odds ratios in oncology applications, when investigating time-to-event or binary outcomes, respectively.

Nevertheless, clarifying the target of inference is also crucial where the measure of effect is collapsible, in order to ensure adequate uncertainty quantification.

For instance, in a linear model with continuous outcomes, the treatment effect (mean difference) is collapsible, and the magnitudes of marginal and conditional effects coincide, on expectation, in the absence of confounding.

If one runs a simple regression of outcome on treatment assignment, the point estimate of the regression coefficient has both a conditional and a marginal interpretation.

However, if one includes further covariates for adjustment, the magnitude of the residuals, residual variance and the standard error of the treatment coefficient are reduced because the covariates, if prognostic of outcome, absorb some outcome heterogeneity that was unexplained in the simple regression.

In this case, interpreting the treatment coefficient of the multivariable regression as a marginal treatment effect would lead to an underestimation of uncertainty.

In HTA, this means that uncertainty will be incorrectly propagated to the wider health economic model.

This is undesirable from the economic modeling point of view and has a negative impact on the “probabilistic sensitivity analysis”, the (often mandatory) process used to characterize the impact of the uncertainty in the model inputs on the decision-making process.

In HTA, when making inferences and policy decisions at the population level, marginal or population-average effect estimates are desirable.

Because the policy is to be uniformly applied to the target population, we want the treatment effect to be conditional on the entire population distribution of covariates, such that we marginalize over the individual-level covariates.

We are interested in a single marginal effect for a specific population, not in a conditional estimate, which is a subgroup-specific measure of effect, allowed to vary across a population at each possible combination of covariates.

This may be the case when making treatment decisions, involving solely clinical efficacy or effectiveness, or reimbursement decisions that also take into account cost considerations.

For treatment decisions, marginal effects may be of interest if deploying the treatment uniformly across the target population.

In reimbursement decisions, marginal effect estimates should inform the mean treatment benefit across the population in a cost-effectiveness analysis.

Conversely, conditional treatment effect estimates are preferred for inference at the individual level.

These are clinically relevant as patient-centered evidence in a physician-patient context, where decision-making relates to the treatment benefit given the specific covariates of an individual patient.

Because treatment effects may be heterogeneous across individuals, a finer resolution is of interest in precision or personalized medicine.

In the health decision sciences, conditional treatment effects would also be of interest for individual-level microsimulation models that simulate the impact of interventions or policies on individual trajectories.

An active area of discussion is whether the primary target of inference in randomized controlled trials should be a marginal or a conditional effect.

When approving new treatments, regulators have traditionally focused on marginal estimates as population-level summaries.

Nevertheless, conditional effect estimates, adjusted for important prognostic covariates, are increasingly reported as the key analysis.

This is due to greater interpretability when applying the results of the trial to individual patients, improved efficiency and statistical power for hypothesis testing, and correcting for (chance) imbalances in baseline characteristics across treatment groups.

The finding that MAIC has unstable model standard errors and high empirical standard errors under poor overlap between studies or small study sample sizes is an important one.

This suggests that the use of MAIC should be avoided when the weights are unstable and dependent on a small number of subjects, i.e., a small effective sample size, a situation that is fairly common in practical applications of MAIC, as evidenced by Phillippo et al. in a recent review [4].

Poor precision and accuracy will be exacerbated in unanchored comparisons, which require accounting for all variables that are prognostic of outcome in order to remove bias.

Almost invariably, the level of between-study overlap will decrease as a greater number of covariates are accounted for.

MAIC is a weighting method that cannot extrapolate where the between-study overlap is insufficient.

However, regression adjustment approaches can extrapolate beyond the covariate space observed in the AB patient-level data.

Therefore, the development of a regression adjustment method for indirect comparisons that targets marginal treatment effects would be impactful.

Currently, to our knowledge, the only peer-reviewed population adjustment methodology that targets marginal effects for indirect treatment comparisons with limited subject-level data is MAIC.

MAIC explicitly targets dAB(AC), and care must be taken to ensure that dAC(AC) is also a marginal treatment effect, so that compatible estimates are combined in the indirect treatment comparison, and to avoid bias when the measure of effect is non-collapsible.

An important weakness of MAIC is that it is only applicable to pairwise indirect comparisons, and it is unclear how to generalize the method for application in larger network structures.

Therefore, a research priority is to extend MAIC, or to adapt ML-NMR, so that marginal treatment effects can be estimated in scenarios where larger networks are required.

Alternatively, the decision-maker could specify a target population for a specific disease area and clinical outcome, and all manufacturers could conduct their indirect comparisons in the target population.

Note that larger networks will be less vulnerable to bias induced by imbalances in effect modifiers, as the heterogeneity “averages out” across the network, and the consistency assumption of network meta-analysis can be relaxed using random effects models.


We thank Remiro-Azócar, Heath, and Baio (R-AHB) for their letter to the editor [1], in response to our recent article presenting a simulation study comparing the performance of methods for population-adjusted indirect comparison [2].

R-AHB discuss the important issue of target estimands with noncollapsible effect measures, expanding upon the discussion in sections 4.3 and 7 of our article [2].

R-AHB distinguish between marginal and conditional treatment effect estimates and explain that matching-adjusted indirect comparison (MAIC) targets marginal effects whereas simulated treatment comparison (STC) and multilevel network meta-regression (ML-NMR) target conditional treatment effects.

They conclude that “methods like MAIC are valid for population-based inference, but not “fit for purpose” when inference is at the individual level, whereas methods like ML-NMR are valid for inference at the individual level, but not designed for population-based inference.” Furthermore, they assert that marginal treatment effect estimates are necessary for population-based inference as required for decision-making in Health Technology Assessment (HTA).


We welcome and encourage debate of these issues, which—despite much discussion in the literature on randomized controlled trials (RCTs) [3-6] and observational epidemiology [7-9]—have largely been overlooked in the literature on population adjustment and meta-analysis to date.

However, whilst we agree with R-AHB that population-based inference is required for HTA, we disagree that methods like ML-NMR are not appropriate to obtain population-average estimates for HTA.

In this response, we further clarify the use of conditional estimates to inform population-average treatment effects and why we believe these are appropriate target estimands for decision making.

We also correct some important inaccuracies in R-AHB’s letter regarding the characterization of the methods (in particular ML-NMR) and interpretation of our simulation study results.


Healthcare decision making requires estimates of average relative treatment effects between each treatment of interest in the decision target population.

Ideally, such estimates would be provided by a well-designed, representative RCT comparing all treatments of interest in the decision target population.

To help solidify ideas and define terminology, let us first consider the analysis of this ideal RCT.


The simplest analysis of this ideal RCT is an unadjusted analysis (eg, ANOVA or regression including only the main effect of treatment).

However, this is not the most efficient approach.

Instead, it is recommended practice to include prespecified prognostic factors in the analysis model (eg, ANCOVA or regression including main effects of covariates) [3-6,10].

The adjusted analysis is more powerful and more efficient because some of the additional variation in the outcome not due to treatment has been conditioned on the covariates [3-6].


The unadjusted analysis results in marginal treatment effect estimates, whereas the adjusted analysis results in conditional treatment effect estimates.

Since only main effects of covariates and no interactions with treatment have been included in the adjusted analysis, the conditional treatment effects apply over the entire study population; they do not vary by covariate values and are not subgroup-specific.

Thus, they can be considered population-average conditional treatment effects, and can be used to make treatment decisions for the entire population represented by this ideal RCT.


When working with noncollapsible effect measures such as odds ratios or hazard ratios, conditional and marginal treatment effects do not in general coincide; conditional effects will lie further from the null [11].

We should also be clear that the everyday usage of “conditional” to mean “depends upon” is misleading here: indeed, marginal estimates are more strongly dependent on the population than these conditional estimates, because the marginal estimates are affected by differences in all prognostic factors (observed and unobserved) whereas the conditional estimates will not be affected by differences in observed prognostic factors [4].


R-AHB are correct that other types of conditional effects depend on specified covariate values when treatment-covariate interactions have been included in the analysis, and are then appropriate for decision-making for individuals.

To avoid further confusion, we refer to these conditional effects from analyses including interactions as individual-level conditional treatment effects, and refer to conditional effects from analyses without interactions as population-average conditional treatment effects.

However, R-AHB conflate these individual-level and population-average conditional treatment effects, which have different interpretations and different uses for decision making.

The ML-NMR model is parameterized in terms of individual-level conditional treatment effects, since treatment-covariate interactions are included in the analysis model in order to adjust for differences in effect modifiers between studies (population adjustment).

However, ML-NMR can still produce estimates of marginal and conditional population-average treatment effects by integration over the covariate joint distribution in the target population, as we have described previously [2,12] and reiterate in the following section.


Both marginal and population-average conditional estimands reflect a “population-average treatment effect” as both apply to the entire population and are averaged over the distribution of covariates in the population.

The marginal estimand reflects the average treatment effect over individuals in the target population without any knowledge of the distribution of prognostic covariates in the sample.

The population-average conditional estimand reflects the average treatment effect over individuals in the target population accounting for the distribution of prognostic covariates.


We consider that the population-average conditional treatment effect is the most appropriate target estimand for decision makers, primarily because it reflects the recommended analysis that would be undertaken in the ideal evidence scenario described above.

Decision makers typically have a well-defined target population in mind, however, the marginal estimand does not make full use of this information.

The population-average conditional estimand is more efficient because it accounts for the known distribution of prognostic factors in the target population [3-6].


Although the ML-NMR model is parameterized on individual-level conditional treatment effects on a given linear predictor scale, ML-NMR can produce population average estimates for a range of quantities of interest in a target population through appropriate use of integration, as described in section 2.5 of Phillippo et al. [ 12].


When the quantities of interest are population-average conditional treatments effects dab(P) between treatments b and a in population P, integration simplifies to plugging in mean covariate values x(P) in the target population since these are defined on the linear predictor scale, as given in equation (9) of our article [2]:


( dab(P) = ∫𝔛 = xT (P) 𝜇(P) + xT ( 𝜷 2,b − 𝜷 1 + 𝜷 ) + 𝛾b − 𝛾a, 2,a ) 2,b ) + 𝛾b f(P)(x)dx − ∫𝔛 ( 𝜇(P) + xT ( 𝜷 1 + 𝜷 2,a ) ) + 𝛾a f(P)(x)dx (1)


where 𝜷 2,b and 𝜷 2,a are coefficients for effect modifier interactions, and 𝛾b and 𝛾a are individual-level treatment effects at the reference level of the covariates x = 0.

𝜷 1 are coefficients for prognostic (main) effects of covariates, 𝜇(P) is a distribution for baseline response in population P, f (P)(⋅) is the joint covariate distribution in population P with support 𝔛. This does not mean that the resulting estimate is only appropriate for individuals with the mean covariate values, as suggested by R-AHB.

It just happens that the population-average conditional treatment effect estimate on the linear predictor scale is equivalent to that for individuals with the mean covariate values.


Health economic models typically require population-average absolute effects, such as average event probabilities pk(P) on treatment k, which can be produced following Phillippo et al. [ 12] and using the notation in our article [2] as


pk(P) = ∫𝔛 g−1 ( 𝜇(P) + xT ( 𝜷 1 + 𝜷 2,k ) ) + 𝛾k f(P)(x)dx, (2)


where g(⋅) is a suitable link function (eg, logit).


Contrary to the assertion of R-AHB that ML-NMR cannot produce marginal population-average treatment effect estimates, ML-NMR can indeed estimate the marginal population-average treatment effect Δab(P) between treatments b and a in population P, simply by working with the population-average absolute effects from (2):


Δab(P) = g(pb(P)) − g(pa(P)).

( 3)


Estimates of other summaries of marginal population-average treatment effects such as risk differences or relative risks can be produced by similar manipulation of pb(P) and pa(P).

Again, we note that dab(P), pk(P), and Δab(P) are not subgroup-specific but apply over the entire target population P, since all covariates (including effect modifiers) have been integrated over.


MAIC directly targets the marginal population-average treatment effect, and cannot estimate the (more efficient) population-average conditional treatment effect unless suitable adjusted estimates are available from the AgD study.

As described by both R-AHB and ourselves [2], STC in typical usage estimates neither the marginal or conditional population-average treatment effect and will be biased for either estimand, because in typical use

STC combines conditional and marginal effects.

Moreover, Equations (2) and (3) make it clear that marginal population-average relative effects depend not only on the distribution of effect modifiers, but also on the distribution of all prognostic variables and the population baseline risk.

Thus, the marginal population-average treatment effects Δab(P) depend more strongly on the population of interest than the population-average conditional treatment effects dab(P), and are less generalizable/transportable as a result [4].

This is an additional concern for MAIC and STC, which produce marginal treatment effect estimates specific to the aggregate study population in a population-adjusted indirect comparison, and may not be representative of the decision target population in either prognostic factors or effect modifiers [13,14].


As we have argued above, population-average conditional treatment effects are appropriate target estimands for efficient decision making.

Our simulation study [2] is therefore designed to evaluate the performance of the methods against the population-average conditional treatment effects dab(P)—not the individual-level conditional treatment effects 𝛾k as R-AHB claim.


R-AHB suppose that much of the observed bias for MAIC in our simulation study is due to evaluating its performance against the wrong estimand, since MAIC targets the marginal population-average treatment effect.

However, standard unadjusted Bucher indirect comparisons also target marginal estimands, and yet MAIC manages to substantially increase the bias compared with these in some scenarios.

Moreover, we would expect STC to perform poorly for the same reasons, since it mixes conditional and marginal estimates.

However, in our simulations STC performed well and was seen to be unbiased when the requisite assumptions were met.

Intuitively, therefore, our simulation scenarios must be such that the differences between marginal and population-average conditional estimands are small.

More formally, we can investigate this claim using the formula of Matthews and Badi [15] for the ratio between the conditional and marginal estimands, which depends on the strength of the covariate effect and the variance within the population.

Using this result, we determine that—even in the worst cases—the difference between marginal and population-average conditional estimands in the scenarios we investigated is less than 0.5%.

Therefore, the performance issues demonstrated for MAIC are not due to any meaningful difference in estimands, but are due to the fundamental inability of MAIC to extrapolate and the resulting bias and instability as population overlap decreases.


R-AHB contrast our results against a simulation study of their own [16], which shows that MAIC can remain unbiased (for the marginal estimand) even with only moderate overlap between populations.

The observed difference in performance is due to R-AHB considering only matching covariate means across populations, whereas we consider matching both means and variances (first and second moments) which is a common approach [14] and follows the original description of the method [17].

The results of R-AHB suggest that matching on covariate means only might be less sensitive to reduced population overlap and may be able to tolerate lower levels of overlap before issues arise, since this is a much less exacting requirement.

This has also been observed in other simulation studies [18,19].

However, the question of when, if at all, it is preferable or necessary to match higher moments between populations for MAIC remains an interesting area for further theoretical research and simulation studies.


We welcome the much-needed discussion of target estimands in the letter from R-AHB, which has largely been overlooked in the population adjustment literature to date.

We hope that our response has served to further clarify the issues surrounding estimands for noncollapsible effect measures.

Here, we have argued the case for considering population-average conditional treatment effects as appropriate targets for efficient decision making.

In their letter, R-AHB state that “methods like ML-NMR are valid for inference at the individual level, but are not designed for population-based inference.” However, we have demonstrated that this is not the case: ML-NMR can indeed support inference at the individual level, but can also provide estimates of both marginal and conditional population-average treatment effects, as well as the population-average absolute effects typically required for health economic modeling.

Moreover, ML-NMR is likely to be a more efficient approach than MAIC even when targeting marginal population-average treatment effects, since regression adjustment is typically more efficient than weighting [20].

We have shown that the results of our simulation study, including the poor performance of MAIC in many scenarios, are not an artifact of noncollapsibility and incompatible estimands, and are pertinent regardless of whether population-average conditional or marginal estimands are of interest.


R-AHB refer to ML-NMR as the “gold standard” for estimating conditional population-adjusted treatment effects from mixtures of IPD and AgD, and suggest further work extending ML-NMR to estimate marginal population-adjusted as a research priority.

However, as we have described above, ML-NMR can be used to obtain marginal population-adjusted treatment effects, and we would suggest therefore that ML-NMR may also be considered the “gold standard” for estimating marginal population-average treatment effects.

Analysts and decision makers should carefully consider which target estimand is most appropriate to their needs and, as we have argued, population-average conditional treatment effects as targeted by a hypothetical “ideal RCT” may be a more efficient choice than marginal estimands which do not account for known population characteristics.


It is estimated that 16.6% of Americans will experience a major depressive episode at some point in their lifetimes (1).

Depression is associated with functional impairment (2), cognitive impairment (3), poor quality of life (4), and in- creased risks of coronary heart disease (5) and type 2 diabetes mellitus (6).

Persons who are depressed also are more likely to abuse drugs and alcohol (7) and to commit suicide (8).

Considering the high prevalence and debilitating effects of depression, as well as the challenges in providing effective treatment to depressed individuals, it is critical to identify modifiable risk factors.

Because more than two-thirds of American adults (68.0%) are overweight or obese (9), re- searchers have begun to question whether being overweight or obese might be 1 such risk factor.


Previous research in this area has yielded unclear conclu- sions (10).

Much of the research has been cross-sectional (10, 11), making it impossible to determine whether overweight/ obesity causes depression or vice versa.

In terms of prospec- tive studies, some researchers have investigated whether de- pression precedes obesity (12, 13) (often hypothesizing that depressed persons gain weight because they eat more and move less than their healthy peers), whereas others have inves- tigated whether obesity precedes depression.


In a recent meta- analysis, Luppino et al. ( 13) found significant associations in both directions.

In 9 prospective studies that examined the po- tential effects of depression on obesity, the pooled odds ratio was 1.58, whereas in 8 prospective studies that examined the relationship in the other direction (the potential effects of obe- sity on depression), the pooled odds ratio was 1.55.


Importantly, most studies have not considered the relation- ships between percentage of body fat or level of abdominal obesity (i.e., waist circumference) and depression but rather have relied solely on body mass index (BMI) to categorize levels of overweight/obesity (10, 13).

Fat tissue, particularly visceral abdominal fat tissue, is closely associated with meta- bolic health status (14, 15), and thus use of these measures might provide a clearer picture of which aspects of obesity (i.e., the extra weight itself or the common metabolic comorbid conditions) are most threatening to mental health.


Most pro- spective studies also have not adequately considered physical activity level or cardiorespiratory fitness (CRF) (10), both of which might be protective against the development of depres- sive symptoms (16, 17).


Previous longitudinal analyses of data from the Aerobics Center Longitudinal Study (ACLS) showed that higher CRF level was associated with a lower incidence of elevated depressive symptoms (16) and that greater odds of incident depression complaints resulted from greater declines in CRF in late middle age (18).

In the present article, we build on these previous findings (16, 18) by examining the indepen- dent and combined associations of overweight/obesity and CRF with the development of elevated depressive symptoms in the ACLS.


The ACLS is an open cohort study in which participants enter and undergo health examinations at the Cooper Clinic (Dallas, Texas) at different time points and complete periodic follow-up health surveys by mail (19).

Participants are pri- marily self-referred or sent by their employer for preventive medical examinations.


Eligible participants for this particular study included men and women 20-100 years of age who had a baseline health examination between 1979 and 1998 (n = 35,392) and who returned at least 1 mail-back health survey in 1990, 1995, or 1999 (19,521 were excluded because they did not return any surveys).

If individuals returned multiple surveys, only their first survey response was used.

Participants were excluded if they did not achieve at least 85% of their age-predicted maximum heart rate during treadmill fitness testing (n = 391) or if they had cardiovascular disease, cancer, or ab- normal resting or exercise electrocardiogram results at baseline (n = 1,602).

Individuals also were excluded if they had any previously diagnosed mental disorder, such as de- pression or anxiety, or if they had a nervous disorder, had thoughts of suicide, or had undergone psychiatric counsel- ing in the past (n = 1,279).

This mental health information was gathered using the standardized Cooper Clinic medical history questionnaire, which included the question, "Please indicate whether you have ever had a significant problem with any of the symptoms or conditions listed below (yes/ no)." Final analyses included 12,599 adults who met the above criteria.

All subjects provided written informed con- sent to participate in the ACLS.

The Cooper Institute's in- stitutional review board approved the ACLS protocol each year.


Trained clinical personnel conducted all baseline examina- tions according to the Cooper Clinic's standardized manual of operations (19).

Participants provided demographic infor- mation, personal and family health histories, and information on personal health habits (e.g., alcohol intake, smoking habits, physical activity level) via standardized Cooper Clinic questionnaires.


Clinical personnel collected height and weight measure- ments with participants wearing light-weight clothing and no shoes.

BMI was then calculated as weight in kilograms di- vided by height in meters squared.

Waist circumference was measured at the level of the umbilicus with a plastic tape mea- sure.

Percentage of body fat was assessed using hydrostatic weighing, the sum of 7 skinfold measurements, or both mea- surements following standardized protocols.

Standard clinical definitions were used for BMI (overweight: 25.0-29.9; obese: ≥30), waist circumference (for women, abdominal obesity was defined as a waist circumference >88.0 cm; for men, it was >102.0 cm), and percentage of body fat (for women, obesity was defined as a percentage ≥30%; for men, it was ≥25%).


Maximal exercise treadmill testing was used to determine CRF (20).

CRF was defined as total symptom-limited time on the treadmill, with the test endpoint being volitional exhaus- tion or termination by a supervising physician.

In our primary analyses, CRF level was grouped using quintiles of the sex- specific distribution of maximal exercise duration in the over- all ACLS population (21).


Participants were asked to complete the 10-item Center for Epidemiological Studies Depression Scale (CES-D) ques- tionnaire as part of a mail-back survey in 1990 (n = 6,657 par- ticipants), 1995 (n = 6,475), or 1999 (n = 7,579) (22, 23).

The aggregate survey response rate was approximately 65% (16, 21).

Nonresponse bias is a concern in epidemiologic surveillance.

This issue has been investigated in the ACLS previously (24).

Baseline health histories and clinical measures were similar between responders and nonresponders and between early and late responders (24).

A score of 8 or higher indicated the presence of elevated symptoms (which corresponds to a cut- point of 16 on the 20-item CES-D).

This measure has been shown to be reliable and valid (22, 23, 25), and a cutpoint of 8 has been used in multiple previous studies (26, 27).


All analyses were sex-specific.

Separate logistic regressions tested the associations between each measure of overweight/ obesity and the onset of elevated depressive symptoms (CES-D score ≥8).

To account for differences in survey response patterns among study participants and for the possibility that Downloaded from https://academic.oup.com/aje/article/181/5/311/195235 by McGill University School of Law user on 16 May 2024 external events might have differentially affected responses to the CES-D during the 3 survey periods, dummy variables were created to indicate whether the outcome measurement was from 1990, 1995, or 1999.

Covariates in model 1 included age and baseline examination year; covariates in model 2 included those in model 1 plus survey response year (1990, 1995, or 1999), current smoking (yes or no), heavy alcohol intake (for men, >14 vs. ≤14 drinks per week; for women, >7 vs. ≤7 drinks per week), physical activity level (active or sedentary), and personal history of hypertension, diabetes, and hypercho- lesterolemia; and covariates in model 3 included those in mod- els 1 and 2 plus CRF (maximal treadmill time).

Models were repeated using CRF as the predictor variable, substituting per- centage of body fat for CRF as a covariate in the full model (model 3).

Additional logistic regression analyses were used to determine the joint association of CRF and overweight/ obesity with onset of depressive symptoms (separate interaction analyses were run for each fatness measure).

In the joint analy- ses, CRF was grouped into a binary variable of unfit (quintile 1, the lowest 20%) and fit (quintiles 2-5, the remaining 80%).

Although no consensus clinical definition of unfit currently ex- ists, the approach we used for defining unfit is a standardized method in the ACLS.

Previous reports from the ACLS have shown that unfit by this definition is an independent predictor of morbidity and mortality (21, 28-30).

In addition, we col- lapsed BMI into categories of <25 and ≥25 in order to main- tain a good number of participants in the cross-tabulation cells.

Additionally, we conducted sensitivity analyses to examine the influence of missing CES-D responses at follow-up.

We used the data set with complete CES-D scores from 1990 to impute the missing CES-D scores using the predicted value from the linear regression, which was adjusted for all variables in model 2.


We then combined the reported data with imputed data and re- peated the main analyses.

Because the magnitude and direction of the associations from the sensitivity analyses were very sim- ilar to those from the initially proposed analyses, we did not re- port them in this article.

Please see the Web Tables based on imputed analyses.

All P values are 2-sided, with an α level of 0.05.

All analyses were performed using SAS, version 9.2 (SAS Institute, Inc., Cary, North Carolina).


The mean baseline age for the study sample was 44.9 (stan- dard deviation, 9.5) years, and 18.4% of participants (n = 2,315) were women (Table 1).

Table 1 also presents sample Table 4), the results of model 1 indicated that the odds of elevated depressive symptom onset were 17% greater for overweight participants and 40% greater for obese participants than for normal-weight participants.

In model 2, the odds of reporting elevated depressive symptoms at follow-up were 16% and 35% higher, respectively.

BMI was not significant in model 3 (which additionally controlled for CRF) for men (P = 0.82).


For women (Table 3), waist circum- ference was not predictive of elevated depressive symptom Downloaded from https://academic.oup.com/aje/article/181/5/311/195235 by McGill University School of Law user on 16 May 2024 onset in any of the 3 models (P > 0.06).

For men (Table 4), in models 1 and 2, the odds of reporting elevated depressive symptoms at follow up were 32% and 27% higher, respec- tively, for those with a waist circumference greater than 102 cm than for those with a waist circumference of 102 cm or less.

Results of model 3 were not significant for men (P = 0.48).


Percentage of body fat.

Results from models 1 and 2 show that women with a body fat percentage of 30% or higher had 38% and 36% higher odds, respectively, of developing ele- vated depressive symptoms than did women with a body fat percentage below 30% (Table 3).

For men (Table 4), results from models 1 and 2 show that those with a body fat percent- age of 25% or higher had 28% and 23% higher odds, respec- tively, of developing elevated depressive symptoms than did those with a body fat percentage less than 25%.

Model 3 showed no significant association between percentage of body fat and onset of depressive symptoms for women or men (P = 0.29 and 0.69, respectively).


CRF level was significantly associated with the onset of el- evated depressive symptoms in all models (Table 5), and thus only results from the full model (model 3) will be reported here.

Compared with women with the lowest levels of CRF (quintile 1), women in the 2 highest CRF groups (quintiles 4 and 5) had 59% and 52% lower odds, respectively, of report- ing elevated depressive symptoms at follow-up.

Compared with men with the lowest levels of CRF (quintile 1), men in the 2 highest CRF groups (quintiles 4 and 5) had 40% and 46% lower odds, respectively, of reporting elevated de- pressive symptoms at follow-up.


Figures 1 and 2 show the joint association of fitness and fat- ness with the risk of developing elevated depressive symptoms.

In interaction analyses using BMI as the fatness measure, men (n = 708) and women (n = 72) who were both unfit and over- weight or obese had 76% and 77% higher odds, respectively, of developing elevated depressive symptoms compared with the normal-weight and fit reference groups.

Using waist circum- ference as the fatness measure, men (n = 425) and women (n = 31) who were both unfit and overweight or obese had 83% and 125% higher odds, respectively, of developing elevated depres- sive symptoms.

Using percentage of body fat, men (n = 542) and women (n = 96) who were both unfit and overweight or obese had 63% and 108% higher odds, respectively, of devel- oping elevated depressive symptoms.


When using waist circumference as the fatness measure, there was no significant difference in the odds of developing elevated depressive symptoms in fit men and women, regard- less of fatness level (i.e., lean and overweight/obese women who were fit had the same odds of developing depressive symp- toms).

However, among overweight and obese men, there was a significant difference in the odds of developing elevated de- pressive symptoms, with unfit/fat men having 63% higher odds than fit/fat men.

There was also no significant difference in the odds of developing elevated depressive symptoms in fit women, regardless of fatness level, when using BMI as the fatness mea- sure.

Conversely, unfit/fat women had 94% higher odds of de- veloping elevated depressive symptoms than did fit/fat women.


In additional comparisons using the fit/fat group as the refer- ence (data not shown), unfit women with BMI less than 25 had 139% higher odds of developing elevated depressive symptoms than did fit women with a BMI of 25 or higher (odds ratio = 2.39; 95% confidence interval: 1.05, 3.58).

Unfit men with per- centage of body fat less than 25% had 38% higher odds than did fit men with body fat percent of 25% or higher (odds ratio = 1.38; 95% confidence interval: 1.02, 1.87).


In the present study, we demonstrated that being overweight or obese is not predictive of the onset of elevated depressive symptoms when CRF level is considered.

This was the case regardless of whether BMI, waist circumference, or percentage of body fat was used to determine adiposity.

Importantly, fat- ness was often predictive of the onset of depressive symptoms when CRF level was not considered.

Because most previous studies that linked fatness and depression did not adequately consider physical activity or CRF (10), the relationship be- tween fatness and depression might often be overestimated.


Consistent with a previous ACLS investigation (16), we found that higher CRF was associated with lower odds of el- evated symptom onset in overweight and obese individuals.

In some (but not all) analyses, fit/fat individuals appeared to have no higher odds of developing elevated depressive symp- toms than did fit, normal weight-individuals, whereas unfit/ fat individuals (i.e., overweight and obese individuals with low CRF) were at significantly higher risk (increased odds rang- ing from 63% to 125%).

Interestingly, fit/fat individuals also had a lower risk of depression than did unfit/not fat individ- uals in multiple analyses, which reiterates that CRF might be more essential to positive mental health than is low fatness.


It is important to clarify that these findings do not indicate that there is no link between obesity and depression; rather, they suggest that CRF might be a better predictor of depres- sive symptom onset and might even be protective against the potential depressive effects of obesity.

Indeed, severely obese individuals might suffer functional impairment because of their excess weight and have more negative thoughts about their health status (31).

Obese women and persons with a high socioeconomic status might also be at a higher risk for depression because of social stigma associated with obesity, body image dissatisfaction, and repeated dieting (31).

Obe- sity is also associated with increased inflammation, possible hypothalamic pituitary adrenal axis dysregulation, and in- creased risk of diabetes mellitus and insulin resistance, all of which might contribute to a link with depression (13).


Many psychosocial and physiological mechanisms could ex- plain how CRF might negate or buffer against the possible mul- tilevel, prodepressive effects of obesity.

Individuals who attain high CRF through regular physical activity might experience increased self-efficacy, self-esteem, and social reinforcement (32, 33).

From a physiological standpoint, regular physical ac- tivity might be associated with increased quantity and activity of key neurotransmitters (e.g., serotonin, norepinephrine, dopa- mine) (34), attenuation of the stress response (e.g., lower corti- sol levels, lower cardiovascular reactivity to mental stressors) (35, 36), and reduced inflammation (37).

Antidepressant prop- erties of physical activity might also be the result of increased levels of key growth factors (e.g., brain-derived neurotrophic factor) and hippocampal neurogenesis (38-40).


A strength of the present study is the use of fatness mea- sures that are indicative of metabolic health (i.e., waist cir- cumference and percentage of body fat in addition to the standard BMI), because in a recent study, Hamer et al. ( 41) showed that poor metabolic health, rather than obesity, was associated with an increased risk of depression.

Other major strengths of the present study include its longitudinal design, large sample size, and use of maximal exercise testing to quantify CRF, which is a better marker of regular physical ac- tivity participation than are self-reported activity question- naires.

Additionally, this study followed participants for an extensive period of time (mean = 9.9 years).


A limitation of this study is its generalizability to other populations, given that the ACLS is composed largely of well-educated, white individuals with a middle-to-high socio- economic status.

Further, a large number of individuals who underwent examinations at the Cooper Clinic between 1979 and 1998 did not return any mail-back health surveys in 1990, 1995, or 1999 and were therefore excluded from analy- ses.

Although the nonresponse issue has been studied in the overall ACLS population (24), the fact that it might have af- fected our results remains a limitation.

Importantly, the mag- nitude and direction of the associations between fitness, fatness, and depressive symptoms did not change when we used imputed data to account for missing CES-D responses.


An additional limitation is the use of a self-reported, non- diagnostic measure of depression (i.e., the CES-D).

However, in a recent meta-analysis of studies that investigated the po- tential antidepressant effects of exercise, Rethorst et al. ( 42) showed that effect sizes did not differ between studies that conducted a clinical interview and those that did not.

Addition- ally, depressive symptoms have been linked to poor health and impaired functioning regardless of whether clinical diagnostic criteria for depression are met (43).

Regardless, we did not use a structured clinical interview to diagnose depression, and in- terpretations should be made accordingly.


It should be noted that physical symptoms of depression might overlap with symptoms related to low fitness levels.

For example, 2 items on the CES-D might be interpreted as reflecting physical fatigue (i.e., "Did you feel that everything you did was an effort?" and "Did you feel you could not get 'going'?") rather than apathy or low motivation.

The relation- ship between low fitness and depressive symptoms, as as- sessed using the CES-D, could theoretically be driven by only the questions that reflect fitness itself; indeed, in a sen- sitivity analysis, the prevalence of elevated depressive symp- toms dropped substantially in our sample when these 2 items were omitted.

Future research should investigate whether composite measures of depression, such as the CES-D, are most appropriate for studying the fitness-depression relation- ship.

Because CES-D data were not available at baseline, only a standardized Cooper Clinic question regarding mental health history was used to exclude individuals with previous mental health concerns.

Thus, it is possible that some individ- uals with high depressive symptoms at baseline might have been inadvertently included in analyses, and therefore reverse causation cannot be entirely ruled out.


Having a low CRF level might be more strongly associated with an increased risk of elevated depressive symptoms than is being overweight or obese; as such, promoting physical ac- tivity to improve CRF rather than weight loss, per se, might Downloaded from https://academic.oup.com/aje/article/181/5/311/195235 by McGill University School of Law user on 16 May 2024 be most effective for reducing the risk of developing elevated depressive symptoms.

It will be important to determine whether gains in CRF are necessary or whether other physiological or psychosocial factors are actually more critical mediators of the antidepressant effects of physical activity.

If improvements in CRF are not necessary, activities at lower intensities (which might be more comfortable for overweight and obese individ- uals) might lower the risk and/or reduce depressive symptoms.

Indeed, results from a 2008 review suggested that relatively low levels of physical activity might protect against depres- sion (44).


Fit/fat individuals might be less likely to develop elevated depressive symptoms than unfit/fat and even unfit/not fat individuals, which suggests that fitness might have antide- pressant effects strong enough to buffer against general de- pressive symptoms, as well as any prodepressive effects of being overweight or obese.

These findings contribute to the established fit versus fat literature (29, 30, 45) by demonstrat- ing that mental health, like physical health, might also be more greatly influenced by CRF than by adiposity.

It should be emphasized that physical activity, the primary determinant of CRF, is firmly established as an essential feature of weight- maintenance and weight-loss strategies.

Thus, in addition to all the plausible pathways linking physical activity directly to improved mental health (irrespective of weight loss), maintaining a healthy body weight or losing weight through a physically active lifestyle might also improve mental health.


Parsing the relative contributions of fitness and fatness to various disease states has long interested investigators, despite their almost certainly intertwined causal relationships.

In this issue of the Journal, Becofsky et al. ( Am J Epidemiol. 2015;181(5):311–320) examined these constructs in relation to depressive symptoms assessed using the Center for Epidemiologic Studies Depression Scale, a widely used and well-validated instrument that ascertains several domains related to depression.


The authors found stronger and more consistent associations of elevated depressive symptoms with objectively measured fitness than with any measure of fatness.

These analyses highlight a serious concern in analyses of risk factors for complex clinical constructs—the likelihood of differential misclassification with respect to the risk factors of interest in scales with multiple, wide-ranging domains.

In this interesting example, the multi-item Center for Epidemiologic Studies Depression Scale, which validly captures the multiple domains of depression, contains measures of psychomotor retardation that could easily reflect fitness itself.


Because an instrument’s validity is uncertain if selected items are removed and gold-standard measures of complex clinical constructs are rarely truly gold, options to remedy this problem in most epidemiologic studies are few.

Nonetheless, their analysis illustrates the particular caution needed when examining exposures that plausibly relate to misclassification in multidimensional outcomes of interest.


As the average body mass index of Americans grows and the prevalence of obesity reaches legitimately epidemic levels, studies that relate obesity to chronic diseases have proliferated in both number and importance.

Although obesity is demonstrably multifactorial, with strong causal links to genetics, physical activity, and caloric intake among many others, particular interest has remained in questions related to the relative importance of physical activity and fitness compared with obesity per se.

In the catchy jargon that undoubtedly fuels the topic, readers of the American Journal of Epidemiology have long had the opportunity to ponder the debate of “fit versus fat” [1].


To some degree, this debate is poorly suited to epidemiologic inquiry.

Because long-term physical activity level is measured with substantial misclassification by standard epidemiologic scales and is an upstream determinant of both fitness and fatness, its role is rarely possible to evaluate fully.

Moreover, activity and obesity are almost certainly causally related to each other, with activity leading to greater caloric expenditure (and weight loss) and obesity leading to a number of cardiopulmonary and musculoskeletal changes that can interfere with activity.

These types of complex interrelationships require appropriate methods that include repeated, accurate measurements in reasonably large samples [2].

Such a burden is particularly unlikely to be met for studies of fit versus fat, in which repeated standardized exercise tests or similar approaches would be required to measure fitness.

As a result, the quantitative contributions of fitness and fatness are likely to be poorly estimated, and their strong interrelationships raise legitimate questions about the clinical utility of trying to parse their separate roles too finely.

Indeed, organizations like the American Heart Association appropriately consider both weight and activity as primary, separate targets in their population health goals [3].


The question of fit versus fat in mental health represents an intriguing problem that has received relatively little direct attention.

In part, this question is particularly interesting because of the inconsistent nature of previous work on obesity, depression, and suicidality.

Cross-sectional and limited longitudinal studies have inconsistently found obesity to be associated with depressive symptoms [4-6].

Paradoxically, obesity has been consistently associated with a lower risk of death from suicide, although this may reflect differences by body mass index in mechanism rather than in suicidal ideation [7].

As a result of this inconsistency, clinicians and public health professionals have been left uncertain about the potential effects of the obesity epidemic on national mental health care needs.


The Aerobics Center Longitudinal Study of the Cooper Clinic represents a particularly well-suited setting in which to examine questions of fitness, as patients at this clinic undergo routine symptom-limited exercise testing.

Although this measure is not synonymous with cardiorespiratory fitness (classically defined as maximal oxygen uptake), it correlates well with other measures and requires much less specialized and intrusive equipment.

As Becofsky et al. noted [8], non-invasive measures of body mass index, waist circumference, and proportion of body fat are also routinely assessed; although bioelectric impedance and other tools to estimate fat mass are not used, these measures of adiposity are clearly strongly correlated in typical ambulatory populations free of overt disease.

Thus, in contrast to most cohort studies, this admittedly highly selected panel of patients has reasonable measures of both fitness and fatness.


To evaluate depression, Becofsky et al. took advantage of repeated administrations of the original version of the Center for Epidemiologic Studies Depression Scale (CES-D) [9], which has since been revised and remains freely available (http://cesd-r.com/) [10].

The CES-D is a remarkably widely used instrument, with strong psychometric properties and remarkable ease of use in even challenging settings and after translation [11-13].

When compared with diagnostic interviews and when using a standard cutpoint of 16 on the 20-item version, it has generally been reported to have a sensitivity and specificity for depression of 0.7–0.8, although lower values have been published and concern has been raised about its ability to separate depression from anxiety [11, 14, 15].

A number of shorter versions have been proposed, including the 10-item version administered by the Cooper Clinic, that generally appear to preserve much of the original scale’s reliability [16].


A quick examination of the scale itself is instructive.

The original and revised versions both explicitly include several domains related to depression, all of which reflect symptoms of depression noted in serial iterations of the Diagnostic and Statistical Manual of Mental Disorders.

Because depression has no single diagnostic finding, its diagnosis is necessarily multidimensional, which the CES-D reasonably captures; the revision added 2 questions on suicidality not included originally.

For diagnostic purposes, the CES-D as a simple scale lacks accuracy because all of the domains are counted equally, and hence weighted scoring has been recommended by some experts [17].

In epidemiologic contexts, however, the CES-D has almost always been used in its simplest version, which totals scores from 0–3 on its 20 questions.


The domains included on the CES-D—as with any scale used for a complex condition like depression that defies simple laboratory testing—represent complementary, interdependent depressive symptoms.

These include depressed mood, feelings of guilt, worthlessness, and helplessness, psychomotor retardation, loss of appetite, and sleep difficulties, all of which are plausibly associated with depression.

However, these symptoms are neither diagnostic of nor specific for depression, and thus the CES-D, although internally consistent and reliable, necessarily estimates depression with error.


Methodologically, the notion that the CES-D estimates depression with error is not of unique concern; few outcomes in large-scale studies are apt to use truly gold-standard measures, and even these may be measured with error, as will be noted further below.

However, the error in, say, measurement of total cholesterol in studies of lipid-lowering agents is commonly nondifferential; we do not expect persons who receive a drug to have their values measured with a net error that differs from those who receive placebo.

Although the spread in error may indeed differ between groups (particularly if absolute error increases with true values and if exposed individuals have systematically higher or lower true values than do unexposed individuals), we routinely and reasonably expect it to remain nondifferential, with observed values distributed equally above and below their true counterparts.


Unfortunately, the question of fitness versus fatness is apt to violate this assumption.

Because the CES-D has multiple, wide-ranging items, a number of exposures—including fitness and fatness—could conceivably lead to differential error.

In this case, specific questions on the 10-item CES-D ask about inability to “get going” and the feeling that everything is an effort.

If these items are influenced by a lack of fitness, as would seem likely, then they lead to differential misclassification, in which fit individuals have systematically lower CES-D values independent of their true level of depression.


Of note, the 20-item CES-D and its revision are unlikely to be any better in comparing fit versus fat, as the former includes both these questions and an item on appetite (i.e., one plausibly associated with obesity per se) and the latter substitutes a question on unintentional weight loss for appetite (an even greater obstacle).

How great of a bias this induces depends on the degree to which those questions are answered affirmatively among apparently depressed and nondepressed fit and unfit individuals and where the threshold for positivity is set; at the extreme, if answered affirmatively only and always among unfit individuals who would have otherwise been classified as nondepressed, such bias could easily overshadow true effects.


Further frustrating these problems is the lack of a true gold standard.

For depression, diagnostic interviews based on the Diagnostic and Statistical Manual of Mental Disorders (now version 5) conducted by expert psychiatrists are commonly considered the most rigorous form of confirmation.

However, even these could conceivably introduce differential error.

If, for example, clinicians believe that morbid obesity ought to lead to depression because it produces social stigmatization, then they may interpret responses to a diagnostic interview in that light and, in borderline cases, err on the side of a positive diagnosis.

Blinding of diagnostic assessors trained to identify nonverbal cues is not possible for conditions like obesity, and hence some differential error may even exist in gold-standard assessment techniques.

A similar concern has been voiced as a potential explanation for the discrepancy in risk of myocardial infarction between the Women’s Health Initiative and earlier observational studies of postmenopausal estrogen administration [18].


Where does this leave us? One reasonable question, particularly given the changes in the CES-D over time, is whether a revised instrument that simply excluded questions conceivably related to fitness itself could be used.

Becofsky et al. rightfully note [8] that the reliability and validity of abbreviated scales cannot be assumed, both because the coverage properties are necessarily diminished and because subtle cognitive biases related to the ordering of questions may be introduced [19].

That is, an ad hoc version of the CES-D that removes questions related to psychomotor retardation cannot be assumed to have the same properties as the original CES-D itself, although concern about whether those questions relate well to the remainder of the CES-D has already been raised elsewhere [20].

Fortunately, at least for the CES-D, efforts to validate shorter versions have been widespread and largely successful, including 4- and 8-item options that exclude those questions most plausibly related to fitness [21]; either of those options could have been used by the authors to address this concern with a reasonable assurance of validity.

However, the CES-D is almost unique in the bewildering array of iterations to which it has been subjected, and this solution will rarely be available for other scales used to measure recalcitrantly complex constructs.


In the end, Becofsky et al found stronger and more consistent associations of fitness than fatness with depressive symptoms, regardless of the measure of fatness used.

A number of difficulties limit the inferences that can be drawn, such as the substantial portion of missing outcome data and its implausibility of missingness at random, but the work by Becofsky et al. illustrates a larger problem likely to become increasingly common as we move to ever more challenging outcomes.

When more dimensions of an outcome exist, the possibility that an exposure is related to misclassification in one of its dimensions naturally grows.

When possible, investigators can assess the consistency of observed relationships across the subscales of larger instruments and in more expansive and shorter versions of the same instrument; relationships limited to single subscales or unique versions should receive appropriate scrutiny, although they still might reflect true associations with just some symptoms of a complex disease (e.g., differences in the effects of antipsychotic drugs on the “positive” and “negative” symptoms of schizophrenia).

That solution will unfortunately not be readily available in most circumstances.

Ultimately, the analysis by Becofsky et al. illustrates the particular caution needed when, like with fit and fat, investigators compare exposures that predict misclassification in addition to (or, worse, instead of) conditions of truly widespread epidemiologic interest.


We appreciate Dr. Mukamal’s thoughtful commentary [1] on our paper [2].

We are in agreement with Dr. Mukamal on a number of points, but wish to re-affirm and clarify our stance on others.


In his commentary, Dr. Mukamal says, “To some degree, this [fit versus fat] debate is poorly suited to epidemiologic inquiry” [1, p. 321].

He stresses that it is difficult to ascertain repeated, quality measurements of fitness in epidemiologic studies.

This point speaks to the importance of our paper and others published on the topic using data from the Aerobics Center Longitudinal Study cohort [3-6], from which “reasonable measures of both fitness and fatness” are available, as Dr. Mukamal notes [1, p. 322].


In speaking of fitness and fatness, Dr. Mukamal notes that “their strong interrelationships raise legitimate questions about the clinical utility of trying to parse their separate roles too finely” [1, p. 321].

Perhaps in terms of clinical utility, his statement is sound; if obese patients begin exercising and lose weight, it is unnecessary to pick apart which physiological adaptations are responsible for the numerous downstream health benefits.

From a messaging perspective, though, we believe it is critical to understand which factor, low fitness or high fatness, is a stronger predictor of negative health outcomes.

As many persons struggling with weight management can attest, being active does not always translate to weight loss.

Alternatively, underactive persons in the “normal” range of body mass index should not be made to believe that “skinny” means “healthy” because of a cultural and clinical emphasis on weight loss.

If fitness is a better predictor of negative health outcomes than is fatness (as has been shown repeatedly [7]), clinicians and public health professionals alike have a responsibility to emphasize that leading an active lifestyle is more important than having a body mass index in the “normal” range.


Dr. Mukamal’s critique of our study focuses on the possibility that responses to 2 items from the Center for Epidemiologic Studies Depression Scale (“inability to ‘get going’” and “the feeling that everything is an effort”) might have been influenced by lack of fitness.

We explicitly acknowledged this possibility in the limitations section of our paper.