Once the power spectrum of the curvature perturbation has been enhanced during inflation from its ∼ 10 −9 value at large scales to ∼ 10 −2 on small scales, and after the fluctuations have been transferred to radiation during the reheating process after inflation, PBHs may form from sizeable fluctuations in the radiation density field if they are able to overcome the resistance of the radiation pressure.


A perturbation of fixed comoving size may not start to collapse till it re-enters the cosmologi- cal horizon.

Therefore, the size of a PBH at formation is related to the horizon length when the corresponding perturbation enters the horizon and collapses.

Fluctuations collapse immediately af- ter horizon re-entry to form PBHs if they are sizeable enough.

Indicating by ζ the gauge-invariant comoving curvature perturbation with power spectrum P ζ , one can define the smoothed version of it ζ R ( x) = d 3 y W (| x − y|, R) ζ( y) (1.1) and its variance σ 2 (R) = d 3 k (2π) 3 W 2 (k, R)P ζ (k), (1.2) where the smoothing out is operated through a window function W (k, R) where one has to select the radius as the comoving horizon length R H = 1/aH. A given region collapses to form a PBH if this variance is larger than a critical value ζ c (0.05 − 1).

The actual value depends on the equation of state w of the fluid when the perturbation re-enters the horizon [29]


For radiation w = 1/3 and one obtains ζ c 0.086.

Larger values are also used in the literature [30-32].

A common value is ζ c 1.3 and we will adopt it as representative from now on.

Notice that one could use also a better criterion [30] based on the density contrast (during the radiation era and on comoving slices) ∆( x) = (4/9a 2 H 2 )∇ 2 ζ( x) for which the critical value is ∆ c 0.45 [33] 1 .

At any rate, our formulae will be valid for both ζ( x) and ∆( x) and their corresponding power spectra on small scales must be rather large if a significant number of PBHs have to form.

The mass of a PBH at formation may be directly related to the number N of e-folds before the end of inflation when the corresponding density fluctuation leaves the Hubble radius M (m 2 P /H)exp(2N ) [35], where m P is the Planck mass.


Under the assumption that the density contrast is a linear quantity obeying a Gaussian statistics, the primordial mass fraction β prim (M ) of the universe occupied by PBHs formed at the time t M is therefore given by P (ζ R H > ζ c ) = β prim (M ) = ζc dζ R H √ 2π σ e −ζ 2 R H /2σ 2 , (1.4) where too large fluctuations have to be disregarded as the collapsing spacetime region behaves as separate closed universe rather than a PBH [2].

The total contribution of PBHs at radiation-matter equality is obtained by integrating the corresponding fraction β(M, t eq ) = a(t eq )/a(t M )β prim (M, t M ) [36] at the time of equivalence Ω PBH (t eq ) = M (teq) Mev(teq) d ln M β(M, t eq ), (1.5) where M ev (t eq ) 10 −21 M is the lower mass which has survived evaporation at equality and M (t eq ) is the horizon mass at equality.

We warn the reader that, being them not the scope of this paper, in this expression various effects are not accounted for, such as the fact that the mass of the PBH is not precisely the mass contained in the corresponding horizon volume, but in reality obeys a scaling relation with initial perturbation [35], that the threshold is shape-dependent [37], or finally that the threshold amplitude and the final black hole mass depend on the initial density profile of the perturbation [38].

Also, we will be dealing here only with the primordial PBH mass fraction.

From the time of equality to now, the PBH mass distribution will be altered by merging and accretion [34, 39], processes which we have nothing to say on.


Now, defining ν(M ) = ζ c σ(M ) , (1.6) the Gaussian mass fraction β prim (M ) for ν ∼ > 5 can be well approximated by


Since PBHs are generated through very large, but rare fluctuations, their mass fraction at formation is extremely sensitive to changes in the tail of the fluctuation distribution and therefore to possible non-Gaussianities in the density contrast fluctuations [19, 40-48].

This implies that non-Gaussianities need to be accounted for within the spiky scenario as they can alter the initial mass fraction of PBHs in a dramatic way.


We are certainly not the first ones to make this point.

For instance, the presence of a scale-invariant local non-Gaussianity, which is nowadays severely constrained by CMB anisotropy measurements [49], can significantly alter the number density of PBHs through mode coupling [50, 51].

In fact here we are rather referring to that non-Gaussianity which is almost inevitably generated at the same small wavelengths where the density perturbations are sizeable and that in the literature is often disregarded, most probably given the associated technical difficulties.


To have the feeling of the level of non-Gaussianity associated to the large perturbations giving rise to the PBHs, let us consider the second-order gauge-invariant comoving curvature perturbation ζ 2 .

During inflation, when the perturbations have already left the Hubble radius, it satisfies the equation [52]ζ 2 = − H (ρ 0 + P 0 ) δP 2 | ρ − 2 (ρ 0 + P 0 ) δP 1 | ρ − 2(ρ 0 + P 0 )ζ 1 ζ 1 , (1.8) where ρ 0 and P 0 are the background energy and pressure densities, δP | ρ is the gauge-invariant non- adiabatic pressure perturbation (on uniform density hypersurfaces) and ζ 1 is the linear gauge-invariant comoving curvature perturbation.


In single-field models the non-adiabatic pressure perturbation van- ishes and the curvature perturbation ζ is conserved at any order in perturbation theory on super-Hubble scales, consistently with Eq. ( 1.8).

Indeed, in single-field models the perturbations are generated at Hubble crossing and Eq. ( 1.8) is not useful to estimate the level of non-Gaussianity associated to the fluctuations responsible for the PBHs and one has to resort to a different way of estimating the level of non-Gaussianity.


However, in the case of spectator fields, the curvature perturbation is not conserved on super-Hubble scales since the non-adiabatic pressure perturbation is not zero.

In such a case, Eq. ( 1.8) signals that the same sizeable linear perturbations which will be responsible for the generation of the PBHs upon horizon re-entry inevitably give rise to sizeable second-order perturbations and therefore to non-Gaussianity, that is ζ 2 = O(1)ζ 2 1 barring cancellations.


This simple argument shows that the level of non-Gaussianity in the large, but rare fluctuations associated to the PBHs nay be not insignificant.


The goal of this paper is to discuss the impact of non-Gaussianity on the initial PBH mass fraction.

In particular,


1. we follow closely Ref.

[ 53] and use a path-integral formulation to provide the probability that the density contrast is larger than a critical value within the horizon at formation time.

The expressions are not original, but maybe known more in the community dealing with halo bias.

They are exact and no approximations are taken;


2. discuss the fine-tuning needed to achieve for the non-Gaussianity to play no role in the deter- mination of the PBH mass function;


3. compute in two popular classes of spiky models the impact of non-Gaussianity, showing that neglecting it leads to blunders in the estimation of the PBH mass fraction;


4. discuss the impact of non-Gaussianity on the clustering of PBHs.


The paper is organised as follows.

In section II we describe the technicalities based on the path- integral approach which allow to write exact expressions for the various quantities of interest in the presence on non-Gaussianity.

In section III we introduce the notion of fine-tuning associated to the non-Gaussianity and in relation to the primordial PBH mass fraction.

Sections IV and V contain two illustrative examples and section VI touches the notion of clustering and non-Gaussianity.

Finally, conclusions are contained in section VII.


This section addresses the calculation of the exact non-Gaussian PBH abundance through a path- integral approach.

It has the virtue of not making any simplifying assumption or approximation, but at a price of being rather technical.

The uninterested reader can jump directly to the final result in Eq. ( 2.28) or its simplified version (2.32).

We follow closely Ref.

[ 53] which we reproduce the results of.

In this sense this section does not contain any new finding, but maybe results are derived in a more transparent way.


Consider the overdensity field ζ( x) with a generic probability distribution P [ζ( x)].

The partition function Z[J] in the presence of an external source J( x) is given by Z[J] = [Dζ( x)]P [ζ( x)]e i d 3 xJ( x)ζ( x) , (2.1) where the measure [Dζ( x)] is such that


The functional Taylor expansion of the partition function Z[J] in powers of the source J( x) specifies the correlators, while the corresponding expansion of W [J] = ln Z[J] generates the connected correlation functions, which we will denote by We have already introduced the smoothed curvature perturbation over a scale R in Eq. ( 1.1).

In a similar manner, we define smoothed connected correlation function ξ (N ) R ( x 1 , · · · , x N ) as ξ (N ) R ( x 1 , · · · , x N ) = ξ (N ) R ( y 1 , · · · , y n ) N i=1 d 3 y i W (| x i − y i |, R), (2.4) so that ξ (2) R ( x, x) = ξ (2) R (0) = σ 2 R , (2.5) is the variance over the scale R. For later use, we also define the normalised correlators


Let us also recall that the functional W [J] is written in terms of the connected correlation functions ξ (N ) as


We now wish to characterise the threshold statistics by requiring that ζ R exceeds a certain threshold ζ c , which we can write as ζ R > νσ R .

Then the peak overdensity ρ ν,R ( x) is defined as ρ ν,R ( x) = Θ(ζ R ( x) − νσ R ), (2.9) where Θ(x) is the Heaviside step-function.

The joint probability that in all points, ( x 1 , · · · , x N ), the overdensity is above the threshold is given by


Since dΘ( x)/dx = δ( x) and 2πδ( x) = ∞ −∞ dφ e iφx , the Θ-function may be expressed as Θ( x) = ∞ −x da ∞ −∞ dφ 2π e iφa , (2.11) and therefore,


where J is defined as J( x) = N i=1 J i with J i ( x i , x) = φ i W (| x − x i |, R), and in the third line in Eq. ( 2.12) we have redefined a i → a i /σ R .

Using Eq. ( 2.8) we may express ln Z[J] as


We may put the last expression in a more explicit form using Eq. ( 2.5)


To calculate the expression (2.18), we have used the formula


where H m−1 are Hermite polynomials and Erfc(x) = √ 2/π ∞ x exp(−t 2 )dt is the complementary error function, we find that Eq. ( 2.21) can be re-expressed as Π (N ) ν,R ( x 1 , · · · , x N ) = ∞ =0 1 !

   { 1 ,··· , N }∈P ,N ! n 1 ! · · · n N !

Ξ { 1 ,··· , N } N r h r (ν)    , (2.23) where Ξ { 1 ,··· , N } denotes the sum of Ξ( x i 1 , · · · , x i )'s at 1 , · · · N points.

For N = 3 and = 4, the possible partitions are (4, 0, 0), (3, 1, 0), (2, 2, 0), (2, 1, 1) and then we have for example,


We are finally ready to collect the results to find the probability for a single density contrast to be above the threshold.

It can be directly calculated by evaluating the one-point statistics


where the argument (0) means that all the correlation functions are computed at the same point, Ξ n (0) = Ξ( x, x, · · · , x), (2.29) and Ξ( x 1 , x 2 , · · · , x n ) is given by Eq. ( 2.20) with ξ (2) R omitted.

For instance, we have


We draw the attention of the reader to the fact that the expression (2.28) is exact and no approxima- tions have been taken along the way.

Nevertheless, it simplifies in the case of large threshold ν 1.

In this limit, using the asymptotic behaviour (ν 1) of the Hermite polynomials and the complementary error function


we find that Eq. ( 2.28) reduces to


where the argument (0) means that all the correlation functions are computed at the same point.

If the smoothing radius is taken to be R H , this expression becomes the mass fraction β prim (M ).

Apart from the limit ν 1, it is exact as it contains all the non-Gaussian correlators for which no assumption needs to be taken.

If one wishes to relax even the assumption of large ν, one can resort to Eq. ( 2.28).


After this technical detour, we are ready to ask under which circumstances non-Gaussianity alters the predictions of the primordial abundance of black holes in a significant way.

To answer this question one of course should look at her/his own preferred model and estimate the amount of non-Gaussianity through the non-vanishing correlators with order larger than two.

However, one can take inspiration by what is routinely done in high energy physics and introduce the notion of fine-tuning, that is how much sensitive is a given observable on the parameters it depends upon.

For instance, in supersymmetric models, where the phenomenon of electroweak symmetry breaking is induced radiatively by the running with energy, one may compute the response of an observable such as the mass squared of the Z-boson m 2 Z to a change of a parameter of the model defined at some high energy [55].


In full similarity, we can ask ourselves how sensitive is β prim (M ) to the various cumulants, defined by the relations


We may therefore define the fine-tuning ∆ n to be the response of the PBH abundance to the intro- duction of the n-th cumulant


Given the particular structure of the non-Gaussian correction to the PBH abundance, one can im- mediately understand the physical significance of this fine-tuning parameter.

Indeed, in the presence of the n-th cumulant, one can express the new non-Gaussian PBH abundance (2.32) in terms of the Gaussian abundance


This implies that the PBH abundance is exponentially sensitive to the non-Gaussianity unless ∆ n is in absolute value smaller than unity


Inspecting Eqs. ( 1.7) and (2.32), we see that


This tells us that non-Gaussianity alters exponentially the Gaussian prediction for the PBH abundance unless


To investigate how restrictive this condition is, we will analyse two representative classes of spiky models, one based on a single-field and the other based on the presence of a spectator field.


In single-field models of inflation the power spectrum of the comoving curvature perturbation is given by [56] P ζ (k) = H 2 8π 2 m 2 P 1 k 3 , (4.1) where = −d ln H/dN is one of the slow-roll parameters and N is the number of e-folds till the end of inflation.

Since the generation of PBHs requires the jumping of the value of the power spectrum of about seven orders of magnitude from its value on CMB scales, without even specifying the single- field model of inflation, one may conclude that there must be a violation of the slow-roll condition |∆ ln /∆N | ∼ > O(1) [32].

In particular, this happens when the inflaton field goes through an inflection point in the scalar potential, see Fig. 1, thus producing a sizeable resonance in the power spectrum of the curvature perturbation.

Correspondingly, slow-roll conditions are broken and the spike in the curvature perturbation may lead to a copious production of PBHs, see for instance Refs. [ 20, 22] for some recent proposals.


When the inflaton experiences a plateau in its potential, since must be extremely small, a short period of Ultra Slow-Roll (USR) is achieved during which the equation of motion of the inflaton background φ 0 reduces to [57-62]φ 0 + 3Hφ 0 0, (4.2) and the piece dV (φ 0 )/dφ 0 is negligible so that there is an approximate shift symmetry in the inflaton field.

The slow-roll parameter η =˙ H = 2φ 0 φ 0 H + 2 −6 (4.3) is therefore sizeable, signalling the breaking of the slow-roll conditions.

During such a phase of USR inflation, = 3 2φ 2 0 V (φ 0 ) ∼ a −6 , (4.4) and the comoving curvature perturbation increases as P ζ ∼ e 6∆N , (4.5) where ∆N parametrises the duration of the USR phase.

Thanks to the approximate shift symmetry as well as the dilation symmetry, one can characterise the bispectrum of the perturbations generated during the USR phase.

The non-Gaussianity turns out to be completely local (δφ is basically a massless scalar field on the de Sitter background in the limit ≪ 1 and non-Gaussianity is generated after Hubble radius crossing) [59, 60, 63]


The corresponding skewness reads


where we have neglected the effect of the window functions in the momentum integrals 2 .

Since the condition (3.6) for non-Gaussianity to play no significant role is S 3 ∼ < 0.06 σ 2 R H 0.02 1.3 ζ c 3 , (4.8) where we have normalised σ 2 R H to the value necessary to get the right abundance of dark matter from PBHs [32] (neglecting accretion), we see that the Gaussian estimate on the PBH mass fraction is hardly trustable.

Of course this result strongly depends on the values of σ 2 R H and ζ c , but it signals a potential danger.


In fact, one can do better at this stage.

We can use the δN formalism [66-68] to evaluate the higher-order cumulants.

In the δN formalism the scalar field fluctuations are quantized on the flat slices and related to the comoving curvature perturbation ζ through the relation ζ = δN .

During the USR phase and in the case of a sharp transition back to the slow-roll phase, one obtains [59] N (φ 0 ,φ 0 ) = 1 3 ln φ 0 φ 0 + 3H(φ 0 − φ e ) , (4.9) where φ e is the value of the field at the end of the USR phase.

Notice in particular that one has to retain the dependence onφ 0 since slow-roll is badly violated.

Since δφ is constant to good accuracy, one can neglect δφ and obtain


where ζ 1 ( x) is the linear curvature perturbation.

If one considers the contribution to the cumulant S n coming from the connected piece of the form


This contribution shifts the Gaussian exponential in the mass fraction in the expression (2.32) from (in this expression one should intend the variance σ 2 R H as the renormalized one once the non-Gaussian contributions are as well summed up [40, 65])


This is not at all an insignificant effect for ν 1.

For instance one can ask what is the impact on the PBH abundance at formation.

We follow Ref.

[ 32] and we set β prim (M ) such that it corresponds today to a totality of the dark matter in the form of PBHs.


Choosing ζ c 1/3, a Gaussian probability would give σ R H ∼ > 0.036, while the non-Gaussian result (4.14) would deliver a result smaller than about ten orders of magnitude.

This shows the large sensitivity of the non-Gaussian result to the cosmological observables.

Of course, this is not the final result as there are other contributions to the cumulants we have not accounted for.


For instance, S 4 receives a connected contribution from the correlator ζ 1 ( x)ζ 1 ( x)ζ 2 1 ( x)ζ 2 1 ( x) .

We have not been able to resum all the pieces.


Despite their incompleteness and the fact that the details have to be worked out model by model, our findings indicate that non-Gaussianity plays a fundamental role in determining the final PBH abundance.


Notice that we have been working under the assumption of a sudden transition between the USR phase and the slow-roll subsequent stage.

It has been recently noticed that the non-Gaussianity is sensitive to the details of the transition from USR phase to the standard slow-roll phase [64].

The skewness takes the generic value S 3 9(4(η V − 3)η V + h 2 + 4η V h)/(2η V + h − 6) 2 , where h = 6 √ V /π e , π = dφ 0 /dN and V and η V = V /3H 2 are evaluated at the slow-roll phase.

For |h| 1 one recovers the sharp transition case, while for a smooth transition or a sharp transition, but small h, one finds the value S 3 η V .

Said in other words, our results show that a smooth transition is preferable in order to have a trustable Gaussian estimate.


The second example we wish to illustrate falls in the category of spiky models where the comoving curvature power spectrum is enhanced at small scales through a spectator field [23].

The idea is that there is a curvaton-like field σ [69] with quadratic potential V (σ) = 1 2 m 2 σ 2 , (5.1) such that the total curvature perturbation during inflation reads (in the flat gauge) ζ = H δρ ρ =ρ φ ρ ζ φ +ρ σ ρ ζ σ , ζ φ,σ = H δρ φ,σ ρ φ,σ , (5.2) where ζ σ is the curvaton perturbation and ζ φ is responsible for the anisotropies in the CMB anisotropies on large scales.

The two contributions are assumed to be comparable at a scale k c Mpc −1 .

The curvaton is in fact an axion-like field.

It is related to the phase θ of a complex field whose modulus is the inflaton field φ 0 with potential V (φ 0 ) = 1 2 λH 2 φ 2 0 , (5.3) with 1 < λ ≤ 9/4.

The inflaton rolls relatively fast towards its minimum φ * which is reached when the wavenumber k * is exiting the Hubble radius and the curvaton field becomes well-defined, σ( x) = φ * θ( x).

For scales k < k * , when the inflaton is still rolling down its potential, the fluctuations of the curvaton inherit a strong scale-dependence, δθ k (H/2πφ 0 (t k )), where t k is the time at which the wavelength 1/k exits the Hubble radius.

The corresponding spectral index is given by n σ − 1 3 − 3 1 − 4 9 λ, (5.4) and one can obtain a blue curvaton power spectrum, n σ O(2 − 4) [23].


After inflation, the curvaton oscillates around the minimum of its potential and it decays into radiation.

If we denote by r the ratio of the curvaton energy density ρ σ over the radiation energy density ρ γ , at small scales the linear curvature perturbation is dominated by the curvaton and becomes


The perturbation is nevertheless getting a non-Gaussian contribution.

Restricting ourselves to the quadratic potential (5.1), the curvature perturbation receives second-order contributions from the (δσ( x)) 2 piece in δρ σ .

A standard computation leads to [70-72]


Neglecting for the moment the window functions, the corresponding skewness is S 3 6 r 3 2 − r 2 , (5.7) which is a decreasing function of r with S 3 = 3 at r = 1.

Imposing again S 3 ∼ < 0.06 σ 2 R H /0.02 (1.3/ζ c ) 3 , one finds solutions only for sufficiently small thresholds or large variances.

If we consider higher-order terms in the curvaton axion-like potential, higher-order cumulants are generated, but they will depend on other parameters, such as the quartic coupling ∼ m 2 /φ 2 * and the background value σ 0 .

We do not expect cancellations to operate and make the Gaussian mass fraction of PBHs trustable.


To estimate the impact of non-Gaussianity, we can use alternatively the density contrast (during the radiation phase) ∆( x) = (4/9a 2 H 2 )∇ 2 ζ( x) for which ∆ c 2 √ 2ζ c /9 0.4(ζ c /1.3).

The corresponding skewness reads


The numerical results are shown in Fig. 2 which makes explicit that the Gaussian evaluation of the PBH abundance does not suffice.

One obvious improvement will be to evaluate the one-loop variance induced by the non-Gaussianity δσ 2 ∆ (R H ) = 2 4 9 2 1 r 2 3 2 − r 2 2 d 3 k (2π) 3 (kR H ) 4 W 2 (k, R H ) d 3 p (2π) 3 P ζ (p)P ζ (| k − p|), (5.9) where the power spectra are of course the tree-level ones.

This improvement will however require an improved computation of the skewness and so on.

We again follow Ref. [ 32] and ask what is the impact of non-Gaussianity if we demand that the primordial PBH abundance corresponds to the totality of the dark matter abundance today.


Let us minimise the influence of non-Gaussianity by setting r = 1 and S 3 3.

Under these circumstances, δσ 2 ∆ (R H ) is still much smaller than the tree-level σ 2 ∆ (R H ), and the Gaussian approach would give σ 2 ∆ (R H ) ∼ > 0.02 [32].

This is in sharp contrast with our condition (3.6) on S 3 which would deliver S 3 ∼ < 0.06.

A value of S 3 would deliver a dark matter abundance smaller by almost twenty orders of magnitude.

This again indicates the sensitivity of the cosmological predictions to the non-Gaussianity.


The PBHs created in the radiation era and originated from spikes in the comoving curvature perturba- tion are highly clustered [34].

Using the peak theory model of bias, one can compute the PBH n-point correlation functions.

For Gaussian fluctuations one can follow Ref. [ 73] and show that the connected two-point correlator of PBHs is given by


One can easily see that the correlator ξ (2) ν,R H ( x 1 , x 2 ) increments for higher values of ν.

In other words, since PBHs are generated by large fluctuations and their mass fraction is exponentially suppressed for ν 1, the fewer there are of them, the more clustered they are.

The technique explained in Ref.

[ 53] allows also to compute the correlators of the density contrasts above a given threshold for a non-Gaussian statistics.

Let us for instance consider the case of W ( x, R) = δ (3) ( x).

It follows immediately that in the large threshold limit the joint probability is Π (N ) ν,R ( x 1 , · · · , x N ) = e −N ν 2 (2πν 2 ) N/2 N i=1 Z[J i ], (6.2) which, by using Eq. ( 2.32) can be written as


Therefore, the N -point peak disconnected correlation functions ξ (N ) disc,ν,R ( x 1 , · · · x N ) defined as ξ (N ) disc,ν,R ( x 1 , · · · , x N ) = N i=1 ρ ν,R ( x i ) ρ ν,R − 1, (6.4) turn out to be


For instance, the non-Gaussian two and three-point peak connected correlation functions read respec- tively


As already stressed in Ref. [ 53] (see also Ref. [ 54]) for the validity of these equations, correlations do not need to be small either with respect to unity or with respect to the variance on the smoothing scale.

The two-point correlator can be rewritten as


The power spectrum in momentum space and over the normalization volume V will read


where the lower cut-off of the integral is due to the finite size of the PBHs.

One can expand in powers the exponentials and obtain


For instance, for the class of single-field model generating PBHs at inflection points described in section IV one finds that the Fourier transform of ν 3 /σ 3 R H ξ (3) R H ( x 1 , x 2 , x 2 ) is given by


that is a factor ∼ 6ζ c times the Gaussian contribution.

One sees again that the non-Gaussian correc- tions can alter the clustering properties significantly.

Since clustering influences the merging of PBHs, this could have a severe impact on the subsequent cosmological evolution of PBHs.

For instance by changing the limits on initial PBH abundance or, as merging enhances the abundance of more mas- sive and long-lived PBHs, by making PBHs the seeds of supermassive BHs [74, 75].

Our preliminary analysis above suggests that non-Gaussianity may not be disregarded in these considerations.

We will defer the study of this subject in a forthcoming publication.


The idea that PBHs may be the dark matter in our universe is a fascinating one.

Equally exciting is the fact that PBHs may owe their origin to large and rare fluctuations generated during inflation, i.e. during the phase of accelerated expansion which might be ultimately responsible also for the large-scale structures we see around us.

If this is the mechanism nature has chosen to permeate the universe with dark matter, it is vital to understand in details the quantitative predictions within a given set-up.

In this paper we have discussed the role of non-Gaussianity in determining the PBH mass fraction at formation time and their clustering properties.

We have done so by discussing a path-integral procedure which leads to exact expressions for the quantities of interest and by estimating the impact of non-Gaussianity in a couple of models.

Our findings show that non-Gaussianity should be taken into account when considering a model for PBH production and the corresponding non-Gaussianity parameters should be evaluated in order to gauge if PBHs are interesting candidates for dark matter and when analysing their clustering properties and subsequent merging and accretion.

Also, it will be interesting to investigate the impact of the non-Gaussianity on the production of the gravity waves from inflation generated by the same large perturbations originating the PBHs [8] and on the limits coming from CMB µ-distortions [76].


We consider mechanisms for producing a significant population of primordial black holes (PBHs) within string inspired single field models of inflation.

The production of PBHs requires a large amplification in the power spectrum of curvature perturbations between scales associated with CMB and PBH formation.

In principle, this can be achieved by temporarily breaking the slow-roll conditions during inflation.

In this work, we identify two string setups that can realize this process.

In string axion models of inflation, sub-leading non-perturbative effects can superimpose steep cliffs and gentle plateaus onto the leading axion potential.

The cliffs can momentarily violate the slow-roll conditions, and the plateaus can lead to phases of ultra slow-roll inflation.

We thus achieve a string motivated model which both matches the Planck observations at CMB scales and produces a population of light PBHs, which can account for an order one fraction of dark matter.

In DBI models of inflation, a sharp increase in the speed of sound sourced by a steep downward step in the warp factor can drive the amplification.

In this scenario, discovery of PBHs could indicate non-trivial dynamics in the bulk, such as flux-antibrane annihilation at the tip of a warped throat.


In the early 1970s, Stephen Hawking proposed that overdense inhomogeneities in the very early Universe could gravitationally collapse to form Primordial Black Holes (PBHs) [1, 2].

Assuming the Planck length as the minimal possible Schwarzchild radius, gives a lower limit for black holes masses of about 10−5g. Moreover, this mechanism – in contrast to the astrophysical process from dying stars – allows for populations of black holes spanning a vast range of masses, from the Planck scale to supermassive and beyond.

Constraints on PBHs abundances continue to improve, but leave viable windows, especially when astrophysical uncertainties are taken into account (see [3, 4] for recent reviews).

PBHs could provide a significant fraction – or indeed all – of the mysterious Dark Matter that dominates cosmic structures in the present day Universe.

Stimulated both by the absence of signatures for well-motivated particle candidates for Dark Matter and by the first detection of gravity waves from merging black holes, this idea has recently taken new flight.

The principal modern process for PBH production is the collapse of curvature perturbations generated during early Universe cosmic inflation, when the relevant scales re-enter the horizon during the radiation era [5].

However, the big challenge here is that the amplitude of these inflationary curvature perturbations at the large scales probed by the CMB has been constrained by observations to be tiny, As ∼ 10−9 at the pivot scale k = 0.05 Mpc−1.

In order for inflationary perturbations to lead to a significant PBH population there must be some mechanism at work that enhances the power spectrum to around As ∼ 10−2 at some smaller scale.

For example, a large enhancement at scales around k ∼ 1012 − 1014 Mpc−1 would lead to PBHs masses in the window 10−17M(cid:12) (cid:46) MP BH (cid:46) 10−13M(cid:12), which could be a dominant component of Dark Matter consistently with current observational constraints1.


Many recent works have explored how such an enhancement could be achieved.

As current CMB observations are consistent with the simplest, single field slow-roll models, much of the focus has been to explore scenarios producing PBHs in single field inflation [7–11].

A possibility is that the inflaton rolls down a slow-roll plateau, probed by the CMB observations, followed by a near-inflection point which leads to a phase of ultra slow-roll.

Roughly, the extreme flattening of the inflationary potential around the inflection point leads to an enhanced amplitude of scalar perturbations.

Whilst such a potential can certainly be manufactured within effective field theory, ultimately its shape – and robustness against quantum corrections – would have to be explained within a fundamental theory, like string theory.

For example, an attempt towards embedding this model within string theory via fine-tuned string-loop corrections in fibre inflation has been made very recently in [12], although it has not yet proved possible to fine-tune a setup such that cosmological perturbations both match the CMB and produce significant PBH Dark Matter.


The purpose of this paper is to identify well-motivated string mechanisms that can lead to an amplification of curvature perturbations and thus significant PBH production in the early Universe. To do so, it is helpful to recall the physical conditions underlying this amplification, first discussed by Leach et al. [ 13, 14]2.

In slow-roll inflation, the equation that governs the curvature perturbation takes the form of a damped harmonic oscillator.

Solutions for super-horizon modes correspond to a linear combination of a constant and a decaying mode.

However, if the slow-roll conditions are broken, then the friction term in the damped harmonic oscillator equation can become a driving term, so that the decaying mode becomes a growing mode, and enhances the curvature perturbation on scales beyond the horizon.

This enhancement continues until the slow-roll conditions are restored.

( A similar mechanism has been used to build single field models with large squeezed non-Gaussianity [16].)


The first possible scenario that we propose is based upon an earlier observation on how subleading, non-perturbative effects can affect axion inflation [17] 3.

In string axion inflation, the perturbative axion shift symmetry is broken spontaneously by background vevs (e.g. fluxes), such as in axion monodromy [23, 24] and/or non-perturbative effects (e.g. string instantons), leading to large field inflation models with monomial or cosine (“natural inflation” [25]) potentials.

In [17], we noted that subleading non-perturbative effects – if sufficiently large – can superimpose periodically steep cliffs and gentle plateaus onto the underlying potential.

The overall effect shown there was to restore axion inflation into the favour of current CMB observations, by allowing sufficient efolds of inflation to be obtained with smaller field ranges and thus lowering the tensor-to-scalar ratio, even achieving natural inflation with sub-Planckian axion decay constants.

Here we note that such an inflationary potential can also quite naturally give rise to an enhancement in the scalar power spectrum at small scales, as the inflaton meets successively shallower plateaus on its roll towards the global minimum.

Depending on the parameters, the cliffs may temporarily halt inflation, and the plateaus may include (near-)inflection points around which ultra slow-roll inflation occurs.

By connecting to the heuristic arguments above, and analysing numerically the cosmological perturbations, we show that such models can both lie within 2σ constraints from Planck results, and produce a population of light PBH (about 10−16 − 10−15M(cid:12)) that provides a signification fraction of Dark Matter.


The second string motivated mechanism for the amplification of power during inflation that we propose is based on single field models of inflation with non-canonical kinetic terms, as for example DBI inflation [26, 27].

To understand what features are required within these models, we extend the Leach et al. argument on amplification of super-horizon modes to include varying speed of sound.

We find that a driving term emerges in the mode equation if the speed of sound increases sufficiently fast.

A simple way to achieve this is if the warp factor experienced by the D-brane sourcing inflation features a sufficiently large4 and steep step during the inflationary trajectory.

A large step in the warp factor could occur for instance, if the D-brane travels down a throat within a throat, sourced by two separated stacks of D-branes [31, 32].

Another scenario could be that during the D-brane’s journey down the throat, an instability towards brane-antibrane or flux-antibrane [33] annihilation occurs at the tip of the throat.

In this scenario, observations of a population of PBHs would reveal interesting properties of branes exploring the geometry of the string compactification.


The paper is organised as follows.

In the following section, we provide a brief review of the basic physical mechanism which leads to the enhancement of the amplitude of curvature perturbation during inflation, subsequently leading to the production of PBHs.

In Section 3, we review and develop models of string axion inflation including subleading non-perturbative effects, describing their background evolution and predictions at the CMB scales.

We then investigate numerically the amplification of scalar power spectrum at small scales.

In particular, we provide two explicit models which both concord with CMB observations and produce PBHs sufficient to explain a significant fraction of Dark Matter.

In Section 4, we turn our attention to DBI models of inflation with a large step feature in the warp factor.

We study the background evolution and the equation governing the curvature perturbations, and thus provide heuristic arguments for the amplification of curvature perturbations, postponing a detailed numerical analysis of the perturbations to future work.

Finally, in Section 5 we present our conclusions and point towards several open questions to which our work leads.


We will use natural units, (cid:126) = c = 1, with reduced Planck mass M 2 pl = (8πG)−1.

Our metric signature is mostly plus (−, +, +, +).

The background metric is a FRW universe with line element ds2 = −dt2 + a2(t) d(cid:126)x2 = a2(τ ) (−dτ 2 + d(cid:126)x2).

The overdots and primes on time dependent quantities denote derivatives with respect to coordinate time t and conformal time τ , respectively.

During inflation, we take a(τ ) = 1/(−Hτ )1+(cid:15) with H is the physical Hubble rate.


Primordial black holes as dark matter may be generated in single-field models of inflation thanks to the enhancement at small scales of the comoving curvature perturbation.

This mechanism requires leaving the slow-roll phase to enter a non-attractor phase during which the inflaton travels across a plateau and its velocity drops down exponentially.

We argue that quantum diffusion has a significant impact on the primordial black hole mass fraction making the classical standard prediction not trustable.


The interest in the physics of Primordial Black Holes (PBHs) and the possibility that they form all (or a fraction) of the dark matter in the universe has risen up [1–10] again after the discovery of two 30M☉ black holes through the gravitational waves generated during their merging [11].

A standard mechanism to account for the generation of the PBHs is through the boost of the curvature perturbation at small scales [12–14].

Such an enhancement can occur either within single-field models of inflation, see Refs [15–19] for some recent literature, or through some spectator field [20–23] which could be identified the Higgs of the Standard Model [24].

In order for this primordial mechanism to occur, one needs an enhancement of the power spectrum of the curvature perturbation from its 10−9 value at large scales to 10−2 on small scales.

Subsequently, these large perturbations are communicated to radiation during the reheating process after inflation and they may give rise to PBHs upon horizon re-entry if they are sizeable enough.

If we indicate by R the comoving curvature power spectrum, a region of size the Hubble radius may collapse and form a PBH if the corresponding square root of the variance σR smoothed with a high-pass filter on the same length scale (comoving momenta larger than the inverse of the comoving Hubble) is larger than some critical value Rc.

Its exact value is sensitive to the equation of state upon horizon re-entry and it is about 0.086 for radiation [25].

However, larger values have been adopted in the literature [26–28].

We will later on use the common representative value Rc ≃ 1.3.

Under the (strong) hypothesis that the curvature perturbation obeys a Gaussian statistics, the primordial mass fraction βprim(M) of the universe occupied by PBHs formed at the time of formation reads βprim(M) = ∫∞Rc dR √2πσR e−R2/2σ2R. (1.1) It corresponds to a present dark matter abundance made of PBH of masses M given by (neglecting accretion) [28] ΩDM(M)h2 0.12 ≃ βprim(M) 10−9 (γ 0.2 )1/2 (106.75 g∗ )1/4 (M☉ M )1/2 , (1.2) where γ < 1 is a parameter accounting for the efficiency of the collapse and g∗ is the effective number of degrees of freedom.

Imposing PBHs to be the dark matter, values γ < 1 require larger values of βprim and therefore to be conservative we impose γ = 1 [28] βprim(M) 10−9 > 3 (M M☉ )1/2 . ( 1.3) For a mass of the order of 10−15 M☉ we find [28] σR > 0.16.


Now, in single-field models of inflation the power spectrum of the comoving curvature perturbation is given by (we set the Planckian mass equal to one from now on) [29] PR1/2(k) = H 2πφ′ , φ′ = dφ dN , (1.5) where N is the number of e-folds, the prime denotes differentiation with respect to N, and H is the Hubble rate.

The generation of PBHs requires the jumping within a few e-folds ΔN of the value of the power spectrum of about seven orders of magnitude from its value on CMB scales.

Without even specifying the single-field model of inflation, one may conclude that there must be a violation of the slow-roll condition as φ′ must change rapidly with time.

This may happen when the inflaton field goes through a so-called non-attractor phase (dubbed also ultra-slow-roll) [30–39] in the scalar potential, thus producing a sizeable resonance in the power spectrum of the curvature perturbation.

When the inflaton experiences a plateau in its potential, since φ′ must be extremely small, a short non-attractor period is achieved during which the equation of motion of the inflaton background φ reduces to φ′′ + 3φ′ + V,φ H2 ≃ 0, where ,φ denotes differentiation with respect to the inflaton field φ with potential V(φ).

The comoving curvature perturbation increases, due to its decaying mode which in fact is growing, as φ′ ∼ e−3N and PR1/2 ∼ e3N.

(1.7) It is this exponential growth which helps obtaining large fluctuations in the curvature perturbation and the formation of PBHs upon horizon re-entry during the radiation phase.

This is also the reason why the power spectrum should be quoted at the end of inflation and not, as usually done in slow-roll, at Hubble crossing: its value at the instant of Hubble crossing differs by a significant factor from the asymptotic value at late times.

Not respecting these rules might lead to an incorrect estimate of the power spectrum and consequently of the PBH abundance at formation, see e.g. Ref.

[ 40] and the subsequent discussions in Refs. [ 28, 36].


Putting aside the strong sensitivity of the PBH mass fraction at formation to possible non-Gaussianities [14, 41–50] which is common to all mechanisms giving rise to PBHs through sizeable perturbations (we will however devote Appendix C for some considerations about non-Gaussianity where we will show that the δN formalism [51] can help in assessing the role of non-Gaussianity for those perturbations generated during the non-attractor phase) in this paper we are interested in another issue, the role of quantum diffusion.

One might be reasonably suspicious that during the non-attractor phase the quantum diffusion becomes relevant [49].

The reason is the following.

The stochastic equation of motion for the classical inflaton field takes into account that each Hubble time the inflaton field receives kicks of the order of ±(H/2π) [52] where ξ is a Gaussian random noise with ⟨ξ(N)ξ(N′)⟩ = 9H2 4π2 δ(N − N′).

( 1.9) During the non-attractor phase, V,φ needs to be tiny enough to allow φ′ to promptly decrease thus violating slow-roll.

For the same reason, one needs to make sure that quantum jumps are not significant in this case.

One could try to impose the condition 2πV,φ 3H3 ≳ 1.

( 1.10) to be satisfied during the non-attractor phase.

In slow-roll, the condition (1.10) could be exactly expressed in terms of the power spectrum and the latter would be required to be smaller than unity, thereby giving a direct constraint on a physical observable.

However, during the non-attractor phase, the bound (1.10) is not directly expressed in terms of the power spectrum (1.5), making the comparison with physical observables more difficult.

One might naively think that during the non-attractor phase the noise is not relevant if φ′ is larger than (H/2π) [17].

However, this is not correct for two reasons.

First because, as we will see, the relevant noise to be evaluated is the one for the inflaton velocity.

Secondly, and above all, because this is not the right criterion to evaluate the strong impact of quantum diffusion onto the PBH mass fraction.

We will first elaborate on the computation of the power spectrum during the non-attractor phase in order to understand some basic features of the power spectrum itself, e.g. its time evolution and the location of its peak.

This will be useful for the considerations about the quantum diffusion.

We will then use the so-called Kramers-Moyal equation [53] to assess the impact of quantum diffusion.

The Kramers-Moyal equation is the suitable starting point as it highlights the importance of the inflaton velocity and it is a generalisation of the Fokker-Planck equation.

Indeed, in general the Kramers-Moyal equation contains an infinite number of derivatives with respect to the inflaton field after having integrated out the velocity, while the Fokker-Planck equation is a truncation of the Kramers-Moyal equation by retaining only two spatial derivatives of the inflaton field.

This is not an irrelevant point: Pawula’s theorem [54] tells us that if we set to zero some coefficient cn of the higher derivative terms with respect to the inflaton field with n ≥ 3, then all the coefficients of the higher derivatives are zero.

It is therefore not consistent to keep some higher derivative term unless all of them are kept.

This means that, whenever the Kramers-Moyal equation may not be solved exactly, a numerical approach is useful to solve it without applying an unjustified truncation.

We will present both analytical and numerical results to quantify the impact of quantum diffusion on the PBH abundance.


In the simplest case of non-attractor phase with an approximately linear or quadratic potential, the system can be solved analytically.

For a linear potential, the stochastic motion of the system is characterised by the fact that the variance of the velocity φ′ rapidly converges towards a stationary value.

For a quadratic potential, the spread in the velocity varies with time, although slowly, away from the stationary point.

However, for more complicated situations, e.g. if during the non-attractor phase the inflaton goes through an inflection point, a numerical analysis is called for and we will show that the spread in the velocity grows with time.

If this growth is too large, classicality as well as information on the PBH abundance is lost.

We will propose two criteria to be respected in order to neglect the quantum diffusion and we will see that the curvature perturbation is severely constrained from above in order to avoid an undesirable spreading of the velocity wave packet.

We will also argue that the capability of the standard (that is classical) calculation to predict the correct dark matter abundance in terms of PBH is severely challenged by the presence of quantum diffusion.


This paper is organised as follows.

In section II we discuss the computation of the curvature perturbation during the non-attractor phase.

We then start our study of the quantum diffusion, both analytically and numerically, in section III and IV respectively.

In sections V and VI we offer two criteria to assess the importance of the diffusion.

Finally, in section VII we offer our conclusions.

The paper contains several Appendices.

Appendix A deals with the curvature perturbation, the Schwarzian derivative and the dual transformation; appendix B with the study of the evolution of the comoving curvature perturbation from the non-attractor phase back to the slow-roll phase; appendix C offers some consideration about non-Gaussianity.


The mean supervision scale score was calculated for each combination of the 65 raters and nine questions.

The Cronbach's alpha was very large (0.977).

The mean score was calculated for each of the 3,421 observed combinations of resident and anesthesiologist.

The logits of the percentage of scores equal to the maximum value of 4.00 were normally distributed (residents, P = 0.24; anesthesiologists, P = 0.50).

There were 20/97 anesthesiologists identified as significant outliers (13 with below average supervision scores and seven with better than average) using the mixed-effects logistic regression with rater leniency entered as a fixed effect but not by Student's t test.

In contrast, there were three of 97 anesthesiologists identified as outliers (all three above average) using Student's t tests but not by logistic regression with leniency.

The 20 vs 3 was significant (P \ 0.001).


Use of logistic regression with leniency results in greater detection of anesthesiologists with significantly better (or worse) clinical supervision scores than use of Student's t tests (i.e., without adjustment for rater leniency).


Contexte Notre département contrô le la qualité de supervision clinique des anesthésiologistes et donne des rétroactions périodiques à chaque anesthésiologiste.

Nous avons émis l'hypothèse qu'une plus grande différentiation dans les scores de supervision des anesthésiologistes serait obtenue en tenant compte de l'indulgence du résident évaluateur.


Méthode Dès le 1 er juillet 2013 et jusqu'au 31 décembre 2015, notre département s'est servi d'une échelle de supervision unidimensionnelle à neuf points, soit l'échelle de Oliveira Filho, afin d'évaluer la qualité de la supervision clinique offerte par les membres du département et telle que notée par les résidents.

Nous avons passé en revue les 13 664 évaluations des 97 anesthésiologistes (les évalués) par les 65 résidents (les évaluateurs).

Nous avons évalué la cohérence interne parmi les réponses aux questions (coefficient alpha de Cronbach étendu [ 0,90) afin d'écarter la possibilité qu'une ou deux questions expliqueraient l'indulgence.

Une régression logistique à effets mixtes a été utilisée pour comparer les évalués tout en contrô lant l'indulgence des évaluateurs vs l'utilisation de tests t de Student sans indulgence de l'évaluateur.


Le score moyen sur l'échelle de supervision a été calculé pour chaque combinaison des 65 évaluateurs et des neuf questions.

Le coefficient alpha de Cronbach était très étendu (0,977).

La note moyenne a été calculée pour chacune des 3421 combinaisons observées de résident et anesthésiologiste.

La distribution des logits du pourcentage de notes égales à la valeur maximale de 4,00 était normale (résidents, P = 0,24; anesthésiologistes, P = 0,50).

Au total, 20/97 anesthésiologistes ont été identifiés comme valeurs aberrantes (13 ayant des notes de supervision au-dessous de la moyenne et sept des notes au- dessus) à l'aide de la régression logistique à effets mixtes avec l'indulgence de l'évaluateur en tant qu'effet fixe, mais pas par le test t de Student.

Par contre, trois des 97 anesthésiologistes ont été identifiés comme valeurs aberrantes (les trois au-dessus de la moyenne) à l'aide des tests t de Student, mais pas par régression logistique avec l'indulgence.

Les 20 vs 3 étaient significatifs (P \ 0,001).


L'utilisation de la régression logistique avec l'indulgence permet une plus grande détection des anesthésiologistes présentant des notes significativement meilleures (ou moins bonnes) de supervision clinique que l'utilisation de tests t de Student (c.-à-d.

sans ajustement pour tenir compte de l'indulgence de l'évaluateur).


Anesthesiologists' clinical competency will be assessed from completion of residency through to retirement.

There are several limitations with using simulation for these assessments.

Mannequin-based and/or multidisciplinary simulations can require expensive travel to a test centre and time away from clinical practice for both ratees and raters.

Simulations often focus on management of a few crises of brief duration.


In contrast, in situ assessments quantify anesthesiologists' clinical performance in the dynamic and unpredictable environment where they personally deliver care.

This environment includes a range of large and unexpected problems, where anesthesiologists' roles include foreseeing and preventing problems and where social, team, and environmental factors influence anesthesiologists' effectiveness.

1,2 Thus, as part of an overall assessment of clinical competency, our department uses in situ assessments of individual anesthesiologists working in operating rooms and other procedural locations (henceforth referred to as ''ORs'') to determine how well they provide clinical supervision of anesthesia residents (Table 1).

3-14 Higher scores for clinical supervision are associated with fewer resident reports of errors with adverse effects on patients (Table 2.15) 11-13 and greater preference for the anesthesiologist to care for the rating resident's family (Table 2.7).

7 Supervision, in this context, refers to all clinical oversight functions directed toward assuring the quality of clinical care whenever the anesthesiologist is not the sole anesthesia care provider.

3-5 The de Oliveira Filho unidimensional nine-item supervision instrument is a reliable scale used to assess the Table 1 de Oliveira Filho et al. 's instrument 6 for measuring faculty anesthesiologists' supervision of residents during clinical operating room care


The faculty provided me timely, informal, nonthreatening comments on my performance and showed me ways to improve


2. The faculty was promptly available to help me solve problems with patients and procedures


The faculty used real clinical scenarios to stimulate my clinical reasoning, critical thinking, and theoretical learning


The faculty demonstrated theoretical knowledge, proficiency at procedures, ethical behaviour, and interest/compassion/respect for patients


The faculty was present during the critical moments of the anesthetic procedure (e.g., anesthesia induction, critical events, complications)


The faculty discussed with me the perianesthesia management of patients prior to starting an anesthetic procedure and accepted my suggestions, when appropriate


The faculty taught and demanded the implementation of safety measures during the perioperative period (e.g., anesthesia machine checkout, universal precautions, prevention of medication errors, etc.)


The faculty treated me respectfully and strived to create and maintain a pleasant environment during my clinical activities


All questions were presented daily in the same sequence.

The wording differs from that developed, 6 only to the extent that a) the word ''faculty'' was used instead of ''instructor'' to be more closely aligned with the role of supervision of the nurse anesthetists, and b) the verb tense was changed to past tense because each evaluation was for a specific date working together 644 F. Dexter et al. 123 quality of supervision provided by each anesthesiologist (Table 1).


The scale measures all attributes of anesthesiologists' supervision of anesthesia residents (Table 2.1) 6,7,10-13 and has been shown, in multiple studies, to do this as a unidimensional construct (Table 2.2).


Low supervision scores are associated with written comments about the anesthesiologist being disrespectful, unprofessional, and/or teaching poorly that day (Table 2.18).


Scores increase when anesthesiologists receive individual feedback regarding the quality of their supervision (Table 2.17).

4 Scores are monitored daily and each anesthesiologist is provided with periodic feedback.


The supervision scale's maximum value is 4.00, which corresponds to a response of 4 (i.e., ''always'') to each of the nine questions (Table 1).

6 Because of the ceiling effect, multiple scores of 4.00 reduce the scale's reliability 7,9,10,15 to differentiate performance among the anesthesiologists, even though such differentiation is mandatory (see Discussion).


We previously asked residents to provide a single evaluation for the overall quality of supervision they received from the department's faculty (i.e., as if intended as an evaluation of the residency program) (Table 2.14).

13 We compared those overall scores pairwise with the mean of each resident's evaluations of all individual anesthesiologists with whom they worked during the preceding eight months.

13 Both sets of scores showed considerable heterogeneity among the residents (e.g., some residents provided overall lower scores than those of other residents).

13 Consequently, our hypothesis was that greater differentiation among anesthesiologists' supervision scores could be obtained by incorporating scoring leniency by the resident (rater) into the statistical analysis (i.e., treating a high score as less meaningful when given by a resident who consistently provides high scores, in other words, lenient relative to other raters).

A


The University of Iowa Institutional Review Board affirmed (June 8, 2016) that this investigation did not meet the regulatory definition of research in human subjects.

Analyses were performed with de-identified data.


From July 1, 2013 to December 31, 2015, our department utilized the de Oliveira Filho supervision scale to assess the quality of clinical supervision by staff anesthesiologists (Table 1).

6,7 The cohort reported herein includes all rater evaluations of all staff anesthesiologists (ratees) over that 2.5-year period chosen for convenience.

We used five six-month periods because we previously showed that six months was a sufficient duration in our department for nearly all ratees to receive evaluations and for an adequate number of unique raters to differentiate reliably among ratees using the supervision scale.


The evaluation process consisted of daily, automated e- mail requests 16 to raters to evaluate the supervision provided by each ratee with whom they worked the previous day in an OR setting for at least one hour, including obstetrics and/or non-operating room anesthesia (e.g., radiation therapy).

4,8-10 Raters evaluated ratees' supervision by logging in to a secure webpage.

8 The raters could not submit their rating until each of the nine questions was answered with their choice of 1-4: 1 = never; 2 = rarely; 3 = frequently; or 4 = always (Table 1).

The ''score'' for each evaluation was equal to the mean of the responses to the nine questions (Table 1).

The scores remained confidential and were provided to the ratees periodically (every six months) only after averaging among multiple raters.


If one or two of the nine questions resulted in leniency among raters, a potential intervention would have been to either modify the question(s) or provide an example of behaviour that should affect the answer to the question(s) (see Discussion).

In contrast, if leniency were present throughout all questions, then an analysis of leniency would need to incorporate the average scores of the raters.

The question whether leniency was present in a few vs all questions was addressed by analyzing the mean ratings for each combination of the 65 raters and nine questions.

These means had sample sizes of at least 37 answers and mean sample sizes of 210 answers (i.e., sufficient to make the ranks of 1, 2, 3, or 4 into interval levels of measurement).

Cronbach's alpha, a test for internal consistency among answers to questions, B was calculated using the resulting 65 9 9 matrix, equally weighting each rater.

The confidence interval (CI) for Cronbach's alpha was calculated using the asymptotic method.

18 A Leniency is the scientific term.

We searched Google Scholar on December 8, 2016.

There were 962 results from ''rater leniency'' OR ''raters' leniency'' OR ''rating leniency'' OR ''leniency of the rater'' OR ''leniency of the raters''.

There were 93.4% fewer results for ''rater heterogeneity'' OR ''raters' heterogeneity'' OR ''heterogeneity of the rater'' OR ''heterogeneity of the raters''.


B See http://FDshort.com/CronbachSplitHalf, accessed February 2017.

For each respondent, select four of the nine questions, calculate the mean score, and calculate the mean score of the other five questions.

Calculate among all raters the correlation coefficient between the pairwise split-half mean scores.

Repeat the process using all possible split halves of the nine questions.

The mean of the cor- relation coefficients is Cronbach's alpha.

This measure of internal consistency provides quantification for the reliability of the use of the score alone.


The statistical distribution of the rater leniency scores was analyzed.

For each observed rater and ratee pair, we calculated the mean score specific to that pair.

8,15,19,C For each rater, we obtained the equally weighted average of the mean scores provided by that rater for all ratees (Fig. 1).

8,C We previously showed that the number of scores per pair differs markedly among raters for each ratee (i.e., there is non-random assignment of residents and anesthesiologists such that leniency will not average out; P \ 0.001).


The same approach was used when calculating the average of the means by ratee.

In order to assess whether the scores of individual ratees were unusually low or high, we compared the averages of the means of each ratee with the value of 3.80 using Student's t test and Wilcoxon signed- rank test.

The value of 3.80 is the overall mean supervision score among all ratees' scores (see Results).

The P values using the Wilcoxon signed-rank test were exact, calculated using StatXact Ò 11 (Cytel Inc., Cambridge, MA, USA).

Student's t test does not adjust for rater leniency.


Stata Ò 14.1 was used (StataCorp LP, College Station, TX, USA) to perform mixed-effects analyses treating the rater as a categorical fixed effect and the ratee as a random effect.

Results of the mixed-effects analyses allowed us to assess the ratees' quality of supervision.

Mixed-effects analyses were carried out for two separate dependent variables, modelled individually: 1) the average score, and 2) the binary variable whether the score equalled the maximum of 4.00 (Fig. 2).

The logistic regression was performed using the ''melogit'' command option of mean- variance adaptive Gauss-Hermite quadrature with 30 quadrature points.

As described later (see Results), several analyses were repeated using other estimation methods, including cluster-robust variance estimation, the unstructured covariance matrix, or estimation by Laplace approximation.

All tests for differences between ratees were performed treating two-sided P \ 0.01 as statistically significant.

The imbalance in the number of resident ratings per ratee is considered in Appendix 1.


Individual questions did not contribute significantly to leniency (i.e., consideration of individual questions could not improve the statistical modelling).

Cronbach's alpha as to the raters' answers to questions was high in value (0.977; 95% CI, 0.968 to 0.985); therefore, the score for each rating could be used (i.e., the mean of the answers to the nine questions in the supervision scale) (Table 1).


We used the 13,664 scores, D with 3,421 observed combinations of the 65 raters and 97 ratees.

In Appendix 2, C The sample sizes are too small to estimate the variance within pairs, and the variances are generally unequal among pairs.

7,8 See the Anesthesia & Analgesia companion papers for mathematical details.

7,8 Even when there are many ratings per rater, using each rating's score minimally influences final assessments clinically.

19 D Residents provided a response for 99.1% (n = 14,585) of the 14,722 requests.

10 For 6.3% (n = 921) of requests, residents responded that they worked with the faculty for insufficient time to evaluate supervision, leaving n = 13,664 ratings.

10 The mean (SD) intraoperative patient care time together was 4.87 (2.53) h day -1 . 10 646 F. Dexter et al. 123 we show lack of validity of the statistical assumptions for a random effects model in the original score scale.

20,21 We treated the rater as a fixed effect to incorporate rater leniency in a mixed-effects logistic regression model.


Fig. 2 shows the distribution among raters of the percentage of scores equal to the maximum value of 4.00 (i.e., all nine questions answered ''always'').

The 65 raters differed significantly amongst each other in terms of the percentages of their scores equal to 4.00 (P \ 0.001 using fixed-effect logistic regression).


The mixed-effects model with rater as a fixed effect and ratee as a random effect relies on the assumption that the distribution of the logits among ratees follows a normal distribution, which it does.

Specifically, no ratee had all scores equal to 4.00 (i.e., for which the logit would have been undefined because of division by zero).

In addition, no ratee had all scores less than 4.00 (i.e., for which the logit would also be undefined).

There were 60 ratees each with C 14 scores lower than 4.00 among their C 32 scores (i.e., sample sizes large enough to obtain reliable estimates of the logits).

8,22 The logits followed a normal distribution [Lilliefors test, P = 0.50; mean (standard deviation [SD]), -0.781(0.491)].


Figures 3 and 4 show each ratee's average score, meaning the average of means, equally weighting each rater (see above ''Statistical distributions of rater and ratee scores'').

4,8,10,15 We use symbols to indicate whether the ratee's average score is significantly (P \ 0.01) different from the average score among all ratees when using a Student's t test (i.e., without adjustment for rater leniency).

We also indicate whether the ratee's percentage of scores \4.00 differed from other ratees when using mixed-effects logistic regression, with rater leniency treated as a fixed effect and ratee as a random effect.

We subsequently refer to that mixed-effects model as ''logistic regression with leniency''.


The principal result is that 20/97 ratees were identified as outliers using the logistic regression with leniency, but not by Student's t tests.

There were 3/97 ratees identified as outliers using the Student's t tests, but not by logistic regression with leniency.

The 20 vs 3 is significant; exact P \ 0.001 using McNemar's test.

Thus, adjusting for rater leniency increased the ability to distinguish the quality of anesthesiologists' clinical supervision.


In Appendix 3, we confirm the corollary that there is less information from scores \ 4.00 vs the percentage of scores equal to the maximum score of 4.00.


In Appendix 4, we show that our previous observation of an increase in supervision score over time with evaluation and feedback (Table 2.17) 4 holds when analyzed using logistic regression with leniency.


In Appendix 5, we show that our previous analyses and publications without consideration of rater leniency were reasonable because initially there was greater heterogeneity of scores among ratees.


In this final section, we examine why incorporating rater leniency increased the sensitivity to detect both below average and above average performance differences among ratees.

Readers who are less interested in ''why'' may want to go directly to the Discussion.


The figures are divided between descriptions of the supervision scores of ratees (i.e., anesthesiologists) with Fig. 5, but no such data points are plotted.

The scale of the vertical axis in Fig. 3 differs from that in Fig. 4 Leniency of raters and anesthesiologist supervision 647 123 scores less than (Figs 3, 5) and greater than (Figs 4, 6) the overall average score of 3.80.

No ratee happened to have an average score equal to 3.80.

Thus, the division reflected different concerns, i.e., identification of ratees potentially performing below average vs those potentially performing above average.


Statistical significance of the logistic regression with leniency depended on the number of scores\4.00, shown on the vertical axes of Figs 3-6 (see Appendix 1 and Appendix 6).

For a given ratee average score, blue circles showing lack of statistical significance are more often present for smaller sample sizes than red triangles and orange squares.


Among the 30 ratees with average scores \ 3.80, 13 were not significantly different from the average of 3.80 using the Student's t test, but were significantly different from the other ratees by logistic regression with leniency (Fig. 3).

For illustration, we consider the ratee with an average score of 3.56, shown by the left-most orange square.

This score was the smallest value not found to be significantly less than the overall average of 3.80 using the Student's t test, but found to differ significantly from the other ratees by logistic regression with leniency.

In Appendix 7, we show that this finding was caused by substantial variability among raters (i.e., residents) regarding how much the ratee's quality of supervision was less than the maximum score (4.00).


Among the 53 ratees who had average supervision scores [ 3.80 and who had at least nine different raters, seven were not significantly different from average as determined by the Student's t test, but were significantly different using logistic regression with leniency (Fig. 4).

There were 3/53 ratees who were significantly different from average by the Student's t test, but not significantly different using logistic regression with leniency.

For illustration, we consider the ratee with the highest average score.

In Appendix 8, we show that logistic regression with, or without, leniency (Fig. 6) lacked statistical power to differentiate this ratee from other anesthesiologists because the ratee had above average quality of supervision and relatively few clinical days (i.e., ratings).


The supervision scores are the cumulative result of how the anesthesiologists perform in clinical environments.

The scores reflect in situ performance and can improve with feedback.

4,15 Supervision scores are used in our department for mandatory annual collegiate evaluations and for at P \ 0.01.

There are three ratees plotted using black Xs.

These were significant by the Student's t test (P B 0.0018) but not by logistic regression with leniency (P C 0.32).

The scale of the vertical axis in Fig. 4 differs from that in Fig. 3 Previous psychometric studies of anesthesiologists' assessments of resident performance have also found significant rater leniency.

28,29 Even with an adjustment of the average scores for rater leniency, the number of different ratings that faculty needed for a reliable assessment of resident performance exceeded the total number of faculty in many departments.

28 Our paper provides a methodological framework for future statistical analyses of leniency for such applications.


Suppose the anesthesiologists were distributed into nine categories.

There are those with a less than average, average, and greater than average annual number of clinical days, thereby receiving a less than average, average, and greater than average number of evaluations of their clinical performance.

There are anesthesiologists who provide less than average, average, and greater than average quality of supervision.

We think that, among these nine (3 9 3) groups, the least institutional cost for misclassifying the quality of clinical supervision (below average, average, above average) would be to consider the group of anesthesiologists providing less than average clinical workload and greater than average quality of supervision as providing average quality of supervision.

Because this was the only group that was ''misclassified'' through use of logistic regression with leniency, we think it is reasonable managerially to use this method to analyze the supervision data.


We showed that leniency in the supervision scale (Table 1) was caused by the cumulative effect of all questions (i.e., leniency was not the disproportionate effect of a few questions).

If an individual question had accounted for variability in leniency among raters, providing examples of behaviour corresponding to an answer could have been an alternative intervention to reduce leniency. Because our department provides OR care for a large diversity of procedures, it is not obvious to us how to provide examples because there are so many different interactions between residents and anesthesiologists that could contribute to less than or greater than average quality of supervision.

1,10 Nevertheless, the finding that leniency arises because of the cumulative effect of all questions shows that the issue is moot.

Variability in rater leniency is the result of the raters' overall (omnibus) assessments of anesthesiologists' performance, without distinction among the nine items describing specific attributes of supervision.


The supervision score is a surrogate for whether a resident would choose the anesthesiologist to care for their family (Table 2.7).

7 Supervision scores for specific rotations are associated with perceived teamwork during the rotation (Table 2.8).

12 Observation of intraoperative briefings has found that sometimes anesthesiologists barely participate (e.g., being occupied with other activities).

30 Team members can ''rate the value'' of the intraoperative briefing performed ''in the OR when the patient is awake''. 31 Thus, we have hypothesized that leniency may Leniency of raters and anesthesiologist supervision 649 123 be related to interaction among organizational safety culture, residents' perceptions of the importance of the intraoperative briefing to patient outcome, and the anesthesiologists' participation (or lack) in the briefings.

Our finding of large internal rater consistency among the nine questions shows that such a hypothesis cannot be supported.

Supervision begins when residents and anesthesiologists are assigned cases together, ends after the day's patient care is completed, and includes inseparable attributes (Table 1).

Future studies could evaluate whether rater leniency is personality based and/ or applies to rating other domains such as quality of life.


Our findings are limited by raters being nested within departments (i.e., residents in one department rarely work with anesthesiologists in other departments).

Consequently, for external reporting, we recommend that evaluation of each ratee (anesthesiologist, subspecialty, 12 or department 11 be performed using the equally weighed average of the scores from each rater.

Results are reported as average scores of equally weighted raters, along with confidence intervals.

8,C In contrast, for assessment and progressive quality improvement within a department, we recommend the use of mixed-effects logistic regression with rater leniency.

Results are reported as odds ratios, along with confidence intervals.

Regardless, in situ assessment of the quality of supervision depends (Figs 4 and 6) on there being at least nine (and preferably more) unique raters for each ratee (Table 2.11).

7 Although this generally holds for operating room anesthesia, it can be a limitation for specialties (e.g., chronic pain) in which residents rotate for weeks at a time and work with one or two attending physicians.


None declared.


This submission was handled by Dr. Hilary P. Grocott, Editor-in-Chief, Canadian Journal of Anesthesia.


Franklin Dexter and Bradley J. Hindman helped design the study.

Franklin Dexter helped conduct the study.

Franklin Dexter and Johannes Ledolter helped analyze the data.

Franklin Dexter, Johannes Ledolter, and Bradley J. Hindman helped write the manuscript.


Appendix 1 Imbalance in numbers of raters among ratees Figures 3-6 control graphically for heterogeneity among ratees in average scores (i.e., horizontal axes) and in sample sizes (i.e., vertical axes), but not for imbalance in the relative proportions of different raters.

Among the 3,411 combinations of rater and ratee, the most common number of ratings was one occasion together (26.2%); whereas the median was three occasions and the 90th percentile was nine occasions.

The imbalance results in substantial residual scatter in the figures, even though the imbalance is taken into account in our inferential analysis.

When all residents evaluated all anesthesiologists during one weekend in a balanced design, there was a statistically significant but quantitatively negligible effect of resident on supervision scores (Table 2.6).

7 Addition of resident class (i.e., ''CA-1,'' ''CA-2,'' or ''CA-3'') as another variable in the logistic regression with leniency failed because the sample sizes were too small to estimate the fixed effects when stratified by six-month period.

A better choice for the horizontal axis would be the marginal estimate of the average score (i.e., adjusting for the relative proportions of different raters among ratees by treating both rater and ratee as fixed effects and then re-estimating the mean using the mean proportions).

Nevertheless, there is substantial collinearity, and estimates are not calculated by Stata Ò .


From among the 13,664 scores, D there were 3,421 observed combinations of the 65 raters (i.e., residents) and 97 ratees (i.e., anesthesiologists).

The mean score was calculated for each of the 3,421 combinations.

The average of these means across ratees was then calculated for each of the 65 raters (Fig. 1).


The distribution among the raters of their n = 65 averages was skewed (P \ 0.0001; skewness [G1] -1.82, standard error 0.30).

20 The 65 averages had sample mean (SD) of 3.80 (0.22).

The corresponding normal distribution had poor fit to the data: Lilliefors test, P \ 0.0001.

After each of six power transformations 21 (X -2.0 , X -1.0 , X -0.5 , ln (X), X 0.5 , and X 2.0 ), still P\0.0001, showing the n = 65 data by rater not following normal distribution.

Thus, treating rater leniency as a random effect would not work reliably because we would need falsely to assume a normal distribution.

These calculations were performed using SYSTAT 13.1 (SYSTAT Software Inc., San Jose, CA, USA).


Next, we considered the probability distribution for the ratee effect.

Mixed-effects modelling of scores with rater treated as a fixed effect and ratee as a random effect makes normality assumptions about the random effect, which needs to be checked.

The average of the means for each combination was calculated for each of the 97 ratees.

The normal distribution had sample mean (SD) of 3.79 (0.17), but poor fit: Lilliefors test, P \ 0.0001.

Normal 650 F. Dexter et al. 123 distributions were not suitable models for ratee averages, even after considering each of the six power transformations (all P \ 0.0001).


Appendix 3 Relative information from scores < 4.00 vs percentage of scores = 4.00


Data presented in the Results suggest less information from scores \ 4.00 vs the percentage of scores equal to the maximum scores of 4.00.

As described in the Section ''Effectiveness of logistic regression with leniency relative to Student's t test'', there were significantly more (20/97) ratees identified as outliers using the logistic regression with leniency but not by Student's t test (P = 0.0005 by McNemar's test).

Also, in the Section ''Influence of leniency on evaluation of ratees with below average supervision'', we provide an example wherein the Student's t test failed to detect a significantly less than the overall average score because of large variability among raters regarding how much the ratees' quality of supervision was less than maximum (4.00).


The most common value of the scores was 4.00 (Table 2.5).

9 The incidence of 4.00 among the 13,664 scores was 68.1%, significantly greater than 50% (P \ 0.00001 by binomial test, StatXact 11).

Consequently, suppose that information from scores \ 4.00 were providing information highly correlated with and supplementing the percentage of scores equal to 4.00.

Then, the next most common score would be the next smaller possible score, 3.89; the 3.89 = (8 questions 9 score of 4 ? 1 question 9 score of 3) / (9 questions).

This was not so.

The next most common score was 3.00, with 8.90% of scores, significantly greater than the 4.30% of scores accounted for by the value of 3.89 (P \ 0.001 by Fisher's exact test).


The reason for this finding is probably that each of the four possible answers for each question represents an associated frequency: 1 = never; 2 = rarely; 3 = frequently; or 4 = always.

Less than ''frequently'' has been associated with ''mistakes that had negative consequences for the patient'' (Table 2.15), 11 ''medication errors (dose or incorrect drug) in the last year'' (Table 2.15), 11 and disrespectful behaviour of the ratee (Table 2.18; P \ 0.0001).

10 An anonymous reviewer recognized an alternative potential explanation for our findings.


Suppose that ratee A has 60% of scores = 4.00, but among scores \ 4.00, the mean equals 2.00.

Suppose that ratee B has 50% of scores = 4.00, but among scores \ 4.00, the mean equals 3.80.

The logistic model would then suggest that ratee A provides greater quality of supervision than ratee B, while the Student t test would suggest the opposite.

The example highlights that the percentage of scores equal to the maximum value of 4.00 could be negatively correlated with the mean of all the ratee's scores \4.00.

This was not so.


Among the 89 ratees with at least nine scores \ 4.00, the Kendall's s b = ? 0.36 (two-sided P\ 0.0001) (StatXact 11).

Furthermore, in the fifth six-month period (i.e., after considerable duration of feedback), among the 58 ratees with at least nine scores, Kendall's s b = ? 0.024 (P = 0.79 relative to 0.00 and P = 0.0003 relative to 0.36).

Thus, as supervision scores increased progressively, there was significantly less information provided by the scores \ 4.00.


We reported previously that, with evaluation and feedback, there was an increase in anesthesiologists' supervision scores from the first to the third six-month period (Table 2.17).

4 We re-evaluated that finding 4 using mixed- effects logistic regression, with rater leniency entered as a fixed effect.

The six-month period, as continuous variable, was entered as a fixed effect.

Each increase of one period was associated with an odds ratio of 1.34 (95% confidence interval [CI], 1.25 to 1.44) for a score equal to 4.00.

Repeating the analysis using robust variance estimation, 95% CI was 1.20 to 1.51, and repeating the analysis using an unstructured covariance matrix, 95% CI was 1.25 to 1.44.

Each of these three methods of analyses had P \ 0.0001.

Therefore, our previous observation of an increase in supervision scores over time 4 held even when rater leniency was included in the analysis.


Although our department had evaluated anesthesiologists' supervision for 2.5 years (five sequential six-month periods), we had not considered rater leniency in our analysis.


The sample estimate of the standard deviation (SD) among ratees in their average of mean scores was calculated for each of the five six-month periods.

During the first six-month period, 57 anesthesiologists received scores from at least nine raters (i.e., residents).

The SD among the raters' averages was 0.154 (n = 57).

Among the subsequent four six-month periods, the SDs were 0.150 (n = 56), 0.120 (n = 50), 0.095 (n = 55), and 0.097 (n = 56), respectively.


Bartlett's test and Levene's test were used to test for inequality in the SDs among these periods as a check on the Leniency of raters and anesthesiologist supervision 651 123 reliability of the results.

That calculation was performed using SYSTAT 13.1.


The lesser variance among ratees was a significant change (Bartlett's test, P = 0.0002 and Levene's test, P = 0.0002), prompting our consideration of statistical methods better able to distinguish among anesthesiologists.


Even using two figures (Figs 3 and 4), there was considerable data point overlap.

In our previous generalizability analysis, when all raters (i.e., residents) evaluated all ratees (i.e., anesthesiologists) on the same weekend in a balanced design, a minimum of nine different raters was needed for reliability (Table 2.2).

7 Among the 97 ratees, there were 14 ratees who received scores from less than nine different raters.

None of these 14 ratees differed significantly from the other ratees when we used mixed-effects logistic regression with rater leniency treated as a fixed effect (all P C 0.041).

Therefore, to reduce clutter in Figs 3 and 4 (and in the subsequent Fig 5 and 6), the results for these 14 ratees are not shown graphically.


Appendix 7 Illustrative example of the ratee shown by the left-most orange square in Fig. 3 We consider the ratee with an average score of 3.56, shown by the left-most orange square in Fig. 3.

This score was the smallest value not found to be significantly less than the overall average of 3.80 by Student's t test but for which the ratee was found to differ significantly from the other ratees by logistic regression with leniency.

To understand the result in the logit scale, we consider the simpler process of logistic regression without consideration of leniency; these findings are shown in Fig. 5.

This ratee had 32.4% of n = 37 scores equal to 4.00, considerably less than the overall percentage of 68.1%.

This is a significant difference with P \ 0.0001, by logistic regression both with and without leniency.

Thus, the relevant question is why this ratee's average supervision score of 3.56 was not significantly different from 3.80 by Student's t test.

This ratee had a sample standard deviation of 0.53.

The number of raters was 20.


Among the 12 of 20 raters providing averages \ 3.80, the median was 3.43, with averages ranging from 2.39 to 3.78.

Among the eight raters providing averages [ 3.80, the median was 4.00, with averages ranging from 3.89 to 4.00.

The distribution of averages below and above 3.80 was not significant by Wilcoxon signed-rank test (P = 0.17).

Thus, the lack of significance of the Student's t test for this ratee was not due to limitations of the assumption of a normal distribution.

Rather, there was substantial variability among raters in how much the quality of supervision provided by the ratee was less than maximum (4.00).


For this illustration, logistic regression models both with and without leniency were equally effective for differentiating the ratee's quality of supervision from other ratees.

This was convenient for understanding the results.

Nevertheless, Fig. 5 shows that the example represented an unusual finding.


Appendix 8 Illustrative example of the ratee with the largest average score (Fig. 4)


We consider the ratee with the largest average score.

The average of the means was equal to 3.97, the standard deviation was 0.068, and n = 16 raters providing 21 individual scores, giving P\0.0001 by Student's t test.

For this ratee, among the averages, 13 were equal to 4.00, and the other three were equal to 3.89, 3.85, and 3.78.

This distribution of averages relative to 3.80 was significant by Wilcoxon signed-rank test (P \ 0.0001).

Thus, identifying this ratee as having greater than the overall quality of supervision was not artifactual, caused by assumptions of the statistical distribution of averages.


Thus, the relevant question is why this ratee's logistic regression with leniency was not significant (P = 0.32).

We rely on the results of logistic regression without leniency (Fig. 6) because that model was easier to interpret and was also not significant (P = 0.57).

There were 16/21 (76.2%) individual scores equal to 4.00.

That percentage (i.e., the 76.2%) differed little from the overall percentage of 68.1%.

To have an 80% power to detect such a difference of a single percentage at a B 0.01 would require a sample size of 327 scores (i.e., greater than tenfold larger).

Thus, use of logistic regression with leniency lacked statistical power to differentiate this ratee with relatively few ratings (i.e., few clinical days) but above average quality of supervision compared with other anesthesiologists.


The same thing applied to logistic regression without leniency.

The observed percentage of scores equal to 4.00 was 76.2%, an 8.1% difference from the overall 68.1%.

Among the 41 of 97 ratees deemed significant (P \ 0.01) using logistic regression with leniency, 11 had absolute differences less than 8.1%.

One of the 11 was also significant by logistic regression without leniency (P = 0.0025), with 61.8% of the scores equal to 4.00.

This was possible because they had 122 scores less than 4.00 (i.e., a much larger sample size).

This ratee is viewable in Fig. 5 at an average score of 3.68.

Otherwise, incorporating leniency mattered.

For example, a ratee had 28 evaluations less than 4.00 and an average score of 3.72 (Fig. 5).

The ratee had 67.1% of scores equal to 4.00, an absolute difference of just 652 F. Dexter et al. 123 1.0%.

P = 0.0012 incorporating leniency vs P = 0.64 without.

For example, a ratee had 20 evaluations less than 4.00 and an average score of 3.85 (Fig. 6).

The ratee had 73.0% of scores equal to 4.00, an absolute difference of 4.9%.

P = 0.0034 incorporating leniency vs P = 0.54 without.

For example, a ratee had 20 evaluations less than 4.00 and an average score of 3.91 (Fig. 6).

P = 0.0004 incorporating leniency and P = 0.46 without.


1. Supervision is a single-dimensional construct that incorporates several different attributes, including participation in perianesthesia planning, availability for help/consultation, presence during critical phases of the anesthetic, and fostering safety measures 6,7,10,12 2.

Supervision can be quantified reliably using an instrument with nine questions, each question assessing a different attribute of supervision.

6 The nine questions take \ 90 sec to complete.

The Cronbach's alpha achieved in routine use was equal to 0.948 ± 0.001 (SE) 10 3.

Raters evaluate how often each attribute is demonstrated by the anesthesiologist (never = 1; rarely = 2; frequently = 3; and always = 4), and the supervision score is the mean of the nine answers.

6 When each anesthesiologist's mean resident and mean nurse anesthetist scores were paired, the means were correlated (P \ 0.0001).

8,9 Thus, the behaviour and attributes used to assess the quality of an anesthesiologist's supervision have significant commonality between residents and nurse anesthetists 8,9 4.

There were very small differences in anesthesiologist supervision scores provided by residents when 1) a resident had more units of work that day with the rated anesthesiologist; (''units together'', s b = 0.083 ± 0.014) or 2) the rated staff anesthesiologist had more units of work that same day with other providers, (''units not together''; s b = -0.057 ± 0.014).

8 Anesthesiologists' mean supervision scores provided by residents and nurse anesthetists were not correlated with anesthesiologists' semi-annual clinical activity (multiple all P[0.65).

4 A very active clinician can provide ineffective supervision, and a less active clinician can be very effective.

4 Supervision served as an independent contributor to the value that an individual anesthesiologist added to the care of the patient 4 5.

The most common supervision score provided by nurse anesthetists was 4.0 (P \ 0.0001), indicating that all of the questions were considered important, including those related to teaching.

9 Anesthesiologist supervision scores provided by residents are even greater (P \ 0.0001).

8,9 The pairwise differences by anesthesiologist are also significantly greater than zero (P \ 0.0001) 8,9 6.

All residents evaluated all anesthesiologists' supervision during a study performed during a single weekend, such that each resident was in one class (e.g., ''CA-1'', ''CA-2'', etc8.

When the supervision instrument was applied to departments 12 (Tables 2.14 and 2.15), the internal consistency (Cronbach's alpha) of the scale was 0.909 ± 0.007.

Convergent validity was based on a positive correlation between supervision and variables related to safety culture (all P \ 0.0001): ''Overall perceptions of patient safety'', ''Teamwork within units'', ''Non-punitive response to errors'', ''Handoffs and transitions'', ''Feedback and communication about error'', ''Communication openness'', and the rotation's ''overall grade on patient safety''.

12 Convergent validity was based on significant negative correlation with variables related to the rater's burnout (all P \ 0.0001): ''I feel burnout from my work'', ''I have become more callous toward people since I took this job'', and ''errors with potential negative consequences to patients (that you have) made and/or witnessed''.

12 Among these variables, supervision was most closely predicted by the same one variable using multiple types of regression trees: ''Teamwork within (the rotation)'' (e.g., ''When one area in this rotation gets busy, others help out'').

12 Discriminant validity was based on absence of rank correlation of supervision score with characteristics of raters and programs (all P [ 0.10): age, hours worked per week, sex, promptness of survey response, number of survey raters from the department, and rotation (specialty) (as random effect) 12 9.

There was no significant association between anesthesiologist supervision score and the number of occasions that a resident rater had worked with the anesthesiologist, based on billing data (by patients, s b = ? 0.01; P = 0.71 and by days, s b = -0.01; P = 0.46) 7 10.

Among anesthesia residents, ''the mean ± standard deviation of staff supervision scores that meets expectations'', neither ''exceeds expectations'' nor is ''below expectations'' was 3.40 ± 0.30. 5 ''Most … residents (94%) perceived that supervision that met their expectations was at least frequent (i.e., a score C 3.0)'' (P \ 0.0001). 5 These values were greater than for nurse anesthetists (P\ 0.0001) 5 11.

Anesthesia departments can measure individual anesthesiologists' supervision with high reliability (i.e., mean score is known with precision) when supervision scores are provided by at least nine different resident raters per anesthesiologist.

7 Monitoring is done by taking each of the raters' mean supervision scores for the anesthesiologist and weighting them equally (i.e., treating each rater's mean as a single observation) 8,15 12.

With residents' evaluations of anesthesiologists, mean supervision scores differed among anesthesiologists based on generalizability analysis (P \ 0.0001) 7 13.

Anesthesiologist performance can be monitored daily using Bernoulli cumulative sum (CUSUM) control charts.

15 A reasonable threshold for low scores is \ 3.0 for residents.

15 The true positive detection of anesthesiologists with incidences of low scores greater than the chosen ''out-of-control'' rate was 14/14.

15 The false- positive detection rate was 0/29. 15 Bernoulli CUSUM detection of low scores was within 50 ± 14 (median ± quartile deviation) days 15 Leniency of raters and anesthesiologist supervision


To the Editor,


I read with interest the paper entitled ‘‘Measurement of faculty anesthesiologists’ quality of clinical supervision has greater reliability when controlling for the lenience of the rating anesthesia resident: a retrospective cohort study’’ that recently appeared in the Journal [1].

I would like to point out what I believe to be a critical logical flaw in its conclusions.


The authors compare a new model for determining the quality of faculty supervision with an old model, concluding that it results in ‘‘greater detection of anesthesiologists with significantly better (or worse) clinical supervision scores.’’ The problem is that the two models are asking fundamentally different questions—the original compares averages of scores, whereas the new one compares the probability of obtaining a perfect score—and are therefore not comparable.

Furthermore, as I illustrate with a simple example, the new model’s definition of quality is oversimplified and could lead to perverse conclusions.

The new model is simply not a good substitute for the old one.


Faculty supervision is measured by the Oliveira Filho supervision scale [1], which asks trainees to fill out a nine-item questionnaire rating a single experience of working with the supervisor in an operating room.

Each item is ranked on a scale of 1–4, with 4 reflecting the best performance.

The overall score is the average of the nine items.

Thus, an individual item’s score can be as high as 4—if the supervisor scores perfectly on all items—or as low as 1.

The original model compared supervisors on the basis of their average score, reasoning that a higher average score implies better supervision.

The second model compares supervisors on their probability of obtaining a perfect score, reasoning that supervisors who obtain perfect scores more often must be better.

To see the problem with this method, consider the following two hypothetical supervisors, John and Mary.


Mary prides herself on her supervision and receives a perfect score 75% of the time.

The remaining 25% of the time she receives a rating of 4 on eight of the nine questions and but has only a 3 on the remaining one, giving her an overall score of 3.89 [(8*4 + 3)/9].

Overall, her average score is 3.97 [(0.75*4) + (0.25*3.89)].

John is a truly excellent supervisor when sober but, tragically, suffers from a crippling substance abuse problem. When he is doing well (80% of the time), John gets uniformly perfect scores of 4.

Sadly, on the other 20% of days, John receives a score of only 1.

Overall, John’s average score is now 3.4 [0.8*4 + 0.2*1].


According to the original model, Mary’s average score of 3.97 appropriately puts her well above John with his average of 3.4.

In the model that Dexter et al. report, which should allow more precise detection of better clinical supervision, John’s 80% probability of getting a perfect score ranks him as a better supervisor than Mary with her probability of only 75%.


Another way to explain the deficiency of the second model is as follows.

The first model uses all the information from a carefully validated performance measurement scale consisting of nine questions, each of which has four possible levels.

With 9^4 = 36 possible values, this measurement scale could represent a full continuum of performance from excellent to very poor.

The second model effectively replaces this sensitive, nuanced scale by a single question: ‘‘Was your staff supervisor’s performance perfect?’’ This quite literally equates Mary’s ‘‘almost perfect’’ (3.97) with John’s ‘‘terrible’’ (1) score and leads to John being judged a better supervisor than Mary.


Conflicts of interest None declared.


We would like to address Dr. Ramsay’s comment on our recent report showing that mixed effects logistic regression can be used for biannual evaluation of anesthesiologists’ supervision while adjusting for the leniency of the raters [1].

Dr. Ramsay’s letter questions the validity of our analyses based on a hypothetical scenario of an anesthesiologist with substance abuse achieving consistently high scores punctuated by very low scores [2].