engine/hazard.py

"""Discrete-time cloglog hazard PD model with competing-risk prepayment.

METHODOLOGY (knowledge/sources/ifrs9_credit_risk_notes.md section 6.2)
----------------------------------------------------------------------
Grouped-duration cloglog. On a loan-quarter panel, the discrete hazard
lambda(t | x_it) = P(T_i = t | T_i >= t, x_it) is fitted as a binary GLM on
at-risk rows with the complementary log-log link:

    cloglog lambda = ln(-ln(1 - lambda)) = alpha(age) + x_it' beta + gamma' m_t

The cloglog link is the EXACT grouped-duration analogue of the continuous-time
Cox proportional-hazards model: if the underlying continuous-time hazard is
proportional (h = h0 * exp(x'beta)) and we only observe interval (quarterly)
survival, the implied discrete-time model is a binomial GLM with cloglog link
and the same beta. exp(beta) is therefore a genuine hazard ratio.

alpha(age) is the baseline age (seasoning) effect, entered as a natural cubic
spline of loan_age -- patsy cr(loan_age, df=5). df=5 gives the curve enough
flexibility to reproduce the retail "seasoning hump" (rise over the first ~2-3
years on book, then decay) without chasing noise; natural cubic splines are
LINEAR beyond the boundary knots, which is the disciplined choice for the
lifetime-PD extrapolation the term structure needs (notes section 6.2:
extrapolation is a flagged model-risk area -- the tail assumption here is
"log-hazard linear in age beyond the observed-age support", plus frozen
covariates; see pd_term_structure).

COMPETING RISKS: cause-specific hazards. Prepayment (status 2) and default
(status 1) compete for the same loan. Each cause-specific hazard is estimated
by treating the OTHER event's occurrence as censoring -- the loan's rows simply
end at the competing event; the competing-event row itself carries y = 0 for
this cause. No special weighting, subdistribution adjustment, or IPCW is
applied: with cause-specific hazards estimated on at-risk rows, the likelihood
factorises across causes and two separate binomial GLMs are the maximum-
likelihood estimator (this is what the notes mean by "estimate cause-specific
hazards"). The two hazards are recombined in pd_term_structure, where survival
requires escaping BOTH exits:  S(t) = prod_{k<=t} (1 - lambda_def_k - lambda_pre_k).

CENSORING / TRUNCATION (first-class, per the build plan):
* Right censoring (loans alive at the end of the window) is handled by
  construction -- censored loans contribute y = 0 at-risk rows and then stop;
  the binomial likelihood on at-risk rows is exactly the discrete-time
  survival likelihood under non-informative censoring.
* Left truncation (orig_time < first observation; loans enter the window
  already seasoned) is handled the same way: a loan entering at age a simply
  contributes at-risk rows from age a onward, which is the correct
  conditional-on-survival-to-entry likelihood for discrete-time hazards.
* Rows after a loan's first terminal event were removed in the panel build
  (data/panel/build_panel.py waterfall).

UNIT / TARGETS: unit = loan-quarter; y = default_event for the default hazard,
y = payoff_event for the prepayment hazard.

COVARIATES (fixed specification; scalings chosen for hazard-ratio readability):
  cr(loan_age, df=5)      baseline seasoning curve
  fico_s                  FICO_orig_time / 100  (HR per 100 FICO points)
  ltv10                   updated_ltv / 10      (HR per 10pp of current LTV;
                          updated_ltv = balance_time-amortised, HPI-indexed
                          collateral LTV built in the panel; uses the CURRENT
                          quarter's HPI -- accepted collateral indexation, see
                          TIMING CONVENTION below)
  prepay_incentive        interest_rate_time - rate_time (pp) -- the note rate
                          vs the CURRENT market rate; the star prepayment
                          driver (accepted state-variable exception, see
                          TIMING CONVENTION below)
  investor_orig_time, REtype_{CO,PU,SF}_orig_time   occupancy / property type
                          flags (baseline: owner-occupied "other" RE type; the
                          three RE flags are mutually exclusive in the data)
  uer_lag1, uer_chg4_lag1 unemployment level and its 4-quarter change, lagged
                          one quarter inside the panel (no lookahead)
  hpi_growth_lag1         lagged one-quarter log HPI growth
  gdp_lag1                lagged GDP growth
  center(ltv10):center(uer_lag1)
                          the DOUBLE-TRIGGER interaction: mortgage default
                          requires both negative equity (ability-to-sell lost)
                          and a cash-flow shock (unemployment); the interaction
                          lets the LTV slope steepen when labour markets crack.
                          The components are CENTERED (patsy stateful center(),
                          training means memorised for prediction). Centering
                          is an exact reparametrisation -- the interaction
                          coefficient itself is IDENTICAL to the uncentered
                          x*y coefficient -- but (a) it removes the near-
                          collinearity between x*y and its main effects that
                          destabilised IRLS on this data, and (b) it makes the
                          main-effect coefficients directly interpretable as
                          marginal effects AT THE TRAINING MEAN of the partner
                          variable (ltv10 = LTV slope at mean unemployment).

TIMING CONVENTION (no contemporaneous MACRO REGRESSORS; two indexed
STATE VARIABLES are deliberate exceptions):
  * All macro regressors -- uer_lag1, uer_chg4_lag1, hpi_growth_lag1,
    gdp_lag1 -- enter LAGGED (publication delay + transmission; built inside
    the panel with groupby(id).shift semantics, gap-safe, only t-k (k>=1)
    values referenced; see data/panel/build_panel.py). No contemporaneous
    macro aggregate appears anywhere in the specification.
  * updated_ltv indexes the collateral value by the CURRENT quarter's HPI.
    This is deliberate and acceptable: it is collateral indexation -- a
    loan-level STATE variable (the live collateral cover that defines the
    negative-equity trigger), not a macro regressor. The macro cycle as a
    systematic driver enters only through the lagged regressors above.
  * prepay_incentive compares the note rate to the CURRENT quarter's market
    rate (rate_time). Same class of exception: the incentive is the
    moneyness of the borrower's prepayment option, a loan-level state
    variable, and market rates -- unlike the UER/GDP/HPI aggregates -- are
    observable in real time with no publication lag, so the current-quarter
    rate is in the information set when the quarter's prepayment decision is
    made. Lagging it would misprice the option against the standard
    prepayment literature. (Residual caveat, accepted: rate_time is a
    within-quarter average, so it partially overlaps the event window.)

Rows in the lag warm-up window (lag_warmup == 1, i.e. time < 6) are dropped
at fit time because the 4-quarter unemployment change is undefined there.

DOCUMENTED DEVIATIONS / NUMERICAL SAFEGUARDS
  * LTV winsorisation: updated_ltv is capped at LTV_CAP = 300 (%) in _prepare,
    at fit AND predict time. 18 of 418,418 training rows exceed the cap (up to
    777% -- residual tiny-balance / data-noise artefacts). Justification: the
    cloglog inverse link saturates (mu -> 1) once the linear predictor passes
    ~ +1, and statsmodels' IRLS has no step-halving -- a handful of eta ~ +15
    rows sent the weights, then the whole fit, to overflow. Capping is the
    standard treatment for pathological LTVs in mortgage PD models and changes
    the economics of 0.004% of rows.
  * Warm start: IRLS is started from the same-formula LOGIT solution. For
    rare events (quarterly hazard ~ 2.7%) the two links agree to first order,
    so the logit optimum sits inside the cloglog basin of attraction; from the
    default start the cloglog IRLS diverged. Convergence is asserted --
    a non-converged fit raises rather than returning silently.

Estimation: statsmodels GLM Binomial(CLogLog), fitted by IRLS. The patsy
formula machinery memorises the spline knots and centering means from the
training data, so predictions on new frames (including ages beyond the
boundary knots) reuse the training basis.
"""

from __future__ import annotations

from dataclasses import dataclass, field

import numpy as np
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf

SPLINE_DF = 5   # natural-cubic-spline df for the loan-age baseline
LTV_CAP = 300.0  # winsorisation cap for updated_ltv (%), see module docstring

# Right-hand side shared by both cause-specific hazards. The prepayment model
# keeps the full set (symmetric specification): incentive is its star driver,
# while credit covariates capture the well-documented fact that impaired
# borrowers cannot refinance (negative equity / low FICO suppress prepayment).
_RHS = (
    f"cr(loan_age, df={SPLINE_DF})"
    " + fico_s + ltv10 + prepay_incentive"
    " + investor_orig_time + REtype_CO_orig_time + REtype_PU_orig_time"
    " + REtype_SF_orig_time"
    " + uer_lag1 + uer_chg4_lag1 + hpi_growth_lag1 + gdp_lag1"
    " + center(ltv10):center(uer_lag1)"
)

#: name of the double-trigger interaction term in the fitted params
DT_TERM = "center(ltv10):center(uer_lag1)"

DEFAULT_FORMULA = "default_event ~ " + _RHS
PREPAY_FORMULA = "payoff_event ~ " + _RHS

#: raw panel columns every scoring frame must carry
REQUIRED_COLS = [
    "loan_age", "FICO_orig_time", "updated_ltv", "prepay_incentive",
    "investor_orig_time", "REtype_CO_orig_time", "REtype_PU_orig_time",
    "REtype_SF_orig_time", "uer_lag1", "uer_chg4_lag1", "hpi_growth_lag1",
    "gdp_lag1",
]


@dataclass
class HazardModel:
    """A fitted cause-specific discrete-time hazard.

    Wraps the statsmodels GLMResults together with the formula and event
    column so downstream code (term structure, validation) never has to
    re-derive the specification.
    """

    result: object                 # statsmodels GLMResultsWrapper
    formula: str
    event_col: str                 # 'default_event' or 'payoff_event'
    name: str                      # 'default' | 'prepay'
    n_obs: int = 0
    n_events: int = 0
    fit_meta: dict = field(default_factory=dict)

    @property
    def params(self) -> pd.Series:
        return self.result.params

    def mcfadden_r2(self) -> float:
        """McFadden pseudo-R2 = 1 - llf / ll(intercept-only)."""
        return float(1.0 - self.result.llf / self.result.llnull)


def _prepare(df: pd.DataFrame) -> pd.DataFrame:
    """Add the scaled covariates the formula expects (non-destructive copy)."""
    missing = [c for c in REQUIRED_COLS if c not in df.columns]
    if missing:
        raise KeyError(f"scoring frame is missing required columns: {missing}")
    out = df.copy()
    out["fico_s"] = out["FICO_orig_time"] / 100.0   # per 100 FICO points
    # per 10pp updated LTV, winsorised at LTV_CAP (module docstring)
    out["ltv10"] = out["updated_ltv"].clip(upper=LTV_CAP) / 10.0
    return out


def _fit(panel_df: pd.DataFrame, formula: str, event_col: str,
         name: str) -> HazardModel:
    d = panel_df
    if "lag_warmup" in d.columns:
        d = d[d["lag_warmup"] == 0]
    d = _prepare(d)
    need = REQUIRED_COLS + [event_col]
    n_before = len(d)
    d = d.dropna(subset=need)
    # IRLS on the cloglog link can diverge from the default start (the inverse
    # link saturates and the working weights explode). Warm-start from a logit
    # fit: for rare events logit and cloglog coefficients nearly coincide
    # (both links agree to first order as lambda -> 0), so the logit solution
    # sits inside the cloglog basin of attraction.
    warm = smf.glm(
        formula, data=d, family=sm.families.Binomial(),
    ).fit()
    model = smf.glm(
        formula, data=d,
        family=sm.families.Binomial(link=sm.families.links.CLogLog()),
    )
    res = model.fit(start_params=warm.params.to_numpy(), maxiter=200)
    if not res.converged:
        raise RuntimeError(f"{name} hazard GLM did not converge")
    return HazardModel(
        result=res, formula=formula, event_col=event_col, name=name,
        n_obs=len(d), n_events=int(d[event_col].sum()),
        fit_meta={"rows_dropped_nan": n_before - len(d),
                  "warm_start": "logit"},
    )


def fit_default_hazard(panel_df: pd.DataFrame) -> HazardModel:
    """Fit the cause-specific DEFAULT hazard on loan-quarter at-risk rows.

    Pass TRAINING rows only (split == 'train'); the function additionally
    drops lag-warm-up rows. Payoff (competing-risk) rows enter with y = 0 and
    the loan's history simply ends there -- that IS the cause-specific
    treatment (module docstring).
    """
    return _fit(panel_df, DEFAULT_FORMULA, "default_event", "default")


def fit_prepay_hazard(panel_df: pd.DataFrame) -> HazardModel:
    """Fit the cause-specific PREPAYMENT hazard (y = payoff_event).

    Same framework as the default hazard with default as the censoring
    competing risk; prepay_incentive (note rate minus current market rate)
    is the star covariate.
    """
    return _fit(panel_df, PREPAY_FORMULA, "payoff_event", "prepay")


def predict_hazard(model: HazardModel, df: pd.DataFrame) -> np.ndarray:
    """Per-row conditional hazard lambda_t = P(event in t | at risk at t).

    Uses the training-memorised spline basis; ages beyond the training
    boundary knots extrapolate linearly on the cloglog scale (natural cubic
    spline tail behaviour -- the documented tail assumption).
    """
    return np.asarray(model.result.predict(_prepare(df)))


def pd_term_structure(models: dict[str, HazardModel],
                      profile_df: pd.DataFrame,
                      horizon: int) -> pd.DataFrame:
    """Competing-risk PD term structure for one or more loan profiles.

    Parameters
    ----------
    models : {'default': HazardModel, 'prepay': HazardModel}
    profile_df : one row per profile carrying REQUIRED_COLS (loan_age is the
        age the loan will have in the FIRST projected quarter; 0 = new loan)
        and optionally a 'profile' name column.
    horizon : number of quarters to project.

    Returns a long DataFrame with, per profile and period t = 1..horizon:
        hazard_default   lambda_d(t)  conditional default hazard
        hazard_prepay    lambda_p(t)  conditional prepayment hazard
        survival         S(t) = prod_{k<=t} (1 - lambda_d(k) - lambda_p(k)),
                         i.e. probability of surviving BOTH default and
                         prepayment (in a discrete competing-risk period the
                         loan exits by at most one cause, so per-period exit
                         probability is the sum of cause-specific hazards)
        marginal_pd      S(t-1) * lambda_d(t)  (unconditional default in t)
        cum_pd           sum_{k<=t} marginal_pd(k)  = P(default by t)

    Simplifications (documented, per MASTER_PLAN rung-1 tail assumption):
    all covariates other than loan_age are FROZEN at the profile's values --
    no amortisation of updated LTV, no macro path; beyond the observed-age
    support the spline baseline extrapolates linearly on the cloglog scale.
    The scenario-conditioning of the macro path is a later rung (satellite
    models + reversion to TTC).
    """
    for key in ("default", "prepay"):
        if key not in models:
            raise KeyError(f"models dict must contain '{key}'")
    frames = []
    for i, row in profile_df.reset_index(drop=True).iterrows():
        name = row.get("profile", f"profile_{i}")
        age0 = float(row["loan_age"])
        grid = pd.DataFrame([row.drop(labels=[c for c in ["profile"]
                                              if c in row.index])] * horizon)
        grid = grid.reset_index(drop=True)
        grid["loan_age"] = age0 + np.arange(horizon, dtype=float)
        grid = grid.astype({c: float for c in REQUIRED_COLS})
        lam_d = predict_hazard(models["default"], grid)
        lam_p = predict_hazard(models["prepay"], grid)
        exit_p = np.clip(lam_d + lam_p, 0.0, 0.999999)
        surv = np.cumprod(1.0 - exit_p)
        surv_prev = np.concatenate([[1.0], surv[:-1]])
        marginal = surv_prev * lam_d
        frames.append(pd.DataFrame({
            "profile": name,
            "period": np.arange(1, horizon + 1),
            "loan_age": grid["loan_age"].to_numpy(),
            "hazard_default": lam_d,
            "hazard_prepay": lam_p,
            "survival": surv,
            "marginal_pd": marginal,
            "cum_pd": np.cumsum(marginal),
        }))
    return pd.concat(frames, ignore_index=True)