#!/usr/bin/env python3
"""
Cause test, exploratory follow-up (October 12, 2026, late evening). EXPLORATORY.

Chosen after the cause test's decision (FAIL, as registered) was known, and run on the same saved runs, so nothing here
is a test, whatever it shows. The decision is not revisited: it stays FAIL, refuted under the tested conditions. The aim
is to find the next discriminating hypothesis, not to rescue the last one.

Read-only. The model is not run (no new simulation; no base run regenerated). Every file read is fingerprinted
(SHA-256) before and after, and the two lists must match. Outputs go to this folder only. A sandbox script kept with the
project; nothing here is served by the site.

Reads
    ../cause-test-2026-10-12/sim_cause4000.npz    the cause test's 42 base runs (starts 19 to 60) and 420 moments
    ../cause-test-2026-10-12/cause_test.json      the report's numbers, to check that this code gives the same ones
    ../cause-test-2026-10-12/design_numbers.json  the benchmark
    the 18 older saved starts at the same setting (starts 1 to 18), through design_numbers.saved_sets(), as a
    cross-check for the base-run analyses only

The list of analyses below was written before any of them was run. Nothing was added after seeing results; anything
that had to change after a first run is listed in the report under "Changes after the first run".

What it works out
  A. The untouched (base) runs: every T = 2,000, 2,005, ..., 5,500 in each start (701 per start). W: the growth rate of
     the gap over (T + 200, T + 500]; C(T): the coherence at T (A and B averaged); g1 to g5: the growth rate in the five
     100-unit windows up to T, as in Test 17 and the benchmark.
     A1. The benchmark slope on the new starts, start by start: W = a + beta C(T) + c1 g1 + ... + c5 g5, as in
         design_numbers.py (this code gives the benchmark's -0.1602 on the old 18 starts, checked), and without g.
     A2. Natural persistence: the slope of the mean coherence over (T, T + 200] and over (T + 200, T + 500] on C(T),
         beside the interventions' own (from the report).
     A3. Candidate carriers known at T. For each feature F: the partial correlation of F with W given g; of F with W
         given g and C(T); and of C(T) with W given g and F (does the coherence's link survive F?). Features: the mean
         coherence over (T - 100, T] and over (T - 500, T]; its trend (least-squares slope over (T - 100, T]); the
         alignment of the groups' mean phases, cos(psi_A - psi_B); the cross term Re(z_A conj z_B); the imbalance
         |R_A - R_B|; the mean field's turning rate |d psi / dt| over (T - 25, T] (average of A and B). Also all of them
         together.
     A4. What comes after T: C(T)'s partial correlation with W given g and (i) the early growth E, the growth rate over
         (T, T + 200]; (ii) the mean coherence over (T, T + 200] and over (T + 200, T + 500]; (iii) all three. Start by
         start: W = a + b_C C(T) + b_E E + c1 g1 + ... + c5 g5; E = a + a_C C(T) + c1 g1 + ... (the early link); and the
         same-window link, W against the mean coherence over (T + 200, T + 500], given g.
  B. The 420 moments, untouched copy at T (the higher harmonics |z2|, |z3|, |z4| are saved only here): W (the untouched
     continuation's growth over (T + 200, T + 500]) against C(T), |z2|, |z3|, |z4|, cos(psi_A - psi_B) and g, each
     start's own means taken out, pooled over the moments; scaled weights; error bars from redrawing the starts.
  C. The interventions: each moment's six branches and its untouched continuation ("none").
     C1. Growth against both controls over (T, T + 200] (immediate), (T + 200, T + 500] (delayed) and in 100-unit
         steps: up - sham, down - sham, sham - none, up - none, down - none; S = (down - up) / 2; P = (up + down) / 2 -
         sham. Per method and pooled.
     C2. The state after the change in the same windows: the coherence C, the alignment cos(psi_A - psi_B) and the
         cross term Re(z_A conj z_B); up - sham, down - sham, sham - none, (up - down) / 2.
         Slopes in one unit (growth per unit of coherence), natural against induced: C(T) against the delayed growth,
         C(T) against the early growth, and the coherence in the delayed window against the growth in the same window.
     C3. Moment by moment within each method (each start's own means taken out, error bars from redrawing the starts):
         a. S against the up/down difference in |z2|, |z3| and |z4| right after the change (with C(T)); and pooled over
            the methods, how much of the dense-sparse difference in S those differences account for.
         b. sham - none against the sham's own change in |z2|, |z3| and |z4| (with C(T)).
         c. S against how much coherence difference was left in the delayed window, (C_up - C_down) / 2 over
            (T + 200, T + 500] (with C(T)).
         d. S against the achieved change in coherence: if coherence were the cause at the benchmark's size, S would
            grow with it at 0.1602 per unit.
     C4. The early growth as a carrier. Start by start, the delayed effect each arm would have against its sham if the
         untouched runs' A4 regression applied to the arms: b_E x (the arm's own early effect) alone, and with
         b_C x (the arm's change in C), against the delayed effect seen. And, moment by moment within each method and
         arm, the delayed effect against the early one.
Error bars: for per-start values, as in the cause test (cause_rule.error_bar: the larger of the resampled and
SD/sqrt(n), the starts redrawn 400 times with seed 20261015); for pooled moment-level weights, the SD of the weights
refitted on the same 400 redraws of the starts.

Usage
    python3 -B explore_cause.py      # explore_cause.txt, explore_cause.json and explore_cause.png, here

Needs Python 3, numpy and matplotlib, and ../cause-test-2026-10-12 and ../coherence-growth-2026-10-11 with their saved
runs, unchanged.
"""
import hashlib
import json
import sys
from pathlib import Path

import numpy as np

HERE = Path(__file__).resolve().parent
CAUSE = HERE.parent / "cause-test-2026-10-12"
CG = HERE.parent / "coherence-growth-2026-10-11"
sys.dont_write_bytecode = True                             # leave every folder as it is
sys.path.insert(0, str(CAUSE))
sys.path.insert(1, str(CG))
import cause_rule as rule  # noqa: E402  (the decision rule's error bars and averages; unchanged)
import design_numbers as dn  # noqa: E402  (the 18 older saved starts; unchanged)
import earlier_growth_check as egc  # noqa: E402  (the earlier growth and partial correlations, as in Test 17)

SAVED = CAUSE / "sim_cause4000.npz"
REPORT, DESIGN = CAUSE / "cause_test.json", CAUSE / "design_numbers.json"
EVERY = 5                                                  # time units per interval of the gap
FIRST, LAST = 2000, 5500                                   # base-run moments T, as in the benchmark
EARLY, LATER = (0, 200), (200, 500)                        # immediate and delayed windows after T
STEPS = ((0, 100), (100, 200), (200, 300), (300, 400), (400, 500))
BIN = 25                                                   # the figure's growth profile
METHODS, ARMS = rule.METHODS, rule.ARMS
FEATURES = (("past100", "mean C over (T - 100, T]"), ("past500", "mean C over (T - 500, T]"),
            ("trend", "trend of C over (T - 100, T]"), ("align", "alignment cos(psi_A - psi_B) at T"),
            ("cross", "cross term Re(z_A conj z_B) at T"), ("imbalance", "imbalance |R_A - R_B| at T"),
            ("turning", "turning rate |dpsi/dt|, (T - 25, T]"))
AFTER = (("E", ("E",), "early growth E over (T, T + 200]"),
         ("C after", ("early_C", "later_C"), "mean C over (T, T + 200] and (T + 200, T + 500]"),
         ("all", ("E", "early_C", "later_C"), "all three"))
OUT_TXT, OUT_JSON, FIGURE = (HERE / f"explore_cause.{x}" for x in ("txt", "json", "png"))


def fingerprints(paths):
    return {str(p.relative_to(HERE.parent)): hashlib.sha256(p.read_bytes()).hexdigest() for p in paths}


def fit(y, columns):
    """Least squares of y on a constant and the columns: the weights (constant first)."""
    design = np.column_stack([np.ones(len(y)), *columns])
    return np.linalg.lstsq(design, y, rcond=None)[0]


def slope(y, x):
    x = x - x.mean()
    return float(x @ (y - y.mean()) / (x @ x))


def summary(values, picks):
    values = np.asarray(values, float)
    bar = float(rule.error_bar(values, picks))
    mean = float(values.mean())
    return dict(mean=mean, error_bar=bar, bars=mean / bar if bar > 0 else float("nan"),
                above=int((values > 0).sum()), n=int(values.size))


def within(y, X, picks):
    """y: (start, row); X: (start, row, k). Each start's own means taken out, least squares pooled over the rows; the
    weights, their SD over the 400 redraws of the starts, and the same scaled (each variable over its pooled spread
    within starts)."""
    yd = y - y.mean(1, keepdims=True)
    Xd = X - X.mean(1, keepdims=True)

    def solve(rows):
        return np.linalg.lstsq(Xd[rows].reshape(-1, X.shape[-1]), yd[rows].reshape(-1), rcond=None)[0]

    weights = solve(np.arange(len(y)))
    redrawn = np.array([solve(p) for p in picks])
    bars = redrawn.std(0)
    scale = Xd.reshape(-1, X.shape[-1]).std(0) / yd.std()
    return dict(weights=weights.tolist(), bars=bars.tolist(), scaled=(weights * scale).tolist(),
                scaled_bars=(bars * scale).tolist())


# ---------------------------------------------------------------------------------------------------------------------
# A: the untouched runs

def base_start(z, logs):
    """Every base-run series for one start at T = 2,000, 2,005, ..., 5,500. z: (group, time), every time unit from
    t = 0; logs: (interval,), interval j covering (5j, 5j + 5]."""
    z = z.astype(np.complex128)
    R = np.abs(z)
    c = R.mean(0)
    T = np.arange(FIRST, LAST + 1, EVERY)
    js = T // EVERY - 1                                    # the interval ending at T
    total = np.concatenate([[0.0], np.cumsum(logs)])

    def growth(a, b):                                      # the growth rate over (T + a, T + b]
        return (total[js + 1 + b // EVERY] - total[js + 1 + a // EVERY]) / (b - a)

    cum = np.concatenate([[0.0], np.cumsum(c)])

    def cmean(a, b):                                       # the mean coherence over (T + a, T + b]
        return (cum[T + b + 1] - cum[T + a + 1]) / (b - a)

    offsets = np.arange(-99, 1)
    recent = c[T[:, None] + offsets]                       # C over (T - 100, T]
    x = offsets - offsets.mean()
    trend = (recent - recent.mean(1, keepdims=True)) @ x / (x @ x) * 100   # change per 100 time units
    turn = np.angle(z[:, 1:] * np.conj(z[:, :-1]))         # each group's mean phase turning over (t - 1, t]
    turned = np.concatenate([np.zeros((2, 1)), np.cumsum(turn, 1)], 1)
    turning = np.abs((turned[:, T] - turned[:, T - 25]) / 25).mean(0)
    za, zb = z[0, T], z[1, T]
    cross = (za * np.conj(zb)).real
    return dict(W=growth(*LATER), E=growth(*EARLY), g=egc.before(logs, js, egc.MAIN), C=c[T],
                past100=cmean(-100, 0), past500=cmean(-500, 0), trend=trend, align=cross / (np.abs(za) * np.abs(zb)),
                cross=cross, imbalance=np.abs(R[0, T] - R[1, T]), turning=turning,
                early_C=cmean(*EARLY), later_C=cmean(*LATER))


def base_numbers(f):
    """One start's numbers for A1 to A4."""
    W, C, g = f["W"], f["C"], f["g"]
    out = dict(beta_held=fit(W, [C, *g.T])[1], beta_plain=fit(W, [C])[1],
               persist_early=slope(f["early_C"], C), persist_later=slope(f["later_C"], C),
               link=egc.partial(C, W, g))
    for key, _ in FEATURES:
        F = f[key]
        out[f"{key}:F|g"] = egc.partial(F, W, g)
        out[f"{key}:F|g,C"] = egc.partial(F, W, np.column_stack([g, C]))
        out[f"{key}:C|g,F"] = egc.partial(C, W, np.column_stack([g, F]))
    out["every feature:C|g,F"] = egc.partial(C, W, np.column_stack([g] + [f[key] for key, _ in FEATURES]))
    for name, keys, _ in AFTER:
        out[f"after {name}:C|g,X"] = egc.partial(C, W, np.column_stack([g] + [f[k] for k in keys]))
    w = fit(W, [C, f["E"], *g.T])
    out["b_C"], out["b_E"] = float(w[1]), float(w[2])
    out["a_C"] = float(fit(f["E"], [C, *g.T])[1])
    out["same_window"] = float(fit(W, [f["later_C"], *g.T])[1])
    out["same_window_r"] = egc.partial(f["later_C"], W, g)
    out["E_W_r"] = egc.partial(f["E"], W, np.column_stack([g, C]))
    return out


# ---------------------------------------------------------------------------------------------------------------------
# the run

def main():
    inputs = [SAVED, REPORT, DESIGN, Path(dn.cg.EARLIER), Path(dn.cg.SAVED), Path(dn.REPLICATION)]
    before = fingerprints(inputs)
    d = np.load(SAVED)
    report = json.loads(REPORT.read_text())
    bench = float(json.loads(DESIGN.read_text())["benchmark"]["beta"])      # 0.1602, the size of the benchmark slope
    starts = [int(s) for s in d["starts"]]
    n = len(starts)
    picks = rule.draws(n)
    carried = np.asarray(d["carried"], bool)
    assert carried.all(), "this script assumes every moment was carried out (all 420 were)"
    T = np.asarray(d["T"], int)
    per = T.shape[1]
    z_base, logs_base = np.asarray(d["z"]), np.asarray(d["logs"], float)
    res = dict(exploratory=("chosen after the cause test's decision (FAIL) was known and run on the same saved runs:"
                            " not a test; the decision stays FAIL as registered"),
               starts=starts, moments=int(carried.sum()), changes_after_first_run=[])

    # A
    new = [base_numbers(base_start(z_base[i], logs_base[i])) for i in range(n)]
    old_runs = [(z[j], logs[j]) for _, z, logs in dn.saved_sets() for j in range(len(z))]
    old = [base_numbers(base_start(z, logs)) for z, logs in old_runs]
    picks_old = rule.draws(len(old))
    assert abs(np.mean([x["beta_held"] for x in old]) + bench) < 1e-9, "the benchmark is not reproduced"

    def both(key):
        return dict(new=summary([x[key] for x in new], picks), old=summary([x[key] for x in old], picks_old))

    A = {key: both(key) for key in new[0]}
    res["A"] = A

    # B
    c_base = np.abs(z_base.astype(np.complex128)).mean(1)                    # (start, time)
    first = T // EVERY

    def none_rate(window):
        low, high = (v // EVERY for v in window)
        return np.array([[logs_base[i, first[i, m] + low:first[i, m] + high].sum() for m in range(per)]
                         for i in range(n)]) / (window[1] - window[0])

    C0 = np.asarray(d["C0"], float)
    assert np.allclose(C0, c_base[np.arange(n)[:, None], T], atol=1e-6), "C0 is not the base run's coherence at T"
    W_none = none_rate(LATER)
    g_m = np.stack([egc.before(logs_base[i], T[i] // EVERY - 1, egc.MAIN) for i in range(n)])   # (start, moment, 5)
    zn0 = {k: np.asarray(d[f"z{k}_0"], float).mean(-1) for k in (2, 3, 4)}
    align0 = np.cos(np.asarray(d["ab0"], float))
    gcols = [g_m[..., j] for j in range(5)]
    b_models = {"C(T) + g": (["C(T)"], [C0]),
                "C(T) + g + |z2|, |z3|, |z4|": (["C(T)", "|z2|", "|z3|", "|z4|"], [C0, zn0[2], zn0[3], zn0[4]]),
                "g + |z2|, |z3|, |z4| (no C)": (["|z2|", "|z3|", "|z4|"], [zn0[2], zn0[3], zn0[4]]),
                "C(T) + g + alignment": (["C(T)", "alignment"], [C0, align0])}
    B = {}
    for name, (labels, cols) in b_models.items():
        out = within(W_none, np.stack(cols + gcols, -1), picks)
        B[name] = {label: dict(scaled=out["scaled"][j], scaled_bar=out["scaled_bars"][j], weight=out["weights"][j],
                               bar=out["bars"][j]) for j, label in enumerate(labels)}
    res["B"] = B

    # C1
    bl = np.asarray(d["branch_logs"], float)

    def rate(x, window):
        low, high = (v // EVERY for v in window)
        return x[..., low:high].sum(-1) / (window[1] - window[0])

    def by_method(v):
        """v: (start, moment, method). Each start's average over its moments, then the methods and their difference."""
        s = v.mean(1)
        return dict(dense=summary(s[:, 0], picks), sparse=summary(s[:, 1], picks), pooled=summary(s.mean(-1), picks),
                    sparse_minus_dense=summary(s[:, 1] - s[:, 0], picks))

    windows = {"immediate": EARLY, "delayed": LATER, **{f"{a}-{b}": (a, b) for a, b in STEPS}}
    G = {name: rate(bl, w).reshape(n, per, 2, 3) for name, w in windows.items()}
    none = {name: none_rate(w)[..., None] for name, w in windows.items()}

    def comparisons(g, nn):
        return {"up - sham": g[..., 0] - g[..., 2], "down - sham": g[..., 1] - g[..., 2], "sham - none": g[..., 2] - nn,
                "up - none": g[..., 0] - nn, "down - none": g[..., 1] - nn, "S": (g[..., 1] - g[..., 0]) / 2,
                "P": (g[..., 0] + g[..., 1]) / 2 - g[..., 2]}

    C1 = {name: {key: by_method(v) for key, v in comparisons(G[name], none[name]).items()} for name in windows}
    assert abs(C1["delayed"]["S"]["pooled"]["mean"] - report["primary"]["pooled"]["S"]["mean"]) < 1e-12
    assert abs(C1["delayed"]["S"]["pooled"]["error_bar"] - report["primary"]["pooled"]["S"]["error_bar"]) < 1e-12
    assert abs(C1["delayed"]["sham - none"]["dense"]["mean"] - report["sham_vs_none"]["later"]["dense"]["mean"]) < 1e-12
    res["C1"] = C1

    # the figure's growth profile, 25-unit bins
    bins = [(a, a + BIN) for a in range(0, 500, BIN)]
    profile = {}
    for a, b in bins:
        for key, v in comparisons(rate(bl, (a, b)).reshape(n, per, 2, 3), none_rate((a, b))[..., None]).items():
            profile.setdefault(key, []).append(v.mean(1).mean(0).tolist())          # (method,) per bin
    res["profile"] = dict(bins=bins, **{key: np.array(v).T.tolist() for key, v in profile.items()})

    # C2: the state after the change
    bz = np.asarray(d["branch_z"]).astype(np.complex128)                    # (start, moment, branch, group, time)
    idx = T[..., None] + np.arange(bz.shape[-1])
    nz = np.stack([np.moveaxis(z_base[i][:, idx[i]], 0, 1) for i in range(n)]).astype(np.complex128)

    def state(zz):
        za, zb = zz[..., 0, :], zz[..., 1, :]
        cross = (za * np.conj(zb)).real
        return dict(C=np.abs(zz).mean(-2), align=cross / (np.abs(za) * np.abs(zb)), cross=cross)

    sb, sn = state(bz), state(nz)
    spans = {"at T": slice(0, 1), "immediate": slice(1, 201), "delayed": slice(201, 501)}
    C2, paths = {}, {}
    for key in ("C", "align", "cross"):
        b = sb[key].reshape(n, per, 2, 3, -1)
        nn = sn[key][:, :, None, :]
        diffs = {"up - sham": b[..., 0, :] - b[..., 2, :], "down - sham": b[..., 1, :] - b[..., 2, :],
                 "sham - none": b[..., 2, :] - nn, "(up - down) / 2": (b[..., 0, :] - b[..., 1, :]) / 2}
        C2[key] = {name: {span: by_method(v[..., s].mean(-1)) for span, s in spans.items()} for name, v in diffs.items()}
        paths[key] = {name: v.mean((0, 1)).tolist() for name, v in diffs.items()}            # (method, time)
    ud = sb["C"].reshape(n, per, 2, 3, -1)
    ud = ud[..., 0, :] - ud[..., 1, :]
    phi = float(ud[..., 1:201].mean() / ud[..., 0].mean())
    left = float(ud[..., 201:501].mean() / ud[..., 0].mean())
    assert abs(phi - report["phi"]["pooled"]) < 1e-9 and abs(left - report["left_over"]["pooled"]) < 1e-9
    res["C2"] = C2
    res["paths"] = paths

    # slopes in one unit: growth per unit of coherence, natural against induced
    C1v = np.asarray(d["C1"], float).reshape(n, per, 2, 3)
    dC = {"up": C1v[..., 0] - C1v[..., 2], "down": C1v[..., 1] - C1v[..., 2]}  # the change in C at T against the sham
    Ge, Gl = G["immediate"], G["delayed"]
    De = {"up": Ge[..., 0] - Ge[..., 2], "down": Ge[..., 1] - Ge[..., 2]}
    Dl = {"up": Gl[..., 0] - Gl[..., 2], "down": Gl[..., 1] - Gl[..., 2]}
    late_C = ud[..., 201:501].mean(-1) / 2                                   # (C_up - C_down) / 2, delayed window

    def induced(num, den):
        """Per start: the moment average of num over the moment average of den, (start, method)."""
        return num.mean(1) / den.mean(1)

    spread = dC["up"] - dC["down"]
    induced_slopes = {"C(T) -> delayed growth": induced(Dl["up"] - Dl["down"], spread),
                      "C(T) -> early growth": induced(De["up"] - De["down"], spread),
                      "C in the delayed window -> growth in it": induced(Dl["up"] - Dl["down"], 2 * late_C)}
    natural_slopes = {"C(T) -> delayed growth": "beta_held", "C(T) -> early growth": "a_C",
                      "C in the delayed window -> growth in it": "same_window"}
    res["slopes"] = {name: dict(natural=A[natural_slopes[name]],
                                induced=dict(dense=summary(v[:, 0], picks), sparse=summary(v[:, 1], picks),
                                             pooled=summary(v.mean(-1), picks)))
                     for name, v in induced_slopes.items()}

    # C3: moment by moment within each method
    zn1 = {k: np.asarray(d[f"z{k}_1"], float).mean(-1).reshape(n, per, 2, 3) for k in (2, 3, 4)}
    h = {k: (zn1[k][..., 1] - zn1[k][..., 0]) / 2 for k in (2, 3, 4)}       # the up/down difference, oriented as S
    q = {k: zn1[k][..., 2] - zn0[k][..., None] for k in (2, 3, 4)}           # the sham's own change
    S_m = (Gl[..., 1] - Gl[..., 0]) / 2
    shamnone_m = Gl[..., 2] - W_none[..., None]
    size_m = (np.abs(C1v[..., 0] - C0[..., None]) + np.abs(C1v[..., 1] - C0[..., None])) / 2

    def per_method(y, cols, labels):
        out = {}
        for k, method in enumerate(METHODS):
            fit_k = within(y[..., k], np.stack([c[..., k] if c.ndim == 3 else c for c in cols], -1), picks)
            out[method] = {label: dict(weight=fit_k["weights"][j], bar=fit_k["bars"][j], scaled=fit_k["scaled"][j],
                                       scaled_bar=fit_k["scaled_bars"][j]) for j, label in enumerate(labels)}
        return out

    def pooled_methods(y, cols, labels):
        """Both methods' moments as rows of each start, with a column that is 1 for the sparse rows."""
        yy = np.concatenate([y[..., 0], y[..., 1]], 1)
        stacked = [np.concatenate([np.zeros((n, per)), np.ones((n, per))], 1)]
        for c in cols:
            stacked.append(np.concatenate([c[..., 0], c[..., 1]], 1) if c.ndim == 3 else np.concatenate([c, c], 1))
        out = within(yy, np.stack(stacked, -1), picks)
        return {label: dict(weight=out["weights"][j], bar=out["bars"][j])
                for j, label in enumerate(["sparse rows"] + labels)}

    hs = [h[2], h[3], h[4]]
    C3 = dict(
        a=dict(per_method=per_method(S_m, hs + [C0], ["h2", "h3", "h4", "C(T)"]),
               pooled_without=pooled_methods(S_m, [C0], ["C(T)"]),
               pooled_with=pooled_methods(S_m, hs + [C0], ["h2", "h3", "h4", "C(T)"]),
               h_levels={f"h{k}": by_method(h[k]) for k in (2, 3, 4)}),
        b=dict(per_method=per_method(shamnone_m, [q[2], q[3], q[4], C0], ["q2", "q3", "q4", "C(T)"]),
               q_levels={f"q{k}": by_method(q[k]) for k in (2, 3, 4)}),
        c=dict(per_method=per_method(S_m, [late_C, C0], ["C left (delayed)", "C(T)"]),
               pooled=pooled_methods(S_m, [late_C, C0], ["C left (delayed)", "C(T)"])),
        d=dict(per_method=per_method(S_m, [size_m], ["achieved change"]),
               pooled=pooled_methods(S_m, [size_m], ["achieved change"]), predicted=bench))
    res["C3"] = C3

    # C4: the early growth as a carrier
    bC = np.array([x["b_C"] for x in new])
    bE = np.array([x["b_E"] for x in new])
    C4 = {}
    for k, method in enumerate(METHODS):
        for arm in ("up", "down"):
            seen = Dl[arm][..., k].mean(1)
            early = De[arm][..., k].mean(1)
            change = dC[arm][..., k].mean(1)
            rebound = bE * early
            full = bC * change + bE * early
            moment = within(Dl[arm][..., k], De[arm][..., k][..., None], picks)
            C4[f"{method} {arm}"] = dict(seen=summary(seen, picks), early=summary(early, picks),
                                         rebound_only=summary(rebound, picks), with_C=summary(full, picks),
                                         seen_minus_with_C=summary(seen - full, picks),
                                         moment_slope=dict(weight=moment["weights"][0], bar=moment["bars"][0]))
    res["C4"] = C4

    after = fingerprints(inputs)
    res["fingerprints"] = dict(inputs=before, unchanged=before == after)
    assert before == after, "an input changed while this script ran"
    res["checks"] = dict(benchmark_old_18=float(np.mean([x["beta_held"] for x in old])),
                         report_S=C1["delayed"]["S"]["pooled"]["mean"], report_phi=phi, report_left_over=left)
    OUT_JSON.write_text(json.dumps(res, indent=1))
    lines = write_report(res, bench)
    OUT_TXT.write_text("\n".join(lines) + "\n")
    figure(res)
    print(f"\nWrote {OUT_TXT.name}, {OUT_JSON.name} and {FIGURE.name}")


# ---------------------------------------------------------------------------------------------------------------------
# the report

def write_report(res, bench):
    lines = []

    def say(text=""):
        print(text)
        lines.append(text)

    def g(s, digits=5):                                    # a growth-rate cell
        return f"{s['mean']:+.{digits}f} ± {s['error_bar']:.{digits}f} ({s['bars']:+.1f})"

    def r(s):                                              # a correlation cell
        return f"{s['mean']:+.4f} ± {s['error_bar']:.4f}"

    def w(x, digits=5):
        return f"{x['weight']:+.{digits}f} ± {x['bar']:.{digits}f}"

    def sc(x):
        return f"{x['scaled']:+.3f} ± {x['scaled_bar']:.3f}"

    A = res["A"]
    say("Cause test, exploratory follow-up: October 12, 2026. N = 4,000 per group, K = 1.90, eps = 0.1")
    say("EXPLORATORY: chosen after the cause test's decision was known and run on the same saved runs, so not a test.")
    say("The decision stands: FAIL as registered (refuted under the tested conditions). Nothing here changes it.")
    say(f"Read-only: the model was not run. Inputs fingerprinted before and after: "
        f"{'unchanged' if res['fingerprints']['unchanged'] else 'CHANGED'}.")
    say(f"Data: the cause test's {len(res['starts'])} new starts (19 to 60) and {res['moments']} moments; for part A also"
        " the 18 older starts (1 to 18) as a cross-check.")
    say("Cells: mean ± error bar (mean in error bars); error bars from the starts, as in the cause test.")
    say("Checks: this code gives the benchmark on the old 18 starts"
        f" ({res['checks']['benchmark_old_18']:+.4f}) and the report's S ({res['checks']['report_S']:+.5f}),"
        f" Phi ({res['checks']['report_phi']:.3f}) and share left ({res['checks']['report_left_over']:.3f}).")
    say()
    say("A. THE UNTOUCHED RUNS (every T = 2,000, 2,005, ..., 5,500; W = growth rate over (T + 200, T + 500])")
    say(f"  {'':<58}{'new 42 starts':<30}old 18 starts")

    def row(label, key, cell=g, digits=4):
        a = A[key]
        left = cell(a["new"], digits) if cell is g else cell(a["new"])
        right = cell(a["old"], digits) if cell is g else cell(a["old"])
        say(f"  {label:<58}{left:<30}{right}")

    say("A1. The benchmark slope (growth per unit of C(T)); the benchmark is -0.1602, from the old 18")
    row("W = a + beta C(T) + c1 g1 + ... + c5 g5: beta", "beta_held")
    row("W = a + beta C(T): beta", "beta_plain")
    row("partial correlation of C(T) with W given g1 to g5", "link", r)
    say("  (Test 17's own measure, -0.1589 on the new starts in the report, is a partial correlation averaged over")
    say("  5-unit lags, not a slope; it is on a different scale from the benchmark and the two should not be compared.)")
    say()
    say("A2. Natural persistence of a coherence difference at T (slope of the later mean C on C(T))")
    row("over (T, T + 200]", "persist_early", g, 3)
    row("over (T + 200, T + 500]", "persist_later", g, 3)
    say(f"  the interventions, from the report: {res['checks']['report_phi']:.3f} over (T, T + 200] and"
        f" {res['checks']['report_left_over']:.3f} over (T + 200, T + 500]")
    say()
    say("A3. Candidate carriers known at T: partial correlations with W (new 42 | old 18)")
    say(f"  {'feature F':<42}{'F with W, given g':<36}{'F with W, given g and C(T)':<36}C(T) with W, given g and F")
    say(f"  {'(nothing added: C(T) with W given g)':<42}{'':<36}{'':<36}{r(A['link']['new'])} | {r(A['link']['old'])}")
    for key, label in FEATURES:
        cells = [f"{r(A[f'{key}:{part}']['new'])} | {A[f'{key}:{part}']['old']['mean']:+.3f}"
                 for part in ("F|g", "F|g,C", "C|g,F")]
        say(f"  {label:<42}{cells[0]:<36}{cells[1]:<36}{cells[2]}")
    s = A["every feature:C|g,F"]
    say(f"  {'every feature together':<42}{'':<36}{'':<36}{r(s['new'])} | {s['old']['mean']:+.3f}")
    say()
    say("A4. What comes after T: C(T)'s partial correlation with W, given g and")
    for name, _, label in AFTER:
        s = A[f"after {name}:C|g,X"]
        say(f"  {label:<58}{r(s['new']):<30}{r(s['old'])}")
    row("W = a + b_C C(T) + b_E E + g terms: b_C", "b_C")
    row("W = a + b_C C(T) + b_E E + g terms: b_E", "b_E")
    row("partial correlation of E with W, given g and C(T)", "E_W_r", r)
    row("E = a + a_C C(T) + g terms: a_C (the early link)", "a_C")
    row("W against the mean C over the same window, given g: slope", "same_window")
    row("  the same as a partial correlation", "same_window_r", r)
    say()
    say("B. THE 420 MOMENTS, UNTOUCHED COPY AT T (W of the untouched continuation; each start's means taken out;")
    say("   scaled weights: W's spread per spread of the variable; error bars from redrawing the starts)")
    for name, entries in res["B"].items():
        say(f"  {name:<40}" + "   ".join(f"{label} {sc(x)}" for label, x in entries.items()))
    say()
    say("C1. GROWTH AGAINST BOTH CONTROLS (per moment, then each start's average; S = (down - up) / 2,"
        " P = (up + down) / 2 - sham)")
    for name in ("immediate", "delayed"):
        span = "(T, T + 200]" if name == "immediate" else "(T + 200, T + 500]"
        say(f"  {name} {span}")
        say(f"    {'':<14}{'dense':<30}{'sparse':<30}{'pooled':<30}sparse - dense")
        for key, v in res["C1"][name].items():
            say(f"    {key:<14}" + "".join(f"{g(v[m]):<30}" for m in ("dense", "sparse", "pooled"))
                + g(v["sparse_minus_dense"]))
    say("  in 100-unit steps after T (pooled over the methods; dense | sparse)")
    say(f"    {'':<14}" + "".join(f"{step:<34}" for step in ("0-100", "100-200", "200-300", "300-400", "400-500")))
    for key in ("up - sham", "down - sham", "sham - none", "S", "P"):
        cells = []
        for step in ("0-100", "100-200", "200-300", "300-400", "400-500"):
            v = res["C1"][step][key]
            cells.append(f"{v['pooled']['mean']:+.5f} ({v['pooled']['bars']:+.1f}) "
                         f"{v['dense']['mean']:+.4f}|{v['sparse']['mean']:+.4f}")
        say(f"    {key:<14}" + "".join(f"{c:<34}" for c in cells))
    say()
    say("C2. THE STATE AFTER THE CHANGE (pooled; dense | sparse means)")
    for key, label in (("C", "coherence C"), ("align", "alignment cos(psi_A - psi_B)"),
                       ("cross", "cross term Re(z_A conj z_B)")):
        say(f"  {label}")
        for name, v in res["C2"][key].items():
            cells = [f"{span}: {g(v[span]['pooled'], 5)} {v[span]['dense']['mean']:+.5f}|{v[span]['sparse']['mean']:+.5f}"
                     for span in ("at T", "immediate", "delayed")]
            say(f"    {name:<17}" + "   ".join(cells))
    say()
    say("   Slopes in one unit, growth per unit of coherence: natural (untouched runs, given g) against induced")
    say("   (arms against each other: the growth difference over the coherence difference, start by start)")
    for name, v in res["slopes"].items():
        say(f"    {name:<42} natural {g(v['natural']['new'], 4)}   induced: dense {g(v['induced']['dense'], 4)},"
            f" sparse {g(v['induced']['sparse'], 4)}, pooled {g(v['induced']['pooled'], 4)}")
    say()
    say("C3. MOMENT BY MOMENT WITHIN EACH METHOD (each start's means taken out; weights ± error bars from redrawing)")
    c3 = res["C3"]
    say("  a. S against the up/down difference in |z2|, |z3|, |z4| right after the change, h_n = (|z_n|_down -"
        " |z_n|_up) / 2, with C(T)")
    for k in (2, 3, 4):
        v = c3["a"]["h_levels"][f"h{k}"]
        say(f"     h{k} level: dense {g(v['dense'], 5)}, sparse {g(v['sparse'], 5)}")
    for method in METHODS:
        say(f"     {method}: " + "; ".join(f"{label} {w(x, 3)} (scaled {sc(x)})"
                                        for label, x in c3["a"]["per_method"][method].items()))
    say(f"     both methods: sparse rows {w(c3['a']['pooled_without']['sparse rows'])} without h;"
        f" {w(c3['a']['pooled_with']['sparse rows'])} with h2, h3, h4")
    say("  b. sham - none (delayed) against the sham's own change in |z_n|, q_n = |z_n|_sham - |z_n| before, with C(T)")
    for k in (2, 3, 4):
        v = c3["b"]["q_levels"][f"q{k}"]
        say(f"     q{k} level: dense {g(v['dense'], 5)}, sparse {g(v['sparse'], 5)}")
    for method in METHODS:
        say(f"     {method}: " + "; ".join(f"{label} {w(x, 3)} (scaled {sc(x)})"
                                        for label, x in c3["b"]["per_method"][method].items()))
    say("  c. S against the coherence difference left in the delayed window, (C_up - C_down) / 2, with C(T)")
    say("     (if coherence in the window slowed growth, the weight would be positive)")
    for method in METHODS:
        say(f"     {method}: " + "; ".join(f"{label} {w(x, 4)} (scaled {sc(x)})"
                                        for label, x in c3["c"]["per_method"][method].items()))
    say(f"     both methods: C left {w(c3['c']['pooled']['C left (delayed)'], 4)}")
    say(f"  d. S against the achieved change (if coherence were the cause at the benchmark's size: {bench:+.4f})")
    for method in METHODS:
        x = c3["d"]["per_method"][method]["achieved change"]
        say(f"     {method}: {w(x, 4)}")
    say(f"     both methods: {w(c3['d']['pooled']['achieved change'], 4)}")
    say()
    say("C4. THE EARLY GROWTH AS A CARRIER (delayed effect against the sham: seen, and what the untouched runs'")
    say("    regression W = a + b_C C(T) + b_E E + g terms would give from each arm's own early effect)")
    say(f"    {'':<14}{'early effect':<30}{'delayed, seen':<30}{'b_E x early':<30}{'+ b_C x change in C':<30}"
        "seen - (b_E early + b_C change)   moment by moment: delayed on early")
    for name, v in res["C4"].items():
        say(f"    {name:<14}{g(v['early']):<30}{g(v['seen']):<30}{g(v['rebound_only']):<30}{g(v['with_C']):<30}"
            f"{g(v['seen_minus_with_C']):<33}{w(v['moment_slope'], 3)}")
    say()
    say("Instrument limits (not saved, so not analysed): the higher harmonics over time and in the base runs (only")
    say("|z2|, |z3|, |z4| magnitudes at T and right after the change, not their phases relative to z1); each")
    say("oscillator's phase and natural rate (so no frequency-resolved or cluster measures); the gap's direction.")
    say("Changes after the first run: " + ("; ".join(res["changes_after_first_run"]) or "none"))
    return lines


# ---------------------------------------------------------------------------------------------------------------------
# the figure

def figure(res):
    import matplotlib
    matplotlib.use("Agg")
    import matplotlib.pyplot as plt

    colors = {"dense": "#377eb8", "sparse": "#e66101"}
    styles = {"up - sham": "-", "down - sham": "--", "sham - none": ":"}
    fig, axes = plt.subplots(2, 2, figsize=(16, 11.5))
    fig.subplots_adjust(left=0.07, right=0.985, top=0.885, bottom=0.11, hspace=0.38, wspace=0.2)
    fig.suptitle("Cause test, exploratory follow-up (October 12, 2026): EXPLORATORY, chosen after the FAIL was known;"
                 " not a test\nN = 4,000 per group, K = 1.90; 42 new starts, 420 moments. The decision stays FAIL as"
                 " registered.", fontsize=13)

    ax = axes[0, 0]
    centers = np.array([(a + b) / 2 for a, b in res["profile"]["bins"]])
    ax.axvspan(200, 500, color="#fff3c4", zorder=0)
    for k, method in enumerate(METHODS):
        for key, ls in styles.items():
            ax.plot(centers, np.array(res["profile"][key])[k], color=colors[method], ls=ls, lw=1.8,
                    label=f"{method}: {key}")
    ax.axhline(0, color="0.5", lw=0.9)
    ax.set(xlim=(0, 500), xlabel="time after the change", ylabel="difference in growth rate (25-unit bins)",
           title="Growth: each arm against its controls (delayed window shaded)")
    ax.legend(fontsize=8, ncol=2, loc="upper right")

    ax = axes[0, 1]
    t = np.arange(501)
    ax.axvspan(200, 500, color="#fff3c4", zorder=0)
    for k, method in enumerate(METHODS):
        for key, ls in styles.items():
            ax.plot(t, np.array(res["paths"]["C"][key])[k], color=colors[method], ls=ls, lw=1.6,
                    label=f"{method}: {key}")
    ax.axhline(0, color="0.5", lw=0.9)
    ax.set(xlim=(0, 500), xlabel="time after the change", ylabel="difference in coherence C",
           title="Coherence: each arm against its controls")
    ax.legend(fontsize=8, ncol=2, loc="upper right")

    ax = axes[1, 0]
    A = res["A"]
    short = dict(past100="C over\nlast 100", past500="C over\nlast 500", trend="C trend", align="alignment",
                 cross="cross\nterm", imbalance="imbalance", turning="turning\nrate")
    later = {"E": "after T:\nearly growth", "C after": "after T:\nlater C", "all": "after T:\nall three"}
    keys = ([("link", "nothing\nadded")] + [(f"{key}:C|g,F", short[key]) for key, _ in FEATURES]
            + [("every feature:C|g,F", "every\nfeature")] + [(f"after {name}:C|g,X", later[name]) for name, _, _ in AFTER])
    for x, (key, label) in enumerate(keys):
        s, o = A[key]["new"], A[key]["old"]
        ax.errorbar(x - 0.1, s["mean"], yerr=s["error_bar"], fmt="D", ms=7, color="#4d4d4d", capsize=3)
        ax.plot(x + 0.15, o["mean"], "D", ms=6, mfc="white", mec="0.5", mew=1.6)
    ax.axhline(0, color="0.5", lw=0.9)
    ax.axhline(A["link"]["new"]["mean"], color="#4d4d4d", lw=0.8, ls=":")
    ax.set_xticks(range(len(keys)))
    ax.set_xticklabels([label for _, label in keys], fontsize=7.5)
    ax.plot([], [], "D", color="#4d4d4d", label="new 42 starts (± error bar)")
    ax.plot([], [], "D", mfc="white", mec="0.5", label="old 18 starts")
    ax.set(ylabel="partial correlation of C(T) with W", title="Untouched runs: C(T)'s link to the delayed growth W,"
           " given the earlier growth g and each added feature")
    ax.legend(fontsize=8, loc="lower right")

    ax = axes[1, 1]
    names = list(res["C4"])
    for x, name in enumerate(names):
        v = res["C4"][name]
        for dx, key, marker, color, label in ((-0.2, "seen", "o", "k", "delayed effect seen"),
                                              (0.0, "rebound_only", "s", "#1b9e77", "b_E x early effect"),
                                              (0.2, "with_C", "^", "#7570b3", "+ b_C x change in C")):
            ax.errorbar(x + dx, v[key]["mean"], yerr=v[key]["error_bar"], fmt=marker, color=color, ms=7, capsize=3,
                        label=label if x == 0 else None)
    ax.axhline(0, color="0.5", lw=0.9)
    ax.set_xticks(range(len(names)))
    ax.set_xticklabels([f"{name}\nagainst sham" for name in names])
    ax.set(ylabel="delayed growth difference, (T + 200, T + 500]",
           title="Delayed effects seen, against what the early growth would carry\n(untouched runs' regression applied"
                 " to each arm's own early effect)")
    ax.legend(fontsize=8.5, loc="best")

    for y, text in ((0.05, "Exploratory: every analysis was listed before it was run, on saved results only (no new"
                           " simulation). Error bars from the starts, as in the cause test."),
                    (0.025, "A toy model: this shows structure for choosing the next question, not proof about anything"
                            " beyond it.")):
        fig.text(0.07, y, text, fontsize=9.5, color="0.25")
    fig.savefig(FIGURE, dpi=110)


if __name__ == "__main__":
    main()
