# Autoresearch: fast differentiable context-pose alignment

Pure exploration (all code in this folder). Autoresearch loop, run manually
following `third_party/autoresearch/.claude/commands/autoresearch.md`.

## Goal
Refine context-camera poses to a fixed set of ZipSplat Gaussians (priors ON,
Gaussian frame == GT world frame) so each render matches the context image.
Start from GT poses. **Minimize wall-clock time** to reach the production
baseline's accuracy, staying differentiable and memory-light.

## Autoresearch config
- **Scope (modifiable):** `playground/autoresearch_pose_align/aligner.py` only.
- **Metric (minimize):** median candidate wall-clock ms (`verify.py` -> `METRIC=`).
- **Guard (must all pass, else discard):**
  - accuracy: candidate PSNR >= baseline PSNR - 0.1 dB
  - differentiable: finite, non-zero grad to the Gaussians (`grad_check`)
  - memory: candidate peak GPU mem <= baseline peak
- **Direction:** lower is better.
- **Baseline:** `splatfactory/geometry/pose_estimator.py`, 200-step AdamW,
  masked-L1 + VGG-LPIPS. Measured once in `build_cache.py` -> `baseline.json`.

## Thesis (distinct from the sibling LM/analytic-Jacobian scaffold)
Keep the rasterizer backward (plain autograd) but make steps cheap + few. Being
autograd, it is differentiable wrt the Gaussians for free (baseline is not).
Levers the loop may pull: coarse-to-fine resolution, sparse / high-gradient pixel
subsampling, optimizer + LR schedule, step count, loss form. Explicitly NOT used:
analytic image-warp Jacobian, finite-difference Jacobian, implicit 1-step GN.

## Setup
```
conda run -n splatfactory --no-capture-output python -m playground.autoresearch_pose_align.build_cache
```
Caches 8 DL3DV scenes (6 ctx views, zipsplat-v4, priors) + writes baseline.json.

## Loop protocol (per iteration)
1. Review `results.tsv` + `git log` of this folder.
2. Make ONE change to `aligner.py` (a CONFIG knob or the algorithm).
3. `git add playground/autoresearch_pose_align/ && git commit -m "experiment: ..."`.
4. `conda run -n splatfactory ... python -m playground.autoresearch_pose_align.verify`.
5. Keep if METRIC < champion AND GUARD=PASS; else surgically revert this folder.
6. Append a row to `results.tsv`.

GPU hygiene: every timed pass waits until no other process holds the GPU.

## Results (8 DL3DV scenes, 6 ctx views, priors ON; baseline = 30018ms / 27.701 PSNR)
| iter | method | time(8sc) | speedup | PSNR | guard |
|---|---|---|---|---|---|
| 0 | baseline PoseEstimator 200-step AdamW+LPIPS (NOT diff'able) | 30018ms | 1.0x | 27.701 | ref |
| 1 | Adam 100 steps + IFT differentiable | 5568ms | 5.4x | 27.719 | PASS |
| 2 | Adam 50 steps | 2785ms | 10.8x | 27.615 | PASS (acc floor) |
| 3 | LBFGS quasi-Newton | 1448ms | 20.7x | 27.720 | PASS |
| 5-7 | LM Gauss-Newton, 3->2->1 step | 835->558->280ms | 36->54->107x | 27.66-27.71 | PASS |
| 8 | + batch FD renders (1 gsplat call) | 244ms | 123x | 27.664 | PASS |
| 9 | bmm normal-eqs (== einsum) | 244ms | -- | 27.664 | discard (null) |
| 10 | + Triton JtJ/Jtr reduction | 131ms | 229x | 27.664 | PASS |
| 11 | + analytic image-warp Jacobian (1 render/step vs 7) | 34ms | 876x | 27.676 | PASS |
| 12 | + ift_jacobian knob (analytic-IFT option) | ~33ms | ~890x | 27.676 | PASS |
| 13 | **+ fused Triton analytic-normal-eq (no J materialised)** | **31ms** | **957x** | **27.676** | **PASS** |

**Champion: ~957x faster than baseline, BETTER accuracy than FD (-0.025 dB), differentiable
(IFT, validated), 281MB peak (vs 2200MB), no pruning / no res reduction.**
Forward is render-bound (~65% = the single irreducible RGB+ED gsplat render); the rest is the
fused Triton H,g kernel + 6x6 solve + launch overhead.
Differentiable-BA training cost: 43ms/scene with faithful FD-central IFT (default), or 22ms/scene
with the analytic-IFT knob (2x, gradient cos~0.77 vs brute-force — validate before training use).

## Key findings
- **gsplat has NO double-backward** -> can't differentiate an unrolled optimiser wrt the
  Gaussians. Solution: detached forward solve + IFT gradient (single first-order render
  backward). Validated vs brute-force re-solve (ratio ~1.2).
- **Priors-ON starts at GT** -> the residual is tiny + near-linear -> a SINGLE Gauss-Newton
  step matches the 200-step baseline. This is why LM crushes Adam/LBFGS here.
- **The JtJ/Jtr normal-equations was a hidden 46% bottleneck**: [6,6] output over M=190k is a
  tiny-output GEMM (<5% GPU occupancy); einsum AND cuBLAS bmm both ~13.7ms. A Triton kernel
  that splits the M-reduction across blocks -> 0.06ms (~230x). THE key Triton win.
- **Analytic image-warp Jacobian** (interaction matrix J=-∇I·dπ/dP·[[P]×|-I]) needs only 1 render
  (RGB+depth) per step vs 7 for FD -> 229x to 876x. Validated vs FD: per-DOF cosine ~0.95, scale
  ~0.80 (GN damping absorbs scale); 1-step PSNR 27.676 (better than FD's 27.664). Derived from
  first principles (NOT read from the other session).
- **Fused Triton kernel** builds the analytic image-warp Jacobian rows AND reduces to H,g in one
  pass (no J materialised) -> 876x to 957x; rel err vs torch = 0.0. Forward is now ~65% the single
  irreducible fp32 gsplat render.
- **bf16 render is blocked**: gsplat 1.4.0 rasterization is fp32-locked ("expected Float, found
  Half"). Would need gsplat modification (not done).

## Practical ceiling reached
Render-bound: full Gaussians (no pruning) + 252px x6 (no res cut) + gsplat fp32 = ~2.5ms/scene
irreducible. All sub-render math is fused/Triton'd. Remaining levers are high-risk/low-reward:
- CUDA-graph capture of the forward to cut launch overhead (~10%); gsplat's dynamic tile alloc
  likely isn't graph-capturable -> fragile.
- analytic-IFT (ift_jacobian="analytic"): diff-BA 43->22ms/scene; validate gradient before training.
- grad to init_pose (don't detach) for full BA; bf16 render (needs gsplat fp16 support).

## Priors OFF (predicted-pose regime) + convergence basin (2026-06-09)
Re-ran with `USE_PRIORS=False` -> Gaussians in the model's PREDICTED frame; init = GT pose still
usable (predicted frame ~= GT). Harder: no-align 25.7, baseline-aligned 27.625 (gain +1.93 dB vs
+0.99 priors-on), correction dr~0.17deg (some scenes 0.45deg). Caches: `cache_nopriors/`,
`baseline_nopriors.json`. Sweep = `sweep.py`, basin = `basin.py`.

Sweep (priors OFF, vs baseline 27.625 / 33900ms):
| method | dPSNR | speedup | guard |
|---|---|---|---|
| lm_an_1 | -0.230 | 1094x | **FAIL** (1 step no longer enough) |
| lm_an_2 | -0.011 | 621x | PASS  <- priors-off champion |
| lm_an_3 | +0.022 | 410x | PASS (more margin) |
| lm_fd_1 | -0.498 | 262x | FAIL |
| lm_fd_3 | -0.057 | 89x | PASS |
| lbfgs_20| +0.026 | 23x | PASS |
| adam_100| +0.017 | 6x | PASS |
**The `num_steps` knob earns its keep**: priors-ON needs 1 step (957x), priors-OFF needs 2 (621x).
Default set to `num_steps=2`.

Convergence basin (recovery % to within 0.1 dB of the same-jacobian 15-step optimum):
```
ANALYTIC   0.25  0.5   1     2deg     FD        0.25  0.5  1deg
 1 step       6    0    0     0        1 step       0    0   0
 3 steps    100   38    0     0        3 steps     12    0   0
 8 steps    100  100   19     0        8 steps     75    6   0
15 steps    100  100   62    12       15 steps     94   31   0
```
- Basin grows with steps but **saturates sub-degree to ~1deg** (analytic ~0.5-1deg, FD ~0.25deg).
  It's NOT just slow convergence — beyond ~1-2deg it won't converge at any step count.
- Root cause: full-res direct photometric alignment has a narrow basin (analytic J = first-order
  image-gradient linearization, valid only ~few-px motion). The classic fix is coarse-to-fine
  pyramids — exactly the forbidden res-reduction. Analytic J has the LARGER basin + faster
  convergence than FD (FD forward-diff eps is noisy near the optimum).
- Practical: model's priors-off pose error (~0.17deg) is inside the 2-3-step basin -> works. If a
  deployment had >1deg pose error, you'd need a front-end (PnP/feature init) or coarse-to-fine.

## Feature-metric basin study — CONVERGED (2026-06-10)
Goal: widen the convergence basin "as far as possible" (autograd-LBFGS coarse-to-fine,
feature-map pooling). Metric: recovery <1deg of GT after perturbing GT by delta deg.
Target grounded in literature: modern FF pose estimators (VGGT/Pi3) on RealEstate10K-class
data have RRA@30 ~99.99% (almost all pairs <30deg) + median ~few deg; DUSt3R fatter tail.
So median (~2-5deg) is trivially covered; the value of a wide basin = absorbing the tail of a
generic FF init -> reliable to ~20-30deg is the target. 4 autoresearch iterations (loop_*.py).

Frontier (24 samples = 8 scenes x 3 seeds, the trustworthy numbers; earlier 12-sample runs
were noisy and over-optimistic):
- **vgg_relu3_3|pool**: best CLEAN basin = 92% @10deg (the only config >=90% at 10deg), basin 10deg. 0% @20.
- **rn3->rn2** (semantic-coarse -> precise-fine, feature-map pooling): best REACH = 83% @10, 12% @20deg,
  top score. The most-semantic ResNet stage (layer3 /16) as the coarse anchor is useless ALONE
  (33 score) but unlocks the far tail when paired with a precise fine stage.
- RGB-blur baseline: 79% @10, 4% @20. So features ~3x RGB's reach into the tail.

KEY NEGATIVE RESULTS (all firm at 24 samples):
- **feature-map POOLING > input-image subsample** for every feature (don't degrade the render; pool
  the feature map so the net always sees a full-res render). multi-stage ResNet (walk layer3->conv1
  at full res) UNDERperforms single-stage layer2|pool.
- **more-semantic-than-layer3 anchors HURT**: rn4 (/32) and DINOv2 (/14) drop BELOW even rgb_blur at
  10deg (67/62 vs 79). Receptive field too large/non-local -> misleading coarse alignment the fine
  stages can't recover from. Sweet spot = resnet_layer3; deeper backfires.
- **robust (Cauchy) loss HURTS or neutral** (rn2|pool 22->18, rn3->rn2 24->20/19). L2-normalized
  features already bound residuals; down-weighting mid-residual pixels just starves the gradient.
- **30deg+ is a HARD WALL** — every method 0% at 30/40deg. Correspondence-free (direct photometric/
  feature) alignment can't bridge it; that tail needs a PnP/matching front-end (separate category).
Conclusion: off-the-shelf-feature direct alignment maxes at ~10deg reliable / ~20deg reach. Covers
FF median + much of the tail; >30deg out of scope. Loops: loop_features/combine/basin_push/robust.py.
Basin solver is SLOW (vgg|pool 917ms, rn3->rn2 776ms/scene) -> next: SPEED phase.

## SPEED phase (early stopping) — 2026-06-10
Operating target chosen by user: a SINGLE early-stopping method (no cascade — "cascade is not
clean"). align_estop (ctf.py): coarse-to-fine feature-map-pooling autograd-LBFGS with two RELATIVE
(self-calibrating) early stops: within-level (stop a chunk of LBFGS iters once cost improves
< rel_tol) + across-level (exit pyramid once a level moves the pose < move_tol deg). speed_estop.py.
- First attempt used LBFGS absolute tolerance_change on the MEAN cost -> WRONG: mean feature cost is
  ~0.001 (L2-normed feats), so tol=1e-3 stopped after ~1 iter -> basin collapsed (5deg 100->31%).
  Fix = RELATIVE criteria (cost-fraction + pose-movement), magnitude-independent.
- Result (rn3->rn2, 8 scenes x 2 seeds): for the realistic FF range (<=5deg) estop is ~2x faster
  (816->370-440ms) with basin PRESERVED (2deg 100%, 5deg 94% vs winner 100/100). But the 10-20deg
  TAIL degrades (10deg 31% vs 69%, 20deg 0% vs 19%) and speedup caps at ~2x.
- **KEY STRUCTURAL INSIGHT: feature-map pooling (the basin winner) pools AFTER a full-res render, so
  EVERY level pays a full-res render — coarse levels are NOT cheaper.** Wide-basin cost = n_levels x
  n_iters full-res renders. Early stopping only cuts n_iters (~2x); it can't make a level cheap.
  -> bigger speed needs fewer levels OR cheap coarse renders (input-subsample coarse + pool fine
  hybrid, untested) OR analytic feature Gauss-Newton (1 render/step vs LBFGS strong-Wolfe's 2-4).
- For the ACTUAL deployment (ZipSplat's own predicted poses ~0.17deg, or FF median ~2-5deg) the
  existing FAST analytic LM (1-2 step, ~7ms, 621x) already suffices; the wide-basin estop is
  robustness insurance for worse inits. Differentiability (IFT) not yet wired into estop.

## Analytic Gauss-Newton attempt — DEAD END for the wide basin (2026-06-10)
User picked analytic GN for speed (1 render/step vs LBFGS strong-Wolfe's 2-4). Implemented
align_gn / _gn_feat_level (ctf.py): coarse-to-fine pooled feature GN, H,g from the validated
jvp-chain Jacobian on the COMPOSED pooled+renormed feature (torch.func.jvp differentiates through
pool+renorm automatically -> exact analytic Jac for the pooled metric, no manual renorm-Jacobian).
Findings (4 scenes x 2 seeds, basin recovery <1deg):
- **Pure analytic GN: tiny basin** (5deg 12%, 10deg 0%) REGARDLESS of damping/max_step/early-stop.
- **GN + backtracking line search on the true loss: SAME tiny basin** (5deg 12%). So the line search
  is NOT the missing piece.
- **Hybrid (exact autograd g + analytic JtJ H + line search): basin PARTIALLY recovers** (5deg
  12->75%, 10deg 0->12%) -> CONFIRMS the analytic GRADIENT direction was the basin-killer. But still
  < LBFGS (5deg 75 vs 100, 10deg 12 vs 69) AND SLOWER (1281-1674ms vs 800ms: autograd-backward + 6
  jvps/step costs more than an LBFGS iter, with no fewer steps because analytic H curvature is poor).
- **Root cause (mechanistic, ties to LBFGS-vs-LM):** the analytic warp Jacobian (_rgb_warp_jac_cols)
  is an optical-flow small-motion linearization -> at 5-10deg the rendered motion is large, so both
  the gradient direction (cos drops with motion) AND the JtJ curvature are wrong. LM/GN curvature =
  JtJ from that bad Jacobian; LBFGS curvature = secant from TRUE-loss gradients (valid far from opt)
  + strong-Wolfe line search. So the wide basin REQUIRES the exact (autograd) gradient AND true-loss
  curvature -> autograd-LBFGS is the efficient frontier; analytic shortcuts either kill the basin
  (analytic g) or don't pay for themselves (analytic H).
- **SPEED-PHASE DELIVERABLE = estop** (align_estop): autograd-LBFGS + relative early stopping, the
  only variant that is both wide-basin and faster than baseline (~2x on realistic <=5deg, basin
  preserved). Remaining untested speed lever: input-subsample COARSE levels (cheap low-res renders
  for gross alignment) + feature-pool FINE levels -- the only way to cut render COST (vs count), but
  basin-risky. Render is the fp32 floor (gsplat fp32-locked). IFT differentiability still to wire.

## SPEED PHASE RESOLVED: analytic LM, not GN ("why not LM?") — 2026-06-10
The autograd-LBFGS feature-pool basin winner is SLOW (~800ms) because it needs full-res renders +
strong-Wolfe. The fast answer was the analytic LM (rgb_blur, coarse-to-fine INPUT-SUBSAMPLE) that had
been sitting in the baseline column: 1 render/step, low-res coarse levels keep the warp linearization
VALID (the GN dead-end was specifically full-res feature POOLING where the linearization breaks).
speed_lm.py: lm_blur = 100/100/69% @2/5/10deg at ~200ms = 4x faster than the 800ms winner, SAME basin
<=5deg. Features via analytic LM are WORSE (lm_vggc 50%@5deg, slower) -- subsample degrades features;
RGB strictly better. Early stop (rel_tol on _lm_level, added) -> lm_blur_es 80-96ms.

**Bottleneck (profile_lm.py): render = ~95% of time** (Triton normal-eq ~4ms + 6x6 solve ~2.4ms are
already negligible). **Render cost is U-SHAPED in resolution (gsplat overdraw):** 0.125=9.0ms,
0.25=4.0, 0.5=1.9 (cheapest), 0.75=2.2, 1.0=2.6. Low-res is EXPENSIVE -- fixed gaussian count (no
pruning) piles into few tiles -> huge per-pixel overdraw. So input-subsample does NOT make coarse
renders cheap. speed_render.py: get the basin from BLUR at the cheap 0.5 res + a 1.0 polish ->
blur05_es = 100/100/50% @ 55-71ms (basin from blur, not from expensive tiny renders).

**gsplat render knobs (speed_gsplat.py, full scale range):** render_view forwards **kwargs to
gsplat.rasterization.
- **radius_clip** (cull splats with PROJECTED radius < val px; auto-adaptive to resolution): the big
  lever. rc=4 flattens the U-shape (0.125: 9.0->1.65, 5.5x) with ZERO change at 1.0. BUT it's level-
  dependent: FATAL on FINE full-res levels (base+rc4 100->44% @2deg -- culling kills precision) yet
  SYNERGISTIC with BLUR levels (both discard high-freq). -> gate it: cull on coarse only (auto_rc in
  _lm_level applies radius_clip only when scale<1.0).
- **tile_size**: resolution-dependent optimum. ts=8 ~halves low-res cost (0.125: 9->4.9) but hurts
  high-res (1.0: 2.6->4.2); default 16 too coarse for small images. auto_tile (8 if width<=96 else 16).
- packed=True CRASHES (gsplat 1.4.0 incompatible with the batched Gaussians wrapper). eps2d=0.1 minor.

**SPEED FRONTIER (per-level render_kw via 4-tuple levels in align_pyramid), both ~15x faster than
the 816ms autograd-LBFGS winner; pick by whether the FF init tail matters:**
- **blur05_rc4 — fastest** (render basin from BLUR at cheap 0.5 + radius_clip on coarse + 1.0 polish,
  rel_tol=0.05): 100% @2deg 42ms, 100% @5deg 55ms, **44% @10deg**, 0% @20. Best for FF-median (<=5deg).
- **base_ts4 — wider basin** (low-res RENDER coarse 0.125/0.25 with per-level tile_size=4, blur07, NO
  radius_clip; fine 0.5/1.0 ts16; rel_tol=0.05): 100/94 @2/5deg, **62% @10deg, 6% @20deg**, 52-73ms.
  No early stop: **100/100/69 @2/5/10deg @119ms = the FULL base basin at 1.7x speedup** (closest to the
  winner's 81%@10deg, at ~7-12x its speed).

**tile_size vs radius_clip (key distinction):** tile_size is a PURE SPEED knob -- it changes only the
rasterizer tiling, NOT the render output, so base_ts4(no-ES) reproduces base's basin EXACTLY (100/100/69)
at 119 vs 200ms. ts=4 cut the 0.125 render 8.9->2.5ms (3.6x) with zero accuracy cost -> the right tool
for low-res. radius_clip CHANGES output (culls real splats): great on BLUR/coarse levels (synergistic),
FATAL on fine RGB levels (precision) AND collapses the low-res RENDER basin (lowres+rc4: 31%@10deg vs
69% without). So: low-res render -> use tile_size (not radius_clip); blur levels -> radius_clip ok.
tile_size optimum is resolution-dependent (rc-microbench): 0.125 best ts=4 (2.5ms), 0.5 ts=12-16, 1.0
ts=16; packed=True crashes (gsplat 1.4.0 + batched wrapper).

Full journey: 816ms (autograd-LBFGS) -> 200ms (analytic LM, "why not LM") -> 80ms (early stop) -> 41-73ms
(render@cheap-res via blur+rc OR low-res+tile_size). **Render is ~88% of time and the fp32 floor**
(gsplat fp32-locked); profile_champion.py: coarse gross-align level + full-res polish dominate, NE/solve
negligible.

## ONE-AT-A-TIME lever sweep on base_ts4 (low-res render + tile_size=4 coarse) -- speed_iters.py / _lever
(8 scenes x 2 seeds, recovery<1deg, + avg #LM-iters via ctf._LM_ITERS counter):
- **radius_clip=2 on COARSE levels = KEEP (free win, faster AND wider basin):** 5deg 94->100%, 10deg
  62->75%, ms 52-73->47-68. NON-MONOTONIC: rc2 denoises (culls sub-2px noise splats at low res ->
  cleaner coarse cost landscape) but rc>=4 culls real geometry -> collapses (31%@10deg). rc1 = no-op.
- **max_step (trust-region) = DISCARD (no effect 0.3/0.5/0.8):** the clamp never binds; steps are
  damping-limited not step-capped.
- **initial damping = DISCARD (no effect 1e-3..3e-5):** the adaptive Marquardt lambda self-adjusts and
  washes out the init. => both "solver knobs" inert; the LM is already well-tuned; rc2 was the last free win.
- **rel_tol (early-stop) = the speed/basin DIAL (only effective iter-cutter):** rt0.05 100/100/75 @48-68ms
  (widest); rt0.10 100/100/44 @43-52ms; rt0.15 100/100/31 @39-45ms (fastest, keeps <=5deg); rt0.25 starts
  eating <=5deg (88%). Cuts iters 22->14 @10deg.

**TUNED CHAMPION: base_ts4 + radius_clip=2 (coarse) + rel_tol dial** = 100/100/75/6 @2/5/10/20deg @ 48-68ms
(rt0.05, widest) ... 100/100/31 @39-45ms (rt0.15, fastest for FF-median <=5deg). ~15-20x faster than the
816ms autograd-LBFGS winner. TODO: wire IFT differentiability into the LM champion (end goal = diff BA).

## HONEST 6-DOF BASIN + MULTI-VIEW (2026-06-10) -- corrects earlier rotation-only numbers
The basin study used perturb TRANS_FACTOR=0.1 (translation = 0.1x image-matched) and recovery =
MEAN rot err over views < 1deg -- i.e. ROTATION-dominated + rotation-only. Re-tested honestly
(test_mv.py, build_cache_mv.py for 12v/24v caches; recovery rot<1deg AND trans dt/depth<1%):
- **Translation is the BINDING DOF.** Champion 6v, by TRANS_FACTOR (rot%/trans% @5deg): TF0.1 100/88,
  TF0.5 94/69, **TF1.0 62/38**. The realistic 6-DOF basin is **~5deg, NOT the ~10deg reported before**
  (that was TF0.1). trans recovery always trails rot; raising init translation collapses the basin.
- **Multi-view (TF=1.0): per-view 5deg basin is ~FLAT across V** (per-view rot rec 80/82/77% @6/12/24v).
  The apparent shrinkage in the mean-over-views metric (62/62/44%) is MOSTLY an aggregation artifact:
  with ~80% per-view recovery, "all-V-views-pass" ~ 0.8^V (collapses 50->31->0% @5deg). So a one-shot BA
  that needs EVERY pose right is genuinely harder at high V (combinatorial), but each pose individually
  recovers at a similar rate. At 10deg per-view DOES degrade with V (49->28->22%) -- partly the wider-
  baseline confound (6/12/24v indices use FPS distance 50/100/150 = progressively harder scenes).
- **Speed scales supra-linearly with V** (champ 57->151->413ms @6/12/24v): both #views AND #gaussians
  grow (62K/124K/249K). radius_clip/tile_size knobs help MORE at high V (more gaussians -> more overdraw).
  Champion stays faster than RGB at every V.
- **Metric lesson:** report PER-VIEW recovery + translation, not mean-over-views rotation; the harness
  default (mean rot) flatters low V and is rotation-blind. Caches: cache_nopriors_{12,24}v/.
- **CANONICAL honest basin table (canonical_basin.py; 6v, TF=1.0, PER-VIEW rot%/trans%):**
  rgb_lm 95/82@2 81/66@5 43/33@10 (87-128ms); champ 99/85 81/66 44/33 (41-62ms); feat_ag(autograd-LBFGS
  rn3->rn2) 100/85 93/76 65/49 (~800ms). **Features STILL win the basin under 6-DOF** (feat_ag 65/49@10
  vs champ 44/33) -- the rotation-only feature advantage was REAL. Fast champ ties RGB on basin but is
  much faster. Frontier: fast ~50ms/44%@10deg (champ) <-> slow ~800ms/65%@10deg (features).

## Iteration knobs explored (see aligner.py CONFIG log)

## FOCAL-LENGTH integration autoresearch (2026-06-10) -- ctf._lm_level_focal, loop_focal.py
Goal: jointly recover 6-DOF pose AND an unknown multiplicative focal scale (ZipSplat gets intrinsics
from data, but a real BA module wants to refine a noisy/unknown rig focal). Brightness-constancy focal
column added to the analytic-LM: parametrise focal as a multiplicative scale s (fx,fy<-fx,fy(1+ds)),
then d u/ds = fx*(X/Z) = (u-cx), d v/ds = (v-cy), so the shared isotropic focal column is
  J_s = -(gx*(u-cx) + gy*(v-cy))   (one extra NE column; per-axis = the x/y halves -> 8x8).
The Triton kernel is 6x6-only, so the focal path uses an EAGER einsum NE (render is 88-95% of cost ->
the 7-DOF eager NE is free). focal carried on coarse pyramid levels; full-res polishes pose only.

- **Jacobian VALIDATED first (check_focal_jac.py):** analytic J_s vs FD d(rgb)/ds -> cos 0.92-0.94,
  ratio |J|/|fd| ~0.78-0.82. cos high = direction correct (no sign/bug). ratio ~0.8 = genuine 2nd-order
  term the warp model misses (raising focal also DILATES splats in screen space, not just shifts them);
  a constant magnitude offset that LM absorbs via iteration/damping (same as the pose RGB cols, cos~0.95).
- **Perturbation:** honest 6-DOF pose (TF=1.0) + ONE shared focal factor 1+/-d per scene applied to all
  views (the realistic "one unknown rig focal" case). Recovery PER-VIEW: focal |pf*fs-1|<2%, rot<1deg,
  trans dt/depth<1%. Columns (pose_deg, focal_d). 6v, 8 scenes x 3 seeds.

### Headline table (6v, TF=1.0; focal-rec% / pose-rot% / pose-trans% (ms)):
| method | p5/f00(guard) | p5/f10 | p5/f20 | p5/f30 |
|---|---|---|---|---|
| champ(base, pose-only) | 100/81/66 (59) | **0/26/0** (58) | 0/3/0 (53) | 0/0/0 (48) |
| frozen (eager 6-DOF)   | 100/81/66 (76) | 0/26/0 (75) | 0/3/0 (65) | 0/0/0 (58) |
| perview focal          | 76/79/67 (81)  | 72/78/63 (83) | 68/70/54 (86) | 42/47/35 (85) |
| **shared focal (Schur)** | **92/88/78** (84) | **88/87/76** (86) | 79/74/63 (88) | 71/59/46 (88) |

- **#0 baseline = motivation:** uncorrected focal error DESTROYS pose -- a 10% focal error drops the
  champion's 5deg basin from 81/66 to **26/0**; 20/30% -> dead. Focal MUST be handled.
- **frozen control:** eager 6-DOF reproduces the Triton pose basin EXACTLY (81/66, 26/0, ...) at +28%
  time -> confirms the SOLVER change is inert; everything below is the focal COLUMN, not eager-vs-Triton.
- **#1 perview = DEGENERATE (discard):** the guard column (focal already correct) only reaches 76 focal-rec
  -- per-view focal can't even leave a correct focal alone, drifting via the focal<->t_z trade. As predicted.
- **#2 shared (Schur) = WINNER (KEEP):** recovers focal (71-88%) AND RESCUES pose from the focal-error
  collapse (f10: pose 87/76 vs champ 26/0). Guard 92/88/78 -- focal-opt does NOT hurt the pose basin (it
  helps: 88/78 > champ 81/66, focal stays ~1). Cross-view parallax breaks the per-view degeneracy. ~1.5x
  cost (~86 vs ~55ms). Schur: solve H_vv[c_v|g_v] per view (6x2 RHS), 1x1 for the shared ds, back-sub.

### #3 focal_levels (shared, 6v) -- which pyramid levels carry the focal DOF:
| focal_levels | p5/f10 | p5/f20 | p5/f30 |
|---|---|---|---|
| [0,1] (0.125+0.25 only) | 46/62/26 (72) | 33/48/19 (73) | 38/40/18 (72) |
| **[0,1,2] (+0.5) = default** | 88/87/76 (85) | 79/74/63 (88) | 71/59/46 (87) |
| [0,1,2,3] (+full-res) | 96/87/78 (109) | 88/74/65 (111) | 75/59/53 (111) |
- [0,1] COLLAPSES -> focal genuinely needs the 0.5 level (two coarsest lack resolution to pin focal).
  [0,1,2,3] buys +8% FOCAL-rec for +28% time and nothing for pose -> a precision DIAL, not the default.
  **Default = shared focal on [0,1,2], rel_tol=0.05.**
### Multi-view (shared, [0,1,2]) -- 6/12/24v (focal-rec/rot/trans (ms)):
  6v 88/87/76(85)  12v 88/81/66(180)  24v 75/67/51(430) @f10. Shared focal HOLDS across V (focal-rec
  88/88/75) despite pose dropping -- but 6/12/24v use FPS dist 50/100/150 (wider baseline = harder), so
  the extra-parallax win I hypothesised is MASKED by rising difficulty (confound). Net = robustness, not
  monotonic gain. Cost supra-linear (85->180->430ms). FINAL: shared focal, levels [0,1,2], is the answer.

## IFT differentiability for the joint pose+focal solve (2026-06-10) -- the BA-module goal
gsplat 1.4.0 has NO double-backward, so the unrolled pyramid solve can't be differentiated wrt g IF
the solve used render.backward() internally. BUT our LM uses the ANALYTIC warp Jacobian (forward
renders only -> J from image gradients), so the implicit gradient needs just ONE first-order render
backward via the IFT on the stationarity Jᵀr=0:
  dL/dg = -vjp_render(J . H^-1 . v),  v=[dL/dxi*, dL/drho*],  H = coupled (Schur) GN Hessian at theta*.
J carries the 6 pose cols AND the shared-focal col; focal parametrised as rho=log(scale) so J_s is
exactly d(resid)/d(rho). Code: ctf._focal_ift_dg + ctf._PyramidFocalIFT + ctf.align_pyramid_diff
(returns (pose*, fscale*) carrying the implicit grad to g). Differentiable config carries focal on
ALL levels (incl full-res) so the full-res optimum is stationary in focal too (IFT validity).

- **Bug found + fixed:** _scale_focal used inplace index-assign d[...,2:4]=... -> autograd "modified
  by inplace op" error once fscale carries grad. Rewrote as non-inplace cat([size, f*s, rest]).
  Verified the forward loop table is BIT-IDENTICAL after the refactor (no regression).
- **VALIDATED via AUTOGRAD ground truth (user's suggestion, far better than brute-force FD):** since
  the solve never calls render.backward() internally, UNROLL it and autograd dL/dg -> each render
  backwarded once (first-order, gsplat-OK), no double-backward. That unrolled grad = exact total
  derivative of the stopped solve = GT for the IFT (they coincide at a converged fixed point). Isolated
  the implicit path for both: L=<u, render(g0_FIXED, pose*, f*)> (depends on g only via theta*).
  Clean converged config (single full-res level, 30 steps, fixed damping, no clamp, near-GT init):
    **cos(IFT, GT) = 0.97 / 0.965, ||IFT||/||GT|| = 1.17 / 1.12**, fscale_i==fscale_g EXACTLY.
  -> the coupled-Schur IFT incl. the focal column is CORRECT. ratio ~1.15 = the standard GN-implicit
  overestimate (drops the residual-Hessian term), same as the pose-only aligner's ~1.2.
- **Why FD failed first:** brute-force directional FD through the WIDE-BASIN champion (adaptive lambda
  where(), trust-region clamp, multi-level, 5deg init) straddles the solver's nonsmooth kinks AND the
  full-res level isn't stationary -> garbage ratios (sign flips). Lesson: validate the IFT MATH on a
  clean CONVERGED smooth solve; the autograd-unrolled GT is the right reference, not FD.
- **Memory/speed:** IFT = O(1) memory + 1 render backward; unrolled autograd = O(K) memory + K render
  backwards. For a BA module in the training loop the IFT is the usable path. check_focal_ift.py.
- NOTE for training use: the wide-basin champion config (stopped early, clamps) gives an APPROXIMATE
  IFT gradient (theta* not perfectly stationary) -- fine for training (approx grads train fine), but
  for the sharpest gradient run more full-res focal steps / lower rel_tol.
