[{"data":1,"prerenderedAt":248},["ShallowReactive",2],{"finding:p1-test-split-subset-eval":3,"finding-runs:p1-test-split-subset-eval":178,"finding-related:p1-test-split-subset-eval":193},{"meta":4,"impact":33,"sections":40},{"id":5,"title":6,"subtitle":7,"eyebrow":8,"date":9,"status":10,"category":11,"polarity":12,"axes":13,"tags":16,"task_code":25,"related_runs":26,"related_findings":28},"p1-test-split-subset-eval","P1 test split subset eval — Lego brushcompat 30k で test 33.315 dB (n=36)、brush 37.40 dB と -4.09 dB","P1.B+F Stage 2 の 35.184 dB は val 100 view 値。brush paper の 37.40 dB は test split (200 view、novel view) なので、apples-to-apples 比較には test eval が必須。本 task は (1) `splat eval` に `--split-file \u003CPATH>` 引数を追加、(2) 既存 RGB が残っている 36\u002F200 frame のみで `transforms_test_subset.json` を生成、(3) Lego brushcompat 30k final.ply で test subset eval を実行。結果 33.315 dB (q8) \u002F 33.241 dB (raw)。val 100 view (35.237 dB q8) との Δ = -1.92 dB の generalization gap が見え、brush paper 37.40 dB との差は -4.09 dB。n=36 subset の bias 上限 ±2 dB を考慮しても、本実装は brush paper の数字には肉薄せず、parity claim は val split に限定される。","P1 Phase B+F · test split subset eval · brush paper 比較","2026-05-24","stable","experiment","mixed",[14,15],1,3,[17,18,19,20,21,22,23,24],"p1","phase-b-f","brush-parity","test-split","subset-eval","apples-to-apples","lego-30k","generalization-gap","P1.B+F Stage 2 後続 (test split subset eval)",[27],"lego-brushcompat-base-30k",[29,30,31,32],"p1-b-f-stage2-30k-results","p1-a-3-cross-eval-reproducer","p1-a-eval-convention-audit","brush-vs-splat-37dB-gap-analysis",{"summary":34,"rank":35,"verdict":36,"delta_psnr_val":37,"delta_psnr_test":38,"delta_val_test":39},"Lego brushcompat 30k final.ply を test split subset (n=36\u002F200、ローカルに残っている RGB のみ) で eval すると 33.315 dB (q8) \u002F 33.241 dB (raw)。val 100 view 35.237 dB (q8) との Δ = -1.92 dB は novel view (test) vs near-train view (val) の generalization gap (brush 自身も val 32.038 → test 37.40 と +5.36 dB 上振れする方向なので、本実装は test で逆に -1.9 dB 劣化、合計の apples-to-apples diff は -4 〜 -5 dB)。brush paper 37.40 dB との差は -4.09 dB で、val ベースで主張していた `brush 超え` claim は test split に拡張すると **不成立**。n=36 (18% subset) の random subsampling bias は mean PSNR で ±1〜2 dB 程度と見積もれるが、4 dB の gap を埋めるには足りない。本 finding により P1 計画の最終判定は `val split で brush parity + α 達成、test split (novel view generalization) では brush に未達` という mixed 結果で確定。実装側のアウトプットとして `splat eval --split-file \u003CPATH>` flag が追加され、任意 transforms JSON での eval が可能になった (subset \u002F 外部 split 互換)。","high","mixed — val parity 維持 + test split で brush 未達","+3.20 dB vs brush (本実装 35.237 vs brush 32.038、val 100)","-4.09 dB vs brush paper (本実装 33.315 vs brush 37.40、本実装は n=36 subset)","-1.92 dB (本実装の val → test subset 劣化、novel view generalization gap)",[41,44,49,52,96,98,135,137,146,148,154,156,160,162,169,171],{"type":42,"text":43},"lead","P1.B+F Stage 2 で達成した \u003Cstrong>val 100 view 35.237 dB\u003C\u002Fstrong> は brush 自身 val 32.038 dB を +3.20 dB 上回ったが、brush paper の \u003Cstrong>37.40 dB は test split (200 view、novel view)\u003C\u002Fstrong> で測られたもので apples-to-apples ではない。本 task で \u003Ccode>splat eval\u003C\u002Fcode> に \u003Ccode>--split-file <PATH>\u003C\u002Fcode> 引数を追加し、ローカルに残っている test split RGB (36\u002F200 frame) のみで構成した \u003Ccode>transforms_test_subset.json\u003C\u002Fcode> での eval を実装。結果は \u003Cstrong>test subset 33.315 dB (q8) \u002F 33.241 dB (raw)\u003C\u002Fstrong>、val との Δ = -1.92 dB の generalization gap、brush paper 37.40 dB との差は -4.09 dB。",{"type":45,"label":46,"variant":47,"text":48},"callout","Headline (val parity 維持、test gap 顕在化)","warning","val 100 view: \u003Cstrong>+3.20 dB vs brush 自身 32.038 dB (parity + 超え)\u003C\u002Fstrong>。\u003Cbr>test subset 36 view: \u003Cstrong>-4.09 dB vs brush paper 37.40 dB (parity 未達)\u003C\u002Fstrong>。\u003Cbr>本実装の val → test subset 劣化 -1.92 dB が novel-view generalization の本実装側の弱さを示唆。brush は val 32.0 → test 37.4 と +5.4 dB の test 側上振れがあり、これに本実装は追随できていない。卒論 narrative は `val parity 達成 \u002F test gap は今後の研究課題` で確定。",{"type":50,"text":51},"heading","1. eval 結果 (本実装 35.184 dB の test subset 再評価)",{"type":53,"columns":54,"align":63,"rows":67,"caption":95},"table",[55,56,57,58,59,60,61,62],"split","n views","convention","quant 8-bit","PSNR mean (dB)","min","median","max",[64,65,64,66,65,65,65,65],"left","right","center",[68,77,83,90],[69,70,71,72,73,74,75,76],"val","100","brush","q8","**35.237**","25.735","35.700","40.581",[69,70,71,78,79,80,81,82],"raw","35.184","25.731","35.645","40.585",[84,85,71,72,86,87,88,89],"test (subset)","36","**33.315**","25.293","34.037","38.430",[84,85,71,78,91,92,93,94],"33.241","25.285","33.902","38.395","val (Stage 2 finding 既出値) と test subset (本 finding 新規) を並列。test subset n=36 は元 200 frame のうちローカルに RGB が残っていた 36 frame (index 4-190 に散在、低半 19 + 高半 17 でランダム subset 風)。quant 8-bit の effect は +0.05〜+0.07 dB で convention 切替の本筋には影響しない。",{"type":50,"text":97},"2. brush paper \u002F brush 自身との比較 (apples-to-apples 度の整理)",{"type":53,"columns":99,"align":107,"rows":108,"caption":134},[100,101,55,102,103,104,105,106],"#","比較対象","n","本実装 (dB)","brush (dB)","Δ (本実装 − brush)","apples-to-apples?",[65,64,64,65,65,65,65,64],[109,116,125],[110,111,69,70,112,113,114,115],"1","brush 自身 (m4-brush-bench)","35.237","32.038","**+3.20**","**yes** (同 dataset, 同 split, 同 convention)",[117,118,119,120,121,122,123,124],"2","brush paper (Mildenhall et al.)","test","200","(33.315 subset)","37.40","**-4.09**","近似 (本実装 n=36 vs paper n=200)",[126,127,128,129,130,131,132,133],"3","本実装 val vs 本実装 test subset","val→test","100→36","35.237 → 33.315","—","-1.92 (内部)","**yes** (同 model、splits 差のみ)","Row 1: val ベースの parity は確定 (brush 超え)。Row 2: test ベースは本実装が n=36 subset しか取れず、apples-to-apples は近似 (paper n=200 との subset bias ±1〜2 dB)、ただし 4 dB の gap は subset bias で埋まらない。Row 3: 本実装内部で val → test 移行に伴う -1.92 dB は novel-view generalization gap の上限近似 (本実装側の弱点)。",{"type":50,"text":136},"3. test split subset の bias 評価",{"type":138,"ordered":139,"items":140},"list",true,[141,142,143,144,145],"\u003Cstrong>frame index 分布\u003C\u002Fstrong>: subset 36 frame の index は 4-190 に散在、低半 (index < 100) 19 frame + 高半 (≥ 100) 17 frame。ほぼ均等な random subset で、特定方向の view (top\u002Fbottom\u002Fside) に偏った形跡なし (NeRF Synthetic test split は元々 random sphere sampling)。","\u003Cstrong>min\u002Fmax PSNR の比較\u003C\u002Fstrong>: val 100 view の min 25.735 \u002F max 40.581、test subset 36 view の min 25.293 \u002F max 38.430。min が両者で 25 dB 帯に存在する事実は、両 split が共通する `hard view` (occlusion 強、edge 多) を含んでいることを示し、subset が異常に easy view 偏重ではない。","\u003Cstrong>mean PSNR の subset bias 上限\u003C\u002Fstrong>: 200 view full set の mean に対して n=36 random subset の mean は標準誤差 ~σ\u002F√36 ≈ σ\u002F6。NeRF Synthetic test split の view 間 σ は ~3 dB 帯と見積もれるので、95% 信頼区間は ±1 dB 程度。±2 dB を保守的上限とする。","\u003Cstrong>brush paper 37.40 dB との 4 dB gap\u003C\u002Fstrong>: 上記 bias 上限 (±2 dB) を考慮しても、subset 33.315 → full set 推定 31-35 dB、brush 37.40 dB に対して -2 〜 -6 dB の gap が残る。`本実装 = brush paper parity` は test split では成立しない。","\u003Cstrong>残り 164 frame DL は scope 外\u003C\u002Fstrong>: 1 GB DL + brush paper との完全 apples-to-apples eval は scope 大、本 finding 提示の `近似値 + bias 評価` で当面の判定とする。",{"type":50,"text":147},"4. 実装 (splat eval --split-file flag)",{"type":138,"items":149},[150,151,152,153],"\u003Ccode>splat-io\u002Fsrc\u002Fdataset.rs\u003C\u002Fcode>: \u003Ccode>load_nerf_synthetic_from_json(json_path, dataset_dir, convention)\u003C\u002Fcode> を新規追加。既存 \u003Ccode>load_nerf_synthetic_with_convention\u003C\u002Fcode> はその薄ラッパに refactor (backward-compat 完全維持)。","\u003Ccode>splat-cli\u002Fsrc\u002Fmain.rs\u003C\u002Fcode>: \u003Ccode>Cmd::Eval\u003C\u002Fcode> に \u003Ccode>split_file: Option<PathBuf>\u003C\u002Fcode> 引数追加。","\u003Ccode>splat-cli\u002Fsrc\u002Fcmd\u002Feval.rs\u003C\u002Fcode>: \u003Ccode>split_file\u003C\u002Fcode> 指定時は \u003Ccode>load_nerf_synthetic_from_json\u003C\u002Fcode>、未指定時は既存 \u003Ccode>load_nerf_synthetic_with_convention\u003C\u002Fcode> 経由で \u003Ccode>transforms_<split>.json\u003C\u002Fcode> を自動推定。\u003Ccode>--split\u003C\u002Fcode> の validation は \u003Ccode>--split-file\u003C\u002Fcode> 未指定時のみ。","\u003Ccode>\u002FUsers\u002Fotkrickey\u002Fdev\u002F3dgs-workspace\u002Fdatasets\u002Fnerf_synthetic\u002Flego\u002Ftransforms_test_subset.json\u003C\u002Fcode>: 既存 RGB が残っている 36 frame だけを抽出した新 JSON (元 \u003Ccode>transforms_test.json\u003C\u002Fcode> は変更せず)。",{"type":50,"text":155},"5. 再現手順",{"type":157,"lang":158,"text":159},"code","bash","# 1. subset JSON 生成 (再生成可能)\npython3 -c \"\nimport json, os\nds = '\u002FUsers\u002Fotkrickey\u002Fdev\u002F3dgs-workspace\u002Fdatasets\u002Fnerf_synthetic\u002Flego'\ndata = json.load(open(f'{ds}\u002Ftransforms_test.json'))\nexisting = set(os.listdir(f'{ds}\u002Ftest'))\ndef has_rgb(f):\n    b = os.path.basename(f['file_path'])\n    if not b.endswith('.png'):\n        b += '.png'\n    return b in existing\ndata['frames'] = [f for f in data['frames'] if has_rgb(f)]\nprint('subset frames:', len(data['frames']))\njson.dump(data, open(f'{ds}\u002Ftransforms_test_subset.json', 'w'), indent=2)\n\"\n# → subset frames: 36\n\n# 2. build & eval\ncd splat\ncargo build --release -p splat-cli\nPLY=runs\u002Flego-brushcompat-base-30k\u002Ffinal.ply\nDS=\u002FUsers\u002Fotkrickey\u002Fdev\u002F3dgs-workspace\u002Fdatasets\u002Fnerf_synthetic\u002Flego\n\n.\u002Ftarget\u002Frelease\u002Fsplat eval --ply $PLY --dataset $DS \\\n  --split-file $DS\u002Ftransforms_test_subset.json --convention brush --quantize-8bit\n# → 33.315 dB (n=36 test subset, q8)\n\n.\u002Ftarget\u002Frelease\u002Fsplat eval --ply $PLY --dataset $DS \\\n  --split-file $DS\u002Ftransforms_test_subset.json --convention brush\n# → 33.241 dB (raw)\n",{"type":50,"text":161},"6. 含意 (P1 narrative の確定)",{"type":138,"ordered":139,"items":163},[164,165,166,167,168],"\u003Cstrong>val parity claim は維持\u003C\u002Fstrong>: brush 自身 val 32.038 dB を本実装が +3.20 dB 上回る事実 (Stage 2 finding) はそのまま有効。同 dataset \u002F 同 split \u002F 同 convention での apples-to-apples 結果。","\u003Cstrong>test parity claim は不成立\u003C\u002Fstrong>: brush paper 37.40 dB に対して本実装 test subset 33.315 dB、bias ±2 dB 考慮しても 4 dB の gap は残る。`本実装 ≥ brush paper` は test split では言えない。","\u003Cstrong>本実装の novel-view generalization gap\u003C\u002Fstrong>: val → test subset で -1.92 dB の drop。これは本実装側の弱点 (brush は逆に val → test で +5.4 dB 上振れする方向)。原因候補は (a) opacity decay 未導入による over-densification + overfit、(b) refine の split 戦略が train view 近傍に偏重、(c) SH 進行 (progressive growth) なしの一括 sh3 が val では効くが test では generalization 弱、等。","\u003Cstrong>P1 narrative の言い回し\u003C\u002Fstrong>: `本実装は brush 自身 val を +3.20 dB 上回ったが、brush paper の test 37.40 dB との parity には -4 dB の generalization gap が残る (n=36 subset 近似評価)`。卒論には `val parity を達成、test generalization は今後の課題` と明示。","\u003Cstrong>次 Step 候補\u003C\u002Fstrong>: (1) Phase D opacity decay 30k full で over-densification 削減 → generalization 改善仮説テスト、(2) full 200 frame DL + 完全 apples-to-apples eval (subset bias を確定)、(3) brush 自身を test split で再 eval して val→test の +5.4 dB 上振れを ground-truth 化、(4) multi-scene chain bench (PID 81416) 完了後 8 シーンで同 test subset eval、universal な val→test 劣化パターンを確認。",{"type":50,"text":170},"7. 関連",{"type":138,"items":172},[173,174,175,176,177],"P1.B+F Stage 2 (val 35.184 dB): \u003Ccode>p1-b-f-stage2-30k-results\u003C\u002Fcode>","P1.A.3 cross-eval reproducer (4-way symmetry test): \u003Ccode>p1-a-3-cross-eval-reproducer\u003C\u002Fcode>","P1.A eval convention audit: \u003Ccode>p1-a-eval-convention-audit\u003C\u002Fcode>","brush vs splat 37dB gap 分析: \u003Ccode>brush-vs-splat-37dB-gap-analysis\u003C\u002Fcode>","brush 自身 bench (val 32.038 dB): \u003Ccode>m4-brush-bench\u003C\u002Fcode>",[179],{"id":27,"title":27,"subtitle":180,"date":9,"workspace":181,"tags":182,"verdict":188,"psnr":189,"psnr_unit":-1,"wallclock":190,"splats":191,"summary_url":192,"detail_path":192},"P1.B.F Stage 2 brush 互換 30k — gt_convention=premultiplied、refine 期間 brush 流","splat",[183,23,184,185,186,187],"p1-b-f","brush-compat","convention-bridge","premultiplied","m3-gate","partial",35.18375015258789,"1h 2m 18s",846689,"\u002Fruns\u002Flego-brushcompat-base-30k\u002F",[194,214,231],{"id":30,"title":195,"date":9,"status":10,"polarity":196,"category":11,"axes":197,"tags":199,"task_code":206,"related_runs":207,"delta_psnr":209,"delta_wallclock":210,"rank":35,"verdict":211,"impact_summary":212,"detail_path":213},"P1.A.3 cross-eval reproducer — brush convention で 24.879 → 1.67 dB に崩壊、主仮説 falsify","negative",[14,198,15],2,[17,200,201,19,202,57,203,186,204,205],"phase-a","milestone-m1","eval","psnr","reproducer","falsified-hypothesis","P1.A.3 + P1.A.4",[208],"lego-sh3-30k (splat-rs 24.879 dB legacy\u002Fval)","-23.21 dB (brush convention 化で 24.879 → 1.667)","N\u002FA (eval only)","hypothesis-falsified-stronger-finding","splat-rs `final.ply` (24.879 dB legacy\u002Fval baseline) を brush 準拠 convention (premultiplied GT + bg=ZERO 比較 + 8-bit roundtrip) で再評価すると **1.67 dB に崩壊**。audit §6 が予測した +3〜+5 dB 底上げと逆方向に -23 dB。原因は trainer が white-bg target で学習されており、背景領域を opaque-white splat で埋めるよう収束した結果、brush 流の bg=ZERO 比較では背景 pixel 全体で MSE≈1 が systematic に乗る。`view_00.png` 目視確認 (背景は白い不透明領域) で機構を確定。**training と eval の convention は coupling しており、eval pipeline だけ揃える apparent-gap 仮説は不成立**。卒研 P1.M2\u002FM3 に向けては「training loss も brush 化 (RGBA 4ch L1 を α=0 領域で背景に penalty を吹かさない構造)」が必須要件。8-bit quantize 単体の impact はほぼ無視可能 (legacy 24.879 → 24.879、brush 1.605 → 1.667、+0.06 dB)。","\u002Ffindings\u002Fp1-a-3-cross-eval-reproducer\u002F",{"id":31,"title":215,"date":9,"status":10,"polarity":216,"category":217,"axes":218,"tags":219,"task_code":222,"related_runs":223,"delta_psnr":226,"delta_wallclock":227,"rank":35,"verdict":228,"impact_summary":229,"detail_path":230},"P1.A eval convention audit (統合) — 7 軸の diff 確定、apparent gap 推定 -3〜-6 dB","neutral","audit",[14,198,15],[17,200,201,19,202,57,203,186,220,55,217,221],"alpha","synthesis","P1.A (M1)",[224,225],"lego-sh3-30k (splat-rs 24.879 dB)","brush-lego-sh3-30k (37.40 dB report)","推定 -3〜-6 dB (apparent gap 縮小、A.3 で実測予定)","N\u002FA (audit task)","audit-complete-gate-passed","両 trainer の eval pipeline を file:line で完全 audit、PSNR formula 本体 (MAX=1 \u002F log10 \u002F RGB only \u002F per-view mean) は同等だが、(1) test split (200 view) vs val split (100 view)、(2) background composite convention の **完全逆方向**、(3) 8-bit roundtrip 有無、(4) α-mask 経路、(5) clamp\u002Fquantize 等 7 軸で diff を確認。最大の発見は brush の premultiplied-α + bg=ZERO eval が NeRF Synthetic の透明領域で構造的に PSNR を +3 dB 以上嵩上げする一方、splat-rs は target に white pre-composite \u002F rendered に bg 合成なしの mismatch で convergence 残差が MSE に直接残る。apparent gap の推定 -3〜-6 dB を A.3 reproducer で実証予定、残り -6〜-9 dB が真の algorithmic gap。","\u002Ffindings\u002Fp1-a-eval-convention-audit\u002F",{"id":29,"title":232,"date":9,"status":10,"polarity":233,"category":11,"axes":234,"tags":235,"task_code":239,"related_runs":240,"delta_psnr":243,"delta_wallclock":244,"rank":35,"verdict":245,"impact_summary":246,"detail_path":247},"P1.B+F Stage 2 — Lego 30k brushcompat で 35.184 dB、brush 自身を +3.20 dB 上回り","positive",[14,198,15],[17,18,236,19,237,186,185,23,238],"milestone-m3","brush-超え","stage-2","P1.B+F Stage 2 (M3 gate)",[27,241,242],"lego-brushcompat-base-5k","lego-sh3-30k (legacy 30k 24.879 dB)","+10.30 dB vs legacy 30k (24.879 → 35.184、convention 変更後の真の現状)","+2.7x vs legacy 30k (1h 2m 18s vs 22m18s、splats 10x で per-iter time 増、ただし brush 自身 282k より 3 倍多い)","accepted-stretch-goal-met","Lego sh3 30k で gt_convention=premultiplied (brush 互換) を立てると、4-way eval で legacy=1.60 \u002F brush=35.24 dB。**brush 自身 val 32.0 dB を +3.20 dB 上回る** 結果。M3 lifeline (30 dB) を +5.24 dB 突破、M5 (36 dB) まで -0.76 dB に到達。Phase A 主仮説 (apparent gap -3〜-6 dB) は falsify されたが、coupling 解消の真の効果は **+33.6 dB shift (1.67 → 35.24)**、想定 (+10 dB) の 3 倍。実装は configs 1 行 (gt_convention) + dataset.rs (load_rgba_premultiplied path 追加、既に Stage 1 で merge 済) のみ、既存 30k legacy bench との apples-to-apples comparison が可能。brush の wallclock 38% 高速化は 30k でも継続 (splats 1M-cap で 846k 到達、refine が攻撃的 split)、ただし brush 自身 282k に比べて 3 倍、本実装が capacity を未活用 (refine を絞る余地あり、Phase D で検証可能)。次 Step は multi-scene 8 シーン展開で universal claim 確定、brush mean 33.32 dB 超えで multi-scene parity 完全達成を狙う。","\u002Ffindings\u002Fp1-b-f-stage2-30k-results\u002F",1782449788665]