Methodology.

How Investment Policy Statements are scored — pipeline, scoring models, and validation

Version 2026-09-14. Reflects the migration to Claude, the anchor-based multi-lens LLM judge, the per-phase model split, the golden-set calibration toolkit, the corpus audit and one-folder-one-policy rule, the stage-review-publish refresh, and withheld scores.

1. What this measures

This system assesses the quality of an institutional Investment Policy Statement (IPS) — primarily U.S. public pension plans — against a structured rubric. Scores reflect policy quality, enforceability, and governance clarity, not investment skill or returns. A high score means the IPS is specific, operational, and enforceable; a low score means it is missing, vague, or aspirational.

Each plan's published IPS PDF is ingested, its policy facts are extracted by an LLM, and the plan is scored across the rubric's domains by two independent methods (a deterministic presence check and an LLM "judge"). Results feed a dashboard, comparative tables, and a research paper.

2. The rubric

The active rubric is config/rubrics/ips_rubric_v1_3.yaml. It defines 15 domains (e.g. Scope & Purpose, Investment Objectives, Risk Tolerance, Strategic Asset Allocation, Rebalancing, Performance Measurement & Benchmarks, Governance, Monitoring & Review, Responsible Investment/ESG). For each domain it specifies:

• required_facts — the concrete elements an IPS should contain for that domain.
• scoring_anchors — definitions for score levels 1 / 3 / 5 on the quality scale.
• mandatory — true, false (optional), or "conditional" (applies only to certain institution types, via applies_to).

The 1–5 scale: 1 = Deficient (missing/aspirational) · 2 = Weak (present but vague) · 3 = Adequate (basic structure, gaps) · 4 = Strong (clear, operational, minor gaps) · 5 = Best Practice (fully specified, enforceable, peer-aligned).

The rubric also defines global_reason_codes grouped into four families — presence/clarity, quantification/enforceability, consistency/structure, governance/process — which the LLM judge uses as diagnostic tags (see §5b).

A domain is "core" when it is mandatory: true or an applicable "conditional" domain. For a public pension plan 12 of the 15 domains are core, and they alone drive the overall score and critical-gap flags:

• 11 mandatory domains, plus the applicable conditional spending / distribution policy = 12 core.
• 2 optional (mandatory: false) — investment philosophy and responsible investment / ESG. Both are scored and reported in full; neither moves the overall score, and neither can raise a critical gap.
• 1 conditional that does not apply — tax considerations, which is not scored for a pension plan at all.

So 14 domains are scored and 12 of them are core. (Until 2026-09-14 this section said 14 of 15 were core, which double-counted the two optional domains; the scorer has always used 12 — judge_score.py excludes mandatory is False from the overall mean.)

3. End-to-end pipeline

  data/raw/IPS/<Plan>.pdf
          |
          v
  +---------------------+
  |  Ingest             |  pypdf + OCR fallback; IPS-validation gate; SHA-256
  |  (ingest_pdf.py)    |  -> page-marked text with [[PAGE n]] markers
  +---------------------+
          | full text
          v
  +---------------------+
  |  Extract facts      |  chunk the text; for each applicable domain, ONE
  |  (extract_only.py + |  Claude call (Haiku) returns each required fact
  |   AnthropicExtractor)|  with supporting evidence (chunk id, page, quote)
  +---------------------+
          | extracted_facts.json  {meta, chunks, domains[]}
          |
          +--------------------> Presence scorer (deterministic) --> scorecard.json
          |                      score.py + generate_scorecard.py
          |
          +--------------------> LLM judge (Sonnet, anchor-based)  --> scorecard_llm.json
          |                      judge_score.score_plan
          |
          v
  generate_detailed_report.py --> assessment_report_detailed.md

The two scorecards coexist for A/B comparison; the judge never modifies scorecard.json. Batch processing is run via scripts/process_all_ips.py (add --judge to also produce scorecard_llm.json); the judge can also be (re)run alone over existing extractions via scripts/judge_score_all.py.

4. Fact extraction

extract_all_facts chunks the ingested text and loops over the applicable domains. For each domain, the AnthropicExtractor (Claude Messages API) is asked only to find facts and cite evidence — not to score. Each returned fact carries its value plus evidence (chunk_id, page, a short verbatim quote). The output (extracted_facts.json) also records provenance: model id, rubric hash, and a hash of the IPS text. Extraction is deliberately the retrieval step, so it runs on the cheaper Haiku model by default (see §6).

Note: extraction uses naive fixed-size chunking, but reads the whole document by default (a --max-chunks cap can restore the pre-2026-08-30 behaviour of the first ~30 chunks ≈ 25 pages). A document too large for a model's context window is trimmed to fit, and meta.n_chunks_used / meta.truncated_to_fit_context record what a given run saw.

5. Scoring — two approaches

5a. Presence scorer (legacy, deterministic)

score.py computes, per domain, the fraction of required_facts that were extracted (treating explicit-absence strings like "Not specified" as missing) and bins it into 1–5 (<20%→1, 20–40%→2, 40–70%→3, 70–90%→4, >90%→5). It is fully reproducible and cheap, but it measures presence, not quality: a vague mention and an enforceable, quantified policy can score the same. Output: scorecard.json.

5b. Anchor-based multi-lens LLM judge (current methodology)

The judge is the quality-aware methodology. It is two-stage: it runs on the frozen extracted_facts.json (no re-extraction) plus the source document, so it is re-runnable and re-scorable. For each domain, AnthropicJudge.score_domain makes one Claude call using tool use (a forced submit_domain_score schema whose reason codes are enum-constrained to the rubric families), scoring four quality lenses 1–5 against the domain's anchors:

  For each applicable domain:

    extracted facts (this domain)        IPS document  (sent as a CACHED
            |                            prefix block, reused across calls)
            +-------------+----------------------+
                          v
              +-----------------------------+
              |  AnthropicJudge (tool use)  |  temperature = 0
              |  Claude Sonnet              |  forced submit_domain_score schema
              +-----------------------------+
                          |  four lens sub-scores (1-5) + reason codes + rationale
                          v
   presence/clarity . quantification/enforceability . consistency/structure . governance/process
                          |  average the APPLICABLE lenses
                          v
                 domain score (float 1-5)
                          |  cited evidence quotes verified against the source text
                          v
   core-domain mean -> overall_score      core domain <= 2 -> critical gap
                          v
                   scorecard_llm.json   (lens detail + reason codes + evidence + diagnostics)

Key properties:

• Lens applicability is judge-decided — presence/clarity always applies; the other three may be marked not-applicable for a domain where they do not fit. The domain score is the average of applicable lenses (a float).
• Evidence is validated — every quote the judge cites is checked (whitespace/case-insensitive, ellipsis-aware) against the source text; the result is recorded as an evidence_integrity count. In the current version this is flag-only — it is reported but does not change the score.
• Per-domain failure isolation — if one domain's judge call fails, that domain is recorded with a null score and the plan is marked partial; the rest still score.
• The legacy presence_ratio is retained per domain as a diagnostic alongside the lens scores.

6. The LLM layer (models, caching, tool use)

The system migrated from OpenAI to Claude (Anthropic). The API key resolves from CLAUDE_API_KEY or ANTHROPIC_API_KEY. The two phases use different default models, chosen by capability vs cost:

Conventions across both phases:

• temperature = 0 for reproducible output.
• Prompt caching — the IPS document is sent as a cached prefix block, so the ~14 per-domain calls for a plan reuse it instead of re-billing the full document each time (a large cost saving).
• Tool use for the judge — a forced structured-output schema guarantees valid, enum-constrained output, removing fragile JSON-repair logic. (Tool-use occasionally returns nested fields as JSON strings; the consumers defensively coerce these.)
• Truncation is surfaced, not retried — a response cut off by max_tokens raises rather than silently looping.

7. Aggregation & flags

• Domain score (judge) = mean of applicable lens scores, 2 dp.
• Overall score = mean of core domain scores (optional domains are scored and shown but excluded from the headline number, per the rubric's critical-gap rule).
• Critical gap = a core domain scoring ≤ 2.
• Recommendations are generated from the weakest lens(es) and reason codes of low-scoring domains.

The dashboard's "Benchmarks" view additionally normalizes the per-plan asset-class benchmark strings into a canonical map (normalize_benchmarks.py deterministic rules + an optional LLM consolidation pass).

8. Validation: golden-set calibration

Because the judge's scores are the product of a model, their credibility rests on measuring agreement with an expert. The calibration toolkit (src/calibration/, scripts/calibration_app.py) does this:

  judge run across all plans (scorecard_llm.json)
          |
          v
  build_calibration_sample.py  -->  sample.json   (stratified ~15-20 plans:
          |                          by judge overall score + plan-type variety
          |                          + forced-in presence-vs-judge "hard cases";
          |                          pins judge model / rubric hash / prompt version)
          v
  calibration_app.py  (expert rates each domain 1-5, BLIND -- judge score
          |             hidden until submit, then revealed)  -->  ratings.json
          v
  metrics harness  -->  report.{json,md}

The expert rates per domain, against the full IPS text (not the judge's extraction, to avoid anchoring), with the judge's score hidden until each rating is submitted. The harness reports, on a defined int/float rule (exact-match/within-1/κ use the judge rounded; MAE/bias/Spearman use the float; null-judge domains excluded):

• Quadratic-weighted Cohen's κ — the headline agreement metric.
• Per-domain within-1 / MAE / bias — where the judge is weak and whether it is systematically lenient or harsh.
• Test-retest ceiling — the rater re-scores a subset blind; judge-vs-expert agreement is reported relative to the rater's own consistency.
• Disagreement log — every domain where judge and expert differ by ≥ 2.

The sample, ratings, and report are durable artifacts tied to a pinned judge configuration, so they double as a regression benchmark: change the prompt or model, re-run against the same sample, and compare κ. The current scope is measure & diagnose only (no automated tuning).

9. Source tracking & refresh

IPS documents change over time. The refresh runs monthly and is deliberately split into three steps, so nothing reaches the published corpus without a human looking at it first:

  download_summary.csv  (plan -> published IPS URL = source of truth)
          |  check_ips_updates.py --stage <dir>
          v
  fetch each URL -> compare SHA-256 to the stored PDF
          |
     UNCHANGED        CHANGED  (outcome STAGED)
          |                |  new document -> data/staging/<stamp>/raw/   (published PDF untouched)
          v                v
  update data/ips_sources.csv  +  per-run log in data/ips_update_runs/

  1. STAGE    monthly_refresh.py  -> data/staging/<stamp>/ + manifest.json + review.md
                                     (fetch and hold; nothing published)
  2. RESCORE  rescore_staged.py <stamp>  -> extraction + judge inside the staging tree,
                                     with the per-document corpus checks and the cost in review.md
  3. PUBLISH  publish_release.py <stamp> -> archive what it replaces into data/releases/<stamp>/,
                                     promote into data/processed/IPS_claude + data/raw/IPS,
                                     rebuild derived tables and the site, sync R2.
                                     --rollback <release> restores the archived state.

Only changed plans are ever rescored. refresh_and_rescore.py is the older single-step, local-only path and is retained for ad-hoc use; the monthly job does not use it.

Aggregator mirrors are never sources. source_policy.py blocks publicplansdata.org: the checker records BLOCKED_DOMAIN without fetching, the site drops the link, and the build fails if one is published. A plan whose own site publishes no IPS keeps its stored PDF with a blank source URL.

9a. Corpus integrity

Three rules govern what is in the corpus at all, enforced by scripts/audit_corpus.py, which gates every scoring run and every publish (exit 1 on any error):

• One folder = one investment policy. Where a single document governs several plans, the corpus keeps one folder; data/plan_members.csv lists every governed plan under it with its AUM. AUM is an attribute, never a weight, and never appears beside a score.
• Every extraction must match its PDF. The audit errors on a folder with no PDF or no extraction, on a processed folder with no registry row, on byte-identical PDFs, on an extraction whose ingest SHA no longer matches the file, and on a broken text layer. Warnings cover encoding damage, glyph-index text, near duplicates, short or low-yield documents, and facts extracted under an older rubric.
• The document has to be an IPS. Beyond the ingest-time validation gate, check_document_type flags a cover page naming another instrument — an agenda, minutes, a statement of beliefs, a set of guidelines, a policy manual — with no policy phrase on it. Four wrong instruments reached the corpus before this existed.

Withheld scores. When a plan's published document turns out to be the wrong instrument, the plan keeps its folder, PDF and extraction (so the audit stays consistent) but publishes no score: its row in data/withheld_scores.csv makes judge_score_all.py skip it and process_all_ips.py --judge refuse it, and the audit errors if a scorecard_llm.json ever appears beside one. Removing the row is a deliberate act that follows replacing the document.

10. Reproducibility & provenance

• Every scorecard_llm.json records judge_model, rubric_hash, prompt_version, and a hash of the source extraction.
• temperature = 0 and the two-stage design make judge re-runs stable and comparable.
• Every score traces to evidence — the judge cites verbatim quotes (validated against the source), and the extractor records per-fact evidence with page references.
• A versioned rubric and prompt_version mean the methodology can be frozen for a paper's claims; changing either invalidates prior scores and triggers a re-run.

11. Limitations (stated honestly)

• Presence ≠ quality. The legacy scorecard.json measures fact presence, not enforceability; it is retained only for A/B comparison. The judge is the quality methodology.
• Calibration is not yet at scale. As of this version the judge has been validated end-to-end on sample data; a full golden-set run (sole expert rater) establishes accuracy and is the credibility prerequisite before scores are treated as authoritative.
• Single rater. Calibration uses one expert with a test-retest ceiling rather than multiple raters — an honest, stated limitation.
• Evidence verification is flag-only and still under-counts quotes that drift a few tokens from the source; it must be improved before scores are ever gated on it.
• Chunking is still naive. Extraction now reads the whole document by default rather than a fixed prefix, but chunk boundaries are still fixed-size windows rather than semantic; per-domain retrieval within the document remains a planned improvement.

12. Outputs & artifacts

Per plan, under data/processed/IPS_claude/<Plan>/ — the Claude-scored universe the dashboard and the public site both build from: extracted_facts.json, scorecard.json (presence), scorecard_llm.json (judge), assessment_report_detailed.md, asset_allocation_report.md, ingested/. (data/processed/IPS/ is the legacy OpenAI-era tree, no longer read by anything.)

Cross-plan: data/plan_registry.csv and data/plan_members.csv (names, types, governed plans, AUM), data/withheld_scores.csv, data/ips_sources.csv, data/ips_update_runs/, data/benchmark_clusters.json, the calibration artifacts under data/calibration/, and the aggregate tables in out/. One monthly refresh lives in data/staging/<stamp>/, what each publish replaced in data/releases/<stamp>/, and everything a retired folder held in data/retired/<stamp>/.

Two consumers: the Streamlit dashboard (scripts/dashboard.py), which surfaces everything including plan names beside scores, and the public static site, which does not — there, a plan name appears only with document facts, scores appear only against Plan N pseudonyms, and a build-time gate fails the build if the two are ever joined on one page.