The Problem with Banking SOPs

If you've ever worked in bank ops, you know the drill. There's a procedure for everything. Loan origination has one. So does wire transfers. Account opening. Fraud investigation. They live in SharePoint folders, sometimes Word docs, sometimes PDFs that haven't been updated since 2019.

The thing nobody talks about: these procedures don't exist in isolation. Your wire transfer SOP references your AML SOP. Your AML SOP references your CIP procedures. Your CIP procedures touch your account opening flow. It's a web. And when regulators update BSA requirements, or when your core banking system changes an API, you're supposed to trace through all of these dependencies manually.

Nobody does. Not really.

What we're building

We think there's a better way. Instead of treating SOPs as static documents, we treat them as nodes in a graph. Each procedure connects to others based on what it references, what it depends on, and what depends on it.

When you upload an SOP, we don't just store it. We read it. We understand the steps, the decision points, the external systems it touches. We embed it semantically so we can find related procedures even when they don't explicitly reference each other. "Suspicious activity" in one doc connects to "SAR filing" in another, even if nobody linked them.

wire_transfer_sop
  ├── references: aml_screening_sop
  ├── references: ofac_check_procedure  
  ├── uses_system: core_banking.wire_module
  └── triggers: daily_bsa_report

aml_screening_sop
  ├── references: cip_verification_sop
  ├── references: sar_filing_procedure
  └── uses_system: lexisnexis.watchlist

cip_verification_sop
  ├── references: account_opening_sop
  └── uses_system: idology.verify

When something changes

Here's where it gets interesting. Say FinCEN updates their SAR filing requirements. You update your SAR procedure. In the old world, that's where it ends. Maybe you remember to check the AML SOP. Probably not.

In our system, updating the SAR procedure automatically highlights every other document that connects to it. Not just the ones that explicitly reference "SAR" by name, but anything semantically related to suspicious activity reporting. The graph shows you the blast radius.

Each connected node gets flagged with what changed and why it might matter. Your AML screening procedure now shows a notice: "SAR filing thresholds updated - review escalation triggers." Your wire transfer SOP shows: "Downstream SAR process changed - verify alert routing still valid."

You're not chasing ghosts through SharePoint. The system tells you what to look at.

Why this matters for agents

We're not building a document management system. We're building AI agents that can actually do the work. But an agent is only as good as the procedures it follows.

When your loan processing agent runs, it needs to know the current state of your underwriting guidelines, your compliance requirements, your exception handling rules. If those procedures are out of sync, the agent makes mistakes. If the agent can see the full graph of how procedures connect, it can catch inconsistencies before they become problems.

More importantly, when you update a procedure, every agent that depends on it knows immediately. No more "we updated the policy but forgot to retrain the model." The knowledge graph is the single source of truth.

How we evaluate

A model that sounds right is not the same as a model that is right. Because these agents touch money and members, we hold every change to an evaluation suite before it ships. Nothing reaches a customer on vibes.

The suite is built from real, de-identified banking tasks and split along the same axis as the context layer. Micro evals test member-level reasoning — does the agent read the right subgraph, apply the right policy, and reach the decision a senior officer would? Macro evals test portfolio-level judgment — pricing under a shifting base rate, trend detection, exposure limits. Each task carries a rubric and a gold answer graded by domain reviewers.

suite                 n     prod   candidate   Δ
micro.decision      420    0.86     0.91     +0.05
micro.policy        260    0.90     0.93     +0.03
macro.pricing       180    0.79     0.88     +0.09
macro.trend         120    0.74     0.82     +0.08
grounding (cite)    980    0.88     0.96     +0.08
hallucination ↓     980    0.061    0.021    -0.040

release gate: pass if no suite regresses > 1.0 pt
                     AND hallucination <= 0.03  ->  PASS

We score three things on every run: task success against the rubric, grounding (is each claim traceable to a node in the context graph?), and calibration. Calibration matters more than raw accuracy here — a lending model that says 90% should be right about 90% of the time. We track it with expected calibration error, binning predictions and measuring the gap between confidence and outcome: ECE = Σ (n_b / N) · |acc(b) − conf(b)|. Our current release sits at ECE = 0.026, down from 0.058 before we added graph grounding.

Every run is offline replay first: candidate models re-decide thousands of recorded cases where we already know what happened, so we can measure lift without touching a live member. High-risk suites also gate on a human-review sample. A release ships only when no suite regresses beyond a point and the hallucination rate stays under 3%.

Technical approach

We use vector embeddings (pgvector in Postgres) to capture semantic relationships between procedures. When you upload a document, we chunk it into logical sections, embed each section, and store both the raw text and its vector representation.

Explicit references (when one SOP mentions another by name) create hard edges in the graph. Semantic similarity creates soft edges, weighted by how closely the content relates. When a node updates, we recalculate affected edges and propagate change notifications based on edge weight.

The result is a living map of your operational knowledge. Not a static org chart, but an actual representation of how information flows through your institution.