Why LLM chat summaries fail credit committee and audit review
A general-purpose chat model can read a borrower's statements and write a fluent memo in seconds. The memo reads well. The trouble starts when someone asks where a number came from.
A credit committee, an LP and an external auditor all ask the same 3 questions about a figure: which document it came from, how it was calculated, and whether the answer would be the same if they ran the analysis again. A chat summary struggles with all 3.
- Numbers that move between runs. The same prompt over the same files can return a different EBITDA or a different DSCR on the next run, or after the provider updates the model.
- Estimated arithmetic. A language model generates ratios as text. It predicts what a division probably returns, and a rounding slip in one step carries into every ratio built on it.
- No page-level source. A summary may name a document, and checking that the figure actually appears there is left to the analyst.
- No version history. When the borrower sends restated accounts, nothing records which version of the file the earlier memo relied on.
- No record of what the model saw. Chat history sits inside the vendor's application, and it can't show which files, in which versions, were in the context when the answer was written.
Each gap sends an analyst back to rebuild the spread by hand before the deal goes to committee. The model saves time on the draft and the team spends it again on checking.
An auditable pipeline splits the work along a clear line. A language model reads and extracts. Code computes. Hashes record the evidence. The rest of this guide walks through each stage of that split.
Stage 1: document intake and cited extraction
Every figure in a spread is only as good as the document behind it. Stage 1 builds a complete, hashed room before any number is extracted.
A borrower portal under your brand
You white-label the borrower ingestion portal with your own brand and choose which checklist templates each applicant sees. The applicant signs in, opens a request, uploads documents and gives consent.
The company checklist asks for 8 required items, from the P&L / Income Statement and Balance Sheet to the Debt Schedule / Credit Report. Each of 11 deal types adds its own product checklist, from trade receivables and CRE income-producing loans to SMB acquisition and crypto-backed lending.
Redact, then classify
PII is redacted before classification, so only redacted text reaches the classifier. A keyword match and an LLM then place each file in a checklist category.
A keyword hit at confidence 0.8 or above returns at once, and a keyword result overrides a low-confidence LLM answer. Files that match no category land in a catch-all folder named "Other".
Unreadable files and the data-room inventory
Each upload is hashed with SHA-256 to check that the file arrived intact. Files the pipeline can't read go on an "unreadable" list for your team to chase, and the borrower can't submit until every document's classification has finished.
The data-room inventory then lists every file in the room, sorted by relative path, with
its size, sha256, extension, MIME type and file class. Re-running it over an
unchanged room produces byte-identical output, apart from the timestamp. The document
receipt counts required items, completed required items and gaps, so missing documents are
as visible as the ones that arrived.
Citations resolved against the inventory
Every documentary figure is stored with 3 parts: the value, a rationale and the source document. The source is resolved against the data-room inventory, and the cited file's SHA-256 is recorded next to the figure.
The spread links each figure back to its source: an analyst clicks a figure in the spread and sees the line in the original statement. The interface shows the source document's name, plus the page wherever the extractor resolved one. A citation can also carry the anchored proof ID and the catalog version ID of its source.
An unresolvable citation is a kill-gate. The run stops, a person fixes the citation, and nothing rewrites it automatically.
Stage 2: deterministic metrics
Stage 2 turns cited figures into ratios. Rekord AI spreads the borrower's financials through a fixed set of named metric functions. Each formula is transcribed from its definition, and a language model plays no part in the arithmetic.
The core formulas
- EBITDA = net income + interest expense + income tax expense + depreciation and amortization.
- DSCR = EBITDA ÷ total debt service. Adjusted DSCR = adjusted EBITDA ÷ total debt service.
- Debt Yield = NOI ÷ loan amount.
- LTV = loan amount ÷ collateral value.
- ABL borrowing base = eligible pool × advance rate − specific reserves, where advance rate = 1 − dynamic credit enhancement, optionally capped by a configured advance-rate cap.
Other named metrics include gross profit, EBITDA margin, net profit margin, YoY revenue growth, current ratio, debt-to-equity, working capital, debt-to-EBITDA, free cash flow, SDE, enterprise value, collateral-to-debt and portfolio loss rate.
Golden test vectors
A golden test vector is a fixed set of inputs paired with the output the formula must return. Each metric function is checked against its vectors.
If a change to a formula alters a result, the matching vector fails and the change is caught. That's what makes a ratio reproducible: the same inputs return the same DSCR, Debt Yield or borrowing base, this quarter and next, for the analyst and for the auditor.
The calculation trace
Each step of the borrowing base lands in a calculation trace with its formula, inputs, unit and source. A reviewer reads the trace top to bottom and can redo every step with a calculator, without access to the model or to the spreadsheet that produced it.
Acceptance thresholds belong to each lender's credit policy. The formulas produce the figures; the policy decides what a given DSCR or LTV means for the deal.
Stage 3: factor-attributed scorecards and the evidence revision manifest
Stage 3 packages the outcome and the evidence behind it into records that a third party can check later.
Factor-attributed scorecards
Every scorecard breaks down to the factors and weights that drove it, ready for a credit committee or a regulator. The committee sees which inputs moved the result and by how much.
For illustration only, a factor could be a coverage ratio taken from the spread, or the share of required checklist items the borrower delivered. The factors, weights and acceptance thresholds in a live mandate follow each lender's credit policy.
Scorecards are serialized as canonical JSON before hashing, and every scorecard emits a cryptographic proof anchored by Rekord Kloud, so any decision can be reconstructed and verified later.
The evidence revision manifest
When you publish an evidence revision, Rekord AI writes a manifest of its member files, serializes it as canonical JSON, hashes it with SHA-256 and anchors it as a proof through Rekord Kloud.
Each revision points to the previous one, so the history of the room is a chain you can walk. When a borrower sends restated accounts, the new revision records the new file version, and the earlier revision still shows what stood behind the earlier decision.
For each member file, the manifest records its logical document ID, proof ID, version ID,
filename, MIME type, fileHash and size. The worked example below shows the shape.
Black-box LLM summarization, manual spreading and Rekord AI compared
The table compares the 3 approaches on the points an investment committee, an LP or an external auditor asks about.
| Criterion | Black-box LLM summarization | Manual spreading | Rekord AI |
|---|---|---|---|
| Source of figures | Read by the model from uploaded text, with no record of which passage it used. | Keyed in by an analyst from the statements. | Extracted by a language model and stored with value, rationale and source document. |
| Arithmetic | Ratios generated as text. | Spreadsheet formulas rebuilt per deal, exposed to broken links and copy errors. | A fixed set of named metric functions with transcribed formulas. |
| Reproducibility | Outputs shift between runs and model versions. | Reviewers re-request files and rebuild the spreadsheet. | Golden test vectors lock each formula, so the same inputs return the same metrics. |
| Citations | Answers may name a document; checking the figure is left to you. | Analyst cell comments, as complete as each analyst makes them. | Source document, the page where resolved, and the file's SHA-256. An unresolved citation stops the run. |
| Version history | Chat history inside the vendor's application. | Access logs and file versions inside the VDR vendor's system. | Evidence revisions, each pointing to the previous one. |
| Audit evidence | Transcripts of prompts and answers. | The spreadsheet and the folder it was built from. | Manifests in canonical JSON, hashed with SHA-256 and anchored as proofs through Rekord Kloud. |
| Data perimeter and privacy | Document text typically sent to the model provider as uploaded. | Files held by the VDR vendor; redaction is a manual step. | Runs inside your perimeter; PII redacted before classification, and only hashes anchored. |
| Human review | An analyst checks the draft figure by figure. | A second analyst re-checks the spreadsheet. | A person fixes any unresolved citation, and the committee reviews factor-attributed scorecards. |
On turnaround, Rekord AI has delivered a 95% reduction in time for SMB acquisition finance underwriting analysis.
Worked example: DSCR, Debt Yield, LTV and the ABL borrowing base
The figures below are round illustrative inputs for 3 separate facilities: a corporate term loan, an income-producing CRE loan and an ABL revolver on trade receivables. Source references are placeholders that show the shape of a citation.
In Rekord AI the calculation trace covers each step of the borrowing base. The DSCR, Debt Yield and LTV rows use the same shape to show how each ratio traces back to its inputs.
| Step | Formula | Inputs | Result | Unit | Source |
|---|---|---|---|---|---|
| 1. EBITDA | Net income + interest expense + income tax expense + D&A | $2,400,000 + $900,000 + $600,000 + $1,100,000 | $5,000,000 | USD | FY2025 audited P&L, p. 4 |
| 2. Total debt service | Scheduled principal + interest | $2,100,000 + $900,000 | $3,000,000 | USD | Debt schedule, p. 2 |
| 3. DSCR | EBITDA ÷ total debt service | $5,000,000 ÷ $3,000,000 | 1.67x | ratio | Steps 1 and 2 |
| 4. Debt Yield | NOI ÷ loan amount | $1,800,000 ÷ $20,000,000 | 9.00% | percent | Rent roll and operating statement, p. 3; term sheet, p. 1 |
| 5. LTV | Loan amount ÷ collateral value | $20,000,000 ÷ $32,000,000 | 62.50% | percent | Term sheet, p. 1; appraisal report, p. 12 |
| 6. Advance rate before cap | 1 − dynamic credit enhancement | 1 − 12.00% | 88.00% | percent | AR aging, p. 1; dilution history, p. 2 |
| 7. Advance rate applied | Lower of step 6 and the advance-rate cap | min(88.00%, 85.00%) | 85.00% | percent | Facility configuration |
| 8. Gross availability | Eligible pool × advance rate | $10,000,000 × 85.00% | $8,500,000 | USD | Borrowing base certificate, p. 1 |
| 9. Borrowing base | Gross availability − specific reserves | $8,500,000 − $400,000 | $8,100,000 | USD | Borrowing base certificate, p. 2 |
Read the trace step by step. EBITDA adds back interest, tax and D&A to net income. Total debt service adds scheduled principal to interest, and DSCR divides the first by the second.
In the borrowing base, the dynamic credit enhancement of 12.00% would allow an advance rate of 88.00%, and the configured cap of 85.00% binds first. The cap is optional; without it, step 7 would carry the uncapped rate.
Every input in the trace points to a document, and every result follows from the row above it. An auditor who disagrees with a figure can name the exact step and the exact source instead of re-running the whole analysis. The thresholds that decide whether these figures pass belong to the lender's credit policy, and none are implied here.
Worked example: verifying an evidence revision manifest
The excerpt below uses the real field names of the evidence revision manifest with
placeholder values. Identifiers prefixed example- are placeholders.
To keep the example checkable, each fileHash is the real SHA-256 of a short
stand-in string in place of the PDF bytes. For the first member, that string is
example bytes of balance-sheet-fy2025.pdf, UTF-8 encoded, and sizeBytes is its
length in bytes.
{
"kind": "underwriting_evidence_revision",
"schemaVersion": 1,
"canonicalizationVersion": 1,
"dataroomId": "example-dataroom-id",
"revisionId": "example-revision-3",
"revisionNumber": "3",
"previousRevisionId": "example-revision-2",
"reason": "example: FY2025 audited statements received",
"publishedBy": "example-analyst-id",
"publishedAt": "…",
"members": [
{
"logicalDocumentId": "example-balance-sheet",
"proofId": "example-proof-balance-sheet",
"versionId": "example-version-balance-sheet",
"filename": "balance-sheet-fy2025.pdf",
"mimeType": "application/pdf",
"fileHash": "e8cc2f7cf833286d5bd67f200bcca2b4bf5732fe142292563545e24271b8b501",
"sizeBytes": "41"
},
{
"logicalDocumentId": "example-income-statement",
"proofId": "example-proof-income-statement",
"versionId": "example-version-income-statement",
"filename": "income-statement-fy2025.pdf",
"mimeType": "application/pdf",
"fileHash": "2dde2c4638ebf9b14652ae610d39a042fcd8924c85d92196aa46fc49b89698d6",
"sizeBytes": "44"
}
]
} How an auditor verifies it
- Take the anchored proof of the revision from Rekord Kloud and the manifest the lender hands over.
- Serialize the manifest as canonical JSON, hash it with SHA-256 and compare the result with the anchored hash. A match shows the manifest is the one that was published.
- For each member, recompute the SHA-256 of the file and compare it with its
fileHash. Any mismatch shows the file changed after publication. - Follow
previousRevisionIdto the earlier revision and repeat, so the full history of the room is checked in order. - Recompute the metrics from the cited figures. The formulas are deterministic, so the same inputs return the same DSCR, Debt Yield or borrowing base.
You can reproduce the first fileHash on any machine with a standard SHA-256
tool over the stand-in string. With real files, the auditor runs the same check over the
PDF bytes the lender provides.
The full architecture walkthrough shows where this step sits between borrower ingestion and on-chain anchoring.
The data stays inside your perimeter
Rekord AI runs inside your own infrastructure. It connects directly to your data sources, bureau feeds and bank integrations, so borrower files stay where your controls already apply.
- Redaction first. PII is redacted before classification, so only redacted text reaches the classifier.
- Hashes only on-chain. Rekord Kloud anchors SHA-256 hashes. A hash holds none of the document content.
- Documents stay with the lender. Financial statements, bank data and legal agreements stay with you.
The anchored record is tamper-detectable. It shows that a given manifest existed at a given time and lets anyone with the files check that nothing has changed since. Reviewers read anchored proofs through the Proof Console in Rekord Kloud.
Rekord AI and Rekord Kloud are built by Rekord AG, Zug, Switzerland. The Rekord AI product page covers the full feature set, including the "Ask the room" copilot that answers questions about the dataroom with numbered citations.