How Should a Boutique Bank Measure AI Pilot Hours, Rework and Throughput Against Its Baseline?

Start with a retained associate-hour baseline

Begin with the manual process as it is actually performed, not an idealized estimate. Select one concrete task such as extracting a company profile from approved source documents, preparing a first-pass buyer list for human review, or assembling a draft marketing-material section. Define start and stop points, the person doing the work, the records used, and what counts as complete. Keep the task narrow enough that the same definition can be applied in the pilot.

Retain the baseline before introducing the AI workflow. For each matched item, record active production time, waiting time if it affects the process, review time, correction time, completed status, and quality failures. Also record task complexity. Useful fields can include source count, page count, number of fields extracted, exception flags, and whether a senior reviewer was required. A short, clean document is not comparable with a long document containing conflicting figures.

Use a stable observation window and write down exclusions. Do not quietly exclude ordinary clarification or source-checking work. If the baseline is incomplete, label the gap. GAO places governance, data, performance, and monitoring together in its AI accountability framework, a reminder that a performance number is only as credible as the data and controls behind it. [2]

Define the matched-task pilot before running it

A matched-task design compares similar work under two conditions: the retained manual process and the supervised AI-assisted process. The unit can be a document, company profile, comparable set, diligence request, or other repeatable work product. Do not compare the fastest manual item with the hardest AI item. Match on relevant complexity fields, or separate results into simple, standard, and exception groups.

Write the pilot protocol in plain language. Specify the task owner, accountable human reviewer, approved source-of-truth records, AI tool or version, permitted inputs, output format, review steps, and stopping conditions. The AI should produce a draft or extraction for review, not send outreach, negotiate, approve a disclosure, or make an investment or transaction decision. FINRA guidance for member firms emphasizes evaluation before deployment and attention to governance, privacy, integrity, reliability, and accuracy. Its applicability depends on the firm and use case, so a boutique bank should assess its own obligations with appropriate counsel or compliance resources rather than treat this article as legal advice. [3]

Build the measurement ledger around usable capacity

The ledger should show more than the time spent generating an output. A practical row has an item identifier, complexity class, condition, production minutes, review minutes, correction minutes, elapsed minutes, completed status, first-pass acceptance, final acceptance, error category, source-check status, reviewer role, and exception notes. Keep setup effort in a separate field or a defined allocation rule. Setup includes prompt or template design, source preparation, permissions review, training, and trial configuration.

Calculate gross time change and usable capacity separately. Gross time change compares observed total time for a matched task under the two conditions. Usable capacity asks whether the team can actually take on more completed work after review, correction, setup, interruptions, and quality failures. A faster draft that creates extensive checking work may reduce neither total labor nor queue time. That conclusion belongs to observed pilot data rather than a forecast.

Choose one formula and keep it consistent. For an item, total pilot effort can be production minutes plus review minutes plus correction minutes, with setup reported separately unless the protocol defines an allocation. Throughput can be completed items divided by measured working time for the same period. Report the denominator, treatment of incomplete items, and whether waiting time was included. Avoid converting a small pilot into an annual savings claim.

Count rework as a first-class result

Rework is not an inconvenient footnote. Record each correction after the initial output and classify its cause. Categories might include missing source support, wrong extraction, unsupported inference, formatting defect, stale or conflicting source, omitted exception, or reviewer disagreement. Count both rework events and minutes required to resolve them. One item with several small edits and one requiring a full rebuild should not look identical in the ledger.

Distinguish preventable rework from normal review. A human reviewer may always need to approve a deal-preparation output. The pilot question is whether the AI workflow changes the amount or type of review, not whether review can be removed. Track first-pass acceptance, final acceptance, and material quality failures separately. Define the material-failure threshold before inspecting results.

When a source cannot be verified, stop the item or route it to a human rather than filling the gap. FINRA’s 2026 oversight guidance recommends ongoing monitoring of prompts, responses, and outputs, along with logs, model-version tracking, validation, human-in-the-loop review, and regular checks for errors or bias. [4]

Measure throughput without hiding quality failures

Throughput should mean finished work that passed the stated review gate, not raw drafts generated. Record items started, completed, paused, rejected, and escalated. If the pilot has several stages, measure throughput at each stage. Calling the generated count throughput would overstate usable production.

Use a small quality panel beside time measures. It can include source support rate, missing-field rate, material-error count, first-pass acceptance, and escalation rate. These are internal pilot measures, not universal industry benchmarks. Set the acceptable range before the pilot, based on the bank’s risk tolerance and the task’s consequences. If a quality threshold is missed, pause and inspect whether the task definition, source set, permissions, prompt, reviewer training, or tool is responsible.

NIST describes its AI Risk Management Framework as voluntary and organizes its core around Govern, Map, Measure, and Manage. [1] For a pilot, that translates into an owner and boundaries, a mapped workflow and data path, measured performance and errors, and a decision to continue, change, or stop. It does not guarantee accuracy, confidentiality, compliance, or time savings.

Make permissions and confidentiality part of the ledger

Before the first item, list what information may be used, where it may be processed, who may access it, and what must remain outside the tool. Use approved source-of-truth records and label whether an input is public, internal, client-provided, or restricted. Do not assume a general-purpose tool has the permissions, retention settings, or contractual terms needed for confidential deal material. A permission review is a control decision, not a productivity metric.

Keep the pilot bounded. Use de-identified or synthetic material only when it tests the task adequately. If real information is necessary, obtain required internal approval and document the permitted purpose. Keep prompts, outputs, source references, reviewer decisions, and version information in the approved recordkeeping location. Avoid pasting confidential documents into an unapproved service. Human review should check source support and disclosure sensitivity before an output enters a client or transaction workflow.

For a regulated firm, applicable rules and supervisory expectations depend on jurisdiction, registration, business activity, and deployment. FINRA’s notice says its rules continue to apply to member-firm GenAI use and that a supervisory system should be reasonably designed for the firm’s business when technology is part of supervision. [3] That is not a blanket rule for every boutique bank. Obtain qualified legal or compliance advice for the bank’s facts.

Set stopping rules before looking at results

A pilot should be able to stop without being treated as a failure. Define stopping rules for material source errors, repeated unsupported outputs, missed review gates, unauthorized data exposure, unexpected access, or a quality metric below the agreed threshold. Also define a pause rule after a significant tool, prompt, source, or model-version change. The accountable owner should have authority to halt the workflow and route work back to the manual process.

At the end of each observation period, reconcile the ledger. Check that every completed item has a reviewer decision, every material correction has a category, and every reported minute uses the same clock definition. Compare like with like by complexity class. Report a range and item count when that is more honest than a single average. Explain missing rows, exceptions, and setup effort. Do not turn observed pilot results into a promise about future capacity or a claim of return on investment.

Use the decision rule to choose one of three outcomes: continue under existing controls, revise the task and run another bounded trial, or stop using AI for that task. A continue decision should name the owner, review gate, source records, permission boundary, monitoring cadence, and reassessment point. A stop decision should preserve the reason so the team does not repeat the same experiment without addressing the issue.

Turn the ledger into a repeatable capacity-planning artifact

The final ledger should let a partner or operations lead answer four questions: what work was tested, what changed in total effort, what quality or control trade-offs appeared, and what can be repeated safely. Include the baseline definition, pilot protocol, data dictionary, observed results, quality findings, exceptions, and decision. Keep raw item-level records available to the accountable reviewer, while the summary uses only detail needed for a defensible decision.

Use the artifact to plan work rather than market a result. If the pilot shows fewer usable completed items, narrow the task or improve source preparation. If it shows lower total effort with stable quality, expand only within the tested boundary and continue monitoring. If the result is mixed, preserve that conclusion. Measurement is valuable because it can reveal that a promising draft step does not improve the whole process.

Acquivo is a software platform that delivers the entire sell-side M&A process, valuation, marketing materials, buyer matching, and deal management, on one platform for business owners, M&A attorneys, and boutique investment banks. In this context, the practical outcome is still the documented pilot measurement ledger, while the bank’s people retain responsibility for permissions, source checks, review, and decisions.