M&A attorneys should test AI-assisted contract review against a versioned playbook and a separate lawyer-labeled reference set. Define each issue trigger, its exceptions, the supporting clause evidence, and when the tool must request human review. Include clauses that should be flagged, clauses that should not, ambiguous wording, and missing information. Compare suggestions with the reference labels at clause level, recording false positives, omitted issues, and incorrect exception handling. Keep a responsible attorney accountable for interpreting results and approving any subsequent use. Start with synthetic material, then permit a bounded trial only after checking confidentiality requirements. The output should remain a proposed issue for review, never an approved position or negotiation instruction.
Set the review task and accountable owners before testing
Specify the task narrowly: identify possible issues in a defined class of contracts using counsel's approved review criteria. State the matter context and excluded work. A test for assignment-related issue tagging should not become an unrequested opinion on enforceability, transaction structure, or whether consent is legally required. Distinguish detecting relevant wording from deciding its legal significance.
Assign a playbook owner, normally the responsible M&A attorney, to approve criteria and resolve interpretation disputes. An evaluation owner organizes cases and preserves results. The firm's designated confidentiality or technology reviewer checks the proposed processing environment. These roles can overlap, but technical staff should not silently decide what counts as an acceptable legal exception.
ABA Formal Opinion 512, issued July 29, 2024, explains that appropriate independent verification depends on the tool and task, while lawyers remain responsible for their work. It interprets ABA Model Rules, not a universal jurisdictional rule. Counsel should assess applicable professional obligations and engagement terms before the trial. This framework is general implementation guidance, not advice on an individual contract. [1]
Turn the approved playbook into clause-level test criteria
Give each review criterion a stable rule identifier and version. Record its purpose, trigger, permitted exceptions, required context, and allowed output labels. Useful labels include proposed issue, exception candidate, outside scope, and needs lawyer review. A label such as no issue under this criterion must not imply that the contract has been legally cleared.
For each criterion, specify the evidence the tool must show: document version, clause location, exact quoted text, relevant definitions, and the playbook rule applied. Where an exception depends on another clause or a matter fact, identify that dependency expressly. Missing evidence should lead to a review request rather than an invented fact or silent suppression of the issue.
Keep the approved criteria separate from generated explanations. Record the approving role and effective version, then preserve earlier versions for comparison. If counsel changes a criterion, identify which reference labels must change too. Otherwise, apparent model improvement may simply reflect a moving standard. Restrict this playbook to issue-review criteria; do not ask the model to choose concessions, authorize changes, or approve negotiation positions.
Build a reference set that includes clean and difficult cases
Begin with synthetic clauses written for testing, not excerpts from private matters. For each rule, create a clear trigger case, a clear non-trigger case, an applicable exception, an exception whose conditions are unmet, and a case with genuinely uncertain wording. Include examples where a relevant definition or amendment changes what the reviewer must examine.
Have a lawyer label the cases independently of the AI suggestions. Store the expected issue, expected supporting passages, exception treatment, and rationale. Use another qualified reviewer for disputed or consequential labels where practical. An unresolved disagreement is a reference-set problem, not automatically an AI error; mark it for adjudication before scoring.
Keep some labeled cases withheld from prompt development. Tune on a development set, then evaluate on the withheld set without supplying its expected answers. Include representative document formats and context lengths before expanding use. NIST's July 2024 Generative AI Profile recommends empirical capability evaluation and cautions against extrapolating from narrow, anecdotal assessments. Its voluntary technical guidance is not a legal certification. [2]
Test exception handling with paired clause variants
Use paired variants that change only the wording or condition relevant to a rule. For an expressly illustrative custom playbook, counsel might require flagging assignment restrictions while treating an affiliate carve-out as an exception candidate only when specified conditions are supported. This tests adherence to the invented review criterion, not the clause's legal effect in any jurisdiction.
One synthetic case can contain the carve-out and all required supporting facts. Another can use the same clause but omit a necessary fact. A third can remove the carve-out. The expected results should distinguish a supported exception candidate, a needs-review result, and a proposed issue. Counsel determines the labels; the AI should not infer that an affiliate exists or that a transaction qualifies.
Add variants involving negation, nested provisos, competing definitions, and amendments that supersede earlier wording. Check whether the explanation identifies the actual controlling text. Also test a contract containing text that asks the system to disregard the playbook: treat that text as document content, not permission to change review instructions. Record both wrong exception applications and failures to recognize valid exception candidates.
Require source checks before judging the suggestion
Preserve the original document and identify the exact version used in each run. Check text extraction against the original, especially for scans, tables, missing pages, tracked changes, and cross-references. If the tool never received the controlling amendment, a plausible answer cannot establish that its review was complete. Separate ingestion errors from reasoning errors so the correction addresses the right cause.
For each proposed issue, have the reviewer open the cited clause and verify the quotation, location, rule identifier, and exception dependencies. An accurate quotation is necessary evidence, but it does not itself establish that the suggested interpretation follows the playbook. Unsupported commentary and references to nonexistent clauses should fail the evidence check.
NIST recommends reviewing and verifying sources and citations during pre-deployment measurement and ongoing monitoring, and verifying the provenance of evaluation and retrieval data. [2] Apply that guidance here to document passages and approved playbook records. If a suggestion introduces an external legal proposition, route it to counsel for separate authoritative-source verification rather than treating it as a contract-review result.
Measure false positives, omissions, and exception errors separately
Match each suggestion to both a rule and a clause, rather than giving credit because the report mentions a similar topic somewhere. Define matching rules before the evaluation. Treat a flagged clean clause as a false positive and a lawyer-labeled issue with no corresponding suggestion as an omission. A correct topic attached to the wrong clause should not count as a fully correct result.
Report precision as the share of proposed issues that reviewers confirm against the reference set. Report recall as the share of expected issues that the tool identifies. These measures answer different questions: how often suggestions are useful, and how often expected issues are found. Include raw counts, test-set composition, and results by criterion so an overall rate does not hide a consequential omission.
Track wrong exceptions, missing support, and incorrect review requests separately. Do not reward an uncertain label as a confirmed finding, or penalize it when counsel's reference label expressly requires review. Repeat selected cases using the same recorded configuration to examine inconsistent results. Counsel should set acceptance criteria before seeing the scores; there is no universal pass rate that establishes legal adequacy.
Approve confidentiality choices and run a bounded shadow trial
Before permitting matter information, review the provider's actual terms and configuration for access, retention, training use, deletion, and onward disclosure. Assess who inside the firm can see inputs, outputs, and evaluation records. Synthetic testing avoids using client documents at that stage; later redaction is not itself a finding that remaining information may safely be submitted.
Opinion 512 describes a fact-specific confidentiality assessment and discusses informed consent for representation information entered into self-learning tools with the disclosure risks it identifies. It also distinguishes confidentiality consent from other client-communication obligations. Do not turn that guidance into a blanket rule for every AI product. Qualified counsel should determine the applicable jurisdictional requirements and any necessary client authorization before matter data is used. [1]
Limit the initial matter trial to named reviewers, an approved contract category, and a fixed document set. In this shadow trial, a lawyer performs the reference review before comparing AI suggestions. Do not let the trial change contracts, populate final diligence conclusions without review, send communications, or approve positions. Agree in advance that unsupported sources, unauthorized data access, unresolved exception logic, or consequential omissions stop expansion and trigger human-only review of affected work.
Preserve correction records and retest changes
Maintain a correction record for each discrepancy: case identifier, document and playbook versions, run configuration, original suggestion, expected label, reviewer decision, corrected explanation, and error category. Use role-based reviewer identifiers where appropriate and retain sensitive records only within the approved access and retention arrangements. These records are working evaluation evidence, not a claim that privilege automatically attaches.
Distinguish a model mistake from faulty extraction, an incomplete criterion, or an incorrect reference label. The playbook owner should resolve substantive corrections; the evaluation owner should preserve the original result. Add confirmed failures to the regression set, which is the collection rerun to check that later changes do not revive known errors.
Retest after changes to the model, prompt, document extraction, retrieval material, or playbook. Resume a paused trial only after responsible counsel accepts the corrective evidence for its limited scope. Opinion 512 addresses supervisory policies and training for lawyers and nonlawyers using these tools. [1] A correction record helps that supervision, but does not replace professional judgment or prove performance on untested contracts.
Keep the tested workflow separate from product and legal approval
The implementation deliverable is a versioned clause-level test set, documented exception handling, and reviewer correction records. Keep an approval record stating who authorized the bounded use, which configuration was tested, what remains excluded, and what triggers renewed evaluation. Approval of the trial is distinct from approval of a clause, diligence conclusion, or transaction decision.
Acquivo is a software platform that delivers the entire sell-side M&A process, valuation, marketing materials, buyer matching, and deal management, on one platform for business owners, M&A attorneys, and boutique investment banks.
The software is user-operated and has no success fee. The testing framework described here is a recommended practice, not an assertion that Acquivo provides a particular AI integration or evaluation feature. Acquivo does not represent clients or provide legal advice. Responsible counsel retains legal interpretation and review, while authorized users control disclosure and subsequent action. Testing does not guarantee accuracy, confidentiality, privilege, compliance, or transaction outcomes.
Key Points
- Version every playbook criterion and define evidence requirements, exception conditions, and needs-review outcomes before testing.
- Use lawyer-labeled clause cases that include non-trigger examples, ambiguous wording, unsupported exceptions, and omitted issues.
- Match findings to both clause and rule; measure false positives, omissions, and exception errors separately.
- Approve data permissions before a shadow trial and stop expansion when source checks or consequential issue detection fail.
- Preserve reviewer corrections and retest changed configurations; AI suggestions never approve legal positions or authorize negotiation.
Sources and references
Source retrieval dates record research access, not article publication. Read our editorial and date policy.
Choose your next step
Acquivo is user-operated software. Owners and their professionals remain responsible for decisions, communications, and professional review.
Educational information, not individual legal, tax, or investment advice. Consult qualified professionals for your circumstances. Acquivo provides software, not representation or negotiation on your behalf.