M&A attorneys should test a proposed AI workflow on synthetic or properly authorized records before permitting client-matter use. Define one task, have qualified lawyers prepare source-linked expected answers, and compare the complete workflow against those answers. Score material omissions separately from unsupported statements, incorrect facts and citation failures. Set task-specific acceptance criteria before seeing results, measure the effort needed to verify and correct outputs, and stop trials when critical errors or permission problems arise. ABA guidance makes review dependent on the tool and task; NIST cautions against extrapolating from narrow tests. A passing benchmark supports only a bounded use decision, not guaranteed accuracy or replacement of lawyer judgment. [1] [2]
Define one M&A task and assign accountable reviewers
Start with a bounded task such as extracting assignment and change-of-control provisions from supplied contracts into a draft diligence worksheet. Specify the document types, required fields and permitted source materials. Distinguish extraction from deciding whether a transaction triggers a provision or whether consent is legally required. Those interpretation questions involve additional facts and professional judgment. A test of clause extraction does not establish fitness for legal research, drafting transaction advice or resolving a consent question.
Appoint a supervising M&A lawyer to own the evaluation and approve its acceptance criteria. A second qualified reviewer should resolve disputed reference answers; a technology lead should record the configuration and input-processing steps. The person responsible for confidentiality permissions should clear the test environment before any matter information enters it. ABA Formal Opinion 512, issued in July 2024 under the Model Rules, describes task-dependent independent verification and continued lawyer responsibility. Applicable jurisdictional rules and engagement obligations require separate review; the opinion is not a universal permission to use AI. [1]
Clear input permissions before building the benchmark
Synthetic contracts are a useful starting point when the firm has not approved a tool for matter data. Write documents specifically for testing, rather than disguising a client's contract by changing names. If using existing records, establish authority to use them for this purpose and in this environment. Check restrictions on disclosure and reuse, including who may see prompts, uploaded files, outputs and evaluation records. Removing identifiers alone should not be treated as resolving confidentiality, privilege or contractual restrictions.
ABA Opinion 512 calls for a fact-specific assessment before inputting representation information and for understanding tool terms, privacy policies and related contractual terms. [1] Have responsible counsel and a qualified technology reviewer assess retention, provider access, training use and internal access arrangements. Decide whether applicable obligations require client communication or informed consent before proceeding. Restrict evaluators to approved materials and disable unnecessary connections where the environment permits. If access or retention terms remain unclear, keep the trial synthetic or pause it. Accuracy testing cannot authorize disclosure or establish privilege protection.
Build representative cases, including missing and conflicting records
Create a case inventory that reflects the intended workload, not just clean examples selected for a demonstration. Include ordinary contracts and difficult inputs such as scanned pages, schedules, cross-references, multiple amendments and incomplete document packages. Record which source versions belong to each case. NIST's Generative AI Profile recommends evaluating performance under conditions similar to deployment and warns that narrow or anecdotal assessments should not be extrapolated into broader capability claims. [2] The benchmark should therefore exercise the firm's proposed document preparation and retrieval process, not only a model answering copied text.
An illustrative synthetic case might contain an original restriction and a later amendment changing its wording. Another might refer to a missing schedule. A third might contain no responsive provision. Expected behavior should differ: use the amended source, identify the missing material, or report that no provision was located in the supplied records. Include cases where the task should return insufficient information rather than a confident answer. Keep some cases aside from prompt development for a final evaluation. Do not choose a sample size merely because a vendor demonstration used it; justify coverage against the intended task's risks.
Prepare lawyer-reviewed expected outputs before running AI
Have the reference lawyer review the full authorized source package and prepare an expected answer for each case before seeing the AI result. For the extraction task, record the relevant document version, section or page, exact supporting text, normalized worksheet fields and any unresolved issue. Mark whether an answer is an express statement, a permitted inference or outside the task. Where the package is incomplete, the expected answer should preserve that limitation rather than fill it with a plausible assumption.
The second reviewer should check material fields and settle disagreements with the evaluation owner. Where qualified reviewers cannot agree, document the ambiguity and require escalation instead of forcing a single artificial answer. Accept equivalent wording when it preserves meaning, but not wording that erases an exception or changes an obligation. ABA Opinion 512 specifically discusses comparing manually reviewed contract subsets with AI summaries as a possible basis for determining appropriate review. [1] That example does not prescribe a minimum sample or accuracy threshold. The reference set remains a lawyer-reviewed benchmark, not an infallible answer key.
Score omissions and unsupported answers separately
Use separate error categories so a polished paragraph cannot conceal a material failure. An omission means an expected provision or exception was left out. An unsupported answer asserts something the authorized source package does not establish. A factual error misstates a party, date, condition or operative wording. A source failure points to the wrong document version, nonexistent passage or irrelevant text. NIST recommends reviewing and verifying sources and citations during pre-deployment measurement. [2] Open the cited record and inspect the supporting passage; a valid link alone does not prove that the claim is supported.
Define the denominator for each measure. Completeness can compare correctly captured required items with all expected items, while unsupported-answer checks examine every substantive assertion in the output. Report both the number of cases affected and the errors within them. Keep material errors separate from harmless formatting differences. For the illustrative missing-schedule case, an invented consent requirement should be recorded as unsupported even if the rest of the worksheet is accurate. Also assess whether the output correctly abstains when evidence is absent. Do not let an aggregate score offset a failure that could materially mislead the reviewing lawyer.
Set task-specific acceptance criteria and trial stopping rules
Approve the evaluation rules before inspecting results. Identify which failures prevent progression and which minor defects can be corrected within the intended review process. A firm might decide that any invented operative clause, wrong amendment controlling a material field, or omitted material exception blocks progression for this extraction task. That is a proposed risk decision, not a bar-mandated numerical threshold. State the required coverage of difficult cases and the maximum review burden the team is willing to accept. Different tasks need different criteria; a drafting aid should not inherit an extraction worksheet's standard.
Make stopping conditions operational. Pause immediately for unapproved disclosure, unexpected access to another matter's information, or unresolved input permissions. Stop progression when a critical accuracy failure occurs; investigate and repeat affected tests before reconsideration. If the system starts using sources outside the authorized package or fails to flag missing records, treat that as a boundary failure rather than a prompt-writing inconvenience. Give the evaluation owner authority to stop the trial. No external communication, negotiation, disclosure or approval should occur during this benchmark, and passing it should not enable those functions automatically.
Run a repeatable trial and measure lawyer review effort
Record the tool and model version where available, prompt, settings, retrieval scope, document preparation and output format. Preserve each case's inputs and results within the approved environment. Test the workflow that lawyers would actually use, including document loading and source retrieval. Run selected cases repeatedly to see whether material answers change. Evaluate held-back cases only after fixing the proposed workflow, and record failures rather than selecting the most favorable run. If the vendor cannot expose relevant configuration details, document that limitation in the decision packet.
Measure active lawyer time spent finding supporting passages, checking completeness, correcting errors and preparing a usable draft. Compare it with the same bounded task completed without AI under reasonably comparable conditions. Record the timing method and distinguish verification from substantive legal analysis. Where practical, have reviewers assess outputs without knowing their origin. Include rework and escalations, not just generation time. These are measurements to collect, not promised savings. NIST identifies gaps between benchmark conditions and real-world use, including sensitivity to prompts; the resulting evidence should be described only within the tested scope. [2]
Document a bounded decision and preserve human review
Produce a reusable packet containing the authorized case inventory, lawyer-approved expected outputs, configuration record, error results, review-effort findings and signed decision. The outcome can be rejection, revision and retest, or progression to a limited supervised pilot. A permitted use should name the task, users, document conditions and mandatory review. New models, changed prompts, different retrieval sources or materially different documents should trigger reassessment under the firm's change criteria. Passing the set does not approve each future output. ABA guidance preserves lawyer responsibility regardless of the selected review level. [1]
Acquivo is a software platform that delivers the entire sell-side M&A process, valuation, marketing materials, buyer matching, and deal management, on one platform for business owners, M&A attorneys, and boutique investment banks. Acquivo is user-operated software with no success fee; it does not represent clients or provide legal advice. This benchmark is a recommended evaluation framework, not a claim about Acquivo's tested AI capabilities. Attorneys remain responsible for permissions, source checks and professional judgment before a proposed workflow enters client-matter use.
Key Points
- Test a defined task and its complete document workflow, not a general claim of legal AI accuracy.
- Use synthetic or properly authorized records and lawyer-reviewed, source-linked expected answers.
- Separate omissions, unsupported assertions, factual errors and citation failures; do not hide critical errors in an average.
- Approve risk-based acceptance criteria, stopping rules and accountable owners before the trial.
- Measure verification and correction effort, preserve repeatable test records, and limit any passing decision to supervised use.
Sources and references
Source retrieval dates record research access, not article publication. Read our editorial and date policy.
Choose your next step
Acquivo is user-operated software. Owners and their professionals remain responsible for decisions, communications, and professional review.
Educational information, not individual legal, tax, or investment advice. Consult qualified professionals for your circumstances. Acquivo provides software, not representation or negotiation on your behalf.