An enterprise dataset can contain thousands of plausible-looking records and still be unsuitable for a particular AI evaluation. The missing element may be the information available at the decision point, the final outcome, or the links that explain how work moved between systems.
A good evaluation therefore begins with the task. What should a model learn or demonstrate? What evidence does the record need to contain? What information must be withheld so the evaluation does not reveal the answer?
This guide provides a repeatable acceptance method and a completed fictional example. It is intended for AI buyers and technical teams assessing a source company or a proposed contribution.
1. Specify the task and decision point
Write the task in operational terms. For example:
Given the support investigation available before escalation, select the next troubleshooting action and explain which evidence supports it.
That is more evaluable than "improve enterprise reasoning." It tells the team which parts of the workflow matter and when the model's information must stop.
Define the decision timestamp, the permitted context and the expected output. Then identify the outcome evidence used for review. An internal resolution note created afterward may be useful as an evaluation reference but inappropriate as an input to the model.
Use the buyer sourcing brief to connect the task to the source-company characteristics, systems, retained history and rights required.
2. Define a minimum useful episode
An episode is the unit of work the buyer intends to evaluate. It might be a support investigation, a sales decision, a code change, a project exception or a financial reconciliation.
| Field or component | Purpose |
|---|---|
| Episode identifier | Keeps the evaluation unit stable across versions |
| Task description | States what the model is being asked to do |
| Source references | Connects included material to its recorded source |
| Event timestamps | Establishes the order of events and the decision cutoff |
| Available context | Contains only the information permitted at the decision point |
| Recorded action | Shows what the people or system did |
| Outcome and outcome time | Supports assessment of what followed |
| Review labels or rubric | Explains how the output will be evaluated |
| Scope and rights reference | Connects the episode to the permitted contribution and use |
| Known gaps | Makes missing or uncertain evidence visible |
The exact schema depends on the task. The principle is to make the episode understandable, traceable and reproducible without depending on an undocumented explanation from the source company.
W3C's provenance model is a useful reference for representing relationships among entities, activities and agents. Dataset documentation frameworks such as Datasheets for Datasets also provide a disciplined way to record composition, collection, intended use and limitations. Neither framework replaces the buyer's own task-specific acceptance criteria.
3. Separate hard gates from quality judgments
Some requirements are necessary before the sample should be used. Others affect how broadly or confidently it can support the task.
Hard gates
Confirm that the sample is within the agreed scope, the proposed use is authorized, the source is identified sufficiently for the agreed review and sensitive material is handled through the approved process. Establish the minimum evidence needed to perform the task at all.
If an essential condition is unresolved, record it as unresolved. A high score on other dimensions should not average it away.
Quality dimensions
Assess whether the usable episodes represent the intended workflow, preserve chronology, include meaningful outcomes and can be evaluated consistently. These are questions about fitness for the defined task, not a universal grade for the company's data.
4. Use a task-specific acceptance scorecard
Agree the criteria before examining the sample where practical. Otherwise, the reviewer may unconsciously change the standard to fit the material available.
| Dimension | Evidence to inspect | Example acceptance question |
|---|---|---|
| Task fit | Episode content and decision | Can the required decision be evaluated from the available record? |
| Chronology | Event and decision timestamps | Is the model context restricted to information available at the cutoff? |
| Outcome quality | Resolution or later outcome evidence | Is the outcome documented and suitable for the evaluation rubric? |
| Cross-system linkage | References among tickets, issues and other records | Can the reviewer follow the workflow without guessing which records belong together? |
| Coverage | Sampling method and episode distribution | Does the sample include the cases the intended use requires, including relevant difficult cases? |
| Leakage control | Context, labels and near-duplicate checks | Does any input reveal the answer or overlap with the held-out evaluation material? |
| Reproducibility | Version identifiers, extraction rules and transforms | Can the source reconstruct the same agreed sample and explain changes? |
| Documentation | Schema, gaps and preparation notes | Can a new reviewer understand the sample without an oral tour? |
Use pass, correction required or unsuitable for the current task, supported by a short evidence note. A numeric score is optional; a documented reason is not.
5. Completed fictional sample review
Assume a buyer requests 20 support-investigation episodes to evaluate next-action selection before engineering escalation. The source provides 20 candidate episodes. The following findings are fictional and illustrate the method.
| Finding | Count | Decision |
|---|---|---|
| Complete task context and usable outcome | 16 | Suitable for the scoped pilot review |
| Outcome absent from the retained record | 2 | Exclude from outcome-based evaluation or replace with suitable episodes |
| Post-decision resolution text included in input | 2 | Rebuild the input at the agreed cutoff and recheck |
| Total candidate episodes | 20 | Do not describe all 20 as accepted |
At the first review, 16 of 20 candidates are usable for the specified task. The two chronology corrections may bring the usable count to 18 if the corrected records pass. The two missing outcomes still need replacement or a different evaluation purpose.
The reviewer should not report "the archive is 80% accurate." The result is narrower: 16 of these 20 candidates met the agreed requirements on this review, and the sampling method determines how much can be inferred beyond them.
A completed review record
| Record field | Example entry |
|---|---|
| Task | Select the next troubleshooting action before engineering escalation |
| Sample version | Fictional review set v1 |
| Candidate count | 20 episodes |
| Usable on first review | 16 episodes |
| Main defects | Two absent outcomes; two inputs containing post-decision information |
| Required correction | Rebuild two input contexts; replace or exclude two outcome-incomplete episodes |
| Next acceptance event | Review the corrected version against the same criteria |
| Commercial implication | Do not treat the full candidate set as an accepted deliverable yet |
This record lets the technical evaluator, source company and commercial owner discuss the same facts. It also prevents a vague request for "better data" from expanding indefinitely.
6. Check sampling and leakage deliberately
Ask how the sample was selected. A handpicked set of unusually complete records may demonstrate possibility without representing the archive. A random sample can still miss rare but important cases. The right selection method follows the intended task and the decision the evaluation is meant to support.
Check for leakage at more than one level. Look for future outcomes in the input, duplicate or near-duplicate episodes across splits, repeated answers embedded in templates and links that expose held-out information.
For a software task, a final patch or review resolution may reveal the intended answer. For a support task, a later incident summary may do the same. The review must follow the actual sequence of work, not merely the export's folder structure.
7. Connect acceptance to delivery and commercial scope
Technical acceptance should describe the deliverable the parties are actually evaluating. If the sample demonstrates a two-year linked workflow, it does not automatically validate an unlinked ten-year archive.
Record the accepted version, the criteria, material exceptions and the next step. If the buyer requests a different period, additional systems or new labels, determine whether that changes the contribution or preparation obligations.
Avelence can help translate a buyer's task into source-company qualification and a clearer initial brief. The buyer's technical team remains responsible for its evaluation criteria and acceptance decision; the source company must understand what it is being asked to provide.
A technically useful sample and a commercially attractive asset are related, but different, findings. The company-data value guide explains how source quality, buyer use and licensing scope meet in the commercial discussion.
Frequently asked questions
How large should the evaluation sample be?
Choose the size from the decision the sample must support, the diversity of the workflow and the cost of review. A small sample can reveal structural problems or demonstrate that a useful episode exists, but it may not support a reliable estimate of full-archive quality. Document the selection method and the limits of the conclusion. Increase or stratify the sample when the decision requires broader coverage.
Are linked outcomes always necessary?
They are necessary for tasks that depend on evaluating what happened after a decision. Other uses may focus on documentation, language, task context or another property. Define the use before deciding that missing outcomes are either fatal or irrelevant. A record unsuitable for one evaluation may still be useful for a different task, provided the contribution and permitted use are clear.
Should we reject a source because the first sample has defects?
Not automatically. Distinguish a correctable preparation problem from a structural absence of the required evidence. A post-decision note can sometimes be removed from the input while preserving the evaluation reference. An outcome that was never recorded may not be recoverable. Give the source a specific correction request and keep the acceptance criteria stable so the next review produces a clear decision.
What should accompany an accepted sample?
At minimum, retain the version, schema, source and transformation notes, sampling method, scope and permitted-use reference, acceptance criteria, review result and known limitations. The detail should be proportionate to the task and agreement. The aim is to let another authorized reviewer understand what was accepted and reproduce the relevant preparation or evaluation without relying on memory.
Sources
- W3C: PROV overview
- Gebru and colleagues: Datasheets for Datasets
- Bender and Friedman: Data Statements for Natural Language Processing
The scorecard and sample review are original Avelence methods and fictional examples. The sources inform provenance and documentation practices; they do not validate the fictional dataset or prescribe its sample size.
About the author
Yananai A. Chiwuta
Yananai A. Chiwuta is the Founder of Avelence. His work covers data-owner relationships, enterprise buyer research, and the commercial decisions behind operational-data licensing.
