What a buyer should ask before requesting the first data sample

Before requesting a data sample, a buyer should be able to state what decision the sample will support, how it will be selected, what it must contain, who can review it, and what permission and access controls apply. A sample request without those answers can create risk while producing little evidence.

What decision should the first data sample answer?

“Send us some records” does not define a test. State the task, the expected input, the relevant context, and the output a reviewer will judge. If the purpose is to assess whether a workflow history can support an agent evaluation, specify which steps, transitions or outcomes need to be present. If it is a schema review, say so. A small sample intended to inspect fields cannot establish overall population coverage or model performance.

Name the evaluation criterion before looking at data. Otherwise reviewers may confuse an interesting example with evidence that the source is broadly useful. Document what a pass, fail or inconclusive result means, and whether the test is exploratory or a procurement gate.

Which records should the sample be selected from?

The sampling frame is the defined set from which records may be selected. Specify source systems, dates, record types, versions, geographies, languages and exclusions. Ask who creates the frame and how gaps, migrations or retention limits are recorded. If the source itself is not yet verified, label the frame as reported rather than confirmed.

Then state the selection method. Random selection, stratified selection, purposive examples and convenience samples answer different questions. A purposive set can help test edge cases but cannot by itself estimate how common those cases are. A random sample can still miss rare conditions, and it may be inappropriate when the sample is too small or the population is clustered. Selection should follow the question, not an assumption that one method always works.

The Datasheets for Datasets paper recommends documenting collection, processing and selection context so readers can interpret a dataset. That is a documentation framework, not a mandatory standard or a universal sample-size rule.

What information belongs in the sample inputs and outcomes?

For each sampled record, determine which inputs were available to the original decision-maker at the time. Separately identify later outcomes, corrections, status changes and annotations. If the goal is to evaluate prediction or action, including information that became available only afterward can create leakage: the test may reveal an answer that the agent would not have known in real use.

If the sample will be split into training and evaluation sets, keep dependent records together when that matches the intended test. The scikit-learn guidance on grouped cross-validation describes group splits for dependent samples. This does not make grouping a universal rule; choose a split that matches the question and explain its limits.

Can a schema or controlled review answer the question first?

Decide whether the first check can happen through a schema, aggregate, synthetic example or controlled demonstration instead of raw records. If identifiable or confidential material is necessary, name the purpose, authorized users, environment, retention period, deletion process, logging and onward-sharing restrictions. Confirm who has authority to disclose the material and who accepts the recipient's access terms.

Removing names is not automatically sufficient. Free text, combinations of attributes, timestamps, attachments and rare events can reveal personal or confidential information. NIST IR 8053 documents re-identification risks across several kinds of material; it is technical guidance, not clearance of a sample. Ask the responsible reviewers to assess the selected fields, recipient context and relevant legal and contractual restrictions before release. A sample request itself does not grant data rights.

Who reviews the sample and what follows the result?

Assign a technical reviewer to check content and quality, a data owner or steward to confirm scope and provenance, and a privacy or legal reviewer where required. State who can make the acceptance decision and what happens if reviewers disagree. Record the version, selection method, review date and limitations so later decisions do not silently treat a sample as representative of a changed source.

Sample request checklist

Before sending a request, include:

• Evaluation question and decision supported
• Required fields, links, time context and output labels
• Source systems and period in the sampling frame
• Selection method and intended interpretation
• Exclusions, known gaps and version information
• Safe-review alternative and access controls
• Named reviewers, acceptance criteria and stop conditions
• Retention, deletion and prohibition on onward use

Suppose a buyer wants to inspect whether work-order histories link an initial report to later inspection and resolution. A useful request would specify the fields and relationships needed, ask for a controlled sample selected across relevant work types, and distinguish event-time facts from later corrections. The example illustrates request design; it is not a real source or evidence that a particular company has these records.

The first sample is an instrument for answering a narrow question. Define that question and the permission boundary first, and the buyer can learn more while asking the source to expose less.

The NIST AI RMF Measure guidance frames measurement as documenting methods, metrics, and limitations for a system context. That supports reporting evaluation conditions alongside a score; it does not prescribe a universal dataset metric.

The buyer sourcing brief can turn those sample criteria into a written request: task, required evidence, review method, access constraints and the next decision. Complete those fields before asking the owner to select records.