When a clean sample does not represent the evaluation population

A clean sample can be useful for checking a format or demonstrating complete cases. It does not establish that it reflects the records an evaluation is intended to cover. If the selection rule removes incomplete, disputed or unusual cases, conclusions about those cases remain unsupported by that sample.

The practical question is what the selection allows you to conclude. Name the target population, explain how the sample was formed and state whether you want a population-like mix, coverage of particular conditions or another purpose. Cleanliness is a property to describe, not a substitute for that decision.

Define the population before describing the sample

Write a plain-language boundary for the records you want the evaluation to say something about. For example, it might concern document-routing requests handled by a particular operation during a defined period, including unresolved requests. That is different from all requests with a recorded final decision and machine-readable text.

Then identify the records that were actually available for selection. Some may have been deleted, excluded for rights reasons or stored in a system outside the review scope. A sample can only be selected from what is available to the selection process. Keep gaps between that available set and the intended population visible.

Do not use representative as a complete explanation. In Data Representativity for Machine Learning and AI Systems, Clemmensen and Kjærsgaard distinguish several meanings of representation, including coverage of relevant possibilities and similarity to a target population's distribution. Those are different objectives. The paper does not supply one method or sample size suitable for every evaluation.

Which conclusion should the selected sample support?

A reviewer may need to answer several questions using different sets of records. Keep the purpose of each set explicit.

This table is a proposed decision aid, not an exhaustive classification of sampling methods. A project may need more than one purpose. If it does, identify which evidence answers which question rather than combining everything under one sample label.

Why do complete-case and difficult-case samples answer different questions?

Imagine a fictional internal document-routing archive. A team first selects records with readable text, all required fields and an agreed final routing decision. These examples help someone understand the expected structure and follow a complete case.

The intended review also includes requests with missing pages, uncertain routing and unresolved status. Those conditions were excluded by the first selection rule. The polished set cannot demonstrate how those excluded conditions are handled merely because every selected record is easy to inspect.

The team then assembles a second set specifically to cover those conditions. That can be useful for inspecting the selected cases. Its deliberately enriched composition should be stated: the team chose certain conditions because it wanted them examined. Their share of this second set is not, on its own, evidence of their share in the full archive.

Both sets can have a useful role. The first is a format and complete-case demonstration. The second is a selected coverage exercise. If the project also needs conclusions about normal operational frequency or aggregate performance, it needs an appropriate sampling design and review for that separate purpose.

All details here are invented. No result, company-wide rate, buyer preference or recommended sample size is implied.

Record the selection rule as it was applied

Write down eligibility criteria, exclusions, filtering, selection method and any manual choices. Distinguish what the team intended to select from what the process actually produced. If a reviewer replaced a difficult record with a cleaner one, record the replacement and its reason.

Data Cards proposes documentation of dataset origins, collection, processing, intended uses and limitations. Its prompts provide useful support for explaining how a sample came into being. Our short selection note applies that documentation idea to one bounded review; it is not a claim that a Data Card certifies representation.

A useful note could identify the population of interest, the available selection set, criteria applied, cases excluded and the question the selected set is meant to answer. It should also name information that is unavailable, such as the number of records excluded for a particular reason. Do not turn an unknown count into zero.

Keep observed rates within their scope

If you report a count or rate, state which records it describes. A rate among reviewed records with complete fields is different from a rate among all selected records, and both may differ from what could be inferred about the target population.

For the fictional routing example, an observation about the complete-case set should be described as an observation about that set. If unresolved requests were excluded, a reviewer cannot use the set's apparent completion to establish how often requests are resolved in the archive. That conclusion would be excluded by the way the sample was constructed.

Keep excluded and unobserved groups in the account even when they are not in the evaluated set. If the objective requires estimating a population frequency, have the sampling design and proposed inference checked for that objective. This article does not provide a statistical estimator, confidence interval or sample-size calculation.

Treat additional cleaning as a change to the sample

Cleaning can help make a permitted record usable. It can also change what the reviewed set contains. If a preparation step removes every record with an uncertain label, the resulting set answers a question about the remaining records, unless the review method supports a broader conclusion.

Record which cases or fields were removed and why. Keep the selection description tied to the version actually reviewed. A before-and-after account helps explain whether the change corrected a formatting problem, altered eligibility or removed a condition the project originally meant to inspect.

This is not an argument to retain unsafe or unauthorized material. If a condition cannot be included within the permitted review, state the exclusion and its effect on the conclusion. A coverage objective does not supply permission to access the missing records.

How should sample findings be reported without overstating coverage?

A useful review summary should say what population or conditions were targeted, how the set was formed and what the observations establish within that scope. It should also say which broader questions remain open. The conclusion may be that a format works, a particular case type deserves further investigation or the available selection does not support the intended inference. Record the population, exclusions and acceptance question in the buyer sourcing brief; that document organizes the requirement, rather than selecting a statistical design.

For the broader question of whether connected evidence supports an evaluation, use the enterprise workflow sourcing guide. This article concerns selection and the reach of the conclusion, rather than the complete evaluation design.

An initial sourcing inquiry can describe the target population, relevant conditions and known exclusions without attaching examples. Avelence can support agreed company sourcing and commercial research. Sample access, handling, permissions and technical acceptance remain separate decisions for the responsible parties.

Frequently asked questions

Are clean samples useful for evaluation?

Yes, for a purpose their selection supports. A complete-case set can demonstrate format or show how a known sequence is represented. If unresolved or incomplete cases were removed, it cannot establish how the system handles those excluded conditions or how frequently they occur in normal operations.

Should a difficult-case sample match normal operational proportions?

Not when its purpose is deliberately to inspect selected failure modes or rare conditions. State that enrichment and explain which cases were chosen. If the project also needs population frequencies or aggregate performance, treat that as a separate sampling and inference question.

Will increasing the sample size fix selection bias?

Not by itself. Adding more examples under the same exclusion rule can leave the relevant population gap intact. First check which records were available, eligible and selected, then decide whether a different selection method or an additional review set is needed for the intended conclusion.