iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Choose metrics for the data’s intended users, decisions, release model, and plausible attackers—not for a universal score. Evaluate task-specific utility alongside disclosure risk, set measurable acceptance levels through governance, and document what the evaluation does not establish.
Start with the work the data must support
Before comparing synthetic data with source data, identify who will use it and what they need to do: estimate a policy effect, fit a regression, test a predictive model, or explore particular subgroups. Those uses determine which errors matter. A dataset can resemble the source on broad summaries and still give misleading results for an analysis that matters to the business.
NIST’s 2021 article Utility Metrics for Differential Privacy: No One-Size-Fits-All recommends choosing utility measures around users’ actual analyses. It quotes Claire McKay Bowen, then Lead Data Scientist for Privacy and Data Security at the Urban Institute: “Instead of seeking the ultimate utility metric, we should ask ourselves, ‘Who are the data practitioners or data users, and what will they use the data and statistics for?’” NIST also cautions that it is impossible to anticipate every analysis and ensure valid results for all of them.
Write down the intended analyses, important populations, decisions that must not change materially, and the release context. Then choose measures that test those requirements. No finite metric suite can certify validity for every possible downstream use.
#1 Best Overall
Keep fidelity, task utility, and privacy distinct
These are related but different questions. Fidelity asks how closely synthetic data reflects selected properties of the source. Task utility asks whether it supports the analyses and decisions users need. Privacy assessment asks what an attacker could learn or reconstruct under stated assumptions. A strong result in one area does not establish a strong result in the others.
For example, similar means or category frequencies do not establish that a regression coefficient, subgroup estimate, or operational decision will be sufficiently accurate. Conversely, a useful result for a particular task is not evidence that records or sensitive attributes are protected. Report the three dimensions separately rather than collapsing them into one score.
Build a layered utility and fidelity evaluation
Use a metric suite that moves from basic summaries to the analyses people actually intend to perform. Select measures that correspond to the data types and decisions at hand; statistical tests are diagnostics, not universal pass/fail rules.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteCompare univariate summaries
Where they matter to users, compare counts, means, rates, quantiles, missingness, and category frequencies. Bias and root mean squared error can summarize differences between corresponding estimates. NIST’s 2021 discussion of U.S. Census Bureau 2020 measures lists mean absolute error, mean numeric error, root mean squared error, mean absolute percent error, coefficient of variation, total absolute error of shares, and counts of percent differences above selected thresholds. These are examples, not a prescribed bundle for every enterprise dataset.
Rank #2
Check relationships and joint distributions
Examine correlations and joint distributions where relationships drive the use case. NIST cites chi-square tests for categorical variables and Kolmogorov–Smirnov tests for continuous variables as examples. A test statistic can flag a discrepancy, but its interpretation depends on the data, sample size, and intended task; it does not by itself establish acceptable utility.
Rerun representative analyses
Where possible, run the analyses stakeholders expect to conduct on both source and synthetic data. Compare estimates, uncertainty, conclusions, or resulting decisions—for example, regression results, policy estimates, or predictive-task performance. Agree in advance which changes would be material to the decision, rather than treating any statistical difference as equally important.
Use global discrimination as a diagnostic
A classifier can be trained to distinguish real rows from synthetic ones. Weak discrimination suggests similarity on the characteristics that the chosen model can detect. The result depends on the classifier and evaluation setup: it neither proves overall fidelity nor measures privacy.
Free tools Windows power users keep installed
One-click scans. No signup required.
Inspect subgroups and uncertainty
Evaluate important populations separately, especially small or vulnerable groups for which aggregate results can conceal large errors. Also examine uncertainty in downstream estimates. NIST SP 800-226, Guidelines for Evaluating Differential Privacy Guarantees (March 2025), notes that synthetic-data generation can add uncertainty and reduce accuracy for subpopulations.
Assess disclosure risk against a stated threat model
Define plausible attackers, what auxiliary information they may have, which fields could act as quasi-identifiers, which attributes are sensitive, and how the data will be released. Select tests that reflect those assumptions. Report the assumptions and tolerances alongside the results; an empirical test only speaks to the attacks and conditions it covers.
Test record replication and apparent matches
- Replicated unique records or percentage of replicated uniques: count synthetic rows that match original unique records on selected quasi-identifiers.
- Apparent Match Distribution: identify synthetic rows that exactly match unique real rows on quasi-identifiers, then compare the sensitive or confidential attributes for those apparent matches.
Test closeness on confidential attributes
Count disclosure and percent disclosure measures can count replicated unique records judged “too close” on confidential variables. The tolerance that defines “too close” is a design assumption: state it, justify it, and report it rather than presenting the result as an objective universal boundary.
Exercise realistic linkage attacks
Where relevant, test direct re-identification and partial matches, pairwise intersections, and other plausible linkage routes. NIST’s Collaborative Research Cycle describes red teaming in terms of whether attributes of targeted individuals can be reconstructed. A useful exercise should reflect the release context and attacker knowledge, not just exact matches on one convenient field set.
Distinguish formal differential privacy from empirical tests
Differential privacy (DP) is a mathematical framework for quantifying privacy loss. If a generation method claims DP, report its guarantee and the assumptions and implementation context needed to interpret that guarantee. Do not substitute a similarity score, classifier result, or attack test for the formal privacy parameters.
NIST SP 800-226 distinguishes DP from the informal guarantees generally available for synthetic data generated without DP. Passing selected attacks is evidence only about those tests; it is not proof against every privacy attack. A DP claim also does not settle utility: the analysis still needs to show whether the data is useful for its intended tasks.
Compare candidate methods across the same decision criteria
Use a common comparison for candidate generation methods and release options. The criteria below synthesize NIST’s use-case-specific utility guidance, de-identification governance guidance, and synthetic-data evaluation resources; they do not prescribe a universal weighting scheme.
| Decision axis | Question to answer |
|---|---|
| Intended analysis | Does the method preserve the outcomes or decisions users need to support? |
| Privacy model | Is there a formal DP guarantee, or only empirical and informal evidence? What threat model is being tested? |
| Sensitive subgroups | Are error, utility, or disclosure risks materially worse for small or vulnerable populations? |
| Uncertainty | How does generation affect variance and downstream inference? |
| Release model | Is public release necessary, or could a query interface or protected enclave meet the need? |
| Operational fit | Can the organization calculate, reproduce, govern, and explain the chosen metrics? |
NIST SP 800-188, De-Identifying Government Datasets: Techniques and Governance (final September 14, 2023), places synthetic data alongside other data-sharing models, including protected enclaves and query interfaces. Compare the release model as part of the decision rather than assuming public release is the only option.
Set acceptance levels through governance
There is no universal enterprise privacy threshold, acceptable fidelity score, or metric bundle established for every dataset and use. NIST SP 800-188 recommends defining goals and risks, setting measurable performance levels before selecting or releasing a de-identification approach, and using re-identification studies as part of governance.
- Specify the context: record the intended analyses and users, sensitive populations and fields, release model, plausible attacker knowledge, and organizational risk tolerance.
- Predefine measures and limits: choose task-specific utility and threat-informed privacy tests, define what counts as a material utility loss or concerning disclosure, and document how those limits were set.
- Evaluate and report separately: show the results for utility, subgroup behavior, uncertainty, and privacy tests, including the assumptions and limitations of each.
- Make a release decision: compare the evidence with the approved limits and assess whether a different release model better meets the need.
Use evaluation tools as aids, not certifications
NIST’s undated Collaborative Research Cycle describes the SDNist Deidentified Data Report Generator as producing more than ten measures, including univariate and multivariate statistics, database distances, PCA, propensity, and basic privacy evaluation. The CRC also provides benchmark data. These resources can support evaluation, but a generated report is not an automatic certification that a dataset is safe for a particular use or release.
The undated HLG-MOS / NIST Synthetic Data Challenge Information Package and Test Drive points to synthpop and SDNist workflows and lists disclosure tests such as replicated uniques, apparent match distribution, and count or percent disclosure. It cautions that training a synthesizer on data without formal DP does not give that synthesizer a formal privacy guarantee.
NIST SP 800-188 also warns that tools that merely mask personal information may not provide sufficient functionality for de-identification, and that its tool list is not an endorsement. Tool output therefore needs to be interpreted in the context of the organization’s goals, threat model, and acceptance criteria.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsQuick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
