Cities should evaluate AI vendors against the city’s specific use, affected residents, legal obligations, and consequences of error—not accept assurances that a product is “fair,” “secure,” or “transparent” at face value. A defensible evaluation asks every bidder for comparable evidence, sets requirements in the contract, and continues monitoring after deployment.
Start with the use, not the vendor’s claims
Before reviewing products, describe the service problem and decide whether AI is necessary to address it. Identify what the system will do, who will use it, whose data it processes, which decisions it may inform, and how much human judgment remains. Include the expected benefit and what could happen if the system is wrong.
Consider alternatives, including a non-AI process. A tool that drafts internal text has a different risk profile from one that could affect a resident’s access to public services. The depth of review should reflect that difference rather than treating every product with an AI label alike.
Set the review team and scope
Bring in the people who understand the service and the risks: procurement, program staff, IT and security, privacy, legal, accessibility or civil-rights expertise, and community perspectives when warranted. Establish the system boundaries, intended and prohibited uses, dependencies, affected groups, and applicable local requirements before comparing bids.
Free tools Windows power users keep installed
One-click scans. No signup required.
#1 Best Overall
NIST’s AI Risk Management Framework (AI RMF) describes trustworthiness across an AI system’s lifecycle and is voluntary. NIST released version 1.0 on January 26, 2023, and says it is being revised; check its current status before using it in a solicitation. It is a useful reference, not a mandatory city standard unless a local rule or contract makes it one.
Ask every bidder for the same evidence
Use a common questionnaire and scoring method so proposals can be compared on substance rather than presentation. Request material that supports each claim, and ask vendors to identify what they cannot provide.
- Purpose and boundaries: intended use, prohibited uses, system components and dependencies, human roles, known limitations, and conditions under which the product should not be used.
- Data handling: data sources and permissions; collection, access, storage, retention, deletion, and sharing; subprocessors; and whether city data may be used for training, testing, evaluation, or product improvement.
- Performance and fairness: test methods, test data and its context, results for relevant groups, acceptance thresholds, known failure modes, and limits on applying results to the city’s population or operating conditions.
- Security and privacy: access management, safeguards for data in transit and at rest, incident response, and the vendor’s responsibilities when a supplier or system component changes.
- Transparency and oversight: technical documentation, descriptions of system behavior and adaptive components, explanations available to staff and residents, monitoring arrangements, and procedures for human review or escalation.
- Operational evidence: references and production examples in settings sufficiently similar to the city’s service, population, data, and deployment conditions. Explain differences that may limit the relevance of those examples.
Georgia’s statewide public-sector RFP guidance recommends practices such as diverse evaluation committees, standardized scoring, review of bias reports, interpretable decisions, data protection, monitoring, and accountability. It can inform a city’s solicitation, but it is not a universal requirement for every city.
Rank #2
Test fairness claims against the city’s context
A vendor’s fairness statement, framework mapping, certification, or self-assessment is not proof that a system will perform fairly in a city’s use. Ask for underlying methods and results, then assess whether the evidence matches the actual population, task, data, and decision consequences.
Questions to ask about bias and error
- What kinds of errors did the vendor measure, and how were the test cases selected?
- Which demographic or other relevant groups were included, and are the results sufficiently detailed to identify uneven performance?
- What were the results for each relevant group, and what uncertainty or limitations apply?
- What failure modes are known, including cases where the system performs poorly or produces misleading outputs?
- How will the vendor and city retest the system when data, populations, models, or operating conditions change?
NIST’s public-sector procurement guidance poses two useful prompts: “What level and type of bias is acceptable in the solution?” and whether acceptance criteria set appropriate accuracy levels. The city should answer these for the specific use and affected population, with legal advice where needed; a threshold is not meaningful just because a vendor proposes it.
Make security and privacy requirements verifiable
Translate broad assurances into questions and obligations the city can check. Map the data flow from collection through processing, storage, access, sharing, and deletion. Specify which data uses are allowed, who can access the information, what safeguards apply, how incidents will be reported, and how the city can verify material commitments.
Address training and product improvement explicitly. Portland’s administrative rule provides a municipal example: within its defined scope, it requires written authorization for use of City data to train, test, or improve vendor AI models, and risk-proportional audit or verification rights. Portland’s rule applies to the City’s defined systems and services; it is not a nationwide rule for all municipalities.
Include the city’s requirements for data retention and deletion, subcontractor controls, incident response, documentation, and verification in the solicitation and contract. Have local counsel and the responsible privacy, security, records, and program teams identify requirements that bind the particular city and use.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Evaluate transparency for staff and residents
Transparency is more than a model card or technical document. Staff need enough information to understand how the system is used, its limits, and when to override or escalate an output. Residents need appropriate notice about the system’s role in a service or decision and a way to seek review when that is relevant to the use.
Rank #4
Ask vendors for technical documentation, descriptions of system behavior and limitations, information about material changes, and explanations that fit the audiences who need them. In the procurement, state what the city will disclose, to whom, and at what point in the process. Do not promise an explanation the vendor’s system cannot support.
Score proposals with a documented rubric
Use the same axes for each bidder, but weight them according to impact and legal context. Separate minimum requirements from preferences: a strong score in one area should not compensate for failure to meet a mandatory security or performance condition. For each score, record the evidence, uncertainties, and person or body accepting any remaining risk.
| Evaluation dimension | Evidence to assess |
|---|---|
| Fit and task performance | Evidence that the system addresses the defined use, including accuracy, error types, and consequences of error. |
| Fairness and evidence quality | Methods, test context, results across relevant groups, thresholds, limitations, and retesting commitments. |
| Security and privacy | Data flows, safeguards, access controls, incident response, approved data uses, retention, and deletion. |
| Transparency and limitations | Technical documentation, explanations suited to users, disclosure plans, and notice of material changes. |
| Oversight and contestability | Human review, escalation, and available routes to question or correct consequential outputs. |
| Operations and accountability | Monitoring, vendor responsibilities, audit or verification rights, update practices, and incident handling. |
| Exit feasibility and public value | Lifecycle cost, transition burden, ability to retrieve or delete data, and the value of the service outcome. |
NIST cautions that trustworthiness characteristics can involve tradeoffs, and that their importance varies by setting. The rubric should make those choices visible instead of implying that one score or certification settles the decision.
Recommended Free Tools
Put the evaluation into the contract
Preserve important proposal claims as enforceable commitments. Depending on the use and applicable rules, contract terms may address:
- Approved purposes, prohibited uses, and permitted data handling, including whether city data may be used for training or product improvement.
- Delivery and updating of technical documentation, test results, and material-change notices.
- Performance, retesting, and monitoring commitments tied to agreed criteria.
- Incident reporting, subcontractor obligations, and responsibilities for errors or system changes.
- Human review and escalation arrangements, plus audit or verification rights proportionate to risk.
- Retention, deletion, transition assistance, and termination rights if the system no longer meets requirements.
Portland’s rule illustrates how a municipality can connect pre-procurement risk assessment with AI-specific disclosures, documentation, limits on data use, and verification rights. A city should adapt requirements to its authority, procurement, use, and legal setting rather than copy another jurisdiction’s rule as if it applied automatically.
Monitor the system after launch
Evaluation continues through deployment and maintenance. Establish a pre-launch baseline and assign named owners to review performance and operational signals. Decide in advance what will trigger investigation, mitigation, retesting, suspension, or public notice.
- Track performance and errors, complaints, access patterns, and security events relevant to the service.
- Review material changes to the model, data, vendor, system dependencies, or intended use.
- Set review intervals and escalation paths, and require the vendor to provide information needed for those reviews.
- Reassess when evidence or operating conditions change, not only at contract renewal.
NIST procurement guidance recommends systematic, continuous risk monitoring through maintenance. The appropriate measures and thresholds depend on the city’s use and obligations; legal requirements and suitable bias-testing approaches can differ by application and context.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Use local rules and frameworks carefully
Neither a framework nor a policy from another jurisdiction replaces review of the city’s own legal duties. NIST’s AI RMF is voluntary, and Georgia’s guidance is statewide public-sector procurement advice. Portland’s rule applies within the City’s defined scope. Before finalizing evaluation criteria, have local counsel and the responsible privacy, security, civil-rights, accessibility, records, and program teams identify requirements for the actual system and service.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
