Asking an LLM to “return valid JSON” is not enough when your application depends on specific fields and rules. Define a schema, use the provider’s structured-output feature when the target model supports it, then validate the result and handle refusals, incomplete responses, and invalid data in application code.
Why “valid JSON” is not a sufficient contract
JSON syntax only tells you whether a response can be parsed as JSON. It does not ensure that the object has the fields, types, or values your application expects. A response can parse successfully and still omit a required field, use an unsupported enum value, or contain a value that violates a business rule.
OpenAI distinguishes JSON mode, which aims to produce valid JSON, from Structured Outputs, which enforces adherence to supported schemas. Its documentation recommends Structured Outputs when the feature is available for the model and API you use: OpenAI Structured Outputs guide.
Define the output contract first
Write down the structure your application accepts before tuning the prompt. Use JSON Schema or an equivalent typed definition to specify required fields, data types, allowed enum values, optional fields, and whether additional properties are permitted.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11#1 Best Overall
Descriptions can clarify what a field means to the model, but they are not a substitute for checks in your code. Some schema rules may not be supported by a provider, and a schema cannot capture every domain constraint your application needs.
Choose the right generation interface
Structured response formatting and function calling solve different problems. A response-format interface shapes content the model returns to the user or your application. Tool or function calling is for asking the model to invoke application behavior. A structured response does not, by itself, authorize an action; your application must still decide what to do with the returned data.
Provider APIs are not interchangeable. Confirm support for your specific model and API path, the schema features and nesting limits, and how the SDK exposes refusals and errors before adopting a contract.
OpenAI
OpenAI’s guide separates JSON mode from Structured Outputs: JSON mode targets valid JSON but does not guarantee a particular schema, while Structured Outputs enforces supported schemas. The guide also describes incomplete-output edge cases, so clients should check whether generation finished before treating a response as complete. See the Structured Outputs documentation.
Google Gemini
Gemini structured output supports a subset of JSON Schema. Google cautions that very large or deeply nested schemas may be rejected and that schema-shaped output may still be semantically wrong. Its guidance is direct: “Always validate the final output in your application code before using it.” Read the Gemini structured output documentation.
Anthropic Claude
Anthropic documents JSON outputs using output_config.format and separately documents strict tool use. Check the current supported schema limits and model availability for your chosen API path rather than assuming that a schema accepted by another provider will work unchanged. See Anthropic structured outputs documentation.
Constrained-decoding research
Provider features are only part of the picture. The JSONSchemaBench paper describes an evaluation across 10,000 real-world schemas, treating efficiency, schema coverage, and output quality as distinct dimensions. That is a useful reminder that syntactic or schema compliance alone does not establish that a result is useful. See the JSONSchemaBench paper abstract.
Validate at the application boundary
Treat every model response as untrusted input, even when you request schema-constrained output. Parse and validate it before passing it to other parts of your system, then apply domain checks that the schema cannot establish.
- Check required fields, types, enum values, and other contract rules.
- Check ranges and relationships between fields, such as a start date preceding an end date.
- Confirm that referenced identifiers exist and that the caller is authorized to use them.
- Apply safety and business rules before using a value to trigger a consequential action.
A syntactically valid, schema-conforming object can still contain false, stale, or unusable information. Google’s Gemini guidance explicitly calls for application validation because semantic correctness is not guaranteed.
Handle failures by category
Build explicit failure paths instead of treating every unsuccessful response as a prompt-writing problem. The recovery should depend on what failed.
| Failure | Application response |
|---|---|
| Schema rejected by the API | Check whether the selected model and API support the schema features and complexity you supplied. Change the contract or deployment; do not repeatedly retry an identical deterministic rejection. |
| Timeout, rate limit, or transport failure | Use the retry and backoff policy appropriate to the specific transient error, and enforce request limits. |
| Interrupted or incomplete generation | Do not parse or use it as a complete result. Detect the provider’s completion state and decide whether a bounded retry is appropriate. |
| Explicit refusal | Route it through a refusal-specific path. Do not mistake it for a malformed object or blindly retry the same request. |
| Parse or schema-validation failure | When using a mode that does not constrain the schema, record the validation failure and consider a bounded repair attempt only if it is safe and useful. |
| Schema-valid but semantically invalid data | Reject or quarantine the value and apply the relevant domain rule; retrying the same request may simply reproduce the same problem. |
Record the failure category and enough diagnostic context to investigate it, while avoiding unnecessary logging of sensitive prompts or outputs. Provider documentation describes specific limitations and edge cases; retry policy and observability are application design decisions.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Test the complete contract, not just parsing
A high parse-success rate can hide objects that are wrong but valid. Exercise the whole path using representative and adversarial inputs, including missing information, boundary values, refusal-triggering cases, long outputs, and schema features near documented limits.
Free tools Windows power users keep installed
One-click scans. No signup required.
- Measure parse success and schema compliance separately.
- Measure semantic and business-rule validity, refusal and interruption rates, and end-to-end task success.
- Test the exact model, API path, SDK, and schema you plan to deploy.
- Recheck the provider’s documentation as model availability and schema support can change.
OpenAI reported that gpt-4o-2024-08-06 scored 100% on its complex JSON Schema-following evaluation, compared with less than 40% for gpt-4-0613. The same August 6, 2024 announcement says the newer model reached 93% on the stated benchmark before a deterministic constrained-decoding layer was added. These are OpenAI-reported results for its named models and evaluation, not a cross-provider comparison or a guarantee of semantic accuracy in production. See OpenAI’s Structured Outputs announcement.
Quick Recap
A practical implementation sequence
- Define the contract: specify the expected object, required fields, types, allowed values, optionality, and additional-property policy.
- Select an interface: use structured response formatting for a structured answer, or tool/function calling when the model needs to request application behavior.
- Verify compatibility: confirm the target model and API path support the required schema features and check the provider’s behavior for refusals and incomplete responses.
- Validate before use: parse and validate the response, then enforce domain rules, authorization, and safety constraints.
- Route failures deliberately: distinguish transient errors from schema rejection, refusal, interruption, and invalid data; retry only under a bounded policy.
- Test and monitor: track schema compliance alongside semantic validity and task outcomes, and revisit the contract when provider support or application needs change.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
