In this blog post we will introduce a method of prompting LLMs that we have chosen to call schema based prompting. At Opper we have found this to be a great method for interacting with LLMs and vLLMs. It has proven to help with developer experience, model interoperability and reliability of AI calls - leading to faster development and better quality.

While structured output is today widely adopted by models, frameworks, libraries and APIs (see Simon Willison's excellent blog on Structured Outputs), using input schemas and schemas based prompting has generally been less discussed.

Unlike traditional methods of prompting where instructions are expressed through natural language, schema based prompting is a technique of expressing tasks through clear data structures. Prompts consists of a clear schema (often in JSON) for both the input data and the expected output data, with only minimal natural language instructions or "prompts".

One way to look at schema based prompting is that we task the model to perform a data transformation from a set of well defined input data structures to well defined output data structures. By doing this we are essentially exploiting the models ability to infer the task from code like objects, instead of having it infer the task from natural language descriptions. As we will see there are some great benefits with this approach. As someone said it: "structured inputs, predictible outputs!"

Before jumping in to the details of schema based prompting, let's first look at the classical way of prompting models through natural language.

Natural language prompting

The classical way of prompting is to provide a free form text prompt and let the model generate a response.

Here is a very simple example of a typical interaction with an LLM:

System: You are a helpful assistant that can translate text to different languages.

User: Translate the text "The capital of Sweden is Stockholm." to Spanish.

Assistant: "La capital de Suecia es Estocolmo."

When programming with LLMs we normally construct these prompts from variables. For example, in this example we are likely to want the input text and the target language to different at every invocation. For this reason, prompts are more likely to include variables and place holders like this:

System: You are a helpful assistant that can translate text to different languages.

User: Translate the text "<some text>" to "<some language>".

Assistant: <response from the model>

Prompts like this can become very complex or many other tasks these "prompts" with variables and parsers can become very complex. Both the System and User prompts often need to be heavily tailored to instruct the model to produce the desired output. These prompts can often become model specific and hard to maintain. And we are really putting our hopes at the output being just the translation and not a lot of other "stuff" that models have a tendency to add, for example:

Assistant: Here is the translation of the text in Spanish: "La capital de Suecia es Estocolmo."

Making parsing the actual output a bit challenging leading us to have to adjust the prompt to more clearly express the output we want.

Schema based prompting

With Schema based prompting, we spend less effort in crafting a perfect natural language instruction. Instead we spend our effort in describing the input we will provide and output data we want. We let the model reason from the expressive schemas and not from a natural language instruction.

The essential part of a schema based prompt is a JSON object. And while JSON objects can be hard to read and write manually there are plenty of well established libraries for this that we program with every day, such as Pydantic in Python or Zod in Typescript.

But let's look at a proper JSON schema first.

A simple JSON schema

Here we define a JSON schema that describes the above translation task. We define an input schema, an output schema, a small instruction and our input data:

{
    "input_schema": {
        "type": "object",
        "properties": {
            "text": {"type": "string"},
            "target_language": {"type": "string"}
        }
    },
    "output_schema": {
        "type": "object",
        "properties": {
            "translation": {"type": "string"}
        }
    },
    "instructions": "Translate the input text to the target language.",
    "inputs": {
        "text": "The capital of Sweden is Stockholm.",
        "target_language": "Spanish"
    }
}

Putting it all together in ChatLM format

Since many LLMs and vLLMs implement the ChatLM standard (often referred to as the OpenAI standard) we often need to cater for the System, User and Assistant roles as seen above. Here is how the above schema based prompt could look like in a ChatLM format and interaction:

System:   

{
    "input_schema": {
        "type": "object",
        "properties": {
            "text": {"type": "string"},
            "target_language": {"type": "string"}
        }
    },
    "output_schema": {
        "type": "object",
        "properties": {
            "translation": {"type": "string"}
        }
    },
    "instructions": "Translate the input text to the target language.",
}

Then we can use the User prompt to provide the input data to the model:

User: 

{
    "inputs": {
        "text": "The capital of Sweden is Stockholm.",
        "target_language": "Spanish"
    }
}

And finally the Assistant prompt will return the output data from the model that conforms to the output schema:

Assistant: 

{
    "translation": "El capital de Suecia es Estocolmo."
}

Changing the target language is as easy as changing the input data:

User: 

{
    "inputs": {
        "text": "The capital of Sweden is Stockholm.",
        "target_language": "Chinese"
    }
}

Assistant: 

{
    "translation": "瑞典的首都是斯德哥尔摩。"
}

As you can see, we have now clearly separated the major parts of the prompt. Everything that is related to the task and the model's role is put into the system prompt. The user prompt is then only used to provide the input data to the model. Futher more, we have clearly expressed both the input schema and the output schema and this avoids a lot of the ambiguity and noise that can be present in natural language prompts.

Making schemas even more expressive

JSON schemas are very powerful and can be used to express a lot of complex data structures. By adding more properties to the schema we can explain more about what the input and output data should look like and what different fields mean.

For example, for the task above we could add a helping description to the input schema to indicate that the text field is a string and the target language field is a string from a set of allowed languages that our Application or Model supports:

{
    "input_schema": {
        "type": "object",
        "properties": {
            "text": {"type": "string", "description": "The text to translate"},
            "target_language": {
                "type": "string",
                "enum": ["Chinese", "Spanish", "French"],
                "description": "The language to translate the text into"
            }
        },
        "required": ["text", "target_language"]
    }
}

When providing this enhanced schema to the model, we are further helping the model to understand the task and the input data.

Validating inputs and outputs

Schemas also allow us to easily validate that we are providing the right input data and getting the right output data. With a few simple steps we can validate that the data us correct - which essentially moves us towards building a contract with the model that we can rely on:

Here is how we can validate the input data:

import json
from jsonschema import validate, ValidationError

try:
    validate(instance=input, schema=input_schema)
except ValidationError:
    raise Exception("Invalid input")

Maybe more importantly, we can validate that the output from the model conforms to the output schema.

import json
from jsonschema import validate, ValidationError

try:
    validate(instance=output, schema=output_schema)
except ValidationError:
    raise Exception("Invalid output")

Now, this is a small example. And one might argue that there is a lot of scaffolding and extra tokens used for this small task. And you would be right! But the since LLM features rarely fails through some marginal increase in tokens or speed, we believe the trade off to be fully worth it in most cases.

How and why schema based prompting works

While the exact inner workings of Large Language Models (LLMs) remain partially opaque, we can make a few informed hypotheses about why schema-based prompting is effective.

In combination of massive amounts of text, the pre-training data for modern LLMs also includes vast amounts of code, documentation, and structured data formats. When we present schemas to these models, we're leveraging their extensive exposure to programming patterns, data structures, and type systems during training. The approach of schema based prompting essentially aligns with how the models have learned to process and generate structured information.

If one would speculate a bit it is likely that when presented with a schema, the model recognizes the formal specification and activates the model's neural pathways trained on code and structured data, enabling it to operate in a more computational mode rather than interpreting ambiguous natural language instructions. From its pre-training the model can understand type systems, constraints like enums and required fields, and identify transformative relationships between input and output schemas without explicit instructions, making the entire process more deterministic and reliable.

Benefits of Schema Based Prompting

Some of the benefits we have found with schema based prompting are:

• Simplified Development: It makes it easier to program with LLMs without needing to develop an intuition for how models interpret natural language instructions, reducing the need for extensive "prompt engineering"
• Easier Quality Assurance: Validating model quality becomes more straightforward as outputs can be automatically validated against the schema and compared to previous examples
• Pipeline Integration: It's ideal for pipelines and agentic workflows where input and output data is well-defined, consistent, and can seamlessly feed into subsequent processes
• Better Practices: It encourages keeping AI interactions task-oriented, modular, and easy to understand - ultimately leading to code that is more maintainable, easier to improve and debug
• Dataset Development: The structured nature creates excellent foundations for building high-quality datasets that can be used for testing, fine-tuning, and optimization and thus effectively enables a future path for fine tuning and distilling models for tasks.
• Model Portability: It's largely model-independent, and works across different LLMs as it minimizes ambiguity and noise in the prompt.

Drawbacks of Schema Based Prompting

There are some drawbacks to schema based prompting as well:

• Readability: The raw schema-based prompts can be harder to read at first glance, but with the right tooling they can be made just as readable as free-form prompts (or even more structured and maintainable)
• Token Overhead: There is an overhead in the tokens used for the schemas, but this cost is typically offset by the benefits in output quality, reliability, and development speed
• Tooling Ecosystem: Natural language prompting remains the most common way to interact with LLMs, so some tooling may lag behind in supporting schema-based approaches (though Opper provides excellent support for this method)
• Output Style: Responses can sometimes appear more structured and less conversational, but this is often beneficial as it grounds the model in clear, precise language rather than allowing potentially over-expressive or verbose outputs

Practicing Schema Based Prompting with Opper

In the Opper API and SDKs we support schema based prompting out of the box with the opper.call() function.

Here is how this works:

import asyncio
from opperai import AsyncOpper
from pydantic import BaseModel, Field

# Initialize the client
opper = AsyncOpper(api_key="your-api-key")

# Define schemas using Pydantic
class RoomInput(BaseModel):
    text: str = Field(description="Text containing room information")

class RoomOutput(BaseModel):
    room: str = Field(description="The extracted room name/number")
    floor: str = Field(None, description="The floor where the room is located")
    time: str = Field(None, description="The meeting time if mentioned")

async def extract_room_details():
    # Example of using opper.call for an extraction task
    result, _ = await opper.call(
        name="extract_room_details",
        input={
            "text": "The meeting will be held in Conference Room B on the 3rd floor at 2:30 PM. Please bring your laptop and the project documents."
        },
        input_type=RoomInput,
        output_type=RoomOutput,
        instructions="Extract the meeting room details from the provided text.",
        model="mistral/mistral-tiny-eu"
    )

    print(f"Room: {result.room}")
    print(f"Floor: {result.floor}")
    print(f"Time: {result.time}")

# Run the async function
asyncio.run(extract_room_details())

Which yields the output:

Room: Conference Room B
Floor: 3
Time: 2:30 PM

For more information on how to use the opper.call() opperation, please refer to the Opper documentation.

Conclusion

In this blog post we introduced Schema-based prompting, a technique that transforms how we can interact with LLMs by using defined data structures instead of natural language instructions, creating a consistent contract between applications and models.

At Opper we have found schema based prompting to be a very powerful technique for building reliable AI calls that can be used to power powerful AI features. A couple of examples of internal projects making heavy use of this is Delvin and Opperator.

Key advantages we have found are:

• Structured inputs and outputs: Clearly defined data shapes improve reliability
• Validation capabilities: JSON schemas enable robust input/output validation
• Model independence: Works across different LLMs using fundamental capabilities
• Developer-friendly: Libraries like Pydantic and Zod make schemas intuitive and type-safe
• Pipeline compatibility: Fits naturally into data pipelines and agent workflows

Despite initial setup overhead, the predictability and maintainability benefits make it worthwhile for production use cases.

If you want to give schema based prompting a try, you can sign up for a free Opper account here.