Validators

One of the biggest roadblocks with using LLMs is when they return invalid content. For example, you may ask for a specific format of JSON, but the LLM returns an explanation that breaks it in your JSON.parse().

To protect against this, we provide a quality control feature called validators. You can specify a schema you want to apply to your prompt, and we will validate the LLM’s response against this schema. If the LLM returns invalid data, we will ask the LLM to fix it. If that fails, we will return an error to your application.

This way, you can rest assured your application will always receive valid data or error gracefully.

const { answer } = step('prompt_completion', {
    prompt: `Analyse this PDF text: ${pdf_text}, and summarise it into a JSON with the following properties: 'title', 'description', 'page'.`,
    validators: [
        {
            _oneof_type_: 'jsonschema',
            schema: {
                type: 'object',
                properties: {
                    title: { type: 'string' },
                    description: { type: 'string' },
                    page: { type: 'number' },
                },
            },
        },
    ],
});

​

The “validators” property

Specify quality control by using the validators property on the prompt_completion step. This receives a list of validators that we want to apply to our LLM’s response, in the format of:

{
    _oneof_type_: 'is_json' | 'regex' | 'jsonschema';
    schema?: // for jsonschema
    pattern?: // for regex
}

Currently there are three options, but we will always be adding more:

• is_json
• regex
• jsonschema

​

is_json

This validates whether the LLM has returned only JSON. It will try to fix it if not. This doesn’t require any additional properties.

​

jsonschema

If you would like to validate a specific format of JSON, you can take advantage of JSONSchema! Provide the schema as the schema property.

{
    type: 'object',
    properties: {
        title: { type: 'string' },
        description: { type: 'string' }
    }
}

​

regex

This validates whether the LLM’s response matches a regex. It will try to fix it if not. This requires a pattern property, in this format:

{
    pattern: "[a-z0-9]",
    flags: 'i'
} // equates to /[a-z0-9]/i