The LLM Tool Step is a versatile Tool step in Relevance AI that allows you to inject the power of large language models into your tools. It supports complex reasoning, classification, summarization, and text generation tasks — tailored to your use case.

Add the LLM Tool step to your Tool

You can add the LLM Tool step to your Tool by:

  1. Creating a new Tool, then clicking ‘LLM’ under Steps
  2. Click ‘Expand’ to see the full Tool step
  3. Add the Prompt you want to the Prompt field
  4. Choose the model you want to use
  5. Click the Settings icon if you want to access Advanced Settings

Advanced Settings

If you’re a pro at using LLMs, you can also access these Advanced Settings.

Fallback Model

You can select a Fallback Model if the model you’ve selected fails for any reason. This is a good setting to select to make sure your Tool works!

System prompt

A system prompt is composed of notes, instruction and guides normally guiding AI to assume a certain role, or to follow a specific format, or limitations.

An example of a System prompt would be: You are an expert on the solar system. Answer the following questions in a concise and informative manner.

Temperature

Temperature is a hyperparameter, ranging in (0,1), that affects the randomness (sometimes referred to as creativity) of the LLMs’ response.

Higher randomness/creativity/diversity is expected of higher temperature values.

However, responses might also lose the right context.

Conversation history

Click on + Add row and you can add lines of conversation taking place between a “user” and “ai”.

The only acceptable roles are “ai” and “user”.

Conversation history is useful in conversational agents and help AI to know more about the situation.

Thinking / Reasoning Configuration

For OpenAI, Claude and Gemini models, you can select a thinking configuration. Enabling this will make the model ‘think’ more before answering, which is useful for complex problems.

Reasoning / thinking tokens are charged as output tokens and will be added to the cost of your LLM Tool step.

If model and vendor options are mismatched, this option will be ignored.

Force response format

For certain models such as OpenAI, response format can be forced using this field.

Common errors