6 min read

May 12, 2025

AgentOS: Total Visibility and Control Over Your AI Workforce

Share this post

https://relevanceai.com/blog/agentos---total-visibility-and-control-over-your-ai-workforce

Jacky Koh

Founder

This week, we launched some of the most ambitious features we’ve ever built:

  • Invent, for turning plain English into custom agents
  • Workforce, for building multi-agent systems visually
  • 2,000+ integrations, for connecting agents to your entire tech stack

Today, we close out AgentDrop with introducing several additional features that help manage your growing AI initiatives, which together form our vision for AgentOS - a framework for scaling your agent operations.

From Agent Creation to Agent Operations

Creating a single agent is just the beginning. As domain experts build more agents, they need better ways to organize and manage them while ensuring their specialized knowledge is properly applied.

That's why we're introducing several improvements to our platform:

  • Improved Knowledge for Agents: Adding domain knowledge to your agent is easier than ever - simply upload documents or connect knowledge bases
  • New Agent UI: A redesigned, more intuitive interface that makes agent creation accessible to non-technical subject-matter experts
  • Embeddable Chat: Chat with your agents directly or embed them where needed to share your expertise
  • Folders: Organize your agents and tools into folders based on domains of expertise
  • Version Control: View and restore previous versions of your agents to track how your knowledge has evolved

These features help domain experts manage the growing complexity as they scale from initial experiments to practical applications.

What AgentOS Means

As subject-matter experts build more agents to share their knowledge, they need better ways to manage them. We've introduced several new features that collectively represent a big milestone towards our vision for AgentOS - a framework that helps domain experts maintain oversight of their AI agents with capabilities like:

  • Scheduling: Control when your knowledge-driven agents run with fixed times or recurring intervals
  • Intelligent Queuing: Prioritize tasks based on domain-specific importance
  • Human in the Loop: Let agents escalate complex questions back to you as the domain expert
  • Metadata: Autonomously extract key information from an agent's task to track how your expertise is being applied
  • Comprehensive Logging: See exactly what happened, when, and why to maintain quality control
  • Performance Analytics: Monitor how accurately your agents represent your domain knowledge
  • Governance and Control: Pause, resume, or override agent behavior when you spot expertise gaps

Bridging the Trust Gap

The path from pilot to production always starts small. A few agents. A few wins. Then it hits a wall - how do we scale this without losing control?

The operational approach we're developing is designed for exactly this moment - when AI stops being a test and starts becoming part of your everyday work.

What we've consistently heard from subject-matter experts isn't that they doubt AI's potential. It's that they can't scale what they can't see and understand.

These new capabilities help experts answer mission-critical questions:

  • What exactly are our agents doing with our domain knowledge?
  • Are the agents properly applying our expertise?
  • Where are they getting stuck or misinterpreting instructions?
  • Can we verify the quality of agent outputs?
  • What needs to improve, and where?

With improved logging, settings, and visibility, experts gain the confidence to make AI truly reliable and accountable to their domain expertise.

Real-World Impact

During testing, we've seen these new features transform how domain experts work - not just by automating repetitive tasks, but by giving them the visibility and control they needed to confidently extend their expertise through AI.

We've watched researchers, legal experts, healthcare professionals, and other specialists go from cautious experimentation to confidently deploying agents that accurately represent their knowledge.

That's the power of having better tools for domain experts to manage their AI agents. Check out our docs to learn more about adding specialized knowledge, escalating complex questions back to experts, and analyzing how your expertise is being applied.

Iterating Towards the Vision

Over the past few weeks, we've shipped several key features designed specifically for domain experts:

  • Improved Knowledge for Agents: Adding specialized context and data more seamlessly
  • New Agent UI: Making agent creation accessible to non-technical experts
  • Folders: Organizing different knowledge domains more effectively
  • Version Control: Tracking how your expertise evolves over time
  • Embeddable Chat: Sharing your expertise through embedded agents
  • Scheduling: Controlling when your knowledge-driven agents run
  • Human in the Loop: Escalating complex questions back to you as the domain expert
  • Metadata: Tracking how your expertise is being applied

These recently shipped features work together with Invent, Workforce, and our 2,000+ integrations to help subject-matter experts share their knowledge more effectively through AI.

Looking Ahead

These new capabilities mark an important step in how domain experts can share their specialized knowledge through AI.

We're building toward a future where AI agents authentically represent subject-matter expertise - where doctors, lawyers, researchers, and specialists of all kinds can extend their reach and impact through agents they trust.

With the backing of our $24M Series B, we're doubling down on making that future real - adding more features that help domain experts maintain the quality and accuracy of their knowledge as it's deployed through AI.

To our users, partners, and the Relevance team: thank you for being part of this journey.We're not just building features - we're building the bridge between human expertise and AI.

Jacky Koh
Co-founder and Co-CEO, Relevance AI

AgentOS: Total Visibility and Control Over Your AI Workforce

This week, we launched some of the most ambitious features we’ve ever built:

  • Invent, for turning plain English into custom agents
  • Workforce, for building multi-agent systems visually
  • 2,000+ integrations, for connecting agents to your entire tech stack

Today, we close out AgentDrop with introducing several additional features that help manage your growing AI initiatives, which together form our vision for AgentOS - a framework for scaling your agent operations.

From Agent Creation to Agent Operations

Creating a single agent is just the beginning. As domain experts build more agents, they need better ways to organize and manage them while ensuring their specialized knowledge is properly applied.

That's why we're introducing several improvements to our platform:

  • Improved Knowledge for Agents: Adding domain knowledge to your agent is easier than ever - simply upload documents or connect knowledge bases
  • New Agent UI: A redesigned, more intuitive interface that makes agent creation accessible to non-technical subject-matter experts
  • Embeddable Chat: Chat with your agents directly or embed them where needed to share your expertise
  • Folders: Organize your agents and tools into folders based on domains of expertise
  • Version Control: View and restore previous versions of your agents to track how your knowledge has evolved

These features help domain experts manage the growing complexity as they scale from initial experiments to practical applications.

What AgentOS Means

As subject-matter experts build more agents to share their knowledge, they need better ways to manage them. We've introduced several new features that collectively represent a big milestone towards our vision for AgentOS - a framework that helps domain experts maintain oversight of their AI agents with capabilities like:

  • Scheduling: Control when your knowledge-driven agents run with fixed times or recurring intervals
  • Intelligent Queuing: Prioritize tasks based on domain-specific importance
  • Human in the Loop: Let agents escalate complex questions back to you as the domain expert
  • Metadata: Autonomously extract key information from an agent's task to track how your expertise is being applied
  • Comprehensive Logging: See exactly what happened, when, and why to maintain quality control
  • Performance Analytics: Monitor how accurately your agents represent your domain knowledge
  • Governance and Control: Pause, resume, or override agent behavior when you spot expertise gaps

Bridging the Trust Gap

The path from pilot to production always starts small. A few agents. A few wins. Then it hits a wall - how do we scale this without losing control?

The operational approach we're developing is designed for exactly this moment - when AI stops being a test and starts becoming part of your everyday work.

What we've consistently heard from subject-matter experts isn't that they doubt AI's potential. It's that they can't scale what they can't see and understand.

These new capabilities help experts answer mission-critical questions:

  • What exactly are our agents doing with our domain knowledge?
  • Are the agents properly applying our expertise?
  • Where are they getting stuck or misinterpreting instructions?
  • Can we verify the quality of agent outputs?
  • What needs to improve, and where?

With improved logging, settings, and visibility, experts gain the confidence to make AI truly reliable and accountable to their domain expertise.

Real-World Impact

During testing, we've seen these new features transform how domain experts work - not just by automating repetitive tasks, but by giving them the visibility and control they needed to confidently extend their expertise through AI.

We've watched researchers, legal experts, healthcare professionals, and other specialists go from cautious experimentation to confidently deploying agents that accurately represent their knowledge.

That's the power of having better tools for domain experts to manage their AI agents. Check out our docs to learn more about adding specialized knowledge, escalating complex questions back to experts, and analyzing how your expertise is being applied.

Iterating Towards the Vision

Over the past few weeks, we've shipped several key features designed specifically for domain experts:

  • Improved Knowledge for Agents: Adding specialized context and data more seamlessly
  • New Agent UI: Making agent creation accessible to non-technical experts
  • Folders: Organizing different knowledge domains more effectively
  • Version Control: Tracking how your expertise evolves over time
  • Embeddable Chat: Sharing your expertise through embedded agents
  • Scheduling: Controlling when your knowledge-driven agents run
  • Human in the Loop: Escalating complex questions back to you as the domain expert
  • Metadata: Tracking how your expertise is being applied

These recently shipped features work together with Invent, Workforce, and our 2,000+ integrations to help subject-matter experts share their knowledge more effectively through AI.

Looking Ahead

These new capabilities mark an important step in how domain experts can share their specialized knowledge through AI.

We're building toward a future where AI agents authentically represent subject-matter expertise - where doctors, lawyers, researchers, and specialists of all kinds can extend their reach and impact through agents they trust.

With the backing of our $24M Series B, we're doubling down on making that future real - adding more features that help domain experts maintain the quality and accuracy of their knowledge as it's deployed through AI.

To our users, partners, and the Relevance team: thank you for being part of this journey.We're not just building features - we're building the bridge between human expertise and AI.

Jacky Koh
Co-founder and Co-CEO, Relevance AI

Contents
Jacky Koh
Tags:
Announcements