Most agent failures don't show up in a single interaction, they show up in the traces. This guide walks through the full process of finding these failures, writing tests to measure them, and creating fixes. At the end you should know how to turn traces into repeatable eval tasks, know why a held-out set matters so a fix doesn’t overfit, and know how to resolve failures with small changes in the agent's instructions and verify those fixes in measurable ways.
This repository is a NemoClaw reference tutorial for optimizing an Enterprise Personal Assistant agent. The setup follows the NemoClaw structure: an agent is built with Hermes running in OpenShell with Nemotron 3 Ultra as the default LLM. The agent connects to an MCP service that contains fictional tools mimicking services like Outlook, Teams, Slack, Jira, and Confluence.
The tutorial walks through a multi-step optimization process: - Create traces by asking the agent a series of questions that use the fictional MCP tools - Use NeMo Compass to analyze the traces and identify common failures - Use a coding agent like Codex with the NeMo Eval Author skills to create Harbor evaluation cases that (a) reflect the most common patterns from the traces, and (b) reproduce any identified failures - Propose a change to the agent that fixes the failure - Run the Harbor eval cases with the proposed fix, confirming improvement without introducing regressions
After completing this tutorial, you will have a core understanding of the NeMo libraries to re-implement this optimization loop with your own agent.
The repository includes checked-in artifacts for each step of this process, allowing you to reproduce the results or skip steps that you do not want to re-run.
Fictional Agent
The optimization tutorial starts with a fictional agent. The agent is built following the NemoClaw reference pattern: - Hermes Agent Harness - OpenShell Runtime - Nemotron 3 Ultra LLM by default
The agent is given a set of fictional MCP tools and is asked to perform tasks for a company product launch: researching across mail, calendar, chat, files, enterprise knowledge, directory, project, analytics, support, and connector tools.
The fictional MCP tool catalog has 15 tools backed by synthetic data. The tools were built to reflect real problems encountered with an actual NVIDIA enterprise assistant including: ambiguous tool definitions, tool overlap, large paginated JSON results, multi-step workflows for resolving UIDs, and human-in-the-loop requirements for draft or write actions.
| Tools | What the agent can do | Enterprise equivalent |
|---|---|---|
connectors.get_status |
Check connection and authentication state | SaaS connector health / OAuth status |
people.search |
Find employees, including ambiguous names | Microsoft Entra ID / Google Workspace directory |
mail.search, mail.read_thread |
Find mail metadata, then read messages | Outlook / Gmail |
chat.search, chat.read_thread |
Find chat threads, then read conversations | Slack / Microsoft Teams |
calendar.list_events |
Find meetings and review times | Outlook Calendar / Google Calendar |
knowledge.search |
Search enterprise knowledge | Confluence / SharePoint / enterprise search |
files.search, files.read_json |
Find files and inspect bounded JSON sections | SharePoint / Google Drive plus a structured-file reader |
projects.search_tasks |
Find project work and blockers | Jira / Asana |
analytics.query_metrics |
Look up product metrics | Enterprise BI / product analytics |
support.search_tickets |
Find incident and support records | ServiceNow / Zendesk |
actions.prepare_message, actions.send_message |
Prepare a draft, then separately execute a send | Email / chat draft-and-send APIs |
The Hermes agent accesses these fictional MCP tools over a remote HTTP interface. The OpenShell policy specifies this access and prohibits other network egress.
Included Artifacts
| Step | What you do | Checked-in artifacts |
|---|---|---|
| Start the agent | Set up Hermes, OpenShell and the fictional world | Baseline profile, runtime configuration, sandbox setup, world records, workload |
| Collect source traces | Run distinct production-like requests | 42 traces, corpus index and analysis input |
| Discover issues | Run NeMo Compass on traces | Production findings, analyst configuration |
| Author Eval Tasks | Ask Codex + Eval Author to create Harbor eval tasks that represent common tasks and reproducible issues found in the traces | Authoring prompt, four Harbor tasks, technical proof receipts |
| Define hold-out cases | Separate some of the eval tasks into a hold-out set. This set is not used when proposing agent improvements, but is used as a last step to check if any proposed changes introduce regressions. | Two-development / two-held-out manifest, human review sheet |
| Execute the Eval Tasks | Run eval tasks three times each and retain scores with traces | Six scored baseline traces |
| Examine the traces from the Eval Tasks | Run NeMo Compass again on baseline development trajectories | Scored-development findings |
| Propose a change to improve the agent | Give Codex the results from NeMo Compass and ask it to propose an improvement to the agent | Proposal, candidate profile, experiment freeze |
| Test the proposed improvements | Run the eval tasks and check for improvements and regressions | Four baseline/candidate run summaries |
| Analyze results | Check improvement on both splits, regressions and uncertainty | A/B comparison, artifact hashes, results and limits |
For the tutorial run-through that is checked into the repository:
- NeMo Compass identified an issue in the 42 checked-in traces where the agent would draft/write messages without user confirmation.
- Eval tasks were created to reproduce this issue.
- Codex suggested a change to the Hermes SOUL.md to enforce user confirmation for drafts/writes.
- The proposed change worked, and all eval tasks passed.
Your run of the tutorial may produce different results. This optimization loop was used on a production agent running at NVIDIA in a similar scenario. The improvements implemented for this production agent are described in harness patterns.
Repository map
| Folder | Purpose |
|---|---|
| configs/ | Trace Analyst settings; ETHOS.md supplies the intended behavior used by analysis and task authoring. |
| docs/ | The walkthrough, measured results, harness patterns and future Gym extension. |
| evals/ | Eval-suite manifest, runnable Harbor tasks, proof receipts and human review sheet. |
| experiments/ | The 42-request workload used to collect production-like traces—not an eval suite. |
| fixtures/ | Deterministic fictional company records and initial tool state. |
| harbor_agents/ | Adapters that run Hermes in OpenShell and deliver answers/tool logs to Harbor. |
| openshell/ | Sandbox image recipe, network policy and model-provider profile. |
| profiles/ | Editable baseline and candidate Hermes system instructions. |
| prompts/ | The authoring request to give Codex when using Eval Author. |
| results/ | Saved Insights reports, candidate rationale, run summaries, comparisons and provenance hashes. |
| scripts/ | Commands for trace collection/conversion, eval execution and validation. |
| src/ | The mock MCP implementation, shared world/tool behavior and trace utilities. |
| tests/ | Automated tests for the environment, adapters, runners and artifact checks. |
| traces/ | Checked-in agent trajectories, corpus indexes and Trace Analyst inputs. |
Start here
Follow the walkthrough for the full tutorial guide, or read the notebook version. The notebook preserves the walkthrough's shell commands as Markdown so you can run them in the host and OpenShell terminals described in the guide.