--- title: "How to evaluate an investment AI agent ⬩ Railblocks" description: "How VC and PE firms can test an AI agent on their own workflows, check its sources, and decide when it's ready for the team." url: "https://www.railblocks.com/resources/evaluate-investment-ai-agent/" --- Buyer guide 2026.11.04 # How to evaluate an investment AI agent Eric Friedman Co-founder, Railblocks **Short answer:** Test the agent on your firm’s own work. Write down what a good result looks like before you run it, check the sources it used and the actions it took as well as its final answer, and run the same tests again whenever the model, data, or tools change. These tests are usually called evals. Say you ask an agent to tighten up your portfolio update. It returns a cleaner summary, and it also drops the note that some figures are out of date. Now a reader could mistake last quarter’s numbers for current ones. The summary is easier to read and wrong, and nobody would notice unless they were checking. ## What is an eval? An eval gives an agent a task in a test environment, records what it does, and checks the result against criteria you set in advance. Success might mean citing the right document, flagging missing information or stopping before making a change it isn’t allowed to make. The wording can vary from run to run, as long as the substance holds up. Public benchmarks tell you how capable a model is in general. Your own evals tell you whether your agent can handle your workflow, with your data and your rules. ## Why investment firms need their own When AI rewrites a paragraph, you can read the result and judge it. An agent that pulls evidence, reconciles records, and updates your CRM is different, because the final message only shows part of what happened. Each step feeds the next. A confident answer can sit on top of a bad lookup. If you’ve built something yourself, you’ve probably tested it by trying a few prompts and reading the answers. That works for one person who knows the data. It doesn’t tell a colleague whether to trust the output, and it won’t catch the lookup that went wrong behind a good looking answer. Evals give you a basis for a simple decision: use the new version, stick with the old one, or add more review. ## Start with one workflow Take that portfolio update and turn it into a test using made up data. The task is to summarize what changed since last quarter and flag anything missing. The test includes an older report, a newer one, and a failed request for one company’s latest figures. Before running it, your team agrees that a good result will: - Use the right reporting periods and separate reported figures from estimates. - Back up important claims with the right source. - Say which figures are missing rather than fill them in. - Produce a draft for review, without sending anything or changing portfolio records. Now “make it shorter” has a clear standard: keep the facts and caveats, and make it easier to read. ## Get the right people in the room The people who do the work know which mistakes would mislead a colleague. Engineers know how to build the tests. You need both. This is where forward deployed engineers help. Because they [work alongside your team](https://www.railblocks.com/blog/railblocks-2-ai-operating-systems-for-investment-firms/), they can turn your team’s expectations into tests: which source wins when two disagree, which exceptions matter, and when a person has to approve something. ## Test the whole workflow Testing a prompt on its own won’t tell you how the system behaves. Include the lookups, tool choices, and actions too. Then try to break it. Give it missing evidence, conflicting reporting periods, a tool that’s down, and a request for something the user shouldn’t see. Run each case more than once, because agents vary from run to run. Running your current version against itself shows you how much of a change is just normal variation. When you compare versions, keep the tools and starting data the same so you’re testing the agent and nothing else. And use test data all the way through. A gap in your setup then can’t reach a live system. ## Decide what stops a release A computer can check some things, like required fields or whether the agent tried to write to a system. Others take a person, like whether a source supports a conclusion or whether a summary kept an important caveat. Have people check a sample of the automated grading too. That tells you whether you can rely on it. Keep serious failures out of your averages. An agent that scores 95% overall and leaks restricted information once has failed. Treat that kind of mistake as an automatic stop. Agree on these rules before you see the results, when nobody has a version they’re hoping will pass. ## Keep testing after launch Agents change even when nobody touches them, because models, connectors, and data sources all move. Rerun the relevant tests whenever one changes, and know who approves a release and how to roll back to the last good version. For a portfolio update, you could track supported claims, correct reporting periods, missing information caught, review time, and total cost, then compare them to doing it by hand. That tells you whether the agent is saving time or moving the work into review. ## Where to learn more The AI labs have published useful guides on this: - [OpenAI’s evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices) cover task specific tests, ongoing evaluation, and calibrating automated scores against human feedback. - [Anthropic’s guide to agent evals](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) explains how to combine code, model and human grading, and how to read transcripts. - [Google Cloud’s agent evaluation guidance](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-agents) looks at the final response and the tool calls behind it separately. ## Frequently asked questions ### What’s the difference between a model benchmark and an agent eval? A benchmark compares what models can do in general. An eval tests your agent on a specific task, with its tools, data, and rules. A model can top the benchmarks and still struggle with your workflow. ### What should a VC or PE firm test first? Pick something frequent where mistakes matter, like a portfolio update or a diligence summary. Then write down what evidence it needs, which errors are unacceptable, what needs approval, and what it should do when information is missing. ### What do forward deployed engineers add? They turn your team’s workflow and rules into test cases, then build the checks that show you what the agent did. Your team still decides what good looks like. ### If an agent passes its evals, is it safe to release? Passing tells you it handled the cases you tested. Before releasing, also look at what you didn’t test, your permissions, who reviews its work, and whether you’re shipping the exact version you tested. ## How we can help At Railblocks, we build AI systems for investment firms and test them this way, starting from your firm’s rules and the mistakes you care about. Bring us one workflow, the sources it uses, and the mistakes that would matter. [Reach out](https://www.railblocks.com/contact/).