Attorneys Spencer Glende and Samuel J. Hartman from the law firm Mayer Brown suggest that companies should evaluate AI agents using a structured pilot program. This approach aims to mitigate risks such as financial loss, legal complications, and reputational damage that can arise from premature deployment or excessive caution.
The authors recommend treating AI agents like job candidates by assessing three key criteria: capability, safety, and return on investment. A formal pilot should begin with a clear job description, defining the agent's specific tasks, data access, and operational boundaries, such as spending limits for purchasing agents.
Testing protocols should focus on task completion rates, accuracy, and the agent's ability to handle complex or fraudulent scenarios. Companies must verify that safety measures, including emergency off switches and privacy protections, function correctly in practice rather than relying solely on contractual language.
Financial assessment is a critical component of the evaluation process. Beyond software fees, companies must account for computing costs and the potential need for additional human supervisors, even if the agent automates certain tasks.
Every pilot program requires a defined end date and a clear decision-making process. The authors emphasize that trials should not transition into full-scale deployment by default. Contracts should specify outcomes for unsuccessful tests, such as extending the trial, implementing further fixes, or opting for a limited rollout.
