Marshall Choy, Chief Business Officer at semiconductor firm Rebellions Inc., argues that the future of AI inference lies in commoditization rather than maintaining its status as a premium, scarce resource. He suggests that lowering costs will drive widespread adoption, similar to the historical expansion of electricity and cloud computing.
Currently, AI is deployed as a luxury product, leading companies to ration token usage and limit API calls to control costs. Choy notes that even Microsoft reportedly restricts AI usage. This approach limits the potential for enterprises to integrate AI into everyday business operations and continuous workflows.
The industry often fears that falling unit costs will shrink the market, but Choy compares this to the chocolate market, where affordable mass-market products expand the customer base. Lower inference costs enable new business models, such as ambient intelligence and autonomous systems, by making them economically viable at scale.
Choy emphasizes that the focus should shift from engineering metrics like throughput or benchmark scores to business outcomes such as task completion rates and cost savings. Infrastructure should be designed for consistency and efficiency to support integration into existing server environments.
Ultimately, the goal is to make AI infrastructure as reliable and accessible as mass-market vehicles. By moving away from high-cost, specialized systems, organizations can stop optimizing individual prompts and focus on the productivity gains that AI enables.
