‘Pretty Crazy’ Token Usage Is Testing Bosses’ Bet on AI

0
2

At the software company 8×8, employees are using Anthropic’s Claude to draft emails, analyze customer feedback, and write code, but so far, their growing reliance on the artificial intelligence chatbot hasn’t troubled the finance team. While other Silicon Valley companies, such as Meta, Uber, and Salesforce, have publicly expressed concerns about the growing cost of generative AI tools and have begun introducing usage caps in some cases, 8×8 says it finds itself in the black.

Over the past 18 months, the company estimates it has saved about $5 million in annual costs by canceling subscriptions to dozens of software and educational tools it deemed unnecessary in part because Claude could provide similar capabilities. So far, 8×8’s annualized bill for Claude is “well below” that figure, says Joel Neeb, the company’s chief transformation and business operations officer.

Neeb expects the savings and costs to eventually even out as 8×8 encourages more employees to adopt AI and it incorporates the tech into more complicated work. But for now, there’s still a huge gap, which “makes my chief financial officer happy,” he tells WIRED. He declined to share exact total spending on generative AI.

As companies pour hundreds of millions of dollars collectively into AI tools for coding, marketing, and customer service, a new obsession has emerged in the tech industry: “tokenomics,” or how to manage the soaring cost of AI usage. (Tokens represent the amount of content an AI model analyzes and generates.)

Last month, Royal Bank of Canada’s CEO disclosed that its token usage surged 500 percent over the past six months. At Cisco, a third of employees are using an internal AI chatbot on a daily basis, so “the token usage is getting pretty, pretty crazy,” CEO Chuck Robbins said on an earnings call. Some top engineers at analytics software developer Amplitude are “spending thousands of dollars a month or more on tokens,” according to its CEO Spenser Skates. Aaron Levine, the CEO of Box, said, “The token budgeting conversation has absolutely taken over as one of the most important” and “heated” topics.

Roughly 300 companies addressed questions or concerns about AI tokens during their earnings calls or in public discussions with financial analysts in April or May, according to a WIRED review of transcripts from the data provider AlphaStreet. That’s a small fraction of the thousands of calls held during the span, but just 93 companies mentioned “token” in April and May a year ago.

Executives at several companies said they are developing or looking to buy systems to help monitor token usage and choose the lowest-priced model for a given prompt. Others said they were still trying to figure out balancing hiring more people and increasing their budgets for tokens to achieve their goals.

Software has rarely come cheap, but the latest generation of AI tools is causing unusual stress in C-suites for a variety of reasons. Prices keep fluctuating. New models that are more powerful—and more expensive—than the last get released every month. And getting entire organizations on board with new ways of working has been a challenge, so AI-fueled productivity gains on one team can lead to bottlenecks for another.

20 Percent

That said, some companies are still encouraging employees to use AI more without worrying about the tab. In April, Long Island, New York-based clothing brand Baseball Lifestyle 101, which expects to generate $250 million in sales this year, told about 50 of its top managers to spend the equivalent of about 20 percent of their salary on AI tokens every month.

Bill Rom, cofounder and chief strategy officer of Baseball Lifestyle 101, tells WIRED the cost is likely to exceed $100,000 a month by the end of the year, but it’s already paying off. Claude recently helped land a $1 million order by identifying that a retailer was running low on some sizes of the company’s popular ice-cream-patterned shorts. “That’s a day and a half of work that can now happen in an hour or two that might make me eight figures of additional revenue over 12 months,” Rom says.

The AI chatbot also helps write financial reports and plan photoshoots, allowing the company to hire fewer junior staffers and to direct investments elsewhere. Rom says it’s important to “inspire people how to use AI” before setting financial ground rules on the technology.

At 8×8, which develops a communications platform to help manage sales and customer service, all of the 1,800 or so full-time employees are encouraged to regularly check a dashboard showing how much they and their colleagues are using Claude in the spirit of not leaving anyone behind. “It’s not punitive in the least; it’s really just so that we all stay tightly packed in this journey,” Neeb says. In May, the product and customer success teams were among the heaviest users, and the sales and finance teams were among the lightest.

Neeb says it’s possible that 8×8 will institute caps on how much staff can use Claude. He discussed the idea for the first time recently with the CFO due to growing internal usage of the Claude Opus 4.8 model, which was released last month and costs nearly 1.7 times more than an offering Anthropic released in February. Though no decisions have been made, access to Opus going forward might require proving that older models can’t get the job done, Neeb says. “Can we downgrade the model a little bit and still get the same outcome?”

Otherwise, Neeb says 8×8 isn’t backing off generative AI in any way. Measures of customer satisfaction and loyalty have been trending higher, and revenue has grown for four consecutive quarters as AI-generated analyses accelerate the work of its sales staff. Attributing the trends to solely AI, or even AI at all, is difficult, but Neeb suspects a connection. “It really is the rising tide that floats all boats when you do this right,” he says.

“Go Faster”

About two years ago, 8×8 began wading into generative AI by giving all of its employees training and support for OpenAI’s ChatGPT and Google’s Gemini. Later, a select few were offered Claude, Neeb says, and it became the companywide standard over the past year.

Management monitors usage and has warned employees that refusing to gain AI fluency will lead to consequences. “If you’re not using AI in some capacity for your role, then you’re missing the opportunity to go faster and get better answers more effectively than your peers,” Neeb says.

Other tech employers have issued similar directives to mixed results. At companies like Amazon and Meta, workers have reported incorporating AI into their work just because they feel they have to or slacking off because AI tools free up their time, both leading to what critics have described as waste. Neeb contends that patience and accountability measures are necessary to orient employees. He doesn’t want them taking longer lunches or sitting on the beach as AI speeds up their work.

Neeb also has wanted more out of what he describes as “laggards in this journey” such as the sales and finance teams, which together account for 28 percent of employees at the company but just 15 percent of token consumption. He’s hoping a recent AI hackathon for the finance team spurs it to automate its extensive manual processes, such as collecting money from customers and generating quarterly accounting.

The operations leader says he has witnessed firsthand how Claude can make his company more efficient. Neeb uses the tool to automate a daily email to the company that summarizes top AI usage tips from industry influencers on YouTube. After noticing that the task used up “a lot of tokens,” Neeb says he asked Claude whether it could operate more affordably. Claude reworked the automation, cutting token usage by 80 percent.

Disclaimer : This story is auto aggregated by a computer programme and has not been created or edited by DOWNTHENEWS. Publisher: wired.com