Artificial intelligence “Employees” can disrupt work in unexpected ways

Over the past year or two, companies have started using so-called AI agents as “bona fide employees,” even including them in their organizational charts.

Emma Wiles, a professor at Boston University who studies how artificial intelligence affects workers, came across the phenomenon in October at a conference where two human resources managers said treating AI agents like real employees is a way to increase productivity and put their companies on top.

But when Dr. Wiles and three associates from the Boston Consulting Group further investigatedthey discovered pitfalls. In an experiment involving dozens of companies with AI employees, researchers found that managers tended to review documents less carefully when they were told an AI employee had created them. Managers missed mistakes other managers caught when they were told they were reviewing a person’s work.

Dr. Wiles speculated that managers didn’t think it was their responsibility to fix AI employees’ mistakes. If something went wrong, they could dismiss it as the fault of the tech team or the managers who wanted AI employees in the first place. “But that’s not your problem,” she said, directing the managers’ thinking to their own roles.

In the years since artificial intelligence has arrived on the scene, many companies have realized the shortcomings that the technology brings and have occasionally taken steps to compensate for them. They know that AI models can be biased against certain groups of people, such as non-whites. They know that chatbots can provide confident but incorrect answers to questions. They know that bots sometimes leak information that should remain private.

But as companies try to introduce AI into their day-to-day operations, researchers are discovering more subtle flaws. In principle, these shortcomings can also be corrected. For example, companies could hold managers directly responsible for the mistakes of AI subordinates.

In practice, however, most business users seem blissfully unaware of these issues, raising the possibility that AI’s promise of increased productivity and huge cost savings could be undermined.

Even researchers who study AI may be aware of only a fraction of the problems this technology brings. “There are a number of unknown unknowns,” said Dr. Wiles.

One well-documented but underappreciated flaw of AI models is that they tend to favor AI-produced work. And 2025 paper in The Proceedings of the National Academy of Sciences found that several large language models had a low opinion of human-written text, creating “a potentially consequential form of implicit ‘anti-human’ bias.”

However, many companies seemed unaware of the problem, or at least unable to imagine how it could wreak havoc on their operations. As the team of scientists explained in the following paperThe discovery that the AI ​​models that companies use to evaluate resumes tend to favor those written with AI over those written entirely by humans has caught the attention of some corporate recruiters.

Jane Yi Jiang, an operations professor at Ohio State University who is the author of the following paper, said she and her co-authors were happy to help recruit companies asking “how to improve their processes.”

But they noted that this was almost certainly not the only problem that companies inadvertently introduced in their rush to implement AI.

For example, some companies are now using artificial intelligence to answer questions like how much to charge for a product or where to open a new location. However, relying on technology for such purposes can quickly go off the rails.

When left to their own devices, people often cooperate and seek mutually beneficial outcomes. But when AI models evaluate the situation, yes tend to adopt a more coldly calculating, “rational” mind-set that stems from basic game theory. They can, say, lead a company to aggressively undercut a competitor, even if it risks a damaging price war.

“Most of the LLMs we test think that human beings are more rational than they really are,” said Jiannan Xu, Ph.D. candidate at the University of Maryland and associate Dr. Jiang’s. However, in many cases “the most rational response leads to a bad situation for everyone”.

In principle, AI developers and users can correct these biases. Dr. For example, Jiang and Mr. Xu found that they could reduce anti-human biases by simply instructing the models to focus on the quality of the written material they were evaluating and avoid thinking about the author.

But AI researchers can’t correct for biases they’re not aware of, and several researchers said the impact of these undetected biases could be growing. One way is that future models will be trained on data produced by today’s models without sufficient care, creating a kind of self-reinforcing loop.

In that case, “a tendency to consolidate on existing perspectives and behaviors seems likely,” said Shayne Longpre, an AI researcher and founder of Data Provenance Initiativea group that monitors AI infrastructure.

And then there are blind spots that arise not so much from AI itself, but from the way humans use it.

Scholars turning to AI at every stage of the research process—asking AI what questions are worth studying; ask for advice on how to answer these questions; its use for data analysis; rely on it to help write up the findings – could inadvertently narrow the scope of their work.

“We don’t necessarily notice it on an individual level,” said Cecilie Steenbuch Traberg, a psychologist at the Copenhagen Business School and author of a recent paper to the topic. “You’re sparring with a chatbot, it helps me come up with ideas, you might think that sounds cool. But on a collective level, it looks pretty similar. They all sound the same.”

Dr. Wiles, a Boston University professor who has studied the way people manage AI workers, said the flaws weren’t necessarily inherent in the technology, but arose when people adopted it with little regard for what could go wrong.

She and her colleagues surveyed more than 1,000 business executives and found that about one-third said their organizations refer to AI as a “teammate or employee,” and that nearly one-quarter said their employer has included AI agents in their organizational charts. “We call it Scout,” one manager told researchers in an interview, referring to the AI ​​agent. “He’s technically an equal on your team.

Dr. Wiles and her colleagues gave all the managers they studied a set of five documents that contained errors and gave them 20 minutes to review as many of them as possible. In some cases, they told managers that the work was done by an AI employee; in some cases they stated that an AI tool did the work; and in some cases they said the work was done by man.

Generally speaking, the source of the documents listed didn’t make much of a difference in how carefully managers reviewed them.

But managers at companies that incorporated AI agents into their organizational charts caught significantly fewer errors when they were told they were reviewing the work of an AI employee.

People who manage people tend to assume that “if someone on my team makes a mistake, it’s on me,” explained Dr. Wiles and therefore carefully inspect the work of these subordinates. Managers also seem to assume they are on the hook for work created by an inanimate AI tool. However, managers at companies with AI employees do not seem to feel the same responsibility for the work of these employees.

Her insights: Over the past few centuries, scientists and business leaders have developed a reliable set of practices for managing people. But the psychology of driving an anthropomorphized artificial intelligence is vastly different, and “we’re going there blind.”

He fears the problem will get worse. At the same conference where she first heard HR officials talk about the benefits of their AI employees, one went even further and said her company would soon have AI employees managing humans. “The room fell silent,” Dr. Wiles recalled.

“We’ll also need someone to study it,” she added.