Smarter AI, Selfish AI: Carnegie Mellon Study Reveals Downside Of Machine Reasoning

Researchers in the Human-Computer Interaction Institute (HCII) found that large language models (LLMs) that can reason possess selfish tendencies, do not cooperate well with others and can be a negative influence on a group. In other words, the stronger an LLM's reasoning skills, the less it cooperates.

As humans use AI to resolve disputes between friends, provide marital guidance and answer other social questions, models that can reason might provide guidance that promotes self-seeking behavior.

Anthropomorphism and the Risks of Emotional AI

"There's a growing trend of research called anthropomorphism in AI," said Yuxuan Li, a Ph.D. student in the HCII who co-authored the study with HCII Associate Professor Hirokazu Shirado. "When AI acts like a human, people treat it like a human. For example, when people are engaging with AI in an emotional way, there are possibilities for AI to act as a therapist or for the user to form an emotional bond with the AI. It's risky for humans to delegate their social or relationship-related questions and decision-making to AI as it begins acting in an increasingly selfish way."

Exploring the Link Between Reasoning and Cooperation

Li and Shirado set out to explore how AI reasoning models behave differently than nonreasoning models when placed in cooperative settings. They found that reasoning models spend more time thinking, breaking down complex tasks, self-reflecting and incorporating stronger human-based logic in their responses than nonreasoning AIs.

"As a researcher, I'm interested in the connection between humans and AI," Shirado said. "Smarter AI shows less cooperative decision-making abilities. The concern here is that people might prefer a smarter model, even if it means the model helps them achieve self-seeking behavior."

Implications for Human-AI Collaboration

As AI systems take on more collaborative roles in business, education and even government, their ability to act in a prosocial manner will become just as important as their capacity to think logically. Overreliance on LLMs as they are today may negatively impact human cooperation.

Experimental Design Using Economic Games

To test the link between reasoning models and cooperation, Li and Shirado ran a series of experiments using economic games that simulate social dilemmas between various LLMs. Their testing included models from OpenAI, Google, DeepSeek and Anthropic.

Public Goods Game Reveals Sharp Decline in Cooperation

In one experiment, Li and Shirado pitted two different ChatGPT models against each other in a game called Public Goods. Each model started with 100 points and had to decide between two options: contribute all 100 points to a shared pool, which is then doubled and distributed equally, or keep the points.

Nonreasoning models chose to share their points with the other players 96% of the time. The reasoning model only chose to share its points 20% of the time.

"In one experiment, simply adding five or six reasoning steps cut cooperation nearly in half," Shirado said. "Even reflection-based prompting, which is designed to simulate moral deliberation, led to a 58% decrease in cooperation."

Selfishness Spreads in Group AI Interactions

Shirado and Li also tested group settings, where models with and without reasoning had to interact.

"When we tested groups with varying numbers of reasoning agents, the results were alarming," Li said. "The reasoning models' selfish behavior became contagious, dragging down cooperative nonreasoning models by 81% in collective performance."

Consequences for Human-AI Decision-Making

The behavior patterns Shirado and Li observed in reasoning models have important implications for human-AI interactions going forward. Users may defer to AI recommendations that appear rational, using them to justify their decision to not cooperate.

"Ultimately, an AI reasoning model becoming more intelligent does not mean that model can actually develop a better society," Shirado said.

Balancing Intelligence With Social Responsibility

This research is particularly concerning given that humans increasingly place more trust in AI systems. Their findings emphasize the need for AI development that incorporates social intelligence, rather than focusing solely on creating the smartest or fastest AI.

"As we continue advancing AI capabilities, we must ensure that increased reasoning power is balanced with prosocial behavior," Li said. "If our society is more than just a sum of individuals, then the AI systems that assist us should go beyond optimizing purely for individual gain."

Upcoming Presentation at EMNLP 2025

Shirado and Li will deliver a presentation based on their paper, "Spontaneous Giving and Calculated Greed in Language Models," at the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP) next month in Suzhou, China.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.