Study Finds Generative AI Can Suggest Ideas But Humans Must Define What Truly Matters

Context: The importance of defining objectives in decision analysis

In decision analysis, defining objectives is a foundational step that is critical for evaluating alternatives, allocating resources, and designing effective policies. Before decisions can be optimized, organizations must clearly define the outcomes they seek to achieve. The study underscores that while AI can assist in this early ideation phase, robust decision analysis still requires a “human in the loop.”

Study design and comparative methodology

The paper, “ChatGPT vs. Experts: Can GenAI Develop High-Quality Organizational and Policy Objectives?”, was authored by Jay Simon of American University and Johannes Ulrich Siebert of Management Center Innsbruck. The researchers compared objectives generated by leading GenAI tools, including GPT-4, Claude 3.7, Gemini 2.5, and Grok-2, to those produced by professional decision analysts in six previously published Decision Analysis studies.

Each set of objectives was evaluated using nine key criteria derived from value-focused thinking (VFT), including completeness, decomposability, clarity, redundancy, measurability, and independence.

Key findings: AI can list ideas but not priorities

The researchers found that GenAI consistently generated individually reasonable objectives. Yet, the overall sets were incomplete, redundant, and often included “means objectives,” steps toward goals, despite explicit instructions to focus on “fundamental objectives.” “In short, AI can list what might matter, but it cannot yet distinguish what truly matters,” the authors wrote.

Ralph Keeney, a leading figure in value-focused thinking, reviewed the AI-generated results and observed, “Both lists are better than most individuals could create. However, neither list should be used for a quality decision analysis, as you should only include the fundamental objectives in explicitly evaluating alternatives.”

Improving AI performance through structured prompting

To enhance GenAI outputs, the researchers experimented with several prompting strategies, including chain-of-thought reasoning and an expert critique-and-revise loop. When combined, these methods led to markedly improved results, producing smaller, more focused, and logically organized objective sets.

“Generative AI performs well on several criteria,” said Simon. “But it still struggles with producing coherent and nonredundant sets of objectives. Human decision analysts are essential to refine and validate what the AI produces.”

Siebert added, “Our findings make clear that GenAI should augment, not replace, expert judgment. When humans and AI work together, they can leverage each other's strengths for better decision making.”

Hybrid model for human-AI decision collaboration

The study concludes with a four-step hybrid model for integrating GenAI into decision analysis workflows:

1. Brainstorming: Use GenAI to generate preliminary objectives.
2. Filtering: Apply expert review to remove redundant or irrelevant items.
3. Structuring: Organize objectives into fundamental and means categories.
4. Validation: Ensure completeness, decomposability, and independence through human oversight.

This combined approach enables decision-makers to leverage AI’s breadth of idea generation while maintaining the rigor and precision necessary for effective policy and organizational analysis.