Fluent, Polite, and (Maybe) Wrong: Conversational Norms and User Evaluation of AI Agents in Complex Reasoning Tasks

Research on conversational artificial intelligence (AI) from a pragmatic perspective shows that users evaluate these systems not only for accuracy and efficiency, but also for communicative qualities such as politeness, coherence, and responsiveness, shaped by expectations from human interaction. This study examines differences in conversational style across AI systems in reasoning tasks, focusing on politeness, user agency, conversational repair, and the effect of interaction order on user evaluations. Using a mixed-methods approach, the study combines quantitative conversational coding with qualitative analysis. Twenty-three participants interacted with two AI systems while completing a structured reasoning task. Interactions were coded for solution divergence, politeness and agency markers, responsiveness, and strategies for managing expectation violations, and complemented by post-task interviews. Findings show stable stylistic differences between systems: one more dialogical and user-oriented, the other more procedural. Politeness and conversational fluency significantly shaped perceived trust and competence, even when logical inconsistencies were present, highlighting a gap between interactional quality and objective task performance. An anchoring effect was also observed: the system encountered first functioned as an implicit evaluative benchmark, with prior familiarity with ChatGPT generating more demanding standards when it was encountered second, while Claude benefited from a novelty effect when experienced first.