{"id":39300,"date":"2025-04-21T11:45:00","date_gmt":"2025-04-21T11:45:00","guid":{"rendered":"https:\/\/www.lifeandnews.com\/articles\/?p=39300"},"modified":"2025-04-23T04:39:43","modified_gmt":"2025-04-23T04:39:43","slug":"popular-ais-head-to-head-openai-beats-deepseek-on-sentence-level-reasoning","status":"publish","type":"post","link":"https:\/\/www.lifeandnews.com\/articles\/popular-ais-head-to-head-openai-beats-deepseek-on-sentence-level-reasoning\/","title":{"rendered":"Popular AIs head-to-head: OpenAI beats DeepSeek on sentence-level reasoning"},"content":{"rendered":"\n

Manas Gaur<\/a>, University of Maryland, Baltimore County<\/a><\/em><\/p>\n\n\n\n

ChatGPT and other AI chatbots based on large language models are known to occasionally make things up, including scientific and<\/a> legal citations<\/a>. It turns out that measuring how accurate an AI model\u2019s citations are is a good way of assessing the model\u2019s reasoning abilities.<\/p>\n\n\n\n

An AI model \u201creasons\u201d by breaking down a query into steps and working through them in order. Think of how you learned to solve math word problems in school.<\/p>\n\n\n\n

Ideally, to generate citations an AI model would understand the key concepts in a document, generate a ranked list of relevant papers to cite, and provide convincing reasoning for how each suggested paper supports the corresponding text. It would highlight specific connections between the text and the cited research, clarifying why each source matters.<\/p>\n\n\n\n

The question is, can today\u2019s models be trusted to make these connections and provide clear reasoning that justifies their source choices? The answer goes beyond citation accuracy to address how useful and accurate large language models are for any information retrieval purpose.<\/p>\n\n\n\n

I\u2019m a computer scientist<\/a>. My colleagues \u2212 researchers from the AI Institute at the University of South Carolina, Ohio State University and University of Maryland Baltimore County \u2212 and I have developed the Reasons benchmark<\/a> to test how well large language models can automatically generate research citations and provide understandable reasoning.<\/p>\n\n\n\n

We used the benchmark to compare the performance<\/a> of two popular AI reasoning models, DeepSeek\u2019s R1 and OpenAI\u2019s o1. Though DeepSeek made headlines<\/a> with its stunning efficiency and cost-effectiveness<\/a>, the Chinese upstart has a way to go to match OpenAI\u2019s reasoning performance.<\/p>\n\n\n\n

Sentence specific<\/h2>\n\n\n\n

The accuracy of citations has a lot to do with whether the AI model is reasoning about information at the sentence level<\/a> rather than paragraph or document level. Paragraph-level and document-level citations can be thought of as throwing a large chunk of information into a large language model and asking it to provide many citations.<\/p>\n\n\n\n

In this process, the large language model overgeneralizes and misinterprets individual sentences. The user ends up with citations that explain the whole paragraph or document<\/a>, not the relatively fine-grained information in the sentence.<\/p>\n\n\n\n

Further, reasoning suffers when you ask the large language model to read through an entire document. These models mostly rely on memorizing patterns that they typically are better at finding at the beginning and end of longer texts than in the middle<\/a>. This makes it difficult for them to fully understand all the important information throughout a long document.<\/p>\n\n\n\n

Large language models get confused because paragraphs and documents hold a lot of information, which affects citation generation and the reasoning process. Consequently, reasoning from large language models over paragraphs and documents becomes more like summarizing or paraphrasing<\/a>.<\/p>\n\n\n\n

The Reasons benchmark addresses this weakness by examining large language models\u2019 citation generation and reasoning. <\/p>\n\n\n\n

Testing citations and reasoning<\/h2>\n\n\n\n

Following the release of DeepSeek R1 in January 2025, we wanted to examine its accuracy in generating citations and its quality of reasoning and compare it with OpenAI\u2019s o1 model. We created a paragraph that had sentences from different sources, gave the models individual sentences from this paragraph, and asked for citations and reasoning.<\/p>\n\n\n\n

To start our test, we developed a small test bed of about 4,100 research articles around four key topics that are related to human brains and computer science: neurons and cognition, human-computer interaction, databases and artificial intelligence. We evaluated the models using two measures: F-1 score, which measures how accurate the provided citation is, and hallucination rate, which measures how sound the model\u2019s reasoning is \u2212 that is, how often it produces an inaccurate or misleading response<\/a>.<\/p>\n\n\n\n

Our testing revealed significant performance differences<\/a> between OpenAI o1 and DeepSeek R1 across different scientific domains. OpenAI\u2019s o1 did well connecting information between different subjects, such as understanding how research on neurons and cognition connects to human-computer interaction and then to concepts in artificial intelligence, while remaining accurate. Its performance metrics consistently outpaced DeepSeek R1\u2019s across all evaluation categories, especially in reducing hallucinations and successfully completing assigned tasks.<\/p>\n\n\n\n

OpenAI o1 was better at combining ideas semantically, whereas R1 focused on making sure it generated a response for every attribution task, which in turn increased hallucination during reasoning. OpenAI o1 had a hallucination rate of approximately 35% compared with DeepSeek R1\u2019s rate of nearly 85% in the attribution-based reasoning task.<\/p>\n\n\n\n

In terms of accuracy and linguistic competence, OpenAI o1 scored about 0.65 on the F-1 test, which means it was right about 65% of the time when answering questions. It also scored about 0.70 on the BLEU test, which measures how well a language model writes in natural language. These are pretty good scores.<\/p>\n\n\n\n

DeepSeek R1 scored lower, with about 0.35 on the F-1 test, meaning it was right about 35% of the time. However, its BLEU score was only about 0.2, which means its writing wasn\u2019t as natural-sounding as OpenAI\u2019s o1. This shows that o1 was better at presenting that information in clear, natural language.<\/p>\n\n\n\n

OpenAI holds the advantage<\/h2>\n\n\n\n

On other benchmarks, DeepSeek R1 performs on par<\/a> with OpenAI o1 on math, coding and scientific reasoning tasks. But the substantial difference on our benchmark suggests that o1 provides more reliable information, while R1 struggles with factual consistency.<\/p>\n\n\n\n

Though we included other models in our comprehensive testing, the performance gap between o1 and R1 specifically highlights the current competitive landscape in AI development, with OpenAI\u2019s offering maintaining a significant advantage in reasoning and knowledge integration capabilities.<\/p>\n\n\n\n

These results suggest that OpenAI still has a leg up when it comes to source attribution and reasoning, possibly due to the nature and volume of the data it was trained on. The company recently announced its deep research tool<\/a>, which can create reports with citations, ask follow-up questions and provide reasoning for the generated response.<\/p>\n\n\n\n

The jury is still out on the tool\u2019s value for researchers, but the caveat remains for everyone: Double-check all citations an AI gives you.<\/p>\n\n\n\n

Manas Gaur<\/a>, Assistant Professor of Computer Science and Electrical Engineering, University of Maryland, Baltimore County<\/a><\/em><\/p>\n\n\n\n

This article is republished from The Conversation<\/a> under a Creative Commons license. Read the original article<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

Manas Gaur, University of Maryland, Baltimore County ChatGPT and other AI chatbots based on large language models are known to occasionally make things up, including scientific and legal citations. It turns out that measuring how accurate an AI model\u2019s citations are is a good way of assessing the model\u2019s reasoning abilities. An AI model \u201creasons\u201d […]<\/p>\n","protected":false},"author":56,"featured_media":39301,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[5,826,277,291,825,25,28,15533,8],"tags":[16280,10656,16281,13297,885,891,886,860,13885,16279,255],"_links":{"self":[{"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/posts\/39300"}],"collection":[{"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/users\/56"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/comments?post=39300"}],"version-history":[{"count":1,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/posts\/39300\/revisions"}],"predecessor-version":[{"id":39302,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/posts\/39300\/revisions\/39302"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/media\/39301"}],"wp:attachment":[{"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/media?parent=39300"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/categories?post=39300"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/tags?post=39300"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}