bloxyen in  
Software Engineer 2 months ago

GPT-4 outperforms its rivals in new AI benchmark suite GPT-Fathom

ByteDance and the University of Illinois researchers have developed an improved benchmark suite with consistent parameters, called GPT-Fathom, that indicates GPT-4, the engine behind the paid version of ChatGPT, significantly outperforms leading LLMs, including its biggest competitor, Claude 2.

For the latest advancements in AI, look here first.

GPT-Fathom's breakthrough

  • The new benchmark suite, GPT-Fathom, addresses consistent settings issues and prompt sensitivity, attempting to reduce inconsistencies in LLM evaluation.
  • In a comparison using GPT-Fathom, GPT-4 outperformed over ten leading LLMs, crushing the competition in most benchmarks, and showing significant performance leaps from GPT-3 to its successors.

Performance specifics

  • The gap in performance was especially pronounced against Claude 2, ChatGPT's biggest rival.
  • GPT-4's Advanced Data Analysis model exhibited superior performance in coding, giving it an edge as compared to LuckLlama 2, the current best-performing open-source model.
  • Llama 2-70B showed comparable or better performance than gpt-3.5-turbo-0613 in safety and comprehension but displayed worse performance in "Mathematics", "Coding", and "Multilingualism".

The seesaw effect

  • The research team noted a 'seesaw effect' where an improvement in one area can lead to degradation in another.
  • For instance, GPT-4 saw a performance drop on the Mathematical Geometry Simple Math (MGSM) benchmark, despite improving its performance significantly on text comprehension benchmark DROP.


P.S. If you like this kind of analysis, I write a free newsletter that tracks the most relevant news and developments in AI. Professionals from Meta, Google, and OpenAI are already reading it.

Subscribe | Supercharged AI

Subscribe | Supercharged AI

Stay on top of the latest AI news, research, and tools with our daily 2-min email. Join 1000+ readers from Google, Meta, OpenAI, and more.