I can’t seem to find an all-in-one solution that can do that right now…
Yes, partly.
There is still no single public site that perfectly answers a plain-English request like “I want natural chat and accurate calculation” and then returns a definitive ranked list. But there are now several places that are much closer to benchmark-first or task-first than the Open LLM Leaderboard, and the best current method is to combine them. (Hugging Face)
The direct answer
The closest things to what you want are:
- Hugging Face OpenEvals Official Benchmarks Leaderboard 2026. It is a unified leaderboard across official Hugging Face benchmarks, and its description says you can filter by selected tasks, model name, and size. That is much closer to benchmark-first browsing than the older Open LLM Leaderboard. (Hugging Face)
- Hugging Face Benchmark Finder. OpenEvals lists it as a Space to “view and inspect all the tasks in Lighteval,” which is explicitly task-oriented. (Hugging Face)
- Hugging Face’s benchmark leaderboard APIs and aggregated dataset. The docs now expose an official dataset-centric API, plus a pre-aggregated
OpenEvals/leaderboard-data Parquet file for cross-benchmark analysis. That is the strongest answer if you want to build your own task-first view. (Hugging Face)
- Arena.ai categories. This is not benchmark-first in the academic sense, but it is very useful for task-first shortlisting because it breaks public preference data into categories like math, instruction following, multi-turn, and creative writing. (arena.ai)
- Artificial Analysis. It is good for model comparison after you know the relevant task axes, because it exposes separate evaluation pages such as MATH-500 and IFBench alongside price, speed, and context comparisons. (Artificial Analysis)
- LiveBench. It is useful as a category-first sanity check because it scores models across math, reasoning, data analysis, language, coding, and instruction following, and it is designed to reduce contamination and rely on objective grading. (ukgovernmentbeis.github.io)
- OpenCompass and HELM. These are more evaluator-oriented than shopper-oriented, but they are strong if you want benchmark browsing and scenario-based comparison rather than one flat rank. OpenCompass says CompassHub is a benchmark browser, and HELM is built around many scenarios and metrics instead of a single score. (GitHub)
The important background
The Open LLM Leaderboard is mostly model-first: start with a model, then inspect benchmark results. Your question is the reverse: start with the task, then ask which models are strong on the relevant benchmarks.
That reverse workflow is now supported better than before. Hugging Face’s own docs explicitly describe a dataset-centric leaderboard API, where you query a benchmark and get ranked models, and they also provide a pre-aggregated multi-benchmark dataset for cross-benchmark views. In other words, the platform now officially supports the exact kind of rearrangement you are asking for. (Hugging Face)
But the broader problem is still only partially solved, because real user needs are not single benchmarks. “Natural chat and accurate calculation” is already a mixed task. It combines at least three separate abilities:
- conversational quality,
- numerical reliability,
- instruction discipline. (arena.ai)
The clean way to think about your example
Your example is not one task. It is a bundle.
1. Natural chat
This usually maps to:
- overall human preference
- multi-turn conversation
- creative writing / prose quality
- instruction following if you care about tone, brevity, or structure
Arena’s category system is unusually useful here because it explicitly separates these. Arena also states that a high overall rank does not mean a model will excel uniformly across every use case. (arena.ai)
2. Accurate calculation
This splits further:
- formal math reasoning, where MATH-500 is useful
- instruction-following under constraints, where IFBench or IFEval matter
- real-world numeracy, which standard math leaderboards do not fully capture
This last part matters a lot. NumericBench was proposed specifically because many existing benchmarks focus on language ability or structured math problems while missing basic real-world numerical tasks like arithmetic, contextual retrieval of numbers, comparison, and numeric summaries. (Artificial Analysis)
There is a second caution too: GSM-Symbolic reports that model performance can drop when only the numerical values are changed, even when the problem structure is otherwise the same. So “high math benchmark score” does not automatically mean “robust numeric behavior in normal use.” (arXiv)
So where is the information organized benchmark-first or task-first?
Best benchmark-first options
1. Hugging Face official benchmark APIs and OpenEvals/leaderboard-data
This is the most literal benchmark-first answer. You can:
- discover official benchmark datasets,
- call a benchmark’s leaderboard endpoint,
- or load one aggregated Parquet file with cross-benchmark scores. (Hugging Face)
2. OpenEvals Official Benchmarks Leaderboard 2026
This is the best current Hugging Face UI for browsing across official benchmarks rather than drilling into one model page. (Hugging Face)
3. OpenCompass CompassHub
This is closer to a benchmark browser for researchers and advanced users. Its own repo says CompassHub is designed to simplify exploring and using a large benchmark collection. (GitHub)
4. HELM
HELM is less about “pick a winner quickly” and more about “see many scenarios and metrics at once.” It is valuable because it rejects the idea that one score is enough. (arXiv)
Best task-first approximations
1. Arena.ai categories
This is the closest public tool to “I know the task shape, not the model name.” You can inspect categories like math, multi-turn, creative writing, and instruction following separately. (arena.ai)
2. Artificial Analysis
This is good after you define your task axes. It lets you compare models by benchmark family and by operational trade-offs like price, speed, and context. (Artificial Analysis)
3. LiveBench
This works well as a category-first objective cross-check because it already separates the capability areas instead of forcing you into one overall number. (ukgovernmentbeis.github.io)
What I would do for your exact use case
I would not ask “which model is best overall?” I would ask:
Which models are strong on conversational preference, instruction following, and numeracy at the same time?
That leads to a much better workflow.
Step 1. Translate your need into benchmark axes
For natural chat + accurate numbers, I would map it like this:
-
Natural chat
- Arena Overall
- Arena Multi-Turn
- Arena Creative Writing
-
Control
- Arena Instruction Following
- IFBench or IFEval
-
Numbers
- MATH-500
- LiveBench Math
- NumericBench if you care about ordinary real-world numbers rather than only contest-style math
-
Long sessions
- LongBench v2 or RULER if continuity across long conversations matters
This “task profile first” approach is the right one. (arena.ai)
Step 2. Generate a shortlist
Use Arena.ai or OpenEvals first, not the Open LLM Leaderboard alone.
- Use Arena if “naturalness” matters most.
- Use OpenEvals if you want benchmark-first filtering.
- Use Artificial Analysis if you also care early about price, speed, and context. (arena.ai)
Step 3. Cross-check category weaknesses
Then inspect the shortlisted models on:
- MATH-500
- IFBench
- LiveBench Math / Instruction Following
- long-context benchmarks if relevant. (Artificial Analysis)
Step 4. Run a tiny private test pack
This is the step public leaderboards do not replace.
Use maybe 12 to 20 prompts:
- 4 chat-only prompts
- 4 calculation-only prompts
- 4 mixed prompts where the model must stay natural and compute correctly
That last group matters most. Many models are separately good at chat and separately decent at math, but weaker when both are required at once.
A practical answer to your “inferencebench” idea
That idea makes sense, but later in the process.
Inference benchmarking tools are useful for latency, throughput, hardware fit, and serving efficiency. They are not the right first filter for “natural chat + accurate numbers.” For example, LLM-Inference-Bench describes itself as a hardware inference benchmarking suite across platforms and inference frameworks. That is valuable after you already know which models are quality candidates. (GitHub)
So the order should be:
- quality shortlist by task
- benchmark cross-check
- your own prompt pack
- inference/performance benchmark for deployment
Not the reverse. (GitHub)
What I would bookmark first
If you want the smallest useful set:
- OpenEvals Official Benchmarks Leaderboard 2026 for cross-benchmark browsing. (Hugging Face)
- OpenEvals Benchmark Finder for browsing tasks. (Hugging Face)
- Hugging Face leaderboard data guide for building your own benchmark-first view. (Hugging Face)
- Arena.ai categories for task-shaped shortlisting. (arena.ai)
- Artificial Analysis for benchmark pages plus price/speed/context comparison. (Artificial Analysis)
- LiveBench for a contamination-aware objective check. (ukgovernmentbeis.github.io)
The bottom line
So the answer is:
-
Yes, the ecosystem now has better benchmark-first and task-filtered options than the Open LLM Leaderboard alone. (Hugging Face)
-
No, there is still no single perfect public tool that takes a natural-language need and gives a fully reliable final answer. (arena.ai)
-
For your example, the best approach is to combine:
- a task-filtered shortlist from Arena.ai or OpenEvals,
- objective checks from MATH-500, IFBench, and LiveBench,
- and a small private eval set for your exact prompts. (arena.ai)
The main conceptual shift is this:
Do not choose a model, then inspect its scores.
Choose the capability profile, then find models that survive it.
That is the task-first version of model selection.
,