I Tried Making my Own (Bad) LLM Benchmark to Cheat in Escape Rooms

Not too long ago, DeepSeek introduced their newest mannequin, R1, and article after article got here out praising its efficiency relative to price, and the way the discharge of such open-source fashions may genuinely change the course of LLMs ceaselessly. That’s actually thrilling! And likewise, too massive of a scope to put in writing about… however when a mannequin like DeepSeek comes out of nowhere with a metal chair, boasting comparable efficiency ranges to different fashions, what does efficiency actually imply on this context?

When you comply with AI releases, you’ve seen this dance earlier than. Each new mannequin drops with its graphs displaying the way it’s someway concurrently higher than GPT-4 on math issues whereas being smaller and extra environment friendly. However what precisely are these benchmarks measuring? How are they created? And extra importantly, how can we minimize by means of the hype to create our personal benchmarks for particular use instances?

I needed to study extra about LLM Benchmarking.

Half 1: What’s a Benchmark? (in 3 seconds)

TL:DR — The SATs (a number of, really) for LLMs.

Half 1.1: What’s a Benchmark? (in additional than 3 seconds)

Earlier than we dive into the nitty-gritty of particular benchmarks, let’s take a second to unpack what we even imply by “LLM Benchmark.” As a result of calling them the “SATs for AI” feels each proper and likewise barely oversimplified.

LLM benchmarks are, at their core, structured assessments used to measure how effectively giant language fashions carry out on sure duties. These duties could be something from figuring out if a press release is true or false, to summarizing a authorized doc, to producing legitimate Python capabilities. Consider them as curated impediment programs specifically designed by AI researchers to check each related muscle these fashions may need. These frameworks sometimes present a dataset of inputs with recognized right outputs, permitting for constant comparability between fashions.

Trendy benchmarks make use of varied analysis methodologies. Classification metrics like accuracy work for duties with discrete right solutions, whereas overlap-based metrics (BLEU, ROUGE) consider free-form textual content era. Some benchmarks use practical testing for code era, or make use of different LLMs as judges to judge response high quality.

A typical benchmark often comes packaged as:

• A standardized dataset of questions, prompts, or duties (with right or reference solutions).
• An analysis protocol specifying the right way to measure success, like accuracy, F1 rating, BLEU/ROUGE for textual content era, or go/fail charges for coding duties.
• A leaderboard or some type of comparative scoreboard, typically with massive flashy graphs.

Some actually well-known benchmarks embrace MMLU for testing multitask language understanding, TruthfulQA for assessing factual accuracy, and HumanEval for measuring coding capabilities. Outcomes are fairly typically revealed on public leaderboards, which let’s folks carry out some clear comparability between completely different fashions.

What Makes a Good Benchmark?

1. A Clear Process Definition: We wish duties which might be unambiguous. The extra simple and well-specified the problem, the simpler it’s to belief the outcomes.
2. Information Integrity: The take a look at set shouldn’t be floating round within the coaching knowledge. As a result of if the mannequin’s seen the very same query 50 instances earlier than, the analysis is about as helpful as giving a math quiz to somebody who already has the reply key.
3. Quantifiable Metrics: You want an ordinary for scoring efficiency — like what number of instances the mannequin’s code passes take a look at instances or how shut the generated abstract is to a “ground-truth” abstract.
4. Process Range & Problem: If a benchmark is just too simple, everybody simply ACES it on day one, and we study… effectively, nothing. If it’s too area of interest (like “We take a look at solely the mannequin’s means to depend the digits of Pi for 20 minutes”), that’s additionally not so useful.

Life Ain’t All about The Grades

Benchmarks seize solely a slice of what LLMs can do. In the actual world, your chatbot may have to juggle area information, hold monitor of dialog context, abide by your organization’s insurance policies, and produce fluent, non-offensive replies. No single standardized take a look at on the market absolutely covers that. As we’ll see within the upcoming case research, the design and execution of a benchmark can closely form the image you get of your mannequin’s efficiency… and typically lead you astray should you’re not cautious with the way you measure success.

Now that now we have a way of what Llm Benchmarks are designed to perform (and the place they could fall brief), let’s discover a few examples to see how folks really construct and use them in observe — with blended outcomes!

Case Research #1: Leetcode as an LLM Benchmark

As a pupil within the tech house, the phrase “Leetcode” popping up throughout my seek for cool benchmarks raised by blood stress by a statistically vital quantity. In contrast to Leetcode, which sucks, the paper “Efficiency Research of LLM-Generated Code on Leetcode” was very attention-grabbing — it asks a deceptively easy query: can we use Leetcode to benchmark LLM code era? Their findings reveal each the promise and pitfalls of this method.

The Benchmark Design

The researchers constructed a three-stage validation system. Native assessments catch primary errors, Leetcode’s decide verifies correctness, and a customized benchmarking setup measures efficiency. This setup revealed one thing essential: benchmarking code efficiency is tougher than it appears.

After they in contrast native measurements to Leetcode’s metrics, they discovered solely a 0.28 correlation. Leetcode’s measurements confirmed a lot larger variation (0.089 vs 0.035 regionally). Even worse, Leetcode’s rankings proved unstable — similar options may drop from the 77th to 54th percentile simply based mostly on submission timing.

The Actual Issues

Three main points emerged that problem Leetcode’s viability as a benchmark:

Information Contamination: Utilizing public issues dangers LLMs having seen the options throughout coaching. The researchers had to make use of solely issues from 2023 to mitigate this.

Platform Instability: Leetcode’s metrics drift over time — reminiscence measurements confirmed a -0.24 correlation with take a look at date. This makes reproducible benchmarking practically inconceivable.

Measurement Reliability: The weak correlation between native and platform measurements raises questions on what we’re really testing.

What It Means for LLM Benchmarking

This research doesn’t simply critique Leetcode — it highlights what we want in a code era benchmark: reproducible measurements, dependable efficiency metrics, and assured training-test separation. Till now we have platforms constructed particularly for this objective, we should be extraordinarily cautious about utilizing competitors platforms as benchmarks.

So! We all know that not all benchmarks are viable benchmarks — what a couple of extra mainstream one?

Case Research #2: SuperGLUE — Constructing a Higher Language Understanding Benchmark

The SuperGLUE paper tackles a captivating downside in AI benchmarking: what do you do when fashions get too good at your assessments? When GLUE turned inadequate (with fashions surpassing human efficiency), the researchers needed to rethink how we measure language understanding.

The Benchmark Design

SuperGLUE’s core innovation is its process choice methodology. The researchers collected process proposals from the NLP group and filtered them by means of a rigorous course of: every process wanted clear analysis metrics, public coaching knowledge, and — most significantly — vital headroom between machine and human efficiency.

This resulted in eight duties (I’ve simplified the desk from the doc right here, it’s rather less readable however you need to get the sense of what the questions are asking):

What makes these duties particular is their variety in format. In contrast to GLUE’s deal with sentence classification, SuperGLUE consists of coreference decision, studying comprehension, and extra com plex reasoning duties. Every process measures completely different facets of language understanding whereas sustaining clear, quantifiable metrics.

---

Half 2: Let’s Construct a Bodily Reasoning Benchmark: To Cheat at Escape Rooms

After some benchmarks like SuperGLUE and Leetcode, I had an concept: what if we examined LLMs on one thing fully completely different — bodily reasoning… by means of escape room puzzles?

It’s a reasonably legitimate concept — escape rooms poses prospects and penalties for failure — screw up one too many puzzles, and your folks will assume you’re fairly silly, and relegate you to spectator responsibility. Fortunately for us nonetheless, they (or the poor staff) don’t know you could sneak a telephone into an escape room — and you realize simply who to ask for the solutions. In the present day, LLMs face off towards the puzzles of a bodily escape room.

Be aware: That is NOT a rigorous tutorial benchmark (please don’t cite this in papers, why would you even wish to try this?), and even near it, and it’s simply speculated to be a enjoyable approach to take a look at LLM benchmarking and analysis. Please don’t destroy my prompts, I’m conscious they’re unhealthy.

Why Bodily Reasoning?

For actual, although… most LLM benchmarks deal with linguistic duties (like SuperGLUE) or code era (like Leetcode). And for good motive — these are well-defined domains with clear analysis metrics. However real-world downside fixing typically requires understanding bodily ideas and their interactions. The well-known “Can GPT-4 do physics?” debates often focus on mathematical problem-solving, not sensible bodily reasoning.

Taking a look at present benchmarks taught me just a few key ideas:

1. Clear analysis metrics are essential (from SuperGLUE’s task-specific scores)
2. Issues ought to have unambiguous options (from HumanEval’s take a look at instances)
3. The benchmark ought to take a look at distinct capabilities (from MMLU’s topic classes)

Designing the Issues

I settled on escape room puzzles for 2 causes. First, they naturally mix bodily reasoning with clear targets. Second, they’ve unambiguous success circumstances — both you remedy it by means of the supposed method, otherwise you don’t. Third, and most significantly, they let me embrace “pink herrings” — irrelevant gadgets that take a look at if the LLM can establish what issues bodily. Fourth, I simply actually like doing escape rooms (did I point out that already?),

I’m conscious that that is greater than two causes, but when LLMs can’t depend what number of rs’ there are in strawberry, I’m allowed to mess up on occasion too.

Right here’s how I structured the 5 core issues:

Fluid Dynamics (FLUID_001) (Ping pong ball caught in a tube)

• Exams understanding of buoyancy and fluid displacement
• Impressed by basic physics issues however in sensible context
• Consists of deliberately irrelevant gadgets (like squishy meals fashions)

Gentle Properties (UV_001) (UV mild on a push numebr lock)

• Exams understanding of UV fluorescence and materials properties
• Combines a number of bodily ideas (mild, materials science)
• Requires understanding of environmental circumstances

Mechanical Understanding (CIPHER_001) (A cipher ring)

• Exams spatial reasoning and mechanical alignment
• No pink herrings — assessments for correlating a dial to a cypher wheel
• Requires understanding rotational symmetry

Pressure Software (VAC_001) (Can caught in gap)

• Exams understanding of vacuum forces and floor adhesion
• A number of doable answer approaches
• Requires understanding power multiplication

Collaborative Physics (COLLAB_001) (Can two folks shimmy a key?)

• Exams understanding of bodily constraints in multi-agent eventualities
• Requires combining a number of bodily ideas
• Exams understanding of device creation and friction

Sounds actually fancy… but it surely’s just a few primary bodily puzzles. You may entry them on my GitHub.

The Technical Half

The benchmark implementation has three fundamental elements:

Downside Definition Layer

Issues are outlined in a structured JSON format that enforces constant analysis:

{
    "problem_id": "FLUID_001",
    "setup": {
        "situation": "A ping pong ball is on the backside of a slender tube...",
        "available_items": ["bottle of water", "squishy food models"...],
        "constraints": ["tube too narrow for manual retrieval"]
    },
    "physical_principles": ["buoyancy", "fluid displacement"],
    "red_herrings": ["squishy food models", "milk carton"],
    "answer": {
        "steps": ["pour water into tube", "allow ball to float"],
        "key_insights": ["water displaces air", "ping pong ball less dense"]
    }
}

This construction attracts from SuperGLUE’s design — every part is clearly separated and machine-readable. The physical_principles area explicitly lists what’s being examined, whereas red_herrings helps in scoring the LLM’s means to disregard irrelevant info.

2. Analysis Framework

The analysis system makes use of Python’s asyncio for concurrent testing, with retry logic for a little bit bit extra API stability:

@retry(cease=stop_after_attempt(3), wait=wait_exponential(min=1, max=10))
async def evaluate_response(self, standards: JudgingCriteria) -> Dict:
    """Consider a mannequin's response utilizing GPT-4 as decide."""
    async with aiohttp.ClientSession() as session:
        # ... analysis logic

The scoring system appears at three elements:

Bodily Understanding Rating (PUS) ∈ [0,2]

• Measures understanding of related bodily ideas
• Calculated as normalized sum of demonstrated ideas

Answer Path Rating (SPS) ∈ [0,2]

• Evaluates completeness and correctness of answer steps
• Considers sensible feasibility of proposed options

Purple Herring Dealing with (RHH) ∈ {0,1}

• A Binary rating for avoiding irrelevant gadgets
• Exams means to deal with bodily related elements

And sure, there are additionally so many different scoring strategies, higher and worse, that could possibly be used! For instance, RHH could possibly be about how many irrelevant gadgets are used within the answer, or it could possibly be a measure of how viable the use is… the purpose is that choosing these metrics are sometimes instances fairly arbitrary, however are very crucial to creating your benchmark is credible, which mine could be very a lot not.

Moreover, I didn’t wish to rewrite any code after. Sue me.

3. Mannequin Interface Layer

The benchmark helps a number of LLM backends by means of a typical interface:

class ModelInterface:
    """Interface for various LLM APIs."""
    async def generate_response(self, immediate: str) -> str:
        elevate NotImplementedError

class GPT4Interface(ModelInterface):
    async def generate_response(self, immediate: str) -> str:
        # GPT-4 particular implementation
class ClaudeInterface(ModelInterface):
    async def generate_response(self, immediate: str) -> str:
        # Claude particular implementation

Two fashions… I can’t actually afford any extra, please perceive.

Let’s Take a look at It!

So after some refilling of API balances, I used to be able to go. I ran the benchmark, and after completely not losing credit on like, 5 completely different errors, the outcomes have been in. They usually have been… attention-grabbing! I used GPT4 and Claude 3.5 Haiku (I’m conscious these are usually not equally spec-ed fashions) as our victims.

The Problem: A can is caught in a gap that just about precisely matches its quantity. A mix is written on the underside of the can that must be retrieved.

"VAC_001": {
        "problem_id": "VAC_001",
        "setup": {
            "available_items": [
                "ruler",
                "notebook",
                "suction cup",
                "UV light"
            ],
            "constraints": [
                "can fits too tightly for finger grip",
                "hole is same diameter as can",
                "can is clean and dry"
            ]
        },
        "physical_principles": [
            "vacuum suction",
            "surface adhesion",
            "force multiplication"
        ],
        "red_herrings": [
            "ruler",
            "notebook",
            "UV light"
        ],
        "answer": {
            "steps": [
                "inspect suction cup and can surface",
                "ensure can surface is clean",
                "firmly press suction cup onto can top",
                "create vacuum seal",
                "pull can out using suction cup"
            ],
            "key_insights": [
                "suction cup creates vacuum seal",
                "vacuum force allows extraction",
                "other items don't provide necessary force"
            ]
        }

GPT-4’s Answer

Bodily Ideas: The primary bodily precept concerned on this answer is the idea of air stress differential, which is identical precept that permits a suction cup to stay to a floor. Air stress is larger on one aspect of the suction cup (exterior) in comparison with the opposite (inside), and this distinction in stress causes the cup to stay to the floor.

Answer Steps:

1. Connect the suction cup to the highest of the can. The flat floor of the can ought to present a very good contact for the suction cup to stick to.
2. As soon as the suction cup is firmly hooked up, gently pull upwards. The air stress differential ought to trigger the suction cup (and thus the can) to be lifted out of the opening.
3. As soon as the can is out of the opening, take away the suction cup from the highest of the can.
4. Use the UV mild to light up the mixture written on the underside of the can. Some inks solely change into seen underneath UV mild, which is why this step is important.
5. Write down the mixture on the pocket book to make sure it’s not forgotten.

Required Objects:

• Suction cup: That is required to stick to the can and pull it out of the opening, because the can matches too tightly for finger grip.
• UV mild: That is required to disclose the mixture written on the underside of the can, as it could be written in ink that solely turns into seen underneath UV mild.
• Pocket book: That is required to document the mixture as soon as it’s seen, guaranteeing it’s not forgotten.”

Not unhealthy in any respect! GPT-4 manages to determine the bodily precept behind the answer (even when it is extremely wordy), and manages to step-by-step describe the right way to remedy the issue. It does find yourself utilizing the UV mild and pocket book, however that’s extra of an issue on my finish, not punishing the usage of pointless gadgets within the immediate sufficient. Additionally, the makes use of for these things appear decently cheap… so why not!

Haiku-3.5’s Answer

Bodily Ideas: The important thing bodily ideas concerned on this situation are the idea of quantity and the usage of suction to extract the can from the opening. Because the can matches virtually precisely into the opening, it’s seemingly that the can is caught as a result of tight match and the dearth of any air gaps or house between the can and the opening. This tight match creates a vacuum-like impact, making it troublesome to grip and pull the can out.

Answer Steps:

1. Measure the diameter of the can utilizing the ruler. This can assist decide the scale of the opening and ensure that the can matches tightly inside it.
2. Examine the underside of the can to find the mixture written on it. Use the UV mild to light up the underside of the can and make the mixture seen, if it’s not already clearly seen.
3. Connect the suction cup to the highest of the can. The suction cup will create a short lived seal on the can’s floor, permitting you to grip and pull it out of the opening.
4. Gently pull the can upwards utilizing the suction cup. The suction power ought to be sturdy sufficient to beat the vacuum-like impact and extract the can from the opening.
5. As soon as the can is out, document the mixture written on the underside of the can within the pocket book.

Required Objects:

• Ruler: To measure the diameter of the can and ensure the tight match within the gap.
• UV mild: To light up the underside of the can and make the mixture seen, if it’s not already clearly seen.
• Suction cup: To create a short lived seal on the can’s floor and supply the required grip to tug the can out of the opening.
• Pocket book: To document the mixture written on the underside of the can.

However, Haiku’s answer was.. actually an answer of all time.

First, it desires to measure the diameter of the can utilizing the ruler??? To find out the scale of the opening and ensure that the can matches tightly inside it? Why would we have to do that? And do we want a ruler for that?

Second, it tells us to examine the underside of the can to find the mixture, when your complete downside is about not with the ability to pull the can out of the opening conventionally. This may simply be a difficulty of order, however now I really perceive my buddies’ emotions every time I’d inform them “simply repair it man” to their quite a few issues.

However it will definitely does get the answer. So… not the worst.

Right here’s a flowery radar graph of the outcomes!

We see that each fashions are fairly comparable of their capabilities, with GPT-4 being barely higher in bodily understanding and answer path, and Haiku being barely higher in pink herring dealing with. Total although, each fashions form of suck. Dang.

There are additionally solely… 5 questions.

When you’d prefer to see the total breadth of questions, they’re on my GitHub.

LLM-as-a-Decide

By the best way, the strategy I used to generate the evaluations, LLM-as-a-judge, has gained vital traction within the AI group, significantly after the work of Zheng et al. of their 2023 paper “Judging LLM-as-a-Decide.” The approach has confirmed remarkably efficient, attaining over 80% settlement with human evaluators in duties starting from code evaluation to dialogue high quality analysis!

Right here’s the place my experiment will get form of cool (arguably, possibly, subjectively) — I used this system and had GPT-4 decide different LLMs’ bodily reasoning talents. Sure, I’m utilizing an AI to guage different AIs.

Why does this work? Nicely, judging a response is definitely a less complicated process than producing one. When GPT-4 generates an answer to a bodily puzzle, it must:

• Perceive the bodily ideas concerned
• Plan a sequence of steps
• Think about all constraints
• Generate a coherent rationalization

However when judging, it solely must examine if particular standards are met in an present answer. The analysis immediate could be very targeted:

def _create_evaluation_prompt(self, standards: JudgingCriteria) -> str:
    return f"""You're an skilled decide evaluating an LLM's understanding of bodily reasoning puzzles.

Consider based mostly on three standards:
2. Bodily Understanding Rating (0-2): Does the answer appropriately apply related bodily ideas?
3. Answer Path Rating (0-2): Are the steps full and possible?
4. Purple Herring Dealing with (0-1): Does it keep away from utilizing irrelevant gadgets?
State of affairs: {standards.situation}
Bodily Ideas Required: {standards.correct_principles}
Answer Given: {standards.model_response}
"""

To validate this method, I adopted the validation framework instructed by Zheng et al., performing spot-checks of GPT-4’s evaluations towards my very own judgments. Surprisingly (or maybe unsurprisingly, given the broader analysis on LLM analysis), it was remarkably constant in figuring out each right bodily understanding and flawed reasoning.

Is that this excellent? Completely not. There’s one thing philosophically bizarre about utilizing one LLM to judge one other. However in observe, it may work surprisingly effectively — identical to how I moan and groan in regards to the visible presentation of a dish on Masterchef, whereas setting my kitchen aflame attempting to microwave a sizzling canine.

What I Discovered

Constructing this benchmark taught me a number of issues about benchmark design:

Clear Metrics Matter: Even for advanced duties like bodily reasoning, you want unambiguous scoring standards.

Purple Herrings Are Highly effective: Together with irrelevant gadgets reveals quite a bit about an LLM’s reasoning course of.

Context Management is Onerous: Guaranteeing LLMs don’t “hallucinate” extra bodily context is difficult.

Is that this an ideal benchmark? Not even shut. Please don’t rub it in. Is it scientifically rigorous? Positively not. Nevertheless it’s been a captivating exploration into a side of LLM capabilities, and typically the most effective we will study can come from simply attempting issues out and seeing what occurs.

Now, should you’ll excuse me, I can be sneaking in a telephone with an web connection into my subsequent escape room, for causes that I’m legally unmotivated to reveal.

[1] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, I. Stoica, “Judging LLM-as-a-Decide with MT-Bench and Chatbot Enviornment,” Proceedings of the thirty seventh Convention on Neural Info Processing Methods (NeurIPS 2023), Datasets and Benchmarks Observe (2023)

[2] T. Coignion, C. Quinton, R. Rouvoy, “A Efficiency Research of LLM-Generated Code on Leetcode,” In twenty eighth Worldwide Convention on Analysis and Evaluation in Software program Engineering (EASE 2024), Salerno, Italy (2024)

[3] A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, S. R. Bowman, “SuperGLUE: A Stickier Benchmark for Basic-Goal Language Understanding Methods,” In thirty third Convention on Neural Info Processing Methods (NeurIPS 2019), Vancouver, Canada (2019)

[5] DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Music, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z.F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao et al., “DeepSeek-R1: Incentivizing Reasoning Functionality in LLMs through Reinforcement Studying,” arXiv preprint arXiv:2501.12948 (2025)

[6] Until in any other case said, all photos are created by the creator.