Our durability score has a blind spot: warping and nonstick failure

What the durability score is

Durability is 25% of our overall score. The scoring standards define it as “long-term performance signals from reviews written 1+ year after purchase”. It is a review-derived composite, not a lab measurement. It is designed to answer “do owners still speak well of this after the honeymoon?” — not “will this pan survive its first month?”

The blind spot

We split cookware-set products into thirds by durability score and compared the problem rates in the top and bottom thirds:

• Warping: 0.22% of reviews in the top third vs 0.15% in the bottom third — 1.5× (63,994 vs 85,819 reviews; adjusted p = 0.006).
• Nonstick failure: 0.29% vs 0.20% — 1.46× (same samples; adjusted p = 0.004).

The absolute differences are small — 0.07 and 0.09 percentage points — but the direction is consistent and statistically significant. The products our score likes are not the products with the fewest early surface failures.

Why this can happen

A long-term signal has a survivorship problem. The people who contribute to a “durable” score are the people who kept using the product long enough to write a year-plus review. The people whose coating failed in month three may have stopped reviewing, or reviewed once and moved on. A product can look durable among its survivors while still producing early failures.

There is also a labelling problem. A reviewer whose pancake sticks may write “sticking”, not “nonstick failure”; a pan that warped may be described as “uneven” or “rocking”. The keyword layer is good at counting what it is trained to count, not at inferring the physical cause.

What we are doing about it

• Publishing the contradiction. A score that hides its own blind spot is worse than a score with a caveat.
• Putting problem rates next to scores. The finding pages already show the rate and the sample; we are making that the first thing a reader sees, not a footnote.
• Reviewing the durability dimension. The current definition is long-term satisfaction, not early-failure resistance. Those are different questions and should not share one number without a label.

What would make the score better

The fix is not to delete the durability dimension; it is to separate two questions that currently share one number. Long-term satisfaction asks whether the people who kept the product still like it. Early-failure resistance asks whether the product survived its first year without a coating or warping complaint. A pan can be excellent on the first and mediocre on the second. We are testing a version that reports both, with the sample size and the failure window attached, and we will publish the change on the scoring-standards page before it goes live. Until then, read the score as a long-term signal and read the problem rates as the early-warning layer.

What this does not tell you

The top and bottom thirds are relative, not absolute quality tiers. The comparison is within cookware sets, not across all cookware. The absolute rates are low, and the review mix differs between the groups: price, usage and product age can all confound the comparison. This is a reason to read the score with the problem rates, not a claim that the score is useless.