Back to stories
parsimony.lasu.dev

Show HN: Parsimony – an open-source benchmark for how much code AI models add

29points bylasu3h
Existing benchmarks measure solve rate and token output/cost, but don't punish excess code, which makes a codebase harder to work with in the long term. I wanted to measure this since it matters most for long-term projects.

Method: it takes the patches that benchmarks already publish (DeepSWE, SWE-bench Verified, SWE-bench Live) and counts "coding units": syntax elements like statements, calls, names and literals. I didn't use lines of code, so cramming code isn't rewarded. Deleted code is subtracted, so it measures net units.

GPT-6 and Claude 5.5 have no public patches yet, so I ran them myself on 3–10 tasks on my own subscriptions (shown as faded circles, not ranked).

Any feedback or model contributions are welcome, especially public patch exports for newer models.

GitHub: https://github.com/La5u/Parsimony

Not affiliated with any of the benchmarks.

Open articleView on Hacker News

Comments

2 total · 2 loaded

Ranked by HN
fastball2hon HN
On the one hand, I absolutely agree that we should try to measure and reduce verbosity that LLMs are slopping into our codebases.

On the other hand, the fact that Astra is "winning" this bench is not great signal for me, since imo Astra produces basically unreadable code. Astra plays code golf. My own revealed preference is that I'd rather have the excessive comments and tests of Claude rather than the inscrutability of Astra.

ndr31mon HN
I keep going in between the two and I'm currently in an Astra season, deeply disliking the tone and modes of Claude.

I wonder if the two will diverge enough that this becomes the tab-vs-spaces/vim-vs-emacs of the LLM era and they'll manage to keep users moving over at some point.