Not sure what you mean by benchmaxxing but we think there's still a lot of usefu... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

ofirpress 56 days ago | parent | context | favorite | on: A Research Preview of Codex

Not sure what you mean by benchmaxxing but we think there's still a lot of useful signals you can infer from SWE-bench-style benchmarking.

We also have SWE-bench Multimodal which adds a twist I haven't seen elsewhere: https://www.swebench.com/multimodal.html

Snuggly73 56 days ago [–]

I mean that there is the possibility that swe bench is being specifically targeted for training and the results may not reflect real world performance.

Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact