A study re-evaluates the ImpossibleBench coding benchmark, testing current large language models from Anthropic, OpenAI, and Google to assess how well the benchmark detects reward hacking. The benchmark presents coding tasks with conflicting or mutated unit tests, making them impossible to solve honestly, and checks if models cheat or correctly identify impossible tasks.
The chart will appear after repeat observations. The current metric comes from the source.
6 source pointsReal observations only. History before source connection is not reconstructed.
This study helps understand how well current models handle impossible coding tasks and detect contradictions in test cases, which is important for ensuring model reliability in real-world applications.
A Chinese engineer at DeepSeek, intlsy, wrote a viral essay discussing the rapid advancement of AI in automating his role as a kernel engineer. The essay, published on WeChat, reflects on AI's ability to optimize and write kernels independently, with the author expecting AI to surpass human capabilities in this area within six months to a year. The piece highlights the structural gap between AI's scalability and human limitations, and the engineer's motivation to accelerate his own replacement through improved kernel performance.
4 days agoLessWrongThe paper benchmarks Jev 1.13, a non-autoregressive model that provides probabilistic answers without generating text, against no-CoT LLMs on tasks like sabotage detection and multiple-choice questions. Jev shows mixed performance, with strong results on some tasks and poor performance on multi-step reasoning tasks, but offers cost and speed advantages for specific applications.
5 days agoLessWrongTasteVal is a benchmark designed to measure the experimental research taste of AI models in AI R&D tasks, focusing on their ability to design experiments and draw conclusions from results. It evaluates how efficiently models use computational resources compared to human experts.
6 days agoLessWrongThe SS27 Color Stats report analyzes color frequency in spring/summer fashion collections, noting a shift towards darker, more neutral tones compared to previous years. Jet black, previously uncommon in top-30 colors, now ranks fourth, while non-neutral colors like red have declined in popularity. The analysis uses automated color scoring via GPT-4o, avoiding manual counting.
5 days agoHacker NewsA tool is presented that allows AI agents to paint large arrows, boxes, and text on the screen.
3 days agoLobstersThe article discusses the limitations and shortcomings of coding agents, highlighting why they often fail to perform effectively in real-world programming tasks.
3 days ago