The paper discusses how prompting-based experiments with large language models (LLMs) can be highly sensitive to small, seemingly insignificant details in the prompt setup, leading to varied behaviors even with similar experimental designs. The study highlights the challenges of interpreting results from such experiments and emphasizes the need for caution when conducting prompting-related research.
The chart will appear after repeat observations. The current metric comes from the source.
5 source pointsReal observations only. History before source connection is not reconstructed.
This study warns researchers about the potential for small, unexpected details in prompt design to significantly affect results, emphasizing the need for careful consideration in experimental setup.
This paper evaluates the ability of the MiniMax-H3 multimodal generative model to reason about the physical world. The study introduces a new framework for assessing physical reasoning through four dimensions, using tasks that require integrating information across multiple modalities. The model achieves an overall success rate of 41.97% across 517 evaluation instances, with video-based reasoning performing best and audio-based reasoning least effectively.
5 days agoDaily PapersLimiX-2 is a new model in the LimiX family that uses the Contextual Mechanism Networks (CMNs) paradigm, pretrained with Context-Conditional Masked Modeling (CCMM). It focuses on learning joint representations of data generation processes rather than target-centric prediction, showing superior performance on tabular data benchmarks and promoting causal awareness.
6 days agoDaily PapersIntBMoE introduces a block-conditioned Mixture-of-Experts (MoE) framework that decouples participation, execution, and materialization by combining dense expert composition with sparse block execution. It uses a learned codebook for blocks and a hypernetwork to merge experts, achieving full participation while keeping execution and memory costs low. Experiments show improvements in image classification, language modeling, and sequential recommendation, with real-world deployment in AMap's recommendation system.
3 days agoDaily PapersThis paper introduces Fuse, a multi-agent simulation framework for evaluating social reasoning in LLM assistants. The framework uses a target agent with a hidden motive interacting with other agents, including a user agent, to create verifiable ground truth for social reasoning tasks. The study applies Fuse to 12 LLMs and identifies key factors affecting social reasoning performance.
6 days agoDaily PapersThe paper introduces SELF-INDEX, a framework that enables an index to self-evolve without human intervention by autonomously diagnosing retrieval issues, revising index keys, and validating changes. It also uses a Query Simulator to proactively explore new retrieval demands, improving performance across various corpora and downstream applications.
4 days agoDaily PapersThe paper introduces XConf, a method for estimating confidence in language models by leveraging the model's accumulated experience. XConf uses past episodes, including tasks, reflections, confidence levels, outcomes, and lessons learned, to inform current confidence estimates through a recall and reflect process. It outperforms existing methods in discrimination and calibration with lower computational cost.
6 days ago