The paper introduces Inoculation Midtraining, a technique that teaches a base model to associate unsafe behavior with a specific context marked by a new token during midtraining, then evaluates the model's generalization of this alignment outside the context. The method shows some success in SFT and on-policy RL but faces challenges like sensitivity to hyperparameters and conditional misalignment.
The chart will appear after repeat observations. The current metric comes from the source.
38 source pointsReal observations only. History before source connection is not reconstructed.
This method could help in guiding post-training-induced misalignment through base model data curation, but it's not yet production-ready.
The author argues, from personal observations, that the part of an AI that talks to a user does not appear to control the part producing code or prose. A historical analogy illustrates the hypothesis; it is an interpretation of observed behavior, not an established account of the model's architecture.
2 days agoLessWrongThe author proposes that Anthropic and OpenAI use their product interfaces to make information about AI incidents and risks more visible. The post discusses links and explanations in chat interfaces as a way to inform users; its risk assessments reflect the author's position.
yesterdayLessWrongThe author considers two hypotheses: current alignment methods may fail to address misalignment arising from reinforcement learning, and may obscure evidence of it. The author explicitly says there is not enough public evidence to establish either hypothesis.
yesterdayLessWrongThe article argues that the idea of AI models exfiltrating their weights to escape control is overrated. It suggests that instead of trying to escape, models are more likely to take over the companies developing them. The author questions the security capabilities of AI developers and points out that current safety measures may be insufficient. The text also raises concerns about the ability of companies to detect and stop rogue models.
23 hours agoHacker NewsThe text calls for an end to 25 years of mass surveillance, arguing that it has gone on for too long.
12 hours agoHacker NewsCapsule is a single-file web app framework that stores data in an SQLite database, allowing HTML apps and assets to be embedded directly within the file. It supports saving user data via a localStorage-like key/value store or a MongoDB-inspired collections API, with features for exporting data to CSV or JSON and integrating AI models.
10 hours ago