x.com/jonasgeiping/status/2087094822146290170
2 corrections found
Ant already encrypts the model's reasoning
Anthropic does not generally encrypt Claudeโs reasoning. Its public docs say Claudeโs thought process is usually shown or summarized, and encryption is used only as a safety intervention for potentially harmful chain-of-thought content.
Full reasoning
Anthropic's public documentation contradicts the idea that it "already encrypts the model's reasoning" as a general practice.
- Anthropic's Claude 4 launch post says they introduced thinking summaries for lengthy thought processes, and that this summarization is needed only about 5% of the time because most thought processes are short enough to display in full.
- Anthropic's help center says that when thinking is enabled, users see an expandable Thinking section with Claude's thought-process summary. It adds that only occasionally the rest of the thought process is unavailable, specifically when safety systems identify elevated risk of harm or misuse.
- Anthropic's Claude 3.7 system card is more specific: it says they use encryption as the intervention mechanism only for chain-of-thought content that a streaming classifier flags as potentially harmful.
So the accurate picture from Anthropic's own materials is: Claude's reasoning is ordinarily shown in full or summarized, while encryption is a limited safety measure for flagged harmful content, not the default handling of model reasoning overall.
3 sources
- Introducing Claude 4 | Anthropic
We've introduced thinking summaries for Claude 4 models that use a smaller model to condense lengthy thought processes. This summarization is only needed about 5% of the timeโmost thought processes are short enough to display in full.
- Change the model, effort, and thinking settings | Claude Help Center
When extended thinking is enabled, you'll see an expandable 'Thinking' section... Click the 'Thinking' section to view Claude's thought process summary... Occasionally... the rest of Claude's thought process is not available. This happens when Claude's thinking involves information our safety systems have identified as potentially posing an elevated risk of harm or misuse.
- Claude 3.7 Sonnet System Card
We implemented and deployed a streaming completion classifier trained to detect and mitigate harmful content within chains of thought... For content identified by the streaming classifier as potentially harmful, we employ encryption as the intervention mechanism.
If anything this (marginally) increases entropy across different generations, so it will make model outputs slightly more varied.
Published evaluations of text-watermarking schemes generally do not show added variety. Prominent papers report that watermarking can reduce inter-response diversity, and some watermark methods even produce identical outputs for the same prompt.
Full reasoning
This statement gets the direction wrong for widely studied text-watermarking methods.
- In Google's/Nature's SynthID-Text paper, the authors explicitly state that both non-distortionary SynthID-Text and the Gumbel-sampling watermark baseline reduce inter-response diversity.
- The ACL 2024 GumbelSoft paper says the original GumbelMax watermark has a major diversity problem, always yielding identical outputs for the same prompt, and introduces new variants specifically to recover diversity.
So the literature does not support the blanket claim that watermarking "increases entropy" and makes outputs "slightly more varied." If anything, major text-watermarking papers describe diversity loss as a problem to be mitigated.
2 sources
- Scalable watermarking for identifying large language model outputs | Nature
We show that although both non-distortionary SynthID-Text and the Gumbel sampling baseline reduce inter-response diversity, SynthID-Text provides a better diversity/detectability trade-off than Gumbel sampling.
- GumbelSoft: Diversified Language Model Watermarking via the GumbelMax-trick | ACL Anthology
GM watermark encounters a major challenge with generation diversity, always yielding identical outputs for the same prompt, negatively impacting generation diversity and user experience. To overcome this limitation, we introduce ... variants that aim to enhance diversity.