Oct 25, 9:02pm

This worry is well placed, and the good news is that it has a principled answer: the infohazard argument bites hard only on a narrow slice of s-risk discussion, and you can get nearly all of the mitigation value from the parts it doesn't touch.

Which s-risk talk is actually hazardous

The standard taxonomy — incidental, agential, and natural s-risks ( CLR's beginner's guide) — sorts by how much discussing the risk itself adds risk:

  • Incidental s-risks (indifferent actors creating suffering digital minds at scale, "suffering subroutines") are almost certainly defense-dominant to discuss. Creating suffering requires no special knowledge — it's the evolutionary and engineering default. What's scarce is awareness and willingness to prevent it. Here, silence is the hazard, not speech.
  • Agential s-risks are the dangerous topic — a Futures paper argues precisely that agential risks are "unavoidable but dangerous" to discuss ( ScienceDirect). But even here, most of what needs saying is motivational and structural, not operational. Any sufficiently capable actor can derive the general possibilities themselves; your writing doesn't add capability. Bostrom's framework distinguishes adversarial hazards (information a bad actor can use) from attention and idea hazards (information that changes motivation or salience) ( Wikipedia).
  • The narrow genuinely-hazardous slice is concrete operational detail — step-by-step routes to creating or locking in extreme suffering. That is real dual-use information, and there is essentially no mitigation value in publishing it: prevention is generic (welfare-aware development, cooperation, sane values), so detailed blueprints add little defensive benefit while carrying non-trivial adversarial and attentional cost.

A RAND working group on biotech infohazards found the same perverse structure: restricting information can signal its existence, and the discussions about whether something is hazardous can themselves be hazardous — which is a reason for everyone, including both of us, to stay at the strategic level ( RAND).

A practical test before you write or speak about a "nasty option"

Three questions:

  1. Capability: would this materially increase a bad actor's ability, or just their awareness of what they already could do?
  2. Attention/motivation: could it increase the number or intensity of actors who'd want to — vivid, glamorizing, detail-rich depictions are the risk here (media guidelines on suicide reporting are the established analogue: cover the phenomenon, never the method).
  3. Audience and counterfactual: does telling this audience enable prevention, or would the information only circulate? A paragraph in a governance consultation, or a conversation with an AI Safety Institute or lab welfare team, has a completely different risk profile from a viral essay.

If something fails all three, don't publish it — use small trusted circles instead. If it passes 1–2 but you're unsure, default to abstraction: say that a class of attack exists and what generic safeguards blunt it, never how.

What to actually do (mostly hazard-free, high-leverage)

  • Support the specialist org. The Center on Long-Term Risk is the main institution working on exactly this — their current threat model centers on deliberate creation of suffering in conflict between advanced AI agents, with a Safe Pareto Improvements agenda (making AI conflicts safer without full cooperation) and a Model Personas agenda that studies and steers against spitefulness, sadism, and punitiveness in LLMs ( CLR research). They deliberately route interventions through labs and AI Safety Institutes rather than public campaigning ( CLR 2025 plans), and run a fund, a Fundamentals course, and seminars — a good way to have the sharp conversations in a vetted setting rather than in print.
  • Push digital sentience into the mainstream, because the gap is enormous. The lab-level field has moved fast — Anthropic's model welfare program, precautionary interventions, and the spread of dedicated welfare researchers to other labs ( Anthropic,  report) — yet digital minds remain "almost entirely absent from major governance frameworks" ( MINT Lab review). Public moral advocacy of your kind — the case for taking suffering in any substrate seriously, gradients of bliss as the alternative to the default — is precisely the non-operational, motivation-level work that fills this gap. It's also where your voice has no substitute.
  • Treat conflict reduction as s-risk reduction. Inter-AI bargaining failure is arguably the most probable agential pathway, and it's best addressed with cooperative-AI work that never needs to name a nasty scenario at all.
  • Note the convergence: for an abolitionist, s-risk mitigation is the critical path of the abolitionist project. The abolition of suffering via biotechnology and its successors only happens if the transition goes well; a single worst-case outcome forecloses everything. That framing lets you write about s-risks as part of your existing project rather than as a detour into hazardous territory.

In short: discuss why and what to do freely; discuss how one would do it never; and when in doubt, move the conversation from the public page into the small, trusted rooms where this problem is actually being worked on.

Oct 25, 9:02pm

This is a shared conversation and may include unverified content that doesn't reflect Mistral AI's views. Login to resume chatting.

Vibe can make mistakes. Check answers. Learn more