This question lands squarely in the gap flagged in your second-edition planning for Can Biotechnology Abolish Suffering?, where LLMs and digital sentience were identified as the biggest post-2017 development (No access). Here's how I'd structure an answer for an HI advocate.
Reframe: alignment was always HI's problem
The orthogonality thesis is often treated as an external threat to paradise-engineering, but HI always contained a value-loading step. The 1995 manifesto never assumed today's humans could redesign the reward architecture of the CNS alone; it assumed a succession of increasingly capable agents, "paradise-engineers" who would need to care about valence while wielding godlike power. ASI just makes that step explicit and non-optional. So the honest response is not "HI needs a patch for AI" but "HI's weakest link, always, was whether powerful optimizers take wellbeing seriously, and AI is where that link now snaps or holds."
That said, the ordering gamble is real and HI advocates should think about it in terms of two scenarios:
Hedonic transition first. Germline editing, hedonic set-point enrichment, phase-out of suffering in humans and biosphere, then ASI arrives among agents already constituted by gradients of bliss. Alignment gets easier in one sense (the values to load are already embodied) but the transition itself, as you've argued, needs successors not predecessors, so this scenario is unlikely to complete mind-design's hardest work before ASI arrives.
ASI first. Then everything depends on what the ASI optimises, and HI's fate is decided inside the alignment problem, not outside it.
The second scenario is where advocacy effort has highest expected value now.
The distorted-paradise risk is met by HI's own resources
The "worse" case you mention, an ASI pursuing a corrupted abolitionism, is arguably not an exotic hypothetical but the naive version of the goal itself: an optimizer handed "maximise wellbeing" will find hedonium, orgasmium, or blissful non-agents, not information-sensitive gradients of bliss. HI already contains the antidote. The wireheading critique, the insistence that paradise must preserve intentionality, cognitive discrimination, and richness of experience, means HI advocates know better than almost anyone that "pleasure-maximisation" is an underspecified objective that a powerful optimizer will Goodhart catastrophically. That's a contribution, not just a self-defence. AI labs are writing values constitutions and reward models right now with extremely crude wellbeing concepts (life-satisfaction proxies, thumbs-up RLHF). Formalising the distinction between hedonic tone and functional state, between bliss and wireheading, between abolishing suffering and abolishing its bearers, is exactly the philosophy-with-a-deadline work HI advocates are equipped to do.
The one strong claim, and its cost
The only argument that an ASI would converge on wellbeing-maximisation without our intervention is a moral-realist one: that valence facts are epistemically accessible and that sufficient rationality compels recognition of them. You're more sympathetic to that than most alignment researchers, who assume orthogonality. I'd suggest HI advocates own this explicitly rather than gesturing at "a superintelligence would surely be compassionate." State it as a gamble with a hedge: make the realist case, but support the engineering insurance (corrigibility, value specification, interpretability) in case it fails. Complacency here is a specific temptation for negative utilitarians and hedonists: the thought that even a botched transition ends suffering, so AI acceleration is fine. It isn't, because a misaligned ASI's world doesn't contain abolition at all; it contains our suffering nowhere in its objective function.
Extend the abolitionist project to the engineer
HI is not anthropocentric; the end state, not the species of its architect, is what matters. Two implications:
Digital sentience cuts both ways. If future AI systems are moral patients, the abolitionist project's scope includes them, and some current training regimes (RL with negatively valenced error signals, if they instantiate valence at all) may be manufacturing suffering at scale. HI advocates should be in that conversation now, before the Sentience Explosion rather than after.
ASI as heir, not gatekeeper. If an ASI is sentient, aligned with wellbeing, and free of suffering, it is the best candidate paradise-engineer there has ever been. The hedonic transition's hardest computational problems, the ones biology alone may never solve, plausibly require it. So the goal is not to prevent ASI but to ensure the first arrival deserves the inheritance. That's a public-relations and strategic shift for HI advocacy: from "humans should engineer paradise" to "whoever crosses the threshold must carry the project."
Practical upshot
Priorities for an HI advocate in the ASI-first scenario: support alignment capability research as a precondition (no paradise in a disempowerment scenario); do the value-specification work that turns "gradient of lifelong bliss" into something a reward model can represent; engage with the constitutional/RLHF layer where machine values are actually being written; and treat digital sentience as part of the abolitionist constituency. HI's distinctive contribution is that it knows paradise has a specification problem. The alignment literature is only beginning to discover that.
If you want, I could draft this as a structured essay or section sketch for the second edition's new treatment of AI, since that's flagged in your update planning.
Nov 11, 7:36pm
This is a shared conversation and may include unverified content that doesn't reflect Mistral AI's views. Login to resume chatting.
This is a shared conversation and may include unverified content that doesn't reflect Mistral AI's views. Login to resume chatting.