Consider this largely a follow-up to Friday's post about a statement aimed at creating common knowledge around it being unwise to build superintelligence any time soon.

Mainly, there was a great question asked, so I gave a few hour shot at writing out my answer. I then close with a few other follow-ups on issues related to the statement.

A Great Question To Disentangle

There are some confusing wires potentially crossed here but the intent is great.

Scott Alexander: I think removing a 10% chance of humanity going permanently extinct is worth another 25-50 years of having to deal with the normal human problems the normal way.

Sriram Krishnan: Scott what are verifiable empirical things ( model capabilities / incidents / etc ) that would make you shift that probability up or down over next 18 months?

I went through three steps interpreting [...]

---

Outline:

(00:30 ) A Great Question To Disentangle

(02:20 ) Scott Alexander Gives a Fast Answer

(04:53 ) Question 1: What events would most shift your p(doom | ASI) in the next 18 months?

(13:01 ) Question 1a: What would get this risk down to acceptable levels?

(14:48 ) Question 2: What would shift the amount that stopping us from creating superintelligence for a potentially extended period would reduce p(doom)?

(17:01 ) Question 3: What would shift your timelines to ASI (or to sufficiently advanced AI, or 'to crazy')?

(20:01 ) Bonus Question 1: Why Do We Keep Having To Point Out That Building Superintelligence At The First Possible Moment Is Not A Good Idea?

(22:44 ) Bonus Question 2: What Would a Treaty On Prevention of Artificial Superintelligence Look Like?

---

First published:

October 27th, 2025

Source:

https://www.lesswrong.com/posts/mXtYM3yTzdsnFq3MA/asking-some-of-the-right-questions

---

Narrated by TYPE III AUDIO.

---