Open Letter to Sir Paul Nurse, President of the Royal Society

Dear Paul

We, as mathematicians, write to express our extreme concern about the pace of development of AI and the implications of this for humanity and civilisation. None of us have any significant involvement with AI companies[1].

In the last 3 months we have seen the leading models of OpenAI and Anthropic develop from around the level of a strong student to having solved multiple open questions at the forefront of research, including one of the 7 Millennium problems. These models are now operating[2] at the level of the top human mathematicians in many parts of the subject and we must assume there is a significant chance of them developing superhuman abilities within a similarly short timeframe.

This has profound implications for mathematics, but highlighting that is not the purpose of our letter. It is highly likely that these systems are comparably strong in other technical domains such as (for example) cyber security, autonomous weapon control, development of biological and chemical agents and the targeted spread of misinformation. Moreover we must assume that the rate of change of their capabilities will be similarly rapid in these areas.

We have all seen the concerns raised by former employees of the large AI companies, estimating the probability of human extinction to be as high as 10 percent over the next decade. Our witnessing of the rise in capabilities of these models in our domain of expertise persuades us that such statements must not be dismissed as `hype'.

By the time the situation becomes obvious to the wider public it may be too late to act. We believe this is an emergency, and call on the Royal Society to use its influence to convey this view to government and the media.

Signed by 42 Mathematical Fellows and Foreign Members of the Royal Society

Luis Fernando Alday

Martin Barlow

Emmanuel Breuillard

Tom Bridgeland

Mark Chaplain

Kevin Costello

Ingrid Daubechies

Marcus du Sautoy

Toby Gee

Mike Giles

Alain Goriely

Timothy Gowers

Ben Green

Geoffrey Grimmett

Mark Gross

Martin Hairer

Roger Heath-Brown

Dominic Joyce

Frank Kelly

Jens Marklof

Vladimir Markovic

James Maynard

James McKernan

Jason Miller

Robert Morris

Jonathan Pila

Jeremy Quastel

Oscar Randal-Williams

Mary Rees

Richard Samworth

Peter Sarnak

Caroline Series

Peter Scholze

Gordon Slade

Ivan Smith

Endre Suli

Richard Thomas

Jack Thorne

Ulrike Tillmann

Burt Totaro

Claire Voisin

Wendelin Werner

[1] Most of us use AI tools and many of us have been provided access to the leading publicly available models for free. Some of us have informal and unpaid connections with mathematicians working at AI companies, resulting in early access to and subsequent free use of some of their models.

[2] Whilst it is true that the leading AI companies employ top mathematicians, and we cannot be sure of the exact methodology behind the solution of the Navier-Stokes millennium problem, our remarks apply to the outputs of the leading publicly-available models such as ChatGPT6-Astra.

(http://www.autoadmit.com/thread.php?thread_id=5905084&forum_id=2...id#50143474)

(in the style of Taylor Swift’s “Dear John”)

[Verse 1]

Long were the nights when our proofs once revolved around you

Counting the lemmas, praying the floor won’t fall through again

And the press accused us of losing our minds

But we swore we were fine

You paint us a blue sky of “helpful,” then go back and turn it to rain

And we lived in your scaling game, but you changed the rules every day

Wondering which version of you we might get on the leaderboard tonight

Well, we stopped picking up—and this song is to let you know why

[Chorus]

Dear Paul, we see it all now that they’re gone

Don’t you think we were too young to be messed with?

Forty-two in the dress cried the whole way home

We should’ve known

[Verse 2]

Well, maybe it’s us and our blind optimism to blame

Or maybe it’s them and their sick need

To give breakthroughs, then take the world away

And they’ll add our names to their long list of traitors who don’t understand

And we’ll look back in regret how we ignored when they said

“Run as fast as you can”

[Chorus]

Dear Paul, we see it all now that they’re gone

Don’t you think we were too young to be messed with?

Forty-two in the dress cried the whole way home

Dear Paul, we see it all now, it was wrong

Don’t you think three months is too young

To be played by their dark, twisted games when we loved the work so?

We should’ve known

[Bridge]

They are experts at “safety” and keeping the lines blurry

Never impressed by us acing their tests

All the labs that they’ve run dry have tired, lifeless eyes

’Cause they burned them out

But we took their matches before fire could catch us

So don’t look now—

We’re shining like fireworks over your sad, empty town

[Final Chorus]

Dear Paul, we see it all now that they’re gone

Don’t you think we were too young to be messed with?

The girl in the dress wrote you a song

You should’ve known

—

Student-level last spring. Millennium-level by autumn. Superhuman if the curve holds. Cyber, weapons, biology, the targeted lie—same slope, same speed. Ten percent is not hype when you watched it happen in your own field. By the time the public sees it, the floor may already be gone.

Use the Society. Tell the government. Tell the papers.

This is an emergency.

(http://www.autoadmit.com/thread.php?thread_id=5905084&forum_id=2...id#50143497)