I hate all the discussions around NVMe vs S3, because they miss all the nuances.
The extremely obvious thing: the read or write latency from NVMe is going to be way smaller than S3 (or S3 Express).
But... most discussions talk about database running in a single machine attached to NVMe. That comparison isn't fair, and here's why:
When you write to S3, it is replicated to multiple machines. But your data is poof if your NVMe is gone.
S3 Standard writes to multiple AZs. S3 Express One Zone writes to multiple machines in a single AZ. Only after the data is secured and durable is it acknowledged as successful. You can't get the same durability guarantees with NVMe attached to a single machine.
If you implement your own quorum (within an AZ) to do the same, it will be around 2–3ms. I implemented a disaggregated storage engine for libSQL using FoundationDB. IIRC, the latency numbers were around 5–6ms with a 3 node FDB cluster in the same AZ. This was an out of the box FDB setup, without any config changes or tuning. I am really handwaving the numbers here, but the point is, when you write to a quorum the latency gonna be way higher than NVMe.
The truth is somewhere in between, this is where DB vendors are fooling you: NVMe isn't as fast as they claim, but still faster than S3.
So yeah, if you are comparing NVMe vs S3, include durability considerations. I'd love to see such benchmarks.