I think we need to differentiate two very different positions here:
Claim 1) Observational studies can stand in for RCTs
Claim 2) Observational studies can have meaningful epistemic causal import even without RCTs.
I do not, nor have I ever believed Claim 1. I also am not aware of anyone seriously claiming this (this would imply if someone had two equally well done studies, one RCT and one observational, come to opposite conclusions, they would maintain agnosticism rather than lean towards the conclusion of the RCT.
But I have always believed claim 2, and this is where I think the discussion should be had. I say this because there are people who think observational studies are completely meaningless and should simply be dismissed of all epistemic import as it pertains to causal epistemic import. I think this view is misguided and, quite silly. But it's a very rampant view.
But notice that the view that observational studies carry meaningful epistemic causal import is NOT the same view that they will ever "stand in" for RCTs. RCTs will still carry more epistemic import than observational studies, all else equal.
If this weren't true, we would expect observational data to be concordant with RCTs somewhere around 50% of the time as any other figure would give us epistemic import. If its say 75%, then we now have more reason to believe an RCT would come to a similar conclusion (not a "stand in", just a reason to nudge the credence of what the results would be). If it was 25%, well then that also carries epistemic import that the RCT would more likely than not find the opposite result (or not find the same result).
Now, in terms of this specific critique, it may very well be that such a paper did not show what they set out to show with RRR. I agree this only means there is no systematic bias in either direction, but this is of course a fundamentally different question as to whether we should expect the same epistemic import from an observational study vs an RCT. It does not follow from RRR=1 that they will just have the same (or similar) results.
But there's still a wealth of background literature that does suggest a well-done observational study having a given conclusion really *does* mean a RCT answering the same question will be more likely than random to arrive at a similar conclusion.
For example, even some of the most unfavorable concordance studies between observational studies and RCTs still show qualitative agreement (if both RCTs and NRSIs found the same direction of effect, that is, a statistically significant increase, a statistically significant decrease, or no statistically significant difference) around 2/3rds of the time, which is roughly the probability of qualitative agreement I myself have seen the last time I manually went through each type of study on an individual level and just counted rates of agreement.
So in term of my epistemology: All else equal, RCTs will trump observational studies. If there are RCTs, we go with the conclusions of the RCTs unless there is a very good reason to call them into question such that we need to turn to observational data.
However, if there are no RCTs, and we only have observational data. We do what we can with observational data, with the understanding that even if it is well done, there is roughly a 1/3rd change that when the RCTs do come along, they may point to a different conclusion.
Not a "stand in", but not epistemically useless in all forms of causal inference.
link.springer.com/article/10…
jamanetwork.com/journals/jam…