Ok, so you got me thinking 😄
I managed to get this to work by swapping the order and making Cookie the default scheme. I thought I had tried that before but maybe didn't have the AddAuthorization.
With CookieAuth the default scheme I now get authed for both.
Order matters!