Since I never actually “used” a mainframe OS, I suppose it would be IMDos (customized version of CP/M for the IMSA 8080, then actual CP/M) before those, I used a version of MS Basic that had no OS, it was the OS, same with the Wang 2200 which had microcode to “boot” directly in to Wang Basic. Then if I recall, Ultrix on the DEC PDP-11