I've often had to choose between a) work more to stay 'on top of stuff' or b) maintain rest, gym, hobbies etc. The few times I've gone for option a), I've come to the edge of burning out completely and being no good to my colleagues and pupils. It's never selfish to rest.