Bhai, even the best of brothers eat together, often work together, but at the end of the day, each goes to his own house. BTW, there is nothing religious about yoga, it is a form of exercise. And a good one at that. Do it in your own home, a yoga class, but certainly not in a place of worship of any religion.