Actually my wife does yoga which is being done by a group of Muslim sisters who take out all the Hindu stuff like the chanting and stuff and focus solely on the exercise part of it. I agree with your overall stance.
Pakistan Defence
Clutch
Chhatrapati
And I'm not against anyone doing it. Our Individual beliefs should not affect others. My fiance also do the pranayama (Breathing exercise) it helps her to stay focused. If yoga gives health benefits like cure or prevent some disease then why not.