I think there is truth to this saying. But it is how one defines "pain" that is the determining factor. In terms of exercise, I think the "pain" is a good thing. It's the burn, struggle, high, and/or sweat equity of the workout. It's what you come for. If it doesn't hurt a little, what's the point? You suffer a little (or even a lot) and feel awesome afterward. And it doesn't matter too much what you do, swim, bike, run, yoga, weight lifting, HIIT, CrossFit, skiiing, whatever!
Hot yoga is my workout of choice and illustrates my point.