I think there is truth to this saying.
Of course, how one defines "pain" is the determining factor. In terms of exercise, I think the "pain" is a good thing. It's the burn, struggle, high, and/or sweat equity of the workout. It's what you come for.
If it doesn't hurt a little, what's the point? You suffer a little (or even a lot) and feel awesome afterward. And it doesn't matter too much what you do, swim, bike, run, yoga, weight lifting, aerobics...