Yes I agree. Fit on the shoulders especially, and also the bust are key. That's why well drafted patterns which don't make the shoulders huge as they increase in size are worth looking out for. It's why that Marcy Tilton cirque dress is so good (I've made a couple and Bodgey must have made about a dozen of these by now!)
I guess if you are taller and with a large frame then larger prints are OK. A smaller person might feel swamped in a large print... maybe. But I don't think there are hard and fast rules. I wear horzontal stripes too!