My Lacey does the same thing - first I thought it was the fabric - I used a very soft cotton batik - hadn't realised its supposed to be for knit!. Then I thought it was too big, so I took 10cms out and put a back zipper in, that seems to have fixed it, for what it's worth. Won't make it again.
Edited to add: Just been googling this problem and asking on another forum and the consensus is that either 1) a forward shoulder adjustment is needed, so that would need a bit more length on the back armhole scye and a bit out of the front one. 2) not the correct high bust size.
Personally I don't have a forward shoulder problem but it was super big on me.