I think I've worked it out
Looking at it with the zip unfastened, it wants to swing over the centre line. The front edge is cut on the straight grain, but because of the waist/underbust seam being at an angle, it doesn't work, ie the angle of the front edge to the waist seam is wrong. It should work, the pattern pieces look like they should work, but in reality it doesn't.
I have used this pattern twice before, and now realise both other jackets have the same issue even though they are buttoned, see pic of tartan one below.