The crucial difference between knit fabric and woven fabric is usually the amount of stretch. You do get some wovens with a bit of stretch and some knits are very stable and hardly stretch at all.
Patterns designed for knits (stretch) usually have some sort of indication as to how much the fabric needs to stretch - ie 4 inches must stretch to 5 inches, or 10 cm must stretch to 15cm, or whatever. If the fabric you use stretches more then the required amount it's usually OK, but if it stretches less it either won't go around you or it won't allow you to move properly!
If your pattern has grown-on sleeves (with no arm scye seam) it is probably relying on the stretch of the fabric to allow you to move your arms forward and when your arms come back the fabric will relax and stay close to the body. If you cut bigger to allow for that movement it will be OK when your arms are forwards, but when down by your sides there will be excess fabric across your back.