The usual advice is to go down one size. Using a knit might make the garment droopier, hence the need to size down and use the stretch, at least in the bust.
Given how loose a fit this top has and how wide the neckline is, I don't think you'll have problems getting in and out of it.