Yes, you choose by your actual chest size. This gives you 3" of ease according to the chart. Have you got a blazer that fits you well? If so measure the chest on that and see how much ease it has.
I've never made a Simplicity Men's pattern, but the women's have always had far too much ease for me, and this has been constant for decades! It would be foolish to make something as time consuming as a jacket without a toile. I always do so with a coat or jacket, doing without will lead to something very ill fitting. Sorry if that's not what you wanted to hear!