May I make a suggestion here, I gather this machine is new to you, so is it brand new i.e. bought from a dealer? If so then why not go back to the dealer and ask for a demonstration, and if you do go back to the dealer take your machine with you, and then get them to demonstrate how to do a buttonhole on your machine, then you have a go whilst there. There is always the chance that something is not quite right with the machine.
I d hope you get this sorted and soon.