Lilacs, the situation with fraying fabric is horrible, because half the time, you don't realise until you wear the garment, that the fabric isn't stable enough.
I have had two instances, one a blouse, frayed a lot, but I did everything right as regards stabilising and finishing. First time of wearing, I came home and the whole seam had come apart! It was overlocked, and looked great, but the fabric just shredded. The second, a pair of cut off trousers, in a loose weave linen type fabric. Worn a couple of times, and not only the side seams, but the zipper seams were just pulling apart.
Wish I could help more, but sometimes the fabric is just not right for the job (it certainly wasn't in my case).
Sorry not to have more positive help, but hoping that your fabric is tamed by normal methods!
Jessie