Soft fabric will often show the first signs of wear on seams that get the most stretch and movement. For your next pair, either put a strip of fusible interfacing along these seams, or sew the seam with cotton tape laid on it so that this can help take the strain.
To save your existing ones, press the back seam flat, apply some strips of fusible or cotton tape (as above) then sew a new seam a few millimetres in from the old one. You can do the same with the darts. This will give them a slightly longer life.