Can you find any matching ribbing for the cuffs, etc. those bits really need to have good recovery as well as stretch, so they don’t go baggy.
Even though I live in joggers, I don’t have this particular pattern, as it’s too low waisted for me. I’ve made other joggers with only an inch or two of ease in less stretchy sweatshirting with no fit issues, though the knees do get baggy by the end of the first day of wear. The Hudson pants for men look to have a good 4” of ease so should be ok.