The question is whether you really want true randomness, or a more pleasing pattern that looks random, but is actually quite controlled.
How about making piles of squares in the different fabrics (ie each pile has the same combo) and taking one from each, not in order, but making sure you use the piles at the same rate.