I've heard of this problem before and am lucky enough that it hasn't happened to me-yet. The looser weave of the fabric contributes to it, as does Quilter's Dream by what I've been told. I use Warm & Natural or Warm & White as they won't beard. When I have mostly black fabrics I use Hobb's Heirloom® Premium 80/20 Black Cotton Blend. It's needle punched and resin bonded, so it has a slightly firm surface and won't beard. I've been pleased with it so far.
I guess you need to use a sweated fuzz shaver to take it off? Lint roller probably pulls more up and you don't really want to cut them all off one by one.
The quilt is lovely! Do we get to see a photo of more of the quilt? Pleezee...
Hate when little things like this go wrong.