Depending on the range of things you may like to do, there are plenty of places to visit or things to do to fill a 3 or 4 day trip away exploring the area around Burnley and East Lancs in general as well as visiting the different fabric outlets, including venturing over into Yorkshire if there's time, energy and any fabric money left.
Towneley Hall Art Gallery & Museum and Towneley Park
Gawthorpe Hall (including a textiles collection)
The Singing Ringing Tree (pick a fair weather day)
The Wayside Arts Trail
Canal Cruises
Ightenhill Park, Scott Park, Queens Park
Pendle Hill and Pendle Heritage Centre
Haworth Art Gallery (+Tiffany Art glass)
Weavers Triangle Visitors Centre, (Burnley Wharf)
Mining Museum
Wycoller Hall
And then it's only a short trip across to Fabworks at Dewsbury and that's a whole other trip.