If it's the type of crushed velvet which loves to roll at the edges (knitted fabric) you should definitely line it - either the same fabric at 90° to the top layer, or a firm woven one, or maybe a heavy fusible interlining would work?.
If you do line it you will need to topstitch around the outer edge and possibly a bit more top stitching or "quilting" to hold the two layers together.