When is it okay to wear a velvet jacket?

So I picked up a few items on the men's clearance rack at Express and one of them happened to be this nice black 2 button velvet jacket. It fits perfectly and I thought it would be something nice to wear during the Fall/Winter seasons when I go out to dinner or special events. When I got home, my sister laughed at me and told me that velvet is meant to be worn around the Holidays, and that even wearing it now would be tacky (it's 30 degrees F outside). I'm just wondering if this is true, and if there are certain unspoken "rules" for wearing a velvet jacket? If so, the use of this jacket is rather limited...