We can craft black, white, and red, but the ONLY way to get a Purple Swashbuckler's Shirt is through a short quest chain. It is a chest piece and not a shirt, so you can't put anything over it, but you can put a shirt under it.
In Dustwallow Marsh, find Mordant Grimsby in the house at Witch Hill.
Pick up [What's Haunting Witch Hill?].
Kill Riven Husks in the area and they'll give you information when they die. Gather 10.
Turn in, pick up [Witch's Bane]. Gather 9 of the plants.
Turn in, pick up [Cleansing Witch Hill]. Torch the dock, fight the demon, return to Grimsby.
When he offers you a choice, get the Mordant's Travel Tunic. It's cloth, so anyone can wear it.
If you already did that quest and are kicking yourself, I'm really sorry. That sucks. I did the quest on at least one character, but I don't remember who. >_>