I use the edge of a house key (more likely bronze rather than brass) as a drift punch.
Just don't use a key that you actually have to use.
If none of these things work....time for a Estwing hammer.
Just kidding!
A brass punch and a big hammer may be the last resort. One on a used 1860 I bought was stuck so bad, I had to work on it for a couple of days before it let go. It had buckled the barrel opening some because it was hammered in so hard.
May have been the factory or previous owner, can't tell.
Another wedge can always be bought, although cleaning up the end with some files and cold blue is a much less expensive option and you'll never know it had a punch on it. Brass punches certainly can dent these soft wedges if you have to hit them hard enough.