This post assumes that you are armed and not so much a helpless victim waiting for death.
First, there can be no substitute for simple eye contact. Turn, make contact, and show no fear. then go about your business.
you've just put them on notice. You know they are there, you're not frightened, and you've given them fair warning that you aren't a sheep.
At that point, displaying a weapon isn't the right thing to do, but if you haven't turned them away, draw it, and hold it in concealment, and if you are confronted, let it be discretely visible. Brandishing is a vague legal term, but in general, before you're arrested, you'll have to have the thing "in action." Showing your hand on the butt of a pistol inside your jacket or wasteband isn't technically "brandishing."
This is my part. If you have been followed for a distance, or fully up to your vehicle or (private) destination, that act probably constitutes at least 50% of proof of intent to harm.
You've been followed in a predatory manner. You're confronted at close range. They were nonverbally warned off, and ignored it. If you see a weapon displayed of any sort, draw and show, and at that point, any, and I do mean any, aggressive moves justifies lethal force. Showing that they intend to attack an armed man justifies immediate defensive actions.
Of course, this will all be reviewed and held to the strictest standards of hindsight by the legal teams that will possibly be lined up with no intention in the world other than destroying your life because you behaved badly towards a couple of fellow model citizens.