Any round with an explosive filler would be a destructive device under the law. It would be the same as having a live grenade, under the law. Generally nobody bothers to prosecute for a handful of collector rounds kept as curios. Usually the most that happens, should the authorities become aware of them is that you surrender them to be disposed of. Particularly when you are unaware that they are a legally prohibited item.
I have seen old artillery shells at yard sales and in collections, still with their explosive projectiles. Scary dangerous, as old artillery shells can be very unstable.
Dummy rounds, training rounds (same solid projectiles) are perfectly legal, and safe. The people who reload and shoot 20mm "rifles" are doing it with solid projectiles, not the explosive ones used in combat.
US dummy ammo (20mm and up) often has a blue colored projectile. But not always. Practice missiles on our warplanes have blue warheads. White ones are "war shots"!
If there are any markings on the "bullet" of your 20mm ammo, you might be able to find out if they are live explosive or dummy. It's worth doing.
And, I think Dynamite is in a different legal class. It is an "explosive", but not a "destructive device". Different rules apply, I believe.