I don't worry as much about brass, though I never leave a shotgun shell behind. It's mainly ideological rather than purely logical - neither one is going to hurt your land, but somehow the brass seems more natural than the polymer. Also, tiny .22 casings fired through a semiautomatic weapon tend to disappear instantly rather than over time - I can never find more than 10% of the damn things if I'm shooting outside and trying to clean up. The natural turnover of the land will bury them. While they'll survive your lifetime, they'll be gone in the blink of an eye on the scale of cosmic time.