My .54 TC Hawken likes 80 grains of 2fg Goex with an .018 pillow ticking patch lubed with Bore Butter 1000+... From the bench I get on average 2" groups from 50 yards.
What you're going to need to do is work up your load. I would suggest going with no less then an .018 lubed patch and start out shooting 3 shot groups off the bench from 25 yards, not 50 yards.
Start again with 60 grains - shoot three shots for group - then up your charge 5 grains and do it again - then repeat this process of upping the charge five grains with every three shot group until you find the tightest group. You'll know because from the tightest three shot group your grouping will start to open up once again.
I would guess you're going to find the best group in the 75 to 90 grain range.
Here is the key however,,, do every thing the same. If you swab between shots continue to do so - doing everything the same. The only thing to change is the powder charge every three shots.
You should see some marked improvements. When this happens it is time to move out to 50 yards and see how you do.
Remember, you are shooting for group while working up your load so where the group appears on the target at this point does not matter. Just get a tight 3 shot group, then adjust your sights at 25 yards to hit center. You should be alright at 50 yards then, with perhaps very little adjustment needed.
Good luck!