I respect your opinion, even though I strongly disagree. You place a great deal of confidence in the security of your doors. I am not nearly as confident in mine. Not to mention windows that provide little real security. The difference between immediate access and "relatively" quick access is enough for me to choose to carry.
We might reach the same conclusion if I did not have a factor that is likely a non-factor for you. I have inquisitive and young children who like to roughhouse and I tend to be their favorite "target" While they are well behaved outside of the home the house is where they are pretty free to be children and run and play and roughhouse. I ditch my gun when inside because I figure it raises the risk of negligent discharge more than it would raises the chances of "winning" against a competent home invasion.