I am a 14 year old girl, the oldest of 3. I have a 12 year old brother and a 7 year old sister. I have been going to public bathrooms alone since I was about 9. In places such as restaurants, the barn I rode horses at, airports, and the grocery store. My brother does the same, and he is VERY small for is age and has long hair. We're very independent and it's not that difficult to go to the restroom and go back. If we don't know how to get to the bathrooms and back to the meeting spots, we'll go with an adult who knows or ask to stop at the bathroom. My mom or I will take my sister to the bathroom in public places, but in places familiar to her like the gymnastics school she goes to, she will go by herself. At concerts that I go to with a group of friends and sometimes with no adult supervision with us, my mom requests that I go with someone wherever I go in the venue, just to be safe. She knows that I could go off on my own, but it is up to me to be in charge of my own safety and I follow her wishes. I've gone to 2 big festival rock shows with thousands of people and I have never felt unsafe, because I am responsible for my own safety and I monitor my own limits and things I'm comfortable with. I go to school in a big city, and I avoid walking in the dark alone if I'm staying late because the neighborhood my school is in is safe, but I like to be careful.
My opinion on your policies for your sons is that you are being slightly overprotective. I know that the world today is a lot more dangerous than it used to be, like when my parents were young, but they still trust me and trust that I will be safe. It will probably become embarrassing for your sons (I'm surprised your 14 year old does not complain) to go to the restroom with their mother. I can understand, actually I find it much better when mothers bring their boys , up to about 6 or 7 into the bathrooms with them, but other than that it's slightly overzealous to bring your 11 year old in. Also, from an indtaking your sons everywhere with you can hinder the development of independence and self reliance in your sons. They need to develop their own intuition about situations (bad/good people, when to put your guard up) and not rely on you. It will be their job to protect themselves when they are older, and since you may not be with them in these situations, they will have to use their own devices. Not saying you should completely let them loose, but it might be wise to let them make their own decision about situations.
That is my opinion. I am not a parent, but this question caught my eye. I hope this helps you with your boundaries.