Do you? If you do, then it is a good idea to stop now.
It is impossible to totally disregard what others have to say about you. After all there are certain people who have more influence in your life than most. Your parents, your spouse, your boss and your close friends. And their opinion about you counts a great deal.
But most people's opinion about you should not.
You cannot leave the life you want if you are going to be too worried about what others think about you. Let them say things about you, don't worry too much about it. in the end it doesn't really matter.
Dr Seuss once said, “Be who you are and say what you feel because those who mind don't matter and those who matter don't mind.” It is a good rule to live by.