You will find, in the Gita, Supreme Lord Sree Krishna first told Arjuna, “Perform your work according to your own competency. Do not attempt to do the work of others who are of a different nature.”
But ultimately, He says, “Sarva-dharman parityajya, mam ekam saranam vraja” (18.66). “Give up all the relative duties I have spoken about earlier and take absolute shelter of Me. You are my own person. You are related to Me. You are part of My own potency. I have hitherto advised you to act according to your own capacity, but you will obtain your actual, eternal welfare when you take absolute shelter of Me.”
When we take absolute shelter of the Supreme Lord, we will not be committing sin by giving up those relative duties. But, take careful note of the words, “mat-krte”: for Me. “Performing household duties or brahmacari duties have become too difficult, so I have given them up.” Thinking like this, you are committing sin. You are violating the vedic injunctions. But if you take shelter of Supreme Lord Sri Krishna, then automatically you are performing service for all.
Generally, we refer to those who have given up their household life as sadhus. But if one thinks “I am married. I have a wife and children. I have no funds to maintain them. So, I have given up household life. I am moving along the road, begging. I have become a sadhu.” then such a person is not a sadhu. He is committing a sin. But if , then all the family members are benefited. “Mat-krte tyakta-karmanas tyakta-svajana-bandhavah”: “For Me.”
— from the lectures of His Divine Grace Srila Bhakti Ballabh Tirtha Goswami Maharaj.