The issue is that they are invisible. No one is sure where they are located without a marker.
My neighbors and I share an imaginary line between our homes. They mow and take care of the grass up to the location where they believe it exists. I do the same.
This arrangement works well for us because I have decent neighbors. They are not trying to take extra property for themselves, and they are not doing anything to hurt our relationship. We all get along and have no real issues.
This situation will only become a difficulty when one of them does something to harm my life and well-being. If they become destructive to my property or cause issues to my family or me, then I will have to reinforce the boundary and make sure they stay on their side.
Your heart and mind function in the same way as your property. There are numerous areas in which you have no problems, so no clearly defined boundary is needed. When something infringes on our good nature, then you must respond. This can be difficult to do as the dividing lines have been blurred for so long, we might not remember where they are located.
I guess that your biggest spiritual struggles are connected to places where the boundaries are unclear. Is one cuss word too many? Are two or three drinks too much? Is talking intimately to a member of the opposite sex okay if you are married? Is one sexual image too many to view? Is it a good idea to miss worship two or three times a month? The list goes on and on.
One way to improve your life is to develop clearly defined lines of behavior. Sometimes a surveyor and a secure fence will prevent issues from happening in the future. Invisible boundaries are fine until they are not.