I am getting the impression that you never liked Charlie from the get go, and probably started everything by shooing him because you didn't want him around you---- this has caused things to escalate to the point where he is on his guard the moment you enter his "territory". I would take him off...