If you are a Christian woman, it could be that the guilt comes from not being able to forgive her. If that is the case, just because you choose to forgive her does not mean you have to keep her, or your brother, in your life. Forgive them, tell them the truth about why you want nothing to do with them & 'kick the dust from your shoes' when you leave.
I have a similar situation with my gram (who is almost 94) & one of my sisters. Nothing physical, but she ran up my Gram;s credit card to over $15,000 & stopped paying the bill. Unlike your brother, she is not familiar with jail. Just the opposite, in fact. She was in law enforcement for about 15 yrs; her husband of 20-some years has worked his way up to captain in the same agency she was with. THEY ARE NOT POOR PEOPLE. There is no excuse for her behavior--or causing my Gram to have to file bankruptcy to get rid of the relentless bill collector calls. I'm sure a lot of us know how nasty they can be. They had her so stressed out that bankruptcy was the only answer for her. My sister quit associating with me, our mom, gram, our other sister & an aunt. Over the years (this has been going on since 2003), I have tried mending fences with her, but she clearly wants nothing to do with me. Her family is not good enough because we dont value material things; I live on a farm, not a fancy house in an upscale neighborhood. I used to feel like you, with all the guilt. It wasnt until I forgave her that I was able to get on with my life & not worry about whether she wanted me in her life or not. I love my life, my God & my family. But I dont feel the need to keep negative people or 'haters' in my life. I have never felt so free since I let go of my baggage.
It's true that you cant pick your family...but you also dont have to keep them in your life either.