Sigh - some people are just that way. My DGF has a SIL who appears to have no desire in life to amount to anything. Both of said SIL's parents have been on disability for years, and so is one of her sisters. No offense to anyone with such a condition, but "chronic fatigue" in a 24 year old woman who was fine before college, sounds highly suspect to me. Especially given the family history.
The brother is an extremely talented chef, but a pitiful excuse for a worker. He got fed up with his last two jobs (working double shifts so the wife could sofa surf) so he stopped going. Their firstborn is due January 9th.
We've had them over to the house, given them food out of our garden and freezer, and will probably share the wealth from the coop as well once the girls stop freeloading. Luckily DGF loathes the SIL, and knows the brother well enough to know what to expect. She does stuff for them because she loves them and they (at least he) are/is family. I do stuff for them, but I guess I also have the luxury of only doing what I am comfortable with doing.
Here's hoping you can sit the DH down and say "Honey, I love you and would move the moon if you asked - your brother, not so much." They are family and they are not going away. My advice would be to work it so you can minimize your own stress, and don't set yourself up to be taken advantage of - give whatever you are comfortable with and truly don't mind having for yourself. Spend your time with the people who enrich your life and whose lives you enrich, spend the holidays with your family
(ouch that came out harsh, but yeah...you get the idea...)