Save the dates causing familial drama.

I never thought I'd be the one to be posting this *family drama* 🤦🏻‍♀️

So to preface this I have an uncle, maternal grandfathers brother, and his wife. They have both been in my life for 28+years however I have maybe seen them 15 times total. Which doesn't sound bad, however they lived within 5 minutes of us growing up and never really saw them. When I would see them my aunt would bully me, as a young child, and my uncle was a sloppy man who drank in excess. Neither of these people I enjoyed being around but saved face and pretended I liked it as a child. Now that I am an outspoken rude adult I don't hide the feelings for them. I am not mean to them, just choose not to engage. I am close with my grandparents, who are brother/SIL to them, so occasionally I get an iPad shoved in my face to say hi to everyone.
Onto the fun bit. For some reason my family was shocked to hear that these people I don't care for arent invited to my wedding, in which I am paying for. However noe that I am finalizing save the dates I am being met with some backlash. My mother, who ive never had a great relationship with, called me to let me know the stress of me not inviting them is "killing grandpa" that his "blood pressure is so high he's talking to his cardiologist." FH is 100% on my side and doesn't want them there either, however now I am debating on just canceling the wedding and eloping. Thus really isn't an option as it feels like we would be punishing his family, and FH wants to have the big wedding celebration.

I don't think I'm looking for advice I just had to get this out. It's driving me insane. Also, No. I will not be inviting them to keep the peace as then my "aunt" is throwing a fit to get her way and personally I don't even care to see her face.