Advice needed on groomsman situation

My soon to be husband would love some advice on how to handle a situation with a groomsman.

Short story is the groomsman expects to make a speech at the wedding, but we decided that we do not want him to make a speech.
Long Story is this groomsman asked about making a speech very early in our planning process before FH officially picked groomsman and FH without thinking about all the intricacies of wedding planning said “sure”. I did not know anything about this until months later when I overhead the groomsman talking about preparing a speech. We only have 4 hours for a reception and already have MOH, 2 best men, a quick word from us, and possibly a parent or two to fit in during the speeches. Also, this groomsman has had a couple of comments over the last year that I feel are incredibly rude/disrespectful to me/mine and FH’s relationship and I am adamant about him not making a speech at the wedding. FH is planning on having a conversation with him about this and offering up the rehearsal dinner as a compromise, but not sure how to approach the subject. And we feel the groomsman will not be understanding.