It sounds to me like 'we're having cupcakes..." was an invitation of sorts. Is there some reason you can't go with the ex being there as well? My husband and I have to be around his ex, but only during times that involved his grandchildren's birthday parties. We don't like having to be around her but we do it for the grandbabies sake. We act polite and civil but frankly are glad when it's over. I'm sure it's just as stressful for her.
My suggestion would be....GO...and be nice for the sake of baby Grace. Unless your husband and his ex threaten to throw down and have it out I see no reason why you can't put up with this for a couple of hours. Again, I'm not clued into their history so there may be good reason why you don't go. Or does his daughter not want both of them together for some reason?

Divorces are hard but not impossible if everyone is willing to give a little for the sake of the children/grandchildren.