If your brother knows your door is always open should he change his heart/mind, then that's good enough for now.

How old are the grandchildren?
It feels it abit strange/toxic it is the grandmother not wanting you to see the children. Something seems amiss here, since it is would be the parents who would make the decision for the children, not the grandmother.

It sounds like a strange, but temporary situation (although that could last for a long time). Maybe at least to suggest at a better time in a few months, that you see children on neutral ground, at a restaurant or country fair, etc.
_________________________
http://cyclewriteblog.wordpress.com/ (How cycling leads to other types of adventures, thoughts)
http://velourbansism.wordpress.com