Assuming good faith on the part of LW2 (a big assumption), they could set the boundary about spending time with their mutual dad (without trampling other boundaries) by:
* Letting Dad know that they and brother are not getting along and need some space, and LW2 would prefer that they and brother not be there at the same time
* Being prepared to leave if brother shows up uninvited
* Proposing outings to Dad that brother is less likely to show up for

And consider the possibility that your father is deliberately trying to make his children get along by engineering the meetings.