Hmm, that's a tough one. It's definitely worth resolving, since he's your brother, and I'm assuming it's not that you're both teenagers and this is a phrase. Based on the information you've given, I would be inclined to try to ask him directly, and privately: Did I offend you so that you act this way? Is something making you uncomfortable?
If this doesn't elicit a real answer: Can you pinpoint the year you last enjoyed spending time with the rest of us? What happened then?

"Privately" because even though it's a family thing, it's also first and foremost a personal thing -- as you mention with the 2.5 hour drive.

I guess from the way you've written your question, you obviously feel this is something your brother has chosen to do rather than, say, a result of some family fight. But you might be surprised at the things people think. Maybe he had a fight with another family member he assumes you know about and is sure you took sides... or whatever. Maybe his partner is really controlling and hates his family so your brother feels uncomfortable loving his own. Basically, whatever the situation, it can't hurt to do a little research, and a rather harmless but direct way to do that is to say, This isn't reasonable behavior so I assume you have your reasons, and I wanted to hear what they were, since we're brothers.

Some related questions are whether you feel there was really a break (were you close as children) or whether there was a gradual but definitive moving apart; and whether you feel comfortable discussing this with the other siblings you mention.

I second what Solveig said, especially the part about talking to him privately.  You may feel hurt and want to call him onto the carpet to embarrass him into talking to you, but I think that would do irreparable harm.  It may be a situation where he feels (correctly or incorrectly) like everyone is against him and therefore he is very guarded in his interactions with all of you; he may be standoffish as a protective mechanism - if he stays aloof from you, you won't have the opportunity to say anything judgmental or hurtful to him, and you can't hurt him.  I UNDERSTAND that you aren't planning to be judgmental or hurtful, but he doesn't know that, and if he has been hurt before by things people have said to him, he's probably hyper-sensitive to any situation that could add to his hurt.

Additionally, I would say that when you talk to him initially, keep all the negative stuff out of it.  The only part of Solveig's response with which I disagree is the conversation starter "This isn't reasonable behavior."  To him, it is reasonable.  I would choose something more neutral like, "This is what I see you are doing; I don't understand it; it hurts my feelings because I want to be close to you; can you explain to me why you act the way you do because I want to understand you and have a good relationship with you."  There's no blame in that kind of language. 

Then, importantly, LISTEN to him.  Try to remember that he sees the world differently from you, even if you are family.  He just may not like to be around other people that much.  He may be setting boundaries for his own comfort.  If that's the case - if he's not limiting his interactions with you because of anything other than his personal preference not to spend much time with his family - then you have to decide how to react to him.  Judging him or trying to change him aren't options - they will only cause a greater rift.  All you can do is decide how his choices impact your relationship with him, respond accordingly, and communicate clearly with him so you both know where you stand.