Although the "birther" claim avoids presenting itself as racism, it's fundamentally a way of constructing a distinction between those who are one of us and those who aren't. I would suggest that the vehemence of those who adhere to this claim owes a lot to its defensive ego-bolstering effect, and when political sentiment is constructed around such an effect, this is enough to acknowledge it as a kind of racism. Yes, it could equally be directed against a president drawn from another racial, ethnic, or cultural group (Hispanic, Jewish, etc.), but if it were directed against an obviously red-blooded American whose birthplace was in question, it would have much less appeal, and it would at any rate disappear as it was soon as shown to be false. It's also worth noting that the backers of this claim overlap to a large extent with those who, in the purest possible voice of racial "tolerance," decry Obama as a Muslim. The danger of the birther claim is that it masks its racism in the language of constitutional technicalities, making it possible for politicians to echo it and lend it an appearance of legitimacy.
In short, it's stupidity (as Leah has said), but I think we can't dismiss it as only stupidity. We would be ignoring one of the strongly problematic currents in recent American political discourse.