Now, I can't speak for all vegans, because there are various reasons behind peoples' dietary decisions, so instead I'll just give you an answer from me personally.
I have been vegetarian all of my life and recently stopped eating dairy as well, both for what you call ethical reasons. When I tell people this, many of the responses that I often get are quite similar to yours. People begin to shoot questions at me, like how I can live with myself for eating at a restaurant that sells meat (even if I don't order any) because I'm supporting the meat industry... or how can I bear to take a walk for fear that I may step on a bug. What you all seem to assume is that I think I am perfect simply because I choose not to consume animal products. I don't think I'm perfect. I realize that many of the things I do and consume have terrible repercussions for animals. I am not "sure my food choices avoid harming cute animals."  Rather, I am quite aware that there is no way to completely avoid harming animals (I could go on about the 'cute' factor as well, but that's a different matter). However, wouldn't it be even more hypocritical of me if, after saying that I hate to see animals in pain or being taken advantage of, I said "well... as bad as the meat industry is, even tofu kills birds and rabbits, so why bother trying?" and sat down to my steak? Also, what if I simply don't like the idea of animals being born and raised with the sole purpose of becoming my dinner? I know that this is a reality, and it will continue to be a reality even though I don't eat meat, but that doesn't mean it is a reality that I want to support and be part of.
It almost seems to me that people are trying to test me, to poke holes in my logic. What do they expect me to say? "Oh yes, you're right... I'm not perfect, I might as well go have a burger!"? I think this wanting to prove vegans wrong may come from you feeling like vegans are trying to prove you wrong. Like I said before, I can't even pretend to talk for all vegans, but when I say that I am vegan for ethical reasons I'm not calling people who aren't  vegan murderous or evil, and I'm not trying to say that I'm morally better than you. I'm just saying that I personally wouldn't feel comfortable eating the way you do. You don't need to defend yourselves by telling me that I'm not perfect, because I don't consider "vegan" synonymous with "angel."

Also, for the record, vegans don't eat dairy, so they probably wouldn't be content living on yogurt and fruit because of the yogurt bit. ;)

How really ironic. I've been reading Michael Pollan, too...disturbing stuff.

Rick -- I think Aliza answered your post quite intelligibly, but I would like to address the Pollan aspects. Pollan frequently gets away with assertions that go unchallenged. He has written, "Humans may not need to eat meat in order to survive, yet doing so is part of our evolutionary heritage, reflected in the design of our teeth and the structure of our digestion." The truth is the complete opposite: over an evolutionary period of at least 10 million years, our teeth and digestive system were designed for eating vegetable matter (and perhaps easy to chew, swallow and digest insects). Our (American/western) modern diet heavy on meat-eating is radically foreign to the vast majority of human evolution and experience. 

He also states, unequivocably, "Under the pressure of the hunt, the human brain grew in size and complexity, and around the fire where the meat was cooked, human culture first flourished." This is sheer speculation to support his thesis. The macho "hunt" played a role in the development of human culture, but certainly no more and probably far less than the gathering of edible plants, tubers and nuts; and there is no evidence whatsoever that hunting directly led to a larger human brain; quite the opposite is the more likely scenario.

Pollan's broadcasting the opinion of Steve Davis, who is an "animal scientist" for the poultry industry, that combine harvesting of vegetarian staples kills millions of wild creatures, is likewise not supported by any factual evidence. I would not deny that some animals may be killed during harvesting of certain crops, but require far more than the word of someone with a vested interest in perpetuating factory meat production for proof that such wanton killing happens at anywhere near the level conjectured by Davis and Pollan. Where are the photographs of such killing fields?

Of course, at least Pollan is thoughtfully considering the ethics of meat-eating, which most meat-eaters refuse to do. He genuinely seems to have some respect for the animals involved, and has called for more humane animal farming practices, while the majority of meat-eaters in the industrialized west don't consider the animal's life at all in their insatiable quest to delight the taste buds and fill the tummy for a few hours.

Yet, in the end, Pollan disingenuously contructs the answer he needs in order to continue eating meat, while failing to really address the varied health, economic, ecological or ethical problems of such a diet.