Great question Morgan! - let's think about it.
Everything is a thought experiment. What they reveal is the place of deliberation, not of decision. You don't care if a tree falling in the woods makes a sound or not, but the thought process, the question, that is what matters.
One which I particularly like is I think from a Woody Allen movie which starts with the question: there is a burning library with the last remaining examples of Shakspeare's works. You run to save them but there is an old lady in the library, and you can either save her or the books not both - which do you save?
It is a stupid question you might say, but what's nice is what it opens up. In how it opens the moral question and how it frames it in a way which you can't run away from. You're there and you need to decide.
Of course you don't need to decide, and you can't decide, but it still has an effect. It has an effect because you do decide and it's a daily decision of how you live your life, who you vote for, and what you support. Every day you decide on whether to save the old lady or the books, only several levels removed from that decision (like a prime minister ordering to go to battle knows people will be killed even if they are not the ones pulling the trigger).
On the other hand, the single most important thing to understand about philosophy (and life) is how important is the question you choose to face. The wrong questions (stupid thought experiments like you mention) will lead you nowhere. Choosing the correct question is the first step, and without the correct direction you can walk and walk and walk and it will lead you nowhere.
So yes, you should always be suspicious of the questions and roads people want you to take. Is this really the question we should be asking? That's where analytic philosophy mostly gets it wrong (though it depends of course). The moment you enter into their domain the fight is already lost. (You are drowned in senseless mind games which is like full gas in neutral.)
Ps. Actually having studied with Davidson in Berkeley, obviously jokes about the swampman and are we talking to him were "surprisingly" prevalent in class.