This is going to be a tough one Charles. It's essential though and I'll try to say something, even if only a small point.

First, it depends on your ambition. As it is clear that your ambition is to be a leader in your field, I will respond to that, but others might want to respond to choosing a direction which will interest people in the field even if it is more minor.

The major way to choose a question/direction should be by an encounter. A calling if you will. You will encounter a question/direction and for some unknown reason it will seem interesting. The reason it has to be an unknown reason is that this problem is like the corkscrew which holds the dam. If you solve it, if you unhinge the corkscrew, a flood of results will follow. But this is not clear ahead of time ,it is only intuition, and the good people of the field know where to go. They feel where this corkscrew is, and how to unhinge it. It might take a while, and things come up - kids, committees - but if you stick with it sometimes it is successful. 

I think this is true in graduate school just as much as after getting tenure. In graduate school of course you need to see you can get some results even if you won't solve the whole things, as otherwise you might end up in grad school for a very long time. And before tenure it is even more dangerous, but waiting till tenure is no solution either.

How to hear that calling? To achieve an encounter? Well, for a while you need to walk aimlessly looking every which way, getting a sense of the field and an intuition for how things move.  You go on your roundabout, listen to conferences lectures, read some, and just generally get a feel of things. Then, hopefully, at some point you will get this calling, hear a problem calling you: "Solve me!"

(I'll mention that sometimes you unscrew the cork and nothing happens. Well, maybe it needs a few more corks before everything will come tumbling down. Maybe you were headed in the wrong direction. Tough to tell. From looking at brilliant people, I think they rarely really go the wrong way, even if long periods of time it sure can look that way.)

Great question, Charles.

By way of reply, there was an interesting discussion here about what abstraction means now in art.  Although it is on some levels a very different question, there are nonetheless similarities in the role painting and theoretical science (at least at a high level) play, particularly in their role of creating frames for thought and giving us an abstract language with which to apprehend, analyze, and represent the world. So one could ask how the artist’s role as technician and master craftsman should merge with the artist’s role as visionary, and how one might balance the practical need to work within an existing representational tradition with what Arthur in a different context called extending the frame.  From various other discussions on modern art here it seems that people feel artists have largely failed in their role as mediators between the world of representation and that of life; that art has become too caught up in its own infinite conversation and not enough concerned with its creative work, with upholding its social contract to give birth to new physical, emotional, abstract languages and images and tools which everyone can use to make sense of the complexities of living. Science is not so unfavorably judged for now, though narrowness and specialization are becoming the norm (in part, simply because of the complexity of the task at hand; but surely not only because of this). 

The subject by itself does not necessarily determine the profundity of the work. Surely many banal novels were written about “a day in the life” before Joyce, and even more afterwards. But what he chose to say was masterful. The deep people can turn almost any mathematical conversation into something profound, tilting the idea slightly or changing its color or feeling what to abstract. Shortly after the Green-Tao result, when some people were going on about the problem having been solved and how wonderful it was to have a proof and be done with it, Tao gave a talk which began with surprising and strange this all was, how little it had to do with the primes in some sense, what enormous and interesting questions it opened up.  The depth of the person, at least as much as the interest of the question or the result, determines whether something is an ending or a beginning.

There are various ways of answering the question of choosing a problem, for different days perhaps. So I’ll give what might seem to be a selfish answer. One should work on the problems which will allow one to become great. Obviously I don’t mean famous.  One should try to be deeply and precisely sensitive to one’s blindness and one’s gift and take on the riddles which allow one to emerge at much higher levels.  The only paradigm shifts, the only questions and solutions which are of any permanent interest are ones which emerge in the pursuit of this kind of understanding.

Molly and Arthur, I find both your answers interesting. However, I don't feel that it completely solves my problems

Let me elaborate a bit. I think that the `call' that Arthur mentioned is what brings us to do science at the first place. We pick what we view as a question that is both interesting, fundamental, and irritating (in the sense, that we know that it has to be solved) and then do our best to attack it. We read other people's works, we borrow tools from other areas, we think hard about it etc. Then, sometimes, we get some results in the way of the big question, we open new ways of attacking the problem, we connect it to problems in other areas and all the other things that are usually called good science. What bothers me these days is that when engaged in research we may be thinking that we are doing a good and fundamental work, but at the end we see that the rock didn't move. We just peeled some dirt off it (which is ok, as it let people see other aspects of the rock), but we did not get any inch closer to solving the big problem. At the process we become better of course. We master something, we are experts, we become leaders and all those nice words, but eventually, this is nothing. 

Well, certainly this is not `nothing', so what do I mean? I think that I am after more than just this kind of an advance. Maybe I want something that will survive time. It is probably too much to ask for, but I can't imagine working on a problem that at the end, even if I solve it, will not mean much in, say, 50 years. I am not saying that you only have to publish immortal works, but rather that immortal problems should always be your goal. But this is difficult! And as I said in my first post, you start working on this immortal problem and you end up working on some problem that is so remote but still difficult. Then a community evolves around this remote problem and before you know it your goals changed and you are working on an unimportant niche. How can we make sure that this does not happen to us? (B.t.w., this reminds me of Coetzee's book disgrace - if I remember correctly, the main figure starts by wanting to write an opera but ends up writing a piece for banjo... ).

I cannot argue with the saying that these things only happen to the mediocre scientists and not to the brilliant ones, or that we don't see it yet but one day the greatness of our work will be revealed, but I still find this an unsatisfactory answer. Maybe one way of making sure that we don't end there is to constantly ask ourselves - "are we doing good science?"...

B.t.w., Molly, I am also not sure that I agree with you last sentence. Why do you think so, I don't remember reading anything about this. I do think that breakthroughs usually happens when one deeply understands, or actually recognizes, a blind spot in a theory, but I am not sure that they way to do good science is to `work on yourself'.