Nice suggestions, Arthur and Sean, thanks. Here are some scattered thoughts.
Shock Corridor is a great suggestion, and maybe we can return to it later.
In response to Sean's remark on the last three films he mentioned (The Shining, Seven, and Audition), I would just say that at least The Shining and Audition may not be very interested in representing mental illness per se. (Maybe Seven isn't either, but it has other similar pretensions, and I'm not keen to defend it.) One of the things I was thinking of when I started this little topic is that there's quite a history of images of madness and the irrational in film, which we could say are used for "sensational" purposes with little pretense of a relation to real mental illness, yet we might want to take this cinematic madness seriously. Nonetheless, as Sean implies, these movies probably do have some effect on the popular imagination of mental illness.
Audition radicalizes the problem of distinguishing reality from illusion, and in spite of this I find it richer and more convincing than Shutter Island. It does try present a certain character psychology, with the male protagonist alternating between a bad conscience and an assertion (justified by social norms, which the movie is probing) that he has done nothing wrong. (Compare Michael Haneke's movie Caché.) The madness here is not exactly the hero's, but I think we sense that it proceeds from him as a consequence or exteriorization of his state.
Audition might not be congenial to some viewers both because of its extreme violence and because it's so involved with certain clichés from recent Japanese pop cinema. The latter point goes for Paprika too, which takes us even farther from western movie imagery, but I might find a copy on DVD and re-watch it in the context of this discussion.
At the risk of introducing the whole horror genre under the rubric of madness, I'd like to suggest that the distorted faces of horror villains -- from Lon Chaney's Phantom of the Opera through the Universal horror films, Val Lewton movies, Mario Bava, and so forth -- could be fruitful for consideration. A naïve psychoanalytic hypothesis: these faces produce something like Freud's uncanny, or especially the version of it elaborated by Julia Kristeva, in which the subject recognizes something ambiguously resembling herself in an image of abjection. Less precisely, these images of the irrational appear as a controlled threat to the viewer's own sanity, a threat often dramatized through a hero whose autonomy and often sanity are more seriously jeopardized.
If this is right at least in outline, then does this pleasure in the uncanny tell us anything about cinema generally? Susan Sontag, in her essay about the death of cinephilia, writes about wanting to sit as close as possible to the movie screen, "ideally [in] the third row center." Many of us can still attest to this. In part, it has to do with a (partly ideological) idea of un-mediated experience, the kind we may associate with the influential French critic André Bazin. (I haven't read the collection that Arthur mentioned, but maybe Bazin's notion of realism generally has something to do with the present topic.) In part, it has to do with the cinephile's proprietary attitude toward the institution and the movie. But as Sontag notices, it is also a matter of subjecting oneself to the image. It seems to me that cinema, more than most forms, encourages the viewer to derive pleasure from giving up the position of rationality, control, and mastery over the image. (Of course, this is not to say that it doesn't also demand a critical relation to the image.)
