There are already a number of great responses here, I thought I’d add a few words about my experience. For the western approaching to LDing, I think LaBerge’s “Exploring The World of Lucid Dreaming” is the ideal starting place. It discusses all the day and night practices that promote lucidity in dreams.
I think there are two types of awareness that play into having lucid dreams: 1) dream awareness; and 2) state awareness.
Dream awareness is awareness of dreams: fully realizing that you do dream every night, being interested in your dreams, having the strong intention to remember your dreams, and becoming very familiar with your own person dreaming rhythm: when, during the course of the night do you tend to have the clearest, most vivid dreams. Also, becoming very familiar with the (typically) repeating patterns (“dream signs”) of experiences that happen in your dreams. And of course, developing increasingly vivid dream recall, and more and more becoming present and aware in the course of your dreaming.
State awareness is being aware of whether your mind (and thus your current experience ) is in the waking or dreaming state.
My motto for (lucid) dreaming is: “pay attention, reflect, recall” – we remember best those experiences to which we pay purposeful attention. Developing a habit of reflecting on these experiences to determine if they are dream-like helps achieve lucidity in dream more often (and helps to incubate the thought of determining your state in dream). Frequently recalling your experiences (both waking and dreaming) builds higher and higher experience-memory and aids greatly in recalling dreams (and waking) experiences.
As you noted, strong intent can be highly stimulating and can interfere with falling asleep. The most important thing about dreaming is being asleep! More sleep time means more dream time and more opportunities to be lucid in the dream state. So Andrew’s motto of “not too tight, not too loose” applies here. Lucid dreaming is all about learning to balance on the tightrope between the extremes of holding on to awareness too tightly (don’t fall asleep) and complete dullness (fall asleep unconsciously with no background intent/awareness to aid in lucidity).
Once one grasps and puts the western LD fundamentals into practice, I recommend adding in the dream yoga aspects: I find Andrew’s work here excellent, as is “The Tibetan Yogas of Dream and Sleep” (TYoDaS). I think the practice section of TYoDaS is the most profound and concise statement of how to go about developing lucidity in dream. I can’t recommend it enough. I even keep a bookmark to the introduction to the practice section to continually remind myself about the goals in the practice:
I find the “pay attention/reflect/recall” approach leads to highly vivid and “present” (“I am here, now”) dreams in time. Consistency over time pays off. Recognizing the dream state (lucid dreams), for me at least, takes long periods of consistently strong intention, mostly during the day. Generally after about 2-3 weeks of very strong intent to be lucid in dreams, I will have a burst of lucid dreams, assuming my dream awareness (recall, vividness, presence) is high enough.
The most effective way to frame intent (again, for me), is to have very concrete, specific goals of things you want to do or notice in the dream state, for which you are highly motivated. For instance, beginning the practice of dream yoga (something specific, like alternating between pushing your hand through a “wall” and then feeling the “hardness” of the wall). “I want to be lucid in dreams” I found is too vague a goal for my dreaming mind to latch on to to get lucid frequently in dreams.
I find the LaBerge practice of “reflection/intention moments” highly effective: where during the waking day you just stop and try to really “feel” yourself as if you were in the dream state right now, and tell yourself, “if I were lucid and dreaming right now, I would … . Tonight, when I’m dreaming, I intend to notice the dream state and do this.”