Ups and downs -- more problems with prayer

You know, this whole process would be a lot easier if I could just have some Constantine-style flaming cross in the sky message and just be completely assured of God's presence, never to doubt again. (Maybe I should change the name of this blog to The Lazy Atheist). I know, I'm confusing almighty God with a cheap parlor magician who does neat tricks for my amusement again, but I've been thrown back into a period of frustration and doubt recently that's bumming me out and just wanted to take a moment to whine that I wish this were easier. :)

There are a few things going on, but the main issue (again) is prayer. I've been following Colleen's suggestion and using the Lord's Prayer as my guide. I've tried to not focus on specifics based on what I think I need (e.g. "please don't let us lose this client"), but rather praying for divine guidance to help me see what God wants me to do in this situation. (A follow-up to the post Colleen commented on: sure enough, the client left us. And it turned out to be the best thing that ever happened to our business.)

After praying in this manner for a few weeks I was very surprised at how well it was going. I truly felt like my prayers were being answered -- God was very clearly guiding me to new insights and perspectives that would have never occurred to me on my own!

...And then there was that annoying materialist voice in the back of my head telling me that I was just seeing what I wanted to see and becoming like "one of those Christians" who fabricated religious experiences to make themselves feel better (the way I used to see all Christians). But I was so happy and the experiences I was having seemed so clearly to be answers to my prayers that I was actually able to ignore that snippy little voice.

And then while watching the Superbowl yesterday I decided to thumb through my copy of Fr. John Hardon's Catholic Catechism to see what it had to say about an issue that had been troubling me lately: what is and is not the will of God. I didn't exactly find the answer to my question, but I came across this passage about prayer:

With human intercession we seek to inform another person of our wants, and then to sway his will on our behalf. Obviously neither of these considerations applies when we make our petitions to God. Our intention is not to divulge our needs and hopes, for God knows all things. Nor can the divine will be persuaded to alter a decision...Prayer is necessary for our sake, to make us reflect on our great needs and arouse our wills to desire what God wishes us to have.

We might add that God has foreseen our prayers from eternity and thus included them in his plan for the universe, to give us (and others) what he knew we would ask for. [emphasis mine]

Maybe it shouldn't have, but that passage took the wind out of my sails about the whole prayer thing. The part about God having seen our prayers from eternity and included it in his plan gave me a headache so I didn't dwell on it too much. It's hard for me to wrap my mind around the whole free will vs. God's plan concept, e.g. that if I decide to let a coin toss determine whether or not I pray today, that the outcome of the toss was part of God's plan and my prayers will be answered (or not answered if the toss lands in favor of not praying) accordingly. That's baffling to me so I'll move on to what I found to be the most troubling part of the passage:

I interpreted the passage to say that a) prayer is not for expressing your needs and wants to God and b) God is not going to do anything based on your prayers since his plan is already laid out; prayer is really just a mental exercise for you to remember who's in charge here. That annoying voice in the back of my head was quick to point out that if the purpose of prayer is just "to make us reflect on our great needs and arouse our wills to desire what God wishes us to have," then my prayers over the past few weeks have not really been answered. What I thought was the hand of God guiding me to see things I had not seen before was really just me using the meditative time of prayer to figure things out for myself.

If I'm interpreting this correctly I guess I could get used to the idea that prayer never results in action on God's part, but that's a whole lot less comforting than what I originally perceived the purpose of prayer to be.

Am I misinterpreting this part of the Catechism? How do most religious people (particularly Catholics) perceive that prayer works? I've heard a lot of people talk about prayer in a cause-and-effect sort of way ("my prayers were answered!") -- were they all mistaken?

-----------------------------

UPDATE:

OK, I just re-read this after I posted it and I can see that the second paragraph of the passage I quoted is indeed encouraging. It seems to be saying that God does answer our prayers...sort of...but it's different since he knew what we were going to ask for and already had it planned...so hopefully you're asking for what was in his plan...although he knew you were going to ask that so maybe he incorporated it into his plan.

Maybe this issue here is that I need to think a bit more about the fact that God doesn't work in the cause-and-effect sort of way that we humans do since he already has the universe all planned out.