I was in a similar situation a few years ago with a "secret sister" group, and it was a terrible feeling. I figured that it would be best for me to just tell the name-drawing coordinator the truth. I could "beat around the bush" and come up with some lame excuse, but I decided the truth would be the best, most forward way to deal with it.
I told the person that I was in a financial bind, and I would be unable to come up with an appropriate, deservimg gift for someone whose name I'd draw. I also said that if money wasn't so tight, I would participate happily. But since these were lean times, I'd like to bow out as graciously as possible.
I thought I'd get a lot of grief for not being a "team player" but amazingly, the coordinator was very gracious.
I hope that it works for you. I know the feeling--like people will think you are party-pooper or a Scrooge--when in reality, that is not the truth at all.