For me, the problem is not so much personal as society-wide. I'm not so concerned that using antibacterial soap will make me sick directly, but when practically everyone everywhere is using it, the bacteria just develop a resistance. These soaps may wipe out "99.9 percent" of bacteria, but the 0.1 percent that survive will spread and be harder to wipe out. So the more we use this stuff (and use antibiotics as preventatatives in animals that aren't sick, and treat humans with antibiotics when they aren't appropriate, etc.), the tougher the germs become.
The direct cause-and-effect link, I think, is more important for parents of small children. That is, if kids aren't exposed to *some* germs when they're young, their immune systems won't develop normally. I believe this is known as the "hygiene hypothesis," and it seems to be gaining increasing acceptance in the mainstream medical community.