you are supposed to climb up and out from bass fishing!!! haha
Been skunked many a time. Keep plugging away. When the riggers don't work switch to lead. We always start with at least one leadcore each in the water. Usually the lead picks up the early fish and as the morning goes on the riggers become more active.
Some days the riggers will not trip at all so it is good to have another method to try.
Good luck and don't give up! It will come but you have to put in the time. Once you start catching (and you will) you can put up with the slower days a little easier.
I think your fishing too close to the down rigger ball. Try back 30 ft. or so.When not catching fish on the riggers. I try longer leads. Some times back more then 30". But there are days the riggers won" catch a thing. Then its leadcore time.