Mould in a finch bird-feeder?

I had a bird-feeder up with finch seed and holes for finch seed, and i hardly ever got any birds, but half the seed sprouted and turned mouldy. is there any way besides taking the feeder down when it rains (since if it starts when i go to work or at night i can never be sure). Is this just a bad feeder for finch seed? should i screw on the larger holes for regular seed and put regular seed into it (tho i don't wanna feed the sparrows and cardinals... i wanna feed the singing house finches!)? it is a bad place for a feeder hanging off apartement wall but i saw 2 finches on it today, which is why i took it down just not cuz it was filled 1/2 way with mould. it was only up for 1 month! can i use this feeder at all or should i get another one? do they all do that and i should just waste seed? annoying...