I have mold in my basement but no one ever goes down there. Can this be a problem?

Yes, it is possible that contaminants can enter small openings in the ventilation system and be distributed to other parts of the home.