So, fungal spores are literally everywhere, and the requirements for fungus to thrive seem to be trivially low; give it a moderately humid environment and it’ll grow on a bare concrete wall ffs eating god only knows what; the dust from the air maybe?
Well, and the great outdoors is full of slightly damp places, many of them downright soggy most of the time - and absolutely rife with organic material to snack on.
Where’s the bottleneck? Why isn’t the world a choking fungal hellscape?
Mold needs the proper conditions to grow.
Probably most important to your question is nutrients. Mold needs certain nutrients in order to grow, which is why it doesnt grow on everything. Some materials that have the nutrients mold needs include various foods, drywall, cotton, and some types of wood.
Mold also need decently specific moisture in its evironment (humidity of over 55%), a proper temperature range, shielding from UV rays that kill it, and if these conditions are met, mold can grow in about 24-48 hours.