|Does the crap trapped in your filter matting cause pollution?
Yes. If not removed by periodic cleaning or by plants, it decomposes consuming oxygen, liberating various noxious gases and carbon dioxide as well as phosphates, which can be relied upon as a reasonable guage of background pollution levels.
Phosphate testing has not come into vogue yet. In many instances, certain elements of your filtration system (water fall) will either blow off these noxious gases or remove them (carbon) - but in other cases it's a good idea to "pond-water-rinse" these pads or matts (or backwash your bead filter) to remove apparent mulm accumulations instead of letting them remain in the water column to decay.
A good frequency to do this would be every 4-6 weeks. Other experts and pundits, some of whom even have kept Koi, believe the solids MUST be removed before they decay. Apparently, decaying waste is not part of the natural biology of Koi on their planet. Obviously if you overclean any media it will be more difficult for beneficial microbes which are responsible for nitrogen reduction to survive.