Implicit statistical learning (ISL)—the discovery of structure using statistical properties of the input—has been documented in multiple modalities including vision, audition, and touch. However, modality-effects have been demonstrated in ISL tasks providing evidence for modality-specific processing. The current work presents the first experimental link between modality effects in ISL tasks and modality-specific processing of attention. In experiments designed to manipulate both modality-specific factors and attention, we report interactions of attention and modality modulated by stimulus timing. Specifically, we find that auditory ISL does not occur for either attended or unattended stimuli in temporally distal conditions. However, under temporally proximal conditions favorable to auditory perceptual grouping, we report significant learning of attended stimuli and marginal learning of unattended stimuli in auditory ISL. There is no corresponding relationship between attention and temporal proximity in visual ISL likely because visual perceptual grouping occurs along spatial rather than temporal dimensions. Thus, this work provides evidence for modality-specific application of attention in ISL tasks, providing insight into the on-going debate as to the role of attention in ISL. More generally, these findings elucidate the manner in which the senses constrain cognitive processing not just in ISL but possibly in learning and memory, more broadly.