Recent studies in normal-hearing listeners have used envelope-vocoded stimuli to show that the masking of speech by noise is dominated by the temporal-envelope fluctuations inherent in noise, rather than just the overall power of the noise. Because these studies were based on vocoding, it was expected that cochlear-implant (CI) users would demonstrate a similar sensitivity to inherent fluctuations. In contrast, we found that CI users showed no difference in speech intelligibility between maskers with inherent envelope fluctuations, and maskers with no fluctuations. Here, these initial findings in CI users were extended to listeners with cochlear hearing loss and the results were compared with those from normal-hearing listeners at either equal sensation level or equal sound pressure level. In CI users, the results support the predictions based of the hypothesis that poor spectral resolution leads to a summation of independently fluctuating temporal envelopes, which in turn leads to “smoother” effective noise temporal envelopes with less energy at modulation frequencies relevant for speech. In hearing-impaired listeners (and in normal-hearing listeners at high sound levels), the results are consistent with a reduction in low-frequency inherent noise fluctuations due to broader cochlear filtering. The reduced effect of inherent temporal fluctuations in noise, due to either current spread or broader cochlear filters, provides a new way to explain the loss of masking release experienced in CI users and hearing-impaired listeners when additional amplitude fluctuations are introduced in noise maskers.