We developed an objective measure to assess neural sensitivity to interaural time differences (ITDs) conveyed in the temporal fine structure (TFS) of low-frequency sounds, and to distinguish it from neural sensitivity to ITDs conveyed in the temporal envelope of amplitude-modulated (AM’ed) high-frequency sounds. Bilateral cochlear implant (CI) users can potentially access the latter cues, but are largely unable to process low-frequency ITD information. To this end, a measure of brain activity that distinguishes between neural pathways coding ITDs in the TFS and the stimulus envelope would be useful in developing stimulus strategies to improve spatial listening in CI users. Using electroencephalography (EEG), we recorded brain activity to sounds in which the interaural phase difference (IPD) of the TFS (or the modulated temporal envelope) was repeatedly switched between leading in one ear or the other. When the amplitude of the tones is modulated equally in the two ears at 41 Hz, the interaural phase modulation (IPM) generates an intracranial percept of a sound moving from one side to the other – and evokes an IPM following-response (IPM-FR) in the EEG signal. For low-frequency signals (520-Hz carrier), IPM-FRs were reliably obtained for a wide range of modulation rates and IPDs, and were largest for an IPM rate of 6.8 Hz and when IPD switches (around 0°) were in the range 45-90°. Increasing the modulation frequency increased the magnitude of IPM-FRs. IPDs conveyed in envelope of high-frequency tones (3-kHz tones AM’ed at 128 Hz, with a second-order modulation of 41 Hz) also generated a reliable pattern of IPM-FRs, but one in which response maxima occurred for IPDs switched between 0° and 180° IPD. The data are consistent with the interpretation that distinct binaural mechanisms generate the IPM-FR at low and high frequencies, and with the reported physiological responses of medial superior olive (MSO) and lateral superior olive (LSO) neurons in other mammals. Low-frequency binaural neurons in the MSO are considered maximally activated by IPDs in the range 45°-90°, consistent with their reception of excitatory inputs from both ears. High-frequency neurons in the LSO receive excitatory and inhibitory input from the two ears receptively – as such maximum activity occurs when the sounds at the two ears are presented out of phase.