The performance of lateralization task based on interaural time or level difference (ITD or ILD) often varies among listeners. This study examined the extent to which this inter-listener variation could be accounted for by the coding efficiency of temporal-structure or level information below the stage of interaural interaction. Young listeners (20s to 30s) and early-elderly (60s) listeners with or without mild hearing loss were tested. The ITD, ILD, TIME, and LEVEL tasks were intended to measure sensitivities to ITD, ILD, the temporal structure of the stimulus encoded by the neural phase locking, and stimulus level, respectively. The performances of the ITD and ILD tasks were not significantly different between the age groups, while the elderly listeners exhibited significantly poorer performance in the TIME task (and in the LEVEL with high frequency stimulus only) than the young listeners. Significant correlations were found between thresholds for the ILD and LEVEL tasks with low- and high-frequency stimuli and for the ITD and TIME tasks for the high-frequency stimulus, implying peripheral coding efficiency as a major factor that determines lateralization performance. However, we failed to find a correlation between the ITD and TIME tasks for the low-frequency stimulus, despite a large range of threshold values of the TIME task. This implies that in a low frequency region, the peripheral coding efficiency of stimulus temporal structure is a relatively minor factor for the ITD-based lateralization performance.