We conducted a cross-linguistic study to evaluate the impact of language experience on midbrain encoding of acoustic dimensions. Midbrain electrophysiological responses were recorded to the four Mandarin tones in native Chinese (N = 10) and English (N = 10) listeners, through a counter-balanced block design. English participants were trained over multiple days to achieve tone categorization accuracy and reaction time equal to that of the Chinese participants. We assessed the extent to which the four Mandarin tones could be discerned from the electrophysiological responses, using a data-driven machine learning approach. The machine learning output was used to generate dissimilarity matrices that were subjected to a multidimensional scaling (MDS) model. A two dimensional MDS solution emerged that corresponded to “pitch height” and “pitch direction” of the Mandarin tones. Findings derived from the individual differences scaling (INDSCAL) method revealed that, initially, pitch direction was weighted more by the Chinese participants relative to the English participants. However, following training, relative weighting on pitch direction was found to increase in the English participants, comparable to the Chinese participants. These results suggest that long-term language experience and short-term training with Mandarin tones selectively enhances weighting of pitch dimensions at the level of the auditory midbrain.