Listeners can account for systematic variability between talkers, which is learned over exposure to multiple talkers. Previous research suggests that listeners can both generalize prior knowledge of phoneme categories from a familiar to a novel talker (Eisner & McQueen, 2005; Kraljic & Samuel, 2006, 2007) and individuate talkers, preventing generalization (Luthra, Mechtenberg, & Myers, 2021; Tamminga, Wilder, Lai, & Wade, 2020). It is unclear how listeners balance these competing demands. Participants (n = 160) learned two novel talkers (one male and one female voice) with a unique voice onset time (VOT) boundary across two days. On each day, participants were passively exposed to a bimodal distribution of VOTs from one talker, then tested on a second talker (uniform distribution). Day 1 assessed how listeners generalize to a novel talker while Day 2 assessed how the talker that was learned on Day 1 is individuated from the new talker. We found evidence for generalization after Day 1 but little evidence of learning after learning both talkers on Day 2. Two follow-up experiments using interleaved designs and supervised learning also showed little evidence for individuation. This suggests that listeners are likely accounting for variability by shifting their VOT boundary to match current input.