Human memories are malleable and often shaped by social interactions. Previous work has demonstrated that not only one's own goals, but also those of a task-partner can facilitate subsequent retrieval of goal-relevant lexical stimuli, known as the Joint Memory Effect (JME). We outline a social-epistemic account of the JME which proposes that this memory enhancement reflects humans' tendency to map out their interaction partners' knowledge states, leading them to cognitively prioritize information relevant to and selectively attended by their partner. This account predicts that the memory enhancement for partner-relevant words should be limited to contexts where task partners are required to process their targets in terms of meaning, instead of attending to a surface feature. Additionally, we predicted that facilitated recall performance for partner-relevant information would be accompanied by enhanced memory for the social context of that information, that is, participants should be able to link the remembered content to the agent acting on it. The results of four experiments support these predictions. We demonstrate that the JME emerges selectively, depending on which stimulus feature (word meaning or presentation color) is attended by the partner, and that it extends to (and may even depend on) memory for the social context of the targets. Prioritizing partner-relevant information in memory may be linked to processes involved in establishing and monitoring common ground.