Building upon the literature on attitudes to immigration, which mainly focuses on Europe and North America, this article explores the role of inter-group interaction in influencing host community attitudes towards refugees in East Africa. It draws upon first-hand quantitative (n = 16,608) and qualitative data collected from refugees and nearby host communities in urban and camp-like contexts in Uganda, Kenya, and Ethiopia. Focusing on the Uganda data, for which host community attitude and interaction data is available, OLS regressions reveal a small positive and significant correlation between refugee-host interaction and the perception of hosts towards refugees. This association disappears when an instrumental variable (IV) approach is used to address endogeneity issues, except when only data from the urban context is used. Combining cross-country data and qualitative data, we highlight some conditions that may shape hosts’ attitudes towards refugees, including the types of interaction, ethno-linguistic proximity, and residence in urban or camp-like contexts. In all contexts, an important part of attitude formation appears to take place at the intra-group level, within households and immediate neighbourhoods, independently of individual interaction with the out-group.