IE can account for a fraction of total traffic, but it still can affect large percentage of users. The same person can use several devices (work, home, mobile) to browse the web at different times. I can't recount how many times I saw some page at work only to send myself the link to read it later. I'm not forced to use IE anywhere, but if I were, loosing that initial visit would also lower the number of visits from other browsers.