The way the buses are laid out in my city is that the seats are directly facing each other. So staring outside could make it seem like your staring at people if it’s too crowded. So it’s more comfortable to pretend to use your phone.
People have a pretty good sense of whether you're staring at them or something just beside or behind them. Not really from the angle of your eyes but the way you react (or not) when the other person looks back.
I wouldn't worry about that so much. And I worry about a lot of social things :)