Listen as if you would have to repeat.
In real life conversations, we listen to people with our reply in mind. As the other person is talking, we are preparing our reply and often can't wait saying it. In the worst case, this leads to us interrupting the other person and in the majority of times it makes us miss parts of what was being said.
I believe one of the worst crimes you can commit on an improv stage, is not being able to repeat what has just been said. Be it by a narrator, your partner in the scene, by people in the scene while you're out or an audience suggestion.
Awareness of every detail is crucial. Moreover, your reply isn't so interesting anyway. We're more interested in your honest reaction, than merely your reply.