It is entirely possible to study a language for several hours every day and immediately upon closing the textbook become engaged in a conversation where the native speaker of studied language will use approximately none of the words you studied.
There's only one real response in a situation like this: a counter-attack in high speed English, delivered monotonously with the mouth hardly opening. Of course, this isn't a friend-winning tactic and is admittedly particularly obnoxious but I'm desperately hankering for just one day here in Japan where I don't feel utterly stupid.