Oauth 2 store access token vs verifying on each call

Let's say my Server is both Authorization Server and Resource server

My Client (for example mobile app) can authenticate in 2 matters:

1. via Resource Owner Password Credentials Grant


2. via Authorization Code Grant / Implicit using facebook for example

Now 1 is trivial - i'm just storing the access token in the server DB after authentication and on each call to protected resource i'm verifying the access token.

But what about case 2: should my server store the access token i got from facebook or on each call for protected resource my server should call facebook api in order to validate the access token?

Are there any drawbacks to keeping an access token which your system didn't generate?

Assumptions:

1. access tokens have ttl


2. access tokens are only used to get protected resources from my Resource Server (not facebook for example)