问题描述:

I'm trying to clean and put in prod-like shape my code but I'm experiencing a lot of problems with the prod env from Google App engine.

Everything is working with no problems in local but every time I try to test on prod I have deadline and timeout issues.

My questions are:

1) What is the best/suggested practise to push the SAME card with the SAME attachment ( a video one ) to many users from my glassware.

This is the classes that is called from the queue:

class batchWorker(webapp2.RequestHandler):

def post(self):

userToPush = self.request.get('user_id')

#posting video

media_link = util.get_full_url(self, '/static/video/short_from_glass_low.mp4')

resp = urlfetch.fetch(media_link, deadline=2000)

media_video = MediaIoBaseUpload(io.BytesIO(resp.content), mimetype='video/mp4',

resumable=False)

users = Credentials.all()

for user in users:

creds = StorageByKeyName(Credentials, user.key().name(), 'credentials').get()

mirror_service = util.create_service('mirror', 'v1', creds)

#first card

timeline_item01 = {'text':'New video from bundle - Test002'}

timeline_item01['bundleId'] = 'video_001'

timeline_item01['isBundleCover'] = 'true'

mirror_service.timeline().insert(body=timeline_item01).execute()

#second card

timeline_item = {'text': 'Text here'}

timeline_item['isBundleCover'] = 'false'

timeline_item['bundleId'] = 'video_001'

mirror_service.timeline().insert(body=timeline_item, media_body=media_video).execute()

logging.info("Posted video for user %s" % user.key().name())

This is how i push it to the queue:

taskqueue.add(url='/worker', params={'user_id': '103012621006129330069'})

Some time i got this done, some others i got log as the followings:

2013-08-01 07:15:37.695 /worker 500 6481ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

I 2013-08-01 07:15:31.676 make: Got type <class 'google.appengine.api.datastore_types.Blob'>

I 2013-08-01 07:15:31.677 validate: Got type <class 'oauth2client.client.OAuth2Credentials'>

I 2013-08-01 07:15:31.711 make: Got type <class 'google.appengine.api.datastore_types.Blob'>

I 2013-08-01 07:15:31.712 validate: Got type <class 'oauth2client.client.OAuth2Credentials'>

I 2013-08-01 07:15:31.713 URL being requested: https://www.googleapis.com/discovery/v1/apis/mirror/v1/rest?userIp=0.1.0.2

I 2013-08-01 07:15:31.770 URL being requested: https://www.googleapis.com/mirror/v1/timeline?alt=json

I 2013-08-01 07:15:32.675 URL being requested: https://www.googleapis.com/upload/mirror/v1/timeline?uploadType=multipart&alt=json

E 2013-08-01 07:15:37.685 The API call urlfetch.Fetch() took too long to respond and was cancelled. Traceback (most recent call last): File "/base/data/home/runtimes/python27

some others I got this:

2013-08-01 07:15:11.066 /worker 500 7239ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

I 2013-08-01 07:15:04.434 make: Got type <class 'google.appengine.api.datastore_types.Blob'>

I 2013-08-01 07:15:04.439 validate: Got type <class 'oauth2client.client.OAuth2Credentials'>

I 2013-08-01 07:15:04.527 make: Got type <class 'google.appengine.api.datastore_types.Blob'>

I 2013-08-01 07:15:04.528 validate: Got type <class 'oauth2client.client.OAuth2Credentials'>

I 2013-08-01 07:15:04.529 URL being requested: https://www.googleapis.com/discovery/v1/apis/mirror/v1/rest?userIp=0.1.0.2

I 2013-08-01 07:15:04.587 URL being requested: https://www.googleapis.com/mirror/v1/timeline?alt=json

I 2013-08-01 07:15:04.620 Refreshing due to a 401

I 2013-08-01 07:15:04.628 make: Got type <class 'google.appengine.api.datastore_types.Blob'>

I 2013-08-01 07:15:04.629 validate: Got type <class 'oauth2client.client.OAuth2Credentials'>

I 2013-08-01 07:15:04.630 Refreshing access_token

I 2013-08-01 07:15:04.833 make: Got type <class 'google.appengine.api.datastore_types.Blob'>

I 2013-08-01 07:15:04.834 validate: Got type <class 'oauth2client.client.OAuth2Credentials'>

I 2013-08-01 07:15:04.839 validate: Got type <class 'oauth2client.client.OAuth2Credentials'>

I 2013-08-01 07:15:04.839 get: Got type <class 'model.Credentials'>

I 2013-08-01 07:15:05.970 URL being requested: https://www.googleapis.com/upload/mirror/v1/timeline?uploadType=multipart&alt=json

E 2013-08-01 07:15:10.985 Deadline exceeded while waiting for HTTP response from URL: https://www.googleapis.com/upload/mirror/v1/timeline?uploadType=multipart&alt=json Traceba

At the moment I'm kind discouraged since I can 100% comprehend the logic but the prod implementation is a pain.

Do I need to use batches calls in order to push the cards one after another?

is there a different approach?

Why I'm hitting the deadline error when the queue processes are supposed to have 10 minutes deadline (the deadline error comes up after 10 second into the execution).

网友答案:

The timeout is actually coming from your upload request, you can fix that by modifying the code that create the service instance (util.create_service) and set a timeout argument when instantiating an httplib2.Http object:

# Instantiate an Http instance
http = httplib2.Http(timeout=2000) # Use a bigger value if you want to stay safe.

Regarding batching the request, unfortunately, the API does not support mixing batch and media upload requests...

网友答案:

I think that the line:

 resp = urlfetch.fetch(media_link, deadline=2000)

is trying to set a deadline for that request of 2000 seconds but:

From the docs: maximum deadline (request handler) 60 seconds So your deadline will be ignored or revert to a default value.

相关阅读:
Top