python - Deferred Task Request deadline exceeded but work never started -

i have task queue handles deferred tasks need processed in real time (i have heard isn't idea can't find supporting documentation on this.). queue hit large influx of tasks (2500/min) , receives nothing minute or two.

queue.yaml

- name: event-message   rate: 200/s   bucket_size: 200   retry_parameters:       task_retry_limit: 2       min_backoff_seconds: 1 

what have noticed "process terminated because request deadline exceeded. (error code 123)" messages tasks aren't started. task sits in queue 10min without being run , times out. thought may related queue configuration have included here.

2015-06-18 10:22:45.842 /_ah/queue/deferred 500 641080ms 0kb appengine-google; (+http://code.google.com/appengine) module=default version=1-0-21 0.1.0.2 - - [18/jun/2015:07:22:45 -0700] "post /_ah/queue/deferred http/1.1" 500 0 "http://*******/_ah/queue/deferred" "appengine-google; (+http://code.google.com/appengine)" "********" ms=641081 cpu_ms=0 queue_name=event-message task_name=0477163503045474472 exit_code=123 app_engine_release=1.9.22 instance=00c61b117c8af1f4df1247032afd14111a19f302 e 2015-06-18 10:22:45.842 process terminated because request deadline exceeded. (error code 123) 

can explain this? ideas how can figure out why these tasks never run?

so looking @ logs more , found additional support theory tasks weren't starting. here "this request caused new process started application" appears after task 10min deadline has been reached.

2015-06-24 17:16:59.554 /_ah/queue/deferred 500 641631ms 0kb appengine-google; (+http://code.google.com/appengine) module=default version=1-0-27 0.1.0.2 - - [24/jun/2015:14:16:59 -0700] "post /_ah/queue/deferred http/1.1" 500 0 "https://integrate.versature.com/api/netsapiens_events/96615760-cac7-40d6-b80d-13a1ffade2c8/" "appengine-google; (+http://code.google.com/appengine)" "integrate.versature.com" ms=641632 cpu_ms=0 queue_name=event-message task_name=59067488044178506841 pending_ms=1061 exit_code=123 app_engine_release=1.9.22 instance=00c61b117c8f2db4ade6be8d30165cb84f0a17ab e 2015-06-24 17:16:59.554 process terminated because request deadline exceeded. (error code 123) 

here another:

i 2015-06-23 13:48:22.970 x-appengine-taskretrycount:0, x-appengine-tasketa:1435081701.5576639, x-appengine-default-namespace:versature.com, x-appengine-taskname:89115825957976957021, x-appengine-taskexecutioncount:0, x-appengine-queuename:event-message, x-appengine-current-namespace:, x-appengine-country:zz  e 2015-06-23 13:58:21.669 traceback (most recent call last): file "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 267, in handle result = handler(dict(self._environ), self._startresponse) file "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__ rv = self.router.dispatch(request, response) file "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher return route.handler_adapter(request, response) file "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__ return handler.dispatch() file "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch return method(*args, **kwargs) file "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 310, in post self.run_from_request() file "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 305, in run_from_request run(self.request.body) file "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 147, in run return func(*args, **kwds) file "/base/data/home/apps/s~versature-integrate-2/1-0-26.385226646541088482/app/api/resources/constructs/netsapiens_event/process_netsapiens_event.py", line 66, in process_data_for_endpoint logging.info("process data specific endpoint. endpoint key: %s", endpoint_key_url_safe) file "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/logging/__init__.py", line 1611, in info root.info(msg, *args, **kwargs) file "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/logging/__init__.py", line 1149, in info self._log(info, msg, args, **kwargs) file "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/logging/__init__.py", line 1268, in _log self.handle(record) file "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/logging/__init__.py", line 1278, in handle self.callhandlers(record) file "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/logging/__init__.py", line 1318, in callhandlers hdlr.handle(record) file "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/logging/__init__.py", line 749, in handle self.emit(record) file "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/app_logging.py", line 79, in emit self._applogslocation()) file "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/logservice/logservice.py", line 457, in write_record logs_buffer().write_record(level, created, message, source_location) file "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/logservice/logservice.py", line 270, in write_record message = cleanup_message(message) deadlineexceedederror  2015-06-23 13:58:21.684 request caused new process started application, , caused application code loaded first time. request may take longer , use more cpu typical request application.