问题描述:

Some context...

I've got a thread that holds my Rabbit consumer class RabbitClient say.

When the client consumes a message from the queue, it feeds it to its CrawlWorker - an object that manages the lifecycle of spiders, and catches scraped items.

To publish back to the queue, it needs to connect a "publish" callback to the CrawlWorker's item_scraped signal, which it does just fine.

However, this callback executes on the main thread - and not the thread that holds the Rabbit client. Is there a way, to make sure the callback is invoked on the thread that holds the Rabbit client?

Here's a very simple example - I think I've boiled the problem down to its essence:

import threading

import logging

from scrapy.signalmanager import SignalManager

from twisted.internet import reactor

logging.basicConfig(level=logging.INFO,

format='(%(threadName)-10s) %(message)s')

SIGNAL = object()

def func():

message = 'Message from {0}!'.format(threading.currentThread().name)

signals.send_catch_log(SIGNAL, message=message)

class ReceiverThread(threading.Thread):

def __init__(self, signals):

self.signals = signals

self.signals.connect(self.receive, signal=SIGNAL)

super(ReceiverThread, self).__init__(group=None, target=None,

name='ReceiverThread', verbose=False)

def receive(self, message):

logging.info('Received message: {0}'.format(message))

def run(self):

# func()

pass

if __name__ == '__main__':

signals = SignalManager()

thread = ReceiverThread(signals)

thread.start()

func()

reactor.run()

If you run this, you get the message:

(MainThread) Received message: Message from MainThread!

Commenting in the func() call in run(), and commenting out the func() call in __main__ will give:

(ReceiverThread) Received message: Message from ReceiverThread!

I'm not that experienced in threaded programming, so these results are probably expected for most people reading this, but is there a way to get:

(ReceiverThread) Received message: Message from MainThread!

相关阅读:
Top