问题描述:

I have some code containing an iterator, which works well:

import multiprocessing

m = [0,1,2,3]

class gener(object):

def __init__(self, m):

self.m = m

self.c = 0

def __iter__(self):

return self

def next(self):

time.sleep(1)

ret = self.m[self.c]

self.c += 1

return ret

tt = gener(m)

def gen(t):

return t.next()

print gen(tt)

print gen(tt)

print gen(tt)

OUT:

0

1

2

But if I try to insert it into a parallel process I don't get the expected results:

import time

import multiprocessing

m = [0,1,2,3]

class gener(object):

def __init__(self, m):

self.m = m

self.c = 0

def __iter__(self):

return self

def next(self):

time.sleep(1)

ret = self.m[self.c]

self.c += 1

return ret

tt = gener(m)

def gen(t):

return t.next()

job1 = multiprocessing.Process(target=gen, args=(tt,))

print job1.start()

job2 = multiprocessing.Process(target=gen, args=(tt,))

print job2.start()

job3 = multiprocessing.Process(target=gen, args=(tt,))

print job3.start()

OUT:

<None)>

<None)>

<None)>

I can't figure out, how could I use this iterator via parallel.

Can anybody help me?

Thank you!

UPDATE:

Following @Anand S Kumar very useful help, I updated my code, and it works fine, except that the output is ambiguous, Currently I'm trying to figure out what is wrong, maybe it will be the subject to another thread, maybe Anand will help me :)):

from threading import Thread, Lock

import time

m = [0,1,2,3]

starter = 0

class gener(object):

def __init__(self, m):

self.m = m

self.c = 0

def __iter__(self):

return self

def next(self):

time.sleep(1)

ret = self.m[self.c]

self.c += 1

return ret

tt = gener(m)

def f(t):

global starter

lock = Lock()

lock.acquire()

try:

starter = t.next()

finally:

lock.release()

t1 = Thread(target=f,args=(tt,))

t1.start()

t2 = Thread(target=f,args=(tt,))

t2.start()

t3 = Thread(target=f,args=(tt,))

t3.start()

t1.join()

print starter

t2.join()

print starter

t3.join()

print starter

Different outputs, with the same code:

0

1

2

2

2

2

2

2

网友答案:

You are trying to print the return value of job.start() function, which does not return anything, hence it prints None .

Instead of printing the return value of job.start() , maybe you can move the print statement into the gen(t) function, something like -

def gen(t):
    print t.next()

And then run the program , without printing the job.start() .

If you want to recieve the return value from the function, you can use Pool from the multiprocessing module. [Documentation]

An example from the documentation -

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    pool = Pool(processes=4)              # start 4 worker processes
    result = pool.apply_async(f, [10])    # evaluate "f(10)" asynchronously
    print result.get(timeout=1)           # prints "100" unless your computer is *very* slow
    print pool.map(f, range(10))

But please note, you are actually creating multiple processes , not threads, they would not share the global variables.

I believe what you want is threads , maybe an example like below would help get you started -

from threading import Thread, Lock
m = [0,1,2,3]
starter = 0

class gener(object):
    def __init__(self, m):
        self.m = m
        self.c = 0

    def __iter__(self):
        return self

    def next(self):
        ret = self.m[self.c]
        self.c += 1
        return ret 

tt  = gener(m)


def f(t):
    global starter
    lock = Lock()
    lock.acquire()
    try:
        starter = t.next()
    finally:
        lock.release()

t1 = Thread(target=f,args=(tt,))
t1.start()
t2 = Thread(target=f,args=(tt,))
t2.start()
t1.join()
t2.join()
网友答案:

Two problems:

1) start() function does not return a value, therefore you're getting None to print out.

2) You're passing the generator object to each process, thereby copying the original gener object (declared in the master process) three times, once to each of the forked processes' stacks. So, even if you change your function to:

def gen(t):
    print t.next()

all you will do is call next() the first and only time on every individual gener object, printing:

0
0
0

In order to get the desired effect, you need to perform the iteration in the master process, passing its result to each spawned process:

job1 = multiprocessing.Process(target=gen, args=(tt.next(),))
#print job1.start()

job2 = multiprocessing.Process(target=gen, args=(tt.next(),))
#print job2.start()

job3 = multiprocessing.Process(target=gen, args=(tt.next(),))
#print job3.start()

Then all your gen function needs to do is print the value:

def gen(t):
    print t

And you get:

0
1
2
相关阅读:
Top