问题描述:

I have a piece of code that queries a DB and returns a set of IDs. For each ID, I need to run a related query to get a dataset. I would like to run the queries in parallel to speed up the processing. Once all the processes are run, then I build a block of text and write that to a file, then move to the next id.

  1. How do I ensure that all the processes start at the same time, then wait for all of them to complete before moving to the page =... and writefile operations?
  2. If run as it, I get the following error: Process object is not iterable (on line 9).

Here is what I have so far:

from helpers import *

import multiprocessing

idSet = getIDset(10)

for id in idSet:

ds1 = multiprocessing.Process(target = getDS1(id))

ds1list1, ds1Item1, ds1Item2 = (ds1)

ds2 = multiprocessing.Process(target = getDS2(id))

ds3 = multiprocessing.Process(target = getDS3(id))

ds4 = multiprocessing.Process(target = getDS4(id))

ds5 = multiprocessing.Process(target = getDS5(id))

movefiles = multiprocessing.Process(moveFiles(srcPath = r'Z://', src = ds1Item2 , dstPath=r'E:/new_data_dump//'))

## is there a better way to get them to start in unison than this?

ds1.start()

ds2.start()

ds3.start()

ds4.start()

ds5.start()

## how do I know all processes are finished before moving on?

page = +ds1+'\n' \

+ds2+'\n' \

+ds3+'\n' \

+ds4+'\n' \

+ds5+'\n'

writeFile(r'E:/new_data_dump/',filename+'.txt',page)

网友答案:

I usually keep my "processes" in a list.

plist = []
for i in range(0, 5) :
    p = multiprocessing.Process(target = getDS2(id))
    plist.append(p)

for p in plist :
    p.start()


... do stuff ...


for p in plist :
    p.join() # <---- this will wait for each process to finish before continuing

Also I think you have an issue with creating your Process. "target" is supposed to be a function. Not the result of a function as it seems you have it (unless your function returns functions).

It should look like this:

p = Process(target=f, args=('bob',))

Where target is the function, and args is a tuple of arguemnts passed like so:

def f(name) :
    print name
相关阅读:
Top