Python: Yield in multiprocessing Pool -


i've parallelize function involves "yield". simple replica of whole program i've work on, sums problems i'm facing. here i'm try understand multiprocessing, apply_async , yield project in example i've used multiprocessing.pool , have used apply_async parallelize. i've put print statements in "parallel" function, aren't getting printed. when replace yield return the print statements getting reflected. i'm not nature of yield. know generator , can used once after returned. please advise on how working.

import multiprocessing mp results=[]  def parallel(x, y, z):     print "aim in parallel"     count=0     result=[]     line in range(10000):         count+=1     result.append(count)     p=x**3+y+z     print " result"     print result     print p     if p > 0:        return result #      yield result, p #      count += 1 #      yield p, result #      count += 1  def collect_results(result):    print "aim in callback"    results.append(result)    #print results   def apply_async_with_callback():     pool    = mp.pool(processes=10)     r = range(10)     [pool.apply_async(parallel, args=(2,5, 7),callback=collect_results) in r ]     pool.close()     pool.join()     print "length"     print len(results)     print results  if __name__ == "__main__":     apply_async_with_callback() 

when function containing yield statement called, doesn't run code returns generator instead:

>>> p = parallel(1, 2, 3) >>> p <generator object parallel @ 0x7fde9c1daf00> 

then, when next value required, code run until value yielded:

>>> next(p) ([10000], 6) >>> next(p) (6, [10000]) 

in case, results contains 10 generators have been created asynchronously, they've never been run.

if want use generator, change code bit target function creates list generator:

def parallel2(x, y, z):     return list(parallel(x, y, z))  def collect_results(lst):    results.extend(lst)  def apply_async_with_callback():     pool = mp.pool()     _ in range(10):         pool.apply_async(parallel2, args=(2, 5, 7),                          callback=collect_results) 

Comments