i've parallelize function involves "yield". simple replica of whole program i've work on, sums problems i'm facing. here i'm try understand multiprocessing, apply_async , yield project in example i've used multiprocessing.pool , have used apply_async parallelize. i've put print statements in "parallel" function, aren't getting printed. when replace yield return the print statements getting reflected. i'm not nature of yield. know generator , can used once after returned. please advise on how working.
import multiprocessing mp results=[] def parallel(x, y, z): print "aim in parallel" count=0 result=[] line in range(10000): count+=1 result.append(count) p=x**3+y+z print " result" print result print p if p > 0: return result # yield result, p # count += 1 # yield p, result # count += 1 def collect_results(result): print "aim in callback" results.append(result) #print results def apply_async_with_callback(): pool = mp.pool(processes=10) r = range(10) [pool.apply_async(parallel, args=(2,5, 7),callback=collect_results) in r ] pool.close() pool.join() print "length" print len(results) print results if __name__ == "__main__": apply_async_with_callback()
when function containing yield
statement called, doesn't run code returns generator instead:
>>> p = parallel(1, 2, 3) >>> p <generator object parallel @ 0x7fde9c1daf00>
then, when next value required, code run until value yielded:
>>> next(p) ([10000], 6) >>> next(p) (6, [10000])
in case, results
contains 10 generators have been created asynchronously, they've never been run.
if want use generator, change code bit target function creates list generator:
def parallel2(x, y, z): return list(parallel(x, y, z)) def collect_results(lst): results.extend(lst) def apply_async_with_callback(): pool = mp.pool() _ in range(10): pool.apply_async(parallel2, args=(2, 5, 7), callback=collect_results)
Comments
Post a Comment