python - Using a Shared Array in multiprocessing to save values -

i'm attempting python multiprocessing working speed code i've written. code looks this:

from multiprocessing import array, pool import numpy np #setting shared memory array global misfit misfit = array('d', np.empty((dim1,dim2,dim3,dim4)).flat)  #looping through values in xrange(0,1):      #setting pool      pool = pool()      p = [pool.apply_async(self.testfunc,args=(somevals,j)) j in xrange(0,1)]      pool.close()      pool.join() 

where self.testfunc looks like:

 def testfunc(self,somevals,j):       #some calculations       k in xrange(0,1):            #some calculations            mn in xrange(0,1):                  #some more calculations                  #save results                  result = i*j*k*mn # example                  misfit[i*j*k*mn] = result 

my problem when run none of values saved in shared array, , remains empty. understand global variable, in simpler program uses exact setup, values saved array. array quite large in full program (4561920000 values). if call function outside of pool, works , values saved.

so question doing wrong here? sending shared array incorrectly?

edit: figured i'd add in code works:

from multiprocessing import array, pool numpy import empty, sin time import time import numpy np  def initarr():   = array('d', empty((5, 50, 80)).flat)   return  def testfunc(i, j, k):   count = (i*50*80) + (j*80) + k   x = sin(k)   a[count] = x   y = np.fft.fft(np.exp(2j*np.pi*np.arange(50000)/50000))   def process(i):   start = time()   pool = pool()   j in xrange(0, 50):     p = [pool.apply_async(testfunc, args=(i, j, k)) k in xrange(0, 80)]   pool.close()   pool.join()   print time() - start   global a = initarr()  in xrange(0, 5):    process(i) 

ok of our department, have version of works, in future viewing question, i'll post solution. haven't used stack overflow sorry if it's bad etiquette answer own question.

we got working using initializer function, had make sure initializer function in same file (module) function being run pool. in 1 module (misc) had:

**misc.py** def testfunc(self,somevals,j):   #some calculations   k in xrange(0,len(krange)):        #some calculations        mn in xrange(0,len(mnrange)):              #some more calculations              #save results              loc = (i*len(jrange)*len(krange)*len(mnrange))+                    (j*len(krange)*len(mnrange))+(k*len(mnrange))+mn              result = i*j*k*mn # example              misfit[loc] = result  def initpool(a):     global misfit     misfit = 

and in main file have:

**main.py** multiprocessing import array, pool misc import initpool, testfunc import numpy np  #setting shared memory array misfit = array('d', np.empty((dim1,dim2,dim3,dim4)).flat)  #looping through values in xrange(0,len(irange)):      #setting pool      pool = pool(initializer=initpool,initargs=(misfit,),processes=20)      p = [pool.apply_async(testfunc,args=(somevals,j)) j in xrange(0,len(jrange))]      pool.close()      pool.join()  print(misfit[0]) 

note that when set array, must named same variable set in initpool, @ least when tested it.

this isn't best way works , other people might find use it!