I have developed a basic Python Library to provide abstraction to parallel programming. This library is based on in built-in library
multiprocessing in python and 3rd party library
ray. In this blog post I will explain in details about the implementations in addition to existing documentations.
CustomMP is the abstraction of
multiprocessing library with a
SharedList is a generalization of commonly used
Manager, pair of
Queue to keep track of task and results.
SharedList implementation take care of this, thus avoid redundant implementation of same
multiprocessing structure each and every time of implementation. Motivation for this implementation is based on the discussion I had in stackoverflow regarding proper way to implement
Sample Use Case of CustomMP
from pyparallel.CustomMP import CMPSystem limit = 4 args = (1,) cmp_sys = CMPSystem(limit) cmp_sys.add_proc(func=child_func, args=(args,)) contents = cmp_sys.run() print(len(contents))