Threading and multiprocessing made easy.
- Free software: Apache-2.0 license
- Documentation: https://lox.readthedocs.io.
- Python >=3.6
Lox provides decorators and synchronization primitives to quickly add concurrency to your projects.
Installation
pip3 install --user lox
For multiprocessing support, install with the optional extra:
pip3 install --user 'lox[multiprocessing]'
Features
- Multithreading: Powerful, intuitive multithreading in just 2 additional lines of code.
- Multiprocessing: Truly parallel function execution with the same interface as multithreading.
- Synchronization: Advanced thread synchronization, communication, and resource management tools.
Todos
- All objects except
lox.processare for threads. These will eventually be multiprocess friendly.
Usage
Easy Multithreading
>>> import lox
>>>
>>> @lox.thread(4) # Will operate with a maximum of 4 threads
... def foo(x,y):
... return x*y
>>> foo(3,4) # normal function calls still work
12
>>> for i in range(5):
... foo.scatter(i, i+1)
-ignore-
>>> # foo is currently being executed in 4 threads
>>> results = foo.gather() # block until results are ready
>>> print(results) # Results are in the same order as scatter() calls
[0, 2, 6, 12, 20]
Or, for example, if you aren't allowed to directly decorate the function you would like multithreaded/multiprocessed, you can just directly invoke the decorator:
>>
>>> def my_func():
... foo_threaded = lox.thread(foo)
... for i in range(5):
... foo_threaded.scatter(i, i + 1)
... results = foo_threaded.gather()
... # foo is currently being executed in default 50 thread executor pool
... return results
...">>>> # Lets say we don't have direct access to this function
... def foo(x, y):
... return x * y
...
>>>
>>> def my_func():
... foo_threaded = lox.thread(foo)
... for i in range(5):
... foo_threaded.scatter(i, i + 1)
... results = foo_threaded.gather()
... # foo is currently being executed in default 50 thread executor pool
... return results
...
... def foo(x, y):
... return x * y
...
>>>
>>> def my_func():
... foo_threaded = lox.thread(foo)
... for i in range(5):
... foo_threaded.scatter(i, i + 1)
... results = foo_threaded.gather()
... # foo is currently being executed in default 50 thread executor pool
... return results
...
This also makes it easier to dynamically control the number of thread/processes in the executor pool. The syntax is a little weird, but this is just explicitly invoking a decorator that has optional arguments:
>>> # Set the number of executer threads to 10
>>> foo_threaded = lox.thread(10)(foo)
>>> foo_threaded = lox.thread(10)(foo)
Easy Multiprocessing
>>> import lox
>>>
>>> @lox.process(4) # Will operate with a pool of 4 processes
... def foo(x, y):
... return x * y
...
>>> foo(3, 4) # normal function calls still work
12
>>> for i in range(5):
... foo.scatter(i, i + 1)
...
-ignore-
>>> # foo is currently being executed in 4 processes
>>> results = foo.gather() # block until results are ready
>>> print(results) # Results are in the same order as scatter() calls
[0, 2, 6, 12, 20]
>>>
>>> @lox.process(4) # Will operate with a pool of 4 processes
... def foo(x, y):
... return x * y
...
>>> foo(3, 4) # normal function calls still work
12
>>> for i in range(5):
... foo.scatter(i, i + 1)
...
-ignore-
>>> # foo is currently being executed in 4 processes
>>> results = foo.gather() # block until results are ready
>>> print(results) # Results are in the same order as scatter() calls
[0, 2, 6, 12, 20]
Progress Bar Support (tqdm)
>>> import lox
>>> from random import random
>>> from time import sleep
>>>
>>> @lox.thread(2)
... def foo(multiplier):
... sleep(multiplier * random())
...
>>> for i in range(10):
>>> foo.scatter(i)
>>> results = foo.gather(tqdm=True)
90%|################################# | 9/10 [00:03<00:00, 1.32it/s]
100%|#######################################| 10/10 [00:06<00:00, 1.46s/it]
>>> from random import random
>>> from time import sleep
>>>
>>> @lox.thread(2)
... def foo(multiplier):
... sleep(multiplier * random())
...
>>> for i in range(10):
>>> foo.scatter(i)
>>> results = foo.gather(tqdm=True)
90%|################################# | 9/10 [00:03<00:00, 1.32it/s]
100%|#######################################| 10/10 [00:06<00:00, 1.46s/it]