mirror of
https://github.com/autistic-symposium/web3-starter-py.git
synced 2025-05-17 14:10:21 -04:00
Add some definitions for parallelism in Python (#24)
This commit is contained in:
commit
09ea5f12cf
1 changed files with 34 additions and 2 deletions
|
@ -1,3 +1,35 @@
|
||||||
### Concurrence in Python
|
## Concurrency and Parallelism in Python
|
||||||
|
|
||||||
Examples for my Medium article: [Python + Concurrence: A Mnemonic Guide🚦](https://medium.com/python-for-the-utopian/python-concurrence-a-mnemonic-guide-7304867cbfb7).
|
|
||||||
|
### Threading
|
||||||
|
|
||||||
|
* Threading is a feature usually provided by the operating system.
|
||||||
|
* Threads are lighter than processes, and share the same memory space.
|
||||||
|
* With threading, concurrency is achieved using multiple threads, but due to the GIL only one thread can be running at a time.
|
||||||
|
* If your code is IO-heavy (like HTTP requests), then multithreading will still probably speed up your code.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
#### Multi-processing
|
||||||
|
|
||||||
|
* In multiprocessing, the original process is forked process into multiple child processes bypassing the GIL.
|
||||||
|
* Each child process will have a copy of the entire program's memory.
|
||||||
|
* If your code is performing a CPU bound task, such as decompressing gzip files, using the threading module will result in a slower execution time. For CPU bound tasks and truly parallel execution, use the multiprocessing module.
|
||||||
|
* Higher memory overhead than threading.
|
||||||
|
|
||||||
|
|
||||||
|
### RQ: queueing jobs
|
||||||
|
|
||||||
|
* [RQ](https://python-rq.org/) is aimple but powerful library.
|
||||||
|
* You first enqueue a function and its arguments using the library. This pickles the function call representation, which is then appended to a Redis list.
|
||||||
|
|
||||||
|
|
||||||
|
### Celery: queueing jobs
|
||||||
|
|
||||||
|
* Celery is one of the most popular background job managers in the Python world.
|
||||||
|
* Compatible with several message brokers like RabbitMQ or Redis and can act as both producer and consumer.
|
||||||
|
* Asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operations but supports scheduling as well.
|
||||||
|
|
||||||
|
### concurrent.futures
|
||||||
|
|
||||||
|
* Using a concurrent.futures.ThreadPoolExecutor makes the Python threading example code almost identical to the multiprocessing module.
|
||||||
|
|
Loading…
Add table
Add a link
Reference in a new issue