Add some definitions for parallelism in Python

This commit is contained in:
bt3gl 2022-03-28 11:12:30 +01:00 committed by GitHub
parent bf8b24bee8
commit fdf2e5219a
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23

View file

@ -1,3 +1,35 @@
### Concurrence in Python
## Concurrency and Parallelism in Python
Examples for my Medium article: [Python + Concurrence: A Mnemonic Guide🚦](https://medium.com/python-for-the-utopian/python-concurrence-a-mnemonic-guide-7304867cbfb7).
### Threading
* Threading is a feature usually provided by the operating system.
* Threads are lighter than processes, and share the same memory space.
* With threading, concurrency is achieved using multiple threads, but due to the GIL only one thread can be running at a time.
* If your code is IO-heavy (like HTTP requests), then multithreading will still probably speed up your code.
#### Multi-processing
* In multiprocessing, the original process is forked process into multiple child processes bypassing the GIL.
* Each child process will have a copy of the entire program's memory.
* If your code is performing a CPU bound task, such as decompressing gzip files, using the threading module will result in a slower execution time. For CPU bound tasks and truly parallel execution, use the multiprocessing module.
* Higher memory overhead than threading.
### RQ: queueing jobs
* [RQ](https://python-rq.org/) is aimple but powerful library.
* You first enqueue a function and its arguments using the library. This pickles the function call representation, which is then appended to a Redis list.
### Celery: queueing jobs
* Celery is one of the most popular background job managers in the Python world.
* Compatible with several message brokers like RabbitMQ or Redis and can act as both producer and consumer.
* Asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operations but supports scheduling as well.
### concurrent.futures
* Using a concurrent.futures.ThreadPoolExecutor makes the Python threading example code almost identical to the multiprocessing module.