From python threading documentation
In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing. However, threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously.
Now I have a thread worker like this
def worker(queue):
queue_full = True
while queue_full:
try:
url = queue.get(False)
w = Wappalyzer(url)
w.analyze()
queue.task_done()
except Queue.Empty:
queue_full = False
Here w.analyze()
doing two things
- Scrape the url using
requests
library - Analyzing the scraped html using
pyv8
javascript library
As far as I know, 1
is I/O bound and 2
is CPU bound.
Does that mean, GIL applied for 2
and my program won't work properly?