3種方式實現(xiàn)python多線程并發(fā)處理

Godtoy 發(fā)布于2019-07-30 18:38 / 1731人閱讀

摘要：對于任務(wù)數(shù)量不端增加的程序，固定線程數(shù)量的線程池是必要的。方法二使用模塊主線程結(jié)束主線程結(jié)束方法三使用模塊參考分組線程池

標(biāo)簽： python奇淫技巧

最優(yōu)線程數(shù)

Ncpu=CPU的數(shù)量

Ucpu=目標(biāo)CPU使用率

W/C=等待時間與計算時間的比率

為保持處理器達(dá)到期望的使用率，最優(yōu)的線程池的大小等于
$$Nthreads=Ncpu*Ucpu*（1+W/C$$

cpu密集型任務(wù)，即$W<

如果希望CPU利用率為100%，則$Nthreads=Ncpu$

IO密集型任務(wù)，即系統(tǒng)大部分時間在跟I/O交互，而這個時間線程不會占用CPU來處理，即在這個時間范圍內(nèi)，可以由其他線程來使用CPU，因而可以多配置一些線程。

混合型任務(wù)，二者都占有一定的時間

線城池

對于任務(wù)數(shù)量不斷增加的程序，每有一個任務(wù)就生成一個線程，最終會導(dǎo)致線程數(shù)量的失控。對于任務(wù)數(shù)量不端增加的程序，固定線程數(shù)量的線程池是必要的。

方法一：使用threadpool模塊

threadpool是一個比較老的模塊了，支持py2 和 py3 。

import threadpool
import time

def sayhello (a):
    print("hello: "+a)
    time.sleep(2)

def main():
    global result
    seed=["a","b","c"]
    start=time.time()
    task_pool=threadpool.ThreadPool(5)
    requests=threadpool.makeRequests(sayhello,seed)
    for req in requests:
        task_pool.putRequest(req)
    task_pool.wait()
    end=time.time()
    time_m = end-start
    print("time: "+str(time_m))
    start1=time.time()
    for each in seed:
        sayhello(each)
    end1=time.time()
    print("time1: "+str(end1-start1))

if __name__ == "__main__":
    main(

方法二：使用concurrent.futures模塊

from concurrent.futures import ThreadPoolExecutor
import time

import time
from concurrent.futures import ThreadPoolExecutor, wait, as_completed

ll = []
def sayhello(a):
    print("hello: "+a)
    ll.append(a)
    time.sleep(0.8)

def main():
    seed=["a","b","c","e","f","g","h"]
    start1=time.time()
    for each in seed:
        sayhello(each)
    end1=time.time()
    print("time1: "+str(end1-start1))
    start2=time.time()
    with ThreadPoolExecutor(2) as executor:
        for each in seed:
            executor.submit(sayhello,each)
    end2=time.time()
    print("time2: "+str(end2-start2))

def main2():
    seed = ["a", "b", "c", "e", "f", "g", "h"]
    executor = ThreadPoolExecutor(max_workers=10)
    f_list = []
    for each in seed:
        future = executor.submit(sayhello, each)
        f_list.append(future)
    wait(f_list)
    print(ll)
    print("主線程結(jié)束")


def main3():
    seed = ["a", "b", "c", "e", "f", "g", "h"]
    with ThreadPoolExecutor(max_workers=2) as executor:
        f_list = []
        for each in seed:
            future = executor.submit(sayhello, each)
            f_list.append(future)
        wait(f_list,return_when="ALL_COMPLETED")
        print(ll)
        print("主線程結(jié)束")

if __name__ == "__main__":
    main3()

方法三：使用vthread模塊

參考：https://pypi.org/project/vthr...

demo1

import vthread
 
@vthread.pool(6)
def some(a,b,c):
    import time;time.sleep(1)
    print(a+b+c)
 
for i in range(10):
    some(i,i,i)

demo2:分組線程池

import vthread
pool_1 = vthread.pool(5,gqueue=1) # open a threadpool with 5 threads named 1
pool_2 = vthread.pool(2,gqueue=2) # open a threadpool with 2 threads named 2

@pool_1
def foolfunc1(num):
    time.sleep(1)
    print(f"foolstring1, test3 foolnumb1:{num}")

@pool_2
def foolfunc2(num):
    time.sleep(1)
    print(f"foolstring2, test3 foolnumb2:{num}")

@pool_2
def foolfunc3(num):
    time.sleep(1)
    print(f"foolstring3, test3 foolnumb3:{num}")

for i in range(10): foolfunc1(i)
for i in range(4): foolfunc2(i)
for i in range(2): foolfunc3(i)