python 3.9.9 x64 去除GIL（全局解释器锁）版——让多线程不再鸡肋
<i class="pstatus"> 本帖最后由 xqyqx 于 2022-5-10 16:03 编辑 </i><br /><br />
<font size="6">0x00 前言</font><br />
<font size="2">最近玩python时想实现多进程多线程混合运行，了解到了GIL（全局解释器锁）的存在，关于GIL，可以参见下面的官方文档</font><br />
https://docs.python.org/zh-cn/3/glossary.html?highlight=gil#term-global-interpreter-lock<br />
也就是说，python的多线程并不能实现真正意义上的并行，虽然多进程没有GIL的问题，但是创建进程的开销、内存占用以及通信效率都是多进程的问题（更何况多线程多进程我全都想要<img src="https://www.52pojie.org/static/image/smiley/default/4.gif" smilieid="918" border="0" alt="" />），所以我开始尝试修改Cpython的源码以去除GIL，但未果。<br />
后来某次在GitHub上发现了一个去除GIL的项目，并且被认为“有希望在未来几年里真正进入 CPython”（项目地址：https://github.com/colesbury/nogil），本版python即基于此源码编译<br />
<font size="6">0x01 过程</font><br />
<font size="2">下载源码之后，查阅相关资料发现python源码可在Linux/Unix上直接编译运行（Linux/Unix的朋友可以直接去那个项目上git clone然后make就行了），但Windows上的编译却十分麻烦（主要是坑太多了），我便萌发了将其制作为安装包形式的想法。</font><br />
<font size="2">查阅官方文档得知编译python源码需要VS2017环境，安装之后进入PCBuild文件夹，先下载依赖项，之后运行build.bat</font><br />
<font size="2">第一次编译就报错了，后来经过多方排查后发现是该项目的作者使用了仅能运行于64位系统上的API函数，32位的编译就只能放弃了，执行命令：.\build.bat -p x64</font><br />
<font size="2">编译完成，运行根目录下的python.bat即可打开python命令行</font><br />
<font size="2">运行项目作者所修改版本中特有的命令：</font><font style="color:rgb(36, 41, 47)"><font style="background-color:transparent"><font face="ui-monospace, SFMono-Regular, &quot;"><font style="font-size:13.6px">import sys; print(sys.flags.nogil)</font></font></font></font>，无报错，编译成功<br />
<ignore_js_op>
<div class="tip tip_4 aimg_tip" id="aimg_2520855_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>2022-05-10_132141.png</strong> <em class="xg1">(55.28 KB, 下载次数: 0)</em></p>
<p>
<a href="https://www.52pojie.cn/forum.php?mod=attachment&aid=MjUyMDg1NXwwMzQ3MmZmNHwxNjUyNTUzMDU3fDB8MTYzNDEwNg%3D%3D¬humb=yes" target="_blank">下载附件</a>
</p>
<p class="xg1 y">2022-5-10 13:22 上传</p>
</div>
<div class="tip_horn"></div>
</div>
</ignore_js_op>
<br />
但是这虽然能在本机上使用，但不便于分发，源码中包含了太多不必要的文件，且文件位置与发行版python不符，因此需要将其做成msi安装包<br />
制作msi安装包过程中的坑特别多，每次编译了半个小时左右就突然给你报错，网上关于这方面的信息少之又少，因此只能自己手动排查，排查过程这里就不再赘述<br />
最终在.\PCBuild\AMD64\en-us目录下生成了安装包与embed包（解压即用）<br />
编译完成留念：<br />
<ignore_js_op>
<div class="tip tip_4 aimg_tip" id="aimg_2520937_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>2022-05-08_134205.png</strong> <em class="xg1">(351.62 KB, 下载次数: 0)</em></p>
<p>
<a href="https://www.52pojie.cn/forum.php?mod=attachment&aid=MjUyMDkzN3wyNmIzYzI2NXwxNjUyNTUzMDU3fDB8MTYzNDEwNg%3D%3D¬humb=yes" target="_blank">下载附件</a>
</p>
<p class="xg1 y">2022-5-10 16:03 上传</p>
</div>
<div class="tip_horn"></div>
</div>
</ignore_js_op>
<br />
<font size="6"><br />
</font><br />
<font size="6">0x02测试</font><br />
<font size="2">安装过程挺顺利的，一次成功</font><br />
<ignore_js_op>
<div class="tip tip_4 aimg_tip" id="aimg_2520857_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>2022-05-08_214951.png</strong> <em class="xg1">(119.08 KB, 下载次数: 0)</em></p>
<p>
<a href="https://www.52pojie.cn/forum.php?mod=attachment&aid=MjUyMDg1N3w5MTU3NWZiZnwxNjUyNTUzMDU3fDB8MTYzNDEwNg%3D%3D¬humb=yes" target="_blank">下载附件</a>
</p>
<p class="xg1 y">2022-5-10 13:28 上传</p>
</div>
<div class="tip_horn"></div>
</div>
</ignore_js_op>
<br />
<ignore_js_op>
<div class="tip tip_4 aimg_tip" id="aimg_2520856_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>2022-05-08_134235.png</strong> <em class="xg1">(104.87 KB, 下载次数: 0)</em></p>
<p>
<a href="https://www.52pojie.cn/forum.php?mod=attachment&aid=MjUyMDg1NnxlZTIwZTQxZXwxNjUyNTUzMDU3fDB8MTYzNDEwNg%3D%3D¬humb=yes" target="_blank">下载附件</a>
</p>
<p class="xg1 y">2022-5-10 13:28 上传</p>
</div>
<div class="tip_horn"></div>
</div>
</ignore_js_op>
<br />
<ignore_js_op>
<div class="tip tip_4 aimg_tip" id="aimg_2520858_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>2022-05-08_215122.png</strong> <em class="xg1">(110.78 KB, 下载次数: 0)</em></p>
<p>
<a href="https://www.52pojie.cn/forum.php?mod=attachment&aid=MjUyMDg1OHxlNDVjMzhhN3wxNjUyNTUzMDU3fDB8MTYzNDEwNg%3D%3D¬humb=yes" target="_blank">下载附件</a>
</p>
<p class="xg1 y">2022-5-10 13:28 上传</p>
</div>
<div class="tip_horn"></div>
</div>
</ignore_js_op>
<br />
<br />
<br />
全家福：<br />
<ignore_js_op>
<div class="tip tip_4 aimg_tip" id="aimg_2520859_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>2022-05-10_132958.png</strong> <em class="xg1">(283.78 KB, 下载次数: 0)</em></p>
<p>
<a href="https://www.52pojie.cn/forum.php?mod=attachment&aid=MjUyMDg1OXxjOTUzMDdlMnwxNjUyNTUzMDU3fDB8MTYzNDEwNg%3D%3D¬humb=yes" target="_blank">下载附件</a>
</p>
<p class="xg1 y">2022-5-10 13:30 上传</p>
</div>
<div class="tip_horn"></div>
</div>
</ignore_js_op>
<br />
<ignore_js_op>
<div class="tip tip_4 aimg_tip" id="aimg_2520860_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>2022-05-10_133057.png</strong> <em class="xg1">(56.88 KB, 下载次数: 0)</em></p>
<p>
<a href="https://www.52pojie.cn/forum.php?mod=attachment&aid=MjUyMDg2MHwyNThiYjY1NHwxNjUyNTUzMDU3fDB8MTYzNDEwNg%3D%3D¬humb=yes" target="_blank">下载附件</a>
</p>
<p class="xg1 y">2022-5-10 13:31 上传</p>
</div>
<div class="tip_horn"></div>
</div>
</ignore_js_op>
<br />
<ignore_js_op>
<div class="tip tip_4 aimg_tip" id="aimg_2520861_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>2022-05-10_133219.png</strong> <em class="xg1">(114.26 KB, 下载次数: 0)</em></p>
<p>
<a href="https://www.52pojie.cn/forum.php?mod=attachment&aid=MjUyMDg2MXw1NzQ0M2EyM3wxNjUyNTUzMDU3fDB8MTYzNDEwNg%3D%3D¬humb=yes" target="_blank">下载附件</a>
</p>
<p class="xg1 y">2022-5-10 13:32 上传</p>
</div>
<div class="tip_horn"></div>
</div>
</ignore_js_op>
<br />
<ignore_js_op>
<div class="tip tip_4 aimg_tip" id="aimg_2520862_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>2022-05-10_133315.png</strong> <em class="xg1">(206.52 KB, 下载次数: 0)</em></p>
<p>
<a href="https://www.52pojie.cn/forum.php?mod=attachment&aid=MjUyMDg2MnxkY2EyYmUwN3wxNjUyNTUzMDU3fDB8MTYzNDEwNg%3D%3D¬humb=yes" target="_blank">下载附件</a>
</p>
<p class="xg1 y">2022-5-10 13:33 上传</p>
</div>
<div class="tip_horn"></div>
</div>
</ignore_js_op>
<br />
<br />
<br />
接下来我们来验证其是否真正去除了GIL<br />
测试代码如下（多线程跑死循环）：<br />
<div style="padding:15px 0;"><div style="font-size:12px;"> <em class="viewsource" style="cursor:pointer;font-size:12px;color:#369 !important;">纯文本查看</em> <em class="copycode" style="cursor:pointer;font-size:12px;color:#369 !important;">复制代码</em></div><pre class="brush: python; gutter: true">from threading import Thread
import sys
def running():
while True:
pass
return True
def main():
thread_array = {}
for tid in range(8):
t = Thread(target=running)
t.start()
thread_array = t
for i in range(8):
thread_array.join()
if __name__ == '__main__':
main()
</pre></div><br />
测试结果如下（对比测试环境为python3.9.5（即normal），测试时系统环境保持不变）：<br />
3.9.9-nogil：<br />
<ignore_js_op>
<div class="tip tip_4 aimg_tip" id="aimg_2520864_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>2022-05-08_202342.png</strong> <em class="xg1">(50.2 KB, 下载次数: 0)</em></p>
<p>
<a href="https://www.52pojie.cn/forum.php?mod=attachment&aid=MjUyMDg2NHw2NjkyNWM2MHwxNjUyNTUzMDU3fDB8MTYzNDEwNg%3D%3D¬humb=yes" target="_blank">下载附件</a>
</p>
<p class="xg1 y">2022-5-10 13:37 上传</p>
</div>
<div class="tip_horn"></div>
</div>
</ignore_js_op>
<br />
3.9.5-normal：<br />
<ignore_js_op>
<div class="tip tip_4 aimg_tip" id="aimg_2520865_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>2022-05-08_220625.png</strong> <em class="xg1">(46.78 KB, 下载次数: 0)</em></p>
<p>
<a href="https://www.52pojie.cn/forum.php?mod=attachment&aid=MjUyMDg2NXw4Mjc1MTA1ZXwxNjUyNTUzMDU3fDB8MTYzNDEwNg%3D%3D¬humb=yes" target="_blank">下载附件</a>
</p>
<p class="xg1 y">2022-5-10 13:37 上传</p>
</div>
<div class="tip_horn"></div>
</div>
</ignore_js_op>
<br />
我的电脑是四核，因此如果带有GIL的话，理论CPU占用率不高于25%，但在nogil版本中cpu占用更是跑满，因此可以验证GIL已成功去除。<br />
<br />
<br />
接下来我们来看一看去除GIL后的性能表现如何<br />
我们用一个计数器来模拟CPU密集型任务：<br />
<div style="padding:15px 0;"><div style="font-size:12px;"> <em class="viewsource" style="cursor:pointer;font-size:12px;color:#369 !important;">纯文本查看</em> <em class="copycode" style="cursor:pointer;font-size:12px;color:#369 !important;">复制代码</em></div><pre class="brush: python; gutter: true">from threading import Thread
import time
import sys
def my_counter():
i = 0
for _ in range(100000000):
i = i + 1
return True
def main():
thread_array = {}
start_time = time.time()
for tid in range(2):
t = Thread(target=my_counter)
t.start()
thread_array = t
for i in range(2):
thread_array.join()
end_time = time.time()
print("Total time: {}".format(end_time - start_time))
if __name__ == '__main__':
main()
</pre></div><br />
结果如下：<br />
normal单线程：<br />
<ignore_js_op>
<div class="tip tip_4 aimg_tip" id="aimg_2520870_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>单线程gil.png</strong> <em class="xg1">(49.07 KB, 下载次数: 0)</em></p>
<p>
<a href="https://www.52pojie.cn/forum.php?mod=attachment&aid=MjUyMDg3MHw3NzgyMmNmMHwxNjUyNTUzMDU3fDB8MTYzNDEwNg%3D%3D¬humb=yes" target="_blank">下载附件</a>
</p>
<p class="xg1 y">2022-5-10 13:42 上传</p>
</div>
<div class="tip_horn"></div>
</div>
</ignore_js_op>
<br />
nogil单线程：<br />
<ignore_js_op>
<div class="tip tip_4 aimg_tip" id="aimg_2520871_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>单线程nogil.png</strong> <em class="xg1">(56.47 KB, 下载次数: 0)</em></p>
<p>
<a href="https://www.52pojie.cn/forum.php?mod=attachment&aid=MjUyMDg3MXwzZTkzMTQzMnwxNjUyNTUzMDU3fDB8MTYzNDEwNg%3D%3D¬humb=yes" target="_blank">下载附件</a>
</p>
<p class="xg1 y">2022-5-10 13:42 上传</p>
</div>
<div class="tip_horn"></div>
</div>
</ignore_js_op>
<br />
normal多线程：<br />
<ignore_js_op>
<div class="tip tip_4 aimg_tip" id="aimg_2520872_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>多线程gil.png</strong> <em class="xg1">(55.84 KB, 下载次数: 0)</em></p>
<p>
<a href="https://www.52pojie.cn/forum.php?mod=attachment&aid=MjUyMDg3Mnw2ZGExNjdmM3wxNjUyNTUzMDU3fDB8MTYzNDEwNg%3D%3D¬humb=yes" target="_blank">下载附件</a>
</p>
<p class="xg1 y">2022-5-10 13:42 上传</p>
</div>
<div class="tip_horn"></div>
</div>
</ignore_js_op>
<br />
nogil多线程：<br />
<ignore_js_op>
<div class="tip tip_4 aimg_tip" id="aimg_2520873_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>多线程nogil.png</strong> <em class="xg1">(56.92 KB, 下载次数: 0)</em></p>
<p>
<a href="https://www.52pojie.cn/forum.php?mod=attachment&aid=MjUyMDg3M3xhNzRhMWI3NnwxNjUyNTUzMDU3fDB8MTYzNDEwNg%3D%3D¬humb=yes" target="_blank">下载附件</a>
</p>
<p class="xg1 y">2022-5-10 13:43 上传</p>
</div>
<div class="tip_horn"></div>
</div>
</ignore_js_op>
<br />
<br />
结果排名：nogil多线程>normal单线程>normal多线程>nogil单线程<br />
可见去除GIL后多线程性能确实增加，但以单线程性能下降作为代价<br />
<br />
再来用一个多任务测试（斐波那契、累加、阶乘，都运用了递归运算）：<br />
<div style="padding:15px 0;"><div style="font-size:12px;"> <em class="viewsource" style="cursor:pointer;font-size:12px;color:#369 !important;">纯文本查看</em> <em class="copycode" style="cursor:pointer;font-size:12px;color:#369 !important;">复制代码</em></div><pre class="brush: python; gutter: true">import threading
import time
class MyThread(threading.Thread):
def __init__(self, func, args, name=''):
threading.Thread.__init__(self)
self.name = name
self.func = func
self.args = args
def getResult(self):
return self.res
def run(self):
self.res = self.func(*self.args)
def fib(x):
if x < 2:
return 1
return (fib(x - 2) + fib(x - 1))
def fac(x):
if x < 2:
return 1
return (x * fac(x - 1))
def sum(x):
if x < 2:
return 1
return (x + sum(x - 1))
funcs =
n = 35
def main():
nfuncs = range(len(funcs))
print('*** SINGLE THREAD')
start_time = time.time()
for i1 in range(3):
for i in nfuncs:
print(funcs(n))
end_time = time.time()
print("Total time: ",format(end_time - start_time))
print('\n*** MULTIPLE THREADS')
start_time = time.time()
threads = []
for i in nfuncs:
t1 = MyThread(funcs, (n, ), funcs.__name__)
t2 = MyThread(funcs, (n, ), funcs.__name__)
t3 = MyThread(funcs, (n, ), funcs.__name__)
threads.append(t1)
threads.append(t2)
threads.append(t3)
for i in range(9):
threads.start()
for i in range(9):
threads.join()
print(threads.getResult())
end_time = time.time()
print("Total time: ",format(end_time - start_time))
print('all DONE')
if __name__ == '__main__':
main()
</pre></div><br />
<br />
结果如下：<br />
normal：<br />
<ignore_js_op>
<div class="tip tip_4 aimg_tip" id="aimg_2520884_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>gil多任务.png</strong> <em class="xg1">(88.39 KB, 下载次数: 0)</em></p>
<p>
<a href="https://www.52pojie.cn/forum.php?mod=attachment&aid=MjUyMDg4NHwxZWRlNTM2OHwxNjUyNTUzMDU3fDB8MTYzNDEwNg%3D%3D¬humb=yes" target="_blank">下载附件</a>
</p>
<p class="xg1 y">2022-5-10 13:47 上传</p>
</div>
<div class="tip_horn"></div>
</div>
</ignore_js_op>
<br />
nogil：<br />
<ignore_js_op>
<div class="tip tip_4 aimg_tip" id="aimg_2520885_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>nogil多任务.png</strong> <em class="xg1">(95.95 KB, 下载次数: 0)</em></p>
<p>
<a href="https://www.52pojie.cn/forum.php?mod=attachment&aid=MjUyMDg4NXwyZjQ5YzMzOXwxNjUyNTUzMDU3fDB8MTYzNDEwNg%3D%3D¬humb=yes" target="_blank">下载附件</a>
</p>
<p class="xg1 y">2022-5-10 13:48 上传</p>
</div>
<div class="tip_horn"></div>
</div>
</ignore_js_op>
<br />
<br />
可见在递归上nogil不管是单线程还是多线程都吊打normal<br />
<br />
<br />
<br />
总的来说如果使用nogil版的python解释器的话，在其中运用多线程会获得不错的性能，但同时也应当在编程中注意避免使用单线程，还要注意线程安全问题<br />
<br />
<font size="6">0x03 结果</font><br />
<font size="2">两个文件，exe为安装包，压缩包为解压即用</font><br />
<font size="2">腾讯哈勃扫描结果：</font><br />
<font size="2">https://habo.qq.com/file/showdetail?md5=ae9b7507e2575b7c1d8f87354819b65f</font><br />
<font size="2">https://habo.qq.com/file/showdetail?md5=c6ab002a9e353cc742add4f4cbb147d4</font><br />
<font size="6"><font color="#ff0000">仅供测试所用，请勿用于实际生产环境</font></font><br />
<ignore_js_op>
<img src="https://s1www.52pojie.org/2022/05/15/text.gif" border="0" class="vm" alt="" />
<span style="white-space: nowrap" id="attach_2520889" onmouseover="showMenu({'ctrlid':this.id,'pos':'12'})">
<a href="https://www.52pojie.cn/forum.php?mod=attachment&aid=MjUyMDg4OXw3MjJjYWU1ZnwxNjUyNTUzMDU3fDB8MTYzNDEwNg%3D%3D" target="_blank">下载地址.txt</a>
<em class="xg1">(150 Bytes, 下载次数: 56)</em>
</span>
<div class="tip tip_4" id="attach_2520889_menu" style="position: absolute; display: none" disautofocus="true">
<div class="tip_c xs0">
<div class="y">2022-5-10 13:58 上传</div>
点击文件名下载附件
<br />下载积分: 吾爱币 -1 CB
</div>
<div class="tip_horn"></div>
</div>
</ignore_js_op>
<div class="quote"><blockquote><font size="2"><a href="https://www.52pojie.cn/forum.php?mod=redirect&goto=findpost&pid=42633955&ptid=1634106" target="_blank"><font color="#999999">zeroxia 发表于 2022-5-12 21:34</font></a></font><br />
第一个测试，normal单线程和nogil多线程，结果都是12秒多。没区别啊。</blockquote></div><br />
第一个测试中只开了两个线程，体现不出多核的优点，仅仅是为了表明去除GIL后单线程性能有所降低 楼主真是牛人啊<img src="https://www.52pojie.org/static/image/smiley/default/42.gif" smilieid="921" border="0" alt="" />，这些流行的语言都会搞，大神， 电脑cpu终结者吗？ 感谢分享 感谢楼主的慷概分享<br />
感谢楼主分享 谢谢分享 感谢楼主分享 这个版本下载来试一下，还没有研究这么深入。
页:
[1]
2