advertisement

Python Performance: Single-threaded, multi-threaded, and Gevent

33 %
67 %
advertisement
Information about Python Performance: Single-threaded, multi-threaded, and Gevent
Technology

Published on February 27, 2014

Author: emptysquare

Source: slideshare.net

Description

A smatter of performance tips for Python programs.
advertisement

What I Know About Python Performance A. Jesse Jiryu Davis
 http://emptysqua.re
 MongoDB

Agenda • Single-threaded • Multi-threaded • Gevent

Single-Threaded Why is Python slow?
 
 • Everything’s an object
 • Lookup

Everything’s an object def f():! i = 2000! i += 1!

Everything’s an object import dis! dis.dis(f)!

Everything’s an object def f():! i = 2000! i += 1 0 3 6 9 12 13 LOAD_CONST STORE_FAST LOAD_FAST LOAD_CONST INPLACE_ADD! STORE_FAST 1 0 0 2 (2000)! (i)! (i)! (1)! 0 (i)!

Everything’s an object def f():! i = 2000! i += 1 malloc! 0 3 6 9 12 13 LOAD_CONST STORE_FAST LOAD_FAST LOAD_CONST INPLACE_ADD! STORE_FAST 1 0 0 2 (2000)! (i)! malloc! (i)! (1)! 0 (i)!

Everything’s an object 12 seconds for 100M numbers: total = 0! for i in longs:! total += i! print float(total) / len(longs)

Everything’s an object 1 second for 100M numbers: print float(sum(longs)) / len(longs)

Everything’s an object 0.2 seconds for 100M numbers: print numpy.mean(array)

Lookup n = 1! ! def f():! print n!

n = 1! ! def f():! print n! 0 LOAD_GLOBAL 3 PRINT_ITEM! 4 PRINT_NEWLINE hashtable lookup 0 (n)!

n = 1! ! def f(n=n):! print n! 0 LOAD_FAST 3 PRINT_ITEM! 4 PRINT_NEWLINE array offset 0 (n)!

Profiling

Profiling import yappi! ! yappi.start(builtins=True)! my_function()! yappi.stop()! stats = yappi.get_func_stats()! stats.save('yappi.callgrind', type='callgrind')!

KCacheGrind

KCacheGrind

Multi-threaded

Multi-threaded When is the GIL released?
 
 • Every 100 bytecodes
 • Acquiring other locks
 • Network or file operations

Multi-threaded Because: Py_BEGIN_ALLOW_THREADS! outlen = recv(s->sock_fd, cbuf, len);! Py_END_ALLOW_THREADS! socketmodule.c

Multi-threaded Because: Py_BEGIN_ALLOW_THREADS! outlen = recv(s->sock_fd, cbuf, len);! Py_END_ALLOW_THREADS! release the GIL! socketmodule.c

Multi-threaded This makes sense: threads = [! threading.Thread(target=download_a_url())! for _ in range(10)]! ! for t in threads:! t.start()! ! for t in threads:! t.join()!

Multi-threaded This makes no sense: threads = [! threading.Thread(target=do_calculations())! for _ in range(10)]! ! for t in threads:! t.start()! ! for t in threads:! t.join()!

Gevent from gevent import monkey! monkey.patch_all()! ! threads = [! threading.Thread(target=download_a_url())]! ! for t in threads:! t.start()! ! for t in threads:! t.join()! actually a greenlet

GreenletProfiler

Links http://bit.ly/pythonperformance

Add a comment

Related presentations

Presentación que realice en el Evento Nacional de Gobierno Abierto, realizado los ...

In this presentation we will describe our experience developing with a highly dyna...

Presentation to the LITA Forum 7th November 2014 Albuquerque, NM

Un recorrido por los cambios que nos generará el wearabletech en el futuro

Um paralelo entre as novidades & mercado em Wearable Computing e Tecnologias Assis...

Microsoft finally joins the smartwatch and fitness tracker game by introducing the...

Related pages

Gevent | LinkedIn

View 659 Gevent posts, presentations, experts, and more. Get the professional knowledge you need on LinkedIn. LinkedIn Home What is LinkedIn? Join Today
Read more

What is gevent? — gevent 1.2.0.dev0 documentation

What is gevent? ¶ gevent is a ... API that re-uses concepts from the Python ... simpler implementation and better performance. Read why others use gevent ...
Read more

gevent 1.1.1 : Python Package Index - PyPI - the Python ...

... , simpler implementation and better performance. Read why others use gevent and check ... Version 1.1 will be the last gevent series to support Python ...
Read more

gevent 1.0.2 : Python Package Index - PyPI - the Python ...

... simpler implementation and better performance. Read why others use gevent and check out the list of the open source ... get gevent. Install Python 2.5, ...
Read more

Python Performance: Links · GitHub

Python Performance: ... Single-Threaded Why is Python slow? Alex Gaynor on the topic: http://vimeo.com/61044810; ... Multi-Threaded.
Read more

multithreading - Does a multithreaded crawler in Python ...

Basically I'm asking is doing a multi-threaded crawler in python really going to buy me much performance vs single threaded? ... Use gevent. It uses async ...
Read more

Python's Hardest Problem - Jeff Knupp

Python's hardest problem is less ... why does my newly multi-threaded Python program ... GIL with no performance penalty to single threaded Python ...
Read more

Multi Threaded | LinkedIn

View 36069 Multi Threaded posts, presentations, experts, and more. Get the professional knowledge you need on LinkedIn. LinkedIn Home What is LinkedIn?
Read more