#89 A tenacious episode that won't give up
Python Bytes - Podcast tekijän mukaan Michael Kennedy and Brian Okken - Maanantaisin
Kategoriat:
Python Bytes 89
Sponsored by Datadog -- pythonbytes.fm/datadog
Brian #1: tenacity
- “Tenacity is a general-purpose retrying library to simplify the task of adding retry behavior to just about anything.”
- Example (Also, nice Trollhunters reference):
import random
from tenacity import retry
@retry
def do_something_unreliable():
if random.randint(0, 10) > 1:
raise IOError("Broken sauce, everything is hosed!!!")
else:
return "Awesome sauce!" # Toby says this frequently
print(do_something_unreliable())
- Features:
- Generic Decorator API
- Specify stop condition (i.e. limit by number of attempts)
- Specify wait condition (i.e. exponential backoff sleeping between attempts)
- Customize retrying on Exceptions
- Customize retrying on expected returned result
- Retry on coroutines
Michael #2: Why is Python so slow?
- Answer this question: When Python completes a comparable application 2–10x slower than another language, why is it slow and can’t we make it faster?
- Here are the top theories:
- “It’s the GIL (Global Interpreter Lock)”
- “It’s because its interpreted and not compiled”
- “It’s because its a dynamically typed language”
- “It’s the GIL”
- Modern computers come with CPU’s that have multiple cores
- For web apps, it might not matter (e.g. https://training.talkpython.fm/ has 16 worker processes, https://talkpython.fm/ has 8 workers)
- “It’s because its an interpreted language”
- I hear this a lot and I find it a gross-simplification of the way CPython actually works.
- JIT vs. NonJIT is interesting (startup time too)
- “It’s because its a dynamically typed language”
- In a “Statically-Typed” language, you have to specify the type of a variable when it is declared. Those would include C, C++, Java, C#, Go.
- In a dynamically-typed language, there are still the concept of types, but the type of a variable is dynamic.
- Not having to declare the type isn’t what makes Python slow
- It’s this design that makes it incredibly hard to optimize Python.
- Conclusion
- Python is primarily slow because of its dynamic nature and versatility. It can be used as a tool for all sorts of problems, where more optimized and faster alternatives are probably available.
Brian #3: Keynoting with Mu
- David Beazley gave his EuroPython talk/demo “Die Threads” using Mu.
- Article also notes that simple tools are great not just for learning, but for teaching, as the extra clutter of a full power editor doesn’t distract too much.
Michael #4: A multi-core Python HTTP server (much) faster than Go (spoiler: Cython)
- Exploring the question, “So, I’ve heard Python is slow… is it?”
- A multi-core Python HTTP server that is about 40% to 110% faster than Go can be built by relying on the Cython language and LWAN C library.
- Just a proof of concept validates the possibility of high performance system programming in the Cython language.
- Primarily interesting as a highlight of Cython
- Cython is both an optimizing static compiler and a hybrid language. It mainly gives the ability to:
- write Python code that can call back and forth from and to C/C++;
- add static typing using C declarations to Python code in order to boost performance;
- release the GIL in some code sections.
- Cython generates very efficient C code, which is then compiled into a module that Python can import. So it is an ideal language for wrapping external C libraries, and for developing C modules that speed up the execution of Python code.
- However, all experiments we are aware that rely on Cython for system programming fail short in at least two ways:
- as soon as some Python code is invoked (as opposed to pure Cython
cdef
code), performance degrades by one or two orders of magnitude; - benchmarks are most of the time provided for single core execution only, which is somehow unfair considering Golang's ability to scale up on multiple cores.
- as soon as some Python code is invoked (as opposed to pure Cython
Brian #5: PyCharm 2018.2 beefs up pytest support
- Honestly, I’m super excited about this release to help my team navigate to all of the fixtures I create on a regular basis.
- This is the release I’ve been waiting for.
- I can now fully utilize the power of pytest from PyCharm
- Here’s the few things that were missing that now work great:
- Autocomplete fixtures from various sources
- Quick documentation and navigation to fixtures
- Renaming a fixture from either the definition or a usage
- Support for pytest’s parametrize
- See also: PyCharm 2018.2 and pytest Fixtures
- But if you really want to understand fixtures quickly, read chapters 3 and 4 of the pytest book.
Michael #6: XAR for Facebook
- XAR lets you package many files into a single self-contained executable file. This makes it easy to distribute and install.
- A
.xar
file is a read-only file system image which, when mounted, looks like a regular directory to user-space programs. This requires a one-time installation of a driver for this file system (SquashFS). - There are two primary use cases for XAR files.
- Simply collecting a number of files for automatic, atomic mounting somewhere on the filesystem.
- By making the XAR file executable and using the xarexec helper, a XAR becomes a self-contained package of executable code and its data. A popular example is Python application archives that include all Python source code files, as well as native shared libraries, configuration files, other data.
- Advantages of XAR for Python usage
- SquashFS looks like regular files on disk to Python. This lets it use regular imports which are better supported by CPython.
- SquashFS looks like regular files to your application, too. You don't need to use
pkg_resources
or other tricks to access data files in your package. - SquashFS with Zstandard compression saves disk space, also compared to a ZIP file.
- SquashFS doesn't require unpacking of
.so
files to a temporary location like ZIP files do. - SquashFS is faster to start up than unpacking a ZIP file. You only need to mount the file system once. Subsequent calls to your application will reuse the existing mount.
- SquashFS only decompresses the pages that are used by the application, and decompressed pages are cached in the page cache.
- SquashFS is read-only so the integrity of your application is guaranteed compared to using virtualenvs or unpacking to a temporary directory.
- Performance is interesting too
Extras:
Brian:
- numpy 1.15.0 just released recently. Switched testing to pytest.
Michael:
- SciPy 2018 videos are out
- PyOhio 2018 videos are out
- Call for papers at PyCon Canada in Toronto
- PyBay 2018 conference in a few weeks
- My latest course, Building data-driven web apps with Pyramid and SQLAlchemy, is out!