Build and Install Python 3.7 on Ubuntu/Debian

At the time of this writing, Python 3.7 doesn't come bundled with Ubuntu 18.04 Bionic and Debian 9 Stretch (it will come bundled with Debian 10 Buster in mid 2019). This tutorial demonstrates how to build Python 3.7 from source on these distributions.

Install Prerequisites

We begin by installing libraries required to build Python. We will install build dependencies of python3 in Ubuntu/Debian:

# apt-get build-dep -y python3

The command will install basic build dependencies but not other development libraries that are needed to build Python. Install development libraries as follows:

# apt-get install --no-install-recommends -y build-essential zlib1g-dev libbz2-dev liblzma-dev libncurses5-dev libsqlite3-dev libssl-dev tk-dev libgdbm-dev libreadline-dev default-libmysqlclient-dev libffi-dev libdb-dev libxml2-dev libxslt1-dev libmpdec-dev uuid-dev

Download Python Source

At the time of this writing Python 3.7.3 is released. Download and extract the tarball:

# cd /usr/src
# wget https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tgz
# tar zxf Python-3.7.3.tgz
# cd Python-3.7.3/

** Building Python**

A lot of tutorials run the configure script only with --enable-optimizations and move on. This is not how I am going to do it. I'd like to build Python with the same configuration parameters that the Python bundled with the distributions is built with. This can be know using the following Python single-liner:

# python3 -c "import sysconfig; print(sysconfig.get_config_var('CONFIG_ARGS'))"

The single-liner yields this output:

'--enable-shared' '--prefix=/usr' '--enable-ipv6' '--enable-loadable-sqlite-extensions' '--with-dbmliborder=bdb:gdbm' '--with-computed-gotos' '--without-ensurepip' '--with-system-expat' '--with-system-libmpdec' '--with-system-ffi' 'CC=x86_64-linux-gnu-gcc' 'CFLAGS=-g   -fstack-protector-strong -Wformat -Werror=format-security ' 'LDFLAGS=-Wl,-Bsymbolic-functions  -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'

Lets streamline the above output and convert it to a ./configure command.

# CC="x86_64-linux-gnu-gcc" CFLAGS="-g -fstack-protector-strong -Wformat -Werror=format-security" LDFLAGS="-Wl,-Bsymbolic-functions -Wl,-z,relro" CPPFLAGS="-Wdate-time -D_FORTIFY_SOURCE=2" ./configure --prefix=/usr --enable-shared --enable-ipv6 --enable-loadable-sqlite-extensions --with-dbmliborder=bdb:gdbm --with-computed-gotos --with-system-expat --with-system-libmpdec --with-system-ffi --without-ensurepip

Now, I am not going to go in long descriptions on what each option means but in summary Python bundled with the distributions is build with:

• Stack overflow protection (-fstack-protector-strong).
• GCC compiler warning options (-Wformat -Werror=format-security -Wdate-time).
• GNU linker options (everything in LDFLAGS)
• More compiler flags (everything in CPPFLAGS)
• Shared Python library (--enable-shared).
• IPv6 support (--enable-ipv6).
• Support for loadable extensions in _sqlite module (--enable-loadable-sqlite-extensions).
• Order in which Python will check for db backends (--with-dbmliborder=bdb:gdbm). This options refers different implementations of dbm libraries.
• Enabling computed gotos (--with-computed-gotos)
• System expat (an XML parsing) library pre-installed (--with-system-expat).
• _decimal module using an installed libmpdec library (--with-system-libmpdec).
• _ctypes module using an installed ffi library (--with-system-ffi).
• Do not install pip bundled with source (--without-ensurepip).

Compile options will be different on Debian Stretch. Thats because Stretch and Bionic have different versions of Python 3. Some build parameters in Python 3.5 (that comes with Stretch) have been deprecated in Python 3.6 and onwards. However, the method used to build Python will remain the same.

You can change --prefix which will install Python wherever you want on the system. By the default there are no optimizations enabled. Hence I will add --enable-optimizations to the configure command and the final command we will run is:

# CC="x86_64-linux-gnu-gcc" CFLAGS="-g -fstack-protector-strong -Wformat -Werror=format-security" LDFLAGS="-Wl,-Bsymbolic-functions -Wl,-z,relro" CPPFLAGS="-Wdate-time -D_FORTIFY_SOURCE=2" ./configure --prefix=/usr/local --enable-shared --enable-ipv6 --enable-loadable-sqlite-extensions --with-dbmliborder=bdb:gdbm --with-computed-gotos --with-system-expat --with-system-libmpdec --with-system-ffi --without-ensurepip --enable-optimizations

The run make and make install:

# make
# make install

With --enable-optimizations parameter, Python will take more time to build. Once make install finishes, Python is installed. Before we can use it, we need to run ldconfig:

# echo '/usr/local/lib' > /etc/ld.so.conf.d/compile_installed.conf
# ldconfig

ldconfig creates the necessary links and cache to the most recent shared libraries found in the directories that are defined in configuration file /etc/ld.so.conf or files under /etc/ld.so.conf.d/. Note that if you change the prefix in the configure command, you will have to add the path of lib directory under the new prefix to /etc/ld.so.conf.d/compile_installed.conf (you can change the name of this file to whatever you like).

Install pip

We could install pip by replacing --without-ensurepip with --with-ensurepip. However, since we have --without-ensurepip in our invocation of the configure script, pip is not installed and we will install pip as follows:

# curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
# python3.7 get-pip.py

This will install pip3.7 in /usr/local/bin.

Virtual Environment

With this build you can create a virtual environment using pyvenv-3.7:

# pyvenv-3.7 /root/venv
# source /root/venv/bin/activate

Alternatively, you can also install virtualenv for Python 3.7:

# pip3.7 install virtualenv

And then create a virtual environment:

# /usr/local/bin/virtualenv /root/venv

Or you can always use the following command to create a virtual environment:

# python3.7 -m venv /root/venv

Build Using Ansible

I wrote an Ansible role to build Python. It can potentially be used to install any verison of Python from source on an Ubuntu/Debian system. Try it out and let me know what you think.

Tagged as: Python Linux Ubuntu Debian Bionic Xenial Stretch