At the time of this writing, Python 3.7 doesn't come bundled with Ubuntu 18.04 Bionic and Debian 9 Stretch (it will come bundled with Debian 10 Buster in mid 2019). This tutorial demonstrates how to build Python 3.7 from source on these distributions.
Install Prerequisites
We begin by installing libraries required to build Python. We will install build dependencies of python3 in Ubuntu/Debian:
# apt-get build-dep -y python3
The command will install basic build dependencies but not other development libraries that are needed to build Python. Install development libraries as follows:
# apt-get install --no-install-recommends -y build-essential zlib1g-dev libbz2-dev liblzma-dev libncurses5-dev libsqlite3-dev libssl-dev tk-dev libgdbm-dev libreadline-dev default-libmysqlclient-dev libffi-dev libdb-dev libxml2-dev libxslt1-dev libmpdec-dev uuid-dev
Download Python Source
At the time of this writing Python 3.7.3 is released. Download and extract the tarball:
# cd /usr/src
# wget https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tgz
# tar zxf Python-3.7.3.tgz
# cd Python-3.7.3/
** Building Python**
A lot of tutorials run the configure script only with --enable-optimizations and move on. This is not how I am going to do it. I'd like to build Python with the same configuration parameters that the Python bundled with the distributions is built with. This can be know using the following Python single-liner:
# python3 -c "import sysconfig; print(sysconfig.get_config_var('CONFIG_ARGS'))"
The single-liner yields this output:
'--enable-shared' '--prefix=/usr' '--enable-ipv6' '--enable-loadable-sqlite-extensions' '--with-dbmliborder=bdb:gdbm' '--with-computed-gotos' '--without-ensurepip' '--with-system-expat' '--with-system-libmpdec' '--with-system-ffi' 'CC=x86_64-linux-gnu-gcc' 'CFLAGS=-g -fstack-protector-strong -Wformat -Werror=format-security ' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'
Lets streamline the above output and convert it to a ./configure command.
# CC="x86_64-linux-gnu-gcc" CFLAGS="-g -fstack-protector-strong -Wformat -Werror=format-security" LDFLAGS="-Wl,-Bsymbolic-functions -Wl,-z,relro" CPPFLAGS="-Wdate-time -D_FORTIFY_SOURCE=2" ./configure --prefix=/usr --enable-shared --enable-ipv6 --enable-loadable-sqlite-extensions --with-dbmliborder=bdb:gdbm --with-computed-gotos --with-system-expat --with-system-libmpdec --with-system-ffi --without-ensurepip
Now, I am not going to go in long descriptions on what each option means but in summary Python bundled with the distributions is build with:
- Stack overflow protection (
-fstack-protector-strong). - GCC compiler warning options (
-Wformat -Werror=format-security -Wdate-time). - GNU linker options (everything in
LDFLAGS) - More compiler flags (everything in
CPPFLAGS) - Shared Python library (
--enable-shared). - IPv6 support (
--enable-ipv6). - Support for loadable extensions in _sqlite module (
--enable-loadable-sqlite-extensions). - Order in which Python will check for db backends (
--with-dbmliborder=bdb:gdbm). This options refers different implementations ofdbmlibraries. - Enabling computed gotos (
--with-computed-gotos) - System expat (an XML parsing) library pre-installed (
--with-system-expat). _decimalmodule using an installed libmpdec library (--with-system-libmpdec)._ctypesmodule using an installed ffi library (--with-system-ffi).- Do not install
pipbundled with source (--without-ensurepip).
Compile options will be different on Debian Stretch. Thats because Stretch and Bionic have different versions of Python 3. Some build parameters in Python 3.5 (that comes with Stretch) have been deprecated in Python 3.6 and onwards. However, the method used to build Python will remain the same.
You can change --prefix which will install Python wherever you want on the system. By the default there are no optimizations enabled. Hence I will add --enable-optimizations to the configure command and the final command we will run is:
# CC="x86_64-linux-gnu-gcc" CFLAGS="-g -fstack-protector-strong -Wformat -Werror=format-security" LDFLAGS="-Wl,-Bsymbolic-functions -Wl,-z,relro" CPPFLAGS="-Wdate-time -D_FORTIFY_SOURCE=2" ./configure --prefix=/usr/local --enable-shared --enable-ipv6 --enable-loadable-sqlite-extensions --with-dbmliborder=bdb:gdbm --with-computed-gotos --with-system-expat --with-system-libmpdec --with-system-ffi --without-ensurepip --enable-optimizations
The run make and make install:
# make
# make install
With --enable-optimizations parameter, Python will take more time to build. Once make install finishes, Python is installed. Before we can use it, we need to run ldconfig:
# echo '/usr/local/lib' > /etc/ld.so.conf.d/compile_installed.conf
# ldconfig
ldconfig creates the necessary links and cache to the most recent shared libraries found in the directories that are defined in configuration file /etc/ld.so.conf or files under /etc/ld.so.conf.d/. Note that if you change the prefix in the configure command, you will have to add the path of lib directory under the new prefix to /etc/ld.so.conf.d/compile_installed.conf (you can change the name of this file to whatever you like).
Install pip
We could install pip by replacing --without-ensurepip with --with-ensurepip. However, since we have --without-ensurepip in our invocation of the configure script, pip is not installed and we will install pip as follows:
# curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
# python3.7 get-pip.py
This will install pip3.7 in /usr/local/bin.
Virtual Environment
With this build you can create a virtual environment using pyvenv-3.7:
# pyvenv-3.7 /root/venv
# source /root/venv/bin/activate
Alternatively, you can also install virtualenv for Python 3.7:
# pip3.7 install virtualenv
And then create a virtual environment:
# /usr/local/bin/virtualenv /root/venv
Or you can always use the following command to create a virtual environment:
# python3.7 -m venv /root/venv
Build Using Ansible
I wrote an Ansible role to build Python. It can potentially be used to install any verison of Python from source on an Ubuntu/Debian system. Try it out and let me know what you think.