Developing in Pythonon Red Hat platforms

Nick CoghlanSenior Software Engineer

Graham DumpletonDeveloper Advocate for OpenShift

June 28th 2016

Using Python on Red Hat Platforms

● Python for Network Services● Python for Applications● Python for System Administration

Current Transitions in the Python Ecosystem

● Migration to Python 3● Modernizing the Python 2.7 Network Security Stack● Defaulting to HTTPS Certificate Verification

Python for Network Services

$ oc new-app https://gitlab.com/osevg/python-django-modwsgi.git--> Found image 772dc19 (4 weeks old) in image stream "python" in project "openshift" under tag "3.4" for "python"

Python 3.4 ---------- Platform for building and running Python 3.4 applications

Tags: builder, python, python34, rh-python34

* The source repository appears to match: python * A source build using source code from https://gitlab.com/osevg/python-django-modwsgi.git will be created * The resulting image will be pushed to image stream "python-django-modwsgi:latest" * This image will be deployed in deployment config "python-django-modwsgi" * Port 8080/tcp will be load balanced by service "python-django-modwsgi" * Other containers can access this service through the hostname "python-django-modwsgi"

--> Creating resources with label app=python-django-modwsgi ... imagestream "python-django-modwsgi" created buildconfig "python-django-modwsgi" created deploymentconfig "python-django-modwsgi" created service "python-django-modwsgi" created--> Success Build scheduled, use 'oc logs -f bc/python-django-modwsgi' to track its progress. Run 'oc status' to view your app.

$ oc expose svc/python-django-modwsgiroute "python-django-modwsgi" exposed

Source to Image (S2I)https://github.com/openshift/source-to-image

$ s2i build https://gitlab.com/osevg/python-django-modwsgi.git centos/python-34-centos7 python-django-modwsgiI0610 10:02:07.422470 84805 docker.go:352] Image "centos/python-34-centos7:latest" not available locally, pulling ...I0610 10:02:36.501637 84805 clone.go:32] Downloading "https://gitlab.com/osevg/python-django-modwsgi.git" ...I0610 10:02:38.831859 84805 install.go:251] Using "assemble" installed from "image:///usr/libexec/s2i/assemble"I0610 10:02:38.831913 84805 install.go:251] Using "run" installed from "image:///usr/libexec/s2i/run"I0610 10:02:38.831943 84805 install.go:251] Using "save-artifacts" installed from "image:///usr/libexec/s2i/save-artifacts"I0610 10:02:38.832495 84805 environment.go:60] Setting 1 environment variables provided by environment file in sources---> Copying application source ...---> Installing dependencies ......---> Collecting Django static files ...I0610 10:03:00.876021 84805 environment.go:60] Setting 1 environment variables provided by environment file in sources

$ docker run --rm -p 8080:8080 python-django-modwsgi---> Running application from Python script (app.py) ...[Fri Jun 10 00:07:33.580013 2016] [mpm_event:notice] [pid 1:tid 139758878566464] AH00489: Apache/2.4.6 (CentOS)mod_wsgi/4.5.2 Python/3.4.2 configured -- resuming normal operations[Fri Jun 10 00:07:33.580149 2016] [core:notice] [pid 1:tid 139758878566464] AH00094: Command line: 'httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost:8080:1001/httpd.conf -D MOD_WSGI_MULTIPROCESS -DMOD_WSGI_WITH_PROXY_HEADERS -D MOD_WSGI_MPM_ENABLE_EVENT_MODULE -DMOD_WSGI_MPM_EXISTS_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_WORKER_MODULE -DMOD_WSGI_MPM_EXISTS_PREFORK_MODULE -D FOREGROUND'

Docker Base Images (S2I Enabled)

● RHEL 7– Python 2.7 → registry.access.redhat.com/rhscl/python-27-rhel7– Python 3.3 → registry.access.redhat.com/openshift3/python-33-rhel7– Python 3.4 → registry.access.redhat.com/rhscl/python-34-rhel7

● CentOS 7– Python 2.7 → docker.io/centos/python-27-centos7– Python 3.3 → docker.io/openshift/python-33-centos7– Python 3.4 → docker.io/centos/python-34-centos7

Migrating from OpenShift 2 to 3

● Converting of existing applications.● Backward compatible S2I builder.● Guidelines for porting applications.● Templates to aid in porting applications.

Python for Applications

Why *Not* Containers?

● Containers are the recommended option for network services● However:

– Container support for rich desktop applications is currently limited– Container runtime may impose unwanted overhead on dedicated systems– Containers may give more isolation than is wanted– Applications may require non-trivial modification to run as a privileged container

● Software Collections aim to offer “minimum viable runtime isolation”– Add new executable directories to front of PATH– Add new shared library directories to front of LD_LIBRARY_PATH

Why Software Collections for Python?

● Use newer Python runtimes without impacting system components● Use a common Python runtime across multiple operating system versions● Python 3 for Red Hat Enterprise Linux 6 & 7!

Red Hat Software Collections

● Platforms– Red Hat Enterprise Linux 6 & Red Hat Enterprise Linux 7– CentOS 6 and 7 (via softwarecollections.org)– Basis for OpenShift language runtimes

● Available versions (as of RHSCL 2.2)– Python 2.7.8 (+ selected backports)– Python 3.5.1 (+ selected backports)– Python 3.4.2 (+ selected backports)– Python 3.3.2 (+ selected backports)

Python Virtual Environments

● Software Collections allow multiple runtimes to share a system without conflicting● Virtual environments allow multiple Python applications to share a runtime● Low or no runtime overhead: just add/replace directories in Python’s import path● Cleanly isolate application dependencies from platform components● Dependencies within the environment managed with pip● Created via:

– python3 -m venv (Python 3.4+)– virtualenv (Python 2.7, Python 3.x)

● Not included in the Red Hat Enterprise Linux System Python

Constructing a Layered Application

● Base platform (via system package manager):– Operating system (e.g. kernel, C runtime)– Language runtime (from Software Collections)– Other external dependencies (e.g. OpenSSL)

● Modify shared library loading (via enabled Software Collection)● Modify Python import configuration (via virtualenv or standard library’s venv)● Inside the virtual environment:

– Application dependencies (managed via pip)– Application source code (managed via pip, or direct from source control)

Deploying a Layered Application (1)

● Application RPM with generated artifacts in SRPM:– Create full installation in representative environment– Bundle entire virtualenv and other desired components into SRPM– Also include scripts to appropriately activate SCL and virtual environment

● Some generated files will end up in the SRPM– Pre-compiled Python files– Executable wrappers for pip managed Python scripts

Deploying a Layered Application (2)

● Application RPM with source-only SRPM:– SRPM contains source for application and any application level dependencies– virtual environment created and configured during RPM build process

● Not yet fully supported in pip– some pip generated metadata will incorrectly include RPM buildroot paths– shouldn’t matter for most RPM based deployment use cases

Managing Application Dependencies

● The Python Package Index is not an App Store!● Designed to minimize barriers to publication:

– No pre-publication review– Publisher Terms of Service ensure right to redistribute, not to run or modify– Publishers may delete (but not replace) previously published versions

● Recommendation: use caching proxies and a component review process– Commercially supported options: JFrog Artifactory, Sonatype Nexus– Community/self-supported options: devpi, Python plugin for Pulp– Check licensing, export restrictions, project governance, ...

● Security monitoring & response also becomes a dev team responsibility!

Python for System Administration

Why *Not* Software Collections?

● Containers are the recommended option for network services● Software Collections are recommended for applications● However:

– some platform bindings are only installed in the main System Python– the System Python is available on all systems by default

● Some system administration tasks are best handled with the System Python

System Python

● Red Hat Enterprise Linux 6– Python 2.6.6 (+ selected backports)

● Red Hat Enterprise Linux 7– Python 2.7.5 (+ selected backports)

● Fedora 23 and later– Recent Python 3.x (rebases rather than backports)– Recent Python 2.7 available as system package, may not have all system bindings

Caveats and Challenges

● Restricted to features of System Python on oldest supported platform● Community maintained libraries and frameworks often require newer runtimes● Conflict between supporting:

– Red Hat Enterprise Linux 5 (Python 2.4 lacks Py3 forward compatibility features)– Fedora 23+ (Python 3.x as System Python)

● May want to consider higher level system abstractions like Ansible

Migration to Python 3

Python 2.7 Support Timeline

● Anticipated community end-of-life for Python 2.7 in January 2020– https://docs.python.org/devguide/#status-of-python-branches

● Supported in Red Hat Enterprise Linux 7 until June 2024– https://access.redhat.com/support/policy/updates/errata

● Anticipate community project support for Python 2 declining sharply post-2020– Already seeing new community projects starting as Python 3 only

Python 3 Migration Techniques

● General “refactoring enablement” techniques:– automated regression testing frameworks– static structural analysis tools

● Recommended approach for applications and network services:– Migrate to the Python 2.7 SCL or OpenShift image (if using the system Python)– Follow https://docs.python.org/3/howto/pyporting.html– Migrate to the latest Python 3.x SCL or OpenShift image

● Recommended approach for system administration tools:– Consider using Fedora 23+ to look for potential pain points

Python 3 Migration Notes

● Common subset of Python 2.6+ and 3.3+ is quite large● Many deprecated idioms can be updated automatically● Key data & workload driven pain points

– Explicit bytes/unicode separation– Removal of implicit cross-type comparisons

● Automated refactoring and compatibility testing tools continue to improve

Modernizing the Python 2.7Network Security Stack

New Security Features in Python 2.7

● https://docs.python.org/2/whatsnew/2.7.html#pep-466-network-security-enhancements-for-python-2-7

● Constant-time comparison (hmac.compare_digest())● Password storage hashing (hashlib.pbkdf2_hmac())● ssl module rebase on Python 3.4 implementation

– Server Name Indication support– SSLContext for SSL configuration– Configuration support for TLS 1.x– Access to system certificate stores– ...

Availability in Red Hat Products

● Red Hat Enterprise Linux 7.2– backported to System Python

● Red Hat Software Collections 2.0+– default in Python 3.4

● Red Hat Software Collections 2.2+– backported to Python 2.7– default in Python 3.5

Third Party Module Compatibility

● ssl module rebase changed several private implementation details● Some third party libraries had used internal APIs instead of requesting public ones● Backport offers greater compatibility than upstream rebase● Testing before upgrading is still recommended● Report problems through the usual channels

Defaulting to HTTPSCertificate Verification

What does “HTTPS” Mean?

● Historical Python standard library answer:– “HTTP connection with SSL/TLS enabled”– didn’t check certificate validity or remote host identification

● Modern Python standard library answer:– “What web browsers say it means”– still a HTTP connection with SSL/TLS enabled–also checks for certificate validity–also checks remote host identification against system certificate store

Verifying HTTPS Certificates

● https://docs.python.org/2/whatsnew/2.7.html#pep-476-enabling-certificate-verification-by-default-for-stdlib-http-clients

● Default behavior of standard library HTTPS clients in:– Python 2.7.9+– Python 3.4.3+– Python 3.5.0+

● Turns a silent security failure into a noisy connection failure● Potential problems:

– Self-signed internal certificates– Expired certificates– Internal CAs not configured on client system

On Red Hat Platforms

● File-based opt-in:– Config setting in /etc/python/cert-verification.cfg– Red Hat Enterprise Linux 7.2+ System Python– Red Hat Software Collections 2.2+ Python 2.7 collection

● Default behavior:– Red Hat Software Collections Python 3.5 collection

● Details in Knowledge Base– https://access.redhat.com/articles/2039753

Future Configuration Options

● https://docs.python.org/2/whatsnew/2.7.html#pep-493-https-verification-migration-tools-for-python-2-7

● Adds new Python 2.7 specific configuration options– ssl._https_verify_certificates() API– PYTHONHTTPSVERIFY environment variable

● Can be used to revert Python 2.7.12+ to Python 2.7.8 behavior● Python 2.7 only, not supported by any version of Python 3

Resources

OpenShift

● OpenShift– https://www.openshift.com

● OpenShift Online Preview– https://www.openshift.com/devpreview/register.html

● OpenShift Container Development Kit– https://developers.redhat.com/products/cdk/overview/

● OpenShift Origin– https://www.openshift.org

● OpenShift Origin All-In-One VM– https://www.openshift.org/vm

Python

● Red Hat Software Collections– http://developers.redhat.com/products/softwarecollections/– Access: https://access.redhat.com/solutions/472793

● Software Collections Upstream– https://wiki.centos.org/SpecialInterestGroup/SCLo– https://www.softwarecollections.org

Related DevNation 2016 Sessions

● OpenShift– OpenShift Enterprise 3 walk-through with Docker and Kubernetes

● Container Development Kit– CDK 2.0: Docker, Kubernetes, and OSE on your desk– Container development for command line developers

● Software Collections– Software Collections: Easy access to the cutting edge