Update the python dependency to python3

* Support opensuseleap, opensusetumbleweed, sles
* Add efs-utils version info field to self-signed certificate
* Check if the localhost stunnel port can be connected before performing the mount operation to avoid connection refused
* Fix the issue that socket is not opened in container network namespace

.circleci/config.yml

@@ -96,7 +96,7 @@ commands:
       - run:
           name: Install package
           command: |
-            zypper install -y --allow-unsigned-rpm build/amazon-efs-utils*rpm
+            zypper --no-gpg-checks install -y build/amazon-efs-utils*rpm
       - run:
           name: Check installed successfully
           command: |
@@ -160,9 +160,6 @@ workflows:
       - test:
           name: python34
           image: circleci/python:3.4.9
-      - test:
-          name: python27
-          image: circleci/python:2.7.13
       - build-deb-package:
           name: ubuntu16
           image: ubuntu:16.04
@@ -211,3 +208,12 @@ workflows:
       - build-suse-rpm-package:
           name: opensuse-leap15.1
           image: opensuse/leap:15.1
+      - build-suse-rpm-package:
+          name: opensuse-leap15.2
+          image: opensuse/leap:15.2
+      - build-suse-rpm-package:
+          name: opensuse-leap-latest
+          image: opensuse/leap:latest
+      - build-suse-rpm-package:
+          name: opensuse-tumbleweed-latest
+          image: opensuse/tumbleweed:latest
\ No newline at end of file

Makefile

@@ -37,10 +37,6 @@ tarball: clean
 
 	tar -czf $(SOURCE_TARBALL) $(PACKAGE_NAME)/*
 
-.PHONY: shebang-support
-shebang-support:
-	./mangle-shebangs.sh
-
 .PHONY: sources
 sources: tarball
 
@@ -53,7 +49,7 @@ rpm-only:
 	cp $(BUILD_DIR)/RPMS/*/*rpm build
 
 .PHONY: rpm
-rpm: shebang-support sources rpm-only
+rpm: sources rpm-only
 
 .PHONY: deb
 deb:

README.md

@@ -6,30 +6,34 @@ Utilities for Amazon Elastic File System (EFS)
 
 The `efs-utils` package has been verified against the following Linux distributions:
 
-| Distribution | Package Type | `init` System | Python Env|
-| ------------ | ------------ | ------------- | --------- |
-| Amazon Linux 2017.09 | `rpm` | `upstart` | Python2 |
-| Amazon Linux 2 | `rpm` | `systemd` | Python2 |
-| CentOS 7 | `rpm` | `systemd` | Python2 |
-| CentOS 8 | `rpm` | `systemd` | Python3 |
-| RHEL 7 | `rpm`| `systemd` | Python2 |
-| RHEL 8 | `rpm`| `systemd` | Python3 |
-| Fedora 28 | `rpm` | `systemd` | Python3 |
-| Fedora 29 | `rpm` | `systemd` | Python3 |
-| Fedora 30 | `rpm` | `systemd` | Python3 |
-| Fedora 31 | `rpm` | `systemd` | Python3 |
-| Fedora 32 | `rpm` | `systemd` | Python3 |
-| Debian 9 | `deb` | `systemd` | Python2 |
-| Debian 10 | `deb` | `systemd` | Python2 |
-| Ubuntu 16.04 | `deb` | `systemd` | Python2 |
-| Ubuntu 18.04 | `deb` | `systemd` | Python3 |
-| Ubuntu 20.04 | `deb` | `systemd` | Python3 |
+| Distribution | Package Type | `init` System |
+| ------------ | ------------ | ------------- |
+| Amazon Linux 2017.09 | `rpm` | `upstart` |
+| Amazon Linux 2 | `rpm` | `systemd` |
+| CentOS 7 | `rpm` | `systemd` |
+| CentOS 8 | `rpm` | `systemd` |
+| RHEL 7 | `rpm`| `systemd` |
+| RHEL 8 | `rpm`| `systemd` |
+| Fedora 28 | `rpm` | `systemd` |
+| Fedora 29 | `rpm` | `systemd` |
+| Fedora 30 | `rpm` | `systemd` |
+| Fedora 31 | `rpm` | `systemd` |
+| Fedora 32 | `rpm` | `systemd` |
+| Debian 9 | `deb` | `systemd` |
+| Debian 10 | `deb` | `systemd` |
+| Ubuntu 16.04 | `deb` | `systemd` |
+| Ubuntu 18.04 | `deb` | `systemd` |
+| Ubuntu 20.04 | `deb` | `systemd` |
+| OpenSUSE Leap | `rpm` | `systemd` |
+| OpenSUSE Tumbleweed | `rpm` | `systemd` |
+| SLES 12 | `rpm` | `systemd` |
+| SLES 15 | `rpm` | `systemd` |
 
 ## Prerequisites
 
 * `nfs-utils` (RHEL/CentOS/Amazon Linux/Fedora) or `nfs-common` (Debian/Ubuntu)
 * OpenSSL 1.0.2+
-* Python 2.7+
+* Python 3.4+
 * `stunnel` 4.56+
 
 ## Installation
@@ -61,6 +65,8 @@ Other distributions require building the package from source and installing it.
 
 - To build and install an RPM:
 
+If the distribution is not OpenSUSE or SLES
+
 ```
 $ sudo yum -y install git rpm-build make
 $ git clone https://github.com/aws/efs-utils
@@ -69,6 +75,26 @@ $ make rpm
 $ sudo yum -y install build/amazon-efs-utils*rpm
 ```
 
+Otherwise
+
+```
+$ sudo zypper refresh
+$ sudo zypper install -y git rpm-build make
+$ git clone https://github.com/aws/efs-utils
+$ cd efs-utils
+$ make rpm
+$ sudo zypper --no-gpg-checks install -y build/amazon-efs-utils*rpm
+```
+
+On OpenSUSE, if you see error like `File './suse/noarch/bash-completion-2.11-2.1.noarch.rpm' not found on medium 'http://download.opensuse.org/tumbleweed/repo/oss/'`
+during installation of `git`, run the following commands to re-add repo OSS and NON-OSS, then run the install script above again.
+
+```
+sudo zypper ar -f -n OSS http://download.opensuse.org/tumbleweed/repo/oss/ OSS
+sudo zypper ar -f -n NON-OSS http://download.opensuse.org/tumbleweed/repo/non-oss/ NON-OSS
+sudo zypper refresh
+```
+
 - To build and install a Debian package:
 
 ```
@@ -157,42 +183,66 @@ By default, when using the EFS mount helper with TLS, it enforces certificate ho
 
 Once you’ve installed the `amazon-efs-utils` package, to upgrade your system’s version of `stunnel`, see [Upgrading Stunnel](https://docs.aws.amazon.com/efs/latest/ug/using-amazon-efs-utils.html#upgrading-stunnel).
 
+## Upgrading stunnel for SLES12
+
+Run the following commands and follow the output hint of zypper package manager to upgrade the stunnel on your SLES12 instance
+
+```bash
+sudo zypper addrepo https://download.opensuse.org/repositories/security:Stunnel/SLE_12_SP5/security:Stunnel.repo
+sudo zypper refresh
+sudo zypper install -y stunnel
+```
+
 ## Enable mount success/failure notification via CloudWatch log
 `efs-utils` now support publishing mount success/failure logs to CloudWatch log. By default, this feature is disabled. There are three
 steps you must follow to enable and use this feature:
 
 ### Step 1. Install botocore
-`efs-utils` uses botocore to interact with CloudWatch log service . Please note the package type and 
-python env from the above table. 
-- To install botocore on RPM
+`efs-utils` uses botocore to interact with CloudWatch log service . Please note the package type from the above table. 
+- Download the `get-pip.py` script
+#### RPM
 ```bash
-# Python2
-sudo python /tmp/get-pip.py
-sudo pip install botocore || sudo /usr/local/bin/pip install botocore
-
-# Python3
-sudo python3 /tmp/get-pip.py
-sudo pip3 install botocore || sudo /usr/local/bin/pip3 install botocore
+sudo yum -y install wget
 ```
-- To install botocore on DEB
+```bash
+if [[ "$(python3 -V 2>&1)" =~ ^(Python 3.5.*) ]]; then
+    sudo wget https://bootstrap.pypa.io/3.5/get-pip.py -O /tmp/get-pip.py
+elif [[ "$(python3 -V 2>&1)" =~ ^(Python 3.4.*) ]]; then
+    sudo wget https://bootstrap.pypa.io/3.4/get-pip.py -O /tmp/get-pip.py
+else
+    sudo wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py
+fi
+```
+#### DEB
 ```bash
 sudo apt-get update
 sudo apt-get -y install wget
-wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py
-
-# Python2
-sudo python /tmp/get-pip.py
-sudo pip install botocore || sudo /usr/local/bin/pip install botocore
+```
+```bash
+if echo $(python3 -V 2>&1) | grep -e "Python 3.5"; then
+    sudo wget https://bootstrap.pypa.io/3.5/get-pip.py -O /tmp/get-pip.py
+elif echo $(python3 -V 2>&1) | grep -e "Python 3.4"; then
+    sudo wget https://bootstrap.pypa.io/3.4/get-pip.py -O /tmp/get-pip.py
+else
+    sudo apt-get -y install python3-distutils
+    sudo wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py
+fi
+```
 
-# On Debian10, the botocore needs to be installed in specific target folder
-sudo python /tmp/get-pip.py
-sudo pip install --target /usr/lib/python2.7/dist-packages botocore || sudo /usr/local/bin/pip install --target /usr/lib/python2.7/dist-packages botocore
+- To install botocore on RPM
+```bash
+sudo python3 /tmp/get-pip.py
+sudo pip3 install botocore || sudo /usr/local/bin/pip3 install botocore
+```
 
-# Python3
+- To install botocore on DEB
+```bash
 sudo python3 /tmp/get-pip.py
 sudo pip3 install botocore || sudo /usr/local/bin/pip3 install botocore
+```
 
-# On Ubuntu20, the botocore needs to be installed in specific target folder
+#### On Debian10 and Ubuntu20, the botocore needs to be installed in specific target folder
+```bash
 sudo python3 /tmp/get-pip.py
 sudo pip3 install --target /usr/lib/python3/dist-packages botocore || sudo /usr/local/bin/pip3 install --target /usr/lib/python3/dist-packages botocore
 ```

amazon-efs-utils.spec

@@ -7,16 +7,9 @@
 #
 
 %if 0%{?amzn1}
-%global python_requires system-python
-
+%global python_requires python36
 %else
-
-%if 0%{?fedora} || 0%{?el8} || 0%{?is_opensuse}
 %global python_requires python3
-%else
-%global python_requires python2
-%endif
-
 %endif
 
 %if 0%{?amzn1} || 0%{?rhel} == 6
@@ -28,11 +21,17 @@
 %if 0%{?is_opensuse}
 %global platform .opensuse
 %else
+
+%if 0%{?sle_version}
+%global platform .suse
+%else
 %global platform %{dist}
 %endif
 
+%endif
+
 Name      : amazon-efs-utils
-Version   : 1.28.2
+Version   : 1.29.1
 Release   : 1%{platform}
 Summary   : This package provides utilities for simplifying the use of EFS file systems
 
@@ -132,6 +131,10 @@ fi
 %clean
 
 %changelog
+* Mon Jan 25 2021 Yuan Gao <ygaochn@amazon.com> - 1.29.1
+- Update the python dependency to python3
+- Support SLES and OpenSUSE
+
 * Thu Oct 8 2020 Yuan Gao <ygaochn@amazon.com> - 1.28.2
 - Fix an issue where fs cannot be mounted with iam using instance profile when IMDSv2 is enabled
 

build-deb.sh

@@ -11,12 +11,9 @@ set -ex
 
 BASE_DIR=$(pwd)
 BUILD_ROOT=${BASE_DIR}/build/debbuild
-VERSION=1.28.2
+VERSION=1.29.1
 RELEASE=1
 DEB_SYSTEM_RELEASE_PATH=/etc/os-release
-UBUNTU18_REGEX="Ubuntu 18"
-UBUNTU20_REGEX="Ubuntu 20"
-DEBIAN11_REGEX="Debian GNU/Linux bullseye"
 
 echo 'Cleaning deb build workspace'
 rm -rf ${BUILD_ROOT}
@@ -31,15 +28,6 @@ mkdir -p ${BUILD_ROOT}/usr/bin
 mkdir -p ${BUILD_ROOT}/var/log/amazon/efs
 mkdir -p ${BUILD_ROOT}/usr/share/man/man8
 
-if [ -f $DEB_SYSTEM_RELEASE_PATH ] && echo "$(grep PRETTY_NAME $DEB_SYSTEM_RELEASE_PATH)" \
-          | grep -e "$UBUNTU18_REGEX" -e "$DEBIAN11_REGEX" -e "$UBUNTU20_REGEX"; then
-    echo 'Correcting python executable'
-    sed -i -e 's/python|python2/python3/' dist/amazon-efs-utils.control
-    # Replace the first line in .py to "#!/usr/bin/env python3" no matter what it was before
-    sed -i -e '1 s/^.*$/\#!\/usr\/bin\/env python3/' src/watchdog/__init__.py
-    sed -i -e '1 s/^.*$/\#!\/usr\/bin\/env python3/' src/mount_efs/__init__.py
-fi
-
 echo 'Copying application files'
 install -p -m 644 dist/amazon-efs-mount-watchdog.conf ${BUILD_ROOT}/etc/init
 install -p -m 644 dist/amazon-efs-mount-watchdog.service ${BUILD_ROOT}/etc/systemd/system

config.ini

@@ -7,5 +7,5 @@
 #
 
 [global]
-version=1.28.2
+version=1.29.1
 release=1

dist/amazon-efs-utils.control

 Package: amazon-efs-utils
 Architecture: all
-Version: 1.28.2
+Version: 1.29.1
 Section: utils
-Depends: python|python2, nfs-common, stunnel4 (>= 4.56), openssl (>= 1.0.2), util-linux
+Depends: python3, nfs-common, stunnel4 (>= 4.56), openssl (>= 1.0.2), util-linux
 Priority: optional
 Copyright: MIT License
 Maintainer: Amazon.com, Inc. <efs-utils@amazon.com>

mangle-shebangs.sh

-#!/bin/bash
-
-RPM_SYSTEM_RELEASE_PATH=/etc/system-release
-RPM_OS_RELEASE_PATH=/etc/os-release
-RHEL8_REGEX="Red Hat Enterprise Linux release 8"
-FEDORA_REGEX="Fedora release"
-CENTOS8_REGEX="CentOS Linux release 8"
-SUSE15_REGEX="openSUSE Leap"
-
-function correct_python_executable() {
-    echo 'Correcting python executable'
-    # Replace the first line in .py to "#!/usr/bin/env python3" no matter what it was before
-    sed -i -e '1 s/^.*$/\#!\/usr\/bin\/env python3/' src/watchdog/__init__.py
-    sed -i -e '1 s/^.*$/\#!\/usr\/bin\/env python3/' src/mount_efs/__init__.py
-}
-
-# RHEL8, Fedora30+ and CentOS8 treat shebangs of the form "#!/usr/bin/env python" as errors
-if [ -f $RPM_SYSTEM_RELEASE_PATH ] && [[ "$(cat $RPM_SYSTEM_RELEASE_PATH)" =~ $RHEL8_REGEX|$FEDORA_REGEX|$CENTOS8_REGEX ]]; then
-    correct_python_executable
-fi
-
-# OpenSUSE use python3 as default python version
-if [ -f $RPM_OS_RELEASE_PATH ] && [[ $(grep PRETTY_NAME $RPM_OS_RELEASE_PATH) =~ $SUSE15_REGEX ]]; then
-    correct_python_executable
-fi

src/mount_efs/__init__.py

-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # Copyright 2017-2018 Amazon.com, Inc. and its affiliates. All Rights Reserved.
 #
@@ -51,10 +51,10 @@ from datetime import datetime, timedelta
 from logging.handlers import RotatingFileHandler
 
 try:
+    from configparser import ConfigParser, NoOptionError, NoSectionError
+except ImportError:
     import ConfigParser
     from ConfigParser import NoOptionError, NoSectionError
-except ImportError:
-    from configparser import ConfigParser, NoOptionError, NoSectionError
 
 try:
     from urllib.parse import quote_plus
@@ -62,12 +62,12 @@ except ImportError:
     from urllib import quote_plus
 
 try:
-    from urllib2 import URLError, HTTPError, build_opener, urlopen, Request, HTTPHandler
-    from urllib import urlencode
-except ImportError:
     from urllib.request import urlopen, Request
     from urllib.error import URLError, HTTPError
     from urllib.parse import urlencode
+except ImportError:
+    from urllib2 import URLError, HTTPError, build_opener, urlopen, Request, HTTPHandler
+    from urllib import urlencode
 
 try:
     import botocore.session
@@ -77,19 +77,20 @@ except ImportError:
     BOTOCORE_PRESENT = False
 
 
-VERSION = '1.28.2'
+VERSION = '1.29.1'
 SERVICE = 'elasticfilesystem'
 
+CLONE_NEWNET = 0x40000000
 CONFIG_FILE = '/etc/amazon/efs/efs-utils.conf'
 CONFIG_SECTION = 'mount'
 CLIENT_INFO_SECTION = 'client-info'
 CLIENT_SOURCE_STR_LEN_LIMIT = 100
-
+# Cloudwatchlog agent dict includes cloudwatchlog botocore client, cloudwatchlog group name, cloudwatchlog stream name
+CLOUDWATCHLOG_AGENT = None
 CLOUDWATCH_LOG_SECTION = 'cloudwatch-log'
 DEFAULT_CLOUDWATCH_LOG_GROUP = '/aws/efs/utils'
 DEFAULT_RETENTION_DAYS = 14
-# Cloudwatchlog agent dict includes cloudwatchlog botocore client, cloudwatchlog group name, cloudwatchlog stream name
-CLOUDWATCHLOG_AGENT = None
+DEFAULT_UNKNOWN_VALUE = 'unknown'
 
 LOG_DIR = '/var/log/amazon/efs'
 LOG_FILE = 'mount.log'
@@ -219,8 +220,43 @@ OS_RELEASE_PATH = '/etc/os-release'
 RHEL8_RELEASE_NAME = 'Red Hat Enterprise Linux release 8'
 CENTOS8_RELEASE_NAME = 'CentOS Linux release 8'
 FEDORA_RELEASE_NAME = 'Fedora release'
-SUSE_RELEASE_NAME = 'openSUSE Leap'
-SKIP_NO_LIBWRAP_RELEASES = [RHEL8_RELEASE_NAME, CENTOS8_RELEASE_NAME, FEDORA_RELEASE_NAME, SUSE_RELEASE_NAME]
+OPEN_SUSE_LEAP_RELEASE_NAME = 'openSUSE Leap'
+SUSE_RELEASE_NAME = 'SUSE Linux Enterprise Server'
+
+SKIP_NO_LIBWRAP_RELEASES = [RHEL8_RELEASE_NAME, CENTOS8_RELEASE_NAME, FEDORA_RELEASE_NAME, OPEN_SUSE_LEAP_RELEASE_NAME,
+                            SUSE_RELEASE_NAME]
+
+
+def errcheck(ret, func, args):
+    from ctypes import get_errno
+    if ret == -1:
+        e = get_errno()
+        raise OSError(e, os.strerror(e))
+
+
+def setns(fd, nstype):
+    from ctypes import CDLL
+    libc = CDLL('libc.so.6', use_errno=True)
+    libc.setns.errcheck = errcheck
+    if hasattr(fd, 'fileno'):
+        fd = fd.fileno()
+    return libc.setns(fd, nstype)
+
+
+class NetNS(object):
+    # Open sockets from given network namespace: stackoverflow.com/questions/28846059
+    def __init__(self, nspath):
+        self.original_nspath = '/proc/%d/ns/net' % os.getpid()
+        self.target_nspath = nspath
+
+    def __enter__(self):
+        self.original_namespace = open(self.original_nspath)
+        with open(self.target_nspath) as fd:
+            setns(fd, CLONE_NEWNET)
+
+    def __exit__(self, *args):
+        setns(self.original_namespace, CLONE_NEWNET)
+        self.original_namespace.close()
 
 
 def fatal_error(user_message, log_message=None, exit_code=1):
@@ -587,22 +623,35 @@ def choose_tls_port(config, options):
         ports_to_try = tls_ports[mid:] + tls_ports[:mid]
         assert len(tls_ports) == len(ports_to_try)
 
+    if 'netns' not in options:
+        tls_port = find_tls_port_in_range(ports_to_try)
+    else:
+        with NetNS(nspath=options['netns']):
+            tls_port = find_tls_port_in_range(ports_to_try)
+
+    if tls_port:
+        return tls_port
+
+    if 'tlsport' in options:
+        fatal_error('Specified port [%s] is unavailable. Try selecting a different port.' % options['tlsport'])
+    else:
+        fatal_error('Failed to locate an available port in the range [%d, %d], try specifying a different port range in %s'
+                    % (lower_bound, upper_bound, CONFIG_FILE))
+
+
+def find_tls_port_in_range(ports_to_try):
     sock = socket.socket()
     for tls_port in ports_to_try:
         try:
+            logging.info("binding %s", tls_port)
             sock.bind(('localhost', tls_port))
             sock.close()
             return tls_port
-        except socket.error:
+        except socket.error as e:
+            logging.info(e)
             continue
-
     sock.close()
-
-    if 'tlsport' in options:
-        fatal_error('Specified port [%s] is unavailable. Try selecting a different port.' % options['tlsport'])
-    else:
-        fatal_error('Failed to locate an available port in the range [%d, %d], try specifying a different port range in %s'
-                    % (lower_bound, upper_bound, CONFIG_FILE))
+    return None
 
 
 def is_ocsp_enabled(config, options):
@@ -709,7 +758,7 @@ def get_system_release_version():
     except IOError:
         logging.debug('Unable to read %s', OS_RELEASE_PATH)
 
-    return 'unknown'
+    return DEFAULT_UNKNOWN_VALUE