Commit 3793a746 authored by Yuan Gao's avatar Yuan Gao
Browse files

Bug fix and enhancement, support fedora

* Fix an issue where subprocess was not killed successfully
* Stop emitting unrecognized init system supervisord if the watchdog daemon has already been launched by supervisor
* Support Fedora
* Change version format to Major-Minor.ReleaseId
parent 7b1f2d0f
...@@ -151,6 +151,12 @@ workflows: ...@@ -151,6 +151,12 @@ workflows:
- build-rpm-package: - build-rpm-package:
name: amazon-linux name: amazon-linux
image: amazonlinux:1 image: amazonlinux:1
- build-rpm-package:
name: fedora28
image: fedora:28
- build-rpm-package:
name: fedora29
image: fedora:29
- build-rpm-package: - build-rpm-package:
name: fedora30 name: fedora30
image: fedora:30 image: fedora:30
......
...@@ -13,6 +13,8 @@ The `efs-utils` package has been verified against the following Linux distributi ...@@ -13,6 +13,8 @@ The `efs-utils` package has been verified against the following Linux distributi
| CentOS 7 | `rpm` | `systemd` | | CentOS 7 | `rpm` | `systemd` |
| RHEL 7 | `rpm`| `systemd` | | RHEL 7 | `rpm`| `systemd` |
| RHEL 8 | `rpm`| `systemd` | | RHEL 8 | `rpm`| `systemd` |
| Fedora 28 | `rpm` | `systemd` |
| Fedora 29 | `rpm` | `systemd` |
| Debian 9 | `deb` | `systemd` | | Debian 9 | `deb` | `systemd` |
| Debian 10 | `deb` | `systemd` | | Debian 10 | `deb` | `systemd` |
| Ubuntu 16.04 | `deb` | `systemd` | | Ubuntu 16.04 | `deb` | `systemd` |
...@@ -20,7 +22,7 @@ The `efs-utils` package has been verified against the following Linux distributi ...@@ -20,7 +22,7 @@ The `efs-utils` package has been verified against the following Linux distributi
## Prerequisites ## Prerequisites
* `nfs-utils` (RHEL/CentOS/Amazon Linux) or `nfs-common` (Debian/Ubuntu) * `nfs-utils` (RHEL/CentOS/Amazon Linux/Fedora) or `nfs-common` (Debian/Ubuntu)
* OpenSSL 1.0.2+ * OpenSSL 1.0.2+
* Python 2.7+ * Python 2.7+
* `stunnel` 4.56+ * `stunnel` 4.56+
......
...@@ -11,8 +11,7 @@ set -ex ...@@ -11,8 +11,7 @@ set -ex
BASE_DIR=$(pwd) BASE_DIR=$(pwd)
BUILD_ROOT=${BASE_DIR}/build/debbuild BUILD_ROOT=${BASE_DIR}/build/debbuild
VERSION=1.25 VERSION=1.25-3
RELEASE=2
DEB_SYSTEM_RELEASE_PATH=/etc/os-release DEB_SYSTEM_RELEASE_PATH=/etc/os-release
UBUNTU18_REGEX="Ubuntu 18" UBUNTU18_REGEX="Ubuntu 18"
DEBIAN11_REGEX="Debian GNU/Linux bullseye" DEBIAN11_REGEX="Debian GNU/Linux bullseye"
...@@ -74,7 +73,7 @@ tar czf data.tar.gz etc sbin usr var --owner=0 --group=0 ...@@ -74,7 +73,7 @@ tar czf data.tar.gz etc sbin usr var --owner=0 --group=0
cd ${BASE_DIR} cd ${BASE_DIR}
echo 'Building deb' echo 'Building deb'
DEB=${BUILD_ROOT}/amazon-efs-utils-${VERSION}-${RELEASE}.deb DEB=${BUILD_ROOT}/amazon-efs-utils-${VERSION}.deb
ar r ${DEB} ${BUILD_ROOT}/debian-binary ar r ${DEB} ${BUILD_ROOT}/debian-binary
ar r ${DEB} ${BUILD_ROOT}/control.tar.gz ar r ${DEB} ${BUILD_ROOT}/control.tar.gz
ar r ${DEB} ${BUILD_ROOT}/data.tar.gz ar r ${DEB} ${BUILD_ROOT}/data.tar.gz
......
...@@ -8,4 +8,4 @@ ...@@ -8,4 +8,4 @@
[global] [global]
version=1.25 version=1.25
release=2 release=3
Package: amazon-efs-utils Package: amazon-efs-utils
Architecture: all Architecture: all
Version: 1.25 Version: 1.25-3
Section: utils Section: utils
Depends: python|python2, nfs-common, stunnel4 (>= 4.56), openssl (>= 1.0.2), util-linux Depends: python|python2, nfs-common, stunnel4 (>= 4.56), openssl (>= 1.0.2), util-linux
Priority: optional Priority: optional
......
...@@ -27,7 +27,7 @@ ...@@ -27,7 +27,7 @@
Name : amazon-efs-utils Name : amazon-efs-utils
Version : 1.25 Version : 1.25
Release : 2%{?dist} Release : 3%{?dist}
Summary : This package provides utilities for simplifying the use of EFS file systems Summary : This package provides utilities for simplifying the use of EFS file systems
Group : Amazon/Tools Group : Amazon/Tools
...@@ -126,6 +126,11 @@ fi ...@@ -126,6 +126,11 @@ fi
%clean %clean
%changelog %changelog
* Tue May 26 2020 Yuan Gao <ygaochn@amazon.com> - 1.25-3
- Fix an issue where subprocess was not killed successfully
- Stop emitting unrecognized init system supervisord if the watchdog daemon has already been launched by supervisor
- Support Fedora
* Tue May 05 2020 Yuan Gao <ygaochn@amazon.com> - 1.25-2 * Tue May 05 2020 Yuan Gao <ygaochn@amazon.com> - 1.25-2
- Fix the issue that IAM role name format is not correctly encoded in python3 - Fix the issue that IAM role name format is not correctly encoded in python3
- Add optional override for stunnel debug log output location - Add optional override for stunnel debug log output location
......
...@@ -68,7 +68,7 @@ except ImportError: ...@@ -68,7 +68,7 @@ except ImportError:
from urllib.error import URLError, HTTPError from urllib.error import URLError, HTTPError
VERSION = '1.25' VERSION = '1.25-3'
SERVICE = 'elasticfilesystem' SERVICE = 'elasticfilesystem'
CONFIG_FILE = '/etc/amazon/efs/efs-utils.conf' CONFIG_FILE = '/etc/amazon/efs/efs-utils.conf'
...@@ -198,7 +198,8 @@ WATCHDOG_SERVICE = 'amazon-efs-mount-watchdog' ...@@ -198,7 +198,8 @@ WATCHDOG_SERVICE = 'amazon-efs-mount-watchdog'
SYSTEM_RELEASE_PATH = '/etc/system-release' SYSTEM_RELEASE_PATH = '/etc/system-release'
RHEL8_RELEASE_NAME = 'Red Hat Enterprise Linux release 8' RHEL8_RELEASE_NAME = 'Red Hat Enterprise Linux release 8'
CENTOS8_RELEASE_NAME = 'CentOS Linux release 8' CENTOS8_RELEASE_NAME = 'CentOS Linux release 8'
SKIP_NO_LIBWRAP_RELEASES = [RHEL8_RELEASE_NAME, CENTOS8_RELEASE_NAME] FEDORA_RELEASE_NAME = 'Fedora release'
SKIP_NO_LIBWRAP_RELEASES = [RHEL8_RELEASE_NAME, CENTOS8_RELEASE_NAME, FEDORA_RELEASE_NAME]
def fatal_error(user_message, log_message=None, exit_code=1): def fatal_error(user_message, log_message=None, exit_code=1):
...@@ -774,6 +775,31 @@ def start_watchdog(init_system): ...@@ -774,6 +775,31 @@ def start_watchdog(init_system):
else: else:
logging.debug('%s is already running', WATCHDOG_SERVICE) logging.debug('%s is already running', WATCHDOG_SERVICE)
elif init_system == 'supervisord':
error_message = None
proc = subprocess.Popen(
['supervisorctl', 'status', WATCHDOG_SERVICE], stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
stdout, stderr = proc.communicate()
rc = proc.returncode
if rc != 0:
if rc == 4: # No such process
error_message = \
'%s process is not started, please use supervisor to launch the watchdog daemon' % WATCHDOG_SERVICE
elif rc == 2: # Supervisorctl is not properly setup
error_message = 'Cannot invoke supervisorctl to check status of %s' % WATCHDOG_SERVICE
else:
error_message = 'Unknown error %s' % stderr
else:
if 'RUNNING' in stdout:
logging.debug('%s is already running', WATCHDOG_SERVICE)
else:
logging.debug('%s is not running', WATCHDOG_SERVICE)
if error_message:
sys.stderr.write('%s\n' % error_message)
logging.warning(error_message)
else: else:
error_message = 'Could not start %s, unrecognized init system "%s"' % (WATCHDOG_SERVICE, init_system) error_message = 'Could not start %s, unrecognized init system "%s"' % (WATCHDOG_SERVICE, init_system)
sys.stderr.write('%s\n' % error_message) sys.stderr.write('%s\n' % error_message)
...@@ -1131,7 +1157,11 @@ def subprocess_call(cmd, error_message): ...@@ -1131,7 +1157,11 @@ def subprocess_call(cmd, error_message):
rc = process.poll() rc = process.poll()
if rc != 0: if rc != 0:
logging.error('Command %s failed, rc=%s, stdout="%s", stderr="%s"' % (cmd, rc, output, err), exc_info=True) logging.error('Command %s failed, rc=%s, stdout="%s", stderr="%s"' % (cmd, rc, output, err), exc_info=True)
process.kill() try:
process.kill()
except OSError:
# Silently fail if the subprocess has exited already
pass
else: else:
return output, err return output, err
error_message = '%s, error is: %s' % (error_message, err) error_message = '%s, error is: %s' % (error_message, err)
......
...@@ -45,7 +45,7 @@ except ImportError: ...@@ -45,7 +45,7 @@ except ImportError:
from urllib.error import URLError from urllib.error import URLError
from urllib.request import urlopen from urllib.request import urlopen
VERSION = '1.25' VERSION = '1.25-3'
SERVICE = 'elasticfilesystem' SERVICE = 'elasticfilesystem'
CONFIG_FILE = '/etc/amazon/efs/efs-utils.conf' CONFIG_FILE = '/etc/amazon/efs/efs-utils.conf'
......
...@@ -13,43 +13,46 @@ try: ...@@ -13,43 +13,46 @@ try:
except ImportError: except ImportError:
from configparser import ConfigParser from configparser import ConfigParser
FILE_LIST = ['src/watchdog/__init__.py', 'src/mount_efs/__init__.py', 'dist/amazon-efs-utils.spec',
'dist/amazon-efs-utils.control', 'build-deb.sh'] SPEC_FILE = 'dist/amazon-efs-utils.spec'
RPM_FILE = 'dist/amazon-efs-utils.spec' NON_SPEC_FILE_LIST = ['build-deb.sh', 'src/watchdog/__init__.py', 'src/mount_efs/__init__.py',
DEB_FILE = 'build-deb.sh' 'dist/amazon-efs-utils.control', 'build-deb.sh']
GLOBAL_CONFIG = 'config.ini' GLOBAL_CONFIG = 'config.ini'
def test_global_version_match(): def test_spec_file_version_release_match():
global_version = get_global_value('version') global_version = get_global_value('version')
for f in FILE_LIST:
version_in_file = get_version_for_file(f)
assert version_in_file == global_version, \
'version in {} is {}, does not match global version {}'.format(f, version_in_file, global_version)
def test_global_release_match():
global_release = get_global_value('release') global_release = get_global_value('release')
version_in_spec_file = get_version_for_file(SPEC_FILE)
release_in_spec_file = get_release_for_file(SPEC_FILE)
assert version_in_spec_file == global_version, \
'version in {} is {}, does not match global version {}'.format(SPEC_FILE, version_in_spec_file, global_version)
assert release_in_spec_file == global_release, \
'release in {} is {}, does not match global release {}'.format(SPEC_FILE, release_in_spec_file, global_release)
release_in_rpm_file = get_release_for_file(RPM_FILE) def test_non_spec_file_version_release_match():
release_in_deb_file = get_release_for_file(DEB_FILE) global_version_release = get_expected_version_release()
assert release_in_rpm_file == global_release, \ for f in NON_SPEC_FILE_LIST:
'release in {} is {}, does not match global release {}'.format(RPM_FILE, release_in_rpm_file, global_release) version_release_in_file = get_version_for_file(f)
assert release_in_deb_file == global_release, \ assert version_release_in_file == global_version_release, 'version-release in {} is {}, does not match global version {}'\
'release in {} is {}, does not match global release {}'.format(RPM_FILE, release_in_deb_file, global_release) .format(f, version_release_in_file, global_version_release)
def test_changelog_version_match(): def test_changelog_version_match():
global_version = get_global_value('version') expected_version_release = get_expected_version_release()
global_release = get_global_value('release')
expected_version_release = global_version + '-' + global_release
version_release_in_changelog = get_version_for_changelog(RPM_FILE) version_release_in_changelog = get_version_for_changelog(SPEC_FILE)
assert version_release_in_changelog is not None and version_release_in_changelog == expected_version_release, \ assert version_release_in_changelog is not None and version_release_in_changelog == expected_version_release, \
'version in {} is {}, does not match expected_version_release {}, you need to add changelog in the spec file'\ 'version in {} is {}, does not match expected_version_release {}, you need to add changelog in the spec file'\
.format(RPM_FILE, version_release_in_changelog, expected_version_release) .format(SPEC_FILE, version_release_in_changelog, expected_version_release)
def get_expected_version_release():
global_version = get_global_value('version')
global_release = get_global_value('release')
return global_version + '-' + global_release
def get_version_for_changelog(file_path): def get_version_for_changelog(file_path):
...@@ -87,8 +90,6 @@ def get_release_for_file(file_path): ...@@ -87,8 +90,6 @@ def get_release_for_file(file_path):
for line in lines: for line in lines:
if line.startswith('Release'): if line.startswith('Release'):
return line.split(':')[1].strip().split('%')[0] return line.split(':')[1].strip().split('%')[0]
elif line.startswith('RELEASE'):
return line.split('=')[1].strip()
return None return None
......
...@@ -41,6 +41,19 @@ def test_systemd_system(mocker): ...@@ -41,6 +41,19 @@ def test_systemd_system(mocker):
assert 'start' in popen_mock.call_args[0][0] assert 'start' in popen_mock.call_args[0][0]
def test_supervisor_system_watchdog_status(mocker):
process_mock = MagicMock()
process_mock.communicate.return_value = ('RUNNING', '', )
process_mock.returncode = 0
popen_mock = mocker.patch('subprocess.Popen', return_value=process_mock)
mount_efs.start_watchdog('supervisord')
utils.assert_called_once(popen_mock)
assert 'supervisorctl' in popen_mock.call_args[0][0]
assert 'status' in popen_mock.call_args[0][0]
def test_unknown_system(mocker): def test_unknown_system(mocker):
popen_mock = mocker.patch('subprocess.Popen') popen_mock = mocker.patch('subprocess.Popen')
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment