Commit 555154b7 authored by Max Beckett's avatar Max Beckett Committed by Jake Luszcz
Browse files

Initial commit of mount.efs

Includes a mount helper utility (mount.efs), an associated watchdog
process, unit tests, and build scripts.

The utility simplifies mounting and using EFS file systems and
optionally enables clients to mount over a TLS tunnel.
parents
*Issue #, if available:*
*Description of changes:*
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
amazon-efs-utils*
!dist/amazon-efs-utils.control
!dist/amazon-efs-utils.spec
.coverage
.pytest_cache
*.pyc
__pycache__/
build/
# Contributing Guidelines
Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional
documentation, we greatly value feedback and contributions from our community.
Please read through this document before submitting any issues or pull requests to ensure we have all the necessary
information to effectively respond to your bug report or contribution.
## Reporting Bugs/Feature Requests
We welcome you to use the GitHub issue tracker to report bugs or suggest features.
When filing an issue, please check [existing open](https://github.com/aws/efs-utils/issues), or [recently closed](https://github.com/aws/efs-utils/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already
reported the issue. Please try to include as much information as you can. Details like these are incredibly useful:
* A reproducible test case or series of steps
* The version of our code being used
* Any modifications you've made relevant to the bug
* Anything unusual about your environment or deployment
## Contributing via Pull Requests
Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that:
1. You are working against the latest source on the *master* branch.
2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already.
3. You open an issue to discuss any significant work - we would hate for your time to be wasted.
To send us a pull request, please:
1. Fork the repository.
2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change.
3. Ensure local tests pass.
4. Commit to your fork using clear commit messages.
5. Send us a pull request, answering any default questions in the pull request interface.
6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation.
GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and
[creating a pull request](https://help.github.com/articles/creating-a-pull-request/).
## Finding contributions to work on
Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/aws/efs-utils/labels/help%20wanted) issues is a great place to start.
## Code of Conduct
This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
opensource-codeofconduct@amazon.com with any additional questions or comments.
## Licensing
See the [LICENSE](https://github.com/aws/efs-utils/blob/master/LICENSE) file for our project's licensing. We will ask you confirm the licensing of your contribution.
We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes.
MIT License
Copyright 2017 Amazon.com, Inc. or its affiliates.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#
# Copyright 2017-2018 Amazon.com, Inc. and its affiliates. All Rights Reserved.
#
# Licensed under the MIT License. See the LICENSE accompanying this file
# for the specific language governing permissions and limitations under
# the License.
#
PACKAGE_NAME = amazon-efs-utils
SOURCE_TARBALL = $(PACKAGE_NAME).tar.gz
SPECFILE = $(PACKAGE_NAME).spec
BUILD_DIR = build/rpmbuild
export PYTHONPATH := $(shell pwd)/src
clean:
rm -rf $(BUILD_DIR)
rm -f $(SOURCE_TARBALL)
rm -f $(SPECFILE)
tarball: clean
mkdir -p $(PACKAGE_NAME)
cp -rp dist $(PACKAGE_NAME)
cp -rp src $(PACKAGE_NAME)
tar -czf $(SOURCE_TARBALL) $(PACKAGE_NAME)/*
specfile: clean
ln -sf dist/$(SPECFILE) $(SPECFILE)
sources: tarball specfile
rpm-only:
mkdir -p $(BUILD_DIR)/{SPECS,COORD_SOURCES,DATA_SOURCES,BUILD,RPMS,SOURCES,SRPMS}
cp $(SPECFILE) $(BUILD_DIR)/SPECS
cp $(SOURCE_TARBALL) $(BUILD_DIR)/SOURCES
rpmbuild -ba --define "_topdir `pwd`/$(BUILD_DIR)" $(BUILD_DIR)/SPECS/$(SPECFILE)
cp $(BUILD_DIR)/RPMS/*/*rpm build
rpm: sources rpm-only
.PHONY: test
test:
pytest
flake8
\ No newline at end of file
efs-utils
Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# efs-utils
Utilities for Amazon Elastic File System (EFS)
The `efs-utils` package has been verified against the following Linux distributions:
| Distribution | Package Type | `init` System |
| ------------ | ------------ | ------------- |
| Amazon Linux 2017.09 | `rpm` | `upstart` |
| Amazon Linux 2 | `rpm` | `systemd` |
| CentOS 7 | `rpm` | `systemd` |
| RHEL 7 | `rpm`| `systemd` |
| Debian 9 | `deb` | `systemd` |
| Ubuntu 16.04 | `deb` | `systemd` |
## Installation
### On Amazon Linux distributions
For those using Amazon Linux or Amazon Linux 2, the easiest way to install `efs-utils` is from Amazon's repositories:
```
$ sudo yum -y install amazon-efs-utils
```
### On other Linux distributions
Other distributions require building the package from source and installing it.
- Clone this repository:
```
$ git clone https://github.com/aws/efs-utils
$ cd efs-utils
```
- To build and install an RPM:
```
$ sudo yum -y install rpm-build
$ make rpm
$ sudo yum -y install build/amazon-efs-utils*rpm
```
- To build and install a Debian package:
```
$ sudo apt-get update
$ sudo apt-get -y install binutils
$ ./build-deb.sh
$ sudo apt-get -y install ./build/amazon-efs-utils*deb
```
#### Run tests
- [Set up a virtualenv](http://libzx.so/main/learning/2016/03/13/best-practice-for-virtualenv-and-git-repos.html) for efs-utils
```
$ virtualenv ~/.envs/efs-utils
$ source ~/.envs/efs-utils/bin/activate
$ pip install -r requirements.txt
```
- Run tests
```
$ make test
```
## Usage
### mount.efs
`efs-utils` includes a mount helper utility to simplify mounting and using EFS file systems.
To mount with the recommended default options, simply run:
```
$ sudo mount -t efs file-system-id efs-mount-point/
```
To mount automatically with recommended options, add an `/etc/fstab` entry like:
```
file-system-id efs-mount-point efs _netdev 0 0
```
To mount over TLS, simply add the `tls` option:
```
$ sudo mount -t efs -o tls file-system-id efs-mount-point/
```
To mount over TLS automatically, add an `/etc/fstab` entry like:
```
file-system-id efs-mount-point efs _netdev,tls 0 0
```
For more information on mounting with the mount helper, see the [documentation](https://docs.aws.amazon.com/efs/latest/ug/using-amazon-efs-utils.html).
#### amazon-efs-mount-watchdog
`efs-utils` contains a watchdog process to monitor the health of TLS mounts. This process is managed by either `upstart` or `systemd` depending on your Linux distribution, and is started automatically the first time an EFS file system is mounted over TLS.
## License Summary
This code is made available under the MIT license.
#!/usr/bin/env sh
#
# Copyright 2017-2018 Amazon.com, Inc. and its affiliates. All Rights Reserved.
#
# Licensed under the MIT License. See the LICENSE accompanying this file
# for the specific language governing permissions and limitations under
# the License.
#
set -ex
BASE_DIR=$(pwd)
BUILD_ROOT=${BASE_DIR}/build/debbuild
echo 'Cleaning deb build workspace'
rm -rf ${BUILD_ROOT}
mkdir -p ${BUILD_ROOT}
echo 'Creating application directories'
mkdir -p ${BUILD_ROOT}/etc/amazon/efs
mkdir -p ${BUILD_ROOT}/etc/init/
mkdir -p ${BUILD_ROOT}/etc/systemd/system
mkdir -p ${BUILD_ROOT}/sbin
mkdir -p ${BUILD_ROOT}/usr/bin
mkdir -p ${BUILD_ROOT}/var/log/amazon/efs
echo 'Copying application files'
install -p -m 644 dist/amazon-efs-mount-watchdog.conf ${BUILD_ROOT}/etc/init
install -p -m 644 dist/amazon-efs-mount-watchdog.service ${BUILD_ROOT}/etc/systemd/system
install -p -m 644 dist/efs-utils.conf ${BUILD_ROOT}/etc/amazon/efs
install -p -m 755 src/mount_efs/__init__.py ${BUILD_ROOT}/sbin/mount.efs
install -p -m 755 src/watchdog/__init__.py ${BUILD_ROOT}/usr/bin/amazon-efs-mount-watchdog
echo 'Copying install scripts'
install -p -m 755 dist/scriptlets/before-upgrade ${BUILD_ROOT}/preinst
install -p -m 755 dist/scriptlets/after-install-upgrade ${BUILD_ROOT}/postinst
install -p -m 755 dist/scriptlets/before-remove ${BUILD_ROOT}/prerm
install -p -m 755 dist/scriptlets/after-remove ${BUILD_ROOT}/postrm
echo 'Copying control file'
install -p -m 644 dist/amazon-efs-utils.control ${BUILD_ROOT}/control
echo 'Creating deb binary file'
echo '2.0'> ${BUILD_ROOT}/debian-binary
echo 'Setting permissions'
find ${BUILD_ROOT} -type d | xargs chmod 755;
echo 'Creating tar'
cd ${BUILD_ROOT}
tar czf control.tar.gz control preinst postinst prerm postrm --owner=0 --group=0
tar czf data.tar.gz etc sbin usr var --owner=0 --group=0
cd ${BASE_DIR}
echo 'Building deb'
DEB=${BUILD_ROOT}/amazon-efs-utils-1.0-1.deb
ar r ${DEB} ${BUILD_ROOT}/debian-binary
ar r ${DEB} ${BUILD_ROOT}/control.tar.gz
ar r ${DEB} ${BUILD_ROOT}/data.tar.gz
echo 'Copying deb to output directory'
cp ${BUILD_ROOT}/amazon-efs-utils*deb build/
#
# Copyright 2017-2018 Amazon.com, Inc. and its affiliates. All Rights Reserved.
#
# Licensed under the MIT License. See the LICENSE accompanying this file
# for the specific language governing permissions and limitations under
# the License.
#
description "Amazon EFS Mount Watchdog"
author "Amazon.com"
# Uncomment these lines to start amazon-efs-mount-watchdog automatically on boot
# start on (runlevel [345] and started network)
# stop on (runlevel [!345] or stopping network)
respawn
respawn limit 0 15
exec /usr/bin/env amazon-efs-mount-watchdog
#
# Copyright 2017-2018 Amazon.com, Inc. and its affiliates. All Rights Reserved.
#
# Licensed under the MIT License. See the LICENSE accompanying this file
# for the specific language governing permissions and limitations under
# the License.
#
[Unit]
Description=amazon-efs-mount-watchdog
[Service]
Type=simple
ExecStart=/usr/bin/env amazon-efs-mount-watchdog
KillMode=process
Restart=on-failure
RestartSec=15s
[Install]
WantedBy=multi-user.target
Package: amazon-efs-utils
Architecture: all
Version: 1.0
Section: utils
Depends: python|python2, nfs-common, stunnel4 (>= 4.56)
Priority: optional
Copyright: MIT License
Maintainer: Amazon.com, Inc. <efs-utils@amazon.com>
Description: This package provides utilities for simplifying the use of EFS file systems
#
# Copyright 2017-2018 Amazon.com, Inc. and its affiliates. All Rights Reserved.
#
# Licensed under the MIT License. See the LICENSE accompanying this file
# for the specific language governing permissions and limitations under
# the License.
#
%if 0%{?amzn1}
%global python_requires system-python
%global with_systemd 0
%else
%global python_requires python2
%global with_systemd 1
%endif
Name : amazon-efs-utils
Version : 1.1
Release : 1%{?dist}
Summary : This package provides utilities for simplifying the use of EFS file systems
Group : Amazon/Tools
License : MIT
URL : https://aws.amazon.com/efs
Packager : Amazon.com, Inc. <http://aws.amazon.com>
Vendor : Amazon.com
BuildArch : noarch
Requires : nfs-utils
Requires : stunnel >= 4.56
Requires : %{python_requires}
%if %{with_systemd}
BuildRequires : systemd
%{?systemd_requires}
%else
Requires(post) : /sbin/chkconfig
Requires(preun) : /sbin/service /sbin/chkconfig
Requires(postun) : /sbin/service
%endif
Source : %{name}.tar.gz
%description
This package provides utilities for simplifying the use of EFS file systems
%prep
%setup -n %{name}
%install
mkdir -p %{buildroot}%{_sysconfdir}/amazon/efs
%if %{with_systemd}
mkdir -p %{buildroot}%{_unitdir}
install -p -m 644 %{_builddir}/%{name}/dist/amazon-efs-mount-watchdog.service %{buildroot}%{_unitdir}
%else
mkdir -p %{buildroot}%{_sysconfdir}/init
install -p -m 644 %{_builddir}/%{name}/dist/amazon-efs-mount-watchdog.conf %{buildroot}%{_sysconfdir}/init
%endif
mkdir -p %{buildroot}/sbin
mkdir -p %{buildroot}%{_bindir}
mkdir -p %{buildroot}%{_localstatedir}/log/amazon/efs
install -p -m 644 %{_builddir}/%{name}/dist/efs-utils.conf %{buildroot}%{_sysconfdir}/amazon/efs
install -p -m 755 %{_builddir}/%{name}/src/mount_efs/__init__.py %{buildroot}/sbin/mount.efs
install -p -m 755 %{_builddir}/%{name}/src/watchdog/__init__.py %{buildroot}%{_bindir}/amazon-efs-mount-watchdog
%files
%defattr(-,root,root,-)
%if %{with_systemd}
%{_unitdir}/amazon-efs-mount-watchdog.service
%else
%config(noreplace) %{_sysconfdir}/init/amazon-efs-mount-watchdog.conf
%endif
/sbin/mount.efs
%{_bindir}/amazon-efs-mount-watchdog
/var/log/amazon
%config(noreplace) %{_sysconfdir}/amazon/efs/efs-utils.conf
%if %{with_systemd}
%post
%systemd_post amazon-efs-mount-watchdog.service
%preun
%systemd_preun amazon-efs-mount-watchdog.service
%postun
%systemd_postun_with_restart amazon-efs-mount-watchdog.service
%else
%preun
if [ $1 -eq 0 ]; then
/sbin/stop amazon-efs-mount-watchdog &> /dev/null || true
fi
%postun
if [ $1 -eq 1 ]; then
/sbin/restart amazon-efs-mount-watchdog &> /dev/null || true
fi
%endif
%clean
#
# Copyright 2017-2018 Amazon.com, Inc. and its affiliates. All Rights Reserved.
#
# Licensed under the MIT License. See the LICENSE accompanying this file
# for the specific language governing permissions and limitations under
# the License.
#
[DEFAULT]
logging_level = INFO
logging_max_bytes = 1048576
logging_file_count = 10
[mount]
dns_name_format = {fs_id}.efs.{region}.amazonaws.com
stunnel_debug_enabled = false
# Validate the certificate hostname on mount. Requires stunnel >= 5.15.
stunnel_check_cert_hostname = false
# Use OCSP to check certificate validity. Requires stunnel >= 5.10.
stunnel_check_cert_validity = false
# Define the port range that the TLS tunnel will choose from
port_range_lower_bound = 20049
port_range_upper_bound = 20449
[mount-watchdog]
enabled = true
poll_interval_sec = 1
unmount_grace_period_sec = 30
if [ $(cat /proc/1/comm) = init ]; then
/sbin/restart amazon-efs-mount-watchdog &> /dev/null || true
elif [ $(cat /proc/1/comm) = systemd ]; then
systemctl try-restart amazon-efs-mount-watchdog.service &> /dev/null || true
fi
if [ $(cat /proc/1/comm) = systemd ]; then
systemctl daemon-reload
fi
if [ $(cat /proc/1/comm) = init ]; then
/sbin/stop amazon-efs-mount-watchdog &> /dev/null || true
elif [ $(cat /proc/1/comm) = systemd ]; then
systemctl --no-reload disable amazon-efs-mount-watchdog.service &> /dev/null || true
systemctl stop amazon-efs-mount-watchdog.service &> /dev/null || true
fi
if [ $(cat /proc/1/comm) = init ]; then
/sbin/stop amazon-efs-mount-watchdog &> /dev/null || true
elif [ $(cat /proc/1/comm) = systemd ]; then
systemctl stop amazon-efs-mount-watchdog.service &> /dev/null || true
fi
attrs==17.4.0
configparser==3.5.0
coverage==4.5
enum34==1.1.6
flake8==3.5.0
funcsigs==1.0.2
mccabe==0.6.1
mock==2.0.0
pbr==3.1.1
pluggy==0.6.0
py==1.5.2
pycodestyle==2.3.1
pyflakes==1.6.0
pytest==3.4.0
pytest-cov==2.5.1
pytest-html==1.16.1
pytest-metadata==1.6.0
pytest-mock==1.6.3
six==1.11.0
[flake8]
max-line-length = 130
exclude = test
[tool:pytest]
addopts =
--verbose
--html build/pytest/index.html
--cov mount_efs
--cov watchdog
--cov-report html:build/coverage
--cov-fail-under 80
#!/usr/bin/env python
#
# Copyright 2017-2018 Amazon.com, Inc. and its affiliates. All Rights Reserved.
#
# Licensed under the MIT License. See the LICENSE accompanying this file
# for the specific language governing permissions and limitations under
# the License.
#
#
# Copy this script to /sbin/mount.efs and make sure it is executable.
#
# You will be able to mount an EFS file system by its short name, by adding it
# to /etc/fstab. The syntax of an fstab entry is:
#
# [Device] [Mount Point] [File System Type] [Options] [Dump] [Pass]
#
# Add an entry like this:
#
# fs-deadbeef /mount_point efs _netdev 0 0
#
# Using the 'efs' type will cause '/sbin/mount.efs' to be called by 'mount -a'
# for this file system. The '_netdev' option tells the init system that the
# 'efs' type is a networked file system type. This has been tested with systemd
# (Amazon Linux 2, CentOS 7, RHEL 7, Debian 9, and Ubuntu 16.04), and upstart
# (Amazon Linux 2017.09).
#
# Once there is an entry in fstab, the file system can be mounted with:
#
# sudo mount /mount_point
#
# The script will add recommended mount options, if not provided in fstab.
import getpass
import json
import logging
import os
import random
import re
import socket
import subprocess
import sys
import threading
from contextlib import contextmanager
from logging.handlers import RotatingFileHandler
try:
import ConfigParser
except ImportError:
from configparser import ConfigParser
try:
from urllib2 import urlopen, URLError
except ImportError:
from urllib.error import URLError
from urllib.request import urlopen
VERSION = 1.0
CONFIG_FILE = '/etc/amazon/efs/efs-utils.conf'
CONFIG_SECTION = 'mount'
LOG_DIR = '/var/log/amazon/efs'
LOG_FILE = 'mount.log'
STATE_FILE_DIR = '/var/run/efs'