README.md 17.7 KB
Newer Older
1
2
[![CircleCI](https://circleci.com/gh/aws/efs-utils.svg?style=svg)](https://circleci.com/gh/aws/efs-utils)

Max Beckett's avatar
Max Beckett committed
3
4
5
6
7
8
# efs-utils

Utilities for Amazon Elastic File System (EFS)

The `efs-utils` package has been verified against the following Linux distributions:

9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
| Distribution | Package Type | `init` System |
| ------------ | ------------ | ------------- |
| Amazon Linux 2017.09 | `rpm` | `upstart` |
| Amazon Linux 2 | `rpm` | `systemd` |
| CentOS 7 | `rpm` | `systemd` |
| CentOS 8 | `rpm` | `systemd` |
| RHEL 7 | `rpm`| `systemd` |
| RHEL 8 | `rpm`| `systemd` |
| Fedora 28 | `rpm` | `systemd` |
| Fedora 29 | `rpm` | `systemd` |
| Fedora 30 | `rpm` | `systemd` |
| Fedora 31 | `rpm` | `systemd` |
| Fedora 32 | `rpm` | `systemd` |
| Debian 9 | `deb` | `systemd` |
| Debian 10 | `deb` | `systemd` |
| Ubuntu 16.04 | `deb` | `systemd` |
| Ubuntu 18.04 | `deb` | `systemd` |
| Ubuntu 20.04 | `deb` | `systemd` |
| OpenSUSE Leap | `rpm` | `systemd` |
| OpenSUSE Tumbleweed | `rpm` | `systemd` |
| SLES 12 | `rpm` | `systemd` |
| SLES 15 | `rpm` | `systemd` |
Max Beckett's avatar
Max Beckett committed
31

32
33
34
35
36
37
The `efs-utils` package has been verified against the following MacOS distributions:

| Distribution | `init` System |
| ------------ | ------------- |
| MacOS Big Sur | `launchd` |

Ian Patel's avatar
Ian Patel committed
38
39
## Prerequisites

40
* `nfs-utils` (RHEL/CentOS/Amazon Linux/Fedora) or `nfs-common` (Debian/Ubuntu)
Ian Patel's avatar
Ian Patel committed
41
* OpenSSL 1.0.2+
42
* Python 3.4+
Ian Patel's avatar
Ian Patel committed
43
44
* `stunnel` 4.56+

45
46
47
48
## Optional

* `botocore` 1.12.0+

Max Beckett's avatar
Max Beckett committed
49
50
51
52
53
54
55
56
57
58
## Installation

### On Amazon Linux distributions

For those using Amazon Linux or Amazon Linux 2, the easiest way to install `efs-utils` is from Amazon's repositories:

```
$ sudo yum -y install amazon-efs-utils
```

59
60
61
62
63
64
65
66
67
68
69
70
71
### Install via AWS Systems Manager Distributor
You can now use AWS Systems Manage Distributor to automatically install or update `amazon-efs-utils`. 
Please refer to [Using AWS Systems Manager to automatically install or update Amazon EFS clients](https://docs.aws.amazon.com/efs/latest/ug/manage-efs-utils-with-aws-sys-manager.html) for more guidance.

The following are prerequisites for using AWS Systems Manager Distributor to install or update `amazon-efs-utils`:

1. AWS Systems Manager agent is installed on the distribution (For `Amazon Linux` and `Ubuntu`, AWS Systems Manager agent
is pre-installed, for other distributions, please refer to [install AWS Systems Manager agent on Linux EC2 instance](https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-install-ssm-agent.html)
for more guidance.)

2. Instance is attached with IAM role with AWS managed policy `AmazonElasticFileSystemsUtils`, this policy will enable your instance to be managed by
 AWS Systems Manager agent, also it contains permissions to support specific features.

Max Beckett's avatar
Max Beckett committed
72
73
74
75
### On other Linux distributions

Other distributions require building the package from source and installing it.

76
- To build and install an RPM:
Max Beckett's avatar
Max Beckett committed
77

78
79
If the distribution is not OpenSUSE or SLES

Max Beckett's avatar
Max Beckett committed
80
```
81
$ sudo yum -y install git rpm-build make
Max Beckett's avatar
Max Beckett committed
82
83
84
85
86
87
$ git clone https://github.com/aws/efs-utils
$ cd efs-utils
$ make rpm
$ sudo yum -y install build/amazon-efs-utils*rpm
```

88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
Otherwise

```
$ sudo zypper refresh
$ sudo zypper install -y git rpm-build make
$ git clone https://github.com/aws/efs-utils
$ cd efs-utils
$ make rpm
$ sudo zypper --no-gpg-checks install -y build/amazon-efs-utils*rpm
```

On OpenSUSE, if you see error like `File './suse/noarch/bash-completion-2.11-2.1.noarch.rpm' not found on medium 'http://download.opensuse.org/tumbleweed/repo/oss/'`
during installation of `git`, run the following commands to re-add repo OSS and NON-OSS, then run the install script above again.

```
sudo zypper ar -f -n OSS http://download.opensuse.org/tumbleweed/repo/oss/ OSS
sudo zypper ar -f -n NON-OSS http://download.opensuse.org/tumbleweed/repo/non-oss/ NON-OSS
sudo zypper refresh
```

Max Beckett's avatar
Max Beckett committed
108
109
110
111
- To build and install a Debian package:

```
$ sudo apt-get update
112
113
114
$ sudo apt-get -y install git binutils
$ git clone https://github.com/aws/efs-utils
$ cd efs-utils
Max Beckett's avatar
Max Beckett committed
115
116
117
118
$ ./build-deb.sh
$ sudo apt-get -y install ./build/amazon-efs-utils*deb
```

119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
### On MacOS Big Sur distribution

For EC2 Mac instances running macOS Big Sur, you can install amazon-efs-utils from the [homebrew-aws](https://github.com/aws/homebrew-aws) respository.
```
brew install amazon-efs-utils
```

This will install amazon-efs-utils on your EC2 Mac Instance running macOS Big Sur in the directory `/usr/local/Cellar/amazon-efs-utils`. At the end of the installation, it will print a set of commands that must be executed in order to start using efs-utils. The instructions that are printed after amazon-efs-utils and must be executed are:

```
Perform below actions to start using efs:
    sudo mkdir -p /Library/Filesystems/efs.fs/Contents/Resources
    sudo ln -s /usr/local/bin/mount.efs /Library/Filesystems/efs.fs/Contents/Resources/mount_efs

Perform below actions to stop using efs:
    sudo rm /Library/Filesystems/efs.fs/Contents/Resources/mount_efs

To enable watchdog for using TLS mounts:
    sudo cp /usr/local/Cellar/amazon-efs-utils/<version>/libexec/amazon-efs-mount-watchdog.plist /Library/LaunchAgents
    sudo launchctl load /Library/LaunchAgents/amazon-efs-mount-watchdog.plist

To disable watchdog for using TLS mounts:
    sudo launchctl unload /Library/LaunchAgents/amazon-efs-mount-watchdog.plist
```

Max Beckett's avatar
Max Beckett committed
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
#### Run tests

- [Set up a virtualenv](http://libzx.so/main/learning/2016/03/13/best-practice-for-virtualenv-and-git-repos.html) for efs-utils

```
$ virtualenv ~/.envs/efs-utils
$ source ~/.envs/efs-utils/bin/activate
$ pip install -r requirements.txt
```

- Run tests

```
$ make test
```

## Usage

### mount.efs

`efs-utils` includes a mount helper utility to simplify mounting and using EFS file systems.

To mount with the recommended default options, simply run:

```
$ sudo mount -t efs file-system-id efs-mount-point/
```

172
173
174
175
176
177
To mount file system to a specific mount target of the file system, run:

```
$ sudo mount -t efs -o mounttargetip=mount-target-ip-address file-system-id efs-mount-point/
```

178
179
180
181
182
183
To mount file system within a given network namespace, run:

```
$ sudo mount -t efs -o netns=netns-path file-system-id efs-mount-point/
```

184
185
186
187
188
189
To mount file system to the mount target in specific availability zone (e.g. us-east-1a), run:

```
$ sudo mount -t efs -o az=az-name file-system-id efs-mount-point/
```

190
To mount over TLS, simply add the `tls` option:
Max Beckett's avatar
Max Beckett committed
191
192

```
193
$ sudo mount -t efs -o tls file-system-id efs-mount-point/
Max Beckett's avatar
Max Beckett committed
194
195
```

196
To authenticate with EFS using the system’s IAM identity, add the `iam` option. This option requires the `tls` option.
Max Beckett's avatar
Max Beckett committed
197
198

```
199
200
201
202
$ sudo mount -t efs -o tls,iam file-system-id efs-mount-point/
```

To mount using an access point, use the `accesspoint=` option. This option requires the `tls` option.
203
The access point must be in the "available" state before it can be used to mount EFS.
204
205
206
207
208
209
210
211
212

```
$ sudo mount -t efs -o tls,accesspoint=access-point-id file-system-id efs-mount-point/
```

To mount your file system automatically with any of the options above, you can add entries to `/efs/fstab` like:

```
file-system-id efs-mount-point efs _netdev,tls,iam,accesspoint=access-point-id 0 0
Max Beckett's avatar
Max Beckett committed
213
214
```

215
For more information on mounting with the mount helper, see the manual page:
Max Beckett's avatar
Max Beckett committed
216
217

```
218
man mount.efs
Max Beckett's avatar
Max Beckett committed
219
220
```

221
or refer to the [documentation](https://docs.aws.amazon.com/efs/latest/ug/using-amazon-efs-utils.html).
Max Beckett's avatar
Max Beckett committed
222

223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
### MacOS 

For EC2 instances using Mac distribution, the recommended default options will perform a tls mount:

```
$ sudo mount -t efs file-system-id efs-mount-point/
```
 or
```
$ sudo mount -t efs -o tls file-system-id efs-mount-point/
```

To mount without TLS, simply add the `notls` option:

```
$ sudo mount -t efs -o notls file-system-id efs-mount-point/
```


242
### amazon-efs-mount-watchdog
Max Beckett's avatar
Max Beckett committed
243

244
`efs-utils` contains a watchdog process to monitor the health of TLS mounts. This process is managed by either `upstart` or `systemd` depending on your Linux distribution and `launchd` on Mac distribution, and is started automatically the first time an EFS file system is mounted over TLS.
Max Beckett's avatar
Max Beckett committed
245

Ian Patel's avatar
Ian Patel committed
246
247
## Upgrading stunnel for RHEL/CentOS

Max Beckett's avatar
Max Beckett committed
248
By default, when using the EFS mount helper with TLS, it enforces certificate hostname checking. The EFS mount helper uses the `stunnel` program for its TLS functionality. Please note that some versions of Linux do not include a version of `stunnel` that supports TLS features by default. When using such a Linux version, mounting an EFS file system using TLS will fail. 
Ian Patel's avatar
Ian Patel committed
249
250
251

Once you’ve installed the `amazon-efs-utils` package, to upgrade your system’s version of `stunnel`, see [Upgrading Stunnel](https://docs.aws.amazon.com/efs/latest/ug/using-amazon-efs-utils.html#upgrading-stunnel).

252
253
254
255
256
257
258
259
260
261
## Upgrading stunnel for SLES12

Run the following commands and follow the output hint of zypper package manager to upgrade the stunnel on your SLES12 instance

```bash
sudo zypper addrepo https://download.opensuse.org/repositories/security:Stunnel/SLE_12_SP5/security:Stunnel.repo
sudo zypper refresh
sudo zypper install -y stunnel
```

262
263
264
265
266
267
268
## Upgrading stunnel for MacOS

The installation installs latest stunnel available in brew repository. You can also upgrade the version of stunnel on your instance using the command below:
```
brew upgrade stunnel
```

269
## Install botocore
270

271
272
273
274
`efs-utils` uses botocore to interact with other AWS services. Please note the package type from the above table and install
botocore based on that info. If botocore is already installed and does not meet the minimum required version, 
you can upgrade the botocore by following the [upgrade botocore section](#Upgrade-botocore).
 
275
276
- Download the `get-pip.py` script
#### RPM
277
```bash
278
sudo yum -y install wget
279
```
280
281
282
283
284
285
286
287
288
289
```bash
if [[ "$(python3 -V 2>&1)" =~ ^(Python 3.5.*) ]]; then
    sudo wget https://bootstrap.pypa.io/3.5/get-pip.py -O /tmp/get-pip.py
elif [[ "$(python3 -V 2>&1)" =~ ^(Python 3.4.*) ]]; then
    sudo wget https://bootstrap.pypa.io/3.4/get-pip.py -O /tmp/get-pip.py
else
    sudo wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py
fi
```
#### DEB
290
291
292
```bash
sudo apt-get update
sudo apt-get -y install wget
293
294
295
296
297
298
299
300
301
302
303
```
```bash
if echo $(python3 -V 2>&1) | grep -e "Python 3.5"; then
    sudo wget https://bootstrap.pypa.io/3.5/get-pip.py -O /tmp/get-pip.py
elif echo $(python3 -V 2>&1) | grep -e "Python 3.4"; then
    sudo wget https://bootstrap.pypa.io/3.4/get-pip.py -O /tmp/get-pip.py
else
    sudo apt-get -y install python3-distutils
    sudo wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py
fi
```
304

305
306
307
308
309
- To install botocore on RPM
```bash
sudo python3 /tmp/get-pip.py
sudo pip3 install botocore || sudo /usr/local/bin/pip3 install botocore
```
310

311
312
- To install botocore on DEB
```bash
313
314
sudo python3 /tmp/get-pip.py
sudo pip3 install botocore || sudo /usr/local/bin/pip3 install botocore
315
```
316

317
318
#### On Debian10 and Ubuntu20, the botocore needs to be installed in specific target folder
```bash
319
320
321
322
sudo python3 /tmp/get-pip.py
sudo pip3 install --target /usr/lib/python3/dist-packages botocore || sudo /usr/local/bin/pip3 install --target /usr/lib/python3/dist-packages botocore
```

323
324
325
326
327
#### To install botocore on MacOS
```bash
sudo pip3 install botocore
```

328
329
330
331
332
333
334
335
336
337
338
339
340
341
## Upgrade botocore
Pass `--upgrade` to the corresponding installation scripts above based on system platform and distribution

```bash
sudo pip3 install botocore --upgrade
```

## Enable mount success/failure notification via CloudWatch log
`efs-utils` now support publishing mount success/failure logs to CloudWatch log. By default, this feature is disabled. There are three
steps you must follow to enable and use this feature:

### Step 1. Install botocore
Follow [install botocore section](#Install-botocore)

342
343
344
345
### Step 2. Enable CloudWatch log feature in efs-utils config file `/etc/amazon/efs/efs-utils.conf`
```bash
sudo sed -i -e '/\[cloudwatch-log\]/{N;s/# enabled = true/enabled = true/}' /etc/amazon/efs/efs-utils.conf
```
346
347
348
349
350

- For MacOS:
```bash
sudo sed -i -e '/\[cloudwatch-log\]/{N;s/# enabled = true/enabled = true/;}' /usr/local/Cellar/amazon-efs-utils/<version>/etc/amazon/efs/efs-utils.conf
```
351
352
353
You can also configure CloudWatch log group name and log retention days in the config file. 

### Step 3. Attach the CloudWatch logs policy to the IAM role attached to instance.
Henrique Rodrigues's avatar
Henrique Rodrigues committed
354
Attach AWS managed policy `AmazonElasticFileSystemsUtils` to the iam role you attached to the instance, or the aws credentials
355
356
357
358
configured on your instance.

After completing the three prerequisite steps, you will be able to see mount status notifications in CloudWatch Logs.

359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
## Optimize readahead max window size on Linux 5.4+

A change in the Linux kernel 5.4+ results a throughput regression on NFS client. With [patch](https://www.spinics.net/lists/linux-nfs/msg75018.html), starting from 5.4.\*, Kernels containing this patch now set the default read_ahead_kb size to 128 KB instead of the previous 15 MB. This read_ahead_kb is used by the Linux kernel to optimize performance on NFS read requests by defining the maximum amount of data an NFS client can pre-fetch in a read call. With the reduced value, an NFS client has to make more read calls to the file system, resulting in reduced performance.

To avoid above throughput regression, efs-utils will modify read_ahead_kb to 15 \* rsize (could be configured via mount option, 1MB by default) after mount success on Linux 5.4+. (not support on MacOS)

This optimization will be enabled by default. To disable this optimization:

```bash
sed -i "s/optimize_readahead = false/optimize_readahead = true/" /etc/amazon/efs/efs-utils.conf
```

To re-enable this optimization

```bash
sed -i "s/optimize_readahead = true/optimize_readahead = false/" /etc/amazon/efs/efs-utils.conf
```

You can mount file system with a given rsize, run:

```bash
$ sudo mount -t efs -o rsize=rsize-value-in-bytes file-system-id efs-mount-point/
```

You can also manually chose a value of read_ahead_kb to optimize read throughput on Linux 5.4+ after mount.

```bash
$ sudo bash -c "echo read-ahead-value-in-kb > /sys/class/bdi/0:$(stat -c '%d' efs-mount-point)/read_ahead_kb"
```

389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
## Using botocore to retrieve mount target ip address when dns name cannot be resolved

`efs-utils` now supports using botocore to retrieve mount target ip address when dns name cannot be resolved, e.g. 
when user is mounting a file system in another VPC. There are two prerequisites to use this feature:

### Step 1. Install botocore
Follow [install botocore section](#Install-botocore)

### Step 2. Allow DescribeMountTargets and DescribeAvailabilityZones action in the IAM policy
Allow the `elasticfilesystem:DescribeMountTargets` and `ec2:DescribeAvailabilityZones` action in your policy attached to 
the iam role you attached to the instance, or the aws credentials configured on your instance. We recommend you attach 
AWS managed policy `AmazonElasticFileSystemsUtils`.

This feature will be enabled by default. To disable this feature:

```bash
sed -i "s/fall_back_to_mount_target_ip_address_enabled = true/fall_back_to_mount_target_ip_address_enabled = false/" /etc/amazon/efs/efs-utils.conf
```

If you decide that you do not want to use this feature, but need to mount a cross-VPC file system, you can use the mounttargetip 
option to do so, using the desired mount target ip address in the mount command.

## The way to access instance metadata
`efs-utils` by default uses IMDSv2, which is a session-oriented method used to access instance metadata. If you don't want to use 
IMDSv2, you can disable the token fetching feature by running the following command:

```bash
sed -i "s/disable_fetch_ec2_metadata_token = false/disable_fetch_ec2_metadata_token = true/" /etc/amazon/efs/efs-utils.conf
```

## Use the assumed profile credentials for IAM
To authenticate with EFS using the system’s IAM identity of an awsprofile, add the `iam` option and pass the profile name to 
`awsprofile` option. These options require the `tls` option.

```
$ sudo mount -t efs -o tls,iam,aws-profile=test-profile file-system-id efs-mount-point/
```

To configure the named profile, see the [Named Profiles doc](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html)
and [Support Config File Settings doc](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-settings)
for more details. If the credentials (e.g. aws_access_key_id) are not configured in `/root/.aws/credentials` or `/root/.aws/config` 
(note that the path prefix may vary based on the root path of sudo), efs-utils will use botocore to assume the named profile. 
This will require botocore is pre-installed, please follow [install botocore section](#Install-botocore) to install botocore first.

Normally you will need to configure your profile IAM policy to make the assume works. For example, if you want to perform a
cross-account mounting, suppose you have established 
[vpc-peering-connections](https://docs.aws.amazon.com/vpc/latest/peering/create-vpc-peering-connection.html) between your vpcs,
next step you need to do is giving permission to account B so that it can assume a role in account A and then mount the file system 
that belongs to account A. You can see 
[IAM doc](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html) for more details.

After the IAM identity is setup, you can configure your awsprofile credentials or config. You can refer to 
[sdk settings](https://docs.aws.amazon.com/sdkref/latest/guide/settings-global.html). For example you can define
the profile to use the credentials of profile `default` to assume role in account A by defining the `source_profile`.

```bash
# /root/.aws/credentials
[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key_id =wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

# /root/.aws/config
[default]
...

[profile test-profile]
role_arn = <role-arn-in-account-A>
source_profile = default
```

Or you can use the credentials from IAM role attached to instance to assume the named profile, e.g.

```bash
# /root/.aws/config
[profile test-profile]
role_arn = <role-arn-in-account-A>
credential_source = Ec2InstanceMetadata
```


Max Beckett's avatar
Max Beckett committed
469
470
471
## License Summary

This code is made available under the MIT license.