Build custom Vault plugins

All Vault auth methods and secrets engines are considered plugins. This simple concept allows both built-in and external plugins to be treated like Legos, and enabled at multiple paths.

This tutorial walks through the basic steps to build, register, and enable external plugins.

Plugins utilize the plugin system to enable third-party secrets engines and auth methods.

It is worth noting that even though database secrets engines operate under the same underlying plugin mechanism, they are slightly different in design than plugins demonstrated in this tutorial. The database secrets engine manages multiple plugins under the same backend mount point, whereas plugins are key-value backends that function as either secret or auth methods.

Tutorial setup

For the demonstration, this tutorial walks through the mock plugins available in the hashicorp-education/learn-vault-plugins GitHub repository.

Clone the hashicorp-education/learn-vault-plugins repository.

$ git clone https://github.com/hashicorp-education/learn-vault-plugins.git

Or download the repository.

Deploy a custom secrets plugin

There are vault-plugin-auth-mock and vault-plugin-secrets-mock directories under learn-vault-plugins. First, walk through the vault-plugin-secrets-mock plugin to learn the workflow.

Compile vault-plugin-secrets-mock

1. Change the working directory to where the mock plugin is located.

   $ cd learn-vault-plugins/vault-plugin-secrets-mock

2. Compile the plugin.

   $ go build -o vault/plugins/vault-plugin-secrets-mock cmd/vault-plugin-secrets-mock/main.go

   This compiles the vault-plugin-secrets-mock plugin and outputs the generated plugin in the vault/plugins/vault-plugin-secrets-mock folder.

   $ tree.├── Makefile├── README.md├── backend.go├── cmd│   └── vault-plugin-secrets-mock│       └── main.go├── go.mod├── go.sum└── vault   └── plugins      └── vault-plugin-secrets-mock

Start Vault

Start a Vault server in -dev mode for demonstration. Also, set the -dev-plugin-dir to ./vault/plugins which is where the vault-plugin-secrets-mock plugin is generated.

$ vault server -dev -dev-root-token-id=root -dev-plugin-dir=./vault/plugins==> Vault server configuration:             Api Address: http://127.0.0.1:8200                     Cgo: disabled         Cluster Address: https://127.0.0.1:8201              Go Version: go1.14.7              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")               Log Level: info                   Mlock: supported: false, enabled: false           Recovery Mode: false                 Storage: inmem                 Version: Vault v1.5.5             Version Sha: 05fe30b1eeadd105312c529c495f107dc54b40c92020-10-26T13:37:16.280-0700 [WARN]  received Unrecognized remote plugin message:This usually means that the plugin is either invalid or simplyneeds to be recompiled to support the latest protocol. attempting as db plugin, attempting as auth/secret pluginWARNING! dev mode is enabled! In this mode, Vault runs entirely in-memoryand starts unsealed with a single unseal key. The root token is alreadyauthenticated to the CLI, so you can immediately begin using Vault.You may need to set the following environment variable:    $ export VAULT_ADDR='http://127.0.0.1:8200'The unseal key and root token are displayed below in case you want toseal/unseal the Vault or re-authenticate.Unseal Key: 9fFg4N48+onMTculzevyvqXmEe5dcPcgVouSsbT5+gI=Root Token: rootThe following dev plugins are registered in the catalog:    - vault-plugin-secrets-mockDevelopment mode should NOT be used in production installations!==> Vault server started! Log data will stream in below:## ...snip...

The output includes the following message:

The following dev plugins are registered in the catalog:    - vault-plugin-secrets-mock

Test the vault-plugin-secrets-mock plugin

The name of the plugin is vault-plugin-secrets-mock and it is a custom secrets engine.

1. Open another terminal session.

2. Set the VAULT_ADDR environmental variable.

   $ export VAULT_ADDR="http://127.0.0.1:8200"

3. Login as root.

   $ vault login root

4. Enable the mock plugin.

   $ vault secrets enable -path=mock-secrets vault-plugin-secrets-mockSuccess! Enabled the vault-plugin-secrets-mock secrets engine at: mock-secrets/

5. Check that the vault-plugin-secrets-mock mock secrets engine is enabled at mock-secrets.

   $ vault secrets listPath             Type                         Accessor                              Description----             ----                         --------                              -----------cubbyhole/       cubbyhole                    cubbyhole_a68ba238                    per-token private secret storageidentity/        identity                     identity_56891537                     identity storemock-secrets/    vault-plugin-secrets-mock    vault-plugin-secrets-mock_beb96f79    n/a## ...snip...

   Each plugin responds to read, write, list, and delete as its own behavior. Let's test the vault-plugin-secrets-mock secrets engine.

6. Write some test data in key-value format.

   $ vault write mock-secrets/test message="Hello World"Success! Data written to: vault-plugin-secrets-mock/test

7. Read the data.

   $ vault read mock-secrets/testKey        Value---        -----message    Hello World

8. You can disable the vault-plugin-secrets-mock secrets engine just like any other secrets engine.

   $ vault secrets disable mock-secretsSuccess! Disabled the secrets engine (if it existed) at: vault-plugin-secrets-mock/

9. Enter Ctrl+C in the other terminal session to terminate the running Vault dev server.

Deploy a custom auth plugin

In this section, you will walk through the mock auth plugin, vault-plugin-auth-mock.

Compile vault-plugin-auth-mock

1. Change the working directory to where the mock auth plugin is located.

   $ cd ../vault-plugin-auth-mock

2. Compile the plugin.

   $ go build -o vault/plugins/vault-plugin-auth-mock cmd/vault-plugin-auth-mock/main.go

   This compiles the vault-plugin-auth-mock plugin and outputs the generated plugin in the vault/plugins/vault-plugin-auth-mock folder.

   $ tree . ├── Makefile ├── README.md ├── backend.go ├── cmd │   └── vault-plugin-auth-mock │       └── main.go ├── go.mod ├── go.sum └── vault     └── plugins         └── vault-plugin-auth-mock

Start Vault

Start a Vault server in -dev mode for demonstration. Also, set the -dev-plugin-dir to ./vault/plugins which is where the vault-plugin-auth-mock plugin is generated.

$ vault server -dev -dev-root-token-id=root -dev-plugin-dir=./vault/plugins

The output includes the following message:

The following dev plugins are registered in the catalog:    - vault-plugin-auth-mock

Test the vault-plugin-auth-mock plugin

The name of the plugin is vault-plugin-auth-mock and it is a custom auth method.

Set the VAULT_ADDR environmental variable.

$ export VAULT_ADDR="http://127.0.0.1:8200"

Login as root.

$ vault login root

Enable the mock auth plugin.

$ vault auth enable -path=mock-auth vault-plugin-auth-mockSuccess! Enabled the vault-plugin-secrets-mock secrets engine at: mock-secrets/

Check that the vault-plugin-auth-mock auth method is enabled at mock-auth.

$ vault auth listPath          Type                      Accessor                                Description----          ----                      --------                                -----------mock-auth/    vault-plugin-auth-mock    auth_vault-plugin-auth-mock_02817f20    n/atoken/        token                     auth_token_be2271f9                     token based credentials

Test the vault-plugin-auth-mock plugin

Create a new user, john with password, password.

$ vault write auth/mock-auth/user/john password=passwordSuccess! Data written to: auth/mock-auth/user/john

List to see that a user, john was created.

$ vault list auth/mock-auth/usersKeys----john

Now, authenticate with Vault as john.

$ vault write auth/mock-auth/login user=john password=passwordKey                  Value---                  -----token                s.VOLpTp2wrxhSxAmY6jcvWlgXtoken_accessor       TOP4LX2WsaNrpJX8d3i6Swt0token_duration       30stoken_renewable      truetoken_policies       ["default" "my-policy" "other-policy"]identity_policies    []policies             ["default" "my-policy" "other-policy"]token_meta_user      john

You can disable the vault-plugin-auth-mock auth method just like any other auth methods.

$ vault auth disable mock-authSuccess! Disabled the auth method (if it existed) at: mock-auth/

Enter Ctrl+C to terminate the dev server that is running.

Setup Vault

The plugin directory is a configuration option of Vault, and can be specified in the configuration file. This setting specifies a directory that all plugin binaries must live. A plugin can not be added to Vault unless it exists in the plugin directory. There is no default for this configuration option, and if it is not set plugins can not be added to Vault.

Set plugin_directory to the desired path in the Vault configuration file. The path should exist and have proper lockdown on access permissions.

Example:

plugin_directory = "/etc/vault/vault_plugins"  # path of plugin binariesstorage "consul" {  address = "127.0.0.1:8500"  path    = "vault"}listener "tcp" {  address     = "127.0.0.1:8200"  tls_cert_file = "/path/to/fullchain.pem"  tls_key_file  = "/path/to/privkey.pem"}# ...

If the Vault server is already running, you need to tell it to reload its configuration by sending SIGHUP. If you stop and start the Vault server, you need to unseal it again.

Register in plugin catalog

Calculate the SHA256 sum of the compiled plugin binary.

$ SHA256=$(sha256sum /etc/vault/vault_plugins/vault-plugin-secrets-mock | cut -d ' ' -f1)

You can now view the calculated value.

$ echo $SHA25625526a18212b3bd85c4df7530bc2f81b20fe090530be6831ffd83c0a6966429b

Use the value to register the plugin using the Vault operator vault plugin.

$ vault plugin register -sha256=$SHA256 secret vault-plugin-secrets-mockSuccess! Registered plugin: vault-plugin-secrets-mock

Enable plugin

Enabling the plugin varies depending on if it's a secrets engine or an auth method.

If it's a secrets engine:

$ vault secrets enable -path=<mount_path> <plugin_name>

If it's an auth method:

$ vault auth enable -path=<mount_path> <plugin_name>

Help and reference

• Custom Plugin Backends
• Plugin System
• API: plugins-catalog & plugins-reload-backend