Hashicorp Vault Deployment in Prod Environment | Hands-On

 Project: Hands-On

• How twtech  deploys (Starts) Hashicorp Vault Server in the Production environment(prod-mode).

Scope:

Step 1,

• Make sure the Hashicorp Vault in the development environment is shutdown completely.

Step2:

• Unset (reset) development token

Step3:

• Create config.hcl  file for vault's storage, listner, api_address, cluster and UI in the prod envirornment. everything will be stored in the disk. 

Step4:

• Starting Hashicorp Vault server in prod environment with server config.

Step5:

• Understand the concepts:  Seal Vault, unseal Vault &  login

Step6:

• Access the UI of vault.

Step7:

• Reset HTTP API of vault from command line interface

Dynamic Secrets commands.

A, 

• Shutdown (stop)  the Hashicorp Vault of the development environment (mode) use the keyboard    

  Ctrl+c  

                      

           

B,  

• twtech Unsets (resets) development token

 unset VAULT_TOKEN

•  twtcch verifies that the dev server in totally shutdown.

vault status

• twtech Unsets (resets) the development Vault token

unset VAULT_TOKEN

• twtech Installs HashiCorp-Vault in Prod-mode on Ubuntu.
• twtech uses Hashicorp Vault to securely managing secrets like: API keys, passwords, and certificates.

Steps A:

• Install Hashicorp Vault on Ubuntu.

Prerequisite:

• A T2.micro instance or above.
• 8GB of storage or above.
• This is cost-saving for any ideal testing purposes.
• A script to bootstrap insallation can be got at the following link. : aivault.sh

https://github.com/Devopspat35/Package-management/blob/master/ai-vault-userdata.sh

• twtech Verifies that the packages are successfully installed (bootstrapped).

vault --version

•  verify that python3 is installed

 pip3 --version

•  verify that bycrpt is installed.

python3 -c "import bcrypt; print(bcrypt.__version__)"

Steps B:

•  We have just install vault in the production environment.
• twtech needs to secure Vault with TLS (transport Layer Security).
•  Create a home directory for Vault.

sudo mkdir /home/vault

•  twtech  needs to Grant ownership of  vault directory to vault and vault-group.

sudo chown -R vault:vault /home/vault

• twtech verifies that the vault directory is now own by vuault user.

sudo ls -al /home/vault

Step C:

•  twtech creates a private key for the Certificate Authority (CA)
• Certificate Authority (CA) is trusted authority that issues digital certificates to verify identities and enable secure connections.

openssl genpkey -algorithm RSA -out ca-key.pem -aes256

• twtech enters a pawssord (pass phrase) to always remember.

• Verify whether the private key( ca-key.pem) was created 

•  twtech Creates a CA certificate.

openssl req -key ca-key.pem -new -x509 -out ca-cert.pem -days 365

• Verify that certificate was created.

• We have now the certificate and ceritificate key.

Next,

• twtech generates the servers private key  that would be used to sign the vault certificate, enable clients to trust vault during connection.

openssl genpkey -algorithm RSA -out vault-server-key.pem

• Virify that the Hashicorp vault server private key is generated.

Next, 

• twtech creates a Certificate Signing Request (CSR).
• The CSR is a file requesting  the Certificate Authorities to sign the certificate.

openssl req -new -key vault-server-key.pem -out vault-server.csr

• twtech Verifies that the Certificate Signing Request (CSR) was created.

Next:

•  twtech creates a SAN configuration file, for CA  to approve the CSR and create Vault’s certificate.
• The certificate needs a Subject Alternative Names (SANs) for securely accessed. 
• That  allows it to associate with specified:  IP addresses, domain names, or hostnames.

sudo vi /opt/vault/tls/vault-san.cnf

[req_ext]
subjectAltName = @alt_names

[alt_names]
IP.1 = 127.0.0.1
IP.2 = 18.92.200.421  #update this to Hashicorp server's publicIP-address
DNS.1 = localhost

• twtech Verifes the contect to vault-san.cnf 

sudo cat /opt/vault/tls/vault-san.cnf

Next:

•  twtech approves the CSR , by using the CA’s key plus the certificate to create a signed certificate for Vault.

sudo openssl x509 -req -in vault-server.csr -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -out vault-server-cert.pem -days 365 -extfile /opt/vault/tls/vault-san.cnf -extensions req_ext

Step D:

• AWS  instances, has a directory for trusted CA certificates. 
• twtech Copies the CA’s certificate into the directory, so that the system  can recognizes and trusts.

For Ubuntu flavor:

sudo cp ca-cert.pem  /usr/local/share/ca-certificates/vault-ca.crt

•  twtech Verifies that the certificate was copied to the trust directory of the Hashicorp vault server ( certificate should be stored in the format .crt)

sudo ls /usr/local/share/ca-certificates/

• twtech Updates CA Certificates

sudo update-ca-certificates

Step E:

• twtech moves all the files created in vault and store in the /opt/vault/tls directory. 

sudo mv * /opt/vault/tls/

• twtech Verifies that all the files were successfully moved.

sudo ls /opt/vault/tls

• twtech updates Vault’s configuration file to point to the new certificates and ensure the api_addr is set to the correct IP address, (Public IP address of Hashicorp Vault server)

sudo vi /etc/vault.d/vault.hcl

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/vault-server-cert.pem"
  tls_key_file = "/opt/vault/tls/vault-server-key.pem"
}

api_addr = "https://18.23.170.32:8200" # this should be twtech server's public IP address

• twtech Grants vault user the necessary permissions for the /opt/vault directory. After that, we'll restart Vault to apply all the changes we've made.

sudo chown -R vault:vault /opt/vault

• twtech Verifies that the permissions are granted:

sudo ls -ll /opt/vault/tls

Step F:

•  twtech Enables vault, Restart vault and Status vault in production mode:

sudo systemctl enable vault

sudo systemctl restart vault

sudo systemctl status vault

Step G:

• twtech Initializes the Hashicorp Vault in Prod mode (env) of ubuntu flavor.
• Switch to the Vault user to start interacting with Vault

sudo -su vault

• twech verifies that user is vault.

whoami

•  twtech Set-up the Vault environment  in Prod-mode (env) & refreshes the Hashicorp Vault server.
• To make it persistent (available every time you open a shell),
• Set environmental veriable to twtech shell profile (depending on the shell): 

echo "export VAULT_ADDR='https://127.0.0.1:8200'" >> ~/.bashrc

source ~/.bashrc

• twtech verifies that the Variables are exported.

echo $VAULT_ADDR

Step H:

• twtech Initializes the Hashicorp-Vault in Prod-mode (env) on Ubuntu flavor:

vault operator init -key-shares=5 -key-threshold=3

NB:

• Vault initialized with 5 key shares and a key threshold of 3. Please securely distribute the key shares printed above. When the Vault is re-sealed,
• restarted, or stopped, you must supply at least 3 of these keys to unseal it before it can start servicing requests.
• Vault does not store the generated root key. Without at least 3 keys to reconstruct the root key, Vault will remain permanently sealed.
• It is possible to generate new unseal keys, provided you have a quorum of existing unseal keys shares. 

Step I:

• twtech Unseals the vault and log in to continue configuring  the configuration  for secure uses.

vault operator unseal

• Unseal Key (will be hidden): <unseal key>

Step J:

•  twtech LogIn to Vault
• twtech Uses root token generate at vault initialization.  
• It’s the only access credential existing, copy and save well.

vault login

Token (will be hidden): <hvs.xxxxxxxxxxxxxxxxxxx>

Step K:

•  twtech Enables Secrets Engine
• twtech Setup the secrets engine to securely store sensitive information like passwords. 
• Vault offers many different secrets engines, but for this project, twtech uses the Key/Value (KV) secrets engine.  
• twtech Enables (creates) KV Secrets Engine (path to the secret)

vault secrets enable -path=secret kv

or:

vault secrets enable -path=twtech-secrets kv

twtech Writes (Stores)  Secrets.

vault kv put secret/path username=engineerpatpat  password=patpatSuperSecret@123

or:

vault kv put twtech-secrets/path username=admin password=twtechengineerpatSecret@123

•  twtech Reads (Retrieves)  Secrets on a secrets engine

vault kv get twtech-secrets/path

or:

vault kv get secret/path

• twtech verifies  vault status:

vault status

Step L:

•  twtech SetUp User Authentication for user with userpass (custom auth method). 
• twtech avoids logging in with the root account  & root token, which is not the best practice. 
• twtech create userpass auth method.
• twtech verifies the exiting auth methods in vault

vault auth list 

• twtech enables (creates) userpass auth methods & verify.
• twtech wants to authenticate custom vault users.

vault auth enable userpass

• twtech enables AWS & other Cloud Secrets Engines.

vault secrets enable -path=aws aws

vault secrets enable -path=azure azure

• twtech Verifies that the path to aws secrets engine is enabled (created)

vault secrets list

• twtech integrates the aws credentials (aws-access token) to the path of aws vault Engine (account-authentication).

vault write aws/config/root access_key=<twtech access key> secret_key=<twtech secret access key> region=us-east-2

• twtech Sets roles and use them to generate dynamic secrets.

# twtech-ec2-role.sh

vault write aws/roles/twtech-ec2-role credential_type=iam_user policy_document=-<<EOF 

{ 

  "Version": "2012-10-17",  

  "Statement": [ 

{ 

        "Sid": "Stmt1426528957000", 

        "Effect": "Allow",

        "Action": [ 

        "ec2:*"

    ], 

   "Resource": [

       "*"

     ] 

   }

  ]

}

EOF

•  twtech reads Dynamic Secrets generated

vault read aws/creds/twtech-ec2-role

• twtech Creates a policies for users and define their respective roles.

vault policy write twtech-users-policy - << EOF

path "secret/data/*" {

 capabilities = ["create", "read", "update", "delete", "list"]
}

path "sys/*" {

    capabilities = ["create", "read", "update", "delete", "list"]
}

path "secret/data/engineer-patpat" { 

   capabilities = ["create", "read", "update", "delete", "list"]

 }

path "secret/data/engineer-foncha" {

   capabilities = ["create", "read", "update", "delete", "list"]

 }

path "secret/data/engineer-abunga" {

   capabilities = ["read"]

 }

EOF

• twtech Verifies exiting & created policy (twtech-users-policy)

vault policy list

• twtech reads the content of a policies (Read Vault policy details). 

vault policy read twtech-users-policy

• twtech  creates and attach token to policies.

 export VAULT_TOKEN="$(vault token create -field token -policy=twtech-users-policy)"

Step M:

•  twtech enables (creates) approle & other auth methods.

vault auth  enable approle 

• Permission denied: we need to login with root token to access vault. 
• How twtech troubleshoots permission denial.

Steps

A. 

• Verify Vault Status
• Check if the Vault server is running and unsealed:

vault status

• If it’s sealed, unseal it using the appropriate unseal keys.

B. 

• Verify that twtech-users Are Authenticated

vault token lookup

•  twtech-user troubleshooting must be authenticated. 
• twtech-user needs to log in appropriately as shown below. (safely without displaying the root-token) 
• Token when pasted is not visible. 

vault login

Token (will be hidden): hvs.IqonxNiCM92qtbabPK6eIxxx 

C. 

• Use Root Token (if necessary) 
• If twtech have the initial root token, try it.

export VAULT_TOKEN=<your-root-token> vault auth list 

• If it works with the root token, that confirms there was a permission issue with the current token.

D, 

• list vault policies:

vault policy list

E. 

•  twtech Lists all auth methods in vault

vault auth list

• With the above steps, twtech  should Successfully list the auth methods in the Hashicorp vault again and enable (create) approle:
• twtech enables (creates) approle & other auth methods.

vault auth  enable approle 

• twtech Verifies exitsting & created auth methods.

vault auth list

• twtech lists existing policies:

vault policy list

• twtech associates the auth methods (approle) with  policies (twtech-users-policy)

vault write auth/approle/role/twtech-role secret_id_ttl=10m token_num_uses=10 token_ttl=20m token_max_ttl=30m  secret_id_num_uses=40 token_policies=twtech-users-policy

• twtech Generates and Export Role ID.
• For external authentication, the username and password is always needed for authentication.

export ROLE_ID="$(vault read -field=role_id auth/approle/role/twtech-role/role-id)"

• twtech Generate and Export Secret ID.

export SECRET_ID="$(vault write -f -field=secret_id auth/approle/role/twtech-role/secret-id)"

• twtech writes configuration using both the role id & Secret id.

vault write auth/approle/login role_id="$ROLE_ID " secret_id="$SECRET_ID "

Step N:

•  twtech authenticates GitHub in hashicorp Vault.

Prerequisite: ( GUI) or User Interface(UI):

• Generate github-pat (personal access token)
• Create an organization in GitHub
• Create teams in that organization in GitHub

UI:

• twtech Create a team with the organization: (Think-with-Tech)

• twtech creates same organizations and same teams in Hashicorp vault .
• Using the command line interface (CLI)
• First, Verify that the authentication auth methods for gitHub is enabled(created):

vault auth list

• twtech enables (create) github auth method.

vault auth enable github

• twtech Verifies that github auth method is enabled (created) 

vault auth list

• twtech creates GitHub Organizations with CLI in vault.

vault write auth/github/config organization=Think-with-Tech

• twtech create GitHub Teams with CLI in vault.

vault write auth/github/map/teams/twtech-dev-team  values=default,application

• twtech login to vault using the github authentication method. GitHub-PAT is needed.

vault login -method=github

• GitHub Personal Access Token (will be hidden)

Step O:

• How twtech Acesses the UI with root-token (it is not recommended in prod)
• Set Up User Authentication
• Before we log into the GUI, setup users accounts  by enabling the userpass authentication.
• This is the simplest way to create a policy in vault dev mode.

We are creating a userpass for : devsecopspat

With policy: twtech-users-policy

And password: changethispassword

vault write auth/userpass/users/engineer-patpat policies=“twtech-users-policy” password=“changethispassword”

• (not safe method in prod-env because the password maybe seen or exposed)

• twtech Writes (Stores) Secrets on the path of a secret engine. 
• This method is not safe for prod because secret is seen.

vault kv put secret/twtech-secret username=admin password=changethispassword

or:

Another account.

vault kv put secret/secret username=devesecopspat  password=changethispassword

 twtech Reads (Retrieves) Secrets from a specified secret engine.

vault kv get secret/twtech-secret

vault kv get secret/secret

• twtech may revokes (delete) authentication from vault.

vault token revoke -mode path auth/github 

•  twtech may disables authentication in vault auth.

vault auth disable github

Step P:

• twtech eventually uses bcrypt,  to hash passwords in Hashicorp vault  changethispassword

python3 -c 'import bcrypt; print(bcrypt.hashpw(b"changethispassword", bcrypt.gensalt()).decode())'

Hash: 

$2b$12$WEnb9.LCSM4q4MlUTw7VH.Qp4xphQpTisV/Mk6U2ywck4vI0xxxxq

• twtech Creates users with hash from password (which makes it stronger).

vault write auth/userpass/users/engineer-patpat policies="twtech-users-policy" password_hash="\$2b\$12\$WEnb9.LCSM4q4MlUTw7VH.Qp4xphQpTisV/Mk6U2ywck4vI0INGxq"

Step Q:

•  twtech gets pubIPaddress of host server (hashicorp vault) to Access the Hashicorp on the browser.

curl ifconfig.me

• twtech curls the vault url to verifies that Hashicorp vault server is access via:  CLI.

curl -v https://<twtech-PubIPaddress>:8200

curl -v https://3.1xx.95.2:8200

• twtech  accesses Hashicorp Vault on the browser: UI

https://<twtech-pubIP-address>:8200

https://3.1xx.95.2:8200