Hashicorp Vault Monitoring and Observability.

 

How twtech enables vault audit log in Json format.  This allow twtech to know who exactly is accessing the Hashicorp vault , their IPaddress, what  they have done and what they have been denied access to do.

The vault_audit.json file comes from Vault’s audit device system, which is crucial for monitoring Vault activities. Let’s break it down:

 What is vault_audit.json

vault_audit.json is typically a file-based audit log in JSON format. It's created when you enable a file audit device in Vault.

Example:

vault audit enable file file_path=/var/log/vault_audit.json

This file then logs every request and response Vault handles (excluding sensitive data like secret values, unless configured otherwise).

 How Vault Audit Works

Vault audit devices capture:

• Who made the request (identity, token, etc.)
• When it was made (timestamp)
• What was requested (e.g., secret read, policy change)
• Where the request was sent (endpoint)
• Response status (success, failure)
• Latency (time taken)

Example Log Entry (JSON format)

# json

{

  "time": "2025-04-15T13:00:00.123456Z",

  "type": "request",

  "auth": {

    "client_token": "hmac-sha256:abc123...",

    "accessor": "hmac-sha256:def456...",

    "display_name": "userpass-twtech-user",

    "identity": "abcd-1234"

  },

  "request": {

    "id": "12345678-90ab-cdef-1234-567890abcdef",

    "operation": "read",

    "path": "secret/data/twtech-user/credentials",

    "remote_address": "192.168.1.100"

  }

}

Note: Vault uses HMAC-sha256 hashes to protect sensitive information in logs.

Benefits of Using vault_audit.json

Benefit	Description
Traceability	See who did what, when, and where — full visibility into Vault operations.
Security Auditing	Detect suspicious activity like brute force attempts, unauthorized reads, or misconfigurations.
Compliance	Helps meet compliance standards (e.g., SOC 2, HIPAA, PCI-DSS) with complete audit trails.
Debugging	Troubleshoot issues like failed authentications, permission errors, or incorrect secret paths.
Analytics Integration	JSON format makes it easy to integrate with tools like Splunk, ELK (Elasticsearch, Logstash, Kibana), or SIEM systems.

 Optional Enhancements

twtech can also:

• Rotate or manage log files using tools like logrotate.
• Forward audit logs to a centralized logging platform (e.g., Fluentd, Logstash).
• Enable multiple audit devices (e.g., file + syslog + socket) for redundancy.

Project

Enable the path to : /var/log/vault_audit.json

vault audit enable file file_path=/var/log/

Troubleshooting Permission denied: vault-user need the needed permissions to access the logs.

Check who is running the Vault service:

ps aux | grep vault

If the file doesn’t exist yet, ensure Vault can write to the directory:

Find out if the log directory exit: (don’t use sudo in vault commands)

ls  /var/log/

Set ownership of the file or directory to that user (e.g., vault): touch the file /var/log/vault_audit.json Then, grant ownership to vault and vault group.

Switch to root user and run the commads.

Exit

# 1. Create the audit log file

sudo touch /var/log/vault_audit.json

# 2. Change ownership to the 'vault' user

sudo chown vault:vault /var/log/vault_audit.json

# 3. Set appropriate permissions

sudo chmod 640 /var/log/vault_audit.json

Verify if file was created 

cd /var/log/

Verify Who owns the file: vault_audit.json

ls -al /var/log

Verify that logs are exported to: /var/log/vault_audit.json

How twtech accesses the log generated (in two approaches):  

# CLI

cat /var/log/vault_audit.json

# GUI:

vault_audit.json file can be downloaded to the local mechine  for observability with ssh client:  mobaxterm.

How twtwch Selects location to download the file to:

Use any compatible json file viewer to open the file ( notepad is okay to open the file)

Elaborate logs about users: