Setting up a Greylog server for central logging with Ansible

I’ve wanted to set up scalable central logging for my home servers for a while now, and Graylog looks like a pretty good tool. At the same time, I’ve been wanting to learn how Ansible works for a while, and this looks like a good opportunity to do so!

Install Ansible

First step is to get Ansible setup and working:

sudo pacman -Syu && sudo pacman -S ansible

Setup ansible inventory, grouping hosts that will belong to the graylog_client group to speed the client setup later on.

nano /etc/ansible/hosts

Using the YAML format as a personal preference

all:
  hosts:
    ma.ax:
  children:
    ubuntu_servers:
      hosts:
        pihole.ma.ax:
        graylog.ma.ax:
        plex.ma.ax:
        yosemite.ma.ax:
    graylog_clients:
      hosts:
        pihole.ma.ax:
        plex.ma.ax:
        yosemite.ma.ax:

Check the Ansible setup, all hosts should appear green:

max@huez ~ % ansible all -m ping      
pihole.ma.ax | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}
....

Setup a VM on Proxmox

Setup a simple VM, 2 CPU cores, 4GB ram, 20GB HDD space running Ubuntu Server 18.04.2 LTS.

Update the VM and install qemu-guest-tools before doing anything on it.

sudo apt update && sudo apt install qemu-guest-tools && sudo apt dist-upgrade

Setup the Graylog server

There’s a pretty comprehensive Ansible playbook available on the Graylog github here. So following that, the server Ansible playbook is:

- hosts: "all"
  remote_user: "max"
  become: True
  vars:
    es_enable_xpack: False
    es_instance_name: "graylog"
    es_heap_size: "1g"
    es_config:
      node.name: "graylog"
      cluster.name: "graylog"
      http.port: 9200
      transport.tcp.port: 9300
      network.host: "127.0.0.1"
    graylog_install_java: False # Elasticsearch role already installed Java
    graylog_password_secret: "*" # generate with: pwgen -s 96 1
    graylog_root_password_sha2: "*"
    graylog_http_bind_address: ":9000"
    graylog_http_publish_uri: "http://:9000/"
    graylog_http_external_uri: "http://:9000/"
  roles:
    - role: "Graylog2.graylog-ansible-role"
      tags:
        - "graylog"

Install the Graylog role, and the dependencies:

ansible-galaxy install -n -p ./roles Graylog2.graylog-ansible-role
ansible-galaxy install -r roles/Graylog2.graylog-ansible-role/requirements.yml -p ./roles

Setup the VM, specifiying the hostname and adding the “-K” flag to prompt for the sudoer password

ansible-playbook server.yml -K -i "graylog.ma.ax,"

Setup Graylog input source

To receive logs, new input sources must be defined. From the Graylog web GUI head over to System -> Input sources. Define the input source as Syslog TCP with the port changed to 9514. I am using port 9514, rather than the default 514 to avoid the privileged execution that is required for ports < 1024.

Setup the Graylog clients

Next step is to configure the clients to forward their syslog entries to the server, again there’s a good and simple guide here. I’ll make a short playbook to automate this for the ‘graylog_clients’ host group.

---
- hosts: graylog_clients
  become: yes
  tasks:  
  - name: install syslog-ng
    apt: 
      name: syslog-ng
  - name: copy graylog syslog-ng configuration to client
    copy:
      src: ./graylog.conf
      dest: /etc/syslog-ng/conf.d/
      owner: root
      group: root
      mode: '0644'
  - name: restart syslog-ng
    service:
      name: syslog-ng
      state: restarted

This installs syslog-ng, copies the config file over, and reloads syslog-ng so the new config is loaded. The graylog.conf is as follows:

# Define TCP syslog destination.
destination d_net {
    syslog("graylog.ma.ax" port(9514));
};

# Tell syslog-ng to send data from source s_src to the newly defined syslog destination.
log {
    source(s_src); # Defined in the default syslog-ng configuration.
    destination(d_net);
};

Run the playbook on the clients:

ansible-playbook client.yml -K

Check if it works

Head back over to System -> Inputs on the Graylog web GUI, click on the “Show received messages” which will direct to a search constrained by the input source. I adjusted my update time to be 1 second on the top of the page, as I only have a couple of log sources.

Generate syslogs if you need too:

logger -p error Test error, please ignore
logger -p warn Test warning, please ignore

It works!

Error checking

If no logs are coming through, check the client and server side logs.

Client syslogs:

tail -f /var/log/syslog

Graylog server logs:

tail -f /var/log/graylog-server/server.log