Log File Analysis with GoAccess

trellis
1

My client is asking about details regarding web traffic. I am used to AWStats on shared hosting.

Go Access looks fairly popular.

I think I have two questions:

  1. If I install Go Access directly on the server, would it be overwritten when I redeploy?
  2. Is there an easier way to show the client what kind of traffic they are seeing?

Thanks.

2
  1. What good would a deploy procedure be if it trashed your entire server and its supporting binaries?
  2. None that involves Roots which I’m aware of.
1 Like
3

Pretty sweet.

sudo apt-get install goaccess

Now I can say:

sudo goaccess -f /var/log/nginx/access.log

And see the analysis in the terminal. However apt-get is installing GoAccess version 0.6 and the current version is 1.0.2. Not sure how to get a newer version with apt-get so followed the manual installation instructions:

$ wget http://tar.goaccess.io/goaccess-1.0.2.tar.gz
$ tar -xzvf goaccess-1.0.2.tar.gz
$ cd goaccess-1.0.2/
$ ./configure --enable-geoip --enable-utf8
$ make
# make install

EXCEPT that I was not able to get --enable-geoip flag to work:

checking for GeoIP_new in -lGeoIP... no
configure: error: *** Missing development files for the GeoIP library

Supposedly this was necessary for the utf8 option:

sudo apt-get install libncursesw5-dev

Wait…

apt-get install php5-geoip php5-dev libgeoip-dev

And:

./configure --enable-geoip --enable-utf8

Yay!

make
sudo make install

NOTE: One could install and use without GeoIP.

Now I need to add a time format to the goaccess.conf file located at:

 /usr/local/etc/goaccess.conf

Uncommented the first option:

# The following time format works with any of the
# Apache/NGINX's log formats below.
#
time-format %H:%M:%S

And Date format, Log format:

date-format %d/%b/%Y
log-format %h %^[%d:%t %^] "%r" %s %b

(or use this flag: --log-format=COMBINED

And finally:

$ sudo goaccess -f /var/log/nginx/access.log.1 -a -o report.html

Still haven’t got the websockets version working. Will aim to report back.

$ sudo goaccess -f /var/log/nginx/access.log -o report.html --real-time-html

Hmmm. Nothing. nothing. nothing. CTRL+C

$ sudo goaccess -f /var/log/nginx/access.log -o report.html --real-time-html
^CSIGINT caught!
Stopping WebSocket server...

Maybe I’m missing a step in:

“To output an HTML report and set the WebSocket server to listen on port 7890 and localhost.”

It seems like there would be some terminal output if there was actually listening going on.

$telnet site.com 7890
telnet: connect to address 45.55.xxx.xxx: Connection refused

Ah. Have to have something listening first.

$ sudo goaccess -f /var/log/nginx/access.log.1 -o /usr/share/nginx/www/report.html \\
--real-time-html --ws-url=domain.com
Parsing... [84] [0/s]

Now at least telnet makes a connection:

telnet domain.com 7890
Trying 45.55.xxx.xxx...
Connected to domain.com.
Escape character is '^]'.
4

I tweaked this ansible role to get goaccess set up, which is really easy to add in to the server.yml playbook (or elsewhere) - I just added the vars in the playbook and the last line with a tag so you can run with --tags "goaccess" if you want to skip all the other steps in the playbook:

- name: WordPress Server - Install LEMP Stack with PHP 7.0 and MariaDB MySQL
  hosts: web:&{{ env }}
  vars:
    goaccess_version: "1.0.2"
    goaccess_tarball: "goaccess-{{goaccess_version}}.tar.gz"
  become: true
  roles:
    - { role: common, tags: [common] }
    - { role: swapfile, swapfile_size: 1GB, tags: [swapfile] }
    - { role: fail2ban, tags: [fail2ban] }
    - { role: ferm, tags: [ferm] }
    - { role: ntp, tags: [ntp] }
    - { role: users, tags: [users] }
    - { role: sshd, tags: [sshd] }
    - { role: mariadb, tags: [mariadb] }
    - { role: ssmtp, tags: [ssmtp, mail] }
    - { role: php, tags: [php] }
    - { role: memcached, tags: [memcached] }
    - { role: nginx, tags: [nginx] }
    - { role: logrotate, tags: [logrotate] }
    - { role: composer, tags: [composer] }
    - { role: wp-cli, tags: [wp-cli] }
    - { role: letsencrypt, tags: [letsencrypt], when: letsencrypt_enabled }
    - { role: wordpress-setup, tags: [wordpress, wordpress-setup, letsencrypt] }
    - { role: goaccess, tags: [goaccess] }

Then I set up main.yml in /roles/goaccess/tasks/ with the following:

---
# These tasks install goaccess from source.
# - name: Update apt cache
#   apt: update_cache=yes cache_valid_time=3600
- name: Apt install required system packages.
  apt: pkg={{item}} state=installed
  with_items:
    - gcc
    - build-essential
    - geoip-database
    - libglib2.0-0
    - libglib2.0-dev
    - libgeoip-dev
    - libncursesw5-dev

- name: Create the goaccess app directory.
  file: state=directory path=/opt/goaccess

- name: Fetching goaccess archive
  get_url: url=http://tar.goaccess.io/{{goaccess_tarball}} dest=/tmp/{{goaccess_tarball}}

- name: Unpack goaccess tarball to app dir.
  command: tar -xzvf {{goaccess_tarball}} --strip-components=1  --no-same-owner -C /opt/goaccess chdir=/tmp creates=/opt/goaccess/configure

- name: Configure.
  command: ./configure --enable-geoip --enable-utf8 chdir=/opt/goaccess creates=/opt/goaccess/Makefile

- name: Make.
  command: make chdir=/opt/goaccess creates=/opt/goaccess/goaccess

- name: Make install.
  command: make install chdir=/opt/goaccess creates=/usr/local/bin/goaccess

I’ll put it together in an ansible-galaxy role so it can be easily added with just one command from terminal :slight_smile:

6 Likes