Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add some dependencies to install and make every shell command copy-pastable #2682

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Add some dependencies to install and make every shell command copy-pastable #2682

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2476cdf
Update README.md
Bbertatum May 17, 2024
8d204c0
add some sudo and make every command copy-pastable README.md
Bbertatum May 17, 2024
6b6692d
Merge branch 'master' into master
Bbertatum May 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 69 additions & 68 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,29 +101,30 @@ GoAccess can be compiled and used on *nix systems.

Download, extract and compile GoAccess with:

$ wget https://tar.goaccess.io/goaccess-1.9.2.tar.gz
$ tar -xzvf goaccess-1.9.2.tar.gz
$ cd goaccess-1.9.2/
$ ./configure --enable-utf8 --enable-geoip=mmdb
$ make
# make install
sudo apt install libncursesw5-dev libmaxminddb-dev
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
sudo apt install libncursesw5-dev libmaxminddb-dev

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's best to keep this part distro-agnostic. We can emphasize the dependencies at the bottom, where they're already listed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that it's not the "straightforward way" to have all this deps in a table like this because per-distro commands to make things work are written above (and also I missed it... thx to point it out) maybe I missed something else but these deps are not "optional" because without it the software do not compile.
But I also agree with your way to see things. Thx for your beautiful software BTW :)

wget https://tar.goaccess.io/goaccess-1.9.2.tar.gz
tar -xzvf goaccess-1.9.2.tar.gz
cd goaccess-1.9.2/
./configure --enable-utf8 --enable-geoip=mmdb
make
sudo make install

### Build from GitHub (Development) ###

$ git clone https://github.com/allinurl/goaccess.git
$ cd goaccess
$ autoreconf -fiv
$ ./configure --enable-utf8 --enable-geoip=mmdb
$ make
# make install
git clone https://github.com/allinurl/goaccess.git
cd goaccess
autoreconf -fiv
./configure --enable-utf8 --enable-geoip=mmdb
make
make install

#### Build in isolated container

You can also build the binary for Debian based systems in an isolated container environment to prevent cluttering your local system with the development libraries:

$ curl -L "https://github.com/allinurl/goaccess/archive/refs/heads/master.tar.gz" | tar -xz && cd goaccess-master
$ docker build -t goaccess/build.debian-12 -f Dockerfile.debian-12 .
$ docker run -i --rm -v $PWD:/goaccess goaccess/build.debian-12 > goaccess
curl -L "https://github.com/allinurl/goaccess/archive/refs/heads/master.tar.gz" | tar -xz && cd goaccess-master
docker build -t goaccess/build.debian-12 -f Dockerfile.debian-12 .
docker run -i --rm -v $PWD:/goaccess goaccess/build.debian-12 > goaccess

### Distributions ###

Expand All @@ -133,62 +134,62 @@ the latest version of GoAccess available.

#### Debian/Ubuntu ####

# apt-get install goaccess
apt-get install goaccess

**Note:** It is likely this will install an outdated version of GoAccess. To
make sure that you're running the latest stable version of GoAccess see
alternative option below.

#### Official GoAccess Debian & Ubuntu repository ####

$ wget -O - https://deb.goaccess.io/gnugpg.key | gpg --dearmor \
wget -O - https://deb.goaccess.io/gnugpg.key | gpg --dearmor \
| sudo tee /usr/share/keyrings/goaccess.gpg >/dev/null
$ echo "deb [signed-by=/usr/share/keyrings/goaccess.gpg arch=$(dpkg --print-architecture)] https://deb.goaccess.io/ $(lsb_release -cs) main" \
echo "deb [signed-by=/usr/share/keyrings/goaccess.gpg arch=$(dpkg --print-architecture)] https://deb.goaccess.io/ $(lsb_release -cs) main" \
| sudo tee /etc/apt/sources.list.d/goaccess.list
$ sudo apt-get update
$ sudo apt-get install goaccess
sudo apt-get update
sudo apt-get install goaccess

**Note**:
* `.deb` packages in the official repo are available through HTTPS as well. You may need to install `apt-transport-https`.

#### Fedora ####

# yum install goaccess
yum install goaccess

#### Arch ####

# pacman -S goaccess
pacman -S goaccess

#### Gentoo ####

# emerge net-analyzer/goaccess
emerge net-analyzer/goaccess

#### OS X / Homebrew ####

# brew install goaccess
brew install goaccess

#### FreeBSD ####

# cd /usr/ports/sysutils/goaccess/ && make install clean
# pkg install sysutils/goaccess
cd /usr/ports/sysutils/goaccess/ && make install clean
pkg install sysutils/goaccess

#### OpenBSD ####

# cd /usr/ports/www/goaccess && make install clean
# pkg_add goaccess
cd /usr/ports/www/goaccess && make install clean
pkg_add goaccess

#### openSUSE ####

# zypper ar -f obs://server:http http
# zypper in goaccess
zypper ar -f obs://server:http http
zypper in goaccess

#### OpenIndiana ####

# pkg install goaccess
pkg install goaccess

#### pkgsrc (NetBSD, Solaris, SmartOS, ...) ####

# pkgin install goaccess
pkgin install goaccess

#### Windows ####

Expand Down Expand Up @@ -261,48 +262,48 @@ configuration file or in the command line.

To output to a terminal and generate an interactive report:

# goaccess access.log
goaccess access.log

To generate an HTML report:

# goaccess access.log -a > report.html
goaccess access.log -a > report.html

To generate a JSON report:

To generate a JSON report file:
goaccess access.log -a -d -o report.json

# goaccess access.log -a -d -o report.json
To generate a CSV file:

To generate a CSV report to stdout:

# goaccess access.log --no-csv-summary -o csv
goaccess access.log --no-csv-summary -o csv > report.csv

GoAccess also allows great flexibility for real-time filtering and parsing. For
instance, to quickly diagnose issues by monitoring logs since goaccess was
started:

# tail -f access.log | goaccess -
tail -f access.log | goaccess -

And even better, to filter while maintaining opened a pipe to preserve
real-time analysis, we can make use of `tail -f` and a matching pattern tool
such as `grep`, `awk`, `sed`, etc.:

# tail -f access.log | grep -i --line-buffered 'firefox' | goaccess --log-format=COMBINED -
tail -f access.log | grep -i --line-buffered 'firefox' | goaccess --log-format=COMBINED -

or to parse from the beginning of the file while maintaining the pipe opened
and applying a filter

# tail -f -n +0 access.log | grep -i --line-buffered 'firefox' | goaccess -o report.html --real-time-html -
tail -f -n +0 access.log | grep -i --line-buffered 'firefox' | goaccess -o report.html --real-time-html -


### Multiple Log files ###

There are several ways to parse multiple logs with GoAccess. The simplest is to
pass multiple log files to the command line:

# goaccess access.log access.log.1
goaccess access.log access.log.1

It's even possible to parse files from a pipe while reading regular files:

# cat access.log.2 | goaccess access.log access.log.1 -
cat access.log.2 | goaccess access.log access.log.1 -

**Note**: the single dash is appended to the command line to let GoAccess
know that it should read from the pipe.
Expand All @@ -311,20 +312,20 @@ Now if we want to add more flexibility to GoAccess, we can use `zcat --force`
to read compressed and uncompressed files. For instance, if we would
like to process all log files `access.log*`, we can do:

# zcat --force access.log* | goaccess -
zcat --force access.log* | goaccess -

_Note_: On Mac OS X, use `gunzip -c` instead of `zcat`.

### Multi-thread Support ###

Use `--jobs=<count>` (or `-j`) to enable multi-thread parsing. For example:

# goaccess access.log -o report.html -j 4
goaccess access.log -o report.html -j 4


And use `--chunk-size=<256-32768>` to adjust chunk size, the default chunk size is 1024. For example:

# goaccess access.log -o report.html -j 4 --chunk-size=8192
goaccess access.log -o report.html -j 4 --chunk-size=8192


### Real-time HTML outputs ###
Expand All @@ -337,25 +338,25 @@ The process of generating a real-time HTML report is very similar to the
process of creating a static report. Only `--real-time-html` is needed to make
it real-time.

# goaccess access.log -o /usr/share/nginx/html/your_site/report.html --real-time-html
goaccess access.log -o /usr/share/nginx/html/your_site/report.html --real-time-html

To view the report you can navigate to `http://your_site/report.html`.

By default, GoAccess will use the host name of the generated report.
Optionally, you can specify the URL to which the client's browser will connect
to. See [FAQ](https://goaccess.io/faq) for a more detailed example.

# goaccess access.log -o report.html --real-time-html --ws-url=goaccess.io
goaccess access.log -o report.html --real-time-html --ws-url=goaccess.io

By default, GoAccess listens on port 7890, to use a different port other than
7890, you can specify it as (make sure the port is opened):

# goaccess access.log -o report.html --real-time-html --port=9870
goaccess access.log -o report.html --real-time-html --port=9870

And to bind the WebSocket server to a different address other than 0.0.0.0, you
can specify it as:

# goaccess access.log -o report.html --real-time-html --addr=127.0.0.1
goaccess access.log -o report.html --real-time-html --addr=127.0.0.1

**Note**: To output real time data over a TLS/SSL connection, you need to use
`--ssl-cert=<cert.crt>` and `--ssl-key=<priv.key>`.
Expand All @@ -369,21 +370,21 @@ Another useful pipe would be filtering dates out of the web log
The following will get all HTTP requests starting on `05/Dec/2010` until the
end of the file.

# sed -n '/05\/Dec\/2010/,$ p' access.log | goaccess -a -
sed -n '/05\/Dec\/2010/,$ p' access.log | goaccess -a -

or using relative dates such as yesterdays or tomorrows day:

# sed -n '/'$(date '+%d\/%b\/%Y' -d '1 week ago')'/,$ p' access.log | goaccess -a -
sed -n '/'$(date '+%d\/%b\/%Y' -d '1 week ago')'/,$ p' access.log | goaccess -a -

If we want to parse only a certain time-frame from DATE a to DATE b, we can do:

# sed -n '/5\/Nov\/2010/,/5\/Dec\/2010/ p' access.log | goaccess -a -
sed -n '/5\/Nov\/2010/,/5\/Dec\/2010/ p' access.log | goaccess -a -

If we want to preserve only certain amount of data and recycle storage, we can
keep only a certain number of days. For instance to keep & show the last 5
days:

# goaccess access.log --keep-last=5
goaccess access.log --keep-last=5

#### Virtual hosts ####

Expand All @@ -402,45 +403,45 @@ To do the same, but also use real-time filtering and parsing:

To exclude a list of virtual hosts you can do the following:

# grep -v "`cat exclude_vhost_list_file`" vhost_access.log | goaccess -
grep -v "`cat exclude_vhost_list_file`" vhost_access.log | goaccess -

#### Files, status codes and bots ####

To parse specific pages, e.g., page views, `html`, `htm`, `php`, etc. within a
request:

# awk '$7~/\.html|\.htm|\.php/' access.log | goaccess -
awk '$7~/\.html|\.htm|\.php/' access.log | goaccess -

Note, `$7` is the request field for the common and combined log format,
(without Virtual Host), if your log includes Virtual Host, then you probably
want to use `$8` instead. It's best to check which field you are shooting for,
e.g.:

# tail -10 access.log | awk '{print $8}'
tail -10 access.log | awk '{print $8}'

Or to parse a specific status code, e.g., 500 (Internal Server Error):

# awk '$9~/500/' access.log | goaccess -
awk '$9~/500/' access.log | goaccess -

Or multiple status codes, e.g., all 3xx and 5xx:

# tail -f -n +0 access.log | awk '$9~/3[0-9]{2}|5[0-9]{2}/' | goaccess -o out.html -
tail -f -n +0 access.log | awk '$9~/3[0-9]{2}|5[0-9]{2}/' | goaccess -o out.html -

And to get an estimated overview of how many bots (crawlers) are hitting your server:

# tail -F -n +0 access.log | grep -i --line-buffered 'bot' | goaccess -
tail -F -n +0 access.log | grep -i --line-buffered 'bot' | goaccess -

### Tips ###

Also, it is worth pointing out that if we want to run GoAccess at lower
priority, we can run it as:

# nice -n 19 goaccess -f access.log -a
nice -n 19 goaccess -f access.log -a

and if you don't want to install it on your server, you can still run it from
your local machine!

# ssh -n root@server 'tail -f /var/log/apache2/access.log' | goaccess -
ssh -n root@server 'tail -f /var/log/apache2/access.log' | goaccess -

**Note:** SSH requires `-n` so GoAccess can read from stdin. Also, make sure to
use SSH keys for authentication as it won't work if a passphrase is required.
Expand Down Expand Up @@ -485,16 +486,16 @@ than the one stored.

##### Examples #####

// last month access log
# goaccess access.log.1 --persist
# last month access log
goaccess access.log.1 --persist

then, load it with

// append this month access log, and preserve new data
# goaccess access.log --restore --persist
# append this month access log, and preserve new data
goaccess access.log --restore --persist

To read persisted data only (without parsing new data)

# goaccess --restore
goaccess --restore

Enjoy!
Loading