Install a private collector

The private collector is a small program you run inside your network. It receives syslog from your servers, network devices and firewalls, writes it to local disk, and sends it to Vigilbase OS over outbound HTTPS. It has no management port, and nothing on the internet connects to it.

What it collects

Each collector has its own generic syslog connection. For every line it receives, Vigilbase keeps the header (priority, time, host, app, process and message IDs) and a SHA-256 of the whole line. The message text is not stored, so it cannot be searched or shown as evidence. BSD-format lines (RFC 3164) have no year or time zone in their timestamp, so they get the time the collector received them.

With every line, from any source, Vigilbase also keeps the address the collector received it from and how (TCP, TLS or UDP). An event shows it as Received from under Evidence & provenance. It tells apart devices that share a listener, or that send lines without a host name. It is evidence, not identity: a UDP source address can be forged, and a NAT, load balancer, syslog relay or port proxy between the device and the collector puts its own address there. After you point a device at the collector, check Received from on one of its events. If it shows a relay or a gateway instead of the device, the address is replaced before it reaches the collector, and the collector checks that address against the listener's allowed senders (see Who may send).

Fortinet FortiGate and Palo Alto Networks firewalls have their own connectors, which parse their native logs. They send to the same collector on their own listener; see FortiGate and PAN-OS firewalls.

Before you start

• Permissions. You need the integration.manage permission. Creating a collector or a new enrollment token asks you to verify your identity.
• A Linux host or VM on the network your devices log to, with Docker, or with systemd 250 or later (the unit keeps the enrollment token as an encrypted credential, which needs systemd 250). Check with systemctl --version.
• Disk for the buffer. The collector writes every record to disk before it sends it, and never deletes data it has not sent. Size the buffer as peak GB per hour × hours of outage to ride out × 1.1. The default, 10 GiB, holds about five hours at 2 GB an hour. Use local SSD: the collector syncs every batch to disk.
• Accurate time. Run NTP (chrony, systemd-timesyncd or your hypervisor's time sync). The collector uses Vigilbase's clock for its credentials, so a wrong host clock does not stop it, but Private collectors shows Host clock … ahead or behind when the host is more than a minute off.
• Outbound HTTPS from the host to ingest.os.vigilbase.com on TCP port 443. The configuration you download fills this in as the Ingest origin. To go through a proxy, set HTTPS_PROXY for the collector. The collector does not follow redirects, so the proxy must pass requests through unchanged. If the proxy inspects TLS, add its CA to the configuration as upload.caFile. Vigilbase does not use client certificates; set upload.clientCertFile and upload.clientKeyFile only if your proxy asks for one.
• A fixed IP address on the host that your devices send to. Every listener binds only that address, not every interface.
• Your devices' IP addresses. Each listener admits only the senders you list for it.
• Inbound syslog from your devices only, on the ports you choose. Do not expose these ports to the internet; also restrict them in the host's or network's firewall where you can.

Create the collector

1. 1

   Open Integrations › Private collectors and choose New collector.

2. 2

   Name it after where it runs, for example London DC syslog, and choose Create enrollment token. Verify your identity if asked.

3. 3

   Save the token now. It is shown once, works once and expires after one hour. Choose Copy token or Download token file. If it expires before the collector uses it, choose New enrollment token in the collector's row actions (…).

4. 4

   Choose where the collector runs: Docker or Linux (systemd).

5. 5

   Enter the Listen address: the host's IP address (IPv4 or IPv6) that your devices send to. Every listener binds only this address.

6. 6

   Check the listeners. A new collector has one listener, for its generic syslog connection, on plain TCP port 5140. Change Protocol or Port if your devices need TLS or UDP. Ports must be 1024 or higher, because the collector runs without privileges.

7. 7

   Enter each listener's Allowed senders: the IP addresses of the devices that send to it, or CIDR ranges such as 192.0.2.0/28 for a group of servers, separated by commas or spaces. For a firewall, list the address it sends from (both peers of an HA pair).

8. 8

   Set Spool size (GiB) and choose Download configuration. You get collector-config.json. It holds no secrets.

The dialog also shows the Collector ID, which firewall connections ask for, and the commands for the deployment you chose. To see them again later, choose the Configuration button (the file icon) on the collector's row.

UDP has no flow control and no delivery guarantee. A UDP listener's configuration says "acceptLoss": true to record that you accept this. Use TCP or TLS wherever the sender supports it.

Who may send

Syslog has no sign-in of its own, and TLS with a server certificate protects the connection, not who opened it. So each listener in the configuration you download has a listen address and a list of allowed senders (allowedSenders):

• A TCP or TLS connection from any other address is closed before the collector reads anything from it, and a UDP datagram from one is discarded. Private collectors shows how many it refused and the latest address, so a device whose address changed is easy to spot.
• A TCP or TLS connection that sends nothing for an hour is closed, so idle connections cannot take every connection slot. A sender that notices the close opens a new connection for its next message. One that does not sends that message into the closed connection, where it is lost: the collector never receives it, so no gap is recorded. Whether FortiGate and PAN-OS notice the close has not been verified. If a device can be quiet for longer than an hour, for example a server that sends only authentication logs or a firewall that forwards only configuration logs, raise the listener's idleTimeout in collector-config.json (up to "24h"). The configuration you download sets "idleTimeout": "1h"; a listener without the setting uses the collector's default of 5 minutes.
• The address checked is the one the collector sees. Docker runs the collector with host networking for this reason: the collector binds the listen address itself and sees each device's own address. A NAT, load balancer or syslog relay in front of the collector replaces the address with its own; list that address only if you trust everything that sends through it.

If you edit the configuration by hand: vigilbase-collector check-config refuses a listener on an address other hosts can reach unless it has allowedSenders, clientCaFile (a TLS listener that requires client certificates from your CA), or "allowAnySender": true (anyone who can reach it may send; only behind a host firewall that does the filtering).

Run it with Docker

The image runs as user 65532 with a read-only root file system. Every file you mount must be readable by that user.

1. 1

   Load the image that Vigilbase support sent you for this host's architecture (linux/amd64 or linux/arm64), tagged vigilbase-collector:local, for example with docker load.

2. 2

   Install the configuration and the token:

   BashCopy

   sudo install -D -m 0644 collector-config.json /etc/vigilbase-collector/config.json
   sudo install -D -m 0400 -o 65532 -g 65532 enrollment-token /run/secrets/vigilbase-collector-enrollment-token
   

3. 3

   For a TLS listener, put its certificate and key at /etc/vigilbase-collector/tls/listener.crt and /etc/vigilbase-collector/tls/listener.key, readable by user 65532. The certificate's name must match the address your devices connect to, and they must trust its CA. A certificate from your internal CA is fine.

4. 4

   Start the collector with the docker run command from the dialog. It uses host networking (--network host), so the listeners bind the listen address directly and see each device's own address; Docker ignores -p with host networking and needs no address translation for it (Host network driver, Docker Engine on Linux). For a new collector with only its generic syslog listener, it is:

   BashCopy

   docker run -d --name vigilbase-collector --restart unless-stopped \
     --read-only --cap-drop ALL --security-opt no-new-privileges \
     -v vigilbase-collector-state:/var/lib/vigilbase-collector \
     -v /etc/vigilbase-collector:/etc/vigilbase-collector:ro \
     -v /run/secrets/vigilbase-collector-enrollment-token:/run/secrets/vigilbase-collector-enrollment-token:ro \
     --network host \
     vigilbase-collector:local run -config /etc/vigilbase-collector/config.json
   docker logs -f vigilbase-collector
   

Host networking means the container shares the host's network stack, as the systemd service does: the collector can connect to services that listen on the host, including on 127.0.0.1, and could open other unprivileged ports on the host's addresses. It opens only the listeners in its configuration and connects only to the ingest origin (or your proxy) and your DNS resolver, and the container keeps a read-only file system, no capabilities and no-new-privileges. If the host runs administration services that trust local connections, give them authentication or run the collector on another host.

Keep the vigilbase-collector-state volume. It holds the collector's identity key, its credential and the buffer. If you lose it, unsent data is lost, and once the collector has delivered data you need a new collector.

A device that can only send to port 514 cannot reach the collector directly: it runs without the privilege to bind ports below 1024. Redirect port 514 on the host to the listener's port with the host firewall (a redirect that changes only the destination port keeps the device's address), or send through a syslog relay on the host and list the relay's address as the allowed sender.

Run it with systemd

The unit runs the collector as a dynamic user with strict sandboxing. That user cannot read root-only files, so the token and any TLS key reach it as systemd credentials.

1. 1

   Check the release in the directory Vigilbase support sent you, then install the binary for this host and the unit:

   BashCopy

   sha256sum --check --ignore-missing SHA256SUMS
   sudo install -m 0755 vigilbase-collector_<version>_linux_amd64 /usr/local/bin/vigilbase-collector
   sudo install -m 0644 vigilbase-collector.service /etc/systemd/system/vigilbase-collector.service
   

   On ARM64, install the linux_arm64 binary instead. SHA256SUMS proves the files match it, not who built them, so get it from Vigilbase support over a different channel from the files.

2. 2

   Install the configuration:

   BashCopy

   sudo install -D -m 0644 collector-config.json /etc/vigilbase-collector/config.json
   

3. 3

   Store the token as an encrypted credential, then delete the plain file:

   BashCopy

   sudo mkdir -p /etc/credstore.encrypted
   sudo systemd-creds encrypt --name=enrollment-token enrollment-token /etc/credstore.encrypted/vigilbase-collector-enrollment-token
   shred -u enrollment-token
   

4. 4

   For a TLS listener, put the certificate and key at /etc/vigilbase-collector/tls/listener.crt and /etc/vigilbase-collector/tls/listener.key (root-only is fine), run sudo systemctl edit vigilbase-collector and add:

   TextCopy

   [Service]
   LoadCredential=listener.crt:/etc/vigilbase-collector/tls/listener.crt
   LoadCredential=listener.key:/etc/vigilbase-collector/tls/listener.key
   

   A configuration downloaded for Linux (systemd) already reads them from /run/credentials/vigilbase-collector.service/.

5. 5

   Check the configuration and start the collector:

   BashCopy

   vigilbase-collector check-config -config /etc/vigilbase-collector/config.json
   sudo systemctl daemon-reload
   sudo systemctl enable --now vigilbase-collector
   journalctl -u vigilbase-collector -f
   

check-config prints configuration OK or says what to fix.

Check that it works

1. 1

   Read the log. It shows collector enrolled once, and a listening line for each TCP or TLS listener. A UDP listener logs a warning that UDP can lose data.

2. 2

   Open Integrations › Private collectors. The collector shows Active, a recent Last heartbeat and its Buffer use. Before it enrolls it shows Awaiting enrollment.

3. 3

   Send a test line from a device you listed under Allowed senders to the generic syslog listener, for example from a Linux server:

   BashCopy

   logger -T -n <listen address> -P 5140 "vigilbase collector test"
   

   A line sent from the collector host itself comes from the listen address, which is refused unless you list it.

   Choose the collector's name to open its generic syslog connection. On its Activity tab, Collection jobs shows the line under Last collected and Records (24h).

4. 4

   Remove the token. It no longer works, and the collector renews its own credential. With Docker, delete /run/secrets/vigilbase-collector-enrollment-token and, the next time you recreate the container, leave out its -v line; the Configuration dialog of an enrolled collector already does. With systemd you can keep the encrypted file, or delete it together with the two lines that name it in the unit.

Send logs to the collector

Point each device at the collector's listen address, on its listener's port and protocol, in the device's native syslog format, from an address in the listener's allowed senders. Over TCP or TLS, use octet-counted framing where the device offers it: with newline framing, a line that has no <priority> and starts with a number and a space (such as 200 OK) is read as a length, so the lines after it are merged into one message or skipped.

Linux servers (rsyslog)

Forward everything over TCP with octet-counted framing, which keeps multi-line messages intact:

*.* action(type="omfwd" target="<collector address>" port="5140" protocol="tcp"
           TCP_Framing="octet-counted" queue.type="linkedList")

Then run sudo systemctl restart rsyslog. For a TLS listener, add rsyslog's TLS settings (StreamDriver, StreamDriverMode="1" and the CA file) as described in the rsyslog omfwd reference.

FortiGate and PAN-OS firewalls

FortiGate and PAN-OS connectors are Alpha: built and tested against recorded logs, not yet proven with a live firewall. Each firewall is its own connection that names the collector.

1. 1

   Copy the Collector ID from the collector's Configuration dialog.

2. 2

   Connect the firewall. In Integrations, choose Add connection, select Fortinet FortiGate or Palo Alto Networks NGFW / Panorama, and enter the collector ID when asked.

3. 3

   Download the configuration again from the collector's Configuration dialog. It now has a TLS listener for the firewall, on the next free port from 6514: enter the firewall's address under its Allowed senders (and the other listeners' senders again). Replace /etc/vigilbase-collector/config.json with it and restart the collector with the dialog's commands. No new token is needed. Until you do, Private collectors shows the firewall's datasets as waiting, then, after 30 minutes, as not receiving with no listener.

4. 4

   Import the CA that signed the listener's certificate into the firewall, then send its logs to the collector on that port, as below.

The dialog normally gives a firewall one listener for all its log types, and says "one listener carries every log type" under it. If it instead shows a separate listener, with its own port, for each of the firewall's log datasets, send each log type only to its own listener's port. A log type sent to another listener's port is quarantined (FortiGate) or refused (PAN-OS), and Private collectors shows its dataset as not receiving. On a FortiGate, set up one syslog server per listener (FortiOS has four: config log syslogd setting to config log syslogd4 setting), each with its own filter (config log syslogd2 filter and so on: the log-type options and free-style categories) that sends only that listener's log types. Reference: config log syslogd2 filter. On PAN-OS, add a syslog server profile per listener in step 1 below, then select the TRAFFIC/THREAT listener's profile in step 2 and the CONFIG/SYSTEM listener's profile in step 3.

FortiGate. In the FortiOS CLI:

config log syslogd setting
    set status enable
    set server "<collector address>"
    set mode reliable
    set port 6514
    set enc-algorithm high
    set format default
end

mode reliable sends syslog over TCP (RFC 6587), enc-algorithm high adds TLS, and format default keeps FortiOS's own key=value fields, which the connector reads. CSV, CEF, JSON and RFC 5424 formats are not supported. Use the port the Configuration dialog shows for the firewall. Reference: config log syslogd setting.

PAN-OS. In the firewall's web interface:

1. 1

   Open Device › Server Profiles › Syslog and choose Add. Add one server: the collector address, Transport SSL, the port the Configuration dialog shows, Format IETF. For SSL, PAN-OS uses TLS 1.2.

2. 2

   Open Objects › Log Forwarding and choose Add. Select the syslog profile for Traffic and Threat logs, then select the log forwarding profile on each security rule (Policies › Security, Actions tab).

3. 3

   Open Device › Log Settings. Select the syslog profile for System and Config logs.

4. 4

   Commit.

Reference: Configure syslog monitoring.

UniFi

The UniFi connector does not read UniFi's syslog (SIEM) export yet. Sent to a generic syslog listener, UniFi logs keep only their headers and hashes.

Monitor the collector

Integrations › Private collectors lists every collector with its Status, Version, Last heartbeat, Buffer use, Oldest unsent record, its Datasets and the UDP records it Dropped total. A collector with no heartbeat for two minutes shows Stale.

Datasets says whether each dataset the collector is registered to deliver is actually arriving, not only that it is registered:

• receiving: a listener carries it and Vigilbase stored a record of it in the last 30 minutes. When a firewall's log types share one listener, the listener's latest record counts for each of them; whether each type still arrives shows on the firewall's connection page.
• waiting: the collector has not enrolled, or the dataset was registered less than 30 minutes ago and has no listener or record yet.
• not receiving, with the reason: the collector is offline or revoked; the dataset is turned off; no listener (the configuration on the host predates it: download it again and restart); listener stopped (Vigilbase refuses the dataset); listener cannot bind (another program holds the address and port; the error is shown); or no records (check the device's syslog settings and that its address is in the listener's allowed senders).

It also shows how many connections and datagrams came from unlisted senders, with the latest address, a listener with every connection slot in use, and a listener that admits any sender (N more listeners admit any sender when the collector's last report left them out).

Vigilbase sends a Coverage degraded notification when a collector reports one of these:

• the buffer is more than 80% full
• data has waited more than 15 minutes to be sent
• UDP data was dropped, by the collector because the buffer was full or by the host kernel
• part of the buffer was damaged
• Vigilbase rejected a record
• a listener's dataset is refused, so the collector stopped that listener
• a listener cannot bind its address
• a listener has every connection slot in use
• a dataset registered more than 30 minutes ago has no listener in the configuration the collector runs
• the host clock is more than a minute off

The feed monitor also marks the collector's datasets Delayed, then Degraded, when records stop arriving, and sends the same notification.

When the buffer is full, the collector stops reading TCP and TLS connections, so senders slow down and nothing is lost. UDP datagrams that do not fit are dropped and counted.

Linux can also drop UDP datagrams before the collector reads them, when a listener's receive buffer is full. The collector counts these too: they are part of Dropped total, which shows them as … by the host kernel. The collector asks for a 4 MiB receive buffer per UDP listener, which Linux caps at net.core.rmem_max; when it gets less, its log says UDP receive buffer is smaller than requested. To allow it, run echo 'net.core.rmem_max=4194304' | sudo tee /etc/sysctl.d/90-vigilbase-collector.conf && sudo sysctl --system, then restart the collector. nstat -az UdpRcvbufErrors shows these drops for the whole host.

Change, move or remove a collector

• Add listeners or firewalls. Download the configuration again from Configuration, replace the file and restart. The collector declares its listeners again when it starts; no new token is needed.
• Fewer firewalls. Keep connectionId on every listener that stays, even when one firewall is left on a dataset. If you remove it, the collector carries that listener's buffer and sequence numbers over only when no other firewall's buffer for the same dataset is left on the host; otherwise that listener stops with sequence conflict (see Troubleshooting).
• Upgrade. Stop the collector (it writes queued records to disk), replace the binary or image, and start it again. It carries on where it stopped.
• New enrollment token. If the token expired, the enrollment was interrupted, or the host was offline for more than seven days, choose New enrollment token in the row actions and give it to the collector on the same host. With Docker, save it as before and recreate the container with the token mounted; with systemd, encrypt it to the same file and restart the service. Keep the state directory: the collector keeps its ID, connections and buffer.
• Move to a new host. Create a new collector there and point firewall connections at it. Never copy identity.key or credential.json between hosts. Once a collector has delivered data, Vigilbase refuses it from a host with a different state directory.
• Rename changes only its name.
• Revoke stops its credentials at once. Records still in its buffer are not delivered, and it cannot be undone.
• Delete revokes the collector, removes its generic syslog connection from Integrations and frees its name. Collected evidence is kept.

Troubleshooting