Solarwinds' Loggly and Papertrail Default to Sending Unencrypted Logs

Solarwinds scrutiny is well-earned these days. Despite being a security company by nature and owning two logging companies - Loggly & Papertrail - I can’t help but notice they aren't taking care of their own business.

For example, if you follow the Loggly or Papertrail default instructions, your syslog will be configured to send everything over the internet as plain-text. Of course, they support TLS and have advanced/optional instructions for configuring that but after decades of experience in software and devops we know the danger of defaults against a motivated attacker.

At Wrble we offer unencrypted channels if you must but default and strongly recommend configuring TLS and it is in there by default - saving you time while strengthening security posture.

This is critical and many customers who otherwise require vendors to secure SOC2 accreditation or fill out endless security questionnaires are relying on companies that don't take it seriously for their own products' day-to-day operation.

How to conduct a self-audit of your logs

First off, do you know what you’re using for logs right now? If you aren't sure what tool you're using or someone other than you (who has maybe moved on to another project, department, or company!) originally configured Syslog on all your servers. Considering what’s at stake, it’s definitely worth a look. We recommend that everyone who sends logs, especially on older systems, review how they're sending and make sure it’s secure as can be.

Here are some tips on how to check:

Whether you're configuring a Syslog daemon, using a custom library, or a custom daemon from a vendor - there are two main ways to get logs off a system: Syslog and HTTP. Now since there are secure and insecure versions of each, we'll run through them and talk about how you can make sure that they're sending over TLS:

Syslog TCP/UDP Verify Configuration

Unencrypted Syslog traffic generally flows over port 514 on either TCP or UDP.

If it's encrypted, you should see one or both of the lines in your conf file:

ActionSendStreamDriver gtls 6514

This config file will probably be a ".conf" file in the /etc/rsyslog.d/ directory, but could also be /etc/rsyslog.conf or /etc/rsyslog.conf. If you're running inside Docker, it might be configured inside the image or on the host system so it's good to check both places.

Check Traffic

Checking configuration is great, but can be complicated and hard to tell which config file is authoritative. Here we'll go over how to check for both unencrypted and encrypted traffic originating from Syslog so you can make sure it's configured correctly:

First, let's check for unencrypted traffic on 514. We'll be using a tool called tcpdump that prints out traffic going over a specific port. You can install it via yum: yum install tcpdump or apt-get: apt-get install tcpdump (you may need to update your package cache with apt-get update).

OK, now that we have that installed here's how to print all unencrypted Syslog traffic on the TCP port:

tcpdump -i any -nn -v port 514

Once that's running it will print out any packets coming or going over TCP 514, trigger something in your application that causes a log line to be sent and see if it shows up there. If it's empty you should be all good, but there are a few more things to check, here's how to look at 514 for UDP traffic:

tcpdump -i any -nn -v udp port 514

Trigger log traffic again and see if anything shows up. If nothing shows up you're in good shape. Now we've verified the negative lack of unencrypted traffic but let's look to see if we can prove it is transiting out of the system securely by watching secure traffic:

TCP: tcpdump -i any -nn -v port 6514

UDP: tcpdump -i any -nn -v udp port 6514

Hopefully, at this point, you'll see some lines come across as logs are triggered. It will be all garbled as you're seeing the TLS packets themselves but their existence and garbled nature are a great sign... congratulations!

HTTP

If you aren't using Syslog, your logs are probably going over HTTP(S) even if it's wrapped up in several layers of vendor libraries or daemons. It's really hard to list all the configurations to check, but we can still audit the traffic with tcpdump.

HTTP(S) traffic can be pretty noisy on many hosts, when we're checking this we'll limit to traffic originating from your host as that should be much less noisy. NOTE: change "10.0.0.10" to your host's IP address in all the tcpdump commands below.

tcpdump -i any src host 10.0.0.10 and port 80

Trigger log sending and see if any traffic shows up, if it's nothing or just other outgoing HTTP connections from your box you're in good shape. Let's also check HTTPS:

tcpdump -i any src host 10.0.0.10 and port 443

Traffic here can be hard to discern from other outbound connections but look for your logging vendor's hostname as a target of one of the packets, if they're showing up here you're in good shape!

Insecure HTTPS

If a vendor is controlling the HTTPS send via a library or daemon it could still be possible for them to "break" the encryption by bypassing TLS security checks when configuring their HTTPS library. This can be especially common on old or embedded system where TLS certs are hard to configure. The most common bypasses are checking the hostname and certificate validity, here are instructions to verify both of those.

This is a little harder to verify but worth your time if you're sending critical data to your logging provider.

First, we'll need to set up an "HTTP echo server" on a separate machine like this:

docker run -d -p 443:443 danthegoodman/https-echoserver

Note that it's listening on 443 and will terminate TLS but it is not a secure server, the TLS certificate is self-signed. This is what we want in this rare case!

Now configure the box that you're application is running on by editing /etc/hosts like this:

10.0.0.1 ingest.loggingprovider.com

but replace 10.0.0.1 with the IP of the machine running your echo server and ingest.loggingprovider.com with the hostname, your logging provider uses to receive logs.

If you now trigger log lines to be sent and they show up in the echo server you'll know that they've disabled hostname and/or certificate checking as we don't have a valid certificate for their domain! Report this to your provider as that’s one big glaring security hole.

Whew, that was a lot of work!

We know security is hard and it's a confusing world out there (Syslog was first released in the '80s when security was an afterthought at best!).

But we hope these tips were able to make it a little less painful and a bit quicker. Reach out directly if I can help walk you through the securing of your systems or to learn