Graylog2 Extractors for FortiGate

UPDATE: I've created the JSON version of this for Graylog 1.0 in a new post here.

Graylog2 is a really powerful log monitor, but it needs some customization when it comes to specific devices.

Pointing a Fortinet FortiGate firewall to Graylog2 results in mediocre usage, due to the syslog field not getting extracted properly. Essentially you end up with all of the data just dumped into the message field.

To point the FortiGate to Graylog2, open the FortiGate console and config the syslog settings:
config log syslogd setting

Then enable and point the logging to your server:
set status enable
set server 192.168.40.50
set port 514
set facility syslog

Once Graylog2 is successfully receiving the logs from the FortiGate, you will need to use extractors for custom processing of the specific syslog messages using the Drools Rule File in Graylog2. You can read more in Graylog2's help file "Custom message rewriting/processing."


To implement the drl file, edit your graylog2.conf file and uncomment the following line, making sure that it points to your drl file:rules_file = /etc/graylog2.drl

Then create your drl file with the rules that you need. Below are the extractors that I use for FortiGate. These are most of the fields that I've run across in the messages that seemed to be of value to me. If there are others, then follow the template and add them. Please notice that some of the fields (like action) don't have quotes around the variable, but others (like app) do, and there are different patterns to match them. There are also a couple of exceptionones (service, status, and vd) that sometimes have quotes and sometimes don't have quotes, so I've got rules in there to take both into account.

The only one that I can't seem to get working correctly is msg one that is supposed to replace the "message" field with the contents of "msg." I'll update here if I can get it working.

To implement the drl file, restart your graylog2-server service, and watch the logfile to make sure that there are no errors on start up.

# Graylog2 Extractors
import org.graylog2.plugin.Message
import java.util.regex.Matcher
import java.util.regex.Pattern
import java.text.DateFormat
import java.text.ParseException

# Fortigate
rule "Fortigate source rewrite"
  when
    m : Message ( message matches ".+devname=.+\\sdevid=.+" )
  then
    Matcher matcher = Pattern.compile("^.+\\sdevname=(\\S+)\\s").matcher(m.getMessage());
    if (matcher.find()) {
      m.addField("source", matcher.group(1));
    }

    Matcher action = Pattern.compile("^.+\\saction=(\\S+)\\s").matcher(m.getMessage());
    if (action.find()) {
      m.addField("action", action.group(1));
    }

    Matcher app = Pattern.compile("^.+\\sapp=\"(\\S+)\"").matcher(m.getMessage());
    if (app.find()) {
      m.addField("app", app.group(1));
    }

    Matcher appact = Pattern.compile("^.+\\sappact=(\\S+)\\s").matcher(m.getMessage());
    if (appact.find()) {
      m.addField("appact", appact.group(1));
    }

    Matcher appcat = Pattern.compile("^.+\\sappcat=\"(\\S+)\"").matcher(m.getMessage());
    if (appcat.find()) {
      m.addField("appcat", appcat.group(1));
    }

    Matcher applist = Pattern.compile("^.+\\sapplist=\"(\\S+)\"").matcher(m.getMessage());
    if (applist.find()) {
      m.addField("applist", applist.group(1));
    }

    Matcher attack = Pattern.compile("^.+\\sattack=\"(\\S+)\"").matcher(m.getMessage());
    if (attack.find()) {
      m.addField("attack", attack.group(1));
    }

    Matcher devid = Pattern.compile("^.+\\sdevid=(\\S+)\\s").matcher(m.getMessage());
    if (devid.find()) {
      m.addField("devid", devid.group(1));
    }

    Matcher dir = Pattern.compile("^.+\\sdir=(\\S+)\\s").matcher(m.getMessage());
    if (dir.find()) {
      m.addField("dir", dir.group(1));
    }

    Matcher dstcountry = Pattern.compile("^.+\\sdstcountry=\"(\\S+)\"").matcher(m.getMessage());
    if (dstcountry.find()) {
      m.addField("dstcountry", dstcountry.group(1));
    }

    Matcher dstintf = Pattern.compile("^.+\\sdstintf=\"(\\S+)\"").matcher(m.getMessage());
    if (dstintf.find()) {
      m.addField("dstintf", dstintf.group(1));
    }

    Matcher dstip = Pattern.compile("^.+\\sdstip=(\\S+)\\s").matcher(m.getMessage());
    if (dstip.find()) {
      m.addField("dstip", dstip.group(1));
    }

     Matcher dtype = Pattern.compile("^.+\\sdtype=\"(\\S+)\"").matcher(m.getMessage());
    if (dtype.find()) {
      m.addField("dtype", dtype.group(1));
    }

    Matcher duration = Pattern.compile("^.+\\sduration=(\\S+)\\s").matcher(m.getMessage());
    if (duration.find()) {
      m.addField("duration", duration.group(1));
    }

    Matcher error_reason = Pattern.compile("^.+\\serror_reason=\"(\\S+)\"").matcher(m.getMessage());
    if (error_reason.find()) {
      m.addField("error_reason", error_reason.group(1));
    }

    Matcher eventtype = Pattern.compile("^.+\\seventtype=(\\S+)\\s").matcher(m.getMessage());
    if (eventtype.find()) {
      m.addField("eventtype", eventtype.group(1));
    }

     Matcher file = Pattern.compile("^.+\\sfile=\"(\\S+)\"").matcher(m.getMessage());
    if (file.find()) {
      m.addField("file", file.group(1));
    }

    Matcher group = Pattern.compile("^.+\\sgroup=\"(\\S+)\"").matcher(m.getMessage());
    if (group.find()) {
      m.addField("group", group.group(1));
    }

    Matcher hostname = Pattern.compile("^.+\\shostname=\"(\\S+)\"").matcher(m.getMessage());
    if (hostname.find()) {
      m.addField("hostname", hostname.group(1));
    }

    Matcher identidx = Pattern.compile("^.+\\sidentidx=(\\S+)\\s").matcher(m.getMessage());
    if (identidx.find()) {
      m.addField("identidx", identidx.group(1));
    }

    Matcher init = Pattern.compile("^.+\\sinit=(\\S+)\\s").matcher(m.getMessage());
    if (init.find()) {
      m.addField("init", init.group(1));
    }

    Matcher locip = Pattern.compile("^.+\\slocip=(\\S+)\\s").matcher(m.getMessage());
    if (locip.find()) {
      m.addField("locip", locip.group(1));
    }

    Matcher locport = Pattern.compile("^.+\\slocport=(\\S+)\\s").matcher(m.getMessage());
    if (locport.find()) {
      m.addField("locport", locport.group(1));
    }

    Matcher logid = Pattern.compile("^.+\\slogid=(\\S+)\\s").matcher(m.getMessage());
    if (logid.find()) {
      m.addField("logid", logid.group(1));
    }

    Matcher mode = Pattern.compile("^.+\\smode=(\\S+)\\s").matcher(m.getMessage());
    if (mode.find()) {
      m.addField("mode", mode.group(1));
    }

    Matcher msg = Pattern.compile("^.+\\smsg=\"(\\S+)\"").matcher(m.getMessage());
    if (msg.find()) {
      m.addField("message", msg.group(1));
    }

    Matcher outintf = Pattern.compile("^.+\\soutintf=\"(\\S+)\"").matcher(m.getMessage());
    if (outintf.find()) {
      m.addField("outintf", outintf.group(1));
    }

    Matcher peer_notif = Pattern.compile("^.+\\speer_notif=\"(\\S+)\"").matcher(m.getMessage());
    if (peer_notif.find()) {
      m.addField("peer_notif", peer_notif.group(1));
    }

    Matcher policyid = Pattern.compile("^.+\\spolicyid=(\\S+)\\s").matcher(m.getMessage());
    if (policyid.find()) {
      m.addField("policyid", policyid.group(1));
    }

    Matcher profile = Pattern.compile("^.+\\sprofile=\"(\\S+)\"").matcher(m.getMessage());
    if (profile.find()) {
      m.addField("profile", profile.group(1));
    }

    Matcher profiletype = Pattern.compile("^.+\\sprofiletype=\"(\\S+)\"").matcher(m.getMessage());
    if (profiletype.find()) {
      m.addField("profiletype", profiletype.group(1));
    }

    Matcher proto = Pattern.compile("^.+\\sproto=(\\S+)\\s").matcher(m.getMessage());
    if (proto.find()) {
      m.addField("proto", proto.group(1));
    }

     Matcher quarskip = Pattern.compile("^.+\\squarskip=\"(\\S+)\"").matcher(m.getMessage());
    if (quarskip.find()) {
      m.addField("quarskip", quarskip.group(1));
    }

    Matcher rcvdbyte = Pattern.compile("^.+\\srcvdbyte=(\\S+)\\s").matcher(m.getMessage());
    if (rcvdbyte.find()) {
      m.addField("rcvdbyte", rcvdbyte.group(1));
    }

    Matcher rcvdpkt = Pattern.compile("^.+\\srcvdpkt=(\\S+)\\s").matcher(m.getMessage());
    if (rcvdpkt.find()) {
      m.addField("rcvdpkt", rcvdpkt.group(1));
    }

    Matcher ref = Pattern.compile("^.+\\sref=\"(\\S+)\"").matcher(m.getMessage());
    if (ref.find()) {
      m.addField("ref", ref.group(1));
    }

    Matcher remip = Pattern.compile("^.+\\sremip=(\\S+)\\s").matcher(m.getMessage());
    if (remip.find()) {
      m.addField("remip", remip.group(1));
    }

    Matcher remport = Pattern.compile("^.+\\sremport=(\\S+)\\s").matcher(m.getMessage());
    if (remport.find()) {
      m.addField("remport", remport.group(1));
    }

    Matcher result = Pattern.compile("^.+\\sresult=(\\S+)\\s").matcher(m.getMessage());
    if (result.find()) {
      m.addField("result", result.group(1));
    }

    Matcher role = Pattern.compile("^.+\\srole=(\\S+)\\s").matcher(m.getMessage());
    if (role.find()) {
      m.addField("role", role.group(1));
    }

    Matcher sentbyte = Pattern.compile("^.+\\ssentbyte=(\\S+)\\s").matcher(m.getMessage());
    if (sentbyte.find()) {
      m.addField("sentbyte", sentbyte.group(1));
    }

    Matcher sentpkt = Pattern.compile("^.+\\ssentpkt=(\\S+)\\s").matcher(m.getMessage());
    if (sentpkt.find()) {
      m.addField("sentpkt", sentpkt.group(1));
    }

    Matcher service = Pattern.compile("^.+\\sservice=\"(\\S+)\"").matcher(m.getMessage());
    if (service.find()) {
      m.addField("service", service.group(1));
    } else {
      Matcher servicenq = Pattern.compile("^.+\\sservice=(\\S+)\\s").matcher(m.getMessage());
      if (servicenq.find()) {
        m.addField("service", servicenq.group(1));
      }
    }

    Matcher srccountry = Pattern.compile("^.+\\ssrccountry=\"(\\S+)\"").matcher(m.getMessage());
    if (srccountry.find()) {
      m.addField("srccountry", srccountry.group(1));
    }

    Matcher srcintf = Pattern.compile("^.+\\ssrcintf=\"(\\S+)\"").matcher(m.getMessage());
    if (srcintf.find()) {
      m.addField("srcintf", srcintf.group(1));
    }

    Matcher srcip = Pattern.compile("^.+\\ssrcip=(\\S+)\\s").matcher(m.getMessage());
    if (srcip.find()) {
      m.addField("srcip", srcip.group(1));
    }

    Matcher srcport = Pattern.compile("^.+\\ssrcport=(\\S+)\\s").matcher(m.getMessage());
    if (srcport.find()) {
      m.addField("srcport", srcport.group(1));
    }

    Matcher stage = Pattern.compile("^.+\\sstage=(\\S+)\\s").matcher(m.getMessage());
    if (stage.find()) {
      m.addField("stage", stage.group(1));
    }

    Matcher status = Pattern.compile("^.+\\sstatus=\"(\\S+)\"").matcher(m.getMessage());
    if (status.find()) {
      m.addField("status", status.group(1));
    } else {
      Matcher statusnq = Pattern.compile("^.+\\sstatus=(\\S+)\\s").matcher(m.getMessage());
      if (statusnq.find()) {
        m.addField("status", statusnq.group(1));
      }
    }

    Matcher subtype = Pattern.compile("^.+\\ssubtype=(\\S+)\\s").matcher(m.getMessage());
    if (subtype.find()) {
      m.addField("subtype", subtype.group(1));
    }

    Matcher transport = Pattern.compile("^.+\\stransport=(\\S+)\\s").matcher(m.getMessage());
    if (transport.find()) {
      m.addField("transport", transport.group(1));
    }

    Matcher type = Pattern.compile("^.+\\stype=(\\S+)\\s").matcher(m.getMessage());
    if (type.find()) {
      m.addField("type", type.group(1));
    }

    Matcher trandisp = Pattern.compile("^.+\\strandisp=(\\S+)\\s").matcher(m.getMessage());
    if (trandisp.find()) {
      m.addField("trandisp", trandisp.group(1));
    }

    Matcher transip = Pattern.compile("^.+\\stransip=(\\S+)\\s").matcher(m.getMessage());
    if (transip.find()) {
      m.addField("transip", transip.group(1));
    }

    Matcher user = Pattern.compile("^.+\\suser=\"(\\S+)\"").matcher(m.getMessage());
    if (user.find()) {
      m.addField("user", user.group(1));
    }

    Matcher utmaction = Pattern.compile("^.+\\sutmaction=(\\S+)\\s").matcher(m.getMessage());
    if (utmaction.find()) {
      m.addField("utmaction", utmaction.group(1));
    }

    Matcher utmevent = Pattern.compile("^.+\\sutmevent=(\\S+)\\s").matcher(m.getMessage());
    if (utmevent.find()) {
      m.addField("utmevent", utmevent.group(1));
    }

    Matcher vd = Pattern.compile("^.+\\svd=\"(\\S+)\"").matcher(m.getMessage());
    if (vd.find()) {
      m.addField("vd", vd.group(1));
    } else {
      Matcher vdnq = Pattern.compile("^.+\\svd=(\\S+)\\s").matcher(m.getMessage());
      if (vdnq.find()) {
        m.addField("vd", vdnq.group(1));
      }
    }

    Matcher virus = Pattern.compile("^.+\\svirus=\"(\\S+)\"").matcher(m.getMessage());
    if (virus.find()) {
      m.addField("virus", virus.group(1));
    }

    Matcher vpntunnel = Pattern.compile("^.+\\svpntunnel=\"(\\S+)\"").matcher(m.getMessage());
    if (vpntunnel.find()) {
      m.addField("vpntunnel", vpntunnel.group(1));
    }
   
    Matcher xauthgroup = Pattern.compile("^.+\\sxauthgroup=\"(\\S+)\"").matcher(m.getMessage());
    if (xauthgroup.find()) {
      m.addField("xauthgroup", xauthgroup.group(1));
    }

    Matcher xauthuser = Pattern.compile("^.+\\sxauthuser=\"(\\S+)\"").matcher(m.getMessage());
    if (xauthuser.find()) {
      m.addField("xauthuser", xauthuser.group(1));
    }

  end

------
Dustin Shaw
VCP