File size: 10,331 Bytes
7c89ed7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 | JSON format
===========
This section describes the structure JSON objects to be sent to MozDef.
Using this standard ensures developers, admins, etc are configuring their application or system to be easily integrated into MozDef.
Background
**********
Mozilla used CEF as a logging standard for compatibility with Arcsight and for standardization across systems. While CEF is an admirable standard, MozDef prefers JSON logging for the following reasons:
* Every development language can create a JSON structure.
* JSON is easily parsed by computers/programs which are the primary consumer of logs.
* CEF is primarily used by Arcsight and rarely seen outside that platform and doesn't offer the extensibility of JSON.
* A wide variety of log shippers (heka, logstash, fluentd, nxlog, beaver) are readily available to meet almost any need to transport logs as JSON.
* JSON is already the standard for cloud platforms like amazon's cloudtrail logging.
Description
***********
As there is no common RFC-style standard for json logs, we prefer the following structure adapted from a combination of the graylog GELF and logstash specifications.
Note all fields are lowercase to avoid one program sending sourceIP, another sending sourceIp, another sending SourceIPAddress, etc.
Since the backend for MozDef is elasticsearch and fields are case-sensitive this will allow for easy compatibility and reduce potential confusion for those attempting to use the data.
MozDef will perform some translation of fields to a common schema but this is intended to allow the use of heka, nxlog, beaver and retain compatible logs.
Mandatory Fields
****************
+-----------------+-------------------------------------+-----------------------------------+
| Field | Purpose | Sample Value |
+=================+=====================================+===================================+
| category | General category/type of event | authentication, authorization, |
| | matching the 'what should I log' | account creation, shutdown, |
| | section below | atartup, account deletion, |
| | | account unlock, zeek |
| | | |
+-----------------+-------------------------------------+-----------------------------------+
| details | Additional, event-specific fields | <see below> |
| | that you would like included with | |
| | the event. Please completely spell | |
| | out a field rather an abbreviate: | |
| | i.e. sourceipaddress instead of | |
| | srcip. | |
+-----------------+-------------------------------------+-----------------------------------+
| hostname | The fully qualified domain name of | server1.example.com |
| | the host sending the message | |
+-----------------+-------------------------------------+-----------------------------------+
| processid | The PID of the process sending the | 1234 |
| | log | |
+-----------------+-------------------------------------+-----------------------------------+
|processname | The name of the process sending the | myprogram.py |
| | log | |
+-----------------+-------------------------------------+-----------------------------------+
| severity | RFC5424 severity level of the event | INFO |
| | in all caps: DEBUG, INFO, NOTICE, | |
| | WARNING, ERROR, CRITICAL, ALERT, | |
| | EMERGENCY | |
+-----------------+-------------------------------------+-----------------------------------+
| source | Source of the event (file name, | /var/log/syslog/2014.01.02.log |
| | system name, component name) | |
+-----------------+-------------------------------------+-----------------------------------+
| summary | Short human-readable version of the | john login attempts over |
| | event suitable for IRC, SMS, etc. | threshold, account locked |
+-----------------+-------------------------------------+-----------------------------------+
| tags | An array or list of any tags you | vpn, audit |
| | would like applied to the event | |
| | | nsm,zeek,intel |
+-----------------+-------------------------------------+-----------------------------------+
| timestamp | Full date plus time timestamp of | 2014-01-30T19:24:43+06:00 |
| | the event in ISO format including | |
| | the timezone offset | |
+-----------------+-------------------------------------+-----------------------------------+
|utctimestamp | Full UTC date plus time timestamp of| 2014-01-30T13:24:43+00:00 |
| | the event in ISO format including | |
| | the timezone offset | |
+-----------------+-------------------------------------+-----------------------------------+
|receivedtimestamp| Full UTC date plus time timestamp in| 2014-01-30T13:24:43+00:00 |
| | ISO format when mozdef parses the | |
| | event. This is set by mozdef upon | |
| | receipt of the event | |
+-----------------+-------------------------------------+-----------------------------------+
Details substructure (mandatory if such data is sent, otherwise optional)
*************************************************************************
+----------------------+--------------------------+---------------------------------+
| Field | Purpose | Sample Value |
+======================+==========================+=================================+
| destinationipaddress | Destination IP of a | 8.8.8.8 |
| | network flow | |
+----------------------+--------------------------+---------------------------------+
| destinationport | Destination port of a | 80 |
| | network flow | |
+----------------------+--------------------------+---------------------------------+
| sourceipaddress | Source IP of a network | 8.8.8.8 |
| | flow | |
+----------------------+--------------------------+---------------------------------+
| sourceport | Source port of a network | 42297 |
| | flow | |
+----------------------+--------------------------+---------------------------------+
| sourceuri | Source URI such as a | https://www.mozilla.org/ |
| | referer | |
+----------------------+--------------------------+---------------------------------+
| destinationuri | Destination URI as in | https://www.mozilla.org/ |
| | "wget this URI" | |
+----------------------+--------------------------+---------------------------------+
| error | Action resulted in an | true/false |
| | error or failure | |
+----------------------+--------------------------+---------------------------------+
| success | Transaction failed/ | true/false |
| | or succeeded | |
+----------------------+--------------------------+---------------------------------+
| username | Username, email, login, | kang@mozilla.com |
| | etc. | |
+----------------------+--------------------------+---------------------------------+
| useragent | Program agent string | curl/1.76 (Windows; 5.1) |
| | | |
+----------------------+--------------------------+---------------------------------+
Examples
********
.. code-block:: javascript
{
"timestamp": "2014-02-14T11:48:19.035762739-05:00",
"hostname": "somemachine.in.your.company.com",
"processname": "/path/to/your/program.exe",
"processid": 3380,
"severity": "INFO",
"summary": "joe login failed",
"category": "authentication",
"source": "ldap",
"tags": [
"ldap",
"adminAccess",
"failure"
],
"details": {
"username": "joe",
"task": "access to admin page /admin_secret_radioactiv",
"result": "10 authentication failures in a row",
"success": false
}
}
|