Log Analysis involves investigating activity stored in data logs by parsing, sorting, searching, correlating and grouping data into meaningful results. Log reviews activities may occur as part of security testing, incident investigation, or forensic auditing. Cybersecurity analyst mostly use automated tools to process massive system logs that can contain structured and unstructured data sorted by timestamps (e.g., server logs, network logs, access control logs, error logs, application logs, etc).

Log Analysis Process:

    1. Collect & Centralize
    2. Cleanse
    3. Correct
    4. Analyze
    5. Generate Results

Key Concepts:

    • Log analysis – process of reviewing, interpreting and understand computer-generated records called logs.
    • Timestamps – establishes the temporal order among a set of events and can be used to link related records from different logs
    • Linux Logs (4 types)
      • Application Logs – track the behavior of a number of applications (e.g., events, errors, warnings, and other messages)
      • Event Logs – record events that take place during the execution of a system as an audit trail and to diagnose potential problems.
      • Service Logs – The Linux OS creates a log file called /var/log/daemon.log which tracks important background services.
      • System Logs – The file /var/log/syslog contains most of the typical system activity logs.

Recommended Tools:

    1. Linux Commands – Quickly analyze, scan and search logs using combinations of Linux commands that can be linked together with the pipe ‘|’ statement ( e.g., Find, Grep, Cmp, Uniq, Wc , More, Less, Touch, Awk, Gawk, etc. ). See ‘Related Linux Commands’ below.
    2. Excel (spreadsheet option for smaller files) – Import log into excel using an Import Wizzard which uses spaces to load column data. Then use excel filters and pivot tables. 
      • Use Import Wizzard:  Data > Get Data > From Text/CSV Use standard Xcel sort, filter and column cleanup functions.
      • Use substring extract to copy part of a column ( ‘=left x’, ‘=right x’ , ‘= mid x-x’, etc)
    3. Use pivot tables to find unique records 

Linux command Example:

Analyze a file (example.log) , remove unneeded column, search for rows with the word ‘key_word’ and create an output file (example2.out).

    1. Use ‘file’ command to identify log type (e.g., data, ascii, etc.)
    2. Peek at a few rows in the file to see the layout ( $ ‘head example.log )
    3. Identify log size ( $ ‘wc -l example.log ‘ )
    4. Use iterations of the Awk command with pipes to search, parse, sort, unique, count rows, etc..
      • ($ awk ‘/keyword’/{print $1, $2}’ example.log | sort | uniq -c | wc -l )
      • search on keyword, print fields 1 & 2 , sort , find unique rows with a count, count number of row from file example.log

Other Options:

    • Programming Option: – See Python references to write custom code.
    • Tools – Splunk, Hadoop, etc.
    • Other – EPOCH Time Converter

Related Linux Commands

The following Linux commands will be useful during a lob analysis challenge:

    • head or tail – show specified # of lines (default =10). (e.g,  ‘tail -n 3 Filename’ )
    • less – show one screen at a time
    • wc – word count (bytes, characters, words, or lines) (e.g., $ wc -l filename) ;  ‘-l’ = lines
    • grep – search for a patterns ; (e.g, $ grep -i “security” filename );   ‘-i’ flag means not case sensitive
    • pipe – combine command with “|” ;(e.g., $ grep -i “security” filename | wc -l)
    • tr –  translate characters ; (e.g, $ grep “20 Jan 2017” filename | tr ‘,’ ‘\t’);  replaces commas with tabs (denoted with ‘\t’).
    • Sort –  on a column (e.g $ sort -nr -t$’\t’ -k8 filename) ;  ‘-nr’ = numeric sort reverse order;  ‘-t$’\t’’ = delimiter is the tab (‘\t’) ;  ‘-k8’ = 8th column  
    • Cmp – Compare command (e.g., cmp -l cattos.jpg kitters.jpg | gawk ‘{printf “%c”, strtonum(0$2)}’ && echo
    • Sed – select specific lines  (e.g., $ sed ‘1 d’ filename > output.txt ); ‘1 d‘ = delete the first line.
    • Cut –  remove a column (e.g., $ cut -d’,’ -f3 filename > output.txt ) ; ‘-d’ = comma-delim;  ‘f3’ = 3rd column
    • Uniq – find uniques (e.g., $ sort filename| uniq -c > authors-sorted.txt
    • Awk – replacement tool (e.g., $ awk ‘/searchword/{print $1, $2, $3}’ filename | sort | uniq -c | wc -l )

References