40 %
60 %
Information about ID172 LOGANALY

Published on December 31, 2007

Author: VolteMort


Building a Logging Infrastructure:  Building a Logging Infrastructure Abe Singer What are we here for?:  What are we here for? Using system & application logs to improve security and reliability on your network Building a logging infrastructure that works, across UNIX & Windows environments How Do We Get There?:  How Do We Get There? Generate Useful Data Collect and Archive Extract Wisdom How do we get there?:  How do we get there? Picking the most efficient place to start Getting the data you need into your logs Understanding the UNIX syslog paradigm, and how it generalizes to other systems Integrating Windows Event Log data into your UNIX log management system How do we get there? cont.:  How do we get there? cont. Managing audit data in a heterogeneous computing environment Reducing log content to human-readable quantities Interpreting the content of log files  Keeping track of what’s going on in your network! Agenda:  Agenda The Log Problem Generating Interesting Data Centralizing your log data Parsing system logs Attack signatures Common mistakes The Log Problem:  The Log Problem The Log Problem:  The Log Problem “Go look at those logs!” Boatloads of data, most of it superfluous The Log Problem:  The Log Problem On most OSes and apps, security events form less than 1% of total volume of log data “Intelligent” security devices – IDS – help, but don’t eliminate the need for archiving host-based logs Ignoring the problem – or the data – doesn’t make it go away The Log Problem cont. :  The Log Problem cont. Conservative minimum amount of operating system log data, for UNIX/NT servers, on a mid-sized corporate network: Not including Web server access logs, mail logs, IDS data, authentication records, etc. 3.8 GB per day The Log Problem cont.:  The Log Problem cont. Successful attacks are often not logged Log messages vary in quality, and not designed for machine parsing What’s “interesting” is very dependent on your environment What does it take?:  What does it take? Automated processing Nominal status data – usage patterns, capacity planning, etc – off-line, batch processing okay Critical event data – security issues, hardware failures – must be handled real-time or close to real-time What does it take?:  What does it take? The common item to look for when reviewing log files is anything that appears out of the ordinary. CERT Coordination Center Intrusion Detection Checklist Generating Interesting Data:  Generating Interesting Data Just starting out?:  Just starting out? What do you need to know? Start small. Pick one or two apps or types of devices. What kinds of events indicate security problems, performance issues or administrative changes? Are your favorite events recorded by the default logging configuration on your device? Always watch for…:  Always watch for… Hardware failures Resource exhaustion Reboots/restarts Patches or changes to system code or firmware or app software (upgrades or downgrades) Failed logins, esp to admin accounts Panic Attack:  Panic Attack Mar 15 23:22:45 enigma unix: panic[cpu2]/thread=2a1001bdd60: Mar 15 23:22:54 enigma unix: dumping to /dev/dsk/c0t2d0s2, offset 1810694144 Mar 15 23:26:08 enigma savecore: reboot after panic: zero Patching Windows:  Patching Windows UNIX Login Attempts:  UNIX Login Attempts Sep 12 10:17:11 kuspy PAM_pwdb[17529]: authentication failure; (uid=0) -> tbird for ssh service Sep 12 10:17:12 kuspy sshd[17529]: log: Password authentication for tbird accepted. Failed Logon to Win2k Domain:  Failed Logon to Win2k Domain <132>EvntSLog:6388: [AUF] Wed Oct 10 10:57:15 2001: OSMOSIS/Security (675) - "Pre-authentication failed: User Name: Administrator User ID: %{S-1-5-21-776561741-2052111302-1417001333-500} Service Name: krbtgt/LAB Pre-Authentication Type: 0x2 Failure Code: 0x18 Client Address: " <132>EvntSLog:6389: [AUF] Wed Oct 10 10:57:15 2001: OSMOSIS/Security (529) - "Logon Failure: Reason: Unknown user name or bad password User Name: Administrator Domain: LAB Logon Type: 2 Logon Process: User32 Authentication Package: Negotiate Workstation Name: OSMOSIS" UNIX System Boot:  UNIX System Boot Jul 8 01:46:52 evileye unix: SunOS Release 5.7 Version Generic_106541-04 [UNIX(R) System V Release 4.0] Windows System Reboot:  Windows System Reboot Windows System Reboot cont.:  Windows System Reboot cont. Cisco IOS restart:  Cisco IOS restart *Mar 1 00:00:24.716 UTC: %SYS-5-RESTART: System restarted – Cisco Internetwork Operating System Software IOS (tm) C3500XL Software (C3500XL-C3H2S-M), Version 12.0(5.4)WC(1), MAINTENANCE INTERIM SOFTWARE Copyright (c) 1986-2001 by cisco Systems, Inc. Compiled Tue 10-Jul-01 12:32 by devgoyal Always watch for::  Always watch for: Creation of new accounts, esp those that “look like” system accounts, or have admin privileges Signatures of known attacks those that crash servers those that don’t crash servers obviously system specific Where do I start?:  Where do I start? What systems to start with? Most vulnerable servers Web servers (public, intranet, extranet) publicly-visible mail servers administrators’ workstations (operating systems, applications, network equipment) Where do I start? cont.:  Where do I start? cont. Critical network infrastructure systems Domain Name servers (or WINS, or whatever) Windows Domain Controllers, NetWare Directory Services Routers & switches Backup servers, network attached storage Internal mail servers Where do I start? cont.:  Where do I start? cont. Perimeter devices intrusion detection systems (theoretically highest bang-for-buck security info) firewalls (often the first machines to detect probes and scans; access control points) remote access servers (account harvesting, brute force attacks) Where do I start? cont.:  Where do I start? cont. Any systems that store proprietary corporate data database servers file servers code repositories data warehouses Monitoring Routers:  Monitoring Routers User entering enable mode Access control list changes Enable/disable/reconfigure interfaces Firmware downgraded/upgraded/patched Conditions that produce Traceback errors rsh, rcp connection attempts Monitoring Firewalls:  Monitoring Firewalls Host OS messages as applicable Configuration changes Adds/deletes/changes of admin accounts Administrative traffic from “unexpected” locations (like the Internet) Connection logs (start/stop/amt of data) Monitoring Database Servers:  Monitoring Database Servers Interactive DB access rather than scheduled jobs or automated processing Access control changes (DBA granting themselves or other DBAs higher level of access to system) DB account access over network Automated reporting of network component versions Monitoring Database Servers cont.:  Monitoring Database Servers cont. Changes to scripts on DB servers Presence (?) and use of non-interactive DB accounts File system full:  File system full set /kernel: pid 801 (mysqld), uid 88 on /var: file system full Monitoring Web Servers:  Monitoring Web Servers Host OS messages Malicious signatures in access logs (artificial ignorance/content inspection) New virtual hosts added New listening ports or virtual IPs added Unusual increase in inbound or outbound traffic (Nimda, anyone?) Monitoring Web Servers cont.:  Monitoring Web Servers cont. New scripts New modules New content Parent or child processes dying with unexpected errors Web server action resulting from client request (i.e. how did that URL map to file system?) Improving the Quality of Logs:  Improving the Quality of Logs Improving the quality of logs:  Improving the quality of logs Complexity of configuration and invisibility of most attacks (especially the successful ones) make monitoring hard A good alarm improves the chances that you’ll see evil quickly, without overwhelming you with false positives Why IDS isn’t Enough:  Why IDS isn’t Enough Jan 2 16:19:23 yyy.yyy.yyy.yyy snort[1260]: RPC Info Query: -> Jan 2 16:19:31 yyy.yyy.yyy.yyy snort[1260]: spp_portscan: portscan status from 2 connections across 1 hosts: TCP(2), UDP(0) Buffer Overflows:  Buffer Overflows Jan 02 16:19:45 rpc.statd[351]: gethostbyname error for ^X÷ÿ¿^X÷ÿ¿^Y÷ÿ¿^Y÷ÿ¿^Z÷ÿ¿^Z÷ÿ¿^[÷ÿ¿^[÷ÿ¿bffff750 804971090909090687465676274736f6d616e797265206520726f7220726f66 bffff718 bffff719 bffff71a bffff71b! !  Buffer Overflown?:  Buffer Overflown? Jan 02 16:20:25 adduser[12152]: new user: name=cgi, uid=0, gid=0, home=/home/cgi, shell=/bin/bash Jan 02 16:22:02 PAM_pwdb[12154]: password for (cgi/0) changed by ((null)/0) Your Network is Talking:  Your Network is Talking Improving quality of OS logs:  Improving quality of OS logs What conditions or “state changes” indicate malicious activity, component failure, or significant admin activity? Do default logging mechanisms detect and record them? If not, can we make them easier to detect? Improving quality of OS logs cont.:  Improving quality of OS logs cont. What kinds of events do we want to record? User logins and logouts, at least for administrators Changes to administrative accounts password change on root/admin account addition of new user with root or admin privileges Improving quality of OS logs cont.:  Improving quality of OS logs cont. Application starts/restarts/shutdowns configuration changes security context (does it run as root or some other user?) network ports in use system files in use Improving quality of OS logs cont.:  Improving quality of OS logs cont. System boot/reboot/shutdown who did it (if appropriate) hardware/software/admin changes Resource issues Network configuration changes IP address, MAC address Access Control Lists Improving quality of OS logs cont.:  Improving quality of OS logs cont. Invalid data input to application what sort of invalid: data not present, too much data, improper format result: did app crash, spawn root shell, recover gracefully Inappropriate privilege transitions in kernel Remember:  Remember You can’t extract interesting information from your logs if it’s not there in the first place Collecting and Archiving:  Collecting and Archiving Collecting...:  Collecting... So now you have all of these systems spewing out tons of wonderfully useful information... Where do you put it? Collecting and Archiving:  Collecting and Archiving Making sure your logs are going somewhere Making sure they’re being received somewhere Making sure they don’t disappear Building a Central Logging Infrastructure:  Building a Central Logging Infrastructure Create good data Be sure you can detect the events you want to see Collect good data Build a loghost, forward device/app logs Extract wisdom from the good data Real-time monitoring for critical events Batch processing for trends, planning Centralizing Your Logs:  Centralizing Your Logs Why? easier to archive easier to correlate log preservation if host is attacked Homogeneous or mixed? Homogeneous: lucky you Built in mechanisms Mixed Centralizing Your Logs cont.:  Centralizing Your Logs cont. Mixed environment: syslog may not be a good choice security reliability syslog may be the only choice most supported logging mechanism So it’s clearly the best choice! syslog & Its Relatives:  syslog & Its Relatives syslogd:  syslogd Consolidated audit mechanism for UNIX kernel and application messages Gives application and OS developers a consistent interface for reporting significant events Allows local or remote storage of messages syslogd cont.:  syslogd cont. /etc/syslog.conf controls how much data is recorded, and what becomes of it syslog.conf format: selector <Tab> action selectors indicate what’s sending the message, and what criticality the message has syslogd cont.:  syslogd cont. facility – the application or system component that generates a log message user – default facility applied if nothing else is specified when message is written kern – messages generated by system processes local0–local7 – facilities available for customized processing syslogd cont.:  syslogd cont. level – the severity of a message on the computer generating it, i.e. emerg – system is or will be unusable if situation is not resolved (most severe) alert – immediate action required notice –a significant but typically normal event that may merit investigation Assigned by the developer who implemented the logging syslogd cont.:  syslogd cont. action – what’s done with a message once it’s received from a facility actions usually represent destinations – message is written to a local file, a syslog daemon on another system, the system console, or a user console syslogd Historical Oddities:  syslogd Historical Oddities Many syslogds require <Tab> as delimiter, not whitespace, & die gory, unpleasant, hard-to-detect deaths if <Tab>s are not present Fixed in SDSC-syslog, syslog-ng, sysklogd, some OS implementations (FreeBSD) Audit Caveats:  Audit Caveats syslog only records what you’ve told it to record Vast majority of events on a system are not recorded – events must generate logs to show up in log monitoring Failed attacks often leave tracks; successful attacks are often only recorded indirectly Audit Caveats cont.:  Audit Caveats cont. Running automated attack tools (nessus, CyberCop Scanner) against base operating systems – 15% of all probes logged by OS or application mechanisms, but at least record genuine system activity IDS, other network alarms really help to identify when further examination is warranted syslogd Issues:  syslogd Issues No default limitations on data sources (users or processes), so all log data is inherently unreliable Nothing to prevent forged data from being inserted into data stream Limited number of actions possible on receipt of a particular message syslogd Replacements:  syslogd Replacements Improved ability to filter and redirect inbound log messages Integrity checks on locally-stored logfiles Store more information about log data and events Fix that whole <Tab> problem Retain compatibility with classic syslog syslogd Replacements cont.:  syslogd Replacements cont. syslog-ng: most popular replacement; allows forwarding over TCP; remembers forwarding addresses; more granular message filtering modular syslog: a syslog replacement that includes data integrity checks, easy database integration, and output redirection using regular expressions syslog the Protocol:  syslog the Protocol syslog messages are sent to central loghost via syslog protocol (UDP/514) Relay architecture supported, but eliminates data from message originator No validation of message (headers or content) Data sent in cleartext: can be sniffed, can be modified in transit Secure Protocol Initiatives:  Secure Protocol Initiatives syslog-reliable: TCP-based for reliable message delivery Add authentication and encryption to protect audit data syslog-sign: authentication of message sender replay protection message integrity and delivery checks Real-World Secure Transmission:  Real-World Secure Transmission SDSC-syslog implements syslog-sign and syslog-reliable nsyslog – TCP over SSL Tunnelling over SSH or SSL Client: netcat -l -u -p syslog | netcat localhost 9999 loghost: netcat -l -p 9999 | netcat localhost -u syslog Serial cables syslog Output:  syslog Output Message format is invented by developer who’s creating logging capability No standard message formats, but usually something like date time host/IP service message Recording facility & level:  Recording facility & level Most UNIX syslogs don’t include facility & level in messages, so hard to determine appropriate filters without pattern matching If you configure syslog.conf to send all emerg messages to logged in users, how do you know you’ll get what you expect? Recording facility & level cont.:  Recording facility & level cont. Solaris 7 and later enables message tagging, controlled in /kernel/drv/log.conf Recording facility & level cont.:  Recording facility & level cont. Enabling message tagging adds fields to syslog messages [ID msgid facility.priority] msgid = hash of message text Also lists specific kernel module for facility, rather than kern Recording facility & level cont.:  Recording facility & level cont. Without tagging: Oct 1 14:07:24 mars unix: alloc: /: file system full With tagging: Oct 1 14:07:24 mars ufs: [ID 845546 kern.notice] alloc: /: file system full Recording facility & level cont.:  Recording facility & level cont. Using syslog-ng: destination my_file { file("/var/log/messages" template("$DATE $FACILITY.$LEVEL $FULLHOST $MESSAGE\n")); }; Recording facility & level cont.:  Recording facility & level cont. Without tagging: Aug 30 01:20:56 bettiepage/bettiepage postfix/smtpd[22956]: disconnect from 2100[] With tagging: Oct 27 11:41:22 bettiepage/bettiepage postfix/smtpd[18020]: connect from smtp8.Stanford.EDU[] syslog Output cont.:  syslog Output cont. UNIX applications that use syslog: amd date ftpd gated inetd sendmail login rlogin named ntpd passwd sudo tcpd vixie-cron lpd nnrpd logger:  logger UNIX command line utility writes arbitrary messages to syslog hathor:/var/log# logger "this space intentionally left blank" hathor:/var/log# Oct 27 13:05:41 local@hathor tbird65: [ID 702911 user.notice] this space intentionally left blank Windows Event Log:  Windows Event Log Windows analog of syslog No integrated capability for remote logging Binary file – no grep! System default – auditing is disabled Windows Event Log cont.:  Windows Event Log cont. System Log: Startup and shutdown messages, system component data, critical services Security Log: Windows auditing system data only, including user & host auth, share access, printing, other Application Log: Nearly everything else Windows Event Log cont.:  Windows Event Log cont. Any process can write to Application and System Event Logs – “should” register message library Only LSA and Event Log Service itself can write to Security Event Log Security log is more reliable forensic information than off the shelf syslog Windows Application Log:  Windows Application Log Application Log messages parsed via message dictionary Should be provided by application developer Frequently isn’t Windows Application Log cont.:  Windows Application Log cont. Windows Event Log cont.:  Windows Event Log cont. Windows Event Log cont.:  Windows Event Log cont. Windows Event Log cont.:  Windows Event Log cont. Windows Event Log cont.:  Windows Event Log cont. logger equivalent for Windows: Win2000 Resource Kit tool logevent Writes an Event ID set by an administrator to the Application Log Message severity is always Informational Adiscon’s MonitorWare agent will forward data added to a Windows text based log to a syslog server Windows Event Log cont.:  Windows Event Log cont. Another logger equivalent for Windows: Kiwi’s Syslog Message Generator Sends manually-generated syslog messages from a Windows command line or GUI to a syslog server Does not read data from Event Log, but useful for testing WinNT Audit Configuration:  WinNT Audit Configuration NT vs. 2000 Audit Categories:  NT vs. 2000 Audit Categories WinNT Win2k User/Group Mgmt. Audit Account Management Logon and Logoff Audit logon events File and Object Access Audit object access Security Policy Changes Audit policy changes Use of User Rights Audit privilege use Audit process tracking Audit process tracking Restart, Shutdown, Audit system events System + Audit account logon events + Audit directory service access Win2k Event Log Details:  Win2k Event Log Details Local policy settings applied first, then domain policy settings, then active directory settings May make local audit setting different from effective audit setting Win2k Audit Configuration:  Win2k Audit Configuration syslog & Relatives Summary:  syslog & Relatives Summary UNIX: syslogd & syslog protocol uncontrolled, unverified datastream most widely implemented logging structure Windows: Event Log reliable security information proprietary format Both require OS configuration, possibly application configuration & tuning Building a Central Logging Infrastructure:  Building a Central Logging Infrastructure Security Surveillance:  Security Surveillance Perimeter devices detect port scans & vulnerability probes  FW, router Improve network attack detection  NIDS Improve host-based attack detection  HIDS Open Source Surveillance Tools:  Open Source Surveillance Tools Monitoring system calls – systrace, St. Jude FW – ipfilter, TCP Wrappers NIDS – Snort HIDS – logdaemon File system integrity checkers – tripwire, Samhain portsentry Log:  portsentry Log Nov 19 00:12:53 hosty portsentry[17645]: [ID 702911 daemon.notice] attackalert: Connect from host: to TCP port: 80 Centralized Logging:  Centralized Logging Building a loghost Popular architectures Have a good time Getting data to the loghost Configuring clients (OS & application) Transport mechanisms Archiving Loghost Decisions:  Loghost Decisions Which operating system? Most experience = easiest to harden vs. Genetic diversity Assuming syslog, which syslog? Assuming syslog-the-protocol, out of the box, or (crypto, authentication) enhanced security? Loghost:  Loghost Bastion system running limited services: archives and processes audit data SSH or other secure protocol for administrative access For real paranoids: hide syslog configuration file Or use a syslog replacement Loghost cont.:  Loghost cont. Separate file systems for log data vs. kernel vs. applications Monitor disk space utilization carefully Store data on write-once media (like CD-ROM drives) if feasible to protect against tampering Document sys admin processes to manage log data Loghost cont.:  Loghost cont. syslog configuration on loghost: are new log messages dumped into common message file, or into specific files based on facility, or files/destinations based on severity? Might want mail, Web (client & server), FW network connection logs handled separately Popular Architectures:  Popular Architectures Relay architecture – remote systems report to loghosts in branch; branch loghosts forward to central location for processing & archiving Stealth logging for DMZ networks – monitoring Web, email servers Logging over SSH for confidential data collection within private network Relay Architecture:  Relay Architecture Relay Architecture cont.:  Relay Architecture cont. Branch office loghosts: receive data from branch office servers, localhost; forward to central loghost Central loghost: receive data from branch office loghosts; write to archive; process data syslog-ng to preserve source info Branch office loghost:  Branch office loghost source branch1-loghost { unix-dgram(“/var/run/log”); internal (); udp (); }: Branch office loghost cont.:  Branch office loghost cont. destination localhost { file(“/var/log/messages”); }: destination central-loghost { tcp(“” port (514)); }: Branch office loghost cont.:  Branch office loghost cont. options {chain_hostnames(yes) }; Branch office loghost cont.:  Branch office loghost cont. log { source(branch1-loghost); destination(localhost); source(branch1-loghost); destination(central-loghost); }; Central office loghost:  Central office loghost source central-loghost { unix-dgram(“/var/run/log”); internal(); tcp(ip( port(514) max-connections(5)); }; Central office loghost conf.:  Central office loghost conf. destination localhost { file(/var/log/messages-all”); }; log { source(central-loghost); destination(localhost): }; Stealth loghost:  Stealth loghost To collect data in places where you need to minimize chance of network-based DoS, or compromise of log server Configure hosts and applications to log to a non-existent but valid IP address on DMZ Stealth loghost cont.:  Stealth loghost cont. Stealth loghost cont.:  Stealth loghost cont. Configure Web servers with bogus arp entry for phantom logserver: Loghost DMZ interface – no IP address, in promiscuous mode, connected to hub or span port on switch arp –s 00:0a:0a:00:bb:77 Stealth loghost cont.:  Stealth loghost cont. tcpdump puts interface into promiscuous mode unless told otherwise. Assume loghost’s stealth interface is exp0 tcpdump –i exp0 –s 1024 –w dst port 514 Non-UNIX syslog servers:  Non-UNIX syslog servers Macintosh netlogger syslog daemon for Windows by Kiwi Serial connections to a VAX, or a mainframe, or a line printer Protect loghost from local users:  Protect loghost from local users # groupadd loggers # chgrp loggers /dev/log # chmod o-rw,ug+rw /dev/log # ls -l /dev/log srw-rw---- 1 root loggers 0 Feb 20 15:56 /dev/log Network Access to Loghost:  Network Access to Loghost Use encryption to limit access (via SSH tunnel, or one of the secure syslog replacements) Built in firewalling on loghost (ipchains, iptables, etc) Have a good time:  Have a good time Accurate time-keeping simplifies analysis and event correlation UNIX and Windows implementations of Network Time Protocol at List of public timeservers at Have a good time cont.:  Have a good time cont. mark facility produces internal timestamps at intervals selected by the admin useful for verifying that logging is up and running, estimating lags in message delivery or other time synchronization problems Getting Data to the Loghost:  Getting Data to the Loghost What information is useful for security and system administration? Configure client OS to forward data to loghost Configure applications to log to syslog Tune logging levels as necessary Log locally and remotely, for consistency checks and redundancy FireWall-1 to loghost:  FireWall-1 to loghost Need to record operating system events, firewall policy configuration changes, network connection logs for thorough monitoring Assumes UNIX host for FW-1 Operating system events: standard syslog configuration for the host OS FireWall-1 to loghost cont.:  FireWall-1 to loghost cont. Firewall policy changes: command line loads are recorded by syslog loads, changes created via GUI tool are recorded in $FWDIR/log/cpmgmt.aud as root, start /bin/sh and type /bin/tail -f $FWDIR/log/cpmgmt.aud | /bin/logger -p > /dev/null 2>&1 & FireWall-1 to loghost cont.:  FireWall-1 to loghost cont. FireWall-1 to loghost cont.:  FireWall-1 to loghost cont. Firewall network connection logs: connection logs are stored in Checkpoint proprietary binary format as root, start /bin/sh and type $FWDIR/bin/fw log -tf | /bin/logger -p Watch the log size! Firewall logging caveat:  Firewall logging caveat FIrewalls don’t usually record why a packet was allowed or denied May record which particular rule caused a particular connection to fail or succeed Makes later correlation challenging unless you explicitly save policies when they change, or name rules transparently (and can write names to logs) Windows to loghost:  Windows to loghost Third-party tools required to send Event Log data to remote loghost Pure syslog clients: BackLog Windows to loghost cont.:  Windows to loghost cont. Other options: Perl module Win32::EventLog – allows external access to EventLog API Discussion based on inexpensive third-party tool, EventReporter Windows Audit Policy:  Windows Audit Policy Windows to loghost cont.:  Windows to loghost cont. Windows to loghost cont.:  Windows to loghost cont. Windows to loghost cont.:  Windows to loghost cont. Windows to loghost cont.:  Windows to loghost cont. Other application considerations:  Other application considerations logger can be used to capture any UNIX log data that can be expressed in text Kiwi’s logger for Windows apps In-house applications? Most UNIX & Windows programming languages include function calls to system logger Mainframes to loghost:  Mainframes to loghost Tivoli NetView Manager for OS 390 Syslog/iX for Hewlett Packard MPE Please e-mail if you know of others! Log Rotation:  Log Rotation Many circumstances can create huge quantities of logs deliberate DoS attempts Nimda forwarding Apache access logs to syslog Protect loghost’s integrity by rotating and archiving logs regularly Log Rotation cont.:  Log Rotation cont. UNIX variants now include log rotation as part of their default install Rotate based on absolute size, disk utilization, age of data Delete old records, or compress the data and store it elsewhere? Managing the Event Log:  Managing the Event Log Archive and storage options default behavior: overwrite old logs save and clear binary files on regular schedule if appropriate or log remotely and avoid whole issue Batch processing to export, dump, view dumpel from WinNT/2000 Resource Kit will dump logs to comma-delimited files Interpreting Log Data:  Interpreting Log Data Wisdom Extraction A Note on Log Reduction:  A Note on Log Reduction Most tools use regular expressions (of varying complexity) to divide logs into “discard” and “investigate” Size of logs also reduced by fixing problems: minor configuration errors & hardware glitches that don’t disrupt service but can be resolved Log Analysis Tools:  Log Analysis Tools Some syslog collectors (like syslog-ng) offer sophisticated parsing mechanisms for real-line processing as data is received Some analysis tools with the ability to handle real-time data streams can be used as collectors but…. Log Analysis Tools cont.:  Log Analysis Tools cont. In practice want to separate the collecting and analyzing capacities even if you use the same code archive raw data for future processing both tasks can be extremely resource intensive, so separating them provides better scalability Automated Log Parsing:  Automated Log Parsing Automated Log Parsing cont.:  Automated Log Parsing cont. Regular expressions to sort data into “nominal status” events known significant events All remaining (new?) messages sent to administrators for research and triage Marcus Ranum’s “artificial ignorance” approach Start Your Ignore List:  Start Your Ignore List cut -f5- -d\ all.log.0 | sort | uniq -c |sort -nr > uniq.sorted.freq Potential Problems:  Potential Problems Once system is tuned, catch new attacks and problems as frequently as human is able to review unmatched data Attacks and problems are typically rare events Can we seed the system with signatures of “known bad” events? Creating attack signatures:  Creating attack signatures Lance Spitzner and others: run the attacks you care about off-line, and use the log data you generate to write swatch or logsurfer filters Same deal with administrative events and system changes Creating attack sigs cont.:  Creating attack sigs cont. Outside sources of attack data: This class The Log Analysis Web site mailing list mailing list Other data sources:  Other data sources Developers – likeliest to work with in-house applications or custom stuff Documentation frequently hard to evaluate (is this log message still in the code execution path?) frequently non-existent start with emerg/warning/alert level messages swatch:  swatch Most frequently-deployed tool for real-time log monitoring Perl-based, single line processing, message consolidation if timestamps available Output colors based on content of what’s being matched swatch cont.:  swatch cont. Configuration file requires patterns for swatch to identify, and actions to be taken when the patterns appear System impact can be significant, especially if swatch starts external programs (like sendmail) when alerts are generated swatch cont.:  swatch cont. For instance, to identify failed logins: watchfor /login.*: .* LOGIN FAILURES ON/ mail=swatcher Write catch-all filter at end of file to identify new messages (have to tune file as per checksyslog first): watchfor /.*/ mail=swatcher,mail=swatch-coder LogSentry:  LogSentry pattern matching to look for significant events based on script from Gauntlet firewall reports on pre-configured security messages, unusual messages via email summary batch processing, not real time Third Party Applications cont.:  Third Party Applications cont. LogSentry keywords determined from: syslog, tcpwrapper and TIS Firewall Toolkit logs Beta testers and end users “guessing” Third Party Applications cont.:  Third Party Applications cont. FreeBSD LogSentry keywords (hacking): EXPN root LOGIN root REFUSED rlogind.*: Connection from .* on illegal port rshd.*: Connection from .* on illegal port sendmail.*: user .* attempted to run daemon logsurfer:  logsurfer Multi-line log event processor Maintains context messages & situations Includes timeouts and resource limits Can change monitoring behavior if situation requires logsurfer cont.:  logsurfer cont. Configuration issues: Severe system impact possible if external programs (like sendmail) are called in response to events Regular expressions must be “good enough” too general matches irrelevant messages too specific misses messages that should be matched logsurfer cont.:  logsurfer cont. # rpcbind #-------------------------------------- ' rpcbind: refused connect from ([^ ]*)' ' connect from [^]*|localhost)' - - 0 CONTINUE open "^.{19,}$2" - 4000 86400 0 ignore ' ([^ .]*)(|) rpcbind: refused connect from ([^ ]*) ' - - - 0 CONTINUE rule before " rpcbind: refused connect from $4" - - - 300 ignore ' ([^ .]*)(|) rpcbind: refused connect from ([^ ]*) ' - - - 0 exec "/usr/local/sbin/safe_finger @4 | /usr/local/sbin/start-mail logsurfer \"$2: rpcbind: (backtrack)\"" Other single line parsing tools:  Other single line parsing tools autobuse colorlogs roottail log_analysis logmuncher logscanner LogWatch and the list goes on.... Baselining:  Baselining What’s normal? How many apps/facilities/systems report to loghost? How many distinct messages from each facility? Top ten most frequent and “top” ten least frequent are a good place to start Baselining cont.:  Baselining cont. Amount of network traffic per protocol: total HTTP, email, FTP etc. Logins/logoffs, access of admin accounts DHCP address management, DNS requests total amount of log data per hour/day number of processes running at any time Thresholding:  Thresholding Once you’ve baselined, what’s weird? Conditions: given a line of data, notify based on the presence of a second line the absence of a second line number of times that event happens in a given time period Or notify when a message doesn’t appear! What’s Interesting? cont.:  What’s Interesting? cont. It depends: Whatever else is pertinent and threatening in your own environment: custom applications, unusual hardware, whatever hit Bugtraq last week… What’s Interesting? cont.:  What’s Interesting? cont. Ugh: in order to identify suspicious behavior, you have to know what expected behavior is Can’t define “weird” unless you know “normal” What’s normal in California may be highly unusual in Kansas Finding the Good Stuff:  Finding the Good Stuff grep, sort, uniq to eliminate nominal status messages, filter things down to the interesting and unknown Or Excel, if you don’t have too much data to process Surprising amounts of info available on obscure messages by searching the Internet Finding the Good Stuff cont.:  Finding the Good Stuff cont. What obscure messages and services? SunMC-SLM eri asclock_applet farmd gnomepager_applet pcipsy multiload_applet uxwdog magicdev netcon_server Finding the Good Stuff cont.:  Finding the Good Stuff cont. Oct 26 03:10:38 [-1]: pyseekd[10906]: Info: [master] deleting directory /cust/SEEKUltra-3.18/master /master/db/118750 Finding the Good Stuff cont.:  Finding the Good Stuff cont. Finding the Good Stuff cont.:  Finding the Good Stuff cont. What to look for:  What to look for Passwords changed by someone other than the user – especially UID 0 users with null logins Processes dying with error code 1 Long messages full of random characters Unexpected configuration changes What to look for cont.:  What to look for cont. The least-frequent messages generated on your network Messages containing the words fatal, panic or password/passwd Sudden increase or decrease in the number of messages received from a host or application What to look for cont.:  What to look for cont. Failed logon from non-local or unknown domain Other Logs to Check:  Other Logs to Check Older rootkits do not backdoor tcp-wrappers, so check for unexpected logins FTPd records logins and is not typically replaced Shell history files in the directory where the compromised server died Other Logs to Check cont.:  Other Logs to Check cont. Web server access logs Proxy server logs Information contained in core dumps from compromised servers Attack signatures:  Attack signatures Log data from miscellaneous attacks & exploits & Intersting things to look for Creating New User – WinNT:  Creating New User – WinNT <133>EvntSLog:423: [AUS] Fri Oct 05 11:59:09 2001: HANDCUFFS/Security (624) - "User Account Created: New Account Name: tbird New Domain: HANDCUFFS New Account ID: S-1-5-21-1647595427-22557637-1039276024-1001 Caller User Name: Administrator Caller Domain: HANDCUFFS Caller Logon ID: (0x0,0x2B79) Privileges - " Creating New User – WinNT cont.:  Creating New User – WinNT cont. <133>EvntSLog:424: [AUS] Fri Oct 05 11:59:09 2001: HANDCUFFS/Security (626) - "User Account Enabled: Target Account Name: tbird Target Domain: HANDCUFFS Target Account ID: S-1-5-21-1647595427-22557637-1039276024-1001 Caller User Name: Administrator Caller Domain: HANDCUFFS Caller Logon ID: (0x0,0x2B79) " <133>EvntSLog:425: [AUS] Fri Oct 05 11:59:09 2001: HANDCUFFS/Security (628) - "User Account password set: Target Account Name: tbird Target Domain: HANDCUFFS Target Account ID: S-1-5-21-1647595427-22557637-1039276024-1001 Caller User Name: Administrator Caller Domain: HANDCUFFS Caller Logon ID: (0x0,0x2B79) " SSH CRC-32 Attack:  SSH CRC-32 Attack sshd[6169]: fatal: Local: Corrupted check bytes on input. sshd[6253]: fatal: Local: crc32 compensation attack: network attack detected sendmail Exploits:  sendmail Exploits Jul 21 01:25:49 ariel sendmail[308]: BAA00307:, ctladdr=":/bin/mail </etc/passwd", delay=00:00:34, mailer=smtp, (, stat=Sent (Ok.) Jul 21 01:35:40 ariel sendmail[545]: setsender: "/bin/mail </etc/passwd": invalid or unparseable, received from [] Jul 21 13:13:04 ariel sendmail[784]: NAA00783: to=\tsutomu,, delay=00:03:09, mailer=local, stat=Sent Nimda: Worm Sign:  Nimda: Worm Sign - - [18/Sep/2001:09:35:19 -0500] "GET /scripts/..%%35%63../winnt/system32 /cmd.exe?/c+dir HTTP/1.0" 400 215 "-" "-" - - [18/Sep/2001:09:35:22 -0500] "GET /scripts/..%%35c../winnt/system32 /cmd.exe?/c+dir HTTP/1.0" 400 215 "-" "-" - - [18/Sep/2001:09:35:23 -0500] "GET /scripts/..%25%35%63../winnt/system3/ cmd.exe?/c+dir HTTP/1.0" 404 - "-" "-“ - - [18/Sep/2001:09:35:23 -0500] "GET /scripts/..%25%35%63../winnt/system32 /cmd.exe?/c+dir HTTP/1.0" 404 - "-" "-" Config Change on Cisco IOS:  Config Change on Cisco IOS %SYS-5-CONFIG: Configured from host1-config by rcp from ACL changes:  ACL changes Sep 12 12:13:39 2000 f_cf a_acladm t_acl_change p_major pid: 21734 ruid: 0 euid: 0 pgid: 21734 fid: 1021694 cmd: 'cf‘ domain: Admn edomain: Admn acl_admin: tbird acl_op: modify acl_table: acl acl_data: {'ignore': 0, 'name': 'ssh_ext_soc'} FW-1: MAD Port Scan Alert:  FW-1: MAD Port Scan Alert Feb 26 17:47:28 root: 26Feb2001 17:47:28 accept localhost >daemon useralert product MAD proto ip src elendil dst moose additionals: attack=blocked_connection_port_scanning iptables: Detecting Specific Attacks:  iptables: Detecting Specific Attacks Create a specific rule to block a known attack, like iptables -A FORWARD -i eth0 -p tcp --tcp-flags ALL SYN -d 0/0 –dport 17300 -j LOG --log-prefix " KUANG2_SCAN " iptables -A FORWARD -i eth0 -p tcp --tcp-flags ALL SYN -d 0/0 –dport 17300 -j REJECT --reject-with icmp-host-unreachable iptables: Detecting Specific Attacks:  iptables: Detecting Specific Attacks Get easy to read logs like Oct 19 07:58:36 gw1 kernel: KUANG2_SCAN IN=eth0 OUT=eth1 SRC= DST= LEN=48 TOS=0x00 PREC=0x00 TTL=115 ID=37626 DF PROTO=TCP SPT=1899 DPT=17300 WINDOW=8192 RES=0x00 SYN URGP=0 Slapper: Linux/SSL worm:  Slapper: Linux/SSL worm Apache/mod-SSL worm discovered 13 Sept 2002; exploits buffer overflow in SSL v2 [error] SSL handshake failed: HTTP spoken on HTTPS port; trying to send HTML error page [error] OpenSSL: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO: http request [Hint: speaking HTTP to HTTPS port!?] Windows Attack Signatures:  Windows Attack Signatures 289010633 2003-01-09 00:18:41.423 AUTH/SEC WARNING 138161:Thu Jan 09 00:18:40 2003: XXXXX/Security (529) - "Logon Failure: Reason: Unknown user name or bad password User Name: administrator Domain: PAFU-EYWAKTYSNO Logon Type: 3 Logon Process: NtLmSsp Authentication Package: NTLM Workstation Name: PAFU-EYWAKTYSNO" CERT Advisory CA-2002-03:  CERT Advisory CA-2002-03 Numerous vulnerabilities in SNMP implementations Denial of service in all vulnerable systems Buffer overflow/root compromise in some vulnerable systems PROTOS test suite publicly available Detecting Use of PROTOS:  Detecting Use of PROTOS Preliminary results for Solaris snmpdx: One of the test packets DoSes daemon Next test case generates syslog msg: Feb 12 23:25:48 mordor snmpdx: agent snmpd not responding PROTOS vs. Solaris SNMP:  PROTOS vs. Solaris SNMP Feb 15 02:06:45 mordor snmpdx: error while receiving a pdu from The message has a wrong version (8355711) Feb 15 02:08:58 mordor snmpdx: SNMP error (UNKNOWN! (65793), 0) sent back to PROTOS signatures: Snort:  PROTOS signatures: Snort alert udp $EXTERNAL_NET any -> $HOME_NET 161 (msg:"Attack using PROTOS Test-Suite-req-app"; content: "|30 26 02 01 00 04 06 70 75 62 6C 69 63 A0 19 02 01 00 02 01 00 02 01 00 30 0E 30 0C 06 08 2B 06 01 02 01 01 05 00 05 00|";) Buffer Overflows - Again:  Buffer Overflows - Again Jun 18 16:54:45 beagle yppasswdd[155]: yppasswdd: user @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ÿ¾û¸ÿ¾üL@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ¿ÿÿ ¿ÿÿÿÿÿàP" À®`î"?ð®àÀ-ÿÿî"?ô®àÀ-ÿþî"?øÀ-ÿÿÀ"?ü ;Ð ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ/bin/shÿ-cÿ echo 'rje stream tcp nowait root /bin/sh sh -i'>z;/usr/sbin/inetd -s z;rm z;: does not exist cachefsd Buffer Overflow:  cachefsd Buffer Overflow May 16 22:46:08 victim-host inetd[600]: /usr/lib/fs/cachefs/cachefsd: Segmentation Fault - core dumped May 16 22:46:22 victim-host inetd[600]: /usr/lib/fs/cachefs/cachefsd: Bus Error - core dumped May 16 22:47:09 victim-host inetd[600]: /usr/lib/fs/cachefs/cachefsd: Hangup Cisco Interface Changing Status:  Cisco Interface Changing Status Feb 26 00:29:50: %LINEPROTO-5-UPDOWN: Line protocol on Interface ATM0/0/1, changed state to down Feb 26 00:29:55: %LINEPROTO-5-UPDOWN: Line protocol on Interface ATM0/0/1, changed state to up Web Server Attack Signatures:  Web Server Attack Signatures Highly visible, frequently attacked Error logs reveal app failures Access logs reveal attempts to retrieve files, traverse directories, may record SQL injection attacks Apache httpd notes:  Apache httpd notes Watch for Slapper, chunked encoding attacks Incorrectly configured cgi-bin: http://host/cgi-bin/lame.cgi?file=../../../../etc/motd Attacks on IIS:  Attacks on IIS Remote access to Windows Shell: http://host/scripts/something.asp=../../WINNT/system32/cmd.exe?dir+e:\ Download the Windows password DB: [drive-letter]:\winnt\repair\sam._ Attacks on IIS cont.:  Attacks on IIS cont. Inappropriate access to server info: http://host/index.asp?something=..\..\..\..\WINNT\system32\cmd.exe?/c+DIR+e:\WINNT\*.txt SQL injection attack on MS-SQL: http://host/cgi-bin/lame.asp?name=john`;EXEC master.dbo.xp_cmdshell'cmd.exe dir c:'-- Code Red:  Code Red - - [28/Sep/2001:00:43:43 -0400] "GET /default.ida?XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u9090%u8190%u00c3%u0003%u8b00%u531b%u53ff%u0078%u0000%u00=a HTTP/1.0" 404 284 "-" "-" Most common logging mistakes:  Most common logging mistakes Most common logging mistakes:  Most common logging mistakes Not ever looking at the logs Not thinking about what sorts of system events you’d like to monitor, before you need to find them Monitoring firewall and/or IDS logs but not database or Web server logs (that is, watching perimeter security systems but not the ultimate targets of attacks) Most common logging mistakes cont.:  Most common logging mistakes cont. Deciding on logging software, analysis software, rotation schedule, archiving system before you’ve determined what sorts of data to collect, how long to store it, how much you’ll get Getting into any conversation on data transfer protocols or XML formatting Synopsis:  Synopsis Need to watch across the network – intrusion detection systems, integrity checks, core system logfiles – and across multiple operating systems, applications To identify attacks in progress, you have to understand the normal day-to-day operation of your network Synopsis:  Synopsis The ratio of interesting to non-interesting audit events is very low Legal situation: not as bad as tech issues suggest, especially if you rely on logs for (non-info-sec) business processes Appendix 1: US Legal Considerations:  Appendix 1: US Legal Considerations Legal Considerations:  Legal Considerations Current “one size fits all” approach Division of computer data into broad categories based on presence of human-generated content Evidentiary requirements “One size fits all”:  “One size fits all” Current case law mostly agrees that computer records maintained as part of regular business procedures are admissible as evidence Computer records classified as hearsay, and then allowed according to business record hearsay exemption – but this isn’t really accurate Computer Data Categories:  Computer Data Categories Data generated by humans that happens to be stored on a computer e-mail messages, documents, static Web page content Computer-generated records network connection logs, ATM receipts Mixed data spread sheets, dynamic Web content Computer Data Categories cont.:  Computer Data Categories cont. Human generated, computer stored – must prove that human statements are trustworthy, and that the records can be reliably associated with a particular individual Computer generated & stored – must assert that program that generated records is behaving properly Computer Data Categories cont.:  Computer Data Categories cont. Both human and computer generated data – must satisfy both sets of requirements Challenges to Computer Records:  Challenges to Computer Records Ease of tampering often used as a reason for discarding computer records Case law supports notion that the mere possibility of tampering does not affect record’s authenticity Opponent must offer substantial proof that tampering occurred Challenges to Computer Records cont.:  Challenges to Computer Records cont. Unreliability of computer programs used as a reason for discarding computer records If the users of an application rely on it as part of their business processes, courts will usually consider its records to be reliable “enough” Challenges to Computer Records cont.:  Challenges to Computer Records cont. Inability to identify author of a computer record used as a reason for discarding data Circumstantial evidence generally used to support claims of authorship Addressing the Challenges:  Addressing the Challenges Ease of tampering Compare local & remote versions of logs before writing to immutable storage media Encrypted & authenticated communications if your network infrastructure allows Addressing the Challenges cont.:  Addressing the Challenges cont. Unreliable applications Document your archiving & review procedures Build log-dependent internal processes (call accounting, policy compliance, capacity planning, incident response) into day-to-day operations Addressing the Challenges cont.:  Addressing the Challenges cont. Unverified identity Strong user, machine authentication Information classification scheme (yeah right) Simplify network topology for user tracking if possible f’r instance, if using DHCP, configure to lock IP address to MAC address on laptops

Add a comment

Related presentations

Related pages

OS Auditng -

– • Centralized Log Host –
Read more

Audit-Evidence-and-Documentation--UDel--PPT | Powerpoint ...

Source : Sponsored Links. About; Contact Us; Family safe; Privacy Policy; Terms Of ...
Read more

Log Infrastructure -

Log Infrastructure • Time Synchronization of log servers ... – • Centralized ...
Read more

Building-a-syslog-Infrastructure--PPT | Powerpoint ...

Source : Sponsored Links. About; Contact Us; Family safe; Privacy Policy; Terms Of ...
Read more

matrix analysis.html PPT Powerpoint Templates ...

Free PPT Templates, Presentations, Lecture Notes, Files for Download, View and Edit - matrix analysis.html, Free PowerPoint templates for download and edit
Read more

Samhain Labs | samhain | ...

Resources for open-source developers and a directory of in-development open-source software.. Get reviews for not samhain. Is ...
Read more