SANS @Night There's Gold in Them Thar Package Management Databases

50 %
50 %
Information about SANS @Night There's Gold in Them Thar Package Management Databases

Published on March 8, 2014

Author: philhagen


 @PhilHagen Image:

WHY ARE WE HERE? Goals: Describe the value of package management databases during the course of a Linux system forensic examination Provide information with immediate benefit! 2

WHO IS THIS GUY? Forensic/infosec consultant Former DoD/IC contractor, USAF Comm Officer (USAFA CompSci) Course lead, FOR572: Advanced Network Forensics & Analysis Linux guy since Slackware needed a stack of floppies and an unsupported SCSI controller meant bootstrapping a kernel compilation 3

OPEN SOURCE: INFINITE POWER (ITTY BITTY LIVING SPACE) Downloading and compiling source code is an amazingly powerful aspect of most open-source software Not viable for the large scale of a production environment Does not address dependencies No common install/uninstall process Fun for the lab or a hobbyist environment, but a headache in an operational environment 4

PACKAGE MANAGEMENT SOFTWARE TO THE RESCUE! Dependencies, file manifests, install/uninstall/upgrade scripts Generally makes software management less of a headache! Many solutions in widespread use among various distributions: opkg (fork of ipkg): Embedded devices like QNAP NAS, etc. dpkg: Debian and Ubuntu tgz: Slackware RPM: RedHat, CentOS, Fedora,many more
 (part of Linux Standard Base) 5

…STILL NOT END-ALL/BE-ALL Most incorporate higher-layer software to handle automatic inclusion of dependencies and other “meta” functions and avoid the much-feared “dependency hell” or “RPM hell” apt-get / aptitude YUM RedHat Network Not focusing on this higher-layer software Some useful artifacts available - bad guys 
 known to “yum install nmap” 6

SCOPE FOR THIS PRESENTATION Just looking at RPM Most concepts apply to other package management standards Notably dpkg All examples created and tested on CentOS 6.5 Ideally: Same OS as subject (RPM library versions, etc) Technically: Same version of RPM, BDB, related libraries May have some success with unmatched versions, but beware!! Presentation notes (including all commands) published soon 7

SOURCES FOR RPM EVIDENCE Populates Berkeley database Lives in /var/lib/rpm/ Includes metadata for every RPM-controlled file User/Group ownership Mode (aka permissions) MD5 (er… SHA256) checksum File size Major/minor number (For entries in /dev/) Symbolic link string (aka “target”) for symlinks Modification time /var/log/yum.log May have useful timestamped history of install/remove/ upgrade actions /var/log/rpmpkgs* 8

PRACTICAL USAGE 1. Mount all partitions from subject filesystem under /mnt/subject/ $ mount | grep subject
 /dev/mapper/vg_centos6vm-lv_root on /mnt/ subject type ext4 (ro,noload)
 /dev/mapper/loop0p1 on /mnt/subject/boot type ext4 (ro,noload) 2. Run RPM commands with “--root /mnt/subject” option Don’t trust the rpm(1) binary from a suspect system! Note: This performs a chroot(2) for all operations - needs root user privileges for validation (not query) actions 9

 FILE ORIGIN $ rpm -qf <filename> Identifies what package owns the specified file Useful to answer “where did this file come from?” or to identify a file as package-less $ rpm --root /mnt/subject -qf /usr/sbin/sshd
 openssh-server-5.3p1-94.el6.x86_64 $ rpm --root /mnt/subject -qf /etc/mail.rc
 Remember chroot! mailx-12.4-7.el6.x86_64 $ rpm --root /mnt/subject -qf /etc/crypttab
 file /etc/crypttab is not owned by any package 10

 PACKAGE VALIDATION (1) $ rpm -V <packagename> Verifies contents of specified package Compares expected (database) to actual (filesystem) Displays files that failed >=1 check, noting which checks failed SM5DLUGT (Size, mode, MD5 (or SHA256!), major/minor nos, link target, user, group, mtime) Shows “?” in output if user running command lacks permission to check (e.g. read access to generate checksums) 11

 PACKAGE VALIDATION (2) WARNING!!! Packages can include “verification scripts” which will execute when the “-V” option is used! You’re not planning to run arbitrary, unknown code on your forensic workstation/VM, are you? The chroot action needs root - these will execute as a child to the sudo process! Use the “--noscripts" option with “-V” to prevent this 12

USE CASE: PACKAGE VALIDATION (3) Some files are expected to change after installation: config files! Still show changes after installation, but denoted with a “c” character Missing files also noted in output as such $ sudo rpm --root /mnt/subject 
 -V openssh-server --noscripts
 <no output> $ sudo rpm --root /mnt/subject 
 -V sudo --noscripts
 S.5....T. c /etc/sudoers 13

USE CASE: PACKAGE VALIDATION (4) $ sudo rpm S.5....T. .M....... S.5....T. ....L.... ....L.... ....L.... ....L.... ..5....T. --root /mnt/subject -Va --noscripts
 c /etc/sudoers
 c /etc/maven/maven2-depmap.xml
 c /etc/pam.d/fingerprint-auth
 c /etc/pam.d/password-auth
 c /etc/pam.d/smartcard-auth
 c /etc/pam.d/system-auth
 c /usr/lib64/security/ $ rpm -root /mnt/subject —V postfix --noscripts
 missing c /etc/postfix/ 14

COOL FEATURE ALERT! Many packages are GPG-signed Independently verify package without using compromised or untrusted system However: RPM database contents not signed: Trojaned RPM package reports no anomalies! How can we use this great feature to our advantage? 15

 AGAINST A PACKAGE FILE (1) The “-p” option runs validation checks between filesystem contents and RPM package file contents Signed package files can be GPG-verified! Avoids an untrusted RPM database entirely Prevents false negative validation from compromised RPM installations Relatively simple process: 1. Download trusted binary RPM file 2. Validate RPM file using GPG 3. Validate filesystem contents against package contents 16

 AGAINST A PACKAGE FILE (2) Consider a system with a compromised RPM database, or a trojaned installation of the Apache web server software $ rpm --root /mnt/subject -V httpd --noscripts
 S.5....T c /etc/httpd/conf/httpd.conf $ wget " x86_64/CentOS/httpd-2.2.3-82.el5.centos.x86_64.rpm"
 $ rpm -K httpd-2.2.3-82.el5.centos.x86_64.rpm
 httpd-2.2.3-82.el5.centos.x86_64.rpm: (sha1) dsa sha1 md5 gpg OK
 $ rpm --root /mnt/subject -Vp httpd-2.2.3-82.el5.centos.x86_64.rpm --noscripts
 S.5....T c /etc/httpd/conf/httpd.conf
 S.5....T /usr/sbin/httpd 17

REAL-WORLD USE CASES (1) Find all non-config files owned by an RPM that fails a verification check $ sudo rpm --root /mnt/subject -Va --noscripts | 
 grep -v " c "
 S.5....T /var/www/awstats/lang/awstats-tt-tr.txt
 S.5....T /var/www/awstats/lang/awstats-tt-tw.txt
 S.5....T /var/www/awstats/lang/awstats-tt-ua.txt
 .......T /var/www/awstats/lib/blacklist.txt
 S.5....T /var/www/awstats/lib/
 S.5....T /var/www/awstats/lib/
 ... 18

REAL-WORLD USE CASES (2) File only config files owned by an RPM, which fail checksum verification $ sudo rpm --root /mnt/subject -Va --noscripts | 
 grep "^..5..... c "
 S.5....T c /etc/pam.d/sshd
 S.5....T c /etc/ssh/sshd_config
 S.5....T c /etc/openldap/slapd.conf
 S.5....T c /etc/sysconfig/ldap
 S.5..... c /etc/sysconfig/saslauthd
 S.5..... c /etc/security/limits.conf
 S.5....T c /etc/logrotate.conf
 SM5....T c /etc/snmp/snmpd.conf
 S.5....T c /etc/sysconfig/snmpd.options
 ... 19

REAL-WORLD USE CASES (3A) Find all files not owned by an RPM (This is going to be slow!) $ cat
 for file in $( sudo find /mnt/subject/etc -type f ); do
 file=$( echo "$file" | sed -e 's//mnt/subject//' )
 rpm --root /mnt/subject -qf "$file" 2>&1 | 
 grep 'package$|directory$' | 
 sed -E 's/^(error: )?file (.*)(: No such file or directory| is not owned by any package)/2/'
 done $ ./
 ... 20

REAL-WORLD USE CASES (3B) Find all files not owned by an RPM (This is going to be slow!) $ ./
 ... 21

COOL PARTY TRICK (1) ! Use the “--queryformat” option to output only relevant/useful fields from the RPM database Provides >150 different tags that can be output for package or for each file in a package Available tags vary by version - online documentation is terrible Use “--querytags” for listing specific to your version of RPM ! Consider “RPMDBtoTimeline”… 22

COOL PARTY TRICK (2)* $ for pkg in $( rpm --root /mnt/subject -qa ) ; do
 rpm --root /mnt/subject -q $pkg --queryformat 
 %{FILEMTIMES}|0|0n]" | sed -e 's/^|/0|/' 
 -e 's/|0|d/|0|d/d/' -e ’s/|0|-/|0|r/r/' 
 4398551f...a13988cf|/usr/share/doc/gamin-0.1.10/ callbacks.gif|0|r/rrw-r—r—|root|root|4514|0|1183556209|0|0
 0|/usr/lib64/|0|lrwxrwxrwx|root|root|15|0| 1282146079|0|0
 0|/usr/share/cracklib|0|d/drwxr-xr-x|root|root|4096|0| 1308983949|0|0
 ... * “Cool” claim not valid at all parties. YMMV. 23

IN CONCLUSION RPM is a pretty cool way to eliminate known files from a Linux system examination Know the shortcomings in the RPM package database so you can mitigate them With a little shell scripting,
 you can develop useful tools to
 quickly and consistently
 minimize input data

Add a comment


HenryCJ | 12/05/16
一般考試,高考,普考,特考 盡在
EnriqueTica | 13/05/16
一般考試,高考,普考,特考 盡在
Victorkn | 16/05/16
一般考試,高考,普考,特考 盡在
MichaelHict | 20/05/16
一般考試,高考,普考,特考 盡在
MichaelHict | 14/06/16
一般考試,高考,普考,特考 盡在
Gregorylor | 29/06/16
RichardOt | 29/06/16
FrancisfuT | 06/07/16
Donaldsi | 22/07/16
一般考試,高考,普考,特考 盡在
RichardOt | 23/08/16
Donaldsi | 07/09/16
一般考試,高考,普考,特考 盡在
MichaelJip | 08/09/16
一般考試,高考,普考,特考 盡在
HenryCJ | 12/09/16
一般考試,高考,普考,特考 盡在
FrancisfuT | 29/09/16

Related presentations

Related pages

There's Gold in them thar package management database with ...

... stored in package management databases for ... Gold in them thar package management database ... There's Gold In Them ...
Read more

There's Gold In Them Thar Package Management Database With ...

There's Gold In Them Thar Package Management Database With Phil Hagen; There's Gold In Them Thar Package Management Database With Phil ... SANS Digital ...
Read more

SANS 2014 (Orlando, FL): Bonus Sessions

SANS 2014 Orlando, FL | Sat, ... SANS 2014. There's *GOLD* in Them Thar Package Management Databases! Phil ... SANS@Night: There's *GOLD* in Them Thar ...
Read more

DFIRCON 2014 (Monterey, CA): Bonus Sessions - SANS Institute

... CA from SANS Institute. ... DFIRCON 2014 Monterey ... DFIRCON 2014. There's *GOLD* in them thar package management databases! Phil Hagen; Thursday ...
Read more

Phil Hagen's Scratch Pad | database

Phil Hagen's Scratch Pad. ... There's GOLD in Them Thar Package Management Databases I still ... Slides from SANS @Night: There’s GOLD in Them Thar ...
Read more

BUDF Club Meeting 3/10/14 - The Onion Router Demo

This is a club meeting for the Bloomsburg ... There's Gold in them thar package management database with Phil ... SANS Digital Forensics 1,251 ...
Read more

Phil Hagen's Scratch Pad | Writings, Postings, Droppings...

By Phil Hagen On December 10, 2014 ... Slides from SANS @Night: There’s GOLD in Them Thar Package ... There's GOLD in Them Thar Package Management ...
Read more

Theres Gold In Those Energy Bills - Education - documents

Share Theres Gold In Those Energy Bills. ... SANS @Night There's Gold in Them Thar Package Management Databases. ... Theres Gold in Them There Hills A GEMS ...
Read more

Package Management | LinkedIn

Package Management. Articles, experts, jobs, and more: get all the professional insights you need on LinkedIn. Learn this skill Sign up Get more ...
Read more