Article Search...

Linux Forensics & Incident Response Introduction

The main purpose of this document will be to combine both incident response and Linux forensics into one single article. Please keep in mind that the information presented in this segment will require intractions with the suspect machine. While we would prefer that analysts utilize the script here: if you feel any other tools would work best for you. Please use those.

Before we begin there are a few things that we need to point out when dealing with a system that may have sustained an incident or is under suspicion of being affected and, needs a forensic analysis. There are a set of questions that you as the investigator must need to ask in order to press onward whether the system needs the analysis or not. To make things easier, we have compiled the following list of questions for you:

  1. What makes you believe the system has been affected?
    1. Did a virus warning go off?
    2. Is the mouse or keyboard acting in erradic ways? (typing, moving, clicking, etc). 
    3. Are new files being created / deleted?
  2. What does the system do?
    1. Is it a business critical system?
    2. Is the device a server or end-point?
    3. Does the device contain sensitive business information?
    4. What department does it belong to?
  3. When did you notice the problem(s)?
    1. Did the problem present itself today or has it been on-going?
    2. Was there an update to the system that has potentially triggered the event(s)?
    3. Did you or anyone else install any new applications, scripts or software?
  4. Who had access to the machine?
    1. Is the machine shared?
    2. Is this machine yours?
    3. When was the last time you logged in?
    4. For how long were you logged in for?
    5. Was the system left unattended whlie authenticated?
  5. Are there any other systems acting the same?
    1. Have you notice other systems in the office doing the same?
    2. Are other users expressing interest over similar issues?
    3. Was there a remote support request or technician who worked on the system last? If so, who?

If you determined that the user who is on the system has not done anything to impact the system but the information you are obtaining warrants an investigation, there are a few things you need to do in order to gather information about the particular event.

While the listing below is going to serve as a cheat sheet for what you will need to obtain, the remainder of the article will go into additional details regarding the commands you will need to utilize in order to obtain the informatin we discussed here.

  • Physical Access to The Machine

    If you have physical access to the machine, you should have a notebook as well as a camera with you when inspecting. Record the date/time of the phone call as well as the date/time in which you have arrived on scene. Take photos of the back of the computer, computer screen and the front ports of each device that is plugged in.

  • Recover Relevant System Information

    At this point, you will need to begin interacting with the suspect machine. When interacting with the suspect machine you will need to write down each command that is being run along with the date/time that it is being processed. In this segment you will need to dump memory, obtain file system information, network information and other details about the system which may aid in the discovery of an event.

  • Running Applications

    From Windows desktop to Linux server we have many years of experience configuring end-point solutions (firewalls, file system detection, malware analysis, threat hunting, etc.) and, server-side configurations. Network Defense Solutions can assist with policy building, alerting and detection measures. We also specialize in securing VPS and shared hosting environment. These service scan range from web site protections, backup, threat detection and more!

  • Logon Sessions / Authentication

    Attempt to obtain the information on who has logged into the system and, on which dates. This will also be useful for tracking who had access and attempt to put together a timeline of when the user noticed the issue he or she called in about.

  • User Accounts

    Falling back on documentation as well as your policies; there should be a trail of which user names are on the systems you provide to your employees. John, Anthony, Joe? Are you seeing EvilEric? This can be a red flag! In this portion you will collect information about the users in /etc/shadow, /etc/passwd, and the homes folder (depending on your *NIX system) and, if they are still logged in!

  • Network Information & Sniffing

    Network information is a very useful set of data to procure. 1) It will provide you as to where the system has been, 2) connections to / from that are foreign and which ports and PID's they are listening on and you can also obtain reputation. Going a step further, you can also put an in-line network tap, use tshark, wireshark or other tools to see where the system is communicating with. If you are using tools such as Bro or Splunk you can start to weed through where on the internet this system has been. Additionally network card information (promiscuous mode, new ap, etc) is another vaulable detail.

  • Kernel Modules

    Has anything been added to the kernel that shouldn't be there? Many times if an attacker is lucky enough to gain root access he/she can then add additional details to the kernel that can hide his or her presence in the system. While detecting these types of attacks is a bit harder and requires external sources (USB Stick with clean binaries) you can attempt to uncover some nefarious acticities.

  • Other Things to Consider

    Network Defense Solutions provides a save method to protect your business by investigating malware attacks and providing full remediation report strategies to help you block, forecast and react to malware and phishing before they become a problem! Combined with training we can assist your environment by dissecting malware and running samples in a safe environment to understand how your adversary is attempting to breach your environment and provide you with meaningful insight and security! We can provide training to your employees and will investigate file attachments, links, e-mails and other objects that your organization deems as suspect.

  • Graceful Shutdown or Pull The Plug?

    There are benefits to each of these methods. However, at the time of your investigation and how you'd like to proceed with your investigation and or the artifacts you need -- this is one of the items / steps to consider.

Warning! Writing any information to a suspect disk can and will lead to a forensic image being dismissed as credible evidence if in the event it does go to court. Additionally almost all states require a P.I. license in order to practice computer forensics! So, if you are doing this professionally we strongly advise against it. If you are doing so for practice. Have fun.

Physical Access to The Machine

Physical access to a desktop, laptop or other type of device is rather crucial. It enables the users to interface with the hardware and software and ultimately allow them to get their work done. Unfortunately this is another method for an attacker to spy on a given end-point or even server if he or she gains access to a server room. The USB devices as well as other devices that are plugged into a computer can tell you quite a few things.

While newer key loggers have come a long way it might not be as prevalant and out there as some of the examples below, but if you see anything similar in nature -- you really should start to scrutinize things further.

Hardware based key logger

If that is not enough, the next thing you might want to scritunize are the cables in use. It's not uncommon for cables to have bluetooth devices hidden within them which can help broadcast the signal of your keystrokes out of the wire. You can see one in action here: USB Hardware Keylogger Wi-Fi.

Although on the physical side of things it doesn't really stop there, keyboards can be replaced which are themselves key loggers. So, if you have had any devices changed out and weird events begin to play out -- don't right away write-off new hardware as a culprit.

While the devices to enable someone to spy on another user can be installed and configured on the box you might be investigating, there are certain things you should also check that wont be ever-so-present with hardware.

Recover Relevant Information

To new security professionals or those starting out in the field, they might forget or skip over quite a few details about a system that might be key to understanding what is going wrong.

Although we have compiled a run-down of questions regarding what to ask in an e-mail or phone call, here are a few other things you should ask when dealing with a frantic caller:

  1. System Information
    1. When was the last time the system was updated?
    2. Do you know the kernel or the patch level of the system?
    3. Are you using any add-on repositories (apt, yum, yast, etc.)
    4. Was the system in the middle of a current update?
    5. What is the date / time that the system is showing?
  2. Security Questions
    1. Was the system installing new anti-malware signatures?
    2. Are the signatures for anti-virus applications up-to-date?
    3. Did the firewall change? Rules added, deleted or modified?
  3. Additional Questions
    1. Did "help desk" call you for assistance?
    2. Did your browser hit a page where it requested you call for remote support?
    3. Has anyone provided support within the last week or last 24 hours?
  4. System Updates
    1. Have you tried to upgrade to the latest OS through the update manager?
    2. Did the problem present itself after a boot?
    3. Is the problem constant or all the time?


Not only are the questions above quite important, the one important question you should ask or attempt to obtain (for threat hunting and additional analysis) you should ask for any logs that the user may have. Or, potentially a time frame in which you can search for between the events; say 01/24/2020 - 2/4/2020. Additionally if you can obtain the information as to where the system is physically located and, the network segment / IP this is also beneficial for your investigation.

If you could record the information in a note pad that would help however, you may also want to get an e-mail detailing the events as well for an audit log if possible!

Running Applications

When dealing with running applications especially in Linux (and Windows, too!) there are a few key things that you can obtain. Of the ways you can obtain running applications you can use the ps command. This command will obtain the process list of the applications running however, there are a few tricks that you can utilize a few switches that can obtain information that will assist in your investigation. If you also combine this information with the network details from a netstat, and a w (who, last) command you can begin to see a broader picture.

For the running application listing you can use some of the commands below:

Display all information that is associated with terminal; x imposes BSD Style

List all processes (-e), with a full format listing (-f)

Combined data from the first command with user info (-u)

Another method that you can use is to utilize the /proc listing for interesting applications that come back. When running a ps -aux command you may see a pid similar to:

In the above code example, we see a few things. While we might not see the process / PID listing (which is the 2'nd column, if we look within proc for the SSH login we have entered the system as, we should see the following:

As you can see from the above the command does return a directory with the PID 1089 which is the pid we have ssh'd into the server with. If we suspect that this process is malicious, there are a number of things we can obtain from the /proc/1089 directory that can help us further examine the system forensically. However, please keep in mind that these techniques are well outside the scope of this documentation. The text is from the output below:

Within more advanced techniques and papers we will explore the information that you see above and what you should be looking for while you are doing your investigation. The purpose is to get you used to collecting the information that you need to provide a meaningful investigation.

While we are on the topic of the /proc directory and the PID that is created within it, there is one take-away that you should pay close attention to. Especially if this is something new or, you want to correlate the events to what the user is citing. If you do a ls -lpa command against the /proc/848 directory you should see some object timestamps. Those time stamps will tell a short story as to when the event started (if the system has not been rebooted or it started the moment you were phoned!):

If you receive a phone call on the 18th of October 2020 at 3PM you can be sure it might have something to do with this!

Login Sessions

Just because we are going off of the ssh login that we created for this example doesn't always mean it will be as straight-forward as we see in this example. So for the sake of investigating a breach, we still need to determine if someone logged in, when the logged in and what times they did so. This is where the w or, who command as well as the last command comes to play!

Running the w or, who command will give you the following output:

If we want to see what the who command will do, we can run the same command but we will get a little less output from the terminal:

Now from this point, if we want, we can also see when the users (in this case serverdude) have logged into the box, or other users have accessed the system. If we issue the last command, we can get a pretty good idea as to how often or how long ago these users have accessed our box:

With that said and done, last can be done in a few ways and we can also combine it to search for specific users such as last -F -i -w which will get us the full times, ip address and the full names. The output should appear as follows:

If you need to isolate some users or a group of users you can either use the grep as follows: last | grep -i "username" or, you can use the egrep tool to find multiple users in one shot: last | egrep -i "serverdude|evildude" and that should give you quite a few users in one shot. If however, you have a select few users you know would be on the box; what you can do is exclude the users. Using both egrep or, grep you can do so as follows: last | grep -i -e "serverdude" with egrep, simply enter the following: last | egrep -e -i "serverdude|devdude" and it should eliminate the common names. However, on a side of caution -- be very careful when using this as many times attackers may log into systems with legitimate names!

Logon Sessions / Authentications

It might not stand out and slap you in the face quickly. However, if you have an attacker knocking at your door -- or you have authentications that are coming from weird locations, this is where you need to start scrutinizing your login and authentication scripts. Be it someone in the office or, someone across the world. There are a few things you must consider when you are looking at the login sessions and or the authentication requests. For this, we are going to set our sights over to the /var/log directory and start scrutinizing a few things.

If we start looking at the /var/log/auth.log account and we issue the following command: grep -i "serverdude" auth.log | grep -i "failed password" we can start to potentially draw a picture that someone has been trying to bruteforce the account in order to get into the system. When the command is run successfully, we should see something like this:

If we do see a number of additional password attempts that attempted access from the offending IP address or the IP pool, we can begin to either deduce something is trying to break in or has (depending on the evidence that we have). From this if they were able to (in some cases) keep the same IP address and get in, establish a connection and do some damage, we can pull that information from a netstat command which we will demonstrate later.

We should point out that at this point if we see an IP address that is outside our corporate range we should do a reputation check on it. If not, and it's internally we need to start scrutinizing the system that is making these requests. Chances are another box that is "trusted" has already been popped and this could be indicative of lateral movement!

User Accounts

Outside the grounds of utilizing an exploit to get into a system -- let us imagine for a moment that a password is guessed, phished or found under the keyboard. Because you know, the sentiment of hand written notes today is a lost art and should be appreciated. Many times attackers will either create a new user. At times the user name might be hidden with a . prefix before the name such as: .myuser instead of the myuser in the /home/ or /Users/ directory (depending on your operating system of choice).

With that said, there are a few locations that you can statr to pry in order to determine if there may be a hidden user on the system you are investigating. First, you can navigate to the home folder. If you're not quite sure where the home(s) are located you can issue the echo $HOME command in order to locate it quickly. We have include some sample output that can help demonstrate this for you:

From this example if you remove the user name, you can see the the users are all homed within the /home/ directory. if we switch into the home directory, we can issue the ls within the /home/ directory and see the users who have home folders in the location we are currently investigating. Now keep in mind there are a few ways we can do this! The first way we can do this is as follows:

Here we see that there are no other users.

In most instances, seeing something like this would be just enough and we can move on, right? Wrong! If we issue the ls -lpa command, we see something is a little off here:

A new user named ".hacker" has been created.

Now that we see a new user account making a guest appearance, there is one thing we need to check. That location would be the /etc/passwd file. Although weeding through this user file can be quite dizzying we're going to show you a few short-cuts that you can do in order to make short work of this. However, before we show you the easy way -- we're going to show you the annoying way:

Now, most of the relevant information within the passwd file wont be present for obvious reasons (it's too large to paste). What we can do is, utilize grep in order to weed through to see if the user hacker is in the passwd file and what shell he/she has. First, we will check to see if the user account of "serverdude" is present and then we will search for hacker to see if that user is present as well:

Now, we can use the same search but with the other users name, such as:

We see from this we return nothing. Which tells us the user is not on the box. However, if the user was in the /etc/passwd file we have bigger issues on our plate. The one thing you need to keep in mind is that sometimes attackers will name accounts similar to the ones we have on the box. For instance. We have the account serverdude, an attacker might create an account called servedude or any combination of the like to throw us off. The lesson from this? Always check the spelling of the accounts on the box and if need be, remove accounts you do not need! However, that is a story for Linux hardening.

Network Information & Sniffing

When we are collecting evidence for an incident or a forensics for that matter we would also want to collect a few key elements from the network. This could be network interface cards and their status, routes and additionally netstat commands to see who or what is connected to our system. Some commands within the Linux and Unix environment may be ambiguous. Meaning oute -n and netstat -rn should produce the same output. If they don't something highly suspect is going on in the system!

First thing is first. First, we want to list out the information about our network adapter and see what can potentially be wrong with is. Our introduction to our interface cards will be ifconfig -a as shown below:

While it does appear that everything is running fine and we have access to the outside world an astute analyst will see something is definitely off on the lin of: UP BROADCAST RUNNING PROMISC MULTICAST this single line is telling us one thing. Someone or, something is snooping as we are in promiscuous mode (promic). This means that packets can be captured if they are utilizing tools such as Wireshark, tshark, tcpdump or others! This is a dangerous situation because someone or something may have already gathered information and whats worse -- if we are engaging with sites that do not use SSL/TLS they now have our credentials. Some companies may see that internally they don't need certificates, here is something that may change your mind. Clear-text passwords can aid in lateral movement! Let that set in.

The next two commands that we need to run would be for our routing. This segment will contain a total of approximately 4 more commands and we can call it quits for the network segments. First we will run route -n and netstat -rn. Both of these will display some output as to what path your traffic takes before it leaves your network and goes to the internet. Here, you'd probably need to know the topology of your environment to make sense of this.

The first command that we will issue will be the route command:

Negating the fact that the information that is displayed is possibly correct, the next command we will issue will be the netstat -rn command. It will become much more clear why we are running these commands in the following segment:

So, we see that the output is the same, correct? Well. Here is a little hint. If the commands did not return the same information, you can begin to deduce one of two things. 1) something is wrong with your network or, 2) there is a rootkit or application that is blocking the command from completing successfully!

Although that the information we provided herein, will tell us the routes, there is another command (netstat) that we need to run to determine the connections that are established to the machine and, potentially the foreign IP addresses which may be malicious (which you will analyze off the box!). While the commands will return a lot of information to mull through, we will provide you with some information that you can utilize in order to shorten your investigation. The command we will be looking at will be netstat -anp:

While this information looks like the wild-west of text. To new analysts or users who are involved in incident response / forensics this is probably quite intimidating. If however, you know what you are looking for or you see that there are foreign IP addresses (in our case it's a we know we can start cross referencing that, or if we know the port the attacker is using (maybe 4444, 22, 80, 443, etc) we can do some filtration. In this example I will show you how you can filter information you might not need however, keep in mind! If you filter things from your initial investigation you might miss a bigger picture so only filter on the data you captured!

To filter some of the information we will attack this in one of three ways. 1) do so by port number, 2) do so by service, 3) do so by foreign addresses only. Lets take a look at how we can accomplish this:

To display only TCP ports we can use the following command: netstat -at, this will provide us with the following output:

From this command, we can see that the ssh service is listening on local host [::] on the IPv6 address space, and we can see that ssh is established on the address (local) but being access from a foreign address The port that you see 6349 is the port multi-plexed port. Unfortunately that is outside the scope of this tutorial and will not be discussed. On the far right column we see that the command is in fact established meaning the threat actor is connected to our machine.

If we wanted to go a step further and, we wanted to do some filtering we could issue the following command: netstat -anp tcp | grep ":22" which would show us only the sessions over ssh/d. If we wanted to get even crazier we could do: netstat -at | egrep -i "ssh|sshd|:22" we could be even extra and do some awk commands but I don't want to complicate the tutorial. After all, we are just utilizing this as a learning piece. Below, we will show the output of both commands so analysts can gather an idea of what they will be looking for with each set:

netstat with at command and grepping for ssh.

netstat utilizing egrep searching for multiple instances of ssh.

Considering this is just a "quick" snap-shot of the output of the netstat command, the thing we need to understand that from the output we really don't get to see the PID that the service or connection is running on. To obtain this information we would need to run the netstat -anp | grep "" and locate the portion that says ESTABLISHED #/sshd: and where, # indicates the PID. We can see this in the following command where we see the PID as 1089

Kernel Modules

Now, there are a few things that we have to state and these need to be taken with a grain of salt -- a large one. Many times if the attacker is smart enough to load the system with a rootkit, some of these commands will either run and return bogus information; or they will do nothing at all! This is where different tactics will come to play when attempting to catch an attacker. Those techniques we will cover in other documentation. For now the purpose of these documents are to get you accustomed to utilizing the commands and hopefully tracking something down.

In order to see which kernel modules are installed, you can use the lsmod command in order to see which modules are loaded. In our environment because we have virtualized the sytem you will see a few things potentially discussing VMWare. But the commands you will normall would use will be lsmod to list modules, insmod to insert modules, rmmod to remove a module and modprobe to probe for a module.

One of the things we will state is that the preservation of logs within /var/log (also depends on your version of Linux) will dictate what modules have been loaded. You can do this with the grep or, the egrep command as we have shown you previously. Before we start demonstrating how to search and scan for the insertion of modules in the kernel, let us demonstrate how to list the modules that the kernel has loaded:

In the following example, we have inserted a module into the kernel called speedstep-lib.ko. if we re-run the lsmod command, we have limited the command to grep just the information for speed step as follows: lsmod | grep -i "speedstep" to show the inserted module. If however, you want to be extra you can issue the command in alphabetical order by entering: lsmod | sort and find it that way. We're just too lazy -- but remember! You want a full output for investigation purposes.

Now we can confirm that speedstep has been inserted into the kernel. However, considering it's there -- where can we look to determine if that was inserted properly? One place that we can look would be in the /proc/modules location by entering cat /proc/modules and grepping for the speed designation as follows:

Now we have double confirmation that the module has been run and inserted. If we need to further scrutinize a module on the system we can run the modinfo command. In this example I have taken the lazy approach to make the system do the work for me, but you can see the information that can be returned:

Since we did briefly discuss the topic of rootkits, we have two methods up our sleeves to potentially determine if a system has been affected by a rootkit or "tainted" (see what we did there?). The first method we can use is to cat the /proc/sys/kernel/tainted file by entering: cat /proc/sys/kernel/tainted and attempting to determine the number of tainted objects returned (or calls) In our example below, we don't really have any:

Before we go into a full blown panic, kernel taint doesn't always mean that there is a rootkit!!! It could also be indicative that the kernel is not in a stable state. It simply indicates that the kernel may be unreliable. Other testing needs to be done before a rootkit can be deemed the culprit so please do additional due diligence before making such a suggestion.

With those warnings out of the way -- we can also do something a bit extra to gether the hashes of the loaded kernel modules, or all the files (if they are within the /lib/modules) directory is utilize the following command: find /lib/modules -type f -exec md5sum {} \; `lsmod | grep -i | awk '{ print $1 }'` this command will produce all the hashes in md5 format so that you can run them against virustotal or other malware hash lookups. You can also pipe the output to a file by doing find /lib/modules -type f -exec md5sum {} \; `lsmod | grep -i | awk '{ print $1 }'` >> kernel_hashes.log

Other Things to Consider

While we really wont bore you with too much text and pictures in this segment you need to understand a few things before you begin examining a system. 1) You will need to know the exact kernal and system you are working on, this is where the uname -a command will come into play, as well as the uname -srm to know the exact kernel that you are running. The reason why you would want this information is so that you can obtain information and binaries for your version and load them to a pen drive! We normally do this as to not invoke any of the binaries of the system so if there is a rootkit on the box we are not running or relying upon a tained system for our investigation information.

Additionally you will want to make a backup copy of your environment variables so that you do not have to play guess work when putting the system to it's original state. The purpose of our Linux Incident Response Script is designed for this sole purpose. Please keep in mind it may still be in beta and we are still testing (that is of this writing). Our script will make a lot of the issues we are discussing here easier for you to accomplish.

DO NOT under any circumstances compile or install applications directly on the affected system, end-point or server! You will only cause additional confusion in the incident or forensics process and potentially trample on important data that will most likely be tossed in a court of competent jurisdiction for improper procurement.

If you can avoid running anything on the box, do so. Additionally have most of the tools you need on a large pen drive and any other large amount of data that you need to copy off the machine do so with tools such as: Pack of 16GB PNY Thumb Drives or, to save larger files / objects you can opt for a 4TB Western Digital USB 3.0 Drive or if you like seagate we recommend: 4TB Seagate with 2 year data rescue. Ideally we prefer ironwolf hard drives with a 3.5" Enclosure USB 3.0. If you will be working off the suspect hard drive after it has been removed you can go for an expensive option and obtain a hard drive hardware write-blocker or you can go the inexpensive route and disable write operations on a Linux system that you will be using solely for incident response / forensics.

In addition to which, we will also be linking other documents into this mix that help you create duplicates of the hard disks, as well as systems that do a 1:1 bit copy of the devices you are working with. However, those will be articles for another time. In those articles we will discuss dd, the forensics DD tool and hardware to make your life easier!

Pull The Plug or, Shut Down?!

It's beginning to feel as though systems should come with a DNR form. The argument for either approach stems from a few key arguments. If you shutdown the system in a clean state, malware has a chance to clean up which can in the process destroy important forensic information. If you are quick to pull the plug and you didn't extract any memory information with the usage of LiME (which we will talk about in another article) you've lost a treasuretrove of information. However, if memory has been captured and you pull the plug you wont lose any additional information!

Keep in mind that if you are going to pull the plug running the sync function before doing so will assist with things being saved to media properly. This will assist when we are creating a disk image of the suspect drive! An additional point to discuss is encryption. If the system is using veracrypt, truecrypt or bitlocker and you have access to the system now you don't have to crack or try to get into it later in certain circumstances.

One last point to make when pulling the plug, you might want to pull the plug directly from the back of the tower / system where it plugs into the Power Supply. This is just in case if you pull the surge protector plug and it also acts as a temp UPS you just sent a shutdown signal to the box.

Print   Email