How to Detect a Hacker Attack

Most computer vulnerabilities can be exploited in a variety of ways. Hacker attacks may use a single specific exploit, several exploits at the same time, a misconfiguration in one of the system components or even a backdoor from an earlier attack.

Due to this, detecting hacker attacks is not an easy task, especially for an inexperienced user. This article gives a few basic guidelines to help you figure out either if your machine is under attack or if the security of your system has been compromised. Keep in mind just like with viruses, there is no 100% guarantee you will detect a hacker attack this way. However, there's a good chance that if your system has been hacked, it will display one or more of the following behaviours.

Windows machines:
Suspiciously high outgoing network traffic. If you are on a dial-up account or using ADSL and notice an unusually high volume of outgoing network (traffic especially when you computer is idle or not necessarily uploading data), then it is possible that your computer has been compromised. Your computer may be being used either to send spam or by a network worm which is replicating and sending copies of itself. For cable connections, this is less relevant - it is quite common to have the same amount of outgoing traffic as incoming traffic even if you are doing nothing more than browsing sites or downloading data from the Internet.
Increased disk activity or suspicious looking files in the root directories of any drives. After hacking into a system, many hackers run a massive scan for any interesting documents or files containing passwords or logins for bank or epayment accounts such as PayPal. Similarly, some worms search the disk for files containing email addresses to use for propagation. If you notice major disk activity even when the system is idle in conjunction with suspiciously named files in common folders, this may be an indication of a system hack or malware infection.
Large number of packets which come from a single address being stopped by a personal firewall. After locating a target (eg. a company's IP range or a pool of home cable users) hackers usually run automated probing tools which try to use various exploits to break into the system. If you run a personal firewall (a fundamental element in protecting against hacker attacks) and notice an unusually high number of stopped packets coming from the same address then this is a good indication that your machine is under attack. The good news is that if your personal firewall is reporting these attacks, you are probably safe. However, depending on how many services you expose to the Internet, the personal firewall may fail to protect you against an attack directed at a specific FTP service running on your system which has been made accessible to all. In this case, the solution is to block the offending IP temporarily until the connection attempts stop. Many personal firewalls and IDSs have such a feature built in.
Your resident antivirus suddenly starts reporting that backdoors or trojans have been detected, even if you have not done anything out of the ordinary. Although hacker attacks can be complex and innovative, many rely on known trojans or backdoors to gain full access to a compromised system. If the resident component of your antivirus is detecting and reporting such malware, this may be an indication that your system can be accessed from outside.

Unix machines:

Suspiciously named files in the /tmp folder. Many exploits in the Unix world rely on creating temporary files in the /tmp standard folder which are not always deleted after the system hack. The same is true for some worms known to infect Unix systems; they recompile themselves in the /tmp folder and use it as 'home'.
Modified system binaries such as 'login', 'telnet', 'ftp', 'finger' or more complex daemons, 'sshd', 'ftpd' and the like. After breaking into a system, a hacker usually attempts to secure access by planting a backdoor in one of the daemons with direct access from the Internet, or by modifying standard system utilities which are used to connect to other systems. The modified binaries are usually part of a rootkit and generally, are 'stealthed' against direct simple inspection. In all cases, it is a good idea to maintain a database of checksums for every system utility and periodically verify them with the system offline, in single user mode.
Modified /etc/passwd, /etc/shadow, or other system files in the /etc folder. Sometimes hacker attacks may add a new user in /etc/passwd which can be remotely logged in a later date. Look for any suspicious usernames in the password file and monitor all additions, especially on a multi-user system.
Suspicious services added to /etc/services. Opening a backdoor in a Unix system is sometimes a matter of adding two text lines. This is accomplished by modifying /etc/services as well as /etc/ined.conf. Closely monitor these two files for any additions which may indicate a backdoor bound to an unused or suspicious port.