Linux admin with experience dating back to early UNIX servers - AMA

[SmokeTree]

Unless we're talking about a windows VM to run scrapebox and the like on, or a special use case that would not be a good fit for Linux, I'd fathom that the vast majority of you are hosting your projects on a Linux server of some sort. That's great and exactly what you want to be doing, but how much do you know about Linux? If someone took your cpanel away, could you survive with just an SSH client, a console and Google?

I live almost exclusively on the console. If I use any kind of cpanel tool to manage servers, it's something light like ajenti (http://ajenti.org/), and that's only for what it can simplify for me. With that said, while this is an AMA thread, I will be honest and say that I'm probably not the best choice to answer the "how do I do X in cpanel" type of questions.

So what say you BuSo? Do you have a server that you're seeing an issue with, questions about different shells, etc? Just have a general Linux question in particular? Post it here

 

[thamonsta]

I have wanted to try out an unmanaged server or digital ocean or something but gotten absolutely confused starting from scratch in the console, but that's all well-documented and that's not my question...

What my fear is that once I do get Wordpress installed and running on the server, what type of backend things do I need to do in terms of security to make sure my server doesn't have a huge neon sign saying "HACK ME!"? Is there a lot to it? I mean, I haven't done much in terms of extra security precautions for managed cPanel-style servers I have had in the past but I was just imaging that going in raw on Linux means there are things I'll need to do.

Basically I don't know what I don't know, so hoping you can shed some insights and hopefully make this less intimidating for me haha.

Thanks for doing this thread I think it's a great idea and it will help a lot of people who have questions about this but don't really know what or where to ask.

 

[eliquid]

I got some stuff for you.

1. Top 5 ways to harden a server. Programs, routines, etc
   
2. How to set up load-balanced MySQL servers so that not only are request balanced, but the servers are updated with the same info correctly.

Ill think of some more...

 

[emp]

I can do fine administrating an Apache server or even bash scripting... The Linux way of handling disks and partitions had me completely stumped, however. Any good tutorials or explanations?
::emp::

 

[SmokeTree]

What my fear is that once I do get Wordpress installed and running on the server, what type of backend things do I need to do in terms of security to make sure my server doesn't have a huge neon sign saying "HACK ME!"? Is there a lot to it?

Hey, great question! I'm going to share some things I hope are both useful and easy to follow. If you do these things, you're well on your way to having a reasonably solid WP install and server environment. @eliquid has asked in another post here about server hardening and I'm going to go into that in a bit more detail about that, which will include some of the more advanced server hardening techniques such as iptables/fail2ban/inotify/etc. Stay tuned for that, but in the meantime, I'm going to touch on some basics and what I feel everyone that is new (or not so new) to running their own VPS or dedicated server should address.

*Disclaimer: This list is not all-inclusive and only meant to be used as a general guideline. Also, this is not meant to replace the advice of a professional Linux Security Consultant. I have done my best to ensure the accuracy of these answers, but please know that what I present to you and the links that I include might not work for your particular use case or Linux distro. Now with all that over...

Keep the server upgraded

This is pretty painless, especially if you are using Ubuntu or other Debian based Linux distro (CentOS and the like can do this with the "yum" command). I'm going to use Ubuntu as an example. Here's what you have to do on the command line:

Make sure you are running as a user that can run commands as root using "sudo". With that said, I don't advise anyone to run as root, always use a non-root user and use sudo. If you are logged on as root, you are leaving an unnecessary shell open logged on to the root account. Any command you run will run with the power of root and the power to destroy worlds, or even worse, your system.

Here's a good example of how to set up the sudoers file for Ubuntu, but CentOS is also included: https://www.digitalocean.com/community/tutorials/how-to-edit-the-sudoers-file-on-ubuntu-and-centos

Once you have a sudo user, the following are the basics of what you'll need to do to keep your server updated. As a non-root user with "sudo" capabilities, run the following commands:

sudo apt-get update && sudo apt-get upgrade

*Pro Tip: The command above uses "&&" which is basically a way to run 2 commands, one after the other. I use this a lot to "chain" certain commands together. The good thing about "&&" is it only runs superceding commands if the command before succeeds (returns 0).

If during the upgrade, you are asked if you want to replace your copy of a config file with a new one, it's best most of the time to keep the local config, or else your config file will be replaced with the new one, over-writting your settings. You don't want this more of the time, and it's something I just want you to be aware of when you see it.

This will help with the current "version" of Ubuntu you have installed. If there are new versions, the steps to upgrade are a bit different but there are a few risks involved and something I'd be glad to touch on in the future if folks are interested.

File Permissions

Highly important! If the wrong file permissions are set, you're leaving a wide open security hole, ripe for many attack vectors.

Here's an article describing the file permissions for WP and how they should be set up. The article also goes into a bit of detail about how the file permission system in UNIX/Linux works: http://www.smashingmagazine.com/2014/05/08/proper-wordpress-filesystem-permissions-ownerships/ Take a look and if there's anything there you'd like me to clarify, feel free to ask. As a general rule, you should stay away from anything/anyone requiring you to change your file permissions to 777, which basically gives the whole world read/write/execute access to the specified file/directory. There is rarely a good reason for this as the problem can be solved with groups and giving full rights to the group.

Key based login only (no passwords)

You'll want to set up SSH on your server to only allow key based authentication. As you can imagine, this is way more secure than allowing access via password: https://www.digitalocean.com/commun...sh-key-based-authentication-on-a-linux-server . One of the primary reasons you want to do this is to keep anyone from trying to brute-force your server by running through dictionary based attacks and other password based attack vectors. If you enforce the use of key only, you've stopped the problem in its tracks because when the hacker bots (or humans) tries to log in with a password, they will be shown the door with a message that says that password based auth isn't allowed.

FTP vs SCP

If for some reason you need to access your server to upload files, never, under any circumstances go with standard FTP. FTP passes passwords in plain text, certainly not what you want. You can either set up a server that implements SFTP or you can use a shell account with an SCP program which pretty much does the whole deal over SSH using the key based authentication described above. The SCP option is the path of least resistance and would be a good choice if you totally trust who is logging on to the server as it uses the "real" shell accounts for this. If the users accessing your system are at all questionable (trust issues), it might be a good idea to set up an SFTP server that will do a "chroot jail", most of the time to their /home/user directory. If you decide SFTP is what you want, just let me know and I'll post on how to do this.

.htaccess file

This file is like a very basic "firewall" of sorts without true firewall capabilities like stateful packet inspection. @CCarter has written pretty extensively about this file and what it can be used for. I'm going to share a link to something pretty broad so you can see exactly what can be done and how it can be used to protect your sites. @CCarter I know there are a few links that exist for your .htaccess file and if you'd like to share the link to the most recent one, please do. In the meantime, this should suffice: http://www.inmotionhosting.com/supp...-unwanted-users-from-your-site-using-htaccess

Check your Plugins & Themes

Check into each and every one of the plugins you'll be using prior to installing them. Here's the thing about plugins. Plugins are all allowed certain permissions on your WP install. They are allowed CRUD operations on database tables and files (Create, Read, Update, Destroy), and can do all sorts of other things you'll definitely want to be aware of.

I recently had a chat with someone who found that attempts were being made to exploit their site, and after investigation, isolated the culprit to a plugin that was doing things it shoudn't. The short version of this is to make sure the plugins you install have some kind of a "rep". I normally use the age of the plugin and I also read reviews before determining whether I should install the plugin. A plugin that has been around for a while and has had several version changes is something I'd be more inclined to "trust" than something that just all of a sudden "surfaces" one day.

You might not be aware, but Themes also have the same rights as plugins and can be exploited in many of the same ways and should be treated just as cautiously as plugins. Here's something important to think about. Plugins and Themes are created by developers who are good and bad, both in intent and skill level. When you invite a theme or plugin into your WP "home", you are trusting that the creator of the plugin has both good intent and good skill, and even then things can go wrong. People make mistakes, that's for sure, and one simple mistake could lead to the site getting exploited.

I hope this helps. Please feel free to ask me to clarify any part(s) of this or if you'd like me to go more in-depth about any part(s) of it. Also, if you have "best practices" of your own you'd like to share, feel free to do that as well.

 

[SmokeTree]

Top 5 ways to harden a server. Programs, routines, etc

Server Security and Hardening - My Take

We're constantly at war. The doors of our servers are constantly getting knocked on, like it's all some type of sick prank. If we open the door just a crack, packets are going to come rushing in like an episode of "The Walking Dead", only these fkers are alive, well and ready to kick your server's ass, if given the chance. If this were our homes, we'd want to put a stop to it quick, so shouldn't we think about our servers the same way? How do we protect our critical servers, the ones that are responsible, in part, for our bank accounts being a bit fatter?

Sometimes we need to get back to basics and break out the "Guns & Dogs"

Keep The Server Updated

This one is a given and I'll expand on what I wrote earlier in this thread. Make sure you're keeping the important stuff upgraded (SSH/Bash comes to mind in particular), but also, it normally doesn't hurt to upgrade the distro you're using either. I see a lot of people that seem to "hold on to dear life" to older distros of Linux and it can only hurt you in the long run when you want to get that updated software package and you run into dependency nightmares that lead into monkey-patching tons of stuff or worse. With Ubuntu, it goes kind of like this:

• sudo apt-get update
• sudo apt-get dist-upgrade
• sudo do-release-upgrade

IPTables

*iptables is a tried-and-true standard and is what I'll talk about here. I'm experimenting with "nftables" and will write about that another time when I feel more proficient and qualified to do so.

When anyone connects to your site, well before they even get to an .htaccess file, iptables is there ready to guard the doors. I normally start with my own server or a client's by identifying a few things:

• What services are running and what ports?
• Who or what needs access to these services?
• Do they have a static IP they will use to access the server, or will they be logging in from other IP address?
• Are there any services that need to be public (e.g. http)?

Based on the answers, I build the appropriate firewall in iptables. Some things to note about iptables:

• The rules are read from top down until a match is found or it runs into the firewall default policy. With that in mind, it's good practice to arrange the rules by the most common. If you're allowing http (tcp 80) globally, put that rule near the top because it will be matched a bit faster
• Iptables is a hell of alot faster than .htaccess for blocking things. As a general rule, if you know the IP or IP range of a user agent, block that in iptables as they won't even reach apache. If you have fat .htaccess files, remember that everytime someone connects to your server it has to go through all that which does add to your overall response time. Use iptables and .htaccess to get a good balance of practicality and best use. There is no "one size fits all", just try to be mindful of what the strengths/weaknesses of each are
  
• Make sure you set your default policy to DROP rather than ACCEPT. You'll want to drop any packets that don't have a matching ACCEPT rule

My thoughts on DROP vs REJECT. I choose to DROP because REJECT generates an answer message. DROP just drops the packet. You'll hear differing opinions about this and mine is to DROP. If you have a reason to REJECT, do that

I'm going to hold off on a link for iptables till the next section, "Intrusion Detection"

Intrusion detection with psad (http://cipherdyne.org/psad/)

Here's a link that explains how to run iptables with psad: https://www.digitalocean.com/commun...t-network-intrusion-attempts-on-an-ubuntu-vps

Implementing iptables is great, but there is one critical element missing. How about a way to detect port scanners, certain hacking attempts (can't catch them all) and other type of suspicious network activity? Psad to the rescue.

Why psad?

• If there are bots (or humans) trying to port scan your server, psad can issue a firewall DROP rule and stop them in their tracks
• If there is suspicious network activity, psad can try to mitigate with firewall rules or just notify an administrator
• You can't watch your network logs 24/7, psad can

So, for those of you that "has a friend that knows a friend" that has used port scanners and has seen like maybe 50 successful ports scanned then it all of a sudden just kinda "dies out", chances are what is happening is a rule in psad or other intrusion detection sys has been triggered. Some admins also run "tarpits", but honestly, that's just a waste of resources to me.


SSH - Configure Key Based Authentication

See above post in this thread for explanation/article on how to do this


Running services on non-standard ports

So you've got SSH running on port 22. Is that necessary? How about changing the port to something a bit higher? In Ubuntu this is as simple as:

• Edit your firewall rules appropriately to allow access to the new port
• Edit the file here (use sudo because you need root for this): /etc/ssh/sshd_config
• Change the port from 22 to something else, preferable a much higher port
  
• Save the file
• Restart ssh with: sudo service ssh restart (make sure you do this from your recovery console, otherwise there may be issues)

Try to log into the new port. If it worked, good job. All those bots pre-configured to brutalize port 22 will pretty much "go away", leaving you with the more sophisticated ones that are going to try to scan ports, but hey, we're good to go because we covered that with psad, right?

File Permissions

So, you have your webserver running and you're wondering how to configure file permissions?
It's easy. If the server needs to write to a file or directory, you give the user permission. If all it needs to do is read and never write to certain files, set permissions accordingly. Just like where I wrote above on how to secure a WP install, the same exact techniques can be applied to a SAAS or just about any other scenario. The short story is to "give only what is needed and nothing else".

Inotify

inotify gives you the ability to detect changes to files and/or directories. This has a TON of uses, only limited by your creativity and imagination, but a bit of coding is involved. I'm sure you can think of many use-cases for this. Some that come to mind are:

• Watching certain directories on WP or any other crucial site directories for changes and taking some kind of action
• Watching certain files that shouldn't be changed and alerting/taking action if the file is changed

Inotify has bindings for most popular languages. The following are for Ruby & PHP.

• For Ruby: https://github.com/nex3/rb-inotify
• For PHP: http://php.net/manual/en/book.inotify.php

Since I work with Ruby mostly, I'll provide a code snippet. To use this, you should have Ruby 1.9 or above and also have the "rb-notify" gem installed by doing "sudo gem install rb-notify". After that, you'll be ready to run the following code. I use a directory path of /test-dir as an example here (make sure it exists first):

#!/usr/local/bin/ruby
require 'rb-inotify'

# make a new notifier object
notifier = INotify::Notifier.new
# set this to be whatever dir you want to watch
dir_to_watch = "/test-dir"

# we are setting up a block here to initialize the notifier, with callbacks.
notifier.watch(dir_to_watch, :modify, :create, :delete) do |event|
  event.flags.each do |flag|

    #get the string value of 'flag'
    flag = flag.to_s

    # here is where we handle the type of modification that is detected.  Replace with callbacks of your own that can do anything you like
    case flag
      when 'create'
        puts "#{Time.now} - #{event.name} created in #{dir_to_watch}"
      when 'delete'
        puts "#{Time.now} - #{event.name} deleted in #{dir_to_watch}"
      when 'modify'
        puts "#{Time.now} - #{event.name} modified in #{dir_to_watch}"
    end

  end
end

# start the notifier and go into an infinite loop of doom...
puts "#{Time.now} - monitoring #{dir_to_watch}"
notifier.run

If you run that code in a console, you'll see a message that states a timestamp and the fact that the directory you set up is being monitored. Loop of doom begins.

This means the program is sitting there, waiting for someone to make a wrong move with files in that directory. At this point, leave what you have running and go into another console and cd to the dir you specified and start doing things like creating files, modifying files and deleting them. When you do each operation, notice how the console that's running the Ruby program tells you what's going on, whether a file was created, modified or deleted. You can replace my callbacks to stdout with actionable callbacks of your own.

Extra Credit

Get out of shell hell. Switch from bash to zsh. Seriously, bash is an oldie and is useful, but let's change with the times, shall we? Why are all the "cool kids" using zsh? http://www.slideshare.net/jaguardesignstudio/why-zsh-is-cooler-than-your-shell-16194692

I'm also a fan of "oh-my-zsh". With this you have all kinds of themes, integration with git, rails, etc and other goodies that you can spend hours messing with: https://github.com/robbyrussell/oh-my-zsh


Good utilities/apps to have

• TMUX - Do you spend a lot of time using SSH with "screen"? Cool, but I can show you something better. Give TMUX a try: http://tmux.sourceforge.net
• HTOP - We all have used the standard top command, but how about something a lot better? Enter htop: http://hisham.hm/htop
• nmap: It's a port scanner. Great for testing your IDS and things like that. http://nmap.org/
• zabbix or something similar to monitor servers: http://www.zabbix.com/ This is another app that can be used to "take action" if certain situations occur

I know I haven't covered all the bases but I do know that doing the above will make your server a lot more secure. As always, please include utilities and best practices of your own. It is my hope that sometime in the future, some of this content and content from the community can be extracted into "mini-guides" and shared with the BuSo community in that form also.

Thanks for taking the time to read this. I'm seriously no "guru" or anything. I'm just someone that has done this since about '95 and stays passionate about it all. Take what you want, leave the rest. I hope that this will help some of you to get a better grasp of Linux security, at least my take on it.

 

[SmokeTree]

The Linux way of handling disks and partitions had me completely stumped, however. Any good tutorials or explanations?

I'll try my best.

Partitions explained

Partitioning a hard drive is analogous to the way one would separate the space in a building into smaller rooms. In this case, the building is the hard drive and the rooms are partitions. Think of files and folders as different areas of the room and "things" within these rooms. It's all about trying to isolate set-aside sections of the drive so they can exist independently of each other.

By default, some Linux distros will ask you if you want to just go ahead and use the whole drive as a single partition, making it mostly "/" with some room put aside for "/boot" which contains your boot images, "/dev" which contains descriptors to your devices, and sometimes "/usr" is made into a separate partition, but not always. You can go with that and keep it like it is, but I'd still recommend a swap partition, which I'll explain in a bit.

There are a few scenarios to consider when partitioning, and I'll offer a bit of explanation as to how things were done in the past:

Speed

With SSD drives this is no longer an issue as the lookup speed is pretty close to instant, whereas the hard drives of yesteryear relied on moving parts and physical location. It's a different ballgame today as it is all done digitally. No more cracked out record player thing going on to worry about. In short, no speed difference is going to exist due to "physical" separation.

File System Limits

A different story, however is if you want to overcome the limits of the file system by spanning multiple partitions (e.g. 2TB limit). This isn't a common use case and will be something I can write about in the future if need be.

Separation between system and user files

Here's what a lot of people are doing and what makes good sense:

• Set up a root partition at "/" and give it a reasonable amount of space. This is where your sys files and such will reside, so it's a good idea to be on the generous side with this.
  
• Set up a separate partition at "/home". Within /home, you'll put your own files and files specific to various users of the system.
• If for some reason the "/" partition gets trashed (OS failure, hey it happens), you can reload the system and leave the "/home" partition intact, thus saving all your files from certain doom. You can give the rest of your space to this one, just leave a bit of room for a swap file.

Wait a second, did I just say "/home"? Wouldn't that be considered part of "/"? This is where things get confusing and was one of the first things I had to wrap my head (he said "head") around. If you go with one huge partition in "/", then "/home" is going to be a part of that main "/" partition. We have no "walls" there separating the two, they are essentially one.

I'll see if I can do this some justice. In Unix/Linux everything is a file, or more accurately "everything is a file descriptor". Your hard drive, cd drive, printer, everything is represented by a file. The file descriptor literally "describes" the type of resource.

To mount the various devices on our system (usually in the /dev dir), we create a blank directory to use to reference the device or file system

When you create partitions, what's happening behind the scenes, is that mount points are being set up so you can conveniently access the resource by, say "/home" or "/home_of_sacred_files". When you access "/", you are really accessing something like "/dev/vda" (I used a VM as an example), which is the true path of that partition.

You might have seen something like this to mount a cdrom (short version):

sudo mkdir /media/cdrom && sudo mount /dev/cdrom /media/cdrom

So what's going on here is we're creating a directory to use as a mount point (/media/cdrom), then were mounting the cdrom at /dev/cdrom using the new directory we created. Now, /media/cdrom is a "connection" of sorts to the CD/DVD drive, so when we do a "cd" to /media/cdrom, we can treat the CD like it was just another directory on our system with the exception of it being read-only (Writing is a bit different).

Partitions are basically no different except that you're setting them up for pre-allocated space. So when you choose a set amount of space for "/home" to exist independently of "/", what's happening behind the scenes is a mount point is created to "/home" and that's why it appears to be a direct part of "/".

Your requirements are how you determine the best partitioning setup. Personally, I always like to reserve a bit of drive space that is either unallocated, or a partition to use as storage space that is separate from the root directory "/". Even if the OS crashes and can't be recovered (only had it happen once and that was slackware MANY years ago), you'll be able to get to "/your_partition" or any other partition you created, by creating a mount point to it using a recovery disk.

Swap file:

Just like with virtual memory in Windows (Paging file), Linux uses a swap file as a memory overflow storage. In many ways, this is like a safety-net of sorts and is a great idea to set up (in many cases a must have). If your system happens to run out of memory, the swap file will fill so the system can continue to run, with the cost being that it will run about as fast as a horse on a pogo stick, as it's trying to use your HD as RAM.

There are opinions on how big this file should be. A general rule of thumb IMHO is at LEAST the size of the RAM you have installed, with a lot of people recommending double the RAM. I'd weigh out the total disk space and then decide how much I want to give up to be used as swap.

If you have an existing system with a VPS, here's a great walkthrough (for Ubuntu as most of my examples are) that will show you how to create and set up the swap file to run at boot and how to configure. This works with existing partitions: https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-14-04

*I mainly use Ubuntu in my examples because it's by far the most popular distro I see in the wild. Also, I post a lot of links to digital ocean for examples. The reason is that I find the walkthroughs on digital ocean to be pretty concise and accurate, and the same info applies to pretty much any VPS or VM.

Here's a good tutorial that explains partitions for both Windows & Linux and describes the "gparted" utility (you'll be using "parted" for a server as "gparted is Gnome based"): http://www.howtogeek.com/184659/beginner-geek-hard-disk-partitions-explained/

I hope this helps explain basic partitioning in Linux. Please, fellow builders, if you have anything to add, by all means feel free. I'm doing most of this off the top of my head and I'm sure there are many things I'm probably leaving out.

 

[emp]

Thank you man!!

::emp::

 

[Calamari]

I'll try my best.

That was a bit understated.