>And as someone who uses virtualisation like this extensively: it is about separating functional concerns. I have images of our "standard" web frontends, and our "standard" database backends and their slaves. We can deploy one of those with a few commands to any of our VM hosts (currently using openvz, but looking at both LXC and KVM and combinations). I don't want them co-mingled because it complicates migrations as needs change.
Once you are at scale, each node, doing one thing, should be able to fill a server,(at scale, you will need a lot more than one server for each service you've got, assuming the app has enough scale to support a team of sysadmins.) - the exception I can think of is 'lopsided' services that require, say, more cpu and less ram than your servers were bought with; in which case, yeah, if you can find another service that is lopsided in another way, you are in good shape.
Until then, sure, you need less than a server, at which point some sort of virtualization makes sense. (And if you own and sysadmin all apps, containers could make a lot of sense, as they are rather lighter weight.) And hey, sometimes even if just one app doesn't have the scale to support a team of sysadmins, the whole lot of them do.
If you are saying that virtualization and containers make a lot of sense for small-scale stuff, or any use case where you need servers that are smaller than the physical servers you buy, we are in violent agreement.
>E.g. some customers could easily fit on a single server when we start deploying our apps for them, but may a year down the line require a complicated multi-server setup. When they do, and it's all in containers, we can migrate the web container to another machine (or duplicate it and load balance), and database container somewhere else, and it's all generic tools. We don't need to start messing around and untangle dependencies on a single server.
This makes it sound like you are as much in the 'service provider' realm as I am. That is an interesting way to think of the cluster administrator role; thinking of the different development groups you are running code for as customers, helping them scale up and down. I actually don't have as much experience with that style of cluster administration. At all my cluster administration jobs, the products I was sysadmining were already many-server affairs by the time babysitting them became my job. In those cases, I (and/or the team) pretty much owned the application, with a little bit of help/support from the devs when we couldn't figure something out, which is different from the customer/provider setup.
>And there's the dependency issue. In my experience, I don't trust for a second that you will end up with a working setup if you ship packages and install them on a pre-existing OS install. It takes about 5 seconds from someone other than a very experienced sysadmin gets an ssh login to a server before it deviates from the servers it is supposed to be identical to, at which points all bets are off whether or not your app will run once deployed to your production servers at all. VM's and containers provide the opportunity to easily guarantee that the images you deploy are unchanged from what you tested on.
Sure, sure.... but generally speaking, if you are using whole-server images? you wipe and re-install the whole goddamn thing every time you get half a chance because of this. With PXE this isn't much more difficult than copying over a new tarball. Most large clusters have some software that let you burn-in and re-install thousands of servers by typing one or two lines.
Now, you can argue that copying a new chroot to a thousand hosts is generally easier, or at least faster to recover from if you screw it up than pxe-installing a thousand hosts, say, if you accidentally target the wrong servers.
>Having them neatly isolated in small manageable chunks also means that we are free to change hardware specs, and move containers around depending on what is most cost-effective rather than having to make sure everything fits.
Sure, different hardware needs different drivers, but I think the difficulty of dealing with that is overstated.
The hard part of having different hardware specs isn't the drivers, it's the different performance characteristics, and virtualization doesn't always help you with that. You've got some servers with 15K sas disks and others with 'intelipower' No amount of virtualization is going to make the two servers perform the same. But then, sometime Virtualization does help with that, for instance if the problem is ram to cpu (or ram to disk) ratio differences; if you have one app that requires a lot of ram and disk but little cpu, and another app that requires a lot of cpu and nothing else, if your new boxes with the best cpu happen to have a lot of ram, you could gain benefit from running them both on the same servers.
Once you are at scale, each node, doing one thing, should be able to fill a server,(at scale, you will need a lot more than one server for each service you've got, assuming the app has enough scale to support a team of sysadmins.) - the exception I can think of is 'lopsided' services that require, say, more cpu and less ram than your servers were bought with; in which case, yeah, if you can find another service that is lopsided in another way, you are in good shape.
Until then, sure, you need less than a server, at which point some sort of virtualization makes sense. (And if you own and sysadmin all apps, containers could make a lot of sense, as they are rather lighter weight.) And hey, sometimes even if just one app doesn't have the scale to support a team of sysadmins, the whole lot of them do.
If you are saying that virtualization and containers make a lot of sense for small-scale stuff, or any use case where you need servers that are smaller than the physical servers you buy, we are in violent agreement.
>E.g. some customers could easily fit on a single server when we start deploying our apps for them, but may a year down the line require a complicated multi-server setup. When they do, and it's all in containers, we can migrate the web container to another machine (or duplicate it and load balance), and database container somewhere else, and it's all generic tools. We don't need to start messing around and untangle dependencies on a single server.
This makes it sound like you are as much in the 'service provider' realm as I am. That is an interesting way to think of the cluster administrator role; thinking of the different development groups you are running code for as customers, helping them scale up and down. I actually don't have as much experience with that style of cluster administration. At all my cluster administration jobs, the products I was sysadmining were already many-server affairs by the time babysitting them became my job. In those cases, I (and/or the team) pretty much owned the application, with a little bit of help/support from the devs when we couldn't figure something out, which is different from the customer/provider setup.
>And there's the dependency issue. In my experience, I don't trust for a second that you will end up with a working setup if you ship packages and install them on a pre-existing OS install. It takes about 5 seconds from someone other than a very experienced sysadmin gets an ssh login to a server before it deviates from the servers it is supposed to be identical to, at which points all bets are off whether or not your app will run once deployed to your production servers at all. VM's and containers provide the opportunity to easily guarantee that the images you deploy are unchanged from what you tested on.
Sure, sure.... but generally speaking, if you are using whole-server images? you wipe and re-install the whole goddamn thing every time you get half a chance because of this. With PXE this isn't much more difficult than copying over a new tarball. Most large clusters have some software that let you burn-in and re-install thousands of servers by typing one or two lines.
Now, you can argue that copying a new chroot to a thousand hosts is generally easier, or at least faster to recover from if you screw it up than pxe-installing a thousand hosts, say, if you accidentally target the wrong servers.
>Having them neatly isolated in small manageable chunks also means that we are free to change hardware specs, and move containers around depending on what is most cost-effective rather than having to make sure everything fits.
Sure, different hardware needs different drivers, but I think the difficulty of dealing with that is overstated.
The hard part of having different hardware specs isn't the drivers, it's the different performance characteristics, and virtualization doesn't always help you with that. You've got some servers with 15K sas disks and others with 'intelipower' No amount of virtualization is going to make the two servers perform the same. But then, sometime Virtualization does help with that, for instance if the problem is ram to cpu (or ram to disk) ratio differences; if you have one app that requires a lot of ram and disk but little cpu, and another app that requires a lot of cpu and nothing else, if your new boxes with the best cpu happen to have a lot of ram, you could gain benefit from running them both on the same servers.