Lessons learned while managing technical operations

 for a cloud based SaaS product, and which might be useful to you if you’re doing the same.

• There is no substitute to knowing your fundamentals. Whatever you’re managing - your own datacenter or a suite of apps on a public cloud - you have to know your Operating Systems, your Computer Networking, your Linux, your VMs.
• Know your tools. Find out what tools you need to monitor, maintain and debug your systems. Know how they work, keep up with updates and play with them often. It will save you time when the crisis hits.
• Know one editor and know it well - vim, emacs or other. Know all common shortcuts, complicated copying pasting routines, tips and tricks - in times of crisis, every second counts.
• Learn a little every day. Share what you learn even if you think nobody’s listening. Soon you’ll find like minded people you can share ideas with.
• Visibility to other teams of what you’re doing is very important. Graph it, present it, blog and talk about it.
• Try to fill your team with the right people. The best people in technical operations have an eye for detail who do not lose sight of the big picture. They are good split-second decision makers and are experts in prioritizing in times of crises. And of course, they know their stuff or are smart enough to figure it out if they don’t.
• Know your industry. Study what others are doing, and why.
• Keep up to date. Know what is new in your field - subscribe to the best newsletters, RSS feeds, podcasts and conferences. There is a lot of noise, so take out the time to sift to the useful parts, adopt what is good for your operations and forget the rest.
• Keep an open mind. Fads will come and go, old ideas will be repackaged and sold with a new coating every few years. Whatever be the case, keep up with trends - they always have something to teach.
• Know your organization’s business. Interface and build relationships with all teams. If you cut away the trappings of the DevOps movement, the most important point that remains is collaboration. How you achieve it depends on you.

DevOps Resources

I have been following the DevOps "movement" since its inception. Like any cultural meme that has value it has led to thousands of blogs, podcasts and now books on topics that are related directly and indirectly to it.

I've added a page on my blog linking to some podcasts that I have been following (on and off for some them), which might be of interest to somebody working in Technical Operations, Infrastructure, System administration or managing and architecting a cloud hosted product. It's linked from the top of my blog header - and also from here.

Multiuser SFTP server setup - the solution

I had to setup an SFTP server on an EC2 instance recently, with multiple users chroot-ed into their own directories (with access to only those directories), and a different set of ssh-enabled users, with key based authentication for sftp as well as ssh.

My first instinct was to do a Google search. Many links came up, none of which solved the complete problem. Some of them did not work (different Linux distro/version) and some ended up disabling ssh when I got sftp working.

I finally found this blog post -

http://blog.famzah.net/2011/02/03/secure-chroot-remote-file-access-via-sftp-and-ssh/

It's the only set of instructions that actually worked, with all the constraints mentioned above.
For the record, the OS was Ubuntu 12.04 LTS. An additional step you need to take on this OS is to disable apparmor, or the ssh stops working after a reboot. I am not a Linux wizard, so I don't know yet why this happens.

On a related note, it turns out that a common mistake that many make is confusing FTP over SSL/TLS with SFTP. FTP over SSL is just FTP over a secure connection, while SFTP is a completely different protocol, with the file transfer happening over an ssh connection.