DevOps at Kobas

I’ve been at Kobas two years now, I previously wrote about my experiences in my first month , so it seemed fitting to do an overview of my experiences since then surrounding the software we use for DevOps.

Jenkins & Continuous Delivery

Jenkins Image At Kobas we use continuous delivery rather than continuous deployment, tool of choice; old man Jenkins .

Jenkins, although riddled with UI/UX issues, is a very helpful tool for us day to day now. We have all sorts of pipelines setup for deploying to our EPoS & Cloud servers. Having the ‘one click’ ability to release to a QA or Production environment and letting Jenkins handle all the steps that go on throughout that build process is a time-saver.

Having the ability to rollback to a previous git tag when a bug has been introduced proves to be a real lifesaver. I can’t imagine going back to releasing projects via script or knowledge.

We keep our Jenkins configuration backed up in Git as the thought of losing all the work put into Jenkins and having to start again is nightmare inducing.

Codeception & Jenkins

My latest win, and one that took the longest to achieve was having Codeception and Jenkins play nicely together for automated testing.

The initial part of that went fine, just getting Codeception to run automatically via the Jenkins build process. But then I decided I wanted metrics like code coverage, the ability to run acceptance tests via Selenium & being able to reset our testing database before each test runs.

Selenium is something I’ve played with a lot on my own machine, so setting up Selenium Grid on a server didn’t cause me much trouble.

Code coverage however has been a pain. Needing c3.php to get remote code coverage working required a number of disgusting hacks / workarounds. This was due to the way our project is setup, the main directory Codeception is in, isn’t even synced to our servers.

The results where worth the pain though I now have the ability of viewing a breakdown of code coverage like this:

Code coverage report

(clearly not an image from our Jenkins, imagine a lot more red).

You also get a dashboard for coverage distribution and showing you the files with the most CRAP (change risk anti-patterns), my new favourite acronym. I can’t seem to find an example of that view online however.

Perhaps later I’ll do a blog post on how to set this up on an open source project and will link to it from here. (Making plans for 2020 already)

Puppet & Server Config

Controlling configuration manually might seem okay when you only have a handful of servers, I certainly manage my personal servers manually still.

At Kobas we have two types of servers, our ‘Cloud’ servers (hosted by AWS) and our ‘EPoS’ servers (hosted by our clients on-site). Currently, we have 4-5 AWS servers, and 170+ EPoS servers. Managing configuration change across the EPoS servers would be impossible without Puppet.

Puppet isn’t the only tool that handles configuration management, there are a number of tools to choose between; Chef and Ansible are two that come to mind. I’ve only used Puppet so can’t comment on the advantages of one over the other. I do recommend getting at least one of these tools setup to manage your server configuration though as the pay-off is huge.

Using a configuration management tool ensures that all your configuration is the same across every environment; development, qa and production. Significantly reducing the ‘But it works on my machine’ issue. It also requires you to put additional thought into making configuration changes.


Posted on September 23, 2018

Free SSL Certificates with LetsEncrypt and Ajenti-v

This is a quick post on how to use LetsEncrypt SSL certificates on your Ajenti-v setup .

Ajenti-v will probably be supporting this natively at some point, there is an open issue on their Github here. But in the meantime you can just follow these steps to start using LetsEncrypt now.

You can click this link and head to ‘Getting Started’ or you can just run these commands to install LetsEncrypt in the folder of your choice:

git clone https://github.com/letsencrypt/letsencrypt
cd letsencrypt

So now LetsEncrypt is installed, LetsEncrypt doesn’t yet support nGinx and since that’s what I’m using I’ll assume that’s what you’re using, the automatic function won’t work for us so we will have to use the ‘certonly’ option.

Run the following command and follow the steps.

service nginx stop
./letsencrypt-auto certonly

Now you should get a message like:

Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/yourdomain.com/fullchain.pem.

You can now restart nginx by running:

service nginx start

Now that we have the cert, it’s time to see how we fit this into Ajenti. Open up the websites tab of Ajenti and open whatever website you’ve decided to do this for, hit the SSL tab and you will get this window:

Ajenti Certs Window

So the first box gets filled in with what the wizard returned to us:

/etc/letsencrypt/live/yourdomain.com/fullchain.pem

The second box gets filled in with the certificates private key:

/etc/letsencrypt/live/yourdomain.com/privkey.pem

Now Ajenti knows what certificate to use, it is time to turn SSL on, lets get the Advanced configuration out of the way first:

ajenti-advanced-cert.png

The ‘Custom top level configuration’ you see will automatically redirect users accessing your website on http:// to the https:// domain.

Lastly we just need to change the website port from the default 80 to SSL’s 443:

ajenti-ssl-ports.png

Apply changes, and check to make sure your website is still redirecting.


Posted on March 21, 2016

The Perfect Web Server - Nginx, Ajenti, Ubuntu

ajenti-dashboard.png

I’ve done a lot of installing of web servers over the last while, some of which have been effortless, others a thorn in my side. I’ve decided to compile a guide for my latest server setup that I’ve fallen in love with;

First a breakdown of what we’ll be installing today;

Nginx : (pronounced Engine X) is a free, open-source, high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server. Nginx doesn’t rely on threads to handle requests. Instead it uses a much more scalable event-driven (asynchronous) architecture. This architecture uses small, but more importantly,predictable amounts of memory under load. Even if you don’t expect to handle thousands of simultaneous requests, you can still benefit from Nginx’s high-performance and small memory footprint. Nginx scales in all directions: from the smallest VPS all the way up to clusters of servers.

Ajenti : “The admin panel your servers deserve.” Easily extensible using Python. Plugin development is fast and pleasant with rich APIs. Includes lots of plugins for system and software configuration, monitoring and management.

Ajenti V : A plugin for Ajenti that makes website setup easy – including app servers, database, and routing.

Ubuntu : If you don’t know what Ubuntu is we’re in trouble.

Now I’m going to take a leap of faith and assume you can either install Ubuntu yourself or figure out how to get a server with it already. A DigitalOcean droplet works perfectly here. So lets log in as root and run all this:

#Insall Ajenti
apt-get update
wget http://repo.ajenti.org/debian/key -O- | apt-key add -
echo "deb http://repo.ajenti.org/ng/debian main main ubuntu" >> /etc/apt/sources.list
apt-get update
apt-get install ajenti
service ajenti restart

# Uninstall Apache2
sudo apt-get autoremove && sudo apt-get remove apache2*

# Install Ajenti-v
apt-get install ajenti-v ajenti-v-nginx ajenti-v-mysql ajenti-v-php-fpm php5-mysql 

# If you <3 Ruby
apt-get install ajenti-v-ruby-unicorn ajenti-v-ruby-puma

# If you need Python
apt-get install ajenti-v-python-gunicorn

# If you need nodeJS
apt-get install ajenti-v-nodejs

# If you want FTP
apt-get install ajenti-v-ftp-pureftpd

# If you want mail
apt-get install ajenti-v-mail

# If you want POP support (for gmail etc.)
apt-get install courier-pop

# Restart All Services
sudo service php5-fpm restart
sudo service nginx restart
sudo service ajenti restart

You should now be able to log in to your Ajenti control panel at https://yourserver.com:8000 with:

username: root
password: admin

Now that’s done you’ll notice if you open /etc/nginx/nginx.conf that files inside /etc/nginx/conf.d/ are loaded before any other .conf files, this is where you should put any additional configuration for Nginx. However if you are just configuring a specific domain or website you should just put the configuration in the Ajenti website configuration’s advanced section.


Posted on February 18, 2015