Capas de automatización

Este último año y medio he estado trabajando intensamente en cuestiones de automatización de pipelines e infraestructura en general, usando principalmente Jenkins, Puppet y herramientas afines.

En particular, estos últimos meses he trabajado en la automatización de despliegues de un producto software bastante complejo. La complejidad en este caso está dada por la gran cantidad de componentes de código e infraestructura involucrados: varias bases de datos relacionales, bus de servicios, sistema de colas, servidor de aplicaciones, servidor web, dispositivos móviles, etc. Al mismo tiempo el producto tiene un requerimiento de correr sobre distintos sabores de infraestructura (Oracle, IBM, OpenSource), lo cual requiere que cada ambiente (desarrollo, test, etc) tenga también varios sabores.

Para la generación de estos ambientes y el deploy de la solución a cada uno de ellos diseñamos una arquitectura de automatización/deploy de 3 capas la cual describo brevemente a continuación.

Capa 1: Sistema operativo

El primer paso es levantar los servidores con el sistema operativo correspondiente (para el caso mencionado CentOS y Oracle Linux) y realizar algunos ajustes de base sobre el mismo (configuración de red, usuarios, etc). En general esto se hace utilizando la API de la plataforma de virtualización y algunos scripts de base. Puntualmente en el mencionado caso usamos el Hypervisor de VMWare y Puppet.

Capa 2: Software de base

Ya con el sistema operativo instalado y servidor 100% funcional hay que instalar software de base: motor de base de datos, el service bus, el web server, etc. Cada uno de estos componentes hay que instalarlos y configurarlos acorde a seteos específicos de la solución. En el caso mencionado esto lo hicimos usando scripts Ansible que se disparan desde RunDeck

Capa 3: Solución

Lo último que resta es instalar la solución en cuestión, lo cual implica crear los schemas de base datos, desplegar las aplicaciones en los servidores/contenedores web, desplegar los flujos en el bus de servicios, etc. En el caso en cuestión esto lo tenemos implementado con simples bash scripts que lanzamos desde Jenkins.

La implementación de esta solución en este caso puntual requirió del trabajo de distintas áreas de la organización: gente de infraestructura (capa 1), gente de operaciones (capa2), gente del grupo de producto (capa3) y obviamente hubo también un cierto esfuerzo de coordinación de todos ellos.

Hay algunas cuestiones de la implementación de este caso que aún hoy no me convencen (como usar Puppet Y Ansible al mismo tiempo, hubiera preferido usar sólo 1 de ellos) y que son consecuencia de decisiones tomadas por los distintas áreas de la organización. Más allá de las herramientas particulares, creo que el esquema general de esta solución es replicable en otros contextos usando incluso otras herramientas, es por ello que decidí compartir estas líneas.

capas_automatizacion

Centralized Log Services: LogEntries and Papertrail

In the cloud era there are several companies offering centralized log services. In general terms these services consist in 2 components:

  1. A client component that takes care of collecting log information and sending it to centralized storage.
  2. A web application that allows you to browse the centralized storage

One company that offers this kind of service is Logentries. I used it for a couple of Ruby applications, some of them hosted on Heroku and others hosted on my own server. In the case of Heroku apps, the integration with LogEntries is transparent (you don’t need to modify your app code), you just have to add logentries plugin to your Heroku app , configure it and that’s all, all log information is sent to LogEntries. For applications running outside Heroku, you can use logentries gem to your app, and just make your app to use LogEntries logger.

Other service I have used is Papertrail. I used it for Ruby and .Net applications hosted on my own servers and in each case I have used a different integration strategy. For .Net apps I like to use Log4Net library which natively integrates with Papertrail (details can be found here). In the case of Ruby apps, I have used a different strategy. I install on my server a client application that takes care of reading local log files and push them to the Papertrial website. This strategy is totally transparent for the application and is application agnostic: no matter what technology your app is built on.

lgentries

From Heroku to Rackspace

It is the third time I am involved in the moving an application from Heroku to a cloud server. The time the server is running Ubuntu and it is hosted on Rackspace. It was not my decision, I was contacted once the the decision was already made, but in this particular case I think it is a good idea.

The requirement was simple: move the application to a Cloud Server running Ubuntu 14. The application should continue using the same infrastructure services it was using on Heroku, that is: Puma, PostgreSQL and Memcached.

Just like the previous cases I was involved, the setup of the infrastructure is really easy, you can do almost everything with apt-get command.

The “not-so-easy” part is when you have to replace some of the services provided by Heroku. No matter how simple your application is, for sure you will have to deal with application deployment and database backups.

In the next days I will post detailed articles to explain how to implemented this concerns.

Primeros pasos con Cucumber-JVM

Hace poco más de dos meses estoy trabajando casi cotidianamente con Cucumber-JVM y como consecuencia de ello he aprendido muchísimo sobre dicha herramienta. Si bien ya tenía bastante experiencia con la implementación Cucumber-Ruby, resulta que la implementación Java tiene sus particularidades.

Para compartir lo que he ido aprendiendo he decidido grabar una serie de videos. He aquí el primero de ellos: Introducción a Cucumber-JVM, espero les resulte útil.

 

Configuration Management in my .NET project

On these days I am working on big system that is built on several components: a couple of websites, some backend services and several shared libraries. At this moment all these components are stored in the same Subversion repository. As you can image, this is a huge repository. At the same time, the system is already running in production mode and some components are being updated/replaced. Because of this situation and in order to simplify configuration management I being working on designing a new configuration management strategy. The solution I have in mind includes 3 key tools:

  • Git: to store source code, in particular I like GitHub service
  • NuGet: a package manager for .NET components (for those not familiar with .NET, it is like a Maven in Java o npm in Node). We know we will need a private Gallery for some components, but we still don’t decide if  we are going to host our own instance or if we will use a cloud service.
  • Jenkins: to provide continuous integration and deployment automation

Now I will try to explain how these components fit together.

Each shared library has its own Git repository. In case a library depends on third party component, it should declare the dependency package.xml for Nuget to download the dependency at build time. At the same time, stable versions of share libraries are published in NuGet Gallery (possible a private one).

Each application has its own Git repository. In case an application depends on third party component, it should declare the dependency in package.xml for Nuget to download the dependency at build time. In case this application depends on a  private share library there two options:

  • If the shared library will be used as it is, then it is managed with NuGet
  • If the shared library needs to be modified, then the shared library repository is added as a submodule of the application repository. This way you will be able to add the shared library project to your Visual Studio solution and will be able to modify it.

At the same time, Jenkins will monitor all repositories and trigger a CI job to compile and run automated tests on each change. For each application there will be additional Jenkins Jobs to perform deployments to the different environments. For each shared library, there will be a Jenkins job to publish the library to the NuGet repository.

scm

Coursera: Curso de Android, Semana #1

Acabo de completar la primer semana de este curso. Me llevó unas 4 horas de estudio entre videos, cuestionarios, tareas de programación y configuración/troubleshooting de las herramientas de desarrollo.

Por ahora viene bastante tranquilo, lo cual es lógico para ser la primer semana, pero pinta realmente muy interesante.

Si quieren aprender Android, creo que aún estan a tiempo de sumarse al curso.

Continuará…

 

Javascript tests running on Jenkins

Nowadays, no matter what technology you use to build your web application, it is almost sure you will need to write some JavaScript code. At the same Javascript script code is not only used to animate the web pages, it is also used to handle validations and application flow. Because of this, everyday is more needed to write unit tests for the Javascript code.

There are several unit testing frameworks for Javascript. In my case I choose Qunit, that is testing framework developed by the jQuery guys.

Of course that in order to be able to write unit tests for your code, you will need to follow some design guidelines, but that is part of another post. Let’s suppose you followed that guidelines and now you want to write some test, these are the steps you should follow to run your tests:

  1. Download qunit.js
  2. Download qunit.css
  3. Write your tests in a javascript file
  4. Create a html page referencing the 3 previous files

With these 4 steps you are almost done, open the html file and you will have your tests executed.

What is missing, is how to run these tests in the build server. The interesting point here is that to run Javascript t tests we need a Javascript engine. In a develop machine, it is not a problem, you can use your browser, but in the build server is not so easy. The approach I took was to use PhantomJS, a tool that among other things, can run Javascript without needing a browser.

So using PhantomJS and MSbuild I was able to have my Jenkins running my Javascript tests.

Here you can download a running example.