Code Sprint November 2013 Report

By: Steffen, Bastian and Michael on behalf of the Server team

On the weekend of November 8-10, the TYPO3 Server Admin Team met in Hamburg for discussion, hacking and socializing.

Thanks to the location and coffee sponsor e-net Consulting we had the opportunity to work concentrated on many topics that have been on our agenda in the last months.

While a lot of tiny things have been done and discussed, we want to summarize only the biggest topics in the following article.

Participants of the meeting were the Server Admin Team members Bastian Bringenberg, Steffen Gebert, Peter Niederlag, Michael Stucki and Christian Trabold and our guest Andri Steiner, a well experienced sysadmin from snowflake. Also we were happy that good old friends and e-net employees Marcus Krause and Christian Kuhn joined our table for their hacking sessions. 

Chef repository refactoring

We are using Opscode Chef to managing our server infrastructure. While Chef is a great product, it does not dictate any workflow to us infrastructure developers. Furthermore, the eco-system around Chef is thriving and new tools and ideas come up frequently.

The need for well-defined workflows and a kind of "standarization" of all the building blocks (roles, cookbooks, wrapper cookbooks, role cookbooks, ..?) is also bothering us. One of the major problems is Chef's current implementation of roles, which lack the ability to be versioned like cookbooks (we're happy to read that during last weeks's Chef Community Summit this was voted as one of the top features of the next version Chef 12). Another hot topic to discuss is how to proceed with community cookbooks that have to be slightly modified for our needs (wrapper cookbook or forking in Git).

Also we've been discussing kind of an "intelligent base role" that would not require us to add several (e.g. data center or operating system specific) roles to the run list, but automatically inheriting everything based on automatically derived Ohai data.

We will summarize our decisions and ideas and present them in a dedicated blog post within the coming weeks, as soon as these refeactorings have happend.

IPv6

Andri Steiner brought up this hot and important topic. Due to the distributed locations of our (100% sponsored) server infrastructure, our connectivity varies from location to location. Currently, 4 of our 5 data centers already provide IPv6 connectivity, the last one following in January.

Motivated by this good news, we added IPv6 addresses to several of more VMs. The most important site that's still missing is typo3.org itself, which will likely follow around February.

As an example you can now connect to http://get.typo3.org/ using IPv6:

$ curl --head --ipv6 http://get.typo3.org/
HTTP/1.1 200 OK

Jenkins Migration

The current Jenkins server running on ci.typo3.org is sponsored and administrated by dkd. However, as we want to make more use of Jenkins in our infrastructure (and would like to see also more use in the TYPO3 project in general), we have received a new server which is kindly sponsored by dkd. We have continued to set up this machine and make first use of it.

Using the help of the TYPO3 QA Team (in fact that's Stefano Kowalke and Andy Grunwald), we want to move the build jobs to the new server very soon. We also want to have more jobs running on our Chef repo for helping us with infrastructure in the future...

Additionally, a problem was fixed related to calculating Sonar's code metrics, as shown on metrics.typo3.org.

An own APT Repository

We are happy Debian users and sometimes require custom `.deb` packages to be built and distributed. Also tiny tools like MySQL tuning primer could be distributed a lot cleaner through an APT repository instead by putting such scripts on our servers through Chef.

Therefore we started building our own repository, which is meant to be used for our own infrastructure. (We also might use this setup to provide packages for TYPO3 products if there will be demand for this by the teams...) 

The setup we are building right now will run all packaging on the Jenkins server, which will build and publish packages out of a Git repository. However, this is still a work in progress...

Archiving old Web Sites

Although we had this problem in mind, we avoided any actions until now, as sites like T3DD08 or T3CON08 (and newer) were not running on our infrastructure, and in fact have been unmaintaned since years. During the sprint, we've built static archives of these sites using httrack and put them on our infrastructure. This way we keep the information online, but avoid the (high) risk of having a *.typo3.org web site being hacked. We thank Marcus Krause from the Security Team and his t3census project for the kind reminder.

For the future, we try to establish a process so that sites from past-year's events will always be archived. 

Forge Update and Git for Everyone

While the preparation for the upgrade of our Forge server is almost finished, some tiny issues are still blocking the new deployment of Forge and the dependent SVN server. We still plan to have the update done this year (really!).

Logging Server

We are collecting log files from all our servers on a central system using rsyslog. We're currently building a server using Graylog2 which will provide a nice web interface for intelligent filtering of log entries (backed by ElasticSearch). A lot of work was done here, however the system isn't ready for prime time yet...

New Tiny Helpers

Monit is a tool for monitoring and supervising processes on the host where it's running. We started with low-hanging-fruits like the postfix daemon, which will now be restarted automatically as soon as Monit detects that it's not running. In contrast to our Zabbix monitoring, which is only used for alerting and statistics, this can avoid human interaction for simplistic manual actions (mostly restarts).

Etckeeper is another nice tool that keeps track of changes in /etc in a local Git repository. This way we will auto-commit changes made by Chef using our Etckeeper Report Handler. Changes introduced by Chef and also manual changes (that should not happen, but are required from time to time for hot-fixes or debugging) can be tracked and reverted this way. 

Having a configuration management solution like Chef in place is a huge benefit for us. Rolling out such tiny helpers to all (or just a group of) servers is really comfortable and useful. 

Summary

We are very happy with the things that we achieved during the meeting. Sadly, some (for us) urgent topics (esp. the Forge update) could not be finished like we would have planned. We preferred to use the opportunity with almost all members in one room for discussing our workflow, which was also really needed and healthy.

Still, we're progressing well on all projects, and we have catched up with a lot of new motivation to work on these things.

Closing words

We'd like to thank Andri Steiner, who was our guest, for being with us and sharing his expertise. We look forward to invite you (and more guests) again to our coming meetings next year...

Thanks to the TYPO3 Association who is covering all travel costs, as well as food and beverages.

Finally, we want to thank the master of events Volker Graubaum, not only for the great chocolate cake (thanks to your wife!), but also for the superb guidance to a delicious restaurant and Hamburg's nightlife. Thank you!

TYPO3 Server Team
Bastian, Christian, Peter, Steffen & Michael

---