Posts

Showing posts from 2014

DevOpsDays Berlin 2014

Image
Update: Read my (German) conference report on heise developer.

Last week I was at the DevOps Days Berlin 2014. This time at the Kalkscheune, a much better location than the Urania from last year. With 250 people the conference was not too full and the location was also well equipped to handle this amount.

Proving DevOps to be more about people and culture, most talks where not so technical but emphasized the need to take along all the people on the journey to DevOps.

An technical bonus was the talk by Simon Eskildsen about "Docker at Shopify" which was the first time that I heard about a successful Docker implementation in production.

Always good to know is the difference between effective and efficient as explained by Alex Schwartz in "DevOps means effectiveness first". DevOps is actually a way to optimize for effectiveness before optimizing for efficience.

Microsoft and SAP gave talks about DevOps in their world - quite impressive to see DevOps beeing main stream…

EuroPython 2014

Image
One full week of Python power is almost more than one can take, but I missing it would be even worse.

This was my first EuroPython and with 1200 participants a big upgrade compared to the previous 2 PyCon.DE events in which I participated. The location (Berlin Congress Center) deserves kudos, along with the perfect organization.

The Wifi worked really well (except for a WAN problem on Tuesday which was fixed quickly) and everybody loved the catering. They even had kosher, helal and vegan food (preordered), which is highly unusual for German conferences. Most amazing was the video crew who managed to upload all videos in about one hour after a talk was given.

I managed to give three talks:

DevOps Risk MitigationHow we use Test Driven Infrastructure at ImmobilienScout24 as part of our general automation to reduce the risk of giving everybody access everywhere. (Access Slides or Watch Video)
YAML ReaderLightning Talk about the yamlreader Python library, which provides a wrapper for the ya…

iPXE - The Versatile Boot Loader

Image
iPXE is a lesser known Open Source PXE boot loader which offers many interesting features:
boot from a web server via HTTP and HTTPSboot from remote block device over iSCSI, FC, FCoE, AoE SANboot from VLAN, WiFi, WAN, Infinibandcontrol the boot process with a scriptuser interaction with menus and login promptsdisplay images and splash screensreplaces NIC firmware or chain load via PXE Talk & Article Since iPXE plays a role in the ImmobilienScout24 boot automation I gave a talk about it at the LinuxTag 2014. The talk is half an hour long and gives a quick introduction into iPXE. It covers build, configuration & scripting and shows how to develop boot scripts in iPXE with a very short feedback cycle.



Download the slides to the talk and the audio recording as a podcast.

At the conference the German Linux Magazin became interested in the topic and asked me to write an article about iPXE:

Der vielseitige Netzwerk-Bootloader I-PXE
Linux Magazin 08/2014

Demo Scripts For the article I cr…

automirror - Automate Linux Screen Mirroring

Image
I do a lot of pair working and many times I connect a large TV or projector to my laptop for others to see what I am doing.

Unfortunately the display resolution of my laptop never matches that of the other display, and Linux tends to choose 1024x768 as the highest compatible resolution. This is of course totally useless for doing any real work.

My preferred solution for this problem is to use X scaling to bridge the resolution gap between the different screens.

Since none of the regular display configuration tools support scaling, I ended up typing this line very often:

xrandr --output LVDS1 --mode 1600x900 --output HDMI3 --mode 1920x1080 --scale-from 1600x900
Eventually I got fed up and decided to automate the process, the result is automirror, a little Bash script that automatically configures all attached displays in a mirror configuration. automirror is available on https://github.com/schlomo/automirror.
Typical Use Cases Connecting a Full HD 1920x1080 display via HDMI to my 1600x…

Granting root access in a DevOps world

At the 2014-06 Berlin DevOps Meetup this week we had an interesting fish bowl discussion about

What is the risk of giving DEVs root access in production?

Since I suggested the topic I was asked to give a short introduction into the topic:


The discussion that followed was suprising in several aspects:
A major concern is safeguarding the production data, but nobody had a really good solution for that. Many people have more problems with Developers seeing live customer data than with Develops changing something in production."Nobody should have root" was proposed by a security specialist, but he had no practical working example for this approach.The question is tightly coupled to the degree of automation. The more automation you have the less need for anybody (Dev or Ops) to use their root privileges.Not everybody having root access knows what to do with it, Developers are sometimes afraid of using their power if granted root.This is mostly a question for larger companies and cla…

My SMART TV - Linux For The Win

Image
I love my "smart" TV - it got Linux inside which is the base for a whole range of nice hacks.
TV Router The most important one is that the TV is actually a wireless router that provides Internet via Ethernet to my TV rack. Usually the Ethernet connection is used by the Playstation or a Raspberry Pi.
The original reason for this hack was simple: The Playstation 3 has a really really bad Wifi reception which made watching Netflix nearly impossible and the unavoidable PS3 updates painfully long. The USB Wifi adapter connected to the TV has a much better reception, sharing it with the PS3 solved all the performance problems.
Samsung Linux TV And here comes the good part. The TV (Samsung LE32C650) runs Linux inside and there is an Open Source project (SamyGO) that "opens up" the TV firmware and extends this Linux with useful tools.

In my case I only had to enable IP forwarding, configure a static IP on the Ethernet interface (eth0) and start a DHCP server on it. The Sam…

Win-Win: Employer Branding and Corporate Social Responsibility

Image
Does your company care about employer branding? Probably yes.

Does your company care about corporate social responsibility? Probably yes.

Does your company combine these two to create a win-win situation? Most likely not!

Take my employer ImmobilienScout24 as a typical example: The about us page mentiones that ImmobilienScout24 is a great place to work (4th in our region) and the CSR team talks about the social engagement, e.g. blood donations or the social day where all employees donate their work time to non-profit organizations.

However, there is no obvious connection between these two things.

I would like to suggest a simple way how to combine both employer branding and corporate social responsibility:

A company should make it a priority to support charitable organizations and social projects related to their own employees.

Examples:
Sponsor non-profit organizations or neighborhood/community projects that employees are involved with.On social day, go to schools and kindergartens wh…

Adding Custom Menus for Linux Desktops

Image
The "Start Menu" of a Linux Desktop usually comes with a predefined set of categories that make up the sub menus. If you have a lot of custom applications then you might want to group them under a dedicated sub menu instead of having them spread out over all the menu categories.

Adding sub menus and new categories on Linux Desktops is defined in the Desktop Menu Specification in Appendix C. It turns out that it is really simple and the following example from ImmobilienScout24 can serve as a base for your own custom menu.

You will need the following parts:
A Desktop file using a custom categoryA Directory file defining the icon and description for the new sub menuThe icon for the sub menuAn XML file describing how to integrate the new sub menu into the menu structure and which categories of Desktop files to show in the new menu The Desktop file describes the menu entry, in this example the VPN client:
The important part here is the Categories entry which specifies a generic …

Simple Video Presentation with Raspberry Pi

Image
Playing videos in an endless loop is a common problem:
Product demos at a trade show or fairInfomercials in a public place or foyerBackground fun at a party... When I faced this problem at the last LinuxTag we did not want to take a full blown computer with us but make do with a Raspberry Pi. The question was how to turn the Pi into a simple video player with a minimum amount of fuss.
The solution is simple and elegant: Install OpenELEC (an Kodi distribution) on a SD cardBoot it up once in the Pi to initialize the storage partitionAdd the following file in the storage partition as .kodi/userdata/autoexec.py
Add any amount of multimedia files in the storage partition under videos/Boot up the Pi and enjoy your videos You can also interrupt the playback and use OpenELEC normally. To go back to the automatic playback simply reboot the system.

And here is our booth with the demo videos in front:

Update 2016-05-13: Adjust for Kodi instead of XBMC. Everything else works as before.

Simple file patching with sed

Patching configuration files is like the bread-and-butter job of every configuration management. In our package-based deployment world we try to minimize the patching to the absolute minimum, usually to "enable" modularized configuration patterns.

The best example is the Apache Webserver, where we have a wrapper RPM package with a %post script that simply replaces (and not patches) the upstream configuration with a few include lines:

Sadly there is still a lot of software that does not support includes in its configuration. For these we of course have to patch the existing configuration and use this short and simple config patcher in our RPM %post scripts, for example like this for sshd_config:

The trick of this snippet is that in the end the changed parts are always at the top of the file. It is also important to always embed some information about the cause of the patch so that one can easily find out who or what is reponsible for the file. The %-variables are filled in …

Automated OpenSSH Configuration Tests

Image
When developing or fine-tuning OpenSSH configurations the testing can be quite tiresome: Change configuration, restart server, run manual tests, repeat. Not to forget the many times when restarting the SSH server does not work and you lock yourself out of your test server.

When writing a Linux Magazin article about SSH key management I wanted to show how to use OpenSSH PKI in a repeatable way. The result is an automated test suite for OpenSSH configuration: $ ./run_demo.sh   ... lots of info output running through ...SSH PKI Demo Test Results:

Succeeded create-ca-key
Succeeded create-host-key
Succeeded sign-host-key
Succeeded create-user-root-key
Succeeded sign-user-root-key
Succeeded create-user-unpriv-key
Succeeded sign-user-unpriv-key
Succeeded test-trusting-known-hosts-via-cert-and-login-with-password
Succeeded test-that-hostname-in-cert-must-match-target-host
Succeeded test-login-with-root-key-trusted-by-cert
Succeeded test-that-username-in-cert-must-match-target-user
Succeeded test-revoked-…

Opening a Window to a Wider World

When I bought a new Chromebook Acer C720 last week I got confirmation that times are changing: It has only an HDMI connector, no more VGA. Luckily, at ImmobilienScout24 we are also adapting and last month our big projector got an upgrade to Full HD with 16:9 Wide Screen. And you can now connect the computer through HDMI, too.

Since me myself so much got used to creating presentations in 4:3 I took the opportunity to remind myself and everybody else why it really pays to pay attention to this little detail.

Video is in German with English subtitles.

SSH with Personal Environment

Image
A colleague, Eric Grehm, raised an interesting challenge:

How to maintain his personal work environment (VIM settings, .bashrc ...) on all servers?

The first thought was putting this somehow into our software distribution, but we quickly realized that this would trigger needless updates on hundreds of servers. The benefit would be that the personal work environment is already on every server upon first access.

The next idea is to switch from a pre-installed personal environment to an on-demand solution where the personal environment is transferred each time a remote connection (over SSH) is established.

A simple implementation would just to a scp before the ssh, but that entails two connections which takes more time and might also bother the user with a double password request.

Side-channel data transfer An alternative is to piggyback the file transfer onto the regular SSH connection so that the personal environment is transferred in a side channel:
On the client create a directory wi…

Rough Measurement for HTTP Client Download Speed

Image
Ever wonder if your website is slow because of the server or because of the clients?
Do you want to know how fast is your clients' connection to the Internet?
Don't want to use external tracking services, injecting JavaScript etc.?

Why not simply measure how long it takes to deliver the content from your webserver to your users? Apache and nginx both support logging the total processing time of a request with a suitably high precision. That gives the time from starting with first byte received from the client and ending after the last byte sent to the client.

To try out this idea I added %D to the log format for access.log of my Apache server and wrote a little Python script to calculate the transfer speeds. With the help of the apachelog Python module parsing the Apache access.log is really simple. This module takes a log format definition as configuration and automatically breaks down a log line into the corresponding values.

The script can be found together with a little …

apt-install

Image
Do you ever get tired of typing
sudo apt-get update && apt-get install <package> just to install one package that you added to your DEB repo? I do and I decided to do something about it. What I really miss is the intelligence of yum which simply updates its repo caches if they are too old.

apt-install (github.com/schlomo/apt-install) is the next best thing. It is a simply Python script that updates the cache and installs the packages given as command line arguments. And it shows a nice GUI with a progress bar:


Turns out that the parts are all there and part of aptdaemon. The only part missing was putting them together into this little script:
Please note that I actually completely don't understand how to write async code. I'll be happy about all feedback with better implementations.

Simple Video Tricks

Image
While working on the new Recorder (see also last posting) I suddenly faced several challanges with the resulting video files:
Many short chunks (50MB each, about 30-60 seconds) need to be mergedExtract the actual talk from a longer recording, e.g. the recorder was on for one hour but the talk was only half an hourConvert the video into another container format because Adobe Premiere does not like AVI filesCreate video thumbnailsConvert videos to be compatible with HTML5 <video> playback Turns out that avconv (or ffmpeg) is the swiss army knife for all of these tasks! I am not qualified to say which is better, for my purposes the avconv that ships with Ubuntu is good enough. The examples given here work with both. When I write avconv I mean both tools.
Since I don't want to degrade the video quality through repeated decode/encode steps I always use -codec copy after the input file to simply copy over the audio and video data without reencoding it.
Concatenate Videos This is p…

Hostname-based Access Control for Dynamic IPs

Image
Sometimes less is more. The most simple way to protect my private web space on my web server is this:

<Location />
    Order Deny,Allow
    Deny from All
    Allow from home.schapiro.org
</Location>

But what to do if home.schapiro.org changes the IP every 24 hours and if the reverse DNS entry (PTR) is something like p5DAE56B9.dip0.t-ipconnect.de? When my computer at home connects to the web server the source IP address is used for a reverse DNS lookup. This lookup returns the above mentioned provider-assigned name and not home.schapiro.org,  the web server will never be able to identify this IP as belonging to my home router.

The solution is to write the IP↔Name mapping for my dynamic IPs into /etc/hosts. That way a reverse lookup on the IP will actually yield the information from /etc/hosts and not ask the DNS system.

Since I don't want to do this manually every time my IP changes, I automate it with this script. It reads host names from /etc/hosts.autoupdate and injects …

Simple UDP Stream Recorder

Image
At the office I got a 3 channel digital Audio/Video Recorder to conveniently record our talks without much human effort. The device has an analog video input for the video camera (standard resolution) and a digital video input (Full HD) and an audio input.
These 3 inputs will be merged into a single side-by-side video where you can see the speaker next to his computer output. The video can be even larger than Full HD, for example 2688x1200 (a 768 pixels wide SD image next to a 1920 pixels wide HD image):

The device is far from cheap (list price is 1840 € + VAT) and can really do a lot. For example, it can create H.264 movies with a bitrate of up to 9 Mbit. It can also upload the videos to a CIFS share, but sadly that works only at a transfer speed of about 4 Mbit! So how could I transfer the videos at really high quality settings (9 Mbit) to the CIFS share? Waiting 2 hours to transfer the videos of a 1 hour talk is no option.


Linux and Open Source to the rescue!
My solution is a simpl…