Posts

Showing posts from 2013

Automated Raspbian Setup for Raspberry Pi

Image
Update (2019):  https://github.com/schlomo/rpi-image-creator  is the new home of the code. Recently we got a whole bunch of Raspberry Pi systems at work - the cheapest platform for building Dashboards . Everybody loves those little cute boxes - but nobody wants to deal with the setup and maintenance. The kiosk-browser package is of course also the base of our Pi-based setup. But how does it get onto the Pi? The solution is a Bash script: rpi-image-creator  available on the ImmobilienScout24 GitHub project . It automates the setup of a Pi with Raspbian by downloading a Raspbian image, customizing it a bit and writing it to a SD card. The reason to write my own script where the following features: It creates the file systems on the SD card instead of dumping a raw image onto the SD card. That way the partitions are all aligned properly and have the right size from the beginning. No later resizing needed It removes all the stuff that Raspbian runs at the first boot so that the

RSH Pitfall: Remote Exit Code

Image
While writing some test scripts that use rsh (see below about why) to run commands on a remote test server I noticed that rsh and ssh have a significant difference: ssh reports the remote exit code and rsh does not .  As a result all my tests did not test anything, the error condition was never triggered: My solution is this rsh wrapper: The reason for using rsh instead of ssh is very simple: In a fully automated environment it provides the same level of security as ssh without the added trouble of maintaining it: I need to make sure that ssh continues to work after all the SSH host keys change (e.g. after I reinstall the target). Also, to allow unattended operation I must deploy the SSH private keys so that in the end others could also extract them and use them. In the end I would be using IP/hostname restrictions on the SSH server side to restrict the use of the private key. With rsh I don't need to worry about deploying or maintaining keys and just configure th

Test Driven Infrastructure

Image
Yesterday I was at the Berlin DevOps meetup and we had a very nice fishbowl about Test Driven Infrastructure (TDI). I used my Lightning Talk from the PyCon in Köln as an introduction to the topic, but quickly realized that the term does not fully explain itself. Test Driven Development  in itself is not a new thing, maybe it is not yet common to apply it to platform operations. As an old Ops guy I had a lot to learn when I started to work at ImmobilienScout24 , which is a real software development company. The bottom line is really simple: Untested = Broken My idea of TDI is to apply most of the basic ideas of TDD also to the development process  of the software that runs our platform. Again, the same thing as the developers do with their code already for a long time. Let's just say that we start to test all  code that goes on a server, no matter who wrote it or what it actually does. Some specific examples that we did in the last month: A service that mounts SAN

Magic ISO Image Booting with GNU GRUB 2

Image
Recently I needed to prepare a USB thumb drive with several Ubuntu installations. A little research quickly yielded many setup instructions, for example like the one from Pendrivesystem.com .  I was really surprised at how well this works and wanted to understand it better. In essence all recipes rely on GNU GRUB in version 2 and the loopback feature that it contains and on the OS's ability to work off an ISO image. The loopback command  mounts a CD or HDD image that contains the kernel and initrd from an ISO image. As a result one can put several ISO images on the boot media without the need to extract them. The OS then also mounts the ISO image and uses that instead of a CD/DVD drive. So here is my version of the recipe, the USB thumb drive is in /dev/sdc  in my examples: 1. partition & format device I prefer to partition the device with parted because it aligns the partition at 1MB so that it leaves enough space for GRUB to embed itself into the first sectors of

Setting hostname from DHCP in Debian

Image
For our team monitors that use the Kiosk Browser on a Raspberry Pi I am building a Raspbian wheezy image. One requirement is that the system will configure the hostname from DHCP so that we can use the same image for all devices. Sadly, setting the hostname from DHCP is not  trivial in Debian, here is the result of my research into the topic. I found 2 things to be essential and learned both of them by analysing the dhclient script. 1. Set the hostname to localhost The first thing to do is to set the system hostname to localhost:     # echo localhost >/etc/hostname 2. Workaround for broken dhclient-script The dhclient-script has (IMHO) a bug: If there is an old DHCP lease with a hostname in the lease database (e.g. in  /var/lib/dhcp/dhclient.eth0.leases ), then the script will not  set the hostname from DHCP even if the system name is still localhost. To circumvent this bug simply create a dhclient enter hook to unset the old host name variable:     #

Idea: Electric Family Van

Image
Today I was walking a bit on the way home. After the third silent taxi (Toyota Prius Hybrid) passed by I started to think about what would be the ideal electric car for my use case. We are a large household and won't do with the standard 4 or 5 seat cars that are now beeing offered as electric or hybrid cars. Since the car makers don't offer what I need I am posting my idea here. This is an approximation of our current family car: It is nice, fairly large, seats 8 people comfortably and even has some extra luggage space in the back. We use it for short trips in the city to go shopping, drive the kids around and do the occasional family sunday trip to the surrounding country side. When we go travelling, we usually need a larger trunk so we have this add-on trunk: This is a trailer which is about half as long as the car and of the same height and width. It fits everything we need even for a long vacation. Or all our bikes for a shorter one. Or the insanely large

Always good for a surprise: PyConDE 2013

Image
I was again at the  PyConDE , this year in Cologne. As before, the conference was a mix between different types of talks. It seems like this year we had less people attending compared to last year, at least the crowd in the main hall looked much smaller (see photo). Many talsk where really interesting, but the lighning talks where the real highlight with lots of funny, useful or astonishing talks.b The conference included a beginners programming competition for school students (13-21), the 2 winners showed their project in the opening keynotes. The project had to do something with Blender and Python and one of the winners (a 13-year old boy!) presented a generated animation of a Skat game with a very solid software design. All that after just 9 months of learning Python was really stunning. Andreas Schreiber  most certainly gave up a lot of personal information in his very practical talk about the " Internet of Things ", in which he showed how to connect vario

Video Converter

Image
Modern video cameras produce beautiful videos - but the resulting files take up a lot of space. For an internal video portal I had to automatically convert such camera videos (.m2ts or .mov at up to 30MBit/s) to "regular" mp4 videos that can be used with web-bases video players like Flow Player . The result is a conversion script that uses HandBrake to convert the videos. I run it as a nightly cron job on the file share with the videos. People drop their camera videos onto the share and my Linux box automatically converts them to reasonable sizes. Example Sizes for 23min video: Size Format Bitrate 1080p Camera Video 3.9 GB M2TS with AVC Video and AC3 Audio 24 MBit/s 320p for iPod 141 MB MP4 with AVC Video and AAC Audio 900 KBit/s 720p for Small Screen 373 MB MP4 with AVC Video and AAC Audio 2.2 MBit/s 1080p for Large Screen 1.2 GB MP4 with AVC Video and AAC Audio 7.4 MBit/s I am no expert on H264 tuning, so these values can be probably optimized quite

Thank You LinuxTag 2013!

Image
LinuxTag is over and I am almost dead. Besides my own talk about Open Source Sponsoring I also gave the Data Center Automation with YADT and the Relax and Recover for UEFI Systems talk, one talk each day. But, despite the smaller venue this year, LinuxTag is still an Open Source highlight for me. Our YADT booth was quite busy, even during the sessions. It is really good to meet so many people again, who come every year. Even better yet, I each time also meet new people who make the whole effort worthwhile. I also think that we nicely managed to raise the general awareness for YADT. So, big thanks to the organization team and see you next year!

Building Eruv at Limmud Festival 2013

Image
I helped building the Eruv at the Limmud Festival 2013 : Ganz in Familie Über 350 Besucher lernen am Werbellinsee http://www.juedische-allgemeine.de/article/view/id/15950 Jüdische Allgemeine, 10.05.2013 We were also filmed by Jewish News One : ( Video on YouTube )

How To Create a Debian/Ubuntu Repository for DEB Packages

Initial Setup Required Debian packages: reprepro Create a directory for the repository and its configuration: $ mkdir -p repo/conf Create a conf/distributions configuration file like this: Put my putinrepo script into the repo or next to it: "Export" the repo to create the metadata files for the empty repo: $ repo/putinrepo.sh $ tree repo repo ├── putinrepo.sh ├── conf │   └── distributions ├── db │   ├── checksums.db │   ├── contents.cache.db │   ├── packages.db │   ├── references.db │   ├── release.caches.db │   └── version └── dists ├── precise │   ├── main │   │   ├── binary-amd64 │   │   │   ├── Packages │   │   │   ├── Packages.gz │   │   │   └── Release │   │   ├── binary-armel │   │   │   ├── Packages │   │   │   ├── Packages.gz │   │   │   └── Release │   │   └── binary-i386 │   │   ├── Packages │   │   ├── Packages.gz │   │   └── Release │   └── Release ├── quantal │   ├──

Simple Command Line Download Speed Test

Image
I got a server that started to have a bad network connection. To help debugging this I needed to collect some data, e.g. of running a download test every 5 minutes. A quick search revealed nothing so I found a simple way to do that with curl and Google Drive. 1. Google Drive Form I set up a simple Google Drive form that accepts two text values: Next I need to find out the HTML form field names of these two fields. This takes a little bit of digging.   There is always a label  attached to each input  field. They are connected via the for  attribute of the label. With this information I can now construct a simple curl script. 2. CURL script to collect some data and post I want curl to try to download a URL and record the download speed. The -w , -y and -Y options come in handy for this purpose: -w lets me output only the information I really care about (the speed of the download in Bytes per second) -y gives a timeout after which curl will abort the download -Y