Schlomo Schapiro

The Open Source Data Center Conference 2016 is a good measure for how the industry changes. Compared to 2014 Cloud topics take more and more space. Both how to build your own on-premise cloud with Mesos , CoreOS or Kubernetes but also how to use the public Cloud. Maybe not surprising, I used the conference to present my own findings from 2 years of Cloud migration at ImmobilienScout24 : After we first tried to find  way to quickly migrate our data centers into the Cloud we now see that a hybrid approach works better. Data center and cloud are both valued platforms and we will optimize the costs between them. Hybrid Cloud - A Cloud Migration Strategy Do you use Cloud? Why? What about the 15 year legacy of your data center? How many Enterprise vendors tried to sell you their "Hybrid Cloud" solution? What actually is a Hybrid Cloud? Cloud computing is not just a new way of running servers or Docker containers. The interesting part of any Cloud offering are mana...

Everywhere there is some data that is relevant either for all applications or for many applications in different parts of the platform. The "obvious" solution to this problem is to make such data internally public or world-readable , meaning that the entire platform can read it. The "obvious" solution to security in this case is actually having no security  beyond ensuring the "are you part of us?" question. Common implementations of this pattern are world-readable NFS shares, S3 buckets readable by all "our" AWS accounts, HTTP APIs that use the client IP as their sole access control mechanism etc. This is approach is really dangerous and should be used with care. The risks include: You most likely don't know who actually needs the data and who not. If you ever need to restrict access you will have a very long and tedious job ahead of you. You don't know who accessed the data for which purpose. After a data leak, yo...

Day two at the microXchg 2016 conference. After listening to yet another talk detailing the pitfalls and dangers of "doing it wrong" I see more and more similarities between the Cloud migration at ImmobilienScout24 and the microservices journey that most speakers present. The Cloud migration moves us from a large data center into many smaller AWS accounts. A (legacy) monolithic application is cut into many smaller microservices. Internal data center communication becomes exposed communication between different AWS accounts and VPCs. Internal function calls are replaced with remote API calls. Both require much more attention to security, necessitate an authentication framework and add significant latency to the platform. A failed data center takes down the entire platform while a failed AWS account will only take down some function. An uncaught exception will crash the entire monolith while a crashed microservice will leave the others running undisturbed. Interna...

Today I was attending the Microxchg 2016 conference in Berlin. I suddenly realized that going to the cloud allows to ask completely new questions that are impossible to ask in the data center. One such question is this: What is the optimum size for a data center?  Microservices are all about downsizing - and in the cloud we can and should downsize the data center! In the world of physical data centers the question is usually goverened by two factors: Ensuring service availability by having at least two  physical data centers. Packing as much hardware into as little space as possible to keep the costs in check. As long as we are smaller than the average Internet giant there is no point to ask about the optimum size. The tooling which we build has to be designed for both large data centers and for having more than one. But in the "1, 2, many" series "2" is just the worst place to be. It entails all the disadvantages of "more than 1" without any o...

Docker provides a convenient way to package entire applications into runnable containers. OTOH in the data center we use RPM packages to deliver software and configuration to our servers. This wrapper build a bridge between Docker appliances and Linux services by packaging a Docker image as a Linux service into an RPM package. The resulting Linux service can be simply used like any other Linux service, for example start the service with service schlomo start . See the GitHub repo at  https://github.com/ImmobilienScout24/docker-service-rpm for code and more details and please let me know if you find this useful.

As ImmobilienScout24 moves to the cloud a recurring topic is the question about the exit strategy. An exit strategy is a plan for migrating away from the cloud, or at least from the chosen cloud vendor. Opinions range from "why would I need one?" to "how can we not have one?" with a heavy impact on our cloud strategy and how we do things in the cloud. When talking about exit scenarios it is worth to distinguish between a forced and a voluntary exit. A forced exit happens due to external factors that don't leave you any choice when to go. A voluntary exit happens at your own choice, both when and how. Why would one be force to have an exit strategy? Simple because running a business on cloud services carries other types of risks compared to running a business in your own data center: Cloud accounts can be disabled for alleged violation of terms Cloud accounts can be terminated There are no guaranteed prices. Running costs can explode as a result of a n...

Today I went to the GRIPS Theater  ( English ) instead of the office. Once a year ImmobilienScout24 donates the work force to social projects, called Social Day . I used the opportunity to catch a glimpse behinde the stage. The theater in turn got a workshop from us about their web site and social media channels. But first we watched a very nice children show ( Ein Fest bei Baba Dengiz ) about a German guy who learned respect for foreigners - from another German with Turkish background. The show was well adapted to the school-age audience. The theater follows a somewhat unusual concept and places the stage in the middle of the audience: Foto mit freundlicher Genehmigung des GRIPS Theaters This was my first visit to the GRIPS Theater, but not the last. Besides a rich children programme the theater also offers shows for adults and is most famously known for the show Linie 1 .

Relax and Recover  is the leading Open Source solution for automated Linux disaster recovery. It was once the pride of my work and is now totally irrelevant at my current job at ImmobilienScout24 . Why? Simply because at ImmobilienScout24 we invest our time into automating the setup of our servers instead of investing into the ability to automatically recover a manually configured system. Sounds simple but this is actually a large amount of work and not done in a few days. However, if you persist and manage to achieve the goal the rewards are much bigger: Don't be afraid of troubles, based on our automation we can be sure to reinstall our servers in a very short time. The following idea can help to bridge the gap if you cannot simply automate all your systems but still want to have a simplified backup and disaster recovery solution: Inject a layer of automation under the running system. The provisioning and configuration of the automation layer should be of cour...

A site to site VPN is the standard solution for connecting several physical data center locations. Going to the Cloud, the first idea that comes to mind is to also connect the Cloud "data center" with a site VPN to the existing physical data centers. All Cloud providers offer such a feature. But is such a VPN infrastructure also a "good idea"? Will it help us or hinder us in the future? I actually believe that for having many data centers a site VPN infrastructure is a dangerous tool. On the good side it is very convenient to have and to set up and it simplifies a lot of things. On the other side it is also very easy to build a world-wide mesh of dependencies where a VPN failure can severly inhibit data center operations or even take down services. It also lures everybody into creating undocumented backend connection between services. The core problem is in my opinion one of scale. Having a small number (3 to 5) of locations is fundamentally different from h...

Since ImmobilienScout24 decided to migrate to a public cloud I have been busy looking at various cloud offerings in detail. Amazon Web Services  (AWS) has a special feature which is interesting: Amazon Linux is a fully supported, "RHEL like", RPM-based Linux distribution. While not beeing a true Red Hat Enterprise Linux clone like CentOS or Scientific Linux (which is the standard OS for the ImmobilienScout24 data centers), it is derived from some Fedora version and comes with a nice choice of current software. To me it feels like "RHEL +" because so far all our internal stuff worked well but a lot of software packages are much newer than on RHEL 6 or RHEL 7. The 2014.09 release  updated a lot of components to very recent versions. On the other hand, we also found packages missing from Amazon Linux, most notably desktop-file-utils . This package is required to install Oracle Java RPMs . I found a thread about this on the AWS Forums and added a request fo...

Update:  Read my (German) conference report on heise developer . Last week I was at the DevOps Days Berlin 2014 . This time at the Kalkscheune , a much better location than the Urania from last year. With 250 people the conference was not too full and the location was also well equipped to handle this amount. Proving DevOps to be more about people and culture, most talks where not so technical but emphasized the need to take along all the people on the journey to DevOps. An technical bonus was the talk by Simon Eskildsen about " Docker at Shopify " which was the first time that I heard about a successful Docker implementation in production. Always good to know is the difference between effective and efficient as explained by Alex Schwartz in " DevOps means effectiveness first ". DevOps is actually a way to optimize for effectiveness before optimizing for efficience. Microsoft and SAP gave talks about DevOps in their world - quite impressive to see DevOps...

One full week of Python power is almost more than one can take, but I missing it would be even worse. This was my first EuroPython and with 1200 participants a big upgrade compared to the previous 2 PyCon.DE events in which I participated. The location ( Berlin Congress Center ) deserves kudos, along with the perfect organization. The Wifi worked really well (except for a WAN problem on Tuesday which was fixed quickly) and everybody loved the catering. They even had kosher, helal and vegan food (preordered), which is highly unusual for German conferences. Most amazing was the video crew who managed to upload all videos in about one hour after a talk was given. I managed to give three talks: DevOps Risk Mitigation How we use Test Driven Infrastructure at ImmobilienScout24 as part of our general automation to reduce the risk of giving everybody access everywhere. ( Access Slides or Watch Video ) YAML Reader Lightning Talk about the yamlreader Python library, which provide...

iPXE is a lesser known Open Source  PXE boot loader which offers many interesting features: boot from a web server via HTTP and HTTPS boot from remote block device over  iSCSI , FC , FCoE , AoE SAN boot from VLAN , WiFi, WAN, Infiniband control the boot process with a script user interaction with menus and login prompts display images and splash screens replaces NIC firmware or chain load via PXE Talk & Article Since iPXE plays a role in the ImmobilienScout24 boot automation I gave a talk about it at the LinuxTag 2014 . The talk is half an hour long and gives a quick introduction into iPXE. It covers build, configuration & scripting and shows how to develop boot scripts in iPXE with a very short feedback cycle. Download the  slides to the talk  and the  audio recording  as a podcast. At the conference the German Linux Magazin became interested in the topic and asked me to write an article about iPXE: Der vielseitige N...

At the 2014-06 Berlin DevOps Meetup this week we had an interesting fish bowl discussion about What is the risk of giving DEVs root access in production? Since I suggested the topic I was asked to give a short introduction into the topic: The discussion that followed was suprising in several aspects: A major concern is safeguarding the production data, but nobody had a really good solution for that. Many people have more problems with Developers seeing live customer data than with Develops changing something in production. "Nobody should have root" was proposed by a security specialist, but he had no practical working example for this approach. The question is tightly coupled to the degree of automation. The more automation you have the less need for anybody (Dev or Ops) to use their root privileges. Not everybody having root access knows what to do with it, Developers are sometimes afraid of using their power if granted root. This is mostly a question for lar...

Does your company care about employer branding ? Probably yes. Does your company care about corporate social responsibility ? Probably yes. Does your company combine these two to create a win-win situation? Most likely not! Take my employer ImmobilienScout24 as a typical example: The about us page mentiones that ImmobilienScout24 is a great place to work (4th in our region) and the CSR team talks about the social engagement, e.g. blood donations or the social day where all employees donate their work time to non-profit organizations. However, there is no obvious connection between these two things. I would like to suggest a simple way how to combine both employer branding and corporate social responsibility: A company should make it a priority to support charitable organizations and social projects related to their own employees. Examples: Sponsor non-profit organizations or neighborhood/community projects that employees are involved with. On social day, go to ...

The "Start Menu" of a Linux Desktop usually comes with a predefined set of categories that make up the sub menus. If you have a lot of custom applications then you might want to group them under a dedicated sub menu instead of having them spread out over all the menu categories. Adding sub menus and new categories on Linux Desktops is defined in the Desktop Menu Specification in Appendix C . It turns out that it is really simple and the following example from ImmobilienScout24 can serve as a base for your own custom menu. You will need the following parts: A Desktop file using a custom category A Directory file defining the icon and description for the new sub menu The icon for the sub menu An XML file describing how to integrate the new sub menu into the menu structure and which categories of Desktop files to show in the new menu The Desktop file describes the menu entry, in this example the VPN client: The important part here is the Categories  entry whic...

Patching configuration files is like the bread-and-butter job of every configuration management. In our package-based deployment world we try to minimize the patching to the absolute minimum, usually to "enable" modularized configuration patterns. The best example is the Apache Webserver, where we have a wrapper RPM package with a %post script that simply replaces (and not patches) the upstream configuration with a few include lines: Sadly there is still a lot of software that does not support includes in its configuration. For these we of course have to patch the existing configuration and use this short and simple config patcher in our RPM %post scripts, for example like this for sshd_config: The trick of this snippet is that in the end the changed parts are always at the top of the file. It is also important to always embed some information about the cause of the patch so that one can easily find out who or what is reponsible for the file. The %-variables are fil...

When I bought a new Chromebook Acer C720  last week I got confirmation that times are changing: It has only an HDMI connector, no more VGA. Luckily, at ImmobilienScout24 we are also adapting and last month our big projector got an upgrade to Full HD with 16:9 Wide Screen. And you can now connect the computer through HDMI, too. Since me myself so much got used to creating presentations in 4:3 I took the opportunity to remind myself and everybody else why it really pays to pay attention to this little detail. Video is in German with English subtitles.

A colleague, Eric Grehm, raised an interesting challenge: How to maintain his personal work environment (VIM settings, .bashrc ...) on all servers? The first thought was putting this somehow into our software distribution, but we quickly realized that this would trigger needless updates on hundreds of servers. The benefit would be that the personal work environment is already on every server upon first access. The next idea is to switch from a pre-installed personal environment to an on-demand solution where the personal environment is transferred each time a remote connection (over SSH) is established. A simple implementation would just to a scp before the ssh, but that entails two connections which takes more time and might also bother the user with a double password request. Side-channel data transfer An alternative is to piggyback the file transfer onto the regular SSH connection so that the personal environment is transferred in a side channel: On the client creat...

Yesterday I was at the Berlin DevOps meetup and we had a very nice fishbowl about Test Driven Infrastructure (TDI). I used my Lightning Talk from the PyCon in Köln as an introduction to the topic, but quickly realized that the term does not fully explain itself. Test Driven Development  in itself is not a new thing, maybe it is not yet common to apply it to platform operations. As an old Ops guy I had a lot to learn when I started to work at ImmobilienScout24 , which is a real software development company. The bottom line is really simple: Untested = Broken My idea of TDI is to apply most of the basic ideas of TDD also to the development process  of the software that runs our platform. Again, the same thing as the developers do with their code already for a long time. Let's just say that we start to test all  code that goes on a server, no matter who wrote it or what it actually does. Some specific examples that we did in the last month: A service that ...