Schlomo Schapiro

Summary:  Use digital identities, trust relationship and access control lists instead of passwords. In the cloud, this is really easy. I strongly believe that static credentials are one of the biggest hazards in modern IT environments. Most information security incidents are somehow related to lost or leaked or guessed static credentials, Instagram's Million Dollar Bug is just one very nice example. Static credentials can be used by anyone who has them - friend or foe are typically very short and can even be brute forced or guessed for machine or service users have to be stored in configuration files from where they can be leaked are hard to remember for humans so that they will write them down somewhere or store them in files typically stay the same over a long period of time don't include any information about the identity of the bearer or user are hard to rotate on a regular base because the change has to happen in several places at the same time All th...

The CoreOS Fest 2016 in Berlin impressed me very much: A small Open Source company organizes a 2 day conference around their Open Source tools and even flies in a lot of their employees from San Francisco. A win both for Open Source and for Berlin. And CoreOS also announced that they got new funding of $28M : Alex Polvi , CEO of CoreOS More interesting for IT people everywhere is the message one can learn here: Container technologies are ready for production. There is a healthy environment of Open Source solutions: Kubernetes , Mesosphere DC/OS , containerd , even OpenStack and others. Commercial editions with vendor support like Tectonic  (CoreOS), Mesosphere Enterprise or Hashicorp Atlas 3rd party tools solving common problems like persistent storage: StorageOS  and  Quobyte In fact, choosing the "right" platform starts to become the main problem for those who still run on traditional Virtualization platforms. On the other hand, IT companies who don't ...

Everywhere there is some data that is relevant either for all applications or for many applications in different parts of the platform. The "obvious" solution to this problem is to make such data internally public or world-readable , meaning that the entire platform can read it. The "obvious" solution to security in this case is actually having no security  beyond ensuring the "are you part of us?" question. Common implementations of this pattern are world-readable NFS shares, S3 buckets readable by all "our" AWS accounts, HTTP APIs that use the client IP as their sole access control mechanism etc. This is approach is really dangerous and should be used with care. The risks include: You most likely don't know who actually needs the data and who not. If you ever need to restrict access you will have a very long and tedious job ahead of you. You don't know who accessed the data for which purpose. After a data leak, yo...

Day two at the microXchg 2016 conference. After listening to yet another talk detailing the pitfalls and dangers of "doing it wrong" I see more and more similarities between the Cloud migration at ImmobilienScout24 and the microservices journey that most speakers present. The Cloud migration moves us from a large data center into many smaller AWS accounts. A (legacy) monolithic application is cut into many smaller microservices. Internal data center communication becomes exposed communication between different AWS accounts and VPCs. Internal function calls are replaced with remote API calls. Both require much more attention to security, necessitate an authentication framework and add significant latency to the platform. A failed data center takes down the entire platform while a failed AWS account will only take down some function. An uncaught exception will crash the entire monolith while a crashed microservice will leave the others running undisturbed. Interna...

Today I was attending the Microxchg 2016 conference in Berlin. I suddenly realized that going to the cloud allows to ask completely new questions that are impossible to ask in the data center. One such question is this: What is the optimum size for a data center?  Microservices are all about downsizing - and in the cloud we can and should downsize the data center! In the world of physical data centers the question is usually goverened by two factors: Ensuring service availability by having at least two  physical data centers. Packing as much hardware into as little space as possible to keep the costs in check. As long as we are smaller than the average Internet giant there is no point to ask about the optimum size. The tooling which we build has to be designed for both large data centers and for having more than one. But in the "1, 2, many" series "2" is just the worst place to be. It entails all the disadvantages of "more than 1" without any o...

Docker provides a convenient way to package entire applications into runnable containers. OTOH in the data center we use RPM packages to deliver software and configuration to our servers. This wrapper build a bridge between Docker appliances and Linux services by packaging a Docker image as a Linux service into an RPM package. The resulting Linux service can be simply used like any other Linux service, for example start the service with service schlomo start . See the GitHub repo at  https://github.com/ImmobilienScout24/docker-service-rpm for code and more details and please let me know if you find this useful.

How many log levels do you know? How many log levels are actually useful? At Relax and Recover we had an interesting discussion about the use of the WARNING log level. I suddenly realized that in a world of automation, I need only two log levels: ERROR and everthing else. ERROR means that I as a human should take action. Everything else is irrelevant for me. So far for the user side. As a programmer the choice of log level is sometimes much more difficult. As a programmer I might not want to decide for the user if some problem is an ERROR or not. The obvious solution is to issue a WARNING in an attempt to shed the responsibility of making a decision. But in an automated world that does not help me as an admin to run the software better. WARNINGS for most cases only create extra manual work because somebody needs to go and check some log file and decide if there actually is a problem. I would rather have the software make that decision and I would be happy to fix or readjus...

Henrik G. Vogel  / pixelio.de Ever wonder if your website is slow because of the server or because of the clients? Do you want to know how fast is your clients' connection to the Internet? Don't want to use external tracking services, injecting JavaScript etc.? Why not simply measure how long it takes to deliver the content from your webserver to your users? Apache and nginx both support logging the total processing time of a request with a suitably high precision. That gives the time from starting with first byte received from the client and ending after the last byte sent to the client. To try out this idea I added %D to the log format for access.log of my Apache server and wrote a little Python script to calculate the transfer speeds. With the help of the apachelog Python module parsing the Apache access.log is really simple. This module takes a log format definition as configuration and automatically breaks down a log line into the corresponding values. ...

Yesterday I was at the Berlin DevOps meetup and we had a very nice fishbowl about Test Driven Infrastructure (TDI). I used my Lightning Talk from the PyCon in Köln as an introduction to the topic, but quickly realized that the term does not fully explain itself. Test Driven Development  in itself is not a new thing, maybe it is not yet common to apply it to platform operations. As an old Ops guy I had a lot to learn when I started to work at ImmobilienScout24 , which is a real software development company. The bottom line is really simple: Untested = Broken My idea of TDI is to apply most of the basic ideas of TDD also to the development process  of the software that runs our platform. Again, the same thing as the developers do with their code already for a long time. Let's just say that we start to test all  code that goes on a server, no matter who wrote it or what it actually does. Some specific examples that we did in the last month: A service that ...

If you work at an agile shop you probably have seen something like this: You also find a lot about how people create those nice dash boards as a web page, but you almost never find out how to actually build the system that runs the dash board. Basically this is a task for digital signage, but "standard" solutions like Xibo seem heavily oversized for running an internal dash board. All we need is a Linux box running a browser and the screen staying on. As my team also wanted to have a dashboard, I  recently hat to build one and decided to put my effort into an Ubuntu package named  kiosk-browser  that can be found on the ImmobilienScout24 github page. The package turns a regular Ubuntu installation into a web-based dashboard. Now the effort to setup a new dashboard system has been reduced to these steps: Install a computer with Ubuntu. Build or download and install the kiosk-browser package.  Log on as a user and run this command: sudo tee /etc/defa...

Going to conferences and meetups is not just for seeing the talks, it is much more for meeting the "right" people. At the recent devopsdays Mountain View I met Ernest Mueller and discovered his theagileadmin.com  blog, where I immediately liked his  DevOps: It’s Not Chef And Puppet blog post. After some talking with Ernest and others at the devopsdays I realized that I believe that the Ops job contains more than most people do with Puppet and Chef: Managing the Operating System patches and upgrades. Read my guest posting  Puppet and Chef do only half the job  for the full story. Update: In the 2012 SysAdvent Blog Miah Johnson suggests in Packages Doing Too Much? to solve this issue by going the opposite way: Deploy the entire operating systems via recipes. She also points to NixOS and Guix as an example of such operationg systems. While I still believe that using the OS packages for custom software and configuration deployment is the cheapest solution, ...

I just got an idea that it might be interesting to combine mod_dims and GPU accelerated image manipulation libraries like NVidia's NPP . Apparently nobody looked into this so far, even though a google search will show a lot of GPU related image processing stuff , NVidia also provides a nice intro to GPU image processing . The idea is that if it would be possible to service scaled images as fast as unscaled images then it would be possible to skip all the solutions that deal with the caching of scaled images and just scale the images always on demand. This depends on the ability to service scaled images really fast, for example with the help of some Tesla boards for the image scalers. It would be an interesting comparison to compare the cost of sufficiently available storage for caching plus the cost for elastic on-demand scaling with the cost of permanent on-demand scaling with GPU acceleration. The advantage of the latter would be, if the costs are in the same region, that i...