Schlomo Schapiro

I strongly believe that you can't be successful in the Cloud without also adopting DevOps. Here is why. My latest definition of DevOps is if every person uses the same tool for the same job codified knowledge: everybody contributes his part to common automation if all people have the same privileges in their tooling if human error is equally possible for Dev and Ops replacing people interfaces by automated decisions and processes but most of all DevOps is the result of doing the right thing and not a process, methodology or even tool set of its own. Looking closely at public cloud vendors and their interfaces I see a close correlation with this DevOps definition. Cloud vendors give every person the same interfaces and tools to work with are API based and make it really simple to code the entire setup give all users the same privileges - that of a customer let all their users make the same mistakes indisccriminately provide most change requests through auto...

Last week I wrote about understanding IAM Roles , let's follow up with some practical aspects. The following examples and scripts all use the  aws-cli  which you should have already installed. The scripts work on Mac and Linux and probably on Windows under  Cygwin . To illustrate the examples I use the case of an S3 backup bucket in another AWS account. For that scenario it is recommended to use a dedicated access role in the target AWS account to avoid troubles with S3 object ownership. AWS Who Am I? The most important question is sometimes to ascertain the identity. Luckily the aws-cli provides an option for that: $ aws sts get-caller-identity {     "Account": "123456789",     "UserId": "ABCDEFG22L2KWYE5WQ:sschapiro",     "Arn": "arn:aws:sts::123456789:assumed-role/PowerUser/sschapiro" } From this we can learn our AWS account and the IAM Role that we currently use, if any. AWS Assume Role Script The following Bash s...

One of the most important security features of Amazon AWS are IAM Roles. They provide a security umbrella that can be adjusted to an application's needs in great detail. As I all the time forget the details I summarize here everything that helps me and some useful tricks for working with IAM Roles. This is part one of two . Understanding IAM Roles From a conceptual perspective an IAM Role is a sentence like Alice may eat apples : It grants or denies permissions (in the form of a access policy ) on specific resources to principals. Alice is the principal, may  is the granting, eat  is the permission (to eat , but not to look at ) and apples  is the resource, in this case any kind of apples. IAM Roles can be much more complex, for example this rather complex sentence is still a very easy to read IAM Role: Alice and Bob from Hamburg may find, look at, smell, eat and dispose of apples № 5 and bananas . Here we grant permissions to our Alice and to some Bo...

Summary:  Use digital identities, trust relationship and access control lists instead of passwords. In the cloud, this is really easy. I strongly believe that static credentials are one of the biggest hazards in modern IT environments. Most information security incidents are somehow related to lost or leaked or guessed static credentials, Instagram's Million Dollar Bug is just one very nice example. Static credentials can be used by anyone who has them - friend or foe are typically very short and can even be brute forced or guessed for machine or service users have to be stored in configuration files from where they can be leaked are hard to remember for humans so that they will write them down somewhere or store them in files typically stay the same over a long period of time don't include any information about the identity of the bearer or user are hard to rotate on a regular base because the change has to happen in several places at the same time All th...

The CoreOS Fest 2016 in Berlin impressed me very much: A small Open Source company organizes a 2 day conference around their Open Source tools and even flies in a lot of their employees from San Francisco. A win both for Open Source and for Berlin. And CoreOS also announced that they got new funding of $28M : Alex Polvi , CEO of CoreOS More interesting for IT people everywhere is the message one can learn here: Container technologies are ready for production. There is a healthy environment of Open Source solutions: Kubernetes , Mesosphere DC/OS , containerd , even OpenStack and others. Commercial editions with vendor support like Tectonic  (CoreOS), Mesosphere Enterprise or Hashicorp Atlas 3rd party tools solving common problems like persistent storage: StorageOS  and  Quobyte In fact, choosing the "right" platform starts to become the main problem for those who still run on traditional Virtualization platforms. On the other hand, IT companies who don't ...

The Open Source Data Center Conference 2016 is a good measure for how the industry changes. Compared to 2014 Cloud topics take more and more space. Both how to build your own on-premise cloud with Mesos , CoreOS or Kubernetes but also how to use the public Cloud. Maybe not surprising, I used the conference to present my own findings from 2 years of Cloud migration at ImmobilienScout24 : After we first tried to find  way to quickly migrate our data centers into the Cloud we now see that a hybrid approach works better. Data center and cloud are both valued platforms and we will optimize the costs between them. Hybrid Cloud - A Cloud Migration Strategy Do you use Cloud? Why? What about the 15 year legacy of your data center? How many Enterprise vendors tried to sell you their "Hybrid Cloud" solution? What actually is a Hybrid Cloud? Cloud computing is not just a new way of running servers or Docker containers. The interesting part of any Cloud offering are mana...

Everywhere there is some data that is relevant either for all applications or for many applications in different parts of the platform. The "obvious" solution to this problem is to make such data internally public or world-readable , meaning that the entire platform can read it. The "obvious" solution to security in this case is actually having no security  beyond ensuring the "are you part of us?" question. Common implementations of this pattern are world-readable NFS shares, S3 buckets readable by all "our" AWS accounts, HTTP APIs that use the client IP as their sole access control mechanism etc. This is approach is really dangerous and should be used with care. The risks include: You most likely don't know who actually needs the data and who not. If you ever need to restrict access you will have a very long and tedious job ahead of you. You don't know who accessed the data for which purpose. After a data leak, yo...

Day two at the microXchg 2016 conference. After listening to yet another talk detailing the pitfalls and dangers of "doing it wrong" I see more and more similarities between the Cloud migration at ImmobilienScout24 and the microservices journey that most speakers present. The Cloud migration moves us from a large data center into many smaller AWS accounts. A (legacy) monolithic application is cut into many smaller microservices. Internal data center communication becomes exposed communication between different AWS accounts and VPCs. Internal function calls are replaced with remote API calls. Both require much more attention to security, necessitate an authentication framework and add significant latency to the platform. A failed data center takes down the entire platform while a failed AWS account will only take down some function. An uncaught exception will crash the entire monolith while a crashed microservice will leave the others running undisturbed. Interna...

Today I was attending the Microxchg 2016 conference in Berlin. I suddenly realized that going to the cloud allows to ask completely new questions that are impossible to ask in the data center. One such question is this: What is the optimum size for a data center?  Microservices are all about downsizing - and in the cloud we can and should downsize the data center! In the world of physical data centers the question is usually goverened by two factors: Ensuring service availability by having at least two  physical data centers. Packing as much hardware into as little space as possible to keep the costs in check. As long as we are smaller than the average Internet giant there is no point to ask about the optimum size. The tooling which we build has to be designed for both large data centers and for having more than one. But in the "1, 2, many" series "2" is just the worst place to be. It entails all the disadvantages of "more than 1" without any o...

As ImmobilienScout24 moves to the cloud a recurring topic is the question about the exit strategy. An exit strategy is a plan for migrating away from the cloud, or at least from the chosen cloud vendor. Opinions range from "why would I need one?" to "how can we not have one?" with a heavy impact on our cloud strategy and how we do things in the cloud. When talking about exit scenarios it is worth to distinguish between a forced and a voluntary exit. A forced exit happens due to external factors that don't leave you any choice when to go. A voluntary exit happens at your own choice, both when and how. Why would one be force to have an exit strategy? Simple because running a business on cloud services carries other types of risks compared to running a business in your own data center: Cloud accounts can be disabled for alleged violation of terms Cloud accounts can be terminated There are no guaranteed prices. Running costs can explode as a result of a n...

This week was my Meetup Marathon: Microservices Meetup Berlin about  Software Memories and Simulated Machines by  William Louth . Berlin DevOps about Scaling Logstash: A Collection of War Stories  by  Pere Urbon-Bayes and "Continuous development with Nix" by  Rok Garbas . Sadly there was no time for the traditional fish bowl discussion. AWS User Group Meetup about  STUPS tools & components platform by Henning Jacobs and Distributed Log Refinement Discussion  by  Christian Kniep . Software Memories and Simulated Machines was above my head. Scaling Logstash made me wonder how many engineers you actually need to run that "properly". Nix is something we hopefully don't need, Rok actually said that if you package everything you don't need it. STUPS  is the "Cloud Ops" stack from Zalando , nicely published on GitHub : The STUPS platform is a set of tools and components to provide a convenient and audit-compliant Platform-...

A site to site VPN is the standard solution for connecting several physical data center locations. Going to the Cloud, the first idea that comes to mind is to also connect the Cloud "data center" with a site VPN to the existing physical data centers. All Cloud providers offer such a feature. But is such a VPN infrastructure also a "good idea"? Will it help us or hinder us in the future? I actually believe that for having many data centers a site VPN infrastructure is a dangerous tool. On the good side it is very convenient to have and to set up and it simplifies a lot of things. On the other side it is also very easy to build a world-wide mesh of dependencies where a VPN failure can severly inhibit data center operations or even take down services. It also lures everybody into creating undocumented backend connection between services. The core problem is in my opinion one of scale. Having a small number (3 to 5) of locations is fundamentally different from h...

Since ImmobilienScout24 decided to migrate to a public cloud I have been busy looking at various cloud offerings in detail. Amazon Web Services  (AWS) has a special feature which is interesting: Amazon Linux is a fully supported, "RHEL like", RPM-based Linux distribution. While not beeing a true Red Hat Enterprise Linux clone like CentOS or Scientific Linux (which is the standard OS for the ImmobilienScout24 data centers), it is derived from some Fedora version and comes with a nice choice of current software. To me it feels like "RHEL +" because so far all our internal stuff worked well but a lot of software packages are much newer than on RHEL 6 or RHEL 7. The 2014.09 release  updated a lot of components to very recent versions. On the other hand, we also found packages missing from Amazon Linux, most notably desktop-file-utils . This package is required to install Oracle Java RPMs . I found a thread about this on the AWS Forums and added a request fo...