Schlomo Schapiro

Update 23.04.2018: Added demo and link to conference talk video/slides at the end of this article Kubernetes is a great way to run applications because it allows us to manage single Linux processes with a real cluster manager. A computer with multiple services is typically implemented as a pod with multiple containers  sharing communication and storage : Ideally every container runs only a single process. On Linux, most applications have three phases with two different programs or scripts: The initialization  phase, typically an init script or a systemd unit file. The run  phase, typically a binary or a script that runs a daemon. The maintenance  phase, typically a script run as a CRON job. While it is possible to put the initialization phase into a Docker container as part of the ENTRYPOINT script, that approach gives much less control over the entire process and makes it impossible to use different security contexts for each phase, e.g. to prevent ...

Last week I wrote about understanding IAM Roles , let's follow up with some practical aspects. The following examples and scripts all use the  aws-cli  which you should have already installed. The scripts work on Mac and Linux and probably on Windows under  Cygwin . To illustrate the examples I use the case of an S3 backup bucket in another AWS account. For that scenario it is recommended to use a dedicated access role in the target AWS account to avoid troubles with S3 object ownership. AWS Who Am I? The most important question is sometimes to ascertain the identity. Luckily the aws-cli provides an option for that: $ aws sts get-caller-identity {     "Account": "123456789",     "UserId": "ABCDEFG22L2KWYE5WQ:sschapiro",     "Arn": "arn:aws:sts::123456789:assumed-role/PowerUser/sschapiro" } From this we can learn our AWS account and the IAM Role that we currently use, if any. AWS Assume Role Script The following Bash s...

One of the most important security features of Amazon AWS are IAM Roles. They provide a security umbrella that can be adjusted to an application's needs in great detail. As I all the time forget the details I summarize here everything that helps me and some useful tricks for working with IAM Roles. This is part one of two . Understanding IAM Roles From a conceptual perspective an IAM Role is a sentence like Alice may eat apples : It grants or denies permissions (in the form of a access policy ) on specific resources to principals. Alice is the principal, may  is the granting, eat  is the permission (to eat , but not to look at ) and apples  is the resource, in this case any kind of apples. IAM Roles can be much more complex, for example this rather complex sentence is still a very easy to read IAM Role: Alice and Bob from Hamburg may find, look at, smell, eat and dispose of apples № 5 and bananas . Here we grant permissions to our Alice and to some Bo...

Who has root access in your IT organizations? Do you "do" DevOps? Even though getting root access was once my personal motivation for pushing DevOps, I never considered the question of the relationship till it was triggered by my last conference visit. Last week I attended the 10. Secure Linux Administration Conference - a small but cherished German event catering to Linux admins - and there where two DevOps talks: DevOps in der Praxis (Practical DevOps) by Matthias Klein and my own DevOps for Everybody talk. I found it very interesting that we both talked about DevOps from a "been there, done it" perspective, although with a very different message. DevOps ≠ DevOps For me DevOps is most of all a story of Dev and Ops being equal, sitting in the same boat and working together on shared automation to tackle all problems. My favourite image replaces humans as gateway to the servers with tooling that all humans use to collaboratively deliver changes to the ser...

The CloudNativeCon + KubeCon North Europe 2017 in Berlin was sold out with 1500 participants. I learned really a lot about Kubernetes and the other new and shiny tools that start to become main stream. To get an introduction into Cloud Native, watch  Alexis Richardson  in the keynote on " What is Cloud Native and Why Should I care " ( slides , video at 12:27). He explained the goal of the Cloud Native Computing Foundation  (CNCF) as avoiding cloud lock-in , which is much more to the point than the official charter  (which talks about "the adoption of a new computing paradigm"). Alexis chairs the Technical Oversight Committee (TOC) of the CNCF. The Foundation is " projects first ", set up similar to the Linux Foundation and already sponsors various Open Source projects . Linux Lock-In His remarks got me to think about the question, especially in comparison with Linux. To me it seems that modern IT in the data center already has a pretty strong ...

SSH keys are considered to be a security feature, but sometimes they make things more complicated than necessary. Especially in automation contexts we use SSH keys without  a pass phrase which degrades the security of the SSH keys to the security level of a plain text password. The only benefit of the SSH keys is the fact that an attacker who gains access to the server won't be able to use the keys found there to login somewhere else. As such SSH keys are still better and more secure than having a regular plain text password. In automation contexts we sometimes have to handle lots of SSH keys, for example with GitHub Deploy Keys . GitHub mandates to use a different SSH key for every repository to ensure that a leaked private key will not lead to a breach of other repositories. I recently had to configure a Go Continuous Delivery server and it turned out that it does not support managing SSH keys at all (like Jenkins or TeamCity do). In order to still be able to us...

I had the honor to attend a new (for me) conference: The spring meeting of the German Unix User Group , this time hosted by the Cybersecurity department of the Darmstadt Technical University . The conference had about 115 participants and orients itself mostly towards admins. The former emphasis on Unix is long gone, all talks except one (about Solaris) where about Linux and Linux-based technologies. Two days of tutorials where followed by 2 days of talks in 2 parallel tracks. Noteworthy talks where the keynote about Jailbreaking WiFi Firmware by  Matthias Schulz , Architecture Pattern for Container and Kubernetes by Thomas Fricke and several talks about software defined storage. Especially the ensuing discussion between the speakers representing competing approaches helped many attendees to sharpen their own opinions. Jailbreaking WiFi Firmware Impressive walk-through of the effort it took to turn Broadcom Wifi chips into WiFi monitors suitable for WiFi hacking. The c...

My company sold old laptops to employees and I decided to use the chance to get an affordable and legally licensed Windows 10 system - a Dell Latitude E6420 . Unfortunately the system has a Broadcom Wifi card and also ships with an NVidia graphics card which require extra work on Ubuntu 16.04 Xenial Xerus . After some manual configuration the system works quite well with a power consumption of about 10-15W while writing this blog article. Switching between the Intel and the NVidia graphics card is simple (with a GUI program and requires a logout-login), for most use cases I don't need the NVidia card in any case. Windows 10 also works well, although it does not support all devices. However, the combined NVidia / Intel graphics systems works better on Windows than on Linux. In detail, I took the following steps to install an Ubuntu 16.04 and Windows 10 dual boot system. Step-by-Step Installation Requirements Either a wired network connection or a USB wifi dongle that ...

Summary:  Use digital identities, trust relationship and access control lists instead of passwords. In the cloud, this is really easy. I strongly believe that static credentials are one of the biggest hazards in modern IT environments. Most information security incidents are somehow related to lost or leaked or guessed static credentials, Instagram's Million Dollar Bug is just one very nice example. Static credentials can be used by anyone who has them - friend or foe are typically very short and can even be brute forced or guessed for machine or service users have to be stored in configuration files from where they can be leaked are hard to remember for humans so that they will write them down somewhere or store them in files typically stay the same over a long period of time don't include any information about the identity of the bearer or user are hard to rotate on a regular base because the change has to happen in several places at the same time All th...

The CoreOS Fest 2016 in Berlin impressed me very much: A small Open Source company organizes a 2 day conference around their Open Source tools and even flies in a lot of their employees from San Francisco. A win both for Open Source and for Berlin. And CoreOS also announced that they got new funding of $28M : Alex Polvi , CEO of CoreOS More interesting for IT people everywhere is the message one can learn here: Container technologies are ready for production. There is a healthy environment of Open Source solutions: Kubernetes , Mesosphere DC/OS , containerd , even OpenStack and others. Commercial editions with vendor support like Tectonic  (CoreOS), Mesosphere Enterprise or Hashicorp Atlas 3rd party tools solving common problems like persistent storage: StorageOS  and  Quobyte In fact, choosing the "right" platform starts to become the main problem for those who still run on traditional Virtualization platforms. On the other hand, IT companies who don't ...

The Open Source Data Center Conference 2016 is a good measure for how the industry changes. Compared to 2014 Cloud topics take more and more space. Both how to build your own on-premise cloud with Mesos , CoreOS or Kubernetes but also how to use the public Cloud. Maybe not surprising, I used the conference to present my own findings from 2 years of Cloud migration at ImmobilienScout24 : After we first tried to find  way to quickly migrate our data centers into the Cloud we now see that a hybrid approach works better. Data center and cloud are both valued platforms and we will optimize the costs between them. Hybrid Cloud - A Cloud Migration Strategy Do you use Cloud? Why? What about the 15 year legacy of your data center? How many Enterprise vendors tried to sell you their "Hybrid Cloud" solution? What actually is a Hybrid Cloud? Cloud computing is not just a new way of running servers or Docker containers. The interesting part of any Cloud offering are mana...

Everywhere there is some data that is relevant either for all applications or for many applications in different parts of the platform. The "obvious" solution to this problem is to make such data internally public or world-readable , meaning that the entire platform can read it. The "obvious" solution to security in this case is actually having no security  beyond ensuring the "are you part of us?" question. Common implementations of this pattern are world-readable NFS shares, S3 buckets readable by all "our" AWS accounts, HTTP APIs that use the client IP as their sole access control mechanism etc. This is approach is really dangerous and should be used with care. The risks include: You most likely don't know who actually needs the data and who not. If you ever need to restrict access you will have a very long and tedious job ahead of you. You don't know who accessed the data for which purpose. After a data leak, yo...

At the microXchg 2016 last week Fred George - who takes pride having been called a hand grenade - gave a very inspiring talk about how all the things that we do right now have one primary goal: Go Faster Reducing cycle time for deployments, automation everywhere, down-sizing services to "microservices", building resilient and fault-tolerant platforms and more are all facets of a bigger journey: Provide value faster and find out faster what works and what not. DevOps DevOps is seen by most developers as beeing an Ops movement to catch on with developers before their jobs become obsolete. Attending various DevOps Days in Germany and the USA, the developers who where also there always complained about the lack of developers and the lack of developer topics. They observed that the conference seems to be by and for Ops people. Consequently, DevOps conferences usually have two tracks: Methods and Tools. Methods teach us how to do "proper" software develop...

Day two at the microXchg 2016 conference. After listening to yet another talk detailing the pitfalls and dangers of "doing it wrong" I see more and more similarities between the Cloud migration at ImmobilienScout24 and the microservices journey that most speakers present. The Cloud migration moves us from a large data center into many smaller AWS accounts. A (legacy) monolithic application is cut into many smaller microservices. Internal data center communication becomes exposed communication between different AWS accounts and VPCs. Internal function calls are replaced with remote API calls. Both require much more attention to security, necessitate an authentication framework and add significant latency to the platform. A failed data center takes down the entire platform while a failed AWS account will only take down some function. An uncaught exception will crash the entire monolith while a crashed microservice will leave the others running undisturbed. Interna...

Today I was attending the Microxchg 2016 conference in Berlin. I suddenly realized that going to the cloud allows to ask completely new questions that are impossible to ask in the data center. One such question is this: What is the optimum size for a data center?  Microservices are all about downsizing - and in the cloud we can and should downsize the data center! In the world of physical data centers the question is usually goverened by two factors: Ensuring service availability by having at least two  physical data centers. Packing as much hardware into as little space as possible to keep the costs in check. As long as we are smaller than the average Internet giant there is no point to ask about the optimum size. The tooling which we build has to be designed for both large data centers and for having more than one. But in the "1, 2, many" series "2" is just the worst place to be. It entails all the disadvantages of "more than 1" without any o...