LinuxCon Berlin 2016

Berlin 4-6 October 2016

Biggest Linux conference in Europe, organised by the Linux Foundation. Lots of Linux kernel developers: Linux Foundation (Greg Kroah-Hartman was there!) , Google , Intel , Amazon (AWS) , Fujitsu. Operating systems: RedHat Linux , SuSE Linux , Oracle Linux , CentOS , CoreOS Lots of container & virtualisation: , Docker developers (and the company) , Kubernetes , Mesos , rkt , OpenShift , QEMU , KVM , Vagrant.

Other Linux subsystems

• systemd and PulseAudio author Lennart Poettering was there
• Kernel security: crypto API and loads of things I didn't understand one iota of 😉
• etcd

Hardware with heavy Linux investment: Dell EMC, IBM, HP, Huawei.

Other notables: , Free Software Foundation , Fedora , ffmpeg Databases: Percona , MariaDB

It was...Awesome!

25 year's celebration @ Charlottenburg palace

The more you know, the more you know you don't know. Aristotle

Google & open source

• Don't use AGPL, "dangerous". Not even contribute to it. Chris DiBona explains this in a The Register article.
• Best way is to start off a project as open source, harder to turn it into open source later.
• Prod, stripped down Debian
• Desktops run Goobuntu
• code.google.com created to take over from sourcefoge when that turned bad (new owners).
• Big git trees on android.googlesource.com
• Openchainproject.org is easier to generate license overviews.
• Apache v2 license clearly gives a patent grant, therefore it's the default license at Google.
• WTFPL and Unlicense cannot be used. Dangerous for a company since they don't have the legal thoroughness to stand up in court. e.g. they don't have necessary disclaimers.

Security

Security in the Linux kernel (way over my head, understood 10%)

• Security model, DAC, designed in the 1960s. Inherited from UNIX.

Linux security modules (LSM)

• SELinux
• Smack
• AppArmor
• Can constrain even what root can do

Integrity management (IMA, EVM)

• CALIPSO IPv6 labelling, can put labels on IP packets

AppArmor

• Seems to have a lot of development
• Policy namespaces
• Poilicy stacking
• Integration with containers
• Check out Jon Johansen's talk on LSS 2016

cgroups

• Limiting what a process can do
• How much CPU, memory++ the process (group) is allowed to use.
• freeze, restore, checkpoint a group

Kernel namespaces

• Give a subset view of what's available.
• e.g. PIDs, user IDs and interfaces

What is a container?

What' the difference between a container and a virtual machine?

Think focus:

• You package up services
• And don't care about the machine on which it runs

Containers are convenient sugar

• Uses old, standard kernel features
• cgroups
• kernel namespaces
• A tarball

demo.start();

$ pstree -a   $(pidof dockerd)
# kill $(pidof sleep)

demo.stop();

Highly recommended

Containerising Java applications

I've talked to a lot of experts, including container & operations experts from HP, RedHat and CoreOS.

These are people that have loads of experience with BIG deployments, LOTS of deployments and commit code to Docker and related technologies like Kubernetes

None knew of a good way to monitor or error hunt a Java application running in a clustered Docker (or other) container.

Containers are great ...

• For some applications (e.g. WordPress)
• Stateless applications ("give me 4 memcaches")
• Popular to install hairy tools into containers. seccomp.
• Even if you 💘 containers, you must remember that it's the orchestration that is hard.
• Error hunting in production is still the pink elephant in the room.
• Using containers for systems with more than one machine, you need orchestration.
• Orchestration is complex.
• Container orchestration is a young discipline.

Containers are not so great ...

• Data and conf files is still hard (how to get these to the container).
• Webshhere likes to set the hostname (!) a problem when running in the container where you don't know the hostname.
• Self modifying code (ruby creates copies of itself, game unique of each user)
• Dynamic IPSec tunnels between each new node (bank)
• Installers (we don't know all what they will do)
• Java kill -QUIT, remote debugging: not easy, no standard way of doing this. Must hack/architect around it. Even more difficult is app server clustering (like jboss or oracle app server)
• Java applications don't containerise well.

Container Orchestration

• I attended 2h workshop with a hands on lab
• Set up docker cluster using Docker Swarm
• Went through setting up Kubernetes (did this last year too at OSCON, attending a 3-4h workshop on just Kubernetes)
• Mesos
• Docker Swarm is built-in. Easy. Young. Immature.
• Kubernetes is a more complete orchestration framework that goes down to the infrastructure level too.
• Mesos is the oldest and most trusted orchestration tool. Difficult to set up.
• A lot of complexity to our architecture
• What gains do we wish to achieve?

We can still make use of some of the plumbing

Some of container related technology could become useful for us.

etcd

• Send updates or messages to all servers
• Guarantees data integrity

Make a container communicate through a socket

• instead of a port locally
• don't need to give the container network access
• comprised containers cannot then call out "home".
• standard kernel feature: to map a port to a socket
• instead of doing the NATing into the container
• can be easily wired up through systemd, where you say that a given socket shall correspond to a service. (my-web-server.socket and my-web-server.service)

An interesting acquaintance: CoreOS

• /usr is mounted r/o
• many files in /etc are just symlinks to /usr
• CoreOS has two /usr partition, A/B, for easy rollback upgrade, not fully utilised yet.
• idea is that a user can rm -rf /, reboot the machine and have a fresh CoreOS environment.
• no framebuffer, wireless drivers. But it's possible to use as a desktop (one person has done it), but that's not their focus. It's a server distribution.
• a distribution optimised for running containers, including but not limited to docker.

rkt instead of dockerd

• Docker is heading in a scary direction: dockerd starts all docker images using runC.
• If dockerd dies, so does the docker instances
• rkt can run container instances without this limitation
• With runC you get Docker running as independent processes that cannot affect eachother.

QA

• SuSE QA process is pretty awesome
• OpenQA
• Uses OpenCV for asserting tests

Chrome OS

• built from the ground up to be jail broken
• you are right at the edge of my knowledge