[Music]
no my hook am i welcome back to lca
2022.
call gel tna i'm joel if you're just
joining us welcome to lca
um we have with us fraser
tweedle who is another fellow red hatter
based out of brisbane uh fraser works on
security and identity solutions at red
hat and today he's going to be telling
us about an upcoming and already
delivered kernel and kubernetes security
features that enable for bitter
container isolation and secure
deployment of systemd based workloads
know my fraser
take it away
so
welcome to my presentation and thanks to
everyone
especially those attending live because
in this particular time slot uh there's
gold in them their tracks
and uh i'm a little resentful to have to
present in this time slot
some of those other prezzos are great
so good luck to them also and let's get
into it so this talk is creative commons
attribution licensed except where
otherwise noted
the slides are available now
and if you're the sort of person who
likes to follow the links
and open up two dozen tabs
to go deep
after or during the prezzo there are
hyperlinks in the pdf
so
go grab the pdf from speaker deck
and you can follow along from there
and i will be available in the chat room
and
in the general
conference chat rooms following my
presentation
if we have questions
that we don't have time to address
during this time slot
so today i will talk
briefly about containers what they are
and container standards
and i will give an overview of
kubernetes and openshift particularly
with respect to their container runtimes
these are huge products there's no way
to cover
even
a sizeable chunk of them in depth so we
just have to focus on this one specific
area of the runtime
then i'll talk about free ipa which is
the application that is the subject of
my team's efforts
and we'll conclude by talking about
systemd based workloads on kubernetes
and openshift what are the challenges
and the workarounds and the solutions to
run them
so what does it contain ask 10 different
people you might get 10 different
answers and they might all be correct
answers because
the concept of a container is really an
abstraction
of over uh isolation and confinement
mechanisms
most commonly when people are talking
about containers they're talking about
some os level virtualization where all
of the
containers or the containerized
processors
share the host kernel so it's the host
kernel running everything but
presenting to the confined processors a
restricted view of the system
and imposing resource limits
non-linux
implementations of os level
virtualization include freebsd jails and
solaris zones
uh it's important to make a distinction
between the container which is actually
the the confinement mechanisms and the
processes running in that environment
from container images so if you're
talking about uh docker files and
building a container
what you're really talking about is
building a container image which defines
the file system contents
intended for use
in or with a container
as well as
metadata
about how that container should be run
such as what process needs to be run
the setting of environment variables and
similar things
on linux containers
usually consist of a bunch of
disparate
security mechanisms offered by the
kernel including namespaces and there
are different kinds of namespaces
namespaces for pids or process ids
mount namespaces network device
namespaces c group namespace and there
are some other kinds
and
each of those kinds of namespace will
present a
restricted view
to the namespace process or processes
for example in a pid name space
uh a process running in that namespace
can see other pids inside that namespace
but nothing outside the name space
mount namespace
has a dedicated mount table for the
processors in that namespace and
manipulating that mount table won't
affect
the mount tables in other mountain name
spaces
and manipulating the system mount table
also
will not affect the
container's mount table
a container may include selinux or
armor
confinement so those are mandatory
access control mechanisms
and a container may use capabilities or
sep comp to restrict
what the
process can do
in terms of system calls so what system
calls
are allowed or what arguments are
allowed in a system call or potentially
modifying the behavior of particular
system calls
now recently the open container
initiative which is uh an initiative of
the linux foundation
has been
uh specifying and developing
specifications for various aspects of
this whole container
ecosystem the one we're going to talk
about today is the runtime specification
they also have specifications for
the image format and other things
the runtime specification is a
specification for a low level runtime
interface
uh it is not linux specific so it um
in encompasses
solaris containers windows containers
even virtual machines treating vms as a
kind of container
within the oci
definition of the container abstraction
oci implementations include run c which
is their reference implementation for
linux c-run is another linux
implementation
and carter containers is an
implementation of the
virtual machine
instantiation of a container
the runtime specification uses a json
configuration format it's quite verbose
but there's a link here to an example if
you want to see that
the general things that
will be specified for any container
include the mounts
the process that should be executed and
its environment
and life cycle hooks for
running commands either on the host on
the container host or inside the
container upon
container start stop creation
destruction etc
and the linux specific
tunables for a container include
capability set so restricting the
capability bounding set for the
container
what namespaces should be used or should
be
created and new for that container
what c group the container should live
in sys controls that should be set
setcomp profile and similar things
so now let's talk about
kubernetes and openshift so kubernetes
is a container orchestration system uh
if you see k8s or cates written that's
an abbreviation for kubernetes
kubernetes
works with a cluster of machines
with distributed configuration and
distributing the
application workloads across the cluster
and it has a declarative configuration
format typically using json or yaml if
you're interacting with the system as a
human
but also there are
apis for most languages you could poke a
stick at in particular go but also
python and haskell
and many other languages
it has integration with
most cloud providers including the the
big obvious ones
all the kubernetes documentation blog
posts
tutorials
guides etc live at kubernetes.io
and
the github homepage is github.com
kubernetes that's where all of the
source code lives
so some of the terminology that we need
to use today and again we're focusing
mainly on this area of the the run time
there are many many many many many other
things that we have no time to cover
today
so a container is
an isolated or confined process or
process tree
uh it may
be a windows container a linux container
a vm based container
that's really up to how that workload
will be sandboxed
a pod is a group one or more related
containers
so a
a typical
uh example would be
some http application and then its
database you know apache and mysql
bundled together in a pod as a single
application but the different components
are actually separate containers within
the pod
a namespace is a scope for
for objects and it's also an
authentication and authorization scope
such as for
a single application or team or project
and
the nodes are the machines in the
clusters where the pods are executed
typically there are two kinds of nodes
one class is the control plane nodes
where
the logic that
is kubernetes that manages all of the
networking and pod scheduling
and auto scaling logic all of that stuff
lives in the control plane or the master
nodes they're also called
and then there are the worker nodes
which is where the
business applications running on the
cluster will be executed
on a node the agent that executes a pod
is called kubelet so kuberlet
observes the distributed configuration
and if it sees that
a particular pod has been scheduled to
it or has been
removed from it then it will affect
those changes on that node
it does that by creating a sandbox
sandbox is the
isolation or confinement mechanism or
mechanisms to be used for a pod
one pod one sandbox
all of the containers for a pod if there
are multiple will run inside the same
sandbox
and the container runtime interface is
the interface
that kuberlette uses to talk to a
container runtime
and tell it to create start stop and
destroy sandboxes and containers
two implementations for linux include
cryo and container d
so visualizing this this whole thing
this whole diagram is one kubernetes
node
on the left we have kubelet and kuberlet
talks to a cri runtime
using cri which has a protobuf
wire format
and then the cri runtime will do
something
to
to create and manage the containers as
requested by kuberlet
so if we instantiate the abstract cri
runtime as cryo now
ben cryo
is a program that uses
an oci runtime
to manage the containers now you can
plug in different oci runtimes so we'll
go a step further and we'll say well
um we can use run c for that and so now
we have kind of this fully instantiated
uh
container runtime setup
uh that is one way that you could do it
in a kubernetes node different
distributions
of kubernetes will use different
cri runtimes and potentially if they're
using an oci runtime
it might be run c or it might be c run
or or it might be something else
i mentioned that the
definitions are declarative
and so here's a very simple pod
definition
we have the kind field which says what
kind of object it is this is this is the
yaml um
serialization of this data but uh you
know you could work with it with json or
many other formats
in the pod spec we have a list of
containers
in this case
the list has one container
and we have to specify a container image
which will be a reference to some image
registry where the cluster can pull the
container image from
and we may also specify the command to
execute
environment variables to set you don't
always need to set these because
these
may have default values specified in the
image metadata
but you can set them in the pod spec or
override them if you need to
open shift
also called openshift container platform
or ocp
is red hat's commercially supported
enterprise container platform it's based
on kubernetes
there's also an upstream distribution
called okd
the latest stable release is version 4.9
and it's no coincidence that in the
diagram previously i used cryo and run c
as the example because those are the
components used in openshift
all existing kubernetes terminology
applies in openshift and there are
additional concepts
in openshift
with respect to the runtime and pods
the two that you need to know about
today are projects
which extend the namespace concept with
additional
attributes
and metadata and security context
constraints which are policies
that affect the se linux context of a
pod
the setcomp profile the capability
bounding set the user ids
that the container can run as and
similar
similar mechanisms
so the openshift runtime environment
today uses se linux for confinement
it creates sandboxes with a set of
namespaces including a unique c group
namespace a pid namespace mount
namespaces
uts namespace which allows setting a
different hostname different kernel view
name and some other attributes
and network devices
each project in an openshift cluster
gets assigned a unique user id range
and by default any container
that is part of a pod
within that project's namespace
must run as a user id from that range
the range is something large you know it
might be
1 billion to 1 billion and 10 000 or
something like that
and it's unique per project in the
cluster
these restrictions can be circumvented
via the run as user
property which is part of the pod spec
you can request that it be run as a
different user
but in creating that pod
you also have to do that via an account
that has permission to use sccs that
allow
that pod to be run as
root on the host or as some other
low value user id on the host rather
than
a uid from the assigned range for that
namespace
this is usually a very bad idea
because
if you're running as
root on the host even with these other
kinds of confinement
if you break out of your se linux
confinement or you can escape your mount
namespace
uh
your root on the
node
uh basically your node and your cluster
are owned
so
it's uh not something that we want to be
doing especially as
people working on and developing a very
security minded and security sensitive
project okay what's that project free
ipa so free ipa is an open source
identity management solution
that means that you can define your
users groups and services
in your organization it provides
authentication mechanisms and lets you
define and enforce access policies
free ipa is made up of a lot of moving
parts
several of which are
very large
mature independent projects in their own
right
so these components include 389ds which
is our database and ldap server
mit kerberos which is an implementation
of the kerberos authentication protocol
we have an http api and also a web ui
and these
are
provided by an application running
behind an apache reverse proxy
we have dog tag pki which is an x509
certificate authority sssd is a client
component
that provides authentication mechanisms
and user lookup facilities
to
applications running on an enrolled host
and
those those are the main moving parts
but there are others
free ipa is available as part of rel and
so is commercially supported as part of
rel and we have
fedora where we do all of our upstream
development
features land there first usually
and we have community support
freeipa.org is the website if you want
to learn more
why do we want to run it
on kubernetes or openshift well
as
organizations move more applications
onto container orchestration systems
like openshift
then
they may also wish to have the identity
management system providing the identity
services that those applications need
running alongside those applications on
the cluster
the cluster itself also has identity
needs
you
need to
have users and groups
within the openshift cluster itself and
you might want to tie that in with your
identity management system for your
organization or you might want to deploy
your identity management system for your
organization
on an openshift cluster
also for access to the nodes directly in
openshift everything
all the nodes are red hat coros nodes as
previously atomic so they're immutable
systems and
currently the way you would log onto a
node is via a debug shell
or via ssh but there's a single user
account
that's not really
adequate for
organizations in highly regulated
industries like telcos banking
government etc
so there is actually a need for
proper identity management and access
control mechanisms
for
logging onto the nodes of the cluster as
well
finally there's this idea that you might
want to offer identity management
systems as a service
so a service provider which in the
future may be red hat maybe someone else
maybe multiple companies uh offering
free ipa as a turnkey managed service
where an organization says yes we're
going to need
free ipa for our organization
i have an account with a cloud provider
i can see it in the service catalog
you know click ok deploy
and a short time later
the identity management system is set up
and we can give the keys to the customer
and typically these will be deployed
on the service provider's own
infrastructure managed by the service
provider and co-tenanted so multiple
different customers
with their
application instances living side by
side on the cluster
so if we want to start doing this sort
of thing
we need containers for free ipa we need
to
put this stuff in a container so we can
run it in a container orchestration
system and our current approach is to
encapsulate the whole
rail or fedora based system
in a container
wrapping everything up together where
pid one the container's entry point will
be system d
and just as if you were deploying on a
vm or bare metal system d will start
and monitor all the services that need
to be brought up
uh
on that
application instance
this is
a very stark contrast from
a microservice architecture or
you might have heard the buzzword cloud
native
uh
where all of the
main components of the application would
be running
as separate containers within within a
pod or maybe even in separate pods uh
wired up to talk to each other to do all
the things they need to do but all of
them running uh separately rather than
together in a single sandbox
uh we're doing the exact opposite we
call it a monolithic container i don't
know if there's any
better or established terminology for
what we're trying to do
i think there might not be because
usually when we talk to people about
this
there are howls of dismay and
why would you do it like that and that's
not how you're supposed to build
applications
but but we have
reasons to
avoid doing it if we can
first of all it's a very big upfront
engineering effort to re-architect free
ipa to be cloud native to to break
everything up as i mentioned a lot of
the components of free ipa are big
legacy projects in their own right
making all kinds of assumptions about
how they're deployed
you know what users they're running as
so
and many of these projects
red hat
is just a contributor to among many
we don't steer
all of the projects that we actually use
as part of free ipa so
it's not only an engineering effort but
also a political effort to make the
changes that we would need
to do this
furthermore
we would face
an increase in ongoing costs
as we support free ipa with two
completely different application
architectures why because free ipa is
available in rel we commercially support
rel
and
the existing releases the near future
releases are going to use the status quo
application architecture and deployment
paradigm
so
we need to support that for five years
minimum probably realistically more like
10 years
so we
face significant ongoing costs
if we have to re-architect free ipa
and
basically build it in a completely
different way
for the cloud
now if we were starting from scratch
today it's a no-brainer of course we
would do it in the cloud-native way
but free ipa is an old project things
were done differently back then it's not
anything to criticize it's just a
reality
of
of where we're at as a team and and the
business
uh consequences
for the different decisions that we have
to make
so to do what we want to do
unsurprisingly there are many challenges
some of the main areas are the runtime
and that's what i'm going deep on in
this presentation some challenges with
volumes and mounts and um
dealing with uh
mounts that uh not necessarily owned by
root in the container and
um yeah just various assumptions about
the file system that that don't
necessarily hold
in a cloud environment
uh and ingress so that's getting traffic
to the cluster free ipa uses some uh
uh
a lot of applications today are just
http and that's it well free ipa uses
ldap and kerberos we use srv records for
service discovery we have services that
use both tcp and udp
to communicate
and
i've got some links there to some blog
posts where i have
written about some of the
challenges that we have with ingress but
that is a whole other talk
maybe next year
so for the runtime what are the
challenges and what are the workarounds
and solutions
the first one is that systemd and other
components in free ipa and you can
abstract this to any kind of
quote-unquote legacy application that
you might want to bundle up
in a monolithic container and run on
kubernetes or openshift
so in these situations for these
applications they may be expecting to
run as root or some other specific uid
and by default
on
kubernetes or an openshift cluster
that's not going to happen it's going to
by default try and run your application
as some unprivileged uid
so a possible solution to this is
username spaces yet another kind of
namespace that
that abstracts
and constrains
the hosts or the parent username space
and
maps those
maps a range of host
user ids to
a range of container uids potentially
starting at a different number say zero
root
so this is supported in the oci runtime
spec and oci runtime implementations
and in
cryo the version that shipped in
openshift47
there is an annotation based username
space feature
so
via a pod annotation and assuming the
cluster is
configured to support this behavior or
to enable this behavior then you can
cause your pod
to be
executed in its own username space
at the moment this requires the pod to
be admitted to the cluster
by an account that
can use the any uid scc or a similar scc
that allows pods to be run as root or as
privileged users on the host
now the workload itself will not run as
those users it's a work around because
the
kubernetes api and the kubernetes data
model
have no awareness yet of username spaces
so visualizing this
you have a host username space which is
0 through 2 to the 32
and
some
slice of this namespace
can be mapped to a particular
process which may be a container so here
we have container a inside the container
there's a range of user ids 0 through
five three five
so if you're running as user two hundred
thousand on the host inside that
username space that will appear to be
root
um or you can think of it vice versa so
for contact for container b
the process that within the container is
running as uid 0 is actually in the host
username space running as user id
26536
uh once your cluster is configured to
support username spaces to actually uh
enable the use of this feature or opt
into it in the pod spec you need to
specify a couple of
annotations um
so this io.openshift build a true this
just
says to the container runtime
i am a container i want to run in
builder mode and in builder mode
username spaces is enabled
and then you also have to specify the
user ns mode annotation which says okay
how do i actually want
to
generate the mapping of host user ids to
contain use user ids one of these modes
is auto that's the most convenient
option and the most secure option
because it just allows you to say
what size
you need
so
i want a uid range
of 65536
and it will find
an available
portion of the host
user id
range and map that to the container
kubernetes itself has no
support for user name spaces but it is a
long running and ongoing discussion
um the first proposal was maybe two or
three years ago and there have been a
few
iterations uh there's a fair bit of
discussion about it at the moment with
this third
proposal number 3065
i still need to catch up on this
this was only sort of over december and
christmas
so i need to get up to date with what's
actually happening in the kubernetes
upstream
uh in terms of username space support
okay so that's user name spaces the next
challenge is c groups so openshift
creates a unique c group for each
container and it creates a c group
namespace for that container which makes
the container c group appear to be the
root namespace the root
sorry not namespace the root c group
inside the container
so when you mount the c group fs
what appears inside the container as
csfsc group will actually be
sys
fsc group
coupe pods dot scope slash coupons
best effort dot scope slash blah blah
blah some container id some deeply
nested a c group for the specific
container
and
in order to bring up the system and do
the service management systemd needs
right access
to the c group to create new scopes and
slices but it doesn't have it so the
solution
well let's modify the runtime
so that it will change the owner of the
c group
to the
user id that's actually running the
container process
so i implemented this uh i submitted a
pull request
and during the discussion and review of
that pull request people were pointing
out well um see run one of the other
runtimes see run already does this and
we were discussing that there were
discrepancies
between what different runtimes were
doing in this regard
the conclusion was well let's define a
semantics for c group ownership in the
oci runtime spec so i shelled that pull
request
went and made proposals to the oci
runtime spec to define this semantics
there was some discussion about it
eventually that pull request was
accepted
um and
after that then we were able to get the
runcpul request merged
so what is the semantics
if and only if c groups v2 is in use and
the container has its own c group
namespace
and the c group of s is to be mounted
read write
then the runtime should change the owner
of the container c group to the host uid
that maps to the
uid of the process
that the
container that the process
that is the container entry point
uh in the user name space
hopefully that makes sense
um tldr
it
will
join
the c group
to
the user that corresponds to root inside
the container's username space
in the common case
it's a little more nuanced than that
what actually gets shown the c group
directory itself so that systemd or the
container process can create new
sub c groups new scopes new slices
and
only the other files mentioned in sys
kernel c group delegate
so that it can move
processes and threads into and out of
the c groups
in
in its secret namespace
uh
why only these ones because
if you turn everything in the container
c group it would allow the container to
elevate its own resource limits and do
other things that it should not be
allowed to do
we need to use c groups v2
and this is required for secure c group
delegation c groups v2 is implemented it
works but it is not yet the default
configuration but it is on the roadmap
so how to configure the cluster in
openshift we use an object
type called machine config
we need to specify kernel arguments to
turn on c groups v2
we need to change a couple of files that
specify
allowed ranges of host uids that can be
mapped to
to child
user id name spaces
and we also need to deploy a
experimental version of run c because
the run changes have not yet made their
way into a release
so we use a systemd unit
to do that and that systemd unit will
run rpmos tree override replace
if the
desired version of run c is not yet
present
on that node
okay
demo time
so i have here a cluster i only deployed
this cluster about an hour ago um
uh so you get um
machine config there is a machine code
thing here that i already deployed this
is one idm-410
so the three slides that i just showed
you that's that machine config
yeah there's there's a lot more
information in the um
when viewing the object uh hang on uh
idm
410
sorry i was showing all machine config
so here's the here's the data that um
that was in the last few slides
uh
i'm going to create a new project
called test
i'm going to create a user called test
i'm going to
assign edit permissions on this project
to user test
to allow to create pods
i'm going to assign it the neuid scc
which is the workaround that we need to
be able to admit
a pod that's using usernamespaces and
running as
uid 0 in the user name space
even though it will map to unprivileged
users
and i will create
a pod with a systemd container just
running nginx
and uh
this is the the same as i showed you on
uh the slide with the example pod spec
for using username spaces in particular
there are those two annotations
so if we have a look at the pod
and particular attributes so the node
name what node is it being scheduled on
and what are the statuses
of the containers
in that pod we can see we're running on
this worker note so worker node a
and the pod is running
and
let's now open a debug shell on that
worker node and have a poke around and
see what's happening under the hood
and we're going to need to copy the
container id or
the prefix of that
we need to route to host on the worker
node
and now we can run
cry control which is a tool for
interacting with
the container runtime
on the worker node
inspect the id
we'll use jq
to
just pull out the pid of the process
now if we do cat
proc
that bit so 65317
uid map
what this file shows us is that uid0
inside the username space of this
process
is mapped to uid 200 000 on the host
it's two hundred thousand because the
modifications to etsy sub uid and that's
the sub gid
that we used
um
that's just the start the allowable
range of assignments
so as the first
container using a namespace that was
created that's where it started
to allocate the
host uids from and the size of the range
is 65536
as requested
now
if we have a look at the processes
running in the container we can do
let's see
pgrep
double dash ns this means show me
all of the processes
running in the same set of name spaces
as
process
six three five one seven
let's pipe that
to x args uh
p s
dash o we'll just have a look at the
user
appeared and the command line
and sort by pid
so this is what's running in the
container init that's system d we see a
bunch of other systemd daemons running
and
then toward the end
nginx the master process and and the
workers for the nginx server
so this is a systemd based container
running in the username space
the last thing to look at is the
c group
so
if we do
oc rsh so remote shell for running
commands inside the container
pod engine x
i think i'm almost out of time so
uh
it will suffice
to
demonstrate that sysfs c group
inside the container
is owned by root
and its
inode
is not
the inode of
cis fsc group
on the host or in the host c group
namespace which is inode one
if you were to
find
this directory from the host point of
view
actually we can have a look here um
uh ai ssfsc group
we can see that
only those files that were allowed to be
shown so init.scope
memory.on.group
the directory itself are owned by root
inside this username space everything
else is no body it's actually root in
the host username space but as that's
unmapped
unmapped uids get interpreted as uid
65534 which is the no body user
okay so that concludes the demo here's a
slide with some links to resources
including the main repo for this project
which doesn't have much in it yet but we
intend to expand that
my experimental run c builds are whoops
at
my
homepage on fedora
team blogs i blog extensively about all
of this research and development that
i'm doing so if you look at the
containers tag on my blog you'll find a
lot of
interesting stuff i have a youtube video
of the demo
with a bit more stuff in it than what i
was able to just show you
and the future
the username space support in kubernetes
itself is an ongoing discussion
on openshift we can use the annotation
based username space support
and we can use sys with that
combined with
the uh
feature to
tone the container c group which
at the moment isn't in an experimental
build
then you can do it as i've just
demonstrated but the question of
official support is still an ongoing
discussion that we're having with the
openshift project with its product
management
and we are looking for allies that is
other teams in a similar boat to us with
legacy applications that would benefit
from these features
being matured and released and
officially supported in openshift
um at the end of the day we might not
get what we want and we may end up
having to re-architect free ipa for the
cloud
and that would not be the end of the
world but uh we're hoping that we can
continue pressing on the path um that
i've been speaking about in this
presentation
and
that concludes thank you very much
fraser i know i've certainly had the the
container root requirement mapping issue
multiple times in the last several years
and as you said ideally you just don't
do that but that's not always the life
we live unfortunately um we have kind of
run out of time so there was a few
questions we'll drop them into the chat
window and
michael
and fraser
hopefully we'll be able to sort that out
as we get prepared for the next speaker
who is going to be coming up very
shortly we've got 10 minutes of
changeover so go grab a drink and we'll
see you back in 10.
thank you