[Music]
good afternoon welcome back
um next up we have uh
project uh
pratik rajesh sampat and gautham chenoy
presenting on cpu namespaces a mechanism
to isolate cpu topology information
in the linux kernel
pratech is a linux kernel developer at
ibm who works primarily with schedulers
in energy management but also on
container primitives and gautham is a
kernel programmer who has been working
on the kernel since 2006 and has
contributed to hot plug process
scheduling rcu lock depth cpu idle and
other things
so please welcome them as they present
on cpu namespaces
thank you thank you uh hello everyone i
am pratik sampat and i work for the
linux technology center at ibm uh with
me i have gotham chennai who works with
the kernel team at amd and today we're
here to talk about the isolation of cpu
information in the linux corner
the agenda for our talk really is uh
that we'll first highlight the question
of what is the purpose of cis and proc
in the world of containers and are there
any implications of exposing this
information next we'll talk about some
existing solutions that help mitigate
this problem we then present our staff
at a solution uh called the cpu
namespace uh some experiments around it
and finally we pose questions around the
challenges that are that exist in this
space
all right motivation
so a short introduction to csfs is that
it's a file system that is used to
expose kernel information to user space
and often it's about kernel subsystems
and the hardware that runs it
applications today can look at this
interface to determine system resources
and they can make decisions based on
that such as allocating resources like
memory and spawning threats
take an example of containers
containerized applications can be
restricted by c groups cpu set however
they can be unaware of these
restrictions and can still look at
traditional interfaces of sys and proc
to make decisions uh about their
applications
and now this problem is not only uh
constrained to the realm of containers
but uh but outside containers as well if
you take say task set like the system
called sched set affinity you can use it
to set cpu restrictions on applications
but applications may still choose to
make their decisions based on looking at
this assistant proc
so the question that arises from all of
this is that
what does cis and proc really mean in
the context of container restriction and
second what are really the implications
of exposing this information when
applications can only use a very small
side of it
for the scope of this discussion we will
stick to the implications of cpu
resources however we will also
periodically call out other potential
problems such as memory as well
so coming to the first implication uh
restrictions can be set through
interfaces like c group cpu set as i
have already said however there are
multiple interfaces which display cpu
information and these control and
display interfaces may be disjoined from
one another
for example uh you can see that if a
host system has about 128 cpus and it
spawns a container with its restriction
set to 32 to 35 cpus now this
restriction is set through the c group
fs itself and uh and viewing this
interface within the container uh yields
uh the right 32 to 35 cpus a task within
this container if it calls the sched get
affinity onto it uh also gets the right
uh view or the right view of the
restriction that has been applied to it
however if it looks at the proxlat stat
which is generally for the load
statistics and applications like uh top
and edge top use it you will see uh you
will see data about all the uh cpus and
similarly for cis fs as well uh you if
you look at this device's system cpu
which is normally used by nproc and lscp
kind of utilities and they also show
that uh that 120 you know eight cpus
exist on this system
another uh you know we will talk about
the potential impact in terms of
performance of of what this really means
in that in that context in in the coming
few slides
uh no next coming to the to another
implication of fair use
so
when an application that is running
within a container is restricted to some
resources should they still be able to
see the entire system resources
and
if so can this knowledge be potentially
misused in any way
now could there be a no user that can
schedule workloads across sockets in
such a way that the bus is now flooded
and other container tenants now
experience a slowdown or could a user
now identify its vicinity from a
peripheral such as a gpu and schedule
themselves closer to get an undue
latency advantage i know compared to the
rest of the you know uh compared to the
rest of the workloads
so
uh so are there any solutions that exist
today that can help mitigate this
problem of inconsistency of information
well turns out there are there are a few
and we've highlighted about it about
three of them so one of the the most
obvious solutions uh out there are hey
just look at cfs so if you need
information about the restrictions that
are imposed on you just look at the
interface that imposes those
restrictions in the first place and
that's a very strong argument to make
however a lot of these applications
legacy or otherwise rely on traditional
interfaces lexis and brock and uh and
asking all these players players to
really change the way they uh they
they interpret information it may be a
difficult task
another problem is also that you know uh
now these applications also need to
interpret newer concepts like uh period
and quota which are cpu restrictions in
time uh and but they're they're used to
interpreting the information in terms of
space and in terms of cpus and threads
uh and how how does this information
really need to be interpreted is is is a
is a difficult uh uh thing to
say
uh and lastly while c groups can be used
to extract information
in principle they are a control
mechanism for the host rather than a
display interface inside the container
and and there really doesn't uh
there really isn't anything that stops
them to change this interface in the
future and uh and maybe like a vc group
v3 comes out and then the applications
have to go uh change the way they look
at information all over again
um uh so there are some user space
innovations in this area as well uh
there's a user space solution called lx
cfs uh by the by the lxc uh current
containers and basically what they do is
uh
it's a user space file system that bind
mounts over the existing system proc fs
and they basically provide consistent
information in accordance to whatever
restrictions that were set on these
applications they're essentially trying
to fake this information in in a way
right
the advantage of is a user space
innovation like this is that it's a very
light easy to use user space tool and we
have we have seen a few articles uh
where it's currently being used uh or
currently being described by uh by
google and uh alibaba as well
uh and and if a user space innovation uh
exists out there to solve solving this
problem this kind of bolsters our
confidence that this problem uh you know
exists in the first place
but a problem with user space
innovations are that they need explicit
setup for applications and they need
explicit setup for applications that
experience this effect of incorrect
information in the first place so uh so
a lot of times inconsistent information
is not going to really crash your
application it's rather going to give
you a a performance hit or it's going to
give you
give you a problem that that is somewhat
of a silent failure and first
identifying that you are facing this
problem in the first place and then
identifying that alex cfs is the right
solution for you can be a bit of a
hassle
uh lastly uh there is
an effort uh for an rfc patch set which
was posted a few months ago which added
uh a proc slash self slash mem info as a
new interface which respects the c group
instruction restrictions and provides
this consistent information uh
for for applications to see
and this is a very good solution as it
introduces standards for exposing and
interpreting this information it is also
a very clean interface as it does not
meddle with the current established
system proc interfaces and
and it kind of
keeps the sanity of of those interfaces
intact
however um just like c group fs it also
faces the problem of a problem that a
lot of applications still look at cis
and clock
instead of c group and the motivation to
use yet another interface may be a
little bit low uh there was a comment
which kind of highlighted of the same in
the same path set as well uh which which
i have you know linked down
on these things
so
what if we could have a solution that
kind of took some good points from all
the three solutions and and built
something around it uh so what if we
could present information about our
restrictions uh we could present them
consistently with all these existing
interfaces of cis and proc
and we could introduce standardization
of how to expose and interpret the
solution by an in-kernel solution
introducing cpu namespace
so we basically try to isolate cpu
information for each task based on
whatever restrictions that have been
applied to it why are
the control and
display interfaces
and and and we make that consistent with
the rest of the interfaces as well so
basically we isolate the cpu information
by maintaining a
translation of these cpus within the
name space and from the namespace pcpu
to a logical cpu we have scrambled the
cpus to help mitigate the problems of
the knowledge of topology that we have
highlighted in one of our previous
slides i'm not an expert here but if it
helps that uh then then that's great uh
in our proof of concept we have
scrambled this map just to show that a
discontiguous cpu numbering works just
right out of the box and if the map if
the mapping does matter we can could
change it to something like a zero to
end map or even make it as a one to one
mapping right
but in summary just like a pid name
space when you view a task cpu resources
within a cpu namespace we can get an
isolated view of the restrictions that
it is bound by
and viewing the tasks resources outside
this namespace will yield the
translations of these
so for for example
if you look if you look at the diagram
which is without the cpu names which is
basically uh the one that the diagram
that we saw uh in one of our earlier
slides you can clearly see that uh you
know procensus uh no variant inc very
consistent in information when there was
a cpu set restriction applied to it uh
when we look at it from a cpu namespace
point of view while when we add the cpu
namespace layer to it we can now see
that a scrambled map is first generated
and c group fs viewing c group fs within
this container now yields a translation
of 5 12 21 23 and similarly all the all
the system calls proc fs as well as sfs
gives this exact view to the system and
which is in accordance to whatever
restrictions that were applied to it of
course uh this this namespace cpus is
going to translate to these real cpus so
where 5121 2023 is going to translate to
32 33 34 and 35.
so basically this is the design of what
a cpu names faces uh the reference link
is also uh is also on the top of this
slide uh where this is where we have
posted the patches and we have had quite
a few interesting discussions uh on on
this as well which we will discuss
further uh in the in the coming slides
so um
well we we showed that there is a
problem of inconsistency of information
and we kind of showed with our solution
that we kind of
have a a a solution that elegantly
brings together all these interfaces to
solving this problem
but is there any performance benefit to
doing this as a spoiler alert but yes uh
in in our experiment we have done this
on an ibm power 9 machine only 44 smt4
containers on 176 cpus uh but these are
experiments are architectural agnostic
uh
as well so our experiment is as follows
we benchmark nginx which is a http
server and which is a fairly modern http
server family at and we benchmark it
with a multi-threaded workload called
work which is a very simple http load
generator
now nginx is configured with a worker
processes auto and this auto really
helps us to enable this application to
manage resources based on the system
configuration that it sees uh nginx
container is configured to cpu set to
four cpus and the work benchmark spawns
about 500 requests in 30 seconds or four
threads which is enough to saturate uh
the
the resources of uh of four cpus that we
have bound our nginx container by
uh on the right hand side if you can see
that there is a small summary graph of
of the percentage of improvement that uh
you really see uh with this experiment
so we have we have a few metrics of
measurements uh first is the is a memory
usage at the initialization and that
when you compare it with a vanilla 5.14
kernel you can see that it is dropped by
about a 91
uh similarly memory at peak drops at
about eighty nine percent uh throttle
drops by about seventy four percent
latency drops by about thirteen percent
and requests per second uh improves and
or increases by about twenty point seven
three percent um we can clearly see that
there is a net net uh improvement in in
providing consistent information
to two applications
but uh why is that uh in the next slide
i aim to just to answer just that and uh
and basically uh if you look at the top
most left side of it which is the pids
you can see that uh uh
on a vanilla kernel it spawns about 177
uh processes whereas on a cpu namespace
kernel just spawns about five 177 is not
a random number uh as we have seen
before uh the system is configured to
have about 176 cpus so basically it
spawns 176 worker threads plus one
master thread and in the in the case of
cpu namespace it spawns four worker
threads plus uh uh one master thread and
uh as a result of that or as a
consequence of that uh the memory usage
in the in the vanilla kernel is pretty
high because now it needs to allocate
memory to to keep track of all these uh
extra pids
uh because of that also the throttle is
pretty high and it's it's as high as
about 97 of throttling and this is also
because that now we are trying to run a
very trying to uh there are a lot of
these resources were trying to contend a
lot of these pids that are trying to
contend for the same exact resource and
that restored this resource is quite
constrained and therefore there's there
is going to be quite a bit of throttling
but even though there is throttling they
are essentially trying to do the same
thing uh do the thing of the same task
and uh and there shouldn't be a lot of
hit on the performance in in that case
right uh but that is not true uh
in that case you're going to be hit by a
scheduler overhead such as context
switch and and that basically kind of
shows us that the requests per second or
the throughput
is quite higher in terms of our cpu name
space and and our latest in c also uh
latency goes lower which is uh where the
lower is better uh
in our cpu namespace as well and this is
this is all because of these extra
overheads that now that the application
needs to really uh uh
or the kernel really needs to uh you
know handle
so
we we kind of showed that there there is
some benefit of doing uh uh of running
these uh of giving consistent
information uh now the proof of the
pudding is you know really eating it so
let's let's show you our implementation
of how we uh really do cpu name space
and uh just so that you have an idea
that uh uh the idea that how it really
works in the linux kernel today so as
you can see there are two tabs um or in
the terminal uh the right hand side is
basically the initial cpu namespace
which is the host outside a container
and the left hand side is the cpu
namespace a um just a is an as an
acronym
within a container like docker right
on the left hand side we basically start
a very simple ubuntu containers uh with
a batch prompt and we we name it say p
example
and uh and we we run it unconstrained so
when we do an ls cpu we should see you
should see all these cpus that that
exist on this system
um similarly if you should do a cat of
you know cpset.cpu's in the c group fs
directory you will also see the entire
list of cpus now when we try to restrict
this container's cpu set and we will try
to restrict it with the docker update
command and we'll restrict it to cpu set
zero to three uh in this case now when
we do an ls cpu now you can see that
there are only four cpus that uh that
exist on the system and uh and the
online cpu is a scrambled map of uh of
four uh no randomized uh cpus of course
uh there are there are a few things like
the new newman node zero and new one
node eight which is not uh really
virtualized uh in information or or or
it's not really uh
shown in in its
true scrambled map but uh this is a
proof of concept and i know we aim to
have a a more fully fledged uh uh
we aim to fully fledge fledge this out
um so next you know same thing if you
try to look at the c group cpu set
interface basically it shows us that
this information is also consistent with
whatever ls cpu saw and it is uh it is
basically just these four cpus in a
scrambled
way
uh next we try to spawn
a stress on one of these available cpus
and we'll test it to uh one of our
available cpus that is cpu 17 and we
stress it to minus c1 which is just to
know cpu is to one cpus and we can do a
task set minus cp to that task to see
whatever the current affinity of that
task is and we can clearly see that it
is uh now 17 which means that the system
call is also coherent
coherently showing this information
if we do a top on to on top of this uh
you can see that now it according to
this container it has only four cpus and
cpu 17 is what is uh consuming a 100
percent of you know utilization
uh on on the similar side if you if you
try to look at the same information from
outside this container if you try to
view this tasks affinity from outside
this container we can do a top and you
know we can see what what consumes
hundred percent of cpu time that is cpu
zero uh and if you try to do this with
this with the task set minus cp by
getting the uh no ps minus ef of crep
stress uh where of course you're not
gonna get like two tasks one is the
parent and the child so parent really
spawns the stressor and we can really
look at either the parent or the child
to see where it is really uh
bounded by uh you can clearly see that
the task set minus cp is really
showing us to be that it is bounded to
cpu zero next
if we try to change this affinity to say
cpu 2 which is which is again in the
permutable limit of whatever cpu set
restrictions we have applied to it uh so
if we tar set it to you know cpu 2
we can we can try to see how this uh
you know varies this information uh in
in the top command uh on both uh within
the container and outside the container
so with this outside the container you
can now see that cpu 2 shows 100
utilization and within the container it
sure it has now migrated from server cpu
17 uh to cpu83
so so that is pretty much uh you know
what it is and uh
in
in the next slide we will talk about a
few challenges and and know what is the
future of of isolation of of information
so while
the solution uh works in a way uh but it
is not perfect and and there are a few
challenges associated with it one of the
most uh foremost challenges that that
exists with this is that until now name
spaces and c groups have been fairly
disjoined from one another uh cpu name
space kind of breaks that and without
cpu or cpu set c groups the cpu
namespace itself loses its meaning and
that brings up the question really that
if
that is a time to now
define interactions between spaces and c
groups uh in in a in a you know it's
reasonable amount of way and and what
does you know what do containers really
mean from that point onwards
um another uh challenge that that exists
with our current design is that the
current design only addresses
restrictions in space which is you know
cpus and threads and pids and so on but
not uh time and not pids by the way uh
the containers also frequently use cf
spirits and quotas and then it's you
know fondly called millicodes in the
kubernetes world and in the cloud world
uh
so how does this information now need to
be exposed for this these restrictions
it can be as simple as some defining
some standards that say that okay if
this is the ratio of period in quota
this is the this is the cpu's worth of
runtime but then uh is ratios the only
factor
for that or not
um
lastly while cpu namespace mitigates
potential misuse stemming from knowledge
of topology by obfuscation of
information the topology can still be
roughly figured out if you know with ipl
latencies to determine who's your
sibling or who's uh or which core is
really far away uh
from
you so
that's that brings us uh to our last
slide of future uh where uh you know the
intention of of these uh of of this
presentation is to spark a discussion on
the problem rather than be the new and
end all of all solutions
if the solution is for applications to
change and look at c group fs or any
other interface there are a few exciting
discussions that are happening around
exporting more useful metrics to entice
applications to change and these were
discussions were happening on the uh uh
uh on the patch set that i had posted um
if the solution is an external user
space program bind mounting like custom
system proc fs then should that be the
norm for the future as well now should
uh sure should user space innovations uh
be encouraged further or should we start
looking at uh you know defining and
standardizing uh a lot of these things
uh for you know within the linux kernel
itself and finally is it a time to you
know
finally define
uh a container as a first class object
uh in linux
so that was pretty much all the
questions i had this is a legal slide
for uh
for attributions and finally some
references
so thank you for uh for this i will look
at if there are any questions around it
and uh
and i will try to i'll try my best to
answer them
yes the first question yeah the first
question is does the new map of the name
spaced cpu still correspond to the
hardware so in the current
implementation that pratik has it does
not so we have not taken that into
consideration but the idea is if we
still want to you know do this whole
permutation thing then we can restrict
the permutation to uh the new uh node so
that uh you know we are consistent at
least with respect to numa but but
if we are going for a random permutation
we will still not be consistent with
other topological information such as
last level caches and
smt siblings for instance
uh yeah the next question is uh why
scramble the cpus rather than just
showing you know zero to four or in
general i think zero to three is what
meant what was meant here since there
are four series now we can we can easily
do that i mean we pick the most uh
generic permutation that one could think
of but it is very easily possible to you
know redefine this map to be just you
know zero to n or or if if that is not
preferable we can just have a one to one
map where you know whatever the host
sees it's the same cpus that you know
you see inside the container as well so
it's just an implementational detail
it's a it's a matter of choice
so i think you want to add anything to
that
no no i think you're absolutely right so
it's just uh it's just a matter of
implementation details and uh and that
was pretty much the first thing that i
implemented uh so like i just also said
before right this could easily be as a
zero to three map or or a one-to-one map
as well right it should show you
whatever cpus that you really have
okay uh
there's one more question
which asks is the idea that you would
scramble the cpu ids
node ids together
in some way where the cpu is on the same
humanoid will still appear on the same
demand node inside the container
yes that that that is the eventual idea
but but it is not present in the current
implementation so current implementation
for instance if you say take two cpus
five and say 130 which happen to be
these are real cpu numbers which happen
to be different um ids when you view it
inside the container they may still
get you know some numbers like
10 and 11
which
inside the container you know mapped to
the same
same you my
id so
that is something that we need to fix
when we are setting this permutation so
currently i think we are taking all the
cpus and then we are
defining a permutation at the start of
the container that could be anything to
anything but then we can partition these
and have this permutations within those
humanities that's that's possible
there's one more question cpu scrambling
hides topology information from the
container but doesn't that mean that
apps that try and optimize their access
patterns for pneuma will actually be
anti-optimized for memory access that is
true that is true and like like i said
and it's it's something that we have not
taken care of in our current application
but it's not very hard to do that
any other questions
any comments because what we are really
looking for is feedback since there are
user space solutions there are
alternative you know kernel uh solutions
what would be the right way forward uh
you know to provide a consistent
information to
applications running inside campaign
what user space pieces are responsible
for setting up the cpu mappings
i
am assuming that this question
you know is restricted to our
implementation of
cpu name space
so pratik you want to take that what
user space pieces are responsible for
setting up the cpu mapping
um so responsible for cpu mappings is
nothing much really we we expose this by
uh by a clone system call and uh we have
just defined a new system called uh or
we just defined a new flag in this clone
system called whereas if you if you call
clone new cpu uh you're you're basically
going to get a new cpu namespace and uh
these these are these are automatically
going to be mapped at the start of
creating this cpu namespace
so yeah in addition to i mean that that
clone is something that in our current
implementation
it is set by default whenever uh pit
namespace is asked for so right so
that's a hack uh but apart from that
there is nothing that the user space
currently needs to do for our
implementation in order to get you know
this restricted information because that
is exposed through rockensis and
utilities such as stop anywhere read
this information so the idea is to
present consistent information
proxies
see group fs and
system calls such as set and get
affinity
yeah it's a hack only because that you
know we want to use your all these
pre-made utilities like docker uh to
really
get going of course you you you can you
can write your own c programs and call
your clone system calls and
get the same thing up and running as
well
all right uh
yeah we still have a minute we can we
can take any questions or comments
yep we've still got uh about one minute
so if anyone has any has one last
question to put in
now we've got one coming in
the question is
once the cpu name space is unshared
how are the additions or removals or
renumberings of the cpus controlled
okay
uh i think you want to take that so
yeah sure uh so basically uh uh when
where so these are these are basically
virtual sort of mappings right uh they
don't really matter uh
they only matter in the sense of that
namespace itself and when you when you
unshare it uh those those mappings just
go away and uh and i and uh of course
the apple the
the the tasks that are that have been
running in those mappings now are mapped
to the translations or the or the real
numberings uh of whatever uh physical
cpus or or logical cpus in terms of
linux is really really mapped to
so so they will see the entire system i
mean if you add a new cpu uh you will
they'll see the you know the permutation
that has been assigned to that new cpu
when you remove a cpu for instance uh uh
then that number goes away and if you if
you unshare the entire namespace i think
the current implementation will give
whatever the host will see so
just get rid of that recommendation
well
i think that brings us to the end of our
time slot so thank you very much to uh
prospect and gotham for their talk
um
and that brings us to afternoon tea uh
so we'll be taking a 30 minute break
until the next talk which will be uh
alice farazi uh talking about merging an
existing framework into kernel ci
that will be coming up at 3 40 pm
see you all then have a good afternoon
tay
thank you thank you
thank you