[Music]
hi everyone today i'd like to talk about
the persistent memory plus rdma new age
remote device
my name is yangxiang i'm a software
engineer and 1950s nanda in china
and have been working on linux and
related oasis for six years
i became a maintainer of linux test
project as the end of 2018
currently i'm focusing on processing the
memory and rdma
this is the agenda of my presentation it
includes five parts
i will explain why persistent memory
with rdma
and the show new specification of rdma
for remote p memory then
i will share how to implement a new
specification on soft rc and live ib
works
and then i will introduce remote
persistent memory access library
finally i will do a conclusion and share
our future work
okay
let me start from why persistent memory
with rdma
what is persistent memory persistent
memory is a high performance and a
better addressable memory device device
which resends on the memory bus
p-member is the short form of processing
the memory
it has many advantages
like for example data is noto volatile
after power interruption
it has nearly the same speed and latency
of drum
it is cheaper than dram and provides the
larger capacity like ssd
user process can access p member in four
modes
fa stacks d redux sector and row
fa stacks and dvdux are good for
improving the performance
because they are decided to access
pmemor directly
certainly it's faster to access local p
memory in either fsdx or dvdux mode
however modern id system and the service
needed to transport the data from or to
remote p memory such as distributed
database distributed file system
key value storm and so on
traditional tcp became the performer's
spotlight
due to a lot of redundant overhead
look look at this figure
for example
copy data between user space and the
kernel space
package data by the software tcp stack
of operating system
in this case we need a faster access way
to remote p memory
rdma is a good solution to access remote
p memory
rdma is the short form of remote
director memory is success
it is a technology that enables
contributors in a network to exchange
data in the main memory without
involving operating system of either
computer
it avoids redundant overhand because of
its advantages
for example
provide a zero copy between kernel space
and user space
bypass the host systems software tcp
stack and the move data results cpu
involvement
by dma engine
is it good enough to access remote p
memory by traditional rdma
note really
rdma has two problems for accessing
remote p memory
the first problem is no guarantee of
data persistency
look at this figure
responder returns acknowledge as soon as
the rdma right reaches the remote unique
the return data will be lost when it has
not been saved into remote p memory and
the remote system is powered down
for data persistency
we need a way to confirm that the data
is actually written returned to remote p
memory
the second problem is no guarantee of
data consistency
two-phase commit is widely used by
distributed database
for example an application writes a
block of data by two-phase commit
like the following figure
step one
write a probe of data into remote
p-member step two mark the data as a
valued by updating an edge batch value
atomically
another application can know if the data
is valued by reading the 8-bet value
rdma doesn't provide an api for atomic
write yet as step 2
so we need a way to update and add
better value atomically
there are two ways to solve these
problems
the first way is to introduce new
specification to extend rdma
it ends rdma flash to guarantee data
persistency and ends are the anatomic
right to guarantee data consistency
the second way is to make new up layer
library it not only guarantees the
persistency and the consistency of data
but also
hence the
complexity of rdma and provides a set of
simple api to applications
in addition it will support a new
specification in the future
ok
i will talk about the both solutions and
our effort next
let me show new specification of rdma4
remote p memory
there are two associations to make a new
specification
ipta and ietf
ibta released the vu 1.5 specification
in august 2021
and it defined the new rdma operations
for remote p memor
ietf released a draft in march 2020 but
didn't update it anymore
it also defined the new rdma operations
for remote p memory
intel has showed the overview of ibts
new specification on storage developer
conference
today i'll talk about ibts new
specification
ibts new specification defines new rdma
flash operation
look at this figure
a new rdma flash can flush all previous
rights or specific
memory regions it guarantees that the
data is pushed pushed to global
visibility or persistency
it will send the rdma reader response
wheels with zero size to request after
the data has been posted
on both request and the responder the
rdma write and rdma flash should be
handled in order
ibts new specification also defines new
rdma atomic write operations
look at this figure
a new rdma atomic write carried and
aligned eight bad value atomically it
will send the rdma read response with
zero size to request
after the eight batch value have been
returned
on both requests and the responder the
rdma flash and the rdma tonic right
should also be handed in order
to support rdma flush and rdm atomic
right operations
what must be extended in the stock of
rdma
as as shown in the figure below
the whole stack of rdma needs to be
extended to support new operations
liver ib
leave ib warps library provides rdma api
to applications and it has no to support
new operations yet
currently
there is no hardware on it and related
driver to support the new operations
the next thing i'm going to talk about
is how to implement a new specification
on soft rlc and live ib works
why use soft rc as i said new
specification requires hardware support
usually
v1.5 specification has been released but
hardware vendors need them to make new
unique
it may be a long time and we don't want
to wait the new arnic
for this reason we are focusing on soft
rc driver which is decided to make a
normal nic support rdma
though it is slower than real arnic
user can experience rdma easily
finally we decided to extend the soft rc
and the live ib works
this figure shows the software
stock of rdma based on soft rce
what is soft rc we have to know rcv2
before introducing soft rc
rc v2 is the short form of rp rootable
rdma over converged ethernet
rc v2 is a network protocol that can
transfer every transport header and the
payload through the traditional ethernet
ip and udp headers
if packets are formatted
by rcv2 they can be forwarded by tcp
routers and switchers
soft rlc is software-based rc v2
it produces ib transport header and
inserts it and the payload into the udp
header by software
the red figure shows the
difference between hardware rc v2 and
soft rce
let me talk about how to implement a new
rdma flash process on soft rce
please see the detailed logical of rdma
flash we implement
step 1
when local soft rc process is a rdma
flash request from user space
it ensures all previous requests have
been set by default all ensures all
previous requests have been completed by
a fast flag the fence plug is used to
ensure the exclusion order of operations
step 2 local soft rc prepares a
rdma flash request package by the
following changes
and the new ibop code rcrdmf rush op
code in base transport header
and the new
flash extended transport handle
including selectivity level and the
placement tab
and the specified the address and lens
to flash in rdma extended transport
handle
step 3
local soft roc sends the rdma ferrari
requested package over udp
step 4 remote soft rlc accepts the rdma
flash request package after executing
all previous requests once
step 5
remote soft rlc flashes the specified
ridge and the dips according to the
content of flash extender transport
header
selectivity level defines the memory
region reaches the rdma flash should
apply on
placement type defines the memory
placement guarantee of this rdma flash
step 6 remote soft rc prepares a rdma
flash response package by the following
changes use ibop code rcrdm may read
response only op code in base transport
handler
set arc or not in
arc extended transport handler
step 7
remote soft roc sends the rdma flash
response package over udp
step 8
local software rlc accepts the rdma
flash response package and generates the
corresponding completion
let me talk about
how to implement a new rdma atomic
writer process on soft rc
please see the detail the logical of
rdma atomic writer we implement
step one when local soft roc
process is a rdm atomic write request
from user space it ensures all previous
requests have been set by default
or ensures all previous requests have
been completed by fs flag
step 2
local soft rlc prepares rdma atomic
writer request package by the following
changes
and the new
ibop called rcrdm atomic write op code
in base transport handler
specify the address and the lens to
atomic right in rdma extended transport
handler
and populate an allied eight better
payload
step 3 local soft rlc sends the rdm
anatomical writer requested package over
udp
step 4 remote soft rlc accepts rdm
atomic writer requested packet
after executing all previous requests
once
step 5 remote soft rlc writes the 8-bat
payload atomically
step 6
remote soft rlc prepares a rdma atomic
writer response package by the following
changes
use ibop code rc rdma reader response
only op code in base transport header
set arc or lock in
arc extended transporter handler
step 7
remote soft rlc sends the rdm atomic
writer response package over udp
step 8
local software rlc accepts the rdma
atomic writer response package and
generates the corresponding completion
okay let's go on to the next how to
implement a new rdma flash api on level
ib works
to support rdma flash in ibv process
send
we defined the new ibv wrdma flash op
code to identify a flash operation
and ended the new structural flash to
transfer the information required by the
flash operation
to support rdma flash in ibv4 cq
we defined the new ibvwc rdma flash op
code to identify a completed flash
operation
the following the following code shows
how application
use
rdma flash api
for example
poster rdma flash request by ibv process
and
get the completion of rdma flash by ibb
por cq
how to implement a new rdma atomic
writer api on live ib works
to support rdma atomic writer in ibv
ports ascent we defined the new ibv wrdn
meta atomic write op code to identify
atomic write operation
and take use of structural rdma to
transfer the information required by
the atomic write operation
to support rdma atomic writing in ibv
poor cq we divided the new ibv wcrdm
atomic writer op code to identify a
completed atomic write operation
the following code also shows how
application use are the metatomic right
api for example
post rdma atomic writer requests by ibv
poster sent get the completion of
the anatomical right by ibb pro cq
new rdma operations are under
development is there any available
solutions
remote persistent memory access library
is an available solution
what is a remote persistent memory
access library it is a new library to
access remote p memory over rdma
libre rpma is the short form of remote
pro system the memory access library
label rpma
provides a complete set of api for
applications to access remote p member
lag rpms sender rpma receive rpma write
and so on
it has rpma flash tool first previous
right into remote p memory
it also has rpma atomic right to mark
the previous red valued atomically after
the previous rpma flash had been
completed
it will support new rdma operations when
they are available
intel and of this are main contributors
to live rpma
let's look at the basic api of libre
rpma i will explain the functions of
some liver rpma api
for memory management
we can register memory reading by rpm mr
rig
and the register memory region by rpma
mrd rig
for connection management
we can create a new outgoing connection
request by rpma call rick new and third
an incoming connection requested by
rpmaep next con rig
for messaging we can send data to remote
site by rpma sender and receive data
from remote sender by rpma receive
for remote p member access
we can write data to remote pmemma by
rpma write flash data into remote p
memory by rpmi flash and write a 8-bet
value to remote p-memor atomically by
rpma writer atomic
in lib rpma community there are 11
examples to show how to use various
liver rpma api together
let's look at the example zero five
flush to per
system for details
look at the following example
client use uses drum to register memory
region by rpma mlr rig
server uses p-member to register memory
region by rpma mr rig
server and
clan and the server
established connection and the transform
provides data by several rpma core
functions
with the connection clad can transfer
data to remote p member by rpma write
and rpma flash
currently flash and atomic write is not
supported by rdma soho liver rpma
implementation
how to implement rpma flush operation
liver rpma implemented rpma flush by
traditional rdma reader
look at this figure
requests requester sends a rdma read as
a rpma flash
the rdma reader reads the completion of
previous router automatically
the rdma reader flash or return data
from arnic to the remote p memo before
reading data from the remote p memory
this way is called a appliance
persistency method
how to implement rpm atomic right
operation
liver rpma implemented the rpma atomic
right by traditional rdma right with a
light hbat value look at this figure
request sends a rdma right with a land 8
bad value and a fast flag as a rpma
atomic write
the rdma right waits the
completion of previous flash bend the
fence flag and then writes the value
the rdma right needs to be flashed to
remote pin member as well
unfortunately the rdma write has to wait
all previous read due to the first flag
one more necessary consideration for
rpma flash operation
intel ddl is a key feature introduced on
the intel xuan e5 professor and the
inters1 e7 professor v2
as the interest document mentions ddl
makes the professor cache
the primary destination and the source
of our data rather than main memory
helping helping to deliver increased
bandwidth
lower latency and reduce power
consumption
with ddl traditional rpma flash using
rdma read can only flash data to the
last level cache of cpu
so remote applications need to trim the
data to p member by themselves
rpma flash has to consider ddio
how to implement rpma flash operation
with the ddio
in this case
liver rpm implemented the rpma flash by
traditional rdma send and rdma receive
look at this figure
requester passes the address and read to
flash to
respond by rdma sender
the rdma center waits the completion of
previous writer automatically and then
flash the written data from arnic to the
arc
responder flash or written data from
area c to p memory according to the
contents received
responder notifies the requester the
data has been returned into p memory by
another rdma standard
libre rpma is an up layer library so we
would like to know how the performance
of live rpma is
how to evaluate the performance of labor
rpma labour rpma introduced the liver
rpma dedicated engine to fio so that we
can use a file to do the performance
test
on our environment
a file is a benchmark to test the io
performance
the table on the left shows comma
configuration of our environment the
example on the right shows the detailed
steps to run the file benchmark on our
environment
for example
client builds the latest file including
live rpma engine
it's a reference sample jobs to create a
new job file for the live rpma client
and then run a file with the client's
job file
server needs to do the similar steps
as well
by fio benchmark we go to the band os
and the latency of remote p memory
access based on label rpma and the
source of low copy memory access as
shown in the tables below
compared with the local p memory xs
the performance of remote p memory
access is slightly worse
but i think liberal pma is still a good
solution to accessory multiple member
and it may provide a higher performance
in the future
okay finally
i will do a conclusion and show our
future work
in this presentation i explained the yp
memo with rdma and show the new
specification of rdma4 remote p memory
i also showed how to implement new rdma
operations on soft rlc and live iv works
and introduced live rpma
in the future we will finish
implementing new rdma operations on soft
rlc and liver iv works
and then pushes them into the kernel and
rdma core
we will also make a live rpma support
new rdma operations
thank you for listening to my
presentation
please contact me by email if you have
any questions about this slide
thanks a lot