[Music]
okay everybody um next talk is from gal
de la cruz uh
about incident response
thank you very much so um once again
thank you for having me here this is my
second time doing a talk
in lca so it's like really good to be
back so before i start i'd like to begin
by acknowledging the traditional owners
of the land on which we meet virtually
today and i would like to pay my
respects to elders past and present
so this particular talk it's based on
the things that i've experienced before
there's always like a lot of questions
so i'm coming from the perspective of
system administrators who are like as
probably thinking why are me why are
they referring to us incident responders
why are they asking me to do this or as
i call this stock adventures in ireland
okay
so first off okay so for those who are
wondering i'm based in melbourne
where we have the best coffee and i'm a
senior security consultant focusing on
incident response and i'm part of the
ibm x-force ir so it's a cool name
exports okay so i shifted the deck in
the early part of this century and i
received my graduate certificate in
incident response from science institute
and went on to go for a masters in cyber
security focusing specifically on
digital forensics and i volunteer for
different organizations so i hang out in
different slacks and discord channel
after work okay
and i do mentoring with different
organizations and if you have any
follow-up questions or anything feel
free to dm me in twitter my dms are open
okay
now so
what will i be talking talking about
like basically just you know these three
things here why why i decided to submit
to a talk again to
this linux conference specifically in
this system
administration mini conference and then
i'm going to be talking about the
incident response life cycle
and then what are the typical activities
and questions okay so
let me just make sure that i have
everything here that i could still see
you know
something here okay
so
um it was last year when i read uh the
x-force threat intelligence index
uh report and one of the things that
actually stuck to my mind was that there
was a proliferation of malware that
targets linux towards the 2020. so the
2021 uh
threat intelligence index actually
covers like the previous year so now
this year 2022 uh the report is being
you know uh generated and finalized so
this particular thought stuck in my mind
and like you know someone who has okay
please
don't play me or something who's you
know basically have windows background
okay and i thought like now um you know
threat actors or attackers are now you
know focusing on you know the linux
operating system and largely because uh
we have you know the cloud uh sort our
resources up there we're in or just like
say up there the cloud uh you know
options where you could like spin up
your servers so there is also that
widespread adaption of linux as the
operating system so
there is actually a rise in the linux
malware families uh specifically what my
colleagues in the x-force ir team and
the intel team actually found out was
that there was a rise in terms of
developing linux cryptominers and
ransomware and because of that move of
the servers to the cloud now it was
quite interesting that some of the
incident response cases that my
colleagues have you know uh have
investigated they found you know linux
variant of the defray 911 or the ransom
exx ransomware and also the s file
ransomware so that stuck in my mind and
i thought like hey i want to make sure
that uh people with the linux background
could sort of like understand why are we
you know asking a lot of questions
during the incident response process
okay
and
one of the things that i want to
highlight especially to you who are
listening in this for uh
to my talk is that you as the system add
means you are actually considered
security incident is incident first
responders whether you like it or not
you are part of like the bigger ir
process and why because you're the one
who
uh have deployed your servers or if not
you are the one who are taking care of
the servers and the applications there
so you are the best person to understand
what is happening in your you know
servers and your environment and if
there's something dodgy going on uh
using your tools or whatever you know
monitoring tools or scripts that you may
have there you will be like alerted so
you will have
you know that first hand knowledge about
your systems okay and so
my goal for this short talk is to walk
you through the ir process so that you
understand why you may get asked certain
questions or you may be asked to
participate in certain activities okay
now
so there is this diagram this is a
classic diagram it actually comes from
the nist there is a document called
computer security incident handling
guide
and think of the ir
as a life cycle and each phase
has its own
distinctive activities okay
now
um
the idea before you actually have an
incident is that you should have like
beard or you haven't had any incidents
yet
that's like really fantastic but
you know you don't you know rest on your
lawrence and say yeah we haven't had
like any incidents or anything or
security breaches
it is a reality and a sad fact that
there will be breaches there will be
like you know incident and you need to
be prepared for that so the first part
of the ir life cycle is that it's
preparation so what is involved in
preparation okay so first of all you
have to have an ir plan okay now um if
you don't have an ir plan in your
organization please start brainstorming
and thinking about it and make sure that
you have a plan that is communicated to
everyone like in the technology teams so
that they are aware of like what are
they supposed to do in case there is a
bridge or a security incident now let me
clarify first the terms about like
incident
so some people get confused about events
and you know incidents so when we talk
about an event that means
there is an observable
uh
uh change or there is uh something that
occurred
so what is an event so it could be that
there was a connection
from
one endpoint like a host you know
connecting to a server so that is an
event
now uh what is an incident incident
basically means in the parlance or
terminology of incident response it
means that
uh there was an event it was it's not a
false positive it's a true positive and
it has breached the security triad so
what's the security detail you have the
cia so you have
not the
agency but confidentiality
integrity and availability okay
so that's how we define incident
now in order to prepare uh uh when you
uh prepare for the response to an
incident so there are things that you
can do okay there is what is known as
tabletop exercise or ttx
okay
so you as the system administrator as
part of the security first responder
team you could possibly be invited to
join or attend a ttx so if they ask you
could you please attend the tabletop
exercise please don't say no please say
yes because this is the chance for you
to actually be involved in a simulated
think of it as a mock incident there's a
mock incident there is an incident that
uh like it's think of it it's like a
scenario based on the most likely you
know incidents that will affect your
industry or your organization prepared
by either someone inside the
organization or it could be that they've
engaged you know a consulting team
so
just to clarify when we talk about
tabletop exercises this is
non-functional because some technical
people they're kind of afraid like oh i
need to like do this and do that and all
those things no it's just think of that
think of it it's like everybody gets
talking you know talking like there's
going to be like a situation provided to
you
and then you'll be asked to talk about
what are you supposed to do
once you see this or you know get this
particular information
and then
after that there's going to be injects
okay injects are like you know
additional information that are provided
to you to move the incident along okay
so once someone asks you to could you
please you know attend the tabletop
exercise just say yes another thing that
you could potentially be asked to do as
part of the preparation phase is to
check
a playbook so what is a playbook
think of this as you know processes or
the procedures that are
um needed in order for the organization
or for your team to respond to security
incidents so sometimes as the subject
matter expert you will be consulted and
asked hey we have this playbook for this
could you please take a look at that
would you please check this so please
say yes and then another thing that you
could potentially be asked to do is
about logs though so us
uh digital forensics you know
investigative store you know incident
responder we love logs okay because logs
give us you know a lot of information
regarding time stamp what has happened
so log analysis is one of the things
that we do as incident responders now
maybe you'll be asked hey could you
check the log retention period how long
do we keep our logs locally and then
aside from local you know um storage you
know you there's also such thing as are
we sending it somewhere um you know it
could be in another data center it could
be in the cloud so the log retention
period is important especially if we
need to go back and check what happened
before a particular incident
okay
now
so that's for the preparation phase the
next phase is called
detection and analysis okay so
it is possible that
uh by the time that you've been like
called to
install an edr and endpoint detection
response agent on the endpoint there is
already an incident and to further
figure out like what's happening across
your environment you may be asked to
suddenly deploy an edr agent but
supposedly this edr agent should have
been put in there during the preparation
phase but sometimes you know you may not
have that but you know
you gotta do what you have to do okay
now another thing that you may be asked
to do is to run certain scripts to
collect a triage package so just to
clarify a triage package think of it as
like
uh
basic you know information needed to
investigate the incident so you will
probably be asked like for example there
is this
um
the velociraptor you could use it as an
offline triage collector or specifically
for
uh linux unix you know os there is the
uac it's uh which stands for you unix
like artifacts collector so you may be
asked to deploy that run this particular
scripts and then once
the script has finished collecting all
those
potential artifacts or data you may be
asked to could you please of course in
the time of the pandemic everybody's
like working remotely
you'll be asked to
uh
save it send it
to a cloud secure cloud storage it used
to be
that when there is an incident before
the pandemic or before times
uh
incident responders will actually go on
site
okay to the client side or to that
particular site and start you know
collecting these
uh sources of evidence but now because
of the challenges with
um traveling and some restrictions you
know sometimes you'll be provided with
the one-page instructions on how to
actually run this and then given a link
to a secure cloud storage and then you
upload the collected artifacts there
and then sometimes it could be that
you'll be asked to collect
certain specific artifacts like for
example the contents of the
slash etc directory or the slash home
directory like for you know figuring out
who are like the users there and
most specially the slash bar directory
there's a lot of logs there so one of
the things that we commonly checked is
that the contents of the wtmp for
successful logins and log out okay and
the btmp for failed login okay and one
thing that
we actually check for btmp for failed
login is that uh it is possible that a
user may have accidentally typed their
password at the user login prompt so we
could actually see that we're gonna
check that the other thing so aside from
that there's also the last log file
which is the most recent use user login
so it really depends on what kind of
incident
are we investigating it could be like
potential insider threat or it could be
that you have attackers who were able to
go into your organization and they've
been moving around and trying to log on
to the different you know servers
okay
so
uh as we go along in the incident uh
process or in the incident you know uh
response uh life cycle
there is this part called containment
eradication and recovery phase
okay so potentially you may be asked
to
do some quarantining or containment of
the server so when we talk about uh
containment okay there are several ways
that you could do that okay
you could do it on a network level ask
the network team hey could you please
shut down the
uh you know that particular switch port
okay or it could be that
uh
you have a an edr which has the
capability to do uh containment or
quarantine wherein the network
communication is going to be stopped
except from
the server
to the edr
server or it could be an edr in the
cloud okay so that's the only network
communication that's allowed
uh
several years ago actually it's more
than a decade ago
believe it or not i had one client who
actually told me in our panic in
containing the server they actually
unplug it and like oh and plug you know
the lan cable uh no the power cable oh
please don't do that because we will
lose certain artifacts that are volatile
like for example things that are in the
memory okay so make sure that you have a
process procedure for containing your
servers
okay
now
uh if let's just say this is a
particular you know incident uh
requiring a freshly installed like for
example it could be that you've been hit
with you know ransomware and
you you know
your choice now is hey we just have to
you know do a freshly installed so
potentially you could be asked to do
that okay or it could be that okay once
we've like done our reinstall you're
gonna be asked hey can we use the
backups okay but before you use the
backup make sure that your back backups
are clean what do i mean by that it
could be
that
when your system was backed up
it also backed up the binaries for the
malware or for the ransomware so
you basically started like a fresh
reinstall and then you put your backup
in and then you get you know infected
again so just make sure that you all you
also check your backups whether they're
clean okay and then another thing is
that are your backups usable so part of
the preparation phase would be in terms
of the backups you always check your
backups too because you may be like
doing a backup but you haven't really
tried a restore
and you know it could be that during
your hour of need that's when you need
to restore your backups you find out
that they're not usable so make sure
that before this happens part of the
preparation is check that your backups
are usable okay
now potentially okay
as you go progress along the way for
this uh incident response you know life
cycle
there are things that you learn like you
know oh we should have done this okay so
at this particular phase it's called the
post-incident activity or sometimes it's
called
lessons learned phase okay
for this okay could be like you'll be
asked questions about
what can we improve
okay what can we improve as a team or in
terms like processes or it could be
maybe there is some challenges in terms
of the technology that you have there or
it could be that you don't have enough
people so typically most of like the
lessons learned will be focus on these
three things people processes and
technology and then what did we learn
from this okay and then what are the
improvements that we need to make in
order to make sure that we are able to
protect our organization our information
assets and have probably let's just say
faster recovery time so basically it's
more about questions like
how can we do better next time okay so
these are the things that would actually
help improve your incidence response
capacity
or i'm sorry capability or sometimes
what happens is that um an incident
happens and you think that oh we're all
set we have the ir plan and then turns
out that the ir plan hasn't been updated
in ages and when you look at it it's not
you know you don't have the correct you
know or contact lease or it could be
like people have moved on so you don't
know who to contact about this
particular information assets and then
you know part of the post incident
activity is to make sure that you also
improve update your incident response
plan
okay now
as you go along here there's definitely
a lot of lessons learned make sure that
you are also documenting each step of
the way like for example if you were
asked by let's just say a third-party
vendor or a service provider as a
security service provider to run certain
scripts or to you know install certain
agents and all those things make sure
that you're also documenting them
because um later on if let's just say
there is a need to do it again at least
you have them ready or it could be that
whatever
uh
you know steps that were
done during the detection analysis the
containment this could potentially be
part of a new playbook so the cycle let
me go back again
okay the cycle here as you can see okay
so you progress along from preparation
to detection and analysis containment
eradication and recovery to post
incident activity so after that okay
whatever you've learned here you go back
to the preparation meaning you
maybe things that you've learned you can
create a mock incident you can have
another tabletop exercise or you could
create another playbook okay
now
my ass from you
okay for after this is that please when
you go back to work think about your
incident response process do you have a
plan or if you have a plan when was the
last time you tested it uh when was the
last time let's just say you updated it
uh do you have
an wall tabletop exercises do you have
playbooks there so you start thinking
about this and slowly but surely you
know you go along and and i know it's
going to be a big ass because aside from
doing your system administration duties
because of you know coveting the
pandemic you may be asked to do other
things you know especially covering for
let's just say other shifts of people
who may be isolating and can't work so
do it you know maybe like on a quarterly
basis you have a plan to do this part
and then after that do another point
okay so with that
okay
so if you're interested in getting a
copy of the x-force threat intelligence
index just go to the ibm website search
for it okay and
you have to provide some you know uh
information about your email and all
those things uh don't worry we have this
you know uh the data privacy you know
regulations and all those things so uh
and you can opt out of marketing just to
get a copy of that report okay and then
there's a good guide from an nist that's
remember it's uh the publication
800-61r2
it's available for free from nist and
then if you are interested into in terms
of specializing in you know let's just
say digital forensics coming from a
linux background there's a very good
book from norseparch press it's called
practical linux forensics that's a very
good book if you need to actually
collect some artifacts you can you know
go to github search for the uac or you
know you can use also velociraptor okay
so if you have any questions after this
conference i have allocated uh you know
a few minutes for some questions okay
you can like ask those questions if not
you can always dm me in twitter okay
so that's it so i'm gonna just minimize
this so i could see if there's any
questions there
yeah there's a couple questions
um so first question is uh do you prefer
people to rebuild servers and restore
data or restore the full
the full image during the recovery
phase um as with all things with in the
ir world uh it depends on
you know how fast you need
the system to be up
so
you're the one who's gonna be like you
know deciding because you're the
information asset owners okay
just make sure that before you start you
know putting back all those backups as i
mentioned before make sure that they're
clean
okay
okay next question is how do you balance
incidents that affect support support
systems versus responses that
potentially affect production systems
how do you go about risk assessment on
the changes involved in
okay
so once again it depends on the
organization's uh risk profile
or what they consider as very critical
to their existence
so if let's just say what is like very
important for their existence is you
know remediating immediately watching
production because once production is
affected that basically affects the
whole business so you start you know
responding there so typically let's just
say a sample scenario would be let's
just say ransomware
okay we're in it could be like several
servers may have been affected and they
immediately ring our ir hotline so we
immediately stand up a lot of streams so
there's several work streams in parallel
happening at the same time so we have
the incident response uh you know
happening we're in there's gonna be a
collection of the artifacts to make sure
that we know what we're dealing with so
we have to make sure that we understand
what is this malware what are the
capabilities and then there would be
some collection of let's just say
certain files because if it's like a new
malware variant we'll have a separate
track for the reverse engineers we're
going to be doing the reverse
engineering on the malware then we have
a separate track wherein we start doing
like you know the remediation so
sometimes what happens is that you have
several tracks happening at the same
time
okay but in order to do that you will
need to make sure that in your
organization if your ir team is in-house
you have enough enough people there and
that's why what i found is that most
organizations will actually call in
a third-party vendor but please make
sure that you have an existing retainer
first because what happened what i've
seen so far is that some organizations
don't have that retainer or like a
contract or something and they start
calling up in
like last last month with the lag4j and
all those things it was very very busy
it may be that
you you call an ir firm and they said
sorry we can't really take you on
because we're like really busy you know
providing all this services to our
existing clientele so so think of like
that retainer as a way for you to make
sure that you have that allocated
resources for you
by the way i'm not a salesperson i'm not
trying to sell you anything i'm just
like talking from experience
okay all right um i think that's all the
questions we've got all right thank you
very much
thank you very much for listening to my
talk i hope you learned something from
this hit me up in twitter if you have
any questions okay so take care and once
again i can't give everybody a hug so
i'm going to give you a wakandan salute
giving take care be well and be safe and
thank you