[Music]
hey christopher
thanks for coming along today um so our
next presentation is christopher biggs
who's going to be talking about
assistive
ar on a budget
and uh this is a topic which is
particularly interesting to me given my
activities over the last couple of years
with assistive technology projects
and i'm fascinated to see what chris has
achieved so
thank you very much chris and take it
away
all right uh greetings earthlings meat
and metal and also to anyone else
receiving this out there in the wider
universe
today i want to confirm that planet
earth is ripe for takeover by exploring
the hilariously poor
state of our cognitive technology
and confirming that the inhabitants of
this planet are still in the ludicrously
early stages of uplifting themselves to
higher sentience
but before i go on some more shameless
self-promotion
my background is in computer science
my sideline as an evil roboticist is
pretty much self-taught
i've always since i was a teenager been
fascinated by the possibility of bodily
and cognitive enhancements
and that fascination has led me to
create accelerando innovation space
which you see behind me
we're a technology consultancy in
brisbane that also provides a co-working
and maker space
and the work we do with our members
ranges from full service implementation
to just providing a roof and some tools
and letting people go for it
so if you're in brisbane and that's
interesting we're also the monthly venue
for the brisbane into the internet of
things meetup event which we call sunday
mad science which is an open lab
afternoon where you can bring your
problems and projects and broken tech
and explore solutions with your peers
and the project i'm going to talk about
today is one of those things that just
arose from sitting around at the mad
science meetup and shooting the breeze
so naturally an environment like
accelerando involves a fair bit of
multitasking and multitasking is
something that which i am quite well
known to be 100
absolutely and undeniably [ __ ] awful
look i've occasionally been known to go
looking for a screwdriver find the exact
one i need and then spend the next hour
wondering where did i put the screw that
was in my hand a minute ago
actually i'm bullshitting a bit there
it's not occasional it's constant that
happens all the time and
the current unpleasantness here in
season 3 of 2020 seems to only make it
worse
now cognitive challenges can be
addressed in a number of ways there's
psychiatric therapy and learning coping
techniques which i like to think of
debugging
for your meat wear
pharmacological treatments are quite
popular and i reckon this is basically
overclocking but today i want to tell
you about prosthetics technology for
improved multitasking
my friend here in brisbane monica
bradley often says that she wishes she
could have an operating system for her
life and that's something that i can
really get behind
so i chose to look at this problem sort
of systematically well not really what i
actually did was read a lot of science
fiction and watched a heap of movies
on the left here is what comics and
movies and novels daniel before us 3d
holograms that obey our tactile and
spoken commands
on the right is the absolute cutting
edge
of wearable technology
as of 1990.
i so envied these guys back in the day i
wanted to be them
so
firstly i want to look at what parts of
the sci-fi future we've already got
what tech is just around the corner and
what currently imaginary tech might be
in the pipeline
then i'm going to go piece by piece
through what someone on the modest
budget that is not tony stark might be
able to assemble today
we'll start with a computer
which is somewhere where we were
actually swamped for choice
screens not so much there's good and
there's cheap and there's very little
intersection and there's very little
compromise
cameras have been a real surprise
there's a
incredibly visceral do not want reaction
from almost everyone to the idea of
heading out of cameras and finally the
bits that motivated me are cognitive
aids and wearable instrumentation that
is the apology the possibility to
augment our data acquisition
and multitasking ability or the lack of
it with technology
some people use the word pre-disabled
with varying degrees of seriousness to
describe those lucky people who have no
physical or cognitive challenges yet
the intent of that term is to highlight
that assistive technology is not a niche
market we are all going to need
something
sooner or later spectacles are
mainstream
hearing aids are more common than you
think
wheelchairs and mobility scooters are
not uncommon but we suck at
accommodating them so here's what i want
for assistive technology
what the market delivers and what i
think is the next level
when i first got my gsm phone in the
1990s i read that it had something known
as short messaging service where you
could send a brief textual message to
another phone but back then it didn't
even work in australia because the
telephone companies hadn't bothered to
bring such an obscure and useless
feature to market
only text messages turned out to be a
killer app
pages and phone booths and voice
messages went extinct pretty much
immediately once people worked out how
much more convenient a quick text
message was
and you might have forgotten that
twitter started as a mobile phone
service for messaging a group of your
friends about what you're up to and
where you are
and then the tail abandoned the dog and
we got a whole new social paradigm
now these text messages are coming to
our smartphone screens or to our smart
watches and to be honest that's pretty
freaking cool that was everything i
wanted in the 1990s
from
always connected network technology a
smartphone gives me probably two-thirds
of my assistive tech utopia and i do
think that in five years or so smart
phones might get us nearly all the way
there
right now we all use our phones as
torches or magnifying glasses and as
barcode scanners to get into every damn
shop that we go to etc and this is
really handy but it's also a handy cap
in that one of your hands is always
consumed in holding the phone
now i'm getting older
and i've been warned that my arthritic
hips and knees are going to give out on
me someday soon so i'm also thinking
about how to save on movement i want to
have my appliances and machines stop
demanding my physical presence so much
and be able to interact with them from a
distance and since we're all
embracing remote working
right now i think this is something that
is only going to
require wider attention
now
if you've met me in real life there's a
good chance i forgot your name within 10
seconds
and please don't think it's because i
don't care but
for an introvert like me meeting someone
new is a heck of a cognitive load
memorizing a face
doing the right bow or elbow bump or
having a bit of handshake smiling and
thinking of something to say pausing
what that person said it's a huge cpu
spike and it seems that in my case the
background process
that moves important information from
short-term memory
to long-term just glitches out for a
moment if i hear the name again i'm good
because after that initial processing
spike everything works okay
so sometimes i wish i could just reround
the last 20 seconds of audio
even
subtitles would be a bonus too
in order to catch something that i
missed or that i failed to record during
a moment of uh cognitive embarrassment
now recording data is another place
where our phones
are both great and awful i bet you've
got a photo roll full of cat pictures
and shopping receipts and white boards
and phone cameras are a fabulous way
to record little snippets of information
but they're a terrible way to then
process that information to return to it
to find it again
a task switching is another area where
i'd love to have mechanical help if i'm
working on three things at once and i
jump from one task to another how can i
use technology to help me get back to
the previous task or even remember what
it was
and the last thing i'm wishing for is
extra sensors the ability
as as the previous pension speaker
mentioned to see in radar to
perhaps see temperature to see
electricity
how can i
uh replace a bunch of
uh
sensor devices that i might normally
carry around
uh throughout the course of a day with
something that is always in my body
so i mentioned
that
i drew inspiration from science fiction
and futurism and i want to call out some
of the milestones in that process
uh early 1980s robocop
apparently he runs ms-dos
not something i'd recommend
but that idea of information overlaid on
a visual field is something that
we've all found compelling
and i think
the market has had a few terribly
unfortunate stabs at
robocop 2 apparently ran rakos which i
can get behind
this is from terminator 2 again the
concept
of uh visual overlay
and
nearly 30 years later we're all still
talking about minority report the idea
of interacting tactilely
with um
with virtual information now there's one
series of books that i've been reading
recently that i want to particularly
call out and this is the uh the series
known as the murderbot diaries and this
is about uh
a uh an enslaved cyborg who uh who
breaks themselves free from their
from their masters
and
one of the um one of the interesting
concepts is of autonomous sensor drones
which go away and look at things and
allow you to
to visualize
the physical orientation of stuff when
you're not physically there
and that's something that i'm going to
come back to when i get a little bit
more into the detail
so that's the setup that's what i want
to do
and why i want to do it and some of the
people who inspired me to try
so let's talk about how do you do that
if you're not
uh mark zuckerberg or
tony stark
now i mentioned that computing is an
area in which we are pretty well served
everybody used to point to the raspberry
pi zero as
the great cheap
unit of compute
and until a few months ago i would have
screamed no no do not not powerful
enough
um the the thing i've been using is
um
a gadget called the banana pi zero which
has a quad-core processor
bluetooth
usb
but just recently it was leapfrogged by
the
new raspberry pi zero two so i withdraw
my objection to raspberry pi and i i can
tell you that if you were going to start
with a project like this today
um something like that that raspberry pi
zero model two
is a great choice
but the one thing that all of those
what i call zero glass boards
uh suffer from is they've only got
512 megabytes of memory only
and that's
pushing it to run a web browser or
even a graphical environment
so i'm going to um
drop some hints later about how to
do things without the the baggage of
browsers and
and even display managers
um the raspberry pi 4 has got eight giga
ram and it's good but it's a bit too big
to go in planting in your body i would
think
um
and
there's
there's also a thing called the
raspberry pi compute module which is
essentially the raspberry pi without all
the ports
um and in our case we don't want too
many of the ports but hdmi is the one
that i can't escape at the moment
and so
i wish i wish i wish there was some way
to
either have a magsafe hdmi or do i have
a wireless hdmi
but we're not quite there um
you know there's continuing rumors about
apple
coming up with some kind of wireless
head mounted display and i'm super
excited to see that come to market
uh but right now unfortunately the um
the video choices which i'll go into a
bit later are just
not great at the low end
operating system-wise if you're running
a raspberry pi use the raspberry pi os
if you're running anything
that is a single board computer that's
not a raspberry pi you're probably going
to want ambien which is
a fabulous community supported
distribution for
all the
all the um
uh single boards out there that aren't
um raspberry pi
um
ambien
uh
comes in either debian or a bunch of
flavors
uh you can now get um vanilla ubuntu
straight on your raspberry pi 4. i'd
really advise not doing that unless
there's something you can't live without
because there's just too much stuff
running
on our standard ubuntu system
for you to
uh get the battery life
and the uh the thermal behavior that
you're gonna need
um so powering powering a
a
a wrist or an arm mounted computer
um
the
the power bank industry has done a solid
there
in that
you can buy a little circuit board
that's that's no wider than your battery
uh that takes care of all your charging
um
should tells you
your charge level gives you a nice solid
five volts
um and you can just plug a usb cable in
to charge it so this is what i'm using
at the moment for my systems uh
a battery that is essentially a phone
battery
uh that has its own short circuit
protection
and a little power bank board which
gives me five volts now screen wise
screen wise uh
in the there's
google glass was all the thing a couple
of years ago
you know it cost three thousand dollars
it had this camera attached to it that
everybody freaked out about
um and then it kind of fell off the
public consciousness now it's still for
sale um with some
restrictions in what's called an
enterprise version
and microsoft has this fabulous piece of
technology called hololens which
um
is bigger than i like it's gorgeous to
wear it looks absolutely incredible from
the inside
but
i don't want to carry
something that big
i need to be able to wear
safety glasses or welding goggles
or what have you from time to time and
not necessarily want to take something
else off
um
and so the least awful solution that
i've found
so far
is this little gadget from a company
called viewfind which costs about 200
u.s around about 300 australian
and what it will do is
it comes with a pair of
fake spectacles if you're not already
encumbered with them and if you are
then it can strap to the to the arm of
your spectacles
and it essentially
gives you
something that looks not unlike a
kind of poor
2014 inch crt monitor
floating in your visual field
and it's intended just to to go right in
the middle of one of your eyes
but i've found that
it also
works quite well
uh just sitting above
your
view line so you can still see forwards
with
with both eyes but if you glance up you
can see your screen
and that's that's a real winner for me
is is to have
something that is
all there all the time but it's not
obscuring your face too badly
and
um you can access it without having to
do anything with your hands
now cameras i've mentioned are a
real turn off for a lot of people i used
to have
a little bluetooth locator that just
stuck to the side of my glasses and all
it did was it beat if i lost my
spectacles because i have a couple of
pairs i switch between them and i always
lose them
and
anytime anybody noticed that they would
always say the same thing they'd say
that's not a camera is it
it's a real
point of terror for people
for someone to have a camera on their
head
and i don't think there's any way around
that but
what i'm gonna do
is put the camera on your hand
because what i want it for is those
instances where i've got some tiny
little circuit board and i need to read
the serial number off of the chip
or i want a
bit of light down a dark hole
um so my approaches here are to use
either the camera module out of a laptop
computer so a little circuit board talks
over usb
or an
endoscope camera
and these endoscope cameras uh they have
terrible resolution
uh they're only about four frames a
second
um
but they're essentially focus free which
means yeah they'll they'll
they'll register a room
they'll register um
the writing on a on an integrated
circuit
just as happily and they've got built-in
lights
so uh this is my current go-to for
um i need
some augmented vision
uh the other thing which is really handy
is is wi-fi cameras
um there's a number of mod models in the
market
and i use these around my home
where they're either permanently
installed there's one on the laundry
which has a view
of the washing machine so we can tell
is it finished
is it loaded is it unloaded
and i have some others which um there's
one here in the office which looks into
reception to tell me if there's anybody
there and i have some that are battery
powered that i can just magnet on so if
i'm working with um
with a machine that's doing a job and i
i want to know it's finished i might
have a
bandsaw that's going to take uh 20
minutes to soar a
10 centimeter wide steel bar in half
i can set up a camera overlooking the
saw and be able to just switch tasks and
occasionally
check in on
on a camera
to see how that task is going
now
the way that i
do this
on a on a limited uh
compute device like a
one of the zero boards
[Applause]
um so use
uh g streamer um and the
the um
[Music]
the frame buffer output so i can
essentially pop up as
one two three or four
video windows on my little virtual
monitor
that
that overlay whatever other information
is being shown
and the other
the other thing that i use all the time
is is a
frame buffer
um
pdf viewer because
my other killer app for head mounted
displays
is being able to read documentation so
if i'm working on a circuit board or i'm
working on a car
i might have or an appliance i might
have the service manual
in digital form or the data sheet in
digital form
and i can throw that up on on screen
again it's a it's a much more
resource appropriate way of viewing a
pdf
than doing it in a um
in a web browser
keyboard-wise um i love various little
um
miniature keyboards which are intended
for use
with smart tvs
um
i i use one which has a keyboard on one
side and
a
gravity mouse on the other side so i can
flip it over and just entire my arm
and that provides me with um
with spatial input or i can
i can type some commands
on the little keyboard gadget and the
way i'm currently operating this is i i
have it attached to my uh attached to my
armband computer with uh
with magnets as we say um
now one of the things i bought while i
was exploring this whole process is it
is a uh
an offshoot of the google cardboard vr
where you plug your uh your mobile phone
into a box
um and it gives you a head up display
these work quite well but
um
you know they're not some they don't
give you through vision so
um if you if you're a virtual space
that's great if you're in a physical
space where you need overlay not so good
but they come with this nifty little
joystick um which is just a bluetooth
device and uh for the 20 bucks of this
gadget cost uh that little joystick
itself is uh is worth the money
now let's talk about cognitive aids
so i like to have
a stack of tasks that i'm doing
and
the mobile phone equivalent is you take
a photo of whatever you were last
working on
um when you switch tasks so you know
where you're up to when you go back
um and so what i
have worked on is
coming up with a with a macro button on
my keyboard which does the same thing so
i can go okay i was here
give me a photo give me a five second
voice note
um and now i'm gonna switch tasks
um and the other thing that i'm i'm
working with
which is kind of unrelated to this
project but it's it's part of my idea
which is just a little
uh flashing led and in this case
you can now buy leds that have all the
flashing circuits being built in so this
is literally just a
lithium battery
and an led and maybe a switch
and i have
about six colors of these
and if i'm interrupted from a task i
turn one on or drop it where i was i go
away
and when i am uninterrupted i turn
around and i look for the flashing light
okay that's what i was doing last
um
as uh as someone who is professionally
frazzled um this is something that is
really helping me to
get that under control
now instrumentation
um as a as an evil robot engineer
i'm often
carrying a multimeter around or a
or a digital storage oscilloscope
or
even
working with blowtorches or other nasty
tools
um what i'd love to be able to measure
uh with my fingertips
is
for one voltages
you know is this thing powered up
for two temperature is that thing still
200 degrees or can i pick it up yet
um
so
how do we do this um
without uh
ending up carrying too much stuff
or without ending up
breaking wires
and just in the last six months or so it
seems that magnetic connections have
gone from
really hard to find to
absolutely dirt cheap and available in
whatever you want so when you've got
um when you've got i squared c sensors
um
that need four wires
um
you know 50 cents will get you a pair of
four pin magnetic connectors which allow
you to swap sensors in or out off of
your
your wearable computing
um and another possibility is bluetooth
i've designed a couple of really small
low-cost bluetooth boards
into which you can plug
whichever sensor you want and you've
basically got a little
matchbox size puck
that can be a temperature sensor or a
voltage sensor or whatever
and they broadcast
their their identity and values via
bluetooth low energy so i have an app on
my gadget which just says hey what
sensors are around me what are they
currently saying
and so i can turn turn a gadget on
uh drop it
in the position near a job
um
and
have that input streaming to my device
so let's look now at what this all
actually looks like in its current uh
prototype state
so here we have
um
a chunk of the eva foam the um
the cosplayers
go to
material
into which we have sewn
a battery
and a
power supply board
and a in this case banana pi zero
and we've got our
keyboard
um slash mouse uh magnetically affixed
to that
um and we've got our head mounted
display
and
here it is in its uh
hilarious
um
physicality
and i think i would probably fit right
in
with uh with those heroes of mine from
the
come back there 1990s perhaps uh
spotted me a little jacket
now
i'm not sure that i would say to you go
out and build one of these right now
because i'm starting to think that maybe
the arm isn't the right place
um there's a youtuber who is uh
who's doing great stuff named or jay
and he's built himself a little
shoulder-mounted uh
companion pet
which which does tricks for him
um
and
i'm really starting to think that either
say the back of the head or on your
shoulder or
the upper arm is a place to put
something perhaps a little bit bigger in
a raspberry pi size computer
that can can use those magnetic fixtures
to um to connect to sensors that are
down at this level
so
that's a a whirlwind tour of where i'm
at in
in um
in wearable sensing and uh data
acquisition
um head mounted cameras um unfortunately
they're just poison
um finger mounted cameras however i
think are a huge win
um gravity and accelerometers is a is a
fascinating input device
head mounted displays right now
you know that viewfinder device is four
years old and there's still nothing that
touches it on the market
um i'm really interested to see what
apple and
samsung come up with with their roommate
products
and there's an old saying that a
computer is a machine to help you think
and that can be whatever it means for
you
whatever
you struggle to stay on top of is what
your computer ought to be doing we have
the same here at salarando
we get computers to worry about things
so that you don't have to
and i hope that
you can find computers to worry about
your
anxieties also
and uh with us being here in in season
three at 2020 um i think remote sensing
and remote monitoring is something that
we all
would like to be doing more of
so that's what i have for you uh thank
you for your attention
um and i think we're probably
running a bit short on time but uh happy
to take questions etc oh great thank you
very much christopher
um yeah there are a couple of questions
we are slightly over time but
questions are good
and the audience always have good things
to ask about
one of the questions is which ar glasses
should i keep an eye on as a wheelchair
user who has been waiting for these for
10 plus years
um
there is another reason there is another
product which i'm struggling to remember
the name of
that's um that's a little bit higher up
the budget tree um my friend pat
catanzariti
um
who you'll find us at patcat on twitter
uh wrote an article last year which is
uh the top 10 ar displays on the market
so
if you go and search uh
for his article that's
probably the uh the best answer i can
give at the moment
okay all of those products unfortunately
were in the thousand price range uh this
viewfinder one is the only uh anyone i
found that's
um
under 500.
yep
another
suggestion was this is going back to the
start of the presentation you were
talking about little sbc single board
computers someone said have you seen the
red shirt right i'm not sure how to
pronounce it redshirt zero which is a
quad a53 four gigabytes of memory and uh
pi zero form factor
no but that sounds fabulous
memory oh my goodness
so that's something to check out and the
final question is are you worried about
the resistance and signal integrity with
those magnetic connectors
um
not so much
particularly using i squared c
um
there is there are some booster chips
that are things that are called active
terminators
i don't know uh how familiar the
audience is with i squared c but it's
it's what's called an open collector
mechanism so
rather than driving a high signal on a
low signal
you've got a resistor which gives you a
sort of weak pull up to your high
voltage
and you drive a zero for the low voltage
and that means that you can have
multiple things connected on a bus
really easily but it also does mean that
it is pretty poor over long distances
but just recently there's some chips
that you can get which
intelligently
slam the bus up and down for you
uh to give you more range and more speed
so
yeah we're actually finding that we can
uh we can go further with those tables
very good
okay um well we had better stop now
because we've got only a few minutes
until our next presentation so thank you
christopher that was fascinating um
everybody else we've got about a five
minute break and then we'll be back with
the next presentation