2018 Ulam Lectures – Cris Moore – Limits of Computers in Science and Society Part 2


good evening everybody so everybody can
hear me fine my name is David walpert I’m a professor up at the Santa Fe
Institute and before anything else some thanks are in order it just disappeared
first off I’d like to thank Thornburg investment management for underwriting
these community lecture series this is always said at the beginning of any
presentation we would like to thank them but really what they do is a great
service not just to the Santa Fe Institute but more generally to you and
I don’t really know that they get the proper mcclain appreciation they really
deserve for this in addition of course there’s the lens ik the lens ik does get
lots of praise and whatnot but it can never be praised too much and
I think that they also really we are all to be very very grateful and a
relatively new contributor to this series of presentations Enterprise
Holdings foundation for their additional support so I’d like to thank them very
very much so these are the coulomb memorial
lecture series I personally never actually knew Stan Gulam but there was a
year that I was the alum fellow up at a small scientific military industrial
complex on the other side of the Rio Grande in the hem astounds and while
there I thought that well hell man I’m the alarm fellow I should learn a little
bit about Stan ulam so I read his memoir called adventures of a mathematician I
highly highly recommend it it’s really a very very well-written very interesting
book on many different angles one of them is that even so to speak ignoring
the factual content just the way it’s presented you start to get a feel for
Stanley LOM as a person and he really comes across as for lack of a better
term a gentle soul he was somebody who was in some ways a boy in a good sense
of its word it was a child still and that’s really what science what all of
the intellectual pursuits of humanity really should be about it’s also
fascinating if you’ve never studied learned before about what the life was
like in the 30s and 40s especially as the institutions that form such a large
part of our current reality were just being born as modern scientific
industrial enterprises were really just developing from the era of the gentleman
scientists and it was very much the gentleman scientist in those days to
word has become now all about the contrast between science and academia in
Europe and in the US but also of course I learned a lot about Coulomb he wore in
a certain sense three hats he was a physicist that’s why he was up at that
little institution involved in that little project that was going on up
there he was also a mathematician and the name
of his memoir adventures of a mathematician but also and I’m a little
bit ashamed to admit didn’t really fully appreciate this
before reading it he was instrumental actually in forming computer science the
third leg of his expertise for example but we now know as the Monte Carlo
algorithm one of the most powerful most important computer algorithms known to
humanity which really lies underneath the guts of so many of the things that
we take for granted in the modern world from being able to fly in airplanes to
being able to talk on these little toys in our pockets all of these things
really rely altum Utley on Monte Carlo algorithms and for that we can give
standalone a round of applause because he was instrumental in actually forming
it but this is of course New Mexico and New Mexico Santa Fe is nothing if not a
place full of karmic resonances so ok this is my kind of spiraling around to
the topic of tonight Stan was a mathematician physicist computer
scientist these are the Elan community lectures the ones that we are privileged
to enjoy tonight last night’s was fantastic and tonight should be riveting
as well though maybe in some ways a bit more depressing well to be seen but
anyway is we are also being presented tonight with somebody who likes Stan was
a physicist computer scientist still is for this is computer scientist and a
mathematician in fact the first time that I ever actually encountered Chris
and his work was with some really very neat for lack of a better word paper
that he had done using some tech showing how a very very simple system in
nonlinear dynamics what’s called the baker’s map and that really is a very
evocative description of how this nonlinear system dynamical system works
it’s just folding things like a baker does when they’re folding pieces of
bread and kneading bread how that simple map inspired by
kitchen Processing’s actually can embody in it Turing machines which we learned
about last night one of the most profound and deep concepts all of
humanity has come across is something that you are accidentally doing every
time you’re kneading bread in the kitchen and analyzing it with a bunch of
really beautiful mathematics these things were all related and ultimately
this is all connected system work that people were doing back then on how
simple physical systems embody Turing machines really cool work and that is
part of what Chris actually started doing from there he went on to do a lot
of he didn’t speak about these things directly last night which was the one
failing I thought that he didn’t talk about his own work more but he went on
from there to do some really groundbreaking stuff involved with
taking the techniques of physics concepts like phase transitions for
example and using them to analyze those very difficult challenging deep kinds of
optimization problems that he was presenting to us
last week last night and then he’s also gone on from that done many many
different things some of them are actually now he’s taking these
techniques of physics and and applying them to things like social networks and
more recently a bunch of stuff in machine learning to actually learn about
what can actually even be learned to get a little bit meta in the whole thing so
he’s really been doing what Stan started to do in an almost much more deep
fundamental way it’s really kind of cool how these things have come together in
Krista’s whole body of work going on from that though Chris in true Santa Fe
style is not just a scientist as I would expect 99% of the people in this room
know he is also a consummate connoisseur of music he knows a lot about it he is
actually quite versatile in the musical arts himself and for example he’s gotten
these little things like television shows that
are all about his deep insights into connections between music and math the
music of the spheres in a different sense of it also as was commented last
night he is don’t hold this too much against him or at least he was a
politician believe it or not there actually have been people who’ve been
involved in political systems who actually have the greater good at heart
and are not doing it all just so they can go around and be on power trips
Chris was a was and I think it’s now past tense at least for the moment a
great example of such a person who is involved in politics for all the best
possible reasons so he’s really a vertebral polymath not just in the kinds
of things that he actually thinks about but in the life that he actually lives
some of the details this is just a quick run-through of some of the highlights he
received his BA in physics math and integrated science as opposed to I guess
differentiated science or something from Northwestern his PhD in physics from
Cornell he was a professor at UNM I think that actually showed last night to
be quite honest he was a professor and in contrast to many other professors he
really cares about the students he cares about the audience that they know what
he’s trying to convey and I think that that really comes through and just how
good a presentation he makes he since 2012 he’s been a resident professor at
the Santa Fe Institute we’ve got the office right next to mine he’s had
visiting positions visiting positions at Ecole Normale Superieure it called poly
a technique University police at the Niels Bohr Institute Northeastern
University of Michigan over a hundred 50 papers blah blah blah
he’s also actually and now we’re getting to things that usually are at the top of
CDs he is a fellow of the American Physical Society of the American
Mathematical Society and of the American Association for the Advancement of
science I’m not really sure all that much what’s left maybe he’ll win a
Pulitzer next year or the Man Booker Prize or could it be a
log listed maybe and of course he’s also as the shamelessly plug to use his own
words last night he is the author with Stephen mertens of a book that is really
a fascinating trove of insights into computer science and mathematics the
nature of computation from oxford university press to sort of bring this
introduction depleted all I started talking about Stan ulam and his great
contributions to computer science the Monte Carlo algorithm and how he himself
was a gentle soul in many ways ah Stan was also actually instrumental in
correcting a flaw in tellers designs for the super and helping to actually come
up with the designs that superseded tellers original work that actually then
bore fruits over some atolls in the Pacific we know the results of this
contribution to humanity usually these days they’re called thermonuclear
weapons the same mind the same person the same humanity that results in the
Monte Carlo algorithms that allow us allows modern human civilization at the
same time intrinsically as part of it this is almost like a yin-yang or
something like that also enables our own destruction what Chris is talking about
tonight it’s a certain level at least as I understand it is how computer
algorithms in general enable so much of our modern civilization but if we’re not
careful about how we use them they actually also have some potential
downsides that we’ve got to be very very aware of and so anybody with all that
introduction I’d put you in the very capable hands of Professor Christopher
Moore Wow
so to help me to help me redefine my ego it would be it would be great if someone
could stand up and say I knew Stan ulam and you sir are no stand along well Stan
ulam is indeed one of our heroes and in fact last night for those of you who are
here we talked briefly about set theory and on top of all the other physics and
mathematics he did he also contributed to that field he also wore excellent
hats so good so thank you all for coming tonight I’ll jump right in artificial
intelligence can do a lot of things whether you call it machine learning
algorithms whatever you like to call it it has scored a lot of impressive
victories over the last few years it’s been a long time now since the champion
chess player on the planet was a human computers have been built beating us a
chest for a while and just a few years ago they became the champions in the
Asian game of Go which is much deeper and more complex than chess and so that
surprised a lot of people including me I thought that achievement was decades
away artificial intelligence is has great promise in medicine so for
instance for many types of cancers a computer program can analyze a medical
scan and identify tumors as well or possibly better than human doctors and
radiologists artificial intelligence can also give us reasonably good
translations from one language to another here is a little sentence from
Voltaire let’s see what Google Translate makes of it doubt is not a very pleasant
state but insurance in insurance is a ridiculous state oh give me give me a
second Oh last year all us of course Voltaire
he didn’t mean insurance he meant certainty one second I can go in and and
fix this your contribution will be used to improve translation quality thank you
okay good just improved Google’s translation
algorithm and I did it for free now of course AI is also capable of doing some
very stupid things like adding bouncy cartoon characters to a picture of your
mother’s grave because someone at Facebook thought it would be great to
identify what your most fun picture was of course this is a this is a bold new a
powerful new combination of human and computer stupidity which will take us to
ever greater heights of stupidity that we have not achieved before all right so
since AI as good as certain things there’s a lot of thought a lot of
controversy as well about AI in the judicial system in the justice system
and let me just toss out a couple of things that you probably already know
about the justice system it is quite biased in many ways certainly against
low-income people who don’t have as good access to quality legal help despite the
heroic work of public defenders and people who with darker skin than mine
are more likely to be arrested when they’re arrested they’re more likely to
be charged when they’re charged they’re more likely to be charged with a felony
with a more serious version of a crime they’re more likely to be given jail
time instead of probation and when they’re given jail time their sentences
are on average longer so and this is even when we try to control for the
similarity of the crime between different defendants there was even a
study in Florida so this is from the Florida a newspaper there called the
Herald Tribune the the orange bars are the average jail sentences given to
african-american defendants the gray bars are for white defendants and so
there’s a lot of bias lurking in society I think you could say in many ways in
our own hearts and it comes to the fore when we make decisions often whether we
intend it to or not there’s both overt and covert bias so this Florida example
was especially troubling because in an effort to try to make this
ferrer they adopted a system where prosecutors would give a number a score
to a crime giving their estimate of how severe it was which the idea was that
then that would be a kind of semi objective suggestion to the judge
guiding how how serious the sentence should be so of course those numbers
themselves could be biased that the prosecutors produced but what was a
little bit more shocking is that in many counties in Florida when a judge looked
at two defendants with the same numerical score given to them by the
prosecutor the black defendant still got on average a longer sentence than the
white defendant so there they tried to adopt a system to kind of stamp out the
bias and were so far unsuccessful at doing that so since we humans are you
know not perfect decision-makers there’s a lot of optimism in certain quarters
that we could use algorithms some kind of mathematical scientific objective
evidence based thing to at least advise us even if a human always has the last
word and that this might make things better so I’m going to focus on a
particular aspect of the justice system which is pretrial detention so you have
been arrested you’ve been arrested but it’s going to be a while yet before your
trial so what happens to you in the meantime
well many of you I expect you already know this typically in most states the
judge will assign to you an amount of bail a certain number of dollars if you
can pay it you walk out the door if you can’t pay it then you’re going to sit
around for a long time and so at this very moment in the United States there
are almost half a million people sitting in jail awaiting their trials this is
more than one out of every thousand Americans including children and is 60
to 70 percent of the local jail population most of the people who are
stuck are stuck on bail of $2,500 or less so I think it’s fair to say that a
lot of us in this room if we really had to scrape together two and a half
thousand dollars to gain our freedom we probably could find some way to do that
possibly with the help of friends and family but a lot of people can’t so
they’re there now in addition to affecting low-income people as you might
imagine this population is disproportionately black and Hispanic
for what it’s worth it costs taxpayers fifteen billion dollars a year to keep
all these people in jail awaiting their trials which we could presumably spend
on something else and you know when I put you in jail your life tends to fall
apart you can’t take care of your kids you can’t go to your job so you’re gonna
get fired your marriage might fall apart you’re gonna fall behind on your house
payments you might lose your home and guess what if six months later we say
you know the charges were dropped or you go to trial we say oh you’re innocent
and you’re back out well your life is kind of in a shambles and it’s not a
surprise that studies have shown that under those circumstances you are now
actually more likely to become a criminal than if I hadn’t done this to
you in the first place so this system probably increases crime
and it certainly wastes a lot of money and disrupts a lot of people’s lives who
don’t really deserve it if you care about sort of your
constitutional rights and things that as a nation were supposed to be very proud
of it’s also notable that if you are in jail it is much harder for you to mount
a good defense you have less access to legal counsel in addition if I send you
to a horrible frightening place like Rikers in New York there’s a good chance
that you’re gonna say you know what I did it give me a sentence upstate get me
out of here this is a terrible place and I think it’s uncontroversial I mean
maybe some would argue this but I think it’s uncontroversial that this system
also produces a lot of false confessions and false plea deals simply to regain
your freedom oh right but if if you can pay the bail no problem
now a lot of states there are trying to move away from this system so recently
as you probably know California did in 2016 here in New Mexico we passed a
constitutional amendment which is an attempt to move away from this system
also New Jersey Kentucky to some extent so a lot of states are trying to get
away from this now there are certainly people who are very bad and dangerous
people and you know if you catch me red-handed about to chop someone’s head
off with an axe you should probably keep me behind bars but that doesn’t seem to
be most people so here is a wonderful article that up
I need technical assistance oh thank you this article came out just a few days
ago and here a foundation decided we’re just going to go take take some of our
money and go bail out a bunch of people from Rikers a bunch of women and
teenagers and this caused some controversy but they’re doing it partly
to help those people and partly to reveal the hypocrisy in the system good
by the way when I first learned about this I was like gee you know this is
kind of interesting I mean constitutionally if you’re
supposed to be innocent until proven guilty exactly why can the government
keep you in jail anyway and of course this is an interesting constitutional
issue and went to the Supreme Court and justice rehnquist of the Reagan era said
well Liberty is the norm detention is the carefully limited exception of
course as some of you may know he was writing for the majority finding that
under certain circumstances we can detain you before your trial Justice
Thurgood Marshall wrote a spirited dissent saying wait a second here you
haven’t gone to trial yet you haven’t been proved proven guilty of a crime but
you’re saying we can keep you in jail indefinitely simply if the prosecutor
convinces a judge that you are likely to commit a crime sometime in the future so
if you like science fiction and I’m sorry I didn’t put in a little clip from
the movie you know there are these movies like Minority Report where there
is a department of pre-crime and they are armed with a very powerful computer
which can predict with like 99% probability that you’re going to do
something terrible and then they go catch you before you do it well as we’ll
see none of the algorithms available to us have anything near that
kind of accuracy people are just not that predictable 99% the the best they
can do is something like say 20% but it’s still a very interesting question
if you could predict criminal activity would would you believe that
constitutionally of the right to grab people and put them away
this is actually kind of a science talk not just an amateur or law talk but I
anyway good so what’s the hope the hope is that
we’re going to use big data and algorithms and what we’re going to do is
identify the small fraction of people defendants who really are dangerous to
society and we’re going to go ahead and detain them which the Supreme Court says
we can do and the rest the people will release and let them go about their
lives take care of their kids do their jobs and so on in the hope that they
will indeed show up for their trials and if we do this right we could do several
things simultaneously we could reduce the jail population we could save money
we could actually reduce crime by detaining the truly dangerous people and
we could stop disrupting the lives of people who don’t really deserve it so
that’s that’s the hope on the other hand there is some skepticism that we may be
trading this for this so the worry is that if we put these algorithms in place
they’ll be handing down decisions or at least recommendations to a human judge
and if you say hey why why why did the algorithm say I should be detained and
if the algorithm is just sort of a black box with a nasty face painted on the
front then the answer will be well it’s it’s not clear but we know it’s very
accurate we fed it all of your data and it says you should be detained so there
you go sorry so that would be a little alarming
and there is also some concern that if these algorithms are trained on
historical data which is in itself biased because the historical data
consists of that the behavior of actual human beings with all our all our quirks
that it might learn some of those biases and encode them
and then make bias decisions but no longer the human judge that you can talk
to and say hey wait a second buddy this is not fair instead it will just be this
gleaming box objective evidence-based and so on and it will be it’ll be
impossible to question its results so I came to this from computer science but I
quickly felt realized that it’s not just a computer science problem there are
some wonderful computer scientists in people in machine learning I could name
Cynthia to work at Harvard Michael Kern’s to give a talk in this theater
last year the year before who are trying to redesign the algorithms that are
standard in machine learning to make them fairer and that’s good work and yet
that’s not enough I think we have to think about how these algorithms are
deployed in the real world we have to think about whether they are transparent
what kinds of recourse we have if we disagree with them we have to think
about all those process issues not just internal mathematical issues of the
algorithm good so how do these algorithms work there are red people and
green people these are not skin colors these are not races these are behaviors
that we’re trying to predict ok the red people if you release them they’re going
to go commit another crime or they’re not going to show up to court we’ll talk
later about all the different reasons people might not show up to court but
the green people are the model citizens they’re going to show up to court when
they’re supposed to they’re not going to be arrested again for anything between
now and their court date you can safely let them go and and everything will be
fine though they’ll come back when they ought to and face the evidence against
them so now what does any algorithm or indeed any decision maker try to do by
the way we keep saying algorithm algorithm an algorithm is simply a
method for solving a problem it could be a method carried out by a human we use
algorithms sometimes consciously or sometimes unconsciously or it could be
something a computer does so here is the ground truth the truth about these
people there are these P who are genuinely good citizens low-risk
people there are these people who are genuinely high-risk people okay and then
an algorithm comes along and tries to classify them into low risk and high
risk and so it puts them on either side of that vertical line now it got some
people right so first of all you see there’s four combinations here of the
truth I know this is very simplistic but let’s say that’s you know they’re just
red or green so there’s the truth and there’s this label given to them by the
algorithm so these are the people who were correctly labeled as high risk
they’re high risk and they were labeled high risk these are the people correctly
labeled low risk but there are also some mistakes here are the false positives
these people are good citizens they will show up to court but the algorithm
thinks they’re high risk so for instance it might recommend detaining them even
though they don’t need to be detained there are also false negatives these
people are the algorithm is going to recommend release but in fact there’s a
good chance they’re going to do something bad okay now these sorts of
mistakes again humans make these mistakes all the time this is not
something that only computers do any decision-making process including human
judges is going to sometimes err on both sides of this unless you release
everyone or detain everyone that’s and that’s the only way to avoid making both
kinds of errors good now which do we care about more false positives or false
negatives I have a complaint which is that whenever you read that someone says
such and such an algorithm is 83% accurate you should jump up and down and
scream that single number means nothing that sentence is not even grammatical
that can doesn’t convey what we need to convey about the algorithm we need to
think about what kinds of mistakes it makes which is worse well this is the
great jurist William Blackstone from England in the 1600s he says that he
would rather let ten guilty people escape then imprison one
innocent person okay so he is this is often quoted and so he is much more
concerned about the false positives here is another opinion the smiling face of
our former vice president and he was much more concerned about the danger of
releasing a bad guy and he didn’t mind if we kept a few innocent people in
Guantanamo so he is much more concerned about false negatives so this is a this
is a matter of debate right and okay so now I’m going to compare in this talk to
of these algorithms or as they’re called in this business risk assessment tools
that have been in the press a lot one of them is called compass it is made by a
private company that was called North Point and now equi vent it is based on a
big questionnaire you sit down with the the defendant and fill out a long
interview and based on that the results are in that questionnaire go into a
secret formula a proprietary formula which is this company’s intellectual
property in which they do not wish to disclose and then that gives various
scores of that will you know try to estimate estimate how risky a person is
then another tool which has been used I think in probably 40 jurisdictions now
except I think I say 38 on the slide including here in New Mexico in
Bernalillo County Court and the second District Court and there are discussions
about taking it statewide so it is generated by a foundation called the
Arnold Foundation which is very concerned about criminal justice reform
I mean my first slide complaining about how terrible the system is could have
been their first slide too so they’re sort of philosophically very interested
in that it takes a much simpler set of data just nine factors from the criminal
record there’s no need for an interview and unlike compass it puts them into a
simple formula that everyone knows it’s just a simple point system that you add
up so it’s transparent rather than pick good let’s let’s see what goes into
compass well let’s see there’s your current charges weapons etc how many
times have you been arrested before including as a juvenile who raised you
your natural parents let’s see if you if they separated how old were you was your
father arrested was your mother ever arrested do you use alcohol and drugs
are you under treatment for alcohol or drugs how long have you lived at your
current residence do you is there a phone at your current residence which
turns out to be used sometimes as a signal of having a stable residential
situation although nowadays cellphones count apparently um what about your
social environment a lot of crime in your neighborhood are there drugs are
there gangs do a lot of your friends feel the need to carry a weapon how did
school go did you complete high school what were your typical grades do you
have a job have you ever been fired and so on now here are some objections one
might raise two this approach one objection that we can’t directly
raise it doesn’t say what color are you right that’s not one of the questions
and and we have pretty clear laws in this country that that couldn’t be one
of the questions so it can’t be directly biased in that sense on the other hand
it does use a lot of types of data that are strongly correlated with race and
we’ve seen some of the dangers of this before in a practice called redlining so
in redlining it was the deliberate practice of not giving mortgages home
improvement loans and so on to african-american families who were
trying to enter the middle class do what middle-class people do build up equity
in a home built up wealth for the next generation and so on now some redline
redlining policies and they went on for decades were explicitly racist but some
were a little subtle they just said don’t give mortgages or loans to that
zip code it’s let’s see it’s um it’s declining its hazardous I guess nowadays
we would say it’s a little too urban that’s the current that’s the current
code for so that way I don’t actually have to say
you know white or black I just you know refer to the neighborhoods where certain
people live and if it’s a strongly segregated community that will do the
job I want to be clear here I mean I’m being snarky it’s late I’m
passionate about this topic I am NOT saying that the designers of compass are
trying to do this they’re really not I I don’t think that their intent is to be
discriminatory or to sneak discrimination in I’m just saying that
we’ve seen before that you can have a decision-making method which does not
use race explicitly but uses lots of other things that are correlated with it
and as a result you get a racially biased result okay so simply you might
if you were if you’d never thought about this before you might think well as long
as it doesn’t ask what the defendants race is it’s not going to be biased but
that’s not the case all right so back to compass it you know
it counts the rests so I complained at the beginning of the talk that
darker-skinned people are much more likely to be arrested in our country so
I’m just I have never gotten arrested hanging out with my friends on a street
corner that doesn’t happen to me I don’t know why so if you count arrests there’s
a real danger here that you’re putting in a racially biased signal there are
also all these environment questions about what kind of neighborhood you live
in and how did you grow up over which you have no control
there’s an interesting ethical question here again not just a math or computer
science question an ethical question it may well be the case that if I look at
the kind of environment you grew up in that that helps me predict whether
you’ll commit a crime but is it ethical for me to use that information even if
it would improve the accuracy of prediction because after all that wasn’t
your choice of where you were born or grew up so these are some of the sorts
of questions that computer scientists don’t think about enough frankly on the
other hand judges don’t think enough about computer science
so then you know then there’s do you have a job obviously that’s correlated
with income I was fired once so for for good reason – and it also looks at the
juvenile record again I think if you did a lot of bad things as a teenager this
probably is helpful in predicting whether you’ll do bad things as an adult
on the other hand in a lot of states you can go to a judge and ask for your
juvenile record to be expunged or sealed so that you get a fresh start a clean
slate so that it won’t count against you so
again do we want to use that data even if it’s available to us there’s also
this question about drug treatment I mean if you’re under treatment for drugs
and alcohol that does mean that you use drugs and alcohol or that you have on
the other hand it might mean that you’re really trying to turn things around and
so it’s not clear whether that should count against you or maybe for you and I
don’t know which it does because the formulas secret okay good now let’s look
at the Arnold Public Safety assessment it just uses nine things interestingly
if it does use your age so I’m sorry young people commit more crime crime
takes a lot of energy it it doesn’t it doesn’t use your gender which is
interesting because compass does use your gender but they decided not to it
uses your past convictions but not your past arrests but by the way I should say
it’s one of these abbreviations it’s trying to predict three different things
FTA means failure to appear that means you didn’t show up to court when you
were supposed to NCA means new criminal activity that means you were arrested
for in before your trial not exactly the same as having done something bad but
maybe that’s the best thing we can do and then there’s new violent criminal
activities that’s what the that’s what this is trying to predict so again oh
and by the way here’s the formula so you see on the right there these point
values so for instance a pending charge at the time of the offense that’s three
points so it’s a simple formula you just add these things up and you get a score
so it’s it’s publicly known it’s a simple point system and it uses past convictions and not arrests
which are hopefully less biased does not use the juvenile record uses age but not
gender employment education or environment so how well does it work
great so you can kind of see where my sympathies lie between these two systems
how well do they actually work what is that accuracy thing mean anyway
well here are a bunch of people I worked very hard on these animations so I hope
you like that here are a bunch of people so what these systems do is they assign
each person a risk score Oh figure out whether they’re red or green so if we
line them all up with the lower risk scores on the left and the higher risk
scores on the right then you might have some policy that says well you’re too
far to the right we’re gonna keep you in jail if you’re too far to if you’re far
to the left we’re gonna let you go and so on now if this score were perfect all
the green people would be below some particular value and all the red people
would be above some particular value in that case just let all the people above
below there go keep all the people above there and you’ve made no mistakes of
either kind no false positives no false negatives but of course the situation is
much more like this and so here are a bunch of false positives so these are
people where if you set your threshold for detention at that dotted line you
would be keeping all those people and maybe you shouldn’t be so if that
concerns you because you’ve been reading that quote from William Blackstone you
could raise your threshold and say well I’ll only detain the people way up there
okay though I’ll fewer false positives then of course when you do that you have
now just let a lot of other people go that maybe you shouldn’t have so you now
have more false negatives so no matter where you put this threshold you’re
going to make some mix of these two kinds of errors and again there’s just
no avoiding that and again this isn’t about computers if a judge consciously
or subconsciously is estimating how risky someone is and in some sense
declaring a threshold they’re in exactly the same pickle right it’s just a fact
about decision-making in the face of uncertainty so so we
could move it all the way it back down have very few false negatives and then
we’re detaining almost everybody I guess that’s the Dick Cheney approach okay now
there is a single number I was railing before against single numbers but there
is a single number that we use to wrap up the accuracy of these scoring methods
it’s called the AUC computer science love these three-letter abbreviations
they drive me up the wall a you C stands for area under the curve
and if we had 15 more minutes I would tell you what curve is the area under
but I’m not going to um I can give you a nice interpretation of it though
although when you see it you’ll think gee that’s sort of odd here’s what we do
we separate the red folks in the green folks and now we ask suppose I chose
randomly a red person a genuinely dangerous person and I chose randomly
one of the green people the AUC is the fraction the percentage of time that the
red person will be correctly scored as more dangerous more risky than the green
person that’s what it is here is another pair that it gets right
here is a pair that it gets wrong so in this case the green person the good
citizen has been incorrectly ranked as riskier than the red person all right
well how good are the AUC s of the things we’ve been discussing well in
fact they’re not that great at least not by the standards of other fields so
according to compass to north points on promotional materials and to follow-up
studies it’s 69 70 about 70 percent so about 70 percent of the time the
genuinely dangerous person be ranked above the genuinely less dangerous
person and about thirty percent of the time it will be the reverse okay so that
means about 30 percent of the time these pairs would be ranked incorrectly the
Arnold PSA according to a follow-up study from Kentucky
I should say to their credit the Arnold Foundation has been very active in
funding follow-up studies they want people to do follow-up studies in
different jurisdictions it’s actually a little lower 64 to 66 percent two thirds
so if we choose a random red person or a random green person that means we’re
twice as likely to rank the red person that’s more dangerous but only twice as
likely one third of the time we get it the other way it’s interesting to
contemplate although I don’t know if this is true that there’s a little bit
of a trade-off here between accuracy and transparency remember I was asking you
all these conundrums conundrum conundrums about well should we use the
juvenile record if it will improve accuracy or shouldn’t we because we feel
that that would be wrong well so maybe this is something to do with the fact
that the Arnold PSA is a simple point system and that it doesn’t use as much
data but the fun thing is there’s this thing called Mechanical Turk where you
can just ask random people on the internet questions and so somebody did
this and they found that random people given roughly the same type of
information that the Arnold PSA is given get as good or a little bit better
accuracy about 70% by the way those random people did not solve the racial
bias problem okay so it’s not that their decisions were glittering ly wonderful
they were problematic too okay good now I just want to mention that in other
fields of artificial intelligence like those medical images and so on I showed
at the beginning we’re used to a you sees that are in the 90s or the high 90s
in medical diagnosis we don’t use a test unless it’s AUC is at least 80% or so so
these numbers don’t look that great to me on the other hand maybe this is the
best we can do these are people we’re trying to predict people so maybe in the
social sciences we just have to accept that there’s going to be a lot of
inaccuracy but we should be aware that there’s a lot of inaccuracy even if this
is the best we can do all right so anyway once you have these scores you
then lump them into categories for instance you might go from the bottom
10% to the top 10% and transform the score into a
1 through 10 that’s what compass does Arnold uses at 136 and then you might
divide these into say low-risk medium risk and high risk categories and then
you might detain the high risk people including some false positives release
some low risk people including false negatives the medium people you might
release them but require some kind of supervision that they check in or even
give them an ankle bracelet to track their movements or something like that
alrighty so now what is what is this debate about fairness that came up so
ProPublica is a data journalism outfit kind of a new type of journalism that’s
very mathematical very data-driven and they had this huge splash of an article
arguing that the compass system it was specifically about the compass system
was racially biased and what they said was if you are a not so risky person
then you are much more likely to be falsely labeled as a risky person if
you’re black and indeed three times as likely so it was a quite a big
difference this caused a huge stir but then there was a very interesting debate
after this this was far from the end of the story
some independent statisticians including many that have been involved in trying
to improve the system came back and said no look you guys measured the wrong
thing this isn’t the right definition of fairness and then there was this
discussion of well is there any way to be fair and what does it mean to be fair
so as I’m gonna show you next even defining what it means to be fair is
trickier than you might think here are some more people new visual
metaphor instead of red and green some of these people have halos good citizens
some of them have horns this is a terrible cartoon this is terrible people
who miss their court dates because they couldn’t find childcare for their kids
and they were terrified of losing their job or they didn’t realize that there
was going to be a warrant out for their arrest if they didn’t show up are not
Devils and they don’t have horns so I’m sorry I was playing with keynote and I
needed some shape and okay all right forgive me but let’s proceed
so the algorithm again classifies some people into low-risk the low-risk
category and some into the high-risk category in reality there are these
gradations but I’m just making two categories to keep the discussions
simple and if we knew the truth about all these people well there would be
some who were correctly labeled so on the left you see the good citizens that
the algorithm correctly labeled as good citizens on the right the people with
horns that it correctly labeled as higher risk and maybe it got on average
you know overall 78% but I told you not to focus on single numbers like that it
also made some mistakes on the left we have these false negatives on the right
we have these false positives okay great so now let’s look at the people that it
has labeled high risk there are three false positives there those folks with
the halos so I hope you can read that that red can you still read that okay
good so out of the 12 people labeled high
risk three of them aren’t actually high risk so we call this this fraction 3
over 12 the false discovery rate some other people talk about sensitivity and
specificity and you can look it all up on Wikipedia and whatever but I’m gonna
call it the false discovery rate so if you’re a judge and you’re looking at
someone who has been labeled high risk in this scenario which of course is
totally made-up the chances would be 25% that the person you were about to treat
as high risk actually deserves to be treated as low risk good here is another
thing though which is a little subtle and I want you to hold on and follow me
here there’s another fraction we could calculate which is look at all the
people who have halos and look at the fraction of them that were labeled
high-risk ok this has the same numerator as before the thing on the top
I know not everybody likes fractions but the numerator is still 3 these 3 false
positives but the denominator is no longer the 12 people that were labeled
high-risk it’s the 15 people who are in fact
low-risk this is called the false positive rate and it’s very easy to get
confused about which about which is which or why they’re different but this
you can think of as if you are a nice person who’s been caught up in the
justice system this is the probability that you will be labeled high-risk ok
subtly different from the fraction we talked about before good so this is all
well and good but now let’s say there are two parts of the population two
groups the purple folks in the green folks so now let’s look at how these
fractions work within each of these subgroups so among the purple folks we
have these two false positives and that’s two out of the six purple people
with halos so that means if you’re a purple person and you have a halo the
chances are one third that you’ll be unfairly labeled high-risk okay that’s
the false positive rate for the purple people on the other hand for the green
people there is one false positive and that’s out of the nine green people with
halos so they’re false positive rate is only one ninth about 11% well this is
basically what pro-public has said they said if you look at data using public
data from Florida and you looked at what compass estimated and there were some
you there are some holes you can poke in what they did if you want but basically
what they said is that compass has a much higher false positive rate among
African American defendants than among white defendants but then came this
counter argument saying no no no that’s not right the real question is if the
judge is looking at someone who has been labeled high-risk how often is that
wrong so here are eight eight purple people
labeled as high-risk two of them two out of eight which is one-quarter 25% have
been labeled wrongly and among the green people
one out of four again 25% have been labeled wrongly so if you are the judge
well then in that sense it’s not biased the chances that it’s wrong or 25%
regardless of the color of this person okay that’s interesting because the
thing before about you’re much more likely to be falsely labeled high-risk
if you’re purple was it was pretty convincing so well which is right I mean
this is what people call predictive parody but this the difference between
these two false positive rates is a case of what we often call disparate impact
it does seem like this system overall is affecting the community of green people
very differently from the community of purple people so which is right well it
as I’ve been trying to say I think it really depends on your point of view
whether you’re the judge or the defendant are you are you the judge
trying to figure out gee have I gotten good information here from this
algorithm because after all I’m about to make a decision or are you a defendant
wondering how am i treated by the system in which I live so again there’s the
judges point of view maybe I went a little crazy with your animations
there’s the defendants point of view okay so here’s the kicker can’t we have
both notions of fairness can’t we get both the false positive rates and the
false discovery rates to be the same for both groups No
except in very artificial situations where the algorithm is perfect which
will never happen or when the two groups are statistically identical with the
same rates of well halos and horns so this kind of knocked a lot of computer
scientists sort of back in their seats a little bit so they had been working on
designing algorithms that achieved predictive parody we’re designing
algorithms that avoided disparate impact and proving theorems about them to show
that they would work to show that they would be fair and then
someone comes along and says actually there is not agreement about which of
these things is the goal which of these things is the proper definition of
fairness and you can’t do both at once and some narrow-minded computer
scientists said well if those ethicists will just figure out what they want then
we can arrange it tell us what you want to optimize well you know give us
whatever you want accuracy plus ten percent of this plus 20 percent of that
constrained by that we can handle it but you guys don’t even know what you want
well of course the more broad-minded computer scientist said gee this is
interesting we should start reading Law Review articles and and philosophy
articles and hopefully those people will still get their PhDs and jobs okay so
where does this leave us where does this leave us what you know as one individual
who hasn’t even published in this area who’s just been trying to educate myself
about all this what do i what do I think we go from here where do I think we go
from here well I think a really basic principle is that there should be no
black boxes in this business I think that there shouldn’t be proprietary
algorithms that are secret I see no excuse for that
certainly not in the justice system and actually I think it’s hard to think of
examples in the public sector is there a problem sorry oh there’s a problem last time we just unplugged it and
plugged it back in which is the cure for all modern technology shall I try that
yes we should try that with the justice system okay okay good yeah I don’t see
why we should use algorithms that are secret I one thing which is definitely
no excuse is well this is the best did this is the business model in the
intellectual property of my company so you can’t reveal the algorithm to which
I think public decision makers like judges and court administrators should
say well then we don’t want it so the public has every right to say we’re
going to use open-source algorithms that are that we can explain to people and I
think there’s an interesting analogy here with voting machines right so some
of the companies that make voting machines do not want to reveal the
software inside of them because they say no no no then people will be able to
hack them well it turns out voting machines are really really easy to hack
and if you want them to be secure than most people like me think the best way
is to make the software completely open-source so that everyone can see how
it works and analyze the potential vulnerabilities and figure out how to
fix it so that kind of openness rather than secrecy is actually a better
approach to security there but of course also if an algorithm is open to everyone
we can do follow-up studies we can have audits by independent data scientists
who can test it for various types of bias check to see if it is accurate on
our local population as it was on the national population for which it was
originally created so I think that’s something we should want now the Arnold
algorithm is publicly known and yet I could I couldn’t complain a little bit
because they trained it they built it around a large national data set and
that data set is not publicly known now to be fair they got this data from lots
of different jurisdictions and so they had agreements with those jurisdictions
and of course there are privacy concerns and yet as an independent scientist I
don’t have my hands on this I’m not able to go in and figure out you know exactly
how well it performs in addition oh geez okay I think it’s I think it’s static
electricity okay good in addition I don’t quite know how they came up with
the algorithm you know how did they come up with what number of points each thing
is worth like having a current charges three points or whatever I’m not sure
how they came up with that and I’d like to know I’m not saying they did it wrong
I just like to know I’d like to know the thinking behind the algorithm good what
else I only have 48 of these points by the way so you’ll get a point number to
embrace uncertainty humans are terrible at dealing with uncertainty we want to
be sure about everything and after we make decisions then we go back and
retroactively lead aside we were really sure even though we weren’t that’s
called confirmation bias so here is how computers are portrayed in popular
culture this is from a really fantastic TV show called person of interest by the
way and in it is this great computer and says see 97% confidence that a violent
crime is about to happen well again that’s just not the case if you look at
even the highest categories that these tools produce even they’re on in one of
the follow-up studies from Kentucky the probability of a violent crime at the
highest category was 3% that’s not nothing
I mean 3 percent of a terrible violent crime is a lot but it’s not 97 and it’s
not even 20 so now should a judge get a piece of paper that says something like
this in a sample that ended in 2015 26% of the defendants that are similar
to the one standing in front of you were really royal and 3 percent of them were
re-arrested for a violent offense studies are ongoing in your local area
to see if this is true of your local population as well well I expect that a
judge who’s very busy would be a little bit ticked off if this was how
things looked on the other hand this is honest this is the situation and instead
what judges see is this this person in front of you was a six they’re red or
orange they have a higher risk of violence recommendation do not release I
really have mixed feelings about this because on the one hand this system
needs to be usable and I understand of course the need to produce things in a
visually attractive form as you can tell from my talk but but this conveys
nothing of the uncertainty behind the algorithm right it doesn’t say well we
think there are six but they might actually be a four or maybe a 4.2 and
bla bla bla it just says there six orange red stop
sign and you know in a perfect world what I would like one thing to happen
from artificial intelligence is to help us reason about uncertainty right to be
kind of the the I’m not sure if they’re angels or Devils are both but things on
our shoulders saying this is probably the right decision but we can only be
sixty percent sure we cannot be 99 percent sure deal with it okay and I
think that as humans we need that kind of thing to make us a little bit more
modest about our ability to predict the future and make perfect decisions
alright here’s another one so we’ve been talking about these
numbers these scores well we saw that it’s not a perfect science but that part
is at least science it is at least statistics but then there’s another
thing that happens which isn’t science at all it’s policy which is that once we
get these scores well we have to translate those into recommended actions
and so there’s a matrix like this where here’s the failure to appear score
here’s the new criminal activity score down here RoR release on their own
recognizance just let them go they’ll come back this okay up here released not
recommended in here various colors corresponding to release with different
levels of supervision now this matrix was not
that they may I think the Arnold foundation may have provided a kind of
template but this is not their algorithm this is something that was formed by
local decision-makers local stakeholders the DA the police chief public defenders
and so on and it’s really these are policy questions what should we do with
someone in this category now the thing about this is I’m a little worried about
the future if politicians get involved and and forgive me if there are any in
the room I respect your work I’m and I’m glad you’re here but so for instance in
Santa Cruz there was already one case where it was a small change but it was a
change they said let’s let’s up the level of supervision at one of the
squares of this decision-making framework that was a local policy
decision the algorithm didn’t change the actions that result from the algorithm
are what changed in New Jersey there is a lawsuit which is actually the Arnold
foundation itself is being sued and what happened was the algorithm recommended
releasing someone because they got a low score according to the algorithm and
then they committed a terrible crime and so there’s now a lawsuit saying you are
liable for this release and one of the kind of upshots of this is the
suggestion that well remember one of the things that went into the algorithm was
was this were you arrested for a violent offense well it’s partly up to the local
jurisdiction to decide what that means so based on this particular case they
said well anytime there’s a gun anywhere nearby that should be a violent offense
gun in your glove compartment violent offense well this would completely
change the scores a lot a lot of people would suddenly up higher scores if we
change the definition that way but you can see what happens right in the
pressure of politics we want to keep everyone safe but we want to often run
on a law-and-order platform just as a cartoon you can imagine coming saying to
keep our kids safe I’ll detain anyone who’s a three or above and my opponent
will let them go and they’ll do terrible things
well I mean that we’ve all seen that kind of thing happen in politics in fact
it happened in the 80s when we tried to make sentencing for drug possession more
standard across the country and to avoid racial disparities we decided we’ll get
these sentencing Commission’s together and they’ll come up with what the
appropriate sentence should be and then politicians said Oh aha crack cocaine
should be a big minimum sentence so it’s tricky once you put these systems in
place if you don’t insulate them a bit from the wins of political change
kind of like the judicial branch is supposed to be insulated from the wins
of political change according to our buddy Alexander Hamilton they could get
warped and weaponized and then we could say ah it’s the algorithm doing it but
actually it was a political tweak that did it so I think this is very tricky I
mean when I look at the when I you know this group of people who got together
and figured this out I trust them but to make balanced decisions I’m just hoping
that no clever highly motivated politician decides to crash the party so
in fact there’s a great example of this our friends at immigrations and Customs
Enforcement have an algorithm it’s called the risk classification
assessment and according to their own internal reporting it’s terrible it was
expensive and useless and didn’t help at all but they have it so they decided hey
let’s remember that threshold let’s move the threshold all the way
down to zero so that the algorithm will recommend detaining everyone the
algorithm said so it said I should detain you so this is the kind of thing
I’m worried about and it’s the danger of as these authors
said giving a scientific veneer to political decisions right so that’s
that’s the question so we have to be very aware of this boundary okay I’m
almost done I’m almost done what about explaining or contesting decisions why
am I in jail the algorithm said so well our friends over in Europe came up with
something called the general data protection regulation or GDP are a month
or two ago you’ve got a lot of really annoying email
asking you to reconfirm reconfirm your permission to receive a lot of really
annoying emails that was the gdpr at work
but it has some other stuff as well it says the data subject that’s you you
humans have the right not to be subject solely to automated processing you
should have a right to obtain human intervention to contest the decision you
should have a obtain an explanation to challenge it and some meaningful
information about the logic behind the decision so what does this mean what is
what is a good explanation well this is tricky because even simple algorithms
combine lots of different factors you know which one put me over the edge well
I talked to I talked to one defense attorney who was able to use the
transparency of the Arnold algorithm in a hearing to say well your honor some of
the information in my clients criminal record is incorrect so run it again with
these inputs instead and see how that changes the result so this is one of the
benefits of having a transparent algorithm it would it allows you or your
defense attorney to challenge the inputs to it you know or you could say if it
was an algorithm that used the rest you could say well that arrest was frivolous
the charges were dropped it shouldn’t count let’s run it again and see what
happens so again this is it’s important that everyone had access to the
algorithm it could actually be a bit of a black box but we all have to have
access to it I’m not I’m gonna skip this but one of the nice things about this is
you can sometimes ask algorithms why did you make a decision and sometimes you
learn interesting things like well I was told to figure out which were the Huskie
pictures in which were the wolf pictures and I realized that the Husky pictures
were the ones with snow in them did I get it right
so this is a good example because sometimes algorithms are finding these
sort of pseudo patterns in the data that we give them which are not actually
learning about the thing we wanted them to learn about okay good why do we care
so much about prediction anyway I mean you know why am I so focused on
predicting whether you will show up to court if I release you
isn’t real point to make sure you show up to
court so you know most most failures to appear are not people jumping on a train
and trying to escape justice they’re not abscond enlike I said they’re you know I
mean yes I’m kind of a liberal guy so maybe I’m a little biased but I’m told
that they’re mostly people with kids people with jobs people who don’t
realize the consequences of not showing up people who were sent a physical
postcard but because they move around a lot they never got it so okay how about
doing things like sending two people text reminders on their phones or
providing free transportation that cost a little money but it’s cheaper than
jail evening or weekend courts a lot of jurisdictions in New Mexico here have
been experimenting with these safe surrender or warrant resolution courts
where you can go in and in a fairly friendly atmosphere find out if there’s
some charge against you and try to resolve it instead of you know running
around and waiting until somebody has to catch you so these all seem like good
things maybe we could even provide child care in courthouses for people with kids
I’m sorry what am I thinking we would never do anything that sensible so you
know I to me this is this is not rocket science it’s more like climate science
what do I mean well of course climate scientists are working very hard to use
more and more powerful computers to get more and more accurate pictures of the
future climate on the other hand we already know what to do we already know
we need to have a low-carbon power grid and a low carbon transportation sector
and we already mostly know how to do it we can already pretty much afford it so
we’re just dragging our feet so this is low-hanging fruit you know know no
reason not to move forward and it seems like in New Mexico and elsewhere there’s
some good creativity here okay finally finally well what’s the worry so I I’m
reasonably happy with for instance the way the the Arnold PSA is being deployed
and so on because I’ve talked to a lot of people involved in it it seems like
good decisions are being made there are still questions that can
be raised but as algorithms spread into more and more areas of human
decision-making I am worried about this thing this new thing called tech washing
tech washing like other sorts of let’s see
well tech washing is it’s when human biases like I said get wrapped up inside
an algorithm and then they’re in this gleaming metal box that everyone thinks
is perfectly objective and it continues it perpetuates the bias that were in the
human that was in the humans that it was trained on that it was trained to learn
from to imitate so this years data effects next year’s algorithm and vice
versa there are feedbacks so one big issue is predictive policing where you
send more police to a certain neighborhood because you think you need
to send more police there well there’s that might be a good idea but there is
also a danger of a feedback where they find out about micro more crime that
neighborhood rises even higher on the list of neighborhoods are concerned
about and you focus too much attention there I’m not saying this might happen
I’m not saying this will happen but it certainly might happen it’s one of the
potential dangers in a feedback between human decisions and computer decisions
so I think the real question is will these feedbacks ease our biases or make
them worse you know psychologists like Daniel Kahneman and
and turski and so on have revealed to us all these biases lurking in our
decision-making eye I hope you’ve read these books books like Thinking Fast and
Slow by Kahneman you know and we we make snap decisions and then defend them to
the death we get focused on one piece of evidence and then ignore all the others
we certainly ignore people who aren’t our friends when they disagree with us
and sometimes our friends in the orab we say you’re not or you’re not my friend
anymore so you know we’re not so great at thinking some of the time at making
decisions we have a lot of biases and so let’s return and this will be the
conclusion to Google Translate here is a good sentence she is a doctor I have
translated it into Turkish Oh better doctor let’s translate it back oh he is
a doctor what happened well don’t be too angry with it the problem is that in
Turkish that thing Oh is a gender-neutral pronoun so when it
translates it back it thinks G based on all the text I’ve
seen from the past which of these two translations is more statistically
likely oh definitely he is a doctor of course it also works with he as a nurse
turns into she is a nurse for the same reason let’s see what happens when we
ask Google Images for images of doctors well it could be worse actually there’s
certainly a lot of salt and pepper there but let’s look at the image search for
nurse okay fairly homogeneous so we are training our algorithms on data from the
past which is completely suffused with all the stereotypes that we’re
constantly fighting against right and so there’s a real danger that these
stereotypes these biases will get written into the algorithms and then
we’ll use the algorithms and assume they’re giving us objective information
and that’s what I’m concerned about just to be fair let’s see what it does with
scientist ah well you can tell first of all that all scientists were white lab
coats that’s certainly true you know I tried this and I got a lot of images of
women and I was happy in a way but then I realized you know what that’s only
because a lot of people are working very hard right now not hard enough but
working hard on changing the perceptions and stereotypes of science in order to
try to diversify science so these images of women are there on purpose because
human beings have worked hard to diversify this picture algorithms are
not going to make things like this better for us this kind of social
improvement will not happen automatically it will only happen if we
make it happen on purpose by changing away from the statistics of the past
instead creating a better future thank you very much okay any questions where’s my water hi
would you see techniques like inverse reinforcement learning as being
something that could be useful there or does an algorithm that has enough
complexity that a single human mind can’t really you know intuitively
understand the whole process turned into too much of a black box okay so the
question was about reinforcement learning which is one of the techniques
for training sorry inverse inverse reinforcement who’s where the AI is
essentially trying to imitate human words a human decision so it’s taking
human decisions and you know determining whether or not it’s an optimum out-out
approach based on what the human tells is well I think it would it would depend
on the human right and I think so if you if you used input from humans who were
for instance very consciously trying to avoid some form of bias then the
algorithm would learn that and if you didn’t it wouldn’t my concern then would
be if the algorithm goes out into the world and we forget this process by
which it was trained if we don’t sort of kind of tag it with by the way it was
trained on this data in this case where the data came from a human behavior then
it might be put out there and then given this sheen of objectivity right sort of
turned into this depersonalized objective thing so it might you know
this is a little bit how people are trying to get self-driving cars to work
right the the the car sort of watched as a human drive for a while is that is
that right is the inverse reinforcement learning used there because I’m not an
expert on it okay III don’t know either so um well
but I think there would be no I haven’t thought about it before but I think
there would be no silver bullet I mean I I think that it would be good to kind of
constantly test to see whether the algorithm is biased in certain ways I
mean I think part of the cultural problem is that for the engineers who
work in machine learning you give them a data set it’s a fixed data set which is
not quite what you’re describing but it’s a fixed data set then these very
clever methods to find patterns in it and then they produce their algorithm
and then they’re done right then they draw a line they say well I produce my
algorithm I did my job I wrote the paper or whatever and now
the algorithm is out there in the world and I think the problem is we need to
have we need to understand the entire lifecycle of these algorithms and once
it’s out in the world we need to constantly monitor it to see if it’s
doing what we hoped it would do and possibly retrain it or add to its
training and keep that lifecycle going but that’s not how the profession of
machine learning right now is sort of set up at least on the academic side
maybe maybe that is how it’s set up on the industrial side I’m not sure but you
know with Google Images and Google face recognition I didn’t show you some of
the worst examples there was one case where some Google employee who happens
to be black was trying out his own face and it said aha I recognize you you’re a
gorilla oh my god so of course that was embarrassing and of course they then
tried to tweak their their databases that their training their image
recognition ons that these databases themselves would be more diverse and so
on but it’s it’s complicated and if you’re naive about it then the
algorithms will make a lot of embarrassing mistakes
they’ll be maybe 90% accurate but the 10% where they’re wrong will be really
embarrassing so anyway I think it’s a very it’s an ongoing
process okay one way over here what about the aspect of the social
repercussions of overtime of the false positives you’ll have the large part of
the population being to distrust the system and be very upset about not want
to cooperate that would be terrible I’m sorry it’s dark humor I mean I I
think that’s kind of what’s happening right I mean I there are a lot of people
who what can I say I mean if I see a police officer coming toward me I’ll be
like hi but for a lot of other people in this society that’s not how they’re
likely to feel about the entire apparatus of law enforcement and and and
the legal system so well I think it’s a it’s a basic issue of justice in our
society of our what if we want our society to function well then everyone
should feel a high degree of faith that they’re going to be treated fairly and I
don’t think that’s true currently so and in particular if there
are parts of the population where I’m sorry I don’t have the statistics at my
fingertips but we’re literally ten twenty percent of them are incarcerated
that’s a really bad thing so yeah does anyone have that actual number like for
young african-american men what fraction of them are in prison it’s it’s shocking
so anyway sorry or have been in yes at some point so and
again I mean even being detained temporarily there studies showing that
studies showing that even being detained for several days has an enormous impact
on your life not just psychologically but practically as you scramble to keep
your home and your family and your jobs from falling apart so question in the
back here Chris hi so my my friend than I eaten upstairs we picked up on a
reference and it may or may not be relevant now but it was Schrodinger’s
cat and it seemed to have been a glitch in the performance but again I would
wanted to ask you I’m sorry I was that when the projector went wonky yes and
there was a box oh yeah someone else made the joke about Schrodinger’s cat it
wasn’t me sorry was the cat oh that was an actual cat yes yes sorry that was not
Schrodinger’s cat that was one of my past cats sadly sadly no longer with us
I wish I wish I could say you know both or neither but I am anyway okay Chris I
have one over here okay the American Institute architects recently had a
conversation about banning the design of solitary confinement facilities as a
moral issue and given the fine line in moral social justice issues and moral
issues has the AI industry contemplated a similar concept of introducing a moral
structure behind the responsibility of being an engineer yeah that’s a great
question there is a lot of foment around this so companies like Google and IBM
that have entertained contracts with intelligence
agencies or the military or whatever often find that then there’s some
fraction of their employees that protest against that at the moment one of the
issues is internal to Google is whether they’re going to make a version of
Google with which the Chinese government would be more comfortable and so there
are a lot of Googlers who are saying no way we can’t we can’t do that that would
violate our ethics so there is a lot of this happening they’re also a bunch of
organizations cropping up one is called AI now one is called the Human Rights
data research group I think and there are also there’s kind of an interesting
cadre of young machine learning and stat and statistics people who are working
for nonprofits that are studying human rights and discrimination and so on so
there’s a lot of that kind of stuff bubbling up from below and it will be
very interesting to see how it how it evolved at the same time if you you know
we could have of course another three-hour discussion about the whole
issue of censorship versus openness on social media another huge discussion so
if you heard about this memo from within Facebook where they were advising their
moderators that white nationalism was okay but white supremacy wasn’t it’s
clear that they’re just in over their heads I mean you know these kids they
have no idea what’s you know what to do and so all of this is at a very early
stage people have built these companies they’ve ballooned into these massive
machines that clearly give a lot of people a lot of enjoyment and then there
are these questions about them that no one is quite sure how to get a hold on
and I think that that’s true in AI as well I was interested in your desire for
transparency in the algorithm for the example you presented I wonder
what you think of transparency and search algorithms such as Google’s
approach by my experience is you can never find what you want because it’s so
far down the search list it’s not even part of the thought process yeah it’s
interesting Google Google when it first started was a very simple and
transparent mathematical algorithm which you know we taught to our students in
computer science class involving these eigenvectors and it was very nice but
for a long time now they’ve added all sorts of things on top of that that are
very different from it and that I certainly have no idea how it works and
part of that was to avoid certain types of abuse certain types of gaming of the
original algorithm part of it was probably to please advertisers there you
know obviously there are lots of lots of motivations at work so indeed yes I have
no idea how Google works now now you know Google is a private company it’s a
huge private company and it is almost a monopoly on search not quite so of
course there’s a lot of discussion about you know should it be you know there’s
some parallel world where it’s a regulated utility right I don’t know how
that would happen exactly it would obviously take an enormous political
will I don’t know if the tax payers would have to buy it that could get
expensive but you know there are media companies
and there are you there are communication utilities and then there
are these new things called platforms that are sort of in a legal limbo that
are not regulated but also not legally responsible for most of the content put
on them no matter how horrible it is except maybe in a few cases and maybe
now there are court cases in Europe and it’s confusing so I don’t know where all
that’s going to go but I can’t help you with Google I’m sorry
I admit it works fairly well for me but uh
you know yes one last question well in fact I have two questions first one
about understanding pick one or combine them somehow into one well I mean first
of all how do you know what what a green guy is because if you put somebody in
jail and then how do you know that you would not have committed to crime you
would not put him in jail and good it’s a more important one it’s one that one
was good and sorry it’s a more important one is that you need to use statistical
correlations if you want to make predictions and the question is which of
ones you are you really are morally entitled to you so of course a twenty
year old man is more likely to committed crimes and a seventy year old woman is
that would that be a good basis of making your prediction maybe a strong
musky a muscular guy is small likely to commit crimes and and a small frail
sickly person but what kind of statistical correlations would you be
allowed to use to make your predictions of course in every individual case they
may be wrong but nevertheless you need to have some basis of judging the
likelihood of a person to commit a crime well let me ask the first question first
so the question was you know if I don’t release you how do I ever find out if
you would have been a good citizen if I had released you well I don’t that’s one
of the problems right I mean I suppose banks must deal with this they don’t I’m
not going to find out if you are a good a good risk for loan unless I actually
give you the loan colleges deal with this I’ll never find out if you would
have succeeded if I had admitted you to my college if I don’t admit you
maybe if I shared information with other colleges I might learn something
I don’t know so there is a one-sidedness to this and there is this there is this
danger that you know I could say well i detained everybody and i didn’t release
anyone by mistake so this is one of the i think this is one of the trap
that we could get into now there was an interesting study done by some people at
Cornell and Stanford where they looked simultaneously at compasses
recommendations and what human judges actually did and there they were trying
to figure out well when judges do release people are they releasing the
right people in other words our judge is actually good at figuring out that even
though the two of you both of a score of five he’s nasty and you’re a good person
who just finished high school and is taking care of your little sister and
deserves a second chance and this particular study suggested that
human judges are not very good at this that that the the fraction of people who
went on to Rhea fend of the same it wasn’t any higher among the people sorry
it wasn’t any lower among the people that the judges use their discretion to
release which was which was a little alarming but yes there is there is a
one-sidedness this your second question about what correlations I mean you know
there’s there are all these classic stories about correlations between
sunspots and hemlines and so on and so what types of correlations should we use
when do we know that they’re genuine and when ethically should we use a
variable even if it is correlated and I think that’s something that like the the
public sector decision-makers need to decide if anything in advance of buying
an algorithm what variables they’re willing to put into it like they could
decide we’re willing to put in these things that are right there in your
criminal record we’re not going to be more invasive than that we’re not going
to dig deeper than that this is what we’re going to do and then
ask the algorithm designers what can you do with this data and only this data how
far does that get you so again it’s part of this this need to not just have the
machine learning people siloed over here doing their engineering and the decision
makers over here doing their stuff but really have both groups of people
understand more about the job of the other all right I think I should
stop there I really appreciate it while coming you

Leave a Reply

Your email address will not be published. Required fields are marked *