For those who are on audio, I want to acknowledge Samuel Noekowsky, who just spoke. This talk is basically what he said, except he said it in the French way with complex theories, while I'm going to give it in the prickly, analytical, English kind of Canadian, dry presentation. But I do want to acknowledge it.
For those on audio, and also for this in the room, this is third in
three talks. It forms a third part of a set. I did a talk just downstairs on
open content. I did a talk at the London School of Economics on Wednesday about
the role of the institution and the changing nature of online learning. This follows
from that. This addresses the specific question of assessment in open learning
and in informal learning.
Faking Our Way to Success
have here is a woman who was a dean at MIT for decades, receiving the
institute's highest honor, the MIT Excellence Award for Leading Change, and it
turns up she didn't have any degrees, not even a bachelor's degree.
Obviously, she's disgraced, et cetera, and she should not have done
that, but the interesting thing here is that she can do an excellent job and be
recognized for this, by a major institute of higher learning, and yet not have
a degree. It raises the question of the value of the degree itself.
On the other hand, we have the
possibility that many of us are just faking our way through. It's not just
faking our way through doing jobs without degrees -- although I do wonder how
many other people are out there doing that. It's faking our cultural knowledge
generally. People acting like they've read "War and Peace" when they
really haven't read War and Peace, or people referring to plot elements in
"Moby-Dick," like I do, without having read Moby-Dick, which I haven't.
Like Learning a Language
When we get into this sort of discussion, we wonder, is there a core, a
common heritage, which we each share, or rather is each individual, as we just
heard, a strand running through the fabric of culture?
It's interesting to think of culture in this perspective, as a
type of language -- culture as something that helps us communicate with
each other. Samuel (Noekowsky ) tried to use Greg Egan (as an example), an
attempt that failed miserably. I nodded at this because I'm familiar with Greg
Egan, but unfortunately I hadn't read that particular book so I felt badly
having nodded. These little cultural misrepresentations.
It's funny, because when he talks about knowledge and language in this
way, we have this sense that we're talking about knowing a language, knowing a
culture, knowing a whole set of facts, and if we could just get all these
facts, then we'd know what people know.
But, in fact -- Wittgenstein here is the expert on this -- a language
isn't just knowing a set of facts or knowing a set of rules. A language is
something much more dynamic, much more behavioral and complex -- but much more
like playing a game.
Looked at from this perspective, what
constitutes success? Playing the game, being a good MIT Dean of Admissions
constitutes success, rather than a demonstration of a set of facts. What is it
for one of us to be a success? Do we count our citations? Do we count the
impact factor of the journals that we're writing? Probably we shouldn't, right?
It raises a lot of serious questions about assessment, understanding
what constitutes success, what constitutes learning. Here's one question: We
have many more ways of finding work today, but it seems harder and harder to
find a job, harder and harder to match ourselves to a position. Why is that the
question: today's students leave a lot of data traces, as we've just heard,
from demographic information to how they read and highlight eBooks, et cetera.
What are the ethics of using this for the purposes of assessment?
Here's another one: everybody's learning experiences are customized.
Everybody is, as my wife likes to say, a special snowflake, where their
experience exactly matches who they are. They study exactly what they best comprehend.
So does this mean everybody gets an A?
Is open online learning or the MOOC, the massive open online course, or
informal learning in general doing the sort of thing that we need? According to
metrics, it's not. People aren't completing their courses. They're not
amassing these sets of facts. They're not proving their knowledge by taking
tests and getting degrees.
Nonetheless, MOOCs and open online learning generally are shifting the
definition of education away from its historical roots to a skills based,
instrumentally defined enterprise -- in other words, very much like speaking a
What We Need
What is it that we need? What is it that we're after? What are we trying
to get at when we're doing assessment?
Figure 1. Skills Gap
One way of looking at it is the skills gap. I don't know whether they
have that here in Europe. It's a big thing in Canada. The point of the skills
gap is you have a bunch of unemployed people. You have a bunch of open
positions. They don't match. The skills the positions need aren't the skills
that the unemployed people have.
Trying to bring people to the point where they can qualify to fill the
jobs. That's one way of looking at it.
Look at it this way. It would make sense for employers to just go out
and say "we'll educate people for the skills that we need, and then we can
hire them," but this
isn't happening. One of the reasons, I think, is employers don't actually
know what these skills are that they need.
They know that they don't have the people. They know that the unemployed
people don't qualify for the job, but they don't know exactly what it is that
would qualify for the job. It's just when they do the interview, when they have
a conversation with the person, it becomes obvious that they're not qualified.
An interesting phenomenon.
What's being recommended?
What has our government and business community come up with? Well, in Canada we
have a thing called the Canada 2020. It's a business oriented group. They're
recommending a learning outcomes assessment program, a council on skills and
higher education. We actually used to have one, called the Canadian Council on
Learning. Now they want it again.
They want education and skills for aboriginal peoples, whatever that
means. They want to "narrow the skills gap between men and women,"
again, whatever that means. They're looking for credential recognition and
skills training for immigrants.
I don't think any of these address the point. I don't think any of these
address the problem. The problem is how do you recognize, first of all, what
you need in society? And secondly, what people actually have in society.
Doing It The Hard Way
This sort of approach is trying to paper it over. They paper over the
cracks with committees and lists of skills, et cetera, but it doesn't get to
the core of the problem.
It's doing it the hard way. Compiling learning
task inventories to define sets of learning activities related to skills.
All kinds of work has been done in that area, although we find that when
people actually use learning task inventories, what they're useful at is
telling people what they don't know. It doesn't help them get to where they do
basis for instructional design. You start with what you want them to learn,
design an experience, cause them to learn it, build in some checks, and see
that this has all happened in the end. A very traditional approach.
That's basically what these councils and these skills programs are going
to do. That's what PISA did. That's what PISA did for 15-year-olds. The PISA
task, for those of you who aren't familiar with them...You take 15-year-olds
around the world. You give them a standard set of test questions (interestingly,
test questions not based on the curriculum of what they learn but rather test
questions based on some definition of what the organizers of PISA think they
should have learned by the age of 15. That's true.)
Apply these tasks, and then you get a league table. You have Finland at
the top, Shanghai -- which for some reason qualifies as a country -- at the
top, all the way down to "not us" at the bottom half of it.
Interestingly, the countries that are doing well in PISA are beginning
to doubt the whole program. Countries like Finland probably always doubted the
whole program. We have Shanghai, and China generally, looking
at whether they really want to focus their efforts on this anyway. Their skills
and qualities, they say, should also be acquired from a variety of activities,
not just studying and testing, such as play, online activities, and games
because they understand that knowing isn't just about getting a set of facts
that you can measure on a test.
Why are we emphasizing the test? Here's one
theory. This is the slide for any who are skeptics.
The per-student cost for testing is currently about $31 a student.
Multiply that by 50 million students in the United States, and you get lots of
money. Interestingly, organizations like "The Washington Post," who
promote testing, also run testing agencies like Kaplan.
is still crucial for economic development, for personal development, and
there is this sense in which skills build on skills. There is this sense in
which education is kind of cumulative. It almost feels like we're piling facts
on each other, even though we know it's not.
We have reports saying the time in school spent by a country's children
is directly related to the productivity of its workers. That appears in the
publisher's Pearson’s White Paper. I'd probably say it differently because I'm
not so concerned about the productivity of its workers. The point nonetheless
remains the same -- more education means happier people and a more developed society.
Outcomes and Process
The problem is we're confusing the outcomes of education, the test
results, et cetera, with the process of education. That's why it's significant --
that it talked about -- the time in school as opposed to, say, the test grade.
If they said the test grades are proportionally related to economic development
or productivity, I think they'd have a much harder time making their case.
Outcomes are hard. I read this
thing from Pearson this morning, "You can be certain and vague, or you
can be precise but not certain. You can't be both." You can be both, but
you can only be...Let me refine that. No, never mind.
The really interesting, useful outcomes like "understand" and
"appreciate" -- so much is built into the word appreciate -- they're pretty
much undefinable. They are, in a certain sense, ineffable. We could not
really express what we mean.
This is what Samuel (Noekowsky) appeals to, the ineffability, and things
like that. The language actually fails us because we think of the language as a
language, as a set of facts, structures, rules, et cetera, and this doesn't
There isn't a nice, neat set of concepts and principals linked to what
we mean by "understand," so you get behavioral outcomes,
"display," "recite," "define," but these are
based in rote. That's the problem with these tests -- you can fake them.
"The complexities matter." Gardner Campbell.
"When confident, simple, plain orderly advice is given about a subject
matter," he says, "I hear the sound of the hatchet replaced by the
sound of wood snapping as the branch I'm sitting on gives way."
It's easy to recommend simple outcomes-based ways of developing
education, but as soon as we begin to do that, we're undermining the
foundations of the educational system. This is the dilemma. One the one hand,
all of that is true. On the other hand, our knowledge is in here, and what's in
here is incredibly difficult to access.
Black Boxes / Making Stuff
Even if you slice open the brain -- and people have. I've heard this --
you still don't see knowledge. You see gray, messy, gooey stuff.
are we going to do? Do we continue
using the black box approach, which is based on "recite,"
"define," et cetera, or do we start making stuff up?
This is where I was worried about Samuel (Noekowsky) for a bit. I was
worried he was going to go into the land of making stuff up. When he started
talking about mathematics and dust, I sort of worried about that, but he's way
better than most.
Here's an example
of what I mean by "making stuff up." This is someone who I won't name
for his own protection. "Our brains need some way of deciding what to
encode and how to encode it, so as to retrieve it in a way which is useful. Our
minds solve this problem by encoding information along the affective context.
No doubt you have studied many papers that sound like this.
This is made up stuff. It seriously is made up stuff. What is the
affective context? I know, you can give me a definition, but can you give me a
definition that is operationalizable in any meaningful way? Can you give me a
definition which is observable in any meaningful way?
When you start saying things like "the brain needs some way of
deciding," how do you know this? Is the brain sitting there and saying "oh,
my, I need to find some way of deciding how I know"? No. It's a
hypothesis. It's making up some kind of activity the brain is doing, but the
brain is not actually doing it.
Male Audience Member:
How do you know that?
We know that because there's no way of knowing that.
Talbott. This is a wonderful paper. If you do get the chance, you should
read it. We have, even in biology -- which is what makes it shameful -- this
tendency to represent the human body, human systems, including thought, even
things like the circulatory system, as though they were machines, when
manifestly they're not.
Figure 2: The Heart as something other than a pump.
He has a wonderful account in this paper about the heart and how we
think of the heart as a pump. Very mechanical, right? That just probably is the
idea that we have in our minds, right? The heart pumps the blood through the
circulatory system -- except that if that's really what happened, the amount of
pressure that the heart would have to exert on the blood would explode the
tiniest veins and arteries in our bodies.
What really happens, he says -- and I don't have a good ground to
disbelieve it, and it is well-referenced -- is that our circulatory system is
more like the tides. It involves some liquid in the blood system but other
liquid elsewhere in the body. It's like the tides that sway back and forth. The
heart is a regulatory mechanism, not the sole thing that makes the whole thing
An example of this is that your veins in your legs actually play a role
in moving the blood back to the heart. The whole body works in this way. The whole
body works together as a mechanism for making the blood move. It's not simply
the heart. The heart is just one part of it. It's a complex, messy -- really
messy. Open a person and you'll see nothing but messiness -- totally integrated
organic system, not a machine.
Same thing for the brain. And that's how I know that.
That takes us to the next part of our discussion, competency-based
education, which is the big thing these days. Competency-based education,
as you know, offers credentials
based on demonstrated
proficiencies. Critics argue that it seems too much like training and it
focuses too much on outcomes.
My response to it is it takes a hard problem (whether somebody is, say,
a dentist) and breaks it into smaller problems, but all of which are just as
hard as the original problem. We've got one hard problem. We've created 10 hard
I wonder, indeed, whether personal learning even requires competencies.
Do we require competencies in order to learn? Is a discussion of our learning
at the same time a requirement for a discussion of the competencies that we've
one sense, yeah. You can't really do personalized learning without common
expectations about competencies.
Figure 3: Competency Pyramid (cf. Dale's Cone) Source: op. cit.
But this is only under certain circumstances, where we require common
expectations as an outcome of personal learning. What if that wasn't the case?
Maybe it isn’t the case. According to this study
from a competency-based education from the Higher Education Quality Council,
systematic comprehensive study indicating that the purported skills from
competency-based education translate into performance.
This Charles Ungerleider and his crew...I've had my criticism of
Ungerleider in the past. I still do, but that's an aside. He is, nonetheless,
correct about it. What are the competencies underlying performance in anything?
What are the competencies underlying a satisfactory high school education,
We think we've got a good handle on what these basic competencies are,
but we don't. We think we do, but it's an illusion. I remember so distinctly
sitting in an auditorium in the Southern United States. A man got up and said,
"We all know what the basics of a fundamental education are. They are
mathematics. They are language, and they are religion."
What's happening now? What are we getting? We are getting alternative
credentials. We used to have a system where we had high school diplomas,
bachelor's degrees, master's degrees, PhDs, and a few professional
certifications. Now, because there's so much information out there and so many
ways of interacting with people, we are getting a ton of alternative
As I say here, a veritable slew -- Qualt, for example, based on courses
developed by the Association of Accounting Technicians, et cetera, and brought
to us by the good people like Google. Harvard has created the Credential
of Readiness, which means that you are ready to take a Harvard education, I
ALT, the Association for Learning Technology, in Britain, is designing
badges as part of its ocTEL MOOC. On top of that, the badge issuing system,
which can establish whether the badges you are issuing are compliant badges. So
we have a credential for a credential.
Udacity, Sebastian Thrun's company, together with AT&T and 1.5
million of their dollars, is launching something called nanodegrees.
In Europe, the European Commission has the VM Pass,
a validation process
based on a combination of peer review and crowd sourcing.
Condé Nast - yes, the magazine publisher - is issuing
college credentials. The experts are going to be the writers and editors from
various magazines. Of course, the publisher will provide some financial
backing. It actually convinced some colleges to go on with the scheme.
The Achievement Standards Network is offering “open access to
machine-readable representations of learning objectives published by education
agencies and organizations including the Common Core State Standards.” Recently
acquired by Desire2Learn.
Learning Locker is an open
source Learning Record Store for tracking learning data.
What Are We Validating?/ Understanding
Grail is finding a sustainable and reliable method for the validation of
But -- and this brings us back to Samuel (Noekowsky) -- are we
validating the learning, or are we validating the person? Who are they hiring
when they hire someone, the learning or the person?
was, remember, jobs, skills, gap. They don't hire learnings. That's why they
don't just use the CV when they're hiring people. They're hiring the person.
You wonder why they would do this. They actually go through a process of
selection, where real people sit down and talk to job candidates. Why does that
We need a basic understanding
of understanding. This is part of my criticism of portfolio approaches.
When you're trying to show that you understand something, the answer to that
isn't really "let me demonstrate." The reason why "let me
demonstrate" isn't the reason is that it's too easy to fake. If you just
show a production of some sort and say "this proves that I know of quantum
physics, because look," people aren't going to believe that.
There is a sense, though, in which understanding quantum physics is more
about doing stuff, as I mentioned at the top of this presentation, than some
mental state, some set of facts in the brain that the brain needs to remember.
To know something -- and I've said this many times -- is to recognize. I
mean that in a very precise sense. What is it to know something? How many of
you are familiar with the books, "Where's Waldo?" Somebody says it's
a different name here. Wally, "Where's Wally?" Thank you.
Open up "Where's Wally?" or "Where's Waldo?" for
those listening in North America, Africa, or Asia. You look and you search and
you search. Finally, you see him. You recognize him. The next time you do the
same thing, it takes you no time at all. There he is. These things only work
Knowing is like finding Waldo. It's being able to recognize him in the
sense that you can't un-know it once you see it. Knowing is like picking out
the face of your spouse or your close friend in a crowd coming over in a train.
Easy to do. Incredibly complex process, but easy to do. You can't not do it.
That's the key.
It's a physical state. It is quite literally the organization of the
connections in our brain. Our brain is a perceptual mechanism. It's a
perceptual mechanism such that if we see something that we recognize or we
know, we get a cascade of neural activation, and we get this characteristic
"Ah, there's Waldo. Ah, there's my wife. Ah, there's a tiger." You
can see how it would be useful.
Knowing and Doing
It's "to do,"
in other words, rather than "to know." This is important. When we're
doing these assessments, when we're doing these measurements, we're after facts
and principles and things like that. Theories. Even language. Even a definition
This is a representation or a model of what we know, rather than what we
know. It's a tool, an aid to learning, an aid to understanding and
comprehension, but it is not literally the thing that we know. The thing that
we know literally is this recognition process.
How do I know that? Because our brain is a pattern-recognition machine,
not a digital computer. It's not even a pattern-recognition machine. It's a
pattern-recognition whole bunch of gooey complex stuff, interconnected neurons,
not a digital computer. The digital computer analogy is simply wrong.
Male Audience Member:
If that [inaudible 31:49] how can you do without [inaudible 31:57] ?
That's easy. Did you attempt to demonstrate something by asking that
question, or were you just asking the question?
Male Audience Member:
You see the distinction. A demonstration is a requirement intended to
satisfy a specific set of criteria. But to do is everything. It's a good
question, and it does lead into the next bit here.
Here's the secret
formula. It's not really a secret formula, but there's a blog over there
called, "The Rapid eLearning Blog," which is all based on "do a
few videos, and learning will magically appear." That's not true.
But this bit is true. How do you become, or establish yourself, or show
that you are an e-learning professional or anything else? First of all you
practice what you do. Secondly, you show examples of what you do. Thirdly, you
show what you do and what you learned.
Not just the examples. You show the process. You show the thinking. You
show what goes on behind the scenes. That's very much how I've structured my
career, and for me it's worked reasonably well.
Figure 4: My LinkedIn Network
Here's me showing what I know. This is my LinkedIn network, and it shows a
lot of connections. You can actually analyze what the different colors mean.
Most colors were actually put in by the computer. I didn't do it. The computer
For example, this green cluster here which sticks down is not India,
even though it looks like it, is... They're all connected to me, but they're
also all connected to each other. It forms a natural cluster. Since it forms a
cluster, the system recognizes "hey, that's a cluster. Let's give it
green," so we have green.
But me showing what I know is me being on this network doing what I do,
whatever I do. All of those people see it, and they talk about it. Maybe they
talk about it, or maybe they ignore it. Who knows? They do what they do.
This is the wave of the future. There's a system called SCROLL, a System for
Capturing and Reusing of Learning Log, and basically it's capturing your
learning as you go along.
Think about what happens when we add learning to the Internet
of Everything. The Internet of Everything is the Internet of people, of
resources, and of things. We're hearing about the Internet of things, but
what's really interesting is the Internet of everything.
I used to talk, many years
ago, about this fishing rod that teaches. All you do is you take your fishing
rod and go out and fish, and the fishing rod has internal sensors that detects
how you cast. Then it remarks to you, "You've never fished before, have
you?" or something like that.
The fishing rod gradually teaches you, but what's more interesting is
that the fishing rod and you are interacting together. It has sensors. It's
detecting what you're doing, and it's using that feedback to help you learn.
It was always theoretical, but a month ago I saw a TV show. They didn't
do a fishing rod. They did a tennis
racket. People who buy a tennis racket will pay more than people who fish,
I suppose. I don't know why they chose a tennis racket. Maybe there's more room
in the handle for the electronics.
The idea here is that as time goes by, we and the machines have to get
skilled, are getting skilled at lumping data and things together and then
filtering and understanding the language. All of this data basically tells, as
Samuel said, the story of who we are. The story of what we do.
It's assessment based on public performance. It's like an essay, and it
can be assessed in exactly the same way an essay is assessed. The way an essay
is assessed now is a person reads it. Sometimes they'll use a rubric but again,
that's taking one hard problem and creating 10 hard problems out of it.
Other times they'll look at it, and the mechanism by which they
determine whether it's an A or an F -- and they've decided that probably very
quickly – maybe even after the first few paragraphs. Certainly when I read
papers -- and I read lots -- I force myself to read the rest of the paper, but
really I know.
You know because you can tell. You can recognize it. See how the words
are used? See how the sentences are constructed? Are they even addressing a
real problem? All of these things are in a certain sense ineffable.
Like Dreyfus would say,
"The knowledge of an expert is intuitive. It's recognized." The
machines are getting smart enough to do this themselves. Neural-network
software is getting smart enough to do it itself.
That's how they mark essays. They don't mark essays using key words or
grammatical constructions. That's a popular fiction.
If you look at the
actual way they do automatic grading of essays, they give a neural net a
training set of five, 100, 1,000, whatever essays all previously marked, and
then that creates the recognition mechanism in the neural network, specifically
the pattern and connection. Then when they put a new essay into the system, it
comes out with a grade based on that previous experience.
Combine two things, the mechanism that creates clusters like we saw on
LinkedIn and the mechanism that can automatically grade essays, and you have
the potential for a system that doesn't actually need to be trained. You just
let it out there into the world, let it find expert discourse for itself, and
then it can associate new discourse with the expert discourse.
A Recognition Task
I've just waved my hand here at what is in fact a very difficult and
challenging problem. We have researchers and engineers working on this
problem...Not easy. But we know the machines are getting closer.
We know this because machines
are passing the Turing test. The whole thing of chatbot which has been going on
since the days of MUDs, IRC, and I think Julia
was the name of it, where people would have conversations with these
bots...That's been continuing since then.
Grading is a recognition task. It's what neural networks do, and it's
what we design and build interfaces for ourselves to do. It's how
we'll respond to the Internet of things. We're not going to examine all of
that data personally. If we could, we'd design systems that present to us maps,
graphs, dashboards, things like that. It's how we're beginning to understand
the world now.
Sometimes it's talked about as intuitive. Sometimes it's talked about as
a language. Sometimes, Don
Tapscott, Marc Prensky, and the rest would talk about it as a culture and a
generation in that. That's what they really mean -- beginning to understand the
world through this process of recognition.
Preskett and Prensky might know that this is what they're saying, but
this is what they're saying. It's not a generational thing. It's not a
"ooh, we can all multitask." It's that if you look at this wide set
of data you can see. You can look at
multiple thread streams, multiple Twitter feeds at once, and see the pattern.
How do you see the pattern? Because you train your brain over time. You
have been exposed
to this data in the past, and it creates a set of connections. Your brain
is a pattern-recognition machine. You see Tweets, LOLcats, whatever, and you
recognize them like you recognize Waldo, like you recognize a pattern.
It's an instinct of knowledge. It doesn't have a name. It doesn't have a
word. It doesn't have a concept. But it's a thing. People know what it is.
We're beginning to become sensitive to these tunes, to these signals,
sometimes even overly sensitive, sometimes even hypersensitive.
Our reaction to some of the stuff that goes on online is proof of our
recognition of some of the stuff that goes on online.
What We Reveal
On the one hand, we reveal
ourselves in our messages. We reveal our thoughts, maybe times a lot more
than we intended. Sometimes these assessments, what we reveal, what other
people look at, can be very
personal, can be very uninhibited. I think we know this.
We're getting sometimes some very brutal -- not necessarily honest but
nasty -- assessments online. There's the whole range of reactions. The range of
reactions is from the very positive to the oh-so-very negative. I've had both,
and I think we all have had both.
We're seeing all of this in other
areas first. We're seeing big data used to analyze all of these Internet
reactions. We have a
project at the National Research Council where they looked at the emotional
significance of Tweets. You have an event. You look at all the Tweets. Are
people generally happy with it? Are they upset with it? et cetera. It's a
project. It's done all by neural network analysis of this published data.
Figure 5. Cat with a cone on its head. Source: http://www.sharenator.com/image/10288/
Sometimes what we reveal is very involuntary. You can sort of get this
reaction. You all saw the poor cat there with the cone on its head, and you
probably all said "aw." This "aw" sensation is revealed in
your later communication. Sometimes you actually type it out, "Aw."
It's interesting because it's not just a one-way thing. There's a
dynamic interplay. I was thinking about this. There's of course this whole
literature about violent video games and that. I've played violent video games.
I don't think that they desensitize you.
The reason that I don't think they desensitize you is that there's no
real emotional attachment to the figures in the game in the first place. It
doesn't trigger the apocryphal mirror neurons. They're
maybe not that apocryphal, but you know what I mean.
But I'm also a devotee of fail videos. That's a little secret. But I
love them because I see myself in those. You see somebody on the bicycle going
over the handlebars, and you go "ah." You do have that feeling. That
feeling is what comes out, and that feeling does get reflected. Then Facebook
analyzes it and monetizes it.
These assessment mechanisms are being built into
the LMS. You require your students to use an LMS. They use an LMS. Your system
now begins to analyze them in more and more and more detail over time. They
probably won't stop at the LMS, will they?
People are beginning to raise
questions about this, and I think these are important and good questions.
They're asking about the methods of exploring the types of data. What kinds of
data is appropriate for an institution to collect? Should an LMS be collecting
information that reveals your emotional state? They're asking about data
fishing. "Let's see what's there."
of privacy and the worlds of analytics interact. This is the problem with
traces. You leave these traces, and they're analyzed. People see into your soul
-- or other non-made-up thing -- do we have the right to do this, or do we have
the right to, as Samuel said, take back our traces, take back our digital
In a certain sense, that's impossible. It's like me trying to take back
the fact that you all saw me 10 minutes ago -- "I'd love to take that
back. It wasn't one of my better moments" -- but it's not possible.
You have heard about this one. I just referred to it. Facebook is doing
experiment on the emotional reactions of 689,003 of its users to show that
emotional states can be transferred to other people. They messed around with
the lists of stories displayed in their news-stream or their feed and then
measured the responses to detect emotional responses to that. Intuitively it's
measuring something that we understand does exist.
You show people nothing but stories of crime -- and there have been
studies on this -- and they think there's a high-crime rate. Even though the
crime rate is going down, because all they see is crime, they think it's going
up. Facebook was doing the same sort of thing, except they didn't tell anybody
they were measuring this.
You've probably heard
about the reaction.
People want to get out of Facebook, and they'll jump to Twitter. But it's not
just Facebook. Yahoo -- among others, I think Google is the other one -- is
dropping the do-not-track mechanism. That's a signal you can put in your
browser so that people don't track you.
They're saying that it's too confusing, but they're the ones who made it
confusing. They're the ones who established the signal in the first place.
last April that they had halted the practice of scanning student
Gmail accounts for potential advertising purposes. Wait a sec, that means
they were scanning student accounts for advertising purposes. They've decided
to stop acting like your creepy uncle. I feel much better now.
accidental. Sometimes the data just gets out, but when it gets out it can
be a bombshell. Like this
one, "University of Virginia Law School collecting and distributing to
potential employers information about grades," OK, "class
ranking," I don't know, "political affiliation, work experience,
recommenders, information about where their girlfriend lives." This is
secret data. Students don't know about this, the employers do.
What do we do? One option
is to delete
all of our social media accounts, but we're not going to do that, are we? People
not leaving Facebook. They're not leaving Google. They're not leaving
LinkedIn. It's not going to happen. These services are too useful.
We learn about our friends. We learn about ourselves. It gives us this
mechanism by which we can recognize what the state of affairs of the world is.
So we're not going to leave the system. At the same time, companies are
beginning to feel the heat.
Figure 6. InBloom. Source: op.cit.
Gates funded a thing called inBloom.
It was based in Atlanta, and basically it was "we will centralize and
store all of your student data for you." Well, we know about that sort of
practice, and we have Rupert Murdoch on the one hand, Gates and Carnegie on the
other hand, and Joe Hacker on the other hand all accessing this data.
Of course it raised a substantial response, a substantial reaction, and
eventually shut down, as it should
be. Ironically the new concern about data is called the Snowden Effect.
That's what you classically call shooting the messenger.
But it's true. Canada
just passed an anti-spam law, which I can report in my case did not change
the flow of email into my inbox. There was a surge before the law was passed of
people asking whether I gave them permission to continue sending messages to
me. I took great delight in not giving them permission.
As of July 1st, the same messages, the same companies...The flow
continued uninterrupted. But there are laws in place now.
This whole concern about privacy is spreading -- and quite rightfully so
in some cases, wrongfully so in other cases. Lecture
capture...You'll notice I did advise everyone that I was recording this
ahead of time. I hope it's still recording. Oh yes, it's still chugging away
People are demanding that the classes they attend not be recorded
because they think that the classroom is a space where you expect privacy. It's
an interesting question.
But it's also becoming more and more the case that when you attend a
class, cameras will be on, if not officially at the front of the room,
unofficially in the back of the room. I could give you a whole bunch of neat
little examples of teachers rampaging on, but I won't.
proposal -- and this comes from Doug Belshaw with Mozilla -- is to ensure
that common spaces are public spaces and not privately owned. There's one
weakness in that proposal -- our public spaces aren't very private either.
It's not simply the case that we can take these services out of the
hands of the billionaires because if we put it in the hands of the government,
we get, again, the Snowden Effect. The real answer here, I think, is personal
privacy. Personal privacy and informal assessment are going to go hand in hand.
They will necessarily go hand in hand.
The schemes -- and you will hear a lot more about these -- big data and
learning analytics are going to flounder on the rock of personal privacy. If
you haven't heard it before, you've heard it here first. I specialize in
There's clear indication that people want this. One example is on a crowdfunding
site called Seedmatch raising money for an NSA-proof
personal server. They asked for...I forget the exact number, but it was
something like €75,000. In 89 minutes, they raised €750,000.
People are moving to privacy
securing personal networks. There are a whole bunch of them. Privatext,
TigerText, Whisper, Cyber Dust, Ansa, Omelette, and Diaspora. Personal
disclosure, I invested in a whole 100 dollars in Diaspora.
A New System of Assessment
Let's put all of this together, wrap it up, and tie it up in a neat bow.
The elements within your system of
assessment are going to involve personal servers. Not stuff you store on the
learning management system, not stuff you store on Facebook or LinkedIn, but
stuff you store on a network you control and own.
There's going to be a public space, the information you are willing to
share with other people, your friends, the assessors that are out there in the
world. That'll be your public face. It'll be the clothes that you put on.
There will be identity management. Us taking control of our own
One of the things about Facebook that's depressing is that -- and this is
Mark Zuckerberg's big objective -- everybody will have one and only one
identity. It's Facebook taking control of identity over your heads.
One of the things we will see is people can have multiple identities.
They can and they will have multiple identities. Why? Because they always have.
All of these things are being developed today. Many of them are being developed
in our own projects in our own labs at NRC.
Figure 7. Personal Production
production. All the stuff we do online publicly feeds into these content
networks. These content networks act, in the first instance, as a global
content filtering system, a global perceptual network. You think of the social
network as an interconnected network of people. It functions in the same way as
a neural network, in the sense that it is a pattern recognizer.
Male Audience Member:
Who are the curators?
People. I don't want to call them curators, necessarily, because I think
people do more than just filter. They interpret, revise, remix, re-purpose,
spin-roll, fold, et cetera. Right? More than just curation, although curation
is a word a lot of people like to use. It's people, right? It's one person
doing things, whatever they do, and these things are seen, recognized, passed
on, commented on, et cetera, by other people.
layer here is the social network. That creates the mechanism for assessing
the qualifications of the individual. Picture it...You can think of it...I'm
trying to draw a picture now, and I don't have a slide for it, which makes it
hard to do. Imagine a social network of quantum physicists. We know all the
major quantum physicists know each other. They communicate with each other,
respond to each other's papers, go to the conferences, call each other names,
et cetera. There's this cluster, just like the cluster of green people in my
LinkedIn network, except they're quantum physicists and therefore have no
color. They're linked to each other.
My qualifications, as a quantum physicist, can be mapped and understood
as my positioning in this cluster. If the other quantum physicists talk to me --
this is putting it very crudely, of course -- I rate as a quantum physicist,
but I can't fool the other quantum physicists because the one person who recognizes
a fake quantum physicist is a quantum physicist.
This human network acts as a perceptual mechanism for filtering the
qualifications of people. That's why the person could fool MIT staff. The
degrees actually don't matter. The way you relate and interact with other
people...That does matter. That's what's happening online today.
As an aside, it has always happened. That's the way it's always worked
except we didn't have a global communications network to pull it off before, so
it was always very local, very individual, very personal.
Professions will coalesce around this. They're online communities.
They're open ended networks. They're similar to vendors' communities in
practice, but they're not just that. They might be gaming communities.
I got my programming chops in a gaming community known as a MUD, Multi
User Dungeon. This is a game, "Untrusted". Same sort
of thing. It's really neat. It's a maze where you solve each step of the maze
All of the artifacts
of our interactions with each other can be measured and understood in terms
of their impact on our qualifications and credentials. Conversations can be
analyzed for meaning and word use.
Interviews. Instead of a person interviewing with 10 different
companies, a person does an interview with a professional interviewer. The
video and transcript are made available online, and then these 10 companies can
look at that interview, use their system to assess that interview according to
their internal criteria, and determine whether or not you get the job. It's a
first-order screening , but it's a very effective one.
Would you hire him? Could a machine detect whether you would hire him?
Assessment of the future will basically redefine
body of work. It used to be formal publication, books, paintings, films,
whatever. Now it will be the mass of communications, my mass of activities,
whatever we do on the public Internet.
That will be a good thing because it will mean that we're able to obtain
a much more accurate, much more precise assessment of people and it will help
to map, first of all, the gaps in knowledge that need to be filled by learning,
which is one of the purposes of assessment.
Secondly, it will help companies, employers, contractors find and
identify the people who most precisely fit what they're looking for.
That's my talk. I thank you very much for your time and attention and
even occasional scepticism.
Male Audience Member: Stephen, I have to
believe there's something wrong [inaudible 60:13] talking about [inaudible
60:17] . Do you get that the biological understanding of the inputs is the
right [inaudible 60:41]?
Stephen: The short answer is yes, but we need to
be careful about how that cashes out because, as I mentioned with the reference
to the once slide with Steve Talbot, what our understanding of biological
systems is needs to be clear, as well. It needs to be, if you will, true to
A lot of people talk about knowledge, the mind, et cetera, as an
ecosystem but when you push them for what they think of as an ecosystem, they
say "well, it's a system, just like a really complex machine."
That's not what I mean. A biology, an ecosystem isn't a mechanical
thing. It's an organic thing. It's the difference between -- to use perhaps the
often overused distinction -- complicated, which has many parts, and complex,
which has multiple simultaneous interactive variables and where the outcomes
are not predictable, except when they're probabilistic or stochastic
Male Audience Member: I really like the
vision [inaudible 61:54] orderly control over the outputs [inaudible 62:16]
while they're in their [inaudible 62:19] .
Stephen: Yeah, two very separate issues, both of
which are important. First of all, as a management, you want personal identity.
One of the reasons why Facebook succeeded -- let's face it, they succeeded --
is because they made something that was originally pretty complex, interacting
with thousands of people online and keeping track of that more or less, and
made it pretty simple.
It replaced a previous very useful system, called thefacebook, which
literally was a book of faces, only now it lets you talk to them as well. This
kind of system where we manage our own personal data will become easier over
time. Some of the things that I referred to -- the personal web server and
things like that, for example -- are things that you will be able to just buy
as a box from Future Shop or Media Markt. It will just be a box. It will cost
You put it in your living room. It'll connect your home network, which
you already have because you have Internet access probably. Then you access it
the way you would access Facebook. It'll be that easy.
There's open source software like ownCloud, for example, that already
allows you to do this. Set up a cloud server, just like Amazon Web Services or
Google Drive, using your own cloud and set one of those up on a personal server
in your living room. It's as easy as using Dropbox or any of the others.
That's that part. The technology will make it easier. As it becomes
easier, the take-up will increase. As the take-up increases, we get these
personally managed social networks as opposed to these centrally managed social
It's like the browser. It used to be that the only way to access the
Internet was to log into a central server. It really was. Lynx was the first
web browser. Lynx was a server application. It would run on your central
computer hosted by an institution. You would log on to Lynx through Telnet,
sign on through the central server, and then you'd access all the other
The personal web browser goes on your computer. It's yours. You can
configure it yourself. It's easy to use. It's a lot easier than Lynx actually.
Nobody uses Lynx anymore. Nobody would use Lynx even if it had pictures, which
it didn't. You can't really justify that for a central server environment.
The second question about security and player data is a good question.
I've done work on DRM, and there are two approaches...Roughly, widely, there
are two ways of looking at DRM, which is a dramatic overstatement of the
distinction. You can either manage the location where the resource is located
or you can manage the resource.
If you manage the resource, then you don't have these kinds of problems.
If you share a resource with an individual where the resource doesn't go beyond
the domain of the individual. [inaudible 65:56] .
Stephen: As an aside, the short answer is there
are technical solutions to this. The longer answer is none of them will work.
It's the Microsoft Darknet paper all over again. Ultimately, you can't
establish security through technical measures. It depends on the interest of
the people involved being better served by preserving the security than
breaking the security.
Edward Snowden had a choice, and the choice that he made wasn't
"oh, I can break this technical measure." He just walked right
through that. It was easy. He was inside of the system. It was "it's
better socially, morally, politically, to release this data [inaudible 67:01]
People are always going to face that at some point. You have a password
for your corporate account. You have a VPN. Why don't you share that with
everyone else? You don't. You don't because it's better for you that you don't.
You have a bank card. What's your PIN? You're not telling me. Why not?
It's not that the technology makes it impossible for you to tell me your
pin. It's that you won't. That's where security comes from in the end.
Companies need to learn this. It's easy to think "oh, we'll just set up a
People talk about military grade security. Have you been to Gagetown?
Gagetown is a military base in New Brunswick, Canada. It's about an hour away
from where I live. Anybody can walk into this military base. Seriously. It's
located right beside the town of Oromocto, by a major highway.
There are signs on the highway saying "you are in a military base.
Please do not stop." The implication being "because we might shell
you." [laughs] You can actually...And I've done this. You drive in, park
in their parking lot, get out of your car, walk into the building. You can just
walk. Nobody challenges you. There's no lighting, no nothing.
The security isn't in creating a fence. The security is all the bits and
pieces. The people inside that base recognize whether or not you belong there.
The people inside that base know that security [inaudible 68:46] , and they
don't just leave classified documents laying around.
Male Audience Member: [inaudible 68:57]
because I think they [inaudible 69:02] and the individual [inaudible 69:07] .
They're based in [inaudible 69:13]. They have a vibrant community over there,
but I don't know how farther you're going to go add and what you're going to
Stephen: Yeah, I'm not familiar with them, but Personal Data Ecosystem Consortium.
Oh, I'll keep that in mind. I'll look for them.
Male Audience Member: [inaudible 69:32]
because technically [inaudible 69:35] . Part of the [inaudible 69:44] and how
can you help to have the resources [inaudible 69:51]. The second one, the
employer is the one who has power. How do you...And get the workers creating
value, like [inaudible 70:03] employer? Why is it better for the employer to
let work through, keep their information [inaudible 70:09]?
Stephen: It's a pretty complex question, yet
again. The "who pays for it" is us because we've always paid for it.
Facebook offers a free service, but as everyone knows you are not Facebook's
client. You are a product. You are what they sell to their real customers, who
are the advertisers, marketers, and corporations. That's who pays for it
Those of us who are good at negotiating deals will get our employers pay
for some of it. You might get a computer from that. You might get a service
from them. Just like my employer pays for my computer in my office, my employer
also pays for laptops, but not this one.
This is for a couple of reasons, one of which is that my employer wants
to give me a Dell with a power supply that is literally this big -- for those
of you on audio, I am indicating the size and shape of a brick. I want to carry
that nice light computer, but also because this is mine. I configured this the
way I want it.
We've got to come up with some sort of accommodation. There are voices
even in my own company, the National Research Council Canada, which is a
government agency, saying "there should be no data whatsoever that goes
outside the corporate infrastructure," but from a practical sense that's
The world is more permeable. The world must be more permeable than that
because you can't run in a completely isolated network and expect to
communicate with the rest of the world. It can't happen.
There isn't going to be an easy answer to your question. Ultimately it
comes out to power. The power relations are shifting. It's outside the scope of
this talk, but the power relations between individuals and employers are
Right now it's really hard to imagine a case where the employee has any
sort of equitable power with the employer. We used to balance that by
unionization, but most of the actions are illegal these days because of a shift
in the balance of power.
What's going to change things is the capacity for people to organize
themselves, to quickly and efficiently find people, individuals, and
contractors to fill individual positions. People will be, because of something
like this, more mobile, more able to market their works to a wide variety of
employers worldwide as opposed to just in your own city or your own plot. That
changed the dynamic.
The relationship between employer and employee becomes one more of a
negotiation between equals rather than the exertion of a corporate dominance by
one over the other. That's what allows for the negotiations to take place
regarding the ownership and the management of the data.
I know employers don't like this. People in power never do, and they
will resist this -- count on it. But the technological answer as opposed to the
social answer will be to try to find [inaudible 73:55] revenue. The social
answer could go any direction.
Male Audience Member: Because of the
transition [inaudible 74:03] competency based employer [inaudible 74:06] . You
should look at [inaudible 74:12] competency standards as of now. Then you could
say a map is very useful to travel, but you don't travel them back. You travel because
you're going to go somewhere, [inaudible 74:24] the richness of the direction.
Maybe one solution would be to...We need maps. There is nothing wrong
with the map. The question is the place of the map within the educational
system. It is usually told to travel -- maybe that's OK -- but you should just
admit the use of the map just to travel the map [inaudible 74:47] .
Stephen: Yeah. I love maps, so I like what
you're saying. But there are maps, and there are maps.
One way of doing a map is to send a guy on a ship out with an astrolabe
and a compass, and to draw pictures of the shorelines. That's what people used
to do. Very bad idea because first of all, really expensive. Secondly, the maps
aren't that accurate.
A better plan is to launch a satellite and take pictures. That is still
not going to be the best map in the world. We could probably still do better,
and it's still only going to be a map. But it's going to be a lot more accurate
and a lot more useful.
When we use a GPS, we use one of the picture kind of maps, rather than
one of the hands-on kind of maps because it's more accurate. It's not perfect --
people have driven into rivers following on GPS. They really shouldn't depend
on them and believe that they are reality, but they're better.
Same story here with competences. The whole enterprise, sending people
out and drafting lists of competencies by hand, is way too expensive and almost
certainly inaccurate. We want the alternate mechanism of automatically
We actually have a project in our program called Automated Competency
Detection and Recognition. The idea here is it's like a satellite photo of an
environment -- in this case a social infrastructure -- taking pictures, understanding
the connection, et cetera, and that gives you your map. But it's just a map.
Once we understand that it's just a map and that it's based on what are
actually real human performances...And it's the real human performances -- and
responses to them which are equally important.
Male Audience Member: Yes, but there is a
list of analyses. You could use them [inaudible 77:04] map. Then that is a
social construction, meaning there is some algorithm designed by some engineer,
who will be very smart, [inaudible 77:16] . It is not a map with us. It is a
map for us. It is [inaudible 77:24].
Stephen: With great tools people will be able to
do this. I don't have any particular opposition to this, but look at
OpenStreetMap versus Google Maps. Right now I use Google Maps. The reason for
that is OpenStreetMap is really detailed in some areas but not too detailed in
other areas, because there are no people there, like places where I live.
I live in a province the size of Belgium with a population of one of its
smaller cities. There are lots of rooms, and there are very few people living
on the roads. None of them are mapping them. That's the problem. Meanwhile, the
Google satellite, or whatever it is, passes over once, takes a picture of the
province, and we've got our map.
I like OpenStreetMap. I like the idea of a map that's created [inaudible
78:24] by the people, but there are right now certain technological and
structural difficulties not [inaudible 78:30] it work. What would I be working
on? A people satellite system. That would be cool but hard.
Male Audience Member: ...But when he
talked about the human network [inaudible 79:04] , what happens if that somehow
[inaudible 79:14] maybe the entire [inaudible 79:18] has some section that's
actually [inaudible 79:22]. How do we then [inaudible 79:26]?
Stephen: That's a good musing, and it's a
relevant musing. I'm not sure. The human perception works basically on the same
principle, and human perception sometimes goes wrong. Sometimes we're color
blind. We only see what we're expecting to see. We don't see the gorilla
walking behind the crowd. Perception is not a perfect mechanism.
I think the first answer to that question is, we need as a society to
understand we've always had these mechanisms. We're [inaudible 80:16]
describing [inaudible 80:17] a giant version of a much more rich and complex
network of social perception and to what we've add in the past.
The one we had in the past is reporters going out, watching things,
taking pictures, and writing about them, publishing in newspapers, and things
like that. It's functioned on the whole very poorly. This is probably better
than that, but it's not perfect. One of the neat things about the new system is
that it's forcing us to question whether our understanding of the world is
correct. That isn't something we've been good at in the past.
In one sense, there isn't a good answer to your question. It's harder to
be self-correcting. We need to develop the global mechanism of self-awareness,
of reflective reflection -- probably should've used a noun there. Just
[inaudible 81:21] the adjective. You know what I mean?
On the other hand, it is a danger, but no matter what we do it's going
to be a danger. I don't have a better option. I wish I did. I wish I could assure
[inaudible 81:42] is perfect and [inaudible 81:43] is perfect, but no, people
will [inaudible 81:47] . People will too often [inaudible 81:53].
Ultimately, with the right mechanisms, I hope we can correct for that.
We did ultimately learn [inaudible 82:03] cannot fly.
Male Audience Member: My question
partially has to do with [inaudible 82:13] .
Stephen: That's right. I didn't trust [inaudible
Stephen: That's a reaction you're supposed to
have. It's interesting the way you described that. What you described with the
triangulation is an attempt to offer a mechanical solution, a logical,
mathematical solution, framed in the name of [inaudible 83:56] to what is in
fact a complex thing. When you depict it as this kind of mechanical, mathematical
problem, it becomes intractable and pretty much impossible to solve.
That's one of the reasons why I favor a neural network approach. In the
neural network, the quality, if you will, of the inference is described in
terms of the structure and the properties of the network. I'm going to
[inaudible 84:38] .
For example, one of the properties of the neural networks is to seek the
set of connections that expresses the lowest potential letter. It's a
methodological [audio cuts] , It might be overall good or might not be overall
good, but as a methodological principle, you can apply that [inaudible 85:05]
past certainly whether or not it is.
That's a fact. What people [inaudible 85:11] are doing right now. But
let's go with that. There's an algorithm, a formula, that describes how a
network settles into the state of [inaudible 85:27] . The sort of thing you're
describing is, what if they fall into a local minimum? A local minimum is a
stable configuration so that it appears through outward appearances like it's
the position of lowest potential energy, but it's not the best possible.
To give you an analogy might help. You throw a rock into a pond or bowl
or something. The water ripples, splashes, and then the water settles. The
methodological principle here is the water settles to the point of lowest
potential energy. But sometimes you get puddles in rocks. The water settles,
but there's a puddle in the rock. That water isn't at the best it could be. It
could be lower, but it isn't because it's in a puddle. It's in a local minimum.
There's a mechanism called a Boltzmann machine where -- imagine this --
you can pick up this pond and shake it. You pick up the pond, shake it --
vigorously at first, but gradually, gradually, gradually less and less
vigorously. Mathematically you can show that, in general, not every time, the
water will splash out of the local minima, and it will all end up in the bottom
of the pond where it belongs.
You can do that same sort of thing with a neural network. You shake up
the connections, and then gradually...For example, when one neuron fires and
another neuron receives signal, it might fire, it might not. That's a
You make the probability really high so that it will fire. You have
neurons firing all over the place. That's like shaking up the network, and then
gradually you lower that probability rule so it's less and less and less likely
to fire. The resulting configuration is the most stable possible.
Your question amounts to, how do you know that will work? My answer is,
first of all, there is a mechanism, and this describes it. It's based on known
principles of thermodynamics. It's something that we can observe with our eyes,
working in the world.
It's the same way we make metal harder -- annealing. We can test this
empirically by building models of this system comparing them to what actually
happens. That's as good as it gets. It doesn't get better. That's my answer.