Droid writes in languages --- it's never seen before...
THIS
CRAFTY ROBOT CAN WRITE IN LANGUAGES IT’S NEVER SEEN BEFORE
SCIENCE 05.27.19 09:00 AM
AMONG THE
MANY things
we humans like to lord over the rest of the animal kingdom is our complex
language. Sure, other creatures talk to one another, but we’ve got all these
wildly complicated written languages with syntax and fun words like defenestrate.
This we can also lord over robots, who, in addition to lacking emotion and the
ability to not fall on their faces,
can’t write novels.
At least not yet. Researchers at Brown University just got a robot to do something as linguistically improbable as it is beautiful: After training to hand-write Japanese characters, the robot then turned around and started to copy words in a slew of other languages it’d never written before, including Hindi, Greek, and English, just by looking at examples of that handwriting. Not only that, it could do English in print andcursive. Oh, and then it copied a drawing of the Mona Lisa on its own for good measure.
Like walking on two
legs, handwriting is one of those seemingly simple human charms that is in fact
elaborate. When you write a word, you have to know where to put down your pen,
how long to draw a line and in which direction, then pick up your pen,
sometimes mid-letter (like with a capital A),
and know where to put it down again.
So to get a kid to write, you can’t just show
them a sample and set them loose—you have to give them instructions on how to
form each letter. “They give you these little algorithms for what strokes to do
and what order to put them in to make the character,” says Brown University
roboticist Stefanie Tellex, who developed the system with Atsunobu Kotani, also
at Brown. “And that's what our algorithm is learning to do.”
Their learning system is split into two distinct models. A
“local” model is in charge of what’s going on with the current stroke of the
pen—so aiming in the right direction and determining how to end the stroke. And
a “global” model is in charge of moving the robot’s writing utensil to the next
stroke of the character.
To train the robot, the researchers fed it a corpus of
Japanese characters, and provided information about how the component strokes
of a character are supposed to work. “From that, it basically learns a model
that looks at pixels of the image and predicts where it needs to go to start
the next stroke, and then where it needs to move while it’s drawing the stroke
to reproduce the image,” Tellex says.
Then they decided to try
to confuse the hell out of the robot by writing hello on a whiteboard in Hindi and Tamil and
Yiddish—all of which use unique scripts. Incredibly, the robot could eyeball
each with machine vision and write its own copies of the words, even though it
only ever trained to write Japanese. Also, they showed it English cursive in
addition to print, and it handled both fine.
Then a gaggle of kindergarteners visited Tellex’s lab.
Surely, the robot couldn’t recognize and replicate their … suboptimal
handwriting? Yeah, it copycatted them with ease. “Just to watch it reproduce
the somewhat wobbly writing of these little 6-year-olds, it was just
incredible, having never seen that before and never trained on that,” Tellex
says.
Surely, the robot wouldn’t be able to
copy a rough sketch that Kotani did of the Mona Lisa on the whiteboard? Well,
this robot is not so easily confused. “That was back in August, and that
picture is still on our whiteboard in our lab,” Tellex says.
But nobody’s perfect. Because the researchers trained the
robot on modern Japanese, which is written left to right, the system could
generalize to English, which is written in the same direction. But it didn’t do
so hot with languages written right to left.
Still, it’s a remarkable
demonstration of the interconnectivity of languages, many different scripts
that nevertheless come from the same human (and now robot) hand. And it’s a
step toward opening up a new line of communication between humans and machines.
Maybe not so much in the
near term, but perhaps one day humanoid robots could leave us handwritten
notes, as opposed to having them spit printouts from their bodies. Ideally not
something ominous like defenestrate! defenestrate! defenestrate!
Comments
Post a Comment