A New Kind Of Search
Has anyone stopped to think about how computers might be different if the first widespread human-personal computer input device was the sketch pad and not the keyboard? I usually try to avoid the googly-eyed “wow” factor of new technologies, but this one, Retrievr, is very cool and has the potential for facilitating access for learners who might have trouble expressing themselves or processing information in words.
Retrievr is an interface which allows you to find Flickr photos by drawing a rough sketch (see the screenshot). It may not be as accurate as, say, a search on Google with words, but it’s also not bad. At one point, as I was attempting to elicit a red apple, a pumpkin was included in the results. I then tried to get the pumpkin back by sketching an orange blob with a short stem: I was unable to get a pumpkin included in the results, but I did get a bowl of spaghetti. It’s also important to keep in mind that it does not recognize sketches as iconic representations. For example, in the screenshot, Retrievr does not recognize my sketch of a rough green triangle with a short brown horizontal line as a “tree.” It recognizes my sketch in its entirety as a rough green triangle with a short brown horizontal line and a blue/green/white background (it took me a bit of experimentation to figure out the background piece of the equation: the background does matter when performing a Retrievr search). In addition, Retrievr currently only draws upon Flickr’s most interesting photos; Retrievr’s developers do invite submissions and a submission interface is in the works.
But enough about Retrievr’s limitations. I think that this method of search holds some promise for learners, as I said above, who may have trouble expressing themselves or processing information in words. It may also be a boon for those who simply prefer images and visualizations to words, especially if you combine Retrievr, Flickr, and, without trying to toot my own horn too much, my Greasemonkey script Trope for Flickr. Learners can start off a search by drawing a sketch in Retrievr and find the most appropriate Flickr photo(s). Because Flickr provides the ability and encourages photographers to assign tags to photos, a (albeit unvetted) descriptive vocabulary is provided for each photograph. With Trope for Flickr installed, icons are placed next to each tag allowing the learner to branch off to find information in other forms of media with the same tag, such as web pages (via del.icio.us), blog entries (via Technorati), podcasts (via Odeo), and academic references (via CiteULike).
Imagine the potential for access and power afforded by Retrievr to those learners who have been kept back from interacting with technology because the primary interaction mode of the World Wide Web is the written word.
January 8th, 2006 at 8:53 pm
When so much of the technology in schools is used simply to recreate what we do already, this is exciting and wondrously provocative thinking. Thank you.
I think you might enjoy Illich writing on the beginning of the monopoly of reading and the book in the 12 Century - In the Vineyard of the Text, if you haven’t already. It disconcerts they way you imagine the world and fits quite well with the ideas you are challenging us to think about here.
In fact, the alphabet is an elegant technology for the visualisation of sounds. Its two dozen shapes trigger the memory of utterances that have been articulated by the mouth, the tongue, or the lips, and filter out what is said by gesture, mime or the guts. Unlike other writing systems, it records sounds not ideas. And in this it is foolproof: readers can be trained to voice things they have never heard of before. p39
January 11th, 2006 at 8:59 pm
When I first saw Retrievr I dismissed it as a game; interesting, but only as a form of amusement. Your post and Artichoke’s comment had me thinking about it while I was reading to my students this morning. From Birchbark House, by Louise Erdrich:
Using a pointed stick to write in wet mud, Angeline showed Omakayas and Nokomis the meaningful signs, which looked like odd tracks.
“What animal would leave these?” Omakayas teased.
“Be patient,” Nokomis counseled. “Let’s find out what your sister has learned.”
“They’re letters,” Angeline said, eager to share her knowledge. “One follows the next. You look at them, just like tracks. You read them. They have a meaning and a sound.”
“Howah!” That’s a good idea! Like our picture writing,” Nokomis said.
…She also knew where certain marks had been placed upon lake rocks long ago. Some of the marks were made by the spirits, some were made by humans; others were drawn by a giant race of people who had lived on earth in the old days and had disappeared (p.190-191).
Music, graphic art, dance, sculpture, etc. have been around since forever. New technologies that enable ancient ways of knowing suggests that digital technology could encourage a return to nonlinguistic media as a dominant form of expression.
Thanks for sharing your thoughts on this.
January 13th, 2006 at 8:46 pm
Thank you both, Artichoke and Doug, for your thoughtful comments comments. Different people have recommended Illich to me in several contexts, so I’ll have to put aside some time to read some of his works. I do believe that the way our technological tools are evolving, we can begin to incorporate other modes of expression and interaction, which would be very empowering to many people. Just think about all the people who are classified as “print disabled” or with “reading-related disabilities.”
There is a very neat page, entitled the Evolution of Alphabets, at http://www.wam.umd.edu/~rfradkin/alphapage.html. It illustrates many of the relationships between pictographs and various single-sound symbols, and then the further evolution into the early sets into some of the modern alphabets. It’s always interesting to see that elements as abstract as alphabetic characters are rooted in something much more concrete.