17
Nov 17

NLP with AIML - Part 1 - Stopwords

Folks learning about chatbots and AIML are likely to ask if AIML is a Natural Language Processing (NLP) implementation. The answer is no. However, can one use AIML to perform NLP functions? ABSOLUTELY! It takes some work and creative use of recursion and predicates, but it can be done.

This should be the first in a series of posts that describe how you can perform NLP tasks with AIML. In this post, we'll tackle a very simple concept: removing stopwords.

In NLP, there is the concept of a pre-processing step whereby very common (and potentially meaningless) words are removed. Examples of these words are: is, the, and, I, will, just, so, than... So an input of "this is a simple sentence" becomes "simple sentence."

In most programming languages, we can conceive of looping through and testing each word. AIML doesn't naturally have this feature (ok, technically in AIML 2.0, there is a tag called , but with the exception of the example in the standard which involves counting numbers, I can't see how to use this in other ways - please comment if you have good examples of ). But we can force AIML to loop using reductions... the tag. The part where we really need to get creative is how we stop the loop. In this case, I basically have a test for when we're at the end.

At this point, I think it's better to show than tell...

Here's a flowchart of the method (no criticizing my flowcharting skills... but please ask questions!):

Flowchart for stopword removal in AIML

And even better than that, you can see this in work in two ways:

1) If you have an account on the pandorabots playground ( playground.pandorabots.com ), search in the clubhouse for "NLP Test Bot" and you can chat directly. Here's a screen shot of a chat:
Chat window with NLP Test Bot

2) I have a zip file of 3 AIML files that you can play around with yourself. These are the same files as what is loaded in the NLP Test Bot on pandorabots playground. A couple things to know... the list of stopwords comes from python's nltk library. It seemed like a good list. However, given that in AIML, we normally expand contractions, half of this list could probably be removed. The other thing to know is that it's AIML v2.0. That's because I am using the and tags.

Download nlpinaiml.z7

2a) Could this be done with AIML v1.0? Yup. Before the and tags came into existence, I had a category that looked something like this:

<pattern>SPLITME * *</pattern>
<template>
<think>
<set name="firstword"><star index="1"/></set>
<set name="remainder"><star index="2"/></set>
</think>
</template>

 

There was also a second category: SPLITME * which, when both of these existed, would match if * only had one word.

Hopefully I'll get around to posting some of the next steps... like part of speech tagging (hint: I do a lot of the same stuff with reduction and setting predicates... the trick is to have a good base of words) and then what really matters: how you can use this to have an interesting chatbot!

Please comment/ask questions...

26
Aug 14

Musings on Ray Kurzweil, Moore's Law, and the not-so-far-off future

I write science fiction. I was with my critique group the other night and one of the gentlemen critiquing my work was concerned with the date I chose for the setting of one of my stories. It was not just the date, but the date combined with the fact that the technology I was positing didn't seem advanced enough.

He cited Ray Kurzweil and Kurzweil's predictions about the integration of non-biological intelligence with human intelligence and was emphatic about it enough so that I decided I needed to re-listen to Kurzweil's TEDtalks.

I did that today.

I like Kurzweil. I really do. But... I think he goes a little too far. At the center of his talks is Moore's Law. Kurzweil emphatically (and correctly) points out how well Moore's law has proven to be true over the years and therefore, it will continue to be true. He also likes to apply the concept of exponential growth to anything digital. Sure. No issue there.

But he makes some unfair and inconsistent extrapolations when he mentions intelligence and our ability to understand intelligence over the coming decades. Just because we are collecting data at an exponential rate and digitizing data at an exponential rate does not imply we are UNDERSTANDING anything, especially intelligence, at anything close to that rate.

While we might have access to an exponentially larger quantity of data than we did in the recent past, while we might be able to compute exponentially faster... we are not exponentially more intelligent.

And we are not going to magically understand intelligence in the coming decades solely based on the rate that our technology is expanding.

Please... don't confuse my assertion that we won't understand intelligence with not understanding brain function. Certainly over the last several decades we've learned a lot about biology and how the brain WORKS. But that's not intelligence. Not by a long shot.

I'm in Jeff Hawkins camp when he writes in his fantastic book "On Intelligence" that we don't yet have a framework for understanding the brain (in terms of intelligence, not biology... different things) and until we do, we won't be making the fantastic leaps in technology that Kurzweil predicts.

(side note: another post that I hope to get out soon will be on Jeff Hawkin's book and how it makes a fantastic case for the AIML)