What's cool vs what's useful

What's really cool is an AI you can have a seamless conversation with. What's useful is a service that can answer your questions.

A long time ago, before everyone was working on a chatbot, a colleague asked me for help on his personal project. He had built a website where he aggregated some data he scrapped from a public server. The data was raw and unfiltered. He painstakingly went through hundreds of CSV files, cleaned up the data, then uploaded it in a database.

Now he wanted to present it as a website where people could query the data. The only issue was he wasn't a programmer. He worked in marketing. He built the website by himself by following tutorials from all over the web. For someone who was not technical, he was pretty good. But how was he supposed to let people search through the database? He built a prototype then sent it to me to help fix it.

What I saw was a text box where the customer could write in plain English whatever data they wanted to get back. For example they could write: "Give me the products that were added between January and March." I wrote that exact query and to my surprise it worked. I instantly became jealous and insecure about my own abilities. "How did he do it?" He worked in marketing, he had little programming insight. How did he write a sophisticated natural language processor? My many years of experience were no match to him. But then I looked at the code that powered it and my ego was appeased.

In the code he had some specifically hard coded conditions that allowed it to work. For example, the string had to start with "Give me," or "Show me."

if (strpos($text, "Give me") !== 0 || strpos($text, "Show me") !== 0) {
return false;
}


Then he looked for the word "product", followed by "between", then an array of months. The year was always set to the current. In fact, the tool was not sophisticated at all. Anything out of the ordinary would break it. For example, if "Give me" was not capitalized, the script would fail. If you added an extra space, it would fail. If you reworded the sentence, it would fail. It wasn't an NLP; it was a bunch of nested conditions. There was no way I could help him debug his script. Instead, I proposed something less cool.

"But I want the user to feel free to type whatever they want. They don't have to be experts in order to query the data." He told me.

Sure, NLP is great. But it is an order of magnitude more difficult to build. Note that spaCy was a few years away from being released. Even so, NLP would have been overkill for what he was trying to do. My solution was to use boring old filter drop downs.

When the user visits your website, they usually have a pretty good idea what they are looking for. Especially when it's a niche website. It may seem like they want to have a conversation with an AI driven chatbot, but the reality is that they have a problem and you have promised to solve it. Whether you use a checklist, an AI or a mind reading device, the result should be the same.

So instead of a free form text box, we ended up with 3 or 4 dynamic drop downs that allowed users to select the product type and the date range. It wasn't sexy, It wasn't complex, and it did not help him get VC funding. But it was simple and practical. That's all that mattered.

It's always fun to get lost in technology and build things in the most complex way possible. But if you actually want to build a tool that is useful from the very beginning, it's best to avoid being fancy.

---

On a related note, here are some interesting articles.

Language is culture not just syntax

I am lucky to have been exposed to many languages as a child. I lived in a city where almost everybody was from a different country. On the average day I would hear at least 4 languages being spoken. French and Fulani at home, English on TV, Arabic on the street and various other languages ranging from Bengali to Swahili from my peers. If you are at least bi-lingual then you know how jokes lose all their substance when translated. A language is more than just being grammatically correct. Words that make up a language not only come with connotation but there is a culture associated with it. With all the baggage it carries, the language makes you think in a certain way. Programming languages do not escape this rule. Each programming lingo comes with it's own set of rules, syntax, and culture.

I charged $18,000 for a Static HTML Page

Not too long ago, I made a living working as a contractor where I would hop from project to project. Some were short term where I would work for a week and quickly deliver my service. Others lasted a couple months where I would make enough money to take some time off. I preferred the short ones because they allowed me to charge a much higher rate for a quick job. Not only I felt like my own boss, but I also felt like I didn't have to work too hard to make a decent living. My highest rates were still reasonable, and I always delivered high quality service. That was until I landed a gig with a large company.

How to write well when you don't know how to

It's easy to decide if a writing is bad. The author uses big words, writes long complicated sentences, and forgets the point he is trying to make. Other common things in bad writing is lack of structure and lack of knowledge in the subject. In a nutshell, when you read it, you don't understand it.

View all articles

Comments