Posts Tagged ‘algorithm’

Interesting News in Biology and Healthcare

November 5, 2011

BioHackers - discover Magazine

Can product management be scientific?

March 12, 2011

I know a spell
That would make you well
Write about love, it could be in any tense, but it must make sense

Belle & Sebastian – Write About Love

Some companies believe that product management can be reduced to scientific experiments.Instead of using intuition and customers interaction one should run experiments and measure results.

While I have great faith in measurable product management , I think that the dream of product management without the human factor is wrong and dangerous.

Everyone seriously involved in pattern recognition and data mining knows that one can’t just throw tons of raw data into an algorithm and expect to gain (artificial) intelligence.

In most cases it is hard to build a large data set to train the algorithm. Once such data set is built , the raw size is too big for any algorithm to train on. As a result ,the raw data needs to be reduced through feature extraction. For example, if we want to build a face recognition algorithm in a video stream we can help the algorithm by removing the soundtrack. While in theory the soundtrack can add information to the algorithm, we guess it is not very helpful.

The process of feature selection and even dataset selection involved intuition and domain knowledge. This is similar to the generic scientific model.

In 2008 wired magazine claimed scientific method is obsolete  in “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete“. The article claims that models are not needed anymore, as data is stronger than models.

However, even the first example is wrong

Google’s founding philosophy is that we don’t know why this page is better than that one: If the statistics of incoming links say it is, that’s good enough

But Google’s early  success was  not just because of the algorithm. The clean UI,text only ads and great performance were crucial. I’m confident it was intuition\product management that led into these decisions. Moreover, the statistics for incoming links  from fraud (link farms) are also very high. The algorithms needs “help” on the features that identify fraud.

Google Suggest brings another example. It is a great feature which interactively “guesses” the search term for the end-user.

For example, typing the word “Robert” suggests the following :

Google Suggest For Robert

Google Suggest For Robert

But looking for the word “Naked” brings no results at all:

Google Suggest For Naked

Google Suggest For Naked

But “Nak”still shows some alternatives:

Google Suggest For Nak

Google Suggest For Nak

Does anyone think the algorithm decided on this feature based on statistics 🙂 ?

Obviously, someone decided that following the real statistics of the human mind would be too dangerous.

I’m very much in flavor of usability research and detailed numerical specs. But in most scenarios, the psychology, human interaction and models are crucial for a building a great product.

Can you make money writing algorithms ? Part I

December 21, 2007

Just a few years ago it didn’t really pay off to be an algorithm focused company or an expert. Life was much better for a company that made pretty straight forward   database-office-automation-process-improvement software. Finally, this is about to change and the really smart people of software programming might start seeing money flowing their way.

Historically, writing sophisticated algorithms didn’t result in great product, system or reward. One of my good friends, who is also one of the brightest guys I know, tried once to sell a great machine learning algorithm.He devoted 5 years of his life to writing it. The best offer he ever heard was 10,000$. That’s more-or-less 200 hours worth of an expert HTML developer in San Francisco , only that this is one of the smartest people in computer science and he spent some 12,000 hours on it.

Things were not much better for others. Let’s try a social game. Name 5 famous Belgians ! Sorry, wrong game.  Name 5 famous algorithms that made their inventors rich. Don’t Peek.

1.       RSA – There is some evidence that considerable financial reward did arrive to some of the inventors.

2.       LZW – Rumor has it that Compuserve made some money out of it.

3.       Can someone Help ? One-Click-Shopping? Window Keyboard Button?  T9 ?

To make things worse even the smart companies didn’t do very well. Take a look the OCR market. There used to be quite a few companies that made great algorithms for image processing and pattern recognition : Ceare, Calera, Xerox, ScanSoft. They are so forgotten you can’t even find most of them on the web.

Today, almost none of them exist and you probably never heard of them, if you are outside of the field. Voice recognition companies didn’t do much better. Dragon, Art,Phonetic Systems. No one was really able to get really huge revenues and they all folded into one company – Nuance.

There are quite a few good reasons for that. First of all, people buy products , services or full solutions. They are not really interested in algorithms. Usually the people who create great algorithms are not so great in product management or system engineering. Google without the text-ad concept and great engineering of scalable data centers would be left with another great information retrieval algorithm.

Second, there are many times in which a better algorithm is less important than implementation and context. In computer science it can mean a lot if something takes N operations to compute or If it takes 2N operating to compute. In real life this is often overshadowed by other considerations such as memory, IO, or just waiting for the users input. Moreover, algorithms need context. Data mining for  shopping is different that basketball analytics. Understanding the unique features of each one involves a lot of domain expertise.

To make matters worse for our algorithm genius, even in places where algorithm can save lots of money such as compression, communication and encryption the trend in recent years has been standardization.
With cellular (GSM) & Security  (AES) innovation being set by standard committees, the potential to make a difference lies more in the implementation of the algorithm, rather than its invention.

In so many words, if you wanted to spend your life writing cool algorithms, you would probably need to focus on academic life or settle for nice day job in medical, communications or defense industry.In next part – Why is it all changing ?