Pattern is a web mining module for the Python programming language.

It bundles tools for data retrieval (Google + Twitter + Wikipedia API, web spider, HTML DOM parser), text analysis (rule-based shallow parser, WordNet interface, syntactical + semantical n-gram search algorithm, tf-idf + cosine similarity + LSA metrics) and data visualization (graph networks).

The module is bundled with 30+ example scripts.

Props to the team behind the library.

My favorite library feature is the pattern.graph module which performs pretty much what you’d expect from a graph library (path of least surprise) and includes a couple of great network node aspects.

• Node weight, which is computed by eigenvector centrality (think Google pagerank) and
• Node centrality, which is the darling of network hotness, betweeness centrality which is how often a node is part of a shortest path

Not long after scanning the documentation I wondered if there was a set of ruby gems or modules that I could quickly take advantage of and integrate into some of my heroku hacks. An alternative is to setup a python server which runs CLiPs^. A few minutes of browsing revealed the Ruby Graph Library (RGL). The graph lib is based on Boost Graph Library which is all good in my book (just started heavily using Boost at work).

One missing piece of the puzzle is a great open source natural language processing gem/lib. In our startup we leveraged third party APIs (Zemanta, Alchemy, OpenCalais) and there are a number of gems which do the same but these are all metered solutions*. I’d much prefer a local option so I can tune and grow the algorithms by leveraging various third parties and add a dash of machine learning. I suspect Google will release a compelling product for giving structure to text based on their talents at machine learning and huge corpus of language (entity extraction/graphs). If you know of a great NLP ruby gem, please point me to it, thanks!

Notes:
^= Flask on a hosted server, or the Google app engine make good targets.

*= The prominent semantic tagging tools have a nice free margin but you’ll quickly run into problems if you want to process individual messages. I described the API costs and free levels in an earlier post.

The past few days I’ve been digging into Open Calais’ API. They have another great semantic tool which we can leverage. The big plus for Calais is there 100k hits per day limit. While debugging our list processing, I’ve already gone beyond the Zemanta API limit of 10k. Alchemy’s Orchestr8 is our goto semantic API, and it gives us 30k hit per day to work with. If necessary we can explore hitting DBPedia directly to identify our own entities from text. Relying on external tools could be more expensive than our limited revenue can practically support. It all depends on how many affiliate sales we can get per API hit. Zemanta’s pricing architecture is $1200/month for 50k hits per day at the time of writing this post. Calais may be our most viable option. Even better than that, in house semantic lookups could be the best long term solution. Lower software and maintenance cost versus API hits, and in theory we could have a massive local database to minimize response time.

code for the above fugly buttons are at this gist

It’s not important what you like

We all like plenty of mediocre crap. The humor that get’s a single syllable laugh, the tech article that get’s a head nod, the hack that connects format a to format b. Tracking knowledge of tolerable information we approve of is a waste of effort. Web companies are better served by replacing like with Fucking Awesome sentiment tracking. They should go ahead and add a Loathe button while they’re at it, even that would be of higher utility than a like button.

The Brief and Questionable History of Sentiment Tracking

There are a number of web services that are doing everything in their power to capture sentiment data. Google and other search engines have leveraged the link graph as a form of approval. Not too long back Friendfeed integrated a like button next to posts in one’s social feed to express appreciation, interest, and to share that information with others. Facebook quickly mimicked Friendfeed’s like button seeing the great potential for categorizing the personalized interests of it’s members, as well as the overall approval by communities. Twitter leveraged the RT user meme which shared interesting tweets and let the original poster know that their update was appreciated*. Hunch skipped the entire social network aspect and dove right after sentiment data.

Sentiment information is highly sought after by businesses. Companies benefit by gaining access to individual tastes, which enables them to strategically plan their marketing efforts. Rapid feedback is obtained by observing real time approval which enables businesses to expend resources and quickly identify successful campaigns. In this way our collective interests serve as advanced sensors for mapping out optimal marketing paths in a highly dynamic and nonlinear terrain.

Notes:
*= new retweets are universally ignored by original posters

I envision the open web as one large database, with various search and services acting as queries against it.

Semantic approaches are pretty nifty. They can connect people, places, or things together with verbs, i.e. “Mark Eats Pancakes”. We can do searches against this type of structured data like “who eats pancakes”. Low and behold I should be on that list. One of the issues with semantic tech is it’s lack of ability to deal with ambiguity. Entities require universal references and it’s challenging to connect many facets of conceptual knowledge through strict relationships. Beyond ambiguities, even mapping or connecting just two semantic databases can result in many conflicts, i.e. which association is correct? We can communicate common understanding and meaning through effort, but assumptions about ivory towers of truth (absolute) fade away as we get to the heart of problems and very particular environments.

Then along comes personalized relevance measures. The right answer for my search on “who like pancakes” may be different than your identical looking query. Over time user controlled and owned (this is important) agents can learn about our interests, tastes, and social connections. Our social network being not just friends, but who we listen to, and who tunes in to us.

Why is it so important for us to own our personal relevance data?

Software as a service companies would love to own all this data, and to access them you’ll be required to click on a license agreement which makes this the case. But clients require control of their own personal search agents, as it will be closely tied to our identity in the future. I don’t see how in the long term any company can best serve my relevance needs better than me, and I’ll be damned if they have the right to sell that information to whoever they wish without my knowledge or approval. Without full control over access to our extended memory and taste graph, we slip towards becoming a society who’s brokered information is controlled by corporate interests.

Here are the routes if you don’t feel like clicking through

• Users: http://imagebrowser.heroku.com/users/<username>
  • Example: http://imagebrowser.heroku.com/users/victusfate
• Lists: http://imagebrowser.heroku.com/
  • Example: http://imagebrowser.heroku.com/lists/victusfate/sharpanglesvcs

WARNING, these are raw search api images. Many of the Google search API top results are unflattering to say the least, but they are the undoctored image search result for a given semantic entity.

Consider the following example: I am. At first glance, this is the simplest of sentences. It covers basic concepts, is two words and describes the existence of the speaker/author. Built into I is the concept of identity. What it means to be a conscious thinking being, an individual. Am can be used as a state of being, or as a connector to another state. It’s worth digging a little further into “am” from a different frame of reference.

I first recognized the intracacies of meaning when taking Spanish in junior high. The verb tener, literally translates as “to have” in English. But when the state of an individual is described “to have” is used instead of the raw being in am (to be in Spanish is ser which is subtly different from am). Instead of saying “I am sick”, I would say “Yo tengo enferma” or “I have sickness”.
Even the simple concept of being requires a depth of understanding that machines are so far incapable of.

Language the Code for Our Minds

This quick review of meaning required a series of more complex thoughts, all of which were based in language. In this case my native language English was used to construct more complex thoughts, and leverage analogies and compare patterns of ideas. Although I’m not limited by language in my ability to imagine, I am restricted by the core foundations of words to build elaborate relationships and communicate them. Writing has always been an enormous challenge for me, words and associations are too detached from experience. I’d prefer to share the raw experience, sensation, or idea and literally share thoughts (Vulcan mind meld).

A concept in my thoughts can be associated to a word, but often times defies characterization by language. Intimacy, love, and orgasm are all concepts that describe feelings or sensations we share as humans, but I am hard pressed to capture the essence of these feelings with language. In essence language has become an abstraction of concepts into words, and is fundamentally connected to assumptions of shared experience. The errors in assumption are both beautiful in an artistic sense and tragic in an efficiency sense.

Compare language with coding for computers. Words capture the role of objects in programming. The relation of structures and their role in architecting more complex software is analogous to how our minds uses language to contemplate and communicate more intricate thoughts. The “place holder” words (love) which reference both specific and abstract collective experiences, only have meaning for an individual when they relate their personal experiences to the concept. The equivalent in software is an object description that’s only completed by each local instantiation of a functional description. The object inherits local scope methods based on where it’s called. This idea has lead me to see that the ideal programming language may be specific to each individual. The description or methods we use to describe a structure or function could be separate from a common language that lies beneath the user interface. I can describe an abstract high level functionality, that local viewers/processing nodes can interpret and fulfill with their translation methods. Default methods can be overwritten with local optimal to the caller.

We attach our own personal meaning “method” to all language we encounter

Personal enriching of semantics is directly responsible for feedback and clarification in communication. If meaning of specific terms is known with high confidence, we could assume successful message transmission. Instead we often repeat a statement followed by a question, “You understand what I’m saying, sir?”

Note: No wonder that communication networks mimick their creators with handshakes. The link in communications is analogous to the shared image of a concept between people.

My biggest hurdle over the past dozen years has been learning a new programming language. After getting comfortable with C++ (and c) I couldn’t imagine starting out all over. Learning how to translate thoughts into data structures, objects, and methods took me a good two years to understand, and five to get very comfortable with. But last year that barrier finally got worn away by a potent force, my enthusiasm to build a highly capable web information tool.

At that point I began to familiarize myself with php and common web programming languages. I had never consciously written code to access a web API before writing a script to access Twitter late last summer. The documentation was rich, but I kept looking for something like an include to dig into source code. That’s how I used to figure interfaces out, often my own source code written years earlier. But APIs only reveal the interface, they are the block box of the web world.

In the fall of last year I toyed with Scala on Lift, and Python with Django. Setting up Scala in Netbeans and Eclipse IDEs was nontrivial for a first timer. To clarify, getting the right compilers/Jre/sdk and mixing this concept with the Google App Engine took some time to straighten out (poor Benjamin Golub was kind enough to put up with my ignorance at the time the friendfeed API v2 was released).

My reliance on IDEs is another hurdle, but I far prefer rich IDEs for code navigation to editors and makefiles (I started coding on SGI unix, VI and emacs are not the ideal coding tools for me). I was clueless how to tweak the sample code frameworks besides obvious logical hints in the code structure. I judge a language by how much sense it makes to me when looking at sample code. If I have to pour over documentation to clarify dense arcane syntax, it’s a bad sign, although I’m becoming acclimated to hitting web docs for help.

Last November Tyler kicked me into learning the basics of Ruby through Rails our current framework for the IMM (Intelligent Media Manager). As a long time engineer who primarily did numerical/algorithm development in C++ I was a little shocked at how slow run times for interpreted languages can be. Why would anyone choose to develop in a language that was an order of magnitude or two slower? Some simple facts I was missing:

• for the type of web processing I do now, the wait time of remote API calls is much greater than local processing time
• abstraction for high level languages allows for rapid construction of functioning prototypes
• Static vs Dynamic typing: (somewhat related to interpreted and compiled languages) I prefer static typing to dynamic, as that’s how I first learned. Declaring types is good practice (its built in documentation) and saves a fair amount of testing. Dynamic typing has it’s own merits but I’m not the right guy to argue this topic. Someone like Dave Pollack can discuss the comparison of each much better. I’ll come back to it in a future post when I have more time to research trade offs (spooky animal)

In contrast to the lack of need for speed with some web development languages, there are fundamental interfaces that we all want to be as responsive as possible. For these (our database and public API) we’ll want things to be hypertuned. In fact if we had to, I’m confident Tyler and I could write the entire site/service over again in a natively faster language now that we’ve gone through it once *ducks the rock Tyler will throw at me for this comment*.

Ruby on Rails is a great tool for learning, developing and understating the process, data, and user flow. There’s still a great deal I can learn about the language (they didn’t choose dynamic typing on a whim). While working with Rails I have even learned a little JavaScript to enable users to do remote hits to the Twitter search API (we use jquery as a support lib). This was the last step for the rudimentary twitter client we now have, Social Gravity. Our tools excel at identifying the important topics of status updates (metadata) and puts us in a position to better invent useful search and interaction tools.

Unlimited API hits aren’t free

The past few days I’ve been digging into Open Calais’ API. They have another great semantic tool which we can leverage. The big plus for Calais is there 100k hits per day limit. While debugging our list processing, I’ve already gone beyond the Zemanta API limit of 10k. Alchemy’s Orchestr8 is our goto semantic API, and it gives us 30k hit per day to work with. If necessary we can explore hitting DBPedia directly to identify our own entities from text. Relying on external tools could be more expensive than our limited revenue can practically support. It all depends on how many affiliate sales we can get per API hit. Zemanta’s pricing architecture is $1200/month for 50k hits per day at the time of writing this post. Calais may be our most viable option. Even better than that, in house semantic lookups could be the best long term solution. Lower software and maintenance cost versus API hits, and in theory we could have a massive local database to minimize response time.

Early last year I experienced a major breakthrough, my Twitter AHA! moment. While watching tweets stream by and following way too many people, I couldn’t really tune into a single user. I witnessed a chorus of voices all sharing their thoughts, ideas and status. Within the net rattle of information, I was able to discover the tech super fan, Robert Scoble.

He was wildly praising friendfeed and why he loved it at the time (powerful aggregation + conversations). What impressed me about my first visit to his blog was a video he created. It captured how he processes social web information. My first instinct was that I could help him filter all that info even better, but I wasn’t sure of the details, it was a gut reaction. The setup Robert was using was a Matrix like interface of displays. He tracked streams of text flowing down the screens, and split his attention among them. There was certainly some Jujutsu on Robert’s behalf, leveraging a distinct lack of focus into an advantage for tech news gathering: Purposeful attention deficit chaos

I got an image of the pulse of the world going through twitter, although it’s social interface only get’s a tiny slice of what people are experiencing. Alas, the ubiquitous state sharing machine will have to wait. Although users get the impression of synchronization, what’s really happening is more like phone conversations, with small but acceptable delays. A message is placed which quickly gets posted to the site. If you subscribe to that person’s updates you receive it in your input stream after a short delay. Wouldn’t it be great if you could filter the history of information for things that are relevant to your interests? It turns out you can! More on that later.

At a coarse level, users of any social service can tune into channels which cover topics that are pertinent or relevant to them. While it’s a good start, almost everybody I listen to has diverse interests. Even focused information streams may share more than most folks have time to consume. While users are free to create multiple input and output streams, this manual processing step unnaturally breaks up their creative flow. Despite that cost, several information nexi (super nodes) have sliced their streams. For examples explore Louis Gray, Robert Scoble, or Mahendra to see how they split their content and curation.

With respect to streams from a single user, I find value in observing various aspects of their personal interests through diverse shares. You can get a feel for how a person thinks, processes, and values information. But there are times when I want to tune into specific topics, and I want that input stream to flow naturally from my reader lists. It got the wheels spinning in my head about information tools to support that form of filtering. Twitter, a massive relational database waiting to happen.

Their’s little relevancy and value in knowing in near real-time if the entire world is eating a bagel. There may be in coordinating a world wide eating event (tea time Tyler?). But businesses benefit by having up to date approximations for how many folks are talking about a certain brand of bagel. Global Bagel Buzz, you heard it here first. The missing piece to provide this type of meta data is user tags (manual) and semantic processing (automated). Knowledge systems like Google search, Hunch, and Stack Overflow all depend on human tagging (or linking) of data. Successful automated tagging of status messages (we’re about ~80% correct with Intelligent Media Manager or IMM) is a gold mine of value. As Mr. Miagi might say “If done right, no can defend”. When implemented the right way, we create the opportunity for fantastic crowd knowledge applications.

I’ve witnessed live applications that are already creating enormous value. Howard Lindzon got the ball rolling with Stocktwits where passionate stock traders share info in the Twitter stream by manually tagging stocks with $ as in $MSFT, and tagging their posts with $$. There is growing crowd knowledge in that stream, and filtering techniques correlated with historic stock values should bring light to trader performance (as well as hot/cold stocks). If you enjoy Howard’s blog as I do, please leave him a comment suggesting he try out the IMM. More on market effects at the end of the post.

Additional Structure/Metadata We Can Add to Status Now

There are plenty of opportunities for meta data assignment algorithms, and accurate user self tagging. At Victus Media we have constructed an opt in system for anyone with a Twitter account. We use semantic algorithms from Zemanta and Alchemy’s Orchestr8 to discover high confidence tags (machine identified categories). We do regular expression pre/post filtering but still get false positives, as well as many empty returns. Current semantic tools combined with our own curating yields ~70-80% matching tags to user interests.

Public internet communications systems leave a wake of user shared information. A ghost in the machine, serving as a trail of interests leading back to an individual or community. It’s time that users begin taking advantage of information that businesses have been leveraging for years (Google Adsense, Gmail,Financial transaction tracking companies, and now Victus Media). The virtual ghost of one’s interests can be a valuable search assistant. It is vital that users have full control of that information. At our site folks can delete or add words they wish associated with their account, and can opt out anytime by disconnecting our service from their Twitter stream. At Victus Media we believe trust is built on open, up front communication and user control.

How the Intelligent Media Manager Works

Certain entities come up in your stream more often, naturally reinforcing the weight of that topic. Other topics you mention only once degrade within a week (coincidentally the time window twitter searches reach back to). We chose twitter because it’s a natural fit to what we’re trying to do (users default to public + straightforward API).

The real magic comes in when these tags empower users to connect on topics that they are passionate about. We look for links and correlations between people and popular trending topics which share categories. While iterating on the software framework , we spent several months brainstorming cool things we could do with the user topics. Tyler recently connected tags to popular trending links from external web sites as a an effective form of social search. My latest contribution is called Social Gravity, and it connects users by topics, creating transient chat rooms based on user chosen topics. I’d like to extend these rooms to a larger index page showing how popular each topic is.

Segway on Markets, They’re on my Mind Today

What better information sharing system to consider than the market, where real-time is a requirement. The number of financial transactions per day is at least in the billions.

Currently NYSE and Nasdaq average approximately 2bn shares per day, and the trend is upwards. In terms of shares per trade, the trend in the last four years is sharply downwards, and currently stands at 300 to 400 shares per trade.

A cautionary note, I’m familiar with market basics, but have no inside experience (hedge funds, or brokerage houses). But as an avid data miner, it’s tough to resist the siren’s call of emerging trends in finance. Traders continuiously look at making profit off of market exchanges. I perceive the system as unpredictable, and a lifelike evolving complex adaptive machine. The promise of the market is to provide liquidity, and fairly value just about everything. But how does it self evaluate?

The system is far from perfect. Any inadequacies are ripe targets for real-time trades. I suspect that a number of hedge funds make their money on defined systematic effects and predictable micro trends. The secretive trading systems (proprietary in the extreme) are automated and become ineffective in highly volatile markets.

I’ve spent some time thinking about how much real wealth and value is generated by systematically exploiting the financial system. It’s difficult to measure the real social value of short term exploits of our financial system. Ok, enough of my morale wealth generating philosophy.

notes:
I don’t enunciate (a comical Dragon speech to text error ).

Related articles by Zemanta

• Monetization for Web2010 (victusspiritus.com)
• RSS, Human Filters And Real Time Streams (regulargeek.com)
• Here’s How To Fix The Twitter Suggested User “Advantage” (centernetworks.com)

The Zemanta team (ala @Andraz) has crafted Galaxy, a relational connector between Zemanta and DBpedia + Freebase. This could save me a jump time from entities to connectors! Just heard about it from twitter, here’s the full blog post with all the cool details. Unfortunately there’s no API for such a connection, so we’ll have to do it ourselves anyway. At least I’m aware of the DBpedia and Freebase now.

Crafting a bot that can read & respond with a personality

Artificial intelligence has come a long way in my lifetime, certainly specialized processing such as chess programs have surpassed human cognitive ability. Late yesterday, inspired by the work of my colleague Tyler, I proceeded down a path few have tread successfully. The challenge chosen is to craft an interactive social bot.

Here are the modules I’m using and why I believe it’s possible:
1) semantic entity extraction allows us to quickly identify a category/type and specific person, place or thing
2) relational semantic lookups give multiple connections between entities in the form of verbs or analogies
3) corpus linguistics or a search API can weave a series of entities and verbs together
4) simple replacement rules can serve as the style, speech, or accent.
5) the spooky hard part is making all these pieces play together. I have some ideas, but I’m interested to see what others are cooking up.

Why?

In an effort to improve the appeal and utility of the IMM, we’re exploring new ways of looking at social search. An interface that one can send and recieve messages from is both desirable and natural.

The Hunch interface, and API could serve as a potential source of answers. Hunch is founded on a basic assumption: if you have the same priorities as other folks, their solutions are likely to be good options for you. If you capture the important factors this architecture works. I appreciate the founder Chris Dixon’s blog and suspect good stuff from this service.

Related articles by Zemanta

• Founder Collective: When Entrepreneurs Form Their Own Seed-Stage Venture Firm (xconomy.com)
• Founder Collective Debuts with $40M (xconomy.com)
• Zemanta (fulltimechris.blogspot.com)
• Web Evolution Nova Spivack Twine (slideshare.net)

Our First Marketing Campaign

It’s time for Victus Media to get serious user input. And to achieve that we’ve decided to ratchet up our visibility by sending single tweets to users with a simple message, an extracted tag, and an opt out choice (to never receive further messages). Spam is not something my cofounder or I support but “cold calling” is a necessary evil of obtaining valued user feedback. At the very least, we’ll determine what fraction of users are open to the idea of supporting the development of our take on social semantic search. Our best hopes are for rich user feedback and common desired features as we move forward. Worst case we’ll make some folks angry for attempting to reach out.

Serendipity from Semantic Search

Meeting new people connected by interest, or discovering a peer reviewed topic expert is a tangible benefit of social media. Dynamic user association by leveraging the IMM database is our first attempt at connecting users by topic.

An early idea is to match a person’s tags with other folks that share common interests. In this way we hope to prompt conversations and new friendships.

Another concept is to relate a users tags to dominant tags in our database, and highlight the match between trending topics and what a user is really interested in.

Our Marketing Agent

@semantic2 is our current agent. It randomly processes a users tweet and sends them an appropriate message every 5 minutes (we overloaded our server and got one Twitter account frozen for sending messages too fast).

I can count on two hands the number of people that know and trust Victus Media. We’re tiny. We’re ghosts. We’re two Jekyll an Hyde founders that are ambitious, and driven to unlock great value from socially shared data. We’re passionate about this space because we see immeasurable value slipping through the cracks. The value is leveraging automated systems to learn more about us as users, in a way we’re comfortable with and in control of.

The trick is, people trust buying from Amazon more than random text links. Why not leverage that brand? The service we provide is simply offering users something they may be interested in, as opposed to dumb ads. Spray and pray advertising (banners, static content related, etc) is very limited compare to targeted offers like Google Adsense.

Amazon Branding at Victus Media

For better or worse we got an overhauled Amazon ad widget for our personalized suggestions widget. I was able to dig through the documentation at Amazon and the scripts on their sample pages to get a clean interface for several choice ad variants (I’ll write them up in detail once we go live with them).

Before this, we had simple links to
products we estimated a certain user may be interested in. This is based on their social/semantic tag cloud. Those links were generic, and easy to ignore. At the very least the ads needed some spicing up. Now we have dynamic rich ads with pricing info.

The flip side is that ad blocker will now hide our widget. That’s fine by me, some folks believe in a paid content powered web. And make no mistake, if you believe in blocking all ads, your pushing for a paid content web. I like the mix of attention/paid content.

Our goal is to match the offerings so well that opt in users find them useful. We won’t stop until ads are valued content. Plus those who choose to opt in will get a cool visual search tool (connects your semantic tags to Google and Twitter real time search). That combined with special offers for those who connect will be an irresistable perk. Even if you choose not to connect, you can look up items a friend may be interested in by entering their Twitter ID.