Ensure All Roads Lead to Your Goal (+ Ruby Threads)

The Zen Abstract Design Side

Our world is a function of pre-existing inertia (environment), uncontrollable events, and my personal favorite the malleable future that we craft together. We are challenged daily to pull our sum experiences together to overcome barriers. The greater our ability to collaborate and scale these walls, the further we can go, and the more wonderous our creations. Take stock of your victories, even if they appear to be the product of luck!

Design Challenge, Calling And Waiting on APIs is too Slow

Last night while working with Tyler on improving extracted keywords/entities we discovered a design tradeoff. In this case it was speed versus quality. We focused on the delays of superior quality entity returns and it occurred to me we could spawn many threads to simultaneously call the processing API to relieve sequential user wait time. Neither of us has done multithreaded Ruby coding before so it was new ground. Background processing of information is our main approach but the Twitter streaming API isn’t fully open, and since we are focused on growth we expect many new users (for which we won’t have preprocessed results).

Ruby just happens to make threading incredibly simple to implement:
threadID = Thread.new {
*threadcode*
}
…
threadID.join()

Literally. in a manner of seconds we had sped up a series of API calls to a single batch call (N x user calls was now the slowest of all the calls, a speedup up of potentially hundreds of seconds for large lists of new users). The only detail is when to rejoin the threads. Naturally collapsing the calls will happen before visualizing their outputs. Additional care must be taken for memory writing and storing their returns in a structure. Threads allow local storage to keep API returns, and we need to ensure that we don’t create a thread for the same user more than once (should have that working this afternoon). The coding experience has been phenomenal compared to threading in C++ (a bit trickier), so kudos to Ruby language designers. Many modern languges keep the simplicity of concurrency at the top of their priority list (even C++ is including simpler handling in the new standard).

Now Back to Zen thanks to Dave

I’ll leave off with a powerful poem passed along by my “cynicism guru” Daveinhackensack (Dave believes the scarcity of optimism boosts its value ).
“It comes from this poem by the great New Jersey man of letters Stephen Crane”

There was set before me a mighty hill,
And long days I climbed
Through regions of snow.
When I had before me the summit-view,
It seemed that my labour
Had been to see gardens
Lying at impossible distances.

May all paths lead to beautiful gardens.

Related articles by Zemanta

• Practical First Rails Threading Example (victusspiritus.com)
• 2010: a fresh new start (zemanta.com)
• Analyzing a FriendFeed group with Ruby and R (revolution-computing.com)
• [Ruby] A few points about Ruby threading… (nandz.blogspot.com)