Build a ‘Klout Klock’, track your influence and time.

Recently, my friend Bertrand wrote an interesting article about how he came to realize that we have become a “Society of Many Tribes”. Go ahead and read the article. It will be worth your time Smile The core concept of the article centers around “Thinking locally first and sharing your ideas with your tribes will eventually result in your impact becoming global”. This is a fascinating idea, begging to be experimented with. But how can one actually measure the breadth, depth and impact of one’s ideas as they travel and evolve through social networks?

Curiously, it turns out that the Klout team has come up with a set of metrics to do just that. If you have a Twitter, Facebook or LinkedIn account, you should try to link them to Klout to understand how this works: in a nutshell, Klout applies a set of algorithms, similar to Google’s page ranking algorithm, to your interactions and relationships within social networks and comes up with a synthetic score on a scale of 1 to 100, representing your overall influence a.k.a your “Klout Score”. A “Klout Score” is itself broken down into multiple data points, described in details here.

Klout’s metrics are interesting from a self-development standpoint because they can be used as a feedback loop: with them, you can evaluate within days or hours the effect of your communications, measuring what works vs. what doesn’t and learn to focus a message in the areas where it will resonate with a target audience. This is a valuable tool for individual bloggers and businesses alike.

Building a Klout Klock

Klout  exposes a web service enabling developers to build mash-up applications around its metrics and all that is required to play is an API key which is easily obtained when registering an application. My application is the “Klout Klock” device and before getting into the details of building it, you can see it how it works in this video:

The clock is built using a Netduino Plus and an AdaFruit ST7735 TFT screen. I have described how to connect them together in a previous post here. In that post, I had indicated that managing such a TFT screen from a Netduino was sub-optimal due to the memory requirements involved. That statement is even more true with a Netduino Plus which has roughly 28KB of RAM available for an application. This means that allocating a 40KB buffer to manage the TFT display as I was doing it previously is out of the question. Also, this time I am using the Netduino Plus’ SD card reader instead of the one built into the AdaFruit ST7735 breakout board. In addition, the screen’s back light is hooked up to a PWN pin to achieve the fade-in / fade-out effect based on the iCuffLink breathing effect 🙂

So, what do you do when you need to fit 10 lbs of crap in a 5 lbs bag? You use a virtual bag, of course! 🙂

Using an SD card as Virtual Memory

The Netduino Plus, in addition to its Ethernet connector, features a handy micro SD card reader for storage which I turned into virtual memory to get around the RAM constraints.

The virtual memory concept is straight forward: instead of reading and writing to/from a RAM buffer matching the size of the AdaFruit TFT display, all reading and writing operations happen against a file whose size matches the size of the display. In this case, that file is exactly 40960 bytes, the size required to hold a 128*160*16 bit/pixel image.

To actually refresh the screen with the image contained in virtual memory, the physical display is updated in ‘segments’, reading the data for each segment into a small RAM buffer, then blasting its content over SPI to the display. For each segment update, a display address window is positioned to cover the proper range of pixels.

The trade-off here is speed since file I/Os are orders of magnitude slower than operations in RAM but speed is not the main concern in this scenario. In addition, the more memory segments are used to chop up a virtual memory block, the slower it gets.

The following diagram shows how the process of updating the display from segmented virtual memory works:

I have added two classes to the Netduino Helpers library, respectively called “VirtualMemory.cs” and “VirtualFrame.cs” which make this process simple. I also refactored the driver that I had previously written for the AdaFruit ST7735 to make use of virtual memory if desired:

In this snippet, a virtual frame the size of the display is created and divided into 16 segments, ensuring that the I/O buffer used to read the image will only require 2560 bytes. The number of segments should be determined by a balance between performance and the overall memory constraints for the rest of the application. The reference to the newly minted VirtualFrame object is then passed to the instance of the display driver. Everything else about the driver works exactly as it did previously when actual RAM was being used.

The VirtualFrame class

This class is the workhorse driving the display and it’s worth explaining its core functions:

  • Copy: this method clones a virtual memory object into the current one. The two copies are identical afterwards.
  • Merge: this method takes a ‘background’ frame and a ‘sprite’ frame and combines them into the current virtual frame, effectively making this method a ‘BitBlit‘ function. Note that sprite transparency is not yet implemented. This method is used to display arbitrary graphics as well as printing strings on the display.
  • GetPathFromCharacter: given a character, this method returns the full path to the file containing the bitmap representing the character. This is heavily used by the Print method to print strings on the display. It relies on the ‘BitmapDirectory’ property to determine the final path.
  • Print: prints a character string at an arbitrary location on the display. Font transparency is not yet supported (see Merge). The dimensions of the font bitmaps must be provided as well. Also, the font is strictly limited to what is required by the clock application.
The following snippet shows how these methods are used to display the time and date on the display:

Strategies employed to dodge the dreaded “Out Of Memory” exceptions

In many ways, the “Klout Klock” application really pushes the envelope of what is possible on a Netduino Plus. To put things into perspective, here are the application requirements that needed to fit in 28KB of RAM:

  • Use NTP time to update the internal clock of the Netduino Plus
  • Use HTTP REST calls to fetch Klout metrics
  • Parse JSON responses containing the Klout metrics
  • Display the date, the time and the Klout metrics
  • Drive the AdaFruit ST7735 TFT screen
To make it all fit, none of the standard .Net Micro Framework classes supporting HTTP requests and JSON parsing could be used without systematically running out of memory and they had to be substituted: HTTP requests are made by the application using plain old TCP socket connections to the Klout web services, caching the responses to the SD card.
Being unable to find a suitable embedded JSON parser written in C#, I implemented a lightweight, non-recursive, JSON parser designed to process data directly from file streams. Finally, it was also necessary to explicitly manage the lifetime of C# objects and invoking the Garbage Collector along the way. In the end, this constant focus on being as lean as possible led to a C# application structured a bit like an application written in C 🙂

Source Code and Configuration

The source code for the “Klout Klock” is part of the netduino Helpers library on CodePlex and is located under the \Samples folder. To get your own “Klout Klock” going, you will need to edit the ‘resources.txt’ file located under \Samples\KloutKlock\KloutKlock\SD Card Resources.
You will need to register a Klout user name and acquire a Klout API key for yourself and substitute them for the user and key values highlighted in red below. The NTP servers referenced in the ntpServers value are tailored for folks living on the West Coast of the U.S, in the Pacific time zone, so you should probably select a set of servers closer to your actual location for best results. Likewise, the timeZone value should reflect your own time zone. Finally, if Daylight Savings Time is ON where you live, you should set the dst value to 1 and 0 otherwise.
* NTP Server list: http://tf.nist.gov/tf-cgi/servers.cgi
string:name=ntpServers;value=nist1-lv.ustiming.org,nist1.aol-ca.symmetricom.com,nist1.symmetricom.com,nist1-sj.ustiming.org,nist1-la.ustiming.org,utcnist.colorado.edu,utcnist2.colorado.edu,ntp-nist.ldsbc.edu
*
string:name=user;value=<KLOUT user name>
string:name=key;value=<KLOUT API Key>
string:name=host;value=api.klout.com
string:name=port;value=80
string:name=timeZone;value=-8
string:name=dst;value=1
Konclusion
For folks interested and actively participating in social networks, for fun or for profit, Klout has valuable insights to offer. As Klout integrates their services with more and more social networks, the relevance and accuracy of their metrics is bound to improve as well. Given their overlap, focus and methods, I would not be surprised if Google acquired Klout one of these days.
 
Now, if you excuse me, I need to go build an ‘EnKlosure’ for my Klock 😉

Links:
Fallout New Vegas

Building a Twitter-enabled Geiger counter with a netduino+

In 1986, when the Chernobyl disaster happened, I was living in France. The whole country was scared of the fallout resulting from the radioactive cloud travelling over Europe. The French government downplayed the  impact of the nuclear accident on the country, manipulated the media and was less than forthcoming about the amount of radiation that we were  absorbing. Not cool. Fast forward to 2011: I now live in Seattle, 4792 miles away from Japan. On March 11th, the Fukushima nuclear accident happens, bringing back many memories from 1986: would the government lie again about the gravity of the situation? Could we trust the data that the EPA is publishing about radiation levels in Washington state? I was determined to find out for myself and started looking for parts to build a Geiger counter.

Integrating a Geiger counter with a netduino

I eventually discovered that Electronic Goldmine was selling a bare bone radiation detection kit at a decent price. At a time when everyone else on the web was sold out of Geiger counters, this was perfect!

I just needed to figure out a way to count the pulses coming from the sensor. The solution that I came up with was simple: the kit features an LED which flashes whenever a pulse is detected by the Geiger-Mueller tube. If I could read each pulse and generate an interrupt from it, it would be easy to count them.

Based on that idea, I built an opto-coupler circuit using parts that I had on hand: 1 photodiode, 3 NPN transistors (2N2222) and 2 LEDs.

When the photodiode detects the light emitted by the yellow LED connected to the Geiger sensor, it energizes the base of the first NPN transistor which in turn amplifies the current and saturates the base of the 2nd transistor to flash the red LED. The base of the 2nd transistor is also connected to the base of the 3rd one, which drives an InterruptPort to ground on the netduino, resulting in a pulse-counting interrupt handler to be called in the application:

            _geigerCounter = new InterruptPort(
                interruptPin,
                true,
                Port.ResistorMode.PullUp,
                Port.InterruptMode.InterruptEdgeLevelLow);
_geigerCounter.OnInterrupt += PulseCounter;

This effectively reproduces the function of an opto-coupler like this one, even though my version is much ‘noisier’ than Vishay’s component.

NOTE: The benefit of this method of communication between the two devices is that they’re electrically isolated from each other: the Geiger-Mueller tube requires ~450 volts to function, which would instantly destroy the netduino+ if somehow the two circuits were connected together by mistake. You obviously need to be careful while working with such high voltages or you could get hurt even if the current is low.

The second part of the circuit consist of a NXP HEF4050 level-shifter used to drive a 5 volt LCD screen through a serial backpack. Everything fits on a DIY shield piggybacking the netduino+:

In addition to counting pulses, the interrupt handler uses the time intervals between pulse pairs to generate a random stream of bits, using a 32 bit integer as the ‘window’ on the bit stream. This will be handy should you require more randomness in your life 😉 This random number generation method is based on John Walker’s excellent article about how his HotBits system works.

            if (_ticksCount < _ticks.Length) {
                _ticks[_ticksCount++] = time.Ticks;
            } else {
                _ticksCount = 0;
                _flipper ^= true;

                var Interval1 = _ticks[1] - _ticks[0];
                var Interval2 = _ticks[3] - _ticks[2];

                if (Interval1 != Interval2) {
                    _randomNumber <<= 1;
                    if (_flipper) {
                        _randomNumber |= (Interval1 < Interval2) ? 0 : 1;
                    } else {
                        _randomNumber |= (Interval1 > Interval2) ? 0 : 1;
                    }
                }
            }

Connecting to Twitter

Every so often, the application reports  data to Twitter:

To achieve this simply, I used Matt Isenhower’s MicroTweet library, which I modified to make it more resilient to networking issues and OutOfMemory exceptions.

I included my changes in a private build which I provided with the source code of this application. Hopefully, this will only be temporary as Matt’s already working on improving MicroTweet.

The configuration of the application is done by reading a set of parameters from a file on the netduino+ microSD card  like this:

* NTP Server list: http://tf.nist.gov/tf-cgi/servers.cgi
string:name=ntpServers;value=nist1-lv.ustiming.org,time-nw.nist.gov,nist1.aol-ca.symmetricom.com,nist1.symmetricom.com,nist1-sj.ustiming.org,nist1-la.ustiming.org,time.nist.gov,utcnist.colorado.edu,utcnist2.colorado.edu,ntp-nist.ldsbc.edu
*
* Register your application @ http://dev.twitter.com/
* To obtain your tokens and secrets check the 'Application details' page and copy them below
*
string:name=accessToken;value=???
string:name=accessTokenSecret;value=???
string:name=consumerKey;value=???
string:name=consumerSecret;value=???
*
* Geiger counter location GPS coordinates
*
string:name=latitude;value=47.6741667
string:name=longitude;value=-122.1202778

To make parsing such configuration files easy, the netduino Helpers library provides a generic resource loader:

        private static void InitializeResources() {
            var resourceLoader = new SDResourceLoader();

            resourceLoader.Load();

            _ntpServers = (string)resourceLoader.Strings["ntpServers"];
            _latitude = (string)resourceLoader.Strings["latitude"];
            _longitude = (string)resourceLoader.Strings["longitude"];

            _twitter = new TwitterClient(
                            (string)resourceLoader.Strings["consumerKey"],
                            (string)resourceLoader.Strings["consumerSecret"],
                            (string)resourceLoader.Strings["accessToken"],
                            (string)resourceLoader.Strings["accessTokenSecret"]);

            resourceLoader.Dispose();

            Debug.GC(true);
        }

The enclosure

The box I selected for this project is made of a thin, light wood perfect for painting and decorating. The bottom part of the case being so thin precludes using standoffs and screws to secure the electronics. The approach I took was to use sticky Velcro strips instead which worked really well: all the parts stay firmly in their place, yet they’re easy to remove from the case if needed:

The final touch to the case was the Fallout 3 theme, courtesy of the artwork shared by Bethesda for Fallout New Vegas. If you’re a gamer, you will likely be familiar with the Fallout universe, where your character’s survival depends on a wearable computer, the PIP Boy 3000, which features a Geiger counter. Being a huge Fallout fan, I felt compelled to pay homage to the game with this project 🙂

Conclusion

When comparing the data samples from this project to the data produced by the EPA’s air radiation monitoring stations in the area, the averages are aligned for the most part, which makes me feel better about the transparency and the integrity of our local government.

Action shot

Links: