cortesi

Introducing pathod: a pathological HTTP server

Tue, 01 May 2012 08:14:00 GMT

Introducing pathod: a pathological HTTP server

01 May 2012

I've just released pathod, a pathological HTTP/S daemon useful for testing and torturing HTTP clients. At its core is a tiny, terse language for crafting HTTP responses. It also has a built-in web interface that lets you play with the response spec language, inspect logs, and access pathod's full help document.

The rest of this post is a quick teaser showing some of pathod's abilities. See the detailed documentation on the pathod site if you want more.

The simplest possible response

The easiest way to craft a response is to specify it directly in the request URL. Lets start with the simplest possible example. Start pathod, and then visit this URL:

http://localhost:9999/p/200

The "/p/" path is the location of the response generator in pathod's default configuration - everything after that a response specification in pathod's mini-language. The general form of a response spec is as follows:

code[MESSAGE]:[colon-separated list of features]

In this case, we're specifying only the HTTP response code - that is, an HTTP 200 OK with no headers and no content, resulting in a response like this:

HTTP/1.1 200 OK

Specifying features

One example of a "feature" is a response header. Lets embellish our response by adding one:

200:h"Etag"="foo"

The first letter of the feature - "h", in this case - is a mnemonic indicating the type of feature we're adding. The full response to this spec looks like this:

HTTP/1.1 200 OK
Etag: foo

Both "Etag" and "foo" are Value Specifiers, a syntax used throughout the response specification language. In this case they are literal values, as indicated by the fact that they are quoted strings. The Value Specification syntax also lets us load values from files or generate random data. For instance, here is a specification that generates 100k of random binary data for the header value:

200:h"Etag"=@100k

Now, binary data in the header value will probably break things in interesting ways, but is unlikely to be read by the client as a valid (but over-long) value. To see if the client really drops off its perch if we feed it a single 100k header, we have to constrain the random data. Here's the same response, but with data generated only from ASCII letters:

200:h"Etag"=@100k,ascii_letters

pathod has a large number of built-in character classes from which random data can be generated.

Pauses and Disconnects

Next, we can disrupt the communications in various ways. At the moment, this means adding pauses and disconnects to a response. Let's start with an HTTP 404 response with a body consisting of a 100k of random binary data:

404:b@100k

Here's the same response, but with a 120 second pause after sending 100 bytes:

404:b@100k:p120,100

And, the same response again, but with hard disconnect after sending 100 bytes:

404:b@100k:d100

Instead of specifying a time explicitly, we can ask pathod to just randomly disconnect at a time of its choosing:

404:b@100k:dr

That's it for the teaser - hopefully it's enough to entice you into looking at pathod's full documentation.

What's next?

pathod is an "airport project" - the first draft was written in its entirety during a 40-hour trip back home from New York (I drew a bad lot in stopovers). I've now firmed it up a bit, but there's still work to be done. In the next month, mitmproxy's test suite will move to pathod, after which there will be a simple, well-documented way to unit test. I also plan to build out the JSON API (which is used to drive pathod in test suites), and expand the mini-language with convenient ways to generate pathological cookies, authentication headers, SSL errors, and cache control.

mitmproxy 0.8

Mon, 09 Apr 2012 16:57:00 GMT

mitmproxy 0.8

09 April 2012

I'm happy to announce the release of mitmproxy 0.8. This release has a few major new features, big speedups, and many, many small bugfixes and improvements. Here are the headlines:

Android interception

The most prominent new feature is that we now have a supported way to intercept Android traffic. What's more, we can do this without a cumbersome transparent proxying rig - see the Android section in the documentation for the details. Special thanks goes to Jim Cheetham for lending me an Android device and helping to get this feature off the ground.

Replacement patterns

Another exceedingly useful new feature is replacement patterns. These consist of a filter, a regular expression and a replacement string, and run continuously while mitmproxy processes requests and responses. You can pass these either on the command-line, or using a built-in replacement pattern editor.

I'm sure you can immediately think of many uses for this flexible feature, but my favourite is to use it during testing as a way to conveniently inject complicated exploits into web traffic. I do this by setting a replacement pattern that swaps a short but likely unique string (say MYXSS) for a long exploit, and then I use simple interaction and front-end tools like Firebug to inject exploits into requests manually based on the short string marker.

Improved pretty-printing of request and response contents

This release of mitmproxy has a completely redesigned subsystem for pretty-printing request and response bodies. For instance, we now extract EXIF tags and other basic information to give you something better than a hex dump when looking at an image:

We also have much improved HTML indenting (using lxml), and a built-in JavaScript beautifier (thanks to JSBeautifier) that teases out compressed and obfuscated scripts into something readable.

Changelog

Detailed tutorial for Android interception. Some features that land in this release have finally made reliable Android interception possible.
Upstream-cert mode, which uses information from the upstream server to generate interception certificates.
Replacement patterns that let you easily do global replacements in flows matching filter patterns. Can be specified on the command-line, or edited interactively.
Much more sophisticated and usable pretty printing of request bodies. Support for auto-indentation of JavaScript, inspection of image EXIF data, and more.
Details view for flows, showing connection and SSL cert information (X keyboard shortcut).
Server certificates are now stored and serialized in saved traffic for later analysis. This means that the 0.8 serialization format is NOT compatible with 0.7.
Add a shortcut key ("f") to load the remainder of a request or response body, if it is abbreviated.
Many other improvements, including bugfixes, and expanded scripting API, and more sophisticated certificate handling.

mitmproxy 0.7

Mon, 27 Feb 2012 20:38:00 GMT

mitmproxy 0.7

27 February 2012

I'm happy to announce the release of mitmproxy 0.7. The biggest visible change is a new structured editor for headers, query strings and form fields. Other new feature include a reverse proxy mode, extended script API that makes many common tasks much easier, and a myriad of improvements to the interface (including a massive increase in speed). Everybody still on 0.6 should upgrade - get it here:

mitmproxy-0.7.tar.gz (docs)

You can also now install mitmproxy using pip, like so:

pip install mitmproxy

In other news, the project has had an amazing month, after a rash of high-profile results obtained using mitmproxy were published. It started with Arun Thampi's discovery that Path uploads users' address books to their servers. Things snowballed from there, and for a few days mitmproxy seemed to be everywhere. Similar findings were made for Hipster, The Verge did a mitmproxy-driven AddressbookGate expose (including vaguely threatening background shots of mitmproxy doing its dastardly work), and lots of people said nice things on Twitter.

To see the impact all of this for the mitmproxy project, you need only look at the Github page - watchers of the repo went from about 200 a month a go, to 950 at the time of this post.

Changelog

New built-in key/value editor. This lets you interactively edit URL query strings, headers and URL-encoded form data.
Extend script API to allow duplication and replay of flows.
API for easy manipulation of URL-encoded forms and query strings.
Add "D" shortcut in mitmproxy to duplicate a flow.
Reverse proxy mode. In this mode mitmproxy acts as an HTTP server, forwarding all traffic to a specified upstream server.
UI improvements - use Unicode characters to make GUI more compact, improve spacing and layout throughout.
Add support for filtering by HTTP method.
Add the ability to specify an HTTP body size limit.
Move to typed netstrings for serialization format - this makes 0.7 backwards-incompatible with serialized data from 0.6!
Significant improvements in speed and responsiveness of UI.
Many minor bugfixes and improvements.

OpenBSD in decline?

Sun, 26 Feb 2012 09:08:00 GMT

OpenBSD in decline?

26 February 2012

My leisurely Sunday activity today is to set up a new OpenBSD firewall for my mobile app testing lab. I haven't done a from-scratch OpenBSD install for years, so I spent some time reading through the change logs for the last few versions to catch up with what's changed. Although the project is clearly still making steady, well-engineered progress, I had the nagging feeling that the rate of change wasn't what it used to be. So, I pulled some numbers from CVS commit message list archives, and graphed them. Here are the number of commits per month from January 2001 to January 2012. The orange line is a simple 12-month moving average:

Now, we should be cautious about interpreting this - the number of commits doesn't tell us anything about the quality, importance or magnitude of code change. Even if it did all of these things, there are other and perhaps better measures of a project's health. Still, the trend is clear, and suggests a sustained decline in activity.

I just bought some T-shirts to help support one of my favourite open source projects. You should too.

Malware

Thu, 05 Jan 2012 22:50:00 GMT

Malware

05 January 2012

Hover and click for more.

The images above are entropy visualizations of samples from a malware database - black is zero entropy, with colour ranging through blue, up to hot pink for maximum entropy. Large areas of very high entropy are usually sections that are packed - encrypted or obfuscated by the malware authors to make the malware hard to detect and reverse engineer. Smaller areas might be keys, passwords, or other chunks of data meant to be hidden from view.

When you hover over an image, you see a character class visualization with the following colors:

	0x00
	0xFF
	Printable characters
	Everything else

Clicking will show you high-detail versions of both visualizations, and let you look up the binary hash to see what it is. I've used a square Hilbert curve layout - the files start in the top-left corner, and pass through the quadrants clockwise.

I spent hours looking through thousands these visualizations today. I find them eerie and rather beautiful - an entirely different perspective from my day-to-day interactions with malware.

Visualizing entropy in binary files

Wed, 04 Jan 2012 05:26:00 GMT

Visualizing entropy in binary files

04 January 2012

Last week, I wrote about visualizing binary files using space-filling curves, a technique I use when I need to get a quick overview of the broad structure of a file. Today, I'll show you an elaboration of the same basic idea - still based on space-filling curves, but this time using a colour function that measures local entropy.

Before I get to the details, let's quickly talk about the motivation for a visualization like this. We can think of entropy as the degree to which a chunk of data is disordered. If we have a data set where all the elements have the same value, the amount of disorder is nil, and the entropy is zero. If the data set has the maximum amount of heterogeneity (i.e. all possible symbols are represented equally), then we also have the maximum amount of disorder, and thus the maximum amount of entropy. There are two common types of high-entropy data that are of special interest to reverse engineers and penetration testers. The first is compressed data - finding and extracting compressed sections is a common task in many security audits. The second is cryptographic material - which is obviously at the heart of most security work. Here, I'm referring not only to key material and certificates, but also to hashes and actual encrypted data. As I show below, a tool like the one I'm describing today can be highly useful in spotting this type of information.

For this visualization, I use the Shannon entropy measure to calculate byte entropy over a sliding window. This gives us a "local entropy" value for each byte, even though the concept doesn't really apply to single symbols.

With that out of the way, let's look at some pretty pictures.

Visualizing the OSX ksh binary

In my previous post, I used the ksh binary as a guinea pig, and I'll do the same here. On the left is the entropy visualization with colours ranging from black for zero entropy, through shades of blue as entropy increases, to hot pink for maximum entropy. On the right is the Hilbert curve visualization from the last post for comparison - see the post itself for an explanation of the colour scheme. Click for larger versions with much more detail:

Entropy	Byte class

Note that this is a dual-architecture Mach-O file, containing code for both i386 and x86_64. You can see this if you squint somewhat at these images - some broad structures in the file are repeated twice. We can see that there are a number of different sections of the ksh binary that have very high entropy. It's not immediately obvious why a system binary would contain either compressed sections or cryptographic material. As it happens, the explanation in this case is quite interesting. Let's have a closer look:

Sections 1 and 2 are a lovely validation of the central idea of this post. These two areas do indeed contain cryptographic material - in this case, code signing hashes and certificates. Rather satisfyingly, they stand out like a sore thumb. It turns out that all of the official OSX binaries are signed by Apple. This is then used in turn to apply a variety of policies, depending on who the signatory is, and whether they are trusted.

You can dump some rudimentary data about a binary's signature using the codesign command (which you can also use to sign binaries yourself):

> codesign -dvv /bin/ksh 
Executable=/bin/ksh
Identifier=com.apple.ksh
Format=Mach-O universal (i386 x86_64)
CodeDirectory v=20100 size=5662 flags=0x0(none) hashes=278+2 location=embedded
Signature size=4064
Authority=Software Signing
Authority=Apple Code Signing Certification Authority
Authority=Apple Root CA
Info.plist=not bound
Sealed Resources=none
Internal requirements count=1 size=92

Section 3 (the two occurrences are the same data repeated for each architecture) is interesting for a different reason - it's a cautionary example of how the simple entropy measure we're using sometimes detects high entropy in highly structured data. A hex dump of the start of the region looks like this:

000d1f00  00 01 00 00 00 02 00 00  00 06 00 00 00 00 00 00  |................|
000d1f10  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000d1f20  00 01 02 03 04 05 06 07  08 09 0a 0b 0c 0d 0e 0f  |................|
000d1f30  10 11 12 13 14 15 16 17  18 19 1a 1b 1c 1d 1e 1f  |................|
000d1f40  20 21 22 23 24 25 26 27  28 29 2a 2b 2c 2d 2e 2f  | !"#$%&'()*+,-./|
000d1f50  30 31 32 33 34 35 36 37  38 39 3a 3b 3c 3d 3e 3f  |0123456789:;<=>?|
000d1f60  40 41 42 43 44 45 46 47  48 49 4a 4b 4c 4d 4e 4f  |@ABCDEFGHIJKLMNO|
000d1f70  50 51 52 53 54 55 56 57  58 59 5a 5b 5c 5d 5e 5f  |PQRSTUVWXYZ[\]^_|
000d1f80  60 61 62 63 64 65 66 67  68 69 6a 6b 6c 6d 6e 6f  |`abcdefghijklmno|
000d1f90  70 71 72 73 74 75 76 77  78 79 7a 7b 7c 7d 7e 7f  |pqrstuvwxyz{|}~.|
000d1fa0  80 81 82 83 84 85 86 87  88 89 8a 8b 8c 8d 8e 8f  |................|
000d1fb0  90 91 92 93 94 95 96 97  98 99 9a 9b 9c 9d 9e 9f  |................|
000d1fc0  a0 a1 a2 a3 a4 a5 a6 a7  a8 a9 aa ab ac ad ae af  |................|
000d1fd0  b0 b1 b2 b3 b4 b5 b6 b7  b8 b9 ba bb bc bd be bf  |................|
000d1fe0  c0 c1 c2 c3 c4 c5 c6 c7  c8 c9 ca cb cc cd ce cf  |................|
000d1ff0  d0 d1 d2 d3 d4 d5 d6 d7  d8 d9 da db dc dd de df  |................|
000d2000  e0 e1 e2 e3 e4 e5 e6 e7  e8 e9 ea eb ec ed ee ef  |................|
000d2010  f0 f1 f2 f3 f4 f5 f6 f7  f8 f9 fa fb fc fd fe ff  |................|

We see that this section contains each byte value from 0x00 to 0xff in order - furthermore this whole block is repeated with minor variations a number of times. There are two things to explain here - why is this detected as "high entropy" data, and what the heck is it doing in the file?

First, we need to understand that the Shannon entropy measure looks only at the relative occurrence frequencies of individual symbols (in this case, bytes). A chunk of data like the one above therefore looks like it has high entropy, because each symbol occurs once and only once, making the data highly heterogeneous.

Now, what earthly use would chunks of data like this be? With a bit of digging, I found the answer in the ksh source code. These sections are maps used for translation between various character encodings. If you're interested, here's the culprit in all its repetitive glory.

The code

As usual, the code for generating all of the images in this post is up on GitHub. The entropy visualizations were created with binvis, a new addition to scurve, my compendium of code related to space-filling curves.

A personal link mill

Fri, 30 Dec 2011 15:42:00 GMT

A personal link mill

30 December 2011

I posted a link to an interesting visualization paper on Twitter today, prompting someone to ask me where I had found it. Sadly, I had to admit that I had no clue where I first saw it referenced, due to the way I consume links I find on the net. So, I thought I'd write a quick blog post to explain myself, and then pitch a product idea that could make my life (and maybe yours) much easier.

First, the problem statement: my aim is to efficiently discover links to interesting stuff on the net. Simple as that. A few years ago, my flow of links came mostly from social news sites (Hacker News and Reddit), and items shared by people I follow on social networks. Over time, I became more and more disenchanted with this way of doing things. The social news approach is to take a torrent of very low quality links (user submissions), and then crowd-source the filtration process through voting. But popularity is not a good measure of information quality, and the result is a bland, lowest-common-denominator view of the world that has no room for anything that doesn't make it to the front page. Don't get me wrong - Reddit and HN do a lot of other things well - but they just don't cut it as primary information sources. Mining links from social networks is a more promising approach, but still problematic. None of the social networks provide the tools needed to extract shared links from the update stream and consume them efficiently. There is also a structural issue - I don't necessarily want to mix my social ties and my information sources, and I definitely don't want to be limited to just one platform. These are separate functions that I feel require separate tools.

My personal link mill

Eventually, I took matters into my own hands. First, I hugely broadened the number of information sources I consumed. The tool I use for this is Google Reader - I now subscribe to about 800 individual feeds, and this number is growing daily. The trick here is to find high-quality, low-volume link sources. The motherlode of good links for me was to be found on social bookmarking sites. About 700 of my subscriptions are to the RSS feeds of individual users on Pinboard and Delicious. This gives me very fine control and a great mix of interests. Plus, getting links from individual curators handily sidesteps the social news group-think problem. The remainder of my subscriptions are split between blogs, some sub-Reddits, a few Twitter users and subsections of arXiv.

So much for how my intake works. Just as important is the way that I consume it. I do my "filtering" in batches, usually in the evening. Using Reeder on my iPad works well for me, letting me flick quickly and comfortably through all the new links of the day. When I find something that looks interesting, I resist the temptation to read it then and there - instead, I batch up all my reading for later. If it's a web page, it goes to Instapaper. If it's a PDF, it gets downloaded into a DropBox folder, which is synced to GoodReader.

Finally, the actual reading. Every morning, I toddle off to a nice cafe with my iPad, and read all the interesting stuff I saved the previous day in a single sitting. I'm ruthless about just skimming things that don't warrant careful attention. If I find something particularly interesting I save it permanently, and perhaps tweet it or mail it to someone I think might be interested.

Problems - and a product idea?

This system works for me, but it has many problems. There's no end-to-end coordination, so by the time I sit down to actually read something, I have no easy way to tell which feed it came from. Google Reader sucks at managing hundreds of low-volume subscriptions. Reeder is a great, but is not tailored to consuming redundant information from many sources. The end result is that maintaining the system I have is a time-consuming pain in the ass. The fact that it's still worth it despite this, makes me think there might be commercial room for a better solution.

Which brings me to a rough product idea - a formalized version of this link mill for people who want to take direct control of their information intake. The business end is a generalized feed consumer, letting you subscribe to RSS feeds, Twitter users, Google+ updates, sub-Reddits and other information sources. Links are extracted from these feeds, keeping track of which links appeared where. The user is then presented with a stream of links to consume, de-duplicated so that those appearing in multiple feeds are presented only once. The system keeps track of links the user marks as "interesting", batching them for later consumption. It also uses this information to score the feeds, letting the user see which feeds are low quality, and should be ditched. Given the right tools, the time needed for a user to maintain and tend their link feed garden would be quite modest, and the rewards would be great.

If someone built this, I for one would gladly fork over some of my hard-earned doubloons to use it. In fact, with some validation of the idea and a few collaborators I might think of building it myself. Does this sound useful to anyone else?

Visualizing binaries with space-filling curves

Fri, 23 Dec 2011 06:02:00 GMT

Visualizing binaries with space-filling curves

23 December 2011

In my day job I often come across binary files with unknown content. I have a set of standard avenues of attack when I confront such a beast - use "file" to see if it's a known file type, "strings" to see if there's readable text, run some in-house code to extract compressed sections, and, of course, fire up a hex editor to take a direct look. There's something missing in that list, though - I have no way to get a quick view of the overall structure of the file. Using a hex editor for this is not much chop - if the first section of the file looks random (i.e. probably compressed or encrypted), who's to say that there isn't a chunk of non-random information a meg further down? Ideally, we want to do this type of broad pattern-finding by eye, so a visualization seems to be in order.

First, lets begin by picking a colour scheme. We have 256 different byte values, but for a first-pass look at a file, we can compress that down into a few common classes:

	0x00
	0xFF
	Printable characters
	Everything else

This covers the most common padding bytes, nicely highlights strings, and lumps everything else into a miscellaneous bucket. The broad outline of what we need to do next is clear - we sample the file at regular intervals, translate each sampled byte to a colour, and write the corresponding pixel to our image. This brings us to the big question - what's the best way to arrange the pixels? A first stab might be to lay the pixels out row by row, snaking to and fro to make sure each pixel is always adjacent to its predecessor. It turns out, however, that this zig-zag pattern is not very satisfying - small scale features (i.e. features that take up only a few lines) tend to get lost. What we want is a layout that maps our one-dimensional sequence of samples onto the 2-d image, while keeping elements that are close together in one dimension as near as possible to each other in two dimensions. This is called "locality preservation", and the space-filling curves are a family of mathematical constructs that have precisely this property. If you're a regular reader of this blog, you may know that I have an almost unseemly fondness for these critters. So, lets add a couple of space-filling curves to the mix to see how they stack up. The Z-Order curve has found wide practical use in computer science. It's not the best in terms of locality preservation, but it's easy and quick to compute. The Hilbert curve, on the other hand, is (nearly) as good as it gets at locality preservation, but is much more complicated to generate. Here's what our three candidate curves look like - in each case, the traversal starts in the top-left corner:

Zigzag	Z-order	Hilbert

And here they are, visualizing the ksh (Mach-O, dual-architecture) binary distributed with OSX - click for the significantly more spectacular larger versions of the images:

Zigzag	Z-order	Hilbert

The classical Hilbert and Z-Order curves are actually square, so for these visualizations I've unrolled them, stacking four sub-curves on top of each other. To my eye, the Hilbert curve is the clear winner here. Local features are prominent because they are nicely clumped together. The Z-order curve shows some annoying artifacts with contiguous chunks of data sometimes split between two or more visual blocks.

The downside of the space-filling curve visualizations is that we can't look at a feature in the image and tell where, exactly, it can be found in the file. I'm toying with the idea (though not very seriously) of writing an interactive binary file viewer with a space-filling curve navigation pane. This would let the user click on or hover over a patch of structure and see the file offset and the corresponding hex.

More detail

We can get more detail in these images by increasing the granularity of the colour mapping. One way to do this is to use a trick I first concocted to visualize the Hilbert Curve at scale. The basic idea is to use a 3-d Hilbert curve traversal of the RGB colour cube to create a palette of colours. This makes use of the locality-preserving properties of the Hilbert curve to make sure that similar elements have similar colours in the visualization. See the original post for more.

So, here's a Hilbert curve mapping of a binary file, using a Hilbert-order traversal of the RGB cube as a colour palette. Again, click on the image for the much nicer large scale version:

This shows significantly more fine-grained structure, which might be good for a deep dive into a binary. On the other hand, the colours don't map cleanly to distinct byte classes, so the image is harder to interpret. An ideal hex viewer would let you flick between the two palettes for navigation.

The code

As usual, I'm publishing the code for generating all of the images in this post. The binary visualizations were created with binvis, which is a new addition to scurve, my space-filling curve project. The curve diagrams were made with the "drawcurve" utility to be found in the same place.

netograph.com - Realtime privacy snapshots of the social web

Thu, 08 Dec 2011 05:39:00 GMT

netograph.com - Realtime privacy snapshots of the social web

08 December 2011

Today, I'm launching Netograph, a new privacy-related site that I've been hacking on over the past few months. The goal of the project is to provide you with a quick overview of the privacy picture for a URL, before you've clicked on the link. At the moment, Netograph scans Reddit, Hacker News, Pinboard, Delicous and Digg - links on these sites should show up within a few minutes of submission.

For more details, head over to netograph.com. There you will also find Firefox and Chrome browser addons that let you view the Netograph report for a URL instantly with a right-click. Enjoy!

guardian.co.uk

techcrunch.com

reddit.com

What's next?

This is just the first step. As I hinted in a previous post, the most interesting results from Netograph are likely to come from aggregating and cross-correlating the data for individual URLs. I'm already hard at work on this - the next iteration of Netograph will aim to shine some light on the sometimes shadowy network of third-parties that track and analyze nearly every URL we visit. I will also be publishing some interesting tidbits from this data corpus on my blog as I go along, so watch this space.

Otago Polytechnic Talk

Mon, 31 Oct 2011 09:05:00 GMT

Otago Polytechnic Talk

31 October 2011

Further reading for the guest lecture I'm giving at Otago Polytechnic today:

The talk I'm not giving: OWASP Top 10
Tools: FireBug, TamperData, Python.
The Myspace Worm, and Samy Kamkar's own explanation of the exploit.
Halvar Flake's Programming and state machines, which is where I first saw the term "programming the weird machine".

Neighborhoods of trust on the web

Tue, 27 Sep 2011 23:23:00 GMT

Neighborhoods of trust on the web

27 September 2011

For the last fortnight I've been hard at work on a new project that aims to examine trust and security on the web at scale. The basic idea is to use a browser instance to render a URL, and then to extract all persistent state with browser forensic techniques afterwards. This gives you a dump of cookies, cache contents, Flash storage, HTML5 databases, and so on. At the same time, all traffic is routed through a specialised version of mitmproxy, and captured for later analysis. The result is a very detailed snapshot of what viewing a given URL actually does. The next step is to do this "at scale" - this means running many instances of this process in parallel on headless servers, decoupling things using queues, backing it all onto a database, and then spending days and days fine-tuning. I'm happy with my progress so far - my infrastructure is now now scanning all the URLs passing through Hacker News, Reddit, Digg, Delicious and Pinboard in realtime, without breaking a sweat.

I am pretty excited about the possibilities for this project, and I'm exploring plans for the future with like-minded security folk. Get in touch if this interests you, and keep an eye on my blog for more news.

After my pilot run, I had 150 gigs of data covering about 120 thousand URLs. Below is a quick peek at one tiny slice of this data - an appetizer for things to come.

Neighborhoods of trust

This graph shows structures that emerge from the way sites use third-party executable resources. In this context, "executable" means means JavaScript, Flash and HTML, and "third-party" means domains other than the URL's own. The nodes in this graph are the third-party domains, and the edges are associations between them via the URLs I crawled. For example, if a site loaded scripts from both Google Analytics and from Doubleclick, that would create (or reinforce) an edge between the nodes "google-analytics.com" and "doubleclick.com". Using this data, I calculated a co-occurrence coefficient for the third-party sources, and then extracted the resulting neighbourhood structures algorithmically. The neighbourhood information was used to colour and lay out the graph, trying to keep nodes that are closely correlated together. Finally, nodes are scaled based on how many URLs reference them.

The result is a rather stunning graph showing neighborhoods of trust - areas of the Internet bound together based on the third parties allowed to run code in users' browsers. I've spent a few hours playing with this data, and the sheer range of interesting structure is surprising. At one end of the spectrum, you can zoom in to the individual node relationships, and find small clusters of surprising sites that cross-load resources from each other, often because they are owned by the same entity. At the other end, countries, language groups, and broad fields of interest aggregate in huge tribes of kinship.

Here are a few of the larger-scale features from the graph:

	Mainstream The most widely used resources dominate in the neighbourhood extraction algorithm, which causes them to cluster together in their own super-community. The top nodes in this cluster, descending order of occurrence are: google-analytics.com, facebook.com, doubleclick.net, fbcdn.net, quantserve.com, twitter.com, google.com, googlesyndication.com, googleapis.com, scorecardresearch.net, facebook.net, addthis.com. These are also the top nodes overall.
	Japanese The main resources are hatena.ne.jp, microad.jp, mixi.jp, yahoo.co.jp, nakanohito.jp. More surprisingly, also in this cluster are topsy.com, appspot.com and postrank.com. Perhaps these resources are especially commonly used on Japanese sites.
	Russian Top resources are yadro.ru, yandex.ru, rambler.ru, vkontakte.ru, openstat.net, userapi.com, shinystat.net, and dt00.net
	Porn And here we have a portion of the web dedicated to pron. The top resources are awempire.com, clickbank.net, picadmedia.com, getresponse.com, adultfriendfinder.com, adultadword.com, phcdn.com, juicyads.com, brazzers.com, etology.com, data-ero-advertising.com and viddler.com. A more surprising inclusion in this group is wufoo.com - I wonder if this is an artifact, or whether Wufoo really does have a use in the adult content world.
	Misc Just to show that it's not all clear-cut, here's an example of a neighbourhood I find harder to explain. The top resources are netdna-cdn.com, amgdgt.com, trafficmp.com, ooyala.com, suitesmart.com, demdex.net, adfrontiers.com, lycos.com and break.com. I speculate that this group might be loosely aligned around a number of big CDNs and analysis suites.

The graph in this post was created, analyzed and pre-processed using graph-tool, a great Python library for dealing with large graphs. The visualization and modularity analysis was done using the ever-wonderful Gephi. If these aren't both in your arsenal of analysis tools, you're missing out.

Why the Apple UDID had to die

Fri, 09 Sep 2011 20:22:00 GMT

Why the Apple UDID had to die

09 September 2011

EDIT: A WSJ Digits article is now up, containing a responses from Zynga and Chillingo. Other networks declined to comment.

A UDID is a "Unique Device Identifier" - you can think of it as a serial number burned permanently into every iPhone, iPad and iPod Touch. Any installed app can access the UDID without requiring the user's knowledge or consent. We know that UDIDs are very widely used - in a sample of 94 apps I tested, 74% silently sent the UDID to one or more servers on the Internet, often without encryption. This means that UDIDs are not secret values - if you use an Apple device regularly, it's certain that your UDID has found its way into scores of databases you're entirely unaware of. Developers often assume UDIDs are anonymous values, and routinely use them to aggregate detailed and sensitive user behavioural information. One example is Flurry, a mobile analytics firm used by 15% of apps I tested, which can monitor application startup, shutdown, scores achieved, and a host of other application-specific events, all linked to the user's UDID. I recently showed that it was possible to use OpenFeint, a large mobile social gaming network, to de-anonymize UDIDs, linking them to usernames, email addresses, GPS locations, and even Facebook profiles.

This post looks at the way UDIDs are used in the broader social gaming ecosystem. The work is based on a simple question: what happens if we swap our UDID for another while communicating with the network? There are a number of ways to do this - in my case I used mitmproxy, an intercepting HTTP/S proxy I developed which lets me re-write the traffic leaving a device on the fly. In most cases this was a simple matter of replacing one string with another, but two networks (Scoreloop and Crystal) prevented UDID substitution using cryptography. Unfortunately, both networks relied on the secrecy of key material distributed in the application binaries to every device. I have verified that it is possible to reverse engineer the application binaries to extract the key material and circumvent the cryptographic protection.

The outcome of this experiment shows that social gaming networks systematically misuse UDIDs, resulting in serious privacy breaches for their users. All the networks I tested allowed UDIDs to be linked to potentially identifying user information, ranging from usernames to email addresses, friends lists and private messages. Furthermore, 5 of the 7 networks allow an attacker to log in as a user using only their UDID, giving the attacker complete control of the user's account. Two networks had further problems that compromised a user's Facebook and Twitter accounts - Crystal lets an attacker take control of a user accounts by leaking API keys, while Scoreloop partially discloses users' friends lists, even if they are private.

	Data leaked	Log in as user	Social Media Accounts
Crystal	Username, friends, Facebook, Twitter, games played, location, email address	Yes	Control of Facebook, Twitter accounts
GameLoft	Username, email address, games played, nationality, friends	Yes	No
Geocade	Username, email address, games played, location	Yes	No
OpenFeint	Username, last played game, online status, friends	Yes	No
Scoreloop	Email address, gender, username, nationality, friends	Yes	Access private Facebook and Twitter friends lists
Plus+	Username	No	No
Zynga	First name, username, friends, in-game messages, mobile number*	Yes*	No

* The starred Zynga findings rely on the fact that other networks can be used to obtain the user's email address using the UDID.

There are two caveats to keep in mind while considering these results. First, the findings are based on the default settings for each social network - some networks may have settings that reduce the amount of information exposed. Second, some of the data leaked is optional - for instance, it's not mandatory for a user to link Facebook or Twitter accounts with any of the networks.

All the affected companies and Apple were notified 5 weeks ago. The Crystal and Scoreloop teams have both repaired the problems that could lead to a follow-on compromise of a user's social network accounts. At the time of writing, it is still possible to log in as a user using only a UDID on five of the vulnerable networks.

The future

A few days after I notified the companies involved, it was revealed that Apple was quietly killing the UDID API. It will still be present in IOS5, but is marked deprecated, and will probably be removed in future. I recommend that developers shift away from using UDIDs now, rather than wait for formal removal of the API.

We can now expect a frenzy of activity as developers look for alternatives. The challenge will be to make sure that the cure isn't as bad as the disease - Apple's recommendation to "create a unique identifier specific to your app" could tempt developers to replicate the UDID mechanism on a smaller scale, flaws and all. Expect more blog posts on this topic soon.

mitmproxy 0.6

Sun, 07 Aug 2011 10:30:00 GMT

mitmproxy 0.6

07 August 2011

I'm happy to announce the release of mitmproxy 0.6, featuring a redesigned scripting API, slew of major new features and a panoply of small bugfixes and improvements.

mitmproxy-0.6.tar.gz

We now have an IRC channel and a mailing list - if you're interested in mitmproxy, come join us!

IRC: #mitmproxy on the OFTC IRC network (irc://irc.oftc.net:6667).

Mailing List: http://groups.google.com/group/mitmproxy

Email:

Changelog

New scripting API that allows much more flexible and fine-grained rewriting of traffic. See the docs for more info.
Support for gzip and deflate content encodings. A new "z" keybinding in mitmproxy to let us quickly encode and decode content, plus automatic decoding for the "pretty" view mode.
An event log, viewable with the "v" shortcut in mitmproxy, and the "-e" commandline argument in both mitmproxy and mitmdump.
Huge performance improvements both in the mitmproxy interface, and loading large numbers of flows from file.
A new "replace" convenience method for all flow objects, that does a universal regex-based string replacement.
Header management has been rewritten to maintain both case and order.
Improved stability for SSL interception.
Default expiry time on generated SSL certs has been dropped to avoid an OpenSSL overflow bug that caused certificates to expire in the distant past on some systems.
A "pretty" view mode for JSON and form submission data.
Expanded documentation and examples.
Many other small improvements and bugfixes.

mitmproxy 0.5

Mon, 27 Jun 2011 17:06:00 GMT

mitmproxy 0.5

27 June 2011

I've just tagged and released mitmproxy 0.5. Everyone should update - this release squelches a few annoying performance killers. You can download it from the project website:

mitmproxy.org

Changelog

An -n option to start the tools without binding to a proxy port.
Allow scripts, hooks, sticky cookies etc. to run on flows loaded from save files.
Regularize command-line options for mitmproxy and mitmdump.
Add an "SSL exception" to mitmproxy's license to remove possible distribution issues.
Add a --cert-wait-time option to make mitmproxy pause after a new SSL certificate is generated. This can pave over small discrepancies in system time between the client and server.
Handle viewing big request and response bodies more elegantly. Only render the first 100k of large documents, and try to avoid running the XML indenter on non-XML data.
BUGFIX: Make the "revert" keyboard shortcut in mitmproxy work after a flow has been replayed.
BUGFIX: Repair a problem that sometimes caused SSL connections to consume 100% of CPU.

UDID media roundup

Fri, 10 Jun 2011 14:38:00 GMT

UDID media roundup

10 June 2011

After a hectic month, I'm finally able to return to the UDID privacy issues I covered in my last few blog posts. I plan to publish some further results soon, but first, a quick roundup of the media coverage of the OpenFeint UDID de-anonymization result.

A post on on the Wall Street Journal tech blog by Jennifer Valentino-DeVries, one of the very few journalists who do good, novel investigative work into issues like UDID privacy.
An interview with La Repubblica, a major Italian daily.
An article in Der Spiegel.
Coverage on CNN online, Wired Gadgetlab and the Huffington Post.
And, last but not least, a nice 30-minute interview with Zach Lanier from the Network Security Podcast. This is your opportunity to get some more details on the OpenFeint issue and find out what a a weird accent I have.

The issue was also mentioned on many, many blogs and smaller publications.

How UDIDs are used: a survey

Thu, 19 May 2011 17:34:00 GMT

How UDIDs are used: a survey

19 May 2011

I recently published some research showing that the OpenFeint social gaming network can be used to link Apple UDIDs to users' real-world identities. To understand why this is a problem, we have to look at the way UDIDs are used in the broader app ecosystem. Once we do this, we see that the vast majority of applications send UDIDs to servers on the Internet, and that UDID-linked user information is aggregated in literally thousands of databases on the net. In this context, UDID de-anonymization is a serious threat to user privacy.

We have one good research paper surveying UDID use - in 2010, Eric Smith looked at the unencrypted portion of app traffic, and found that 68% of tested apps send UDIDs upstream in the clear. I was curious to see what the figures would look like if encrypted (HTTPS) traffic was included, so I decided to do my own survey, using mitmproxy to analyse all traffic from the 94 applications I had installed on my iPhone. Below is a set of graphs highlighting the main facts. I've also published a list of all applications and the domains they contacted here - it makes for interesting reading.

Apps are noisier than you think they are

84% of apps tested contacted one or more domains during use. At the extreme end, iDestroy contacted 14 domains, including 3 different ad networks and OpenFeint.

... and send your UDID to more places than you expect

74% of apps tested sent the device UDID to one or more domains.

... often without encryption

46% of apps that transmitted UDIDs did so in the clear. 54% of apps transmitting UDIDs used encryption for all UDID traffic¹.

A few big UDID aggregators dominate

Three big aggregators of UDID-related data dominate: Apple, Flurry, and OpenFeint. Each one of these companies has the vast majority of UDIDs on file, linked to a rich set of privacy-sensitive information. OpenFeint's ubiquity is one of the reasons why UDID de-anonymization using their API is so serious.

... behind them are a long tail of smaller aggregators

Here is a list of all the remaining domains that had UDIDs transmitted to them - a mixture of ad networks, analytics firms, individual developer sites, and online services.

ads.mp.mydas.mobi	analytics.localytics.com	api.dropbox.com
bayobongo.com	bbc.112.2o7.net	beatwave.collect3.com.au
catalog.lexcycle.com	data.mobclix.com	init.gc.apple.com
msh.amazon.com	notifications.lexcycle.com	promo.limbic.com
soma.smaato.com	www.chimerasw.com	www.phasiclabs.com
www.trainyard.ca	api.twitter.com	ngpipes.ngmoco.com
npr.122.2o7.net	ws.tapjoyads.com

Methodology

For each application, I started a logging instance of mitmdump, like so:

mitmdump -w appname

I then started up the application, interacted with anything that might elicit network traffic, and shut it down. The collected data was analyzed with a simple script, that used the libmproxy API to traverse the traffic dumps and extract the needed information.

The fact that 54% of UDID-using apps would have gone undetected by Smith's study seems to indicate that there should be a much greater difference between our results - Smith found 68% of apps use UDIDs vs my 74%. The discrepancy can be accounted for by the fact that we used different samples - Smith used predominantly applications in Apple's "Top Free" lists, whereas I used both paid and unpaid applications that happened to be on my phone. ↩

De-anonymizing Apple UDIDs with OpenFeint

Wed, 04 May 2011 19:30:00 GMT

De-anonymizing Apple UDIDs with OpenFeint

04 May 2011

Every iPhone, iPad and iPod touch has an associated Unique Device Identifier (UDID). You can think of the UDID as a serial number burned into the device - one that can't be removed or changed¹. This number is exposed to app developers through an API, without requiring the device owner's permission or knowledge.

Few Apple users realise just how widely their UDIDs are used. Research shows that 68% of apps silently send UDIDs to servers on the Internet. This is often accompanied by information on how, when and where the device is used. The most common destination for traffic containing a user's UDID is Apple itself, followed by the Flurry mobile analytics network and OpenFeint, a mobile social gaming company. These companies are uber-aggregators of UDID-linked user information, because so many apps use their APIs. Trailing behind the big three are thousands of individual developer sites, ad servers and smaller analytics firms. Users have no way to stop their device from offering up their UDID, telling who their data is being sent to, or even telling that it's happening at all. This situation has caused wide-spread concern, including coverage in the Wall Street Journal, and two lawsuits aimed at Apple.

The saving grace is that your device UDID is not linked to your real-world identity. If it were possible to de-anonymize UDIDs, the result would be a serious privacy breach. Apple is well aware of this, and explicitly tells developers that they are not permitted to publicly link a UDID to a user account.

I recently published a tool called mitmproxy, a man-in-the-middle proxy that allows one to intercept and monitor SSL-encrypted HTTP traffic. Using mitmproxy to view the encrypted traffic sent by my own iOS devices, I was able to observe protocols and data flows that have clearly received very little external review. A slew of interesting security results followed (keep an eye on this blog), but by far the most alarming was the fact that it was possible to use OpenFeint to completely de-anonymize a large proportion of UDIDs.

De-anonymizing UDIDs with OpenFeint

Linking UDIDs to OpenFeint user accounts

When an OpenFeint-enabled app is first fired up, it submits the device's UDID to OpenFeint's servers, which then return a list of associated accounts:

https://api.openfeint.com/users/for_device.xml?udid=XXX

This is a completely unauthenticated call - you can try it out by cutting and pasting it into your browser, replacing XXX with your own UDID. Here's an example of the response for my UDID, with sensitive information removed:

<?xml version="1.0" encoding="UTF-8"?>
<resources>
  <user>
    <chat_enabled>true</chat_enabled>
    <gamer_score>XXX</gamer_score>
    <id>XXX</id>
    <last_played_game_id>187402</last_played_game_id>
    <last_played_game_name>tiny wings</last_played_game_name>
    <lat>XXX</lat>
    <lng>XXX</lng>
    <online>false</online>
    <profile_picture_source>FbconnectCredential</profile_picture_source>
    <profile_picture_updated_at>XXX</profile_picture_updated_at>
    <profile_picture_url>http://XXX>
    <uploaded_profile_picture_content_type nil="true">
    </uploaded_profile_picture_content_type>
    <uploaded_profile_picture_file_name nil="true">
    </uploaded_profile_picture_file_name>
    <uploaded_profile_picture_file_size nil="true">
    </uploaded_profile_picture_file_size>
    <uploaded_profile_picture_updated_at nil="true">
    </uploaded_profile_picture_updated_at>
    <name>XXX</name>
  </user>
</resources>

Included is my latitude and longitude, the last game I played, my chosen account name, and my Facebook profile picture URL.

Linking UDIDs to GPS co-ordinates

If the user has opted to allow OpenFeint to use their location, latitude and longitude is returned in the profile results. This lets us trivially associate a UDID with GPS co-ordinates.

The location leak was fixed by OpenFeint after my report. Although some portions of the OpenFeint API still returns a user location, it seems that it is no longer served for direct profile requests.

Linking UDIDs to Facebook profiles

If the user registered a Facebook account with OpenFeint, a profile picture URL hosted by the Facebook CDN was returned in the user's profile data. Facebook profile picture URLs include the user's Facebook ID, directly linking it to their Facebook account.

For example, here's Bruce Schneier's Facebook profile picture URL:

http://profile.ak.fbcdn.net/hprofile-ak-snc4/41795_60615378024_8092_n.jpg

The 11-digit number in this URL is his Facebook user ID. We can now view his profile using a URL like this:

http://www.facebook.com/profile.php?id=60615378024

This final step represents a complete de-anonymization of the UDID, directly linking the supposedly anonymous identifier with a user's real-world identity.

The Facebook ID leak was fixed by OpenFeint after my report.

OpenFeint's response

I reported this problem to OpenFeint on 5th of April. I did not hear back from them immediately, but I knew they were working on the problem because their API stopped returning GPS coordinates and Facebook profile picture URLs. On the 12th, I received an email from Jason Citron, OpenFeint's CEO, who wanted to set up a phone conversation with me, him and an OpenFeint legal representative. We spoke on the evening of the 20th of April. I recapped my findings and expressed concern that their API still linked UDIDs to user accounts. They thanked me for the vulnerability report, confirmed that they had tightened their API in response to it, and asked for more time to consider the issue before I released anything. The following morning, it was announced that OpenFeint had been bought by GREE for $104 million.

Last week I received what I assume is OpenFeint's last word on the matter, in the form of an email from Jason Citron: "We will continue to pay attention to the issues you raised and will continue to adjust our practices as necessary." At the time of writing, OpenFeint's API still allows you to associate a UDID with a private user information.

Impact

Testing with a small corpus of UDIDs gathered from my own and friends' devices, I was able to link roughly 30% of UDIDs to GPS co-ordinates, 20% of users to a weak identity (e.g. OpenFeint profile picture, user-chosen account name), and 10% of UDIDs directly to a Facebook profile. I stress that my sample was small and probably unrepresentative - only OpenFeint knows what the real numbers are. None the less, we can make a broad guess at the magnitude of the problem, based on the fact that OpenFeint claims to have 75 million users:

This would mean that about 7.5 million users may have had Facebook accounts linked publicly to their UDIDs until OpenFeint stopped returning profile picture URLs a few weeks ago.
About 22.5 million users may have had GPS co-ordinates linked publicly to their UDIDs until the issue was corrected.
About 15 million users may still have identifying information like profile pictures and user-chosen account names (that can often be used to identify users) exposed.
All 75 million users still have personal details like the last OpenFeint-enabled game they played and whether they are online (i.e. logged in to the OpenFeint network) exposed.

Although the Facebook and GPS de-anonymization issues have been repaired, we have to consider the possibility that these vulnerabilities have already been used to de-anonymize a database of UDIDs.

Conclusion

I want to stress that the problem here is not primarily with OpenFeint. By designing an API to expose UDIDs and encouraging developers to use it, Apple has ensured that there are literally thousands of databases linking UDIDs to sensitive user information on the net. A leak from any one of these - or worse a large-scale de-anonymization like the OpenFeint one - inevitably has serious consequences for user privacy.

I should note that this is not quite accurate. The UDID is actually a computed value - a hash calculated over a set of identifying hardware attributes. In a sense, it only really exists as an API call. ↩

subscount: Counting RSS feed subscribers

Sat, 02 Apr 2011 12:22:00 GMT

subscount: Counting RSS feed subscribers

02 April 2011

A couple of months ago, I wrote a post following one of my blog posts through the the social news grist mill. In it, I bemoaned the fact that social news seems to be displacing more old-fashioned person-to-person connections on the 'net - 33,000 unique visitors to my blog resulted in only 41 new Google Reader subscribers. Google Reader is so completely dominant in this space that ignoring everything else was good enough as a first approximation, but I made a mental note to come up with a more complete figure.

So, yesterday I hacked up a little tool called subscount to help. It parses parses Apache-style logs to make a best guess at feed subscriber numbers, and emits a snippet of JavaScript that can be used to show subscriber numbers on statically rendered sites like my blog.

Estimating feed subscribers from web server logs

Broadly speaking, there are four different groups of feed retrievers we need to deal with:

Well behaved aggregators that report a feed ID and the number of end subscribers in the user agent string. In my case, this is Google Reader, FriendFeed and NetVibes. There's no standard governing this, but there are so few significant players that I just catered manually for all the variations.
Poorly behaved aggregators that report a subscriber number, but no feed ID. An example here is PostRank. Again, there are a small number of these, so subscount handles them with a hand-coded set of rules.
Individual subscribers using tools like Akregator and NetNewsWire. In this case, we distinguish between subscribers by IP address, which should be good enough as long as we keep the analysis time window to a day or so.
A myriad of automated feed consumers. These are mostly poorly behaved, and rarely identify themselves properly. Weeding them out would be nearly impossible, so we treat them just like individual subscribers.

When subscount traverses a log file, it calculates a unique identifier and a number of subscribers for each retriever of the feed. For individual subscribers, the ID is the IP address, and the number of subscribers is 1. For aggregators, we use the reported feed ID, and the reported number of subscribers. We use the unique ID to make sure we don't count anyone more than once, and simply tot up the numbers at the end.

Needless to say, the figure we come up with is just an estimate - but I think it's probably a reasonable one. As expected, the figures show that Google Reader alone is responsible for 68% of my subscribers.

Deploying subscount

I thought it would be neat to report the number of feed subscribers I have next to the feed icon on my blog, so I extended subscount to help. The -j flag to subscount takes a DOM element ID, like so:

./subscount -p "/rss.xml" -j subscriber_div /var/log/mylog

And then prints a snippet that modifies the specified tag with the subscriber number, like this:

function _subs(){
    var subsdiv = document.getElementById("subscriber_div");
    if (subsdiv)
        subsdiv.innerHTML = ("947");
};
window.onload = _subs;

I run subscount from cron just after log rotation every night (it can read gzipped log files directly), and pipe the output to a file in my blog's web root. I then simply source this file in a script tag, and voila! - dynamically updated subscriber numbers for my statically rendered site. You can see the results in my sidebar.

The code

As usual, the code is available on GitHub.

mitmproxy: Breaking Apple's Game Center with replay

Thu, 31 Mar 2011 18:23:00 GMT

mitmproxy: Breaking Apple's Game Center with replay

31 March 2011

This is the second in the series of tutorials I'm writing for mitmproxy. You can find the first one - a 30 second tutorial on client replay - here. There will be more to come in the next few days.

The setup

In this tutorial, I'm going to show you how simple it is to creatively interfere with Apple Game Center traffic using mitmproxy. To set things up, I registered my mitmproxy CA certificate with my iPhone - there's a step by step set of instructions for doing this in the mitmproxy docs. I then started mitmproxy on my desktop, and configured the iPhone to use it as a proxy.

Taking a look at the Game Center traffic

Lets take a first look at the Game Center traffic. The game I'll use in this tutorial is Super Mega Worm - a great little retro-apocalyptic sidescroller for the iPhone:

After finishing a game (take your time), watch the traffic flowing through mitmproxy:

We see a bunch of things we might expect - initialisation, the retrieval of leaderboards and so forth. Then, right at the end, there's a POST to this tantalising URL:

https://service.gc.apple.com/WebObjects/GKGameStatsService.woa/wa/submitScore

The contents of the submission are particularly interesting:

<plist version="1.0">
<dict>
    <key>category</key>
    <string>SMW_Adv_USA1</string>
    <key>score-value</key>
    <integer>55</integer>
    <key>timestamp</key>
    <integer>1301553284461</integer>
</dict>
</plist>

This is a property list, containing an identifier for the game, a score (55, in this case), and a timestamp. Looks pretty simple to mess with.

Modifying and replaying the score submission

Lets edit the score submission. First, select it in mitmproxy, then press enter to view it. Make sure you're viewing the request, not the response - you can use tab to flick between the two. Now press e for edit. You'll be prompted for the part of the request you want to change - press b for body. Your preferred editor (taken from the EDITOR environment variable) will now fire up. Lets bump the score up to something a bit more ambitious:

<plist version="1.0">
<dict>
    <key>category</key>
    <string>SMW_Adv_USA1</string>
    <key>score-value</key>
    <integer>2200272667</integer>
    <key>timestamp</key>
    <integer>1301553284461</integer>
</dict>
</plist>

Save the file and exit your editor.

The final step is to replay this modified request. Simply press r for replay.

The glorious result and some intrigue

And that's it - according to the records, I am the greatest Super Mega Worm player of all time.

Curiously, the top competitors' scores are all the same: 2,147,483,647. If you think that number seems familiar, you're right: it's 2^31-1, the maximum value you can fit into a signed 32-bit int. Now let me tell you another peculiar thing about Super Mega Worm - at the end of every game, it submits your highest previous score to the Game Center, not your current score. This means that it stores your highscore somewhere, and I'm guessing that it reads that stored score back into a signed integer. So, if you were to cheat by the relatively pedestrian means of modifying the saved score on your jailbroken phone, then 2^31-1 might well be the maximum score you could get. Then again, if the game itself stores its score in a signed 32-bit int, you could get the same score through perfect play, effectively beating the game. So, which is it in this case? I'll leave that for you to decide.

mitmproxy: A 30-second client playback example

Thu, 31 Mar 2011 09:58:00 GMT

mitmproxy: A 30-second client playback example

31 March 2011

Yesterday I published version 0.4 of mitmproxy - an intercepting proxy for HTTP/S traffic. The tool already has pretty complete documentation, but I've decided to write a series of less formal tutorials to showcase its abilities. Below is the first, and simplest, of these - keep an eye on the blog for more in the coming days.

A 30-second client playback example

My local cafe is serviced by a rickety and unreliable wireless network, generously sponsored with ratepayers' money by our city council. After connecting, you are redirected to an SSL-protected page that prompts you for a username and password. Once you've entered your details, you are free to enjoy the intermittent dropouts, treacle-like speeds and incorrectly configured transparent proxy.

I tend to automate this kind of thing at the first opportunity, on the theory that time spent now will be more than made up in the long run. In this case, I might use Firebug to ferret out the form post parameters and target URL, then fire up an editor to write a little script using Python's urllib to simulate a submission. That's a lot of futzing about. With mitmproxy we can do the job in literally 30 seconds, without having to worry about any of the details. Here's how.

1. Run mitmdump to record our HTTP conversation to a file.

> mitmdump -w wireless-login

2. Point your browser at the mitmdump instance.

I use a tiny Firefox addon called Toggle Proxy to switch quickly to and from mitmproxy. I'm assuming you've already configured your browser with mitmproxy's SSL certificate authority.

3. Log in as usual.

And that's it! You now have a serialized version of the login process in the file wireless-login, and you can replay it at any time like this:

> mitmdump -c wireless-login

Embellishments

We're really done at this point, but there are a couple of embellishments we could make if we wanted. I use wicd to automatically join wireless networks I frequent, and it lets me specify a command to run after connecting. I used the client replay command above and voila! - totally hands-free wireless network startup.

We might also want to prune requests that download CSS, JS, images and so forth. These add only a few moments to the time it takes to replay, but they're not really needed and I somehow feel compelled trim them anyway. So, we fire up the mitmproxy console tool on our serialized conversation, like so:

> mitmproxy wireless-login

We can now go through and manually delete (using the d keyboard shortcut) everything we want to trim. When we're done, we use S to save the conversation back to the file.

mitmproxy 0.4 has been released

Wed, 30 Mar 2011 14:44:00 GMT

mitmproxy 0.4 has been released

30 March 2011

I've just tagged and released mitmproxy 0.4. You can download it from the new project website:

mitmproxy.org

This is a huge update, with dozens of new features, and improvements to almost every aspect of the project. A few highlights are:

Complete serialization of HTTP/S conversations
On-the-fly generation of SSL interception certificates
Ability to replay both the client and the server side of HTTP/S conversations
mitmdump has grown up to be a powerful tcpdump-like commandline tool for HTTP/S
Scripting hooks for programmatic modification of traffic using Python
Many, many user interface improvements, bug fixes, and minor features
Better documentation.

Special thanks go to Henrik Nordstr�m for many great contributions to this release. I'd love more contributors to join the project - if you feel like hacking on mitmproxy, take a look at the todo file at the top of the tree for ideas.

Over the next week I will write a series of tutorials to showcase mitmproxy's abilities, ranging from simple to quite complex. Keep an eye on the blog for these - they will be published here first, before making their way into the official documentation.

A self-portrait, drawn on an iPad

Tue, 29 Mar 2011 17:27:00 GMT

A self-portrait, drawn on an iPad

29 March 2011

I've just returned from a trip I made to my parents' home to see my grandmother. She's 92, and had braved a long international journey to come to see her great-grandchildren for the first, and probably the last, time. My grandmother is a truly remarkable woman who led a truly remarkable life - she speaks five languages, lived on every continent bar Antarctica, made dozens of trips to Egypt as an amateur Egyptologist, and once mounted a canoe expedition up the Amazon river. And she had talent to match her sense of adventure - she was a great artist, winning many international prizes for her wood carvings. Most important to me, though, was the fact that she was the eccentric grandmother every child deserves to have. She'd seen enough and knew enough not to care what anyone else thought, living life absolutely (and sometimes infuriatingly) by her own rules. My fondest memories of childhood are of visits to her, filled with books, brilliant schemes, and the smell of sawdust in her studio.

The 8 years since I last saw her have been cruel. Physically, she seems almost unchanged, but her memory is fading fast. She loved playing with my baby son, but would often ask me who he was. I would re-explain that he was her great-grandson, and she would be delighted all over again. She was overwhelmed by the helter-skelter of family conversations and small children, and clearly yearned to be back home, where she spends her days in quiet reminiscence among the curios and collected oddities of 70 years of travel. Her days as an artist have long been over - she hasn't been able to hold a pencil for a decade, much less a wood-chisel or a jigsaw.

On a whim, a few days into the visit, I started up a sketch app on my iPad and and handed it to her. She was doubtful at first, but quickly became deeply engrossed. She sat for hours, hunched over, using the side of her index finger to draw. And draw. Saving a sketch and starting a new one was too complicated, so she used the eraser to clear space instead, drawing each image over the top of the previous one. At the end of the day she handed the iPad back to me with the final sketch of a beautiful, sad young woman still on the screen.

A self-portrait, she said with a smile.

Truly, the tragedy of life is not that we grow old, it's that we stay young.

Indenting XML-ish markup: a code snippet

Sun, 06 Feb 2011 16:45:00 GMT

Indenting XML-ish markup: a code snippet

06 February 2011

I've been hacking on mitmproxy recently, gearing up to a new release in the next week. One of the features I needed was to pretty-print XML-ish markup (HTML, SOAP, etc.) to make it easier to quickly scan through traffic not formatted for human eyes. I needed this function to cope robustly with incomplete or malformed data, which ruled out proper XML parsers like ElementTree. I also needed it to be fast on large-ish files, which ruled out BeautifulSoup. On the upside, I didn't need it to be perfect - and as long as it didn't lose or corrupt data, getting the indentation mostly right would be good enough.

Today I sat down and hacked up my own solution. This turns out to be just 40 lines of code, somewhat gnarled and ugly after being fine-tuned against a few dozen real-world data samples:

import re, textwrap

TAG = r"""
        <\s*
        (?!\s[!"])
        (?P<close>\s\/)?
        (?P<name>\w+)
        (
            [^'"\t >]+ |
            "[^\"]"['\"] |
            '[^']'['\"] | 
            \s+
        )*
        (?P<selfcont>\s\/\s)?
        \s>
      """
UNI = set(["br", "hr", "img", "input", "area", "link"])
INDENT = " "4
def pretty_xmlish(s):
    """
        A robust pretty-printer for XML-ish data. 
        Returns a list of lines.
    """
    data, offset, indent, prev = [], 0, 0, None
    for i in re.finditer(TAG, s, re.VERBOSE|re.MULTILINE):
        start, end = i.span()
        name = i.group("name")
        if start > offset:
            txt = []
            for x in textwrap.dedent(s[offset:start]).split("\n"):
                if x.strip():
                    txt.append(indentINDENT + x)
            data.extend(txt)
        if i.group("close") and not (name in UNI and name==prev):
            indent = max(indent - 1, 0)
        data.append(indentINDENT + i.group().strip())
        offset = end
        if not any([i.group("close"), i.group("selfcont"), name in UNI]):
            indent += 1
        prev = name
    trail = s[offset:]
    if trail.strip():
        data.append(trail)
    return data

(snippet.py)

Little snippets of code like this are too trivial to spin out into an independent library, but I'd like to put them up somewhere public where other folks could use them. Right now I don't know of a good place to do this. There's snipplr, but they went with the wrong kind of "social" when they made a social snippet repository, ending up with a social-news-like site focused on upvotes and popularity. This just seems to be a total mismatch to the problem space. What I really want is some combination of asymmetric follow, change tracking, tags, powerful search and good curation tools - more like delicious.com (may it rest in peace) than Reddit. Github's gists are structurally much closer to this, but aren't quite there on curation and search. I also suspect that the fact that gists are full-fledged Git repos is overkill for snippet tracking, much as I love Git (and Github) for larger projects.

I'll no doubt be fine-tuning this function in the days to come - if you're interested, you'll have to keep an eye on this file here, which is less than ideal. If anyone knows of a better snippet sharer, though, let me know...

Social news eats a blog post

Mon, 24 Jan 2011 20:43:00 GMT

Social news eats a blog post

24 January 2011

This is the second post in which I try to add some data to my nagging doubts about the technical news ecosystem. In my previous post, I showed off a visualisation of how the proggit front page changes over time. In this post, I take a look at the flip-side of the coin - what happens to a specific post as it passes through the short, fickle social news cycle? To do this, I'll take a deep dive into my own server logs, looking at a recent post of mine that appeared briefly on both Hacker News and proggit. I'd guess that nearly all posts follow more or less the same trajectory as they are extruded through the social news mill, so this should be interesting to more people than just me. At the risk of making things a bit dry and descriptive, I'm saving speculation and interpretation for a future post.

The scene is set at about 10pm New Zealand time, when I put the finishing touches to my blog post, and fire off an rsync up to my server. I quickly double-check that the blog and the RSS feed have updated OK, tweet a link to the post, and go to bed. While I sleep, the post creeps onto both Hacker News and proggit, ultimately getting 41000 hits over the next 5 days or so. The graphs below show only the first 50 hours of the post's lifetime - everything after that is just a long, slow d�nouement as it dwindles into obscurity.

Our real-time robot overlords

The action starts almost as soon as I click the "tweet" button. Within seconds, the post is retrieved by Twitterbot. One second later, Googlebot appears, and almost simultaneously I get hit by Jaxified, Njuice, LinkedIn and PostRank. In all, 10 bots read my blog post within the first minute, handily beating the first human, who slouches lethargically into view at a tardy 90 seconds.

Below is a list of the bots that retrieved my post before the first submission to a social news site. These are the realtime robots, presumably hoovering up the Twitter firehose and indexing all the links they find. The cast of characters is a mixture of the expected big fish, stealth startups, and skunkworks projects at well-known companies. Bot identity was gleaned from HTTP user-agent headers when they were provided, or by checking the ownership of the responsible IP through reverse DNS resolution and whois lookups when they weren't. Most of the real-time bots were well behaved, identifying themselves clearly with a URL in the user-agent string.

minutes after publication	bot
1	Twitter
	Google
	Jaxified
	NJuice
	LinkedIn
	PostRank
	Unidentified bot from a Microsoft-owned IP
	Yahoo! Slurp
	Unidentified bot from a BBC RAD labs IP.
	OneRiot
2	FriendFeed
	Kosmix
	Topsy Butterfly
	Unidentified bot from marban.com subdomain. (PoPUrls?)
3	metauri.com
3	msnbot
6	Summify
6	Bot identifying itself just as "NING", can't confirm that it's the Ning.
9	tineye
26	spinn3r.com
27	backtype.com
47	facebookexternalhit

Enter the heavyweights: Hacker News and Reddit

48 minutes after the post was published, the first hit from a social news site appears: hello Hacker News. The post quickly makes it onto the front page, and HN traffic peaks at 399 hits per hour in the second hour after publication. All told, the post got 2337 hits with a HN referrer header.

news.ycombinator.com

Two hours and three minutes after publication, the real monster of social news arrives: the first hit from Reddit appears. The Reddit traffic peaks in the sixth hour after publication at 3025 hits per hour, and delivers a total of 23807 hits in the 51 hours after publication.

reddit.com/r/programming

The long tail

Reddit accounted for the vast majority of the post's traffic, dwarfing all other sources combined. In all, I received only 2300 hits with specified referrer headers that weren't Reddit or HN. Here are all the referrers that were responsible for more than 10 hits to the post:

hits	site
456	popurls.com
359	Google Reader
282	Twitter
196	jimmyr.com
183	delicious
153	pop.is
139	Google Search
82	wired.com
56	Facebook
36	longurl.com
36	glozer.net/trendy
30	oursignal.com
28	hackurls.com
24	Yahoo Pipes
18	www.netvibes.com
15	dzone.com
11	www.freshnews.org

It's interesting to see that I got nearly 200 hits from delicous.com. By contrast, pinboard.in - which seems to be delicous.com's anointed successor - sent me only two hits. Then again, my post was published in late November 2010, about a month before Yahoo spectacularly hobbled their bookmarking property. I wonder what those figures would look like today.

The thin end of the long tail are the 200 hits from 94 sites that were responsible for 10 or fewer hits each. We can break this motley crew up into a few different classes:

Sites that provide some sort of social news analysis, piggy-backing off HN, Reddit and delicious.com. For example, popacular.com, seesmic.com, hotgrog.com.
URL shorteners like j.mp and unshorteners like untiny.me
Social media-ish services like FriendFeed, StumbleUpon, pinboard.in
Tiny personal blogs.
And, surprisingly - a number of sites that just provide an alternative interface or URL for Hacker News: hackerne.ws, ihackernews.com, hacker-newspaper.gilesb.com, www.icombinator.net.

Robot scavengers of the social news ecosphere

Let's take a look at overall bot traffic, separating out our silicone friends by looking for non-human and non-standard user-agent headers. The moment the post hits the HN front page bot traffic spikes, and this spike continues as the post is submitted to Reddit and starts its climb up the proggit front page.

all bots

Enter the robot scavengers of the social news ecosphere - a set of second-tier aggregators that monitor social news and Twitter for hot stories. Here's a sample of bot visitors, taken more or less at random from the logs:

inagist.com	www.netvibes.com	chattertrap.com	twingly.com
coder.io	newsmagpie.com	worio.com	www.myvbo.com
www.zemanta.com	embed.ly	brandwatch.net	www.flipboard.com
paper.li	rivva.de	attribyte.com	diffbot.com
yoono.com	hatena.net.jp	hourlypress.com	longurl.org
untiny.me	goo.ne.jp	www.baidu.com	sharethis.com
ideashower.com	pannous.info	wikiwix.com	pipes.yahoo.com
mustexist.com	pics.fefoo.com	cyber.law.harvard.edu	seatgeek.com
metadatalabs.com	moreover.com	thinglabs.com	stufftotweet.com
chilitweets.com	bkluster.hut.edu.vn	wikio.com	Yahoo Pipes
zite.com	zelist.ro	buzzzy.com	intravnews.com

At this point, I'd like to bitch a bit about how astonishingly badly behaved some of the automated systems skulking around today's web are. The vast, vast majority don't provide any clue about the responsible entity in the user-agent string. The list above consists of responsible bots that do identify themselves, and less responsible ones that I could identify through reverse domain resolution. Most of the irresponsible bots come from Amazon Web Services, which seems to be a right wretched hive of scum and villainy. The worst performers here boggle the mind - about a dozen hosts from AWS retrieved the blog post more than 200 times a day, all using full GET requests, without an If-Modified-Since header, and with no identification. The arch-villain hit the post 600 times in its first 24 hours - that's about once every 2.5 minutes.

Referrer-less viewers and stealthy bots

I was surprised to see that almost 20% of requests not identified as bot requests had no specified referrer, a much greater percentage than I would have anticipated. Here's a graph showing the number of referrer-less requests per hour:

requests without a referrer

It looks like the double-peak in this graph coincides with the traffic peaks from HN and Reddit. This suggests that the majority of these hits do in fact come (perhaps indirectly) from HN and Reddit users. One possibility is that a chunk of this referrer-less traffic comes from non-browser Twitter clients.

A fraction of the referrer-less traffic also comes from stealthy bots sending user-agent strings that match those of desktop browsers. About 5% of these requests, for example, come from the Amazon EC2 cloud, so are unlikely to be real browsers. One Internet darling that does this is Instapaper, which seems to use the requesting client's user-agent string rather than frankly confessing itself to be a bot. It also appears to re-request an article in full for each user, rather than simply checking if there's been a change and using a cached copy. On the upside, this means that I know that 131 readers used Instapaper to view my post.

Aftermath

After the post drifts off the proggit and HN front pages, traffic dies down. There's a dwindling tail of stragglers that bothered to flip through to the second or third page of top stories, and a tiny dribble of users who discovered the link through other sources. A month later, the post gets about 60 hits per day, of which more than a third are from bots. Non-bot traffic is still dominated by Reddit, presumably from people searching or idly flicking through Reddit's history.

So, in the end, after my once-thrumming server quiets down, what has the lasting effect been on my own social graph? I had a small surge of Twitter follows, going from 230 to 245 followers. There was a minor blip of subscribers to my RSS feed, with Google Reader reporting subscriptions going from about 510 to 551. Out of 33,000 unique visitors 56 decided to cultivate a more permanent relationship of some sort to my blog. That's 1 in 600. If you remember only one figure from this post, this should be it.

A journey through the bowels of proggit

Wed, 12 Jan 2011 20:51:00 GMT

A journey through the bowels of proggit

12 January 2011

I've had a nagging sense of dissatisfaction with my information diet lately, and it's becoming clear that over-reliance on social news sites like Reddit and Hacker News (much as I love them) lies at the heart of my discontent. For the past few months, I've been gathering data to help me come up with a coherent explanation for my malaise. I'm still working on it, so this post will have no conclusions, only repulsive metaphors and pretty pictures.

For a week or so in November I logged the slow, peristaltic progress of stories through the bowels of proggit, watching them get nudged this way and that by the malodorous, hot gas of public opinion before finally being shunted on to the colon of the second page of results. In other words, I sampled the top 25 stories every 5 minutes through the RSS feed. One of the things I was interested in was how submission rankings changed over time, so I visualised the dataset using the same technique I came up with to visualise sorting algorithms. The image above shows 4 hours of proggit, with each submission represented by a line. The lines are coloured based on the average rank the story achieves over its lifetime in the top 25, ranging between upvote orange for top stories, and downvote blue for bottom stories.

Here's a bigger sample - 72 hours of data embedded in a widget to let you zoom and pan around. The busy cut-and-thrust of life on reddit is all here. The meteoric rise, inevitably followed by long, slow decay. The sudden, mysterious, mid-flight disappearances. The jostling and writhing among the bottom submissions that never quite manage to make it into the big leagues. Heady stuff. Click to view:

Perhaps I'll do an expanded version that lets you view submission titles, times and so forth later on.

Cyclesort - a curious little sorting algorithm

Mon, 22 Nov 2010 21:36:00 GMT

Cyclesort - a curious little sorting algorithm

22 November 2010

One of the nice things about building sortvis.org and writing the posts that led up to it is that people email me with pointers to esoteric algorithms I've never heard of. Today's post is dedicated to one of these - a curious little sorting algorithm called cyclesort. It was described in 1990 in a 3-page paper by B.K. Haddon, and has become a firm favourite of mine.

Cyclesort has some nice properties - for certain restricted types of data it can do a stable, in-place sort in linear time, while guaranteeing that each element will be moved at most once. But what I really like about this algorithm is how naturally it arises from a simple theorem on symmetric groups. Bear with me while I work up to the algorithm through a couple of basic concepts.

Cycles

Lets start with the definition of a cycle. A cycle is a subset of elements from a permutation that have been rotated from their original position. So, say we have an ordered set [0, 1, 2, 3, 4], and a cycle [0, 3, 1]. The cycle defines a rotation where element 0 moves to position 3, 3 to 1 and 1 to 0. Visually, it looks like this:

We can apply a cycle to an ordered set to obtain a permutation, and we can then reverse that cycle to re-obtain the original set. Here's a Python function that applies a cycle to a list in-place:

def apply_cycle(lst, c):
    # Extract the cycle's values
    vals = [lst[i] for i in c]
    # Rotate them circularly by one position
    vals = [vals[-1]] + vals[:-1]
    # Re-insert them into the list
    for i, offset in enumerate(c):
        lst[offset] = vals[i]

Here's an interactive session showing the function in action:

>>> lst = [0, 1, 2, 3, 4]
>>> c = [0, 3, 1]
>>> apply_cycle(lst, c)
>>> lst
[1, 3, 2, 0, 4]
>> c.reverse()
>> apply_cycle(lst, c)
>> lst
[0, 1, 2, 3, 4]

Permutations

Now, it's a fascinating fact that any permutation can be decomposed into a unique set of disjoint cycles. We can think of this as analogous to the factorization of a number - every permutation is the product a unique set of component cycles in the same way every number is the product of a unique set of prime factors. Taking this as a given, how could we calculate the cycles that make up a permutation? One obvious way to proceed is to pick a starting point, and simply "follow" the cycle in reverse until we get back to where we started. We know from the result above that the element is guaranteed to be part of a cycle, so we must eventually reach our starting point again. When we do, hey presto, we have a complete cycle. If we keep track of the elements that are already part of a known cycle, we can skip to the next unknown element and repeat the process. Once we reach the end of the list we're done.

This scheme can only work if we know where in the ordered sequence any given element belongs, because this is the way we find the "previous hop" in a cycle. In the examples above, we worked with lists that consist of a contiguous range of numbers 0..n, which gives us a short-cut: the element's value is its offset in the ordered list. In the code below I've factored this out into a function key, which takes an element value, and returns its correct offset - in this case key is simply the identity function.

Here's a Python function that finds all cycles in permutations of numbers ranging from 0..n:

def key(element):
    return element

def find_cycles(l):
    seen = set()
    cycles = []
    for i in range(len(l)):
        if i != key(l[i]) and not i in seen:
            cycle = []
            n = i
            while 1: 
                cycle.append(n)
                n = key(l[n])
                if n == i:
                    break
            seen = seen.union(set(cycle))
            cycles.append(list(reversed(cycle)))
    return cycles

Running it on our example permutation produces the cycle we used to produce it:

>>> find_cycles([1, 3, 2, 0, 4])
>>> [[3, 1, 0]]

Here's find_cycles run on a longer, randomly shuffled list:

l = [0, 5, 6, 8, 7, 4, 9, 1, 3, 2]
>>> find_cycles(l)
>>> [[7, 4, 5, 1], [9, 6, 2], [8, 3]]

And here's a handsomely colourful graphical version of the output above:

A sorting algorithm emerges

Let's take a closer look at the find_cycles function above. We keep track of elements that are already part of a cycle in the seen set, so that we can skip them as we proceed through the list. The seen set can be as large as the list itself, so we've doubled the memory requirement for the algorithm. If we're allowed to destroy the input list, we can avoid explicitly tracking seen elements by relocating elements to their correct position as we work our way around each cycle. All the cycles are disjoint and we traverse each cycle only once, so doing this won't affect the function's output. We can then tell that we need to skip an element we've already seen by checking whether it's in the correct sorted position. Here's the result:

def key(element):
    return element

def find_cycles2(l):
    cycles = []
    for i in range(len(l)):
        if i != key(l[i]):
            cycle = []
            n = i
            while 1: 
                cycle.append(n)
                tmp = l[n]
                if n != i:
                    l[n] = last_value
                last_value = tmp
                n = key(last_value)
                if n == i:
                    l[n] = last_value
                    break
            cycles.append(list(reversed(cycle)))
    return cycles

But... at the end of this process, the original list is sorted! Tada: cyclesort pops out of the shrubbery almost as a side-effect of efficiently finding all cycles. If we're only interested in sorting, we can strip the code that saves the cycles, which leaves us with a nice, pared-back sorting algorithm:

def key(element):
    return element

def cyclesort_simple(l):
    for i in range(len(l)):
        if i != key(l[i]):
            n = i
            while 1: 
                tmp = l[n]
                if n != i:
                    l[n] = last_value
                last_value = tmp
                n = key(last_value)
                if n == i:
                    l[n] = last_value
                    break

The cyclesort_simple algorithm only works on permutations of sets of numbers ranging from 0 to n. There are other fast ways to sort data of this restricted kind, but all the methods I know of require additional memory proportional to n. Cyclesort can do it without any extra storage at all, which is a neat trick.

Visualising cyclesort

At this point, we have enough information to visualise the algorithm, so let's take a look at the beastie we're working with. I've had to make some little adjustments to the usual sortvis.org visualisation process to cope with cyclesort. In the algorithm above, the first element is duplicated into the second position of each cycle, and that duplicate remains in play until it's over-written by the last element of the cycle. I changed the algorithm slightly to write a null placeholder at the start of the cycle to avoid duplicates, and taught the sortvis.org visualiser to deal with "empty" slots. The resulting weave visualisation looks like this:

This is quite satisfying - you can tell where each cycle begins and ends by the gaps, which span each cycle exactly. It's immediately clear that the permutation above, for instance, contained five cycles. Within each cycle, you can follow along as each element replaces the next, until we finally close the gap by placing the last element in the first slot.

The dense visualisation is less informative because the gaps are too small to see at a single-pixel width, and the algorithm doesn't have much other large-scale structure. It still looks neat, though:

Generalising cyclesort

Cyclesort works whenever we can write an implementation of the key function, so there's quite a bit of scope for clever exploitation of structured data. The Haddon paper presents a solution for one common case: permutations whose elements come from a relatively small set, where the number of occurances of each element is known. The insight is that the key function can have persistent state, letting us calculate the positions of elements incrementally as we work through the list.

We begin by adding an extra argument to our sort function: a list (element, count) tuples telling us a) the order of the keys, and b) the frequency with which each key occurs.

[("a", 10), ("b", 33), ("c", 18), ("d", 41)]

Now, in the sorted list, we know that there will be a contiguous blog of 10 "a"s, followed by a contiguous block of 33 "b"s, and so forth. We can use this information to calculate the offset of each contiguous block up front:

def offsets(keys):
    d = {}
    offset = 0
    for key, occurences in keys:
        d[key] = offset
        offset += occurences
    return d

The key function uses this offset dictionary to look up the current index for any element. Each time we insert an element into position, we increment the relevant offset entry - next time we get to an element of the same type, we will place it in the next position in the contiguous block. We also make a small modification to the algorithm to cater for the progressive position increment process: we start a cycle only when the element is equal to or above the position where it ought to be. Here's a Python implementation:

def offsets(keys):
    d = {}
    offset = 0
    for key, occurences in keys:
        d[key] = offset
        offset += occurences
    return d


def key(o, element):
    return o[element]


def cyclesort_general(l, keys):
    o = offsets(keys)
    for i in range(len(l)):
        if i >= key(o, l[i]):
            n = i
            while 1: 
                tmp = l[n]
                if n != i:
                    l[n] = last_value
                last_value = tmp
                n = key(o, last_value)
                o[last_value] += 1
                if n == i:
                    l[n] = last_value
                    break

This algorithm runs in O(n + m), where n is the number of elements and m is the number of distinct element values. In practice m is usually small, so this is often tantamount to being O(n).

The code

As usual, the code for these visualisations have been incorporated into the sortvis project. I've also added the visualisations above to the sortvis.org website.

What Stuxnet means

Mon, 15 Nov 2010 15:54:00 GMT

What Stuxnet means

15 November 2010

The last bit of evidence is now in - it appears that the mysterious Stuxnet worm was indeed aimed at Iran's nuclear capability. This means that we now know for sure that Stuxnet was an event of great significance - the first example of a type of sophisticated interstate warfare that we can expect to see a lot more of in future. It neatly ties together a number of trends that we've been talking about to clients at Nullcube for years:

The worm as a targeted delivery platform. Stuxnet spread indiscriminately, waiting until it infected its intended target before springing into action. This is a marvelous delivery platform with excellent deniability. When executed with flair - using multiple previously unknown vulnerabilities, spreading through both physical media and networks - it can be incredibly hard to defend against. Look for a Stuxnet-like worm that exfiltrates data from targeted systems next.
Internet security is a national concern. There's a tendency to view the Internet as an internationally homogeneous network. Stuxnet makes it (even more) clear that the Internet is a domain for contest between nation states, and that national differences in security readiness and technology populations matter. Look for more direct government involvement in tracking and improving the security of local networks. I suspect we'll also see the rise of national perimeter defenses in some countries in the next few years.
Embedded systems are a target. Embedded systems are everywhere, are often ignored when security is considered, and are opaque, difficult to inspect, and difficult to monitor. This is a malware nirvana. Whether they are directly or indirectly connected to a network, embedded systems are a target. My prediction: soon, we'll see a Stuxnet-like worm that spreads directly from embedded system to embedded system, most likely affecting DSL modems. In fact, we've already seen a clumsy precursor of this in Psyb0t, discovered at the beginning of 2009.

There's a lot about this incident that we will most likely never know. We're unlikely to find out who's behind Stuxnet (although Israel and the US seem to be the only real possibilities). We're unlikely to find out if Stuxnet ever repayed the immense technological capital its creators invested. But we do know that it's a sign of things to come.

Tau: is it worth switching?

Mon, 04 Oct 2010 18:54:00 GMT

Tau: is it worth switching?

04 October 2010

The mailing list for my local LUG recently had a small flurry of posts on The Tau Manifesto, a proposal to replace of the constant π with τ, equal to 2π. Pro- and anti- camps quickly emerged, and much beer will likely be spilt over the issue at our next meeting.

Disregarding for the moment any conceptual elegance or expanatory power that Tau might have, I was interested to know if the move would really reduce redundancy in common mathematical expressions. Lets say (rather arbitrarily) that Tau simplifies a mathematical expression whenever π is preceded by an even constant - that means that 2π becomes τ, and 4π becomes 2τ, and so forth. I had a vague intuition that the majority of occurances of π in the wild fell into this category, which might indicate that τ is a more natural (or at least parsimonious) constant to use. Was my hunch right? This, I felt, was something I could quantify.

Methodology

I wrote a small script to crawl all the articles linked to from the Wikipedia List of Equations page. For each page, I extracted all mathematical expressions, and checked the LaTeX source of each for occurances of the symbol π. A little bit of light parsing was then done to check if the symbol was directly preceded by an integer constant. Finally, I rendered the LaTeX source back to images to produce the equation tables below.

Of course, anyone of sound judgement will disregard what follows entirely, due to the many obvious shortcomings of this procedure and its underlying assumptions. Readers of my blog, on the other hand, may find the results interesting.

Results

I found a total of 3173 equations, of which 133 contained the symbol π. Of these 133 equations, the distribution of constant factors preceding π looked like this:

I call this a straight win for Tau - the vast majority of expressions using π (119 of 133) are preceded by even integer constants.

Equations

Below are all the expressions that included π, plus the detected constant factor. The headings point to the Wikipedia pages from which the equations were taken.

If nothing else, this list is a nice reminder of the mysterious ubiquity of a constant involving the diameter and circumference of a circle in all aspects of physics and higher math.

Relativistic wave equations

constant	expression
8

Sine-Gordon equation

constant	expression
2
2

Fokker–Planck equation

constant	expression
2
2
2
2

Euler's equation

constant	expression
None

Friedmann equations

constant	expression
8
4
8
8
8
4
8
8
8
8

Vlasov equation

constant	expression
4
4
4

Screened Poisson equation

constant	expression
4
2
2
4
4

Quadratic equation

constant	expression
4
2

Stokes-Einstein relation

constant	expression
6
6
6

Fisher equation

constant	expression
None
None
None
None

Einstein's field equation

constant	expression
8
8
8
8
8
8
8
8
8
8
8
8
4
4
4
8
8

Sackur-Tetrode equation

constant	expression
4

Laplace's equation

constant	expression
None
4
4
4
2
4
4

Cauchy-Riemann equations

constant	expression
2

Cubic equation

constant	expression
2
2
2
4
2

Partial differential equation

constant	expression
2
2
2
2
None
2
None

Lane-Emden equation

constant	expression
4

Heat equation

constant	expression
None
None
None
None
2
4
4
4
4
4
4
4
4
4
4
2
2

Wave equation

constant	expression
4
4
4
2

Primitive equations

constant	expression
None
None

Quintic equation

constant	expression
2
2
2
2
2

Black–Scholes equation

constant	expression
2
2
2

Fredholm integral equation

constant	expression
2

Poisson's equation

constant	expression
2
4
4

Helmholtz Equation

constant	expression
4

Van der Waals equation

constant	expression
4
4
2
4
2

Lorentz equation

constant	expression
4
4
4

Maxwell's equations

constant	expression
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4

iPad: the perfect computing device for children?

Sun, 05 Sep 2010 13:43:00 GMT

iPad: the perfect computing device for children?

05 September 2010

This is my 10 week old son playing with Uzu on the iPad like a tiny, mad concert pianist. Remarkably, this was the first time we saw him intentionally manipulating the external world with his hands. That was a week ago - since then it's like a switch has been flicked somewhere in his fuzzy little cranium, and now his previously aimless flailing has turned into a methodical tactile exploration of everything around him.

Watching this minor domestic miracle, it struck me that the iPad may well be the perfect computing device for small children. The multitouch interface couldn't be more simple and direct, and it works equally well for adult hands and tiny stubby fingers. All the complexity of the desktop-and-windows metaphor has been stripped away - the iPad does only one thing at a time, so when an app is running it becomes the device. The result is the clearest, most unmediated computing experience so far. It's also interesting to consider that what makes the iPad so great for children is also what makes it so great for adults.

As I've written before, I'm more than slightly terrified by just how good Apple's devices are.

Sea lions and lifestyle change

Thu, 02 Sep 2010 12:59:00 GMT

Sea lions and lifestyle change

02 September 2010

About a year and a half ago, after dinner at a favourite local restaurant, and having entered into that zone of philosophical clarity that sets in around the dessert wine, my wife and I had the sudden simultaneous realisation that it was time for a change. For most of our adult lives, we had lived in the suburb of Newtown in Sydney - a hyper-urban jungle densely packed with coffee shops and theatres, inhabited by a thronging mixture of students and bohemians with counterculturally-correct hairdos. It was all beginning to seem a bit tired and same-ish. We needed more time and more space. We needed to get back to the essentials of life.

Four weeks later our furniture was in a shipping container en-route to Dunedin, a small university town near the southern tip of New Zealand. We decided to work together from home, keeping our schedules flexible to make time for walks, reading, cooking, and (more recently) spending time with our son. It was a huge risk - it was quite possible that the isolation would impose a punishing work travel regime on me, or put a crimp in my wife's very specialised career in linguistics. It took enterprise, determination and a no small amount of possibly-foolish optimism, but it's all worked out. Our leap of faith has turned out to be one of the best decisions we've ever made. Dunedin is a breathtakingly beautiful place to live - I still can't quite believe that I can get up from my desk, and within 20 minutes be on a deserted beach littered with lazy sea lions basking in the winter sun.

My advice to you is this: when your life begins to seem a bit stuffy and constricted, when you begin to feel you've lost sight of something more fundamental and get the urge to refactor - just do it. There has never been a better time in history for people who choose to march to a different drum.

To prove what a lucky fellow I am, here are two photos from my walk yesterday morning - click to view in a lightbox.

It's not clear from the picture, but this is a massive New Zealand Sea Lion bull - about 400 kilograms of apparently boneless muscle and blubber.

It's hard to believe that this sleek female is the same species as the dumpy, snub-nosed chap above. New Zealand Sea Lions are the rarest species of sea lion in the world - it's an immense privilege to be able to share a beach with them.

cortesi

Introducing pathod: a pathological HTTP server

01 May 2012

The simplest possible response

Specifying features

Pauses and Disconnects

What's next?

mitmproxy 0.8

09 April 2012

Android interception

Replacement patterns

Improved pretty-printing of request and response contents

Changelog

mitmproxy 0.7

27 February 2012

Changelog

OpenBSD in decline?

26 February 2012

Malware

05 January 2012

If you subscribe to my RSS feed, please visit this article directly. The table below has interactive elements that won't work in most feed readers.

Visualizing entropy in binary files

04 January 2012

Visualizing the OSX ksh binary

The code

A personal link mill

30 December 2011

My personal link mill

Problems - and a product idea?

Visualizing binaries with space-filling curves

23 December 2011

More detail

The code

netograph.com - Realtime privacy snapshots of the social web

08 December 2011

What's next?

Otago Polytechnic Talk

31 October 2011

Neighborhoods of trust on the web

27 September 2011

Neighborhoods of trust

Mainstream

Japanese

Russian

Porn

Misc

Why the Apple UDID had to die

09 September 2011

The future

mitmproxy 0.6

07 August 2011

Changelog

mitmproxy 0.5

27 June 2011

Changelog

UDID media roundup

10 June 2011

How UDIDs are used: a survey

19 May 2011

Apps are noisier than you think they are

... and send your UDID to more places than you expect

... often without encryption

A few big UDID aggregators dominate

... behind them are a long tail of smaller aggregators

Methodology

De-anonymizing Apple UDIDs with OpenFeint

04 May 2011

De-anonymizing UDIDs with OpenFeint

Linking UDIDs to OpenFeint user accounts

Linking UDIDs to GPS co-ordinates

Linking UDIDs to Facebook profiles

OpenFeint's response

Impact

Conclusion

subscount: Counting RSS feed subscribers

02 April 2011

Estimating feed subscribers from web server logs

Deploying subscount

The code

mitmproxy: Breaking Apple's Game Center with replay