cortesi

mitmproxy 0.9.1

Sun, 16 Jun 2013 16:48:00 GMT

mitmproxy 0.9.1

16 June 2013

I'm happy to announce the release of mitmproxy 0.9.1. This is a bugfix release, with no significant changes in behaviour.

As hinted in my previous release note, the project itself is also evolving. As of this release, mitmproxy and its sister projects (pathod and netlib) are housed under a separate organization on Github, rather than my own personal space:

github.com/mitmproxy

I'm also very happy to welcome the first external core developer to the mitmproxy projext: Maximilian Hils. Max is the author of HoneyProxy, a web analysis front-end for mitmproxy. In the next few months, he'll be working on integrating and expanding his work to become mitmproxy's official web interface. Max's efforts will be sponsored by Google under their Summer of Code program, and will be mentored by the HoneyNet Project.

Changelog

Use "correct" case for Content-Type headers added by mitmproxy.
Make UTF environment detection more robust.
Improved MIME-type detection for viewers.
Always read files in binary mode (Windows compatibility fix).
Correct PyOpenSSL dependency declaration.
Some developer documentation.

Skout: a devastating privacy vulnerability

Fri, 31 May 2013 09:08:00 GMT

Skout: a devastating privacy vulnerability

31 May 2013

I've become a bit weary of the process of public vulnerability disclosure - I'm much more likely nowadays to just drop companies an anonymous notice and move on. Every so often, though, I come across an issue so egregious that talking about it publicly seems like an imperative. This is one of them.

First, some background. Skout is a location-based mobile social network. The idea is to allow people to meet others in their area, semi-anonymously, get to know them, and then perhaps line up a meeting in meatspace. As far as I can tell, a huge fraction of the userbase are singles, using Skout as an ad-hoc dating app. Skout's scale is significant - they don't release exact user numbers, but I've seen claims of more than 10 million users, and a growth rate of a million users per month.

In 2012, Skout went through a major PR catastrophe, when its service was linked to no fewer than 3 separate rapes of children by adult men posing as teenagers. Skout immediately suspended the service for teenagers and went through a security re-vamp. A month later, teens were allowed back, with Skout making much of its new safety system, "advanced, proprietary algorithms" to weed out stalkers, and its long-term commitment to community safety.

Given this background, the problem I found is simple but devastating. The Skout mobile application talks to Skout's servers through a simple API. When a user's profile is viewed an unencrypted, plain-HTTP request is made to to a path like this:

http://i22.skout.com/services/ServerService/getProfile

What's returned is a blob of XML containing the user's complete profile data. In fact, the profile data is too complete, including some bits of data information that is never actually used by the app. For example, we can see the user's exact date of birth:

<ax213:birthdayDate>xx/xx/1995</ax213:birthdayDate>

... but only the user's age in years is actually displayed. Most serious, however, is the high-precision location information that is returned in the ax213:homeLocation and ax213:location tags:

<ax213:latitude>-xx.xxx</ax213:latitude>
<ax213:longitude>xxx.xxx</ax213:longitude>

The three decimal places of precision in the co-ordinates is enough to locate a user to within about 110 meters north-south, and substantially less than that east-west depending on the distance from the equator. Here's what that looks like in a hypothetical example:

I used mitmproxy to observe Skout's traffic, but because the request is unencrypted any tool that allows you to inspect network traffic would be enough. The result is a stalker's wet dream - click on an anonymous profile, watch your network traffic, and find out exactly where the victim lives. I've also seen minors located at malls where they hang out, and at their schools... Given the scale of Skout's userbase and the ease with which the data can be obtained, I think there's a high likelihood that this issue has already been used for unsavoury purposes.

I reported the vulnerability to Skout on the 24th of May. I'm happy to report that they immediately realised the seriousness of the situation, and their API stopped returning exact lat/long values a few hours later. Subsequent correspondence with Niklas Lindstrom, Skout's CTO, confirmed that they were taking steps to tighten security. I've encouraged Skout to speak about this publicly - their userbase needs to know about the issue, and need to be reassured that action is being taken to ensure that this type of privacy breach won't ever recur.

How mitmproxy works

Thu, 16 May 2013 08:53:00 GMT

How mitmproxy works

16 May 2013

I started work on mitmproxy because I was frustrated with the available interception tools. I had a long list of minor complaints - they were insufficiently flexible, not programmable enough, mostly written in Java (a language I don't enjoy), and so forth. My most serious problem, though, was opacity. The best tools were all closed source and commercial. SSL interception is a complicated and delicate process, and after a certain point, not understanding precisely what your proxy is doing just doesn't fly.

The text below is now part of the official documentation of mitmproxy. It's a detailed description of mitmproxy's interception process, and is more or less the overview document I wish I had when I first started the project. I proceed by example, starting with the simplest unencrypted explicit proxying, and working up to the most complicated interaction - transparent proxying of SSL-protected traffic¹ in the presence of SNI.

Explicit HTTP

Configuring the client to use mitmproxy as an explicit proxy is the simplest and most reliable way to intercept traffic. The proxy protocol is codified in the HTTP RFC, so the behaviour of both the client and the server is well defined, and usually reliable. In the simplest possible interaction with mitmproxy, a client connects directly to the proxy and makes a request that looks like this:

GET http://example.com/index.html HTTP/1.1

This is a proxy GET request - an extended form of the vanilla HTTP GET request that includes a schema and host specification, and it includes all the information mitmproxy needs to relay the request upstream.

1	The client connects to the proxy and makes a request.
2	Mitmproxy connects to the upstream server and simply forwards the request on.

Explicit HTTPS

The process for an explicitly proxied HTTPS connection is quite different. The client connects to the proxy and makes a request that looks like this:

CONNECT example.com:443 HTTP/1.1

A conventional proxy can neither view nor manipulate an SSL-encrypted data stream, so a CONNECT request simply asks the proxy to open a pipe between the client and server. The proxy here is just a facilitator - it blindly forwards data in both directions without knowing anything about the contents. The negotiation of the SSL connection happens over this pipe, and the subsequent flow of requests and responses are completely opaque to the proxy.

The MITM in mitmproxy

This is where mitmproxy's fundamental trick comes into play. The MITM in its name stands for Man-In-The-Middle - a reference to the process we use to intercept and interfere with these theoretically opaque data streams. The basic idea is to pretend to be the server to the client, and pretend to be the client to the server, while we sit in the middle decoding traffic from both sides. The tricky part is that the Certificate Authority system is designed to prevent exactly this attack, by allowing a trusted third-party to cryptographically sign a server's SSL certificates to verify that they are legit. If this signature doesn't match or is from a non-trusted party, a secure client will simply drop the connection and refuse to proceed. Despite the many shortcomings of the CA system as it exists today, this is usually fatal to attempts to MITM an SSL connection for analysis. Our answer to this conundrum is to become a trusted Certificate Authority ourselves. Mitmproxy includes a full CA implementation that generates interception certificates on the fly. To get the client to trust these certificates, we register mitmproxy as a trusted CA with the device manually.

Complication 1: What's the remote hostname?

To proceed with this plan, we need to know the domain name to use in the interception certificate - the client will verify that the certificate is for the domain it's connecting to, and abort if this is not the case. At first blush, it seems that the CONNECT request above gives us all we need - in this example, both of these values are "example.com". But what if the client had initiated the connection as follows:

CONNECT 10.1.1.1:443 HTTP/1.1

Using the IP address is perfectly legitimate because it gives us enough information to initiate the pipe, even though it doesn't reveal the remote hostname.

Mitmproxy has a cunning mechanism that smooths this over - upstream certificate sniffing. As soon as we see the CONNECT request, we pause the client part of the conversation, and initiate a simultaneous connection to the server. We complete the SSL handshake with the server, and inspect the certificates it used. Now, we use the Common Name in the upstream SSL certificates to generate the dummy certificate for the client. Voila, we have the correct hostname to present to the client, even if it was never specified.

Complication 2: Subject Alternative Name

Enter the next complication. Sometimes, the certificate Common Name is not, in fact, the hostname that the client is connecting to. This is because of the optional Subject Alternative Name field in the SSL certificate that allows an arbitrary number of alternative domains to be specified. If the expected domain matches any of these, the client will proceed, even though the domain doesn't match the certificate Common Name. The answer here is simple: when extract the CN from the upstream cert, we also extract the SANs, and add them to the generated dummy certificate.

Complication 3: Server Name Indication

One of the big limitations of vanilla SSL is that each certificate requires its own IP address. This means that you couldn't do virtual hosting where multiple domains with independent certificates share the same IP address. In a world with a rapidly shrinking IPv4 address pool this is a problem, and we have a solution in the form of the Server Name Indication extension to the SSL and TLS protocols. This lets the client specify the remote server name at the start of the SSL handshake, which then lets the server select the right certificate to complete the process.

SNI breaks our upstream certificate sniffing process, because when we connect without using SNI, we get served a default certificate that may have nothing to do with the certificate expected by the client. The solution is another tricky complication to the client connection process. After the client connects, we allow the SSL handshake to continue until just after the SNI value has been passed to us. Now we can pause the conversation, and initiate an upstream connection using the correct SNI value, which then serves us the correct upstream certificate, from which we can extract the expected CN and SANs.

There's another wrinkle here. Due to a limitation of the SSL library mitmproxy uses, we can't detect that a connection hasn't sent an SNI request until it's too late for upstream certificate sniffing. In practice, we therefore make a vanilla SSL connection upstream to sniff non-SNI certificates, and then discard the connection if the client sends an SNI notification. If you're watching your traffic with a packet sniffer, you'll see two connections to the server when an SNI request is made, the first of which is immediately closed after the SSL handshake. Luckily, this is almost never an issue in practice.

Putting it all together

Lets put all of this together into the complete explicitly proxied HTTPS flow.

1	The client makes a connection to mitmproxy, and issues an HTTP CONNECT request.
2	Mitmproxy responds with a 200 Connection Established, as if it has set up the CONNECT pipe.
3	The client believes it's talking to the remote server, and initiates the SSL connection. It uses SNI to indicate the hostname it is connecting to.
4	Mitmproxy connects to the server, and establishes an SSL connection using the SNI hostname indicated by the client.
5	The server responds with the matching SSL certificate, which contains the CN and SAN values needed to generate the interception certificate.
6	Mitmproxy generates the interception cert, and continues the client SSL handshake paused in step 3.
7	The client sends the request over the established SSL connection.
7	Mitmproxy passes the request on to the server over the SSL connection initiated in step 4.

Transparent HTTP

When a transparent proxy is used, the HTTP/S connection is redirected into a proxy at the network layer, without any client configuration being required. This makes transparent proxying ideal for those situations where you can't change client behaviour - proxy-oblivious Android applications being a common example.

To achieve this, we need to introduce two extra components. The first is a redirection mechanism that transparently reroutes a TCP connection destined for a server on the Internet to a listening proxy server. This usually takes the form of a firewall on the same host as the proxy server - iptables on Linux or pf on OSX. Once the client has initiated the connection, it makes a vanilla HTTP request, which might look something like this:

GET /index.html HTTP/1.1

Note that this request differs from the explicit proxy variation, in that it omits the scheme and hostname. How, then, do we know which upstream host to forward the request to? The routing mechanism that has performed the redirection keeps track of the original destination for us. Each routing mechanism has a different way of exposing this data, so this introduces the second component required for working transparent proxying: a host module that knows how to retrieve the original destination address from the router. In mitmproxy, this takes the form of a built-in set of modules that know how to talk to each platform's redirection mechanism. Once we have this information, the process is fairly straight-forward.

1	The client makes a connection to the server.
2	The router redirects the connection to mitmproxy, which is typically listening on a local port of the same host. Mitmproxy then consults the routing mechanism to establish what the original destination was.
3	Now, we simply read the client's request...
4	... and forward it upstream.

Transparent HTTPS

The first step is to determine whether we should treat an incoming connection as HTTPS. The mechanism for doing this is simple - we use the routing mechanism to find out what the original destination port is. By default, we treat all traffic destined for ports 443 and 8443 as SSL.

From here, the process is a merger of the methods we've described for transparently proxying HTTP, and explicitly proxying HTTPS. We use the routing mechanism to establish the upstream server address, and then proceed as for explicit HTTPS connections to establish the CN and SANs, and cope with SNI.

1	The client makes a connection to the server.
2	The router redirects the connection to mitmproxy, which is typically listening on a local port of the same host. Mitmproxy then consults the routing mechanism to establish what the original destination was.
3	The client believes it's talking to the remote server, and initiates the SSL connection. It uses SNI to indicate the hostname it is connecting to.
4	Mitmproxy connects to the server, and establishes an SSL connection using the SNI hostname indicated by the client.
5	The server responds with the matching SSL certificate, which contains the CN and SAN values needed to generate the interception certificate.
6	Mitmproxy generates the interception cert, and continues the client SSL handshake paused in step 3.
7	The client sends the request over the established SSL connection.
7	Mitmproxy passes the request on to the server over the SSL connection initiated in step 4.

I use "SSL" to refer to both SSL and TLS in the generic sense, unless otherwise specified. ↩

pathod 0.9

Thu, 16 May 2013 08:41:00 GMT

pathod 0.9

16 May 2013

I've just released pathod 0.9, my toolset for crafting malicious and interesting HTTP traffic. Apart from the usual range of stability improvements and bugfixes, this release introduces a major new set of features: proxy support. Pathoc, the client, has sprouted support for vanilla proxy connections, and is also able to tunnel through proxies using CONNECT. Pathod, the server, will now respond to proxy requests as well as straight HTTP, and will treat CONNECT requests as SSL with on-the-fly generation of dummy certificates.

The Pathod changes in particular open a whole new range of possibilities for fuzzing and other mischief. Any client with proxy support can be directed at Pathod, which can then impersonate the upstream server and return the creatively malicious response of your choice.

There have also been some organizational changes. This is the first release based on netlib, the gonzo networking library pathod now shares with mitmproxy. Over the next while, pathod and mitmproxy will move closer together. As a sign of this, the major version numbers between these projects are now synchronized.

mitmproxy 0.9

Wed, 15 May 2013 11:43:00 GMT

mitmproxy 0.9

15 May 2013

I'm happy to announce the release of mitmproxy 0.9. This is a major release, with huge improvements to mitmproxy pretty much across the board. So much has happened in the year since the last release that it's difficult to pick out the headlines. Mitmproxy is now faster, more scalable, and works in more tricky corner cases than ever before. Full transparent mode support has landed for both Linux and OSX. Content decoding is much nicer, with a slew of new targets like AMF and Protocol Buffers. We now have a WSGI container that allows you to host web apps right in the proxy. In addition to this, there is a myriad of new features, bugfixes and other small improvements.

There are also changes afoot in the project itself. As a first step, I've moved mitmproxy from the GPLv3 to an MIT license. I hope that this will make it easier for people to use the project in more contexts. Keep an eye out for more changes along these lines soon, geared to broadening participation in the project.

Changelog

Upstream certs mode is now the default.
Add a WSGI container that lets you host in-proxy web applications.
Full transparent proxy support for Linux and OSX.
Introduce netlib, a common codebase for mitmproxy and pathod (http://github.com/cortesi/netlib).
Full support for SNI.
Color palettes for mitmproxy, tailored for light and dark terminal backgrounds.
Stream flows to file as responses arrive with the "W" shortcut in mitmproxy.
Extend the filter language, including ~d domain match operator, ~a to match asset flows (js, images, css).
Follow mode in mitmproxy ("F" shortcut) to "tail" flows as they arrive.
--dummy-certs option to specify and preserve the dummy certificate directory.
Server replay from the current captured buffer.
Huge improvements in content views. We now have viewers for AMF, HTML, JSON, Javascript, images, XML, URL-encoded forms, as well as hexadecimal and raw views.
Add Set Headers, analogous to replacement hooks. Defines headers that are set on flows, based on a matching pattern.
A graphical editor for path components in mitmproxy.
A small set of standard user-agent strings, which can be used easily in the header editor.
Proxy authentication to limit access to mitmproxy

Google, destroyer of ecosystems

Thu, 14 Mar 2013 16:15:00 GMT

Google, destroyer of ecosystems

14 March 2013

Google has finally shut down a service I actually care about - Google Reader will die a graceless, undignified death on July 1, 2013. The only way Google could inconvenience me more would be to shut down search itself, and yet - I'm not angry that Google is shutting Reader down. I'm furious that they ever entered the RSS game at all. Consider this quote from a TechCrunch article in January 2006. Here, Michael Arrington ends an article about the shutdown of a feed reader service with a statement that seems truly bizarre today:

The RSS reader space is becoming hyper competitive, with dozens of different choices for readers.

A hyper competitive space with dozens of choices? Reader made its first public appearance a couple of months before this, in October 2005. I remember this period well - it was a time of immense excitement, when RSS seemed to be the future, the news ecosystem was vibrant, and this thing called the blogosphere, fueled by peer subscription, was doubling in size every six months. It was into this magic garden that Google wandered, like a giant toddler leaving destruction in its wake. Reader was undeniably a good product, but it's best quality was also its worst: it was free. Subsidized by Google's immense search profits, it never had to earn its keep, and its competitors started to die. Over time, the "hyper competitive" RSS reader market turned into a monoculture. Today, on the eve of its shutdown, RSS more or less means "Google Reader" to a large fraction of readers, to the extent where even the best feed readers on IOS are just Google Reader clients¹.

The sudden shock of Reader's closure will harm a news ecosystem that I already believe to be deeply ill. Google Reader is not just a core part of my information diet - it's also the most direct channel I have to readers of this blog. As of today, the Reader subscriber count for corte.si stands at about 3 times the total number of other subscribers combined. Some of these readers will migrate to other services and stay in touch, but many will inevitably abandon the idea of direct subscription to blogs entirely. In the next few months, tens of thousands of small blogs will lose direct contact with a large fraction of their readers.

The truth is this: Google destroyed the RSS feed reader ecosystem with a subsidized product, stifling its competitors and killing innovation. It then neglected Google Reader itself for years, after it had effectively become the only player. Today it does further damage by buggering up the already beleaguered links between publishers and readers. It would have been better for the Internet if Reader had never been at all.

Yes, I'm aware that there are a few hardy outliers still playing in this place. My own logs show that their reach is insignificant, though, and when I tried to shift my subscriptions about a year ago, there was nothing as good as Reader itself. Once NewsBlur's servers have recovered, I definitely plan to give it another shot. ↩

Things I found on GitHub: aspell custom dictionary entries

Tue, 26 Feb 2013 09:07:00 GMT

Things I found on GitHub: aspell custom dictionary entries

26 February 2013

I've been doing a series of posts looking at data gathered with ghrabber, a simple tool I wrote that lets you grab files matching a search specification from GitHub. Last week, I looked at shell history in the broad, and then specifically at pipe chains. Today, I move on to something different - custom aspell dictionaries. When aspell finds a word it doesn't recognize, the user is prompted to correct it, ignore it, or add it to a custom dictionary so that it will be recognized as correct in future. These words are written to the user's custom dictionary - a file named .aspell_en_pw that lives in the user's home directory. It turns out that 30 people have checked aspell dictionaries into GitHub, containing a total of 9501 custom words. The chart below shows the top 50 words, with the X-axis showing the percentage of files the word appeared in.

There were a few requests for the raw data behind the previous two posts, so this time round you can also download a CSV file with the occurrence totals for each word in the dataset.

Things I found on GitHub: pipe chains

Fri, 22 Feb 2013 09:59:00 GMT

Things I found on GitHub: pipe chains

22 February 2013

Earlier this week I published ghrabber, a simple tool that lets you grab files matching an arbitrary search specification from GitHub. I used ghrabber to retrieve all the bash_history and zsh_history files accidentally checked in to repos, and took a light look at the dataset with some simple graphs. In total, I obtained 234 shell history files with 165k individual command entries. This is a very rare opportunity to "shoulder-surf", to actually see what people do at the command prompt, and perhaps get some insights into how to improve things.

Along those lines, today's post looks at pipe chains - that is, compound commands that pipe the output of one command to another. The pipe operator lies at the core of the Unix command-line philosophy. The fact that we can easily compose complex operations is the reason why we are able to write small tools that "do one thing well" without losing generality. The shell history data on Github can give us some real data about what people do with composed commands, and how they do it.

It turns out that about 2% of all commands issued on the command-line use pipes. The graph above shows the prevalence the most common pipe chains - that is, what percentage of the user in my sample used each chain. There's a lot of fascinating stuff we can read straight from this image.

Starting at the top, the first thing we notice is how widely used the ps | grep chain is. About 17% of users in my sample used this chain - given the type of data we have, the real-world prevalence would surely be higher still. I've just been extolling the virtues of small tools and composability, but in this case practicality should beat purity. I suggest that everyone should have a command-alias similar to this in their shell configuration:

alias pg="ps aux | grep"

I've added this to my .zshrc today, and I've already used it twice.

Next up, we have the ls | grep pipes. The vast majority of uses here could actually be accomplished using the shell's filename generation mechanism. This ranges from simple redundancies like grepping for file extensions, to performing quite complex matching operations that could be done using the shell's advanced glob operations. I'm guilty of this myself - I rarely use features like recursive globbing, expansions using character ranges, case insensitive globbing, and so forth. I've brushed up on filename expansion for my chosen shell, and perhaps you should too.

The last thing I want to point out is a pattern that's genuinely dangerous - curl | bash, along with its cousins curl | sh and wget | sh. Unfortunately, this has become the recommended installation pattern for some tool - the vast majority of invocations here are for RVM and Yeoman. I don't think it's a good idea to pipe anything from the web straight into a local shell, but the situation is made particularly dire by the fact that almost half of these invocations are either over plain HTTP or explicitly turn certificate validation off.

I'll stop here, although There are interesting things to say about nearly every entry in the graph above. Next week, I'll move on from the shell history sample, look at some other juicy datasets extracted using ghrabber.

Things I found on GitHub: shell history

Tue, 19 Feb 2013 10:37:00 GMT

Things I found on GitHub: shell history

19 February 2013

Github recently introduced hugely improved code search, one of those rare moments when a service I use adds a feature that directly and measurably measurably improves my life. Predictably, there was soon a flurry of breathless stories about the security implications. This shouldn't have been news to anyone - by now, it should be clear that better search in almost any context has security or privacy implications, a law of the universe almost as solid as the second law of thermodynamics. We saw this with Google's own code search, as well as Google proper, Facebook's Graph Search and even Bing. A certain fraction of people will always make mistakes, and and any sufficiently powerful search will allow bad guys to find and take advantage of the outliers.

After the dust had settled a bit I started wondering what else we could do with Github's search - other than snookering schmucks who checked in their private keys. I'm always enticed by data, and the combination of search and the ability to download raw checked-in files seemed like a promising avenue to explore. Lets see what we can come up with.

ghrabber - grab files from GitHub

First, some tooling. I've just released ghrabber, a simple tool that lets you grab all files matching a search specification from GitHub. Here, for instance, is an obvious wheeze - fetching all files with the extension ".key":

./ghrabber.py "extension:key"

Downloaded files are saved locally to files named user.repository. Existing files with the same name are skipped, which means that you can reasonably efficiently stop and resume a ghrab.

Shell history files

I've been having a lot of fun exploring Github with ghrabber. I'll return to this in future posts - today I'll start with a quick illustration of what can be done. One type of difficult-to-find information that is sometimes checked in to repos is shell history. Two simple ghrabber commands for the two most popular shells is all we need:

./ghrabber.py "path:.bash_history"

and

./ghrabber.py "path:.zsh_history"

After cleaning the data a bit, I had 234 history files varying in length from 1 line to just over 10 thousand, containing a total of 165k entries. I fed this into Pandas for analysis, parsing each command using a combination of hand-hacked heuristics and the built-in shlex module. The remainder of this post is a light exploration of some approaches to this dataset, steering clear of the obvious and tediously well-covered security implications.

One way to slice the data is to look at the percentage of history files a given command appears in. This gives us a nice listing of the top commands by user prevalence, which you can see in the graph on the left above. On the right, I've taken the same list of commands, and checked how many invocations are preceded by a man lookup for the command. This gives us an idea of which commonly-used commands have difficult or unintuitive interfaces. It's interesting that ln is right at the top of the list, considering how simple the command syntax is. My theory is that everyone forgets the order of the source and target files.

Since we have a list of the most widely used commands, it's also trivial to do silly popularity comparisons. Above is the obvious look at the state of the editor wars (vim is winning, folks), and a check on how tmux is doing in supplanting screen (the faster the better).

Another interesting thing to do is to look at the most commonly used flags to commands. I think having "real data" of command use may well guide us to design better command-line interfaces. I'd love to know the most common invocation flags for some of the tools I write.

I'll stop there. The data pool in this case is very deep, and there are a huge range of interesting bits of command-line ethnography that could be done. Stay posted for more in the coming weeks.

The trouble with social news

Thu, 24 Jan 2013 11:01:00 GMT

The trouble with social news

24 January 2013

There is something terribly awry with the social news ecosystem. This is a feeling that's been growing on me over the last few years, and is the reason why I've cut both Reddit and Hacker News (who together constitute pretty much all of "social news") out of my information diet. Although I've mulled over things in various conversations, I've never actually tried to put my feeling of unease in writing, until today. What's spurring me into action is a proposal by Yann LeCun that a model similar to social news be adopted for scientific peer review - self-assembled Reviewing Entities voting on streams of submitted papers, regulated by a reputation system for authors and reviewers. Basically, this is science a la Reddit: complete with subreddits, karma and upboats. I find the idea frankly terrifying.

I guess it's time, then, to put finger to keyboard and lay out what disquiets me about social news.

Karma Corrupts

You start by introducing a reputation mechanism like karma to improve some outcome - say, to increase the quality of comments, or to apply a threshold to restrict voting to trustworthy community members. This seems like a plausible and even elegant mechanism at first, until you discover the terrible side-effects.

Humans are fundamentally status-seeking social apes, and you've now introduced a visible measure of social worth that people will be driven to maximize. In the real world, we have a word for those who spend their lives accumulating karma - we call them politicians. And so, within karma communities, we see the rise of a political class - persuasive centrists who cater (perhaps unconsciously) to a constituency, and who express (perhaps eloquently) opinions calculated to appeal to the masses and avoid controversy. Hacker News and many subreddits are dominated by people like this, whose comments are largely predictable and rarely add anything new or unexpected to the conversation.

At the bottom end of the food chain, we have a different class of creature with the same basic aim as the politicians, but without the persuasive charm needed to pull off the political approach. These are the karma whores, who use a mixture of frank pandering, provocation and calculated outrage to achieve the same aims.

The karma maximization game often acts contrary to the goals we aimed to achieve by introducing karma in the first place: the tenor of the community suffers, the diversity of opinion declines, and the karma whores post pictures of their cats everywhere.

The Lossy Sieve

Go and have a look at the new story submission queue on Hacker News. Scroll through a few pages, and pay attention to the stories stuck at one vote - they will most likely never receive another upvote and will die in obscurity. Now, go look at the front page. When I do this exercise I'm struck by the fact that there's plenty of crap on the front page, and quite a bit of good stuff in the submission queue languishing in obscurity. So, quality can't be the sole metric here - what determines what gets onto the front page and what doesn't?

Lets try a thought experiment. First, set up a small number of voting accounts - say, 10 or so. Now, in the new submission queue, pick 5 random stories every hour, and give them a small number of upvotes soon after they are submitted. I predict that you will find that stories that received this small initial boost are vastly more likely to end up on the front page. If I'm right, then chance dominates story selection - as long as an article exceeds some basic quality threshold, it all depends on who happens to see the story soon after it is submitted, and whether the spirit moves them to vote. Note that this is not the case at the extremes - frankly bad content won't be upvoted, and really important stories will usually find their way to the top. The lossy sieve phenomenon affects everything in between.

What this boils down to is that social news doesn't provide an effective filter - good content gets lost, and mediocre content finds its way onto our screens.

The Pinhole Effect

In social news, the front page is king. Most users never go beyond the first or second page of top stories. However, front-page real estate is incredibly limited compared to the volume of submissions on most popular subreddits and on Hacker News. The effect of this is that we're looking at a fast-flowing river of information through a pinhole. Even assuming that the selection mechanism works flawlessly, what you see on the front page is a small sliver of the total, chosen through a consensus mechanism that takes no account of individual variation in tastes and interests. The news you see is not tailored to you - it's tailored to some abstract, average participant, with all the rough edges of individuality smoothed away. The effect of this is that even at its best, the stories that emerge from the social news system feel like a predictable pablum dished up by the hivemind. The subreddit system tries to improve this by allowing communities to self-assemble around interests, but the pinhole effect still dominates in busy subreddits like /r/programming.

Gaming The System

Social news systems are eminently gameable, and cheating is rife. Part of the reason for this is that a story's destiny depends on a relatively small number of votes. If your story has any merit at all, you significantly increase the likelihood that it will end up on the front page by giving it a small nudge at the beginning of its life. If it has no merit whatsoever, you can still force it onto people's screens with a few tens or hundreds of votes. Conversely, you can use the same effect to censor and oppress views you disagree with if your social news site has downvotes. Anyone who's kept an eye on these things can rattle off examples of gaming in action: the voting rings, the "social media consultants", the vigilante thought-polizei, the political operators, and dozens of other types of manipulation and villainy. What's more - these visible scandals are just the tip of the iceberg. Eyeballs are valuable, and there's an active arms race with social news sites on the one side, and a dark army of spammers, scammers and true believers on the other. How much of what we see is affected by this type of cheating? We just don't know, but my suspicion is that the effect is significant.

The point here is broader than any particular instance of gaming. It's that social news sites are structurally susceptible to manipulation in ways that can't be fixed without changing the core of their operation. A system like this might be good enough to deliver rage comics, but I feel queasy trusting it any further.

Community Collapse Disorder

My final beef with social news is a problem that it shares with pretty much all online communities, especially technical ones. We're all familiar with the life-cycle of technical forums. They start with a small community of insiders who create value, which then attracts more people to participate, which then dilutes the quality of the contributions (and often introduces a few pathological bad actors), which then causes the good contributors to move on, which causes the magic well to dry up. Everyone then take their toys and move to the next community, and the cycle repeats. We saw this with Usenet and the original C2 wiki, and we are seeing it now with Hacker News and many technical subreddits all at various points in this life-cycle.

I believe that Community Collapse Disorder is one of the Big Problems online that we don't yet have a satisfactory solution to. People are trying, though. Hacker News, for instance, seems to be rather poignantly aware of its own decline, with some of the best of the old-timers calling for an alternative. Paul Graham himself recognizes the issue, and has been tweaking things in various ways to combat the phenomenon, without much success.

At the moment, we just don't know how to build online communities that are both inclusive and stable. Democracy, here, seems to lead inevitably to decline, and social news sites are no exception.

A better way forward?

A big part of the reason I don't use social news anymore is that my existing social networks have become so much more effective at turning up good content. The absolute best source of news for me is simply the set of links shared by the folks I follow on Twitter. I follow people who post interesting content, and whom I trust to act as information filters for me. Most of them share my technical interests, but some are interesting because they are from my home town, or because they share some more esoteric pursuit with me. So, the news stream I see is exactly tailored to me. At the same time, there is also room idiosyncrasy - if someone I follow shares something left-field that tickles their fancy, I'll see it. In turn, I try to be a responsible information filter for those who follow me - I find a link or two worth tweeting on most days.

There are still things I miss - Twitter is great for sharing links, but is an awful medium for technical discussion. Google+ could be a better alternative, but just doesn't seem to have achieved liftoff for me. I would also love better tools for aggregating and harvesting links from my social network. At the moment I use Flipboard and Prismatic, but I have issues with both. On the whole, though, these are quibbles. It seems to me that using social networks to filter news is a better way forward - if I was tackling the social news problem, I'd be building tools to support this process.

Go: a nice language with an annoying personality

Fri, 18 Jan 2013 11:51:00 GMT

Go: a nice language with an annoying personality

18 January 2013

Last week, I had the pleasure of attending Dropbox's annual company hack fest. It was a great opportunity to get a look at how Dropbox works internally, and mingle with the smart and driven folks who make one of my favourite products. In the spirit of hack week, me and my friend @alexdong decided to do our project in Go. We'd both wanted to explore the language, but had never quite been able to make time - a week-long code holiday seemed to be the perfect opportunity. I was hopeful that Go would turn out to hit a magical sweet spot: a light set of abstractions hugging close to the machine, while still providing the indoor plumbing and civilized conveniences of life that I had grown used to with languages like Python. Five days of furious hacking later, I can report that Go might well deliver on this promise, but has enough annoying personality quirks that I will think twice about basing any more projects on it.

My main beef with Go has nothing to do with fundamental language design, and may seem almost inconsequential at first glance. The Go compiler treats unused module imports and declared variables as compile errors. This is great in theory and is something you might well want to enforce before code can be committed, but during the actual process of producing code it's nothing but an irksome, unnecessary pain in the ass. Let's look at a concrete example, starting with a snippet of code as follows ¹

import (
    "io/ioutil"
)
...
...
    m, err := ioutil.ReadFile(path)
    if err != nil {
        return nil, err
    }
...
...
    DoSomething(m)

I'm a firm believer that printing stuff to screen is a programmer's best debugging tool, so say we're hacking away and want to print the value of m while running our unit tests. We change the code as follows, adding an import for the "fmt" module and a call to Print:

import (
    "io/ioutil"
    "fmt"
)
...
...
    m, err := ioutil.ReadFile(path)
    if err != nil {
        return nil, err
    }
    fmt.Print(m)
...
...
    DoSomething(m)

Now we keep hacking, and want to comment out the print statement for a moment like so:

import (
    "io/ioutil"
    "fmt"
)
...
...
    m, err := ioutil.ReadFile(path)
    if err != nil {
        return nil, err
    }
    //fmt.Print(m)
...
...
    DoSomething(m)

This is a compile error. We have to switch contexts, move to the top of the module, also comment out the import, and then move back to the spot we're really hacking on:

import (
    "io/ioutil"
    //"fmt"
)
...
...
    m, err := ioutil.ReadFile(path)
    if err != nil {
        return nil, err
    }
    //fmt.Print(m)
...
...
    DoSomething(m)

A few seconds later, we want to re-enable the Print statement - so up we go again to the top of the module to re-enable the import. This is even worse when we want to, say, comment out the DoSomething call while hacking:

import (
    "io/ioutil"
)
...
...
    m, err := ioutil.ReadFile(path)
    if err != nil {
        return nil, err
    }
...
...
    //DoSomething(m)

This is also a compile error because now m is unused. We have to hunt up in our code to find the declaration, which could be explicit or implicit using an := assignment. So, in this case we find the declaration, and use the magic underscore name to throw the offending value away:

import (
    "io/ioutil"
)
...
...
    _, err := ioutil.ReadFile(path)
    if err != nil {
        return nil, err
    }
...
...
    //DoSomething(m)

That should fix it, right? Well, no. It turns out we've previously declared and used err (a very common idiom), so this is still a compile error. We're using the "declare and assign" syntax, but have no new variables on the left-hand side of the ":=". So we need to make another tweak:

import (
    "io/ioutil"
)
...
...
    _, err = ioutil.ReadFile(path)
    if err != nil {
        return nil, err
    }
...
...
    //DoSomething(m)

Five seconds later, we want to re-enable DoSomething, and now we have to unwind the entire process.

The cumulative effect of all this is like trying to write code while someone next to you randomly knocks your hands off the keyboard every few seconds. It's a pointlessly pedantic approach that adds constant friction to your write-compile-test cycle, breaks your flow, and just generally makes life a little harder for very little benefit. There's no way to turn this mis-feature off, no flag we can pass to the compiler to temporarily make this a warning rather than an error while hacking².

The irony of the situation is that I agree with the sentiment behind this. I don't want dangling variables or imports in my codebase. And I agree that if something is worth warning about it's worth making it an error. The mistake is to confuse the state we want at the conclusion of a unit of hacking³, with what we need at every point in between, during the write-compile-test cycle. This cycle is the core of the process of actually producing code, and the exhilarating sense of weightlessness that you get when hacking in Python is largely due to the fact that the language works really, really hard to optimize this process. Go has given away this feeling of exhilaration, basically for nothing.

Despite all this, it's still possible that the benefits of Go do outweigh its irritating personality. Interfaces, memory management, first-class concurrency and static type checking is a knockout combination, and the language in general has something of the taut practicality that I love in C. So, despite the rantiness of this post, I'll keep hacking on our project and make sure I produce a few thousand more lines of code before making a final call on the language. Look for a project release and a blog post along these lines in the coming months.

Ellipses indicate "an arbitrary amount of intervening code" ↩
I edited this paragraph a bit for tone. I originally accused the Go documentation of being faintly smug about all of this - which is not fair, and doesn't add anything to the argument. ↩
Why don't we have a word for this? By "unit of hacking", I mean the work that goes on between starting to hack on a change-set and doing a commit. At the beginning and at the end, the code is in a clean state, but in between there are many periods of transition where cleanliness requirements are relaxed. ↩

Released: pathod 0.3

Fri, 16 Nov 2012 15:25:00 GMT

Released: pathod 0.3

16 November 2012

I've just released pathod 0.3, which beefs up pathoc's fuzzing capabilities, improves the spec language and includes lots of bugfixes and other small tweaks. Get it while it's hot!

Better fuzzing

A major focus of this release is to improve pathoc's capabilities as a basic fuzzing tool. I've had fun breaking webservers with pathoc, and it's even come in handy in my Day Job. Here's a quick summary of how things have changed.

The -x flag tells pathoc to explain its requests. This prints out an expanded pathoc query specification, with all randomly generated content and query modifications resolved. If you trigger an exception, you can precisely replay the offending query using this explanation.
The options for outputting requests and responses have been expanded hugely. First, the -q and -r flags tell pathoc to dump complete records of requests and responses respectively. This data is sniffed by instrumenting the socket, so is canonical regardless of our ability to interpret returned data. The -x option makes pathod dump this data in hexdump format (otherwise unprintable characters are escaped to preserve your terminal).
A number of options have been added to let you ignore expected responses. -C takes a comma-separated list of response codes to ignore. -T ignores server timeouts. This lets you hone in on the exceptional responses that you care about, and ignore the rest.

Language improvements

I've simplified response specifications by making the response message a standard component with the "r" mnemonic.
I've added the "u" mnemonic to request specifications, as a shortcut for specifying the User-Agent header:

get:/:u"My Weird User-Agent"

We also have a small library of representative User-Agent strings that can be used instead of specifying your own. For example, this specifies the GoogleBot User-Agent string:

get:/:ug

The list of available shortcuts are in the docs, and can be listed from the commandline using the --show-uas flag to pathoc:

> ./pathoc --show-uas
User agent strings:
   a android
   l blackberry
   b bingbot
   c chrome
   f firefox
   g googlebot
   i ie9
   p ipad
   h iphone
   s safari

pathoc: break all the Python webservers!

Thu, 27 Sep 2012 09:46:00 GMT

pathoc: break all the Python webservers!

27 September 2012

A few months ago, I announced pathod, a pathological HTTP daemon. The project started as a testing tool to let me craft standards-violating HTTP responses while working on mitmproxy. It soon became a free-standing project, and has turned out to be incredibly useful in security testing, exploit delivery and general creative mischief. In the last release, I added pathoc - pathod's malicious client-side twin. It does for HTTP requests what pathod does for HTTP responses, and uses the same hyper-terse specification language.

In this post, I show how pathoc can be used as a very simple fuzzer, by finding issues in a number of major pure-Python webservers. None of the tested servers failed catastrophically - they all caught the unexpected exception and continued serving requests. None the less, I think it's reasonable to say that we've triggered a bug if a) the server returns an 500 Internal Server Error response or terminates the connection abnormally, and b) we see a traceback in our logs. In fact, by this definition, I found bugs in every pure-Python server I tested.

All of the problems I list below are simple failures of validation - what they have in common is that somewhere in the project code is called with input that it doesn't expect and can't handle. This matters - in fact, I'd argue that the majority of security problems fall in this category. It's interesting to ponder why this type of issue is so ubiquitous in Python servers. I have no doubt that part the answer lies in Python's use of exceptions - errors that would be explicit in other languages can be implicit in Python, and code that seems clean and intuitive might in fact be buggy. I think this is especially relevant right now, given the recent flurry of discussion surrounding the Go language and its error handling. It's pretty instructive to read Russ Cox's recent riposte to this post criticizing Go's explicit approach, while looking at the bugs below. I love Python and I think it's a fine language, but I also think the designers of Go probably made the right choice.

Basic fuzzing with pathoc

My methodology for these tests was very simple indeed. I launched each server in turn, and used pathod to fire corrupted GET requests at the daemon until I saw an error. I then looked at the logs, and boiled the distinct cases down to a minimal pathoc specification by hand. This exercises a rather shallow set of features in the server software - mostly parsing of the HTTP lead-in and request headers. It's possible to give software a much, much deeper workout with pathoc, but I'll leave that for a future post.

My pathoc fuzzing command looked something like this:

pathoc -n 1000 -p 8080 -t 1 localhost 'get:/:b@10:ir,"\x00"'

The most important flags here are -n, which tells pathoc to make 1000 consecutive requests, and -t, which tells pathoc to time out after one second (necessary to prevent hangs when daemons terminate improperly). The request specification itself breaks down as follows:

get	Issue a GET request
/	... to the path /
b@10	... with a body consisting of 10 random bytes
ir,"\x00"	... and inject a NULL byte at a random location.

It's that last clause - the random injection - that makes the difference between simply crafting requests and basic fuzzing. Every time a new request is issued, the injection occurs at a different location. I varied the injected character between a NULL byte, a carriage return and a random alphabet letter. Each exposed different errors in different servers. For a complete description of the specification language, see the online docs.

Results

For each bug, I've given a traceback and a minimal pathoc call to trigger the issue. The tracebacks have been edited lightly to shorten file paths and remove irrelevances like timestamps.

pathoc -p 8080 localhost 'get:/:b@10:h"Content-Length"="x"'

ENGINE ValueError("invalid literal for int() with base 10: 'x'",)
Traceback (most recent call last):
  File "cherrypy/wsgiserver/wsgiserver2.py", line 1292, in communicate
    req.parse_request()
  File "cherrypy/wsgiserver/wsgiserver2.py", line 591, in parse_request
    success = self.read_request_headers()
  File "cherrypy/wsgiserver/wsgiserver2.py", line 711, in read_request_headers
    if mrbs and int(self.inheaders.get("Content-Length", 0)) > mrbs:
ValueError: invalid literal for int() with base 10: 'x'

pathoc -p 8080 localhost 'get:/:i4,"\r"

ENGINE TypeError("argument of type 'NoneType' is not iterable",)
Traceback (most recent call last):
  File "cherrypy/wsgiserver/wsgiserver2.py", line 1292, in communicate
    req.parse_request()
  File "cherrypy/wsgiserver/wsgiserver2.py", line 580, in parse_request
    success = self.read_request_line()
  File "cherrypy/wsgiserver/wsgiserver2.py", line 644, in read_request_line
    if NUMBER_SIGN in path:
TypeError: argument of type 'NoneType' is not iterable

pathoc -p 8080 localhost 'get:/:b@10:h"Content-Length"="x"'

[E 120927 11:42:26 iostream:307] Uncaught exception, closing connection.
    Traceback (most recent call last):
      File "tornado/iostream.py", line 304, in wrapper
        callback(*args)
      File "tornado/httpserver.py", line 254, in _on_headers
        content_length = int(content_length)
    ValueError: invalid literal for int() with base 10: 'x'
[E 120927 11:42:26 ioloop:435] Exception in callback 
    Traceback (most recent call last):
      File "tornado/ioloop.py", line 421, in _run_callback
        callback()
      File "tornado/iostream.py", line 304, in wrapper
        callback(*args)
      File "tornado/httpserver.py", line 254, in _on_headers
        content_length = int(content_length)
    ValueError: invalid literal for int() with base 10: 'x'

pathoc -p 8080 localhost 'get:/:h"h\r\n"="x"'

[E iostream:307] Uncaught exception, closing connection.
    Traceback (most recent call last):
      File "tornado/iostream.py", line 304, in wrapper
        callback(*args)
      File "tornado/httpserver.py", line 236, in _on_headers
        headers = httputil.HTTPHeaders.parse(data[eol:])
      File "tornado/httputil.py", line 127, in parse
        h.parse_line(line)
      File "tornado/httputil.py", line 113, in parse_line
        name, value = line.split(":", 1)
    ValueError: need more than 1 value to unpack
[E ioloop:435] Exception in callback 
    Traceback (most recent call last):
      File "tornado/ioloop.py", line 421, in _run_callback
        callback()
      File "tornado/iostream.py", line 304, in wrapper
        callback(*args)
      File "tornado/httpserver.py", line 236, in _on_headers
        headers = httputil.HTTPHeaders.parse(data[eol:])
      File "tornado/httputil.py", line 127, in parse
        h.parse_line(line)
      File "tornado/httputil.py", line 113, in parse_line
        name, value = line.split(":", 1)
    ValueError: need more than 1 value to unpack

pathoc -p 8080 localhost 'get:/:b@10:h"Content-Length"="x"'

[HTTPChannel,4,127.0.0.1] Unhandled Error
    Traceback (most recent call last):
      File "twisted/python/log.py", line 84, in callWithLogger
        return callWithContext({"system": lp}, func, *args, **kw)
      File "twisted/python/log.py", line 69, in callWithContext
        return context.call({ILogContext: newCtx}, func, *args, **kw)
      File "twisted/python/context.py", line 118, in callWithContext
        return self.currentContext().callWithContext(ctx, func, *args, **kw)
      File "twisted/python/context.py", line 81, in callWithContext
        return func(*args,**kw)
    ---  ---
      File "twisted/internet/selectreactor.py", line 150, in _doReadOrWrite
        why = getattr(selectable, method)()
      File "twisted/internet/tcp.py", line 199, in doRead
        rval = self.protocol.dataReceived(data)
      File "twisted/protocols/basic.py", line 564, in dataReceived
        why = self.lineReceived(line)
      File "twisted/web/http.py", line 1558, in lineReceived
        self.headerReceived(self.__header)
      File "twisted/web/http.py", line 1580, in headerReceived
        self.length = int(data)
    exceptions.ValueError: invalid literal for int() with base 10: 'x'

pathoc -p 8080 localhost 'get:"/\0"'

Exception happened during processing of request from ('127.0.0.1', 54029)
Traceback (most recent call last):
  File "lib/python2.7/SocketServer.py", line 284, in _handle_request_noblock
    self.process_request(request, client_address)
  File "lib/python2.7/SocketServer.py", line 310, in process_request
    self.finish_request(request, client_address)
  File "lib/python2.7/SocketServer.py", line 323, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "lib/python2.7/SocketServer.py", line 638, in __init__
    self.handle()
  File "python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "lib/python2.7/BaseHTTPServer.py", line 328, in handle_one_request
    method()
  File "lib/python2.7/SimpleHTTPServer.py", line 44, in do_GET
    f = self.send_head()
  File "lib/python2.7/SimpleHTTPServer.py", line 68, in send_head
    if os.path.isdir(path):
  File "lib/python2.7/genericpath.py", line 41, in isdir
    st = os.stat(s)
TypeError: must be encoded string without NULL bytes, not str

pathoc -p 8080 localhost 'get:/:i16," "'

ERROR:waitress:uncaptured python exception, closing channel 
 
(
    :list index out of range 
        [lib/python2.7/asyncore.py|read|83] 
        [lib/python2.7/asyncore.py|handle_read_event|444]
        [lib/python2.7/site-packages/waitress/channel.py|handle_read|169]
        [lib/python2.7/site-packages/waitress/channel.py|received|186]
        [lib/python2.7/site-packages/waitress/parser.py|received|99]
        [lib/python2.7/site-packages/waitress/parser.py|parse_header|158]
        [lib/python2.7/site-packages/waitress/parser.py|get_header_lines|247]
)

Edit: The first version of this post had examples that were due to the test WSGI application, not waitress. I've replaced them with the traceback above, which has been reformatted for clarity.

pathoc -p 8080 localhost 'get:/:h"Host"="n\r\0"'

Traceback (most recent call last):
  File "flask/app.py", line 1518, in __call__
    return self.wsgi_app(environ, start_response)
  File "flask/app.py", line 1507, in wsgi_app
    return response(environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/werkzeug/wrappers.py", line 1082, in __call__
    app_iter, status, headers = self.get_wsgi_response(environ)
  File "werkzeug/wrappers.py", line 1070, in get_wsgi_response
    headers = self.get_wsgi_headers(environ)
  File "werkzeug/wrappers.py", line 986, in get_wsgi_headers
    headers['Location'] = location
  File "werkzeug/datastructures.py", line 1132, in __setitem__
    self.set(key, value)
  File "werkzeug/datastructures.py", line 1097, in set
    self._validate_value(_value)
  File "werkzeug/datastructures.py", line 1065, in _validate_value
    raise ValueError('Detected newline in header value.  This is '
ValueError: Detected newline in header value.  This is a potential security problem

Limits of data visualization with space filling curves

Thu, 20 Sep 2012 12:30:00 GMT

Limits of data visualization with space filling curves

20 September 2012

I recently wrote a series of posts using the Hilbert curve to visualize binaries, culminating in a gallery showing regions of high entropy in malware.

The fact that the Hilbert curve has excellent locality preservation means that one dimensional features are preserved (as much as they can be) in the two-dimensional layout. This lets us visually pick out features of interest, and makes it possible, for instance, to quickly identify different malware packers just based on their layout characteristics.

An obvious next step is to ask if it's possible to extend this idea to let us visually compare binaries, creating a sort of visual diff. Unfortunately, we now bump our heads against the limitations of space-filling curve visualization. I made the animation below after a recent conversation along these lines, and I think it illustrates the main issues nicely. It shows a single contiguous stretch of data (the black area) being shifted progressively through a binary. At each timestep, the only thing that changes is the starting location of the data block:

Two things are immediately clear:

The block of data doesn't retain its shape at different offsets - identical stretches of data can look totally different depending on their locations.
There's no way to quickly see where in the binary a piece of information lies. Unless you are very familiar with the particular curve and know its exact orientation, you can't say, for instance, when the data block lies a third of the way through the binary.

It's often worthwhile to trade off these things for locality preservation, but it definitely scotches certain use cases. I do wonder if it might be possible to tune the trade-off somewhat - sacrificing some locality preservation for better shape retention and offset estimation. I've toyed with some ideas along these lines (see the unrolled layouts in the binary visualization post), but I still don't have a satisfying solution. If anyone out there knows of one, drop me a line.

Findng the UDID leak: a guessing game

Fri, 07 Sep 2012 21:59:00 GMT

Findng the UDID leak: a guessing game

07 September 2012

It's become quite a popular parlor game to guess who is responsible for the recent Antisec UDID leak. I've now seen no less than six separate apps named as the probable source (two of which came from Marco Arment). Before we pick the next culprit, I think it's worth taking a step back to consider the list of things we don't know:

We don't know that we're dealing with just one source. The Antisec dump may well be an amalgam of data from various sources.
We don't know that we're looking for just one app, or even a set of apps by one developer. The leak may well come from one of the myriad of 3rd party services which could be included in thousands of apps.
We don't know that Antisec is being truthful about the scale of the database, or the additional data they claim is associated with the UDID/APNS records.
We certainly don't know that the data was filched from an FBI laptop or that the NCFTA was in any way involved.

Given all of these unknowns, I think a simple process-of-elimination approach to tracking down the leak will probably be fruitless, or worse, result in the finger being pointed at even more innocent parties. The one entity that may already have the answer to this question is Apple. They have a list of a million affected UDIDs, and they presumably have records of all apps that have ever used the associated push tokens. Given a large and precise sample like this, it should be possible to find the origin(s) of the leak reasonably easily. Indeed, if Apple is on the ball they may already have done this.

Now for some frank speculation of my own. Let's assume for a moment that Antisec has been entirely truthful about the data, and that we're dealing with a single source. In that case, we're looking for:

... an app or third-party service integrated into multiple apps
... with 12 million or more users
... that is APNS-enabled
... which also gathers user data like real names and zip codes.

I'll throw my hat in the ring and say that my money is on a third-party service, not a single app. If my hunch is right, the list of possible culprits is actually rather short.

The UDID leak is a privacy catastrophe

Tue, 04 Sep 2012 18:12:00 GMT

The UDID leak is a privacy catastrophe

04 September 2012

Something I've been worrying about for a long time has just happened: Antisec has leaked a database with more than a million UDIDs. The UDID issue has been a bit of a white whale of mine - I've written many blog posts about it and spent more hours than I care to think negotiating responsible disclosure with companies misusing UDIDs. Let's recap some of the posts I've written about this:

In May 2011, just before its sale to Gree was announced, I showed that OpenFeint was misusing UDIDs in a way that allowed you to link a UDID to a user's identity, geolocation and Facebook and Twitter accounts. I didn't discuss it openly at the time, you could also completely take over an OpenFeint account, and access chat, forums, friends lists, and more using just a UDID. This resulted in a class-action lawsuit against OpenFeint, which has since petered out.
Later that month, I published a survey looking at how UDIDs are used in practice. The data is now slightly out of date, but shows just how widely UDIDs are used and misused.
In September 2011, I published the most troubling news so far, which paradoxically also got the least coverage in the press. I looked at all the gaming social networks on IOS - basically OpenFeint and its competitors - and found catastrophic mismanagement by nearly everyone. The vulnerabilities ranged from de-anonymization, to takeover of the user's gaming social network account, to the ability to completely take over the user's Facebook and Twitter accounts using just a UDID.

As serious these problems are, I'm afraid it's just the tip of the iceberg. Negotiating disclosure and trying to convince companies to fix their problems has taken literally months of my time, so I've stopped publishing on this issue for the moment. It's disheartening to say it, but some of the companies mentioned in my posts still have unfixed problems (they were all notified well in advance of any publication). I will also note ominously that I know of a number of similar vulnerabilities elsewhere in the IOS app ecosystem that I've just not had the time to pursue.

When speaking to people about this, I've often been asked "What's the worst that can happen?". My response was always that the worst case scenario would be if a large database of UDIDs leaked... and here we are.

Defiler

Sun, 26 Aug 2012 13:14:00 GMT

Defiler

26 August 2012

I've been living out of a bag for the last 3 weeks, working hard on a series of intense but fun audits. After running in high gear for a while I find that I need a mental palate cleanser - something to help me refocus and stop me from getting snowblind. I then grab my camera, strap on my macro rig, and walk out the door to try to catch the local wildlife in the act. It's become a bit of a game - the aim is to catch creatures in their natural setting and leave them completely undisturbed when I go, with no posing, prodding or other disturbances. Getting a usable shot of a 5mm target sitting on a twig swaying in the wind is a fun challenge.

Today I find myself in Sydney, working in a part of the town that is shot through with unreasonably beautiful walking tracks. The place is also blessed with a huge diversity of invertebrate life that makes my adopted home town seem barren by comparison. I walked along a nearby track until I found a quiet, leafy spot, geared up, and leopard-crawled through the underbrush. Not long after, I came face-to-face with this imposing little chap sitting on the tip of a fern frond.

This is a Lymantriid caterpillar of some variety, probably one of the tussock moths native to Australia. "Lymantria" means "defiler" - some species of this family can cause huge damage to foliage, and are considered to be destructive pests. So much so, that when a single male Gypsy Moth (Lymantria dispar) was discovered in Hamilton, New Zealand, they sprayed the entire city with a caterpillar-specific bacterial insecticide.

No need for drastic measures with this particular fellow, though - he's native to this ecosystem, and the only pest is me and my camera. He was head down munching away when I found him, and paid absolutely no attention to me when I moved in close to get these shots. He's got reason to be cocksure, too - those tufts of hair on his back contain hollow, poison-filled spines that can cause a pretty unpleasant reaction when touched.

An few hours exploring and photographing is a very effective brain-cleaner, leaving me ready to deal with spiny, venomous defilers of the digital variety.

pathod 0.2: the daemon gets an evil twin

Wed, 22 Aug 2012 14:02:00 GMT

pathod 0.2: the daemon gets an evil twin

22 August 2012

I've just pushed pathod 0.2 out the door. This is a huge release, with many new features:

pathoc, pathod's evil client-side twin.
libpathod.test, a framework for using pathod in your unit tests.
Improved mini language, including many new abilities and improvements.
A rewrite of the networking core.

The project also has a new website at pathod.net. Yes, pathod is now self-hosting, so you can try out both pathod and pathoc specifications right on the website. There's also a new public pathod instance, which I'm sure everyone will use entirely responsibly.

get it here

Introducing pathod: a pathological HTTP server

Tue, 01 May 2012 08:14:00 GMT

Introducing pathod: a pathological HTTP server

01 May 2012

I've just released pathod, a pathological HTTP/S daemon useful for testing and torturing HTTP clients. At its core is a tiny, terse language for crafting HTTP responses. It also has a built-in web interface that lets you play with the response spec language, inspect logs, and access pathod's full help document.

The rest of this post is a quick teaser showing some of pathod's abilities. See the detailed documentation on the pathod site if you want more.

The simplest possible response

The easiest way to craft a response is to specify it directly in the request URL. Lets start with the simplest possible example. Start pathod, and then visit this URL:

http://localhost:9999/p/200

The "/p/" path is the location of the response generator in pathod's default configuration - everything after that a response specification in pathod's mini-language. The general form of a response spec is as follows:

code[MESSAGE]:[colon-separated list of features]

In this case, we're specifying only the HTTP response code - that is, an HTTP 200 OK with no headers and no content, resulting in a response like this:

HTTP/1.1 200 OK

Specifying features

One example of a "feature" is a response header. Lets embellish our response by adding one:

200:h"Etag"="foo"

The first letter of the feature - "h", in this case - is a mnemonic indicating the type of feature we're adding. The full response to this spec looks like this:

HTTP/1.1 200 OK
Etag: foo

Both "Etag" and "foo" are Value Specifiers, a syntax used throughout the response specification language. In this case they are literal values, as indicated by the fact that they are quoted strings. The Value Specification syntax also lets us load values from files or generate random data. For instance, here is a specification that generates 100k of random binary data for the header value:

200:h"Etag"=@100k

Now, binary data in the header value will probably break things in interesting ways, but is unlikely to be read by the client as a valid (but over-long) value. To see if the client really drops off its perch if we feed it a single 100k header, we have to constrain the random data. Here's the same response, but with data generated only from ASCII letters:

200:h"Etag"=@100k,ascii_letters

pathod has a large number of built-in character classes from which random data can be generated.

Pauses and Disconnects

Next, we can disrupt the communications in various ways. At the moment, this means adding pauses and disconnects to a response. Let's start with an HTTP 404 response with a body consisting of a 100k of random binary data:

404:b@100k

Here's the same response, but with a 120 second pause after sending 100 bytes:

404:b@100k:p120,100

And, the same response again, but with hard disconnect after sending 100 bytes:

404:b@100k:d100

Instead of specifying a time explicitly, we can ask pathod to just randomly disconnect at a time of its choosing:

404:b@100k:dr

That's it for the teaser - hopefully it's enough to entice you into looking at pathod's full documentation.

What's next?

pathod is an "airport project" - the first draft was written in its entirety during a 40-hour trip back home from New York (I drew a bad lot in stopovers). I've now firmed it up a bit, but there's still work to be done. In the next month, mitmproxy's test suite will move to pathod, after which there will be a simple, well-documented way to unit test. I also plan to build out the JSON API (which is used to drive pathod in test suites), and expand the mini-language with convenient ways to generate pathological cookies, authentication headers, SSL errors, and cache control.

mitmproxy 0.8

Mon, 09 Apr 2012 16:57:00 GMT

mitmproxy 0.8

09 April 2012

I'm happy to announce the release of mitmproxy 0.8. This release has a few major new features, big speedups, and many, many small bugfixes and improvements. Here are the headlines:

Android interception

The most prominent new feature is that we now have a supported way to intercept Android traffic. What's more, we can do this without a cumbersome transparent proxying rig - see the Android section in the documentation for the details. Special thanks goes to Jim Cheetham for lending me an Android device and helping to get this feature off the ground.

Replacement patterns

Another exceedingly useful new feature is replacement patterns. These consist of a filter, a regular expression and a replacement string, and run continuously while mitmproxy processes requests and responses. You can pass these either on the command-line, or using a built-in replacement pattern editor.

I'm sure you can immediately think of many uses for this flexible feature, but my favourite is to use it during testing as a way to conveniently inject complicated exploits into web traffic. I do this by setting a replacement pattern that swaps a short but likely unique string (say MYXSS) for a long exploit, and then I use simple interaction and front-end tools like Firebug to inject exploits into requests manually based on the short string marker.

Improved pretty-printing of request and response contents

This release of mitmproxy has a completely redesigned subsystem for pretty-printing request and response bodies. For instance, we now extract EXIF tags and other basic information to give you something better than a hex dump when looking at an image:

We also have much improved HTML indenting (using lxml), and a built-in JavaScript beautifier (thanks to JSBeautifier) that teases out compressed and obfuscated scripts into something readable.

Changelog

Detailed tutorial for Android interception. Some features that land in this release have finally made reliable Android interception possible.
Upstream-cert mode, which uses information from the upstream server to generate interception certificates.
Replacement patterns that let you easily do global replacements in flows matching filter patterns. Can be specified on the command-line, or edited interactively.
Much more sophisticated and usable pretty printing of request bodies. Support for auto-indentation of JavaScript, inspection of image EXIF data, and more.
Details view for flows, showing connection and SSL cert information (X keyboard shortcut).
Server certificates are now stored and serialized in saved traffic for later analysis. This means that the 0.8 serialization format is NOT compatible with 0.7.
Add a shortcut key ("f") to load the remainder of a request or response body, if it is abbreviated.
Many other improvements, including bugfixes, and expanded scripting API, and more sophisticated certificate handling.

mitmproxy 0.7

Mon, 27 Feb 2012 20:38:00 GMT

mitmproxy 0.7

27 February 2012

I'm happy to announce the release of mitmproxy 0.7. The biggest visible change is a new structured editor for headers, query strings and form fields. Other new feature include a reverse proxy mode, extended script API that makes many common tasks much easier, and a myriad of improvements to the interface (including a massive increase in speed). Everybody still on 0.6 should upgrade - get it here:

mitmproxy-0.7.tar.gz (docs)

You can also now install mitmproxy using pip, like so:

pip install mitmproxy

In other news, the project has had an amazing month, after a rash of high-profile results obtained using mitmproxy were published. It started with Arun Thampi's discovery that Path uploads users' address books to their servers. Things snowballed from there, and for a few days mitmproxy seemed to be everywhere. Similar findings were made for Hipster, The Verge did a mitmproxy-driven AddressbookGate expose (including vaguely threatening background shots of mitmproxy doing its dastardly work), and lots of people said nice things on Twitter.

To see the impact all of this for the mitmproxy project, you need only look at the Github page - watchers of the repo went from about 200 a month a go, to 950 at the time of this post.

Changelog

New built-in key/value editor. This lets you interactively edit URL query strings, headers and URL-encoded form data.
Extend script API to allow duplication and replay of flows.
API for easy manipulation of URL-encoded forms and query strings.
Add "D" shortcut in mitmproxy to duplicate a flow.
Reverse proxy mode. In this mode mitmproxy acts as an HTTP server, forwarding all traffic to a specified upstream server.
UI improvements - use Unicode characters to make GUI more compact, improve spacing and layout throughout.
Add support for filtering by HTTP method.
Add the ability to specify an HTTP body size limit.
Move to typed netstrings for serialization format - this makes 0.7 backwards-incompatible with serialized data from 0.6!
Significant improvements in speed and responsiveness of UI.
Many minor bugfixes and improvements.

OpenBSD in decline?

Sun, 26 Feb 2012 09:08:00 GMT

OpenBSD in decline?

26 February 2012

My leisurely Sunday activity today is to set up a new OpenBSD firewall for my mobile app testing lab. I haven't done a from-scratch OpenBSD install for years, so I spent some time reading through the change logs for the last few versions to catch up with what's changed. Although the project is clearly still making steady, well-engineered progress, I had the nagging feeling that the rate of change wasn't what it used to be. So, I pulled some numbers from CVS commit message list archives, and graphed them. Here are the number of commits per month from January 2001 to January 2012. The orange line is a simple 12-month moving average:

Now, we should be cautious about interpreting this - the number of commits doesn't tell us anything about the quality, importance or magnitude of code change. Even if it did all of these things, there are other and perhaps better measures of a project's health. Still, the trend is clear, and suggests a sustained decline in activity.

I just bought some T-shirts to help support one of my favourite open source projects. You should too.

Malware

Thu, 05 Jan 2012 22:50:00 GMT

Malware

05 January 2012

Hover and click for more.

The images above are entropy visualizations of samples from a malware database - black is zero entropy, with colour ranging through blue, up to hot pink for maximum entropy. Large areas of very high entropy are usually sections that are packed - encrypted or obfuscated by the malware authors to make the malware hard to detect and reverse engineer. Smaller areas might be keys, passwords, or other chunks of data meant to be hidden from view.

When you hover over an image, you see a character class visualization with the following colors:

	0x00
	0xFF
	Printable characters
	Everything else

Clicking will show you high-detail versions of both visualizations, and let you look up the binary hash to see what it is. I've used a square Hilbert curve layout - the files start in the top-left corner, and pass through the quadrants clockwise.

I spent hours looking through thousands these visualizations today. I find them eerie and rather beautiful - an entirely different perspective from my day-to-day interactions with malware.

Visualizing entropy in binary files

Wed, 04 Jan 2012 05:26:00 GMT

Visualizing entropy in binary files

04 January 2012

Last week, I wrote about visualizing binary files using space-filling curves, a technique I use when I need to get a quick overview of the broad structure of a file. Today, I'll show you an elaboration of the same basic idea - still based on space-filling curves, but this time using a colour function that measures local entropy.

Before I get to the details, let's quickly talk about the motivation for a visualization like this. We can think of entropy as the degree to which a chunk of data is disordered. If we have a data set where all the elements have the same value, the amount of disorder is nil, and the entropy is zero. If the data set has the maximum amount of heterogeneity (i.e. all possible symbols are represented equally), then we also have the maximum amount of disorder, and thus the maximum amount of entropy. There are two common types of high-entropy data that are of special interest to reverse engineers and penetration testers. The first is compressed data - finding and extracting compressed sections is a common task in many security audits. The second is cryptographic material - which is obviously at the heart of most security work. Here, I'm referring not only to key material and certificates, but also to hashes and actual encrypted data. As I show below, a tool like the one I'm describing today can be highly useful in spotting this type of information.

For this visualization, I use the Shannon entropy measure to calculate byte entropy over a sliding window. This gives us a "local entropy" value for each byte, even though the concept doesn't really apply to single symbols.

With that out of the way, let's look at some pretty pictures.

Visualizing the OSX ksh binary

In my previous post, I used the ksh binary as a guinea pig, and I'll do the same here. On the left is the entropy visualization with colours ranging from black for zero entropy, through shades of blue as entropy increases, to hot pink for maximum entropy. On the right is the Hilbert curve visualization from the last post for comparison - see the post itself for an explanation of the colour scheme. Click for larger versions with much more detail:

Entropy	Byte class

Note that this is a dual-architecture Mach-O file, containing code for both i386 and x86_64. You can see this if you squint somewhat at these images - some broad structures in the file are repeated twice. We can see that there are a number of different sections of the ksh binary that have very high entropy. It's not immediately obvious why a system binary would contain either compressed sections or cryptographic material. As it happens, the explanation in this case is quite interesting. Let's have a closer look:

Sections 1 and 2 are a lovely validation of the central idea of this post. These two areas do indeed contain cryptographic material - in this case, code signing hashes and certificates. Rather satisfyingly, they stand out like a sore thumb. It turns out that all of the official OSX binaries are signed by Apple. This is then used in turn to apply a variety of policies, depending on who the signatory is, and whether they are trusted.

You can dump some rudimentary data about a binary's signature using the codesign command (which you can also use to sign binaries yourself):

> codesign -dvv /bin/ksh 
Executable=/bin/ksh
Identifier=com.apple.ksh
Format=Mach-O universal (i386 x86_64)
CodeDirectory v=20100 size=5662 flags=0x0(none) hashes=278+2 location=embedded
Signature size=4064
Authority=Software Signing
Authority=Apple Code Signing Certification Authority
Authority=Apple Root CA
Info.plist=not bound
Sealed Resources=none
Internal requirements count=1 size=92

Section 3 (the two occurrences are the same data repeated for each architecture) is interesting for a different reason - it's a cautionary example of how the simple entropy measure we're using sometimes detects high entropy in highly structured data. A hex dump of the start of the region looks like this:

000d1f00  00 01 00 00 00 02 00 00  00 06 00 00 00 00 00 00  |................|
000d1f10  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000d1f20  00 01 02 03 04 05 06 07  08 09 0a 0b 0c 0d 0e 0f  |................|
000d1f30  10 11 12 13 14 15 16 17  18 19 1a 1b 1c 1d 1e 1f  |................|
000d1f40  20 21 22 23 24 25 26 27  28 29 2a 2b 2c 2d 2e 2f  | !"#$%&'()*+,-./|
000d1f50  30 31 32 33 34 35 36 37  38 39 3a 3b 3c 3d 3e 3f  |0123456789:;<=>?|
000d1f60  40 41 42 43 44 45 46 47  48 49 4a 4b 4c 4d 4e 4f  |@ABCDEFGHIJKLMNO|
000d1f70  50 51 52 53 54 55 56 57  58 59 5a 5b 5c 5d 5e 5f  |PQRSTUVWXYZ[\]^_|
000d1f80  60 61 62 63 64 65 66 67  68 69 6a 6b 6c 6d 6e 6f  |`abcdefghijklmno|
000d1f90  70 71 72 73 74 75 76 77  78 79 7a 7b 7c 7d 7e 7f  |pqrstuvwxyz{|}~.|
000d1fa0  80 81 82 83 84 85 86 87  88 89 8a 8b 8c 8d 8e 8f  |................|
000d1fb0  90 91 92 93 94 95 96 97  98 99 9a 9b 9c 9d 9e 9f  |................|
000d1fc0  a0 a1 a2 a3 a4 a5 a6 a7  a8 a9 aa ab ac ad ae af  |................|
000d1fd0  b0 b1 b2 b3 b4 b5 b6 b7  b8 b9 ba bb bc bd be bf  |................|
000d1fe0  c0 c1 c2 c3 c4 c5 c6 c7  c8 c9 ca cb cc cd ce cf  |................|
000d1ff0  d0 d1 d2 d3 d4 d5 d6 d7  d8 d9 da db dc dd de df  |................|
000d2000  e0 e1 e2 e3 e4 e5 e6 e7  e8 e9 ea eb ec ed ee ef  |................|
000d2010  f0 f1 f2 f3 f4 f5 f6 f7  f8 f9 fa fb fc fd fe ff  |................|

We see that this section contains each byte value from 0x00 to 0xff in order - furthermore this whole block is repeated with minor variations a number of times. There are two things to explain here - why is this detected as "high entropy" data, and what the heck is it doing in the file?

First, we need to understand that the Shannon entropy measure looks only at the relative occurrence frequencies of individual symbols (in this case, bytes). A chunk of data like the one above therefore looks like it has high entropy, because each symbol occurs once and only once, making the data highly heterogeneous.

Now, what earthly use would chunks of data like this be? With a bit of digging, I found the answer in the ksh source code. These sections are maps used for translation between various character encodings. If you're interested, here's the culprit in all its repetitive glory.

The code

As usual, the code for generating all of the images in this post is up on GitHub. The entropy visualizations were created with binvis, a new addition to scurve, my compendium of code related to space-filling curves.

A personal link mill

Fri, 30 Dec 2011 15:42:00 GMT

A personal link mill

30 December 2011

I posted a link to an interesting visualization paper on Twitter today, prompting someone to ask me where I had found it. Sadly, I had to admit that I had no clue where I first saw it referenced, due to the way I consume links I find on the net. So, I thought I'd write a quick blog post to explain myself, and then pitch a product idea that could make my life (and maybe yours) much easier.

First, the problem statement: my aim is to efficiently discover links to interesting stuff on the net. Simple as that. A few years ago, my flow of links came mostly from social news sites (Hacker News and Reddit), and items shared by people I follow on social networks. Over time, I became more and more disenchanted with this way of doing things. The social news approach is to take a torrent of very low quality links (user submissions), and then crowd-source the filtration process through voting. But popularity is not a good measure of information quality, and the result is a bland, lowest-common-denominator view of the world that has no room for anything that doesn't make it to the front page. Don't get me wrong - Reddit and HN do a lot of other things well - but they just don't cut it as primary information sources. Mining links from social networks is a more promising approach, but still problematic. None of the social networks provide the tools needed to extract shared links from the update stream and consume them efficiently. There is also a structural issue - I don't necessarily want to mix my social ties and my information sources, and I definitely don't want to be limited to just one platform. These are separate functions that I feel require separate tools.

My personal link mill

Eventually, I took matters into my own hands. First, I hugely broadened the number of information sources I consumed. The tool I use for this is Google Reader - I now subscribe to about 800 individual feeds, and this number is growing daily. The trick here is to find high-quality, low-volume link sources. The motherlode of good links for me was to be found on social bookmarking sites. About 700 of my subscriptions are to the RSS feeds of individual users on Pinboard and Delicious. This gives me very fine control and a great mix of interests. Plus, getting links from individual curators handily sidesteps the social news group-think problem. The remainder of my subscriptions are split between blogs, some sub-Reddits, a few Twitter users and subsections of arXiv.

So much for how my intake works. Just as important is the way that I consume it. I do my "filtering" in batches, usually in the evening. Using Reeder on my iPad works well for me, letting me flick quickly and comfortably through all the new links of the day. When I find something that looks interesting, I resist the temptation to read it then and there - instead, I batch up all my reading for later. If it's a web page, it goes to Instapaper. If it's a PDF, it gets downloaded into a DropBox folder, which is synced to GoodReader.

Finally, the actual reading. Every morning, I toddle off to a nice cafe with my iPad, and read all the interesting stuff I saved the previous day in a single sitting. I'm ruthless about just skimming things that don't warrant careful attention. If I find something particularly interesting I save it permanently, and perhaps tweet it or mail it to someone I think might be interested.

Problems - and a product idea?

This system works for me, but it has many problems. There's no end-to-end coordination, so by the time I sit down to actually read something, I have no easy way to tell which feed it came from. Google Reader sucks at managing hundreds of low-volume subscriptions. Reeder is a great, but is not tailored to consuming redundant information from many sources. The end result is that maintaining the system I have is a time-consuming pain in the ass. The fact that it's still worth it despite this, makes me think there might be commercial room for a better solution.

Which brings me to a rough product idea - a formalized version of this link mill for people who want to take direct control of their information intake. The business end is a generalized feed consumer, letting you subscribe to RSS feeds, Twitter users, Google+ updates, sub-Reddits and other information sources. Links are extracted from these feeds, keeping track of which links appeared where. The user is then presented with a stream of links to consume, de-duplicated so that those appearing in multiple feeds are presented only once. The system keeps track of links the user marks as "interesting", batching them for later consumption. It also uses this information to score the feeds, letting the user see which feeds are low quality, and should be ditched. Given the right tools, the time needed for a user to maintain and tend their link feed garden would be quite modest, and the rewards would be great.

If someone built this, I for one would gladly fork over some of my hard-earned doubloons to use it. In fact, with some validation of the idea and a few collaborators I might think of building it myself. Does this sound useful to anyone else?

Visualizing binaries with space-filling curves

Fri, 23 Dec 2011 06:02:00 GMT

Visualizing binaries with space-filling curves

23 December 2011

In my day job I often come across binary files with unknown content. I have a set of standard avenues of attack when I confront such a beast - use "file" to see if it's a known file type, "strings" to see if there's readable text, run some in-house code to extract compressed sections, and, of course, fire up a hex editor to take a direct look. There's something missing in that list, though - I have no way to get a quick view of the overall structure of the file. Using a hex editor for this is not much chop - if the first section of the file looks random (i.e. probably compressed or encrypted), who's to say that there isn't a chunk of non-random information a meg further down? Ideally, we want to do this type of broad pattern-finding by eye, so a visualization seems to be in order.

First, lets begin by picking a colour scheme. We have 256 different byte values, but for a first-pass look at a file, we can compress that down into a few common classes:

	0x00
	0xFF
	Printable characters
	Everything else

This covers the most common padding bytes, nicely highlights strings, and lumps everything else into a miscellaneous bucket. The broad outline of what we need to do next is clear - we sample the file at regular intervals, translate each sampled byte to a colour, and write the corresponding pixel to our image. This brings us to the big question - what's the best way to arrange the pixels? A first stab might be to lay the pixels out row by row, snaking to and fro to make sure each pixel is always adjacent to its predecessor. It turns out, however, that this zig-zag pattern is not very satisfying - small scale features (i.e. features that take up only a few lines) tend to get lost. What we want is a layout that maps our one-dimensional sequence of samples onto the 2-d image, while keeping elements that are close together in one dimension as near as possible to each other in two dimensions. This is called "locality preservation", and the space-filling curves are a family of mathematical constructs that have precisely this property. If you're a regular reader of this blog, you may know that I have an almost unseemly fondness for these critters. So, lets add a couple of space-filling curves to the mix to see how they stack up. The Z-Order curve has found wide practical use in computer science. It's not the best in terms of locality preservation, but it's easy and quick to compute. The Hilbert curve, on the other hand, is (nearly) as good as it gets at locality preservation, but is much more complicated to generate. Here's what our three candidate curves look like - in each case, the traversal starts in the top-left corner:

Zigzag	Z-order	Hilbert

And here they are, visualizing the ksh (Mach-O, dual-architecture) binary distributed with OSX - click for the significantly more spectacular larger versions of the images:

Zigzag	Z-order	Hilbert

The classical Hilbert and Z-Order curves are actually square, so for these visualizations I've unrolled them, stacking four sub-curves on top of each other. To my eye, the Hilbert curve is the clear winner here. Local features are prominent because they are nicely clumped together. The Z-order curve shows some annoying artifacts with contiguous chunks of data sometimes split between two or more visual blocks.

The downside of the space-filling curve visualizations is that we can't look at a feature in the image and tell where, exactly, it can be found in the file. I'm toying with the idea (though not very seriously) of writing an interactive binary file viewer with a space-filling curve navigation pane. This would let the user click on or hover over a patch of structure and see the file offset and the corresponding hex.

More detail

We can get more detail in these images by increasing the granularity of the colour mapping. One way to do this is to use a trick I first concocted to visualize the Hilbert Curve at scale. The basic idea is to use a 3-d Hilbert curve traversal of the RGB colour cube to create a palette of colours. This makes use of the locality-preserving properties of the Hilbert curve to make sure that similar elements have similar colours in the visualization. See the original post for more.

So, here's a Hilbert curve mapping of a binary file, using a Hilbert-order traversal of the RGB cube as a colour palette. Again, click on the image for the much nicer large scale version:

This shows significantly more fine-grained structure, which might be good for a deep dive into a binary. On the other hand, the colours don't map cleanly to distinct byte classes, so the image is harder to interpret. An ideal hex viewer would let you flick between the two palettes for navigation.

The code

As usual, I'm publishing the code for generating all of the images in this post. The binary visualizations were created with binvis, which is a new addition to scurve, my space-filling curve project. The curve diagrams were made with the "drawcurve" utility to be found in the same place.

netograph.com - Realtime privacy snapshots of the social web

Thu, 08 Dec 2011 05:39:00 GMT

netograph.com - Realtime privacy snapshots of the social web

08 December 2011

Today, I'm launching Netograph, a new privacy-related site that I've been hacking on over the past few months. The goal of the project is to provide you with a quick overview of the privacy picture for a URL, before you've clicked on the link. At the moment, Netograph scans Reddit, Hacker News, Pinboard, Delicous and Digg - links on these sites should show up within a few minutes of submission.

For more details, head over to netograph.com. There you will also find Firefox and Chrome browser addons that let you view the Netograph report for a URL instantly with a right-click. Enjoy!

guardian.co.uk

techcrunch.com

reddit.com

What's next?

This is just the first step. As I hinted in a previous post, the most interesting results from Netograph are likely to come from aggregating and cross-correlating the data for individual URLs. I'm already hard at work on this - the next iteration of Netograph will aim to shine some light on the sometimes shadowy network of third-parties that track and analyze nearly every URL we visit. I will also be publishing some interesting tidbits from this data corpus on my blog as I go along, so watch this space.

Otago Polytechnic Talk

Mon, 31 Oct 2011 09:05:00 GMT

Otago Polytechnic Talk

31 October 2011

Further reading for the guest lecture I'm giving at Otago Polytechnic today:

The talk I'm not giving: OWASP Top 10
Tools: FireBug, TamperData, Python.
The Myspace Worm, and Samy Kamkar's own explanation of the exploit.
Halvar Flake's Programming and state machines, which is where I first saw the term "programming the weird machine".

Neighborhoods of trust on the web

Tue, 27 Sep 2011 23:23:00 GMT

Neighborhoods of trust on the web

27 September 2011

For the last fortnight I've been hard at work on a new project that aims to examine trust and security on the web at scale. The basic idea is to use a browser instance to render a URL, and then to extract all persistent state with browser forensic techniques afterwards. This gives you a dump of cookies, cache contents, Flash storage, HTML5 databases, and so on. At the same time, all traffic is routed through a specialised version of mitmproxy, and captured for later analysis. The result is a very detailed snapshot of what viewing a given URL actually does. The next step is to do this "at scale" - this means running many instances of this process in parallel on headless servers, decoupling things using queues, backing it all onto a database, and then spending days and days fine-tuning. I'm happy with my progress so far - my infrastructure is now now scanning all the URLs passing through Hacker News, Reddit, Digg, Delicious and Pinboard in realtime, without breaking a sweat.

I am pretty excited about the possibilities for this project, and I'm exploring plans for the future with like-minded security folk. Get in touch if this interests you, and keep an eye on my blog for more news.

After my pilot run, I had 150 gigs of data covering about 120 thousand URLs. Below is a quick peek at one tiny slice of this data - an appetizer for things to come.

Neighborhoods of trust

This graph shows structures that emerge from the way sites use third-party executable resources. In this context, "executable" means means JavaScript, Flash and HTML, and "third-party" means domains other than the URL's own. The nodes in this graph are the third-party domains, and the edges are associations between them via the URLs I crawled. For example, if a site loaded scripts from both Google Analytics and from Doubleclick, that would create (or reinforce) an edge between the nodes "google-analytics.com" and "doubleclick.com". Using this data, I calculated a co-occurrence coefficient for the third-party sources, and then extracted the resulting neighbourhood structures algorithmically. The neighbourhood information was used to colour and lay out the graph, trying to keep nodes that are closely correlated together. Finally, nodes are scaled based on how many URLs reference them.

The result is a rather stunning graph showing neighborhoods of trust - areas of the Internet bound together based on the third parties allowed to run code in users' browsers. I've spent a few hours playing with this data, and the sheer range of interesting structure is surprising. At one end of the spectrum, you can zoom in to the individual node relationships, and find small clusters of surprising sites that cross-load resources from each other, often because they are owned by the same entity. At the other end, countries, language groups, and broad fields of interest aggregate in huge tribes of kinship.

Here are a few of the larger-scale features from the graph:

	Mainstream The most widely used resources dominate in the neighbourhood extraction algorithm, which causes them to cluster together in their own super-community. The top nodes in this cluster, descending order of occurrence are: google-analytics.com, facebook.com, doubleclick.net, fbcdn.net, quantserve.com, twitter.com, google.com, googlesyndication.com, googleapis.com, scorecardresearch.net, facebook.net, addthis.com. These are also the top nodes overall.
	Japanese The main resources are hatena.ne.jp, microad.jp, mixi.jp, yahoo.co.jp, nakanohito.jp. More surprisingly, also in this cluster are topsy.com, appspot.com and postrank.com. Perhaps these resources are especially commonly used on Japanese sites.
	Russian Top resources are yadro.ru, yandex.ru, rambler.ru, vkontakte.ru, openstat.net, userapi.com, shinystat.net, and dt00.net
	Porn And here we have a portion of the web dedicated to pron. The top resources are awempire.com, clickbank.net, picadmedia.com, getresponse.com, adultfriendfinder.com, adultadword.com, phcdn.com, juicyads.com, brazzers.com, etology.com, data-ero-advertising.com and viddler.com. A more surprising inclusion in this group is wufoo.com - I wonder if this is an artifact, or whether Wufoo really does have a use in the adult content world.
	Misc Just to show that it's not all clear-cut, here's an example of a neighbourhood I find harder to explain. The top resources are netdna-cdn.com, amgdgt.com, trafficmp.com, ooyala.com, suitesmart.com, demdex.net, adfrontiers.com, lycos.com and break.com. I speculate that this group might be loosely aligned around a number of big CDNs and analysis suites.

The graph in this post was created, analyzed and pre-processed using graph-tool, a great Python library for dealing with large graphs. The visualization and modularity analysis was done using the ever-wonderful Gephi. If these aren't both in your arsenal of analysis tools, you're missing out.

Why the Apple UDID had to die

Fri, 09 Sep 2011 20:22:00 GMT

Why the Apple UDID had to die

09 September 2011

EDIT: A WSJ Digits article is now up, containing a responses from Zynga and Chillingo. Other networks declined to comment.

A UDID is a "Unique Device Identifier" - you can think of it as a serial number burned permanently into every iPhone, iPad and iPod Touch. Any installed app can access the UDID without requiring the user's knowledge or consent. We know that UDIDs are very widely used - in a sample of 94 apps I tested, 74% silently sent the UDID to one or more servers on the Internet, often without encryption. This means that UDIDs are not secret values - if you use an Apple device regularly, it's certain that your UDID has found its way into scores of databases you're entirely unaware of. Developers often assume UDIDs are anonymous values, and routinely use them to aggregate detailed and sensitive user behavioural information. One example is Flurry, a mobile analytics firm used by 15% of apps I tested, which can monitor application startup, shutdown, scores achieved, and a host of other application-specific events, all linked to the user's UDID. I recently showed that it was possible to use OpenFeint, a large mobile social gaming network, to de-anonymize UDIDs, linking them to usernames, email addresses, GPS locations, and even Facebook profiles.

This post looks at the way UDIDs are used in the broader social gaming ecosystem. The work is based on a simple question: what happens if we swap our UDID for another while communicating with the network? There are a number of ways to do this - in my case I used mitmproxy, an intercepting HTTP/S proxy I developed which lets me re-write the traffic leaving a device on the fly. In most cases this was a simple matter of replacing one string with another, but two networks (Scoreloop and Crystal) prevented UDID substitution using cryptography. Unfortunately, both networks relied on the secrecy of key material distributed in the application binaries to every device. I have verified that it is possible to reverse engineer the application binaries to extract the key material and circumvent the cryptographic protection.

The outcome of this experiment shows that social gaming networks systematically misuse UDIDs, resulting in serious privacy breaches for their users. All the networks I tested allowed UDIDs to be linked to potentially identifying user information, ranging from usernames to email addresses, friends lists and private messages. Furthermore, 5 of the 7 networks allow an attacker to log in as a user using only their UDID, giving the attacker complete control of the user's account. Two networks had further problems that compromised a user's Facebook and Twitter accounts - Crystal lets an attacker take control of a user accounts by leaking API keys, while Scoreloop partially discloses users' friends lists, even if they are private.

	Data leaked	Log in as user	Social Media Accounts
Crystal	Username, friends, Facebook, Twitter, games played, location, email address	Yes	Control of Facebook, Twitter accounts
GameLoft	Username, email address, games played, nationality, friends	Yes	No
Geocade	Username, email address, games played, location	Yes	No
OpenFeint	Username, last played game, online status, friends	Yes	No
Scoreloop	Email address, gender, username, nationality, friends	Yes	Access private Facebook and Twitter friends lists
Plus+	Username	No	No
Zynga	First name, username, friends, in-game messages, mobile number*	Yes*	No

* The starred Zynga findings rely on the fact that other networks can be used to obtain the user's email address using the UDID.

There are two caveats to keep in mind while considering these results. First, the findings are based on the default settings for each social network - some networks may have settings that reduce the amount of information exposed. Second, some of the data leaked is optional - for instance, it's not mandatory for a user to link Facebook or Twitter accounts with any of the networks.

All the affected companies and Apple were notified 5 weeks ago. The Crystal and Scoreloop teams have both repaired the problems that could lead to a follow-on compromise of a user's social network accounts. At the time of writing, it is still possible to log in as a user using only a UDID on five of the vulnerable networks.

The future

A few days after I notified the companies involved, it was revealed that Apple was quietly killing the UDID API. It will still be present in IOS5, but is marked deprecated, and will probably be removed in future. I recommend that developers shift away from using UDIDs now, rather than wait for formal removal of the API.

We can now expect a frenzy of activity as developers look for alternatives. The challenge will be to make sure that the cure isn't as bad as the disease - Apple's recommendation to "create a unique identifier specific to your app" could tempt developers to replicate the UDID mechanism on a smaller scale, flaws and all. Expect more blog posts on this topic soon.

cortesi

mitmproxy 0.9.1

Changelog

Skout: a devastating privacy vulnerability

How mitmproxy works

Explicit HTTP

Explicit HTTPS

The MITM in mitmproxy

Complication 1: What's the remote hostname?

Complication 2: Subject Alternative Name

Complication 3: Server Name Indication

Putting it all together

Transparent HTTP

Transparent HTTPS

pathod 0.9

mitmproxy 0.9

Changelog

Google, destroyer of ecosystems

Things I found on GitHub: aspell custom dictionary entries

Things I found on GitHub: pipe chains

Things I found on GitHub: shell history

ghrabber - grab files from GitHub

Shell history files

The trouble with social news

Karma Corrupts

The Lossy Sieve

The Pinhole Effect

Gaming The System

Community Collapse Disorder

A better way forward?

Go: a nice language with an annoying personality

Released: pathod 0.3

Better fuzzing

Language improvements

pathoc: break all the Python webservers!

Basic fuzzing with pathoc

Results

Limits of data visualization with space filling curves

Findng the UDID leak: a guessing game

The UDID leak is a privacy catastrophe

Defiler

pathod 0.2: the daemon gets an evil twin

Introducing pathod: a pathological HTTP server

The simplest possible response

Specifying features

Pauses and Disconnects

What's next?

mitmproxy 0.8

Android interception

Replacement patterns

Improved pretty-printing of request and response contents

Changelog

mitmproxy 0.7

Changelog

OpenBSD in decline?

Malware

If you subscribe to my RSS feed, please visit this article directly. The table below has interactive elements that won't work in most feed readers.

Visualizing entropy in binary files

Visualizing the OSX ksh binary

The code

A personal link mill

My personal link mill

Problems - and a product idea?

Visualizing binaries with space-filling curves

More detail

The code

netograph.com - Realtime privacy snapshots of the social web

What's next?

Otago Polytechnic Talk

Neighborhoods of trust on the web

Neighborhoods of trust

Mainstream

Japanese

Russian

Porn

Misc

Why the Apple UDID had to die

The future