Aehso’s Output

Taking Personal Data Out Of Social Networks?

As we’ve seen recently screen scraping Facebook pages violates their Facebook Terms Of Use.  Dare suggests that the Facebook Platform APIs can be used to get some (but not all) of a users data but I think he’s forgetting about the conditions governing Storable Information which does not permit storing friends IDs (amongst other things).  Also, in order to use the Facebook Platform in the first place, developers have to agree to the Developer Terms Of Service which clearly indicate that any data gathered (above and beyond that defined as Storable Information) while using the Facebook APIs can only be stored for 24 hours (section 2.A.4).  The TOS definition of a data repository is fairly all-encompassing:

any spreadsheet, database, physical document, server, network, or other repository of information, whether centralized or distributed.

Pretty all-encompassing eh!  I’ll bet there are more than a few facebook applications that are actively breaking this term of service. (Aside: The 24 hour restriction can be avoided if, and only if, the application explicitly asks the user to opt-in - see section 2.A.6. I wonder does the OutSync tool that Dare uses do that?) 

I am of course picking bones here - I could go on but enough said about the minutiae of Facebook legal mumbo-jumbo.  A much bigger and much more important question is. How did we end up in the situation whereby we need to take personal data out of social networks?  The answer of course is that we allow multiple web services and social networks to indefinitely store overlapping subsets of our personal data as they see fit.

Let me put it another way - what would happen if we inverted the location of your personal data?  What if social networks had to (periodically) contact your identity provider to get your personal information and social graph? Then this type of problem would not exist and everyone would have far greater data and service portability. 

However, there are several large barriers to this happening:

• We don’t yet have an established global identity scheme for storing the critical personal and social graph information that social network websites need to operate.  OpenID and OAuth provide the low level plumbing for such a scheme but a  higher level standardized portable personal information protocol is required to allow 3rd parties to find out more about a user with an OpenID.
• Assuming the above existed, it would be impossibly difficult for 99% of the internet users to manage/use/understand unless it (their identity service) was managed on their behalf by the organization their work for or their broadband provider.  I was going to initially say ‘was built into their OS’ but nowadays people use multiple computers that have no fixed public internet address so that’s not even close to an option.
• No large social network will ever willingly volunteer to support this.  Legislation/Regulation will be required to force the existing social networks to evolve onto this identity model.

The last point is probably the biggest barrier and is likely the reason why no big player is expending significant effort to developing standards for user owned identity profiles.  Given the relative lack of voice that average internet users, or even groups of users, now have (Scoble aside) legislation and/or regulation is IMHO the only way to compel the incumbents to change how the whole social network operates.

HitWise : Social Network traffic surpasses Webmail traffic in UK.

Assuming the stats behind these graphs are accurate, the UK appears to have passed yet another tipping point in the relentless growth of social network websites.

• Social networks accounted for 5.17% of all UK Internet visits, compared to 4.98% for webmail services:
  
• Social networks referred more traffic to other websites than webmail services:
  

Email is truly dying a slow and painful death. (Yep, I know these figures are not including non-webmail email but…)

(Attribution: all graphs and stats are from Hitwise)

OpenSocial : Critiques

Three recent posts on OpenSocial that I’ve come across that all touch on points I raised in my last two posts that are worth sharing:

• Terms (Shelley Powers) - comments on the very important issue of the terms and conditions attached to usage of the OpenSocial APIs - I had completely overlooked this. T&C are normally attached to proprietary products, not open standards. So it looks like we are really looking at Google APIs, not open standards here.
• Where the hell is the Container API? (Russell Beattie). Short, to the point, and bang on the money. I’m not in a rush so I’m happy to wait for important web API specifications to be drafted, discussed, refined, voted upon and published via a credible authority. But the OpenSocial development process is not open (a newsgroup of pleading users does not make it so).
• Google OpenSocial: Technical Overview and Critique - Dare Obasanjo. Too much to summarize here, go read it.

All in all I’m beginning to think the use of the term Open in OpenSocial is terribly misleading marketing speak. I’d like to think this is an accident (after all Google care about the continued growth of an open internet right?) but there is such a monstrous gap in the process behind and the function of OpenSocial and how other open APIs and standards are developed that I can only assume that this is all a marketing exercise to misdirect attention away from Facebook at the cost of ushering out a half-baked alternative. The Campfire One video only re-enforces this - it is incredibly corny and lacking in substance!

APIs and standards are sometimes hacked together by partnerships in order to try address immediate market share concerns - this is beginning to look like yet another of these efforts. Hopefully I’ll be proven wrong when the Container API documentation is published but I’m not holding my breath…

OpenSocial Doc Review Part 1 : Data APIs are all AtomPub based.

Starting to read through the OpenSocial API documentation the most striking feature is that ALL of the underlying data APIs are Atom and AtomPub based.

• The People Data API exposes people (viewer, owner, friends) via Atom feed documents and Atom entry documents. However, there is no AtomPub support for this feed so people cannot be created/updated/deleted via API. I suspect this will cause some debate as it means some fairly critical data remains locked into the underlying OpenSocial containers
• The Activities Data API exposes activities (news feeds etc) as a Atom feeds. The Activities data feed can be fully manipulated (CRUD) via Atom/AtomPub requests.
• The Persistence Data API exposes name/value pair data (e.g. configuration params) as a Atom feeds. The persistence data feed can be fully manipulated (CRUD) via Atom/AtomPub requests.

I’m off for a curry but more later today. Here’s a teaser though - authentication is a concern….

Open Social - Disruptive?

Update: Marc Andreessen has a great writeup on Open Social. Spin warning: Ning is a launch partner of Open Social so take it with a little pinch of salt.

Update2: Bebo and MySpace are also joining. That leaves just Microsoft and Facebook out in the cold.

No wonder Google didn’t invest in Facebook, they were busy setting up OpenSocial (site goes live tomorrow) with a who’s who of companies that are not Microsoft, Facebook or MySpace.

It’s an admirable and ambitious project and it kicks Facebook right in the FBML-tender-spot but as ever the devil will be in the detail. There seems to be some concerns about how truly portable applications would be but I’m far more interested in portability of user data. I’m looking forward to reviewing the API details when they are published, hopefully they will also incorporate some of the emerging de-facto standards like OpenID and OAuth.

Facebook’s next major move will be interesting. Some of their key application developers (Rock You, Slide) will of course divert resources to work on OpenSocial based applications.

Alex Iskold on Read Write Web.

I completely agree with Brad and Fred - Alex Iskold’s The Structured Web and Facebook: What If More Is Less? contain some great concise analysis of the how the web and the networks within it are evolving.

Well worth a read if you’re looking for some food for thought and can spare 10 minutes….

Brad Fitzpatrick’s Social Graph Problem.

Ronan pointed me to an interesting Wired blog entry on Brad Fitzpatrick’s social graph problem. The social graph problem requires many of solutions that Dion’s Dream River needs so I was going to just comment that it will take a long time for any of this to happen (thankfully Brads essay openly calls this out).

Then I read that Brad might be joining Google. So I’ve changed my comment to “it might happen just a little bit sooner”.  This is just the type of problem that the folks at Google love to try solve…

Dion’s Dream River

Now that I’ve been using Facebook, LinkedIn and Twitter for a while I’m beginning to notice fragmentation of my contact network and event streams.  The problem goes a little deeper than “should I twitter this or/and update my FB status?” or “should I add her as a friend in Facebook or LinkedIn?”. 

Then there is also network replication to deal with - I’m getting multiple friend invites for the same people, many of whom are already in my (private) contacts list.  Don’t get me wrong, I like connecting with people!  I’m just finding myself wishing that there was a better mechanism through which all this stuff can be synchronized though.  It turns out Dion Almer is finding himself dreaming up a solution he calls The River:

In the dream I developed an application called “The River” that was a filthy rich UI that showed your one main river of events, and various rivers that would run into the main one. The river would contain: Incoming email, Status events from fb, twitter, pownce, Cool.Next, Event invites, Blog entries via rss, Photos uploaded, etc.

Dion adds a nice kicker when he suggests the following feature:

When a user signs up, they are asked for their FB/gmail/linkedin id to find people / invite them. Most services do that right now. However, there is an extra check box which states “if you see me connected on another network, automatically connect me here”. This means that you don’t have to keep accepting friend invites from the same people again and again. You should virally, quite quickly, get all of your contacts sync’d up.

I think this is an excellent suggestion but I’d take it one step further.  The River needs to do this continuously, effectively acting as an intelligent agent.  Unfortunately there are two major barriers to this ever happening:

• The agent needs to be able to verify user identities.  For example it needs to be sure that the John O’Shea in your network is me, not the footballer, the filmmaker or the humanitarian.  I can and do play football, shoot home movies and donate to charities but I’m not those people.  One problem here is none of the major social networks support associating an OpenID with a user profile so by consequence none have the required APIs.  A second problem is not many people have OpenIDs yet (that reminds me…)
• The agent needs read/write  to the users contacts and event streams within each social network.  Read can be done via RSS feeds, but write needs HTTP API access.  Of the networks I mentioned above, Facebook(api) and Twitter(api) provide this LinkedIn doesn’t (yet - they are working on APIs though).  The walled garden social network sites like MySpace and Bebo really won’t want to ever provide this.

If OpenID gains more momentum then I think all of the above is do-able.  I am of course ignoring the walled garden networks but over time they will seem more and more antiquated to their users and their influence will diminish.

This is, of course, all an extension of the older problem of synchronizing contact information between multiple email accounts and mobile phones and that nut still isn’t completely cracked.  iSync solved this to an extent but it (still) doesn’t have any decent Google mail/calendar integration. (I’ve tried ABGMerge but it created a mess to the extent that I had to revert to backups. This is becoming a real pain for me as I ditched my .mac subscription last year and I’m mainly using gmail now - suggestions welcome!)

Meanwhile something is going to have to give, and I think the first service I’ll drop is Twitter…