This is a quick writeup of an application of RDF query tools to the problems of weblog discovery and filtering. It shows off RDF's ability to merge XML-encoded data from multiple sources, answering queries which couldn't be answered by considering each source separately.

Imagine you are looking for Weblog URLs (and perhaps their associated RSS feeds) of people who have written Perl software that is in CPAN. At our disposal are two data sources: a data dump listing the SHA1 hashes of CPAN author email addresses. For each author, we have an @cpan address, plus one of their normal addresses (cpan2foaf.pl). A second RDF/XML source of FOAF data is a document listing other information about several people who happen to write for CPAN. That file is a stand-in for their real FOAF files, which would typically be collected from each of their sites individually. For simplicity, we show that data as a single document.

The first datasource mentions several hundred people who contribute to CPAN, and enough information to identify them (by hash of email address). The second datasource contains some of the same identifiers (their hashed home email address) plus, more interestingly, information about their names, homepages, weblogs and suchlike. How can we get answers to questions like "find me weblog addresses for CPAN contributors?".

So... we load the RDF up into an (SQL-backed) RDF database, make sure to merge the data (more on which another time), and it is available for query.

This RDF query language is called Squish as it is loosly SQL-ish in the way it is used. We send the database a question asking for a selection of fields, and we get back a table of results. Here are name/homepage/weblog 'hits' corresponding to the rows returned by the query system:

Here is a simple Ruby lookup script which extracts the tabular result set from the database. The query extracts information (in this case weblog address etc) about people who contribute to CPAN and are known to the database.

I used the dc:contributor property incorrectly in this demo. It should relate a document to one of its contributors. I used it backwards, and slightly incorrectly, by relating a person to a homepage of a project they contribute to. I should use foaf:project instead.

The demo links are also a bit inconsistent, in that I added in a clause to the query which also asks for the RSS feed of a weblog, but didn't commit the new resultset output to the website yet. I'll try to get that fixed.

This brief writeup assumes perhaps a bit too much background knowledge for those new to RDF to follow in detail, also things like SHA1 'hashing' could do with elaboration. Still I hope the basic approach is clear: RDF allows us to merge disparate datasources and get back answers that depend on their combination. And that we can use that for filtering and discovery of weblogs, given a description of some weblogs and their authors.

Spring provides a canvas-style desktop interface, replacing the ageing 'documents and folders' metaphor with a UI based on several content types (people, places, products, books, music, ...), with appropriate kinds of action associated with each type of thing. So you might invite a person to an event, draw a map of a location, etc.

This support is improved in v1.3.1: a "Refesh From Web" feature allowing FOAF-derived person entries in Spring to pick up the changes from remote FOAF files.

So needless to say, I've been experimenting with Spring on my Mac laptop, and it looks pretty interesting and a hint at what desktops might evolve into. I really like the way it associates actions with entity-types in a way that takes advantage of remote Web services. That, coupled with the "refresh from Web" approach to using remote RDF/XML descriptions (FOAF etc.) is quite a compelling combination.

So in practice, I found the FOAF support worked with some but not all FOAF files from the public FOAFBulletinBoard. I'm not sure what the issue is, perhaps something to do with the mime-types these documents are served up as, or perhaps Spring has certain expectations for the syntactic form of a FOAF document. If others are trying out the FOAF support, perhaps leave comments here or at the UserCreations article, perhaps we can figure out how to help.

Ken MacLeod has announced a new tool, 'FOAF Check', which reads a FOAF file, checks its PGP signature and extracts portions of the RDF, eg. for inclusion in weblog comments (via. Movable Type cookies etc.).

This approach provides a way to avoid re-entering the same data over and again across multiple sites - you just point them at your FOAF document and let them read what you've already said about yourself. Ken's writeup points to previous work on FOAF/PGP-based decentralised profiles.

This is really cool, basically. Looking at the detail it does suggest we need to do a little more work on the FOAF vocabulary for clarifying which of several people mentioned in any given FOAF file is the topic of that file. There are currently several ways we could indicate this; one needs to be picked and documented more clearly in the FOAF spec. It also isn't immediately clear (to me!) how we get from knowing that a FOAF file validates against some PGP key to knowing that the PGP identity is the one that our application wants it to be. But it's a step in the right direction...

Also the combination of foafbot and tools like foaf-check provide good reason for people to actually start using PGP (or GNU Privacy Guard (GPG), the Free GNU version), That can only be a good thing.

One other topic this throws up: is it reasonable to expect people to remember and quote their FOAF URLs? Or should we ask for their homepage, and look for a LINK REL in their homepage markup pointing to their FOAF data? ----- EXCERPT: ----- KEYWORDS: ----- COMMENT: AUTHOR: Dan Brickley EMAIL: danbri@rdfweb.org IP: 10.0.2.16 URL: http://rdfweb.org/people/danbri/ DATE: 06/24/2003 09:11:51 PM See also Ken's notes in the FOAF wiki, http://rdfweb.org/topic/FoafIdentityAssurance for more details on how this might be used for a challenge/response style of authentication. Nearby, FOAF's wiki homepage (recently tidied up somewhat), http://rdfweb.org/topic/FoafProject ----- -------- AUTHOR: danbri TITLE: New version of XML::FOAF in CPAN STATUS: Publish ALLOW COMMENTS: 2 CONVERT BREAKS: 0 ALLOW PINGS: 0 PRIMARY CATEGORY: Programming CATEGORY: Programming DATE: 06/25/2003 11:22:11 AM ----- BODY:

Ben Trott has uploaded version 0.02 of XML::FOAF to CPAN. This is a Perl module designed to make it easier for programmers to work with FOAF documents. It builds upon the RDF::Core module from Ginger Alliance and can use Crypt::OpenPGP to check digitally signed FOAF files. Version 0.02 of XML::FOAF adds autodiscovery support. The documentation shows how to load FOAF from a URI, which can either be a directly named FOAF file, or an HTML page which references a FOAF file using the following markup:

Anywhere there is an element describing a Person (or for that matter a company, group etc), you can include a sub-element that mentions their weblog(s):

It doesn't matter exactly where you put the foaf:weblog entry, so long as it is immediately 'inside' the foaf:Person element. You can have the foaf:homepage bit first, or foaf:img, foaf:workplaceHomepage etc. there too., all alongside each other in any order.

This same technique works if you are describing your friends and collaborators in a FOAF file; just add in a foaf:weblog property inside the foaf:Person section that describes them.

That's all there is to it. Your FOAF file now describes the address of your weblog, making it easier to find for FOAF and other Semantic Web tools.

Note that If you have multiple weblogs, list them each separately, one after another. Note also that each foaf:weblog described in FOAF has to be the weblog of one 'thing' (whether Person, Company or whatever). So describing collaboratively edited weblogs is a topic I'll come back to in a future article.

The FOAF autocreation script takes two links - one to your current FOAF file and one to your OPML blogroll - and produces a brand new FOAF file for you by attempting FOAF autodiscovery on every site your OPML file mentions. See the autofoaf page for more details on this tool.

So this is yet more reason to take a few seconds to make an auto-discovery link to your FOAF file in from the <head> of your Web page, to give software tools a hint about how to find it. This auto-discovery syntax is detailed in the FOAF spec, but here it is once more as a reminder:

...where "foaf.rdf" is a relative link to your FOAF file, with the name changed appropriately.

The upcoming TypePad weblog hosting service from Six Apart also makes use of this auto-discovery markup, as does the XML::FOAF Perl software library, so it is increasingly worthwhile adding in the FOAF autodiscovery link.

The basic idea for FOAF auto-discovery is to remove the need to remember (and type in) a separate URL for your FOAF file. Instead of remembering both your homepage and FOAF address, just know the former and make sure it references your FOAF file in a way that software tools can follow. Less work for humans, more work for machines, just as it should be.

Actually it was published back in March, but I missed the chance to write about it then. Better late than never. The article introduces the basic concepts of the Semantic Web and RDF using their FOAF browser, built using Java and PHP. The article goes on to talk about the potential business value of such work, giving a brief case study of the Ecademy networking site.

The other publications on SemaView's site are worth a read too. They've made an effort to provide a friendly overview of Semantic Web technologies, and to provide a business-oriented perspective as well as a technical one.

Edd Dumbill writes with news of his latest work on FOAFbot, the chat-based FOAF aggregator.

FOAFbot is now based on the 'twisted' framework, opening up possibilities for various new interfaces to the data beyond the current IRC one. This is interesting as it makes it easier for people to adapt FOAFbot, eg. for instance messenger or HTTP interfaces, or to hook it up to alternate front-ends such as foafnaut. Edd's weblog provides more details on all this, including links to source code and a teaser screenshot of FOAFbot data showing up in Gnome Dashboard. Nice work! :)

There is plenty of work to do on the new site, but now seems as good a time as any to cut across to the new version. The old FOAF homepage had a (somewhat random) collection of links and information on FOAF. A version of that is included in this 'getting started' article; for more comprehensive links to FOAF-related materials, see the collaboratively maintained FoafProject wiki site.

The current www.foaf-project.org site is pretty minimalistic. Nicole Sullivan has been working on some ideas for a more complete FOAF site. One of the things we need to figure out before doing that is a sensible balance between fairly accessible end-user materials (www.foaf-project.org) and the more obscure technical and developer content familiar from rdfweb.org.

The basic recipe for getting started with FOAF is as follows. First, you'll need a basic idea of what FOAF is, and is for. Short version: FOAF is all about creating and using machine-readable homepages that describe people, the links between them and the things they create and do.

For a more in depth version, the evolving FOAF Specification should be the authoritative reference, accompanied by less technical documents such as Edd Dumbill's FOAF intro article. The FOAF FAQ has links to some other media articles, including 'Metadata Mark II' in Web Monkey Magazine which describes FOAF alongside some related efforts.

If you're still interested, the best thing to do next is to make yourself a FOAF file. For this, most people use Leigh Dodd's handy foaf-a-matic tool, which is now available in eight (8!) languages. So make yourself a FOAF file, save it online somewhere, and add an auto-discovery link to it from your homepage so that machines have an easy way to find it. If you mess about with the FOAF markup, eg. to add your own extensions, the W3C RDF Validator is a useful tool for checking the file is still in legal RDF/XML format.

Then what? Well hopefully your data will start to show up in the various FOAF navigators and aggregators, some links to which are included in this article...

Here are a few useful links relating to FOAF: FOAFNaut (and SVG-based navigator), FOAF Explorer (server-based navigator using XSLT bookmarklet), eikeon.com FOAF Web View (another server-based navigator, written in Python), JabFOAF (utilities for integrating FOAF and Jabber).

Some more Links and background... this list is not complete, but repeated here so I can reference it from the new more minimalist FOAF site.

or "Friend Of A Friend" - The idea is that everyone puts a page on their site describing who they know and where you can find them (and yourselves). A nice idea. I got most of the information from this

A few people have asked how we can use FOAF to express that a page is about a person, or whether there is a way within FOAF to write testimonials about other people (presumably along the lines of various popular dating sites).

The former is easy; the latter isn't something the FOAF vocabulary currently supports. Not because it would have been hard to add, but because it would have opened a can of worms that is best left to ripen a little while longer.

But FOAF does have a way of describing the topics of pages, which is a way of providing similar functionality.

So, let's imagine you've written a whole HTML page all about one of your bestest friends, and you want to share that fact with your world by adding something to your FOAF file. How to do that?

The basic idea is one that recurs throughout FOAF. We use the foaf:topic property, which is a relationship between a Document and something that the document is about. In your FOAF file, you can mention that you know someone, and at same time mention that there's a page you made that's about them. Here's some example markup:


<!-- imagine you're Alice and this is the markup in your FOAF about you -->
<foaf:Person>
  <foaf:nick>Alice</foaf:nick>  
  <foaf:knows rdf:nodeID="bob"/> <!-- a pointer to the 'bob' entry below -->
  <foaf:made rdf:resource=""/>       <!-- this says: I made this FOAF file -->
  <foaf:made rdf:resource="http://alice.example.com/why-bob-is-great.html"/>       
</foaf:Person>

<!-- some markup describing a document, and its topic -->
<foaf:Document rdf:about="http://alice.example.com/why-bob-is-great.html">
  <dc:title>A page all about Bob, by Alice</dc:title>
  <foaf:topic>  
    <!-- here is a chunk of data about bob -->
    <foaf:Person rdf:nodeID="bob">
      <foaf:nick>Bob</foaf:nick> 
      <!-- other stuff about bob could go here -->
    </foaf:Person>
  </foaf:topic>
</foaf:Document>

What this basically says is "There is a Person whose nick is 'Alice' and who made this current FOAF document and who knows a Person whose nick is 'Bob'; there also is a Document whose title is 'A page all about Bob, by Alice' that Alice made and that is about Bob."

So one question facing us in the FOAF design is finding a trade-off between simplicity and expressiveness. It is possible to use FOAF to say some fairly complex things, but we have our work cut out to improve the tools and tutorials that explain how to create FOAF files that say what we want them to say.

Here's a diagram of the above example markup, generated by the W3C RDF validator. It may be useful for illustrating the underlying pattern of relationships we're trying to describe in the FOAF markup...

In brief, foaf:knows was designed to be relatively harmless. It's use (and non-use) should be inoffensive. We dropped foaf:friend and foaf:knowsWell a year or two ago as they proved undeployably awkward, both when used and more interestingly, when not used.

Things are heating up in the battle for the hearts, minds and contact lists of introverted Web geeks. While an Introvertster craze sweeps through workplaces and homes, FOAF's under-used foaf:myersBriggs support seems to be slowly catching on, revealing a disturbing trend. According to members of the Myers Briggs personality classification cult, individuals branded as IN** (ie INTJ, INFJ,INTP,INFP) should number no more than about 4 or 5% of the population. Yet a recent sampling by RDFWeb's FOAF scutter shows an alarming outburst of introversion amongst FOAF early adopters...

While I generally find MBTI a bit contentious, including foaf:myersBriggs has been an interesting exercise and even from the rather small dataset we have currently, there appear to be a distinct leaning towards the IN** side of things.

Dusting off my sociologists hat, there are various hypotheses we could investigate. Maybe FOAF's current audience is mainly geeky early adopters, people who show up as IN** when they go through the annoying-but-probing online questionnaires which try to approximate a full (tm)-compatible MBTI analysis. Maybe the character summaries for INTP are more flattering and ego-stroking than for other 'types' (or at least, more flattering to the kinds of people who show up as INTP in the surveys). Or at least, more likely to evoke a 'heheh, yeah that is sorta like me' reaction. Maybe INTP's like FOAF as a non-extraverted way of describing what they're like, and perhaps this extends to attaching a 'beware I'm an INTP / introverted freak, don't expect me to talk to you' health warning to their machine-readable homepage. Maybe INTP-ish people are the only ones obsessive and nit-picky enough to deal with RDF and XML syntax, while other types have malformed FOAF files which don't get past the RDF parser? Maybe the survey sample is too statistically insignificant to tell us anything.

The results mentioned above come from an incomplete (but 24hrs+) run of my FOAF harvester ('scutter') from a couple days ago, ie. are based on a bunch of source-attributed RDF statements stored in a PostgreSQL database. I'll give the code example first, then the results, summarising all the foaf:myersBriggs properties that the database knows about. It uses my rough-n-ready RubyRdf library, and might some day be nicely packaged for others to use. In the meantime, here's the rough idea...

Not much more to add really, thought this might be a fun mini-tutorial on RDF query. But I would be interested in pointers to other statistical analysis of MBTI data. In particular I reckon it would be worthwhile to feed the raw answers to the online quizes to statisticians, to see if the 16 types proposed by MBTI-advocates actually reflect clusterings within the data...

The FOAF specification now includes a new (and long anticipated) property, foaf:gender. Like all FOAF properties, 'gender' is optional - nobody is required to describe themselves in any more detail than they choose to, and the new text in the spec goes to some lengths to stress this, amongst other concerns. I announced it to the FOAF mailing list (rdfweb-dev) earlier today following editorial discussions in #foaf, our IRC channel.

Typical values for foaf:gender are 'female' and 'male', which can be added to most FOAF files simply by including an element such as <foaf:gender>male</foaf:gender> inside any foaf:Person element(s) in the document. See the foaf:gender definition for details and accompanying documentation. Feedback on the design and documentation is welcome, either through weblog comments or by email.

The addition of a 'gender' property to FOAF is obvious, but also not something to do lightly. I have tried in the documentation to highlight a few issues that users and developers should be aware of. FOAF is often compared to centralised databases such as Six Degrees (the old pre-crash version; the domain name seems to have been recycled) or Friendster. However, the FOAF "data set" (ie. the collection of all Web documents and services which use FOAF vocabulary) is highly distributed, ie. not under central control. FOAF is being put to uses that are literally out of control; there is no "FOAF sysadmin" who can delete, edit or update all FOAF profiles. The data is scattered throughout the Web, under the control of the people who publish it.

Consequently we need to proceed with a certain amount of caution when increasing the expressiveness of FOAF. Adding gender information to FOAF means that whenever someone is looking for a way to describe a person's gender in an RDF/XML document, there is a simple piece of FOAF markup that will do the job. We can't control what other information this will be mixed with, whether the information is accurate, or the uses to which it will be put. An obvious application people have been discussing is 'dating', and doubtless we'll see additional vocabularies to accompany FOAF which will go into more detail on that topic than FOAF itself will ever attempt. Other application areas include the (again long-awaited) treatment of 'family tree' data within FOAF, where foaf:gender could be used to help define other concepts, like 'brother', 'aunt', etc.

It's also worth noting that this addition to FOAF has been defined in general terms so that it is very widely applicable. We can talk about the foaf:gender of things that aren't human, eg. other animals. This is a good example of unexpected re-use, something often seen with RDF-based data formats such as FOAF. By adding the foaf:gender property, and combining FOAF with other vocabulary such as Wordnet, we could for example use FOAF and RDF to catalogue digital photos and say things like "this is a photo of a female Anteater". A random example, but one that illustrates the variety of unexpected uses that a single addition to an RDF vocabulary can facilitate. It will be interesting to see how foaf:gender gets used...

OK, I'm trying NetNewsWire as a way of writing here while offline... Let's see if it works.

The FOAF developer site (rdfweb.org) has a number of useful, but overlapping facilities to support the development of FOAF and related projects. In fact, there are too many. We use IRC chat, weblogs, Wiki and email for communication; each has its value, but it is sometimes not clear where to look to figure out what's going on. One tool we have used in a half-hearted way, but which hasn't really caught on, is Bugzilla. We're therefore in process of shutting down our Bugzilla database and migrating the content across into the Wiki.

Bugzilla is great for large, collaborative software projects such as Mozilla. For the FOAF spec, it was overkill. and discouraging since it required you to set up and remember a password before scribbling notes in the issue-tracking system.

I've started a page in the Wiki as a replacement IssueTracker. The content isn't fully migrated out of the Bugzilla system yet, and the template/layout/etc may change. But hopefully this gives us one less place for information to hide away, and better integration with the other FOAF documentation that lives in the FOAF wiki.

Leigh Dodds has written an excellent Introduction to FOAF for XML.com. It gives a good technical overview of the whole project, working through some examples on data-merging and relationship typing. Thanks Leigh!

I've been talking with Mike Linksvayer of Creative Commons and Robert Kaye of MusicBrainz about the idea of a foaf:tipjar property that relates an 'agent' to a 'document' which serves as some form of 'tip jar' page, so that artists, content creators etc could use RDF to make statements that help them get rewarded for works they make available online. This initially wouldn't be very machine-readable, just a normal human-oriented Web page perhaps pointing to PayPal or Amazon wishlist pages. We can start simple and get fancy later.

The details need further discussion, but I like this idea a lot. It brings together concerns from the FOAF, CreativeCommons and MusicBrainz communities, and -- from a technical perspective -- allows us to explore the use of RDF to combine information expressed using the FOAF, Creative Commons and MusicBrainz datasets. One way to implement this would be to have a common notion of 'agent' shared across the three datasets/vocabularies. Leigh Dodds has been working on the RDF schema for MusicBrainz; we could define MM:Artist as a sub-class of foaf:Agent, so that FOAF tools would have some partial understanding of the MusicBrainz dataset. Mike also mentioned some discussions within CC of having an RDF representation of the notion of an agent.

How might this work in practice? One idea was for tools which generate CreativeCommons markup describing online content to have an option to include a declaration of the creating agent's FOAF description, including a link to a foaf:tipjar page. From a MusicBrainz perspective we could try to associate tipjar information with as many artists as possible, ideally alongside their homepage URLs and other handy metadata (photos, weblogs, ...). This would hopefully assist users of content in finding their way to a page that explains how to reward the content-creator (with online payments, Amazon gifts, etc.).