Creating the XMPP Network Graph

January 24, 2024
by Guus der Kinderen

At the risk of sounding like an unhinged fanboy: XMPP is pretty awesome!

I've been involved in one way or another with XMPP, the network protocol that is an open standard for messaging and presence, for the last two decades. Much of that revolves around development of Openfire, our XMPP-based real-time communications server.

TL;DR:

An elaborate network graph of XMPP domain nodes.

Decentralisation with XMPP

There's much to say about what I like about XMPP, but let me focus on one thing in this text: decentralisation. Not only the protocol, but the entire XMPP ecosystem - with all its different server and client implementations - is based on the principle of federation. This allows anyone to set up their own solution for instant messaging, voice and/or video conferencing, data sharing, and much, much more. All of this is done without creating a dependency on any central entity or organisation. At the same time, you’ve created a solution that allows you to communicate with others that are not part of your own domain.

Some of the benefits of decentralisation are obvious: you get to control your own data. When you’re not sharing data with one monolithic IM solution provider, then there’s a lot less to worry about with regards to their privacy policies, marketing strategies, cookie policies and data security.

Another benefit of using a decentralised approach is diversity. I know of at least seven or eight different XMPP server implementations - all of which are mature projects that have a proven track-record of interoperability for many, many years. Each of the server implementations have their own strengths. Some favour raw network performance, others offer a more complete social media feature set. Some focus on being active in the Small and Medium-sized Enterprises segment, others try to cater to family & friends type of communities. Some are open source, others are commercial products. There are products that offer a turn-key, no-config-needed full blown instant messaging solution, while others can act as a development platform that is useful when you’re looking to develop your own networking application. As you might be able to tell from all this, diversity gives you the option to select the best software suite for your needs.

I digress. XMPP federation is based on server-to-server connections. Whenever a user on one domain starts to interact with a user on another domain, servers of both domains will connect to each other in the background to facilitate this interaction. As you might imagine, when enough users start to interact with each other, this leads to interesting webs of interconnected domains.

Reviving an old idea: creating a network graph!

Over the last holiday season, I remembered an old, now defunct project by Thomas Leister, that generated a visual representation of interconnected servers. Its visuals were pretty amazing. I remember Tom’s solution to be based on an out-of-band exchange of data (through a simple webservice), and recalled his desire to replace this with a solution that used XMPP’s own protocol and functionality. His stated goal was to use XMPP’s Publish/Subscribe functionality, but never seemed to have been able to get around to implementing that. I had some spare time over the holidays and challenged myself to build this new approach. I started work on a new version of that project, aiming to build a web application that renders a semi-live network graph of XMPP domains with their connections to other XMPP domains.

The path from prototype to a fully working solution was an interesting one, involving a couple of different aspects of development within Openfire, but also the XMPP community as a whole.

Using Openfire as a development platform

Perhaps unsurprisingly, but I love working with Openfire. It’s so incredibly versatile when it comes to adding new features and functionality.

For this new project, I needed a couple things.

  1. An API to add functionality to Openfire. Check. Openfire’s Plugin API gives you structured access to pretty much everything in Openfire. It’s easy to use, yet very versatile.
  2. A web server. Check. Openfire ships with Eclipse Jetty, an embedded web server. It’s used for Openfire’s Administration Console, but can just as easily be instantiated to serve different content. Openfire’s BOSH connectivity makes use of this, but also plugins like the Random Avatar Generator and Presence Service that expose user-specific, as well as the inVerse and JSXC plugins, that each serve a full-fledged web based XMPP client.
  3. A Publish-Subscribe service. Check. Openfire out of the box implements XMPP’s pub-sub standard.
  4. Database storage. Check. Openfire ships with support for most popular relational databases. Crucially, it allows a plugin to define and manage its own database structures.
  5. A javascript graphing engine. From a quick search, various applicable projects popped up. I opted to go with vis.js, for no other reason than that it was the first thing that popped up, looked reasonably mature and had easy-to-follow documentation. I later added VivaGraph, which offers WebGL support. Turns out that if you render thousands of nodes in a network, CPUs tend to get busy. Who knew? WebGL helped make things more efficient.
  6. Basic HTML and CSS design skills. 😬 I am many things, a good designer is not one of them.

My first prototype wrapped all of this into a solution that:

  • Periodically iterated over all server-to-server connections
  • Stored all information in a simple data structure
  • Persisted the data structure in the database
  • Created a web service that exposes the data as ‘nodes’ and ‘edges’ to be consumed by the graphing software
  • Have a simple, static webpage that consumes that webservice, and renders the graph using the third-party graphing engine.

In all, I was pretty proud to have been able to write all this in a single evening!

The approach above gave me a nice hub-and-spoke graph, where my server was the hub, showing spokes to every connected remote domain.

To be able to install this on more than one domain, I separated the plugin into two:

  1. One plugin that aggregates the connectivity data, to be installed on all servers on the network
  2. Another one that generates the website installed only on the server that acts as the public interface to the website.

I’ve used the XMPP’s Publish-Subscribe feature to bridge the gap between the two plugins. After some quick modifications, the first plugin creates a pub-sub node on the local pub-sub service, to which the second plugin subscribes. The second plugin then aggregates all of the data in its database, and uses that to populate the webservice, just as before.

Using this mechanism, it is pretty straight-forward to have many servers feeding one website. With a bit more work, I was even able to write a quick crawler, that tries to find pub-sub nodes with similar information on all of the XMPP domains that are reported as being remotely-linked domains, which removed the need to have every server sign up to the website manually.

Finally, I paid a hoster a little bit of extra money to have a new server to host a new Openfire server that would act as the public website, going through the motions of having a domain name and corresponding TLS certificate. Having done this before, I automated most of that, allowing me to create a new Openfire server from scratch in about ten minutes. I manually installed the new plugin, installed a reverse proxy to serve web content on standard ports, and, presto! The XMPP Network Graph suddenly became a publicly available service!

Some of the community members at IgniteRealtime.org were happy to install my plugin, which quickly contributed to the network graph growing.

Working with the XMPP community

To be able to grow the XMPP network graph, it is desirable to have support added to more server implementations than just the one for Openfire. As luck would have it, the XMPP ecosystem, as stated above, thrives on diversity.

To allow for a great deal of extensibility and flexibility, and to optimise interoperability, the XMPP Standards Foundation manages a pretty nifty process for extending the XMPP, through documents aptly named XMPP Extension Protocols (XEPs). The full process is documented in the very first XEP-0001. Have a read, if you’re interested.

The standardised way to get XMPP entities to interoperate on a bit of functionality is simple:

  • Write a XEP to document the functionality
  • Submit the XEP to the XSF for review and publication
  • Incentivise others the adopt the XEP

I did just that, and found the added value of this process to be unexpectedly high.

A submitted XEP makes for a convenient discussion subject. My original document quickly drew feedback.

Although I was aware of Thomas’ implementation, others apparently also toyed with creating network graphs of the XMPP network. Seems that I’m even further from having had an original idea than what I expected.

The feedback from the XMPP community showed the expertise and experience that lies within that community. Several technical issues were discussed, which led to improvements of the protocol. Probably the most important bit of feedback that was given related to privacy concerns, which we discussed at length.

The XMPP ecosystem consists of servers of all sizes. There are various XMPP service providers that each have many thousands of users. There are also plenty of servers that are running for family and friends, or even a single user. It is these servers that were the subject of the privacy concern.

If a connection is visible between two of these small servers, it becomes reasonably evident that two specific individuals are communicating with each-other. If both individuals agree to have this information published, then there’s no privacy concern - but what if only one individual does so? If John makes public that they’re connecting to Jane, then the fact that Jane is communicating with John is implicitly made public too. If other friends of Jane (Jack, Jill and Johan) similarly publish all their connections, then determining who Jane’s friends are becomes pretty straightforward - without Jane having consented to any data publication.

This, rightly, got flagged in early feedback from XSF members. We’ve discussed the impact of this problem, the need to address it, and various strategies to resolve the issue. We ended up with a solution that allows any server to publicise their connections, but require them to automatically verify that their peer opts-in to having a connection to their server be identifiable (those that do not show up as anonymous blips in the graph).

Based on the feedback, this and other improvements were quickly made to the XEP and my Openfire implementation. Now that there was a stable-ish protocol definition, it became easy for compatible implementations to be created for other XMPP servers. To date, there are implementations for Tigase, Prosody and ejabberd - and there’s mine for Openfire, of course. Not a bad score, after only a few weeks of development!

Wrapping up.

My XMPP Network Graph project has been maturing nicely in the last few weeks, as you can see from the screenshot above. You can have a look at and interact with the network graph at xmppnetwork.goodbytes.im. At the time of writing, it contains over 6,600 domains. It is pretty powerful to see how many people are interacting over XMPP, and that only in the small part of the network that is being mapped by the graph!

You can now add your own XMPP server to the graph! The plugin that I created for Openfire can be found here. Plugins or modules are available for other XMPP servers too. Have a look at the FAQ section of the XMPP Network Graph for instructions on how to add your server to the network graph!

I've enjoyed the process of setting all this up. Having most of the development pieces already in place, as mentioned above, allowed for rapid development. To me this is a testament to the power of not only Openfire as a development platform but also XMPP as the versatile Swiss Army knife of network protocols.

I'd love to learn what you make of this! Do you have success stories of your own to share? I’d like to hear from you!

If you want to comment on this article, please do so as a reaction to the blog post that corresponds with the article!