[{"title":"A bullet-journal workflow","url":"/bullet-journal-workflow/","content":["Making task lists can be fantastically useful, but it's all too easy to lose them.","My paper lists get buried under reams of doodles and random notes, or lost in the depths of a full notebook that simply has too much in it.","Even digital lists get lost, and I find I lack the discipline to open a mammoth text file every morning.","Paper is much better for immediate reference, even if it is easy to miss-place.","Productivity Hacking","The quest for elegant ‘productivity hacks’ is both fantastically useful and fantastically counter-productive.","You can waste days reading about how not to waste your time.","Finally, however, it seems that all the hours I've spent reading about productivity-hacks weren't wasted after all; my current system finds me being more efficient and productive than I've ever been before.","And at the hub of my new-found momentum lies my Bullet Journal.","The Bullet Journal Concept","Devised by inspired Brooklynite Ryder Carrol, the Bullet Journal is little more than a seemingly-simple method of writing out a ‘to-do’ list using a pen and a notebook.","I've experimented with similar list-hierarchies in the past, with mixed results, and while this new one is elegant, it's not exactly revolutionary.","But there's one aspect of the Bullet Journal system that I've found to be a real revelation: the way the list rolls-over from day-to-day and month-to-month.","What first drew me to the Bullet Journal was the systematized nature of its bullet-point organization, but what's kept me using it is the way it stays relevant.","Simplicity.","I love simplicity, and the system itself is fairly straightforward – each list item is marked with either a square, circle or mid-dot.","The square boxes denote tasks, circles events, and dots are standard notes (things to remember, but not explicitly actionable).","At the start of every month I draw up a list of long-term goals, and then each morning I set out my daily tasks.","I number the pages as I go, and keep an index at the front of the notebook.","This makes assessing progress on long-term goals a piece of cake.","There's only ever one active list.","At the start of every day I write out a new daily list.","Any items unchecked from the day before are either struck-through (which denotes they've become irrelevant) or marked with an arrow and copied into the new daily list.","This way you never have to refer back to old lists hidden in the depths of the notebook, and it's painfully obvious when a task is being continually put-off (which forces you to act: either do it or mark it as irrelevant).","There's still long-term structure.","While there's only the one active list for you to deal with at any one time, the monthly lists are useful for reference, making it harder for day-to-day tasks to swamp your ‘big picture’ aims.","It's flexible.","The Bullet Journal system is loose enough that it never gets in the way of my day-to-day workflow.","If i need a date-agnostic list on a specific topic, I just make it and add the page number to the index.","Little additions like this make the whole process expandable and fairly all-encompassing, and the day-to-day running of the list is never disturbed by all these additions.","It's really easy.","Keeping the journal up to date sounds like rather an onerous task, what with all the indexing, page-numbering, and three-tiered bullet system.","In reality, keeping on top of all my lists is no hassle at all.","Updating the list takes about 10 minutes every morning, and up to half-an-hour on the first day of every month (as I update my calendar, and re-set my long-term goals).","My Bullet Journal","The journal isn't solely responsible for my productivity upswing, but it's definitely helped.","By way of a disclaimer, making the switch to a Bullet Journal workflow wasn't the only change I've made recently.","A few months ago I quit my day-job and struck out on my own; an intrepid freelancer cum entrepreneur bound for riches and glory!","So now that any-and-all money that I make is directly proportional to how much work I do, it'll come as no surprise that I'm working harder and longer than ever before (and would be with or without the Bullet Journal).","But using the Bullet Journal has undeniably helped my workflow.","My days now have structure, and (most importantly) I'm now much less likely to overlook an important task."]},{"title":"Simple is hard","url":"/simple-is-hard/","content":["“Simplify, simplfy!”","we're often told.","Your service isn't appealing to your customers?","The answer's simple: simplify!","Your song isn’t connecting with your fanbase?","Simplify!","You aren’t shifting as many units as Apple?","Simplify!","It’s an easy mantra to mock, and even easier to swallow hook, line, and sinker.","What we need to do is distinguish between ‘simple’ and ‘simplistic’.","It's fairly obvious that content suffers when overrun by crowded, cluttered, un-navigable interfaces.","If what you’re saying has value, you don’t need the kitchen sink to help you say it.","But what do you need?","And how much can you afford to throw out?","Only keep what you need.","Deleting everything that you don't need only works when you know for sure what you do need.","It's tempting to think of simplifying a design as merely stripping away layers of visual ornamentation, but true simplicity comes more from the user experience than the look of the thing.","Hiding an unsightly menu behind a hamburger icon is an easy way to de-clutter an app’s interface, for example, but does that count as simplification?","All that achieves is forcing the user to make an extra click when they want to navigate somewhere.","Even the Government's Digital team agree that this is both important and hard (and the new gov.uk design is really impressive from a usability perspective).","Clearing away useless clutter is generally a good tactic, but there's a distinct difference between simplifying a product and merely making it's function more obtuse.","Is one button with five functions really better than five buttons that each do one task?","Is an aesthetically-pure minimalist interface really ideal if you then need to teach every new user how it works?","As with any task worth doing, there are many levels of nuance to simplification.","It takes judgement; an assessment of what functions are important to your product, your app, your idea.","Strip away anything extraneous and put anything the user needs front-and-centre, but don’t confuse the two or you’ll live to regret it.","This website is all about words.","The page you’re reading now went through many iterations before settling on the look you see today.","Early drafts were swamped with full-page background images and clever CSS3 animation effects, but no matter how fancy I got with the coding these early designs never felt quite right.","Then I stumbled upon Justin Jackson's inspired ‘This Is A Web Page’ article and was reminded that what was important for this site was the words themselves.","It goes deeper than that, even: what's important to this site is the content.","It could be read aloud by a screen-reader, translated into sign-language, even transmitted telepathically for all I care.","As long as the message can be absorbed, this page will have done its job.","What I was doing with my early designs was trying to build a ‘great website’, when what I should really have been focusing on was building a great content delivery system.","It seems video is the de rigueur medium for getting messages across quickly, and for a while I did toy with the idea of making this page into a vlog, but ultimately words are purer.","Video is harder to create than written content, and would skew the message with all the baggage brought along by seeing my face (beautiful) or hearing my accent (perfectly inflected Queen's English).","With the written word I have more chance of capturing the ‘white heat of inspiration’ and of being able to communicate my thoughts clearly.","Why use any styling at all?","So why not go all-out and strip out all the styling from this page completely?","Properly written semantic HTML is nothing but content, so surely stripping away everything else would be the most efficient way to get the message across?","Well, quite frankly, the default styling applied by web browsers is ugly, and that's not as shallow a complaint as you might think.","Concepts like line-height, line-length, leading & kerning each have a huge impact on the readability of text.","To trust all those to the random whims of a browser is madness.","It’s often said that we read best what we read most, and my what my friends and I read most are books.","Books are fantastically efficient, and the best ones are things of beauty that wrap their content in a nourishing blanket of aesthetic proportions and finely-wrought typefaces.","Given how many books I own solely on the topic of type design and typography, I can’t stand idly by and leave my content at the mercy of the ‘web-safe’ default fonts.","I have to believe that good typography and good design makes the reading experience better.","Simple doesn’t have to be ugly.","It seems obvious, but websites still have a lot to learn from books.","Just because we can swamp everything in technical wizardry doesn’t mean we should, but it equally doesn’t mean we should abandon design altogether.","‘Simple’ doesn’t (and shouldn’t) mean ‘ugly’, and the simple act of thinking about line-lengths and visual hierarchies can turn an indigestible splurge of HTML into something readable, and in the right hands even something beautiful.","Finding the right balance between visual simplicity, simplicity of interaction and a pleasing aesthetic is a real challenge, and I’ll be the first to admit I’ve still a long way to go.","But at least now I’m pointing in the right direction; simple is important, simple is necessary, and simple is really h"]},{"title":"Futura for the win","url":"/futura-for-the-win/","content":["Picking the right typeface for this site took a long time, and my final choice wasn’t one I was expecting.","Ask 3-months-ago-me if I'd set a website entirely in Futura and I'd have laughed at you.[","1]","How the dropcap looked when this post was written","My initial ‘Contemporary Drop Cap’ concept was much more florid and overtly ornamental than what you see today (as of April 2014).","I burnt through many drafts experimenting with ornate serif fonts before I finally remembered that what I was trying to build was a portfolio; a neutral backdrop for my designs, not something that would distract from the work on show.","In short, I needed a sans.","Neutral doesn't mean flavourless.","Even the most generic-seeming typefaces have their own distinct flavour, as my Helvetica-fuelled “Hey look!","I just discovered Swiss Design” phase will attest (now commonly known as my “God-awful Superdry Period”).","I've developed a deep and abiding love for a perfectly circular “o”, so when choosing a sans I normally start with the geometrics.","I'm particularly fond of the classically-British ‘modern’ fonts of Edward Johnston and the saucy Eric Gill, but I've used them far too often already.","If I used them in my portfolio site as well, I'd start to look like a one-trick pony.","H&FJ(RIP)'s Gotham looked lovely, but I can’t help but feel it’s on fashion's downward slope now.","Far too many people are using it without thinking, and of course it carries a certain politicised air with it now, too (thanks, Obama!).","So keeping with the geometric “o” theme, the next obvious choice was Futura.","Retro futurism","I'd like to say there is some elaborate over-arching theoretical underpinning to my choice of Futura, but the reality is more prosaic.","Sure I could waffle on about the retro-futurist stylings of the “W”s and “M”s, and how they fit into the dialectics of 20th Century design praxis, but in truth it was simply next on the list of “things to try”.","Once I saw my generously-margined headings set in Futura Bold I was smitten.","Couple that with the not-too-shouty elegance of the all-caps italic, and I'd found the lynchpin for my design.","Flawed Futura","Unlike many others, I actually quite like Futura's much-maligned question mark, for all it's eccentric meanderings.","There are, though, some very visible problems with the ParaType cut of the font (the one I use via Typekit), and quite possibly with the typeface as a whole, too.","The lowercase “J” looks stylish and bizarre in equal measure, and the semicolon is oddly top-heavy.","In body copy the x-height is a tad lower than I find comfortable and the letter spacing after an apostrophe is massive.","Not an issue in a programme that lets you kern, but an ugly cross to bear on the web.","In fact, all the apostrophes and quotation marks look the same.","It’s all well and good training yourself to press alt shift ] every time you want an apostrophe, but Futura renders such efforts irrelevant.","How many different characters can you count in the following sentence?","Futura quotes etc. - small","Tough, eh?!","It's a little easier at larger sizes:","Futura quotes etc. - large","There are actually seven discrete glyphs there.","There are differences, which at large sizes are delightfully subtle and nuanced, but at anything less than 72pt they all blur into one...","A designer's website is like a builder's house; always the last to be finished.","This blog-cum-portfolio site of mine has been through a seemingly infinite number of variations.","What you see in April 2014 is only the third full ‘theme’ to make it all the way into code and deployment, but if you count Illustrator mockups then the number of iterations trebles.","Count paper sketches – even limiting yourself just to fully drawn out and wire-framed designs – and the total skyrockets.","Only time will tell how long this current designs sticks around for, but it already feels like this version is a more substantial website than any of my previous attempts.","Picking Futura and sticking with it, despite the issues and imperfections, has meant I've been able to turn my attention to the finer details: the little moments of interaction design, the negative space, and above all the content.","This post was written in 2014.","As you've no doubt already noticed, I've moved on from Futura as the typeface for this site.","↩︎"]},{"title":"Fullpage screenshots in Firefox","url":"/fullpage-screenshots-in-firefox/","content":["Every now and then I need to take a screenshot of an entire webpage, generally to use in my portfolio.","Manually positioning the cursor for a cmd shift 4 screenshot (on a Mac) feels a little imprecise for my tastes, and I want to capture the whole of a site, not just the area visible in the viewport.","All the browser extensions I tried turned out a little buggy.","Areas of the page would be repeated, like a visual interpretation of a CD skipping, or fonts would render strangely, as if they'd been run through an IE8 simulator or something.","Finally I stumbled upon this neat little trick.","Sadly it doesn't help me get girls or lose weight like all the “one weird trick” pop-up ads keep promising me, but it does mean I can take a screenshot of an entire webpage using one line of code.","The answer!","Navigate to the page you want to shoot and open up Firefox's Developer Tools command line using shift f2 (Mac users might need to remember to press fn as well, if their keyboard rebinds f2 to increase the monitor's brightness).","Then all you need to do is type the following line:","screenshot --fullpage YourFileNameHere","And voilà!","A screenshot has been taken of the current page in its entirety – not just the visible area!","If you're super lazy, you can even leave off the file name; Firefox will just use the current date and time to save the image under.","I found I needed to crop off a sliver of transparency on the right hand side when I opened the shots up in Photoshop, but that's a small price to pay for the time saved not having to stitch together various peepshow-views of a page.","See it in action","You can see some of the screenshots I took using this method over on my portfolio page.","~The ‘phone’ view of the Atlantic Brewery page is particularly long and thin – an image I'd have struggled to create using any other method.","~"]},{"title":"Inspirational Web People","url":"/inspirational-web-people/","content":["There's an episode of the Boagworld podcast where Marcus and Paul list the ten ‘web people’ they've found most inspiring.","This has in turn inspired me to publish my own list.","The web-creators community is packed with generous people willing to share their insights and experiences.","Through blog posts and podcasts, conference talks and tutorials, these people have had a direct impact on my outlook and my career choices.","01: Steve Lovegrove","In my experience everyone has that one person that inspired them when they were just starting out.","For some it's a teacher, or someone who gave them their first break, or a professional who mentored them.","For me, it was my friend Steve.","Steve's always been miles ahead of me in intellect and ability, and most of what he tried to teach me went straight over my head.","But even so, he was willing to let me spend hours peering over his shoulder while he coded.","And because we would also talk through what he was working on, some of it rubbed off.","Steve taught me just enough to be dangerous; just enough to get something on the web and see how good it felt.","He's long since left the web behind to do ‘real’ science, but having a patient and generous friend like Steve was probably the single biggest factor in shaping the direction my career has taken.","02: Chris Coyier and Dave Rupert","Anyone who's ever Googled a CCS question has probably benefited from Chris' sterling work on his website CSS Tricks, but it's his video tutorials (on Lynda.com and the “Lodge” section of his own website) that have impacted my world the","most.","His approach of getting-it-wrong-on-camera-and-then-fixing-it-(probably) was a welcome break from other more sanitised tutorials.","Not everything installs perfectly the first time, and no one is immune from typos.","Seeing how professionals deal with these problems is almost as useful as the main focus of the lessons.","Chris also does sterling work in the podcast field with his partner in crime Dave Rupert.","If you're not entertained by the combination of in-depth Q-and-A, stupid sound effects, and friendly banter that they put out in the Shop Talk Show, then you're probably not interested in front-end development.","As an added bonus, their mantra is probably the single most useful bit of advice any web professional can give:","Just.","Build.","Websites.","03: Brad Frost","I'm never more aware of (or excited by) the unique nature of designing for the web than when I listen to Brad Frost.","If every colleague and every client could take the time to read his blog and watch his talks, the standard of our work and the quality of our work-lives would raise ten-fold across the board.","The idea of designing ‘paintings of websites’ (a.k.a.","PSD mockups) is completely outdated; we can't just design for one ideal viewport size, or two, or ten, or a hundred.","“Atomic Design” might sound like a buzz word, but the principles are applicable to all web work.","The basic concept of designing components first and pages last is quite simply How It Should Be Done Now.","04: Mark Brickey, James Flames, Billy Bauman","I love to hear people talking candidly and in-depth about fields they are experts in.","It almost doesn't matter what the topic is (which is one of the reasons I love Radio 4's In Our Time so much), but when the topic is illustration I love it even more.","The Adventures In Design podcast is irreverent and unhinged, but always honest and informative.","They always have good guests, but more importantly Mark, Billy and James know what to do with them.","They trigger interesting and inspirational discussion, no matter who the guest is.","The resulting interviews are often challenging, often controversial, and always riveting listening.","Another reason I like AID so much is that team aren't ashamed of making a living: this is a podcast about the business of design.","It just so happens that the design field they work in is the super-cool one of gig posters and illustration.","05: Mike Monteiro","Anyone working in the world of design who deals with clients should watch Mike's *F*ck You.","Pay Me.* talk.","He explains why it's important to take yourself and your business seriously, and even brings his lawyer on stage.","His book, Design Is a Job, is less caustic, but just as inspiring.","His book, Design Is a Job, is less caustic, but just as inspiring.","Neither the book nor the talk are rants against stupid clients (well, maybe a little...).","They focus instead on building productive client relationships where no-one is taken advantage of.","It's hard to admit that sometimes a problem doesn't stem from the client but from yourself.","Even so, having that in the back of my mind has often saved my freelancing-bacon.","Following Mike's advice has often turned tricky clients into excellent ones.","For all his acetic bluster, Mike's core message is simple; be friendly and use common sense.","06: Jessica Hische","Jessica is a letterer, and all her work is stunning.","Before I was aware of her, I didn't even know ‘lettering’ was an actual job that you could do.","Back in the day, Jessica was a jack-of-all-trades (like many of us), but she realised that You Get The Work You Show.","She filled her portfolio with self-initiated lettering work, and was soon able to specialise professionally.","Art directors (and all clients, in fact) aren't blinkered and lacking in imagination.","With so much talent out there, they can find someone with exactly the right experience and demonstrable expertise.","If you think lettering is your strongest skill but stack your portfolio with e-commerce websites, guess what kind of work you're more likely to be hired for?","If you want to specialise in something, you have to get off the mat and do it.","Don't wait for the “dream commission” to come walking through the door; get out there and chase it down.","Jessica makes her own luck, and makes me think I might be able to make my own, too.","07: Trent Walton","There are only so many ways to present seriffed text on a light background, but somehow Trent manages to make his blog look nicer than everyone else's.","His art-directed posts (each with their own custom styling) are a benchmark for bloggers everywhere.","Somewhat annoyingly, what he writes about is also several orders of magnitude better than anything I could manage.","I don't know if he became so respected by writing well about “big picture” subjects or whether he writes well because he's so respected, but either way, respected he is.","His posts come fairly infrequently, but when they do they're invariably beautiful and interesting.","08: Ben Schwarz","Task runners (like Grunt and Gulp) have revolutionised my workflow over the past year.","Grunt trickled into my sphere of awareness over a period of months, much like SASS.","SASS I adopted more by osmosis than through any deliberate choice, but I can pin-point my adoption of Grunt to one specific incident.","Ben has made a fantastic CSS-only Slider, and I bought his accompanying screencast tutorial.","It might be the best £10 I’ve ever spent.","The video explains his intricate SASS, but also walks through a Grunt installation and shows how useful it is.","Up to that point, I was aware of Grunt but didn’t understand it or see the point of it.","To see someone as consummately professional as Ben using it and extolling its virtues sealed the deal.","I’ve since switched from Grunt to Gulp, but using any task runner at all has been the real quantum leap for my workflow.","09: Sean Johnson & Liz Elcoate","Quitting a job to go freelance is a daunting proposition.","Making the leap myself, I found comfort in the curious trend for freelance-web-folk-focused podcasts.","There are plenty of great offerings out there, but by far the most relevant to my past situation was The Freelance Web.","Liz and Sean aren't afraid to deep-dive into the details of running a web business; both the good parts and the bad.","Their advice is useful and reassuring, and the fact that they talk about specifics makes it all the more valuable.","I've since moved from freelancing to an agency position, but I still enjoy The Freelance Web.","The willingness to share experiences and knowledge is one of the best characteristics of the web community.","It's one I find it hard to believe any other industry could match, and Liz and Sean are true embodiments of that generous spirit.","10: Daryll Doyle","On a day-to-day basis, by far my biggest inspiration is my team mate, Daryll.","He's very much from the school of thought that says “if you're not moving forwards, you're moving backwards”.","Pretty much constantly, Daryll is showing me some new tool he's discovered, technique he's mastered, or framework he's learnt.","His dedication to self-improvement is nothing short of inspiring.","I know a couple of the people on this list IRL, and a couple more I'd class as ‘digital acquaintances’, but most are just web personalities who's work, public speaking, or blogging has taught me something new, steered me when my course has","wandered, opened my eyes to new ways of thinking, or just been inspiring in some more ineffable manner.","People like this are what makes the web such a fantastic industry to be a part of, and I owe them all a debt of gratitude."]},{"title":"Getting started with inline SVG icons","url":"/inline-svg-icons/","content":["I've long been an advocate of using icon fonts.","They're resolution-independent, light-weight, and stylable with CSS.","It turns out they're not the best option: inline SVG icons are better in almost every way.","We should all know by now that using raster images for icons is a bad idea.","Limiting ourselves to one set resolution is not how we work in a multi-device ecosystem.","It's not responsive, it's not performant, it's not nice.","For a while it seemed that Icon Fonts were the answer.","They're vector based and therefore resolution-independent.","You can style them with CSS.","They have smaller file sizes than .","JPGs or .","PNGs, so they improve a site's performance.","A font file is one asset to download (more http requests = poorer site performance), so we don't need to mess around with image sprites any more.","And on top of all that, icon font creation can be easily automated with your task-runner of choice: simply drop your icon .","SVGs into a folder and you're good to go.","/* Calling an icon font icon as a pseudo element within CSS.","*/",".elementName:before {","font-family: "iconfont";","content: "\\e001"; // The unicode value of the font character you want.","color: #000; // The icon is text, so you can style it using regular CSS","}","If it ain't broke, why fix it?","Lately the winds of change have been blowing.","Fewer and fewer big-name sites are using icon fonts, and more and more are publicly coming out in favour of inline SVGs.","At first I was sceptical: "that problem's already been solved", I thought.","Actually inlining raw SVG code into a site seemed overly complicated, and harder to maintain and automate than an icon font setup.","But was it?","And were icon fonts as great as they first appeared?","When icon fonts fail, things get weird.","Sometimes there's no telling what crazy character a browser will display in place of an icon that hasn't loaded.","There are inexplicable rendering inconsistencies.","Firefox in particular seemed, like an overly generous grandparent, determined to fatten my icons up.","Faux-bold on regular fonts is ugly and annoying, but when it happens to brand assets like icons it's excruciating (and something clients always pick up on).","Automation isn't so seamless after all.","Oh hey, my npm install keeps failing.","I wonder what the problem is: oh yeah, my gulp-icon font is throwing errors all up in my grill.","Again. #facepalm","By contrast, pure SVG is a much more consistent format.","I've been using SVGs within <img /> tags since 2006, and I've never had any trouble with setting SVGs as CSS background images.","/* SVG CSS background-image.","*/",".elementName {","height: 1em;","width: 1em;","background-image: url(path/to/icon.png); // Fallback for browsers that don't like SVG.","background-image: url(path/to/icon.svg);","}","The obvious simple solution is to just use SVGs like that: set them with CSS or use <img> elements.","That gives you scaleable vector graphics that work well in most environments.","The trouble is, you're loading in a new file for every icon.","Every icon would represent another HTTP request to the server, and would be a performance nightmare.","No matter how "squishy" your design, if your site is slow to load it is not "Responsive".","So we're back to the same multiple-requests problem we had in the bad old days of raster iconography.","The answer back then was to use image sprites.","Concatenate all the icons into one file, load them all with just one HTTP request, and use CSS to crop the image when you want to display a particular icon.","It turns out we can do that with SVG too, except now we can do it better!","Inline SVG Sprites","SVGs are made of paths – collections of co-ordinates that describe the individual vectors of the image.","Open an SVG in your favourite text editor and you can see those paths.","What's more, you can add classes and IDs to them.","Even more mindblowing: if you wrap the individual paths in a symbol tag you can combine multiple images into a single SVG file, and selectively pull them out using their IDs.","<!-- Load the SVG file into your document... -->","<svg","xmlns="http://www.w3.org/2000/svg"","xmlns:xlink="http://www.w3.org/1999/xlink"",">","<symbol viewBox="0 0 10 10" id="iconID">","<path d="{ vector data }" />","</symbol>","</svg>","<!-- ...and then 'use' a specific icon -->","<svg class="icon">","<use xlink:href="#iconID" />","</svg>","Now you might be thinking that this all looks like a whole lot of work, and you'd be right.","Making sprites by hand sucks.","Thankfully the arrow of progress is on our side.","We may not have jetpacks and flying cars yet, but we do have task runners.","I used to use Grunt, but now I use Gulp (being a JS nerd, I find the syntax easier to follow) and there are SVG spriting plugins for both of them.","I use gulp-svg-sprite, and so far it's worked like a charm (so far I've put it to work on a half-dozen private projects and a couple of client sites).","In fact, it's proving to be much more reliable than any of the Gulp plugins for icon fonts ever were.","It takes a bit of configuring to get the drop-an-icon-into-a-folder-and-let-Gulp-do-the-rest workflow I'd maintained with icon fonts, but as long as you remember to specify that you want a symbol based sprite, you can't go too wrong.","// My Gulp config","var svgConfig = {","shape: {","dimension: {","// Set maximum dimensions","maxWidth: 32,","maxHeight: 32","}","},","mode: {","symbol: true // Activate the «symbol» mode","}","};","// The task itself","gulp.task("svg", function () {","return gulp",".src("_uncompressed/icons/**/*.svg")",".pipe(svgSprite(svgConfig))",".pipe(gulp.dest("_includes"));","});","Inline SVG in Jekyll","This site is built using the static-site generator Jekyll.","Transferring this site's (admittedly meagre) icon set from an icon font to an inline SVG sprite couldn't have been easier.","Make sure your task-runner outputs your sprite into Jekyll's _includes directory and you can pull in your sprite with a {% include /symbol/svg/sprite.symbol.svg %}.","And because Jekyll compiles before deployment, you don't even have to make a single HTTP request to get your icons: it's all truly inline.","Inline SVG in Wordpress","Jekyll's all well and good if your clients are happy with working in code and using the command line to update their site.","Sadly the world hardly ever makes things that easy for us.","Wordpress sites don't have the advantage of being compiled before deployment, which means we need to use PHP to include the sprite.","Including an automatically generated SVG sprite using PHP is a little trickier than it first appears. <?","php include_once("path/to/svg/sprite.svg"); ?","> would seem like the obvious solution, but all that will give you is a face full of errors (if you've got WP_DEBUG set to true).","It turns out the issue is with the XML and DOCTYPE added to the sprite by the gulp task, and the trick is to use file_get_contents() instead of include_once().","SVG & CSS","Styling the icons had a couple of little hurdles that I wasn't expecting.","One of the joys of icon fonts was how easy they were to manipulate with CSS, and that's a feature that SVG sprites were promising too.","Instead of color: #000; we use fill: #000; but, as with all CSS, watch out for specificity!","If the paths themselves have explicit fills and strokes defined, then you won't be able to override them with external CSS.","Some of my legacy icons have in-built colours, and going through and stripping them out would be a real pain.","Thankfully my buddy Daryll was on hand to build me a tidy RegEx to programatically strip out any rogue fills:","<?","php","// Save the SVG file in a variable...","$rawSVG = file_get_contents(get_template_directory_uri() . "/path/to/svg/sprite.svg");","// ...then filter out any fills.","echo preg_replace( '/fill=("|\\')(#)?([","a-fA-F0-9]*)("|\\')/i', '', $rawSVG );","?",">","When I'm using Jekyll I still have to manually manage explicit in-file styling, but Jekyll projects tend to require a more fine-grained approach anyway.","Another Big Win for CSS styling and SVGs comes from being able to add classes and IDs to specific paths within individual icons.","This provides us with the hooks we need to style (and animate and manipulate) almost every aspect of an icon discretely.","Icon fonts were limited to one flat colour, but with SVG sprites we can colour every shape in an icon differently (should we need to).","For a long time I was very skeptical of using inline SVG for icon sets, primarily because of the perceived effort it would take to set up such a system.","I'm happy to report that SVG sprites are considerably easier to get started with than I'd ever thought.","I can't imagine ever using an icon font again."]},{"title":"Why doesn’t everyone have an Internet Fridge?","url":"/internet-fridge/","content":["The great white hope of the Internet Of Things is the internet-connected fridge.","It's the go-to example when trying to explain IOT to lay people. "Imagine what you could do if your fridge was connected to the web!","" So why don't we have them yet?","And do we really want them?","Or even need them?","The "Internet of Things" is not about adding the internet to devices.","Being able to access the internet through a fridge gives us nothing.","There's no earthly benefit to having your social-network connections "like" your fridge, and there are few scenarios where you'd want to "share" your #fridge.","More accurately, there are few occasions when I'd want you to share your fridge with me.","In person, it's different.","I definitely want you to share: I'll have a cold beer, please, and maybe a sandwich?","Digitally, however?","No way.","The noughties idea of an internet-connected device was simple.","We can access the internet wherever we want: access email through the TV, get the news from your microwave, do your shopping through the fridge!","Simple, but pointless once smartphones achieved ubiquity.","Why wait for the fridge to boot-up when my phone's already in my pocket?","There's no aspect of the old, clichéd example of an "internet-fridge" that you couldn't better achieve by glue-gunning an iPad onto the front of a regular fridge.","And that would be more effective and cheaper than a monstrous over-priced, internet-enabled fridge.","Where's the win?","Where things get really interesting, and where IOT really comes into its own, is when we switch from accessing the internet through our devices to accessing our devices through the internet.","If a component moves, it will heat up.","If it heats up, it will distort and eventually wear-out.","Tiny sensors are now cheap and easily available, and connecting them to the internet is also cheap and easy.","It's the perfect-storm to herald the age of IOT.","Put a sensor on something that moves and connect it to the internet, and suddenly you can make efficiency savings.","At scale, this is for wheels and gears on industrial machinery.","If you know when a component will break, you can replace or fix it before it breaks.","This knowledge will save you time and money.","In the consumer marketplace, the wins are less tangible, but more relateable.","How would this manifest itself in a fridge?","The cooling mechanism inside the fridge has broken: sound the alarm!","The freezer door has been left open for more than 10 minutes: text alert!","You're shopping, but can't remember what you have already: a webcam snapshot of the inside of your fridge gets sent straight to your phone.","You live in the future, where every consumer good has a sensor inside the packaging.","Your fridge acts as a hub that sends useful data out to you.","The milk carton is 32% full.","The quiche is 2-days 7-hours away from its Best Before date.","The veg drawer is giving off a high proportion of noxious gases: I guess the peppers have gone off; here's a photo so you can decide for yourself.","Why hasn't it happened yet?","1: The runway for white-goods is too long.","Iteration takes decades.","Traditional IOT needs a much much shorter iteration cycle.","By the time a traditional incumbent gets a product to market it has become so overwhelmed by extra features and catches for obscure edge-cases that the primary objectives have been lost.","Simplicity is just not an option for these companies, and that it why they will always fail.","2: People buy fridges very infrequently.","If an incumbent manufacturer get their product in your home, it’s going to be there for ten years or more.","Give that product the ability to cross-sell other products, and that incumbent has a marketing beachhead in your kitchen.","That’s not something they’ll be able to resist.","If you get an Internet Fridge from an established white-goods manufacturer, your fridge will spam you.","3: White-goods manufacturers are not software companies.","“Smart” thermostats, for example, have been around for years but have always been ugly and nigh-on unusable.","Nest are not doing anything particularly new, but they have an eye for details and user experience, and are confident in their own value.","It’s the Apple effect all over again.","Only one company in every thousand is actually any good.","Getting “normal” people to interact seamlessly with technology is a tough sell — can your mum use the video recorder yet?","Making user-friendly software is hard.","Making user-friendly hardware is hard.","Doing both together is even harder, by many orders of magnitude.","The unintuitive interface from the depths of hell that I have to decipher every time I want to bump up the heat, and a Nest thermostat that makes me wish my life was more like the fancy people I see in magazines.","4: White-goods manufacturers do not sell to consumers: they sell to retailers.","They don’t understand their end-users in the way that a regular customer-facing company does.","Without that understanding, they’ll never make an innovative product that can truly excite us.","5: It increases the cost across the board.","Adding IOT capability makes the products more expensive, and only a fraction of the current market would see any real value from them.","I know from personal experience that only a small percentage of people can see the potential of the internet , let alone the Internet of Things.","Commercial success in this field is going to take a revolution in consumer-mindset.","6: The business model doesn’t suit the incumbents.","Another often-overlooked aspect of “The Internet” is that it’s not free and it does exist in a physical world.","It needs infrastructure and servers, and what that means in business terms is recurring costs.","Suddenly your white-goods manufacturer has to deal with support and maintenance.","That’s not what they signed up for, that’s not where their expertise lies.","They’ve spent decades implementing “planned obsolescence”.","To justify implementing IOT into their products, these companies will have to be sure of generating the recurring revenue to match the recurring costs.","Which brings us back to the fridge that sends you spam…","So we need a start-up to make the Internet Fridge?","To create the Internet Fridge we all want needs serious innovation, and all the evidence suggests the big incumbent white-goods manufacturers just aren’t up to the task.","This is where the mythical Start Up™ can help.","Those guys live for innovation and disruption, right?","Well, maybe…","To bootstrap an idea into existence you need a short iteration cycle.","Try something, watch it fail, learn from the failure, improve the next version.","Try something, fail, learn, improve.","Try, fail, learn, improve.","And so on.","It’s a pattern that has worked for almost every successful start-up out there.","This approach works really well in the software and internet spheres where you can ship updates and bug-fixes as often as you like.","With hardware, things get much messier.","There’s no continuous deployment or integration for fridges.","If your first version doesn’t work, you need a product recall.","And just getting to that first version in the first place is hard, for all the reasons we outlined above.","The lead-times are long and manufacturing is only cost-effective at scale, so rapid-prototyping is both hard and expensive.","What we need is an Apple for white-goods.","We need a large established company that values good design, leads the field in innovation, and isn’t afraid of changing a premium for quality.","They need to have experience in manufacturing, and also be at the top of their game with software too.","They need deep pockets and a long-term vision.","There are companies out there that could do it.","Occulus are working at the intersection of hardware and software: investing in their headsets and the digital systems that power them.","Nest are pushing the boundaries of what we expect from a home appliance: beautiful and functional.","Traditional white-goods — fridges, freezers, dishwashers, washing-machines — they all suffer from being both hard to build and not very sexy.","Whoever creates the perfect Internet Fridge will no doubt make a pretty penny, but with such imposing obstacles in the way, I can’t see anyone getting there any time soon."]},{"title":"Notes from ThingMonk: Day One","url":"/thingmonk-day-one/","content":["[1]ThingMonk was wall-to-wall interesting and inspiring.","I took lots of notes, but naturally focused on the areas that interested me most, and was often too busy trying to take it all in to write anything down...","ThingMonk styles itself as a conference “for developers, designers, data wranglers and decision-makers that want to turn ideas and concepts into industrial scale systems”.","I was there with my web-developer hat on, but was keeping my eyes peeled for hints at the future of business-tech in general.","I'm already deeply embedded in data-visualisation and dynamic analytics, but eager to connect my digital skills to a hardware framework.","Day One","Bullet-Proof & Profitable Things","Ubuntu guru and part-time space tourist Mark Shuttleworth made for a high-profile, big-picture opening speaker.","It was interesting to hear that his idea of "scale" was not massive super-computers costing exorbitant amounts of money, but rather lots and lots of cheap, tiny micro-computers all working in unison over a network.","It takes five to ten years to make an overnight success, and now is the time to be getting deep into IoT.","Wait 12 months and the train will have left the station.","Real disruption comes from riding a wave of change, and the disaggregation and componentisation we're seeing in the IoT sphere at the moment will inevitably lead to the "hockey stick" graph of innovation.","Data can be a liability - you must have a responsible security strategy when handling lots of data.","One mistake or hack can cripple a big-data company.","The next version of Ubuntu will be highly modular.","Apps will be isolated and contained, and will each have a predefined area they can write to.","This allows for robust version control across the system, and no more fear when updating the OS.","The “Internet of Toys” is going to be more influential than people realise.","It's easier to innovate with things that won't kill people.","Long-term projects will have to use standards already set by short-term projects.","Coherent UX for Distributed Systems","User-experience expert Claire Rowland had some fascinating insights into the challenges of managing UX in a world of connected "things".","We don't expect "things" to behave like the internet.","Light bulbs shouldn't "buffer".","Interfaces (if they are honest) should introduce the concept of failure to users.","Don't fake success when a user clicks a button in a low-connectivity area: show in interstitial "loading" state wherever possible.","You often have to choose between fuzzy-and-current or accurate-but-old.","A cat-tracker can either say "Mrs.","Tibbles was at this exact location 3 hours ago" or "Mrs.","Tibbles is somewhere within this range right now".","Linking physical and digital interfaces takes special thought.","A traditional dimmer-switch always knows it's ‘state’, where as a remotely-controlled version has to account for the fact that the ‘state’ of the bulb might have changed since it last checked.","From 2 to 2 billion: How to Design for Scale.","Sam Winslet & Sophie Riches from IBM presented us with the "Seven Deadly Sins for Design of IoT":","Greed: designing for yourself.","Envy: being a copycat.","Gluttony: focusing on features & functions.","Wrath: assuming your users know what IoT is.","Lust: IoT for the sake of it, without any real ideas.","Pride: don't over-complicate and don't over-simplify.","IBM use a strategy they call “Progressive Disclosure”: where an interface reveals deeper levels of detail as a user interacts with it.","Sloth: assuming users have ∞ patience or care about you/your product.","Hacking NFC","Nick Ludlum from Moo gave (with a little help from colleague Kai Turner) one of the more engaging and tech-focused presentations of the event.","The team at Moo have (after considerable time spent in R&D) figured out how to print NFC chips into business cards.","Every attendee at the conference was given a Moo business card on arrival, which was embedded with an NFC chip with tokens for free coffee and beer.","For NFC to work, you need a chip and an antenna.","The chip is tiny – literally mm across – and the antenna gets printed between to laminated sheets of paper using conductive silver ink.","This means the finished business card looks and feels just like a regular card - after all, it's still mostly just card and ink.","To build a reader, all you need is a "smarter" chip, an antenna, and a power source.","Android phones have the ability to be NFC readers, but IOS ones do not.","nfcpy is the Python library for NFC.","The cards can store a variety of MIME-type information (webpages, lat. & lng. data, etc) but URLs are the most versatile (and can be re-purposed without reprogramming the card itself).","Security is still ongoing project, and the Moo team would welcome any insights or help in that area that the developer community can provide.","The Things Network: London","Mark Hill, from Open TRV (a company that makes smart Thermostatic Radiator Valves) got the short-straw: his talk was right before lunch, and therefore had to be shortened because the conf. was running over time.","The Things Network is a low-power, wide-area network.","Crowdsourced a complete city-wide IoT data network in Amsterdam.","Launching the network in cities across the world.","London had its network launched this month.","Powered by gateways installed on roofs of businesses, accessible to all. ($1000 - $1500 per gateway).","Data Gravity & Time-series","Dave McCrory, allegedly famous for for coining the term “data gravity”, delivered an engaging talk with a charming hand-drawn slide-deck.","He touched briefly on how data !","= information, but his main point was that dealing with time (in an IoT setting) is important but very, very hard.","Data Gravity: data has a 'weight' and can be hard to deal with.","Some datasets might take 40 years to even upload, let alone process.","0 is just data.","0°C is data with context.","0°C in London today is information.","When you are working with real-time analytics, how do you deal with that data?","System clocks become important - do you trust your machine, your cloud server, or the machine in the field?","Do you deliver a constant stream of data?","Sometimes it might be better to time-stamp it and package it up, then send packets of data at regular intervals.","That leads to less chance of drops in connection causing data loss.","The Secret Life of Buildings","It was interesting to head Yodit Stanton's insights into working with “connected“ buildings, particularly some of the pitfalls and unexpected issues.","Her company opensensors.io deals with commercial spaces with pre-existing Building Management Systems (BMSs).","Automation in a machine-to-machine sense (M2M) has been around for years: office lights that turn on when you enter the room, automatic doors, etc.","Learning about a building's 'health' in the form of lighting-levels, environmental factors, footfall, etc. can give you insight into employee absenteeism, efficiency of your workforce, and on and on...","Average cost of one desk in London is £12k p.a.","Small percentage increases in efficiency can reap big rewards.","Legacy protocols (bacnet, dali, modbus, knx) are building-data standards and will be around for a while.","Contracts are typically 15-20 years long, and cost between £50k and £200k.","They won't be thrown out anytime soon.","They lock down their data too much.","None of those services have an API.","Open Source integrations are more sustainable and easier to manage.","Don't use WiFi - there are “edge dependencies” on people (in one example, an office manager unplugged the wi-fi router every night!).","It's none of your effing business","The award for most curmudgeonly speaker at the event goes to Boris Adryan (according to ThingMonk host James Governor).","Boris devoted a large portion of his talk to dissecting an article depicting a near-future IoT-fueled wakeup routine.","The post is well-worth a read.","He also touched on machine learning, and I started craving a cup of tea and took fewer notes...","In short, we need our systems to be smart because people instinctively lack an understanding of statistical p-values.","Things, People, and Beer","Craig Cmehil, from SAP, subtitled his talk “voice as an alternative user interface for analytics”.","He gave a demonstration of the voice-activated Amazon Echo, essentially a programmable Siri in a large box.","He ran a mini competition, too, where he gave away an Echo (not yet available in Europe) which involved some live-coding: always a brave mood in a live presentation...","IoT Device Management","Juan Perez (from Microsoft) said some stuff about Azure (a new MS Cloud platform or something-or-other).","I may have slept through this one...","Welcome to the Conversation","The official father of the term “Silicon Roundabout”, Matt Biddulph, closed day one with a talk about how 'conversation' can help solve some of the main issues with many IoT devices.","He was very coy about his new venture, Thington, but hinted that it will be using this concept of conversation in conjunction with the “social graph” when it launches in early 2016.","Traditional light-bulbs are an interface an messaging bus.","A light-switch is a state messaging transport.","Adding connectivity to a bulb (a la Philips Hue) adds issues:","latency","state","consensus","trust","location","power","There's an 'old' scholarly article that outlines the 8 fallacies of distributed computing:","The network is reliable.","Latency is zero.","Bandwidth is infinite.","The network is secure.","Topology doesn't change.","There is one administrator.","Transport cost is zero.","The network is homogeneous.","Matt's “New fallacies of Distributes Services”:","Devices are powered.","Devices are reachable.","Devices stay in one place.","Messages can be trusted.","Causality is unambiguous.","All device state is known.","There is one clock.","How do we solve these issues?","Simple: a conversation.","If the devices can 'talk' to one-another, they can iron out inconsistencies and neutralize many of the problems.","At some point in the next week or so (mid-late Dec 2015) I'll post my notes from day 2 of ThingMonk.","I'll add the link here, but if you're really desperate not to miss out, drop me a line (tom@tomhazledine.com) and I'll add you to the slowly-growing list of people who I'll email when I've published the notes.","[2]","These are my notes from day one of the conference.","You can read my notes from the second day here.","↩︎","You can read my notes from the second day here.","↩︎"]},{"title":"Notes from ThingMonk: Day Two","url":"/thingmonk-day-two/","content":["[1]ThingMonk is a conference “for developers, designers, data wranglers and decision-makers that want to turn ideas and concepts into industrial scale systems".","If day one of ThingMonk was mostly about the internet of things at near-scale - IoT for our daily routine, in our houses and in our offices – day two had a more “big picture” focus to it.","The headline talks were about large thorny environmental problems, and how big-industry is using IoT to make enormous efficiency savings.","Day Two","The Ambient Kettle Project","Andy Stanford-Clark, from IBM, kicked-off day two by introducing us to the Hy Pi Zero.","The original Hy Pi had debuted at ThingMonk 2014 and was a Raspberry Pi powered by a hydrogen fuel cell.","The Hy Pi Zero was the same concept, only this time using the newly-released Raspberry Pi Zero.","It had taken the team at Arcola all the preceding day to get it up-and-running, and was now powering the slide-deck for the presentation.","He then handed us over to his colleague, Laura Cowen, who talked us through the "Ambient Kettle" she and Andy had been working on.","It had started when Laura made her kettle post to a custom twitter account every time she made a cup of tea.","Her mum then started following the kettle's account, and Laura tried to think of ways to streamline the "ambient intimacy" her and her mum were experiencing.","By abstracting the fundamental characteristics of the experience, Laura and Andy then experimented putting those elements together independently of the act of making a cup of tea itself.","They identified many UX facets: the sound of the water boiling and the click of the kettle, the shape and feel of the physical object, the colour of the kettle's indicator lights as it boiled, the fact that the experience needed no","configuration at all.","Iterating over various prototypes, they built a pseudo-kettle that severed as a remote indicator that someone (presumably someone you care about) has just made a cup of tea.","The end result was a tiny dolls-house scale 3D-printed kettle with an LED and speaker.","The Convergence of IoT and Energy","The real show-stopper talk of the conference was given by Tom Raftery, an analyst at GreenMonk (the environment-focused arm of RedMonk; the event organisers).","Energy networks are built around spikes and "peak load" (Bake Off finishes, and everyone up and down the country puts the kettle on and flushes the loo).","2°C = cap for global warming set by 2009 Copenhagen Climate Accord.","565GtCO2 = Gigatons of CO2 needed to reach a 2°C increase (a carbon budget that leaves us with 20-30 years at current emissions rates).","2795GtCO2 = Proven reserves.","We've now passed the 400 parts-per-million CO2 threshold in the atmosphere, and will never cross it again.","The stats from the Scripps Institution of Oceanography make for scary reading.","Electricity Grids are dumb.","They only learn about outages when customers phone up to complain.","They then need a truck-roll (apparently the biz-speak for “sending a truck”) to diagnose the problem, and then a second truck-roll with the specialist equipment to deal with the issue.","The grid is provisioned for 'hits' – the two weeks of the year with peak-demand dictate the infrastructure for the whole system.","Wind and solar are unpredictable – you can't just order-up more capacity to match demand.","However, Denmark already meets 140% of its energy demand with wind-power, allowing them to sell-on excess supply.","Nuclear doesn't scale easily: there's a 10-20 year runway for new projects.","1GW is roughly the output of one nuclear power station.","China are building 700GW worth of solar production.","Industrial IoT is often divided into two concerns: Predictive Maintenance and Demand Management.","Dynamic management of energy demand could be the silver-bullet to deal with the problem of un-environmental energy production.","Connected devices can look at energy prices, and when the price drops suck up all the energy they can store.","Increased storage capacity helps even out the miss-match between supply and demand.","Energy from renewables is always cheaper: fixed costs mean 'old' providers drop out of the market when the price drops below a certain point.","Renewables can sell at any price.","Becoming a Digital Industrial Company","Jeremiah Stone from GE spoke about how his company is transitioning from a traditional heavy-industry & manufacturing company into one that fully realises the potential of 'digital'.","His colleague Stephan King then gave a demo of their new IoT cloud platform (all buzz-words, little substance, very boring).","GE are adding analytics infrastructure to their massive fleet of trains * Data fuels insight: using their new analytics, they discovered they needed to wash jet engines that operate in dusty environments, saving themselves expensive","replacements (and avoiding potential fleet-grounding from the FAA) and saving $7m p.a. in fuel costs.","Every extra 1mph in capacity generated on the US rail network equates to $200k in savings.","In windfarms, turbines create a cone of turbulence behind them.","Using 'smart' blades that can adjust their angle based on live data means they can reduce turbulence and increase efficiency across the whole farm.","Log, I Am Your Father","James Hodge & Matt Davies came from Splunk (one of the event sponsors).","Splunk deal with data analytics.","A lot of data already exists, unused, in log files: we just don't know we have it.","Data like this can fuel the “Build->Measure->Learn” cycle: Have an idea, build a product, measure the data that product creates, and learn from that data to generate new ideas.","We Need to Talk About Telco","Keith O'Byrne works at Asavie, a large-scale telecommunications infrastructure company.","Connectivity is a big issue for IoT projects.","No-one will run a copper cable into a power station.","In that scenario, connectivity must be wireless.","All IoT plans start like this: “step one: assume internet”.","Tesla (the electric car people) addressed this problem badly - they routed the in-car WiFi hotspot through Spain.","It seemed an economical solution, but had side-effects: Google was in Spanish, and location-based services (such as BBC iPlayer in the UK) would not work.","If you need to use a SIM in your project, use a Global Sim.","They have better connectivity because they can use any network, the support and API is much better, and there's safety in numbers - most big systems use them.","The only downside is they don't have a phone number, so take a bit more tech-savvy to get them working.","Industry Things at Scale","Redmonk analyst Tomas Grassl spoke about the difference between consumer and business IoT.","Consumer IoT has many areas (health, household appliances, toys, etc.) but the business space is where the revolutionary work is being done.","Big ports cannot simply build new roads into big cities.","They need to leverage efficiencies in order to grow.","E.g. routing trucks to overflow carparks based on live data.","Smart bathrooms in sports-stadia allow for efficient management of supply (water, soap, loo-roll etc.) to deal with massive demand in short time-windows.","$447bn is spent annually on maintenance.","Pirelli tires for trucks now measure speed, heat, pressure, etc.","They can be replaced before they fail, and the company has changed to a new business model: leasing the tires rather than selling them.","James Gosling's IoT talk is well worth a watch, apparently.","Brought to You by the Number Ten","Recent AWS aqui-hiree Kyle Roche recreates the Eames' famous Powers of Ten video, but talking about IoT tech at the different scales, with various insights along the way.","When automatic gunshot detectors were installed in New Jersey, authorities discovers that only 38% of gunshots were reported by humans.","The F-35 jet's HUD allows the pilot to see 360° 'through' the plane.","It also orders its own replacement parts.","From source to house, the US mains loses 16% of its water.","From Killer Robot to Killer Product","Pat Patterson is a “Developer Evangelist Architect” for event-sponsor SalesForce.","His job sounds amazing: traveling from event to event showcasing interesting techy things.","A snazzy demo can be very useful and powerful as a tool to drive engagement (his main example was an industrial robotic arm, normally used for manufacturing, that he'd taught to do a dance).","We're living in an age of cheap components - creating a working prototype for an IoT project can be very quick and very affordable.","[2]On the whole, ThingMonk 2015 was amazing.","I met loads of people working on all sorts of interesting projects, and it hammered home to me that the IoT revolution has already started.","I've come away inspired to get even more stuck-in to my IoT side-projects, and with a feeling that it is now essential to start bringing IoT concepts into my day-to-day work too.","These are my notes from day two of the conference.","If you want to catch-up, read my notes from the first day of ThingMonk.","↩︎","If you missed them, don't forget to check out my notes from day one of ThingMonk.","↩︎"]},{"title":"Which do you choose: native app or web app?","url":"/web-apps/","content":["Native apps are the ones you download to a device and run as a discrete application (hence the name).","So-called web-apps perform similar tasks, but run in your browser.","But whether we can even consider a website to be an “app” is an altogether more contentious issue.","Despite what some people might say, I do think a “web-app” is a real thing.","Not only that, I think web apps are one of the best things a startup (or anyone, for that matter) can invest their time in.","There's no denying that the boundaries between web-apps and web-sites are blurry.","But so are most things we try to define.","Arguing about it is nothing more than a distraction.","Why is the existence of web apps contentious?","It's easy to understand why people see websites and native apps as separate entities.","They have different underlying technologies and usage patterns, after all.","And it is the usage that I think sways the argument in favour of web apps.","They are websites, but we use them as if they are apps.","Of course, the difference only matters when you're arguing about it.","Everyone else just gets on with making things.","Whether you call it an ‘app’ or a ‘site’ has no impact at all on what your creation actually does.","I find I just know which is which.","Most websites don't have that instinctive "app feel".","E-commerce sites often have complex user-login areas that feel app-like.","But I would class those as web-sites.","Post some new content using WordPress and you'll encounter many full-blown app characteristics.","You need to login; you're accomplishing user-centric tasks; you have your own personalised dashboard.","Despite all this, I'd still call the vast majority of WordPress installs web-sites.","I pleased to report that the web-apps I concern myself with don't lie in this grey-area.","I'm not saying there isn't an issue here, I'm just kicking it down the path.","The fuzziness is someone-else's problem; I've got work to do.","Why choose one over the other?","One main reason native apps are so popular on iPads and iPhones is the iOS browsing experience.","It sucks, big-time.","Want to switch back to a tab you were on seconds ago?","Sorry, you're going to have to load the page again!","Looked away for a nanosecond?","Load the page again!","Native apps don't have this problem.","Build a native app and you have full control over your users' experience.","Native apps can also tap into device features with more ease than something running in a browser can.","Want to access files, sync with the cloud, or authenticate using an external service?","All much more straightforward (for the end user) in an app.","It's easy to see the appeal of a native app.","But there are pitfalls and gotchas for a tech startup building their first product.","When using a real browser on a real computer, the web wins hands-down every time.","Why not have both?","In most cases, an app will have a browser version and a native app version.","This is a sign of a mature product: the users can do what they need to do wherever they feel most comfortable.","Where the distinction comes into play is in the early stages of a new venture.","Resources are scarce early on.","Budgets and deadlines will be tight.","Your (small) team's technical skills will generally fall into a few key competencies.","In these cases, you need to make hard decisions, if only for the sake of expediency.","Don't develop your minimum viable product for every platform available.","That would be a staggering waste of resources.","Maintaining two (or more) separate codebases will slow you down at the best of times.","When you're still working on product/market fit, that slowness can be debilitating.","This is where web apps come into their own.","The web just works.","Sure, there may be some technical challenges.","Keeping a consistent user experience across platforms is hard, especially when using cutting-edge features.","But get it right and your app will run anywhere that can open a website.","And that's pretty much every device out there.","As soon as you choose a native platform you're closing off your app to a whole swathe of the market.","Develop an iOS app, and now no-one on an Android device can use your product.","Or anyone on a desktop, or anyone on Windows or Linux or whatever else they might be using.","Make your app as a website and it will run everywhere (if you've done it right).","How easy is it to publish updates?","It's all well and good releasing Version One of your app, but you'll always need to make updates.","And if you're still working towards product/market fit, regular updates are essential.","The ability to stay nimble is crucial if you're working in a Lean or Agile environment.","If you've made a native app, pushing updates is tedious at best.","Every time you want to make a change you need to go through the rigmarole of submitting your new code to the App Store.","And even once you've passed the App Store code review and approval, that's not the end of the story.","You still need your users to download the update (and it's far from certain they'll actually do that at all).","If you're publishing your own web-app, deploying a new update or bugfix is much simpler.","Just push your code repository live (and maybe wait for your caches to clear).","No outside approval needed.","You don't even need your users' permission.","With a web-app, the power lies in your own hands.","Planning for the future and dealing with growth","At some point you will need to choose one path over another.","When you do, you should think about the time when you'll be able to branch out again.","If you only have the resources to build a web-app or a native app, don't forget to look ahead to when you can have both.","If your app is a success, you'll want to spread to as many platforms as possible.","The ultimate aim is, of course, to give your customers as much flexibility and convenience as you can.","Alas, bouncing from one platform to another is not a straightforward process.","You can't just take code written for iOS and run it on an Android device, for instance.","One of the steps in the creation of any decent app is the creation of a decent API.","The API will power the bulk of your app's functionality, and you can plug any front-end you like into it.","When you expand to other platforms, having a good API means you need not rebuild the entire app from scratch.","All you need to do is build a new endpoint.","The API approach helps whether your going native or web-based first.","But it's not a silver bullet: building the endpoint will still take time and effort.","But developing an API is always easier if your endpoint prototype is web-based.","So when prototyping is over, why not reuse the code you've already written for Version One of your app?","Hiring","When it comes to hiring developers to work on your project, the ubiquity of the web is your friend.","Talent and experience in web apps are far more common traits than native app skills.","Sticking to the fundamental technologies of the web gives you more hiring options.","Web Apps are the best option for Getting Stuff Done™.","If you have validated your great idea and are ready to build your MVP, take a serious look at building a web app.","It may well save you a lot of time and effort, and will be a strong step in your journey to greatness."]},{"title":"Don't turn your problem into your users' problem","url":"/your-problem-is-not-your-users-problem/","content":["I'm noticing a tendency for organisations to pass their problems along to their customers, and I wish it didn't happen.","At its simplest, this issue can be demonstrated by a basic form asking a user to input their name.","It's common to see two separate fields for first-name and surname, as opposed to one box.","As a developer it's easier to parse and save structured data when it's more precise, so I can see why I'd want to receive a user's name in two parts[1].","But all I'm doing is passing that task to the user and saving myself the tricky task of deciphering their potentially-vague input.","Generally speaking, anything worthwhile has complexity.","The complexity has to live somewhere, and most of the time it's up to developers to decide where it goes.","Any user-facing interface should strive to make the task as simple as possible for the user, above all other things.","This is particularly relevant for e-commerce, where the simplicity of the users' experience correlates directly to money-made.","Every obstacle you put in the users' path (an extra form-field to fill out, another button to press, an extra decision for them to make) can have a percentage-point impact on the conversion rate.","The simpler your process, the fewer people will give up partway through.","Holiday letting companies are particular offenders in this area.","Tied to systems that rely on fixed changeover-days but eager to maximise their availability, they resort to complex short-break rules.","You can book on a Friday for three nights, provided that the preceding Monday is already booked and the booking doesn't cross a price-band boundary in the company's admin system... ouch!","Making these clear to the user is a non-trivial task, and slap-dash implementations often leave the user stabbing in the dark and praying they aren't presented with yet another incomprehensible "your requested booking does not comply","with short-break rules, please try again" message.","Conversely, sometimes breaking the input into two-parts can help> the user.","There's an argument to made for directing our users to the desired input: when presented with an ambiguous box, the user might not know whether you want their full name or just their first or last name, or their title, or maybe just their","initials?","Giving them a set of discrete input-fields can make their task clearer to them.","This issue can also be addressed by adding sensible placeholder text to the one-box approach.","This might all sound overly simplistic, but the same principles apply at all levels of complexity.","↩︎"]},{"title":"You are only as good as your README","url":"/a-good-readme/","content":["Our programming languages substitute the complex logical underpinnings with easy-to-remember words.","Typing for or while is an abstraction that suits our brains better than a string of ones and zeros.","We program using languages we understand.","Code is not for computers, it's for us.","And our code is not just written for ourselves.","Other developers will need to read our code and understand it.","If we are writing something useful, other people will want to incorporate it in their own work.","If we are writing code for a client, somewhere down the line someone else will have to support that code.","If you want your code to have a life beyond your own computer, other people will need to be able to read and understand it.","We write code for other developers.","This is why we comment our code.","This is why we use human-readable names for our variables (and let our compilation scripts reduce them to memory-efficient abstractions for us).","This is why we respect coding standards and make damned-sure all our indentation is consistent and accurate.","This is also why the most important file in any project is not your index.html or your package.json or your composer.lock.","The most important file in your project is your README.md.","The README file is the simplest file in your project.","It doesn't control your routing or manage dependencies or run any code at all.","It's nothing more than a simple text file.","Your README is where you tell people what you were doing, what tools you used for what you were doing, and how you did what you were doing.","And if you're feeling really generous you can tell people how they can add to what you were doing.","For a new set of eyes looking at your project, the README is their orientation.","It's the first place they look, and it tells them where they should look next.","When we write code we are writing for other developers, and if our READMEs aren't useful we may as well not be writing at all."]},{"title":"n-minute read: calculating an average reading speed","url":"/calculating-reading-speed/","content":["I'm interested in measuring what effect, if any, displaying an approximate reading-time for articles will have on readers of my site[1].","Will people be put off by knowing in advance that an article will take a long time to read?","Will they gravitate to shorter pieces, or avoid them because they're looking for more in-depth content?","Will it have no effect at all?","My hypothesis is this: adding a read-time to post summaries will increase click-throughs from the archive page to full-post views.","It reduces the "mystery meat" problem by providing users with objective data about their options, meaning they are better prepared to make a decision about which post to read.","Because this is an increasingly common idea[2], there are many examples of reading-time-calculation functions out there.","I've borrowed and adapted concepts from all over the place, but diverged from most I saw in two main ways.","Firstly, where others have focused on concision, I have opted for a more verbose style.","One of my goals in writing pieces like this is to fully understand the code I'm working with.","Breaking down each discrete operation into its own line (and compulsively commenting everything) ensures I know what every part of the code is doing (and hopefully makes it easier for readers to understand, too).","Secondly, I've chosen to separate the calculation-logic from display-logic.","The working-out of the length of time the content will take to read just needs to output a number.","Turning that number into human-readable text is another process, and thus another function in my solution.","The "Separation of Concerns" is a fundamental programming concept, and ties in nicely with the WordPress rubric: content-manipulation is for plugins, display-stuff is for themes.","If I were planning to release this code, I could make a plugin to handle the calculation and let theme creators outputting the result any which way they choose.","Calculating the time","Here's my final function for calculating the read-time in seconds.","This example is written in JavaScript, but only because that's what I like to write in.","The general gist (count the words, divide by the reading-speed) can be applied to any language.","const readingTime = content => {","// Predefined words-per-minute rate.","const wordsPerMinute = 225;","const wordsPerSecond = wordsPerMinute / 60;","// Count the words in the content.","const wordCount = content.split(" ").length;","// How many seconds (total)?","seconds = Math.floor(wordCount / wordsPerSecond);","return seconds;","};","Displaying the time","When it comes to outputting this calculation to the screen, there are a few steps we need to take.","Firstly, my guess is that very few people have an innate understanding of time when it's expressed as seconds.","Thirty-seconds is a chunk of time I can comprehend, but when we go much above sixty, my instant-comprehension drops dramatically.","Without thinking, do you know how long 438 seconds is?","I certainly don't.[","3]","The simplest option would be to convert our time-in-seconds into mm:ss format.","And "Minutes and seconds" is a pretty standard way to show short periods of time, so it would solve the intuitive-comprehension problem.","In essence, this boils down to converting a decimal number (in base ten) to a sexagesimal number (base sixty).","Getting the number of minutes is easy: we just divide the number of seconds by 60 (the number of seconds in a minute), and round down to the nearest whole number:","const minuteCount = Math.floor(seconds / 60);","Getting the number of seconds is a little tricker, but here modular arithmetic is our friend.","What we want is the remainder (a.k.a. the modulus) from the minute-calculation, which we get by using the modulo operator:","const minuteRemainder = seconds % 60;","Getting fancy with our output","Simply showing users the minutes-and-seconds that our calculation spits out doesn't feel right to me.","This is, after all, a very rough approximation of reading-time, as no two people read at exactly the same speed.","I'd far rather show a more human result.","The simplest (but most verbose) way to get the effect I was after was to simply set a custom message if the number of seconds fell within a certain range:","let message;","if ( seconds < 30 ) {","message = 'hardly any time at all.'",";","} elseif  ( seconds < 50 ) {","message = 'less than a minute.'",";","} elseif  ( seconds < 55 ) {","message = 'nearly a minute.'",";","// ...","But this could get tiresome pretty quickly, and I much prefer a more dynamic approach.","After a bit of tinkering I finally settled on a system that created a message based around how close the total-length was to a round minute.","If the result is within two seconds of a minute (e.g. 122 seconds, or 239 seconds), then the result reads "minuteCount minutes, on the nose", and so on and so forth.","// ...","} else if ( minuteRemainder < 2 || minuteRemainder > 58 ) {","// If we're within +/- 2 seconds of a minute:","message = `${minuteCount} minutes, on the nose.","`;","} elseif ( minuteRemainder > 50 ) {","// If we're within less than 10 seconds short of any minute:","message = `just shy of ${minuteCount} minutes.","`;","// ...","The last challenge was to convert the raw integers (1, 2, 3...) into strings ("one", "two", "three"...).","To do this I've simplified Karl Rixon's much more comprehensive solution.","His function handles virtually every number conceivable, with support for positives, negatives, decimal-points - the works!","All we need to do for our current goal is convert the basic integers into strings, and we don't even need to go up very far up the number-line, either.","It's unlikely I'll be writing anything that takes more than ten or fifteen minutes to read.","const dictionary  = {}","0: 'zero',","1: 'one',","2: 'two',","3: 'three',","4: 'four',","// ... etc.",");","const string = dictionary[number];","View the full convertNumberToWords() function I used in this Gist.","This version uses PHP because it was written for a WordPress site.","Putting it all together","With all our functions created, all that's left is to put it all together and hook the whole thing up to the content we want to measure (in this instance the WordPress post content):","// Get reading time.","const readingTime = readingTime(content);","const readingTimeString = parseReadTime(readingTime);","Whether or not this will actually achieve anything remains to be seen.","Will people click-through more when they know how much time they'll need to commit up-front?","Maybe, maybe not.","Either way, I've enjoyed diving into the problem (probably too deep).","Medium is the obvious example, but I've seen it used effectively on other sites, too.","The most recent example I saw was on the Boagworld site.","↩︎","This is, of course, on a very trivial scale.","But the idea that displaying more information (provided it's easily digestible) leads to more decisive action by users has wider implications.","Most of my day-to-day work is focused around "moving the needle" on conversion rates.","A percentage-point increase in click-throughs to a checkout represents a Job Well Done.","↩︎","A big assumption in this project is that there is a uniform reading rate, which there really isn't.","Most code examples I looked at stuck to a standard words-per-minute rate of 200, but I (very unscientifically) tested my own reading-speed and found it to be closer to 300wpm.","The real task at hand here is to give people a rough-estimate, so erring on the side of caution (going at the pace of the slowest) is probably the best option.","I eventually settled on 225wpm as a usable middle-ground.","↩︎"]},{"title":"Well-written HTML doesn't need any styling. Except that it does.","url":"/html-doesnt-need-any-styling/","content":["One of the best things about HTML is that it just works.","As with much of the web, things only get weird when designers and developers start adding things.","By default, a raw HTML file will render quite nicely in every browser on every device.","Hierarchy is clear, lists look like lists, tables look like tables, we have bold text and italics, and links (the ‘hyper’ in HTML) are obvious.","Lately I've discovered a few bloggers who have embraced this quality.","Dan Luu, for instance, is such a developers'-developer that he doesn't even have a stylesheet on his blog.","All that matters is the content, free from artifice or sneaky designer-tricks.","His writing stands or falls on its own merits.","HTML has everything he needs, so why include anything else?","Myself, I disagree.","Reading his articles is hard, even though they are expertly written.","The aesthetic works well for him - it suits the style of his writing, and emphasises his credentials - but there's one aspect that makes his pages nigh-on unreadable on all but a few devices: the lines are simply too long.","My monitor at home is bigger than most, but not huge by modern standards.","At this size, most paragraphs appear as just a couple of lines stretching the width of the screen.","There's a reason books are the size and proportions that they are: there is an upper limit to the line-length that a person can comfortably read.","Counter to what you might expect, our eyes don't smoothly follow the lines on the page.","When we read our eyes are actually jumping back and forth in very small movements[1].","If the lines get too long, our brain struggles to tell which line is which and the movements become unaligned: everything scrambles, and we have to work hard to find our place again.","Below a certain length (usually somewhere between 55 and 75 characters) our eyes can easily discern where the next line begins.","The spacing between the lines also plays a part here (known as "leading" after the strips of lead placed between the lines of text in letterpress printing).","Too wide or too tight a space, and our eyes struggle to get their bearings.","The best books have just the right balance of leading and line-length.","Raw HTML, however, is not at all bothered by the lengths of its lines.","Super widescreen display?","HTML will keep those lines running from edge to edge, readability be damned.","There is, of course, an easy fix.","All we need to do is add three lines of CSS to our HTML page, and everything looks much better.","body {","max-width: 50em;","padding: 2em 0.5em;","margin: 0 auto;","}","Here we're setting a max-width: if we used a simple width, we'd force the page to be that wide even on small screens (so we'd have to scroll sideways to see all our content).","This runs counter to the spirit of HTML, which fits to any available space, so using max-width is better (if the screen is larger than 50em, it will cap the line-length, otherwise it lets HTML do its thing).","Notice, too, that we're using em units.","These are linked to the size of the type (literally the width of a letter "m"), so we can be sure the width will be proportional to the text.","As for the rest of the rules, the padding is making sure the text is never jammed uncomfortably close to the edges of the screen, and the margin: 0 auto; rule then centres the block in the middle of the screen.","That's not a lot of styling, but it has a tremendous impact on the readability of a web page.","One of the benefits of presenting your content in nothing but HTML is that the browser only has to make a single file request to the server.","Traditional CSS lives in its own file, and is therefore another request.","Normally that's no problem at all, but in this context it's literally doubling the amount of load the page places on the server.","The simple get-out for this is to in-line the CSS rules within the HTML file.","It's not stylistically "pure" or "best practice", but does actually work.","So next time you're swept up in the machismo of a HTML-purist rallying cry, spare a thought for your readers and think about your line-length.","The tiny eye movements are called saccades.","Our brain is taking-in the whole block of text at once, not just the specific word we've got to.","It's similar to how we perceive paintings: our eyes can only focus on one tiny part at any one time, but we "see" the picture as a whole.","As an interesting aside, the size of the in-focus area you can see at any one time is easy to demonstrate.","Hold your arm out in front of you with your thumb pointing up: the in-focus area is roughly the size of your thumbnail.","Our brains fill in the rest while our eyes dart around faster than we can perceive.","↩︎"]},{"title":"Bullet Journal Revisited","url":"/bullet-journal-revisited/","content":["My original analogue attempt at maintaining a Bullet Journal","The "Bullet Journal"[1] is a system for Getting Things Done.","One nested pen-and-paper list that gets rewritten every morning.","My first post on this site was about the success I was having with the Bullet Journal.","Over the last few weeks I've begun to use the journal system again, so I figured it was time for an update.","I used the Bullet Journal daily for about ten months (covering most of my time as a freelance developer).","When an agency recruited me, I stopped using it.","They had their own task-management tools, and I no longer had such an urgent need to organize my own time.","Fast-forward several years and I was beginning to feel the lack of a personal organization system.","I was getting stuff done in work hours, for sure, but momentum was slipping on all my "side projects".","I don't get people who struggle to come up with ideas for projects: I'm overwhelmed by ideas.","What I find hard is execution.","Getting Things Done is my Achilles heel.","It was time to revisit the Bullet Journal.","Going digital.","When I kept the daily task-list, I had a dedicated paper notebook.","Keeping things analogue (as per the "pure" Bullet Journal method) was a way of maintaining focus.","But now I'm much more comfortable keeping everything in the "cloud".","I keep all my notes in a private Dropbox folder so I can access them wherever I am, regardless of what device I'm using.","It also allows me to use my favourite writing apps[2].","Constant access to my files has removed the downsides of keeping a digital list.","In 2013 I was more comfortable carrying a physical notebook around with me.","Now, I find it more convenient to keep a folder full of markdown files in the cloud.","Because I spend so much time working with markdown documents, and because markdown has such an elegant way of handling bulleted lists (with checkboxes!), adapting the Bullet Journal method was a natural choice.","My evolved Bullet Journal system.","The central tenets of my journal are the same as the "official" system.","I have one task list, and that gets refreshed every morning.","I remove completed items, and anything still outstanding gets re-evaluated[3].","Because the list is digital, I don't need to worry about adding a "migrated" marker when a task rolls-over into the next day.","If it's done, it gets deleted.","If I want it to "migrate" to the next day's list, it stays where it is.","I've also moved away from the other markers used in the original setup.","Sticking to the nested list works nicely for me, and I found the extra markers for events etc. made my lists over-complicated and overwhelming.","I nest my list by topic.","By way of example, here's a snippet of a list I made for adding a new feature to this very site:","-   [ ] My personal website // This is the "master" list item","-   [ ] readtime calculation","_ [x] calculate seconds based on words-per-minute // I have completed this item.","_ [ ] parse time into human-readable string \\* [ ] write a blog post about the process + [ ] another feature","// ...","When I kept my analogue journal, I would create "super objectives" for every month.","The idea was that I could refer to those pages and make sure I was keeping up with my "big picture" goals.","This was always a cumbersome process, and these super-objectives were often overlooked.","Space is not a factor for my digital journal, so I can afford to add as many items as I like.","I even break those items into groups if the mood takes me.","Right now I have "Big goals", "Technologies to learn (and master?)","", "Dream goals", and "Short-term tasks".","Big goals is at top of the page and has all my serious objectives for the year ahead (this is a new addition for 2017 - it's early days yet, so we'll see how this pans out...).","My hope is that by keeping these items front-and-center means I can't ignore them.","I've discovered there are few things more disheartening than finding a year-old task list and not being able to tick-off a single item.","Technologies to learn (and master?)","helps me keep on top of the things I'm trying to learn.","Sample entries at the moment include vue.js & web audio API.","Dream goals is more whimsical.","These are things I'd like to do, but am unlikely to find time for.","But by writing them down I can justify a major change in tack if a great opportunity comes along.","(“Sure I can blow some money on a fancy copper alembic-still; make some whisky is on my task list!”).","It also helps to keep my procrastination in-check.","If I find myself wasting time on some shiny new idea, I can't use the old "well, I've always dreamt of doing this" excuse.","If I cared enough about it, I'd have put it on the list!","Short-term tasks is the most mundane section, but also the most used and the most useful section.","Lately I've been focusing on improving this website.","As a result, my list is full of items like setup LetsEncrypt on server and filter out lang-spam in G.","Analytics.","Most of the items in this section are getting checked off, which is a signal that I'm close to my goal[4].","Time to start thinking about the next short-term goal...","In the past, it has always been the time in-between projects when my focus has dipped.","And once I lose my focus, I find it very hard to get back into the groove again.","Predicting an imminent "danger zone" helps me sustain a higher level of productivity.","Update:","Coincidentally, it appears I'm not the only one who's been investigating the Bullet Journal lately:","trends.embed.renderExploreWidget(\"TIMESERIES\", {\"comparisonItem\":[{\"keyword\":\"bullet journal\",\"geo\":\"\",\"time\":\"today 5-y\"}],\"category\":0,\"property\":\"\"},","{\"exploreQuery\":\"q=bullet%20journal\",\"guestPath\":\"https://www.google.co.uk:443/trends/embed/\"});","After this post was written, I checked Google's Search Trends tool and saw a surprising spike.","My original 2013 article always sees a uplift in traffic around the new year, but the last twelve months have seen consistent growth that correlates with the general Google trend.","My hope is that everyone who's trying out the Bullet Journal (myself included) can turn that interest into a genuine boost for productivity.","Credit for the Bullet Journal concept goes to digital product designer Ryder Carrol.","↩︎","My favourite writing apps are Sublime Text on my laptops and desktop, and Byword on phone and tablet.","I do all my writing in Markdown (be it drafts for complicated emails, notes on work projects, or aide-mémoire).","↩︎","The daily re-evaluation of all uncompleted tasks keeps the list fresh.","↩︎","The "new year refresh" has passed the 80% complete mark.","That's farther than I've ever got before.","Time to move on to the next project!","↩︎"]},{"title":"Getting to grips with SVG markup","url":"/svg-markup/","content":["Here is a side-by-side comparison of an icon.","The image on the left is an SVG, the one on the right is a raster PNG.","They are both roughly the same file size (~1Kb), and at 64px x 64px they look pretty similar.","But see how those same files look at double-scale:","*If you're reading this post through some kind of syndication feed (RSS, etc.) you may need to visit the original post to view this image-demo correctly.","*","An .","SVG and .","PNG icon, side-by-side","The same files, increased to double-size","...and to 10x the original size","We can use the same SVG file for all sizes, and produce consistently clear and crisp results.","To get the same effect with rasterized images, we'd need to create separate files for each size.","Each increase in visible size would result in an increase of file size, too.","We'd not only be pushing up the number of HTTP requests but also the total page-weight as well.","The basics of the SVG markup language","SVG as a format needn't be scary or intimidating.","SVG stands for Scalable Vector Graphic, and the language of SVGs is an XML-derived language like RSS feeds and XHTML[1].","Once you know what tags to look out for, the whole thing gets a lot simpler.","Take this simplified SVG markup, for example:","<svg viewBox="0 0 100 100">","<path d="M10 10H90V90H10Z"/>","<circle cx="50" cy="50" r="40"/>","</svg>","Even without being able to read the more technical aspects, there are clues to pick up on.","It's clear that the SVG above contains a path (a.k.a. a line) and a circle.","Dive a little deeper and you'll see that it's a circle inside a square.","The circle has a diameter of 80 ("r" == "radius"), and sits inside a 90 x 90 square (the line joins up with itself to form a shape).","Both of these lie in the centre of a 100 x 100 artboard (a.k.a "view box").",".svg_demo_1_a{fill:#E3E1E1;}",".svg_demo_1_b{fill:none;stroke:#000000;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:10;stroke-dasharray:0,4;}","Real-world SVGs are more complex, of course.","There are plenty of attributes and DOCTYPE declarations to worry about.","But the general principle is the same.","There is an opening tag (<svg>) and closing tag (</svg>) just like <html>.","And we build shapes within the SVG using tags like <path/> and <circle/> and <rect/> (rectangle)[2].","Styling SVGs with CSS","SVGs are complex, for sure, but that very complexity gives them their power.","And we don't need to know the intricacies of the co-ordinate system to harness that power.","All we need to do is recognize the different elements.","This is a language much like HTML, after all, so once we can find the different shapes in an icon we can add classes to them.","<svg viewBox="0 0 100 100">","<path class="outer_square" d="M10 10H90V90H10Z"/>","<circle class="inner_circle" cx="50" cy="50" r="40"/>","</svg>","...which we can then style with CSS:","svg {","background: yellow;","}",".outer_square {","stroke: red;","stroke-width: 1px;","}",".inner_circle {","fill: blue;","}",".svg_demo_2_a{fill:#FCEE21;}",".svg_demo_2_b{fill:none;stroke:#FF0000;stroke-miterlimit:10;}",".svg_demo_2_c{fill:#0000FF;}","This gives us a more fine-grained level of control than a single character in an icon font would.","We can manipulate the styles of each discrete part of an icon with CSS.","Transitions and animations provide an extra level of detail.","Our positioning can be much more precise.","We have a defined and predictable co-ordinate system to work with.","No more fiddling about with baseline alignment of invisible-hitboxed font characters!","There are some fantastically clever and artistic examples of SVG in the wild.","The only limits appear to be the creativity and imagination of the developers who build them.","A browse through CodePen reveals stunning work, but I'll end with a more simple demonstration of my work.","I've always had a soft-spot for loading graphics, and love to build them myself.","I built this one with a two-path SVG and a dash of CSS animation:","See the Pen Hourglass Loader by Tom Hazledine (@tomhazledine) on CodePen.","This is not the place for a debate about the nuances of XHTML vs HTML vs SGML.","If that kind of language-geekery is for you, you can learn more about the *ML "family" from Computerphile.","↩︎","You can read more about the inner-workings of SVGs - including a detailed look at how to construct paths using co-ordinates – on the MDN SVG tutorial pages.","↩︎"]},{"title":"Inline SVG icon sprites are (still) not scary.","url":"/inline-svg-icon-sprites/","content":["You can see SVG icon sprites in use across the high-performance web.","Lonely Planet1, GitHub, & CodePen all use an SVG icon sprite, and there are plenty of advocates for SVG icons across the development community.","These examples come from the cutting-edge of front-end development, but SVG icons are not "hard" or "complicated".","In fact, using an SVG sprite is one of the easiest ways to maintain an icon system.Vector-based images are much better than raster formats (JPG & PNG) for creating icon systems for websites.","They're easy to style with CSS, and have small file-sizes.","The old way of dealing with this was to use icon fonts (where the icons were glyphs in a typeface), but a couple of years ago a better method emerged: SVG icon sprites.Using an SVG spriteWe could add SVG icons to our sites using standard","image-tag syntax – <img src="our_icon.svg"> – but that's not the best option.","Every new <img> we add to the page creates a new HTTP request, which is bad for performance.","The more requests you make, the longer the site takes to load2.","As in the bad-old-days of image-sprites, it makes sense to group all our icons into a single request.","Or even better: no extra requests at all.","We can do this with SVG icon sprites.We call them "sprites", but SVG sprites are much more elegant and programmatic than their rasterized forebears.","The SVG language allows us to group blocks of icon-code in a single file for individual use later.","All our disparate icon files get smushed into one big SVG, which we load like an asset.","Then whenever we want to display a particular icon, we reference the relevant part of the master SVG file.","No matter how many icons we use, or how often we use them, we're only ever including the complex SVG code once.The reason we can do this is because SVG is a XML-like language.","We can mark out different elements using tags in much the same manner as we'd mark-up our HTML.","Instead of using <h1> tags for headings and <a> tags for links, we use tags for lines and shapes.","We can also use tags to group our shapes together.","The <g> tag ("g" == "group") is the obvious choice, but the <symbol> tag work best for our purposes3.","The following code-snippet shows two separate icons within one SVG file:<svg xmlns="http://www.w3.org/2000/svg">","<symbol viewBox="0 0 100 100" id="square_icon">","<path d="M10 10H90V90H10Z"/>","</symbol>","<symbol viewBox="0 0 100 100" id="circle_icon">","<circle cx="50" cy="50" r="40"/>","</symbol>","</svg>","Notice how we've added a standard HTML id attribute to each <symbol>.","This is the "secret sauce" that makes SVG sprites useful.","We can now load our entire SVG at the top of our page and extract the symbols whenever we need them.With the sprite-code in place, using individual icons becomes much simpler.","We can include them whenever and wherever we need them by taking advantage of SVG's <use> tag.<svg>","<use xlink:href="#square_icon"/>","</svg>","And of course we can still add classes to the <svg> tags, allowing us to style an icon in as many different ways as we like.<ul class="example_svg_list">","<li>","<svg class="example_svg_1"><use xlink:href="#example_svg" /></svg>","</li>","<li>","<svg class="example_svg_2"><use xlink:href="#example_svg" /></svg>","</li>","<li>","<svg class="example_svg_3"><use xlink:href="#example_svg" /></svg>","</li>","</ul>","Building an SVG sprite with GulpHand-coding SVG icon sprites can be a real chore.","It gives your site's users the best experience possible, but it's not fun for developers.","Even combining simple SVGs into <symbol> elements inside a single <svg> is not a fun way to spend your time.","And real-world SVGs are never as easy to read as the examples show here.","What we need is a way to automate the sprite-creation process.I automate other front-end tasks using a task runner.","If you've made it this far into an article about creating SVG icon sprites, then I'm going to assume you use one too.","Gulp is my task-runner of choice, but you might be using something like Grunt4.","Gulp already handles my javascript and css, so adding my icons into the mix is a natural progression.Adding an SVG icon sprite to my Gulp setup was quick & painless, and the same task has been serving me well for over two years now.","The gulp-svg-sprite module runs error-free without much need for configuration.","If you've already got a gulpfile.js in your project, you can install the module like this:npm install gulp-svg-sprite --save","That module takes individual SVG files and combines them into a single <svg>.","Each file becomes a <symbol> with an ID matching the original filename.So a folder structure like this:SVG_folder","- icon_one.svg","- second_icon.svg","Becomes a sprite like this:<svg xmlns="http://www.w3.org/2000/svg">","<symbol id="icon_one">","<path d="/* path data */"/>","</symbol>","<symbol id="icon_two">","<path d="/* path data */"/>","</symbol>","</svg>","Once we've installed the module, we need to build our task.","The important things to set are the source folder and destination folder.","I.e. where the task will find the raw SVGs, and where it will put the sprite.","// Load the module.","var svgSprite = require("gulp-svg-sprite");","// Set our desired configuration values.","svgConfig = {","mode: {","// Make sure we're combining icons","// using the <symbol> method.","symbol: true","},","// Some more settings to keep","// the SVG's code clean:","svg: {","xmlDeclaration: false,","doctypeDeclaration: false,","// By default the module wants to namespace","// all our IDs and classes.","We're grownups","// so we want to preserve our settings.","namespaceIDs: false,","namespaceClassnames: false","}","};","// Define our task.","gulp.task("svg", function () {","// Set the source folder.","gulp.src("uncompressed/icons/**/*.svg")","// Include our options.",".pipe(svgSprite(svgConfig))","// Set the destination folder.",".pipe(gulp.dest("assets/icons"));","});","With this task in place, maintaining an SVG icon sprite is easy.","Whenever we add a new icon to our source folder we just need to run gulp svg.","That takes our raw files, cleans them up, and combines them into a single file.Keeping the intermediate files.When creating our sprite, the module first cleans-up our SVGs.","It removes empty tags, strips out any unnecessary attributes, and minifies the files.","This can come in handy if you need easy access to nice, neat SVG code for an individual icon.If you want to target discrete elements within an icon, you may need to include the raw markup in your page.","This can be necessary when you want to build complex animations or transitions.","Being able to copy/paste the nicely-configured code is useful in these situations.You can configure the module to create a folder with the individual SVGs in after they've been optimized and cleaned-up by the module.","You can do this by adding the following rule to the svgConfig object:shape: {","// Choose a folder to store the","// intermediate SVG files in.","dest: "intermediate";","}","Including the sprite in a siteOnce Gulp has created the sprite file for us, we need to add that to our page below your opening <body> tag.","Copy-pasting the contents of sprite.symbol.svg would do the trick.","But we can automate this process by including a reference to our sprite file.Including an SVG icon sprite in a Jekyll siteIn the past I've worked a lot with Jekyll - a static-site generator.","When it comes to performance and security it's hard to beat static HTML files.","Jekyll compiles a site before it gets put on a server.","There are no databases to hack, and no server-side build-steps to slow the site down.","Because Jekyll is already combining lots of files together, adding a sprite into the mix is easy.We can get Gulp to output the sprite into Jekyll's _includes directory by changing the path in gulp.dest().","So .pipe( gulp.dest( 'assets/icons' ) ); becomes .pipe( gulp.dest( '_includes' ) );.","Then you can pull in your sprite with a simple "include" statement:{% include /symbol/svg/sprite.symbol.svg %}","Remember to use Liquid's {% raw %} tag to ensure the SVG code doesn't get escaped.Including an SVG icon sprite in a WordPress siteI built this site in WordPress, which has its own conventions for including files.","When I want to inject code into the markup, `get_template_part()` is the preferred method.","The only downside is that `get_template_part()` only works for PHP files.I get around this obstacle with the following Gulp task:gulp.task("svg_rename", function () {","return gulp",".src("assets/icons/symbol/svg/*.svg")",".pipe(rename("iconsprite.svg.php"))",".pipe(gulp.dest("assets/icons"));","});","That task renames the sprite and saves the renamed file in the assets/icons directory.","Note that we're adding two file extensions: .svg and .php.","Once we've generated this file we include it in our code like this:<?","php get_template_part('assets/icons/iconsprite.svg'); ?",">","Hiding the spriteWhen we load a page with an SVG icon sprite in it, the browser will try to render the sprite code.","This can create a large empty space at the top of our page.","The only time we want our SVG code to appear is when we reference it with a  tag, so we want to hide the raw sprite.","Remember to add a style of `display:none;` to the sprite or it's container.Building and using SVG icon sprites sounds complicated, and it is.","But maintaining a performant and scalable icon system is a non-trivial task.","That complexity has to live somewhere, and icon sprites are by far the best of the options5.","Couple the sprite with a sensible workflow and the whole process becomes second-nature.I've been using SVG icon sprites in production for over two years now.","I have yet to see a simpler solution.","Now that I've got by Gulp setup how I like it, I can be confident that all the tools I need are at my fingertips.","Whenever I need to add a new icon to a site, it's a two-step process.","I save an SVG to my uncompressed/icons folder and drop a corresponding <use> element in my markup.Like many good ideas before them, SVG icon sprites are spreading across the web.","And like Tyler Sticka said: Don't be Table Guy.Ian Feather wrote a great article about when Lonely Planet switched from using an icon-font to SVG icon sprites.","↩The number of requests is a big deal with regular HTTP connection.","HTTP-2, however, changes things; but you'll have to find out more about that elsewhere.","↩We could just use a <g> tag – or even reference individual paths and shapes directly – but <symbol> allows us to add a viewBox attribute.","This means we only have to declare the view-box once for each icon, rather than repeating ourselves every time we use the icon in our page.","↩Note that the raw svg-sprite Node module doesn't read or write to the file system, so you'll need to use something to help you out.","↩It is worth looking at the counter argument in favour of icon fonts, because SVG sprites may not be appropriate for everybody.","Ben Frain probably makes the best case against SVG sprites, but his argument is about changing from fonts to sprites.","Kind of a "if it ain't broke" thesis.","↩"]},{"title":"What is a decibel, anyway?","url":"/what-is-a-decibel-anyway/","content":["The decibel has always confused me.","Sometimes people use decibels as an absolute value where "0" is silence and anything over "120" is very loud.","Journalists do this when describing how loud a band is1.","Sound engineers do it too, when boasting about the power of their speaker-setup. "It's louder than a jet engine at 50 paces, man".","But when I look at the volume meter on a mixing desk, things are different.","The desk shows the loudest value as zero, all the way down to silence at "-96" (or sometimes even "∞").","See what I mean?","Confusing.The mixer section of Logic X (a typical DAW - Digital Audio Workstation)For a long time I've been at least tangentially connected to the world of "pro" audio.","As such, I'd feel uneasy when I overheard a lay-person talking about volume.","I would often catch myself thinking "I know they are getting it wrong, but I don't know why".","So after years of pretty much taking the decibel for granted, I finally had enough.","I did some research, and it turns out the decibel is actually pretty interesting.A decibel is a relative unit.","That means it expresses a ratio, and not an explicit value.Decibels are measured using a logarithmic scale2.","They are a good unit for expressing very small and very large differences, and can do both on the same scale.","This works because they are relative and logarithmic.Decibels are a technical measure of intensity and/or pressure.","Humans don't hear all frequencies equally, so "loudness" is a subjective concept.When decibels are used to express "loudness", (for instance, by journalists) they are used with a fixed point of reference.","In this context, the unit we're using is dBSPL (decibels relative to sound pressure level).dBSPL represent an absolute value of sound pressure.","Just like 20 metres is the same length wherever you go, 20 dBSPL always represents the same level of sound pressure.","To use a decibel this way, we need a fixed point of reference.","The reference point for dBSPL is 20μPa (micropascals) - roughly the limit of sensitivity of the human ear.Levels of >120 dBSPL will damage our ears, but our pain threshold is somewhere around 150 dBSPL.","So be careful, you could do lasting damage without realizing.If you want to reference "loudness" using decibels, then you need to apply a filter of some kind.","This more accurately describes the idea of "volume" as humans understand it.The most common filter is the "A scale" (dBA).","This is a "band-pass" filter, which is less sensitive to very high and very low frequencies.","It's simplistic, but "close-enough" to have become a standard.A less-simplistic approximation of what our ears hear would be a "hearing response curve".","These are much more complex than the "A" scale, with peaks and troughs at different frequencies.10100100010k100k10100100010k100k0+10+20-10-20-30-40-50Gain (dBFS)The graph on the left shows an "A" scale frequency response","curve, while the graph on the right shows one that more closely matches the response of an average human ear.Now that I've done my research, the dB markings on a mixing desk look far more logical.","I can see now that using a simple low-to-high dB meter wouldn't make any sense.","Any dB scale requires a defined reference point, which doesn't exist in this context.","A mixer-channel cannot have a fixed frame of reference because it gets mixed.","A mixer routes all channels to the "master bus" output channel, which has its own volume control.","Knowing the value of a signal at the channel-level tells us nothing about the final output value.But if we flip things around, a channel strip can have a fixed reference point.","That fixed point is the maximum level that it can output.","Because decibels are relative units, we can use them to express how far below the max-level a signal is.","By putting 0dB at the top of the meter (and negative values on the downward scale) we can gain useful information about the level of the signal.There is a technical term for this kind of decibel-usage - dBFS (decibel full scale).","The highest possible level of sound produced by audio equipment is 0 dBFS.","All other levels are then expressed in negative numbers.","Absolute silence in dBFS is -∞ (minus infinity), but in practical terms -96 makes for a useable bottom point.Putting dBFS to use on the web.Much like a mixing desk, the Web Audio API does not use dBSPL.","The final volume depends on both the level set by your operating system and the level of your speakers.","Your computer has no way of knowing what your audio-out connects to (in an analogue setting, at least).","It could be going to a set of tiny tinny earbuds, or a 500w amplifier and massive speaker stack.","Providing a dBSPL reading of the level would be both meaningless and impossible.","For this reason the Web Audio API uses dBFS.In this scenario the dBFS value is useful information.","It lets us know if a channel is "clipping", and gives us the information we need to keep on top of our gain-staging.","It also gives us a value to compare against other channels on the same system.","We may not know the volume of the final destination, but we can compare the individual channels.","If we look at a drum track and a guitar track, for instance, we can use the dBFS values to guide how we mix the levels of the two.Update, 29th March 2017:After it was first published this post made it onto the homepage of Hacker News, where","it generated an interesting comment thread.","I'm grateful to the HN community for pointing out a couple of typos and omissions, as well as raising some interesting points of their own.By nature of my specialties and interests, this post is very web-centric, which means it deals","almost exclusively with digital audio.","Once we make the transition into the analogue world, we start to see positive values for dBFS.","This is known as "headroom", and accounts for the fact that analogue signals are commonly specified in dBV or dBu (0 references of 1 volt and ~0.775 volts, respectively).","“So 0dBFS signal going into your USB audio interface might come out as a +16dBu analog signal, while the same digital signal going into a professional mixing console might come out as a +24dBu analog signal.” korethrThe medium through which","sound (pressure) travels has an effect on how we define decibels.","“In air, dB SPL is referenced to 20 micropascals.","In water, it's 1 micropascal, so the medium matters.” hprotagonistDecibels are not just used in an audio context.","I've now heard tales of them being used in spectrum analysis and anywhere that "pressure" needs to be measured.","“Decibels are just a way to easily express ratios on a logarithmic scale.","It's handy for all sorts of things.” pitajBonus points go to AndrewKemendo for opening my eyes to the bonkers world of "Car Audio SPL Drag Racing".","Apparently some car owners compete to see which car-stereo can reach the highest dBSPL level when playing a single frequency.","“It was instructive to compare frequency and spl because some cars resonant frequencies (fs) were better suited to different fs values of drivers (speakers) and then further down the line certain amplifiers optimized certain frequencies”.","Just bananas!","If you want to learn more about these interesting decibel side-stories, the comment thread for this post on HN makes a good jumping-off point.For instance, this from Wikipedia: The heavy metal band Manowar is one claimant of the title of",""loudest band in the world", citing a measurement of 129.5 dB in 1994 in Hanover.","↩For a deeper dive into the science, check out this fantastically thorough explanation from Australia's UNSW.","↩"]},{"title":"Living with Alexa: the problems with \"voice\" as an interface","url":"/echo-alexa-voice-interface-problems/","content":["Alexa represents a growing trend.","The big tech players are investing heavily in voice-control (in order of current competence: Amazon, Google, Microsoft, Apple).","They see voice as the next step logical interface for technology.","Gone are they days, they argue, when we had to do anything quite so quaint as type on a keyboard.","And in many ways, they're right: being able to say what you want and get an immediate answer can be fantastically useful.","But it doesn't always beat a good-old-fashioned keyboard.","I've lived with Alexa (the virtual assistant held within the Amazon Echo) for six months now.","The experience has changed how I feel about voice-as-an-interface.","Here are a few of my observations:","Failure states need a lot of work, and Alexa hasn't got it right yet.","The "I'm sorry, I'm having trouble understanding right now" message gets more annoying than a 404 or 500 error page ever does.","Also, defaulting to a web-search is not the best option for most queries.","Precisely when you didn't want to have to mess about, the interaction becomes longer and more complicated.","Text input is often more precise (and certainly more concise).","Natural Language Processing has its limits, and sometimes you don't want the hassle of converting your abstract thought into prose.","You can't browse as easily - it's quicker to read than to listen, and easier to bounce-between and assess options.","You can't "skim-listen".","Half the time you feel like an idiot talking aloud to your devices.","It's not discreet: I often want to multi-task, particularly in boring meetings.","(This is more of a future-worry, as I currently only have voice-enabled stuff in my house.)","Alexa isn't able to distinguish between individual voices.","We want (and expect) her to work in all scenarios: whether we've got a cold, or are whispering, or shouting, or in the next room.","As a result, natural-language interpretation is very good, but it responds to any voice.","I think Alexa needs to know who her "boss" is...","Following on from that, security is non-existent.","My friends can hijack the party playlist.","Adverts can inadvertently purchase things from Amazon (this has not happened to us yet, but Alexa has woken up because of wake-words being spoken on the telly).","These is a palpable shortage of "skills" (the Alexa-speak term for what other platforms would call apps), and the ones that are available are either very clunky or too US-centric.","A16Z analyst Benedict Evans thinks it's not surprising that most 'skills' go unused.","“It's hard enough to remember what apps you have when you can see them on a screen - having to remember to ask for them by name is the new command line”.","Amazon are doing their best to kick-start a "skills" ecosystem.","They have an enthusiastic developer-outreach programme with a big marketing budget (based on the targeted adds I can't seem to escape from...).","They've also opened up the technology, so developers can run the (Java-based) Alexa code on their own machines.","Music has been the killer app for us.","Being able to select a song, pause, and skip without having to open any apps or type anything is great.","It's eliminated a friction I didn't even know was there.","Selecting music manually now feels archaic.","Porting my (large) music library into Amazon Music was a pain (and I'm still not finished syncing everything).","It also cost £25 to get enough storage, but being able to select specific artists and songs was totally worth it.","The default music library you get with Prime is fairly comprehensive, but Amazon definitely wants us all to sign-up for their music streaming service (not my bag at all).","I was surprised how often I'd simply say “Alexa, play some music”.","After a little training period (“Alexa, I don't like this!”)","the resulting playlists were actually pretty good.","The "always on" nature of the Echo, while creepy, is one the most game-changing aspects.","It's the factor that tips Alexa from a novelty to a utility.","I just speak into the air, and she responds.","No boot-up time, no log-in required.","She just works.","The speech detection is excellent.","We have the Echo in the living room (tucked behind the telly) and it can still hear me when I'm in the kitchen, even when there are other people in both rooms.","Alexa gets easily confused when multiple people try to talk at once (or talk over each other, which happens a lot when the family are visiting).","But I get easily confused when that happens too, so I can't judge the Echo too harshly on this score.","Getting answers to trivia & general-knowledge questions might be the most useful feature, but the music features are still more impressive when showing off to friends."]},{"title":"I changed my site's font to Comic Sans as an April Fool. It was a disaster.","url":"/comic-sans-april-fool-disaster/","content":["I like to think of myself as a mature typographer[1].","I've spent years obsessing of the minutia of line-lengths and leading and kerning-pairs.","Bringhurst's Elements is always within arm's reach of my desk, but of course it doesn't need to be because I've got it memorized.","I've even reached acceptance in the Kübler-Ross 5-stages when it comes to Comic Sans.","Denial - "Nobody will use Comic Sans now that there are so many good alternatives to choose from."","Anger - "Why are all these idiots still using Comic Sans?!",""","Bargaining - "Let's popularise tools that erase Comic Sans from people's computers.","If it's not available as a default, they won't use it."","Depression - "Why do we even bother designing things?","Most people will choose Comic Sans.","I don't want to live on this planet anymore."","Acceptance - "Comic Sans is actually a well-constructed typeface, and is appropriate in certain situations."","Making the change","For this year's April Fools' Day I thought I'd take that acceptance one step further.","Throwing caution to the wind, I set all the text on my website in Comic Sans.","Thanks to my fancy front-end workflow, it was a simple task.","My styles are all pre-compiled with SASS, so things like colours and font-families are all declared as $variables[2].","// Fonts","$text: "ff-tisa-web-pro", georgia, serif;","$code: Menlo, Monaco, "Andale Mono", "Lucida Console", "Courier New", monospace;","Changing to Comic Sans (or more specifically Comic Sans MS) meant swapping a couple of lines of code.","// Fonts","$text: "Comic Sans MS", cursive, sans-serif;","$code: "Comic Sans MS", cursive, sans-serif;","My front-end tooling makes pushing these changes to my live site a quick process, too.","I committed my "hilarious" update into GIT and pushed that to the remote repository.","Then I connected to my server by ssh, ran git pull to get the changes from the repo.","The final step was to run gulp sass to compile the production-ready CSS.","Writing it all down like that makes it look more complicated than it is.","Once you've run through the process a couple of times, deploying an update takes <5 minutes.","Overkill for a quick two-liner like this, perhaps, but it works for updates of any size.","My site, but set in Comic Sans.","A bit of background","I'm by no means the first person to have a little fun at the expense of Comic Sans' terrible reputation.","In 2012 CERN spokesperson Fabiola Gianotti released some results from the ATLAS project.","There was an outcry from type-nerds because the slide-deck for the announcement was set entirely in Comic Sans.","Yes, you heard right: the discovery of the Higgs Boson was announced in Comic Sans.","CERN had the last laugh a couple of years later.","In 2014 they announced they had rebranded - and the new official typeface of the European Organization for Nuclear Research was Comic Sans.","They made the announcement on their website - and of course the entire site was set in Comic Sans.","The date of that announcement, you ask?","April the 1st.","What I learned from this","My little prank was not a resounding success.","I got a couple of "tears-of-joy" emoji when I released the update - which is all I was hoping for, in fairness.","But it soon became clear that it only worked when visitors to my site already had Comic Sans MS installed on their machines.","Visit from an iOS device, and all you'd see were the fallback fonts.","If I'm honest this came a quite a shock, but wasn't all that surprising once I thought about it.","I'd been working on the assumption that all the classic "web safe" system fonts from my youth would still be web safe.","I hadn't taken into account how much the web ecosystem has changed since Apple became one of the dominant hardware providers.","Inevitably Apple were too "design sensitive" to bundle the much-maligned Comic Sans with their mobile operating system.","The practical upshot of this is slightly ironic; if I wanted my prank to be truly cross-browser I'd have to use a Comic Sans MS web font.","A good design should work with any typeface","I'm pleased to report that even when clad in Comic Sans this site remained lean and functional.","I even tried switching to Comic Sans on a couple of our more visual projects, and they too looked, well... okay.","It certainly made for a humourous minute or two in the office.","And the designs did still hold up.","I often hear that "good design" works in all situations, no matter what typeface you wrap it in.","The reality, I think, is a little more nuanced.","The typeface always sets the tone for the content.","And on a site like mine the typeface has a massive impact because most of the content is words.","Although I'm now a full time developer and haven't been an "official" designer for years.","Typography is a crucial part of my job, but makes up a much smaller percentage of my time than it used to: 30% > ~5% ↩︎","To keep down requests and page-size I only use one custom web-font, FF Tisa.","All the fallbacks in my font-stack are system-fonts.","I'm not too fussy about which monospace gets used in <code> blocks, so I've stuffed that with "nice to haves".","↩︎"]},{"title":"Introducing Picobel.js - an audio player you can style with css","url":"/picobel-js-css-audio-player/","content":["I've launched my first open source project.","Well, okay, so it's not my first and it's also not 100% launched yet.","Maybe I should explain...","What is Picobel.js?","Picobel.js (pronounced peek-o-bell, as in decibel) is a lightweight dependency-free Javascript tool that converts html <audio> tags into styleable markup.","Why I built it","Back in the day, I ran a music-review blog[1] that made heavy use of on-page audio players.","At the time I was also diving deep into css.","If you could style it, I would style it.","Few things were more satisfying than getting a raw html element to look exactly how I wanted it to look.","Before long most elements were bending to my will.","Checkboxes and select inputs presented no problems.","Range selectors were tricky, but eventually looked exactly how I wanted them.","But the <audio> element always eluded me.","Picobel is my attempt to get around the restrictions of the native html5 audio player.","In it's simplest form, Picobel finds any audio elements on a page and replaces them with "regular" markup (divs and spans etc.).","I can then apply my own styles, and avoid the cross-browser inconsistencies of the "shadow DOM".","For good measure I've also included a selection of optional "themes".","If css trickery isn't something that excites you, then you can pick a pre-made style and use that.","My open source debut","Ever since I first joined GitHub I've been storing the code for my personal projects "in the open".","But there is a big difference between technically public and actually launching something.","This is the first time I've moved beyond "something that works for me" towards "something others can use".","With Picobel.js I've thought a lot about ease of use.","Rather than being tightly coupled to my esoteric workflow, it works in any situation.","I've also made a conscious effort to document my process.","Rather than needing to examine the inside of my head, potential users can now simply read the docs.","Picobel has a demo page complete with detailed installation instructions.","There's also a CodePen collection that serves as a gallery for all the pre-made themes[2].","The GitHub project includes an official open source license (the MIT one, for now).","And there is a widget built into the README that shows the current build-status.","That's right; this project even has tests!","How far off is version 1.0?","In my mind Picobel is almost feature-complete.","The only gaps I can think of involve edge-cases and failure states.","The "buffering" state is currently rather primitive.","There's also no catch for when a file fails to load completely.","But I'm happy to save these enhancements for Version 2.0.","As it stands, Picobel works.","It's much better to get it published and in the hands of users.","So Picobel is on the cusp of hitting the 1.0 milestone.","I "soft launched" it on Reddit a couple of weeks ago and gained a few early users.","With another week or so of beta testing I'll be ready to call version one of Picobel finished.","Now I need your help.","If you could try out Picobel and let me know what you think (good or bad), I'll be eternally grateful.","Eaten by Monsters ran for about five years.","Slowly customising every aspect of that site is what drew me into web-development.","So it's got a lot to answer for...","↩︎","Picobel lets developers create their own css styles for audio players.","But by providing a range of pre-made themes the utility of this project is much increased.","Having a "plug and play" option allows people to get results without mucking around with any css first.","↩︎"]},{"title":"Recommended Listening: my favourite podcasts","url":"/podcasts-recommended-listening/","content":["I have a long commute, so blast through at least a couple of hours of podcasts every day.","I enjoy polished efforts like those from Gimlet Media and NPR, but for me the real gold lies in the more unstructured efforts.","Without any constraints of format or time, the best podcasts get to dive deep into topics that wouldn't otherwise get covered.","The recent "podcast renaissance" has felt like a vindication.","I've been trying to get my peers to listen to podcasts for years (I even made my own podcast back in '06, but the less said about that the better...).","Following the mainstream breakthrough of Serial and the prevalence of shows with high production values (like those from Gimlet), the format is finally getting the attention it deserves.","Tech Podcasts","As a developer, I'm naturally drawn to the more technical end of the spectrum.","I'm happiest when listening to an hour-long discussion about the intricacies of a new file format...","ShopTalk Show","Answering listener questions about front-end web development - alternate topic-based interviews and "rapid fire" Q&A shows.","Listen first: The Web-VR episode is a good example of the general tone of ShopTalk, but the best-ever episode was when host Chris interviewed the guy who hacked his site.","ATP","Discussion show about all things Apple.","If you like geeking-out about file systems and enjoy endless debate about the next Mac Pro, this is the show for you.","Listen first: Best to just jump in at the latest episode, but I they go "big" on WWDC (Apple's developer conference), so don't start with one of those as they're not representative of the rest of the episodes.","CodePen Radio","The team behind the CodePen web app give a behind the scenes look at what it's like to run a tech startup.","Listen first: The episode about animation makes a good jumping-off point, but they cover a wide range of topics from server-setups to getting VC-funding to hiring to managing an online "community", etc...","a16z","Archetypal Silicon Valley VC firm Andreessen Horowitz (a16z) present discussions on the hot tech topics of the day.","Listen first: A recent episode about Open Source was pretty good, but the episodes that feature the company founders, Marc Andreessen and Ben Horowitz are worth digging for.","The Creative Coding Podcast","A mix of game-dev tips'n'tricks and insights into the world of hardware hacking with Rpi-zeros, Arduinos, etc. (the clue is in the name!).","Listen first: I learnt a lot about how old-school Nintendo light-guns (Duck Hunt FTW!)","worked from this episode.","Startups & Business Podcasts","I've long since discarded any genuine plans of entrepreneurship in favour of the life of a happy drone.","But somewhere deep down the itch is still there, and I keep it in check by living vicariously through the exploits of others.","Startup","This is the big daddy.","Not only a good series about what it's like to start a business, but also a look into the inner machinations of podcasting itself.","Meta.","The more recent seasons look at other businesses, but season one was all about the founding of Gimlet Media.","Host Alex Blumberg had a vision for what a podcast company could be, and documented his journey to get there.","First listen: This is a show that requires starting at the very beginning.","Listening to Alex mess up a pitch to infamous VC Chris Sacca is radio gold, and the whole story is expertly told.","The Tim Ferriss Show","Tim's approach to life is very easy to mock, and in many ways I find him a frustrating person to listen to.","Yet he obliquely manages to get more out of his guests than most interviewers.","He asks the questions I'd like to ask, and always pushes for more detail when I want to hear more detail.","First listen: The show works best when Tim gets total buy-in from his guest.","The Seth Godin episodes are always good listens, but Seth is all about inspiring great work in others.","I prefer to be inspired by stories straight from the horses mouth, so my favourite episode so far is probably the first Matt Mullenweg interview.","Planet Money","This one straddles the line between "business" and "general interest".","The stories are always interesting, and always have an economics angle to them.","Much less "actionable" that the insights you get from the Tim Ferriss Show, for instance, but learning more about how the world around us works never did me any harm.","First listen: Episode #762 "The Fine Print" tells three short stories about what happens when you actually read the fine print.","All the episodes are short, but generally tell one story.","This trio serves as a nice potted introduction to the show's style.","Boagworld","I used to class this show as a tech podcast, but in reality it's much more focused on the mechanics of running a web business.","Regular hosts Paul and Marcus are joined by a revolving door of guests, and each season has an over-arching theme that guides the conversations.","First listen: The 15th season was all about digital project management.","When the season-wide topic was announced I groaned - could they have found a more boring subject to drag out over thirteen episodes?!","To my surprise, however, it lead to a whole series of fascinating discussions.","General Interest Podcasts","I'm not a 100%-mission-focused robot, so I do occasionally listen to podcasts for nothing more than the sheer enjoyment of it.","Reply All","New episodes of Reply All jump straight to the top of my "must listen" playlist.","The subject matter is","First listen: The episode on phishing (appropriately called "What Kind of Idiot Get's Phished?","") was a real eye-opener, but was presented by one of the show's producers.","The long-term charm of the show comes down to the interactions between regular hosts Alex & PJ, so listen to a few episodes to get a true flavour.","More or Less","In a similar vein to Planet Money, but a little more irreverent and more interested in unpicking topical issues.","A look at the numbers and statistics in the news and in life.","First listen: A recent episode on the Economics of Overbooking provided an interesting perspective on the United scandal (where a poor passenger was assaulted by staff on an overbooked flight).","To The F'ing Future","Epitome of American "Bros" talking nonsense, but some interesting discussions if you can get past the personalities.","The basic idea is that they talk about "the future" of a different topic each week.","Had a short run in 2013 and then stopped, which was a shame.","Listen first: Episode #5 (with Rob Auten) is an interesting look at what the state-of-the-art in video-game plot and characterisation was like in 2013, and what they thought the future would hold.","Even more interesting when viewed from 2017; we can see what played out and what sank without trace.","Episode #6 about tattooing was also fascinating.","Song Exploder","A geeky behind-the-scenes look at specific songs.","Uses the original mixer-tracks to dig into how a song was constructed (often looking at early demos etc., too).","If any of you remember the TV show Classic Albums, this is a lot like that but just for a single song.","Listen first: Episode #22: "Smells Like Content" by The Books","In Our Time","The BBC at its best: three experts around a table explaining a topic to Melvin Bragg.","A normal Radio 4 programme, but having the archive available as podcasts is a fantastic resource.","I tend to find the maths ones the most interesting, but part of the appeal is being introduced to a topic I'd never have learnt about otherwise.","Listen first: Take your pick - the topics are so varied it's hard to mark any individual episode as "representative".","The IOT archive is a great place to start browsing."]},{"title":"You can now install Picobel using NPM","url":"/you-can-now-install-picobel-using-npm/","content":["I’ve finally published Picobel as a node package.","Picobel is my open source JavaScript tool for turning HTML <audio> tags into styleable markup (a.k.a. regular divs and spans that you can target with your CSS).","For a while now it’s been possible to use Picobel by directly including the script in your project (just like all those old-school jQuery plugins).","But it’s been a long time since I included a script or lib like that.","My package.json rules the roost now, and almost everything I use gets installed with npm install PACKAGE_NAME.","No doubt your workflow has advanced in a similar manner.","With that in mind, it seemed foolish not to update Picobel to fit in with a more modern development workflow.","What’s changed?","For the time being, there are no breaking changes to Picobel’s core functionality.","If you have any <audio> tags on a web page, call Picobel(); in your script and Picobel will replace those audio elements with markup that you can style directly.","The only difference is how you get access to that Picobel function.","You can still include the script directly if you like, but now you’ll need to reference the picobel.legacy.js file instead.","But for all you cool cats on the NPM bandwagon, things are much nicer...","Install Picobel in your project with npm install picobel —save and then import it into your JS file with import Picobel from ‘picobel’;.","Then you’re free to call the Picobel() function just like before.","For more information about using options and themes with Picobel, take a look at the README.md on the project’s GitHub page.","What’s on the horizon?","This update to Picobel is the precursor to quite a few changes.","It’s part one of an ongoing project to convert Picobel to ES6, expand the test coverage to all Picobel’s internal methods, and get around to adding some requested new features and themes."]},{"title":"Writing well is essential. Try your best to get good at it","url":"/writing-well/","content":["I’ve worked hard on my writing style, and continually reap the benefits of having done so.","Email and instant messages take more work than face-to-face communication, and being able to convey a point clearly and concisely is an essential skill.","Writing documentation is even harder, but just as important.","I’ve had to navigate poorly documented code in the past, and try my best to ensure no-one has the same experience when trying to decipher code I’ve written.","I’m not saying that I’m a good writer - I’ve got a long long way to go on that front.","But over the last few years I’ve definitely improved, and I can quantify that improvement by looking at the decrease in communication-based foul-ups at work.","In the past my communications with clients would often be misinterpreted.","I can attribute some of the blame to fact that clients are often fundamentally uninterested in the minutiae of the product they’ve commissioned.","Busy CEOs probably don’t need to understand the nuances of browser caching[1], but they do need to know how to be sure they’re looking at the correct version of their website.","Explaining technical details to people untrained in the technology is a fundamentally hard problem.","There’s no escaping email.","The people I need to talk to are busy, and I’m a socially awkward developer who’ll do anything to dodge actually picking up a phone.","But I do need to teach the client to use the new features we’ve built for them, and they need to make me understand the problem they’re really having (rather than just the symptom they’re able to see and describe).","Professional communication is a non-trivial problem, but over the years I’ve developed a couple of coping strategies.","I’ve had to master two main skills that appear to be mutually exclusive: brevity, and tactical repetition.","Keep your writing short","Everyone’s heart sinks when they open an email and are presented with an essay.","The dreaded “wall of text” will put off even the most keen communicator.","Particularly when explaining a fix for a particularly tricky bug, my instinct is to go deep and detailed[2].","Learning to cut out the unnecessary details and get straight to the point took me a long time.","The people I’m writing to aren’t as interested in the nuances as I am.","They want to know what they need to do, and how they can do it quickly.","Tactically repeat the important points","Alongside this newfound concision, I’ve also got great mileage from learning when to tactically repeat myself.","If there’s a particularly complex point to get across, the key is to say the same thing in two different ways.","The scope of clients to misinterpret a set of instructions is seemingly limitless, but I’ve found they’re less likely to misinterpret two different sentences in the same way.","Tell them something once, and they’ll just live with whatever garbled interpretation hits their brain first.","Tell them the same thing in two different ways, and they’ll either get it straight away (the happy path!)","or at the very least notice that their interpretation is confusing.","Then they’ll actually tell you they don’t understand, rather than just muddling along.","If they don’t understand but also let you know they don’t understand, that’s fine.","Sometimes things need repeating or unpicking (and sometimes my explanations are simply too confusing, and I need to work harder to find the real issue at hand).","If these confusions are picked up early, everyone ends up better off.","As a bonus tip, I picked up a useful phrase from the world of solicitors: “For the avoidance of doubt…”.","That one comes in handy when you need to repeat something back to someone, but want them to know why you’re doing it.","Soften it if you need to, but never hesitate to say “so just to make sure I’ve got this straight…” or something similar.","Using these two strategies has vastly improved my work life; the return on investment from improving my writing is far greater than from learning a new programming language.","Being good at your job is, of course, essential.","But a marginal gain in writing ability will give better results than a marginal gain in coding ability.","And it doesn’t just help with clients.","Most work-place communication (for me) happens over Slack now (other messaging platforms are available).","And let’s not forget every developer’s favourite task; creating the docs.","Writing good documentation for your code saves hours and hours of developer-time in the long run.","It’s not enough to ensure every method has it’s parameters written down: you have to explain how the whole system fits together.","Explaining hard things is hard, but we need to do it.","People misunderstand each other all the time, and if it goes unnoticed the consequences can be time consuming, frustrating, and expensive.","If I can give one piece of advice to anyone starting out in a technical field, it’s this: work on your communication skills as much as you can.","Write, write, write.","Good luck if you ever have to hear a CEO pronounce “cache”.","↩︎","As Carl Sagan said, “if you wish to make an apple pie from scratch, you must first invent the universe.”","↩︎"]},{"title":"Using world ranking to predict the results of the 2019 Rugby World Cup pool stages","url":"/rugby-world-cup-2019/","content":["This year (2019) is a rugby world cup year.","I like data visualisation, and I like rugby.","So here's my primitive attempt to calculate the results of the pool stages of the 2019 World Cup.","My simple heuristic is that world ranking (as of Aug 22nd 2019) is a predictor of a team's success.","In effect, my simple algorithm states that a team with a higher ranking will always beat a team with a lower ranking.","Just here for the predictions?","skip to the end","Note: I'm not trying to predict who will win a match, just who is expected to win.","When the inevitable "upsets" occur (like Japan vs.","South Africa in 2015), I want to be able to say "wow!","They only had a {N}% chance of winning, but they did it!",""","So let's translate that calculation into code:","const chanceOfWinning = (teamRank, oppositionRank) => {","const combinedRanks = teamRank + oppositionRank;","const invertedRanking = combinedRanks - teamRank; // Becasue a rank of 1 is the best","const percentage = (invertedRanking / combinedRanks) * 100;","return percentage.toFixed(1); // Round to 1 decimal place","};","chanceOfWinning(2, 13); // 86.7","// NZL are ranked `2`, and ITA are ranked `13`","// Therefore NZL have an 86.7% chance of beating ITA","With this primitive algorithm, I can produce a "likelihood of winning" percentage for any pairing of teams.","And on first inspection it looks pretty good (based on my own subjective opinion of who should win a given match).","New Zealand are currently ranked #2 in the world, and should be expected to crush a #13 side like Italy.","Samoa and Russia (#16 and #20 respectively) should be a much closer match, but you'd expect Samoa to emerge victorious.","2nzl","86.7%","13.3%","ita13","16sam","55.6%","44.4%","rus20","There are problems with using rank","But this method starts to look a bit shaky when we include the #1 ranked team in the world (a crown recently claimed by Wales at the time of writing).","Not because Wales are particularly special, but because this algorithm massively favours lower rankings.","I'd expect Wales to crush Uraguay (#19 in the world), but would not expect them to have such an easy time against Australia (ranked #6).","The ranking-based algorithm predicts both matches would be walkovers:","1wal","95%","5%","ura19","1wal","85.7%","14.3%","aus6","And there's another problem with using world rankings.","Rankings, by their very nature, are ordinal.","By ranking alone, the difference between #1 and #2 is the same as the difference between #2 and #3, and so on...","Whereas in reality, some teams are much closer than their mere ranking would suggest.","Using points rather than ranking","A better metric to use as the base for our calculation would be points.","Word Rugby, the sport's governing body, uses a points system to determine the world rankings.","These points are based on match performance, and range from zero to one hundred (the top side generally has a rating of somewhere near 90 points).","In late August 2019, Wales have 89.43 points and New Zealand have 89.40 - it's tight at the top!","Australia are on 84.05 and Uraguay have 65.18 points.","Using points rather than rank changes our algorithm slightly (we no longer need to invert the team's value, as higher points are better).","const chanceOfWinning = (teamPoints, oppositionPoints) => {","const combinedRanks = teamPoints + oppositionPoints;","const percentage = (teamPoints / combinedRanks) * 100;","return percentage.toFixed(1);","};","Plumbing our examples into this calculation produces a much tighter set of matches.","The end results are still the same (in this system, a team with higher points will always beat a team with lower points, in just the same way as the team with the better ranking always wins).","2nzl","55.4%","44.6%","ita13","16sam","51.6%","48.4%","rus20","1wal","57.8%","42.2%","ura19","1wal","51.6%","48.4%","aus6","These results look a little better than the ranking-only method.","The delta between WAL/URA and WAL/AUS looks more realistic, and whoever is in the #1 spot has less of an unfair advantage.","But now the amounts look wrong.","Any theory that gives Italy a 44.6% of beating New Zealand must be inaccurate.","Increasing the weighting","The points-based system is a better reflection of the team's relative chance of winning, but to my eyes the results aren't extreme enough.","It gives too much credit to the lower-tier teams, and not enough to the top-tier ones.","For the calculation to better match my expectations, it needs to favour the teams at the top of the rankings.","Not only that, but it needs to do it progressively - so a team in the middle gets a bit of a boost, but not as much as those at the top get.","I need to write a function that will adjust the points value of each team.","The easiest way to get the result I'm after is to multiply each team's points by a power.","const adjustment = num => Math.pow(num, 5);","-->","const chanceOfWinning = (teamPoints, oppositionPoints) => {","const combinedRanks = adjustment(teamPoints) + adjustment(oppositionPoints);","const percentage = (adjustment(teamPoints) / combinedRanks) * 100;","return percentage.toFixed(1);","};","I started with 2 as the exponent, and that was better than nothing, but still not enough. 10 was too extreme, and in the end I settled on 5.","Increasing each team's points by a power of 5 gave me a set of probabilities that looked about right.","That formula added just enough of a notch in the middle of the graph - and thereby increasing the likelihood of a top-tier team beating a lower-tier one.","2nzl","74.6%","25.4%","ita13","16sam","57.9%","42.1%","rus20","1wal","82.9%","17.1%","ura19","1wal","57.7%","42.3%","aus6","Results for all the pools","This is of course only based on my experience of rugby and my own highly subjective opinions.","But it is still anchored in reality because I'm using the points as a starting point, and treating each team equally (as much as I want to give England a boost, the algorithm doesn't support it).","Ironically, this calculation shows that the draw for this world cup does give England a slight boost.","The top 8 teams make it through to the quarter finals as you would expect.","But when it comes to the semis, 4th ranked South Africa miss out, while 5th ranked England manage to sneak in.","A side-effect of the pools being drawn years before the event.","On the other hand, it probably shows that the draw-process works fairly well if, given all the top 8 make it into the quarters (or at least shows that the rankings have been comparatively static).","I'm not expecting these predictions to come true - there's a lot more to success in rugby than simple rankings.","But I do find this kind of objective analysis useful for setting expectations.","Looking at these predictions, I'll make more of an effort to see matches I might otherwise have passed on.","Tonga vs.","USA, for instance, looks like it'll be a close one.","As do Scotland vs.","Japan and New Zealand vs.","South Africa (although after this year's Championship you don't need an algorithm to tell you that'll be a real grudge match!).","Pool A matches","Pool A results","Pos.","Team","Wins","1st","Ireland","4 wins","2nd","Scotland","3 wins","3rd","Japan","2 wins","4th","Samoa","1 win","5th","Russia","0 wins","Pool B matches","Pool B results","Pos.","Team","Wins","1st","New Zealand","4 wins","2nd","South Africa","3 wins","3rd","Italy","2 wins","4th","Canada","1 win","5th","Namibia","0 wins","Pool C matches","Pool C results","Pos.","Team","Wins","1st","England","4 wins","2nd","France","3 wins","3rd","Argentina","2 wins","4th","United States","1 win","5th","Tonga","0 wins","Pool D matches","Pool D results","Pos.","Team","Wins","1st","Wales","4 wins","2nd","Australia","3 wins","3rd","Fiji","3 wins","4th","Georgia","1 win","5th","Uraguay","0 wins","-->"]},{"title":"Algorithmically predicting the results of the 2019 Rugby World Cup","url":"/rugby-world-cup-predictions/","content":["I've been working on a rugby prediction algorithm for a while now.","My basic premise is that the World Rugby rankings are a good indicator of past performance, but don't give the full picture when predicting future results.","I've already visualised my predictions based on form alone, and the results look reasonable - so is there room for improvement?","My hypothesis is that a more accurate prediction can be made if we take into account short-term form and past performance against specific opponents.","With this in mind, my new model combines four key metrics:","All-time performance - as shown by current world ranking","Recent Form: tier 1 - how well have the team performed in their last 10 matches against tier 1 opposition.","Recent Form: tier 2 - how well have the team performed in their last 10 matches against tier 2 opposition.","History against opponent - how well have the team performed against the specific opponent we're predicting the result for?","There's obviously plenty more that I can add to this algorithm, but with the World Cup actually starting soon (tomorrow, at time of writing), I've got a hard deadline for "just shipping" whatever I've got.","The predictions","-->"]},{"title":"Rugby prediction: retrospective","url":"/rugby-prediction-retrospective/","content":["Making predictions is a dangerous game, especially if you publicise them.","Before this year's (2019's) Rugby World Cup, I was chewing the ear off anyone who'd sit still long enough: "I know how the world cup will turn out.","I've made an algorithm!","" Not only did I bore all my friends, exhaust my colleagues, and write up my predictions here on my blog...","Not only that, but I actually put my hard-earned money on the line.","So confident was I in my predictions, I made a series of bets. "Sure things", I thought them.","Pride comes before a fall","As anyone who saw my predictions will know already, my algorithm was no match for reality.","After a perfect first weekend, things began to unravel from the second weekend of matches.","I became very clear very quickly that my predictions were way off.","So what went wrong?","I underestimated the specific impact of the players on the pitch and overestimated the record of the team as a whole.","Some (but not all) teams are defined by one or two star players, and subsequently highly susceptible to the whims of injury and individual form.","Japan surpassed all my expectations, proving that the tier system lacks nuance.","A large part of my algorithm hinged on the idea of "tiers": the gulf in budget and schedule and player-pool mean that "tier one" teams (like the 6 Nations and SANZAAR) are heavily favoured over "tier two"","teams.","If nothing else, 2019 was the year a new "tier one" team was born.","But Japan's ascendancy aside, the two-tier model has been shown time and time again to be a flawed indicator of performance.","I can now see evidence of four tiers, at least.","And the big question for my algorithm is this: should the model use tiers as part of the calculation, or are tiers the output of the model?","Past performance is not a reliable indicator of future performance.","Much like the players themselves, training and strategy have a big impact on performance (this is maybe the biggest failing of my algorithm - of course a team's strategy has an impact on their results #facepalm).","As much as I may previously dismissed their influence, coaches have a big impact.","They sometimes field "experimental" teams in the run up to a big tournament; it helps to blood inexperienced players, and provides a playground for experimentation in game-management and set-pieces.","And on top of all this, certain teams will target certain games.","England saved some tricks for their (in their eyes) inevitable showdown with New Zealand, South Africa held back some devastating weapons for the final (in a staggering display of self-belief), and Japan poured everything into their pool","match against Ireland (who in turn fielded a second-string team of their own, naïvely writing the fixture off as a "sure thing").","If the 2019 world cup has taught me anything, it's that there are no "sure things" on a rugby pitch.","It's easy to mock, but some teams do have a history of choking on big occasions.","Not to pick on Ireland, but "Ireland-crash-out-in-the-quarter-finals" has become a potent meme.","And how do you factor in the French?","They always over-perform at world cups (and almost as inevitably do something mind bogglingly stupid, too).","You can't build a reliable algorithm on past scores alone.","There are so many other factors that impact how a team performs.","You might be able to quantify most (if not all) of them, but that information is not hidden in the scores of past games.","Do I mind?","Of course not.","I might have put real money on the line, but it was (thankfully) money I could afford to lose.","In fact, the added excitement of having something at stake in every match makes me think I got my money's worth, and then some.","The 2019 world cup was the most surprising and entertaining world cup that I've ever followed.","The upsets and close-calls that so confounded my algorithm were the very things that made the whole event such fun.","Will I continue to work on my algorithm?","You betcha, I will.","Roll on the 2020 Six Nations!"]},{"title":"Static site generators: Hugo vs Jekyll vs Gatsby vs 11ty","url":"/eleventy-static-site-generator/","content":["I have an obsession with static site generators.","They help turn simple markdown files into a lean, mean, super-fast website.","They're rendered at compile-time, so there's no server-side activity to slow things down.","And because all the browser ever sees is static files, there's much less chance of your site being hacked.","My favourite SSG is Eleventy (11ty), but I've been on a long journey to get here...","I tried WordPress, vanilla HTML, Jekyll, Gatsby, and Hugo, but 11ty is my favourite","I tried WordPress","Before I discovered SSGs, I was a WP ninja for years.","People often complain about the speed of WordPress, but after running several massive-traffic, high-complexity, image-heavy sites for clients, I'm happy to report that WordPress is not the performance bottleneck that people think it is.","(You just have to be very careful!)","I also loved the versatility of Custom Post Types and the magic UI that comes with Advanced Custom Fields Pro, but never felt that WordPress as a whole aligned with my preferred workflow.","Despite the power of CPTs and ACF, the WP dashboard was always a point of friction.","It could do too much, while at the same time not being customisable enough 😬","I tried vanilla HTML","Eugh!","That was a fun intellectual challenge, and it's something I believe that every frontender should be able to do.","But once you get beyond a couple of pages it was a massive pain!","I missed Markdown and partials and data-management and, well, all the things people use frameworks and systems for!","I tried Jekyll","After the over-engineering of WordPress, Jekyll was a breath of fresh air.","Comprehensible flat files!","No database to slow things down!","Markdown support that makes sense!","(Good markdown support is hard to manage in WP even to this day).","Even though I quickly fell out with Jekyll (too much Ruby, not enough flexibility for shared metadata and categorisation, and getting Jekyll to play nicely with my workflow automation was a major PITA), this was the moment that I fell in","love with statically generated websites.","The concept made perfect sense for the simple blogs I was creating at the time, and meant I jumped straight to the top of the my-site-loads-faster-than-yours leaderboards.","I tried Gatsby","After I finally mastered React (maybe; it's a long journey, I suspect), it was inevitable that my React-all-the-things!","instinct would carry over to my blogs.","Gatsby was an easy sell.","All the power of a React SPA, with all the benefits of a statically generated site?","Yes, please!","The image-handling and page-prerendering still blow my mind, but in the end it was goddam GraphQL that drove me away.","The frontmatter's right there!","Why do I need to change three files to get it!","I don't want to add the data, then add a schema too!","Fuuuuuuuuu...","Needless to say, I've retreated again to less over-engineered pastures.","I tried Hugo","Hugo is awesome, and being able to tell people I use GoLang is fun.","But if I'm brutally honest with myself, the street-cred is the only real advantage over Jekyll 😋","It has all Jekyll's benefits, but also all of the same limitations (for me, at least).","We use it for our documentation hub at my Jobby Job, and I really enjoy working with it (we might even roll out a Hugo build of our currently-WP brochure site, too).","But most of the time I want more control over shortcodes and snippets and advanced features.","This could all be improved if I was better at Go, I'm sure, but if there's a JavaScript option out there, that's the one I'll inevitably gravitate towards.","11ty is my favourite","And so we come to Eleventy (a.k.a. 11ty).","It has all the benefits of Hugo and Jekyll, but it's written in JavaScript.","Flat files?","Check!","No excess cruft?","Check!","Markdown is king?","Check!","I can customise to my heart's content, and add all the shortcodes and layout tweaks I want.","The real selling point for me is the intuitive folder structure - it's so simple and elegant, and I grokked it almost instantly.","As an added benefit, it integrates very nicely indeed into my existing workflow automation.","I'm not penalised for using WebPack for assets, and relying on GitHub and Netlify for CI is a breeze.","I've rebuilt my site many times use many different tools, but the 11ty rebuild was by far the fastest","There are still friction points; pagination's a real pain, as is any attempt to simulate different post types, and the documentation is still a work in process (although that almost makes me more excited because there's a genuine","opportunity for me to contribute something meaningful to the project).","So, all told, I could still be lured away by a newer and shinier SSG.","But after over a year of use I'm still happy with 11ty.","It does the majority of things I'd want a site-generator to do, and gets out of the way when I want to add in my own crazy things.","If anything, my travels through the land of SSGs have taught me that the later point is more important - a static site generator cannot get in the way of my workflow.","As soon as it does, I'll most likely try another.","And there are plenty out there still to try."]},{"title":"The things I use","url":"/things-i-use/","content":["Every now and then I get "nerd sniped".","Something (a link, an idea, anything) comes out of left-field and instantly becomes an obsession.","Take, for example, Universal Paperclips.","I stumbled on that link who-knows-where, and BAM!","Three days' productivity lost, just like that.","Thankfully the most recent example is more benign.","Wes Bos' uses.tech site is a catalogue of tech bloggers' "uses" pages.","I saw that, and had to drop everything to make my own: tomhazledine.com/uses","What is a "uses" page?","In short, a "uses" page is a list of the things you use.","Beyond that, it's really up to you.","I've chosen to list the hardware I own and use on a regular basis.","I covers the basics like computers and laptops, but also other useful things.","I have a section for video- and audio-related tech (which I use for podcasting and my occasional coding videos), and also a section that covers my art and design tools (ranging from fancy sketchbooks to The World's Best Pencil[1]).","Future plans","I think having (and maintaining) a Uses page is a valuable thing.","I love reading about the tools other professionals use, so hopefully someone else (maybe you!)","will find value in my list.","So far I've only scratched the surface (I really love things, it turns out) and there are already plenty of omissions.","The tools I use day-to-day fall in and out of favour periodically, so regular updates will be required.","And I haven't even started to list the software I use!","I will revisit my Uses page as often as I can, and will actively seek-out the Uses pages of others.","You should make one of your own, and then Tweet the link to me.","I'd love to see it!","It's almost embarrassing how much I love this pencil.","The feature set, the weight of it, the finger-feel of the exquisite knurling.","Everything about it is perfect.","↩︎"]},{"title":"Twitter Cards with Nunjucks and 11ty","url":"/twitter-cards-with-nunjucks-and-eleventy/","content":["When sharing links on Twitter, there's a handy feature to make those links look prettier.","Twitter "cards" will turn your simple text link into a rich card complete with an image and structured title and description data.","Here's a recent tweet from my podcast's[1] account.","When writing the tweet, we simply included the url https://aquestionofcode.com/52-what-gear-do-you-use/ in the text, and Twitter automatically added the card content.","🎙️52: What gear do you use?","This week we take a look at all the tech and tools that we use everyday. #techgear #techpodcast #CodeNewbieAvailable on your podcatcher of choice!","https://t.co/xRyxuXAAW2— A Question Of Code (@aQoCode) March 3, 2020","There are several types of card, including summary, player, and summary with large image.","They all serve different purposes and are activated in different ways.","You can find the full details on Twitter's developer overview.","Adding support for Twitter Cards to your own site","Does this happen to all links?","Sadly not.","To activate this feature, the page you're linking to needs to include some Twitter-specific metadata in it's <head>.","Here's what that metadata for this page looks like:","<meta name="twitter:card" content="summary" />","<meta name="twitter:creator" content="{{site.authorTwitterUrl}}" />","<meta property="og:url" content="{{site.url}}{{page.url}}" />","<meta property="og:title" content="{{title}}" />","<meta","property="og:description"","content="{% if excerpt %}{{excerpt}}{% else %}{{site.summary}}{% endif %}"","/>","<meta name="twitter:image" content="{{site.url}}/images/pages_stack_bg.jpg" />","<meta property="og:image" content="{{site.url}}/images/pages_stack_bg.jpg" />","Adding the metadata in 11ty","My site is built with Eleventy (a.k.a. 11ty), and I use the Nunjucks templating engine.[","2] I chose Nunjucks because it's the de-facto standard in all the code example in the 11ty docs, and it's about as lightweight and intuitive as any other templating language.","That same block of metadata that I showed before looks like this in a Nunjucks file:","<meta name="twitter:card" content="{{cardType}}" />","<meta name="twitter:creator" content="{{site.authorTwitterUrl}}" />","<meta property="og:url" content="{{site.url}}{{page.url}}" />","<meta property="og:title" content="{{cardTitle}}" />","<meta","property="og:description"","content="{% if excerpt %}{{excerpt}}{% else %}{{site.summary}}{% endif %}"","/>","<meta name="twitter:image" content="{{cardImage}}" />","<meta property="og:image" content="{{cardImage}}" />","Because the way 11ty templates and layouts are put together, I had to add the metadata to my site's master main.njk layout file.","This is the main layout that wraps all my pages, so I need to include some custom logic to account for a few scenaris:","Pages that don't have an image of their own (most pages on my site, in fact) fall back to using a generic site-logo image.","If we're using the generic site-logo image, I switch the card's type from summary_with_large_image to simply summary, which creates a smaller card that is better suited to a icon or simple logo.","Gotchas","I have to confess that it took me several attempts to get the images loading correctly[3].","Twitter provides a helpful Card validator.","Having access to that meant I could test and debug without having to spam my followers with tweets full of broken card links (which was a relief, I can say).","It still needs a valid real-world url to work, however, and that's not something my current dev-setup can provide for this site (where I normally just rely on a localhost url for testing and development).","I just YOLO'd it by repeatedly deploying to my Netlify instance.","Probably not "best practice", but it got the job done.","#shamelessPlugA podcast, you say?","Why yes!","be sure to check out A Question of Code in your podcatcher of choice.","↩︎","Convienently, the syntax is the same for Liquid (which is the other popular templating language used with Eleventy) ↩︎","Note to self, Twitter cards don't like relative paths for images!","↩︎"]},{"title":"CSS Naked Day","url":"/css-naked-day/","content":["As a developer who came of age at the dawn of Web 2.0, supporting semantic markup is something that I've always been in favour of.","I've written in the past about how HTML doesn't need (much) styling to be perfectly functional.","So I'm really enjoying the fact that today (April 9th) is "officially" CSS Naked Day.","If you're reading this on the 9th, then my whole site will be untouched by CSS.","If you're reading on any other day, then it's only this page that's unstyled.","Apart from being a fun exercise, there is a serious point to be made here.","A lot of people experience the web via nothing more than the markup (be that through assistive tech like screen readers, or just automated delivery mechanisms like web-scrappers and RSS feeds).","The way we structure our documents matters."]},{"title":"Installing acoustic panels","url":"/installing-acoustic-panels/","content":["In case it's somehow escaped your attention, I've been having a lot of fun recording a podcast lately.","It's a weekly show that we have been running for nearly a year and half now (which is something I'm rather proud of).","This week I installed some acoustic panels in my office to improve the sound quality of the podcast.","Unboxing my new acoustic panels","Running our show is very much a hobby, but we take it seriously and aim for the best quality we can achieve.","We don't run adverts or have any external funding, so the only budget we have is what we're willing to spend (knowing it's just for fun).","Marginal gains are better than massive single investments","With all that in mind, I do sometimes splash out and "invest" in the 'cast[1].","I could blow hundreds of pounds on a microphone upgrade, but a more sensible approach is to aim for marginal gains.","There are plenty of small changes that can improve audio quality, and most involve spending effort rather than money.","Improving your microphone technique is the biggest thing you can do to improve your audio quality, regardless of the gear you use.","And all that takes is a little research and plenty of practice and discipline.","After working on your mic technique, the next best thing you can do is to improve the acoustic qualities of the space you record in.","It makes sense: the quality of the sound you capture in you mic is the single biggest factor in the quality of the overall sound.","Sometimes you can "fix it in post", but it's far easier, quicker, and cheaper to fix it before you record.","Dealing with room noise","So what is your microphone picking up?","Two things: your voice, and the sounds of your environment.","Hopefully you're able to record in a quiet space with little or no background noise, so it's very likely that the majority of the "room noise" you record will be reflections and reverberations of your own voice.","Every hard, flat surface in your room will bounce you voice back at the microphone.","In most cases this manifests as a "boxy" sounding end result.","The cure to this problem?","Soften the hard surfaces and break up the reflections.","Soft furnishings will help a lot here (sofas, heavy curtains, bookshelves, etc.) but every now and then you find a large, flat surface that you can't do anything else with.","That was the case in my new home office - a big flat wall right behind my desk; perfectly poised to reflect the maximum amount of sound directly at my microphone.","And that's where acoustic panels come in.","Acoustic panels","Acoustic panels are a fairly simple concept: pads of foam that you stick to your wall to reduce reflections.","Compared to a lot of audio "gear", they're not too expensive.","I picked up a pack of sixteen 30cm x 30cm squares for about £36.","But all it is is cleverly shaped foam, so really it's actually quite expensive.","But I'm not much of a DIY-er, and I'm generally happy to pay for convenience.","In "proper" studios, you'll find fancy panels with wooden frames, nice fabric covers, and dense rock-wool stuffing.","They are obviously more effective than some simple foam glued to the wall, but like I said; my DIY skills are weak.","Installation (maybe)","My first attempt at attaching the foam panels to my wall did not go well.","I had a set of double-sided sticky pads, recommended by the Amazon Q&A on the page that I got the panels from.","After some careful positioning (with a tape measure and a spirit-level, you'll be pleased to hear), I was pretty pleased with the results.","I'd yet to do any recording, but I could test that in the morning...","...jump-cut to the next morning, and many of the panels were strewn across the room or wedged between my computer and the wall.","In short - not where I'd left them 🙁","The sticky pads just weren't sticky enough","The sticky pads were not enough to hold the panels by themselves.","The pads had stuck to the wall fine.","In fact, if I ever want to take them off I'm sure it'll be a devil of a job!","The trouble was that the pads weren't reliably adhering to the foam itself, so the panels would just drop off.","Not at first (no!","they were happy to lull me into a false sense of security), but gradually, over a period of hours or days, they would fall to the floor.","Installation - take two","For take-two, I took things up a notch with some serious spray mount.","After doing some research, it emerged that the best course of action was to attach the panels to some kind of backing (plywood is best, but cardboard works fine).","This time I was taking no chances and both the back of the panel and the cardboard backing got a liberal dousing in spray-fix.","I even weighted them down and gave them plenty of time to dry.","Cutting cardboard to act as backing for the panels.","Glueing that cardboard in place.","Weighing down the glued panels with a heavy box while the glue dries.","Attaching the new cardboard-backed panels to the wall was even quicker and easier than the first attempt.","The sticky pads were still attached (and still sticky) so the cardboard adhered very thoroughly.","Job done.","Or so I thought...","Installation - take three?","Thankfully the cardboard is still stuck fast.","That theory, at least, was sound.","And this time most of the panels have stayed in place.","In general the glue fixing the foam to the cardboard is still pretty strong, but a couple of panels fell down overnight.","Cue day three of trying to get these darned things to stay up 😬","Useful advice","If you're planning on installing acoustic foam panels, the biggest tip I can give is this: apply a sturdy backing to the foam, and make sure it's stuck fast before attempting to attach it to the wall.","It's save you a lot of bother.","But was it all worth it?","Stay tuned to future episodes of the A Question of Code podcast and see if you can hear the difference...","The end result","One of the first signs you've become a "podcast idiot" is you start calling them 'casts.","↩︎"]},{"title":"Spiraling out of control? Open up the Bullet Journal again","url":"/bullet-journal-redux/","content":["Believe it or not, I've thought of something new to say about Bullet Journalling.","I won't dive into the mechanics of what and how, because I've written about Bullet Journalling before (several times, in fact).","If you're unfamiliar with the concept of Bullet Journalling, those posts are better places to start.","What I've noticed lately is the pattern behind my Bullet Journal usage.","I've picked up the journalling habit several times now, and dropped it just as many times too.","My journal was a constant presence when I was freelancing, but it fell by the wayside when I became an employee at a company.","I picked it up again when I needed to find a new job, and it was a massively useful tool in that process.","But once gainfully employed again, the journal soon began to gather dust.","And when a company I was working at shut down suddenly, out came the journal again and the same pattern repeated.","I've started using my Bullet Journal again recently, too.","But this time I'm not in need of a new job, and I'm not freelancing.","This time the entire world is on fire and falling apart around us[1].","And that's the pattern; when things feel like they're slipping beyond my control, Bullet Journalling helps me stay on course and stay (or get back to being) productive.","What I didn't realise before is that journalling is a transactional process for me.","It's not the life-long everyday habit that I first thought it would be.","It's a useful mechanism to help deal with times of crisis.","Pick it up, reap the benefits for a few weeks, and then discard it again.","When I do maintain a Bullet Journal, I'm pretty thorough and update it every morning.","But I'm still using the first notebook I bought for the job (from a pack of three I picked up in 2013!).","Flicking through the pages, I can see a written record of all my periods of uncertainty, transition, and crisis[2].","I'm not saying that everyone should take up journalling when their lives start slipping, but this is certainly an effective coping mechanism for me.","And being aware of journaling's function in my life is only going to make the tool more useful.","I'll no longer feel guilty when I inevitably stop maintaining the journal - it's not a chore that "must be done", its a tool that has done its job.","And maybe I'll adopt the journaling habit sooner next time; maybe get things back on-track before things bubble over into a productivity crisis.","I'm hoping some of you are reading this from a time that's not gripped by a global pandemic.","Sic transit gloria and all that.","↩︎","Made much easier by the fact that I'm maintaining a detailed "contents" page, like the nerd that I am.","↩︎"]},{"title":"Dark mode: hard mode","url":"/dark-mode/","content":["Dark mode has been around in operating systems and web browsers for a while now, but it wasn't something I'd ever really cared about.","But that all changed when my brother implemented a dark-mode stylesheet for his (seldom updated but excellent) blog, edthecoder.dev.","In a true spirit of "anything you can do, I can do better", the sibling rivalry kicked in and I knew I had to implement dark mode on my own site.","If a "mere" backend dev could implement a solid dark mode, surely a "frontend expert" such as myself should be able to do the same (but better).","The basics: simple dark-mode with the prefers-color-scheme media query.","All in one place: defining dynamic themes with CSS custom properties.","Live switching: toggling dark-mode without having to change a system-wide preference.","The basics","At its simplest, "dark mode" can be applied to your stylesheet by using the prefers-color-scheme media query.","This behaves in much the same way as more familiar size-related queries (for example @media (min-width: 960px) {}), and will only apply the styles within it if the browser detects that the system has the prefers-color-scheme: dark flag set.",".thing {","/* "normal" colour rules */","color: black;","background: white;","@media (prefers-color-scheme: dark) {","/* These colour rules will only be","applied if the site is loaded on a","machine with dark-mode enabled */","color: white;","background: black;","}","}","All in one place","I've previously been declaring the colours in my stylesheet using scss variable (for example, $black: #4d4d4d).","This would be fine if I was happy adding @media rules to handle dark-mode throughout my stylesheet, but that comes with a few issues:","The dark-mode specific rules would be spread all over the place, introducing unnecessary complexity and creating a maintenance headache.","Sass variables are not dynamic (a.k.a. they are defined once, at compilation time), and therefore wouldn't be able to adjust on-the-fly depending on the light/dark setting of the client.","What I need are colour variables that can change value depending on their context.","What I need are CSS Custom Properties.","CSS Custom Properties","Custom Properties are native CSS variables, and they are supported in all major browsers (RIP in peace, IE).","Unlike Sass variables, they are evaluate by the client at runtime.","This means that our stylesheet can adapt to whatever mode the client is using (light or dark), and we could even change that setting dynamically with JavaScript.","Note: Unlike variables in scss, custom properties need to be declared inside a wrapper.","Common practice is to use the :root pseudo class to ensure the custom properties are accessible in the entire cascade (:root represents the  element and is identical to the selector html, except that its specificity is higher).","/* a static Sass variable */","$primary: red;","/* a dynamic custom property */",":root {","--primary: red;","}",".unchanging_thing {","/* This value is fixed when the stylesheet is compiled */","color: $primary;","}",".changing_thing {","/* This value will change on-the-fly if --primary is changed */","color: var(--primary);","}","For a long time now, I've declared all colour variables in a single scss file (_settings.colours.scss, in case you were wondering) which makes the job of maintaining a unified "theme" across a complex site a relatively simple","task.","Having to spread prefers-color-scheme queries throughout the various parts of the stylesheet messes this up big time.","Using custom properties allows us to abstract the theme-specific parts of the stylesheet away from specific elements.","Crucially, it allows us to add a single prefers-color-scheme query into the existing colours.scss partial.","Nice and tidy!","/* Default colours */",":root {","--primary: red;","--secondary: blue;","/* etc...","*/","@media (prefers-color-scheme: dark) {","/* Dark-theme colour alternatives */","--primary: maroon;","--secondary: navy;","/* etc...","*/","}","}","Gotchas when using custom properties and Sass","Something to watch out for when using custom properties inside Sass is that things get weird when you try to use them inside functions.","Sass is evaluated when the file is complied, so things like darken($colourVar, 10%) are set in stone as soon as you hit save (and therefore not dynamic when the file is loaded by a browser).","Using a custom property in this instance will result in an error, and the scss will not compile.","If you're locked-in to (pseudo-)dynamically adjusting colour values, then CSS' native filter rule might be an option for you.","Because filter adjusts the entire element, I find it much simpler to manually set the colour value I want.","So rather than using $red: #ff0000; and $red--dark: darken(red, 20%);, we would need to write --red: #ff0000; and --red--dark: #990000;","Another thing to watch out for is how specific you need to be when formatting colours.","With Sass variables, I've grown accustomed to being able to declare a colour as a hex value and have it "just work" wherever I want to use it.","I could declare a variable as #ff0000 and use that variable inside an rgba() function to apply opacity: rgba($red, 0.3).","In that instance, rather than having to convert my hex colour into the RGB equivalent (255, 0, 0), Sass would do the conversion work for me.","With native CSS custom properties, we don't have this convenience.","/* Works in Sass */","$red: #ff0000;","$red--translucent: rgba($red, 0.3);","/* Does not work in Sass (or anywhere else) */","--red: #ff0000;","--red--translucent: rgba(--red, 0.3);","/* Works everywhere */","--red: #ff0000;","--red--translucent: rgba(255, 0, 0, 0.3);","This meant that I had to do a lot of hex-to-rgb conversion when converting my stylesheet to use custom properties.","Turns out I was a heavy user of both RGBA colours and declared all my colours as hex values.","Oops.","Live switching","By doing all that we've done so far, we've successfully implemented dark mode!","After a trivial long and hard refactor I've now reached feature-parity with my brother's site.","But the goal here was to surpass Ed's efforts, and the best way I can think of to achieve that is really lean in to the dynamic aspect of the custom properties I've taken such pains to add.","What we've done so far is to switch colour-theme based on the prefers-color-scheme value, but that's not the only thing we can use to toggle between themes.","With our custom properties in place, I could add as many "themes" as I want.","For now I'm happy with "light mode" and "dark mode", but to prove the concept it would be great to be able to toggle between the modes on-the-fly.","Currently, unless a visitor to the site already has dark-mode enabled on their system, there's no way for them to know that the dark version of the site even exists.","Allowing users to dynamically toggle between themes has a few requirements:","We'll need to detach the theme-switching logic from the prefers-color-scheme state (but still respect that setting if it exists).","We'll need a UI element to activate the toggling, with display states for both themes.","We'll need to persist the user's setting between page loads (requiring them to re-set the value on every page load would be really obnoxious).","Here comes the JavaScript!","The first step is easy.","We can move the dark-mode custom properties out of the @media query, and use a data attribute instead:",":root {","--primary: #00b7c6;","/* Normal colours...","*/","}",":root[data-theme="dark"] {","--primary: #ff851b;","/* Dark-mode colours...","*/","}","The next step is to create a JavaScript function to handle changing that data attribute.","This setDarkMode() function will accept a boolean parameter that decides if we're applying light or dark mode.","We'll also set a localStorage value so that the page remembers which mode the user has chosen.","const setDarkMode = (active = false) => {","const wrapper = document.querySelector(":root");","if (active) {","wrapper.setAttribute("data-theme", "dark");","localStorage.setItem("theme", "dark");","} else {","wrapper.setAttribute("data-theme", "light");","localStorage.setItem("theme", "light");","}","};","With that function in place, we then need to run it when the page loads.","This is where we need to detect the system preference (the JS equivalent of the CSS prefers-color-scheme value).","We can do this with the browser's matchMedia() method - which returns the value of a given media query.","So if we retrieve this value when the page loads, we can call our setDarkMode() function with the correct boolean.","As an added bonus, we can attach an event listener to the query which will fire whenever the system setting changes (meaning that if someone switches dark mode on or off while the page is open, the change will be applied instantly).","const query = window.matchMedia("(prefers-color-scheme: dark)");","setDarkMode(query.matches);","query.addListener(e => setDarkMode(e.matches));","This is made a little more complicated when we account for the setting from localStorage, which we want to be able to override the system preference (this is the user's explicit choice, after all).","const query = window.matchMedia("(prefers-color-scheme: dark)");","const themePreference = localStorage.getItem("theme");","let active = query.matches;","if (themePreference === "dark") {","active = true;","}","if (themePreference === "light") {","active = false;","}","setDarkMode(active);","query.addListener(e => setDarkMode(e.matches));","But even with this plumbing set up and ready, we're still missing a key part of the puzzle: the ability for user's to actually set their preference on the page itself.","For that, we'll need a <button> in our HTML and an event listener to handle to toggling of the theme.","const toggleDarkMode = () => {","const theme = document.querySelector(":root").getAttribute("data-theme");","// If the current theme is "light", we want to activate the dark theme","setDarkMode(theme === "light");","};","// ".js__dark-mode-toggle" is the unique class we've added to the <button>","const toggleButton = document.querySelector(".js__dark-mode-toggle");","toggleButton.addEventListener("click", toggleDarkMode);","Putting it all together","The final piece of the puzzle is to add some CSS sparkle to the button element (I ended up settling for a cheesy single-div sun/moon combo), and then we've got all the pieces we need!","You can see all this code in action live on this very site; click the icon in the top right of the screen to toggle the light/dark modes.","You can also see the full code (and mess about with it to your heart's content) in the CodePen demo embedded below:","See the Pen","Dark Mode Toggle by Tom Hazledine (@tomhazledine)","on CodePen."]},{"title":"Building a delay effect with the Web Audio API","url":"/web-audio-delay/","content":["In this post I'm going to show you how to use JavaScript to recreate a delay pedal.","A delay pedal is something that you'd place between a guitar and an amplifier to add a delay effect to the signal from the guitar.","They're a relatively common piece of musical kit, and I'm going to recreate one in the browser.The audio contextTo get started, we will need an audio context.","The audio context is our gateway into the audio power of a browser.const context = new window.AudioContext();","Having created a new AudioContext object (which in this instance we're naming "context"), we can then use that context object to do all sorts of things: we can create audio sources, connect audio-creating-things to","audio-listening-things, all of that good stuff.And of course because we live in the real world we need to fudge things a little to make sure that it works in as many different browsers as possible:const context = new (window.AudioContext ||","window.webkitAudioContext)();","Once we've got that AudioContext set up, we can then start using it.","And the way that we're going to use it is to recreate the idea of an analogue signal path, but in code.The signal pathA good example of a "signal path" is a how a band sets up on stage.","It's the path that sound takes from an instrument to a mixing desk and then out to some speakers (so the audience can hear it).","InstrumentMixerSpeakersAnd in audio-context terms, Instrument > Mixer > Speakers","translates to oscillator > gainNode > context.destinationThe instrument becomes an oscillator (we'll be using an oscillator to create some sounds)The mixer will become a gainNode (which is a way for us to control the level of","that signal - the "volume", if you will)The speakers will become our context.destinationNow let's work backwards through this path in more detail...3.","The destinationThe context.destination is available on the context object that we created, and the "destination" will be either the speakers in your laptop or your headphones or some kind of Bluetooth connection.","In short: wherever your sound goes when it leaves your computer, that'll be your context's "destination".2.","GainThe gainNode is a little bit more complicated.","This is a way of controlling the "volume" of our signal.","We'll use our context object to create a "gain node".","We'll call the new node "master" (because it will act as our master volume control), and we'll need to set a value for it.const master = context.createGain();","master.gain.value = 0.8;","master.connect(context.destination);","The sound comes in at a raw level, essentially (if it comes from an oscillator it's just coming in at the default level), but once it gets to a gain node we can set what level we want that signal to be.","The range runs from 0 to 1, and in our example we're using 0.8 (just slightly less than full volume).","Then on the last line we're connecting the master gain node to our destination, which will be the the output of the sound.So we've got the sound coming in, we're setting that volume level with a gain node, and then connecting","it with .connect (which is a method on all audio objects that we create from our context).1.","OscillatorMoving to the very start of that signal path (where the sound actually comes from!), we're going to simulate a "voltage-controlled oscillator" (a.k.a. a "VCO").","Of course it's not voltage controlled because we're not in the analogue world and we're not actually plugging electricity into things and wiring things up and soldering them.","But we can simulate a VCO with the context's createOscillator() method.const VCO = context.createOscillator();","VCO.frequency.value = 440.0;","VCO.connect(master);","That creates an oscillator object.","The object has has a frequency value which represents the pitch of the note that will be created.","Here we're setting a value of 440, which translates to 440 hertz (Hz).","That is the frequency value of "Middle A", which in an orchestral setup is the note that everyone tunes to.We're then connecting that to master (the gain node that we're treating as our "master volume"), which","then sends our signal off to the to the destination.Now that we've created an oscillator we can test that it's working using two simple methods: start and stop.","// Start the oscillator","VCO.start();","// Stop the oscillator","VCO.stop();","Putting these methods into action might look a little like this:let buttonState = "off";","const button = document.querySelector("#our-button");","button.addEventListener("click", () => {","if (buttonState === "off") {","VCO.start();","buttonState = "on";","} else {","VCO.stop();","buttonState = "off";","}","});","You can test this out using the button below.","Clicking once will call .start(), and you should hear a (probably quite horrible) noise.","Then clicking a second time will call .stop(), which will end the sound.If we look at the frequency graph of what's happening, you can see when we start the pitch we get a peak at 440Hz, which is the "middle A" that we are","hoping for.This code gives us a constant tone which is not massively useful for adding a delay to.","The delays will start layering up and overlapping, so we're not going to be able to hear what's going clearly.","What will be more useful for us is to have a pulse, so let's look at how to have the button trigger a pulse of sound:// Start the oscillator now","VCO.start(context.currentTime);","// Stop the oscillator in .25 seconds time","VCO.stop(context.currentTime + 0.25);","Now when we hit "start" we're not just a calling .start() with no parameters: we're calling it with our current time and then straight away calling the .stop() function with a different time.","This means our code will know to start and stop at these given times.We're getting time from context.currentTime, which is a more reliable way of getting a time-value than using a setTimeout() like we would normally use in JavaScript.","The audio context has its own internal clock which is much more precise.Controlling a VCO with a gain node (a.k.a. a VCA)So now we get a pulse of sound that lasts for 0.25 seconds, but it's still kind of sudden.","What we will probably ought to do now is "soften" this slightly so it's not quite such a jarring sound.","It just turns on and off.","What we want is to have a way of ramping up the volume gradually and then ramping that down again.We want it to fade in and fade out, and we can achieve that by using another gain node.We've already used a gain node for our master","volume, and now we're using a new gain node paired just to our oscillator.const note = {","vco: context.createOscillator(),","vca: context.createGain()","};","note.vco.connect(note.vca);","note.vca.connect(master);","const button = document.querySelector("#our-button");","button.addEventListener("click", () => {","// Start the oscillator gradually","note.vca.gain.exponentialRampToValueAtTime(1, context.currentTime + 0.2);","// Stop the oscillator gradually","note.vca.gain.exponentialRampToValueAtTime(","0.0001,","context.currentTime + 0.5",");","});","Here we've made a regular JavaScript object called note, and we've given it a VCO and a VCA.Whereas a VCO is a voltage controlled oscillator, a VCA is a voltage controlled amplitude.","That's analogue-synthesiser-speak for a node that controls "volume".","Here our VCA is going to be a gain node.We connect those two together, then we set the values using a function called exponentialRampToValueAtTime() which, surprise surprise, exponentially ramps to a value at a","given time.We want to ramp up to the volume value of 1 (a.k.a. maximum volume) and we want that to take 0.2 seconds.","To stop the pulse, we then ramp all the way down to a value of 0 half a second later.Now our pulse is starting to sound a little bit more sonorous.","We've got a gentle pulse that more closely mimics sound that we'd hear in the real world (rather than the artificial on/off of the raw oscillator).","Randomly selecting a "note"Now we can spice things up a little by randomly choosing the pitch of the note that is generated.const getRandomInt = (min, max) =>","Math.floor(Math.random() * (max - min + 1)) + min;","note.frequency.value = getRandomInt(220, 880);","The getRandomInt() function creates a random integer between 220 and 880 (which is the frequency range, in hertz, that we want to hear).","Now every time we press the "pulse" button we will hear a random pitch.We talked about it being a bit more sonorous earlier, but because these pitches are random there is no sense of musicality.","To enhance this a little bit we can make use of an array of note values that each have a name (so that we know what note we're dealing with) and a value (which is the","important part).const C_Maj = [","{ name: "C4", value: 261.63 },","{ name: "D4", value: 293.66 },","{ name: "E4", value: 329.63 },","{ name: "F4", value: 349.23 },","{ name: "G4", value: 392.0 },","{ name: "A4", value: 440.0 },","{ name: "B4", value: 493.88 },","{ name: "C5", value: 523.25 },","{ name: "D5", value: 587.33 },","{ name: "E5", value: 659.26 },","{ name: "F5", value: 698.46 },","{ name: "G5", value: 783.99 }","];","Now that we have a list of all the notes in a C major scale, we can use that same random number generator.","This time, however, we'll use it to generate a key for the array of notes.const noteNumber = getRandomInt(0, C_Maj.length - 1);","note.frequency.value = C_Maj[noteNumber].value;","Now we are still randomly creating notes, but they're all within the bounds of a C major scale (which should in theory sound a little bit nicer).","It's still a bit random but it is within the bounds of a normal scale so it sounds like of how we'd expect music to sound.Doubling-up our noteNext, let's get even more fancy and duplicate the notes for a "fuller"","sound.","We can create a second VCO and VCA for our note:const note = {","...note,","vco2: context.createOscillator(),","vca2: context.createGain()","};","And rather than manually choosing the note value we can "transpose" the value of our first VCO.const transpose = (freq, steps) => freq * Math.pow(2, steps / 12);","const startingPitch = note.vco1.frequency.value;","note.vco2.frequency.value = transpose(startingPitch, 7);","note.vco2.connect(note.vca2);","note.vca2.connect(master);","This transpose() function takes in a frequency and the number of "steps" in either direction that we want to go.","Say you want to transpose from a C to a D; that's two semitones (musical steps) up.What the transpose function does is work that out for us.","It calculates how we should mathematically transform our starting pitch, and does this by multiplying the frequency by a power law.By adding a second VCO, we are doubling up the notes that we hear when we hit the "pulse" button.","But rather than simply duplicating the sound, we're transposing the second VCO to be seven steps above the pitch of the first VCO.","Seven steps up equates to a "fifth" in musical terminology (there are 12 steps but eight notes, so seven steps up is actually five notes up the scale).","You can see see the double peak on the frequency graph when we trigger a pulse:The FX loopLet's get to the business at hand of actually adding in our effects unit.This will be a unit that slots into our signal path and adds an effect","to the signal.The "normal" signal path (as we've already established) is "instrument" -> "mixer" -> "speaker".","The FX ("effects", for non-nerds) loop sits into that path between the instrument and the mixer.{% include "delay/signalpath-fx.njk" %}This little module takes a bit of the signal, does something to it, and then sends","that","signal back up to the mixer along with the original signal itself.Creating a delay object in JavaScript is nice and easy to do because the audio context has the concept of "delay" built into it.const delay = context.createDelay();","delay.delayTime.value = 0.4;","delay.connect(master);","The delayTime value determines how much time the delay node will delay the signal","before sending it on again.Anything that we connect to that delay will then be paused for whatever value we set (0.4 seconds, in this example) before then being sent on to the master output.If we hook our notes' VCOs to the delay node,","then our pulse will be heard twice (audio nodes can have multiple connections, so here our notes are connected directly to the master output and indirectly via the delay node).note.vca1.connect(master);","note.vca2.connect(master);","note.vca1.connect(delay);","note.vca2.connect(delay);","delay.connect(master);","Success!","We have now created a delay.","But we don't have the full picture yet; this is only applying the delay once.","When we press the "pulse" button, we hear the original note and a single repetition of that note forty milliseconds (0.4 seconds) later.FX FeedbackWhat we're missing is the idea of "feedback".","Feedback is the mechanism by which a small part of the delayed signal is fed back into the delay module.{% include "delay/fx-loop.njk" %}We could connect the delay node to itself directly (delay.connect(delay)), but this would","send the entire signal back into the delay.","This would be an infinite loop, so let's not do that because it would sound terrible.What we're going to do is create a new gain node.","We've used a lot of game nodes so far, so we're pretty familiar with them at this point.We eliminate the problem of infinite feedback by giving the feedback node a very small value: 0.3 (which is equivalent to saying "30% of","the signal").","Then we put this new feedback node in between the delay output and the delay inputconst feedback = context.createGain();","feedback.gain.value = 0.3;","delay.connect(feedback);","feedback.connect(delay);","delay.connect(master);","The amount of signal that we're passing back in controls how many times we hear the pulse repeated; that's the audible delay.","The combination of the delay duration (delay.delayTime.value) and the level of the feedback (feedback.gain.value) combine to create the full effect.Making it interactiveThere's one final step required to fully mimic the behaviour of a",""real" delay pedal.","To finish off, I'll plug the two crucial factors (the duration and the feedback) into some HTML range elements.","This allows us to tweak those values "on the fly".","The higher the feedback level, the longer the repetition of the pulse will continue for.","The longer the delay duration, the larger the gaps between the repetitions will be.Have a play yourself, and see what crazy sounds you can make.","You can even set the feedback level to 100%, but be warned - weird things will happen!","This post has, hopefully, shown you some of the power of the audio context as it exists in the browser.","We've just scratched the surface; creating a tone and manipulating it in small ways is pretty simple stuff, but there is no end to the amount of creativity that can be had from playing with these tools.","I hope this has inspired you to do something of your own.If you've enjoyed this or you have questions please do get in touch on Twitter and let me know if you've done something crazy with the audio context.","I'd love to see what you've come up with."]},{"title":"RSS in 2021 (yes, it's still a thing)","url":"/adding-rss/","content":["I'm trying out a "recipe blogger" style of writing[1].","That is to say, I'm going to share a simple set of technical instructions but first I'll tell you a rambling story about how this recipe reminds me of how butterflies sip dew from the golden whatever high atop the thing...","Unlike food bloggers, however, I'm self-aware enough to include handy section links so you can choose your own adventure.","Rambling story: Why I'm an RSS fanboy","Actually useful content: How I added an RSS feed to my Eleventy blog","My finished feed: tomhazledine.com/feed.xml (go ahead, follow me!)","Why I'm an RSS fanboy","Strangely, RSS is one of my favourite parts of the web.","I say "strangely" because I'm primarily a front end developer who loves having control over how a website looks, and it's hard to imagine a medium that gives you less control over the display of your content than RSS does.","It's purely a content-based medium.","As it says in the name; it's really simple.","So why do I love it so?","It's the underpinning tech that makes podcasts possible.","You might not have heard, but I'm really in to podcasting and think it's one of the single greatest innovations of my lifetime.","And part of the power of the medium is that the distribution is 100% decentralised.","Literally anyone (with access to the bare minimum tech requirements) can distribute their show on the same footing as giant media companies.","Here at the start of 2021 it looks like that might be in danger of changing, but the core concept of "what I think a podcast is" still feels solid and robust and defensible.","People will still be making the kind of podcasts I like decades from now, thanks in part to RSS.","It's a direct delivery to me from writers I want to read.","So much of modern content delivery is predicated on virallity and share-ability, but sometimes there are writers (dare we even say bloggers) who I know in advance I want to read (or at least be made aware of) everything they publish.","RSS makes that process (you guessed it) Really Simple.","There are no algorithms to filter what I'm shown, and no adverts and holding-pages injected into the flow to drive me crazy.","It's just the content.","And while for lazy readers like me there are timelines to wade through, provided your reader-app is a good one this process is straightforward and manageable.","How I added an RSS feed to my Eleventy blog","So given that I love RSS so much, why don't I have an RSS feed for my own blog?","Well, you'll be pleased to hear that now I do.","Part of the fun of maintaining your own blog is that you tend not to get too much for free, especially if (like me) you've chosen to roll your own site with minimal help from pre-made themes or templates.","Adding features like RSS feeds (or search, or favicons, or meta tags) is all in the journey.","Thankfully I'm not mad enough to have rolled my own static site generator (yet), so with a little help from the folks at Eleventy I was able to add an RSS feed with ~very little fuss at all~ some fuss after all.","1.","Create a feed.xml page","Add a feed.md file in your content directory, and use the permalink frontmatter to tell Eleventy that you want the page to be created as an XML file, and that you want the page to be excluded from collections (note that I write my","frontmatter in yaml).","permalink: feed.xml","excludeFromCollections: true","2.","Create the template structure for your feed","The aim here is to build the XML document that will serve as your RSS feed.","This is very similar to generating a regular archive or category page, except we want the output to be XML rather than HTML.","The differences are subtle, but there are a few things to watch out for if we want our feed to be valid.","Dates, for instance, need to be formatted in a very specific way.","The Eleventy ecosystem is on hand to help us out with here with the RSS Plugin.","It doesn't do everything for us, but does give us access to some handy RSS/XML-related functions: getNewestCollectionItemDate and dateToRfc3339 being particularly useful when it comes to correctly formatting date-info for our feed.","The docs for the Eleventy RSS Plugin also provide a sample template file, but I'd advise treading with caution here.","While helpful, converting the sample template to match my exact project was trickier than I'd first anticipated.","Be sure to go through the sample template line-by-line and make sure you know what each line does.","This is especially true if you've done anything beyond the basics when it comes to "collections" within your content[2].","3.","Account for any weirdness in your posts","I think it's important that the articles I publish on my site are accessible by RSS.","But sometimes RSS isn't the best tool for the job.","I have several posts that rely on interactive elements within the page (most recently, my Web Audio API article includes plenty of illustrative web-audio-dependent examples), and these simply don't translate to a text-and-image-only medium.","I've sidestepped this problem by adding a not_rss_friendly flag to the frontmatter of these articles.","When that flag is true, my RSS feed adds a caveat to the top of the content block.","This allows me to explain that some of the content may not work on the reader's tool of choice, and they might want to check out the original web version.","And what's more, this has no effect whatsoever on my main site; these caveats only appear in the RSS version.","4.","Validate and publish","The feed is easy enough to test locally.","I have my Eleventy dev server pointing at http://localhost:1337, so to view my new feed I can look at http://localhost:1337/feed.xml to ensure everything has built as expected.","With a format as pernickety as XML, it's worth doing some proper validation, too.","To test my local version I copy/pasted view-source:http://localhost:1337/feed.xml into the W3C validator, which caught a few minor issues for me (it also flagged issues with non-matching URLs between local and live which I was happy to","ignore at this stage).","For added pre-deployment comfort I turned to one of my favourite features of Netlify (the service I use to build and host my website).","Their Deploy Previews are a fantastic tool.","Every time I create a PR on my GitHub repo, Netlify spools up a matching development server with a real URL.","These previews are full, working versions of my site, and because the preview URL is a proper live domain on the actual internet I can submit that preview version of my feed into any feed-reader app and see a real version of what my changes","will look like.","And because Netlify is just the best tool ever, all I need to do to deploy my new feed is to merge my PR.","Boom.","Nice and easy.[","3]","With that done, my new feed is live at tomhazledine.com/feed.xml.","If you like to consume your content using RSS, you can add that feed to your reader of choice and never miss a post from this site (lucky you!).","My full feed.md template file:","<?","xml version="1.0" encoding="utf-8"?",">","<feed xmlns="http://www.w3.org/2005/Atom">","<title>{{ site.title }}</title>","<subtitle>{{ site.summary }}</subtitle>","<link href="{{ site.url }}/{{ permalink }}" rel="self"/>","<link href="{{ site.url }}"/>","<updated>{{ collections.articles | getNewestCollectionItemDate | dateToRfc3339 }}</updated>","<id>{{ site.url }}/</id>","<author>","<name>{{ site.author }}</name>","<email>{{ site.authorEmail }}</email>","</author>","{%- for post in collections.articles | reverse %}","{% set absolutePostUrl %}{{ post.url | url | absoluteUrl(site.url) }}{% endset %}","<entry>","<title>{{ post.data.title }}</title>","<link href="{{ absolutePostUrl }}"/>","<updated>{{ post.date | dateToRfc3339 }}</updated>","<id>{{ absolutePostUrl }}</id>","<content type="html">","{%- if post.data.not_rss_friendly %}","{{ site.rssCaveat | markdown }}<p>View the original here: <a href="{{ absolutePostUrl }}">{{ absolutePostUrl }}</a></p>","{%- endif %}","{{ post.templateContent | markdown | htmlToAbsoluteUrls(absolutePostUrl) }}","</content>","</entry>","{%- endfor %}","</feed>","Before we start, let's have a moment's silence for the passing of Google Reader.","I know at this point it's been over seven years since it was so cruelly taken from us, but I'm still a bit cross about it.","↩︎","I've got to be honest, even when using Eleventy's "debug" mode this was a hard template to put together.","Becasue there are various level of content-escaping going on, I became very familiar with the "Opening and ending tag mismatch" error message.","↩︎","Now all that's left to do is decide where to put the feed-link icon...","🤔 ↩︎"]},{"title":"The year of writing","url":"/year-of-writing/","content":["For 2019 I experimented with an alternative to a traditional new year's resolution: a "Yearly Theme".","It was such a success that I repeated the process in 2020, and now I'm laying the foundations for my "theme" for 2021.","Themes are not resolutions","To be blunt, I've come to view new year's resolutions as a load of nonsense.","They're often small and frivolous goals set with little thought and even less expectation of success.","But I am a big believer in occasionally taking-stock and assessing my progress.","Progress in what, you may ask?","Well, pretty much everything.","And because everything takes a pause at the end of the year (by design, as Christmas is a big deal in my house), the January-December cadence works well as a natural period to look back on and plan ahead for.","Although perfectly arbitrary, the incrementing of a number on the calendar is a useful boundary.","What are Yearly Themes?","Inspired by CGP Grey and Myke Hurley from the Cortex podcast, a yearly theme is a guiding idea or principle that you apply to your life for the whole year.","The "theme" concept is deliberately vague enough to mean different things to different people.","So what does a yearly theme mean to me?","It should be specific enough to provide guidance when faced with a decision.","You should be able to envisage a situation where your theme would be useful when faced with a "should I do x or y" decision.","It should, by nature, be strategic, and therefore have a logical opposite.","The theme is all about choosing a specific path for yourself, so it would have no value if it was something that you would naturally do anyway.","A theme like "the year of getting older one day at a time" is meaningless, because it's something you're inevitably going to do: there is no viable alternative.","In a similar vein, I don't think that "the year of being successful" is a good theme because no one sets out to not be successful, and in many ways success is a measure, not a specific goal.","You can't plan to be successful; success is an outcome of your plans.","It must be memorable, because you should have it at the front of your mind throughout the year.","It's also helpful if other people in you life can remember it, too.","That helps with accountability and also gives those around you an insight into why you choose to do the things you do.","Past themes:","This is not my first rodeo, and I've had a theme for each of the last couple of years.","2019: The year of shipping.","My plan here was to increase the amount of projects that I finished.","By default I'm a tinkerer, and rarely see things through to completion, so setting this as an explicit goal meant that I shipped more things.","In many ways it was a great success[1] and heralded a change in mental attitude, but there's still more work to be done.","I do ship more now that I used to, but I still err on the side of Infinite Tinkering.","2020: The year of action.","I wasn't happy with the name for this one, but @ed_the_coder convinced me to roll with "action".","In my mind it was more akin to the (less catchy) "year of personal responsibility".","In many, many ways this ended up being a great theme for 2020.","2021: The year of ???","For 2021 I'm hedging my bets and choosing a dual-sided theme.","Last year I got into modular synths in a big way, and as a result I've fallen back in love with music[2].","So I'm determined to write and release some of my own music this year.","I'm also doubling down on blogging as a concept that I think is important.","Some of my favourite websites are blogs, and as someone who writes code for a living, have an outlet for my more creative technological experiments is valuable both for my career and as a general learning tool.","So this year I'll be writing music.","And what is blogging if not writing?","2021: the year of music?","The year of blogging?","Why not both?","2021 is going to be my Year of Writing.","Ask me in 2022 how I got on...","In 2019 I upped my workplace productivity, and consistently released weekly episodes of the A Question of Code podcast (where we have talked at length about Yearly Themes before).","↩︎","How I fell out of love with music is an interesting tale, but one for another time.","↩︎"]},{"title":"Falling back in love with music","url":"/falling-back-in-love-with-music/","content":["Over the last year I've fallen in love with making music again, and it's all thanks to modular synthesisers.","But to understand why this is such a big step for me, you need to know just how surprising it is that I'm now "in to" electronic music.","My growing eurorack modular synthesiser (complete with obligatory succulent)","Falling out of love with music","I've always been "into" music, but I used to be really into music.","I played in lots of bands, tried my hand as a session musician, tried to start a record label, ran a music blog (for, like, years!).","But in 2013 it all sort of fizzled out.","In retrospect, I ran out of steam.","On paper things were going in the right direction - I could legitimately call myself a session musician, I sold a couple of commercial compositions, and I even had a song of mine on the radio (that £6 royalty cheque was a Big Deal, let me","tell you!), but I never really saw enough success in any area to persevere.","Except, maybe, the blog - that worked and was fun.","Making that blog was so fun, in fact, that 12 years later I now have a pretty decent career as a frontend engineer making things for the web.","But the "music" aspect soon fell by the wayside.","I shuttered the blog in 2013 I never looked back, focusing instead on frontend engineering.","The wilderness years","For a long time I was more interested in writing code than writing music.","A couple of my projects touched on music but the focus was always on the technology, never on actually creating music.","The Web Audio API featured heavily in my CodePen tinkering and I built a whole conference talk around my experiments recreating a guitarists' delay pedal in JavaScript.","But at no point was I making music.","Not even in private or just for fun.","If I remembered, I'd run a few drills on the guitar to maintain my muscle memory but I wasn't ever composing or creating anything new.","Guitar pedals: the ultimate gateway drug","This story hinges on a single moment: I saw something and was instantly hooked.","But the groundwork for that one incident was laid over a long, long time.","How did I get into electronica?","It's a big leap for an ex-musician who always focused on acoustic instruments (heck, I wrote my undergraduate dissertation on the evolution of English folk music, and my post-grad thesis was an extremely contrived deep-dive into the",""album" as a medium).","But there were two things that lit the spark of interest that would ignite into a full-blown obsession with modular synths: GAS and the Web Audio API.","GAS - a.k.a.","Gear Acquisition Syndrome.","As in, "oh boy, those new Strymon guitar pedals have triggered my GAS.","RIP my bank account".","Guitar pedals are a fantastic introduction into the world of modular synths: small units that do one thing to an audio signal.","It already sounds very "modular", am I right?","And I've always loved guitar pedals.","In fact, my GAS covers all guitar-related gear.","Microphones, audio interfaces, cables, pedals, the guitars themselves.","I'm not going to lie; I get just as much (if not more!)","pleasure from owning all these nice shiny things than I do from actually using them.","The second factor that paved the way for an interest in modular was the Web Audio API.","A side effect of playing around with using JavaScript to create and manipulate audio was that (almost by a process of osmosis) I was exposed to concepts and terms that originated in the world of hardware synthesisers.","VCAs and VCOs became part of my lexicon, along with more familiar concepts like filtering and mixing.","Bit by bit, without even being aware of it at the time, I was learning how to synthesise sound.","Enter stage right: modular synthesisers","So what was that single moment that opened my eyes to the world of modular synths?","Quite simply, it was a YouTube video.","I've kept a weather eye on Andrew Huang's musical adventures for a while, but I stumbled on his Modular Synthesis Explained video while looking for something cool to include in my newsletter.","Specifically, the shot of his mega system at work at 15:30 mins touched something deep inside me.","I didn't know what it was, or what it was doing, or how it worked, but I knew I wanted one for myself.","Andrew has made lots of videos about his eurorack system over the years (it's fascinating to see the system grow over time) and he's made some fantastic "explainer" videos.","YouTube, it turns out, is a great place to ramp up your GAS to extreme levels. there are so many great creators making so many great videos, both showing off their great modular music but also making fantastic tutorial content.","Why has modular made me fall back in love with making music?","I'm about a year into my modular journey at this point.","It was twelve-ish months ago that I first learned what a modular synth was, and another few months of research and obsessive YouTube bingeing before I bit the bullet and bought my first modules.","Now my system is getting to the point where I can really make some fun pieces with it, and I'm excited about what the future holds.","I've embraced the idea of "learning in public", so I'm sharing all my experiments and clumsy, early attempts at music making on my Instagram feed.","This is totally unlike anything I ever did with any other instrument, and goes against my instinct of getting things as close to perfect as possible before sharing them with anyone.","And it feels great!","There's no pressure to be "good", because everyone knows I'm making this stuff up as I go along.","And the feedback I've already had is just amazing - there's so much more validation in being "okay" at something in public than there is in being amazing at it in private.","I should have adopted this approach a long time ago.","And when (on the rare occasion) I make something I'm really proud of, I'll take the time to edit it and put it on YouTube.","YouTube feels more "permanent" than the 'gram, and already I'm seeing that the life of a YT video continues long (so far!)","after the initial launch.","Whereas on Instagram, once a post is off the front page it is gone forever (as far as engagement goes).","As for why I'm enjoying it so much, there are several reasons:","I love the sounds I can make with this machine.","I've been listening to a lot of ambient music made on modular systems, and I'm relishing the challenge of recreating that style all by myself (and I still have a lot of learning and practicing to do!).","I love that making music on a modular doesn't require a computer at all.","Manipulating sounds with real tactile controls and making connections with physical wires is exponentially more fun than using a software alternative.","Sure, you might be able to create the same sounds "in the box" (maybe), but you can't create the same experience.","Modular synths are playable.","You can build muscle memory, you can manipulate two parameters at the same time (try doing that with a mouse).","You can "ride" a fader or pot with so much more precision than you can in a software GUI.","I love that I get to build the instrument that I want to play.","There are so many modules out there that there's a high chance that even my small system is unique.","I get to choose the exact tools required to make the kinds of sounds I want to make.","I love that I can make make sounds that are impossible (or at least highly impractical) to make on a computer.","There are some things that you can do on a modular synthesiser that you just cannot do in software.","I love how the instrument looks.","I'm not going to lie, the aesthetic of modular is probably a good third of my motivation for getting into it.","The arrays of potentiometers, the vast selection of I/O, the blinking LEDs (oh, the blinking LEDs!), and the general space-ship-control-panel vibe.","I love it all.","There are plenty more reasons, too.","And looking at that list written down, I'm now thinking that each one of those points could be an article all by itself (so stay tuned for more modular nerdery here soon).","But suffice it to say, I've fallen hard for modular synthesisers.","If you're interested in my modular journey, be sure to checkout my YouTube channel where I post "completed works" and my Instagram feed where I post my experiments and test things out."]},{"title":"Learning (and doing) in public","url":"/learning-and-doing-in-public/","content":["I've always struggled with finishing things, but lately I've found that forcing myself to release projects early has had some amazing effects.","Flushed with that success, I'm going one step further and making myself publicly accountable for things I'm planning to do.","What's been working?","As I mentioned in my last post, I've been making music again.","A big part of the enthusiasm I've maintained for music lately comes from the fast feedback loop that comes from learning in public.","With my past musical endeavours I would ticker for ages in private, building up to a "grand reveal" that often simply never happened.","This time around I've made the conscious decision to share early; I'm posting experiments and practices to my instagram account, and releasing pieces onto YouTube as soon as their "done".","This has taken two big leaps for me:","I've never shared "practice" before.","I'm objectively bad at the instrument I'm currently learning (my modular synth), and my practices could be seen by experts as embarrassing.","I'm teaching myself not to care about this.","I've had to change my definition of "done".","The pieces I'm releasing are not finely crafted opuses.","I'm releasing as soon as I have ~4 minutes of sound with a beginning, middle, and end.","The results so far have been awesome.","It's great to get positive feedback; hearing that people like my work drives me to make more of it.","Negative feedback would be useful, too, but I've (mercifully?)","not had much of that yet.","Although on social media a lack of any feedback is a strong signal.","The good stuff gets a reaction, and the bad stuff sinks into obscurity.","Either way, I'm learning what works and what doesn't work, and that guides my future experiments and practice sessions.","And in the spirit of "doubling down on what seems to be working", I'm now expanding my "do all the things in public!","" policy to some other areas.","What am I committing myself to?","Firstly, some simple rolling objectives.","I've had these goals in place since January as part of my Year of Writing:","A written piece on this blog, once a month.","Twelve videos about my modular synth on YouTube by the end of the year.","Fifty-two modular-themed posts on Instagram by the end of the year.","They sound modest, but just look back at my blogging history to see how consistent (or should I say inconsistent) my track record is.","I can be very productive in short bursts, but keeping up momentum for a full calendar year is not something I often manage.","Secondly, I'm setting myself some stretch goals.","These are things I want to get done Soon™️, and would love to get shipped before the end of the summer.","But if I get to December and haven't hit any of these goals but have met my rolling objectives, I'll still call the year a "Win".","Sort out the newsletter.","Should this site (should I) have a newsletter?","I started a podcast-focused one last year, and it flopped was less of a success than I'd hoped for.","I'm currently on the fence about whether I should have a general newsletter for all the content published on this site, or whether it should focus solely on my Modular Synth activities.","What I do know, is that this site should have a proper "call to action" for people who enjoy the content.","Release Picobel.js as a React component.","I (still!)","really enjoy working in React, and Picobel feels like the perfect use-case for the <Component /> pattern.","It also gives me more control over state changes (like "playing" or "paused" or "loading") which lead to a nicer (and more reliable) experience for the end user.","Make an interactive JavaScript version In C.","This one's is just for me, and just for fun.","I'd love to recreate Terry Riley's seminal minimalist musical composition, In C, on my modular.","Alas, both my skills and sequencing equipment aren't up to the task just yet.","As a stopgap, I'm going to recreate it in the browser using the Web Audio API.","What are the consequences of failure?","In all honesty, the consequences of missing my targets are mercifully slight.","Given the amount of readers this blog gets (very few) and the number of people clamouring for my new projects (none), there are no tangible consequences for missing my "deadlines".","But failure stings, even if the bounds of success are arbitrary and imaginary.","Fear of failing to meet my rolling goals has already pushed me to publish more frequently here and on IG and YouTube.","Ask me in six months how this experiment has gone.","I've reasonably confident that I'll feel good about it by then."]},{"title":"Line graphs with React and D3.js","url":"/line-graphs-with-react-svg-d3/","content":["A while ago I wrote an article about the Web Audio API with an interactive demo.","The primary visualisation for that demo was a series of graphs that displayed the live frequency data of the sounds made by the demo.","Since then, several people have asked about how the graphs were made and the answer is way too confusing when expressed at tweet-length, so this post is a full explanation.The core principle is to use React to generate an SVG that updates","("reacts"?)","when the data changes, and to use certain features of D3.js to make these calculations easier.Push the "pulse" button to make some bleepy-bloopy sounds.","The frequency data will be shown in the graph in real-time.In this articleWhat are the tools we'll be using?","The basics: converting an array of data into an SVG path Step 1. parsing the data Step 2. creating a React component with an SVG in it Step 3. mapping our data to the visual domain using D3.js Make it dynamic Why is this graph so different","from the original demo?","Core ideas to use in your own work What are the tools we'll be using?","SVG1: using an SVG path feels (to me) like a really intuitive way to draw our graph lines.","You can inspect the results in any web-inspector just like any other DOM node, and we can style the SVG with CSS in the same way we would style any other component.React: the React "virtual DOM" makes this kind of graph really","easy.","When the data changes, React efficiently handles the re-rendering of the graph.","Because we're using a stream of real-time audio data that updates several times a second, we don't need to worry about "tweening" or animating the graph line.D3.js: D3 is a powerful tool that could handle the creation of","the whole graph.","I find React more intuitive for this kind of component work, but D3 has some amazing data-wrangling tools that we can leverage to make our life easier.","Specifically, the mapping of data into a layout-friendly domain and the creation of the actual path values that will form the main part of the graph.The aim is to optimize for an easy workflow that leans into the tools and techniques that I","already use when creating websites.","I already use React for anything that needs to be "dynamic".","SVG uses markup that looks familiar to anyone who knows HTML, and can be styled with the same (S)CSS workflow that I use for every web project.","D3 is the only graph-specific tool I'm using, and that's because it has some very specific utilities that make the rest of the process much easier.The basics: converting an array of data into an SVG pathSo what's happening in","the example at the top of the page?","At the most basic level, the Web Audio API code is giving us an array of numbers for each moment in time.","These numbers are then converted to a format that can be used as line data in an SVG.","That SVG is then rendered on the page.","Here's a simpler example, with all the "decorative" elements (axes, labels, etc.) removed, and using a simpler (and static) data set:A simple static line graph drawn using SVG.Step 1. parsing the dataTo build this reduced","version of the graph, we'll start off with some representative data.","We'll use fewer numbers than the real audio data, but the basic format is the same.","The Web Audio API "analyser" gives us an array of frequency data for each moment in time2, so what we're using for this demo is functionally the same:const exampleData = [34, 44, 32, 78, 184, 221, 171, 26, 62, 5];","The obvious thing that we're missing from this data is a second axis.","Each "node" in the graph line represents an x and y coordinate, and our data only contains a single dimension.With the real audio data, the array of numbers represents the volume of the signal at a series (an array, even?)","of specifc frequencies.","This level (ranging between 0 and 255) becomes what we plot on the y-axis.","Because each value represents an evenly-spaced slice of the frequency data we can distribute our points evenly across our graph.","We can use the index-value from the array to create the x-axis data we need.To convert our raw array of numbers into a format that can be used to draw a graph, we want each item to be an object with an x and y value.","A quick map() over our array creates this for us:const exampleData = [34, 44, 32, 78, 184, 221, 171, 26, 62, 5];","const cleanData = exampleData.map((item, i) => ({ x: i, y: item }));","// cleanData = [","//     {x: 0, y: 34},","//     {x: 1, y: 44},","//     {x: 2, y: 32},","//     {x: 3, y: 78},","//     etc...","// ];","Step 2. creating a React component with an SVG in itNow we're armed with some useable data, we can create a React component to draw this data onto the page.What we want is to draw an rectangular <svg> element with a single","<path> within it.","An SVG <path> gets it's shape from a data prop called d.","We'll create a line state value that will populate this prop, and hard code this value for now (and add our data in the next step).","Note that we're doing something triksy with the SVG's viewBox property here.","Because in future steps we'll be mapping our data to coordinates within the SVG, we need a fixed set of dimensions for our image.","But we also want our graph to be responsive and fit whatever screen it's being shown on (even in narrow contexts!)","so we're setting absolute values for our viewBox and a relative value (a percentage) for our width.","Combined with the preserveAspectRatio="none" prop, this means that we only need to do our coordinate calculations once, and CSS will work it's magic to ensure the graph responds to different widths correctly.import React, {","useState } from "react";","const ExampleLineGraph = ({ data }) => {","const [line, setLine] = useState("M0,200 L100,100 L400,100 L500,0");","return (","<svg","className="graph--example"","width="100%"","height="200"","viewBox="0 0 500 200"","preserveAspectRatio="none"",">","<path className="graph__data" d={line} />","</svg>",");","};","export default ExampleLineGraph;","We'll also want to set some CSS rules so our SVG looks the way we want it to:// Declare our colour custom properties (a.k.a.","CSS variables)",":root {","--grey: #dad8d2;","--primary: #00b7c6;","}","// Set the border for the whole graph",".graph--example {","border: 1px solid var(--grey);","}","// Set the colour of the line (and remove any defualt "fill" our line may have)",".graph__data {","fill: none;","stroke: var(--primary);","}","Then all that's left is to mount our example graph on the page.","Don't forget that even though we're passing in our cleanData value here, we're not actually using it yet (that will come in step #3):const exampleData = [34, 44, 32, 78, 184, 221, 171, 26, 62, 5];","const cleanData = exampleData.map((item, i) => ({ x: i, y: item }));","ReactDOM.render(","<ExampleLineGraph data={cleanData} />,","document.getElementById("simple-line-graph")",");","A hard-coded SVG <path>.","Step 3. mapping our data to the visual domain using D3.jsWith our basic line being successfully drawn within the SVG, the next step is to hook in our "real" data.","Currently we're passing in an array of data via the data prop, but we're not using it yet.","We need to convert the array of x/y objects into a format that can be understood by the d property of an SVG <path> element.This is where D3.js comes in handy.","The D3 graphing library is often used to generate entire graphs, but all we need in this instance are a couple of helper functions to make our data-conversion a little easier:import { line, scaleLinear } from "d3";","D3's line() function will handle the formatting of our data: turning it into a valid SVG d value.D3's scaleLinear() function will help us convert our raw data into spatially-aware values (a.k.a. mapping our frequency values into","pixel values)The key concept at work here is that of ranges and domains.","Our raw data exists in a frequency "domain", and the values are frequency values (i.e.","34 Hz, 44 Hz, 32 Hz etc.",").","To draw this data with our graph, we'll need to convert those frequencies into pixel values; we need to transform them from the frequency domain to the spatial "range".","This is what we'll use scaleLinear for.Defining the spatial range is relatively straightforward.","In our first pass at the graph, we hardcoded the viewBox values for our graph.","To make life easier for ourselves (and to avoid accidentally changing an important value in one place but missing it in another) we'll define our width and height values in a layout object that we can reference every time we need to","use a layout value.","We'll then update the height and viewBox props to use these values:const layout = {","width: 500,","height: 200","};","height={layout.height}","viewBox={`0 0 ${layout.width} ${layout.height}`}","The next piece of the puzzle is to setup our D3 functions as values we can use in the rest of our code.","As well as our line generator (using the D3 line() function), we'll need to scales3: one for x and one for y.","For each scale we'll set a range from 0 to the corresponding layout value (width for x and height for y):const graphDetails = {","xScale: scaleLinear().range([0, layout.width]),","yScale: scaleLinear().range([layout.height, 0]),","lineGenerator: line()","};","There are two more things we need to setup for our line generator before we can use it.","Firstly we need to define the domain of our data, and secondly we need to tell the generator which values to use for which axis.Setting the data domain looks similar to how we set the range.","We're defining the minimum and maximum values our data might be.","The x domain is the simpler of the two, being as it's the number of items in our array (more complicated datasets would need more complicated domains, of course).","As for the y domain, we know that our audio analyser provides a maximum frequency value of 255 Hz, so we can hardcode this value (I like to add a bit of "headroom" to the graph for purely visual reasons, so I've bumped the","value from 255 to 280).graphDetails.xScale.domain([0, data.length - 1]);","graphDetails.yScale.domain([0, 280]);","Assigning the correct data values to the line generator is one of the more esoteric parts of D3 (it confused me for a long time!).","Things made a lot more sense when I realised that we're defining a function that will be run for each item in our dataset when the line generator is actually used.Our data (provided to this component via the data prop) is in object","format (e.g. {x: 3, y: 78}).","We've handily named our values "x" and "y" but they could be called anything, so we need to tell our line generator which values to use.","This is also the point where the values are run through our scale functions.","So for each item (d) in our dataset, we're passing our d["x"] value into the xScale function, and returning that computed value. (and ditto for y).graphDetails.lineGenerator.x(d => graphDetails.xScale(d["x"]));","graphDetails.lineGenerator.y(d => graphDetails.yScale(d["y"]));","And now that the line generator is all set up, we can finally use it when we intialise our lineData state, and then we can plumb that into the <path> element within our SVG:const [lineData, setLineData] = useState(() =>","graphDetails.lineGenerator(data)",");","<path className="graph__data" d={lineData} />","At this point, the full component looks like this:import React, { useState } from "react";","import { line, scaleLinear } from "d3";","const ExampleLineGraph = ({ data }) => {","const layout = {","width: 500,","height: 200","};","const graphDetails = {","xScale: scaleLinear().range([0, layout.width]),","yScale: scaleLinear().range([layout.height, 0]),","lineGenerator: line()","};","graphDetails.xScale.domain([0, data.length - 1]);","graphDetails.yScale.domain([0, 280]);","graphDetails.lineGenerator.x(d => graphDetails.xScale(d["x"]));","graphDetails.lineGenerator.y(d => graphDetails.yScale(d["y"]));","const [lineData, setLineData] = useState(() =>","graphDetails.lineGenerator(data)",");","return (","<svg","className="graph--example"","width={"100%"}","height={layout.height}","viewBox={`0 0 ${layout.width} ${layout.height}`}","preserveAspectRatio="none"",">","<path className="graph__data" d={lineData} />","</svg>",");","};","export default ExampleLineGraph;","The line renders the real data 🎉Make it dynamicThe beauty of using React for a project like this is that there aren't many more steps required to make the graph dynamically respond to changing data.","Because we've already initialised our data with a useState hook, we can make React watch for changes in the data with a standard useEffect hook.useEffect(() => {","if (data) {","// Calculate the data line","const newLine = graphDetails.lineGenerator(data);","setLineData(newLine);","}","}, [data]);","With this hook in place, whenever the data prop changes the path d will be recalculated and re-rendered.","Note again that we're not digging into the specifics of how the data value is changed.","That would require an entire article on Web Audio API analyser nodes.","This example just covers the graphing component, not the actual data generation.Push the "pulse" button to make some bleepy-bloopy sounds.Why is this graph so different from the original demo?","There are a couple of obvious differences between this simplified demo and full-featured frequency graph at the top of this page.The first graph uses a logarithmic scale for the x-axis (note how on the simple example, the peaks occur","squished toward the left of the graph).","Instead of using D3's scaleLinear() function, the original graph uses scaleLog() combined with actual frequency values (rather than the evenly spaced indexes that we've used for the simpler demo).","If you're implementing this for yourself, you can apply the same concepts that we used on our y-axis: i.e. mapping between a visual range and a data domain.The first graph has tick marks (the helpful guiding lines that make it clear","where each frequency goes).","These are implemented with <line> nodes in SVG, drawing simple lines from one x/y coordinate to another using the x1, y1, x2, and y2 props.The first graph has labels for the axes.","Truth be told, I've dodged some complexity here by only marking the minimum and maximum frequency values.","This allows me to add the labels with traditional markup and CSS.","If you wanted to mark incremental values along either axis, you would need to use the yScale or xScale functions to find the visual positions of the exact values you wanted to mark.Core ideas to use in your own workAs I just mentioned, in","this article we've made a much-simplified version of the original frequency graph.","This is deliberate, because every graph will inevitably have it's own quirks and idiosyncrasies that make it different from all other graphs (and thus hard to explain in a basic manner).","I wish it wasn't so, but you can only abstract so far before your all-singing-all-dancing reusable graphing component has so many options that it's actually harder to use that building every graph individually from scratch.","This article is an attempt to show a few core concepts that can be reused in lots of different contexts.","Those core concepts are:Use whatever tools that you are most comfortable with (but sometimes add in a little bit of something new to make your life easier).","I'm comfortable using React and SVG, so I'm mostly just using those.","Graph-maths can get hard, though, so I'm using as little D3 as possible to make my life easier.SVG and React are great tools for creating visualisations.","If visualising data is something that you want to do on a regular basis, these tools are great ones to learn.","They're versatile enough to handle all sorts of data-vis and the skills you learn will also be useful in the wider web ecosystem (so you're not wasting too much time learning an esoteric system that's only useful for one","thing).","Speaking of esoteric systems, D3.js is really powerful but can be daunting too.","It can do so much, but also has it's own quirky ways of doing a lot of things.","My response is to just use the parts that I actually need, which in this case is just the line generator and linear scale calculator.","Then I'm free to let the more conventional tools (a.k.a.","React and hand-coded vanilla SVG) to do the bulk of the work.If SVGs are new to you (or you just need a quick refresher on their syntax) I've written a primer on SVG markup that might be useful.","↩A quick note about getting audio data: we're skipping the details of getting raw frequency data from the Web Audio API.","The important concept to Google is the createAnalyser method. createAnalyser() is available on any audio "context" (a concept we touched on in detail in the original Web Audio API post).","↩Note that for the yScale we're setting the range in reverse (from height to 0).","This is so that our line starts at the bottom of the graph.","↩"]},{"title":"Podcasting: what gear do you need?","url":"/podcasting-equipment/","content":["We're in the setup-and-planning phase of launching a new podcast at my new job, and once again I found myself sharing my Podcast Equipment 101 tips with a team who are interested in podcasting but have no prior experience of working with","audio.","This is not the first time I've written a guide like this, so I'm publishing this one here for posterity (mostly so I don't have to type it all out again next time).","I've broken this guide into three sections: an overview of the essential equipment that you'll need to record a podcast, my personal recommendation for what I think you should use (if you're looking for a TL;DR, this is it), and a few quick","hacks for better recordings","Essential equipment","To record a good podcast, there are three things that you will unequivocally[^1] need.","Yes, this is just my opinion.","Yes, you can make "a" podcast with less than this.","No, I don't think that podcast would be a "good" one.","The quality bar is low, but there is a bar.","Get these three things sorted and your quality is guaranteed to be "good enough".","A "good" microphone","A connection between your mic and your computer","Software to record the audio from your mic","1.","Microphones","It is possible to record a podcast with anything; your phone's earbuds, a dictaphone, your laptop's built in speaker, anything you can think of.","The content is of course more important than the equipment you use.","However, the biggest distinction between amateur and professional podcasts is the audio quality, and the single best investment you can make is to get a decent microphone.","The difference between even a "cheap" dedicated microphone and the basic methods outlined above is massive.","There are three types of microphones that are good for podcasting:","Dynamic mics Good examples of these are the basic Shure SM58 (~£85) and the fancy SM7B (~£350).","Dynamic mics have a small diaphragms (the membrane that converts sound waves to electrical signals), and only pick up noise that is very close to them.","Dynamic mics are good for stage performers, but also useful for people recording in noisy environments (although you'll have to remember to keep your mouth very close to the mic).","Condenser mics Good examples of these are the Rode NT1-A (~£150) and the Neumann U87 (~£2,500) These are often "studio grade" microphones, and are great for recording musical instruments and crisp, clear voices.","Most entry-level podcasting mics are condensers with large diaphragms, which means they pick up every sound in the room (be careful that you're not recording in a noisy environment).","These will require a shock mount to eliminate noise from vibrations in your desk and a pop shield to remove the "bangs" that plosive sounds ("a","b", "p", etc.) make when you speak directly into a","sensitive microphone.","Lavalier mics Good examples of these are the Rode Lavelier Go (~£50) and the Sennheiser MKE40-4 (~£300).","These are the tiny clip-on mics that people often use at conferences, and as a result are often part of a "wireless" setup.","These are not common for "normal" podcasting, but can be useful for setups with multiple non-technical people in the same room.","Mic technique is less of an issue with these, but the sound quality suffers a little compared to standard desk-mounted mics.","2.","Connections","Once you've got your mic, you need a way to connect it to your computer so you can record it.","For most microphones, this will involve running an XLR cable from the mic into an "audio interface" that connects to your computer via USB.","There are many, many types of audio interfaces that range in price and features, but you can't go wrong with the Focusrite Scarlett Solo (~£100) which allows you to connect a single mic to your computer, or the Scarlett 2i2 (~£150) that","allows you to connect two mics at the same time.","Some podcast-focused microphones don't require an audio interface, and can be connected to the computer directly via USB.","Good examples of these are the Rode NT-USB (~£140) and the Audio-Technica AT2020USB+ (~$130).","You'll also need a stand for your microphone.","Boom arms give you the best flexibility for an at-the-desk setup (and also give you that drive-time radio DJ vibe), but small tripods that sit directly in front of you are fine.","Remember, though, that for best results your mouth needs to be as close to the mic as possible, and this can be awkward if you're trying to use a keyboard with a desk-mount between you and the keys.","Boom arms offer the best UX.","Depending on what mic you choose, you may need some extra peripherals.","The Shure SM7B, for example, needs a pre-amp to boost the signal before it goes into the audio interface.","These are uncommon, however, and most common setups don't require anything beyond the items outlines in this doc.","3.","Software","Audio software is an overwhelming mess of options, so I'll do my best to be brief here.","What software you use depends on your role in the podcast.","If you're going to be doing any editing, you'll need a full Digital Audio Workstation.","If you're just a host or guest, then all you need to worry about is capturing the audio from your microphone and there are plenty of simple options for this.","A Digital Audio Workstation (often referred to as a DAW, which can be said "dee, a, double-u" or "door" depending on how nerdy you're feeling) is a full-featured audio editing suite.","With a DAW you can do everything from trimming the ends of an audio file to pitch-shifting your guest's voice so it sounds like they're singing.","Good examples are the Mac-native Logic Pro (~£180), the cross-platform industry stalwart Pro Tools (~$30 per month), and the open-source Audacity (free).","These are often expensive and can be intimidating to new users, but a good DAW will give you everything you need to record, edit, and export your podcast episodes.","If you're releasing a podcast, someone on your team will need to have a DAW and know how to use it, but it's definitely overkill to expect guests and non-editing hosts to buy one and learn it.","While you can "just record" with a DAW (or even a halfway-house like Garage Band), any simple audio-capture software is fine.","If you're using a Mac, then Quicktime is perfect.","I'm sure there are Windows/Linux equivalents to Quicktime, but they've never crossed my radar.","The key features are being able to select which input device you're recording from (a.k.a. the fancy mic you've just plugged in) and to make sure that you're recording in high quality (without any of the compression that tools like Zoom and","Skype introduce to save bandwidth).","So what should I use?","If you're not the person doing the editing, then get a decent USB microphone (either of the two mentioned above would be fine).","Get a cheap boom arm and record with Quicktime.","If you want the "best" podcasting setup, then get an SM7B instead of the USB mic.","You'll then also need a Cloudlifter preamp, an XLR cable, and the Scarlett Solo audio interface.","Okay, so maybe The BestTM is subjective, but this setup is at least the Industry StandardTM.","And if you're the designated editor, then I'd recommend buying (and learning) Logic Pro.","Logic is the DAW I use (after having tried a lot of the alternative options over the years) and you can see a list of some of the other bits of podcasting equipment that I use in the Audio Gear section of my /uses page","Good technique is more important than good equipment","The most important caveat to all of this is that if you don't use the equipment properly there's no point using it at all.","Even if you spend thousands on mics and peripherals, if you're pointing the mic in the wrong direction and recording in an echoey room then your audio will still sound terrible.","Quick hacks for better recordings:","Always always always use headphones.","You'll probably be talking to someone over Skype or Zoom, and you only want your microphone to pick up the noises that you make.","Keep your mouth as close to the mic as possible, and make sure you're talking into the correct part (most dynamic mics record through their top, whereas condensers often record from the side).","Don't record somewhere noisy, and especially if you're using a condenser mic be sure to turn off any humming computers or air-conditioners etc.","Minimize the amount of flat, reflective surfaces near where you are recording.","Sound reflections from large flat walls can cause feedback loops or just bad-sounding echoes.","The worst place to record is in a perfectly square room with bare walls.","Pull the curtains, and try to record in a room with lots of soft furnishings.","The surface directly behind the mic (from your perspective) can have a big impact on the sound: if your mic is on your desk in front of you, put a jumper or cushion between the mic and your computer to help minimize unwanted reflections.","There's a lot more to podcasting than I've covered here (hosting, editing, marketing, actually making good content) but this guide feels appropriately focused on the equipment side of things.","Perhaps I'll come back to this well for more how-to articles in future.","If that's something you'd be interested in, let me know on Twitter (I'm @thomashazledine).","Thanks for reading, and good luck with your podcasting adventure.","Be sure to share your show with me when you've launched it, I can't wait to hear it."]},{"title":"Humility in software development","url":"/humility-in-tech/","content":["I'm in the getting-to-know-you phase of a new job, and it's got me thinking about the aspects of my work-personality that I've deliberately chosen.","I'm sure I've got plenty of inadvertent character traits that I've no idea of or control over (for good, I hope, but also inevitably for bad).","But there are few I've consciously tried to foster, and the biggest of them is humility[1].","It's maybe not the perfect word to describe the concept I'm grasping at, and there's certainly no getting away from the "ever so humble" Uriah Heep references or the image of the 45th US president saying "I am humble.","I think I'm much more humble than you would understand".","But it is the best word I've been able to come up with to encapsulate an idea that has been specifically valuable to me in the context of my career in tech.","What do I mean by "humility"","I guess the multi-word version of what I'm trying to express is a sentiment I've heard a few times: never be the smartest person in the room.","That is great advice for anyone who wants to get better at what they do, and sets up a brilliant atmosphere for learning.","And for it to work, there's a prerequisite for possessing the humility to admit to the limits of your knowledge and expertise.","It can be tough to admit when you don't know something, but opening yourself up to that level of exposure is (in my experience) a sure-fire way to level-up your own skills.","I also extend this principle to non-skills-specific aspects of my work life.","I've never suffered when working with someone who downplays their ability or is not confident in their own skills.","I have, however, absolutely hated working with people who think they know it all.","When I say "be humble" I mostly mean "don't be a person that people don't want to be around".","I aspire to be a >1✕ developer, and I want my work to be the best it can be.","But the concept of a "10✕ developer" is toxic.","I don't want to do the work of ten developers; I want to enable the other devs in my team to do better work.","I want to be a "force multiplier" to the rest of my team; my colleagues should be able to do better work because I'm around.","When has humility helped my career?","Everyone learns differently, but having someone directly explain a new concept or technique to me is my preferred way to learn.","By exposing the gaps in my knowledge and actually asking for help, I get to move to the next level of understanding so much faster.","In meetings, I always try to ask the "dumb" questions.","I'm sure someone else in the room will also be wondering what that TLI[2] means, or what the unspoken objectives are.","No none has ever shouted at me for being an idiot, and in fact quite the opposite has happened.","People thank me for engaging and asking questions (both in public and in private).","When reviewing PRs I try to never, ever, ever, be dismissive of the submission another developer has spent time on.","Even if my gut instinct is to literally throw my laptop out of the window in the hope I never have to view that disgusting code again, I step back and take a breath and remind myself how often my gut has been wrong in the past.","Plainly put, my gut has terrible instincts.","A simple friendly interrogation can often expose the intention that I've missed, or the bug I didn't realise needed fixing.","No-one writes bad code on purpose, and even if I've got a lot to say and many changes to suggest, I always try to present my ideas as possibilities rather than edicts.","A good PR should be a conversation, and the ultimate responsibility lies with the author.","When digging through "legacy" code, it always pays to have humility.","It's so easy to think my ideas are so much better than the garbage code that I've been forced to work with.","The temptation to refactor everything into my style and to flow the way I think it should flow is real.","Every now and then I'm right - my version is better.","But that is so rarely the case.","If I start with the understanding that the people who came before me knew what they were doing, I'm more able to spot the subtle bugs they've avoided or the elegant performance improvements they've made by not doing things the way I would","expect.","I wish I'd learnt that last point so much sooner - it would have saved a lot of time and painful on-call refactoring!","When has humility hurt my career?","Another mantra I live with is that "a plan is not a strategy unless the opposite approach is also valid". "Eat food" is not a strategy, because not eating is not a valid option. "Don't eat meat" is a strategy,","because eating meat is perfectly normal.","Each option has pros and cons, but both are valid: it's a strategic choice.","So what is the opposite approach to humility?",""Arrogance", I guess.","And yes, sadly, in tech choosing to be arrogant can be a beneficial strategy.","Act like an expert, and some people will treat you like an expert.","Honestly, a lot of this tech-career malarkey is a perception game.","If you make a good first impression and your superiors decide that you are "a smart person with good ideas" then you can go a long way without having to actually prove yourself to anyone.","If, like me, your whole schtick is "your ideas are better than mine; tell me more about them", then some people might take that to mean that you don't have any ideas.","And a bad impression sticks much more thoroughly than a good one.","A CEO's good opinion, once lost, is lost forever.","I've had to fight this a couple of times.","But even though I'm asking the "stupid" questions and deferring to others' expertise, I know exactly how good I am.","I also know how valuable this tactic has been to my long-term career advancement (and general mental wellbeing!), so I've made it my policy never to compromise.","A healthy team instinctively knows the value of humility already.","And an unhealthy team; well, that's not a team I want to be a part of.","The elephant-in-the-room caveat","I'm a white straight male with a private school education[3].","That alone gives me more leeway to risk appearing "lesser" than the people I'm working with.","Despite having done nothing to earn it, people are more likely to give me the benefit of the doubt and assume I'm "playing devil's advocate" or just "digging into the nitty gritty details" rather than seeing me as",""slow" or having "not grasped the overall concept".","People with different backgrounds may well find the suggestion to "show more humility" to be unhelpful and downright insulting.","So I think my advice only applies to people who never find their abilities questioned.","If you've had to fight tooth and nail just to get in the room, then you probably need (and deserve) to show confidence.","It's those of us who get all that respect for free that need to tone it down a bit.","If more developers in my position showed a bit more humility, it would open up a lot of doors for other people.","Is it a useful concept for guiding my actions?","I've gone from self-taught and self-employed to junior-dev to mid- to senior- to whatever-you-want-to-call-what-I-am-now.","I've not always been able to live up to my ideals of humility, and have learnt painfully from those experiences.","But when I've been able to live up to (or at least near) my lofty ambitions - those are the times I've done my best work.","Those are the times I've made the best friends, had the best experiences, and made the best advancements in my career.","My new colleagues know very little about me beyond my CV, so if all I can do in these early stages is show that I'm open to conversations and happy to learn wherever possible, then I'll call that a win.","I know, I know!","But trust me; keep reading.","The idea is less gross than it sounds.","I promise.","↩︎","A TLI is a Three Letter Initialism.","Har har.","↩︎",""Lord, grant me the confidence of an average white man." - I know this is a thing, and it sucks.","↩︎"]},{"title":"Improving my Wordle opening words using simple node scripts","url":"/wordle-node-script/","content":["Let’s be honest, by this point you’re either addicted to Wordle1 or sick to death of hearing about it.","And yes, I'm shamelessly using Wordle as click-bait for this post.","Having lured you in with the promise of Wordle-insight (which I will deliver!)","I'm going to try and get you excited about writing tiny scripts.Tiny scripts that you run in your terminal can help with, well, pretty much anything involving large amounts of data.","They can help you with the stuff that humans are generally bad at but computers are great at.","Like analysing the 12,972 words that Wordle uses in its code.I want to find a sequence of guesses that covers as many of the most common consonants as possible.My strategy is to hit the most common letters as quickly as possible, but at the","same time be able to get through as many guesses as possible without repeating letters.","I’m not “solving” Wordle; I just want to optimise my opening sequence of guesses to make playing it more fun.2The practical considerations of my approach:Try to minimise the amount of vowels in the first couple of guesses.","Once you get past guess #3 words are harder to form without repeating letters, so keeping some handy word-makers in my pocket is helpful.It would be really helpful to know which letters actually are the most common (rather than just relying","on my flawed gut-feel).","I need a "complete" dictionary of words to choose from (a.k.a. my own vocabulary is not good enough).","I can solve all three issues by writing a little bit of code that I'll run on the command-line in my terminal app.Scripts can be gross and "bad", as long as they workOne joy of tiny scripts is that they're just for me","and all I really care about is the output.","That means I can use whatever language I like.","I spend most of my time writing JavaScript so I'll use Node for my scripts, but you can 100% get the same results with Python or PHP or bash or whatever you like.It also means that I can play fast and loose with "best","practices".","Working something out for thousands of items (in this case the Wordle words) is practically impossible by hand, but easy for a script, so even if my code is objectively terrible (with multiple inefficiencies and O(n log n) complexity), the","computer can still calculate the result in fractions of a second.I also don't need to worry about extensibility or re-use.","As long as it works for the data-set I'm working with right now, that's fine!","I'll just throw this script away when I'm done anyway.Where does Wordle get its words from?","The source code of Wordle3 contains two arrays of words which I'm calling "winners" and "guesses". winners has 2,315 items and shows all the potential winning words, whereas guesses is longer (10,657) and contains","all possible valid guesses.export const winners = ["cigar", "rebut", "sissy", "humph", "awake", /* etc.","*/ ],","export const guesses = ["aahed", "aalii", "aargh" /* etc.","*/];","// Combine the two lists into one array","export const words = [...winners,...guesses];","I've saved these lists in a js file called `/dictionary.js`Which letters are most common?","Working out the frequency of letters is a three-step process.","The first step is to count how many times each letter appears in the list of words.","Then I need to sort those letters into a useful order (most frequent at the start, least frequent at the end).","The third step is to work out the relative frequency of each letter from the raw count.The initial counting is done by importing our word list4 and flattening it to a single string of letters with words.join('').","We then turn this into an array of letters with a "spread": [...words.join('')].","The we run that array through a reduce() operation to create an object where every letter is a key with a the letter's count as a value.import { words } from "./dictionary.js";","const letterFrequency = letters =>","letters.reduce((total, letter) => {","total[letter] ?","total[letter]++ : (total[letter] = 1);","return total;","}, {});","const allLetters = [...words.join("")];","const unsortedFrequencies = letterFrequency(allLetters);","That letterFrequency() function produces an output that looks like this:unsortedFrequencies: {","h: 1371,","d: 2060,","l: 2652,","// etc...","}","Sorting is a bit fiddly.","Because our unsorted letters are in object-form, we'll need to use Object.keys(frequencies) to allow us to map through the keys and apply a .sort() to them.const sortFrequencies = frequencies =>","Object.keys(frequencies)",".map(key => ({","letter: key,","count: frequencies[key]","}))",".sort((a, b) => frequencies[b] - frequencies[a]);","const sortedFrequencies = sortFrequencies(unsortedFrequencies);","The third step is to calculate the percentage that each letter occurs in the full list of letters.","I'm aiming for accuracy to two decimal places here, hence the * 100 and / 100 dance I need to do to get the rounding to work.const calculatePercentages = (totalCount, frequencies) =>","frequencies.map(frequency => {","const percentage =","Math.round((frequency.count / totalCount) * 100) / 100;","return { ...frequency, percentage };","});","const percentages = calculatePercentages(allLetters.length, sortedFrequencies);","Letter frequency resultsSo what does this little script discover?","I've saved it in a file called frequencies.js, so to run it I open my terminal, cd to the directory that contains the script, and run:node frequencies","For my real version I "wasted" some time building a little function to print the results as a graph directly into my terminal (see the screenshot), but for this post I've prettied-up the results a little more.My first run of","the script found the most common letters in the entire Wordle "dictionary".","Unsurprisingly the vowels all ranked pretty high, but the most commonly occurring letter was S.5%10%15%20%01:S10% (6665)02:E10% (6662)03:A9% (5990)04:O7% (4438)05:R6% (4158)06:I6% (3759)07:L5% (3371)08:T5% (3295)09:N5% (2952)10:U4%","(2511)The 10 most common letters in the Wordle dictionaryFor my second run of the script, I filtered out the vowels.","This doesn't change the values for any of the letters, but lets me focus on the letters that I think might be more "valuable" when choosing Wordle guesses.5%10%15%20%01:S10% (6665)02:R6% (4158)03:L5% (3371)04:T5% (3295)05:N5%","(2952)06:D4% (2453)07:Y3% (2074)08:C3% (2028)09:P3% (2019)10:M3% (1976)The 10 most common consonants in the Wordle dictionaryFor completeness, and to assuage my curiosity, I also ran the script against just the "winning" words.","This one is potentially a "spoiler" and some could call it cheating, so feel free to skip these results if you like.","It was interesting, however, to see how the frequencies changed for this reduced set of words!","the spoiler5%10%15%20%01:E11% (1233)02:A8% (979)03:R8% (899)04:O7% (754)05:T6% (729)06:L6% (719)07:I6% (671)08:S6% (669)09:N5% (575)10:C4% (477)11:U4% (467)12:Y4% (425)13:D3% (393)14:H3% (389)15:P3% (367)16:M3% (316)17:G3% (311)18:B2%","(281)19:F2% (230)20:K2% (210)21:W2% (195)22:V1% (153)23:Z0% (40)24:X0% (37)25:Q0% (29)26:J0% (27)The most common letters in the "winning" Wordle word listFinding the best sequence of wordsThe next script I need will be my word","finder, which I'll create in a file called wordfinder.js.","To apply the letter-frequency information from the previous step, I'll need a script that can do two things:Find all the words from the list that contain a given set of letters (a.k.a. find all words in the list that contain",""a", "b", and "c")Find all the words that do not include any of the letters in a given set (a.k.a. find me all the five-letter words that do not contain: "abcdefghijk")// Find words that contain *all*","input letters.","const inclusiveMatches = (input, words) => {","const letters = [...input];","const matches = words.filter(word => {","const hits = letters",".map(letter => word.includes(letter))",".filter(hit => hit);","return hits.length == letters.length;","});","return matches;","};","// Find words that do *not* contain any of the input letters.","const exclusiveMatches = (input, words) => {","const alphabet = "abcdefghijklmnopqrstuvwxyz";","const allowedLetters = [...input].reduce(","(a, l) => a.replace(l, ""),","alphabet",");","const match = new RegExp(`^[${allowedLetters}]+$`);","const matches = words.filter(word => match.test(word));","return matches;","};","I could just input the full dictionary from the last script, but there are a couple of other steps I can add to make my search more focused.One of my objectives is to maximise the amount of the alphabet I cover with my opening Wordle","guesses, so I know straight away that I'm not interested in words with repeated letters.","A word where every letter is different is known as an "isogram", so with a little regEx magic we can filter out any isograms from our list.const isIsogram = string => !","/(.).","*\\1/.test(string);","const noRepeatedLetters = words.filter(isIsogram);","Once we have our list of words that match our criteria (words that contain all the MUST_HAVE_LETTERS but none of the MUST_NOT_HAVE_LETTERS), I would find it really useful to sort them by the amount of vowels.","Particularly when selecting our first or second Wordle guess, there could be a lot of options (we're dealing with a word-list with >12k entries, after all!).","One of my heuristics is that I want my early guesses to include as few vowels as possible in order to maximise the word options on the later guesses.Sorting by vowel-count is a two step process.","I'll need a function to count the number of vowels in a given word, and I'll need a function to sort the words by that count.const countVowels = (word, vowels = "aeiou") => {","const letters = [...word.toLowerCase()];","const justVowels = letters.filter(char => vowels.indexOf(char) > -1);","return justVowels.length;","};","const orderByVowels = words =>","words",".map(word => ({ word, vowelCount: countVowels(word) }))",".sort((a, b) => (a.vowelCount > b.vowelCount ? -1 : 1))",".map(object => object.word)",".reverse();","Putting it all togetherAssuming we've imported the word list and the aforementioned functions, the final flow of the script looks like this:const onlyIsograms = words.filter(isIsogram);","const set = inclusiveMatches(MUST_HAVE_LETTERS, onlyIsograms);","const subset = exclusiveMatches(MUST_NOT_HAVE_LETTERS, set);","const sorted = orderByVowels(subset);","sorted.map(word => console.log(word));","The only missing ingredients are the MUST_HAVE_LETTERS and MUST_NOT_HAVE_LETTERS variables.The script could probably be automated to loop through all possible sequences of words and populate these values dynamically, but that's a bit","more effort than I'm willing to expend.","And besides, this is the stage that requires (in my mind, at least) some external judgement.","I want to be able to pick Wordle guesses that feel right (otherwise I'll end up playing "yokul", "speug", or "phpht" - which definitely feel like cheat words even though they are genuine words from the","Wordle allowed-list!).","I can get the best of both worlds by adding support for arguments into my script.","If I can run node wordfinder include=abc exclude=xyz, for example, then I can quickly iterate over the options I like the look of manually. "Job done", in my opinion!","Adding arguments to a node scriptAdding support for arguments in my script involves setting up a config object with my default values, and then looping through the process.argv array (process is a variable that is built into Node and argv","is a handy property which makes all the arguments available).","For each argument, I'll split by "=" to give me a key and value for each one.","So running node myscript foo=one bar=two will give me key/values of foo/one and bar/two.","If I include arguments that match the keys of my config options (in this case, inc and exc) then I can overwrite those values, and for the rest of my script config.inc will equal the value I passed into inc=abc.const config = {","inc: "",","exc: "",","max: 10","};","process.argv.map(arg => {","const argParts = arg.split("=");","const key = argParts[0];","const value = argParts[1];","config[key] = value;","});","Do it!","Finally, after all this work, I can actually begin my search for a suitable sequence of opening guesses for Wordle!","The node frequencies script told me that the most frequently occurring consonants in the Wordle dictionary were S R L T N.","To see if (by some miracle) there's a valid five letter word comprised of those exact five letters, I can run my wordfinder script.node wordfinder inc=srltn","Unsurprisingly, there's no word that matches that criterion 😢.","But ditching the n returns three words: "rotls", "slart", and "tirls".","Hmmm, not exactly the "feels-right" words I was hoping for.","But given that I've already ruled out vowels from my must-have letters on the basis that they're more useful later on, surely s is just as handy a letter to have up my sleeve late-game.","Knocking that off the list opens up the options further.Here are the opening options I experimented with:Must-havesResult countResult wordssrltn0srlt3rotls, slart, tirlsrltnd0rltn1larntrltd1trildrlnd0rtnd3trend, drant, drentOf these, only","trend really feels right to me, so my next step is to run the script again to find my options for the second guess.","This time I'll include the exc argument to exclude the letters from my chosen first guess.node wordfinder inc=lycph exc=trend","Must-havesResult countResult wordslycpm0lycph0lycp0lyc13xylic, hylic, cymol, cylix, colby, cloys, clays, calyx, calmy, acyls, scaly, lucky, coalylypm6lymph, plumy, palmy, lumpy, imply, amplyI can then repeat the process, picking new target","letters for the inc argument and adding the previous word to the exc argument.","This is, in itself, a fun little word game that I've now created for myself, and my little node scripts are what have made it possible.I've uploaded the complete version of these scripts to GitHub, so you can download them and","play with them yourself (and tweak them, if you like).","What words did you end up choosing as your opening Wordle guesses?","Be sure to tell me about them on Twitter!","TakeawaysIf you've made it this far, then hopefully you'll feel empowered to write some fun little scripts to help you out with day-to-day problems.","Scripts like this one can save you loads of time (and can actually be used for useful things and not just silly word games).","The idea of writing and running my own helper scripts used to feel daunting to me, but knowing they can be written in whichever language I fancy and that there are no stakes whatsoever makes it all seem more accessible.So next time","you're faced with a repetitive task that might take you hours to complete by hand, why not burn a few days writing a script to automate it instead...","If you’re looking for the most mathematically optimal programatic solution, I’d strongly recommend Three Blue One Brown’s excellent video about solving Wordle using information theory.","↩I'm not very good at Wordle, but have never "failed" yet.","Fewer guesses would be great but my main aim is coverage.","I enjoy "clearing" the keyboard - that's the part of Wordle that I find most therapeutic.","Getting three or more correct letters in the opening guess is actually quite stressful - the pressure's then on to get a good score!","↩Wordle is blissfully open source in the best tradition of the web.","You can open a web inspector, and the all the code is right there for you to see.","And for those of you wondering if anything changed when the NYT took over, the only difference to the word list was that they removed a couple of tricky words.","↩Here I've used ESM syntax and exported and imported using export and import, but that required setting up a package.json in my script's directory and declaring the script to be a module with "type": "module".","You can save this hassle by using the CJS module.exports = words; and const { words } = require('./dictionary.js') pattern, but I think that's ugly.","Confused?","↩"]},{"title":"Oblique Strategies via npx","url":"/oblique-strategies-via-npx/","content":["In 1975 Brian Eno and Peter Schmidt released their Oblique Strategies; "Over One Hundred Worthwhile Dilemmas"","The Oblique Strategies constitute a set of over 100 cards, each of which is a suggestion of a course of action or thinking to assist in creative situations.","These famous cards have been used by many artists and creative people all over the world since their initial publication.","Fifth edition 2001.","You can buy your own set of the cards on Brian Eno's shop (www.enoshop.co.uk/product/oblique-strategies), but you can also get a mini taste of oblique inspiration in your terminal by running the following npx command:","npx oblique-strategy","Examples","┌────────────────────────┐","│                        │","│  Remove a restriction  │","│                        │","│    - Edition 4 (1996)  │","│                        │","└────────────────────────┘","┌─────────────────────────────┐","│                             │","│  Is the tuning appropriate  │","│                             │","│         - Edition 1 (1975)  │","│                             │","└─────────────────────────────┘","┌──────────────────────────────────┐","│                                  │","│  Fill every beat with something  │","│                                  │","│              - Edition 2 (1978)  │","│                                  │","└──────────────────────────────────┘","About the code","The full code for this npm package is viewable on my GitHub at github.com/tomhazledine/oblique-strategy, but there's not really too much to it.","There are more lines dedicated to drawing the box around the output than there are for actually picking the strategy.",""proof" that I own a legit copy of the strategies","The strategies themselves were all found on other GitHub repos.","I've deliberately not credited any of those projects because, well, they're clearly breaching some kind of copyright 🤷‍♂️ - as too, I should note, am I.","As recompense for my flagrant disregard of others' work, I've bought a physical copy of the Oblique Strategies for the princely sum of £50 and I'll take this repo down the second anyone complains.","In truth this was more of a learning exercise for me.","I wanted to master the npx process and learned a lot about package.bin in the process, so I'm considering this project a success."]},{"title":"Adding client side search to a static site","url":"/client-side-search-static-site/","content":["In the bad old days (a.k.a. "a couple of years ago") adding search functionality to a statically-generated site required third-party services that provided a search backend (like Algolia or Elastic or whatever).","Thankfully we can now run a (simple) search using only client-side tech thanks to libraries like Fuse.js.","As Fuse say in their docs: "you don’t need to setup a dedicated backend just to handle search."So how does it all work?","Why do we even want site search functionality?","And what are the steps required to add a decent search experience to a statically-generated site like mine?","Why add search to a blog like this?","It's a good question, especially when you consider that Google exists already.","But a built-in search function does significantly improve the user experience for a site like this.It offers readers a fast and direct way to find the specific content they're interested in, eliminating the need to navigate through","multiple pages or understand the overall structure of the site.","This ease of use not only enhances the site's accessibility but also increases engagement, as readers are more likely to stay longer and explore more content if they can easily locate the bits that interest them.Search fundamentalsAt","its simplest, a site-search function is a tool that allows users to search for specific content within a website.","When a user enters a search query into the search box on the site, the search function scans the content of the site to find pages that match the query and returns a list of results.This is normally a two-step process that consists of",""indexing" and "querying".","To start, an index is created of the content.","The index is a representation of all the information on the site that could be relevant to a search and is specifically created to enable efficient searching.","The index should contain metadata about each page, such as its title, description, and keywords, as well as the content of the page itself.Then, when a search is made, the search engine queries this index to find pages that match the query.","The results are then displayed to the user in a list, ordered by relevancy.","In the context of a content-search function for a website, the relevance of search results could be determined by factors such as the presence of keywords in the page title, the frequency of keywords in the page content, and the overall","quality of the content itself.So to build an effective search engine for a site, we need to have two key features:A way to create an index of the site's content.A way to query that index to return relevant results.The two options:","server-side and client-sideThere are two primary options when it comes to implementing search.","The traditional approach is "server-side", where the search is executed on a server.","The alternative, which we'll be focusing on, is "client-side", where the search is performed in the user's web browser.There are benefits and trade-offs for both approaches, but in my view client-side search is the best","option for a site like mine.Why not use server-side search?","In the past when creating static sites that needed a search function I've used an external service.","These have been server-side solutions like Elastic or Algolia, which index your site's content for you and provide API endpoints for performing searches.1.","CostThe biggest obstacle to adding server-side search to a static site is cost.","Even though my static site doesn't require a server, if I'm using server-side search there has to be server somewhere.","While there are free tiers available for many server-side services, it's likely that I'd need a paid plan.","Costs can escalate quickly depending on the amount of data indexed and the number of search queries made, and for a "continual tinkerer" like myself the amount of fine-tuning and customisation I'd want would inevitably result","in expensive bills.2.","FunFor a real business deciding between building or buying a complicated second-order function like search, it generally makes sense to let a third party handle the complexity in exchange for money.","As highlighted in the point above, I'm not keen on splashing too much cash on this project (aside from ethereal "career" benefits, this blog generates no income at all) but there are other benefits to building it myself.For","starters, having a proper project to work on is a great way to reinforce things I've learnt.","So by building my own search function I'll inevitably learn more about building search functions.","Plus I actually enjoy these kinds of nitty-gritty software challenges.Building it myself will also give me more control over the end result.","I'll be able to fine-tune and customise as much as I like, without having to fight too much against a system that has it's own conventions and defined way of doing things.","And that's what my whole career has been built on thus far; learning how to do things myself to get around the limitations of off-the-shelf products.But I'm not willing to compromise on the "staticness" of this site, so","if I'm determined not to use an external service then I'll have to find an option that works client-side.How does client-side search work?","For a static content site, client-side search should be a two step process.","The first step is to generate an index whenever a site is built.","The second step is to provide frontend components to perform a query against that index.The index-creation should be handled by the SSG, and the query functionality would be handled by component-level JavaScript.Creating the indexPart of","the power of using a Static Site Generator is that you can interact with all your content at build time.","By generating a search index when the site is built, I'm able to leverage this power to address one of the biggest problems with any client-side search: generating an index is a costly operation.To perform a search, the search engine","needs an index to query against, and to make an index requires the code to be able to "see" all the content within a site.","Server-side search engines traditionally "crawl" all the pages of a website to generate an index that is then stored server-side, but client-side search engines often generate their index in the client at the time a search is","made.","Using an SSG allows me to get the best of both worlds by generating the index at build time.This approach comes with an extra benefit, too: because the index is generated at build time, whenever the site's content changes (and thus","triggers a rebuild) the index is automatically regenerated.","No more waiting for crawlers or manually-triggered re-indexing.","Any search results always represent the most up-to-date content on the site.In a client-side context, the best format for a search index is a JSON array.","Each page in the site becomes an object in the array, each with a title, url, and content keys.","These are the keys I've chosen to use, but the beauty of "rolling your own" is that you can easily tailor this to your own content.","Are "tags" crucial to your information architecture?","Include them in the search index.","The same goes for anything else you think might contribute to the relevancy of search results; categories, keywords, meta-data, etc.","If you think it will produce better results, include it in the index (and then test and refine over time).","Once created, I need a way for my client-side JavaScript to be able to read the index, so I need to make it available to the browser.","The easiest way to do this is to save the index as a JSON file and include it in the site's build output.","This way the index is available to the browser at the same time as the rest of the site's content.","You can see the index for this site here: search-data.json[","{",""title": "Adding client side search to a static site",",""url": "/client-side-search-static-site/",",""content": [",""In the bad old days (of, like, a couple of years ago) adding search functionality to a statically-generated site required third-party services that provided a search backend (like Algolia or Elastic or whatever). ",",""Thankfully we can now run a (simple) search using only client-side tech thanks to libraries like Fuse.js. "","// etc...","]","},","{",""title": "Oblique Strategies via npx",",""url": "/oblique-strategies-via-npx/",",""content": [",""In 1975 Brian Eno and Peter Schmidt released their Oblique Strategies; "Over One Hundred Worthwhile Dilemmas"",",""The Oblique Strategies constitute a set of over 100 cards, each of which is a suggestion of a course of action or thinking to assist in creative situations."","// etc...","]","}","// etc...","]","Note: Because of the way our matching engine works (see the section on Fuse.js later in this article), the content body is split into an array of sentences.","This is achieved using Intl.Segmenter:const segmenter = new Intl.Segmenter("en", { granularity: "sentence" });","const segments = [...segmenter.segment(content)]",".map(segment => segment.segment)","// Cap sentences at 240 chars",".map(line => chunk(line, 240))",".flat()","// Remove trailing whitespace",".map(line => line.trim())","// Remove empty lines",".filter(line => line.length > 0);","Querying the indexWith the index handled, the next step is to write code to query that index.","At the core of the search engine is a "matching" function that consumes the index and any given search term and returns a list of results ordered by relevancy.A simple "matching" function would search the index for exact","matches of a search term, but in practice this does not produce useful results.","A more sophisticated matching engine should account for several things:It should consider variations of the search term, such as singular versus plural forms or different verb tenses, which can help to find more accurate and relevant","results.It should account for synonyms or related terms that might be relevant to the search query, allowing for a more comprehensive set of results.","For example, a search for "car" might also include results about automobiles, vehicles, or transportation.It should account for "stop words", which are common words often excluded from search queries because they are","considered to have little semantic value or meaning.Stop words are particularly tricky, as sometimes they should be included and sometimes omitted.","Stop words are often articles, prepositions, and conjunctions, such as "a", "an", "the", "in", "on", "and", and "or".","A naive search function would treat them like any other word in the search query which can lead to search results that are cluttered with irrelevant pages that happen to contain the stop words.","For example, if a user searches for "the best coffee in town," a naive search function may return results for every page that includes the words "the", "best", "coffee", and "in" regardless","of whether the page is actually about coffee or has any useful information about the best coffee in town.A smarter search function would take stop words into account and exclude them from the search query, focusing instead on the more","meaningful keywords.","But it would also be able to handle cases were users are searching for an exact match.","For example, if a user searches for "red shoes" (in quotes), the search engine will only return results that contain that exact phrase, rather than returning any content that contains either "red" or "shoes"","separately.In short, writing a matching engine is a big project and not one to take lightly.","After several POC projects where I compared hand-written options to off-the-shelf options, there was one clear winner: Fuse.js.Fuse.jsFuse.js is a JavaScript library that focuses on what it calls "fuzzy" searching.Generally","speaking, fuzzy searching (more formally known as approximate string matching) is the technique of finding strings that are approximately equal to a given pattern (rather than exactly).","Fuse performed well in head-to-head tests against other options (including a complex hand-rolled function) and has good documentation, wide usage, and is under active development.As with anything code-related, specific implementation","details get complicated quickly, but the basic implimentation of Fuse is a straightforward process.","We initialise a new instance of Fuse with our index data and a configuration object, and then we can call the search method on that instance with a search term.","The search method returns an array of results, each of which contains a reference to the original item in the index and the matched text.const options = {","includeMatches: true,","findAllMatches: true,","minMatchCharLength: 2,","threshold: 0.3,","ignoreLocation: true,","keys: [","{ name: "title", weight: 2 },","{ name: "content", weight: 1 }","]","};","const index = await fetch("/search-data.json").then(res => res.json());","const fuse = new Fuse(index, options);","const results = fuse.search("search term");","The configuration options are powerful and allow for a lot of fine-tuning.","The keys option is particularly useful, as it allows us to specify which fields in the index should be searched and how much weight should be given to each field.","In the example above, we're telling Fuse to search the title field with twice as much weight as the content field.","This means that a match in the title field will be considered twice as relevant as a match in the content field.I'm also setting the threshold to 0.3 (meaning that only results with a score of 0.3 or higher will be returned) which is a","comparatively low value.","My site doesn't have thousands of pages, so I'm happy to return a lot of results and let the user decide which is most relevant.","If you have a larger site, you may want to increase this value to return fewer results.OptimizationUltimately there are some trade-offs and compromises that come from choosing client-side over server-side search.","These are few, thankfully, and I've mitigated as many of these as I feasibly can.While the configuration options are powerful, they are not as powerful as a fully server-side stack and Elastic (or other similar services).","A client-side option is always going to be constrained by the power of the browser it's running inside.For deep sites, the index can get quite large (sometimes up to multiple megabytes for sites with thousands of pages).","For the search to work that index needs to be loaded into the client.","I've mitigated this by lazy-loading the index and leveraging caching, but there's no way to search an index without having the index.","This is one area where server-side power can really speed things up.","But on the flip-side, a server-side search will always require a network round-trip to fetch the results, so while the client-side search is slower on first load, every actual search feels pleasingly fast.ConclusionI'm really happy","with the search experience on my site.","It's fast, it's accurate, and it's easy to use.","It's also a great example of how powerful modern browsers are and how much can be achieved with a little bit of JavaScript.And because I've built it myself, I can tweak and improve it as I see fit.","I can add new features, change the UI, and make it work exactly how I want it to.","I can also use it as a learning tool to better understand how search engines work and how to improve them."]},{"title":"What if minesweeper kept getting harder?","url":"/what-if-minesweeper-kept-getting-harder/","content":["I've burnt thousands of hours playing Minesweeper over the years, and my preferred method of keeping things interesting was to add a mine everytime I completed the grid.","I often wondered if someone would automate this process, and (as is the way of all software nerds) I inevitably decided to do it myself.Play my incrementally harder version hereThis soon turned into quite the project, so I'm explaining","myself here.","But before I get to that, it's worth explaining how Minesweeper works (for those of you who didn't sink thousands of hours into Windows 3.11/95/98).","How to play MinesweeperPlaying Minesweeper involves clicking cells on a grid to reveal their contents.","If you click a cell that contains a mine, you lose.","If you clear all the cells that don't contain a mine, you win.The basic controls:Left-click (or a tap on a touch-enabled device) reveals a cell.Right-click (or a long-press) flags1 or unflags a cell.If the cell you click is adjacent to","any mines, it will show you the number of mines that it is adjacent to.","For example, if a cell has a 1 on it, there is 1 mine next to that cell.","This could be above, below, left, right, or diagonal to the numbered cell.This means you can use logic to deduce where the mines are.","For example, if a 8 is surrounded by eight unrevealed cells you can deduce that every surrounding cell contains a mine.If you click a cell that is blank (i.e. contains no number), all the surrounding cells will be revealed.","If any of those cells are also blank, all the surrounding cells of those cells will be revealed, and so on.","This is called a "cascade" and is a key part of keeping the gameplay speedy.Take this example 4x4 grid version of the game...","For the first move, the player clicks to reveal cell B3.","It contains a 1, so the player knows that there is one mine adjacent to that cell, but it isn't a particularly useful opening move.","There's no clue which of the adjacent cells contains the mine.The player then clicks to reveal cell A3.","This cell is blank, so the "cascade" kicks in and reveals all the connected black cells and the numbered cells that are adjoining.","This tells a smart player that cells C3 and D3 must contain mines.Our player, however, is not smart and clicks on C3.","This is a mine, and they lose the game.How does my version differ from "classic" Minesweeper?","1.","Difficulty (the main point of making my version)In my version, the difficulty increases as you play.","This is achieved by adding a mine to the grid every time you complete it.","This means that the grid gets more difficult to complete as you play, and the number of mines becomes your score.2.","SizeFor my version, I've constrained things to a 10 x 10 grid.","The primary reason for this is that I wanted to make the game playable on mobile devices, and ten-by-ten is about the largest I could fit on a phone screen and still be able to play comfortably with my fat and clumsy fingers.4.","You always start with a clear cellNo matter which cell you click first, it will always be a clear cell.","This is to avoid the frustration of losing on the first click, which is a common occurrence in the original game.5.","FlaggingAt first I left the concept of "flagging" or "marking" cells out of my version, taking the view that "flagging is for chumps".","But early playtesters were adamant I add it, so I caved to the pressure.How well will you do?","I've been tinkering with this version for about a year now, and my all-time high score is 36.","Can you beat that?","If so, take a screenshot and show me on Mastodon (@tomhazledine@mastodon.social).","Note: "flagging" (or "marking" or "chording" in some versions of the game) is for chumps who aren't confident enough to yolo their way to victory.","↩"]},{"title":"Improving SVG chart interactivity with Voronoi diagrams","url":"/improving-svg-chart-interactivity-with-voronoi-diagrams/","content":["Scatter plots are a fun way to visualize data, showing both the relationship between two variables and the distribution of data points.","And they're reasonably trivial to create in web-friendly SVG (especially if you get a library like D3.js involved).","One issue, though, is hover interactions.I've made so many graphs that include hover-triggered tooltips, and the sheer precision required to target those points with a cursor can be infuriating.","You can solve this UX problem by making the points (and thus the hover-targets) bigger, but that inteferes with the design of the chart as a whole.","For charts with complex datasets, it's often not feasible to make the points big enough to be easily targeted.Tiny points require precision to target with a cursorAdding a hitbox to our pointsWhen I'm building charts like these","I'm using SVG, so there are plenty of options to try.","One approach is to really dive into the "hitbox" analogy from video-games.","The visual part of the datapoint doesn't have to be the only part you can target with the mouse!","What is a "hitbox"?","In video games, hitbox is a term used to describe the area in which a character can be hit by an attack.","While a character's appearance can be a complex shape, the hitbox is often much simpler; a primitive circle or rectangle that roughly covers the area of the character.Visual size `!","==` hitbox size.Illustration of how a hitbox can extend beyond the visual portion of a characterMake the logo hitbox biggerWith SVG, we can use the <g> tag to "group" several elements together.","This means we can replace a datapoint's single <circle> element with two circles and treat them as a single element.In this example, we have a circle with a radius of 2 that fires the handleHover() event when hovered.","(Note we're assuming the SVG is being writen as a React component in JSX, with d as the data point and handleHover as the event handler.)const DataPoint = ({ d, handleHover }) => (","<circle r={2} cx={d.x} cy={d.y} onMouseOver={handleHover} />",");","To add a bigger hitbox to this point, we can add a second circle with a larger radius and no fill.","Because we're adding fill: "none" as a style rule this circle will be invisible, but it will still trigger the handleHover() event when hovered if we set pointerEvents to "all".const DataPoint = ({ d, handleHover })","=> (","<g>","<circle r={2} cx={d.x} cy={d.y} />","<circle","r={10}","cx={d.x}","cy={d.y}","onMouseOver={handleHover}","style={{ fill: "none" }}","pointerEvents="all"","/>","</g>",");","So now we can render a scatter graph with the original design we wanted, but with the added benefit of a larger hitbox for each point.The hitbox is now larger than the visual portion of the datapointMore hitboxes, more problemsThis approach","works well for charts with sparse datasets, but it doesn't scale well.","We've now effectively added a buffer zone around each of our points that will obscure anything behind it.","This means that if we have a lot of points, we'll end up with a lot of overlapping hitboxes.","And overlapping hitboxes means that we could easily be trying to hover over one point, but accidentally trigger the hover event for a different point.Ideally we'd like to have hitboxes that are as large as possible without overlapping","any other points.","What we need, it turns out, is a Voronoi diagram.What is a Voronoi diagram?","At its simplest, a Voronoi diagram divides our chart area into "cells" that are each assigned to a single datapoint.","The cell for a given datapoint is the area of the chart that is closer to that datapoint than any other datapoint.Example Voronoi diagramVoronoi diagrams are named after the Russian mathematician Georgy Voronoy, who introduced the concept","in 1908 (although the concept can be traced all with way back to Descartes).","Mathematically, the process of creating Voronoi diagrams is a complicated procedure involving Delaunay triangulation and "circumcircles" (all a mystery to a mere code monkey like myself!), but thankfully we don't have to deal","with that directlty when creating our charts in SVG.Voronoi diagrams in D3.jsIf you're not familiar with D3.js, it's a JavaScript library for creating data visualizations in SVG (it can do canvas stuff too, but all I ever use it","for is generating path data to pipe into SVGs).","I use it all the time for converting data into charts and graphs.It's a powerful library, but it can be a bit daunting at times and a lot of the demo code you can find (which is plentiful and inspiring) assumes a D3-only workflow.","My approach is a bit more peacemeal, as I just use the parts of D3.js that I need to generate path data and coordinates that I then "manually" add to SVG in JSX.The circles in the example-code above aren't using D3 directly,","but my normal workflow is to use D3 to map my data to a pixel domain, so the cx and cy data will have been run through a D3 "scale" function.","I'll write more about how I use D3 scales in a future post.So, to create a Voronoi diagram in D3, we need to use the Delaunay and Voronoi modules.","We can then use the Delaunay.from() method to create a Delaunay triangulation from our data, and the voronoi() method to create a Voronoi diagram from that triangulation.import React from "react";","import { Delaunay } from "d3-delaunay";","const HoverTargets = ({ data, scales, layout, handleHover }) => {","const delaunay = Delaunay.from(","data.map(d => [scales.x(d.x), scales.y(d.y)])",");","const voronoi = delaunay.voronoi([","layout.graph.left,","layout.graph.top,","layout.graph.right,","layout.graph.bottom","]);","const shapes = data.map((d, i) => {","const path = voronoi.renderCell(i);","return (","<path","key={`hover-target-${i}`}","pointerEvents="all"","d={path}","onMouseOver={e => handleHover(e, d)}","/>",");","});","return <g>{shapes}</g>;","};","export default HoverTargets;","In this example we created a HoverTargets component that will map over our data and return an SVG <path> for each point (this will be the voronoi "cell").","Within the component we create a Delaunay triangulation object from our data points using the Delaunay.from() method.","The data.map(d => [scales.x(d.x), scales.y(d.y)]) part of the code uses D3 scales to translate our data points into pixel positions.Once we have the Delaunay triangulation, we create a Voronoi diagram by calling delaunay.voronoi() with","the boundaries of our graph area as an argument.","The result is a Voronoi diagram that perfectly fits within the confines of our chart, and it automatically divides the space into cells that correspond to our data points.The we populate our <path> into the shapes array, with the d","value for each path based on the Voronoi cells.","When a cell is hovered over, the handleHover function is called, and the associated data point is passed as an argument.","This gives us a perfect hitbox for each point.Voronoi diagram incorporated into a scatter plotMaking it interactiveThe final step is to make the Voronoi diagram interactive by adding an onMouseOver event listener to each Voronoi cell.","Then, when the cell is hovered that listener passes the associated data point to the component's state, which we can then use to apply our hover effects to the correct point on the graph.","This can include stylistic effects (like visually highlighting the active point, as in the example below) or functional effects like showing a tooltip or adjusting other parts of the graph.Using the Voronoi shapes as hitboxes for the","points.","The first diagram shows the hitboxes, and the second shows the points with the hitboxes hidden.I'll acknowledge that this overview does contain a few "draw the rest of the owl" moments.","I've assumed a certain amount of familiarity with D3.js and React, but if you already know how to attach event listeners and toggle class names based on state, then this article should give you all you need to get cracing with Voronoi","diagrams.By using Voronoi diagrams, we can drastically improve the interactivity of SVG charts, making them more user-friendly and accurate.","The best part is that it's all possible with just a few lines of additional code when you're already using D3.js.","So the next time you find yourself grappling with hover issues on your scatter plot or any other kind of chart, give Voronoi diagrams a try.To summarize:When building SVG graphs don't rely on hover interactions on visually small","elements - they are hard to target with a cursor.You can use invisible elements to extend the "hitbox" of your points, but the quick-and-dirty approach (i.e. just makeing the points bigger) can lead to overlapping hitboxes and","inaccurate hover interactions.Voronoi diagrams are a great way to create hitboxes that are as large as possible without overlapping any other points.If you're using D3.js, the d3-delaunay package is a great way to create Voronoi","diagrams from your data."]},{"title":"TomBot2000: automatically finding related posts using LLMs","url":"/llm-related-posts/","content":["I've been having a lot of fun lately with embeddings.","They are a somewhat-overlooked sideshow to the current GPT/LLM/AI circus, but in my view they are even more interesting than the headline-grabbing chat apps.","And unlike the the chat interfaces, embeddings are a way to make LLMs actually useful without any of the worries about "hallucination" or just-plain-wrong content.A large part of what I've been using embeddings for is",""natural language search" which (at the simplest level) involves defining how similar two bits of text are.","Give the robots two different pieces of content and they then decide how similar the "meanings" of the two pieces are on a scale of 0.0 to 1.0.","This is great for search applications where it neatly sidesteps the need for complex algorithms and complicated matching engines (rendering a lot of my previous search work charmingly obsolete!), but it's also a neat solution to a","problem that I've been halfheartedly grappling with for years: finding "related posts" for blog content.In this articleWhy do I care about related posts?","How have I tried and failed to generate related posts in the past?","Failed attempt #1: the manual approach Failed attempt #2: leaning on CMS features So what are embeddings?","Using embeddings for "natural language search" Can embeddings be used to calculate relations between our content?","Calculating the related posts Gotchas Script optimisation Ship it!","Why do I care about related posts?","When I talk about "related posts", what I'm referring to is the section at the end of an article that points the reader to other, similar content.","The promise being made to any reader that reaches the end of an article or post is: "If you liked reading that, then you'll probably be interested in this too".","Partly I want to make my sites more useful to people who visit them, but mostly I just want to keep readers on my site.","I've been working on blog-style content sites for well over a decade-and-a-half at this point, and I'm keenly aware of how tough a challenge "audience retention" can be.The default option of "next post" and",""previous post" links (I default option I've often used myself) doesn't really provide much value.","Only the most ardent completionists are going to work their way through a site from the very start to the very end.","If you can surface related content instead, that's inevitably going to be more useful.How have I tried and failed to generate related posts in the past?","Failed attempt #1: the manual approachThe "easiest" way to add related content to the bottom of a post is to do it manually.","By which I mean picking a couple of posts that you have personally decided are related to the post you've just written.","Then you type the links out at the end of the post's content yourself.","You, the post's author, do the choosing.","You find the links.","You write the description.","In many ways this is probably the best way to do it, but there are a few issues:You can only choose from content that already exists.","This is because the related content you select is limited to the articles or posts you've already published, leaving no room for future publications to be included.","In theory, you could revisit every post every time you add a new one and update the related links, but this quickly becomes impractical and time-consuming.The links you choose might break.","Once you have hand-written a related post link, that link is locked in place.","If the permalink for the related post changes (for example, if you update the title of the post or change the structure of your site) then you need to remember to come back and update the link (or at the very least setup a 303 redirect).","Again, this is lots of work, and an almost certain recipe for "dead links" once your site reaches any kind of scale.It's a lot of work.","Both issues 1. and 2. involve a lot of manual effort, but there's also the cognitive overhead of keeping all your posts' content in your head.","Plus it takes a lot of time and brainpower to come up with the "relations" in the first place.In short, choosing related posts is a process that cries out for automation.Failed attempt #2: leaning on CMS featuresOne massive time","saving that can be made is to define "related posts" in whatever CMS you're using.","Back (way back!)","when I used WordPress, it was relatively trivial to add a custom field to the edit screen that allowed me to select any existing post from a dropdown.","With that mechanic in place it's possible to quickly select a "related" post (or several) whenever a new post is written.This is still a slightly manual process because you still have to choose the relations, but it does","avoid some of the pitfalls of an entirely manual process:By selecting a post from within the CMS rather than by pasting a URL string, the relation is dynamically linked to the post object (meaning if the title or permalink change, the",""related" link inherits those changes).","Because this approach uses the post object there are also more possibilities at the presentational level, as templates can access any part of the post (meaning the "related posts" section of the page can include post excerpts,","thumbnail images, or any other content associated with the post).","Another potential enhancement is to fully automate the process by letting the CMS automatically select related posts for you.","In WordPress land there are no doubt plenty of plugins that offer this functionality (but as with all WP plugins, proceed with caution.","Here be dragons!).","I've used a couple myself over the years, and they've been fine.","As far as I can tell they use a combination of metadata matching (are these posts in the same category, for example?)","and "traditional" search queries to compare posts (although how they build their queries I do not know - if I was building a plugin like this I'd probably use some combination of post-title and excerpt to get the best balance","of accuracy and performance).","But static sites don't use a CMS...","The biggest caveat to this whole approach (and the reason I haven't used it in years) is that a static site doesn't have a CMS.","And, in case you missed it, I'm a huge advocate of static sites and static site generators.","Sure, some of you might be using a fancy headless CMS or whatever, but for me one of the main appeals of running a static site is that all my content is nothing more than a folder full of markdown files.The one major downside to just using","markdown files is that if you want any custom metadata (such as links to related posts) you have to write it in yourself.","The format is a bit more structured than yolo-ing it in the main content flow as-per the fully manual approach (if you can call YAML structured #shotsFired), but still comes with all the same downsides.So what are embeddings?","Embeddings power my 100%-automated related posts workflow.","The general concept of "embeddings" is an offshoot of the Large Language Model (LLM) technology that makes tools like ChatGPT work.","The basic idea is that you can take a piece of text (a blog post, for example) and turn it into a vector (an array of numbers).","This vector is called an "embedding" and it represents the "meaning" of the text.","It's a weird concept to get your head around at first, but you can learn more about embeddings in detail in Simon Willion's excellent explainer, "Embeddings: What they are and why they matter".","The way I think about it is like this: an embedding is a mathematical representation of the meaning of a chunk of content.","And because the embedding is an array of numbers it can be treated as a set of coordinates.","If the embedding was a simple one with only two numbers, you could plot it on a 2D graph and the the position of that point on the graph would represent the meaning of the text used to create the embedding.","In essence, by using embeddings you're creating a "map" of the meaning of your content.Of course it's more complex than that as the embeddings are actually a lot longer than two numbers, but the principle is the same.","The longer the vector, the more dimensions you have to plot the point in.","The more dimensions you have, the more accurate the representation of the meaning of the text.","I've been using OpenAI's ada-002 model to create my embeddings, and the embeddings it creates are made up of 1536 numbers.","To plot these vectors on a graph you'd need to (somehow) visualise a 1536-dimensional space.","Like I said, it's a tricky concept to get your head around.Using embeddings for "natural language search"It is the "semantic mapping" aspect of embeddings that make them useful.","If you have a set of embeddings created from different strings of text, their "position" in 1536-dimensional space represents their meaning.","If they are close, then the meanings are similar.","If they are far apart, then the meanings are different.The practical application of this "closeness" is that you can measure how "close" multiple embeddings are and compare them against each other.","If one of your strings happens to be a "search query", then tada: you've built a search engine!","Rig up an input field to generate an embedding of whatever question the user asks, and then compare that embedding to the embeddings of all your content.That simple search engine will work just fine with "normal" queries that","you'd type into any old search box, but it would also work really well with natural language queries.And the similarity is computationally "easy" to calculate, too.","If you've already done the work to create the embeddings, then the similarity can be worked out with a function known as cosine similarity.export const cosineSimilarity = (a, b) => {","const dotProduct = a.reduce((acc, cur, i) => acc + cur * b[i], 0);","const magnitudeA = Math.sqrt(a.reduce((acc, cur) => acc + cur ** 2, 0));","const magnitudeB = Math.sqrt(b.reduce((acc, cur) => acc + cur ** 2, 0));","const magnitudeProduct = magnitudeA * magnitudeB;","const similarity = dotProduct / magnitudeProduct;","return similarity;","};","There's a lot of (to my eyes) complicated maths going on in that cosineSimilarity function, but thankfully you don't need to understand exactly how it works to use it effectively.","(Although of course you can just copy/paste that function into ChatGPT and get a pretty decent explanation).","Complex or not, calculating cosine similarity is a lot less work than creating a fully-fledged search algorithm, and the results will be of similar quality.","In fact, I'd be willing to bet that the embedding-based search would win a head-to-head comparison most of the time.There are some caveats to this, as how you've divided up the content before creating your embeddings will effect","the results (did you embed every sentence, or every paragraph, or the whole article as a single embedding?","Those choices will have consequences).","Handy fact: hallucination is not a problem when working with embeddingsAnyone who's used LLMs for any amount of time will have hit upon the biggest obstacle to using them for "proper work": LLMs hallucinate.","Yep, they just make stuff up all the time, giving you "facts" that sound plausible but may or may not actually be true.","This is an issue with all chat-based LLM interactions where the LLM is generating text.The beauty of using LLM embeddings is that at no point is any text being generated.","An LLM generates an embedding of text that we explicitly gave it.","The "AI magic" is in how it turns the meaning of the text into a list of numbers.","We don't need it to generate any text for us, so there's no scope for making stuff up.Can embeddings be used to calculate relations between our content?","In short: yes, yes it can.","That same cosineSimilarity function that we used to compare a post's embedding to an embedded search term can also be used to compare one post to another.","If we do that for all the posts in our blog's archive, then we compare the similarities to find the N most similar posts.I've also added some GPT-powered sugar on top of the standard "related posts" concept by getting","GPT4 to tell us why the two posts are similar.","This lets me flesh out the related-posts section with useful information that will hopefully entice readers into exploring more articles on my site.Calculating the related postsI've written a node script that does this for my blog","posts every time I publish a new article.","The script works like this:It reads all the markdown (.md or .mdx) files in my content directory, and extracts the relevant content (i.e. the title, subtitle, and body-copy of the post).","This post content is then sent to the OpenAI embeddings API to generate a single embedding for that whole post.I then use the completions API to summarise the content of each post.","This will help me in later steps because the API has limits on how much text it can parse at any one time.","Sending over two full blog posts' worth of content will mean I quickly hit the limits and can't generate the "why are these posts similar?","" text.","By creating a shorter summary of each post at this stage, I can then use those summaries in my later prompts.For each post, the script then finds the top two most-similar posts based on the cosine similarity of the embedding vectors.For","each "similar post" the script then compares the posts' summary with the summary of the starting post, and sends these to GPT4 to generate the "why these are similar" text.The final step is to write this",""similar posts" data back into the frontmatter of the original markdown files.Because this script just needs the markdown files of my blog posts, it can live outside the normal build pipeline of my site.","In fact, it could work just as well with any Static Site Generator - if you've got a folder of markdown files, this script would work just fine.GotchasSeeing the script laid out step-by-step like that makes it all look rather simple,","but there were a few obstacles to overcome that turned an hour's POC into several days' worth of development effort.Gotcha #1: API Rate limitsIt turns out that you can't just spam the OpenAI API with hundreds of requests all","at once, so I had to add pauses in my script to account for this.","When in "dev mode" the script will stop any wait for user input ("press y to continue") after every API call, but also has a simpler mode for automatically running through all the requests: it just waits for 6 seconds","after every API call to before continuing.Gotcha #2: API token limitsAs I noted in step #3, the API also has a "context limit" - a.k.a. how much content can it process at once.","This is measured in "tokens" (which roughly works out to a token or two per word in a sentence).","For GPT3.5 the token limit is 4,097 tokens, and for GPT4 the limit is 8,192.","Gotcha #3: GPT4 pricingThe OpenAI API is not free to use.","Thankfully the embeddings requests are really cheap, and in the process of building and debugging my script I generated literally hundreds of embeddings (using the text-embedding-ada-002 model) and never got my usage above even a single","dollar.","GPT3 (using the gpt-3.5-turbo model) is a little more expensive, and GPT-4 is even more so!","While the cost of generating the embeddings was negligible, generating the "why are these posts similar" text cost about two and half dollars per run of the complete script.","This was for my personal blog which has about ~50 posts, and includes generating the summaries and then using those summaries to generate the final text.For my own sanity I've spent a lot of time implementing caching to avoid running","scripts multiple times for the same content, so future runs (i.e. when new posts are added to my site) will cost a lot less (generally somewhere between a few cents and a dollar).","Gotcha #4: Prompt tweakingAgain, the embeddings side of things doesn't require any trickery or shenanigans (it just works!), but generating the GPT summaries and "why are these posts similar" content required a bit of what","the hype-merchants call "prompt engineering".","In my experience, the GPT models can be a bit enthusiastic when generating text, so it took a little bit of careful prompt phrasing to stop them creating over-the-top SEO-fodder (the kind of stuff that marketers and Google seem to love, but","real humans hate).","If you're interested, here's the full prompt I used to generate the "why are these two posts similar" text:You are an automatic recommendation engine for my technical blog.","At the end of each blog post, you recommend other posts that readers may be interested in.","I'll provide the summaries for two blog posts.","Can you describe why someone who has just read the first post might be interested in reading the second post?","Here is the first post:","${postSummary}","Here is the second post:","${similarPostSummary}","Here are some additional instructions to help you write the recommendation:","* Focus on the facts included in the post and the author's opinions.","* Start every summary with "This is similar to what you've just read because"","* Limit responses to single sentence.","* Avoid hyperbole (such as "It's an enlightening read" or similar).","Just describe the similarities of the post and why it might be interesting to someone who has just read the first post.","There a lot of hand-wavey stuff going on there, but after a lot of experimentation that was the prompt that gave me the most reliable results.Gotcha #5: Inconsistent outputAnnoyingly, even with a super-specific prompt the output from GPT","generation is not always consistent.","For instance, it sometimes adds quote marks around the final text, and other times it doesn't.","Not a huge issue, but something that does need to be accounted for at the code level.Gotcha #6: HallucinationEven when mostly relying on hallucination-less embeddings, and even when being super-explicit with my prompts and providing all the","relevant content in the API requests, even then, hallucination is a bit of a problem.","When generating the summaries it can be subtly wrong about the content of the post it's summarising, and when describing why two summaries are similar it occasionally went off-piste and misunderstood the meaning of a post.After a lot","of prompt-tweaking and experimentation I've ended up with a script that I'm happy with for this specific application, but it does highlight the general problem with hallucination.","Now I've spent more time working with these LLMs it's really apparent to me that the hallucinations are the biggest obstacle to doing "useful work" with LLM content generation.Gotcha #7: Post deletion (requires","recalculating similarities)Predictably (although I didn't predict it when writing the first iteration of the script!), when a post is deleted or changed you need to re-generate (or at least check) all the similar posts for all of the","posts.","Easy enough to handle when the content has changed, but deleting a markdown file did result in my script exploding in annoying and hard to debug ways.","Got it sorted in the end, but it was enough of a headache to include on this list!","Gotcha #8: Node memoryThis was the biggest bit of uncharted territory for me.","Storing all that content in Node's memory meant I eventually hit the memory ceiling!","Keeping the post content in memory wasn't an issue, and nor was storing the object that contained the summaries and the similarities.","The killer (from Node's perspective) was keeping all that and the embeddings (i.e. a whole load of arrays each with 1536 floating points) and not being smart about how many times I map and reduce the data and trying to write them to a","file.","Probably a problem I would have avoided entirely if I had a CS degree and knew more about what "Big O" meant, but I got there in the end.Script optimisationIn the end I did quite a lot of "optimisation" work to get the","script to a place where I could rely on it for all the scenarios in my blog workflow (calculating the initial "related posts" for a folder of markdown files, regenerating when new posts are added, handling changes to old post","content, and removing and renaming posts).","CachingTo avoid making repeated calls to the (expensive) API for content I'd already generated, I saved the results to a local cache file.","If an embedding already existed in my cache, then I could skip fetching that API response a second time.","This saved on both execution time and money spent.","To invalidate the cache, I generated a sha256 hash from the post's content (being sure to hash just the posts' title and content and not the frontmatter, as the script saved it's results in the posts' frontmatter and","would therefore instantly invalidate the cache).","Separate cache storage for embeddingsThe embedding arrays, being as long as they are, were a big memory-management concern.","In the end, I cached those separately to the rest of the data and just saved a unique key for each embedding in the main cache.","That meant that only the embeddings that I actually needed would be loaded into memory.Regenerate only when content has actually changedWith the cache/hash pattern in place, the script was able to tell if a post's content had changed","since the last time the script had been run.","This meant I could skip API calls for any post that hadn't changed, but a subtle "gotcha" was that I still needed to regenerate the related posts for a given post if any of the relations had changed.","Thankfully the embedding-based similarity calculation was fast and cheap and could be done without calling any APIs, so I only needed to regenerate the GPT-generated content if the relations had actually changed (I'd re-calculate the","relations for every post, but then if the resulting relations were the same for a post, they could be safely skipped).","Ship it!","So with the script complete the only thing left to do now is use the thing!","The script is triggered by running the command yarn related, and the output looks like this:Screenshot of the terminal output for the script that generates related posts for my blog.That output is a little verbose, but it's useful for","debugging and for seeing what's going on under the hood.","The script also updates the frontmatter of the markdown files with the related posts data, so the final result looks like this:related:","- relativePath: 2021-01-17-adding-rss","permalink: /adding-rss/","date: 2021-01-17","tags:","- articles","categories:","- code","title: RSS in 2021 (yes, it's still a thing)","excerpt: Adding an RSS feed to an Eleventy site is (mostly) easy peasy.","summary:","- This is similar to what you've just read because it also deals with the","topic of automated blog content distribution, specifically diving into","the process of integrating an RSS feed, a channel which - like the","LLM-based automatic posting from the first post - also bypasses","algorithmic interferences.","score: 0.7949880662192309","- relativePath: 2022-02-25-wordle-node-script","permalink: /wordle-node-script/","date: 2022-02-25","tags:","- articles","- featured","categories:","- code","title: Improving my Wordle opening words using simple node scripts","excerpt:","Crafting command-line scripts to calculate the most frequently used","letters in Wordle (and finding an optimal sequence of starting words).","summary:","- This is similar to what you've just read because it delves into another","practical application of scripting and data manipulation, this time","focusing on a word game, which could interest readers who enjoy seeing","real-world implications of coding and automated processes.","score: 0.7938704968029567","That YAML frontmatter gives me enough content to build a nice little template wrapper for the related posts section of my site.","And with all that complete, you should be able to view the real-world results of this script at the bottom of this very page.","The "related posts" functionality is live on this blog, and I'm pretty happy with the results.","I've been using it for a few weeks now and it's been working well.How much does it cost to run?","For the record, running that script with a single new blog post resulted in four calls to the GPT4 completions API endpoint, which cost roughly thirty cents (prices in USD as that's what OpenAI use in their billing).","Is the script available for anyone to use?","Update: Yes, yes it is.","I've opend-sourced the script and published it to NPM.","You can find it here: github.com/tomhazledine/related-posts.","Currently needs an OpenAI API key to work, but I'm working on a way to hook it up to other LLMs.","If you have any questions or suggestions feel free to raise a PR or @ me on Mastodon (I'm @tomhazledine@mastodon.social).","Not yet, but it could be soon.","If you're interested in implementing something similar for your own site, I'm not far off packaging the script into something universal that I can open-source on NPM.","So if that would be useful to you, then @ me on Mastodon (I'm @tomhazledine@mastodon.social).","If more than a couple of people ask me, I'll happilly put in the work to make the script more generic and publish it.","It's 90% done already, but as with all software projects I'm anticipating the final 10% of the work will take 90% of the time!"]},{"title":"Publishing on npm is weird","url":"/npm-publishing/","content":["This is a list of a few things I’ve learned while publishing my JavaScript shenanigans on npm.","Or, more accurately, it's a list of things I’ve banged my head against while trying to publish on npm.","Some of it is logical, some of it is inscrutable.","All of it was necessary.","I’m documenting it here so that next time I go to publish something I can avoid going down a hundred-open-tabs search-engine rabbit hole (again!).","The scenario","I really enjoy writing scripts in Node.","I long ago accepted the fact that JavaScript is the language I’m most fluent in, and now I “lean in” to being able to write JS everywhere.","Other people might use Bash or Python to solve their little problems but when I want to automate something or inspect some data, I turn to Node.js.","And when I want to share my work, the npm registry is the first place I think of.","It’s great to be able to run yarn add my-thing (or npm install my thing if you prefer) and instantly be able to reuse my little scripts.","So what has consistently tripped me up?","And what have I learned after publishing a few things?","Put it in the bin","If a script is to be run by an end user (i.e. they write npm run your-cool-script or yarn your-cool-script), you need to add that script to the bin section of your package.json","{",""bin": {",""your-cool-script": "./path/to/the/script.js"","}","}","Type = module","If you’re writing using ESM syntax (import foo, { bar } from “baz”, etc...), you need to declare your package as a “module” in your package.json","{",""type": "module"","}","Shebang","If you want your script to run correctly on Unix-like systems (which MacOS and Linux both are) then the file should start with a “shebang”: #!","/usr/bin/env node.","This is a comment that specifies which interpreter the system should use for the script, and uses the user’s env command to find the node interpreter in the system’s PATH.","Note that bin is again being used as a place to store executable scripts, this time at a system level.","Shebang get’s its name from a concatenation of “sharp” (i.e. #) and “bang” (i.e. !)","as interpreter comments all start with #!.","If you’re using a tool like esbuild to compile your script from multiple source files, then you’ll need to make use of the “banner” configuration option.","Setting a “banner” is a way to inject arbitrary text at the start of your final built file.","const esbuildConfig = {","// ...","banner: {","js: "#!","/usr/bin/env node"","}","};","esbuild’s format and platform","Speaking of esbuild, there are a couple of other configuration options that you should set: format and platform.","You need to set format to be “esm” because even now at the end of 2023 ESM is still not the default.","CommonJS just will not die, and it will never cease to confound me as to why.","Setting platform to “node” makes more sense, however, as the esbuild is primarily a tool for building things for the web.","By setting this config value we’re telling it not to worry too much about built-in Node packages like path and fs (which it would freak out about if we tried to compile those for the web).","const esbuildConfig = {","// ...","format: "esm",","platform: "node",","banner: {","js: "#!","/usr/bin/env node"","}","};","Let the npm CLI do the work","Versioning is an important part of reliably releasing your code, especially if you’re like me and constantly George Lucas-ing your creations.","SemVer (a.k.a. semantic versioning) is our friend here, so stick to the {major}.{minor}.{patch} format.","I very quickly learned to stop manually tweaking the version value in my package.json.","It’s much more straightforward to run npm version {major|minor|patch}.","This takes care of bumping the package version and cutting a new “tag”.","If these get out of sync, then you’ll have problems when pushing the code to npm, so the CLI is a much more reliable way to handle this process.","Just remember to git push --tags to ensure the new tag gets pushed to your repo.","Note: Watch out if you’re in a sub-folder of a monorepo, npm version won’t do the tagging and commit for you!","Always start at zero","As an aside to versioning, I find it helpful to always start with a “major” version of 0 (so commit #1 is usually version 0.1.0).","When in the early stages of a new project I’m frequently adding “breaking changes” as I find my feet, so if I struck strictly to SemVer and began at v1.0.0 I’d be on version three hundred before my code ever saw the light of day.","The pre-version-one freedom to play fast-and-loose with backwards compatibility is essential, in my opinion.","Go forth and publish","This is not a comprehensive list of All The Things You Need To Know To Publish On npm™️, but everything include here is something I’ve had to learn through trial and error.","I’ll hit more obstacles, I’m sure, and when I do I might remember to add them to this page (maybe).","But at least I’ve got these ones out of my working memory and into long term storage.","And is that not the main purpose of technical blogging after all?"]},{"title":"Mapping LLM embeddings in three dimensions","url":"/mapping-llm-embeddings-in-3d/","content":["Embeddings are long lists of seemingly inscrutable numbers created by an LLM, but if you're prepared to lose a little precision you can force them into 3D space where they're a little easier to comprehend and explain.fig #1:","Headlines from a galaxy far, far away.Plotting the semantic "meaning" of strings of text that have been converted to embeddings by an LLM.","Hover over points to see the original text value and use the "rotation" slider to create a parallax effect.What am I trying to show here?","I talked a lot about "embeddings" in a recent article about finding related posts within my blog archive.","In that article I showed how embeddings could be used to calculate similarities between different chunks of text, and while writing it I had to work out how best to describe what embeddings actually are.The most obvious description I found","for embeddings was the one most closely tied to how I was using them; I was using them to calculate how "close" two piece of text were to each other, so that naturally led to relying on physical space as a useful metaphor.","Sure, the embeddings I use are coordinates in fifteen-hundred-dimensional space, but if you squint your eyes just so you can shrug and say "coordinates are coordinates".","As I said in that article:"by using embeddings you're creating a 'map' of the meaning of your content"Me, a few weeks agoOver the past few months I've found this to be a really helpful way to think about","embeddings, and I've been describing this theoretical "map" whenever I've mansplained embeddings to people (apologies if you've met me lately and been bored stiff by my constant LLM talk).","How can we actually visualise this map?","It's all well and good to describe a theoretical multi-dimensional "map of content", but quite another to come up with a way to visualise it without access to a tesseract or any other n-dimensional hypercube.","As I mentioned in the previous article, I've been using OpenAI's ada-002 model to create my embeddings, and the embeddings it creates are "vectors" made up of 1,536 numbers.","To plot these vectors on a graph you'd need to (somehow) visualise 1,536-dimensional space.","That's an impossibility, so the only practical option is to reduce the number of dimensions we're dealing with.","Thankfully there is a mathematical tool we can put to use to achieve this, provided we're happy to lose some data in the process.Principle component analysis is a "dimensionality reduction" technique that allows us to take a","large-dimensional dataset and convert it into a smaller set of dimensions.","This is a new technique to me (and I copy/pasted most of the code I used) but to the best of my understanding a "principal component" can be seen as a "direction" in multidimensional space.","In a 3D space, the directions are up/down, left/right, forward/backward (the classic x, y, z coordinates).","In 4D, 5D, n-D space, the same idea applies.","Principle Component Analysis (PCA) works out which dimensions have the most variance and ignores the ones with less variance.It's an inherently lossy processes as we're literally ignoring parts of the data, but it's a good","way to get the general "vibe" for a dataset.","I'm happy listening to MP3s and looking at JPEGs online as the compression benefits outweigh the loss in fidelity.","And the embeddings I've been working with represent such a small dataset that I'm happy to lose some detail in exchange for the added comprehensibility.The practical implementation of PCA into my project was actually fairly","trivial thanks to the ml-pca Node.js package, so now I can take my list of embeddings (with each embedding being an array of 1,536 numbers) and transform them into arrays of just three numbers.import { PCA } from "ml-pca";","import { embeddings } from "./my-local-data.js"","const pca = new PCA(embeddings);","// Get the first three principal components","const { data } = pca.predict(embeddings, { nComponents: 3 });","I did originally include an example of a full embedding here, but it was just too much scrolling even if it was funny to see the full before/after comparison.","Suffice it to say, this processes turns a big array into a small array.What does this tell me about my data?","Given that I'd already created an embedding for each of my blog posts, the next natural step was to run a PCA on them and plot them in a 3D scatter graph.","The labels on the graph show just the post titles, but the embeddings were calculated from the entire content of each post.","The primary category each post belongs to is manually set by me when I write each post, and the colours of the point on the graph each correspond to a specific category.fig #1: A "map" of my blog postsEach post's position in","space is determined by the embedding of the content (a.k.a. they're arranged by semantic meaning).","This graph is pretty, but currently just serves as a novelty.","It was a lot of fun to make, as I've done a lot of 2D SVG graphing but never anything that tried to show three dimensions like this (and I'm rather pleased with how it turned out).","I find it fascinating to see "objective" relationships between content I've written over the last ten years, especially the little clusters of topics...","There are a few massive outliers where I spent a month writing about sports prediction.","There's a pleasing "podcasting zone" over to the right and nice little typography corner down the bottom, too.","While not being intrinsically useful yet, it's already got me rethinking how I categorise my archive, and also got me thinking about clusters I'd like to flesh out more (and gaps that might need filling!).","Importantly, what it does do is show visually how my cosine similarity calculations were working out which posts were similar.","A LOT of the dimensions have been removed, but the basic premise is still clearly shown.","**This graph is a much more concrete illustration of "embedding space" than my previous hand-wavey, back-of-a-napkin attempts ever were.","It also has the benefit of being based on real data (even if it is greatly optimised).","When I'm describing embeddings as a "map of content" I can now show them this graph to make my point more clearly than my bumbling words ever could.This is not a true "map" of embedding spaceIn some ways this is not","the post I set out to write.When I was first experimenting with PCA, I assumed that the outputs represented absolute values.","I though that an embedding for a string of text would always result in the same 3D coordinates and PCA would "simplify" all embedding data in the same way.","I naively thought that PCA would allow me to map any embedding into the same 3D space.","I was wrong about this.Because the principal components are determined by the variance in the data, different data sets will have more or less variance in different dimensions.","The PCA "space" that gets created is unique to each set of embeddings that the PCA has been run with.","The "map" of my blog posts will be drawn in a very different space to the map of your blog posts, so we couldn't combine the two sets of 3D coords.","But if we combined our embeddings and ran a PCA for all of them at the same time then we would have a map of our combined posts but the positions wouldn't match the original independent maps.My original plan was to build the graph","framework and have a form input where anyone could type in some text and see it added into the embedding space.","I thought I could make a real map of all embeddings.","PCA just doesn't work like that.","I could (and might yet?!)","create a page where you can input a whole set of text strings and have it calculate the embeddings and PCA'd 3D map in real-time, but what with building the SVG cube graph from scratch I've already burned more time on this post","that I intended.","That'll have to wait for a future project.What next?","Aside from building the interactive version of the graph, what are the next steps for this?","Crucially, how can I turn this curiosity into something genuinely useful?","Fancy data visI love a good piece of data visualisation, and embeddings open up a host of new areas for me to experiment with.","And if I use embeddings and PCA to generate "spatial" graphs and charts, it's a small step to pair that with other factors to tell interesting stories with data.Search index size optimisationFollowing on from my "related","posts" experiments, I've been testing out embeddings as the basis for a genuine site-search engine.","It "kind of" works currently, but the biggest issue is that embeddings are so damn large!","My current set of "whole file" embeddings comes in at about 1.6 MB, which is more than three times the size of my real search index (which, at 419 KB contains all the text content of all my pages).","Yep, the JSON file with a single embedding for each post is larger than the actual content of all those posts.","And to get search results that compete with, say, Fuse.js' matching engine I'd need even more embeddings; ideally one per paragraph of text.","Totally un-useable for the statically-generated client-side-only approach I prefer.Applying principle component analysis to those embeddings might (maybe) change that.","I doubt three dimensions will produce good search results, but perhaps even halving the number of dimensions might still produce useable results while being a more manageable size.","I'll definitely be doing more experimentation around this in the coming weeks.Wherever I go with this next, I'm having a lot of fun and I'm definitely not finished yet."]},{"title":"So long, and thanks for all the Sass","url":"/so-long-and-thanks-for-all-the-sass/","content":["A strange thing just happened.","I was starting a new web project and setting up my build pipeline (and yes, I'm the kind of nerd who enjoys that kind of thing), and I didn't import my boilerplate .scss templates.","This is probably the first time I've not started a project by importing Sass in, oh, about ten years.In this articleWhat is Sass/SCSS and why did I need it in the first place?","Why don't I need to use Sass now?","What did I need to change and refactor?","Comments Variables Breakpoints mixin Imports and partials Nesting Colour functions Maybe don’t just ship it, though...","It’s probably still worth bundling your assets And while you’re at it, why not minify the code as well My esbuild “MVP” config for bundling and minifiying CSS files What have I learnt?","What is Sass/SCSS and why did I need it in the first place?","CSS is how I got into "programming" in the first place (thank you, MySpace!), and I've always loved doing fun and weird and over-engineered things with CSS.","But a big part of what made me confident that CSS was a professional tool I could build a career around was being able to enhance my workflow with "pre-processing".","CSS was always powerful, but it came with quirks and edge-cases and a lot of tiny papercuts.","Using a preprocesser took a lot of that pain away and made the developer experience delightful.","Sass helped me fall in love with frontend development.Simply put, Sass was a tool that let me write the CSS I wanted to write and would then transform it into a "proper" stylesheet that any web browser could understand.","Rather than writing styles.css, I'd name my file styles.scss and get instant access to all sorts of fancy extra features.Side note: SCSS was the Sass syntax I always preferred, because it more closely resembled vanilla CSS in its","formatting - all CSS is valid SCSS, whereas the more uncommon .sass syntax removed braces and semicolons.Using Sass meant I could use variables for colours and other often-repeated values.","I could use logic and perform calculations and write "mixins" that meant I could reuse blocks of functionality without repeating myself too much.","And I could nest selectors and break my code into smaller files that were easier to read and maintain.So much of that sounds trivial nowadays, but back in 2012 when I was first learning about these concepts it felt revolutionary.","I'd already been tinkering with CSS for years at that point, but learning Sass was the first time I ever felt more like a "programmer" than a "designer".","Why don't I need to use Sass now?","Over the years my use of Sass evolved to the point where I'd built up a solid "starter kit" of Sass conventions and tricks that I found useful in almost every project.","Rather than revisiting a load of decisions and choices with every new project, I could get to "meaningful work" much faster.","I'd update the starter kit periodically as I learnt new things or discovered new ways to tackle old problems.","But, as is often the case with foundational things that become part of your muscle memory, I tended to be a bit slow in removing deprecated or redundant parts.","It "just worked" so why invest time in changing it too much?","In the early 2010s it felt like vanilla CSS moved at a glacially slow pace.","After the influx of new features in CSS3, things took a long time to trickle out to all browsers and supporting "legacy" browsers was a big deal.","But thankfully things are different now.","The switch to "evergreen" browsers (ones that automatically update without their users having to do anything) was a massive step forward, as were the introduction of features like custom properties (a.k.a. variables) and grid","systems and imports.","CSS is great now, and we can pretty much use all the fun stuff everywhere!","I could probably have dropped my Sass starter-kit a few years ago, and a lot of my work projects have been Sass-free for even longer.","But this week was the first time I started a new side project and actively avoided adding Sass.","Rather than blindly importing the starter-kit just as I always had done before, I instead refactored the whole thing to use vanilla CSS.What did I need to change and refactor?","The Sass-to-CSS refactor was delightfully straightforward, but it wasn't just a case of renaming the file extensions.CommentsThe first big find-and-replace job was to tackle code comments.","In Sass you can use a double forward slash to mark any following text on the line as a comment just like in JavaScript: // text.","But in vanilla CSS you can only use /* text */.","Crucially, the vanilla version requires the closing */ at the end of the comment.","Honestly this might be the biggest thing I miss from Sass, and I was surprised by how much the closing-tag requirement annoyed me.","But on the plus side it's encouraging me to be more diligent about removing lines I'd commented-out "temporarily" (and which had inevitably remained in my code for years).","VariablesIn Sass a variable is defined with the $name: value; syntax. 10 years ago this was an absolute game changer, but vanilla CSS has had "custom properties" for years now.","Because custom properties are part of the vanilla spec, they can be altered and scoped at run-time, which actually makes them more powerful than Sass variables.There is a case to be made that complex calculations using variables are better","done in Sass because all the work happens at build-time (whereas custom properties are worked out by browsers on the fly when the code is run), but in the vast majority of use cases (and, in fact, in all of mine) CSS custom properties are a","better choice.Aside from having to add a "scope" to the variable declarations (Sass vars can be defined anywhere and everywhere within a SCSS file, whereas CSS custom properties need to be defined within a selector) this was a","simple refactor.","In fact, this is a change I made a long time ago (back when "dark mode" became a common requirement).","Since then, all my colour and type vars have been custom properties, and this recent refactor of my starter kit caught a few extra spacing variables that I had forgotten to convert.$colour-red: #ff4136;",".foo {","color: $colour-red;","}",":root {","--colour-red: #ff4136;","}",".foo {","color: var(--colour-red);","}","Breakpoints mixinEarly in my Sass days I adopted a breakpoint "mixin" strategy for standardising my responsive breakpoints.","A mixin is a reusable block of code that can be included (or "mixed in") throughout a stylesheet, so rather than YOLO'ing a new media query every time I needed one I could use my bp($size) mixin to target specific","breakpoints.","// My lengthly (and now completely redundant) breakpoint mixin:","@mixin bp($point) {","// "min-width" breakpoints (screens getting bigger)","@if $point == u1 {","@media (min-width: 960px) {","@content;","}","} @else if $point == u2 {","@media (min-width: 1200px) {","@content;","}","} @else if $point == u3 {","// etc...","}","// "max-width" breakpoints (screens getting smaller)","@else if $point == d1 {","@media (max-width: 1000px) {","@content;","}","} @else if $point == d2 {","// etc...","}","}","// The mixin in use:",".related-llm-icon {","width: 80%;","@include bp(u2) {","width: 50%;","}","}","At first glance, this seems like an ideal use for custom properties.","At the expence of a few extra characters (i.e. writing a full @media (min-width... statement) I could have pretty much the same functionallity as with the original mixin.","(Beware - this is a trap!)","/* Breakpoint variables: */","/* Note: THIS DOES NOT WORK!","*/",":root {","--breakpoint-1: 960px;","--breakpoint-2: 1200px","}","/* Using a breakpoint: */","/* Note: THIS DOES NOT WORK!","*/",".responsive-thing {","width: 80%;","@media (min-width: var(--breakpoint-2)) {","width: 50%;","}","}","⚠️ I was totally wrong about this!","You cannot use custom properties in media queries.","I spent way longer than I'd care to admit debugging why my media queries weren't being applied.At the time of writing, I haven't cracked this issue.","I'd love to be able to set my breakpoint values in a single place without having to resort to any pre- or post-processing, but for now I'm back to copy/pasting good ol' Magic Numbers..responsive-thing {","width: 80%;","@media (min-width: 1200px) {","width: 50%;","}","}","Imports and partialsI was expecting some difficulty when restructuring my Sass partials into vanilla CSS files, but it was actually the easiest part of the whole process.","Vanilla CSS's @import syntax is identical to that of Sass, so all I needed to do was add the .css file extension to each import path.Importing lots of partials into a single production file was a "big win" when I switched","from vanilla to Sass, but looking back at the caniuse compatibility table for @import makes me wonder why I thought I needed Sass for this.","I don't regret my choice, as I think build-time bundling is still a good idea (see my later section about bundling) and back in 2012 my Sass pipeline was the only way I knew how to do that kind of thing.","Thankfully now (as we'll see in a few paragraphs) there are much simpler ways to bundle and minify your CSS.NestingThe ability to nest selectors in Sass was a much bigger deal for me when I started than it is now.","Back then, the idea of nesting selectors aligned with my mental model for structuring my style rules.","Over time, that mental model has shifted; first to limiting nesting to 2 or 3 levels deep, and then with my adoption of BEM it pretty much disappeared completely.If you'd asked me last week, I'd have said I hardly ever nest","selectors at all anymore.","That said, it was interesting to see during this latest refactor exactly how much nesting I still had in my starter kit.","Not loads, but enough that I had to spend a bit of time copy/pasting root classes onto the start of selectors.","None of this substantially changed the structure of my stylesheets, but maybe a little bit of nesting does still fit with how I think about CSS.","With that in mind, I might be open to a future where I set up the tooling to reintroduce nesting into my workflow.Colour functionsSass has some build-in functions for manipulating colours, and I used them a lot.","The most common ones I used were darken() and lighten, which would take a colour (hex-code, string name, whatever you were using would work) and make it darker or lighter by the specified amount. darken($colour, 20%) was my go-to way to set","the colours for hover states and dropshadows and meant I didn't have to manually calculate the new colour values.I do miss the interoperability of colour formats in Sass (being able to mix and match #ff4136 and red and rgb(255, 65, 54)","within the same context was great), but I've gradually retrained myself to simply put in a bit more work when defining my colour variables.","In places where I would have used darken() or lighten() I now use hsl(), where the "lightness" value of the hue/saturation/lightness format behaves just the same as using a percentage value in darken or lighten.","And with tools like Copilot on hand to do the conversion from hex to hsl for me, it's not even that much extra work.Maybe don’t just ship it, though...","Despite the fact that you can responsibly ship un-processed CSS files with your project, for any large (normal?)","sized project you’d be leaving money on the table if you didn’t put some effort into optimisation.It’s probably still worth bundling your assetsIf you’re not bundling your code, every @import statement means the browser has to make another","request.","Bundling (a.k.a. smushing all your code together into one Big Chungus of a file) means the browser needs to make fewer requests to your server in order to render your page, so in most cases “fewer requests” means “faster page load”.","Obviously part of the joy of switching to a mostly-vanilla workflow is that you don’t have to faff about with build tools anymore, so adding a bundling step is adding complexity to your project.","But the options are so much simpler, quicker, and easier to setup than they were ten years ago.I recommend esbuild for your bundling/post-processing tasks, but I strongly believe that if you’re already using another tool (for your","JavaScript, or images, or whatever) then you should add your stylesheet processing into that tool rather than spend ages refactoring.Vanilla CSS optimisation and processing is table-stakes for tools like webpack and Parcel and Vite and","Snowpack and all the rest...","Whichever you choose, you’ll be fine.And while you’re at it, why not minify the code as wellIf you’ve gone to the effort of bundling, it’s generally a trivial extra step to add minification to your build as well.","Minification reduces the file size of your CSS by eliminating unnecessary characters (whitespace, line breaks, comments, etc.) without altering the functionality.Some unminified CSS:/* Main container style */",".container {","width: 80%;","margin: 0 auto;","padding: 20px;","}","/* Header style */",".header {","background-color: #f3f3f3;","padding: 10px;","font-size: 24px;","border: 1px solid #d1d1d1;","}","The same code, after being minified *(note that as far as the computers are concerned, they’re the same picture):.container{width:80%;margin:0 auto;padding:20px}.header{background-color:#f3f3f3;padding:10px;font-size:24px;border:1px solid","#d1d1d1}","My esbuild “MVP” config for bundling and minifiying CSS filesMy setup uses esbuild, and compared to my previous pipelines is hilariously simple (and replaces ~140 lines of webpack config or ~200 of gulpfile.js if you’re feeling really","nostalgic).// build.js","import { build } from "esbuild";","build({","entryPoints: ['app.css'],","outfile: 'out.css',","bundle: true,","minify: true,","});","Okay, okay... my real setup is a bit more involved than that.","It has flags to handle “development” and “production” builds differently, and will watch files for changes and create source maps and smartly rename multiple source files and a whole load of other useful things.But the point is: that small","snippet is all you need.","Install esbuild and add that build.js file and you’re pretty much good-to-go.","Rename the entryPoints and outfile paths to match your folder structure and you can create production-ready CSS by running node build.js.It’s not as straightforward as shipping vanilla, un-processed CSS files, but on balance probably still","worth the effort (and it’s a hell of a lot less effort than it used to be!).","What have I learnt?","My biggest takeaway from this exercise is that I should have done this much sooner!","Sass is one of those tools that I'd been using for so long that it had become second nature, and I'd stopped even thinking about it.","Let alone thinking about if I still needed it.","It's so easy to sleep on these "base layer" parts of my tooling, even though it's something I'd interact with almost every day.","Now I need to evaluate the rest of my setup and see if there are any other parts in need of a good spring cleaning.But either way, Sass was an important part of my developer journey and I'll always have fond memories of us ing it.","So long, Sass, and thanks for all the fish."]},{"title":"RSS is Awesome","url":"/rss-is-awesome/","content":["RSS is “having a moment” right now, and I’m 100% here for it.","Well, maybe folks aren’t explicitly excited about RSS specifically, but an independent and open Web (a Weird Web, even?)","is being talked about a lot.","Burned by recent trends in [looks around] everything, we’ve realised that having some degree of control over the way our content is distributed is actually quite useful.We almost lost itWhen I wrote a post about RSS in 2021 it felt so","off-trend that I added a “yes, it’s still a thing” caveat right in the post’s title.","And honestly, I wasn’t super confident that it was still “a thing”.","So it’s massive relief to see that something I care a lot about is growing in relevance.In short, blogs are back!","And with the demise of algorithmic feeds (or at least their massively reduced reach) we’re all looking for reliable way to find all this content.","And it turns out RSS was pretty much the best tool for the job all along.Skin in the gameA principle I’ve been trying to live by lately is that if I think something’s important I should be demonstrably involved.","In some form or other I should have some skin in the game.","To paraphrase a saying tied to musicians’ muscle memory: it’s not just enough to know something “in the head” - if I care about it I should know it “in the hand” too.I think podcasting is simply the best medium, so I’ve got a podcastI think","YouTube is an amazing outlet for the world’s creativity and knowledge, so I’ve got a YouTube channel to share some things I’m super-nerdy aboutI’m excited about the potential of Web Components (maybe we don’t all need React after all?!), so","I’ve released one of my open-source tools as a Web ComponentMy projects are not setting the world alight, but I’m involved.","If podcasting or YouTube come up in conversation, I can talk from experience.","I can legitimately say “I was there”.","So how does that apply to RSS?","Well, for starters I’ve been blogging in some form or other since 2008 so I’ve definitely put my fair share of RSS feeds out into the world.","But that feels a bit passive.","If I truly want to have some skin in the RSS game, I need to be contributing.","I need to give something back, something that helps other folks (not just burdening them with something more of mine to read).","So I’ve built an RSS reader.Err, don’t we have those already?","True, there are loads of RSS reader apps out there.","And some of them are really good, too.","But none are exactly what I’m looking for.","And what’s the use of being a developer if you can’t build things you like?","Building a thing is the best way to learn a thing.In classic “how hard can it be?”","tradition, I learned a lot while building my app.What we colloquially think of as RSS is actually a quite fragmented ecosystem with many flavours (RSS/Atom/JSON/etc.) that all have subtly different specifications.","So writing a feed parser that can handle any feed a user might throw at it needs to account for the fact that YOU CANNOT ASSUME ANY “REQUIRED” FIELDS WILL BE PRESENT.A properly useful feed reader consumes A LOT of data.","I naively assumed that because all I’d be storing would be text content, the payloads would be small.","But even with a limited set of test feeds, my local state soon hit tens of megabytes.Everyone has a different set of priorities for what makes a “good” feed reader experience.","Some people want notifications and statuses up the wazoo, some folks want the most minimal interface possible.And because RSS has been around for so long, there are a lot of “nice to have” features in pretty much every reader:Auto-discovery","of feed URLs.","Being able to submit a plain ol’ URL for a site and have the reader-app find the feed URLs is table-stakes at this point.Once you open the door to auto-discovery of feed URLs, you need your UI to be able to handle sites that have multiple","different feeds.Resolving relative URLs within feed content (for links, images, etc.).","Despite the best practice of making all URLs in feeds absolute, comparatively few feeds actually implement this (so dead links and missing images are super common when browsing a naively-parsed feed).","Universal import and export via OPML file.","OPML is a markup format that allows groups of feed URLs (or podcasts) to be exported and imported into pretty much any mainstream app that deals with RSS.Prioritising the features I care most aboutSo building my RSS app was a useful","exercise solely from a learning perspective, but I was also able to throw in some features purely because they’re what I’m interested in 😆An (almost) serverless app.No accounts.","I (currently) have very little interest in creating a CRUD app or in dealing with all the user-management and server-side shenanigans that come with such an app, so from the get-go my USP for this project was “no accounts”.","Users can just hit the homepage and get started using the reader functionality.Local storage.","So if I’m not running a server and database, where does all the feed content go?","The add needs some persistence of data, so naturally the browser’s “local storage” was the first thing I tried.","Plain old localStorage turned out to be too small for all the data a feed reader needs to store, so I ended up using localStorage's beefier buddy, IndexedDB (there's an entire database platform built right into the browser.","Who knew?!).","Static hosting.","Having no backend means that deploying my reader app is miraculously simple, especially with static hosting platforms like Netlify.Edge functions.","When I said the app was static and serverless, I lied.","Thanks to (perfectly sensible and secure) CORS restrictions, client-side JavaScript can't just go around downloading content from arbitrary URLs willy nilly.","And reading content from arbitrary URLs is pretty much the one key feature of an RSS reader.","To get around this, the "read the content of the URL" part of my reader app lives on a Netlify "edge function".","This is a simple Node.js function that accepts a feed URL, reads it, and returns the data to the client-side JS.The reading experience I want.Another big win from a selfish standpoint is that by building my own feed reader app I get to","craft whatever esoteric and bespoke reading experience I want.I've written before about how little styling it takes to make vanilla HTML an excellent way to read content, but I do like to put my own spin on things.","Alongside basic typographic best-practices (like sensible line-lengths, line-heights, and font sizes) I appreciate a well-crafted <hr> element and a confidently opinionated <blockquote>.","And then there's the "reader app" specifics around feed management.","Can the feeds be organised in groups?","Can you drag-and-drop from one group to another?","Can you fine tune your "unread" and "new" notifications?","Add to that my (perhaps unusual) preference for monospaced typefaces and loud accent colours in both light and dark themes, my ideal reading experience isn't something I can realistically expect anyone else to create for me.You can use","it tooSo now I’ve sung my reader app’s praises, where can you go to actually use it?","Well, my final flourish was to find a domain name I thought summed up how I felt about the whole experience:rssisawesome.com"]},{"title":"Known Pleasures: SVG line art","url":"/known-pleasures-svg-line-art/","content":["The cover art of Joy Division's debut album, Unknown Pleasures, is about as good as it gets.","It's instantly recognisable and a perfect example of how a simple, abstract image can become iconic.","And because it's both simple and data-driven, it feels like a great candidate for recreating with code.Joy Division - Unknown Pleasures ( FACT 10, 1979)After all, it's just a series of lines, right?","And I've already written a lot about how to draw lines with SVG.","How hard could it be?","Step 1: drawing a lineIn my post on line graphs with React, SVG, and D3 I covered how to use D3.js to convert "real world" data into a format compatible with SVG's <path> element.","To recap, the process is as follows:Find the bounds of your "range" and "domain"Create a "line generator" functionBuild the SVG markupFind the bounds of your "range" and "domain"For both x","and y axes, define the domain of your data (i.e. the minimum and maximum values) and the range of your graph (i.e. the width and height of the SVG canvas).","The domain and range are the two things you need to generate a D3 "scale" function.This example code assumes we're creating an SVG canvas that is 100px wide and 100px tall and dataset that is an array of ten random data","points ranging from 0 to 10.import { scaleLinear } from "d3";","const xScale = scaleLinear()",".domain([0, 10])",".range([0, 100]);","const yScale = scaleLinear()",".domain([0, 10])",".range([100, 0]);","Don't forget to flip the Y axis!","SVG's coordinate system has the origin at the top-left corner, whereas most data visualisations have the origin at the bottom-left corner.","If you don't flip the Y axis ([100, 0] rather than [0,100]), your graph will be upside-down.Create a "line generator" functionArmed with your xScale and yScale functions, you can now create a line-generator function.","This is a function that will convert an array of data into a string of M and L commands.","These strings are how an SVG defines a line.import { line } from "d3";","const lineGenerator = line()","// Here we're using the index of the data point as the x-coordinate",".x((_, d) => xScale(d))",".y(d => yScale(d));","Build the SVG markupWe can then use this getPathData function to generate the d attribute of a <path> element in our SVG:const data = generateRandomBellCurveData(40);","// [","//     4.761029986699451,","//     0,","//     5.327995603655005,","//     7.1660931217039066,","//     etc...","// ]","const path = lineGenerator(data);","console.log(path);","// "M0,1L12.5,50L25,20L37.5,30L50,35L62.5,70L75,100L87.5,90L100,30"","Generating "random" data I've written a small function to generate "random-ish" data that roughly follows a "normal distribution" (i.e. a bell curve).","I won't include it here, but you can view my generateRandomBellCurveData() function in this GitHub Gist.<svg width="400" height="400" viewBox="0 0 100 100">","<path d="M0,1L12.5,50L25,20L37.5,30L50,35L62.5,70L75,100L87.5,90L100,30"></path>","</svg>","A simple line graph using an SVG <path> element.Step 2: applying margins to the graphWe might have successfully used some data to draw a line in an SVG, but there's still a long way to go yet.One big difference between our simple","line and the Unknown Pleasures cover art is whitespace.","Peter Saville's graphic has lots, and ours has none.Adding margins and padding to elements with CSS is very straightforward, but in SVG-land things are a bit more complicated.","This is because our path element is defined in terms of absolute coordinates relative to the SVG canvas.","If we want to add margins, we then also need to adjust the position of the path element.","This can be achieved in a few different ways:Translate the path: This is the simplest way to move the path around.","You can use the transform attribute to move the path around.","For example, transform="translate(10, 10)" would move the path 10px to the right and 10px down, allowing us to safely add a 10px margin to the SVG (by making the SVG's width and height 120px).","Use the viewBox attribute: This is a bit more complicated, but allows you to define a "virtual" canvas that is larger than the actual SVG element.","If our path is drawn with x/y coordinates between 0 and 100, we could set the viewBox attribute to "-10 -10 120 120" to add a 10px margin to all sides of the SVG.Adjust the path data: This is the most complicated option, but also","the most flexible.","You could adjust the path data itself to add margins.","For example, if you wanted to add a 10px margin to the top and left of the SVG, you could add M-10,-10 to the start of the path data or (and this is the approach I favour) you can adjust the x and y scales to add margins to the data before","it's converted into a path.(","0, 0)(-100, -100)(200, 200)Our line graph with a 100px margin, showing a 100×100 SVG canvas positioned within a viewBox of "-100 -100 300 300"Step 3: stacking multiple linesThe next step is to "stack" multiple lines on","top of each other.","This is where things start to get a bit more complicated, as we need to generate multiple paths and then layer them on top of each other.In code-terms, this means working with an array of rows, and then mapping over that array to get our","path data.","// Create an array of ten items, each of which is an array of 40 points","const rows = Array.from({ length: 10 }, () => generateRandomBellCurveData(40));","// Run each row through the line generator to get the path data","const paths = rows.map(row => lineGenerator(row));","When rendering our SVG, it's easy enough to map over the paths array to generate multiple <path> elements:Note: for the rest of this article will use JSX for rendering SVG markup.","JSX is the React templating language, but React is not at all required for anything shown in this article.","I just find JSX to be a more readable way to write SVG markup.const pathMarkup = lines.map((line,i) => <path key={i} d={line} />);","But if we do this, we'd end up with all the lines aligned to the same baseline.","Each line would bleed into the others and we'd be left with a mess.All the lines have the same baseline, so they all bleed into one.What we want is to evenly distribute the lines across the canvas, so that they don't overlap.","This could be done with a transform on each <path>, but a more flexible alternative is to let the yScale function do the heavy lifting.The trick is to work out the baseline for each row, and then use that to generate a new yScale","function for each row.","The baseline can be calculated by dividing the height of the canvas by the number of rows and then multiplying that by the row index, which provides the px value for the bottom of each row.","Feeding this into the range of the yScale in (where we had previously hard-coded 100) will then distribute the lines evenly across the canvas.Offsetting the lines' baseline doesn't completely solve the issue, however.","The lines themselves still are still the same height as the whole canvas.","This means each successive line creeps more and more over the top of the graph's boundary.","This is fixed by constraining the height of the lines, which can be done by setting the yRange to be a fixed height rather than the full height of the canvas.","(For the example code, rowHeight is set to be 20px.)const xScale = scaleLinear()",".domain([0, data[0].length - 1])",".range([0, 100]);","const rowGap = 100 / (data.length - 1);","const rowHeight = 20;","const rows = data.map((row, i) => {","const rowBaseline = 100 - i * rowGap;","const yRange = [rowBaseline, rowBaseline - rowHeight];","const yScale = scaleLinear().domain([0, 100]).range(yRange);","const lineGenerator = line()",".x((_, d) => xScale(d))",".y(d => yScale(d));","return { line: lineGenerator(row), rowBaseline };","});","We can evenly spread the lines vertically in our graph, but then they overflow the edge of the canvas.By constraining the vertical "range" of each line, they all stay within the bounds of the canvas.There's one last thing to","be done to complete the effect, and that is to handle the "fill" of the lines.","In the original Unknown Pleasures cover art, the lines are filled in such a way that they obscure each other.","This is achieved by using a separate "area" for each line, rather than a single <path> element.Instead of running the data through a "line generator" function, we can use a "area generator" function as","well.","This is similar to the line generator, but instead of just generating a line, it generates a closed path that fills the area between the line and the baseline.A area generator is created in much the same way as a line generator, but uses","D3's area function in place of the line function.","The area generator allows us to define both a y0 and y1 function, which are used to define the top and bottom of the area.","So the data (d) gets passed into y1 and the baseline (0) gets passed into y0, giving us a filled area between the line and the baseline.const areaGenerator = area()",".x((_, d) => xScale(d))",".y0(() => yScale(0))",".y1(d => yScale(d));","Then when the data is mapped over, we can generate both a line and an area for each row.const rows = data.map(row => {","// ...","return {","line: lineGenerator(row),","area: shapeGenerator(row)","};","});","And then when rendering the SVG, we can use both the line and area strings to render a line and an area for each row.{","rows.reverse().map((row, i) => (","<g key={i}>","<path className="area" d={row.area} />","<path className="line" d={row.line} />","</g>","));","}","Note the .reverse() added to the data array.","This is necessary to ensure the lines are drawn in the correct order.","Unlike CSS, SVG has no concept of z-index, so source-order is king..line {","fill: none;","stroke: black;","stroke-width: 1;","}",".area {","fill: white;","stroke: none;","}","Sometimes the lines overlap each other.Filling the areas under each line hides this.Matching the fill to the background cleans things up.Step 4: using real dataSo far the chart has being using the pseudo-random data that","generateRandomBellCurveData() spits out.","To really recreate the iconic cover art, it would be great to use the actual data that inspired it.Pulsar PSR B1919+21The image is a visualisation of the first pulsar ever discovered, PSR B1919+21.","The data was collected at the Arecibo Observatory in Puerto Rico in 1970, and the image was published in the scientific journal Nature in 1971.","The image shows the radio signal emitted by the pulsar, which is a rapidly rotating neutron star.","The signal is a series of evenly spaced pulses, which is why the image looks like a series of peaks and troughs.How it came to be used in the cover art for Unknown Pleasures is quite the story and well worth looking into.Getting the data","into a JavaScript array was a quick task, but plugging it into the graph component required a bit of extra tweaking.The graph was already calculating the offset for each line based on the number of lines in the data (const rowGap =","layout.height / (data.length - 1);).","So even though our original test data had ten rows and the pulsar data has eighty rows, the line-offsets worked nicely without any tweaking.The same goes for the xScale, which was already by calculated based on the number of points in each","row: .domain([0, data[0].length - 1]).","Another freebie!","The yScale, however, needed a little attention.","That scale was previously hard-coded to account for a minimum data value of 0 and a max of 100.","To allow for any data, this needs to be changes to calculate the min and max values of the data and then use those to set the domain of the yScale.const dataMax = data.reduce(","(acc, row) => Math.max(acc, Math.max(...row)),","0",");","const dataMin = data.reduce(","(acc, row) => Math.min(acc, Math.min(...row)),","Infinity",");","Using pixels in SVGs can be a bit weird.","SVGs don't have an intrinsic unit of measurement, so when you define a width or height in an SVG the default unit is "user units", which are relative to the viewBox of the SVG.","This means we can style an SVG line with CSS and set it to 1px, but what's really being set is 1 user unit, which could be any number of pixels depending on the viewBox and the outer dimensions of the SVG element.","Add a responsive percentage width to the SVG (which I always do) and suddenly the connection between CSS px and rendered pixels in the final SVG becomes even less clear.Owing to the fact there there are so many more rows in the real data","than in our test data, the overall scale of the graphic had to be tweaked too.","The SVG canvas was being defined as 100 which meant that a "1 pixel" width line was being rendered as a 1% width line.","With 80 rows to consider, the final image was just a blobby mess.","Bumping the scale of the viewBox up to 400 fixed this.CP 1919318 MHzFig #1: Extract from Radio Observations of the Pulse Profiles and Dispersion Measures of Twelve Pulsars by Harold D.","Carft, Jr.The final graphic (with captions from the original paper).","Bonus: different dataHaving gone to the effort of making the graph component work with arbitrary data, the next obvious step is to throw other data sets at it and see what happens.My daily step count (broken down by hour) actually works","pretty well in this format.00:0023:59Daily steps, March 2024A lot of the "fun" from these stacked graphs comes from having data that fits a couple of basic requirements:It looks best with lines that start low and end low, with a","peak in the middle.The data should be noisy, but not too noisy.","Each line should ideally overlap some of the others while still maintaining the general trend.The original Pulsar data is a perfect example of these requirements, but that kind of data is surprisingly hard to find.I thought weather data","would be a great fit for this, and the UK's Met Office provides a Climate Data Portal that lets you download average monthly temperatures for any 2km square in the UK.","I grabbed the data for my local area and, well, it doesn't quite look as good as I was hoping.","The data is just too uniform.JanDecAverage monthly temperature, Cornwall UK (1991 - 2020)In retrospect, I guess this is to be expected.","There isn't that much variation in the temperature where I am right now and the temperature a couple of KM in either direction.","To get a better looking chart, I should cherry-pick the temperatures from locations further apart.To close the loop on this little experiment, it seemed only fitting to style the pulsar data to look as close to the original Joy Division","album cover as possible.My best SVG recreation of the cover for Unknown Pleasures Joy Division, 1979SourcesI got a .csv of the Pulsar data from this Gist by Borgar, which I found in this D3 demo by D3-creator Mike Bostock which in turn I","found via this excellent post that also recreates the Unknown Pleasures graphic using code.","For what it's worth, I only found those posts when looking for the data after I'd written my code (although a LOT of my D3 knowledge comes from other Observable demos by Bostock).","I learned a lot about the history of the original graphic from this fantastic article in Scientific American, which also contains some great dynamic visualisations that explain how pulsar data is collected and interpreted.I got the idea to","embark on this project by seeing this TikTok which was made by the author of the Scientific American article referenced above, Jen Christiansen.","Thanks, Jen!"]},{"title":"Stacked Sparklines web component","url":"/stacked-sparklines-web-component/","content":["Web components are finally viable!","Well, I think they've been viable for a long time but I'm only just discovering how fun and useful they can be.","I've just published a new one: a stacked sparklines component.In this articleOverview Demo Gallery Example 1: Using all the options to generate a pulsar chart Example 2: My daily steps for 2023 OverviewMy recent SVG recreation of the","Unknown Pleasures album cover quickly expanded into a resusable JSX component.","That worked well for this site, because all my components are rendered to static HTML at build-time.","But what about those folks who don't want to add the wieght of React to their projects?","This was a perfect excuse to experiment with stripping out all the React from a compoment and turning it into a web component.The <stacked-sparklines> component is available on npm (npmjs.com/package/stacked-sparklines) or you can","download the source directly from the GitHub repository (github.com/tomhazledine/stacked-sparklines), and the total uncompressed size is 30kb (the lion's share of which is the D3 scale and line functions).yarn install","stacked-sparklines","import 'stacked-sparklines';","<stacked-sparklines data-data="DATA_GOES_HERE"></stacked-sparklines>","Demo GalleryClick on an example to view the full-size version and see the code used to generate it.Example 1Example 2Example 1: Using all the options to generate a pulsar chartThis example recreates the original pulsar chart that inspired","the Unknown Pleasures album cover.","It makes use of all three text labels: data-label-left and data-label-right, as well as the data-caption-html attribute (an alternative option to a plain-text data-caption) that allows the inclusion of HTML elements in the caption.Because","the data is so dense (~80 lines stacked together), the data-scale and data-row-height attributes have been manualy set.Full details on the available options are available in the Stacked Sparklines README on GitHub.Note: The web component","expects the data to be passed as a string.","This is because web components can only accept strings as attributes.","To pass an array of data, you'll need to stringify it first and inject it using whatever templating language you're using.const DATA_STRING = JSON.stringify(pulsarData);","<stacked-sparklines","class="kp-pulsar-wc"","data-data="{{DATA_STRING}}"","data-row-height="0.2"","data-size="600"","data-baseline="0"","data-scale="4"","data-margin="0.4"","data-label-left="CP 1919"","data-label-right="318 MHz"","data-caption-html="<strong>Fig #1:</strong> Extract from <em>Radio Observations of the Pulse Profiles and Dispersion Measures of Twelve Pulsars</em> by Harold D.","Carft, Jr."","></stacked-sparklines>","Example 2: My daily steps for 2023Each line represents a day, with steps measured at hourly intervals.The code for this example uses JSX to show how the component can be used in a React-like environment.","The key attribute is React-specific, and the self-closing tag is JSX-specific.","The component itself is a standard web component and can be used in any environment that supports them.import stepsData from "./data/2023-steps.js";","const monthNames = [",""January",",""February",",""March",",""April",",""May",",""June",",""July",",""August",",""September",",""October",",""November",",""December"","];","export default () => (","<div className="yearly-steps">","{stepsData.map((month, i) => (","<stacked-sparklines","key={`month_${i}`}","class={`yearly-steps__${monthNames[i].toLowerCase()}`}","data-data={JSON.stringify(month)}","data-caption={monthNames[i]}","data-background="#00b7c6"","data-foreground="#fdfdfa"","data-size="300"","data-margin="0.2"","data-scale="1.5"","data-row-height="0.1"","data-baseline="0"","/>","))}","</div>",");","This example renders multiple <stacked-sparklines> components into a grid, each representing a month of the year.","The data is passed as a stringified array, and the data-caption attribute is used to label each chart with the month name. data-background and data-foreground are used to set the colours of the chart.","The SVGs rendered by the component are fully styleable with CSS, but these attributes allow for quick and easy customisation.I'll keep adding to this gallery as-and-when I create new examples.","If you have any ideas for cool things to do with this component, let me know over on Mastodon: @thomashazledine@mastodon.social!"]},{"title":"Subsetting my font files reduced their size by more than 60%","url":"/subsetting-my-fonts-reduced-their-size-by-sixty-percent/","content":["Despite often describing myself as a "typography nerd", there's not all that much evidence of my font-nerdery on these pages.","And given that I'm a vocal proponent of making sites small and fast, it's doubly damning that I've never given much thought to the way webfonts are applied to my own site.In this articleShift Happens Universal Principles of","Typography This site's typographic timeline Can variable fonts help me?","What is font subsetting?","How can you easily subset a font file?","Even more optimisation I've recently finished reading two books that have reignited my interest in typography; Shift Happens by Marcin Wichary, and Universal Principles of Typography by Elliot Jay Stocks.1.","Shift HappensMarcin Wichary's epic book about typewriters, Shift Happens, came with a digital copy of Gorton Perfected (view the specimen PDF), a re-jiggering of the classic Gorton typeface used on typewriter key caps and all over","miscellaneous vintage hardware.I used Gorton to typeset a data visualisation that I posted on r/dataisbeautiful, and we all agreed the typeface looked awesome.","Ever the magpie, I started to dream about what this website would look like if I switched to Gorton Perfected.2.","Universal Principles of TypographyReading Elliot Jay Stocks' excellent new book, Universal Principles of Typography, reminded me that variable fonts were a thing that I'd been meaning to try for years but had still not gotten","around to.Variable fonts allow you to change aspects of a font dynamically.","For example, you could have one single font file that has a variable axis for weight.","By changing the weight variable, you change the display weight of the font (i.e. how light or bold the font appears)For several years I've been using multiple font files to handle the different font styles on this site:",""normal", "normal italic", "heavy" (a.k.a. bold), and heavy italic were each being provided via a separate font file.","Time to get to workGorton Perfected has a variable axis for "weight", so switching to Gorton could potentially not only breathe new life into my site's look-and-feel, but could also bring a much-needed performance boost (by","reducing the number of font files a browser would need to download to correctly render my site).","Now it was time to seriously look at changing things up.This site's typographic timeline2013 - 2016: FuturaSomething of an obvious choice, perhaps, but one I thought long and hard about.","I used Typekit (now known as Adobe Fonts) to provide the webfonts, essentially as a "Fonts as a Service" deal.Nov 2016 - 2019: Brandon Grotesque and FF TisaI think of this as my site's "fancy era" (check it out on","the way-back-machine) and thus needed suitably fancy typefaces.","Brandon Grotesque was a huge upgrade on Futura for the headings, and FF Tisa made for a beautiful accompaniment on all the body copy.","In fact, I'm half temped to revisit this pairing as it worked so well.","Both were provided, again, through a Typekit subscription.Jan 2019 - current: CartographIn 2019 I'd been spending way too much time on https://danluu.com/ and stripped my site's styling way back.","As a CSS addict I couldn't bring myself to remove all styles, so what I settled on was what you see today (as of May 2024).","The design has barely changed at all in the intervening years.","This is when I ditched Typekit and bought a standalone license for Cartograph by Connary Fagen.","I still love this font, and use it in as many places as I can.","Having lost the convenience of Typekit's FaaS hosting, this is when I switched to self-hosting the font files myself.@font-face {","font-family: "Cartograph Sans";","src: url("/fonts/CartographSansCF-Medium.eot");","src: url("/fonts/CartographSansCF-Medium.eot?","#iefix")","format("embedded-opentype"),","url("/fonts/CartographSansCF-Medium.woff") format("woff"),","url("/fonts/CartographSansCF-Medium.ttf") format("truetype");","font-weight: normal;","font-style: normal;","font-display: swap;","}","/* Repeat for all font files */",":root {","--type-family--text: "Cartograph Sans", "ff-tisa-web-pro", georgia, serif;","--type-family--code: "Cartograph Mono", Menlo, Monaco, "Andale Mono", "Lucida Console", "Courier New", monospace;","--type-weight--book: 400;","--type-weight--bold: 700;","}","That's a lot of font files!","To be able to cover all the scenarios that I regularly use I need five separate font files, and each comes at a cost.Cartograph .woff sizes:Font variantWeightSizeSansMedium37.8 kBSansMediumItalic42.0 kBSansHeavy38.9 kBSansHeavyItalic42.8","kBMonoMedium35.4 kBCombined download "cost" for all font files: 196.9 kBFor context, my core CSS stylesheet (app.css) is 73 kB and my loaded-on-every-page core JavaScript file is 9.8 kB.","So while those font-file sizes may not seem to egregious in isolation, when viewed as a percentage of my total page weight they are a significant burden.","For a JS-light page (such as this blog post), the fonts are more than 50% of the total payload a reader has to download.Can variable fonts help me?","Spoiler alert: maybe in the future, but not right now.One of the inciting incidents that sent me down this rabbit hole was the idea that Gorton might make a good choice for this site.","I tried it, and wasn't in love with the results for a few reasons.Gorton looks awesome when setting small caps (or just generously-letterspaced all-cap headlines).","For long sections of body copy it's not quite so elegant.Gorton has no italic.","While there's a variable axis for weight, there's no equivalent for skew.","This one is a deal breaker.","My writing style needs italics.So how about a different typeface, one with a variable weight and italics?","This is where the "maybe in the future" comes from.I'm super picky about typefaces, and there aren't many that I'd be happy with for something as important (to me) as my own website.","And I love Cartograph, so whatever replaces it had better be absolutely amazing and an upgrade on all fronts.I've come this far.","Now I need to fix somethingDespite deciding the now is not the time to change to a different typeface, I still had work to do.","When it comes to webfont sizes I've been burying my head in the sand for too long, but now I've actually looked up the cold hard numbers I have to fix it.While variable fonts might be a dead-end (for now), I do still have some","options.","I've head about something called "subsetting" before, but never properly looked into it.","So with that in mind I dug up an old Zach Leatherman post from my bookmarks: 23 minutes of work for better font loading.","In that post Zach walks through the steps required to subset a webfont and improve how the fonts are loaded.What is font subsetting?","Font subsetting is the process of removing all the characters from a font file that you don't need.","If you're only using the Latin alphabet, for example, you can remove all the other characters from the font file and save a lot of space.To subset a font you need two things: the original .ttf font file, and a list of the characters","you want to keep.TrueType Font (TTF) files are the standard font file format, and can be converted to other formats (like .woff and .woff2) for use on the web.","If you've got the .ttf you can create all the other formats you need.How can you easily subset a font file?","Subsetting is one of those things that I've know about for a long time but always put off because it sounded so complicated.","Editing raw font files sounds complex, and how do I even work out which characters I need?","Maybe I could do it for a headings-only font, but for body copy?","That's a lot of characters to work through.Thankfully the aforementioned Zach has a handy little tool called glyphhanger that can help with this.","Give glyphhanger a font file and some html pages, and it will work out which characters you need to keep and then generate the new files for you.","It even has a built-in spider that can follow links on your site and check all the pages for font usage.Installing glyphhangerGlyphhanger can be installed as a node package (npm install glyphhanger), but it also requires a few other","non-node things to be installed too: fonttools, brotli, and zopfli.","The docs say to use pip install fonttools which in my experience never works on modern Macs without a lot of fiddling.","I guess the best option would be pip3 install fonttools for modern Mac OS, but I used Homebrew for all three and had no issues (brew install fonttools etc.).","The I ran this command:glyphhanger http://localhost:1337/archive/","--US_ASCII","--subset=src/public/fonts/raws/*.ttf","--output=src/public/fonts/subsets","--spider","--jsdom","There's a lot going on in that command so let's break it down:http://localhost:1337/archive/ is the URL of the site I want to subset the fonts for.","I'm running a local server on port 1337 to serve the site, which glyphhanger can read without any trouble.","I've linked to the archive page as it contains links to every article I've published (see the --spider option later).--US_ASCII is the character set I want to include in the subset.","My post archive probably contains all the characters I need, but I do have other non-post pages, so including the basic ASCII set covers most of my bases.--subset=src/public/fonts/raws/*.ttf is the path to the raw font files I want to","subset.--output=src/public/fonts/subsets is the path to save the subsetted font files.--spider tells glyphhanger to follow links on the site and check them for font usage.","This means by using the /archive URL glyphhanger can quickly check all the important pages on my site for font usage.--jsdom tells glyphhanger to use JSDOM to render the page and check for font usage.","The default mode uses Puppeteer/headless Chrome, but given that my site is static (and therefore just plain HTML files and not JS-generated content) I can skip this and the command will run much faster.The resultsThe size stats for the","subset woff files look much better:Font variantWeightSizeSansMedium16.7 kBSansMediumItalic18.6 kBSansHeavy17.0 kBSansHeavyItalic18.8 kBMonoMedium15.8 kBAnd what's more, glyphhanger also generates woff2 files, which are even","smaller:Font variantWeightSizeSansMedium13.7 kBSansMediumItalic15.2 kBSansHeavy14.0 kBSansHeavyItalic15.5 kBMonoMedium12.9 kBSo now the combined download "cost" for all font files is 71.3 kB (a saving of 125.6 kB).","My updated @font-face rules now look like this:Note I've also taken this opportunity to ditch the .eot files, as they're not needed for modern browsers.@font-face {","font-family: "Cartograph Sans";","src: url("/fonts/subsets/CartographSansCF-Medium-subset.woff2")","format("woff2"),","url("/fonts/subsets/CartographSansCF-Medium-subset.zopfli.woff")","format("woff"),","url("/fonts/subsets/CartographSansCF-Medium-subset.ttf")","format("truetype");","font-weight: normal;","font-style: normal;","font-display: swap;","}","Even more optimisationBecause it's been so long (years!)","since I last looked at my font loading strategy, there were a couple of other things I could do to improve the situation.Ditch IE!","Out of gotta-support-old-Internet-Explorer habit, I've been serving .eot files for years.","I've ditched IE support in my other assets a long time ago, so I can safely remove the .eot from my asset pipeline and my CSS.","It also has the added aesthetic benefit of cleaning up my @font-face rules, too.@font-face {","/* ...","*/","/* These lines can go!","*/","src: url("/fonts/CartographSansCF-Medium.eot");","src: url("/fonts/CartographSansCF-Medium.eot?","#iefix")format("embedded-opentype");","/* ...","*/","}","Preloading.","Another tip from Zach's posy was to set the font links in my site's <head> to "preload"`.","This makes the requests start earlier and reduces FOIT ("Flash of Invisible Text") and FOUT ("Flash of Unstyled Text").","Previously I'd been relying on the stylesheet to instigate the font download, but by preloading in the core HTML of the site I can get them downloading as soon as possible.<link rel="preload"","href="/fonts/subsets/CartographSansCF-Medium-subset.woff2" as="font" type="font/woff2" crossorigin ></link>","ConclusionI'm really happy with the results of this work.","I've saved a significant amount of bandwidth by subsetting my font files, and I've also taken the opportunity to clean up my font loading strategy.","It's a shame I can't find a suitable variable font right now, but I'm happy with the results of this work.","It was well worth the deep-dive."]},{"title":"What even is a week? (dates are hard)","url":"/what-is-an-iso-week/","content":["Anyone who's ever worked with dates in code will know that they can be, well, unpredictable.","Timezones are the classic bête noire of many a programmer and there are many, many horror stories of bugs caused by daylight savings time transitions.My most recent instalment of "dates are hard" comes courtesy of a simple","question:What even is a week?","Why care about weeks?","It's easy to ignore "weeks" as a concept when working with dates in code.","They rarely feature when we write dates; year-month-day is the international standard, although a lot of non-techie folks still often prefer "day-month-year" (and don't even get me started on the madness that is","month-day-year).","But when was the last time anyone asked you what week it was?","Days of the week are important, sure, but does it really matter if the 1st of June is in the 22nd week of the year or the 23rd?","Business folks (you know, the suits that talk about "roadmaps" and say things like "Q3 is right around the corner") obviously care about week numbers.","But a situation where weeks are important to folks like me (i.e. demand-avoidant developers) is in visualising date information.","Most calendars are laid out in a grid, with weeks as the rows and days as the columns.","And if you're creating a calendar UI with code, knowing which week "row" a given day falls in is pretty helpful.The classic example of this is the GitHub contribution graph: a grid-view of an entire calendar-year where each","column represents a week.Surely I didn't really deface my public GitHub contribution history?","So how does this translate into the world of frontend JavaScript?","Is there a function we can call to get the week number of a given date?","Native JS date methodsIn theory (note: this is foreshadowing) getting numerical values for "day", "month", "year", etc. is straightforward when using the JS Date object.","We can create a date object with the new Date() constructor, and then use the various get methods to extract the relevant information.To get a day number (Sunday = 0, Monday = 1, Saturday = 6) we can use the getDay() method.const date = new","Date('2020-12-25');","// Fri Dec 25 2020 00:00:00 GMT+0000 (Greenwich Mean Time)","const day = date.getDay();","console.log(day); // 5 (a.k.a.","Friday)","For months it's the same: we can use the getMonth method to return a numerical representation of the month.","Like the days, these are zero-indexed so January = 0, February = 1, and December = 11.const date = new Date('2020-12-25');","const month = date.getMonth();","console.log(month); // 11 (a.k.a.","December)","And the same can be done for years:const date = new Date('2020-12-25');","const year = date.getYear();","console.log(year); // 120","Wait, what?!","Why is the year 120?","Silly me, of course getYear() returns the number of years since 1900.","Totally logical.Thankfully (if somewhat confusingly), getYear is deprecated, and we should use getFullYear instead:const date = new Date('2020-12-25');","const year = date.getFullYear();","console.log(year); // 2020 (phew!)","There is no native getWeek()So we can get the day, month, and year of a date with the native JS Date object.","But what about the week?","Well, there is no getWeek() method.This is where date libraries come in.","For a long time, the go-to library for date manipulation in JavaScript was moment.js.","But in an admirable display of humility, the maintainers of moment.js have deprecated the library in favour of smaller libraries like date-fns.There are lots of good reasons to not use a library (they can be heavy, for starters, and often","you can do everything you want with the native Date object anyway) but they do make some things a lot easier.","Like getting the week number of a date.date-fns is my current go-to when I need a date library.","It's small, works well with tree-shaking, and it's date objects are immutable (a massive source of weirdness with both Moment and native date objects).","And date-fns has a getWeek method!","import { getWeek } from 'date-fns';","const date = new Date('2020-12-25');","const week = getWeek(date); // 52","Great!","Now we can start to build our grid; from the first day of the year to the last day of the year.","// Week values for the first and last days of 2024","const first = getWeek(new Date("2024-01-01"));","console.log(first);// 1","const last = getWeek(new Date("2024-12-31"));","console.log(last);// 1","Say what now?","The first day of the year is in week 1, and so is the last day of the year?","That can't be right.","Does the same thing happen for all years?","// Week values for the first and last days of 2023","const first = getWeek(new Date("2023-01-01"));","console.log(first);// 1","const last = getWeek(new Date("2023-12-31"));","console.log(last);// 1","// Week values for the first and last days of 2022","const first = getWeek(new Date("2022-01-01"));","console.log(first);// 1","const last = getWeek(new Date("2022-12-31"));","console.log(last);// 53","Fifty-three?!","Something fishy is going on.","Clearly there are some (literal) edge-cases at play here.","Why are the first and last days of 2023 and 2024 in the same week?","The key thing to get our heads around here is that weeks don't necessarily start on the first day of the year.","We can say a week starts on a Monday, but not every year will start on a Monday. 365 (or 366, of course) does not evenly divide by seven, so a given date will inevitably fall on a different day-of-the-week each year.This means that","sometimes the first week of the year starts before the first day of the year.Thus the last day of 2024 is actually in the first week of 2025, so returning 1 when we ask for the week number makes sense.","We'll just need to account for the overflow by checking the year when getting the week number, and if the year is actually "next year", rather than returning 1 we should add it on to the end of the week count (by adding 52).","Why is the last day of 2022 in week 53?","This is also an artefact caused by weeks always starting on the same day (traditionally Monday or Sunday).","The first week of the year is "officially" (we'll dive into this "officialness" in more detail soon) defined as the week that contains the first Thursday of the year.","This means that the first week of the year can have up to three days from the previous year in it (assuming a week-start of Monday).","Even taking into account that the following year can "steal" a few days from the previous year, the days still occasionally fall so that a year will have 53 numbered weeks.","In general, "week number 53" is accurate and not a bug, so we just make sure our design accounts for it (e.g. by making sure our grid has 53 rows).","Finding an immutable reference pointSo we've got it sorted, right?","Hahaha, no.","We've forgotten to account for timezones and localization!","The Date object in JavaScript is based on the user's local timezone.","If you initialize a day with new Date('2020-12-25') you're actually creating a date object that represents midnight on the 25th of December 2020 in whatever timezone the user is in.","So you could theoretically get different week numbers for the same date, depending on where in the world the user is.","And if you can't rely on the date to be consistent, you can't rely on the week to be consistent either.Local Timezone vs.","UTCSuppose we have a date, December 25, 2020.","Depending on the user's local timezone, this date could correspond to different times, which might push the date into the next day or even the next week.","// User in New York (UTC-5)","const dateNY = new Date('2020-12-25T00:00:00-05:00');","console.log(dateNY.toISOString()); // Outputs: 2020-12-25T05:00:00.000Z","// User in Tokyo (UTC+9)","const dateTokyo = new Date('2020-12-25T00:00:00+09:00');","console.log(dateTokyo.toISOString()); // Outputs: 2020-12-24T15:00:00.000Z","Notice how the same local date corresponds to different UTC times.","This discrepancy can lead to different week numbers if the week is calculated based on local time.Different Locales, Different Week StartsAnd to add another level of complexity, different locales have different definitions of what a",""week" is.","In the UK, weeks start on a Monday and end on a Sunday.","In the US, weeks start on a Sunday and end on a Saturday (And in Iran, Afghanistan, and Somalia weeks start on a Saturday and end on a Friday).import { getWeek } from 'date-fns';","// Assuming weeks start on Sunday","const weekUS = getWeek(new Date('2020-12-25'), { weekStartsOn: 0 });","console.log(weekUS); // Outputs: 52","// Assuming weeks start on Monday","const weekUK = getWeek(new Date('2020-12-25'), { weekStartsOn: 1 });","console.log(weekUK); // Outputs: 52","// Assuming weeks start on Saturday","const weekIR = getWeek(new Date('2020-12-25'), { weekStartsOn: 6 });","console.log(weekIR); // Outputs: 53","Here, the same date can belong to different weeks depending on the locale.","This inconsistency can be problematic when you need a reliable week number for global applications.Standardizing with UTC and ISO WeeksThe first step in any date-related code should be to find an immutable reference point.","For dates and times, this is UTC format.A UTC time is the same no matter where you are in the world, and it's a sensible approach for any date-related code to get your dates into UTC format as soon as possible.","// Unpredictable localized date:","const date = new Date(`2020-12-25`);","// Predictable UTC date:","const date = new Date(Date.UTC(2020, 11, 25));","There are a couple of things to note about Date.UTC:Firstly, months are zero-based which means 11 is December.","Date.UTC(2020, 12, 25) would roll over to the next year (effectively "month thirteen") and give you January 25th, 2021.","It's logical if you take into account the zero-based indexing, but it does make the declarations hard to read at a glance.","In normal usage, 2020, 11, 25 looks a lot like the 25th of November.Secondly, Date.UTC returns a timestamp (the number of milliseconds since the Unix epoch) rather than a Date object.","This means it doesn't have any of the get methods that a Date object has.","This is why the example above wraps Date.UTC in a new Date() constructor. date-fns functions, however, will happily accept a timestamp or a date object as an argument.What is an ISO week?","Getting dates nailed down in UTC format is great, but how can we do the same for weeks?","The week-equivalent of this standardization is the ISO week.","If a (UTC) day falls on ISO week 03, it will always be the third week of the year, no matter where you are in the world.We mentioned earlier that weeks are "officially" defined as starting on a Monday and ending on a Sunday.","But defined by whom?","Our old friend The International Standards Organization has us covered with ISO 8601, the standard covering the worldwide exchange and communication of date and time-related data.","ISO 8601 lays out the concept of an ISO week which has "several mutually equivalent and compatible descriptions" of week 01:the week with the first business day in the starting year (considering that Saturdays, Sundays and 1","January are non-working days),the week with the starting year's first Thursday in it (the formal ISO definition),the week with 4 January in it,the first week with the majority (four or more) of its days in the starting year, andthe","week starting with the Monday in the period 29 December - 4 January.date-fns, of course, has a handy getISOWeek method, and combining that with a UTC timestamp gives us a reliable, predictable, and immutable reference point for weeks.","Hurrah!","import { getISOWeek } from 'date-fns';","const date = Date.UTC(2020,11,25);","const week = getISOWeek(date); // 52","One final spanner in the works: what day does the week start on?","We've already touched on the fact that weeks start on different days in different locales.","The ISO week, by definition, always starts on a Monday.","But what if you want the bug-free predictable experience of using UTC and ISO week numbers, but want your week to start on a Sunday?","I ran into this exact scenario recently when recreating the GitHub contribution graph.","As a reminder, the graph is a 7x52 (or 53 sometimes!)","grid, with each column representing a week and each row representing a day.","Because I've been burned by dates before, I used UTC dates and ISO weeks, but I wanted the graph to start on a Sunday, not a Monday.","This meant I needed to write a function that gave me all the immutable predictability of an ISO week number, but offset by a day.The steps for that function were as follows:Work out the adjustment needed to get the week to start on the","desired day.","I.e. by how many days would I need to offset the date to get the week to start on a Sunday?","0 for Monday (as the ISO week already starts on a Monday) or 1 for Sunday.Apply that adjustment to the date.","Date-fns's handy addDays() function allows us to shift a date object by any given number of days.Get the ISO week number of the adjusted date.","This is the easy part, we just use getISOWeek() on the adjusted date.Check if the ISO week is in the correct year.","If the ISO week is from the preceding or succeeding year, we need to ensure we use sequential week numbers.","If the ISO week is from the preceding year, we return 0.","If the ISO week is from the succeeding year, we return ISOWeek + 52.","If the ISO week is from the correct year, we return the ISO week number.import { addDays, getISOWeek, getISOWeekYear } from "date-fns";","const getAdjustedISOWeek = (date, weekStart = 7, year) => {","const adjustments = {","1: 0, // isoWeek already starts on Monday","2: 6, // Add 6 days to start the week on Tuesday","3: 5, // Add 5 days to start the week on Wednesday","4: 4, // Add 4 days to start the week on Thursday","5: 3, // Add 3 days to start the week on Friday","6: 2, // Add 2 days to start the week on Saturday","7: 1 // Add a day to start the week on Sunday","};","const adjustment = adjustments[weekStart];","const adjustedDay = addDays(date, adjustment);","const ISOWeekYear = getISOWeekYear(adjustedDay);","const ISOWeek = getISOWeek(adjustedDay);","// If the ISO week is from the preceding or succeeding","// year, ensure we use sequential week numbers.","if (ISOWeekYear < year) return 0;","if (ISOWeekYear > year) return ISOWeek + 52;","return ISOWeek;","};","The one downside to this approach is that when you run the function for every day in a year, some years will start at week 0 and others on week 1.","This is because the first day of the year might fall on a Sunday, and the first week of the year might start on the following Monday.","// Assuming `days` is an array of objects that each have a `week` value created by `getAdjustedISOWeek`","const normaliseWeekNumbers = days => {","// If the first day of the year is in week 0, increment all week numbers by 1","if (days[0].week === 0) {","return days.map(day => ({ ...day, week: day.week + 1 }));","}","// If the first day of the year is in week 1, do nothing","return days;","};","Working with dates in JavaScript is always going to be hard.There are so many edge cases and inconsistencies that it's impossible to cover them all.","Libraries like date-fns help a lot, and the upcoming Temporal API will hopefully make things much easier.","In the meantime, my recommendations when working with dates in JavaScript are:If you need to handle and store date information, get your dates into UTC format as early as possible.","If you need to use week numbers, use ISO weeks.Both these strategies help you avoid the pitfalls of timezones and locale-specific.","Or at least, they help you avoid the most common issues.","As sure as the sun will rise tomorrow, you will always be able to find more date-related bugs."]},{"title":"How do you test the quality of search results?","url":"/assessing-search-quality/","content":["The site-search for this blog needs improving, but I have no idea how to measure how "good" or "bad" it is.","If I can't measure it, then how can I tell if any future changes improve it?","With this in mind I've finally done some research and put together a handy toolset for assessing the quality of my site's search functionality.","Strap in, this is a deep dive!","In this articlePart 1: learning the basic concepts Precision, Recall, and the F-score Average Precision Normalized Discounted Cumulative Gain “Ground truth” and subjectivity Summarising search-assessment techniques Part 2: implementing the","assessment in JavaScript The test framework Calculating an F-score in JavaScript Calculating Mean Average Precision in JavaScript Calculating Normalized Discounted Cumulative Gain in JavaScript Part 3: running the tests The results What","does this tell me?","Next steps Part 1: learning the basic conceptsIt turns out that "Information Retrieval" is rich field of academic study, and comes with a wealth of approaches for evaluating results.","I'm not inventing anything new here, just documenting my own mini voyage of discovery.","Precision, Recall, and the F-scoreWhen assessing the quality of search results, the two key factors to consider are precision and recall.Precision is the proportion of retrieved documents that are relevant.Recall is the proportion of","relevant documents that are retrieved.Precision and recall are both equally important, so it's useful to combine these metrics into a single factor: the F-score.","This is a score between 0 and 1, with 1 meaning "the results are perfect" and 0 meaning "the results are terrible".","F1 Score=2×Precision×RecallPrecision+Recall\\text{F1 Score} = 2 \\times \\frac{\\text{Precision} \\times \\text{Recall}}{\\text{Precision} + \\text{Recall}}F1 Score=2×Precision+RecallPrecision×Recall​The F-score (or, more accurately, the F1-score1)","uses the harmonic mean rather than a simple mathematical average.","This gives a balanced measure that penalises extreme values, so both precision and recall need to be high to achieve a high F1-score.Average PrecisionAn F-score shows the trade-off between precision and recall, but gives no consideration to","the ranking of the results.","And in the real world the position of a relevant document in the results list is kind of important (just ask anyone who's website appears on the second page of Google).","An assessment measure that does account for ranking position is Average Precision (AP).","Similar to the F-score, AP measures the quality of search results on a 0–1 scale.","Unlike the F-score, however, AP considers the order in which relevant documents appear, giving more weight to relevant documents that appear earlier.Average Precision is calculated by averaging the precision at each position where a","relevant document is retrieved.AP=1R∑k=1nP(k)⋅rel(k)\\text{AP} = \\frac{1}{R} \\sum_{k=1}^{n} P(k) \\cdot \\text{rel}(k)AP=R1​∑k=1n​P(k)⋅rel(k)where:𝑅 is the total number of relevant documents for the query.𝑛 is the total number of retrieved","documents.𝑃(𝑘) is the precision at position 𝑘.rel(𝑘) is a binary indicator function that is 1 if the document at position 𝑘 is relevant, and 0 otherwise.Normalized Discounted Cumulative GainTo go even deeper, we can also use a concept","called Normalized Discounted Cumulative Gain (NDCG).","Where AP decreases the weighting of results linearly as they appear further down the list, Discounted Cumulative Gain (DCG) does this logarithmically.","This means that DCG gives exponentially higher weights to top-ranked documents.2DCG is computed by summing the relevance-scores of documents in the order they are retrieved, discounted logarithmically by their position in the ranking","(ensuring higher-ranked relevant documents contribute more to the score).","DCGp=∑i=1p2reli−1log⁡2(i+1)\\text{DCG}_p = \\sum_{i=1}^{p} \\frac{2^{\\text{rel}_i} - 1}{\\log_2(i + 1)}DCGp​=∑i=1p​log2​(i+1)2reli​−1​where:𝑝 is the rank position up to which the documents are considered.rel𝑖 is the relevance score of the","document at position 𝑖.log2(𝑖+1) is the logarithm base 2 of (𝑖+1).","As its name suggests, NDCG normalizes the DCG score.","It does this against an ideal DCG score (IDCG), allowing for better comparison of different queries.","IDCG is found with the same formula, but using the ideal order of documents instead of the retrieved documents.","The two values, DCG and IDCG, are then divided to give us NDCG.NDCG=DCGIDCG\\text{NDCG} = \\frac{\\text{DCG}}{\\text{IDCG}}NDCG=IDCGDCG​“Ground truth” and subjectivityF1, AP, and NDCG are all very official sounding science-y terms, but one","aspect I've avoided addressing so far is this: an assessment of search performance will always be subjective.All of those “objective” metrics rely on the same thing: a preset definition of what an “ideal” search looks like.","Put plainly, to measure how many relevant results have been returned for a query I need to know which pages would be relevant in the first place.Common practice is to define a “ground truth dataset” that contains example queries and their","ideal results.","But I have to make this judgment myself.","It's a mini halting problem: to programatically judge whether a search is perfect, I'd have to build a perfect search algorithm.The content I'm searching is my blog, so each "relevant document" is a page on my site,","represented in the dataset by it's URL.3{",""data visualisation": [",""/known-pleasures-svg-line-art/",",""/mapping-llm-embeddings-in-3d/",",""/improving-svg-chart-interactivity-with-voronoi-diagrams/"","// etc...","],",""static site generator": [",""/eleventy-static-site-generator/",",""/llm-related-posts/",",""/client-side-search-static-site/",","// etc...","],","// etc...","};","Summarising search-assessment techniquesFor my n00b perspective, the important takeaway is that I now have a set of three values to compare, each of which is conveniently expressed as a number between zero and one.The F1-score tells me how","well the algorithm balances precision and recall.The Average Precision tells me how well the algorithm retrieves relevant documents.The Normalized Discounted Cumulative Gain tells me how well the algorithm ranks relevant documents by their","importance.Part 2: implementing the assessment in JavaScriptTo put all these ideas into practice I'll need to turn them into JavaScript functions that I can use.The test frameworkMy long-term plan is to test multiple different search","algorithms, so it makes sense to build a framework that can be reused with different implementations.I'll need to ensure each algorithm I test returns results in the same format, which in my case will be an array of document URLs.","For now, I'll just use a single example algorithm: myAwesomeSearch().","This code assumes that I've imported my JSON index as index and my example search algorithm as myAwesomeSearch.4// Example search algorithm","const exampleSearch = {","title: "Algo #1",","fn: myAwesomeSearch(index),","parser: result => result.item.url","};","const searches = [","exampleSearch,","// Add more search algorithms here...","];","F1, AP, and NDCG all need to be run on a per-query basis, and then averaged across all queries to give a final score.For each search algorithm this is a three-stage process where we 1. get the search results for each ground-truth query, 2.","run the per-query assessments for each set of results, and 3. get the average value (a.k.a. the mathematical mean) for each metric.","// Get results for each search algorithm","const finalResults = searches.map(search => {","// Run search for each query in ground-truth","const searchResults = Object.keys(groundTruth).map(query => {","const results = search.fn.search(query);","const parsedResults = results.map(result => search.parser(result));","return {","query,","documents: groundTruth[query],","results: parsedResults","};","});","// Get per-query metrics","const metrics = searchResults.map(item => {","const F1 = getF1(item.results, item.documents);","const AP = getAP(item.results, item.documents);","const NDCG = getNDCG(item.results, item.documents);","return { ...item, F1, AP, NDCG };","});","// Get average results","const meanF1 = metrics.reduce((acc, item) => acc + item["F1"], 0) / metrics.length;","const meanAP = metrics.reduce((acc, item) => acc + item["AP"], 0) / metrics.length;","const meanNDCG = metrics.reduce((acc, item) => acc + item["NDCG"], 0) / metrics.length;","return {","title: search.title,","metrics,","meanF1","meanAP,","meanNDCG,","};","});","// Output results","console.log(finalResults);","The only parts missing now are the actual assessment functions: getF1(), getAP(), and getNDCG().","Calculating an F-score in JavaScriptThe getF1 function takes two arguments:The set of results retrieved by the search algorithm for a query.The ideal list of relevant results for that query from the ground truth dataset.const F1 =","getF1(retrievedDocs, relevantDocs);","The function itself works by comparing the two sets of results to generate precision and recall values which are then fed into the F1 formula:F1 Score=2×Precision×RecallPrecision+Recall\\text{F1 Score} = 2 \\times \\frac{\\text{Precision}","\\times \\text{Recall}}{\\text{Precision} + \\text{Recall}}F1 Score=2×Precision+RecallPrecision×Recall​const f1Score = 2 * (precision * recall) / (precision + recall);","The full getF1 function looks like this:const getF1 = (retrievedDocs, relevantDocs) => {","const relevantSet = new Set(relevantDocs);","const relevantRetrieved = retrievedDocs.filter(doc =>","relevantSet.has(doc)",").length;","// Bail early to avoid division by zero","if (relevantRetrieved === 0) return 0;","const precision = relevantRetrieved / retrievedDocs.length;","const recall = relevantRetrieved / relevantDocs.length;","const f1Score = (2 * (precision * recall)) / (precision + recall);","return f1Score;","};","Calculating Average Precision in JavaScriptgetAP takes the same two arguments as getF1: the actual results and the ideal results.const AP = getAP(retrievedDocs, relevantDocs);","The calculation itself is more complicated.AP=1R∑k=1nP(k)⋅rel(k)\\text{AP} = \\frac{1}{R} \\sum_{k=1}^{n} P(k) \\cdot \\text{rel}(k)AP=R1​∑k=1n​P(k)⋅rel(k)const getAP = (retrievedDocs, relevantDocs) => {","const isRelevant = doc => relevantDocs.includes(doc);","const precisionAtK = (retrievedDocs, k) => {","const relevantRetrieved = retrievedDocs",".slice(0, k)",".filter(isRelevant).length;","return relevantRetrieved / k;","};","const relevantPrecisions = retrievedDocs",".map((doc, k) =>","isRelevant(doc) ?","precisionAtK(retrievedDocs, k + 1) : 0",")",".filter((_, k) => isRelevant(retrievedDocs[k]));","const sumPrecision = relevantPrecisions.reduce(","(sum, precision) => sum + precision,","0",");","const R = relevantDocs.length;","return R > 0 ?","sumPrecision / R : 0;","};","Calculating Normalized Discounted Cumulative Gain in JavaScriptCalculating NDCG requires two functions: getDCG and getNDCG.getDCG compares the results of a search to the “relevant documents” ground","truth:DCGp=∑i=1p2reli−1log⁡2(i+1)\\text{DCG}_p = \\sum_{i=1}^{p} \\frac{2^{\\text{rel}_i} - 1}{\\log_2(i + 1)}DCGp​=∑i=1p​log2​(i+1)2reli​−1​const getDCG = (retrievedDocs, relevantDocs) => {","return results.reduce((acc, result, i) => {","const relevance = relevantDocs.includes(result) ?","1 : 0;","return acc + relevance / Math.log2(i + 2);","}, 0);","};","The getNDCG function uses getDCG to get the DCG values from both the retrieved documents and the ideal results, and then divides them to get the normalized value.NDCG=DCGIDCG\\text{NDCG} = \\frac{\\text{DCG}}{\\text{IDCG}}NDCG=IDCGDCG​export","const getNDCG = (retrievedDocs, relevantDocs) => {","const dcgValue = getDCG(retrievedDocs, relevantDocs);","const idcgValue = getDCG(relevantDocs, relevantDocs);","return dcgValue / idcgValue;","};","Part 3: running the testsTo sum up so far, I've worked out some metrics to test for and built the JS framework to allow me to test any search algorithm I want.","The next step is to run the tests and see how my current search functionality performs.I've constructed my ground truth dataset by looking at the most common search terms that lead folks to my site, plus sprinkling in a few",""obvious" terms that I know should return a single result.","(You can view the full JSON file in this Gist).","I've kept my list of test queries quite short, but for future tests will aim to be more comprehensive.The resultsmean F1mean APmean NDCG0.42840.54390.5597I also logged each query's individual F1, AP, and NDCG values:full","resultsPer-query results (rounded to 4 decimal places to keep the table legible)QueryF1APNDCG"data visualisation"0.71430.80830.9270"static site generator"0.66671.00001.0000"web","component"0.80000.80610.9088"wordle"1.00001.00001.0000"11ty vs hugo"0.00000.00000.0000"eleventy vs jekyll"0.00000.00000.0000"llm embedding"1.00001.00001.0000"llm embedding","visualization"0.00000.00000.0000"bullet journal"0.85711.00001.0000"react d3 line chart"0.00000.00000.0000"picobel"0.80001.00001.0000"add delay to audio online"0.00000.00000.0000"how to","calculate reading speed"0.00000.00000.0000"decibel"0.16001.00001.0000Hide full resultsWhat does this tell me?","I've actually found these results surprising!","Due to the small scale of the content being searched over there are quite a few queries that return 1s across the board.","This part is to be expected, as any specific query with only one or two matches will either be found or not found.","More concerning and interesting are the queries that don't get a full match.Partial matches.","There are a few queries that show a decimal value denoting a partial-match with the ground-truth, and that seems to correlate with queries that have a lot of matched documents.","The inference here is that either the order of the results is less than ideal, or that the search algorithm is returning too many results.Total misses.","More concerning are the total misses; for such a small number of test queries there are a lot of them that return no results at all.","This is a clear sign that the search algorithm is not working as intended.","I'd definitely expect "llm embedding visualization" to return a result, but the "-ize" vs "-ise" spelling difference is enough to throw it off.","I thought my matching algorithm was smart enough to account for this, but clearly I was wrong.Next stepsFrom the point of view of "actionable insights", this is great.","I have very specific things to address, and a set way to test if my improvements have improved things.","I'll also work on expanding the ground-truth dataset to include more queries, and to make sure that the queries are as varied as possible.As for the overall mean results (the mean F1, the MAP, and the mean NDCG), there's not much","I can glean from them at this stage other than "I need to make the numbers go up".","They'll be more useful when it comes to tracking changes to the algorithm, or comparing different algorithms entirely.Thankfully there was some value to conducting this deep-dive, and I'm looking forward to seeing how the search","functionality (and my understanding of the search functionality) improves over time.In an F1-score the "1" denotes that precision and recall are equally weighted.","The more generic Fβ-score allows for variable weighting of precision and recall.","↩NDCG can also be used in scenarios with graded relevance (i.e. documents A and B are both relevant, but B is more relevant).","↩Thankfully my site is relatively small and I've got a pretty good mental map of all the articles.","I imagine that building the "ground truth" for a bigger, more complex set of documents would be a tedious and slow process.","↩The parser function will be different for each search algorithm.","It extracts the URL from the search result object so all searches present their results in the same format.","For this example we're assuming that myAwesomeSearch returns results that each have an item with a url.","↩"]},{"title":"How does cosine similarity work?","url":"/cosine-similarity/","content":["Look up "how to compare vectors" and cosine similarity will be the most common (if not the only) approach you will see.","I've been working with vectors a lot lately in the context of LLM embeddings, and being able to measure how similar any two embeddings are has become an important part of my workflow.","But how does the cosine similarity process actually work?","I've been relying on copy/pasting cosine similarity code without really understanding how it works.","To give myself a deeper understanding, I want to answer the following questions:How does the cosine similarity formula work?","What do the different parts of it mean?","Why is this a useful method for comparing LLM embeddings?","What do we mean by "vectors"?","Before getting too deep, it's worth clarifying what we mean by "vectors".","For my projects I'm using the terms "embedding" and "vector" interchangeably.","To quote a previous post of mine:The general concept of "embeddings" is an offshoot of the Large Language Model (LLM) technology that makes tools like ChatGPT work.","The basic idea is that you can take a piece of text (a blog post, for example) and turn it into a vector (an array of numbers).","This vector is called an "embedding" and it represents the "meaning" of the text.In short, embeddings are vectors and vectors are lists of numbers.","If, like me, you translate all code into JavaScript, we're talking about arrays. const vector = [0.1, 0.2, 0.3, 0.4, 0.5]; is a vector.The vectors created for LLM embeddings are very long arrays of numbers.","An embedding created by OpenAI's ada-002 model contains 1,536 numbers.","Mathematically that can be described as a vector in 1,536-dimensional space.","2D vectors (that is to say, arrays with only 2 numbers) are much less useful in the context of embeddings, but the same principles apply, and we can use them to illustrate how cosine similarity works.Why is cosine similarity useful when","comparing vectors?","Being able to compare how similar two vectors are is a key part of working with embeddings.","Cosine similarity is the recommended way to do this.If we picture a super-simplified 2D space where all our vectors have only two values, the angle between two vectors is the angle between the two lines they represent.","The lines are drawn from the origin (0, 0) to the end of the vector, treating the two vector numbers as x/y co-ordinates.Note: if we had vectors with three values, we'd be working in 3D space.","Four values, 4D space, and so on.","The principles of cosine similarity remain the same no matter how many dimensions you're working in.Simplified examples showing the angle θ between two vectors in 2D space.The θ (theta) value is the angle between the two vectors.","This is the angle required to rotate one vector to align with the other.","The cosine of that angle (cos(θ)) gives us the cosine similarity: a number between -1 and 1.","If the directions of the vectors are identical, the cosine similarity is 1.","If they're orthogonal (at right angles), the cosine similarity is 0.","If they're opposite, the cosine similarity is -1.","Cosine similarity ignores the magnitude of the vectorsThe cosine similarity formula only cares about the angle between the vectors, not their length.","This means that vectors of different lengths can still have a cosine similarity of 1 if they're pointing in the same direction.Digging into the cosine similarity formulaThe mathematical formula itself (as cribbed from Wikipedia) looks","like this:Cosine Similarity=cos⁡(θ)=A⋅B∣∣A∣∣×∣∣B∣∣\\text{Cosine Similarity} = \\cos(\\theta) = \\frac{\\mathbf{A} \\cdot \\mathbf{B}}{||\\mathbf{A}|| \\times ||\\mathbf{B}||}Cosine Similarity=cos(θ)=∣∣A∣∣×∣∣B∣∣A⋅B​where:A⋅B is the dot product of","vectors A and B.","∣∣A∣∣ is the magnitude of vector A.","∣∣B∣∣ is the magnitude of vector B.θ is the angle between the two vectors.That seems straightforward enough, but to translate it into a useable JavaScript function I'll need to answer a few followup questions:What is the "dot","product" of two vectors?",""Dot product" basically means "multiply and add".","The dot product of two vectors is the sum of the products of their corresponding elements.A⋅B=a1b1+a2b2+…+anbn\\mathbf{A} \\cdot \\mathbf{B} = a_1 b_1 + a_2 b_2 + \\ldots + a_n b_nA⋅B=a1​b1​+a2​b2​+…+an​bn​For example, the dot product of the","vectors [1, 2, 3] and [4, 5, 6] is 1 × 4 + 2 × 5 + 3 × 6 = 4 + 10 + 18 = 32.","Given vectors a and b, the dot product can be calculated in JS like this:const dotProduct = a.reduce((acc, cur, i) => acc + cur * b[i], 0);","What is the "magnitude" of a vector?","In simple language, the "magnitude" of a vector is its length.","In mathematical terms, it's the square root of the sum of the squares of its elements.∥v∥=v12+v22+…+vn2\\|\\mathbf{v}\\| = \\sqrt{v_1^2 + v_2^2 + \\ldots + v_n^2}∥v∥=v12​+v22​+…+vn2​​For example, the magnitude of the vector [1, 2, 3] is","√(12 + 22 + 32) = √(1 + 4 + 9) = √14.","Given a vector a, the magnitude can be calculated in JS like this:const magnitude = Math.sqrt(a.reduce((acc, cur) => acc + cur ** 2, 0));","But aren't we ignoring the magnitude of the vectors?","Calculating the magnitudes of the vectors feels counterintuitive given we want to ignore it.","But by including the product of the magnitudes the formula can normalize the dot product, ensuring the similarity measure is independent of the vectors' lengths.The full Cosine Similarity function in JavaScriptNow that we've","unpicked all the parts of the formula, we can put them together to create a JavaScript function that calculates the cosine similarity of any two vectors.cos⁡(θ)=A⋅B∣∣A∣∣×∣∣B∣∣\\cos(\\theta) = \\frac{\\mathbf{A} \\cdot \\mathbf{B}}{||\\mathbf{A}||","\\times ||\\mathbf{B}||}cos(θ)=∣∣A∣∣×∣∣B∣∣A⋅B​export const cosineSimilarity = (a, b) => {","const dotProduct = a.reduce((acc, cur, i) => acc + cur * b[i], 0);","const magnitudeA = Math.sqrt(a.reduce((acc, cur) => acc + cur ** 2, 0));","const magnitudeB = Math.sqrt(b.reduce((acc, cur) => acc + cur ** 2, 0));","const magnitudeProduct = magnitudeA * magnitudeB;","if (magnitudeProduct === 0) return 0; // Prevent division by zero","const similarity = dotProduct / magnitudeProduct;","return similarity;","};","Why doesn't this function use Math.cos()?","Given how much we've been talking about "cosines", it's surprising that our cosineSimilarity function doesn't make use of JavaScript's built-in Math.cos() function.","This is because our function directly computes the cosine of the angle without needing to determine the angle itself.","Using Math.cos() would require first calculating the angle using an inverse trigonometric function like Math.acos(), which is unnecessary and computationally more expensive.So why is cosine similarity useful for comparing LLM embeddings?","There's a reason cosine similarity is the recommended way to compare LLM embeddings.","The important part of an embedding is its direction, not its length.","If two embeddings are pointing in the same direction, then according to the model they represent the same "meaning".","Because cosine similarity measures the similarity of two vectors based on their direction, ignoring their length, it's the perfect tool for comparing embeddings.","It's also computationally cheap, which is a bonus.","And, as we've seen, can be implemented in just a few lines of JavaScript.The power of embeddings comes from their multidimensionality.","The vectors are long, and the relationships between the numbers in the vectors are complex.","But the principles of cosine similarity remain the same, no matter how many dimensions you're working in.","It's a simple and effective way to compare vectors.Are there alternatives to cosine similarity?","Cosine similarity is the most common way to compare vectors, but it's not the only way.","I'll be exploring alternatives in next month's post, so pop your email in the box at the bottom of this page if you want to be notified when it's published.Update: The follow-up post is now live, Alternatives to cosine","similarity.","Click through to learn about the mechanics of Euclidean, Manhattan, and Chebyshev distances, and how they compare to cosine similarity."]},{"title":"Alternatives to cosine similarity","url":"/cosine-similarity-alternatives/","content":["Last month we looked at how cosine similarity works and how we can use it to calculate the "similarity" of two vectors.","But why choose cosine similarity over any other distance function?","Why not use Euclidean distance, or Manhattan, or Chebyshev?","In this article we'll dig in to some alternative methods for comparing vectors, and see how they compare to cosine similarity.","Are any of them faster?","Are any of them more accurate?","In this articleOverview The distance functions Cosine similarity Euclidean distance Manhattan distance Chebyshev distance Other ways to measure similarity Testing the functions Performance results Accuracy results Which is the best choice","for comparing LLM embeddings?","OverviewComparing vectors is a powerful tool when working with LLM embeddings.","It's often the technical underpinning of RAG pipelines (Retrieval Augmented Generation), for instance, where related content is "found" and injected into the context of a message passed to an LLM.","Across the web, cosine similarity is the most frequently recommended way to compare vectors.OpenAI themselves say as much in their documentation:Which distance function should I use?","We recommend cosine similarity.","The choice of distance function typically doesn’t matter much.But there are plenty of other distance functions, so why is cosine similarity the most recommended?","And are there any other functions that might be better suited to comparing LLM embeddings?","What are we comparing?-1-0.500.51-1-0.500.510.5, 0.50.7, -0.3-1-0.500.51-1-0.500.510.2, 0.4-0.7, -0.2-1-0.500.51-1-0.500.510.3, 0.80.4, -0.2Three pairs of simple vectors in 2D space.","We'll use these to illustrate the different distance functions later in this article.The end goal for all these similarity functions is to compare vectors.","A vector is an array of numbers, and we refer to them as "vectors" because they can be conceptualized as points in space.","A "two dimensional" vector (i.e. an array with two numbers) can be thought of as a point on a graph.","A "three dimensional" vector (an array with three numbers) can be thought of as a point in 3D space.","Their magnitude is the distance from the origin (0, 0 for 2D, or 0, 0, 0 for 3D) to the point.In practice, I use similarity functions to compare LLM embeddings which are vectors with ~1,5k dimensions.","For the purposes of this article, we'll stick to visualizing the different functions in two dimensions, but all the concepts covered are applicable to multidimensional vectors.Note: in the context of LLM embeddings, a vector is a list","of numbers that represents the "meaning" of a piece of text.","The more "similar" the vectors, the more similar the meanings of the text.","For more detail, see my post about mapping LLM embeddings in three dimensions.The distance functionsCosine similarity Euclidean distance Manhattan distance Chebyshev distance Cosine similarityIn case you missed the first article,","here's a quick recap of how cosine similarity works:d(A,B)=cos⁡(θ)=A⋅B∣∣A∣∣×∣∣B∣∣\\mathit{d}(A,B) = \\cos(\\theta) = \\frac{\\mathbf{A} \\cdot \\mathbf{B}}{||\\mathbf{A}|| \\times ||\\mathbf{B}||}d(A,B)=cos(θ)=∣∣A∣∣×∣∣B∣∣A⋅B​In this and","following formulae, we'll use d(A,B) to represent the distance d between vectors A and B.In cosine similarity, the final value is the cosine of the angle between two vectors.","The similarity value is calculated by taking the dot product of the two vectors (A⋅B) and dividing it by the product of the magnitudes of the two vectors (∣∣A∣∣×∣∣B∣∣).","You can learn more about what "dot product" means (and how to calculate it) in the previous article.-1-0.500.51-1-0.500.51θ-1-0.500.51-1-0.500.51θ-1-0.500.51-1-0.500.51θCosine similarity uses the angle θ between two vectors in 2D","space.Vectors created by OpenAI embeddings are normalized to a magnitude of 1.","The full cosine similarity function does it's own normalization (by dividing by the magnitudes of the vectors), but the fact that the vectors are already normalized means we can simplify the function to just the dot","product:Normalizedcos⁡(θ)=A⋅B\\text{Normalized} \\cos(\\theta) = A \\cdot BNormalizedcos(θ)=A⋅BAs the OpenAI documentation says:OpenAI embeddings are normalized to length 1, which means that:Cosine similarity can be computed slightly faster","using just a dot productCosine similarity and Euclidean distance will result in the identical rankingsIn JavaScript, calculating the dot product can be a simple one-liner (if you think reduce() functions are simple):const dotProduct = (a,","b) => a.reduce((acc, cur, i) => acc + cur * b[i], 0);","Euclidean distanceBy comparison to cosine similarity, Euclidean distance is a much simpler concept to visualize.","It's a measure of the straight-line distance between two points in space.","It's calculated by taking the square root of the sum of the squared differences between the two vectors.d(A,B)=∑i=1n(Ai−Bi)2\\mathit{d}(A,B) = \\sqrt{\\sum_{i=1}^n (A_i - B_i)^2}d(A,B)=∑i=1n​(Ai​−Bi​)2​Finding Euclidean distance d for","vectors A and B of length n.-1-0.500.51-1-0.500.51-1-0.500.51-1-0.500.51-1-0.500.51-1-0.500.51Euclidean distance is the straight-line distance between two points.const euclideanDistance = (a, b) => Math.sqrt(","a.reduce((acc, curr, i) => {","const diff = curr - b[i];","return acc + diff * diff;","}, 0)",");","Is Euclidean distance better than cosine similarity for use with LLM embeddings?","When the vectors are normalized to a magnitude of 1 (as OpenAI embeddings are), the results of cosine similarity and Euclidean distance will be equivalent.","This means that the rankings of the vectors will be the same, even if the actual values are different.There's not much to separate the functions when viewed from the perspective of time complexity (i.e. using "Big O"","notation), as both functions have O(n) complexity.","But the dot product uses simpler operations (one multiplication and one addition per iteration) than the Euclidean calculation (which performs subtraction, squaring, and addition for each iteration and includes an additional square root","operation at the end).","For measuring the similarity of LLM embeddings, cosine similarity is more often chosen because dot-product-based cosine similarity is slightly faster to calculate.Manhattan distanceUnlike Euclidean distance, which measures the straight line","distance between two points, Manhattan distance measures the sum of the absolute differences between the two vectors.","It gets its name from the fact that it's the distance a taxi would have to travel to get from one point to another in a city grid (where travel is limited to moving along the grid lines, rather than the as-the-crow-flies Euclidean","distance).d(A,B)=∑i=1n∣A1−B1∣\\mathit{d}(A,B) = \\sum_{i=1}^n |A_1 - B_1|d(A,B)=∑i=1n​∣A1​−B1​∣Finding Manhattan distance d for vectors A and B of length n.-1-0.500.51-1-0.500.51-1-0.500.51-1-0.500.51-1-0.500.51-1-0.500.51Manhattan distance","is the sum of the absolute differences between two points.const manhattanDistance = (a, b) => a.reduce((acc, curr, i) => acc + Math.abs(curr - b[i]), 0);","Is Manhattan distance better than cosine similarity for use with LLM embeddings?","The numerical results of Manhattan distance will be different to cosine similarity, but (as with Euclidean distance) there should not be much variation in the rankings when sorting vectors by similarity.","As with the other two functions, the time complexity of Manhattan distance is also O(n), but the operations are simpler than Euclidean distance (just one subtraction and one addition per iteration) but a little more costly than the dot","product calculation (with the "absolute" Math.abs() calculation being the key differentiator).","When compared directly to Euclidean distance, Manhattan distance is often considered to be less affected by the "curse of dimensionality".","This is a phenomenon where the distance between points in high-dimensional space becomes less meaningful as the number of dimensions increases.","Manhattan distance is less affected by this because it doesn't square the differences between the points, which also makes it less sensitive to outliers (the squared differences in Euclidean distance can be heavily influenced by a","single large difference).","Chebyshev distanceChebyshev distance is the maximum of the absolute differences between the two vectors.","While useful in certain scenarios, the fact that it discards the other differences between the vectors makes it less useful for comparing vectors in the context of LLM embeddings.The Chebyshev distance is more useful for things like","warehouse logistics, where it can measure the time an overhead crane takes to move an object (as the crane can move on the x and y axes at the same time but at the same speed along each","axis).d(A,B)=max⁡(∣A1−B1∣,∣A2−B2∣,…,∣An−Bn∣)\\mathit{d}(A,B) = \\max(|A_1 - B_1|, |A_2 - B_2|, \\ldots, |A_n - B_n|)d(A,B)=max(∣A1​−B1​∣,∣A2​−B2​∣,…,∣An​−Bn​∣)Chebyshev","distance-1-0.500.51-1-0.500.51-1-0.500.51-1-0.500.51-1-0.500.51-1-0.500.51Chebyshev distance is the longest of the absolute differences between two points.const chebyshevDistance = (a, b) => 1 - Math.max(...a.map((curr, i) =>","Math.abs(curr - b[i])));","Other ways to measure similarityThere are a lot of distance functions out there.","For this article, I've leaned into the "geometrical" view of vectors - i.e. treating them as points in multidimensional space.","This is because LLM embeddings are built to represent the "meaning" of text, and the crucial part of that is the "directional" relationships between the vectors.","Embeddings are all normalized and have no sense of "magnitude" (i.e. the length of the vector), so the "direction" is the only thing that matters.Other geometric distance functionsMinkowski distance and Canberra distance","are functions that build on the foundation of the Euclidean and Manhattan functions.","Minkowski is a generalization of both that can generate either of those distances or something in between, and Canberra is a weighted version of Manhattan distance.Mahalanobis distance is a different process entirely, which measures the","distance between a single point and a distribution.","This is useful for intra-vector analysis, but not helpful for comparing vectors to one another.Correlation-based comparison functionsThese measures focus on the relationship between variables rather than direct vector comparison.","They can be useful for analyzing trends or patterns in data, but are less useful for comparing vectors.Pearson correlation coefficient measures the linear relationship between two variables.Spearman's rank correlation coefficient","measures the strength and direction of the relationship between two variables.Kendall's tau is a measure of the ordinal association between two measured quantities.Set-based comparison functionsThese are more suitable for comparing","sets or binary vectors, which may not be directly applicable to continuous-valued embeddings.","Sets, unlike vectors, are unordered collections of unique elements.Jaccard similarity measures the similarity between two sets.Hamming distance measures the number of positions at which the corresponding elements are different.Testing the","functionsTesting methodI put these functions to the test with some of my own data, and the results were generally in-line with what I expected.I created an embedding vector for the titles of all my blog posts.","I then created embeddings for each query in my search-test "ground truth dataset" and compared each query to every post title (98 queries each tested against 79 document titles, with each query and title represented by a","1,536-dimensional vector).","I ran this test five times for each distance function, and averaged the results of each run.The execution time for each comparison call (for example cosinesimilarity(vector1, vector2)) was measured with the JS performance.measure()","method.The "accuracy" was measured using a combination of F-score, Average Precision (AP), and Normalized Discounted Cumulative Gain (NDCG).","You can read more about how these measures work in my post, "How do you test the quality of search results?","".","Hide Testing methodPerformance resultsExecution-time distribution for different distance functions (using JavaScript).00.050.10.150.20.250.3051015202530Cosine Similarity median time: 0.12msEuclidean median time: 0.08msManhattan median time:","0.10msChebyshev median time: 0.10msDot Product median time: 0.07msExecution time (ms)Cosine SimilarityDot ProductEuclideanManhattanChebyshevI was expecting a little variation in execution time for each comparison, but what I wasn't","expecting was the bimodal nature of the results.","For each function, there were two distinct groups of execution times.","These peaks existed for all the function types at consistent relative spacings, which suggests that certain vector comparisons took longer than others no matter which distance function was being used.The main takeaway, however, is as","expected.","Calculating the dot product was the fastest calculation, but not my a large margin.","Euclidean distance was pretty close, followed by Manhattan distance and finally Chebyshev distance.Accuracy resultsFunctionF-scoreAPNDCGCombinedCosine similarity0.23660.76020.80670.6012Dot Product0.23660.76020.80670.6012Euclidean","distance0.23660.76020.80670.6012Manhattan distance0.23640.75540.80410.5986Chebyshev distance0.20760.62970.69080.5093 The "combined" score is a simple average (mean) of the F-score, AP, and NDCG scores.As predicted by the theory,","the cosine similarity, dot product, and Euclidean distance functions all produced identical results.","The Manhattan distance function was slightly less accurate, and the Chebyshev distance function was the least accurate.","I was actually surprised to see Chebyshev score as highly as it did, and I suspect that as the size of the dataset increases the Chebyshev scores would drop off considerably.Note: to ensure an apples-to-apples comparison, these tests","measured how well each function ranked the entire list of document titles for each query - essentially just sorting the documents by similarity.","This is why the F-scores in particular are quite low (by including all the documents in the ranking, the precision is diluted).","Which is the best choice for comparing LLM embeddings?","The crucial thing to keep in mind when choosing a distance or similarity function is this: what kind of difference do you care about?","In the case of LLM embeddings, what we ultimately care about is "how similar are the meanings of these bits of text".","For our embedding vectors, the "meaning" is represented by the direction of the vector, so the best comparison function would be one that focuses on the angle between the vectors and ignores other factors.The "full"","cosine similarity function is clearly the most accurate measure of this, given that it focusses exclusively on the directionality of the vectors it compares.","This is why we see cosine similarity emerge as the most frequently recommended choice.","And yet in practice Euclidean distance could potentially produce identical results with less complex (and therefor faster!)","computation.So is Euclidean distance what we should all be using?","Again, we need to look at what we're measuring.","For OpenAI embeddings, the vectors are all normalized.","This means the similarity calculation can be simplified to just the dot product, and the dot product is marginally more efficient to compute than the full Euclidean distance.TL;RD: For normalized vectors like those used by LLM embeddings,","calculating the dot product is the optimal way to determine their similarity."]},{"title":"Home","url":"/","content":["More about me","I tinker with all sorts of things but my current interests are AI-powered data processing, data-visualisation, and modular synthesis.","I like building things","I think RSS is important, so I built an RSS reader app: RSS is Awesome.","I think static sites are important, so I buit an ESM-first static site generator: JS.SSG.","I think being able to style everything with CSS is important, so I made a styleable audio player: Picobel.","I enjoy building funny little interactive charts, and lately I've gone deep into the world of Eurorack modualar synths (I love making music without computers).","But first-and-foremost I make things for the web.","I work as a lead software engineer at J.P.Morgan, and was previously in charge of frontend development at Cronofy.","I co-host the (currently dormant) A Question of Code podcast and I spend most of the day glued to the Fediverse at @thomashazledine@mastodon.social."]},{"title":"Sign up to my newsletter","url":"/newsletter/","content":["I started sending out a semi-regular newsletter as a fun way of nagging my friends about the great podcasts I'd been listening to, and now I use it update people when new content gets released on this site."]},{"title":"Almost done!","url":"/newsletter/pending/","content":[]},{"title":"You've successfully signed up!","url":"/newsletter/success/","content":["Thanks so much for signing up to my newsletter.","This started as a fun way of nagging my friends about the great podcasts I'd been listening to, and now I use it update people when new content gets released on this site.","To have people volutarily sign-up really does mean a lot!","You can find out more about my old podcast-newsletter project on the "Podcasts for Nerds" landing page (where you can also read all the old newsletters), and you can find links to the most popular articles in my general archive","here:Where to startI'd say these posts are the most "representative" of the work I like to do.","They're all deep-dives into something JavaScript-related, and follow a loose "tutorial/explainer" structure.","How does cosine similarity work?","When working with LLM embeddings, it is often important to be able to compare them.","Cosine similarity is the recommended way to do this.Improving SVG chart interactivity with Voronoi diagramsHow I used Delaunay triangulation and Voronoi diagrams to fix hover issues in my SVG charts (with React and D3.js).","Building a delay effect with the Web Audio APIAn introduction to the power of JavaScript's Web Audio APIThanks again!"]},{"title":"Prescience, new beginnings, and modern linguistics","url":"/podcasts-for-nerds/01-prescience-beginnings-linguistics/","content":["Crystal balls and Coronavirus, The Numberphile Podcast (45mins, 2020-04-10)","Numberphile is a YouTube channel for maths-nerds, and the spin-off podcast interviews leading mathematicians.","In this episode TV-mathematician Hannah Fry looks back on the lessons learned (or not learned) from the 2018 BBC show Contagion (which she presented).","That show was a remarkably prescient look at what a modern pandemic would look like.","Find it here","The idea, Backstage (28mins, 2020-04-27)","Myke and Stephen run Relay FM, a podcast network that produces loads of good shows I listen to all the time.","They're starting a new show that's a "behind the scenes" look at creating a podcast (meta, I know!).","The whole show will be behind a members-only paywall, but the first episode is free for all and really interesting.","Find it here","The grammar of singular they, Lingthusiasm (42mins, 2020-04-17)","Everyone loves shows about linguistics, right?","I'll fight people about less vs fewer, oxford commas, etc., but this episode convinced me that I'm probably in the wrong about "singular they" (substituting "him" or "her" with "they" when referring","to non-gendered people).","It sounds odd, but we need to move with the times.","Trigger warning: American voices, and liberal use of "like" - but they're linguistics experts, so they get to decide what's right and wrong.","Find it here"]},{"title":"Tannoys, Tragic Pitches, and Business Traction","url":"/podcasts-for-nerds/02-tannoys-pitches-traction/","content":["The Final Destination, Tales from the Tannoy (32mins, 2020-04-21)","We probably hear the work of voice-over artists every day, but don't even notice them.","Tales from the Tannoy is presented by Elinor Hamilton, the voice of the London Underground.","The show interviews people whose voices we probably know but whose lives we know nothing about.","This episode features Jon Briggs, the original voice of Apple's Siri and the voice inside many sat navs.","He's charming and (obviously) has a fantastic voice, and his story takes a rather unexpected turn.","Find it here.","How not to pitch a billionaire, Startup (25mins, 2014-04-05)","Host Alex Blumberg had a vision for what a podcast company could be, and documented his journey to get there.","The first episode, above all others, is podcasting at its best.","The part where Alex messes up a pitch to infamous VC Chris Sacca is a must-listen.","Even if you're not into tech or business stories, this is an episode I think you should listen to: IMO this marked the point at which podcasting started to really take itself seriously.","Find it here.","A publishing platform built for independent writers, The Business of Content (43mins, 2020-04-13)","I've heard John O'Nolan tell the story of how he founded his publishing platform before.","How he started Ghost is an object lesson in how to capitalise on a flash of viral popularity (step 1: be an expert in a field and come up with an idea for something everybody wants).","In addition to the fantastic origin story, this interview addresses the problems many news organisations are having (still!)","getting readers to pay for content online.","Find it here.","Not a podcast, but...","I didn't even know people out there were adding salt to their coffee, but apparently it is "a thing".","James Hoffmann has done some testing in his video on The Magic of Salt in Coffee.","Things get really entertaining when the Nescafé makes an appearance...","James' YouTube channel is full of coffee-related nerdery, and is definitely worth exploring.","Find it here.","Happy listening!","Tom."]},{"title":"Soundscapes, a grounding, and a year in isolation","url":"/podcasts-for-nerds/03-soundscapes-grounding-isolation/","content":["Thanks for all the great feedback on the format of this newsletter!","Keep it coming; it's really helpful.","I promised myself I'd only make a website for this site if we could reach certain number of readers in the first two weeks.","Thanks to you we've passed that mark, so now this list has a name ("Podcasts for Nerds") and a (placeholder) website: podcastsfornerds.com.","3 great podcast episodes I've listened to this week:","Pew Pew, Twenty Thousand Hertz (38mins, 2020-05-13)","This sound-design podcast is full of gems, and we'll no doubt be returning to it in future editions of Podcasts for Nerds.","The most recent episode looks at the work of legnedary sound designer Ben Burtt, who created the soundscapes for Star Wars.","A real highlight is when the 20kHtz team deconstruct and recreate some sounds that will be familiar to all of us.","Who knew that elphant growls were so terrifying?!","Find it here","Danger: rocks ahead!, Cautionary Tales (36mins, 2019-11-15)","Plan Continuation Bias (also known by pilots as "get-there-itis") can have catastrophic effects.","Economist Tim Harford takes a look at what we can learn from disasters, starting with the sorry tale of the Torrey Canyon.","Find it here","This is the way up, The Habitat (25mins, 2018-04-18)","There's a lot of stuff that needs to be thoroughly tested before we can go to Mars, but the biggest unknown is not what equipment we'll take or what propulsion system we'll use.","The most important thing to test is, well, us!","The Habitat documentsone of NASA's year-long HI-SEAS experiments to see how a team of humans cope with being isolated for a full year.","For science!","Find it here","Not a podcast, but...","If useful maths is something that appeals, then 3Blue1Brown is a YouTube channel for you.","With the help of some elegant animation, the recent "Simulating an epidemic" video digs into the theories behind different lockdown strategies.","Eye-opening and informative, but still remains fact-based and theoretical.","Find it here","Thanks again for being on this list.","Some of you are already recommending episodes to me (thanks, Mike!), and that is strongly encouraged!","What episode do you think should appear in a future edition of Podcasts for Nerds?","Cheers,","Tom."]},{"title":"Cracking, hacking, and phishing","url":"/podcasts-for-nerds/04-cracking-hacking-phishing/","content":[""Hacker" is a word with several meanings.","In the vernacular, it's taken to mean a tech-savvy bad-actor; "a hacker broke into the bank's database", "the NSA mainframe was hacked", etc.","But in the start-up sphere, it just means someone who is creative with technology; "she hacked together her MVP in the garage", "did you see that fantastic hack she made to fix that bug?","".","This week's issue of Podcasts for Nerds contains both types.","3 great podcast episodes I've listened to this week:","One on One with a Hacker, Shoptalk Show (56mins, 2014-03-24)","After having had his website "hacked", Shoptalk host Chris Coyier was able to actually interview the person who hacked the site.","Seriously, folks; be on your guard for Behavioural Engineering.","Find it here.","Hacking classic Nintendo guns and going viral, The Creative Coding Podcast (46mins, 2016-09-07)","Seb Lee-Delisle is a digital marker/artist/inventor who's laser events are the stuff of legend.","In this episode of Creative Coding (RIP), he does a deep-dive on how he reverse-engineered a Nintendo light-gun for his own creative ends.","Find it here","What kind of idiot gets phished?","Reply All (30mins, 2017-05-18)","In a break from the show's usual format, one of Reply All's producers takes over and attempts to "phish" her colleagues (and bosses!).","A eye-opening look at how subtle some attackers can be; even if you're already on the look out!","Find it here","Not a podcast, but...","Jay Forman's eccentric look at the history of London's boroughs is full of great pub-quiz level trivia and also manages to be hilarious (if you like that sort of thing).","The two-part series starts with the 11min YouTube video "Why does London have 32 boroughs?","", and I recommend you watch them both.","Find them here","Thanks again for reading.","I'm investigating different ways to manage this list, which is why this issue looks a little more "fancy" than last week's.","What do you think?","Do you like the new look, or did you prefer the plain-text version?","Cheers,","Tom."]},{"title":"Podcast or not? You decide","url":"/podcasts-for-nerds/05-podcast-or-not/","content":["The boundary between podcasts and radio shows is blurry.","If the BBC release one of their shows online, does it become a podcast?","Objectively, yes; if it has an RSS feed of audio files then it is a podcast.","But it's not really what I think of when I think of "podcasting".","The constraints of Old Media are still there, whereas the main appeal of podcasts (in my view) is the lack of constraints.","They can be as long as the creators feel they need to be (ideally letting the content dictate the length), and they can go deeper into topics than any show aimed at a "general" audience could ever dare to go.","Conversely, they can be less focused and more rambling and more amateurish than any show on the radio.","If someone somewhere finds it interesting or entertaining, then it is a valid podcast.","With that in mind, this week includes two BBC shows that do pass my internal "podcasting duck-test" (If it looks like a podcast and quacks like a podcast, it's probably safe to call it a podcast).","And to even things out, the final recommendation is a show that could only exist as a podcast.","Three great podcast episodes I've listened to this week:","Jon Ronson, Grounded with Louis Theroux (58mins, 2020-05-09)","Louis Theroux can't make documentaries because of the lockdown, so he's been having conversations with his famous friends instead.","In this episode, Louis and fellow documentarian Jon Ronson compare notes on their surprisingly similar careers.","Lighthearted and entertaining, this episode's highlight is the surprisingly nuanced discussion of "de-platforming" controversial figures like Alex Jones and David Icke.","Find it here.","The fourth astronaut, 13 minutes to the moon (48mins, 2019-06-12)","The second season of 13MTTM has just finished.","It told the story of Apollo 13, and is a highly-recommended listen.","If you've not heard about this podcast before, however, you'll be best-off starting with series 1, which is all about Apollo 11's final 13-minute descent to the surface of the moon.","The show focusses on the team behind the missions, and includes remarkable insights into the specifics of every tiny moment in the flight recordings of that historic descent.","For nerds like me, this episode about the onboard computer is particularly interesting.","13MTTM has just about the highest production value of any podcast I've heard, no doubt mostly owing to the original score by Hans Zimmer.","Find it here.","Kenneth Finnegan, On the metal (65mins, 2020-01-06)","As the show's name suggests, this podcast goes deep into the weeds of CoLo servers, ISPs, and computer networking.","Kenneth's tale has elements of Forrest Gump in the sheer good luck and fortuitousness of his decisions (but with none of FG's sadness).","Regardless of how much you know about the tech in question, hearing how all the pieces fell together makes for a great story.","Find it here.","Not a podcast, but...","It feels a bit odd to continue writing about tech and podcasts when the world is on fire.","I think it's important to show our support for those in peril in the US, but also to not talk over the voices of people who actually know what they're talking about.","Donate if you can #BLM","Stay safe out there (or in there, if you're still locked down).","Tom."]},{"title":"Getting wet, getting loud, getting better","url":"/podcasts-for-nerds/06-wet-loud-better/","content":["At the start of the lockdown I asked my Twitter followers how their podcast listening habits had changed.","The majority of responders said they were listening to fewer podcasts during the "new normal".","A lot of people use their commute as podcast-time!","Personally I've replaced the ~10 hours a week travel-time with ~10 hours a week staring-at-a-blank-wall-musing-on-the-futility-of-existence time, so I'm actually listing the same volume of podcasts as before.","How have your listening habits changed over the past months?","Reply to this email and tell me.","Three great episodes to listen to this week:","Antediluvian, Floodlines (32mins, 2020-03-12)","Vann R.","Newkirk II is quite a name!","Floodlines is his beautifully told examination of the "unnatural disaster" that occurred when the coasts of Louisiana and Mississippi were hit by a Category 5 tropical cyclone in August, 2005.","This is a story about Hurricane Katrina, but it's not all about the meteorology...","Find it here.","That beat, that beat right there, Mogul (38mins, 2017-07-16)","The story of music-manager Chris Lighty is tied to the story of hip-hop, and Mogul tells that story.","Lighty was a rough-and-tumble entrepreneur, and his sorry tale makes for strangely compelling listening.","The series charts his rise and inevitable fall, so I'd recommend starting at episode one to get the full arc.","Find it here.","Things can only get better, About Race (25mins, 2018-03-22)","Well produced and well thought out, this podcast was an instant hit when it was released in 2018 and has been hovering around the top of the charts again this week.","In episode #1, Host Reni Eddo-Lodge (author of Why I’m no longer talking to white people about race) looks at the golden era of multiculturalism in 90's Britain, and asks what went wrong.","Find it here.","Not a podcast, but...","If you like memes, you've probably seen Andrew Huang's work (if not, be sure to check out his "99 red balloons played with red balloons" video).","But as well as making viral music clips he also shares his immense music production knowledge.","His studio is epic, so if you're interested in synths and music making in general, you'll love his Modular synthesis explained video.","Find it here.","Thanks for reading.","If you've enjoyed this week's issue of Podcasts for Nerds, feel free to forward it to a friend or two.","I'm in "growth hacking" mode now, so I'll be extra grateful for any bonus promotion you can give me.","And if you've been forwarded this email (by someone awesome, no doubt), you can sign up for yourself at podcastsfornerds.com to be sent all future issues.","Thanks again,","Tom."]},{"title":"Maximal, minimal, optimal","url":"/podcasts-for-nerds/07-maximal-minimal-optimal/","content":["Most of the recommendations in this newsletter have been at or around the thirty-minutes-long mark.","Many of you have commented that this is "just right" and probably "the best length of podcast to be recommending".","So naturally I'm mixing it up for this issue.","There's a long rambling chat about supply chains (my personal favourite podcast format!), and a short snappy tidbit that's been really well produced.","Plus a show at the "optimal" length.","But you tell me: reply to this email and let me know what your favourite length is for a podcast.","Three great episodes to listen to this week:","In pursuit of tea, Sourceress (52mins, 2019-09-04)","Sourceress is a podcast about supply chains, and this episode is an interview with a New York based tea importer.","It's a long chat conducted during a tea ceremony, but stick with it because there's some fascinating stuff in here.","Learn all about how the picking process determines the type of tea, and also why you can’t sell Americans tea that tastes “shrimpy”.","Find it here.","Dinosaur, Science Diction (12mins, 2020-03-10)","Richard Owen won acclaim as the world's first palaeontologist and inventor of the word "dinosaur".","Trouble is, Owen himself was the one doing the acclaiming.","With a talent for naturalism, and an even bigger talent for self promotion, Owen was a controversial figure who picked a fight with one of the world's most revered scientists.","Find it here.","He's having second thoughts, Dead Eyes (33mins, 2020-01-23)","What do you do when you've been fired by the most likeable man in the world?","Actor and comedian Connor Ratliff's career was almost over before it had begun, and twenty years later he's digging into what happened when Tom Hanks decided he had "dead eyes".","This is an odd story, hilariously told.","Find it here.","Not a podcast, but...","There aren't many of my fellow nerds that are as interested in late-nineties BMWs as I am, but most nerds I know are interested in cameras and the craft of filmmaking.","This film is a quick look at the concept of the "chase car": a fast but practical car that has a monstrous camera rig strapped to it and is used to get all the fantastic shots of supercars travelling at speed.","Even if you're not as excited by an E39 M5 as I am, you've got to admit the footage is stunning.","Find it here.","As always, feel free to forward this email to someone you think might like it.","And if you've been forwarded this email (by someone awesome, no doubt), you can sign up for yourself at podcastsfornerds.com to be sent all future issues.","Thanks again,","Tom."]},{"title":"Chaos, productivity, and more chaos","url":"/podcasts-for-nerds/08-chaos-productivity-more-chaos/","content":["We often hear that "discovery" is the biggest unsolved problem in podcasting.","I'm trying to address that problem with this newsletter, and the very fact that you've subscribed means you're actively seeking out new podcasts to listen to.","But beyond knowing what I do, I don't really know how people find out about shows.","So tell me.","Reply to this email and let me know the most common way you hear about new shows.","I'll share the best tactics next week (spoiler alert: tactic #1 is "subscribe to Podcasts for Nerds", so be sure to tell your friends).","Three great episodes to listen to this week:","Chaos Engineering at Netflix, WeAreNetflix (51mins, 2019-02-04)","I find the idea of "chaos monkeys" captivating.","A chaos monkey is an automated program that will turn off pieces of a company's key infrastructure without warning; essentially the digital equivalent of running through a data centre and unplugging random cables.","In theory it's a great way to test the durability of an online system, and the idea has been championed by the engineering team at Netflix (one of the biggest instances of online infrastructure in the world).","I'm not certain who this podcast is for - I suspect it's mostly a recruiting exercise aimed at attracting engineers - but it's very interesting nonetheless.","Find it here.","Productivity 101, Cortex (110mins, 2020-05-11)","There are a handful of shows where I never miss an episode, but they're hard to recommend to people because the appeal of the show comes from having gotten to "know" the hosts after listening for ages (years, in some cases).","Cortex is one of those; beyond "start at the beginning and see if you like it", it's tough to come up with definitive episodes for people to try out.","Thankfully a recent instalment has (sort-of) addressed this by handily distilling all their productivity tips into a single episode.","(And if you like this one, my best advice is to go back to their first episode and follow along from there.)","Find it here.","The Earthquake, The Big One (30mins, 2019-01-10)","If you got a kick out of the real-time-disaster-reporting of Floodlines (recommended in issue #6), then you'll love The Big One.","Thankfully the premise of this show is speculative (for now).","The city of LA sits on a massive geological fault line, and will be hit by a massive and potentially devastating earthquake.","But no-one can say when, so preparations are maybe not as thorough as they should be.","This excellently-produced show charts what would happen when the The Big One finally hits.","Not a podcast, but...","I'm guessing you've all seen the videos of Spot; simultaneously the cutest and the creepiest robot around.","Well now Spot's creators, Boston Dynamics (odds-on favourites for instigating the inevitable robot uprising), are selling Spots to anyone with a spare $74,500 lying around.","The most engaging "comercial" application so far is seeing Spot trialed as a sheep dog in New Zealand.","Watch the madness here.","As always, feel free to forward this email to someone you think might like it.","And if you've been forwarded this email (by someone awesome, no doubt), you can sign up for yourself at podcastsfornerds.com to be sent all future issues.","Thanks again,","Tom."]},{"title":"Dress smart, hear the echo, and ask great questions","url":"/podcasts-for-nerds/09-smart-echo-questions/","content":["Favour time!","I would love to be able to include some testimonials on the Podcasts for Nerds homepage.","If you're enjoying these emails, it'd be great if you could tweet one thing you like about them; "I like this newsletter because it summarises podcasts I'm too lazy to listen to" - that sort of thing.","Include the hashtag #podcastsForNerds, and I'll embed them on the PFN site.","Three great episodes to listen to this week:","Suits, Articles of Interest (33mins, 2020-05-26)","Podcasting heavy-weight 99 Percent Invisible will no doubt feature a lot in this newsletter; their deep-dives into niche topics are prime fodder for us nerds.","For now, treat yourself to an episode from their clothing-focused mini-series, Articles of Interest.","Avery Trufelman covers the history of the suit, from Beau Brummell through to the modern day.","Find it here.","Sound Break: Hagia Sophia, The World According to Sound (4mins, 2020-04-09)","This one sent shivers down my spine.","In this remarkable four-minute show you get to hear some normal-sounding chanting, but then you get to hear it again as it would have sounded in the Hagia Sophia - a building with an 11 second (!)","reverb.","Find it here.","How to make millions by writing online (Sam Parr of the Hustle), Indie Hackers (60mins 2020-05-08)","People who have the self confidence to drag a profitable business out of literally nothing (like, the absolute tiniest shred of an idea) tend to not be the kind of people I'd want to hang out with, and I'm pretty sure Sam Parr is one of","those guys.","All of this interview is interesting on an academic level, but I find Parr himself quite annoying.","However (and here's the real value-proposition of this newsletter) I've trawled through the whole thing to be able to tell you about the great section in the middle of this podcast where Parr outlines one of his key superpowers: asking","questions.","The kicker is that he then demonstrates it perfectly by grilling the host, Courtland Allen.","Find it here.","Not a podcast, but...","Tom Scott's "Things you might not know" series has been running for years, and all of them are great (if fact, his whole channel is consistently awesome in the most nerdy way possible).","His latest, Why You Can Spot Bad Green Screen is particularly relevant to those of us who've been messing about with Zoom's "virtual backgrounds" lately (as well as simply highlighting why some bad TV looks so bad).","Watch it here.","As always, feel free to forward this email to someone you think might like it.","And if you've been forwarded this email (by someone awesome, no doubt), you can sign up for yourself at podcastsfornerds.com to be sent all future issues.","And I'm serious about the Twitter thing; it'd really help me out.","Thanks again,","Tom."]},{"title":"Way more writing than you signed up for","url":"/podcasts-for-nerds/10-way-more-writing-than-you-signed-up-for/","content":["This week heralds yet more evidence that podcasting is becoming a serious mainstream medium.","Podcasting has been bubbling under the surface for years (decades?)","as a niche nerdy pursuit, but following Spotify's $230 million acquisition of Gimlet Media (a pioneering podcast studio who are responsible for quite a few shows that have featured in this newsletter) early last year, the stakes have felt a","little bit higher.","Then in May Spotify dropped $100 million to bring the execrable Joe Rogan podcast into their fold (arguably making him the highest paid broadcaster in the world).","But now it's not just Spotify who are flashing their cash in the world of podcasting.","US digital radio station SiriusXM bought the Stitcher podcasting studio this week for $300 million.","I don't think we've featured any Stitcher shows here before.","My money was on "How did this get made?","" being the first Stitcher show to be included here, but "Blowback" has beaten them to it (see below).","We're in a time where there's clearly more interest in podcasting from companies with deep pockets, and it cuts both ways.","There's more funding and a bigger market for podcasters, but also the existential threat of platform lock-in (Spotify are big proponents of shows that only appear on their platform).","As we've touched on before, is a podcast really a podcast if it doesn't have an RSS feed?","Maybe...","I'm still undecided.","Three great episodes to listen to this week:","Their dark materials, 99 Percent Invisible (40mins, 2020-01-21)","Following on from last weeks' appearance of Articles of Interest, this week I'm including a "proper" episode of 99PI.","The "feud" between artists Anish Kapoor and Stuart Semple is both hilarious and nerdy.","Vantablack is (or was?)","the world's "blackest black", but the carbon nanotube technology was exclusively licensed by Kapoor, thus prompting an inevitable backlash.","Hot drama with the lowest of stakes; I love it.","Find it here.","Rosebud, Blowback (58mins 2020-06-14)","I don't know how to classify this show.","Is it a "two guys casually talking nonsense to themselves" deal?","Yes.","Is it a tightly produced and well researched history show?","Also yes.","Curious tone aside, this show is a detailed look at the events that lead to the 2003 invasion of Iraq.","But it's also a little bit funny?","As you can tell, I'm finding it all a bit confusing.","But it's strangely compelling story telling, and I'll keep listening to the full series.","Find it here.","Jason Cohen - Learning to Hire and Manage a Team, Full Stack Radio (55mins 2020-07-01)","FSR host Adam Wathan's 2-man business is doing okay.","His book Refactoring UI has earned $2.2m since December 2018 (I own the book; it's great) and their "new" CSS UI library Tailwind has earned $1.8m since February this year.","Now they're thinking about hiring someone, and Jason Cohen (the founder of massive hosting company WP Engine) is on hand to give him some much needed advice.","Find it here","Not a podcast, but...","Sometimes I want a deep dive into a densely technical topic, and other times all I want to do is watch ten years' worth of footage of the Sun spinning around.","Thanks to NASA, I can do that now.","Watch it here.","As always, feel free to forward this email to someone you think might like it.","And if you've been forwarded this email (by someone awesome, no doubt), you can sign up for yourself at podcastsfornerds.com to be sent all future issues.","Thanks again,","Tom."]},{"title":"Music. Music? Music!","url":"/podcasts-for-nerds/11-music-music-music/","content":["I've got another loosely-themed issue for you.","This week's podcast recommendations are all about music.","That said, I've often received a pretty strong "that's not music" response every time I've tried to play tracks by The Books to friends and family.","And as for bleepy-bloopy modular-synth based stuff, the disgust and repulsion was even stronger.","So your milage may vary but I think all this stuff is bona-fide music, and I love it.","Three great episodes to listen to this week:","The Books: Smells Like Content, Song Exploder (16mins, 2014-11-12)","If forced to compile a list of the Top Ten "Best" Podcasts, Song Exploder would certainly be on it.","The show ticks all my boxes: a deep dive into a niche topic?","Check.","Beautifully produced?","Check.","An array of guests that aligns with my tastes?","Check.","An obsessive interest in picking apart the minutiae of a musical track, piece by piece?","Check.","(Okay, I'll admit there aren't too many shows that tick that last box).","The show's creator, Hrishikesh Hirway, has proven on other shows to be a charming and sympathetic interviewer, but in Song Exploder he's worked hard to be invisible: he lets the musician and the music tell the story.","This episode in particular is a great place to start because Smells Like Content is song that was written at the mixing desk: a collage of found-sounds and unusual production tricks.","Find it here.","Tom Whitwell: Music Thing Modular, Why We Bleep (81mins, 2018-01-21)","This newsletter is inadvertently charting my descent into an obsession with modular synthesis (kickstarted by the Not A Podcast from issue #6).","The latest manifestation of my new interest is a willingness to hoover up any modular-related content, and having thankfully some of this comes in podcast-form.","But if you don't share my interest in synth hardware, fear not: this discussion between performer and modular-YouTuber Mylar Melodies (real name: Alex) and module-creator/builder Tom Whitwel (real job: Digital Editor of The Times) covers","wider and more interesting ground than just modular synthesis.","Highlights include a section on "generative" music and some great anecdotes about encounters with the OG musical minimalist, La Monte Young.","Find it here.","Maisie Peters, Ditty In A Dash (22mins, 2020-06-22)","Despite considering myself to be (on some level, at least) a musician, traditional song-writing and lyric-creation has always been a blind spot of mine (as evidenced by my interest in bleepy-bloopy instrumental nonsense).","This new podcast, Ditty in a Dash is a wholesome look behind the curtain at that very process.","Host Frances challenges her music friends to create a song in fifteen minutes (or less) with just the barest bones of starting inspiration.","The results are surprisingly effective, and the whole experience is thoroughly entertaining.","Be warned, however: if the sound of young people being happy drives you crazy, this might not be a podcast for you.","Find it here.","Not a podcast, but...","If the beeps and bloops and talented young people are all getting too much for you, you'll probably want to hit the wine.","But what if you're not a traditional "wine snob", and find the lingo confusing?","As always, YouTube's got your back.","Video creator Sabrina Cruz neatly explains the problem, and recruits her editor Melissa Fernandes to help dig into the topic properly.","The whole video is great, but be sure not to miss the excellent "flavour maps".","This week's Not A Podcast was recommended by loyal newsletter reader Adam (do we need a label for us?",""Nerds" feels like it's already taken.","PFNers sounds a little too contrived.","Suggestions on a postcard) Watch it here.","As always, feel free to forward this email to someone you think might like it.","And if you've been forwarded this email (by someone awesome, no doubt), you can sign up for yourself at podcastsfornerds.com to be sent all future issues.","Thanks again,","Tom."]},{"title":"Late of this parish","url":"/podcasts-for-nerds/12-late-of-this-parish/","content":["This week I've been thinking a lot about podcasts that are no longer with us.","With the current boom in podcasting (are we at "peak podcast" yet?","Maybe...)","I keep seeing lots of faux-stats about the survival rate of most shows.","Proportionally, very few make it past more that a couple of episodes, and hardly any make it past ten.","The long tail is long, but not very consistent.","There have been plenty of shows, however, that had really good runs and made quite an impact on me, but which have now ended (either officially killed-off or on indefinite hiatus).","Three great episodes to listen to this week:","Rob Auten (Video Game Writer), To The F'ing Future (122mins, 2013-04-17)","The epitome of American "Bros" talking nonsense, but To The F'ing Future contained some interesting discussions (if you can get past the host's personalities).","They had a short run in 2013, talking about "the future" of a different topic each week.","Episode #5 (with Rob Auten) is an interesting look at what the state-of-the-art in video-game plot and characterisation was like way back in 2013, and what they thought the future would hold.","Which is even more interesting when viewed from 2020; we can see what played out and what sank without trace.","If gaming isn't your thing, episode #6 was about tattooing (hear me out) and perhaps a little more "timeless".","As with lots of podcasts I love, I have no interest in tattooing, but hearing an expert talk in depth about their work is exactly the kind of content I like the best.","Find it here.","Renay Richardson, Broccoli Content, The Wolf Den (50mins, 2019-01-31)","This podcast about the business of podcasting was very "inside baseball" but I loved it.","Seemingly in the "indefinite hiatus" camp, this show interviewed some key figures in the behind-the-scenes world of podcasting, and always revealed new and interesting details about an industry that I find fascinating.","Find it here.","Art Directing the Web with Dan Mall, Unfinished Business (65mins, 2018-04-18)","When I "pivoted" into my tech career back in 2013, podcasts helped me feel like I was connected to the industry.","Hearing talented and enthusiastic designers and developers talk at length about their work was massively helpful, and this played right into the hands of my podcast addiction.","Unfinished Business was a must-listen show for me, and taught be a lot about the web in general, and designing for the web specifically.","This episode with returning-guest Dan Mall is from a later brief revival of the show (and therefore slightly more current), but there are some gems in the show's back catalogue.","Find it here.","Not a podcast, but...","Sadly in keeping with this week's theme of shows that are no longer running, I wanted to throw a shout-out to MPJ, the creator of the Fun Fun Function YouTube channel.","He announced in an emotional video this month that the show is shutting down, and I'm really bummed out about it.","MPJ's enthusiasm for programming was infectious, and I got a lot from his mission to teach fundamental skills in an entertaining way.","If you like JavaScript and are thinking about getting into "functional" programming, then his original 2015 video series is a great place to start.","Watch it here.","As always, feel free to forward this email to someone you think might like it.","And if you've been forwarded this email (by someone awesome, no doubt), you can sign up for yourself at podcastsfornerds.com to be sent all future issues.","Thanks again,","Tom."]},{"title":"Lots of things. No theme.","url":"/podcasts-for-nerds/13-lots-of-things-no-theme/","content":["Bit of an eclectic mix this week, but I think that's fine.","Sometimes have a unifying theme can be interesting, but more often than not it's just an arbitrary limitation.","There's also been more big-money acquisition news in the world of podcasting this week... but hey, sometimes it's good to focus on enjoying the content rather than obsessing over the business model.","(I will definitely return to obsessing over the business model next week 😛)","Three great episodes to listen to this week:","Offbeat/Psychedelic, Got it (3mins, 2020-01-17)","Another example of stretching the idea of what a podcast can be, this one is simply a two-person parlour game that's been recorded.","Kind of fun to listen to, but mostly useful as inspiration to try it out myself next time I get a chance.","Got it?","Find it here.","Wakandan Econ, Welcome to Geektown (28mins, 2018-08-31)","If you've seen any of the recent Marvel films, you might be wondering if Wakanda (home of the Black Panther) could ever really exist.","Could a super-secret but highly-developed country become the most technologically advanced nation on the planet?","The podcast looks at that question from an economic perspective.","This one might cross the line from "nerd" to "geek", but the inclusion of actual economic theory from an actual economics professor means it's something I'm comfortable including here.","Find it here.","Python in supply chains: oil rigs, rockets, and lettuce, Talk Python (122mins, 2020-06-25)","This week my brother Ed and I had the pleasure of chatting to Michael Kennedy, host of the Talk Python podcast (the chat was for an episode of our own show, A Question of Code - you should check it out).","He's a charming and eloquent guy, and his show is well worth a listen.","It highlights fascinating real-world applications of the Python programming language, lending a dash of context to a subject that can so often be presented in a clinical and abstract manner.","Find it here.","Not a podcast, but...","I got properly nerd-sniped this week by a video about painting restoration.","Because I spend so much of my time working in front of a computer screen I'm always attracted to tales of creative and skillful "real world" craftsmanship - especially when they happen in a cool-looking workshop.","Even if art conservation isn't your bag, stick with it until the presenter starts building a customised work table out of extruded aluminium.","Hat-tip to my Dad for pointing me in the direction of this one.","Watch it here.","As always, feel free to forward this email to someone you think might like it.","And if you've been forwarded this email (by someone awesome, no doubt), you can sign up for yourself at podcastsfornerds.com to be sent all future issues.","Thanks again,","Tom."]},{"title":"Riding the modular wave","url":"/podcasts-for-nerds/14-riding-the-modular-wave/","content":["I've noticed I have to make a conscious effort to not have every recommendation be music-related.","Thanks to discovering the world of modular synthesis I've fallen in love with music and music making all over again, and this is reflected in the podcasts I'm listening to.","Is that a bad thing?","My hobbies often come and go in waves, so we'll probably swing back to another topic in a few weeks.","Three great episodes to listen to this week:","Mahler Symphony No.","6, Sticky Notes: The Classical Music Podcast (42mins, 2020-07-02)","In this three-part series of Sticky Notes, we hear Mahler's 6th dissected and examined.","I'm of the school of thought that believes knowing more about how a piece was made deepens the enjoyment of it, and that holds doubly true for the classical greats.","Structure and technique are such fundamental parts of Mahler's music, and nowhere is that more apparent than in his 6th Symphony.","Find it here.","Face Off: LIVE!, How Did This Get Made?","(94mins, YYYY-MM-DD)","There's a manic energy to all HDTGM episodes which can be "too much" way too easily.","But when that mania is mirrored in the subject of the episode, the format really works and the energy becomes infectious.","The tagline for this show is "Have you ever seen a movie so bad that it’s amazing?","", and 1990's action blockbuster Face:Off absolutely fits that bill.","Find it here.","WeWork: Miguel McKelvey, How I Built This (48mins, 2018-09-02)","With all the madness and hype surrounding WeWork's founder (and all-round bad-egg) Adam Neumann, it's easy to overlook that he was actually the co-founder.","The actual seeds of the idea, and the domain expertise to make it happen, came from architect Miguel McKelvey.","Whereas Neumann is more likely to appear in an episode of Behind the Bastards (we'll cover that show here soon, I'm sure), McKelvey's story is far more wholesome: it's a tale of a good idea executed well at the right time.","WeWork has become, well, "controversial" of late, but the origin story is genuinely interesting.","Find it here.","Not a podcast, but...","Those of you following along with my modular-synth adventure (which has it's seeds in the early issues of this very newsletter) would be forgiven for thinking it's all bleeps and bloops and GAS (Gear Acquisition Syndrome).","But at its core is the fact that modular makes making music fun again (for me, at least).","Nowhere is that better articulated in this video from Red Means Recording.","There are bleeps, for sure, but the driving force behind this video is the music.","It's really well put together and I've watched it at least twenty times already.","Watch it here.","As always, feel free to forward this email to someone you think might like it.","And if you've been forwarded this email (by someone awesome, no doubt), you can sign up for yourself at podcastsfornerds.com to be sent all future issues.","Thanks again,","Tom."]},{"title":"Quirks mode","url":"/podcasts-for-nerds/15-quirks-mode/","content":["Announcement: from this week onwards, Podcasts for Nerds is switching to a fortnightly format.","I want to stress that this is because I'm lazy, and not because I think you're lazy.","A lot of the feedback I'm getting is that you folks don't have enough time to listen to the recommendations - and that's fine!","The whole premise of this newsletter is that you can cherrypick the shows that appeal to you.","Don't feel bad about saying to me "I don't really listen to many of them, but that one that I did listen to was awesome!","": that's literally the whole point of this newsletter.","Three great episodes to listen to this week:","Tailwind, Where to Find Inspiration, SVG Corrections, and Web Workers, ShopTalk Show (55mins, 2020-08-10)","This is probably the podcast I've listened to for the longest.","At first it was a great way to learn more about frontend development, but over time (as is often the way with podcasts) I've come to feel that I know the hosts.","Now it feels like I'm just hanging out with friends.","This feeling intensified this week when STS co-host Chris Coyier joined me and the li'l bro on a recording of the A Question of Code podcast.","Now I know for sure that Chris is as fun to hang out with as he sounds.","We've included a "special" episode of STS on this list before, but this week's episode (number 425!)","is a better example of what STS is really all about: frontend web chat.","Find it here.","1.01: PILOT, The West Wing Weekly (46mins, 2016-03-22)","I considered not including TWWW in this newsletter because even though I'm a generic nerd, I'm aware this show is very much a particular quirk of mine.","But it ended up being one of my all time favourite podcasts, and it was really successful so there must be plenty of other weirdos like me out there.","The premise?","Can you even say you've binge-watched an old series if you didn't consume the watchalong podcast too?","The West Wing Weekly is just that: a discussion about an episode of the West Wing TV show, one a week from start to finish.","I suspect you probably do have to be a fan of the West Wing to properly "get" the show, but I think this podcast is great and just gets better as it goes along.","I binged all ~200 episodes in a couple of months (along with the obligatory WW re-watch, too).","Find it here.","Gregor, Heavyweight (47mins, 2016-09-24)","Heavyweight is a very highly reviewed and exquisitely produced show, and I've only ever listened to one episode.","I thought it was brilliant, and great example of non-fiction storytelling in audio - but I'm pretty sure I mostly liked it because of the premise. "20 years ago, Gregor lent some CDs to a musician friend.","The CDs helped make him a famous rockstar.","Now, Gregor would like some recognition.","But mostly, he wants his CDs back." See?","And it's better than it sounds, too.","I doubt the other episodes are a strong as this, so I've avoided them.","Reply to this email if you've heard the other episodes and think I'm wrong; I'd hate to be missing out on an awesome show.","(Hang on, doesn't that completely reverse the point of this newsletter?!","Oops.)","Find it here.","Not a podcast, but...","Uh oh - CGP Grey was wrong on the internet.","He released a video with (in his opinion) a massive error.","His video explaining and deconstructing how the mistake happened is actually more entertaining than the original video (which was quite interesting to start with).","It's very "inside baseball", but you know by now that "inside baseball" is my favourite genre of content by far.","* Watch it here.","As always, feel free to forward this email to someone you think might like it.","And if you've been forwarded this email (by someone awesome, no doubt), you can sign up for yourself at podcastsfornerds.com to be sent all future issues.","Thanks again,","Tom.","* There is no actual baseball content in any of these recommendations."]},{"title":"Podcasts for Nerds","url":"/podcasts-for-nerds/","content":["Signup to my newsletterJoin the dozens (dozens!)","of people who get my writing delivered directly to their inbox.","You'll also hear news about my miscellaneous other projects, some of which never get mentioned on this site.SubscribeWhat is Podcast for Nerds?","💌One email with three recommended podcast episodes, sent every Wednesday.","🎙️Each recommendation has a short summary explaining why you might like it.","👀Read issue #12 right now: Late of this parish📮Subscribe to read issue #15Issues are archived online after a few weeks, but the only way to get them "fresh" is to sign up to the newsletter.","If you subscribe you'll get to read the most recent issue straight away, and all future issues will wing their way directly to your inbox.Why am I writing this newsletter?","I'm probably overly obsessed by podcasts.","I'm currently subscribed to way more shows than anyone else I know, with even more individual episodes queued-up for review.","Every (sensible) person I know listens to a few, but I definitely listen to too many.So now I'm trying to turn that bug into a feature.I can save you time by summarising the best episodes I've listened to lately.","There's loads of fantastic stuff out there, but you have to listen to a lot of average episodes to be sure of hearing the good bits.","Let me do that digging for you.Recent issues:#15.","Quirks mode (sign up now to read this)#14.","Riding the modular wave#13.","Lots of things.","No theme.","#12.","Late of this parish#11.","Music.","Music?","Music!","#10.","Way more writing than you signed up for#9.","Dress smart, hear the echo, and ask great questions#8.","Chaos, productivity, and more chaos#7.","Maximal, minimal, optimal#6.","Getting wet, getting loud, getting better#5.","Podcast or not?","You decide#4.","Cracking, hacking, and phishing#3.","Soundscapes, a grounding, and a year in isolation#2.","Tannoys, tragic pitches, and business traction#1.","Prescience, new beginnings, and modern linguisticsSign up to read the latest issue (and get a new one straight into you inbox every Wednesday)."]},{"title":"Search","url":"/search/","content":[]},{"title":"Uses","url":"/uses/","content":["This is a list of (some of) the hardware I own and use on a regular basis.","Read more about why I maintain this list in my "Things I Use" article.","Hardware","Laptop: MacBook Pro (15-inch, 2018) 16GB","Desktop: iMac (21-inch, 2012) 8GB","Griffin Elevator Laptop Stand","Bose QuietComfort 35 (Series II) Noise Cancelling Wireless Headphones","WD 3TB External Desktop Hard Drive (with metal enclosure)","Audio Gear","Focusrite Scarlett 2i2 (2nd Gen) USB Audio Interface","InnoGear Pop Filter","Beyerdynamic DT770 PRO Headphones (250 Ohm)","Tabletop Tripod Mic Stand","KORG nanoKEY2 - USB MIDI Keyboard (25 Key)","KORG nanoKONTROL2 - USB MIDI Controller","Zoom H5 Handy Recorder","Video Gear","Neewer Dimmable LED Light","Foldable Roll-Flex LED Light","Canon EF-S 24 mm f/2.8 STM Lens","Canon EF 50 mm f/1.8 II Lens","Canon EOS 60D Digital SLR","Neewer Slim Fast Micro USB Battery Charger","Design","Rotring 800 0.5mm Black Barrel Mechanical Pencil","Kuretake Sumi Brush Pen","iPad Pro (11-inch)","Apple Pencil","FIELD NOTES Notebooks","Moleskine Classic Dotted Paper Notebook","UNI-BALL PIN fineliner MARKER 0.8mm (black)","Sakura Pens Pigma Micron"]}]