Blog: Tech

Most of these posts were originally posted somewhere else and link to the originals. While this blog is not set up for comments, the original locations generally are, and I welcome comments there. Sorry for the inconvenience.

Facebook outage

As the whole Internet knows, Facebook and other stuff they own were all down for several hours a few days ago. They were off the network entirely: DNS couldn't resolve their host names. A post from Cloudflare describes what happened from the outside, including explaining how some of the key parts work (like BGP and Autonomous Systems, terms I learned this week), and a post from Facebook explains what happened inside. Read more…

Trope Trainer

Trope Trainer is a software package for working with Hebrew cantillation (trope). You can use it to view, listen to, or print the weekly Torah reading (or parts thereof), weekday readings, holiday readings, etc. As the "trainer" in the name implies, one of its purposes is to teach the cantillation system -- or, I should say, systems, because there are regional and other variations.

I didn't use it for that because I already know (my community's) cantillation system; while occasional curiosity might lead me to ask it "hey, how does the Lithuanian tradition chant this?", in practice I haven't. No, what I use Trope Trainer for is to print legible copies with the vowel markers and trope markers. These are useful for practicing and, when I know in advance so I can print it, for checking the reader during the service, because the scroll used for readings does not have vowels and trope marks. (There is always somebody following along during a Torah reading to correct the reader in case of mistakes.)

Back in August, somebody in my minyan asked me to be his checker the following Shabbat, so I launched the program to print a copy. But the program was stuck at "checking for updates", a state that had previously passed so quickly that I wasn't used to seeing it. If I cancelled, the program crashed. Repeatedly. A little digging revealed the probable cause: the company went out of business and their domain isn't there any more. Presumably the software is checking a now-dead URL and the programmers didn't handle failures. (There are other reasons the service might not be available, so this isn't just "didn't consider the company might die".) Read more…

For want of a nail, the kingdom was...delayed

I have an open-source project I am very enthusiastic about (Codidact). Mostly my role does not involve the code directly: I'm the community lead (i.e. primary talker-with-people-who-use-it and triager of feature requests), and I do some design of features, workflows, wireframes, internal documentation, and stuff like that. And I beat up on the test server a lot when there's work in progress to poke at. We have infrastructure to support all that.

But sometimes I'd like to get a little closer to the code, mostly for my own education and partly so I can maybe help do smaller things because our team is pretty small still. And there was that one time that I really wanted to fix a front-end bug that I admitted was limited in scope; it was bothering me, but not something to drag a developer off of something else for. And it was in the Javascript code, which I can bumble my way through, so ok, I figured, I can do this. (And there was that weird thing about dates in Javascript, but I digress.) But I didn't have a dev environment to test it with, and ended up putting it in a userscript to test and then asking somebody else to plug it in for real, which meant I needed help from one of the developers after all, and I shouldn't be that lame. Read more…

Hiring dark pattern

We're hiring, so I've had recent occasion to see a hiring "dark pattern". I don't know how widespread this is outside of the tech industry, but I see it a lot in tech and it needs to stop.

The pattern: asking for "number of years of experience in (insert tool, language, or skill here)".

Jobs are multi-faceted. I'm seeing people who don't know how to choose a number, and just as I see the ones that overshot, I know there must be ones who undershot and were filtered out. I saw a resume recently from someone with a broad role, who listed "N years experience" in something where N was the length of the job, even though the job involved many other things and wasn't primarily about that skill. That's "N years of experience" in the same way that I have "20 years of Java experience". Spoiler alert: I don't.

Recruiters and hiring managers put people in this position: we have this checklist, tell us how many years of each, and we need a number not an explanation. Maybe we're making you fill out a web form and you can't even add a comment or ask questions. In the absence of guidance about primary and secondary skills, people will apply different weighting factors. It's a mess.

I often don't know how to answer either. In the last four years I've gained four years of tech-writing experience, the focus of my job, and four years of git experience, meaning I have all the basics and can untangle a merge conflict (but I've never rebased successfully). Those "four"s don't mean the same thing; I don't spend all day using git. Tech writing is fundamental to my role and I'd have no qualms about fully claiming those four years, but for git I would want to say that I've used it as part of my job for four years. That's different. I don't have the same knowledge that an infrastructure person whose job revolves around maintaining git servers and stuff for the same amount of time has. "How many years of experience?" questions don't allow for that nuance.

People who are trying to be thoughtful and honest run the risk of getting filtered out, and people who count "everything" get filtered in. I wonder how the success rate (meaning "acceptable hire", since you'll never know if you got the "best hire") of this exercise compares to recruiters just not filtering on skill-specific numbers at all (just general experience) and having the hiring team evaluate more people. I'm fortunate enough to be sufficiently "long in the tooth" that I don't need to worry about this gatekeeping -- either you're looking for a top-notch tech writer with all that implies, or you aren't -- but this hurts early-career people where the difference between "2" and "4" can make or break a deal.

As for the Java: I spent years documenting and helping to design Java APIs, wrote some examples, but haven't written anything deep and complicated. Someone with a year or two of full-time actual Java development experience beats me. Would the recruiter be able to tell?

The source post on Dreamwidth has a number of thoughtful comments.

The future of Stack Overflow

Yesterday Stack Overflow was bought by Prosus, a tech company based in the Netherlands, for a jaw-dropping $1.8B (yes billion). In the world of recent tech acquisitions that might be small change, but it's about three times what I thought their current valuation was. It's kind of a mystery what Prosus (yeah, I'd never heard of them before either) is getting out of this.

I might have more to say about this later, but for now I'm going to post here what I wrote on Reddit (which I joined for other reasons a couple months ago but hadn't posted on before), in response to a comment referring to "SO’s bonkers relationship with its moderator community" and suggesting that getting bought by a mega-corp would make that even worse.

I don't know how the sale will affect their disastrous relationship with the people they rely on to donate and curate content for their financial gain. Often a new owner doesn't understand what it's bought and makes things worse by meddling. On the other hand, the claim is that Stack Overflow will still operate independently and make its own decisions. In the acquisition of a successful company that would be good news (they can keep doing what they're doing), but in a declining company that shouldn't keep doing what it's doing because it's not working, pressure from the new owner could help, if Prosus will actually apply that pressure.

Stack Overflow and the Stack Exchange network have been in decline for several years (since at least 2017 by my reckoning, some say longer). Some of that decline is due to outside factors and a lot is due to the company's actions. The good news is that most of the architects of those bad decisions are gone now, so the company could take the opportunity to say "y'know, we've been doing it wrong and we need to fix that" without anybody still there having to eat crow. The bad news is that, historically, this is not what Stack does; they double down on bad decisions, I assume because admitting mistakes is embarrassing. Several people still there who weren't part of those decisions now appear to be endorsing them -- whether due to internal pressure or because they drank the kool-aid I don't know.

Thus, the future is pretty unclear to me when it comes to how Stack Overflow treats its moderators and users. If Prosus allows them to operate independently, I expect they'll keep mistreating people even though they no longer have to placate departed leaders. If Prosus takes a closer look at what they've bought, they could make things either worse or better depending on what they decide and how well they execute it. On the current trajectory, I would expect the community, people's willingness to become moderators, and the quality of content to continue their current decline, and the invasiveness of ads and promotion of their Teams and Enterprise products to accelerate. SO is the gateway to the company's for-sale products; it doesn't matter to them independently. The company doesn't need quality and it does need to overcome SO's reputation of hostility, so they're willing to sacrifice the former to attempt the latter. The sad thing is that they could end up with neither even though it's actually possible to get both.

Oh, for crying out loud (thank you Javascript)

We have a widget on our site that displays the current Hebrew date. We're using HebCal's REST API to get the data, which comes back a month at a time -- JSON blob that has one entry for each day, with the secular date, the Hebrew date, and some other information. We use the Javascript Date class to get today's date, extract a string of the right format from that, and look up "today" in that returned collection. Easy, right? The API documentation for Date makes a point of saying that it uses local time, not UTC, which is what we want.

(Because the Hebrew day actually starts at sundown the previous day, we've got some "best effort" code in there that starts the day at 8PM. Not ideal, but we're not looking up local sunsets and suchlike... We also display something like "22 Sivan (Tuesday night and Wednesday)" to make it clear.)

Yeah ok, so we have the local date, and we need to look up an entry in the calendar data. Dates there use ISO format (YYYY-MM-DD). So we get our Date object, which is supposed to be local, and call toISOString() on it.

Guess what. Date.toISOString does not use the local time. FFS. I've been banging my head against a wall for two hours thinking it was something more esoteric, maybe having to do with that "start at 8PM" code (which I had actually commented out to eliminate that possibility). I finally went to an online Javascript tester, one of those sites where you can type in code and run it, and confirmed that right now today.toISOString() says it's June 2 but today.getDay() says it's June 1.

So now I have to write Google for code that constructs the proper format for the local time, which is down to padding numbers with leading zeros if the month or day number is less than 10 and string-munging and...ugh. (Found it easily enough, but sheesh!)

Brain trust: talk to me about hosted content?

I've written lots of stuff in a variety of places online -- (LJ to) Dreamwidth and Medium and SE and one-offs in handwritten HTML and (heaven help us) Twitter and probably some others. Some of it was transient, but some of it is stuff I'd like to keep available and together. Read more…

Social-media platforms and their manipulations

A friend, in a locked post, talked about leaving Facebook because of their manipulation of what you see. "That's a lot of power for one company to wield", the person said. I wrote the following in a comment:

I was never comfortable with Facebook. I had an account for about two minutes, under the belief that if I created and then deleted an account, nobody else would be able to create a fake account to impersonate me. Ah, those days of naivete.

I don't trust platforms that give me a selective view of activity without letting me turn the knobs. On Dreamwidth (and LiveJournal before it turned evil), I know that if I read back until I hit something I recognized, I've seen everything on my subscription list (modulo author-deleted posts). I wasn't going to miss stuff. On Facebook, and Twitter and Google+, that isn't/wasn't true. Twitter now has a control for "show me stuff in order" versus "show me selections based on what's likely to interest me", but I don't know that the former doesn't also filter stuff out. I use Twitter, but knowing that I can't rely on it for consuming stuff. For the people I want to see everything from, I get notifications. That doesn't scale, so I miss a lot, and therefore it's not reliable. But I'm there because that's where some important-to-me connections are, presumably like why you were on Facebook.

Not being on Facebook has cut me off from some stuff, I know. Too many people think "well, I posted it on Facebook so you should know about it". I can only hope that, now that more people are becoming aware of how Facebook filters content out, some people might stop assuming that. We'll see.

When you use someone else's platform you're at risk of them flaking out or changing their rules or them just being dishonest. Facebook, Twitter, Stack Exchange, Reddit, LiveJournal... they control the servers so, ultimately, they control your activity. They can be arbitrary (recently saw someone get booted without appeal from Twitter for content that their algorithm said was bad, that wasn't), they can be capricious (Stack Exchange, need we say more?), they can be evil (LiveJournal, now in thrall to the Russian government)... we make the best decisions we can at the time about where to participate, and sometimes learn we were wrong and have to move.

But striking out on your own has costs too. If you set up a blog on your own, a few of your friends will subscribe by RSS but you won't have the community aspect. When LJ imploded I didn't set up my own blog; I moved to Dreamwidth. I decided to trust Dreamwidth, and believe they are more worthy of that trust than LJ was. I like to think that Codidact is more worthy of trust than Stack Exchange is, and that people can participate with more confidence that they won't be treated sneakily and viciously. Disclosure: I'm one of the people in charge at Codidact, so I'm biased. But operating in the open and not having stockholders makes a big difference too, I think.

If we want to be part of Internet-connected communities, whether small groups of friends or huge international groups, then we have to either build our own platforms or rely on others. Usually the former is impractical, so it comes down to remaining vigilant about the practices of the providers on whom we depend. And sometimes we need to pick up and move (or leave), disruptive as it is.

GitHub graph

My GitHub history got a lot less sparse in 2020, and especially in the last few months of the year. It's great to be a productive member of my first open-source team!

activity graph

Gmail update

Someone who can self-identify if desired shared Google's summary of the recent email outages (PDF). This is the outage that caused my address (and many others) to start sending permanent bounce messages.

Background: The Gmail SMTP inbound service uses a configuration system that allows specific service options and flags to be changed while the service is already deployed in production. The "" domain name is specified as one of these configuration options. An ongoing migration was in effect to update this underlying configuration system to meet Google internal best practices.

A configuration change during this migration shifted the formatting behavior of a service option so that it incorrectly provided an invalid domain name, instead of the intended "" domain name, to the Google MTP inbound service. As a result, the service incorrectly transformed lookups of certain email addresses ending in "(at)" into non-existent email addresses. When the Gmail user accounts service checked each of these non-existent email addresses, the service could not detect a valid user, resulting in SMTP error code 550.


To guard against the issue recurring and to reduce the impact of similar events, we are taking the following actions:

  • Update the existing configuration difference tests to detect unexpected changes to the SMTP service configuration before applying the change.
  • Improve internal service logging to allow more accurate and faster diagnosis of similar types of errors.
  • Implement additional restrictions on configuration changes that may affect production resources globally.
  • Improve static analysis tooling for configuration differences to more accurately project differences in production behavior.


Fixing things in production systems is hard. I've been there; things can go wrong, sometimes badly wrong. I'm used to thinking of Google as having near-infinite resources, including a replica of their production system to test changes on. Perhaps that's unrealistic.