Caffeinspiration

Timeline and Early Days

Sun, 16 Nov 2025 16:27:10 -0500

Timeline and discovery: the early days

The simple timeline of my journey sounds deceptively easy at first:

I found out about the issue after an echocardiogram on January 23rd, 2025.
I had open heart surgery on Tuesday, June 24th
I was sent home from the hospital three days later on Friday, June 27th
I hit the 12-week mark on Tuesday, September 16th
It’s now Sunday, November 16th

Just a quick ten months in which I had to process, learn, understand, talk to experts, learn some more, imagine worst case scenarios, cope, walk, constantly be aware of my heart, try not to think about it, try to carry on, get more scans, get second, third, and even fourth opinions, become an expert at navigating these medical appointments, make difficult decisions, make the call and pick a date, get a list of all of the financial accounts and their passwords just in case, get covid, get better, invite family to town, take medical leave, show up at the hospital, put one foot in front of the other, take a quick nap, wake up, spend three grueling days at the hospital, make it home, start recovery, build up my walking again, not pick anything more than ten pounds up for three months, get better, starting picking things up again, start jogging again, and return closer and closer to a normal life.

Not bad!

Naturally, the full timeline is much more complex. Looking back, there has definitely been some time dilation going on here in my memory, between the rawness and the flurry of activity pre-surgery, the overwhelming intensity of the 3-5 days of the surgery and hospital stay, and the floating repetitiveness of my recovery months. Really, it’s as if the last ten months are separated into three even chunks of time in my memory, which above all else really underlines just how intense the few days around the surgery really were:

But for now, let’s return to the start.

The longer story really starts much earlier. My family has a complex medical history, particularly around cardiac health, with issues on either side. Most immediately relevant to my own experience is my father’s history, where he had to have a mitral valve repair, then a repair of a repair. He ended up passing away from a presumed cardiac incident a little over two years ago.

With that in mind, my siblings and I all share these genes, and all of us got echocardiograms, where a technician measures the various sizes, diameters, blood flows, and efficiencies of your heart, valves, and vessels. Having one done when you’re younger can provide a good baseline to compare against as you get older, allowing you to measure things like changes in sizes or valve efficiency over time.

That was the mindset I had when I scheduled an echocardiogram in January; let’s measure so we can have the data points to compare against if health issues start to show up in my 50s or 60s. Calm as a cucumber, I went in for the echo, and, knowing the smallest bit of heart anatomy from a random heart physics class I took in college, I even joked with the tech about one of my valves having three leaflets, just as it should. I had an appointment with my cardiologist immediately after the echo, and I was expecting a short and sweet “all’s well, see you later” that I had come to expect from most of my visits.

In retrospect, the first sign that something wasn’t right might have been that they came in and let me know they decided they wanted to get an EKG as well. I remember it was a chilly 20 degree day that I had gotten bundled up for to walk to my appointment, and I had to roll back my perfectly tucked socks and wool long underwear so they could put a lead on my leg. The tech even apologized for disrupting my perfect layering. It’s funny what little things stick out in your memory.

When the cardiologist came in and shared the results of my echo, the main things I remember clearly are the words " aortic aneurysm", “likely need surgery to fix”, and “refer you to a cardiac surgeon”.

…!

If you’ve ever been shocked with bad news, you probably know the sensation of immediately being completely overwhelmed. It was almost as if everything else shut down while my brain worked to re-process those words, each of which I knew, but also knew they didn’t apply to me. Surely it wasn’t little old me at 32 and relatively good health — this was not how these appointments were supposed to go.

If you’d like to see what it roughly felt like, I have an easy four step process to try it out:

First, turn on an annoyingly high pitched noise to set the mood (tinnitus-enjoyers can skip this step)
Second, plug your ears so the only thing you can hear is your breathing and the beating of your own heart
Third, close your eyes, because you couldn’t
Finally, repeat in a whisper to yourself: “you need heart surgery, you might die, and your life will never be the same”

Fun!

I motioned from my neck down my sternum and asked if it would be heart surgery heart surgery, and my cardiologist said it would be up to the surgeon, but likely yes. They mentioned a follow-up CT scan to get even better images and measurements, and they recommended stopping any strenuous physical exercise, as spikes in blood pressure can worsen the aneurysm and lead to tears, which would be very, very bad.

Physical exercise could be bad, you say? Funny you should say that, because three weeks before, I had been doing very strenuous physical exercise in Joshua Tree, rock climbing and scrambling with no cellphone service, even occasionally going out on solo morning scrambles deep in the park on random, infrequently visited areas. There easily could have been a parallel timeline where something happened during one of those scrambles, and that would have been it for me.

My Joshua Tree alternate timeline resting place

I walked home from that appointment in an absolute daze. I remember that my main thought was how I was going to break this news to my wife. We had been joking and silly before I left, and coming back with this kind of news was guaranteed to cloud the rest of the day. There was a fleeting impulse to not tell her at all — let’s just keep this terrible news to myself and not ruin anyone else’s sunny January day. I floated past the rest of the 25 minute walk, with each foot bringing me a little closer to the difficult thing I’d inevitably have to soon do.

On the way back, I experienced this odd sensation of forcing myself to put one foot in front of another, making myself move forward even though I was scared, uncertain, and not looking forward to it. I knew what I would have to do — return to my apartment and share the news with my wife — and I just had to go through the motion as best I could. This would end up being a recurring theme throughout my entire journey; knowing what I have to do, I just have to go step by step.

Biblically accurate map of Somerville with my walk to the appointment and back

Naturally, I did tell my wife, and naturally, it did fuck up her day too.

Once you know something like that, you can’t go unlearn it. The pre-aneurysm times were over, even though we didn’t even know at the time how lucky we were to be in the pre-aneurysm times. Life was immediately divided into a “before” and “now”, with the new chapter starting with a bang. Funny enough, it was only the new knowledge that was causing this split. It was likely that nothing had physically changed from the day before, but now we knew about it, this new cursed fact infecting every part of our thoughts.

Along with the knowledge, there were so many questions. What did this actually mean? Would I really need surgery? What if it isn’t really a problem and we can just wait it out? Why did this happen to me? What happens next? As you can tell, the story goes well, but at the time, we were very far from knowing it would.

This is how the story began.

(Link back to heart topics)

On having heart surgery at 32

Sun, 09 Nov 2025 13:00:16 -0500

Author’s note: this is a departure from topics that I usually write about. It’s not related to tech, learning, or anything I’ve written about before; it’s my personal experience going through heart surgery at 32.

Reader, be warned that there are personal and real details. There won’t be any gory images, but I won’t be sanitizing the experience or pretending it was just sunshine and rainbows and not incredibly difficult. I’m writing this for myself, and though it might be difficult at parts, you’ll find it to be true.

This is my website, after all.

With that said, enjoy!

In June 2025, at 32 years of age, I had to have open heart surgery. I had an aortic aneurysm, which means that my aorta (the big tube that transports blood pumped from your heart to the rest of your body) was wider than it should have been in one critical section. This can be an important problem to fix, because it can grow larger and tear, at which point you have a very, very big problem.¹

I’m incredibly fortunate that we caught it, met some wonderful doctors and surgeons, and were able to have it fixed. Everything went extremely well, and my recovery has been tremendous. I’m fully fixed, and life is returning to normal.

This was one of the most difficult things I’ve ever gone through in my life, and as you can imagine, I’ve gone through the entire spectrum of emotions, from deep grief from facing death to deep gratitude for having found it in the first place and joy at having had everything go as well as it has.

I’ve found that writing is one of the best methods I have for processing things, and in the months since my surgery, I’ve been filled with a desire to write about my experience. I’m largely doing this for myself, but I’ve found joy sharing my experience with others, especially if I can spread some humor at the same time (one of my key coping mechanisms!). On top of that, if this helps at least one other person dealing with a similar situation, then that’ll just be the cherry on top.

I’m likely going to approach it by writing little vignettes about various parts of the experience. There may some day be some chronological order to it, but for now, I’m just going to write about whatever I feel like for that day. I’ll be adding them here over time so they don’t take up my entire landing page post list, since I’ll likely be continuing with regular programming as well. ²

Cathy

Let’s start it off with a bang, remembering my dear friend Cathy the catheter — I’ll never forget the time we spent close together, for better or for worse.

I warned you about it being personal!

Timeline and Early Days

Next up is a nice, relaxing echocardiogram and cardiologist appointment on a chilly January day: Timeline and Early Days

This is a good time to mention that I’m not a medical doctor, and obviously nothing written here is medical advice. ↩︎
Though maybe I’ll start having this site be more of my personal writing in general as a place to share it. Especially these days, I don’t care about it helping with my career hustling. ↩︎

Cathy: my dear, helpful catheter

Sun, 09 Nov 2025 13:00:11 -0500

Listen, there isn’t much in this world that’s as luxurious as being able to pee whenever you want to. Imagine floating in the ocean, free at the drop of a hat, except it’s anywhere. A simple luxury, but a freeing one. Or, at least it would be, if you weren’t confined to a hospital bed.

I’m talking, of course, about wearing a Foley catheter.

The only small downside to this absolute luxury is having a tube constantly down your urethra. You notice it constantly; every slight repositioning of your legs means a gentle, calming, pulling sensation as if your bladder was being pulled out the long way, with extra friction the whole way out.

Funny enough, that’s also how it stays in there. Theres a little balloon at the end that gets inflated once it has made its way all the way to your bladder. Like a ship in a bottle, it’ll stay in place, resisting being tugged.

And tugged it was. Mine was the plain old gravity version, which meant that the drainage would slowly make its way out, helped by the general downhill slope of the tubing, but occasionally, one of the nurses would have to adjust it to help it on its way.

No matter — what’s a slight, uncomfortable tugging among friends? It was far from the most uncomfortable thing I was going through at the time, and unlike some of hte other sensations, at least it was pretty predictable.

Predictable, at least, while I was awake.

There are many terrible ways to wake up in this life. Fire alarms and the smell of smoke is one, or maybe waking up to the sound of a kid or pet about to vomit. I’d like to offer another for consideration: the catheter tug.

Sitting in my hospital bed, I must have dozed off at some point during a lull in activity, having not been sleeping well, because I was suddenly brought crashing back to this conscious realm by a well intentioned nurse adjusting my catheter to help it drain. There’s something so guttural about being pulled from the inside out, and the sensation shocked me awake faster than any coffee ever could. A mix of adrenaline, shock, and friction, especially in so private an area, immediately took my full attention, easily beating out any oxy fever dream I might have been briefly entertaining.

If you’ve ever struggled to wake up to your first alarm, then I’ve got the perfect product for you!

As time went on, as convenient as it was, I was looking forward to not having my catheter in any more. We had been through some difficult times together, and she (Cathy) had been a great convenience, but I knew these times couldn’t last.

I was not, however, looking forward to the removal. Though directionally it aligned better with the usual direction of flow that I’m used to down there (thank god I was out when it was inserted), I knew it wasn’t going to be pleasant.

If you somehow got access to my 3AM oxy-and-insomnia-fueled googlings in the early morning of the 26th, you’d see “hwo are foly cathetrs removed”, quickly autocorrected and answered by Google. It’s the opposite order: deflate the balloon, then pull it on out. The nurse was honest, and she was kind. She told me it was going to be brief, but uncomfortable.

It was both.

But, then it was done. While still smarting, I realized that for the first time in days, I could cross my legs without that telltale tugging sensation. I immediately stretched my legs with a huge feeling of relief, like the first time you’re able to stretch after a long flight in the middle seat. Only you hadn’t been in the middle seat — you were in a hospital bed (and sometimes a recliner) with a tube all the way in you.

Having it out made standing, sitting, and moving so much easier. I still had my drainage tubes and pacemaker wires going in through my belly, but it was a relieving improvement and a great step in the recovery process. My entire process was a series of these incremental steps, each a small action, though they each felt huge, and each victory brought me one step closer to discharge, to getting home, and to getting better. It made the process a series of gradual improvements that helped harden my resolve and make me readier for the next step — I made it through that part, crossing it off my list, and now I can move on to the ever reducing list of next steps.

Afterward, my nurse gave me a portable urinal and told me I had to pee on my own within six hours before they’d start to worry. I’ve never been one to shy away from a challenge, and armed with my resolve and newly found mobility freedom, I took it in stride, making my way through a couple cups of ice water. The urinal was awkward at first, and I’m sure the office workers in the building across from my window saw more than they had bargained for, but we made it work and were home free. Or, if not home free, at least free to move on with everything else I had to do, free from any of those eyebrow raising tugs.

But, I’ll never forget the sensations, especially that wake up call. And that’s why, Sharks, I’d like to return to my alarm clock idea, and with an initial investment for 10% of the company…

(Link back to heart topics)

How much oranger do red orange bags make oranges look?

Sun, 13 Apr 2025 00:39:45 -0400

Look at this orange:

Now look at this orange:

It’s the same orange.

But, look how much more orange it looks with the red mesh on top of it:

If you buy bags of oranges (at least at many places in the US), they frequently come in this red mesh bag. This bag makes the oranges look more orange. Oranger.

Here’s what that looks like at a local grocery store¹:

Ripe oranges are usually oranger, so this bag makes the oranges look better than they may actually be. Maybe the secret is to never buy bagged fruit, since it’s harder to evaluate the quality of each orange.²

This made me wonder — how does the bag change how we perceive the color?

I thought this difference would be visible if we did some quick and tricky digital math: what if we had a picture of the orange with and without the bag under the same light and camera conditions, then checked the average pixel?

Here are the results from 11 different orange photos, with and without the mesh:

Orange	Without bag	With bag	Avg. color without	Avg. color with
1
2
3
4
5
6
7
8
9
10
11

There are a few interesting things here. First, the average pixel is not what I would expect it to be at all, to be honest. I even ran the average pixel calculation a second time with more advanced calculations, including some orange-only-masking to avoid non-orange colors, but I got similar results. They’re all much more brown than my eyes would assume when I look at the images.

Weirdly, that kind of makes sense when you look at each image closely. Here’s a big trypophobia warning, but you can open the spoiler below.

Click to see a close up photo of orange skin, which is kinda weird to be honest

Look how much brown there really is when you look closely! Also, kind of gross.

Kinda weird, right? This kind of makes sense though — this whole thing was motivated by the feeling that our eyes are tricked by colors, so it makes sense that our eyes are much less analytical than my computer averaging over all of the pixels.

The other interesting thing is that the addition of the red mesh clearly adds a warmth to each of the average colors. We can see a clear shift, even for those showing up as brown.

We see the RGB shift mostly in the green, interestingly enough. The average change to RGB values is around (-15, -20, -4) with the bag, with some larger shifts in the green. That’s a little hard to visualize, but that’s the difference between this first pale yellow and the second, more robust orange:

OK, maybe not exactly a robust orange, and not exactly more appetizing, but again, I think our mind is probably playing more tricks on us. There’s also probably a better way to think about color shifts that I’m not familiar with, but even as a basic measure, we can see this clear shift with the average pixels side by side.

Of course, as expected, eyes are incredibly complex, and the answer is much more nuanced than the average pixel value: our eyes adapt to the environment, remember the color of things, and change dynamically.

If you’re interested in a rabbit hole, I’d recommend this NASA Research Center Color Usage Research Lab article on simultaneous and successive contrast and these this Wikipedia article on chromatic adaptation, color appearance model, and color constancy. It obviously extends well past a simple average pixel color!

Given that the trick is happening in our eyes, I think a better experiment would be a human-focused experiment for how we perceive the average color. Maybe we could have two groups, with bag and without, and we show them the cropped photos and have them pick the average (or most dominant?) color they perceive in the photo. We’d then be able to compare across the groups to confirm that the with-bag photos skew redder.

Maybe another day. I think I’ve already been staring at pictures of oranges for too long.

Anyways, here’s how I set this up.

The experimental setup, with the author’s attentive assistant

I took 11 different photos of various oranges in the same position, with and without the red mesh, and cropped the same section of each photo.

I found the pixel locations of the square I wanted, then I translated those coordinates into specific offsets for a sips command (Scriptable Image Processing System), which I learned about today. It made this programmatic cropping very easy. For example, cropping two photos of Orange 1 in the same position, with and without mesh, as two files, orange-1-with.jpeg and orange-1-without.jpeg:

for f in orange-1*.jpeg; 
  do sips -c 788 740 --cropOffset 1519 1083 "$f"; 
done

This let me go from these two photos:

To these two photos

Assuming I put the mesh on without disturbing the orange, this meant that we would be doing an exact comparison between the two.

After I did this for all of the photos, with and without mesh, I then used magick to calculate the average pixel value:

$ for f in *-cropped.jpeg; 
>   do   echo -n "$f: ";   
>   magick "$f" -resize 1x1 txt:- | grep -o '#[A-Fa-f0-9]\{6\}'; 
> done
orange-1-with-cropped.jpeg: #B9310A
orange-1-without-cropped.jpeg: #D0530A
...

Pretty neat!

Once I found that everything was showing up a lot more brown, I also experimented with a basic Python script that leveraged OpenCV. This script creates a mask for each image that excludes non-orange-ish pixels, defined by a range that I define. It can then take the average over just the orange pixels that fall outside of the mask.

for file in files:
    image = cv2.imread(file)

    # Convert to HSV color space (better for color detection)
    hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)

    # Define the range for orange in HSV. This took some tinkering to get the right values.
    lower_orange = np.array([3, 150, 150])
    upper_orange = np.array([20, 255, 255])

    # Mask the image to get only the orange parts.
    mask = cv2.inRange(hsv, lower_orange, upper_orange)
    orange_pixels = cv2.bitwise_and(image, image, mask=mask)

    # For debugging, I saved the binary mask to visualize them.
    mask_filename = os.path.join(output_mask_dir, os.path.basename(file).replace(".jpeg", "_mask.png"))
    cv2.imwrite(mask_filename, mask)

    # I also saved just the orange parts to visualize it.
    orange_only_filename = os.path.join(output_orange_only_dir, os.path.basename(file).replace(".jpeg", "_orange.png"))
    cv2.imwrite(orange_only_filename, orange_pixels)

    # Now, take the mean of the orange pixels with the mask, which means we're (hopefully) ignoring all of the browner 
    # pixels when calculating the mean.
    bgr_avg = cv2.mean(orange_pixels, mask=mask)[:3]
    # Then, translate to RGB (and HSV for debugging).
    rgb_avg = tuple(reversed(bgr_avg))
    hsv_avg = cv2.cvtColor(np.uint8([[bgr_avg]]), cv2.COLOR_BGR2HSV)[0][0]

    print(file, "Average RGB orange color:", rgb_avg, "HSV:", hsv_avg)

This was pretty neat, because it meant that I could mask away any non-orange pixels (like very dark shadows). That ended up looking something like this, with the original photo, the mask, and just the orange parts that would be used for the average:

I must confess, I was cheating and trying to get the CSS boxes in the table above to look more orange. This isn’t how our eyes work, and these ended up looking more muted anyways. Maybe because I messed something up in the translation? The average pixel values ended up being very, very similar though, so I ended up just using the magick ones in the table above. Fun to experiment with though!³

This was also a great example of how much easier this experimentation is with LLMs — being able to easily discover tools for cropping or pixel evaluation meant that the time from idea to proof of concept was very, very short.

Even with these lackluster brown average pixels, I’m convinced that the red mesh bags make the oranges look oranger. It’s not big enough to call the FTC for, but it is an effective little trick, a small ripeness deception that we all have to live with.

This post is not affiliated with Sumo Citrus, though the author would be very interested in a sponsorship deal. ↩︎
Stores also do this with green bags for avocados, and others. That’ll be the follow up paper that cites this one. ↩︎
New addition to the resume: “proficient in computer vision” ↩︎

Visualizing my strengths and areas for growth with RPG radar charts

Sat, 12 Apr 2025 12:47:16 -0400

Sometimes I just love a good visualization, and there’s something about radar charts that makes them so approachable. Maybe it’s the asymmetry or the mapping to 2D space, but they give an immediate view across multiple dimensions.

The other day, I was chatting with a colleague of mine about different areas you can grow in as a software engineer, and I was reminded of something I had been thinking about for a while: what would it look like to visualize your strengths and weaknesses with something like a radar chart? With my colleague, I sketched out a rough diagram of what that could look like for a few charts on a whiteboard: six different axes for technical skills, six different axes for non-technical skills, and six different axes for focus areas.

We went through each one together, thinking through where each of us would fall on different axes. How would I rate myself on coding, technical design, mentoring, or writing? How would he rate himself, and how would I rate him?

After we had filled it in and discussed where we thought we were, we were left with pretty good visualizations of our skill sets, where we were strong, and where we could grow more. It was an interesting exercise, and I wanted to think through it a little more here. What makes it a good exercise? Where does it fall short? What could make it even better?

The good part: evaluating my own strengths and areas for growth

If you came to just see my strengths and weaknesses, you’ve come to the right place. I regularly take some time to reflect and think through how I’ve grown and where I can continue to grow, and I’ve organized some recent thoughts into a few different groupings, each with a few different axes that came to mind, visualized in radar charts.¹

Technical skills

For my technical skills, I think I’m pretty good across the board. I would say my strengths lie in debugging, creativity, and MVPs/prototypes. I would say my coding and code review are likely the main areas for growth; for coding, I find myself only wanting to learn just enough of the programming language to solve the problem at hand, and for code review, I could be more diligent about both the depth of the reviews and the latency.

Soft(er) skills

For the softer skills, I think my main strengths are in writing, mentoring, technical talks, and fostering psychological safety. I’m pretty good at leading projects and planning, but I know I can improve at off the cuff discussions (I usually do better with writing or prepared talks) and navigating the subtleties of big company politics. Some of my other main growth areas are networking (like meeting more people across the company and staying in touch with people I’ve worked with in the past, not like TCP) and awareness of the industry at large, much as I may read HN.

The start of the asterisks

There are a few caveats that are worth mentioning here that we can discuss later:

There’s always room to grow, even for the ones I’ve ranked higher.
What’s the scale? What does 30% of the way to the edge mean?
Why did I select those items? What about this other one I’m missing?
How accurate is my own assessment?
What do these mean? Why aren’t you 10/10 at code review?
What am I doing evaluating my technical skills in my free time?

Those are all very valid points. Even with those asterisks, assuming I can trust my own evaluation (more on this later too), this is a neat way to see at a glance where my strengths and weaknesses are, and the visualization makes it clear where there’s room to grow.

How effective is this?

The arbitrary scale

The first thing that comes to mind is the arbitrary scale, like skill bars on a resume. If Bjarne Stroustrup rates himself at 7/10 for C++, where does that leave the rest of us? Zero is easy, but 100% is obviously subjective. Clearly I’m not at 100% for anything, except maybe for self-awareness and knowing I shouldn’t rate myself at 100% for anything.

The critical part is that I know what the scale is (or, if working with someone else, we calibrate what we think these are together). I find the value to be in evaluating the space to grow, and it really doesn’t matter if your end marker is some threshold you want to meet or the highest attainable goal in the universe. There’s no cheating here, only lying, and the important part for each rating is the honest reflection.

A fair evaluation

There’s also the problem of fairly evaluating yourself, particularly since you may not know your own blind spots. I think I’m pretty self-aware, but of course I’d think that. I also find it very helpful to anchor with feedback I’ve gotten from others. While I think I’m pretty good at writing and mentorship, I’ve also gotten very positive feedback about both, which helps me comfortably set those higher.

On the other end of that spectrum, I’m also likely my harshest critic, and I may be ranking some areas unfairly lower. Take my technical abilities, for example, where I’m likely subconsciously comparing myself to extremely gifted engineers that I’ve worked with or seen in the past. Ironically this also seemed like a good spot to cite the Bjarne Stroustrup anecdote, but I hesitated at first, feeling that my own self-evaluation skills must be much worse than his!

Honesty and vulnerability

Both of the above points have an underlying emphasis on honesty and vulnerability, a key part of doing this exercise. On your own, I think it’s a great way to reflect, but as an exercise with someone else, it requires a good level of trust and psychological safety, with that extra mental step to clearly call out your shortcomings. It’s OK to have room to grow, and it should be OK with whoever you’re working with as well.

That does make it feel like it wouldn’t translate well into relationships without trust or with power imbalances. For example, if I was a junior engineer sending out my resume to hiring managers, I wouldn’t want to tell anyone that my coding skills are a 1/10. If I didn’t have psychological safety and was worried that my manager was going to fire me for any displayed weakness, I would be much less comfortable being honest.

I’ve shared my own evaluation because I value transparency and I’m in a very fortunate position to lead by example for psychological safety. I have a strong work history and confidence in my abilities, and really if anyone reads this and doesn’t want to hire me because I didn’t rate my coding abilities as 10/10, it likely wouldn’t have been a good fit anyways!

What do you even include for the axes?

It’s clearly not valuable to put everything up here, and I’ve omitted things like professionalism, patience, or time management. For myself, I can assume those are pretty much all taken care of, though my time management definitely didn’t used to be as good as it is now. For someone else, maybe time management is something they’re actively working on and they should track it for themselves.

There are also things that might just not be worth including. Should I include “computer graphics” if it’s just going to be marked as a zero anyways? Maybe if I’m doing some introspection on my familiarity with different areas in computer science, but maybe not if I’m evaluating my abilities with the lens of my current role.

In case it’s not clear by now, this is also not “Alex’s exhaustive list of the things that every software engineer needs”. These are skills and areas that I’ve noticed in my day to day job that come into play. The ones I’ve included are different for different roles, and they’re not all needed at different companies. They also come with huge asterisks; who cares about office politics really? I don’t care in the grand scheme of things, but I care in the scope of understanding how the game works so I can ensure a coworker of mine gets proper recognition for the important work they’re doing or help a junior colleague get promoted.

Along those lines, I think it’s also an interesting exercise to do this for different categories. I included the ones I’ve done for technical skills and soft skills, with a subset of the axes for each, but there are a few other ones that would be great to work through:

Different services your team manages and interacts with
Different frameworks, technologies, and subareas for your role.
- For a networking role, this could include Wireshark, Netty, HTTP/2, QUIC, security, DDoS attacks, and more
Various focus areas within software engineering
- E.g. databases, data engineering, front-end, mobile development, etc
Various focus areas within computer science
- E.g. compilers, networking, graphics, etc)

Where does the radar chart fall short, and how could it be better?

I think it’s a pretty neat visualization, but there are some clear drawbacks.

One is that there’s some obvious crossover between axes that isn’t represented in these charts. Empathy and communication skills are the building blocks for mentoring, writing, technical talks, leading, and much more. Breaking them up into these more concrete areas makes it easier to think about how to grow them; it’s easier to work with “I want to practice my technical talks” than an amorphous “I want to improve my general communication skills”. But having them as discrete axes loses some of that foundational, crossover effect, where improving at one will likely make you better at another.

Another drawback is the implied relationships between axes on the chart. For one, it appears to give the same weight to two areas that may not have the same importance. I’ve included both “networking” and “mentoring” above on the same graph, but being a good mentor is so much more important to me. Additionally, things next to each other feel like they should be related, even though the axes above are independent.

I think an interesting thought exercise is to think about the order of the categories. Maybe grouping similar items together would allow that crossover effect to be more visible, or maybe you can think about steps in a process. For example, you could look at the different parts of a project and how good you are at the different areas (these are arbitrary values, not necessarily my own evaluation):

With those caveats, I think it’s a neat visualization. Like writing, one of the real benefits of this exercise is the thinking that it forces you to do, particularly as you do an honest evaluation.

There’s no perfect way to do it, but I like these visuals.

Shout out to charts.livegap.com for their free radar graph generator tool that I used for this post. I’m not affiliated, and for full disclosure, I used their site with an adblocker. You can try it out for yourself here. ↩︎

Harry Chapin and RATM: the problem with bleeding interests

Sun, 28 Apr 2024 14:55:20 -0400

Note: this post requires a basic familiarity with the songs discussed. If you’re not familiar with them, I’d recommend you listen to at least a portion of the following three songs:

Free Bird by Lynyrd Skynyrd (YouTube)
Cat’s in the Cradle by Harry Chapin (YouTube)
Killing In The Name by Rage Against The Machine (YouTube)

There are many musical genres I like, and what I listen to usually depends on my mood at the time. Some days, I want to listen to Miles Davis, and some days, I want to listen to Slayer.

Some days, I just want to play some normcore Southern rock. The other day, I opened Spotify, went to Free Bird, and selected “Go to radio”. It gave me a playlist, and after playing on shuffle for a few songs, I realized something was wrong.

What’s the problem here? It’s a little tag at the top of the playlist, just out of sight in that screenshot: “Made for Alex”.

Let me start the rest of this complaint with full acknowledgement that I have nearly zero knowledge of recommendation algorithms. But, to me, this is a problem with personalized recommendations that are over-indexed on your interests. In particular, when these interests are very disparate; I don’t think I would ever expect a human to create a playlist with nearly polar vibe opposites Cat’s in the Cradle and Killing In The Name. Though they can be loosely connected to the song the playlist was based off of, they clearly don’t fit together.

I’ve probably searched for and clicked play on every song in that screenshot, and there’s the problem. They have a clear, explicit signal that these are songs I’m likely to like, and there’s no signal for interest like an explicit signal from the user. But, by focusing on this, this means that the source, this Free Bird track, counts less. Worse still, it means that it’s quick to bleed interests, genres, and vibes, with songs that not only don’t fit, but harshly clash.

It could come back to the philosophical question about the goal of a song radio or playlist, but I don’t think that even fully explains it. If you’re using it as a way to listen to songs you like, this could work, minus the genre clashing. If you’re using it as a way to discover new, similar music, then it really doesn’t work. It leans towards things I already know about, and by prioritizing those, it’s omitting songs I don’t know I don’t know. In this case, I wish there was a separate way to generate these playlists: “ignore everything I’ve ever told you before and just give me songs that are similar to this one”.

In all of my Googling so far, there doesn’t seem to be an easy way to turn this off. I’m also not the only one asking about this.

Spotify’s new daylist sounds like it could be what I’m looking for, as it’s a more “match the vibe” playlist. For example, it says I’ve previously listened to pop art and soundscape on Sunday afternoons, and it created a playlist with “siren, haunting, delicate, soothing, vocal, and songwriter”. All great adjectives, but what if this Sunday I’m driving in Boston traffic instead of doing some delicate vibing?

This is all likely a sign that there are better tools out there for new music discoverability, and I should go discover them too.

In closing, please enjoy:

All of the writing I did in a week as a software engineer

Sat, 18 Nov 2023 11:57:12 -0500

The other day, I was thinking about less-obvious skills that I find helpful for working as a software engineer. Some skills are obvious, like understanding technical topics, learning new things, and thinking like a computer, but there’s a separate class of unobvious skills that an outsider may not immediately think of when they imagine a career in software.¹

Empathy and communication are the main ones that come to mind.² ³

The other day, I was thinking through what communication looks like for my day job. Much of it is talking with other people, be it occasional on-site hallway conversations, 1:1s, meetings, or presentations. A great deal of it is also in writing, especially as I work from home. I have a general feeling that I do a lot of writing, but as an experiment, I thought it would be interesting to look at all of the writing I did in a given week.

Quick caveat that I’m also pretty biased towards writing. I like to write, so I write more, so I get a lot of practice writing, so I get better at the process, so it’s easier for me to write, and I end up liking it more. You can see where I’m going with this.

A day in the life

For a little context on my work, I currently work as a software engineer on the Cloud Gateway team at Netflix. Our team runs the big proxies at the front door — we write services that are like automated switchboard operators, routing your browser’s request for www.netflix.com to the right place. We have a few different proxies, and I’m roughly the tech lead for one of them. My time is split between current work (both as an individual contributor and project lead), forward looking work, partnerships, mentorship, security, incidents, and on-call.

Alongside my regular day job work, I was also on call this last week⁴, which entails answering support questions (our team’s immediate users are internal teams), responding to incidents, and keeping an eye on general software health for our services. This led to a little more writing than usual, as you’ll see.

cat_typing.gif

In the last week, I wrote:

Zero emails

I actually don’t write many emails for work, as most of our async communication is around Slack. This last week, I wrote exactly 0.

A few dozen lines of code

This was a pretty light coding week, mostly because 1) I was on call and 2) I’m currently working on one of those “investigate for a week, then fix with a few lines of code” problems.⁵ This writing included the code, tests, comments, and PR descriptions, as well as comments I left in PR reviews.

A handful of Jira tickets

We loosely track our work items in Jira, and I added a few, mostly summarizing a few future work items. I also provided updates on existing tickets. Not much to write home about here.

A feedback form for the new grad on our team

I’m the mentor for a new grad that joined our team in August, and this week I had to submit a feedback form for their first three months and the new grad program as a whole. This was pretty straightforward, as we’ve had plenty of feedback conversations and much of it was top of mind already.

A couple of investigation & debugging docs

I’m a huge fan of taking notes while debugging. They help me be methodical about my investigations and capture context with what I’ve seen, my train of thought, and what I intend to try next (and why). Though they’re mostly for myself, I find it’s also helpful for sharing that context with others. I especially like showing the thought process for folks who are newer to the kind of deep debugging that we do, just to show one of the ways to dive into a gnarly problem.

This week I wrote ~15 Google doc pages of investigation notes, with the caveat that these included many diagrams that padded my page count. I also usually use the pageless format, since I have lots of images that would otherwise lead to weird blank space. These docs are a lot of things like “Here’s what I’m seeing. My current hypothesis is X for these reasons — what about if we try Y? How does Z work?”. As I mentioned before, my main work this week was spent mostly focused on one gnarly problem in an area I wasn’t previously super familiar with, leading to a lot of investigation.

As part of this, I also put together a few diagrams summarizing the system’s architecture, mostly for my own knowledge. This is tangentially writing — it’s more like writing a name in a box, then putting it in the right place and drawing arrows to its neighbors.

For more on this, see Julia Evans’ masterful zine on debugging.

A few technical docs

I find design documents extremely helpful for thinking through a hard problem and sharing context and ideas with others. I find that if I can’t clearly explain a problem and my solution, I don’t really understand it. Again, this is probably my bias towards writing, but I find it relatively easy to knock out a doc, and the benefit for me greatly outweighs the cost.

There are many kinds of technical docs. This week, I wrote the following:

A forward looking strategy doc for one of our services
A forward looking “what if we built this feature” doc following a conversation with a partner
A rough draft of a design doc to explain exactly what pieces we’ll need for a new integration
A technical design doc exploring a problem we’re seeing and some potential solutions
Minor updates to a previous design doc based on feedback

These added up to ~15 Google doc pages or so total, with some diagrams in there as well.⁶

In this category, I’d also include comments on docs, both for documents I’ve reviewed and comments I’ve responded to on my own docs. It’s a little hard to quantify these, but I’d say in the low tens, ranging from one-liners to paragraphs.

397 Slack messages

Ahh, so this is where my time this week went. We have a Slack-heavy culture, but 400 still seems a lot. The breakdown was as follows:

109 messages in our support channels.
- These are mostly answering questions, explaining things, asking clarifying questions, and sharing metrics and traces. This entails a lot of translating our team-internal things to folks on other teams.
68 messages in our team channel
- We tend to have a lot of async discussions in our team channel. I also err on the side of sharing context with the team, leaving breadcrumbs for anyone’s future searches.
- A healthy amount of these were just emojis, too.
18 messages in our team’s notifications channel, which we use for alerts, CI/CD notifications, and general on-call things
174 direct messages
- I’m pretty sure a lot of this is just pasting links or snippets for people while I’m in 1:1s with them.
- This includes 20 messages in a group chat helping to debug a gnarly issue
- This also includes 10 messages to myself, which I use as a scratchpad for saving things for later
28 channels in miscellaneous channels

Key takeaways

The biggest takeaway is that my split keyboard was a very worthwhile investment.

Other than that, I don’t think there are any, really, other than the fact that I do a lot of thinking and typing. This is just an interesting experiment to see how much writing I actually do in a given week. This week was weighted pretty heavily towards Slack (due to the on-call support) and investigation docs, but it’s not out of the ordinary.

I find writing immensely helpful, and I’ll always recommend it for thinking through a problem and sharing your thoughts with others. If you find yourself wanting to get better at writing, I would absolutely encourage you to do so. Read a book on it, write more, and edit more.

But do get an ergonomic keyboard setup.

It’s like explaining to my mom that even though I write code, I still talk with other humans quite a bit! ↩︎
I’m also of the opinion that the most important part of communication is empathy, but that’s a post for another day. ↩︎
Also maybe stubbornness? But not stubbornness like “I’m right and I have to be right and I won’t listen to you”, but more “I’ve spent two days digging into this slowly making progress and I’m not going to give up” ↩︎
I’m also on call through the rest of the weekend. Please watch Netflix responsibly. ↩︎
Again, stubbornness! ↩︎
This is now motivating me to push for being paid by the page, both the writing and the on-call. ↩︎

Brute-forcing the NYT Digits game

Tue, 11 Jul 2023 22:06:31 -0400

You’re given a list of integers and a target number. You can add, subtract, multiply, or divide numbers, and you can use the result for future operations. Once you have used a number in an operation, you may not use it again. Your task is to find a combination of arithmetic steps to get to the target number.

For example, if you’re given [1, 2, 3] and the target is 9, a valid path would be the following:
Current numbers: [1, 2, 3]
(1 + 2) -> 3, current numbers now [3, 3]
(3 * 3) -> 9, current numbers now [9], target reached
Resulting in the path: [(1 + 2), (3 * 3)]
Given the input [5, 7, 11, 19, 23, 25] and the target number 438, find a path to the target.

Is this your third algorithm interview question in a hellish day of interviewing for a programming job where you won’t really see problems like this?

No!

This is the new New York Times game, Digits:

This game recently came out, and after playing for a while, I got to thinking more about the game.

Part of the game is that you progress through 5 different combinations of input lists and targets. For example, in the above screenshot, you can see that I already solved the 156 puzzle. When I first started playing, I assumed the difficulty would progress as you went through them, with 62 being the easiest and 438 being the hardest.

This got me thinking: this is a very non-linear game, where it can sometimes feel like you make very little progress. How would you define difficulty for these problems?

Digits & difficulty

In the real game, you get 1-3 stars depending on how close you get to the target, with 3 being an exact match. For now, let’s just look at getting exactly to the target. I’m mostly interested in the 3-star solutions, and there are plenty of other areas to add more complexity to if we really wanted to.

The first thing that came to mind is that there must be something related to the number of solutions for each combination of operations. You have to find a combination, not the shortest combination, so as long as you find one combination that gets you to the target number, you get full points. That made me think that the more possible combinations that get you to the target there are, the easier the problem would feel.

For example, let’s say you’re given [1, 2, 3], and your target is 6. Depending on how we do the order and deduplication, there are multiple ways to get there:

2 * 3 -> 6 ✓

1 + 2 -> 3
3 + 3 -> 6 ✓

1 + 3 -> 4
2 + 4 -> 6 ✓

2 + 3 -> 5
5 + 1 -> 6 ✓

But, let’s say you’re given the same input, but your target is 7. There’s only really one path to get there, if we consider 2*3 to be the same as 3*2:

2 * 3 -> 6
6 + 1 -> 7 ✓

Does that mean the first one is easier? I think that could be a rough heuristic, though it doesn’t really take into account the size of the number (with smaller numbers seeming easier) or the complexity of the operations, where it seems easier to just add all of the numbers together (just spamming the + button) than it does to multiple two numbers, multiply two other numbers together, then add the products.

What would this look like for the real problem? How would we go about estimating the size of the solution space?

What would the computer say?

Because I spend a lot of my time trying to convince computers to do what I want, my next thought was “how do I get the computer to do it for me?”

I joked earlier about it being like an algorithm interview problem, but its saving graces were that 1) it was something I was curious about, 2) it wasn’t something I was being asked to do, and 3) there was no cursed relationship between this toy problem and my ability to buy expensive cheese. With these three pillars of playful coding established, I thought a bit about how I’d go about solving this.

Brutal force

My first thought was some weird combination of calculating all of the possible representations of these operations. Doing 2 * 3 -> 6 and 6 + 1 -> 7 really looked like this if you write it out with the order of operations:

(2 * 3) + 1 -> 7

What would all possible paths look like? Maybe we could think about how to programmatically create the string, then feed it into eval and call it a day. We can also swap in an op placeholder for the operators, since we’d potentially have to try all 4. This was getting pretty weird, pretty quickly:

1 op (2 op 3)
(1 op 2) op 3
(1 op 3) op 2
1 op (3 op 2)
... 8 other combinations

Now, some of these combinations wouldn’t be valid. Digits doesn’t let you do 3 / 2 or 2 - 3, as it restricts you to dealing with just positive integers here. But still, this is already getting hairy for just 3 numbers in the input list, and once I started thinking about what it would look like for 6 numbers, it got even worse. Nested parentheses, since you could keep multiplying, and maybe you’d have to try swapping each operand too, since order matters for / and -.

I spent a little time thinking through what this would look like. Maybe some sort of odd graph? Visions of state machines came to mind, some cursed thing like the following with a bunch of conditional transitions:

Gross. We could probably figure something out here, but do you really want to keep track of everything for all of the combinations for six digits?

Recursion and representing the game as a tree

It was around here that I took my dog for a short walk, and I found myself asking the dangerous question: “what if we used recursion?”

If you start with a list of numbers like [1, 2, 3], once you do an operation, you’re really left with the same problem, just with a smaller list. For example,

Current list: [1, 2, 3], target: 7
2 * 3 -> 6

Current list: [1, 6], target: 7
1 + 6 -> 7

Current list: [7], target: 7, target has been found

That’s the happy path, but we can imagine branching at every step for 1) each combination of operands and 2) each operator. It would look like this diagram, but much more complicated — each subproblem would have many more branches off of it for each (operand, operator, operand) combination.

This is actually similar to how you play the game. You start with [1, 2, 3], you add 1 and 3, then you’re left with [4, 2], and you can go from there. There are many paths through this tree with many outcomes that don’t lead to the target number — some are included in light red — but there are paths (at least in this case) that do end up with the target number.

Are we guaranteed to have a path? For the Digits game, I’m sure they always pick a target that you can get to in a number of ways, but mathematically speaking, there’s no guarantee at all. We could have tried 8 as the target above, which would have no solution.

Thinking with trees and recursion made this problem much easier to visualize and translate to a computational solution. Breaking this down, we’ll basically do the following for each subproblem in our function:

Check if the target is in the list of numbers. If it is, we’re done.
Otherwise, go through every pair of numbers.
For each operator for each pair, calculate the new resulting number and call the function recursively with that number in the list instead of the pair.

The pseudocode would look like the following:

operators = ['+', '-', '*', '/']

def check_combo(numbers, target, steps_so_far):
  # Check if we already have the target
  if target in numbers:
    print('Found solution!', steps_so_far)
    return

  # Try each of the combinations of operands with each operator
  for index, number in enumerate(numbers):
    for second_index, second_number in enumerate(numbers):
      # Though not with itself.
      if index == second_index:
        continue

      # Get a list of the other numbers to pass to the recursive call
      other_numbers = numbers.copy()
      other_numbers.remove(number)
      other_numbers.remove(second_number)

      for operator in operators:
        # We can do a basic check here to make sure that
        # the operation is valid
        if not is_valid(number, operator, second_number):
          continue

        # Turn it into '2 * 3'
        operation_string = '{} {} {}'.format(number,
                                           operator,
                                           second_number)
        # Do not ever do this in any code that isn't
        # low-effort, throwaway blog post code.
        new_number = int(eval(operation_string))
        check_combo(other_numbers + [new_number],
                    target,
                    steps_so_far + [operation_string])


# We'll want to ensure that the operations for the given
# operands are valid for the rules of the game.
def is_valid(first_operand, operator, second_operand):
  # No negatives allowed
  if operator == '-':
    return first_operand > second_operand
  # Also, integers only
  if operator == '/':
    return first_operand % second_operand == 0
  return True

check_combo([1, 2, 3], 6, [])

Oh wait, that’s not pseudocode, it’s Python! Running it yields the following:

Found solution! ['1 + 2', '3 + 3']
Found solution! ['1 + 2', '3 + 3']
Found solution! ['1 * 2', '3 * 2']
Found solution! ['1 * 2', '2 * 3']
Found solution! ['1 + 3', '2 + 4']
Found solution! ['1 + 3', '4 + 2']
Found solution! ['1 * 3', '2 * 3']
Found solution! ['1 * 3', '3 * 2']
Found solution! ['2 + 1', '3 + 3']
Found solution! ['2 + 1', '3 + 3']
Found solution! ['2 * 1', '3 * 2']
Found solution! ['2 * 1', '2 * 3']
Found solution! ['2 / 1', '3 * 2']
Found solution! ['2 / 1', '2 * 3']
Found solution! ['2 + 3', '1 + 5']
Found solution! ['2 + 3', '5 + 1']
Found solution! ['2 * 3']
Found solution! ['3 + 1', '2 + 4']
Found solution! ['3 + 1', '4 + 2']
Found solution! ['3 * 1', '2 * 3']
Found solution! ['3 * 1', '3 * 2']
Found solution! ['3 / 1', '2 * 3']
Found solution! ['3 / 1', '3 * 2']
Found solution! ['3 + 2', '1 + 5']
Found solution! ['3 + 2', '5 + 1']
Found solution! ['3 * 2']

These all look like valid solutions, but there are two pretty clear problems. First, some of these operations are redundant. 1 * 2 is technically a valid step before doing 2 * 3, but we can ignore redundant ones to keep our solution set a little crisper. Additionally, a lot of these operations are commutative — 3 + 2 is the same as 2 + 3, so we can add some basic checks for that as well.

With those in place, it may look something like this:

operators = ['+', '-', '*', '/']

def check_combo(numbers, target, steps_so_far):
  # Check if we already have the target
  if target in numbers:
    print('Found solution!', steps_so_far)
    return

  # Keep a set of the commutative operations we've tried already
  checked_commutative_operations = set()

  # Try each of the combinations of operands with each operator
  for index, number in enumerate(numbers):
    for second_index, second_number in enumerate(numbers):
      # Though not with itself.
      if index == second_index:
        continue

      # Get a list of the other numbers to pass to the recursive call
      other_numbers = numbers.copy()
      other_numbers.remove(number)
      other_numbers.remove(second_number)

      for operator in operators:
        # We can do a basic check here to make sure that
        # the operation is valid and not redundant
        if (not is_valid(number, operator, second_number)
            or is_redundant(number, operator, second_number)):
          continue

        # Turn it into '2 * 3'
        operation_string = '{} {} {}'.format(number,
                                           operator,
                                           second_number)
        # Do not ever do this in any code that isn't
        # low-effort, throwaway blog post code.
        new_number = int(eval(operation_string))

        # If it's commutative, see if we've already checked that combo.
        # If so, skip it. If not, add it to the set for future pairs.
        if is_commutative(operator):
          canonical_form = get_canonical_form(number, operator, second_number)
          if canonical_form in checked_commutative_operations:
            continue
          checked_commutative_operations.add(canonical_form)

        # Recursive call
        check_combo(other_numbers + [new_number],
                    target,
                    steps_so_far + [operation_string])


# We'll want to ensure that the operations for the given
# operands are valid for the rules of the game.
def is_valid(first_operand, operator, second_operand):
  # No negatives allowed
  if operator == '-':
    return first_operand > second_operand
  # Also, integers only
  if operator == '/':
    return first_operand % second_operand == 0
  return True


# If we're multiplying or dividing and either operand is a 1, we can just skip it.
def is_redundant(first_operand, operator, second_operand):
  return ((operator == '*' or operator == '/')
          and (first_operand == 1 or second_operand == 1))


def is_commutative(operator):
  return operator == '*' or operator == '+'


# 2 + 3 is the same as 3 + 2, but we can put them
# in the same 'smaller operator larger' form
def get_canonical_form(first_operand, operator, second_operand):
  return '{} {} {}'.format(min(first_operand, second_operand),
                           operator,
                           max(first_operand, second_operand))


check_combo([1, 2, 3], 6, [])

Now, we have a nice, crisp output:

Found solution! ['1 + 2', '3 + 3']
Found solution! ['1 + 3', '2 + 4']
Found solution! ['2 + 3', '1 + 5']
Found solution! ['2 * 3']

That matches what we had before, and just to check for 7, we can do check_combo([1, 2, 3], 7, []), which gives us the single, expected answer:

Found solution! ['2 * 3', '1 + 6']

Trying on the real problem

We can try this on one of the real problems now to see how it does:

Calling check_combo([1, 2, 4, 5, 10, 25], 94, []) results in… a lot:

That’s only a small subset of the answers as the repl was firing away — there are clearly a ton, even after our basic improvements.

Instead of printing them, let’s do a few things:

We can count the total number of solutions
We can also save them by length, and inspect just one of each

I won’t include all of the code here, but the changes would be like this:

operators = ['+', '-', '*', '/']

all_solutions = []
total_checked = 0

def check_combo(numbers, target, steps_so_far):
  global total_checked
  total_checked += 1
  # Check if we already have the target
  if target in numbers:
    # print('Found solution!', steps_so_far)
    all_solutions.append(steps_so_far)
    return


# ... all of the other code, then at the end:

def metrics():
  counts_by_length = {}
  short_solution = None
  for solution in all_solutions:
    solution_length = len(solution)
    if solution_length in counts_by_length:
      counts_by_length[solution_length] += 1
    else:
      counts_by_length[solution_length] = 1

    # Also keep around one of the shortest solutions to see later
    if short_solution is None or solution_length < len(short_solution):
      short_solution = solution

  print('We checked', total_checked, 'subproblems')
  print('We have', len(all_solutions), 'total solutions!')
  print('The number of solutions by number of operations:', counts_by_length)
  print('One of the shortest solutions:', short_solution)


check_combo([1, 2, 4, 5, 10, 25], 94, [])
metrics()

Running this with the metrics yields the following after a bit:

We checked 777019 subproblems
We have 1319 total solutions!
The number of solutions by number of operations: {5: 1119, 4: 194, 3: 6}
One of the shortest solutions: ['1 + 5', '4 * 25', '100 - 6']

No wonder this took a while — we ended up checking hundreds of thousands of subproblems. There were over a thousand ways to get to the target with 5 operations, around two hundred that got there with 194, and just six that got there in 3 operations. I printed out one of those “get it in 3” solutions, and it’s actually a pretty natural one for coming up by hand — usually I try to get somewhere close to the total with multiplication with the bigger numbers, then look for adding/subtracting from there.

That brings up another question: why are we able to solve this as humans, with our puny little brain calculators, without trying every solution? It’s because we use strategy to avoid the full tree. Things like targeting large intermediate numbers that are close to the solution or knowing that if you’re in the thousands already from multiplying big numbers and that there’s no real point going farther than that. It’s interesting to think how we might translate some of these solutions to a programmatic approach, but I’ve left that for another day.

What does the difficulty look like?

I ran this for the rest of today’s problems to see what the solution space looked like for each one.

(obviously, game spoilers below)

checking [2, 3, 5, 10, 15, 25] 195
We checked 1250803 subproblems
We have 1844 total solutions!
The number of solutions by number of operations: {5: 1294, 4: 510, 3: 39, 2: 1}
One of the shortest solutions: ['3 + 10', '15 * 13']

checking [3, 5, 7, 9, 11, 20] 251
We checked 1077566 subproblems
We have 375 total solutions!
The number of solutions by number of operations: {4: 102, 5: 268, 3: 5}
One of the shortest solutions: ['3 * 7', '11 * 21', '20 + 231']

checking [4, 5, 6, 9, 11, 20] 382
We checked 991830 subproblems
We have 106 total solutions!
The number of solutions by number of operations: {5: 87, 4: 19}
One of the shortest solutions: ['4 * 9', '11 * 36', '6 + 396', '402 - 20']

checking [3, 13, 19, 20, 23, 25] 456
We checked 938291 subproblems
We have 989 total solutions!
The number of solutions by number of operations: {5: 916, 4: 73}
One of the shortest solutions: ['3 * 13', '19 * 23', '39 - 20', '437 + 19']

Overall, it seems like they’re trending fewer solutions to the right, save for the very last one. This generally makes sense if we think about our wetware approaches as well, as these big numbers also seem a little more intimidating. I’m much more familiar with my multiples below 15, and I feel like I have much more intuition for the smaller numbers. For the last problem, I think this is where more of the complicated difficulty concepts come into play. There may be more ways to get to 456, but 1) our starter numbers are generally much larger and 2) the larger target feels trickier to get to.

Pretty neat.

Improving this further

Even with these improvements, it’s still a brutally brute-force solution, and each problem takes ~30 seconds when I run it in the repl. We could still probably improve it further — for example, we’re definitely duplicating work, since the code above isn’t looking at commutative across steps (or even if the result of the step is used later in the solution — who cares about an intermediate number if it doesn’t show up in the rest of the steps). The counts are also definitely not a perfect reflection of difficulty, as '6 + 396', '402 - 20' is the same as both '396 - 20', '376 + 6' and '20 - 6', '402 - 14'.

This works though, which meets my bar for fun, exploratory code. I’ll leave further improvements as an exercise for the reader, as this is good enough for now.

Thank you again for attending our onsite interview process. Based on your interview performance, we have chosen to not move you forward in the process. In particular, interviewers mentioned your lack of interest in further optimizing your solutions and your constant mentioning of naps and your desire to go home to play with your dog.

We will keep your resume on file for future openings.

You can play with the code (at your own risk) at https://replit.com/@lxls/JointRedundantSquare, with the caveat that it’s just throwaway blog post code.

The dog hair and the computer mouse

Thu, 13 Apr 2023 08:01:09 -0500

Because I spend my entire workday on the computer, my mouse is very important to me. With my basic optical mouse, I’m far from a connoisseur, but there are a few things that I’m just so used to that I notice any changes immediately. For example, it’s easy for me to notice changes to my mouse acceleration, and even the slightest change in acceleration throws me off. I notice it so much that I have a program on my laptop to disable any mouse acceleration — even the default lowest value isn’t good enough for me.

I’m also very, very far from a hardware expert. My physical object debugging skills include unplugging and plugging it back in, turning it off and back on, and then trying it with different hardware if possible. This is usually enough to solve 99% of the problems I encounter, but I do fear that other 1%.

So you can imagine my dismay when one day, my computer mouse stopped working.

It had been working fine all morning. I had clicked, scrolled, panned, dragged, and opened in a new tab all day without any issue. All of a sudden, it was jittery. The cursor would jump weirdly, sometimes respecting my movements, but sometimes jumping across the screen at random.

I went through the usual steps. Unplug, plug back in. Unplug from USB hub and plug directly in. Switch out the USB-A to USB-C adaptor. Try a different USB port. Quit the mouse acceleration program, just in case, then restart it. Check to make sure I don’t have the mouse on a reflective surface. Try to make do for a bit without the mouse, but then struggle when trying to effectively tab through a new Google search page. Restart the computer, just in case.

After a few unsuccessful minutes of this, I happened to look at the bottom of my mouse.

There was a small dog hair¹ stuck in the little laser area. Optical mice work by shining a light on the surface below and using a sensor to compare how the pattern it sees changes from one instant to another. If the small pattern on your mouse pad has moved a millimeter, it’s likely that the mouse has moved a millimeter, and it can send that signal. But, if there’s a dog hair in the way, there may be some wildly different patterns as the hair is dragged across the front of the sensor. You may just end up with weird, jittery motion.

I removed the hair, and the mouse worked fine.

Facing little problems you’ve seen before

This has since happened a few times, and every time, I’ve (usually correctly) checked for dog hair first. It’s top of mind, and it’s now my first go-to thing when checking².

The other day at work, when walking a new team member through some of our systems, I found myself using the metaphor of “scar tissue” for little problems you’ve seen and built up experience with. I think about this a lot when working with software, as we see similar instances of small problems we’ve seen before. As you learn from seeing something weird, from making a mistake, or from spending time debugging, you slowly build this arsenal of problems you’ve seen before. Most of them are really just nameless echoes of time you’ve spent digging into hairy bugs or things that have burned you in the past.

“Ah, a 502? That probably means something related to security groups when talking to the upstream service, and I’d check there first.”

“For that new feature, it’ll be helpful to add metrics and logging _before_ you may need them.”

“The text fits in the div when it’s in English, but have you tried when the page is in German?”

If you ask me how I know these things, it’s because I’ve already found a hundred dog hairs in a hundred different mice. I’ll probably find hundreds more.

Small dog tax ↩︎
I also furminator my dog quite a bit, but to no avail. He is a scruffy, grumpy, hair-creating machine. ↩︎

Things that are hard to measure, things that are easy to measure

Sun, 26 Feb 2023 15:29:21 -0500

There are things that are hard to measure, and there are things that are easy to measure. I think about this duality a lot when it comes to software engineering, especially when it comes to the difficult task of measuring developer productivity and engineer effectiveness¹.

Taking a very naive approach, there are many easily measurable aspects of the job that you could use²:

Jira tickets closed
Number of pull requests submitted (or reviewed)
Number of lines of code
Number of interviews given
Number of design documents written
Features shipped
Money coming in because of those features (if there’s a direct connection)
Time engineer spends at the office

What about the parts of the job that are harder to measure?

The quality of those PRs, PR reviews, lines of codes, or interviews
How well those features are built
Mentoring and helping other engineers grow
Giving technical talks and spreading knowledge
Reducing technical debt
Deciding to not build features (when appropriate)
The monetary value of engineering that’s not directly connected to revenue
The knowledge accumulated by building an initial feature that you can use as a foundation for future work
Even just the value of two team members just sitting down for coffee

Some of these can be correlated to things across the divide. If you’re an effective engineer shipping new features, you may write a fair bit of code. If you’re able to effectively build consensus about solving a difficult problem with cross team buy-in, you may end up writing a few design documents. If you’re able to narrow down your team’s focus to just those features the team can (and should) build, you’ll likely be able to better ship those features.

But, this positive relationship doesn’t hold up in the other direction, and that’s the danger. The number of lines of code an engineer writes is not indicative of the value of that code — we’re in the business of elegant solutions to complex problems after all. You can’t build consensus by sheer volume of documents and meetings alone, unless you’re hoping that the other party resigns out of fatigue by the time the decision has to be made. If you do a bunch of interviews, is it valuable if you’re a jerk in those interviews?

On the other hand, does the negative relationship hold up in the other direction? Well, it’s nuanced. Maybe the project was exploration-heavy, and that didn’t translate to many PRs. Maybe it was a critical one line fix, but it took a month of intensive investigation to find it. Or, maybe I really do have a second remote job, and I’ve been too busy with it this quarter to get to all of my Jira tickets³.

It’s not possible to measure the full picture by just the easily measured metrics alone. Doing so ignores the real depth of the job.

More so, if 1) you’re just using the easy ones and 2) the way you evaluate is visible to the engineers under review⁴, they’re going to game it. How could they not? If they’re being evaluated on the number of new features they ship and this evaluation has a direct relationship to the money they receive or their career trajectory, of course they’re going to optimize their time and effort towards the new features. It’s your fault for hiring a bunch of smart cookies!

So how do you measure how effective an engineer is? This is where I’m going to take the easy way out — that’s well beyond the scope of this short post, written by a lowly engineer. All that I ask is that if you’re in the position where you’re evaluating engineers, you look beyond the easy-to-measure metrics, as there’s so much more to the work. It’s hard, but this is the way.

Though I’m not convinced it’s a decidable problem. ↩︎
Quick caveat that neither of these are exhaustive lists. ↩︎
Quick caveat that I don’t have a second remote job. ↩︎
As it should be. ↩︎

In the desert, no service, and your partner's late to the rendezvous. What now?

Thu, 16 Feb 2023 22:50:02 -0500

Let’s say, purely hypothetically, that you’re in Joshua Tree National Park. Your partner dropped you off in the morning, and you’ve spent the day hiking and scrambling in pretty populated areas of the park. You arranged to meet your partner at 4:45PM at the place where they dropped you off. You’re currently in possession of a cell phone (though without any service), some snacks and water left over from your day, a notebook, and a pen. You’re 7 miles from the entrance of the park, and there’s a decent number of other visitors around.

You’re standing at the meeting spot, and you glance at your watch. It’s 5:02. What now?

At that moment, you realize how much you’ve taken cell phone service for granted. You also realize you hadn’t discussed any contingency plans in case anything goes wrong. You had let them know which rocks you’d be scrambling on, but you hadn’t discussed worst case scenarios in case they couldn’t get to you.

The good news is you have options. The bad news is that none of them are great.

Do you wait indefinitely? You don’t really have enough gear to spend the night comfortably, though you’ve got plenty of warm layers thanks to a layering habit from growing up in a colder climate. You’d also likely want to hitchhike out before it gets too dark, especially as your likelihood of being picked up for a ride will probably drop as it gets darker — the long beard probably won’t help. How long do you wait? Maybe another 30 minutes? If it’s something calamitous and they’re at urgent care or something, thirty minutes won’t make much of a difference. But what if you hitchhike but then pass them on the way out? Do you ask your kind ride-sharer to drop you off, then you walk back to the spot? Maybe we pair that with some frantic waving?

Ah, we have our notebook. Maybe we can leave a note under a rock on a little signpost here. Not exactly leaving no trace though, and it could easily blow away (or be picked up as litter by another visitor). They’re not very big pages though, and would they even see it? Maybe it’s worth it anyways, and when you do meet up with your partner, you can come back and pick it up.

What if you hitchhike out but miss them as you pass them? Then you’d be at the park entrance, and they’d be waiting at the meeting point. Their not being there on time brings to mind images of a flat tire, but you not being there an hour after the meeting time hints at a broken leg. Would they try to walk out to the rocks you told them you’d be near? You remember that you have the newly purchased family headlamp, and it’s probably not great for them to be out there with just their phone flashlights looking for you, especially if you’re not even out there. Maybe you should leave that note.

This feels like an interview problem. “What’s the best strategy to ensure you rendezvous with your partner?” The clear answer is that you arrange a plan if things go wrong ahead of time and stick to that plan, but that’s not what they want to hear. They just want to see how you think through a problem, though it would be nice if it compiled and passed some tests at the end.

Great.

It does all feel very game theory-ish, though. If I had to assign things values, I’d say that the worst case would be them spending the night out here looking for me. Let’s call that value -99. If I had to spend the night, it wouldn’t be terrible, but it would be uncomfortable and they’d worry about me. Let’s call that -5. Walking 7 miles in the waning light is also unideal, especially since I don’t have that much water left. Let’s say that’s an additional -3. Leaving the note is a slight hit to the ethic, so let’s call that -0.5.

So the worst case would be if I walked 7 miles, still slept out here, and they missed me and spent the night searching, for a whopping value of -107. Let’s leave that note, though it pains me. Maybe what I can do is hitchhike out to the entrance (skipping the 7 mile walk) while keeping a very diligent eye out for their rental car — I think it was a gray Toyota Corolla — and I wave out the window or something to catch their eye. I hope I can see their car in the evening light.

If I make it to the park entrance and still haven’t seen them, at least I’ll have cell service and can call them, and if I still can’t reach them, I’ll hope that they’re back at the rental and not out looking for me (hopefully they saw the note in the dark). It’s getting pretty cold, and I’m not sure they would have brought their heavier jacket to pick me up. But what action do I think they’ll take?

You glance at your watch. It’s now 5:07. As you look up, you see your partner’s car pull into the parking lot.

They were just a little late.

Practice being a beginner

Tue, 01 Nov 2022 20:28:03 -0400

I’m a big fan of trying new things. Part of the process that I’ve come to enjoy is facing and getting over my initial hesitations and reaffirming that it’s OK to be bad at things.

Trying something new for the first time can be scary. There’s an element of vulnerability, where you’re clearly showing that you’re not good at something. It’s like I’m going to show up at the rock climbing gym and they’re all going to judge me for being the out of shape noob.

“Who’s dad is this?”

Realistically though, that’s not the case. It’s so OK to not be good, and it’s an obvious part of learning and improving. It’s even a little silly to explicitly write that out — of course every expert was a beginner at some point!

That’s not what we commonly see though, and I easily fall into this trap as well. It’s part of the curse of social media where the highlights reign. We generally don’t see the hours of practice that went into the minute-long video of the virtuoso guitar player. We don’t see the many, many hours of prep and focus that let the chess grandmaster play so well. The Beatles showed up in storm, without a trace of their leather-jacket marathon playing Hamburg days where they practiced their craft for hours a day.

It’s kind of like learning a foreign language and getting over your hesitations about talking with a native speaker. There’s the trope of someone claiming they speak their second language better after a few drinks — I’ve certainly heard (and maybe claimed) this a few times. My theory is that it’s because that feeling of hesitation and embarrassment slips away, and your mind is free to focus on communication, not the anxiety of what the other people will think of you if you mispronounce or get a word wrong.

I think it’s a worthwhile thing to practice. I find it too easy to stick to things that I’m already pretty good at, where I feel comfortable and skilled. It’s like a well-traveled rut of expertise, since I can stay in my comfort zone, maybe slinging a new coding project or taking a new class. I don’t think there’s anything wrong with either that or building expertise, but for me, there’s so much more out there, and there are so many things that I’ve come to enjoy after giving them a try. This has been especially pertinent for me as I consciously try to move away from the grind and a focus on work/career.

These new things double as humbling exercises, too, if you ever find yourself feeling a little too high and mighty. Go try something new. Try rock climbing, then nurse your sore muscles the next day and appreciate how damn good the pros are. Put on a helmet (and maybe pads, if it’s truly your first time) and try rolling around slowly on a skateboard, then go watch some skate videos from the 90s to see what they were mastering over 30 years ago.

I may not be very good yet, but I certainly have a lot of fun doing it.

Learning by working on problems just outside of your reach

Sun, 02 Oct 2022 21:51:56 -0400

There’s a common pattern I’ve seen when learning something new, where you can effectively move towards a larger goal by working on problems just outside of your reach. I did a little thinking about what that would look like and how to find the right steps to take.

Circles of knowledge: what I do and do not know

Let’s say you have some general knowledge in some areas. I sometimes think about knowledge and “things you know and understand” as some sort of multidimensional blob that can grow, contract, and connect in many different ways as you learn and practice. For the sake of easy visualization, we can think of it as a circle of knowledge, dividing everything into two categories: things you know and things you don’t know.

My knowledge is a weird mix of things I’ve picked up over the last few decades. There are some things I know well, like the names of all 50 states in alphabetical order (thanks to the song), but there are many things that only partially fall within my circle with much of their depth unknown to me.

There are so many things I don’t know, and there are frequently things that I want to learn. Maybe it’s something relatively small, like a new JS web API I’m curious about, but maybe it’s something big and amorphous, like “distributed systems”. If we think about this boundary, where the rubber of the new thing meets the road of what I already know, we can imagine that it’s just outside this circle.

Let’s take distributed systems, for example, where I may have already known some parts of a few languages, some general programming concepts, and some networking concepts. Staring me in the face is this intimidating box, “distributed systems”.

In reality, this broad “distributed systems” is made up of many things, including concepts around distributed computation, RPCs, fault tolerance, consensus, debugging, and many things off the screen to the bottom.

You don’t learn the big box at once. You work your way towards the goal, slowly expanding your new beachhead of knowledge by working problems always just outside of your reach. Much like the race to the edge of human knowledge in the illustrated guide to a Ph.D., it’s like continuously adding a single additional piece of unknown as you take steps towards the goal. Maybe you start with MapReduce to learn more about distributed computation and ideas around splitting up work.

Suddenly, your pretty circle of knowledge has a blemish where you’ve expanded out into the unknown. The border of the known has shifted a little, and picking off RPCs grows it even further:

Bumpy! You’ve picked up a few more things, the bumpy surface of your understanding now including a few more things, and you’re slowly building towards that broader goal. What you learn builds off of what you’ve already learned, and you grow your knowledge by picking off concepts and ideas.¹

And just like that, suddenly you’ve got pretty good coverage across this broad area. Of course, it isn’t a slowly expanding circle, and these things rarely are fully in your circle of knowledge. It’s some sort of amorphous blob of connections, strengthened understanding, and jutting things you don’t know. Everything relates to everything else, especially in programming, and everything is so complex.

The concepts above are also pretty interchangeable in their order, but that’s not always the case for working your way through. As an exercise left to the reader, think about what this would look like for learning a foreign language, ramping up a new engineer on a code base, or learning a new programming language.

Working on problems just outside of reach

One of the most helpful things about this is that it gives more immediate and appropriately scoped goals; as a junior engineer new to a codebase, I shouldn’t be trying to learn the whole codebase in one go, but instead I should learn parts of it by fixing bugs and adding features with the scope increasing as I understand more and more. Someone new to programming shouldn’t be setting out to “learn programming”, but should instead start with small programs, like printing the sum of the first ten integers, then spend the next ten years working on more complicated projects. I find it also helps to give a guided path towards these goals, especially if you’re able to start your first steps with fundamentals that later steps will build off of, much like you probably spent a lot of time doing basic arithmetic before learning algebra.

How do you find those right steps to take? This is where guidance from a third party can be tremendously helpful, as appropriate scoping for steps can really help with your movement towards the eventual goal.² Books are nice, as they frequently give a path and steps for you to follow. Better yet is if you have someone that knows the path well and is able to guide you along it, crafting those steps for your current knowledge and checking to see where you are. This is where classes are great, as they (hopefully) offer a nice balance between a well-trod path through a subject and a helping hand from the professor and/or TAs to guide you through it.

It’s one of the things I really enjoy about mentorship, where you can help create these paths and guide people along them. It’s one of the things I most enjoyed about my previous life working as a tutor, long before I got into tech. It was a game of figuring out where the student’s understanding was and what the best way to get them to the next step would be. It would be a constant game of empathy and translation, both seeing where they were and exploring the best way to map the subject material to something that they would understand and be able to explore themselves. This is where I think the personal touch really matters, as the single path presented in the textbook may not be the one that makes the most sense to your brain and where your current understanding is.

It’s a very tricky problem, but it’s a wonderful one to solve.

Outside the scope of this section is the best way to learn new things. I’d recommend a mix of reading, building, experimentation, and talking to smart people. It’s really up to you to discover what works best for yourself! ↩︎
Of course, you can always figure it out by yourself as you go. I’d recommend at least some guidance from someone with experience on the path, since it can be so hard without knowing what you don’t know. ↩︎

iPhones and action discoverability, or "How the hell was I supposed to know that?"

Sat, 24 Sep 2022 11:26:15 -0400

Discoverability

I spend a lot of my time thinking about UX and design, and I try to consciously think about it in the wild when I can. One of the things that always catches my eyes is the idea of discoverability, or how a user is able to discover that action X does some thing Y.

A basic way to do so is to rely on patterns that we’re all familiar with. For example, this link is blue, and if you hover over it with your mouse, you can see the cursor change. That is a helpful context clue, since you’ve probably seen and interacted with many links before, and because it has the same pattern, you can expect it to do the same thing.

Sometimes there are things that are new to you but are still intuitive when you first explore them, like using a touch screen for the first time. We touch things all the time, and when you first used a good touch screen, you were probably able to quickly understand how it worked. You moved your finger, the page scrolled, and that feedback helped you to quickly solidify the connection between your gestures and the screen’s actions. This is like discoverability through action, where you’re able to figure it out as you try it.

I also think about the category of things that aren’t intuitive and require some guidance, like Googling steps to show a cursor in an iOS screencast or putting together IKEA furniture. Woe is the fool who tries to discover a complicated IKEA piece through action! (Side note: there should be competitive IKEA furniture speed runs). This discoverability is aided by a guide, and although the quality of these guides can vary, sometimes it is helpful to first RTFM.

What iOS is doing these days

This brings us to a weird area: what about things that are unintuitive and unguided? Or, as I like to say, how the hell was I supposed to know it would do that?¹

I’d like to present the following features for your consideration, with an exaggerated pointer to show the interactive actions:

Deleting a number on the iOS calculator

To delete the last digit from a number you entered in the iOS calculator, you can swipe the numbers to delete.

To better navigate around a text document, you can long press the spacebar, then navigate around.

Switching between Safari pages by swiping the URL bar

To go back and forward in Safari, you can swipe the edges of the screen. To switch between tabs, you can swipe the URL bar left and right.

Going back after opening a new Safari tab

This is more of an unintuitive nitpick than a feature, but when you swipe back from a new tab, you silently close that tab and return to the old tab, unless you’ve already closed the old tab.

Undiscoverability

I don’t think any of these are intuitive or easily discoverable. You’ll sometimes see these features pop up in life hack or “I never knew this hidden feature existed” tweets, and that novelty and surprise is indicative of the lack of discoverability for these features. They’re at an odd intersection of 1) not following previous expected patterns, 2) not being intuitive, and 3) not being clearly explained anywhere, unless you happen to stumble upon them.

This is where you frequently see some cliché comments like “Steve Jobs would fire the people who made these”. Yeah maybe, but who am I to make that claim, much less to argue that it would be a good thing.

Good design for very complicated systems is hard, and these are just a few points where it fell short. It ends up being a double edged sword as well, because the complicated features (that must have taken many hours to build, debug, test, and launch) are hard to discover — how many users end up swiping their calculator or using the spacebar navigation? I’d love to see those metrics.

As much as I like to point out design shortcomings, it’s always funny to me to think about the number of very, very complicated things we take for granted now, with things only popping up as notable when they aren’t perfect. It’s like modern front ends — yes it’s complicated, but the expectations are very high as well. ↩︎

The hobby buyer's fallacy: "I'd really pursue my hobby if I just bought X"

Sat, 24 Sep 2022 10:13:28 -0400

There’s a trap I used to fall into that has happened more times than I’d like to admit¹. The typical scenario would be something like this:

There’s a hobby I’m into (or starting to get into)
I find myself wanting to do that hobby more than I currently am
I find some THING that seems like it would help
I somehow convince myself that having that THING would make the difference
I purchase that THING
The THING only slightly helps, and I may or may not do the hobby more

Much like the scroll in Kung Fu Panda, the secret was really inside myself the whole time. But, as I mentioned, I’ve fallen into this a few times before:

“If I only had the perfect notebooks, I would write much more by hand”

“If I had that drawing board, it would make it that much easier to practice drawing and I would draw so much more”

“The only thing standing between me and painting is those new paint pens”

As you can imagine, the new shiny things were not the only things standing between me and doing my hobbies.

It’s so tempting, though. Maybe that thing is what the pros use, and of course it would make sense to upgrade my equipment if I want to get better (spoiler: it isn’t the shoes). Maybe I’ve been more tired and burnt out recently without the energy to sit down and do my hobbies and I fool myself into thinking this new drawing board would make the difference. Maybe it’s only $40 and Amazon will deliver it tomorrow.

At some point I started to see the pattern, and I started fooling myself less and less and built up defenses as I became more aware of it. I have a “nice to have” shopping list I add to when I have that spur-of-the-moment feeling — if it would really be helpful, it’ll still be there in a month. I think about whether I’m fooling myself with this old pattern, and having fallen into the trap a few times, I find it easier to take a moment to think about whether I’m starting to think like this. Even recognizing the pattern has helped me avoid it, like I’ve given it a name and can now point to it, as if it was a gremlin in the back of my head whispering foolish thoughts.

Maybe the gremlin’s name is Purchasa. No, Purchasa, the more expensive guitar won’t make me play more or play better.

I’ve also accepted that I won’t be 100% into all of my hobbies at the same time, and it’s OK if my interest in something ebbs in favor of other things. If I’m not pursuing any of my hobbies, there’s probably something else going on that I need to take a look at, but if it’s just that I’ve been playing more guitar than piano recently, that’s totally fine. There just isn’t enough time for everything.

For new hobbies, there are little gut checks that help as well, like “try without it for a month and see if you’re still doing your new hobby, then re-decide from there”. Maybe we buy a single month pass at the climbing gym to make sure we really like it before we buy the full year. Maybe the one-month pass is a little more expensive, but it’s like a “don’t fool yourself” fee that frequently ends up more than worth it.

There are a few things that don’t fall under this, though. You really do need a skateboard to skateboard, though bonus points if you can borrow from a friend. If you’re going to be spending a lot of time in front of your computer for your hobby, a good ergonomic setup is worth it. There are points at which an equipment upgrade actually makes a real difference, but when I’m honest with myself, I’m usually far from that point. Until we get there, let’s stick with what we’ve got.

I still need practice to get to Carnegie Hall, and buying that thing probably won’t be the difference.

Quick caveat that this is by far one of the most first world problems to have — I’m lucky to be blessed with such cushy problems. ↩︎

Stubbornness and programming

Sun, 11 Sep 2022 16:28:14 -0400

Every now and then, there will be the type of problem to solve where I know it will require an extra cup of coffee, a long block of dedicated time, and a lot of careful reading, debugging, note taking, and going for walks to think.

This is often “subtle bugs that are hard to repro but we need to fix” territory; it’s not fun greenfield development with a lot of brand new code (which, of course, would be a problem-free fresh start…), but instead, probably a small PR with only a few lines changed that completely undersells the deep investigation required.

It’s like the old joke: $5 for the hammer and $95 for knowing where to hit the machine with the hammer.

There’s some sort of stubbornness that helps with these. Not stubborn like “I’m always right and won’t listen to your opinion”, but stubborn like “it’s hour 3 of debugging and I’m slowly getting closer and I still want to keep digging until I find that needle in the haystack”. Persistence, maybe? Focus? Perseverance? Motivation from a quick glance at your paycheck? Whatever it is, it’s some amount of ability to stick with it and keep digging — whatever gets you slowly closer to a solution, until finally you shine some light on that small subtlety that was out of place.

It’s the computers that are the obstinate ones, inflexibly doing exactly what you tell them to do. They’re the ones not bending; in order to work together, it’s our job to take our human minds and make ourselves think the way the computers think. It’s that weird layer between our abstract and the computer’s specificity, mentally translating what should happen into those careful instructions for the machine to follow.

Very rude of them, when you think about it.

This kind of perseverance can be very helpful when working with computers. You build it up naturally over time, starting with your own little programs, then larger ones, then ones other people wrote, then maybe even ones other people wrote over many years and ones that work across many computers. You can get better at it too, much like anything else, with practice, focus on the process, and some metacognition as you think about how your brain works as you go through the process. At the risk of recommending Julia Evans’ zines and site in every post, I highly recommend her Some ways to get better at debugging.

I wonder if there’s some room for more focused learning around debugging. Imagine something like a class-like program where you’re given a code base that has a number of bugs in it. Much like a developer ramping up on a project, you could go through the ramp-up process, starting by fixing a few trivial starter bugs, then working on larger and larger bugs. One of the things I didn’t see too much of in the CS classes I’ve taken is the experience of working through other peoples’ code, and I wonder if something like this would be a good way to build those skills (or at least to give some preview of the nitty gritty you’ll get to deal with on the job).

Another interesting shift I’ve seen recently is some amount of focus on debugging and working with realistic scenarios as part of the job interview process. It seems like it’s much harder to quantify, but given that it’s such a big part of what we do, it’s an interesting area to explore.

Writing a toy WebSocket server from scratch

Sun, 28 Aug 2022 12:06:08 -0400

WebSocket (WS) connections are pretty neat. I’ve gotten to play around with them before, both for personal and professional projects, but my view has mostly been at a high level and concerned mainly with the things that I can build on top of them. I haven’t spent much time digging through the protocol.

I thought it would be pretty interesting to build a toy WS server from scratch. Like many other projects, I find building a toy system to be such a good way to learn more, and I’ll be talking about this toy WS server (and some of those things I learned) in the rest of the post.

WebSocket

The main idea behind the WebSocket protocol is bi-directional communication. Rather than the usual request/response lifecycle, with a client asking a server for resources, WebSockets allow for messages to be sent in either direction. Think of a chat application, where the client should be notified by the server that there’s a new message for them. You could have the client long poll, continuously asking the server if there’s anything new, but with a WebSocket connection, the server can independently send those new messages to the client, allowing the client to passively receive when it needs to.

The WebSocket connection flow

WebSocket connections are initiated via a regular HTTP request from the client that indicates it would like to switch, or upgrade, to talking via a WebSocket connection. It’s kind of like sending someone a Slack message asking if they want to hop on a quick video call — it’s still communicating over the computer, and you’re using one method to initiate the next.

The steps to the flow are the following:

Client sends an HTTP request to the server indicating it would like to switch to WS
Server responds saying it’s OK to switch to WS
Client and server can start sending WS frames back and forth

The real steps are of course a little more complicated, including the origin, what and how to upgrade, protocols, extensions, and a few other pieces of data. A typical handshake request may look like this:

GET /chat HTTP/1.1
Host: server.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Origin: http://example.com
Sec-WebSocket-Protocol: chat, superchat
Sec-WebSocket-Version: 13

And a typical response:

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

If you’re interested in more details, I’d recommend reading through RFC6455 — it’s not too bad to dig through!

What’s this Sec-WebSocket-* business? It’s one of the ways that the server can prove to the client that it received the handshake and it’s responding to that request. The server does so by hashing the key with a well-known magic string, 258EAFA5-E914-47DA-95CA-C5AB0DC85B11, then base64 encoding it and sending it back as the handshake response. If you’re curious, you can see section 1.3 for more details, and I’ll have an example in the later code as well.

This means the full conversation looks a little more like this:

A toy websocket server

I started with a basic idea: what’s the smallest server we can build that can handle an incoming websocket message? For example, what would it take to handle the following, which is the most basic way to initiate WS communication via JavaScript:

const exampleWS = new WebSocket("ws://localhost:5006/websocket");
exampleWS.send("foo");

These two little lines are deceiving; there’s a lot going on under the hood.

First, for this toy project, it’s helpful for myself to define two things:

Goal: write a basic server that can handle the WS handshake and parse an incoming WS frame to see the message sent from the client

Non-goals: writing a robust or real HTTP server, writing a fully compliant WS server, or handling all edge cases

With that in mind, the steps will look something along these lines:

Make a very basic HTTP server to handle incoming HTTP requests
Handle the initial handshake request and the connection upgrade
Actually parse the incoming WS frames

Starting with HTTP

We can start by writing a very basic HTTP server that can handle incoming requests. This example is very much inspired by Beej’s Guide to Network Programming¹, and the flow is the following:

Have one socket that listens for incoming connections
When there’s a new connection on it, create a new socket for listening to it
When the new sockets are ready to read from, handle the incoming requests

This isn’t very exciting, but we can roll a very rough server in a big handful of lines:

"""
Very basic HTTP server that just listens on a port and responds.
"""
import socket
import select
TCP_IP = '127.0.0.1'
TCP_PORT = 5006
BUFFER_SIZE = 1024 * 1024

DEFAULT_HTTP_RESPONSE = (
    b'''<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">\r\n
<TITLE>200 OK</TITLE></HEAD><BODY>\r\n
<H1>200 OK</H1>\r\n
Welcome to the default.\r\n
</BODY></HTML>\r\n\r\n''')


def main():
    '''
    Creates the front-door TCP socket and listens for connections.
    '''
    # We can have a main socket that listens for initial connections.
    tcp_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    tcp_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    tcp_socket.bind((TCP_IP, TCP_PORT))
    tcp_socket.listen(1)
    print('Listening on port: ', TCP_PORT)
    input_sockets = [tcp_socket]
    output_sockets = []  # Maybe use, we'll see
    xlist = []  # Not using
    while True:
        # Get the sockets that are ready to read (the first of the
        # three-tuple).
        readable_sockets = select.select(input_sockets,
                                         output_sockets,
                                         xlist,
                                         5)[0]
        for ready_socket in readable_sockets:
            # Make sure it's not already closed
            if (ready_socket.fileno() == -1):
                continue
            if ready_socket == tcp_socket:
                print('Handling main door socket')
                handle_new_connection(tcp_socket, input_sockets)
            else:
                print('Handling regular socket read')
                handle_request(ready_socket, input_sockets)


def handle_new_connection(main_door_socket, input_sockets):
    # When we get a connection on the main socket, we want to accept a new
    # connection and add it to our input socket list. When we loop back around,
    # that socket will be ready to read from.
    client_socket, client_addr = main_door_socket.accept()
    print('New socket', client_socket.fileno(), 'from address:', client_addr)
    input_sockets.append(client_socket)


def handle_request(client_socket, input_sockets):
    print('Handling request from client socket:', client_socket.fileno())
    message = ''
    # Very naive approach: read until we find the last blank line
    while True:
        data_in_bytes = client_socket.recv(BUFFER_SIZE)
        # Connection on client side has closed.
        if len(data_in_bytes) == 0:
            close_socket(client_socket, input_sockets)
            input_sockets.remove(client_socket)
            client_socket.close()
            return
        message_segment = data_in_bytes.decode()
        message += message_segment
        if (len(message) > 4 and message_segment[-4:] == '\r\n\r\n'):
            break
    print('Received message:')
    print(message)

    (method, target, http_version, headers_map) = parse_request(message)

    print('method, target, http_version:', method, target, http_version)
    print('headers:')
    print(headers_map)

    # For now, just return a 200. Should probably return length too, eh
    client_socket.send(b'HTTP/1.1 200 OK\r\n\r\n' + DEFAULT_HTTP_RESPONSE)
    close_socket(client_socket, input_sockets)


# Parses the first line and headers from the request.
def parse_request(request):
    headers_map = {}
    # Assume headers and body are split by '\r\n\r\n' and we always have them.
    # Also assume all headers end with'\r\n'.
    # Also assume it starts with the method.
    split_request = request.split('\r\n\r\n')[0].split('\r\n')
    [method, target, http_version] = split_request[0].split(' ')
    headers = split_request[1:]
    for header_entry in headers:
        [header, value] = header_entry.split(': ')
        # Headers are case insensitive, so we can just keep track in lowercase.
        headers_map[header.lower()] = value
    return (method, target, http_version, headers_map)


def close_socket(client_socket, input_sockets):
    input_sockets.remove(client_socket)
    client_socket.close()
    return


if __name__ == '__main__':
    main()

Not very robust, but life is short and toy projects are for fun! What does that look like from the client side?

~: $ curl -v 127.0.0.1:5006
*   Trying 127.0.0.1:5006...
* Connected to 127.0.0.1 (127.0.0.1) port 5006 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:5006
> User-Agent: curl/7.77.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
* no chunk, no close, no size. Assume close to signal end
<
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>200 OK</TITLE></HEAD><BODY>
<H1>200 OK</H1>
Welcome to the default.
</BODY></HTML>

* Closing connection 0

Nice. And on the server?

websockets-from-scratch: $ python3 foo.py
Listening on port:  5006
Handling main door socket
New socket 4 from address: ('127.0.0.1', 60294)
Handling regular socket read
Handling request from client socket: 4
Received message:
GET / HTTP/1.1
Host: 127.0.0.1:5006
User-Agent: curl/7.77.0
Accept: */*


method, target, http_version: GET / HTTP/1.1
headers:
{'host': '127.0.0.1:5006', 'user-agent': 'curl/7.77.0', 'accept': '*/*'}

Nice! This is basically all the HTTP we need, since we’ll be responding to a basic GET request and parsing the headers, so no need to get fancier.

Handling the initial handshake request

As I mentioned before, the initial handshake includes a few indicators that the client would like to switch over to talking over WebSocket. We can do a few basic checks to see if an incoming request is something along those lines, starting with a check to see if it’s a valid request:

def is_valid_ws_handshake_request(method, target, http_version, headers_map):
    # There are a few things to verify to see if it's a valid WS handshake.
    # First, the method must be get.
    is_get = method == 'GET'
    # HTTP version must be >= 1.1. We can do a really naive check.
    http_version_number = float(http_version.split('/')[1])
    http_version_enough = http_version_number >= 1.1
    # Finally, we should have the right headers. This is a subset of what we'd
    # really want to check, but it works for now!
    headers_valid = (
        ('upgrade' in headers_map and
         headers_map.get('upgrade') == 'websocket') and
        ('connection' in headers_map and
         headers_map.get('connection') == 'Upgrade') and
        ('sec-websocket-key' in headers_map)
    )
    return (is_get and http_version_enough and headers_valid)

If it is a valid handshake request, we’ll need to do that trick with the incoming Sec-WebSocket-Key header, where we concatenate it with the Magic String, sha1 hash it, and base64 encode it before returning it as the Sec-WebSocket-Accept header.

    ...
MAGIC_WEBSOCKET_UUID_STRING = '258EAFA5-E914-47DA-95CA-C5AB0DC85B11'
    ...

def generate_sec_websocket_accept(sec_websocket_key):
    # We generate the accept key by concatenating the sec-websocket-key
    # and the magic string, Sha1 hashing it, and base64 encoding it.
    # See https://datatracker.ietf.org/doc/html/rfc6455#page-7
    combined = sec_websocket_key + MAGIC_WEBSOCKET_UUID_STRING
    hashed_combined_string = hashlib.sha1(combined.encode())
    encoded = base64.b64encode(hashed_combined_string.digest())
    return encoded

Finally, we just need to tie this together with a check on the target and a handler for the handshake that returns the appropriate response:

    ...
WS_ENDPOINT = '/websocket'
    ...

def handle_request(client_socket, input_sockets, ws_sockets):
    print('Handling request from client socket:', client_socket.fileno())
    message = ''
    # Very naive approach: read until we find the last blank line
    while True:
        data_in_bytes = client_socket.recv(BUFFER_SIZE)
        # Connection on client side has closed.
        if len(data_in_bytes) == 0:
            close_socket(client_socket, input_sockets, ws_sockets)
            return
        message_segment = data_in_bytes.decode()
        message += message_segment
        if (len(message) > 4 and message_segment[-4:] == '\r\n\r\n'):
            break
    print('Received message:')
    print(message)
    (method, target, http_version, headers_map) = parse_request(message)
    print('method, target, http_version:', method, target, http_version)
    print('headers:')
    print(headers_map)
    # We will know it's a websockets request if the handshake request is
    # present.
    if target == WS_ENDPOINT:
        print('request to ws endpoint!')
        if is_valid_ws_handshake_request(method,
                                         target,
                                         http_version,
                                         headers_map):
            handle_ws_handshake_request(
                client_socket,
                ws_sockets,
                headers_map)
            return
        else:
            # Invalid WS request.
            client_socket.send(b'HTTP/1.1 400 Bad Request')
            close_socket(client_socket, input_sockets, ws_sockets)
            return
    ...

def handle_ws_handshake_request(client_socket,
                                ws_sockets,
                                headers_map):
    # Save this socket in the WS sockets list so we will know to speak WS with
    # it in the future.
    ws_sockets.append(client_socket)

    # To handle a WS handshake, we have to generate an accept key from the
    # sec-websocket-key and a magic string.
    sec_websocket_accept_value = generate_sec_websocket_accept(
        headers_map.get('sec-websocket-key'))

    # We can now build the response, telling the client we're switching
    # protocols while providing the key.
    websocket_response = ''
    websocket_response += 'HTTP/1.1 101 Switching Protocols\r\n'
    websocket_response += 'Upgrade: websocket\r\n'
    websocket_response += 'Connection: Upgrade\r\n'
    websocket_response += (
        'Sec-WebSocket-Accept: ' + sec_websocket_accept_value.decode() + '\r\n')
    websocket_response += '\r\n'

    print('\nresponse:\n',websocket_response)

    client_socket.send(websocket_response.encode())

Finally, when we’re checking a socket that’s read to read from, we can see if it’s a socket we’re chatting WS with:

def main():
    '''
    Creates the front-door TCP socket and listens for connections.
    '''

    # We can have a main socket that listens for initial connections.
    tcp_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    tcp_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    tcp_socket.bind((TCP_IP, TCP_PORT))

    tcp_socket.listen(1)
    print('Listening on port: ', TCP_PORT)

    input_sockets = [tcp_socket]
    output_sockets = []  # Maybe use, we'll see
    xlist = []  # Not using

    # NEW: keeping track of which sockets we're talking to over WS
    ws_sockets = []

    while True:
        # Get the sockets that are ready to read (the first of the
        # three-tuple).
        readable_sockets = select.select(input_sockets,
                                         output_sockets,
                                         xlist,
                                         5)[0]

        for ready_socket in readable_sockets:
            # Make sure it's not already closed
            if (ready_socket.fileno() == -1):
                continue
            if ready_socket == tcp_socket:
                print('Handling main door socket')
                handle_new_connection(tcp_socket, input_sockets)
            ### NEW CODE HERE
            elif ready_socket in ws_sockets:
                print('this is where we would handle the websocket message')
                handle_websocket_message(ready_socket, input_sockets,
                                         ws_sockets)
            ### END NEW CODE
            else:
                print('Handling regular socket read')
                handle_request(ready_socket, input_sockets, ws_sockets)
...
def handle_websocket_message(client_socket, input_sockets, ws_sockets):
    print('Handling WS message from client socket:', client_socket.fileno())
    message = b''
    # TODO: handle websocket message

You can see the full commit for this step here.

Trying this out, we can see the successful handshake in action:

~: $ curl -v 127.0.0.1:5006/websocket -H "Upgrade: websocket" -H "connection: Upgrade" -H "sec-websocket-key: dGhlIHNhbXBsZSBub25jZQ=="
*   Trying 127.0.0.1:5006...
* Connected to 127.0.0.1 (127.0.0.1) port 5006 (#0)
> GET /websocket HTTP/1.1
> Host: 127.0.0.1:5006
> User-Agent: curl/7.77.0
> Accept: */*
> Upgrade: websocket
> connection: Upgrade
> sec-websocket-key: dGhlIHNhbXBsZSBub25jZQ==
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 101 Switching Protocols
< Upgrade: websocket
< Connection: Upgrade
< Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
<

And on the server side:

websockets-from-scratch: $ python3 foo.py
Listening on port:  5006
Handling main door socket
New socket 4 from address: ('127.0.0.1', 54357)
Handling regular socket read
Handling request from client socket: 4
Received message:
GET /websocket HTTP/1.1
Host: 127.0.0.1:5006
User-Agent: curl/7.77.0
Accept: */*
Upgrade: websocket
connection: Upgrade
sec-websocket-key: dGhlIHNhbXBsZSBub25jZQ==


method, target, http_version: GET /websocket HTTP/1.1
headers:
{'host': '127.0.0.1:5006', 'user-agent': 'curl/7.77.0', 'accept': '*/*', 'upgrade': 'websocket', 'connection': 'Upgrade', 'sec-websocket-key': 'dGhlIHNhbXBsZSBub25jZQ=='}
request to ws endpoint!

response:
 HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

What if we try with JS?

const exampleSocket = new WebSocket("ws://localhost:5006/websocket");
exampleSocket.send('foo')

Listening on port:  5006
Handling main door socket
New socket 4 from address: ('127.0.0.1', 55842)
Handling regular socket read
Handling request from client socket: 4
Received message:
GET /websocket HTTP/1.1
Host: localhost:5006
Connection: Upgrade
Pragma: no-cache
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36
Upgrade: websocket
Origin: http://localhost:5006
Sec-WebSocket-Version: 13
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Sec-WebSocket-Key: mTNcafNQdQ6VrDiH5/ytoA==
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits


method, target, http_version: GET /websocket HTTP/1.1
headers:
{'host': 'localhost:5006', 'connection': 'Upgrade', 'pragma': 'no-cache', 'cache-control': 'no-cache', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36', 'upgrade': 'websocket', 'origin': 'http://localhost:5006', 'sec-websocket-version': '13', 'accept-encoding': 'gzip, deflate, br', 'accept-language': 'en-US,en;q=0.9', 'sec-websocket-key': 'mTNcafNQdQ6VrDiH5/ytoA==', 'sec-websocket-extensions': 'permessage-deflate; client_max_window_bits'}
request to ws endpoint!

response:
 HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: RfGS7Se2ChrHy1OSvUyWZmsOiJU=


this is where we would handle the websocket message
Handling WS message from client socket: 4

Nice! It looks like we’ve gotten all the way to having the data read into the handle_websocket_message call. Now comes the hard part.

The WebSocket frames

Now that we can actually get the web sockets connected and send data on it, we need to take the incoming bytes and translate them into a WebSocket frame.

The WebSocket frame looks like this:


      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-------+-+-------------+-------------------------------+
     |F|R|R|R| opcode|M| Payload len |    Extended payload length    |
     |I|S|S|S|  (4)  |A|     (7)     |             (16/64)           |
     |N|V|V|V|       |S|             |   (if payload len==126/127)   |
     | |1|2|3|       |K|             |                               |
     +-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - +
     |     Extended payload length continued, if payload len == 127  |
     + - - - - - - - - - - - - - - - +-------------------------------+
     |                               |Masking-key, if MASK set to 1  |
     +-------------------------------+-------------------------------+
     | Masking-key (continued)       |          Payload Data         |
     +-------------------------------- - - - - - - - - - - - - - - - +
     :                     Payload Data continued ...                :
     + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
     |                     Payload Data continued ...                |
     +---------------------------------------------------------------+

This looks complicated, but the main idea is that there are four sections: flags, payload length, the masking key (maybe), and the payload data. You can read about everything in section 5.2, but I’m going to cover a few particularly interesting ones here.

Masking

First, a quick note about masking. Traffic can be (and sometimes must be) masked with a masking key, to prevent the WS traffic from mimicking other traffic and HTTP requests, which has the potential to cause problems in intermediary devices.

For example, let’s say your request goes through a cache that doesn’t know about WebSocket connections and it caches responses based on their Host header. If you’re communicating via WebSockets, you could make the payload look like a regular HTTP response, and the cache may try to cache it. If you were nefarious, this would let you poison the cache and potentially redirect requests at will.²

WebSocket gets around this by forcing the data to be masked with a random number, making it hard to deterministically shape the traffic on the wire, with an algorithm that works in either direction:

Octet i of the transformed data (“transformed-octet-i”) is the XOR of

octet i of the original data (“original-octet-i”) with octet at index

i modulo 4 of the masking key (“masking-key-octet-j”):

Or, in (hopefully) more clear terms, for each byte of the payload, we bitwise OR it with the corresponding byte of the random number, wrapping around with modulo.

masking_key = ... # A 4-byte random number from the message
decoded_payload = []
for index, byte in enumerate(encoded_payload):
  bitmask = masking_key[index % 4]
  decoded_byte = byte ^ bitmask  # Bitwise OR
  decoded_payload.append(decoded_byte)

This works in the other direction, too, since we’re really just doing bitwise ORs here.

Back to the frame

But, the payload isn’t always masked, though for client->server it must be. The MASK flag, the first bit of the second byte, tells us whether or not it’s present.

This is the first hint that the layout of the frame can change based on its contents. An IPv4 header can vary depending on the options, but normally we’d expect the fields to be where the fields are. Interestingly enough, that’s not the case for websockets.

Along with the mask flag, there’s also the variation due to the varying number of bytes we use to determine the payload length. This is a weird one, but it depends on the initial 7 bits that are always in the same spot. The logic is as follows:

def get_payload_length():
  initial_payload_len = last 7 bits of the 2nd byte
  if initial_payload_len < 126:
    return initial_payload_len
  if initial_payload_len == 126:
    return those 7 bits with the next 2 bytes
  if initial_payload_len == 127:
    return those 7 bits with the next _8_ bytes

So where do we find our payload data? It depends! It depends on both having a mask and the initial payload length from those 7 bits. There are only 6 different cases and we can write some code for it (and I did, which you can read later), but it feels odd to look at. What do you mean the fields aren’t always in the same place — isn’t that what the spec is for? How many bytes are we really saving here?

Parsing the frame

We can start thinking about how to translate these incoming bytes into some representation of a WebSocket frame. Let’s say that we have some class WebsocketFrame, and we can populate it from a message made up of bytes. Let’s assume for now that our bytes represent exactly one full frame. We can start with the flags:


class WebsocketFrame:
  ...
  def _parse_flags(self, data_in_bytes):
      first_byte = data_in_bytes[0]
      # Bad python formatting, but it helps to see where each one is.
      self._fin    = first_byte & 0b10000000
      self._rsv1   = first_byte & 0b01000000
      self._rsv2   = first_byte & 0b00100000
      self._rsv3   = first_byte & 0b00010000
      self._opcode = first_byte & 0b00001111

      second_byte = data_in_bytes[1]
      self._mask = second_byte & 0b10000000

That’s not so bad — by visually lining it up, we can see where each one is.

Next up would be parsing the payload length — this one is odd, because the actual bytes we’ll translate into a number depend on the initial 7 bits that we read, but if we need to combine those bits with later bytes, we need to only use the 7 (and not the masking bit). Speaking of masking, because the payload length also determines where we’ll find our mask key (as the position will vary), we can keep track of that at the same time.


class WebsocketFrame:
    ...
    def _parse_payload_length(self, data_in_bytes):
        # The payload length is the first 7 bits of the 2nd byte, or more if
        # it's longer.
        payload_length = (data_in_bytes[1]) & 0b01111111
        # We can also parse the mask key at the same time. If the payload
        # length is <126, the mask will start at the 3th byte.
        mask_key_start = 2
        # Depending on the payload length, the masking key may be offset by
        # some number of bytes. We can assume big endian for now because I'm
        # just running it locally.
        if payload_length == 126:
            # If the length is 126, then the length also includes the next 2
            # bytes. Also parse length into first length byte to get rid of
            # mask bit.
            payload_length = int.from_bytes(
                (bytes(payload_length) + data_in_bytes[2:4]),
                byteorder='big')
            # This will also mean the mask is offset by 2 additional bytes.
            mask_key_start = 4
        elif payload_length == 127:
            # If the length is 127, then the length also includes the next 8
            # bytes. Also parse length into first length byte to get rid of
            # mask bit.
            payload_length = int.from_bytes(
                (bytes(payload_length) + data_in_bytes[2:9]),
                byteorder='big')
            # This will also mean the mask is offset by 8 additional bytes.
            mask_key_start = 10
        self._payload_length = payload_length
        self._mask_key_start = mask_key_start

Now that we have our flags, our payload length, and where the mask starts, we can parse the mask (if it exists). It’ll be the next 4 bytes from that position:


class WebsocketFrame:
    ...
    def _maybe_parse_masking_key(self, data_in_bytes):
        if not self._mask:
            return
        self._masking_key = data_in_bytes[
            self._mask_key_start:self._mask_key_start + 4]

Finally, we have our flags, we have our payload length, we know where it starts, we know if we’re masked, and we have the mask. We can finally go through the payload data, decode like before, and generate the final data (and add a getter for it):


class WebsocketFrame:
    ...
    def _parse_payload(self, data_in_bytes):
        # All client data_in_bytes should be masked
        payload_data = b''
        if (self._payload_length == 0):
            return payload_data
        if self._mask:
            # Get the masking key
            payload_start = self._mask_key_start + 4
            encoded_payload = data_in_bytes[payload_start:]
            # To decode the payload, we do a bitwise OR with the mask at the
            # mask index determined by the payload index modulo 4.
            decoded_payload = [
                byte ^ self._masking_key[i % 4]
                for i, byte in enumerate(encoded_payload)]
            payload_data = bytes(decoded_payload)
        else:
            # If we don't have a mask, the payload starts where the mask would
            # have.
            payload_start = self._mask_key_start
            payload_data = data_in_bytes[payload_start:]
        self._payload_data = payload_data

    def get_payload_data(self):
        return self._payload_data

If we put that all together, we can update our handle_websocket_message:


def handle_websocket_message(client_socket, input_sockets, ws_sockets):
    # Let's assume that we get a full single frame in each recv (may not
    # be true IRL)
    data_in_bytes = client_socket.recv(BUFFER_SIZE)

    websocket_frame = toy_websocket_frame.WebsocketFrame()
    websocket_frame.populateFromWebsocketFrameMessage(data_in_bytes)

    print('Received message:', websocket_frame.get_payload_data().decode('utf-8'))
    return

And finally, we can run it in the browser console:

You’ll notice I also included a much longer message, with over 127 bytes, just to make sure we were parsing correctly. It took a little debugging, but it works!

The code is rough because it’s just for fun, but you can see the full thing here.

Fun with toy programs

Here’s a short and incomplete list of the things I haven’t touched on:

Sending data back
Message fragmentation: what if I don’t have a full message? I would need to keep track of the payload length received so far.
Threading: what would be a good threading model for these messages? How much would you have to keep in shared buffers as you wait for the rest of incomplete messages?
How should we handle opcodes? Closing the connection? Origin checks?
Extensions? Subprotocols?
How should we actually encode our data?
WSS and encrypting our WS connections?
And probably many, many more

I’m so grateful we’ve had smart people working on these things already. Much like my post on RPCs, writing this from scratch made me really appreciate the libraries we already have to do this work for us. But, even with a tiny slice of reality, this gave us some insight into the handshake, the details of the handshake, the frame, how these things are built, and the problems they have to solve.

Beej’s Guide to Network Programming ↩︎
See something like this for an even better explanation. ↩︎

What happens if you point two CNAMEs at each other? Not much, really

Sun, 14 Aug 2022 20:46:39 -0400

I dabble with DNS for work, and I’m frequently checking if CNAMEs are properly configured. CNAMEs are Canonical NAMEs, kind of like nicknames, that indicate that one domain name is a nickname for another domain name. For example, you probably encounter CNAMEs that map from the classic www subdomain:

www.example.com CNAME example.com

This entry indicates that example.com is the Canonical NAME for www.example.com. When you go to www.example.com, your DNS resolver sees this entry and looks up the right side instead, example.com, which then (hopefully) resolves to the actual destination. This is pretty neat, and it can be handy for doing all sorts of interesting things with subdomains.

If you aren’t familiar with CNAMEs or DNS, I’d highly, highly recommend Julia Evans’ How DNS Works! Zine.

I was thinking about CNAMEs the other day and a silly question popped into my head: if CNAMEs are used to define domain names that are the canonical names for other domain names, what if we defined another CNAME that just pointed back to the first? What if there was a cycle?

TL;DR nothing terribly exciting happens. They thought about this back in 1993.

We can see what this looks like with a quick dig:

blog: $ dig a.alexanderell.is

; <<>> DiG 9.10.6 <<>> a.alexanderell.is
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 29976
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;a.alexanderell.is.		IN	A

;; Query time: 172 msec
;; SERVER: 2001:558:feed::1#53(2001:558:feed::1)
;; WHEN: Sun Aug 14 21:36:47 EDT 2022
;; MSG SIZE  rcvd: 46

Well that’s not very exciting. There was a SERVFAIL on the query.

What does the full query look like starting from the root servers?

(Feel free to skip to the end of the output, where I walk through it query by query).

blog: $ dig +trace +all a.alexanderell.is

; <<>> DiG 9.10.6 <<>> +trace +all a.alexanderell.is
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45480
;; flags: qr ra ad; QUERY: 1, ANSWER: 14, AUTHORITY: 0, ADDITIONAL: 27

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;.				IN	NS

;; ANSWER SECTION:
.			483007	IN	NS	c.root-servers.net.
.			483007	IN	NS	d.root-servers.net.
.			483007	IN	NS	e.root-servers.net.
.			483007	IN	NS	f.root-servers.net.
.			483007	IN	NS	g.root-servers.net.
.			483007	IN	NS	h.root-servers.net.
.			483007	IN	NS	i.root-servers.net.
.			483007	IN	NS	j.root-servers.net.
.			483007	IN	NS	k.root-servers.net.
.			483007	IN	NS	l.root-servers.net.
.			483007	IN	NS	m.root-servers.net.
.			483007	IN	NS	a.root-servers.net.
.			483007	IN	NS	b.root-servers.net.
.			483007	IN	RRSIG	NS 8 0 518400 20220827050000 20220814040000 20826 . oQpyr4pIhGLgcV1GUSYSasiMsvbkTkZ6ZkXyjDQsA5nMob/9YnXX7gAX aTODvODzX68dG6au5/lFHkuH+gbbatHMNQfLND9B1Fah2UVVRKkd10N/ UVCoh4QE3g8ktWvdBriKUfgPC/0iizk1mx/vS0I8M1n9ey9zN8r+vkfd i9izcHE7sX6OCe5OaTTILMDbBZPSp6fJ9Idci3iToPJ3Y23esRanMEqp 0aY2UNpf/oFYW8u/PduugJmyfvp4WZ1lqvWCxx43oSVI/b9QVG9o08Hr 6dQOFmYa7mui65nP7kFW/lWvwP43lNHdpFOu7L9KCzZOenEAT3lTXMUA b7+4fQ==

;; ADDITIONAL SECTION:
a.root-servers.net.	313522	IN	A	198.41.0.4
a.root-servers.net.	313582	IN	AAAA	2001:503:ba3e::2:30
b.root-servers.net.	313628	IN	A	199.9.14.201
b.root-servers.net.	313558	IN	AAAA	2001:500:200::b
c.root-servers.net.	313615	IN	A	192.33.4.12
c.root-servers.net.	313577	IN	AAAA	2001:500:2::c
d.root-servers.net.	313623	IN	A	199.7.91.13
d.root-servers.net.	313621	IN	AAAA	2001:500:2d::d
e.root-servers.net.	313720	IN	A	192.203.230.10
e.root-servers.net.	313578	IN	AAAA	2001:500:a8::e
f.root-servers.net.	313629	IN	A	192.5.5.241
f.root-servers.net.	313544	IN	AAAA	2001:500:2f::f
g.root-servers.net.	314027	IN	A	192.112.36.4
g.root-servers.net.	313600	IN	AAAA	2001:500:12::d0d
h.root-servers.net.	313616	IN	A	198.97.190.53
h.root-servers.net.	313577	IN	AAAA	2001:500:1::53
i.root-servers.net.	313614	IN	A	192.36.148.17
i.root-servers.net.	313614	IN	AAAA	2001:7fe::53
j.root-servers.net.	313614	IN	A	192.58.128.30
j.root-servers.net.	313561	IN	AAAA	2001:503:c27::2:30
k.root-servers.net.	313615	IN	A	193.0.14.129
k.root-servers.net.	313561	IN	AAAA	2001:7fd::1
l.root-servers.net.	313617	IN	A	199.7.83.42
l.root-servers.net.	313617	IN	AAAA	2001:500:9f::42
m.root-servers.net.	313517	IN	A	202.12.27.33
m.root-servers.net.	313515	IN	AAAA	2001:dc3::35

;; Query time: 22 msec
;; SERVER: 2001:558:feed::1#53(2001:558:feed::1)
;; WHEN: Sun Aug 14 21:41:50 EDT 2022
;; MSG SIZE  rcvd: 1097

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4649
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 8, ADDITIONAL: 13

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;a.alexanderell.is.		IN	A

;; AUTHORITY SECTION:
is.			172800	IN	NS	sunic.sunet.se.
is.			172800	IN	NS	durinn.rhnet.is.
is.			172800	IN	NS	bes.isnic.is.
is.			172800	IN	NS	tg-tldsecondary01.isnic.is.
is.			172800	IN	NS	ht-tldsecondary01.isnic.is.
is.			172800	IN	NS	sab.isnic.is.
is.			86400	IN	DS	179 8 2 D45BB61D3BC69EF89B877439444854151B3EC3A2A9E3E740A3D89DBB 5B5CD15B
is.			86400	IN	RRSIG	DS 8 1 86400 20220827170000 20220814160000 20826 . KYXFNCkA+RqpWwEnNH/dpCVdTCQ5b51nU9bFighuq7OkG9LKQS4esY6z hpFfpTkeZSOPXeVQV6Ijyw/T4egEFFkpD3yst5kroHcHNwsoQJ9VTX8R 5b9Y/VVO1KSDWRHz1ImReHA+FyndqWOv+vCvfSIDc4ntdMdq7xpbjsjH HqsgslYdn+5yyb1Y/O4ioMyPfC0W2bCmmUwiwpazZBXmwAFHyYhtVMSb vCEX4DwshK6q3QrN86o/PC33xdMJB84bXSSvd7vqRylNOcMVoy9T6l6u eBWi50XccRv58GUB/WHO3CwF9quMtw/qTe9hhF0hax0HpDUJzDxRW/dD k42EvA==

;; ADDITIONAL SECTION:
tg-tldsecondary01.isnic.is. 172800 IN	A	185.93.156.154
ht-tldsecondary01.isnic.is. 172800 IN	A	185.93.156.10
durinn.rhnet.is.	172800	IN	A	130.208.16.20
sunic.sunet.se.		172800	IN	A	192.36.125.2
sab.isnic.is.		172800	IN	A	194.146.106.58
bes.isnic.is.		172800	IN	A	204.61.216.116
tg-tldsecondary01.isnic.is. 172800 IN	AAAA	2001:67c:6c:f056::154
ht-tldsecondary01.isnic.is. 172800 IN	AAAA	2001:67c:6c:56::10
durinn.rhnet.is.	172800	IN	AAAA	2a00:c88:10:16::20
sunic.sunet.se.		172800	IN	AAAA	2001:6b0:7::2
sab.isnic.is.		172800	IN	AAAA	2001:67c:1010:14::53
bes.isnic.is.		172800	IN	AAAA	2001:500:14:6116:ad::1

;; Query time: 90 msec
;; SERVER: 202.12.27.33#53(202.12.27.33)
;; WHEN: Sun Aug 14 21:41:50 EDT 2022
;; MSG SIZE  rcvd: 812

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4229
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 5, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;a.alexanderell.is.		IN	A

;; AUTHORITY SECTION:
alexanderell.is.	86400	IN	NS	ns3.digitalocean.com.
alexanderell.is.	86400	IN	NS	ns1.digitalocean.com.
alexanderell.is.	86400	IN	NS	ns2.digitalocean.com.
ffmia098sr21vlstesk9tcas3o8r2kbi.is. 1800 IN NSEC3 1 0 0 - FFN0KB1I2OUM49H3U27J2Q5TIJ2ID6U9  NS
ffmia098sr21vlstesk9tcas3o8r2kbi.is. 1800 IN RRSIG NSEC3 8 2 1800 20220823181002 20220809164002 27694 is. jAg9RbYPjOHbxBPjJaBe9v2/y1A8h8mBHXzvliihyXw/ZPJa7nFA2q/O rdmYNgHMbF25qePTv0lQXsohVHXcvTxCY6lVND0YDu3FfsQ2zdfBSpzz wOq2mUogLduIAavbx6EmNow4nvbGR4Lg3aiFVjVI+l9ra8WVrPWrOf62 wx6JfezdbXNimgRMRVn8ndSHvJPSQfyRpAjjxiAa4WQqfqRuPujSNOjD IwGk6k2hfTambCbDHXY5QKOPoE0fVAXtCKS4YbcJlIgI6m/POE1Pz92Y kK223LnTTPQULyeuzd57V1imk6gKDtH4SiW2KhuxsuUhzq3ZryWSHHwu 0DtE4Q==

;; Query time: 117 msec
;; SERVER: 2001:67c:6c:56::10#53(2001:67c:6c:56::10)
;; WHEN: Sun Aug 14 21:41:51 EDT 2022
;; MSG SIZE  rcvd: 482

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 28117
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;a.alexanderell.is.		IN	A

;; Query time: 97 msec
;; SERVER: 198.41.222.173#53(198.41.222.173)
;; WHEN: Sun Aug 14 21:41:51 EDT 2022
;; MSG SIZE  rcvd: 46

The dig output can look gnarly, but let’s break this down. dig is recursively asking “Where do I find this domain?”, starting with the root servers and asking the DNS server in the answer. You can imagine it walking down a tree of DNS servers as they get more and more specific for the domain in question.

You can see the first request in the first QUESTION SECTION: . IN NS, or “I’m looking for any domain. Who are the root servers?”. This query did fine, and it returned the root servers and where to find them.

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45480
;; flags: qr ra ad; QUERY: 1, ANSWER: 14, AUTHORITY: 0, ADDITIONAL: 27

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;.				IN	NS

;; ANSWER SECTION:
.			483007	IN	NS	m.root-servers.net.
... More root servers

;; ADDITIONAL SECTION:
m.root-servers.net.	313522	IN	A	202.12.27.33
... where to find other root servers

The next question was to ask the root servers about this domain. We can see it asked the server with the IP 202.12.27.33, which corresponds to m.root-servers.net above. We asked it how to find a.alexanderell.is, and it pointed us towards the .is TLD name servers (and a Swedish name server, see the .se). Again we can see that there was no error, which makes sense.

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4649
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 8, ADDITIONAL: 13

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;a.alexanderell.is.		IN	A

;; AUTHORITY SECTION:
is.			172800	IN	NS	sunic.sunet.se.
is.			172800	IN	NS	durinn.rhnet.is.
... More Icelandic servers

;; ADDITIONAL SECTION:
durinn.rhnet.is.	172800	IN	A	130.208.16.20
sunic.sunet.se.		172800	IN	A	192.36.125.2

;; Query time: 90 msec
;; SERVER: 202.12.27.33#53(202.12.27.33)

Next up, we can see the query to one of the .is TLD name servers, 2001:67c:6c:56::10, which corresponds to ht-tldsecondary01.isnic.is. It responded with the DigitalOcean name servers, because I originally set up my Icelandic hostname with them to point at a droplet, though I now just use DigitalOcean for DNS things, though I probably don’t need them any more. Regardless, again we see no error.

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4229
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 5, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;a.alexanderell.is.		IN	A

;; AUTHORITY SECTION:
alexanderell.is.	86400	IN	NS	ns3.digitalocean.com.
... Other DigitalOcean name servers

;; Query time: 117 msec
;; SERVER: 2001:67c:6c:56::10#53(2001:67c:6c:56::10)

Finally, we can see the query to the DigitalOcean name server for this silly setup. We got nothing back, other than a final result that there was a SERVFAIL, which, as you may have guessed, means there was a failure on the server.

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 28117
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;a.alexanderell.is.		IN	A

;; Query time: 97 msec
;; SERVER: 198.41.222.173#53(198.41.222.173)
;; WHEN: Sun Aug 14 21:41:51 EDT 2022
;; MSG SIZE  rcvd: 46

I think it’s pretty likely that the DigitalOcean name server started to look this up and quickly realized they were dealing with me, a fool, who pointed a CNAME to a CNAME that pointed back to the first CNAME. This seems like it could be a pretty basic cycle check, and it makes sense that they’re guarding against such cases.

What if we went outside of DigitalOcean?

Maybe the issue is that we’re giving the DigitalOcean name server too much freedom to squash our silly attempts. What if we split the CNAMEs across name servers? I just happened to have a Google domain for a side project I’ll get around to someday, and it’s pretty easy to point them at each other.

Google Domains DNS setup:

DigitalOcean DNS setup:

This is some seriously silly stuff right here. Giving this a shot, we can see that spreading these CNAMEs across two name servers made a difference, and we can now see both CNAME entries from a single dig:

blog: $ dig a.alexanderell.is

; <<>> DiG 9.10.6 <<>> a.alexanderell.is
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58919
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;a.alexanderell.is.		IN	A

;; ANSWER SECTION:
a.alexanderell.is.	42393	IN	CNAME	a.reviewscheduler.com.
a.reviewscheduler.com.	3600	IN	CNAME	a.alexanderell.is.

;; Query time: 358 msec
;; SERVER: 2001:558:feed::1#53(2001:558:feed::1)
;; WHEN: Sun Aug 14 22:30:49 EDT 2022
;; MSG SIZE  rcvd: 95

If you look at the traces (exercise left to the reader), you can see that there’s a little more going on under the hood, in particular what looks like proactive lookup ahead of time on Google’s side. Importantly though, these servers find that the end result is another CNAME pointing back to the original, and they go no further.

If we now try to curl one of these hosts, the DNS resolution times out after a while, as it’s now probably trying these CNAMEs back to back, looping as it makes futile attempts to resolve. Sorry, curl.

blog: $ curl -v a.alexanderell.is
... 30 seconds later
* Could not resolve host: a.alexanderell.is
* Closing connection 0
curl: (6) Could not resolve host: a.alexanderell.is

What does the DNS spec say?

If we go back to 1993, RFC 1536 explored some of these issues:

2. Recursion Bugs

   When a server receives a client request, it first looks up its zone
   data and the cache to check if the query can be answered. If the
   answer is unavailable in either place, the server seeks names of
   servers that are more likely to have the information, in its cache or
   zone data. It then does one of two things. If the client desires the
   server to recurse and the server architecture allows recursion, the
   server chains this request to these known servers closest to the
   queried name. If the client doesn't seek recursion or if the server
   cannot handle recursion, it returns the list of name servers to the
   client assuming the client knows what to do with these records.

   The client queries this new list of name servers to get either the
   answer, or names of another set of name servers to query. This
   process repeats until the client is satisfied. Servers might also go
   through this chaining process if the server returns a CNAME record
   for the queried name. Some servers reprocess this name to try and get
   the desired record type.

   However, in certain cases, this chain of events may not be good. For
   example, a broken or malicious name server might list itself as one
   of the name servers to query again. The unsuspecting client resends
   the same query to the same server.

   In another situation, more difficult to detect, a set of servers
   might form a loop wherein A refers to B and B refers to A. This loop
   might involve more than two servers.

That’s us!

FIXES:

      a. Set an upper limit on the number of referral links and CNAME
         links you are willing to chase.

         Note that this is not guaranteed to break only recursion loops.
         It could, in a rare case, prune off a very long search path,
         prematurely.  We know, however, with high probability, that if
         the number of links cross a certain metric (two times the depth
         of the DNS tree), it is a recursion problem.

      b. Watch out for self-referring servers. Avoid them whenever
         possible.

      c. Make sure you never pass off an authority NS record with your
         own name on it!

      d. Fix clients to accept iterative answers from servers not built
         to provide recursion. Such clients should either be happy with
         the non-authoritative answer or be willing to chase the
         referral links themselves.

No surprises here — these all make sense as ways to defend. This does bring up another interesting question though: what’s the longest valid DNS chain we can make? Another exercise left to the reader.

What other things are out there?

I’m no expert on DNS servers (I’m more of an “application layer well after DNS” guy), but after a quick search, I found an interesting article about DNS loops and potential attack vectors, where they found a real-world example where NS record cycles led to a magnification of queries to the authoritative servers. In their words:

Last May, we publicly disclosed tsuNAME, a DNS vulnerability that could be exploited to mount DDoS attacks, where resolvers, clients and/or forwarders send endless queries to authoritative DNS servers. Although earlier RFCs have documented the existence of DNS name loops, none of them have fully addressed the problem. To fix that, we have proposed a new IETF draft to the DNS Operations Working Group (DNSOP WG).

From their technical report on tsuNAME:

On 2020-02-01, two domains (DomainA and DomainB) under .nz had their NS records misconfigured to be cyclically dependent. DomainANS records were pointed to ns[1,2].DomainB.nz, while DomainB NS records pointed to ns[1,2].DomainA.nz. This configuration error led to a 50% surge in query volume at .nz authoritative servers (Figure 1) . The .nz operators manually fixed this misconfiguration on 2020-02-17, after which the queries to return to normal levels

From the Emulating TSUNAME section, they describe recreating the issue with their own authoritative servers. They used probes to send ~18,000 DNS queries to their first hop resolvers, and those requests were eventually magnified to ~8,000,000 queries to the authoritative servers.

Recommend reading through the rest of the technical report if you’re curious — it’s pretty interesting stuff.

Glad I stopped at CNAMEs.

Sharpening the axe for programming

Sat, 06 Aug 2022 15:46:51 -0400

A woodsman was once asked, “What would you do if you had just five minutes to chop down a tree?” He answered, “I would spend the first two and a half minutes sharpening my axe.”

When it comes to solving a problem with code, I’ve found a few things that help make the process more efficient. The code is the output, but like sharpening the axe, there are a few things I do outside of writing code that makes it easier. I’ll be walking through a few of them below.

Quick note that this isn’t a guide to sharpening your own programming axe; that you’ll have to figure out on your own! While I encourage everyone to think deliberately about how they work, this is just a few things I’ve found work for my own brain.

Explaining myself

I’ve found that one of the best ways to think through a tricky problem is to explain it first. Sometimes it’s a formal design document for other people to spread awareness and build consensus. Sometimes, it’s a rubber-duck debugging session as I explain the problem to my dog. Sometimes, it’s even a scribbled diagram on a throwaway piece of paper that helps me visualize the solution.

Putting the problem into words forces you to think through it and requires that you be exact about it. There’s nothing quite like attempting to go through a detailed explanation to force you to really understand what’s going on (extra points if you’re explaining it to someone else), and you quickly uncover areas you don’t fully understand. Those areas you don’t understand are the ones that can make coming up with a solution and implementing it difficult, and working through them ahead of time can save you from so many later troubles.

Additional extra points if you have someone already familiar with the problem take a look at what you’ve written to confirm your understanding. It’s a great way to learn, and it’s a strong benefit to going through the writing process.

Deliberately exploring and chipping away

When working through a tricky problem, it can be so tempting to noodle, and I’d be lying if I said I had never rerun a test after randomly changing a line or two without fully thinking through it. I try not to, though. I’m a huge fan of keeping a debugging lab notebook, and I often find myself writing summaries and updates of what I’ve tried and what I’m seeing to help keep my exploration guided, even though no one will ever read them. Most of the time, they’re a winding path of things I uncover and think about as I work through a problem, including the next steps I’m planning to take.

It’s a combination of writing it down (like earlier) and being deliberate that helps me. Keeping track of what I’ve done, where I’ve been, and the immediate next steps helps me deliberately build my understanding, as I write down questions and dig in to the answers. It’s like a targeted exploration of the problem, where you take individual steps to build your understanding, explore the problem, and get closer to a solution.

I really enjoy this Julia Evans comic about debugging: just focus on one question at a time, and slowly build understanding from there!

debugging strategy: come up with one question pic.twitter.com/2Lytzl4laQ
— 🔎Julia Evans🔍 (@b0rk) August 1, 2022

When faced with ambiguous problems and a difficult search space, I find it helpful to take those individual steps. I write down a few questions, then go through answering each one. They often result in more questions, but they also result in progress.

Much like writing notes throughout the day, I find that the best benefit is from the process and not saving my steps for the future. Usually my debugging notebooks waste away as bytes on a Google Drive server somewhere, but occasionally I’ll see a similar problem and be able to go back and see what steps I took before. Like visiting old code, it’s also sometimes fun to go back with my current knowledge and see how I thought about a problem at the time.

Passing it off to a background thread

One of the best ways for me to make forward progress is to step away from the computer and let my brain passively think about the problem. It can be difficult to remember to do sometimes, especially when I’m stuck deep in a problem whose solution feels like it’s right around the corner: what if I just tried this one other thing?

In the moment, it can feel counterintuitive, especially when you’re deep in the weeds. I’ve never regretted taking a break, and more often than not, something in the background will fire and make a connection that makes sense and helps uncover some mystery.

If none of this works, I’ve found that one of the best ways to move forward with a problem is to get away from the problem. If it’s halfway through the day, going for a walk around the block (importantly, without just going on my phone) or going for a swim will often let me realize something I missed that proves to be the key when I get back to my desk. If it’s close to the end of the day, heading home and letting the problem sit overnight is often enough to unblock me for the next morning. I’ve lost track of how many times I’ve realized a key insight on the train home, only 15 minutes of unfocused day dreaming since I was just looking intently at the code.

Part of the game for me is making it easy for my brain to do the difficult work. I once saw a question about whether or not typing speed was important to be a proficient programmer, and I don’t think it is. For programming, typing the code in is the easy part; it’s understanding systems and the existing code, deciding which code to write, and figuring out how to write it that are the hard parts. Luckily, I’ve found a few things that make it all easier.

Funny artifacts in a 3D house scan

Sat, 30 Jul 2022 13:16:31 -0400

I like to window shop houses on Zillow. I really enjoy the recent trend of 3D interior scans, as they help give you a feeling for what the place is like, and it allows for another way to explore the space without seeing it in person, which is key for a good cross-country window shopping experience.

The other day, I was looking at a 4 story townhouse, and I noticed a few weird things on its 3D model. These models are frequently weird to look at because of the exterior wall artifacts, but this one was extra weird.

Is that a bed floating in the void?

Why yes, I think it is! But the bed looks familiar: it’s a weird copy of the bed in the room next to it.

Is this the bed’s shadow self? The Upside Down?

I’ll give you a moment to think about why this may be happening. When you’re ready, click the spoiler tag below.

Spoiler: why there is a floating bed

It looks like the 3D camera was bouncing its sensing waves off the mirror! It saw a bed through the mirror, and it couldn’t tell whether the thing it saw was a real bed, maybe through a window, or a bed’s reflection in the mirror. Additionally, because the vantage point of the camera was a limited section of the bed through the mirror, it thought it was a triangle shaped bed — the right side of the ghost bed is truncated along the camera’s line of sight at the edge of the mirror.

Here’s the position the camera was in when it was doing the 3D scan of this room:

It’s so cool that the camera can create a (mostly working) 3D model from just a single vantage point, but you can see how important the additional scans are for verifying and completing the models.

There are a few more in here, too, like these windows that are translated… scarily.

Here’s a link if you want to play around with the 3D model yourself. It’s fun to think about how these things work!

My false start when learning to program

Sat, 16 Jul 2022 16:35:10 -0400

I got into programming after college, but there was a brief period of time when I had first tried to learn how to program. It didn’t stick, and I was thinking the other day about why that first attempt didn’t keep.

It was during one of my college summers (2012 or so) when I was working as an intern at a small law firm in Vermont. My day to day duties varied heavily, with some days being very busy and other days being filled with nothing to do, and in some of that downtime, I decided to learn to program. I had met a few people at school who were studying computer science, and although I knew next to nothing about what it actually entailed, I thought it may be something that would be good to do.

I had heard about the introductory JavaScript classes on Codecademy, and I started following the first few modules. I remember working through a few of the starter courses, but it got difficult fast. There were a few concepts I didn’t immediately understand, and after floundering for a bit, I fell off the learning wagon. With the braggadocio of a 19-year-old, I cast it aside as something I wasn’t interested in. I didn’t take a single CS class in college, and I didn’t pursue more programming until much later.

Joke’s on me — I now do it for my day job, and I really enjoy programming.

In retrospect, I think the main problem was motivation, in multiple ways. It was during a period of “I don’t really know what to do, so maybe I’ll try to learn to program, or maybe learn to speak Korean, or what if I got into…”, where I was exploring broadly, but without clear goals or much practice learning on my own. At the time, I didn’t even know what I wanted to major in. Because there wasn’t much real interest backing the initial exploration, I backed out when I ran into some resistance and it required a little more effort, and I instead moved on to the next thing (spoiler alert: I also didn’t learn Korean). I wasn’t very good at putting in the work it takes to learn hard things, and I hadn’t had a lot of practice studying, though I somehow passed (most of) my freshman year classes without good study skills.

I think that starting with the broad, amorphous goal of “learn to program” didn’t include tangible goals that I think would have helped motivate 19-year-old me, like building an actual website, writing a useful app, or making a basic game. You need the fundamentals to build off of, but ideally you’d have something to build towards. Multiplying numbers for the sake of multiplying numbers wasn’t enough, and for an autodidactic approach (without a lot of autodidactic practice), there wasn’t strong enough internal motivation towards a goal to stick with it. Looking at it with more experience, it’s also clearly an impossible goal — you’re never really done learning.

That lack of inspiration wasn’t helped by an aversion to putting in the work. At the time, I hadn’t developed strong study skills and had been able to mostly coast through my freshman year, with a few bumpy general chemistry Cs offset by my liberal arts As and Bs. I had been able to make it through a lot of my earlier schoolwork with pretty good grades without learning to seriously study, and I hadn’t yet developed the muscle to work hard on something when it doesn’t immediately come naturally. Thinking like a computer with the rigid rules of a programming language was a new way of thinking for my brain, and when it started to get hard, there was that nagging feeling to just move on, and move on I did.

One of the unfortunate side effects was that I assumed I wasn’t interested in programming or computer science, even though I had a very, very narrow view of it and next to no context on what it was actually like.

Fortunately, I did end up coming back to it much later. When learning the second time around, a large part of the initial motivation was to make a career shift, and it was much easier to direct my energy and focus, aided by a little more maturity and practice learning as well. There was nothing quite like fielding customer support calls to help motivate me to study after work, and I was able to stick with it much better. I ran into similar sticking points, but I worked through them, asking for help and giving it time. I built up the muscles of putting in the work to keep trying at a problem and sticking with something difficult until it comes into focus.

It went much better than my first attempt. As I kept learning, the career shift motivation gave way to a general interest and enjoyment of both computer science and learning. It also naturally developed into a positive feedback loop, where I put in the work to learn more, it got easier to put that work in and learn, and I enjoyed the process and wanted to keep learning.

When people ask me about learning to code, I try to think of ways that they can leverage their interests — maybe if you love cooking, you can learn enough to build a basic cooking app to scratch your own itch. You’ll run into some tough spots, and having that inspiration can help you get through them, hopefully until you’re well practiced in getting through those tough spots. Better yet, if you’re able to work through it with other people, you can get through those tough spots together. I’ve found that a lot of learning difficult things comes down to sticking with it and putting in the work to understand; there will always be tough spots, but you get better at dealing with them.

I’m glad I stuck with it.

This post was inspired by a few thoughts I put together for a Hacker News thread: “Ask HN: Why did you quit learning programming?”

Work context, home context

Sun, 10 Jul 2022 12:38:39 -0400

When I usually think about context in the meta, non-technical part of being a software engineer, I usually think of context switching and swapping your brain’s focus and working memory from one mode to another, switching from focused programming to a status meeting or trying to fit in a little coding time in the 30 minute blocks in between meetings.

Another aspect of context I’ve been thinking about recently is the separation of work context and home context. I do best when there are clear distinctions between the two, where there is a separate context for Work and a separate context for Life (or at least the important rest of life that isn’t about work). It makes it easier for me to get into work mode, and conversely, it makes it easier to transition out of work mode into non-work mode. It’s like the world’s simplest decision tree:

If I’m in my work context, do work things, otherwise, do non-work things.

This “in my work context” part is a little fuzzy these days, though. It had vanished for me two and a half years ago. In 2020, my wife and I were living in a 1-bedroom loft apartment in an urban area, close to the trains that brought me to the office. It was a smaller space, but we were used to spending a lot of time outside of it. When the forced WFH happened, it was suddenly not only my home space, but also my work space, recreation space, and really, my only space.

Since then, I’ve worked mostly remotely, and it has been an interesting exercise in exploring these contexts and their relationships with my work and life. I’ve found that physical boundaries help me better build mental boundaries, with one of the best improvements being when we moved to a 2-bedroom apartment and I was able to have a separate space to devote to a home office. I could even shut the door and not see my desk after hours or on the weekends. It’s a little funny to put it into words, but it’s really like I was fooling myself with some regressed object permanence; out of sight, out of mind!

I would be lying if I said every day was (or is) a good day, though. Especially during the early pandemic times when I was also facing school context, the ratio of good days to bad days fluctuated drastically. Some days it was easy to get it going, but there were other days where I wouldn’t really be able to get in the groove until the late afternoon, leading to a disruption of my normal hours and a clobbered work/life balance. Over time, as I better developed my tricks and habits (and got a second bedroom), the ratio improved, but I’ve been constantly on the lookout for both slippage and ways to improve it further.

There were a few periods where I had the option of heading back into the office, and after I did a few times, I was struck by how much it helped reinforce those boundaries. Although half of my days were spent catching up and drinking coffees, when it came to sitting down and getting some focused work done, it was easy. We were at work, so of course I would just knock out some work! I’m also fortunate enough to be able to slip into a focused mode with a pair of noise canceling headphones and some good music, even with a cluttered visual field, allowing even a noisy open office to exist outside of my little pond of stillness and focus. Even though the environment was much busier than my private office at home, it was the act of being in that environment itself that made it easier for it all to flow.

More importantly, though, I found that I was better able to leave work at work. I would log off, close the laptop, maybe leave it there, and walk out of the building. My commute was only around 30 minutes or so, a mix of walking and easy trains, letting my brain wander and gracefully transition away from the worries of work. I would sometimes still think about it, but there was a much cleaner break.

Oh, work? Yeah, I’ll deal with that when I’m back tomorrow.

I’ve since left that office job and started working fully remotely. Mindful of these mental tricks that help me effectively strike a good balance, I’ve deliberately focused on building up the physical barriers and routines that help. I have a dedicated bedroom office, strict hours, commute-like dog walks before and after work to clear the cobwebs, and a silly but strict policy of putting my work laptop away immediately after my workday is done.

For an extra step, I’ve recently started renting a private office in a local coworking space. It has been a great addition so far, giving me that additional physical separation of a place for work and a place for the rest of life. With a clearly dedicated space, people to talk to, and the extra physical context, it has reminded me of just how helpful the separation can be. Maybe someday I’ll be out in the countryside with a separate work/office shack I can retreat to during my work day, but for now, the private office is a wonderful balance for our current city living, and I’m excited to have that space and be a part of the community.

I’m always envious of the people that don’t need these extra steps to effectively strike healthy balances, but I know it’s harder for me. It has made me appreciate the wider range of working styles, and it has made me more skeptical of broad stroke declarations of How You Should Work, with the nuances showing themselves in the different experiences I’ve seen and had myself.

A big part of the game for me is to understand how my brain works, then work with it. For me, the physical context switching helps my brain, and I’ll be leveraging that as much as I can.

How to break the Envoy CI at head

Sun, 03 Jul 2022 18:16:17 -0400

I like to talk about small mistakes I’ve made. I’m a big proponent of psychological safety, and as a champion for it, I think it’s very important to talk about times you’ve slipped up or done something dumb. We all make mistakes, and there are times when you feel silly for doing something.

This is one of those times.

I recently merged a large pull request in Envoy that required a lot of work. It was so satisfying to see the approvals come in, and that final moment where the PR finally merged was so gratifying after many months, many delays, and many revisions.

It was, at least, until I got a few GitHub notifications:

@AlexanderEllis we’re reverting this because it broke CI on main. Can you figure out what broke and re-submit the PR with a fix? The easiest-to-review way to do that is to revert this revert, then add another commit with whatever fix needs to be made.

Ah, that satisfying feeling, quickly replaced by “oh shit I broke the build for a big public project”, the satisfaction vanishing into the afternoon breeze. Not the end of the world, but I hate that feeling, especially as it meant other PRs were blocked.

The problem

Scoping out the error was illuminating, but mostly because I’ve been desensitized to this sort of error message.

...
[ RUN      ] GrpcAccessLoggerCacheImplTest.LoggerCreationResourceAttributes
unknown file: Failure
C++ exception with description "Protobuf message (type opentelemetry.proto.collector.logs.v1.ExportLogsServiceRequest reason INVALID_ARGUMENT:(resource_logs.instrumentation_library_logs[0]) logs: Cannot find field.) has unknown fields" thrown in the test body.
test/extensions/access_loggers/open_telemetry/grpc_access_log_impl_test.cc:51: Failure
Actual function call count doesn't match EXPECT_CALL(*async_client, startRaw(_, _, _, _))...
         Expected: to be called once
           Actual: never called - unsatisfied and active
Stack trace:
  0xf9f9ff: testing::internal::GoogleTestFailureReporter::ReportFailure()
  0xfa25ca: testing::internal::UntypedFunctionMockerBase::VerifyAndClearExpectationsLocked()
  0x6bab7e: testing::internal::FunctionMocker<>::~FunctionMocker()
  0x6bac9d: Envoy::Grpc::MockAsyncClient::~MockAsyncClient()
  0x6bacbe: Envoy::Grpc::MockAsyncClient::~MockAsyncClient()
  0x6b5961: Envoy::Extensions::AccessLoggers::Common::GrpcAccessLogger<>::~GrpcAccessLogger()
  0x6b6b81: Envoy::Extensions::AccessLoggers::Common::GrpcAccessLoggerCache<>::ThreadLocalCache::~ThreadLocalCache()
  0x90b59a: Envoy::ThreadLocal::MockInstance::SlotImpl::~SlotImpl()
  0x90b5fe: Envoy::ThreadLocal::MockInstance::SlotImpl::~SlotImpl()
  0x6b41aa: Envoy::Extensions::AccessLoggers::OpenTelemetry::(anonymous namespace)::GrpcAccessLoggerCacheImplTest::~GrpcAccessLoggerCacheImplTest()
  0x6b425e: Envoy::Extensions::AccessLoggers::OpenTelemetry::(anonymous namespace)::GrpcAccessLoggerCacheImplTest_LoggerCreationResourceAttributes_Test::~GrpcAccessLoggerCacheImplTest_LoggerCreationResourceAttributes_Test()
  0xfb49dc: testing::internal::HandleExceptionsInMethodIfSupported<>()
  0xfb5894: testing::TestInfo::Run()
  0xfb65e9: testing::TestSuite::Run()
... Google Test internal frames ...

This means that an exception was thrown because there was an unknown field, the log field, in a resource_logs.instrumentation_library_logs proto. Because that exception was thrown, this meant that the gmock expectation on line 51 of the async_client.startRaw function being called was not being fulfilled, as the exception stopped everything immediately.

Much like the actual function call, I was also very unsatisfied.

This test in question, test/extensions/access_loggers/open_telemetry/grpc_access_log_impl_test.cc, was parsing YAML strings as protos to ensure the right fields were being populated correctly, with something like the following:

const std::string expected_message_yaml = R"EOF(
  resource_logs:
    instrumentation_library_logs:
      - log:
          - severity_text: "test-severity-text"
  )EOF"
opentelemetry::proto::collector::logs::v1::ExportLogsServiceRequest expected_message;
TestUtility::loadFromYaml(expected_message_yaml, expected_message);
// ... more C++ goodness that verifies that the expected message is what shows up

The error above meant that it was running into trouble when parsing this YAML, purportedly because the log field didn’t exist.

The diagnosis

This area of the code was actually only tangentially related to the changes I was making. I was adding a new OpenTelemetry tracing extension, but this error was in the OpenTelemetry access logger. Although it was a separate extension altogether, it immediately rang a bell as a suspicious character I had seen before.

As part of the PR, I had bumped the version of the OpenTelemetry Proto library that Envoy uses. One of the included changes in this bump was a breaking change that renamed the log field in resouce_logs.instrumentation_library_logs to log_records. I had bumped the version, and I had already done a little work to deal with the new field name, even though I was doing nothing related to logging.

Unfortunately for Envoy’s CI, with hilariously bad timing, around 6 hours before my multi-month PR was merged, a different PR had added a new test with the log field in a YAML string that it parsed. I hadn’t pulled that commit into my branch, and when my PR was merged, this new test immediately started throwing exceptions when it was being run, as this log field no longer existed with the new version of the library.

Luckily, this meant that it was only a quick one line forward fix to get things back in business.

Happy little accidents

These silly mistakes are always helpful, if only for little reminders to yourself down the line. If you’re looking for a few recommendations, this is a good little reminder about a few things:

Be more careful about merging main (rookie mistake, and I blame myself for getting out of practice after 3.5 years of not using git for work)
Be careful about unstable library upgrades
Decouple library upgrades from other functional changes and avoid large PRs
As I mention in that other post, do everything you can to not have multi-month PRs!

May your builds be ever green and your tests be ever passing.

Long running pull requests

Sun, 26 Jun 2022 11:45:27 -0400

There’s nothing quite like merging a pull request that has been pending for months. It’s so satisfying.

At Google, they have little badges for your internal user profile. One of the ones you can get celebrates the longest time delta in between creating a change and submitting it. For example, if you create a PR on August 1st and submit it on September 1st, you’d probably get the 1-month PR badge.

I think I would have broken my record for a recent pull request I submitted. As part of my MSCS thesis work, I added native OpenTelemetry tracing to Envoy. Even before the PR, there was a ton of work to write a proposal, chat with folks, ramp up on Envoy tracing internals, ramp up on OpenTelemetry, figure out the best way to do it, write a prototype, do some testing, get some metrics, write the actual thesis (a casual sprint marathon), defend it, and finally, polish the code to production level and write some solid tests. By nature of it being a side project, the work unfortunately took a back seat to other priorities, like day-job work that pays the bills.

I finally opened a PR on Mar 9, and it was finally merged on June 22. 105 days, 15 weeks, or 3 months and 13 days.

If your brain works anything like mine, a pending pull request lives in the back of your head, stealing cycles that you’d otherwise spend thinking about other things. Worse yet is when you don’t have the time to work on it — comments and reviews slip from “I need to do this” territory into “I’m embarrassed I haven’t done this yet” territory. Going back to the code feels like dusting off old library books — I made notes to myself, but of course with the context I had in my head at the time.

I also switched jobs in that timeframe. With everything involved taking up more time (day job, interviewing, wrapping up day job, ramping up at new day job), the backpressure was real, and the 20% project open source work packets got dropped.

In case you’re new to software, the most obvious problem here is that you never want to be in a position where your PR takes months. It should be smaller and self-contained, making the job of reviewing it easy. You should respond to comments promptly and iterate quickly to avoid losing your context. It’s much better to work consistently, rather than bursting code with weeks of delay in between. Or, to put it another way, do as I say, not as I do!

This PR was the majority of the work to get this feature in place, and future PRs will be smaller, more self-contained, and easier to shepherd through the review process. I’m looking forward to writing those, but I’m just glad to have this one all set.

As a final note, merging the PR actually broke Envoy at head due to a subtle issue with an updated field in parsed YAML in a test. Nothing like a quick forward fix to keep you on your toes!

How far behind a plane is its noise?

Sun, 12 Jun 2022 10:03:10 -0400

If you’re ever near a relatively low flying airplane, you may have noticed that it sounds like it’s in a different spot in the sky than where it actually is. You can hear the loud engines, and your ears tell you it should be in one place, but your eyes tell you it’s clearly well ahead of its noise.

It’s a funny visual trick, and, living near the Boston airport flight path, I was thinking the other day about how it works.

What does it mean to be “behind”?

The first key point is that the concept of the noise being “behind” the plane relies on you, as an observer, trying to locate the plane. These planes are generating noise from their engines constantly, and that noise spreads out from the engines. From the plane’s point of view, there’s no real “behind” — the engines are just constantly making noise¹.

Once you’re a little farther away from the plane, it gets a little more interesting. Because noise and light travel at different speeds through air, they can look like they’re out of sync. Let’s say you’re observing a flying plane. At some instant, that plane’s loud engines make some noise, which travels through the air like a ripple on a pond.

A small amount of time later, the sound wave will also have continued on its journey. The plane will also have continued on its journey, and it will be creating new sound waves. For now, let’s just think about the sound that the plane created at that earlier point in time.

The plane is now some distance from where it was when it created that noise — marked by the semi-transparent plane in the diagram. The sound hasn’t reached you yet, but the effect is already in motion.

This will continue as the plane continues to fly.

Finally, the sound wave from the original spot reaches you. By this time, the plane has zoomed ahead, and it’s far from its original position. Because we’re so used to the things we hear being where we see them, it looks to our ears like the plane is in the original position, but it has clearly moved on!

As the plane continues to fly, we as observers hear the ripples that the plane created in the past. Because these sound waves arrive continuously from where the plane used to be, it looks as if the plane’s sound is actually behind it.

Visualizing this with the pictures above, you can see that a few things would affect this effect:

The speed of the plane
The speed of the sound (and how loud it is)
The distance from the observer to the plane

If the plane was moving very slowly, it wouldn’t outpace its sound by much. If the speed of sound was faster, it would get to us faster, leading to less of an effect. If the noise wasn’t loud enough or the plane was too far away, the noise would also dissipate into the air before it got to us (like planes at cruising altitude). If the distance between us and the plane was smaller, there wouldn’t be as much of a dramatic difference between the two speeds.

Luckily for us, planes move fast, they make a lot of noise, sound is relatively slow (compared to light), and planes are usually far enough away.

What a funny trick!

Light versus sound

We’re able to see the plane because of the light reflecting off of it, and we’re able to hear the plane because of the noise from the engine. Light travels really fast, so we can assume that the plane is exactly where it looks like it is. Because the light and the sound travel the distance to us at different speeds, there ends up being this funny gap between the two. Because the plane is moving fast and light gets to us very, very quickly, by the time the sound finally makes it to us, the plane has moved on.

Part of the reason this ends up being such a funny feeling is because our ears are usually so good at visualizing where something is from its sound. If you’re able to, try having someone walk around the room and snap their fingers while you have your eyes closed — even without the visual clues from your eyes, your ears are working overtime to help you “see” where they are in 3D space. You could imagine this being very helpful evolutionarily — when I hear the prey I’m hunting and am able to visualize where it is, it makes it that much easier to see it and get it.²

Even though this phenomenon is the easiest to visualize with sound since the speed gap is wide, it also shows up with light, though it’s a little weirder. If something is moving fast enough and the distance between us and the object is large enough, there can be a gap between where it looks like the object is and where it actually is.

For example, the sun isn’t where the sun looks like it is — it takes about 8 minutes for the light to get to us, so we’re seeing the sun where it used to be, 8 minutes ago. In reality, much like our plane from earlier, the sun has moved on. This one is a little harder to visualize, because it decouples the light from something and that something’s position, but it’s the same exact effect, though without the additional ability to easily verify where something is. This is a neat part of astrophysics, and it’s a little more mind bending — if we’re pretty used to things being where they sound like they are, then we’re very, very used to things being where they look like they are!

Napkin math

We can easily calculate how far the sound is behind the plane with a few approximations.

Like any good physicist, we’ll fudge the numbers with some reasonable approximations. Let’s say that the speed of sound is constant in the air between us and the plane (it’s not). Let’s say that the speed of light is instantaneous (it’s not), so the plane is always where it looks like it is (it slightly isn’t). Let’s assume the speed of the plane is constant (it probably isn’t), and let’s assume it’s flying in a straight line through space (it probably isn’t). We can also look at right angles to help visualize the math easier.

The math is pretty simple: in the time it takes for the sound to reach us, how far does the plane go? We can start with some basic values for everything. Let’s say sound travels at exactly 343 meters per second. Let’s say the plane is at a usual landing speed, 265 kilometers per hour, which is around 73.6 meters per second. Let’s say the observer is 10km from the plane (you can make your own triangles from altitude and GPS coordinates if you want).

With this, it looks like the sound would be around 2.1km behind the plane — definitely enough to notice!

How could you estimate this quickly when you’re out and about? One easy way to do it would be to keep your eyes locked on a single position where the plane is at one instant, then, keeping your eyes on that same spot, count seconds mentally until the sound “looks” like it has also reached that spot. Multiply that by some rough speed of the airplane, maybe 100m/s or so (around 360kph or 224mph, which may be a good guess for a landing/taking off aircraft³), and that will tell you roughly how far behind the sound is. Multiply it by the speed of sound (350m/s or so for napkin math), and that will tell you roughly how far away you are from the plane.

This means that if a plane takes 15 seconds for its sound to catch up, it will “look” like the plane is 1.5km ahead of its noise, and you can also tell you’re around 5km away. Pretty neat!

Mach speeds notwithstanding — although flying faster than the speed of sound doesn’t change the effect we’re talking about here, it does get into all sorts of other interesting sound questions. ↩︎
As a quick aside, dogs tilt their heads when listening intently, as it allows them to better echolocate the source of the noise. Here’s some dog tax of this in action. ↩︎
Details left as an exercise to the aeronautically-minded reader. ↩︎

I only care about the helpful notifications, not the promotional ones

Sun, 05 Jun 2022 10:54:51 -0400

Here’s a scenario. First, you download an app for a service that has some sort of in-person component, like picking up an order or meeting a driver at the curb. When using that app for the goal, be it that order or that meeting, it’s very helpful to have notifications, since they give you the current status and it can be important to coordinate.

You’d like to get a notification for when your order is ready, since that takes away the need for you to repeatedly poll the app and refresh constantly. This makes sense.

You get your notification, you meet your driver or pick up your order, and all’s well. Those notifications helped you coordinate during the transaction, and hopefully you were able to pick up your order before it got too cold. Maybe you get a follow up notification to rate your experience, but other than that, these notifications were helpful. You probably don’t even think about disabling them, since maybe the next time you use the app you’ll want those helpful pings again.

Now it’s a week later, and you haven’t used the app since. Your phone lights up with a notification from the same app.

Ahh, tricky. You want notifications during the transaction, the goal sequence, the thing you downloaded the app for. They’ve got you to enable notifications, since they can be helpful, but now they’ve co-opted that notification consent to send you marketing notifications.

I don’t want to be notified that there are only 9 days left to try Devin Booker’s salad (no offense to Devin Booker). I want to be notified when there are things that need my attention in real time during this business transaction we’re engaged in.

What are you going to do? The only control available (for me, at the time of writing) is to disable notifications at the app level. That seems coarse grained, especially since I do want some of those notifications. Am I really going to enable and disable them every time I use the app? Maybe, but that’s additional toil, and maybe I won’t always remember to. It would be nice if there were finer grained or more accessible controls, like if there was an option when long-pressing the app on the home screen to quickly enable or disable notifications.

Better yet, it would be great to have some sort of “disable marketing notifications” inside the app. Great for us users, but not as great for the companies that are knowingly using this channel for both useful and unuseful notifications and their business engagement metrics. I say “knowingly”, because it’s pretty clear when you look at something like Starbucks’ notifications options:

Dear @Starbucks, this is actually very simple: if you put promotions and order status into the same notifications bucket then you will not be allowed to deliver either. pic.twitter.com/6t64veJyUQ
— Malte Ubl (@cramforce) May 26, 2022

Tricky. The only way to disable notifications for promotions is to also disable notifications for order status, the latter being the actual useful ones. By removing the user’s explicit choice between the two, they’re grouping the helpful with the unhelpful, and I’m sure they’re getting that many more engagements because of it.

Is it a disconnect between user intent and the app’s actual behavior? By not giving the user an explicit choice, they naively accept that “yes to all notifications” option, the only option, when really I think we all know what notifications are the most helpful to the user. It’s like pretending that because you want all of your mail delivered, you also want junk mail delivered. That’s why I find the Starbucks example so interesting — they’re clearly showing the strict pairing of the two!

I’ll be keeping an eye out for this channel re-use. For now I’ll take on the toil to disable them as needed, but it would be nice if we didn’t have to.

Playing with a 3D representation of RGB color space

Sat, 28 May 2022 12:29:44 -0400

I’m working on a color picking/finding game, and part of it will be asking the user to pick a color from a range of colors. This got me thinking: what are some ways we can represent RGB color space?

RGB

One way to think of colors, especially colors on computers, is as a combination of three different primary colors: Red, Green, and Blue. If we say how much of each color we want, we can build a color from the values. For example, maybe we want all Red and no Green or Blue - we could represent this as (100%, 0%, 0%), corresponding to (R, G, B). That would look like this:

That’s an OK red, but it gets more interesting when we mix in some other colors. For instance, here’s 100% red, 33% green, and 33% blue:

Now we’re talking!

2D, but with a spectrum

If you Google “color picker”, you get a widget that provides you with a spectrum of colors to pick from. I think these are pretty intuitive, since you’re able to see what colors are close and similar to your colors. That being said, it’s a little unintuitive how the actual values map to the layout on the grid - try clicking and dragging left and right, and see what the RGB values do.

Give it a shot here.

The y-axis looks like it’s red, as going up and down on the far left and far right only changes the red value. But, go to the middle and go up and down, and all three values change. Put your picker in the middle, then change the spectrum from red to blue - you can see there are discrete sections where only one value or another is increasing. The display is intuitive, but the numbers aren’t!

3D RGB color space

Because each color is composed of those three separate values, you could imagine it as if it were a 3D chart, with one axis for red, one axis for green, and one axis for blue. As you go up to 100% for each axis, the color changes accordingly.

SharkD, CC BY-SA 3.0 https://creativecommons.org/licenses/by-sa/3.0, via Wikimedia Commons

You could, if you wanted to, actually make one of these cubes. That’s what Tauba Auerbach did in the RGB ColorspaceAtlas, where the pages in a book correspond to a slice of that cube, so that every color in the entire space is printed. Pretty neat!

How do we see this represented?

Oftentimes when working with computers and colors, you’ll want to pick a color from a selection of colors. An interesting question here is how the colors are represented — on one unhelpful end, we could just type in the color codes and see the color, while on the other end, we’d somehow fly through the 3D color space to find the right one. An interesting consideration is how well the representation lets you compare the color you’ve chosen with its colorful neighbors. If you’re just typing in an exact number, you won’t have any context of the colors around it. If you’re selecting from a spectrum, you’d have much more context, though this gets interesting when you think in 3D.

As a side note, we usually talk about each color ranging from 0 to 255, with each color represented by 8 bits. I’ll use percentages below, but keep that in mind for the actual code and color pickers.

Exploring in 3D

Let’s start with what looks like a very color picker, where you can pick a value for each color and see how it changes. As you go through, you can also visualize in the 3D space below, where one axis is for red, one is for green, and one is for blue. I’ll let you figure out which is which! Feel free to change the sliders and see how the color changes.

For an extra bonus, try enabling DVD screensaver mode below — you can still mess with the sliders while it’s running.

Red: 50%

Green: 50%

Blue: 50%

DVD screensaver mode:

Off On

Not bad! It’s almost like an ant crawling through the inside of that cube above, reporting back the color it finds at each position.

But I think there are some key drawbacks with this single-color visualization:

It’s hard to visualize what changing a value does to the color, and you can’t see the neighbors before you select the color.
You’re only able to see the exact color at a time, so searching is more difficult.
You have three axes to change, but you’re only able to do one slider at a time.
DVD mode repeats itself frequently

I wonder what this could look like if you had a plane of color across two of the axes, then adjusted the third axis? It would kind of be like fluidly switching between pages in the book above.

This is already rambling enough, so I’ll have to explore that another time :)

You can see the source code here, though it’s very rough around the edges. It’s built with D3 and X3DOM, though it was mostly scraped together from examples.

Wikipedia Over WhatsApp

Sat, 21 May 2022 17:37:15 -0400

In-flight messaging

I was on a Delta flight the other day, and I was thinking about how they offer free messaging from your smartphone.

How do they limit the wifi? Maybe they have some basic filter that says “deny everything except the urls for these 3 messaging apps”, which makes sense conceptually.

What if we don’t want to pay for wifi and we still want to be able to browse the web?

Tunneling through WhatsApp

If the wifi is letting WhatsApp messages through, what if we used WhatsApp as a vehicle for the information we really care about? Much like we encapsulate the rest of our networking objects in higher-level objects, we could encapsulate web pages inside of WhatsApp messages.

We could start with a simple case, like Wikipedia. It’s mostly text-based, so there should be no issue with sending images (which are blocked in the free messaging over wifi). What if we built a basic service that allowed users to query Wikipedia pages and read them over WhatsApp?

Maybe it looks something like this, where a user sends a message to a WhatsApp number. If we hook that number into a WhatsApp automated backend (probably via Twilio), we can have that backend parse the message, query Wikipedia, and send the response.

There are two main components to this Wikipedia Over WhatsApp, or WoW service:

The parsing of the messages and making requests to Wikipedia
The WhatsApp interface for sending/receiving messages

Requesting articles from Wikipedia

Wikipedia has a pretty approachable API that allows you to do a few things. We can keep our service very simple: let’s say the user has to send the title of a Wikipedia article. If there’s an article that matches, we respond with it. If there isn’t an article that matches, we can do a search for the given text and return the top 10 articles.

A very basic Python version might look something like this:


import urllib.parse
import urllib.request
import json

# URL to query a page, format JSON, extract the text, and give us plain wiki text.
PAGE_WIKI_REQ_STRING = 'https://en.wikipedia.org/w/api.php?action=query&format=json&prop=extracts&list=&formatversion=2&exlimit=1&explaintext=1&exsectionformat=wiki&titles='
# URL to search for articles for a search term.
SEARCH_WIKI_REQ_STRING = 'https://en.wikipedia.org/w/api.php?action=query&format=json&list=search&formatversion=2&srsearch='


# This is a simple function for getting articles from wikipedia
def get_wiki_page(page_name):
    # Encode the article name and build the URL.
    encoded_page_name = urllib.parse.quote(page_name.encode('utf8'))
    page_req_url = PAGE_WIKI_REQ_STRING + encoded_page_name

    page_result = urllib.request.urlopen(page_req_url)
    result_body = page_result.read()
    result_json = json.loads(result_body.decode('utf-8'))

    # If it's an article, we should have gotten an 'extract', so we can send it.
    # In our case, this will be the entire article.
    query_result = result_json['query']['pages'][0]
    if 'extract' in query_result and query_result['extract'] != '':
        return query_result['extract']
    else:
        # If we don't have an article, we can instead search for articles for
        # that term and return the top 10 results.
        result_string = 'I didn\'t find a perfect match for ' + \
            page_name + ', but you can try one of the following:\n'
        # Try to search for it now.
        search_req_url = SEARCH_WIKI_REQ_STRING + encoded_page_name
        search_result = urllib.request.urlopen(search_req_url)
        search_result_body = search_result.read()

        search_result_json = json.loads(search_result_body.decode('utf-8'))
        for result in search_result_json['query']['search']:
            result_string += result['title'] + '\n'
    return result_string

To test this, we could even throw it together for a quick CLI tool:

... code from above
def main():
    while(True):
        page_name = ''
        while page_name.strip() == '':
            page_name = input('Please enter a Wikipedia page title.\n')
        get_wiki_page(page_name)
        print('\n\n')


if __name__ == '__main__':
    main()

Running that gives the following:

That’s a good start! Given a page name, we can search for articles or get the requested article. The formatting is wonky, but we’re just having fun here.

The WhatsApp interface

Like a good engineer, I started with a Google search, and one of the first results was exactly on the money:

How to Receive WhatsApp Messages in Python Using Flask and Twilio

I won’t bore you with the details, but the TL;DR is that it’s pretty easy to set up a sandbox Twilio WhatsApp environment that you can send and receive messages with. I hooked this up with a few changes, and the relevant parts are in the following diagram:

The flow is roughly like this:

The user sends a message to the WhatsApp number that corresponds with a Twilio number
That Twilio number is connected to Twilio hooks that say “when I receive a message, post it to this URL”
That URL corresponds to an ngrok URL fronting my WoW service running on my computer
My WoW service receives the message and passes it to the get_wiki_page code above
Once we have a response, we can send it back through Twilio to the user

The only issues ended up being that 1) WhatsApp messages are limited to 1600 characters, while Wikipedia pages are frequently much longer and 2) the basic free accounts I was using rate-limit to ~1QPS, so if you try to chunk up a message and send it as many messages, you may get throttled. To handle those, we can do some basic chunking and delay between messages. You’re on a plane after all; you’ve got nowhere to go!

This simple WoW service, with its message handling, ended up looking something like this:

import os
from dotenv import load_dotenv
from flask import Flask, request
from twilio.twiml.messaging_response import MessagingResponse
from twilio.rest import Client
import time

import wow

# Leave some room for the progress meter, like `(1/20) `
MESSAGE_SIZE = 1590

load_dotenv()

app = Flask(__name__)
TWILIO_ACCOUNT_SID = os.environ.get('TWILIO_ACCOUNT_SID')
TWILIO_AUTH_TOKEN = os.environ.get('TWILIO_AUTH_TOKEN')
client = Client(TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN)


# https://www.twilio.com/blog/receive-whatsapp-messages-python-flask-twilio
def respond(request, full_message):
    # Break the message up into chunks that fall under the WhatsApp character
    # limit and send them with a delay in between.
    chunks = (len(full_message) // MESSAGE_SIZE) + 1
    for i in range(chunks):
        start = i * MESSAGE_SIZE
        end = start + MESSAGE_SIZE
        progress = '({}/{}) '.format(i + 1, chunks)
        message_chunk = progress + full_message[start:end]
        client.messages.create(
            body=message_chunk,
            from_=request.form.get('To'),
            to=request.form.get('From'),
        )
        time.sleep(1)

    return str(MessagingResponse())


@app.route('/message', methods=['POST'])
def reply():
    message = request.form.get('Body')
    if message:
        result = wow.get_wiki_page(message)
        return respond(request, result)

Once I wired everything up and debugged a little, it ended up working pretty smoothly!

As you can see, I now have 20 full messages of the entire Wikipedia page on Cheese to read, and this will keep me plenty busy for the flight!

What would this need to go past a prototype?

For one, like most of my side projects, my artificial limit here was not paying for anything, so I’m rate limited, using the sandbox, and running the service off of my own computer.

The user experience is also pretty rough - maybe we’d want to allow for an easy way to get to the next page? Maybe paginating so you don’t get the full article all at once? Maybe sending a section at a time, so if you’re really just interested in the history of the harmonica, you don’t have to get to the rest of the page?

Nothing quite kills the fun of a silly side project like having to do anything real for it, and as soon as I started to read the guidelines for the WhatsApp Business API and integrating with other Meta services, I realized I wasn’t going to go through with setting this up as a publicly-available service. Feel free to set your own up, though!

Back to the original question, though: would this really work? I think so off the top of my head, but I’ll just have to try setting up a sandbox again before my next flight.

Either way, a fun little toy :)

A basic ear training note matching game

Sat, 14 May 2022 01:03:02 -0500

JavaScript is needed to run this game. If you don’t have it enabled, I totally get it — I often don’t like it either.

This is a simple ear-training game for matching notes. Play along with your favorite instrument!

You can also try singing/whistling, but note that the pitch detection is hand rolled and rough around the edges — apologies if it makes any mistakes, especially for lower notes.

Here’s how to play:

Press the start button to begin
Listen to the notes, then play them yourself
(Optional): Hit Replay to listen to the notes again
Play around with the options and continue as long as you want :)

Options (they’ll apply to the next round):
Number of notes:
Number of repeated success required:
Min note duration (the one you play):
Show notes

Total correct: 0

Perfect streak: 0

Best perfect streak: 0

"I don't know the numbers": a math puzzle

Sun, 08 May 2022 15:29:23 -0400

I saw a math puzzle the other day on Hacker News. It reads as follows:

Two numbers are chosen randomly, both are positive integers smaller than 100. Sandy is told the sum of the numbers, while Peter is told the product of the numbers.

Then, this dialog occurs between Sandy and Peter:

Peter: I don’t know the numbers.
Sandy: I don’t know the numbers.
Peter: I don’t know the numbers.
Sandy: I don’t know the numbers.
Peter: I don’t know the numbers.
Sandy: I don’t know the numbers.
Peter: I don’t know the numbers.
Sandy: I don’t know the numbers.
Peter: I don’t know the numbers.
Sandy: I don’t know the numbers.
Peter: I don’t know the numbers.
Sandy: I don’t know the numbers.
Peter: I don’t know the numbers.
Sandy: I don’t know the numbers.

Peter: I do know the numbers.

What are the numbers?

At first glance this looks impenetrable, but I’d encourage you to give it a think if you’d like to. I’ll be walking through exactly how it works and how to solve it (with the help of a computer).

Spoilers below, be warned

Again, at first glance, it doesn’t look like any of this information would be helpful, but the trick is that each time they say they don’t know the answer, the other party is able to narrow down the options for what the numbers could be. If you do this enough times, there’s only one pair of numbers left, at which point Peter knows the answer.

I found it helpful to think about this in a few different ways. First, let’s say we have a list of all of the pairs of numbers - we can think about those as our candidates. To simplify, let’s say that the order doesn’t matter ((1, 2) is the same as (2, 1)) and we just keep track of one of the duplicates.


candidates: [
  (1, 1),
  (1, 2),
  (1, 3),
  (1, 4),
  ...
  (99, 99)
]

Let’s also imagine a big representation of all of the possible sums and products and the pairs of numbers that make up those sums and products. Let’s say they’re maps, where the keys are the sums/products and the values are lists of the pairs that make up those sums/products. Furthermore, let’s say that both Sandy and Peter have full knowledge of these maps.


sums: {
  2: [(1, 1)],  # The only pair that adds up to 2 is (1, 1)
  3: [(1, 2)],
  4: [(1, 3), (2, 2)],  # For 4, we have two pairs: (1, 3) and (2, 2)
  5: [(1, 4), (2, 3)],
  6: [(1, 5), (2, 4), (3, 3)],
  # And so on.
  ...
  198: [(99, 99)]
}

products: {
  1: [(1, 1)],  # The only pair that multiply together to make 1 is (1, 1)
  2: [(1, 2)],
  3: [(1, 3)],
  4: [(1, 4), (2, 2)],  # For 4, we have two pairs: (1, 4) and (2, 2)
  # And so on.
  ...
  9801: [(99, 99)]
}

The trick is in those entries that are only made up of a single pair.

Let’s look at Peter first. Peter knows the product of the two numbers. Peter can look up this product in the products map - if there is only one pair that makes up that product, then those must be the numbers! When Peter tells us that he doesn’t know the numbers, it means that the product must not be any product that only has one pair. This means that every time Peter doesn’t know, we can remove each pair that is the only pair for a product, and those pairs can be removed from the list of potential candidates.

Let’s look at those above lists and Peter’s first claim. At the start, we have a full list of candidate pairs, a full sums map, and a full products map. Because Peter knows the product, when he says “I don’t know the numbers”, it means that he looked at the possible pairs that made up this product and found multiple potential pairs - otherwise he would immediately know the numbers. This means that we can then disqualify products that are made up of one pair. 1 as a product, for instance, is only made from multiplying the pair (1, 1). Since we know that Peter doesn’t immediately know it as the solution, it means that it can’t be (1, 1).

Similarly, if the product was 9801, Peter would have immediately known the pair was (99, 99). Because he can’t yet say for certain, it means that the numbers must not be (99, 99).

We’re able to now remove all candidates that were the only factors that made up a product. For the first round, this is notably 1 times each prime and each prime multiplied by another prime that results in a number greater than 100. Importantly, we can also remove them and their products/sums from the products map and the sums map.


# After culling the candidates from the first round
candidates: [
  (̶1̶,̶ ̶1̶)̶,̶
  (̶1̶,̶ ̶2̶)̶,̶
  (̶1̶,̶ ̶3̶)̶̶,̶
  (1, 4),
  ...
  (̶9̶9̶,̶ ̶9̶9̶)̶
 ]

sums: {
  2̶:̶ ̶[̶(̶1̶,̶ ̶1̶)̶]̶,
  3̶:̶ ̶[̶(̶1̶,̶ ̶2̶)̶]̶,
  4: [(̶1̶,̶ ̶3̶)̶, (2, 2)],
  5: [(1, 4), (2, 3)],
  6: [(̶1̶,̶ ̶5̶)̶, (2, 4), (3, 3)],
  # And so on.
  ...
  198: [(99, 99)]
}

products: {
  1̶:̶ ̶[̶(̶1̶,̶ ̶1̶)̶]̶,̶
  2̶:̶ ̶[̶(̶1̶,̶ ̶2̶)̶]̶,̶
  3̶:̶ ̶[̶(̶1̶,̶ ̶3̶)̶]̶,̶
  4: [(1, 4), (2, 2)],
  # And so on.
  ...
  9̶8̶0̶1̶:̶ ̶[̶(̶9̶9̶,̶ ̶9̶9̶)̶]̶
}

# Note: the above uses strikethrough code - if it looks funny on your screen, I apologize!
# In that case, the culling is an exercise left to the reader.

This is the trick. Because the space of possible candidates has been reduced, it means the possible candidates that make up each sum and product have also been reduced. Let’s look at the updated candidates, sums, and products, filling in a few more previously omitted values:


candidates: [
  (1, 4),
  (1, 6),
  (1, 8),
  ...
  (88, 90)
]

sums: {
  4: [(2, 2)],
  5: [(1, 4), (2, 3)],
  6: [(2, 4), (3, 3)],
  # And so on.
  ...
  178: [(88, 90)],
  179: [(80, 99)]
}

products: {
  4: [(1, 4), (2, 2)],  # For 4, we have two pairs: (1, 4) and (2, 2)
  # And so on.
  ...
  7920: [(88, 90), (80, 99)]
}

Now, it’s Sandy’s turn to say that she doesn’t know the numbers. Because Sandy knows the sum, again it must mean that the sum can’t be any sum with only one pair as an option in sums, so we can again cull those pairs. In the updated sums structure, we can see that 4 is now only made up of a single candidate pair, (2, 2). If the sum was 4, Sandy would know immediately what the pair was, but since she doesn’t, it means the pair cannot be (2, 2), and we can remove it from the candidates.

Every round, we’re disqualifying potential candidates. Because we’re also able to remove them from the sums and products as potential options, it means that each protagonist is working on a new collection every time, giving a little more information every time they say they don’t know the number.

This pattern will continue, with a certain number of options being removed every time. At some point, Peter finally says that he does know the numbers - finally, there is only one product left with one potential pair, and that is the answer. Importantly, it has to go down to a single product - otherwise there would be multiple answers! That’s why this version has Peter claiming to know after 14 rounds, because it limits the answer to a single pair (although there are other options - see later).

This would be a little rough to do by hand, but luckily we can have the computer do it for us.


N = 100


# Helper functions to regenerate the sums/products data structures every round.
# Not optimal but hey this is for fun.
def generate_sums(candidate_pairs):
    sums = {}
    for tuple_pair in candidate_pairs:
        (i, j) = tuple_pair
        sum = i + j
        if sum not in sums:
            sums[sum] = [tuple_pair]
        else:
            sums[sum].append(tuple_pair)
    return sums


def generate_products(candidate_pairs):
    products = {}
    for tuple_pair in candidate_pairs:
        (i, j) = tuple_pair
        product = i * j
        if product not in products:
            products[product] = [tuple_pair]
        else:
            products[product].append(tuple_pair)
    return products


def find_pairs():
    # First, generate all possible pairs from (1, 1) to (N-1, N-1).
    candidate_pairs = set()
    for i in range(1, N):
        for j in range(i, N):
            candidate_pairs.add((i, j))

    # Swap between hearing from Peter or Sandy every round (product or sum)
    do_not_know_product = True
    round = 1
    while True:
        sums = generate_sums(candidate_pairs)
        products = generate_products(candidate_pairs)
        # First, check if there's a single product left with one pair.
        if round == 15:
            for product in products:
                if len(products[product]) == 1:
                    print('Peter: I do know the numbers')
                    print(products[product][0])
                    return

        if do_not_know_product:
            print('Peter: I don\'t know the numbers')
            # Go through products. For any product that only has a single pair,
            # we can remove those from the candidates for the next round.
            for product in products:
                if len(products[product]) == 1:
                    tuple_pair = products[product][0]
                    # Remove from candidates.
                    candidate_pairs.remove(tuple_pair)
            do_not_know_product = False
        else:
            print('Sandy: I don\'t know the numbers')
            # Go through sums. For any product that only has a single pair,
            # we can remove those from the candidates for the next round.
            for sum in sums:
                if len(sums[sum]) == 1:
                    tuple_pair = sums[sum][0]
                    # Remove from candidates.
                    candidate_pairs.remove(tuple_pair)
            do_not_know_product = True
        round += 1


if __name__ == '__main__':
    find_pairs()

Running that leads to the following, just like the puzzle, though now it includes an answer:


Peter: I don't know the numbers
Sandy: I don't know the numbers
Peter: I don't know the numbers
Sandy: I don't know the numbers
Peter: I don't know the numbers
Sandy: I don't know the numbers
Peter: I don't know the numbers
Sandy: I don't know the numbers
Peter: I don't know the numbers
Sandy: I don't know the numbers
Peter: I don't know the numbers
Sandy: I don't know the numbers
Peter: I don't know the numbers
Sandy: I don't know the numbers
Peter: I do know the numbers
(77, 84)

There’s the pair! 77 and 84.

Interestingly, as part of debugging this solution, I found that there could be different answers at different numbers of rounds. As long as there is a single product made up of a single pair, that could be the answer. We can update the find_pairs function to check for potential solutions at various round lengths (and stop before we go to infinity):


def find_pairs():
    # First, generate all possible pairs from (1, 1) to (N-1, N-1).
    candidate_pairs = set()
    for i in range(1, N):
        for j in range(i, N):
            candidate_pairs.add((i, j))

    # Swap between hearing from Peter or Sandy every round (product or sum)
    do_not_know_product = True
    round = 1
    while True:
        sums = generate_sums(candidate_pairs)
        products = generate_products(candidate_pairs)
        name = 'Peter:' if do_not_know_product else 'Sandy:'

        potential_final = None
        # First, check if there's a single product left with one pair.
        if do_not_know_product:
            count_unique_product_pairs = 0
            for product in products:
                if len(products[product]) == 1:
                    count_unique_product_pairs += 1
                    potential_final = products[product][0]
            if count_unique_product_pairs == 1:
                print(name, 'I could know the numbers after',
                      round, 'rounds:', potential_final)
        else:
            count_unique_sum_pairs = 0
            for sum in sums:
                if len(sums[sum]) == 1:
                    count_unique_sum_pairs += 1
                    potential_final = sums[sum][0]
            if count_unique_sum_pairs == 1:
                print(name, 'I could know the numbers after',
                      round, 'rounds:', potential_final)

        num_candidates_before_cull = len(candidate_pairs)
        if do_not_know_product:
            # Go through products. For any product that only has a single pair,
            # we can remove those from the candidates for the next round.
            for product in products:
                if len(products[product]) == 1:
                    tuple_pair = products[product][0]
                    # Remove from candidates.
                    candidate_pairs.remove(tuple_pair)
            do_not_know_product = False
        else:
            # Go through sums. For any product that only has a single pair,
            # we can remove those from the candidates for the next round.
            for sum in sums:
                if len(sums[sum]) == 1:
                    tuple_pair = sums[sum][0]
                    # Remove from candidates.
                    candidate_pairs.remove(tuple_pair)
            do_not_know_product = True
        round += 1
        # If we didn't remove any pairs, we may be in an infinite loop and want
        # to stop.
        if len(candidate_pairs) == num_candidates_before_cull:
            return

Using this version, we now get the following:


Sandy: I could know the numbers after 6 rounds: (72, 99)
Peter: I could know the numbers after 7 rounds: (81, 88)
Sandy: I could know the numbers after 8 rounds: (70, 99)
Peter: I could know the numbers after 9 rounds: (77, 90)
Sandy: I could know the numbers after 10 rounds: (72, 95)
Peter: I could know the numbers after 11 rounds: (76, 90)
Sandy: I could know the numbers after 12 rounds: (70, 96)
Peter: I could know the numbers after 13 rounds: (80, 84)
Sandy: I could know the numbers after 14 rounds: (66, 98)
Peter: I could know the numbers after 15 rounds: (77, 84)

This means that the puzzle could be a number of different round combinations, with either Sandy or Peter saying they know the answer! In my opinion, showing 15 “I don’t know"s makes the puzzle all the more confuddling, which adds to the puzzleness.

How many rounds does it go? Here, we see that it can progress until there are no culled candidates - for N = 100, we go 15 rounds before we stop. What does that look like for a different value of N? We can explore that easily, and I think there’s a fun question here: what’s the lowest value of N that has an answer?

For the lowest, we can just do the following, adding n as an argument:


def find_pairs(n):
    # First, generate all possible pairs from (1, 1) to (N-1, N-1).
    candidate_pairs = set()
    for i in range(1, n):
        for j in range(i, n):
            candidate_pairs.add((i, j))
    # ... everything else the same


if __name__ == '__main__':
    for i in range(100):
        print('N =', i)
        find_pairs(i)
        print('*****')

N = 0
*****
N = 1
*****
N = 2
Peter: I could know the numbers after 1 rounds: (1, 1)
*****
N = 3
*****
N = 4
*****
N = 5
*****
N = 6
*****
N = 7
*****
N = 8
*****
N = 9
*****
N = 10
Peter: I could know the numbers after 3 rounds: (1, 4)
Sandy: I could know the numbers after 4 rounds: (2, 3)
Peter: I could know the numbers after 5 rounds: (1, 6)
Sandy: I could know the numbers after 6 rounds: (3, 4)
Peter: I could know the numbers after 7 rounds: (2, 6)
Sandy: I could know the numbers after 8 rounds: (4, 4)
Peter: I could know the numbers after 9 rounds: (2, 8)
*****
... a lot more

It looks like the lowest is N = 2, which is not a lot of fun, because the only pair is (1, 1). Not a very interesting puzzle! The next one is N = 10 for values 1-9, which can be done in 3-9 turns. That’s pretty interesting, since we could formulate the riddle differently, though the large number of numbers and rounds makes it all the more befuddling.

I think the other interesting thing here is that for some values of N, there is no solution! Let’s look at N = 4, where the numbers are 1-3. These tables are pretty easy to write out by hand:


candidates: [
  (1, 1),
  (1, 2),
  (1, 3),
  (2, 2),
  (2, 3),
  (3, 3)
]

sums: {
  2: [(1, 1)],
  3: [(1, 2)],
  4: [(2, 2), (1, 3)],
  5: [(2, 3)],
  6: [(3, 3)]
}

products: {
  1: [(1, 1)],
  2: [(1, 2)],
  3: [(1, 3)],
  4: [(2, 2)],
  6: [(2, 3)],
  9: [(3, 3)]
}

OK shoot, if we remove all of the products that are made up of a single pair, we don’t have any pairs left, which makes sense. Because not all values of N lead to a solution, this also means that trying to reason through the problem for low values of N is extra tricky, since it doesn’t actually lead you to a solution!

Overall, a fun puzzle. Here are a few closing extra credit questions for you:

Does it matter whether Peter or Sandy talks first?
What would happen if we had a third person that knew the absolute value of the numbers subtracted from each other? Would the order of the three matter?
What if that third person knew the numbers subtracted, where the order of the numbers mattered?

Getting comfortable with being uncomfortable

Sun, 01 May 2022 00:28:44 -0400

Software engineering is cursed with uncertainty.

When we’re solving big new problems with software, if the exact problem has been solved before, we could just reuse the solution that already exists. Frequently though, the exact problem (or combination of problems) hasn’t been solved before, and we need to come up with a way to solve it.

Have you felt that feeling? That moment of uncertainty, where you don’t know what the solution will look like. You’ve solved many other problems before this one (and so far they keep paying you), but now you have to reach into the creative ether again to come up with a way to solve this new one.

(As an aside, one of my pet theories is that this is part of the reason behind all of our imposter syndromes, since every new problem is a new chance to face this uncertainty. What if I can’t solve this new problem? Is this going to be the one where they finally figure out I’m a fraud?)

It can be uncomfortable. It’s like diving into a new codebase, where at first it’s all a series of files and unfamiliar names. It’s like learning a new technology, where you’re trying to map these new concepts to the languages you’ve seen before while also seeing what’s new. It’s these moments of uncertainty, where everything is still an amorphous, blurry mist that has yet to come fully into focus.

Slowly, over time, the mist starts to clear up. Suddenly you can see how things are connected. After a chat with a coworker, a deep-dive into the code, or a walk around the block, something clicks, and some of the pieces start falling into place. The mist transforms into an outline, the outline into a conversation, the conversation into a diagram, the diagram into a few pull requests, the pull requests into follow up pull requests, and finally this vague problem has been translated from that amorphous blob into concrete lines of code.

This process is one of the most interesting parts of software development. I’m lucky to be able to work on interesting problems and spend a lot of time learning, and I’ve spent a lot of my time in this uncomfortable zone. I’ve gotten used to it and, dare I say, almost started to enjoy it. There’s still that moment of uncertainty, but I have my ways to get through it. I think it’s an important skill to build, this familiarity with the unknown and the uncertainty, and finding ways that are effective for you to get through it can be very empowering¹.

We’ve made it through every other problem so far, and chances are good we can make it through this next one, too.

I particularly enjoy Walter Bright’s method: “1. load my brain with all the context of the problem 2. go out for a run, which bounces it all around in my brain until things fall into place 3. write the solution when I get back” ↩︎

Opinionated variable names

Sun, 17 Apr 2022 13:08:57 -0400

I think a lot about the human part of code, especially the work our brains do when writing and reading it and interfacing with the computer. Rachelbythebay had a great take on a way to inject some brain-processing into a technical task - when you’re running a command to affect n servers, maybe it would be good to type that number in manually.

The idea is to force you to take in that number through your usual input devices (I’d say eyes, but some people are using text-to-speech stuff or similar, and they count too), chew on it with your wetware, and then feed it back into the computer somehow. Adding a few extra steps like this will hopefully activate enough of your brain to make you stop short before blowing off your entire leg with a giant foot-gun.

Rachelbythebay: Type in the exact number of machines to proceed

I like this idea about a mental check, even if it’s just a quick second of consideration, and it’s interesting to see how it can show up in naming, as well.

Take dangerouslySetInnerHtml, for instance. It’s a function in React that, as the name implies, sets the inner HTML of an HTML element on the page. As the name also implies, this is generally scary and discouraged because of how easy it makes it to accidentally surface attack vectors for cross-site scripting. By prepending that adverb and explicitly calling it out as a potential footgun, it (hopefully) adds a layer of self-reflection when typing it in to use it, and it helps to raise a red flag that’s easy to pick up on when reviewing changes.

It doesn’t all have to be so serious, though. One of Google’s helpful frameworks for building websites is named after a character from the Wizard of Oz universe. When running your local development server, when the server was ready, it would print an ASCII drawing of that character to your terminal. It even scaled the drawing to your terminal size, though the drawing was a little more abstract at smaller sizes. You could disable the drawing if you wanted to though, using an aptly named flag that was something like --my_terminal_space_is_more_important_than_your_artistic_expression.

Petty? Definitely. Whimsical and lighthearted? Absolutely.

It’s fun to think about how these names we type out can impart meaning and context beyond the usual “what it is” or “what it does” naming. Like most things, a healthy serving of moderation is key. I don’t want to read every opinion in every function (nor would anyone want to see mine), but it’s fun to see from time to time.

What is a minute, really?

Sun, 10 Apr 2022 11:44:07 -0400

What’s in a minute? It can be sixty long seconds, or it can be just a blink of the eye.

Easy ways to slow time down

If you’re trying to slow time down, see how long a minute feels for any of the following:

If you’re able to, hold a plank for 60 seconds
Read the Elementary Rules of Usage section from The Elements of Style
If you’re able to, sprint at full speed for 60 seconds
After asking a question to a group and hearing no response, wait for 60 seconds before offering the answer
Try to hold your breath for 60 seconds

Easy ways to speed time up

If you’re trying to speed time up, see how long a few minutes feels for any of the following:

Go on TikTok (or your time-waster of choice)
Take a nap
Get immersed in a good book
Try to complete a 20-part multiple choice quiz in 10 minutes
Be stuck in traffic while late for a flight

It’s funny how our perception of time can vary so greatly.

The tech interview prep industry

Sun, 03 Apr 2022 17:55:05 -0400

Before I got into tech, I worked at a tutoring company in the Northern Virginia suburbs of Washington, D.C., doing a mix of schoolwork-related tutoring and standardized test prep.

From a business perspective, it made so much sense - you have kids taking standardized tests and parents who wanted their kids to do well on them. Here we have the SAT, a measurable evaluation that can be an immediate help to your future if you do well on it (though whether or not it’s truly a good evaluation is well beyond the scope of this essay). For better or for worse, it makes up a non-trivial part of the criteria used to evaluate college applications.

With that in mind, let’s play economist: if your SAT score allowed you better future prospects, how much would it be worth it to raise it by 100 points? Sure, we’d have to quantify those future prospects and the probabilities, but it generally seems like it would be a good thing. Furthermore, if you could pay someone to help your kid raise their score by 100 points, how much would you pay?

Even if there isn’t a direct value we can calculate for future benefit on SAT score increase, it seems like one of those things that should be true, at least. Talk about easy marketing! From a business perspective, this seems like the perfect fit; for every graduating student, there’s a new student who will be taking these tests soon enough, frequently with parents that are more than happy to pay for help along the way.

It’s a funny pattern:

A standardized test is developed
Doing well on that test is beneficial
People are willing to pay for help to do better, especially if it’s for their kids
Businesses pop up to fill that need

You can see many examples of this pattern in many standardized tests in the US: the SAT, ACT, PSAT, GRE, MCAT, LSAT…

You see this with tech interviews as well.

I don’t know who first came up with this idea of having a candidate work through a tricky algorithm problem as part of their interview, but it has created a beast. Like a butterfly flapping its wings across the world to create a hurricane, we’re now in the depths of the algorithm interviews.¹

(Note: whether or not these kinds of interviews are good is also beyond the scope of this essay)

It’s really an odd game, these interviews. The rules are mostly there for you to see before you play, with books detailing the process, online sites dedicated to the questions asked, and many avenues to practice. Interestingly, some companies also tell you about the rules and what to expect, even going as far as recommending specific subject areas to study (or, helpfully, that they won’t be asking any dynamic programming questions). Some of the companies even recommend specific sites to practice on.

Where did these sites come from? I think it’s the same pattern as before:

Companies started doing algorithm interviews
Doing well on those interviews can be very beneficial
People are willing to pay for ways to do better
Businesses pop up to fill that need

The similarities to the test prep industry are very interesting. Is an algorithm interview reflective of how an engineer works in their day-to-day job? Along those lines, what is the SAT really measuring? There’s this odd gap of artificiality between the test and the actual thing it’s purportedly testing for. Much like how bootcamps have popped up to fill in some of the gap between studying CS in college and the skills you need to successfully create websites, these businesses have popped up to help you bridge this gap between what you have to do as an engineer (or student) and what you have to do in the interview (or standardized test).

Look at timing - even a little strategy about how to tackle questions in each SAT section can help students do better. Maybe you aren’t used to writing essays in 50 minutes, but that’s OK! With a little practice and timing (be sure to wear a watch), you can get more used to it. Is that really reflective of life after the test? Maybe you’ll get a multiple choice test in college you have to hurry for or have to write manually during an exam, but the hard part is all of the other learning you’ll have to do.

For interviews, you can practice the timing, too. These businesses can help you get used to solving problems quickly as you identify general patterns. Working as an engineer, will your bugs ever be timeboxed to a time limit? Maybe you have to hurry a little bit every now and then, but I certainly hope it isn’t always the case that you’re working under 45-minute time limits, writing everything from scratch.

The other compelling similarity is that doing well after passing the test is not necessarily related to your performance on the test itself. When my wife was studying for the LSAT (the Law School Admission Test in the US), she found a tutor who had done tremendously well on the LSAT and pivoted that into a job helping other people study for the LSAT. I found this so interesting, because the mismatch between the test and working as a lawyer is just as striking (not to mention the mismatch between law school and working as a lawyer, but that’s also outside the scope here). Here was a test for preparedness to do well in law school (mostly in name, of course), but it clearly doesn’t take a lawyer to teach it.

With the market for helping others pass interviews, suddenly if you’re a few years into your career, there’s another avenue to build a brand doing interview prep. Just look up “coding interview tips” on YouTube - if you’re pretty good at interviews, you can leverage that into building a personal brand, and you get bonus points if you have some of the big companies already on your resume.

As a quick aside, I just wanted to note that there’s no disrespect meant towards those who are making money off of this; I’m more interested in this in a cultural way.

There are also a few interesting differences between this interview/business cycle and the standardized tests above. First, financially, the math is much clearer in favor of spending now to get a positive result later, as doing very well on these interviews can have a direct mapping to future benefits. If you’re able to land a job that increases your salary by $50,000, suddenly the $40 you spend on a book and the $35 you spend on an online subscription pale in comparison to the upside.

Additionally, there’s a strong self-fulfilling cycle here. Let’s say Company A does algorithm interviews and they know that there’s a gap between an engineer’s day-to-day responsibilities (such as writing good code to solve real business needs) and what you have to do in these interviews, so maybe they recommend a few test-prep sites to the candidate so they can get a little practice in. This would be like if signing up for the SAT came with a recommendation for a Kaplan study book - free marketing!²

What are we really measuring here?

If you’re measuring how someone communicates and thinks through a problem, what do you do when there’s a book that walks them through how companies want you to communicate and think through a problem in an interview?

If you’re trying to see how someone thinks about building complicated systems, what do you do when there are YouTube videos that walk them through how companies want them to work through building a complicated system?

If your company has to recommend practicing in advance of the interview, is that indicative that the interview could be better? What about when there’s an entire industry dedicated to being better at technical interviews?

What a weird balance.

It makes me simultaneously want to do all of the following:

Start an interview prep startup/personal brand and make money off of it, because shovels sell well in a gold rush and we might as well make a buck. Ideally we get some of these big companies to recommend our product, because that’s great marketing.³
Swear off technical interviews, throw away all of my computers, and live in the woods without electricity.
Play the dang game and do the stupid practice problems, because the upside is huge and real estate is expensive.
Work hard to find better and more equitable ways of interviewing candidates in a kind and human-focused way.

I definitely don’t have all the answers. I think I’ll just focus on #4.

Luckily, some companies are moving away from the usual format, with new debugging and pair-programming interviews. Many companies are still in deep. ↩︎
New startup idea: sell ad space in recruiter prep emails to interview prep companies. Extra points if we dogfood it and do that for the startup’s recruiting emails. ↩︎
Now seeking investors for the world’s worst interviewing site - since we’re in a race to the bottom, let’s be the first to define that lower limit! ↩︎

Keeping a debugging lab notebook

Sun, 27 Mar 2022 13:09:00 -0400

I like to keep a debugging lab notebook when I’m working through a particularly tricky problem.

It’s nothing fancy, just a new tab visit to docs.new and a quick @today to add today’s date. I usually try to describe the problem and the specific issue I’m having, including any reproduction steps or screenshots (frequently copied over from the issue tracker). I try to summarize the issue and my understanding of how things work, though this frequently leads to a handful of open questions for me to go through to really understand what’s happening.

What usually follows is a freeform jumble of writing down hypotheses, results, error messages, screenshots, explanations, and log statements (something like print('alex here 7.5') is a classic, since if you add another log statement between existing ones, you can add another decimal place to show you’re in between). I try to err on the side of including too much information so that I can shift everything out of my immediate working memory onto something a little more permanent, giving a clear history of what I’ve attempted as I work through the problem, but also importantly, what open questions remain and what things I should attempt next.

This helps me in two main ways. First, it encourages me to take a minute to explain in writing what I’m seeing — if I can’t explain it, I probably don’t fully understand it. Much like rubber duck debugging, this makes me think it through and work through what’s happening, which is frequently enough to help me realize an important insight.

The second main benefit ties into this as well, as the process of working through and writing things down encourages me to be more deliberate about the debugging path I take. It can be so tempting to just make a quick edit as a shot in the dark and rerun the test, and I’ve definitely done this plenty of times before. I find that I’m much more effective when I’m more deliberate about things, taking the time to explore each result and the next potential steps before moving forward.

It’s like the joke about the best way a junior engineer can level up: actually reading the error messages!

Many of these benefits show up as well when you’re pair programming or even just explaining the issue to your rubber duck (or curious pet). The other thing I like about this written approach is that it’s saved for you. When you come back to it later, you can retrace your steps, copy the same saved output or rerun a test, and page the context back into your working memory. My brain works much better as a processing machine than as a storage device, and the more I can keep written down and off of my busy brain’s responsibilities, the better.

At some point, finally, the problem clicks, and you realize the key insight that helps you solve it. You write up a quick summary, update the issue, and throw a quick explanation into the PR. The working notebook, full of roundabout exploration and steps taken down many false paths, has done its job. Occasionally I’ll look up how I solved a similar problem before or ran a test in a certain way, but generally, I rarely ever come back to them. For these debugging logs, that’s ok; it’s much more about how they’ve helped with the journey through.

Detecting pitch with the Web Audio API and autocorrelation

Sun, 20 Mar 2022 13:00:15 -0400

I’ve been playing with the Web Audio API recently, and I made a basic app that will detect the pitch of incoming tones using your microphone. You can try it by pressing “Start” (be sure to try both the sine wave and the frequency displays), and I’ll be walking through how it works below.

Press start to begin

Rounding options:

No rounding
Round to nearest Hz
Round to nearest note

Smoothing options:

No smoothing
Basic smoothing
Very smoothed (note must be more consistent and held for longer)

Display (click Start after you switch):

Sine wave
Frequency

Here’s what the pitch detection looks like.

How it works

This basic app relies on the Web Audio API, a powerful module in our browsers that lets us directly work with noise with JavaScript. I also leveraged this when making Morse Codle and my Morse code games, which you can read about here.

The Web Audio API works by connecting a series of nodes to an AudioContext. For this setup, I wanted to connect the user’s microphone as an incoming stream to an AudioContext. I also wanted to attach an AnalyserNode to the stream, which would allow me to access real-time data about the stream.

This is all pretty straightforward, especially if we follow some existing examples from MDN on visualizing the Web Audio API. First, we can access the microphone via getUserMedia. We can check if getUserMedia is valid and, if so, try to access the user’s microphone, which creates the permissions check for the user. If that’s successful, then we can connect our nodes and start the application.

var source;
var audioContext = new (window.AudioContext || window.webkitAudioContext)();
var analyser = audioContext.createAnalyser();
analyser.minDecibels = -100;
analyser.maxDecibels = -10;
analyser.smoothingTimeConstant = 0.85;
if (!navigator?.mediaDevices?.getUserMedia) {
  // No audio allowed
  alert('Sorry, getUserMedia is required for the app.')
  return;
} else {
  var constraints = {audio: true};
  navigator.mediaDevices.getUserMedia(constraints)
    .then(
      function(stream) {
        // Initialize the SourceNode
        source = audioContext.createMediaStreamSource(stream);
        // Connect the source node to the analyzer
        source.connect(analyser);
        visualize();
      }
    )
    .catch(function(err) {
      alert('Sorry, microphone permissions are required for the app. Feel free to read on without playing :)')
    });
}

For the main application loop via visualize, we want to do the following:

Draw the sound data on the canvas
Process the incoming sound data to calculate the frequency

Drawing the sound data on the canvas

Because I was more interested in the processing side, I leaned on the examples in the MDN Visualizations with Web Audio API page, particularly the examples from the Voice-change-O-matic. You can see the code snippets I borrowed here.

Processing the incoming sound data

The real difference between that example app and this one is that we want to calculate the main frequency of the incoming sound, and that’s where the interesting part of it is.

Can we just use the AnalyserNode’s getByteFrequencyData?

The AnalyserNode actually already has all of the data we need, and it even exposes the getByteFrequencyData method, which populates a buffer with the frequency bucket intensity level via Fast Fourier transform.

The main problem is that if we’re trying to get a specific frequency, we can’t just look at the buckets (shout out to this stackoverflow question). If we have a FFT size of 2048, sample rate of 48,000 Hz, our AnalyserNode will generate frequency bins across the range from [0Hz, 24000Hz], with each bin being max_frequency / frequencyBinCount, where frequencyBinCount is FFT size divided by 2. In this case, the bins would 24000 / 1024 == 23.4Hz wide. In other words:

fData[0] is the strength of frequencies from 0 to 23.4Hz.
fData[1] is the strength of frequencies from 23.4Hz to 46.8Hz.
fData[2] is the strength of frequencies from 46.8Hz to 70.2Hz.
fData[3] is the strength of frequencies from 70.2Hz to 93.6Hz.
...
fData[511] is the strength of frequencies from 11976.6Hz to 12000Hz.
fData[512] is the strength of frequencies from 12000Hz to 12023.4Hz.
...
fData[1023] is the strength of frequencies from 23976.6Hz to 24000Hz.

The issue here is that these buckets wouldn’t be exact enough to differentiate between notes in the lower ranges (you can see the note to frequency mapping here). For example, if we want to tune the top string of our guitar in the regular tuning, E, we would want to tune it close to 82.41 Hz (assuming we want the exact match - we won’t be getting into any more advanced theory here). If we want to tune it to F instead, we would want to tune it to 87.3 Hz. If all we know about the sound of the string is that it’s in the 70.2-93.6Hz bucket, we wouldn’t be able to tell which one we’re closer to.

One way to get around this would be to decrease bucket size by increasing the FFT size - we could do this, but our calculations would be more expensive, we’d require more space for this, and we’d have to bump the size up 10-fold to even get closer to to fine-tuned distances. Additionally, we don’t actually need these small bucket sizes for the higher frequency notes, which means we would have a ton of wasted bucket space.

Autocorrelation: we can do some quick math ourselves

Instead of relying on the getByteFrequencyData, we can instead access the sound information directly and calculate the frequency ourselves. One simple way to do this is autocorrelation, a technique where we compare part of a signal, in this case a sound wave, with a delayed copy of itself. By looking at how the signal compares to itself with a given offset, then varying that offset, we can calculate the offset at which the pattern roughly repeats. That would represent the period of the sound wave, and we could easily then calculate the frequency.

Let’s say we have a very simple data series representing the signal at different time steps:

[0.0, 0.5, 1, 0.5, 0.0, -0.5, -1, -0.5, 0.0, 0.5, 1, 0.5, 0.0, -0.5, -1, -0.5]

That might look something like this, where the x-axis is time and the y-axis is amplitude:

We want to compare this signal against what it would look like at different offsets, for example, with offsets 1 and 2:

Original: [ 0.0,  0.5, 1.0, 0.5, 0.0, -0.5, -1.0, -0.5,  0.0,  0.5, 1.0, 0.5, 0.0, -0.5, -1.0, -0.5]
Offset 1: [       0.0, 0.5, 1.0, 0.5,  0.0, -0.5, -1.0, -0.5,  0.0, 0.5, 1.0, 0.5,  0.0, -0.5, -1.0]
Offset 2: [            0.0, 0.5, 1.0,  0.5,  0.0, -0.5, -1.0, -0.5, 0.0, 0.5, 1.0,  0.5,  0.0, -0.5]

You can kind of imagine it like this, with multiple graphs at offsets 1 and 2 respectively:

To see where our graph repeats itself, we can look at each potential offset value and look at what it would look like if we multiplied each point on the graph by its offset value. Because we’re expecting a sinusoidal wave, the sum of those products would be highest when the wave repeats itself, as it would basically be squaring each value.

For example, for Offset 2, we would multiply each original value by the value of the offset graph:

Original: [ 0.0,  0.5, 1.0, 0.5, 0.0, -0.5, -1.0, -0.5,  0.0,  0.5, 1.0, 0.5, 0.0, -0.5, -1.0, -0.5]
Offset 2: [            0.0, 0.5, 1.0,  0.5,  0.0, -0.5, -1.0, -0.5, 0.0, 0.5, 1.0,  0.5,  0.0, -0.5]
Product:  [            0.0, .25, 0.0, -.25,  0.0,  .25,  0.0, -.25, 0.0, .25, 0.0, -.25,  0.0,  .25]
Sum: 0.25

Or, another way to think about this, we would multiple each value in the original by its value two timesteps later:

This yields a total of 0.25 for this offset - not a strong contender!

You could imagine doing this for each offset. At some point, if our wave is regular, we’ll reach the point where the wave repeats itself. In this case, with offset 8, we would be multiplying the points by themselves, as the full offset means the graph repeats.

Original: [ 0.0,  0.5, 1.0, 0.5, 0.0, -0.5, -1.0, -0.5,  0.0, 0.5, 1.0, 0.5, 0.0, -0.5, -1.0, -0.5]
Offset 8: [                                              0.0, 0.5, 1.0, 0.5, 0.0, -0.5, -1.0, -0.5]
Product:  [                                              0.0, .25, 1.0, .25, 0.0,  .25,  1.0,  .25]
Sum: 3.0

With a sum of 3, this would be the largest sum, and we’d know that our wave repeats itself every 8 timesteps. If our total time data length is 1 second cut into 15 time steps, then our frequency would be 15 / 8 = 1.875 Hz.

What would this look like in code? We could do the following:

// Start with a bufferLength of the analyser's FFT size
var bufferLength = analyser.fftSize;
// Actually create the buffer
var buffer = new Float32Array(bufferLength);
// Populate the buffer with the time domain data
analyser.getFloatTimeDomainData(buffer);

// Create a new array of the sums of offsets to do the autocorrelation
var offsetSums = new Array(bufferLength).fill(0);
// For each potential offset, calculate the sum of each buffer value times its offset value
for (let offset = 0; offset < bufferLength; offset++) {
  for (let j = 0; j < SIZE - i; j++) {
    offsetSums[offset] = offsetSums[offset] + buffer[j] * buffer[j+offset]
  }
}

// Calculate the offset with the highest value
var maxValue = -1;
var bestOffset = -1;
for (var i = 0; i < offsetSums.length; i++) {
  if (offsetSums[i] > maxValue) {
    maxValue = offsetSums[i];
    bestOffset = i;
  }
}

// Once we have the best offset for the repetition, we can calculate the frequency from the sampleRate
var frequency = sampleRate / bestOffset

That’s pretty much it! All that’s left is some UX to smooth the display changing (we don’t want to flash a new value every animation frame, though you can try it above) and translate the frequency into a note.

You can see the full source code here.

Can we do better?

One of the drawbacks of this basic autocorrelation is that it has trouble with sounds with multiple stacked notes - think of a piano note that actually has that same note layered across different octaves. The basic detector does well with my 440Hz tuning fork, but when I try to tune the A string on my guitar and play a more complicated sound wave (the guitar string vibrating), it will switch between detecting the 110Hz tone and 440Hz, depending on how close I am to the microphone. Another drawback is that it’s not as efficient as it could be; since we’re calculating each offset across the entire sample, leading to quadratic complexity.

There are also many ways to tune this basic detection further, such as trimming the sound data to only run on interesting parts of the tone and doing parabolic interpolation, where we assume that our peak isn’t a sharp point on the graph but instead a smooth parabola between the points on either side - we can use the highest point of the parabola for even more accuracy.

I chose this basic approach because 1) it’s easy to implement and follow along with the code and 2) it works pretty well for my use case. If you’re interested in more of the intricacies, I highly recommend checking out the pitch detection algorithm page on Wikipedia; definitely a rabbit hole!

This will eventually be part of an interactive ear training app, where you will listen to a tone and have to repeat it — this pitch detection will be the backbone of processing the incoming sound and translating it to the note.

It’s a good start for now!

I prefer my biographies in chronological order, thank you very much

Sun, 13 Mar 2022 18:52:21 -0400

Usually, biographies go through someone’s story in roughly chronological order.

OK, that’s a little too linear, like a page-a-day journal for your entire life. Realistically, you’re going to focus on different parts of the subject’s life more than others.

You can imagine that broken up into chapters, too. Maybe our subject’s childhood is generally passed over in favor of the more interesting middle-life.

This general flow makes sense, because you can do some fun foreshadowing by mentioning things that happen in the subject’s future (orange in the diagram below), and you can remind the reader of things that happened in the subject’s past with past references (purple in the diagram below). This makes sense, since you, the reader, are generally learning about things in the subject’s life as they happen.

I recently read a biography of Thomas Edison that turned this model around. Each chapter was a decade of Edison’s life, but it was as if you chopped the chronological timeline into chunks and reversed them. Here’s what the contents page looks like:

You could imagine us drawing this like before, roughly like the following, where the first chapter was Edison as an old man and the last chapter is about the years following Edison’s birth.

If we drew in the line following from birth to death, we’d have to have these dotted line jumps from the end of the chapter to the beginning of the previous chapter:

This meant that those two I mentioned earlier, foreshadowing in orange (talking about events that happen later in Edison’s life) and past references in purple (talking about events that happened earlier in Edison’s life), looked a little odd, like the following:

This meant that some references to things in Edison’s past were actually pointers to later parts of the book, while some of the forward-looking foreshadowing referred to parts you had already read. Not all references were switched, though, since time progressed forward within each chapter.

This made for some initial confusion on my part when reading it at first. Edison would have a relationship with a person that I hadn’t met yet, or there would be a reference to a chronologically-earlier but book-later event that I didn’t yet know about. It also made the chapter transitions feel like sudden cuts, as you’d meander from 1900 to 1909 throughout the course of a chapter, then suddenly spring back to 1890 at the start of the next chapter, suddenly faced with making your way back towards 1899.

As much as I joke that a wayward junior editor mixed up the chapter order in the final manuscript right before it went to the printers, I do think this approach adds to the story in an interesting way. It was a little like learning about your grandparent’s early life, as they already exist in your mind as grown adults, but learning about their childhood adds depth beyond the adult you know. This nonlinear splitting also meant that some “foreshadowing” was even more interesting, as you could see the ripples of this curious young man’s mind having already flowed through to future inventions with his later life already established in your head.

Nonlinear storytelling can be such an interesting component of a story, where the jumbled timeline plays with our expectations of story and causality. Consider Memento or Slaughterhouse-Five and how their nonlinearity adds such depth and flavor. As dismissive as I may sound in the title, I do think this was an interesting approach I hadn’t seen much before (though I’m far from an expert in the art of biography), but in this case, I think I would have enjoyed this shift more if I had already read another detailed Edison biography. With my mind pre-seeded with the character list and the major events, it would have been easier to follow the threads and connect the dots.

I look forward to seeing how it feels when I come back to it in the future.

My lizard brain is no match for infinite scroll

Sun, 06 Mar 2022 17:28:14 -0500

I’m declaring defeat. Infinite scroll has won.

There is an endless supply of interesting things to consume online, and there are teams and teams of very smart people working very hard to get me to keep watching. One more video means one more metric counter incrementing, one more engagement click, one more step towards growth, ad revenue, product success…

The danger for me is that I fall in. It’s so easy to open an app in an automatic movement of muscle memory, filling up an empty moment without conscious thought. Suddenly that content is there, and it’s so easy to go to the next image, post, or video. 30 seconds later, once you’ve reached the end, they’ll even scroll to the next one for you.

It’s so easy to get stuck in it. Time melts away. I wasn’t planning to spend 30 minutes lying on the couch looking at my phone, but here we are. It’s far too easy, and since they know exactly what I find interesting, there’s almost no barrier to keep looking.

I think my main problem with it is that it makes me a passive recipient in my interactions with this device, instead of an active participant. I didn’t know I wanted to see how these hand-made skis are made and finished, but then again, I also didn’t know I’d be down here on the couch for 45 minutes.

When I take a minute to think about the things I enjoy doing with my devices, it helps me realize that they’re the ones where I’m deliberately using it. Talking to people I know, for example. Watching that movie I had been looking forward to. Looking up the origin of an oddly spelled word. Creating, rather than just consuming, and using it as a tool to improve my life, even if that little improvement is a one word answer to a tiny question that had been bugging me.

But the content — it’s a default. It’s a comfort when you’re tired, when it’s just easier to look. Worse yet, by defaulting to this little screen, it gets in the way of me spending my time deliberately. I really enjoy drawing, and usually, even if I’m tired, if I grab a pencil and some paper, I end up having so much fun scratching out little sketches. If I had gotten up and started drawing, I probably would have remembered how fun it is. But, I’m a little tired. Going for a walk sounds tiring, and it’s not quite bedtime yet, so I’ll just take a glance at a few more videos since I’m already here. My feet are a little cold from being here on the couch for an hour, but luckily I’ve got a blanket as well.

Leisure time is so important, and I have no time for hustle culture. My sacred down time is under attack by these patterns, all from apps that want me spending more time on them with plenty of ways to hook me in. I enjoy turning off my brain and relaxing, but I’ve noticed it’s so much better when it’s on my terms. When I think about how I feel after 30 minutes of letting my mind wander on a walk versus 30 minutes of scrolling, it’s night and day.

I don’t have any quick fixes or easy answers. I’ve struggled with this for a very long time. I’ve gotten much better at dealing with it, but I find I have to remain conscious of it. That’s where admitting defeat helps; I know how my brain works, and I can work with it. Let’s not install that app with the infinite scroll, since we can probably get by with just the mobile web version. Let’s not log in, unless there’s a reason you need to, since they’re after you with recommendations for your account. Let’s try to be conscious of how much time you end up spending on certain sites.

There’s a broad pattern these days of mobile sites being much worse and more restrictive, since they’re trying to push you towards logging in or downloading the app. Ironically, being conscious of my struggle, these moments of friction actually help push me towards closing the site instead. For that, I’m grateful for Reddit’s mobile UI.

I don’t think I’ll be downloading the app today.

I miss easily seeing how my coworkers use tools

Sun, 27 Feb 2022 13:58:57 -0500

It’s one of the little things I miss about working in-person with other people.

Writing software can involve so many different tools. Before my current remote working times, one of the ways I picked up on efficient ways to use some of the tools involved was from seeing how coworkers navigated them, especially when they had a lot more practice with them.

A colleague and I would be looking at their screen together, working with a tool that I had an OK understanding of, when they’d use a feature or shortcut I didn’t even know about. It may have been muscle memory to them, but it would be brand new to me.

Wait, how did you just do that?

One of the most staggering things about ramping up at Google was the number of internal tools and systems that were brand new to me. I learned many of them by trying them out and following patterns of how I saw others using them, slowly over time building my familiarity with them until things became second nature. Even after I was comfortable, there would still be power-user techniques that I only picked up on from seeing a coworker do them on their screen.

What button was that again?

I’ve noticed that this discoverability is harder for me since I’ve been working remotely. It still happens as we share screens, record talks, and screenshot pages and buttons for each other, but it has never felt quite as natural. Maybe because there’s some inherent cognitive load to screen sharing, requiring at least a few mental cycles to even just keep up with what someone is doing. Add in any delays, sudden window changes, or small fonts, and it’s even trickier.

Maybe it’s that we often only share a single screen or window, with text blown up so we can all see in our browser windows or Zoom clients. I wonder if having the exact same monitor setup as the other person and having their monitors mirrored on mine would help, since I would have the full context of what they’re doing.

Sorry, do you mind zooming in a bit?

There may also be some natural barrier in my head to looking at a screen together remotely. It’s probably latent Zoom fatigue, but there’s a difference between a text chat back and forth and hopping on a call. Working in-person, it’s likely that a discussion would be us talking about it already, so maybe the jump to “can I come see what error you’re getting?” doesn’t have as much overhead to it.

I’ve actively worked to push against this barrier. When working with more junior engineers or interns, I would force myself to remove it completely for them, jumping straight into chats and debugging without hesitation. For my regular individual contributor work with others, though, I still feel it, and I always try to stay conscious of it.

Want to hop on a quick call?

I think about this discoverability a lot for junior engineers or new team members ramping up. There are so many things to learn, and reducing barriers to sharing that knowledge can only help. As we continue to move forward with widespread remote work, how do we better enable easy and casual collaboration?

Morse Codle

Wed, 16 Feb 2022 21:03:02 -0500

oftendon't like

MORSE CODLE

Enter

Writing Morse code games in JavaScript using the Web Audio API

Sat, 12 Feb 2022 12:36:19 -0500

Last week, I posted a pair of games I made to play with Morse code in your browser. I wanted to take some time to describe how I made them and some of the challenges that came up along the way.

The first game is a listening game, where it plays the Morse code representation of a random target and you have to enter it in. The second game is a “speaking” game, where it gives you a random target string and you have to enter it in Morse code. The games include some options, including the speed that the Morse code plays (ranging from slow to fast for both the Farnsworth method and extending all times) and the difficulty (with the target ranging from a single letter to a number of words).

These games are written in vanilla JavaScript, and the source code is available here and here. It’s unpolished, but it works :)

Playing sound with the Web Audio API

The first thing that comes to mind for these games is the ability to play sound. You can do so many things with JavaScript, and luckily, this is one of them. The main interface for sound in the browser is the Web Audio API. I won’t go into too much detail about it here, but I highly recommend the MDN docs and the tutorials there for playing around with it.

For both of these games, I want to be able to play a single tone for a certain amount of time. The first step is playing the tone, and we can worry about the times later. One way to do this is to set up a new AudioContext (“an audio-processing graph built from audio modules linked together”) with an OscillatorNode (“an AudioNode audio-processing module that causes a given frequency of wave to be created”) and a GainNode (“an AudioNode audio-processing module that causes a given gain to be applied to the input data before its propagation to the output”) for controlling volume. Simply put, we’ll want to be able to create a tone at a certain frequency using the OscillatorNode, and we’ll want to start and stop the tone by adjusting the volume with the GainNode.

Before we do anything, we’ll need to initialize the AudioContext, add the nodes, and connect them to the context. We can set up the OscillatorNode with a frequency of 440hz, A, which is a reasonably pleasant Morse code tone. This might look something like the following:

const FREQUENCY = 440;

let note_context;
let note_node;
let gain_node;
let audioContextInitialized = false;

function initializeAudioContext() {
  note_context = new AudioContext();
  note_node = note_context.createOscillator();
  gain_node = note_context.createGain();
  note_node.frequency.value = FREQUENCY.toFixed(2);
  gain_node.gain.value = 0;
  note_node.connect(gain_node);
  gain_node.connect(note_context.destination);
  note_node.start();
  audioContextInitialized = true;
}

With that audio context, we can then make functions for starting and stopping playing, which really adjust the gain value in the GainNode:

function startNotePlaying() {
  // Pass a start time of 0 so it starts ramping up immediately.
  gain_node.gain.setTargetAtTime(0.1, 0, 0.001)
}

function stopNotePlaying() {
  // Pass a start time of 0 so it starts ramping down immediately.
  gain_node.gain.setTargetAtTime(0, 0, 0.001)
}

Additionally, we’ll want to keep track of whether or not the audio context is initialized, so the first time we’re planning to play a sound we can initialize it if it hasn’t been already. As a side note, for mobile users, it’s important that this initialization is tied to a user action; we can’t just start playing sound without the user’s input (I know nothing about the code involved, but I’d bet this is why YouTube videos on mobile Safari are initially muted).

You can try these out with the following buttons:

Not too bad! The next step is being able to play them for a given amount of time, since Morse code is made up of short signals (dots) and longer signals (dashes). For this, we can do a subtle trick with JavaScript’s async and Promises. JavaScript doesn’t really have a sleep method, but we can simulate one by creating a Promise that resolves itself after a certain amount of time. We can then await that function in an async function; this will cause that async function to pause at that function call until the amount of time has passed. Because of JavaScript’s event loop, it isn’t an exact timing, but it’s close enough for this!

To play a note for a certain amount of time, we can start the note playing, wait for the given time, then stop playing the note. That could look something like the following:

// Times in milliseconds; note that Morse code dash time is 3 * dot time
var DOT_TIME = 60;
var DASH_TIME = DOT_TIME * 3;

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

async function playDash() {
  startNotePlaying();
  await sleep(DASH_TIME);
  stopNotePlaying();
}

async function playDot() {
  startNotePlaying();
  await sleep(DOT_TIME);
  stopNotePlaying();
}

You can try these out with the following buttons:

Playing Morse code

Morse code timing

Speaking of those time delays, let’s take a moment to look the timing of Morse code. The canonical ratios are as follows (from MorseCode.World):

The timing in Morse code is based around the length of one “dit” (or “dot” if you like). From the dit length we can derive the length of a “dah” (or “dash”) and the various pauses:

Dit: 1 unit
Dah: 3 units
Intra-character space (the gap between dits and dahs within a character): 1 unit
Inter-character space (the gap between the characters of a word): 3 units
Word space (the gap between two words): 7 units

Or, in code:

var DOT_TIME = 60;  // If our unit is 60ms.
var DASH_TIME = DOT_TIME * 3;
var SYMBOL_BREAK = DOT_TIME;
var LETTER_BREAK = DOT_TIME * 3;
var WORD_BREAK = DOT_TIME * 7;

Playing letters, words, and sentences

Using our async and sleep tricks from earlier, we can start to build higher order functions to play our Morse code letters. For a given character, we’ll want to play each symbol with a symbol break in between. We can write a playLetter function that takes in a letter’s Morse code representation (e.g. ".-." for "r") and plays it.

/**
 * letter is something like '---'
 */
async function playLetter(letter) {
  if (!audioContextInitialized) {
    initializeAudioContext();
  }
  for (let i = 0; i < letter.length; i++) {
    if (letter[i] == '-') {
      await playDash(currentPlayCounter);
    } else if (letter[i] == '.') {
      await playDot(currentPlayCounter);
    }
    await sleep(SYMBOL_BREAK);
  }
}

playLetter('.-.');  // "r" is ".-." in Morse code

We can then build this up one level higher, into playing a word. Now, we’ll want to play each letter, then wait for the duration of the letter break:

// Word is an array of letters as Morse code, like ['.', '.-', '-']
async function playWord(word) {
  for (let i = 0; i < word.length; i++) {
    await playLetter(word[i]);
    await sleep(LETTER_BREAK);
  }
}

playWord(['.', '.-', '-']);  // "eat" is ". .- -" in Morse code

Finally, we can do one more higher order function to play a sentence, again reusing what we’ve already done. We’ll play each word, followed by a word break.

// Sentence is an array of words.
// Ex. "dog is good" -> [['-..', '---', '--.'], ['..', '...'], ['--.', '---', '---', '-..']]
async function playSentence(sentence, currentPlayCounter) {
  for (let i = 0; i < sentence.length; i++) {
    await playWord(sentence[i], currentPlayCounter);
    await sleep(WORD_BREAK);
  }
}

playSentence([['-..', '---', '--.'], ['..', '...'], ['--.', '---', '---', '-..']]);  // "dog is good" in Morse code

Writing each word out in Morse code is pretty rough, though. We can write a translation layer to translate from a string (anything from a letter to a sentence) to the Morse code representation, starting with a map of letter to Morse:

const MORSE_MAP = {
  'a': '.-',
  'b': '-...',
  'c': '-.-.',
  'd': '-..',
  'e': '.',
  'f': '..-.',
  'g': '--.',
  'h': '....',
  'i': '..',
  'j': '.---',
  'k': '-.-',
  'l': '.-..',
  'm': '--',
  'n': '-.',
  'o': '---',
  'p': '.--.',
  'q': '--.-',
  'r': '.-.',
  's': '...',
  't': '-',
  'u': '..-',
  'v': '...-',
  'w': '.--',
  'x': '-..-',
  'y': '-.--',
  'z': '--..',
};
// asciiChar is something like 'd'
// Assumes [a-z0-9]
function convertAsciiCharToMorse(asciiChar) {
  return MORSE_MAP[asciiChar];
}

// asciiWord is something like 'dog'
// Assumes [a-z0-9]
function convertAsciiWordToMorse(asciiWord) {
  return asciiWord.split('').map(convertAsciiCharToMorse);
}

// asciiSentence is something like 'dog is good'
// Assumes words are separated by spaces.
function convertAsciiSentenceToMorse(asciiSentence) {
  let splitSentence = asciiSentence.toLowerCase().split(' ');
  return splitSentence.map(convertAsciiWordToMorse);
}

We can then use these higher order functions to translate our strings first.

let targetString = 'dog is good';
playSentence(convertAsciiSentenceToMorse(targetString));

With that, we’re able to play Morse code! We’ve built our playing up from playing a single tone for a duration to translating entire sentences and playing them.

Adjusting the speed

One of the things I wanted to do for these games was to make the speed adjustable, since it’s much easier to go slow when you’re first learning. This brought up an interesting question though: how should you slow down Morse code to learn?

My naive approach was to just adjust the dot value and keep the ratios all the same. This would (I presumed) give you plenty of time to listen carefully and learn.

Try out the following values with a very slow 300ms dot:

It turns out this isn’t actually helpful, since a large part of learning Morse code is being able to recognize and parse the sound of entire letters and words. If you slow everything down, you don’t build the familiarity with identifying the real sounds at the real speeds. Instead, there’s a technique called the Farnsworth method, where the individual letters are kept at the same speed, but the space between them varies. This gives you the benefit of hearing the letters at full speed with a longer break to process in between characters.

Thanks to the commenters on Hacker News for recommending this; I had never heard about it!

If we have some difficulty input, we can do something like the following to vary the spacing in between characters and words:

function updateSpeed(difficulty) {
  // Get the difficulty and update the constants.
  DOT_TIME = 60;
  DASH_TIME = DOT_TIME * 3;
  SYMBOL_BREAK = DOT_TIME;
  let letterBreakMultiplier;
  switch (difficulty) {
    case 'easy':
      letterBreakMultiplier = 24;
      break;
    case 'medium':
      letterBreakMultiplier = 12;
      break;
    case 'hard':
      letterBreakMultiplier = 6;
      break;
    default:
      letterBreakMultiplier = 3;
      break;
  }
  LETTER_BREAK = DOT_TIME * letterBreakMultiplier;
  WORD_BREAK = DOT_TIME * (letterBreakMultiplier * 2.5);
}

Now, try listening to "eat" again, now with the Farnsworth method. Listen to how each letter is at full speed, but you have space to think in between!

With all of the playing and the timing figured out, we can now move to the games.

The Listening Game

Now that we have the ability to play an arbitrary phrase, ranging from one letter to complete sentences, we can use that to make the first game, the listening game. This game works as follows:

Get a random target phrase depending on the difficulty
Play the phrase
Wait for the user to type it in
- If they get it, tell them and give option to play again
- If not, play the sound again

First, let’s look at how we’d get a random target. Let’s say we have an input with a few different radio buttons:

function getTarget(difficulty) {
  // Get the difficulty and assign a random target.
  let target;
  switch (difficulty) {
    case 'easy':
      target = getRandomLetter();
      break;
    case 'medium':
      target = getRandomWord();
      break;
    case 'hard':
      target = getRandomEasyWords();
      break;
    default:
      target = getRandomWords();
      break;
  }
  return target;
}

That’s not too bad! We just have to implement those child functions:

// Note: in the game, EASY_WORDS and ALL_WORDS are long arrays of strings.
// Here are a Shown here for example.
const EASY_WORDS = ['cat', 'ate', 'dog'];
const ALL_WORDS = ['cat', 'ate', 'dog', 'interest', 'complexity'];

function getRandomLetter() {
  const randomIndex = Math.floor(Math.random() * 26);
  return Object.keys(MORSE_MAP)[randomIndex];
}

function getRandomWord() {
  const randomIndex = Math.floor(Math.random() * EASY_WORDS.length);
  return EASY_WORDS[randomIndex];
}

function getRandomEasyWords() {
  const totalLength = Math.floor(Math.random() * 3) + 2;
  let finalSentence = ''
  for (let i = 0; i < totalLength; i++) {
    finalSentence += EASY_WORDS[Math.floor(Math.random() * EASY_WORDS.length)];
    if (i < totalLength - 1) {
      finalSentence += ' ';
    }
  }
  return finalSentence;
}

function getRandomWords() {
  const totalLength = Math.floor(Math.random() * 4) + 2;
  let finalSentence = ''
  for (let i = 0; i < totalLength; i++) {
    finalSentence += ALL_WORDS[Math.floor(Math.random() * ALL_WORDS.length)];
    if (i < totalLength - 1) {
      finalSentence += ' ';
    }
  }
  return finalSentence;
}

With these in hand, we can write the basic game functions:

// The listening game
let target = '';

function startGame() {
  // Initialize audio context if it isn't already
  if (!audioContextInitialized) {
    initializeAudioContext();
  }
  // Get a random target, assume we're on medium mode
  target = getTarget('medium');
  // Play the target phrase
  playTarget();
}

function playTarget() {
  playSentence(convertAsciiSentenceToMorse(target));
}

function guess(enteredGuess) {
  if (enteredWord == target) {
    console.log('Correct! call `startGame()` to play again.');
  } else {
    console.log('Not quite! Try again.');
    playTarget();
  }
}

That’s the main functionality right there! You can actually play this minimal version in your console on this page:

All that’s left to do with this is to clean it up and hook it up to the user input (buttons and keyboard shortcuts) — since that’s mostly event listeners, I’m omitting it here, but you can see the full source here; look for the ListeningGame class.

Replaying the target

One subtle thing that came up with this game is how to allow the user to replay the target phrase, and mostly what would happen if they tried to replay it before the playthrough was finished.

I initially tried to manage this with a global STOPPED variable and a few checks; if the playSentence function saw that everything was stopped, it wouldn’t try to play any more. This quickly ran into problems related to the different sleeps I have in there: since I set it up so that a new playthrough would set STOPPED to false, this meant it could be reset while the earlier playSentence was still sleeping, leading to the original playthrough still continuing. This meant that two playthroughs would play at the same time, which quickly led to a terribly messing sounding audio.

I ended up fixing this by introducing a monotonically increasing playthrough counter global variable and introducing an invariant that only the most recent playthrough can be played. By incrementing this variable when a playthrough is initiated and passing the value to the child play calls, each child function can perform a check if its passed playthrough ID still matches the global current playthrough ID; if not, it means that there’s a newer playthrough and the current one should be discarded.

This looked something like this:

// Global strictly increasing play counter to avoid multiple replays at the same time.
var playCounter = 0;

async function playDash(currentPlayCounter) {
  if (currentPlayCounter != playCounter) { return; } // New line to short circuit if out of date
  ... rest of function
}
async function playDot(currentPlayCounter) {
  if (currentPlayCounter != playCounter) { return; } // New line to short circuit if out of date
  ... rest of function
}
async function playLetter(letter, currentPlayCounter) {
  if (!audioContextInitialized) {
    initializeAudioContext();
  }
  for (let i = 0; i < letter.length; i++) {
    if (currentPlayCounter != playCounter) { return; }
    ... rest of function
  }
}
async function playWord(word, currentPlayCounter) {
  for (let i = 0; i < word.length; i++) {
    if (currentPlayCounter != playCounter) { return; }
    ... rest of function
  }
}
async function playSentence(sentence, currentPlayCounter) {
  if (currentPlayCounter != playCounter) { return; }
  ... rest of function
}
playTarget() {
  playCounter += 1;
  playSentence(convertAsciiSentenceToMorse(this.target), playCounter);
}

This way, if playTarget was called again, the new playthrough would supersede the old one, and the old one would stop. Not the most elegant, but it removed the duplicate plays!

(Thanks to sirmarksalot on Hacker News for pointing out this bug!)

The “Speaking” game

The second game was a little trickier to write, as it’s much more dynamic than the first one. It requires the player to input the Morse code representation of a given target (again, depending on the difficulty, ranging from a single letter to a whole sentence).

You can imagine the rough steps are like this:

Get a random target phrase depending on the difficulty
Listen for user inputs
- Translate user input into signals and pauses
- Translate signals and pauses into letters and words
- Check if the user input the target correctly

Much more of a dynamic problem!

Parsing the incoming signal into Morse code

For an initial consideration, we can imagine just the part of parsing user signals. Since we’d be constructing letters from symbols, we can approach it like other “build up as we go” problems and use a short stack to keep track of what we’ve seen so far. Let’s imagine it going left to right, since that’s how we’d read it in English, and let’s imagine a user inputting "r", which in Morse code is ".-.".

At some point, we’ll process the symbols on the stack (more on that later) into a letter. We’ll want to build the letters up over time, so we can imagine another stack of letters:

If we’re dealing with sentences, you could imagine there being an additional stack for the words, too.

Like any interesting stack, the real question is when to read from it. This is where the pauses and silences come into play. Earlier we talked about the ratios for the pauses (the SYMBOL_BREAK, LETTER_BREAK, and WORD_BREAK from earlier). We can abstract away the actual input and imagine a decision tree like the following:

That gives us four events to handle: handleDot, handleDash, handleLetterBreak, and handleWordBreak. With our stacks from earlier, those could do the following, where we parse the symbol stack in handleLetterBreak and the letter stack in handleWordBreak:

let symbolStack = [];
let letterStack = [];
let wordStack = [];

function handleDot() {
  symbolStack.push('.');
}

function handleDash() {
  symbolStack.push('-');
}

function handleLetterBreak() {
  const letterAsMorse = symbolStack.join('');
  // Assume we have a mapping of Morse:letter,
  // the reverse of the earlier MORSE_MAP
  const letter = lookUpMorse(letterAsMorse);
  letterStack.push(letter);
  // Clear the symbol stack.
  symbolStack = [];
}

function handleWordBreak() {
  const word = letterStack.join('');
  wordStack.push(word);
  // Clear the word stack.
  wordStack = [];
}

This looks pretty good. We’ll be incrementally building up our inputs as we see them, from symbol all the way up to words, pushing to stacks and parsing them when appropriate. Now, how do we translate user input into these chunks?

Let’s start by imagining our signal as an electrical signal (much like our oscillation from before) as a square wave, to make it easy to see. Let’s say we have "r" as an input.

Parsing the high signal seems pretty easy, as we can just look at the time when the signal starts and the time when the signal stops. We can say it’s a dot if it’s shorter than a DASH_TIME, and if it’s longer, then it’s clearly a dash.Since humans aren’t perfect, we can be a little generous about the time.

let signalStartTime = 0;

function handleSignalStart() {
  signalStartTime = Date.now();
}

function handleSignalEnd() {
  // How long was this signal?
  let timeDelta = Date.now() - signalStartTime;
  if (timeDelta < DASH_TIME) {
    handleDot();
  } else {
    handleDash();
  }
}

If we tie these in to the points where the signal starts and stops, we can parse input into dots and dashes.

Now, how can we parse the silences? One initial approach could be to look backwards whenever a signal starts. Silence is the absence of signal, and every time a signal starts, we can look to see how long it has been since the last signal ended. This is the pause between signals, and we can parse it appropriately. We can even ignore the first silent section, since we don’t want a leading space before any input.

To add to the previous code, we now need to also keep track of the timestamp when the signal ended.

let signalStartTime = 0;
let signalEndTime = 0;

function handleSignalStart() {
  let currentTime = Date.now();
  // Keep track of signal start time as before.
  signalStartTime = currentTime;
  // Now, calculate how long that silence was.
  let silenceTimeDelta = currentTime - signalEndTime;
  if (signalEndTime == 0) {
    // This is the first silence and we can ignore it.
  } else if (silenceTimeDelta < LETTER_BREAK) {
    // This is just a symbol break and we can ignore it.
  } else if (silenceTimeDelta < WORD_BREAK) {
    // Anything more than a letter break and less than a word break counts
    // as a letter break.
    handleLetterBreak();
  } else {
    // We have a word break!
    handleWordBreak();
  }
}

function handleSignalEnd() {
  let currentTime = Date.now();
  // How long was this signal? Calculate delta as before
  let timeDelta = currentTime - signalStartTime;
  if (timeDelta < DASH_TIME) {
    handleDot();
  } else {
    handleDash();
  }
  // We also need to mark the timestamp that this signal ended.
  signalEndTime = currentTime;
}

Now, our parsing for "r" will look like this:

Close, but there’s a problem. If we’re relying on a new signal to know when the current silence is done, what if we don’t get a new signal? If the player has entered the last symbol, we should be able to automatically know that enough time has passed for a letter break. Additionally, if we’re dynamically processing the user’s input, we should be able to show it to them as it’s parsed. If we have to wait for the next signal to know that the current symbol is done, there would be a weird effect where you’d only see the current letter once you start the signal for the next letter.

For example, what if the player had to input "a"?

We need a way to proactively parse the input when enough time has passed. Instead of the backwards-looking approach above, we can instead use a forward looking approach. When we see a signal end, we can start two timers: one the for time period of a letter break, and one for the time period of a word break. When these timers go off, if we haven’t seen a new signal, we’ll know that we can handle the letter breaks and word breaks appropriately. If we do see a signal, we can just cancel the timers. This also allows us to show the parsed input to the user as soon as it has been parsed, which feels much more responsive.

Now, when a user inputs the word "at", we can proactively handle the silence.

To do this in the code, we can use a combination of creating timeouts and clearing them as we need to.

let signalStartTime = 0;

// Timeout IDs we'll keep around so we can clear them when needed.
let letterTimeout = null;
let wordTimeout = null;

function handleSignalStart() {
  signalStartTime = Date.now();
  // We also want to clear any pending silence timeouts.
  clearTimeout(letterTimeout);
  clearTimeout(spaceTimeout);
}

function handleSignalEnd() {
  // How long was this signal?
  let timeDelta = Date.now() - signalStartTime;
  if (timeDelta < DASH_TIME) {
    handleDot();
  } else {
    handleDash();
  }
  // Start the timeouts
  letterTimeout = setTimeout(handleLetterBreak, LETTER_BREAK);
  wordTimeout = setTimeout(handleWordbreak, WORD_BREAK);
}

Now that we have this, we can put it all together:

That’s really the hard part.

Hooking it up to user input

To make this feel dynamic, we can hook up the handleSignalStart function to run on keyDown for the spacebar and mousedown/touchstart on the Tap button and the handleSignalEnd function to run on keyUp for the spacebar and mouseup/touchend on the Tap button.

To make it feel real, we want to play a tone when the user is pressing down on those inputs. Since we did all of the tone playing before, all we have to do is add a simple call to startNotePlaying in handleSignalStart and the corresponding stopNotePlaying in handleSignalEnd.

The last step is adding a similar flow to the first game, where we generate a target, listen and parse input, and check if it matches. All that’s left to do with this is to clean it up, add a little more functionality for clearing, and hook it up to the user input (buttons and keyboard shortcuts) — since that’s mostly event listeners, I’m again omitting it here, but you can see the full source here; look for the InputGame class.

Finishing thoughts

I also cleaned up the games and put them in classes to keep things a little more cleanly separated. I also spent some time on some nice-to-haves, like auto-focusing on the text input when the first game starts and keyboard shortcuts to clear/reset.

You can play the games here and see the full source code here and here.

Feel free to reach out if you have any questions about this, and happy coding!

Little games to play with Morse Code in your browser

Fri, 04 Feb 2022 22:32:54 -0500

Be sure to also check out my other Morse code game, Morse Codle, a daily Morse code Wordle puzzle.

This is a pair of games for playing with Morse Code. Since the games play the signals as sound, I recommend headphones (wired if possible for the latency) and a low volume. Game options are available towards the bottom.

Morse Code Listening Game

In this game, it will play a word in Morse Code, and you have to type in the word.

Press Start to begin.

How to play the listening game

As you hear the Morse Code, enter the letters (or words!) that you hear. Press Enter (either the button or on your keyboard) to see if you got it, and press Replay (or Control on your keyboard) to hear it again.

For doing Morse to letter lookups, I find it helpful to look at a tree-shaped diagram, where you can follow from one node to the next based on whether it’s a dit or a dah. Source

Morse Code “Speaking” Game

For the speaking game, it will show a word, and you’ll have to enter it in Morse Code.

Press Start to begin.

How to play the speaking game

Press start to reveal the target. Enter Morse Code with your spacebar or the Tap button. Press Control or the Try Again button to clear your input and start over.

The game will assume you’re done with a letter after the letter gap interval (see options below), and it will automatically insert a space for you after the word interval; be careful you don’t take too long in between dits!

For doing letter to Morse lookups, I find it helpful to look at an alphabetical list with the matching representation. This makes looking up a letter faster. Source

Game options

After changing settings, you have to stop and restart the above games.

Difficulty

Please select your preferred difficulty:

Easy (letters)
Medium (short words)
Hard (multiple short words)
Very Hard (multiple words)

Speed

Please select your preferred speed:

Slow
Medium
Fast
Faster

Use the Farnsworth method?

Yes
No

The speeds match the following from Wikipedia:

The duration of a dah is three times the duration of a dit. Each dit or dah within an encoded character is followed by a period of signal absence, called a space, equal to the dit duration. The letters of a word are separated by a space of duration equal to three dits, and words are separated by a space equal to seven dits.

Speed	Dit (ms)	Dash (ms)	Letter gap (ms)	Word gap (ms)
Slow	300	900	900	2100
Medium	200	600	600	1400
Fast	100	300	300	700
Very Fast	60	180	180	420

But, that being said, the Farnsworth method varies the intervals between the letters and the word gap instead of varying the length of the dits, dashes, or character gaps. It is more helpful for learning the sounds of each letter at a real speed you would hear it.

All times are in milliseconds. I recommend starting with slow and giving yourself plenty of time for the dahs, then increasing the speed as you go!

Issues/questions

Something went wrong?

You may need to refresh — it’s a little rough around the edges.

Something went wrong on iOS?

First, make sure your phone is unsilenced (turns out a silenced phone blocks the web audio APIs). You also may need to tap Start and Stop a few times, refresh, or start with the “speaking” game. I ran into an issue with initializing the AudioContext, and although I think I got it right, I’m sorry if it’s giving you trouble.

Why is it so quick with the spaces when listening?

Sorry, it was easiest to be super strict about the timing when making it. Try slowing the speed down or practicing to go faster!

How does it work?

You can read about it here, and you can see the source code here and here, though it’s very unpolished.

Any recommendations for learning Morse Code for real?

I’m far from an expert, but so far I’ve seen a few recommendations (thanks to curiousfab and SaberTail on HN):

Writing a toy traceroute from scratch

Sun, 30 Jan 2022 10:59:16 -0500

traceroute is a tool you can use to trace the route of packets from your computer to another computer. It lets you see each step that your packets take along the way.

For example, the following is the result of running traceroute from my computer in Massachusetts to “example.com”:

You can see that it took 8 hops to get from my computer to the destination IP (93.184.216.34), and you can see the associated timing for three different attempts for each step. If we wanted to, we could map out these IPs to see the rough physical path of our packets too; from a quick whois, it looks like example.com’s server is based in Los Angeles, and there are quite a few ways to get there!

traceroute is made possible by a side effect of features in the Internet Protocol (IP). Every IPv4 packet header includes a number of fields:

One of the fields in the header is a “Time To Live” value, or TTL. The original DARPA RFC defined Time to Live in 1981 as follows:

The Time to Live is an indication of an upper bound on the lifetime of an internet datagram. It is set by the sender of the datagram and reduced at the points along the route where it is processed. If the time to live reaches zero before the internet datagram reaches its destination, the internet datagram is destroyed. The time to live can be thought of as a self destruct time limit.

In theory, this would be measured in seconds, representing a true self-destruct timer, but in practice, the TTL actually reflects the number of hops that the packet can still take on its way to the destination. At each router along the packet’s path, this value is decremented, and if it hits zero, the packet goes no further and an ICMP error message is returned to the sender. This helps prevent unintended behavior in a complicated network, such as infinite looping. Without the TTL, you could imagine orphaned and lost packets being forwarded indefinitely.

The ICMP “Time exceeded” message that is sent back to the original sender includes some basic information, and with the message, the address of the router that sent it is also visible.

traceroute

traceroute leverages this decrementing TTL functionality in a clever way to explore the network. Let’s say we’re trying to trace the route to C in the following network:

traceroute relies on artificially manipulating the TTL of outgoing packets, incrementally sending packets with incrementing TTL values. It starts with a TTL of 1; when that packet gets to the first router, the TTL will be decremented to 0, and the router will send back an ICMP message.

traceroute can then send a packet to the destination with a TTL of 2, and this packet will make it to the second router along the path before an ICMP is sent back.

Each packet will go one hop further than the last, and by plumbing the router at each hop depth and keeping track of the routers it receives messages from along the way, it can reconstruct a rough path to the destination host.

What would a basic implementation look like?

In order to recreate this, we’ll want to write some code that does the following:

Gets the destination IP of the example destination, say “example.com”
Creates a socket that allows it to send UDP packets
Creates a socket on which it’ll listen for ICMP messages
Initializes loop variables for keeping track of the current hop
Inside a loop while we haven’t reached the destination:
- Sends a packet with an incrementing TTL
- Receives an incoming ICMP packet
- Keeps track of the current hop and the address of the received ICMP message

That doesn’t sound too bad! This is all pretty approachable, and in Python, it may look something like this:


import socket

DESTINATION = 'example.com'
DESTINATION_PORT = 33434
MESSAGE = 'foo'


def main():
    # Get the destination ip address.
    destination_ip = socket.gethostbyname(DESTINATION)
    print('Tracing the route to {0}'.format(destination_ip))

    # Prepare a socket to send UDP packets.
    sending_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sending_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

    # Prepare a socket to listen for ICMP messages.
    # Note: the middle argument _may_ be SOCK_RAW, but since I'm running this
    # on a mac, I had to use SOCK_DGRAM to avoid needing root privileges.
    # See https://apple.stackexchange.com/questions/312857/how-does-macos-allow-standard-users-to-ping.
    receiving_socket = socket.socket(
        socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_ICMP)

    # Show the ICMP header, since that's where the router address is.
    receiving_socket.setsockopt(socket.SOL_IP, socket.IP_HDRINCL, 1)

    # Initialize the variables we'll use for seeing if we're done and keeping
    # track of the current hop.
    received_ip = None
    current_hop = 1
    while received_ip != destination_ip:
        # Set the socket's TTL to the current hop so that the packet just
        # reaches it before being stopped.
        sending_socket.setsockopt(
            socket.IPPROTO_IP, socket.IP_TTL, current_hop)

        # Attempt to send a UDP packet to the destination ip.
        sending_socket.sendto(bytes(MESSAGE, 'utf-8'),
                              (destination_ip, DESTINATION_PORT))

        # Receive any incoming ICMP packet. We can ignore the first return
        # value from recvfrom, which would be the included data.
        _, addr = receiving_socket.recvfrom(1500)
        received_ip = addr[0]
        print('Current hop {0}: ICMP message received from {1}'.format(
            current_hop, received_ip))
        current_hop = current_hop + 1


if __name__ == '__main__':
    main()

There we go! If I go ahead and run this, I get the following:

blog: $ python3 content/posts/toy-traceroute/toy_traceroute.py
Tracing the route to 93.184.216.34
Current hop 1: ICMP message received from 10.1.10.1
Current hop 2: ICMP message received from 96.120.66.197
Current hop 3: ICMP message received from 24.124.212.221
Current hop 4: ICMP message received from 96.108.45.74
Current hop 5: ICMP message received from 96.110.22.85
Current hop 6: ICMP message received from 69.241.119.54
Current hop 7: ICMP message received from 152.195.232.129
Current hop 8: ICMP message received from 93.184.216.34
blog: $

This matches exactly the path that traceroute uncovered earlier, which is very heartening! We’ve successfully plumbed the route in between my computer and the destination host.

Complications that make it interesting

There are many ways to improve this basic program, including at least the following:

Timing: how long did each hop take?
What if we sent multiple tries to have more timing samples?
What do we do if a hop doesn’t respond?

traceroute handles these, but it’s always interesting to think about how you’d solve the problem yourself.

There are also a few interesting cases about tracing the route in general:

What if a router drops UDP packets? What if it doesn’t send ICMP packets?
What if two subsequent requests take a different path through the network? Would the path we’re tracing really exist?

Seeing how things work through a toy problem

traceroute is a simple but clever use case of a built-in feature of the internet, and it’s fun to see how it works through a basic example.

Misc. recommended reading:

traceroute Wikipedia article
- traceroute is also turning 35 this year!
traceroute source code
Cloudfare’s blog: What is the Internet Protocol?
LIFEGUARD: Practical Repair of Persistent Route Failures, another clever reuse of functionality with BGP to avoid route failures

(1) New Message: How websites catch your attention with JavaScript

Sat, 22 Jan 2022 11:59:21 -0500

There are many good things about JavaScript. There are also bad things about JavaScript.

There are many cases of well-intentioned features in JavaScript being used for unhelpful purposes. These features can help support rich web applications with impressive features and capabilities, but they can often be used antagonistically, in the sense that they aren’t helpful for the user (like intercepting your clipboard commands). I find these cases interesting, and I wanted to explore a few of these attention-grabbing examples.

Note: these were all tested on desktop in Chrome, Firefox, and Safari. If you’re on mobile or an unsupported browser, I apologize that you won’t get to see the weirdness, and I’ve included screenshots and screencasts for your enjoyment.

(1) New Message

One of the things you have probably seen is how pages play with the page’s title, like prepending “(1) New Message” to the title. There are some valid and helpful use cases for this, such as an online chat app or an email inbox showing the number of unread messages.

Where this gets a little fuzzier is when the page updates it in a way to catch your eye with movement. This is doubly dubious when the message is something eye-catching that you didn’t opt in to, such as an unprompted sales message from a chat bot. At that point, it feels like a blinking advertisement desperate to get your attention. As the developer, if you want to be extra sneaky, you can even make it only happen when the user tabs away from the page, creating an extra little push to get them back to your site.

Give it a shot with the radio button below: turn it on, then open a new tab!

Blinking the title when hidden: Off On

Spoiler: Blinking the title when hidden

When I was playing around with this with other values for the title, I even fooled myself — I had tabbed away from my draft blog post for a minute, and my eye was immediately grabbed by the “(1)” in the title! Funny to see how quickly your conditioned mind gets fooled.

How does this work?

The main reason this works is because the page’s title is accessible through the document element via document.title. Because you can assign directly to it, like document.title = "foo";, you can easily put it in a continuously running interval with two options, the default title and a new title. Page visibility is also accessible in JavaScript via the document element. In this case, you can perform a check on document.hidden to see whether or not the page is currently hidden. If it is, you can go ahead with the title update; if not, you can lay low for now.

Click to view code


  var titleBlinkOff = document.querySelector("#title-blink-off");
  var titleBlinkOn = document.querySelector("#title-blink-on");
  var originalTitle = document.title;
  var newTitle = "HELP I'M TRAPPED IN A WEBSITE";

  var titleBlinkInterval;
  titleBlinkOff.onclick = () => {
    document.title = originalTitle;
    clearInterval(titleBlinkInterval);
  }
  titleBlinkOn.onclick = () => {
    // Change the title every second
    titleBlinkInterval = setInterval(() => {
      if (document.hidden) {
        document.title = document.title == originalTitle? newTitle : originalTitle;
      }
    }, 1000);
  }
  // If you click back in, go back to original title.
  document.addEventListener("visibilitychange", function() {
    if (!document.hidden) {
      document.title = originalTitle;
    }
  })

More reading: document.title MDN page, document.hidden MDN page.

What about the favicon?

Another thing you can do, though this is a little more cursed, is changing the page’s favicon. I don’t think I’ve seen this in the wild, thankfully, but depending on your browser (should work in Chrome and Firefox), it may be possible (though you may need an incognito window if your browser has already cached my favicon?). Give the following a try!

Blinking the favicon: Off On

Spoiler: Blinking the favicon

How does this work?

This one is a little weirder. As far as I can tell, because of the way the browsers parse favicon icons, the last icon element in the HTML head that hasn’t already been parsed is used. You can easily create a new link element, give it an icon ref and an appropriate href, and add it to the page’s head. This works for changing from the default icon to a new icon, but I ran into an issue when trying to go back to the default, where 1) I couldn’t just delete the new one, as the browser wouldn’t go back to parse the original and 2) I couldn’t just append a new version of the old one. Instead, I’m doing a weird trick where I’m recreating and removing the favicon icon links accordingly to re-parse them; this ends up looking as if it’s switching between the two. You can see that in the debug HTML:

As I mentioned earlier, definitely a little more cursed!

Click to view code


  // This is so dumb, I'm sorry
  // Used some of https://stackoverflow.com/questions/260857/changing-website-favicon-dynamically to create favicon.
  var faviconBlinkOff = document.querySelector("#favicon-blink-off");
  var faviconBlinkOn = document.querySelector("#favicon-blink-on");
  var headElement = document.getElementsByTagName("head")[0];

  var childAppended = false;
  var blankFaviconHref = "blank-favicon.ico";
  var originalFaviconHref = "/images/favicon.ico";
  var getNewFavicon = (blank) => {
    var newFavicon = document.createElement("link");
    newFavicon.rel = "icon";
    newFavicon.href = blank ? blankFaviconHref : originalFaviconHref;
    return newFavicon;
  }
  var faviconBlinkInterval;
  var blankFavicon;
  var originalFavicon;
  var resetFavicon = () => {
    if (blankFavicon && headElement.lastChild == blankFavicon) {
      headElement.removeChild(blankFavicon);
    }
    originalFavicon = getNewFavicon(false);
    headElement.appendChild(originalFavicon);
  }
  var addNewFavicon = () => {
    if (originalFavicon) {
      headElement.removeChild(originalFavicon);
    }
    blankFavicon = getNewFavicon(true);
    headElement.appendChild(blankFavicon);
  }
  faviconBlinkOff.onclick = () => {
    clearInterval(faviconBlinkInterval);
    resetFavicon();
  };
  faviconBlinkOn.onclick = () => {
    faviconBlinkInterval = setInterval(() => {
      if (childAppended) {
        childAppended = false;
        resetFavicon();
      } else {
        childAppended = true;
        addNewFavicon();
      }
    }, 1000);
  };

Further reading: Wikipedia article on favicons.

Are you sure you want to leave?

Sometimes when you try to navigate away from a page, you’ll get a popup asking you if you’re sure you want to go. There are some cases where this can be helpful. Maybe you’re halfway through a long comment on your favorite social media site and you accidentally hit control-w. Rather than lose your unsaved progress, that helpful intermediate step can let you cancel the action and get back to your typing.

The downside is that like everything else, there’s nothing stopping pages from doing this when it isn’t helpful. For example, maybe I want to do everything I can to keep people from leaving my blog (note: I don’t — please leave at your leisure!). Try turning on the following radio option and closing this tab.

Confirm before leaving: Off On

How does this work?

This one is much more straightforward. By adding a function that fires for the beforeunload event, we can ask the browser to check with the user before they leave the page. Browsers frequently require you to first interact with the page, but if you clicked the radio button above, you would have already done so. We can mess with this event easily by setting the window.onbeforeunload function to do a simple return true;.

Click to view code

  // Shout out to https://stackoverflow.com/questions/1119289/how-to-show-the-are-you-sure-you-want-to-navigate-away-from-this-page-when-ch
  var leaveConfirmationOff = document.querySelector("#leave-confirmation-off");
  var leaveConfirmationOn = document.querySelector("#leave-confirmation-on");

  leaveConfirmationOff.onclick = () => {
    window.onbeforeunload = null;
  }

  leaveConfirmationOn.onclick = () => {
    window.onbeforeunload = function() {
      return true;
    };
  }

Further reading: MDN page on WindowEventHandlers.onbeforeunload and beforeunload event.

What can we do with the user agent?

Sometimes when you’re on a website, you can see what looks like an alert from your computer. For example, if you’re surfing the web on a Mac, you may see what looks like a Mac alert about software being out of date on your computer:

Screenshot from the Adobe Support Forums.

To an unknowing user, it looks like your Mac is telling you that your software is out of date. In fact, it actually looks like my “Leave site?” alert above, which came from my trusted computer’s Chrome. Unlike most websites, your Mac is generally a trustable friend that frequently warns you about risks and tries to keep you safe.

What’s probably actually happening is that the website has checked what operating system you’re on (either in the server or in client code) and shown an alert with custom styles that look like that system’s alerts. You could do that a number of ways, but an easy way would be with JavaScript if you wanted to do it with a dynamic pop up. Click the button below; I’m just doing an alert, but you could imagine I could easily add an HTML element with the appropriate CSS classes to mimic your system.

How does this work?

This one is also not as exciting. The userAgent is easily available via navigator.userAgent, and we can check a few values to see what would match. For instance, we could look for the string "Mac" in the userAgent, and if it’s there, we could add an element with Mac styles. The trick here would be in the CSS, since we’d (presumably) want to make it match the system and not be too out of date.

Click to view code

  var userAgentButton = document.querySelector("#show-user-agent");
  var userAgents = [
    "iPhone",
    "iPad",
    "Mac",
    "Android",
    "Linux",
    "Windows",
  ]
  userAgentButton.onclick = () => {
    for (const userAgent of userAgents) {
      if (navigator.userAgent.indexOf(userAgent) !== -1) {
        alert("This is when I'd add a popup HTML element with " + userAgent + " styles.");
        return;
      }
    }
    alert("Sorry, I wasn't checking for your userAgent. You're safe today!" );
  }

Further reading: MDN page on Navigator.userAgent.

Be aware

I’ve seen most of these out in the wild, and I’m sure you have too. Like many things, JavaScript can be used for good or for bad. These capabilities allow for a powerful and feature filled web, but they also leave room for abuse. It’s already hard to protect your attention, but it’s helpful to think about how people try to pry at yours even more.

How do you visualize code?

Mon, 17 Jan 2022 17:11:31 -0500

The other day I was reading about the development of personal computers and the desktop GUI, and I was thinking about how comfortable we all have become with the analogy of the “desktop” for our personal computers. We place files in folders and keep them on our desktops. There’s a lot of physical action going on.

Humans are very good at understanding spaces, especially when it comes to remembering physical spaces, and it got me thinking about how we commonly visualize code. Is there any good way to tap into that when thinking about and visualizing code?

How do you visualize code?

This made me think about how I tend to visualize code, and it’s a little hard to describe. I think it usually lives in my head in a variety of ways, depending on the level of abstraction and specificity, and there’s some combination of all of them there at the same time. Depending on the need, I can jump between them pretty well, especially if I’m very familiar with the codebase.

For example, when I’m imagining the interactions between various microservices, it’s helpful to draw big boxes around each service, treating each one like a big unit of work with RPC interactions in between.

Architecture diagram of a simple microservice-oriented car rental service and the accompanying distributed trace that might represent a path through it.

On the other end of the spectrum, if I’m worried about the minuscule specifics of how the computer is reading what I’ve given it, it’s helpful to zoom all the way in to a representation of the physical memory.

Screenshot of a Google Sheets page I once made with memory addresses and assembly in the various cells. This was helpful for walking through it very, very carefully.

Luckily I spend most of my time somewhere in the middle, with some balance between reading the actual code (luckily higher level than assembly), thinking about chunks of code as large units, and working with architecture diagrams and inter-system communication. Even code on its own has a lot of physical relationships already built in; think of directory paths, namespaces, line indentation, and the linear ordering of lines of code,

How well do these visualizations do?

Thinking about this made me consider a few different visualization techniques, each of which is useful in different circumstances. Consider the following incomplete list:

Architecture diagrams
Dependency graphs
Distributed traces
Sequence diagrams
Class diagrams
Print statements
Flame graphs
Reading the source code

How would we compare these? First, there seems to be a natural axis of abstractness, ranging from low level code reading to high level architecture diagrams. There seem to be a few other axes we could order them on as well.

Maybe we could rank them on how well they represent the larger system? Architecture diagrams would do well there, but a flame graph would only represent a single execution path. Maybe frequency of change? That would be an interesting one, with architecture diagrams (hopefully) staying pretty static as the source code changes frequently.

What if we thought about how well the visualization represents the actual code execution through the system? In this case, maybe a high level architecture diagram wouldn’t score very high, as it would abstract away many of the details inside the service boxes. Distributed tracing could do better, though the level of specificity would depend on the number of tracepoints you have. Class diagrams, while helpful for visualizing the relationship between classes, may not represent the actual paths taken through. Flame graphs would show a clear execution path, but again they would only show a single code path without giving visibility to the larger system.

Graphing these might look something like the following, (though each of these dots may be better represented as a range):

Link to a larger version of the graph

This made me think about what that upper right section would be like. What would a helpful visualization look like that provided clear insight into the details? Is there a way to visualize the path through the larger system while staying at a low level?

What would it look like if we tried to use space as well?

Looking at a diagram still feels like reading a map. There’s that moment of spatially translating what you see in the code to the diagram, much like you’d orient yourself with a map in an unfamiliar area. Like the desktop metaphor for things on our computers, I wonder if there’s another way to visualize code as things that exist in the world to make the translation easier.

What if we had a toy visualization where we represented code in physical space?

Let’s say we have the following basic example, where we have a Counter class that keeps track of a private count_ variable and allows various access methods. Maybe it’s accessed in a simple main function to do some counting.

#include <iostream>

class Counter {
public:
  void reset() {
    count_ = 0;
  }

  int get() {
    return count_;
  }

  void incr() {
    ++count_;
  }

  void set_count(int x) {
    count_ = x;
  }

private:
  int count_;
};

int main() {
  Counter counter;
  counter.reset();
  counter.incr();
  counter.incr();
  std::cout << counter.get();
}

What if we represented different sections of the code as 3D structures? What if we represented the Counter class as a big room with incantations written on the wall? You could imagine the possible paths through the code, connections between variables and the things they represent, as pink string:

Link to a larger version

What could we see with this? The first is that count_ is clearly used appropriately as a private variable, as there aren’t any pink strings from it that leave the room. The public methods are all pretty clear as well, with their pink connections to things outside of the room.

More interestingly, what would it look like with this room connected to another? The main function could be another room, with its incantation lines connected accordingly:

Link to a larger version

Now we’ve got cross-room talk; where the main function calls counter.reset(), we could have a connection from the caller in main to the callee in the Counter class. You could even imagine a debugger stepping through this, watching pulses being sent across the wires with arguments and return values. Imagine being able to zoom out to a different area to see local state and values, then follow the call paths back through to the active area.

Would this be helpful?

Would something like this be helpful? I’m not sure. It would be interesting to walk through a familiar codebase with something like this, especially if you were able to explore spatially in a 3D representation (or in VR?), being able to zoom in and out as you’d like. I wonder if it would be especially useful when exploring a new codebase for the first time to see how things are connected, though I’ve definitely dealt with some codebases that I imagine would be very intimidating in a view like this. I’d love to see what a diff looked like in this view, where a new class is introduced with new connections here and there.

That being said, I think you’d run into at least the following problems when making something like this:

Complicated code is hard enough to reason through. It might be cathartic to visualize the spaghetti in spaghetti code, but how busy would this be with very complicated code?
How would we be able to represent things like two threads executing at the same time?
How would you represent passing by reference instead of passing by value?
How would you represent asynchronous work?
How would you represent recursion? Rooms all the way down?
How would you prevent something from this from getting stale and out of date? At the very least, it would need to be automatically generated.

Problems at the intersection of space and code

There are a few considerations that make this tricky. One is that physical locations usually change on much longer timeframes than code does. Part of the reason it’s useful to use a physical location you know well for a memory palace is because it’s the same every time in your memory. If you’re memorizing a deck of cards, you can temporarily store the ace of clubs behind the cupboard door, but the cupboard door will still be there the next time you need to store a card.

If your codebase is changing frequently, your map of how things are laid out spatially may change, regardless of whether those maps are 3D generated or purely mental. It’s like returning to an area you used to know, except instead of just the landmarks changing, imagine if the roads were rerouted as well. Even if we’re naturally gifted at remembering things spatially, if that space changes, would we still be able to benefit from visualizing that space if we had to relearn it?

It would be interesting to see how helpful something like this would be for both exploring a new code base (like using a map to explore a new city) and revisiting that codebase as it changes over time (like going back to your hometown after a long time away).

Visualizing Code

I’m not very familiar with this area of data visualization (nor the rest of data visualization, really), but in a brief search (AKA 30 minutes of Googling), I found a few projects that seem like they’re doing something similar:

SoftVis3D: where a "‘code city’ view provides a visualization for the hierarchical structure of the project".
Code Park: A New 3D Code Visualization Tool (2017), a “novel tool for visualizing codebases in a 3D game-like environment” with code represented as “code rooms” with code on the walls (sounds very similar now that I read this).
Code Structure Visualization Using 3D-Flythrough (2016), with spatial metaphors and first-person exploration of code.
Primitive, a VR collaboration startup with a Matrix-looking “Immersive Development Environment” with “new tools for visually analyzing software in 3D”.

Edit: here are a few more that readers have pointed out:

AppMap, an automated code analysis tool that includes dependency maps and trace views.
plurid, a framework for visualizing and debugging code in a 3D explorable structure.
fsn (file manager), an experimental application to view a file system in 3D (featured in Jurassic Park).

If you’re aware of any others, please get in touch; I’d love to learn more!

Fun to imagine

This concept is clearly not groundbreaking, but I think it’s a fun way to think about how we work with our tools and, importantly, how we might do better. There must be better ways out there, and it’s fun to imagine what they might be.

A Children's Book Story About Distributed Systems

Wed, 12 Jan 2022 20:07:04 -0500

Editor’s note: the following document was found in a collection of papers and assignments, apparently dating from late 2020 when the author was stuck inside studying distributed systems. An additional note, written in green and red Crayon on the back of a CVS receipt, was found stapled to the document: “it is now winter . this work features only approximate explanations. it favors silliness over accuracy in some sections. it should be taken with a light hearted grain of salt” [sic].

The following is a verbatim reproduction.

Karl and the Three Dilemmas

Karl sends a letter

Karl is a small terrier who lives in Cambridge, Massachusetts. One wintery day in Cambridge, Karl was looking for something to do. He had already eaten a snack, gone for a walk, chewed a toy, played fetch, and taken a nap, but it was only 2:00PM! He was very bored.

As he was day-dreaming about great snacks he had in the past, he remembered his good friend, Jake, who he hadn’t seen in a while. “I know!” exclaimed Karl. “I’ll see how Jake is doing!”

Unfortunately, Karl didn’t remember Jake’s phone number, but he did remember where he lived, since he would always meet him outside to go for walks together. How would he get in touch with Jake?

Karl remembered another friend, Andrew Birrell, who was great at talking to friends, even when they were far apart. Karl called Andrew to see if he had any ideas.

“Hello?”

“Hi Andrew, it’s Karl!”

“Hi Karl, how are you?”

“I’m doing well! I had a question for you: I want to ask my friend Jake how he’s doing, but I don’t remember his telephone number. I only have his address. Do you have any ideas for how to get in touch?”

“Sure, Karl, it’s easy! First, you’ll want to figure out what question you want to ask, and write that down on a piece of paper. That way, Jake will know what you want to know. After that, you’ll want to give the letter to your mom, who will put your letter in an envelope and write Jake’s address on it and your address smaller. Don’t forget to put a stamp on it!”

“I won’t!”, said Karl.

“Next,” said Andrew, “you’ll want to go put it in a mailbox. A postal worker will come get it, then it’ll get sent across the state to different warehouses and finally to Jake’s house. Once it gets to Jake’s house, his mom will open the letter and give it to Jake, who will read your question, respond, and send it right back the same way. It’ll almost be as if you’re passing notes to Jake in the same room!”

“That sounds so easy!” said Karl. “So we don’t have to worry about how it gets to Jake?”

“Nope!” replied Andrew. “You just have to write your question, then your mom will take care of writing Jake’s address and getting it to the post office! It’s really handy to have the post office, since they do the mail delivery for everyone. Otherwise, every time you wanted to write a letter to a friend, you’d have to worry about how to get it there yourself!”

“That sounds great! Thank you so much Andrew!”

“You’re welcome Karl!”

So Karl went and got a piece of paper, wrote his question, handed it to his mom who then got an envelope and a stamp, put the letter the envelope, wrote Jake’s address on the envelope, wrote his own address smaller in the top left, put the stamp on, and dropped it off at the post office.

A week later, a letter came for Karl!

“Dear Karl, thank you for writing! I was really happy to get your letter. I am doing very well! My mom knit me a sweater for Christmas. Hope you’re doing well too! Your friend, Jake.”

It was just like they were in the same room together!

Karl borrows a book from the library

One day on his holiday break from puppy-school, Karl decided he wanted to read a book, but he had already read all of the books in his house. He had heard about a new book, Go, Dog. Go!, from his friend Shiro, and he asked his mom if they had a copy.

“I don’t think so, Karl. Why don’t you see if you can rent it from the library?” asked his mom.

The library! Karl had walked past it before, but he had never been inside. Since his mom was busy, Karl decided to ask his friend Sanjay about renting books from the library, since he knew a lot about libraries, filing, and how to organize things. Karl gave Sanjay a call.

“Hello?”

“Hi Sanjay, it’s Karl!”

“Hi Karl! How are you?”

“I’m good, thank you very much! I had a question for you: how do you rent a book from the library?”

“That’s a good question Karl. It’s great to borrow a book from your local library. You used to be able to go to the library and pick a book in person, but it’s a little different these days.”

“Oh yeah?” asked Karl.

“Yeah,” replied Sanjay. “Now, you have to write a letter to the head librarian and ask if you can check out the book. They will look up your library card information to see if you can borrow the book, and if you can, they will write back with the address of the library branch that has that book. You can then write a letter to that library asking for the book, and they’ll send it to you.”

“That sounds pretty easy,” said Karl.

“It definitely works!” said Sanjay. “Then, if you ever want to borrow the same book again after you return it, you can write another letter directly to the library branch that has it. I’ve rented the same book many times.”

“That’s great!” said Karl. “I’ve heard really good things about Go, Dog. Go!, and I may want to read it a lot. I am curious though: how would the libraries get new books?”

“That’s a good question, Karl,” said Sanjay. “If someone wanted to give the library copies of a new book, they’d write a letter to the head librarian. If the library can accept the books, the librarian will send back the address of a branch library to send the books to. Once the library gets the books, it can forward on a copy to a few different branches, and those libraries will let the librarian know they have a copy. That way, if anyone else wants to borrow the book, the librarian can send them the address of the closest library with a copy in stock.”

“That’s pretty neat!” said Karl. “That sounds like a lot of letters back and forth, and I’m really glad the post office knows how to deliver all of those letters. Now that I think about it, it’s really not that different from borrowing a book from one library through the mail, is it?”

“That’s exactly right!” responded Sanjay. “Just like writing a letter to your friend is like passing them a note, it’s like we’re borrowing a book from one big library, even though it’s actually a bunch of small libraries working together with the head librarian.”

“That makes sense! I think I’m going to go write the librarian a message now. Thanks Sanjay!”

“No problem Karl!”

So Karl went and wrote the librarian a letter asking if he could borrow Go, Dog. Go!.

A little later, the librarian responded saying he could borrow it, and that he should write a letter to the Central Square branch of the Cambridge Public Library. He wrote them a letter, and they sent him the book.

He read it the same day, and he decided it was his new favorite book.

Karl makes an advent calendar

One day in late November, Karl wrote a letter to his local library to borrow Go, Dog. Go! again, since it was his favorite book. This time, when the book came, a note came with it:

“Have a good idea for an advent calendar of pictures? Send in a quarter and 25 things to draw, and we’ll send you a hand drawn calendar.”

An advent calendar of pictures, how exciting! Karl liked to draw, but every drawing took him a long time since he had trouble holding the pencil.

But he had a ton of good ideas for drawings! “I wonder what this library advent calendar is all about?” wondered Karl. He then remembered that his friends Jeff and Sanjay were really good at working together and splitting up work, so he decided to give them a call to see if they knew anything about it.

“Hello?”

“Hi Jeff, it’s Karl! How are you?”

“Hi Karl, I’m great! Just working on a problem with Sanjay.”

“Hi Karl!”, said Sanjay.

“Hi Sanjay!”, said Karl. “I had a question for you two. I saw that the local library will make you a calendar if you send in things to draw. Do you two know how it works?”

“Sure!”, said Jeff. “It’s actually very simple. You can send them a list of things to draw, one for each day, and then they’ll have people draw them and send you back a calendar made up of the drawings.”

“That’s so fun!” said Karl. “So one of the assistant librarians draws all of the pictures for me?”

“Not quite!”, said Sanjay. “If one librarian does all of the drawing, they’ll have a lot of work to do by themself, even if they’re fast at drawing. Instead, they send out the ideas to a lot of different librarians at different branches! That way, each librarian only has to draw one picture, and they can do them all at the same time.”

“It’s a great idea!” said Jeff. “That way, the assistant librarian can send out a letter with the day and something to draw to each librarian, who can draw it and send it back. Once the assistant librarian has gotten all of the drawings back from the other branches, they put them in order for the calendar. Once they’re in the right order, they paste it all together, put a cover on it, and send it right back to you.”

“That’s fun!” exclaimed Karl. “That sounds like it will be much better than drawing every picture one after another, and I just happened to learn the other day how to send a letter from my friend Andrew, which means it’ll be very easy to write to the librarian.”

“That’s right,” said Sanjay. “And since all of the libraries are talking to each other to keep track of which library books are where, it’s easy for them to keep track of all of the different drawings. It’s like your drawings are the books, and they’re asking each other for them. That way, the assistant librarian handles all of the coordination between all of the other branches, and you just have to tell them what to do for each day!”

“That makes sense!” said Karl. “Thanks so much!”

“No problem Karl! Bye!” said Jeff and Sanjay.

Karl then made a list of 25 things to draw. With his mom’s help, he sent out the list (and a quarter) to the assistant librarian. A week later, he got the full calendar! The assistant librarian even included a note apologizing for the delay, explaining that they had to wait a little for the very last one, since the librarian at the Brighton branch draws very slowly.

Karl sent a picture of the calendar to Jeff and Sanjay. “Looks good to me!”, they said.

Karl put the calendar right next to his bed so he could see it when he went to sleep.

Editor’s note: this photo was also found with the document with a sticky note attached that said “dog tax”.

Completing a Part-Time Master's in Computer Science While Working

Sat, 08 Jan 2022 17:33:34 -0500

“I’m going to be 29 anyways, so I might as well be 29 with a Master’s.”

— me 5 years ago debating whether I should take classes

On June 5, 2017, I started taking classes at Tufts University while working full time. On January 4, 2022, I submitted the final draft of my Master’s thesis. Much like my past self had hoped, I’ve made it 4 years in the future, and I’m done with my Master’s program. At the same time, I’m both very happy I did it and very happy to be done, and I’m going to describe a few thoughts about my experience.

For a quick visualization, this is what my work and school looked like for the last few years:

small startup -> Google, while on the school side, my classes are listed." />

(Fall 2020 was a busy time for me)

For a little background, I didn’t do any programming in college (though in retrospect I wish I had tried it), instead studying physics, French, and political science. I did a lot of self-studying on my own from roughly January 2015 onwards.

By the numbers

Classes

Overall I took 11 courses, 10 at Tufts and 1 at Oregon State University (online), and completed 2 semesters of a Master’s thesis. In rough order:

Data Structures
Discrete Math
Algorithms
Artificial Intelligence
Theory of Computation
Machine Architecture & Assembly (Oregon State)
Networks
Introduction to Machine Learning
Cloud Computing
Compilers
Debugging Cloud Computing
Thesis (Summer)
Thesis (Fall)

The first five classes counted towards Tufts’ Post-Baccalaureate certificate program, which is roughly the equivalent of a CS minor, and the the second grouping of classes (and a few of the Post-Bacc classes) counted towards the Master’s degree. I took the assembly class at Oregon State because the Tufts equivalent was 10x as expensive (see later) and wouldn’t count towards graduate credit, and I could make up that requirement by taking a class that had the assembly class as a prerequisite. I took a small break from classes in Spring and Summer 2019 after finishing the Post-Bacc, then decided to complete the Master’s. I took another small break in the summer of 2020 while I co-hosted interns on my team at work.

Time

It took me a little over 4 years to do the Tufts CS Post-Bacc and the Master’s, at an average rate of 1 class per semester/summer.

I took these classes while working full time. I transitioned from a tiny company to a very small startup a week after starting classes (June 2017), and I went from that startup to Google in September 2018. I ended up getting a promotion in Fall 2020, and I transitioned from the Hotels team to the Cloud Networking team in April 2021.

In spring 2020, both my work and classes transitioned to be fully remote, and both have been remote since (with a small window of a few in-person meetings in late 2021).

Cost

When I started the classes, tuition was a whopping $5,052 per class for part-time students. After my first 4 classes, the department switched from per-class tuition ($5,052 per class) to per-credit hour tuition ($1,700 per credit). Classes ranged from 3-5 credits, and luckily I was able to finish the program with only 3-credit classes after that.

Google has an education tuition reimbursement benefit for classes taken while working there, and I was able to use this benefit for all classes I took after starting at Google.

Total cost: ~$61,000
Google’s help: ~$23,000
My final cost: ~$38,000

Thoughts, feelings, and reactions

The Good

Classes really helped me in my CS career

Even though there’s a gap between academic CS and working in tech, the classes I took had a very positive influence on my career with computers.

When I got my first real programming job at the small startup, they hired me as an inexperienced junior frontend engineer who would be willing to learn fast, and the classes helped show that I was serious about learning more. This helped me transition away from doing some development work at the tiny non-tech company to being a Real Engineer at a real (albeit small) tech company. That alone was easily worth the price of admission.

I took the Algorithms and AI courses before interviewing at Google, and the in-depth coverage of various algorithms helped build the fundamentals I used in my interviews and on the job since. I had done algorithm interviews before the algo class, and I highly recommend taking the class first. (Warning: confirmation bias) I didn’t go in with a bunch of LeetCode problems memorized, but instead a pretty good foundation of algorithms, data structures, and different types of problems I understood after carefully working through them, and these classes were very helpful for building that understanding. Again, easily worth it.

The other way these classes helped me is my transition at work from working on front end work to more backend and distributed systems work. When I started at Google, I worked mostly on the front end, but slowly over time I transitioned deeper into the stack, eventually transitioning teams to work on the Cloud Networking team on load balancing, deep in the C++ and networking stack. The classes I took on networking, distributed systems, and debugging distributed systems were incredibly helpful for building my background knowledge and helping me find an area I was really interested in, and I even did my master’s thesis on an area tangentially related to my day job work (adding a new flavor of distributed tracing to Envoy, an open source proxy that we use at work). These classes made this transition easier, and I think there’s also some value in the narrative around them too, being able to point to them in conversations as evidence of my interest in learning about this new area.

Shining light on the mysterious caves of CS

Interviewing and getting jobs aside, one of the best benefits of my classes is that they helped shed light on a wide selection of various areas in computer science. This served two great purposes: helping demystify areas that initially seemed scary and showing the true depth of some of these areas.

Coming into computer work without a solid CS background meant that there were many gaps in my knowledge. I had taken a few online classes and done some reading, but there were still many areas that I knew of without knowing how they worked. These can be described as the known unknowns; I’ve heard of it, and I know I don’t know about it. For example, there were many things in networking I had known about from cursory reading; I roughly knew about TCP/IP (partly from seeing the option in StarCraft when connecting to Battlenet in 2001), but I wasn’t familiar with any of the details. This is where the classes were especially helpful, as they represented a guided and structured exploration into these areas.

From networking to algorithms, the in-depth view helped to shed light on the mysterious unknowns, and it made these areas much more approachable. It helped build my portfolio of solved problems, building up my tool chest (or treasure chest?) of problems I had solved in the past, helping to further convince me that my future problems are solvable too. Having built a basic compiler, it wasn’t nearly as scary any more, and it helped show me that I could work hard at an area to understand it.

The other side of this equation is that having seen the basics in introductory classes, I got a better appreciation for the absolute depth of complexity of some of the areas. These can be described as the previously unknown unknowns; I now have a greater appreciation for just how much I don’t know. Like the small specialty in the classic illustrated guide to a PhD, these classes helped me see some of the depth to the edge, and that distance is still vast for me for many, many subfields in CS. It makes me very grateful that we have smart people working on these problems!

Learning more about CS academia

Another solid benefit from my Master’s was being able to get a better picture of research, academia, and CS academia. I didn’t do any research in undergrad, and when deciding on doing the Master’s after the initial 5-class Post-Bacc, I wanted to get a little exposure to the academic side to see if it was something I liked.

While doing the program, I had the opportunity to work in the lab of one of my professors, who ended up being my advisor for my MS thesis. I was able to help out on a published research project (and get a very casual Erdős number of 4). I also had the option of doing the pure 10-class Master’s or doing a thesis at the end, and I chose to do the thesis to see what it was like. I ended up doing my thesis on the perfect intersection between work (Envoy, a distributed proxy), my advisor’s work (distributed tracing), and open source (all open source, so much easier to write about with work’s approval). This was another reason I chose to do an in-person program; although there are many cheaper online-only options, I had read that getting involved in research wasn’t as much of an option for those.

This was all hard to do while working full time, and nearly every time I left a class or a research meeting, all I could think about was how nice it would be to do it full time without the additional distraction of my 9-5. I really enjoyed the collaborative and explorative aspects of the research, and there was a feeling of doing something new for the first time that I greatly liked. The classes and the research work also really scratched my itch to learn, which was very satisfying. I think it was especially helpful when I was working on front end work for my job, as I started to enjoy (and understand!) distributed systems, which was separated from what I was doing for my day-to-day. I’ve since moved into an area where I do that for my day job too, and I find that itch scratched by my 9-5 work as well.

I’ve thought about continuing with the PhD (Tufts even has a part-time PhD program), but I’m not going to do so right now. I’m going to take some time to have a little break outside of my day job, and my day job is giving me plenty to learn and keep my learning-gremlin happy. Someday I may taper down to part-time at work and go back to school, but I’m going to enjoy my free time for a little bit. One other barrier to doing the PhD full time is financial; my big tech pay check allows me to keep up my expensive cheese habits, and things would be very different on the grad student stipend. I’ve heard that one of the reasons to go straight from undergrad to grad school is that you don’t get tempted by that sweet taste of tech salary, and for better or for worse, I’ve both tasted it and gotten used to it.

Structure

Learning on your own is hard. I’ve started many more online classes than I’ve finished, and although I would love to imagine myself as the diligent autodidact who can sit down and hammer out an online class on a whim, realistically it’s really hard to do.

Because so much of learning is about consistently showing up and doing the work, I’ve found that it’s easiest for me to do so with some amount of structure, deadlines, and accountability. I think that’s why learning for my day job is pretty doable for me, as all three are built in (and I often need to do so to support my cheese habit). But this is where the in-person classes shined; there was clear structure with a built-in physical context of being at school to learn, clear deadlines, and built-in accountability with professors I got to know. There was also the cost aspect — having paid for it, I was much more motivated to do it to the best of my ability. I find it’s important to harness the fallacies that fool your brain!

I’ve taken (or tried to take) online classes for free with various levels of structure to various levels of success. Classes like CS50 with solid structure and clear assignments were doable, though they took me much longer than a semester. Classes like MIT’s Intro to Algorithms was much harder, and I only ever got about halfway through. I’ve also taken a single online class that I paid proper tuition for, and this was the assembly class at Oregon State University. I also did many in-person classes that I paid (a lot) for. I found the in-person classes to be the best fit for how my brain works, and I’ve found that I do my best when it’s live, I can raise my hand and ask questions in class, and I build connections with my professor and peers. A non-trivial part of it is also going through it with fellow students, which was a great aspect.

How to learn

One of the odd benefits from taking classes while working full time was that it strongly encouraged me to develop better study and learning habits.

Those habits and skills were not the best during my undergrad studies, which included a lot of skipped readings, procrastination, and many last minute cramming finishes, often resulting in all nighters the night before a deadline or exam. During my recent studies, it took exactly one all nighter the night before a work day to realize that it wasn’t a tangible option long term, as I had to zombie my way through a long day of meetings and coding at far less than 100%.

Combined with a little more maturity and interest in the subject matter, I focused on building better study habits and skills. I owe a lot to the Learning How To Learn class and the accompanying A Mind For Numbers book, as they helped me build a very strong foundation with practical advice and knowledge about how learning works. It turns out that doing the reading, starting early on assignments, and studying consistently really does help.

I was able to finish all of my classes well and efficiently, getting better grades than I ever did in my undergrad without any additional all night study sessions. There were a few late nights, but they were generally few and far between. Overall it gave me a better appreciation for healthy learning habits that I’ll keep close moving forward. In particular, I’ll continue to go for long walks after a long session of research or any time I’m particularly stuck on a problem.

The Not So Good

Where did my money go?

As noted above, my final cost for this was nearly $40,000. The Tufts classes were very expensive, at roughly $5,000 per class. Halfway through my Post-Bacc (the initial 5 classes), Tufts changed part-time tuition to be per-credit instead of per-class, with each class being 3-5 credits. I had actually planned to take Tufts’ assembly class, but at 5 credits, it would have been even pricier at $8,500+. I ended up not taking the class at Tufts, instead getting the background knowledge via Oregon State University’s Assembly class and making up the Master’s Assembly requirement with a grad-level class that had an Assembly class pre-req.

There were a few other small tuition hacks that helped. First, I took the maximum number of classes in the Post-Bacc that would count towards the MS (3 classes). I also opted out of signing up for an additional credit for the more advanced version of a class, though I ended up doing the advanced work anyways. While nominally the credit was supposed to represent the extra work the student had to go through for the more advanced work, for part-time students like myself it would mean an extra $1,700 in tuition, and luckily I didn’t need the credit.

Small one-off optimizations like this helped reduce the price tag by $5-10k, which was absolutely fine with me.

I’ve worked very hard, but I’m also very fortunate to be in a position where this tuition was manageable. I spent many months at the tiny company saving up for the first class, and I was very fortunate to have a few job and pay changes that made the tuitions even more palatable as I went. Ironically, my Google job came with the biggest pay increase, but with the tuition reimbursement, it also meant that the overall cost was greatly reduced.

With the very positive impact on my day job(s), the benefit greatly outweighs the cost for me, even without considering the many other intangible benefits.

Where did my time go?

Taking classes while working full time led to a very busy schedule. A typical week day for me (pre-pandemic) would be me getting into work around 9, working for a few hours, commuting to class, going to a class (or two), commuting back to the office, and working late to make up the missed time. Add in the commute to and from the office and it got very busy very quickly, especially with projects, homeworks, readings, and group meetings.

Fortunately, my wife was similarly busy with graduate school for the first three years of my studies, and I was able to go through it with a buddy. We would go to the library together, work from a coffee shop, and even have little study halls at home. The solidarity was a huge help.

The busy schedule did mean that a lot of my non-day-job time was spent studying or working. I have a wide variety of interests and hobbies, and many of them have been neglected more than I would have liked in favor of school work. This is another reason I’m not immediately jumping into a 5-year part-time PhD program; I’d like to find a little more balance in my life.

I’d also like to find a little more balance at my job. It’s really nice to start work at 10 and go home at 5 or 6, but with the many interruptions during the day due to school, I would frequently be shifting my hours and working later to make up the time. Context switches are expensive, and I’m very much looking forward to fewer of them.

As an aside, the busy back and forth also meant that I was less likely to stick around for office hours after class or student group meetings, as I wanted to go finish my work for the day so I could go home. I think this also contributed to some feelings of disconnected “He doesn’t even go here” vibes as a part-time student.

The So-So

Where did my in-person interactions go

For a number of reasons, I chose to do an in-person MS instead of doing an online program (such as Georgia Tech’s MSCS, if I could get in). I knew it would be greatly more expensive, but I was hoping it would allow for the benefits of in-person classes, such as meeting peers and professors, potential opportunities to get involved in research, and the benefit of a physical context dedicated to learning.

While that all went pretty well for the first few years, everything went fully remote in the spring of 2020. All of a sudden, everything was at home over video chat. The transition was hard for both work and school, and even though I was able to do my classes, ask questions in class, and get involved with research, it was very different. It was missing a lot of what I enjoyed from going in-person. Furthermore, I think it contributed to my feelings of burnout and zoom fatigue, as I was now spending both work time and school time on video chats, and without the physical context switches of going to work and going to school, I was then attempting (and failing) to balance home, work, and life from the same living room. This was a difficult period for me, and the busy schedule didn’t help.

That being said, there were two positives. First, I didn’t have a long commute anywhere any more, so I had a bit of time saved throughout the day, though this led to even faster context switches as I would have to jump from one video chat to another. The other large benefit was that during a time where it felt like everything had stopped moving forward and time stood still, I still had tangible forward progress towards my academic goals. The structure of the classes meant that I was moving forward in part of my life, though the rest of it was standing still. With that feeling of “let’s keep moving forward”, I kept up a pretty good pace, and as I got better at doing everything from home, I was able to strike a better balance and take on more amorphous and ambiguous work (such as getting involved with research and working on the thesis).

Was it worth it?

To me, taking the classes and completing the MS was absolutely worth it, and I’m very glad I saw it through to completion. At the same time, I’m very happy to be done. I’m looking forward to spending more time writing, drawing, playing music, cooking, and making puppets. There may be more classes later on in my future (there’s that learning itch again), but for now, I’m going to enjoy a little more balance and a little more free time.

The Strange Path of Accepting How Your Brain Works

Sun, 31 Oct 2021 00:10:11 -0400

I recently read Turing’s Cathedral, a book about the early history of computers, and it largely focused on the life and brilliant times of John von Neumann, who was, to put it simply, a genius. It is both fascinating and disappointing to learn about a brain that works so much better than your own.

Fascinating because the wide spectrum of human ability has led to a brain so capable. That we (in the general, humanity sense of “we”) are able to perform such feats is incredible. Genius exists, living in one of our heads and manifesting its abilities through photographic memory, mastery of over half a dozen languages, and imaginative and original creative work.

But, to be perfectly honest, there’s something somewhat disappointing too, especially if your mind is prone to drawing comparisons with itself. Reading about John von Neumann’s photographic memory while having trouble remembering a birthday is unsatisfying, as is reading about him needing only 4 hours of sleep while feeling the tiring effects of only getting 7 hours yourself. It is humbling, if only by giving perspective into the actual width of the ability gap. How nice it would be to have all of that!

Before getting into tech, I worked as a tutor for high school students. One student I worked with on his Algebra 2 work wasn’t always the best at completing his assigned work. We ended up striking a deal where if he completed his worksheets, I would watch his favorite anime series or movie and we could spend a small part of every weekly tutoring session talking about it. In a win-win, he ended up completing more assignments, and I ended up watching a few extra shows and movies in my free time.

One of the movies I ended up watching was an anime about a high school math genius who had to help defend a virtual world against an AI that had turned evil. At one point near the end of the film, the student is hacking away at defeating the AI, but the AI keeps blocking his digital path with complicated cryptography problems. The protagonist would take a few minutes of wild scribbling as he solved the equations by hand on a stack of scratch paper, typing in the solution to unblock his digital path and continue his attempt at saving the day.

Just when time is running out, the evil AI throws one more problem at the student, even more complicated than the last, but there isn’t enough time to solve it by hand. The kid seems to unfocus his eyes as he transfixes on the problem, sweating from the stress, and drops his pencil as he fully absorbs himself in the problem. After a moment, he is miraculously able to complete the problem entirely in his head, and he types in the correct answer, letting him enter a few final keystrokes to finally help save the day at the last moment.

Back in real life a little while after watching the film, I was working with my real student on a math assignment. We had talked about some strategies for solving a certain type of problem, and I had encouraged him to carefully write out each step, explaining that it’s one of the best ways to make sure you’re taking your time to not miss anything and that it would help him (and his teacher) see and confirm his process and understanding.

A few problems in, I noticed him pause when starting on a new problem, and after a brief moment, he wrote down an answer without any intermediate steps or work. I could see that he was skipping the process of writing down the steps we had talked about, trying to work through it entirely in his head. It was almost as if he was his favorite character, faced with the complicated problem, but after staring at it and absorbing it he could arrive at a complete solution entirely in his mind.

Part of the reason I could tell he was trying to take the shortcut was because he had missed a step he had previously gotten by writing out the steps. I’m not “anti mental math”; more that I’m very “pro practicing and building the understanding and fundamentals”! I encouraged him to work through the steps for the problem, and he ended up finding his mistake after writing out the process.

Having seen the anime, I later wondered if he was playing as if he was the protagonist, and I remember smiling at the thought; I too wish I could just do it all in my head! As I think back, it’s a very human thing to do, and they’re very natural thoughts to think.

“What if I was able to do complicated math problems in my head on a whim? What if I could keep track of every tricky step in a convoluted algorithm solely in my mind? How nice would that be?”

“What if I had the memory of von Neumann?”

I don’t think it happened at any specific moment, and maybe it’s a part of growing up, but I’ve come to accept the realities of what my brain can and cannot do.

It’s funny to write that out even now, because it seems at the same time both obvious and final. Of course I don’t have photographic memory, as much as I’d love to imagine I did. How wonderful it would be to not have to write out my work! But I know that I can’t keep it all in my head. It would be delightful to have perfect pitch, but I have to find my 440hz with a tuning fork just like everyone else.

At the same time, there’s a certain finality to it. How fun it is to imagine! Can you imagine what you would do if you could remember every page you’ve ever read after a single reading? I’ve thought about it before. How much easier school would be, or remembering peoples’ names, or those tricky French verb tenses I always trip up on, or…

But, I can’t. Somehow they slip easily through my mind, maybe catching on and sticking in a neural pathway somewhere, but maybe not. Il faut que tu… soie? sois? soies??

More importantly though, it’s OK that my memory isn’t perfect, and I know that about my brain. It is what it is.

I think the most powerful thing about acknowledging your brain’s limitations is 1) the acceptance of it and 2) your ability to work with your brain with those limitations in mind. I don’t have perfect memory, but I can learn about how memory works and leverage tools to help me move past what my brain can do on its own. It’s why I find checklists, mnemonics, and spaced repetition so helpful, as they let me augment what I’m working with. We’re lucky to have enough tools at our disposal (and the ability to build our own tools), and much like Steve Jobs’ metaphor of the computer as a bicycle for the mind, once you know how to work with your brain, you can help it to go even further.

As much as I tease AI startup landing pages for listing “AI + Human”, I think that’s one area where the combination of humans and computers does especially well. For me, there are certain things that a computer helps with, such as keeping track of some detail I’ll need in the future or letting me know when 30 minutes has passed. It can handle the specifics of the spaced repetition algorithm, and all I have to do is tell it how good I felt about the card I just got correct. That being said, I don’t think computers are the fix for everything, at least for my own lizard brain. I know I’m a sucker for infinite scroll and interesting content, and since I know I do best after at least 8 hours of sleep, part of the battle is even to get off the damn computer!

So part of the good news is that we can work with tools to augment our own abilities to get closer to what some of these gifted brains can do from just their natural ability. Maybe it’s not fair that they get it by default, but it’s not worth it to worry too much about “fair”. The people we look up to feel the same way about people they look up to too, and if Nobel Prize winners look up to the Martians as otherworldly, that’s enough to make me feel better already.

The other good news is that a lot of what looks like natural ability is the result of a great deal of hard work and practice. As easy as it is to discount a skill gap as due to some ingrained natural aptitude, the reality is that they’ve probably worked at it a lot more than I have. There’s a vignette I really enjoy from Richard Hamming’s You and Your Research talk, where he talks about drive:

I worked for ten years with John Tukey at Bell Labs. He had tremendous drive. One day about three or four years after I joined, I discovered that John Tukey was slightly younger than I was. John was a genius and I clearly was not. Well I went storming into Bode’s office and said, “How can anybody my age know as much as John Tukey does?’’ He leaned back in his chair, put his hands behind his head, grinned slightly, and said, “You would be surprised Hamming, how much you would know if you worked as hard as he did that many years.” I simply slunk out of the office!”

So the good news is twofold: you can work with your brain, and many of those you admire for having natural talent are probably building their abilities by putting in a great deal of work.

As a quick aside, this seems like a good point to note the difference between putting in the work and day-job work or “hustle culture”, in that I strongly reject the idea that you should always be working harder and harder for money in every facet of your life. It’s wonderful to have a hobby, but if that hobby is basket weaving and you want to get better at basket weaving, it’s OK to want to work on getting better at it.

To stave off the risk of rambling, I’m going to cut it here. Knowing and accepting your brain is half the battle, and working with it is even better.

I’m going to go review a few French flashcards.

Misc. recommended reading:

John von Neumann’s Wikipedia article (but don’t get too down on yourself)
Wikipedia article about the Hungarian Martians
Richard Hamming’s You and Your Research
Turing’s Cathedral by George Dyson

Writing an RPC From Scratch

Sun, 18 Apr 2021 13:15:13 -0400

There are many ways for computers to talk to other computers. A commonly used approach is a Remote Procedure Call, or an RPC. An RPC allows for the abstraction of calling another computer’s procedure as if it were a local one with all of the transmission and communication taken care of.

Let’s say we’re writing some math program on a single computer and we have some procedure or function that handles checking if a number is prime. We can use this function as we’d like, passing it numbers and getting an answer, and it can live in our computer.

This is fine, and for doing many things, it’s very helpful to keep things nearby. It’s easy to call, and since it’s right alongside the rest of our code, there’s nearly no delay when we have to send it numbers.

But, there are cases when it isn’t as helpful to keep it local. Maybe we want to have it run on a really powerful computer with many cores and lots of memory so it can check really, really big numbers. Well, that’s not so bad. We can run our main program on this big computer as well, so even though the rest of the program might not need it, the prime finder can use as much of the computer as it can. What if we want to let other programs reuse the prime finder? We could make it a library that we can share between programs, but we might need a lot of memory resources for every computer that runs the prime finding library.

What if we just ran the prime function on a separate computer, then talked to that computer when we needed to check a number? That way, we can beef up this prime finder computer, and we can share it with other programs running on other computers.

The downside here is more complexity. Computers fail, networks are unreliable, and we have to worry about sending the answer back and forth. If you’re just writing a math program, you may not want to worry about network details, resending lost packets, or even how to find the computer that’s running the prime finder. If your job is writing the best prime-finding program, you may not want to worry about how to listen for requests or check for closed sockets.

This is where remote procedure calls come into play. We can wrap the complexity of inter-computer communication in the center, and we can surface a simple interface, called a stub, to either side of the conversation. To the math-program writer, we can make it look like you’re just calling the function that lives on the other computer. To the prime-finding program writer, we can just make it look like your function is being called. If we abstract away the middle, each side can focus on their respective details while still being able to enjoy the benefit of splitting this computation across multiple computers.

The main job of an RPC call is to handle this middle section. Part of it has to live on the math program computer, where it has to take the arguments, package them, and send them to the other computer. When it gets a response, it has to unpackage it and pass it back. The part that lives on the prime finder computer has to wait for requests, unpackage the arguments, pass them to the function, get the result, package it, and send it back to whoever asked for it. Importantly, both the math program and the prime finder program have a clean interface between them and their respective stubs that allows for this abstraction.

Diagram from Implementing Remote Procedure Calls, Birrell & Nelson, 1981

For more details, I’d highly recommend the 1981 paper Implementing Remote Procedure Calls by Andrew D. Birrell and Bruce Jay Nelson¹, where they walk through the use of RPCs at Xerox PARC.

Writing an RPC from scratch…

What would this look like if we tried to write it?

(With a big old caveat that I’m far from an expert in C, but I can scratch enough down with a toy problem to (hopefully) get a concept across. Imagine a big asterisk after nearly every line that says “would be different for production code”. These examples are very much inspired by Beej’s Guide to Network Programming².)

We can start with our basic math program. Maybe for the sake of simplicity, it’s a command line tool that takes some input and checks whether or not it’s prime. It has a separate method, is_prime, for the actual checking.


// basic_math_program.c
#include <stdio.h>
#include <stdbool.h>

// Basic prime checker. This uses the 6k+-1 optimization
// (see https://en.wikipedia.org/wiki/Primality_test)
bool is_prime(int number) {
  // Check first for 2 or 3
  if (number == 2 || number == 3) {
    return true;
  }
  // Check for 1 or easy modulos
  if (number == 1 || number % 2 == 0 || number % 3 == 0) {
    return false;
  }
  // Now check all the numbers up to sqrt(number)
  int i = 5;
  while (i * i <= number) {
    // If we've found something (or something + 2) that divides it evenly, it's not
    // prime.
    if (number % i == 0 || number % (i + 2) == 0) {
      return false;
    }
    i += 6;
  }
  return true;
}

int main(void) {
  // Prompt the user to enter a number.
  printf("Please enter a number: ");
  // Read the user's number. Assume they're entering a valid number.
  int input_number;
  scanf("%d", &input_number);

  // Check if it's prime
  if (is_prime(input_number)) {
    printf("%d is prime\n", input_number);
  } else {
    printf("%d is not prime\n", input_number);
  }

  return 0;
}

OK, some potential issues there, and the author clearly didn’t handle all edge cases, but maybe they had some pressing deadlines. Either way, it’s using this function, and it looks like it’s working pretty well:

So far so good. Maybe we want to split this up into different files, so is_prime would be reusable between programs on the same computer. That’s easy enough. Let’s start by making a separate library for is_prime:


// is_prime.h
#ifndef IS_PRIME_H
#define IS_PRIME_H

#include <stdbool.h>

bool is_prime(int number);

#endif


// is_prime.c
#include "is_prime.h"

// Basic prime checker. This uses the 6k+-1 optimization
// (see https://en.wikipedia.org/wiki/Primality_test)
bool is_prime(int number) {
  // Check first for 2 or 3
  if (number == 2 || number == 3) {
    return true;
  }
  // Check for 1 or easy modulos
  if (number == 1 || number % 2 == 0 || number % 3 == 0) {
    return false;
  }
  // Now check all the numbers up to sqrt(number)
  int i = 5;
  while (i * i <= number) {
    // If we've found something (or something + 2) that divides it evenly, it's not
    // prime.
    if (number % i == 0 || number % (i + 2) == 0) {
      return false;
    }
    i += 6;
  }
  return true;
}

We can now include it and call it from our main program:


// basic_math_program_refactored.c
#include <stdio.h>
#include <stdbool.h>

#include "is_prime.h"

int main(void) {
  // Prompt the user to enter a number.
  printf("Please enter a number: ");
  // Read the user's number. Assume they're entering a valid number.
  int input_number;
  scanf("%d", &input_number);

  // Check if it's prime
  if (is_prime(input_number)) {
    printf("%d is prime\n", input_number);
  } else {
    printf("%d is not prime\n", input_number);
  }

  return 0;
}

Trying it again, it looks like it still works! Though it would have been nice if the original author had written some tests…

This is OK so far, but now we’ve come to that junction we mentioned earlier, where we want to distribute this across computers. We need to write the following:

Caller stub, which has to…
- Pack the argument
- Transmit argument
- Receive the result
- Unpack the result
Callee stub, which has to…
- Receive the argument
- Unpack argument
- Call the function
- Pack the result
- Transmit the result

Our example is a pretty simple one, since we’re just packing and sending a single int as the argument and receiving a single byte as a result. For the caller library, we can pack the data, create a socket, connect to a host (let’s assume localhost for now), send the data, wait to receive, unpack, and then return. Here’s what the header file looks like for the caller library:


// client/is_prime_rpc_client.h
#ifndef IS_PRIME_RPC_CLIENT_H
#define IS_PRIME_RPC_CLIENT_H

#include <stdbool.h>

bool is_prime_rpc(int number);

#endif

The astute (or just conscious) reader will notice that the interface is actually the exact same as when it was just the library, and this is the point! The caller doesn’t have to worry about anything other than the business logic it’s trying to send (but see caveats below). The implementation, on the other hand, is a little more complex:


// client/is_prime_rpc_client.c

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <netdb.h>

#define SERVERPORT "5005"  // The port the server will be listening on.
#define SERVER "localhost"  // Assume localhost for now

#include "is_prime_rpc_client.h"

// Packs an int. We need to convert it from host order to network order.
int pack(int input) {
  return htons(input);
}

// Gets the IPv4 or IPv6 sockaddr.
void *get_in_addr(struct sockaddr *sa) {
  if (sa->sa_family == AF_INET) {
    return &(((struct sockaddr_in*)sa)->sin_addr);
  } else {
    return &(((struct sockaddr_in6*)sa)->sin6_addr);
  }
}

// Gets a socket to connect with.
int get_socket() {
  int sockfd;
  struct addrinfo hints, *server_info, *p;
  int number_of_bytes;

  memset(&hints, 0, sizeof hints);
  hints.ai_family = AF_UNSPEC;
  hints.ai_socktype = SOCK_STREAM;  // We want to use TCP to ensure it gets there
  int return_value = getaddrinfo(SERVER, SERVERPORT, &hints, &server_info);
  if (return_value != 0) {
    fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(return_value));
    exit(1);
  }

  // We end up with a linked-list of addresses, and we want to connect to the
  // first one we can
  for (p = server_info; p != NULL; p = p->ai_next) {
    // Try to make a socket with this one.
    if ((sockfd = socket(p->ai_family, p->ai_socktype, p->ai_protocol)) == -1) {
      // Something went wrong getting this socket, so we can try the next one.
      perror("client: socket");
      continue;
    }
    // Try to connect to that socket.
    if (connect(sockfd, p->ai_addr, p->ai_addrlen) == -1) {
      // If something went wrong connecting to this socket, we can close it and
      // move on to the next one.
      close(sockfd);
      perror("client: connect");
      continue;
    }

    // If we've made it this far, we have a valid socket and can stop iterating
    // through.
    break;
  }

  // If we haven't gotten a valid sockaddr here, that means we can't connect.
  if (p == NULL) {
    fprintf(stderr, "client: failed to connect\n");
    exit(2);
  }

  // Otherwise, we're good.
  return sockfd;
}

// Client side library for the is_prime RPC.
bool is_prime_rpc(int number) {

  // First, we need to pack the data, ensuring that it's sent across the
  // network in the right format.
  int packed_number = pack(number);

  // Now, we can grab a socket we can use to connect see how we can connect
  int sockfd = get_socket();

  // Send just the packed number.
  if (send(sockfd, &packed_number, sizeof packed_number, 0) == -1) {
    perror("send");
    close(sockfd);
    exit(0);
  }

  // Now, wait to receive the answer.
  int buf[1];  // Just receiving a single byte back that represents a boolean.
  int bytes_received = recv(sockfd, &buf, 1, 0);
  if (bytes_received == -1) {
    perror("recv");
    exit(1);
  }

  // Since we just have the one byte, we don't really need to do anything while
  // unpacking it, since one byte in reverse order is still just a byte.
  bool result = buf[0];

  // All done! Close the socket and return the result.
  close(sockfd);
  return result;
}

As mentioned earlier, this client code needs to pack the argument, connect to the server, send the data, receive the data, unpack it, and return it. This is relatively simple for our example, since we just need to ensure the byte order of the number is in the network order.

Next, we need to run the callee library on the server. It will call the is_prime library we wrote earlier, which now lives entirely on the server.


// server/is_prime_rpc_server.c

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>
#include <arpa/inet.h>
#include <sys/wait.h>
#include <signal.h>

#include "is_prime.h"

#define SERVERPORT "5005"  // The port the server will be listening on.

// Gets the IPv4 or IPv6 sockaddr.
void *get_in_addr(struct sockaddr *sa) {
  if (sa->sa_family == AF_INET) {
    return &(((struct sockaddr_in*)sa)->sin_addr);
  } else {
    return &(((struct sockaddr_in6*)sa)->sin6_addr);
  }
}

// Unpacks an int. We need to convert it from network order to our host order.
int unpack(int packed_input) {
  return ntohs(packed_input);
}

// Gets a socket to listen with.
int get_and_bind_socket() {
  int sockfd;
  struct addrinfo hints, *server_info, *p;
  int number_of_bytes;

  memset(&hints, 0, sizeof hints);
  hints.ai_family = AF_UNSPEC;
  hints.ai_socktype = SOCK_STREAM;  // We want to use TCP to ensure it gets there
  hints.ai_flags = AI_PASSIVE;  // Just use the server's IP.
  int return_value = getaddrinfo(NULL, SERVERPORT, &hints, &server_info);
  if (return_value != 0) {
    fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(return_value));
    exit(1);
  }

  // We end up with a linked-list of addresses, and we want to connect to the
  // first one we can
  for (p = server_info; p != NULL; p = p->ai_next) {
    // Try to make a socket with this one.
    if ((sockfd = socket(p->ai_family, p->ai_socktype, p->ai_protocol)) == -1) {
      // Something went wrong getting this socket, so we can try the next one.
      perror("server: socket");
      continue;
    }
    // We want to be able to reuse this, so we can set the socket option.
    int yes = 1;
    if (setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(int)) == -1) {
      perror("setsockopt");
      exit(1);
    }
    // Try to bind that socket.
    if (bind(sockfd, p->ai_addr, p->ai_addrlen) == -1) {
      // If something went wrong binding this socket, we can close it and
      // move on to the next one.
      close(sockfd);
      perror("server: bind");
      continue;
    }

    // If we've made it this far, we have a valid socket and can stop iterating
    // through.
    break;
  }

  // If we haven't gotten a valid sockaddr here, that means we can't connect.
  if (p == NULL) {
    fprintf(stderr, "server: failed to bind\n");
    exit(2);
  }

  // Otherwise, we're good.
  return sockfd;
}

int main(void) {

  int sockfd = get_and_bind_socket();

  // We want to listen forever on this socket
  if (listen(sockfd, /*backlog=*/1) == -1) {
    perror("listen");
    exit(1);
  }
  printf("Server waiting for connections.\n");

  struct sockaddr their_addr;  // Address information of the client
  socklen_t sin_size;
  int new_fd;
  while(1) {

    sin_size = sizeof their_addr;
    new_fd = accept(sockfd, (struct sockaddr *)&their_addr, &sin_size);
    if (new_fd == -1) {
      perror("accept");
      continue;
    }

    // Once we've accepted an incoming request, we can read from it into a buffer.
    int buffer;
    int bytes_received = recv(new_fd, &buffer, sizeof buffer, 0);
    if (bytes_received == -1) {
      perror("recv");
      continue;
    }

    // We need to unpack the received data.
    int number = unpack(buffer);
    printf("Received a request: is %d prime?\n", number);

    // Now, we can finally call the is_prime library!
    bool number_is_prime = is_prime(number);
    printf("Sending response: %s\n", number_is_prime ? "true" : "false");

    // Note that we don't have to pack a single byte.

    // We can now send it back.
    if (send(new_fd, &number_is_prime, sizeof number_is_prime, 0) == -1) {
      perror("send");
    }
    close(new_fd);
  }

}

Finally, we can update our main function that runs on the client to use the new RPC library call:


// client/basic_math_program_distributed.c
#include <stdio.h>
#include <stdbool.h>

#include "is_prime_rpc_client.h"

int main(void) {
  // Prompt the user to enter a number.
  printf("Please enter a number: ");
  // Read the user's number. Assume they're entering a valid number.
  int input_number;
  scanf("%d", &input_number);

  // Check if it's prime, but now via the RPC library
  if (is_prime_rpc(input_number)) {
    printf("%d is prime\n", input_number);
  } else {
    printf("%d is not prime\n", input_number);
  }

  return 0;
}

The RPC in action:

If we run the server, we can run the client to distribute our prime check! Now, when the program is calling is_prime_rpc, all network business happens in the background. We’ve successfully distributed the computation, and the client really is calling a remote procedure.

…so you can appreciate the libraries

This is a toy example, and although it shows some of the ideas, it’s really only a toy. Real frameworks (such as gRPC³) are understandably much more complex. Our implementation could use (at least) the following improvements:

Discoverability: in the toy example above, we assumed the server was running on localhost. How would the RPC library know where to send the RPC? We would need some way to discover where the servers that can handle this RPC call live.
RPC type: we’re dealing with a very simple server for a single RPC call. What if we wanted our server to serve two different RPCs, e.g. is_prime and get_factors? We would need a way to differentiate between the two in the requests that we’re sending to the server.
Packing: Packing an integer is easy, and packing a single byte is even easier. What if we had a complicated data structure we wanted to send across the wire? What if we wanted to compress the data to save some bandwidth?
Generating code automatically: we don’t want to hand write all of the packing and networking code every time we write a new RPC. It would be great if we could just define our interface once and let the computer do the work for us, giving us just the stub to work with. This is where something like protocol buffers⁴ come into play.
Multiple languages: along the previous lines, if we’re automatically generating the stubs, we might as well let it generate it in multiple languages, so cross-service and cross-language communication is still as easy as calling a function.
Error & timeout handling: what happens if the RPC fails? What if the network goes down, the server stops, the wifi drops… What if we want to have a timeout to ensure we’re not waiting all day?
Versioning: let’s say you have all of the above, but you want to make a change to an RPC call that already has code generated and is running on multiple computers. How do you do it safely?
All of the other caveats that go along with running servers: threading, blocking, multiplexing, security, encryption, authorization…

Computer science is the business of standing on the shoulders of those that have come before us. It’s things like this that make you appreciate the libraries that do a lot of work for us already.

You're Reading This in the Future

Fri, 16 Apr 2021 19:39:25 -0400

You’re reading this in the future. Isn’t that weird?

Even I’m reading this in the future. Let’s say I write the last “C” in this sentence at 7:41:50 EDT in the evening of Friday April 16, 2021, or exactly 1618616510 seconds since January 1, 1970, UTC. Even as I started typing this next sentence, let alone editing this and pushing this to my blog, that time has passed, and we’ve all moved on.

It’s one of the most wonderful things about writing. Not only are we able to conjure up images in each others’ brains across vast distances (quick, imagine two flamingos doing a crossword puzzle), but we’re able to do it across time as well. It’s like a mental bridge to exactly what the other person bothered to write down, hopefully edit, and share. It’s often even a circular bridge to your own mind, showing up in the form of shopping lists and reminders (note to self: don’t forget to buy dog food on your way home).

Video, music, and pictures do this well too. I’ve recently gotten into watching reruns of What’s My Line, a TV game show that started in the 1950s. The only problem with it is that every episode is available on YouTube¹, and with 16 years of weekly half-hour shows, I can easily spend a large chunk of my free time watching these old snapshots of the 50s and 60s.

It’s writing, though, that goes back even farther. One of the early panelists on What’s My Line was Fred Allen, who had a successful career in vaudeville as a poor juggler and a fine comedian before transitioning to another successful career in radio. Many of Fred’s television and radio appearances were recorded, and those recordings live on for us to listen to today. His work in vaudeville, on the other hand, is much more ephemeral. Vaudeville itself is ephemeral, existing largely before things were recorded, dying in the early 1900s, and living on through acts that were able to transition to radio or movies.

I recently read Fred Allen’s autobiography about growing up in Boston and entering vaudeville². Between the vaudeville theaters, the circuits, and the countless talents, it’s a snapshot into an era that is largely otherwise inaccessible. I’m glad he wrote about his early life, because it helps that time to live on in a way that other mediums don’t.

(Scollay Square³, home to many of the theatres where Fred performed in Boston, was later demolished in the late 1950s)

It’s also a human connection, where you can see yourself and the modern world in epitaphs for beloved pets in Ancient Greece and Rome or a cuneiform complaint about customer service. Nearly everything has changed, but these motifs through time help us see what hasn’t.

At the same time, one of the joys is having a snapshot of an earlier time and viewing it with a modern lens. You’re reading this in the future, much like we’re both in the future reading about Ken Olsen saying in 1974 that he couldn’t see any use for a computer in someone’s home⁴. Some day, maybe even already in your future, we’ll be looking back on this snapshot with some mix of hindsight and incredulity, and it’s fun to think about what future readers might question:

“You could just get in your car and drive as fast as it could go on the highway? No wonder so many people died in car accidents before self driving cars.”

“Why did the number of states in the US plateau at 50 for so long before going up again?”

“Your computers only dealt with 1s and 0s? No wonder they weren’t powerful enough.”

I hope your future is going well.

Checking if the Pi is done

Tue, 10 Nov 2020 20:48:00 -0400

I recently put together a Raspberry Pi cluster to try out some toy distributed systems projects, and I wanted to start with a basic monitoring program to get my feet wet and work out the following:

Getting each node running and set up with software
Running Go on each machine
Deploying code to each machine
Nodes listening & responding to RPCs
Sending/receiving RPCs between laptop and nodes

As I’ve previously mentioned, this isn’t meant to be production grade, since I would just grab one off the shelf if that was the goal. Instead, I want to work through the problem myself to get a feel for design decisions and making it work.

System design

We can set up a basic monitoring system by running a server on each of the nodes that listens for a heartbeat RPC. As a first step, we can have the servers respond with an OK status, and anything other than an OK response can signal a problem. We can have our main node — the one that will be monitoring the others — make requests to the child nodes at a certain interval and report the results.

A quick sketch would look like this:

This means that there will be two main components to our code: a server running on each child node and a client running on the main node.

Child node code

Each child node will run a small server that waits for a heartbeat request and responds immediately with “OK”. We can keep this very simple, first by defining the RPC arguments and response:

// Args is the empty input to the heartbeat RPC.
type Args struct{}

// Reply is a simple boolean response.
type Reply struct {
  OK bool
}

We can then create an RPC server that listens for a heartbeat RPC on a given port. We’ll want our server to do the following:

Listen for incoming RPCs on a certain port
When we receive a heartbeat RPC, respond with OK
Print some debug information

This comes together quickly with Go’s RPC package:

/*
This is a simple heartbeat server we can run on the cluster nodes. It just waits
for any heartbeat and responds.
*/

package main

import (
  "log"
  "net"
  "net/http"
  "net/rpc"
  heartbeat "pi-cluster-monitoring/rpc"
)

// Server is a basic heartbeat server.
type Server int

// Heartbeat is a simple check to see if the server is still alive and responding.
func (t *Server) Heartbeat(args *heartbeat.Args, reply *heartbeat.Reply) error {
  reply.OK = true
  log.Println("Received Heartbeat. Sending OK.")
  return nil
}

func main() {
  heartbeatServer := new(Server)
  rpc.Register(heartbeatServer)
  rpc.HandleHTTP()
  l, err := net.Listen("tcp", ":1234")
  if err != nil {
    log.Fatal("listen error:", err)
  }

  log.Println("Listening on port 1234")
  go http.Serve(l, nil)
}

We can then run this on each of the child nodes, and they are now ready to be contacted.

Main node code

Next, we can build a small client that will periodically check on our listening child nodes by doing the following:

Make an RPC to each of the child nodes and report the status, either OK or an error if something went wrong
Wait a little bit
Repeat

/*
This is a simple heartbeat client we can run on a monitoring computer (i.e. not
the nodes). It wil periodically query the nodes and report their status.
*/

package main

import (
  "fmt"
  "net/rpc"
  "os"
  heartbeat "pi-cluster-monitoring/rpc"
  "time"
)

// HeartbeatIntervalInSeconds determines how often the heartbeats are sent.
var HeartbeatIntervalInSeconds = 5

func sendHeartbeat(serverAddress string) {
  client, err := rpc.DialHTTP("tcp", serverAddress+":1234")
  if err != nil {
    fmt.Println("Server "+serverAddress+" is not OK. dialing: ", err)
    return
  }

  args := &heartbeat.Args{}
  var reply heartbeat.Reply
  err = client.Call("Server.Heartbeat", args, &reply)
  if err != nil {
    fmt.Println("Server "+serverAddress+" is not OK. error calling Heartbeat: ", err)
    return
  }

  if !reply.OK {
    fmt.Println("Server "+serverAddress+" is not OK.")
    return
  }
  fmt.Println("Server "+serverAddress+" is OK.")
}

func main() {
  allServerIPs := []string{
    os.Getenv("PI1"),
    os.Getenv("PI2"),
    os.Getenv("PI3"),
    os.Getenv("PI4")}

  for true {
    fmt.Println("\nSending heartbeats")
    for _, server := range allServerIPs {
      go sendHeartbeat(server)
    }

    time.Sleep(time.Duration(HeartbeatIntervalInSeconds) * time.Second)
  }
}

A few notes:

I’ve passed in the addresses of the child nodes as environment variables, so this assumes an unchanging set of child nodes we know about in advance
We can easily send the requests (roughly) in parallel to the child nodes via goroutines (though for N=4 this is just for fun)

Once this is started on the main node (my laptop) with the child nodes already running, it immediately begins querying the child nodes and reporting their statuses:

Potential improvements

This is a very simple approach, but it was helpful for the goals I mentioned above. The basic system is set up, the nodes are communicating, and the basic monitoring will be helpful to use for future projects. Personally, I find it better to implement the minimal version that achieves the goals and concepts, as I could easily spend hours tinkering with it. Kind of like when you toil away tweaking the CSS on your personal blog instead of really writing: it’s important to actually do what you set out to do.

That being said, there are a few things I’d love to do to improve this:

Include some CPU and memory information in the heartbeat to see how busy the Pis are
Run the server on the nodes on startup so they’re always available in the background
Add some nicer reporting in the client.go output and history logging
Improve the deployment strategy, which is currently ssh’ing into each one then git pulling

More complexity

If this was supposed to be a robust monitoring system for a busy cluster, what would we have to add? A few (non-exhaustive) thoughts on possible problems and tradeoffs:

The list of nodes is static. How would we make a monitoring system that allowed for dynamically adding/removing nodes?
- We could keep the current setup but also have the main node run an RPC server with a “Register” RPC, or something similar, that would allow new nodes to add themselves to the main node’s list of children. We could also have a “Deregister” RPC for removing nodes from this list that could be callable from a CLI, for instance, when we knew a node had been removed from the system.
- Right now we have the client initiating the RPC to the nodes, which requires it to know about each node before checking in. If we switched this and had the child nodes make requests to the main node, we could just have each child know the location of the main node.
The monitoring client is on the same network as the nodes themselves, so if I was surfacing this information in a webpage served from my laptop and my home network went down, the monitoring system would go down too.
There’s only one monitoring client, and if my laptop dies, I wouldn’t be able to see if they’re up. Ideally there would be another avenue for checking.

There are plenty of improvements to make, but this is a good start.

Building a Raspberry Pi Cluster for Building Toy Distributed Systems

Sat, 24 Oct 2020 20:48:06 -0400

I’ve recently been getting more interested in distributed systems, and I wanted to get experience building some of the concepts I’ve read about.

My interest lies more on the software and system design side – more “how would we design and build an example of a distributed file system across four computers?” than “how do I make a reliable home server to play media?” – which is why my focus is on playing around with some toy software problems instead of worrying about what a real production hardware setup would look like. As with anything, it’s almost too easy to get caught up in the minor details instead of what you meant to work on, and I feel like I could spend days agonizing over hardware choices.

That being said, I did want to work with real hardware, which makes it more interesting than running VMs. There’s something about physically disconnecting ethernet switches to see the effect on the software that’s intriguing, and doubly so if the software can handle it gracefully.

I want to get a little more experience with the concepts behind some distributed systems papers and projects I’ve learned about. I’ve heard that the true devil of distributed systems lies in the details at scale, which isn’t really possible for my tiny cluster, but I’m planning to work through these projects to get a better feel for some of the general ideas. I’ve found that programming an implementation is one of the best ways to understand a concept, even if it’s a toy implementation, and I’m looking forward to doing so for some of the papers I’ve been reading.

I’m planning to write most of the software in Go because I’ve enjoyed working with its RPC library in the past and I want to get more experience working with the language. I do mostly C++/Java/JavaScript at work, and it’s always nice to explore outside of the comfort zone.

Anyways, back to the cluster; I ended up picking N=4 for my little cluster and drew up a quick diagram of what my plan:

The planned cluster, with 4 Raspberry Pis connected to power on the left and a network switch on the right

This ended up actually being pretty close to reality, which is always nice for design diagrams. I’ll spare you the order details (I’ve found these can quickly get out of date), but after the components trickled in, I wired up the Pis from 1-4 from bottom to top with a helpful case to keep them separate.

The cluster in action

So far I’ve done the following:

Installed the Raspberry Pi OS Lite on each MicroSD & enabled SSH
Grabbed the IPs and SSH’ed into each
Had to re-image a MicroSD after I didn’t save the password I generated
Did some basic ssh/scp/config setup to install .vimrc/tmux/vim/git/Go
Wrote up a basic heartbeat monitoring program to get them talking to each other
Spent 15 minutes debugging before I realized the switch was turned off

My next steps are probably something like this, time and motivation permitting:

Flesh out and write up the heartbeat monitoring program
Design and implement a basic distributed file system that I can reuse for future projects
- Probably will follow along with GFS but with some appropriate adjustments
Work through a few papers and adapt them to the mini-cluster
- Tentatively thinking MapReduce, some KV store (implementing consistent hashing), Raft, etc.

Stay tuned for more :)

Debugging a launch-blocking issue

Sun, 23 Aug 2020 23:14:15 -0400

It was around 4PM a few Fridays ago when I heard that something had gone wrong, and I volunteered (was volunteered?) to help fix it. There was a broken internal configuration that blocked future immediate work for an upcoming launch that same day. I hadn’t worked on this particular system, just similar things before, but because I’m decent at visualizing and debugging these things, my colleague asked me for help with it.

In retrospect, it was something that probably said “this should never happen” in the documentation. I scoped it out with my colleague who was more familiar with it, since he had been working on it right before things went south and was convinced he had caused the bug in one of his setup steps. He was more familiar with a similar but slightly different configuration where this issue wasn’t even possible, and he wasn’t aware that an extra fail-safe was needed to avoid what he thought had gone wrong.

I looked at the system, confirmed the problem matched what he thought it was, and started thinking about how to fix it. Initially I tried using a basic tool, but it didn’t work (issue of scale). After a quick search online, I found a few potential solutions for the same issue, but they all seemed either too complicated or involved some tooling we didn’t have readily available.

After a little more digging and thinking, it turned out we had some existing tool that we could reuse to fix the incorrect configuration, even though it was designed for a different use (who knows why it was still lying around). After some paired work on the problem, we were able to go inside the system and correct the configuration, which unblocked my colleague and the subsequent launch.

To help encourage our blameless culture, after some very light teasing I brought up the fact that anyone could have easily made this mistake – myself very much included – and helped ensure that the extra fail-safe was in place for whoever used it next.

I’ve been intentionally vague with the story because it wasn’t a software bug at work; it was a lake-side sailboat problem. My cousin had accidentally pulled one end of his sailboat’s halyard up through the mast while rigging the boat, and the end he needed threaded through at the bottom was instead at the top of the 20 foot mast.

Visualization of what went wrong (but not to scale)

This boat, a 420, had a mast where the halyard went through the middle of the mast, while other 420s he had sailed just had it going up the side. He needed to be able to pull it back through the mast to hoist the sail (blocking his late-afternoon sailboat launch). This was pretty easy to confirm, and after trying my first attempt with a short wire (an issue of scale…) and finding that most of the solutions online were kind of complicated (“tie the end of the rope to a bolt and use a magnet to feed it through”), we found some sturdy long wire we reused to feed the end of the rope through the mast. Afterwards, for the failsafe, he tied a few figure 8 knots at the end to ensure it wouldn’t get pulled up again.

I’ve been thinking a lot recently about why fixing problems with physical things is so satisfying to me, and I was thinking through how familiar the debugging experience was between this real-life fix and and most of the software debugging I do. It’s really all the same!

Code Review and Criticism

Wed, 03 Jun 2020 00:07:04 -0400

I’ve been working in software for a few years, and I’ve come to take it for granted that I am not my work. My work exists in some separate sphere, where it can easily be critiqued, dissected, refactored, or improved by me or anyone else. It’s as if it’s in a separate pond from my private one, where any ripples don’t cross the pebbly barrier to the rest of me. I’m still proud of the things I make and still take ownership over code, projects, and problems, but those things are separate from me.

This makes working with software much easier. Any comment or constructive criticism is clearly not about me; it’s about this thing I created. If it can be improved, that means there’s something to learn. If there’s something wrong with it, it means that there’s something to remember the next time around. It removes any kind of validation or personal valuation from code reviews and ensures it’s a collective effort towards solving problems and improving the codebase.

This may sound self-evident, but it wasn’t always like this for me. I remember my first proper code review pretty clearly. After learning to code mostly through solo side projects without any external input, let alone code reviews, and working a basic role doing basic dev work, also without code reviews, I ended up at a proper startup with more senior developers and a modern development process.

I remember sending out my first nontrivial code for review, then getting it back with what felt like a ton of comments. It’s funny to think back to, because now it would easily be no sweat to get a comment-heavy review, especially if it was a new codebase or a new language for me. At the time, it was a lot of sweat. I remember feeling embarrassed that there was so much wrong with it. Had I been learning wrong this whole time? Was I really qualified to be paid for this? My manager said it was a good start and was very supportive, but the imposter syndrome in the back of my head told me it didn’t matter. It felt like I had learned a word only by seeing it in writing, then I mispronounced it when trying to show that I understood the concept. This was one of the first times that I was opening up code I had written for feedback, and not being experienced with criticism of my code, it felt like it was a reflection onto me personally.

Of course, it ended up being OK. The comments were all helpful, and there was nothing personal about it. I got through that first review, and as one code review faded into ten, then dozens, then hundreds, this muscle built up over time, growing the mental separation between myself and my output.

I think putting creative work that feels like a part of yourself out there is scary at first. I’ve experienced it with having others review my code, cooking for other people, sharing a drawing I made with someone, or even writing publicly like this for anyone to read. At first it feels like it’s a part of you, and any negativity about this thing you’ve created is a direct reflection onto you. Like anything, I think practice building that barrier is important. Of course, this requires good faith on behalf of the reviewer and assumes that any feedback is constructive, if not at least correct.

This odd skill of separating your work from yourself and getting comfortable with criticism isn’t something I ever saw mentioned when I was learning to code, and I didn’t consciously think about it once I had built it up. It grew over time, like a callous on your hand you didn’t notice until you happened to think about your first blister in the same spot.

This came to mind recently as I was walking through the importance of healthy code reviews with the summer interns on our team at work, where working to foster a supportive and constructive environment reminded me of my first experiences with this odd balance. It makes me think about the other skills you build over time subconsciously, and it makes me wonder which ones I’ve forgotten I had to learn. I guess I’ll have to do my best to remember them.

Don't touch my clipboard

Mon, 17 Feb 2020 18:01:11 -0500

Looking for an em dash

I was recently trying to find an em dash character to use in a piece of writing. Since I don’t have the shortcut memorized—for future reference, shift+option+minus on Mac—I did a quick search for it. I didn’t see one on the search results page to copy, so I clicked into the first result: “The Punctuation Guide”:

From https://www.thepunctuationguide.com/em-dash.html

I’m a sucker for a good copy and paste, so I selected the em dash character, copied it, and pasted back into my doc. One problem though:

Lorem ipsum dolor sit amet, consectetur adipiscing elit—

From https://www.thepunctuationguide.com/em-dash.html
© 2020 thepunctuationguide.com

Immediately suspicious. It looks like this site is overriding the copied text somehow, and though I select a single character, it’s augmenting my copied text with a citation and copyright information.

JavaScript and client-side code

Two of the good things about JavaScript are that it runs on your computer and it’s easily inspectable. I’ve worked with JavaScript quite a bit, but I’ve never modified a user’s copy command. Out of curiousity I inspected the page to see if I could figure how they were doing this.

My first instinct was that they were using some generically named “copy-copyright” library somewhere, so I inspected the HTML. JavaScript tags usually come at the end of the page, so straight to the bottom.

Opening this up shows the 15 responsible lines of JS:

// JavaScript Document
function addLink() {

    var selection = window.getSelection(),
        pagelink = '<br /><br /> From ' + document.location.href + '<br />Â© 2020 thepunctuationguide.com',
        copytext = selection + pagelink,
        newdiv = document.createElement('div');


    newdiv.style.position = 'absolute';
    newdiv.style.left = '-99999px';


    document.body.appendChild(newdiv);
    newdiv.innerHTML = copytext;
    selection.selectAllChildren(newdiv);

    window.setTimeout(function () {
        document.body.removeChild(newdiv);
    }, 100);
}

document.addEventListener('copy', addLink);

From https://www.thepunctuationguide.com/js/copyright.js at the time of writing

This creates a function addLink that does the following:

Get the object that represents the user’s current selection
Build a string of HTML elements with the current page’s link and copyright info
Concatenate the user’s selection and that HTML
Create a new empty div element on the page
Move it well off the screen to the left
Add the new div element to the page with that the user’s selection and that HTML
Update the current selection to be the children of that div (aka select the concatenated string of user selection and attribution)
In 100ms, get rid of that element

It then adds this function as a listener on the copy event, so every time the browser hears that you’re copying, this is run first. The magic is from window.getSelection(), which gives access to the string that the user has selected. It then updates this selection with selection.selectAllChildren, which updates the selected text to be the text in the newly created div.

Now that I know how it’s done, another unusual thing makes sense—when I highlight text and hit copy, the highlight goes away. This is because the focus and selection has shifted to that other div.

Interestingly, this function must run before the copy event is fully bubbled up. This means that it’s intercepting the copy event, modifying the text that’s selected, and passing it on, like a man-in-the-middle attack against your clipboard.

This is bad

This falls into the category of not respecting the user’s actions. Attribution is important, but changing things outside of what the site should be changing is a bad pattern. It’s like smooth scrolling, where the page commandeers the user’s scroll actions. I’m strongly of the opinion that you shouldn’t do things behind the scenes that diverge from the user’s normal expectations.

One other thing I’m also curious about is the validity of the copyright for small text selections. I don’t know much about copyrights, but I’d be curious if an atomic selection from a creative work (like a single character, “—”) would fall under the same copyright as the whole work.

Ironically, a little reverse searching reveals that this code was copied verbatim without attribution from this StackOverflow post (see “Manipulating the selection” from the top answer). Maybe copy/paste isn’t so bad?

Harnessing the Power of Shower Thoughts

Sat, 16 Nov 2019 21:43:06 -0500

One of the best ways to learn and come up with new ideas is to focus intensely on a problem, then let your mind wander. I’ve enjoyed doing this when problem solving and learning, and recently I’ve been thinking about how weird it is to build trust in your unconscious to do the work for you, especially when it comes to technical work.

Letting your mind focus and letting your mind wander

The first recommendation I have for anyone taking a meta look at learning is Barbara Oakley’s Learning How To Learn. I’ve found it to be a tremendous resource for building better study habits and learning skills, and recommending the course in various threads about learning is easily the majority of my comments on Hacker News.

One of my biggest takeaways from the course was the balance between actively focusing and letting your mind wander, or the focused and diffuse modes of thinking. The focused mode is for filling your mind with information, and the diffuse mode is great at processing that information, forming connections between existing concepts, and developing new ideas.

In the course, one of the examples of consciously switching between the two modes to develop new ideas is the nap strategy employed by Salvadore Dali, where after a long period of focused work on a project, he would retire for a brief nap while holding a key in one hand above a plate. As he drifted off, his mind would wander, forming new thoughts. As soon as he entered proper sleep, his grip would relax and the key would fall to the plate, making enough noise to wake him and call him back to his work, the new ideas fresh in his head. Thomas Edison would also reportedly do the same with a ball bearing in his hand instead, harnessing the amorphous diffuse mode to come up with new ideas.

In a recent New Yorker article, The Myth and Magic of Generating New Ideas, Dan Rockmore goes into detail about working as a mathematician and coming up with new insights while jogging or lifting weights after focusing hard on the problem: “it’s just a feeling of being free, of forgetting for a moment that we are bound by gravity and logic and convention, of letting the magic happen.”

Letting the magic happen in practice

I started paying more attention to this balance around the time when I was learning to program, and I’ve been able to use these two modes when learning how to program, learning in an academic environment, and solving problems that come up in my daily work.

Working in software has graciously given me the opportunity to often have no immediate idea how to solve the problem in front of me. There’s a certain kind of cruel balance to the work; if it was exactly the same problem we were solving before, we’d just copy and paste what we did there. It’s usually not the same problem, so it usually needs a new solution.

Sometimes after investigating a problem, thinking through the edge cases, and trying an attempt or two, the right solution isn’t immediately there. Sometimes you look at a problem and something comes to you, while other times, you draw a blank (and occasionally this is where the nagging imposter syndrome thoughts sneak out of their grungy caves). My usual next steps involve writing things down, trying some code, searching for similar problems, rubber-ducking, or asking a coworker.

If none of this works, I’ve found that one of the best ways to move forward with a problem is to get away from the problem. If it’s halfway through the day, going for a walk around the block (importantly, without just going on my phone) or going for a swim will often let me realize something I missed that proves to be the key when I get back to my desk. If it’s close to the end of the day, heading home and letting the problem sit overnight is often enough to unblock me for the next morning. I’ve lost track of how many times I’ve realized a key insight on the train home, only 15 minutes of unfocused day dreaming since I was just looking intently at the code.

I’ve found that this has also been very helpful for learning new concepts. Focusing on something new in a lecture or problem set and giving it some time to soak while my mind is floating seems to let the ideas settle better than if I was just reviewing my notes.

My own recipe for striking the balance

Take some time to focus entirely on the problem
Take some time to not focus on any problem

This sounds deceivingly simple, but like any advice you read in the “How To” books, the hard part isn’t understanding it’s a good idea. The hard part is actually applying it and seeing if it helps at all.

Ways I’ve been able to let my mind wander

There are a few ways in particular that I’ve found to be especially good at shifting my mind towards the diffuse mode.

Going for a walk without my phone
Running
Taking a nap
Taking a shower
Swimming

Things I’ve found don’t work

Going on my phone mindlessly
Playing video games
Checking my email
Refreshing Hacker News, Twitter, Instagram or (insert favorite site)

It’s very easy to fool myself into thinking I’m harnessing this weird power when really I’m just noodling time away. These usually create some middle stage, with my brain distracted but not fully relaxed enough to sink into diffuse thinking. I’m also convinced that half of the reason shower thoughts are a thing is because it’s one of the only times that it’s especially hard to fill the time with a screen. Archimedes wasn’t on Instagram in the bath.

Learning to live with the magic

There’s a weird balance here between the highly logical and analytical skills and work involved in programming and the complete unknown that is the unconscious and how new ideas are generated. There’s something unsatisfying about resorting to a black box of neurons and brain matter that tosses different ideas around in the background without you, like a clothes dryer filled of ideas that’s tumbling while you sleep. When it’s done, you might be missing a sock, but somehow these nebulous thoughts came together into a solution.

This uncertainty is made worse by the fact that you can’t tell how your mind surfaced the idea. How did you come up with this solution? How do you know you’ll come up with a solution next time? The idea that you can trust yourself to come up with another idea in the future is strange.

I’m learning to live with the idea that I can’t force it, but I can guide it. I’m more comfortable with the idea now, and more than anything, I’m just glad it works.

Dark pattern for autoplaying videos: clickjacking pause

Thu, 24 Oct 2019 20:40:52 -0400

I recently saw a common clickjacking dark pattern when trying to pause a video the other day.

On a Bloomberg news autoplaying video, the sound is initially muted.

Clicking anywhere on the video will unmute it. It would be pretty easy to have an invisible element overlaying the video that listens for a click, and it looks like that's probably what we have here:

What's interesting is that the control panel (pause, play, scrubbing, etc.) at the bottom of the video is initially shown for a few seconds. This makes it look like you can click it.

If you try to click pause, your click will instead be intercepted by that covering layer and unmute the video. Only then is the control panel clickable.

User action: mouse click on pause button
Expected result: pause video
Actual result: unmute video

Classic misdirection. News websites are notorious for bad UX, and Bloomberg lives up to hype.

Clickbait Snail Mail

Sat, 19 Oct 2019 20:47:19 -0400

Junk mail is so interesting at its odd intersection between design, advertising, and psychology.

I usually try to pay attention to the ways that things around me are designed, and recently I’ve been saving some junk letters that are particularly interesting in their attention-grabbing efforts. Here are a few of the notable envelopes I’ve received in the past few months.

What if we didn’t put any information on it and made it look exotic?

Here’s a letter I received with a colorful design and no information:

You can tell by the pixelation that it’s all just printed on, and it’s actually a request for donations for a company that provides loans (but not donations?) to farmers in Africa.

There are a few things that this design does.

No return address: I don’t immediately know it’s junk from a non-profit, and it could be from someone I know. This could make me less likely to toss it immediately.
Air mail stamps and old school border design: makes it look more interesting than your usual mail, which adds to the allure. Best case scenario it was really routed through Brussels.

What can we do with color?

Next up is a bright blue letter that stood out from its all-white neighbors.

Bright blue color and embossed “hello”: these catch your eye with color and texture.
Pre-qualified: classic.
Lower case “hello”: signaling that it’s casual, not just corporate.

Mailing to people who registered dogs in Cambridge

I got a political mailer that stood out to me. It’s less about opening a letter, but it’s still advertising.

Dog-focused: they clearly know I have a dog, so they sent me the dog mailer.
Photo with a dog: Craig is clearly dog-friendly.
Endorsed by Mass Voters for Animals and Sierra Club: lots of positive signaling.

Save a child and get a map

This letter asks for your help and shows you a child that you could help. They also advertise a free gift, which is oddly a map of the United States.

Personalized: this is no “To Whom It May Concern”; it’s you that can help, and they call that out.
Showing you a child you can help: this helps make the issue real, since you can directly see who you could help.
Free map: this one is pretty odd to include, but when you get a gift by just opening the letter, opening the letter becomes even more appealing.

Action required for your electricity account

I have an account with Eversource, the utilities company that provides the electricity I’m using to write this, and I received this letter from another company, which is importantly not Eversource.

Attention Eversource Customer
Action Required by: 2019-11-30: this vague notice about a required action is a ploy to get you to open the letter, since you’d would want to know about any issue with your utilities immediately. The “required” action is actually just a deadline for a special rate to switch your electricity to be from clean energy. Important, but surely not required.

Offering offers

This one’s a classic: offering a limited time bonus for signing up and displaying it prominently.

~~30,000~~ 60,000: Double the miles! Except it’s a little unclear if the usual offer is 30,000. This is something you could easily just say, like raising a price before a “sale”.
Limited time offer: classic. Better act fast.
Float Away/photo of pool/Implying “This could be you.”: Yeah right, maybe if I looked good in a one-piece.

Keep it simple: just tell me to look inside

SoFi keeps it simple and adds some text that could easily be taken straight from a clickbait article:

Slight hyperbole

Coming soon on FiveThirtyEight: Christmas catalog power rankings.

Discounts and visible gift cards

I found this one interesting. It’s pretty common to give a special deal for signing up, and this front is no different. What sets this apart is the back of the envelope; it’s semi-translucent, and you can see the gift cards within.

9 Free Meals including free shipping: Classic free stuff, including free shipping.
Take grocery shopping off your plate: fun meal pun, fun meal company.
Visible gift cards: this gives an extra appeal of “I can see those and I want them.” They’re physical gift cards, which makes the discounts feel even realer.

I wonder how much more expensive this is to make?

Notable mentions not featured here

There are a few that I’ve gotten in the past but haven’t saved since I thought of doing this:

Mailing me a nickel and putting it in clear plastic so I can see it
Credit card offer that looks like a check (including the tearable sides like a pressure sealed check)
Credit card offer that was just a manila envelope with no return information

What’s the point?

I’ll preface this by saying I have no familiarity with advertising by mail, so the following is all speculation.

In these examples, we can see a huge push just to get you to even open the letter.

I would imagine that there’s a huge drop off in the recipient pipeline between receiving the letters and opening, then another big drop between opening and following through (signing up, sending donations, etc). There’s no tracking pixel that will report back that I’ve opened their letter, but I could see some A/B testing where they keep the contents the same and vary the envelope, or vice-versa, then see if the recipient was more or less likely to follow through.

In that lens, I could imagine these being ways to improve the open-rate, but at what cost? Like an ad click that doesn’t end up in a sale, I could see some of these resulting in more open envelopes, but it feels like a race to the bottom. At least it gives us an interesting view into how they’re designing these envelopes to be opened.

I’ll keep my eyes out for any interesting ones in the future. In the mean time, I’ll be here enjoying my free gifts.

MapReduce in Simple Language

Sat, 09 Feb 2019 00:00:00 -0500

MapReduce Explained Using the Ten Hundred Words People Use Most Often

MapReduce is a good way to take a big job that would take a long time on one computer and break it up into smaller jobs that can be done by simple computers.

If you can break down a big job into smaller jobs, you can run the smaller jobs on many simple computers. If you have a lot of simple computers, you can do a lot of the smaller jobs at the same time, and this can make your job go a lot faster. This can also help if your computers ever break. Instead of having to get a new big computer, if one of your simple computers breaks, you can just get another simple computer. This can help you save money.

The problem of breaking down a big job into smaller jobs is not simple though. How do you decide how to break up the big job? How do you decide when to send the jobs to the simple computers, and who plans it? This can be hard.

MapReduce is one way to do this. It does the planning and the sending for you, and it lets you say what the smaller jobs look like and what to do with them. Since it takes care of the hard parts, breaking up a big job is much easier.

The Cat Saw The Dog

What if we were given a book and we had to count the number of times each word appears? What if we had a lot of books to count words in? This a good problem for MapReduce because doing it by hand would be very boring, and although a single computer could probably do it, if we had a big number of books to look through, it could take a long time.

Let’s say we have a very simple book that is made up only of the words “the cat saw the dog.” Counting the words would give us this:

the: 2
cat: 1
saw: 1
dog: 1

To do this in smaller jobs, we could do something like this:

Break up the book into pages
For each page, for each word, make a card with that word on it
Group all the cards by word
For each word pile, count how many cards are in the pile
Write down the final count for each word

How would we give these to simple computers? It would be nice to have each simple computer work on one page (or even smaller parts). This means that we could break up the work between them to get it all done faster. It would also be nice if they could do the counting for each word, because then we could break that up too. If we’re able to break these down into small jobs, we can spread out the jobs to many simple computers, and these computers can work together to finish the big job much faster than if we had to do it on one computer (and a lot faster than doing it by hand).

MapReduce is a way to break this job down into small jobs that simple computers can do easily, and it takes care of the hard parts here, which are breaking up the book into pages (step 1), grouping all the cards by words (step 3), and writing down the end count for each word (step 5). The other hard part is making sure everything happens at the right time, which is not written down above; how do we know which simple computer to give what page, and when? How does the simple computer know what to do? When should a computer get cards to count, and how does it get a new pile when it has finished? MapReduce does this too, which is nice.

What about steps 2 and 4? These are the things we have to tell it about, and they are called the map and the reduce (this is why it is called MapReduce). We just have to tell it how to do these two jobs. It tells the simple computers how to do their jobs, and it figures out which computers do which jobs.

Step 2 is the map, and it has to say what to do with what is put in and how to give the answer as pairs that can be grouped and later used by the reduce. In the case above, you would be making a single card each time a word happens. It takes a page, and it gives a group of pairs made up of the key (the word) and count (the card, which acts as a 1 for counting).

Let’s see that:

'the cat saw the dog' -> [('the', 1), ('cat', 1), ('saw', 1), ('the', 1), ('dog', 1)]

MapReduce then groups the words before sending them to the computers that are doing the reduce. It knows that the first ('the', 1) and the second ('the', 1) are both for the same word, so it can group them together as the word and all of its counts: ('the', [1, 1]). This is like sorting the word cards into one pile for each word.

Step 4 is the reduce, and it has to say what to do with all of the things that came from the map for a single key. It takes a key (in this case, the pile’s word) and a group of things from the map (here, a pile of cards) and gives out what you want for that key (here, the number of cards in the pile).

Let’s see our reduce for a pile of two “the” cards (really, two “1” counts):

('the', [1, 1]) -> ('the', 2)

Once we had the counts, we could just write them all down and we’d be done. This would let us break up the job and count words in a lot of books really fast.

Looking for things

The word counting is not too exciting, and we could do even more interesting things. Let’s look at how we could do something like searching for every time a word appears in a book. This is another job that could take a while on one computer for a big number of books; let’s give it to the simple computers!

Let’s look through this book:

’Twas brillig, and the slithy toves
      Did gyre and gimble in the wabe:
All mimsy were the borogoves,
      And the mome raths outgrabe.

And let’s look for the lines that have the word “and.” Here’s one way to do the map and the reduce to do this job:

Map: Take in a line from the book, and if the word appears in the line, give the line and a 1 (key: line, message: 1). Since we aren’t counting, we can call this a message, because it’s more of a sign that we have it. If the line doesn’t have the word, we don’t give anything.

'Did gyre and gimble in the wabe' -> [('Did gyre and gimble in the wabe', 1)]
'All mimsy were the borogoves,' -> []

MapReduce then groups all of the line/message pairs, but since each line that has the word will be the key, they end up in piles of 1. If we had two lines that are the same, we could also put their line number in the key to make sure we don’t group them together.

Reduce: Take in a line and just return the line.

('Did gyre and gimble in the wabe', [1]) -> 'Did gyre and gimble in the wabe'

What would we get at the end? Since our map would only give out the line if the word appears in the line, only lines with that word would be sent to the reduce. Our reduce just gives what it gets, so we’d end up with the lines that have the word in them.

This would allow us to break down a large number of books into many lines and give them to many computers. If we have a lot of computers, we could do this very quickly!

How many of you are there?

What about something like seeing how often people go on our home page and other pages? Maybe we have a simple history of people going to pages like this:

alexanderell.is           01/01/2017
alexanderell.is           01/01/2017
alexanderell.is/about     01/01/2017
alexanderell.is           01/02/2017
alexanderell.is/about     01/02/2017
alexanderell.is           01/02/2017

It would be interesting to see how many times people went to each page. This would be very possible with MapReduce!

Map: Take in a line from the history and send out a 1 for each page (key: page, count: 1).

'alexanderell.is           01/01/2017' -> [('alexanderell.is', 1)]

Reduce: Take in a page and the group of counts of people going to it (that MapReduce grouped for us) and give out the whole count.

('alexanderell.is', [1, 1, 1, 1]) -> ('alexanderell.is', 4)

This would let us count it all very fast.

Why is this good?

Showing simple jobs for this can be boring, since they wouldn’t be hard to do even by hand. It gets interesting when you’re working with big, big numbers. What if there were more lines of the book? What if there were a hundred books? What if there were ten hundred? Ten hundred times more than that? What about ten hundred times more than that? And ten times more? That’s more books than there are people on Earth. My computer does a lot for me, but it would take a long time to look through that many things!

If you’re working with a very large number of things, that’s when breaking it up and giving it to the smaller computers works the best. MapReduce does the hard parts and lets you do just the two simple map and reduce parts, which lets you work on what you want to do and not worry about how exactly it gets done.

[/simple]

I highly recommend reading through the original paper, and you can find it here.

This was done using the XKCD SimpleWriter with the only non-simple words being “MapReduce” and “reduce.” I couldn’t think of a good way to replace “reduce,” given that it’s in the name, but if you can think of a good way to do it, please let me know!

Good Design: Double Sided Park Signs

Mon, 27 Aug 2018 23:52:22 -0400

I often walk my dog around a small reservoir near my apartment. It’s a perfect distance for tiring a small terrier, and it’s a beautiful area.

There are a number of signs around the reservoir that tell you to not swim in the water (as it’s a backup emergency water supply) and to not walk on the ice when it’s frozen over in the winter (as it’s very, very cold if you fall in). These signs are switched for the appropriate season.

On one of my walks, I noticed that one of the signs was partially hidden by its support.

I remembered having seen this sign displayed in the winter, but I didn’t remember the post in front of it. Checking the reverse, I saw that the other side was the summer-time version: “No Swimming.”

If you have two signs, each for roughly half of the year, where do you keep the other sign when it’s not in use? Here is a simple answer: on the back of the other one. Switching the seasonal signs is as easy as removing it, reversing it, and bolting it back on.

Every sign to be swapped can be visited with just a handful of tools. You don’t have to worry about storing, bringing, counting, misplacing, or manufacturing the other signs. It’s the Mass Department of Conservation and Recreation, and they want to save the extra metal!

It’s not the cleanest looking solution, but it is simple and it works. It seems like an obvious choice when you notice it, as good design should.

Why does a spray bottle work?

Fri, 08 Jun 2018 10:27:44 -0400

Exploring a simple machine

I took apart a simple spray bottle to see how it kept fluid flowing only in one direction from the reservoir to the nozzle. It does so with simple check valves made of two pieces of plastic, a spring, and a small sphere. It’s a simple way to solve a tricky problem.

A simple spray bottle

One way flow

It’s often useful to restrict flow to only one direction. One simple use case is in a spray bottle. Let’s think of a simplified spray bottle with three main components: the reservoir where we store the fluid, the staging area for the fluid that’s about to be sprayed, and the nozzle that connects it to the space outside where we’ll dispense our fluid. We want to take fluid from the reservoir, put it in the staging area, and dispense it to the outside.

Fluid sounds like a technical term, but it’s an important one in this context. Fluid refers to anything that flows, meaning its particles aren’t fixed. This means that both gases and liquids are fluids. We want to pay attention to particles that can move, as they’re the ones that move and push around on their own.

This sounds very abstract. Here are those sections on a spray bottle:

Here’s a simplified picture:

The main user interface for the spray bottle is the trigger. This is a simple lever with a plunger that extends into the staging area. When the trigger is pressed, it pushes on the fluid that is in the staging area, which then travels out of the nozzle. When the trigger is relaxed, a spring pushes it back out, creating a vacuum effect that pulls fluid from the reservoir into the staging area, allowing the cycle to continue with the next pull of the trigger.

Why this works

There are two important things happening here. First, as the plunger applies pressure to the fluid, it must move only through the nozzle to the exterior. If the fluid only went back to the reservoir, we would have a simple machine that draws water from its reservoir and pushes it back in. Not very helpful.

Also, when the trigger is drawn back out, it must refill the staging area from the reservoir, not from any fluid from the outside. In the case of a water spray bottle, the outside fluid is air. If it refilled the staging area with air, attempting to spray would just shoot the air back out of the nozzle. This would also be a simple machine that just sprayed a small amount of air. Again, not very helpful.

Solving these problems

These two problems are solved with the use of simple connections that allow fluid to move in only one direction. If we put one in between the reservoir and the staging area, we will never have to worry about fluid going from staging back to reservoir. This solves our first issue. If we also put one in the nozzle such that fluid can only move from staging to the nozzle, the air would never come into the staging area when we relax the trigger.

Very helpful. We can construct a very simple one-way connection using three main components: a ball, a spring, and a cylinder with a hole in one end.

The spring keeps the ball pushed against the circular hole. This means that any fluid on the spring side doesn’t fit through the hole, as the ball is in the way. Both the pressure from the spring and any pressure from fluid on the right pushes the ball tighter against the hole, ensuring that there’s no space for fluid to flow to the left.

Now, let’s apply pressure on the left side with a plunger, which causes the fluid on the left to push against the ball. If that pressure is more than the pressure from the right (from the spring and the right fluid), it will push the ball back, allowing some fluid from the left to escape around it through the hole. Fluid now successfully flows from the left to the right. When we stop applying pressure, the pressure from the right once again dominates, and the ball is pushed tightly back against the hole forming a seal.

Note that we’re talking about pressure from a fluid. This pressure could be from water, oil, air, or any other fluid. The general idea of pressure is the same for each; many small particles are bouncing against surfaces.

If we’re able to apply pressure on one side, we can make it so that our fluid only flows one direction.

Let’s put a one way connection that allows flow from the staging area to the nozzle. When we push on water in the staging area, it will force its way through the nozzle, but when we relax the trigger, air will not come back in. Here, the force that pushes against the spring to open the valve is the pressure created from pulling the trigger.

We also want to ensure that fluid can only flow from the reservoir to the staging area. Let’s add a one-way connection there too.

This also makes sense. When we relax the trigger, we want it to pull water from the reservoir, but when we push on the same water, we don’t want it to go back to the reservoir. Here, the force that pushes against the spring to open the valve is the pressure created from the vacuum pulling on the spring side.

Here's a flow diagram with the one way check-valves. The arrows denote the direction of the flow for each valve.

With those two check-valves in place, we can now exactly control the flow direction of the fluids, and we can successfully spray our water.

Here’s the check valve from the nozzle of the spray bottle. The cylinder on the left goes over the group on the right, and the ball is pressed into the underside of the hole.

Check valve from the nozzle. Which way should the ball point?

Schrader Valves

What if we wanted to be able to open the valve manually? This is actually the same principle behind air valves on car and bicycle tires. Instead of a ball, a specially shaped pin is forced by a spring to form a seal. Part of that pin extends past the valve so that you can push it from the other side of the valve. Below is an example of a Schrader valves, and it works in a very similar manner. Take a look at the animation; how is it similar to the ball check valve from earlier? How is it different?

Thanks to Staffanlincoln for the Schrader valve GIF.

Update: this post got quite a few comments on Hacker News, and I recommend reading through for some interesting discussion! https://news.ycombinator.com/item?id=17273649

Second update: this post has been translated into Japanese if you’re interested: https://gigazine.net/news/20180611-why-does-spray-bottle-work/

Letting the User Guide You: Analyzing Store Design

Wed, 06 Jun 2018 09:02:23 -0400

I enjoy looking at design in everyday life and the balance between how things are designed and how people try to use them.

A store I visited recently had bad customer flow, and the signs were there. There was literally a sign telling customers where to line up for the checkout line. Customers didn’t naturally line up there, and an employee was correcting customers who missed the sign. This is bad design.

Here’s a quick sketch of the layout of the checkout corner of the store (not quite to scale):

A curved counter extended from one wall towards the back of the store. Two cash registers (1 and 2 on the diagram) were on the bottom-right section of the curve. There was a display of small checkout trinkets at B and calligraphy materials on the right wall at A.

Where would you line up? Where would you want customers to line up?

Imagine you’re a customer seeing this for the first time. It would be natural to line up at B after you approach the cash registers from the rest of the store. That’s where the majority of the supplies are, so that’s probably where you spent most of your time. You could spend your line-waiting time looking at the checkout trinkets (“I should buy dog stickers for $2.99!”). Once you check out, you could turn to the right and follow the wall to the exit.

However, this store had the line start at A. They had a “Line Starts Here” sign hanging from the ceiling. Customers who didn’t see the sign would line up at B, and one of the employees would let them know that the line began at A. Judging from the employee’s exasperated tone, this happened often.

Customers would then walk over to A. If there were already other customers waiting, the arriving customer would walk to the back of the store to the end of the line. A’s corridor is only around two people wide, which meant there was congestion as people shuffled to the rear with even more if someone was looking at the calligraphy section.

This is wasted energy. The customer has to think about where to go, and their experience is affected. If their experience is anything but easy, it’s not right. For the employees, having to worry about how customers are lining up is time and energy that could be better spent elsewhere.

Disclaimer: I was not one of the disgruntled customers. I noticed the odd design when looking for the checkout, and the sign stood out to me. I noticed the sequence above happen multiple times while I was waiting to check out. Life is too short to worry about things like getting to check out first, but life is not too short to think about peoples’ experiences.

How can we tell this is bad design? We can tell it’s not intuitive, as customers are not lining up at A on their own. If it was intuitive, you wouldn’t need to put up a sign. Even with the sign it’s not natural; the employee had to tell people where to go.

People are generous with their intuitions. They will show you what they expected as they try to do that first, and you have to listen. Seeing a user interact with a design underlines the differences between the assumptions of the designer and the assumptions of the user, and it lets you build empathy for your user.

In this case, the customers are showing their assumptions and are already doing what’s natural. Let them line up at B. Ditch the sign. Let your employees not worry about directing, and let customers go to the rear of the store on their own if they want to wrap gifts. Keep it simple. Design isn’t easy, but it’s not always hard.

PS: If you can identify and tell me what store this is, I’ll buy you a beer/coffee/tea/notebook if you’re ever in Boston! Side note: the store may be in the Boston area.

Building a Simple Cache Server in Python

Sat, 26 May 2018 10:24:05 -0400

I’m a fan of small toy projects that have the sole purpose of demonstrating a concept. This is a small project that demonstrates how a cache server works.

I think it’s important to balance toy projects with getting exposure to production-level code. While this project has been helpful for understanding the idea, an open source project like [Squid-Cache] (https://github.com/squid-cache/squid) is definitely on my reading list!

At the very simplest level, an HTTP server responds to requests. Let’s take a look at a simple server that can respond to GET requests:

Here’s a pseudo-code for a very basic HTTP server that can respond to GET requests:

- Wait for a request
  - When a request comes in, check what file it's looking for.
  - If we have it:
    - Great! Send it.
  - If we don't have it:
    - Send back a 404.

Caching

At the very simplest level, a cache server is responsible for serving items that have already been requested from the main server. Doing this gives the main server a break from having to respond to every request, and the workload can be shared by the cache server.

Here’s the pseudocode for a very basic cache server:

- Wait for a request
  - When a request comes in, check what file it's looking for.
  - If we have it stored in the cache:
    - Send it.
  - If we don't have it in the cache:
    - Request it from the main server
    - If the main server has it:
      - Store a copy in the cache
      - Send the file to the client
    - If the main server does not have it:
      - Send a 404 to the client

For instance, here’s a diagram of fetching an item from a cache server when the cache server has not already saved the item and the item exists on the main server.

Here are the steps for the first time an item is requested:

Client requests item
Cache server checks if it’s stored in the cache
The item is not found
The cache server requests the item from the main server
The main server sends the item back
The cache server saves a copy of the item
The cache server sends the client the item

We could definitely swap 6 and 7, especially if we’re trying to send the item back to the client quickly.

Here’s a diagram of a subsequent request for the same item:.

Here are the steps for the future request:

Client requests item
Cache server checks if it’s stored in the cache
The item is found
The cache server sends the client the item

Although it looks like the main server is lonely to the right, it didn’t have to worry about the client’s request at all. Lucky main server!

Here’s another way of looking at the problem. Here, the vertical axis is time increasing as you go down, and the horizontal arrows are the requests back and forth. The same actions are taken in the same exact order, but the actions are separated vertically by when they happen:

This isn’t too bad! Let’s build a very simple version in python. We’ll use joaoventura’s simple python webserver as our simple http server and a template for our cache server.

Here’s how we can set the server up to abstract away the caching in the main process:

# cacheproxy.py

"""
    Implements a simple cache proxy
"""

import socket
from urllib.request import Request, urlopen, HTTPError
import argparse


def main():
    # Get port command line argument
    parser = argparse.ArgumentParser()
    parser.add_argument('port')
    args = parser.parse_args()

    # Define socket host and port
    SERVER_HOST = '0.0.0.0'
    SERVER_PORT = int(args.port)

    # Initialize socket
    server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_socket.bind((SERVER_HOST, SERVER_PORT))

    server_socket.listen(1)

    print('Cache proxy is listening on port %s ...' % SERVER_PORT)

    while True:
        # Wait for client connection
        client_connection, client_address = server_socket.accept()

        # Get the client request
        request = client_connection.recv(1024).decode()
        print(request)

        # Parse HTTP headers
        headers = request.split('\n')

        top_header = headers[0].split()
        method = top_header[0]
        filename = top_header[1]

        # Index check
        if filename == '/':
            filename = '/index.html'

        # Get the file
        content = fetch_file(filename)

        # If we have the file, return it, otherwise 404
        if content:
            response = 'HTTP/1.0 200 OK\n\n' + content
        else:
            response = 'HTTP/1.0 404 NOT FOUND\n\n File Not Found'

        # Send the response and close the connection
        client_connection.sendall(response.encode())
        client_connection.close()

    # Close socket
    server_socket.close()

Depending on how fetch_file is implemented, this could just be a regular server that serves local files. We can make this more interesting by implementing the caching at this point:

def fetch_file(filename):
    # Let's try to read the file locally first
    file_from_cache = fetch_from_cache(filename)

    if file_from_cache:
        print('Fetched successfully from cache.')
        return file_from_cache
    else:
        print('Not in cache. Fetching from server.')
        file_from_server = fetch_from_server(filename)

        if file_from_server:
            save_in_cache(filename, file_from_server)
            return file_from_server
        else:
            return None

This matches up exactly with our pseudocode. If we have it stored in the cache, return it. If we don’t have it in the cache, request it from the main server. If we get it from the main server, save a copy in the cache and return it, but if we don’t get it back from the main server, return None.

Since above we’re checking on the return value from this function to decide if we return a 200 or 404, returning None from this function will correctly return a 404 to the client.

Let’s take a look at what a simple fetch_from_cache implementation might look like:

def fetch_from_cache(filename):
    try:
        # Check if we have this file locally
        fin = open('cache' + filename)
        content = fin.read()
        fin.close()
        # If we have it, let's send it
        return content
    except IOError:
        return None

This is about as simple as it gets! Check the local cache/ directory for the file, read it, and return it. If we run into any “file not found” issue, we’ll just return None.

Next, let’s take a look at a simple implementation of fetch_from_server:

def fetch_from_server(filename):
    url = 'http://127.0.0.1:8000' + filename
    q = Request(url)

    try:
        response = urlopen(q)
        # Grab the header and content from the server req
        response_headers = response.info()
        content = response.read().decode('utf-8')
        return content
    except HTTPError:
        return None

Once again, pretty simple! It creates a url to request from the main server running on localhost:8000, then it attempts to get the file from that endpoint. If it runs into a 404 HTTPError, it simply returns None, which is then passed up and results in a 404 being returned from the cache server.

Finally, let’s take a look at save_in_cache, which allows us to store a local copy to serve on the next duplicate request:

def save_in_cache(filename, content):
    print('Saving a copy of {} in the cache'.format(filename))
    cached_file = open('cache' + filename, 'w')
    cached_file.write(content)
    cached_file.close()

Very simple! It takes the filename and content and writes it in the cache/ directory.

Putting this all together, we have a simple cache server! Here’s what it looks like when you run it for the first time:

Note that the main server in the top left responds to the first curl, but the cache handles the second one!

How could we improve this further? First, our simple server only responds to GET requests, and it doesn’t pay any attention to cache-related headers. Furthermore, there’s no expiration date for the files it saves, so the index.html it saves today would be served from the cache forever. For showing the main functionality, this works, and we can add more complexity in the future.

Here’s the full source code on GitHub.

Investigating JavaScript's RegExp Lookbehind

Sat, 28 Apr 2018 13:22:25 -0400

Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems.

Jamie Zawinski

Regular Expression Lookbehinds

I recently came across an interesting regular expressions use case that led to an interesting look into regular expressions, JavaScript engines, and release versions.

In particular, I was trying to find a certain substring from a string that did not contain a substring preceding it.

Let’s say I have the strings 'aabbcc' and 'bbccdd'. I might want to be able to locate the substring 'cc' only if it occured at some point after the substring 'aa'. This is called a positive lookbehind, and this would only match the 'cc' in the first string.

On the other hand, if I wanted to match 'cc' only if it does not come after 'aa', this would be a negative lookbehind. This would only match the 'cc' in the second string.

Using lookbehinds in JavaScript

If you’re using Chrome 62 or higher, Node 6.0.0 or higher (with the --harmony flag), or Node 8.1.10 or higher (see below for finding these details), you can use negative lookbehinds through Google’s V8 engine. Here’s how you would do those cases above:

~: $ nvm use 8.10.0
Now using node v8.10.0 (npm v5.6.0)

~: $ node

// Positive lookbehind checking 'aabbcc' for 'cc' preceded by 'aa' at some point
> /(?<=aa.*)cc/.exec('aabbcc')
[ 'cc', index: 4, input: 'aabbcc' ]

// Positive lookbehind checking 'bbccdd' for 'cc' preceded by 'aa' at some point
> /(?<=aa.*)cc/.exec('bbccdd')
null

// Negative lookbehind checking 'aabbcc' for 'cc' not preceded by 'aa' at some point
> /(?<!aa.*)cc/.exec('aabbcc')
null

// Negative lookbehind checking 'bbccdd' for 'cc' not preceded by 'aa' at some point
> /(?<!aa.*)cc/.exec('bbccdd')
[ 'cc', index: 2, input: 'bbccdd' ]

Looking into compability

In my case, I was trying to do a negative lookbehind for my replacement. I was also attempting to do so with just a regular expression and no additional parsing through a function, as I was passing it to another module. After a few attempts by hand, I did some searching to see if it was possible. The results were not encouraging, mostly indicating in older posts that it was not supported but there were many hacks to work around it.

I then came across 2ality’s blog post on negative lookbehinds, which looked much more promising.

The proposal “RegExp Lookbehind Assertions” by Gorkem Yakin, Nozomu Katō, Daniel Ehrenberg is at stage 4. This blog post explains it.

This post featured a great explanation, and of note, an interesting link to a Feb 2016 blog post from the Google V8 team at the bottom: V8 JavaScript Engine: RegExp lookbehind assertions. The gist of that post is that the lookbehind assertions were very valuable, and although the TC39 proposal was in its early stages, it was already being rolled out in V8 version 4.9 and Chrome 49.

At this point, I tried the posted examples in both my local Node (version 6.11.1) and Chrome (66). They threw an error in Node, but worked in Chrome. My usecase was part of a Node build script, so although the version was throwing an error, it seemed there was hope for a later version. At this point, I could have just used the most recent version of Node or used the --harmony flag, but I wanted to confirm versions to ensure that I was able to be exact about the required version for future use.

I then found a more recent blog post by the V8 team from July 2017 featuring upcoming features. In particular:

Lookbehind Assertions

Lookahead assertions have been part of JavaScript’s regular expression syntax from the start. Their counterpart, lookbehind assertions, are finally being introduced. Some of you may remember that this has been part of V8 for quite some time already. We even use lookbehind asserts under the hood to implement the Unicode flag specified in ES2015.

Then, an even more recent V8 release post from September 2017 for V8 6.2:

More regular expressions features

…

Lookbehind assertions, another new regular expression feature, are now available by default.

This meant that V8 6.2 had them available, which is why it worked in my Chrome 66. The next check was to the Node releases page. Here’s the relevant section:

Version	LTS	Date	V8

Node.js 8.10.0	Carbon	2018-03-06	6.2.414.50
Node.js 8.9.4	Carbon	2018-01-02	6.1.534.50

Node.js 6.0.0		2016-04-26	5.0.71.35
Node.js 5.12.0		2016-06-23	4.6.85.32

This means that V8 version 6.2 made it into Node on March 6, 2018 in version 8.10.0, and V8 version 4.9 (allowing this with the --harmony flag) made it into Node 6.0.0. After switching to 8.10.0, my negative lookbehinds successfully ran on Node, and I was able to finish the build script.

JavaScript: Writing your own Array.reduce

Sun, 01 Apr 2018 09:51:00 -0400

I’ve found that one of the best ways for me to really understand anything is to implement it myself. Between the ECMAScript spec and a few lines of JavaScript, you can build a chunk of knowledge for a very specific area of JavaScript. Doing this often is a great way to gain a deeper understanding of the language, and it’s a great exercise in reading the docs, reading code, and creating your own solution.

In this post, I’ll walk through Array.reduce and my own implementation. Although it’s easy to read my code, I highly recommend that you go through this process yourself.

Deep knowledge of how Array.reduce works is useful to have in your JavaScript toolkit. It allows you to succinctly break down an array using a function of your choice. The idea of reducing, especially when paired with mapping, is a common one that can be found in MapReduce, Redux, mongoDB, and many other areas.

Let’s start by looking at the spec for Array.reduce:

22.1.3.18 Array.prototype.reduce ( callbackfn [ , initialValue ] )

NOTE callbackfn should be a function that takes four arguments. reduce calls the callback, as a function, once for each element present in the array, in ascending order.

When the reduce method is called with one or two arguments, the following steps are taken:

1. Let O be ToObject(this value).
2. ReturnIfAbrupt(O).
3. Let len be ToLength(Get(O, "length")).
4. ReturnIfAbrupt(len).
5. If IsCallable(callbackfn) is false, throw a TypeError exception.
6. If len is 0 and initialValue is not present, throw a TypeError exception.
7. Let k be 0.
8. If initialValue is present, then
  a. Set accumulator to initialValue.
9. Else initialValue is not present,
  a. Let kPresent be false.
  b. Repeat, while kPresent is false and k < len
    i. Let Pk be ToString(k).
    ii. Let kPresent be HasProperty(O, Pk).
    iii. ReturnIfAbrupt(kPresent).
    iv. If kPresent is true, then
      1. Let accumulator be Get(O, Pk).
      2. ReturnIfAbrupt(accumulator).
    v. Increase k by 1.
  c. If kPresent is false, throw a TypeError exception.
10. Repeat, while k < len
  a. Let Pk be ToString(k).
  b. Let kPresent be HasProperty(O, Pk).
  c. ReturnIfAbrupt(kPresent).
  d. If kPresent is true, then
    i. Let kValue be Get(O, Pk).
    ii. ReturnIfAbrupt(kValue).
    iii. Let accumulator be Call(callbackfn, undefined, «accumulator, kValue, k, O»).
    iv. ReturnIfAbrupt(accumulator).
  e. Increase k by 1.
11. Return accumulator.

The length property of the reduce method is 1.

Overall it’s not too bad! Let’s write this out in pseudocode:

1. Keep track of input object with O
2. If there are any issues doing so, return
3. Keep track of the length of the input object with variable len
4. If there are any issues doing so, return
5. If we can't call callbackfn, eg it isn't a function, throw a TypeError exception (we need a function to call)
6. If we have an empty input but don't have an initial value, throw a TypeError exception (we need to have a value to start with)
7. Initialize a counter variable k to be 0
8. If we have an initial value,
  a. Initialize our accumulator variable to be that initial value
9. If we don't have an initial value,
  a. Initialize a present indicator variable kPresent to be false
  b. While we haven't found an item and our counter variable is less than the length of our input,
    i. Get the string version of our counter variable to use for indexing
    ii. Check whether or not our input object O has an element at the current index
    iii. If there's any problem indexing in to our input element, return
    iv. If our input object O has an element at the current index,
      1. Initialize our accumulator variable to be that element
      2. If there's any problem accessing that element, return
    v. Increase our counter variable to the next index
  c. If we've gone through all the indexes of the input object and didn't find a value, throw a TypeError exception (since we need a value to start with)
10. While our counter variable k is less than the length of our input object,
  a. Get the string version of our counter variable to use for indexing
  b. Check whether or not our input object O has an element at the current index
  c. If there's any problem checking, return
  d. If there is an element at the current index,
    i. Assign that element to a variable kValue
    ii. If there's a problem accessing or assigning, return
    iii. Call our callback function with the accumulator variable, the current element kValue, the current index k, and our input O, then assign that value to the accumulator variable
    iv. If there was any problem calling the callback, return
  e. Increase our counter variable by one
11. We're done! Return the accumulator variable

That ended up much longer than it needs to be, in part because the specification is very specific. Let’s put this in broad brush strokes:

1. Make sure we have an object with length, a callback function, and a value to start with
2. Loop through the rest of the values and update the accumulator variable with the result of our callback function called with the accumulator and the value
3. Return the accumulator variable

That doesn’t look so bad! If you’re looking for an interesting exercise, try writing your own reduce function. Here’s one way to do it in JavaScript:

function reduce (input, callback, initialValue) {
  let O = input;
  const inputLength = O.length;


  if (typeof callback !== 'function') {
    throw new TypeError('callback must be a function');
  }

  if (!inputLength && !initialValue && initialValue !== 0) {
    throw new TypeError('We need an initial value');
  }

  let k = 0;

  let accumulator;
  let Pk;
  let kPresent;

  if (initialValue) {
    accumulator = initialValue;
  } else {
    kPresent = false;
    while (!kPresent && k < inputLength) {
      Pk = String(k);
      kPresent = O.hasOwnProperty(Pk);

      if (kPresent) {
        accumulator = O[Pk];
      }
      k++;
    }

    if (!kPresent) {
      throw new TypeError('If no initial value is given, the input must have a value to use as the initial value');
    }
  }

  while (k < inputLength) {
    Pk = String(k);
    kPresent = O.hasOwnProperty(Pk);

    if (kPresent) {
      let kValue = O[Pk];
      accumulator = callback(accumulator, kValue, k, O);
    }

    k++;
  }

  return accumulator;
}


console.log(reduce([1, 2, 3], (total, current) => total + current));
> 6

console.log(reduce([1, 2, 3], (total, current) => total + current, 10));
> 16

Success!

If you’ve written your own, I highly recommend comparing to both my version and the MDN Polyfill version. What’s different? What choices did you make that are different?

typeof null: investigating a classic JavaScript bug

Mon, 19 Mar 2018 08:40:17 -0400

In my last post, I looked into some JavaScript casting and explored why 0 <= null evaluates as true.

For this post, I’d like to investigate another unexpected behavior in JavaScript: why typeof(null) evaluates as 'object'.

This is a well-known bug, and we’ll investigate first in the ECMAScript specification followed by a deep dive into an early implementation of JavaScript to see the bug in its natural habitat.

The main idea is that the code assigned each item some bits for use as flags for different types, but null was different. Objects had a flag of 000, so an object’s last 3 bits were 000. null was previously defined as 32 0 bits: 00000000000000000000000000000000. When the code tried to check null’s flag, the last three bits of null (000) matched the Object flag (000), so it was incorrectly determined to be an object.

Let’s take a deeper look!

As before, for a proper background and preparation for this post, please watch (or rewatch!) wat.

To start, let’s take a look at the ECMAScript spec for typeof:

The typeof Operator

Runtime Semantics: Evaluation

UnaryExpression : typeof UnaryExpression

1. Let val be the result of evaluating UnaryExpression.
2. If Type(val) is Reference, then
  a. If IsUnresolvableReference(val) is true, return "undefined".
3. Set val to ? GetValue(val).
4. Return a String according to Table 35.

So when we call typeof null, null is passed in as the UnaryExpression. It is evaluated in step 1, and val gets its value, null. Type(null) evaluates to the Null type, so Type(val) is not Reference and step 2 is false. Next, step 3 calls GetValue on null. We saw get value before, but let’s take another look to confirm nothing odd is going on:

GetValue ( V )

1. ReturnIfAbrupt(V).
2. If Type(V) is not Reference, return V.
3. Let base be GetBase(V).
4. If IsUnresolvableReference(V) is true, throw a ReferenceError exception.
5. If IsPropertyReference(V) is true, then
  a. If HasPrimitiveBase(V) is true, then
    i. Assert: In this case, base will never be undefined or null.
    ii. Set base to ! ToObject(base).
  b. Return ? base.[[Get]](GetReferencedName(V), GetThisValue(V)).
6. Else base must be an Environment Record,
  a. Return ? base.GetBindingValue(GetReferencedName(V), IsStrictReference(V)) (see 8.1.1).

Calling GetValue with null will only reach the second step, as Type(null) is the Null type, and GetValue will return null.

Finally we reach step 4: “Return a String according to Table 35.” Here’s Table 35:

Type of val	Result
Undefined	“undefined”
Null	“object”
Boolean	“boolean
Number	“number”
String	“string”
Symbol	“symbol”
Object (ordinary and does not implement [[Call]])	“object”
Object (standard exotic and does not implement [[Call]])	“object”
Object (implements [[Call]])	“function
Object (non-standard exotic and does not implement [[Call]])	Implementation-defined. Must not be “undefined”, “boolean”, “function”, “number”, “symbol”, or “string”.

Not very illuminating. It says that typeof null should just return the string 'object'. But why? Let’s dive a little deeper.

I came across the 2ality post about the history of typeof null, and it answers the question very well with reference to an early implementation. As Brendan Eich wrote in the top comment, the code comes from the 1996 Spider Monkey implementation, not the very original 10-day Mocha, although the bug existed in the original as well. Let’s take a look!

Although the link from that post is deprecated, we can still see the classic version of the code on https://dxr.mozilla.org/classic/source/js/src/.

Two main C files will be of interest here at first: classic/js/src/jsapi.h and classic/js/src/jsapi.c. Let’s take a look at the relevant part of jsapi.c first:

JS_PUBLIC_API(JSType)
JS_TypeOfValue(JSContext *cx, jsval v)
{
    JSType type = JSTYPE_VOID;
    JSObject *obj;
    JSObjectOps *ops;
    JSClass *clasp;

    CHECK_REQUEST(cx);
    if (JSVAL_IS_VOID(v)) {
        type = JSTYPE_VOID;
    } else if (JSVAL_IS_OBJECT(v)) {
        obj = JSVAL_TO_OBJECT(v);
        if (obj &&
            (ops = obj->map->ops,
             ops == &js_ObjectOps
             ? (clasp = OBJ_GET_CLASS(cx, obj),
                clasp->call || clasp == &js_FunctionClass)
             : ops->call != 0)) {
            type = JSTYPE_FUNCTION;
        } else {
            type = JSTYPE_OBJECT;
        }
    } else if (JSVAL_IS_NUMBER(v)) {
        type = JSTYPE_NUMBER;
    } else if (JSVAL_IS_STRING(v)) {
        type = JSTYPE_STRING;
    } else if (JSVAL_IS_BOOLEAN(v)) {
        type = JSTYPE_BOOLEAN;
    }
    return type;
}

Let’s walk through what this does.

First, it checks JSVAL_IS_VOID(v). That is defined in jsapi.h as follows:

#define JSVAL_IS_VOID(v)        ((v) == JSVAL_VOID)

And JSVAL_VOID is defined as:

#define JSVAL_VOID              INT_TO_JSVAL(0 - JSVAL_INT_POW2(30))

So the void value, what we know in JavaScript as undefined, is defined as -(2^30), an integer outside of the integer range. The typeof call first checks if that’s what we’re dealing with. If so, the line type = JSVAL_VOID is called, and then that is returned. Good so far!

If we aren’t dealing with a void value, the code then evaluates JSVAL_IS_OBJECT(v). This is defined as follows in jsapi.h:

#define JSVAL_IS_OBJECT(v)      (JSVAL_TAG(v) == JSVAL_OBJECT)

So this checks a tag on the value and sees if it matches the object tag. Here are the tag definitions from jsapi.h:

/*
 * Type tags stored in the low bits of a jsval.
 */
#define JSVAL_OBJECT            0x0     /* untagged reference to object */
#define JSVAL_INT               0x1     /* tagged 31-bit integer value */
#define JSVAL_DOUBLE            0x2     /* tagged reference to double */
#define JSVAL_STRING            0x4     /* tagged reference to string */
#define JSVAL_BOOLEAN           0x6     /* tagged boolean value */

This means that objects will have 000 as the bits for its tag, integers will have 001, doubles will have 010, strings will have 100, and booleans will have 110.

Here’s JSVAL_TAG and its called JSVAL_TAGMASK:

#define JSVAL_TAGBITS           3
#define JSVAL_TAGMASK           JS_BITMASK(JSVAL_TAGBITS)
#define JSVAL_TAG(v)            ((v) & JSVAL_TAGMASK)

JS_BITMASK comes from a different file, jstypes.h, and it builds the correct value to bitmask against tag. This means that it grabs the last 3 bits from the value, and those three bits correspond to the tag.

For instance, if we were working with a very simplified version with integers of 5 bit length and the last 3 bits were the tag, we may have a value like 00001001. Here, the first 5 bits, 00001, would correspond to the value, and the last 3 bits, 001, wold be the tag (here, for an integer). To get just the tag, we would do a bitwise AND comparison between our value 00001001 and a mask with 1s for the last three bits, 111, to get just the tag, 001.

Value:           00001001
Bitmask:         00000111
Value & bitmask: 00000001

So far so good! Let’s see what we can find about the null value. A quick search reveals the definition for null from jsapi.h:

/*
 * Well-known JS values.  The extern'd variables are initialized when the
 * first JSContext is created by JS_NewContext (see below).
 */
#define JSVAL_VOID              INT_TO_JSVAL(0 - JSVAL_INT_POW2(30))
#define JSVAL_NULL              OBJECT_TO_JSVAL(0)
#define JSVAL_ZERO              INT_TO_JSVAL(0)
#define JSVAL_ONE               INT_TO_JSVAL(1)
#define JSVAL_FALSE             BOOLEAN_TO_JSVAL(JS_FALSE)
#define JSVAL_TRUE              BOOLEAN_TO_JSVAL(JS_TRUE)

Here, JSVAL_NULL is OBJECT_TO_JSVAL(0). Let’s follow this down:

jsapi.h:

#define OBJECT_TO_JSVAL(obj)    ((jsval)(obj))

jspubtd.h:

typedef jsword    jsval;

jscompat.h:

typedef JSWord jsword;

jstypes.h:

/*
** A JSWord is an integer that is the same size as a void*
*/
typedef long JSWord;
typedef unsigned long JSUword;

This means that null, OBJECT_TO_JSVAL(0), is actually just creating a long with just 0 as the value. This is just 00000000000000000000000000000000 (32 bits, all 0).

Now, let’s go back to our JSVAL_IS_OBJECT call

#define JSVAL_IS_OBJECT(v)      (JSVAL_TAG(v) == JSVAL_OBJECT)

Now, what happens if we call JS_TAG on our null value?

Value:           00000000000000000000000000000000
Bitmask:         00000000000000000000000000000111
Value & bitmask: 00000000000000000000000000000000

No surprises there: this means that the resulting calculated tag is just 0x0.

This is where we have the problem! We then compare null’s calculated 0x0 tag with JSVAL_OBJECT, 0x0. They match!

This means our JS_TypeOfValue call then checks if its a function, and when it isn’t, it assigns type to be JSTYPE_OBJECT and returns type. JSTYPE_OBJECT is the string 'object', and we get typeof null === 'object'.

As mentioned in the 2ality blog, interestingly enough there is a function (in this version of the code, at least) that checks for a null value:

#define JSVAL_IS_NULL(v)        ((v) == JSVAL_NULL)

Maybe this was added in after the fact? Maybe this was in there and overlooked? Regardless, this bug was in the original, and it remains to this day.

“In general, typeof seems like a mess that will be hard to reform sensibly. “

— Brendan Eich 2006/03/31 15:13

Thanks to Brendan Eich for writing JavaScript, Mozilla for hosting the classic code, and Dr. Axel Rauschmayer for the very helpful blog post about this topic.

Investigating JavaScript Casting Behavior

Sun, 18 Mar 2018 11:14:19 -0400

The disconnect between “what seems like it should happen” and “what happens” causes you to push doors that are to be pulled, tap ads that load in the place of content, and cast null into 0 by accident in JavaScript.

For more on JavaScript’s unintuitive behavior, please watch (or rewatch) wat.

Much of this behavior can be explained by JavaScript’s casting. From ECMAScipt: “The ECMAScript language implicitly performs automatic type conversion as needed.” Since knowing is half the battle, I’ve found it helpful to know of the unexpected behavior so that it’s in the back of my mind when writing new code.

I’ve worked with and am reasonably comfortable with JavaScript, but occasionally unintuitive behavior will still show up unannounced. For instance, I recently ran into the following unexpected code:

> 0 == null
false
> 0 === null
false
> 0 < null
false
> 0 <= null
true

At first glance, not knowing the exact casting behavior, this seems to make no sense at all (especially without an idea of what’s going on under the hood). If 0 isn’t equal to null, either with equality or strict equality, and 0 is not less than null, why would 0 <= null be true?

Diving into the ECMAScript docs tells us more:

RelationalExpression:RelationalExpression<=ShiftExpression

1) Let lref be the result of evaluating RelationalExpression.
2) Let lval be ? GetValue(lref).
3) Let rref be the result of evaluating ShiftExpression.
4) Let rval be ? GetValue(rref).
5) Let r be the result of performing Abstract Relational Comparison rval < lval with LeftFirst equal to false.
6) ReturnIfAbrupt(r).
7) If r is true or undefined, return false. Otherwise, return true.

So this evaluates the left side, gets the value, evaluates the right side, gets the value, and performs Abstract Relational Comparison rval < lval with LeftFirst equal to false.

Note how the right and left values are swapped in the Abstract Relational Comparison. This uses the idea that a <= b is the same as !(b < a). If b is not less than a, it must be greater than or equal to a, or flipped, a must be less than or equal to b.

Let’s take a look at the Abstract Relational Comparison x < y:

Abstract Relational Comparison x < y

1. If the LeftFirst flag is true, then
    a. Let px be ? ToPrimitive(x, hint Number).
    b. Let py be ? ToPrimitive(y, hint Number).

2. Else the order of evaluation needs to be reversed to preserve left to right evaluation,
    a. Let py be ? ToPrimitive(y, hint Number).
    b. Let px be ? ToPrimitive(x, hint Number).

3. If both px and py are Strings, then
    a. If py is a prefix of px, return false. (A String value p is a prefix of String value q if q can be the result of concatenating p and some other String r. Note that any String is a prefix of itself, because r may be the empty String.)
    b. If px is a prefix of py, return true.
    c. Let k be the smallest nonnegative integer such that the code unit at index k within px is different from the code unit at index k within py. (There must be such a k, for neither String is a prefix of the other.)
    d. Let m be the integer that is the code unit value at index k within px.
    e. Let n be the integer that is the code unit value at index k within py.
    f. If m < n, return true. Otherwise, return false.

4. Else,
    a. Let nx be ? ToNumber(px). Because px and py are primitive values evaluation order is not important.
    b. Let ny be ? ToNumber(py).
    c. If nx is NaN, return undefined.
    d. If ny is NaN, return undefined.
    e. If nx and ny are the same Number value, return false.
    f. If nx is +0 and ny is -0, return false.
    g. If nx is -0 and ny is +0, return false.
    h. If nx is +∞, return false.
    i. If ny is +∞, return true.
    j. If ny is -∞, return false.
    k. If nx is -∞, return true.
    l. If the mathematical value of nx is less than the mathematical value of ny —note that these mathematical values are both finite and not both zero—return true. Otherwise, return false.

For our case comparing 0 <= null, the inputs are evaluated in the second step (as LeftFirst was passed in as false) with ToPrimitive with a hint of Number. Since they aren’t both strings, step 3 is skipped, and and later ToNumber is called on them in step 4. This seems promising!

Let’s take a look at ToPrimitive to see what’s happening here:

ToPrimitive ( input [ , PreferredType ] )

1. Assert: input is an ECMAScript language value.
2. If Type(input) is Object, then

    a. If PreferredType was not passed, let hint be "default".
    b. Else if PreferredType is hint String, let hint be "string".
    c. Else PreferredType is hint Number, let hint be "number".
    d. Let exoticToPrim be ? GetMethod(input, @@toPrimitive).
    e. If exoticToPrim is not undefined, then
        i. Let result be ? Call(exoticToPrim, input, « hint »).
        ii. If Type(result) is not Object, return result.
        iii. Throw a TypeError exception.

    f. If hint is "default", set hint to "number".
    g. Return ? OrdinaryToPrimitive(input, hint).

3. Return input.

I expected the Number hint to be the smoking gun after seeing it before (and referenced on StackOverflow), but I don’t think that’s what’s causing this behavior. When null is passed to ToPrimitive, the first step correctly asserts that it’s a language value. The second step conditional, however does not evaluate as true, as null is the Null type, not the Object type. (TODO: future blog post on why typeof(null) evaluates to 'object'.)

So it actually looks like ToPrimitive immediate goes to step 3, where it just returns null. This means we can look back at the 4th step in Abstract Relational Comparison. The next call is Let nx be ? ToNumber(px). This is where the magic is happening. ToNumber does what you’d (mostly) expect and returns +0 for an argument type of Null, which is what we’ve passed in with null.

This is finally where we’re comparing 0 and +0. +0 is not less than 0, so our call to Abstract Relational Comparison returns false, and our original <= RelationalExpression returns true, evaluating 0 <= null as true.

Returning to our original troublesome code, we can connect why this leads to our 0 <= null being true. The only things left to look at are the equality comparisons.

Abstract Equality Comparison

The comparison x == y, where x and y are values, produces true or false. Such a comparison is performed as follows:

1. If Type(x) is the same as Type(y), then
    a. Return the result of performing Strict Equality Comparison x === y.
2. If x is null and y is undefined, return true.
3. If x is undefined and y is null, return true.
4. If Type(x) is Number and Type(y) is String, return the result of the comparison x == ToNumber(y).
5. If Type(x) is String and Type(y) is Number, return the result of the comparison ToNumber(x) == y.
6. If Type(x) is Boolean, return the result of the comparison ToNumber(x) == y.
7. If Type(y) is Boolean, return the result of the comparison x == ToNumber(y).
8. If Type(x) is either String, Number, or Symbol and Type(y) is Object, return the result of the comparison x == ToPrimitive(y).
9. If Type(x) is Object and Type(y) is either String, Number, or Symbol, return the result of the comparison ToPrimitive(x) == y.
10. Return false.

Strict Equality Comparison

The comparison x === y, where x and y are values, produces true or false. Such a comparison is performed as follows:

1. If Type(x) is different from Type(y), return false.
2. If Type(x) is Number, then
    a. If x is NaN, return false.
    b. If y is NaN, return false.
    c. If x is the same Number value as y, return true.
    d. If x is +0 and y is -0, return true.
    e. If x is -0 and y is +0, return true.
    f. Return false.
3. Return SameValueNonNumber(x, y).

For the Abstract Equality, we can see that there is no ToNumber call that would make null +0, and we can also see that the Strict Equality immediately returns false when Type(0), Number, is different from Type(null), Null.

> 0 == null
false // No casting in Abstract Equality
> 0 === null
false // No casting in Strict Equality
> 0 < null
false // Similar casting as above, and 0 < +0 is false
> 0 <= null
true // Main casting question, and 0 <= +0

Definitely tricky, but good to keep in the back of your mind!

Investigating a Bug in Moment.js

Sat, 06 Jan 2018 11:37:37 -0500

Investigating a bug in Moment.js

In this post, I walk through the steps I took while investigating a bug in Moment.js, an open source JavaScript library for dealing with dates and times.

Rather than write about what would have ideally happened (the steps of which would look like 1) see bug, 2) know issue, 3) investigate fix, 4) open PR), I wanted to include the entire process, which included some roundabout work. When reading about someone else’s work, I’ve found that it’s very valuable to see both what worked and what didn’t work.

One of my recent goals has been to contribute more to open source. OSS makes my life easier, and the least I can do is to give back with small bug fixes or documentation updates.

I’ve used Moment.js a number of times, both for side projects and for my day job. Recently, I’ve been occasionally visiting the issues page to see if there was anything I could help out with. Since I work with JavaScript a lot, I figured something like this would be the most approachable for me. I also wanted to learn about how it works internally, and I knew that this would be a great chance to dig through the code and learn more.

The issue

The other day, I came across this issue marked as a bug:

Hello,
I'm reviewing tutorial
https://momentjs.com/docs/#/i18n/locale-data/

I got an issue for the following:
localeData = moment.localeData();
localeData.longDateFormat();
Uncaught TypeError: Cannot read property 'toUpperCase' of undefined

Also recently found.
localeData.relativeTime();
Uncaught TypeError: Cannot read property 'replace' of undefined

localeData.week();
Uncaught TypeError: Cannot read property 'year' of undefined

Please resolve this issue.
Thank you.

This looked like something I could investigate! I wasn’t sure if I could completely fix it or if it was a good first bug, but I decided to see what I could do with it.

I started by reproducing the issue on a jsfiddle:

If you click on Result and check your console, you’ll see the TypeError. This was good to confirm that the error existed. I next confirmed the error appeared for relativeTime() and for week(). Both of those appeared as well, so I was confident the error was real. I also checked the linked Moment docs and confirmed that the method signatures at that link were indeed localeData.longDateFormat(), localeData.relativeTime(), and localeData.week().

(At this point, how would you debug further? I missed a few things here that seem obvious in retrospect and went on a longer route, and I’ll review the different paths towards the end.)

I then forked and cloned the Moment repository so that I could have a local copy. This let me dig around in the code and search for specific things.

As a quick check, I searched for a few things using grep to see how often they came up. localeData, .replace, and .year appeared often, but toUpperCase() appeared only twice:

moment: $ grep -r toUpperCase src
src/lib/locale/formats.js:        formatUpper = this._longDateFormat[key.toUpperCase()];
src/test/moment/normalize_units.js:        fullKeyCaps = fullKey.toUpperCase();

Two results is much easier to look through! Since I was already on the lookout for lognDateFormat, I took a look at src/lib/locale/formats.js and saw the following function:

/*
  src/lib/locale/formats.js
*/
...
export function longDateFormat (key) {
    var format = this._longDateFormat[key],
        formatUpper = this._longDateFormat[key.toUpperCase()];

    if (format || !formatUpper) {
        return format;
    }

    this._longDateFormat[key] = formatUpper.replace(/MMMM|MM|DD|dddd/g, function (val) {
        return val.slice(1);
    });

    return this._longDateFormat[key];
}
...

Since the TypeError said Cannot read property 'toUpperCase' of undefined, my guess was that longDateFormat was getting an undefined key passed to it, which would throw the error when formatUpper is being assigned. This looked promising! I wrote it down to later investigate who was calling longDateFormat with what argument as I went through the rest of the code.

I started to really dig in by opening src/moment.js, the main file for the project. Taking a look at what it imports, I saw the following:

/*
  src/moment.js
*/
...
import {
    defineLocale,
    updateLocale,
    getSetGlobalLocale as locale,
    getLocale          as localeData,
    listLocales        as locales,
    listMonths         as months,
    listMonthsShort    as monthsShort,
    listWeekdays       as weekdays,
    listWeekdaysMin    as weekdaysMin,
    listWeekdaysShort  as weekdaysShort
} from './lib/locale/locale';
...

This let me know that localeData comes from getLocale, which lives in src/lib/locale/locale.js. Warmer!

I also noticed that there was another locale directory with the various configurations for different locales. The first one, af.js, is the Afrikaans configuration. The files in this directory were all similarly named, so the localeData code probably wouldn’t live in there.

Here’s part of src/lib/locale/locale.js:

/*
  src/lib/locale/locale.js
*/
...
import {
    getSetGlobalLocale,
    defineLocale,
    updateLocale,
    getLocale,
    listLocales
} from './locales';
...

src/lib/locale/locale.js didn’t have much in it, but getLocale, which src/moment.js imports as our friend localeData, is imported from src/lib/locale/locales.js. Note the “s” on the end! Definitely made me double check what file I had open a few times.

Inside src/lib/locale/locales.js, I found the getLocale function I had been looking for:

/*
  src/lib/locale.locales.js
*/
...
// returns locale data
export function getLocale (key) {
    var locale;

    if (key && key._locale && key._locale._abbr) {
        key = key._locale._abbr;
    }

    if (!key) {
        return globalLocale;
    }

    if (!isArray(key)) {
        //short-circuit everything else
        locale = loadLocale(key);
        if (locale) {
            return locale;
        }
        key = [key];
    }

    return chooseLocale(key);
}
...

At first glance, I didn’t see anything particularly helpful in there. I knew that when I had been calling moment.localeData(), I hadn’t been passing in any argument, so the key for getLocale would be undefined. That meant it would return globalLocale about halfway through the function. Not super helpful, but let’s take a look at globalLocale. I did a quick search, and the only reference not in another function was at the top:

/*
  src/lib/locale.locales.js
*/
...
var globalLocale;
...

Hmm, not very helpful. At this point, I was thinking that I must be missing something. moment.localeData() was clearly not returning undefined, as I was able to call localeData.week() and others, although not successfully.

My next step was to take a break from digging and write a failing test to confirm that this error was happening locally. I took a look in the tests, and I found src/test/moment/locale.js, which had a few tests for localeData:

/*
  src/test/moment/locale.js
*/
...
test('library localeData', function (assert) {
    moment.locale('en');

    var jan = moment([2000, 0]);

    assert.equal(moment.localeData().months(jan), 'January', 'no arguments returns global');
    assert.equal(moment.localeData('zh-cn').months(jan), '一月', 'a string returns the locale based on key');
    assert.equal(moment.localeData(moment().locale('es')).months(jan), 'enero', 'if you pass in a moment it uses the moment\'s locale');
});
...

Since I figured it would throw an error anyways, I wasn’t too worried about having a perfect test written. I added assert.equal(moment.localeData().week(), 1, 'Quick week test') and ran the tests:

Errors:

Module: locale Test: library localeData
Died on test #1     at global.test.QUnit.test (/path/to/moment/node_modules/qunit/lib/child.js:146:21)
    at /path/to/moment/build/umd/test/moment/locale.js:239:1
    at i (/path/to/moment/build/umd/test/moment/locale.js:4:43)
    at Object.<anonymous> (/path/to/moment/build/umd/test/moment/locale.js:7:2)
    at Module._compile (module.js:556:32)
    at Object.Module._extensions..js (module.js:565:10)
    at Module.load (module.js:473:32)
    at tryModuleLoad (module.js:432:12): Cannot read property 'year' of undefined
TypeError: Cannot read property 'year' of undefined
    at weekOfYear (/path/to/moment/build/umd/moment.js:1204:41)
    at Locale.localeWeek [as week] (/path/to/moment/build/umd/moment.js:1262:12)
    at Object.<anonymous> (/path/to/moment/build/umd/test/moment/locale.js:244:38)

Global summary:
┌───────┬───────┬────────────┬────────┬────────┬─────────┐
│ Files │ Tests │ Assertions │ Failed │ Passed │ Runtime │
├───────┼───────┼────────────┼────────┼────────┼─────────┤
│ 1     │ 3262  │ 130247     │ 1      │ 130246 │ 25097   │
└───────┴───────┴────────────┴────────┴────────┴─────────┘
Warning: 1 tests failed Use --force to continue.

Nice! So this was happening locally too. I took a look at build/umd/moment.js at lines 1204 and 1262:

/*
  build/umd/moment.js
*/
...
function weekOfYear(mom, dow, doy) { // Line 1204
    var weekOffset = firstWeekOffset(mom.year(), dow, doy),
        week = Math.floor((mom.dayOfYear() - weekOffset - 1) / 7) + 1,
        resWeek, resYear;

    if (week < 1) {
        resYear = mom.year() - 1;
        resWeek = week + weeksInYear(resYear, dow, doy);
    } else if (week > weeksInYear(mom.year(), dow, doy)) {
        resWeek = week - weeksInYear(mom.year(), dow, doy);
        resYear = mom.year() + 1;
    } else {
        resYear = mom.year();
        resWeek = week;
    }

    return {
        week: resWeek,
        year: resYear
    };
}
...
function localeWeek (mom) { // Line 1262
    return weekOfYear(mom, this._week.dow, this._week.doy).week;
}
...

Interesting! Not super helpful to view it in the built code, so I grepped localeWeek and found these results (along with others):

moment: $ grep localeWeek -r src
src/lib/locale/prototype.js:import { localeWeek, localeFirstDayOfYear, localeFirstDayOfWeek } from '../units/week';
src/lib/locale/prototype.js:proto.week = localeWeek;

Getting warmer! I opened up src/lib/locale/prototype.js.

/*
  src/lib/locale/prototype.js
*/
import { Locale } from './constructor';

var proto = Locale.prototype;
...
// Week
import { localeWeek, localeFirstDayOfYear, localeFirstDayOfWeek } from '../units/week';
proto.week = localeWeek;
proto.firstDayOfYear = localeFirstDayOfYear;
proto.firstDayOfWeek = localeFirstDayOfWeek;
...

I can see here that prototype is importing a function named localeWeek from src/lib/units/week.js and assigning it to the week key. This is where the structure started to make more sense. prototype.js defined the methods available to the Locale constructor, so that when the Locale is created, it has access to everything defined here. The default locale must create a Locale object that we then can call .week() from. That makes more sense!

The question remains why the .week() function isn’t firing right. I opened up src/lib/units/week.js and found the localeWeek function in question:

/*
  src/lib/units/week.js
*/
...
// LOCALES

export function localeWeek (mom) {
    return weekOfYear(mom, this._week.dow, this._week.doy).week;
}
...

Alright, we’ve seen this one before! So this is the function being given to the prototype, which is accessible to the localeData as .week(). The only problem is that localeWeek is called with a mom argument, and it passes it to weekOfYear, which appears in src/lib/units/week-calendar-utils.js :.

/*
  src/lib/units/week-calendar-utils.js
*/
...
export function weekOfYear(mom, dow, doy) {
    var weekOffset = firstWeekOffset(mom.year(), dow, doy),
        week = Math.floor((mom.dayOfYear() - weekOffset - 1) / 7) + 1,
        resWeek, resYear;

    if (week < 1) {
        resYear = mom.year() - 1;
        resWeek = week + weeksInYear(resYear, dow, doy);
    } else if (week > weeksInYear(mom.year(), dow, doy)) {
        resWeek = week - weeksInYear(mom.year(), dow, doy);
        resYear = mom.year() + 1;
    } else {
        resYear = mom.year();
        resWeek = week;
    }

    return {
        week: resWeek,
        year: resYear
    };
}
...

So here, we see that the mom being passed in is used in the first line, where it is accessed with mom.year(). This means that when we call localeData.week() without an argument, week’s mom parameter is undefined, so that when it passes it to weekOfYear, weekOfYear calls mom.year(), which is really (undefined).year(), which causes the Uncaught TypeError: Cannot read property 'year' of undefined. This makes sense! We’re missing the argument in week().

It took me longer than it should have to realize that mom meant a Moment object. I confirmed this in the jsfiddle:

Clicking on result should successfully alert 1, which makes sense. It’s current January 6 for me, and in the default locale, en, this is the first week. This will change accordingly if you’re reading this in a later week of the year.

I investigated longDateFormat() and relativeTime() in a similar way (left as an exercise for the reader), and I found that longDateFormat should have a date format string passed to it (like longDateFormat(String)), and relativeTime should be called with a number, a withoutSuffix boolean, a key string, and an isFuture boolean (like relativeTime(Number, Boolean, String, Boolean)).

Armed with this new knowledge, I returned to the GitHub issue and wrote up a response detailing that it was an error in the docs, not an error in the code, and that I would be more than happy to update the docs. Right before I hit submit, I checked the docs one more time. To my dismay, I noticed the following information a couple of mouse scrolls below the method signatures:

localeData.longDateFormat(dateFormat);  // returns the full format of abbreviated date-time formats LT, L, LL and so on
localeData.relativeTime(number, withoutSuffix, key, isFuture);  // returns relative time string, key is on of 's', 'm', 'mm', 'h', 'hh', 'd', 'dd', 'M', 'MM', 'y', 'yy'. Single letter when number is 1.
localeData.week(aMoment);  // returns week-of-year of aMoment

In these three lines we have the info I just found through investigating the code. This was one of the steps I mised earlier! I’m not sure how I missed this the first time around, but the answer was right there. This did confirm what I found though, and the fact that it didn’t match the above method signatures was a sign that the docs were indeed lacking. If this new user could make this mistake, followed by me making the mistake, then something could be improved.

I ended up replying to the issue, opening an issue in the Moment.js docs repository, and submitting a PR with updated method signatures. Since I knew where to look for these functions, I was able to confirm my understanding of any unclear types in the code.

Postmortem

Digging through the code did give me a better feel for how the library works, and it was great to practice contributing to a project. While looking through for localeData, I was able to get a feel for how the library is put together. I’ll also be better prepared for future issues in this library, as I’ll already have my basic mental model for the library and can write tests and investigate earlier.

I did miss the docs, and I also realized that I could have followed the TypeError console traceback after recreating the issue in a jsfiddle. This would have led me directly to longDateFormat attempting to call toUpperCase() on key. I ended up getting there anyways, but it did take me a bit longer. Debugging takes practice, and I’ll have to keep these in mind the next time I’m digging through an issue.

Moving Forward

I didn’t end up submitting a PR for the code itself, but I was able to contribute to the documentation and learn more about the library. Throughout the search, I had that uncomfortable feeling of “I don’t know what’s going on in this library”, especially when I switched from investigating localeData to creating the temporary test. I know that the uncomfortable feeling is a signal that I’m learning, and although it isn’t easy, embracing that feeling lets me know that I’m pushing my comfort zone in the right direction.

You can check out the Moment.js code here and read through the docs here.

Update April 4, 2018

My PR got merged today, and the docs should be updated with their next release. Great!

Connecting Arduino with a Node Server

Mon, 01 Jan 2018 09:59:52 -0500

Arduinode

I’ve been working through the Arduino Starter Kit projects recently, and I wanted to try a new project that was outside of the pre-made booklet. Working with an offline Arduino (intranet of things?) has let me learn a great deal, and I wanted to see how difficult it would be to connect my Arduino to a simple Node server, which would get me ready to connect it in any future projects.

Since I’m most comfortable with JavaScript, I decided to create a simple connection between the Arduino and a Node server. Since a classic Hello, World Arduino project is to light up an LED, I decided I would make a very basic program that would send a button press from the Arduino to the server, which would then send an action response back to the Arduino to turn an LED on or off.

Here’s what the schematic looks like:

After a little research, I came across the serialport package, which would allow a simple connection between the server and the Arduino. Easy enough!

Hello, Arduino

My first test was to get an LED shining from a message from the Node server. I decided to keep it very simple and only send 1 byte at a time, either a 1 or a 0. For now, the server would send a 1, and the Arduino would turn the LED on. I started with very basic Arduino code:

/*
  project.ino
*/
int LEDPin = 6; // LED connected to pin 6
int incomingByte; // for incoming serial data

void setup() {
  // Initialize the LED pin to be an output
  pinMode(LEDPin, OUTPUT);
  // Give the pin low voltage, which turns the LED off
  digitalWrite(LEDPin, LOW);
  // Initialize the Serial connection
  Serial.begin(9600);
}

void loop() {
  // Check if we have a message available
  if (Serial.available() > 0) {
    // read the incoming byte:
    incomingByte = Serial.read();

    // If we have an incoming byte of 1, turn light on.
    if (incomingByte == 1) {
      digitalWrite(LEDPin, HIGH);
    }
  }
}

Not too bad! My server code was similarly simple. One thing to note is that there’s a slight delay between opening the Serial port and being able to send a message. Apparently when the port is opened, the Arduino code restarts, leading to a slight delay. I got around this by adding in a 2 second delay before the server sends the 1.

/*
  server.js
*/
// Import SerialPort
var SerialPort = require('serialport');

// Create a new connection with the correct path to the Arduino and a baudRate that matches
var port = new SerialPort('/dev/cu.usbmodemFD131', { baudRate: 9600 }, function(err) {
  if (err) {
    console.log('Error: ', err.message);
  }
});


// When the connection is open, send the Arduino a 1
port.on('open', function() {
  setTimeout(function() {
    port.write([1]);
  }, 2000);
});

After connecting and uploading the Arduino code, I started the server and the LED turned on!

Hello, Server

The next step was to get the Arduino talking to the server. Since I had the server to Arduino code already set, I added an event listener to the server that would listen for new data on the Serial port. I kept it simple, sending a 1 if it heard a 1 and a 0 if it heard a 0.

/*
  server.js
*/
var SerialPort = require('serialport');

var port = new SerialPort('/dev/cu.usbmodemFD131', { baudRate: 9600 }, function(err) {
  if (err) {
    console.log('Error: ', err.message);
  }
});


port.on('open', function() {
  // Add event listener for data
  port.on('data', function(data) {
    // Log the incoming data so we can see it
    console.log(data[0]);
    // If we get a 1, send a 1 back
    if (data[0]) {
      port.write([1]);
    } else {
      // Otherwise, send a 0
      port.write([0]);
    }
  });
});

Now, I needed to make the Arduino listen for a button press change. I kept track of the state of the switch, whether it’s open or closed, and the previous state. This allowed me to only send a message to the server when something new happened, instead of constantly sending messages. Essentially, pressing the switch down would send one message to the server, instead of many messages while the switch is open and many messages once the button was pressed down. Given how often the Arduino runs its loop, I wanted to keep the flow simple.

/*
  project.ino
*/
int LEDPin = 6;
int incomingByte; // for incoming serial data

// Initialize variables to show that button starts not pressed
int switchState = 0;
int previousSwitchState = 0;

void setup() {
  pinMode(LEDPin, OUTPUT);
  // Initialize the button pin to input
  pinMode(2, INPUT);
  digitalWrite(LEDPin, LOW);

  Serial.begin(9600);
}

void loop() {
  // Read the voltage coming in from the button
  switchState = digitalRead(2);

  if (switchState == HIGH && switchState != previousSwitchState) {
    // If the switchState is pressed and it didn't used to be, send message to server
    Serial.write(1);
    previousSwitchState = switchState;
  } else if (switchState == LOW && switchState != previousSwitchState) {
    Serial.write(0);
    previousSwitchState = switchState;
  }

  // Check if we have a message available
  if (Serial.available() > 0) {
    // read the incoming byte:
    incomingByte = Serial.read();

    if (incomingByte == 1) {
      digitalWrite(6, HIGH);
    } else {
      digitalWrite(6, LOW);
    }
  }
}

Now, inside the main loop, the Arduino will check if the button is pressed and if it’s different than the last time it checked. If it is different, it will send either a 1 or a 0 to the server. The server, seeing that 1 or 0, will respond appropriately, and the Arduino will then turn the LED on or off.

This worked! I then realized that it could be working because the Arduino is reading the byte that it just sent, which would mean that it was really bouncing these messages off of a mirror to turn the light on. As a sanity check, I stopped the node server, at which point the light stopped turning on or off. This left me confident that the server was in fact playing its role correctly.

All that was left was some refactoring to clean up the code:

/*
  server.js
*/
/*
  This is the very simple server code for listening to and responding to the arduino's button press.
  It listens for data from the arduino, and if it receives an ON signal, it returns an ON signal. Likewise for OFF.
*/
var SerialPort = require('serialport');
var PATH = '/dev/cu.usbmodemFD131';

var port = new SerialPort(PATH, { baudRate: 9600 }, function(err) {
  if (err) {
    console.log('Error: ', err.message);
  }
});

port.on('open', function() {
  port.on('data', function(data) {
    console.log(data[0]);
    if (data[0]) {
      port.write([1]);
    } else {
      port.write([0]);
    }
  });
});

/*
  project.ino
*/
/*
Setup:
  Initialize I/O pins
  Initialize Serial port
Loop:
  Check for button press change and send to server with Serial
  Check for incoming message and turn LED on/off accordingly
*/

int LEDPin = 6;
int incomingByte; // for incoming serial data
int switchPin = 2;
int switchState = LOW;
int previousSwitchState = LOW;

void setup() {
  pinMode(LEDPin, OUTPUT);
  pinMode(switchPin, INPUT);

  // Start with LED turned off
  digitalWrite(LEDPin, LOW);

  Serial.begin(9600);
}

void loop() {
  // Read button press
  switchState = digitalRead(switchPin);

  // Check previousSwitchState so we aren't constantly writing back and forth without a button change
  if (switchState != previousSwitchState) {
    Serial.write(switchState);
    previousSwitchState = switchState;
  }

  // Check if we have a message available
  if (Serial.available() > 0) {
    // read the incoming byte:
    incomingByte = Serial.read();
    digitalWrite(LEDPin, incomingByte);
  }
}

Next steps

This project served as a good way to test out the communication between Arduino and computer, and it was an easy way to get something up and running with serialport. With this project done, it’ll be very easy to integrate the Arduino into any future Node project.

You can see the full code on my GitHub page here.

MBTA Bus Mirrors

Wed, 06 Dec 2017 08:51:05 -0500

I take the MBTA 86 bus as part of my commute in the morning. It runs from Reservoir in Brighton to Sullivan Station in Cambridge, and it stops at Harvard on the way, which is where I get off.

Right after the Eliot street stop, there are two ways for the bus to get underground to the Harvard stop. One way is to go down Bennett St., take a right on University Road, take a right on Mt. Auburn, and then take a left into the bus tunnel. The other way is to go through Bennett Alley and continue straight into the bus tunnel. This way is shorter, and it’s the preferred route of every driver I’ve ridden with.

Bennett Alley is for MBTA traffic only, and the the bus driver usually decides on the route to take based on how many buses are already in Bennett Alley. Often times, there will be one bus parked to the left and one parked to the right, and the bus driver is able to drive through the middle. If there are already three buses in the alley or one of the two is parked at an angle reducing the space left for our bus, the driver will instead take the longer route around.

If I had to guess how wide Bennett Alley is, I would say it’s just larger than 3 busses wide. When our bus goes between two others, there’s often inches to spare. One of the reasons our bus is able to make it through is because of the mirrors.

The left and right mirrors on every MBTA bus are vertically offset to allow two parallel buses to overlap their mirror space without hitting each other.

If the two buses had mirrors at the same height, and additional ~8 inches would be required on either side. This would increase the width required for three buses by ~16 inches. This doesn’t sound like much, but even a foot is often the difference between being able to drive through Bennett Alley and taking the slightly longer way around.

It’s a small design decision that saves just a bit of space. It’s not helpful in the large majority of situations, but given Boston’s often tight streets, it gives just a little more breathing room for the drivers.

Temperature Sensing with Arduino

Sun, 13 Aug 2017 15:16:13 -0400

I worked through the third basic project in the Arduino Starter Kit Projects Book. I like to write about the projects to reinforce my own understanding.

The third project, the “Love-O-Meter”, creates a simple circuit with a temperature sensor. Using the Analog-to-Digital converter, the Arduino reads in the value coming from the temperature sensor, then lights up a combination of 3 LEDs depending on that value.

Here’s what my circuit looked like:

You can see the temperature sensor in the top right connected to the positive, the analog A0 pin, and the ground. 3 LEDs are each connected to one of the digital ports on the right side of the arduino.

To begin, two constant variables were created for the sensor pin and the baseline temperature. In the setup function, output pins 2, 3, and 4 were put in output mode and were sent low voltage (keeping the LEDs off). The serial port was also initialized to send logging information back to the computer. 9600 refers to sending the computer 9600 bits per second.

const int sensorPin = A0;
const float baselineTemp = 25.0;

void setup() {
  Serial.begin(9600);
  for (int pinNumber = 2; pinNumber < 5; pinNumber++) {
    pinMode(pinNumber, OUTPUT);
    digitalWrite(pinNumber, LOW);
  }
}

The middle wire of the temperature sensor was connected to the A0 analog in-pin. Inside the main loop, the Arduino receives receives a value between 0-1023 from the sensor through this pin.

const int sensorPin = A0;
int sensorVal = analogRead(sensorPin);

Because the total voltage available to the sensor is 5 volts, the value between 0-1023 can be mapped to a voltage between 0 and 5 volts.

float voltage = (sensorVal / 1024.0) * 5.0;

The voltage output can then be used to convert to temperature. The data sheet for the TMP36 temperature sensor has information about the conversion, and the relevant table is below:

This means that there is a 0.5 V offset for the readings, and 10 mV (or 0.01 V) corresponds to a 1°C change. We can confirm this with the output voltage at 25°C. With an output of 750 mV, we can subtract the offset voltage (0.5 V, or 500 mV), which then gives us 250 mV. Dividing by 10 mV/°C gives us the correct value of 25°C.

We could also do this conversion in volts (instead of millivolts) by subtracting 0.5 V and dividing by 0.01 V/°C, which is the same as multiplying by 100. In other words:

float temperature = (voltage - 0.5) * 100;

We can send these variables to the computer for logging using Serial.print:

Serial.print("Sensor Value: ");
Serial.print(sensorVal);
Serial.print(", Volts: ");
Serial.print(voltage);
Serial.print(", degrees C: ");
Serial.println(temperature);

The function can then turn on a combination of the LEDs based on the difference between the temperature reading and the baseline temperature. By assigning the output voltage to be HIGH, the Arduino sends the high value of voltage (5 V) out of the digital output pin, which then flows through and shines the corresponding LED.

  if (temperature < baselineTemp) {
    digitalWrite(2, LOW);
    digitalWrite(3, LOW);
    digitalWrite(4, LOW);
  } else if (temperature >= baselineTemp && temperature < baselineTemp + 2) {
    digitalWrite(2, HIGH);
    digitalWrite(3, LOW);
    digitalWrite(4, LOW);
  } else if (temperature >= baselineTemp + 2 && temperature < baselineTemp + 4) {
    digitalWrite(2, HIGH);
    digitalWrite(3, HIGH);
    digitalWrite(4, LOW);
  } else if (temperature >= baselineTemp + 4) {
    digitalWrite(2, HIGH);
    digitalWrite(3, HIGH);
    digitalWrite(4, HIGH);
  }

At the end of the loop, a very slight delay is included to create a pause in between the analog-to-digital readings to ensure that the Analog-to-Digital Converter (ADC) can reset successfully before the next reading.

  delay(10);

Finally, the result:

You can see the full code here.

Noticing UI: Shazam's loading screen

Sat, 05 Aug 2017 21:29:38 -0400

Shazam’s loading screen

I was pleasantly surprised by Shazam’s iPhone app loading screen, and I took a look into what makes it stand out.

Here’s what it looked like:

This felt different, and it made me look at the experience of opening apps. I started by spending a bus ride home from work opening and closing apps to get a feel for why this wasn’t like other apps.

Loading…

A loading screen tells a user that the app or other result is not ready yet. This is good. The user can’t try to interact with anything before it’s ready. There are no non-working buttons or menus to click and get frustrated with. It means that there is a pause in between asking for the app and being able to use it, which means a delay in the user’s experience.

Companies often use this in-between view to show the company’s logo (just in case the user forgot what app they opened). Some companies use this as a chance to help the user connect the company to their logo (just in case the logo has no words or markings about which company it is). I saw this for Uber, Audible, and YNAB.

You can open five different apps from your phone/tablet and see how they chose to handle the in between time. My experience was that most of them were a simple separate view featuring the company’s logo and name.

“A++ WOULD OPEN AGAIN”

One thing that I enjoyed about Shazam’s take is that the overall structure between the loading page and the ready app is the same. There are no jumps to different views or new screens. Instead, the app feels like it comes alive under your thumb. The faded S logo lights up with the animated S and transforms into a button. The shifted gradient makes the circle pop more like the optical illusion of background color changing an object’s color, and the changing circle size gives further depth.

Part of the reason this works best for Shazam is that pressing the button has been designed as the main functionality of the app. The button allows Shazam to listen to and find a match for your audio, and pressing that button is likely the user’s goal when opening the app. Not only has the app loaded when the button comes alive, but you’re now able to go directly to what you’re looking for. The My Shazam and Discover sections are secondary and accessible after the button has been presented. I like this. The extra clicks add up.

This doesn’t deviate from normal app behavior. It’s still showing the logo while loading, and it’s still reinforcing the connection between the company and the slanted S (or is that a slanted “u” and an “n” on top of each other? No, I opened Shazam, probably an S). The additional transition to the button, an extension of the logo, allows the user to pass easily from the usual loading view to the ready app.

This connection would not be as clear for a different app. For a notetaking app, the importance of reviewing notes may match starting a new note. If you tried to connect the app’s logo loading screen with one button, which would you pick? For something like Facebook, how would you connect the loading screen to the timeline?

One similar example that I noticed was Instagram’s app, which loads with the header (with logo), a blank viewport for the content, and the footer area (without menu options). This also builds continuity between the structure of the loading screen and the structure of the ready app.

Keep It Simple Shazam

Shazam is able to build a simple loading user experience with a smooth animation that connects their logo to the app’s purpose. It’s not for every case, but for the right case, it’s a good move.

Caffeinspiration

Timeline and Early Days

Timeline and discovery: the early days

On having heart surgery at 32

Cathy

Timeline and Early Days

Cathy: my dear, helpful catheter

How much oranger do red orange bags make oranges look?

1

2

3

4

5

6

7

8

9

10

11

Anyways, here’s how I set this up.

Visualizing my strengths and areas for growth with RPG radar charts

The good part: evaluating my own strengths and areas for growth

Technical skills

Soft(er) skills

The start of the asterisks

How effective is this?

The arbitrary scale

A fair evaluation

Honesty and vulnerability

What do you even include for the axes?

Where does the radar chart fall short, and how could it be better?

Harry Chapin and RATM: the problem with bleeding interests

All of the writing I did in a week as a software engineer

A day in the life

cat_typing.gif

Zero emails

A few dozen lines of code

A handful of Jira tickets

A feedback form for the new grad on our team

A couple of investigation & debugging docs

A few technical docs

397 Slack messages

Key takeaways

Brute-forcing the NYT Digits game

Digits & difficulty

What would the computer say?

Brutal force

Recursion and representing the game as a tree

Trying on the real problem

What does the difficulty look like?

Improving this further

The dog hair and the computer mouse

Facing little problems you’ve seen before

Things that are hard to measure, things that are easy to measure

In the desert, no service, and your partner's late to the rendezvous. What now?

Practice being a beginner

Learning by working on problems just outside of your reach

Circles of knowledge: what I do and do not know

Working on problems just outside of reach

iPhones and action discoverability, or "How the hell was I supposed to know that?"

Discoverability

What iOS is doing these days

Deleting a number on the iOS calculator

Using the spacebar as textbox navigation

Switching between Safari pages by swiping the URL bar

Going back after opening a new Safari tab

Undiscoverability

The hobby buyer's fallacy: "I'd really pursue my hobby if I just bought X"

Stubbornness and programming

Writing a toy WebSocket server from scratch

WebSocket

The WebSocket connection flow

A toy websocket server

Starting with HTTP

Handling the initial handshake request

The WebSocket frames

Masking

Back to the frame

Parsing the frame

Fun with toy programs