Blogs on kieranhealy.org

Zero Sum Problems

Thu, 21 May 2026 15:50:40 -0400

Over at Daring Fireball, John Gruber makes a passing observation about the Apple Sports app:

I’ve got some gripes about certain specific aspects of Apple Sports. Like, where does one even start to explain how much is wrong with their zero-sum visualization of team stats? Has anyone ever even seen a presentation like that before? Anyone?

That “Anyone” link lands over here. Hi everyone! The team stats image is quite confusing. It’s a summary of a game between the San Antonio Spurs and the Oklahoma City Thunder. I don’t know much about basketball, but I do know a bit about data visualization and in a pleasing coincidence my former student Josh Fink is the A-VP of Basketball Data Science for the Spurs. Here is the image that John objected to:

I had to look at it for a while as well.

I just finished driving a very long way up the side of the country, so I’m kind of tired. But even allowing for that, boy, this way of representing things really is quite confusing. Not being an Apple Sports user I had to look at it for a bit to understand what was happening. But, now that it has given me a headache, I can kind of see why whoever designed this ended up in the undoubtedly bad place they did.

Before I get to why I have some sympathy for the designer, why did I find this representation of these numbers so disorienting? It’s not just just because I’ve been driving for nine hours. John is right to call the picture a “Zero Sum” representation. The design strongly suggests to the viewer that, within each row, we’re looking at each team’s share of a total. Each pair of black and blue lines seem to be vying for control of their whole row, with the longest line being the “winner” in each case.

This sort of representation would make perfect sense for a measure that really was zero sum. Take an example from a properly good sport, like rugby. There, like in basketball, to a first approximation a team either has the ball or it doesn’t.¹ But there’s no shot clock in rugby, and possession routinely gets turned over without the game stopping. So, knowing that Team A had 65% possession is not only informative, it also immediately entails that Team B had 35%. You could show that with a representation like one of the rows above.

Literally none of the measures in the Basketball data above are zero-sum in this way. Both teams could shoot 100% from the free throw line, or zero percent. But because the first three measures shown are percentages, this reinforces the zero-sum impression given by the lines. It certainly did that in my case. But then, starting with Assists, the remaining rows are just absolute numbers. When I started looking at the absolute numbers, I got confused a second time by the length of the lines. “Oh so it’s not a share, it’s the value” I thought—but no, they do correspond in terms of relative proportions to the teams share within each row. But they’re not really shares they’re just magnitudes. But they have to be shown in a fixed space and we want to make them relatively comparable somehow so … Argh.

It would be nice if there were One Weird Trick to fully fix this figure. But I’m not sure that there is. For example, at a minimum we could redraw these numbers to reflect the fact that they’re not zero-sum. Keep each measure as a row (i.e. on the y-axis) but have the lines, or columns, be side by side within each category instead of facing off. Like this:

Team Stats side by side for each measure.

This view at least lets you immediately see who “won” each measure. The viewer can just directly compare the length of the bars in each category. People are really good at doing that accurately. In that sense it’s much less confusing than the original. But there’s still a lot wrong with it. The core problem is that when we draw a graph like this, we’re usually putting the same kind of thing (e.g. countries, or religious groups, or sports teams) on the y-axis, and then seeing how different their scores are on some single measure (e.g. GDP, or number of adherents, or average points scored per game), which we put on the x-axis. Maybe we use color to break things out by some third measure as well.² In this case, I’ve just labeled the x-axis as generically as possible. “Value” covers the range of all the measures. The lowest value is 5, in Largest Lead. The highest is 88, in Free Throw %. But these numbers are not meaningfully comparable. The graph encourages us to compare across as well as within categories. But while within-category comparisons are meaningful, the between-category ones are not. There were way more Bench Points than Blocks in the game. But that is not a useful thing to know.

Team Stats side by side and ordered from absolute highest to lowest, whatever that means.

Knowing who won each measure isn’t nothing. It can be informative about how the game went, maybe especially when a team won the game but “lost” on a number of the measures. If you really wanted to lean in to that aspect, you could sort of justify the zero-sum view, and maybe look for a way to sort and order by “how much” a team “won” each category. But again, what’s the right denominator for those measures? For instance, do we care about a team’s share of all Defensive Rebounds in the game? Or do we care about the share of Defensive Rebounds a team won relative to every opportunity it had to make a Defensive Rebound? How meaningful is ordering our rows by those kinds of shares? Even worse, some measures (notably Fouls) are bad to “win”, so we’d have to do something about those.

Our fundamental problem is that we just have two cases (the teams) and fifteen different measures, or variables. Each variable, except for the three percentages, is in effect on its own scale. There’s no direct way to make comparisons across them. Sure, some of these measures are probably going to be associated with one another—e.g. Turnovers and Points Off Turnovers—but the numeric values aren’t directly comparable in general. If you know a lot about basketball you might have some informative rules of thumb about each one of these measures, or some of them in combination. But at that point the lines in this particular graph are not going to be doing any work for you; you’ll just end up looking directly at the numbers. If we had data on all these measures for every NBA game for a whole season then we could of course do much more with them, because then each measure would have a distribution across all games and across all teams.

As it is, the purpose of the “Stats” screen in Apple Sports is just to summarize information from a single game. The other thing I could think of to do with the numbers as kind of graph is something like this:

A back-to-back column chart.

This is marginally more helpful than the one before just because, again, it gets rid of the unhelpful zero-sum look of the original. As I hope you can immediately see, it creates many other difficulties. It also doesn’t do away with the core problem. That problem is principally one of information design rather than data visualization. What I mean is that what we’re trying to organize is, in effect, fifteen pairs of related but fundamentally distinct numbers. If we had fifteen cases and two variables things would be simple. But with fifteen variables and two cases … well, this is not the kind of thing you can make a single effective and non-confusing graph out of. That’s why I kind of sympathize with the designer. In a constrained space they have to show thirty numbers (thirty two, including the score). Lots of information. A straight table seems like it would be boring. Surely there’s some way to thematically integrate the numbers in a visually appealing manner that brings out some of the relationships across the rows. That’s what graphs do; it seems like the right thing to reach for. But at its heart this information is not a graph. It just sort of looks like one, and that ends up confusing people.

Modulo some measurement decisions about how to determine when possession is turned over while the ball is in play. ↩︎
Here’s an example of a graph with a categorical measure on the y-axis, a continuous measure on the x-axis, and an additional categorical feature shown with color. ↩︎

Bad Weather and the Subway

Sat, 02 May 2026 08:59:15 -0400

Snow in Inwood, New York. Photograph by the author.

Recently I’ve been looking at hourly ridership data from the New York City Subway. Last time we learned that people go to work in the morning and come home in the evening, for example. (All together now: “Only in New York, baby!”) Today we’ll learn that bad weather makes people stay at home. Except, sometimes it doesn’t.

Regular readers will recall that the subway system carries a lot of passengers every day. The ridership data for the whole of 2025 represents just over 1.3 billion entries into the system via an OMNY tap or Metrocard. It’s available aggregated to hourly resolution by station complex. With that data in hand, we can calculate average hourly ridership for every day of the week. This gives us a profile of what, for example, a Monday or a Wednesday typically looks like, by hour. When calculating the average day-of-the week profile we exclude holidays and the like.

Meanwhile, the National Weather Service provides data on severe weather events that affected the New York City region in 2025. We could get more fine-grained if we wanted to, but for now we’ll just use the general list of events the NWS provides. Then we plot the Subway ridership profile for that specific date against the average profile for whatever day of the week the event happened on.

Bad weather suppresses Subway ridership, in general. But not always.

The gray lines are the baseline. The red ones are the bad weather day. The basic shape of the gray lines (and many of the red ones) is set by the rhythm of daily life. The sharp double-peak pattern is what someone I’ve shown too many of these graphs to has taken to calling “The Giant Cat-Ears of Employment”. The cat-ear shapes vary by work day (which might be the topic of another post), but are most sharply-contrasted with the weekends, which look more like little hillocks or drumlins.

We can see a few different cases in the panels. First are days when the weather event put no dent at all in people’s day. This is because ~~of the incredible toughness and resilience of New Yorkers, something they are surprisingly very modest about~~ even though there was a weather event in the region that day, it just didn’t impinge on the city much, or at all. The light snow on February 11th or the heavy rain on March 6th are examples here. People just continued to go about their business.

Second are cases where there’s a lot of travel suppression but it’s not really—or not wholly—the weather that’s responsible. The winter storm on Friday December 26th is a case of this. Bad weather; strongly suppressed travel profile; but that’s not a regular Friday. Many people were staying at home anyway, because it’s the day after Christmas.

Third are cases where the weather does seem to have suppressed travel. These are days like the snow on January 19th, or the shitty weather on Sunday February 16th. These events look like they made people stay at home. Some of these are more severe than others. The strongest example is the flash flooding on Thursday July 31st. That happened in the back half of the day and affected the evening commute directly.

Our fourth and final category is my favorite one. Sometimes snow makes no difference at all, especially if it’s on a workday. Sometimes it’s snowy on the weekend but you’re kind of sick of it, maybe because it’s late in the winter, so you’re either going about your business as usual or you’re just staying indoors. But there’s another kind of snow day.

Let’s go exploring.

The weekend of December 13th and 14th 2025 brought the city’s first measurable snow of the year, and in decent amounts, too—between four and eight inches of accumulation. Reports remarked on how long it had been in arriving. The result was that, over the weekend, ridership on the subway went up. Maybe on the Saturday it was to go out and buy the mandatory bread, milk, and eggs.¹ But maybe it was also just to be out in the snow. The next day, the people who didn’t have to go work slept in as usual. But that day, too, across the afternoon, more people than usual headed outside and took the subway somewhere. I’d like to think a bunch of them had a sled under their arm.

Maybe, at least for some New Yorkers, it was because it made more sense to take the subway than drive. Though this probably wouldn’t be all that many people. It’d be somewhat possible to investigate this with the data at hand, especially if e.g. outlying stations showed higher ridership rates. ↩︎

Hourly Subway Station Flows

Sat, 25 Apr 2026 11:12:39 -0400

Pie charts are bad, as any fule kno. We’re not as good at judging relative differences between angles and areas as we are at judging relative differences in lengths on a common baseline. This is especially true when we have more than two things to compare at the same time. So, as a rule, you shouldn’t use them. You should figure out some other way of viewing your data instead. On the other hand, I just made 424 animated pie charts because if you’re going to break a rule you should break it good and hard.

A view of the New York City Subway System (excluding the SIR). We’ll animate this in just a minute.

The New York City Subway system is very large and carries a lot of passengers every day. The MTA makes quite a bit of data available about the subway, including data on hourly flow through the system. Now, the MTA can’t track individual pathways people take through the subway. If you use an OMNY card (or before that, a Metrocard) to enter the system, this signals the start of a trip from some specific station or station complex. But unlike some systems, you don’t need to “tag out” of the subway, you just exit through a turnstile. So the system doesn’t know where you exit it. In addition, while many stations are just on a single line, some (like 34 St/Penn Station, or Fulton Street) are station complexes that serve many lines and allow transfers between them.

However, the MTA does publish hourly Origin-Destination estimates for all pairs of stations. These are their best guess about the flow of traffic from any particular station to any other. Because there are so many combinations, visualizing that sort of data is quite tricky. Even then, you don’t get information about routes through the system, just start and end points. Transit analysts and planners can go further by introducing some further assumptions about subway users. For example we might assume that commuters take the most efficient route between any given pair of entry and exit stations, and build from there to a picture of flow through the system.

I do something rather more simple here. I use the MTA’s hourly origin-destination estimates and aggregate them on a station-by-station basis to calculate in-and-out flows across 424 subway stations or station complexes. These specific numbers are averaged over all Mondays in 2025. For each hour of the we calculate the total passenger volume at the station, and the share of that volume that are estimated arrivals and departures. Then we draw a pie chart for each station, coloring it yellow for departures, purple for arrivals. The circle size reflects total volume and the pie slice proportions show the flow balance.

The flow data is pretty bulky. The original dataset has about 121 million rows. But working with it is pretty straightforward, thanks to the magic of parquet files, duckdb, and duckplyr. Having patiently downloaded the data via its API, I put it in a parquet file. The CSV is about 17GB but the parquet file boils it down to 1.5GB. Then I made a small R package that bundled that data with a few convenience functions. This lets me use the data without copying it into any single project. So I can write, e.g.,

nycsubwayodr::nyc_subway_odr()
#> # A duckplyr data frame: 15 variables
#>     year month day_of_week hour_of_day timestamp           day_of_month origin_station_complex_id
#>    <int> <int> <chr>             <int> <dttm>                     <int>                     <int>
#>  1  2025     1 Monday                1 2025-01-06 01:00:00            6                       189
#>  2  2025     1 Monday                1 2025-01-06 01:00:00            6                       313
#>  3  2025     1 Monday                1 2025-01-06 01:00:00            6                       611
#>  4  2025     1 Monday                1 2025-01-06 01:00:00            6                       125
#>  5  2025     1 Monday                1 2025-01-06 01:00:00            6                       313
#>  6  2025     1 Monday                1 2025-01-06 01:00:00            6                       154
#>  7  2025     1 Monday                1 2025-01-06 01:00:00            6                       167
#>  8  2025     1 Monday                1 2025-01-06 01:00:00            6                       612
#>  9  2025     1 Monday                1 2025-01-06 01:00:00            6                       272
#> 10  2025     1 Monday                1 2025-01-06 01:00:00            6                       167
#> # ℹ more rows
#> # ℹ 8 more variables: origin_station_complex_name <chr>, origin_latitude <dbl>, origin_longitude <dbl>,
#> #   destination_station_complex_id <int>, destination_station_complex_name <chr>,
#> #   destination_latitude <dbl>, destination_longitude <dbl>, estimated_average_ridership <dbl>

From there, we lazily query the data and duckdb does the work of doing the calculations. The whole table is never loaded into your R session, and duckdb is very fast. From there, we take our hourly flow summaries, join them to a tibble of station and line data, and export the result to some JSON files that D3js animates for us.

Here’s the result. There are three views. Initially, you see just the schematic subway map. If you click the “Map” button in the top left, it will switch to the ticking pie-chart view, which puts a pie on every station complex, with each tick being an hour of the day. The pies pile up on one another in the geographic view (in a not wholly uninformative way), but click again to have them expand to a somewhat more abstracted, force-directed network view of the system. Then click again to go back to the map. You can hover over or tap on nodes to get information about the bit of data it’s currently showing.

Now, you might reasonably say, Kieran, that’s a lot of data to show that people go to work in the morning and come home in the evening. I’m not saying there’s nothing to that criticism. But there are quite a few interesting details in there as the data pick up traffic to different parts of town. The big interchanges naturally dominate the view, but even here there are things of interest about the balance of flow, as e.g. Penn Station has people coming in on New Jersey Transit during morning rush hour and then entering the subway, which does a lot to balance its net flow during rush-hour and even tip it towards net departures. But more importantly, who doesn’t want to sit back and contemplate more than 400 pie charts, each one pulsing with life as another hour ticks by?

New York City Hexmaps

Sun, 19 Apr 2026 10:05:54 -0400

The five boroughs of New York City can be informally or formally carved up into many different pieces, depending on what it is that you’re doing. As part of an ongoing project, I recently made an R package, nycmaps, that lets you draw maps of some of these geographies. Things being what they are, these spatial units don’t necessarily overlap in compatible ways. City, State, and Congressional Districts, School Districts, Police Precincts, Fire Companies, Election Precincts, Municipal Court Districts, Zip Codes … there are loads of them. Some of them are quite straightforward; others patiently lie in wait to trap unwary analysts (I’m looking at you, Zip Codes / ZCTAs).

Mapping Tracts and NTAs

Two classifications of particular interest to people like me are Census Tracts and Neighborhood Tabulation Areas (NTAs). Census Tracts are defined by the Census Bureau and form part of a nested set of geographical units that go from the smallest unit the Census keeps track of (the Block) up to the largest (the whole country). Blocks aggregate to Block groups, Block groups aggregate to Tracts. Tracts aggregate to Counties. There are of course several complications. Ideally, the Census would like tracts to be contiguous, sub-county geographical areas with about 4,000 people in them, or at least between 1,200 and 8,000 people. This means tracts can vary considerably in geographical area. They generally follow visible features of the environment, whether physical or built. Uninhabited areas also get tract designations, so in principle we can get a full tract-level map of an area with no gaps. Here’s what a tract-level map of New York City looks like:

ggplot(nycmaps::nyc_census_tracts_2020_sf) +
  geom_sf(aes(fill = boro_name), color = "black", linewidth = 0.1) +
  scale_fill_brewer(palette = "Set2") +
  labs(fill = "Borough") +
  theme_void()

2020 NYC Census Tract boundaries.

You can see the variation in tract size (compare e.g. Staten Island tracts with those in lower Manhattan). And you can also see features that are included on the map but, at least to a first approximation, don’t have permanent residents. That big roundy blob in the southeast corner of Queens, for instance, is JFK Airport. There are about 2,300 Census Tracts in New York City. (Naturally, their number and spatial layout changes from decennial census to decennial census, because why should life be easy?)

Neighborhood Tabulation Areas, meanwhile, are not official Census units. They are one of several subdivisions used by New York City government. The idea is to aggregate tracts into units that roughly correspond to neighborhoods that people conventionally refer to. This is, of course, an impossible task, because people don’t agree on neighborhood boundaries. But the idea is good. You want something bigger than a tract because those are small enough to be noisy on many measures produced by the main source of tract-level data, the American Community Survey. But you want something smaller than the next level up, which is a Community District Tabulation Area. Presently, there are 262 NTAs. They look like this:

ggplot(nycmaps::nyc_nta20_sf) +
  geom_sf(aes(fill = boro_name), color = "black", linewidth = 0.1) +
  scale_fill_brewer(palette = "Set2") +
  labs(fill = "Borough") +
  theme_void()

2020 Neighborhood Tabulation Areas

NTAs have recognizable names:

`1`	`nycmaps::nyc_nta20_sf \|> select(nta2020, nta_name, nta_abbrev)`

Simple feature collection with 262 features and 3 fields
Geometry type: MULTIPOLYGON
Dimension:     XY
Bounding box:  xmin: 913175.1 ymin: 120128.4 xmax: 1067383 ymax: 272844.3
Projected CRS: NAD83 / New York Long Island (ftUS)
First 10 features:
   nta2020                            nta_name nta_abbrev
1   BK0101                          Greenpoint      Grnpt
2   BK0102                        Williamsburg   Wllmsbrg
3   BK0103                  South Williamsburg  SWllmsbrg
4   BK0104                   East Williamsburg  EWllmsbrg
5   BK0201                    Brooklyn Heights      BkHts
6   BK0202 Downtown Brooklyn-DUMBO-Boerum Hill   DwntwnBk
7   BK0203                         Fort Greene      FtGrn
8   BK0204                        Clinton Hill    ClntnHl
9   BK0261                  Brooklyn Navy Yard   BkNvyYrd
10  BK0301           Bedford-Stuyvesant (West)    BdSty_W

When we have a table like this, we can get tract-level data from the Census, for example on educational attainment, and aggregate it to the NTA level. Then we can join that data to the simple feature collection that has our geometries in it. With a little polishing (which you can read all about in what I personally think of as a very useful book), we get something like this:

BA degrees or higher within NTAs.

Nominally zero-population NTAs get grayed out (JFK, LGA, various parks and cemeteries, Brooklyn Navy Yards, the United Nations, etc).

Here’s what this data looks like at the tract level:

BA degrees or higher within tracts.

Hexgrids

Sometimes we want a more schematic representation of geographies, because choropleth maps can be tricky to work with. There are many possibilities here, including not drawing maps at all. One option is to make a kind of cartogram by turning our map polygons into a tessellated grid where each unit gets a single tile. My nychex package provides hexagonal and square tilings for NTAs and tracts in New York City. Turning geographically accurate polygons into regular tiled grids can be a bit tricky, especially when the polygons you are trying to tile contain “holes”. But thanks to the tilemaps and rmapshaper packages we can get reasonably far in a semi-automated way, and then tweak things by manually nudging tiles around. Our baseline NTA hexmap looks like this:

ggplot(nychex::nyc_nta20_hex_sf) +
  geom_sf(aes(fill = boro_name), color = "black", linewidth = 0.3) +
  scale_fill_brewer(palette = "Set2") +
  labs(fill = "Borough") +
  theme_void()

The opening gameboard for my upcoming strategy game, Ticket to Ridgewood

And here is what our BA prevalence map looks like when mapped with it:

Labeled NTA hexmap

Here I’ve labeled the hexes with the (sometimes highly) abbreviated version of their NTA name. Maps like this don’t magically solve the difficulties of spatially representing population-based data, but they have their uses. They can be handy if you want to quickly get a sense of variation across units while retaining a roughly spatial layout. They also make some kinds of small-multiple or faceted plots a little easier.

We don’t have to stop at NTA-level resolution. We can do a tract-level one, too. Here’s a tract-level base hexmap:

Ticket to Ridgewood, advanced edition

This map makes a series of compromises with city geography in order to make room for each tract’s hexagon. In particular, northern Manhattan is more detached from the Bronx than is ideal, and it was also necessary to sever Brooklyn from Queens along their shared border. (Our simplify-and-tile algorithm had a very hard time with the undifferentiated Brooklyn/Queens landmass.) With a fully hand-drawn hexmap we could probably avoid most of these problems, but that would involve quite a substantial amount of work. Even when we generate the main components algorithmically, quite a bit of hand-adjustment in the overall positioning and layout is still required (especially for things like the Rockaways and other islands or quasi-islands). Sadly there’s no magic way to integrate the main borough polygons while preserving the orientation of all the hexes. Still, the result isn’t bad.

Here’s our BA map in tract-level hexagonal form:

Tract-level BA map.

The usual benefits and disadvantages of regularized choropleths are in evidence here. Lower Manhattan’s population gets a fairer shout, as do parts of Brooklyn and the Bronx. The geography still mostly works. The difficulties flow mostly from the map being at the tract-level in the first place, rather than the tiling. That is, some tracts have unusual shapes that result in quite noisy estimates. For example, sometimes a tract will consist mostly of a park or an industrial area, but have a few residential segments. This can mean it ends up being measured too high or too low on the thing we’re counting. Or, by contrast, a tract might be almost entirely one kind of entity, like a retirement home, producing results that might seem odd if you don’t know what’s in that spot. You can see why the City aims at the NTA level for a lot of its summaries. It has ten times fewer units, but things get smoothed out in a way that may be more useful. Any real-world method of measurement comes with some rate of error, which the Census helpfully provides estimates of. Nice maps tempt you to reify observations and spin yarns about what you see, whether it’s a finely-detailed spatial polygon or a pleasingly regular hexagon. The finer the observational grain, the more important it is for you to know about the situation on the ground. Literally.

Subway Sign

Sat, 28 Mar 2026 14:25:52 -0400

After the parking signs last time, here is a subway sign.

A borough-specific sign to display

Here’s a direct link to the PNG and the PDF. Once again, put it onna stick and exercise the constitutional rights to freedom of expression, speech, and assembly enjoyed by everyone in the United States.

Confessors in Harmony

Mon, 23 Mar 2026 10:54:57 -0400

Charles Fourier was an early socialist utopian, part of the French tradition of thinkers who came up with various schemes for the complete reorganization of society on more rational grounds, and whose views now read to us as an unstable admixture of obviously sensible notions, delusional crankery, and things that seem to encapsulate both of those elements at once in a way that brings out the weirder aspects of our own dominant forms of social organization. Fourierist communities, known as phalansteries, got off the ground in several countries, including a few in the United States. The basic social unit he had in mind was called a phalanx; in his mind, this would consist of 1,620 people, because of course there are 810 types of personality and you need two of each kind. A phalanstery is thus the large housing complex that houses a phalanx. Horace Greeley founded a couple, including the Sylvania Colony in Pennsylvania that eventually became the town of Greeley, PA.

A phalanstery.

Reading Fourier in his context is of course a matter of careful intellectual history, which we are going to skip here. I just wanted to make a note of a passage I came across this morning where Fourier discusses aspects of his plans for relations between the sexes. Fourier had many interesting views here that flow from his rejection of the constraints of “civilization”.¹ He coined the term “feminism”. His vision was of a kind of liberation of human creative energies and capacities of all kinds, from work to sex, that would be organized in a manner at once fulfilling to individuals and socially productive and harmonious. As is often the case in utopian visions, there’s a desire to have all of the benefits of an advanced division of labor while avoiding its overall problems and rejecting the social mechanisms we seem to have to coordinate it (which is to say, usually, an oppressive state or an open market). French utopian socialism, in particular, often ends up with schemes that are at once clearly hierarchical, or at least occupationally differentiated, highly integrated and solidaristic, but also fully accommodating of individual freedoms and liberties. A magic wand that grants some theory of human nature, or posits a general conversion to some sort of religion of science, is often waved by the author to accomplish the hard parts.

A side effect of this line of thought is that the twin beliefs in rational administration of the social order and the harmonious satisfaction of individual needs and desires can result in ideas that seem strangely familiar to us. For example, in this passage Fourier is discussing the problem of how people should pair up in phalansteries, and in particular how a traveler from one settlement to another should be able to make the best use of their time in matters of love. His answer is that we clearly need some sort of organized matching system that optimizes on the combination of people’s present sexual wants and their basic personality types:

We will now take up one of the most interesting branches of the calculus of the passions: the art of enabling anyone anywhere, even in places where he is a total stranger, to make instant contact with people with whom he is in complete sympathy. If the theory of attraction offered no other advantage, would it not still be a boon to all mankind? Would it not be a blessing to the people of civilization who often spend years in a city without encountering sympathetic partners in love, friendship or any of the passions? In Harmony any traveler will make such acquaintances on the very day of his arrival in a city. … The art of the sympathist, which is unknown in civilization, provides the means for the instant matching of personalities and sympathies, anywhere and under any circumstances.

Let us first consider the enormity of the calculations entailed by sympathetic matching and the speed with which they must be performed. … Let us suppose that a horde of one thousand adventurers and adventuresses has arrived in a Phalanx at four o’clock in the afternoon. They are immediately served light refreshments and then, even before they take time to wash, they rush off to confession. The most skillful confessors and confessoresses of the region have been gathered. Their job is to examine these thousand knights errant, each of whom must submit a written declaration concerning his or her most recent adventures. The confessors go over these declarations as well as the reports of previous confessors; they study the immediate inclinations and physical needs of each individual, and attempt to provide him or her with appropriate sympathetic relationships. Putting all the relevant information together, the confessors determine, by means of an equation, the balance of contrasts and identities that will be most attractive to each of the adventurers.

When an individual is still basking in the enthusiasm of a romantic passion, the delicious contrast provided by a sympathy in the composite mode, should the confessor intervene to provide a diversion? Yes, very likely. For someone who has just concluded a sympathetic relationship in the composite would probably be incapable of duplicating the experience immediately. Then should the confessor provide the visitor with a cabalistic liaison by offering him a choice among several candidates of identical character? Or should an appeal be made to the visitor’s alternating sympathies, his penchant for variety and contrast in both the moral and physical realms? Such are the first questions considered by the confessor. For there are three basic types of sympathies, and the initial problem is to decide which should be employed. A variation may be in order. Or it may be advisable to continue on the same scale of sympathies. For if an individual’s appetite has not been exhausted by his previous adventures, he will be capable of engaging the same sympathies in a higher or lower degree.

That is the heart of the matter; now let us turn to subordinate problems. No matter which of the three sympathies is brought into play, should it be presented directly or should it be preceded by transitions or complements? In the latter case, should the individual be subjected to a direct unitary sympathy or to an inverse unitary sympathy or perhaps even a diffracted sympathy? Should simple movement be relied upon?² This is sometimes a wise course of action, though rather inglorious, as simple movement always is; but it may be necessary in exceptional cases which will be determined by the confessors and the fairies. It is not for the individual to choose between these alternatives. He is too much absorbed by his recent memories. It is up to the calm and judicious confessor to determine the type of charm that will arouse his enthusiasm. On the basis of his declarations the confessor will decide what sort of relationship is most certain to engage his sympathies. Then the fairies with the best knowledge of the Phalanx will designate the matching individuals from whom he may make his choice.

All of this work, which involves the use of algebraic formulas, should be completed for the thousand adventurers within the short time of two or three hours. For if each visitor was not systematically informed concerning his sympathies, if there was no one to inform him about the individuals with whom he could instantaneously establish sympathetic relationships, he would run the risk of getting involved in purely sensual intrigues, intrigues wholly lacking in illusion. Like the people of civilization, he would fall back upon trivial, simple sex. (Sometimes this is necessary, but only as a respite from composite relationships or, as a transition, in moments of hesitation and of overabundant pleasure.) After a two days’ visit, at the moment of his departure, he might accidentally encounter people with whom he was truly sympathetic. Then he would regret having spent his two days without having known their merit and without having formed a liaison which might have charmed him. He would leave the Phalanx with a feeling of resentment, with bad memories instead of delicious illusions.

This goes to show that the functions of the confessor are most important and that a skilled confessor is an invaluable member of any Phalanx. The job is not one that can be confided to the first comer, as it is in civilization; it requires the greatest degree of tact, human understanding, and familiarity with local circumstances. Women will excel more than men in this sort of work, and as a rule in Harmony there will be two confessoresses for every one confessor. They will be magnificently paid for their services and promoted to the highest ranks. As for the formulas used by the confessors in their work of sympathetic matching, they cannot be explained just yet. It will first be necessary to classify the 810 personality types, any one of which may be represented in an equation of sympathy and may appear in a number of cases.

This translation is from The Utopian Vision of Charles Fourier, edited by Jonathan Beecher and Richard Bienvenu (Beacon, 1971, pp. 378–80). They note, somewhat dryly, that “Fourier does in fact go on to explain—at length—the process whereby the confessors do their matching. Unfortunately, his ‘algebraic’ formulas do not lend themselves to translation.”

Here’s a taste of that, from volume eleven of Oeuvres complètes de Charles Fourier:

I wouldn’t rely on it myself.

In any case, I like the idea of a full filing system containing an information card for each person, classified according to a system of 810 personality types, and including regularly updated information about needs and recent activities, curated and administered by experts, and which people consult upon arriving in a new town or city when they are considering how best to match up with someone it would be enjoyable to hook up with. Obviously a completely unrealistic and impossible-to-implement scheme that bears no resemblance whatsoever to how anything in contemporary society works.

Always a derogatory term for Fourier, in contrast with “Harmony”, the overall name for his utopia. ↩︎
Here he means what is in his view the simplest kind of attraction, that of opposites. ↩︎

Using Quarto to Write a Book

Mon, 09 Mar 2026 09:34:53 -0400

I’ve spent the last couple of months revising my Data Visualization book for a second edition that, ideally, will appear some time in the next twelve months. As with the first edition, I’ve posted a complete draft of the book at its website. The production process hasn’t started yet, so it’s not ready to pre-order or anything, but the site has a one-question form you can fill out that asks for your email address if you’d like to be notified with one (and only one) email when it’s available. A lot has changed since the first edition, reflecting changes both in R and ggplot specifically, and in the world of coding generally. I may end up highlighting some of those new elements in other posts. But here, I want to focus on some nerdy details involved in getting the book to its final draft. I’ll discuss Quarto, the publishing system I used, its many advantages, and its current limits with respect to the demands I made of it.

A detail from facing pages in Chapter 4, in the PDF version.

The book is about doing good data visualization using R and ggplot. The book contains many figures, almost all of which are written using the code the book shows and explains.

Reasonable Demands

My baseline list of requirements for the book manuscript was as follows:

The entire text of the book is in some kind of plain-text format.
Figures in the book that are the result of R code should be directly produced by R code in the actual document; no cutting and pasting of code snippets and separately-produced figures. Doing that is a recipe for error.
The scholarly machinery of the book—chapter, section, table, and figure numbering; cross-references; in-text bibilographical references; the bibliography itself and its formatting, and so on—should be automatically handled. No manual numbering and renumbering of figures, etc.
It should be straightforward to repeatedly generate a fully-formatted and laid-out version of the book manuscript as I go, ideally in any of several output formats (e.g. PDF, HTML), despite it all being written in plain text.

These requirements are reasonable because, for projects like this, working in plain text is the right thing to do. We are writing and revising text and our code; we keep the text in a version control system; we don’t want the results of the code to come apart from the code that generated it; and we need to deliver outputs that consist both of fully-formatted material and replication packages that allow other people do see what we did. PDF is of course the worst, but we still need to target it as one of our output formats.

Despite being reasonable, these requirements are in truth quite demanding. Once you start thinking about what all the pieces entail you realize there’s a lot to keep track of. Systems for doing some or all of this have been developed in whole or in part over the years. Newer ones sometimes escape the constraints of older ones; sometimes they inherit their legacies. I’m not going to review them here. This time around I used Quarto.

Quarto is a publishing system focused on documents of different kinds (articles, presentations, books, websites), written as plain-text sources that mix prose and code in any of several languages (R, Python, Julia, others), destined to be fully-finished outputs in any of several formats (PDFs, HTML, or Word files). Quarto builds on and extends many tools, notably pandoc for getting from Markdown to any number of other output formats. It’s a spiritual descendant of literate programming approaches for dealing with code that needs to be run in the context of prose. In the R world these descendants include Sweave and RMarkdown/knitr. These broadly “notebook” approaches to writing and discussing code have benefits and also sharp limits if your focus is full-on software development and its documentation, or complex data analysis involving many interrelated steps.¹ But they’re very useful if you are primarily writing longer-form text that periodically requires things like figures and tables to be programatically generated in a reproducible fashion.

If you just want to know whether you can write long-form projects like articles, books, or websites using Quarto and R, the answer is absolutely yes. A long time ago I wrote parts of my dissertation and several articles using Sweave. A few years ago I wrote the first edition of Data Visualization using RMarkdown. I wrote the second edition using Quarto. Each one was better than the previous version in terms of flexibility and power. Quarto eliminated several pain-points that I had to deal with for the first edition of this book. It’s very well-documented and continually improving. Its defaults are sensible and produce good-looking output. You can stop reading now.

A schematic overview of how Quarto orchestrates its document processing.

Unreasonable Demands

I had a very clear idea about how I wanted the first edition of the book to look in print. I also knew that I wanted to make it available as a website. I was fortunate enough to be able to have both of these things work out. This time around, I did the same again but I really wanted there to be as little as possible post hoc work with the website version. I knew that wouldn’t be the case with the PDF, for reasons I will discuss in a moment. I’m pleased that Quarto performed so well with the whole process. I wrote two pretty heavily-customized output formats (one for PDF and one for HTML) that specified the layout of the book. Quarto’s LaTeX-based book pipeline uses the scrbook class from the KOMA-script bundle, which has many nice features, though I find its documentation a tiny bit eccentric. (This might be because I wrote my first book using the memoir class.) I also wrote a couple of R packages that managed the themes and some other details of how PNG and especially PDF figures were produced. A version of the theme is in the development version of the socviz package that accompanies the book.

The PDF design is a two-column “Tufte-style” layout with wide margins for side-notes and figures. It works very well for a book of this kind as we can show small figures alongside the code that generates them, but also have figures break out of the main text column if needed.

Facing pages with a figure that runs the full width of one of the pages.

A layout like this can’t be rigidly ported over to a website, especially in an era of widely-varying screen sizes. So the HTML version of the book has a broadly responsive layout that arranges things differently at different sizes. Organizing and tweaking it this time around was made a lot easier by Quarto’s much better support for margin notes and marginal figures. It certainly wasn’t without its headaches. Marginal figures and notes are quite annoying to deal with in both HTML and PDF formats, for different reasons. In the PDF case, it’s tricky to get captions right, and there are still a few hacks in there to make it work. But it’s much cleaner than what I had to do in RMarkdown for the first edition, which was in effect a lot of regular expression substitution for things I could only add after the .tex file was produced. That’s gone now.

Here’s a screenshot of a facing page layout with some code, some marginal notes, and two kinds of figures, one in the margin and one full page-width:

Gapminder figures in the PDF version.

And here’s some of the same material as seen on the website:

Gapminder figures in the HTML version

Here’s a direct link to the same section. In the website version the marginal figures appear more marginal. There’s also a little bit of conflict to be worked out between the navigation guides and the marginal notes. In addition, the intrinsic variability of the web layout means that the positioning of the marginal notes is less precisely controllable than it is in the PDF output. But the overall result is pretty good. And I have to say it’s very satisfying to be able to produce a good website and a clean PDF (and also an ePub!) from the same folder of qmd files, with the text written in Markdown, the bibliography managed by Zotero and BBT, interspersed with the code that makes all the figures.

Let the Professionals do a Professional Job

I should say “less precisely controllable without substantial further adjustment”. Because this is the crux of the customization biscuit. There’s no end to it. One of the benefits of being in a position to do a second edition—something I really am very grateful for—is that it allowed me to have a much better sense of the production process for the hard copy of the book. This in turn placed sharp limits on what I was willing to do when it came to customizing the PDF version myself. Camera-ready files for books published by proper Presses are produced in many different ways. My most recent book, which is all prose and no code, was designed and typeset using Adobe InDesign. For the first edition of Data Visualization I sent the Press a set of LaTeX files and PDF image assets. The LaTeX files produced a very good facsimile of the design we’d agreed on. Then the Press’s typesetter got to work on it.

You might think that they just took my files, lightly edited them here and there, and added the trim, bleed, registration, and color marks for the physical print job. That’s not how it went. Book layouts are very hard to get just right, especially layouts that have many different-sized images and notes and other paraphernalia. They’re fragile. Moving something slightly here, or editing a sentence there, can cause a cascade of unwanted effects. Even ordinary pages of text will have issues with excessive or insufficient spacing around paragraph and section breaks, or widows and orphans, or rivers, and many other infelicities that most people won’t notice explicitly, but which cumulatively convey bad vibes even to people who don’t much care about design.

Some of this can be automated. That’s what layout algorithms do. The Knuth-Plass box-and-glue algorithm, which is the thing that causes TeX to emit those Underfull \hbox (badness 10000) complaints, is a real marvel. But it can’t quite work miracles. In my case, the professional typesetter took my LaTeX file, threw away my document class and substituted their own custom one (and some custom style files). Like any document class it defined the layout and all the features of the book, but it also included a variety of commands that allowed her to finely adjust the text as needed on any particular page. Tightening up the spacing here; forcing a break there; very slightly expanding or contracting the page size when needed to make sure that the layout didn’t break in a visible way on the next page, and so on. Here’s an example from the first edition:

And another:

Those uses of {\break}, \enlargethispage, \vspace{}, and the non-breaking space in this~way are all done by hand, based on rendering and re-rendering the document as it’s built to make sure each page meets the Press’s standards. An automatically-produced PDF can get you eighty five or ninety percent of the way to this but, if you really want to get things right, that last stretch will inevitably mean a bunch of adjustment by hand in whatever the final format is. That’s not something you can incorporate into your reproducibility pipeline.

Fortunately, you don’t need to. Most of the time we don’t require anything like that level of attention to detail. It’s worth producing and circulating material in accessible and readable formats that also don’t look like garbage. And it’s gratifying to be able to reliably generate pretty high-quality versions of those outputs from plain-text sources. That’s more than good enough in almost all cases. When writing papers that end up as PDFs, for example, I use a template that’s almost 20 years old. I only touch it when something breaks. By the same token, while my amateur interests compel me to run up polished custom Quarto book formats for my book projects, I also know that the people who set type for a living know a lot more about the fine grain of that work than I know, or need to know. But once in a while it’s nice to see how far you can push things.

The trick is to have the code chunks in your document be short and sweet, and have structured scripts and properly-documented packages manage the heavy lifting in any analysis. ↩︎

Data Visualization, Second Edition

Fri, 06 Mar 2026 06:52:36 -0500

I’ve written a second edition of Data Visualization: A Practical Introduction, which ideally should come out with Princeton University Press later this year. As with the first edition, a full draft of the book is available at https://socviz.co. The production process is just getting started so there’s no new cover yet, and there isn’t a link to pre-order. But (also like last time) I’ve put up a link to a form that lets you add your email if you’d like to be notified when it’s available to buy. You’ll only get one email (from me personally, not a marketing department) if you do; no spam or anything.

The revised edition is a pretty thorough rewrite. Naturally all the code is brought up to date for ggplot 4 and R version 4.5 and higher. The code from the first edition still runs, but you’ll get warnings and so on; those are all now gone. The back half of the book has been pretty thoroughly redone to reflect big changes in the availability of software for maps, (the sf package) and extracting results from models (the marginaleffects package). Meanwhile, several years of teaching this material (and getting feedback from others) have resulted in shifts of emphasis here and there to introduce just a little bit more on data wrangling. As the book goes on I also shift from an “object-based” approach to writing plots to a more “pipeline-based” one.

The recent rise of LLMs and coding agents gets some discussion, too. There the question is “Why can’t I just have a robot write all the code for me?” I don’t dismiss this question out of hand, and I don’t pretend that agents aren’t very powerful. My feeling about this is summed up in the Preface:

Perhaps you have a robot to help you write your code now. Large Language Models (LLMs) and coding agents are now part of the workflow of code generation and evaluation. They can do a great deal; so much so that it might seem superfluous to spend any time with the iterative, write-try-redo approach to visualization that this book presents. Can’t the robot write all the code instead? Not quite. It’s not that I believe repeatedly doing repetitive and error-prone tasks yourself is a virtue. To the contrary, that’s what computers are for. This book is full of examples where we end up automating something in order not to worry about it. But I also want you, the reader, to learn how to do good graphical work in a reproducible way. That means having a keen eye for quality and a good nose for error. Cultivating those senses requires practice and a vocabulary to express them. It seems faintly absurd to have to say explicitly but, whatever tools you use, your work will be better if you know what you are doing and understand why you are doing it. This book teaches you ggplot specifically, but it is not trying to lock you in to a particular framework. It’s just that, the way you acquire a general skill or a wide-ranging taste is by first learning some more specific version of those things, and then practicing them. Automation can come a later. In the words of the author Ann Leckie, you don’t learn how to do something by not doing it. For that reason, this book remains a hands-on introduction.

Or to put it another way, the book is an introduction to how to do something. One feature of books like it is that they tend to have two audiences: people who don’t know anything about the topic, and who’d like to learn something about it, and people who know a lot, at least in relative terms, and who have forgotten what it’s like not to know it. When the first edition came out, one of the early Amazon reviews was a complaint that the book seemed “pretty introductory” in its content. I mean, my Brother in Christ, that is right there in the title.

As with any corner of the vast division of labor that is human society, not everyone has to know about any specific thing in great detail. We’re all taking huge amounts of stuff for granted at any moment. But if you want to be proficient in some piece of that enormous web, it’s better that you know rather than not know what’s what. There’s nothing wrong with using tools that give you tremendous leverage. You do it every time you use a stand mixer in the kitchen, or a sander in the garage. You do it every time you turn your computer on, in fact. But you still need to develop the capacity to tell good work from bad, or correct from incorrect output, or safe uses from dangerous ones. That way you can take advantage of the power tools without being at risk of slicing your own or anyone else’s arm off.

Ordinal Exchanges

Thu, 13 Nov 2025 12:52:40 -0500

The Socio-Economic Review has published a book symposium on The Ordinal Society with contributions from Nitsan Chorev, J.P. Pardo-Guerra, and Greta Krippner, followed by a reply from Marion and myself. (Here’s a PDF of the exchange.) There’s also a symposium at the Journal of Cultural Economy with contributions from Hatim A. Rahman, Juan M. del Nido, Julien Migozzi, and Michelle Jackson, again with a reply from us. Finally there’s also a new review from Michael Sauder in ASQ (PDF), which follows on reviews from Barbara Kiviat in Social Forces (PDF), and Laura Nelson in Acta Sociologica.

As we say more than once in our replies, it’s very gratifying to have your peers in the field take the time to read your work.

Trustworthy Data Visualization

Mon, 10 Nov 2025 08:26:45 -0500

This past September I gave the closing keynote at posit::conf; it’s now on YouTube to watch. Keen-eyed observers will note from the title that it’s about trustworthy data visualization. But it’s also about trust a bit more generally, and how we should think about it in a world where researchers are faking results, AIs are enthusiastically confabulating, and government is destroying data infrastructure. When you find yourself giving a talk with a little tiny microphone stuck to the side of your head you have to ask yourself some hard questions, but the talk was partly about that.

One of the nice things about the opportunity to give a talk like this to a large audience is that you can tell the audience not just about your own stuff, but also the work you build from and rely on. This is, as you’ll see if you watch, one of the themes of the talk in the first place. In my case, in addition to all the good things done by people in the general area of data visualization and data science, I was able to talk about some ideas from social theory and, in addition, some of the work of Katherine Hawley on trust and commitment. The fulcrum of the talk is essentially an idea about commitment that she developed in her book How to be Trustworthy. I got to know Katherine and her family a little bit over a few summers in the 2010s when my own family used to visit St Andrews, where she was Professor of Philosophy. She died of cancer at the age of fifty in 2021. She was a wonderful person; a clear-minded philosopher of enviable intellectual gifts but also immense personal kindness. It was good to be able to think of her while writing this talk and, in however small a way, introduce some of her ideas to a new audience when I gave it.

Mamdani vs Sliwa and Cuomo

Thu, 06 Nov 2025 12:57:44 -0500

Mamdani’s victory in the New York City mayoral election gave me the opportunity to draw a few maps, and also to learn a bit about incorporating additional spatial data into maps drawn in R. R is not a specialized piece of GIS software. ESRI’s ArcGIS is the 800lb gorilla in this world and QGIS the GIMP to its Photoshop, so to speak.

Still, you can do a lot of spatial stuff in R, grounded in the sf package and its many friends. Plus you get the benefit of all the data manipulation and analysis that R is really good at. So, having gotten the precinct-level results for the election, some maps from New York City (e.g., the clipped borough boundaries map), and GTFS data from the MTA describing the structure of the subway system, I was able to draw some things. I strongly approve of the existence of the GTFS, by the way. It’s a spec for encoding transit data and lots of cities use it. Really handy.

Anyway, here’s a map. For each precinct with more than twenty voters, I combined the Sliwa/Cuomo vote into what we might call (purely for compactness reasons) the Slimo vote, calculated the Mamdani and Slimo vote shares as a proportion, and subtracted the former from the latter. That gets us a score raning from -1 to +1. I then cut that into bins, ten on each side of the zero line, to get deciles in each direction. That’s what we fill the precincts with. For the map I also drew the subway stations and lines. Several lines that are right next to each other are in effect drawn on top of one another in several places, e.g. the A and the C, or the 1 and the 3, etc, but that doesn’t matter for this map. (You may have heard that drawing transit maps meant for navigation is a really hard information design challenge.) We then use a discrete, diverging scale. Here’s the result.

Oh, choropleths

I chose a blue-green color gradient partly to experiment with it and partly because neither the election nor this particular cut of the data is quite the usual Blue vs Red, Democrat vs Republican. For one thing we have amalgamated two candidates on one side of the spectrum. For another, Cuomo is in some sense a Democrat, so the way voters were split is trickier than it normally would be. Andrew Gelman has some more thoughts on this today.

You can see a bunch of nice neighborhood patterns, such as e.g. the Hasidic communities in Brooklyn who voted strongly for Cuomo. And you can also see evidence of the characteristic weakness of choropleths, which is the way they can force a number characterizing some number of persons to be represented by a shape representing some area of space.

One solution to this is to make a dot-density map. You put a dot on the map for every person (or maybe every n people) you want to represent. Here’s what that looks like, with a 1-to-1 representation of dots to voters.

Dot dot dot.

This is better than a choropleth in some key ways. For one thing, you can see that—even in New York City—some places are both much more densely populated than others and also more likely to turn out to vote. To be clear, when I say “each dot represents one vote”, it’s not as if I know the identity of every voter, or how they voted, or can precisely locate them to their home address. I promise I don’t know that. Only the NSA and the Phone Company know that stuff. What I do know is how many votes were cast for each candidate within each of the 4,200 or so precincts. And I have a polygon that represents the shape of each precinct. So I spatially sample without replacement within each polygon to randomly place a dot for each voter within their precinct. With two million or so votes the pointillist effect ends up being quite effective.

A slice across Manhattan in the upper 50s and lower 60s and over into Queens.

The main reason for doing this is that, from the point of view of the election, precincts aren’t real. They are closely related to the social geography of the city, which is one of the reasons we want to draw a map like this, but in and of themselves they are at best proxies for the thing we care about, so we should take care not to reify them. The sampling method still brings out the basis of the data, so you can see the precinct-patchwork that forms the underlying grid, especially in densely-populated areas. But those polygons are filled in proportion to the number of people who actually voted. I saw someone on social media observe that this “conflated” partisan lean and population density. But again, precincts aren’t real. The distribution of partsian lean across population density is what we’re trying to bring out with this quasi-person-level approach.

Detail of the PDF.

There are costs, of course. Color choice is harder, especially with more than one category of dot at once. The image tends to look less bright because you’re not using a big swatch of paint across a large area. The visual impression is also very sensitive to small changes in the size of the dots and to settings like their alpha transparency. The order that the layers are drawn also matters a great deal at this sort of resolution. In addition, when you draw a few million dots on a large grid of pixels then your file size gets big real fast. The main dot-density image above is a 300dpi 4,500 by 4,500 pixel PNG and it’s 12mb in size before being crushed down further (with optipng) to 9mb or so. I could make a JPG of course, which would be a lot smaller, but then you start running into the question of why you made a dot-density map in the first place, because you lose detail in the raster. In fairness, you’d still get the benefit of having larger and less densely-populated (or lower-turnout) precincts not appear fully filled-in, even if you wouldn’t be able to zoom in at all.

Meanwhile, thanks to the wonders of multiplication, a 600dpi version of the same image is much sharper to zoom in on but is also about 35mb in size. A PDF, which is a vector rather than a raster format, is even bigger, weighing in at 63MB. Rendering a PDF with that many vector elements does not make Preview or Illustrator happy, let me tell you. The benefit of course is that you can scale it up to any size you like without loss, as shown above. Because browsers are crazy and so is javascript, it’s possible to serve up dot density maps like this in real time in a way that makes them zoomable and fluid, but that’s not my department.

GSS Release

Tue, 28 Oct 2025 11:14:15 -0400

Trends in the immameco question.

Release 2 of the 2024 GSS cross-section and 1972-2024 culumative data are now available. I’ve updated gssr and gssrdoc to incorporate them. There are quite a few changes in the data and variables, thanks in part to some changes in data collection methods and a privacy/disclosure review.

The gssr and gssrdoc packages are the nicest way to get General Social Survey data up and running in R. The figure above shows (survey-weighted) trends derived from the immameco question.

Manhattan Plot of Manhattan

Sat, 25 Oct 2025 11:38:02 -0400

Here I continue my efforts to design visualizations that are as poorly-suited as possible to being displayed on phones. It looks pretty good on a big monitor, or six feet wide on a wall.

I made a version of this plot a few years ago. I ended up revisiting it this morning because I’m updating various datasets and code. A Manhattan plot is a term sometimes used to describe a kind of scatter plot where the x-values are fairly continuous, and the y values have distributions with long tails, so the plot looks like a skyline. This one here is a bar chart rather than a scatter plot but it’s still a kind of Manhattan plot of Manhattan.

The plot shows the heights of almost all currently-existing buildings¹ in Manhattan (on the y-axis) by their year of construction on the x-axis. What I want is a plot that gives a sense of the distribution of building heights over time. To make the plot work I play a few tricks. First, the resolution of the x-axis is only to the year,which would result in way too much overplotting. (We have almost 35,000 buildings to draw.) So we add a small amount of random noise to the x-values, which makes buildings distribute themselves around their year of construction. There’s still overplotting, but now it works in our favor. It contributes to a feeling of building density.

Second, there are so many buildings that we can’t plot everything as solid, filled rectangle. Instead, we make make the outlines of each rectangle a very thin white line, so everything looks like a vector-driven video game from 1981. Then we make a variable that bins buildings by height into ten categories, one for every hundred feet of additional roof height. We map that to fill color of the rectangles (darker purples for shorter buildings through bright yellow for taller ones), which means the taller a building the brighter it looks. But again, we can’t just plot those as solid colors. So we also take the roof heights and rescale them to a range of 0.4 to 0.85. Then we map that number directly to the alpha channel of the fill color. So taller buildings are not only brighter in color, they are more opaque. Alpha runs from 0 (fully transparent) to 1 (fully opaque). The specific values of 0.4 to 0.85 are just from trial and error. We want the taller buildings to stand out, and because there are fewer of them they need to be more opaque. Whereas the fills of many more shorter buildings will overlap, so they can be individually more transparent.

You can really see the recent rise of supertall fancy apartment buildings in Manhattan in the last ten years or so—buildings like the curséd 111 West 57th and 53 West 53rd. The cursédness 111 W57th extends to dataset:

manhat |>
  filter(heightroof == 1428)
#> Simple feature collection with 1 feature and 19 fields
#> Geometry type: POLYGON
#> Dimension:     XY
#> Bounding box:  xmin: 990374 ymin: 217856.8 xmax: 990559.1 ymax: 218081.1
#> Projected CRS: NAD83 / New York Long Island (ftUS)
#>   name     bin cnstrct_yr date_lstmo   time_lstmo lststatype doitt_id heightroof feat_code groundelev shape_area shape_len   base_bbl
#> 1 <NA> 1023728       1924 2021-01-04 00:00:00.000     Merged  1260269       1428      2100         58          0         0 1010100025
#>   mpluto_bbl geomsource BoroCode  BoroName Shape_Leng Shape_Area                       geometry
#> 1 1010107507 Other (Man        1 Manhattan   359993.1  636620786 POLYGON ((990461.3 217856.8...

It’s recorded as having been completed in 1924, instead of 2019. Why? There’s a hint in the lststatype column, which says “Merged”. Technically the building took over and “renovated” Steinway Hall, which was built in 1924. There are a few cases like this in the dataset for buildings that are going to be salient in the figure—i.e. very tall ones. There’s also some missing data, with eight buildings over 600 feet tall where the construction year is not available. At least two of those are because the building was still under construction when the data were recorded. As always, 90% of data analysis is data cleaning.

Data on building heights and construction years come from the NYC Open Data portal. The data are restricted to buildings constructed after 1899 that are currently standing in Manhattan. ↩︎

gssrdoc Updates

Sun, 19 Oct 2025 10:50:04 -0400

Regular readers know that I maintain gssr and gssrdoc, two packages for R. The former makes the General Social Survey’s annual, cumulative and panel datasets available in a way that’s easy to use in R. The latter makes the survey’s codebook available in R’s integrated help system in a way that documents every GSS variable as if it were a function or object in R, so you can query them in exactly the same way as any function from the R console or in the IDE of your choice. As a bonus, because I use pkgdown to document the packages, I get a website as a side-effect. In the case of gssrdoc this means a browsable index of all the GSS variables. The GSS is the Hubble Space Telescope of American social science; our longest-running representative view of many aspects of the character and opinions of American households. The data is freely available from NORC, but they distribute it in SPSS, SAS, and STATA formats. I wrote these packages in an effort to make it more easily available in R. If you want to know the relationship between these various platforms, I have you covered. But the important thing is that R is a free and open-source project, and the others are not.

This week I spent a little time updating gssrdoc a bit to clean up how the help pages looked and make some other improvements. Inside R, you can say, e.g., ?govdook at the console and have this pop up in the help:

Yeah govdook is short for ‘Gov Do OK’, not ‘Go v Dook’.

The package also includes gss_doc, a data frame containing all of the information that the help pages are built from. I included it because it can be useful to work with directly, as when you might want to extract summary information about a subset of variables.

library(tibble)
library(gssrdoc)

gss_doc
#> # A tibble: 6,694 × 10
#>    variable description                           question         value_labels var_yrtab yrballot_df module_df subject_df norc_id norc_url
#>    <chr>    <chr>                                 <chr>            <chr>        <list>    <list>      <list>    <list>       <int> <chr>   
#>  1 year     GSS year for this respondent          "GSS year"       "[NA(d)] do… <chr [1]> <tibble>    <tibble>  <tibble>         1 https:/…
#>  2 id       Respondent id number                  "Respondent id … ""           <chr [1]> <tibble>    <tibble>  <tibble>         2 https:/…
#>  3 wrkstat  labor force status                    "Last week were… "[1] workin… <tibble>  <tibble>    <tibble>  <tibble>         3 https:/…
#>  4 hrs1     number of hours worked last week      "Last week were… "[89] 89+ h… <chr [1]> <tibble>    <tibble>  <tibble>         4 https:/…
#>  5 hrs2     number of hours usually work a week   "Last week were… "[89] 89+ h… <tibble>  <tibble>    <tibble>  <tibble>         5 https:/…
#>  6 evwork   ever work as long as one year         "Last week were… "[1] yes / … <tibble>  <tibble>    <tibble>  <tibble>         6 https:/…
#>  7 occ      R's census occupation code (1970)     "A. What kind o… "[NA(d)] do… <chr [1]> <tibble>    <tibble>  <tibble>         7 https:/…
#>  8 prestige r's occupational prestige score(1970) "A. What kind o… "[NA(d)] do… <tibble>  <tibble>    <tibble>  <tibble>         8 https:/…
#>  9 wrkslf   r self-emp or works for somebody      "A. What kind o… "[1] self-e… <tibble>  <tibble>    <tibble>  <tibble>         9 https:/…
#> 10 wrkgovt  govt or private employee              "A. What kind o… "[1] govern… <tibble>  <tibble>    <tibble>  <tibble>        10 https:/…
#> # ℹ 6,684 more rows

The gss_doc object has regular columns but also a series of list-columns to (insert meme here, you know the one) put data frames inside your data frames. (They’re labeled as “tibbles” here; basically the same thing).

Why a list-column? Why a list? Well, a list is one of the fundamental ways to store data of any sort. Lists are useful because they can contain heterogeneous elements:

items <- list(
  todo_home = c("Laundry", "Clean bathroom", "Feed cat", "Bring out rubbish bins"),
  important_dates = as.Date(c("1776-07-04", "1788-06-21", "2025-01-18")),
  keycode = 8675309,
  storage_tiers = c(128, 256, 512, 1024)
)

items
#> $todo_home
#> [1] "Laundry"                "Clean bathroom"         "Feed cat"               "Bring out rubbish bins"
#> 
#> $important_dates
#> [1] "1776-07-04" "1788-06-21" "2025-01-18"
#> 
#> $keycode
#> [1] 8675309
#> 
#> $storage_tiers
#> [1]  128  256  512 1024

One thing to notice about a list like this is that it doesn’t really make sense to represent it as a table. This is partly because the elements of the list are of different lengths, but really it’s because if we did represent it as a table, it would not mean anything to read across the rows:

items_df
#> # A tibble: 4 × 4
#>   todo_home              important_dates keycode storage_tiers
#>   <chr>                  <date>            <int>         <int>
#> 1 Laundry                1776-07-04      8675309           128
#> 2 Clean bathroom         1788-06-21           -            256
#> 3 Feed cat               2025-01-18           -            512
#> 4 Bring out rubbish bins -                    -           1024

The rows don’t form “cases” of anything. We just have four unrelated categories with various pieces of information in them.

Lists are also useful because they lend themselves easily to being nested:

items <- list(
  todo_home = list(
    tasks = c("Laundry", "Clean bathroom", "Feed cat", "Bring out rubbish bins"),
    tobuy = c("Cat Food", "Burritos"), 
    wifi_password = "p@ssw0rd!"
  ),
  important_dates = as.Date(c("1776-07-04", "1788-06-21", "2025-01-18")),
  keycode = 8675309,
  storage_tiers = list(
    ssd = c(128, 256, 512, 1024),
    ram = c(1, 4, 8)
  )
)

items
#> $todo_home
#> $todo_home$tasks
#> [1] "Laundry"                "Clean bathroom"         "Feed cat"               "Bring out rubbish bins"
#> 
#> $todo_home$tobuy
#> [1] "Cat Food" "Burritos"
#> 
#> $todo_home$wifi_password
#> [1] "p@ssw0rd!"
#> 
#> 
#> $important_dates
#> [1] "1776-07-04" "1788-06-21" "2025-01-18"
#> 
#> $keycode
#> [1] 8675309
#> 
#> $storage_tiers
#> $storage_tiers$ssd
#> [1]  128  256  512 1024
#> 
#> $storage_tiers$ram
#> [1] 1 4 8

In its bones, R is a LISP/Scheme-like list-processing language fused with features of classic array languages like APL. This is because, in the world of data analysis, what we deal with all the time are rectangular tables, or arrays, where rows are cases and columns are different sorts of variables. The wrinkle is that, unlike a beautiful array of pure numbers, each column might measure something (a date, a True/False answer, a location, a score, a nationality) that we’d prefer not to represent directly as a number. Sure, underneath in the computer everything is all just ones and zeros. (Or rather, electromagnetic patterns in some physical substrate that we can interpret as meaning ones and zeros.) And if we want to do any sort of data analysis that involves treating our table as a matrix then we’ll want numeric representations of all the columns. But for many uses we’d like to see “France” or “Strongly Agree” instead of “33” or “5”. Just a table of rows and columns, where different things can be represented across columns, but any particular column is all the same kind of thing.

A rectangular table like that is called a data frame. One way to think of a data frame is just as a special case of a list. A data frame is a list where you can put all the list elements side by side and treat them as columns, and where all these elements are made of vectors of the same length. Beyond that, it’s a list where the nth element of each vector refers to some property of the same underlying entity, i.e. the thing that’s in the row, or case; the thing the columns are showing you measurements or properties of. You can have empty entries if needed, as when some bit of data is missing. The important thing is that each column has as many slots as there are cases, and you fill in the values for each case in the same slot in each column. Whenever you look at any table of data, one of your first questions should always be “What is a row in this table?” In this case, each row is a variable in the full GSS dataset, and each column describes some property of that variable.

library(tibble)
library(gssrdoc)

gss_doc
#> # A tibble: 6,694 × 10
#>    variable description                           question         value_labels var_yrtab yrballot_df module_df subject_df norc_id norc_url
#>    <chr>    <chr>                                 <chr>            <chr>        <list>    <list>      <list>    <list>       <int> <chr>   
#>  1 year     GSS year for this respondent          "GSS year"       "[NA(d)] do… <chr [1]> <tibble>    <tibble>  <tibble>         1 https:/…
#>  2 id       Respondent id number                  "Respondent id … ""           <chr [1]> <tibble>    <tibble>  <tibble>         2 https:/…
#>  3 wrkstat  labor force status                    "Last week were… "[1] workin… <tibble>  <tibble>    <tibble>  <tibble>         3 https:/…
#>  4 hrs1     number of hours worked last week      "Last week were… "[89] 89+ h… <chr [1]> <tibble>    <tibble>  <tibble>         4 https:/…
#>  5 hrs2     number of hours usually work a week   "Last week were… "[89] 89+ h… <tibble>  <tibble>    <tibble>  <tibble>         5 https:/…
#>  6 evwork   ever work as long as one year         "Last week were… "[1] yes / … <tibble>  <tibble>    <tibble>  <tibble>         6 https:/…
#>  7 occ      R's census occupation code (1970)     "A. What kind o… "[NA(d)] do… <chr [1]> <tibble>    <tibble>  <tibble>         7 https:/…
#>  8 prestige r's occupational prestige score(1970) "A. What kind o… "[NA(d)] do… <tibble>  <tibble>    <tibble>  <tibble>         8 https:/…
#>  9 wrkslf   r self-emp or works for somebody      "A. What kind o… "[1] self-e… <tibble>  <tibble>    <tibble>  <tibble>         9 https:/…
#> 10 wrkgovt  govt or private employee              "A. What kind o… "[1] govern… <tibble>  <tibble>    <tibble>  <tibble>        10 https:/…
#> # ℹ 6,684 more rows

Because R was designed by statisticians—R is a descendant of S, which like everything else in computing traces its origins to Bell Labs—it has this concept of a data frame built-in to its core instead of being bolted-on afterwards, which is extremely handy. Normally data frames are just ordinary rectangles, but there’s no reason why any particular column can’t itself be thought of as a list of something else. That’s what we have here. The yr_vartab column contains data frames of crosstabs of the answers to each question by year. Except where it doesn’t (e.g. for id), and this is fine because lists don’t have to be internally homogeneous. Similarly yrballot_df has a little table of which ballots, or internal portions of the survey, a question was asked on for each year it was asked.

The upshot is that having assembled the gss_doc object we can use it to emit, like, seven thousand pages of documentation on the GSS’s many, many questions over the years. We can build them as standardized R help pages, as above. On the website that pgkdown builds for us, we get this:

Website view.

The cross-referencing to other relevant variables in the “See Also” section is new in this version. It comes courtesy of the GSS’s own information about survey modules and an ad hoc topic index they keep for the variables. I just use a subset of possible cross-references as we don’t want, e.g., every single question in the GSS core to be cross-referenced to every other core question on any particular help page. On the website, I gather these into a single page:

Topic index page.

The GSS has its own handy data explorer which is very useful for quickly checking on particular trends and getting a quick graph of what the data look like, or a summary view of the content of particular variables. Each help page in gssrdoc now links to the GSS Data Explorer page for that variable, in case you want to hop over and take a look there. Of course, the gssrdoc package doesn’t and isn’t meant to replace the Data Explorer; it’s just a different view of the same information, with a different use-case in mind.

Parking Signs

Mon, 13 Oct 2025 16:01:57 -0400

If you want to print out a poster for October 18th, here are two; fully-guaranteed countrywide but maybe especially suitable for in and around New York.

Two signs to display

Here’s a direct link to the No King Anytime JPG and the Two Term Limit JPG. These are available as PDFs, as well (vector outlines; no worries about fonts): here’s the No King Anytime PDF and the Two Term Limit PDF. Put them onna stick and exercise the constitutional rights to freedom of expression, speech, and assembly enjoyed by everyone in the United States.

Halloween in the Round

Wed, 08 Oct 2025 09:38:09 -0400

Last year I wrote about cleaning some data from the NHTSA FARS database, the system that tracks information about road accidents in the United States. I did that again this year for my Modern Plain Text Computing class. I won’t repeat the cleaning details, which are more or less the same. One question is how to aggregate data like this, if at all, and how to draw a picture of it. There are, as always, lots of options. The data as obtained (with a slightly different query from last year; a small lesson in itself about repeated measures) arrive as counts of pedestrian fatalities (in motor vehicle accidents) for each day of the year, within months, for each year from 2009 to 2023.

The data in the spreadsheet you get after a specific query to FARS on the web.

What I was interested in was patterns of daily counts, so I wanted to average by day across years. This sort of aggregation of course gets rid of other things we might be interested in, like trends over time by year. Averaging by day means we won’t see, for example, any tendency for the number of pedestrian deaths to decrease over time. One way to draw this is a column chart with time (as day-of-the-year) on the x-axis and the average count on the y-axis.

Wide version.

This works fine. Nice and compact. One thing people tend to underestimate about graphs like this (whether as bars or lines) is that you can often compress the vertical axis a lot without loss of information. Indeed, often with long time series that’s the right thing to do anyway, because there’s nothing better for making your trends look dramatic than narrowing the horizontal part of the aspect ratio. Here the focus is not on a trend, but on the one day that really stands out from the others. In any case, wide is good.

On the other hand, most people are looking at things on their phones. Maybe a vertical view can be better there? We could do something like this, like last year:

Tall version.

Here we facet by month and stack them. It works OK. This would be more useful for data where you expect a bit more structure down the rows as well as across the columns of the table you’re graphing. For example, accidents vary by day of the week, especially pedestrian accidents. There are more people out at the weekend. And then for some parts of the country the seasonality of pedestrian activity would be stronger than in others, because it’s nicer to be outside some months of the year than others. (You can in fact get counts by day of the week from FARS, but I didn’t; exercise for the reader, etc.) Even in this one you can see some evidence of that, as Summer and (back to school) Fall have more fatalities than January, for instance.

Finally, and the point of this post, we can also experiment with the fact that the year is cyclical, and use polar coordinates. When we do this we take our x-axis and wrap it around a circle. Position on the x-axis measured as a distance becomes position on a polar or radial axis measured as an angle. We call it theta and measure in degrees or radians or whatever. In ggplot coord_polar() is available as a transformation to do this. One of the nice things about ggplot’s grammar-of-graphics approach is that it makes it easier to show how the same graph can change under various transformations. A standard xy plot has its coordinates set up by the coord_cartesian() function. We usually never write this unless explicitly want to tweak some aspect of it, but it’s there. But we can just replace it altogether with a different coordinate transformation altogether. This can be a polar system or something else, as when we draw a map and systematically transform points on a sphere into a flat surface via some projection.

Version 4 of ggplot came out last month and introduced coord_radial() as an improved version of coord_polar(). We can use it to draw our plot. Here we’ll use a FARS dataset that shows fatal crashes that involve pedestrians aged 17 and under. (The difference from the ones above is that pedestrians were “involved” in the crash but weren’t necessarily killed.) We write code that looks like this:

m_breaks <- cumsum(as.integer(diff(seq(as.Date("2016-01-01"),
                                       as.Date("2017-01-01"), by = "month")))) - 16  

arrow_segment_df <- tibble(
  x = 10, y = 3.4, xend = 61, yend = 3.2)
  

p_out  <- ggplot(data = fars_involved_agg,
       mapping = aes(x = day_ind, y = n, color = flag, fill = flag)) +
    geom_point(group = 1, size = 2.5, shape = 21) +
    geom_textsegment(
      data = arrow_segment_df, 
      mapping = aes(x = x, y = y, xend = xend, yend = yend), 
      label = "Calendar Day", color = "cornflowerblue",
      family = "Socviz Condensed",
      arrow = arrow(length = unit(0.2, "cm"), type = "closed"), 
      inherit.aes = FALSE, linewidth = 0.5) + 
    annotate("text", x = 305, 
                y = 4, label = "Halloween", size=5, hjust = -0.12,
                color="darkorange2", family = "Socviz Condensed",
             ) + 
    scale_color_manual(values = c("gray10", "gray5")) +    
    scale_fill_manual(values = c(NA, "darkorange2")) + 
    coord_radial(expand = FALSE, rlim = c(0,4.25), inner.radius = 0.25, 
                 r.axis.inside = TRUE) + 
    scale_x_continuous(breaks = m_breaks, labels = month.name, minor_breaks = m_breaks - 15) +
    guides(color = "none", fill = "none", theta = "axis_textpath") + 
    labs(x = NULL,
         y = NULL,
         title = "Fatal Motor Vehicle Crashes involving Child Pedestrians",
         subtitle = "Daily Means, 2009-2023",
         caption = "Figure: Kieran Healy / Data: NHTSA Fatality Analysis Reporting System") + 
  theme(panel.grid.major.x = element_blank(),
        panel.grid.minor.x = element_line(color = "gray10"),
        panel.grid.major.y = element_line(color = "gray50"),
        panel.grid.minor.y = element_blank(),
        axis.text = element_text(face = "bold"), 
        axis.ticks.theta = element_blank())

This is well beyond the minimum necessary to get a servicable plot, but I got carried away polishing the thing. The key bit is the call to coord_radial():

coord_radial(expand = FALSE, 
             rlim = c(0,4.25), 
             r.axis.inside = TRUE, 
             inner.radius = 0.25)

This says: draw the x-axis as a circle; don’t expand the margins (this has the effect of keeping January at the top rather than being slightly decentered); set the y-axis limits (i.e. the length of the radius) to be slightly more than the maximum of the data; put the radial axis inside the plot rather than next to it; and make the circle more like a donut by putting a circular “hole” in the middle of the graph that’s 25 percent of the total length of the radius. Donut-style versions of polar plots are often easier to read, as opposed to having everything go into the very center of the circle.

The word “Halloween” is put in with annotate(). The call to geom_textsegment() is from the very handy geomtextpath package. This lets us put text or labels on or alongside lines in a way that follows their shape. You can do this for ordinary trend lines in xy plots but it understands polar coordinates too. In addition, ggplot 4’s radial system knows about geomtextpath. If you’re using radial coordinates and geomtextpath is loaded then you can write guides(theta = "axis_textpath"). This will change the labels of the theta axis (which is to say, the polar-transformed x-axis). Instead of sticking out horizontally they will pleasingly follow the path of the circle. Nice. We also use the distinction between minor and major breakpoints for the labels and panel grid lines to put the month names on the minor breaks and leave the major breaks empty (and turn off their tick marks). That puts the month names in the middle of each monthly wedge, which is better than having them at the beginning. The only other slightly fancy thing is deliberately picking a shape for the points that lets me have them be empty rings, except for the Halloween point, which is filled. Here’s what we get.

Roundy version.

Not bad. You have to be careful with polar coordinates. People are much better at judging relative lengths than relative angles. This is why pie charts are hated by dataviz nerds. A pie chart is, after all, just a bar or column chart whose x-axis has been spun into a circle. In this case, our humble wide-aspect column plot does perfectly well for much less effort, and has the advantage of being immediately comprehensible by more people. There are cases where truly seasonal data can really benefit from being represented on or in a circle. This data isn’t quite one of those, as we’re just saying “Hey, look how this one day is different from all the others”. But by the same token, we’re not really asking the viewer to judge angular offsets or to estimate the area of a pie wedge. Instead it’s just distance from the inner circle. Seeing the points for all the other days cluster close to the inner ring while Halloween stands out a ways does a decent job of getting the content across, and maybe in a way friendlier to people looking at the plot on a phone.

PS: One other thing. Remember, as we also noted last year, that any inference about how dangerous Halloween is for child pedestrians shouldn’t depend just on the number of fatalities observed but also on the exposure, which we don’t see directly. There are lots more children—perhaps orders of magnitude more—wandering around on the evening of Halloween than on a typical night, which will change the meaning of the count of pedestrian fatalities.

Iterating some sample data

Fri, 03 Oct 2025 05:32:15 -0400

I’m teaching my Modern Plain Text Computing course this semester and so I’m on the lookout for small examples that I can use to show some of the ordinary techniques we regularly use when working with tables of data. One of those is just coming up with some example data to illustrate something else, like how to draw a plot or fit a model or what have you. This is partly what the stock datasets that come bundled with packages are for, like the venerable mtcars or the more recent palmerpenguins. Sometimes, though, you end up quickly making up an example yourself. This can be a good way to practice stuff that computers are good at, like doing things repeatedly.

This happened the other day in response to a question about visualizing some evaluation data. The task goes like this. You are testing a bunch of different LLMs. Say, fifteen of them. You have trained them to return Yes/No answers when they look at repeated samples of some test data. Let’s say each LLM is asked a hundred questions. You have also had an expert person look at the same hundred questions and give you their Yes/No answers. The person’s answers are the ground truth. You want to know how the LLM performs against them. So for each LLM you have a two-by-two table showing counts or rates of true positives, true negatives, false positives, and false negatives. (This is called a “confusion matrix”.) You want to visualize LLM performance for all the LLMs. An additional wrinkle is that, from the point of view of your business, responses are variably costly. Correct answers (true positives or true negatives) cost one unit. Then, say, a False Negative costs two units and a False Positive is worst, costing four units.

The questioner wanted some thoughts on what sort of graph to draw. You can of course just picture what the data would look like and figure out which of your many stock datasets has an analogous structure. Or you’d sketch out an answer with pen and paper. In this case, even though they have problems in general, I thought a kind of stacked bar chart (but flipped on its side) might work. OK, done. But half the fun—for some values of “fun”—is generating data that looks like this. And as I said, I’m on the lookout for data-related examples of iteration, i.e. where I repeatedly do something and gather the results into a nice table.

When we want to repeatedly do something, we first solve the base case and then we generalize it by putting in some sort of placeholder and use an engine that can iterate over an index of values, feeding each one to the placeholder. In imperative languages you might use a counter and a for loop. In a functional approach you map or apply some function.

We’ve got a hundred questions and fifteen LLMs. We imagine that the LLMs can range in accuracy from 40 percent to 99 percent in one percent steps. We’ll pick at random from within this range to set how good any specific LLM is.

set.seed(100125) # so we get the same 'random' results each time

n_runs <- 100
n_llms <- 15
accuracy_range <- seq(0.4, 0.99, 0.01)

Our baseline is n_runs human answers with some given distribution of Yes/No answers. Let’s say 80% No, 20% Yes. It doesn’t matter what they are; the person is the ground truth. We sample with replacement a hundred times from “N” or “Y” at that probability.

human_evals <- sample(c("N", "Y"), n_runs, replace = TRUE, prob = c(0.8, 0.20))

human_evals
#>   [1] "N" "N" "N" "Y" "N" "Y" "N" "N" "N" "Y" "N" "Y" "N" "N" "N" "Y" "N" "N" "N" "N" "N" "N" "N" "N" "Y" "N" "Y" "N" "N" "N" "Y" "N" "N"
#>  [34] "N" "N" "N" "N" "N" "N" "Y" "N" "N" "N" "Y" "N" "N" "Y" "N" "Y" "N" "N" "N" "N" "N" "N" "N" "N" "N" "N" "N" "N" "Y" "N" "N" "N" "N"
#>  [67] "N" "N" "N" "N" "N" "N" "N" "N" "N" "Y" "N" "Y" "N" "N" "Y" "Y" "N" "Y" "Y" "N" "Y" "N" "N" "N" "N" "N" "N" "N" "N" "N" "N" "N" "N"
#> [100] "N"

For each of our fifteen LLM what we want to do is generate a string of its one hundred Y/N answers in the same way, but with its particular idiosyncratic distribution of Ys and Ns, and then evaluate it against the human baseline. And we’d like to gather all the answers into a single data frame so we can keep everything tidy.

Our evaluation function looks like this:

eval_llm <- function(llm_evals, human_eval) {
  case_when(
    llm_evals == "Y" & human_eval == "Y" ~ "True Positive",
    llm_evals == "Y" & human_eval == "N" ~ "False Positive",
    llm_evals == "N" & human_eval == "Y" ~ "False Negative",
    llm_evals == "N" & human_eval == "N" ~ "True Negative",
  )
}

We generate a string of responses for an imaginary LLM just like we did for the imaginary person. The single case would look like this:

`1`	`llm_01 <- sample(c("N", "Y"), n_runs, replace = TRUE, prob = c(0.7, 0.3))`

Which we would then feed to eval_llm() along with the vector of human answers. But we want to do this fifteen times, with varying values for prob and also we want to put each LLM in its own column in a data frame. So we replace the values with variables. Then we evaluate all of them.

First we generate a vector of LLM names. We use str_pad to get sortable numbers with a leading zero:

llm_id <- paste("LLM", str_pad(1:n_llms, width = 2, pad = "0"), sep = "_")

llm_id
#>  [1] "LLM_01" "LLM_02" "LLM_03" "LLM_04" "LLM_05" "LLM_06" "LLM_07" "LLM_08" "LLM_09" "LLM_10" "LLM_11" "LLM_12" "LLM_13" "LLM_14" "LLM_15"

Now we’re going to create a little table of LLM parameters. We already created a vector of probabilities for the LLMs, accuracy_range. We’ll sample from that to get fifteen values. The number of runs is fixed at a hundred. R’s naturally vectorized way of working will take care of the table getting filled in properly.

llm_df <- tibble(
  llm_id = llm_id,
  p_yes = sample(accuracy_range, n_llms),
  p_no = 1 - p_yes,
  n = n_runs
)


llm_df
#> # A tibble: 15 × 4
#>    llm_id p_yes   p_no     n
#>    <chr>  <dbl>  <dbl> <dbl>
#>  1 LLM_01  0.49 0.51     100
#>  2 LLM_02  0.91 0.09     100
#>  3 LLM_03  0.46 0.54     100
#>  4 LLM_04  0.88 0.12     100
#>  5 LLM_05  0.45 0.55     100
#>  6 LLM_06  0.52 0.48     100
#>  7 LLM_07  0.55 0.45     100
#>  8 LLM_08  0.6  0.4      100
#>  9 LLM_09  0.65 0.35     100
#> 10 LLM_10  0.61 0.39     100
#> 11 LLM_11  0.93 0.0700   100
#> 12 LLM_12  0.53 0.47     100
#> 13 LLM_13  0.94 0.0600   100
#> 14 LLM_14  0.69 0.31     100
#> 15 LLM_15  0.92 0.0800   100

Next we need a function that can accept each row as a series of arguments and use it to generate a vector of LLM answers:

run_llm <- function(llm_id, p_yes, p_no, n_runs) {
  tibble(
    {{ llm_id }} := sample(c("N", "Y"), n_runs, replace = TRUE, prob = c(p_yes, p_no))
  )
}

This function returns a data frame that has one column and a hundred rows of Y/N answers sampled at a given probability of yes and no answers. There are two tricks. The first is, we want the name of the column to be the same as the llm_id. To do this we have to quasi-quote the llm_id argument. This is what the {{ }} around llm_id does inside the function. It lets us use the value of llm_id as a symbol that’ll name the column. Normally when using tibble() to make a data frame we create a column with col_name = vector_of_values. We did that when we made llm_df a minute ago. But because we’re quasi-quoting the LLM name on the left side of a naming operation, for technical reasons having to do with how R evaluates environments we can’t use = as normal. Instead we have to assign the name’s contents using the excellently-named walrus operator, :=. If we were quasi-quoting with {{ }} on the right-hand side, an = would be fine.

Now we’re ready to go. We feed the llm_df table a row at a time to the run_llm function by using one of purrr’s map functions. Specifically, we use pmap, which takes a list of multiple function arguments and hands them to a function. We have written our llm_df columns so that the columns are named and ordered the way that our run_llm() function expects, so it’s nice and compact.

llm_outputs <- pmap(
  as.list(llm_df),
  run_llm
) |> 
  bind_cols()

llm_outputs
#> # A tibble: 100 × 15
#>    LLM_01 LLM_02 LLM_03 LLM_04 LLM_05 LLM_06 LLM_07 LLM_08 LLM_09 LLM_10 LLM_11 LLM_12 LLM_13 LLM_14 LLM_15
#>    <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr> 
#>  1 N      N      N      N      Y      N      Y      Y      N      N      N      N      N      N      N     
#>  2 N      N      N      N      N      N      N      N      N      N      N      Y      N      N      Y     
#>  3 N      N      N      N      N      N      N      N      N      N      N      N      Y      N      Y     
#>  4 N      Y      N      N      N      N      N      N      Y      N      N      N      N      N      Y     
#>  5 N      N      N      Y      Y      N      Y      N      Y      Y      N      N      N      N      Y     
#>  6 N      N      N      Y      Y      N      N      Y      N      N      N      N      Y      Y      N     
#>  7 N      Y      N      N      Y      Y      Y      Y      Y      N      N      N      N      Y      N     
#>  8 N      N      N      N      N      N      N      Y      Y      N      N      N      N      N      Y     
#>  9 N      N      N      N      N      N      Y      Y      Y      N      N      N      N      N      N     
#> 10 N      N      Y      N      Y      Y      N      Y      Y      N      N      N      N      N      Y     
#> # ℹ 90 more rows

We write as.list(llm_df) because pmap() wants its series of arguments as a list. (A data frame is just a list where each list element—each column—is the same length, by the way.) It returns a list of fifteen LLM runs, which we then bind by column back into a data frame. Nice.

Now we can evaluate all these LLMs against our human_evals vector in the same way, by mapping or applying the eval_llm() function we wrote earlier. This time we can just use regular map() because there’s only one varying argument, the LLM id. We take the llm_outputs data frame and use an anonymous or lambda function, \(x) to say “pass each column to eval_llm() along with the non-varying human_evals vector”. (You could write this without a lambda, too, but I find this syntax more consistent.) At the end there I deliberately convert all these character vectors to factors for a reason I’ll get to momentarily.

llm_results <- map(llm_outputs, \(x) eval_llm(x, human_evals)) |>
  bind_cols() |> 
  mutate(
    across(everything(), as.factor))

llm_results
#> # A tibble: 100 × 15
#>    LLM_01         LLM_02         LLM_03         LLM_04         LLM_05  LLM_06 LLM_07 LLM_08 LLM_09 LLM_10 LLM_11 LLM_12 LLM_13 LLM_14 LLM_15
#>    <fct>          <fct>          <fct>          <fct>          <fct>   <fct>  <fct>  <fct>  <fct>  <fct>  <fct>  <fct>  <fct>  <fct>  <fct> 
#>  1 False Positive False Positive False Positive True Negative  True N… True … False… True … True … True … True … True … False… True … True …
#>  2 True Positive  False Negative False Negative False Negative False … False… False… True … False… False… False… False… False… True … False…
#>  3 False Negative True Positive  False Negative False Negative False … False… True … False… False… False… False… False… False… True … False…
#>  4 False Positive False Positive True Negative  True Negative  True N… True … True … True … True … True … True … True … True … False… True …
#>  5 True Negative  True Negative  True Negative  True Negative  False … False… False… True … False… True … True … True … True … False… False…
#>  6 False Positive True Negative  True Negative  False Positive False … False… True … False… False… True … True … True … False… True … True …
#>  7 False Positive False Positive True Negative  False Positive True N… True … False… False… False… True … True … True … True … True … True …
#>  8 False Negative True Positive  False Negative False Negative True P… False… False… False… False… False… False… False… False… True … True …
#>  9 False Negative True Positive  False Negative True Positive  True P… False… True … False… True … False… False… False… False… False… False…
#> 10 True Negative  True Negative  False Positive True Negative  True N… True … False… False… False… True … True … True … False… False… False…
#> # ℹ 90 more rows

Now we’re done; each LLM has been compared to the ground truth and we can construct a confusion matrix of counts for each column if we want. Let’s summarize the table, adding a cost code for the bad answers:

llm_summary <- llm_results |>
  pivot_longer(everything()) |>
  group_by(name, value, .drop = FALSE) |>
  tally() |>
  mutate(prop = n/sum(n),
         cost = case_when(
           value %in% c("True Positive", "True Negative") ~ 1L,
           value == "False Negative" ~ 2L,
           value == "False Positive" ~ 4L)
  ) 

llm_summary 
#> # A tibble: 60 × 5
#> # Groups:   name [15]
#>    name   value              n  prop  cost
#>    <chr>  <fct>          <int> <dbl> <int>
#>  1 LLM_01 False Negative    18  0.18     2
#>  2 LLM_01 False Positive    19  0.19     4
#>  3 LLM_01 True Negative     59  0.59     1
#>  4 LLM_01 True Positive      4  0.04     1
#>  5 LLM_02 False Negative    16  0.16     2
#>  6 LLM_02 False Positive    22  0.22     4
#>  7 LLM_02 True Negative     56  0.56     1
#>  8 LLM_02 True Positive      6  0.06     1
#>  9 LLM_03 False Negative    19  0.19     2
#> 10 LLM_03 False Positive    17  0.17     4
#> # ℹ 50 more rows

Now, why did I convert the LLM results table from characters to factors? It’s because of how dplyr handles table summaries. It’s possible that an LLM could get e.g. all True Positive answers, leaving the other three cells in its confusion matrix empty, i.e. with zero counts in those rows. By default, when tallying counts of character vectors, dplyr drops empty groups. For some kinds of tallying that’s fine, but for others you definitely want to keep a tally of zero-count cells. With factors we can tell dplyr explicitly not to drop them. (You can also set this option permanently for a given analysis.) The alternative is to ungroup and complete the table once its been created, explicitly adding back in the implicitly missing zero-count rows.

Now that we have our table, we can graph it. As I said at the beginning, stacked bar charts are not great in many cases but it’s fine here, and better than trying to repeatedly draw fifteen confusion matrices. We don’t really care about the difference between true positives and true negatives anyway. We take the results table, merge it with the summary table, and draw our graph ordering the LLMs by performance weighted by average cost. We use a manual four-value color palette to distinguish the broadly bad from the broadly good answers.

llm_results |>
  pivot_longer(everything()) |>
  left_join(llm_summary) |>
  mutate(name = str_replace(name, "_", " ")) |>
  ggplot(aes(y = reorder(name, cost), fill = value)) +
  geom_bar(color = "white", linewidth = 0.25) +
  scale_fill_manual(values = fourval_pal) +
  guides(fill = guide_legend(reverse = TRUE,
                             title.position = "top",
                             label.position = "bottom",
                             keywidth = 3,
                             nrow = 1)) +
  labs(x = "N Questions", y = NULL,
       fill = "Compared with a Person the LLM Yielded ...",
       title = "LLM Performance Relative to Human Baseline",
       subtitle = "False Positives are twice as costly as False Negatives") +
  theme_minimal() +
  theme(legend.position = "top")

Ordered and stacked bar chart of imaginary LLM performance.

The Road to Selfdom

Thu, 21 Aug 2025 07:38:35 -0400

Marion and I have an essay in Aeon today:

What is happening here is more than an abstract flow of information. It is more than a means of surveillance. It is more than a price mechanism. Rather, it’s as if the air traffic control and insurance commission functions of the IBM 650 have been fused, shrunk, and wholly generalised. This is the real computing revolution. Much of what we do is immediately authenticated as we do it, stored as data, classified or scored on some sort of scale, and deployed in real time to modulate some outcome of interest – usually, the behaviour of a person, or a machine, or an organisation. … Because of this transformation, our sense of who we are is assembled in a strange and tangled fashion. The machinery of ordinalisation attends carefully to individuals rather than coarse classes or groups. By doing so, it appears to liberate people from the constraints of social affiliations and to judge them for their distinctive qualities and contributions. It promises incorporation for the excluded, recognition for the creative, and just rewards for the entrepreneurial. And yet this emancipatory promise is delivered through systems that classify, sort and, above all, rank people with ever-greater precision and on a previously unimaginable scale. The resulting social order is a sort of paradox, characterised by constant tensions between personal freedom and social control, between the subjective elan of inner authenticity and the objective forces of external authentication. It gives rise to a certain way of being, a new kind of self, whose experiences are defined by the push for personal autonomy and the pull of platform dependency.

You should go check it out.

Blueberry Hill

Thu, 07 Aug 2025 20:07:52 -0400

ChatGPT 5 was released today.

ChatGPT-maker OpenAI has unveiled the long-awaited latest version of its artificial intelligence (AI) chatbot, GPT-5, saying it can provide PhD-level expertise. Billed as “smarter, faster, and more useful,” OpenAI co-founder and chief executive Sam Altman lauded the company’s new model as ushering in a new era of ChatGPT. “I think having something like GPT-5 would be pretty much unimaginable at any previous time in human history,” he said ahead of Thursday’s launch. GPT-5’s release and claims of its “PhD-level” abilities in areas such as coding and writing come as tech firms continue to compete to have the most advanced AI chatbot.

I had a chat with it. Yes I know the questions surrounding AI are tricky. But I do not pretend to address those here; I merely report the following conversation.

The thesis outlined.

The thesis patiently explained, defended in the face of objections, and further elaborated.

The collegial chat and a final effort at rebuttal.

In fairness to GPT5, in my career I have indeed encountered PhDs with this level of commitment to their particular blueberry. And many have also had that blithe confidence — the use of “Ah”, the “Let’s slow it down” (to your two-B level), the “Exactly” (Now you see my genius), the confidently colloquial “Yep” and “Nope” … actually I retract my earlier skepticism; the lad has the makings of a fine philosopher.

Very quick update: Posting this on social media led to some interesting responses. While certainly well-meant, I’m afraid I can’t say I am especially impressed by helpful suggestions that I spend time to do a little prompt engineering in order to get GPT5 to emit the right answer to the question “How many times does the letter b appear in blueberry”. Meanwhile, my excellent student Andrés got a different, and correct, response from his attempt. So score one for replicability, I guess.

Update 2: Look, these tools are amazing in lots of ways. But if you’re gonna pitch and sell—or foist—them on the world at large, or your employees, by saying “This is your PhD level genius expert buddy you can ask things in natural language”, don’t get all pissy when it comically bombs tasks people reasonably think of as trivial.

Update 3, now in meme form:

I don’t think you get to have it both ways. That is, you don’t get to, as it were, borrow charisma from all the hype and then disavow every failure to live up to it as someone else’s naive mistake for believing the hype.

The Sound of Silence

Tue, 22 Jul 2025 07:36:20 -0400

A GitHub Issue on OpenAI’s Whisper, which is a good speech-recognition and transcription model with support for a large number of languages. A lot of people use it. The issue:

Complete silence is always hallucinated as “ترجمة نانسي قنقر” in Arabic which translates as “Translation by Nancy Qunqar”

In the comments, people note that this class of error has been known for a while and there are equivalents or counterparts in other languages:

I found a similar thing happens in German where it says “Untertitelung des ZDF für funk, 2017.”

That would be “Subtitling by ZDF”. Although another commenter notes “In german it’s ‘Vielen Dank’ (Thank you very much)”. In Romanian,

i’ve noticed multiple instances where the transcripts ends with “nu uitati sa da-ti like si subscribe” which, as you might easily infer , translates to “don’t forget to like and subscribe”.

You can see what’s happening. The model learns that silence is the end of the recording. As KillerX says,

this seems to be an artifact of the fact that Whisper was trained on (amongst other things) YouTube audio + available subtitles. Often subtitlers add their copyright notice onto the end of the subtitles, and the end of the videos are often credits with music, applause, or silence. Thus whisper learned that silence == ‘copyright notice’."

Amusingly,

In English there is always applause