Blogs on kieranhealy.org

Characteristic NYC Neighborhood Complaints

Tue, 27 Oct 2026 06:51:50 -0400

New York City month continues here at Kieran Healy Industries, an effort that will eventually produce something properly scholarly in nature. But in the meantime—and with my apologies to the many great urbanists of times past—we can enjoy ourselves by messing around with some data. Specifically, New York’s 311 Services and Complaints data. The 311 phone line and website is the the main point of contact for city services when it’s not an emergency. The city logs all the calls it gets and makes anonymized data available publicly via NYC OpenData. I got every record from 2010 to 2025 and packaged it up for R. The package is really just a front-end for a big parquet file managed by DuckDB. DuckDB offers a very powerful in-memory database engine that is optimized for the kinds of thing we want to do in data analysis. That is, we want to efficiently query a big dataset lots of different ways, rather than, say, repeatedly update its records as fast as possible. We can work in R and write tidy dplyr-like queries that get handled by the database. It’s great.

The dataset has about 43 million rows, each one of which is a service request (i.e., a complaint). I interact with it via a function that returns a DuckDB table. It looks like this:

nyc_311()
#> # A duckplyr data frame: 37 variables
#>    unique_key created_date        closed_date         agency agency_name   problem problem_detail location_type incident_zip incident_address street_name cross_street_1 cross_street_2 address_type city  status
#>         <dbl> <dttm>              <dttm>              <chr>  <chr>         <chr>   <chr>          <chr>         <chr>        <chr>            <chr>       <chr>          <chr>          <chr>        <chr> <chr> 
#>  1   45278137 2019-12-31 23:59:11 2020-01-01 03:07:18 NYPD   New York Cit… Noise … Banging/Pound… Club/Bar/Res… 11229        3939 EMMONS AVE… EMMONS AVE… PLUMB    3 ST… BELT PARKWAY … <NA>         BROO… Closed
#>  2   45281589 2019-12-31 23:58:54 2020-01-01 05:37:03 NYPD   New York Cit… Noise … Loud Music/Pa… Residential … 10025        865 COLUMBUS AV… COLUMBUS A… WEST  100 STR… WEST  103 STR… <NA>         NEW … Closed
#>  3   45283574 2019-12-31 23:58:03 2020-01-01 01:26:50 NYPD   New York Cit… Noise … Loud Music/Pa… Residential … 10040        558 WEST  192 S… WEST  192 … AUDUBON AVENUE ST NICHOLAS A… <NA>         NEW … Closed
#>  4   45278249 2019-12-31 23:57:19 2020-01-01 09:33:55 NYPD   New York Cit… Noise … Loud Music/Pa… Residential … 11372        35-18 91 STREET  91 STREET   35 AVENUE      37 AVENUE      <NA>         JACK… Closed
#>  5   45282332 2019-12-31 23:56:52 2020-01-01 03:06:36 NYPD   New York Cit… Blocke… No Access      Street/Sidew… 11226        2310 CORTELYOU … CORTELYOU … EAST   23 STR… BEDFORD AVENUE <NA>         BROO… Closed
#>  6   45281555 2019-12-31 23:56:34 2020-01-01 08:22:26 NYPD   New York Cit… Noise … Loud Music/Pa… Residential … 11375        72-72 112 STREET 112 STREET  72 DRIVE       73 AVENUE      <NA>         FORE… Closed
#>  7   45281053 2019-12-31 23:56:34 2020-01-01 02:47:53 NYPD   New York Cit… Noise … Loud Music/Pa… Street/Sidew… 10128        EAST   89 STREET EAST   89 … 2 AVENUE       1 AVENUE       <NA>         <NA>  Closed
#>  8   45282921 2019-12-31 23:56:32 2020-01-01 02:18:56 NYPD   New York Cit… Noise … Loud Music/Pa… Residential … 11433        168-70 93 AVENUE 93 AVENUE   168 PLACE      170 STREET     <NA>         JAMA… Closed
#>  9   45283114 2019-12-31 23:56:28 2020-01-01 05:58:35 NYPD   New York Cit… Noise … Loud Music/Pa… Residential … 11373        42-21 KETCHAM S… KETCHAM ST… BAXTER AVENUE  PETTIT AVENUE  <NA>         ELMH… Closed
#> 10   45281055 2019-12-31 23:55:56 2020-01-01 11:01:07 NYPD   New York Cit… Noise … Loud Talking   Street/Sidew… 11435        85-85 148 STREET 148 STREET  85 DRIVE       86 AVENUE      <NA>         JAMA… Closed
#> # ℹ more rows
#> # ℹ 21 more variables: resolution_description <chr>, resolution_action_updated_date <dttm>, community_board <chr>, police_precinct <int>, bbl <chr>, borough <chr>, open_data_channel_type <chr>,
#> #   park_facility_name <chr>, park_borough <chr>, latitude <dbl>, longitude <dbl>, created_year <dbl>, created_month <dbl>, created_day <dbl>, created_dow <chr>, created_hour <dbl>, closed_year <dbl>,
#> #   closed_month <dbl>, closed_day <dbl>, closed_dow <chr>, closed_hour <dbl>

We can ask all kinds of questions of data like this. We also have to be careful with it, just because any dataset that’s this large and has grown over this long a period will have all kinds of quirks in it. For example, the classification scheme the city uses to categorize service requests isn’t static, and for some categories it gets completely redone. At least once over the period the city changed the service provider for data management, which affected measurement in some ways. Meanwhile, on the complaints side, one feature of service databases like this is that some people really like to complain, even by the high standards of New Yorkers. Complaining also becomes easier after the introduction of the web portal, because you can just quickly report something via a form rather than having to make a phone call. So we need to think about how to deal with things like that.

Complaints are ultimately made by individual people, and our dataset is a record of individual reports. But here I’m going to aggregate things to the level of neighborhoods. You can slice up the city in many different ways. New York often uses Neighborhood Tabulation Areas, or NTAs, as its de facto neighborhood unit. The idea is to get something that’s larger than a Census Tract, of which there are about 2,300 in New York, but smaller than one of the City’s 59 Community Districts. There are 262 NTAs, of which 197 are residential. Here’s a map of them, with their names.

So, how can we get from our complaint records to an NTA-level map, and what do we want to see? As I say, here we’re going more or less for data-driven neighborhood trolling, so here’s what I did to characterize each neighborhood. First, I did a fair amount of work harmonizing the complaint categories over the whole dataset, but in addition to that here I’ll just restrict myself to complaints from 2022 to 2025, which is about 13.5m records. Second, we want to moderate the role played by people we can politely call “super-reporters”, so we count distinct addresses within complaint categories, within residential NTAs, over the time period we’re looking at. Each address counts only once in each category it complained about, so we get about 4.6m address-category pairs from about one million unique addresses. (The addresses don’t include identifying information like house or apartment numbers, by the way.) The average address complains in about four or five categories over the four years we’re looking at.

Now, when it comes to complaints, we are all trying to get through this thing we call life and many of the trials we face, we all share. Which is to say that the most common complaints in any given neighborhood are more likely to be the most common complaints in other neighborhoods, too. These are things like the heat and hot water being out, noise of various kinds, and illegal parking. But what we’re interested in is the most distinctive complaints in each neighborhood. So we take every complaint and figure out its share of all complaints citywide. Then we do the same but within each neighborhood. And then for each neighborhood we ask, what category of complaint in this neighborhood most exceeds its citywide share? Actually we do two things. We do that first by general category of complaint, like “Sanitation and Dumping”, or “Parking, Driveways, and Vehicles”, or “Buildings and Construction”. And then we do it for the most fine-grained category we have, like “Traffic: Drag Racing”, or “Appliance: Refrigerator”, or “Lost Property in Taxi or For-Hire Vehicle: Wallet”. Then we color our NTA map in by the broad category and label each neighborhood by its most-specific category. Here’s what that looks like.

I think the results are pretty good. Sorry about the small print and fine detail. This is the sort of thing that works best as a poster. But zoom in and take a look. And of course Tag Yourself. Are you the Badly Behaved Nightlife Crescent from the Upper East Side to Lower Manhattan who got drunk and lost your wallet or keys? (I like how right at the top of the UES they don’t lose their wallets but they do complain about the driver.) Are you the brownstone Brooklyn Bicycle Police? Or the South Brooklyn Curb Cutters who are installing illegal carports? Or maybe you are just out in Woodside, Elmhurst, or Corona, drinking in public. Or abandoning a vehicle somewhere. There’s a lot going on in the city.

Bear in mind that there’s a kind of duality to the map, in the sense that at first glance it shows you what seems to be happening in these NTAs, but only through the lens of what people complain about in these neighborhoods. So, indulge your neighborhood stereotypes, but maybe don’t lean too hard on them. Though I do like those people in Forest Hills who just call the cops for unspecified Other Non-Emergency Police Matters. You know, for a chat or something. What else are you gonna do out there on a Friday?

Dogs of New York, Shapes and Sizes Edition

Sun, 25 Oct 2026 13:57:06 -0400

My recent Dogs of New York, post got circulated a little. On Friday I got a very nice note from William Kornblum, an urban and community studies sociologist who these days is emeritus at CUNY. In the 1980s, Bill studied the people who use Central Park and wrote a report for the Central Park Conservancy. Some of that work counted dog-walkers.

He reminded me that parks in New York City see a lot of dogs. NYC Parks estimates about 1.5 million canine visits to Central Park each year. That is about 4,100 visits a day, to one park. He also asked:

Is there a size gradient perhaps? Larger dogs out on the periphery where there is more room? Smaller dogs in the dense apartment neighborhoods?

This is a good question. The data I have can almost answer it. The main problem is that the license data contained in nycdogs does not record the size or weight of any particular dog. Licenses do record the breed that the owner reports (though, as we saw before the modal breed reported is “Unknown”). But let’s see how far we can get by leveraging our knowledge of breeds.

To begin we need to give each breed a size. For this I used the American Kennel Club’s breed weights. The AKC lists a weight range for each breed it recognizes, from its breed standards. Tiago Filho collected these into one table for 277 breeds in 2020. I had to make a few adjustments to reconcile it with the breeds recorded in nycdogs, most notably for two popular city breeds. First, the AKC lists one Dachshund range that covers both sizes within this breed. Owners often record “Miniature” in the breed name, so I gave those dogs the AKC miniature limit of 11 pounds. Second, Pit Bulls are among the most common dogs in the city, but to my knowledge the AKC does not register them. I used the United Kennel Club’s American Pit Bull Terrier standard instead.

I assigned each breed a “typical weight”, which is the middle of its range. Some standards give only one figure, such as “not more than 6 pounds” for a Chihuahua. In those cases I used that number. This isn’t ideal. A breed standard describes a show dog, not the dog you see in the park. (Though, of course, in some NYC parks those might be the same. But not in general.) Real dogs will be heavier or lighter than their standard. Still, a Great Dane is reliably bigger than a Yorkie, and it’s the relative differences we’re interested in.

The second problem is that many dogs have no breed we can use. As in the earlier post, we’re working with a subset of the nyc_license table that’s our best-guess approximation of a “dog-level” data set, one that cuts down as much as possible on double-counting and just has one record per actual dog. This table consists of about 358,000 dogs. I can only assign a breed size to about 60 percent of those:

About 22 percent are crossbreeds, such as “Labrador Retriever Crossbreed”. A mix could be any size. I chose not to guess.
About 10 percent have no breed recorded, mostly “Unknown”.
About 6 percent are “designer” breeds, such as the up-and-coming Goldendoodles, Maltipoos, and Morkies. No kennel club sets a standard for these.
About 1.5 percent are Poodles with no size given. A Toy Poodle and a Standard Poodle differ by about 50 pounds, so again, I chose to drop these.

We end up, for this analysis, with about 215,000 dogs. Now, if the missing dogs were spread evenly across the city, the missingness would not matter all that much. We can try to check that. Here is a map showing the share of dogs in each ZIP code that have a breed size.

Coverage runs from about half to about two-thirds of dogs in each ZIP code. It is highest on the West Side of Manhattan and in parts of eastern Queens. It is lowest in central Brooklyn. So, dogs are not MCAR in the city. The maps and analysis below is restricted to describing dogs with a known breed, and that group is not quite the same in every part of the city.

Where the big dogs are

For each ZIP code with at least 50 dogs of known size, I calculated the median typical weight. (As before, I use the city’s MODZCTAs, which are ZIP code areas with some small ones merged.)

I didn’t try to make the weight categories too fine-grained. Again, the weights are derived from the breed standard, not individual dogs, and so the distribution is quite lumpy. Each dog is counted as having the typical weight of its breed. Very popular breeds therefore pile up on single values. Many ZIP codes have a median of exactly 12.5 pounds, for example. (That’s the Shih Tzu’s weight.)

Even so, we still see a clear pattern. The ZIP codes with the biggest dogs are in lower Manhattan and brownstone Brooklyn. Park Slope is at the top, with a median of 29 pounds. Woodlawn and City Island in the Bronx, and the Rockaways in Queens, also have big dogs. The smallest dogs are in most of the Bronx, southeast Queens, and East New York and Brownsville in Brooklyn. There, the median dog weighs 12.5 pounds.

So Bill’s idea is partly supported by the evidence. Some of the city’s edges, where people have more room on average, do have bigger dogs. But, on the other hand, some of the densest apartment neighborhoods in the city have the biggest dogs of all.

Size classes

A median can hide a lot. I sorted the breeds into five size classes. I used the weight bands that you see on bags of dog food: Toy (up to about 9 pounds), Small (9 to 22), Medium (22 to 55), Large (55 to 99), and Giant (over 99). This is kind of arbitrary, and not even arbitrary in the “AKC Standard” sense of arbitrary. But I don’t think it’s unreasonable.

Across the city, here is how the dogs with a known size divide up:

About a quarter are Toy dogs.
About a third are Small.
About a quarter are Medium.
14 percent are Large.
Fewer than 2 percent are Giant.

Here is the share of each class within each ZIP code.

Toy dogs are most common in the Bronx and in parts of Brooklyn and Queens. Medium dogs cluster in somewhat fancier parts of Brooklyn and lower Manhattan. Large dogs are most common on Staten Island, in the Rockaways, and (again) in parts of Manhattan. Giant dogs are rare everywhere.

The same numbers aggregated up to the level of boroughs are a little easier to compare:

Small dogs are about a third of all licensed dogs in every borough. The differences are at the ends. A third of the Bronx’s dogs are Toy dogs, compared with about a fifth of Manhattan’s. Only 9 percent of the Bronx’s dogs are Large, compared with 16 percent in Manhattan and 17 percent on Staten Island. Staten Island, the least densely-populated borough, has the largest share of big dogs, but only by a percentage point more than Manhattan.

Mo Money, Mo Doggo

Manhattan and brownstone Brooklyn are dense, but they are also rich. The Bronx is dense too, but much poorer. Happily, thanks to both NYC OpenData and the Census Bureau, we have relevant housing and income measures available that we can construct for each ZIP code.

Floor area per home. This comes from PLUTO, the city’s database of every tax lot. I added up the residential floor area and the number of homes on every lot in each ZIP code, and divided. This is gross floor area, so it includes halls and stairs. It’s a rough measure of apartment size. You can use my nycpluto package to access these data in R.
Percent of homes in one- or two-family houses. This is also from PLUTO, using each lot’s building class.
Median rooms per home, percent of homes that are owner-occupied, and median household income. These come from the Census Bureau’s American Community Survey, using the five-year estimates for 2020 to 2024. You can query the Census API directly for these, or use my nycdemog package to get them.

The Census reports these figures for ZCTAs, its version of ZIP code areas. Most MODZCTAs contain only one ZCTA. Where a MODZCTA contains more than one, I added up the counts. For the medians, I took an average of the ZCTA medians, weighted by the number of households. The Census caps median income at $250,000. A few of the richest ZIP codes in Manhattan hit that cap.

I fit a regression with this data where the outcome is the typical weight of the dogs in each ZIP code. I use an average rather than the median here, because the median is so lumpy. Specifically, I average the logged weights, which is the same as taking the geometric mean. Breed weights are very skewed. The middle dog weighs about 17 pounds, but a few breeds weigh well over 100. And differences in weight are proportional. Five pounds is a big difference between two toy dogs, but not between two retrievers. On the log scale, effects become percent differences in weight. The predictors are the five measures above, plus the borough. There are 177 ZIP codes in the model.

The five predictors use different units: dollars, square feet, rooms, and percentages. This makes the coefficients hard to compare directly. What we can do is ask “If we take a ZIP code that is low on this measure (at the 25th percentile of ZIP codes) and move it to a high value (the 75th percentile), holding everything else constant, by what percent do its dogs get bigger or smaller?” For the boroughs, the figure shows the difference between each borough and Manhattan.

Income is the clearest result. A ZIP code with a median household income of $116,000 has dogs that are about 17 percent heavier, on average, than one with a median income of $67,000. The housing measures make much less difference once we know income, and they do not point the way Bill expected. The share of homes in one- or two-family houses and the share of owners have effects close to zero. Median rooms has a small positive effect, maybe, with an interval that includes zero. Floor area per home has a small but clearly negative effect. Moving from about 910 to about 1,100 square feet per home goes with dogs about 3 percent smaller, at the same income. Together, the four housing measures do improve the model a little, by more than we would expect by chance. But mostly that is the floor area effect, and it runs the wrong way for the idea that more room means bigger dogs.

Among the boroughs, Staten Island’s dogs are about 10 percent heavier than Manhattan’s at the same levels of income and housing. That fits the idea of room on the periphery, although the interval is wide, running from about 1 to 20 percent. The other boroughs do not differ much from Manhattan.

In terms of how much variation we can explain, income alone accounts for about 53 percent of the variation in typical dog size across ZIP codes. Adding the borough raises that to 54 percent. Adding all four housing measures raises it to 57 percent.

Here is the income effect on its own. This is a partial regression plot. For both dog size and income, we remove the part that the borough and the housing measures predict. What remains shows the relationship between income and dog size among ZIP codes with similar housing in the same borough.

The relationship is pretty good. Each doubling of median household income goes with dogs about 22 percent heavier. The points come from every borough. So there is a size gradient, as Bill speculated, but it’s not purely geography that’s at work here, it’s money. Some of the biggest dogs in New York live in dense, expensive neighborhoods such as Park Slope, SoHo, and the West Village. The smallest dogs live in the Bronx, where incomes are much lower. Once we know a ZIP code’s income, its housing tells us only a little more about its dogs, and not in the expected direction. If anything, more floor area per home goes with slightly smaller dogs. Staten Island is the one place where extra room may add to dog size, beyond what income predicts.

Again, as a paid-up social science rather than infographic guy, I feel like I have to close with the caveats. Remember these are breed weights, not dog weights. Four in ten dogs have no breed size. And, like we said before, these are the licensed dogs only. Licensing rates probably vary with income too. Finally, this is a comparison of ZIP codes, not of households. It tells us that richer ZIP codes have bigger dogs on average. It does not tell us that richer people choose bigger dogs, although that seems likely.

Why would money matter more than space? Big dogs cost more to feed and to treat at the vet. They are harder to keep in buildings that limit pets by weight. And their owners may need to pay someone to walk them while they are at work. Perhaps those many daily dog visits to Central Park are part of the answer too. If you live near a very large park, maybe it’s easier to have a very nice apartment that’s comparatively small for your very large dog.

The Decoupling of Outlet from Impact

Mon, 19 Oct 2026 10:11:49 -0400

Recently I had the happy task of contributing to a forthcoming issue of the American Behavioral Scientist honoring my PhD advisor, Paul DiMaggio. The issue engages with different aspects of Paul’s very varied work in organizations, culture, institutional analysis, and other topics. My contribution, which you can read a draft of here, puts his most-cited paper, DiMaggio and Powell (1983), in the context of other highly-cited articles in sociology since the 1930s and tries to see what that can tell us. It tries to do that through the lens of the paper itself.

“The Iron Cage Revisited” is one of the most successful articles in the history of sociology. Web of Science says it has accumulated over thirty-two thousand citations; Google Scholar puts the figure at over seventy-seven thousand. It is the most-cited paper ever published in the American Sociological Review, and the most-cited sociology paper written in the 1980s. Its core ideas—coercive, normative, and especially mimetic isomorphism—have traveled well beyond sociology, most notably into organization studies and management.

Sociology’s top-ten most-cited papers by decade

Here’s the main figure from my paper:

The top 10 most-cited Sociology papers from each decade, 1930s-2010s.

To make it, I took a cohort-based approach to the Web of Science citation data. That is, I divided the period from 1930 to 2019 into decades and, for each decade, found the ten sociology articles published in that decade that have the highest number of lifetime citations.¹ Each label shows the journal the paper appeared in. American Sociological Review (ASR) articles are in blue. The American Journal of Sociology (AJS) is in gold. Everything else is in gray. Looking at the very top of the field this way, a few things jump out.

First, there is the sheer volume of citations accumulated by papers from the 1970s and 1980s. For example, Granovetter (1973) is “The Strength of Weak Ties”. It has over 24,000 lifetime citations. The most-cited paper from the 1930s, Merton’s “Social Structure and Anomie” (1938), has about 3,800 by comparison, despite a thirty-five-year head start. “The Iron Cage Revisited” has over 32,000. Papers from more recent decades are of course at a disadvantage simply because they’ve had less time to accumulate citations. But the advantage held by the 1970s and 1980s cohorts isn’t just a matter of time. These are the papers that benefited most from the postwar expansion of the university system, and from the subsequent growth of the citation base as more and more faculty began writing more and more papers.

Second, there’s the character of the papers themselves. Across the decades, most of the top-ten papers are either primarily theoretical or primarily methodological. They are not, for the most part, reports of findings from a particular empirical study. “The Strength of Weak Ties” does briefly sketch Granovetter’s dissertation research, but only schematically, as one illustration amongst several of the main theoretical argument. “The Iron Cage Revisited” is wholly a theory paper. So is Swidler (1986) on culture as a toolkit, West and Zimmerman (1987) on doing gender, and Collins (2015) on intersectionality. On the methods side, we have Freeman (1978) on network centrality, Bentler and Chou (1987) on structural equation models, Corbin and Strauss (1990) on grounded theory, Raftery (1995) on Bayesian model selection, Heckathorn (1997) on respondent-driven sampling, and more besides. The papers that become the most widely cited present either theories or tools. Specific findings, however important, tend not to reach this (admittedly rarefied) level of influence.

The third pattern is the one the paper is mostly about. In the 1930s, five of the top-ten papers appeared in AJS. (ASR’s first issue was in 1936 but it still managed to get two papers in the top ten from that decade.) In the 1940s and 1950s, ASR dominates, with seven of the top ten in each decade and AJS accounting for most of the rest. The 1960s look much the same: one paper from Social Problems (Glaser 1965 on the constant comparative method) and the rest from ASR or AJS. Beginning in the 1970s, though, things start to fragment, and the two flagship journals decline in prominence decade by decade. By the 2000s, just one of the ten most-cited papers comes from ASR, and none from AJS. By the 2010s, neither journal has a paper in the top ten at all. The journals that take their place—the Annual Review of Sociology, Gender & Society, Sociological Methods & Research, Social Networks, Sociological Theory—are specialist outlets or review journals, not competing general-interest journals like Social Forces. This is a complete reversal of the midcentury pattern.

“The Iron Cage Revisited” sits right at the hinge of this change. It’s a pure example of a once fairly common kind of article: a theoretically focused, self-consciously agenda-setting piece published in one of the discipline’s two leading generalist journals. It is also the high-water mark of that combination of form and outlet. Now, the raw counts in the figure favor the 1970s and 1980s. It’s natural to ask what happens if we express each paper’s citations relative to some denominator. One option is to express the citation counts as a percentage of that decade’s citation base: that is, the total number of citations, from anywhere and at any time, to all sociology articles published in that decade. Here is each decade’s most-cited paper expressed as a share of that total.

Decade	Authors	Title	Journal	Share
1930s	Merton	Social Structure and Anomie	ASR	15.49%
1940s	Mills	Situated Actions and Vocabularies of Motive	ASR	5.08%
1950s	Sykes & Matza	Techniques of Neutralization	ASR	4.11%
1960s	Gouldner	The Norm of Reciprocity	ASR	4.56%
1970s	Granovetter	The Strength of Weak Ties	AJS	4.63%
1980s	DiMaggio & Powell	The Iron Cage Revisited	ASR	3.32%
1990s	Corbin & Strauss	Grounded Theory Research	ZS	0.68%
2000s	McPherson, Smith-Lovin & Cook	Birds of a Feather	ARS	0.62%
2010s	Collins	Intersectionality’s Definitional Dilemmas	ARS	0.26%

Because the citation base grows sharply almost every decade, normalizing this way tends to favor older papers. Merton’s “Social Structure and Anomie” (1938) accounts for over fifteen percent of all citations to sociology papers from the 1930s. From the 1940s through the 1980s the top paper’s share sits in a band of roughly three to five percent. About one in every thirty citations to a sociology paper published in the 1980s is to “The Iron Cage Revisited.” Then, from the 1980s to the 1990s the top paper’s share drops by about an order of magnitude, while the citation base grows by about fifty percent. After that point it’s no longer possible for even the most-cited paper to command the kind of share available to the 1980s cohort.

Why did this happen?

So, from the 1920s to the 1970s, sociology’s most-cited papers were published in its two most important generalist journals. Now, ASR and AJS remain the most prestigious journals in the discipline, and publishing in them is still vitally important for establishing and consolidating a career. But, at least at the very top of the distribution, these journals no longer publish the discipline’s most-cited work. Impact has become decoupled from venue. Why?

It’s plausible to think that the shift is broadly institutional in character, and driven by the forces that DiMaggio and Powell identified. The postwar expansion of American higher education happened in two phases. First came the rapid growth of universities from the late 1940s to the end of the 1960s, bringing many more sociology departments, faculty positions, PhD programs, and journals. Then came the arrival of the students educated during that expansion into the ranks of the faculty themselves. This makes the 1970s and 1980s the era of what you might call “Boomer Sociology”, when the first postwar generation was coming into its own and the discipline reached its largest size.

At this point, there were far more academics writing far more papers, and many more places to publish them. But the most prestigious journals had been founded in earlier waves of institutionalization, and they didn’t much change their format, frequency, or the number of articles they published. As job markets tightened after the initial wave of expansion, the career payoff to publishing in a top journal rose. That changed the character of the gatekeeping role the flagships played. Journals became focal points for career competition.

Over time, ASR and AJS responded with—or gravitated toward; the particular mechanism doesn’t matter much here—a model that demanded more of the papers it published. They had to make a significant contribution on several fronts at once: a substantial empirical finding and an advance in either theory or methods, while at the same time still on a topic of interest to a wide swathe of the field. The idea of a “full package” paper became the expectation. This is a high bar! Run-of-the-mill findings couldn’t clear it. But neither, increasingly, could papers whose main contribution was a theory or a method. You can see this in the figure. After the 1980s there’s only one purely big-idea theory paper from ASR or AJS in the top ten (Emirbayer and Mische 1998’s “What Is Agency?”). The other flagship papers that make the list—Uzzi (1996) on embeddedness, Inglehart and Baker (2000) on modernization and cultural change, and Burt (2004) on structural holes and good ideas—are empirically focused studies with a strong theoretical idea attached. They are exemplars of the newer sort of ASR or AJS paper. But they are also the last papers on these lists to appear in either journal.

It’s tempting to see this as a recognizable case of normative and mimetic isomorphism. Professionalization is one of the main engines of organizational homogenization. In a field whose members are trained in similar programs, credentialed in similar ways, competing for a limited number of positions, and evaluated by somewhat vague but widely shared criteria, organizations and their outputs will tend to converge in form. Sociology’s relative lack of disciplinary consensus may have made this process more intense at the level of the elite journals in particular. (Sociology has always been the least disciplinary of the social sciences, with both the most internal disagreement on theory and methods and—relative to the other social sciences—the most representation of different positions across the field.) Editors were more likely to see themselves as setting and modeling standards for the whole field through peer review. Meanwhile, in departments, a flagship publication could act as a point of consensus amongst colleagues who otherwise disagreed strongly about theory and method. It’s hard to dismiss the value of an ASR or an AJS. (John Levi Martin has referred to this tendency as the Pax Wisconsana.)

Research literatures still principally prize good ideas and useful tools, however. People didn’t stop writing theory and methods papers, and they certainly didn’t stop citing them. It’s just that those papers stopped appearing in the flagships. Big-idea theory papers found homes in Theory and Society (for a while) or Sociological Theory. Methods contributions went to Sociological Methods & Research or Sociological Methodology. Papers developing concepts within subfields like gender, race, culture, or networks went to journals like Gender & Society or Social Networks. And the Annual Review of Sociology emerged as one of the most important venues for highly cited work in the 2000s and 2010s. Reviews have an obvious citation advantage, as they can absorb credit from the work they summarize, though the evidence on how much this matters is mixed.² Moreover, the most-cited Annual Review pieces tend to frame a problem, set an agenda, or introduce an organizing concept. That is, they are often theory papers disguised as literature reviews.

The upshot is that the demographic expansion and steady professionalization of sociology transformed the expectations attached to people facing an often-uncertain job market. Conditions distinctive to the discipline may have increased the gatekeeping power of its flagship journals. The resulting standards were reinforced through PhD training, the job market, and the tenure process. As a result, the average quality of work in the flagships went up, papers in those journals became more similar to one another, and what ASR and AJS were likely to accept became decoupled from what, at the very highest end of the distribution, the field found most valuable to cite. Elite journals remained elite, but the character of their importance shifted along with the composition of their content. “The Iron Cage Revisited” both marks that shift and helps explain it.

“Sociology articles” means articles published in any journal classified as “Sociology” in the Social Sciences Citation Index. Citation counts are cumulative across all Web of Science databases, so they include citations from any field. Data and code to reproduce the analysis are available on GitHub. ↩︎
The worry goes back at least to Michael H. MacRoberts and Barbara R. MacRoberts, “Problems of Citation Analysis: A Critical Review,” Journal of the American Society for Information Science 40, no. 5 (1989): 342–349. For mixed evidence, see Christian Lachance, Steve Poirier, and Vincent Larivière, “The Kiss of Death? The Effect of Being Cited in a Review on Subsequent Citations,” Journal of the Association for Information Science and Technology 65, no. 7 (2014): 1501–1505; and Ruben Miranda and Esther Garcia-Carpintero, “Overcitation and Overrepresentation of Review Papers in the Most Cited Papers,” Journal of Informetrics 12, no. 4 (2018): 1015–1030. ↩︎

Dogs of New York

Wed, 14 Oct 2026 13:17:28 -0400

I recently updated nycdogs, a data package for R that I originally made for teaching purposes. It bundles up New York City’s dog licensing and dog bite datasets into an R package. The data is already provided by the city in a directly usable way (as CSV files), but packaging it up makes it easier to integrate with other packages and get things moving a little more quickly in class.

The nyc_license table looks like this:

nyc_license
#> # A tibble: 819,323 × 12
#>    animal_name animal_gender animal_birth_year breed_name       breed_rc zip_code zip   license_issued_date license_expired_date extract_year borough city 
#>    <chr>       <chr>                     <int> <chr>            <chr>    <chr>    <chr> <date>              <date>                      <int> <chr>   <chr>
#>  1 Paige       F                          2014 American Pit Bu… Pit Bul… 10035    10035 2014-09-12          2017-09-12                   2016 Manhat… New …
#>  2 Yogi        M                          2010 Boxer            Boxer    10465    10465 2014-09-12          2017-10-02                   2016 Bronx   Bronx
#>  3 Ali         M                          2014 Basenji          Basenji  10013    10013 2014-09-12          2019-09-12                   2016 Manhat… New …
#>  4 Queen       F                          2013 Akita Crossbreed Akita C… 10013    10013 2014-09-12          2017-09-12                   2016 Manhat… New …
#>  5 Lola        F                          2009 Maltese          Maltese  10028    10028 2014-09-12          2017-10-09                   2016 Manhat… New …
#>  6 Ian         M                          2006 Unknown          Unknown  10013    10013 2014-09-12          2019-10-30                   2016 Manhat… New …
#>  7 Buddy       M                          2008 Unknown          Unknown  10025    10025 2014-09-12          2017-10-20                   2016 Manhat… New …
#>  8 Chewbacca   F                          2012 Labrador Retrie… Labrado… 10013    10013 2014-09-12          2019-10-01                   2016 Manhat… New …
#>  9 Heidi-bo    F                          2007 Dachshund Smoot… Dachshu… 11215    11215 2014-09-13          2017-04-16                   2016 Brookl… Broo…
#> 10 Massimo     M                          2009 Bull Dog, French French … 11201    11201 2014-09-13          2017-09-17                   2016 Brookl… Broo…
#> # ℹ 819,313 more rows

The dataset is good for teaching with for several reasons. First, people like dogs. Second, it has about 800,000 records, which is a lot in many ways but also trivial for a modern laptop to work with. Third, the data are pretty clean, but they’re not that clean. There are various things going on in there that invite us to explore it but also to be careful about what we can and can’t be confident about.

The first thing we ask about any data set like this is: what is each row in this table? In the case of nyc_license, each row is a record of one time-limited license for a dog. A row is not necessarily the record of a unique dog. When an owner renews a dog’s license, the dog will appear one time for each license period. In New York City, dog licenses can be issued for one year or five years. The table does not provide any dog_id column that ties sequences of licenses to individual dogs.

Second, the table contains several “extracts” of the license data. It seems like the way the provider updates the NYC Open Data record is to periodically pull from its internal database every year or so and append it to the existing, previously-released version. The extract_year column identifies the extract, so at least we know that. But it’s also the case that a license that was active in more than one extract year appears in each extract. As a result, the table has a large number of duplicate rows.

Third, license information is provided by dog owners. This means that there’s a fair amount of noise in the records, because people make typos and other errors. For instance, the ZIP code data has incomplete ZIPs. Some people provide invalid ZIP codes (e.g. ones that are fewer than five digits long). Others (about 3,300 records) have ZIP codes that are not in New York City. Eighty percent of these (about 2,700) are ZIP codes located elsewhere in New York state. Another 77 are located in Florida, which I kind of like. Five of them live in Boca Raton, for example. Their kids never call. You’d think it would be easy for them to just pick up the bone, I mean phone, but you never hear from them.

About 650 rows have valid ZIP codes that are in Manhattan, but these ZIPs are not ZCTAs. The trouble with ZIP codes is that people think of them as areas, as polygons, but in fact they are lines. They are defined by the Post Office as postal routes, i.e. as journeys a mail carrier takes. While they tend to form closed polygons, they are absolutely not guaranteed to. But people know their ZIP code and all kinds of analysts (from marketers to political operatives) like thinking in terms of ZIP codes. So the Census defines ZIP Code Tabulation Areas that are a kind of best-effort at answering the question “What if this ZIP code were a closed polygon?” ZCTAs exclude every P.O. box and single-address ZIP nationwide. Manhattan has several buildings that have their own ZIP code. New York City also defines MODZCTAs that merge some low- or no-population ZCTAs with their neighbors. I use these below.

Fourth, beyond the ZIP codes, owners also provide information about the age and sex of their dogs, the dog’s name, and the dog’s breed. Again, this kind of information tends to be a little noisy. For example, people vary in how they spell names. They also vary in how they classify their dog by breed. Beyond the list of “recognized” breeds (itself something that dog owners variably conform to), some people might describe their dog as, e.g., “Labrador-Sheltie mix” while others might just say “Unknown”. The dataset includes the breed_name column from the original dataset and also breed_rc, which is a recoding by me to clean up equivalent terms (“Yorkshire Terrier” and “Terrier, Yorkshire”, for example) along with a few other coding decisions.

Finally, and only because I’m choosing to stop here, the table just shows the number of licensed dogs in New York City. That is definitely not the same as the number of dogs there. Moreover, license compliance varies quite a bit in ways that are related to characteristics of the owner that we can’t observe, and also ones that we partially can, like where they live.

All of which is related to the fact that one of the lolsob-mantras of courses I teach is “Now you know the true meaning of data”. In the graphs and maps below, I’ve done the best I can, where appropriate, to uniquely identify individual dogs in the dataset (even though ultimately we cannot).

Popular Breeds

With that out of the way, it’s time to indulge your prejudices about Dogs, New Yorkers, and New York Dogs. The most common “breed” of dog in New York is “Unknown”, as befits a city of immigrants, hustlers, scrappy up-and-comers, and dogs seeking to leave their small-town past behind and remake their identities. Somewhere between eight and ten percent of records each year have “Unknown” recorded as their breed. There’s some change over time (a declining share), but I think this is an artifact of how breeds are getting recorded rather than a change in the composition of the dog population. If we leave the Unknowns out, here are the trends in annual licenses issued to the twenty most popular breeds.

Some breeds are pretty steadily popular. Smaller breeds do well in a city full of apartments. We can also see some evidence here of trends in popularity. However, these annual numbers are what we might call the prevalence of dogs, rather than the incidence of new dogs. If we want to get a better sense of fashion trends in dog breeds, we should look at puppies. (Who doesn’t want to look at puppies?) Because of the structure of the data we can’t quite do this. The best we can do, based on the dog’s age, is to examine licenses issued to dogs younger than a year old. Let’s pick out some breeds that seem to be rising, a few that seem relatively steady, and a few that might be declining. I’ve indexed these changes relative to each breed in 2016 by setting the value for that year to 100. Even though our range is narrow, we use a log scale on the y-axis because, e.g., a rise in a breed’s index from 100 to 200 reflects the same relative change as a fall from 100 to 50.

Maltipoos and Goldendoodles seem to be doing pretty well for themselves. Golden Retrievers remain steadily popular. Meanwhile, Shiba Inus, beloved fad-breed of the 2010s, seem to have been affected by their association with crypto and are strongly in decline. Much drop. Very diminish. Wow. Beagles are also doing poorly. Good grief.

Where Breeds Live

We can look at the relative prevalence of breeds by their ZIP code (strictly speaking, by their MODZCTA). I hesitate a little on this just because, even with 800,000 licenses, we’ll run out of degrees of freedom fairly quickly if we break things down by breed and ZIP. Relatedly, we should ask what sort of metric we’re trying to show here. What’s the denominator? (A surprising amount of Statistics, I claim, is essentially exploring the question “What happens if I divide by this number instead of that number?”) Popular dogs are, to a first approximation, pretty popular all over the city. Most ZIP codes have a solid contingent of Yorkies. If, for each popular breed, we calculate the share of that breed as a percentage of all dogs in the ZIP code, we’ll see some ZIP-by-ZIP variation come out.

It would also be nice to adjust for the fact that different ZIP codes have quite different shares of the total number of dogs licensed in the city. (Which, again, is not the same as share of the total number of dogs in the city.) So we calculate the within-ZIP share and make a ratio of it by dividing that number by the breed’s citywide share. The ratio is 1 when the breed is just as common in that ZIP code as it is across the whole city. If it’s 2, the breed is twice as common in that ZIP as it is across the city. If it’s 0.5, the breed is half as common in that ZIP as it is in the city as a whole. The main downside of ratios like this is that they can exaggerate differences a little, especially if the base rates are small. We have enough breeds in each zip code to make me feel OK about doing this. But if you refer to the bar charts above you can see that even for disproportionately popular breeds in zips, you’re looking at changes like “twice as common with respect to a baseline of five percent of all dogs”. Here’s the map:

We see some plausible evidence that breeds are distributed across ZIP codes in a manner that tracks the demography of owners. French Bulldogs in Chelsea, Soho, and TriBeCa. Havanese on the Upper East Side. Pit Bulls in the Bronx. German Shepherds on Staten Island. Goldendoodles and King Charles Spaniels in Manhattan south of 96th.

The Diffusion of Goldendoodles

Let’s take a closer look at the Goldendoodles. In 2015, Goldendoodles weren’t that common. We can ask: where were they initially popular? Did they diffuse across New York City from focal points where people were more likely to set trends in breed fashion? Restricting ourselves to young dogs, we can try to look at each ZIP’s share of all Goldendoodle puppies licensed each year from 2015 to 2025. Here’s what that looks like:

In 2015, Goldendoodle puppies are licensed in a relatively small number of ZIP codes, mostly in Manhattan. Most ZIP codes are doodle-free. Over the next five years, they gradually spread across the city. By 2023 or 2024, most—though still not all—ZIP codes are licensing young Goldendoodles.

Again, the underlying numbers aren’t huge. So take all of this with a delicious treat. I mean, grain of salt. I also wonder what the next few years are going to look like in terms of dog popularity. Based on the maps above, I think Australian Shepherds are likely to continue to rise in popularity amongst owners looking to acquire a fashionable breed. And while they’re not on any of the maps above, the data also suggest that the Jindo, a Korean breed that is a relatively recent arrival in the US, has been sharply on the rise. If you ask me, though, if you’re in New York City and planning to get a dog, you should throw in your lot with Unknown and get a good, solid, dog-shaped mutt. Preferably one with strong views on the state of the Subway these days, or who knows a guy who can get you a deal on some shirts, or the special ramen, or tickets. Not tickets for anything in particular. Just, like, tickets. Works out of an upstairs bar downtown on Water St. Trust me.

Steven Lukes, The Diversity of Morals

Thu, 08 Oct 2026 06:27:40 -0400

In August I was a discussant on a panel about Steven Lukes’s recent book, The Diversity of Morals. Here’s an extended version of what I said. A PDF version is available as well.

The Diversity of Morals asks the questions philosophers ask about morality: what is “the moral”? How wide and deep is its diversity? Is there a standpoint from which we can make our judgments? It then refuses to answer them by philosophical methods alone. Instead, it draws on the resources of both social theory and the social sciences, from primatology and anthropology to sociology and political science, in an effort to construct a defensible answer. I do not think it quite succeeds. But I am also unsure of what success would look like.

Before any critique, let me state the argument as I understand it. I confess that I found it a little tricky to reconstruct because of the way the book lays things out. The text is dense with quotations, sometimes quite lengthy, and prefers to proceed by picking out representatives of the various positions investigated, letting us hear them speak, and maybe selecting some piece of their argument to be partially endorsed and carried forward. It reminded me of being in an expertly-taught graduate seminar where the participants are working their way through a wide literature. In a seminar we would be able to stop at many points to discuss each new contributor’s ideas, to make connections in an expanding web of ideas. But because this is a book, at times it was a little too much like a whistle-stop tour of other people’s views. I found myself wanting a clearer sense of the author’s own argument. To be fair, Lukes acknowledges early on that he does not always take sides, and that sometimes “such reluctance may be the appropriate way to recognize the sheer difficulty of reaching a convincing conclusion” (xii).¹

Hume, Herder, and Smith

There is a positive line of argument, though. The book argues that the word “moral” names not a domain or perspective sitting alongside the legal, economic, or political, but the standpoint from which all such standpoints are judged. It’s the point of view from which we ask what the most serious, central things in life are. There is no “outside” to this standpoint. The unmaskers of morality, from Thrasymachus to Bernard Williams, are moralists in spite of themselves. The distinctively human capacity underlying it is “Smithian empathy”, that is, an effort to understand others embedded in a second-person point of view where we both demand and owe justification. This perspective-taking is not the Humean “contagion” that we share with other primates, but a kind of projection that triangulates what you would feel, what you should feel, and what an impartial “anyone” would feel (58-62; 71-3).

This Smithian empathy is used to overcome the broad contrast that the book starts out with. On the philosophical, or Humean, side is the intuition that there can be correct answers to moral questions, possibly grounded in the conviction that people are in the end all very much the same, perhaps thanks to some intrinsic human nature. On the social-scientific, or Herderian, side is the intuition that the sheer heterogeneity of moral beliefs and practices across societies makes a nonsense of efforts to derive and defend some general substantive morality. The best we can do is document its variety, and perhaps try to naturalistically explain its origins in particular cases.

This broad distinction, as the book itself makes clear, breaks down almost immediately. There are plenty of error-theorists and moral nihilists in philosophy. And in the social sciences, there is a lot of good empirical work on the most general aspects of “moral” social behavior, which generally means the emergence of generalized exchange, or altruism, or “pro-sociality”. A little more narrowly, and where many sociologists land, we have examinations of the character “moral” beliefs or actions, of “values” of all kinds and the practices or norms associated with them. These are all “social” morals, in the book’s terminology.

The Diversity of Morals agrees that the natural and social sciences can explain the emergence and persistence of morals in that second sense, but argues that they cannot naturalistically explain the universalization of morals—what the book calls “critical” morals. This is both the expansion of moral concern past every boundary of group and culture, and an evaluative attitude toward the moral perspective itself. It seems to be a contingent historical achievement. The book explores its possible sources in Axial-age reflexivity, Enlightenment dignity, and the Durkheimian sacralization of the individual in general, even as it is repeatedly threatened by counter-sacralizations that confine empathy to the bounds of nation, volk, race, or party. Here the question is how we should think about this critical moral capacity.

For Lukes, “values” are what are important, desirable, and justifiable to ourselves and others (137). The extreme test of the relation between merely social and properly critical morals is atrocities like the Holocaust and other genocides. The book resists the “situationist consensus” that says anyone in that particular situation would have acted in those ways (99-110). But it also insists that we cannot let ourselves off the hook by saying that what went on in the minds of the people who perpetrated these atrocities is completely severed from our own morality. We need a “bridgehead” of recognizable motivation or shared rationality (116). Smithian empathy has to apply here, too, somehow.

The book finds its way out with the help of the empirical literature (from Schwartz and D’Andrade) finding that personal values vary surprisingly little across societies. Practices differ, because “what counts as” realizing shared values differs widely, but the values themselves do not. And so we end with two claims. Morals narrow along a continuum from “mere” conventions to ultimate values, with the latter being more shared than might first appear. Morals also deepen, not in the sense of becoming richer or more fruitful, but in the way the ocean deepens into utter darkness: morals deepen into non-recognition, where the second-person standpoint, the need to justify one’s values to others, fails or falls away, and the moral community shrinks to the like-minded, to tribalism. While never losing its essentially sober outlook, the book does try to end somewhat optimistically, by arguing that contemporary political polarization might be successfully overcome. Much of it, Lukes suggests, is largely manufactured non-recognition rather than truly irreconcilable differences in values.

Width and Abstraction

My first worry is that the “width” thesis is an artifact of abstraction due to measurement. The claim that value diversity is narrow rests principally on D’Andrade’s study and the Schwartz surveys (D’Andrade, 2008; Schwartz & Bardi, 2001). Across the United States, Japan, and Vietnam, and across dozens of national samples, people affirm strikingly similar hierarchies of thin value-words. We all value responsibility, honesty, self-fulfillment, treating human life as precious. Within-society variation dwarfs between-society variation. The puzzle of why practices differ so much more than values is then solved with Searle’s formula: societies differ over what counts as realizing values they share (Searle, 2010).

As the book itself notes in Chapter 1, however, Nelson Goodman points out that similarity is relative to context. Whether two things count as being alike depends on what we are looking for. If we describe values thinly enough, everything converges to similarity. Every society values “responsibility”, every society values “care”. But this is not a discovery about the true shallowness of moral disagreement, it’s a feature of the level of description we are operating at. If all the moral action is in the specification—if what responsibility is, what it demands, whom it covers, is what different cultures differ on—then the formula “shared values, divergent realizations” is not an explanation of their disagreement. It is a redescription of it.

Take the book’s own abortion example. We are told both sides agree on the values of responsibility, privacy, and self-determination, and differ only in how these are linked to the world. But these rival “conceptions” of responsibility have such different practical implications that they might as well be simply rival values. The survey data can’t really arbitrate between these readings, because the surveys only ever ask about the thin word. The work of Jerolmack & Khan (2014), which the book cites approvingly, insists that values must be viewed in context, tied to norms and practices. But the book’s claims about width rely on a process of decontextualization that it elsewhere disapproves of. The book does offer “intelligibility” and “seriousness” as a means of bounding the space of valid values. But even here, it admits that an awful lot can be left over, especially if we take the time to understand why something seemingly unintelligible or unserious at first blush might turn out to be just the opposite on further reflection. Moreover, even if these criteria can delineate a space of values it doesn’t follow that this space is small. And the optimistic parts of the Coda, the parts about about deliberation and polarization, need that space to be small, rather than a large field of genuine and irreconcilable conflict. How confident are we that that the range of real value diversity is narrow?

Depth and Disagreement

A version of this question arises in the case of the depth of values, too. By the end of the book, depth is redefined as non-recognition, as an accidental or deliberate failure of Smithian empathy. But I wonder if that is what deep value conflict meant to Isaiah Berlin and Max Weber, the value pluralists discussed in Chapter 5. For Weber, at least, deep value conflict was intractability given mutual recognition. The method of verstehen is very close to the practice of Smithian empathy. The Christian of the Sermon on the Mount and the man of worldly honor, in Weber’s example, are not failing to take up each other’s second-person standpoint. The ethic of conviction and the ethic of responsibility are held by people who share a great deal in common. Weber’s claim was that the conflict between them cannot be resolved, only decided. That is the old, frightening sense of depth: irreconcilable differences among people who recognize one another, and reject what they see.

The book puts its redefinition of depth as non-recognition to work in Chapter 4, with some success. And it is necessary in the Coda as well. But I wonder whether substituting it for the old concept discreetly retires the question the old concept named. Are there rationally irresolvable conflicts among values that remain fully mutually intelligible, among people who fully recognize each other’s standing? If the answer is yes, that is, if some conflicts survive recognition, complete information, and scrupulous Smithian empathy, then deliberation, however well designed, will not dissolve them. It can only stage them. That is Weber’s position, and I do not think the book confronts it.

Chapter 5 ends with a contrast between thinking of values as merely trade-offs (Berlin), opportunity costs within choice-sets, or truly sacrifices between incommensurable ends (Weber). The Coda returns us to that question. If the depth of morality just is non-recognition, restoring recognition through some method of deliberation seems to solve the problem of depth simply by definition. In that case, experiments in deliberation aren’t just encouraging, they’re guaranteed to work. That can’t be right. If we observe preference moderation under conditions of accountability, we can interpret that as the discovery of shared values. But this is also compatible with the softening of positions that people were never deeply invested in, while genuinely deep conflicts—deep in the Weberian sense—remain untouched. The Irish case of Citizens Assemblies cited in the Coda is indeed encouraging. But, even there, one can read the Citizens’ Assembly as an already-shared value of responsibility being jointly renegotiated, or as one side prevailing under procedural conditions favorable to it.

Solving all these problems is easier the more common ground you share. The general “anyone” of Smithian empathy, the “impartial spectator”, is one way those shared values can be smuggled into the argument. The impartial spectator’s views must come from somewhere. If we disagree, and try to empathize with one another, and your impartial spectator shares many of the same views as mine, our chances of moral agreement will be much higher. If our impartial spectators seem to differ a great deal, though, things will go much worse. Again, there is some acknowledgment of this in the book. For instance, a note quoting a communication from Samuel Fleischacker, remarks that while we need to posit a “perspective-free impartiality” in order to get Smithian empathy off the ground, “We don’t need to say we have actually achieved such impartiality”, just “hold it out as a possibility to which we aspire” (224).

Metaethics and Sacralization

The author’s own values seem clear in the book. I think I share most of them. He sides with a liberal naturalism over its various rivals, with Smith over Hume on empathy, with de Swaan and Geras over the situationists and Rorty, and so on. One place the book conspicuously does not choose is between Berlin and Weber, that is, on the question of what values are, and what it really means for them to be plural. This is a hard question for the philosophers, of course, but also for social scientists who want a meaningful link between the research they do and the values they espouse.

Consider, for example, the question of the sacralization of the person. That is, the question of how the equal dignity of all humans became, in Durkheim’s sense, sacred. Critical morals (in the book’s sense) and human rights rest on that sacralization. Was its development a discovery about human beings? Or was it a contingent, collective representation that one tribe (our own) successfully consecrated? The book’s normative voice often sounds like it wants to pick the first option. It’s a discovery. Himmler’s justifications at Poznan are a failure, even an obscenity. The rescuers were right. Bezalel Smotrich’s standpoint is moral only in the anthropological sense. The counter-sacralizers and tribalists and Schmittians are wrong. And so on. Moral universalism thus has an authority that outruns its genealogy.

Durkheim, whom Lukes knows better than anyone, plainly says it’s the second. These sacralized beliefs about human dignity are, in the end, a contingent collective representation that grew out of an advancing division of labor. He says this even as he believes that the cult of the individual is something we should be committed to and take advantage of for the purposes of social policy. In its much more bleakly Nietzschean way, Weber’s view is on this side of the argument as well. Value commitments can have deep consequences for social organization, and this is why theory begins by trying to understand why people act from their own point of view. But in the end those values are not discoveries, they are primordial commitments.

This makes for a persistent tension in sociology as a discipline. Different wings of the field do of course want to advocate for particular moral views, or views with a moral element. Sometimes it’s by way of a focus on practical advocacy or policy reform, where the model of advocacy rests on claims to substantive expertise. Sometimes it explicitly rests on some standpoint, where MacIntyre’s “prison of our own upbringing” (MacIntyre, 2013) is reimagined as a source of wisdom and insight. More grandly, though these days less often, it’s by way of a claim to have grasped—and to speak for—where history itself is going. But, as a discipline, sociology is largely defined by the insight that morality as such is downstream of contingent social facts. Again, not everyone in the field agrees. But when sociologists want to emphasize what is most distinctive about their own disciplinary perspective, for the most part they reach for classics that emphasize the historical contingency of social structure, the immense diversity of social forms, and the dreaded “sociological imagination” that allegedly allows us to connect both to individual experience. In this emphasis on the variability and historicity of social life the field tends to side with anthropology (and of course disciplinary history), rather than with philosophy and economics.

When it comes to developing a position on these matters, as opposed to preferring not to think about them, The Diversity of Morals is a very useful book because it brings out just how difficult the terrain is underfoot, and just how little we have to stray from well-trodden paths in order to get bogged down trying to coherently and defensibly answer questions about the nature of morality as such, and how it relates to the findings and practice of social science. In the end, the book knows it cannot answer its biggest questions. Lukes remarks more than once that some disputes are better continued than resolved. Perhaps that is the best we can do: to at least keep talking, in a civilized manner. I will claim one extended quotation of my own. I am reminded, a little absurdly, of a scene from At the Villa of Reduced Circumstances, Alexander McCall Smith’s gentle comedy of academic manners. The scene is a ceremonial dinner at Cambridge:

The Master then rose to give a short address. ‘Dear guests of the College,’ he began, ‘dear Fellows, dear undergraduate members of this Foundation: William de Courcey was cruelly beheaded by those who could not understand that it is quite permissible for rational men to differ on important points of belief or doctrine. The world in which he lived had yet to develop those qualities of tolerance of difference of opinion which we take for granted, but which we must remind ourselves is of rather recent creation and is by no means assured of universal support. There are amongst us still those who would deny to others the right to hold a different understanding of the fundamental issues of our time. Thus, if we look about us we see people of one culture or belief still at odds with their human neighbours who are of a different culture or belief; and we see many who are prepared to act upon this difference to the extent of denying the humanity of those with whom they differ. … ‘Here in this place of learning, let us remind ourselves of the possibility of combating, in whatever small way we can, those divisions that come between man and man, between woman and woman, so that we may recognise in each other that vulnerable humanity that informs our lives, and makes life so precious; so that each may find happiness in his or her life, and in the lives of others. For what else is there for us to hope for? What else, I ask you, what else?’ (McCall Smith, 2003, pp. 56–57)

What else, indeed. Almost all of the people in these conversations are committed to something like the universalization story in their everyday lives, whether its origins lie in Axial transcendence, Kantian dignity under the shadow of God, Hans Joas’s soul and gift, Durkheimian sacredness, or something else. Philosophical error theorists and moral nihilists do not, on the whole, want to live in a state of vicious anarchy. The same goes for academic Marxists, debunking Evolutionary Psychologists, hopelessly confused Critical Realists, biogenomic bean-counters, Scandinavian DBO enumerators, tradition-marinaded MacIntyreans, authenticity-obsessed standpoint epistemologists, and have-your-cake-and-eat-it pragmatists. All of them are people you are likely to encounter at conferences. They are all in some sense already in the same conversation as you. But the problem, in the end, is that you have to make peace with your enemies, not your friends.

References

D’Andrade, R. (2008). A study of personal and cultural values: American, Japanese, and Vietnamese. London: Palgrave Macmillan.

Jerolmack, C., & Khan, S. (2014). Talk is cheap: Ethnography and the attitudinal fallacy. Sociological Methods & Research, 43(2), 178–209.

Lukes, S. (2025). The diversity of morals. Princeton: Princeton University Press.

MacIntyre, A. (2013). On having survived the academic moral philosophy of the twentieth century. In F. O’Rourke (Ed.), What happened to and in moral philosophy in the twentieth century (pp. 17–34). Notre Dame, IN: University of Notre Dame Press.

McCall Smith, A. (2003). At the villa of reduced circumstances. Edinburgh: Polygon.

Schwartz, S., & Bardi, A. (2001). Value hierarchies across cultures: Taking a similarities perspective. Journal of Cross-Cultural Psychology, 32(3), 268–290.

Searle, J. (2010). Making the Social World: The Structure of Human Civilization. New York: Oxford University Press.

Page numbers with no further bibliographical reference are to Lukes (2025). ↩︎

Trump Tourism Effects

Thu, 01 Oct 2026 20:25:09 -0400

In the past day or two on social media I kept seeing this graph from a Semafor story on the declining volume of international visitors to the United States.

Semafor Graph.

It seems to show, and was being remarked on as showing, that the second Trump administration has been responsible for a drop in visitors from abroad from some countries on a scale comparable to the effects of the Covid-19 pandemic. My first thought was “that can’t be true”. And indeed it is not true. I mean, the pandemic shut down air travel almost completely for a while. The current administration’s actions have been terrible in many ways and led to direct, negative consequences for tourism, too. But they haven’t been on that scale. Something’s wrong with the graph.

These numbers come from International Trade Administration’s I-94 arrivals data. They maintain a spreadsheet that counts all international visitors by country of residence who arrive in the U.S., spend at least one night here, are among qualified visa holders, including tourist visas and waivers and so forth. This is a monthly series. Right now, we only have data up to August of 2026, and for a couple of countries in the graph only up till June. I think—I am only guessing—that what the person who made this graph did was aggregate to yearly totals, plot those points, and then smooth them. (That’s why the y-axis values are so large in comparison to the ones on the monthly graphs I’m about to show you.) But the data for 2026 are only for half the year, hence the giant drop.

I went and got the data myself and took a look at it. Here’s the raw monthly series for the same countries as in the one above:

The monthly series.

It’s noisy because tourism is highly seasonal. Even so, the catastrophic effects of the pandemic are obvious, as is the fact that whatever is going on in the past year isn’t nearly as severe as that. Though something is going on. We can smooth the monthly data directly by calculating a trailing average, which is just a moving average except the window we’re moving across the data doesn’t extend beyond the current month. That way, we don’t illegitimately “look ahead” of the current month in a way that would make our moving average seem to anticipate the future. If we used an ordinary moving average it might seem like the trend was able to anticipate the Covid-19 downturn, for example. Here’s the result:

The smoothed monthly series.

So, no Covid-scale drop. But still, not nothing. The Canadian dip, following the trade war Trump started for no good reason, is very substantial. Just eyeballing it, it seems about twice as large as the downturn that followed the global financial crisis of the Great Recession, which is quite remarkable. It also looks like French tourism has declined, but I’d need to look a little more closely at that series by itself, as it’s on quite a different scale.

The general lesson, I suppose, is just that it’s always worth trying to cultivate some sort of rough comparative sense of how big or small an effect you should be expecting to see in your data, so that when something goes wrong (as things so often do) you’re more likely to be brought up short by it.

Speaking of which, you might also wonder about the remarkable leap in visitors from Mexico around 2004-2005 that’s visible in both the original graph and my version of it. What happened there? The answer is not some sudden shift in tourism preferences, but our old friend, measurement change. Prior to 2005, visitors from Mexico originating within 25 miles of the border weren’t counted as visitors. After 2005, they were. Hence the jump. In a smaller but still noticeable way, data for 2014 aren’t fully comparable either, because overseas one-night-stay visitors were included that year. There is some documentation of this on this old BTS page. If you want to measure change, you can’t change the measure.

Update: Now that I have this data and am messing around with it, here’s one more. This is monthly data averaged with a 12-month trailing window, showing all travel to the U.S., broken out by different “regions”. Canada and Mexico are large enough to each be their own region. Looking at things this way deliberately smooths away all the seasonality. The size of the Canadian reaction to the Trump administration’s attack on them really is remarkable. Elbows up, Canada.

All visitor travel to the U.S.

Research Workflows

Mon, 07 Sep 2026 08:55:03 -0400

I’m teaching MPTC again this semester and have been developing a few workflow slides. Here’s a popular one.

The Scientific Process.

Now if you’ll excuse me there’s a mandatory webinar I need to click through.

socviz 2.0.0 on CRAN

Fri, 21 Aug 2026 07:25:44 -0400

In anticipation of the second edition of Data Visualization, which is coming later this year from Princeton University Press, version 2.0.0 of my socviz package is now on CRAN. The update removes some functions that aren’t needed anymore and adds a couple of sample datasets.

Ihaka Lecture

Thu, 13 Aug 2026 12:09:27 -0400

I gave the first of this year’s Ihaka Lectures at the University of Auckland this past July. It revisited and partially updated things I’ve been talking about for the past eighteen months or so on trust in data visualization (and by extension in scientific work) in the light of the many challenges we presently face.

Zero Sum Problems

Thu, 21 May 2026 15:50:40 -0400

Over at Daring Fireball, John Gruber makes a passing observation about the Apple Sports app:

I’ve got some gripes about certain specific aspects of Apple Sports. Like, where does one even start to explain how much is wrong with their zero-sum visualization of team stats? Has anyone ever even seen a presentation like that before? Anyone?

That “Anyone” link lands over here. Hi everyone! The team stats image is quite confusing. It’s a summary of a game between the San Antonio Spurs and the Oklahoma City Thunder. I don’t know much about basketball, but I do know a bit about data visualization and in a pleasing coincidence my former student Josh Fink is the A-VP of Basketball Data Science for the Spurs. Here is the image that John objected to:

I had to look at it for a while as well.

I just finished driving a very long way up the side of the country, so I’m kind of tired. But even allowing for that, boy, this way of representing things really is quite confusing. Not being an Apple Sports user I had to look at it for a bit to understand what was happening. But, now that it has given me a headache, I can kind of see why whoever designed this ended up in the undoubtedly bad place they did.

Before I get to why I have some sympathy for the designer, why did I find this representation of these numbers so disorienting? It’s not just just because I’ve been driving for nine hours. John is right to call the picture a “Zero Sum” representation. The design strongly suggests to the viewer that, within each row, we’re looking at each team’s share of a total. Each pair of black and blue lines seem to be vying for control of their whole row, with the longest line being the “winner” in each case.

This sort of representation would make perfect sense for a measure that really was zero sum. Take an example from a properly good sport, like rugby. There, like in basketball, to a first approximation a team either has the ball or it doesn’t.¹ But there’s no shot clock in rugby, and possession routinely gets turned over without the game stopping. So, knowing that Team A had 65% possession is not only informative, it also immediately entails that Team B had 35%. You could show that with a representation like one of the rows above.

Literally none of the measures in the Basketball data above are zero-sum in this way. Both teams could shoot 100% from the free throw line, or zero percent. But because the first three measures shown are percentages, this reinforces the zero-sum impression given by the lines. It certainly did that in my case. But then, starting with Assists, the remaining rows are just absolute numbers. When I started looking at the absolute numbers, I got confused a second time by the length of the lines. “Oh so it’s not a share, it’s the value” I thought—but no, they do correspond in terms of relative proportions to the teams share within each row. But they’re not really shares they’re just magnitudes. But they have to be shown in a fixed space and we want to make them relatively comparable somehow so … Argh.

It would be nice if there were One Weird Trick to fully fix this figure. But I’m not sure that there is. For example, at a minimum we could redraw these numbers to reflect the fact that they’re not zero-sum. Keep each measure as a row (i.e. on the y-axis) but have the lines, or columns, be side by side within each category instead of facing off. Like this:

Team Stats side by side for each measure.

This view at least lets you immediately see who “won” each measure. The viewer can just directly compare the length of the bars in each category. People are really good at doing that accurately. In that sense it’s much less confusing than the original. But there’s still a lot wrong with it. The core problem is that when we draw a graph like this, we’re usually putting the same kind of thing (e.g. countries, or religious groups, or sports teams) on the y-axis, and then seeing how different their scores are on some single measure (e.g. GDP, or number of adherents, or average points scored per game), which we put on the x-axis. Maybe we use color to break things out by some third measure as well.² In this case, I’ve just labeled the x-axis as generically as possible. “Value” covers the range of all the measures. The lowest value is 5, in Largest Lead. The highest is 88, in Free Throw %. But these numbers are not meaningfully comparable. The graph encourages us to compare across as well as within categories. But while within-category comparisons are meaningful, the between-category ones are not. There were way more Bench Points than Blocks in the game. But that is not a useful thing to know.

Team Stats side by side and ordered from absolute highest to lowest, whatever that means.

Knowing who won each measure isn’t nothing. It can be informative about how the game went, maybe especially when a team won the game but “lost” on a number of the measures. If you really wanted to lean in to that aspect, you could sort of justify the zero-sum view, and maybe look for a way to sort and order by “how much” a team “won” each category. But again, what’s the right denominator for those measures? For instance, do we care about a team’s share of all Defensive Rebounds in the game? Or do we care about the share of Defensive Rebounds a team won relative to every opportunity it had to make a Defensive Rebound? How meaningful is ordering our rows by those kinds of shares? Even worse, some measures (notably Fouls) are bad to “win”, so we’d have to do something about those.

Our fundamental problem is that we just have two cases (the teams) and fifteen different measures, or variables. Each variable, except for the three percentages, is in effect on its own scale. There’s no direct way to make comparisons across them. Sure, some of these measures are probably going to be associated with one another—e.g. Turnovers and Points Off Turnovers—but the numeric values aren’t directly comparable in general. If you know a lot about basketball you might have some informative rules of thumb about each one of these measures, or some of them in combination. But at that point the lines in this particular graph are not going to be doing any work for you; you’ll just end up looking directly at the numbers. If we had data on all these measures for every NBA game for a whole season then we could of course do much more with them, because then each measure would have a distribution across all games and across all teams.

As it is, the purpose of the “Stats” screen in Apple Sports is just to summarize information from a single game. The other thing I could think of to do with the numbers as kind of graph is something like this:

A back-to-back column chart.

This is marginally more helpful than the one before just because, again, it gets rid of the unhelpful zero-sum look of the original. As I hope you can immediately see, it creates many other difficulties. It also doesn’t do away with the core problem. That problem is principally one of information design rather than data visualization. What I mean is that what we’re trying to organize is, in effect, fifteen pairs of related but fundamentally distinct numbers. If we had fifteen cases and two variables things would be simple. But with fifteen variables and two cases … well, this is not the kind of thing you can make a single effective and non-confusing graph out of. That’s why I kind of sympathize with the designer. In a constrained space they have to show thirty numbers (thirty two, including the score). Lots of information. A straight table seems like it would be boring. Surely there’s some way to thematically integrate the numbers in a visually appealing manner that brings out some of the relationships across the rows. That’s what graphs do; it seems like the right thing to reach for. But at its heart this information is not a graph. It just sort of looks like one, and that ends up confusing people.

Modulo some measurement decisions about how to determine when possession is turned over while the ball is in play. ↩︎
Here’s an example of a graph with a categorical measure on the y-axis, a continuous measure on the x-axis, and an additional categorical feature shown with color. ↩︎

Bad Weather and the Subway

Sat, 02 May 2026 08:59:15 -0400

Snow in Inwood, New York. Photograph by the author.

Recently I’ve been looking at hourly ridership data from the New York City Subway. Last time we learned that people go to work in the morning and come home in the evening, for example. (All together now: “Only in New York, baby!”) Today we’ll learn that bad weather makes people stay at home. Except, sometimes it doesn’t.

Regular readers will recall that the subway system carries a lot of passengers every day. The ridership data for the whole of 2025 represents just over 1.3 billion entries into the system via an OMNY tap or Metrocard. It’s available aggregated to hourly resolution by station complex. With that data in hand, we can calculate average hourly ridership for every day of the week. This gives us a profile of what, for example, a Monday or a Wednesday typically looks like, by hour. When calculating the average day-of-the week profile we exclude holidays and the like.

Meanwhile, the National Weather Service provides data on severe weather events that affected the New York City region in 2025. We could get more fine-grained if we wanted to, but for now we’ll just use the general list of events the NWS provides. Then we plot the Subway ridership profile for that specific date against the average profile for whatever day of the week the event happened on.

Bad weather suppresses Subway ridership, in general. But not always.

The gray lines are the baseline. The red ones are the bad weather day. The basic shape of the gray lines (and many of the red ones) is set by the rhythm of daily life. The sharp double-peak pattern is what someone I’ve shown too many of these graphs to has taken to calling “The Giant Cat-Ears of Employment”. The cat-ear shapes vary by work day (which might be the topic of another post), but are most sharply-contrasted with the weekends, which look more like little hillocks or drumlins.

We can see a few different cases in the panels. First are days when the weather event put no dent at all in people’s day. This is because ~~of the incredible toughness and resilience of New Yorkers, something they are surprisingly very modest about~~ even though there was a weather event in the region that day, it just didn’t impinge on the city much, or at all. The light snow on February 11th or the heavy rain on March 6th are examples here. People just continued to go about their business.

Second are cases where there’s a lot of travel suppression but it’s not really—or not wholly—the weather that’s responsible. The winter storm on Friday December 26th is a case of this. Bad weather; strongly suppressed travel profile; but that’s not a regular Friday. Many people were staying at home anyway, because it’s the day after Christmas.

Third are cases where the weather does seem to have suppressed travel. These are days like the snow on January 19th, or the shitty weather on Sunday February 16th. These events look like they made people stay at home. Some of these are more severe than others. The strongest example is the flash flooding on Thursday July 31st. That happened in the back half of the day and affected the evening commute directly.

Our fourth and final category is my favorite one. Sometimes snow makes no difference at all, especially if it’s on a workday. Sometimes it’s snowy on the weekend but you’re kind of sick of it, maybe because it’s late in the winter, so you’re either going about your business as usual or you’re just staying indoors. But there’s another kind of snow day.

Let’s go exploring.

The weekend of December 13th and 14th 2025 brought the city’s first measurable snow of the year, and in decent amounts, too—between four and eight inches of accumulation. Reports remarked on how long it had been in arriving. The result was that, over the weekend, ridership on the subway went up. Maybe on the Saturday it was to go out and buy the mandatory bread, milk, and eggs.¹ But maybe it was also just to be out in the snow. The next day, the people who didn’t have to go work slept in as usual. But that day, too, across the afternoon, more people than usual headed outside and took the subway somewhere. I’d like to think a bunch of them had a sled under their arm.

Maybe, at least for some New Yorkers, it was because it made more sense to take the subway than drive. Though this probably wouldn’t be all that many people. It’d be somewhat possible to investigate this with the data at hand, especially if e.g. outlying stations showed higher ridership rates. ↩︎

Hourly Subway Station Flows

Sat, 25 Apr 2026 11:12:39 -0400

Pie charts are bad, as any fule kno. We’re not as good at judging relative differences between angles and areas as we are at judging relative differences in lengths on a common baseline. This is especially true when we have more than two things to compare at the same time. So, as a rule, you shouldn’t use them. You should figure out some other way of viewing your data instead. On the other hand, I just made 424 animated pie charts because if you’re going to break a rule you should break it good and hard.

A view of the New York City Subway System (excluding the SIR). We’ll animate this in just a minute.

The New York City Subway system is very large and carries a lot of passengers every day. The MTA makes quite a bit of data available about the subway, including data on hourly flow through the system. Now, the MTA can’t track individual pathways people take through the subway. If you use an OMNY card (or before that, a Metrocard) to enter the system, this signals the start of a trip from some specific station or station complex. But unlike some systems, you don’t need to “tag out” of the subway, you just exit through a turnstile. So the system doesn’t know where you exit it. In addition, while many stations are just on a single line, some (like 34 St/Penn Station, or Fulton Street) are station complexes that serve many lines and allow transfers between them.

However, the MTA does publish hourly Origin-Destination estimates for all pairs of stations. These are their best guess about the flow of traffic from any particular station to any other. Because there are so many combinations, visualizing that sort of data is quite tricky. Even then, you don’t get information about routes through the system, just start and end points. Transit analysts and planners can go further by introducing some further assumptions about subway users. For example we might assume that commuters take the most efficient route between any given pair of entry and exit stations, and build from there to a picture of flow through the system.

I do something rather more simple here. I use the MTA’s hourly origin-destination estimates and aggregate them on a station-by-station basis to calculate in-and-out flows across 424 subway stations or station complexes. These specific numbers are averaged over all Mondays in 2025. For each hour of the we calculate the total passenger volume at the station, and the share of that volume that are estimated arrivals and departures. Then we draw a pie chart for each station, coloring it yellow for departures, purple for arrivals. The circle size reflects total volume and the pie slice proportions show the flow balance.

The flow data is pretty bulky. The original dataset has about 121 million rows. But working with it is pretty straightforward, thanks to the magic of parquet files, duckdb, and duckplyr. Having patiently downloaded the data via its API, I put it in a parquet file. The CSV is about 17GB but the parquet file boils it down to 1.5GB. Then I made a small R package that bundled that data with a few convenience functions. This lets me use the data without copying it into any single project. So I can write, e.g.,

nycsubwayodr::nyc_subway_odr()
#> # A duckplyr data frame: 15 variables
#>     year month day_of_week hour_of_day timestamp           day_of_month origin_station_complex_id
#>    <int> <int> <chr>             <int> <dttm>                     <int>                     <int>
#>  1  2025     1 Monday                1 2025-01-06 01:00:00            6                       189
#>  2  2025     1 Monday                1 2025-01-06 01:00:00            6                       313
#>  3  2025     1 Monday                1 2025-01-06 01:00:00            6                       611
#>  4  2025     1 Monday                1 2025-01-06 01:00:00            6                       125
#>  5  2025     1 Monday                1 2025-01-06 01:00:00            6                       313
#>  6  2025     1 Monday                1 2025-01-06 01:00:00            6                       154
#>  7  2025     1 Monday                1 2025-01-06 01:00:00            6                       167
#>  8  2025     1 Monday                1 2025-01-06 01:00:00            6                       612
#>  9  2025     1 Monday                1 2025-01-06 01:00:00            6                       272
#> 10  2025     1 Monday                1 2025-01-06 01:00:00            6                       167
#> # ℹ more rows
#> # ℹ 8 more variables: origin_station_complex_name <chr>, origin_latitude <dbl>, origin_longitude <dbl>,
#> #   destination_station_complex_id <int>, destination_station_complex_name <chr>,
#> #   destination_latitude <dbl>, destination_longitude <dbl>, estimated_average_ridership <dbl>

From there, we lazily query the data and duckdb does the work of doing the calculations. The whole table is never loaded into your R session, and duckdb is very fast. From there, we take our hourly flow summaries, join them to a tibble of station and line data, and export the result to some JSON files that D3js animates for us.

Here’s the result. There are three views. Initially, you see just the schematic subway map. If you click the “Map” button in the top left, it will switch to the ticking pie-chart view, which puts a pie on every station complex, with each tick being an hour of the day. The pies pile up on one another in the geographic view (in a not wholly uninformative way), but click again to have them expand to a somewhat more abstracted, force-directed network view of the system. Then click again to go back to the map. You can hover over or tap on nodes to get information about the bit of data it’s currently showing.

Now, you might reasonably say, Kieran, that’s a lot of data to show that people go to work in the morning and come home in the evening. I’m not saying there’s nothing to that criticism. But there are quite a few interesting details in there as the data pick up traffic to different parts of town. The big interchanges naturally dominate the view, but even here there are things of interest about the balance of flow, as e.g. Penn Station has people coming in on New Jersey Transit during morning rush hour and then entering the subway, which does a lot to balance its net flow during rush-hour and even tip it towards net departures. But more importantly, who doesn’t want to sit back and contemplate more than 400 pie charts, each one pulsing with life as another hour ticks by?

New York City Hexmaps

Sun, 19 Apr 2026 10:05:54 -0400

The five boroughs of New York City can be informally or formally carved up into many different pieces, depending on what it is that you’re doing. As part of an ongoing project, I recently made an R package, nycmaps, that lets you draw maps of some of these geographies. Things being what they are, these spatial units don’t necessarily overlap in compatible ways. City, State, and Congressional Districts, School Districts, Police Precincts, Fire Companies, Election Precincts, Municipal Court Districts, Zip Codes … there are loads of them. Some of them are quite straightforward; others patiently lie in wait to trap unwary analysts (I’m looking at you, Zip Codes / ZCTAs).

Mapping Tracts and NTAs

Two classifications of particular interest to people like me are Census Tracts and Neighborhood Tabulation Areas (NTAs). Census Tracts are defined by the Census Bureau and form part of a nested set of geographical units that go from the smallest unit the Census keeps track of (the Block) up to the largest (the whole country). Blocks aggregate to Block groups, Block groups aggregate to Tracts. Tracts aggregate to Counties. There are of course several complications. Ideally, the Census would like tracts to be contiguous, sub-county geographical areas with about 4,000 people in them, or at least between 1,200 and 8,000 people. This means tracts can vary considerably in geographical area. They generally follow visible features of the environment, whether physical or built. Uninhabited areas also get tract designations, so in principle we can get a full tract-level map of an area with no gaps. Here’s what a tract-level map of New York City looks like:

ggplot(nycmaps::nyc_census_tracts_2020_sf) +
  geom_sf(aes(fill = boro_name), color = "black", linewidth = 0.1) +
  scale_fill_brewer(palette = "Set2") +
  labs(fill = "Borough") +
  theme_void()

2020 NYC Census Tract boundaries.

You can see the variation in tract size (compare e.g. Staten Island tracts with those in lower Manhattan). And you can also see features that are included on the map but, at least to a first approximation, don’t have permanent residents. That big roundy blob in the southeast corner of Queens, for instance, is JFK Airport. There are about 2,300 Census Tracts in New York City. (Naturally, their number and spatial layout changes from decennial census to decennial census, because why should life be easy?)

Neighborhood Tabulation Areas, meanwhile, are not official Census units. They are one of several subdivisions used by New York City government. The idea is to aggregate tracts into units that roughly correspond to neighborhoods that people conventionally refer to. This is, of course, an impossible task, because people don’t agree on neighborhood boundaries. But the idea is good. You want something bigger than a tract because those are small enough to be noisy on many measures produced by the main source of tract-level data, the American Community Survey. But you want something smaller than the next level up, which is a Community District Tabulation Area. Presently, there are 262 NTAs. They look like this:

ggplot(nycmaps::nyc_nta20_sf) +
  geom_sf(aes(fill = boro_name), color = "black", linewidth = 0.1) +
  scale_fill_brewer(palette = "Set2") +
  labs(fill = "Borough") +
  theme_void()

2020 Neighborhood Tabulation Areas

NTAs have recognizable names:

`1`	`nycmaps::nyc_nta20_sf \|> select(nta2020, nta_name, nta_abbrev)`

Simple feature collection with 262 features and 3 fields
Geometry type: MULTIPOLYGON
Dimension:     XY
Bounding box:  xmin: 913175.1 ymin: 120128.4 xmax: 1067383 ymax: 272844.3
Projected CRS: NAD83 / New York Long Island (ftUS)
First 10 features:
   nta2020                            nta_name nta_abbrev
1   BK0101                          Greenpoint      Grnpt
2   BK0102                        Williamsburg   Wllmsbrg
3   BK0103                  South Williamsburg  SWllmsbrg
4   BK0104                   East Williamsburg  EWllmsbrg
5   BK0201                    Brooklyn Heights      BkHts
6   BK0202 Downtown Brooklyn-DUMBO-Boerum Hill   DwntwnBk
7   BK0203                         Fort Greene      FtGrn
8   BK0204                        Clinton Hill    ClntnHl
9   BK0261                  Brooklyn Navy Yard   BkNvyYrd
10  BK0301           Bedford-Stuyvesant (West)    BdSty_W

When we have a table like this, we can get tract-level data from the Census, for example on educational attainment, and aggregate it to the NTA level. Then we can join that data to the simple feature collection that has our geometries in it. With a little polishing (which you can read all about in what I personally think of as a very useful book), we get something like this:

BA degrees or higher within NTAs.

Nominally zero-population NTAs get grayed out (JFK, LGA, various parks and cemeteries, Brooklyn Navy Yards, the United Nations, etc).

Here’s what this data looks like at the tract level:

BA degrees or higher within tracts.

Hexgrids

Sometimes we want a more schematic representation of geographies, because choropleth maps can be tricky to work with. There are many possibilities here, including not drawing maps at all. One option is to make a kind of cartogram by turning our map polygons into a tessellated grid where each unit gets a single tile. My nychex package provides hexagonal and square tilings for NTAs and tracts in New York City. Turning geographically accurate polygons into regular tiled grids can be a bit tricky, especially when the polygons you are trying to tile contain “holes”. But thanks to the tilemaps and rmapshaper packages we can get reasonably far in a semi-automated way, and then tweak things by manually nudging tiles around. Our baseline NTA hexmap looks like this:

ggplot(nychex::nyc_nta20_hex_sf) +
  geom_sf(aes(fill = boro_name), color = "black", linewidth = 0.3) +
  scale_fill_brewer(palette = "Set2") +
  labs(fill = "Borough") +
  theme_void()

The opening gameboard for my upcoming strategy game, Ticket to Ridgewood

And here is what our BA prevalence map looks like when mapped with it:

Labeled NTA hexmap

Here I’ve labeled the hexes with the (sometimes highly) abbreviated version of their NTA name. Maps like this don’t magically solve the difficulties of spatially representing population-based data, but they have their uses. They can be handy if you want to quickly get a sense of variation across units while retaining a roughly spatial layout. They also make some kinds of small-multiple or faceted plots a little easier.

We don’t have to stop at NTA-level resolution. We can do a tract-level one, too. Here’s a tract-level base hexmap:

Ticket to Ridgewood, advanced edition

This map makes a series of compromises with city geography in order to make room for each tract’s hexagon. In particular, northern Manhattan is more detached from the Bronx than is ideal, and it was also necessary to sever Brooklyn from Queens along their shared border. (Our simplify-and-tile algorithm had a very hard time with the undifferentiated Brooklyn/Queens landmass.) With a fully hand-drawn hexmap we could probably avoid most of these problems, but that would involve quite a substantial amount of work. Even when we generate the main components algorithmically, quite a bit of hand-adjustment in the overall positioning and layout is still required (especially for things like the Rockaways and other islands or quasi-islands). Sadly there’s no magic way to integrate the main borough polygons while preserving the orientation of all the hexes. Still, the result isn’t bad.

Here’s our BA map in tract-level hexagonal form:

Tract-level BA map.

The usual benefits and disadvantages of regularized choropleths are in evidence here. Lower Manhattan’s population gets a fairer shout, as do parts of Brooklyn and the Bronx. The geography still mostly works. The difficulties flow mostly from the map being at the tract-level in the first place, rather than the tiling. That is, some tracts have unusual shapes that result in quite noisy estimates. For example, sometimes a tract will consist mostly of a park or an industrial area, but have a few residential segments. This can mean it ends up being measured too high or too low on the thing we’re counting. Or, by contrast, a tract might be almost entirely one kind of entity, like a retirement home, producing results that might seem odd if you don’t know what’s in that spot. You can see why the City aims at the NTA level for a lot of its summaries. It has ten times fewer units, but things get smoothed out in a way that may be more useful. Any real-world method of measurement comes with some rate of error, which the Census helpfully provides estimates of. Nice maps tempt you to reify observations and spin yarns about what you see, whether it’s a finely-detailed spatial polygon or a pleasingly regular hexagon. The finer the observational grain, the more important it is for you to know about the situation on the ground. Literally.

Subway Sign

Sat, 28 Mar 2026 14:25:52 -0400

After the parking signs last time, here is a subway sign.

A borough-specific sign to display

Here’s a direct link to the PNG and the PDF. Once again, put it onna stick and exercise the constitutional rights to freedom of expression, speech, and assembly enjoyed by everyone in the United States.

Confessors in Harmony

Mon, 23 Mar 2026 10:54:57 -0400

Charles Fourier was an early socialist utopian, part of the French tradition of thinkers who came up with various schemes for the complete reorganization of society on more rational grounds, and whose views now read to us as an unstable admixture of obviously sensible notions, delusional crankery, and things that seem to encapsulate both of those elements at once in a way that brings out the weirder aspects of our own dominant forms of social organization. Fourierist communities, known as phalansteries, got off the ground in several countries, including a few in the United States. The basic social unit he had in mind was called a phalanx; in his mind, this would consist of 1,620 people, because of course there are 810 types of personality and you need two of each kind. A phalanstery is thus the large housing complex that houses a phalanx. Horace Greeley founded a couple, including the Sylvania Colony in Pennsylvania that eventually became the town of Greeley, PA.

A phalanstery.

Reading Fourier in his context is of course a matter of careful intellectual history, which we are going to skip here. I just wanted to make a note of a passage I came across this morning where Fourier discusses aspects of his plans for relations between the sexes. Fourier had many interesting views here that flow from his rejection of the constraints of “civilization”.¹ He coined the term “feminism”. His vision was of a kind of liberation of human creative energies and capacities of all kinds, from work to sex, that would be organized in a manner at once fulfilling to individuals and socially productive and harmonious. As is often the case in utopian visions, there’s a desire to have all of the benefits of an advanced division of labor while avoiding its overall problems and rejecting the social mechanisms we seem to have to coordinate it (which is to say, usually, an oppressive state or an open market). French utopian socialism, in particular, often ends up with schemes that are at once clearly hierarchical, or at least occupationally differentiated, highly integrated and solidaristic, but also fully accommodating of individual freedoms and liberties. A magic wand that grants some theory of human nature, or posits a general conversion to some sort of religion of science, is often waved by the author to accomplish the hard parts.

A side effect of this line of thought is that the twin beliefs in rational administration of the social order and the harmonious satisfaction of individual needs and desires can result in ideas that seem strangely familiar to us. For example, in this passage Fourier is discussing the problem of how people should pair up in phalansteries, and in particular how a traveler from one settlement to another should be able to make the best use of their time in matters of love. His answer is that we clearly need some sort of organized matching system that optimizes on the combination of people’s present sexual wants and their basic personality types:

We will now take up one of the most interesting branches of the calculus of the passions: the art of enabling anyone anywhere, even in places where he is a total stranger, to make instant contact with people with whom he is in complete sympathy. If the theory of attraction offered no other advantage, would it not still be a boon to all mankind? Would it not be a blessing to the people of civilization who often spend years in a city without encountering sympathetic partners in love, friendship or any of the passions? In Harmony any traveler will make such acquaintances on the very day of his arrival in a city. … The art of the sympathist, which is unknown in civilization, provides the means for the instant matching of personalities and sympathies, anywhere and under any circumstances.

Let us first consider the enormity of the calculations entailed by sympathetic matching and the speed with which they must be performed. … Let us suppose that a horde of one thousand adventurers and adventuresses has arrived in a Phalanx at four o’clock in the afternoon. They are immediately served light refreshments and then, even before they take time to wash, they rush off to confession. The most skillful confessors and confessoresses of the region have been gathered. Their job is to examine these thousand knights errant, each of whom must submit a written declaration concerning his or her most recent adventures. The confessors go over these declarations as well as the reports of previous confessors; they study the immediate inclinations and physical needs of each individual, and attempt to provide him or her with appropriate sympathetic relationships. Putting all the relevant information together, the confessors determine, by means of an equation, the balance of contrasts and identities that will be most attractive to each of the adventurers.

When an individual is still basking in the enthusiasm of a romantic passion, the delicious contrast provided by a sympathy in the composite mode, should the confessor intervene to provide a diversion? Yes, very likely. For someone who has just concluded a sympathetic relationship in the composite would probably be incapable of duplicating the experience immediately. Then should the confessor provide the visitor with a cabalistic liaison by offering him a choice among several candidates of identical character? Or should an appeal be made to the visitor’s alternating sympathies, his penchant for variety and contrast in both the moral and physical realms? Such are the first questions considered by the confessor. For there are three basic types of sympathies, and the initial problem is to decide which should be employed. A variation may be in order. Or it may be advisable to continue on the same scale of sympathies. For if an individual’s appetite has not been exhausted by his previous adventures, he will be capable of engaging the same sympathies in a higher or lower degree.

That is the heart of the matter; now let us turn to subordinate problems. No matter which of the three sympathies is brought into play, should it be presented directly or should it be preceded by transitions or complements? In the latter case, should the individual be subjected to a direct unitary sympathy or to an inverse unitary sympathy or perhaps even a diffracted sympathy? Should simple movement be relied upon?² This is sometimes a wise course of action, though rather inglorious, as simple movement always is; but it may be necessary in exceptional cases which will be determined by the confessors and the fairies. It is not for the individual to choose between these alternatives. He is too much absorbed by his recent memories. It is up to the calm and judicious confessor to determine the type of charm that will arouse his enthusiasm. On the basis of his declarations the confessor will decide what sort of relationship is most certain to engage his sympathies. Then the fairies with the best knowledge of the Phalanx will designate the matching individuals from whom he may make his choice.

All of this work, which involves the use of algebraic formulas, should be completed for the thousand adventurers within the short time of two or three hours. For if each visitor was not systematically informed concerning his sympathies, if there was no one to inform him about the individuals with whom he could instantaneously establish sympathetic relationships, he would run the risk of getting involved in purely sensual intrigues, intrigues wholly lacking in illusion. Like the people of civilization, he would fall back upon trivial, simple sex. (Sometimes this is necessary, but only as a respite from composite relationships or, as a transition, in moments of hesitation and of overabundant pleasure.) After a two days’ visit, at the moment of his departure, he might accidentally encounter people with whom he was truly sympathetic. Then he would regret having spent his two days without having known their merit and without having formed a liaison which might have charmed him. He would leave the Phalanx with a feeling of resentment, with bad memories instead of delicious illusions.

This goes to show that the functions of the confessor are most important and that a skilled confessor is an invaluable member of any Phalanx. The job is not one that can be confided to the first comer, as it is in civilization; it requires the greatest degree of tact, human understanding, and familiarity with local circumstances. Women will excel more than men in this sort of work, and as a rule in Harmony there will be two confessoresses for every one confessor. They will be magnificently paid for their services and promoted to the highest ranks. As for the formulas used by the confessors in their work of sympathetic matching, they cannot be explained just yet. It will first be necessary to classify the 810 personality types, any one of which may be represented in an equation of sympathy and may appear in a number of cases.

This translation is from The Utopian Vision of Charles Fourier, edited by Jonathan Beecher and Richard Bienvenu (Beacon, 1971, pp. 378–80). They note, somewhat dryly, that “Fourier does in fact go on to explain—at length—the process whereby the confessors do their matching. Unfortunately, his ‘algebraic’ formulas do not lend themselves to translation.”

Here’s a taste of that, from volume eleven of Oeuvres complètes de Charles Fourier:

I wouldn’t rely on it myself.

In any case, I like the idea of a full filing system containing an information card for each person, classified according to a system of 810 personality types, and including regularly updated information about needs and recent activities, curated and administered by experts, and which people consult upon arriving in a new town or city when they are considering how best to match up with someone it would be enjoyable to hook up with. Obviously a completely unrealistic and impossible-to-implement scheme that bears no resemblance whatsoever to how anything in contemporary society works.

Always a derogatory term for Fourier, in contrast with “Harmony”, the overall name for his utopia. ↩︎
Here he means what is in his view the simplest kind of attraction, that of opposites. ↩︎

Using Quarto to Write a Book

Mon, 09 Mar 2026 09:34:53 -0400

I’ve spent the last couple of months revising my Data Visualization book for a second edition that, ideally, will appear some time in the next twelve months. As with the first edition, I’ve posted a complete draft of the book at its website. The production process hasn’t started yet, so it’s not ready to pre-order or anything, but the site has a one-question form you can fill out that asks for your email address if you’d like to be notified with one (and only one) email when it’s available. A lot has changed since the first edition, reflecting changes both in R and ggplot specifically, and in the world of coding generally. I may end up highlighting some of those new elements in other posts. But here, I want to focus on some nerdy details involved in getting the book to its final draft. I’ll discuss Quarto, the publishing system I used, its many advantages, and its current limits with respect to the demands I made of it.

A detail from facing pages in Chapter 4, in the PDF version.

The book is about doing good data visualization using R and ggplot. The book contains many figures, almost all of which are written using the code the book shows and explains.

Reasonable Demands

My baseline list of requirements for the book manuscript was as follows:

The entire text of the book is in some kind of plain-text format.
Figures in the book that are the result of R code should be directly produced by R code in the actual document; no cutting and pasting of code snippets and separately-produced figures. Doing that is a recipe for error.
The scholarly machinery of the book—chapter, section, table, and figure numbering; cross-references; in-text bibilographical references; the bibliography itself and its formatting, and so on—should be automatically handled. No manual numbering and renumbering of figures, etc.
It should be straightforward to repeatedly generate a fully-formatted and laid-out version of the book manuscript as I go, ideally in any of several output formats (e.g. PDF, HTML), despite it all being written in plain text.

These requirements are reasonable because, for projects like this, working in plain text is the right thing to do. We are writing and revising text and our code; we keep the text in a version control system; we don’t want the results of the code to come apart from the code that generated it; and we need to deliver outputs that consist both of fully-formatted material and replication packages that allow other people do see what we did. PDF is of course the worst, but we still need to target it as one of our output formats.

Despite being reasonable, these requirements are in truth quite demanding. Once you start thinking about what all the pieces entail you realize there’s a lot to keep track of. Systems for doing some or all of this have been developed in whole or in part over the years. Newer ones sometimes escape the constraints of older ones; sometimes they inherit their legacies. I’m not going to review them here. This time around I used Quarto.

Quarto is a publishing system focused on documents of different kinds (articles, presentations, books, websites), written as plain-text sources that mix prose and code in any of several languages (R, Python, Julia, others), destined to be fully-finished outputs in any of several formats (PDFs, HTML, or Word files). Quarto builds on and extends many tools, notably pandoc for getting from Markdown to any number of other output formats. It’s a spiritual descendant of literate programming approaches for dealing with code that needs to be run in the context of prose. In the R world these descendants include Sweave and RMarkdown/knitr. These broadly “notebook” approaches to writing and discussing code have benefits and also sharp limits if your focus is full-on software development and its documentation, or complex data analysis involving many interrelated steps.¹ But they’re very useful if you are primarily writing longer-form text that periodically requires things like figures and tables to be programatically generated in a reproducible fashion.

If you just want to know whether you can write long-form projects like articles, books, or websites using Quarto and R, the answer is absolutely yes. A long time ago I wrote parts of my dissertation and several articles using Sweave. A few years ago I wrote the first edition of Data Visualization using RMarkdown. I wrote the second edition using Quarto. Each one was better than the previous version in terms of flexibility and power. Quarto eliminated several pain-points that I had to deal with for the first edition of this book. It’s very well-documented and continually improving. Its defaults are sensible and produce good-looking output. You can stop reading now.

A schematic overview of how Quarto orchestrates its document processing.

Unreasonable Demands

I had a very clear idea about how I wanted the first edition of the book to look in print. I also knew that I wanted to make it available as a website. I was fortunate enough to be able to have both of these things work out. This time around, I did the same again but I really wanted there to be as little as possible post hoc work with the website version. I knew that wouldn’t be the case with the PDF, for reasons I will discuss in a moment. I’m pleased that Quarto performed so well with the whole process. I wrote two pretty heavily-customized output formats (one for PDF and one for HTML) that specified the layout of the book. Quarto’s LaTeX-based book pipeline uses the scrbook class from the KOMA-script bundle, which has many nice features, though I find its documentation a tiny bit eccentric. (This might be because I wrote my first book using the memoir class.) I also wrote a couple of R packages that managed the themes and some other details of how PNG and especially PDF figures were produced. A version of the theme is in the development version of the socviz package that accompanies the book.

The PDF design is a two-column “Tufte-style” layout with wide margins for side-notes and figures. It works very well for a book of this kind as we can show small figures alongside the code that generates them, but also have figures break out of the main text column if needed.

Facing pages with a figure that runs the full width of one of the pages.

A layout like this can’t be rigidly ported over to a website, especially in an era of widely-varying screen sizes. So the HTML version of the book has a broadly responsive layout that arranges things differently at different sizes. Organizing and tweaking it this time around was made a lot easier by Quarto’s much better support for margin notes and marginal figures. It certainly wasn’t without its headaches. Marginal figures and notes are quite annoying to deal with in both HTML and PDF formats, for different reasons. In the PDF case, it’s tricky to get captions right, and there are still a few hacks in there to make it work. But it’s much cleaner than what I had to do in RMarkdown for the first edition, which was in effect a lot of regular expression substitution for things I could only add after the .tex file was produced. That’s gone now.

Here’s a screenshot of a facing page layout with some code, some marginal notes, and two kinds of figures, one in the margin and one full page-width:

Gapminder figures in the PDF version.

And here’s some of the same material as seen on the website:

Gapminder figures in the HTML version

Here’s a direct link to the same section. In the website version the marginal figures appear more marginal. There’s also a little bit of conflict to be worked out between the navigation guides and the marginal notes. In addition, the intrinsic variability of the web layout means that the positioning of the marginal notes is less precisely controllable than it is in the PDF output. But the overall result is pretty good. And I have to say it’s very satisfying to be able to produce a good website and a clean PDF (and also an ePub!) from the same folder of qmd files, with the text written in Markdown, the bibliography managed by Zotero and BBT, interspersed with the code that makes all the figures.

Let the Professionals do a Professional Job

I should say “less precisely controllable without substantial further adjustment”. Because this is the crux of the customization biscuit. There’s no end to it. One of the benefits of being in a position to do a second edition—something I really am very grateful for—is that it allowed me to have a much better sense of the production process for the hard copy of the book. This in turn placed sharp limits on what I was willing to do when it came to customizing the PDF version myself. Camera-ready files for books published by proper Presses are produced in many different ways. My most recent book, which is all prose and no code, was designed and typeset using Adobe InDesign. For the first edition of Data Visualization I sent the Press a set of LaTeX files and PDF image assets. The LaTeX files produced a very good facsimile of the design we’d agreed on. Then the Press’s typesetter got to work on it.

You might think that they just took my files, lightly edited them here and there, and added the trim, bleed, registration, and color marks for the physical print job. That’s not how it went. Book layouts are very hard to get just right, especially layouts that have many different-sized images and notes and other paraphernalia. They’re fragile. Moving something slightly here, or editing a sentence there, can cause a cascade of unwanted effects. Even ordinary pages of text will have issues with excessive or insufficient spacing around paragraph and section breaks, or widows and orphans, or rivers, and many other infelicities that most people won’t notice explicitly, but which cumulatively convey bad vibes even to people who don’t much care about design.

Some of this can be automated. That’s what layout algorithms do. The Knuth-Plass box-and-glue algorithm, which is the thing that causes TeX to emit those Underfull \hbox (badness 10000) complaints, is a real marvel. But it can’t quite work miracles. In my case, the professional typesetter took my LaTeX file, threw away my document class and substituted their own custom one (and some custom style files). Like any document class it defined the layout and all the features of the book, but it also included a variety of commands that allowed her to finely adjust the text as needed on any particular page. Tightening up the spacing here; forcing a break there; very slightly expanding or contracting the page size when needed to make sure that the layout didn’t break in a visible way on the next page, and so on. Here’s an example from the first edition:

And another:

Those uses of {\break}, \enlargethispage, \vspace{}, and the non-breaking space in this~way are all done by hand, based on rendering and re-rendering the document as it’s built to make sure each page meets the Press’s standards. An automatically-produced PDF can get you eighty five or ninety percent of the way to this but, if you really want to get things right, that last stretch will inevitably mean a bunch of adjustment by hand in whatever the final format is. That’s not something you can incorporate into your reproducibility pipeline.

Fortunately, you don’t need to. Most of the time we don’t require anything like that level of attention to detail. It’s worth producing and circulating material in accessible and readable formats that also don’t look like garbage. And it’s gratifying to be able to reliably generate pretty high-quality versions of those outputs from plain-text sources. That’s more than good enough in almost all cases. When writing papers that end up as PDFs, for example, I use a template that’s almost 20 years old. I only touch it when something breaks. By the same token, while my amateur interests compel me to run up polished custom Quarto book formats for my book projects, I also know that the people who set type for a living know a lot more about the fine grain of that work than I know, or need to know. But once in a while it’s nice to see how far you can push things.

The trick is to have the code chunks in your document be short and sweet, and have structured scripts and properly-documented packages manage the heavy lifting in any analysis. ↩︎

Data Visualization, Second Edition

Fri, 06 Mar 2026 06:52:36 -0500

I’ve written a second edition of Data Visualization: A Practical Introduction, which ideally should come out with Princeton University Press later this year. As with the first edition, a full draft of the book is available at https://socviz.co. The production process is just getting started so there’s no new cover yet, and there isn’t a link to pre-order. But (also like last time) I’ve put up a link to a form that lets you add your email if you’d like to be notified when it’s available to buy. You’ll only get one email (from me personally, not a marketing department) if you do; no spam or anything.

The revised edition is a pretty thorough rewrite. Naturally all the code is brought up to date for ggplot 4 and R version 4.5 and higher. The code from the first edition still runs, but you’ll get warnings and so on; those are all now gone. The back half of the book has been pretty thoroughly redone to reflect big changes in the availability of software for maps, (the sf package) and extracting results from models (the marginaleffects package). Meanwhile, several years of teaching this material (and getting feedback from others) have resulted in shifts of emphasis here and there to introduce just a little bit more on data wrangling. As the book goes on I also shift from an “object-based” approach to writing plots to a more “pipeline-based” one.

The recent rise of LLMs and coding agents gets some discussion, too. There the question is “Why can’t I just have a robot write all the code for me?” I don’t dismiss this question out of hand, and I don’t pretend that agents aren’t very powerful. My feeling about this is summed up in the Preface:

Perhaps you have a robot to help you write your code now. Large Language Models (LLMs) and coding agents are now part of the workflow of code generation and evaluation. They can do a great deal; so much so that it might seem superfluous to spend any time with the iterative, write-try-redo approach to visualization that this book presents. Can’t the robot write all the code instead? Not quite. It’s not that I believe repeatedly doing repetitive and error-prone tasks yourself is a virtue. To the contrary, that’s what computers are for. This book is full of examples where we end up automating something in order not to worry about it. But I also want you, the reader, to learn how to do good graphical work in a reproducible way. That means having a keen eye for quality and a good nose for error. Cultivating those senses requires practice and a vocabulary to express them. It seems faintly absurd to have to say explicitly but, whatever tools you use, your work will be better if you know what you are doing and understand why you are doing it. This book teaches you ggplot specifically, but it is not trying to lock you in to a particular framework. It’s just that, the way you acquire a general skill or a wide-ranging taste is by first learning some more specific version of those things, and then practicing them. Automation can come a later. In the words of the author Ann Leckie, you don’t learn how to do something by not doing it. For that reason, this book remains a hands-on introduction.

Or to put it another way, the book is an introduction to how to do something. One feature of books like it is that they tend to have two audiences: people who don’t know anything about the topic, and who’d like to learn something about it, and people who know a lot, at least in relative terms, and who have forgotten what it’s like not to know it. When the first edition came out, one of the early Amazon reviews was a complaint that the book seemed “pretty introductory” in its content. I mean, my Brother in Christ, that is right there in the title.

As with any corner of the vast division of labor that is human society, not everyone has to know about any specific thing in great detail. We’re all taking huge amounts of stuff for granted at any moment. But if you want to be proficient in some piece of that enormous web, it’s better that you know rather than not know what’s what. There’s nothing wrong with using tools that give you tremendous leverage. You do it every time you use a stand mixer in the kitchen, or a sander in the garage. You do it every time you turn your computer on, in fact. But you still need to develop the capacity to tell good work from bad, or correct from incorrect output, or safe uses from dangerous ones. That way you can take advantage of the power tools without being at risk of slicing your own or anyone else’s arm off.

Ordinal Exchanges

Thu, 13 Nov 2025 12:52:40 -0500

The Socio-Economic Review has published a book symposium on The Ordinal Society with contributions from Nitsan Chorev, J.P. Pardo-Guerra, and Greta Krippner, followed by a reply from Marion and myself. (Here’s a PDF of the exchange.) There’s also a symposium at the Journal of Cultural Economy with contributions from Hatim A. Rahman, Juan M. del Nido, Julien Migozzi, and Michelle Jackson, again with a reply from us. Finally there’s also a new review from Michael Sauder in ASQ (PDF), which follows on reviews from Barbara Kiviat in Social Forces (PDF), and Laura Nelson in Acta Sociologica.

As we say more than once in our replies, it’s very gratifying to have your peers in the field take the time to read your work.

Trustworthy Data Visualization

Mon, 10 Nov 2025 08:26:45 -0500

This past September I gave the closing keynote at posit::conf; it’s now on YouTube to watch. Keen-eyed observers will note from the title that it’s about trustworthy data visualization. But it’s also about trust a bit more generally, and how we should think about it in a world where researchers are faking results, AIs are enthusiastically confabulating, and government is destroying data infrastructure. When you find yourself giving a talk with a little tiny microphone stuck to the side of your head you have to ask yourself some hard questions, but the talk was partly about that.

One of the nice things about the opportunity to give a talk like this to a large audience is that you can tell the audience not just about your own stuff, but also the work you build from and rely on. This is, as you’ll see if you watch, one of the themes of the talk in the first place. In my case, in addition to all the good things done by people in the general area of data visualization and data science, I was able to talk about some ideas from social theory and, in addition, some of the work of Katherine Hawley on trust and commitment. The fulcrum of the talk is essentially an idea about commitment that she developed in her book How to be Trustworthy. I got to know Katherine and her family a little bit over a few summers in the 2010s when my own family used to visit St Andrews, where she was Professor of Philosophy. She died of cancer at the age of fifty in 2021. She was a wonderful person; a clear-minded philosopher of enviable intellectual gifts but also immense personal kindness. It was good to be able to think of her while writing this talk and, in however small a way, introduce some of her ideas to a new audience when I gave it.

Mamdani vs Sliwa and Cuomo

Thu, 06 Nov 2025 12:57:44 -0500

Mamdani’s victory in the New York City mayoral election gave me the opportunity to draw a few maps, and also to learn a bit about incorporating additional spatial data into maps drawn in R. R is not a specialized piece of GIS software. ESRI’s ArcGIS is the 800lb gorilla in this world and QGIS the GIMP to its Photoshop, so to speak.

Still, you can do a lot of spatial stuff in R, grounded in the sf package and its many friends. Plus you get the benefit of all the data manipulation and analysis that R is really good at. So, having gotten the precinct-level results for the election, some maps from New York City (e.g., the clipped borough boundaries map), and GTFS data from the MTA describing the structure of the subway system, I was able to draw some things. I strongly approve of the existence of the GTFS, by the way. It’s a spec for encoding transit data and lots of cities use it. Really handy.

Anyway, here’s a map. For each precinct with more than twenty voters, I combined the Sliwa/Cuomo vote into what we might call (purely for compactness reasons) the Slimo vote, calculated the Mamdani and Slimo vote shares as a proportion, and subtracted the former from the latter. That gets us a score raning from -1 to +1. I then cut that into bins, ten on each side of the zero line, to get deciles in each direction. That’s what we fill the precincts with. For the map I also drew the subway stations and lines. Several lines that are right next to each other are in effect drawn on top of one another in several places, e.g. the A and the C, or the 1 and the 3, etc, but that doesn’t matter for this map. (You may have heard that drawing transit maps meant for navigation is a really hard information design challenge.) We then use a discrete, diverging scale. Here’s the result.

Oh, choropleths

I chose a blue-green color gradient partly to experiment with it and partly because neither the election nor this particular cut of the data is quite the usual Blue vs Red, Democrat vs Republican. For one thing we have amalgamated two candidates on one side of the spectrum. For another, Cuomo is in some sense a Democrat, so the way voters were split is trickier than it normally would be. Andrew Gelman has some more thoughts on this today.

You can see a bunch of nice neighborhood patterns, such as e.g. the Hasidic communities in Brooklyn who voted strongly for Cuomo. And you can also see evidence of the characteristic weakness of choropleths, which is the way they can force a number characterizing some number of persons to be represented by a shape representing some area of space.

One solution to this is to make a dot-density map. You put a dot on the map for every person (or maybe every n people) you want to represent. Here’s what that looks like, with a 1-to-1 representation of dots to voters.

Dot dot dot.

This is better than a choropleth in some key ways. For one thing, you can see that—even in New York City—some places are both much more densely populated than others and also more likely to turn out to vote. To be clear, when I say “each dot represents one vote”, it’s not as if I know the identity of every voter, or how they voted, or can precisely locate them to their home address. I promise I don’t know that. Only the NSA and the Phone Company know that stuff. What I do know is how many votes were cast for each candidate within each of the 4,200 or so precincts. And I have a polygon that represents the shape of each precinct. So I spatially sample without replacement within each polygon to randomly place a dot for each voter within their precinct. With two million or so votes the pointillist effect ends up being quite effective.

A slice across Manhattan in the upper 50s and lower 60s and over into Queens.

The main reason for doing this is that, from the point of view of the election, precincts aren’t real. They are closely related to the social geography of the city, which is one of the reasons we want to draw a map like this, but in and of themselves they are at best proxies for the thing we care about, so we should take care not to reify them. The sampling method still brings out the basis of the data, so you can see the precinct-patchwork that forms the underlying grid, especially in densely-populated areas. But those polygons are filled in proportion to the number of people who actually voted. I saw someone on social media observe that this “conflated” partisan lean and population density. But again, precincts aren’t real. The distribution of partsian lean across population density is what we’re trying to bring out with this quasi-person-level approach.

Detail of the PDF.

There are costs, of course. Color choice is harder, especially with more than one category of dot at once. The image tends to look less bright because you’re not using a big swatch of paint across a large area. The visual impression is also very sensitive to small changes in the size of the dots and to settings like their alpha transparency. The order that the layers are drawn also matters a great deal at this sort of resolution. In addition, when you draw a few million dots on a large grid of pixels then your file size gets big real fast. The main dot-density image above is a 300dpi 4,500 by 4,500 pixel PNG and it’s 12mb in size before being crushed down further (with optipng) to 9mb or so. I could make a JPG of course, which would be a lot smaller, but then you start running into the question of why you made a dot-density map in the first place, because you lose detail in the raster. In fairness, you’d still get the benefit of having larger and less densely-populated (or lower-turnout) precincts not appear fully filled-in, even if you wouldn’t be able to zoom in at all.

Meanwhile, thanks to the wonders of multiplication, a 600dpi version of the same image is much sharper to zoom in on but is also about 35mb in size. A PDF, which is a vector rather than a raster format, is even bigger, weighing in at 63MB. Rendering a PDF with that many vector elements does not make Preview or Illustrator happy, let me tell you. The benefit of course is that you can scale it up to any size you like without loss, as shown above. Because browsers are crazy and so is javascript, it’s possible to serve up dot density maps like this in real time in a way that makes them zoomable and fluid, but that’s not my department.