Blogs on kieranhealy.org

Dogs of New York

Wed, 14 Oct 2026 13:17:28 -0400

I recently updated nycdogs, a data package for R that I originally made for teaching purposes. It bundles up New York City’s dog licensing and dog bite datasets into an R package. The data is already provided by the city in a directly usable way (as CSV files), but packaging it up makes it easier to integrate with other packages and get things moving a little more quickly in class.

The nyc_license table looks like this:

nyc_license
#> # A tibble: 819,323 × 12
#>    animal_name animal_gender animal_birth_year breed_name       breed_rc zip_code zip   license_issued_date license_expired_date extract_year borough city 
#>    <chr>       <chr>                     <int> <chr>            <chr>    <chr>    <chr> <date>              <date>                      <int> <chr>   <chr>
#>  1 Paige       F                          2014 American Pit Bu… Pit Bul… 10035    10035 2014-09-12          2017-09-12                   2016 Manhat… New …
#>  2 Yogi        M                          2010 Boxer            Boxer    10465    10465 2014-09-12          2017-10-02                   2016 Bronx   Bronx
#>  3 Ali         M                          2014 Basenji          Basenji  10013    10013 2014-09-12          2019-09-12                   2016 Manhat… New …
#>  4 Queen       F                          2013 Akita Crossbreed Akita C… 10013    10013 2014-09-12          2017-09-12                   2016 Manhat… New …
#>  5 Lola        F                          2009 Maltese          Maltese  10028    10028 2014-09-12          2017-10-09                   2016 Manhat… New …
#>  6 Ian         M                          2006 Unknown          Unknown  10013    10013 2014-09-12          2019-10-30                   2016 Manhat… New …
#>  7 Buddy       M                          2008 Unknown          Unknown  10025    10025 2014-09-12          2017-10-20                   2016 Manhat… New …
#>  8 Chewbacca   F                          2012 Labrador Retrie… Labrado… 10013    10013 2014-09-12          2019-10-01                   2016 Manhat… New …
#>  9 Heidi-bo    F                          2007 Dachshund Smoot… Dachshu… 11215    11215 2014-09-13          2017-04-16                   2016 Brookl… Broo…
#> 10 Massimo     M                          2009 Bull Dog, French French … 11201    11201 2014-09-13          2017-09-17                   2016 Brookl… Broo…
#> # ℹ 819,313 more rows

The dataset is good for teaching with for several reasons. First, people like dogs. Second, it has about 800,000 records, which is a lot in many ways but also trivial for a modern laptop to work with. Third, the data are pretty clean, but they’re not that clean. There are various things going on in there that invite us to explore it but also to be careful about what we can and can’t be confident about.

The first thing we ask about any data set like this is: what is each row in this table? In the case of nyc_license, each row is a record of one time-limited license for a dog. A row is not necessarily the record of a unique dog. When an owner renews a dog’s license, the dog will appear one time for each license period. In New York City, dog licenses can be issued for one year or five years. The table does not provide any dog_id column that ties sequences of licenses to individual dogs.

Second, the table contains several “extracts” of the license data. It seems like the way the provider updates the NYC Open Data record is to periodically pull from its internal database every year or so and append it to the existing, previously-released version. The extract_year column identifies the extract, so at least we know that. But it’s also the case that a license that was active in more than one extract year appears in each extract. As a result, the table has a large number of duplicate rows.

Third, license information is provided by dog owners. This means that there’s a fair amount of noise in the records, because people make typos and other errors. For instance, the ZIP code data has incomplete ZIPs. Some people provide invalid ZIP codes (e.g. ones that are fewer than five digits long). Others (about 3,300 records) have ZIP codes that are not in New York City. Eighty percent of these (about 2,700) are ZIP codes located elsewhere in New York state. Another 77 are located in Florida, which I kind of like. Five of them live in Boca Raton, for example. Their kids never call. You’d think it would be easy for them to just pick up the bone, I mean phone, but you never hear from them.

About 650 rows have valid ZIP codes that are in Manhattan, but these ZIPs are not ZCTAs. The trouble with ZIP codes is that people think of them as areas, as polygons, but in fact they are lines. They are defined by the Post Office as postal routes, i.e. as journeys a mail carrier takes. While they tend to form closed polygons, they are absolutely not guaranteed to. But people know their ZIP code and all kinds of analysts (from marketers to political operatives) like thinking in terms of ZIP codes. So the Census defines ZIP Code Tabulation Areas that are a kind of best-effort at answering the question “What if this ZIP code were a closed polygon?” ZCTAs exclude every P.O. box and single-address ZIP nationwide. Manhattan has several buildings that have their own ZIP code. New York City also defines MODZCTAs that merge some low- or no-population ZCTAs with their neighbors. I use these below.

Fourth, beyond the ZIP codes, owners also provide information about the age and sex of their dogs, the dog’s name, and the dog’s breed. Again, this kind of information tends to be a little noisy. For example, people vary in how they spell names. They also vary in how they classify their dog by breed. Beyond the list of “recognized” breeds (itself something that dog owners variably conform to), some people might describe their dog as, e.g., “Labrador-Sheltie mix” while others might just say “Unknown”. The dataset includes the breed_name column from the original dataset and also breed_rc, which is a recoding by me to clean up equivalent terms (“Yorkshire Terrier” and “Terrier, Yorkshire”, for example) along with a few other coding decisions.

Finally, and only because I’m choosing to stop here, the table just shows the number of licensed dogs in New York City. That is definitely not the same as the number of dogs there. Moreover, license compliance varies quite a bit in ways that are related to characteristics of the owner that we can’t observe, and also ones that we partially can, like where they live.

All of which is related to the fact that one of the lolsob-mantras of courses I teach is “Now you know the true meaning of data”. In the graphs and maps below, I’ve done the best I can, where appropriate, to uniquely identify individual dogs in the dataset (even though ultimately we cannot).

Popular Breeds

With that out of the way, it’s time to indulge your prejudices about Dogs, New Yorkers, and New York Dogs. The most common “breed” of dog in New York is “Unknown”, as befits a city of immigrants, hustlers, scrappy up-and-comers, and dogs seeking to leave their small-town past behind and remake their identities. Somewhere between eight and ten percent of records each year have “Unknown” recorded as their breed. There’s some change over time (a declining share), but I think this is an artifact of how breeds are getting recorded rather than a change in the composition of the dog population. If we leave the Unknowns out, here are the trends in annual licenses issued to the twenty most popular breeds.

Some breeds are pretty steadily popular. Smaller breeds do well in a city full of apartments. We can also see some evidence here of trends in popularity. However, these annual numbers are what we might call the prevalence of dogs, rather than the incidence of new dogs. If we want to get a better sense of fashion trends in dog breeds, we should look at puppies. (Who doesn’t want to look at puppies?) Because of the structure of the data we can’t quite do this. The best we can do, based on the dog’s age, is to examine licenses issued to dogs younger than a year old. Let’s pick out some breeds that seem to be rising, a few that seem relatively steady, and a few that might be declining. I’ve indexed these changes relative to each breed in 2016 by setting the value for that year to 100. Even though our range is narrow, we use a log scale on the y-axis because, e.g., a rise in a breed’s index from 100 to 200 reflects the same relative change as a fall from 100 to 50.

Maltipoos and Goldendoodles seem to be doing pretty well for themselves. Golden Retrievers remain steadily popular. Meanwhile, Shiba Inus, beloved fad-breed of the 2010s, seem to have been affected by their association with crypto and are strongly in decline. Much drop. Very diminish. Wow. Beagles are also doing poorly. Good grief.

Where Breeds Live

We can look at the relative prevalence of breeds by their ZIP code (strictly speaking, by their MODZCTA). I hesitate a little on this just because, even with 800,000 licenses, we’ll run out of degrees of freedom fairly quickly if we break things down by breed and ZIP. Relatedly, we should ask what sort of metric we’re trying to show here. What’s the denominator? (A surprising amount of Statistics, I claim, is essentially exploring the question “What happens if I divide by this number instead of that number?”) Popular dogs are, to a first approximation, pretty popular all over the city. Most ZIP codes have a solid contingent of Yorkies. If, for each popular breed, we calculate the share of that breed as a percentage of all dogs in the ZIP code, we’ll see some ZIP-by-ZIP variation come out.

It would also be nice to adjust for the fact that different ZIP codes have quite different shares of the total number of dogs licensed in the city. (Which, again, is not the same as share of the total number of dogs in the city.) So we calculate the within-ZIP share and make a ratio of it by dividing that number by the breed’s citywide share. The ratio is 1 when the breed is just as common in that ZIP code as it is across the whole city. If it’s 2, the breed is twice as common in that ZIP as it is across the city. If it’s 0.5, the breed is half as common in that ZIP as it is in the city as a whole. The main downside of ratios like this is that they can exaggerate differences a little, especially if the base rates are small. We have enough breeds in each zip code to make me feel OK about doing this. But if you refer to the bar charts above you can see that even for disproportionately popular breeds in zips, you’re looking at changes like “twice as common with respect to a baseline of five percent of all dogs”. Here’s the map:

We see some plausible evidence that breeds are distributed across ZIP codes in a manner that tracks the demography of owners. French Bulldogs in Chelsea, Soho, and TriBeCa. Havanese on the Upper East Side. Pit Bulls in the Bronx. German Shepherds on Staten Island. Goldendoodles and King Charles Spaniels in Manhattan south of 96th.

The Diffusion of Goldendoodles

Let’s take a closer look at the Goldendoodles. In 2015, Goldendoodles weren’t that common. We can ask: where were they initially popular? Did they diffuse across New York City from focal points where people were more likely to set trends in breed fashion? Restricting ourselves to young dogs, we can try to look at each ZIP’s share of all Goldendoodle puppies licensed each year from 2015 to 2025. Here’s what that looks like:

In 2015, Goldendoodle puppies are licensed in a relatively small number of ZIP codes, mostly in Manhattan. Most ZIP codes are doodle-free. Over the next five years, they gradually spread across the city. By 2023 or 2024, most—though still not all—ZIP codes are licensing young Goldendoodles.

Again, the underlying numbers aren’t huge. So take all of this with a delicious treat. I mean, grain of salt. I also wonder what the next few years are going to look like in terms of dog popularity. Based on the maps above, I think Australian Shepherds are likely to continue to rise in popularity amongst owners looking to acquire a fashionable breed. And while they’re not on any of the maps above, the data also suggest that the Jindo, a Korean breed that is a relatively recent arrival in the US, has been sharply on the rise. If you ask me, though, if you’re in New York City and planning to get a dog, you should throw in your lot with Unknown and get a good, solid, dog-shaped mutt. Preferably one with strong views on the state of the Subway these days, or who knows a guy who can get you a deal on some shirts, or the special ramen, or tickets. Not tickets for anything in particular. Just, like, tickets. Works out of an upstairs bar downtown on Water St. Trust me.

Steven Lukes, The Diversity of Morals

Thu, 08 Oct 2026 06:27:40 -0400

In August I was a discussant on a panel about Steven Lukes’s recent book, The Diversity of Morals. Here’s an extended version of what I said. A PDF version is available as well.

The Diversity of Morals asks the questions philosophers ask about morality: what is “the moral”? How wide and deep is its diversity? Is there a standpoint from which we can make our judgments? It then refuses to answer them by philosophical methods alone. Instead, it draws on the resources of both social theory and the social sciences, from primatology and anthropology to sociology and political science, in an effort to construct a defensible answer. I do not think it quite succeeds. But I am also unsure of what success would look like.

Before any critique, let me state the argument as I understand it. I confess that I found it a little tricky to reconstruct because of the way the book lays things out. The text is dense with quotations, sometimes quite lengthy, and prefers to proceed by picking out representatives of the various positions investigated, letting us hear them speak, and maybe selecting some piece of their argument to be partially endorsed and carried forward. It reminded me of being in an expertly-taught graduate seminar where the participants are working their way through a wide literature. In a seminar we would be able to stop at many points to discuss each new contributor’s ideas, to make connections in an expanding web of ideas. But because this is a book, at times it was a little too much like a whistle-stop tour of other people’s views. I found myself wanting a clearer sense of the author’s own argument. To be fair, Lukes acknowledges early on that he does not always take sides, and that sometimes “such reluctance may be the appropriate way to recognize the sheer difficulty of reaching a convincing conclusion” (xii).¹

Hume, Herder, and Smith

There is a positive line of argument, though. The book argues that the word “moral” names not a domain or perspective sitting alongside the legal, economic, or political, but the standpoint from which all such standpoints are judged. It’s the point of view from which we ask what the most serious, central things in life are. There is no “outside” to this standpoint. The unmaskers of morality, from Thrasymachus to Bernard Williams, are moralists in spite of themselves. The distinctively human capacity underlying it is “Smithian empathy”, that is, an effort to understand others embedded in a second-person point of view where we both demand and owe justification. This perspective-taking is not the Humean “contagion” that we share with other primates, but a kind of projection that triangulates what you would feel, what you should feel, and what an impartial “anyone” would feel (58-62; 71-3).

This Smithian empathy is used to overcome the broad contrast that the book starts out with. On the philosophical, or Humean, side is the intuition that there can be correct answers to moral questions, possibly grounded in the conviction that people are in the end all very much the same, perhaps thanks to some intrinsic human nature. On the social-scientific, or Herderian, side is the intuition that the sheer heterogeneity of moral beliefs and practices across societies makes a nonsense of efforts to derive and defend some general substantive morality. The best we can do is document its variety, and perhaps try to naturalistically explain its origins in particular cases.

This broad distinction, as the book itself makes clear, breaks down almost immediately. There are plenty of error-theorists and moral nihilists in philosophy. And in the social sciences, there is a lot of good empirical work on the most general aspects of “moral” social behavior, which generally means the emergence of generalized exchange, or altruism, or “pro-sociality”. A little more narrowly, and where many sociologists land, we have examinations of the character “moral” beliefs or actions, of “values” of all kinds and the practices or norms associated with them. These are all “social” morals, in the book’s terminology.

The Diversity of Morals agrees that the natural and social sciences can explain the emergence and persistence of morals in that second sense, but argues that they cannot naturalistically explain the universalization of morals—what the book calls “critical” morals. This is both the expansion of moral concern past every boundary of group and culture, and an evaluative attitude toward the moral perspective itself. It seems to be a contingent historical achievement. The book explores its possible sources in Axial-age reflexivity, Enlightenment dignity, and the Durkheimian sacralization of the individual in general, even as it is repeatedly threatened by counter-sacralizations that confine empathy to the bounds of nation, volk, race, or party. Here the question is how we should think about this critical moral capacity.

For Lukes, “values” are what are important, desirable, and justifiable to ourselves and others (137). The extreme test of the relation between merely social and properly critical morals is atrocities like the Holocaust and other genocides. The book resists the “situationist consensus” that says anyone in that particular situation would have acted in those ways (99-110). But it also insists that we cannot let ourselves off the hook by saying that what went on in the minds of the people who perpetrated these atrocities is completely severed from our own morality. We need a “bridgehead” of recognizable motivation or shared rationality (116). Smithian empathy has to apply here, too, somehow.

The book finds its way out with the help of the empirical literature (from Schwartz and D’Andrade) finding that personal values vary surprisingly little across societies. Practices differ, because “what counts as” realizing shared values differs widely, but the values themselves do not. And so we end with two claims. Morals narrow along a continuum from “mere” conventions to ultimate values, with the latter being more shared than might first appear. Morals also deepen, not in the sense of becoming richer or more fruitful, but in the way the ocean deepens into utter darkness: morals deepen into non-recognition, where the second-person standpoint, the need to justify one’s values to others, fails or falls away, and the moral community shrinks to the like-minded, to tribalism. While never losing its essentially sober outlook, the book does try to end somewhat optimistically, by arguing that contemporary political polarization might be successfully overcome. Much of it, Lukes suggests, is largely manufactured non-recognition rather than truly irreconcilable differences in values.

Width and Abstraction

My first worry is that the “width” thesis is an artifact of abstraction due to measurement. The claim that value diversity is narrow rests principally on D’Andrade’s study and the Schwartz surveys (D’Andrade, 2008; Schwartz & Bardi, 2001). Across the United States, Japan, and Vietnam, and across dozens of national samples, people affirm strikingly similar hierarchies of thin value-words. We all value responsibility, honesty, self-fulfillment, treating human life as precious. Within-society variation dwarfs between-society variation. The puzzle of why practices differ so much more than values is then solved with Searle’s formula: societies differ over what counts as realizing values they share (Searle, 2010).

As the book itself notes in Chapter 1, however, Nelson Goodman points out that similarity is relative to context. Whether two things count as being alike depends on what we are looking for. If we describe values thinly enough, everything converges to similarity. Every society values “responsibility”, every society values “care”. But this is not a discovery about the true shallowness of moral disagreement, it’s a feature of the level of description we are operating at. If all the moral action is in the specification—if what responsibility is, what it demands, whom it covers, is what different cultures differ on—then the formula “shared values, divergent realizations” is not an explanation of their disagreement. It is a redescription of it.

Take the book’s own abortion example. We are told both sides agree on the values of responsibility, privacy, and self-determination, and differ only in how these are linked to the world. But these rival “conceptions” of responsibility have such different practical implications that they might as well be simply rival values. The survey data can’t really arbitrate between these readings, because the surveys only ever ask about the thin word. The work of Jerolmack & Khan (2014), which the book cites approvingly, insists that values must be viewed in context, tied to norms and practices. But the book’s claims about width rely on a process of decontextualization that it elsewhere disapproves of. The book does offer “intelligibility” and “seriousness” as a means of bounding the space of valid values. But even here, it admits that an awful lot can be left over, especially if we take the time to understand why something seemingly unintelligible or unserious at first blush might turn out to be just the opposite on further reflection. Moreover, even if these criteria can delineate a space of values it doesn’t follow that this space is small. And the optimistic parts of the Coda, the parts about about deliberation and polarization, need that space to be small, rather than a large field of genuine and irreconcilable conflict. How confident are we that that the range of real value diversity is narrow?

Depth and Disagreement

A version of this question arises in the case of the depth of values, too. By the end of the book, depth is redefined as non-recognition, as an accidental or deliberate failure of Smithian empathy. But I wonder if that is what deep value conflict meant to Isaiah Berlin and Max Weber, the value pluralists discussed in Chapter 5. For Weber, at least, deep value conflict was intractability given mutual recognition. The method of verstehen is very close to the practice of Smithian empathy. The Christian of the Sermon on the Mount and the man of worldly honor, in Weber’s example, are not failing to take up each other’s second-person standpoint. The ethic of conviction and the ethic of responsibility are held by people who share a great deal in common. Weber’s claim was that the conflict between them cannot be resolved, only decided. That is the old, frightening sense of depth: irreconcilable differences among people who recognize one another, and reject what they see.

The book puts its redefinition of depth as non-recognition to work in Chapter 4, with some success. And it is necessary in the Coda as well. But I wonder whether substituting it for the old concept discreetly retires the question the old concept named. Are there rationally irresolvable conflicts among values that remain fully mutually intelligible, among people who fully recognize each other’s standing? If the answer is yes, that is, if some conflicts survive recognition, complete information, and scrupulous Smithian empathy, then deliberation, however well designed, will not dissolve them. It can only stage them. That is Weber’s position, and I do not think the book confronts it.

Chapter 5 ends with a contrast between thinking of values as merely trade-offs (Berlin), opportunity costs within choice-sets, or truly sacrifices between incommensurable ends (Weber). The Coda returns us to that question. If the depth of morality just is non-recognition, restoring recognition through some method of deliberation seems to solve the problem of depth simply by definition. In that case, experiments in deliberation aren’t just encouraging, they’re guaranteed to work. That can’t be right. If we observe preference moderation under conditions of accountability, we can interpret that as the discovery of shared values. But this is also compatible with the softening of positions that people were never deeply invested in, while genuinely deep conflicts—deep in the Weberian sense—remain untouched. The Irish case of Citizens Assemblies cited in the Coda is indeed encouraging. But, even there, one can read the Citizens’ Assembly as an already-shared value of responsibility being jointly renegotiated, or as one side prevailing under procedural conditions favorable to it.

Solving all these problems is easier the more common ground you share. The general “anyone” of Smithian empathy, the “impartial spectator”, is one way those shared values can be smuggled into the argument. The impartial spectator’s views must come from somewhere. If we disagree, and try to empathize with one another, and your impartial spectator shares many of the same views as mine, our chances of moral agreement will be much higher. If our impartial spectators seem to differ a great deal, though, things will go much worse. Again, there is some acknowledgment of this in the book. For instance, a note quoting a communication from Samuel Fleischacker, remarks that while we need to posit a “perspective-free impartiality” in order to get Smithian empathy off the ground, “We don’t need to say we have actually achieved such impartiality”, just “hold it out as a possibility to which we aspire” (224).

Metaethics and Sacralization

The author’s own values seem clear in the book. I think I share most of them. He sides with a liberal naturalism over its various rivals, with Smith over Hume on empathy, with de Swaan and Geras over the situationists and Rorty, and so on. One place the book conspicuously does not choose is between Berlin and Weber, that is, on the question of what values are, and what it really means for them to be plural. This is a hard question for the philosophers, of course, but also for social scientists who want a meaningful link between the research they do and the values they espouse.

Consider, for example, the question of the sacralization of the person. That is, the question of how the equal dignity of all humans became, in Durkheim’s sense, sacred. Critical morals (in the book’s sense) and human rights rest on that sacralization. Was its development a discovery about human beings? Or was it a contingent, collective representation that one tribe (our own) successfully consecrated? The book’s normative voice often sounds like it wants to pick the first option. It’s a discovery. Himmler’s justifications at Poznan are a failure, even an obscenity. The rescuers were right. Bezalel Smotrich’s standpoint is moral only in the anthropological sense. The counter-sacralizers and tribalists and Schmittians are wrong. And so on. Moral universalism thus has an authority that outruns its genealogy.

Durkheim, whom Lukes knows better than anyone, plainly says it’s the second. These sacralized beliefs about human dignity are, in the end, a contingent collective representation that grew out of an advancing division of labor. He says this even as he believes that the cult of the individual is something we should be committed to and take advantage of for the purposes of social policy. In its much more bleakly Nietzschean way, Weber’s view is on this side of the argument as well. Value commitments can have deep consequences for social organization, and this is why theory begins by trying to understand why people act from their own point of view. But in the end those values are not discoveries, they are primordial commitments.

This makes for a persistent tension in sociology as a discipline. Different wings of the field do of course want to advocate for particular moral views, or views with a moral element. Sometimes it’s by way of a focus on practical advocacy or policy reform, where the model of advocacy rests on claims to substantive expertise. Sometimes it explicitly rests on some standpoint, where MacIntyre’s “prison of our own upbringing” (MacIntyre, 2013) is reimagined as a source of wisdom and insight. More grandly, though these days less often, it’s by way of a claim to have grasped—and to speak for—where history itself is going. But, as a discipline, sociology is largely defined by the insight that morality as such is downstream of contingent social facts. Again, not everyone in the field agrees. But when sociologists want to emphasize what is most distinctive about their own disciplinary perspective, for the most part they reach for classics that emphasize the historical contingency of social structure, the immense diversity of social forms, and the dreaded “sociological imagination” that allegedly allows us to connect both to individual experience. In this emphasis on the variability and historicity of social life the field tends to side with anthropology (and of course disciplinary history), rather than with philosophy and economics.

When it comes to developing a position on these matters, as opposed to preferring not to think about them, The Diversity of Morals is a very useful book because it brings out just how difficult the terrain is underfoot, and just how little we have to stray from well-trodden paths in order to get bogged down trying to coherently and defensibly answer questions about the nature of morality as such, and how it relates to the findings and practice of social science. In the end, the book knows it cannot answer its biggest questions. Lukes remarks more than once that some disputes are better continued than resolved. Perhaps that is the best we can do: to at least keep talking, in a civilized manner. I will claim one extended quotation of my own. I am reminded, a little absurdly, of a scene from At the Villa of Reduced Circumstances, Alexander McCall Smith’s gentle comedy of academic manners. The scene is a ceremonial dinner at Cambridge:

The Master then rose to give a short address. ‘Dear guests of the College,’ he began, ‘dear Fellows, dear undergraduate members of this Foundation: William de Courcey was cruelly beheaded by those who could not understand that it is quite permissible for rational men to differ on important points of belief or doctrine. The world in which he lived had yet to develop those qualities of tolerance of difference of opinion which we take for granted, but which we must remind ourselves is of rather recent creation and is by no means assured of universal support. There are amongst us still those who would deny to others the right to hold a different understanding of the fundamental issues of our time. Thus, if we look about us we see people of one culture or belief still at odds with their human neighbours who are of a different culture or belief; and we see many who are prepared to act upon this difference to the extent of denying the humanity of those with whom they differ. … ‘Here in this place of learning, let us remind ourselves of the possibility of combating, in whatever small way we can, those divisions that come between man and man, between woman and woman, so that we may recognise in each other that vulnerable humanity that informs our lives, and makes life so precious; so that each may find happiness in his or her life, and in the lives of others. For what else is there for us to hope for? What else, I ask you, what else?’ (McCall Smith, 2003, pp. 56–57)

What else, indeed. Almost all of the people in these conversations are committed to something like the universalization story in their everyday lives, whether its origins lie in Axial transcendence, Kantian dignity under the shadow of God, Hans Joas’s soul and gift, Durkheimian sacredness, or something else. Philosophical error theorists and moral nihilists do not, on the whole, want to live in a state of vicious anarchy. The same goes for academic Marxists, debunking Evolutionary Psychologists, hopelessly confused Critical Realists, biogenomic bean-counters, Scandinavian DBO enumerators, tradition-marinaded MacIntyreans, authenticity-obsessed standpoint epistemologists, and have-your-cake-and-eat-it pragmatists. All of them are people you are likely to encounter at conferences. They are all in some sense already in the same conversation as you. But the problem, in the end, is that you have to make peace with your enemies, not your friends.

References

D’Andrade, R. (2008). A study of personal and cultural values: American, Japanese, and Vietnamese. London: Palgrave Macmillan.

Jerolmack, C., & Khan, S. (2014). Talk is cheap: Ethnography and the attitudinal fallacy. Sociological Methods & Research, 43(2), 178–209.

Lukes, S. (2025). The diversity of morals. Princeton: Princeton University Press.

MacIntyre, A. (2013). On having survived the academic moral philosophy of the twentieth century. In F. O’Rourke (Ed.), What happened to and in moral philosophy in the twentieth century (pp. 17–34). Notre Dame, IN: University of Notre Dame Press.

McCall Smith, A. (2003). At the villa of reduced circumstances. Edinburgh: Polygon.

Schwartz, S., & Bardi, A. (2001). Value hierarchies across cultures: Taking a similarities perspective. Journal of Cross-Cultural Psychology, 32(3), 268–290.

Searle, J. (2010). Making the Social World: The Structure of Human Civilization. New York: Oxford University Press.

Page numbers with no further bibliographical reference are to Lukes (2025). ↩︎

Trump Tourism Effects

Thu, 01 Oct 2026 20:25:09 -0400

In the past day or two on social media I kept seeing this graph from a Semafor story on the declining volume of international visitors to the United States.

Semafor Graph.

It seems to show, and was being remarked on as showing, that the second Trump administration has been responsible for a drop in visitors from abroad from some countries on a scale comparable to the effects of the Covid-19 pandemic. My first thought was “that can’t be true”. And indeed it is not true. I mean, the pandemic shut down air travel almost completely for a while. The current administration’s actions have been terrible in many ways and led to direct, negative consequences for tourism, too. But they haven’t been on that scale. Something’s wrong with the graph.

These numbers come from International Trade Administration’s I-94 arrivals data. They maintain a spreadsheet that counts all international visitors by country of residence who arrive in the U.S., spend at least one night here, are among qualified visa holders, including tourist visas and waivers and so forth. This is a monthly series. Right now, we only have data up to August of 2026, and for a couple of countries in the graph only up till June. I think—I am only guessing—that what the person who made this graph did was aggregate to yearly totals, plot those points, and then smooth them. (That’s why the y-axis values are so large in comparison to the ones on the monthly graphs I’m about to show you.) But the data for 2026 are only for half the year, hence the giant drop.

I went and got the data myself and took a look at it. Here’s the raw monthly series for the same countries as in the one above:

The monthly series.

It’s noisy because tourism is highly seasonal. Even so, the catastrophic effects of the pandemic are obvious, as is the fact that whatever is going on in the past year isn’t nearly as severe as that. Though something is going on. We can smooth the monthly data directly by calculating a trailing average, which is just a moving average except the window we’re moving across the data doesn’t extend beyond the current month. That way, we don’t illegitimately “look ahead” of the current month in a way that would make our moving average seem to anticipate the future. If we used an ordinary moving average it might seem like the trend was able to anticipate the Covid-19 downturn, for example. Here’s the result:

The smoothed monthly series.

So, no Covid-scale drop. But still, not nothing. The Canadian dip, following the trade war Trump started for no good reason, is very substantial. Just eyeballing it, it seems about twice as large as the downturn that followed the global financial crisis of the Great Recession, which is quite remarkable. It also looks like French tourism has declined, but I’d need to look a little more closely at that series by itself, as it’s on quite a different scale.

The general lesson, I suppose, is just that it’s always worth trying to cultivate some sort of rough comparative sense of how big or small an effect you should be expecting to see in your data, so that when something goes wrong (as things so often do) you’re more likely to be brought up short by it.

Speaking of which, you might also wonder about the remarkable leap in visitors from Mexico around 2004-2005 that’s visible in both the original graph and my version of it. What happened there? The answer is not some sudden shift in tourism preferences, but our old friend, measurement change. Prior to 2005, visitors from Mexico originating within 25 miles of the border weren’t counted as visitors. After 2005, they were. Hence the jump. In a smaller but still noticeable way, data for 2014 aren’t fully comparable either, because overseas one-night-stay visitors were included that year. There is some documentation of this on this old BTS page. If you want to measure change, you can’t change the measure.

Update: Now that I have this data and am messing around with it, here’s one more. This is monthly data averaged with a 12-month trailing window, showing all travel to the U.S., broken out by different “regions”. Canada and Mexico are large enough to each be their own region. Looking at things this way deliberately smooths away all the seasonality. The size of the Canadian reaction to the Trump administration’s attack on them really is remarkable. Elbows up, Canada.

All visitor travel to the U.S.

Research Workflows

Mon, 07 Sep 2026 08:55:03 -0400

I’m teaching MPTC again this semester and have been developing a few workflow slides. Here’s a popular one.

The Scientific Process.

Now if you’ll excuse me there’s a mandatory webinar I need to click through.

socviz 2.0.0 on CRAN

Fri, 21 Aug 2026 07:25:44 -0400

In anticipation of the second edition of Data Visualization, which is coming later this year from Princeton University Press, version 2.0.0 of my socviz package is now on CRAN. The update removes some functions that aren’t needed anymore and adds a couple of sample datasets.

Ihaka Lecture

Thu, 13 Aug 2026 12:09:27 -0400

I gave the first of this year’s Ihaka Lectures at the University of Auckland this past July. It revisited and partially updated things I’ve been talking about for the past eighteen months or so on trust in data visualization (and by extension in scientific work) in the light of the many challenges we presently face.

Zero Sum Problems

Thu, 21 May 2026 15:50:40 -0400

Over at Daring Fireball, John Gruber makes a passing observation about the Apple Sports app:

I’ve got some gripes about certain specific aspects of Apple Sports. Like, where does one even start to explain how much is wrong with their zero-sum visualization of team stats? Has anyone ever even seen a presentation like that before? Anyone?

That “Anyone” link lands over here. Hi everyone! The team stats image is quite confusing. It’s a summary of a game between the San Antonio Spurs and the Oklahoma City Thunder. I don’t know much about basketball, but I do know a bit about data visualization and in a pleasing coincidence my former student Josh Fink is the A-VP of Basketball Data Science for the Spurs. Here is the image that John objected to:

I had to look at it for a while as well.

I just finished driving a very long way up the side of the country, so I’m kind of tired. But even allowing for that, boy, this way of representing things really is quite confusing. Not being an Apple Sports user I had to look at it for a bit to understand what was happening. But, now that it has given me a headache, I can kind of see why whoever designed this ended up in the undoubtedly bad place they did.

Before I get to why I have some sympathy for the designer, why did I find this representation of these numbers so disorienting? It’s not just just because I’ve been driving for nine hours. John is right to call the picture a “Zero Sum” representation. The design strongly suggests to the viewer that, within each row, we’re looking at each team’s share of a total. Each pair of black and blue lines seem to be vying for control of their whole row, with the longest line being the “winner” in each case.

This sort of representation would make perfect sense for a measure that really was zero sum. Take an example from a properly good sport, like rugby. There, like in basketball, to a first approximation a team either has the ball or it doesn’t.¹ But there’s no shot clock in rugby, and possession routinely gets turned over without the game stopping. So, knowing that Team A had 65% possession is not only informative, it also immediately entails that Team B had 35%. You could show that with a representation like one of the rows above.

Literally none of the measures in the Basketball data above are zero-sum in this way. Both teams could shoot 100% from the free throw line, or zero percent. But because the first three measures shown are percentages, this reinforces the zero-sum impression given by the lines. It certainly did that in my case. But then, starting with Assists, the remaining rows are just absolute numbers. When I started looking at the absolute numbers, I got confused a second time by the length of the lines. “Oh so it’s not a share, it’s the value” I thought—but no, they do correspond in terms of relative proportions to the teams share within each row. But they’re not really shares they’re just magnitudes. But they have to be shown in a fixed space and we want to make them relatively comparable somehow so … Argh.

It would be nice if there were One Weird Trick to fully fix this figure. But I’m not sure that there is. For example, at a minimum we could redraw these numbers to reflect the fact that they’re not zero-sum. Keep each measure as a row (i.e. on the y-axis) but have the lines, or columns, be side by side within each category instead of facing off. Like this:

Team Stats side by side for each measure.

This view at least lets you immediately see who “won” each measure. The viewer can just directly compare the length of the bars in each category. People are really good at doing that accurately. In that sense it’s much less confusing than the original. But there’s still a lot wrong with it. The core problem is that when we draw a graph like this, we’re usually putting the same kind of thing (e.g. countries, or religious groups, or sports teams) on the y-axis, and then seeing how different their scores are on some single measure (e.g. GDP, or number of adherents, or average points scored per game), which we put on the x-axis. Maybe we use color to break things out by some third measure as well.² In this case, I’ve just labeled the x-axis as generically as possible. “Value” covers the range of all the measures. The lowest value is 5, in Largest Lead. The highest is 88, in Free Throw %. But these numbers are not meaningfully comparable. The graph encourages us to compare across as well as within categories. But while within-category comparisons are meaningful, the between-category ones are not. There were way more Bench Points than Blocks in the game. But that is not a useful thing to know.

Team Stats side by side and ordered from absolute highest to lowest, whatever that means.

Knowing who won each measure isn’t nothing. It can be informative about how the game went, maybe especially when a team won the game but “lost” on a number of the measures. If you really wanted to lean in to that aspect, you could sort of justify the zero-sum view, and maybe look for a way to sort and order by “how much” a team “won” each category. But again, what’s the right denominator for those measures? For instance, do we care about a team’s share of all Defensive Rebounds in the game? Or do we care about the share of Defensive Rebounds a team won relative to every opportunity it had to make a Defensive Rebound? How meaningful is ordering our rows by those kinds of shares? Even worse, some measures (notably Fouls) are bad to “win”, so we’d have to do something about those.

Our fundamental problem is that we just have two cases (the teams) and fifteen different measures, or variables. Each variable, except for the three percentages, is in effect on its own scale. There’s no direct way to make comparisons across them. Sure, some of these measures are probably going to be associated with one another—e.g. Turnovers and Points Off Turnovers—but the numeric values aren’t directly comparable in general. If you know a lot about basketball you might have some informative rules of thumb about each one of these measures, or some of them in combination. But at that point the lines in this particular graph are not going to be doing any work for you; you’ll just end up looking directly at the numbers. If we had data on all these measures for every NBA game for a whole season then we could of course do much more with them, because then each measure would have a distribution across all games and across all teams.

As it is, the purpose of the “Stats” screen in Apple Sports is just to summarize information from a single game. The other thing I could think of to do with the numbers as kind of graph is something like this:

A back-to-back column chart.

This is marginally more helpful than the one before just because, again, it gets rid of the unhelpful zero-sum look of the original. As I hope you can immediately see, it creates many other difficulties. It also doesn’t do away with the core problem. That problem is principally one of information design rather than data visualization. What I mean is that what we’re trying to organize is, in effect, fifteen pairs of related but fundamentally distinct numbers. If we had fifteen cases and two variables things would be simple. But with fifteen variables and two cases … well, this is not the kind of thing you can make a single effective and non-confusing graph out of. That’s why I kind of sympathize with the designer. In a constrained space they have to show thirty numbers (thirty two, including the score). Lots of information. A straight table seems like it would be boring. Surely there’s some way to thematically integrate the numbers in a visually appealing manner that brings out some of the relationships across the rows. That’s what graphs do; it seems like the right thing to reach for. But at its heart this information is not a graph. It just sort of looks like one, and that ends up confusing people.

Modulo some measurement decisions about how to determine when possession is turned over while the ball is in play. ↩︎
Here’s an example of a graph with a categorical measure on the y-axis, a continuous measure on the x-axis, and an additional categorical feature shown with color. ↩︎

Bad Weather and the Subway

Sat, 02 May 2026 08:59:15 -0400

Snow in Inwood, New York. Photograph by the author.

Recently I’ve been looking at hourly ridership data from the New York City Subway. Last time we learned that people go to work in the morning and come home in the evening, for example. (All together now: “Only in New York, baby!”) Today we’ll learn that bad weather makes people stay at home. Except, sometimes it doesn’t.

Regular readers will recall that the subway system carries a lot of passengers every day. The ridership data for the whole of 2025 represents just over 1.3 billion entries into the system via an OMNY tap or Metrocard. It’s available aggregated to hourly resolution by station complex. With that data in hand, we can calculate average hourly ridership for every day of the week. This gives us a profile of what, for example, a Monday or a Wednesday typically looks like, by hour. When calculating the average day-of-the week profile we exclude holidays and the like.

Meanwhile, the National Weather Service provides data on severe weather events that affected the New York City region in 2025. We could get more fine-grained if we wanted to, but for now we’ll just use the general list of events the NWS provides. Then we plot the Subway ridership profile for that specific date against the average profile for whatever day of the week the event happened on.

Bad weather suppresses Subway ridership, in general. But not always.

The gray lines are the baseline. The red ones are the bad weather day. The basic shape of the gray lines (and many of the red ones) is set by the rhythm of daily life. The sharp double-peak pattern is what someone I’ve shown too many of these graphs to has taken to calling “The Giant Cat-Ears of Employment”. The cat-ear shapes vary by work day (which might be the topic of another post), but are most sharply-contrasted with the weekends, which look more like little hillocks or drumlins.

We can see a few different cases in the panels. First are days when the weather event put no dent at all in people’s day. This is because ~~of the incredible toughness and resilience of New Yorkers, something they are surprisingly very modest about~~ even though there was a weather event in the region that day, it just didn’t impinge on the city much, or at all. The light snow on February 11th or the heavy rain on March 6th are examples here. People just continued to go about their business.

Second are cases where there’s a lot of travel suppression but it’s not really—or not wholly—the weather that’s responsible. The winter storm on Friday December 26th is a case of this. Bad weather; strongly suppressed travel profile; but that’s not a regular Friday. Many people were staying at home anyway, because it’s the day after Christmas.

Third are cases where the weather does seem to have suppressed travel. These are days like the snow on January 19th, or the shitty weather on Sunday February 16th. These events look like they made people stay at home. Some of these are more severe than others. The strongest example is the flash flooding on Thursday July 31st. That happened in the back half of the day and affected the evening commute directly.

Our fourth and final category is my favorite one. Sometimes snow makes no difference at all, especially if it’s on a workday. Sometimes it’s snowy on the weekend but you’re kind of sick of it, maybe because it’s late in the winter, so you’re either going about your business as usual or you’re just staying indoors. But there’s another kind of snow day.

Let’s go exploring.

The weekend of December 13th and 14th 2025 brought the city’s first measurable snow of the year, and in decent amounts, too—between four and eight inches of accumulation. Reports remarked on how long it had been in arriving. The result was that, over the weekend, ridership on the subway went up. Maybe on the Saturday it was to go out and buy the mandatory bread, milk, and eggs.¹ But maybe it was also just to be out in the snow. The next day, the people who didn’t have to go work slept in as usual. But that day, too, across the afternoon, more people than usual headed outside and took the subway somewhere. I’d like to think a bunch of them had a sled under their arm.

Maybe, at least for some New Yorkers, it was because it made more sense to take the subway than drive. Though this probably wouldn’t be all that many people. It’d be somewhat possible to investigate this with the data at hand, especially if e.g. outlying stations showed higher ridership rates. ↩︎

Hourly Subway Station Flows

Sat, 25 Apr 2026 11:12:39 -0400

Pie charts are bad, as any fule kno. We’re not as good at judging relative differences between angles and areas as we are at judging relative differences in lengths on a common baseline. This is especially true when we have more than two things to compare at the same time. So, as a rule, you shouldn’t use them. You should figure out some other way of viewing your data instead. On the other hand, I just made 424 animated pie charts because if you’re going to break a rule you should break it good and hard.

A view of the New York City Subway System (excluding the SIR). We’ll animate this in just a minute.

The New York City Subway system is very large and carries a lot of passengers every day. The MTA makes quite a bit of data available about the subway, including data on hourly flow through the system. Now, the MTA can’t track individual pathways people take through the subway. If you use an OMNY card (or before that, a Metrocard) to enter the system, this signals the start of a trip from some specific station or station complex. But unlike some systems, you don’t need to “tag out” of the subway, you just exit through a turnstile. So the system doesn’t know where you exit it. In addition, while many stations are just on a single line, some (like 34 St/Penn Station, or Fulton Street) are station complexes that serve many lines and allow transfers between them.

However, the MTA does publish hourly Origin-Destination estimates for all pairs of stations. These are their best guess about the flow of traffic from any particular station to any other. Because there are so many combinations, visualizing that sort of data is quite tricky. Even then, you don’t get information about routes through the system, just start and end points. Transit analysts and planners can go further by introducing some further assumptions about subway users. For example we might assume that commuters take the most efficient route between any given pair of entry and exit stations, and build from there to a picture of flow through the system.

I do something rather more simple here. I use the MTA’s hourly origin-destination estimates and aggregate them on a station-by-station basis to calculate in-and-out flows across 424 subway stations or station complexes. These specific numbers are averaged over all Mondays in 2025. For each hour of the we calculate the total passenger volume at the station, and the share of that volume that are estimated arrivals and departures. Then we draw a pie chart for each station, coloring it yellow for departures, purple for arrivals. The circle size reflects total volume and the pie slice proportions show the flow balance.

The flow data is pretty bulky. The original dataset has about 121 million rows. But working with it is pretty straightforward, thanks to the magic of parquet files, duckdb, and duckplyr. Having patiently downloaded the data via its API, I put it in a parquet file. The CSV is about 17GB but the parquet file boils it down to 1.5GB. Then I made a small R package that bundled that data with a few convenience functions. This lets me use the data without copying it into any single project. So I can write, e.g.,

nycsubwayodr::nyc_subway_odr()
#> # A duckplyr data frame: 15 variables
#>     year month day_of_week hour_of_day timestamp           day_of_month origin_station_complex_id
#>    <int> <int> <chr>             <int> <dttm>                     <int>                     <int>
#>  1  2025     1 Monday                1 2025-01-06 01:00:00            6                       189
#>  2  2025     1 Monday                1 2025-01-06 01:00:00            6                       313
#>  3  2025     1 Monday                1 2025-01-06 01:00:00            6                       611
#>  4  2025     1 Monday                1 2025-01-06 01:00:00            6                       125
#>  5  2025     1 Monday                1 2025-01-06 01:00:00            6                       313
#>  6  2025     1 Monday                1 2025-01-06 01:00:00            6                       154
#>  7  2025     1 Monday                1 2025-01-06 01:00:00            6                       167
#>  8  2025     1 Monday                1 2025-01-06 01:00:00            6                       612
#>  9  2025     1 Monday                1 2025-01-06 01:00:00            6                       272
#> 10  2025     1 Monday                1 2025-01-06 01:00:00            6                       167
#> # ℹ more rows
#> # ℹ 8 more variables: origin_station_complex_name <chr>, origin_latitude <dbl>, origin_longitude <dbl>,
#> #   destination_station_complex_id <int>, destination_station_complex_name <chr>,
#> #   destination_latitude <dbl>, destination_longitude <dbl>, estimated_average_ridership <dbl>

From there, we lazily query the data and duckdb does the work of doing the calculations. The whole table is never loaded into your R session, and duckdb is very fast. From there, we take our hourly flow summaries, join them to a tibble of station and line data, and export the result to some JSON files that D3js animates for us.

Here’s the result. There are three views. Initially, you see just the schematic subway map. If you click the “Map” button in the top left, it will switch to the ticking pie-chart view, which puts a pie on every station complex, with each tick being an hour of the day. The pies pile up on one another in the geographic view (in a not wholly uninformative way), but click again to have them expand to a somewhat more abstracted, force-directed network view of the system. Then click again to go back to the map. You can hover over or tap on nodes to get information about the bit of data it’s currently showing.

Now, you might reasonably say, Kieran, that’s a lot of data to show that people go to work in the morning and come home in the evening. I’m not saying there’s nothing to that criticism. But there are quite a few interesting details in there as the data pick up traffic to different parts of town. The big interchanges naturally dominate the view, but even here there are things of interest about the balance of flow, as e.g. Penn Station has people coming in on New Jersey Transit during morning rush hour and then entering the subway, which does a lot to balance its net flow during rush-hour and even tip it towards net departures. But more importantly, who doesn’t want to sit back and contemplate more than 400 pie charts, each one pulsing with life as another hour ticks by?

New York City Hexmaps

Sun, 19 Apr 2026 10:05:54 -0400

The five boroughs of New York City can be informally or formally carved up into many different pieces, depending on what it is that you’re doing. As part of an ongoing project, I recently made an R package, nycmaps, that lets you draw maps of some of these geographies. Things being what they are, these spatial units don’t necessarily overlap in compatible ways. City, State, and Congressional Districts, School Districts, Police Precincts, Fire Companies, Election Precincts, Municipal Court Districts, Zip Codes … there are loads of them. Some of them are quite straightforward; others patiently lie in wait to trap unwary analysts (I’m looking at you, Zip Codes / ZCTAs).

Mapping Tracts and NTAs

Two classifications of particular interest to people like me are Census Tracts and Neighborhood Tabulation Areas (NTAs). Census Tracts are defined by the Census Bureau and form part of a nested set of geographical units that go from the smallest unit the Census keeps track of (the Block) up to the largest (the whole country). Blocks aggregate to Block groups, Block groups aggregate to Tracts. Tracts aggregate to Counties. There are of course several complications. Ideally, the Census would like tracts to be contiguous, sub-county geographical areas with about 4,000 people in them, or at least between 1,200 and 8,000 people. This means tracts can vary considerably in geographical area. They generally follow visible features of the environment, whether physical or built. Uninhabited areas also get tract designations, so in principle we can get a full tract-level map of an area with no gaps. Here’s what a tract-level map of New York City looks like:

ggplot(nycmaps::nyc_census_tracts_2020_sf) +
  geom_sf(aes(fill = boro_name), color = "black", linewidth = 0.1) +
  scale_fill_brewer(palette = "Set2") +
  labs(fill = "Borough") +
  theme_void()

2020 NYC Census Tract boundaries.

You can see the variation in tract size (compare e.g. Staten Island tracts with those in lower Manhattan). And you can also see features that are included on the map but, at least to a first approximation, don’t have permanent residents. That big roundy blob in the southeast corner of Queens, for instance, is JFK Airport. There are about 2,300 Census Tracts in New York City. (Naturally, their number and spatial layout changes from decennial census to decennial census, because why should life be easy?)

Neighborhood Tabulation Areas, meanwhile, are not official Census units. They are one of several subdivisions used by New York City government. The idea is to aggregate tracts into units that roughly correspond to neighborhoods that people conventionally refer to. This is, of course, an impossible task, because people don’t agree on neighborhood boundaries. But the idea is good. You want something bigger than a tract because those are small enough to be noisy on many measures produced by the main source of tract-level data, the American Community Survey. But you want something smaller than the next level up, which is a Community District Tabulation Area. Presently, there are 262 NTAs. They look like this:

ggplot(nycmaps::nyc_nta20_sf) +
  geom_sf(aes(fill = boro_name), color = "black", linewidth = 0.1) +
  scale_fill_brewer(palette = "Set2") +
  labs(fill = "Borough") +
  theme_void()

2020 Neighborhood Tabulation Areas

NTAs have recognizable names:

`1`	`nycmaps::nyc_nta20_sf \|> select(nta2020, nta_name, nta_abbrev)`

Simple feature collection with 262 features and 3 fields
Geometry type: MULTIPOLYGON
Dimension:     XY
Bounding box:  xmin: 913175.1 ymin: 120128.4 xmax: 1067383 ymax: 272844.3
Projected CRS: NAD83 / New York Long Island (ftUS)
First 10 features:
   nta2020                            nta_name nta_abbrev
1   BK0101                          Greenpoint      Grnpt
2   BK0102                        Williamsburg   Wllmsbrg
3   BK0103                  South Williamsburg  SWllmsbrg
4   BK0104                   East Williamsburg  EWllmsbrg
5   BK0201                    Brooklyn Heights      BkHts
6   BK0202 Downtown Brooklyn-DUMBO-Boerum Hill   DwntwnBk
7   BK0203                         Fort Greene      FtGrn
8   BK0204                        Clinton Hill    ClntnHl
9   BK0261                  Brooklyn Navy Yard   BkNvyYrd
10  BK0301           Bedford-Stuyvesant (West)    BdSty_W

When we have a table like this, we can get tract-level data from the Census, for example on educational attainment, and aggregate it to the NTA level. Then we can join that data to the simple feature collection that has our geometries in it. With a little polishing (which you can read all about in what I personally think of as a very useful book), we get something like this:

BA degrees or higher within NTAs.

Nominally zero-population NTAs get grayed out (JFK, LGA, various parks and cemeteries, Brooklyn Navy Yards, the United Nations, etc).

Here’s what this data looks like at the tract level:

BA degrees or higher within tracts.

Hexgrids

Sometimes we want a more schematic representation of geographies, because choropleth maps can be tricky to work with. There are many possibilities here, including not drawing maps at all. One option is to make a kind of cartogram by turning our map polygons into a tessellated grid where each unit gets a single tile. My nychex package provides hexagonal and square tilings for NTAs and tracts in New York City. Turning geographically accurate polygons into regular tiled grids can be a bit tricky, especially when the polygons you are trying to tile contain “holes”. But thanks to the tilemaps and rmapshaper packages we can get reasonably far in a semi-automated way, and then tweak things by manually nudging tiles around. Our baseline NTA hexmap looks like this:

ggplot(nychex::nyc_nta20_hex_sf) +
  geom_sf(aes(fill = boro_name), color = "black", linewidth = 0.3) +
  scale_fill_brewer(palette = "Set2") +
  labs(fill = "Borough") +
  theme_void()

The opening gameboard for my upcoming strategy game, Ticket to Ridgewood

And here is what our BA prevalence map looks like when mapped with it:

Labeled NTA hexmap

Here I’ve labeled the hexes with the (sometimes highly) abbreviated version of their NTA name. Maps like this don’t magically solve the difficulties of spatially representing population-based data, but they have their uses. They can be handy if you want to quickly get a sense of variation across units while retaining a roughly spatial layout. They also make some kinds of small-multiple or faceted plots a little easier.

We don’t have to stop at NTA-level resolution. We can do a tract-level one, too. Here’s a tract-level base hexmap:

Ticket to Ridgewood, advanced edition

This map makes a series of compromises with city geography in order to make room for each tract’s hexagon. In particular, northern Manhattan is more detached from the Bronx than is ideal, and it was also necessary to sever Brooklyn from Queens along their shared border. (Our simplify-and-tile algorithm had a very hard time with the undifferentiated Brooklyn/Queens landmass.) With a fully hand-drawn hexmap we could probably avoid most of these problems, but that would involve quite a substantial amount of work. Even when we generate the main components algorithmically, quite a bit of hand-adjustment in the overall positioning and layout is still required (especially for things like the Rockaways and other islands or quasi-islands). Sadly there’s no magic way to integrate the main borough polygons while preserving the orientation of all the hexes. Still, the result isn’t bad.

Here’s our BA map in tract-level hexagonal form:

Tract-level BA map.

The usual benefits and disadvantages of regularized choropleths are in evidence here. Lower Manhattan’s population gets a fairer shout, as do parts of Brooklyn and the Bronx. The geography still mostly works. The difficulties flow mostly from the map being at the tract-level in the first place, rather than the tiling. That is, some tracts have unusual shapes that result in quite noisy estimates. For example, sometimes a tract will consist mostly of a park or an industrial area, but have a few residential segments. This can mean it ends up being measured too high or too low on the thing we’re counting. Or, by contrast, a tract might be almost entirely one kind of entity, like a retirement home, producing results that might seem odd if you don’t know what’s in that spot. You can see why the City aims at the NTA level for a lot of its summaries. It has ten times fewer units, but things get smoothed out in a way that may be more useful. Any real-world method of measurement comes with some rate of error, which the Census helpfully provides estimates of. Nice maps tempt you to reify observations and spin yarns about what you see, whether it’s a finely-detailed spatial polygon or a pleasingly regular hexagon. The finer the observational grain, the more important it is for you to know about the situation on the ground. Literally.

Subway Sign

Sat, 28 Mar 2026 14:25:52 -0400

After the parking signs last time, here is a subway sign.

A borough-specific sign to display

Here’s a direct link to the PNG and the PDF. Once again, put it onna stick and exercise the constitutional rights to freedom of expression, speech, and assembly enjoyed by everyone in the United States.

Confessors in Harmony

Mon, 23 Mar 2026 10:54:57 -0400

Charles Fourier was an early socialist utopian, part of the French tradition of thinkers who came up with various schemes for the complete reorganization of society on more rational grounds, and whose views now read to us as an unstable admixture of obviously sensible notions, delusional crankery, and things that seem to encapsulate both of those elements at once in a way that brings out the weirder aspects of our own dominant forms of social organization. Fourierist communities, known as phalansteries, got off the ground in several countries, including a few in the United States. The basic social unit he had in mind was called a phalanx; in his mind, this would consist of 1,620 people, because of course there are 810 types of personality and you need two of each kind. A phalanstery is thus the large housing complex that houses a phalanx. Horace Greeley founded a couple, including the Sylvania Colony in Pennsylvania that eventually became the town of Greeley, PA.

A phalanstery.

Reading Fourier in his context is of course a matter of careful intellectual history, which we are going to skip here. I just wanted to make a note of a passage I came across this morning where Fourier discusses aspects of his plans for relations between the sexes. Fourier had many interesting views here that flow from his rejection of the constraints of “civilization”.¹ He coined the term “feminism”. His vision was of a kind of liberation of human creative energies and capacities of all kinds, from work to sex, that would be organized in a manner at once fulfilling to individuals and socially productive and harmonious. As is often the case in utopian visions, there’s a desire to have all of the benefits of an advanced division of labor while avoiding its overall problems and rejecting the social mechanisms we seem to have to coordinate it (which is to say, usually, an oppressive state or an open market). French utopian socialism, in particular, often ends up with schemes that are at once clearly hierarchical, or at least occupationally differentiated, highly integrated and solidaristic, but also fully accommodating of individual freedoms and liberties. A magic wand that grants some theory of human nature, or posits a general conversion to some sort of religion of science, is often waved by the author to accomplish the hard parts.

A side effect of this line of thought is that the twin beliefs in rational administration of the social order and the harmonious satisfaction of individual needs and desires can result in ideas that seem strangely familiar to us. For example, in this passage Fourier is discussing the problem of how people should pair up in phalansteries, and in particular how a traveler from one settlement to another should be able to make the best use of their time in matters of love. His answer is that we clearly need some sort of organized matching system that optimizes on the combination of people’s present sexual wants and their basic personality types:

We will now take up one of the most interesting branches of the calculus of the passions: the art of enabling anyone anywhere, even in places where he is a total stranger, to make instant contact with people with whom he is in complete sympathy. If the theory of attraction offered no other advantage, would it not still be a boon to all mankind? Would it not be a blessing to the people of civilization who often spend years in a city without encountering sympathetic partners in love, friendship or any of the passions? In Harmony any traveler will make such acquaintances on the very day of his arrival in a city. … The art of the sympathist, which is unknown in civilization, provides the means for the instant matching of personalities and sympathies, anywhere and under any circumstances.

Let us first consider the enormity of the calculations entailed by sympathetic matching and the speed with which they must be performed. … Let us suppose that a horde of one thousand adventurers and adventuresses has arrived in a Phalanx at four o’clock in the afternoon. They are immediately served light refreshments and then, even before they take time to wash, they rush off to confession. The most skillful confessors and confessoresses of the region have been gathered. Their job is to examine these thousand knights errant, each of whom must submit a written declaration concerning his or her most recent adventures. The confessors go over these declarations as well as the reports of previous confessors; they study the immediate inclinations and physical needs of each individual, and attempt to provide him or her with appropriate sympathetic relationships. Putting all the relevant information together, the confessors determine, by means of an equation, the balance of contrasts and identities that will be most attractive to each of the adventurers.

When an individual is still basking in the enthusiasm of a romantic passion, the delicious contrast provided by a sympathy in the composite mode, should the confessor intervene to provide a diversion? Yes, very likely. For someone who has just concluded a sympathetic relationship in the composite would probably be incapable of duplicating the experience immediately. Then should the confessor provide the visitor with a cabalistic liaison by offering him a choice among several candidates of identical character? Or should an appeal be made to the visitor’s alternating sympathies, his penchant for variety and contrast in both the moral and physical realms? Such are the first questions considered by the confessor. For there are three basic types of sympathies, and the initial problem is to decide which should be employed. A variation may be in order. Or it may be advisable to continue on the same scale of sympathies. For if an individual’s appetite has not been exhausted by his previous adventures, he will be capable of engaging the same sympathies in a higher or lower degree.

That is the heart of the matter; now let us turn to subordinate problems. No matter which of the three sympathies is brought into play, should it be presented directly or should it be preceded by transitions or complements? In the latter case, should the individual be subjected to a direct unitary sympathy or to an inverse unitary sympathy or perhaps even a diffracted sympathy? Should simple movement be relied upon?² This is sometimes a wise course of action, though rather inglorious, as simple movement always is; but it may be necessary in exceptional cases which will be determined by the confessors and the fairies. It is not for the individual to choose between these alternatives. He is too much absorbed by his recent memories. It is up to the calm and judicious confessor to determine the type of charm that will arouse his enthusiasm. On the basis of his declarations the confessor will decide what sort of relationship is most certain to engage his sympathies. Then the fairies with the best knowledge of the Phalanx will designate the matching individuals from whom he may make his choice.

All of this work, which involves the use of algebraic formulas, should be completed for the thousand adventurers within the short time of two or three hours. For if each visitor was not systematically informed concerning his sympathies, if there was no one to inform him about the individuals with whom he could instantaneously establish sympathetic relationships, he would run the risk of getting involved in purely sensual intrigues, intrigues wholly lacking in illusion. Like the people of civilization, he would fall back upon trivial, simple sex. (Sometimes this is necessary, but only as a respite from composite relationships or, as a transition, in moments of hesitation and of overabundant pleasure.) After a two days’ visit, at the moment of his departure, he might accidentally encounter people with whom he was truly sympathetic. Then he would regret having spent his two days without having known their merit and without having formed a liaison which might have charmed him. He would leave the Phalanx with a feeling of resentment, with bad memories instead of delicious illusions.

This goes to show that the functions of the confessor are most important and that a skilled confessor is an invaluable member of any Phalanx. The job is not one that can be confided to the first comer, as it is in civilization; it requires the greatest degree of tact, human understanding, and familiarity with local circumstances. Women will excel more than men in this sort of work, and as a rule in Harmony there will be two confessoresses for every one confessor. They will be magnificently paid for their services and promoted to the highest ranks. As for the formulas used by the confessors in their work of sympathetic matching, they cannot be explained just yet. It will first be necessary to classify the 810 personality types, any one of which may be represented in an equation of sympathy and may appear in a number of cases.

This translation is from The Utopian Vision of Charles Fourier, edited by Jonathan Beecher and Richard Bienvenu (Beacon, 1971, pp. 378–80). They note, somewhat dryly, that “Fourier does in fact go on to explain—at length—the process whereby the confessors do their matching. Unfortunately, his ‘algebraic’ formulas do not lend themselves to translation.”

Here’s a taste of that, from volume eleven of Oeuvres complètes de Charles Fourier:

I wouldn’t rely on it myself.

In any case, I like the idea of a full filing system containing an information card for each person, classified according to a system of 810 personality types, and including regularly updated information about needs and recent activities, curated and administered by experts, and which people consult upon arriving in a new town or city when they are considering how best to match up with someone it would be enjoyable to hook up with. Obviously a completely unrealistic and impossible-to-implement scheme that bears no resemblance whatsoever to how anything in contemporary society works.

Always a derogatory term for Fourier, in contrast with “Harmony”, the overall name for his utopia. ↩︎
Here he means what is in his view the simplest kind of attraction, that of opposites. ↩︎

Using Quarto to Write a Book

Mon, 09 Mar 2026 09:34:53 -0400

I’ve spent the last couple of months revising my Data Visualization book for a second edition that, ideally, will appear some time in the next twelve months. As with the first edition, I’ve posted a complete draft of the book at its website. The production process hasn’t started yet, so it’s not ready to pre-order or anything, but the site has a one-question form you can fill out that asks for your email address if you’d like to be notified with one (and only one) email when it’s available. A lot has changed since the first edition, reflecting changes both in R and ggplot specifically, and in the world of coding generally. I may end up highlighting some of those new elements in other posts. But here, I want to focus on some nerdy details involved in getting the book to its final draft. I’ll discuss Quarto, the publishing system I used, its many advantages, and its current limits with respect to the demands I made of it.

A detail from facing pages in Chapter 4, in the PDF version.

The book is about doing good data visualization using R and ggplot. The book contains many figures, almost all of which are written using the code the book shows and explains.

Reasonable Demands

My baseline list of requirements for the book manuscript was as follows:

The entire text of the book is in some kind of plain-text format.
Figures in the book that are the result of R code should be directly produced by R code in the actual document; no cutting and pasting of code snippets and separately-produced figures. Doing that is a recipe for error.
The scholarly machinery of the book—chapter, section, table, and figure numbering; cross-references; in-text bibilographical references; the bibliography itself and its formatting, and so on—should be automatically handled. No manual numbering and renumbering of figures, etc.
It should be straightforward to repeatedly generate a fully-formatted and laid-out version of the book manuscript as I go, ideally in any of several output formats (e.g. PDF, HTML), despite it all being written in plain text.

These requirements are reasonable because, for projects like this, working in plain text is the right thing to do. We are writing and revising text and our code; we keep the text in a version control system; we don’t want the results of the code to come apart from the code that generated it; and we need to deliver outputs that consist both of fully-formatted material and replication packages that allow other people do see what we did. PDF is of course the worst, but we still need to target it as one of our output formats.

Despite being reasonable, these requirements are in truth quite demanding. Once you start thinking about what all the pieces entail you realize there’s a lot to keep track of. Systems for doing some or all of this have been developed in whole or in part over the years. Newer ones sometimes escape the constraints of older ones; sometimes they inherit their legacies. I’m not going to review them here. This time around I used Quarto.

Quarto is a publishing system focused on documents of different kinds (articles, presentations, books, websites), written as plain-text sources that mix prose and code in any of several languages (R, Python, Julia, others), destined to be fully-finished outputs in any of several formats (PDFs, HTML, or Word files). Quarto builds on and extends many tools, notably pandoc for getting from Markdown to any number of other output formats. It’s a spiritual descendant of literate programming approaches for dealing with code that needs to be run in the context of prose. In the R world these descendants include Sweave and RMarkdown/knitr. These broadly “notebook” approaches to writing and discussing code have benefits and also sharp limits if your focus is full-on software development and its documentation, or complex data analysis involving many interrelated steps.¹ But they’re very useful if you are primarily writing longer-form text that periodically requires things like figures and tables to be programatically generated in a reproducible fashion.

If you just want to know whether you can write long-form projects like articles, books, or websites using Quarto and R, the answer is absolutely yes. A long time ago I wrote parts of my dissertation and several articles using Sweave. A few years ago I wrote the first edition of Data Visualization using RMarkdown. I wrote the second edition using Quarto. Each one was better than the previous version in terms of flexibility and power. Quarto eliminated several pain-points that I had to deal with for the first edition of this book. It’s very well-documented and continually improving. Its defaults are sensible and produce good-looking output. You can stop reading now.

A schematic overview of how Quarto orchestrates its document processing.

Unreasonable Demands

I had a very clear idea about how I wanted the first edition of the book to look in print. I also knew that I wanted to make it available as a website. I was fortunate enough to be able to have both of these things work out. This time around, I did the same again but I really wanted there to be as little as possible post hoc work with the website version. I knew that wouldn’t be the case with the PDF, for reasons I will discuss in a moment. I’m pleased that Quarto performed so well with the whole process. I wrote two pretty heavily-customized output formats (one for PDF and one for HTML) that specified the layout of the book. Quarto’s LaTeX-based book pipeline uses the scrbook class from the KOMA-script bundle, which has many nice features, though I find its documentation a tiny bit eccentric. (This might be because I wrote my first book using the memoir class.) I also wrote a couple of R packages that managed the themes and some other details of how PNG and especially PDF figures were produced. A version of the theme is in the development version of the socviz package that accompanies the book.

The PDF design is a two-column “Tufte-style” layout with wide margins for side-notes and figures. It works very well for a book of this kind as we can show small figures alongside the code that generates them, but also have figures break out of the main text column if needed.

Facing pages with a figure that runs the full width of one of the pages.

A layout like this can’t be rigidly ported over to a website, especially in an era of widely-varying screen sizes. So the HTML version of the book has a broadly responsive layout that arranges things differently at different sizes. Organizing and tweaking it this time around was made a lot easier by Quarto’s much better support for margin notes and marginal figures. It certainly wasn’t without its headaches. Marginal figures and notes are quite annoying to deal with in both HTML and PDF formats, for different reasons. In the PDF case, it’s tricky to get captions right, and there are still a few hacks in there to make it work. But it’s much cleaner than what I had to do in RMarkdown for the first edition, which was in effect a lot of regular expression substitution for things I could only add after the .tex file was produced. That’s gone now.

Here’s a screenshot of a facing page layout with some code, some marginal notes, and two kinds of figures, one in the margin and one full page-width:

Gapminder figures in the PDF version.

And here’s some of the same material as seen on the website:

Gapminder figures in the HTML version

Here’s a direct link to the same section. In the website version the marginal figures appear more marginal. There’s also a little bit of conflict to be worked out between the navigation guides and the marginal notes. In addition, the intrinsic variability of the web layout means that the positioning of the marginal notes is less precisely controllable than it is in the PDF output. But the overall result is pretty good. And I have to say it’s very satisfying to be able to produce a good website and a clean PDF (and also an ePub!) from the same folder of qmd files, with the text written in Markdown, the bibliography managed by Zotero and BBT, interspersed with the code that makes all the figures.

Let the Professionals do a Professional Job

I should say “less precisely controllable without substantial further adjustment”. Because this is the crux of the customization biscuit. There’s no end to it. One of the benefits of being in a position to do a second edition—something I really am very grateful for—is that it allowed me to have a much better sense of the production process for the hard copy of the book. This in turn placed sharp limits on what I was willing to do when it came to customizing the PDF version myself. Camera-ready files for books published by proper Presses are produced in many different ways. My most recent book, which is all prose and no code, was designed and typeset using Adobe InDesign. For the first edition of Data Visualization I sent the Press a set of LaTeX files and PDF image assets. The LaTeX files produced a very good facsimile of the design we’d agreed on. Then the Press’s typesetter got to work on it.

You might think that they just took my files, lightly edited them here and there, and added the trim, bleed, registration, and color marks for the physical print job. That’s not how it went. Book layouts are very hard to get just right, especially layouts that have many different-sized images and notes and other paraphernalia. They’re fragile. Moving something slightly here, or editing a sentence there, can cause a cascade of unwanted effects. Even ordinary pages of text will have issues with excessive or insufficient spacing around paragraph and section breaks, or widows and orphans, or rivers, and many other infelicities that most people won’t notice explicitly, but which cumulatively convey bad vibes even to people who don’t much care about design.

Some of this can be automated. That’s what layout algorithms do. The Knuth-Plass box-and-glue algorithm, which is the thing that causes TeX to emit those Underfull \hbox (badness 10000) complaints, is a real marvel. But it can’t quite work miracles. In my case, the professional typesetter took my LaTeX file, threw away my document class and substituted their own custom one (and some custom style files). Like any document class it defined the layout and all the features of the book, but it also included a variety of commands that allowed her to finely adjust the text as needed on any particular page. Tightening up the spacing here; forcing a break there; very slightly expanding or contracting the page size when needed to make sure that the layout didn’t break in a visible way on the next page, and so on. Here’s an example from the first edition:

And another:

Those uses of {\break}, \enlargethispage, \vspace{}, and the non-breaking space in this~way are all done by hand, based on rendering and re-rendering the document as it’s built to make sure each page meets the Press’s standards. An automatically-produced PDF can get you eighty five or ninety percent of the way to this but, if you really want to get things right, that last stretch will inevitably mean a bunch of adjustment by hand in whatever the final format is. That’s not something you can incorporate into your reproducibility pipeline.

Fortunately, you don’t need to. Most of the time we don’t require anything like that level of attention to detail. It’s worth producing and circulating material in accessible and readable formats that also don’t look like garbage. And it’s gratifying to be able to reliably generate pretty high-quality versions of those outputs from plain-text sources. That’s more than good enough in almost all cases. When writing papers that end up as PDFs, for example, I use a template that’s almost 20 years old. I only touch it when something breaks. By the same token, while my amateur interests compel me to run up polished custom Quarto book formats for my book projects, I also know that the people who set type for a living know a lot more about the fine grain of that work than I know, or need to know. But once in a while it’s nice to see how far you can push things.

The trick is to have the code chunks in your document be short and sweet, and have structured scripts and properly-documented packages manage the heavy lifting in any analysis. ↩︎

Data Visualization, Second Edition

Fri, 06 Mar 2026 06:52:36 -0500

I’ve written a second edition of Data Visualization: A Practical Introduction, which ideally should come out with Princeton University Press later this year. As with the first edition, a full draft of the book is available at https://socviz.co. The production process is just getting started so there’s no new cover yet, and there isn’t a link to pre-order. But (also like last time) I’ve put up a link to a form that lets you add your email if you’d like to be notified when it’s available to buy. You’ll only get one email (from me personally, not a marketing department) if you do; no spam or anything.

The revised edition is a pretty thorough rewrite. Naturally all the code is brought up to date for ggplot 4 and R version 4.5 and higher. The code from the first edition still runs, but you’ll get warnings and so on; those are all now gone. The back half of the book has been pretty thoroughly redone to reflect big changes in the availability of software for maps, (the sf package) and extracting results from models (the marginaleffects package). Meanwhile, several years of teaching this material (and getting feedback from others) have resulted in shifts of emphasis here and there to introduce just a little bit more on data wrangling. As the book goes on I also shift from an “object-based” approach to writing plots to a more “pipeline-based” one.

The recent rise of LLMs and coding agents gets some discussion, too. There the question is “Why can’t I just have a robot write all the code for me?” I don’t dismiss this question out of hand, and I don’t pretend that agents aren’t very powerful. My feeling about this is summed up in the Preface:

Perhaps you have a robot to help you write your code now. Large Language Models (LLMs) and coding agents are now part of the workflow of code generation and evaluation. They can do a great deal; so much so that it might seem superfluous to spend any time with the iterative, write-try-redo approach to visualization that this book presents. Can’t the robot write all the code instead? Not quite. It’s not that I believe repeatedly doing repetitive and error-prone tasks yourself is a virtue. To the contrary, that’s what computers are for. This book is full of examples where we end up automating something in order not to worry about it. But I also want you, the reader, to learn how to do good graphical work in a reproducible way. That means having a keen eye for quality and a good nose for error. Cultivating those senses requires practice and a vocabulary to express them. It seems faintly absurd to have to say explicitly but, whatever tools you use, your work will be better if you know what you are doing and understand why you are doing it. This book teaches you ggplot specifically, but it is not trying to lock you in to a particular framework. It’s just that, the way you acquire a general skill or a wide-ranging taste is by first learning some more specific version of those things, and then practicing them. Automation can come a later. In the words of the author Ann Leckie, you don’t learn how to do something by not doing it. For that reason, this book remains a hands-on introduction.

Or to put it another way, the book is an introduction to how to do something. One feature of books like it is that they tend to have two audiences: people who don’t know anything about the topic, and who’d like to learn something about it, and people who know a lot, at least in relative terms, and who have forgotten what it’s like not to know it. When the first edition came out, one of the early Amazon reviews was a complaint that the book seemed “pretty introductory” in its content. I mean, my Brother in Christ, that is right there in the title.

As with any corner of the vast division of labor that is human society, not everyone has to know about any specific thing in great detail. We’re all taking huge amounts of stuff for granted at any moment. But if you want to be proficient in some piece of that enormous web, it’s better that you know rather than not know what’s what. There’s nothing wrong with using tools that give you tremendous leverage. You do it every time you use a stand mixer in the kitchen, or a sander in the garage. You do it every time you turn your computer on, in fact. But you still need to develop the capacity to tell good work from bad, or correct from incorrect output, or safe uses from dangerous ones. That way you can take advantage of the power tools without being at risk of slicing your own or anyone else’s arm off.

Ordinal Exchanges

Thu, 13 Nov 2025 12:52:40 -0500

The Socio-Economic Review has published a book symposium on The Ordinal Society with contributions from Nitsan Chorev, J.P. Pardo-Guerra, and Greta Krippner, followed by a reply from Marion and myself. (Here’s a PDF of the exchange.) There’s also a symposium at the Journal of Cultural Economy with contributions from Hatim A. Rahman, Juan M. del Nido, Julien Migozzi, and Michelle Jackson, again with a reply from us. Finally there’s also a new review from Michael Sauder in ASQ (PDF), which follows on reviews from Barbara Kiviat in Social Forces (PDF), and Laura Nelson in Acta Sociologica.

As we say more than once in our replies, it’s very gratifying to have your peers in the field take the time to read your work.

Trustworthy Data Visualization

Mon, 10 Nov 2025 08:26:45 -0500

This past September I gave the closing keynote at posit::conf; it’s now on YouTube to watch. Keen-eyed observers will note from the title that it’s about trustworthy data visualization. But it’s also about trust a bit more generally, and how we should think about it in a world where researchers are faking results, AIs are enthusiastically confabulating, and government is destroying data infrastructure. When you find yourself giving a talk with a little tiny microphone stuck to the side of your head you have to ask yourself some hard questions, but the talk was partly about that.

One of the nice things about the opportunity to give a talk like this to a large audience is that you can tell the audience not just about your own stuff, but also the work you build from and rely on. This is, as you’ll see if you watch, one of the themes of the talk in the first place. In my case, in addition to all the good things done by people in the general area of data visualization and data science, I was able to talk about some ideas from social theory and, in addition, some of the work of Katherine Hawley on trust and commitment. The fulcrum of the talk is essentially an idea about commitment that she developed in her book How to be Trustworthy. I got to know Katherine and her family a little bit over a few summers in the 2010s when my own family used to visit St Andrews, where she was Professor of Philosophy. She died of cancer at the age of fifty in 2021. She was a wonderful person; a clear-minded philosopher of enviable intellectual gifts but also immense personal kindness. It was good to be able to think of her while writing this talk and, in however small a way, introduce some of her ideas to a new audience when I gave it.

Mamdani vs Sliwa and Cuomo

Thu, 06 Nov 2025 12:57:44 -0500

Mamdani’s victory in the New York City mayoral election gave me the opportunity to draw a few maps, and also to learn a bit about incorporating additional spatial data into maps drawn in R. R is not a specialized piece of GIS software. ESRI’s ArcGIS is the 800lb gorilla in this world and QGIS the GIMP to its Photoshop, so to speak.

Still, you can do a lot of spatial stuff in R, grounded in the sf package and its many friends. Plus you get the benefit of all the data manipulation and analysis that R is really good at. So, having gotten the precinct-level results for the election, some maps from New York City (e.g., the clipped borough boundaries map), and GTFS data from the MTA describing the structure of the subway system, I was able to draw some things. I strongly approve of the existence of the GTFS, by the way. It’s a spec for encoding transit data and lots of cities use it. Really handy.

Anyway, here’s a map. For each precinct with more than twenty voters, I combined the Sliwa/Cuomo vote into what we might call (purely for compactness reasons) the Slimo vote, calculated the Mamdani and Slimo vote shares as a proportion, and subtracted the former from the latter. That gets us a score raning from -1 to +1. I then cut that into bins, ten on each side of the zero line, to get deciles in each direction. That’s what we fill the precincts with. For the map I also drew the subway stations and lines. Several lines that are right next to each other are in effect drawn on top of one another in several places, e.g. the A and the C, or the 1 and the 3, etc, but that doesn’t matter for this map. (You may have heard that drawing transit maps meant for navigation is a really hard information design challenge.) We then use a discrete, diverging scale. Here’s the result.

Oh, choropleths

I chose a blue-green color gradient partly to experiment with it and partly because neither the election nor this particular cut of the data is quite the usual Blue vs Red, Democrat vs Republican. For one thing we have amalgamated two candidates on one side of the spectrum. For another, Cuomo is in some sense a Democrat, so the way voters were split is trickier than it normally would be. Andrew Gelman has some more thoughts on this today.

You can see a bunch of nice neighborhood patterns, such as e.g. the Hasidic communities in Brooklyn who voted strongly for Cuomo. And you can also see evidence of the characteristic weakness of choropleths, which is the way they can force a number characterizing some number of persons to be represented by a shape representing some area of space.

One solution to this is to make a dot-density map. You put a dot on the map for every person (or maybe every n people) you want to represent. Here’s what that looks like, with a 1-to-1 representation of dots to voters.

Dot dot dot.

This is better than a choropleth in some key ways. For one thing, you can see that—even in New York City—some places are both much more densely populated than others and also more likely to turn out to vote. To be clear, when I say “each dot represents one vote”, it’s not as if I know the identity of every voter, or how they voted, or can precisely locate them to their home address. I promise I don’t know that. Only the NSA and the Phone Company know that stuff. What I do know is how many votes were cast for each candidate within each of the 4,200 or so precincts. And I have a polygon that represents the shape of each precinct. So I spatially sample without replacement within each polygon to randomly place a dot for each voter within their precinct. With two million or so votes the pointillist effect ends up being quite effective.

A slice across Manhattan in the upper 50s and lower 60s and over into Queens.

The main reason for doing this is that, from the point of view of the election, precincts aren’t real. They are closely related to the social geography of the city, which is one of the reasons we want to draw a map like this, but in and of themselves they are at best proxies for the thing we care about, so we should take care not to reify them. The sampling method still brings out the basis of the data, so you can see the precinct-patchwork that forms the underlying grid, especially in densely-populated areas. But those polygons are filled in proportion to the number of people who actually voted. I saw someone on social media observe that this “conflated” partisan lean and population density. But again, precincts aren’t real. The distribution of partsian lean across population density is what we’re trying to bring out with this quasi-person-level approach.

Detail of the PDF.

There are costs, of course. Color choice is harder, especially with more than one category of dot at once. The image tends to look less bright because you’re not using a big swatch of paint across a large area. The visual impression is also very sensitive to small changes in the size of the dots and to settings like their alpha transparency. The order that the layers are drawn also matters a great deal at this sort of resolution. In addition, when you draw a few million dots on a large grid of pixels then your file size gets big real fast. The main dot-density image above is a 300dpi 4,500 by 4,500 pixel PNG and it’s 12mb in size before being crushed down further (with optipng) to 9mb or so. I could make a JPG of course, which would be a lot smaller, but then you start running into the question of why you made a dot-density map in the first place, because you lose detail in the raster. In fairness, you’d still get the benefit of having larger and less densely-populated (or lower-turnout) precincts not appear fully filled-in, even if you wouldn’t be able to zoom in at all.

Meanwhile, thanks to the wonders of multiplication, a 600dpi version of the same image is much sharper to zoom in on but is also about 35mb in size. A PDF, which is a vector rather than a raster format, is even bigger, weighing in at 63MB. Rendering a PDF with that many vector elements does not make Preview or Illustrator happy, let me tell you. The benefit of course is that you can scale it up to any size you like without loss, as shown above. Because browsers are crazy and so is javascript, it’s possible to serve up dot density maps like this in real time in a way that makes them zoomable and fluid, but that’s not my department.

GSS Release

Tue, 28 Oct 2025 11:14:15 -0400

Trends in the immameco question.

Release 2 of the 2024 GSS cross-section and 1972-2024 culumative data are now available. I’ve updated gssr and gssrdoc to incorporate them. There are quite a few changes in the data and variables, thanks in part to some changes in data collection methods and a privacy/disclosure review.

The gssr and gssrdoc packages are the nicest way to get General Social Survey data up and running in R. The figure above shows (survey-weighted) trends derived from the immameco question.

Manhattan Plot of Manhattan

Sat, 25 Oct 2025 11:38:02 -0400

Here I continue my efforts to design visualizations that are as poorly-suited as possible to being displayed on phones. It looks pretty good on a big monitor, or six feet wide on a wall.

I made a version of this plot a few years ago. I ended up revisiting it this morning because I’m updating various datasets and code. A Manhattan plot is a term sometimes used to describe a kind of scatter plot where the x-values are fairly continuous, and the y values have distributions with long tails, so the plot looks like a skyline. This one here is a bar chart rather than a scatter plot but it’s still a kind of Manhattan plot of Manhattan.

The plot shows the heights of almost all currently-existing buildings¹ in Manhattan (on the y-axis) by their year of construction on the x-axis. What I want is a plot that gives a sense of the distribution of building heights over time. To make the plot work I play a few tricks. First, the resolution of the x-axis is only to the year,which would result in way too much overplotting. (We have almost 35,000 buildings to draw.) So we add a small amount of random noise to the x-values, which makes buildings distribute themselves around their year of construction. There’s still overplotting, but now it works in our favor. It contributes to a feeling of building density.

Second, there are so many buildings that we can’t plot everything as solid, filled rectangle. Instead, we make make the outlines of each rectangle a very thin white line, so everything looks like a vector-driven video game from 1981. Then we make a variable that bins buildings by height into ten categories, one for every hundred feet of additional roof height. We map that to fill color of the rectangles (darker purples for shorter buildings through bright yellow for taller ones), which means the taller a building the brighter it looks. But again, we can’t just plot those as solid colors. So we also take the roof heights and rescale them to a range of 0.4 to 0.85. Then we map that number directly to the alpha channel of the fill color. So taller buildings are not only brighter in color, they are more opaque. Alpha runs from 0 (fully transparent) to 1 (fully opaque). The specific values of 0.4 to 0.85 are just from trial and error. We want the taller buildings to stand out, and because there are fewer of them they need to be more opaque. Whereas the fills of many more shorter buildings will overlap, so they can be individually more transparent.

You can really see the recent rise of supertall fancy apartment buildings in Manhattan in the last ten years or so—buildings like the curséd 111 West 57th and 53 West 53rd. The cursédness 111 W57th extends to dataset:

manhat |>
  filter(heightroof == 1428)
#> Simple feature collection with 1 feature and 19 fields
#> Geometry type: POLYGON
#> Dimension:     XY
#> Bounding box:  xmin: 990374 ymin: 217856.8 xmax: 990559.1 ymax: 218081.1
#> Projected CRS: NAD83 / New York Long Island (ftUS)
#>   name     bin cnstrct_yr date_lstmo   time_lstmo lststatype doitt_id heightroof feat_code groundelev shape_area shape_len   base_bbl
#> 1 <NA> 1023728       1924 2021-01-04 00:00:00.000     Merged  1260269       1428      2100         58          0         0 1010100025
#>   mpluto_bbl geomsource BoroCode  BoroName Shape_Leng Shape_Area                       geometry
#> 1 1010107507 Other (Man        1 Manhattan   359993.1  636620786 POLYGON ((990461.3 217856.8...

It’s recorded as having been completed in 1924, instead of 2019. Why? There’s a hint in the lststatype column, which says “Merged”. Technically the building took over and “renovated” Steinway Hall, which was built in 1924. There are a few cases like this in the dataset for buildings that are going to be salient in the figure—i.e. very tall ones. There’s also some missing data, with eight buildings over 600 feet tall where the construction year is not available. At least two of those are because the building was still under construction when the data were recorded. As always, 90% of data analysis is data cleaning.

Data on building heights and construction years come from the NYC Open Data portal. The data are restricted to buildings constructed after 1899 that are currently standing in Manhattan. ↩︎

gssrdoc Updates

Sun, 19 Oct 2025 10:50:04 -0400

Regular readers know that I maintain gssr and gssrdoc, two packages for R. The former makes the General Social Survey’s annual, cumulative and panel datasets available in a way that’s easy to use in R. The latter makes the survey’s codebook available in R’s integrated help system in a way that documents every GSS variable as if it were a function or object in R, so you can query them in exactly the same way as any function from the R console or in the IDE of your choice. As a bonus, because I use pkgdown to document the packages, I get a website as a side-effect. In the case of gssrdoc this means a browsable index of all the GSS variables. The GSS is the Hubble Space Telescope of American social science; our longest-running representative view of many aspects of the character and opinions of American households. The data is freely available from NORC, but they distribute it in SPSS, SAS, and STATA formats. I wrote these packages in an effort to make it more easily available in R. If you want to know the relationship between these various platforms, I have you covered. But the important thing is that R is a free and open-source project, and the others are not.

This week I spent a little time updating gssrdoc a bit to clean up how the help pages looked and make some other improvements. Inside R, you can say, e.g., ?govdook at the console and have this pop up in the help:

Yeah govdook is short for ‘Gov Do OK’, not ‘Go v Dook’.

The package also includes gss_doc, a data frame containing all of the information that the help pages are built from. I included it because it can be useful to work with directly, as when you might want to extract summary information about a subset of variables.

library(tibble)
library(gssrdoc)

gss_doc
#> # A tibble: 6,694 × 10
#>    variable description                           question         value_labels var_yrtab yrballot_df module_df subject_df norc_id norc_url
#>    <chr>    <chr>                                 <chr>            <chr>        <list>    <list>      <list>    <list>       <int> <chr>   
#>  1 year     GSS year for this respondent          "GSS year"       "[NA(d)] do… <chr [1]> <tibble>    <tibble>  <tibble>         1 https:/…
#>  2 id       Respondent id number                  "Respondent id … ""           <chr [1]> <tibble>    <tibble>  <tibble>         2 https:/…
#>  3 wrkstat  labor force status                    "Last week were… "[1] workin… <tibble>  <tibble>    <tibble>  <tibble>         3 https:/…
#>  4 hrs1     number of hours worked last week      "Last week were… "[89] 89+ h… <chr [1]> <tibble>    <tibble>  <tibble>         4 https:/…
#>  5 hrs2     number of hours usually work a week   "Last week were… "[89] 89+ h… <tibble>  <tibble>    <tibble>  <tibble>         5 https:/…
#>  6 evwork   ever work as long as one year         "Last week were… "[1] yes / … <tibble>  <tibble>    <tibble>  <tibble>         6 https:/…
#>  7 occ      R's census occupation code (1970)     "A. What kind o… "[NA(d)] do… <chr [1]> <tibble>    <tibble>  <tibble>         7 https:/…
#>  8 prestige r's occupational prestige score(1970) "A. What kind o… "[NA(d)] do… <tibble>  <tibble>    <tibble>  <tibble>         8 https:/…
#>  9 wrkslf   r self-emp or works for somebody      "A. What kind o… "[1] self-e… <tibble>  <tibble>    <tibble>  <tibble>         9 https:/…
#> 10 wrkgovt  govt or private employee              "A. What kind o… "[1] govern… <tibble>  <tibble>    <tibble>  <tibble>        10 https:/…
#> # ℹ 6,684 more rows

The gss_doc object has regular columns but also a series of list-columns to (insert meme here, you know the one) put data frames inside your data frames. (They’re labeled as “tibbles” here; basically the same thing).

Why a list-column? Why a list? Well, a list is one of the fundamental ways to store data of any sort. Lists are useful because they can contain heterogeneous elements:

items <- list(
  todo_home = c("Laundry", "Clean bathroom", "Feed cat", "Bring out rubbish bins"),
  important_dates = as.Date(c("1776-07-04", "1788-06-21", "2025-01-18")),
  keycode = 8675309,
  storage_tiers = c(128, 256, 512, 1024)
)

items
#> $todo_home
#> [1] "Laundry"                "Clean bathroom"         "Feed cat"               "Bring out rubbish bins"
#> 
#> $important_dates
#> [1] "1776-07-04" "1788-06-21" "2025-01-18"
#> 
#> $keycode
#> [1] 8675309
#> 
#> $storage_tiers
#> [1]  128  256  512 1024

One thing to notice about a list like this is that it doesn’t really make sense to represent it as a table. This is partly because the elements of the list are of different lengths, but really it’s because if we did represent it as a table, it would not mean anything to read across the rows:

items_df
#> # A tibble: 4 × 4
#>   todo_home              important_dates keycode storage_tiers
#>   <chr>                  <date>            <int>         <int>
#> 1 Laundry                1776-07-04      8675309           128
#> 2 Clean bathroom         1788-06-21           -            256
#> 3 Feed cat               2025-01-18           -            512
#> 4 Bring out rubbish bins -                    -           1024

The rows don’t form “cases” of anything. We just have four unrelated categories with various pieces of information in them.

Lists are also useful because they lend themselves easily to being nested:

items <- list(
  todo_home = list(
    tasks = c("Laundry", "Clean bathroom", "Feed cat", "Bring out rubbish bins"),
    tobuy = c("Cat Food", "Burritos"), 
    wifi_password = "p@ssw0rd!"
  ),
  important_dates = as.Date(c("1776-07-04", "1788-06-21", "2025-01-18")),
  keycode = 8675309,
  storage_tiers = list(
    ssd = c(128, 256, 512, 1024),
    ram = c(1, 4, 8)
  )
)

items
#> $todo_home
#> $todo_home$tasks
#> [1] "Laundry"                "Clean bathroom"         "Feed cat"               "Bring out rubbish bins"
#> 
#> $todo_home$tobuy
#> [1] "Cat Food" "Burritos"
#> 
#> $todo_home$wifi_password
#> [1] "p@ssw0rd!"
#> 
#> 
#> $important_dates
#> [1] "1776-07-04" "1788-06-21" "2025-01-18"
#> 
#> $keycode
#> [1] 8675309
#> 
#> $storage_tiers
#> $storage_tiers$ssd
#> [1]  128  256  512 1024
#> 
#> $storage_tiers$ram
#> [1] 1 4 8

In its bones, R is a LISP/Scheme-like list-processing language fused with features of classic array languages like APL. This is because, in the world of data analysis, what we deal with all the time are rectangular tables, or arrays, where rows are cases and columns are different sorts of variables. The wrinkle is that, unlike a beautiful array of pure numbers, each column might measure something (a date, a True/False answer, a location, a score, a nationality) that we’d prefer not to represent directly as a number. Sure, underneath in the computer everything is all just ones and zeros. (Or rather, electromagnetic patterns in some physical substrate that we can interpret as meaning ones and zeros.) And if we want to do any sort of data analysis that involves treating our table as a matrix then we’ll want numeric representations of all the columns. But for many uses we’d like to see “France” or “Strongly Agree” instead of “33” or “5”. Just a table of rows and columns, where different things can be represented across columns, but any particular column is all the same kind of thing.

A rectangular table like that is called a data frame. One way to think of a data frame is just as a special case of a list. A data frame is a list where you can put all the list elements side by side and treat them as columns, and where all these elements are made of vectors of the same length. Beyond that, it’s a list where the nth element of each vector refers to some property of the same underlying entity, i.e. the thing that’s in the row, or case; the thing the columns are showing you measurements or properties of. You can have empty entries if needed, as when some bit of data is missing. The important thing is that each column has as many slots as there are cases, and you fill in the values for each case in the same slot in each column. Whenever you look at any table of data, one of your first questions should always be “What is a row in this table?” In this case, each row is a variable in the full GSS dataset, and each column describes some property of that variable.

library(tibble)
library(gssrdoc)

gss_doc
#> # A tibble: 6,694 × 10
#>    variable description                           question         value_labels var_yrtab yrballot_df module_df subject_df norc_id norc_url
#>    <chr>    <chr>                                 <chr>            <chr>        <list>    <list>      <list>    <list>       <int> <chr>   
#>  1 year     GSS year for this respondent          "GSS year"       "[NA(d)] do… <chr [1]> <tibble>    <tibble>  <tibble>         1 https:/…
#>  2 id       Respondent id number                  "Respondent id … ""           <chr [1]> <tibble>    <tibble>  <tibble>         2 https:/…
#>  3 wrkstat  labor force status                    "Last week were… "[1] workin… <tibble>  <tibble>    <tibble>  <tibble>         3 https:/…
#>  4 hrs1     number of hours worked last week      "Last week were… "[89] 89+ h… <chr [1]> <tibble>    <tibble>  <tibble>         4 https:/…
#>  5 hrs2     number of hours usually work a week   "Last week were… "[89] 89+ h… <tibble>  <tibble>    <tibble>  <tibble>         5 https:/…
#>  6 evwork   ever work as long as one year         "Last week were… "[1] yes / … <tibble>  <tibble>    <tibble>  <tibble>         6 https:/…
#>  7 occ      R's census occupation code (1970)     "A. What kind o… "[NA(d)] do… <chr [1]> <tibble>    <tibble>  <tibble>         7 https:/…
#>  8 prestige r's occupational prestige score(1970) "A. What kind o… "[NA(d)] do… <tibble>  <tibble>    <tibble>  <tibble>         8 https:/…
#>  9 wrkslf   r self-emp or works for somebody      "A. What kind o… "[1] self-e… <tibble>  <tibble>    <tibble>  <tibble>         9 https:/…
#> 10 wrkgovt  govt or private employee              "A. What kind o… "[1] govern… <tibble>  <tibble>    <tibble>  <tibble>        10 https:/…
#> # ℹ 6,684 more rows

Because R was designed by statisticians—R is a descendant of S, which like everything else in computing traces its origins to Bell Labs—it has this concept of a data frame built-in to its core instead of being bolted-on afterwards, which is extremely handy. Normally data frames are just ordinary rectangles, but there’s no reason why any particular column can’t itself be thought of as a list of something else. That’s what we have here. The yr_vartab column contains data frames of crosstabs of the answers to each question by year. Except where it doesn’t (e.g. for id), and this is fine because lists don’t have to be internally homogeneous. Similarly yrballot_df has a little table of which ballots, or internal portions of the survey, a question was asked on for each year it was asked.

The upshot is that having assembled the gss_doc object we can use it to emit, like, seven thousand pages of documentation on the GSS’s many, many questions over the years. We can build them as standardized R help pages, as above. On the website that pgkdown builds for us, we get this:

Website view.

The cross-referencing to other relevant variables in the “See Also” section is new in this version. It comes courtesy of the GSS’s own information about survey modules and an ad hoc topic index they keep for the variables. I just use a subset of possible cross-references as we don’t want, e.g., every single question in the GSS core to be cross-referenced to every other core question on any particular help page. On the website, I gather these into a single page:

Topic index page.

The GSS has its own handy data explorer which is very useful for quickly checking on particular trends and getting a quick graph of what the data look like, or a summary view of the content of particular variables. Each help page in gssrdoc now links to the GSS Data Explorer page for that variable, in case you want to hop over and take a look there. Of course, the gssrdoc package doesn’t and isn’t meant to replace the Data Explorer; it’s just a different view of the same information, with a different use-case in mind.