Natural Spirit of Good Company

two different perspectives on an investigation

2010-07-10T14:12:00.000-07:00

Is it noteworthy not to look at the emails in question in context? And not to ask questions such as: "Prof Jones, did you delete any e-mails?" Some think so. Others think not. If only there were any commonly accepted standards by which we could judge one of them to be absurdly wrong...

"destitute, hopelessly stagnant proletariat" in 1971 South Korea

2010-06-28T07:00:00.000-07:00

another 1970s time capsule, this time from Super-Economy:

The book is hints at how crazy the ideological atmosphere was in 1971. As I wrote, Villy Bergström was a brilliant economist, and considered a centrist Social Democrat. Yet he writes in one point, favorably comparing North Korea with other nations: "[Classical] liberalism and capitalism in South Korea has led to fascism and an upper class in ruthless luxury, with a destitute, hopelessly stagnant proletariat. This has happened in South Korea, Taiwan, South Vietnam, Pakistan, South America and southern Italy."

I am quite impressed with the epic fail of choosing 2/4 of the Asian Tigers to illustrate "hopelessly stagnant proletariat."

(And incidentally, I don't think that that sharp disconnect from economic reality discredits the other remarks here. Most of the other remarks in the passage are not as sharp as "stagnant [vs. a reality of dramatic economic change]," leaving room for reasonable people to disagree about how correct they are, and I even agree that some of them are correct. I do disagree about how "classically liberal" these societies were. I also have a narrower disagreement with "fascist" not because it's overharsh, but because it's overspecific. I don't see how the 1971 snapshots can be classified with Hitler but not with Stalin, or with Mussolini but without 1900 Japan or 1900 Russia or 1920s Russia. Thus I'd prefer a term less misleadingly specifically referring to the enemies of the Social Democrats, perhaps "tyranny" or "absolutism.")

A MISTAKE TO MAKE ONLY ONCE

2010-06-26T12:35:00.000-07:00

Or, how I learned that (AWHEN (AND (BLOG-COMMENTING-P) (PASTING-P) (STRING-UPCASE-P IT) (NDOWNCASE IT))).

I cut capitalized page titles from an EconLog web form and an EconLog posting-delay-notification screen and pasted them into my remarks in two blog comments. OOPS!

I have spent a lot of time working with program comments and plain ASCII documentation files like this, using a convention where ALL CAPS indicates not yelling but quoting of fragments of computer programs, rather like italics can indicate not emphasis but quoting of title text like The Wealth of Nations. This seems to've created a blind spot in my proofreading skillz, since the yelling interpretation wasn't glaringly obvious to me. The yelling interpretation was certainly glaringly obvious to EconLib Ed., though: "If you can't wait so much as five or ten minutes before griping and screaming and yelling, you are pretty hair-trigger, eh?".

Sadly, perhaps EconLib Ed.'s assessment that I'm a ranting loon is uncontroversially true. More positively, though, perhaps I should take this a sort of double hint from fate (first that this occurred in my comment about how it's technically easier to do something on my own blog and second that this occurred in a blog post advising "Get Your Own Blog") reminding me that even if I am a ranting loon I can still post here! (What could possibly go wrong?)

Meanwhile I should probably try to remember to proofread more carefully the next time that I need to identify a web screen URL-suitable global meaning so that I am tempted to identify it by giving its title and furthermore its title happens to be capitalized. Too bad the brain is too inexpressive to support OAOO implementation of this as (DEFMETHOD MAKE :AROUND ((M MISTAKE)) (UNLESS (ALREADY-MADE-P M) (CALL-NEXT-METHOD)))...

On maintaining the appearance of balance

2010-06-26T12:16:00.000-07:00

I'm surprised that both Tyler Cowen and Julian Sanchez wrote lengthy blog posts about the severity of Dave Weigel's problems with his Journolist emails, but didn't mention the coincidence that this is in the wake of the controversy over apparent maneuvering to try to protect Rep. Etheridge. (A "any video of a member [of Congress] acting strangely, no matter how grainy" forsooth!) When fire from the right catches you just as you're heeled over that far to the left, it tends to strike below the waterline.

UPDATE: also observed at Colby Cosh

On Hacker News on New Scientist on Living in Denial

2010-05-18T12:26:00.000-07:00

(This is an extended version of a comment posted on a Hacker News article .)

(No, I don't remember the customary capitalization rules for different parts of speech in titles, why do you ask?)

From the article in question: "Vaccine denial: Umbrella term for a disparate movement claiming that certain vaccines either (1) do not work or (2) are harmful"

I have no particular sympathy with any anti-vaccine activism that I'm aware of. But I wonder how, other than by not being an important faction in the appropriate big political tent, "anti-vaccine denial" ended up on this article's excrement list along with Holocaust deniers, while "nuclear power denial" and "genetic engineering denial" didn't.

My impression is that political opposition to nuclear power plants, to nuclear waste facilities, and to GM crops have had at least as much economic impact as political opposition to vaccines. Thus, it seems to me that they shouldn't be left off this list because they're unimportant.

Perhaps the columnist thinks that the anti-nuke and anti-GM-crops political movements don't belong on the list because they have achieved their success primarily by honestly making valid technical points? Granting for the sake of argument that that is a tenable position, then why ignore them? Wouldn't the anti-nuke and anti-GM movements make useful examples to clarify his position by comparing and contrasting? Wouldn't describing what is healthy and good about the thinking of the anti-GM and anti-nuke coalitions help us understand better what is exactly is so characteristically diseased and vile about anti-vaccine folk to justify grouping them with Holocaust deniers?

Now, I vaguely remember that long ago, when I first encountered "politically correct" used in the usual modern sense, I laughed pretty hard. Cruelly. Since then I've used the term repeatedly, trying to criticize suitably dishonest tactics with the nasty reminder that they seem not only dishonest, but significantly parallel to dishonest tactics which were used to support murderous totalitarian regimes. So maybe I should recognize that being labelled a "denier" for arguing against the IPCC version of AGW (as opposed to, say, little-feedback "lukewarming") should be considered karmic justice. (Pretty crude justice, I think, since the parallel seems broken in various ways. E.g., the left regards Che t-shirts rather more fondly than right regards Rommel t-shirts. If you want a recognizable parallel, it would work a lot better to make a nasty reference to how Confederate sympathies are widely tolerated. It's not that there are no real problems to be nasty about, just that being nasty about the particular problem of Nazi sympathies is delusional.)

Perhaps what's going on is a rhetorical declaration of political factional loyalty by politically correct use of a partisan barb, not an actual assessment that people are violating some objective standard of analysis and discourse. Perhaps then I should accept that, since I've confessed to politically-correct-for-libertarians[*] use of the barb "politically correct." However, if indeed that's what's going on, I think it would be nice if the New Scientist would be honest about it. If the tables were turned, I'd be pretty disgusted with a magazine article, either an openly partisan one or a nominally objective one, which published a lot of text based on a definition of "politically correct" in neutral terms, but mysteriously happened to choose only targets on the left when illustrating those terms, avoiding e.g. any school boards which have made embarrassingly right-wing curriculum or library choices, or various episodes of narrowly doctrinaire in-group infighting weirdness among libertarian groups, even when discussing parallel kinds of misbehavior on the left.

On David J C Mackay on AGW

2010-05-16T13:23:00.000-07:00

(Maybe this weblog is still dead. The current unnatural stirring of its <body> is basically an elaboration of a comment I left on http://bishophill.squarespace.com/blog/2010/5/16/david-mackay-at-oxford.html, about David Mackay's talk about Mackay's freely-downloadable book Sustainable Energy --- without the hot air . I am well aware that it is insufficiently edited, and if more than three people read it, maybe I will feel bad about that. Or maybe not. And I refuse to feel bad about Blogger's endearing HTML-plus-homebrew-randomness treatment of blank lines, of paragraph tags, and of the interaction between the two, which might anyway have changed in the last year since I never was able to find it documented anywhere, so if this looks weird, then yup, I forgot my workaround, but I do remember I was never able to make my workaround less than clunky, and part of the fun was that the Preview always had different pathology than the published version, so I stand innocent whatever presentation hell I have sent my text into. And as for the logical hell it may seem to've issued from, I am vast, as vast as a swarm of subordinate clauses chewing on prepositional phrases and bleeding distracting observations, and sometimes losing the point, and indeed sometimes vaster than that. I contain vast sentences, and generally it takes time and care for me to shorten them, and it's a really nice day outside, and now the merest hundreds of lines of draft prose into this article I find it harder than before to care so much that someone is wrong on the Internet, and anyway this must be submitted as a blog post before I can submit a blog comment linking to it, and blog comments even more than blog posts should be timely, so it follows logically that my sentences contain multitudes, so deal.)

Mackay's other downloadable book, Information Theory, Inference, and Learning Algorithms, is impressive. I downloaded that book five years ago in order to study several sections of it. Hence my particular interest in seeing what Mackay had to say now about AGW.

Having just skimmed the first 15 pages or so of Sustainable Energy, I think Mackay is guilty of flaky preaching to the choir about the underlying AGW problem. He spends multiple pages in a lovingly detailed victory dance over the counterargument that CO2 concentration hasn't risen. He spends much less time on the counterarguments about the rather more important question of CO2 sensitivity --- disposing of them by saying it's complicated and then uncritically endorsing IPCCish results as a scientific consensus and thus a reasonable estimate of CO2 sensitivity. I don't see any nonpartisan way to justify that. As far as I can tell, disputes about the existence of a rise of CO2 level are completely marginal compared to disputes about temperature sensitivity. (CO2 level disputes seem to be a distant fourth behind at least three other disputes, about 1. temperature sensitivity, 2. temperature measurements, and 3. credibility of current-generation climate models.) Mackay has plenty of expertise in the fundamentals of statistical reasoning, and it would be nice if he'd write even a third as much about back-of-the-envelope cross-checks of his confidence in the IPCC temperature sensitivity as he spent on such cross-checks of CO2 level rise.

In particular, it'd be interesting to know how Mackay can justify appealing to a scientific consensus that circles the wagons around the original Mann hockey stick articles, and around the IPCC process which made those articles the flagship of sufficiently strong evidence and sufficiently sound analysis to justify punting previous ideas about large preindustrial climate fluctuations. Mackay has done a book's worth of research on quantitative sanity checks related to the AGW controversy, enough work to have published pages of cross-checks addressing a quaternary controversy. And before that, he published quite a good book on (more or less) the fundamentals of statistical inference. Thus, a lack of interest in quantitative sanity checks on the central CO2 controversy consensus seems out of place.

Perhaps Mackay subscribes to the view that the preindustrial temperature evidence is unimportant because the modellers have statistically sound demonstrations of sufficient ability to make quantitative predictions from first principles and recent measurements? Or that the statistical problems of the original hockey stick aren't important because later studies to defend its conclusions were done with fundamentally sound statistics, honestly accounting for what seems to have been a strong political temptation to, e.g., give rather heavier statistical weight to trees in a small dataset giving palatable results than to trees in larger datasets with less palatable result? I don't know how Mackay can justify deferring to the IPCC estimate of CO2 sensitivity without subscribing to one of those two possible views. However, I also don't know how he can easily subscribe to either. Conversely, it's sort of fascinating, but in a rather sad creepy way, when he finesses this by dropping from previous pages of physicist-speaking-to-physicist analysis --- performing sanity checks on the fundamentals by numerate back-of-the-envelope/elevator-pitch analysis --- to breezy chatter about "complex, twitchy beasts" and "Bad Things."

What kinds of cross-checks am I dreaming of here? As cross-checks of arguments for smallness of preindustrial climate changes, I nominate five examples. 1. How statistically reasonable is it to handle what is in effect a multisensor fusion problem by giving zero weight to our scattered incomplete hard temperature data (historical lake/river freezing times in Europe and various of its colonies, e.g.), calculating our result purely in terms of more indirect proxies (because of their compensating advantages like longer time series). 2. Roughly how sensitive might the post-Mann IPCC-camp temperature results be to outright cherry-picking and/or softer irregularities like giving heavier weight to a tree in an 8-tree dataset than to a tree in a larger dataset? 3. Given the level of local variation we observe in climate in naively-comparable sites today (e.g., comparable in altitude and latitude) roughly how often do we expect to see purely-local fluctuations of the level of the LIA/MWP observations? 4. How numerically reasonable was the Wegman network-analysis critique, and to what extent does it apply to the various generations of IPCC-favored analyses? 5. How information-theoretically reasonable is it to be pointedly uninterested (e.g., a long-standing pattern not publishing raw data and details about its collection, and of not energetically remeasuring and rechecking the proxies as the passage of time adds new tree rings or mudlayerwiggles or whatever; and the recently-controversial masterstroke of not graphing them either, in the famous "hide the decline" trick) in key details of at least the most heavily weighted proxies? Is the observed level of interest consistent with the hypothesis of a technical community honestly reaching a scientific consensus about a statistical reconstruction?

(I don't claim that each of those cross-checks would torpedo the IPCC position below the waterline. I do think that #1, #2, and #5 are serious criticisms. I also think that #2 and #5 are sufficiently common criticisms that addressing either or both instead of "CO2 concentration is not rising" would be much more unlike beating a strawman. I have mixed feelings about #4; quick-and-crude quantification of fundamentally messy things like social relationships doesn't appeal to me, but on the other hand, claims of "consensus" and "independent studies" are fundamentally equally crude quantifications also. Thus, to the extent that it's worth addressing such a crude simplification of a messy system, Wegman's calculation seems like a natural enough way for a statistician to try to do it. And I'm quite curious about #3, and I don't know why I have never run across a reference to such a calculation. I'm unlikely to do such a calculation myself, and less likely to write it up, since I think it would take me at least two weeks to get sufficiently up to speed on the data sources to get a result I'd be unembarrassed to put on a webpage. But many dozens of people are already very familiar with the data, and many of them might be able to do it in a weekend, and many of them write at least dozens of pages a year on similar subjects.)

It's harder to dream up direct quantitative cross-checks for the validity of IPCC modeller consensus. None of the AGW forecast data I've ever heard seem to be friendly to such back-of-the-envelope checks on any reasonable timescale. Thus my nominations for sanity checks here are not calculations, but questions "why [is it hard to find such cross-checks]?" and "what [the hell are we thinking then]?" 1. If modellers have the situation under control, why are they unable to (or unmotivated to?) pick easily-measurable numbers where their models make clear interesting predictions near-term predictions? 2. If they're not doing this, what is the strongest line of argument that they are reasoning clearly about having the situation under control, as opposed to, say, peddling overfitted nonsense?

To elaborate on point #1 here, it's a common situation in modelling that there are economically-important questions that we care about (e.g., how often a new kind of telephone exchange will have to refuse/drop calls because of congestion) which are expensive to measure directly. (First build the expensive piece of equipment, then wire it up to a bunch of customers to use them as guinea pigs...) Of course ultimately you *do* test this kind of thing on the poor guinea pigs, whether you like it not, but it's usually worth doing a lot of work before you get to that stage. If you have a model which purports to demonstrate a surprising result about behavior in full-scale customers-running-live mode, it's good to have evidence in support of that model before you actually test the thing on customers. (Note that "surprising" doesn't need to be terribly surprising, either: if you want to scale up a supermarket by 80% relative to the largest supermarket your chain has built so far, and claim that various size-dependent properties will be accurately (+/-15%, say) predicted by linear extrapolation from the sizes of existing stores, that might not be surprising in informal terms, but in this context it's surprising enough that you'd like some justification before you build it and let the guinea pigs in.) So if you have a model which is good enough to make "surprising" predictions in the expensive large, in my experience, it tends also to be good enough to make comparable predictions in the small.

In my experience in chemistry, a question which might matter economically is the value of a binding constant under some exotic conditions that will be very difficult (time-consuming, expensive...) to set up. If it's too difficult to set up unless the model is correct, then how then can your model prove its worth now, so that you know it's correct to do the expensive thing? The model doesn't prove its worth by stubbornly claiming that it really can calculate the one number that we ultimately care about, you damnable denier, but by successfully predicting surprisingly accurate results for dozens or hundreds of other numbers in related problems where measurement is much more practical. E.g., it might make a boatload of predictions about spectroscopic changes of the bound molecule in related conditions. In CO2 climate sensitivity questions I don't know enough to guess what should be simultaneously easy to predict, easy to measure, and surprisingly significantly different from naive extrapolation, but roughly the kind of thing I'd expect is Mackay writing "a famous early example was the Tarsasku model B 1997 "beach bunny" curve predicting the change in the power spectrum of coastal/inland nocturnal barometric fluctuations, which was clearly vindicated by 2002; Zer's microfoundations review article of 2003 gives 14 such predictions which met his 98% confidence level, and today we have approximately 300."

(Maybe that power spectrum example sounds unrealistically complicated? or unreasonably simpleminded? I don't mind simple predictions at all --- e.g., differential tropospheric warming? excellent in its simplicity. But my impression from other modelling is that in pursuit of results which are easy to predict, surprising, and easy to measure sufficiently accurately --- and from what I've seen of the troposphere controversy, tropospheric warming measurement accuracy is at best marginal --- one tends naturally to end up with esoteric predictions. Commonly, in fact, one ends up with very esoteric predictions, and I'd cheerfully accept predictions of the north/south hemisphere deviation of the El-Nino-corrected coastal/inland nocturnal barometric fluctuations, or results hairier than that, as long as they're precisely defined in terms of measurements which are routinely made to sufficient accuracy. Compared to the hairiness of tunneling down through all the layers of equipment and calculation to the actual physical reality of spectroscopic experiments (like 2D NMR, or various nonlinear superfast laser stuff), that seems almost tame. But I'm not impressed with hindcasting small noisy datasets with enormous computer programs, and I'm not impressed with any prediction which even today, after all the years which have passed since the science was settled, are hard to sharply distinguish from the supernaive naive rival hypothesis of "zero trend, not even no-feedback lukewarming, just the usual reddish noise drift" over the past decade, and which won't be clearly distinguishable from historical trend extrapolation for many years yet.)

And to elaborate on point #2, I'm venting, but fundamentally it's a serious question. And its seriousness doesn't depend on worries about left/right/enviro political subtexts, about professional clannishness, or about professional or financial incentives to reach particular kinds policy conclusions. Without any of those incentives, modellers can very easily be spontaneously guilty of overfitted nonsense; it seems to be very human to fall in love with the modelling approach one has chosen, and to believe its results with more confidence than one should. Relatedly, there is a strong human tendency to resent being suspected of merely falling in love with an unrealistic model, and to react by pumping out results which demonstrate that the model really can predict surprising stuff accurately, even if the experimentalists (did we mention that when attending institutions where people who qualify are able to become theorists, they "chose" to go into experiment? need we say more?) haven't yet been able to measure relevant quantities to sufficient accuracy to confirm our accurate surprising prediction for the result that people are economically motivated to care about. The modellers are human, if they aren't pumping out those results, I judge that it's alarmingly likely to be because their models simply aren't valid enough to produce forecasts sharply distinguishable from weak models like linear extrapolation.

(It looks to me as though preindustrial temperature fluctuations were large compared to deviations from the post-1800 warming trend. I know no strong reason to believe that non-CO2-driven fluctuations have calmed down since 1800. Thus, I have independent reason to suspect that modellers are peddling overfitted nonsense. But even if I didn't have that reason, perhaps in some alternative universe where written history only began in 1815, I'd still consider the lack of focus sharp specialized test predictions somewhere between "inexplicable" and "damning." )

(BRAAAAAIIINS!)

Happy upcoming Father's Day!

2009-06-03T14:20:00.000-07:00

Several times over the past ten years or so my father has mentioned an old Horizon magazine article written by a British Labour MP. Sometimes he has paraphrased passages from memory, e.g.,

The point I am making is that although the doomwatchers say they are upset at the prospect of having to sacrifice freedom, their disregard of all serious argument against the need to do so, the indecent haste with which they embrace the authoritarian option, and the self-righteous passion with which they try to ram it down everyone's throat belie their words.

and sometimes wished he could check his memory of it. As is probably obvious from the wording of that quote, I have now successfully tracked down the article ("Getting Along With Doomsday" by Bryan Magee in the summer 1975 issue of Horizon magazine). Primarily I am smugly preparing to mail a photocopy as a more-thoughtful-than-usual-for-me Father's Day gift. Secondarily, though, I'll take the opportunity to remark about the article here.

The article is about general properties common to potential catastrophes which particularly alarm leftists, and common to the political agitation around them. It seems to me that the generalizations have held up very well over almost 35 years, so well that I'm surprised that no one has been motivated to reprint the article on the web. Instead the article seems to be completely invisible: my Google search for the quoted title string "getting along with doomsday" finds only two hits, both merely the tables of contents of entire issues of Horizon magazine. So I am apparently the first WWW author to recommend the article; go me!

Unable to hyperlink to it, I shall at least quote another paragraph, from the first page:

Having lived in the period when these views were popular, I am struck by several peculiarities about them. First, although most of them are logically unconnected, and some are mutually contradictory, they were all accepted and promoted by roughly the same people. And --- this may be merely a comment on the circles I happen to move in, but I do not think so --- their appeal seemed to be preponderantly to people of a certain left-wing persuasion. Many of my acquaintances moved from one to the next as each in its turn became fashionable. Some embraced two or more simultaneously. A few heroically muddled individuals tried to believe all of them at once.

Really someone should put this entire article properly on the web, blast it! It was hard to decide which of about ten paragraphs in a row (starting from this one) I wanted most to quote; and there are various rival paragraphs in other sections of the article too.

Besides securing the rights to post the original on the web --- well worth doing, I think, because part of the force of the article today is that it's a time capsule whose frozen analysis can be compared against developments since then --- it might be interesting if some thoughtful person wrote a generalized update. The generalization that I'd particularly like to see would be not only to critique the stereotypical leftist doomsaying as Magee does, but also to critique stereotypical vaguely-classical-liberal doomsaying. (Here by vaguely classical liberalism I mean a pretty big tent, including e.g. the authors of The Federalist). After all, we have a checkered record too. E.g., classical liberals have worried about standing armies making republican states hopelessly unstable, and about growth of state power being a one-way ratcheting garrote. Neither of those has been a really good predictor for the twentieth century. (The one-way ratchet isn't a terribly bad rule of thumb, but in the twentieth century as in the three centuries before it the exceptions were very important, and we're not doing very well at predicting the exceptions.)

Vaguely-classical-liberal folk have also done some doomsaying about absolute socialism causing absolute poverty. Such doomsaying seems now to be popularly discredited, considered to be overblown scare stories falsified by history. As far as I know, though, the most influential scare stories involving absolute poverty also involved truly absolute socialism, including things like absolutely eliminating money, and not merely Soviet-style 90% collectivization of agriculture, but absolute 100% collectivization of agriculture. Thus, it seems to me that the verdict of history is slightly unclear on the effects of such absolute socialism. The much clearer verdict is on how the ratchet tends to stop before you reach such absolute socialism, leaving grey market arrangements like private farming plots to play an important part in the economy.

The twentieth century does seem to have supported the vaguely-classical-liberal doomsaying about the dangers of strong states. States so strong that they wipe out most independent power centers have been extremely dangerous to their own subjects over the past 100 years; it's hard to use the twentieth century to support the proposition that there is any threat which justifies strengthening the state to the point that independent power centers start getting wiped out.

You could try to justify a very high level of state control by invoking the threat of conquest by an equally illiberal state. However, it's not so obvious that a supercentralized state is militarily stronger than a more liberal state. Nazi Germany is the obvious scary example to support the idea that supercentralized states can be ferociously strong in high intensity war. However, knowledgeable people seem to judge most of their relative effectiveness to have been due to getting tactics and doctrine right. The history of the last fifty years seems to support the idea that they're correct, and that the things the Wehrmacht got right are largely independent of fighting for a supercentralized state.

Incidentally, my wish for a more balanced update isn't intended as a criticism of Magee writing in 1975. Given his venue, his article was plenty long and ambitious. Also, the classical liberal bugaboos are mostly claims about economics and political science under different kinds of political systems. Today we have considerably more relevant and undisputed data on those subjects than we did in 1975: the passage of time has not only created much more economic data, but also unlocked previously hidden and disputed Soviet and Chinese economic data. So I'm only wishing for someone to take advantage of the analysis opportunities that we have in 2009, not complaining that Magee improperly left his analysis incomplete in 1975.

Finally, let me acknowledge that of course bad arguments or suspect motives do not suffice as a logical justification to deny an argued-for conclusion. Therfore, even if Magee correctly identified patterns of bad arguments or suspect motives, that doesn't suffice to disprove any conclusions. Sometimes, however, a pattern of bad arguments or suspect motives can suffice as a justification for becoming exasperated.

some sniping at peer-reviewed AGW science

2009-05-31T14:29:00.000-07:00

The discussion of AGW at Overcoming Bias refers to a Physical Review Letters paper by Verdes. The first link to the paper in the Overcoming Bias post doesn't work for me, the second link costs money, but in the comments, commenter "g" gave a variant of the first link which works and is free.

The fit in the Verdes figure reproduced by Robin Hanson looks pretty good, probably about as good as can be expected given the quality of the experimental data. But it is not clear to me that it can remain so good outside Verdes' chosen interval.

At the high end: cutting off around 2001 in a paper submitted in 2006 seems peculiar, especially when linear extrapolation suggests that by 2005 the model might be diverging by more than it does in any of the years when Verdes chose to test it. And today we have data up to 2008, and it might be interesting to calculate what the Verdes model infers the CO2 level have been up to 2008. I'd rather expect it to diverge further (from skimming his description of his model, and noting that global temperatures have remained well below the strong-AGW projected long-term trend).

At the low end: while it seems fairly natural to stop fitting sometime in the 19th century (as data quality is falling fast in that period) it is not obvious that one should also stop testing the model sometime in the 19th century. There is a controversy about how much climate variability there has been in the last 1000 years, fanned by IPCC AR3 and An Inconvenient Truth promoting the famous low-variability "hockey stick" reconstruction. Since those heady days of settled science, IPCC has backed off to the variability illustrated in Fig. 6.10 of the AR4 report. It's not obvious that before 1800, when anthropogenic CO2 forcing is negligible, the Verdes model can produce as much variability as AR4 estimates to have existed. (And various IPCC critics, like me, doubt that AR4 has gone far enough: I hope that Robin Hanson will put his econophysicist hat on and make a nice time machine for betting markets, so that we can make and cleanly settle bets about pre-1800 variability.:-)

Also, setting aside concerns about the arbitrariness of the test window, is the fit good enough to justify the title "Global Warming Is Driven by Anthropogenic Emissions," and the concluding sentence of the abstract?

Here we show, using two independent driving force reconstruction techniques, that the combined effect of greenhouse gases and aerosol emissions has been the main external driver of global climate during the past decades.

That's a pretty strong statement. If we quantified the goodness of the fit over the three wiggles of the low-frequency signal within the window, would it be enough decibels of evidence to fairly paraphrase as "show ... has been the main external driver"? If I were an advocate seeking to make the most of this analysis, I might paraphrase it as "strongly support" the conclusion, rather than "show" the conclusion.

nifty nonlinearities

2009-05-21T10:31:00.000-07:00

Snow rollers. Who knew? Not me, anyway.

(h/t Watt's Up With That)

two very nifty complexity theorems

2009-05-20T15:26:00.000-07:00

theorems #2 and #3 here

(also one theorem, #1, that I can't appreciate and can't follow the proof of; and two theorems, #4 and #5, where the definitions used are new to me and I'm having trouble seeing the significance)

Incidentally, I don't find them sidesplitting as Aaronson does. YMMV, but if I were looking for a word for what he seems to be trying to get at, I'd probably call them something like "twisty" or "subversive," not sidesplitting.

fascinating artifacts, difference in perspective

2009-05-20T09:46:00.000-07:00

Reading John McWhorter's TNR post on the significance of the "prehistoric pin-up" reported in Nature I was struck by McWhorter's remark that

It has actually been long established that the earliest evidence of artistically conscious humans has been found in, as we might expect, Africa, given that it's where our species emerged. Specifically, South Africa, in Blombos Cave. There were beads made from shells, and geometric engravings on ochre --- i.e. slam-dunk "modern" tokens, unimaginable of even the smartest dog, parrot, chimp, or even Australopithecine "Lucy." And this stuff dates back to 75,000 to 80,000 years ago. No bosomy figurines, sure --- but if what got dug up in Germany was jewelry and etchings instead, we can sure there would be the same claims that here was the birth of advanced thought.

I knew very little about prehistoric art, so I had to do a little searching before I could compare the Blombos beads and engravings to Nature's new figurine. I was surprised to find that what McWhorter derides in his article as "socially unsavory," the "fetishization of artistic tokens dug up in Europe from a few tens of thousands of years ago," is exemplified by John Noble Wilford referring to "inspiration and symbolism behind the rather sudden flowering" behind the difference between this and this. I don't know about "sudden" (as per my caveat below about archaeological evidence trends being tenuously related to real trends) but referring to that difference with "flowering" or even stronger words seems quite reasonable to me.

To me, the new figurine looks like something that none of the characters in Lord of the Flies would have been able to make, while the Blombos etchings look like something the characters in Lord of the Flies wouldn't have bothered to make. And the Blombos beads are nifty, and significant, but (1) it is not obvious to me how to compare the significance of decoration to the significance of representational art, and (2) I don't know how hard it is to punch holes in shells using primitive tools, but I wouldn't be terribly impressed if a Lord of the Flies character made them in a few afternoons.) Therefore, when McWhorter says "if what got dug up in Germany was jewelry and etchings instead, we can be sure there would be the same claims" I think it seems to say more about the intellectual dishonesty of the accuser than of the accused.

When I look at artifacts like the new figurine, I feel the same puzzlement I feel when I read about ancient China or Rome: it makes me start scratching my head, wondering what developments were missing to keep the next revolution from starting. What kept the erotic figurine makers from going on to an agricultural revolution? What kept China or Rome from going on to an industrial revolution? (I do know some of the stock answers to the second question, but I don't consider the question completely settled, especially for China.) Looking at the Blombos artifacts doesn't give me the same feeling --- the artifacts seem very impressive and significant in the way that firemaking or weaponmaking or clothing are very impressive and important, but they don't seem very much like the work of modern humans marooned in the Stone Age.

(disclaimer: I'm not trying to argue against all of McWhorter's points, just his "fetishization" smear of what I see as an obvious conspicuous difference. In particular, I basically agree with his point about geographical distribution of artifacts not necessarily resembling the underlying historical reality. Accidents and unexpected cross-correlations can skew the distribution of artifact-like evidence very dramatically, and the problem can get worse when researchers start seeing what they expect to see and/or publishing what others want to hear. Such caution is important in lots of observational sciences, not just paleoarchaeology, e.g., trying to understand astronomical trends or climate trends or ecological trends or economic trends from the accidentally biased data that we have.)

This being the Internet

2009-05-13T14:10:00.001-07:00

Since I have been dumping on The Economist, it is now my solemn duty to try to be snarky about silly usage errors on their part.

"There is a widely pedalled myth that China’s growth depends on American consumers."

(h/t Carpe Diem) And then as the great mythic wheel of Internet karma is peddled, I will presumably be pwned for usage errors too.

an interesting line of questioning

2009-05-13T06:59:00.000-07:00

Mankiw channels Popper:

Going forward, what macroeconomic data would you have to observe before you concluded that the stimulus bill has been a failure? Or will you conclude, no matter how bad things get, that the economy would have been in even worse shape without the stimulus? And if the latter is the case, aren't these quarterly reports just a bit surreal?

With friends like these

2009-05-12T18:14:00.000-07:00

Besides my remarks earlier about the surprising applause lines, I was surprised by The Economist's extraordinarily limp defense of "the more liberal Anglo-Saxon model." "This newspaper stands firmly on [its] side," the authors declared in their p. 13 editorial, the editorial explaining the cover illustration of a French model grinning down at the Anglo-Saxon model from head and shoulders and indeed shoes above. And, in an article devoted to how unfolding economic outcomes have discredited the Anglo-Saxon model, the authors could find nothing about the current economic outcomes to say in defense of the model. Such loyalty, to loyally announce that you are paid to defend it! So damning of the Anglo-Saxon model that even those who take money to defend it can find no defense!

To pick on one passage, "Mr. Obama is right to admit that in some ways Continental Europe has coped well. Tough job-protection laws have slowed the rise in unemployment." This is probably literally true. But it's hard to imagine why people firmly on the side of the more liberal Anglo-Saxon model would choose to describe it in this extraordinary way. I don't know where to find comprehensive time series data, so I Googled for "France unemployment" and "USA unemployment" pages dated within the last month. Evidently in March official unemployment was 8.2% in France and 8.5% in the USA. This isn't good for the USA, but neither does it deserve mention as how Continental Europe has coped well. An 0.3% difference is well within the uncertainty of how unemployment should be defined, and measured, and compared across significantly different populations. (E.g., the two countries don't have the same age distribution, and the US doesn't participate in anything much like the intra-EU labor migration system.) In my opinion, it would be essentially as accurate to say "the crisis has caused US employment rates to rise as high as the chronically high rates of France." It might be worse for other Anglo-Saxon countries, but (1) Obama was specifically mentioned here, and (2) I don't see the definitive list of which countries are included in the Anglo-Saxon model basket and (3) even if the USA were the only one whose unemployment comparable to France, the USA will be a pretty sizable fraction of any possible basket. Thus, this statistic is a bizarre way to support the elevation of the French model.

There are all sorts of economic uncertainties about how bad the current economic problems will get. There are also all sorts of political uncertainties about how much further Obama and the Democratic Congress will move the USA away from "the Anglo-Saxon model." Perhaps in a few months the Anglo-Saxon economies will have suffered much more, and the Continental economies will have suffered much less. Perhaps over those months the US system will not have become significantly more Continental (e.g., not much more politicization of capital allocation, and no passage of card check with mandatory wage arbitration). If so, then perhaps a dirigiste triumphalist article like this could make a lot of sense. At this time, though, not so much, and it seems quite strange that people who profess to analyze the economy, much less people who profess to favor the Anglo-Saxon model, should choose to justify a cover image like with an article built on comparisons like this.

Which side is your choir on, boys?

2009-05-12T17:52:00.000-07:00

Which side is it on?

I skimmed The Economist last week and this week. I found myself surprised by some of their throwaway partisan applause lines. The lines seem like sloppy surely-we-are-all-on-the-correct-side-of-this-issue preaching to the choir. That, alas, doesn't surprise me. (I grant the The Economist can be less cavalier about that than many magazines. That's damning with faint praise if there ever was.) But I am surprised at their implicit assumption about the partisan sympathies of their choir. What happened while I wasn't looking? The choir they seem to be expecting doesn't match what my mental model for who subscribes to The Economist.

One surprising line is on page 13 of the May 9 issue. "Thirty years after Thatcherism began to work its cruel magic in Britain (see page 60), continental Europe still tends to favor a larger state." Are they casually taking a side-swipe at the obvious overall cruelty of a smaller state and/or of Thatcherism? Some audiences would consider that to be controversial, not obvious. Damned few would claim there were no downsides to Thatcherism, of course. But some would dispute that they were particularly cruelly distributed, especially since the outcomes of the kinds of policies that Thatcher fought against have been seriously flawed as well.

E.g., high unemployment in France's rigidly controlled labor market has not been good for the poor. Even the same editorial acknowledges the very high unemployment concentrations in youth (over 20 percent) and various Muslim areas (over 40 percent). That outcome alone bears comparison with various negative outcomes associated with Thatcherism. If such unemployment were present in England (or the USA) and absent in France, I expect it would rank very high on any passionate leftist's list of flaws of Thatcherism (or the USA). (And indeed, now that economic upheaval has raised USA unemployment to the level of France, the paid defenders of the Anglo-Saxon model at the The Economist refer to this this as a notable disadvantage of the USA's approach; that is, a notable disadvantage relative to France! See my following post.)

Conceivably the "cruel magic" line is intended to be ironic. The authors could be using a light touch to express what could be heavy-handedly expressed as "after 'cruel' Thatcherism began to work its magic." But if so, the touch is so light that I can't feel it even when I try. It really seems to me that that line is an intended as an earnest reminder that obviously Thatcherism is particularly cruel.

(Now, denying passports and work visas and whatnot to subjects in Hong Kong as the lease expired, that was particularly cruel and particularly shoddy. But that's not what people usually mean by Thatcherism.)

Another surprising line is in the April 25th issue, in the cover editorial on p. 13: "The Depression showed how damaging it can be if governments don't step in when the rest of the economy seizes up." This line might be less controversial than the "cruel magic" line. Still, it's a funny thing to state as simple fact if you know the kind of person who reads Milton Friedman, or Amity Shlaes, or real business cycle theory, or George Selgin. Those are not terribly popular views, but they are not terribly marginal or terribly ignorant either.

Also, while right-thinking people have long considered the Depression to be clear evidence for the importance of governments artificially stimulating economies, the very length of this history can be a bit of a problem, because stimulus fans who argued for their models by back-fitting the Depression have had several sieze-ups since then when their policy prescriptions could be tried. They famously encountered difficulties in the 1970s in industrialized countries. As far as I know it's hard to consider their record in the Third World in any decade to be a success. And while there's considerable disagreement about what lessons to draw from Japan's lost decade, I think one plausible lesson is that even by 1990 even the broadest outline of how to un-stick a stuck economy was not very well understood.

Against that history, a possible triumph is that governmental manipulations broadly consistent with the Great Depression stimulus lesson might deserve much of the credit for "The Great Moderation". But as Bernanke was at pains to acknowledge in the given link, as of 2004 there was some difficulty in assigning credit. And measuring possible costs is even harder. At one extreme, continuing "small" costs (in the sense of being too small to measure easily) could be quite significant in economic outcomes. E.g., US macroeconomic policy has tended to discourage private savings and investment. It's hard to be sure that hasn't caused a penalty rather more than 0.1% in annual economic growth; that would add up. And at the other extreme, occasional huge costs can also be proverbially tricky to estimate: how often will the steamroller catch you just as you are trying to get away with your latest shiny penny?

What is wrong with short encryption passwords?

2009-04-03T14:16:00.000-07:00

What is wrong with short encryption passwords (a.k.a. "keys")?

It is natural to want to use a short convenient password to control access to something like a bank account, a locked door, or an encrypted document. Banks and multiuser computer systems get away with using login passwords that are short enough to be easy to remember: 4-5 digits, or a single randomly chosen dictionary word. Why can't we use such short convenient keys to protect the secrets in an encrypted document?

The basic problem is that when someone gets access to an encrypted document, you (mostly) can't keep them from using their computer to try very many guessed encryption keys very fast. In this post, first I will write about why (for encrypted documents, as opposed to for some other applications of secret passwords) we can't stop people from trying guessed encryption keys very fast. Then I will write, or just do some arithmetic, to show how severe this vulnerability is for short passwords for encrypted documents.

If you try to guess the password on a stolen ATM card, the bank's ATM can make this difficult for you in two ways. First, the ATM can keep you from testing any guess quickly --- you could train yourself to type the key very quickly, but the ATM need not cooperate by giving you the answer very quickly. And second, the ATM can become suspicious (cooperating even less, and/or raising red flags for security people) when you make too many incorrect guesses.

(Multiuser computer systems can do similar things when people try to guess login passwords.)

(ATMs also have other security advantages that I won't discuss. For example, you have to be physically present at an ATM to try a guess.)

An encrypted document just sits there. Unlike an ATM or a multiuser computer system, an encrypted document (mostly) can't stop someone from trying guesses quickly. And an encrypted document absolutely can't become suspicious and change its behavior when someone tries guesses. An ATM or a computer system is a machine which draws electrical power and thus is capable of having interesting behavior which interferes with breakin attempts. An encrypted document can have no behavior. Typically it is stored in some fundamentally passive way, like a pattern of magnetization on a disk; even when it is not being stored that way, the need to make it passively storable that way keeps it from having behavior.

(I wrote "(mostly)" about not being able to stop someone from trying guesses quickly. There is one class of exceptions to that: an encrypted document can be encrypted in such an inconvenient way that it takes a large amount of computer time to decode it even when you have the correct key. This is generally impractical, because people mostly don't want to work with such inconveniently slow encryption systems. However, it is practical in a few niche applications; the main niche I am aware of is in encrypting the files which contain computer login passwords, partly by using the trivial technique of inconveniently slow encryption systems and partly by using more devious "salt" techniques.)

So now, having tried to convince you that someone who misappropriates an encrypted document will be able to quickly try guessed passwords against it, let me try to sketch how severe the problem is.

In order to estimate how fast a guess can be tried by a computer, we need to estimate two numbers. First, how much text will the computer need to decode in order to detect that it has made a bad guess, so it can go on to the next guess? It might be as few as 10 characters, but 100 characters is a more conservative guess. Second, how fast can a computer decode those 100 characters? We will assume that it can do that about as fast as it can read characters from a hard disk, because it is inconvenient to use encryption schemes which are much slower than that, so people tend to avoid them. In round numbers, then, a typical laptop computer can decrypt on the order of 10 million characters per second, so it can try about 100,000 guessed keys per second.

(Will someone who misappropriates an encrypted document be able to run the software to try these guesses? Maybe not. But the population of people who can run software is pretty large. The population of people who can write the software is smaller, but it only needs to be written once. (It is also pretty simple software, easy for an experienced programmer to write, and possible for a motivated teenager to write.) Useful software tends to get widely distributed. Software useful for mischief --- decryption programs, keystroke loggers, trojan horse programs, etc. --- gets distributed in some highly visible ways, but various less visible ways too. It is seldom safe to assume that someone who is motivated to break into an encrypted document will not have access to such software.)

What are the implications of guessing 100,000 keys per second? It takes one second to try all possible 5-digit keys. It takes a few hours to try all possible 9-digit keys. It takes about one second to try all dictionary words. It takes a few hours to try all sequences of six letters and numbers, such as "jrr6d9".

Roughly, then, the security you get from encrypting a document with a short easy-to-remember key is comparable to the security you get from locking a physical box with a padlocked hasp whose screws aren't protected by the hasp, so that the box can be opened simply by unscrewing the screws. (Since decryption software isn't as common as an ordinary or Phillips screwdriver, perhaps the screws should be something less common like TORX screws.) Such an arrangement does make it clear that you intend the message to be private, but it causes no more than a minute of inconvenience to someone who chooses to ignore your intent.

So now what? If expressing but not enforcing the intent to privacy is enough, use a short password and be happy. If instead you want to use encryption passwords as a serious obstacle to unintended reading, it is a well-studied problem, and I am neither an expert nor motivated to write a complete survey of what I do know, but some possibilities are: annoyingly long random keys (e.g., 16 or more random letters and numbers), passphrases (a messy subject, but think 5 randomly generated dictionary words), various schemes using "public key cryptography" to do an end run around the problem, or various schemes replacing the hard encrypt-a-document problem with the easier (ATM-like) authenticate-a-user problem. A useful programmer's reference for many problems related to this is Schneier's book Applied Cryptography; unfortunately I don't happen to know a comparably useful nonprogrammer's reference.

ICFP 2008 Rules: A Geeky Disappointment

2008-07-11T10:08:00.000-07:00

The annual ICFP Programming Contest is one of the highest profile programming contests in the world. They publish a task --- e.g., write a program which processes a contrived formal language or which implements a strategy to win a novel game --- and give 3 days to solve it. (Or 1 day; see below.) While the contest is associated with the International Conference on Functional Programming, it is open to anyone in the world.

Many years I have tuned in to check the announced task, and decided whether it looks like enough fun to do. Only once has a fun task and a good mood coincided, and I entered the "lightning" subtournament (with only 24 hours to produce a solution) for the 2004 contest. This year the task will be announced this afternoon, so I checked the website to look at the rules. And ugh.

For years, the contest designed have done clever things to dodge the difficulties of running all conceivable user-submitted program on their test machines. It's not a trivial problem. There are the usual problems of malicious programs. The contest encourages entries in all possible programming languages --- one of the main "prizes" is an announcement, at the conference, of bragging rights for the language used in the winning programs. And the contest has the usual anti-credentialist esthetic of the mathematical sciences, proudly taking entries from anywhere, even some marginal surplus MacII kept running in a junior high-school outside Jakarta.

Many entries in various previous years have finessed the problem by (roughly speaking) requiring entrants to send in the results of their programs, rather than sending in executable programs in standard programming languages. Alas, I find that this year they've switched to a brute force solution that I don't much like like in principle, and which in practice looks like too much trouble for me. Now they want the participants to download a standardized version of Linux and get their programs running under that. It's not a terrible solution, but it isn't particularly elegant either, and it would call for me to spend at least an hour or two downloading and preparing before the contest. I can't be bothered, given that I probably only had a 25% chance of being motivated by this year's task anyway.

Can I redirect my frustrated geeky energies to something more obviously productive? Um.

Fun With Lists: Between a Programmer's Bookends

2008-04-06T15:34:00.000-07:00

I read grokcode's top 9 list of programming books, and I partially agree with it.

What nine and a half books would I recommend for a programmer's bookshelf?

I heartily agree with grokcode's choice of (1) Cormen et al.'s Introduction to Algorithms. I worked for about a decade with scientific/simulation software mostly in Fortran 77. This gave me plenty of exposure to the work product of extremely smart self-taught programmers who never got around to studying even trivial data structures like linked lists, and I say with deep feeling, don't let this happen to you! Reading Introduction to Algorithms, and keeping it handy, is an excellent way to prevent it.

I would add (2) at least one more book oriented toward whatever specialized algorithms and data structures suit your work. Such a second book might be Russell and Norvig's Artificial Intelligence: A Modern Approach, or Okasaki's Purely Functional Data Structures, or Motwani and Raghavan's Randomized Algorithms, or Dechter's Constraint Processing, or Press et al.'s Numerical Recipes. (Incidentally, use the last one with caution: I have heard other people complain about poor judgment calls in it, and I myself wasted more than a week of work in grad school by using a flaky pseudoRNG recommended in the then-current edition. But I don't know any comparably clear and comprehensive overview of the subject, and its potentially-flaky summary advice comes backed with a solid bibliography, so I still consider the book very useful.) Or the second book might be something more specialized: something like Schneier's Applied Cryptography, or Clarke's Model Checking, or a book on compiler construction techniques like grokcode's recommendation of Aho et al. Compilers: Principles, Techniques, or one of various newer compiler books with rather different emphases, e.g., Queinnec's Lisp in Small Pieces.

Grokcode's recommended Structure and Interpretation of Computer Programs is a very good book to study and even to reread, but it isn't the kind of book I actually refer back to with any regularity, and so it got squeezed off my list.

Grokcode's recommended The Mythical Man-Month is also very good, but when limiting myself to nine and a half, I think I prefer some recent books. How about (3) Hunt and Thomas's The Pragmatic Programmer and (4) Fowler's Refactoring? Between them, these two also squeeze out three other basically-worthy books on the grokcode list: Programming Pearls, Code Complete 2, and Design Patterns.

I agree with grokcode's recommendation of (5) The C Programming Language. But I think there should be at least one book on a language with much more support for abstraction than C. (And Structure and Interpretation of Computer Programs isn't my first choice for such a book.) Somewhat more than fifty percent of the code I write is so straightforward that it hardly matters what language it in. E.g., Fortran 77 is in many ways horribly inexpressive, and I worked in it for years and had many periods of intense frustration, but even in f77 many things remain easy to write. However, that's 50+% of code seems to take about 25% of my time. I find the 10-30% of hard code in a project often takes more than 50% of my time, and I find that for such code, choice of programming language can make a considerable difference.

(Language choices should come with strong disclaimers, both "horses for courses" and "de gustibus et de coloribus non disputandum est;" consider it done.:-) In principle, I can easily see the point of choosing C++, Common Lisp, Haskell, OCaml, Scheme, or SML. And in practice, given tradeoffs involving availability of implementations and libraries, I can see the point for various other choices in some cases: e.g., Java, Javascript, Mathematica or Maple, Perl, Prolog, Python, or Ruby. Still and all...

I find I prefer CL for about 70% (weighted by time spent) of projects. (Too bad about all the cruft in CL, too bad about CL being a niche language with the attendant weaknesses in implementation and library support, and too bad that CL's design makes routine program-analysis jobs fundamentally undecidable. More important than those drawbacks is that CL makes it easier to express things OAOO than any other practical language I know. It's also a significant plus that CL does unusually well at implementing hard and soft layers in the same language, and at letting very hairy toolboxes be naturally interactive.) And I find I prefer C++ for about 15% (again, by time spent) of projects. (C++ goes about as far in abstraction as anyone has managed while remaining a systems programming language naturally suitable for implementing OSes and for compiling onto a $0.40 AVR embedded microcontroller. And too bad about all the cruft, again, and about people choosing C++ much too often for complicated applications from about 1985 to 2000, and about how entire books can be written about C++'s idiosyncrasies and outright gotchas. Still and all, C++ libraries written with taste can be pretty good, and did I mention that C++ is a systems programming language? Today it is no longer a particularly good idea to write things like compilers or mail processing systems or web browsers or visual editors in systems programming languages. But for things which still IMO truly should be written in systems programming languages, so that IMO the serious competition is limited to Ada and C and some languages that I'm only vaguely aware of --- BLISS? Modula? Oberon? --- C++ starts to look pretty good.) So I recommend (6) any of several good books on CL: Seibel's Practical Common Lisp, or Graham's Common Lisp, or Norvig's Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp, and (7) Stroustrup's The C++ Programming Language. Note, too, that the Norvig book and the Stroustrup book are not only good books on their chosen languages, but also pretty good books on various aspects of programming in general.

To my taste, grokcode's Unix Power Tools doesn't really belong on the list: certainly the Unix tools are very useful, but learning them from the man pages seems to work adequately well. A book on Unix system administration might be worthy of being on the list, instead ... but you only get nine books. So you decide: if you think you're going to spend more of your time fooling around with system administration than you will spend fooling around with embedded microcontrollers and high-performance graphics and FFTs and BDDs and whatnot, then replace Stroustrup's book with a book like Frisch's Essential System Administration.

You also probably want a book on your preferred default way to provide GUIs and/or web interfaces to your software. Just as I didn't have a definitive choice for languages, I don't have a definitive choice here: it seems to depend strongly on your tastes, on how interactive your software tends to be, on how stably portable your software must be, and on what environment you typically run under. If your tradeoffs look like mine do currently, then (8a) JavaScript: The Definitive Guide might be a reasonable choice. But, e.g., someone with higher-bandwidth interaction requirements might laugh condescendingly and replace it with (8b) a book on something like OpenGL. Or, if your code always resides so deep in the system that you laugh condescendingly at the lightweights whose software provides any GUI at all, then replace this with (8c) a good manual on the most important API/architecture that your code talks to: something like the Stevens books Advanced Programming in the UNIX Environment or Unix Network Programming, or the reference manual for the CPU that your compiler produces code for, or maybe a book on relational databases or some such thing.

Finally, I'd nominate (9) a book on something reasonably fundamental and promisingly important for your work. For me currently this would most likely be a book related to machine learning: e.g., Mitchell's Machine Learning, or Norvig and Russell's aforementioned AIMA, or Gruenwald's The Minimum Description Length Principle. For someone else, it might be a book on modern techniques for expressing algorithms and data structures so that they scale to parallel hardware, or a book on ways to prove correctness of systems (like the aforementioned Model Checking, or like Bertot and Casteran's Interactive Theorem Proving and Program Development), or a book on virtual worlds, or a book on online reputation systems, or whatever.

And even more finally, vaguely in the spirit of the "and one half" in the title of the grokcode article, I'll nominate not-quite-a-book: choose (9.5) some significant open-source software project that you're impressed with, and study it enough that you understand the design choices and tradeoffs. Or if you prefer an actual whimsical book recommendation parallel to grokcode's recommendation of Hitchhiker's Guide to the Galaxy, then while I certainly enjoyed Hitchhiker's Guide, I think its connection to programming is weak enough that I'd prefer to nominate either the first three of Rick Cook's "Wizardry" books, or Vernor Vinge's The Peace War.

(And what books have been physically between the bookends on my desk and referred to in the last month or two? Three of the books I listed above: Introduction to Algorithms, Purely Functional Data Structures, and Javascript: The Definitive Guide. Besides those, lab-book-style handwritten logbooks for various of my computers, and ancient editions of Lamport's LaTeX: A Document Preparation System, Wall et al.'s Programming Perl, and The American Heritage Dictionary. A CL book would be there too, except that these days I'm sufficiently familiar with the language that when I need to look something up I can find it in the ANSI standard, and I use the HTML version of that.)

edit: added one leftangle-P-rightangle HTML tag before each paragraph after being burned for the Nth time by Blogger's inspired combination of not-quite-HTML formatting rules and not-quite-preview-function

my new snark idol

2007-11-21T12:22:00.000-08:00

My new snark idol is Tanta. (With considerable extra credit for (1) being sharp and knowledgeable and generously sharing the knowledge with the world and (2) being openly and justifiably grouchy about the general seriousness of the situation, and specifically about how many people like her were undervalued and poorly treated in the runup to this, but avoiding using that to rationalize presenting impassioned-but-shaky arguments. Number 1 is not too rare on the Internet but still not to be taken for granted; number 2 seems too rare.)

State of Flow

2007-11-08T10:13:00.000-08:00

I just emerged from however long it took me to write about four pages of reasonably-mathematical LaTeX text (about four hours, I think) to finally notice that I had been listening to the first Goldberg variation the whole time (where my intent had been the milder repeat-them-all mode). As I twitch rhythmically I wonder whether I have rediscovered technology from the brainworld in A Wrinkle in Time (M. L'Engle, RIP).

Griefers of Urban Dead: Some Design Suggestions

2007-09-23T10:37:00.000-07:00

I think Urban Dead is an impressive game. I've played it with four characters for somewhat more than a week. I played as a newbie, of course, but an unusually well-read and thoughtful one: not only did I spend hours carefully reading more than 100 pages of strategy guides and other information on the Urban Dead Wiki, I have also started reading Bartle's very interesting book, Designing Virtual Worlds.

While the game is impressive, it seems to have a unusually large problem with "griefers," people who play to annoy other people, not by struggling within the advertised worldview of the game, but by exploiting marginal niches of the rules and declaring victory when they cause harm to players who are nominally on their side. I'm not really looking for a long-term gaming habit, but even if I were, I think the level of griefing would be enough to drive me away soon. I'm really impressed with the basic design of the game, though, and I wonder whether two simple tweaks might be enough to dramatically reduce the level of griefing. Before I (probably) leave, then, I'm motivated to point out two griefing patterns, and to suggest two ways to fix them without (I think) messing up the intended dystopic fun of the game.

In Urban Dead, players are survivors or zombies in a quarantined city after a zombification plague. The main theme of the game is the usual zombie movie mythos, where survivors struggle against zombies. Since a large fraction of players are zombies, though, the mythos is seen from both sides, and a lot of design work went into giving zombies interesting things to do and interesting skills to learn by doing so. As usual in a massively multiplayer online roleplaying game, death is somehow not permanent. In Urban Dead, the impermanence is achieved by having dead survivors rise as zombies, and by providing ways for scientists among the survivors to cure zombies so that they become survivors.

Both griefing patterns I will discuss have to do with players whose characters are survivors, but whose fun as players comes from betraying the interests of other survivors. Of course that is dystopic, and I don't think it should be forbidden by the rules. Unfortunately, the rules not only permit it, but favor it by allowing it to be undetectable, and by letting any survivor easily learn things which need to be concealed from the zombies (but which the griefer sends to zombies through of out-of-game communication channels). I think the first, especially, follows from a design mistake, and that the second could be handled better. I will try to explain why these patterns are bad at all, explain what in the rules encourages them, and suggest rule tweaks that I think would help fix the problems.

Betrayal is a great dystopic theme, perfectly suitable for the zombie mythos. However, endless betrayal with no risk of detection is something else. You could make a gripping psychodrama about endless betrayal with no risk of detection: a movie where the drama comes from the thoughts and decisions of the invincible undetectable betrayer. (It's a psychodrama because there's little drama in the action. The betrayer succeeds again! What a surprise.) But whether or not the invincible betrayer psychodrama had zombies in it, it would not be "a zombie movie." It would be a movie about the corrupting temptation when the usual real-world consequences of dishonesty have been magicked away. In Kevan's virtual world where the rules eliminate important real-world constraints on betrayal, there are in effect two games, one corresponding to the zombie movie and one corresponding to the psychodrama. For the ordinary customer, trying to play a game corresponding to a zombie movie, most of the rules are cleverly set up for a fun game in a zombie movie world, but the world turns out to have so many griefers in it that in practice the griefers seem to be as much of a threat as the zombies. And unlike the carefully-crafted zombie threat, the griefer threat is no fun, just a headache. For the griefers, though, it's Christmas come early: a psychodrama where they get to star as the invincible betrayer, and again and again defeat the real live humans that they know are playing the bit parts.

How much of a headache are the griefers of Urban Dead? I see two significant headaches, and the first one is not just significant but huge. I played carefully, using all sorts of information from player guides. From time to time my characters died anyway. And how did they die?

Memenie the consumer: died once, killed by PKer "not l33t"
Olliman Thrucke the zombie: started as a zombie, was revivified by a wandering lab assistant before he even learned a single new zombie skill, and afterwards played as a human without getting killed so far
Visseck the medic: died once, killed by PKer "Grymlock"
Jadette the lab assistant: not killed so far

So I don't know what the long-run risk is, but if 2/2 of my deaths as a survivor were from randomly malicious PKers, it suggests that randomly malicious PKers are currently the biggest threat to a careful survivor: it's not a city overrun by zombies so much as a zombie game overrun by griefers. And if I'm wrong that the griefers are the biggest threat (possible, because of course a sample size of 2 is pretty small), at least they seem to be more than 30% of the threat. I'm quite confident that non-PKers are in the large majority of players, and that a large majority of them would agree that the high PK risk is not a fun feature, just an annoying bug.

Now, of course much the fun of UD comes from dystopic challenges, so I should defend my "just an annoying bug" claim a little before I go on. So...

Is PKing a fun challenge for the PKee? No. It is an obstacle to succeeding in the game, but that doesn't make it a fun challenge. A bad Internet connection would be an obstacle to succeeding in the game, but that doesn't mean that technical problems on one's Internet connection are a fun challenge. Coping with zombie challenges by playing a hide-the-pea game and by working with barricades and by developing skills and local knowledge and associations? That is paranoid dystopic fun: bravo to Kevan. Finding a flak jacket and learning the body building skill and taking care to sleep in out-of-the-way buildings that are slightly less convenient for a free-running PKer to search, so that one can reduce one's PK risk by 30% or so? No, that is unsatisfyingly ineffective, and even if were effective it wouldn't be fun. It's about as satisfying as buying a UPS to increase the reliability of the service provided by the company to whom the local government has granted a monopoly on power distribution.

The dystopic atmosphere is fun, and uncontrolled PKing is all tangled up with the dystopic atmosphere. But as above, it screws up the zombie movie flavor of the dystopia. If you expect most of your customers to be happy playing bit parts in a movie starring a minority of invincible sociopaths, it would probably be best to advertise the game that way.:-| And even if you advertised for them, I think you'd find them scarce. It may not be normal for someone to enjoy playing a bit part in a zombie movie, hiding from zombies, building improvised defenses, and cooperating with other survivors. But it's further up on the normality scale than enjoying a bit part as someone unable to defend oneself effectively.

In my opinion, the key problem is that it's impossible to identify the characters responsible for the PKing. Does that sound absurd? Yes, of course you know who actually shot you, and groups of survivors do have well-developed PK lists. Yes, those lists are apparently effective in giving griefers a real risk of being killed by bounty hunters, and are effective at giving griefers a seriously reduced chance of being revived in the ordinary way (at "revive points") by other survivors. But it's hard for bounty hunters to keep griefers down: as usual in Urban Dead, the only really effective way to interfere with survivors is to interfere with their revives. And unfortunately, interfering with ordinary revives at revive points is not nearly enough to interfere with the revives of customers who enjoy starring in an invincible betrayer movie.

Once you remember that to really interfere with a survivor, you must interfere with revives, it follows that the characters ultimately responsible for persistent grief PKing are the secretely-griefer lab technicians who revive the openly griefer PK button men. And that's what I mean about the PKee not being able to identify the responsible character: the game provides no way for PKees ever to detect such revives, and it's almost impossible to find such secretly-griefer characters otherwise. (Anti-zerging rules mean two or more griefer players must cooperate to do this. The main ways I can see for ordinary survivors to interfere with it at all involve setting up triple agent decoys to interfere with the process of two griefers finding each other in-game to set up such cooperation. A triple agent could remain loyal to the survivors while pretending to offer such betrayal-by-backscratching arrangements, and then report on any mere double agent who takes them up on it. Clever, perhaps. Certainly devious. But I think probably only very slightly effective.)

So if survivors can't fix it, can Kevan? Without gross rule changes which would mess up Malton's atmosphere of zombie movie dystopic fun? (I.e., without changes like a "PK flag", or simply making it impossible for survivors to harm other survivors.) I bet it could be fixed simply by making revives not anonymous. I think the most natural in-mythos way would be for any lab technician's revive record to be publicly accessible at any Necrotech terminal, but other ways would work too. A really convenient way for players would be to have the history of revives performed upon a player, or by a player, to be visible in his profile. A moderately convenient way would be for the history of revives performed upon a player to be visible whenever a DNA scanner is used on him.

Even with such a change, it would still be possible for griefers to set up revives bypassing the revive points. The obvious way to do it would be for a group of griefers to continuously generate disposable alts who begin as Necrotech employees, are quickly trained up to learn the skill of reviving zombies, then discarded once they get a reputation for reviving known griefers. If it were sufficiently easy to make such disposable alts, then despite nonanonymous revives, possibly griefers would remain so common that newbies like me would still encounter 2/2 deaths from them. But I doubt it. I'm no expert on human psychology, but I'd guess that part of the reason there are so many highly active PKers in the current system is that the activity required to prop up the openly-PK character isn't considered work, but in fact is part of the fun for a griefer type. Maintaining a betraying alt takes a little time, but it's time spent on constant invincible betrayal. To a certain type of mind, this is winning. Winning! All the time! Creating and levelling new alts might take only five times more time (or whatever). But I think the time spent on that would probably be just tedious, not malicious fun and reminders of what a winner one is. I bet that would make a big difference.

(So much for my ideas about the first problem.)

The second problem I see is probably less severe, and it hasn't bitten me even once. But I'll mention it anyway, partly because judging from the Wiki it seems to be a real problem, partly because it seems to be another perverse riff on the same invincible-anonymity theme, and partly because my proposed solution interacts with the PK griefing problem.

The basic mechanics of the game depend heavily on survivors hiding from zombies in barricaded buildings which survivors can get into easily, but which which zombies can't see into, and which take work for the zombies to break into. It doesn't take all that much work for a group of zombies to break into any given building, though. And if a group of zombies knows that a group of survivors is in a building, the game mechanics basically favor a zombie group breaking in and chowing down. Effectively fighting zombies is pretty hard for survivors, and much of the protection for survivors is derived not from protecting a given building, but from hiding in several buildings, trying to make the zombies waste much of their effort on breaking into buildings which turn out to be empty. However, this protection goes away almost completely when zombie groups ally themselves with survivor alts whose role is to betray the other survivors by scooting through buildings looking for other survivors for zombies to eat, and telling the zombies by out-of-game channels. Again, the zerg rules stop a single player from using a survivor betraying alt to help his own zombie character. But again, it's trivial for multiple players to cooperate to achieve it by using their betraying alts to aid each other. Is there any reasonable rule change that would reduce this problem?

The game won't work unless survivors can get into buildings quickly, so it's hard to stop zombie-aligned survivor alt spies from getting into buildings quickly. But is there any need for survivors to notice other survivors quickly? How about making it possible for humans in buildings to hide from other survivors? It could even be the default, but my proposal would be instead to make it require an explicit action costing an action point. Then after you hide, other survivors entering the building won't automatically see you. This needn't unbalance the game much at all; in particular, if a zombie is able to enter the building, then hiding needn't be any defense against it, because a zombie can smell harman brahnz so it instantly knows who is in the building with it. Hiding doesn't need to be absolute concealment from other survivors, either: perhaps for every action point spent searching, a survivor can find one survivor who is hiding. But I think even such a weak hiding system (require action point, can't hide from zombies, revealed to other humans by search) would demote free-running zombie scouts from their starring role in their own personal elite unstoppable sociopath betrayal movie: they'd be back in the same zombie movie as the ordinary customers.

I think neither of these two rule tweaks (nonanonymous revives, survivors hiding from other survivors) would be enough to interfere with the main, advertised zombie movie theme of the game, but I hope that they would be enough to screw up these two popular griefer strategies. As far as I can see, the main downside risk of the tweaks is that the second tweak (hiding) could change survivor social interactions in various ways, many of them impossible to predict. It could change shallow interactions like looking someone who needs healing, and it could also change deeper ones like noticing who hangs out in an area and who thus might be worth developing long-term relationships with. It would also interact with the PK griefing problem in various ways which I can't fully analyze: e.g., it would make it slightly harder for the griefer to find victims, but also make it substantially harder for bounty hunters to hunt PKers.

On Slack: One Reason Why Software Maintenance is Hard

2007-03-25T14:00:00.000-07:00

Software maintenance tends to be hard in part just because software tends to be hard. Most of our effort is spent on quite complicated software, running to hundreds or thousands of pages in printed form. And software tends to be fragile: for example, any small typographical error anywhere in all those pages tends to invalidate the program, or just to subtly corrupt it, so it behaves incorrectly occasionally.

However, I wish here to remark about software maintenance in particular, the problem of taking old software and making it better, or adapting it to a new purpose. And in particular, I want to talk about adapting old software to a new purpose, as opposed to finding and fixing lingering mistakes from when the software was built for its original purpose.

I have spent a lot of my mental effort over the past few years on "incremental algorithms." Such algorithms efficiently maintain a solution for a problem as the problem changes. For example, an ordinary sorting algorithm, when given input [101,8,4,11,14,88,89,16,12], might return [4,8,11,12,14,16,88,89,101]. An incremental algorithm allows an operation like "now insert 55 in that sequence," and efficiently returns [4,8,11,12,14,16,55,88,89,101] in that case.

Of course, even without any special incremental algorithm, you can just use an ordinary sort algorithm to find that answer: insert 5 into the input sequence, then restart the sort algorithm from scratch to build a new output sequence. However, for enormous input data sets, an incremental algorithm can be enormously more efficient. For exmaple, if you maintain a telephone directory covering much of the USA, containing perhaps a hundred million phone numbers, then re-running an ordinary (non-incremental) sort operation from scratch will tend to require at least checking the order of a hundred million pairs of numbers. (More or less, the algorithm must at least re-check the order as it runs.) An incremental algorithm needs less than one hundred comparisons to insert a new element in an existing ordered set of one hundred million elements.

One thing I have noticed, which I'm sure others have noticed and even perhaps formally analyzed, is that incremental algorithms need to have some slack in their representation of the problem. Consider the incremental sorting algorithm again, but this time not for telephone numbers but for library books in card catalog order. Any library will have some gaps on the shelves, quite intentionally, for libraries face the problem of inserting new books in the order. When a new book is delivered, a librarian wants to walk to the appropriate location, slide aside a few dozen books, and insert the new book. A library could economize on space by eliminating all the gaps. However, then inserting a new book in order could require quite a lot of work, sliding about half the library's books aside.

Similar things happen in filing systems inside your computer. On a computer's hard disk, where new files may be created, and old files may be expanded or deleted, usually quite a lot of space (20% or more) is reserved for such slack. This is not just a property of the way computers read files, for such slack does not usually exist on something like a CD-ROM, which will never be changed. It is instead a property of "who knows what the future will bring?"

It is an interesting challenge finding the appropriate form of slack for a given set of anticipated changes. First, though, note that the appropriate slack really does depend on the anticipated changes. To go back to the library example, the gaps in the shelf aren't just common sense, agnostic about the kinds of anticipated changes. Instead, they are more nearly slack specifically designed to cope with the anticipated problem of inserting a new book. Consider an unanticipated challenge, when Tom Gander seizes power, decrees that T and G are now the first two letters of the alphabet, and starts exterminating librarians who don't comply. At that point, the gaps on the shelves don't help, if you are going to comply efficiently, you need is wheels on the bookcases. (Or perhaps you should think outside the box and maintain an arsenal or cross-border escape tunnel in the library basement.)

I've been thinking about the problem especially in the last few weeks because I finally succeeded in finding a promising form of slack for a problem I've been working on, after thinking about for (not full time, but off and on) only a few years. It's a slightly tricky problem, related to the one solved here (where given any directed graph, one must return a minimal directed graph which is congruent to it). But compared to the problem solved by programmers (where given any problem specification, one must return a piece of software which solves the problem) it is hardly tricky at all. Thus, since it was hard for me to find a suitable form of slack to cope with one anticipated set of changes in my slightly-tricky problem of directed graphs, it now makes a lot of sense to me that it can be really hard to find suitable forms of slack to cope with anticipated small changes to programming problems so that it's easy to make a small change the program to handle the changed problem. (That's the main point of this post, any programmers reading this can stop now.)

Besides that, for any non-programmers reading this post, I'll illustrate an extra point, using the magnificent foundation of examples that I have constructed: unanticipated changes in software can be surprisingly hard. (You programmers already knew this, why didn't you go home?) If you don't know anything about the inside of a library, it might not be clear that changing T and G to the first two letters of the alphabet is harder than shelving a dozen boxes of new books. After all the T-and-G change is so simple that it can be described in a single sentence, while the dozen-boxes problem requires many pages to describe. (I.e., the receipt listing the titles of all the new books is many pages long.) However, in an ordinary library, the T-and-G change would probably be a lot more work. So if you are ever tempted to speculate how easily a software system can easily be changed to accommodate a simple change, remember that the answer is "it depends." By that I don't only mean that it depend on whether the programmers are lazy morons (though it does:-), but that it depends on design choices inside the software system, and those design choices depend in part on whether the change was anticipated around the time of the original design. (This is related to this; not quite the same thing, but said enough better than I can that nonetheless it deserves a nod.)

For example, the SBCL project (a free compiler that I set up years ago, and whose maintenance I still formally head) experiences chronic headaches because its design (made over twenty years ago in the CMU CL system that I copied for the original SBCL code) calls for most of its code to be stored at a fixed address in the computer's memory. Twenty years is a long time in the computer world. In the kinds of computers that SBCL usually runs on (general-purpose computers running operating systems like Linux and Microsoft Windows, not special-purpose computers like the ones which control antilock brake systems), it is now pretty abnormal for application software to expect to be stored at a fixed address in memory. Thus, this design now causes various kinds of friction. However, it looks like it would be a lot of work to rework the system to use an approach which is less surprising to modern operating systems. It's not an impossible amount of work, but it's enough that I find it unsurprising that despite all the headaches no one has done it yet.

Google vs. Intellectuals

2007-03-18T12:09:00.000-07:00

It's annoying to me to find smart, well-educated people arguing things which are inconsistent with what a non-specialist can find with an hour of searching Google — and not addressing the inconsistency.

Now, I appreciate that there can be rarefied truths which are hard for a layman to check, or even to understand. After all, I did my Ph.D. work on simulating quantum mechanical systems, in systems where the primary difficulty was the Pauli exclusion principle, and I don't know of anyone who has much success discussing such stuff with nonspecialists. My point, though, is that even rarefied truths should correspond to simple reality. My specialist experience doesn't much tempt me to waffle on that, but instead tends to make me a hard-liner.

For example, en route to my Ph.D. I was in an undergraduate quantum mechanics class. I questioned the formula which had just been derived for reflection of objects at a boundary where potential energy changes. I pointed out that the formula seemed to say that an ordinary macroscopic ball — such as a tennis ball — would have a high probability of bouncing back when one attempted to roll it over the edge of a table. (There is a standard way that quantum mechanical effects tend to fade away for large objects, but in this formula they didn't fade away that way.) How could that be right, since that doesn't happen in real life? The professor didn't respond that the question was uninteresting because it was so mundane and un-quantum-mechanical. Instead, he addressed it very seriously, as essentially any physicist would. (And ultimately we reached an answer which didn't change the formula, but which made it explicit that the formula applied only when the boundary was sharp compared to the quantum wavelength of the reflected object, in a way that macroscopic boundaries never are in practice.)

Life is short, and I wouldn't want to require intellectuals to be compulsive about descending to check every mundane implication of their ideas. Thus, I have no problem with a physics professor lecturing on a particular reflection formula without having thought out how it applies in the limit of tennis balls rolling off tables. But I do expect intellectuals to take such mundane questions seriously, and I do expect them either to be able to handle such questions (as in the reflection case, where the observed behavior still followed from ordinary quantum mechanics once more details of the problem were included) or to revise their position.

I also had the interesting experience of testing out of an inorganic chemistry requirement for graduate school by taking a multiple choice test. My undergraduate background was a biology degree with some extra physics courses, and some research experience in simulating biomolecules in solution. That left a sizable hole where a chemistry graduate school normally expects one's inorganic chemistry qualifications to be. But I had read a lot — essentially no inorganic chemistry specifically, but lots of things about what materials are technologically significant, and what was developed when, and so forth. I was amused that that — history and fiction and engineering and physics and so forth — was enough to constrain the answers to the questions enough to pass the test. That can be interpreted in lots of ways, of course: a common interpretation is "multiple choice tests are silly." But one interpretation is that even a specialized field can touch on common knowledge in so many places that it is highly constrained by the need to conform to nonspecialist knowledge.

Thus, it is annoying to find things like the examples which follow. Perhaps the first example could be somewhat excused by the difficulty of stating one's points precisely within a one-page opinion piece. (Though, ahem, trimming some of the padding like "noisy champions today" might help with the length constraints, hmm?) Perhaps the second example could be resolved by aggressively going after the details of problem, more or less in the same spirit as my physics professor did (but likely by poking around in a research library instead of by reanalyzing more carefully with pencil and paper and calculus). But they still grate, since I doubt that either excuse actually holds.

Example I: Paul Johnson wrote in Forbes (June 20, 2005) that (among other things)

Neither Darwin nor any of his followers — nor his noisy champions today — was a historian. None of them thought of time historically or made their calculations chronologically. Had they done so, they'd have seen that natural selection works much too slowly to fit into the time line allowed by the ages of the universe and our own planet.

Eh? If you were to search Google for Darwin's own most prominent work ("On the Origin of Species...") and surf around a bit, you'd have any number of routes into the original text. The natural one for me is "project gutenberg origin of species" which gives you a free downloadable copy of the text. Search for the text "here we encounter a formidable objection" and note that Darwin himself is already using numerical bounds on the age of the earth, committing himself to within a factor of ten or so (not unreasonable before knowledge of things like radioisotope dating and genetic clocks), and seriously addressing criticism based on those estimates.

Furthermore, this seems to be a case of the pot calling the sun black. Paul Johnson doesn't seem terribly interested giving in any estimate within a factor of ten how long it would take ordinary (non-divine-intervention) natural selection to evolve organisms comparable to modern lifeforms. That is: Johnson claims "natural selection works much too slowly to fit into the time line allowed by the ages of the universe and our own planet." Then how much too slowly, and using what estimate of the ages? Would it take fifty billion years for natural selection to evolve organisms comparable to modern ones? Fifty quadrillion years? Charitably assuming that Johnson the historian troubled to familiarize himself with Origin of Species, he must be criticizing Darwin only committing himself to within a factor of ten or so. Darwin had rather less to go on than we do: mostly taxonomy, folk genetics, and the fossil record, I think. In this age, when many details of mutation and genetic relatedness are well-known down to the last atom, wouldn't it be reasonable to expect Johnson to commit himself too? If not quite as precisely as Darwin did, then at least within a factor of a million?

Example II: The collective-right interpretation of the second amendment has been bouncing around the blogosphere recently, since it turns up (renounced by the majority and embraced by the minority) in the recent Parker v. District of Columbia decision. A moment's Googling brings up other high-profile bills of rights: anyone who had a civics class in the USA might know states tend to have bills of rights, so it's interesting to check big states like NY (right? what right?), PA (individual), VA (universal militia). (All are reachable with obvious queries like "virginia bill of rights".) Anyone who knows a little more history might know there was debate about ratification of the Constitution, that the US Bill of Rights was not unrelated to that debate, and that there's something called the English Bill of Rights, too. Given that those were in the air around the time of ratification, it's bit odd that even with the resources of research libraries, the collective rights folk seem unable to come up with contemporary documents which interpreted the document as having even a possible ambiguous collective right interpretation, whether to support it or to criticize it. After all, the Federalism debate generated quite a lot of writing. Why does doesn't some fraction of it turn on that? Imagine if we had a few historical documents like Paine (or, at the opposite religious end, some Catholic) arguing for the inadequacy of the guarantee protecting the right of local governments to choose who to arm. And, perhaps, Hamilton arguing that it was not only adequate but much wiser than the old-style guarantee to all individuals (of the religion on the winning side, anyway) in the English Bill of Rights. To judge from the tenor of the arguments from the collective rights folk, you might be forgiven for thinking that we do have such documents. To my rather confident knowledge, we don't. Certainly, if we do, collective-rights advocates are inexplicably reluctant to remind us of them.

From the extreme individual-rights point of view where the words of the amendment have suspiciously-convenient unmodern interpretations (like "well-regulated" meaning "smoothly-operating"), of course this is a non-puzzle. But is it explicable for a collective-rightist? Sadly, despite having followed the debate off and on for years, I have never seen a collective-rightist willing to tackle the question the way that my quantum mechanics professor tackled the tennis ball puzzle. If it has been tackled, I'd rather expect it to be findable by chasing citations from Parker v. District of Columbia. However, I rather doubt that in fact it is findable there. Certainly no one on the collective side pointed it out in the many pages of post-ruling web debate that I read.

I'm not trying to prove either conclusion here, that natural selection is a good theory or that handgun bans are bad law. But I am claiming that the particular arguments that I am picking on seem like pretty bad arguments.

I'm not trying to hold intellectuals to impossibly high standards in defending their complicated ideas against simplistic critiques. For what it's worth, I think decent examples of specialists responding with specialist detail to simple layman's critiques can be found close to the controversies from which I took my examples: responses to "how can you possibly reconcile evolution by natural selection with the second law of thermodynamics?" and "jurisdiction A has lots of legal guns and lots of murder, jurisdiction B has little of either, so how can you possibly deny the fact that legal guns are a danger to public safety?"

I also vaguely plan to write my own essay "What is Entropy Like?" which will be, in some part, a response to the thermodynamic critique of evolution. Entropy arises naturally in the statistical mechanics which was used in the work I used to do, so I'm pretty comfortable with the concept, and I think a lot can usefully be said about it without "hard" math like logarithms, and without truly-hard math like taking partial derivatives or integrating over phase space, or physics like counting the number of quantum states. I am thinking of explaining and illustrating how entropy is the complement of an order which behaves much like commonsense intuitions about wealth. (More precisely, wealth over any ordinary very short period, like 10 minutes, where wages and compound interest and such are negligible.) It comes in various forms like cash and deeds to land, and of course you can shift it around or destroy it, but don't expect to increase it. Also, use a suitably broad but not-too-broad definition or expect to get nonsense.)

Just because one's ideas are ever so very, very intellectual doesn't mean the the ideas should be held above simple reality checks by nonspecialists. John W. Gardner said something of the sort more memorably: "The society which scorns excellence in plumbing because plumbing is a humble activity, and tolerates shoddiness in philosophy because philosophy is an exalted activity, will have neither good plumbing nor good philosophy. Neither its pipes nor its theories will hold water."

Too Many Econ Blogs I: Friedman's Factoids

2006-12-26T13:14:00.000-08:00

Reading too many economics blogs may be bad for one's blood pressure, and I predict I will be driven to vent at least once more after this.

There is a class of factoid that is so plausible that invoking it tends to force one's opposition to concede immediately, without trying to squabble about nuances like how it was collected. From the left, e.g., wait for an argument about how much of women's historical difficulty in becoming tenured university professors is because of their inferior job performance, perhaps due to their tendency to work fewer hours and have more career interruptions and such. Then bring up one of those nasty little studies about how people rate the exact same paper higher when it has a man's name on top than when it has a woman's name on the top. Lo, one's opponents tend to concede the relevance of pure discrimination. At some level they basically knew it already, even if that knowledge hadn't stopped them from closed-mindedly pegging you as a closed-minded fanatic before.

From the libertarian direction, studies by Milton Friedman and others on how prices increased as doctors were given the power to limit the number of competing doctors seem to have something of this invincible factoid character. Once presented, even people who previously sincerely believed that the regulations didn't raise prices scurry to some other point to defend. It seems to be hard to summon one's courage to search for data to defend one's belief that "no, it didn't raise prices when we gave the doctors broad legal power to choke off their competition." (For another Friedman-on-the-AMA factoid, see the block quote in the last paragraph of this post.)

This is a somewhat understandable game when played in casual conversations of nonexperts, but it becomes more than a little irritating when experts are involved. If an expert on racial or sexual discrimination says that university grading is a system of pure merit, then even if he backed down immediately when reminded of studies on how changing the name on top of a paper can have a significant effect, one could still tend to go away mad that he had said it in the first place, even if he said it carefully in measured tones without resorting to sarcasm and without previous maneuvers that suggested perhaps he was well aware of the studies and was just hoping that no one else knew.

I read several blogs by ideologically sympathetic academics and journalists. In an effort not to lose my perspective completely, I also read DeLong and Thoma. So presently I stumbled upon two rather different treatments of the same question of how free the health care market is.

Commenting on the complaint of John Derbyshire and the followup of Kevin Drum remarking how this shows the importance of his preferred policies, "Jane Galt" wrote

Kevin Drum offers John Derbyshire's experience as an argument for national health care. [...] I gather that Mr Derbyshire lives in New York State. If so, I can cheerfully attest that the problem is not "the market", but "the government", whose thicket of minimum coverage, community pooling, and like regulations has made New York State's health insurance the most expensive in the nation.

while Brad DeLong wrote

John Derbyshire encounters the free market in health care, and recoils in horror. [...] Now we liberals have lots of reasons and arguments for why we would not expect free markets in health insurance to work very well, and Derbyshire has just encountered one of them: it looks as if his particular health plan is entering an adverse-selection death spiral.

The juxtaposition between "Galt's" take and DeLong's take was bad for my blood pressure. "The free market in health care," hmm? Brad DeLong is a professor of economics at a prestigious university and spent a year or so working in government on health care policy. He is not excused from knowing about the heavy-handed government interventions in the insurance layer of the market in health care described by Galt, and he is not excused from knowing about the strict barriers to entry in the provision of health care services which are familiar to damned near everyone. (It is not so easy to find people who think it would be legal to just hang up a shingle and open a little medical business, perhaps starting small with something like vaccinations at a convenient location. Especially for anything involving prescription drugs, restrictive licensure is a rather in-your-face system, and "gosh, doctors get to charge a lot of money [I wish I could be one]" is not such a complicated thought that only an economics rocket scientist can think it.) I wouldn't mind someone saying "the market" about health care, and I am not such a free market fanatic that I would mind someone saying "free market" about most aspects of the computer hardware or software market, but for the US health care market, as for the old USSR election system, using "free" as a qualifier is Orwellian given that the choices are so strongly restricted by the government.

However, it also gets a little worse, DeLong happens to be the author of the Slate obituary for Friedman. I well remembered this particularly because of the way DeLong wrote

He [Friedman] scorned government licensing of professionals -- especially doctors, who heard over and over again about how their incomes were boosted by restrictions on the number of doctors that made Americans sicker.

He didn't mention how Friedman's scorn was backed up by his empirical work (including his doctoral thesis, also published as a book Income From Independent Professional Practice coauthored with his advisor Kuznets). The question of how much regulation there should be is highly politically charged, and it's common for people to scorn some policy choice or other, all too often with inadequate justification. A more noteworthy fact here is that Friedman won professional respect among economists for his statistical work both in general and in this area in particular. I had a nasty suspicion that DeLong intended that his readers would combine his formulation of Friedman's position with their their confident belief that heavy government regulation is vitally important, and reach the logical conclusion that Friedman was a blindered ideological extremist. Admittedly that intent is not so obvious, and perhaps I should even be ashamed of my nasty suspicion that DeLong had a political ax to grind and planted it in the back of the dead. However, it is at least very obvious that in November, DeLong had no fear that puzzled readers would think "restrictions on the number of doctors? what restrictions? in the free market in health care!?" You get extra demerits in my book for referring to the "free elections" in Xamplestan when just the previous month you had written

He [Fredmannov] scorned government selection of electoral candidates --- especially by the Party, and Party members heard over and over again that their power was boosted by restrictions on nonmembers entering government elections, making government worse.

Possibly Fredmannov could be being dogmatic when he makes these claims of corrupt interest of Party members and reduction in quality of government. Possibly Fredmannov's statistical work that you declined to mention could be misleading or incompetent. But whether or not their unfreeness benefits the general population of Xamplestanians, those are definitely not free elections.

Caveat: Actually, I have never looked at the Friedman/Kuznets book, and my local university library is closed for the Christmas holidays, so take my claims about the contents of Friedman's thesis work with a grain of salt. But it's clear from what others have written that a lot of it was empirical work on the effects of licensing. E.g., the NY Times obituary mentions "one finding of the book was that the AMA exerted monopolistic pressure on the income of doctors," and that the AMA "forced the publisher to delay publication" though "the book was eventually published, unchanged." And before the library closed for the holidays, I was able to check out Capitalism and Freedom. (I had asked the library to recall it for me shortly after Friedman's death, because some tributes had made me curious how Friedman's evaluations of and predictions about oft-cited free-economy/mixed-economy examples like Hong Kong and Sweden might have changed over time.) I still have it checked out, and peering past various post-publication scrawled annotations (such as "This title is an oxymoron", "Apologist Reactionary", "Utopia", "Fascist!", "Perhaps hell! What about Germany, or don't you know what happened?" and "I'm surrounded by unthinking hulks") I can see that Friedman wrote a chapter on occupational licensure. The chapter is full of empirical stuff which seems impossible to reconcile with "free market" even when embellished with Berkeley rocket scientific concepts like the death spiral. E.g.,

A dramatic piece of evidence on the power and potency of the [American Medical] Association as well as on the lack of relation to quality is proved by one figure that I have always foundstriking. After 1933, when Hitler came to power in Germany, there was a tremendous outflow of professional people from Germany, Austria, and so on, including of course, physicians who wanted to practice in the United States. The number of physicians trained abroad who were admitted to practice in the United States after 1933 was the same as in the five years before. This was clearly not the result of the natural course of events. The threat of these additional physicians led to a stringent tightening of requirements for foreign physicians which imposed extreme costs on them.

Capitalism and Freedom doesn't have many footnotes, and there is no footnote for that 1933-effect figure, but it is my impression (from something read long ago) that that figure is the primary result of a research paper in economics that Friedman himself published back in the day.

Reasoning From Authority

2006-11-09T15:53:00.000-08:00

It is probably dangerous to disagree too strongly with Jane Galt, Julian Sanchez, and The Who. But note also Aesop's pro-gridlock take on unexpected fallout from regime change! I have tried to set my bar low: I hope I shall be able to remain pleased that Santorum has been removed. Of course, even with that small goal, I could have been rather unhappy about Santorum for SecDef. Fortunately, I didn't hear about that until it was safely over. And anyway I don't think that was the kind of perverse consequences that the wise ones are warning about.